禾息
e8a26827ae
Refactor CUDA stream context management in CosyVoice2Model
...
- Replaced the use of torch.cuda.stream with a direct context management approach for improved clarity and performance during inference.
- This change simplifies the stream handling code while maintaining efficient resource utilization.
2025-04-16 16:04:40 +08:00
禾息
7f4c9a2c64
Refactor CosyVoice inference methods to streamline CUDA stream management
...
- Removed the queue-based stream pool and integrated direct CUDA stream usage for improved performance.
- Simplified inference methods by eliminating unnecessary synchronization and stream management code.
- Enhanced logging for better tracking of synthesis operations and performance metrics.
- Updated the model class to support CUDA stream context management, ensuring efficient resource utilization during inference.
2025-04-16 14:15:14 +08:00
禾息
62e04e8856
Enhance CosyVoice with CUDA stream management and estimator handling
...
- Introduced a queue-based system for managing CUDA streams to improve inference performance.
- Updated inference methods to utilize CUDA streams for asynchronous processing.
- Added an EstimatorWrapper class to manage TensorRT estimators, allowing for efficient execution context handling.
- Modified model loading functions to support estimator count configuration.
- Improved logging and performance tracking during inference operations.
2025-04-16 11:16:28 +08:00
qihua
90b666ea20
初步合并vllm支持，异步推理的通道处理还存在bug
2025-03-07 20:26:19 +08:00
lyuxiang.lx
07e477519b
add llm bistream
2025-01-23 10:12:06 +08:00
lyuxiang.lx
0b75c3a03f
update
2025-01-14 22:55:13 +08:00
lyuxiang.lx
b95f18909e
add empty cache
2025-01-10 14:14:32 +08:00
lyuxiang.lx
1cfc5dd077
add online trt export
2025-01-10 13:55:05 +08:00
huzetao.hzt
b6a1116d15
support online onnx to trt conversion
2025-01-07 17:20:06 +08:00
lyuxiang.lx
77d8cf13a3
update
2025-01-02 10:55:59 +08:00
lyuxiang.lx
b9ddcba5fd
add some instruction and assert
2024-12-30 16:41:57 +08:00
Xiang Lyu
0d1e562f1d
Merge pull request #736 from FunAudioLLM/dev/lyuxiang.lx
...
fix cosyvoice1.0 bug
2024-12-17 16:06:41 +08:00
lyuxiang.lx
b00d8a073c
fix cosyvoice1.0 bug
2024-12-17 16:05:37 +08:00
Xiang Lyu
c4c8050532
Merge pull request #717 from FunAudioLLM/dev/lyuxiang.lx
...
fix lint
2024-12-16 10:38:31 +08:00
lyuxiang.lx
3581caec76
fix lint
2024-12-16 10:37:10 +08:00
Xiang Lyu
94d6ce1006
Merge pull request #715 from FunAudioLLM/dev/lyuxiang.lx
...
Dev/lyuxiang.lx
2024-12-16 09:55:27 +08:00
lyuxiang.lx
ac70560364
fix lint
2024-12-16 09:54:24 +08:00
lyuxiang.lx
2511a49a72
update
2024-12-12 18:48:25 +08:00
lyuxiang.lx
c693039d14
update
2024-12-12 16:46:28 +08:00
lyuxiang.lx
3e381002d7
add cosyvoice2
2024-12-11 16:14:19 +08:00
boostarea
3411e1f599
feat.release flow_cache_dict to prevent potential memory leaks in long-running processes.
2024-10-28 12:02:07 +08:00
Tao Liu
18b9a8c844
Update model.py
...
The torch.tensor() function does not have a dim parameter
2024-10-17 11:53:09 +08:00
lyuxiang.lx
a4db3db8ed
update flow cache
2024-10-16 15:24:47 +08:00
Xiang Lyu
ace734def8
Merge pull request #455 from boji123/bj_dev_stream_fix_promptcache
...
[debug] support flow cache, for sharper tts_mel output (handle prompt bug)
2024-10-16 14:12:40 +08:00
lyuxiang.lx
6b7286eb62
fix typo
2024-10-16 13:57:27 +08:00
lyuxiang.lx
7e6d60c24c
fix bug
2024-10-16 13:30:13 +08:00
lyuxiang.lx
789ee9e5e7
add hifigan train
2024-10-16 11:37:32 +08:00
boji123
c9acce1482
[debug] support flow cache, for sharper tts_mel output
2024-09-29 16:26:11 +08:00
lyuxiang.lx
ffa28e3bbd
update token args
2024-09-29 10:35:10 +08:00
lyuxiang.lx
ba3d9693da
load jit to device
2024-09-26 14:55:03 +08:00
lyuxiang.lx
06934c38c7
update vc code
2024-09-26 14:46:24 +08:00
lyuxiang.lx
72b89a52fb
update vc/tts code
2024-09-26 11:53:10 +08:00
lyuxiang.lx
f65eca6723
add speech fade in out
2024-09-19 18:02:42 +08:00
Xiang Lyu
cd26f11859
Merge pull request #379 from boji123/bj_dev_stream_fix
...
[debug] fix badcase, add fade on speech output
2024-09-19 17:32:04 +08:00
lyuxiang.lx
e19e80fcd8
update tempo change
2024-09-18 16:13:40 +08:00
liubaiji
9e0b99e48e
[feature] fix badcase, add fade on speech output
2024-09-11 10:41:50 +08:00
lyuxiang.lx
122df8c420
set onnx to false as last chunk rtf unstable
2024-09-06 17:10:54 +08:00
lyuxiang.lx
90433f5373
fix lint
2024-09-05 16:15:34 +08:00
lyuxiang.lx
2ce724045b
add onnx export
2024-09-04 18:15:33 +08:00
禾息
752103a307
Merge remote-tracking branch 'origin/inference_streaming' into inference_streaming
2024-09-03 11:13:25 +08:00
禾息
a801416805
mirror modify
2024-09-03 11:07:47 +08:00
禾息
fadb22086f
export onnx
2024-09-03 11:06:24 +08:00
Xiang Lyu
ee988420f3
Merge branch 'inference_streaming' into flow_tensorrt
2024-08-30 14:20:06 +08:00
禾息
18599be8d5
mirror modify
2024-08-30 14:15:24 +08:00
zhoubofan.zbf
29408360fb
fix bug
2024-08-30 13:43:54 +08:00
禾息
6e7f5b922a
update
2024-08-30 13:14:44 +08:00
lyuxiang.lx
1ab3186799
revert trt TODO
2024-08-29 23:35:19 +08:00
zhoubofan.zbf
5f21aef786
add flow decoder tensorrt infer
2024-08-29 23:35:07 +08:00
lyuxiang.lx
1d881df8b2
fix vocoder speech overlap
2024-08-29 19:10:08 +08:00
lyuxiang.lx
f1e374a9bb
add trt script TODO
2024-08-29 10:44:04 +08:00
禾息