Text-to-Audio
Transformers
Safetensors
English
dasheng_audiogen
feature-extraction
audio-generation
text-to-speech
text-to-music
sound-effects
diffusion
custom_code
Instructions to use mispeech/Dasheng-AudioGen with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use mispeech/Dasheng-AudioGen with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-to-audio", model="mispeech/Dasheng-AudioGen", trust_remote_code=True)# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("mispeech/Dasheng-AudioGen", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
Jiahao mei commited on
Upload folder using huggingface_hub
Browse files- README.md +47 -27
- README_zh.md +60 -29
- modeling_dasheng_audiogen.py +18 -11
README.md
CHANGED
|
@@ -52,9 +52,30 @@ pip install torch torchaudio "transformers<5" einops
|
|
| 52 |
|
| 53 |
> Tested with Python 3.10, torch 2.8.0+cu128, transformers 4.57. Not compatible with transformers 5.x.
|
| 54 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 55 |
## Quick Start
|
| 56 |
|
| 57 |
-
###
|
|
|
|
|
|
|
| 58 |
|
| 59 |
```python
|
| 60 |
import torchaudio
|
|
@@ -62,15 +83,6 @@ from transformers import AutoModel
|
|
| 62 |
|
| 63 |
model = AutoModel.from_pretrained("mispeech/Dasheng-AudioGen", trust_remote_code=True).cuda()
|
| 64 |
|
| 65 |
-
audio = model.generate("A dog barking in a park")
|
| 66 |
-
torchaudio.save("output.wav", audio.cpu(), 16000)
|
| 67 |
-
```
|
| 68 |
-
|
| 69 |
-
### Aspect-wise Prompt
|
| 70 |
-
|
| 71 |
-
Use `compose_prompt` to describe different audio aspects separately:
|
| 72 |
-
|
| 73 |
-
```python
|
| 74 |
prompt = model.compose_prompt(
|
| 75 |
caption="A gritty detective narrating over the sound of heavy rain and a melancholic solo jazz saxophone.",
|
| 76 |
speech="gritty deep male voice",
|
|
@@ -83,17 +95,31 @@ audio = model.generate(prompt)
|
|
| 83 |
torchaudio.save("output.wav", audio.cpu(), 16000)
|
| 84 |
```
|
| 85 |
|
| 86 |
-
|
|
|
|
|
|
|
| 87 |
|
| 88 |
```python
|
| 89 |
-
|
| 90 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 91 |
)
|
|
|
|
|
|
|
| 92 |
```
|
| 93 |
|
| 94 |
### Batch Inference
|
| 95 |
|
| 96 |
```python
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 97 |
prompts = [
|
| 98 |
model.compose_prompt(caption="A cat meowing softly.", sfx="Soft cat meow."),
|
| 99 |
model.compose_prompt(caption="Thunder rolling in the distance.", env="Stormy night ambience."),
|
|
@@ -108,27 +134,21 @@ for i, audio in enumerate(audios):
|
|
| 108 |
### Generation Parameters
|
| 109 |
|
| 110 |
```python
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 111 |
audio = model.generate(
|
| 112 |
-
prompts=
|
| 113 |
num_steps=25, # number of denoising steps (default: 25)
|
| 114 |
guidance_scale=5.0, # classifier-free guidance scale (default: 5.0)
|
| 115 |
sway_sampling_coef=-1.0, # sway sampling coefficient (default: -1.0, 0 for linear)
|
| 116 |
)
|
|
|
|
| 117 |
```
|
| 118 |
|
| 119 |
-
## Prompt Format
|
| 120 |
-
|
| 121 |
-
Dasheng-AudioGen uses structured tags to describe different audio aspects:
|
| 122 |
-
|
| 123 |
-
| Tag | Description |
|
| 124 |
-
|-----|-------------|
|
| 125 |
-
| `<\|caption\|>` | Overall audio scene description |
|
| 126 |
-
| `<\|speech\|>` | Speaker identity and speaking style |
|
| 127 |
-
| `<\|asr\|>` | Spoken transcript / dialogue |
|
| 128 |
-
| `<\|sfx\|>` | Sound effects |
|
| 129 |
-
| `<\|music\|>` | Background music |
|
| 130 |
-
| `<\|env\|>` | Environmental ambience |
|
| 131 |
-
|
| 132 |
## Acknowledgments
|
| 133 |
|
| 134 |
Dasheng-AudioGen was developed with contributions from **XIAOMI LLM PLUS** and **SJTU X-LANCE**.
|
|
|
|
| 52 |
|
| 53 |
> Tested with Python 3.10, torch 2.8.0+cu128, transformers 4.57. Not compatible with transformers 5.x.
|
| 54 |
|
| 55 |
+
## Prompt Format
|
| 56 |
+
|
| 57 |
+
Dasheng-AudioGen uses structured tags to describe different audio aspects. A valid prompt **must start with the `<|caption|>` tag**, which provides the overall scene description. Other tags are optional and can be included as needed.
|
| 58 |
+
|
| 59 |
+
| Tag | Description | Required |
|
| 60 |
+
|-----|-------------|:--------:|
|
| 61 |
+
| `<\|caption\|>` | Overall audio scene description | Yes |
|
| 62 |
+
| `<\|speech\|>` | Speaker identity and speaking style | No |
|
| 63 |
+
| `<\|asr\|>` | Spoken transcript / dialogue | No |
|
| 64 |
+
| `<\|sfx\|>` | Sound effects | No |
|
| 65 |
+
| `<\|music\|>` | Background music | No |
|
| 66 |
+
| `<\|env\|>` | Environmental ambience | No |
|
| 67 |
+
|
| 68 |
+
**Rules:**
|
| 69 |
+
- The prompt must begin with `<|caption|>` — prompts without it will be rejected.
|
| 70 |
+
- Only include tags that are relevant; omit tags with no content (e.g., skip `<|music|>` if there is no music).
|
| 71 |
+
|
| 72 |
+
> **Multilingual note:** When using the multilingual model, all descriptive tags (`caption`, `speech`, `sfx`, `music`, `env`) should be in **English**. Only the `<|asr|>` field (the actual speech content to synthesize) uses the target language.
|
| 73 |
+
|
| 74 |
## Quick Start
|
| 75 |
|
| 76 |
+
### Usage 1: Aspect-wise Composition
|
| 77 |
+
|
| 78 |
+
Pass each aspect as a named argument. The `caption` field is required; all other fields are optional.
|
| 79 |
|
| 80 |
```python
|
| 81 |
import torchaudio
|
|
|
|
| 83 |
|
| 84 |
model = AutoModel.from_pretrained("mispeech/Dasheng-AudioGen", trust_remote_code=True).cuda()
|
| 85 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 86 |
prompt = model.compose_prompt(
|
| 87 |
caption="A gritty detective narrating over the sound of heavy rain and a melancholic solo jazz saxophone.",
|
| 88 |
speech="gritty deep male voice",
|
|
|
|
| 95 |
torchaudio.save("output.wav", audio.cpu(), 16000)
|
| 96 |
```
|
| 97 |
|
| 98 |
+
### Usage 2: Pre-formatted Prompt String
|
| 99 |
+
|
| 100 |
+
Pass a complete tagged string via the `prompt` parameter. The string must start with `<|caption|>`.
|
| 101 |
|
| 102 |
```python
|
| 103 |
+
import torchaudio
|
| 104 |
+
from transformers import AutoModel
|
| 105 |
+
|
| 106 |
+
model = AutoModel.from_pretrained("mispeech/Dasheng-AudioGen", trust_remote_code=True).cuda()
|
| 107 |
+
|
| 108 |
+
prompt = model.compose_prompt(
|
| 109 |
+
prompt="<|caption|> A gritty detective narrating over the sound of heavy rain and a melancholic solo jazz saxophone. <|speech|> gritty deep male voice <|asr|> The city never sleeps, but it sure knows how to cry. <|sfx|> heavy rain hitting pavement <|music|> melancholic solo saxophone <|env|> distant urban ambience"
|
| 110 |
)
|
| 111 |
+
audio = model.generate(prompt)
|
| 112 |
+
torchaudio.save("output.wav", audio.cpu(), 16000)
|
| 113 |
```
|
| 114 |
|
| 115 |
### Batch Inference
|
| 116 |
|
| 117 |
```python
|
| 118 |
+
import torchaudio
|
| 119 |
+
from transformers import AutoModel
|
| 120 |
+
|
| 121 |
+
model = AutoModel.from_pretrained("mispeech/Dasheng-AudioGen", trust_remote_code=True).cuda()
|
| 122 |
+
|
| 123 |
prompts = [
|
| 124 |
model.compose_prompt(caption="A cat meowing softly.", sfx="Soft cat meow."),
|
| 125 |
model.compose_prompt(caption="Thunder rolling in the distance.", env="Stormy night ambience."),
|
|
|
|
| 134 |
### Generation Parameters
|
| 135 |
|
| 136 |
```python
|
| 137 |
+
import torchaudio
|
| 138 |
+
from transformers import AutoModel
|
| 139 |
+
|
| 140 |
+
model = AutoModel.from_pretrained("mispeech/Dasheng-AudioGen", trust_remote_code=True).cuda()
|
| 141 |
+
|
| 142 |
+
prompt = model.compose_prompt(caption="A dog barking in a park")
|
| 143 |
audio = model.generate(
|
| 144 |
+
prompts=prompt,
|
| 145 |
num_steps=25, # number of denoising steps (default: 25)
|
| 146 |
guidance_scale=5.0, # classifier-free guidance scale (default: 5.0)
|
| 147 |
sway_sampling_coef=-1.0, # sway sampling coefficient (default: -1.0, 0 for linear)
|
| 148 |
)
|
| 149 |
+
torchaudio.save("output.wav", audio.cpu(), 16000)
|
| 150 |
```
|
| 151 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 152 |
## Acknowledgments
|
| 153 |
|
| 154 |
Dasheng-AudioGen was developed with contributions from **XIAOMI LLM PLUS** and **SJTU X-LANCE**.
|
README_zh.md
CHANGED
|
@@ -4,11 +4,24 @@
|
|
| 4 |
[](https://huggingface.co/mispeech/Dasheng-AudioGen)
|
| 5 |
[](https://huggingface.co/spaces/mispeech/Dasheng-AudioGen)
|
| 6 |
[](https://nieeim.github.io/Dasheng-AudioGen-Web/)
|
|
|
|
| 7 |
|
| 8 |
[**English**](./README.md) | [**中文**](./README_zh.md)
|
| 9 |
|
| 10 |
**Dasheng-AudioGen** 是一个统一的音频生成模型,能够根据文本描述同时合成**语音、音乐、音效和环境声**。
|
| 11 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 12 |
## 模型
|
| 13 |
|
| 14 |
| 模型 | HuggingFace | 文本编码器 | 语言支持 |
|
|
@@ -24,9 +37,30 @@ pip install torch torchaudio "transformers<5" einops
|
|
| 24 |
|
| 25 |
> 已在 Python 3.10、torch 2.8.0+cu128、transformers 4.57 上测试通过。已知不兼容 transformers 5.x。
|
| 26 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 27 |
## 快速开始
|
| 28 |
|
| 29 |
-
###
|
|
|
|
|
|
|
| 30 |
|
| 31 |
```python
|
| 32 |
import torchaudio
|
|
@@ -34,15 +68,6 @@ from transformers import AutoModel
|
|
| 34 |
|
| 35 |
model = AutoModel.from_pretrained("mispeech/Dasheng-AudioGen", trust_remote_code=True).cuda()
|
| 36 |
|
| 37 |
-
audio = model.generate("A dog barking in a park")
|
| 38 |
-
torchaudio.save("output.wav", audio.cpu(), 16000)
|
| 39 |
-
```
|
| 40 |
-
|
| 41 |
-
### 分项 Prompt
|
| 42 |
-
|
| 43 |
-
使用 `compose_prompt` 分别描述不同的音频维度:
|
| 44 |
-
|
| 45 |
-
```python
|
| 46 |
prompt = model.compose_prompt(
|
| 47 |
caption="A gritty detective narrating over the sound of heavy rain and a melancholic solo jazz saxophone.",
|
| 48 |
speech="gritty deep male voice",
|
|
@@ -55,17 +80,31 @@ audio = model.generate(prompt)
|
|
| 55 |
torchaudio.save("output.wav", audio.cpu(), 16000)
|
| 56 |
```
|
| 57 |
|
| 58 |
-
|
|
|
|
|
|
|
| 59 |
|
| 60 |
```python
|
| 61 |
-
|
| 62 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 63 |
)
|
|
|
|
|
|
|
| 64 |
```
|
| 65 |
|
| 66 |
### 批量推理
|
| 67 |
|
| 68 |
```python
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 69 |
prompts = [
|
| 70 |
model.compose_prompt(caption="A cat meowing softly.", sfx="Soft cat meow."),
|
| 71 |
model.compose_prompt(caption="Thunder rolling in the distance.", env="Stormy night ambience."),
|
|
@@ -80,29 +119,21 @@ for i, audio in enumerate(audios):
|
|
| 80 |
### 生成参数
|
| 81 |
|
| 82 |
```python
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 83 |
audio = model.generate(
|
| 84 |
-
prompts=
|
| 85 |
num_steps=25, # 去噪步数(默认:25)
|
| 86 |
guidance_scale=5.0, # 无分类器引导强度(默认:5.0)
|
| 87 |
sway_sampling_coef=-1.0, # sway 采样系数(默认:-1.0,设为 0 使用线性调度)
|
| 88 |
)
|
|
|
|
| 89 |
```
|
| 90 |
|
| 91 |
-
## Prompt 格式
|
| 92 |
-
|
| 93 |
-
Dasheng-AudioGen 使用结构化标签来描述不同的音频维度:
|
| 94 |
-
|
| 95 |
-
| 标签 | 描述 |
|
| 96 |
-
|------|------|
|
| 97 |
-
| `<\|caption\|>` | 整体音频场景描述 |
|
| 98 |
-
| `<\|speech\|>` | 说话人身份和说话风格 |
|
| 99 |
-
| `<\|asr\|>` | 语音转写内容 / 对话文本 |
|
| 100 |
-
| `<\|sfx\|>` | 音效 |
|
| 101 |
-
| `<\|music\|>` | 背景音乐 |
|
| 102 |
-
| `<\|env\|>` | 环境音 |
|
| 103 |
-
|
| 104 |
-
> **多语言 prompt 规范:** 使用多语言模型时,所有描述性标签(`caption`、`speech`、`sfx`、`music`、`env`)应使用**英文**填写,仅 `<|asr|>` 字段(实际要合成的语音内容)使用目标语言。
|
| 105 |
-
|
| 106 |
## 致谢
|
| 107 |
|
| 108 |
Dasheng-AudioGen 由**小米 LLM PLUS** 和 **上海交通大学 X-LANCE** 联合开发。
|
|
|
|
| 4 |
[](https://huggingface.co/mispeech/Dasheng-AudioGen)
|
| 5 |
[](https://huggingface.co/spaces/mispeech/Dasheng-AudioGen)
|
| 6 |
[](https://nieeim.github.io/Dasheng-AudioGen-Web/)
|
| 7 |
+
[](https://colab.research.google.com/#fileId=https://huggingface.co/mispeech/Dasheng-AudioGen/resolve/main/notebook.ipynb)
|
| 8 |
|
| 9 |
[**English**](./README.md) | [**中文**](./README_zh.md)
|
| 10 |
|
| 11 |
**Dasheng-AudioGen** 是一个统一的音频生成模型,能够根据文本描述同时合成**语音、音乐、音效和环境声**。
|
| 12 |
|
| 13 |
+
<p align="center">
|
| 14 |
+
<video
|
| 15 |
+
src="https://github.com/user-attachments/assets/497f5688-8731-4830-8ee7-b9cf4234d900"
|
| 16 |
+
controls
|
| 17 |
+
autoplay
|
| 18 |
+
muted
|
| 19 |
+
loop
|
| 20 |
+
playsinline
|
| 21 |
+
width="85%">
|
| 22 |
+
</video>
|
| 23 |
+
</p>
|
| 24 |
+
|
| 25 |
## 模型
|
| 26 |
|
| 27 |
| 模型 | HuggingFace | 文本编码器 | 语言支持 |
|
|
|
|
| 37 |
|
| 38 |
> 已在 Python 3.10、torch 2.8.0+cu128、transformers 4.57 上测试通过。已知不兼容 transformers 5.x。
|
| 39 |
|
| 40 |
+
## Prompt 格式
|
| 41 |
+
|
| 42 |
+
Dasheng-AudioGen 使用结构化标签来描述不同的音频维度。有效的 prompt **必须以 `<|caption|>` 标签开头**,用于描述整体音频场景。其他标签为可选项,按需使用。
|
| 43 |
+
|
| 44 |
+
| 标签 | 描述 | 是否必需 |
|
| 45 |
+
|------|------|:--------:|
|
| 46 |
+
| `<\|caption\|>` | 整体音频场景描述 | 是 |
|
| 47 |
+
| `<\|speech\|>` | 说话人身份和说话风格 | 否 |
|
| 48 |
+
| `<\|asr\|>` | 语音转写内容 / 对话文本 | 否 |
|
| 49 |
+
| `<\|sfx\|>` | 音效 | 否 |
|
| 50 |
+
| `<\|music\|>` | 背景音乐 | 否 |
|
| 51 |
+
| `<\|env\|>` | 环境音 | 否 |
|
| 52 |
+
|
| 53 |
+
**规则:**
|
| 54 |
+
- Prompt 必须以 `<|caption|>` 开头,否则会报错。
|
| 55 |
+
- 仅包含有实际内容的标签;没有对应内容的标签请省略(例如没有音乐则不传 `<|music|>`)。
|
| 56 |
+
|
| 57 |
+
> **多语言 prompt 规范:** 使用多语言模型时,所有描述性标签(`caption`、`speech`、`sfx`、`music`、`env`)应使用**英文**填写,仅 `<|asr|>` 字段(实际要合成的语音内容)使用目标语言。
|
| 58 |
+
|
| 59 |
## 快速开始
|
| 60 |
|
| 61 |
+
### 用法一:分维度组装
|
| 62 |
+
|
| 63 |
+
通过命名参数分别传入各个维度的描述。`caption` 为必填项,其他字段可选。
|
| 64 |
|
| 65 |
```python
|
| 66 |
import torchaudio
|
|
|
|
| 68 |
|
| 69 |
model = AutoModel.from_pretrained("mispeech/Dasheng-AudioGen", trust_remote_code=True).cuda()
|
| 70 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 71 |
prompt = model.compose_prompt(
|
| 72 |
caption="A gritty detective narrating over the sound of heavy rain and a melancholic solo jazz saxophone.",
|
| 73 |
speech="gritty deep male voice",
|
|
|
|
| 80 |
torchaudio.save("output.wav", audio.cpu(), 16000)
|
| 81 |
```
|
| 82 |
|
| 83 |
+
### 用法二:传入完整 Prompt 字符串
|
| 84 |
+
|
| 85 |
+
通过 `prompt` 参数传入预格式化的标签字符串,该字符串必须以 `<|caption|>` 开头。
|
| 86 |
|
| 87 |
```python
|
| 88 |
+
import torchaudio
|
| 89 |
+
from transformers import AutoModel
|
| 90 |
+
|
| 91 |
+
model = AutoModel.from_pretrained("mispeech/Dasheng-AudioGen", trust_remote_code=True).cuda()
|
| 92 |
+
|
| 93 |
+
prompt = model.compose_prompt(
|
| 94 |
+
prompt="<|caption|> A gritty detective narrating over the sound of heavy rain and a melancholic solo jazz saxophone. <|speech|> gritty deep male voice <|asr|> The city never sleeps, but it sure knows how to cry. <|sfx|> heavy rain hitting pavement <|music|> melancholic solo saxophone <|env|> distant urban ambience"
|
| 95 |
)
|
| 96 |
+
audio = model.generate(prompt)
|
| 97 |
+
torchaudio.save("output.wav", audio.cpu(), 16000)
|
| 98 |
```
|
| 99 |
|
| 100 |
### 批量推理
|
| 101 |
|
| 102 |
```python
|
| 103 |
+
import torchaudio
|
| 104 |
+
from transformers import AutoModel
|
| 105 |
+
|
| 106 |
+
model = AutoModel.from_pretrained("mispeech/Dasheng-AudioGen", trust_remote_code=True).cuda()
|
| 107 |
+
|
| 108 |
prompts = [
|
| 109 |
model.compose_prompt(caption="A cat meowing softly.", sfx="Soft cat meow."),
|
| 110 |
model.compose_prompt(caption="Thunder rolling in the distance.", env="Stormy night ambience."),
|
|
|
|
| 119 |
### 生成参数
|
| 120 |
|
| 121 |
```python
|
| 122 |
+
import torchaudio
|
| 123 |
+
from transformers import AutoModel
|
| 124 |
+
|
| 125 |
+
model = AutoModel.from_pretrained("mispeech/Dasheng-AudioGen", trust_remote_code=True).cuda()
|
| 126 |
+
|
| 127 |
+
prompt = model.compose_prompt(caption="A dog barking in a park")
|
| 128 |
audio = model.generate(
|
| 129 |
+
prompts=prompt,
|
| 130 |
num_steps=25, # 去噪步数(默认:25)
|
| 131 |
guidance_scale=5.0, # 无分类器引导强度(默认:5.0)
|
| 132 |
sway_sampling_coef=-1.0, # sway 采样系数(默认:-1.0,设为 0 使用线性调度)
|
| 133 |
)
|
| 134 |
+
torchaudio.save("output.wav", audio.cpu(), 16000)
|
| 135 |
```
|
| 136 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 137 |
## 致谢
|
| 138 |
|
| 139 |
Dasheng-AudioGen 由**小米 LLM PLUS** 和 **上海交通大学 X-LANCE** 联合开发。
|
modeling_dasheng_audiogen.py
CHANGED
|
@@ -30,7 +30,7 @@ TAG_ORDER = OrderedDict([
|
|
| 30 |
|
| 31 |
|
| 32 |
def compose_prompt(
|
| 33 |
-
|
| 34 |
caption: str | None = None,
|
| 35 |
speech: str | None = None,
|
| 36 |
asr: str | None = None,
|
|
@@ -38,10 +38,21 @@ def compose_prompt(
|
|
| 38 |
music: str | None = None,
|
| 39 |
env: str | None = None,
|
| 40 |
) -> str:
|
| 41 |
-
if
|
| 42 |
-
|
| 43 |
-
if
|
| 44 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 45 |
|
| 46 |
values = {
|
| 47 |
"caption": caption, "speech": speech, "asr": asr,
|
|
@@ -54,10 +65,6 @@ def compose_prompt(
|
|
| 54 |
value = str(value).strip()
|
| 55 |
if value:
|
| 56 |
chunks.append(f"{tag} {value}")
|
| 57 |
-
if not chunks:
|
| 58 |
-
raise ValueError(
|
| 59 |
-
"No prompt content provided. Pass `content` or at least one aspect field."
|
| 60 |
-
)
|
| 61 |
return " ".join(chunks)
|
| 62 |
|
| 63 |
|
|
@@ -231,7 +238,7 @@ class DashengAudioGenModel(PreTrainedModel):
|
|
| 231 |
|
| 232 |
@staticmethod
|
| 233 |
def compose_prompt(
|
| 234 |
-
|
| 235 |
caption: str | None = None,
|
| 236 |
speech: str | None = None,
|
| 237 |
asr: str | None = None,
|
|
@@ -240,7 +247,7 @@ class DashengAudioGenModel(PreTrainedModel):
|
|
| 240 |
env: str | None = None,
|
| 241 |
) -> str:
|
| 242 |
return compose_prompt(
|
| 243 |
-
|
| 244 |
asr=asr, sfx=sfx, music=music, env=env,
|
| 245 |
)
|
| 246 |
|
|
|
|
| 30 |
|
| 31 |
|
| 32 |
def compose_prompt(
|
| 33 |
+
prompt: str | None = None,
|
| 34 |
caption: str | None = None,
|
| 35 |
speech: str | None = None,
|
| 36 |
asr: str | None = None,
|
|
|
|
| 38 |
music: str | None = None,
|
| 39 |
env: str | None = None,
|
| 40 |
) -> str:
|
| 41 |
+
if prompt is not None:
|
| 42 |
+
prompt = str(prompt).strip()
|
| 43 |
+
if not prompt:
|
| 44 |
+
raise ValueError("The `prompt` string is empty.")
|
| 45 |
+
if not prompt.startswith("<|caption|>"):
|
| 46 |
+
raise ValueError(
|
| 47 |
+
"The `prompt` string must start with the <|caption|> tag. "
|
| 48 |
+
f"Got: {prompt[:50]!r}..."
|
| 49 |
+
)
|
| 50 |
+
return prompt
|
| 51 |
+
|
| 52 |
+
if caption is None or not str(caption).strip():
|
| 53 |
+
raise ValueError(
|
| 54 |
+
"The `caption` field is required and cannot be empty."
|
| 55 |
+
)
|
| 56 |
|
| 57 |
values = {
|
| 58 |
"caption": caption, "speech": speech, "asr": asr,
|
|
|
|
| 65 |
value = str(value).strip()
|
| 66 |
if value:
|
| 67 |
chunks.append(f"{tag} {value}")
|
|
|
|
|
|
|
|
|
|
|
|
|
| 68 |
return " ".join(chunks)
|
| 69 |
|
| 70 |
|
|
|
|
| 238 |
|
| 239 |
@staticmethod
|
| 240 |
def compose_prompt(
|
| 241 |
+
prompt: str | None = None,
|
| 242 |
caption: str | None = None,
|
| 243 |
speech: str | None = None,
|
| 244 |
asr: str | None = None,
|
|
|
|
| 247 |
env: str | None = None,
|
| 248 |
) -> str:
|
| 249 |
return compose_prompt(
|
| 250 |
+
prompt=prompt, caption=caption, speech=speech,
|
| 251 |
asr=asr, sfx=sfx, music=music, env=env,
|
| 252 |
)
|
| 253 |
|