Jiahao mei commited on
Commit
60b4f97
·
verified ·
1 Parent(s): ba94c6e

Upload folder using huggingface_hub

Browse files
Files changed (3) hide show
  1. README.md +47 -27
  2. README_zh.md +60 -29
  3. modeling_dasheng_audiogen.py +18 -11
README.md CHANGED
@@ -52,9 +52,30 @@ pip install torch torchaudio "transformers<5" einops
52
 
53
  > Tested with Python 3.10, torch 2.8.0+cu128, transformers 4.57. Not compatible with transformers 5.x.
54
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
55
  ## Quick Start
56
 
57
- ### Basic Usage
 
 
58
 
59
  ```python
60
  import torchaudio
@@ -62,15 +83,6 @@ from transformers import AutoModel
62
 
63
  model = AutoModel.from_pretrained("mispeech/Dasheng-AudioGen", trust_remote_code=True).cuda()
64
 
65
- audio = model.generate("A dog barking in a park")
66
- torchaudio.save("output.wav", audio.cpu(), 16000)
67
- ```
68
-
69
- ### Aspect-wise Prompt
70
-
71
- Use `compose_prompt` to describe different audio aspects separately:
72
-
73
- ```python
74
  prompt = model.compose_prompt(
75
  caption="A gritty detective narrating over the sound of heavy rain and a melancholic solo jazz saxophone.",
76
  speech="gritty deep male voice",
@@ -83,17 +95,31 @@ audio = model.generate(prompt)
83
  torchaudio.save("output.wav", audio.cpu(), 16000)
84
  ```
85
 
86
- You can also pass a pre-formatted string with tags directly:
 
 
87
 
88
  ```python
89
- audio = model.generate(
90
- "<|caption|> A helicopter passing overhead. <|sfx|> Rhythmic helicopter blade sounds. <|env|> Open sky ambience."
 
 
 
 
 
91
  )
 
 
92
  ```
93
 
94
  ### Batch Inference
95
 
96
  ```python
 
 
 
 
 
97
  prompts = [
98
  model.compose_prompt(caption="A cat meowing softly.", sfx="Soft cat meow."),
99
  model.compose_prompt(caption="Thunder rolling in the distance.", env="Stormy night ambience."),
@@ -108,27 +134,21 @@ for i, audio in enumerate(audios):
108
  ### Generation Parameters
109
 
110
  ```python
 
 
 
 
 
 
111
  audio = model.generate(
112
- prompts="A dog barking in a park",
113
  num_steps=25, # number of denoising steps (default: 25)
114
  guidance_scale=5.0, # classifier-free guidance scale (default: 5.0)
115
  sway_sampling_coef=-1.0, # sway sampling coefficient (default: -1.0, 0 for linear)
116
  )
 
117
  ```
118
 
119
- ## Prompt Format
120
-
121
- Dasheng-AudioGen uses structured tags to describe different audio aspects:
122
-
123
- | Tag | Description |
124
- |-----|-------------|
125
- | `<\|caption\|>` | Overall audio scene description |
126
- | `<\|speech\|>` | Speaker identity and speaking style |
127
- | `<\|asr\|>` | Spoken transcript / dialogue |
128
- | `<\|sfx\|>` | Sound effects |
129
- | `<\|music\|>` | Background music |
130
- | `<\|env\|>` | Environmental ambience |
131
-
132
  ## Acknowledgments
133
 
134
  Dasheng-AudioGen was developed with contributions from **XIAOMI LLM PLUS** and **SJTU X-LANCE**.
 
52
 
53
  > Tested with Python 3.10, torch 2.8.0+cu128, transformers 4.57. Not compatible with transformers 5.x.
54
 
55
+ ## Prompt Format
56
+
57
+ Dasheng-AudioGen uses structured tags to describe different audio aspects. A valid prompt **must start with the `<|caption|>` tag**, which provides the overall scene description. Other tags are optional and can be included as needed.
58
+
59
+ | Tag | Description | Required |
60
+ |-----|-------------|:--------:|
61
+ | `<\|caption\|>` | Overall audio scene description | Yes |
62
+ | `<\|speech\|>` | Speaker identity and speaking style | No |
63
+ | `<\|asr\|>` | Spoken transcript / dialogue | No |
64
+ | `<\|sfx\|>` | Sound effects | No |
65
+ | `<\|music\|>` | Background music | No |
66
+ | `<\|env\|>` | Environmental ambience | No |
67
+
68
+ **Rules:**
69
+ - The prompt must begin with `<|caption|>` — prompts without it will be rejected.
70
+ - Only include tags that are relevant; omit tags with no content (e.g., skip `<|music|>` if there is no music).
71
+
72
+ > **Multilingual note:** When using the multilingual model, all descriptive tags (`caption`, `speech`, `sfx`, `music`, `env`) should be in **English**. Only the `<|asr|>` field (the actual speech content to synthesize) uses the target language.
73
+
74
  ## Quick Start
75
 
76
+ ### Usage 1: Aspect-wise Composition
77
+
78
+ Pass each aspect as a named argument. The `caption` field is required; all other fields are optional.
79
 
80
  ```python
81
  import torchaudio
 
83
 
84
  model = AutoModel.from_pretrained("mispeech/Dasheng-AudioGen", trust_remote_code=True).cuda()
85
 
 
 
 
 
 
 
 
 
 
86
  prompt = model.compose_prompt(
87
  caption="A gritty detective narrating over the sound of heavy rain and a melancholic solo jazz saxophone.",
88
  speech="gritty deep male voice",
 
95
  torchaudio.save("output.wav", audio.cpu(), 16000)
96
  ```
97
 
98
+ ### Usage 2: Pre-formatted Prompt String
99
+
100
+ Pass a complete tagged string via the `prompt` parameter. The string must start with `<|caption|>`.
101
 
102
  ```python
103
+ import torchaudio
104
+ from transformers import AutoModel
105
+
106
+ model = AutoModel.from_pretrained("mispeech/Dasheng-AudioGen", trust_remote_code=True).cuda()
107
+
108
+ prompt = model.compose_prompt(
109
+ prompt="<|caption|> A gritty detective narrating over the sound of heavy rain and a melancholic solo jazz saxophone. <|speech|> gritty deep male voice <|asr|> The city never sleeps, but it sure knows how to cry. <|sfx|> heavy rain hitting pavement <|music|> melancholic solo saxophone <|env|> distant urban ambience"
110
  )
111
+ audio = model.generate(prompt)
112
+ torchaudio.save("output.wav", audio.cpu(), 16000)
113
  ```
114
 
115
  ### Batch Inference
116
 
117
  ```python
118
+ import torchaudio
119
+ from transformers import AutoModel
120
+
121
+ model = AutoModel.from_pretrained("mispeech/Dasheng-AudioGen", trust_remote_code=True).cuda()
122
+
123
  prompts = [
124
  model.compose_prompt(caption="A cat meowing softly.", sfx="Soft cat meow."),
125
  model.compose_prompt(caption="Thunder rolling in the distance.", env="Stormy night ambience."),
 
134
  ### Generation Parameters
135
 
136
  ```python
137
+ import torchaudio
138
+ from transformers import AutoModel
139
+
140
+ model = AutoModel.from_pretrained("mispeech/Dasheng-AudioGen", trust_remote_code=True).cuda()
141
+
142
+ prompt = model.compose_prompt(caption="A dog barking in a park")
143
  audio = model.generate(
144
+ prompts=prompt,
145
  num_steps=25, # number of denoising steps (default: 25)
146
  guidance_scale=5.0, # classifier-free guidance scale (default: 5.0)
147
  sway_sampling_coef=-1.0, # sway sampling coefficient (default: -1.0, 0 for linear)
148
  )
149
+ torchaudio.save("output.wav", audio.cpu(), 16000)
150
  ```
151
 
 
 
 
 
 
 
 
 
 
 
 
 
 
152
  ## Acknowledgments
153
 
154
  Dasheng-AudioGen was developed with contributions from **XIAOMI LLM PLUS** and **SJTU X-LANCE**.
README_zh.md CHANGED
@@ -4,11 +4,24 @@
4
  [![Hugging Face Model](https://img.shields.io/badge/HuggingFace-Model-orange?logo=huggingface)](https://huggingface.co/mispeech/Dasheng-AudioGen)
5
  [![Hugging Face Demo](https://img.shields.io/badge/HuggingFace-Demo-orange?logo=huggingface)](https://huggingface.co/spaces/mispeech/Dasheng-AudioGen)
6
  [![Web Demo](https://img.shields.io/badge/Website-Demo-181717?logo=google-chrome)](https://nieeim.github.io/Dasheng-AudioGen-Web/)
 
7
 
8
  [**English**](./README.md) | [**中文**](./README_zh.md)
9
 
10
  **Dasheng-AudioGen** 是一个统一的音频生成模型,能够根据文本描述同时合成**语音、音乐、音效和环境声**。
11
 
 
 
 
 
 
 
 
 
 
 
 
 
12
  ## 模型
13
 
14
  | 模型 | HuggingFace | 文本编码器 | 语言支持 |
@@ -24,9 +37,30 @@ pip install torch torchaudio "transformers<5" einops
24
 
25
  > 已在 Python 3.10、torch 2.8.0+cu128、transformers 4.57 上测试通过。已知不兼容 transformers 5.x。
26
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
27
  ## 快速开始
28
 
29
- ### 基本用法
 
 
30
 
31
  ```python
32
  import torchaudio
@@ -34,15 +68,6 @@ from transformers import AutoModel
34
 
35
  model = AutoModel.from_pretrained("mispeech/Dasheng-AudioGen", trust_remote_code=True).cuda()
36
 
37
- audio = model.generate("A dog barking in a park")
38
- torchaudio.save("output.wav", audio.cpu(), 16000)
39
- ```
40
-
41
- ### 分项 Prompt
42
-
43
- 使用 `compose_prompt` 分别描述不同的音频维度:
44
-
45
- ```python
46
  prompt = model.compose_prompt(
47
  caption="A gritty detective narrating over the sound of heavy rain and a melancholic solo jazz saxophone.",
48
  speech="gritty deep male voice",
@@ -55,17 +80,31 @@ audio = model.generate(prompt)
55
  torchaudio.save("output.wav", audio.cpu(), 16000)
56
  ```
57
 
58
- 也可以直接传入包含标签的完整字符串:
 
 
59
 
60
  ```python
61
- audio = model.generate(
62
- "<|caption|> A helicopter passing overhead. <|sfx|> Rhythmic helicopter blade sounds. <|env|> Open sky ambience."
 
 
 
 
 
63
  )
 
 
64
  ```
65
 
66
  ### 批量推理
67
 
68
  ```python
 
 
 
 
 
69
  prompts = [
70
  model.compose_prompt(caption="A cat meowing softly.", sfx="Soft cat meow."),
71
  model.compose_prompt(caption="Thunder rolling in the distance.", env="Stormy night ambience."),
@@ -80,29 +119,21 @@ for i, audio in enumerate(audios):
80
  ### 生成参数
81
 
82
  ```python
 
 
 
 
 
 
83
  audio = model.generate(
84
- prompts="A dog barking in a park",
85
  num_steps=25, # 去噪步数(默认:25)
86
  guidance_scale=5.0, # 无分类器引导强度(默认:5.0)
87
  sway_sampling_coef=-1.0, # sway 采样系数(默认:-1.0,设为 0 使用线性调度)
88
  )
 
89
  ```
90
 
91
- ## Prompt 格式
92
-
93
- Dasheng-AudioGen 使用结构化标签来描述不同的音频维度:
94
-
95
- | 标签 | 描述 |
96
- |------|------|
97
- | `<\|caption\|>` | 整体音频场景描述 |
98
- | `<\|speech\|>` | 说话人身份和说话风格 |
99
- | `<\|asr\|>` | 语音转写内容 / 对话文本 |
100
- | `<\|sfx\|>` | 音效 |
101
- | `<\|music\|>` | 背景音乐 |
102
- | `<\|env\|>` | 环境音 |
103
-
104
- > **多语言 prompt 规范:** 使用多语言模型时,所有描述性标签(`caption`、`speech`、`sfx`、`music`、`env`)应使用**英文**填写,仅 `<|asr|>` 字段(实际要合成的语音内容)使用目标语言。
105
-
106
  ## 致谢
107
 
108
  Dasheng-AudioGen 由**小米 LLM PLUS** 和 **上海交通大学 X-LANCE** 联合开发。
 
4
  [![Hugging Face Model](https://img.shields.io/badge/HuggingFace-Model-orange?logo=huggingface)](https://huggingface.co/mispeech/Dasheng-AudioGen)
5
  [![Hugging Face Demo](https://img.shields.io/badge/HuggingFace-Demo-orange?logo=huggingface)](https://huggingface.co/spaces/mispeech/Dasheng-AudioGen)
6
  [![Web Demo](https://img.shields.io/badge/Website-Demo-181717?logo=google-chrome)](https://nieeim.github.io/Dasheng-AudioGen-Web/)
7
+ [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/#fileId=https://huggingface.co/mispeech/Dasheng-AudioGen/resolve/main/notebook.ipynb)
8
 
9
  [**English**](./README.md) | [**中文**](./README_zh.md)
10
 
11
  **Dasheng-AudioGen** 是一个统一的音频生成模型,能够根据文本描述同时合成**语音、音乐、音效和环境声**。
12
 
13
+ <p align="center">
14
+ <video
15
+ src="https://github.com/user-attachments/assets/497f5688-8731-4830-8ee7-b9cf4234d900"
16
+ controls
17
+ autoplay
18
+ muted
19
+ loop
20
+ playsinline
21
+ width="85%">
22
+ </video>
23
+ </p>
24
+
25
  ## 模型
26
 
27
  | 模型 | HuggingFace | 文本编码器 | 语言支持 |
 
37
 
38
  > 已在 Python 3.10、torch 2.8.0+cu128、transformers 4.57 上测试通过。已知不兼容 transformers 5.x。
39
 
40
+ ## Prompt 格式
41
+
42
+ Dasheng-AudioGen 使用结构化标签来描述不同的音频维度。有效的 prompt **必须以 `<|caption|>` 标签开头**,用于描述整体音频场景。其他标签为可选项,按需使用。
43
+
44
+ | 标签 | 描述 | 是否必需 |
45
+ |------|------|:--------:|
46
+ | `<\|caption\|>` | 整体音频场景描述 | 是 |
47
+ | `<\|speech\|>` | 说话人身份和说话风格 | 否 |
48
+ | `<\|asr\|>` | 语音转写内容 / 对话文本 | 否 |
49
+ | `<\|sfx\|>` | 音效 | 否 |
50
+ | `<\|music\|>` | 背景音乐 | 否 |
51
+ | `<\|env\|>` | 环境音 | 否 |
52
+
53
+ **规则:**
54
+ - Prompt 必须以 `<|caption|>` 开头,否则会报错。
55
+ - 仅包含有实际内容的标签;没有对应内容的标签请省略(例如没有音乐则不传 `<|music|>`)。
56
+
57
+ > **多语言 prompt 规范:** 使用多语言模型时,所有描述性标签(`caption`、`speech`、`sfx`、`music`、`env`)应使用**英文**填写,仅 `<|asr|>` 字段(实际要合成的语音内容)使用目标语言。
58
+
59
  ## 快速开始
60
 
61
+ ### 用法一:分维度组装
62
+
63
+ 通过命名参数分别传入各个维度的描述。`caption` 为必填项,其他字段可选。
64
 
65
  ```python
66
  import torchaudio
 
68
 
69
  model = AutoModel.from_pretrained("mispeech/Dasheng-AudioGen", trust_remote_code=True).cuda()
70
 
 
 
 
 
 
 
 
 
 
71
  prompt = model.compose_prompt(
72
  caption="A gritty detective narrating over the sound of heavy rain and a melancholic solo jazz saxophone.",
73
  speech="gritty deep male voice",
 
80
  torchaudio.save("output.wav", audio.cpu(), 16000)
81
  ```
82
 
83
+ ### 用法二:传入完整 Prompt 字符串
84
+
85
+ 通过 `prompt` 参数传入预格式化的标签字符串,该字符串必须以 `<|caption|>` 开头。
86
 
87
  ```python
88
+ import torchaudio
89
+ from transformers import AutoModel
90
+
91
+ model = AutoModel.from_pretrained("mispeech/Dasheng-AudioGen", trust_remote_code=True).cuda()
92
+
93
+ prompt = model.compose_prompt(
94
+ prompt="<|caption|> A gritty detective narrating over the sound of heavy rain and a melancholic solo jazz saxophone. <|speech|> gritty deep male voice <|asr|> The city never sleeps, but it sure knows how to cry. <|sfx|> heavy rain hitting pavement <|music|> melancholic solo saxophone <|env|> distant urban ambience"
95
  )
96
+ audio = model.generate(prompt)
97
+ torchaudio.save("output.wav", audio.cpu(), 16000)
98
  ```
99
 
100
  ### 批量推理
101
 
102
  ```python
103
+ import torchaudio
104
+ from transformers import AutoModel
105
+
106
+ model = AutoModel.from_pretrained("mispeech/Dasheng-AudioGen", trust_remote_code=True).cuda()
107
+
108
  prompts = [
109
  model.compose_prompt(caption="A cat meowing softly.", sfx="Soft cat meow."),
110
  model.compose_prompt(caption="Thunder rolling in the distance.", env="Stormy night ambience."),
 
119
  ### 生成参数
120
 
121
  ```python
122
+ import torchaudio
123
+ from transformers import AutoModel
124
+
125
+ model = AutoModel.from_pretrained("mispeech/Dasheng-AudioGen", trust_remote_code=True).cuda()
126
+
127
+ prompt = model.compose_prompt(caption="A dog barking in a park")
128
  audio = model.generate(
129
+ prompts=prompt,
130
  num_steps=25, # 去噪步数(默认:25)
131
  guidance_scale=5.0, # 无分类器引导强度(默认:5.0)
132
  sway_sampling_coef=-1.0, # sway 采样系数(默认:-1.0,设为 0 使用线性调度)
133
  )
134
+ torchaudio.save("output.wav", audio.cpu(), 16000)
135
  ```
136
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
137
  ## 致谢
138
 
139
  Dasheng-AudioGen 由**小米 LLM PLUS** 和 **上海交通大学 X-LANCE** 联合开发。
modeling_dasheng_audiogen.py CHANGED
@@ -30,7 +30,7 @@ TAG_ORDER = OrderedDict([
30
 
31
 
32
  def compose_prompt(
33
- content: str | None = None,
34
  caption: str | None = None,
35
  speech: str | None = None,
36
  asr: str | None = None,
@@ -38,10 +38,21 @@ def compose_prompt(
38
  music: str | None = None,
39
  env: str | None = None,
40
  ) -> str:
41
- if content is not None:
42
- content = str(content).strip()
43
- if content:
44
- return content
 
 
 
 
 
 
 
 
 
 
 
45
 
46
  values = {
47
  "caption": caption, "speech": speech, "asr": asr,
@@ -54,10 +65,6 @@ def compose_prompt(
54
  value = str(value).strip()
55
  if value:
56
  chunks.append(f"{tag} {value}")
57
- if not chunks:
58
- raise ValueError(
59
- "No prompt content provided. Pass `content` or at least one aspect field."
60
- )
61
  return " ".join(chunks)
62
 
63
 
@@ -231,7 +238,7 @@ class DashengAudioGenModel(PreTrainedModel):
231
 
232
  @staticmethod
233
  def compose_prompt(
234
- content: str | None = None,
235
  caption: str | None = None,
236
  speech: str | None = None,
237
  asr: str | None = None,
@@ -240,7 +247,7 @@ class DashengAudioGenModel(PreTrainedModel):
240
  env: str | None = None,
241
  ) -> str:
242
  return compose_prompt(
243
- content=content, caption=caption, speech=speech,
244
  asr=asr, sfx=sfx, music=music, env=env,
245
  )
246
 
 
30
 
31
 
32
  def compose_prompt(
33
+ prompt: str | None = None,
34
  caption: str | None = None,
35
  speech: str | None = None,
36
  asr: str | None = None,
 
38
  music: str | None = None,
39
  env: str | None = None,
40
  ) -> str:
41
+ if prompt is not None:
42
+ prompt = str(prompt).strip()
43
+ if not prompt:
44
+ raise ValueError("The `prompt` string is empty.")
45
+ if not prompt.startswith("<|caption|>"):
46
+ raise ValueError(
47
+ "The `prompt` string must start with the <|caption|> tag. "
48
+ f"Got: {prompt[:50]!r}..."
49
+ )
50
+ return prompt
51
+
52
+ if caption is None or not str(caption).strip():
53
+ raise ValueError(
54
+ "The `caption` field is required and cannot be empty."
55
+ )
56
 
57
  values = {
58
  "caption": caption, "speech": speech, "asr": asr,
 
65
  value = str(value).strip()
66
  if value:
67
  chunks.append(f"{tag} {value}")
 
 
 
 
68
  return " ".join(chunks)
69
 
70
 
 
238
 
239
  @staticmethod
240
  def compose_prompt(
241
+ prompt: str | None = None,
242
  caption: str | None = None,
243
  speech: str | None = None,
244
  asr: str | None = None,
 
247
  env: str | None = None,
248
  ) -> str:
249
  return compose_prompt(
250
+ prompt=prompt, caption=caption, speech=speech,
251
  asr=asr, sfx=sfx, music=music, env=env,
252
  )
253