FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

add run time and an interface · speechbrain/speechbrain@c14048b · GitHub

/ speechbrain Public

Commit c14048b

Browse files
BenoitWang
committed
add run time and an interface
1 parent 045d2ed commit c14048b

3 files changed

Lines changed: 221 additions & 9 deletions

File tree

‎recipes/ZaionEmotionDataset/README.md‎

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,18 +1,16 @@
11
# Speech Emotion Diarization (SED)
22

3-
Speech Emotion Diarization ([arXiv link](https://arxiv.org/pdf/2306.12991.pdf)) aims to predict the correct emotions and their temporal boundaries within an utterance. For now, the model was trained with audios that contain only 1 non-neutral emotion event. The output is a dictionary of emotion components (neutral/happy/angry/sad) and their boundaries such as:
3+
[Speech Emotion Diarization](https://arxiv.org/pdf/2306.12991.pdf) is a technique that focuses on predicting emotions and their corresponding time boundaries within a speech recording. The model, described in the research paper titled "Speech Emotion Diarization" ([available here](https://arxiv.org/pdf/2306.12991.pdf)), has been trained using audio samples that include neutral and a non-neutral emotional event. The model's output takes the form of a dictionary comprising emotion components (*neutral*, *happy*, *angry*, and *sad*) along with their respective start and end boundaries, as exemplified below:
44

5-
```
5+
```python
66
{
7-
'example.wav':
8-
[
9-
{'start': 0.0, 'end': 1.94, 'emotion': 'n'}, # n -> neutral
10-
{'start': 1.94, 'end': 4.48, 'emotion': 'h'} # h -> happy
11-
]
7+
'example.wav': [
8+
{'start': 0.0, 'end': 1.94, 'emotion': 'n'}, # 'n' denotes neutral
9+
{'start': 1.94, 'end': 4.48, 'emotion': 'h'} # 'h' denotes happy
10+
]
1211
}
1312
```
1413

15-
1614
## Dependencies
1715

1816
First, please install the extra dependencies, do `pip install -r extra_requirements.txt`
@@ -64,6 +62,8 @@ The EDER (Emotion Diarization Error Rate) reported here was averaged on 5 differ
6462
|:-------------:|:---------------------------:|
6563
| WavLM-large | 30.2 ± 1.60 |
6664

65+
It takes about 40 mins/epoch with 1xRTX8000(40G), reduce the batch size if OOM.
66+
6767
## Inference
6868

6969
The pretrained models and a easy-inference interface can be found on [HuggingFace](https://huggingface.co/speechbrain/emotion-diarization-wavlm-large).

‎speechbrain/pretrained/interfaces.py‎

Lines changed: 212 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3028,3 +3028,215 @@ def transcribe_batch(self, wavs, wav_lens):
30283028
def forward(self, wavs, wav_lens):
30293029
"""Runs full transcription - note: no gradients through decoding"""
30303030
return self.transcribe_batch(wavs, wav_lens)
3031+
3032+
3033+
class Speech_Emotion_Diarization(Pretrained):
3034+
"""A ready-to-use SED interface (audio -> emotions and their durations)
3035+
3036+
Arguments
3037+
---------
3038+
hparams
3039+
Hyperparameters (from HyperPyYAML)
3040+
3041+
Example
3042+
-------
3043+
>>> from speechbrain.pretrained import Speech_Emotion_Diarization
3044+
>>> tmpdir = getfixture("tmpdir")
3045+
>>> sed_model = Speech_Emotion_Diarization.from_hparams(source="speechbrain/emotion-diarization-wavlm-large", savedir=tmpdir,) # doctest: +SKIP
3046+
>>> sed_model.diarize_file("speechbrain/emotion-diarization-wavlm-large/example.wav") # doctest: +SKIP
3047+
"""
3048+
3049+
MODULES_NEEDED = ["input_norm", "wav2vec", "output_mlp"]
3050+
3051+
def __init__(self, *args, **kwargs):
3052+
super().__init__(*args, **kwargs)
3053+
3054+
def diarize_file(self, path):
3055+
"""Get emotion diarization of a spoken utterance.
3056+
3057+
Arguments
3058+
---------
3059+
path : str
3060+
Path to audio file which to diarize.
3061+
3062+
Returns
3063+
-------
3064+
dict
3065+
The emotions and their boundaries.
3066+
"""
3067+
waveform = self.load_audio(path)
3068+
# Fake a batch:
3069+
batch = waveform.unsqueeze(0)
3070+
rel_length = torch.tensor([1.0])
3071+
frame_class = self.diarize_batch(
3072+
batch, rel_length, [path]
3073+
)
3074+
return frame_class
3075+
3076+
def encode_batch(self, wavs, wav_lens):
3077+
"""Encodes audios into fine-grained emotional embeddings
3078+
3079+
Arguments
3080+
---------
3081+
wavs : torch.tensor
3082+
Batch of waveforms [batch, time, channels].
3083+
wav_lens : torch.tensor
3084+
Lengths of the waveforms relative to the longest one in the
3085+
batch, tensor of shape [batch]. The longest one should have
3086+
relative length 1.0 and others len(waveform) / max_length.
3087+
Used for ignoring padding.
3088+
3089+
Returns
3090+
-------
3091+
torch.tensor
3092+
The encoded batch
3093+
"""
3094+
if len(wavs.shape) == 1:
3095+
wavs = wavs.unsqueeze(0)
3096+
3097+
# Assign full length if wav_lens is not assigned
3098+
if wav_lens is None:
3099+
wav_lens = torch.ones(wavs.shape[0], device=self.device)
3100+
3101+
wavs, wav_lens = wavs.to(self.device), wav_lens.to(self.device)
3102+
3103+
wavs = self.mods.input_norm(wavs, wav_lens)
3104+
outputs = self.mods.wav2vec2(wavs)
3105+
return outputs
3106+
3107+
def diarize_batch(self, wavs, wav_lens, batch_id):
3108+
"""Get emotion diarization of a batch of waveforms.
3109+
3110+
The waveforms should already be in the model's desired format.
3111+
You can call:
3112+
``normalized = EncoderDecoderASR.normalizer(signal, sample_rate)``
3113+
to get a correctly converted signal in most cases.
3114+
3115+
Arguments
3116+
---------
3117+
wavs : torch.tensor
3118+
Batch of waveforms [batch, time, channels].
3119+
wav_lens : torch.tensor
3120+
Lengths of the waveforms relative to the longest one in the
3121+
batch, tensor of shape [batch]. The longest one should have
3122+
relative length 1.0 and others len(waveform) / max_length.
3123+
Used for ignoring padding.
3124+
batch_id : torch.tensor
3125+
id of each batch (file names etc.)
3126+
3127+
Returns
3128+
-------
3129+
torch.tensor
3130+
The frame-wise predictions
3131+
"""
3132+
outputs = self.encode_batch(wavs, wav_lens)
3133+
averaged_out = self.hparams.avg_pool(outputs)
3134+
outputs = self.mods.output_mlp(averaged_out)
3135+
outputs = self.hparams.log_softmax(outputs)
3136+
score, index = torch.max(outputs, dim=-1)
3137+
preds = self.hparams.label_encoder.decode_torch(index)
3138+
results = self.preds_to_diarization(preds, batch_id)
3139+
return results
3140+
3141+
def preds_to_diarization(self, prediction, batch_id):
3142+
"""Convert frame-wise predictions into a dictionary of
3143+
diarization results.
3144+
3145+
Returns
3146+
-------
3147+
dictionary
3148+
A dictionary with the start/end of each emotion
3149+
"""
3150+
results = {}
3151+
3152+
for i in range(len(prediction)):
3153+
pred = prediction[i]
3154+
lol = []
3155+
for j in range(len(pred)):
3156+
start = round(self.hparams.stride * 0.02 * j, 2)
3157+
end = round(start + self.hparams.window_length * 0.02, 2)
3158+
lol.append([batch_id[i], start, end, pred[j]])
3159+
3160+
lol = self.merge_ssegs_same_emotion_adjacent(lol)
3161+
results[batch_id[i]] = [
3162+
{"start": k[1], "end":k[2], "emotion": k[3]} for k in lol
3163+
]
3164+
return results
3165+
3166+
def forward(self, wavs, wav_lens):
3167+
"""Runs full transcription - note: no gradients through decoding"""
3168+
return self.transcribe_batch(wavs, wav_lens)
3169+
3170+
def is_overlapped(self, end1, start2):
3171+
"""Returns True if segments are overlapping.
3172+
3173+
Arguments
3174+
---------
3175+
end1 : float
3176+
End time of the first segment.
3177+
start2 : float
3178+
Start time of the second segment.
3179+
3180+
Returns
3181+
-------
3182+
overlapped : bool
3183+
True of segments overlapped else False.
3184+
3185+
Example
3186+
-------
3187+
>>> from speechbrain.processing import diarization as diar
3188+
>>> diar.is_overlapped(5.5, 3.4)
3189+
True
3190+
>>> diar.is_overlapped(5.5, 6.4)
3191+
False
3192+
"""
3193+
3194+
if start2 > end1:
3195+
return False
3196+
else:
3197+
return True
3198+
3199+
def merge_ssegs_same_emotion_adjacent(self, lol):
3200+
"""Merge adjacent sub-segs if they are the same emotion.
3201+
Arguments
3202+
---------
3203+
lol : list of list
3204+
Each list contains [utt_id, sseg_start, sseg_end, emo_label].
3205+
Returns
3206+
-------
3207+
new_lol : list of list
3208+
new_lol contains adjacent segments merged from the same emotion ID.
3209+
Example
3210+
-------
3211+
>>> from speechbrain.utils.EDER import merge_ssegs_same_emotion_adjacent
3212+
>>> lol=[['u1', 0.0, 7.0, 'a'],
3213+
... ['u1', 7.0, 9.0, 'a'],
3214+
... ['u1', 9.0, 11.0, 'n'],
3215+
... ['u1', 11.0, 13.0, 'n'],
3216+
... ['u1', 13.0, 15.0, 'n'],
3217+
... ['u1', 15.0, 16.0, 'a']]
3218+
>>> merge_ssegs_same_emotion_adjacent(lol)
3219+
[['u1', 0.0, 9.0, 'a'], ['u1', 9.0, 15.0, 'n'], ['u1', 15.0, 16.0, 'a']]
3220+
"""
3221+
new_lol = []
3222+
3223+
# Start from the first sub-seg
3224+
sseg = lol[0]
3225+
flag = False
3226+
for i in range(1, len(lol)):
3227+
next_sseg = lol[i]
3228+
# IF sub-segments overlap AND has same emotion THEN merge
3229+
if self.is_overlapped(sseg[2], next_sseg[1]) and sseg[3] == next_sseg[3]:
3230+
sseg[2] = next_sseg[2] # just update the end time
3231+
# This is important. For the last sseg, if it is the same emotion then merge
3232+
# Make sure we don't append the last segment once more. Hence, set FLAG=True
3233+
if i == len(lol) - 1:
3234+
flag = True
3235+
new_lol.append(sseg)
3236+
else:
3237+
new_lol.append(sseg)
3238+
sseg = next_sseg
3239+
# Add last segment only when it was skipped earlier.
3240+
if flag is False:
3241+
new_lol.append(lol[-1])
3242+
return new_lol

‎tests/recipes/ZaionEmotionDataset.csv‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,2 @@
11
Task,Dataset,Script_file,Hparam_file,Data_prep_file,Readme_file,Result_url,HF_repo,test_debug_flags,test_debug_checks
2-
Emotion_Diarization,ZaionEmotionDataset,recipes/ZaionEmotionDataset/emotion_diarization/train.py,recipes/ZaionEmotionDataset/emotion_diarization/hparams/train.yaml,recipes/ZaionEmotionDataset/emotion_diarization/zed_prepare.py,recipes/ZaionEmotionDataset/README.md,https://www.dropbox.com/sh/woudm1v31a7vyp5/AADAMxpQOXaxf8E_1hX202GJa?dl=0,https://huggingface.co/speechbrain/emotion-diarization-wavlm-large/,--data_folder=tests/samples/ASR/ --train_annotation=tests/samples/annotation/ASR_train.json --valid_annotation=tests/samples/annotation/ASR_dev.json --test_annotation=tests/samples/annotation/ASR_dev.json --number_of_epochs=2 --skip_prep=True --wav2vec2_folder=tests/tmp/wav2vec2_checkpoint,
2+
Emotion_Diarization,ZaionEmotionDataset,recipes/ZaionEmotionDataset/emotion_diarization/train.py,recipes/ZaionEmotionDataset/emotion_diarization/hparams/train.yaml,recipes/ZaionEmotionDataset/emotion_diarization/zed_prepare.py,recipes/ZaionEmotionDataset/README.md,https://www.dropbox.com/sh/woudm1v31a7vyp5/AADAMxpQOXaxf8E_1hX202GJa?dl=0,https://huggingface.co/speechbrain/emotion-diarization-wavlm-large,--data_folder=tests/samples/ASR/ --train_annotation=tests/samples/annotation/ASR_train.json --valid_annotation=tests/samples/annotation/ASR_dev.json --test_annotation=tests/samples/annotation/ASR_dev.json --number_of_epochs=2 --skip_prep=True --wav2vec2_folder=tests/tmp/wav2vec2_checkpoint,

0 commit comments

Comments
 (0)
Back | FazBrowse Home | New Git URL