This major release extends SpeechBrain's support for SpeechLLMs and introduces several new features, recipes, and improvements.

Highlights

• Feature Caching — Save extracted features (e.g. wav2vec embeddings) to disk and load them on the fly, skipping recomputation. This powers our first ASR SpeechLLM recipe on LibriSpeech, enabling LLM-based training with pre-computed embeddings.
• New Recipes — SpeechLLM for ASR and translation, streaming SSL, FocalCodec, and SENSE models.

Along with internal improvements and bug fixes. Here follows a changelog of the main changes (omitting some minor bugfixes):

What's Changed

• Reforging LLaMA lobe (code from Samsung AI Center-Cambridge) by @TParcollet in #2850
• Streaming recipe for BestRQ by @Chaanks in #2790
• Librilight data preparation for SpeechBrain SSL (code from Samsung AI Center Cambridge) by @shucongzhang in #2765
• Alignment with CTC ASR models powered by k2. by @ZhaoZeyu1995 in #2772
• SpeechLLM (with LLaMA) and Conformer recipe for speech translation on CoVoST (Code from Samsung AI Center Cambridge) by @TParcollet in #2865
• Feature caching proposal: CachedDynamicItem by @pplantinga in #2985
• Adapting transducer greedy decoding by @younessdkhissi in #2975
• Replace torchaudio I/O with soundfile-based audio_io wrapper by @Copilot in #2989
• FocalCodec [NeurIPS 2025] by @lucadellalib in #3000
• Caching: add compression + filename + closing/loading by @Adel-Moumen in #3005
• Implement per-key padding configuration in PaddedBatch by @Adel-Moumen in #3008
• SpeechLLM LibriSpeech recipe by @Adel-Moumen in #2885
• Remove CTC CUDA + Move transducer loss in integrations by @Adel-Moumen in #3028
• Adding SENSE models by @MaryemBouziane in #2998

New Contributors

• @emmanuel-ferdman made their first contribution in #2900
• @ofiryaish made their first contribution in #2871
• @omidiu made their first contribution in #2923
• @svecjan made their first contribution in #2934
• @nouranali made their first contribution in #2855
• @OscarFree made their first contribution in #2988
• @younessdkhissi made their first contribution in #2975
• @jordanozang made their first contribution in #2982
• @jrochdi made their first contribution in #2947
• @seohyunjun made their first contribution in #2996
• @Daheer made their first contribution in #3022
• @raotnameh made their first contribution in #2617
• @Mr-Neutr0n made their first contribution in #3029
• @georgesabr made their first contribution in #3038
• @MaryemBouziane made their first contribution in #2998

Full Changelog: v1.0.3...v1.1.0

What's Changed

• Fix broken links due to change in voxlingua107 location by @pplantinga in #2877
• Reforging LLaMA lobe (code from Samsung AI Center-Cambridge) by @TParcollet in #2850
• Attempt to fix self.device for AMP by @Adel-Moumen in #2882
• fix convolutions docstrings by @gfdb in #2883
• Attempt fix for input_norm test by @pplantinga in #2887
• Fix dropout at inference by @Adel-Moumen in #2889
• Avoid running tests on draft PRs by @pplantinga in #2890
• Fix input normalization and global normalization variance calculation by @pplantinga in #2835
• Bump version of pyroomacoustics for recipes/DNS/enhancement by @rogiervd in #2899
• Fix argument to rec_loss in (Variational)AutoencoderLoss (from Samsung AI Center Cambridge) by @rogiervd in #2902
• Ensure that x is not unbound in WordEmbeddingEncoder (from Samsung, AI Center, Cambridge) by @rogiervd in #2905
• Fix bug in ModuleList.insert (from Samsung, AI Center, Cambridge) by @rogiervd in #2903
• Fix bug constructing ValueError in Conv2d (from Samsung, AI Center, Cambridge) by @rogiervd in #2904
• Make types correct (from Samsung, AI Center, Cambridge) by @rogiervd in #2912
• Use correct signature for torch.tensor.type() (from Samsung, AI Center, Cambridge) by @rogiervd in #2910
• Update handling of dtype in read_pkl to torch.Tensor (from Samsung, AI Center, Cambridge) by @rogiervd in #2907
• Make arpa_to_fst work when arguments are str (from Samsung, AI Center, Cambridge) by @rogiervd in #2919
• Make f_max integer (from Samsung, AI Center, Cambridge) by @rogiervd in #2918
• Account for all combinations of arguments in IterativeCSVWriter.write (from Samsung, AI Center, Cambridge) by @rogiervd in #2916
• Migrate to modern Python Logger API by @emmanuel-ferdman in #2900
• Do not forget to raise ValueError (from Samsung, AI Center, Cambridge) by @rogiervd in #2909
• Replace axis= by dim= for calls to Torch (from Samsung, AI Center, Cambridge) by @rogiervd in #2913
• Remove unused value (from Samsung, AI Center, Cambridge) by @rogiervd in #2914
• Make Scores.repr actually return something by @rogiervd in #2897
• Remove deprecated abstractproperty (from Samsung, AI Center, Cambridge) by @rogiervd in #2915
• Use "cls" instead of "self" in classmethod by @rogiervd in #2908
• Change location from which to import tqdm by @rogiervd in #2921
• Fix type annotations (from Samsung, AI Center, Cambridge) by @rogiervd in #2911
• Add test for various length mask functions (code from Samsung, AI Center, Cambridge) by @rogiervd in #2894
• Change test for make_padding_mask to expose mistake by @rogiervd in #2895
• fix bug in reverberate with rescale_amp by @ofiryaish in #2871
• docs: Fix contributing.md by @omidiu in #2923
• Renaming largescale ASR to the Loquacious Set by @TParcollet in #2928
• Move files with optional dependencies to integrations folder by @pplantinga in #2782
• Streaming recipe for BestRQ by @Chaanks in #2790
• Librilight data preparation for SpeechBrain SSL (code from Samsung AI Center Cambridge) by @shucongzhang in #2765
• Re-ran tutorial to remove errors by @pplantinga in #2925
• Create FetchConfig for standardizing use of fetch by @pplantinga in #2828
• Fix issue 'GlobalNorm with DDP' by @svecjan in #2934
• Bump torch from 2.6.0 to 2.7.1 in /docs by @dependabot[bot] in #2939
• Solves #2846 Refactor run_opts into a @DataClass by @nouranali in #2855
• Make run_on_main and main_process_only return the result to all proce… by @rogiervd in #2943
• Saveable Generator: initial import by @flexthink in #2937
• Alignment with CTC ASR models powered by k2. by @ZhaoZeyu1995 in #2772
• Fix forced alignment tutorial link to google colab by @pplantinga in #2951
• Fix missing defaults in run options by @pplantinga in #2952
• Ad...

Major Changes

• Add People's Speech (30,000 hours) Conformer ASR (Code from Samsung AI Center Cambridge) by @TParcollet in #2767
• Audio and Music SSL by @poonehmousavi in #2755
• Add new Audio Tokenziers by @poonehmousavi in #2751
• Libriheavy (Code from SAIC-Cambridge) by @shucongzhang in #2781
• Conformer recipe for LargeScaleASR (code from Samsung AI Center Cambridge) by @TParcollet in #2806
• Rotary Position Embedding (RoPE) for ASR (code from Samsung Cambridge) by @shucongzhang in #2799
• Voice analysis functions by @pplantinga in #2689

New Contributors

• @rogiervd made their first contribution in #2734
• @benniekiss made their first contribution in #2746
• @mirofedurco made their first contribution in #2762
• @kit1980 made their first contribution in #2797
• @IliasMAOUDJ made their first contribution in #2574

Full Changelog: v1.0.2...v1.0.3

This is a minor update which includes some new features and recipes, internal improvements, bugfixes and improved tutorials.

Here follows a changelog of the main changes (omitting some minor bugfixes):

Notable changes

• Added support for Adapters, see new tutorial (#2563)
• Added BEST-RQ with a recipe for LibriSpeech (#2309)
• Added GigaSpeech recipes for ASR, with a Conformer RNN-T and WavLM CTC (#2421)
• Overhauled fetch and Pretrained, greatly reducing the use of symlinks out of the box as much as possible (see down below)
• Migrated all tutorials to the SpeechBrain repository, merged them into the main docs, improved documentation (updated outdated information, broken links, etc.).
• Added new tutorials:
  • Metrics for Speech Recognition
  • Neural Network Adapters for faster low-memory fine-tuning
  • Streaming Speech Recognition with Conformers
• TensorFloat32 is now enabled by default when available (#2682)

New features

• Added layer dropout support for TransformerASR (#2309)
• Added the sign flip augmentation for ASR/EEG/potentially more (#2636)
• Improved reproducibility by adding seed_everything, improved DDP handling for seeding (#2654)
• Added "quirks" to centralize overriden PyTorch defaults and workarounds (among other things) in an easy-to-find fashion, with proper logging (#2558)

Bugfixes

• Improved performance for VAD inference (#2683)
• Fixed various issues with DDP handling (#2682)
• Fixed broken augmentation integration tests (#2628)
• Fixed error when processing newer CommonVoice (#2647)
• Fixed concat bug in augmentation (#2717)
• Removed EOS in G2P inference which was incorrectly introduced (#2718)
• ... and some more

New fetching semantics

We have made a number of changes that affect how fetch works, and this affects various things in a way you should be aware of.

• In various fetching-related code, such as inference interfaces' from_hparams, the savedir refers to the directory where files should be collected. It is now optional and defaults to None.
  • When fetching files (models, audio, etc.) from local paths or HuggingFace repositories, you no longer need to specify a target directory.
    • For local file fetching, the path is returned directly.
    • For HF fetching, HF cache is used directly.
    • For URL fetching, you still need to specify the savedir.
  • Inference interfaces will no longer pollute your working directory with symlinks by default when loading audio files.
  • Avoiding symlink creation by default is now much friendlier to Windows compatibility. We also added some warnings to help.
  • If you do specify a savedir, behavior should be largely unchanged (although more robust).
• In various fetching-related code, you can now specify a fetch_strategy.
  • The fetching strategy dictates what to do when a file is found locally, but not in the desired savedir. For instance, if some model file is available in HuggingFace cache, you can now choose between copying, symlinking or not linking the file in the savedir.
• fetch now has an allow_network parameter, which defaults to True. When disabled, this fails if the URI is not available locally, or if the file isn't found locally in HF cache.
• fetch also now has an allow_updates parameter, which defaults to True. It interrogates HF even if a local path is present in order to update any model files (or switch revisions), if required.

This is a minor update which includes some new features and recipes, internal improvements, bugfixes, compatibility improvements, and wider Python backwards compatibility.

NOTE: both v1.0.0 and v1.0.1 were released earlier than this date on GitHub. These releases were accidentally marked as drafts.

Notable changes

• We now advertise support from Python 3.8 to 3.12 (instead of 3.9-3.11) and improved our testing in that regard.
• Major improvements to Whisper integration with various tasks supported, fine-tuning fixes, performance improvements, and more (#2450)
• Improved model parameter info printing (#2470)
• Added new metrics mostly targeted at speech recognition (#2451)
• Added backwards compatibility for old speechbrain.pretrained imports that were broken following a v1.0 refactor (#2485)
• Updated BibTeX citation. You may find the latest one here at all times.

Recipes and other features

• Added a VoxPopuli transducer recipe (#2421)
• Upgraded CommonVoice transducer and transformer recipes (#2433, #2465, #2560) with various improvements
• Refactored ESC50 recipes and added FocalNets (#2499), added single-sample inference for interpretation (#2616)
• Speechtokenizer integration (#2497)
• Added support for HiFi-GAN to work with new SSL Discrete Tokens, with support for bitrate-scalable training on LJSpeech and LibriTTS (#2571)
• Added recipe for Listenable Maps for Audio Classifiers (#2538)

Fixes

• Fixed errors with ctc_segmentation (#2505)
• Fixes and refactors for RelPosEncXL (#2498)
• Fixed input normalization incorrectly applying in-place to the user inputs in some cases (#2504)
• DDP fixes (#2506, #2633)
• Fixed backwards compatibility with older torchaudio version when not using streaming stuff (#2532)
• Fixed Separation and Enhancement recipes behavior when NaN is encountered (#2524)
• Fix for too aggressive SpecAugment in LibriSpeech transducer recipe (#2548)
• Fix for double file conversion in CommonVoice data preparation (#2557)
• Fixed SpectrogramDrop errors in some cases (#2564)
• Potential fix for Windows install issues when using the --editable flag (#2541)
• Improvements to SSL Discrete Tokens and refactoring (#2509)
• Fixes and improvements for quaternion networks (#2464)
• Fixed AISHELL models and added backwards compatibility warning for causal in TransformerASR (#2606)
• ... and a few more

Internal changes

• Improved code quality with the inclusion of spellchecking, include sorting, and stronger documentation linting in the CI pipeline.
• Significantly improved CI performance for a better PR development experience.
• Refactored the module structure of SpeechBrain to use lazy-loading when possible, reducing import times and greatly reducing circular import headaches.
• Introduced some infrastructure to enable some preprocessing when importing state dicts](https://gist.github.com/asumagic/1289f391acf849ca01b8ca7f9c5dd069)

Please, help our community project. Star on GitHub!

🚀 What's New in SpeechBrain 1.0?

📅 On February 2024, we released SpeechBrain 1.0, the result of a year-long collaborative effort by a large international network of developers led by our exceptional core development team.

📊 Some Numbers:

• SpeechBrain has evolved into a significant project and stands among the most widely used open-source toolkits for speech processing.
• Over 140 developers have contributed to our repository, getting more than 7.3k stars on GitHub.
• Monthly downloads from PyPI have reached an impressive 200k.
• Expanded to over 200 recipes for Conversational AI, featuring more than 100 pretrained models on HuggingFace.

🌟 Key Updates:

• SpeechBrain 1.0 introduces significant advancements, expanding support for diverse datasets and tasks, including NLP and EEG processing.

• The toolkit now excels in Conversational AI and various sequence processing applications.

• Improvements encompass key techniques in speech recognition, streamable conformer transducers, integration with K2 for Finite State Transducers, CTC decoding and n-gram rescoring, new CTC/joint attention Beam Search interface, enhanced compatibility with HuggingFace Models (including GPT2 and Llama2), and refined data augmentation, training, and inference processes.

• We have created a new repository dedicated to benchmarks, accessible at here. At present, this repository features benchmarks for various domains, including speech self-supervised models (MP3S), continual learning (CL-MASR), and EEG processing (SpeechBrain-MOABB).

For detailed technical information, please refer to the section below.

🔄 Breaking Changes

People familiar with SpeechBrain know very well that we do our best to avoid backward incompatible changes. While SpeechBrain has consistently prioritized maintaining backward compatibility, the introduction of this new major version presented an opportunity for significant enhancements and refactorings.

1. 🤗 HuggingFace Interface Refactor:

   • Previously, our interfaces were limited to specific models like Whisper, HuBERT, WavLM, and wav2vec 2.0.
   • We've refactored the interface to be more general, now supporting any transformer model from HuggingFace including LLMs.
   • Simply inherit from our new interface and enjoy the flexibility.
   • The updated interfaces can be accessed here.

2. 🔍 BeamSearch Refactor:

   • The previous beam search interface, while functional, was challenging to comprehend and modify due to the combined search and rescoring parts.
   • We've introduced a new interface where scoring and search are separated, managed by distinct functions, resulting in simpler and more readable code.
   • This update allows users to easily incorporate various scorers, including n-gram LM and custom heuristics, in the search part.
   • Additionally, support for pure CTC training and decoding, batch and GPU decoding, partial or full candidate scoring, and N-best hypothesis output with neural LM rescorers has been added.
   • An interface to K2 for search based on Finite State Transducers (FST) is now available.
   • The updated decoders are available here.

3. 🎨 Data Augmentation Refactor:

   • The data augmentation capabilities have been enhanced, offering users access to various functions in speechbrain/augment.
   • New techniques, such as CodecAugment, RandomShift (Time), RandomShift (Frequency), DoClip, RandAmp, ChannelDrop, ChannelSwap, CutCat, and DropBitResolution, have been introduced.
   • Augmentation can now be customized and combined using the Augmenter interface in speechbrain/augment/augmenter.py, providing more control during training.
   • Take a look here for a tutorial on speech augmentation.
   • The updated augmenters are available here.

4. 🧠 Brain Class Refactor:

   • The fit_batch method in the Brain Class has been refactored to minimize the need for overrides in training scripts.
   • Native support for different precisions (fp32, fp16, bf16), mixed precision, compilation, multiple optimizers, and improved multi-GPU training with torchrun is now available.
   • Take a look at the refactored brain class here.

5. 🔍 Inference Interfaces Refactor:

   • Inference interfaces, once stored in a single file (speechbrain/pretrained/interfaces.py), are now organized into smaller libraries in speechbrain/inference, enhancing clarity and intuitiveness.
   • You can access the new inference interfaces here.

🔊 Automatic Speech Recognition

• Developed a new recipe for training a Streamable Conformer Transducer using Librispeech dataset (accessible here). The streamable model achieves a Word Error Rate (WER) of 2.72% on the test-clean subset.
• Implemented a dedicated inference inference to support streamable ASR (accessible here).
• New models, including HyperConformer andd Branchformer have been introduced. Examples of recipes utilizing them can be found here.
• Additional support for datasets like RescueSpeech, CommonVoice 14.0, AMI, Tedlium 2.
• The ASR search pipeline has undergone a complete refactoring and enhancement (see comment above).
• A new recipe for Bayesian ASR has been added here.

🔄 Interface with Kaldi2 (K2-FSA)

• Integration of an interface that seamlessly connects SpeechBrain with K2-FSA, allowing for constrained search and more.
• Support for K2 CTC training and lexicon decoding, along with integration of K2 HLG and n-gram rescoring.
• Competitive results achieved with Wav2vec2 on LibriSpeech test sets.
• Explore an example recipe utilizing K2 here.

🎙 Speech Synthesis (TTS)

• Improvements to FastSpeech2.
• Development of the DiffWave Vocoder. See the recipe here.
• Development of a Zero-Shot TTS baseline based on Tacotron. See the recipe here.

🌐 Speech-to-Speech Translation:

• Introduction of new recipes for CVSS datasets and IWSLT 2022 Low-resource Task, based on mBART/NLLB and SAMU wav2vec.

🌟 Speech Generation

• Implementation of diffusion and latent diffusion techniques with an example recipe showcased on AudioMNIST.

🎧 Interpretability of Audio Signals

• Implementation of Learning to Interpret and PIQ techniques with example recipes demonstrated on ECS50.

😊 Speech Emotion Diarization

• Support for Speech Emotion Diarization, featuring an example recipe on the Zaion Emotion Dataset. See the training recipe here.

🎙️ Speaker Recognition

• Introduction of a new Speaker Recognition recipe on Voxceleb Speaker, based on ResNET.

🔊 Speech Enhancement

• Release of a new Speech Enhancement baseline based on the DNS dataset.

🎵 Discrete Audio Representations

• Support for pretrained models with discrete audio representations, including EnCodec and DAC.
• Support for discretization of continuous represen...

SpeechBrain 0.5.16 will be the last minor version of SpeechBrain before the major release of SpeechBrain 1.0.

In this minor version, we have focused on refining the existing features without introducing any interface changes, ensuring a seamless transition to SpeechBrain 1.0 where backward incompatible modifications will take place.

Key Highlights of SpeechBrain 0.5.16:

Bug Fixes: Numerous small fixes have been implemented to enhance the overall stability and performance of SpeechBrain.

Testing and Documentation: We have dedicated efforts to improve our testing infrastructure and documentation, ensuring a more robust and user-friendly experience.

Expanded Model and Dataset Support: SpeechBrain 0.5.16 introduces support for several new models and datasets, enhancing the versatility of the platform. For a detailed list, please refer to the commits below.

Stay informed and get ready for the groundbreaking SpeechBrain 1.0, where we will unveil substantial changes and exciting new features.

Thank you for being a part of the SpeechBrain community!

Commits

• [cea36b4]: Update README.md (Mirco Ravanelli) #1599
• [cead130]: Updated README.md (prometheus) #975
• [779c620]: Update README.md (Mirco Ravanelli) #2124
• [32af2ac]: update requirement (to avoid deprecation error) (Mirco Ravanelli) #975
• [b039df1]: small fixes (Mirco Ravanelli) #975
• [07e7c73]: small fixes (Mirco Ravanelli) #975
• [dac6842]: Update README.md (Mirco Ravanelli) #975
• [75f4c66]: Update README.md (Mirco Ravanelli) #975
• [327a3f5]: Fixed SSVEP yaml file (prometheus) #975
• [067d94e]: Fixed conflicts (prometheus) #975
• [331741d]: Fixed read/write conflicts mne config file when training many models in parallel (prometheus) #975
• [0f25d5b]: Added hparam files for other architectures (prometheus) #975
• [9ba76e3]: Updated LMDA, forcing odd kernel size in depth attention (prometheus) #975
• [6336200]: Fixed activation in LMDA (prometheus) #975
• [1593cc4]: Fixed issue in deepconvnet (prometheus) #975
• [2f0f5f0]: Fixed issue with shallowconvnet (prometheus) #975
• [8f70136]: Fixed issue with lmda (prometheus) #975
• [ac4f9e4]: Merge remote-tracking branch 'origin/develop' into fixeval (Adel Moumen) #2123
• [cdce80c]: fix ddp issue with loading a key (Adel Moumen) #2128
• [66633a0]: Added template yaml files (prometheus) #975
• [6f631a7]: minor additions for tests (pradnya-git-dev) #2120
• [331acdb]: add notes on tests with non-default gpu (Mirco Ravanelli) #2130
• [091b3ce]: fixed hard-coded device (Mirco Ravanelli) #2130
• [cc72c9e]: fixed hard-coded device (Mirco Ravanelli) #2130
• [c60e606]: fixed hard-coded device (Mirco Ravanelli) #2130
• [253859e]: Resolve paths so relative works too (Aku Rouhe) #2128
• [8a98401]: small fix on orion flag (Mirco Ravanelli) #975
• [7da9a95]: extend fix to all files (Mirco Ravanelli) #975
• [4b09ff2]: fix style (Mirco Ravanelli) #975
• [ced2922]: Merge remote-tracking branch 'upstream/develop' into eeg_decoding (Mirco Ravanelli) #975
• [5e070a2]: fix useless file (Mirco Ravanelli) #975
• [46565cf]: Merge branch 'develop' of https://github.com/speechbrain/speechbrain into develop (xuechenliu) #2142
• [19235f2]: Merge remote-tracking branch 'upstream/Adel-Moumen-revert_commit_ddp' into revert_commit_ddp (Adel Moumen) #2128
• [2fb247f]: Save the checkpoint folder and meta only on the main process and communicate to all procs (Peter Plantinga) #2132
• [f37d433]: Only broadcast checkpoint folder if distributed (Peter Plantinga) #2132
• [e23da7d]: Initialize external loggers only on main process (Peter Plantinga) #2134
• [67b1255]: fixes (BenoitWang) #2119
• [70d8901]: Merge branch 'develop' into fs2_internal_alignment (Yingzhi WANG) #2119
• [5565073]: Add file check on all recipe tests (#2126) (Mirco Ravanelli) #2126
• [76923a4]: removeused varibles, add exception types (BenoitWang) #2119
• [[0a18729](https://github.com/speechbrain/speechbrain/commit/0a187291fa95323929ac...

SpeechBrain 0.5.15 Release Notes

We are thrilled to announce the release of SpeechBrain version 0.5.15! This new version represents a significant step forward for our open-source Conversational AI toolkit. The core team, along with a rapidly growing network of contributors, has worked diligently to enhance and expand the toolkit while addressing various issues.

What's New?

This release marks a crucial point as it will likely be the final minor version before the highly anticipated SpeechBrain 1.0, scheduled for release in the coming months. We have achieved notable milestones in this version, and a summary of the key achievements is presented below. For a comprehensive list of all changes, please refer to the detailed notes at the end.

Notable Achievements

1. Benchmark Repository:
   We are proud to introduce the benchmark repository, which aims to provide a standard recipe for researchers to benchmark and compare different techniques and models. Currently, the following benchmarks are available:

   • CLMASR: Evaluates continual learning techniques for speech recognition in new languages.
   • MP3S Benchmarks: Assesses speech self-supervised representation across various tasks and with different downstream models (multi-probe).

2. Enhanced User Experience:
   We've made it more convenient for our users to access logs and checkpoints by migrating the logs and output folders from Gdrive to Dropbox.

3. New Models with Improved Performance:
   We implemented a modified Fastspeech 2.0. This offers efficiency and pretty high performance. We've made significant steps in enhancing performance on Librispeech, thanks to the implementation of better Conformers and Branchformers. Additionally, we've introduced a performant Conformer Transducer and the SLI-GRU model.

4. Post-doc Interpretability Techniques:
   We now offer improved support for post-doc interpretability techniques. Refer to the ESC50 recipe for more information.

5. New Datasets:
   We've incorporated recipes for new datasets, including the recently released RescueSpeech (speech recognition in rescue and domain environments) and the Zaion Emotion Dataset for Speech Emotion Recognition.

6. Enhanced Korean ASR:
   We've made improvements to KsponSpeech for Korean Automatic Speech Recognition.

7. Improved Recipe Tests:
   We've taken steps to enhance recipe tests, ensuring better reliability and performance.

8. ** Whisper Fixes**:
   We've fixed Whisper recipes and interfaces in a way that maintains backward compatibility. This was necessary to address interface changes made in the original model.

9. Various Fixes:
   In addition to the above achievements, we've addressed several other issues, including gradient accumulation and various minor fixes.

Thank you to our dedicated community of contributors and users for making this release possible! We invite you to explore the new features and improvements in SpeechBrain 0.5.15 and look forward to the upcoming release of SpeechBrain 1.0. Happy SpeechBrain-ing!

For a complete list of changes, please refer to the detailed release notes below.

Commits

• [4aff580]: update new model at 2.8 (Titouan Parcollet) #1782
• [2c75924]: Merge branch 'develop' of https://github.com/speechbrain/speechbrain into libri_conformer_large (Titouan Parcollet) #1782
• [a6d5d54]: add testing (Titouan Parcollet) #1782
• [6cb2b25]: push new results (Titouan Parcollet) #1782
• [fefdcfb]: Merge branch 'develop' into fastspeech2 (Mirco Ravanelli) #1572
• [9937302]: download automatically allignments (Mirco Ravanelli) #1572
• [9b9c6c2]: Merge branch 'speechbrain:develop' into RescueSpeech (Sangeet Sagar) #2017
• [555de27]: add hf, dropbox, recipes.csv (Sangeet Sagar) #2017
• [9447794]: fix pre-commit conflicts (Sangeet Sagar) #2017
• [6707d09]: add field- noise_wav (Sangeet Sagar) #2017
• [c604b89]: only kept the WeightedSSLModel class in this PR (salah-zaiem) #2047
• [61d4af1]: removed useless changes (salah-zaiem) #2047
• [042fd1e]: Update README.md (Mirco Ravanelli) #2017
• [cdda921]: Fix with the solution used by Luca in the Benchmarks (Mirco Ravanelli) #2016
• [07e2458]: Update README.md (Mirco Ravanelli) #2017
• [ce53199]: Fix backward compatibility issues (Luca Della Libera) #2016
• [9eadc9e]: add transformers (Sangeet Sagar) #2017
• [3f1959f]: add clarity to training data; add computing power details (Sangeet Sagar) #2017
• [a8265bf]: Update extra_requirements.txt (Parcollet Titouan)
• [2905a52]: unknown token fix (pradnya-git-dev) #1572
• [2d5b40f]: Update README.md (Adel Moumen) #2064
• [d6bc4a8]: manage paths and reduce SOX warnings (BenoitWang) #2048
• [672af7a]: fixes (BenoitWang) #2048
• [045d2ed]: info of input/output, add links, add to recipes (BenoitWang) #2048
• [9f841f1]: Update README.md (Mirco Ravanelli) #2048
• [379772a]: Update ZaionEmotionDataset.csv (Mirco Ravanelli) #2048
• [9cde9cc]: small fix (Mirco Ravanelli) #1572
• [c14048b]: add run time and an interface (BenoitWang) #2048
• [6d0d6ab]: merge fix (BenoitWang) #2048
• [5a9df26]: fix (BenoitWang) #2048
• [6a5237d]: adjusting n_symbols for unknown tokens (pradnya-git-dev) [#1572](https://github.com/speechbrain/speech...

This release is a minor yet important release. It increases significantly the number of features available while fixing quite a lot of small bugs and issues. A summary of the achievements of this release is given below, while a complete detailed list of all the changes can be found at the bottom of this release note.

Notable achievements

• 22 new contributors, thank you so much, everyone!
• 31 new recipes (ASR, SLU, AST, AER, Interpretability, SSL).
• FULL automatic recipe testing.
• Increased coverage for the continuous integration over the code, URLs, YAML, recipes, and HuggingFace models.
• New Conformer Large model for ASR.
• Integration of Whisper for fine-tuning or inference.
• Full pre-training of wav2vec2 entirely re-implemented AND documented.
• Low resource Speech Translation with IWSLT.
• Many other novelties... see below.

What's Changed

• fix 1522 by @anautsch in #1526
• bug-fix: fixed OPEN_RIR data preparation process conflict. by @xin-w8023 in #1536
• add noise and reverberance version for BinauralWSJ0Mix by @huangzj421 in #1502
• fix distributed namespace by @anautsch in #1566
• feat: use member field instead of hard-code by @xin-w8023 in #1567
• Update logo to new version by @pplantinga in #1575
• IWSLT 2022 speech translation recipe by @mzboito in #1475
• Fix Issue #1277 timit recipe missing uppercase option by @Adel-Moumen in #1564
• Update README.md by @qanastek in #1577
• Output hiddens states from all the transformer layers of huggingface_wav2vec by @BenoitWang in #1570
• Fix bugs of update_learning_rate by @wangxin22 in #1578
• Fix to use output of unsqueeze() in Tacotron2 parse_decoder_outputs() by @jqug in #1525
• wav2vec2 pretraining implemented with speechbrain by @RuABraun in #1312
• In filter_ctc_output(), remove redundant filtering by @olvb in #1584
• Fixed output_all_hiddens for hubert in huggingface_wav2vec by @gorinars in #1587
• Fix return value of batch_evaluation for separation recipes by @z-wony in #1555
• fix endless doctest despite no example by @anautsch in #1591
• Fix documented min python version to 3.7 by @asumagic in #1595
• Conformer separation by @ycemsubakan in #1519
• Add CTC recipe to AISHELL-1 by @BenoitWang in #1576
• Add templates for issues by @Adel-Moumen in #1588
• Added workaround for CyclicLR saving by @Gastron in #1683
• scikit-learn import and comment fix by @underdogliu in #1485
• Adding recipe for HiFiGAN training using LibriTTS dataset by @pradnya-git-dev in #1621
• fix LibriSpeech CTC pretrainer by @BenoitWang in #1594
• wav2vec German model added by @sangeet2020 in #1557
• issue 1615 typo fix by @sharmadhiraj86 in #1700
• typo in TransformerASR.py by @Adel-Moumen in #1704
• Causality in Conv2d by @fpaissan in #1608
• Switchboard Recipe by @dwgnr in #1460
• read_audio fixes and docs cleanup by @asumagic in #1592
• Fix path flake8 in pre-commit by @Adel-Moumen in #1721
• Added german_cleaners by @padmalcom in #1642
• fixing issue 1707 by @TParcollet in #1728
• explicit fetch args & download-only option by @anautsch in #1735
• fix sorting bug by @anautsch in #1730
• remove discussions references by @Adel-Moumen in #1737
• Fix torchaudio mel_normalized for Tacotron2&HifiGAN by @BenoitWang in #1740
• Whisper finetuning by @Adel-Moumen in #1717
• loss must be avg when BS>1 when calling evaluate_batch() by @sangeet2020 in #1744
• [FIX] Flush gradients and save memory for validation. by @MartinKocour in #1739
• add coloring in tqdm progress bar by @sangeet2020 in #1573
• Fix librispeech transformer recipe by @TParcollet in #1775
• 🖍️ improving type-hints in speechbrain/pretrained/interfaces.py by @jonasvdd in #1725
• Enabling the retrieval of whisper's hidden states by @Hguimaraes in #1751
• Added fix to use DDP with hifi_gan training on ljspeech by @padmalcom in #1781
• Fix wav2vec2 masking by @TParcollet in #1799
• fix #1794 by @Adel-Moumen in #1805
• refactor: recipe testing CSVs by @anautsch in #1600
• fix 1788 by @BenoitWang in #1842
• fix docstring for pooling by @BenoitWang in #1843
• Whisper finetunng common voice by @poonehmousavi in #1809
• fixing the convtasnet causal=True bug by @ycemsubakan in #1851
• Fix Whisper doc + improve max_decode_ratio by @Adel-Moumen in #1858
• Rewrite multi-GPU documentation by @asumagic in #1861
• SLU Media recipe by @GaelleLaperriere in #1172
• edits for refactoring check tool by @anautsch in #1838
• minor fixes for recipe testing by @anautsch in #1872
• Fix Whisper avoid_if_longer_than never used by @Adel-Moumen in #1882
• Starting a recipe for ESC50 by @ycemsubakan in #1605
• Fix for #1886 by @anautsch in #1890
• fix batch_to_right by @anthony-wss in #1884
• Fixes for pre-release testing by @anautsch in #1895
• Fix Conformer Instabilities and add Large Model by @TParcollet in #1892
• Downsampling by @salah-zaiem in #1888
• fix core.py bf16 by @Adel-Moumen in #1898
• S2SGreedySearcher : Do not continue decoding when EOS token was generated for all samples from a batch by @Jeronymous in #1899
• quick fixes before minor by @anautsch in #1896

New Contributors

• @xin-w8023 made their first contribution in #1536
• @mzboito made their first contribution in #1475
• @qanastek made their first contribution in #1577
• @wangxin22 made their first contribution in #1578
• @jqug made their first contribution in #1525
• @olvb made their first contribution in #1584
• @gorinars made their first contribution in #1587
• @z-wony made their first contribution in #1555
• @asumagic made their first contribution in #1595
• @pradnya-git-dev made their first contribution in #1621
• @sangeet2020 made their first contribution in #1557
• @sharmadhiraj86 made their first contribution in #1700
• @fpaissan made their first contribution in #1608
• @dwgnr made their first contribution in #1460
• @padmalcom made their first contribution in #1642
• @MartinKocour made their first contribution in #1739
• @jonasvdd made their first contribution in https://github...

This is a minor release with better dependency version specification. We note that SpeechBrain is compatible with PyTorch 1.12, and the updated package reflects this. See the issue linked next to each commit for more details about the corresponding changes.

Commit summary

• [edb7714]: Adding no_sync and on_fit_batch_end method to core (Rudolf Arseni Braun) #1449
• [07155e9]: G2P fixes (flexthink) #1473
• [6602dab]: fix for #1469, minimal testing for profiling (anautsch) #1476
• [abbfab9]: test clean-ups: passes linters; doctests; unit & integration tests; load-yaml on cpu (anautsch) #1487
• [1a16b41]: fix ddp incorrect command (=) #1498
• [0b0ec9d]: using no_sync() in fit_batch() of core.py (Rudolf Arseni Braun) #1449
• [5c9b833]: Remove torch maximum compatible version (Peter Plantinga) #1504
• [d0f4352]: remove limit for HF hub as it does not work with colab (Titouan) #1508
• [b78f6f8]: Add revision to hub (Titouan) #1510
• [2c491a4]: fix transducer loss inputs devices (Adel Moumen) #1511
• [4972f76]: missing space in install command (pehonnet) #1512
• [6bc72af]: Fixing shuffle argument for distributed sampler in core.py (Rudolf Arseni Braun) #1518
• [df7acd9]: Added the link for example results (cem) #1523
• [5bae6df]: add LinearWarmupScheduler (Ge Li) #1537
• [2edd7ee]: updating scipy version in requirements.txt. (Nauman Dawalatabad) #1546

Release Notes - SpeechBrain v0.5.12

We worked very hard and we are very happy to announce the new version of SpeechBrain!

SpeechBrain 0.5.12 significantly expands the toolkit without introducing any major interface changes. I would like to warmly thank the many contributors that made this possible.

The main changes are the following:

A) Text-to-Speech: We developed the first TTS system of SpeechBrain. You can find it here. The system relies on Tacotron2 + HiFiGAN (as vocoder). The models coupled with an easy-inference interface are available on HuggingFace.

B) Grapheme-to-Phoneme (G2P): We developed an advanced Grapheme-to-Phoneme. You can find the code here. The current version significantly outperforms our previous model.

C) Speech Separation:

1. We developed a novel version of the SepFormer called Resource-Efficient SepFormer (RE-Sepformer). The code is available here and the pre-trained model (with an easy inference interface) here.
2. We released a recipe for Binaural speech separation with WSJMix. See the code here.
3. We released a new recipe with the AIShell mix dataset. You can see the code here.

D) Speech Enhancement:

1. We released the SepFormer model for speech enhancement. the code is here, while the pre-trained model (with easy-inference interface) is here.
2. We implemented the WideResNet for speech enhancement and use it to mimic loss-based speech enhancement. The code is here and the pretrained model (with easy-inference interface) is here.

E) Feature Front-ends:

1. We now support LEAF filter banks. The code is here. You can find an example of a recipe using it here.
2. We now support SincConv multichannel (see code here).

F) Recipe Refactors:

1. We refactored the Voxceleb recipe and fix the normalization issues. See the new code here. We also made the EER computation method less memory demanding (see here).
2. We refactored the IEMOCAP recipe for emotion recognition. See the new code here.

G) Models for African Languages:
We now have recipes for the DVoice dataset. We currently support Darija, Swahili, Wolof, Fongbe, and Amharic. The code is available here. The pretrained model (coupled with an easy-inference interface) can be found on SpeechBrain-HuggingFace.

H) Profiler:
We implemented a model profiler that helps users while developing new models with SpeechBrain. The profiler outputs a bunch of potentially useful information, such as the real-time factors and many other details.
A tutorial is available here.

I) Tests:
We significantly improved the tests. In particular, we introduced the following tests: HF_repo tests, docstring checks, yaml-script consistency, recipe tests, and check URLs. This will helps us scale up the project.

L) Other improvements:

1. We now support the torchaudio RNNT loss*.
2. We improved the relative attention mechanism of the Conformer.
3. We updated the transformer for LibriSpeech. This improves the performance from WER= 2.46% to 2.26% on the test-clean. See the code here.
4. The Environmental corruption module can now support different sampling rates.
5. Minor fixes.