SLAM-LLM

SLAM-LLM is a deep learning toolkit that allows researchers and developers to train custom multimodal large language model (MLLM), focusing on Speech, Language, Audio, Music processing. We provide detailed recipes for training and high-performance checkpoints for inference.

1. News
2. Installation
3. Usage
   • List of Recipes
   • Configuration Priority
4. Features
5. Acknowledge
6. Citation

• [Update Jan. 15, 2026] The SLAM-LLM Framework was accepted by IEEE Journal of Selected Topics in Signal Processing (JSTSP), where we put the framework design and some unpublished experiments and insights for speech, audio, and music MLLM.
• [Update Apr. 24, 2025] We have supported large-scale industrial training, suitable for datasets on the order of 100,000 hours. Its main features include:
  • Support for multi-task training: Designed to support tasks such as ASR and ST through a unified data format.
  • Dynamic prompt selection: Supports random selection from multiple prompts.
  • Iterative dataset: Uses an iterative dataset format to reduce startup time for large datasets.
  • Deepspeed training: Supports DeepSpeed training to significantly reduce memory usage.
  • Multi-machine multi-GPU inference: Supports distributed inference across multiple machines and GPUs to reduce evaluation time.
  • Dynamic frame batching: Dynamically combines frames based on audio size rather than using a fixed batch size, significantly reducing training and evaluation time (reduces training time by 3/4 for 100,000 hours of data).
• [Update Apr. 24, 2025] We have supported the Deepspeed, checkout the instruction #Fine-tuning using Deepspeed at here.
• [Update Jan. 22, 2025] 🔥🔥🔥 Full reproduction (including all data preparation, model training, and inference) for SLAM-Omni has been supported.
  
  • SLAM-Omni is a timbre-controllable voice interaction system that requires only single-stage training and minimal resources to achieve high-quality, end-to-end speech dialogue, supporting multi-turn conversations in both Chinese and English. (paper, demo)
  • We have fully reproduced the training and inference processes of SLAM-Omni and open-sourced all related training datasets. The provided code framework theoretically supports all codec-based spoken dialogue models. Additionally, we offer the reproduction code for Mini-Omni.

• [Update Nov. 17, 2024] Recipes for LLM-Based Contextual ASR have been supported.
• [Update Nov. 5, 2024] Recipes for speech emotion captioning (SEC) with emotion2vec as the encoder has been supported.
• [Update Oct. 12, 2024] Recipes for SLAM-AAC with EAT as the encoder have been supported.
• [Update Sep. 28, 2024] Recipes for CoT-ST have been supported.
• [Update Sep. 25, 2024] Recipes for DRCap have been supported.
• [Update Jun. 12, 2024] Recipes for MaLa-ASR have been supported.
• [CALL FOR EXAMPLE] We sincerely invite developers and researchers to develop new applications, conduct academic research based on SLAM-LLM, and pull request your examples! We also acknowledge engineering PR (such as improving and speeding up multi-node training).
• [Update May. 22, 2024] Please join slack or WeChat group. We will sync our updates and Q&A here.
• [Update May. 21, 2024] Recipes for Spatial Audio Understanding have been supported.
• [Update May. 20, 2024] Recipes for music caption (MC) have been supported.
• [Update May. 8, 2024] Recipes for visual speech recognition (VSR) have been supported.
• [Update May. 4, 2024] Recipes for zero-shot text-to-speech (TTS) have been supported.
• [Update Apr. 28, 2024] Recipes for automated audio captioning (AAC) have been supported.
• [Update Mar. 31, 2024] Recipes for automatic speech recognition (ASR) have been supported.

git clone https://github.com/huggingface/transformers.git
cd transformers
git checkout tags/v4.35.2
pip install -e .
cd ..
git clone https://github.com/huggingface/peft.git
cd peft
git checkout tags/v0.6.0
pip install -e .
cd ..
pip install torch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 --index-url https://download.pytorch.org/whl/cu118
git clone https://github.com/ddlBoJack/SLAM-LLM.git
cd SLAM-LLM
pip install  -e .

For some examples, you may need to use fairseq, the command line is as follows:

# you need to install fairseq before SLAM-LLM
git clone https://github.com/pytorch/fairseq
cd fairseq
pip install --editable ./

We also provide a docker image for convenience:

# build docker image
docker build -t slam-llm:latest .

# run docker image with gpu
docker run -it --gpus all --name slam --shm-size=256g slam-llm:latest /bin/bash

We provide reference implementations of various LLM-based speech, audio, and music tasks:

• Speech Task

  • Automatic Speech Recognition (ASR)

    • SLAM-ASR

  • Contextual Automatic Speech Recognition (CASR)

    • Mala-ASR
    • LLM-Based Contextual ASR

  • Visual Speech Recognition (VSR)

  • Speech-to-Text Translation (S2TT)

    • CoT-ST

  • Text-to-Speech (TTS)

    • VALL-E-X

  • Speech Emotion Captioning (SEC)

  • Voice Interaction System

    • SLAM-Omni

• Audio Task

  • Automated Audio Captioning (AAC)

    • SLAM-AAC
    • DRCap

  • Spatial Audio Understanding

    • BAT

• Music Task

  • Music Caption (MC)

We provide hierarchical configuration inheritance relationships as follows:

command-line (shell file) > Hydra configuration (yaml file) > dataclass configuration (Python file)

• Easily extend to new models and tasks.
• Detailed recipes for training and high-performance checkpoints for inference.
• Mixed precision training which trains faster with less GPU memory on NVIDIA tensor cores.
• Multi-GPU training with data and model parallel, supporting DDP, FSDP and deepspeed (still need to be improved).
• Flexible configuration based on Hydra and dataclass allowing a combination of code, command-line and file based configuration.

• We borrow code from Llama-Recipes for the training process.
• We borrow code from Fairseq for deepspeed configuration.
• We thank the contributors for providing diverse recipes.

SLAM-LLM Framework:

@article{ma2026slam,
  title={SLAM-LLM: A Modular, Open-Source Multimodal Large Language Model Framework and Best Practice for Speech, Language, Audio and Music Processing},
  author={Ma, Ziyang and Yang, Guanrou and Chen, Wenxi and Gao, Zhifu and Du, Yexing and Li, Xiquan and Zheng, Zhisheng and Zhu, Haina and others},
  journal={Proc. IEEE Journal of Selected Topics in Signal Processing},
  year={2026}
}

SLAM-ASR:

@article{ma2025speech,
  title={Speech Recognition Meets Large Language Model: Benchmarking, Models, and Exploration},
  author={Ma, Ziyang and Yang, Guanrou and Yang, Yifan and Gao, Zhifu and Wang, Jiaming and Du, Zhihao and Yu, Fan and Chen, Qian and Zheng, Siqi and Zhang, Shiliang and others},
  journal={Proc. AAAI},
  year={2025}
}

Mala-ASR:

@article{yang2024mala,
  title={MaLa-ASR: Multimedia-Assisted LLM-Based ASR},
  author={Yang, Guanrou and Ma, Ziyang and Yu, Fan and Gao, Zhifu and Zhang, Shiliang and Chen, Xie},
  journal={Proc. INTERSPEECH},
  year={2024}
}

LLM-Based Contextual ASR:

@article{yang2024ctc,
  title={CTC-Assisted LLM-Based Contextual ASR},
  author={Yang, Guanrou and Ma, Ziyang and Gao, Zhifu and Zhang, Shiliang and Chen, Xie},
  journal={Proc. SLT},
  year={2024}
}

SLAM-Omni:

@article{chen2024slam,
  title={SLAM-Omni: Timbre-Controllable Voice Interaction System with Single-Stage Training},
  author={Chen, Wenxi and Ma, Ziyang and Yan, Ruiqi and Liang, Yuzhe and Li, Xiquan and Xu, Ruiyang and Niu, Zhikang and Zhu, Yanqiao and Yang, Yifan and Liu, Zhanxun and others},
  journal={Proc. ACL},
  year={2025}
}

CoT-ST:

@article{du2024cot,
  title={CoT-ST: Enhancing LLM-based Speech Translation with Multimodal Chain-of-Thought},
  author={Du, Yexing and Ma, Ziyang and Yang, Yifan and Deng, Keqi and Chen, Xie and Yang, Bo and Xiang, Yang and Liu, Ming and Qin, Bing},
  journal={Proc. ACL},
  year={2025}
}

SLAM-AAC:

@article{chen2025slam,
  title={SLAM-AAC: Enhancing Audio Captioning with Paraphrasing Augmentation and CLAP-Refine through LLMs},
  author={Chen, Wenxi and Ma, Ziyang and Li, Xiquan and Xu, Xuenan and Liang, Yuzhe and Zheng, Zhisheng and Yu, Kai and Chen, Xie},
  journal={Proc. ICASSP},
  year={2025}
}

DRCap:

@article{li2025drcap,
  title={DRCap: Decoding CLAP Latents with Retrieval-augmented Generation for Zero-shot Audio Captioning},
  author={Li, Xiquan and Chen, Wenxi and Ma, Ziyang and Xu, Xuenan and Liang, Yuzhe and Zheng, Zhisheng and Kong, Qiuqiang and Chen, Xie},
  journal={Proc. ICASSP},
  year={2025}
}

BAT:

@article{zheng2024bat,
  title={BAT: Learning to Reason about Spatial Sounds with Large Language Models},
  author={Zheng, Zhisheng and Peng, Puyuan and Ma, Ziyang and Chen, Xie and Choi, Eunsol and Harwath, David},
  journal={Proc. ICML},
  year={2024}
}