JoyAI-Video-Edit is a real-time, instruction-guided video editing system for open-ended video streams. Given a live camera stream or uploaded video and a natural-language edit instruction, it edits frames causally as they arrive, without waiting for the full video, requiring a predefined video length, or revisiting future frames. In our deployment benchmark, the full end-to-end pipeline reaches 30 FPS at 720 × 1248, pushing video editing from offline batch processing toward interactive streaming generation.

The system combines an MLLM-based condition encoder, a causal video VAE, and a 16B-parameter multimodal diffusion transformer. It is trained and deployed as an autoregressive diffusion editor, then accelerated with aligned autoregressive distribution matching distillation, long-horizon optimization, bounded KV-state inference, and deployment-oriented scheduling to sustain high-throughput 720p editing while reducing train-inference mismatch and accumulated temporal drift.

• 2026.08.24: 🎉 Consumer GPU support landed — real-time streaming video editing on a single GeForce RTX 5090 (32 GB): 840 × 480 @ 24 FPS. Deployment Guide
• 2026.08.15: 🎉 Live demo released — real-time streaming video editing on a single RTX PRO 6000 (Blackwell) GPU: 840 × 480 @ 24 FPS or 720p @ 16 FPS. Try HuggingFace Demo
• 2026.08.14: 🎉 Released an upgraded checkpoint with significantly stronger reference-image-guided video editing (RV2V), delivering better subject and identity preservation, more faithful reference conditioning, and improved temporal consistency across long streams. Grab the new DiT weights.
• 2026.08.05: 🎉 We release the deployment code, technical report, and JoyAI-Video-Edit checkpoints. Please check the links above for details.

• Real-time open-ended editing. Edits live or uploaded videos as frames arrive, without requiring the full sequence upfront.
• Diverse instruction control. Supports subject edits, local edits, background changes, style transfer, motion changes, and reference-guided editing.
• Autoregressive diffusion design. Combines an MLLM condition encoder, causal video VAE, and MMDiT backbone for streaming video editing.
• High-throughput 720p deployment. Reaches 30 FPS end-to-end throughput at 720 × 1248 with bounded KV-state inference and stable per-chunk compute.

Point your webcam at yourself, type an instruction, and watch the edit stream back in real time — try it in the browser: Hugging Face Demo. The screenshots below are captured live from it (left: camera source, right: streamed result).

• Stronger model version in progress. A more powerful version is under active development, with a particular focus on advancing reference-image-guided video editing (RV2V) capabilities.
• Consumer GPU support. Optimize deployment for consumer-grade GPUs such as GeForce RTX 5090.
• Diffusers support. Provide a 🤗 Diffusers pipeline for JoyAI-Video-Edit to streamline loading and inference.
• LongV2VBench release. Release LongV2VBench for long-form video-to-video editing evaluation.
• Release full training and data pipelines. Open-source the complete training framework and data generation pipeline.

JoyAI-Video-Edit is designed for broad video editing tasks, including global appearance changes, local object edits, subject add/remove/replace, background replacement, style transfer, and reference-guided edits.

Source	Prompt	Edited
	Transform the people, hairstyles, and interior into a British castle aristocratic style.
	Turn the video into a watercolor wash style.
	Make all dogs white, add colorful hats, and turn the sunglasses hot pink.
	Dress the girl in a brown down jacket and blue baseball cap.
	Remove the two white cats in pink clothes on both sides.

RV2V: Put the coat/dress/shirt ... from Image 1 on the model in the video

Download the released JoyAI-Video-Edit weights from Hugging Face, then place them under:

deploy/deps/checkpoints/JoyAI-Video-Edit/
|-- dit/
|   `-- joyai_video_edit_dit_0811.pth
`-- vae/
    |-- config.json
    `-- diffusion_pytorch_model.safetensors

conda create -n joyai-video-edit python=3.10 -y
conda activate joyai-video-edit
python -m pip install -r deploy/requirements.txt

Then build the in-tree CUDA op library deploy/joyomni_ops (required — the DiT calls its fused kernels at runtime) and, per GPU, SageAttention / flash-attn-4: follow DEPLOYMENT.md §2.

Download the released weights from the Hugging Face link above. MiMo-VL is a required external dependency; the ONNX detectors are optional (missing files just disable the corresponding gates). See DEPLOYMENT.md for details.

cd deploy
bash run_server.sh

Then open:

http://localhost:8080

The server binds 0.0.0.0:8080 by default — for remote machines, just open the port (or use SSH port forwarding).

If JoyAI-Video-Edit is useful for your research or product prototype, please cite:

@article{xiao2026joyai,
  title={JoyAI-Video-Edit: Real-Time Open-Ended Video Editing with Autoregressive Diffusion},
  author={Xiao, Yicheng and Dai, Wenxun and Qin, Xinran and Song, Lin and Zhang, Maoquan and Xu, Hang and Chen, Yukang and Li, Yitong and Zhang, Guohui and Zhang, Yuan and Zhang, Xuying and Zhang, Tommy and Yuan, Jianlong and Li, Peihao and Lu, Shuai and Fu, Siming and Zhao, Chuyang and Han, Xin and Huang, Jie and Li, Wenbo and Ma, Guoqing and Huang, Wei and Qi, Xiaojuan and Huang, Haoyang and Duan, Nan},
  journal={arXiv preprint arXiv:2608.03974},
  year={2026}
}

JoyAI-Video-Edit is licensed under Apache 2.0.