New work with Ron Weiss, Eric Battenberg, Soroosh Mariooryad, Durk Kingma -- finally achieving what Yuxuan Wang and I set out to do in 2016 before switching to spectrograms: direct waveform generation from characters. (1/7) abs: arxiv.org/abs/2011.03568 samples: google.github.io/tacotron/publi…

Understanding the Difficulty of Training Transformers (arxiv.org/pdf/2004.08249…, Liyuan Liu (Lucas)) Studies instability of transformer. Dives into the dark arts of NN stability, impact of layer norm / residuals. Isolates residual paths as a main cause. Trains 60 layers transformers.

[new VC paper candidate] "Utilizing Self-supervised Representations for MOS Prediction" arXiv: arxiv.org/abs/2104.03017

[paper version up] "Utilizing Self-supervised Representations for MOS Prediction" arXiv: arxiv.org/abs/2104.03017

Three years ago, when we first tried to use GAN to realize ​​unsupervised ASR (arxiv.org/abs/1804.00316), I thought the idea was sci-fi. But a few days ago, Facebook AI pushed the idea of ​​using GAN for unsupervised ASR to 5.9% WER on Librispeech (ai.facebook.com/blog/wav2vec-u…).

Accepted to @INTERSPEECH2021 as 1st author!

Honor to cooperate with the researchers of Facebook, CMU, MIT, and JHU to develop SUPERB. When you pre-train an LM like BERT on text, you use GLUE to evaluate its performance. How about speech? You can use SUPERB, which will be the speech version of GLUE. superbbenchmark.org

Two tutorials at INTERSPEECH'22. Self-Supervised Representation Learning for Speech Processing slides: docs.google.com/presentation/d… Neural Speech Synthesis slides: github.com/tts-tutorial/i…

🚨 Call for Papers – ASRU 2025 Special Session 🎤 Responsible Speech & Audio Generative AI 📍 Honolulu, Hawaii · Dec 2025 Join us to tackle accountability, fairness, and trust in generative speech/music/audio systems! 👉 Deadline: May 28, 2025 🔗 Detail: codecfake.github.io/RespSA-GenAI/

We at NVIDIA and GAMMA UMD are excited to release Audio Flamingo 3, the most powerful, open, and capable large audio-language model to date! Paper: arxiv.org/abs/2507.08128 Open-source model, code, and data: research.nvidia.com/labs/adlr/AF3/ Try it out here: huggingface.co/spaces/nvidia/…

1/7 🔗 Introducing STITCH: our new method to make Spoken Language Models (SLMs) think and talk at the same time. Paper link 👉 arxiv.org/abs/2507.15375