Temporal Latent Bottleneck combines recurrence and self-attention in an unified way. Recurrence integrates information over time, and self-attention models local dependencies in "short" context. arxiv.org/abs/2205.14794

Discrete Key-Value Bottleneck (Updated) Compresses the information of a pre-trained model in learnable "key-value" codebook such that knowledge can be quickly adapted in a continual learning fashion. arxiv.org/abs/2207.11240

Work from Danilo J. Rezende, Shakir Mohamed and Daan Weistra. proceedings.mlr.press/v32/rezende14.… Credit assignment is difficult.

Interesting progress from Rafael Rafailov @ NeurIPS and FTP et. al following our work (applied to mathematical and commonsense reasoning): Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learning arxiv.org/abs/2405.00451 (Also discussed in Llama-3 paper, AI at Meta )

Boosting LLM Performance with Dynamic Skill Selection! 1/ 🚀 What if LLMs could get better at solving math problems by understanding the skills they need? We explored this idea by having LLMs identify and label the skills required for each problem. arxiv.org/abs/2405.12205

To me, diffusion LMs work because they remove unnecessary inductive biases. The left-to-right inductive bias is natural for human but is unlikely to be natural for AI. This gives more capacity to our models like Transformer having a bigger capacity than LSTM. Our experiment