We're announcing, together with Eric Schmidt: Superalignment Fast Grants.

$10M in grants for technical research on aligning superhuman AI systems, including weak-to-strong generalization, interpretability, scalable oversight, and more.

Apply by Feb 18! openai.com/blog/superalig…

RLHF works great for today's models. But aligning future superhuman models will present fundamentally new challenges.

We need new approaches + scientific understanding.

New researchers can make enormous contributions—and we want to fund you!

Apply by Feb 18!

My view is that what makes super-alignment 'super' is ensuring we can safely scale the capabilities of AIs even though we can't scale their human supervisors. For this, it is imperative to study the 'weak teacher strong student' setting. Paper shows great promise in this area!

i'd particularly like to recognize Collin Burns for today's generalization result, who came to openai excited to pursue this vision and helped get the rest of the team excited about it!

Large pretrained models have excellent raw capabilities—but can we elicit these fully with only weak supervision?

GPT-4 supervised by ~GPT-2 recovers performance close to GPT-3.5 supervised by humans—generalizing to solve even hard problems where the weak supervisor failed!

new paper! one reason aligning superintelligence is hard is because it will be different from current models, so doing useful empirical research today is hard. we fix one major disanalogy of previous empirical setups. I'm excited for future work making it even more analogous.

New direction for AI alignment — weak-to-strong generalization.

Promising initial results: we used outputs from a weak model (fine-tuned GPT-2) to communicate a task to a stronger model (GPT-4), resulting in intermediate (GPT-3-level) performance.

Extremely excited to have this work out, the first paper from the Superalignment team! We study how large models can generalize from supervision of much weaker models.

twitter.com/OpenAI/status/…

Kudos especially to Collin Burns for being the visionary behind this work, Pavel Izmailov for all the great scientific inquisition, Ilya Sutskever for stoking the fires, Jan Hendrik Kirchner and Leopold Aschenbrenner for moving things forward every day. Amazing ✨

I’m extremely excited to finally share the first paper from the OpenAI Superalignment team :)

In it, we introduce a new research direction for aligning superhuman AI systems. 🧵

twitter.com/OpenAI/status/…

Super excited about our new research direction for aligning smarter-than-human AI:

We finetune large models to generalize from weak supervision—using small models instead of humans as weak supervisors.

Check out our new paper:
openai.com/research/weak-…

In the future, humans will need to supervise AI systems much smarter than them.

We study an analogy: small models supervising large models.

Read the Superalignment team's first paper showing progress on a new approach, weak-to-strong generalization: openai.com/research/weak-…

❤️

Sam Altman is back as CEO, Mira Murati as CTO and Greg Brockman as President. OpenAI has a new initial board. Messages from Sam Altman and board chair Bret Taylor openai.com/blog/sam-altma…

There exists no sentence in any language that conveys how happy I am:

Returning to OpenAI & getting back to coding tonight.

🚢ing

i love openai, and everything i’ve done over the past few days has been in service of keeping this team and its mission together. when i decided to join msft on sun evening, it was clear that was the best path for me and the team. with the new board and w satya’s support, i’m…

I deeply regret my participation in the board's actions. I never intended to harm OpenAI. I love everything we've built together and I will do everything I can to reunite the company.