A deep dive into how DeepSeek-R1 was trained from scratch and how this open-source research will accelerate AI progress like never before.
Image generated with DALL-E 3
It’s incredible to see how far AI has progressed in the last decade.
Most of this progress came after Google released their groundbreaking paper called “ Attention Is All You Need ” in 2017.
It is then that other companies worked on the ideas discussed in this paper (the Transformer architecture) to build powerful LLMs.
One of these companies, OpenAI, before the release of transformers, was heavily focused on reinforcement learning (RL) and moved its trajectory towards LLMs, benefiting exponentially from Google’s open-sourced research.
Although OpenAI started as an organisation to democratise AI, it ended up making its research and products proprietary.
The last open-source model released by OpenAI was GPT-2, which was made publicly available in November 2019.
Since then, all their model advancements have been kept a secret.
With OpenAI releasing ‘o1’ a few months ago, they have really discovered something…