On June 20th, Anthropic, an AI company built by former OpenAI employees, released their new LLM (Large Language Model): Claude 3.5 Sonnet.

In this study, we evaluate the impact of this release on the popularity of the best LLMs (Claude, Gemini, GPT, Mistral & Llama).

We compared the data from Mammouth AI of the ten days before Anthropic’s new release, under the previous version of Claude 3 Opus, with the ten days after the release of Claude 3.5 Sonnet.

Claude’s popularity rises from 9.3% to 19.4% of total requests

LLM popularity BEFORE the release of Claude 3.5

LLM popularity before the release of Claude 3.5 Sonnet

LLM popularity AFTER the release of Claude 3.5

LLM popularity after the release of Claude 3.5 Sonnet

Other top LLMs benefit from Claude’s release except for GPT

• Gemini also benefits from the release of Claude, increasing from 10% to 12% of total requests
• Mistral Large maintains steady popularity at 4.8%
• GPT-4 is the only victim of Claude's success, as its usage dropped from 30.5% to 16.5%

How should we interpret Llama 3's score?

• Llama 3 was selected as the default LLM in the study (see methodology below). Therefore, the 45% & 47% can’t be considered an unbiased metric to evaluate the popularity of Llama 3. It is most likely overestimated.
• However, the outstanding speed of Llama 3 powered by Groq makes it a serious contender for the crown of the most popular AI model. (The Mammouth team selected it as the default LLM because it was their personal favorite).
• The team is considering reviewing the default setup on Mammouth (the interface where the study was conducted) to provide an accurate figure for Llama 3 in our next study.

Will this popularity remain constant or evolve in the future?

To keep in touch with our next study, follow Mammouth on LinkedIn or X.