Mistral Launches 8X22B Mixture of Experts AI Model in Open Source, Benchmarks Reveal Big Upgrades

Mistral released its latest artificial intelligence (AI) model, 8X22B, on Wednesday. The mixture-of-expert AI model follows the 8X7B model released in December 2023 and offers a larger parameter size. The company known for its fully open-source AI models released the latest model unconventionally, without any announcement posts or blog posts accompanying it. While there are no benchmarks provided by the AI firm itself, the Hugging Face community users tested it and posted benchmark scores of the 8X22B model. The results appear to close the gap with the closed models from OpenAI and Google.

The official X (formerly known as Twitter) account of Mistral released the 8X22B AI model via a torrent magnet link, continuing its unconventional method of dropping AI models without an announcement and directly for people to download. Mistral is also one of the only truly open-source platforms that not only makes the weights open but also the entire architecture. However, it should be noted that most devices are not equipped to run powerful AI models on-device, and it might cause the device to malfunction. The total file size is 262GB.

Mistral's 8X22B is an autocomplete AI model. These are typically different from instruct or chat variants of AI models. OpenAI's ChatGPT and Google's Gemini AI are chat models, whereas Meta's Code Llama 7B and 13B are instruct models. Chat models are those which can understand natural language and contextual queries to provide the correct response. Instruct models are mainly used by developers who ask the AI model to perform a specific task. In contrast, an autocomplete model completes the sentence that has been provided in the prompt.

Early benchmark scores based on a Hugging Face community post show that the latest Mistral model offers significant upgrades over its predecessors and closes the gap with other major models. In the Hellaswag benchmark, Mistral 8X22B scored 88.9, closely trailing GPT-4 at 95.3, Claude 3 Opus at 95.4, and Gemini 1.5 Pro at 92.5. However, it outscores GPT-3.5 at 85.5 and Gemini 1.0 Ultra at 87.8.

Comparing the benchmarks with Mistral's own models, the 8X22B falls between Mistral-medium and Mistral-large models. With a later release of an instruct or chat variant, it might even fare similarly to Mistral-large.