Microsoft Phi-3.5 Open-Source AI Models Released, Claimed to Outperform Gemini 1.5 Flash, GPT-4o Mini

Microsoft has released Phi-3.5 MoE, Mini, and Vision AI models.

Advertisement
Written by Akash Dutta, Edited by Siddharth Suvarna | Updated: 21 August 2024 13:18 IST
Highlights
  • All three AI models are available on Hugging Face
  • The Phi-3.5 MoE supports 1,28,000 tokens context length
  • The Phi-3.5 Vision has 4.2B parameters and an image encoder
Microsoft Phi-3.5 Open-Source AI Models Released, Claimed to Outperform Gemini 1.5 Flash, GPT-4o Mini

Running the Microsoft Phi-3.5 MoE model was trained on the Nvidia H100 80GB GPU

Photo Credit: Reuters

Microsoft released the family of Phi-3.5 artificial intelligence (AI) models on Tuesday, as the successor of the Phi-3 models introduced in April. The new release comprises Phi-3.5 Mixture of Experts (MoE), Phi-3.5 Vision, and Phi-3.5 Mini models. These are instruct models, so they will not work as a typical conversational AI but will require users to add specific instructions to get the desired output. The open-source AI models are available to download from the tech giant's Hugging Face listings.

Microsoft Releases Phi-3.5 AI Models

The release of the new AI models was announced by Microsoft executive Weizhu Chen in a post on X (formerly known as Twitter). The Phi-3.5 models offer upgraded capabilities over the predecessor, but the architecture, dataset and training methods largely remain the same. The Mini model has been updated with multilingual support, and the MoE and Vision models are new inclusions in the AI model family.

Coming to technicalities, the Phi-3.5 Mini has 3.8 billion parameters. It uses the same tokeniser (a tool that breaks down text into smaller units) and a dense decoder-only transformer. The model only supports text as input and supports a context window of 1,28,000 tokens. The company claims it was trained using 3.4 trillion tokens between June and August, and its knowledge cut-off is October 2023.

One key highlight of this model is that it now supports multiple new languages including Arabic, Chinese, Czech, Danish, Dutch, English, Finnish, French, German, Hebrew, Hungarian, Italian, Japanese, Korean, Norwegian, Polish, Portuguese, Russian, Spanish, Swedish, Thai, Turkish, and Ukrainian.

Advertisement

The Phi-3.5 Vision AI model has 4.2 billion parameters and it includes an image encoder that allows it to process information within an image. With the same context length as the Mini model, it accepts both text and images as input. It was trained between July and August on 500 billion tokens of data and has a text knowledge cutoff of March.

Finally, the Phi-3.5 MoE AI model has 16x3.8 billion parameters. However, only 6.6 billion of them are active parameters when using two experts. Notably, MoE is a technique where multiple models (experts) are trained independently and then combined to improve the accuracy and efficiency of the model. This model was trained on 4.9 trillion tokens of data between April and August, and it has a knowledge cutoff date of October 2023.

Advertisement

On performance, Microsoft shared benchmark scores of all of the individual models, and based on the data shared, the Phi-3.5 MoE outperforms both Gemini 1.5 Flash and GPT-4o mini in the SQuALITY benchmark which tests the readability and accuracy when summarising a long block of text. This tests the long context window of the AI model.

However, it should be mentioned that it is not a fair comparison since MoE models use a different architecture and require more storage space and more sophisticated hardware to run. Separately, the Phi-3.5 Mini and Vision models have also outperformed relevant competing AI models in the same segment in some metrics.

Advertisement

Those interested in trying out the Phi-3.5 AI models can access them via Hugging Face listings. Microsoft said that these models use flash attention which will require users to run the systems on advanced GPUs. The company has tested them on Nvidia A100, A6000, and H100 GPUs.

 

For the latest tech news and reviews, follow Gadgets 360 on X, Facebook, WhatsApp, Threads and Google News. For the latest videos on gadgets and tech, subscribe to our YouTube channel. If you want to know everything about top influencers, follow our in-house Who'sThat360 on Instagram and YouTube.

Advertisement

Related Stories

Popular Mobile Brands
  1. Nothing Phone 3 to Get New Glyph Matrix Interface on the Rear Panel
  2. Poco F7 5G to Be Equipped With a Snapdragon 8s Gen 4 SoC
  3. Oppo Reno 14 5G Series Teased to Launch in India Soon
  4. Samsung Galaxy M36 5G India Launch Date and Key Features Revealed
  1. Fast Radio Bursts Reveal Universe’s Missing Matter Hidden in Cosmic Intergalactic Fog
  2. Apollo Astronauts Found Orange Glass Beads on the Moon, Scientists Now Know Why
  3. World’s Oldest Tailored Dress Found in Egyptian Tomb Dates Back Over 5,000 Years
  4. Ancient Footprints in White Sands Confirm Humans Reached America 23,000 Years Ago
  5. Humanoid Robot Achieves Controlled Flight Using Jet Propulsion and AI Systems
  6. Curiosity Rover Reaches Uyuni Quad, Begins New Mars Mapping and Surface Analysis Campaign
  7. NASA to Gather Reentry Imagery of European Commercial Capsule Using High-Altitude Aircraft
  8. ESA's Proba-3 Unveils First-Ever Artificial Solar Eclipse Images from Precision Satellite Formation
  9. My Hero Academia Final Season OTT Release Date Revealed: Everything You Need to Know
  10. NASA Study Reveals Correlation Between Earth’s Magnetic Field and Atmospheric Oxygen
Gadgets 360 is available in
Download Our Apps
Available in Hindi
© Copyright Red Pixels Ventures Limited 2025. All rights reserved.