Small and Cool: Inside Microsoft’s New Phi-4-Mini and Phi-4-Multimodal

Author(s): Jesus Rodriguez

Originally published on Towards AI.

I recently started an AI-focused educational newsletter, that already has over 175,000 subscribers. TheSequence is a no-BS (meaning no hype, no news, etc) ML-oriented newsletter that takes 5 minutes to read. The goal is to keep you up to date with machine learning projects, research papers, and concepts. Please give it a try by subscribing below:

Introduction The Phi-4 series represents a significant advancement in small language models (SLMs), demonstrating that carefully curated and synthesized data can enable highly competitive performance despite a smaller number of parameters. Building on the success of the Phi family, Microsoft has introduced Phi-4-Mini and Phi-4-Multimodal, extending their capabilities to handle vision and audio modalities. Phi-4-Mini is a 3.8-billion-parameter language model that excels in multilingual support, reasoning, and mathematics, with the added functionality of function calling. Phi-4-Multimodal is a multimodal model integrating text, vision, and speech/audio inputs. These models can be deployed on edge devices, enabling generative AI in resource-constrained environments.

Key Capabilities Phi-4-Mini

• Enhanced Multilingual Support: Phi-4-Mini features an expanded vocabulary size of 200K tokens to better support multilingual applications.
• Reasoning and Mathematics: The model exhibits strong reasoning and mathematical capabilities, outperforming open-source models of similar size and matching the performance of models twice its size.
• Function Calling: Phi-4-Mini supports function calling, extending its text-processing capabilities by integrating search engines and connecting various tools.

Phi-4-Multimodal

• Unified Multimodal Support: Phi-4-Multimodal efficiently handles multiple modalities (text, images, speech/audio) within a single model checkpoint.
• Vision and Language Understanding: The model demonstrates significant improvements in vision-language understanding, surpassing previous models and outperforming baseline models of similar sizes. It excels in chart understanding and science reasoning tasks.
• Speech and Audio Performance: Phi-4-Multimodal achieves strong performance in multilingual speech recognition and translation tasks and is the first open-sourced model with speech summarization capability. It surpasses expert ASR models and exhibits competitive performance in speech summarization compared to larger models.
• Advanced Reasoning: Retains strong reasoning capabilities, even with a reduced parameter size. Can generate structured project code based on image content and prompts.

Model Architecture Language Model Architecture Phi-4-Mini and Phi-4-Multimodal share the same language model backbone, consisting of 32 Transformer layers with a hidden state size of 3,072. The models utilize the tiktoken tokenizer with a vocabulary size of 200,064, supporting multilingual and multimodal input and output. The architecture is based on a decoder-only Transformer and supports a 128K context length based on LongRoPE. Key architectural features include:

• Group Query Attention (GQA): Optimizes key and value memory (KV cache) usage for long-context generation. The model employs 24 query heads and 8 key/value heads, reducing KV cache consumption to one-third of its standard size.
• Fractional RoPE Dimension: Ensures that 25% of the attention head dimension remains position-agnostic, supporting smoother handling of longer contexts.

Multimodal Architecture Phi-4-Multimodal employs a novel “mixture of LoRAs” technique, integrating modality-specific LoRAs while keeping the base language model entirely frozen. This approach allows multiple inference modes combining various modalities without interference. The architecture includes:

• Vision Modality:
• A pre-trained vision encoder processes image inputs.
• A vision projector (2-layer MLP) maps vision features to the text embedding dimension.
• A LoRA adapter is added to all linear layers in the language decoder and deployed in the supervised fine-tuning (SFT) stage.
• Speech and Audio Modality:
• A pre-trained audio encoder consists of convolution layers and conformer blocks.
• An audio projector (2-layer MLP) maps speech features to the text embedding space.
• LoRA is applied to all attention and MLP layers in Phi-4-Mini to improve speech/audio benchmark performance while preserving text capability.

Innovations Mixture of LoRAs The “mixture of LoRAs” technique is a key innovation in Phi-4-Multimodal. Different LoRAs are trained to handle interactions between different modalities. This approach enables multimodal capabilities while minimizing interference between modalities. The design is highly extensible, allowing seamless integration of new LoRAs to support additional modalities without impacting existing ones.

Training Data and Methods

• High-Quality Data: Both Phi-4-Mini and Phi-4-Multimodal are trained on high-quality web and synthetic data. The synthetic data recipe emphasizes high-quality math and coding datasets for Phi-4-Mini.
• Data Filtering: Improved data filtering techniques are used to enhance the quality of the pre-training data. An enhanced quality classifier, trained on a larger curated dataset, leads to better filtering quality across multiple languages.
• Reasoning Training: The reasoning-optimized version of Phi-4-Mini undergoes a three-stage training process:

1. Pre-training on extensive reasoning data.
2. Fine-tuning on a smaller, carefully curated dataset.
3. DPO training using preference samples.

Training Pipeline The multimodal training pipeline includes vision training, speech/audio training, and vision-speech joint training.

• Vision Training: Involves pre-training with interleaved image-text documents and SFT using a combination of text SFT datasets and multimodal instruction tuning datasets.
• Speech and Audio Training: A two-stage paradigm includes pre-training with ASR data to align the audio encoder and Phi-4-Mini, followed by post-training with curated speech and audio SFT samples.
• Vision-Speech Joint Training: Involves fine-tuning the vision adapter LoRA, vision encoder, and vision projector while freezing the language base model, audio encoder, and audio projector.

Performance Language Benchmarks Phi-4-Mini demonstrates strong performance across different language understanding benchmarks, outperforming similar-sized models and performing on par with models twice its size. It excels in math and reasoning-related benchmarks.

Coding Benchmarks Phi-4-Mini exhibits strong coding performance, outperforming most 3B and 8B models on various coding benchmarks.

Multimodal Benchmarks

• Vision-Language: Phi-4-Multimodal demonstrates significant improvements over previous models and outperforms baseline models of similar sizes. It surpasses some closed-source models in chart understanding and science reasoning tasks.
• Vision-Speech: Phi-4-Multimodal significantly outperforms InternOmni and Gemini-2.0-Flash on vision-speech benchmarks.
• Speech and Audio: Phi-4-Multimodal achieves very strong ASR and AST performance, surpassing expert models. It is the first open-sourced model with speech summarization capability.

Safety Phi-4-Mini and Phi-4-Multimodal were developed in accordance with Microsoft’s responsible AI principles. The approach included safety alignment in post-training, red-teaming, and automated testing. Systematic safety evaluations were carried out using Microsoft’s Azure AI Evaluation SDK. The models demonstrate robustness to jailbreaks and are effective at refusing to answer harmful prompts.

Weaknesses and Limitations

• Model Size: The model’s size limits its ability to remember specific facts.
• Multilingual Capability: Multilingual capability is limited by the number of model parameters.
• Potential for Undesirable Content: Like other models, Phi-4-Mini and Phi-4-Multimodal can sometimes output undesirable content.
• Not Designed for Biometric Categorization: Phi-4-Multimodal is not designed to be used as a biometric categorization system.

Conclusion Phi-4-Mini and Phi-4-Multimodal represent significant advancements in small language models, showcasing the potential of carefully curated data and innovative architectural designs. With strong capabilities in language understanding, reasoning, coding, and multimodal processing, these models offer a compelling solution for various applications, particularly in resource-constrained environments. The “mixture of LoRAs” technique and the emphasis on high-quality training data are key innovations that contribute to the performance and versatility of the Phi-4 series. These models are well-aligned with responsible AI principles, incorporating safety measures to mitigate potential risks. As SLMs continue to evolve, the Phi-4 series exemplifies the importance of data quality, architectural innovation, and safety considerations in achieving high performance with limited resources.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI