Introduction to Multimodality With LLaVA

Author(s): Marcello Politi

Originally published on Towards AI.

Learn how to implement multimodal AI on low-resource hardware

In the last couple of years, I have worked mainly with large language models, training, fine-tuning, prompting and so on, since this was highly requested in the market and by users. But I believe that LLMs that work mainly on text is only the beginning of GenAI. At a certain point, everybody will want physical AI, where models can see, hear, feel, and reason in a more grounded, human way.

This article introduces the concept of multimodality and focuses on implementing the LLaVA model, which integrates both images and text for generating responses. It explains how to set up the environment, download the necessary components, and fine-tune the model for effective use in low-resource scenarios. It also discusses specific instructions on installing required libraries, processing images and text, and structuring the training data for the LLaVA model, guiding readers through practical coding examples and offering insights on improving model efficiency and performance.

Read the full blog for free on Medium.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI