Coupling Four AI Models to Deliver the Ultimate Experience in Immersive Visualization and Modeling

Author(s): LucianoSphere (Luciano Abriata, PhD)

Originally published on Towards AI.

Read on about how this new free app called HandMol achieves immersive, collaborative visualization and modeling of molecular structures by integrating 6 modern computer technologies -all in the web and multi-device
HandMol integrates WebXR, AI-powered hand tracking native to the VR headset, AI-powered speech recognition assisted by an AI large language model, AI-based molecular mechanics, and WebRTC to allow immersive two-user interaction in molecular graphics and modeling. Figure composed by the author from snapshots taken in his own software.

In this new blog entry, I describe how we created a web-based tool called HandMol that integrates various AI-based tools and technologies to enable one or more users to view and manipulate molecular systems in immersive 3D with their hands and assisted by orally requested commands and molecular mechanics. My point is not to describe HandMol’s power for chemists (which is covered in dedicated preprint) but to explain how it blends six very modern computer technologies, four of them AI-based: WebXR, AI-powered hand tracking, AI-powered speech recognition assisted by a large language model, AI systems for quick and accurate simulation of molecular physics, and WebRTC for communication between multiple users. Moreover, the whole program runs entirely on the web, more specifically on the client, which has multiple advantages, as discussed.Beyond the many technical explanations you will find in this article, I hope it will achieve its broader goal of inspiring a similar integration… Read the full blog for free on Medium.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI