Click, Scroll, Type — The AI Evolution

Author(s): Myra Roldan

Originally published on Towards AI.

Anthropic’s latest update to their AI models, Claude 3.5 Sonnet and Claude 3.5 Haiku, has introduced a new feature called “computer use.” An interesting choice of name for the service and a bit confusing on how that may work. Now, Imagine an AI tool that can see your screen and do thinks like write documents and switch between applications screens on its own without you needing to guide it. Sounds like a Ghost in the Machine type situation, minus the serial killer. Maybe this sounds just a little too futuristic for many? For others, this breakthrough is a brave new world, while it’s leading others to raise an eyebrow.

The new “computer use” feature, which is in beta right now, lets developers direct Claude to interact with computers just like a human. As of the writing of this article, this capability is still in its early experimental stages. According the a post on the Anthropic site, computer use is sometimes clunky, and has a few quirks; but, the potential is different than other Generative AI tools.

Think of computer use like giving Claude a virtual set of eyes and hands that knows how to interact with a computer just like a human. So, instead of just processing data or generating text behind the scenes like we’ve gotten used to; Claude can now interact with software like a human does. Yes, that means Claude can navigate apps, click buttons, scroll around the screen, and type text. Imagine an AI that can handle a task requiring dozens (or hundreds) of steps to complete without your guidance. While this is a groundbreaking shift in how AI can assist with complex workflows, it also introduces new questions about security, privacy, and the AI’s reliability.

Computer use is an evolution of Generative AI’s that is a game-changer in the making. But, let’s set expectations. The computer use capability is still in its early stages, currently available in public beta, and it’s far from the plug-and-play simplicity we’ve become used to with Generative AI. For now, computer use is designed for software developers with the technical know-how and programming skills to create automations, build and test software, and conducted open-ended tasks. It’s powerful but experimental, and not something the average end-user can immediately dive into without technical skills. That said, the potential for businesses that have the in-house software development talent and are ready to invest in experimentation with computer use can be substantial.

Right now, the practical applications are for more straightforward processes, but as computer use matures, we can imagine it being able to handle increasingly nuanced tasks. What would this look like? Let’s put on our futures thinking caps to image Claude being able to not only pull sales numbers into reports but intuitively follow multi-step workflows that require judgment, adaptiveness, or even a bit of creativity. Over time, this could evolve into a capability that helps businesses with real-time analysis, multi-system integrations, or complex support tasks that previously required human hands on keyboards.

But remember, with great power, comes great responsibility — cliche but true. The computer use feature is designed to allow AI to interact directly with your laptop, desktop, or servers where it will have access to potentially sensitive business software and data. The first words that come to my mind are the iconic warnings of Blinky from the 70’s series Lost in Space, “Danger Will Robinson, Danger!” There are inherent risks, like data security, potential misuse, potential programming errors, and even operational disruptions if the AI interactions aren’t well-monitored. Anthropic says that it integrated safety measures into computer use and is refining the tech to make it safer and more reliable as it evolves. At the end of day, its still a system that suffers from the black box problem meaning we can’t always see how these powerful models make their decisions, making it tough to explain their results. So, I recommended, early adopters should start with lower-risk experiments and be prepared for the computer use feature’s imperfections.

Looking ahead, the implications can be huge. For businesses, this means that AI might not just augment human work but start handling those tedious processes that aren’t the best use of your employees time. We’re witnessing the beginning of Generative AI’s evolution, which makes it difficult to predict, “what’s next?” Claude’s computer use feature could be the the start of a new relationship between AI, computers, humans, and business operations. It’s an exciting time, and as we step into this frontier, the question isn’t just what Claude can do today but will companies use this Generative AI evolution to scale and grow their businesses.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI