Voice Mode AI Assistant on Apple Watch Through FaceTime.

Author(s): Evgeny Basisty

Originally published on Towards AI.

Recently, I’ve begun using AI assistants with voice mode, and I find them incredibly useful. I often use my Apple Watch with an eSIM during walks or workouts outdoors. In such situations, I usually don’t carry my iPhone because it adds unnecessary weight.

I attempted to locate an Apple Watch app that could offer voice mode for any of the major LLM providers, but unfortunately, I couldn’t find any. There are some apps specifically designed for Apple Watch, but they can be quite challenging to use. Ultimately, most of these apps require you to type on a tiny watch keyboard, which can be quite difficult. I even attempted to use Ant Browser and open Chat GPT or Perplexity. The sites would load, but I couldn’t figure out how to start the voice assistant. I’m certain that dedicated watch applications from major LLMs will soon be available on Apple Watches, but watch apps generally aren’t a top priority for developers due to their smaller user base. If you’re using any WatchOS app that provides true voice mode interaction, so we can speak to LLMs just like we would with a real person, please let me know in the comments.

So, I decided to find a way to interact with Perplexity AI voice assistant using my Apple Watch. From my experience, the app that works best for voice transmission on Apple Watch is FaceTime. Is it possible to call Perplexity using FaceTime?

Turns out, it’s easy, but you need a Mac with LoopBack installed. LoopBack offers powerful features for creating virtual audio devices and routing audio channels. Although it’s not free, the trial mode is sufficient for our purposes.

First, create a new user on your Mac. You’ll also need to create a new Apple ID to enable calls from your watch user to this new Mac user.

Install LoopBack from here. You’ll need to create two virtual devices in the app:

1. Facetime:

2. Safari:

This is it. At this point you can already test how your AI assistant works. Make a call from your Mac to your Watch. In FaceTime make sure that you selected Safari as your microphone sound source and FaceTime as your speaker:

Open Perplexity https://www.perplexity.ai/on your Mac in Safari and start voice mode:

Now try speaking to your watch. Perplexity should respond to you as if you were talking to it through microphone.

Of course, this setup doesn’t make sense if you’re away from your Mac and can’t make a call from it. You’d want to call your Mac from your watch and initiate a conversation with Perplexity. After some trial and error, I managed to force my Mac to accept incoming FaceTime calls and start Perplexity in Safari. I’ve published a script on GitHub that solves this problem.

Download the script and run it in the Terminal of your Mac:

python3 vai.py 

The script waits for FaceTime to start calling. When the call is detected the script starts Safari, opens Perplexity web site and launches voice mode. After that the script accepts the call by pressing space. This only works if Full keyboard Access is enabled in your Mac Settings:

The script relies on hardcoded delays which you might need to adjust depending on your system. I’ve configured the delays for my Mac mini M4.

You also need to grant microphone permissions to Perplexity in Safari Settings:

When you run the script you would want to prevent your Mac from going to sleep in the settings:

Now as long as script is running you can call your Mac and it will connect you to the voice assistant. Of course you can modify the script a little bit and use another voice assistant instead of Perplexity. As soon as you finish your call the script will kill Safari and FaceTime applications. The script will be ready to accept new calls in a few seconds.

Hope you will find this mode of interaction with LLMs useful in some scenarios! For myself I noticed that I started using voice mode much more often with this setup. The latency is ok and the experience is comparable to the experience that you get when you lunch the assistant from your phone. One major drawback of the method is that you cannot clear context window — start a new conversation and ai assistant gets confused when you jump from topic to topic. To start a new chat you will have to end the call and call the assistant again.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI