Using ARKit to Develop Mobile Apps Powered by AI

Last Updated on May 15, 2020 by Editorial Team

Author(s): Victoria M.

Programming

The creation and development of mobile applications is a large and rapidly developing industry. In the past few years, it has been modified significantly due to the introduction of novel futuristic technologies. Augmented reality and AI have been around for quite some time, and now it’s time to apply them in the mobile app creation.

What is ARKit?

ARKit is Apple’s development tool for the creation of AR apps. For this technology to work, Apple uses the device’s built-in camera, its processor, and some of its sensors to analyze the environment. The device automatically detects horizontal surfaces and allows users to place suitable objects on them. Apple also takes care of all the calculations associated with rendering shadows and all movements when moving the camera.

To enable augmented reality, ARkit uses a processor (a separate subsystem is allocated in the new A11 Bionic), an integrated device camera, and a set of sensors for analyzing the surrounding space.

Now the iPhones and iPads will be able to use the data set coming from cameras, gyroscopes, accelerometers, and other sensors to determine their location and position in the current situation, taking into account the objects surrounding the device. One of the critical applications for all this data is to find flat objects or lines suitable for the current task on which information or a virtual object can be placed.

The platform can recognize the dimensions of the surrounding space and take into account the lighting conditions to integrate virtual objects into real life as reliably as possible.

Besides, Apple now has the fastest and most economical mobile processors, and ARKit can get full access to their resources. Other feature sets, such as Metal and SceneKit, can also be used to build scenes. Compatibility with the vast majority of modern iOS devices makes ARKit the most massive augmented reality platform to use for mobile app development.

Building Mobile Apps with ARKit

Apple ARKit’s main function is to provide a set of tools to application developers so that they can use augmented reality (AR) in the process of mobile app creation. Regardless of whether it is planned to use the created application for entertainment or practical tasks, the tools used are the same:

• device cameras receive a visual image;
• the microphone and other sensors get sound, read information about vibrations and other changes;
• the processor analyzes the incoming data and coordinates it with the commands that are submitted directly from the application.

Equipped with a set of tools, the AR iPhone and iPad now can independently detect horizontal and vertical surfaces, determine light sources and shadows, distinguish voices and faces, and much more. The latest processors will fully cope with the use of incoming information and create augmented reality.

The possibilities of the new toolkit include:

1. free navigation on maps and obtaining information about the world;
2. measuring distances and other indicators;
3. three-dimensional drawing and modeling;
4. simulators of a ride, flight, other processes;
5. entertaining and gaming applications, one way or another based on interaction with the real world.

For example, let’s take a look at the process of placing a 3D object in the AR app built on ARKIt. It only takes several simple steps. The first move is setting up a scene and running it in theviewWillAppear method:

func setupScene() {
    let scene = SCNScene()
    sceneView.scene = scene
}
    
func setupConfiguration() {
    let configuration = ARWorldTrackingConfiguration()
    sceneView.session.run(configuration)
}

Next, you need to add a 3D object to the scene (let’s call it “Object”):

class Object: SCNNode {
    func loadModel() {
        guard let virtualObjectScene = SCNScene(named: "Object.scn") else { return }
        let wrapperNode = SCNNode()
        for child in virtualObjectScene.rootNode.childNodes {
            wrapperNode.addChildNode(child)
        }
        addChildNode(wrapperNode)
    }
}

Once we configure the SCNNode, lets’ start the initialization an object of the Object class and supplement it after setting up configuration:

func addObject() {
    object.loadModel()
    sceneView.scene.rootNode.addChildNode(object)
}

Combining AI and AR with ARKit

Artificial Intelligence has been a subject of interest to many developers. Whether you are developing a personal assistant, a program for computing, or just a game, introducing this novel technology into your project will make it modern, engaging, and definitely in demand. Leveraging AI in mobile app development, you can also increase data security, improve customer service, and win the ever-increasing competition between application developers.

The most apparent method to merge AR and AI designs is to capture pictures or sound from a scene, operate that information in an AI model, and utilize that result to impose a particular effect. For instance, consider the following patterns:

• Tagging an image oDr a scene: you can process the camera frames by running it through an AI model that will analyze/classify the picture. Once the image has been classified, the location will get its AR-tag.
• Target exposure: A camera frame is transferred to an AI model that determines the situation and size of an object. Location data is later utilized to make hitboxes that ease the interaction of real and digital objects.
• Position evaluation: the AI model determines the poses of the objects utilized to manage AR content.
• Text identification and interpretation: An AI model can recognize, read, and translate content. AR can then transfer it to the 3D environment.
• Sound identification: the AI model listens to certain words that urge the AR effect. For instance, if you say the word “Hat” then a virtual hat will appear on your head.

Final Word

As you can see, with modern tools such as ARKit, building an engaging mobile app is turning into an easy and enjoyable endeavor. The combination of a well-chosen direction of development and your passion for your project will undoubtedly lead to the creation of a successful and unique product.

---

Using ARKit to Develop Mobile Apps Powered by AI was originally published in Towards AI — Multidisciplinary Science Journal on Medium, where people are continuing the conversation by highlighting and responding to this story.

Published via Towards AI