Getting Started with Form Recognizer — Labeling Tool

Author(s): Dayo Bamikole

Originally published on Towards AI.

TLDR

• Background — Story of needing Form Recognizer
• Overview — About Form Recognizer
• How does it work / What is the labeling tool
• How to Deploy the Labeling Tool

Background

So, at some point in life, we all have had to fill out a form either physically or online. From the front-end as a customer or responder, that is the end of the process, but for the back-end as a business or reviewer, a lot more goes into the process. Typically, the responses from these forms need to be extracted and analyzed for making sound business decisions, or even personal choices if you are like me. (Fun fact: I take pictures of my receipts, so I can go back to them to assess my spending habits and be more financially-responsible). Forms are documents that come in different formats e.g., PDFs, JPGs, PNG, or TIFF. The text in these documents can become difficult to work with if they cannot be extracted, searched upon, or appropriately analyzed. To the rescue comes in Form Recognizer!

Overview

Form Recognizer is part of Microsoft Azure Cognitive Service. What is Cognitive Services? It is pre-built AI features that can be easily used via an API call. They can be split into 5 categories: Decision, Language, Speech, Vision, and Web Search. Check out the different features in each of these categories here. Form Recognizer falls under the vision category, and it’s’s goal is to

extract text, key-value pairs, and tables from documents

Here is an example of key-value pair extraction from a receipt

You also get the form extraction in JSON format, which can be easily consumed by different databases (e.g., NoSQL) or tools that use JSON format.

Hopefully, by now, you’re excited about the possibilities of Form Recognizer. The next question is, how do I get this working.

Pre-requisite

The only pre-requisite is to have an Azure Subscription.

How does it work?

How does Form Recognizer work? According to Azure Docs, it uses machine learning technology to extract data from the documents. As a result, it needs to at least 5 forms to create a model that subsequent forms can use as a reference point for labeling purposes (for more details, please visit the Forms Recognizer Document here). The tool used to make this happen called the Form Recognizer Labeling Tool. This is what I will like to focus on for the rest of the blog. I will show you the most efficient way to set it up such that you can use it for all your document extraction.

The tool is available via Docker. You can install Docker on your laptop or a VM (Virtual Machine). This is done with any operating system: Linux, Windows, or macOS.

While this is an excellent method, I prefer to use a PaaS approach to install the labeling tool because it guarantees that my labeling tool will always be available whenever without having to rely on my laptop or VM to be on.

To deploy the labeling tool using a PaaS approach, we will be an Azure Web App.

Deploy the Labeling Tool

Head over to the Azure Portal

1. Select “Create a Resource”
2. Select “Web App”
3. On the Basics Tab. Fill in the following information:

• Resource Group: DemoFormRecognizer
• Name: “”type a unique name””
• Publish: Docker Container
• Region: Central US
• SKU and size: “_choose Dev/Test for a cheaper or free size _.”

4. On the Docker Tab. Fill in the following information:

• Image Source: Docker Hub
• Image and Tag: mcr.microsoft.com/azure-cognitive-services/custom-form/labeltool
• Startup Command: docker run -it -p 3000:80 mcr.microsoft.com/azure-cognitive-services/custom-form/labeltool eula=accept

5. Select Review + Create

6. Access the tool by going to the URL

Final Thoughts

I hope this helped deploy the Form Recognizer Labeling Tool. Now you can begin to label your forms for training and testing purposes. The next steps will be to follow the instructions in the Microsoft Docs below to Create a Project in the Labeling Tool. You will need to have provisioned a Form Recognizer resource in Azure as well as an Azure Storage account.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI