AI Builder – Form Processing Layouts

Microsoft introduced a small but important change to AI Builder last week. Prior to this, a form processing model could only be trained to extract data from documents with the same layout. This was somewhat cumbersome when dealing with documents such as invoices where the layout varies from supplier to supplier.

What is Form Processing?

Form Processing in AI Builder allows you to create a model to extract data from documents such as PDFs or images. The model is trained to recognize and extract pieces of text from specific locations in a document. You train the model by uploading five sample documents and highlighting the data you want to be extracted. All the AI Builder functionality in the Power Platform is built on Azure empowering anybody to take advantage of AI capabilities.

How to use Form Processing

Step 1: Create an AI Model

From the AI Builder menu select Form Processing and give your new AI model a name.

Step 2: Enter the fields to be extracted

The next step is to enter the name of the fields you want to extract from the document. There are no field data types as all the data is extracted as text. You can convert the fields after extraction as required.

Step 3: Upload Document Collections

You can now upload a sample of five documents for each of the different layouts you want to process. You can also name each collection to help track which invoice layout type the collection relates to.

Step 4: Tagging Documents

When you move on to the next step AI Builder it will take a little time to analyze your collections of documents. Once this is completed you need to train the model by viewing each document in turn, highlighting the text to be extracted, and selecting the field it relates to. As you move through the documents in a collection AI Builder begins to learn how to extract the fields. After you have identified the fields in one or two documents you should find you only need to confirm it has identified the fields correctly. You cannot add fields at this stage so you will need to return to step 2 if you need to add or delete fields.

Step 5: Missing Fields/Tags

If a document is missing a field you can flag the field as unavailable. I would recommend uploading documents that have all the required tags to make the training easier.

Step 6: Train your Model

The final step is to submit your model for training.

Step 7: Quick Test

Once your model is trained it will appear as published and you run a quick test on it by uploading a sample document. It will process the document and display its confidence scoring for each field identified.

Conclusion

Once published your AI model can be used from either Power Apps or Power Automate. Invoice processing is a typical usage scenario for a form processing model with a variety of layouts. It’s easy to create a Power Automate flow to use your AI Model to grab email attachments, extract the invoice fields, and use the fields to update a back end system. Invoices from different suppliers in a variety of formats can now be processed by the same AI model. There is no longer a need to create multiple AI Models and Power Automate flows to process your invoices. Hopefully, this post shows how this AI Builder improvement can help with generic form processing.