Extracting and Parsing Multi-Receipts Tutorial
Overview
When to use this?
It may be a common occurrence for you or your users that taking one picture of a bunch of receipts at once might be way faster than taking a bunch of different pictures. In one such case, you may find that it may be tough to split the ensuing picture into sub-files before sending them to the Receipt API. No worries, the Multi-Receipts Detector API takes care of that.
The underlying principle is simple, an AI scans the contents of a file and identifies the coordinates of the receipts it sees.
Those coordinates then can be used to extract those receipts from the initial picture into their own files.
Note: for PDFs the principle applies at a page-level: i.e. each page will be treated as a separate picture. The results can be retrieved either all at once or on a page-by-page basis.
Initial Setup
The goal of this tutorial is to use the Multi-Receipts Detector API to isolate receipts and then parse them on the Receipts OCR.
To do this, we'll be using the following dummy file:
Before uploading, be sure that you have taken the following into account:
- Receipts are clear, unstained and properly unfolded
- Receipts don't overlap, and they are not spilling out of the picture
- Receipts are aligned in roughly the same direction
- No other type of documents are mixed in with the receipts
The Multi-Receipts Detector API is quite robust, but it will not distinguish between receipts and other similar pieces of paper if you mistakenly add some.
Subscribing to the Multi-Receipts API
For this tutorial, you'll need to be in possession of a valid and up-to-date API key( and a valid subscription to the Multi-Receipts API as well as the Receipt OCR.
If you aren't sure whether your subscription is enabled or not, go on the API page of the main interface, click on Utilities:
Then click on the Multi Receipts Detector button. That's it, your subscription to the API should be enabled.
To do the same for the receipts OCR, simply head to the Document Catalog:
Then, like in the previous step, click on our product of interest (Expense Receipt).
Note: be aware that you need to enable both subscriptions to get this whole tutorial working.
Calling the API
Time to get coding!
For this tutorial, we'll be using the official Mindee client library for Node.js.
Other programming languages are supported, check the list.
Let's create a directory to store our project and install the Mindee client:
mkdir multi_receipts_tutorial
cd multi_receipts_tutorial
npm install -s mindee
touch demo_multi_receipts.js
To get started with the code, we'll head on over to the Documentation page by using the link on the left of the interface.
From there we'll click on your language (NODEJS), then Select an API key, and finally use the copy code button:
Using your IDE, open the demo_multi_receipts file we created, and paste this code. We'll use it as a base.
Make sure the API key is correctly filled in, if not, simply grab one from the Mindee interface.
Find the line:
const inputSource = mindeeClient.docFromPath("/path/to/the/file.ext");
And put in a real file path on your drive. You should test this file in the live interface before checking.
Now run the code:
node demo_multi_receipts.js
The result should look something like this:
########
Document
########
:Mindee ID: d7c5b25f-e0d3-4491-af54-6183afa1aaab
:Filename: default_sample.jpg
Inference
#########
:Product: mindee/multi_receipts_detector v1.0
:Rotation applied: Yes
Prediction
==========
:List of Receipts: Polygon with 4 points.
Polygon with 4 points.
Polygon with 4 points.
Polygon with 4 points.
Polygon with 4 points.
Polygon with 4 points.
Page Predictions
================
Page 0
------
:List of Receipts: Polygon with 4 points.
Polygon with 4 points.
Polygon with 4 points.
Polygon with 4 points.
Polygon with 4 points.
Polygon with 4 points.
You'll notice that receipts only have a description of a polygon here. That is because this is only a summary of what's going on, but fret not, our receipts coordinates are of course accessible!
Image extraction code
The method to call to extract the images is called in the following way:
const { imageOperations } = require("mindee");
//...
const someResult = imageOperations.extractReceipts(myInputFile, myInferenceResult);
We'll add the method call to the response handling part of the script.
You should have something like this:
const { Client, imageOperations } = require("mindee");
const mindeeClient = new Client();
//... rest of the basic implementation
// Handle the response Promise
apiResponse.then((resp) => {
const someResult = imageOperations.extractReceipts(myInputFile, resp.document.inference);
someResult.then((extractionResp) => {
//some code...
});
});
This is not very pretty, though.
We can make this a bit more legible and usable by switching to something like this.
const { Client, product, imageOperations } = require("./../dist/src");
const { setTimeout } = require("node:timers/promises");
async function parseReceipts() {
// fill in your API key or add it as an environment variable
const mindeeClient = new Client();
const multiReceiptsFile = mindeeClient.docFromPath("path/to/your/file.ext");
const resp = await mindeeClient.parse(product.MultiReceiptsDetectorV1, multiReceiptsFile);
let receipts = await imageOperations.extractReceipts(multiReceiptsFile, resp.document.inference);
}
parseReceipts();
Parsing the Extracted Files
Now one last piece is missing: what to do with our receipts? Well, parse them, of course!
The code for this part is mostly up to you, but for this tutorial we'll use a simple loop to parse the documents and then display the results:
for (const receipt of receipts) {
const respReceipt = await mindeeClient.parse(product.ReceiptV5, receipt.asSource());
console.log(respReceipt.document.toString());
await setTimeout(1000); // wait some time between requests as to not overload the server
}
This should print something like this:
########
Document
########
:Mindee ID: 8506a82c-bc1f-4d5b-abd1-174f60459091
:Filename: receipt_p0_0.pdf
Inference
#########
:Product: mindee/expense_receipts v5.1
:Rotation applied: Yes
Prediction
==========
:Expense Locale: fr-FR; fr; FR; EUR;
:Purchase Category: miscellaneous
:Purchase Subcategory:
:Document Type: CREDIT CARD RECEIPT
:Purchase Date: 2022-08-24
:Purchase Time: 12:45
:Total Amount: 16.50
:Total Net:
:Total Tax:
:Tip and Gratuity:
:Taxes:
+---------------+--------+----------+---------------+
| Base | Code | Rate (%) | Amount |
+===============+========+==========+===============+
:Supplier Name:
:Supplier Company Registrations: 81187803200010
:Supplier Address:
:Supplier Phone Number:
:Line Items:
Page Predictions
================
Page 0
------
:Expense Locale: fr-FR; fr; FR; EUR;
:Purchase Category: miscellaneous
:Purchase Subcategory:
:Document Type: CREDIT CARD RECEIPT
:Purchase Date: 2022-08-24
:Purchase Time: 12:45
:Total Amount: 16.50
:Total Net:
:Total Tax:
:Tip and Gratuity:
:Taxes:
+---------------+--------+----------+---------------+
| Base | Code | Rate (%) | Amount |
+===============+========+==========+===============+
:Supplier Name:
:Supplier Company Registrations: 81187803200010
:Supplier Address:
:Supplier Phone Number:
:Line Items:
########
Document
########
:Mindee ID: f8e5b080-533e-43b6-b4bb-ef77208a74a3
:Filename: receipt_p0_1.pdf
Inference
#########
:Product: mindee/expense_receipts v5.1
:Rotation applied: Yes
Prediction
==========
:Expense Locale: fr-FR; fr; FR; EUR;
:Purchase Category: food
:Purchase Subcategory: restaurant
:Document Type: EXPENSE RECEIPT
:Purchase Date: 2022-08-25
:Purchase Time: 12:43
:Total Amount: 21.50
:Total Net: 19.55
:Total Tax: 1.95
:Tip and Gratuity:
:Taxes:
+---------------+--------+----------+---------------+
| Base | Code | Rate (%) | Amount |
+===============+========+==========+===============+
| 19.55 | TVA | 10.00 | 1.95 |
+---------------+--------+----------+---------------+
:Supplier Name: CHES
:Supplier Company Registrations: 87880027500017
:Supplier Address: 35 rue charlot 75003 paris
:Supplier Phone Number: 0100000000
:Line Items:
+--------------------------------------+----------+--------------+------------+
| Description | Quantity | Total Amount | Unit Price |
+======================================+==========+==============+============+
| Repas | 1.00 | 21.50 | |
+--------------------------------------+----------+--------------+------------+
Page Predictions
================
Page 0
------
:Expense Locale: fr-FR; fr; FR; EUR;
:Purchase Category: food
:Purchase Subcategory: restaurant
:Document Type: EXPENSE RECEIPT
:Purchase Date: 2022-08-25
:Purchase Time: 12:43
:Total Amount: 21.50
:Total Net: 19.55
:Total Tax: 1.95
:Tip and Gratuity:
:Taxes:
+---------------+--------+----------+---------------+
| Base | Code | Rate (%) | Amount |
+===============+========+==========+===============+
| 19.55 | TVA | 10.00 | 1.95 |
+---------------+--------+----------+---------------+
:Supplier Name: CHES
:Supplier Company Registrations: 87880027500017
:Supplier Address: 35 rue charlot 75003 paris
:Supplier Phone Number: 0100000000
:Line Items:
+--------------------------------------+----------+--------------+------------+
| Description | Quantity | Total Amount | Unit Price |
+======================================+==========+==============+============+
| Repas | 1.00 | 21.50 | |
+--------------------------------------+----------+--------------+------------+
########
Document
########
:Mindee ID: 6812060a-b9e4-4e45-8492-00aef14de36d
:Filename: receipt_p0_2.pdf
Inference
#########
:Product: mindee/expense_receipts v5.1
:Rotation applied: Yes
Prediction
==========
:Expense Locale: fr-FR; fr; FR; EUR;
:Purchase Category: food
:Purchase Subcategory: restaurant
:Document Type: EXPENSE RECEIPT
:Purchase Date: 2022-08-22
:Purchase Time: 12:27
:Total Amount: 13.50
:Total Net:
:Total Tax:
:Tip and Gratuity:
:Taxes:
+---------------+--------+----------+---------------+
| Base | Code | Rate (%) | Amount |
+===============+========+==========+===============+
:Supplier Name: LA CERISE SUR LA PIZZA
:Supplier Company Registrations: 51092131500063
:Supplier Address: 14 rue froissart 75003 paris france
:Supplier Phone Number:
:Line Items:
+--------------------------------------+----------+--------------+------------+
| Description | Quantity | Total Amount | Unit Price |
+======================================+==========+==============+============+
| EMPORTER LA CASTE | 1.00 | 11.50 | 11.50 |
+--------------------------------------+----------+--------------+------------+
| CANETTE SODA | 1.00 | 2.00 | 2.00 |
+--------------------------------------+----------+--------------+------------+
Page Predictions
================
Page 0
------
:Expense Locale: fr-FR; fr; FR; EUR;
:Purchase Category: food
:Purchase Subcategory: restaurant
:Document Type: EXPENSE RECEIPT
:Purchase Date: 2022-08-22
:Purchase Time: 12:27
:Total Amount: 13.50
:Total Net:
:Total Tax:
:Tip and Gratuity:
:Taxes:
+---------------+--------+----------+---------------+
| Base | Code | Rate (%) | Amount |
+===============+========+==========+===============+
:Supplier Name: LA CERISE SUR LA PIZZA
:Supplier Company Registrations: 51092131500063
:Supplier Address: 14 rue froissart 75003 paris france
:Supplier Phone Number:
:Line Items:
+--------------------------------------+----------+--------------+------------+
| Description | Quantity | Total Amount | Unit Price |
+======================================+==========+==============+============+
| EMPORTER LA CASTE | 1.00 | 11.50 | 11.50 |
+--------------------------------------+----------+--------------+------------+
| CANETTE SODA | 1.00 | 2.00 | 2.00 |
+--------------------------------------+----------+--------------+------------+
########
Document
########
:Mindee ID: ac5a337b-7ca2-4acd-8a3f-a2cf54697ede
:Filename: receipt_p0_3.pdf
Inference
#########
:Product: mindee/expense_receipts v5.1
:Rotation applied: Yes
Prediction
==========
:Expense Locale: fr-FR; fr; FR; EUR;
:Purchase Category: food
:Purchase Subcategory: restaurant
:Document Type: EXPENSE RECEIPT
:Purchase Date: 2022-08-24
:Purchase Time: 12:45
:Total Amount: 16.50
:Total Net: 15.00
:Total Tax: 1.50
:Tip and Gratuity:
:Taxes:
+---------------+--------+----------+---------------+
| Base | Code | Rate (%) | Amount |
+===============+========+==========+===============+
| 15.00 | TVA | 10.00 | 1.50 |
+---------------+--------+----------+---------------+
:Supplier Name: BUR
:Supplier Company Registrations: FR95811878032
81187803200010
:Supplier Address: 22 rue rambuteau 75003 paris france
:Supplier Phone Number:
:Line Items:
+--------------------------------------+----------+--------------+------------+
| Description | Quantity | Total Amount | Unit Price |
+======================================+==========+==============+============+
| Repas | 1.00 | 16.50 | 16.50 |
+--------------------------------------+----------+--------------+------------+
Page Predictions
================
Page 0
------
:Expense Locale: fr-FR; fr; FR; EUR;
:Purchase Category: food
:Purchase Subcategory: restaurant
:Document Type: EXPENSE RECEIPT
:Purchase Date: 2022-08-24
:Purchase Time: 12:45
:Total Amount: 16.50
:Total Net: 15.00
:Total Tax: 1.50
:Tip and Gratuity:
:Taxes:
+---------------+--------+----------+---------------+
| Base | Code | Rate (%) | Amount |
+===============+========+==========+===============+
| 15.00 | TVA | 10.00 | 1.50 |
+---------------+--------+----------+---------------+
:Supplier Name: BUR
:Supplier Company Registrations: FR95811878032
81187803200010
:Supplier Address: 22 rue rambuteau 75003 paris france
:Supplier Phone Number:
:Line Items:
+--------------------------------------+----------+--------------+------------+
| Description | Quantity | Total Amount | Unit Price |
+======================================+==========+==============+============+
| Repas | 1.00 | 16.50 | 16.50 |
+--------------------------------------+----------+--------------+------------+
########
Document
########
:Mindee ID: 2fc54836-61eb-4c0f-b3f3-35717505f7aa
:Filename: receipt_p0_4.pdf
Inference
#########
:Product: mindee/expense_receipts v5.1
:Rotation applied: Yes
Prediction
==========
:Expense Locale: fr-FR; fr; FR; EUR;
:Purchase Category: food
:Purchase Subcategory: restaurant
:Document Type: EXPENSE RECEIPT
:Purchase Date: 2022-08-23
:Purchase Time: 12:52
:Total Amount: 14.50
:Total Net:
:Total Tax: 0.21
:Tip and Gratuity:
:Taxes:
+---------------+--------+----------+---------------+
| Base | Code | Rate (%) | Amount |
+===============+========+==========+===============+
| | TVA | 5.50 | 0.21 |
+---------------+--------+----------+---------------+
:Supplier Name: CHEZ ALAIN MIAM MIAM
:Supplier Company Registrations: 83833640200014
:Supplier Address: 26 rue char lot 75003 paris france
:Supplier Phone Number:
:Line Items:
+--------------------------------------+----------+--------------+------------+
| Description | Quantity | Total Amount | Unit Price |
+======================================+==========+==============+============+
| MIAM MIAM | 1.00 | 10.50 | 10.50 |
+--------------------------------------+----------+--------------+------------+
| Limonade | 1.00 | 4.00 | 4.00 |
+--------------------------------------+----------+--------------+------------+
Page Predictions
================
Page 0
------
:Expense Locale: fr-FR; fr; FR; EUR;
:Purchase Category: food
:Purchase Subcategory: restaurant
:Document Type: EXPENSE RECEIPT
:Purchase Date: 2022-08-23
:Purchase Time: 12:52
:Total Amount: 14.50
:Total Net:
:Total Tax: 0.21
:Tip and Gratuity:
:Taxes:
+---------------+--------+----------+---------------+
| Base | Code | Rate (%) | Amount |
+===============+========+==========+===============+
| | TVA | 5.50 | 0.21 |
+---------------+--------+----------+---------------+
:Supplier Name: CHEZ ALAIN MIAM MIAM
:Supplier Company Registrations: 83833640200014
:Supplier Address: 26 rue char lot 75003 paris france
:Supplier Phone Number:
:Line Items:
+--------------------------------------+----------+--------------+------------+
| Description | Quantity | Total Amount | Unit Price |
+======================================+==========+==============+============+
| MIAM MIAM | 1.00 | 10.50 | 10.50 |
+--------------------------------------+----------+--------------+------------+
| Limonade | 1.00 | 4.00 | 4.00 |
+--------------------------------------+----------+--------------+------------+
########
Document
########
:Mindee ID: dfb53ec3-7f05-4e6a-a26a-c727a52b8984
:Filename: receipt_p0_5.pdf
Inference
#########
:Product: mindee/expense_receipts v5.1
:Rotation applied: Yes
Prediction
==========
:Expense Locale: fr-FR; fr; FR; EUR;
:Purchase Category: miscellaneous
:Purchase Subcategory:
:Document Type: CREDIT CARD RECEIPT
:Purchase Date: 2022-08-23
:Purchase Time: 12:52
:Total Amount: 14.50
:Total Net:
:Total Tax:
:Tip and Gratuity:
:Taxes:
+---------------+--------+----------+---------------+
| Base | Code | Rate (%) | Amount |
+===============+========+==========+===============+
:Supplier Name:
:Supplier Company Registrations: 83833640200022
:Supplier Address:
:Supplier Phone Number:
:Line Items:
+--------------------------------------+----------+--------------+------------+
| Description | Quantity | Total Amount | Unit Price |
+======================================+==========+==============+============+
| NO AUTO: MONTANT | | 14.50 | |
+--------------------------------------+----------+--------------+------------+
Page Predictions
================
Page 0
------
:Expense Locale: fr-FR; fr; FR; EUR;
:Purchase Category: miscellaneous
:Purchase Subcategory:
:Document Type: CREDIT CARD RECEIPT
:Purchase Date: 2022-08-23
:Purchase Time: 12:52
:Total Amount: 14.50
:Total Net:
:Total Tax:
:Tip and Gratuity:
:Taxes:
+---------------+--------+----------+---------------+
| Base | Code | Rate (%) | Amount |
+===============+========+==========+===============+
:Supplier Name:
:Supplier Company Registrations: 83833640200022
:Supplier Address:
:Supplier Phone Number:
:Line Items:
+--------------------------------------+----------+--------------+------------+
| Description | Quantity | Total Amount | Unit Price |
+======================================+==========+==============+============+
| NO AUTO: MONTANT | | 14.50 | |
+--------------------------------------+----------+--------------+------------+
Despite showing a filename, the results shouldn't needlessly create temporary files, as everything is managed through file buffers.
A working implementation of this code can be found in the example/multiReceiptsTutorial.js file on the Node.js repo.
Custom Implementation
If you are using a non-supported language, or simply want to implement things your own way, here's a gist of how things work under the hood...
But first, if you are not familiar with other products yet, know that a response from the API will contain a document field, which in turns contains the following attributes of interest: inference > prediction and inference > pages. pages being a list of objects, all containing their own prediction attributes. Think of the inference's prediction as an aggregate of these.
Note implementation might change depending on the language you use, but the structure will always follow that of the raw json response, which you can find the scheme of in this page.
Why are these predictions relevant? Because they contain our receipts' coordinates. Which in turn contains a bounding box, which is simply a series of 4 coordinates. The coordinates have a very simple structure: a list of four points, which are each a list of two elements (x position and y position).
Those coordinates represent an offset to the top-left origin, displayed in percentage (float or equivalent between 0.0 and 1.0). Which means that in order to get a receipt's coordinates, you need to multiply its x coordinate by the width of the file, and its y coordinate by the height of the file.
Important: Be aware that some libraries flip the y side and consider the bottom-right point to be the origin, which will result in upside-down images if you aren't careful!
The next step is pretty straightforward: use the bounding boxes of each receipts to cut out a piece of the original file, and then either save those file or send them to the Receipt OCR for parsing!
Updated 4 months ago