Financial Documents OCR
Mindee’s Financial Documents OCR API uses deep learning to automatically, accurately, and instantaneously parse both invoices and expense receipts in your applications using the same endpoint. It takes the API a few seconds to extract data from your PDFs or photos. The API extracts data such as:
- Line items
- Due date
- Invoice date
- Invoice number
- Reference numbers
- Locale & currency
- Payment details (IBAN, Swift, Bic, Account number...)
- Supplier company registrations (SIRET, EIN, VAT number...)
- Supplier name
- Supplier address
- Supplier phone number
- Taxes details
- Total amount including taxes
- Total net amount (excluding taxes)
- Customer address
- Customer company registrations
- Customer name
- Category
- Sub Category
- Tip & Gratuity (typed and handwritten)
- Time
Supported Countries
We officially support the following countries, which doesn't prevent you to use the API for invoices from different locations: you may get the country-specific data like taxes.
North America
United States & Canada
Europe
Austria, Belgium, Croatia, Czech Republic, Denmark, Estonia, Finland, France, Germany, Hungary, Iceland, Ireland, Italy, Latvia, Luxembourg, Monaco, Netherlands, Norway, Poland, Portugal, Romania, Slovakia, Slovenia, Spain, Sweden, Switzerland & United Kingdom
Set up the API
Important
Before making any API calls, you need to have created your API key.
-
You'll need to get a document : an invoice or a receipt. This can be a recently received financial document or you can google for free invoice samples that you can download to test with. Alternatively, you can use the sample invoice provided below.
-
Access your Financial Documents API by clicking on the Financial Documents card in Explore APIs.
-
From the left navigation, go to documentation > API reference, you'll find sample code in popular languages and command line.
import requests
url = "https://api.mindee.net/v1/products/mindee/financial_document/v1/predict"
with open("/path/to/my/file.ext", "rb") as file_handle:
files = {"document": file_handle}
headers = {"Authorization": "Token my-api-key-here"}
response = requests.post(url, files=files, headers=headers)
print(response.text)
// works for NODE > v10
const axios = require('axios');
const fs = require("fs");
const FormData = require('form-data');
async function makeRequest() {
let data = new FormData();
data.append('document', fs.createReadStream('./file.jpg'))
const config = {
method: 'POST',
url: 'https://api.mindee.net/v1/products/mindee/financial_document/v1/predict',
headers: {
'Authorization':'Token my-api-key-here',
...data.getHeaders()
},
data
}
try {
let response = await axios(config)
console.log(response.data);
} catch (error) {
console.log(error)
}
}
makeRequest();
curl -X POST \
https://api.mindee.net/v1/products/mindee/financial_document/v1/predict \
-H 'Authorization: Token my-api-key-here' \
-H 'content-type: multipart/form-data' \
-F [email protected]/path/to/your/file.png
# tested with Ruby 2.5
require 'uri'
require 'net/http'
require 'net/https'
require 'mime/types'
url = URI("https://api.mindee.net/v1/products/mindee/financial_document/v1/predict")
file = "/path/to/your/file.png"
http = Net::HTTP.new(url.host, url.port)
http.use_ssl = true
request = Net::HTTP::Post.new(url)
request["Authorization"] = 'Token my-api-key-here'
request.set_form([['document', File.open(file)]], 'multipart/form-data')
response = http.request(request)
puts response.read_body
<form onsubmit="mindeeSubmit(event)" >
<input type="file" id="my-file-input" name="file" />
<input type="submit" />
</form>
<script type="text/javascript">
const mindeeSubmit = (evt) => {
evt.preventDefault()
let myFileInput = document.getElementById('my-file-input');
let myFile = myFileInput.files[0]
if (!myFile) { return }
let data = new FormData();
data.append("document", myFile, myFile.name);
let xhr = new XMLHttpRequest();
xhr.addEventListener("readystatechange", function () {
if (this.readyState === 4) {
console.log(this.responseText);
}
});
xhr.open("POST", "https://api.mindee.net/v1/products/mindee/financial_document/v1/predict");
xhr.setRequestHeader("Authorization", "Token my-api-key-here");
xhr.send(data);
}
</script>
using System;
using System.IO;
using System.Net.Http;
using System.Net.Http.Headers;
class Program
{
static void Main(string[] args)
{
var url = "https://api.mindee.net/v1/products/mindee/financial_document/v1/predict";
var filePath = @"/path/to/my/file";
var token = my-api-key-here;
var file = File.OpenRead(filePath);
var streamContent = new StreamContent(file);
var imageContent = new ByteArrayContent(streamContent.ReadAsByteArrayAsync().Result);
imageContent.Headers.ContentType = MediaTypeHeaderValue.Parse("multipart/form-data");
var form = new MultipartFormDataContent();
form.Add(imageContent, "document", Path.GetFileName(filePath));
var httpClient = new HttpClient();
httpClient.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Token", "my-api-key-here");
var response = httpClient.PostAsync(url, form).Result;
Console.WriteLine(response.Content.ReadAsStringAsync().Result);
}
}
- Replace my-api-key-here with your new API key, or use the "select an API key" feature and it will be filled automatically.
- Copy and paste the sample code of your desired choice in your application, code environment or terminal.
- Replace
/path/to/my/file
with the path to your receipt.
API Key
Always remember to replace your API key!
- Run your code. You will receive a JSON response with the document details (Invoice or Receipt).
API Response
Here is the full JSON response you get when you call the API:
{
"api_request": {
"error": {},
"resources": [
"document"
],
"status": "success",
"status_code": 201,
"url": "https://api.mindee.net/v1/products/mindee/financial_document/v1/predict"
},
"document": {
"id": "cc875b84-e83e-4799-aabe-4b2c18f44b27",
"name": "sample_document.jpg",
"n_pages": 1,
"is_rotation_applied": true,
"inference": {
"started_at": "2022-11-06T16:37:28+00:00",
"finished_at": "2022-11-06T16:37:29+00:00",
"processing_time": 1.121,
"pages": [
{
"id": 0,
"orientation": {"value": 0},
"prediction": { .. },
"extras": {}
}
],
"prediction": { .. },
"extras": {}
}
}
}
You can find the prediction within the prediction
key found in two locations:
- In
document > inference > prediction
for document-level predictions: it contains the different fields extracted at the document level, meaning that for multi-pages PDFs, we reconstruct a single receipt object using all the pages. - In
document > inference > pages[ ] > prediction
for page-level predictions: it gives the prediction for each page independently. With images, there is only one element on this array, but with PDFs, you can find the extracted data for each PDF page.
Each predicted field may contain one or several values:
- a
confidence
score - a
polygon
highlighting the information location - a
page_id
where the information was found (document level only)
prediction": {
"category": {
"confidence": 0.67,
"value": "miscellaneous"
},
"customer_address": {
"confidence": 0.31,
"polygon": [
[
0.035,
0.304
],
[
0.214,
0.304
],
[
0.214,
0.353
],
[
0.035,
0.353
]
],
"value": "1954 Bloon Street West Toronto, ON, M6P 3K9 Canada"
},
"customer_company_registrations": [],
"customer_name": {
"confidence": 0.84,
"polygon": [
[
0.035,
0.284
],
[
0.098,
0.284
],
[
0.098,
0.296
],
[
0.035,
0.296
]
],
"value": "JIRO DOI"
},
"date": {
"confidence": 0.99,
"polygon": [
[
0.842,
0.305
],
[
0.931,
0.305
],
[
0.931,
0.319
],
[
0.842,
0.319
]
],
"value": "2018-09-25"
},
"document_type": {
"value": "INVOICE"
},
"due_date": {
"confidence": 0.93,
"polygon": [
[
0.84,
0.326
],
[
0.943,
0.326
],
[
0.943,
0.34
],
[
0.84,
0.34
]
],
"value": "2018-09-25"
},
"invoice_number": {
"confidence": 0.99,
"polygon": [
[
0.843,
0.265
],
[
0.861,
0.265
],
[
0.861,
0.28
],
[
0.843,
0.28
]
],
"value": "14"
},
"line_items": [
{
"confidence": 0.97,
"description": "Platinum web hosting package Down 35mb, Up 100mb",
"polygon": [
[
0.035,
0.501
],
[
0.965,
0.501
],
[
0.965,
0.536
],
[
0.035,
0.536
]
],
"product_code": null,
"quantity": 1,
"tax_amount": null,
"tax_rate": null,
"total_amount": 65,
"unit_price": 65
},
{
"confidence": 0.97,
"description": "2 page website design Includes basic wireframes, and responsive templates",
"polygon": [
[
0.035,
0.563
],
[
0.964,
0.563
],
[
0.964,
0.598
],
[
0.035,
0.598
]
],
"product_code": null,
"quantity": 3,
"tax_amount": null,
"tax_rate": null,
"total_amount": 2100,
"unit_price": 2100
},
{
"confidence": 0.97,
"description": "Mobile designs Includes responsive navigation",
"polygon": [
[
0.035,
0.626
],
[
0.964,
0.626
],
[
0.964,
0.662
],
[
0.035,
0.662
]
],
"product_code": null,
"quantity": 1,
"tax_amount": null,
"tax_rate": null,
"total_amount": 250,
"unit_price": 250
}
],
"locale": {
"confidence": 0.93,
"currency": "CAD",
"language": "en"
},
"orientation": {
"confidence": 0.99,
"degrees": 0
},
"reference_numbers": [
{
"confidence": 0.91,
"polygon": [
[
0.84,
0.284
],
[
0.914,
0.284
],
[
0.914,
0.298
],
[
0.84,
0.298
]
],
"value": "AD29094"
}
],
"subcategory": {
"confidence": 0,
"value": null
},
"supplier_address": {
"confidence": 0.48,
"polygon": [
[
0.756,
0.128
],
[
0.964,
0.128
],
[
0.964,
0.162
],
[
0.756,
0.162
]
],
"value": "156 University Ave, Toronto ON, Canada M5H 2H7"
},
"supplier_company_registrations": [],
"supplier_name": {
"confidence": 0.72,
"polygon": [
[
0.164,
0.087
],
[
0.4,
0.087
],
[
0.4,
0.147
],
[
0.164,
0.147
]
],
"value": "TURNPIKE DESIGNS CO."
},
"supplier_payment_details": [],
"supplier_phone_number": {
"confidence": 0,
"polygon": [],
"value": null
},
"taxes": [
{
"base": null,
"code": null,
"confidence": 0.76,
"polygon": [
[
0.785,
0.745
],
[
0.965,
0.745
],
[
0.965,
0.758
],
[
0.785,
0.758
]
],
"rate": 8,
"value": 193.2
}
],
"time": {
"confidence": 0,
"polygon": [],
"value": null
},
"tip": {
"confidence": 0,
"polygon": [],
"value": null
},
"total_amount": {
"confidence": 0.99,
"polygon": [
[
0.887,
0.84
],
[
0.97,
0.84
],
[
0.97,
0.858
],
[
0.887,
0.858
]
],
"value": 2608.2
},
"total_net": {
"confidence": 0,
"polygon": [],
"value": null
},
"total_tax": {
"confidence": 0.76,
"polygon": [],
"value": 193.2
}
}
}
Depending on the field type specified, additional attributes can be extracted from the document object. Using the above invoice example, the following are the basic fields that can be extracted.
- Line items
- Invoice Number
- Reference numbers
- Customer Information
- Supplier Information
- Category
- Subcategory
- Document type
- Dates
- Time
- Locale & currency
- Orientation
- Taxes
- Total tax
- Tip & Gratuity
- Total amount
- Total net
Line items
- line_items: In the JSON response below, we have the list of items included in the body of the invoice or receipt. Each item may contain:
- description (String Generic): The description of the item
- product_code (String Generic): the code that identifies the product in the line. It can be a reference number, a product ID, an Article code… - Invoice Only
- quantity (number)
- unit_price (number)
- total_amount (number): The line total amount
- tax_amount (number) - Invoice only
- tax_rate (number): Returned in % - Invoice only
{
"line_items": [
{
"confidence": 0.97,
"description": "Platinum web hosting package Down 35mb, Up 100mb",
"polygon": [
[0.035, 0.501],
[0.965, 0.501],
[0.965, 0.537],
[0.035, 0.537]
],
"product_code": null,
"quantity": 1,
"tax_amount": null,
"tax_rate": null,
"total_amount": 65,
"unit_price": 65
},
{
"confidence": 0.97,
"description": "2 page website design Includes basic wireframes, and responsive templates",
"polygon": [
[0.035, 0.562],
[0.964, 0.562],
[0.964, 0.599],
[0.035, 0.599]
],
"product_code": null,
"quantity": 3,
"tax_amount": null,
"tax_rate": null,
"total_amount": 2100,
"unit_price": 2100
},
{
"confidence": 0.96,
"description": "Mobile designs Includes responsive navigation",
"polygon": [
[0.035, 0.626],
[0.964, 0.626],
[0.964, 0.662],
[0.035, 0.662]
],
"product_code": null,
"quantity": 1,
"tax_amount": null,
"tax_rate": null,
"total_amount": 250,
"unit_price": 250
}
]
}
Reference numbers
- reference_numbers: In the JSON response we have a list of variable size with references extracted including PO numbers, reference numbers or more rarely project numbers.
{
"reference_numbers": [
{
"confidence": 1,
"polygon": [
[0.628, 0.374],
[0.703, 0.374],
[0.703, 0.383],
[0.628, 0.383]
],
"value": "PO-00001827"
},
{
"confidence": 1,
"polygon": [
[0.156, 0.901],
[0.211, 0.901],
[0.211, 0.908],
[0.156, 0.908]
],
"value": "1C5H17N"
}
]
}
Customer Information
Customer information fields are only available for Invoices
- customer_name: In the JSON response, the value of the customer name as found on the invoice.
{
"customer_name": {
"confidence": 0.84,
"page_id": 0,
"polygon": [[0.035, 0.284], [0.098, 0.298], [0.098, 0.296], [0.035, 0.296]],
"value": "JIRO DOI"
}
}
- customer_address: In the JSON response, the value of the customer address as found on the invoice.
{
"customer_address": {
"confidence": 0.3,
"page_id": 0,
"polygon": [[0.035, 0.304], [0.214, 0.304], [0.214, 0.353], [0.035, 0.0353]],
"value": "1954 Bloon Street West Toronto, ON, M6P 3K9 Canada"
}
}
- customer_company_registrations (string): In the JSON response below, the list of the customer company identifiers. Each item may contain:
- type (String Generic): The following company registration numbers are supported: VAT NUMBER, SIRET, SIREN, NIF, CF, UID, STNR, HRA/HRB, TIN (includes EIN, FEIN, SSN, ATIN, PTIN, ITIN), RFC, BTW, ABN, UEN, CVR, ORGNR, INN, DPH, GSTIN, COMPANY REGISTRATION NUMBER (UK), KVK, DIC
- value (String): Value of the company identifier
{
"customer_company_registrations": [
{
"confidence": 0.99,
"page_id": 0,
"polygon": [[ 0.515, 0.962 ], [ 0.59, 0.962 ], [ 0.59, 0.973 ], [ 0.515, 0.973 ]],
"type": "SIRET",
"value": "XXX81125600010"
},
{
"confidence": 0.99,
"page_id": 0,
"polygon": [[ 0.658, 0.963 ], [ 0.729, 0.963 ], [ 0.729, 0.973 ], [ 0.658, 0.973 ]],
"type": "VAT NUMBER",
"value": "FR44837811XXX"
}
]
}
Invoice Number
- invoice_number : in the JSON response bellow, the value of the Invoice number.
{
"invoice_number": {
"confidence": 0.99,
"polygon": [[0.5293,0.8128],[0.176,0.092],[0.5812,0.4886],[0.4846,0.1325]],
"value": "1547854214"
}
}
Supplier Information
- supplier_payment_details: On some invoices (Null on receipts), there are many payment details written such as account number, IBAN, routing number, BIC etc. Our Financial documents OCR extracts all of them. In the JSON response below, we have the list of the suppliers' payment details, each item contains different fields and they are set to null or filled with the right value depending on the invoice / receipt:
{
"supplier_payment_details": [
{
"account_number": "XXXX",
"confidence": 0.95,
"iban": "XXXX",
"page_id": 0,
"polygon": [[0.035, 0.304], [0.214, 0.304], [0.214, 0.353], [0.035, 0.0353]],
"routing_number": "XXX",
"swift": "XXX"
}
]
}
- supplier_company_registrations: List of detected supplier's company registration number. Each company number object contains an extra attribute:
- type (String Generic): The following company registration numbers are supported: VAT NUMBER, SIRET, SIREN, NIF, CF, UID, STNR, HRA/HRB, TIN (includes EIN, FEIN, SSN, ATIN, PTIN, ITIN), RFC, BTW, ABN, UEN, CVR, ORGNR, INN, DPH, GSTIN, COMPANY REGISTRATION NUMBER (UK), KVK, DIC
- value (String): Value of the company identifier
{
"supplier_company_registrations": [
{
"confidence": 0.99,
"page_id": 0,
"polygon": [[0.515, 0.962], [0.59, 0.962], [0.59, 0.973], [0.515, 0.973]],
"type": "SIRET",
"value": "XXX81125600010"
},
{
"confidence": 0.99,
"page_id": 0,
"polygon": [[0.658, 0.963], [0.729, 0.963], [0.729, 0.973], [0.658, 0.973]],
"type": "VAT",
"value": "FR44837811XXX"
}
]
}
- supplier_name: In the JSON response below, we have the value of the supplier name as written in the invoice or receipt document
{
"supplier_name": {
"confidence": 0.11,
"page_id": 0,
"polygon": [[0.165, 0.089], [0.385, 0.089], [0.385, 0.145], [0.165, 0.145]],
"value": "DESIGNS TURNPIKE CO"
}
}
- supplier_address: In the JSON response, the value of the supplier address as found on the invoice.
{
"supplier_address": {
"confidence": 0.49,
"page_id": 0,
"polygon": [[0.756, 0.128], [0.964, 0.128], [0.964, 0.162], [0.756, 0.162]],
"value": "156 University Ave, Toronto ON, Canada M5H 2H7"
}
}
- supplier_phone_number: in the JSON response the value of the supplier phone number. This field is only available for receipts.
Category
- category: The API makes a prediction on the type of purchase. In the following example, it is 99% sure it is food. The possible categories that can be extracted are:
- For receipt:
toll, food, parking, transport, accommodation, gasoline, telecom, miscellaneous
- For Invoices:
accommodation, telecom, transport, miscellaneous
{
"category": {
"confidence": 0.99,
"value": "food"
}
}
Subcategory
- subcategory: The API returns different values depending on the value of the
category
field.- If
category
=transport
Thesubcategory
field can return values specifying what type of transport the receipt is:plane
,train
,taxi
- If
category
=food
(limited to receipt documents)
Thesubcategory
field can return values specifying what type of food the receipt is about:restaurant
,shopping
- If
{
"category": {
"confidence": 0.97,
"value": "transport",
},
"subcategory": {
"value": "taxi" or "plane" or "restaurant",
"confidence": 0.54
}
}
{
"category": {
"confidence": 0.87,
"value": "food",
},
"subcategory": {
"value": "shopping" or "restaurant",
"confidence": 0.54
}
}
Document type
- document_type: is a classification field, that can return 4 values : 'EXPENSE RECEIPT' , 'CREDIT CARD RECEIPT', 'INVOICE', 'CREDIT NOTE'
{
"document_type": {
"confidence": 0.77,
"value": "CREDIT CARD RECEIPT"
}
}
Orientation
- orientation: The orientation field is only available at the page level as it describes whether the page image should be rotated to be upright. The rotation value is also conveniently available in the JSON response at:
document > inference > pages [ ] > orientation > value
.
If the page requires rotation for correct display, the orientation field gives a prediction among these 3 possible outputs:- 0 degree: the page is already upright
- 90 degrees: the page must be rotated clockwise to be upright
- 270 degrees: the page must be rotated counterclockwise to be upright
{
"orientation": {
"confidence": 0.99,
"degrees": 0
}
}
All
polygon
fields across the JSON response are already rotated accordingly!
Date
- date: In the JSON response below, the value of the document date in an ISO format(yyyy-mm-dd).
{
"date": {
"confidence": 0.99,
"page_id": 0,
"polygon": [[0.84, 0.305], [0.932, 0.305], [0.932, 0.318], [0.84, 0.318]],
"value": "2018-09-25"
}
}
- due_date: In the JSON response below,the value of the due_ date in an ISO format(yyyy-mm-dd). Limited to Invoices.
{
"due_date": {
"confidence": 0.86,
"page_id": 0,
"polygon": [[0.841, 0.323], [0.941, 0.323], [0.941, 0.338], [0.841, 0.338]],
"raw": "Upon receipt",
"value": "2018-09-25"
}
}
Time
- time: ISO formatted time as seen on the document in HH: MM format. Limited to Receipts.
{
"time": {
"confidence": 0.99,
"page_id": 0,
"polygon": [[0.62, 0.173], [0.681, 0.173], [0.681, 0.191], [0.62, 0.191]],
"value": "15:20"
}
}
Locale & currency
- **language** (String): Language code in ISO 639-1 format as seen on the document. The following language codes are supported: `ca`, `de`, `en`, `es`, `fr`, `it`, `nl`, and `pt`.
- **currency** (String): Currency code in ISO 4217 format as seen on the document. We support the following 44 currencies:
Europe(13) | North America (3) | South America (4) | Middle East (5) | Asia (12) | Africa (6) | Australia (1) |
---|---|---|---|---|---|---|
EUR, GBP, CHF, CZK, NOK, SEK, HUF, RON, PLN, RUB, DKK, XPF, TRY | USD, CAD, MXN | COP, BRL, CLP, ARS | AED, SAR, QAR, ILS, OMR | CNY, PHP, SGD, HKD, JPY, MYR, KRW, TWD, THB, VND, IMR, IDR | DZD, MAD, TND, XOF, ZAR, XAF | AUD |
{
"locale": {
"confidence": 0.82,
"currency": "GBP",
"language": "en"
}
}
Taxes
- taxes: In the JSON response below, the list of taxes detected in the document. Each of these items includes:
- value: the amount of the tax line in the receipt currency
- rate: float between 0 and 1
-basis: the amount used to calculate the tax (Null for Invoices) - code: the tax code (Null for Invoices). The output is one of the following:
VAT
,TVA
,MWST
,IVA
,BTW
,UST
,GST
,QST
,PST
,HST
,RST
,MERCHANT TAX
,GOVERNMENT TAX
,TRANSPORTATION TAX
,EXCISE TAX
,COUNTY TAX
,CITY TAX
,STATE TAX
,LOCAL TAX
,SALES TAX
,ACCOMMODATION TAX
,FOOD TAX
,BEVERAGE TAX
,FLIGHT TAX
,TAX
{
"taxes": [
{
"confidence": 0.98,
"page_id": 0,
"polygon": [[0.19, 0.403], [0.698, 0.403], [0.698, 0.432], [0.19, 0.432]],
"rate": 20.0,
"code": null,
"basis":null,
"value": 1.7
}
]
}
Total tax
- total_tax: Total tax amount of the purchase.
{
"total_tax": {
"confidence": 0.79,
"page_id": 0,
"polygon": [[0.549, 0.619], [0.715, 0.619], [0.715, 0.64], [0.549, 0.64]],
"value": 2.42
}
}
Tip & Gratuity
- tip: Total amount of tip and gratuity. Both typed and handwritten characters are supported.
{
"tip": {
"confidence": 0.29,
"page_id": 0,
"polygon": [[0.549, 0.619], [0.715, 0.619], [0.715, 0.64], [0.549, 0.64]],
"value": 12.00
}
}
Total amount
- total_amount: Perhaps the most important part of a financial document is the total spent including taxes, discounts, fees, tips, and gratuity. In the case the document is a receipt, the Financial Document OCR API supports both typed and handwritten characters for this field.
{
"total_amount": {
"confidence": 0.99,
"page_id": 0,
"polygon": [[0.549, 0.619], [0.715, 0.619], [0.715, 0.64], [0.549, 0.64]],
"value": 10.2
}
}
Total net (excluding taxes)
- total_net: Total amount excluding taxes
{
"total_net": {
"confidence": 0.89,
"page_id": 0,
"polygon": [[0.549, 0.619], [0.715, 0.619], [0.715, 0.64], [0.549, 0.64]],
"value": 14.27
}
}
Questions?
Join our Slack
Updated 14 days ago