Receipt OCR

Automatically extract data from receipts and bills

Mindee’s receipt OCR API uses deep learning to automatically, accurately, and instantaneously parse your receipt details.

In under a second, the API extracts a set of data from your pdfs or photos of receipts, including:

Total amount
Receipt date
Receipt time
Merchant name
Taxes details
Locale & currency
Expense category

API Prerequisites

  1. You’ll need a free Mindee account. Sign up and confirm your email to log in.
  2. A receipt. Look in your bag/wallet for a recent one, or do a Google Image search for a receipt and download a few to test with.

Set up the API

Log into your Mindee account and access your Expense Receipt API environment by clicking the Expense Receipts card:

Mindee Expense receipt OCR APIMindee Expense receipt OCR API

Mindee Expense receipt OCR API

When clicking this card, you land on the dashboard page - where you can quickly see API usage (you have none right now, but that will change). On the left navigation, there are links to “Documentation”, “API Keys” and “Live Interface”. The docs tab has all of the technical details you’ll need to build for the receipts API endpoint, and the Live Interface is a cool interactive demo.

Rather than try out the demo, we want to build with the API, so click on API Keys to create an API token.

Click on the Create a new API key button and name your API key:

Now, we are ready to make an API call. You can find sample codes for the most popular languages in the "Documentation" tab:

curl -X POST \
  https://api.mindee.net/v1/products/mindee/expense_receipts/v3/predict \
  -H 'Authorization: Token my-api-key-here' \
  -H 'content-type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW' \
  -F [email protected]/path/to/your/file.png
import requests

url = "https://api.mindee.net/v1/products/mindee/expense_receipts/v3/predict"

with open("/path/to/my/file", "rb") as myfile:
    files = {"document": myfile}
    headers = {"Authorization": "Token my-api-key-here"}
    response = requests.post(url, files=files, headers=headers)
    print(response.text)
// works for NODE > v10
const axios = require('axios');
const fs = require("fs");
const FormData = require('form-data')

async function makeRequest() {
    let data = new FormData()
    data.append('document', fs.createReadStream('./file.jpg'))
    const config = {
        method: 'POST',
        url: 'https://api.mindee.net/v1/products/mindee/expense_receipts/v3/predict',
        headers: { 
          'Authorization':'Token my-api-key-here',
          ...data.getHeaders()
           },
        data
    }

    try {
      let response = await axios(config)
      console.log(response.data);
    } catch (error) {
      console.log(error)
    }

}

makeRequest()
# tested with Ruby 2.5
require 'uri'
require 'net/http'
require 'net/https'
require 'mime/types'

url = URI("https://api.mindee.net/v1/products/mindee/expense_receipts/v3/predict")
file = "/path/to/your/file.png"

http = Net::HTTP.new(url.host, url.port)
http.use_ssl = true
request = Net::HTTP::Post.new(url)
request["Authorization"] = 'Token my-api-key-here'
request.set_form([['document', File.open(file)]], 'multipart/form-data')

response = http.request(request)
puts response.read_body
<form onsubmit="mindeeSubmit(event)" >
  <input type="file" id="my-file-input" name="file" />
  <input type="submit" />
</form>

<script type="text/javascript">
const mindeeSubmit = (evt) => {
  evt.preventDefault()
  let myFileInput = document.getElementById('my-file-input');
  let myFile = myFileInput.files[0]
  if (!myFile) { return }
  let data = new FormData();
  data.append("document", myFile, myFile.name);

  let xhr = new XMLHttpRequest();

  xhr.addEventListener("readystatechange", function () {
    if (this.readyState === 4) {
      console.log(this.responseText);
    }
  });

  xhr.open("POST", "https://api.mindee.net/v1/products/mindee/expense_receipts/v3/predict");
  xhr.setRequestHeader("Authorization", "Token my-api-key-here");
  xhr.send(data);
}
</script>

Simply replace {my-api-key-here} with your new API token and /path/to/your/file/png with the path to your receipt.

📘

You can also copy this code right from the documentation tab of the API with your API token inserted for you.

In this example, we'll use this receipt:

Receipt imageReceipt image

Receipt image

Paste the cURL sample into your terminal, hit enter, and about a second later, you will receive a JSON response with the receipt details. Since the response is quite verbose, we will walk through the various fields section by section.

API Response

Here is the full json response you get when you call the API:

{
  "api_request": {
    "error": {},
    "resources": [
      "document"
    ],
    "status": "success",
    "status_code": 201,
    "url": "http://api.mindee.net/v1/products/mindee/expense_receipts/v3/predict"
  },
  "document": {
    "annotations": {
      "labels": []
    },
    "id": "cc875b84-e83e-4799-aabe-4b2c18f44b26",
    "inference": {
      "finished_at": "2021-05-06T16:37:29+00:00",
      "pages": [
        {
          "id": 0,
          "prediction": {
            "category": {
              "confidence": 0.99,
              "value": "food"
            },
            "date": {
              "confidence": 0.99,
              "polygon": [
                [
                  0.479,
                  0.173
                ],
                [
                  0.613,
                  0.173
                ],
                [
                  0.613,
                  0.197
                ],
                [
                  0.479,
                  0.197
                ]
              ],
              "raw": "26-Feb-2016",
              "value": "2016-02-26"
            },
            "locale": {
              "confidence": 0.82,
              "country": "GB",
              "currency": "GBP",
              "language": "en",
              "value": "en-GB"
            },
            "orientation": {
              "confidence": 0.99,
              "degrees": 0
            },
            "supplier": {
              "confidence": 0.71,
              "polygon": [
                [
                  0.394,
                  0.068
                ],
                [
                  0.477,
                  0.068
                ],
                [
                  0.477,
                  0.087
                ],
                [
                  0.394,
                  0.087
                ]
              ],
              "value": "CLACHAN"
            },
            "taxes": [
              {
                "code": null,
                "confidence": 0.98,
                "polygon": [
                  [
                    0.19,
                    0.403
                  ],
                  [
                    0.698,
                    0.403
                  ],
                  [
                    0.698,
                    0.432
                  ],
                  [
                    0.19,
                    0.432
                  ]
                ],
                "rate": 20.0,
                "value": 1.7
              }
            ],
            "time": {
              "confidence": 0.99,
              "polygon": [
                [
                  0.62,
                  0.173
                ],
                [
                  0.681,
                  0.173
                ],
                [
                  0.681,
                  0.191
                ],
                [
                  0.62,
                  0.191
                ]
              ],
              "raw": "15:20",
              "value": "15:20"
            },
            "total_incl": {
              "confidence": 0.99,
              "polygon": [
                [
                  0.549,
                  0.619
                ],
                [
                  0.715,
                  0.619
                ],
                [
                  0.715,
                  0.64
                ],
                [
                  0.549,
                  0.64
                ]
              ],
              "value": 10.2
            }
          }
        }
      ],
      "prediction": {
        "category": {
          "confidence": 0.99,
          "value": "food"
        },
        "date": {
          "confidence": 0.99,
          "page_id": 0,
          "polygon": [
            [
              0.479,
              0.173
            ],
            [
              0.613,
              0.173
            ],
            [
              0.613,
              0.197
            ],
            [
              0.479,
              0.197
            ]
          ],
          "raw": "26-Feb-2016",
          "value": "2016-02-26"
        },
        "locale": {
          "confidence": 0.82,
          "country": "GB",
          "currency": "GBP",
          "language": "en",
          "value": "en-GB"
        },
        "supplier": {
          "confidence": 0.71,
          "page_id": 0,
          "polygon": [
            [
              0.394,
              0.068
            ],
            [
              0.477,
              0.068
            ],
            [
              0.477,
              0.087
            ],
            [
              0.394,
              0.087
            ]
          ],
          "value": "CLACHAN"
        },
        "taxes": [
          {
            "code": null,
            "confidence": 0.98,
            "page_id": 0,
            "polygon": [
              [
                0.19,
                0.403
              ],
              [
                0.698,
                0.403
              ],
              [
                0.698,
                0.432
              ],
              [
                0.19,
                0.432
              ]
            ],
            "rate": 20.0,
            "value": 1.7
          }
        ],
        "time": {
          "confidence": 0.99,
          "page_id": 0,
          "polygon": [
            [
              0.62,
              0.173
            ],
            [
              0.681,
              0.173
            ],
            [
              0.681,
              0.191
            ],
            [
              0.62,
              0.191
            ]
          ],
          "raw": "15:20",
          "value": "15:20"
        },
        "total_incl": {
          "confidence": 0.99,
          "page_id": 0,
          "polygon": [
            [
              0.549,
              0.619
            ],
            [
              0.715,
              0.619
            ],
            [
              0.715,
              0.64
            ],
            [
              0.549,
              0.64
            ]
          ],
          "value": 10.2
        }
      },
      "processing_time": 1.121,
      "product": {
        "features": [
          "locale",
          "category",
          "date",
          "time",
          "total_incl",
          "taxes",
          "supplier",
          "orientation"
        ],
        "name": "Mindee-Demo/expense_receipts",
        "type": "standard",
        "version": "3.0"
      },
      "started_at": "2021-05-06T16:37:28+00:00"
    },
    "n_pages": 1,
    "name": "sample_receipt.jpg",
    "ocr": {}
  }
}

Extracted fields

Under the api_request key of the JSON response, you can find some metadata about the request.

What is probably most important to you is the extracted data. Under the document key, you can find a structure like this:

{
    "annotations": {
      "labels": []
    },
    "id": "cc875b84-e83e-4799-aabe-4b2c18f44b26",
    "inference": {
      "finished_at": "2021-05-06T16:37:29+00:00",
      "pages": [
        {
          "id": 0,
          "prediction": {
            "category": {...},
            "date": {...},
            "locale": {...},
            "orientation": {...},
            "supplier": {...}
        }
      ],
      "prediction": {
        "category": {... },
        "date": {...},
        "locale": {...},
        "supplier": {...},
        "taxes": [...],
        "time": {...},
        "total_incl": {...},
      "processing_time": 1.121,
      "product": {
        "features": [
          "locale",
          "category",
          "date",
          "time",
          "total_incl",
          "taxes",
          "supplier",
          "orientation"
        ],
        "name": "Mindee-Demo/expense_receipts",
        "type": "standard",
        "version": "3.0"
      },
      "started_at": "2021-05-06T16:37:28+00:00"
    },
    "n_pages": 1,
    "name": "sample_receipt.jpg",
    "ocr": {}
  }

The extracted data appears in two different elements of the list.

Document-level prediction: document > inference > prediction is the document level prediction. It contains the different fields extracted at the document level, meaning that for multi-pages pdfs, we reconstruct a single receipt object using all the pages.

Page-level prediction: document > inference > pages[] > prediction is an array, containing the extracted data from each page. For images, there is only one element on this array, but for pdfs, you can find the extracted data for each pdf page.

Each predicted field contains a confidence_score as well as a polygon when the information is located in the image.

Category

"category": {
    "confidence": 0.99,
    "value": "food"
}

The API makes a prediction on the type of purchase. In this case, it is 99% sure it is food. The possible categories are [toll, food, parking, transport, accommodation, gasoline, miscellaneous].

Date


Identified from the text on the receipt and converted into ISO format. This purchase was made on February 26, 2016, and the model is 99% confident in that choice. The segmentation bounding box provides 4 (x,y) coordinates indicating where the data was pulled from the receipt [(0,0) is the upper left corner, (1,1) is the bottom right corner].

"date": {
    "confidence": 0.99,
    "page_id": 0,
    "polygon": [
      [
        0.479,
        0.173
      ],
      [
        0.613,
        0.173
      ],
      [
        0.613,
        0.197
      ],
      [
        0.479,
        0.197
      ]
    ],
    "raw": "26-Feb-2016",
    "value": "2016-02-26"
  }

Locale

Using data from the receipt, the API can predict where the purchase was made, the language, and the currency. Check the documentation for the latest support. At the time of writing, support is centered on Europe and North America.

"locale": {
  "confidence": 0.82,
  "country": "GB",
  "currency": "GBP",
  "language": "en",
  "value": "en-GB"
}

In the case of my receipt, it is 82% confident that the purchase is in GB, and in £.

Merchant

"supplier": {
  "confidence": 0.71,
    "page_id": 0,
      "polygon": [
        [
          0.394,
          0.068
        ],
        [
          0.477,
          0.068
        ],
        [
          0.477,
          0.087
        ],
        [
          0.394,
          0.087
        ]
      ],
    "value": "CLACHAN"
}

The API correctly predicted (it was 71% sure) that it was a CLACHAN store. Again, four (x,y) points mark the location of the text naming the Merchant on the image.

Orientation

🚧

This field is not extracted on the document-level prediction. Getting a document orientation is not possible as it depends on pages only.

"orientation": {
  "confidence": 0.99,
  "degrees": 0
}

Did the document require rotation before parsing? Measured in 90 degree increments [0.90.180.270]. In this case, it did not require any rotation.

Taxes

"taxes": [
  {
    "code": null,
    "confidence": 0.98,
    "polygon": [
      [
        0.19,
        0.403
      ],
      [
        0.698,
        0.403
      ],
      [
        0.698,
        0.432
      ],
      [
        0.19,
        0.432
      ]
    ],
    "rate": 20.0,
    "value": 1.7
  }
]

The API detects taxes from line items. It's then returned as an array of items. Each of those items contains:

  • value: the amount of the tax line in the receipt currency
  • rate: (optional) float between 0 and 1
  • code: (optional) the tax code (HST, GST... for Canadian; City Tax, State tax for US, etc..)

Time

"time": {
  "confidence": 0.99,
    "polygon": [
      [
        0.62,
        0.173
      ],
      [
        0.681,
        0.173
      ],
      [
        0.681,
        0.191
      ],
      [
        0.62,
        0.191
      ]
    ],
  "raw": "15:20",
  "value": "15:20"
}

Time the receipt was printed in HH:mm format.

Total

Perhaps the most important part of the receipt, the total spent including taxes, along with confidence and the box indicating the location on the receipt.

"total_incl": {
  "confidence": 0.99,
    "polygon": [
      [
        0.549,
        0.619
      ],
      [
        0.715,
        0.619
      ],
      [
        0.715,
        0.64
      ],
      [
        0.549,
        0.64
      ]
    ],
    "value": 10.2
}

Updated about a month ago


Receipt OCR


Automatically extract data from receipts and bills

Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.