Receipt OCR

Automatically extract data from receipts and bills

Mindee’s receipt OCR API uses deep learning to automatically, accurately, and instantaneously parse your receipt details. In under a second, the API extracts a set of data from your PDFs or photos of receipts, including:

  • Expense category
  • Locale & currency
  • Receipt date
  • Receipt time
  • Supplier
  • Taxes details
  • Total amount

Set up the API

  1. You'll need a receipt. You can use the sample receipt provided below.

sample receiptsample receipt

  1. Log into your Mindee account and access your Expense Receipt API dashboard by clicking the Expense Receipts card.
    Expense receipts API card on the rightExpense receipts API card on the right

  2. On the left navigation, click on API Keys, and on the page click on the Create a new API key button
    API keys on the left navAPI keys on the left nav

  3. Name your API key and click on the Create API key button.
    create a new API key buttoncreate a new API key button

  4. From the left navigation, go to documentation > API Reference, you'll find sample code in popular languages and command line.
    receipt API sample code in the API Reference sectionreceipt API sample code in the API Reference section

curl -X POST \
  https://api.mindee.net/v1/products/mindee/expense_receipts/v3/predict \
  -H 'Authorization: Token my-api-key-here' \
  -F [email protected]/path/to/your/file.png
import requests

url = "https://api.mindee.net/v1/products/mindee/expense_receipts/v3/predict"

with open("/path/to/my/file", "rb") as myfile:
    files = {"document": myfile}
    headers = {"Authorization": "Token my-api-key-here"}
    response = requests.post(url, files=files, headers=headers)
    print(response.text)
// works for NODE > v10
const axios = require('axios');
const fs = require("fs");
const FormData = require('form-data')

async function makeRequest() {
    let data = new FormData()
    data.append('document', fs.createReadStream('./file.jpg'))
    const config = {
        method: 'POST',
        url: 'https://api.mindee.net/v1/products/mindee/expense_receipts/v3/predict',
        headers: { 
          'Authorization':'Token my-api-key-here',
          ...data.getHeaders()
           },
        data
    }

    try {
      let response = await axios(config)
      console.log(response.data);
    } catch (error) {
      console.log(error)
    }

}

makeRequest()
# tested with Ruby 2.5
require 'uri'
require 'net/http'
require 'net/https'
require 'mime/types'

url = URI("https://api.mindee.net/v1/products/mindee/expense_receipts/v3/predict")
file = "/path/to/your/file.png"

http = Net::HTTP.new(url.host, url.port)
http.use_ssl = true
request = Net::HTTP::Post.new(url)
request["Authorization"] = 'Token my-api-key-here'
request.set_form([['document', File.open(file)]], 'multipart/form-data')

response = http.request(request)
puts response.read_body
<form onsubmit="mindeeSubmit(event)" >
  <input type="file" id="my-file-input" name="file" />
  <input type="submit" />
</form>

<script type="text/javascript">
const mindeeSubmit = (evt) => {
  evt.preventDefault()
  let myFileInput = document.getElementById('my-file-input');
  let myFile = myFileInput.files[0]
  if (!myFile) { return }
  let data = new FormData();
  data.append("document", myFile, myFile.name);

  let xhr = new XMLHttpRequest();

  xhr.addEventListener("readystatechange", function () {
    if (this.readyState === 4) {
      console.log(this.responseText);
    }
  });

  xhr.open("POST", "https://api.mindee.net/v1/products/mindee/expense_receipts/v3/predict");
  xhr.setRequestHeader("Authorization", "Token my-api-key-here");
  xhr.send(data);
}
</script>
  • Replace my-api-key-here with your new API key, or use the "select an API key" feature and it will be filled automatically.
  • Copy and paste the sample code of your desired choice in your application, code environment or terminal.
  • Replace /path/to/my/file with the path to your invoice.

❗️

Don't forget to replace your token! Alternatively, you can use our ready-to-use Client Libraries Python SDK and Node SDK

  1. Run your code. You will receive a JSON response with the receipt details.

API Response

Here is the full JSON response you get when you call the API:

{
  "api_request": {
    "error": {},
    "resources": [
      "document"
    ],
    "status": "success",
    "status_code": 201,
    "url": "https://api.mindee.net/v1/products/mindee/expense_receipts/v3/predict"
  },
  "document": {
    "id": "cc875b84-e83e-4799-aabe-4b2c18f44b27",
    "name": "sample_receipt.jpg",
    "n_pages": 1,
    "is_rotation_applied": true,
    "inference": {
      "started_at": "2021-05-06T16:37:28+00:00",
      "finished_at": "2021-05-06T16:37:29+00:00",
      "processing_time": 1.121,
      "pages": [
        {
          "id": 0,
          "orientation": {"value": 0},
          "prediction": { .. },
          "extras": {}
        }
      ],
      "prediction": { .. },
      "extras": {}
    }
  }
}

You can find the prediction within the prediction key found in two locations:

  • In document > inference > prediction for document-level predictions: it contains the different fields extracted at the document level, meaning that for multi-pages PDFs, we reconstruct a single receipt object using all the pages.
  • In document > inference > pages[ ] > prediction for page-level predictions: it gives the prediction for each page independently. With images, there is only one element on this array, but with PDFs, you can find the extracted data for each PDF page.

Each predicted field may contain one or several values:

  • a confidence score
  • a polygon highlighting the information location
  • a page_id where the information was found (document level only)
{
  "prediction": {
    "category": {
      "confidence": 0.99,
      "value": "food"
    },
    "date": {
      "confidence": 0.99,
      "page_id": 0,
      "polygon": [[0.479, 0.173], [0.613, 0.173], [0.613, 0.197], [0.479, 0.197]],
      "raw": "26-Feb-2016",
      "value": "2016-02-26"
    },
    "locale": {
      "confidence": 0.82,
      "country": "GB",
      "currency": "GBP",
      "language": "en",
      "value": "en-GB"
    },
    "orientation": {
      "confidence": 0.99,
      "degrees": 0
    },
    "supplier": {
      "confidence": 0.71,
      "page_id": 0,
      "polygon": [[0.394, 0.068], [0.477, 0.068], [0.477, 0.087], [0.394, 0.087]],
      "value": "CLACHAN"
    },
    "taxes": [
      {
        "code": null,
        "confidence": 0.98,
        "page_id": 0,
        "polygon": [[0.19, 0.403], [0.698, 0.403], [0.698, 0.432], [0.19, 0.432]],
        "rate": 20.0,
        "value": 1.7
      }
    ],
    "time": {
      "confidence": 0.99,
      "page_id": 0,
      "polygon": [[0.62, 0.173], [0.681, 0.173], [0.681, 0.191], [0.62, 0.191]],
      "raw": "15:20",
      "value": "15:20"
    },
    "total_incl": {
      "confidence": 0.99,
      "page_id": 0,
      "polygon": [[0.549, 0.619], [0.715, 0.619], [0.715, 0.64], [0.549, 0.64]],
      "value": 10.2
    }
  }
}

Additional Attributes

Depending on the field type specified, additional attributes can be extracted from the receipt object. Using the above receipt example the following are the basic fields that can be extracted.

Category

  • category: The API makes a prediction on the type of purchase. In the following example, it is 99% sure it is food. The possible categories that can be extracted are:
    • accommodation
    • food
    • gasoline
    • toll
    • parking
    • transport
    • toll
{
  "category": {
    "confidence": 0.99,
    "value": "food"
  }
}

Date

  • date: In the JSON response below, we have both the raw and value of the identified date in ISO 8601 format.
{
  "date": {
    "confidence": 0.99,
    "page_id": 0,
    "polygon": [[0.479, 0.173], [0.613, 0.173], [0.613, 0.197], [0.479, 0.197]],
    "raw": "26-Feb-2016",
    "value": "2016-02-26"
  }
}

Locale

  • locale: From the receipt, the API can predict where the purchase was made, the language, and the currency. Check the documentation for the latest support. At the time of writing, support is centered on Europe and North America.
    • language (String): Language code in ISO 639-1 format as seen on the invoice. The following language codes are supported: ca, de, en, es, fr, it, nl, and pt.
    • country (String): Country code in ISO 3166-1 alpha-2 format as seen on the invoice. The following country codes are supported: CA, CH, DE, ES, FR, GB, IT, NL, PT, and US.
    • currency (String): Currency code in ISO 4217 format as seen on the invoice. The following country codes are supported: CAD, CHF, GBP, EUR, USD.
{
  "locale": {
    "confidence": 0.82,
    "country": "GB",
    "currency": "GBP",
    "language": "en",
    "value": "en-GB"
  }
}

Orientation

  • orientation: The orientation field is only available at the page level as it describes whether the page image should be rotated to be upright. The rotation value is also conveniently available in the JSON response at:
    document > inference > pages [ ] > orientation > value.
    If the page requires rotation for correct display, the orientation field gives a prediction among these 3 possible outputs:
    • 0 degree: the page is already upright
    • 90 degrees: the page must be rotated clockwise to be upright
    • 270 degrees: the page must be rotated counterclockwise to be upright
{
  "orientation": {
    "confidence": 0.99,
    "degrees": 0
  }
}

📘

All polygon fields across the JSON response are already rotated accordingly!

Supplier

  • supplier_name: In the JSON response below, we have the value of the merchant as found on the receipt.
{
  "supplier": {
    "confidence": 0.71,
    "page_id": 0,
    "polygon": [[0.394, 0.068], [0.477, 0.068], [0.477, 0.087], [0.394, 0.087]],
    "value": "CLACHAN"
  }
}

Taxes

  • taxes: In the JSON response below, we have the list of taxes detected in the receipt. Each of these items include:
    • value: the amount of the tax line in the receipt currency
    • rate: (optional) float between 0 and 1
    • code: (optional) the tax code (HST, GST... for Canadian; City Tax, State tax for US, etc..)
{
  "taxes": [
    {
      "code": null,
      "confidence": 0.98,
      "page_id": 0,
      "polygon": [[0.19, 0.403], [0.698, 0.403], [0.698, 0.432], [0.19, 0.432]],
      "rate": 20.0,
      "value": 1.7
    }
  ]
}

Time

  • time: In the JSON response below, we have both the raw and the value of the time as seen on the receipt in HH: MM format.
{
  "time": {
    "confidence": 0.99,
    "page_id": 0,
    "polygon": [[0.62, 0.173], [0.681, 0.173], [0.681, 0.191], [0.62, 0.191]],
    "raw": "15:20",
    "value": "15:20"
  }
}

Total

  • total_incl: Perhaps the most important part of the receipt is the total spent including taxes, along with confidence and the box indicating the location on the receipt.
{
  "total_incl": {
    "confidence": 0.99,
    "page_id": 0,
    "polygon": [[0.549, 0.619], [0.715, 0.619], [0.715, 0.64], [0.549, 0.64]],
    "value": 10.2
  }
}

 

Questions?

Slack Logo IconSlack Logo Icon  Join our Slack


Did this page help you?