/> Case study - Document Data Extraction
Document Parser
A time saving service that turns freight documents into usable, consistent JSON.
- Role
- Sole architect & developer
- Domain
- Data Extractionfreight & logistics
- Status
- In usepending broader deployment · early 2027
- Surface
- DocTR + Claudewith candidate glue
Problem, Meet Solution. The Story
Tackling dreaded data entry so it actually gets done.
There was a period of time between 2019 and early 2025 where I was a partner in a trucking company. Within it, I lived the day to day of someone in the trenches of logistics. Cancellations, delays, tracking, shortages, rerouting, disagreements, etc. It's an essential industry, but it is an industry of malarkey.
That being said one particular task I despised with every inch of my being was sorting out scanned or even physical paperwork and associating it with shipments in order to collect payment. My disdain for this task led to billing delays that very quickly would stretch into five figures on a regular basis. I eventually ended up hiring this exact task out because bookkeeping frankly exhausted me. But I carried this forward as the motivation to just solve the problem. I am an engineer, after all. So out of this, the freight document parser described here was created.
Originally called DocumentML (and technically still called that) because I had intended to label every bit of paperwork I had in order to create a data set and turn it into my own NER model on SpaCy. I initially didn't want to use an LLM like GPT or Claude. So it started its life as a prototype script in a Jupyter notebook, using Label Studio for annotation. I'd let my PC sit overnight some days to just read documents with DocTR and generate slices of these documents for me to annotate later. However, the "building a NER" path was one that would take far longer than I'd have liked between picking up new skills, labeling what amounted to thousands of slices, and my desire to just solve the problem that I'd felt a personal vendetta against for draining my energy.
So through this journey, I created a system that would extract text from given documents through either pre-existing text or OCR, and turn it into structured JSON that I could actually programmatically use, no matter the shape of the document.
The Process
The Step by Step Process of the Extraction Service.
Upload
The user uploads a file to the service, these files are either PDFs or images prepared for extraction.
The OCR Process
Checks first to see if the file already has text before running a more expensive OCR process.
Candidate Extraction
Once we have the text, we go through the text blocks and gate what is or is not potentially relevant.
LLM Step
Extracted candidates and their context is sent to Claude Haiku along with a prompt and schema.
Validation
Ensuring the response is valid JSON, parsed against the Pydantic Schema, hallucination check against candidate values.
Enhancement
Using Open Cage to fill in empty address model fields, validate addresses, and produce coordinates.
By the end, you get a consistent JSON output that you can use in real world applications.
Is more expensive really more better?
How do you know it's working?
When I had initially created DocumentML, my "testing" and "evaluation" consisted of feeding it a few documents, looking at the output and comparing it to the actual document, then saying something along the lines of "looks good to me!"
After researching more about how people were actually testing and evaluating AI outputs, I came to the conclusion that eyeballing it wasn't enough for me. So I setup Promptfoo as an evaluation harness to determine which model was actually performing better, or at least equally to what I was already using, which was Sonnet. I started off by going through the painstaking process of manually annotating 30 freight documents, getting down details like reference numbers, pick up orders, dates, shipment details, etc. After I set this all up, I ran my first test. Sonnet vs Haiku, and the results were surprising.
Something to note is that I have this as part of the language in my prompt: "Never invent values. If the document does not contain a field, OMIT that field entirely...". When I ran my tests, I was pleasantly surprised to see Sonnet hallucinating more than Haiku on one of my fixtures. When it came to the linear feet of a shipment, Haiku correctly omitted the value because it was not present in the provided candidates, but Sonnet had said the shipment was 53 linear feet. My guess is that this is an assumption the model made off of the equipment type, a Van trailer.
Van trailers in the modern day in a full truckload context are typically 53 feet long, so are many other types of trailers. So Sonnet actually just used this knowledge to just assume the shipment was 53 ft. Catching this detail is important because for all it could have known because it wasn't specified, this could be going on a 28' pup trailer and still be classified as a van shipment. So flagging a non-existent linear shipment length is the correct move.
Finally, another change that I made that was able to cut down on time significantly was changing the output of my LLMs in a meaningful way. Instead of requiring missing fields to output null in the JSON, I asked that they be omitted entirely. This was able to drive down cost because every unneeded "null" was more tokens consumed and more time spent. So dropping empty fields actually helped significantly; it's also what helped catch the previous hallucination.
Saving AI Tokens and Cost with Evals
Right fitting a model
Through my evaluations I was able to determine that Haiku is better than Sonnet for this use-cause because it is better at following explicit directions without making assumptions. Not only that, but it's cheaper too! Cutting cost by roughly 63%.
| Model | Total Runs | Avg Cost Per Run | Total Cost |
|---|---|---|---|
| Sonnet claude-sonnet-5 | 60 | $0.132 | $7.92 |
| Haiku claude-haiku-4-5 | 60 | $0.049 | $2.94 |
| Difference | (0.083) | (4.98) |
Model costs are rounded up or down for readability.
From 60+ seconds to 3-6 seconds.
Performance Optimizations on the OCR Stage
When I had first prototyped this in Jupyter notebooks, it started off by slicing each image and running OCR on each. The idea was to be able to eventually distribute these across OCR processors, but my method of doing this proved to be far too slow between the slicing and reads. Not only this, but I was running accuracy-first models with DocTR (db_resnet50 + master) on all of these slices.
Suffice it to say, each page took about 40-60 seconds + waiting for LLM output, which is an awfully slow experience. I needed a better process, which is why this was entirely rewritten after I had the concepts down from the prototyping stage.
My initial change was moving away from the slicing mechanism and just doing whole pages at once. I also switched to lighter models, namely linknet_resnet18 and crnn_mobilenet_v3_small, which for my use case produced sufficient results when OCR was needed.
The biggest possible gain I was able to make was just using any pre-existing text in the PDF. OCR actually being needed was the less common case, but I left it that way because my candidate processor relied on the positioning and text block outputs that DocTR provided. This changed when I introduced the "fast path".
Rather than write a second extraction path, I normalized the PDF's existing text into the same structure DocTR's export() produces. That way the candidate generator runs identically whether the text came from OCR or straight from the PDF. As a result, if I ever need to make changes to the candidate generation strategy (which is more likely than changing the extraction itself), I don't need to do it in two places.
It should be noted that the lighter-weight models worked better for this use case because this feature oftentimes included rate confirmations that are clean, generated PDFs. A heavier model should be used for matching documents scanned by a cell phone to existing shipments in the system. I would also like to add that all of the OCR was done locally on my Nvidia GTX 1080 GPU, which is hardly current hardware, but it made my optimization work matter more.
Together, all of these changes took a page from 40-60 seconds down to 3-6 through the process, LLM Response Included.
In short
A Document Pipeline to Cut Down Data Entry
DocumentML is evidence that I am able to incorporate LLMs and modern day AI into real solutions that can help businesses quickly bill and input data into their TMS.
If you want to start incorporating AI into your applications, that's something I can probably help you with.