One of the recent papers that came across my X feed was a researcher digitizing a corpus of old Sears catalogs to estimate inflation more accurately:
The technical skills to do this work are closely related to many of the projects I am working on at Gainwell – large-scale document processing using LLMs. So if you are a PhD student and your thesis involves work like this, I would likely want to hire you for a six-figure job as a data scientist at Gainwell.
I wanted to have a quick blog post on some of the tools to process documents. So this researcher processed 180 Sears catalogs that are typically well over 1000 pages, so they likely dropped well over $10k on this project.
Like I said in the X post, for this type of volume processing, you shouldn’t be using the main models. You should first consider the cheaper models, like the flash-lite models from Gemini (not even the flash for this), or the mini/nano models from OpenAI. Using these cheaper models, which I suspect would be of similar accuracy, would reduce the cost to more like $2k in this project.
For those even more budget-conscious (I am processing this volume often daily at Gainwell for different projects), here are a few of my notes on open source models. (Also I am concerned OpenAI will entirely drop the mini/nano models in the future, hence I want to make sure I have plenty of options.)
OCR and Markdown
So this project used the images to help classify the information, but if you are dealing with pure text, a common approach is to first OCR the document, and then apply structured extraction from the text. This works well to save costs even if using served models, as we are talking about an image of a page being 10k tokens, but the extracted text often being well under 1k tokens.
In LLMs for Mortals, I show using the docling library to do this:
If the PDF is already OCRed (it has the underlying text) there are Python libraries to directly read the text. This example uses pypdf, but a better default library is pypdfium2 (it is much faster).
The difference between converting to Markdown vs. reading the text is that Markdown conversion will be a bit nicer in converting tables, graphs, and page chrome (e.g. footers/headers).
I have had good success with docling (which you can have it convert to Markdown with just the page text), but it does load in a PyTorch model (which adds a bit of latency). You can turn it off to only use the text in the PDF (and not the image), which does save time but still has a fair bit of latency.
If the PDF has the text embedded and you do not want a big model, I have been happy with the results of the liteparse library. This does not use an LLM at all to do the conversion to Markdown, so is quite fast.
Due to the increased cost, if you do need a served model, (sometimes you just have latency requirements and you don’t want to deal with your own server), Mistral OCR is worth checking out. This is a good cost-effective alternative if you are using AWS Textract and need things like table extraction.
There are many different models and providers coming out in the space (OvisOCR2 is one of the recent models for OCR, for served models the group that created liteparse (LlamaIndex) also has served model options).
Structured Extraction
OK, so now you have your pages in text that you can use. At this stage, you will want to extract out information. My go-to local model for named entity recognition (NER) is GLiNER, but the GLiNER2 library has examples that do the full structured extraction.
For a primer on the distinction between NER and structured extraction, pretend you had a free-text narrative “PA note: Vitals checked: HR 78 bpm, SpO₂ 98% on RA. Pt tolerated assessment well, no acute distress noted.”.
For NER, if you extracted vitals you would get back something like:
from gliner import GLiNER
# load GLiNER model
model = GLiNER.from_pretrained("urchade/gliner_small-v2")
# Physician Assistant narrative
narrative = """PA note: Vitals checked: HR 78 bpm, SpO₂ 98% on RA.
Pt tolerated assessment well, no acute distress noted."""
# define entities to extract
labels = ["heart_rate","oxygen"]
# extract entities
entities = model.predict_entities(narrative, labels)
print(entities)And this prints out:
[{'start': 25, 'end': 34, 'text': 'HR 78 bpm', 'label': 'heart_rate', 'score': 0.9038965106010437}, {'start': 36, 'end': 43, 'text': 'SpO 98%', 'label': 'oxygen', 'score': 0.8429999351501465}]So it identifies the specific text. Structured extraction can actually identify additional information. So here I just say collect all vitals, and identify if the note is from a physician assistant or nurse.
from gliner2 import GLiNER2
# Load GLiNER2 model
extractor = GLiNER2.from_pretrained("fastino/gliner2-base-v1")
# Physician Assistant narrative
narrative = """PA note: Vitals checked: HR 78 bpm, SpO₂ 98% on RA.
Pt tolerated assessment well, no acute distress noted."""
schema = {"clinical_note": [
"clinician_role::[nurse|physician assistant]::str::Role of the healthcare professional",
"vitals::list::Vital signs collected"
]}
# Define structured extraction schema
results = extractor.extract_json(narrative,schema)
print(results)And this prints out:
{'clinical_note': [{'clinician_role': 'physician assistant', 'vitals': ['SpO 98% on RA', 'HR 78 bpm']}]}So it filled in the clinician role without the actual words “physician assistant” being in the actual note (it smartly inferred that from the “PA note” part).
NER is actually one of the techniques that local models are really the only viable approach, but structured extraction is still a case where I default to the small frontier models. But small open source models are worth testing (these can easily be run on a CPU).
They are just different use cases. NER is good for redacting info and then forwarding to an LLM (see OpenAI’s model, although I have had good success using GLiNER for this exact use case as well). But most of the time I want actual structured extraction – so turning PDFs/images into actual data you can put into a table.
If you want to skip the OCR part entirely and just submit images, see the NuExtract model (and follow Gio’s work, he is often writing examples of these).
If you are using Databricks, it has similar structured extraction tasks compiled down to UDFs that are directly available in SQL. (Also FYI, Teradata has a similar option to take models as an ONNX or PMML file and create a UDF function.)
Advice
So I talked a lot; let’s try to recap in some simple advice.
-
Do not use the main frontier models (e.g. Sonnet, Sol) to do document processing and extraction tasks. Should default to trying models like Gemini Flash-Lite and gpt-nano at first to see if they work. Frontier models are both more expensive and often not any better than the tinier models.
-
If you do not need to extract info out of images, converting documents to text first (and maybe converting that text to markdown) is a way to make extraction cheaper/faster than using images directly. There are multiple local models that run reasonably fast on CPU that can do this.
-
If you need NER or structured extraction, check out GLiNER2. I would still default to using the cheaper frontier models for this often. But if they are too expensive (or you are a cheap grad student), the local models may be totally sufficient.
I am often using the frontier models at work, as I have throughput requirements and I default to not running a server as much as I possibly can. For production APIs, the latency requirements are often hard.
But in the example you are a grad student and have a corpus of 30k pages you want to churn through, you can just have your laptop burn through them overnight for a week if you want to cut costs.
Andrew Wheeler 