Jerry Liu
@jerryjliu0
Parsing the world's hardest PDFs @llama_index. cofounder/CEO
LlamaParse:
Enterprise:
1.6K Following 87K Followers
GPT 6.1 Terra is a strong frontier model baseline for document parsing.
✅ It spikes at table parsing, coming in 5th on our overall leaderboard, ahead of 6 Sol and behind Opus 5.5 / Astra
✅ It is decent at content faithfulness and reading order
✅ It struggles on layout, charts, and formatting
Coming in at ~1c per page, it's not a bad option for table-heavy pages! You can use these alongside specialized OCR solutions (like LlamaParse) that will give more consistent performance across a variety of dimensions.
Check it out on ParseBench:
Show more
We benchmarked Gemini 4 Argon, Muse Wonder, and Muse Spark comprehensively on hard document OCR tasks.
They're all pretty good!
- Muse Spark and Gemini 4 Argon* are SOTA at their respective price points for table understanding among frontier models.
- Wonder and Argon are also quite good at chart understanding, reading order, and formatting recognition
It's exciting that frontier models are starting to ramp up on visual understanding capabillities.
We're committed to open benchmarking of all available models even as we aim to build the best specialized models for document parsing and extraction.
You can check out the full set of models on OpenDocRouter:
If you want a dedicated OCR solution without choosing, come check out LlamaParse:
Show more
These days, you can pretty much solve any task by defining an eval and hillclimbing over it, instead of directly defining the deterministic/agentic workflow to solve it.
The data provider companies' entire job is to define evals for all economic activity to make the frontier models capable of doing anything.
Your job then becomes pointing frontier intelligence in the right direction - defining what to solve, and how to measure what good looks like.
Agent application interfaces will evolve to capture this. The most complex processes will still need some sort of explicit workflow builder interface, but most tasks can be compressed into goals and eval instructions.
Show more
your ai productivity is strongly correlated with your wpm
idk what cosign is but help a girl out with an invite
We comprehensively evaluated 16 recent frontier VLMs - including Opus 5.5 and GPT-6 Sol/Luna - on whether higher effort led to better document parsing performance.
Higher effort typically leads to improvements on other benchmarks (coding, knowledge work), but up until recently it wasn't obvious that this natively improved capabilities for reading PDFs.
Results:
✅ Out of the frontier models, Opus 5.5 has the best performance relative to its price. It's especially good at parsing tables.
✅ Astra is also quite good, but starts at a more expensive price than Opus.
✅ GPT-6 Luna is more compelling at the cheaper end of doc parsing
If you're parsing documents at scale, you'll still want a dedicated OCR solution like LlamaParse ( that has better performance at a cheaper price.
But if you're parsing docs "in the agent loop" within an app like Codex/Claude code, and you're too lazy to integrate a dedicated solution, then Opus 5.5 is currently the leader.
Full results on ParseBench:
Show more
blank cells in tables can mess up agent decision making
many document OCR tools struggle with reading multiple blank cells in a document and cause values to be shifted from the original column and rows. this causes the wrong numerical interpretation downstream.
This is a nice illustration of what happens!
If you have complex documents where table accuracy is extremely important, come check out LlamaParse:
Show more
A blank cell can change the meaning of a forecast.
Four times a year, the Fed's 18 top policymakers each put their forecasts on paper: where growth, jobs, inflation, and interest rates are headed. This is September 2026's edition, home of the "dot plot" that markets treat as the Fed tipping its hand. This release is the closest thing to the Fed saying what it plans to do. Most analysts will want to throw this documents to an AI agent, but this messy doc is dense: full of complex tables and charts that hold valuable context. All things that frequently trip up raw LLM APIs.
The Fed’s September 2026 projections table includes a 2029 column, but its June comparison row leaves that cell empty. This is a common but silent failure point for document parsers.
We parsed page 2 with LlamaParse and checked the displayed GDP median excerpt against the original PDF. All nine numbers matched and the data stays aligned in the returned HTML.
Try LlamaParse on a table where headers and missing cells matter.
Source:
Show more
i always appreciate some great writing, we have too little of it these days
Here’s to the wanting. Here’s to the dreaming. Here’s to everyone.
A blank cell can change the meaning of a forecast.
Four times a year, the Fed's 18 top policymakers each put their forecasts on paper: where growth, jobs, inflation, and interest rates are headed. This is September 2026's edition, home of the "dot plot" that markets treat as the Fed tipping its hand. This release is the closest thing to the Fed saying what it plans to do. Most analysts will want to throw this documents to an AI agent, but this messy doc is dense: full of complex tables and charts that hold valuable context. All things that frequently trip up raw LLM APIs.
The Fed’s September 2026 projections table includes a 2029 column, but its June comparison row leaves that cell empty. This is a common but silent failure point for document parsers.
We parsed page 2 with LlamaParse and checked the displayed GDP median excerpt against the original PDF. All nine numbers matched and the data stays aligned in the returned HTML.
Try LlamaParse on a table where headers and missing cells matter.
Source:
Show more
i like opus 5.5 a little more than astra, but i like the codex app more than the claude app
Muse making me think we should’ve done it 😮💨
come join us at SF @Techweek_ !
building in the agent space? this is your room at SF @Techweek_.
think modern gatsby: black tie, red carpet, cocktails, live jazz & art deco glamour.
300+ founders, founding operators & investors across agent products, infra & dev tools.
5 cohosts sending exclusive invites:
@llama_index - ai agents for doc ocr
@thelinqapp - messaging & voice infra
@lemma_ - observability & evals
@Tiny_Fish - web infra
@obvious - the ai agent that builds with you
supported by @SpaceXAI, @MongoDB, @daytonaio & @arceuslegal.
dropping 30k for you to meet your next customer, investor, partner or hire. maybe even your soulmate. have a good time doing it.
want in? reach out to a cohost for an invite.
Show more
in the spirit of sus mascots, this was in our drafts a few months ago
(we did not proceed with this)
predicting model uncertainty is a hard problem.
perfectly "calibrated" confidence scores are exact probability values on whether the output is correct.
this is extremely important for agentic decision making, including document extraction. i made a sick video below showing how confidence scores can be used to choose decision thresholds and vary precision / recall.
if you set a really high threshold, then you automate less, but more of the automated extraction is correct. if you set a low threshold, then you automate more, but there's more errors in the extraction.
check out our blog!
Show more
confidence scores only matter if they help you decide what to automate.
for document extraction, that usually means knowing how much work you can safely accept at a given precision target. in our latest post, we look at confidence scoring through that lens, including:
✅️ confidence cutoffs
✅️ precision vs. recall
✅️ score coverage
✅️ score granularity
✅️ human review volume
using ExtractBench, we compare how different extraction systems perform after confidence filtering. at a 97% precision target, LlamaParse Agentic Plus reached 66.48% recall on expected fields after filtering.
the useful part of a confidence score isn’t the number itself. it’s whether you can use it to control automation and review in production.
👉️ read the full post:
Show more
This is a fantastic blog post from our research team on the importance of calibrated confidence scores.
Here it's in the context of document extraction, it's also extremely important for more general agentic decision making (eg with Jev)
When you set a confidence threshold, you can choose to automatically accept values above that threshold and do HITL review of values below the threshold.
The higher the confidence threshold, the higher precision you're able to guarantee (e.g. confidence of 0.7 could mean 95% precision, confidence of 0.9 could mean 98% precision), but of course the more human review you'd have to do on false negatives.
We've put in a lot of work to make sure our confidence scores are well calibrated and represents real uncertainty over complex documents in production.
Come check out our blog:
LlamaParse:
Show more
confidence scores only matter if they help you decide what to automate.
for document extraction, that usually means knowing how much work you can safely accept at a given precision target. in our latest post, we look at confidence scoring through that lens, including:
✅️ confidence cutoffs
✅️ precision vs. recall
✅️ score coverage
✅️ score granularity
✅️ human review volume
using ExtractBench, we compare how different extraction systems perform after confidence filtering. at a 97% precision target, LlamaParse Agentic Plus reached 66.48% recall on expected fields after filtering.
the useful part of a confidence score isn’t the number itself. it’s whether you can use it to control automation and review in production.
👉️ read the full post:
Show more
confidence scores only matter if they help you decide what to automate.
for document extraction, that usually means knowing how much work you can safely accept at a given precision target. in our latest post, we look at confidence scoring through that lens, including:
✅️ confidence cutoffs
✅️ precision vs. recall
✅️ score coverage
✅️ score granularity
✅️ human review volume
using ExtractBench, we compare how different extraction systems perform after confidence filtering. at a 97% precision target, LlamaParse Agentic Plus reached 66.48% recall on expected fields after filtering.
the useful part of a confidence score isn’t the number itself. it’s whether you can use it to control automation and review in production.
👉️ read the full post:
Show more
building in the agent space? this is your room at SF @Techweek_.
think modern gatsby: black tie, red carpet, cocktails, live jazz & art deco glamour.
300+ founders, founding operators & investors across agent products, infra & dev tools.
5 cohosts sending exclusive invites:
@llama_index - ai agents for doc ocr
@thelinqapp - messaging & voice infra
@lemma_ - observability & evals
@Tiny_Fish - web infra
@obvious - the ai agent that builds with you
supported by @SpaceXAI, @MongoDB, @daytonaio & @arceuslegal.
dropping 30k for you to meet your next customer, investor, partner or hire. maybe even your soulmate. have a good time doing it.
want in? reach out to a cohost for an invite.
Show more
sorry to every other sf tech week host in advance,
but we may have gotten carried away planning our event...
we're about to host the biggest and most lavish black-tie gala sf tech has ever seen.
live jazz. drinks under the art deco lights of the famous redwood room. the best founders and investors from across the AI agent economy will be dressed like its the roaring 20s.
oct. 9. rsvp below 🥂
Show more
we've built the world's most advanced engine for document extraction over complex documents
the video below shows an overview of LlamaExtract Agentic Plus in action. long tables, giant forms, calibrated confidence scores, and grounded extraction.
if you have complex document extraction use cases and existing vendors aren't cutting it, let us know:
come sign up to LlamaParse today:
Show more
Excited for next Tuesday, Sept 29, and our monthly @daytonaio AI Builders Demo Night in SF!
We’re hosting at @WorkOS SF Office in partnership with @llama_index.
Join us for an evening of great demos, conversations, and networking with the AI builder community.
Event details & RSVP ⬇️
Show more
