On a badly scanned copy of Enviva Partners’ 2018 Schedule K-1, a dash sits in the box for one item of income. The best AI model in our test read that dash into the box next to it, reported the result with a confidence of 95 out of 100, and moved on. Nothing in its output looked any different from the hundreds of values it had read correctly.
That small miss matters now because K-1 extraction is arriving in ordinary tax software. Drake’s SmartExtract added Schedule K-1 (Form 1065) on October 13 [1], and the vendors’ advice on accuracy is to “review and verify all extracted data” [2], which is the work the tool was supposed to save. We ran five vision-language models and two OCR pipelines over 30 sample K-1s and brokerage 1099s with known answers, 11 times each [3]. The best model was right on nearly every value, and it was exactly as sure of itself when it was wrong: its confidence score never flagged one of its own mistakes. What caught every error was a second model that had to agree with the first, plus a rule that sends every amount the issuer moved to an attached statement straight to a person. A review queue built on the model’s confidence is checking the wrong values.
How accurate the best AI model was on sample K-1s
Gemini 3.8 Flash read all 676 values right in our clean run on October 18, 2026, at $0.0064 a document [3]. The set held three real Form 1065 K-1 packages that Enviva Partners and The Blackstone Group publish for their unitholders, nine simulated scans of them, and 18 made-up 1120-S and 1041 K-1s and consolidated 1099s, each scored against an answer key. It’s a small set with simulated scans, so these numbers describe these documents rather than a firm’s client files.
| Pipeline | Values right, clean run | Wrong values accepted without review, by run | Cost per document |
|---|---|---|---|
| Gemini 3.8 Flash | 676 of 676 | 0 in seven runs, 2 in four | $0.0064 |
| Qwen 3.8 27B (open weights) | 636 of 708 | 25 to 91 | $0.0054 |
| Gemma 4 31B (open weights) | 380 of 740 | 197 to 278 | $0.0068 |
| olmOCR 2 text, read by Gemini 3.8 Flash | 550 of 703 | 68 to 87 (nine runs) | $0.0071 |
| Gemini 3.8 Flash and Qwen 3.8 27B, must agree | 676 of 708 | 0 in all 11 | $0.0118 |
The open models aren’t safe on their own. Gemma 4 31B accepted around 200 wrong values in every run, and Qwen 3.8 27B’s count swung between 25 and 91 on the same documents and the same code.
Gemini’s confidence never flagged its own K-1 misreads
Gemini 3.8 Flash’s confidence score never sent a value to review in any run. Everything it flagged was an amount marked STMT or one it couldn’t parse. In four of the 11 runs it accepted a misread instead: a dash read into box 4 rather than box 5 on a bad scan of the Enviva 2018 K-1, accepted at confidence 95, and in other runs a box shift and a wrong code letter [3]. Each put two wrong values into the output marked as settled.
Research on other documents finds the same. Perfios’s ExtractConf study tested the usual confidence signals on DocILE invoices, where frontier models get 26% of fields wrong. Token probabilities, the model’s stated confidence and agreement across five samples all scored between 0.69 and 0.74 ROC AUC, and all “collapse toward all-positive behaviour at practical thresholds” [4].
It also changes how people check the work. In an experiment with 119 professional auditors, Christian Peters found that auditors “reduce effort and are less effective in reviewing work conducted by automated tools compared to reviewing identical work conducted by a human colleague” [5]. A K-1 value the software marks as certain gets the lightest check from a person, and in our runs those were exactly the values that were wrong.
A second model and the K-1’s STMT rule caught every error
Two simpler checks did the work. When Gemini 3.8 Flash had to agree with the open-weight Qwen 3.8 27B before a value was accepted, nothing wrong got through in any of the 11 runs [3]. The pair sent 87 of 708 values to review, about 12%, and cost $0.0118 a document, nearly double Gemini alone. Most of the extra review came from rows Qwen skipped or filed under the wrong box, so the second model’s errors cost review time and never accuracy.
The rest of the queue comes from the form. The IRS tells partnerships that for “items that can’t be reported as a single dollar amount,” they enter a code with an asterisk and “STMT” to show “the information is provided on an attached statement” [6]. That makes a rule with no model judgment in it: every STMT amount goes to a person, with the statement page beside it. All 24 values Gemini sent to review in the clean run were STMT amounts, and it read all 62 statement lines behind them correctly.
ExtractConf reached the same answer from the research side. Its best signal came from reading each document twice, in two ways that fail differently, and treating disagreement as the warning. That lifted ROC AUC to 0.928 [4].
Reading the K-1 image beats reading OCR text
A vision model reading the page image placed amounts far better than the same model reading OCR text. We gave Gemini 3.8 Flash the text that olmOCR 2, a strong open OCR model, produced from the same pages, and it was right on 550 of 703 values [3]. olmOCR got the characters right. The errors were placement: amounts one box off and coded rows under the wrong box, because once a K-1 becomes text, the grid that says which box an amount belongs to is gone.
This doesn’t make OCR useless. An AWS team building the ConfBench benchmark found that giving a model both OCR text and the image produced the best confidence estimates [7]. The lesson for K-1s is to keep the page image in front of the model.
What to ask before trusting AI to read K-1s this season
Our earlier guide to extracting K-1s and brokerage statements with a review queue recommended sending fields below a confidence threshold to a person, alongside rule-based checks. Our measurements say to put the weight on the rules and on a second reading. Before committing to a tool this season, ask four things.
What sends a value to review? If it’s the model’s confidence, what checks the model? Is every STMT amount and coded row routed to a person by rule? Does a second, independent reading have to agree before a value goes into the return? And on your own documents, how many wrong values were accepted without review?
Every published K-1 performance figure we found is a vendor’s, without a method behind it; Wolters Kluwer, for example, cites efficiency gains of 30 to 70 percent [8]. A firm can get a better number in a week by running a tool over 50 of last season’s K-1 packages, including the hedge-fund, state and badly scanned ones our sample lacked, and counting the wrong values that would have reached the return unflagged.
The best AI models now read K-1s well, and they don’t know when they’re wrong. A firm that builds its review around that will catch the errors a confidence score lets through.
The code, sample documents and answer keys behind these numbers are public in the K-1 extraction demo, and the whole test runs with one command, so a firm can point the same scoring at its own packages.
Sources
This article draws on our K-1 extraction test, Drake Software’s and TaxAct’s SmartExtract pages, the ExtractConf and ConfBench papers on arXiv, the Foundation for Auditing Research study of auditors, the IRS instructions for Form 1065, and CPA Practice Advisor’s report on Wolters Kluwer.
- Drake Software: Drake Tax – Drake SmartExtract (Beta) (October 13, 2026)
- TaxAct: SmartExtract (Beta) Powered by Drake Software (October 2026)
- Fulton Ring: K-1 extraction with a review queue, agent-content-demos (October 18, 2026)
- arXiv: Beyond Logprobs: A Multi-Signal Confidence Engine for LLM-Based Document Field Extraction (June 2026)
- Foundation for Auditing Research: Auditor automation usage and professional skepticism (June 17, 2025)
- IRS: 2025 Instructions for Form 1065 (2025)
- arXiv: Can You Trust the Confidence? ConfBench for Vision-Language Models on Document Extraction (August 3, 2026)
- CPA Practice Advisor: Expanded CCH Axcess Expert AI to Deliver AI-Powered Scan and K-1 Automation (June 29, 2026)
