Nodding along to AI work feels like review. Two months ago, 49 of my pieces passed three automatic checks (prose style, format and search) with no flags. A reader who was not in my meetings still could not follow one of them.
Then a draft I asked a model to polish passed every check again. It lacked every embed, every tagged creator and the short summary. Each check looked for what was wrong. None asked what was missing.
Work you must re-check by hand each week is rented work, because you pay again and never own it. Daniel Kahneman's book Thinking, Fast and Slow names the two modes behind it. System 1 is fast, automatic thinking. System 2 is slow, effortful thinking. AI supplies the fast kind in seconds. Teams moving to AI native work must supply the slow kind themselves.
The short version.
AI is System 1. It answers in seconds and sounds sure.
Skimming its answer is fast thinking checking fast thinking.
Slow thinking must happen twice. Write the definition of good first, then review the output against it.
Every miss becomes one new line in the file the AI reads next.
Start with one task and a short checklist. It takes under 30 minutes.
Reading AI Output Is Not System 2 Thinking
The puzzle Kahneman made famous takes 10 seconds. A bat and a ball cost $1.10 together. The bat costs $1 more than the ball. What does the ball cost?
The answer that jumps out is 10 cents. That jump is System 1. The correct answer is 5 cents. You only reach it when you slow down.
AI sounds sure, and that tells your brain the answer is fine. Kahneman found that System 2 is lazy, so it often approves what System 1 offers.
A Wharton study named the result cognitive surrender, which means people accept what the AI says, even when it is wrong. Alberto Romero says the first cost is that "you let more wrong answers pass as correct".
Advait Sarkar's TED talk asks how to keep AI from killing your critical thinking.
A Judge Model Is Another System 1
The usual fix is a judge model, a second model that grades the first. It scales well and still thinks fast.
Cameron R. Wolfe, Ph.D. names the flaw: "the judge tends to favor responses that are generated by itself".
A NeurIPS paper found the same effect. Models scored their own text higher, and human raters judged the same texts equal.
Slow thinking needs a mechanism the output cannot sway. A definition of good written before the output exists is the first. A test that runs the code is another. A list of required parts is the third.
Write the Definition of Good Before You Prompt
The first slow step runs before the AI does. Write a short checklist with three parts.
Good looks like. What a right result shows.
Bad looks like. Real misses from past runs.
Evals. Yes or no checks you run on every output. Teams use this word for scored checks on AI work.
The evals are the part my checks lacked. They only hunted for flaws. A draft missing its embeds has no flaws to find, so it scored zero. An eval that asks "is the video there?" fails a draft for what it lacks.
Hugo Bowne-Anderson puts the order plainly: "you need to know what the product is supposed to do, who will use it, and what a successful result looks like".
Aman Khan says a judge needs the same input. You must "clearly define what success and failure look like" for it.
@realmadhuguru says teams treat evals as a late QA step, when evals are the product spec.
Write the checklist before you see any output, so the output cannot set your standard. Bad inputs cause AI slop, meaning machine-shaped prose a checker can measure.
Every Correction Must Leave One Rule Behind
The second slow step comes after the output. Review it against the checklist, never by feel. Each miss becomes one new line.
This loop is how a person becomes an expert. Kahneman and Gary Klein agreed that expert intuition earns trust in a regular environment with fast, honest feedback.
The model does not carry your review to the next run. The file it reads next time does. The checklist makes the environment regular, and each review is the feedback. Put it in the file your tool reads first, as this post shows.
When a rule stops failing, move it into a test or a judge model. Jev is a model built for System 1 judgments that returns a choice and a score for each question you ask. That puts System 2 inside System 1, and the speed stays.
A review that leaves nothing behind is rented work. You pay for it again next week.
I have not run a controlled test of this loop. It rests on two failures of mine. One added a clarity check, and the other added a gate that fails on what is missing. The Wharton label is contested. Tina Austin warns it "sounds like a diagnosis, even though the underlying mechanisms remain poorly specified".
Tonight, Review One Task and Leave One Rule
Below is the checklist for an AI image task. Swap in the task your team re-prompts most and fill it in 20 minutes. Run it once and review it against the list. Write the first miss as one new line, then run it again.
TASK: product photo of a ceramic mug for a shop page
WHAT GOOD BAD
Shape one handle, round rim extra handles, warped rim
Color matches the brief washed out or off color
Text none fake logo, garbled words
Backdrop plain, soft shadow busy, props steal the eye
Framing mug fills the frame cropped or tiny
EVALS, run on every image. Any "no" fails it.
1. Is there exactly one handle?
2. Is the color the one in the brief?
3. Is the image free of text and logos?
4. At thumbnail size, is it clearly a mug?
5. Is the whole mug inside the frame?
AFTER EACH RUN: add one line per failA working checklist means the same miss stops returning.
