Slack chat where Atharva calls AI slop solvable

Nodding along to AI work feels like review. Two months ago, 49 of my pieces passed three automatic checks (prose style, format and search) with no flags. A reader who was not in my meetings still could not follow one of them.

Then a draft I asked a model to polish passed every check again. It lacked every embed, every tagged creator and the short summary. Each check looked for what was wrong. None asked what was missing.

Work you must re-check by hand each week is rented work, because you pay again and never own it. Daniel Kahneman's book Thinking, Fast and Slow names the two modes behind it. System 1 is fast, automatic thinking. System 2 is slow, effortful thinking. AI supplies the fast kind in seconds. Teams moving to AI native work must supply the slow kind themselves.

The short version.

AI is System 1. It answers in seconds and sounds sure.
Skimming its answer is fast thinking checking fast thinking.
Slow thinking must happen twice. Write the definition of good first, then review the output against it.
Every miss becomes one new line in the file the AI reads next.
Start with one task and a short checklist. It takes under 30 minutes.

Six-panel stickman comic: a person nods at fast AI work, then a checklist and a test catch the gap

Reading AI Output Is Not System 2 Thinking

The puzzle Kahneman made famous takes 10 seconds. A bat and a ball cost $1.10 together. The bat costs $1 more than the ball. What does the ball cost?

The answer that jumps out is 10 cents. That jump is System 1. The correct answer is 5 cents. You only reach it when you slow down.

AI sounds sure, and that tells your brain the answer is fine. Kahneman found that System 2 is lazy, so it often approves what System 1 offers.

A Wharton study named the result cognitive surrender, which means people accept what the AI says, even when it is wrong. says the first cost is that "you let more wrong answers pass as correct".

Advait Sarkar's TED talk asks how to keep AI from killing your critical thinking.

A Judge Model Is Another System 1

The usual fix is a judge model, a second model that grades the first. It scales well and still thinks fast.

names the flaw: "the judge tends to favor responses that are generated by itself".

A NeurIPS paper found the same effect. Models scored their own text higher, and human raters judged the same texts equal.

Slow thinking needs a mechanism the output cannot sway. A definition of good written before the output exists is the first. A test that runs the code is another. A list of required parts is the third.

Three reviewers compared by what each one can catch

Write the Definition of Good Before You Prompt

The first slow step runs before the AI does. Write a short checklist with three parts.

Good looks like. What a right result shows.
Bad looks like. Real misses from past runs.
Evals. Yes or no checks you run on every output. Teams use this word for scored checks on AI work.

The evals are the part my checks lacked. They only hunted for flaws. A draft missing its embeds has no flaws to find, so it scored zero. An eval that asks "is the video there?" fails a draft for what it lacks.

puts the order plainly: "you need to know what the product is supposed to do, who will use it, and what a successful result looks like".

says a judge needs the same input. You must "clearly define what success and failure look like" for it.

@realmadhuguru says teams treat evals as a late QA step, when evals are the product spec.

Write the checklist before you see any output, so the output cannot set your standard. Bad inputs cause AI slop, meaning machine-shaped prose a checker can measure.

Every Correction Must Leave One Rule Behind

The second slow step comes after the output. Review it against the checklist, never by feel. Each miss becomes one new line.

The loop between System 2 and System 1, with rules moving into tests

This loop is how a person becomes an expert. Kahneman and Gary Klein agreed that expert intuition earns trust in a regular environment with fast, honest feedback.

The model does not carry your review to the next run. The file it reads next time does. The checklist makes the environment regular, and each review is the feedback. Put it in the file your tool reads first, as this post shows.

When a rule stops failing, move it into a test or a judge model. Jev is a model built for System 1 judgments that returns a choice and a score for each question you ask. That puts System 2 inside System 1, and the speed stays.

A review that leaves nothing behind is rented work. You pay for it again next week.

I have not run a controlled test of this loop. It rests on two failures of mine. One added a clarity check, and the other added a gate that fails on what is missing. The Wharton label is contested. warns it "sounds like a diagnosis, even though the underlying mechanisms remain poorly specified".

Tonight, Review One Task and Leave One Rule

Below is the checklist for an AI image task. Swap in the task your team re-prompts most and fill it in 20 minutes. Run it once and review it against the list. Write the first miss as one new line, then run it again.

TASK: product photo of a ceramic mug for a shop page

WHAT       GOOD                      BAD
Shape      one handle, round rim     extra handles, warped rim
Color      matches the brief         washed out or off color
Text       none                      fake logo, garbled words
Backdrop   plain, soft shadow        busy, props steal the eye
Framing    mug fills the frame       cropped or tiny

EVALS, run on every image. Any "no" fails it.
1. Is there exactly one handle?
2. Is the color the one in the brief?
3. Is the image free of text and logos?
4. At thumbnail size, is it clearly a mug?
5. Is the whole mug inside the frame?

AFTER EACH RUN: add one line per fail

A working checklist means the same miss stops returning.

How I Ship 200+ Pieces at AI Speed Without…