In short. You asked an AI to do the same job ten times and got ten slightly different results. That job does not belong to the AI. AI is good at judgement; programs are good at repetition. This post gives one test that separates the two. It also shows what to do with the work that lands on the program side.
A free app I built myself. Give it a try.
There is one test - can the answer be fixed by a rule?
Say the same input must always give the same answer. If you can write that answer down as a rule, it is a program. If the input differs every time and the answer depends on context, it is AI. That sounds abstract. In practice one question settles it: "Can I write the pass condition in code right now?"
| Question | Belongs to a program | Belongs to an AI |
|---|---|---|
| Same input, same answer | Required | Not required |
| Stating the rule | Written in code | The exceptions never end |
| When it is wrong | A bug; fix it and it stops | A misjudgement; it can recur |
| Cost per run | Close to zero | Tokens on every call |
| Auditability | Left in a diff | Left only in a conversation |
The last two rows weigh the most in practice. When the rule lives in code, the diff shows who changed it and when. When the rule lives only in a prompt, nothing explains why today's verdict differs.
Programs win at counting
My blog skill has one script that measures the length of a draft. It converts a notation such as 1~2p into a character count. Then it measures the finished draft against that target range. The conversion gives 1,615 to 3,570 characters. It gives the same number however many times you run it.
What happens if you hand that calculation to an AI is the point. In each conversation it estimates "a page is roughly 1,700 characters" again. PASS and SHORT are then separated on top of that estimate. The boundary moves from session to session, and the reason it moved is recorded nowhere. Counting, converting, looking up, comparing and searching all sit on the program side.
The AI builds that tool too
Deciding that a program is the better fit does not mean "go and write it yourself". Have the AI write the script, and from then on the AI only calls it. Anything that runs straight from a shell will do. Pick whatever is already installed in the target environment from bash, PowerShell, batch or Python.
python blogstat.py post.md --target "1~2p" # PASS / SHORT / LONG, exit code 0 or 1With an exit code attached, the AI does not interpret the output; it just branches on it. One convention follows from this. Do not copy the numbers into the instruction document. Write the same number in two places, code and prose, and they will diverge. After that nobody knows which one is right. The document says only "ask the script".
AI wins at judgement
The rule ends where the script reports SHORT. What to add is not a rule. Padding the same point fills the count and makes the writing worse. That distinction does not get written in code.
I saw the same boundary in a keyword research script. Fetching search volume and document counts is an API call, so it is a program. Is the person who typed that query looking for information, or trying to reach a particular site? That is a judgement. I approximated that judgement with a suffix rule and put it behind an option. It stays an approximation, and the final call belongs to a person or an AI.
Inferring a cause from an error log goes in the same column. So do weighing design alternatives and naming things.
The answer is an order
Real work usually mixes both qualities. So it is not a matter of choosing one side but of deciding the order.
- Split off the part that has hardened into a rule and make it a script.
- Leave the AI only the judgements the script cannot make.
- When the AI makes the same judgement three times, look again at whether it has become a rule.
Step 3 is the entrance to the loop. A judgement that came out the same three times is already a rule. At that point it goes back down to step 1.
Summary
- If the same input must give the same answer it is a program; if the answer turns on context it is AI.
- For the program side, have the AI write the script, and after that only call it.
- Keep the rule in one place. Write the same number in the document and in the code and they will diverge.
Open a recent conversation with an AI. Find the place where you repeated the same instruction twice or more. Asking for that instruction to be turned into a script is the next step.