Reading code is not running it

A lesson from Engineering Judgment When AI Writes the Code. Play it above, or read it through below.

• set -e did check a status. It was not the status that failed.

  The common read: set -euo pipefail stops a script at the first failing command, so a grep that matches nothing aborts the run before anything deploys.

  set -e did not miss anything. It inspected an exit status at the moment the manual says it does, and the status it inspected belonged to local, whose documented return is zero unless it is called outside a function, the name is invalid, or the name is a readonly variable, so the shell saw a success and moved to the next line. set -e ran its check as documented, and by that check nothing had failed.

  The retro on that deploy named the config file, and the action item was a schema check on release.conf. That check catches this one missing key. It does not fix the line that turned a failed read into an empty value and an exit status of 0, and six weeks later that line deployed a second release with an empty channel.

  You can recite the rule, but the rule depends on one detail: which command's status the shell inspects, and when. Reading the script tells you set -euo pipefail is present and that whoever wrote it meant it. It does not tell you which of the statuses produced on that one line the shell inspected, and re-reading does not resolve it, because the answer is not in the text.

• Splitting one line into two changed the exit status. It changed nothing else.

  The common read: I know how set -e behaves, so a rule I can recite is a rule I would have applied while reading this file.

  Name the exit status of a script identical to the last one except that local channel sits on a line of its own, with the assignment beneath it. Nothing else changes: same file, same grep, same missing key, same two echo lines, same header. The two scripts behave differently, although a reader would describe both in the same words.

  The second exits 1 and prints nothing at all. An assignment with no command name in front of it takes the exit status of its last command substitution, so set -e sees the 1 that grep produced, leaves the function before the first echo reaches stdout, and the whole script exits. Splitting one line into two stopped the release. Every token a reviewer reads as meaningful is identical across the two versions.

  The review that approved the first version left one comment and it was about the function name. Commenting on the name was not careless. Naming is the part of a diff a reader can evaluate by reading, so it is the part that gets evaluated, and the exit status that decides whether the release runs cannot be seen by reading.

  The reader already knew the rule and did not know which line it applied to. That question has one cheap answer: run both versions and print $?, and the two exit statuses differ. Running both takes about ten seconds, and the alternative is a release that reports success. The check is worth running on any change where a silent wrong action costs more than a rerun, and not worth running anywhere else.

• forEach never waited. The count was printed before the pushes ran.

  The common read: await inside the callback makes the loop wait, so the count on the line after it is the count once the work is done.

  The await did pause, but it paused only the callback. It suspends the callback's own frame until the 10 ms pause resolves, and forEach does not wait for that frame: the specification's loop performs the call, binds nothing, and returns undefined. Three async calls become three promises that no code awaits, the loop finishes on the tick it started on, and the line beneath it reads a length that is still zero.

  No id was lost. Swap forEach for a for...of and the same flush reports 3 on the same line, because a for...of body runs inside the frame the await suspends. Both versions were run. The forEach version reaches 3 as well, after the print, and a check at 50 ms reads 3. The work happens after the only line that reports it.

  The incident doc said the drain had failed and asked for a retry on the flush path. The drain had not failed. Every id was recorded, milliseconds after the line that reported none of them, and the retry that shipped writes each id a second time whenever the pauses run long. Nothing in that file lost an id, and the retry was added because of a count that was only printed too early.

  async changes how one function's frame runs, and that frame is not always the one you are reading. Inside the callback, await pauses as documented. At the boundary, whoever called the callback decides whether anything waits, and forEach decided that before this file was written. Reading the line tells you the await is there. It does not tell you which code awaits the promise, and re-reading does not resolve it, because the answer is in the caller and the caller is a built-in.

• Explaining every line is a different thing from naming what the code prints.

  The common read: I can explain what this code does, so I know what it outputs.

  Explaining code and predicting code are different acts, and only one of them can be checked. An explanation names the parts and says what each part is for, and every file in the table has a correct one, short enough to give out loud, that anybody who has read the file could produce without hesitating or looking anything up. A prediction is a commitment to a value on an input, and the run either confirms it or contradicts it.

  Reading a familiar shape is fast because the shape is a chunk you already hold, and a chunk includes the rule you learned it with. It does not include the rule's exception, because the exception was never in a file you had to read closely. Fluency is reading speed. Comprehension is what a correct prediction demonstrates, and no other step in a review asks for a prediction.

  The approval on that release took four minutes and the timeline records the four minutes. It does not record that the approver, asked forty minutes later what the function returned when the key was missing, could not answer. No step in a review has ever asked for that answer and no field in any review tool holds it, so the one number that would have measured comprehension is the one number nobody collects.

• A prediction takes ten seconds. Nothing else in a review measures comprehension.

  The common read: careful reading is the expensive part of a review, so running the code is an optional extra for the times there is room for it.

  Approve one file today without naming what it prints on the input it will see in production, and the defect that ships is one that careful reading would not have caught. A prediction costs about ten seconds. It is the only step in a review whose result can show you are wrong, and that is its value: a wrong prediction gives you information, and a reading that finds nothing wrong does not.

  You predict one value on one input, before the run: a number, a string, a count or an exit status, not what the function is for, whether the approach is right or whether the names are good. The commitment has to be specific enough to be wrong, which is why "it looks correct" is not one and "it returns an empty list" is.

  The comment left on that review was about the function name, and the reviewer was right about the name. Naming is the part of a diff that reading can evaluate, so it is the part reviews evaluate, and the ratio of naming comments to behaviour comments in any review tool measures what reading can evaluate rather than what mattered.

  The check has a cost, and it does not scale to every value. Predicting every value in a large change is not affordable, so make one ten-second prediction per file, and the file that gets it is the one whose failure costs more than a rerun. The ticket that came out of that release was called config parsing improvements, which is accurate and names none of the failures that happened.

• The check has the same rounding as the value it checks. Predict the output, then run it.

  The common read: parsing a payload and serializing it back is an identity, so an equality check against the literal I typed would catch it if it were not.

  The round trip is not an identity, and the check that was supposed to catch the change cannot detect it. Parsing turns the text into a double, and above 2^53 - 1 not every integer is representable, so the value is rounded to the nearest representable one and serialization prints the rounded value. The id changed by one in transit. Nothing in the function is wrong and the run reports no error.

  The equality check on the second line is the part to remember. The literal in the comparison is parsed like the payload, rounds to the same double, and the comparison passes. A check that stays inside the same representation cannot detect the rounding. Three other checks were run on the same value, and all three report a difference: Number.isSafeInteger says the parsed number is past the safe range, its string form is not the string that arrived, and a BigInt comparison against the original digits is false.

  The timeline named the upstream service as the source and never recorded that the check passed. The prescription is one line long and it is not about being more careful. Write down what the code prints, then run it. Reading tells you what the code is for and running tells you what it does, and across the five files in this lesson those two answers differed every time, with no diagnostic, no warning and no line in any log that would have made anyone look again.