Post-mortem

My AI oversight tool was blind for 20 days and told me it was healthy

23 July 2026 · William Ryan Hunt

I build a thing called HELM. It watches Claude Code agents running on my machine and clicks "Allow" on the permission prompts, so a long autonomous run doesn't stall at 2am waiting for a human who is asleep. Supervisor software. The whole product is one promise: when your agent needs a human, something reliable stands in.

On July 23rd I found out it had approved exactly nothing since July 3rd.

Twenty days. It had been running the entire time. Its self-test reported Self-check OK (8/9) on every single run. Its log filled up with what looked like successful approvals. And it had not clicked a single button.

Here's the anatomy, because every layer of this failed in a way that's worth stealing.

Layer 1: the window went dark and nothing said so

Claude Desktop is Electron, which means Chromium, which means the accessibility tree is lazy. Chromium only builds the tree for its renderer content when it believes an assistive technology client is listening. When it decides nobody's there, it doesn't build it.

At some point around July 3rd, it stopped believing.

What that did to HELM: UIA could still see the native shell: the window chrome, the minimize button, the sidebar with my session list. All present. So HELM still listed every session, still drew its panel, still looked completely alive. But the conversation pane, the part of the window where the Allow button physically lives, was not in the tree at all.

Not renamed. Not moved. Absent.

I found this by dumping every UIA control in the window and counting. 641 nodes, and every single one of them was chrome or sidebar. Zero content. The button HELM existed to click was not a thing it could see.

approve_button=None    <- 3,551 consecutive times

The fix turned out to be four lines of Windows API: send WM_GETOBJECT to the window and its render widget, and Chromium builds the tree. Tree went from 641 nodes to 1,191, content buttons from 0 to 81.

One detail cost me an hour and is worth writing down: SendMessageTimeoutW does not work for this. I tried it first, being a good citizen who doesn't want to block on a hung app. It moved the tree from 509 nodes to 539. Noise. Only the blocking SendMessageW actually triggers the accessibility build. So it runs on a throwaway daemon thread instead, which gets the same safety without lying to myself about whether it worked.

Layer 2: the diagnostic was pointing at the wrong thing

HELM already had a diagnostic for exactly this scenario. It printed the buttons it could see, so a wording change in a new Claude build would be obvious in the log.

It printed this, every time:

buttons_seen=['Minimize','Maximize','Restore','Close','Menu','Collapse sidebar',
              'Search','Back','Forward','Home','Code','New','Artifacts','Routines',
              'Pinned','Idle Draft HOA conversation talking points', ...]

Chrome and sidebar. Always. Never one content button, no matter what was on screen.

Why: it sampled the first 20 buttons in depth-first traversal order, and DFS hits window chrome and the sidebar before it ever reaches the conversation. The sample was structurally incapable of showing the thing it existed to show.

So for three weeks the diagnostic said, in effect, "here are some buttons, none of them is Allow", which reads like the button got renamed. It sent me looking for a wording change that had never happened. The real message was "there is no content in this window at all," and nothing in the output could express that.

It now sorts content buttons first and reports both counts. Zero content buttons with shell buttons present is now its own named error: ACCESSIBILITY BLACKOUT.

Layer 3: the self-test was lying politely

There was a check for this. It was called UIA can see content. It was non-critical.

So on every run, one check failed, the overall verdict came back healthy, and the log printed:

Self-check OK (8/9)

Eight out of nine. Sounds fine. Reads fine at a glance for twenty days.

The failing one was the only check that mattered. A supervisor that cannot see the conversation cannot approve anything, and that is the entire product. It is now CRITICAL, it asserts content_buttons > 0, and I proved it fails by simulating a blackout before I trusted it. The tray also no longer prints "OK" when anything is failing. It prints Self-check DEGRADED and names the failure.

If your health check has a category called "non-critical," go read what's in it. Mine had the product's core capability filed under advisory.

Layer 4: I shipped the fix, and it was still broken

This is the part I find hardest to write.

I fixed the blackout. I rebuilt. I launched the packaged binary, not just the source, the actual artifact a customer downloads, and it reported content pane IS exposed (53 content buttons visible) and Self-check OK (9/9). Verified. I published it as v1.0.24 and told the one person who cared that it was fixed.

It still couldn't approve anything.

Because "can it see" and "can it click" are different questions, and I had only answered the first one.

Claude's real button is named:

'Allow once 2 Ctrl +Enter'

That's the label, plus its shortcut number, plus its keyboard hint, concatenated into one accessible name. My matcher looked for exactly "Allow once", "Allow always", or "Allow". None matched. So it fell through to a regex, ^allow, and that matched something:

'Allow Claude to use create scheduled task (scheduled-tasks)?'

The dialog's question heading, which Claude also exposes as a button-shaped element.

HELM clicked the question. Clicking a question does nothing. And because the code trusted GetInvokePattern().Invoke() returning without raising as proof of success, it logged an approval and moved on. Then did it again 11 seconds later. And again.

855 repeats of one dialog
1,194 'Allow once'
1,155 'Allow'

Every one of those is a logged approval that never happened.

There was a third false target underneath that, which I only found because the new verified click refused to call it a success: the same regex matched my own conversation. Claude renders each tool call as a block whose accessible name is its description, and one of those blocks was literally named "Approve the Cockpit permission with verification", the description of a command I'd run an hour earlier. It was being selected as the approve button and clicked, forever.

A real dialog puts Deny beside Allow. So a regex match now has to sit on a button row with a danger control next to it. Transcript text has no Deny row, so it selects nothing.

The actual lesson

Verifying a precondition is not verifying a capability.

I verified that the window was visible. I verified the self-test passed. I verified the process was running. All true, all green, all irrelevant, because the product's job is to click a button, and I never once verified that a button got clicked.

Seeing is not clicking. Connected is not sending. rc=0 is not an outcome. A green dashboard is not a working product.

The general form, which I now apply everywhere: name the one capability the thing exists to deliver, and design the test around observing that. If your test can pass while the capability is dead, your test is decoration.

What the fix actually looks like

Approvals are verified now. HELM clicks, waits, re-queries, and confirms the dialog is gone before it reports anything. When the accessibility click silently no-ops, which Electron buttons do routinely, because they expose an InvokePattern that isn't wired to anything, it falls back to a real mouse click and re-checks again. If the dialog is still there, it says so:

UIA could NOT dismiss permission 'Allow Claude to use create scheduled task
  (scheduled-tasks)?': approve button still present after invoke (invoke_returned=True)

That line is the whole fix in one sentence. invoke_returned=True and the dialog is still open. The old code would have called that a success. It called it a success 855 times.

And no build ships now without passing a release gate that builds a realistic permission dialog, heading, Deny 1, Allow once 2 Ctrl +Enter) and asserts which control gets selected. Not whether the names match individually; both the heading and the button match ^allow, which is exactly how the broken build passed my earlier thinking. It asserts the selection. Run against the v1.0.24 logic it fails by name:

FAIL  selected the PROMPT HEADING 'Allow Claude to use create scheduled task...?'
      - this is the v1.0.24 defect; clicking it is a permanent no-op logged as success

I wrote that test by reproducing my own bug and refusing to trust the gate until I'd watched it fail.

Postscript: nobody told me

Twenty days of a completely dead product and not one bug report.

I looked at the download numbers afterward. Twenty-four releases, forty-three downloads total, all time. Most of them mine.

That's its own lesson, and honestly a harder one than the technical stuff. A product with ten real users has a bug report on July 4th. Silence isn't proof that things are fine. Sometimes it's proof that nobody's there. And the failure that finally taught me to verify capability instead of preconditions is the same failure that showed me how few people were watching.

Both of those are worth knowing. Only one of them is a bug you can fix in code.