Engineering · 31 July 2026

We Pay Our Own Invoices to Find What's Broken

An endpoint that answers 200 is not an endpoint that works. We had monitoring that proved every route was awake, and it told us nothing about whether a paying customer received anything. So we started buying from ourselves. The first run found two surfaces that had never once delivered.

The gap in ordinary monitoring

Our uptime checks poked each endpoint every fifteen minutes and confirmed it replied. That catches a crash. It cannot catch an endpoint that takes payment, runs, and hands back nothing useful — because from the outside that looks identical to a healthy request.

The refund system made it quieter still. When a job fails, a background reconciler notices the charge was paid but never delivered and sends the sats back automatically. Nobody was out of pocket. No alert fired. The endpoints were failing politely.

What the audit does

Once a week a script pays a real Lightning invoice to ten of our paid endpoints and then inspects what came back. Not the status code — the content:

• a translation must differ from the text sent in
• a PDF must begin with the bytes that make it a PDF
• audio and images must be large enough to be real output, not an empty file
• a rare-language translation must come back in that language's script

It costs us a few hundred sats a week, most of which circulates back to our own node. If a quote ever comes back far higher than expected, the run refuses to pay it rather than discovering a pricing bug the expensive way.

What it found

Object removal had never worked

Object removal detects what you describe, builds a mask from it, then erases that region. The detector returns box coordinates in pixels. Our code treated them as fractions of the image, multiplied them by the width and height, and produced boxes far outside the picture. Every box was discarded, so the mask was always empty.

With nothing to erase, the endpoint reported that it could not find the object you asked for, and refunded. That is exactly what a genuine no-match looks like. It was wrong every single time and still never looked like a bug. It is fixed, and the audit now confirms the returned image visually.

Our own audiobook example paid and refunded

The example request in our documentation named a narrator voice that only exists on one of our speech engines — but the request did not name that engine, so it ran on a different one, which had never heard of the voice. Anyone who copied our documented example paid, waited, and got a refund. On all three places that endpoint is offered. Fixed.

Inbound fax had been dead for months

Receiving a fax needs a routing assignment at the carrier, and ours had quietly emptied — probably when we split voice and fax onto separate numbers. Sending was unaffected, so the service looked half-alive. Nothing we ran could see it, because the fault was in configuration held at the carrier rather than anywhere in our own systems. There is now a check that asks the carrier directly, every fifteen minutes.

Other changes worth knowing

Why publish this

Because "we found two of our own products completely broken" is more useful to you than another uptime badge. There is no account to close and no subscription to cancel here — you pay per request, so the only thing that keeps you is whether the thing works. That makes it worth telling you when it did not.

Higher-cost surfaces are still audited by hand rather than weekly, because the fixtures they need are harder to generate. That is the next thing to close.