The boring half of an AI image pipeline

QuickStaging takes a photo of an empty room and returns the same room, furnished. The call to the image model is one line. This post is about the other lines: what happens before the model sees a photo, what happens while it works, and what happens when it gets something wrong.

Before the model

Real estate photos come from phones, cameras and other tools, and all of them send in odd files. HEIC from iPhones. AVIF from editing software. PDFs of floor plans. PNGs with a transparent background. Photos that look upright but are stored sideways with an EXIF flag that says "rotate me".

The browser converts what it can before uploading. The server rotates every image upright, re-encodes it and checks its size. None of that is interesting until it's missing. An AVIF upload once passed every check, failed quietly further down, and left a listing with zero photos. Now unusual formats are converted on the client before anything else happens.

My favourite bug from this layer came from an image library. One quality check runs edge detection on the result. On images with an alpha channel, the edge map came back all zeros, because the channel was only removed at the end of the pipeline, after the convolution. The check passed every image that had transparency. The fix was one call in the right place, and I confirmed it changed nothing on 238 images without transparency.

While the model works

Generation takes anywhere from seconds to several minutes, which is far longer than an HTTP request should stay open. So every job is asynchronous. The API claims credits, starts the job, returns at once, and the provider calls a webhook when it's done. The page polls the database, not the provider.

Async jobs fail in more ways than sync calls, so errors are sorted by what to do about them:

• Permanent (bad input, a rejected request): stop and refund.
• Throttled or transient: retry within the request, with backoff and jitter, but never past the route's time limit.
• Out of capacity: the provider is busy. The job goes into a queue that retries over almost two hours, and the credit stays held until it lands or is refunded.

When there's a second provider, the fallback signs its webhook exactly the way the first one does. The completion code can't tell which provider did the work, and it doesn't need to.

When the webhook never comes

Webhooks get lost. A deploy restarts the server, a request times out, the network drops one. Early on, a lost webhook meant a paid render nobody ever saw.

Two things fix that. A scheduled job looks for anything stuck in "processing" for 15 minutes. Before refunding anything, it asks the provider what actually happened and settles the job through the same code the webhook would have run. Only jobs the provider has no record of are written off. The job also claims each record with a guarded update before touching credits, so two runs can't refund the same image twice.

The clock starts when processing starts, not when the record was created. Using the creation date once reaped regenerations of old images that were working fine.

Checking the output

Image models fail confidently. The result looks like a photo, but a window has moved or a doorway is gone. So every result goes through two checks: cheap pixel measurements first, then a vision model comparing before and after.

The rules that matter:

• A pixel check on its own can only mark a result "suspect". It takes the vision model's confirmation to hold a result back.
• A rejected result is regenerated a limited number of times for free. If it still fails, the user gets a refund.
• The checker fails open, because a broken checker shouldn't block every user. Every verdict also records why it passed. I added that after a usage-limit error was silently logged as a pass for a stretch of time.

How the credits behave around all this is its own topic, covered in a separate post.

What I deleted

The pipeline got better mostly by getting smaller. A large prompt builder lost a blind comparison to something much shorter. A "structure lock" step that was supposed to protect the room did worse than a plain retry on the same set of rooms. A photo "look" pass still sits in the code, but nothing calls it.

Each of those felt like progress when I wrote it. Measuring is what removed them. These days a new step has to beat a test set before it ships.