How to test subsurface agents without overfitting to a few fields

IFI Gatehouse, 0 stars: a fence contractor's estimating tool where AI reads the bid package and a named person approves every change

The AI proposes findings from drawings, specs and addenda, each tied to the sheet, page and quoted words, and nothing enters the scope until a person accepts it. The motto is AI interprets, code calculates, humans authorize.

For estimators and preconstruction managers: a concrete way to use AI on bid documents without letting it fill gaps with confident numbers.

Why does AI code like a human? An analytical chemist asks for AI that runs offline and gives the same answer every time

By his own measurements, more than half of what Claude did in a session was drift away from what he asked for. His point for plants: refineries and labs will not hand their data to a model in someone else's cloud, so AI they trust has to run on a laptop, air-gapped, and give the same answer every time.

Image: Medrado Michael Leal, Pathfinding

Industrial Agent Evaluation Lab: replayable tests for maintenance agents, with fault injection and scoring

The agent has to work out which pump readings and maintenance revision to trust, find a qualified technician, reserve the right part, ask for supervisor review and draft a work order. A deterministic evaluator grades its tool calls and the final database state, with injected timeouts, rate limits, duplicate deliveries and a lost work-order response.

For reliability teams and vendors: prove a maintenance agent recovers from bad data and dropped responses, not just the happy path.