A plant assistant, and what it cost to know whether it was right
A large industrial company had its operating knowledge split between dense manuals and the memory of its most experienced people. We built an assistant that answers from the company’s own systems and walks through the diagnosis step by step. This page tells how it is put together and what it cost us to show that it was getting the answers right.
What this page covers
01 · The problem
Nobody reads the manual with the machine stopped
The knowledge of a plant lives in two places and neither is at hand when you need it. Part of it is in long manuals, written to be read with time and calm. The other part is in the heads of the people who have worked there for years.
Somebody standing in front of a stopped machine can reach neither. They are not going to read forty pages, and the person who knows may be on another shift or not in that day.
What this company wanted was not a search engine. A search engine returns documents and leaves the reading to whoever has the least time. They wanted an answer, with the steps to follow, taken from their own systems.
02 · How it works
Half a dozen agents and a single point of contact
Underneath there is not one assistant but several. Half a dozen specialized agents, each with its own ground, coordinated by an orchestrator that decides which of them a query belongs to.
From outside none of that shows. Whoever asks writes once and gets one answer, without picking a recipient and without knowing there is a division of labor in there.
The division exists for a practical reason. An agent covering a narrow area answers better on that area than a general model trying to cover all of them at once.
But that architecture solves one problem and creates another. A decision appears that did not exist before, which is getting right which agent each question belongs to. If that decision fails, it makes no difference how good the agent was that the query never reached.
03 · What it cost to know
The first measurement came out at 72.8%
That decision has to be measured, and measuring it is work. We gathered 118 real queries, the kind actually asked on the plant floor. Then we checked one by one whether the question had reached the agent it belonged to.
And how many landed where they should? 72.8%. Something more than one in four queries was ending up at the wrong agent.
A number like that is uncomfortable to show and it is the only door there is to improving. Without measuring, what you have is the impression that it works, which is exactly what everybody has before they measure.
With the 118 in front of you, you can see where it breaks. We corrected the routing and ran the same queries again, and a second round of corrections left it at 91.5%.
The same 118 every round. Changing the exam between one measurement and the next turns the comparison into decoration.
Why we stopped near 92%
And how far do you keep going? From 91.5% to 100% there is a stretch, and the temptation to walk it is strong, because a round number shows far better.
We decided to stop near 92% and say so out loud. Past a certain point, what improves is no longer the system but the exam.
When you chase a perfect score, every case that fails pushes you to adjust the test until it stops failing. The number goes up, the system stays where it was and what you have built is an exam cut to fit whoever passes it.
The same discipline took away an idea we liked. We tried a cheaper model for the routing and it lost ten points overall, which is already plenty.
What mattered was in the tie cases, the ones where two agents could both fit and you have to pick well. There it fell from 89% to 44%. The saving was on the model bill and the cost was on the hard queries, which are the ones that make somebody ask in the first place.
04 · What holds the rest up
Whoever asks is in charge, not the assistant
There are two decisions you cannot see from outside, and they are the ones that let this run inside a large company.
The first is about permissions. The assistant has no access of its own to anything. When it queries a system it does so with the identity of the person asking, so each one sees what their role allows them to see and not one field more.
The second is about exactness. When the answer has to include a reference text, the model does not write it. It returns a key and the code goes and fetches that text as it is stored, in its own words and with no variation.
Both come from the same idea, the one that orders everything else. Anything that can have consequences is not left to a model’s judgment, it is settled in code.
Does your operating knowledge live in manuals and in three people’s memory?
Tell us your challenge and we reply within one business day. If we don’t see a return, we’ll tell you.