A Two-Week AI Agent Pilot: What You Can Really Get Done

Most “AI pilots” in companies die the same way: three months of approvals, a demo on pre-prepared questions, excitement at the steering committee — and zero in production a year later. The problem is not the models. The problem is that a pilot is designed as a small project instead of an experiment with a pre-defined failure criterion. Below — how we assemble a pilot in 14 calendar days, which decisions we make on which day, and why two weeks is not a speed sprint but a tool against self-deception.

Day 0. What we refuse to do

A pilot does not answer the question “can AI solve our task”. The answer to that is almost always “yes, to some extent”, and it is useless. A pilot answers a different one: at what quality and cost-per-request metrics the agent pays for itself in our process — and whether those metrics are achievable on our data. So on day one we fix three numbers: the target accuracy on a real sample (usually 85–92% depending on the risk of an error), the acceptable share of escalations to a human, and the maximum cost of one processed request. If the client cannot name these numbers, we help derive them from the current economics of the process — but without them the pilot does not start.

Days 1–3. Data matters more than the model

Three days is a hard limit for getting data. Four things are needed: an export of real dialogs or tickets for the last 3–6 months (at least 500, better 2–5 thousand), an up-to-date knowledge base in any form (Confluence, Word regulations, site pages, an FAQ from the senior operator’s head), access to a test environment of the target system (amoCRM, Bitrix24, 1C, helpdesk), and one person on the client side who answers “how should it be” within an hour, not a week.

In parallel we do what almost nobody does: we manually label 150–300 real requests together with the client’s expert. For each — a reference answer and a category. This is the evaluation dataset everything else will be measured on. Without it, any assessment of the agent degrades into “well, it seems to answer fine”, and that is not a metric. We set half of the dataset aside and show it to nobody — neither us nor the model — until the final day.

If the data has not been received in three days — the pilot stops, and that is a normal outcome. It means the company is not organizationally ready, and no model will fix that. Better to learn it on day three than in month three.

Days 4–6. Boring architecture that works

For the pilot we almost always assemble the same scheme: RAG over the knowledge base, one or two tools with side effects (create a deal, set a task, request an order status via API), a deterministic intent router at the input and an escalation rule at the output. No multi-agent setups, no “autonomous” reasoning loops — all of that makes a request 5–10x more expensive and is almost never needed on the first line.

Model selection is driven by data requirements, not benchmarks. If data may leave the perimeter — Claude or OpenAI via API: they follow instructions best. If it may not — Qwen 2.5/3 72B or GigaChat in a closed perimeter; by our measurements on support tasks they lag by 3–6 percentage points, which is acceptable for most processes. Orchestration — Pydantic AI or OpenAI Agents SDK: both give typed tools and tracing out of the box, which is critical for error analysis at the next stage.

The most expensive bug of a pilot is not a model hallucination but a badly cut document in the index. 70% of the errors on the first iteration we find exactly in retrieval.

Days 7–10. Iterations on the open half of the dataset

We run the agent on the first half of the labeled requests, get accuracy — usually 60–75% on the first pass — and start breaking errors down by class. Typical distribution: 40% — the right fragment was not found in the knowledge base (fixed by chunking, metadata, hybrid search), 25% — found but answered in the wrong tone or format (fixed by prompting and few-shot examples), 20% — the question was out of scope but the agent tried to answer (fixed by the router and a confidence threshold), 15% — genuine model errors. After three or four iterations, accuracy on the open half rises to 85–90%.

At the same time we measure the economics: p50/p95 latency (up to 6 seconds p95 is acceptable for chat), the cost of a request in tokens and money, and the share of requests that reached a tool call. If the cost per request comes out above the target — that is a signal to change the model or the architecture now, not after launch.

Days 11–13. The held-out sample and live traffic

The final run is on the half of the dataset nobody has seen. The gap between the open and the held-out sample shows how much we “overfitted” the prompt to known questions; a normal gap is 2–4 percentage points, more than 8 means the iterations went the wrong way. In parallel, if the client is ready, we switch the agent to shadow mode on live traffic: it answers, the operator sees the answer and either sends it or edits it. The share sent without edits is the most honest metric you can get in two weeks.

Day 14. Three outcomes, and all three are a result

• We scale. Metrics met, the economics add up. The pilot code becomes the production base — we write it from day one so it will not be thrown away.
• Fixing the bottleneck. Accuracy 80% instead of 88%, and we know exactly which error class is responsible. Another week or two with a concrete plan.
• Wrapping up. The data is too dirty, the process too variable, or the economics do not add up. The client spent two weeks instead of six months and knows what to fix before returning to the topic.

What the client gets in any case: a report with metrics by error class, a labeled evaluation dataset (an asset — any next vendor or internal team will use it), the pilot source code, and an estimate of the cost and timeline of a full rollout. None of this requires continuing to work with us — and that is exactly why most clients continue.