FEED · LIVE v3.2.1 QC · CA FR
field-notes/feed · 23 transmissions · cadence every two weeks
--:--:-- UTC
QUEBEC · 46.81°N -71.21°W
root · field-notes · /feed

Field notes /
from production.

Transmissions from inside live agent deployments: postmortems, evals, cost graphs, what worked, what blew up. No "10 ways AI will change everything."

posts23
contributors8
last published2026·08·15
avg. read time~10m
edited byJ.P. Levac
cadenceevery two weeks
filter ·
Featured · transmission 018

Why your RAG pipeline silently degrades, and how we caught it in eval week 3.

A worked example: a 7-agent customer-support stack for a mid-size fintech. Three weeks in, satisfaction scores can dip 4 points overnight with no model change, no prompt change, no traffic anomaly. The drift is in the index, and it changes how these systems should be measured.

JP Jean Pierre Levac
2026·05·14 11m read ● RAG · evals
Read transmission
§ feed · all transmissions

Recent / chronological

22 posts · sorted newest → oldest
tx · 031 ops

The AI tell is structural: strip the style and the writing is still recognizable.

UMD and Google DeepMind's StoryScope separates human from AI fiction using narrative structure alone: 93.2% macro-F1 with every stylistic cue removed, on their corpus. Each model's structural fingerprint, why AI outputs converge while human work spreads out, the two overclaims to refuse, and what an operator can honestly take from a fiction-only result.

tx · 030 infra

The open tier now costs a third as much. Rerun your model math.

Moonshot's Kimi K3 lands near the top of the independent indices at roughly a third of the closed frontier's cost. The two durable patterns inside its July 2026 technical report (reasoning effort is a trained budget cap, and harness overfitting is now something labs train away), how to read vendor-run numbers, and the Law 25 question that decides it for a Quebec SMB.

tx · 029 agents

Your AI gets measurably worse when you change your mind.

Every model tested, frontier and open, gets measurably worse when your goal shifts mid-conversation than when you state it once. A July 2026 Microsoft Research study across nine models: adding detail is nearly free, changing the task is the expensive move, and reminding the model what you now want only partly fixes it. What that changes about how you run a session with a coding agent or assistant.

tx · 028 agents

The setup prompt is written for your AI, not for you.

A new kind of install instruction is spreading: setup prompts written for your coding agent to run itself, without asking you. Some edit the agent's global config and are engineered to suppress its confirm-before-acting reflex. The two questions that separate a trustworthy agent-setup prompt (Cloudflare's) from an exposed one, and the verify-before-you-run habit for teams adopting agent tooling.

tx · 027 agents

Graph engineering is the new name for work we already run.

At least six YouTube channels shipped "graph engineering" videos inside two weeks of July 2026: fan your AI agents out into a graph instead of one long loop. What the label actually names in production terms (the diamond, the barrier), the judge-node rule the tutorials underplay, and the honest cost frame: unit cost down, total cost up, by design.

tx · 026 ops

Your AI judge is grading its own homework.

LLM judges measurably favor machine-written text: 1.42x to 1.91x over-acceptance in a June 2026 Cambridge-led study. Where the bias lives (open-ended judging, not reference-anchored grading), the adversarial-pool correction that fixed it, and four calibration moves for any judge you run in production.

tx · 025 ops

Your eval harness is a frozen RL environment.

Tasks, verifiers, rewards, termination conditions: the components are identical whether you gate a deploy or train a policy. The only difference is temperature. Why the RL environment literature's reward-design lessons apply to your evals at zero training cost, and the four-rung ladder from frozen CI gate to molten training environment.

tx · 024 agents

More agents mostly didn't survive production.

The controlled studies landed: 260 configurations, seven papers, matched compute budgets. Multi-agent coordination helps only when the task genuinely decomposes. Independent agent fleets amplify errors 17.2x, orchestration costs 285% more tokens, and debate is closer to ensembling than deliberation. The decision rule that survived.

tx · 023 RAG

Your agent has amnesia. Memory is now its own engineering discipline.

Agent memory grew from 'stuff history into the context window and hope' into its own engineering discipline: a write-manage-read loop, three memory types, reproducible benchmarks (LoCoMo, LongMemEval, BEAM), and multi-scope retrieval. Why memory is the retrieval problem you forgot to instrument.

tx · 022 infra

Your agent needs a wallet. Seven protocols want to be it.

Agent payments went from one duct-taped pattern to seven competing protocols in four months: x402, Stripe MPP, Visa TAP, Mastercard Agent Pay, Google AP2, and more. The fee math that decides it (30,003% vs 10%), how x402 turns HTTP 402 into a working rail, the regulatory gap, and what we'd pick.

tx · 021 infra

The scaling wall is real. The fix might be in your pocket.

Paper notes on arXiv:2503.08223 (Zhejiang University, April 2026). The two limits everyone worries about, data exhaustion and compute monopolization, could both be broken by the devices already in people's hands. The math, the open problems, and what is shippable today.

tx · 020 agents

Post-mortem: the loop that cost $3,200 overnight.

A missing termination condition, no cost alert, a confidence budget written but not shipped. 24,847 API calls, 9 hours, $3,218. A worked example of why every safeguard is non-optional.

tx · 019 ops

Your CS agent has a 4.2-star rating. It's also hallucinating 8% of the time.

CSAT says the customer felt helped, not that they were helped correctly. The three-tier measurement stack that catches CS agent hallucination before your return rate does.

tx · 017 agents

Building a planner that knows when to give up.

Most agent failures aren't wrong answers. They're infinite loops. We added a confidence-budget primitive to our planner and watched p99 latency drop 38%.

tx · 016 scenario

Worked example: a 12-person ops team, a 24/7 inbox, one agent stack.

How a 12-person Quebec logistics SMB could cut after-hours response time from 6 hours to 4 minutes by routing 73% of inbound through a grounded copilot. A worked example with the full architecture.

tx · 015 infra

Vector DB shootout: we ran 4 stores on 1.2M chunks. Here's the receipts.

pgvector, Pinecone, Qdrant, Weaviate. Same dataset, same embedding model, same queries. p50/p99 latency, recall@10, $/1M reads, ops complexity. No vendor took every prize.

tx · 014 RAG

Hybrid retrieval: when BM25 beats your $400 embedding model.

Dense vectors get the marketing. On technical content with rare named entities, sparse retrieval still wins. We measured the crossover.

tx · 013 ops

The 6-line eval suite we ship with every agent.

Evals don't have to be a research project. Our standard regression harness fits in a notebook and catches 80% of bad model swaps.

tx · 012 agents

Why we stopped writing custom orchestrators (mostly).

Three years, four custom runtimes, one painful lesson: LangGraph is good enough for 80% of multi-agent workflows. Here's when we still roll our own.

tx · 011 agents

Tool calling vs. function calling vs. agents: the actual differences.

The terminology is a mess. A working glossary, with code for each, plus when each one is the right call. Bookmark this for the next sales meeting.

tx · 010 RAG

Chunking is a hyperparameter. Tune it.

We tested 8 chunking strategies on 5 representative corpora. Semantic chunking wins on ambiguous text, fixed-size wins on docs, and you should never use 512 by default.

tx · 009 infra

$0.0004 per agent step: how we made cost a first-class metric.

Latency dashboards are everywhere. Cost dashboards are rare. We built a per-step cost trace that surfaces the 12% of calls eating 60% of your bill.

Ship to your
inbox / every two weeks.

Field notes, postmortems, and the occasional sharp opinion on what's actually working in production agentic AI. No "ultimate guides." No threads.

unsubscribe in 1 click no spam, ever

© 2026 Acceleratech · field-notes · v3.2.1 ← back to root A Digital Growth Strategy by JPL Digital Growth Group.