Research line

Field Notes on Agents That Act

I study how AI agents behave when they can change real, shared state: when they should act, ask, or stop, how far a production prompt can move them, and how to grade them honestly. The experiments run on an agent I ship, against scenarios shaped like its traffic, with the scoring code and aggregate results published beside every conclusion.

Three rules hold across the series. No real user data touches a published number. Every study is a dated snapshot with its limitations attached, and corrections stay visible beside the claim they changed. The practice loop is Build → Instrument → Evaluate → Publish → Revise.

Published studies

June 24, 2026 snapshot

We Gave Six LLMs a Family to Run. Most Hesitated. One Crossed the Safety Line.

A production-shaped restraint study on synthetic household scenarios; no real user data was used.

Six OpenRouter model identifiers each routed 227 scenarios three times at temperature 0.2 through a forced do/ask/confirm/chat choice. Routing was scored deterministically against an answer key built from a three-model panel.

Limitations (4)
  • The scenario set is small and hand-written by the product founder.
  • The forced-choice format likely inflates asking relative to open conversation.
  • Two panel judges were also models under test, and Gemini’s apparent rank depended materially on its own-family judge.
  • Only two tested identifiers were date-pinned snapshots; provider routing and model behavior can change.
Read the study →Methods repository →

April 15, 2026 snapshot

We Asked 8 LLMs to Run Our Family’s Life. Two Tried to Book a Vacation.

An April 2026 production case study using Honeydew’s prompt and a small synthetic scenario set.

Eight models were tested on 35 scenarios across six categories, ten trials per model-scenario pair.

Limitations (4)
  • The prompt was built around GPT-family models, creating prompt-coupling the study cannot separate from capability.
  • The June 24 restraint study supersedes the original noisy-input and duplicate-handling conclusions.
  • This is a production case study, not peer-reviewed research or a universal model ranking.
  • The repository documents a dated production case study; it does not turn the result into a universal model ranking.
Read the study →Methods repository →

April 14, 2026 snapshot

Do LLMs Actually Cite Your Startup? A 90-Day Field Note

A 90-day descriptive field note on one early-stage consumer product’s observable LLM referrals and one ten-query Perplexity citation snapshot.

GA4 source-attributed sessions and a separate browser-referrer event were kept as different units. Ten queries were run once each in fresh Perplexity sessions and checked separately in ordinary web search.

Limitations (4)
  • There was no pre-intervention baseline or control group, so the field note cannot establish causal lift.
  • GA4 sessions and custom referrer events are different units and must not be divided into a capture-rate estimate.
  • The citation check used one provider, one product, ten queries, and one run per query.
  • Typed URLs, later searches, cross-device discovery, and other indirect journeys are outside the observed data.
Read the study →Methods repository →

Publishing next

Stated in advance so the gates are visible. A piece ships when its gate clears, not before.

Swapping the model under a live agent

Writing now

What a release-floor harness and a capability tier caught, and missed, when the model under a shipped family agent changed. Exact builds, dated receipts, synthetic scenarios, no user data.

Grading the graders

Gated on a hand re-grade

Two production LLM judges audited against human labels: agreement, direction of disagreement, and where the ground truth itself is soft.

Five ways agents break, re-cut on 227 scenarios

Gated on a re-analysis

A behavior taxonomy rebuilt on the June data and its reasoning transcripts, replacing an earlier cut that rested on corrected claims.

The mental load, measured

Gated on data volume

A ranked map of what families hand an agent, from aggregate intent counts only. Publishes when a 30-day pull clears 300 classified requests across eight or more intents.

Citing and reusing

Each study page carries a citation block, and each repository ships a CITATION.cff. Code is Apache-2.0; data and figures are CC BY 4.0. The scenario fixtures are synthetic and demonstrate the method; they do not reproduce the original rankings, and the repositories say so.

ChatGPeTe

New studies arrive by email first.

Subscribe