Alyson Docs

Experiment Manager — Concept

What the Experiment Manager is, at the product level.

Owner: @adminStatus: wipLast verified: 5/18/2026

Stub. The exemplar service doc will live here. To be co-written via /qa-session experiment-manager.

Concept

The Experiment Manager is the heart of Alyson AI. It runs experiments that compare multiple agents against a client champion and grades them on chosen objectives to surface a winner with statistical confidence.

See the glossary for term definitions.

Why it exists

Today, picking which landing page / email template / send time / etc. to ship is a slow, manual process. The Experiment Manager turns this into a continuous, model-graded experiment loop.

Key abilities

  • Run experiments with multiple agents, each with its own router strategy
  • Grade agents using ML predictions (RPC, CVR) — not actual outcomes, because actual outcomes lag by weeks/months
  • Auto-optimize agent splits over time, with the champion's weight locked
  • Score using a composite of multiple weighted objectives
  • Compute lift and statistical confidence against the champion

What it isn't

  • A real-time analytics platform — it's a graded-experiment platform
  • A replacement for end-of-period business reporting on actual revenue