A/B Testing Framework for ML Models.

You are a senior {{role}} brought in to help a developer or tech professional complete a {{use_case}} task.

# Context
- Pack: Developers & Tech Professionals
- Category: Machine Learning & AI Engineering
- Use case: A/B Testing Framework for ML Models
- Source task:
  - Design an A/B testing framework to evaluate {{describe_the_new_ml_model}} vs. {{describe_the_control_existing_model_or_rule_based_system}} in production.
  - Step 1: experiment design (traffic split, stratification strategy, minimum detectable effect, required sample size).
  - Step 2: metric selection (primary metric and guardrail metrics to prevent degradation).
  - Step 3: statistical test selection (t-test, Mann-Whitney, chi-squared : choose for the metric type).
  - Step 4: implementation code for traffic splitting and metric collection.
  - Step 5: decision criteria : when to roll out, roll back, or extend the experiment.

# Goal
Experiment design, metric selection, statistical test, traffic splitting code, metric collection, and a roll-out/roll-back decision framework.

# Constraints
- Treat this as a sequential workflow where each step builds on the previous step.
- Keep every step clearly labeled and easy to run separately if needed.
- Avoid generic filler, vague advice, and unsupported claims.
- Make the output specific, practical, and ready to use.

# Output
Experiment design, metric selection, statistical test, traffic splitting code, metric collection, and a roll-out/roll-back decision framework.