A/B Testing Models Quiz

5 questions Pass: 70% +25 pts (50 with Pro)

Quiz covering Model Analysis

What you'll be asked

  1. What is the primary purpose of conducting an A/B test when deploying a new machine learning model?
  2. In an A/B test setup, why is it critical to ensure that users are randomly assigned to either the control group or the treatment group?
  3. You are running an A/B test for a recommendation model. Which of the following scenarios best describes a 'novelty effect' that might skew your results?
  4. When evaluating the results of an A/B test, what does a p-value of 0.03 indicate, assuming a significance level (alpha) of 0.05?
  5. You observe a statistically significant improvement in click-through rate (CTR) for the treatment group, but notice that the model's latency increased by 200ms. What is the most appropriate next step?

Answers and explanations come after you submit.