Multivariate Testing in Minutes

Multivariate testing is a technique for testing a hypothesis in which multiple variables are modified. In this tutorial, we will explain how a multivariate test differs from an A/B Test, how to create and conduct a multivariate test, and what questions you should be asking of your test. The goal of multivariate testing is to determine which combination of variations performs the best out of all of the possible combinations.

Hi, and welcome to this quick introduction on multivariate testing.
So, what is a multivariate test? We saw that with A/B testing we were comparing two
distinct versions of an item, or variants. So, a red versus a green call-to-action
Multivariate testing is when you’re trying different possibilities. You have
just an image, or you try a sign up, or you add a video with an image, or you add a
video with the sign up. Or maybe you are testing four buttons, two that are blue
and two that are orange. One blue and one orange buttons say RSVP and
another blue and orange button say sign up. So, what are some common advantages
and disadvantages to doing a/b vs. multivariate testing? When running an a/b
test it will be simple in design and a small sample size may be okay.
The limitations to doing an a/b test are that you’re only testing one alternative.
When conducting a multivariate test, you can test many combinations at once.
However, you will need a much larger sample size to run your experiment on.
For example, you have 200 users and you run an a/b test. So, you test one version
on 100 people and the second version on another 100 people. But if you have a
multivariate test, where you’re testing out different combinations at once, you
will have a much more limited sample size to test each version on. With A/B
testing you are trying one alternative, but with multivariate testing you can
try many combinations at once. And for that you’ll need a better understanding
of interactions. Since there are so many variations the data can get a bit messy.
It can happen with drug testing. One is a control, one is a test. One is a placebo
and one is the actual drug. But what if the pharmaceutical company is in a rush
and they are testing with cholesterol medication, hair loss medication and
blood pressure medication. What if some people only get two medications or some
get all of the medications? You will need a better understanding of how these
experiences interact with each other. So, now you know the difference between an
a/b test and a multivariate test. Before deciding what kind of experiment to run
ask yourself these questions: What are the factors or levels you plan on
changing? How big is your sample size? How long will you conduct your experiment?
What is the business question you want to answer? What are your metrics and
expected outcome? And who is in your experiment? Answering these questions are
the first step in experimentation for your a/b or multivariate test. Thanks for
watching, give us a like if you found this useful or you can check out our
other tutorials at

Check out another online experimentation video:
What is A/A Testing?

Next Video:
Steps in Experimentation

More Data Science Material:
[Video] Community Talk: Online Experimentation and A/B Testing
[Video] Introduction to Online Experimentation and A/B Testing
[Blog] Ethics in A/B Testing


Blair Heckel
About The Author
- Blair holds a Bachelors degree in Marketing from Washington State University and has a background of leading data-driven marketing campaigns.


You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>