1. Home
2. Solutions
3. Tutorials

Instructions

The principle of A/B testing is that you have one or more 'tests' (such as "how many articles should be displayed on each page"), and each test has two or more 'buckets' (such as "10", "15" and "20"). In each test, the user should be randomly assigned to a bucket, taking into account whether some buckets are weighted more or less than others, but when running more than one test at the same time, also ensuring that the tests don't influence each other. The same user should, once assigned to a set of buckets, stick with them, so that they don't perceive that the website is changing for no reason.

Let's get started.

Make sure the solution executes in the right place

Fastly services support a number of features that can cause subroutines to be executed more than once, such as shielding and restarts. You will want to ensure that your code only runs once. Start with this code at the end of vcl_recv:

# Only enable...

if (
  fastly.ff.visits_this_service == 0 && # on edge nodes (not shield)
  req.restarts == 0 # on first VCL pass
) {
  # Remainder of this tutorial's RECV code goes here
}

This code checks that the request is not on a shield machine, and that it is on a first pass though the configuration (prior to any restart). You could also use this opportunity to add an on/off switch linked to a key in a configuration table.

Now that you have sufficient guardrails in place, you can add the logic inside of the if block that you just created.

Give the user an ID

Consistently mapping a user to the same set of A/B test buckets depends on the user having the same identity from one request to the next. If you are already doing authentication in your Fastly service, you can use the ID that you get from that, but if you want to include anonymous users in your A/B testing, you'll likely still need to create a tracking ID for them.

First check in the vcl_recv subroutine for an existing usable ID from a cookie, and if it doesn't exist, generate one. Also generate a cookie to store the new ID in:

# Allocate the user a unique identifier if they don't already have one

if (req.http.Cookie:ab) {
  set req.http.Fastly-ABTest-UserID = req.http.Cookie:ab;
} else {
  set req.http.Fastly-ABTest-UserID = uuid.version4();
  set req.http.abtest_new_cookie = "ab=" + req.http.Fastly-ABTest-UserID + "; max-age=31536000; path=/; secure; httponly";
}

Before the response goes to the browser, set the cookie if it wasn't received in the incoming request:

# If the user's AB testing ID is not already in a cookie, send them one

if (req.http.abtest_new_cookie) {
  add resp.http.Set-Cookie = req.http.abtest_new_cookie;
  set resp.http.Cache-Control = "no-store";
  unset req.http.abtest_new_cookie;
}

Notice that you're using a temporary HTTP header to store the cookie that you want to set. This is a useful pattern - the wrapper if statement in vcl_recv is making sure the solution is restart and shield safe, so, rather than repeating that logic here, you can treat the abtest_new_cookie header as a queue for cookies waiting to be set.

Define some tests

Fastly needs to know some things about the tests you want to run:

• The number of tests and the name of each one
• The number of possible buckets for each test and the relative weighting of each
• The name of each bucket

You can lay out this information efficiently in a VCL table, which can then be managed independently of your configuration using an edge dictionary. If you are testing it in a fiddle, write the dictionary definition manually:

table solution_abtest {
  "tests": "itemcount, buttonsize",
  "test-itemcount": "AB",
  "bucket-itemcount-A": "10",
  "bucket-itemcount-B": "15",
  "test-buttonsize": "AAAAAAABBBCC",
  "bucket-buttonsize-A": "small",
  "bucket-buttonsize-B": "medium",
  "bucket-buttonsize-C": "large"
}

VCL is designed to allow us to compile your configuration into extremely fast code with a predictable execution cost. That comes with a few limitations, such as a lack of enumeration or loops. You can get creative to work around this, while retaining the high performance. Here the solution is to define the buckets and relative weights by writing a string of single characters, where each character represents one of the AB test options, and then repeating characters as needed to create the right relative weights.

So, to run a test called logovariant with three buckets, where you want 25% of users in the first, 25% in the second, and 50% in the third, the test is defined as "test-logovariant": "ABCC".

Once you have defined the test, you need to map the buckets (A, B, C) to the values you want to assign to them. For this, you can use a constant namespace (i.e., bucket) to prefix your dictionary key, to separate these bucket entries from the test definitions. Add the name of the test, and finally the bucket key, e.g., "bucket-logovariant-A": "badger". You should create one entry for each unique character in your test string, so if your test string was "ABCC", you need A, B and C buckets.

We'll show later how you can set the limit for the number of tests that you can run, but for now, feel free to configure 2-3 example tests.

Retrieve the list of tests and run them

Your test definitions are outside of any subroutine, so they are parsed at compile time and loaded just once at startup. But you need to read and apply the tests when each request is received. In the vcl_recv subroutine, after the user ID code that you wrote earlier, insert:

set req.http.Fastly-ABTest-Queue = table.lookup(solution_abtest, "tests");

call solution_abtest_allocate;
call solution_abtest_allocate;
call solution_abtest_allocate;
call solution_abtest_allocate;
call solution_abtest_allocate;
call solution_abtest_allocate;
call solution_abtest_allocate;
call solution_abtest_allocate;
call solution_abtest_allocate;
call solution_abtest_allocate;

unset req.http.Fastly-ABTest-Queue;

Here you are reading the current list of tests to run, placing that value into an HTTP header, and calling a custom subroutine 10 times. Since you (potentially) have more than one test, it makes sense to put the code that processes a test into a subroutine that you can call, rather than having to repeat the same larger block of code. Remember, VCL does not support loop constructs. VCL local variables are scoped to a single subroutine, so in order for the custom subroutine solution_abtest_allocate to have access to the queue of tests to be processed, the temporary request header Fastly-ABTest-Queue is used.

Also note that since the subroutine calls must be resolved at compile time, it's not possible to call solution_abtest_allocate the same number of times as the number of tests you have defined. You must decide what is the likely maximum number of tests you will want to run (and you can always change this later by adjusting the number of times you call the subroutine).

Allocate buckets for each test

Now, define the solution_abtest_allocate custom subroutine, which goes in the INIT space:

sub solution_abtest_allocate {
  declare local var.thisTest STRING;
  declare local var.testSplits STRING;
  declare local var.userTestSlot INTEGER;
  declare local var.bucketKey STRING;
  declare local var.bucketValue STRING;
  declare local var.result STRING;
  declare local var.lookupKey STRING;

  if (req.http.Fastly-ABTest-Queue ~ "^\s*(\w+)\s*(?:,(.\*))?\$") {
    set var.thisTest = re.group.1;
    set req.http.Fastly-ABTest-Queue = re.group.2;

    # <-- The rest of the code included in the current tutorial step goes here.

  }
}

The process of extracting the bucket options and finding the correct bucket will require a few local variables. Define those first - we'll explain what they are for in a moment. The key part here is that the header containing the list of tests to process (which you populated in the last step) can be matched against a regular expression to extract the first item in the list. That first item is then assigned to var.thisTest and the header is mutated to remove that first item. This effectively does the same as an array 'shift' method.

Now you have the name of a test in var.thisTest, you can get the bucket allocation string for that test:

set var.lookupKey = "test-" var.thisTest;
set var.testSplits = table.lookup(solution_abtest, var.lookupKey);

var.testSplits now contains something like 'ABCC', and you need to pick one of those letters to be the bucket you are assigning to the current request:

set var.userTestSlot = randomint_seeded(
  1,
  std.strlen(var.testSplits),
  std.strtol(
    substr(digest.hash_md5(req.http.Fastly-ABTest-UserID var.thisTest), 0, 8),
    16
  )
);

set var.userTestSlot -= 1;
set var.bucketKey = substr(var.testSplits, var.userTestSlot, 1);

Taking this from the middle outwards:

1. digest.hash_md5 creates an MD5 hash of the user's ID combined with the name of the test. This results in a string such as "79a54025255fb1a26e4bc422aef54eb4", and for the same input, will always produce the same output (i.e., it's deterministic).
2. Ultimately you want a number between 1 and the length of var.testSplits, which can be mapped to a single character. The hash is therefore used as a seed for a random number generator, but converting this long hex string into an integer will produce a number that exceeds Fastly's maximum integer size, so truncate the hash to just the first 8 characters (e.g., "79a54025").
3. This can now be converted to an integer using std.strtol, which would produce 2040872997 in this example.
4. The integer is used as a seed for a random number generator configured to choose a number between 1 and the length of var.testSplits.
5. Strings in VCL are zero-indexed, so deduct one from the result to place it in a range that starts at 0.
6. Use this index to extract a single character from var.testSplits and assign it to var.bucketKey. That value is, in this example, either 'A', 'B', or 'C'.

Now use the bucket key to find the corresponding bucket value:

set var.lookupKey = "bucket-" + var.thisTest + "-" + var.bucketKey;
set var.bucketValue = table.lookup(solution_abtest, var.lookupKey);
set var.result = "test-name=" + var.thisTest + ", bucket=" + var.bucketValue;

At the end of this, var.result contains the allocation for this test, for this user, e.g., test-name=buttonsize, bucket=small.

Send test allocation to origin in an HTTP header

For your origin server to be able to act on the test allocations, you need to send the allocation data along with the request, to the origin server. It's important to include only one test's allocation in each header, so that Fastly can store your content with the minimum number of permutations. Later, you'll see how to indicate which tests were actually used in the construction of a page, using the Vary header. For now, you need to find a 'spare' header, and use it to assign the current test's data.

if (!req.http.Fastly-ABTest) {
  set req.http.Fastly-ABTest = var.result;
} elseif (!req.http.Fastly-ABTest2) {
  set req.http.Fastly-ABTest2 = var.result;
} elseif (!req.http.Fastly-ABTest3) {
  set req.http.Fastly-ABTest3 = var.result;
} elseif (!req.http.Fastly-ABTest4) {
  set req.http.Fastly-ABTest4 = var.result;
} elseif (!req.http.Fastly-ABTest5) {
  set req.http.Fastly-ABTest5 = var.result;
} elseif (!req.http.Fastly-ABTest6) {
  set req.http.Fastly-ABTest6 = var.result;
} elseif (!req.http.Fastly-ABTest7) {
  set req.http.Fastly-ABTest7 = var.result;
} elseif (!req.http.Fastly-ABTest8) {
  set req.http.Fastly-ABTest8 = var.result;
} elseif (!req.http.Fastly-ABTest9) {
  set req.http.Fastly-ABTest9 = var.result;
} elseif (!req.http.Fastly-ABTest10) {
  set req.http.Fastly-ABTest10 = var.result;
}

This might seem clunky, but remember that VCL does not support runtime variable declarations: we need to know what headers you might set at compile time.

Remove the user ID from origin requests

The ID that you are using to identify the user for the purposes of A/B testing shouldn't be sent to your origin servers, because if the origin server were to generate different content based on the ID, rather than the bucket allocation, then Fastly can't cache the resulting output efficiently. To remove that possibility, add the following code to both the vcl_miss and vcl_pass subroutines:

unset bereq.http.Cookie:ab;
unset bereq.http.Fastly-ABTest-UserID;
unset bereq.http.abtest_new_cookie;

Use the allocations on your origin server

The information about the bucket allocations is presented to your origin server as HTTP headers, prefixed Fastly-ABTest. In the examples above, you created tests called itemcount and buttonsize, so you will see two headers, Fastly-ABTest, and Fastly-ABTest2, one with test-name=itemcount and one with test-name=buttonsize. You can use this information to adjust the response you generate.

It's vital that you tell Fastly which tests you used to determine the content of the page, so we can cache multiple variants of the response where appropriate. Sometimes, the user might request some resource that is not affected by any A/B tests - perhaps an image - and in this case you don't need to do anything. We will cache just one copy of the resource, and use it to satisfy all requests. However, if you do use any Fastly-ABTest headers to decide what to output, then you need to tell us that you did this, using a Vary header:

Vary: Fastly-ABTest2

In this case, you're saying that in producing the response, you checked for the test defined in the Fastly-ABTest2 header. If that is the buttonsize test, Fastly needs to keep 3 copies of this resource, one for each of the possible button size buckets (small, medium and large). However, you're also implicitly saying that the page does not have a list of items on it and therefore the itemcount test is irrelevant.

To indicate that you used multiple tests in generating the page, just comma-separate the header names in your Vary header:

Vary: Fastly-ABTest, Fastly-ABTest2

Now, Fastly needs to store six variations of the resource: two itemcount possibilities, for each of three buttonsize possibilities (2 x 3 = 6). Clearly, the more permutations, the more variants must be stored.

Next steps

Where you include a Vary header citing an AB test on the response, you could consider adding a surrogate key as well. This will allow pages that have been subject to particular tests to be purged without affecting other resources (e.g., if you decided to make the 'large' buttonsize even larger. However, since the bucket allocation and bucket value are part of the request and can be considered by the cache, you don't need to purge in order to update either your weightings or bucket values (e.g., if you decided to introduce a new itemcount of 25).

Related content

VCL reference:

• std.strtol
• digest.hash_md5
• randomint_seeded

Blog posts:

• A/B testing at the edge (covers other techniques)
• GOV.UK guide to how A/B testing works (customer blog)

Quick install

The embedded fiddle below shows the complete solution. Feel free to run it, and click the INSTALL tab to customize and upload it to a Fastly service in your account:

Once you have the code in your service, you can further customize it if you need to.

All code on this page is provided under both the BSD and MIT open source licenses.