Do Not Start an Incrementality Test During Cyber Week
· Tutorials/Guides

Every Q4, a founder asks us to prove what Meta and Google are worth before the biggest spend week of the year. The timing feels logical because the most budget is at stake that week, but we tell them to wait until January.
An incrementality test measures the revenue your ads cause by comparing a group that sees them against a holdout group that does not. The whole method rests on one assumption, that both groups would behave the same without the ads. Cyber Week breaks that assumption in every direction at once.
Data Druid, Taikun's account intelligence platform, tracks spend across our client accounts. Among the 12 accounts with enough history, the median account spent 3.78x a normal August day on its peak day of the 2025 season. The top quartile spent 9.93x or more. Promotions, email, SMS, organic search and word of mouth surge in the same window, so any lift you measure belongs to the calendar as much as to your ads.
Below is why the results mislead, what a holdout costs you on your highest-revenue days, and what we do instead to read Cyber Week performance.
Cyber Week rewrites your baseline
A holdout test reads lift against a baseline. During Cyber Week that baseline moves faster than your test can measure it.
The distortion also runs longer than the 5 days from Thanksgiving through Cyber Monday. Across the same 12 accounts, the worst 30 consecutive days of 2025 season spend ran at a median of 1.79x a normal August day. A standard 2 to 4 week test sits entirely inside that window.
Three things change at once, none of them caused by your ads:
Conversion rate. Sitewide discounts pull purchases forward, so customers who would have bought in January buy now regardless of exposure.
Customer mix. Deal hunters, lapsed customers and gift buyers flood the site, so the people in your test do not match the people you will market to in Q1.
Auction cost. CPMs climb as every advertiser bids up the same inventory, which changes how much reach each test dollar buys from one day to the next.
A test result is only useful if it predicts what happens next quarter. A lift number built on Cyber Week buyers describes a week that will not repeat for another 12 months.
Holdouts lose their signal inside a demand spike
Your holdout group still shops during Cyber Week. Those customers get your promo emails, see your SMS blasts, search your brand name and hear about the sale from friends.
That pushes holdout conversion rates up for reasons unrelated to paid media. The relative gap between test and control shrinks, so the test can understate what your ads drive in a normal week.
The noise problem compounds it. Daily revenue swings harder during Cyber Week than any other stretch of the year. Bigger swings need a bigger sample to detect the same lift, so a test run at the length you would use in October leaves the result buried inside the confidence interval.
Geo holdouts carry extra risk in this window:
Shipping cutoff dates hit regions differently in the final days before the holidays.
Retail foot traffic and weather shift demand by market in ways your matched regions did not show in September.
Regional promotions from retailers that stock your product change local demand without any change to your ads.
Any one of these can produce a false read. During Cyber Week you get all of them in the same test.
Your own Cyber Week playbook contaminates the test
A clean test holds the treatment steady. Cyber Week is the one window where every operator touches the account daily.
On a typical BFCM account we raise budgets several times in a week, launch promo-specific campaigns, swap in offer creative and loosen tROAS or tCPA targets to capture volume. Each change alters what the test group receives halfway through the test. Large budget edits on Meta can also push ad sets back into the learning phase, which changes delivery for reasons unrelated to incrementality.
Cross-channel spillover makes it worse. Your Meta holdout regions still see the YouTube, Performance Max and branded search pushes you scaled for the same week. The control group stops being a control.
If a test cannot survive the changes you plan to make, you have two bad options:
Freeze the account to protect the test and leave Cyber Week revenue on the table.
Run your playbook and accept a result you cannot trust.
We pick neither and schedule the test for a window where the account can sit still.
A holdout on your biggest days is the most expensive test you can run
Every holdout withholds ads from part of your market. That cost scales with what those days are worth, so Cyber Week puts the price at its yearly high.
Here is a simple example. Take a brand that normally spends $20k a day on Meta and plans a 20% geo holdout. At the fleet median peak-day multiple of 3.78x, that day carries $75.6k of planned spend. The holdout pulls $15.1k of it out of the market on the day buyers are most ready to purchase.
The revenue you give up is the incremental revenue that spend would have produced. Run the same holdout in a quiet February week and the same test costs a fraction as much.
New customer acquisition adds to the bill. Brands with strong repeat rates use Cyber Week to acquire customers who pay back over the next 6 to 12 months. A holdout cuts that cohort in the held-out regions, so the cost lands on your LTV/CAC long after the test ends.
You pay the highest price of the year for the least reliable read of the year. We would not make that trade with our own money.
Run the test in a quiet window instead
The right window is one where demand is stable, budgets can stay flat and your promo calendar is empty. For most ecommerce brands that means late January through early March.
Black Friday falls on November 27 this year, with Cyber Monday on November 30. A 3 to 4 week test started today would end inside early Black Friday promotions, so the pre-season window for 2026 has closed. Put the test on the calendar for Q1 now.
Here is how we set one up:
Pick the window. Block 4 to 6 weeks with no sitewide sales, no launches and no major budget shifts. Skip gifting peaks like Valentine's Day if your category depends on them.
Freeze the treatment. Agree in writing which campaigns, budgets and targets stay fixed for the full test. Every edit you skip protects the read.
Size the test on normal weeks. Use the variance from a stable pre-period to set the holdout size and test length. Sizing on Q4 data overstates the sample you need.
Match regions on recent behavior. Pair geo cells on the 8 weeks before the test, after holiday returns settle.
Read lift against contribution margin. Translate incremental revenue into incremental contribution margin so the result answers whether the spend made money.
A Q1 test also gives you a clean incremental ROAS number to plan the next 3 quarters against. That is worth more than a Cyber Week number you cannot reuse.
How we read Cyber Week without a live test
Skipping the test does not mean flying blind. We triangulate Cyber Week performance from 5 sources, each with a known blind spot.
Calibrated platform numbers. Apply the incrementality factor from your last clean test to what Meta and Google report. If your last test showed Meta claiming 2x its true contribution, halve the platform number before you act on it.
MER against last year. Total revenue divided by total ad spend, compared to the same days last year, shows whether the whole system got more efficient. It ignores channel credit, which is the point.
Post-purchase surveys. Tools like Fairing and Triple Whale ask buyers where they heard about you. Survey data catches the YouTube and creator influence that click-based attribution misses.
New customer share and CAC. Track first-time buyers as a share of orders each day. If new customer CAC climbs while returning orders hold, your scaled spend is mostly paying for people who would have bought anyway.
CPMr as an early warning. CPMr, or Cost Per Mille Reached, is CPM x frequency, the cost to reach 1,000 unique people. Phil Kiel, our Director of Paid Social, popularized it internally as a saturation signal. When CPMr climbs, check which input moved. If frequency rises while CPM holds flat, you are cycling the same audience. Rising CPM with flat frequency means auction pressure, which is expected in Cyber Week and calls for a different fix.
When 3 or more of these sources point the same direction, we act on it. If they disagree, we hold budgets steady and wait for the Q1 test to settle the question.
If you already have a test running
Check the end date against your promo calendar today. Your next move depends on where the two overlap.
It ends before your first early-access or Black Friday promo. Let it finish. Read it on the clean weeks only.
It runs into your promo ramp. End it the day before the first promo goes live. Analyze the clean weeks you have. A shorter clean test beats a longer contaminated one.
It was designed to span Cyber Week. Shut it down now. Turn the held-out regions back on before the auction gets expensive, so you do not lose the peak days in those markets.
Do not stitch pre-Cyber Week and post-Cyber Week data into one read to rescue the sample. The 2 periods describe different customers in different auctions. Average them and the result describes neither.
Incrementality testing is the most honest read you can get on paid media. That honesty depends on a stable baseline, a steady treatment and a control group that behaves like your test group without ads. Cyber Week removes all 3 at once while charging you peak prices for the holdout.
Use this season to scale with the data you already trust. Calibrate platform numbers against your last clean test, watch MER and new customer CAC daily, and track CPMr to catch saturation early. Then put a Q1 test on the calendar this week, with the account frozen and the sample sized on normal weeks.
The brands that know their true incremental ROAS in March plan Q4 2027 with real numbers instead of platform claims.