- Published
- Nov 1, 2026
- Reading time
- 11 min read
Geo Holdout Too Noisy? Audit Power and Detectable Lift
Assess noisy geo holdouts with lift denominators, historical simulations, power assumptions and uncertainty before treating a point estimate as a result.

In this guide (9 sections)
If a planned geo experiment can reliably detect only effects larger than the effect you need to measure, reassess the design before running it or using its point estimate in reporting. First verify what the minimum detectable effect means, which outcome and denominator it uses, and whether historical simulations represent the proposed intervention. Raw sales volatility is not itself the experiment's detection threshold.
A wide interval after the study also needs an honest interpretation. An estimate can be useful evidence about uncertainty without being precise enough to become a fixed channel-lift assumption. More spend, smaller regions, a longer test or different software each changes a different part of the problem; none is a universal repair.
The real question: when the intended effect is smaller than the MDE#
A public question about noisy geo holdouts describes volatile purchase totals and a GeoLift simulation whose minimum detectable effect appears larger than the anticipated channel effect. The poster wants a sufficiently reliable estimate for internal paid-media reporting. Replies suggest longer tests, changed markets, historical checks and alternative comparisons, but do not establish a validated result for that business.
The opened original is an older discussion. Some captured replies have newer relative-age labels, but exact creation and latest substantive-activity timestamps were not verified here. The percentages and vendor criticisms in that thread are poster assertions, not independently measured campaign findings.
This guide addresses the design and reporting task using official references checked on 1 November 2026. GeoLift's homepage identifies it as an open-source research tool, not a Meta product. Its documentation can guide an analyst's investigation; it does not certify a particular study or replace checking its assumptions.
Check whether the experiment can detect the effect you need
An MDE larger than the effect of interest may make the design uninformative. Discuss the outcome and denominator behind the estimate.
Discuss geo measurementA short inquiry to scope relevant options; no campaign access is needed.
1. Write the question and effect unit before changing the model#
“Can this channel create lift?” leaves several incompatible experiments possible. Specify the intervention: introducing advertising, withholding an existing channel, or changing a defined campaign bundle. Name the geographic assignment unit, eligible outcome, observation period and implementation boundaries.
Then define the effect the business needs to distinguish. Relative purchase lift, additional orders and incremental return on ad spend are different units. Record the denominator and any value or spend inputs rather than copying an unexplained MDE percentage from a chart.
| Item | Definition to retain | Ambiguity to remove |
|---|---|---|
| Outcome | Accepted purchases, units or a stated revenue measure | Purchases mixed with page visits or modeled conversion credit |
| Geography | Stable assignment and outcome-location rules | Exposure location silently substituted for purchase location |
| Intervention | Exact campaigns and changes in each group | “Media on” without specifying which media |
| Effect | Absolute units or relative change against a defined counterfactual | Percentage lift without its baseline |
| Decision | The effect range that would change the decision | A positive point estimate treated as sufficient |
Google's Conversion Lift overview separates absolute incremental conversions from value and return measures. Its controlled-study description is a useful boundary: attributed conversions alone do not supply the outcome that would have occurred without the intervention. Use the metric definitions guide when the underlying outcome is still being confused with clicks, reach or views.
2. Separate raw volatility from counterfactual error#
Large changes in sales do not automatically make a geo test impossible. The comparison asks how well the untreated information explains the treatment group's outcome under the chosen design. Shared seasonal movement and a treatment-specific disruption create different diagnostic problems.
Build a complete location-by-time outcome panel. Preserve the original extraction, date boundaries, timezone, unit definitions and geographic mappings. Distinguish a verified zero from a missing row. Investigate tracking outages, returns-policy changes, distribution gaps and geographic remapping before interpreting them as demand variation.
The GeoLift walkthrough starts with location, time and outcome data, warns about incomplete inputs and describes prospective simulation. Inspect the imported panel and its actual retained regions; a completed function call does not establish that all intended data survived preparation.
As an analyst's diagnostic, plot observed and predicted outcomes over the historical comparison, then inspect the residual difference. Record the periods and markets where that difference changes. A strong average fit can conceal a holiday, promotion or outage that matters for the planned test dates. Keep unresolved structural changes in the design review instead of deleting them merely to improve a fit metric.
3. Check sensitivity across plausible historical windows#
The relevant question is whether the proposed setup detects an effect of the size that matters under the chosen simulation and inference assumptions. Record the treatment-market set, duration, effect grid, significance setting, test direction, model specification and historical windows together.
GeoLift's lookback-window explanation describes shifting simulated experiments through historical periods. That helps reveal sensitivity to timing. It also describes a tradeoff: looking further back leaves less pre-period information for each simulated fit. A larger lookback value is not automatically a more faithful forecast.
Include zero-effect simulations and inspect their behavior as well as the intended effects. Also review historical placebo windows where no planned intervention was recorded. These checks can expose implausible detections or unstable timing, but do not prove that every historical comparison had zero real-world disturbance. Annotate promotions, regional shocks and other media changes that could invalidate that interpretation.
Keep all candidate results with their specifications. Do not present only the market combination with the most convenient simulated MDE. If different windows produce materially different sensitivity, report that range and investigate its cause before reducing the design to one favorable threshold.
4. Treat power and MDE as design-dependent outputs#
Power concerns the chance that the specified procedure detects a specified effect when the simulation's assumptions hold. MDE is a sensitivity threshold under a stated design and target-power criterion. Neither is a universal property of the business, nor a guarantee about the eventual result.
Before approving a study, request the power curve and the choices behind it. State the anticipated effect as an assumption, not a promised outcome. If the design has weak power at that effect, an inconclusive result remains plausible even when the intervention has a real effect.
GeoLift best practices emphasize stable historical information, matching on the outcome, purchase-cycle coverage and other media activity. These are design considerations to assess with the analyst, not a universal test-duration or budget prescription.
| Possible design change | What to investigate | What the change cannot guarantee |
|---|---|---|
| Different treatment/control markets | Historical predictive relationship and feasible targeting | Smaller regions always have less noise |
| Longer observation period | Purchase cycle, seasonal coverage and operational stability | More days remove a structural break |
| Stronger intervention contrast | Actual difference between groups and operational constraints | More spend creates positive incremental return |
| Broader campaign bundle | Whether a larger combined effect answers the decision | Separate channel effects become identifiable |
| Different outcome | Reliable measurement and relevance to the business question | A visit-lift result becomes a purchase-lift result |
Changing the outcome changes the question. A more detectable website response may be useful, but keep it as that outcome rather than relabeling it as recovered sales evidence.
5. Verify the geographic intervention independently of the statistics#
Freeze the approved group list and intervention schedule before launch. Retain actual campaign settings and delivery by compatible regions and periods. Log changes in other channels, pricing, stock availability, site operation and local promotions. Record deviations even if they make the result harder to explain.
Exposure and outcome geography can differ. Google's geographic Conversion Lift guidance discusses contamination when exposure and conversion cross group boundaries. It also limits availability to eligible accounts; an open-source study is not automatically equivalent to that managed offering.
For a separate targeting discrepancy, follow the geo-delivery diagnostic. That audit helps establish what location evidence means. It does not by itself validate experimental assignment or reveal what would have happened without advertising.
Keep an intervention-integrity record beside the outcome panel. If a control region receives relevant exposure, document its scope and ask the analyst how it affects interpretation. Do not quietly move that region after seeing the outcome and then report the revised comparison as the original design.
6. Report the estimate and uncertainty together#
The inference record needs the outcome and effect unit, study population, dates, method, confidence level and any model warnings. GeoLift's confidence-interval explanation describes distinct inferential methods and a fallback path. Retain the method actually used rather than assuming every output has identical statistical semantics.
Hypothetical arithmetic only: suppose the estimated no-intervention purchase total for the treatment regions is 10,000, while their observed total is 10,600. The estimated difference is 600 purchases. Relative to that same counterfactual baseline, the point estimate is 600 / 10,000 = 6%. It is not six percentage points of a user conversion rate.
Suppose a compatible uncertainty analysis gives an illustrative interval from −600 to +1,800 incremental purchases, expressed against that fixed illustrative baseline as −6% to +18%. This example is invented; it is not a GeoLift run or an AdProof result. The point estimate is positive, but the interval spans negative, zero and positive effects under the stated assumptions. Reporting only “6% incremental sales” discards the central limitation.
Do not interpret the interval as a probability distribution over the true effect. Do not infer a p-value from an unrelated interval or compare confidence levels without checking their definitions. If the result is too imprecise for a fixed modeling input, keep it as uncertain evidence and document the proposed next experiment or sensitivity treatment.
Alternative comparisons can expose sensitivity. They should not become a search for whichever model reports the preferred lift. Define their role in advance, keep the primary result visible and ask a qualified analyst to assess disagreements. Agreement among several related models does not independently prove the counterfactual correct.
Acceptance checklist before using the study in reporting#
- Freeze the intervention, geographic units, outcome and effect denominator.
- Verify the outcome panel's completeness, mappings and historical changes.
- Preserve simulated effects, power criteria, timing windows and all reviewed candidates.
- Document the actual treatment contrast, contamination and concurrent changes.
- Retain the final method, estimate, interval, confidence level and warnings.
- State which decision the precision supports and what remains inconclusive.
Keep campaign delivery, audience overlap, attributed outcomes and estimated causal effects in separate reporting rows. The linear-and-streaming overlap guide explains why additional measured reach is not the same as additional purchases.
For supported campaign-delivery investigation, AdProof can export campaign/event/report records scoped by dates, timezone and available dimensions. Use those records to document the delivered population alongside a separately owned experimental outcome dataset. AdProof does not provide a GeoLift integration, calculate a study's power by exporting impressions, or certify causal sales lift.
The anonymized fashion-retail delivery case establishes a delivery baseline, not experiment sensitivity or incremental revenue. Discuss a scoped delivery review with the intervention and reconciliation question when that evidence is missing from the study packet.
Frequently asked questions#
Does high sales volatility mean geo testing cannot work?
Not by itself. Inspect how well the historical comparison explains the treatment outcome and test the proposed design's sensitivity. Raw outcome variation and the remaining counterfactual error are different quantities.
Is an MDE above my anticipated lift proof that the channel has no effect?
No. It indicates a sensitivity problem under the specified design and simulation assumptions. A small real effect can remain difficult to distinguish. Reassess the design or retain an inconclusive finding rather than converting it into a zero-effect claim.
Should I simply run longer or spend more?
Evaluate those changes prospectively against power, purchase-cycle coverage, contamination and operational stability. Neither change repairs every data or design defect, and neither guarantees a positive result or acceptable economics.
Can I use a positive point estimate as a fixed channel-lift multiplier?
Report its uncertainty and decision limits first. A point estimate with a broad interval is not a precise, portable constant. Keep the study's scope, method and assumptions with any modeling use, and have the analyst document how uncertainty is handled.
Check whether the experiment can detect the effect you need
An MDE larger than the effect of interest may make the design uninformative. Discuss the outcome and denominator behind the estimate.
Discuss geo measurementA short inquiry to scope relevant options; no campaign access is needed.
