Active Learning Sampling Design: Glossary and FAQ

Why this matters

Household surveys and phone polls often suffer from unequal response rates across demographic groups. If some strata respond much more than others, estimates can be biased or unnecessarily variable. Active Learning Sampling Design (ALSD) adapts the next wave of sampling toward strata that are under-represented or have lower response propensities, aiming to reduce variance after weighting while still meeting a budget of completed interviews.

Key terms in plain English

• Design effect (from weighting): When we weight respondents to represent the population, some respondents get larger weights than others. This inflates the variance of weighted estimates compared to a simple random sample. The Kish design effect is a standard summary: with weights normalized to mean 1, $D_\mathrm{eff} = \mathrm{mean}(w^2) = 1 + \mathrm{var}(w)$.
• Sample weights: Factors used to make the achieved sample look like the population. Post-stratification weights are proportional to population share divided by sample share within each stratum.
• Traditional methods: Fixed quotas and proportional allocation try to “hit” population shares but ignore heterogeneous response rates; SRS ignores strata entirely. Both can yield large weights (and high design effect) when response rates differ across groups.
• Raking (iterative proportional fitting): A common weighting adjustment that forces weighted margins (e.g., by gender and region) to match population totals. It reduces bias in means but can increase variance if weights become highly variable. ALSD is complementary: allocate the next wave to reduce the need for extreme weights in the first place.

What ALSD does

ALSD models (or directly chooses) next-wave sampling probabilities by stratum. It uses a principled objective that approximates the variance inflation from post-strat weights: the expected Kish design effect after adding the next wave. The optimisation chooses probabilities that minimize this expectation, subject to a budget (the expected number of completes in the wave).

Two flavours included here:

1. alsd_simple: A proportional-to-population baseline that allocates next-wave interviewing to close gaps between current respondent counts and population shares.
2. alsd_modelled: A gradient-based optimiser that directly minimises an objective. By default this is expected Kish design effect; we also expose an alternative cross-entropy objective (KL) that encourages the respondent distribution to track the population distribution.

How we evaluate it

• Expected vs realised: We optimise an expected quantity (given current response propensities). After we draw and observe a wave, we log realised design effect from post-stratification weights on the respondents. This helps compare the optimisation target to what actually happened.
• Error vs. sample size: We follow the survey estimate’s absolute error as respondents accrue, comparing ALSD variants to an SRS proxy and to the simple proportional baseline.
• Determinism and constraints: We make the optimisation reproducible (ADAM with fixed settings) and enforce the wave’s expected completes using a logit-intercept shift, so planning aligns with budget.

Where these ideas come from

The core objective functions and gradients are derived from decomposing the expected respondent totals per stratum in the next wave, translating them into implied post-strat weights, and then aggregating the expected Kish design effect. The cross-entropy (KL) alternative compares the expected respondent distribution to the population distribution. Both are natural, convex-like targets in practice and produce intuitive allocations.

Try it quickly

See the walkthrough vignette (vignettes/walkthrough.Rmd) for an end-to-end simulation that compares alsd_simple, alsd_modelled, and an SRS proxy, plotting error vs. respondents and logging realised design effect.

Notes and caveats

• Realised vs expected $D_\mathrm{eff}$ can differ because real responses are stochastic; our logging computes Kish $D_\mathrm{eff}$ from respondent-level post-strat weights with mean normalization.
• Legacy plots sometimes show $\mathrm{var}(w)$ without the “+1”; we report Kish $D_\mathrm{eff} = \mathrm{mean}(w^2)$. Both can be computed if needed.
• Raking after the fact is still useful; ALSD tries to make the raking step gentler by avoiding extreme weights.

What’s missing (roadmap)

• Weighted summaries: Optionally report weighted (post-stratified or raked) estimates alongside unweighted ones in the simulation outputs.
• Diagnostics: Show optimizer/convergence diagnostics and the per-stratum selection probabilities by wave.
• Dual $D_\mathrm{eff}$ metrics: Allow logging $\mathrm{var}(w)$ in addition to Kish $D_\mathrm{eff} = \mathrm{mean}(w^2)$ for backwards comparability.
• Costs: Expose a cost-aware penalty or constraint if strata have heterogeneous per-complete costs.

# Minimal example: run a short simulation
library(activesampling)
data(zimbabwe_minisample)
res <- run_alsd_simulation(zimbabwe_minisample, c("gender","region"),
                           outcome_var = "fcs_insuff_flag",
                           method = "alsd_modelled", total_budget = 200,
                           wave_size = 50, initial_sample_size = 100,
                           enforce_target = TRUE, optimizer = "adam")
res$summary