![]() ![]() The SPRT is quite successful in that it reduces the average sample size required by 40-50% compared to a fixed sample test with an equivalent design in terms of type I and type II error control. ![]() Since a lot of industrial testing is destructive for the tested item, and takes a fair bit of effort, the goal was to minimize the number of items inspected, in order to either clear a batch of items, or to declare it unfit. The SPRT aimed to reduce the cost of assuring the quality of the different equipment, vehicles, ammunition, etc., produced during the war effort. In 1943 Abraham Wald, then working as part of the Statistical Research Group at Columbia University, developed the Sequential Probability Ratio Test (SPRT). While there were some applications of sequential testing procedure before that, the first significant development in modern sequential testing happened during the Second World War. Early development of sequential statistical testsĮarly development of sequential statistical tests.Online A/B testing remains the focus throughout. The article ends with a brief note on the related concept of tests of power of one. The article will start with a bit of history to show the evolution of sequential testing which will help put the rest of the article into context. It draws heavily on Chapter 10 (“Sequential testing: continuous monitoring of data”) of my book “Statistical Methods in Online A/B Testing” as well my 2017 white paper “Efficient A/B Testing in Conversion Rate Optimization: The AGILE Statistical Method”, but also contains insights not shared before, including commentary on some potentially misleading claims by A/B testing software vendors. This article will present the main available options, namely fully sequential and group sequential tests, their strengths and weaknesses, and attempts to outline what test to use in which scenario. What is the best design for a statistical test with sequential evaluation of the data at multiple points in time? This is a question anyone who has realized that unaccounted for peeking with intent to stop is the bane of A/B testing eventually comes to ask.
0 Comments
Leave a Reply. |