Advice - Random stats drawer
Gist
Just random stats advice placed in this drawer until they’re better organized
Simulation is great for understanding
Simulation of random variables is important in applied statistics for several reasons. First, we use probability models to mimic variation in the world, and the tools of simulation can help us better understand how this variation plays out. Patterns of randomness are notoriously contrary to normal human thinking—our brains don’t seem to be able to do a good job understanding that random swings will be present in the short term but average out in the long run—and in many cases simulation is a big help in training our intuitions about averages and variation. Second, we can use simulation to approximate the sampling distribution of data and propagate this to the sampling distribution of statistical estimates and procedures. Third, regression models are not deterministic; they produce probabilistic predictions. Simulation is the most convenient and general way to represent uncertainties in forecasts
From Regression and other stories (Gelman, Hill and Vehtari 2022)
Within-group analysis is wrong!
Using separate paired tests against baseline and inter- preting only one being significant as indicating a differ- ence between treatments is a frequent practice. It is conceptually wrong, statistically invalid, and conse- quently highly misleading. When the null hypothesis between groups is true, the Type I error can be as high as 50%, rather than the nominal 5%, and even higher when more than two groups are compared.
From Bland and Altman 2011
Again the failure to reject the null hypothesis does not mean the null hypothesis is true!
Failure to observe a statistically signifiant difference between treatments does not mean that they are similar
A common inferential mistake is to assume that failure to observe a statistically significant difference in a conventional superiority trial is evidence that the two treatments being compared are similar.