In this paper, Gardner and Altman discuss three main points for either moving away from or supplementing statistical reporting with p-values:<\/p>\n
- \n
- Research often focuses on null hypothesis significance testing with the goal being to identify statistically significant results.<\/li>\n
- However, we are often interested in the magnitude of the factor of interest.<\/li>\n
- Given that research deals with samples of a broader population, the readers are not only interested in the observed magnitude of the estimand but also the degree of variability and plausible range of values for the population. This variability can be quantified using confidence intervals.<\/li>\n<\/ol>\n
Aside from the paper providing a clear explanation of the issue at hand, their appendix offers the equations to calculate confidence intervals for means, differences in means, proportions, and differences in proportions. Thus, I decided to compile the appendix in an R script for those looking to code confidence intervals (and not have to rely on pre-built functions).<\/p>\n
All of this code is available on my GITHUB page<\/a><\/span><\/strong>.<\/p>\n
Confidence Intervals for Means and Differences<\/strong><\/span><\/p>\n
Single Sample<\/em><\/strong><\/p>\n
- \n
- Obtain the mean<\/li>\n
- Calculate the standard error (SE) of the mean as SE = SD\/sqrt(N)<\/strong><\/em><\/li>\n
- Multiply by a t-critical value specific to the level of confidence of interest and the degrees of freedom (DF) for the single sample, DF = N – 1<\/strong><\/em><\/li>\n<\/ol>\n
The confidence intervals are calculated as:<\/p>\n
Low = mean – t_{crit} * SE<\/strong><\/em>
\nHigh = mean + t_{crit} * SE<\/strong><\/em><\/p>\nExample<\/em><\/p>\n
We collect data on 30 participants on a special test of strength and observe a mean of 40 and standard deviation of 10. We want to calculate the 90% confidence interval.<\/p>\n
The steps above can easily be computed in R. First, we write down the known info (the data that is provided to us). We then calculate the standard error and degrees of freedom (N – 1). To obtain the critical value for our desired level of interest, we use the t-distribution is specific to the degrees of freedom of our data.<\/p>\n
\r\n## Known info\r\nN <- 30\r\navg <- 40\r\nSD <- 10\r\n\r\n## Calculate DF and SE\r\nSE <- SD \/ sqrt(N)\r\nDF <- N - 1\r\n\r\n## Calculate the confidence level of interest (90%), the amount of data in each of the the two tails ((1 - level of interest) \/ 2) and the t-critical value\r\nlevel_of_interest <- 0.90\r\ntail_value <- (1 - level_of_interest)\/2\r\n\r\nt_crit <- abs(qt(tail_value, df = DF))\r\nt_crit\r\n\r\n## Calculate the 90% CI\r\nlow90 <- round(avg - t_crit * SE, 1)\r\nhigh90 <- round(avg + t_crit * SE, 1)\r\n\r\ncat("The 90% Confidence Interval is:", low90, " to ", high90)\r\n<\/pre>\n![\"\"](\"https:\/\/optimumsportsperformance.com\/blog\/wp-content\/uploads\/2024\/01\/Screenshot-2024-01-31-at-8.46.25\u202fPM-1024x92.png\")
<\/a><\/p>\nTwo Samples<\/em><\/strong><\/p>\n
- \n
- Obtain the sample mean and standard deviations for the two samples<\/li>\n
- Pool the estimate of the standard deviation:s = sqrt(((n_1 – 1)*s^2_1 + n_2 – 1)*s^2_2) \/ (n_1 + n_2 – 2))<\/strong><\/em><\/li>\n
- Calculate the SE for the difference:SE_{diff} = s * sqrt(1\/n_1 + 1\/n_2)<\/strong><\/em><\/li>\n
- Calculate the confidence interval as:<\/li>\n<\/ol>\n
Low = (x_1 – x_2) – t_{crit} * SE_{diff}<\/strong><\/em>
\nHigh = (x_1 – x_2) + t_{crit} * SE_{diff}<\/strong><\/em><\/p>\nExample<\/em><\/p>\n
The example in the paper provides the following info:<\/p>\n
- \n
- Blood pressure levels were measured in 100 diabetic and 100 non-diabetic men aged 40-49 years old.<\/li>\n
- Mean systolic blood pressure was 146.4 mmHg (SD = 18.5) in diabetics and 140.4 mmHg (SD = 16.8) in non-diabetics.<\/li>\n<\/ul>\n
Calculate the 95% CI.<\/p>\n
\r\n## store the known data\r\nN_diabetic <- 100\r\nN_non_diabetic <- 100\r\ndiabetic_avg <- 146.4\r\ndiabetic_sd <-18.5\r\nnon_diabetic_avg <- 140.4\r\nnon_diabetic_sd <- 16.8\r\n\r\n## calculate the difference in means, the pooled SD, and the SE of diff\r\ngroup_diff <- diabetic_avg - non_diabetic_avg\r\n\r\npooled_sd <- sqrt(((N_diabetic - 1)*diabetic_sd^2 + (N_non_diabetic - 1)*non_diabetic_sd^2) \/ (N_diabetic + N_non_diabetic - 2))\r\n\r\nse_diff <- pooled_sd * sqrt(1\/N_diabetic + 1\/N_non_diabetic)\r\n\r\n## Calculate the confidence level of interest (95%), the amount of data in each of the the two tails ((1 - level of interest) \/ 2) and the t-critical value\r\nlevel_of_interest <- 0.95\r\ntail_value <- (1 - level_of_interest)\/2\r\n\r\nt_crit <- abs(qt(tail_value, df = N_diabetic + N_non_diabetic - 2))\r\nt_crit\r\n\r\n## Calculate the 95% CI\r\nlow95 <- round(group_diff - t_crit * se_diff, 1)\r\nhigh95 <- round(group_diff + t_crit * se_diff, 1)\r\n\r\ncat("The 95% Confidence Interval is:", low95, " to ", high95)\r\n<\/pre>\n![\"\"](\"https:\/\/optimumsportsperformance.com\/blog\/wp-content\/uploads\/2024\/01\/Screenshot-2024-01-31-at-8.53.48\u202fPM-1024x85.png\")
<\/a><\/p>\nConfidence Intervals for Proportions<\/strong><\/span><\/p>\n
Single Sample<\/em><\/p>\n
- \n
- Obtain the proportion for the population<\/li>\n
- Calculate the SE of the proportion, SE = sqrt((p * (1-p)) \/ N)<\/strong><\/em><\/li>\n
- Obtain the z-critical value from a standard normal distribution for the level of confidence of interest (since the value for a proportion does not depend on sample size as it does for means).<\/li>\n
- Calculate the confidence interval:<\/li>\n<\/ol>\n
low = p – z_{crit} * SE<\/strong>
\nhigh = p + z_{crit} * SE<\/strong><\/p>\nExample<\/em><\/p>\n
We observe a basketball player with 80 field goal attempts and a FG% of 39%. Calculate the 90% CI.<\/p>\n
\r\n## Store the known info\r\nN <- 80\r\nfg_pct <- 0.39\r\n\r\n## Calculate SE\r\nse <- sqrt((fg_pct * (1 - fg_pct)) \/ N)\r\n\r\n## Calculate z-critical value for 50% confidence\r\nlevel_of_interest <- 0.95\r\ntail_value <- (1 - level_of_interest) \/ 2\r\nz_crit <- qnorm(p = tail_value, lower.tail = FALSE)\r\n\r\n## Calculate the 95% CI\r\nlow95 <- round(fg_pct - z_crit * se, 3)\r\nhigh95 <- round(fg_pct + z_crit * se, 3)\r\n\r\ncat("The 95% Confidence Interval is:", low95, " to ", high95)\r\n<\/pre>\n![\"\"](\"https:\/\/optimumsportsperformance.com\/blog\/wp-content\/uploads\/2024\/01\/Screenshot-2024-01-31-at-8.56.01\u202fPM-1024x95.png\")
<\/a><\/p>\nTwo Samples<\/em><\/p>\n
- \n
- Calculate the difference in proportions between the two groups<\/li>\n
- Calculate the SE of the difference in proportions:SE_{diff} = sqrt(((p_1 * (1-p_1)) \/ n_1) + ((p_2 * (1 – p_2)) \/ n_2))<\/strong><\/em><\/li>\n
- Calculate the z-critical value for the level of interest<\/li>\n
- Calculate the confidence interval as:<\/li>\n<\/ol>\n
low = (p_1 – p_2) – (z_{crit} * se_{diff})<\/strong><\/em>
\nhigh = (p_1 – p_2) + (z_{crit} * se_{diff})<\/strong><\/em><\/p>\nExample of two unpaired samples
\n<\/em><\/p>\nThe study provides the following table of example data:<\/p>\n
\r\ndata.frame(\r\n response = c("improvement", "no improvement", "total"),\r\n treatment_A = c(61, 19, 80),\r\n treatment_B = c(45, 35, 80)\r\n)\r\n<\/pre>\n![\"\"](\"https:\/\/optimumsportsperformance.com\/blog\/wp-content\/uploads\/2024\/01\/Screenshot-2024-01-31-at-8.58.32\u202fPM.png\")
<\/a><\/p>\nThe difference we are interested in is between the proportion who improved in treatment A and the proportion of those who improved in treatment B.<\/p>\n
\r\n## Obtain the two proportions from the table and total sample sizes\r\npr_A <- 61\/80\r\nn_A <- 80\r\npr_B <- 45\/80\r\nn_B <- 80\r\n\r\n## calculate the difference in proportions\r\ndiff_pr <- pr_A - pr_B\r\n\r\n## Calculate the SE\r\nse_diff <- sqrt((pr_A * (1 - pr_A))\/n_A + (pr_B * (1 - pr_B))\/n_B)\r\n\r\n## Get z-critical value for 95% confidence\r\nlevel_of_interest <- 0.95\r\ntail_value <- (1 - level_of_interest) \/ 2\r\nz_crit <- qnorm(p = tail_value, lower.tail = FALSE)\r\n\r\n## Calculate the 95% CI\r\nlow95 <- round(diff_pr - z_crit * se_diff, 3)\r\nhigh95 <- round(diff_pr + z_crit * se_diff, 3)\r\n\r\ncat("The 95% Confidence Interval is:", low95, " to ", high95)\r\n<\/pre>\n![\"\"](\"https:\/\/optimumsportsperformance.com\/blog\/wp-content\/uploads\/2024\/01\/Screenshot-2024-01-31-at-8.59.44\u202fPM-1024x80.png\")
<\/a><\/p>\n<\/p>\n
Example for two paired samples<\/em><\/p>\n
We can organize the data in a table like this:<\/p>\n
\r\ndata.frame(\r\n test_1 = c("Present", "Present", "Absent", "Absent"),\r\n test_2 = c("Present", "Absent", "Present", "Absent"),\r\n number_of_subjects = c("a", "b", "c", "d")\r\n)\r\n<\/pre>\n![\"\"](\"https:\/\/optimumsportsperformance.com\/blog\/wp-content\/uploads\/2024\/01\/Screenshot-2024-01-31-at-9.01.18\u202fPM.png\")
<\/a><\/p>\nLet’s say we measured a group of subjects for a specific disease twice in a study. A subject either has the disease (present) or does not (absent) in the two time points. We observe the following data:<\/p>\n
\r\ndat <- data.frame(\r\n test_1 = c("Present", "Present", "Absent", "Absent"),\r\n test_2 = c("Present", "Absent", "Present", "Absent"),\r\n number_of_subjects = c(10, 25, 45, 5)\r\n)\r\n\r\ndat\r\n\r\n## total sample size\r\nN <- sum(dat$number_of_subjects)\r\n<\/pre>\nIf we care about comparing those that had the disease (Present) on both occasions (both Test1 and Test2) we calculate them as:<\/p>\n
p_1 = (a + b) \/ N<\/strong><\/em>
\np_2 = (a + c) \/ N<\/strong><\/em>
\nDiff = p_1 – p_2<\/strong><\/em><\/p>\nThe SE of the difference is:<\/p>\n
SE_{diff} = 1\/N * sqrt(b + c – (b-c)^2\/N)<\/strong><\/em><\/p>\n
\r\n## Obtain the info we need from the table\r\np1 <- (10 + 25) \/ N\r\np2 <- (10 + 45) \/ N\r\ndiff_prop <- p1 - p2\r\ndiff_prop\r\n\r\n## Calculate the SE of the difference\r\nse_diff <- 1 \/ N * sqrt(25 + 45 - (25+45)^2\/N)\r\n\r\n## Get z-critical value for 95% confidence\r\nlevel_of_interest <- 0.95\r\ntail_value <- (1 - level_of_interest) \/ 2\r\nz_crit <- qnorm(p = tail_value, lower.tail = FALSE)\r\n\r\n## Calculate the 95% CI\r\nlow95 <- round(diff_prop - z_crit * se_diff, 3)\r\nhigh95 <- round(diff_prop + z_crit * se_diff, 3)\r\n\r\ncat("The 95% Confidence Interval is:", low95, " to ", high95)\r\n<\/pre>\n![\"\"](\"https:\/\/optimumsportsperformance.com\/blog\/wp-content\/uploads\/2024\/01\/Screenshot-2024-01-31-at-9.03.32\u202fPM-1024x88.png\")
<\/a><\/p>\nAs always, all of this code is available on my GITHUB page<\/span><\/a><\/strong>.<\/p>\n
And, if you’d like to read the full paper, you can find it here:<\/p>\n
- Calculate the SE for the difference:SE_{diff} = s * sqrt(1\/n_1 + 1\/n_2)<\/strong><\/em><\/li>\n
- Multiply by a t-critical value specific to the level of confidence of interest and the degrees of freedom (DF) for the single sample, DF = N – 1<\/strong><\/em><\/li>\n<\/ol>\n