Tests and methods available in add_p() and add_difference()

The tests below are available in add_p() and add_difference(), by name. They are computed with the stats package (and survival for survival tables), so nothing beyond base R is needed to run them. Tests listed with ... accept additional arguments through test.args; for example, to compute a p-value from t.test() assuming equal variances, use tbl_summary(trial, by = trt) |> add_p(age ~ "t.test", test.args = age ~ list(var.equal = TRUE)).

The test argument also accepts the name of a function in your session, or a function itself; see the section on custom functions.

`tbl_summary() |> add_p()`

alias	description	pseudo-code	details
`"t.test"`	t-test	`t.test(variable ~ as.factor(by), data = data, conf.level = 0.95, ...)`	`by` must have two levels. Welch's test unless `var.equal = TRUE` is passed in `test.args`.
`"paired.t.test"`	Paired t-test	`t.test(x = variable in by level 1, y = variable in by level 2, paired = TRUE, conf.level = 0.95, ...)`	Observations are paired on `group`.
`"oneway.test"`	One-way ANOVA	`oneway.test(variable ~ as.factor(by), data = data, ...)`	Welch's ANOVA unless `var.equal = TRUE` is passed in `test.args`.
`"kruskal.test"`	Kruskal-Wallis test	`kruskal.test(data[[variable]], as.factor(data[[by]]), ...)`
`"wilcox.test"`	Wilcoxon rank sum test	`wilcox.test(as.numeric(variable) ~ as.factor(by), data = data, conf.int = TRUE, conf.level = 0.95, ...)`	`by` must have two levels.
`"paired.wilcox.test"`	Paired Wilcoxon rank sum test	`wilcox.test(x = variable in by level 1, y = variable in by level 2, paired = TRUE, conf.int = TRUE, conf.level = 0.95, ...)`	Observations are paired on `group`.
`"mood.test"`	Mood two-sample test of scale	`mood.test(variable ~ as.factor(by), data = data, ...)`	`by` must have two levels.
`"chisq.test"`	chi-square test of independence	`chisq.test(x = data[[variable]], y = data[[by]], ...)`	With Yates' continuity correction for 2 x 2 tables.
`"chisq.test.no.correct"`	chi-square test of independence	`chisq.test(x = data[[variable]], y = data[[by]], correct = FALSE, ...)`
`"fisher.test"`	Fisher's exact test	`fisher.test(x = data[[variable]], y = data[[by]], conf.level = 0.95, ...)`	For large tables pass `test.args = list(simulate.p.value = TRUE)`.
`"prop.test"`	Test for equality of proportions	`prop.test(x = successes by level, n = observations by level, conf.level = 0.95, ...)`	Dichotomous variables only; `by` must have two levels.
`"mcnemar.test"`	McNemar's test	`mcnemar.test(x = table(variable in by level 1, variable in by level 2), ...)`	Observations are paired on `group`.
`"ancova"`	ANCOVA	`lm(variable ~ by + adj.vars, data = data)`	`by` must have two levels; the estimate is the difference between its first and second level, adjusted for `adj.vars`.

The method names reported in the footnote are R's, with three shortened: "Wilcoxon rank sum test", "Pearson's Chi-squared test" and "Fisher's exact test".

`tbl_summary() |> add_difference()`

alias	description	pseudo-code	details
`"t.test"`	t-test	`t.test(variable ~ as.factor(by), data = data, conf.level = conf.level, ...)`	The difference in means.
`"wilcox.test"`	Wilcoxon rank sum test	`wilcox.test(as.numeric(variable) ~ as.factor(by), data = data, conf.int = TRUE, conf.level = conf.level, ...)`	The Hodges-Lehmann estimate of the difference in location.
`"paired.t.test"`	Paired t-test	`t.test(x = variable in by level 1, y = variable in by level 2, paired = TRUE, conf.level = conf.level, ...)`	Observations are paired on `group`.
`"prop.test"`	Test for equality of proportions	`prop.test(x = successes by level, n = observations by level, conf.level = conf.level, ...)`	Dichotomous variables only; the difference in proportions.
`"ancova"`	ANCOVA	`lm(variable ~ by + adj.vars, data = data)`	The adjusted difference in means, with `confint()`'s interval. "One-way ANOVA" in the footnote without `adj.vars`.
`"smd"`	Standardized mean difference	`smd::smd(x = data[[variable]], g = data[[by]], std.error = TRUE)`, computed in base R	Continuous variables use the population variance of each group; factors use the multinomial covariance of the level proportions (Yang and Dalton, 2012). The interval is normal. No p-value.
`"cohens_d"`	Cohen's d	`effectsize::cohens_d(variable ~ by, data = data, ci = conf.level, ...)`, computed in base R	Numeric variables; `by` must have two levels. The difference in means over the pooled standard deviation, or over the root mean of the two variances with Welch's degrees of freedom when `test.args` passes `pooled_sd = FALSE`; `mu` and `alternative` are also accepted. The interval is the noncentral t interval. No p-value.
`"hedges_g"`	Hedges' g	`effectsize::hedges_g(variable ~ by, data = data, ci = conf.level, ...)`, computed in base R	Cohen's d and its limits times the exact small-sample correction.
`"paired_cohens_d"`	Paired Cohen's d	`effectsize::cohens_d(x = variable in by level 1, y = variable in by level 2, paired = TRUE, ci = conf.level, ...)`, computed in base R	Observations are paired on `group`; the mean of the pair differences over their standard deviation. `mu` and `alternative` are accepted in `test.args`.
`"paired_hedges_g"`	Paired Hedges' g	`effectsize::hedges_g(x = variable in by level 1, y = variable in by level 2, paired = TRUE, ci = conf.level, ...)`, computed in base R	Paired Cohen's d and its limits times the exact small-sample correction.

Not available in ltsummary: "emmeans" (emmeans), "lme4" and "ancova_lme4" (lme4).

`tbl_continuous() |> add_p()`

alias	description	pseudo-code	details
`"anova_2way"`	Two-way ANOVA	`lm(continuous_variable ~ variable + by, data = data)`, the omnibus F-test	Both predictors as factors, the include variable first. The default when the table has a `by` variable.
`"t.test"`, `"oneway.test"`, `"kruskal.test"`, `"wilcox.test"`, `"ancova"`	The `tbl_summary()` tests with the roles swapped	as above, with `variable = continuous_variable` and `by = variable`	Cannot be used when the table has a `by` variable; the p-value is left missing.

A custom function receives the include variable as variable =, the by variable as by = and the summarized variable as continuous_variable =. Not available in ltsummary: "aov", "lme4" and "ancova_lme4".

`tbl_survfit() |> add_p()`

alias	description	pseudo-code
`"logrank"`	Log-rank test	`survival::survdiff(Surv(.) ~ variable, data, rho = 0)`
`"petopeto_gehanwilcoxon"`	Peto & Peto modification of Gehan-Wilcoxon test	`survival::survdiff(Surv(.) ~ variable, data, rho = 1)`
`"survdiff"`	G-rho family test	`survival::survdiff(Surv(.) ~ variable, data, ...)`
`"coxph_lrt"`	Cox regression (LRT)	`survival::coxph(Surv(.) ~ variable, data, ...)`, the likelihood ratio test of `summary()`
`"coxph_wald"`	Cox regression (Wald)	`survival::coxph(Surv(.) ~ variable, data, ...)`, the Wald test of `summary()`
`"coxph_score"`	Cox regression (Score)	`survival::coxph(Surv(.) ~ variable, data, ...)`, the score test of `summary()`

Custom functions

To report a p-value from a test that is not listed, write a function. It returns a one-row data frame, or a list of scalars, with a column named "p.value" and optionally "method" (shown in the footnote), "estimate", "statistic", "std.error", "parameter", "conf.low" and "conf.high". This is the shape of broom::tidy() applied to an htest object, so a function ending in broom::tidy() works as is.

Example calculating a p-value from a t-test assuming a common variance between groups.

ttest_common_variance <- function(data, variable, by, ...) {
  res <- t.test(data[[variable]] ~ as.factor(data[[by]]), var.equal = TRUE)
  list(p.value = res$p.value, method = "t-test with equal variances")
}

trial |>
  tbl_summary(by = trt, include = age) |>
  add_p(test = age ~ "ttest_common_variance")

The function can be passed by name, as above, or as the function object, add_p(test = age ~ ttest_common_variance).

Function arguments

The custom function is called with the arguments fun(data =, variable =, by =, group =, type =, value =, test.args =, conf.level =, tbl =, adj.vars =). A function that accepts ... receives all of them; otherwise only the arguments it declares are passed, so function(data, variable, by) is enough for most tests. Include ... to stay compatible with arguments added in the future.

argument	value
`data=`	The data frame passed to `tbl_summary()`, without rows with a missing `by` value
`variable=`	String, the variable name
`by=`	String, the `by` variable name
`group=`	String, the `add_p(group =)` variable name, or `NULL`
`type=`	String, the summary type of the variable
`value=`	The level shown for a dichotomous variable, or `NULL`
`test.args=`	The list given for the variable in `add_p(test.args =)`, or an empty list
`conf.level=`	Confidence level, `0.95` in `add_p()` and `add_difference(conf.level =)`
`tbl=`	The `tbl_summary` object
`adj.vars=`	Character vector, the `adj.vars` variable names, or `NULL`
`continuous_variable=`	String, the summarized variable of a `tbl_continuous()` table, or `NULL`

Warnings and errors raised by a test are collected, reported once after all tests have run, and the p-value is left NA.

Examples

# Example 1 ----------------------------------
# pass arguments to a built-in test
trial |>
  tbl_summary(by = trt, include = c(age, marker), missing = "no") |>
  add_p(test = everything() ~ "t.test", test.args = all_tests("t.test") ~ list(var.equal = TRUE))

# Example 2 ----------------------------------
# paired tests: `group` identifies the pairs, here a made-up patient id, and
# the data must hold complete pairs, one row per treatment per patient
trial_paired <- trial[c("trt", "marker", "response")]
trial_paired$id <- ave(seq_len(nrow(trial_paired)), trial_paired$trt, FUN = seq_along)
trial_paired <- trial_paired[complete.cases(trial_paired), ]
trial_paired <- trial_paired[trial_paired$id %in% names(which(table(trial_paired$id) == 2)), ]

trial_paired |>
  tbl_summary(
    by = trt,
    include = c(marker, response),
    label = list(marker = "Marker Level (ng/mL)", response = "Tumor Response"),
    missing = "no"
  ) |>
  add_p(test = list(marker ~ "paired.wilcox.test", response ~ "mcnemar.test"), group = id)

# Example 3 ----------------------------------
# a custom test function
ttest_common_variance <- function(data, variable, by, ...) {
  res <- t.test(data[[variable]] ~ as.factor(data[[by]]), var.equal = TRUE)
  list(p.value = res$p.value, method = "t-test with equal variances")
}

trial |>
  tbl_summary(by = trt, include = c(age, marker), missing = "no") |>
  add_p(test = all_continuous() ~ ttest_common_variance)

Tests and methods available in add_p() and add_difference()

tbl_summary() |> add_p()

tbl_summary() |> add_difference()

tbl_continuous() |> add_p()

tbl_survfit() |> add_p()