rlsmeansemmeans

Perform hypothesis testing on coefficients from separate linear models

Imagine I have two separate lm objects

data(mtcars)

lm1 <- lm(mpg ~ wt, data = mtcars)
lm2 <- lm(mpg ~ wt + disp, data = mtcars)

In this case I'd like to compare both wt coefficients, and perform a hypothesis test on the null that the coefficients in both models are equal(for technical reason I need to actually have two models, rather than just including an interaction)

Solution

  • Since you want to perform a hypothesis test on the estimates, I suggest a fully Bayesian model, which will get you the full posterior distribution of every variable.

    rstanarm is based on Stan, and offers convenient functions that mimic the usual lm, glm syntax; if you want to know more about Stan/RStan, see here.

    Based on the posterior distributions of every variable, we can then perform e.g. a t test and Kolmogorov-Smirnov test to compare the full posterior densities for every variable.

    Here is what you could do:

    1. Perform the model fits.

      library(rstanarm);
lm1 <- stan_lm(mpg ~ wt, data = mtcars, prior = NULL);
lm2 <- stan_lm(mpg ~ wt + disp, data = mtcars, prior = NULL);

      Note how easy it is to run a fully Bayesian linear model with rstanarm.

    2. Extract the posterior densities for all shared coefficients (in this case, the (Intercept) and wt).

      library(tidyverse);
shared.coef <- intersect(names(coef(lm1)), names(coef(lm2)));
shared.coef;
#[1] "(Intercept)" "wt"
df1 <- lm1 %>%
    as.data.frame() %>%
    select(one_of(shared.coef)) %>%
    mutate(model = "lm1");
df2 <- lm2 %>%
    as.data.frame() %>%
    select(one_of(shared.coef)) %>%
    mutate(model = "lm2");

      The posterior densities for 4000 MCMC draws are stored in two data.frames.

    3. We plot the posterior densities.

      # Plot posterior densities for all common parameters
bind_rows(df1, df2) %>%
    gather(var, value, 1:length(shared.coef)) %>%
    ggplot(aes(value, colour = model)) +
        geom_density() +
        facet_wrap(~ var, scale = "free");

      enter image description here

    4. We compare the posterior density distributions of every shared parameter in a t test and a KS test. Here I'm using library broom to tidy-up the output. 

      # Perform t test and KS test
library(broom);
res <- lapply(1:length(shared.coef), function(i)
    list(t.test(df1[, i], df2[, i]), ks.test(df1[, i], df2[, i])));
names(res) <- shared.coef;
lapply(res, function(x) bind_rows(sapply(x, tidy)));
#$`(Intercept)`
#   estimate estimate1 estimate2 statistic p.value parameter  conf.low conf.high
#1 -4.497093  30.07725  34.57434 -104.8882       0  7155.965 -4.581141 -4.413045
#2        NA        NA        NA    0.7725       0        NA        NA        NA
#                              method alternative
#1            Welch Two Sample t-test   two.sided
#2 Two-sample Kolmogorov-Smirnov test   two-sided
#
#$wt
#   estimate estimate1 estimate2 statistic      p.value parameter  conf.low
#1 0.1825202 -3.097777 -3.280297  9.120137 1.074479e-19  4876.248 0.1432859
#2        NA        NA        NA  0.290750 0.000000e+00        NA        NA
#  conf.high                             method alternative
#1 0.2217544            Welch Two Sample t-test   two.sided
#2        NA Two-sample Kolmogorov-Smirnov test   two-sided
#
#There were 12 warnings (use warnings() to see them)

      (The warnings originate from unequal factor levels when binding rows, and can be ignored.)