Question: Write down the null and alternative hypotheses that represent this question. Which hypothesis do you deem more likely?

Question: Have all your data been loaded in correctly? That is, do all data points substantively make sense? If you are unsure, go back to the .csv-file to inspect the raw data.

describe(dataPHD)

##       vars   n    mean     sd median trimmed    mad min  max range  skew
## diff     1 333    9.97  14.43      5    6.91   7.41 -31   91   122  2.21
## child    2 333    0.18   0.38      0    0.10   0.00   0    1     1  1.66
## sex      3 333    0.52   0.50      1    0.52   0.00   0    1     1 -0.08
## age      4 333   31.68   6.86     30   30.39   2.97  26   80    54  4.45
## age2     5 333 1050.22 656.39    900  928.29 171.98 676 6400  5724  6.03
##       kurtosis    se
## diff      5.92  0.79
## child     0.75  0.02
## sex      -2.00  0.03
## age      24.99  0.38
## age2     42.21 35.97

1.Do you understand the priors?

Before actually looking at the data we first need to think about the prior distributions and hyperparameters for our model. For the current model, there are four priors:

• the intercept
• the two regression parameters (\(\beta_1\) for the relation with AGE and \(\beta_2\) for the relation with AGE2)
• the residual variance (\(\in\))

We first need to determine which distribution to use for the priors. Let’s use for the

• intercept a normal prior with \(\mathcal{N}(\mu_0, \sigma^{2}_{0})\), where \(\mu_0\) is the prior mean of the distribution and \(\sigma^{2}_{0}\) is the variance parameter
• \(\beta_1\) a normal prior with \(\mathcal{N}(\mu_1, \sigma^{2}_{1})\)
• \(\beta_2\) a normal prior with \(\mathcal{N}(\mu_2, \sigma^{2}_{2})\)
• \(\in\) an Inverse Gamma distribution with \(IG(\kappa_0,\theta_0)\), where \(\kappa_0\) is the shape parameter of the distribution and \(\theta_0\) the rate parameter

Next, we need to specify actual values for the hyperparameters of the prior distributions. Let’s say we found a previous study and based on this study the following hyperparameters can be specified:

• intercept \(\sim \mathcal{N}(-35, 20)\)
• \(\beta_1 \sim \mathcal{N}(.8, 5)\)
• \(\beta_2 \sim \mathcal{N}(0, 10)\)
• \(\in \sim IG(.5, .5)\) This is an uninformative prior for the residual variance, which has been found to perform well in simulation studies.

It is a good idea to plot these distribution to see how they look. To do so, one easy way is to sample a lot of values from one of these distributions and make a density plot out of it, see the code below. Replace the ‘XX’ with the values of the hyperparameters.

par(mfrow = c(2,2))
plot(density(rnorm(n = 100000, mean = XX, sd = sqrt(XX))), main = "prior intercept") # the rnorm function uses the standard devation instead of variance, that is why we use the sqrt
plot(density(rnorm(n = 100000, mean = XX, sd = sqrt(XX))), main = "effect Age")
plot(density(rnorm(n = 100000, mean = XX, sd = sqrt(XX))), main = "effect Age^2")

par(mfrow = c(2,2))
plot(density(rnorm(n = 100000, mean = -35, sd = sqrt(20))), main = "prior intercept") # the rnorm function uses the standard devation instead of variance, that is why we use the sqrt
plot(density(rnorm(n = 100000, mean = .8,  sd = sqrt(5))),   main = "effect Age")
plot(density(rnorm(n = 100000, mean = 0,   sd = sqrt(10))),  main = "effect Age^2")

We can also plot what the expected delay would be (like we did in the brms regression assignment) given these priors. With these priors the regression formula would be: \(delay=-35+ .8*age + 0*age^2\). These are just the means of the priors and do not yet qualify the different levels of uncertainty. Replace the ‘XX’ in the following code with the prior means.

years <- 20:80
delay <- XX + XX*years + XX*years^2
plot(years, delay, type = "l")

years <- 20:80
delay <- -35 + .8*years + 0*years^2
plot(years, delay, type =  "l")

2. Does the trace-plot exhibit convergence?

First we run the anlysis with only a short burnin period of 250 samples and then take another 500 samples.

The following code is how to specify the regression model:

# 1) set the priors
priors_inf <- c(set_prior("normal(.8, 2.24)", class = "b", coef = "age"),
               set_prior("normal(0, 3.16)", class = "b", coef = "age2"),
               set_prior("normal(-35, 4.47)", class = "b", coef =  "intercept"),
                 set_prior("inv_gamma(.5,.5)", class = "sigma"))

# 2) specify the model
model_few_samples <- brm(formula = diff ~ 0 + intercept + age + age2, 
                         data    = dataPHD,
                         prior   = priors_inf,
                         warmup  = 250,
                         iter    = 500,
                         seed    = 12345)

Now we can plot the trace plots.

modeltranformed <- ggs(model_few_samples) # the ggs function transforms the BRMS output into a longformat tibble, that we can use to make different types of plots.
ggplot(filter(modeltranformed, Parameter %in% c("b_intercept", "b_age", "b_age2", "sigma")),
       aes(x   = Iteration,
           y   = value, 
           col = as.factor(Chain)))+
  geom_line()+
  facet_grid(Parameter ~ .,
             scale     = 'free_y',
             switch    = 'y')+
  labs(title = "Trace plots",
       col   = "Chains") +
  theme_minimal()

Alternatively, you can simply make use of the built-in plotting capabilities of Rstan.

stanplot(model_few_samples, type = "trace")

## No divergences to plot.

It seems like the trace (caterpillar) plots are not neatly converged into one each other (we ideally want one fat caterpillar, like the one for sigma). This indicates we need more samples.

We can check if the chains convergenced by having a look at the convergence diagnostics. Two of these diagnostics of interest include the Gelman and Rubin diagnostic and the Geweke diagnostic.

• The Gelman-Rubin Diagnostic shows the PSRF values (using the within and between chain variability). You should look at the Upper CI/Upper limit, which are all should be close to 1. If they aren’t close to 1, you should use more iterations. Note: The Gelman and Rubin diagnostic is also automatically given in the summary of brms under the column Rhat
• The Geweke Diagnostic shows the z-scores for a test of equality of means between the first and last parts of each chain, which should be <1.96. A separate statistic is calculated for each variable in each chain. In this way it check whether a chain has stabalized. If this is not the case, you should increase the number of iterations. In the plots you should check how often values exceed the boundary lines of the z-scores. Scores above 1.96 or below -1.96 mean that the two portions of the chain significantly differ and full chain convergence was not obtained.

To obtain the Gelman and Rubin diagnostic use:

modelposterior <- as.mcmc(model_few_samples) # with the as.mcmc() command we can use all the CODA package convergence statistics and plotting options
gelman.diag(modelposterior[, 1:4])

## Potential scale reduction factors:
## 
##             Point est. Upper C.I.
## b_intercept       3.92       6.76
## b_age             3.78       6.50
## b_age2            3.02       5.06
## sigma             1.04       1.11
## 
## Multivariate psrf
## 
## 3.3

gelman.plot(modelposterior[, 1:4])

To obtain the Geweke diagnostic use:

geweke.diag(modelposterior[, 1:4])

## [[1]]
## 
## Fraction in 1st window = 0.1
## Fraction in 2nd window = 0.5 
## 
## b_intercept       b_age      b_age2       sigma 
##       5.334     -11.443       7.479       2.465 
## 
## 
## [[2]]
## 
## Fraction in 1st window = 0.1
## Fraction in 2nd window = 0.5 
## 
## b_intercept       b_age      b_age2       sigma 
##      -2.057       1.559      -1.456       1.105 
## 
## 
## [[3]]
## 
## Fraction in 1st window = 0.1
## Fraction in 2nd window = 0.5 
## 
## b_intercept       b_age      b_age2       sigma 
##     0.42201     0.08127    -0.60839    -0.25410 
## 
## 
## [[4]]
## 
## Fraction in 1st window = 0.1
## Fraction in 2nd window = 0.5 
## 
## b_intercept       b_age      b_age2       sigma 
##      0.3407     -0.1883     -0.4051      3.7839

geweke.plot(modelposterior[, 1:4])

These statistics confirm that the chains have not converged. Therefore, we run the same analysis with more samples.

model <- brm(formula = diff ~ 0 + intercept + age + age2, 
             data    = dataPHD,
             prior   = priors_inf,
             warmup  = 2000,
             iter    = 4000,
             seed    = 12345)

Obtain the trace plots again.

stanplot(model, type = "trace")

## No divergences to plot.

Obtain the Gelman and Rubin diagnostic again.

modelposterior <- as.mcmc(model) # with the as.mcmc() command we can use all the CODA package convergence statistics and plotting options
gelman.diag(modelposterior[, 1:4])

## Potential scale reduction factors:
## 
##             Point est. Upper C.I.
## b_intercept          1          1
## b_age                1          1
## b_age2               1          1
## sigma                1          1
## 
## Multivariate psrf
## 
## 1

gelman.plot(modelposterior[, 1:4])

Obtain Geweke diagnostic again.

geweke.plot(modelposterior[, 1:4])

Now we see that the Gelman and Rubin diagnostic (PRSF) is close to 1 for all parameters and the the Geweke diagnostic is not > 1.96.

3. Does convergence remain after doubling the number of iterations?

As is recommended in the WAMBS checklist, we double the amount of iterations to check for local convergence.

model_doubleiter <- brm(formula = diff ~ 0 + intercept + age + age2, 
                        data    = dataPHD,
                        prior   = priors_inf,
                        warmup  = 4000,
                        iter    = 8000,
                        seed    = 12345)

You should again have a look at the above-mentioned convergence statistics, but we can also compute the relative bias to inspect if doubling the number of iterations influences the posterior parameter estimates (\(bias= 100*\frac{(model \; with \; double \; iteration \; – \; initial \; converged \; model )}{initial \; converged \; model}\)). In order to preserve clarity we just calculate the bias of the two regression coefficients.

You should combine the relative bias in combination with substantive knowledge about the metric of the parameter of interest to determine when levels of relative deviation are negligible or problematic. For example, with a regression coefficient of 0.001, a 5% relative deviation level might not be substantively relevant. However, with an intercept parameter of 50, a 10% relative deviation level might be quite meaningful. The specific level of relative deviation should be interpreted in the substantive context of the model. Some examples of interpretations are:

• if relative deviation is < |5|%, then do not worry;
• if relative deviation > |5|%, then rerun with 4x nr of iterations.

Question: calculate the relative bias. Are you satisfied with number of iterations, or do you want to re-run the model with even more iterations?

round(100*((posterior_summary(model_doubleiter)[,"Estimate"] - posterior_summary(model)[,"Estimate"]) / posterior_summary(model)[,"Estimate"]), 4)

## b_intercept       b_age      b_age2       sigma        lp__ 
##     -0.3279     -0.2810     -0.3135     -0.1298     -0.0033

4. Does the posterior distribution histogram have enough information?

By having a look at the posterior distribution histogram plot(fit.bayes, pars=1:4, plot.type="stan_hist"), we can check if it has enough information.

Question: What can you conclude about distribution histograms?

stanplot(model, type = "hist")

## `stat_bin()` using `bins = 30`. Pick better value with `binwidth`.

stanplot(model_few_samples, type = "hist")

## `stat_bin()` using `bins = 30`. Pick better value with `binwidth`.

5. Do the chains exhibit a strong degree of autocorrelation?

To obtain information about autocorrelation the following syntax can be used:

stanplot(model, pars = 1:4, type = "acf")

Question: What can you conclude about these autocorrelation plots?

6. Do the posterior distributions make substantive sense?

We plot the posterior distributions and see if they are unimodel (one peak), if they are clearly centered around one value, if they give a realistic estimate and if they make substantive sense compared to our prior believes (priors). Here we plot the posteriors of the regression coefficients. If you want you can also plot the mean and the 95% Posterior HPD Intervals.

stanplot(model, pars = 1:4, type = "dens")

Question: What is your conclusion; do the posterior distributions make sense?

7. Do different specification of the variance priors influence the results?

So far we have used the –\(\in \sim IG(.5, .5)\) prior, but we can also use the –\(\in \sim IG(.01, .01)\) prior and see if doing so makes a difference. To quantify this difference we again calculate a relative bias.

Question: Are the results robust for different specifications of the prior on the residual variance?

# 1) set the priors
priors_inf2 <- c(set_prior("normal(.8, 2.24)", class = "b", coef = "age"),
               set_prior("normal(0, 3.16)", class = "b", coef = "age2"),
               set_prior("normal(-35, 4.47)", class = "b", coef=  "intercept"),
               set_prior("inv_gamma(.01,.01)", class="sigma"))

# 2) specify the model
model.difIG <- brm(formula = diff ~ 0 + intercept + age + age2, 
                   data    = dataPHD,
                   prior   = priors_inf2,
                   warmup  = 2000,
                   iter    = 4000,
                   seed    = 12345)

posterior_summary(model.difIG)

##                  Estimate   Est.Error          Q2.5         Q97.5
## b_intercept -3.622203e+01 4.223666351 -4.441874e+01 -2.785579e+01
## b_age        2.143473e+00 0.210379606  1.731300e+00  2.554017e+00
## b_age2      -2.065084e-02 0.002681358 -2.576232e-02 -1.542095e-02
## sigma        1.405130e+01 0.543962535  1.305633e+01  1.520437e+01
## lp__        -1.363654e+03 1.427742809 -1.367363e+03 -1.361893e+03

8. Is there a notable effect of the prior when compared with non-informative priors?

The default brms priors are non-informative, so we can run the analysis without any specified priors and compare them to the model we have run thus far, using the relative bias, to see if there is a large influence of the priors.

Question: What is your conclusion about the influence of the priors on the posterior results?

model.default <- brm(formula = diff ~ 0 + intercept + age + age2, 
                     data    = dataPHD,
                     warmup  = 1000,
                     iter    = 2000,
                     seed    = 123)

posterior_summary(model.default)

Question: Which results do you use to draw conclusion on?

9. Are the results stable from a sensitivity analysis?

If you still have time left, you can adjust the hyperparameters of the priors upward and downward and re-estimating the model with these varied priors to check for robustness.

From the original paper:

“If informative or weakly-informative priors are used, then we suggest running a sensitivity analysis of these priors. When subjective priors are in place, then there might be a discrepancy between results using different subjective prior settings. A sensitivity analysis for priors would entail adjusting the entire prior distribution (i.e., using a completely different prior distribution than before) or adjusting hyperparameters upward and downward and re-estimating the model with these varied priors. Several different hyperparameter specifications can be made in a sensitivity analysis, and results obtained will point toward the impact of small fluctuations in hyperparameter values. […] The purpose of this sensitivity analysis is to assess how much of an impact the location of the mean hyperparameter for the prior has on the posterior. […] Upon receiving results from the sensitivity analysis, assess the impact that fluctuations in the hyperparameter values have on the substantive conclusions. Results may be stable across the sensitivity analysis, or they may be highly instable based on substantive conclusions. Whatever the finding, this information is important to report in the results and discussion sections of a paper. We should also reiterate here that original priors should not be modified, despite the results obtained.”

For more information on this topic, please also refer to this paper.

10. Is the Bayesian way of interpreting and reporting model results used?

For a summary on how to interpret and report models, please refer to https://www.rensvandeschoot.com/bayesian-analyses-where-to-start-and-what-to-report/

summary(model)

##  Family: gaussian 
##   Links: mu = identity; sigma = identity 
## Formula: diff ~ 0 + intercept + age + age2 
##    Data: dataPHD (Number of observations: 333) 
## Samples: 4 chains, each with iter = 4000; warmup = 2000; thin = 1;
##          total post-warmup samples = 8000
## 
## Population-Level Effects: 
##           Estimate Est.Error l-95% CI u-95% CI Eff.Sample Rhat
## intercept   -36.40      4.26   -44.66   -28.06       2409 1.00
## age           2.15      0.21     1.74     2.56       2288 1.00
## age2         -0.02      0.00    -0.03    -0.02       2662 1.00
## 
## Family Specific Parameters: 
##       Estimate Est.Error l-95% CI u-95% CI Eff.Sample Rhat
## sigma    14.04      0.55    13.01    15.15       3093 1.00
## 
## Samples were drawn using sampling(NUTS). For each parameter, Eff.Sample 
## is a crude measure of effective sample size, and Rhat is the potential 
## scale reduction factor on split chains (at convergence, Rhat = 1).

In the current model we see that:

• The estimate for the intercept is -36.4 [-44.66 ; -28.06]
• The estimate for the effect of \(age\) is 2.15 [1.74 ; 2.56]
• The estimate for the effect of \(age^2\) is -0.02 [-0.03 ; -0.02]

We can see that none of 95% Posterior HPD Intervals for these effects include zero, which means we are can be quite certain that all of the effects are different from 0.

Remember how we plotted the relation between delay and years based on the prior information? Now, do the same with the posterior estimates.

years <- 20:80
delay <- -35 + 2.13*years - 0.02*years^2
plot(years, delay, type= "l")

References

Depaoli, S., & Van de Schoot, R. (2017). Improving transparency and replication in Bayesian statistics: The WAMBS-Checklist. Psychological Methods, 22(2), 240.

Link, W. A., & Eaton, M. J. (2012). On thinning of chains in MCMC. Methods in ecology and evolution, 3(1), 112-115.

van Erp, S., Mulder, J., & Oberski, D. L. (2017). Prior sensitivity analysis in default Bayesian structural equation modeling.

Van de Schoot, R., & Depaoli, S. (2014). Bayesian analyses: Where to start and what to report. European Health Psychologist, 16(2), 75-84.