---
title: "PRDA: Prospective and Retrospective Design Analysis"
description: "This vignette offers a general overview of the PRDA package, introducing the concept of design analysis and providing information about the main functions in the package."
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{PRDA}
  %\VignetteEncoding{UTF-8}
  %\VignetteEngine{knitr::rmarkdown}
bibliography: PRDA.bib
editor_options: 
  chunk_output_type: console
---

```{r, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>"
)
library(PRDA)
```


{PRDA} allows performing a prospective or retrospective design analysis to evaluate inferential risks (i.e., power, Type M error, and Type S error) in a study considering Pearson's correlation between two variables or mean comparisons (one-sample, paired, two-sample, and Welch's *t*-test). 

## Introduction to design analysis

The term *Design Analysis* was introduced by @gelmanPowerCalculationsAssessing2014 as a broader definition of Power Analysis. Traditional power analysis has a narrow focus on statistical significance. Design analysis, instead, evaluates together with power levels also other inferential risks (i.e., Type M error and Type S error), to assess estimates uncertainty under hypothetical replications of a study.

Given an *hypothetical effect size* and  the study characteristics (i.e., sample size, statistical test directionality, $\alpha$ level), design analysis evaluates:

- **Power**: the probability of the test rejecting the null hypothesis.
- **Type M error** (Magnitude): the factor by which a statistically significant effect is on average exaggerated, also known as *Exaggeration Ratio*.
- **Type S error** (Sign): the probability of finding a statistically significant result in the opposite direction to the hypothetical effect.

Moreover, @gelmanPowerCalculationsAssessing2014 distinguished between two types of design analysis according to the study phase:

- **Prospective Design Analysis**: if the analysis is performed in the planning stage of a study to define the sample size needed to obtain a required level of power.
- **Retrospective Design Analysis**: if the analysis is performed in a later stage when the data have already been collected. This is still useful to evaluate the inferential risks associated with the study.

It is important to do not mistake a retrospective design analysis for post-hoc power analysis. The former defines the hypothetical effect size according to previous results in the literature or experts indications, whereas the latter defines the hypothetical effect size based on the same study results and it is a widely-deprecated practice [@goodmanUsePredictedConfidence1994; @lenthStatisticalPowerCalculations12007; @Senn1304].

### Enhancing researchers awareness

Although Type M error and Type S error depend directly on power level, they underline valuable information regarding estimates uncertainty that would otherwise be overlooked. This enhances researchers awareness about the inferential risks related to their studies and helps them in the interpretation of the results.

Despite the lower chances, a statistically significant result could be obtained even in an underpowered study (e.g., power = 20%). This might seem a promising finding, and researchers might think that getting a statistically significant result in an underpowered study means the results must be reliable. Therefore, they would probably be even more confident in the interpretation of their results.

However, in this scenario statistically significant results are almost certain to be an overestimation of the population effect. As pointed out by @gelman_hill_vehtari_2020 "a key risk for a low-power study is not so much that it has a small chance of succeeding, but rather that an apparent success merely masks a larger failure" (p.292). This is also referred as the "Winner's curse", indicating that the apparent win in terms of a statistically significant result is an actual loss as the obtained estimate is inflated.

For example, in a study considering a two-sample *t*-test with 30 participants per group, if the hypothetical population effect size is small (e.g., Cohen's *d* of .25) the actual power is only 16%.  The associated Type M error is around 2.60 and  the Type S error is 0.01. That means, statistical significant results are on average an overestimation of 160% of the hypothesized population effect and there is a 1% probability of obtaining a statistically significant result in the opposite direction. 

In this scenario, knowing the type M and S errors, researchers would be much more cautious in interpreting the results and might consider carrying out a replication study to obtain more reliable results.

### More on design analysis

To know more about design analysis consider @gelmanPowerCalculationsAssessing2014. While, for an introduction to design analysis considering examples in psychology see @altoeEnhancingStatisticalInference2020 and  @bertoldoDesigningStudiesEvaluating2020.

## The package

Given a plausible value of effect size, {PRDA} performs a prospective or retrospective design analysis to evaluate the inferential risks (i.e., power, Type M error, and Type S error) related to the study design.

{PRDA} package can be used for Pearson's correlation between two variables or mean comparisons (i.e., one-sample, paired, two-sample, and Welch's t-test) considering an hypothetical value of $\rho$ or Cohen's *d* respectively. See [`vignette("retrospective")`](retrospective.html) and [`vignette("prospective")`](prospective.html) to know how to set function arguments for the different effect types. 

### Install

You can install the released version of PRDA from [CRAN](https://CRAN.R-project.org/package=PRDA) with:

``` r
install.packages("PRDA")
```

And the development version from [GitHub](https://github.com/ClaudioZandonella/PRDA/tree/master) with:

```{r setup, eval = F}
# If devtools is not installed yet: 
# install.packages( "devtools" )  
devtools::install_github("CaludioZandonella/PRDA",
                         build_vignettes = TRUE)
library(PRDA)
```

### Functions

In {PRDA} there are two main functions:

- **`retrospective()`**.
Given the hypothetical population effect size and the study sample size, the function `retrospective()` performs a retrospective design analysis. According to the defined alternative hypothesis and the significance level, the inferential risks (i.e., Power level, Type M error, and Type S error) are computed together with the critical effect value (i.e., the minimum absolute effect size value that would result significant). To know more about function arguments and examples see the function documentation `?retrospective` and  [`vignette("retrospective")`](retrospective.html).

- **`prospective()`**.
Given the hypothetical population effect size and the required power level, the function `prospective()` performs a prospective design analysis. According to the defined alternative hypothesis and the significance level, the required sample size is computed together with the associated Type M error, Type S error, and the critical effect value (i.e., the minimum absolute effect size value that would result significant).  To know more about function arguments and examples see the function documentation `?prospective` and [`vignette("prospective")`](prospective.html).

### Hypothetical effect size

The hypothetical population effect size can be defined as a single value according to previous results in the literature or experts indications. Alternatively, {PRDA} allows users to specify a distribution of plausible values to account for their uncertainty about the hypothetical population effect size.  To know how to specify the hypothetical effect size according to a distribution and an example of application see [`vignette("retrospective")`](retrospective.html).



## Case study

@eisenbergerDoesRejectionHurt2003 claimed that social and physical pain seem to share similar neural underpinnings. Their experiment included 13 participants, and they found a statistically significant correlation between perceived distress due to social exclusion and activity in the brain area associated with physical pain. However, the magnitude of the estimated correlation ($r = .88$) is beyond what could be considered plausible. In this field correlations are likely to be around $\rho = .25$ [for a complete discussion see @bertoldoDesigningStudiesEvaluating2020].

### Retrospective design analysis

The function `retrospective()` can be used to evaluate the inferential risks associated with the study.

```{r, retro}
set.seed(2020) # set seed to make results reproducible

retrospective(effect_size = .25, sample_n1 = 13, test_method = "pearson")
```

In the output, we have the summary information about the hypothesized population effect, the study characteristics, and the inferential risks. We obtained a statistical power of almost 13% that is associated with a Type M error of around 2.6 and a Type S error of 0.03. That means, statistical significant results are on average an overestimation of 160% of the hypothesized population effect and there is a 3% probability of obtaining a statistically significant result in the opposite direction.

To know more about function arguments and examples see the function documentation `?retrospective` and  [`vignette("retrospective")`](retrospective.html).

### Prospective design analysis

Considering the previous results, researchers might consider planning a replication study to obtain more reliable results. The function `prospective()` can be used to compute the sample size needed to obtain a required level of power (e.g., power = 80%).

```{r, pro}
prospective(effect_size = .25, power = .8, test_method = "pearson", 
            display_message = FALSE)
```

In the output, we have again the summary information about the hypothesized population effect, the study characteristics, and the inferential risks. To obtain a power of around 80% the required sample size is $n = 126$, the associated Type M error is around 1.10 and the Type S error is approximately 0.

To know more about function arguments and examples see the function documentation `?prospective` and [`vignette("prospective")`](prospective.html).

## References



