Keywords
null hypothesis testing - bayesian analysis - Bayes factor - frequentist - statistical analysis - p-value - dichotomization - bayesian hypothesis testing - null hypothesis significance testing - JASP
Palabras Clave
prueba de hipótesis nula - análisis bayesiano - factor de Bayes - frecuentista - análisis estadístico - valor p - dicotomización - prueba de hipótesis bayesiana - prueba de significación de hipótesis nula - JASP
Introduction
Many scientific publications base their quantitative analyses on null hypothesis significance testing and p-values, a theoretical framework developed almost a century ago by Ronald Fisher,[1] Jerzy Neyman, and Egon Pearson.[2] However, an increasing number of authors have pointed out the limitations of this so-called ‘frequentist’ approach, mainly related to the low reproducibility of the studies.[3]
[4]
[5]
[6]
In 2018, three major urological magazines, the Journal of Urology, the British Journal of Urology, and European Urology, published the ‘Guidelines for reporting of statistics for clinical research in urology’, which state that we should not continue describing the studies as ‘positive’ or ‘negative’ according to p-values.[7]
[8]
[9] They encourage all researchers to follow the American Statistical Association statement on p-values, published in 2016,[10] which declares that the probability that a hypothesis is true does not depend on the p-value, and scientific conclusions should not be based on specific p-value thresholds. The referred document proposes alternatives like Bayesian estimations and methods that emphasize estimation over testing, among others.[10]
Despite the mentioned publications, problems related to the frequentist and Bayesian hypothesis test methods remain difficult to understand for many physicians who are not familiar with statistics and mathematics.[11] In light of this, the present narrative review offers an intuitive explanation of the subject. Throughout the text, the concepts are exemplified by comparing the mean International Prostate Symptom Score (IPSS) between two groups.
Frequentist Hypothesis Testing
Frequentist Hypothesis Testing
Sampling Distribution and Sampling Error
Sampling distribution and sampling error are essential concepts to understand the frequentist method. The sampling distribution is the hypothetical distribution of data from samples obtained from a population.[12]
[13]
[14]
[15]
[16] Suppose a hypothesis testing to compare the mean IPSS between a group of patients who take medicine A and a group who take medicine B. The population is the entire group of people about whom we want to carry out the study, and is defined by the inclusion and exclusion criteria. Say we draw different samples over and over from the population to compare the effect of medicine A and medicine B. We will obtain a value (a difference between means, dm) from each one of these samples. The sampling distribution is the curve obtained from these values ([Figure 1]).
Fig. 1 Sampling distribution. The circle on the left shows the population (P) or group of people suitable to take medicine A or B, according to inclusion and exclusion criteria. In the center, each small circle corresponds to a sample obtained from the population. Each letter corresponds to individuals who take medicine A or medicine B. We obtain a difference between means (dm) from each sample, which is the difference between the mean IPSS with medicine A and the mean International Prostate Symptom Score (IPSS) with medicine B. On the right, the results are grouped in a curve, which is the sampling distribution curve. In the example, the curve is normal, but in other scenarios it might have a different configuration.
The frequentist method is based on the assumption that there are no differences between the groups in the population. Therefore, in the case of a dm, the frequentist considers that the population parameter is zero (the null hypothesis, H0, is true). If that is the case, and we draw different hypothetical samples from a population (in which H0 is true), we would expect in each one of them a difference between groups equal to zero. However, most results will be different from zero due to the so-called sampling error: the inaccuracy of working with a sample and not with the entire population.[17]
[18]
[19] We can build a hypothetical sample distribution curve despite the sampling error thanks to statistical methods and mathematical laws. The ‘law’ for a dm says that the sampling distribution curve will be Gaussian, the mean will be zero, and most samples will fall around zero. These features are part of the Central Limit Theorem, described more than 200 years ago, and applied when sample sizes are larger than 60[16]
[20] ([Figure 2] depicts an example).
Fig. 2 The Central Limit Theorem. The circle on the left shows the population (P) of patients suitable to take medicine A or medicine B. Hypothesis testing assumes that in this population there are no differences between the mean IPSS with medication A and the mean IPSS with medication B (that is, the dm is equal to zero: H0 is true). In the center, we draw many samples with at least 60 individuals (n ≥ 60) from the population P. We calculate in each sample a dm between A and B, which is not necessarily zero, due to the sampling error. On the right, all the differences (dm) obtained are grouped in a sampling distribution curve, which will have a normal distribution. The mean of the curve is equal to the difference in means in the population P, that is, zero.
Obtaining and Understanding the p-value
Unlike the examples in [figure 1] and [2], researchers do not work with the entire population or with many different samples, but with a single sample. Once the result is obtained, it is placed into the hypothetical sampling distribution curve thanks to statistical methods such as the Student t-test. That is possible because we can predict the shape of the sampling distribution curve, as discussed before. Then, we calculate the percentage of hypothetical samples with an equal or higher value under the sampling distribution curve: this percentage is the p-value. The p-value is the probability of finding the same result or higher within the hypothetical sampling distribution curve, assuming that the sample comes from a population with no differences between the two groups, that is, assuming that H0 is true[11]
[12]
[13] ([Figure 3]). The p-value is a conditional probability because its calculation depends on the assumption (condition) that H0 is correct and does not indicate the probability that H0 is true, as many erroneously think.
Fig. 3 Calculation of the p-value, one tale test. The circle on the left shows the population P of patients suitable to take medicine A or B. We assume no differences between the two groups in the population (that is, the dm is zero: H0 is true). In the center, a sample is obtained, with a size ≥ 60. We obtain a dm from the sample. On the right, according to the Central Limit Theorem, it is known in advance that the hypothetical sample distribution follows a normal curve and has a mean equal to the population mean, which is zero. Thanks to these premises, the Student t statistical test enables the location of the study's result within the sampling distribution curve. Also, we calculate the percentage of hypothetical samples whose results are equal to or higher than the result obtained in the studied sample. The percentage mentioned is the p-value. In the example, the percentage was of 15%, that is, p = 0.15.
Suppose we conduct a single study to compare medication A and medication B in terms of the mean IPSS, and the difference obtained is 5 points. Then, we apply the Student t-test to place the outcome in the sampling distribution curve. Say the result is at one end of the sampling distribution curve, and from that point are located 15% of the possible hypothetical samples: the p-value, in this case, is 0.15. In other words, if we assume that there are no differences between the two groups in the population, and we perform the study repeatedly, 15% of the studies will show a difference of 5 points or more ([Figure 3]).
A p-value of 0.05 means that the sample is at an extreme of the sampling distribution because only 5% of the theoretical studies carried out in the population will show a similar result or higher. Traditionally, 0.05 is the threshold to suspect that the H0 is less likely to be the case, and there are actual differences between the two groups in the population, namely, the population parameter is not the null value (the alternative hypothesis, H1, is true).[13] The 0.05 threshold is called the level of statistical significance.[11]
[12]
[13]
[21]
The p-value is usually reported along with a 95% confidence interval (CI). As aforementioned, the sample estimate might be different from the population parameter due to the sampling error. The 95% CI expresses the inaccuracy caused by the sampling error and is a range in which the population parameter might lie.[22] If we draw samples over and over from the population, 95% of them will have a CI that contains the population parameter.[22]
[23] For example, if the population parameter is a dm = 3 (H1 true), and we obtain an infinite number of samples from the population, for 95% of these samples the 95% CI will contain the value 3. It is wrong to state that there is a probability of 0.95 that the 95% CI contains the population parameter. The probability of 0.95 refers to the proportion of infinite samples whose CI contains the population parameter and does not apply to a single CI.[22]
The p-value dictates how rare it is to obtain a value equal to or higher than the outcome of a single study if we assume no real differences between the groups compared in the population. Unfortunately, the p-value does not answer the fundamental question: what is the probability of a difference in the population between the compared groups? The frequentist method focuses on the wrong question: given a population with no differences between the groups, what is the probability of a result equal or more extreme than the one obtained? The real question is: given the data, what is the probability of a real difference between groups in the population? (What is the probability of H1?) As we discuss now, the Bayesian method offers the answer.
Bayesian Hypothesis Testing
Bayesian Hypothesis Testing
Bayesian statistics calculate the probability that an event is true considering the results obtained (the data) and the knowledge that one has of this probability before developing a study (the prior probability). It is based on the Bayes theorem, formulated by Thomas Bayes, an 18th-century English statistician.
The Bayesian framework deals with conditional probability, which is the probability of an event given that another event has occurred.[24]
[25] Conditional probability is expressed as p(A | B), and means ‘the probability of A, given that B occurs’. Bayes' theorem enables us to find a conditional probability when we are provided with its inverse: if we know p(A | B), we can calculate p(B | A).[25] That is the case of hypothesis testing: we seek the probability of a hypothesis given the data p(H | data), and our data (the result of our study) enables us to retrieve the probability of the data, given the hypothesis p(data | H). In the example mentioned before of medicine A versus medicine B, we want to know if there is in the population a difference in terms of the IPSS given the outcome of a study [p(H1 | data)], and the study provides the probability of the outcome obtained if we assume that H1 is true [p(data | H1)].
Bayes formula for two rival hypothesis (H1 and H0) is expressed in the form of odds:[26]
p ( H1 ∣ data ) p ( H1) p ( data ∣ H1 )
-------------------------------- = -------------------------- X ------------------------
p ( H0 ∣ data ) p ( H0 ) p ( data ∣ H0 )
posterior odds for H1vsH0 prior odds for H1vsH0 Bayes Factor
In words, the posterior odds of H1 versus H0 given the data is the product of the prior odds for H1 versus H0 times the Bayes Factor (Bf). We can turn odds into probability (p) by applying the formula p = odds/1 + odds.[27] The prior odds are what we believe before the study about the probability of H1 versus H0. The Bf is equal to the probability of the data, given H1, divided by the probability of the data, given H0, and represents the strength of evidence that the data provide for H1 versus H0. For example, if we think a priori that the rival hypotheses have the same probability [p(H1)/p(h0) = 0.5/0.5 = 1], and the Bf is 20 (that is, the observed data is 20 times more likely under H1 than under H0), the posterior odds are 1 × 20 = 20. Then, we turn odds into probability: p = 20/20 + 1 = 0.95, and we conclude that the data has increased the probability of H1 from 0.5 to 0.95. Note that, unlike the frequentist method, this time we have answered the main question: what is the probability of H1?
Prior Odds
Prior odds are researcher beliefs about how plausible are the hypotheses.[27]
[28]
[29]
[30] It is background knowledge that is relevant when we make any conditional prediction, not only in hypothesis testing. For instance, suppose two 65-year-old male patients with a prostate-specific antigen (PSA) level of 5 ng/ml. They live in the same geographical area. We want to know the risk of prostate cancer given the PSA. The first man is white, does not have a family history of prostate cancer, and his prostate volume is of 80 mL. The second man is black, has a family history of prostate cancer, and his prostate volume is of 20 mL. Despite their similar PSA level (the data), the black man has a higher probability of developing prostate cancer because of his higher pretest odds.
Prior odds are a subjective way to quantify uncertainty and depend on the result of previous studies in the field, theoretical considerations, biological plausibility, and basic physiological knowledge.[28]
[29]
[31]
[32] When there is skepticism towards H1, the researcher assigns low prior odds that H1 is true. When it is more plausible, the prior odds will be high. A study that compares a new selective alpha-blocker and placebo to treat lower urinary tract symptoms secondary to prostatic enlargement, for example, will have high prior odds in favor of H1 because the benefit of alpha-blockers in this scenario has been previously proven.
Bayes Factor
The Bf is a comparison of how well two hypotheses predict the data. It is a measure of how likely the data is in a hypothesis compared to another hypothesis.[28]
[32] Therefore, the Bf is not a probability but a ratio of probabilities.[33] For instance, if the Bf = 10 when we compare H1 versus H0, data are 10 times more likely to have occurred under H1 than under H0.
The English statistician Sir Harold Jeffreys proposed a classification of the evidence for a hypothesis in terms of specific Bf intervals[28] ([Table 1]). According to the scheme, the Bf describes ‘anecdotal’, ‘moderate’, ‘strong’, ‘very strong’, or ‘extreme’ relative evidence for a hypothesis. This set of rigid labels facilitates scientific communication, but there is no specific threshold to prefer H1 and reject H0.[28]
[30]
[34] The Bf does not yield a dichotomous decision (reject or not reject H0) but a relative comparison of the hypothesis. Its value must be interpreted along with the prior probability to predict how likely is H1. Most Bayesian studies describe the Bf but not the prior odds nor the posterior odds to enable the reader to make their own conclusions according to their beliefs.
Table 1
Evidence categories for the Bayes Factor in favor of the alternative hypothesis (H1) over the null hypothesis (H0)[28]
|
Bayes Factor
|
Interpretation
|
|
> 100
|
Extreme evidence for H1
|
|
30–100
|
Very strong evidence for H1
|
|
10–30
|
Strong evidence for H1
|
|
3–10
|
Moderate evidence for H1
|
|
1–3
|
Anecdotal evidence for H1
|
|
1
|
No evidence
|
|
0.33–1
|
Anecdotal evidence for H0
|
|
0.33–0.1
|
Moderate evidence for H0
|
|
0.1–0.03
|
Strong evidence for H0
|
|
0.03–0.01
|
Very strong evidence for H0
|
|
< 0.01
|
Extreme evidence for H0
|
The Bf depends on three elements: data likelihood, the prior distribution, and the H0 specification.
Data likelihood. Likelihood is a term used to describe the probability of observing data that have already been collected.[25]
[27] Suppose a study comparing the mean IPPS between an alpha-blocker and placebo with a final result of 10 points. The parameter, in this case, is the dm. Data likelihood estimates the probability of our data given the infinite possible values of the parameter. If we consider dm = 10 points, the probability of the study's outcome (our data, 10 points) will be high. If we consider dm = 5 points, the probability of the outcome (our data, 10 points) will be low. We make the same calculation for every possible value of the parameter to draw a likelihood profile curve (also called likelihood function curve).[23]
[27]
[35] The curve says how likely is the data for every value of the parameter. [Figure 4] illustrates the example.
Fig. 4 Likelihood profile curve for a dm. The x-axis is the unknown parameter dm. We calculate how likely are different values of dm given an observed dm = 10. The parameter value dm = 10 has the highest likelihood. The parameter value dm = 5 has a low likelihood.
The prior distribution. A primary characteristic of the Bayesian method is the uncertainty about the value of the population parameter, unlike the frequentist analysis, in which the parameters are considered fixed.[36] When we calculate the sample size for a frequentist comparison between means, we assume that the dm is exactly 0 for H0 and has an exact prespecified value for H1. The Bayesian comparison between means, instead, assumes uncertainty about those values. In the case of the H1 parameter, we assign a range of possible values. The distribution of those values is called the prior distribution.[23]
[28]
[37] Common sorts of prior distribution are normal distribution and t-distribution ([Figure 5]).
Fig. 5 Examples of prior distributions. The x-axis corresponds to the parameter values. The y-axis, the density, refers to how frequent is each parameter value.
A common mistake is considering the Bayesian parameters ‘random’, in opposition to the fixed point by the frequentist.[36] The parameter in the Bayesian model is not random but uncertain. Uncertainty in the Bayesian method is usually misinterpreted as ‘randomness’.[36]
There are two different ways to choose the prior distribution in Bayesian analysis.[27]
[28]
[38]
[39] The first is the subjective way, which is based on the researcher's beliefs about the parameters. These beliefs are supported by previous theoretical information, and the prior distribution is therefore called ‘informative’. The informative distribution describes a specific range for the parameter with the more possible values. The second is the objective way, not based on the researcher's beliefs, which yield a ‘non-informative’ distribution. It is called objective because there is no bias toward any specific value: the distribution displays a wide range of values of the parameter because all of them are equally likely, or almost equally likely, from the researcher's point of view. For instance, if we plan to compare the difference in the mean IPPS between a new alpha-blocker and placebo, and a previous meta-analysis describes a difference of 10 points, we can set an ‘informative’ prior distribution centered on 10 and with a narrow standard deviation. If there is no previous information about the topic, we would have to set a ‘non-informative’ prior with a wide range of values for the difference in means ([Figure 5]).
Different authors advocate non-informative priors, especially for scenarios in which there is no previous reliable information about the research topic.[28]
[40]
[41] In that sense, ‘default’ priors are recommended. ‘Default’ prior non-informative distributions are prior distributions with limited preference for particular parameter values. They are recommended because they increase the objectivity of the analyses and facilitate communication, because they provide a standard reference to compare Bfs from different studies.[42]
[43] The flat prior (uniform prior, with no biases towards any parameter value) is a sort of non-informative distribution not recommended because it leads to Bfs in favor of the H0 even if the data suggest a difference in favor of the H1.[23]
[40]
[44] Therefore, non-informative priors adopted by researchers span a wide range of parameter values but are not entirely flat. The Cauchy distribution is the default prior for the t-test, and it is similar to the Gaussian normal distribution but has fatter tails and less central mass[28]
[29]
[38]
[40]
[42] ([Figure 5]).
It is important to emphasize two concepts. First, the prior distribution must be specified before (not after) the data collection.[23]
[28]
[29]
[39]
[40] Once the researcher knows their study's data, the prior distribution's construction is prone to bias.[29]
[39] However, redefining the prior distribution is allowed during data obtention if additional information becomes available.39 Second, the prior odds and the prior distribution are not the same.[28]
[41] The prior odds are the researcher's prior beliefs about the probability of H1 in relation to H0. The prior distribution is a range of possible values of the parameter in H1, assigned by the researcher.
H0 specification. In Bayesian analysis, the H0 is usually described as a single fixed point equal to zero (no differences between the groups).[28]
[29]
[37]
[45] Conceptually, however, the Bayesian method acknowledges that considering H0, the population parameter is never exactly zero, but a value near zero; therefore, the specification of the H0 can be an interval around 0.[29]
[37]
[46]
The Posterior Distribution
The posterior distribution is the distribution of possible values of the H1 parameter obtained after updating the prior distribution based on the data.[23]
[27]
[28]
[37]
[38]
[41]
[42] It combines our beliefs before knowing the data (the prior distribution) and the information provided by the data (the likelihood).[28] In many cases, the information provided by the data can reduce the uncertainty of the H1 parameter.
The posterior distribution is calculated by multiplying the data likelihood by the prior distribution[23]
[28]
[37]
[42] ([Figure 6]). The range obtained is described by central tendency measures such as mean, median, and mode.[28] The dispersion around the central tendency measurement is described with an interval that includes 95% of the parameter values.[28] The mentioned interval is called the 95% credible interval and means that we are 95% certain that the actual parameter value falls in the interval assuming that the alternative hypothesis
is true (H1).[41]
[42] Recall that the probability of a true H1 is not expressed by the credible interval nor the posterior distribution. As aforementioned, the probability H1 versus H0 is given by the posterior odds.
Fig. 6 Calculation of the posterior distribution. The posterior distribution (blue line) is obtained by the multiplication of the prior (dotted line) and the data likelihood (yellow line). CI = credible interval.
Calculation of the Bayes Factor
As aforementioned, the Bf is the probability of the data, given H1, divided by the probability of the data, given H0:
Bf = p(data | H1)/p(data | H0).
The probability of the data given H0 refers to how likely are the data given the parameter value of no differences (the point null), that is to say, the height of the likelihood profile curve for the point null.[28]
[44]
[45]
The probability of the data given H1 is a more complex calculation because H1 prior distribution is not a point null, but a range of values.[27]
[28]
[29]
[44] In that sense, we need to integrate the data likelihood to H1 prior distribution, which is, in simple terms, multiplying (averaging) the prior distribution by the data likelihood for each parameter value and adding all the obtained values.[28]
[47] The outcome is called H1 marginal likelihood.
The Bf calculation based on the H1 marginal likelihood and point-null likelihood can be graphically illustrated (August 9, 2015 posting by Alexander Etz to ‘Understanding Bayes: Visualization of the Bayes Factor’ blog; unreferenced, retrieved from https://alexanderetz.com/2015/08/09/understanding-bayes-visualization-of-bf). The parameter's continuous distribution is interpreted as a set of many points spaced very close together. We calculate the likelihood ratio for every point and multiply the likelihood ratio by the respective H1 prior density. Then, we do the sum of all calculations, and finally, we divide by the total number of points ([Figure 7]).
Fig. 7 Calculation of the Bayes Factor (Bf). a) The x-axis is the parameter value (dm). The yellow curve is the likelihood profile curve, and the dashed curve is the prior distribution. b) The parameter's continuous distribution can be thought of as a set of many point hypotheses spaced very close together. c) We determine the likelihood ratio (LR) for the different dm point values relative to the point null (H0), and then we weight the LR on the prior distribution. The LR is the ratio of the heights of the likelihood curve (yellow curve). For instance, we calculate the LR for dm = −10 relative to H0. The height is 0.1 for dm = −10 and 0.5 for the point null, so LR = 0.1/0.5 = 0.2. The LR is multiplied by the density assigned to dm = −10 in the prior distribution to weight the LR based on the prior distribution. In the example, the density in the prior distribution for dm = −10 is 0.8, so the weighted LR = 0.2 * 0.8 = 0.16. A similar calculation must be done for every single dm point hypotheses values of the x-axis. The Bf is obtained by adding all the weighted LR and dividing by the total number of points.
An alternative way to obtain the Bf is by dividing the posterior distribution height by the prior distribution height at the null point, with no calculation of the H1 marginal likelihood[28]
[45] ([Figure 8]). The method corresponds to the Savage–Dickey density ratio, and is suitable for nested models.[28]
[45]
[46]
[47]
[48] In hypothesis testing, we say that we have nested models when we can obtain H0 by constraining the parameters of H1;[48] in other words, when H0 is a subset of H1. That is the case for most Bayesian hypothesis testing because the ‘point null’ of H0 can be obtained from H1 by setting the parameter equal to 0.[28]
Fig. 8 Calculation of the Bf, the Savage–Dickey method. The Bf is the ratio of the heights of the orange dots, which correspond to the height at the null point of the prior and posterior distributions.
Bayesian hypothesis testing is also feasible for complex models with many parameters or non-precise prior distributions.[28] If that is the case, we can apply computational methods like the Markov chain Monte Carlo (MCMC) to obtain the posterior distributions over the parameters.[37]
[48]
[49]
Sensitivity Analysis
The main challenge of Bayesian analysis is its dependence on the prior distribution. As aforementioned, the Bf calculation takes into consideration the H1 prior distribution, and the latter is in some sense arbitrary. One way to prevent a misleading Bf is to collect appropriate knowledge to set the best informative prior, but that is not always possible.[28] A second alternative is the so-called sensitive analysis, in which we check how the Bf is affected by changes in the width of the prior distribution.[28]
[30]
[36] If the Bf does not fall below certain limits despite different prior widths, we can conclude that we have a reliable and trustworthy assessment. If we obtain a Bf of 40 in our study, meaning strong evidence in favor H1, for example, and then different width values do not yield a Bf below 10, we are confident about the robustness of our research outcome.
Stop Rule and Sample Size in Bayesian Hypothesis Testing
The frequentist analysis specifies the sample size, and the study cannot be finished until the planned number of participants has been included. Bayesian methodology, instead, does not specify a sample size, and the Bf can be monitored as the data come in.[29]
[41]
[50] Bayesian researchers are allowed to stop the study whenever they want, especially if the evidence is compelling.[50] For example, some authors plan to stop the research as soon as Bf ≥ 10. Other ways to establish limits are deadlines for the recruitment process or fixing a maximum number of participants per group (a ‘maximum sample size’, N).[29]
[41]
Example with JASP
JASP is a free statistical software program developed by researchers from the University of Amsterdam. Its name is an acronym for Jeffreys Awesome Statistics Package, referring to Sir Harold Jeffreys, who played a central role in developing the Bayesian methodology. JASP can be easily downloaded at https://jasp-stats.org/download/, and it includes both frequentist and Bayesian tests. Different tutorials are available online, and a guideline has been published recently.[34]
[40]
[41]
Let us suppose a hypothetical new pharmacological treatment for lower urinary tract symptoms, different from the traditional alpha-blockers. Let us call this new product the ‘x-blocker’. There is biological plausibility from a physiopathological perspective, but we do not have previous research regarding the product. Suppose we conduct a clinical trial comparing the ‘x-blocker’ and placebo in benign prostatic hyperplasia (BPH) patients in terms of the IPSS after four weeks of treatment. These are the data, following a normal distribution, including 60 patients per group:
‘X-blocker’ group, IPSS, 60 patients: 12,8,13,7,13,12,14,8,11,10,5, 10,11,10,9,10,5,10,16,10,10,4,3,10,11,17,11,8,11,9,9,12,12,14,12,12,8,8,13,13,9,13,7,7,9,8,7,9,7,14,14,6,6,11,6,6,15,15,11,9. The mean is 10.0, and the standard deviation is 3.0.
Placebo group, IPSS, 60 patients : 13,10,6,12,11,12,14,9,11,12,13,4,17,3,12,8,12,9,3,12,16,11,12,10,12,12,11,14,5,4,19,11,13,15,15,11,8,7,14,2,11,13,13,10,10,13,16,16,17,18,13,11,12,9,12,13,15,16,7,13. The mean is 11.3, and the standard deviation is 3.7.
We can set the hypothesis in terms of the Cohen d (δ), which is a way to express the effect size when we compare means.[40]
[41] Cohen d values indicate the difference between the two groups in standard deviations. Values around 0.2 and 0.5 are considered small and medium effects respectively. Vales ≥ 0.8 are considered a large effect. For example, Cohen d = 1.2 is a large effect and says that the groups compared differ by 1.2 standard deviations. Cohen d = 0 means no difference between the two groups. Therefore, H0 assumes δ = 0, and H1 assumes δ ≠ 0.
To obtain the frequentist comparison, we first load the data into JASP. Then, we select ‘T-test’ and click on ‘Classical independent samples T-test’. In the options panel, we click on ‘Student’ on the test option. Finally, we select our alternative hypothesis option: if we do not have a directional prediction about H0 versus H1, we choose ‘Group 1≠Group 2’. [Table 2] presents the results.
Table 2
Frequentist outcome of a hypothetical comparison of the IPSS between two groups. Independent samples t-test (Student t-test)
|
t
|
p
|
Cohen d
|
95% onfidence interval for the Cohen d
|
|
Lower
|
Upper
|
|
International Prostate Symptom Score (IPSS)
|
2.21
|
0.029
|
0.404
|
0.042
|
0.765
|
Note: The analysis was performed with the JASP software.
The p-value = 0.02 describes how rare is the result in a population with no differences between the groups: only 2% of hypothetical samples will have a similar or more extreme outcome. Since the data are unlikely given H0 and the p-value is lower than 0.05, we reject the H0. Cohen d = 0.4 equates to a small/medium effect. The 95% CI says that we are confident that the population's Cohen d value will fall within the interval 0.04-0.75 because that will be the case in 95% of samples if we redo the study many times. As aforementioned, the probability of H1 (the probability of a difference between the groups in the population) is not specified by the frequentist approach.
To perform the JASP Bayesian analysis, we go to ‘T-test’ and then to ‘Bayesian independent samples T-test’. In the options panel, we choose the alternative hypothesis option, ‘Group 1≠Group 2’. In the ‘Bayes Factor’ option, we choose ‘BF10’ to obtain a Bf in favor of H1 over H0 (the program offers the option ‘BF01’ if we prefer a Bf describing the evidence in favor of H0 over H1). In the ‘Prior’ option, we select the prior H1 distribution: since our example establishes a lack of solid background evidence about the ‘x-blocker’, we choose the default ‘Cauchy’ distribution. The default Cauchy is centered on zero with an interquartile range r = 0.70, meaning we are 50% confident that the actual effect size lies between Cohen d = -0.7 and Cohen d = 0.7. In the ‘Plots’ option, we click on ‘Prior and posterior’ and ‘Bayes factor robustness check’, and we obtain [figures 9] and [10].
Fig. 9 Bayesian outcome of a hypothetical comparison of the mean IPSS between two groups. The analysis was performed with the JASP software.
Fig. 10 Sensitivity analysis or Bf robustness plot of a hypothetical comparison of the mean IPSS between two groups. The analysis was performed with the JASP software.
[Figure 9] features the main results of the Bayesian analysis. The BF10 is 1.73, meaning H1 predicts the data 1.73 times better than H0. The circle in red and white is called ‘the probability wheel’, and is a graphical representation of the Bf. The more evidence in favor of H1 (the higher the Bf, for example), the greater the red/white radius. The dashed line curve represents the Cauchy prior distribution, and the solid line curve represents the posterior distribution. Both curves present densities for different Cohen d values. The grey dots represent the height of the curves at the point null of no effect, and their values are used to calculate de Bf with the Savage–Dickey method, as aforementioned. The 95% CI = 0.02-0.7 refers to the 95% credibility interval, meaning a 95% probability that the population's Cohen d value falls in the interval, with a median of 0.36, as long as the H1 is true.
From the data of our example, we might predict the probability of H1. Suppose the researcher thinks the prior odds of the H1 versus H0 are 60/40. Given the obtained Bf = 1.73, the posterior odds of H1versus H0 are 60/40 × 1.73 = 2.59. We turn the posterior odds into probability (p) applying the formula p = odds/1 + odds = 2.59/1 + 2.59 = 0.72. The probability of a true H1 in our example is of 72%.
[Figure 10] displays a sensitivity analysis or Bayes factor robustness check. As seen, Bf lies in the interval 1-3 despite different Cauchy prior widths, meaning the Bf provides anecdotal evidence for the null hypothesis relative to the alternative hypothesis irrespective of the prior widths. The Bf decreases as the width is wider, indicating data with low robustness.
We can establish a parallel between the frequentist and the Bayesian method based on the example. The frequentist yields a dichotomous outcome accepting or rejecting H0: we rejected the null because the p-value was lower than 0.05. The probability of a real H1 cannot be determined. The Bayesian, instead, accepts uncertainty about H1 versus H0, and its prediction is continuous scales: the BF10 was 1.73, indicating anecdotal evidence in favor of H1, and the posterior probability of H1 was of 72%. The frequentist 95% CI says that 95% of repeated samples will have a CI that contains the population parameter, but we do not know if the obtained 95% CI (Cohen d: 0.04-0.76) corresponds to one of those samples. The obtained Bayesian 95% credibility interval, in turn, predicts that, in the case of a true H1, the population parameter lies between Cohen d 0.02-0.7.
Conclusion
The present manuscript synthesizes the main concepts of the frequentist and the Bayesian methodology. The frequentist approach has been used for many decades and is still the preferred method to analyze and report results despite its limitations related to the simplistic acceptance or rejection of the null hypothesis depending on the 0.05 p threshold, with no consideration of the probability of H1. Bayesian inference provides a compelling continuous measure of how likely is H1 in terms of the Bf and the posterior odds, and researchers should be encouraged to become familiar with this technique. Hopefully, the implementation of friendly tools like JASP will enable us to report Bayesian results more frequently, alone or in combination whit the frequentist conclusions.
