Introduction to statistics

Regression analysis is a statistical method used to model and explain the relationship between variables, and to make predictions about one variable based on one or more others. Different types of regression are required depending on the type of variables involved, and this page outlines two of the most common types.

Linear regression

Linear regression is used to model the relationship between one or more independent variables, which can be continuous or categorical, and a continuous dependent variable. It builds on the idea of correlation because it also describes a linear relationship between variables, but it allows this relationship to be expressed as an equation that can be used to make predictions. Furthermore, while simple linear regression involves only one independent variable (as with correlation), most of the time there are actually multiple factors that may influence the outcome - and these can all be accounted for in multiple linear regression.

For example, multiple linear regression can be used to examine how factors such as heart rate before exercise, age and gender relate to heart rate after exercise. It provides information about how well these variables explain heart rate after exercise overall, as well as the effect of each variable individually while accounting for the other variables in the model.

Before conducting linear regression, you need to check that the following assumptions are met:

In addition, it is important to have a sufficiently large sample size relative to the number of independent variables included in the regression model. A commonly used rule of thumb is to have around \(10–15\) cases for each independent variable.

If the assumptions for linear regression are met and the analysis is conducted using statistical software, such as SPSS in this example, the results should include the following statistics:

The first output table is the Model Summary, which includes \(R^2\) and adjusted \(R^2\) values. Both provide an indication of how well the model explains variation in the dependent variable. However, adjusted \(R^2\) takes into account the number of independent variables in the model and provides a more accurate estimate of how well the model would perform in the population, so it is typically the value used. In this case, the adjusted \(R^2\) value of \(.131\) indicates that \(13.1\%\) of the variation in heart rate after exercise can be explained by the independent variables in the model (heart rate before exercise, age and gender).

The second output table is the ANOVA table, which includes the overall \(p\) value for the model. This is used to determine whether the model as a whole is statistically significant. In this case, the \(p\) value of \(.045\) indicates that the model is statistically significant at the \(.05\) level.

The final output table is the Coefficients table, which shows the effect of each independent variable on the dependent variable while accounting for the other variables in the model. It includes both unstandardised coefficients (\(B\)) and standardised coefficients (\(\beta\)), along with their associated \(p\) values. While both types of coefficient provide information about the relationship between the independent and dependent variables, the unstandardised coefficients are typically used for interpretation. They indicate the expected change in the dependent variable for a one-unit increase in the independent variable, while accounting for the other variables in the model.

In this example, heart rate before exercise has a positive coefficient (\(B = 0.76\)) and is statistically significant (\(p = .020\)). This indicates that, after accounting for age and gender, heart rate after exercise is expected to increase by \(0.76\) beats per minute for each additional beat per minute in heart rate before exercise.

Age also has a positive coefficient (\(B = 0.22\)) but is not statistically significant (\(p = .346\)). This indicates that, after accounting for the other variables, heart rate after exercise is expected to increase by \(0.22\) beats per minute for each additional year of age, although there is insufficient evidence that this relationship exists in the population.

Gender also has a positive coefficient (\(B = 7.24\)) but is not statistically significant (\(p = .066\)). Given that gender is coded as \(1\) for male and \(2\) for female, this indicates that, after accounting for the other variables, heart rate after exercise is expected to be \(7.24\) beats per minute higher for females than males. However, there is insufficient evidence that this difference exists in the population.

Overall, while the model is statistically significant, heart rate before exercise is the only independent variable with a statistically significant relationship with heart rate after exercise after accounting for age and gender.

Logistic regression

Logistic regression is used to model the relationship between one or more independent variables, which can be continuous or categorical, and a categorical dependent variable. Most commonly, the dependent variable has two categories, such as yes/no or pass/fail, although logistic regression can also be extended to outcomes with more than two categories. Like linear regression, logistic regression allows the effects of multiple independent variables on an outcome to be examined while accounting for the other variables in the model. However, rather than modelling a linear relationship, it models the probability of an outcome occurring.

For example, logistic regression can be used to examine how factors such as study time, attendance and gender relate to whether a student passes or fails a unit. It provides information about how well these variables explain pass status overall, as well as the effect of each variable individually while accounting for the other variables in the model.

Before conducting logistic regression, you need to check that the following assumptions are met:

In addition, it is important to have a sufficiently large sample size relative to the number of independent variables included in the regression model. A commonly used rule of thumb for logistic regression is to have at least \(10\) outcome events for each independent variable.

If the assumptions for logistic regression are met and the analysis is conducted using statistical software, such as SPSS in this example, the results should include the following statistics:

The first output table is the Omnibus Tests of Model Coefficients table, which includes the overall \(p\) value for the model in the ‘Model’ row. This is used to determine whether the model as a whole is statistically significant. In this case, the \(p\) value is less than \(.001\), indicating that the model is statistically significant at the \(.05\) level.

The second output table is the Model Summary table, which includes pseudo \(R^2\) values (Cox & Snell and Nagelkerke \(R^2\)). Both provide an indication of how well the model accounts for the outcome, however, Nagelkerke \(R^2\) is scaled to range from \(0\) to \(1\) which makes it easier to interpret. In this case, the Nagelkerke \(R^2\) value of \(.61\) indicates that the model accounts for a substantial amount of the variation in pass status, although some variation remains unexplained.

The final output table is the Variables in the Equation table, which shows the effect of each independent variable on the dependent variable while accounting for the other variables in the model. It includes the regression coefficients (\(B\)), their associated \(p\) values, and odds ratios (\(Exp(B)\)). While the regression coefficients are used to construct the model, the odds ratios are typically easier to interpret because they show how the odds of the outcome change for a one-unit increase in an independent variable.

In this example, study time has an odds ratio of \(2.13\), but is not statistically significant (\(p=.661\)). This indicates that, after accounting for attendance and gender, each additional hour of study time is associated with approximately twice the odds of passing the unit. However, there is insufficient evidence that this relationship exists in the population.

Attendance has an odds ratio of \(1.07\), but is not statistically significant (\(p=.895\)). This indicates that, after accounting for the other variables, each additional percentage point in attendance is associated with a small increase in the odds of passing. However, there is insufficient evidence that this relationship exists in the population.

Gender has an odds ratio of \(2.60\), but is also not statistically significant (\(p=.351\)). Given that gender is coded as \(1\) for male and \(2\) for female, this indicates that, after accounting for the other variables, females have approximately \(2.6\) times the odds of passing compared with males. However, there is insufficient evidence that this difference exists in the population.

Overall, while the model is statistically significant, there is insufficient evidence that any of the individual variables are significantly associated with pass status once the other variables are taken into account.