What is Multicollinearity? Why Does it Matter? When can it be Ignored? What Models are Affected? How is it Diagnosed and Corrected?

Dr. Gordy Fairchild (CEO of Synergy Data Science)

Misinformation about multicollinearity is commonplace. In short, there is disagreement among practitioners about many facets of multicollinearity. Both new-school and old-school data scientists often handle it in different ways.

Inexperienced new-school practitioners who have been trained quickly in a data science boot camp typically do not have a well-developed understanding. Big Data-driven analyses often leave out any exploration of multicollinearity due to highly complex feature sets with vast quantities of real-time data. There is no time to fully evaluate multicollinearity with 1,000 features in an 18-billion row dataset, along with daily or continuous fine-tuning runs. Automation can help but there is an art and a science involved that can evade automation. Additionally, numerous methods and algorithms exist that can often forecast better or where multicollinearity does not apply.

Yet, old-school practitioners who model human behavior may labor in a world of “smaller” data and the need to follow the Scientific Method. They ask both ”Why” and “What” to yield business-centered results, individual variable importance, or tested hypotheses. In this world especially, dealing with multicollinearity is both an art and a science. It is a science because it is a phenomenon that is defined by equations and quantitative diagnostics. But an art since there are qualitative factors where the model does not “feel” right and there are fine touches about what to do (if anything) about the problem and when.

Multicollinearity exists when there is a strong linear relationship among two or more independent variables (features), making it difficult to separate their individual effects on the dependent variable (target).

Multicollinearity occurs only for certain modeling methods. Regression models (linear, logit, Cox, or probit) are most often thought of. Additionally, other models are subject to the problem: Naïve Bayes, KNN (difficulty with feature importance), decision tree regression (instability in tree-building process), time series models including ARIMA and VAR, Poisson models, SVM (Support Vector Machines), and certain neural network algorithms.

1. Causes of Multicollinearity

Multicollinearity (which may often be a data problem) also results from:

Variables that are nearly measuring the same thing.
Two or more variables may be based on a common time trend.
Poorly constructed derived variables.
Lagged variables with a common time trend.
Small or narrowly based samples.
Unexpected relationships among semantically different independent variables.
Data collection bias
Measurement error
Overfitting

It can also occur when:

Variables are powers, multiples, interaction effects, ratios, or certain dummy variable patterns from one underlying variable.
Structural arrangements of the data add another cause since drug control study data is in a different structure than vintage analysis in finance or marketing. Methods that do not take data structure into account can yield extremely multicollinear and incorrectly specified models that may use the wrong solution algorithm.

2. Consequences of Multicollinearity: Ignore or Diagnose

The existence of multicollinearity does NOT always mean multicollinear models are useless. Forecasts and predictions that result from multicollinear models can still be valid. If the goal of the model is to predict or forecast, then doing nothing may be a perfectly reasonable action. Note that this point is hotly debated. For example, AI sources (on Poe) state: “Multicollinearity should not be ignored in forecasting tasks. While multicollinearity primarily affects the interpretation and inference of regression models, it can still have implications for forecasting accuracy and reliability. Ignoring multicollinearity can lead to unreliable and misleading forecasts.” Yet old-school authors like Peter Kennedy argue “In general, multicollinearity does not severely affect forecasting so long as the collinearity characterizing the data to be forecast remains similar to the collinearity in the data used for estimation” (a Guide to Econometrics, 6th ed., p.338)

Also, it is to be expected that variables that are powers, multiples, interactions, or ratios of a given variable would yield some multicollinearity that can typically be ignored. Dummy variables with 3 or more categories can be another example of cases where doing nothing beyond dummy redefinition is warranted.

Outside of these special cases, multicollinearity can be serious because the value of individual coefficients prevents identification of the relevant magnitude of influence among the independent variables themselves and the degree of impact each variable has on the dependent variable. Additionally, the descriptive or explanatory power of independent variables is further weakened by artificially high variances. Hypothesis testing is systematically biased in such a way that you can’t exclude the possibility that every variable’s coefficient is insignificant (equal to zero via t-test). This can lead to the seemingly impossible conclusion that although the regression as a whole is significant via the joint F-test (at least 1 variable coefficient does not equal zero), yet each coefficient can be equal to zero via individual t-tests.

3. Symptoms and Diagnosis of Multicollinearity

The symptoms of multicollinearity include:

Small changes in data values can create large swings in the parameter estimates,
Coefficients can have magnitudes or signs that are bizarre,
The standard errors of the coefficients are high with low significance despite the fact they are jointly significant, and adjusted R² is artificially high. (Greene, Econometric Analysis, p. 94)

These (and other) symptoms are evaluated in the context of several diagnostic criteria:

Correlation Coefficient. There are various rules of thumb proposed about the magnitude of correlation coefficients that help diagnose multicollinearity. Values from 0.5-0.9 (and -.5 to -.9) are proposed as rules to define “high” correlation. NOTE: a high correlation coefficient does not automatically mean there is multicollinearity. Instead, other confirmatory tests are needed to properly diagnose the issue. Some practitioners throw out ALL variables with correlation coefficients greater than the rule of thumb. This technique is entirely unsupported by statistical evidence and is incorrect unless the correlation coefficient equals 1 (the case of perfect multicollinearity). An even more serious error is when data that are used for models not subject to multicollinearity are blindly used to pre-cleanse all datasets.
Per variable Variance Inflation Factors (VIFs). If the VIF equals 1 there is no multicollinearity for that variable. But VIFs over 5 signify a problem and VIFS over 10 (the number varies according to NS vs. OS) a serious problem according to one rule of thumb. Tolerance (reciprocal of VIF) is sometimes used instead.
Condition Indices. By performing high-level matrix algebra operations, the condition index is produced. There is one condition index per feature with values over 30 typically creating cause for concern according to rules-of-thumb.
Variance proportions when combined with condition indices can be used to isolate each variable’s contribution to variance to allow intelligent variable removals and feature engineering.

Such qualitative and quantitative diagnostic criteria may give clues about how to proceed with correcting or at least mitigating the problem, but they are subject to subjective rules of thumb.

4. Mitigating Multicollinearity

There are at least 7 ways multicollinearity can be handled.

Do nothing. Perhaps the multicollinearity is initially suspected through “high” correlation coefficients (>0.85 or <-0.85) but passes rules of thumb with VIFs and condition indices. Such non-severe multicollinearity can be safely ignored.
Eliminate some variables from modeling. Comparing VIFs or decomposing condition indices by variance proportion source, comparing model metrics, or continuous variable bucketing can help inform removals rather than dual variable removals or guessing.
Use dimensionality reduction techniques such as PCA (Principal Component Analysis) or Factor Analysis.
Use creative variable reduction techniques that will still display multicollinearity but lower the magnitude. Examples include products, interactions, or alternative transformations. A polynomial regression, marginality-based variables such as rates of changes, an alternative specification of lopsided dummy variables (extreme frequencies and many levels), or grouping values via one hot encoding can all help.
Get more data. Often multicollinearity is a data problem and not a theoretical or practical one. It is caused by a dataset that is too small, does not cover a broad range of variable values, or is due to flawed sampling methods.
Another approach uses mainly newer algorithms that can handle the situation well. Regularization algorithms, including Ridge Regression, Lasso Regression, Elastic Net Regression, Linear Regression with Polynomic features, and Random Forest approaches are applied to construct methods that either mitigate or eliminate multicollinearity.
Make sure any underlying data structures are properly accounted for when engineering new features. Cross-sectional data, Vintage Data, Survey Data, Fixed Effects, Mixed Effects, and Random Effects models, HLOC data (High Low Open Close), Dummy variable levels and frequencies, Lagged Effects, Time Series Data, TS Model Structure, sort orders, and GLM distributional requirements are some examples.

5. New-School and Old-School Perspectives.

5A. OLD SCHOOL PHILOSOPHY.

Small sample sizes are often the rule.
Use BLUE (Best Linear Unbiased Estimate) statistical proofs to support cases where multicollinearity can be ignored for regression if the goal is prediction or forecasting.
Use artistic means such as the look and feel of the model according to scientific domain or business judgment.
Asking why, what matters most, or estimating ROI or other business metrics impact
Use The Scientific Method.
Cross-sectional data vs. lags and time series if not structured can create incorrect models.
Highly domain-dependent when measuring people against behavioral theories and not signals and impulses against hard science theories.
Delineation between perfect multicollinearity (p=+/- 1) and degrees of multicollinearity.
Correlation coefficients can help to diagnose multicollinearity along with more rigorous criteria. Correlation Coefficients are not a threshold to remove muticorrelated variables.
When possible, use cost-benefit financial analysis to determine the dollar impact of multicollinearity.
When to ignore multicollinearity other than in forecasting.
Thinking is often focused on OLS (linear regression) and logistic regression.
Fewer alternative models and fixes than NS.
Use VIFs, condition indices, variance proportions, or other diagnostic metrics to identify strong multicollinearity and assess which variables are causing the issue so informed corrective action can be taken.

5B. NEW SCHOOL PHILOSOPHY:

Billions of observations of up to real-time data mean multicollinearity symptom ID and diagnostic criteria are resource-intensive and do not allow the fast turnaround needed to fine-tune solution iterations. Especially true with wide data.
No time for art.
Often not asking why.
In addition to linear regression, multicollinearity can impact other models such as Naïve Bayes, KNN, logit/probit/Cox regression, SVM (Support Vector Machines), time series models including ARIMA and VAR, Poisson models, and neural nets.
Not affected …many new school methods, clustering, Decision Forests, MLP (Multi-Layer Perceptron), etc.
Improve forecast effectiveness via feature engineering.
The tendency to say multicollinearity should NOT be ignored in prediction/forecasting.
Some use rule-of-thumb correlation coefficient thresholds to exclude variables in automated feature engineering.
- NOTE: This may be done for ALL data and not just data that is used to estimate models that are subject to multicollinearity. This can weaken the application of multicollinearity-free models and methods.
Trivial, mainly legacy issue.
More alternative models and fixes than OS.
Use new algorithms designed to mitigate or remove multicollinearity’s impact.