Are outliers bad news for data scientists? Should they be thrown out of modeling datasets or models themselves?

By Dr. Dean G. “Gordy” Fairchild (CEO, Synergy Data Science), poe.com AI (Assistant), poe.com Data-Scientist-GPT3, Univ of California Berkely Statistics Glossary, Data Camp Data Science Glossary

The answer to both questions depends on old-school vs. new-school viewpoint as well as the context of the problem and what type of outlier is being described. One thing is for sure they are not always bad, nor should they always be thrown out.

(Old-School, OS) An outlier is an observation that is many Standard Deviations from the mean or possibly above the 99th percentile or below the first percentile. It is sometimes tempting to discard outliers, but this is imprudent unless the cause of the outlier can be identified, and the outlier is determined to be spurious. Also, if the problem at hand is to examine in the top 1% of income, wealth, or net worth for studies of affluent consumers for example. Otherwise, discarding outliers can cause one to underestimate the true variability of the measurement process. Sometimes, listing the top 50 or bottom 50 observations will allow the data scientist to see a natural break above (or below) which signifies extreme outliers or influential observations. In old-school data science, several methods are used to detect outliers:

• Visual Inspection: In old-school data science, one of the simplest methods to detect outliers is through visual inspection of data using scatter plots, box plots, or histograms. Outliers may appear as data points that are significantly distant from the majority of the data.
• Z-Score: The Z-score method calculates the number of standard deviations an observation is away from the mean. Observations with a Z-score greater than a certain threshold, usually set as 2 or 3, are considered potential outliers.
• Modified Z-Score: The modified Z-score is a variation of the Z-score method that uses the median and median absolute deviation (MAD) instead of the mean and standard deviation. It is more robust to outliers.
• Range Rule: The range rule is a simple rule of thumb where observations beyond a certain range from the mean are considered outliers. This range is often defined as mean ± a certain number of standard deviations.

Outlier measures for specific models can also be used. In the context of regression models, there are several statistics and graphs that can be used to evaluate the power and influence of outliers. These techniques help assess how influential outliers are in terms of their impact on the regression model’s coefficients, goodness-of-fit measures, and overall model performance. Here are a few commonly used methods:

• Cook’s Distance: Cook’s distance measures the influence of each observation on the regression model’s coefficients. It quantifies how much the model’s predictions change when a particular observation is removed. Larger values of Cook’s distance indicate greater influence of the corresponding data point on the model.
• DFFITS: DFFITS measures the influence of each observation on the fitted values or predicted response. It calculates the standardized difference between the predicted response values with and without each observation. DFFITS values larger than a threshold, typically ±2√(p/n), where p is the number of predictors and n is the sample size, are considered influential.
• DFBETAS: DFBETAS assess the influence of each observation on the regression coefficients. It calculates the standardized difference between the estimated coefficient values with and without each observation. DFBETAS values larger than a threshold, typically ±2/√n, where n is the sample size, are considered influential.
• Residual Plots: Residual plots provide a visual assessment of outliers’ impact by examining the patterns of the model’s residuals. Outliers may appear as data points with large positive or negative residuals, indicating substantial deviations from the predicted values. Common residual plots include scatter plots of standardized residuals against predicted values or against the independent variables.
• Influence Plots: Influence plots combine both graphical and statistical measures to assess the influence of individual observations. They often display measures such as Cook’s distance or standardized residuals against the observation index. Points outside a certain threshold are considered influential outliers.

(New School, NS) An outlier is an abnormal value in a dataset that deviates considerably from the rest of the observations. Outliers can be evidence of a measurement error or extraordinary event. Often NS automatic cleansing algorithms will automatically remove the top or bottom percentile or data above (or below) a set number of standard deviations from the mean (clipping). At first glance, this is unwise based on OS practice, but outliers can significantly skew results, especially for certain kinds of models (like regressions) so that coefficient values are misleading. It can depend on the nature of the problem as when you are interested in the middle classes or skewed distributions. It can be too time-consuming to do a physical inspection of outliers in cases where speed-to-result is important or data quality is known to be error-prone. Or, techniques such as using Median-based accuracy measures in forecasting are less sensitive to outliers.

In general, robust statistical models are less sensitive to outliers compared to traditional models. They are designed to handle data with outliers or influential observations without significantly affecting the model estimates. Here are a few models that are known for their robustness to outliers:

• Robust Regression: Robust regression methods, such as RANSAC (RANdom SAmple Consensus) and Huber regression, are less influenced by outliers compared to ordinary least squares (OLS) regression. These methods downweight or ignore outliers, resulting in more robust parameter estimates.
• Decision Trees: Decision trees are less sensitive to outliers because they partition the data based on feature values and thresholds rather than relying on the overall distribution of the data. Outliers may not strongly influence the tree-building process as long as they are not the majority within a specific split.
• Random Forests: Random forests, being an ensemble of decision trees, inherit the robustness of decision trees. They aggregate the predictions of multiple trees, reducing the impact of outliers on the overall model prediction.
• Support Vector Machines (SVM): SVMs are generally robust to outliers due to the use of support vectors, which are the data points closest to the decision boundary. Outliers that are far away from the support vectors have less influence on the model.
• Quantile Regression: Unlike ordinary least squares regression, quantile regression estimates the conditional quantiles of the response variable. It is less sensitive to outliers because it focuses on estimating different parts of the distribution rather than the mean.

It’s important to note that while these models are considered robust to outliers, extreme outliers can still have some influence on the model. Additionally, the definition of an outlier may vary depending on the context and the specific problem at hand. Preprocessing techniques such as outlier detection or data transformation may be necessary to handle outliers effectively in any model.

Also in new-school data science, certain methods can be used to detect outliers:

• Box Plot and Interquartile Range (IQR): Box plots provide a visual representation of the distribution of the data, including the median, quartiles, and potential outliers. Observations outside a certain range, typically defined as 1.5 times the IQR, are considered outliers.
• Local Outlier Factor (LOF): LOF is an algorithm that measures the local density deviation of a data point compared to its neighbors. Points with a significantly lower density compared to their neighbors are considered outliers.
• Isolation Forest: The isolation forest algorithm isolates outliers by randomly selecting a feature and then randomly selecting a split value between the maximum and minimum values of that feature. Outliers can be identified by the number of splits required to isolate them.
• Robust Mahalanobis Distance: Mahalanobis distance measures the distance of an observation from the mean of a multivariate distribution, accounting for the covariance structure. Robust Mahalanobis distance is less sensitive to outliers and can be used to detect them.

In new-school models, such as machine learning algorithms or advanced statistical techniques, there are several outlier detection statistics and graphs that can be used to evaluate the power of outliers on model results. These methods help assess the influence of outliers on model performance, accuracy, and generalization. Here are a few commonly used techniques:

• Residual Analysis: Residual analysis is a powerful method to evaluate outliers in new-school models. It involves examining the difference between the predicted and actual values. Outliers may be identified as data points with large residuals, indicating significant deviations from the model’s predictions.
• Mahalanobis Distance: Mahalanobis distance measures the distance of each observation from the mean of a multivariate distribution, taking into account the covariance structure. Outliers are identified as data points with large Mahalanobis distances, which indicate they are far away from the central distribution of the data.
• Local Outlier Factor (LOF): LOF is a popular outlier detection algorithm that measures the local density deviation of a data point compared to its neighbors. Points with significantly lower density compared to their neighbors are considered outliers. LOF provides a measure of the outlier score for each observation.
• Isolation Forest: Isolation Forest is another algorithm used for outlier detection. It isolates outliers by randomly selecting a feature and then randomly selecting a split value within the range of that feature. Outliers are identified as data points that require fewer splits to isolate them. Influence Plots: Influence plots combine graphical and statistical measures to evaluate the influence of individual observations on the model. They often display measures such as Cook’s distance, leverage, or standardized residuals against the observation index. Points outside certain thresholds are considered influential outliers.
• Receiver Operating Characteristic (ROC) Curve: In classification problems, ROC curves can be used to evaluate the power of outliers. By plotting the true positive rate against the false positive rate, the ROC curve provides a measure of the model’s performance in detecting outliers. A higher area under the curve (AUC) indicates better outlier detection capability.

These statistics and graphs help assess the power of outliers in new-school models and guide further analysis or model improvement. It’s important to note that outlier detection is a complex task and requires consideration of the specific problem, data characteristics, and modeling assumptions.