DRAFT UNDER LIVE CONSTRUCTION

By Dr. Dean G. “Gordy” Fairchild (CEO, Synergy Data Science), poe.com AI (Assistant)

New-School Sampling

In new-school problems, practitioners typically use all the available data to perform many kinds of analyses even if there are billions of observations. The standard approach is to draw a training sample from which a model (or models) is/are developed and a test sample to evaluate whether there is over or under-fitting, obtain accuracy metrics, and further tune hyperparameters. Under this method, a random training sample (often about 70%) of the data is distinct from a random testing sample (often 30%). Both are drawn from the full dataset. The selection of the samples is typically SRS (Simple Random Sample), although other methods can be used. There is sometimes a third type of sampling called a validation or cross-validation sample. Typically, cross-validation is a resampling method when training machine learning models that splits labeled data into training and test sets. In each iteration of cross-validation, different parts of the data are used to train and test the model. The training set is used to train a model, and the test set is used to make predictions and compare them with the actual labels for those entries. Afterward, an overall accuracy metric is calculated to estimate the predictive performance of the resulting model. [Data Camp Glossary & Dr. Gordy Fairchild]

Models are estimated (trained) using training sample data only. The resulting model is then applied to the testing sample to see if there is “overfitting”, “underfitting”, or how well the model predicts with different data. Overfitting means that the model is so dependent on the training sample that it does not fit the test sample very well. Underfitting refers to a scenario where the model being used is too simple or has too few parameters to capture the underlying patterns and complexity in the data. The model can be fine-tuned based on result differences between train and test to be more accurate (or to be optimal on non-accuracy metrics) .

Every single data point post-cleanse is “sampled” and used, often at great time and resource cost. The problem is that the 70/30 split sample method violates the best practices of the statistical concept of sampling.

However, new-school practitioners point out the following:

• This process is standard and results might not be accepted if the process was changed.
• Many kinds of models (especially unsupervised machine learning models) are not very (or at all) statistical, so this argument about using statistical sampling theory is moot for those cases. After all many kinds of learning models need to “learn” from using all of the data as it occurs in time and space to optimize their effectiveness. The rise of machine learning, particularly deep learning, has shifted the focus of many data science practitioners away from traditional statistical theory. Machine learning models often operate as “black boxes” and can achieve high performance without relying on explicit statistical assumptions or modeling. Note that within the Blackbox advanced statistical concepts may be used. But typically, many new-school practitioners are unfamiliar with details.
• If the goal of the model is the best prediction, it can be argued that the legacy nuances of statistic theory are not required to obtain the best answer or “answer set”. In many data science applications, the primary goal is to achieve the best predictive performance, rather than to build interpretable statistical models. This has led to a greater emphasis on the use of powerful, but less transparent, machine learning algorithms over classical statistical techniques.
• The availability of large, complex datasets has made traditional statistical methods less practical or even infeasible. The assumptions and requirements of many statistical techniques may not hold true for the scale and complexity of modern data.
• New-school data science often involves an iterative, exploratory approach to problem-solving, which may not align well with the more formal, theoretically grounded statistical paradigm.
• The widespread availability of user-friendly machine learning libraries and frameworks (e.g., TensorFlow, PyTorch, sci-kit-learn) has made it easier for data scientists to apply advanced machine learning techniques without in-depth statistical knowledge. Some packages and algorithms already use more theoretically pure stat methods to solve parts of the problem inside what is otherwise a “black box”.
• In many data science projects, there is a greater emphasis on delivering practical solutions and meeting deadlines, which may take precedence over a rigorous statistical approach.
• Much of the focus on using stat sampling theory in practice is focused on specific kinds of problems like survey sample sizes for questionnaires, the behavior and the use of random numbers deep in an algorithm, or cases where there is a small number of data observations, We are in the era of Big Data.

But, from a new-school perspective, a 70/30 test-train split is often argued against for a mix of statistical and non-statistical reasons. The key reasons are:

• Lack of Representativeness: A 70/30 split means that 70% of the data is used for training, while only 30% is held out for testing. This can lead to an unrepresentative test set, as the small size may not capture the full distribution and diversity of the data. The test set needs to be large enough to provide a reliable estimate of the model’s performance on unseen data.
• Overfitting Risks: With a 70/30 split, the training set is quite large compared to the test set as mentioned. This increases the risk of overfitting, where the model performs well on the training data but fails to generalize to new, unseen data. A smaller training set and a larger test set can help identify and mitigate overfitting issues.
• Unreliable Performance Evaluation: The small test set in a 70/30 split may not provide a stable or reliable estimate of the model’s performance. This is especially a property of the volume of data in observations and features. New-school practitioners may be unused to smaller datasets of under 10,000 observations or down to a sample of 30 or so. The performance metrics calculated on a small test set can be highly sensitive to the specific samples in the test set, leading to inconsistent and unreliable results.
• Lack of Cross-Validation: A 70/30 split does not allow for effective cross-validation, which is a crucial technique for evaluating model performance and tuning hyperparameters. Cross-validation requires multiple train-test splits to provide a more robust and reliable assessment of the model. Cross-validation makes considerable use of the statistical theory of the sampling distributions which is mentioned more below.
• Instead of a 70/30 split, it is generally recommended to use a more balanced split, such as a 60/40 or a 50/50 split, or to employ techniques like k-fold cross-validation. These approaches provide a more representative and reliable assessment of the model’s performance, helping to ensure the model can generalize well to new, unseen data.

Some think that either new-school or old-school alone is the only correct philosophy. Not both. The freshest six-week BootCamp new-school intern may literally have been brainwashed that a certain process is the only option and is not to be questioned. The most experienced star Ph.D. econometrician has also been brainwashed but in a much different way. Such elites are exposed to a larger amount of information and practical experience to shape their techniques, but they often restrict themselves to a hardcore theoretical domain and use a handful of “proven” techniques. Such people would rather not risk learning anything outside their insular paradigms. Most other practitioners see the new-school/old-school dichotomy as an effective way to improve the craft by discussing best practices for each school to improve the overall quality of data science.

Old-School Sampling

Old-school practitioners would argue that more traditional sample sizes provide additional reasons a 70/30 split is undesirable and that using ALL the data to test, train, and validate can actually harm the quality of conclusions and model effectiveness, bias, accuracy metrics and findings. For one thing, old-school practitioners often object to calling 70-30 train & test splits “random samples” since using all the data is hardly sampling and those proportions are unnecessarily large for many kinds of problems. Typically, NS practitioners do not perform summary statistical analyses to compare the mean, median, skewness, kurtosis, and standard deviation/standard error for each variable in each sample including the binary target (numeric 0,1). The value of that exercise is to see whether there even are statistically significant differences between train and test. With big data (millions to billions of observations) this would typically reveal the samples are approximately equal on the key metrics of comparison such as mean, median, and standard deviation. Splits that use all the data would reveal no significant numerical difference between test and train (a quantifiable conclusion) and are thus both a waste of time and a strain on computer resources in cases with billions of observations.

In reality, if the objective is to minimize overfitting, statistically-based samples of 1-10% may be superior. There are established formulas for desirable statistical sample sizes. Not using all the data avoids the inclusion into the models of spurious or extremely minor variables or flukes. Preventing these low magnitude and minor issues can prevent overfitting the training data. However, then there would be a risk of underfitting the data as well. However, the argument is circular. If the cardholder’s astrological sign causes a $0.0001 increase in the probability of default the result is too insignificant to matter. You could argue the model is overfit with this variable but underfit without it. The most important variables and the simplest models that predict well are often simple with comparatively fewer variables and less complexity. Yet, such models do have a higher possibility of underfitting unless cross-validation on a variety of separate samples is used.

Additionally, splits that use all data (such as the standard new-school 70-30 random sample for train and test) are unlikely to show significant differences between the two groups on descriptive statistics with Big Data datasets. Thus you are not setting up a fair test. The size of the test dataset is so large (millions to billions of observations) that you are testing a “sample” that cannot usually be distinguished from the train “sample”. With smaller random samples of 1-10% the two samples are more likely to differ statistically so the overfitting criteria are more robust. The statistical approach leverages facts about specialized random samples to save time and cost in training, testing, validating, and evaluating models. Ultimately, the model can be used in production to score all new data in real-time or short turnaround types. As an aid to validation, the final model(s) can be analyzed using all the data.

Thus, for many old-schoolers true sampling is NOT a 70/30 split of a gargantuan dataset. Statistical sampling methods would be employed to lower computing burden and avoid overfitting/underfitting by leveraging true (smaller) random samples for train, test, cross-validation, time series holdout, and feature engineering.

Note that one caveat is in order, unbalanced datasets. With fraud or anti-money laundering (AML) data, there may be billions of transactions, but only 100 that are known to be fraudulent. Here, in so-called rare event data, all 100 bads may be oversampled and specialized procedure used to evaluate effectiveness and accuracy netrics.

Formulating appropriate research questions and hypotheses
Selecting appropriate statistical models and techniques
Ensuring valid inferences and interpretations
Quantifying uncertainty and assessing statistical significance
Designing effective experiments and sampling strategies

SAMPLING DISTRIBUTION: The sampling distribution of an estimator is the probability distribution of the estimator when it is applied to random samples. The tool on this page allows you to explore empirically the sampling distribution of the sample mean and median given any sample percentage of random draws with or without replacement draws from a box of numbered tickets. Sampling error (difference between population metric and sample metric) varies with each sample. The more times a sample is redrawn, means and medians come closer to the true population and a standard error is used to determine whether there is a statistically significant difference.

In the 3 graphs below, we compare the sample distribution of the scores estimated by a logistic model with a dependent binary variable.

SAMPLING METHOD:

SAMPLING SNIPPETS
Sampling may be a 70/30 split of the entire data set or statistical sampling methods employed to lower computing burden and avoid overfitting/underfitting by leveraging true (smaller) random samples for train, test, validation, time series holdout, and feature engineering.

Non-Representative Sampling Bias
Non-Representative Sampling Bias
Non-representative sampling bias is also referred to as selection bias. This inaccuracy occurs because of implementing random methods during the selection process. It results in an excess representation of some of the elements in the population. All the samples collected using convenience sampling are caused by bias. These type of situation are called under coverage bias.

BOOTSTRAPPING:

BOOTSTRAPPING:

(OS) any test or metric that relies on random sampling with replacement. Bootstrapping allows assigning measures of accuracy (defined in terms of bias, variance, confidence intervals, prediction error, or some other such measure) to sample estimates. (https://en.wikipedia.org/wiki/Bootstrapping_(statistics))

Confidecne intervals depend on sample size and the dispersion of the variable.

CROSS-VALIDATION:

Measurement error

Data reduction

Synergistic Data Science blends the best of both schools. New-school creativity and new-school coding brilliance is based on Big Data problems, AI, and the use of models first based on machine behavior. The business-friendly old-school Scientific Method and old-school diversity of thought and practice sit on top of a focus on human (not machine) behavior. Having all of that in one Synergistic Method brings great returns. The fact that viewpoints on sampling techniques, traditions, and methods differ should be a source of celebration and not mistrust.