Evaluating Data Statistics and Properties

When you first receive a new dataset, it is tempting to jump straight into building models or creating visualizations. However, experienced data professionals know that the most critical phase of any project happens before a single line of modeling code is written. This phase is data profiling—the process of examining your data, identifying its structure, and calculating statistics to understand its underlying properties.

Think of data profiling as a detective's initial investigation of a crime scene. You are looking for clues, inconsistencies, and patterns that will tell you what happened and what you can expect to find later. If you skip this step, you risk building your entire project on a foundation of "dirty" or misunderstood data. Evaluating data statistics and properties allows you to catch errors early, understand the limitations of your information, and make informed decisions about how to clean and transform the data for the best possible results.

In this lesson, we will explore the fundamental properties of data, how to calculate and interpret key statistics, and how to use these insights to prepare your data for analysis or machine learning.

Understanding Data Properties and Metadata

Before we look at the numbers, we must understand the structure of the data itself. Data properties, often referred to as metadata, describe the characteristics of the variables in your dataset. Understanding these properties helps you determine which statistical tests are appropriate and how you should handle different columns.

Syntactic vs. Semantic Properties

There is a difference between what the computer thinks the data is (syntactic) and what the data actually represents in the real world (semantic).

• Syntactic Properties: These are the technical data types assigned by your programming environment or database. Examples include integers, floats, booleans, and strings. For example, a "Customer ID" might be stored as an integer, but mathematically, it doesn't behave like one.
• Semantic Properties: These describe the meaning of the data. Even if "Customer ID" is an integer, it is semantically a unique identifier. You wouldn't calculate the average Customer ID because that number has no meaning. Similarly, a column for "Temperature" and a column for "Price" might both be floats, but they have different physical constraints (e.g., prices usually cannot be negative).

Data Levels of Measurement

Understanding the level of measurement is the first step in profiling. This tells you what kind of mathematical operations are legal for a specific column.

1. Nominal: Categorical data with no inherent order (e.g., Color, Country, Eye Color). You can count them and find the mode, but you cannot rank them.
2. Ordinal: Categorical data with a clear ranking or order (e.g., Education Level, Customer Satisfaction Rating). You know that "High" is better than "Low," but you don't know the exact mathematical distance between them.
3. Interval: Numerical data where the distance between values is meaningful, but there is no "true zero" (e.g., Temperature in Celsius or Fahrenheit). 20 degrees is not "twice as hot" as 10 degrees.
4. Ratio: Numerical data with a true zero point (e.g., Height, Weight, Income). Here, 100 dollars is exactly twice as much as 50 dollars.

Callout: Why Levels of Measurement Matter

If you treat an ordinal variable (like a 1-5 star rating) as a ratio variable, you might calculate an average of 4.2. While common, this can be misleading because the "distance" between a 1-star and a 2-star review might not be the same as the distance between a 4-star and a 5-star review. Always consider the level of measurement before choosing your summary statistics.

Descriptive Statistics for Numerical Data

When we talk about "evaluating statistics," we usually start with descriptive statistics. These provide a high-level summary of the distribution and characteristics of your numerical columns.

Measures of Central Tendency

Central tendency tells us where the "middle" of the data lies.

• Mean (Average): The sum of all values divided by the number of values. It is highly sensitive to outliers. If you have ten people earning $50,000 and one billionaire in the room, the mean income will be skewed significantly higher than what most people actually earn.
• Median: The middle value when the data is sorted. It is much more "robust" than the mean because it isn't pulled away by extreme outliers. In the billionaire example, the median would still be $50,000.
• Mode: The most frequently occurring value. This is useful for identifying common values in discrete numerical data.

Measures of Dispersion

Dispersion tells us how spread out the data is. A dataset could have a mean of 50 because every value is 50, or because half the values are 0 and half are 100. Dispersion helps us distinguish between these two scenarios.

• Range: The difference between the maximum and minimum values. While simple, it is entirely dependent on outliers.
• Variance: The average of the squared differences from the mean. It gives a sense of the "spread," but because it's squared, it's not in the same units as the original data.
• Standard Deviation: The square root of the variance. This is the most common measure of spread because it is in the same units as the data. A low standard deviation means the data points are close to the mean; a high standard deviation means they are spread out.
• Interquartile Range (IQR): The difference between the 75th percentile (Q3) and the 25th percentile (Q1). This represents the "middle 50%" of the data and is an excellent way to identify the spread without being influenced by outliers.

Shape and Distribution

Beyond the middle and the spread, we need to know the "shape" of the data.

• Skewness: This measures the asymmetry of the distribution.
  • Positive (Right) Skew: The tail on the right side is longer. Most data points are clustered on the left (e.g., household income).
  • Negative (Left) Skew: The tail on the left side is longer. Most data points are clustered on the right (e.g., age at death).
• Kurtosis: This measures the "peakedness" or "tailedness" of the distribution. High kurtosis means the data has heavy tails and a sharp peak, often indicating the presence of frequent outliers.

Note: A "Normal Distribution" (the bell curve) has a skewness of 0 and a kurtosis of 3. Many statistical models assume your data follows this distribution. If your profiling shows significant skewness, you may need to apply transformations (like a log transform) before modeling.

---

Practical Implementation: Profiling with Python

Let’s look at how we actually perform these evaluations using Python and the Pandas library. Pandas is the industry standard for data manipulation and provides built-in methods for quick profiling.

Step 1: Basic Schema Inspection

First, we load our data and check the syntactic properties.

import pandas as pd

# Load a sample dataset
df = pd.read_csv('customer_data.csv')

# Check data types and non-null counts
print(df.info())

The .info() method is your first line of defense. It tells you:

1. How many rows and columns you have.
2. The data type of each column (int64, float64, object).
3. How many missing (null) values exist in each column.

Step 2: Generating Summary Statistics

Next, we use .describe() to get the numerical summary.

# Detailed statistics for numerical columns
numerical_summary = df.describe()
print(numerical_summary)

By default, .describe() provides the count, mean, standard deviation, min, max, and the 25th, 50th (median), and 75th percentiles.

Pro Tip: To see the distribution of categorical data, you can use df.describe(include=['O']). This will show you the number of unique values, the most frequent value (top), and its frequency.

Step 3: Detecting Outliers with IQR

We can use the statistics we've gathered to programmatically find outliers. A common industry standard is the "1.5 * IQR rule."

# Calculate Q1 and Q3
Q1 = df['Annual_Income'].quantile(0.25)
Q3 = df['Annual_Income'].quantile(0.75)
IQR = Q3 - Q1

# Define bounds for outliers
lower_bound = Q1 - 1.5 * IQR
upper_bound = Q3 + 1.5 * IQR

# Identify outliers
outliers = df[(df['Annual_Income'] < lower_bound) | (df['Annual_Income'] > upper_bound)]
print(f"Number of outliers detected: {len(outliers)}")

Step 4: Measuring Skewness and Kurtosis

# Check for skewness in the 'Spending_Score' column
skew = df['Spending_Score'].skew()
kurt = df['Spending_Score'].kurt()

print(f"Skewness: {skew}")
print(f"Kurtosis: {kurt}")

If the skewness is greater than 1 or less than -1, the distribution is highly skewed. This is a signal that you might need to handle those values differently during the cleaning phase.

---

Evaluating Categorical Data Properties

Numerical data gets a lot of attention, but categorical data profiling is just as important. When evaluating categorical properties, we focus on cardinality and frequency.

Cardinality

Cardinality refers to the number of unique values in a column.

• Low Cardinality: A column with few unique values (e.g., "Gender" or "Subscription Tier"). These are easy to encode for machine learning.
• High Cardinality: A column with many unique values (e.g., "Zip Code" or "User Agent"). High cardinality can be problematic because it can lead to "overfitting" in models if not handled correctly.

Frequency and Mode

For categorical data, we want to know:

1. What is the most common category?
2. Is there a "class imbalance"? For example, if you are looking at credit card fraud, 99.9% of your data might be "Normal" and only 0.1% "Fraud." Standard statistics won't show this as an "error," but it is a vital property to understand before you start cleaning or modeling.

# Checking the distribution of a categorical column
print(df['Subscription_Tier'].value_counts(normalize=True))

Using normalize=True gives you the percentage of the total for each category, which is often more useful than the raw count.

---

Identifying Data Quality Issues Through Statistics

Evaluating statistics isn't just about understanding the "shape" of the data; it's about finding what's broken. Here are the common issues that statistical profiling reveals:

1. Missing Values (The "Swiss Cheese" Problem)

If the count in your summary statistics is lower than the total number of rows, you have missing values. You need to decide:

• Are they missing at random?
• Is the missingness itself a piece of information? (e.g., a missing "Date of Marriage" might simply mean the person is single).

2. Zero-Variance Variables

If the standard deviation of a column is 0, every single value in that column is the same. This column provides no information for a model and should usually be removed during the cleaning process.

3. Impossible Values

Summary statistics often reveal data entry errors.

• A "Min" age of -5.
• A "Max" height of 300 inches.
• A "Min" price of 0 for a luxury item. These are "impossible" based on domain knowledge and indicate that the data needs cleaning.

4. Data Leakage Indicators

Data leakage occurs when information from the "future" or the "target" is accidentally included in your features.

• High Correlation: If a feature has a 99% correlation with the target variable, it might be "leaking" the answer. For example, if you are predicting if a customer will churn, and you have a column for "Reason for Cancellation," that column is a leak because it only exists for people who have already churned.

Callout: Correlation vs. Causation

During profiling, you might find that "Ice Cream Sales" and "Drowning Incidents" are highly correlated. This doesn't mean ice cream causes drowning. Both are correlated with a third variable: "Hot Weather." Statistical profiling identifies the relationship; your domain knowledge identifies the reason.

---

Comparison Table: Statistical Measures and When to Use Them

Measure	Use Case	Sensitivity to Outliers
Mean	Best for symmetric data without outliers.	High
Median	Best for skewed data or data with outliers.	Low
Mode	Best for categorical or discrete numerical data.	Low
Std Dev	Best for understanding typical variance in normal data.	High
IQR	Best for understanding spread in skewed data.	Low
Range	Quick check of the total span of data.	Extreme
Skewness	Checking if the data needs transformation.	Moderate

---

Step-by-Step Guide: Evaluating a New Dataset

When you sit down with a new dataset, follow this workflow to ensure you've evaluated its properties thoroughly.

Step 1: Load and Inspect Metadata

Start by checking the shape of the data and the data types. Ensure that dates are recognized as dates and numbers as numbers. If a "Price" column is being read as a "String," you've already found your first cleaning task.

Step 2: Check for Completeness

Calculate the percentage of missing values for every column.

• Rule of Thumb: If a column is missing more than 50-60% of its data, it might be worth dropping entirely unless the missingness is meaningful.

Step 3: Analyze Central Tendency and Spread

Compare the mean and the median for every numerical column.

• If Mean > Median, the data is right-skewed.
• If Mean < Median, the data is left-skewed.
• Check the Min and Max to see if they make sense in a real-world context.

Step 4: Evaluate Categorical Diversity

Look at the unique counts for categorical columns. Identify high-cardinality columns (like IDs or addresses) that won't be useful for general patterns. Look for "near-zero variance" categories where one value represents 99% of the data.

Step 5: Check for Relationships (Correlation)

Create a correlation matrix. Look for variables that are highly correlated with each other (multi-collinearity). If two variables are 95% correlated, you probably don't need both.

# Correlation matrix
correlation_matrix = df.corr()
print(correlation_matrix)

Step 6: Document Findings

Create a "Data Profile Report." Note which columns have outliers, which are skewed, and where data is missing. This report will serve as the roadmap for your cleaning and transformation steps.

---

Common Pitfalls and How to Avoid Them

Even seasoned data scientists make mistakes during the evaluation phase. Here are the most common traps:

1. Trusting the Mean Blindly

The "Average" is one of the most misused statistics in the world. Always look at the median and the distribution alongside the mean. If you're looking at a dataset of "Average House Prices" in a city, a few multi-million dollar mansions will make the neighborhood look much more expensive than it actually is for the average resident.

2. Ignoring the "Long Tail"

In many real-world scenarios (like web traffic or wealth), a small number of observations account for the vast majority of the values. This is known as a Power Law distribution. If you try to treat this data as "Normal," your models will fail. Always check for a long tail in your distributions.

3. Misinterpreting Null Values

Don't assume a null value means "zero." In a medical dataset, a null "Blood Pressure" doesn't mean the patient has no blood pressure; it means it wasn't recorded. Treating nulls as zeros will drastically skew your statistics and lead to incorrect conclusions.

4. Overlooking Data Types

Sometimes numerical data is actually categorical. For example, "Store ID 101" and "Store ID 102." If you calculate the average Store ID, you are performing a meaningless calculation. Always map your syntactic types (integers) to semantic types (categories) during evaluation.

5. Failing to Check for Duplicates

Duplicate rows can artificially inflate your statistics and lead to over-optimistic results in machine learning. Always check for and evaluate duplicates. Are they true duplicates (the same event recorded twice) or just similar observations?

# Checking for duplicate rows
duplicate_count = df.duplicated().sum()
print(f"Total duplicate rows: {duplicate_count}")

---

Best Practices for Data Profiling

To make your data evaluation more effective, follow these industry standards:

• Use Visualizations Early: Numbers can be deceiving. A histogram or a box plot can reveal patterns (like bimodal distributions—where there are two "humps" in the data) that summary statistics might hide.
• Automate Where Possible: Use profiling libraries like pandas-profiling or ydata-profiling. These tools generate comprehensive HTML reports that include correlations, missing value heatmaps, and detailed statistics for every column with a single line of code.
• Incorporate Domain Knowledge: Statistics alone can't tell you if a value is "wrong." You need to know the context. Talk to subject matter experts to understand what the "expected" range of values should be.
• Profile Every Subset: Sometimes the statistics for the whole dataset hide important patterns in subsets. For example, the average purchase price might be $50, but if you profile by "Region," you might find it's $20 in one area and $200 in another.
• Maintain a Data Dictionary: As you evaluate properties, document them. Define what each column means, its units (e.g., kilograms vs. pounds), and its expected range. This is invaluable for anyone else who works with the data later.

---

Summary and Key Takeaways

Evaluating data statistics and properties is the "due diligence" of the data world. It is the process that transforms raw data into a known quantity, allowing you to proceed with confidence. By understanding the shape, spread, and nuances of your data, you avoid the "Garbage In, Garbage Out" trap that ruins so many projects.

Here are the key takeaways from this lesson:

1. Distinguish Between Syntactic and Semantic Types: Don't just trust the data types assigned by your software. Understand what the data represents in the real world to choose the right statistics.
2. The Mean is Not Enough: Always use the median and standard deviation to get a true sense of central tendency and spread, especially in the presence of outliers or skewed distributions.
3. Identify the Level of Measurement: Know whether your data is nominal, ordinal, interval, or ratio. This determines which mathematical operations are valid.
4. Use the 1.5 * IQR Rule for Outliers: This is a robust, standard way to identify extreme values that might need to be removed or capped during the cleaning phase.
5. Check for Cardinality and Class Imbalance: In categorical data, high cardinality and skewed class distributions can significantly impact the performance and fairness of your models.
6. Look for Missing Patterns: Missing data is rarely random. Use statistics to determine the extent of missingness and investigate if there is a systematic reason behind it.
7. Correlation is a Red Flag for Leakage: High correlation between features or between a feature and the target is a signal to investigate further for data leakage or multi-collinearity.
8. Visualize to Verify: Always complement your statistical summaries with visual tools like histograms and box plots to catch nuances like bimodal distributions or subtle skewness.

By mastering these evaluation techniques, you set yourself apart as a data professional who doesn't just "run models," but truly understands the information they are working with. This foundation is what makes the subsequent steps of cleaning, feature engineering, and modeling successful.