Exploratory Data Analysis — A Repeatable Workflow

A step-by-step EDA checklist: data quality, distributions, segments, relationships and anomalies, ending in written findings.

Est. 40 min Updated Suggest a fix
// Part 01 — What EDA is for

Explore to Learn What the Data Can Support

Exploratory data analysis (EDA) is the disciplined process of understanding a dataset before drawing conclusions or building a model. It helps you discover what was measured, find data-quality issues, see distributions, compare relevant groups, and generate questions for further investigation. It does not by itself prove a cause or establish that a pattern will persist.

Imagine that an operations team reports an increase in delivery time. The first step is not to pick a chart; define the target population, date window, unit of observation, time field, and delivery-time definition. Then preserve the input, check its structure, and explore in an order that avoids treating every interesting pattern as a discovery. Write down decisions as you go so the analysis can be repeated after the next data refresh.

An ordered workflow makes exploration more useful than a gallery of charts.
EDA stageQuestionOutput
ScopeWhat do these rows represent?Population, grain, period and definitions
QualityCan the fields be trusted?Missingness, duplicates, ranges and types
ShapeHow are values distributed?Counts, quantiles and visual summaries
SegmentsWhere do patterns differ?Comparable subgroup summaries
FindingsWhat is supported and what remains unknown?Evidence, limitation and next check
💡 Note
EDA generates hypotheses and checks data. Use a separate confirmatory design or additional evidence before treating an observed association as an explanation.
// Part 02 — Scope and schema

Confirm the Grain, Fields and Time Window

Write a data inventory before summarising. Identify the source, extract date, intended population, row grain, primary key, time zone, units, categorical fields, numeric measures, and any known exclusions. Check that the row count is plausible and that expected columns are present. An order table can have one row per order, while a delivery event table may have many events per order; merging them without a plan changes the grain.

eda_inventory.py
python
import pandas as pd

orders = pd.read_csv("data/orders.csv")
print("shape:", orders.shape)
print("columns:", list(orders.columns))
print("types:
", orders.dtypes)
print("sample:
", orders.head(5))
print("unique order IDs:", orders["order_id"].nunique())

Compare the unique key count with row count and investigate any difference. A duplicate may be a valid multiple event or an accidental repeated record; do not deduplicate until the intended grain is known. Similarly, inspect date parsing and reporting timezone before deciding what belongs to a day or week.

⚠️ Important
If the data dictionary or owner cannot tell you what a field means, mark its interpretation as unknown. Do not infer a definition from a column name alone.
// Part 03 — Quality checks

Measure Missingness, Duplicates and Implausible Values

Count missing values, duplicate keys, invalid categories, and values outside known operational bounds. Look for impossible timestamps such as a delivery before order placement, but verify whether time zones or asynchronous ingestion explain the apparent anomaly. A suspicious value deserves investigation, not automatic deletion. Track both the number and share of affected records, and determine whether issues cluster in one source, region, or period.

eda_quality.py
python
required = ["order_id", "city", "status", "order_date", "delivery_minutes"]
print("missing by column:
", orders[required].isna().sum())
print("duplicate IDs:", orders["order_id"].duplicated().sum())
print("statuses:
", orders["status"].value_counts(dropna=False))

impossible = orders["delivery_minutes"].lt(0)
print("negative delivery times:", int(impossible.sum()))
print("missing city share:", round(orders["city"].isna().mean(), 3))

Validate metric-specific completeness. If delivery duration is missing only for cancelled orders, the missingness may be expected; if it is absent for a particular city after a date, an upstream pipeline may have failed. Compare raw and cleaned record counts and retain an exclusion log. A clean-looking aggregate does not guarantee that the included observations are representative.

  1. Check required fields and their actual types.
  2. Quantify missing values and duplicate keys by time and important segment.
  3. Inspect impossible and extreme values against domain rules and source records.
  4. Record a disposition: keep, correct, exclude, or investigate further.
// Part 04 — Distributions

Inspect One Variable Before Comparing Groups

For a numeric measure, review count, minimum, maximum, median, quartiles, and spread. A histogram shows how values are distributed across bins; a box plot gives a compact view of centre and spread but can hide multiple clusters. For categories, use counts and shares and include missing or unknown values. Choose bins and scales consciously: their settings affect what a chart appears to show.

eda_distributions.py
python
measure = orders["delivery_minutes"].dropna()
print(measure.describe(percentiles=[0.1, 0.25, 0.5, 0.75, 0.9]))
print("status share:
", orders["status"].value_counts(normalize=True, dropna=False))

ax = measure.plot.hist(bins=20, edgecolor="white", figsize=(8, 4))
ax.set(title="Delivery-time distribution", xlabel="Minutes", ylabel="Orders")

A long right tail is common for wait times, spend, and support resolution; it can make the mean much higher than the median. Multiple peaks may indicate different operating contexts or data sources. Do not label every tail point an outlier: it may represent the most important failures to understand. Report the sample count and use units in both text and charts.

🎯 Pro Tip
Inspect a histogram and summary statistics together. A single mean cannot tell you whether the data are symmetric, clustered, bounded, or dominated by a few extreme observations.
// Part 05 — Segments

Compare Like with Like

Segment by a business-relevant dimension such as city, service type, weekday, or new versus returning customer. Show group size next to the measure: a small group can have a volatile average. Compare equivalent time windows and definitions, and be alert to composition changes. If one city serves a different mix of orders or has different operating hours, an unadjusted comparison may combine several effects.

eda_segments.py
python
completed = orders.loc[orders["status"].eq("completed")].copy()
segment_summary = (
    completed.groupby("city", dropna=False)
    .agg(
        orders=("order_id", "nunique"),
        median_minutes=("delivery_minutes", "median"),
        p90_minutes=("delivery_minutes", lambda values: values.quantile(0.90)),
    )
    .sort_values("orders", ascending=False)
)
print(segment_summary)

A grouped table can reveal questions worth investigating, but it does not automatically explain why a group differs. Inspect data coverage and denominators, and compare within similar periods or operating conditions where possible. Predefine a manageable set of meaningful segments rather than slicing repeatedly until one group looks unusual.

Use a measure and denominator that fit the segment question.
ComparisonUseful viewImportant check
Delay by cityMedian and p90 with order countOperating mix and volume
Orders by weekdayDaily counts over comparable weeksDifferent number of days in the window
Cancellation by channelRate and eligible-order denominatorChannel coverage and status definition
Basket value by customer typeMedian and distributionNew/returning definition and time window
// Part 06 — Relationships and anomalies

Explore Associations Without Claiming Causes

A scatter plot can show whether two numeric variables move together, while a correlation coefficient summarizes a particular kind of linear association. Neither establishes that one variable caused the other. A third variable, selection effect, reverse direction, measurement error, or nonlinear relationship can shape the pattern. Examine units, missingness, outliers, and the scatter plot before interpreting a single coefficient.

eda_relationship.py
python
paired = orders[["delivery_minutes", "distance_km"]].dropna()
print("paired observations:", len(paired))
print("Pearson correlation:", paired.corr(method="pearson").iloc[0, 1])

ax = paired.plot.scatter(
    x="distance_km", y="delivery_minutes", alpha=0.35, figsize=(7, 4)
)
ax.set(title="Distance and delivery time", xlabel="Distance (km)", ylabel="Minutes")

An anomaly is a record or pattern that differs from expectations; it is not automatically a bad record. Check the original source and nearby events, then decide whether it points to data quality, a rare but valid event, or a new operational process. Keep a note of the investigation outcome so the same record is not rediscovered and removed without context every month.

📌 Real World Example
A sudden rise in reported cancellations may be a product change, a source-system status mapping change, or a real operational issue. Compare event definitions and ingestion changes before presenting it as customer behaviour.
// Part 07 — Write findings

Separate Evidence, Interpretation and Next Checks

A clear finding states the observation, the population and period, the comparison, the evidence, and a limitation. For example: "Among completed deliveries recorded in the four-week window, the p90 delivery time rose in two high-volume cities; order mix and reporting coverage have not yet been checked." That sentence reports a pattern and a next step without claiming a cause.

Keep an analysis log with the source, query or notebook version, filters, cleaning decisions, and chart definitions. Share concise tables and charts with context, not a pile of exploratory outputs. If the team needs a causal answer, propose a follow-up that could distinguish competing explanations, such as validating a system change, comparing an appropriate control group, or designing an experiment.

  • What was observed, in which population and period?
  • How large is the difference, and what is the denominator?
  • What quality or composition issue could change the interpretation?
  • Which conclusion is supported, and what remains a hypothesis?
  • What follow-up would reduce the most important uncertainty?
// Part 08 — Pitfalls

Common EDA Mistakes

Treating an unusual observation as an error by default

Why: Extreme values may represent genuine rare failures or customer behaviour.

Fix: Check provenance and domain rules, document the decision, and compare results with and without the record when appropriate.

Comparing groups with different denominators or time windows

Why: Volume, exposure, calendar coverage, or eligibility differ across groups.

Fix: Align definitions and periods, report group sizes, and choose a denominator matching the question.

Searching through many segments until one looks significant

Why: Repeated slicing increases the chance of a chance pattern.

Fix: Use meaningful pre-specified comparisons and confirm surprising results with new data or a designed analysis.

Writing that a correlation explains an outcome

Why: Association alone cannot distinguish confounding, reverse direction, or coincidence.

Fix: Describe the relationship as an association and list plausible alternative explanations and next tests.

// Part 09 — Practice

Run an EDA Review for Delivery Operations

Practice

Produce a short, defensible EDA memo

  1. Write down the row grain, population, reporting window, primary key, time zone, and definitions for status and delivery duration.
  2. Report missingness, duplicate keys, category counts, and impossible values; investigate one issue in its source context.
  3. Summarise the overall distribution with count, median, upper percentile, and a histogram with labelled units.
  4. Compare delivery-time summaries by city with order counts and flag low-volume groups.
  5. Inspect a relationship between distance and delivery time, report paired sample count, and avoid causal language.
  6. Write two findings, one limitation, and one follow-up that would test a plausible explanation.

Hint: Use the pandas workflow from the previous lesson. A small synthetic or approved public dataset is enough to practise the checklist.

🎯 Key Takeaways

  • ✓EDA is a structured way to understand scope, quality, distributions, segments, and relationships before drawing conclusions.
  • ✓Confirm row grain, field definitions, units, timezone, population, and period before summarising.
  • ✓Quantify missingness, duplicates, invalid categories, and extreme values; investigate rather than automatically deleting.
  • ✓Inspect distributions and denominators alongside averages and rates.
  • ✓Compare meaningful groups with aligned definitions, periods, and coverage.
  • ✓Association and anomalies generate questions; they do not prove causes.
  • ✓Write findings with evidence, population, period, limitations, and a clear next check.

Next

What comes next

EDA reveals the shapes and questions in a dataset. Next, learn the statistical summaries that describe those distributions accurately.

Lesson 05 · Descriptive Statistics and Distributions →

HELP US IMPROVE

How was this lesson?

Your note is shared with the Chaduvuko team for review. Please don’t include personal or sensitive information.

Choose your feedback

Spotted a mistake or something unclear? Suggest a fix on GitHub or email us. Select the text first and it is quoted for you.

Share

Discussion

0

Have a better approach? Found something outdated? Share it — your knowledge helps everyone learning here.

Continue with GitHub
Loading...