Linux and Shell Scripting for Data Engineers

Every Data Pipeline Runs on Linux

Almost every server that runs a data pipeline — cloud VMs, Docker containers, Kubernetes pods, Airflow workers, Spark executors — runs Linux. When a pipeline fails at 3 AM, you SSH into a Linux box and diagnose it. When a disk fills up and kills a pipeline, you find the culprit with Linux commands. When you need to quickly inspect a 10 GB log file without loading it into Python, you use Linux tools that do it in seconds.

Linux proficiency for a data engineer is not about memorising every command. It is about being comfortable in a terminal, knowing which tools solve which problems, and being able to write shell scripts that automate the repetitive operational tasks that surround every data pipeline. This module is built around a single thread you will follow the whole way through: a real orders pipeline for a company called FreshCart, and every tool introduced along the way is one you will actually use on it — not a detached command reference you have to mentally translate into a real situation later.

Moving Around and Finding What You Need

You have just been given SSH access to pipeline-01, the server that runs FreshCart's nightly orders pipeline. The first thing any data engineer does on an unfamiliar server is get their bearings — where am I, what is here, and how big is it.

Where am I, and what's here

pwd

/home/pipeline_user

pwd ("print working directory") always tells you exactly where you are — the single most useful command to run the moment you feel lost. Next, move to where the pipeline actually lives and see what's there.

cd /data/pipelines
ls -lah

total 24K
drwxr-xr-x 5 pipeline_user data_team 4.0K Mar 17 06:00 .
drwxr-xr-x 3 root          root      4.0K Jan  4  2026 ..
-rwxr-xr-x 1 pipeline_user data_team  892 Mar 17 06:00 run_orders.sh
drwxr-x--- 2 pipeline_user data_team 4.0K Mar 17 06:14 logs
drwxr-x--- 2 pipeline_user data_team 4.0K Mar 16 23:00 pipeline

-l gives the long listing (permissions, owner, size, date — all things you will need constantly), -a shows hidden files (anything starting with a dot), and -h makes sizes human-readable (4.0K instead of 4096). This three-flag combo is worth memorising as one habit: ls -lah is the default way any experienced engineer looks at a directory.

Finding files without knowing exactly where they are

You need last night's log file, but you don't remember the exact path.find searches a directory tree by name, age, or size — and it is almost always faster than clicking through folders.

find /data -name "orders_*.log" -mtime -1

/data/pipelines/logs/orders_20260317.log

-name matches a filename pattern; -mtime -1 means "modified in the last 1 day" — the minus sign means less than. A few more variations you will reach for constantly:

find /data -size +1G                         # files larger than 1 GB
find /data -empty                            # empty files — often a sign something failed
find /tmp -name "*.tmp" -mtime +7 -delete    # delete .tmp files older than 7 days

Viewing file content without opening an editor

tail -n 20 /data/pipelines/logs/orders_20260317.log

2026-03-17 06:00:04 INFO  Starting orders_pipeline
2026-03-17 06:00:05 INFO  Connected to database
2026-03-17 06:02:41 INFO  Batch 1 complete: 10000 rows
2026-03-17 06:04:58 INFO  Batch 2 complete: 10000 rows
2026-03-17 06:07:12 WARNING DEBUG_MODE=true detected — writing full row dump

That last line is worth remembering — it becomes important later in this module. tail shows the end of a file (the default is 10 lines; -n 20 asks for 20). Its most useful mode for a live pipeline is -f, which follows the file and streams new lines as they are written — the command you leave running in a terminal while a pipeline executes:

tail -f /data/pipelines/logs/orders_20260317.log | grep --line-buffered ERROR

This follows the log in real time but only prints lines containing ERROR — everything else scrolls by silently. head is the mirror image of tail, showing the first N lines instead of the last — useful for checking a CSV's header without printing the whole file with cat, which for a 10 GB file would flood your terminal and do nothing useful.

How much disk space is actually left

df -h /data

Filesystem  Size  Used  Avail  Use%  Mounted on
/dev/sdb1   500G  460G    40G   93%  /data

df ("disk free") reports space at the filesystem level — this is the first command to run any time a pipeline behaves strangely, because a nearly-full disk causes symptoms that look like almost anything else: writes hang, processes stall, jobs that used to take 5 minutes suddenly run for hours. Once you know a disk is nearly full, du ("disk usage") tells you what is taking the space:

du -sh /data/* | sort -rh | head -5

312G  /data/raw
 92G  /data/processed
 34G  /data/logs
  8G  /data/tmp
  2G  /data/pipelines

-s summarises each argument to one total instead of listing every file inside it, and piping through sort -rh (reverse, human-readable-numeric) puts the biggest consumer first. You can repeat this one level deeper on whichever directory turns out to be the culprit — that exact drill-down is exactly how the 7 AM incident later in this module gets solved.

The rwx Model — Reading and Fixing "Permission Denied"

A data engineer who does not understand permissions will spend hours debugging "Permission denied" errors that take seconds to fix once the model is understood. Every file and directory on Linux carries exactly this information:

ls -lah /data/pipelines/run_orders.sh

-rwxr-x--- 1 pipeline_user data_team 892 Mar 17 06:00 run_orders.sh

Break the permission string apart and it reads as four independent pieces:

Each of the three access groups (owner, group, others) carries the same three possible permissions: r (read — can view contents, or list a directory), w (write — can modify a file, or create files inside a directory), and x (execute — can run a file as a program, or enter a directory with cd). This file is rwx for the owner, r-x for the group, and nothing at all for everyone else.

Changing permissions with chmod

Each permission has a numeric value — r=4, w=2, x=1 — and you add them together per group. rwx is 4+2+1=7, r-x is 4+1=5, and chmod takes one digit per group in owner-group-others order:

chmod +x run_orders.sh        # add execute permission, keep everything else
chmod 600 db_credentials.env  # secrets: owner read/write, nobody else anything

Diagnosing a permission error, systematically

When you see Permission denied, resist the urge to guess — three commands tell you exactly what's wrong:

ls -lah /data/output/orders.parquet   # what permissions does the FILE have?
id                                    # what user and groups am I actually in?
stat /data/output/orders.parquet      # full detail: owner, group, exact mode

-rw-r----- 1 other_service data_team 4.2G Mar 17 06:12 orders.parquet
uid=1001(pipeline_user) gid=1002(data_team) groups=1002(data_team)

Reading this output: the file is owned by other_service, not pipeline_user — but pipeline_user is a member of the data_team group, and the group permission is r-- (read only, no write). That fully explains a "permission denied" on any attempt to write to this file — the fix is either to change the group permission (chmod g+w) or the ownership entirely:

chown pipeline_user:data_team orders.parquet     # change owner and group
chmod -R 750 /data/secrets/                      # recursively set 750 on a whole tree

grep — Searching Logs Without Opening Them

Linux text processing tools are the fastest way to investigate pipeline logs and answer quick questions without writing a line of Python. A data engineer who knows grep, awk, sed, and cut can diagnose most pipeline failures in minutes, straight from the terminal.

The basics

grep "ERROR" orders_20260317.log

2026-03-17 06:41:02 ERROR Connection to database timed out after 30s
2026-03-17 06:41:33 ERROR Retry 1/3 failed

The plainest possible use — every line containing the literal text ERROR. Three flags cover most of what you need day to day:

Context lines — seeing what happened around an error

A single matching line rarely tells the whole story. You almost always want to see what happened immediately before or after it too:

grep -B 2 -A 5 "ERROR" orders_20260317.log

2026-03-17 06:40:58 INFO  Attempting database connection...
2026-03-17 06:40:58 INFO  Connection pool: 5/5 in use
2026-03-17 06:41:02 ERROR Connection to database timed out after 30s
2026-03-17 06:41:02 INFO  Retry scheduled in 5s
2026-03-17 06:41:07 INFO  Retrying database connection...
2026-03-17 06:41:33 ERROR Retry 1/3 failed
2026-03-17 06:41:33 INFO  Retry scheduled in 10s

-B 2 shows 2 lines before each match, -A 5 shows 5 after — instantly turning one alarming line into the full story: the connection pool was maxed out right before the timeout. -C 5 is the shorthand for equal context on both sides.

Regex patterns for real log formats

grep -E "ERROR|CRITICAL" orders_20260317.log      # either word, extended regex
grep -E "order_id=[0-9]+" orders_20260317.log     # order_id followed by digits
grep -E "^2026-03-17 06:4" orders_20260317.log    # lines starting with this timestamp

-E turns on extended regular expressions, which is what lets | mean "or" and [0-9]+ mean "one or more digits." Without -E, grep's basic mode requires escaping these characters, which is easy to get wrong under pressure — reach for -E by default.

Searching many files, and scripting a check

grep -r "CRITICAL" /var/log/pipelines/           # search every file, recursively
grep -l "ERROR" /var/log/pipelines/*.log          # list which FILES contain it (not lines)

/var/log/pipelines/orders_20260317.log
/var/log/pipelines/inventory_20260317.log

grep -q (quiet — no output at all, just a pass/fail exit code) is what makes grep useful inside a script's own logic, not just for reading logs yourself:

if grep -q "CRITICAL" orders_20260317.log; then
    echo "Critical error found — alerting team"
fi

awk for Columns, sed for Find-and-Replace

Where grep finds lines, awk works with columns inside those lines — exactly what you need for a CSV or any consistently-delimited text.

awk — column extraction and calculation

awk -F',' '{print $3}' orders.csv | head -5

amount
24.99
89.50
12.00
156.75

-F',' sets the field delimiter to a comma, and $3 refers to the third column ($1, $2… and $0 for the whole line). awk becomes genuinely powerful once you add a condition or a running calculation:

awk -F',' 'NR>1 {sum += $3} END {print "Total:", sum}' orders.csv

Total: 48291.35

NR is the current line number, so NR>1 skips the header row; sum += $3 runs on every remaining line; END marks a block that runs once, after every line has been processed. This one-liner is doing the same thing a full Python script with a CSV reader and an accumulator variable would do — in a single terminal command.

awk -F',' 'NR>1 {counts[$4]++} END {for (s in counts) print s, counts[s]}' orders.csv

delivered 412
cancelled 18
pending 31

sed — find, replace, and delete in a text stream

sed ("stream editor") is built around one core operation: substitution.

sed 's/old-db-host/new-db-host/g' config.yaml

s/find/replace/g — substitute, and g means every occurrence per line, not just the first. Printed to the screen like this, sed changes nothing — it only shows you what the result would look like. Add -i to actually modify the file in place:

sed -i.bak 's/old-db-host/new-db-host/g' config.yaml

Two more sed patterns come up constantly around data files:

sed '1d' orders.csv | wc -l              # strip the header, then count remaining rows
sed -n '10,20p' pipeline.log              # print ONLY lines 10 through 20

cut — when you just need columns, nothing more

For simple column extraction with no calculation involved, cut reads more clearly than awk:

cut -d',' -f1,3 orders.csv | head -3

id,amount
1001,24.99
1002,89.50

sort, uniq, and Pipes — The Real Power of the Shell

Every tool covered so far does exactly one thing. The reason the Linux shell is genuinely powerful for a data engineer is not any single command — it's that | (pipe) sends the output of one command straight into the input of the next, letting you chain small, simple tools into something that would otherwise take a real script to write.

sort and uniq, individually

sort -t',' -k3 -rn orders.csv | head -3

4821,ST003,899.00,delivered
1052,ST001,650.25,delivered
3390,ST002,512.10,cancelled

-t',' sets the field separator, -k3 sorts by the 3rd field, -rn reverse-numeric — so this shows the highest-value orders first. uniq has one sharp edge worth knowing before you use it: it only removes adjacent duplicate lines, which is exactly why it is almost always used right after sort.

The pattern: cut | sort | uniq -c | sort -rn

This exact four-stage pipe is one of the highest-value one-liners in the entire module — it turns any column into a ranked frequency count:

cut -d',' -f4 orders.csv | sort | uniq -c | sort -rn

  412 delivered
   31 pending
   18 cancelled

Read left to right: cut pulls out just the status column, sort groups identical values next to each other (required before uniq can work), uniq -c collapses each run of duplicates into one line prefixed with its count, and the final sort -rn puts the highest count first. Four tools, each doing one job, composed into an answer that would otherwise mean opening the file in pandas.

awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -10   # top 10 IPs hitting a server

Processes, Signals, and Running Things in the Background

Data pipelines are processes. Understanding how Linux manages them lets you run a pipeline in the background, watch its resource usage, and kill a stuck job cleanly instead of guessing.

Finding and inspecting a running process

ps aux | grep orders_pipeline

pipeline 18734 98.2  4.1  python3 orders_pipeline.py --date 2026-03-17

ps aux lists every process; piping through grep narrows it to the one you care about. That 98.2 is the CPU percentage — worth watching, because a healthy pipeline batch job is usually well under 100%, and a number pinned there for a long stretch is a signal something is spinning instead of progressing. To see exactly how long it has actually been running:

ps -p 18734 -o pid,etime,pcpu,pmem,cmd

PID    ELAPSED  %CPU  %MEM  CMD
18734  02:14:32  98.2   4.1  python3 orders_pipeline.py

Two hours and fourteen minutes for a job that should finish in thirty is a real signal, not a coincidence — this is the exact reading that kicks off the diagnosis in the Real World section later in this module.

Killing a process — and why the signal you send matters

kill 18734              # SIGTERM — ask nicely
sleep 5
kill -0 18734 2>/dev/null && kill -9 18734    # still alive? force it

kill -0 is a neat trick worth knowing on its own — it sends no signal at all and only checks whether the process still exists, which is exactly what you need before deciding whether escalation to SIGKILL is actually necessary.

Running something that survives you logging out

nohup python3 pipeline.py > output.log 2>&1 &
echo $!

[1] 21044
21044

Four things are happening on that first line: nohup means the process keeps running even after you close the SSH session; > output.log redirects normal output to a file; 2>&1 sends error output to that same place instead of the screen; and the trailing & runs it in the background so your terminal is immediately free again. echo $! prints the process ID of that last background command — worth capturing immediately if you'll need to check on or kill it later.

scp, rsync, curl, and S3 — Choosing the Right Tool

Data engineering involves constant movement of files — from a source server to a data lake, between cloud regions, from an external partner's SFTP drop to a processing node. Four tools cover almost every case, and the right one depends entirely on what you're actually doing.

scp — simple and immediate

scp orders.csv user@pipeline-01:/data/landing/
scp -i ~/.ssh/pipeline_key.pem orders.csv ec2-user@54.1.2.3:/data/

rsync — the right choice for anything repeated or large

rsync only transfers files that have actually changed, which is the difference between a nightly sync taking 40 minutes versus 40 seconds once most of the data is already in place.

rsync -avz --dry-run /data/local/ user@server:/data/remote/    # preview first — nothing moves
rsync -avz /data/local/ user@server:/data/remote/               # then actually run it

-a is archive mode (recursive, preserves permissions and timestamps), -v is verbose, -z compresses during transfer. Always run with --dry-run once first on anything you haven't run before — it shows exactly what would be transferred, without transferring anything, which is a cheap safety check before a sync that touches thousands of files.

curl — for APIs and HTTP downloads

curl -H "Authorization: Bearer $API_TOKEN" https://api.example.com/data > data.json

Moving data to and from S3

aws s3 cp orders.csv s3://freshcart-data/raw/orders.csv
aws s3 sync /data/local/ s3://freshcart-data/processed/

cp moves one object; sync behaves like rsync for an entire prefix — only uploading what's new or changed.

Cron — Automating a Pipeline on a Schedule

For a pipeline that doesn't yet warrant a full orchestration tool like Airflow, cron is the fastest, most reliable way to schedule it. And even once you are using Airflow, its schedule strings use this exact same syntax — so cron is never wasted knowledge.

Reading cron syntax

A * means "any value" for that field. So this reads: minute 0, hour 6, any day of month, any month, any day of week — every day at 6:00 AM, exactly.

Editing and reading the crontab

crontab -e      # edit your crontab (opens in $EDITOR)
crontab -l      # list what's currently scheduled

Three habits every production crontab entry needs

# Fragile — will likely fail silently under cron:
0 6 * * * run_orders.sh

# Production-ready:
0 6 * * * /data/pipelines/run_orders.sh >> /var/log/pipelines/orders.log 2>&1

Three specific things changed: an absolute path to the script (cron does not use your normal shell's PATH — more on exactly why in Part 12), and output redirected to a real log file so a failure leaves a trace instead of vanishing. This exact gap — a script that runs fine by hand but fails silently under cron — is common enough that it gets its own dedicated explanation later in this module.

Debugging a cron job that "isn't running"

sudo systemctl status cron              # is the cron daemon even running?
grep CRON /var/log/syslog | tail -20     # what has cron actually attempted?

Variables, Conditionals, and Loops

Bash scripts wrap a pipeline with the operational logic Python alone doesn't handle well: checking preconditions, logging, alerting on failure, preventing duplicate runs. Every production pipeline is wrapped in at least a basic bash script — the rest of this module builds one, piece by piece.

Variables

name="FreshCart"
today=$(date +%Y-%m-%d)      # command substitution — capture a command's output
echo "Company: $name, today: $today"

Company: FreshCart, today: 2026-03-17

$(...) runs a command and substitutes its output — the single most-used piece of bash syntax in real pipeline scripts, since almost every script needs "today's date" or "the result of some check" captured into a variable.

Conditionals

if [[ -f "/data/orders.csv" ]]; then
    echo "File exists"
else
    echo "File is missing"
fi

One conditional worth knowing specifically: testing whether a command itself succeeded, which is how a script checks its own dependencies before doing real work.

if psql "$DATABASE_URL" -c "SELECT 1" > /dev/null 2>&1; then
    echo "Database is reachable"
else
    echo "Cannot reach database — aborting"
    exit 1
fi

Loops

stores=("ST001" "ST002" "ST003")
for store in "${stores[@]}"; do
    echo "Processing store: $store"
done

retry=0
max_retries=3
while [[ $retry -lt $max_retries ]]; do
    if python3 pipeline.py; then
        echo "Success on attempt $((retry+1))"
        break
    fi
    retry=$((retry+1))
    echo "Attempt $retry failed — retrying in $((2**retry))s"
    sleep $((2**retry))
done

$((2**retry)) is exponential backoff written directly in bash arithmetic — 2, 4, 8 seconds — the same pattern you'd reach for in Python, expressed in the shell.

Functions

check_disk_space() {
    local path="${1:-/data}"
    local min_gb="${2:-10}"
    local free_gb
    free_gb=$(df -BG "$path" | awk 'NR==2 {print $4}' | tr -d 'G')

    if [[ $free_gb -lt $min_gb ]]; then
        echo "ERROR: only ${free_gb}GB free at $path (need ${min_gb}GB)"
        return 1
    fi
    return 0
}

check_disk_space /data 10 || exit 1

local keeps a variable scoped to the function instead of leaking into the rest of the script — worth using by habit inside every function you write. Notice this function also reuses the exact df + awk combination from Part 02 — a small sign of how a handful of core tools recombine into everything else in this module.

Parsing Filenames and Computing Dates in Pure Bash

Pipeline scripts constantly need to pull a piece out of a filename, or compute "yesterday" for a backfill — bash can do both without calling out to Python.

filename="/data/orders_2026_03_17.csv"

echo "${filename##*/}"     # orders_2026_03_17.csv  — strip everything up to the last /
echo "${filename%.*}"      # /data/orders_2026_03_17 — strip the extension
echo "${filename##*.}"     # csv — just the extension

##*/ and %.* look cryptic at first but follow one rule: # strips from the front, % strips from the back, and doubling the symbol (##, %%) makes it greedy (match as much as possible) instead of matching the shortest possible piece.

today=$(date +%Y-%m-%d)
yesterday=$(date -d 'yesterday' +%Y-%m-%d)     # Linux
# yesterday=$(date -v-1d +%Y-%m-%d)            # macOS — different flag, same result
log_suffix=$(date +%Y%m%d_%H%M%S)

today: 2026-03-17
yesterday: 2026-03-16
log_suffix: 20260317_081432

Why a Script That Works By Hand Can Fail Under Cron

This is the single most common "it works on my machine" problem a data engineer hits, and it has one root cause: cron runs your script in aminimal environment, not the rich one your interactive shell sets up for you.

echo $PATH

# In your interactive shell:
/usr/local/bin:/usr/local/sbin:/usr/bin:/bin:/home/pipeline_user/.local/bin

# Under cron:
/usr/bin:/bin

That difference is the whole problem. If your script calls python3 and it happens to live in /usr/local/bin (very common), your interactive shell finds it instantly — but cron's minimal PATH does not include that directory at all, so the exact same script fails under cron with "command not found," even though it just ran perfectly when you tested it by hand two minutes earlier.

set -a              # auto-export every variable that gets set below
source /etc/pipeline_environment
set +a              # stop auto-exporting

This pattern — set -a, source the file, set +a — is the cleanest way to load a whole file of configuration into a script's environment without hand-writing an export line for every single variable in it.

Assembling Everything Into One Real Pipeline Wrapper

Every tool in this module so far has been a piece. Now they come together — the same FreshCart orders pipeline, wrapped in a real production-grade bash script, built up one addition at a time so each piece is understood before the next one lands on top of it.

Step 1 — the non-negotiable first line

#!/usr/bin/env bash
set -euo pipefail

This is the correct second line of every production bash script, without exception. -e exits immediately the moment any command fails, instead of bash's default of quietly continuing to the next line. -u turns a typo like $DATABSE_URL into an immediate, clear error instead of silently substituting an empty string. -o pipefail makes a pipe fail if any stage of it fails — without it, bad_command | good_command reports success as long as the last command in the chain succeeds, even if the first one silently produced nothing.

Step 2 — configuration and logging

readonly LOG_DIR="/var/log/pipelines"
readonly LOG_FILE="${LOG_DIR}/orders_$(date +%Y%m%d).log"
readonly PIPELINE_SCRIPT="/data/pipelines/pipeline/orders_ingestion.py"

log() {
    local level="$1"; shift
    echo "$(date '+%Y-%m-%d %H:%M:%S') [${level}] $*" | tee -a "$LOG_FILE"
}
info()  { log "INFO"  "$@"; }
error() { log "ERROR" "$@"; }

tee -a is doing double duty here — it prints the message to the screen and appends it to the log file at the same time, so you get live feedback when running the script by hand and a permanent record when cron runs it unattended. The info/error wrapper functions exist purely so the rest of the script reads as info "message" instead of repeating the full log call everywhere.

Step 3 — a lock file, so cron can never run two copies at once

readonly LOCK_FILE="/tmp/orders_pipeline.lock"

if [[ -f "$LOCK_FILE" ]]; then
    pid=$(cat "$LOCK_FILE")
    if kill -0 "$pid" 2>/dev/null; then
        error "Another instance is already running (PID $pid). Exiting."
        exit 1
    fi
    echo "Stale lock file found — removing"
fi
echo $$ > "$LOCK_FILE"

A pipeline that normally takes 30 minutes but occasionally runs long (from the earlier scenario: 2 hours) is a real risk if cron fires the next scheduled run before the first one finishes — now there are two copies writing to the same output. This lock file, checked with the same kill -0 trick from Part 07, makes that structurally impossible: a second invocation sees the lock, confirms the original process is genuinely still alive, and exits immediately instead of racing it.

Step 4 — cleanup that always runs, success or failure

cleanup() {
    local exit_code=$?
    rm -f "$LOCK_FILE"
    if [[ $exit_code -ne 0 ]]; then
        error "Script exited with code $exit_code"
    fi
}
trap cleanup EXIT

trap cleanup EXIT registers cleanup to run automatically no matter how the script ends — a normal finish, an exit 1 from an earlier check, or a crash from set -e catching a failed command. This is what guarantees the lock file from Step 3 is always removed — without it, one failed run would permanently lock out every future run.

Step 5 — preconditions, then the actual pipeline

main() {
    info "==== Starting orders pipeline ===="

    : "${DATABASE_URL:?DATABASE_URL is required}"
    [[ -f "$PIPELINE_SCRIPT" ]] || { error "Script not found: $PIPELINE_SCRIPT"; exit 1; }
    check_disk_space /data 10 || { error "Insufficient disk space"; exit 1; }

    local run_date="${1:-$(date -d 'yesterday' +%Y-%m-%d)}"
    info "Processing date: $run_date"

    python3 "$PIPELINE_SCRIPT" --date "$run_date" 2>&1 | tee -a "$LOG_FILE"

    info "==== Finished orders pipeline ===="
}

main "$@"

: "${DATABASE_URL:?DATABASE_URL is required}" is a bash idiom worth learning once: : is a no-op that does nothing with its argument, and ${VAR:?message} makes bash exit with that exact message if the variable is unset — a one-line, readable precondition check. Notice this step also calls check_disk_space, the exact function written in Part 10 — the whole script is built from pieces this module already taught, not new syntax appearing out of nowhere.

Diagnosing a Failed Pipeline at 7 AM Using Only Linux Commands

ps aux | grep orders_pipeline

pipeline 18734 98.2  4.1  python3 orders_pipeline.py --date 2026-03-17

ps -p 18734 -o pid,etime,pcpu,pmem,cmd

PID    ELAPSED  %CPU  %MEM  CMD
18734  02:14:32  98.2   4.1  python3 orders_pipeline.py

df -h /data

Filesystem  Size  Used  Avail  Use%  Mounted on
/dev/sdb1   500G  499G   512M   99%  /data

du -sh /data/raw/2026/03/* | sort -rh

288G  /data/raw/2026/03/17
 24G  /data/raw/2026/03/16

ls -lth /data/raw/2026/03/17/ | head -5

-rw-r--r-- 1 pipeline pipeline 288G Mar 17 06:28 orders_debug_dump.csv

grep -i debug orders_20260317.log

2026-03-17 04:32:14 WARNING DEBUG_MODE=true detected — writing full row dump

kill 18734                              # SIGTERM first
sleep 5
kill -0 18734 2>/dev/null && kill -9 18734

rm /data/raw/2026/03/17/orders_debug_dump.csv
sed -i 's/DEBUG_MODE=true/DEBUG_MODE=false/' /etc/pipelines/orders.env

nohup python3 /data/pipelines/pipeline/orders_ingestion.py --date 2026-03-17 \
    >> orders_20260317.log 2>&1 &
echo "Restarted with PID $!"

tail -f orders_20260317.log | grep -E "INFO|ERROR"

07:03:41 INFO Batch 1 complete: 10000 rows
07:04:28 INFO Batch 2 complete: 10000 rows

Five Misconceptions About Linux and Shell Scripting

5 Interview Questions — With Complete Answers

Mistakes Beginners Make Constantly

Errors You Will Hit — And Exactly Why They Happen