File Systems and I/O

Files, directories, and inodes, allocation methods, journaling, buffering and caching, and disk scheduling.

Est. 35 min Updated Suggest a fix
// Part 01 — Files and names

A Directory Name Is Not the File Itself

A file system organizes persistent data into named objects and directories. A pathname is resolved component by component: directory entries connect names to filesystem objects. On Unix-like filesystems, an inode stores metadata such as ownership, permissions, timestamps, size, and references to data blocks; the filename-to-inode association lives in a directory entry. Hard links are additional names for an inode, while symbolic links store a path-like target.

Directory trees are a naming convention over filesystem objects. Mounting attaches a filesystem at a directory in a larger namespace. Exact metadata and allocation details differ across filesystem types and operating systems.

ConceptStores / representsImportant distinction
Directory entryName and object referenceName lookup, not file contents
Inode (Unix-like)Metadata and data-block referencesUsually not the filename itself
Hard linkAnother name for the same inodeSame filesystem; links share contents
Symbolic linkA path targetCan cross filesystems; target can disappear

File allocation strategies describe how a filesystem lays out file blocks. Contiguous allocation stores a file in adjacent blocks, enabling simple sequential and direct access but making growth and external fragmentation challenging. Linked allocation chains blocks and grows easily, but random access may require following links. Indexed allocation keeps block pointers in an index structure, supporting direct lookup with metadata overhead. Modern filesystems often use extents, trees, and copy-on-write techniques rather than these textbook schemes in isolation.

Textbook allocation models explain trade-offs; real filesystem layouts are more sophisticated.
MethodStrengthLimitation
ContiguousFast sequential access and simple offsetsGrowth can require relocation or larger extents
LinkedEasy to append blocksPoor random access and pointer overhead
IndexedDirect block lookup through indexIndex metadata consumes space
Extent-basedEfficient runs of adjacent blocksFragmentation and metadata still need management
💡 Note
A file descriptor is a process-local handle returned by opening a file-like resource; it is not the inode number or pathname.
// Part 02 — File operations

Open, Read, Write, and Close

Opening a path asks the OS to resolve names and check access rules. The process receives a descriptor referring to an open-file description, which tracks state such as the current offset and status flags. Descriptors may be inherited across process creation or deliberately duplicated. Several descriptors can refer to related open state, so sharing and offsets should be understood explicitly.

copy_file.c
c
#include <fcntl.h>
#include <unistd.h>
#include <stdio.h>

int main(int argc, char **argv) {
    if (argc != 3) { fprintf(stderr, "usage: copy SRC DST
"); return 2; }
    int in = open(argv[1], O_RDONLY);
    if (in < 0) { perror("open source"); return 1; }
    int out = open(argv[2], O_WRONLY | O_CREAT | O_TRUNC, 0644);
    if (out < 0) { perror("open destination"); close(in); return 1; }
    char buf[4096];
    ssize_t n;
    while ((n = read(in, buf, sizeof buf)) > 0) {
        ssize_t sent = 0;
        while (sent < n) {
            ssize_t w = write(out, buf + sent, (size_t)(n - sent));
            if (w <= 0) { perror("write"); close(in); close(out); return 1; }
            sent += w;
        }
    }
    if (n < 0) perror("read");
    close(in); close(out);
    return n < 0 ? 1 : 0;
}

This example handles partial writes and common errors, but a production copy tool must also consider metadata, symlinks, permissions, interruption, durability requirements, and safely choosing a destination.

// Part 03 — Consistency

Journaling Helps Recover After a Crash

Updating a file can require several metadata and data writes. A crash between those writes may leave an inconsistent filesystem. A journal records intended changes or transaction information so recovery can replay or roll back operations after a crash. Many journaling filesystems primarily journal metadata; journaling does not necessarily mean every recent user-data byte is durable.

Buffered writes may first land in memory and reach storage later. APIs such as fsync request durability for file data and relevant metadata, but exact guarantees depend on filesystem, device, mount options, and correct ordering (including parent-directory updates for some rename/create patterns). Durability is a contract to verify for the platform, not something inferred from a successful buffered write.

⚠️ Important
Journaling improves structural recovery; it is not a backup, version history, or protection against accidental deletion and every hardware failure.
// Part 04 — I/O path

Buffering and Caching Reduce Expensive Trips

Buffering groups small operations to reduce call or device overhead; caching retains data to serve future accesses faster. The page cache keeps file data in memory on many systems. A cache hit can make a read appear fast even though the same data would need device I/O after eviction or a cold start. A user-space stdio buffer is another layer and is distinct from the kernel page cache.

Sequential access benefits from read-ahead; write-back can defer writes. These improve average performance but create flush and durability considerations. Applications should choose flush and sync behavior based on their correctness needs rather than calling expensive sync operations blindly for every byte.

bash
# Inspect filesystem space and mounted filesystems
df -h
findmnt

# Inspect a file's inode and metadata on Linux
stat notes.txt
ls -li notes.txt

# Trace a short file operation when strace is available
strace -e trace=openat,read,write,close cat notes.txt
// Part 05 — Devices

DMA Lets Devices Move Data Efficiently

For I/O, a device driver programs a controller and coordinates completion with the kernel. Direct Memory Access (DMA) allows a capable device to transfer data between a device and memory with less CPU copying for each byte. The CPU still configures and manages the operation; DMA does not mean the device can access arbitrary memory without controls. Interrupts or polling can notify the CPU when work completes.

A typical buffered read goes from storage through a controller/DMA path into kernel-managed memory, then is copied or mapped for the application depending on the API and path. Real implementations include caches, mapping constraints, and zero-copy alternatives. The main lesson: application read/write calls describe intent, while driver and device details are handled below them.

TermPurposeTrade-off to remember
BufferingCombine or stage I/OData may not yet be durable
Page cacheReuse recently read/written file dataMemory pressure and eviction affect latency
DMATransfer blocks with less per-byte CPU workStill requires safe setup and completion handling
InterruptNotify CPU about device eventHigh rates can add overhead
📌 Real World Example
A dashboard reading a recently accessed configuration file may hit the page cache. A cold read may wait on storage. Measuring only the warm run overstates cold-start performance.

For rotating disks, request ordering can affect seek distance. FCFS follows arrival order; shortest-seek-time-first chooses a nearby request but may starve distant requests; SCAN sweeps in one direction then reverses, while C-SCAN returns to one end before sweeping again. These textbook policies are less directly relevant to flash devices, but they teach how device geometry and fairness influence I/O scheduling.

Classic rotating-disk scheduling models; real device queues and SSDs change the costs.
PolicyRequest choiceTrade-off
FCFSArrival orderFair order, potentially long movement
SSTFNearest pending trackGood local movement; starvation risk
SCANSweep then reverseMore regular service across the disk
C-SCANOne-direction sweep then wrapMore uniform waiting under the model
// Part 06 — Operational practice

Measure the I/O Your Workload Actually Performs

Linux exposes tools for observation, not perfect explanations. iostat (from the sysstat package where installed), vmstat, and process-level tools can help identify utilization, queueing, or memory effects. Avoid assuming that a high I/O count means a slow disk: small random reads, writeback, cache misses, and application serialization can all contribute.

safe_write.py
python
from pathlib import Path

path = Path("result.txt")
with path.open("w", encoding="utf-8") as output:
    output.write("completed
")
    output.flush()  # flush Python's user-space buffer to the OS
    # For durability-sensitive work, an OS-level sync request is also needed.

Python's flush() drains its own buffer to the OS; it does not alone promise persistence on physical media. When durability matters, use the platform's sync interface and confirm the filesystem/device guarantees required by the application.

  1. Identify the API layer: language buffer, kernel, filesystem, or device.
  2. Decide whether you need throughput, low latency, atomicity, or durability.
  3. Trace or measure representative warm and cold workloads.
  4. Document what the measurement tool observes and which layers it cannot see.
// Part 07 — Pitfalls

Answer with Precision

Giving a memorized slogan without assumptions

Why: A scheduling or replacement answer changes with the workload and model.

Fix: State arrivals, frame count, preemption, and tie-breaking before calculating.

Confusing two neighboring concepts

Why: Similar terms describe different layers or guarantees.

Fix: Define each concept and provide one contrast before adding detail.

Treating an observation as a universal guarantee

Why: A command samples one system at one moment.

Fix: Describe the observation limits and what additional evidence is needed.

Ignoring trade-offs and failure modes

Why: An idealized definition omits the engineering constraints.

Fix: Name one cost, edge case, or recovery path for every mechanism.

// Part 08 — Practice

Apply It to a Small System

Practice

Trace, calculate, and explain

  1. Draw the state, resource, or address transitions for a realistic workload, including a wait or failure path.
  2. Run a relevant Linux observation command from this lesson and explain what it can and cannot prove.
  3. Implement the small C or Python exercise, change one input, and predict the result before executing it.
  4. Explain one design choice with a precise invariant, a concrete example, and a trade-off.

Hint: Write assumptions first, then use a trace or calculation to support your explanation.

🎯 Key Takeaways

  • ✓On Unix-like systems, directory entries map names to inode-like objects; the inode is not generally the filename.
  • ✓File descriptors are process-local handles to open-file state, not pathnames or inode numbers.
  • ✓Journaling can restore filesystem consistency but does not guarantee every recent data write is durable.
  • ✓Buffering, page caching, and DMA improve I/O efficiency at different layers and have distinct effects.
  • ✓Measure warm and cold workloads, and match sync behavior to explicit durability requirements.

Next

What comes next

HDDs versus SSDs, disk scheduling algorithms, RAID levels and their trade-offs, and how storage performance affects applications.

Lesson 08 · Disk Scheduling, RAID and Storage Devices →

HELP US IMPROVE

How was this lesson?

Your note is shared with the Chaduvuko team for review. Please don’t include personal or sensitive information.

Choose your feedback

Spotted a mistake or something unclear? Suggest a fix on GitHub or email us. Select the text first and it is quoted for you.

Share

Discussion

0

Have a better approach? Found something outdated? Share it — your knowledge helps everyone learning here.

Continue with GitHub
Loading...