File Systems and I/O
Files, directories, and inodes, allocation methods, journaling, buffering and caching, and disk scheduling.
A Directory Name Is Not the File Itself
A file system organizes persistent data into named objects and directories. A pathname is resolved component by component: directory entries connect names to filesystem objects. On Unix-like filesystems, an inode stores metadata such as ownership, permissions, timestamps, size, and references to data blocks; the filename-to-inode association lives in a directory entry. Hard links are additional names for an inode, while symbolic links store a path-like target.
Directory trees are a naming convention over filesystem objects. Mounting attaches a filesystem at a directory in a larger namespace. Exact metadata and allocation details differ across filesystem types and operating systems.
| Concept | Stores / represents | Important distinction |
|---|---|---|
| Directory entry | Name and object reference | Name lookup, not file contents |
| Inode (Unix-like) | Metadata and data-block references | Usually not the filename itself |
| Hard link | Another name for the same inode | Same filesystem; links share contents |
| Symbolic link | A path target | Can cross filesystems; target can disappear |
File allocation strategies describe how a filesystem lays out file blocks. Contiguous allocation stores a file in adjacent blocks, enabling simple sequential and direct access but making growth and external fragmentation challenging. Linked allocation chains blocks and grows easily, but random access may require following links. Indexed allocation keeps block pointers in an index structure, supporting direct lookup with metadata overhead. Modern filesystems often use extents, trees, and copy-on-write techniques rather than these textbook schemes in isolation.
| Method | Strength | Limitation |
|---|---|---|
| Contiguous | Fast sequential access and simple offsets | Growth can require relocation or larger extents |
| Linked | Easy to append blocks | Poor random access and pointer overhead |
| Indexed | Direct block lookup through index | Index metadata consumes space |
| Extent-based | Efficient runs of adjacent blocks | Fragmentation and metadata still need management |
Open, Read, Write, and Close
Opening a path asks the OS to resolve names and check access rules. The process receives a descriptor referring to an open-file description, which tracks state such as the current offset and status flags. Descriptors may be inherited across process creation or deliberately duplicated. Several descriptors can refer to related open state, so sharing and offsets should be understood explicitly.
#include <fcntl.h>
#include <unistd.h>
#include <stdio.h>
int main(int argc, char **argv) {
if (argc != 3) { fprintf(stderr, "usage: copy SRC DST
"); return 2; }
int in = open(argv[1], O_RDONLY);
if (in < 0) { perror("open source"); return 1; }
int out = open(argv[2], O_WRONLY | O_CREAT | O_TRUNC, 0644);
if (out < 0) { perror("open destination"); close(in); return 1; }
char buf[4096];
ssize_t n;
while ((n = read(in, buf, sizeof buf)) > 0) {
ssize_t sent = 0;
while (sent < n) {
ssize_t w = write(out, buf + sent, (size_t)(n - sent));
if (w <= 0) { perror("write"); close(in); close(out); return 1; }
sent += w;
}
}
if (n < 0) perror("read");
close(in); close(out);
return n < 0 ? 1 : 0;
}This example handles partial writes and common errors, but a production copy tool must also consider metadata, symlinks, permissions, interruption, durability requirements, and safely choosing a destination.
Journaling Helps Recover After a Crash
Updating a file can require several metadata and data writes. A crash between those writes may leave an inconsistent filesystem. A journal records intended changes or transaction information so recovery can replay or roll back operations after a crash. Many journaling filesystems primarily journal metadata; journaling does not necessarily mean every recent user-data byte is durable.
Buffered writes may first land in memory and reach storage later. APIs such as fsync request durability for file data and relevant metadata, but exact guarantees depend on filesystem, device, mount options, and correct ordering (including parent-directory updates for some rename/create patterns). Durability is a contract to verify for the platform, not something inferred from a successful buffered write.
Buffering and Caching Reduce Expensive Trips
Buffering groups small operations to reduce call or device overhead; caching retains data to serve future accesses faster. The page cache keeps file data in memory on many systems. A cache hit can make a read appear fast even though the same data would need device I/O after eviction or a cold start. A user-space stdio buffer is another layer and is distinct from the kernel page cache.
Sequential access benefits from read-ahead; write-back can defer writes. These improve average performance but create flush and durability considerations. Applications should choose flush and sync behavior based on their correctness needs rather than calling expensive sync operations blindly for every byte.
# Inspect filesystem space and mounted filesystems
df -h
findmnt
# Inspect a file's inode and metadata on Linux
stat notes.txt
ls -li notes.txt
# Trace a short file operation when strace is available
strace -e trace=openat,read,write,close cat notes.txtDMA Lets Devices Move Data Efficiently
For I/O, a device driver programs a controller and coordinates completion with the kernel. Direct Memory Access (DMA) allows a capable device to transfer data between a device and memory with less CPU copying for each byte. The CPU still configures and manages the operation; DMA does not mean the device can access arbitrary memory without controls. Interrupts or polling can notify the CPU when work completes.
A typical buffered read goes from storage through a controller/DMA path into kernel-managed memory, then is copied or mapped for the application depending on the API and path. Real implementations include caches, mapping constraints, and zero-copy alternatives. The main lesson: application read/write calls describe intent, while driver and device details are handled below them.
| Term | Purpose | Trade-off to remember |
|---|---|---|
| Buffering | Combine or stage I/O | Data may not yet be durable |
| Page cache | Reuse recently read/written file data | Memory pressure and eviction affect latency |
| DMA | Transfer blocks with less per-byte CPU work | Still requires safe setup and completion handling |
| Interrupt | Notify CPU about device event | High rates can add overhead |
For rotating disks, request ordering can affect seek distance. FCFS follows arrival order; shortest-seek-time-first chooses a nearby request but may starve distant requests; SCAN sweeps in one direction then reverses, while C-SCAN returns to one end before sweeping again. These textbook policies are less directly relevant to flash devices, but they teach how device geometry and fairness influence I/O scheduling.
| Policy | Request choice | Trade-off |
|---|---|---|
| FCFS | Arrival order | Fair order, potentially long movement |
| SSTF | Nearest pending track | Good local movement; starvation risk |
| SCAN | Sweep then reverse | More regular service across the disk |
| C-SCAN | One-direction sweep then wrap | More uniform waiting under the model |
Measure the I/O Your Workload Actually Performs
Linux exposes tools for observation, not perfect explanations. iostat (from the sysstat package where installed), vmstat, and process-level tools can help identify utilization, queueing, or memory effects. Avoid assuming that a high I/O count means a slow disk: small random reads, writeback, cache misses, and application serialization can all contribute.
from pathlib import Path
path = Path("result.txt")
with path.open("w", encoding="utf-8") as output:
output.write("completed
")
output.flush() # flush Python's user-space buffer to the OS
# For durability-sensitive work, an OS-level sync request is also needed.Python's flush() drains its own buffer to the OS; it does not alone promise persistence on physical media. When durability matters, use the platform's sync interface and confirm the filesystem/device guarantees required by the application.
- Identify the API layer: language buffer, kernel, filesystem, or device.
- Decide whether you need throughput, low latency, atomicity, or durability.
- Trace or measure representative warm and cold workloads.
- Document what the measurement tool observes and which layers it cannot see.
Answer with Precision
Giving a memorized slogan without assumptions
Why: A scheduling or replacement answer changes with the workload and model.
Fix: State arrivals, frame count, preemption, and tie-breaking before calculating.
Confusing two neighboring concepts
Why: Similar terms describe different layers or guarantees.
Fix: Define each concept and provide one contrast before adding detail.
Treating an observation as a universal guarantee
Why: A command samples one system at one moment.
Fix: Describe the observation limits and what additional evidence is needed.
Ignoring trade-offs and failure modes
Why: An idealized definition omits the engineering constraints.
Fix: Name one cost, edge case, or recovery path for every mechanism.
Apply It to a Small System
Practice
Trace, calculate, and explain
- Draw the state, resource, or address transitions for a realistic workload, including a wait or failure path.
- Run a relevant Linux observation command from this lesson and explain what it can and cannot prove.
- Implement the small C or Python exercise, change one input, and predict the result before executing it.
- Explain one design choice with a precise invariant, a concrete example, and a trade-off.
Hint: Write assumptions first, then use a trace or calculation to support your explanation.
🎯 Key Takeaways
- ✓On Unix-like systems, directory entries map names to inode-like objects; the inode is not generally the filename.
- ✓File descriptors are process-local handles to open-file state, not pathnames or inode numbers.
- ✓Journaling can restore filesystem consistency but does not guarantee every recent data write is durable.
- ✓Buffering, page caching, and DMA improve I/O efficiency at different layers and have distinct effects.
- ✓Measure warm and cold workloads, and match sync behavior to explicit durability requirements.
Next
What comes next
HDDs versus SSDs, disk scheduling algorithms, RAID levels and their trade-offs, and how storage performance affects applications.
Lesson 08 · Disk Scheduling, RAID and Storage Devices →HELP US IMPROVE
How was this lesson?
Your note is shared with the Chaduvuko team for review. Please don’t include personal or sensitive information.
Spotted a mistake or something unclear? Suggest a fix on GitHub or email us. Select the text first and it is quoted for you.
Discussion
0Have a better approach? Found something outdated? Share it — your knowledge helps everyone learning here.
