How Linux Works

Goal: Understand the core abstractions that make Linux an operating system — processes, files, permissions, and services — so you can navigate and debug an embedded system.

Related Tutorials

For hands-on practice, see: Processes and IPC | Threads and Synchronization | SSH Login | Shell Scripting | ELF and Program Execution | Debugging Practices

---

You SSH into your Raspberry Pi. You run ls /dev and see 200 entries. You run ps aux and see 80 processes. You try to read a sensor and get "Permission denied."

On a microcontroller, there is no filesystem, no processes, no permissions — just your code running on bare metal. Linux adds all of these layers. Before you can write drivers or build custom images, you need to understand what these layers are and why they exist.

---

1. Everything Is a Process

A process is a running program with its own memory space, file descriptors, and permissions.

Concept	Description	Command
PID	Every process has a unique ID	ps aux
Parent/child	Every process is started by another (except PID 1)	pstree
Foreground/background	Terminal processes vs daemons	jobs, &, fg
Signals	How processes communicate (kill, stop, continue)	kill -l

Every process is in one of these states at any moment:

State	Symbol	Meaning
Running	R	Actively executing on a CPU or waiting in the run queue
Sleeping	S / D	Waiting for an event: S = interruptible (can be killed), D = uninterruptible (waiting for I/O, cannot be killed)
Stopped	T	Paused by a signal (SIGSTOP or Ctrl+Z), can be resumed with fg or SIGCONT
Zombie	Z	Process has finished but its parent has not yet read its exit status — the entry persists in the process table until wait() is called

# See all processes
ps aux

# Tree view — who started what
pstree -p

# Interactive view with CPU/memory usage
htop

Why This Matters for Embedded

On a bare-metal microcontroller, you have one program. With an RTOS, you have multiple tasks but they share one address space. On Linux, you have dozens of isolated processes competing for CPU, memory, and I/O. Understanding this is essential for:

• Debugging "why is my sensor read slow?" (another process is hogging CPU)
• Writing drivers that work alongside other kernel code
• Designing appliances where exactly the right services run at boot

1.1 How a Program Becomes a Process

You type ./sensor_reader and press Enter. What happens between that keystroke and main()?

graph TD
    A[Shell] -->|fork| B[Child Shell]
    B -->|execve| C[Kernel: check ELF]
    C -->|mmap segments| D[Load into memory]
    D -->|invoke| E["ld.so (dynamic linker)"]
    E -->|resolve symbols| F["_start → __libc_start_main"]
    F --> G["main()"]

    style C fill:#4CAF50,color:#fff
    style E fill:#FF9800,color:#fff

1. The shell calls fork() — creates a copy of itself (the child process).
2. The child calls execve("./sensor_reader", ...) — asks the kernel to replace itself with the new program.
3. The kernel checks the file's ELF magic bytes (7f 45 4c 46) to confirm it is a valid executable.
4. The kernel maps the ELF segments (code, data) into the process's virtual address space via mmap.
5. If the binary is dynamically linked, the kernel invokes the dynamic linker (ld-linux-armhf.so.3) which loads shared libraries and resolves symbols.
6. Control jumps to _start → __libc_start_main → your main().

See it live with strace — the fastest way to answer "what files did it open?" and "which system call failed?" without modifying the program or even having its source code:

# Trace fork and exec across processes
strace -f -e trace=clone,execve ./sensor_reader

# What files does a program open?
strace -e trace=openat ./sensor_reader 2>&1 | grep -v ENOENT

# Which call failed? (look for return value -1)
strace ./sensor_reader 2>&1 | grep "= -1"

Aspect	MCU (bare-metal)	Linux
Entry point	Reset vector → main() directly	_start → dynamic linker → main()
Code loading	Burned into flash at compile time	Kernel maps ELF segments via mmap at runtime
Libraries	Statically linked (or none)	Dynamically linked (.so files loaded at runtime)
Address space	Physical addresses, shared	Virtual addresses, private per process

After fork(), the child gets a copy of the parent's memory and file descriptors (FD 0, 1, 2). Both processes then run independently — the child typically calls execve() to replace itself with a new program.

Tip

Why fork + exec instead of a single "spawn" call? Because the shell needs to set up redirects, pipes, and environment variables in the child before the new program replaces it. This two-step pattern is fundamental to Unix.

1.2 ELF Format and Dynamic Linking

Every binary, shared library, and kernel module on Linux is an ELF (Executable and Linkable Format) file.

Key ELF sections:

Section	Contents	Analogy
.text	Machine code (your compiled functions)	The program instructions in flash
.data	Initialized global variables	RAM with initial values
.bss	Zero-initialized globals (not stored in file)	Zeroed RAM at startup
.rodata	Read-only data (string constants, lookup tables)	const data in flash
.dynamic	Dynamic linking information	N/A on MCU (no dynamic linking)

Inspection commands:

# What kind of file is it?
file ./sensor_reader

# ELF header — architecture, entry point, type
readelf -h ./sensor_reader

# Section table — all sections with sizes
readelf -S ./sensor_reader

# Which shared libraries does it need?
ldd ./sensor_reader

# Disassemble — see the generated ARM instructions
objdump -d ./sensor_reader | head -60

Runtime memory layout — see where everything is loaded:

# For a running process
cat /proc/self/maps

Note

Why this matters for embedded: Buildroot/BusyBox uses static linking to create tiny binaries with no .so dependencies. Understanding static vs dynamic linking explains why a Buildroot ls is 800 KB but a Debian ls is 130 KB (plus 2 MB of shared libraries). It also explains "Exec format error" (wrong architecture) and missing .so failures.

1.3 Inter-Process Communication (IPC)

On a microcontroller, your single program communicates between functions via global variables and ISR flags. On Linux, processes are isolated — they have separate address spaces and cannot see each other's memory. They need explicit mechanisms to share data.

Mechanism	Direction	Data Type	Typical Use
Pipes / FIFOs	One-way	Byte stream	Shell pipelines, parent-child communication
Signals	One-way	Notification only	Process control (SIGTERM, SIGKILL, SIGINT)
Shared memory	Bidirectional	Raw memory region	High-speed data sharing (sensor buffers)
Unix sockets	Bidirectional	Byte stream / datagrams	D-Bus, systemd, local services
TCP/UDP sockets	Bidirectional	Byte stream / datagrams	Network communication (MQTT, HTTP)
Message queues	Bidirectional	Discrete messages	Task queues with priority ordering
D-Bus	Bidirectional	Typed messages	Desktop/system service communication

MCU comparison — what replaces what:

MCU Pattern	Linux Equivalent
Global variable shared between tasks	Shared memory (with synchronization)
ISR flag / event bit	Signal (SIGUSR1) or eventfd
UART between two chips	TCP/UDP socket
FreeRTOS queue	POSIX message queue or pipe

Practical guidance: Start with pipes (simplest, built into the shell). Use Unix sockets for local service communication. Use TCP/UDP sockets for network communication (covered in Networking and Security). Use shared memory only when performance requires it — it needs careful synchronization to avoid race conditions.

Tip

inotify is a Linux-specific mechanism for monitoring filesystem changes — for example, watching a config file for modifications. It is not traditional IPC but is useful for loose coupling between services that communicate via files.

1.3.1 Pipes in C

The shell pipe operator | uses the pipe() system call underneath. A pipe is a one-way byte stream with two file descriptors: one for writing, one for reading.

int fd[2];
pipe(fd);           /* fd[0] = read end, fd[1] = write end */
pid_t pid = fork();

if (pid == 0) {
    /* Child: write to pipe */
    close(fd[0]);
    write(fd[1], "26750", 5);
    close(fd[1]);
    _exit(0);
} else {
    /* Parent: read from pipe */
    close(fd[1]);
    char buf[64];
    int n = read(fd[0], buf, sizeof(buf) - 1);
    buf[n] = '\0';
    printf("Received: %s\n", buf);
    close(fd[0]);
    waitpid(pid, NULL, 0);
}

 Child process           Kernel              Parent process
 ┌────────────┐     ┌──────────────┐     ┌────────────┐
 │ write(fd[1])├────►│  pipe buffer ├────►│ read(fd[0])│
 └────────────┘     │  (64 KB)     │     └────────────┘
                    └──────────────┘

This is the same mechanism the shell uses for dmesg | grep spi — the shell creates a pipe, forks twice, and connects stdout of dmesg to stdin of grep via the pipe file descriptors (see Section 7).

1.3.2 Signals

A signal is a software interrupt delivered to a process. The kernel or another process sends a signal number, and the receiving process either handles it, ignores it, or dies.

Signal	Number	Default	Catchable?	Embedded Use Case
SIGINT	2	Terminate	Yes	Ctrl+C — graceful stop
SIGTERM	15	Terminate	Yes	systemctl stop — clean shutdown
SIGKILL	9	Terminate	No	Last resort — force kill
SIGUSR1	10	Terminate	Yes	Application-defined: reload config
SIGCHLD	17	Ignore	Yes	Child exited — parent calls wait()
SIGSTOP	19	Stop	No	Freeze process (debugger)
SIGCONT	18	Continue	Yes	Resume stopped process

Register a handler with sigaction() (not the deprecated signal() function):

volatile sig_atomic_t got_signal = 0;

void handler(int sig) {
    got_signal = 1;     /* Only set a flag — no printf here! */
}

struct sigaction sa = { .sa_handler = handler };
sigaction(SIGUSR1, &sa, NULL);

/* Main loop checks the flag */
while (1) {
    if (got_signal) {
        got_signal = 0;
        printf("Received SIGUSR1 — reloading config\n");
    }
    /* ... do work ... */
}

Note

Signal handlers are the user-space equivalent of ISRs. Both must be fast — set a flag, return. Just as you keep MCU ISR code minimal and defer work to the main loop, signal handlers should only set volatile sig_atomic_t flags. This pattern reappears in kernel interrupt handlers (Lab 7).

Warning

Async-signal-safety: Only a small set of functions are safe to call inside a signal handler (write(), _exit(), signal()). Calling printf(), malloc(), or any function that takes locks can cause deadlocks or corruption. The safe pattern is always: set a flag in the handler, act on it in the main loop.

1.3.3 Choosing an IPC Mechanism

Start simple, escalate only when needed:

Need	Mechanism	Complexity
Parent <-> child data stream	pipe()	Low
Unrelated processes, one-way	Named pipe (FIFO)	Low
Notify a process (no data)	Signal (kill())	Low
Local service communication	Unix socket	Medium
Network communication	TCP/UDP socket	Medium
High-speed shared buffer	shm_open() + mmap()	High (needs synchronization)

Rule of thumb: Use pipes for parent-child data flow. Use signals for notifications. Use Unix sockets when you need bidirectional communication between unrelated processes. Use shared memory only when latency measurements prove it's necessary.

Tip

For a hands-on walkthrough of pipes, signals, and shared memory in C, see Tutorial: Processes and IPC in C. For threads, mutexes, condition variables, semaphores, and atomics, see Tutorial: Threads and Synchronization.

1.4 Processes vs Threads

Threads share the same address space and file descriptors — cheaper to create than processes but with no memory isolation between them. Linux implements threads as lightweight processes (clone() with shared memory flags).

Aspect	Process	Thread
Memory	Separate address space (full isolation)	Shared address space (no isolation)
File descriptors	Separate copy after fork()	Shared with all threads in the process
Creation cost	High (copy page tables, FD table)	Low (new stack + registers only)
Crash impact	Only that process dies	Entire process (all threads) dies
Linux implementation	fork() → separate mm_struct	clone() with CLONE_VM → shared mm_struct

MCU tie-in: RTOS tasks are similar to threads — they share one address space and have no memory isolation. A pointer bug in one task can corrupt another. Linux processes give full isolation at higher cost.

1.4.1 fork() and wait()

fork() creates a near-exact copy of the calling process. The return value tells you which copy you are:

pid_t pid = fork();

if (pid == 0) {
    /* Child process — do work */
    printf("Child PID: %d, parent: %d\n", getpid(), getppid());
    _exit(0);
} else if (pid > 0) {
    /* Parent process — wait for child */
    int status;
    waitpid(pid, &status, 0);
    printf("Child exited with status %d\n", WEXITSTATUS(status));
} else {
    perror("fork failed");
}

Return value	Meaning
0	You are the child
> 0	You are the parent (value = child's PID)
-1	fork() failed (out of memory or process limit)

Zombie processes: If a child exits but the parent never calls wait() or waitpid(), the child becomes a zombie (Z state in ps). The kernel keeps its exit status in the process table, waiting for the parent to collect it. Zombies consume a PID and a process table slot — in an embedded system with limited PIDs, this matters. Always call waitpid() or handle SIGCHLD.

1.4.2 POSIX Threads (pthreads)

Threads share the same address space — cheaper than processes but with shared-memory risks:

#include <pthread.h>

int counter = 0;
pthread_mutex_t lock = PTHREAD_MUTEX_INITIALIZER;

void *worker(void *arg) {
    for (int i = 0; i < 100000; i++) {
        pthread_mutex_lock(&lock);
        counter++;
        pthread_mutex_unlock(&lock);
    }
    return NULL;
}

int main(void) {
    pthread_t t1, t2;
    pthread_create(&t1, NULL, worker, NULL);
    pthread_create(&t2, NULL, worker, NULL);
    pthread_join(t1, NULL);
    pthread_join(t2, NULL);
    printf("Counter: %d (expected 200000)\n", counter);
}

Compile with gcc -pthread -o thread_demo thread_demo.c. Without the mutex, the counter will be less than 200000 due to the race condition — both threads read, increment, and write the same variable simultaneously.

Note

Under the hood, Linux uses clone() for both.

• fork() = clone() with separate mm_struct (separate page tables, separate memory)
• pthread_create() = clone() with CLONE_VM | CLONE_FS | CLONE_FILES (shared memory, shared filesystem info, shared file descriptors)

This is why Linux process and thread creation share the same kernel code path. The flags determine the level of sharing.

1.4.3 When to Use What

Criterion	Processes (fork)	Threads (pthread)
Memory isolation	Full — crash in child cannot corrupt parent	None — bug in one thread can corrupt all
Data sharing	Explicit IPC (pipe, shared memory)	Direct — shared heap and globals
Startup cost	Higher (copy page tables via COW)	Lower (new stack only)
Crash impact	Only that process dies	Entire process (all threads) dies
Debugging	Easier — isolated state	Harder — race conditions, deadlocks
Best for	Isolation-critical tasks, untrusted code	Low-latency data sharing, parallel computation

Embedded guideline: Use processes when isolation matters (a sensor reader crashing should not kill the display). Use threads when threads in the same program need to share large data structures with low latency (e.g., a sensor-read thread feeding a display-render thread).

---

2. Everything Is a File

This is Linux's most powerful abstraction. Devices, kernel interfaces, and inter-process communication all look like files:

Path	What It Is	Example
/dev/spidev0.0	SPI bus device	Read/write SPI data
/dev/fb0	Framebuffer (display)	Write raw pixels
/dev/i2c-1	I2C bus	Talk to sensors
/sys/class/gpio/	GPIO control via sysfs	Set pin direction, read value
/proc/cpuinfo	CPU information	Check hardware model
/proc/meminfo	Memory usage	Check available RAM

# Read CPU info — it's just a file
cat /proc/cpuinfo

# Check memory — it's just a file
cat /proc/meminfo

# See all devices — they're just files
ls /dev/

The Virtual File System (VFS)

How does Linux make "everything is a file" actually work? The answer is the Virtual File System (VFS) — a kernel layer that provides a single API (open, read, write, close) regardless of whether the target is a disk file, device, pipe, or kernel data structure.

User-space write() → VFS sys_write() → filesystem's write method → physical media. Your code uses the same API whether it's writing to ext4 on an SD card or to a device node in /dev/.

Internally, VFS tracks three key objects:

VFS Object	What It Stores	Why It Matters
inode	Metadata (owner, permissions, size, timestamps)	Identifies the file regardless of name
dentry	Name → inode mapping (directory entry)	Maps a path like /dev/i2c-1 to its inode
FILE (file object)	Open state (current position, access mode)	Allows the same file to be opened twice with different read positions

VFS switch: when you open() a file, VFS creates a file object, looks up the dentry (cached for speed), and finds the inode. Each filesystem (ext4, procfs, sysfs) implements its own inode operations — VFS provides the uniform interface above them.

This is why cat /proc/cpuinfo works — procfs implements the VFS interface, so the kernel generates CPU data on-the-fly when you call read(). And it's why you can echo 1 > /sys/class/leds/led0/brightness — sysfs implements write() to call a driver function instead of storing data on disk.

The Three Pseudo-Filesystems

Filesystem	Mount Point	Purpose	Persistence
devfs	/dev/	Device nodes — talk to hardware	Created at boot by udev
sysfs	/sys/	Kernel object attributes — configure and query	Kernel-generated, not on disk
procfs	/proc/	Process and system information	Kernel-generated, not on disk

None of these exist on disk. The kernel creates them in memory and presents them as files through VFS. This is why you can cat /proc/cpuinfo even though there is no file called cpuinfo stored anywhere — procfs creates a virtual inode that generates data when read.

Tip

When you write a kernel driver later in this course, you will create entries in /sys/ and /dev/. Understanding that these are kernel interfaces disguised as files is the key mental model.

Inodes and Links

Every file on a Linux filesystem has an inode — a metadata structure that stores everything about the file except its name. The inode number is the file's true identity in the filesystem. The name you see in ls is just a label in a directory entry that points to an inode number.

The inode contains: file type and permissions (mode), owner UID/GID, file size, timestamps, link count, and pointers to data blocks. The filename is NOT stored in the inode — it's stored in the directory entry (dentry).

What's Inside an Inode

You can inspect any file's inode with stat:

$ stat /etc/hostname
  File: /etc/hostname
  Size: 8          Blocks: 8          IO Block: 4096   regular file
Device: 179,2   Inode: 131074      Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2025-03-15 08:12:01.000000000 +0000
Modify: 2025-01-20 14:30:00.000000000 +0000
Change: 2025-01-20 14:30:00.000000000 +0000

Inode Field	Meaning	Example
File type	Regular file, directory, symlink, device, pipe, socket	-rw-r--r-- (regular)
Permissions	Owner/group/other read/write/execute bits	0644
Owner (UID/GID)	Who owns this file	root:root
Size	File content size in bytes	8
Link count	Number of directory entries pointing to this inode	1
Timestamps	Access (atime), modify (mtime), change (ctime)	See below
Block pointers	Which disk blocks store the actual data	Managed by the filesystem

Note

Three timestamps, not two. Linux tracks mtime (content modified), ctime (inode metadata changed — permissions, owner, link count), and atime (last read). On embedded systems, atime updates on every read cause unnecessary SD card writes. Most embedded images mount with noatime or relatime to avoid this wear.

Inode Number = File Identity

The inode number is what the kernel uses to identify a file — not its name. A directory is just a table mapping names to inode numbers:

Directory "/etc/":
    .           → inode 2
    ..          → inode 2
    hostname    → inode 131074
    passwd      → inode 131076
    fstab       → inode 131080

This separation of names from metadata is fundamental. It explains why renaming a file within the same filesystem is instant (just update the directory entry, the inode doesn't move), and why you can have multiple names for the same file.

# Show inode numbers
ls -i /etc/hostname /etc/passwd
# 131074 /etc/hostname
# 131076 /etc/passwd

Hard Links

A hard link is a second directory entry pointing to the same inode. Both names are equally "real" — there is no "original" and "copy." Deleting one name decrements the link count; the inode and data blocks are freed only when the link count reaches 0 and no process has the file open:

# Create a hard link
echo "sensor data" > file.txt
ln file.txt hardlink.txt

# Both show the same inode number, link count = 2
ls -li file.txt hardlink.txt
# 12345 -rw-r--r-- 2 user user 12 Jan 15 10:00 file.txt
# 12345 -rw-r--r-- 2 user user 12 Jan 15 10:00 hardlink.txt

# Delete the original name — data survives through the hard link
rm file.txt
cat hardlink.txt     # still works

Hard links cannot cross filesystem boundaries (the inode number is only unique within one filesystem) and cannot point to directories (to prevent cycles in the directory tree).

Symbolic (Soft) Links

A symbolic link is a small file whose content is a path string. The kernel follows the path when you access the symlink. It can break if the target is deleted:

# Create a symbolic link
ln -s file.txt symlink.txt

# Different inode numbers — symlink is its own file
ls -li file.txt symlink.txt
# 12345 -rw-r--r-- 1 user user 12 Jan 15 10:00 file.txt
# 67890 lrwxrwxrwx 1 user user  8 Jan 15 10:00 symlink.txt -> file.txt

# Delete the target — symlink breaks (dangling link)
rm file.txt
cat symlink.txt     # error: No such file or directory

	Hard Link	Symbolic Link
Inode	Same as target	Own inode
Cross filesystem	No	Yes
Link to directory	No	Yes
Target deleted	Data survives	Dangling (broken)
Common use	Backup references, log rotation	/dev/ aliases, library versioning

Virtual Inodes

On-disk filesystems (ext4, f2fs) store inodes in a fixed table on the partition. But /proc/, /sys/, and /dev/ are not on disk — the kernel creates inode-like objects in memory on demand.

# Real inode on disk (ext4)
stat /etc/hostname         # Inode number in the ext4 inode table

# Virtual inode in memory (procfs)
stat /proc/cpuinfo         # Inode exists only while the kernel runs

# Virtual inode in memory (sysfs)
stat /sys/class/thermal/thermal_zone0/temp   # Created by the thermal driver

When you cat /proc/cpuinfo, the kernel doesn't read a disk block — procfs generates the content on the fly from kernel data structures. The inode is a lightweight in-memory object that implements read() by calling a kernel function. This is the same mechanism your kernel drivers will use in later labs.

Why this matters for embedded: On an embedded system, /dev/ entries are created by udev at runtime — not stored on disk. /proc/ and /sys/ entries have no disk blocks at all — VFS creates inode-like objects in memory. Understanding this distinction (real inodes on disk vs virtual inodes in memory) explains why not everything in the filesystem is on the disk.

Device Files and Major/Minor Numbers

Device files in /dev/ are how user-space programs access hardware. Each device file has a major and minor number that the kernel uses to route system calls to the correct driver:

$ ls -la /dev/i2c* /dev/spidev* /dev/fb0 2>/dev/null
crw-rw---- 1 root i2c  89, 1 Jan 15 10:00 /dev/i2c-1
crw-rw---- 1 root spi 153, 0 Jan 15 10:00 /dev/spidev0.0
crw-rw---- 1 root video 29, 0 Jan 15 10:00 /dev/fb0

Device	Major	Minor	Meaning
/dev/i2c-1	89	1	I2C adapter, bus 1
/dev/spidev0.0	153	0	SPI bus 0, chip select 0
/dev/fb0	29	0	Framebuffer, display 0

The major number identifies the driver. The minor number identifies the device instance. When you call open("/dev/i2c-1", ...), the kernel looks up major 89, minor 1 and dispatches to the I2C driver.

udev: Dynamic Device Management

Device files don't exist until the kernel discovers the hardware. udev is the daemon that creates /dev/ entries when the kernel announces new devices:

# /etc/udev/rules.d/99-i2c.rules
SUBSYSTEM=="i2c-dev", MODE="0666"

Before Rule	After Rule
crw-rw---- root i2c /dev/i2c-1	crw-rw-rw- root i2c /dev/i2c-1
Requires sudo or group membership	Any user can access

You can also create stable symlinks — SYMLINK+="my_sensor" ensures /dev/my_sensor always points to the right device, regardless of discovery order.

Tip

Understanding udev explains why you needed sudo before adding a udev rule but not after. When you write kernel drivers, you'll control the default permissions that udev applies to your device nodes.

---

3. Filesystem Hierarchy

Every Linux system follows the same directory layout. Knowing where things live saves hours of debugging:

/
├── bin/          Essential user commands           ← on disk (SD card)
├── boot/         Kernel, device tree               ← on disk
├── dev/          Device nodes                      ← IN MEMORY (created by udev)
├── etc/          System configuration              ← on disk
├── home/         User home directories             ← on disk
├── lib/          Shared libraries                  ← on disk
├── proc/         Process and kernel info           ← IN MEMORY (procfs)
├── root/         Root user's home                  ← on disk
├── sys/          Kernel objects and attributes     ← IN MEMORY (sysfs)
├── tmp/          Temporary files                   ← on disk (or tmpfs = memory)
├── usr/          User programs and libraries       ← on disk
└── var/          Variable data (logs, databases)   ← on disk

Note

If you pull the SD card and mount it on your laptop, you will see /bin, /etc, /home, etc. — but /dev, /proc, and /sys will be empty directories (or missing entirely). They only exist when the Linux kernel is running and creates them in RAM.

For embedded, the important ones are:
/boot/	Kernel image, device tree, config.txt (on Pi)
/dev/	Your sensor and display devices
/sys/	Driver attributes (sysfs)
/etc/	Network config, service configuration
/var/log/	System logs (dmesg, syslog, journal)

MCU vs RTOS vs Linux Comparison

Aspect	Bare-Metal MCU	RTOS (e.g., FreeRTOS)	Linux
Programs running	One (main() loop)	Multiple tasks (threads), single binary	Multiple processes, each isolated
Storage	Flash, flat address space	Flash, flat or simple FS (LittleFS, FAT)	Filesystem hierarchy (ext4, overlayfs)
Configuration	#define in code	#define or config struct	Files in /etc/
Hardware access	Direct register writes	Direct or through HAL	Through /dev/ and /sys/
Logging	UART printf	UART or task-local buffer	journalctl, /var/log/
Scheduling	None (or manual round-robin)	Priority-based preemptive scheduler	CFS + optional RT scheduler
Memory protection	None — a bug corrupts everything	Usually none (shared address space)	Full MMU isolation per process
Boot time	Microseconds-milliseconds	Milliseconds	Seconds (5-35 s typical)
Kernel/OS size	None (0 kB)	10-50 kB typical	5-20 MB typical

Note

An RTOS gives you multitasking and deterministic scheduling but not memory protection — all tasks share one address space, so a pointer bug in one task can crash everything. Linux gives full process isolation at the cost of higher overhead. This is the fundamental trade-off between RTOS and Linux in embedded systems.

---

4. Users and Permissions

Linux enforces who can access what. Every file has an owner, a group, and permission bits:

-rw-r--r-- 1 root root  4096 Jan 15 10:00 /etc/hostname
 │││ │││ │││
 │││ │││ └── others: read only
 │││ └── group: read only
 └── owner: read + write

Permission	File	Directory
r (read)	View contents	List entries
w (write)	Modify contents	Create/delete entries
x (execute)	Run as program	Enter directory

Why Permissions Matter for Embedded

# This fails without root
cat /dev/spidev0.0
# Permission denied

# This works
sudo cat /dev/spidev0.0

# Better: set a udev rule so your user can access it
echo 'KERNEL=="spidev*", MODE="0666"' | sudo tee /etc/udev/rules.d/99-spi.rules

When you write kernel drivers, you control the default permissions of /dev/ and /sys/ entries. Getting this right means your application can run without sudo — which is important for security and for systemd services.

---

5. Services and Init

After the kernel boots, PID 1 (the init system) starts everything else. On modern Linux, this is systemd.

graph TD
    A[Kernel] --> B[systemd PID 1]
    B --> C[networking.service]
    B --> D[ssh.service]
    B --> E[your-app.service]
    B --> F[journald.service]

    style B fill:#4CAF50,color:#fff
    style E fill:#FF9800,color:#fff

Key systemd commands:

# List running services
systemctl list-units --type=service

# Check a specific service
systemctl status ssh

# Start/stop a service
sudo systemctl start my-app
sudo systemctl stop my-app

# Enable at boot / disable
sudo systemctl enable my-app
sudo systemctl disable bluetooth

Services vs Bare-Metal

Bare-metal (MCU)	Linux (systemd)
main() runs at power-on	systemd decides what runs and when
One infinite loop	Multiple services, each with its own process
No restart on crash	Restart=always in service file → auto-recovery
No dependency management	After=network.target → ordered startup

Note

In the Boot Timing Lab, you will use systemctl disable to remove unnecessary services and measure the impact on boot time. Understanding what each service does is the first step.

---

6. Kernel vs User Space

The most fundamental boundary in Linux:

	Kernel Space	User Space
Runs as	Part of the kernel (ring 0)	Normal process (ring 3)
Access	All hardware, all memory	Only through system calls
Crash impact	Entire system crashes (kernel panic)	Only that process dies
Language	C only	Any language
Examples	Drivers, scheduler, filesystems	Your application, Python scripts

graph TB
    subgraph User Space
        A[Your Application]
        B[Python Script]
        C[systemd]
    end

    subgraph Kernel Space
        D[SPI Driver]
        E[I2C Driver]
        F[Framebuffer Driver]
        G[Scheduler]
    end

    subgraph Hardware
        H[BMI160 IMU]
        I[MCP9808]
        J[Display]
    end

    A -->|read /dev/bmi160| D
    B -->|read /dev/i2c-1| E
    A -->|write /dev/fb0| F
    D --> H
    E --> I
    F --> J

Your application never touches hardware directly. It asks the kernel (via system calls, /dev/, or /sys/), and the kernel talks to the hardware. This separation is why:

• A buggy application cannot crash the whole system
• Multiple applications can share the same hardware safely
• Drivers can be updated without changing applications

GNU/Linux: Kernel vs Userland

"Linux" is technically just the kernel. The tools you use daily — gcc, bash, ls, grep, coreutils — come from the GNU project, which is why the full system is sometimes called "GNU/Linux." On a Buildroot image, GNU tools are often replaced by BusyBox — a single binary providing smaller alternatives. This is why a BusyBox ls may behave slightly differently from the Debian version.

glibc — The Bridge Between User Space and Kernel

User-space programs don't call system calls directly. They call functions in the C library, which wraps the system call interface:

printf("Hello") → glibc printf() → write() syscall → kernel → hardware

The C library is the interface between your code and the kernel. Different C libraries exist for different needs:

	glibc	musl	uClibc-ng
Size	~10 MB	~1 MB	~600 KB
POSIX conformance	Full	Nearly full	Partial
Thread safety	Full	Full	Partial
Default in	Debian, Ubuntu	Alpine Linux	Legacy Buildroot
Best for	Full-featured systems	Minimal containers, embedded	Legacy/tiny systems

The size difference matters in practice: on a device with 16 MB of flash, glibc alone would consume 60% of the storage. Switching to musl saves ~9 MB — enough to fit the entire root filesystem.

Tip

When your Buildroot image uses musl instead of glibc, some programs may behave differently (locale handling, DNS resolution). Always test on the actual image.

Kernel Architecture: Why Monolithic?

Linux is a monolithic kernel — drivers, filesystems, networking, and scheduling all run in kernel space. This is a deliberate design choice with trade-offs:

Type	Runs in Kernel	Runs in User Space	Examples
Monolithic	Everything (drivers, FS, networking, scheduling)	Applications only	Linux, FreeBSD
Microkernel	Only IPC, scheduling, memory management	Drivers, FS, networking as services	QNX, MINIX 3, seL4
Hybrid	Core + some drivers in kernel	Some services in user space	Windows NT, macOS (XNU)

In the early 1990s, Andrew Tanenbaum argued that microkernels were the superior design. Torvalds chose monolithic for performance and pragmatism — every driver call in a microkernel requires an IPC round-trip, adding latency. Linux mitigates the risks of a monolithic design with loadable kernel modules — you can add and remove drivers at runtime without rebooting.

Embedded relevance: QNX (microkernel) is popular in safety-critical automotive and medical systems where formal verification matters. Linux (monolithic) dominates general embedded because of its ecosystem, driver support, and performance. When you load a kernel module in the driver tutorials, you are using the mechanism that makes the monolithic approach practical.

---

7. Shell Tools: Pipes, Redirects, and Scripts

The shell is your primary debugging tool on an embedded Linux system.

File Descriptors

Every open file, device, pipe, or socket is represented by a file descriptor (FD) — a small integer that the kernel uses to track the resource. Every process starts with three:

FD	Name	Default	Purpose
0	stdin	Keyboard / pipe input	Where the program reads input
1	stdout	Terminal	Where normal output goes
2	stderr	Terminal	Where error messages go

The Three-Table Architecture

File descriptors are part of a three-level system inside the kernel:

Each process has its own FD table. Each FD points to an entry in the system-wide file table (which tracks the current read/write position and access mode). File table entries point to inodes (the actual file metadata). When two processes open the same file, they get separate file table entries — this is why they can read at different positions independently.

Redirections: Rewiring File Descriptors

Redirects and pipes work by reassigning these file descriptors. The key insight: the programs themselves don't change — only where their FDs point.

Baseline state: FD 0 reads from the keyboard, FD 1 and FD 2 write to the terminal, additional FDs (3, 4, ...) connect to open files.

cmd < file — the shell opens file and assigns it to FD 0 before calling execve(). The program reads from disk instead of the keyboard, without knowing anything changed.

cmd > file — the shell opens file and assigns it to FD 1. The program's output goes to disk instead of the terminal.

cmd > file 2>&1 — both stdout and stderr point to the same file. This is how you capture all output including error messages.

cmd1 | cmd2 — the shell creates a kernel pipe buffer and connects FD 1 of cmd1 to FD 0 of cmd2. Data flows between processes without touching disk.

Summary of redirections:

• > file — redirect FD 1 (stdout) to a file
• 2>&1 — send FD 2 (stderr) to wherever FD 1 currently points
• echo "ERROR: ..." >&2 — explicitly write to stderr (Section 10 uses this pattern)
• cmd < input.txt — connect FD 0 (stdin) to a file

This is why dmesg 2>&1 | grep error captures both normal output and errors — it merges stderr into stdout before the pipe.

Operators

Three operators connect commands together:

Operator	What It Does	Example
\| (pipe)	Send one command's output to another's input	dmesg \| grep spi
> (redirect)	Write output to a file (overwrite)	cat /proc/cpuinfo > cpu.txt
>> (append)	Write output to a file (append)	echo "test" >> log.txt
< (input)	Read input from a file	wc -l < /etc/passwd
2>	Redirect stderr to a file	./sensor_app 2> errors.log
2>&1	Redirect stderr to stdout (merge)	make 2>&1 \| tee build.log
&>	Redirect both stdout and stderr to a file	./app &> all_output.log

The number before > is a file descriptor number: 1 = stdout, 2 = stderr. So 2>&1 means "send FD 2 (stderr) to wherever FD 1 (stdout) currently goes." This is essential for capturing error messages in log files or piping them through grep:

# Capture both normal output and errors in one file
gcc -Wall main.c 2>&1 | tee build.log

# Discard errors silently (common in scripts that probe for optional features)
ls /dev/i2c-* 2>/dev/null

# Separate stdout and stderr into different files
./sensor_app > data.csv 2> errors.log

Pipes Are Everywhere

Most embedded debugging involves chaining commands:

# Find SPI-related kernel messages
dmesg | grep -i spi

# Count running services
systemctl list-units --type=service --state=running | wc -l

# Find which process owns a device
lsof /dev/i2c-1

# Watch a sensor value change in real time
watch -n 1 cat /sys/class/hwmon/hwmon0/temp1_input

Environment Variables

Programs inherit configuration from the environment:

# See all environment variables
env

# Set a variable for one command
SDL_VIDEODRIVER=kmsdrm ./my_app

# PATH — where the shell looks for commands
echo $PATH

Tip

In the SDL2 tutorials later, you will use SDL_VIDEODRIVER=kmsdrm to force DRM/KMS mode. Understanding environment variables explains why this works — SDL2 checks the variable at startup.

---

8. How Scheduling Works

With 80 processes but only 4 CPU cores, Linux must decide who runs when. The kernel scheduler makes this decision thousands of times per second.

The Completely Fair Scheduler (CFS)

Linux's default scheduler gives every process a fair share of CPU time. It tracks how much time each process has used and picks the one that has run the least. But "fair" does not mean "equal" — you can influence how much CPU time a process receives by setting its nice value.

Nice Values and Process Priority

Every normal (non-real-time) process has a nice value ranging from -20 to +19. The name comes from how "nice" the process is to others: a high nice value means the process yields more CPU time to others; a low (negative) nice value means it demands more time for itself.

Nice Value	Priority	Typical Use Case
-20	Highest (least nice)	Time-sensitive sensor reading, control loops
-10	High	Interactive UI, display rendering
0	Normal (default)	Most user processes
10	Below normal	Compilation, batch processing
19	Lowest (most nice)	Background logging, backups, indexing

The kernel translates nice values into scheduling weights. The relationship is not linear — each nice level changes the CPU share by roughly 10%. A process at nice -20 gets about 40x more CPU time than one at nice +19 when they compete for the same core.

# See process priorities (NI = nice value, PRI = kernel priority)
ps -eo pid,ni,pri,comm --sort=-pri | head -10

# Start a process with a specific nice value
nice -n 10 ./my_background_task

# Change the nice value of a running process
renice -n -5 -p 1234

Note

Only root (or processes with CAP_SYS_NICE) can set negative nice values. Any user can make their own process nicer (higher nice value), but making it less nice (lower value) requires privileges. This prevents unprivileged users from starving the system.

When to use nice: If your embedded system runs a sensor logger alongside a data compression task, set the compressor to nice -n 15 so it only gets CPU time when the logger is idle. If your display renderer stutters because a background apt update is running, renice -n 19 the update process.

Real-Time Scheduling

Nice values only adjust priority within the CFS scheduler. For time-critical tasks that must always preempt normal processes regardless of nice values, Linux offers real-time scheduling policies. An RT task at the lowest RT priority (1) still preempts every CFS task, even one at nice -20.

Policy	Priority Range	Behavior	When to Use
SCHED_OTHER	Nice -20 to +19	Default CFS, fair sharing	Normal applications
SCHED_FIFO	1-99	Run until done or preempted by higher RT priority	Sensor polling loops, control tasks
SCHED_RR	1-99	Like FIFO but with time slices between same-priority tasks	Multiple RT tasks at equal priority
SCHED_DEADLINE	N/A (EDF)	Earliest deadline first — kernel guarantees deadline	Tasks with known period and WCET

# Run a command with SCHED_FIFO at RT priority 50
sudo chrt -f 50 ./sensor_reader

# Check a process's scheduling policy and priority
chrt -p $(pidof sensor_reader)

# Run with SCHED_DEADLINE (period=10ms, runtime=2ms, deadline=10ms)
sudo chrt -d --sched-runtime 2000000 --sched-deadline 10000000 \
    --sched-period 10000000 0 ./control_loop

The priority hierarchy looks like this:

Priority (higher = preempts lower)
───────────────────────────────────
  SCHED_FIFO / SCHED_RR  99  ← highest RT (interrupt threads)
                          ...
                          50  ← typical sensor/control task
                          ...
                           1  ← lowest RT (still beats all CFS)
─────────── RT boundary ──────────
  SCHED_OTHER          nice -20  ← most aggressive CFS
                       nice   0  ← default
                       nice +19  ← most yielding CFS
───────────────────────────────────

Tip

For the formal theory behind RT scheduling — Rate Monotonic Analysis, utilization bounds, and priority inversion — see Real-Time Systems: Schedulability Analysis.

Why This Matters

When your level display stutters, it might be because a lower-priority process is stealing CPU time from your render loop. Understanding scheduling helps you:

• Set correct nice values or RT priorities for sensor and display tasks
• Know when nice is sufficient vs when you need chrt with a real-time policy
• Understand why PREEMPT_RT kernel helps (covered in Real-Time Systems and Real-Time Graphics)
• Diagnose performance problems with top (NI column), htop, and chrt -p

---

9. Virtual Memory and the MMU

On a microcontroller, your program accesses physical memory directly — address 0x4000_0000 is always the same RAM location. Linux is fundamentally different.

Address Translation

Every process sees its own virtual address space. The CPU's Memory Management Unit (MMU) translates virtual addresses to physical addresses:

graph LR
    A[Process A<br/>0x0000-0xFFFF] --> MMU
    B[Process B<br/>0x0000-0xFFFF] --> MMU
    MMU --> C[Physical RAM<br/>scattered pages]

    style MMU fill:#FF9800,color:#fff

Both processes can use address 0x1000, but the MMU maps them to different physical pages. This gives each process isolation — one process cannot read or corrupt another's memory.

Why This Enables Kernel/User Space

Without MMU (MCU)	With MMU (Linux)
Any code can access any address	Kernel marks its pages as privileged
A bug can overwrite anything	A user process that accesses kernel memory gets SIGSEGV (crash)
No memory protection	Each process isolated from every other
One program, all memory visible	Kernel controls which physical pages each process can see

The kernel/user space split from Section 6 is not just a software convention — it is enforced by hardware. The MMU checks every memory access and traps violations.

Swap and Overcommit

# See virtual vs physical memory
cat /proc/meminfo | grep -E "MemTotal|MemFree|SwapTotal"

# See per-process memory
ps aux --sort=-%mem | head -5

On embedded systems, swap is usually disabled (SD cards are too slow and wear out). This means if you run out of physical RAM, the OOM (Out-Of-Memory) killer terminates processes. Design your application to stay within available memory.

Note

Raspberry Pi 4 has an MMU (Cortex-A72). The Raspberry Pi Pico (Cortex-M0+) does not have an MMU — this is why it runs bare-metal or RTOS, not Linux. The MMU is the hardware feature that makes Linux possible.

---

10. Shell Scripting for Embedded

Section 7 introduced pipes, redirects, and environment variables. This section covers scripting — writing reusable programs in bash that automate embedded Linux tasks.

10.1 Variables and Quoting

# Variable assignment — no spaces around '='
SENSOR="mcp9808"
THRESHOLD=75000

# Use double quotes to prevent word splitting
echo "Reading from $SENSOR"

# Command substitution — embed command output
TEMP=$(cat /sys/class/thermal/thermal_zone0/temp)
TIMESTAMP=$(date +%Y-%m-%dT%H:%M:%S)
echo "$TIMESTAMP: temperature is $TEMP"

# Arithmetic
TEMP_C=$(( TEMP / 1000 ))
NEXT=$(( TEMP_C + 1 ))

Tip

Always quote your variables: "$VAR". Without quotes, a filename with spaces (my file.txt) splits into two words and breaks your script.

10.2 Conditionals

# Check if a device file exists before reading it
if [[ -e /dev/i2c-1 ]]; then
    echo "I2C bus available"
else
    echo "ERROR: I2C bus not found" >&2
    exit 1
fi

# Numeric comparison — is temperature above threshold?
TEMP=$(cat /sys/class/thermal/thermal_zone0/temp)
if [[ $TEMP -gt 75000 ]]; then
    echo "WARNING: CPU temperature is $(( TEMP / 1000 ))°C"
fi

Operator	Tests	Example
-f	File exists and is regular file	[[ -f /boot/config.txt ]]
-d	Directory exists	[[ -d /sys/class/gpio ]]
-e	Any path exists	[[ -e /dev/spidev0.0 ]]
-z	String is empty	[[ -z "$RESULT" ]]
-n	String is non-empty	[[ -n "$SENSOR" ]]
-eq, -lt, -gt	Numeric comparison	[[ $TEMP -gt 50000 ]]
==, !=	String comparison	[[ "$MODE" == "active" ]]

10.3 Loops

# Iterate over all I2C devices
for dev in /dev/i2c-*; do
    echo "Found I2C bus: $dev"
done

# Process a log file line by line
while read line; do
    echo "LOG: $line"
done < /var/log/syslog

# Polling loop — read temperature every second
while true; do
    TEMP=$(cat /sys/class/thermal/thermal_zone0/temp)
    echo "$(date +%H:%M:%S) $(( TEMP / 1000 ))°C"
    sleep 1
done

The while true + sleep pattern is the most common embedded scripting pattern — polling a sensor, checking a condition, or monitoring a log.

10.4 Functions

# Reusable function to read a sysfs sensor
read_temp() {
    local path="$1"
    local raw
    raw=$(cat "$path" 2>/dev/null) || return 1
    echo $(( raw / 1000 ))
}

# Call it
CPU_TEMP=$(read_temp /sys/class/thermal/thermal_zone0/temp)
echo "CPU: ${CPU_TEMP}°C"

• Use local to keep variables from leaking into the global scope.
• $? holds the exit status of the last command (0 = success, non-zero = failure).
• Functions can return an exit code with return N but use echo for data output.

10.5 Text Processing Pipeline

Three tools handle most embedded log parsing:

Tool	Purpose	Example
grep	Filter lines matching a pattern	dmesg \| grep -i spi
sed	Substitute text or extract fields	sed 's/.*temp=\([0-9]*\).*/\1/'
awk	Column extraction and arithmetic	awk '{print $1, $3}'

Regex basics for grep:

Pattern	Matches	Example
^	Start of line	grep '^Error'
$	End of line	grep 'failed$'
.*	Any characters	grep 'spi.*probe'
[0-9]	A digit	grep 'i2c-[0-9]'

A practical pipeline:

# Find the 10 most recent error-related kernel messages
dmesg | grep -i error | awk '{print $1, $NF}' | tail -10

# Count errors by type in syslog
grep -i error /var/log/syslog | awk '{print $5}' | sort | uniq -c | sort -rn

10.6 Error Handling and Robustness

Scripts on embedded systems often run unattended — at boot, from cron, or as systemd services. Silent failures are dangerous.

#!/bin/bash
set -euo pipefail

# set -e   → exit immediately if any command fails
# set -u   → treat unset variables as errors
# set -o pipefail → a pipe fails if ANY stage fails, not just the last

# Cleanup on exit (normal or error)
cleanup() {
    echo "Cleaning up temporary files..."
    rm -f /tmp/sensor_data.tmp
}
trap cleanup EXIT

# Now if any command fails, the script exits and cleanup() runs
TEMP=$(cat /sys/class/thermal/thermal_zone0/temp)
echo "$TEMP" > /tmp/sensor_data.tmp

Practice	Why
set -euo pipefail	Catch errors early — don't run 100 lines after a failure
trap 'cleanup' EXIT	Release resources even if the script crashes
cmd \|\| echo "fallback"	Handle expected failures gracefully
echo "ERROR: ..." >&2	Send error messages to stderr, not stdout

Quick Reference: Common Embedded Scripting Patterns

Task	Pattern
Read a sysfs value	VALUE=$(cat /sys/class/thermal/thermal_zone0/temp)
Check device exists	[[ -e /dev/i2c-1 ]] \|\| exit 1
Parse kernel log	dmesg \| grep -i "pattern" \| awk '{print $NF}'
Periodic sensor poll	while true; do read_sensor; sleep 1; done
Safe file write	echo "$DATA" > /tmp/file.tmp && mv /tmp/file.tmp /data/file && sync
CSV logging	echo "$(date +%s),$TEMP,$LOAD" >> /data/log.csv

---

11. Debugging and Testing

Embedded Linux debugging spans three layers — each with its own tools and mental model.

11.1 Three Layers of Debugging

Layer	What Breaks	Primary Tools
Application	Logic errors, segfaults, wrong output	GDB, strace, valgrind, printf/logging
System	Services won't start, libraries missing, permissions	journalctl, systemctl, ldd, file
Kernel / Hardware	Driver probe failures, bus errors, boot hangs	dmesg, printk, JTAG/SWD, oscilloscope

Start at the top (application) and work down. Most bugs are application-level — don't reach for dmesg until you've ruled out your own code.

11.2 GDB and Remote Debugging

GDB (GNU Debugger) lets you stop a program mid-execution, inspect variables, and step through code line by line.

# Compile with debug symbols
gcc -g -O0 -o sensor_reader sensor_reader.c

# Basic GDB session
gdb ./sensor_reader
(gdb) break main
(gdb) run
(gdb) next          # step over
(gdb) print temp    # inspect variable
(gdb) backtrace     # show call stack
(gdb) continue

Remote debugging is the norm in embedded — you compile on your laptop (x86) and debug on the Pi (ARM):

# On the Pi (target):
gdbserver :9000 ./sensor_reader

# On your laptop (host):
gdb-multiarch ./sensor_reader
(gdb) target remote <pi-ip>:9000
(gdb) break main
(gdb) continue

This cross-development workflow is fundamental: the target has limited resources, but your host has a full IDE, fast disk, and plenty of RAM.

11.3 QEMU: Test Without Hardware

QEMU emulates ARM (or any architecture) on your x86 laptop. Two modes:

Mode	What It Does	Command
User-mode	Run a single ARM binary on x86	qemu-arm ./sensor_reader
System	Boot an entire ARM Linux (kernel + rootfs)	qemu-system-arm -M vexpress-a9 ...

QEMU + GDB — full debugging without target hardware:

# Start program under QEMU, waiting for GDB on port 1234
qemu-arm -g 1234 ./sensor_reader

# In another terminal, attach GDB
gdb-multiarch ./sensor_reader
(gdb) target remote :1234
(gdb) break main
(gdb) continue

When to use QEMU:

• CI/CD testing — run ARM tests on x86 build servers
• Early development — start coding before hardware arrives
• Kernel debugging — test kernel changes without flashing SD cards
• Reproducibility — same environment every time, no hardware quirks

11.4 In-Circuit Debuggers (JTAG/SWD)

JTAG and SWD are hardware debug interfaces — a physical connection to the CPU's debug port, independent of the OS.

Feature	GDB over SSH	QEMU + GDB	JTAG / OpenOCD
Needs target HW	Yes	No	Yes + debug probe
OS must be running	Yes	Yes (user-mode)	No
Can debug early boot	No	Yes (system mode)	Yes
Can debug kernel panic	No	No	Yes
Can halt CPU	No	Yes	Yes
Setup complexity	Low	Medium	High

Raspberry Pi 4 exposes JTAG on GPIO 22-27. OpenOCD bridges the debug probe to GDB. This is essential for:

• Debugging before the kernel boots (bootloader, device tree issues)
• Investigating kernel panics where dmesg is lost
• Hardware bring-up on new boards

Tip

You rarely need JTAG for application development. Start with GDB + SSH. Reach for JTAG only when the OS itself is broken.

11.5 Kernel Logging

The kernel has its own logging system, independent of user-space:

printk levels (highest to lowest priority):

Level	Macro	When to Use
0	KERN_EMERG	System is unusable
3	KERN_ERR	Error conditions
4	KERN_WARNING	Warning conditions
6	KERN_INFO	Informational (driver loaded, device found)
7	KERN_DEBUG	Debug-level messages

In drivers, use the dev_* family (dev_info, dev_err, dev_dbg) — they automatically prefix the device name.

Reading kernel logs:

# Human-readable timestamps
dmesg -T

# Only errors and warnings
dmesg --level=err,warn

# Follow in real time (like tail -f)
dmesg -w

# Via journalctl — kernel messages only
journalctl -k

# Filter by service
journalctl -u my-app.service

# Recent messages
journalctl --since "5 min ago"

# Only errors
journalctl -p err

---

Quick Checks

• What is the difference between /dev/, /sys/, and /proc/?
• Why does cat /dev/spidev0.0 fail without sudo?
• If you run ps aux | wc -l and see 80 processes, is that normal for an embedded system? What would you expect on a Buildroot minimal image?
• What happens to a process when its parent dies?
• What is the difference between SCHED_OTHER and SCHED_FIFO? When would you use each?
• Why can't Linux run on a Cortex-M0+ (like the Pi Pico)?
• What does the pipe operator | do? How is it different from >?
• What does set -euo pipefail do, and why should every embedded script start with it?
• Write a one-line command that reads the CPU temperature, checks if it exceeds 70C, and prints a warning. (Hint: combine command substitution, arithmetic, and if.)
• What are the six steps between typing ./sensor_reader and reaching main()?
• When would you use QEMU instead of running directly on the Pi?
• Name one scenario where GDB over SSH is insufficient and you would need JTAG.
• What is the difference between SIGTERM and SIGKILL? Why can't you catch SIGKILL?
• When would you use a pipe vs shared memory for IPC?
• What happens if a parent process never calls wait() on its children?

---

Mini Exercise

SSH into your Raspberry Pi and answer these questions by exploring the system:

Part A -- Processes and Filesystem

1. How many processes are running? (ps aux | wc -l)
2. Which process uses the most memory? (ps aux --sort=-%mem | head -5)
3. What CPU does your Pi have? (cat /proc/cpuinfo | grep model)
4. How much RAM is free? (free -h)
5. What services are running? (systemctl list-units --type=service --state=running)
6. Find your I2C device in /dev/. What are its permissions? Who owns it?
7. Find the kernel version (uname -r) and compare it to the kernel file in /boot/

Part B -- Scheduling and Memory

1. What is the nice value of the ssh process? (ps -eo pid,ni,comm | grep ssh)
2. Start sleep 300 & and find it with ps. What is its scheduling policy? (chrt -p <PID>)
3. How much virtual vs physical memory does your Pi have? (cat /proc/meminfo | grep -E "MemTotal|SwapTotal")
4. Run dmesg | grep -i mmu — does the kernel mention the MMU at boot?

Part C -- Shell Tools

1. Use a pipe to count how many files are in /dev/: ls /dev/ | wc -l
2. Redirect dmesg output to a file: dmesg > ~/boot_log.txt. How large is the file? (ls -lh ~/boot_log.txt)
3. Use watch -n 2 uptime to see the system load update in real time. What are the three load average numbers?

Part D -- Program Execution and Debugging

1. Run file /usr/bin/ls — what architecture is it compiled for? Is it dynamically linked?
2. Run ldd /usr/bin/ls — how many shared libraries does it depend on?

Fill in:

Question	Your Answer
Running processes
Top memory user
CPU model
Free RAM
Running services count
I2C device permissions
Kernel version
SSH nice value
Sleep scheduling policy
Total RAM / Swap
Files in /dev/
System load average
/usr/bin/ls architecture
ls shared library count

---

Key Takeaways

• Linux adds processes, files, permissions, and services on top of hardware — layers that don't exist on bare-metal MCUs.
• Everything is a file — devices, kernel info, and configuration all appear in the filesystem.
• Permissions control access to hardware — understanding them prevents "Permission denied" frustrations.
• systemd manages services — you'll use it to control boot time and run your applications.
• The shell (pipes, redirects, environment variables) is your primary debugging interface — most embedded Linux debugging is done by chaining commands.
• The scheduler decides who runs when — understanding priorities and RT scheduling explains why your sensor loop might stutter.
• The MMU enforces kernel/user space isolation in hardware — it's the reason Linux requires a Cortex-A (with MMU) and cannot run on a Cortex-M (without MMU).
• Shell scripts automate repetitive embedded tasks — sensor polling, log parsing, device checks, and health monitoring all benefit from scripting with proper error handling.
• ELF and dynamic linking — understanding how Linux loads programs explains "Exec format error", missing library failures, and the static vs dynamic trade-off in Buildroot.
• Debugging is layered — start with application tools (GDB, strace), escalate to system tools (journalctl, ldd), and reach for kernel/hardware tools (dmesg, JTAG) only when needed.

---

Hands-On

Explore these concepts in practice:

• Tutorial: SSH Login — connect to your Pi and run the Mini Exercise above
• Tutorial: Shell Scripting for Embedded Linux — apply Section 10 with hands-on scripting exercises
• Tutorial: ELF and Program Execution — inspect ELF binaries, trace program loading, cross-compile with QEMU (Sections 1.1-1.2)
• Tutorial: Debugging Practices — GDB, remote debugging, QEMU, and kernel logging (Section 11)
• Tutorial: Processes and IPC in C — fork, pipe, signals, and pthreads (Sections 1.3-1.4)