Lesson 2: How Linux Works

Óbuda University — Linux in Embedded Systems

80 processes, 200 device files, permission denied. What is all this?

Today's Map

Block 1: Processes, program execution, everything is a file, filesystem hierarchy, permissions, kernel vs user space, shell tools, scheduling & MMU, debugging overview.
Block 2: Lab 2 preview: SSH in and explore processes, filesystem, permissions, scheduling; then fill in the comparison table.

Where to run commands: # [RPi] = run on the Raspberry Pi · # [Host] = run on your laptop · # [Host/RPi] = works on either · no label = concept only

1. Everything Is a Process

A process is a running program with its own memory space, file descriptors, and permissions.

Concept	Description	Command
PID	Every process has a unique ID	`ps aux`
Parent/child	Every process started by another (except PID 1)	`pstree`
Foreground/background	Terminal processes vs daemons	`jobs`, `&`, `fg`
Signals	How processes communicate	`kill -l`

# [RPi]
ps aux          # See all processes
pstree -p       # Tree view — who started what
htop            # Interactive CPU/memory view

MCU vs Linux: One Program vs Many

 MCU (Pico)                    Linux (RPi 4)
 ┌────────────────┐            ┌─────────────────┐
 │                │            │  ssh   systemd  │
 │   main()       │            │  dbus  journal  │
 │   while(1) {}  │            │  dhcp  avahi    │
 │                │            │  cron  udev     │
 │   (one loop,   │            │  your_app  ...  │
 │    all yours)  │            │  (80+ processes │
 │                │            │   sharing CPU)  │
 └────────────────┘            └─────────────────┘

Understanding this matters for debugging: "Why is my sensor read slow?" — another process may be hogging CPU.

How a Program Becomes a Process

You type ./sensor_reader and press Enter. What happens?

 ┌──────────┐   fork()    ┌──────────┐   execve()   ┌──────────┐
 │  Shell   │ ──────────► │  Child   │ ───────────► │  Kernel  │
 │  (bash)  │             │  Shell   │              │ check ELF│
 └──────────┘             └──────────┘              └────┬─────┘
                                                         │
                                                    mmap segments
                                                         │
                                                    ┌────▼─────┐
                                                    │ ld.so    │
                                                    │ (dynamic │
                                                    │  linker) │
                                                    └────┬─────┘
                                                         │
                                                    resolve symbols
                                                         │
                                                    ┌────▼─────┐
                                                    │  _start  │
                                                    │  main()  │
                                                    └──────────┘

The Six Steps to main()

Shell calls fork() — creates a copy of itself (the child process)
Child calls execve("./sensor_reader", ...) — asks the kernel to replace itself
Kernel checks ELF magic bytes (7f 45 4c 46) — is this a valid executable?
Kernel maps ELF segments (code, data) into virtual memory via mmap
If dynamically linked, kernel invokes ld-linux-armhf.so.3 to load .so libraries
Control jumps to _start then __libc_start_main then your main()

Why fork + exec instead of a single "spawn"? Because the shell needs to set up redirects, pipes, and environment variables in the child before the new program replaces it. This two-step pattern is fundamental to Unix.

# [RPi]
# See it live:
strace -f -e trace=clone,execve ./sensor_reader

fork() Creates a Copy

center height:400px

fork() Creates a Copy

center height:400px

After fork(), parent and child have separate memory (mm_struct) and separate FD tables. The child then calls execve() to replace itself with a new program.

center

Process state transitions in Linux: running, sleeping, stopped, and zombie. The scheduler moves processes between states based on events and resource availability.

ELF Format and Program Loading

Every binary, library, and kernel module on Linux is an ELF (Executable and Linkable Format) file.

Section	Contents	MCU Analogy
`.text`	Machine code (your functions)	Program in flash
`.data`	Initialized global variables	RAM with initial values
`.bss`	Zero-initialized globals	Zeroed RAM at startup
`.rodata`	Read-only data (string constants)	`const` data in flash
`.dynamic`	Dynamic linking information	N/A on MCU

Aspect	MCU (bare-metal)	Linux
Entry point	Reset vector then `main()`	`_start` then linker then `main()`
Code loading	Burned into flash at compile	Kernel maps ELF segments at runtime
Libraries	Static (or none)	Dynamic (`.so` loaded at runtime)
Addresses	Physical, shared	Virtual, private per process

2. Everything Is a File

This is Linux's most powerful abstraction. Devices, kernel interfaces, and IPC all look like files:

Path	What It Is	Example
`/dev/spidev0.0`	SPI bus device	Read/write SPI data
`/dev/fb0`	Framebuffer (display)	Write raw pixels
`/dev/i2c-1`	I2C bus	Talk to sensors
`/sys/class/gpio/`	GPIO control	Set pin direction
`/proc/cpuinfo`	CPU information	Check hardware model

# [RPi]
cat /proc/cpuinfo       # Read CPU info — it's just a file
cat /proc/meminfo       # Check memory — it's just a file
ls /dev/                # See all devices — they're just files

The Virtual File System (VFS)

center height:400px

VFS provides one API (open/read/write/close) for all file types. Your code does not need to know whether it's reading a disk file, a sensor, or kernel data.

write() → VFS sys_write() → filesystem's write method → physical media

Inside VFS

center height:350px

VFS tracks three objects per open file:

Object	What it stores
inode	Metadata (owner, size, permissions, block pointers)
dentry	Name → inode mapping (cached for speed)
file object	Open state (current position, access mode, FD)

Every filesystem (ext4, procfs, sysfs) registers its own operations (read, write, open). VFS dispatches your read() call to the right filesystem — you never need to know which one.

VFS: One Interface, Many Backends

  User space:   cat /proc/cpuinfo       cat /etc/hostname
                    │                        │
  ────────── syscall boundary (read) ──────────────
                    │                        │
  VFS layer:     lookup dentry → find inode → call inode's read()
                    │                        │
  Filesystem:    procfs                   ext4
                 (generates data           (reads disk
                  from kernel memory)       blocks)

This is the power of VFS: same read() call, completely different implementations underneath.

cat /proc/cpuinfo → procfs generates text from kernel data structures
cat /etc/hostname → ext4 reads blocks from the SD card
cat /sys/class/thermal/thermal_zone0/temp → sysfs calls the thermal driver

Your code always uses open() / read() / write() / close(). VFS routes to the right place.

The Three Pseudo-Filesystems

Filesystem	Mount Point	Purpose	On disk?
devfs	`/dev/`	Device nodes — talk to hardware	No — created by udev
sysfs	`/sys/`	Kernel object attributes	No — kernel-generated
procfs	`/proc/`	Process and system info	No — kernel-generated

None of these exist on disk. The kernel creates them in memory and presents them as files.

This is why you can cat /proc/cpuinfo even though there is no file called cpuinfo stored anywhere.

When you write a kernel driver later, you will create entries in /sys/ and /dev/. These are kernel interfaces disguised as files.

sysfs Walkthrough

Navigate a real sysfs tree for a temperature sensor:

# [RPi]
$ ls /sys/class/hwmon/hwmon0/
device  name  power  subsystem  temp1_input  uevent

$ cat /sys/class/hwmon/hwmon0/name
mcp9808

$ cat /sys/class/hwmon/hwmon0/temp1_input
26750

One value per file — this is the sysfs design rule. Each attribute is a separate file.

Approach	How You Read Temperature
MCU	Read register `0x05` directly over I2C bus
Linux user space	`cat /sys/class/hwmon/hwmon0/temp1_input`
What happens underneath	VFS → hwmon subsystem → driver → I2C transaction → register read

The file is a thin interface to a chain of kernel subsystems. The driver populates it; you just read it.

Inodes: What's Under a Filename

center height:280px

Every file has an inode — its true identity. The inode number is what the kernel uses, not the filename.

Inodes

Inode field	What it stores
File type & permissions	`-rw-r--r--`, `drwxr-xr-x`, `crw-rw----`
Owner (UID/GID)	Who owns this file
Size	Content size in bytes
Timestamps	mtime (modified), ctime (metadata changed), atime (accessed)
Link count	How many directory entries point here
Block pointers	Which disk blocks hold the data

A directory is just a table: { "hostname" → inode 131074, "passwd" → inode 131076, ... }

stat /etc/hostname — try it! Shows all inode fields for any file.

Hard Links vs Symbolic Links

Hard link:                     Symbolic link:

  "file.txt"  ──┐               "file.txt"  ── inode 12345 ── [data]
                ├── inode 12345 ── [data]
  "backup.txt" ─┘               "link.txt"  ── inode 67890 ── "/path/to/file.txt"
                                                  (just a string)

	Hard link	Symbolic link
Same inode?	Yes — same file, two names	No — own inode
Cross filesystem?	No	Yes
Target deleted?	Data survives (link count > 0)	Broken (dangling)
Can link to directory?	No	Yes

ls -li file.txt backup.txt    # same inode number = hard link
ls -li file.txt link.txt      # different inode = symlink

Virtual inodes: /dev/, /proc/, /sys/ have no disk blocks — the kernel creates inodes in memory on demand. This is how a driver's read() function gets called when you cat /sys/.../temp.

Device File Ecosystem

# [RPi]
$ ls -la /dev/i2c* /dev/spidev* /dev/ttyUSB* /dev/fb0 2>/dev/null
crw-rw---- 1 root i2c  89, 1 Jan 15 10:00 /dev/i2c-1
crw-rw---- 1 root spi 153, 0 Jan 15 10:00 /dev/spidev0.0
crw-rw---- 1 root tty 188, 0 Jan 15 10:00 /dev/ttyUSB0
crw-rw---- 1 root video 29, 0 Jan 15 10:00 /dev/fb0

Major and minor numbers identify the driver and device instance:

Device	Major	Minor	Meaning
`/dev/i2c-1`	89	1	I2C adapter, bus 1
`/dev/spidev0.0`	153	0	SPI bus 0, chip select 0
`/dev/fb0`	29	0	Framebuffer, display 0

The major number selects the driver. The minor number selects the instance. The kernel uses these to route open()/read()/write() calls to the correct driver function.

udev Rules

Linux names devices dynamically — /dev/i2c-1 does not exist until the kernel discovers the I2C adapter.

udev is the daemon that creates /dev/ entries when the kernel announces new devices.

A one-line rule controls permissions:

# [RPi]
# /etc/udev/rules.d/99-i2c.rules
SUBSYSTEM=="i2c-dev", MODE="0666"

Before Rule	After Rule
`crw-rw---- root i2c /dev/i2c-1`	`crw-rw-rw- root i2c /dev/i2c-1`
Requires `sudo` or group membership	Any user can access

"This is why you needed sudo before but not after."

You can also use udev to create stable symlinks: SYMLINK+="my_sensor" → /dev/my_sensor always points to the right device, regardless of discovery order.

3. Filesystem Hierarchy

Every Linux system follows the same directory layout (FHS):

center

Key Directories for Embedded

/
├── bin/          Essential commands           ← on disk (SD card)
├── boot/         Kernel, device tree          ← on disk
├── dev/          Device nodes                 ← IN MEMORY (udev)
├── etc/          System configuration         ← on disk
├── lib/          Shared libraries             ← on disk
├── proc/         Process and kernel info      ← IN MEMORY (procfs)
├── sys/          Kernel objects               ← IN MEMORY (sysfs)
├── tmp/          Temporary files              ← on disk (or tmpfs)
├── usr/          User programs                ← on disk
└── var/          Variable data (logs)         ← on disk

Pull the SD card, mount it on your laptop → you see /bin, /etc, /home. But /dev, /proc, /sys are empty — they only exist when the kernel is running.

MCU vs Linux: Where Things Live

Aspect	Microcontroller	Linux
Storage	Flash memory, flat address space	Filesystem hierarchy
Configuration	`#define` constants in code	Files in `/etc/`
Hardware access	Direct register writes	Through `/dev/` and `/sys/`
Logging	UART printf	`journalctl`, `/var/log/`
Multiple programs	No (single main loop)	Yes (processes, services)

4. Users and Permissions

Linux enforces who can access what. Every file has an owner, a group, and permission bits:

-rw-r--r-- 1 root root  4096 Jan 15 10:00 /etc/hostname
 │││ │││ │││
 │││ │││ └── others: read only
 │││ └───── group: read only
 └──────── owner: read + write

Permission	On a file	On a directory
`r` (read)	View contents	List entries
`w` (write)	Modify contents	Create/delete entries
`x` (execute)	Run as program	Enter directory

Why Permissions Matter for Embedded

# [RPi]
# This fails without root:
$ cat /dev/spidev0.0
Permission denied

# This works:
$ sudo cat /dev/spidev0.0

# Better: set a udev rule so your user can access it
$ echo 'KERNEL=="spidev*", MODE="0666"' | \
    sudo tee /etc/udev/rules.d/99-spi.rules

When you write kernel drivers, you control the default permissions of /dev/ and /sys/ entries.

Getting this right means your application can run without sudo — important for security and for systemd services.

5. Kernel vs User Space

The most fundamental boundary in Linux:

	Kernel Space	User Space
Runs as	Part of the kernel (ring 0)	Normal process (ring 3)
Access	All hardware, all memory	Only through system calls
Crash impact	Entire system crashes	Only that process dies
Language	C only	Any language
Examples	Drivers, scheduler, FS	Your app, Python scripts

Protection Rings and Architecture

center

CPU privilege levels (rings 0–3). Linux uses ring 0 for the kernel and ring 3 for user-space applications. Hardware enforces the boundary.

Linux Architecture

center

Layered Linux architecture: hardware at the bottom, kernel (drivers, scheduler, VFS) in the middle, user-space applications at the top.

The System Call Boundary

Your application never touches hardware directly. It asks the kernel via system calls:

 ┌─────────────────────────────────────────────────┐
 │  User Space                                     │
 │  ┌────────────┐  ┌────────────┐  ┌────────────┐ │
 │  │ Your App   │  │ Python     │  │ systemd    │ │
 │  │ read()     │  │ open()     │  │ socket()   │ │
 │  └─────┬──────┘  └─────┬──────┘  └─────┬──────┘ │
 ├────────┼───── system call interface ───┼────────┤
 │  Kernel Space                          │        │
 │  ┌─────▼──────┐  ┌────────────┐  ┌─────▼──────┐ │
 │  │ SPI Driver │  │ I2C Driver │  │ Net Driver │ │
 │  └─────┬──────┘  └─────┬──────┘  └─────┬──────┘ │
 ├────────┼───────────────┼───────────────┼────────┤
 │  ┌─────▼──────┐  ┌─────▼──────┐  ┌─────▼──────┐ │
 │  │ BMI160 IMU │  │ MCP9808    │  │ Ethernet   │ │
 │  └────────────┘  └────────────┘  └────────────┘ │
 └─────────────────────────────────────────────────┘

Why This Separation Matters

Three critical benefits of kernel/user space isolation:

A buggy application cannot crash the whole system
Segfault kills only your process, not the OS
Multiple applications can share the same hardware safely
The kernel serializes access to devices
Drivers can be updated without changing applications
The /dev/ and /sys/ interface is stable

On an MCU, any bug can corrupt anything — there is no protection boundary.

glibc and Kernel Architecture

User-space programs call C library functions, which wrap system calls:

printf("Hello") ──► glibc printf() ──► write() syscall ──► kernel ──► HW

	glibc	musl	uClibc-ng
Size	~10 MB	~1 MB	~600 KB
Used in	Debian, Ubuntu	Alpine, Buildroot	Legacy embedded

Linux is a monolithic kernel — drivers, FS, networking all in kernel space. Mitigated by loadable kernel modules (add/remove at runtime).

Type	In Kernel	Examples
Monolithic	Everything	Linux, FreeBSD
Microkernel	Only IPC, sched, MM	QNX, seL4
Hybrid	Core + some drivers	Windows NT, macOS

musl vs glibc

The C library is the gateway between your application and the kernel. Two choices dominate:

	glibc	musl
Size	~10 MB	~700 KB
Static linking	Problematic (NSS, iconv)	Clean and supported
Features	Full POSIX + GNU extensions	Strict POSIX, minimal extras
Locale support	Complete (adds ~30 MB)	Minimal (UTF-8 only)
Used in	Debian, Ubuntu, Raspberry Pi OS	Alpine, Buildroot (default)

Why Buildroot defaults to musl: On a 16 MB flash target, 10 MB for glibc leaves 6 MB for your kernel, rootfs, and application. musl at 700 KB gives you room.

Trade-off: Some software assumes glibc (NSS for user lookups, dlopen quirks). Test your application with musl early — not the night before the demo.

6. Shell Tools: Pipes, Redirects, Environment

The shell is your primary debugging tool on embedded Linux.

Operator	What It Does	Example
\|	\| (pipe)	dmseg \| grep spi
`>` (redirect)	Write output to file	`cpuinfo > cpu.txt`
`>>` (append)	Append output to file	`echo "test" >> log.txt`
`<` (input)	Read input from file	`wc -l < /etc/passwd`
`2>`	Redirect stderr to file	`./app 2> errors.log`
`2>&1`	Merge stderr into stdout	`make 2>&1 \\| tee build.log`

2 = stderr file descriptor. 2>&1 = "send stderr wherever stdout goes." Essential for logging.

# [RPi]
dmesg | grep -i spi                          # Find SPI kernel messages
systemctl list-units --state=running | wc -l  # Count services
watch -n 1 cat /sys/class/hwmon/hwmon0/temp1_input  # Live sensor
SDL_VIDEODRIVER=kmsdrm ./my_app               # Set env var for one cmd

File Descriptors

Every open file, device, pipe, or socket is a file descriptor (FD) — a small integer the kernel uses to track the resource.

Every process starts with three: FD 0 (stdin), FD 1 (stdout), FD 2 (stderr).

FD Architecture: Three Tables

center height:400px

Per-process FD table → system-wide file table → inode table. When two processes open the same file, they get separate FDs and separate file table entries (separate read positions), but both point to the same inode.

Baseline: Default File Descriptors

center height:400px

FD 0 from keyboard, FD 1+2 to display, FD 3+ for any files the process opens. This is the starting state for every process.

Write into a file

center height:400px

Input Redirection: `cat file`

center height:400px

The shell opens file and assigns it to FD 0 before execve(). The program reads from disk instead of the keyboard — without the program knowing anything changed.

Output redirection: `prog 2>&1`

center height:400px

The shell opens file and assigns it to FD 1. The program's printf() output goes to disk instead of the terminal. Same program, different wiring.

Pipes: `cmd1 | cmd2`

center height:400px

The shell creates a kernel pipe buffer, connects FD 1 of process 1 to FD 0 of process 2. Data flows between processes without touching disk.

Key insight: Redirections and pipes are just reassigning file descriptors. The programs themselves don't change — only where their FDs point.

Try It Now: File Descriptors (5 min) `[RPi]`

Inspect FDs of a live process, then use redirects and pipes on real hardware data:

# [RPi]
ls -l /proc/$$/fd            # Your shell's open FDs
dmesg | grep -i i2c > ~/i2c_log.txt   # Redirect + pipe
cat /sys/class/thermal/thermal_zone0/temp | awk '{print $1/1000, "°C"}'

Tutorial: Exploring Linux — Section 5: Pipes, Redirects, and Watching Theory: Section 7: Shell Tools — File Descriptors

Pipes Are Everywhere

Most embedded debugging involves chaining commands:

 ┌────────┐      ┌────────┐      ┌────────┐
 │ dmesg  │ ───► │ grep   │ ───► │ tail   │
 │        │ pipe │ -i spi │ pipe │ -10    │
 │ stdout │      │ filter │      │ last N │
 └────────┘      └────────┘      └────────┘

# [RPi]
# Find which process owns a device
lsof /dev/i2c-1

# Parse kernel log for errors
dmesg | grep -i error | awk '{print $1, $NF}' | tail -10

# Count errors by type
grep -i error /var/log/syslog | awk '{print $5}' | \
    sort | uniq -c | sort -rn

7. How Scheduling Works

With 80 processes but only 4 CPU cores, Linux must decide who runs when.

The Completely Fair Scheduler (CFS)

CFS gives every process a fair share of CPU time. It tracks how much time each process has used and picks the one that has run the least.

You influence CFS with nice values — how "nice" a process is to others:

Nice Value	Priority	Typical Use	CPU Share
−20	Highest (least nice)	Sensor reading, control loops	~40× of nice +19
−10	High	Display rendering, interactive UI	~10× of nice +19
0	Normal (default)	Most user processes	baseline
10	Below normal	Compilation, batch jobs	~1/10 of nice −10
19	Lowest (most nice)	Background logging, backups	minimal

Nice levels

A higher "niceness" value means the process is more polite, taking less CPU time.

Each nice level changes CPU share by ~10%. Negative nice requires root (CAP_SYS_NICE).

# [RPi]
# See process priorities (NI = nice value)
ps -eo pid,ni,pri,comm --sort=-pri | head -10

# Start a process with lower priority
nice -n 10 ./my_background_task

# Change nice value of a running process (need root for negative)
sudo renice -n -5 -p 1234

Real-Time Scheduling

Nice values only adjust priority within CFS. For time-critical tasks, Linux offers RT scheduling that always preempts normal processes — even nice −20:

Policy	Priority	Behavior	When to Use
`SCHED_OTHER`	Nice −20…+19	Default CFS, fair sharing	Normal applications
`SCHED_FIFO`	RT 1–99	Run until done or preempted by higher RT	Sensor loops, control tasks
`SCHED_RR`	RT 1–99	Like FIFO with time slices	Multiple RT tasks at same priority
`SCHED_DEADLINE`	EDF	Earliest deadline first	Tasks with known period and WCET

# [RPi]
# Run with real-time FIFO priority 50 (range 1-99)
sudo chrt -f 50 ./sensor_reader

# Check a process's scheduling policy
chrt -p $(pidof sensor_reader)

When your level display stutters, it might be because a lower-priority process is stealing CPU time. Fix: nice the background task, or chrt the display task.

Priority Hierarchy

Priority (higher preempts lower)
─────────────────────────────────
  SCHED_FIFO / RR     99   ← interrupt threads, safety-critical
                      ...
                      50   ← typical sensor/control task
                      ...
                      1    ← lowest RT (still beats ALL of CFS)
──────── RT boundary ────────
  SCHED_OTHER     nice -20  ← most aggressive CFS process
                  nice   0  ← default
                  nice +19  ← yields to everyone
─────────────────────────────────

Key rule: RT priority 1 always preempts nice −20. The boundary is absolute.

# [RPi]
# Nice: influence CFS share (any user can increase, root to decrease)
nice -n 15 ./compressor         # runs only when CPU is idle

# chrt: jump above CFS entirely (requires root)
sudo chrt -f 80 ./control_loop  # preempts everything below RT 80

Theory 8 goes deep on real-time scheduling, RMA bounds, and priority inversion.

8. Virtual Memory and the MMU

On a microcontroller, address 0x4000_0000 is always the same RAM location.

On Linux, every process sees its own virtual address space. The CPU's MMU translates virtual to physical:

 Process A                    Process B
 ┌──────────────┐            ┌──────────────┐
 │ 0x0000       │            │ 0x0000       │
 │ 0x1000 ──────┼──┐     ┌───┼── 0x1000     │
 │ 0xFFFF       │  │     │   │ 0xFFFF       │
 └──────────────┘  │     │   └──────────────┘
                   │     │
                   ▼     ▼
              ┌──────────────┐
              │     MMU      │
              │  (hardware)  │
              └──────┬───────┘
                     │
              ┌──────▼───────┐
              │ Physical RAM │
              │ (scattered   │
              │  pages)      │
              └──────────────┘

Both use address 0x1000, but the MMU maps them to different physical pages.

MMU

center

Process memory on 32bit

center

MMU Enables Isolation

Without MMU (MCU)	With MMU (Linux)
Any code can access any address	Kernel marks its pages as privileged
A bug can overwrite anything	Bad access causes `SIGSEGV` (crash)
No memory protection	Each process isolated
One program, all memory visible	Kernel controls page visibility

The kernel/user space split is not just a software convention — it is enforced by hardware. The MMU checks every memory access.

Raspberry Pi 4 has an MMU (Cortex-A72). Raspberry Pi Pico (Cortex-M0+) does not — this is why it cannot run Linux. The MMU is the hardware feature that makes Linux possible.

Process vs Thread

center height:350px

	Process	Thread
Memory	Separate address space	Shared address space
File descriptors	Separate copy after `fork()`	Shared
Creation cost	High (copy page tables)	Low (new stack only)
Crash impact	Only that process dies	All threads die
Linux syscall	`fork()`	`clone()` with `CLONE_VM`

RTOS tasks are like threads — shared address space, no isolation. Linux processes give full isolation at higher cost.

Inter-Process Communication

On an MCU, functions share data via global variables. On Linux, processes are isolated — they need explicit mechanisms:

Mechanism	Direction	Typical Use
Pipes / FIFOs	One-way	Shell pipelines
Signals	One-way	Process control (SIGTERM)
Shared memory	Bidirectional	High-speed sensor buffers
Unix sockets	Bidirectional	Local services (D-Bus)
TCP/UDP sockets	Bidirectional	Network (MQTT, HTTP)

MCU Pattern	Linux Equivalent
Global variable between tasks	Shared memory (with sync)
ISR flag / event bit	Signal or eventfd
FreeRTOS queue	POSIX message queue or pipe

fork() in Code

pid_t pid = fork();

if (pid == 0) {
    /* Child process */
    printf("Child PID: %d\n", getpid());
    _exit(0);
} else if (pid > 0) {
    /* Parent process */
    int status;
    waitpid(pid, &status, 0);
    printf("Child exited: %d\n", WEXITSTATUS(status));
}

Return Value	Meaning
`0`	You are the child
`> 0`	You are the parent (value = child PID)
`-1`	Error (out of resources)

Both processes resume from the same line. Only the return value tells you which one you are.

fork() Memory Separation

int counter = 0;             /* global variable */

pid_t pid = fork();          /* both get a COPY */

if (pid == 0) {
    counter += 100;
    printf("Child:  counter = %d (addr %p)\n", counter, &counter);
    _exit(0);
}
waitpid(pid, NULL, 0);
printf("Parent: counter = %d (addr %p)\n", counter, &counter);

Child:  counter = 100 (addr 0x55a3b4)    ← child modified its copy
Parent: counter = 0   (addr 0x55a3b4)    ← parent unchanged!

Same virtual address, different physical pages. The MMU maps them separately. This is Copy-on-Write (COW): the kernel shares pages until one process writes, then copies.

Compare this to threads → shared memory, no copy, race conditions possible.

Try It Now: fork() (5 min) `[Host/RPi]`

Compile and run fork_demo.c — observe parent/child PIDs and exit status collection.

Then comment out waitpid(), recompile, and check for zombies with ps aux | grep Z.

Tutorial: Processes and IPC — Section 1: fork() Theory: Section 1.4.1: fork() and wait()

pipe() — Data Between Processes

int fd[2];
pipe(fd);               /* fd[0] = read, fd[1] = write */
pid_t pid = fork();

if (pid == 0) {          /* Child writes */
    close(fd[0]);
    write(fd[1], "26750", 5);
    close(fd[1]);
    _exit(0);
} else {                  /* Parent reads */
    close(fd[1]);
    char buf[64];
    int n = read(fd[0], buf, sizeof(buf) - 1);
    buf[n] = '\0';
    close(fd[0]);
    waitpid(pid, NULL, 0);
}

This is what the shell does for cmd1 | cmd2 — now you see the system calls.

Try It Now: pipe() (5 min) `[Host/RPi]`

Compile and run pipe_sensor.c — child reads the CPU temperature sensor, sends it through a pipe, parent displays it.

Tutorial: Processes and IPC — Section 2: pipe() Theory: Section 1.3.1: Pipes in C

Signals

A signal is a software interrupt delivered to a process.

Signal	Default	Catchable	Embedded Use
`SIGINT` (2)	Terminate	Yes	`Ctrl+C`
`SIGTERM` (15)	Terminate	Yes	`systemctl stop`
`SIGKILL` (9)	Terminate	No	Force kill
`SIGUSR1` (10)	Terminate	Yes	Reload config
`SIGCHLD` (17)	Ignore	Yes	Child exited

kill -SIGUSR1 $PID        # From bash

volatile sig_atomic_t flag = 0;
void handler(int sig) { flag = 1; }  /* Set flag only! */
struct sigaction sa = { .sa_handler = handler };
sigaction(SIGUSR1, &sa, NULL);

Key rule: Signal handlers must be fast — set a flag, return. Same principle as MCU ISRs.

Try It Now: Signals (5 min) `[Host/RPi]`

Compile and run signal_demo.c & — then send kill -SIGUSR1 and kill -SIGINT from another terminal.

Tutorial: Processes and IPC — Section 3: Signals Theory: Section 1.3.2: Signals

Threads — Shared Memory, Shared Risk

center height:250px

pthread_mutex_t lock = PTHREAD_MUTEX_INITIALIZER;
pthread_mutex_lock(&lock);
counter++;
pthread_mutex_unlock(&lock);

	Process (`fork`)	Thread (`pthread`)
Memory	Separate	Shared
Crash	Only that process	All threads die
Sync	IPC (pipe, signal)	Mutex, condition var

POSIX Threads Architecture

center height:400px

POSIX threads share code, data, heap, and file descriptors. Only the stack and registers are private. Compile with -pthread.

Try It Now: Threads (5 min) `[Host/RPi]`

Compile thread_demo.c with gcc -pthread — run it and observe the race condition without mutex vs correct result with mutex.

Tutorial: Threads and Synchronization — Section 1: Threads Theory: Section 1.4.2: POSIX Threads

Condition Variables — "Wait Until Ready"

A mutex protects data. A condition variable avoids busy-waiting — a thread sleeps until another thread signals it.

pthread_mutex_t lock = PTHREAD_MUTEX_INITIALIZER;
pthread_cond_t  cond = PTHREAD_COND_INITIALIZER;
int data_ready = 0;

/* Producer thread */                    /* Consumer thread */
pthread_mutex_lock(&lock);               pthread_mutex_lock(&lock);
buffer[head] = value;                    while (!data_ready)
data_ready = 1;                              pthread_cond_wait(&cond, &lock);
pthread_cond_signal(&cond);              process(buffer);
pthread_mutex_unlock(&lock);             pthread_mutex_unlock(&lock);

Mechanism	When to Use
Mutex only	Protect shared data, threads poll or sleep with `usleep()`
Mutex + cond var	Producer-consumer: consumer sleeps until data arrives

pthread_cond_wait atomically unlocks the mutex and sleeps. When signaled, it re-locks and returns.

Producer-Consumer Pattern

The most common threading pattern in embedded systems:

 ┌─────────────┐    ┌──────────────────────┐    ┌──────────────┐
 │  Producer   │    │   Circular Buffer    │    │  Consumer    │
 │  (sensor)   ├───►│  [.][.][X][X][.]     ├───►│  (logger/    │
 │             │    │   ↑head    ↑tail     │    │   display)   │
 └─────────────┘    └──────────────────────┘    └──────────────┘
                     protected by mutex + cond

Embedded example	Producer	Consumer
Data logger	Sensor read thread	CSV write thread
Motor control	Encoder thread	PID loop thread
Dashboard	Multiple sensor threads	Display thread

This is what your sys_dashboard exercise already does — but with polling. The tutorial shows how to upgrade it with condition variables.

Debugging Threads

# Show all threads of a process (LWP = Light Weight Process = thread)
ps -eLf | grep sys_dashboard

# Interactive: see threads in htop
htop    # press H to toggle thread view

# Detect race conditions automatically
gcc -g -fsanitize=thread -o app app.c -pthread
./app   # ThreadSanitizer reports data races at runtime

# Valgrind alternative (slower but catches more)
valgrind --tool=helgrind ./app

Tool	What it finds
`ps -eLf`	Thread count, TIDs, CPU usage per thread
`-fsanitize=thread`	Data races (compile-time instrumented)
`helgrind`	Lock order violations, missing locks
`strace -f`	System calls from all threads (`-f` follows forks/threads)

Choosing an IPC Mechanism

Need	Mechanism	Complexity
Parent ↔ child data	`pipe()`	Low
Unrelated processes	Named pipe (FIFO)	Low
Notification (no data)	Signal (`kill()`)	Low
Local service	Unix socket	Medium
Network	TCP/UDP socket	Medium
High-speed buffer	`shm_open()` + `mmap()`	High

Start simple (pipe), escalate only when needed.

Try It Now: Sensor Monitor (10 min) `[Host/RPi]`

Compile and run sensor_monitor.c — it combines fork + pipe + signals. Test: run in background, send SIGUSR1 for stats, send SIGTERM for graceful shutdown, check the CSV.

Tutorial: Processes and IPC — Section 4: Sensor Monitor Theory: Sections 1.3–1.4 combined

9. Services and Init (systemd)

After the kernel boots, PID 1 starts everything else. On modern Linux, this is systemd.

center

systemd as PID 1: manages service dependencies, starts daemons in parallel, and supervises running processes with restart policies.

systemd Commands

# [RPi]
# List running services
systemctl list-units --type=service

# Check a specific service
systemctl status ssh

# Start/stop a service
sudo systemctl start my-app
sudo systemctl stop my-app

# Enable at boot / disable
sudo systemctl enable my-app
sudo systemctl disable bluetooth

MCU (bare-metal)	Linux (systemd)
`main()` runs at power-on	systemd decides what runs
One infinite loop	Multiple services, each a process
No restart on crash	`Restart=always` for auto-recovery
No dependency management	`After=network.target` for order

Designing a systemd Service

Your data logger needs to start at boot, restart on crash, and wait for the network.

[Unit]
Description=Temperature Data Logger
After=network.target

[Service]
Type=simple
ExecStart=/usr/local/bin/data_logger
Restart=on-failure
RestartSec=5
WatchdogSec=30

[Install]
WantedBy=multi-user.target

Field	Purpose
`Type=simple`	Process stays in foreground — systemd tracks its PID
`Restart=on-failure`	Restart automatically if it crashes (exit code ≠ 0)
`RestartSec=5`	Wait 5 seconds before restarting (avoid crash loops)
`WatchdogSec=30`	If the process does not ping systemd within 30 s, kill and restart it
`After=network.target`	Do not start until networking is ready

This is how your data logger stays alive — the init system is your watchdog.

10. Debugging Overview — Three Layers

Layer	What Breaks	Primary Tools
Application	Logic errors, segfaults	GDB, strace, valgrind
System	Services, missing libs	journalctl, systemctl, ldd
Kernel / HW	Driver probes, bus errors	dmesg, printk, JTAG

Start at the top and work down. Most bugs are application-level.

# [RPi]
# Application layer
strace ./sensor_reader           # Trace system calls
gdb ./sensor_reader              # Step through code

# System layer
journalctl -u my-app.service     # Service logs
ldd ./sensor_reader              # Missing libraries?

# Kernel layer
dmesg -T                         # Kernel messages with timestamps
dmesg --level=err,warn           # Only errors and warnings
dmesg -w                         # Follow in real time

Remote Debugging and QEMU

Embedded norm: compile on laptop (x86), debug on Pi (ARM).

 [Host] Your Laptop               [RPi] Raspberry Pi
 ┌──────────────────┐              ┌──────────────────┐
 │  gdb-multiarch   │◄── TCP ────►│  gdbserver :9000  │
 │  ./sensor_reader │   port 9000 │  ./sensor_reader  │
 └──────────────────┘              └──────────────────┘

QEMU — test without hardware: emulates ARM on x86.

Feature	GDB over SSH	QEMU + GDB	JTAG / OpenOCD
Needs target HW	Yes	No	Yes + probe
OS must run	Yes	Yes (user)	No
Debug early boot	No	Yes (system)	Yes
Debug kernel panic	No	No	Yes
Setup complexity	Low	Medium	High

Start with GDB + SSH. Reach for JTAG only when the OS itself is broken.

Shell Scripting Essentials

Scripts on embedded systems often run unattended. Silent failures are dangerous.

# [RPi]
#!/bin/bash
set -euo pipefail
# set -e       → exit immediately if any command fails
# set -u       → treat unset variables as errors
# set -o pipefail → pipeline fails if ANY command fails, not just the last

TEMP=$(cat /sys/class/thermal/thermal_zone0/temp)
TEMP_C=$(( TEMP / 1000 ))

if [[ $TEMP -gt 75000 ]]; then
    echo "WARNING: CPU at ${TEMP_C}C" >&2
fi

Practice	Why
`set -euo pipefail`	Catch errors early
`trap 'cleanup' EXIT`	Release resources on crash
`echo "ERROR" >&2`	Errors go to stderr, not stdout

strace for Embedded Debugging

When a sensor script fails silently, strace shows exactly what happened:

# [RPi]
$ strace -e openat,read,ioctl python3 read_sensor.py
openat(AT_FDCWD, "/dev/i2c-1", O_RDWR) = 3
ioctl(3, I2C_SLAVE, 0x18)              = 0
read(3, "\x01\xa8", 2)                 = 2

Syscall	What It Tells You
`openat` → `= 3`	File opened successfully (FD 3)
`openat` → `= -1 EACCES`	Permission denied — need udev rule or `sudo`
`ioctl(I2C_SLAVE)` → `= -1 EBUSY`	Another process has the device open
`read` → `= -1 EIO`	I2C transaction failed — wiring or address wrong

strace intercepts every system call your program makes. It is the fastest way to answer: "what files did it try to open?" and "which call failed?"

The Big Picture: MCU vs Linux

	MCU (Pico)	Linux (RPi 4)
Programs	1 (your `main()`)	80+ processes
Files	None (flat flash)	Filesystem hierarchy
Permissions	None	Owner/group/other
Hardware access	Direct registers	Through `/dev/`, `/sys/`
Scheduling	Your loop (or RTOS)	CFS + RT policies
Memory	Physical, shared	Virtual, isolated (MMU)
Crash isolation	None — everything dies	Per-process
Boot time	< 100 ms	5-30 s
Debugging	UART printf, SWD	GDB, strace, dmesg

Block 1 Summary

Key takeaways:

Processes — Linux runs dozens of programs simultaneously, each isolated
fork + exec — the two-step pattern for launching programs; threads share memory, processes don't
VFS — one API for all file types; the kernel layer that makes "everything is a file" work
Inodes — metadata lives in the inode, names live in the directory entry; hard vs soft links
FHS — standard directory layout; /dev, /proc, /sys are in memory, not on disk
File descriptors — redirections and pipes are just reassigning FDs; three-table architecture
Permissions — control access to hardware; understand them to avoid "Permission denied"
Kernel vs user space — enforced by MMU hardware, not just convention
Shell tools — pipes, redirects, environment variables are your debugging interface

Block 2

Lab 2 Preview: Exploring Linux

What You Will Do

SSH into your Raspberry Pi and explore the real system:

# [RPi]
# Processes
ps aux | wc -l                    # Count processes
ps aux --sort=-%mem | head -5     # Top memory users
pstree -p                         # Process tree

# Hardware info
cat /proc/cpuinfo | grep model    # CPU model
free -h                           # RAM usage
uname -r                          # Kernel version

# Services
systemctl list-units --type=service --state=running

# Filesystem and permissions
ls /dev/i2c*                      # Find your I2C device
stat /dev/i2c-1                   # Check permissions

# Scheduling
ps -eo pid,ni,comm | grep ssh     # SSH nice value

Fill In This Table

Question	Your Answer
Running processes
Top memory user
CPU model
Free RAM
Running services count
I2C device permissions
Kernel version
SSH nice value
Files in `/dev/`
System load average

Compare with your neighbor. Are the numbers similar? Different? Why?

Quick Checks

Test your understanding:

What is the difference between /dev/, /sys/, and /proc/?
Why does cat /dev/spidev0.0 fail without sudo?
What happens to a process when its parent dies?
Why can Linux not run on a Cortex-M0+ (like the Pi Pico)?
What does the pipe operator | do? How is it different from >?
What are the six steps between typing ./sensor_reader and reaching main()?

Key Takeaways

Linux adds processes, files, permissions, and services on top of hardware
Everything is a file — the single most important Linux abstraction
The MMU enforces kernel/user space isolation in hardware
The shell (pipes, redirects, env vars) is your primary debugging interface
Debugging is layered — start at application level, work down to kernel
Understanding these layers is essential before writing drivers or building custom images

Next teaching block: Lesson 3 — Device Tree & Drivers. How does Linux discover MCP9808 and expose data to apps?

Tutorials: Start with Exploring Linux today. Extensions: Shell Scripting, ELF and Program Execution, Debugging Practices.

Lesson 2: How Linux Works

Today's Map

1. Everything Is a Process

MCU vs Linux: One Program vs Many

How a Program Becomes a Process

The Six Steps to main()

fork() Creates a Copy

fork() Creates a Copy

ELF Format and Program Loading

2. Everything Is a File

The Virtual File System (VFS)

Inside VFS

VFS: One Interface, Many Backends

The Three Pseudo-Filesystems

sysfs Walkthrough

Inodes: What's Under a Filename

Inodes

Hard Links vs Symbolic Links

Device File Ecosystem

udev Rules

3. Filesystem Hierarchy

Key Directories for Embedded

MCU vs Linux: Where Things Live

4. Users and Permissions

Why Permissions Matter for Embedded

5. Kernel vs User Space

Protection Rings and Architecture

Linux Architecture

The System Call Boundary

Why This Separation Matters

glibc and Kernel Architecture

musl vs glibc

6. Shell Tools: Pipes, Redirects, Environment

File Descriptors

FD Architecture: Three Tables

Baseline: Default File Descriptors

Write into a file

Input Redirection: cat file

Output redirection: prog 2>&1

Pipes: cmd1 | cmd2

Try It Now: File Descriptors (5 min) [RPi]

Pipes Are Everywhere

7. How Scheduling Works

The Completely Fair Scheduler (CFS)

Nice levels

Real-Time Scheduling

Priority Hierarchy

8. Virtual Memory and the MMU

MMU

Process memory on 32bit

MMU Enables Isolation

Process vs Thread

Inter-Process Communication

fork() in Code

fork() Memory Separation

Try It Now: fork() (5 min) [Host/RPi]

pipe() — Data Between Processes

Try It Now: pipe() (5 min) [Host/RPi]

Signals

Try It Now: Signals (5 min) [Host/RPi]

Threads — Shared Memory, Shared Risk

POSIX Threads Architecture

Try It Now: Threads (5 min) [Host/RPi]

Condition Variables — "Wait Until Ready"

Producer-Consumer Pattern

Debugging Threads

Choosing an IPC Mechanism

Try It Now: Sensor Monitor (10 min) [Host/RPi]

9. Services and Init (systemd)

systemd Commands

Designing a systemd Service

10. Debugging Overview — Three Layers

Remote Debugging and QEMU

Shell Scripting Essentials

strace for Embedded Debugging

The Big Picture: MCU vs Linux

Block 1 Summary

Block 2

Lab 2 Preview: Exploring Linux

What You Will Do

Input Redirection: `cat file`

Output redirection: `prog 2>&1`

Pipes: `cmd1 | cmd2`

Try It Now: File Descriptors (5 min) `[RPi]`

Try It Now: fork() (5 min) `[Host/RPi]`

Try It Now: pipe() (5 min) `[Host/RPi]`

Try It Now: Signals (5 min) `[Host/RPi]`

Try It Now: Threads (5 min) `[Host/RPi]`

Try It Now: Sensor Monitor (10 min) `[Host/RPi]`