Skip to content

Data Logger Appliance (Read‑Only Root + Logging)

Time estimate: ~30 minutes Prerequisites: SSH Login, Enable I2C

Learning Objectives

By the end of this tutorial you will be able to:

  • Configure a read-only root filesystem with overlayfs
  • Build a data logging script that survives power loss
  • Create a systemd service for automatic startup
  • Explain why read-only root is standard in embedded products
Reliability in Embedded Linux: Read-Only Root, Watchdogs, and Updates

Embedded devices lose power unexpectedly -- lightning, breaker trips, cable pulls. A writable root filesystem can corrupt on unclean shutdown, leaving the device unable to boot. The standard solution is overlayfs with a read-only root:

  • The lower layer (rootfs on SD card) is mounted read-only -- it cannot be corrupted by writes or power loss
  • An upper layer (RAM-backed tmpfs) captures runtime changes using copy-on-write -- modifications never touch the original files
  • On reboot, the RAM overlay is discarded and the system starts clean from the known-good base image
  • Persistent data (sensor logs) is written to a separate writable partition with explicit flush() and fsync() calls to guarantee durability

Even with a reliable filesystem, software can hang. A hardware watchdog is a timer built into the SoC that forces a hard reboot if your application stops "kicking" it within the timeout period. Combined with systemd's service-level watchdog (WatchdogSec=), this creates two layers of automatic recovery.

For field updates, the A/B partition layout writes new firmware to an inactive partition, verifies its checksum, then switches boot. If the new image fails, the watchdog triggers a rollback to the previous known-good partition -- the device is never bricked by a bad update.

For a deeper treatment of reliability patterns, see the Reliability, Updates, and Watchdogs reference.

Introduction

Embedded devices in the field lose power unexpectedly — lightning strikes, breaker trips, cable pulls, battery depletion. A standard Linux filesystem mounted read-write can corrupt on unclean shutdown: journals may be incomplete, writes may be partial, and the system may not boot next time.

The solution is overlayfs with a read-only root:

  • The lower layer (rootfs) is mounted read-only — it cannot be corrupted by writes or power loss
  • An upper layer (writable overlay, usually in RAM via tmpfs) captures runtime changes using copy-on-write: when a process modifies a file, overlayfs copies it from the lower layer to the upper layer and applies the change there — the original file is never touched
  • On reboot, the upper layer (in RAM) disappears and the system starts clean from the known-good lower layer
  • Persistent data (like sensor logs) is written to a separate writable partition with explicit flush() and fsync() calls

This architecture is used in nearly every commercial embedded Linux product — from routers to industrial gateways to vehicle computers.

1. Read‑Only Root + Overlay

Concept: Read‑only rootfs prevents corruption; overlay provides a writable layer.

Enable overlayfs for writable areas while keeping rootfs read‑only.

On Raspberry Pi OS:

sudo raspi-config

Enable Overlay FS (read‑only root).

Reboot.

Checkpoint

After reboot, verify root is read-only: 

mount | grep "on / "
You should see ro (read-only) in the mount options. If you see rw, the overlay was not enabled correctly.

Stuck?
  • "Can't write to filesystem" — this is expected! The root filesystem is now read-only. Writable data must go to an explicitly writable location.
  • Need to make changes? — temporarily disable the overlay in raspi-config to modify system files, then re-enable it.

2. Logging Script

Concept: A simple logger proves the data path end‑to‑end.

python3 - <<'PY'
import time

log = "/home/pi/logs/data.csv"
with open(log, "a") as f:
    f.write("ts,temp_c\n")
    for _ in range(100):
        with open("/sys/class/thermal/thermal_zone0/temp") as t:
            temp = int(t.read())/1000
        f.write(f"{time.time()},{temp:.2f}\n")
        f.flush()
        time.sleep(1)
PY
Warning

The f.flush() call is critical. Without it, data stays in Python's buffer and may be lost on power cut. For even stronger guarantees, add os.fsync(f.fileno()) after flush to force the OS to write to disk.

Checkpoint

After the logger runs for a few seconds, verify the CSV has data: 

head /home/pi/logs/data.csv
You should see a header line and timestamped temperature readings.

3. Add systemd Service

Concept: systemd ensures the logger survives reboots and crashes.

Create data-logger.service:

[Unit]
Description=Data Logger Appliance
After=multi-user.target

[Service]
ExecStart=/usr/bin/python3 /home/pi/logs/logger.py
Restart=always

[Install]
WantedBy=multi-user.target

Enable:

sudo systemctl enable data-logger
sudo systemctl start data-logger
Stuck?
  • "Service won't start" — check logs with journalctl -u data-logger
  • "Permission denied" on log directory — ensure the writable partition is mounted and the service user can write to it
  • Logs disappear after reboot — the log directory must be on a writable partition, not the overlay (which is cleared on reboot)

What Just Happened?

You built a sensor logging appliance with the reliability pattern used in commercial embedded products:

graph LR
    A[SD Card: rootfs] -->|read-only| B[overlayfs merge]
    C[RAM: overlay] -->|writable, volatile| B
    D[Separate partition] -->|writable, persistent| E[data.csv]
    B --> F[Running system]
  • The system is protected from corruption (read-only root)
  • Runtime changes (temp files, caches) live in the volatile overlay
  • Sensor data is explicitly written to persistent storage with proper flushing

This is why your WiFi router still boots after a power outage — its firmware uses the same pattern.

Challenges

Challenge 1: Real Sensor Data

Replace the thermal zone reading with an actual I2C sensor (MCP9808) using smbus: 

import smbus
bus = smbus.SMBus(1)
raw = bus.read_word_data(0x18, 0x05)
raw = ((raw << 8) & 0xFF00) + (raw >> 8)
temp = (raw & 0x0FFF) / 16.0

Challenge 2: Power-Cut Test

With the logger running, pull the power cable. Reconnect and verify: - System boots cleanly (read-only root is intact) - Last few log entries may be missing (expected — they were in the write buffer) - All data before the last flush is preserved

Deliverable

  • Log file (data.csv) with at least 100 entries
  • Proof of recovery after power loss: system boots, previous data intact
  • Brief explanation: Why is f.flush() not enough for guaranteed durability? (Hint: OS write cache)

Course Overview | Next: Debugging Practices →