Skip to content

Jitter Measurement and RT Comparison

Time estimate: ~45 minutes Prerequisites: Level Display: SDL2, PREEMPT_RT Latency

Learning Objectives

By the end of this tutorial you will be able to:

  • Quantify sensor jitter and frame jitter with statistical analysis
  • Compare standard vs PREEMPT_RT kernel performance
  • Understand why worst-case matters more than average for real-time
  • Use CPU stress to reveal scheduling differences
Understanding Jitter Measurement

Jitter is the variation in timing between consecutive events that should be equally spaced. For example, a loop targeting 1 ms per iteration that actually varies between 0.8 ms and 3.2 ms has 2.4 ms of jitter. In embedded systems, jitter matters because a single late event can cause a visible stutter, a missed control deadline, or a sensor reading gap. The tool cyclictest measures worst-case scheduling latency by repeatedly sleeping for a fixed interval and recording how late the wakeup actually occurs. On a standard Linux kernel, worst-case latency under load can reach several milliseconds. With the PREEMPT_RT patch, nearly all kernel code becomes preemptible, reducing worst-case latency to tens of microseconds. Plotting a histogram of measured latencies reveals the distribution — a tight cluster near the target with a long tail indicates occasional scheduling delays. The length of that tail is what determines whether the system meets real-time requirements.

For the full theory on real-time scheduling, preemption models, and latency analysis, see the Real-Time Systems Reference.

1. Enable CSV Logging

Run the SDL2 level display application with CSV logging enabled:

sudo SDL_VIDEODRIVER=kmsdrm ./level_sdl2 -l baseline.csv

Each row in the CSV contains:

Column Unit Description
timestamp_ns ns Absolute monotonic timestamp
sensor_dt_ns ns Delta between consecutive sensor reads
frame_dt_ns ns Delta between consecutive frame presentations
latency_ns ns Sensor-to-display latency
roll_deg degrees Filtered roll angle
pitch_deg degrees Filtered pitch angle

The sensor_dt_ns reveals how consistently the sensor thread runs. The frame_dt_ns shows how consistently the render loop meets VSync deadlines. The latency_ns measures the total pipeline delay from reading the sensor to presenting the frame.

Note

For the statistical theory behind jitter analysis — why worst-case matters, percentile definitions, how many samples you need, and outlier classification — see Real-Time Systems Reference § Jitter Statistics.

2. Baseline Test

First, establish a baseline on the standard kernel with no competing load.

Close any unnecessary applications and services:

sudo systemctl stop lightdm 2>/dev/null  # stop desktop if running

Run the test for 2 minutes:

sudo SDL_VIDEODRIVER=kmsdrm ./level_sdl2 -l baseline.csv &
sleep 120
kill %1

Verify the data:

wc -l baseline.csv
head -3 baseline.csv

You should have approximately 7,200 data points (60 fps x 120 seconds).

Checkpoint

baseline.csv exists and contains roughly 7,000+ rows of data.

3. Stress Test

Now repeat the measurement while generating CPU load. The stress-ng tool creates synthetic load on specific cores:

sudo apt install stress-ng

Run the stress test on 3 out of 4 CPU cores, leaving one core for the application:

stress-ng --cpu 3 &
STRESS_PID=$!

sudo SDL_VIDEODRIVER=kmsdrm ./level_sdl2 -l stress.csv &
APP_PID=$!

sleep 120

kill $APP_PID
kill $STRESS_PID

During the test, you can observe CPU load in another terminal:

mpstat -P ALL 1

You should see 3 cores at 100% and one core running the level display application.

Checkpoint

stress.csv exists with roughly 7,000+ rows. You may have noticed occasional visual stutters during the stress test.

Stuck?
  • If stress-ng is not available, use stress or run while true; do :; done & three times
  • If the app crashes under load, check available memory with free -m

4. Install PREEMPT_RT

The PREEMPT_RT kernel patch transforms Linux from a general-purpose OS to a real-time capable system. It makes nearly all kernel code preemptible, reducing worst-case latency from milliseconds to microseconds.

Follow the existing PREEMPT_RT Latency tutorial to install and boot the RT kernel. Then return here.

Verify you are running the RT kernel:

uname -r

The kernel version should contain rt in the name (e.g., 6.1.77-rt24-v8+).

cat /sys/kernel/realtime

This should output 1 on an RT kernel.

Checkpoint

uname -r shows an -rt kernel version and /sys/kernel/realtime reads 1.

5. RT Baseline

Repeat the baseline test on the RT kernel with no load:

sudo SDL_VIDEODRIVER=kmsdrm ./level_sdl2 -l rt_baseline.csv &
sleep 120
kill %1
Tip

Make sure you have rebuilt or copied the level_sdl2 binary after switching kernels if the kernel headers changed. SDL2 user-space binaries typically do not need rebuilding, but the BMI160 kernel module does.

6. RT Stress Test

Apply the same CPU stress on the RT kernel:

stress-ng --cpu 3 &
STRESS_PID=$!

sudo SDL_VIDEODRIVER=kmsdrm ./level_sdl2 -l rt_stress.csv &
APP_PID=$!

sleep 120

kill $APP_PID
kill $STRESS_PID

You now have four CSV files representing the complete test matrix:

File Kernel Load
baseline.csv Standard None
stress.csv Standard 3-core stress
rt_baseline.csv PREEMPT_RT None
rt_stress.csv PREEMPT_RT 3-core stress

7. Run Analysis Script

The analysis script computes statistics and generates comparison plots:

python3 src/embedded-linux/scripts/jitter-measurement/analyze_jitter.py \
    baseline.csv stress.csv rt_baseline.csv rt_stress.csv \
    --plot

This produces:

  • A histogram of sensor dt and frame dt for each test condition
  • A time series plot showing jitter spikes
  • A summary table printed to the console
  • Plots saved as jitter_analysis.png
Stuck?
  • Install dependencies if needed: pip3 install numpy matplotlib
  • If --plot fails on headless Pi, add --save-only to write plots to file without displaying
  • You can also analyze manually with simple shell commands: 
    # Average frame dt in ms
awk -F',' 'NR>1 {sum+=$3; n++} END {print sum/n/1e6}' baseline.csv
# Max frame dt in ms
awk -F',' 'NR>1 {if($3>max) max=$3} END {print max/1e6}' baseline.csv

8. Fill Comparison Table

Using the output from the analysis script, fill in this table with your measured values:

Metric Standard Standard + Load RT RT + Load
Sensor dt avg (us)
Sensor dt max (us)
Frame dt avg (ms)
Frame dt max (ms)
Dropped frames (>25 ms)
Latency 99th pct (ms)
Tip

Typical results you might expect:

  • Sensor dt avg should be similar across all conditions (~5000 us)
  • Sensor dt max is where RT shines: standard kernel under load might show spikes of 10,000+ us, while RT stays below 6,000 us
  • Frame dt avg should be ~16.6 ms everywhere (hardware-paced by VSync)
  • Frame dt max shows dropped frames — standard + load might hit 33+ ms, RT + load stays closer to 17 ms
  • Dropped frames count tells the practical story
Checkpoint

Your comparison table is filled with measured data. The RT kernel should show noticeably better worst-case numbers, especially under load.

What Just Happened?

RT kernel reduces worst-case sensor jitter but may not improve average frame timing (display is hardware-paced). Under load, the difference becomes dramatic — RT keeps the sensor loop deterministic while the standard kernel shows spikes.

Why does this matter? In real-time systems, worst-case matters more than average. A system that averages 5 ms sensor reads but occasionally spikes to 50 ms is worse than one that consistently delivers 6 ms reads. The spike causes a visible stutter or, in safety-critical systems, a missed control deadline.

The frame timing is less affected because SDL_RenderPresent() with VSync is fundamentally hardware-paced. The display controller generates VBlank interrupts at a fixed 60 Hz regardless of CPU load. However, if the render thread misses its deadline (takes more than 16.6 ms to prepare a frame), you get a dropped frame — and the standard kernel drops more frames under load than the RT kernel.

Challenges

Challenge: CPU Isolation

Add isolcpus=1,2,3 to the kernel command line (/boot/firmware/cmdline.txt), reboot, and pin the sensor thread to an isolated core: 

sudo taskset -c 1 SDL_VIDEODRIVER=kmsdrm ./level_sdl2 -l isolated.csv
stress-ng --cpu 3 --taskset 0,2,3 &
Compare the isolated-core results to your RT + Load results. Does CPU isolation alone match or beat the RT kernel?

Deliverable

  • [ ] Filled comparison table with measured values from all four test conditions
  • [ ] Histogram plots from the analysis script (jitter_analysis.png)
  • [ ] Written explanation (2-3 sentences) of why worst-case jitter matters more than average in embedded systems

Course Overview | Next: SPI DMA Optimization →