What is real-time Linux? Part III

Welcome to the final post of this three-part series on real-time Linux. As market adoption of real-time computing continues to rise, real-time in the Linux kernel is emerging as a valuable solution on the software side.

A real-time Linux kernel on its own, however, will not necessarily make a system real-time.   We clarified this point in Part I and listed the common misconceptions and stack components. We then concluded Part I by discussing the role of real-time Linux in target verticals and applications, from industrial factories and telco workloads to automotive and healthcare.

Part II tackled how to achieve real-time in the kernel by introducing User Space to Kernel Space transitions and preemption. This is the final chapter of the series, where we will start by discussing the challenges to bringing preemption into the Linux kernel and the key benefits of the PREEMPT_RT real-time patches.

Challenges to implementing preemption

The challenge in making a preemptible kernel lies in identifying all the locations where preemption must not occur. These are critical sections where kernel developers must disable preemption.

In the example scenario in Figure 1 below, two processes with different priority levels operate on shared variables and make decisions based on their values. Process A is accessing and working on some shared data in the kernel. At this point, a higher-priority Process B preempts the running, lower-priority Process A. Process B then changes the value of the shared data structure. A process switch then occurs, with Process A resuming its operations after the prior interruption. Once Process A gets to run again, however, it will make a decision based on a value determined by Process B. In such a scenario, where two processes access the same data structures in critical sections, kernel preemption must be disabled.

The difficult tradeoff is that developers must make the Linux kernel preemptible to reduce latency and achieve real-time computing capabilities. At the same time, they must disable preemption around critical sections to avoid resulting in corrupted data.

Such a balance is hard to strike, and it involves poring over the entire kernel source code base to assess which data to protect from concurrency. For real-time processing in Linux to be a reality, the kernel must be able to preempt the current thread of execution while placing checks at strategic locations. How do Linux developers overcome this riddle?

Preemption options in the Linux kernel

A few preemption models are available in the mainline Linux kernel’s scheduler. In this section, we’ll go over the key features and target applications for each [1].

PREEMPT_NONE

The traditional Linux preemption model maximises the kernel’s raw processing power, irrespective of scheduling latencies.

As the default behaviour in standard kernels, the no-preemption case for server workloads is optimised for overall throughput for systems making intense computations like a server. Whereas PREEMPT_NONE can still provide decent latencies, there are no guarantees, and occasional longer delays can occur, as interrupts, exceptions, and system calls in the kernel code are never preempted.

PREEMPT_VOLUNTARY

The PREEMPT_VOLUNTARY preemption mode provides quicker application reactions to user input and targets desktop use.

A low-priority process can voluntarily preempt itself even when executing a system call in kernel code. By enabling voluntary preemption points in kernel code, PREEMPT_VOLUNTARY reduces the maximum latency of rescheduling at the price of slightly lower throughput. As a result, faster application reactions, even when under load, are perceived to run more smoothly.

PREEMPT

Just like PREEMPT_VOLUNTARY, PREEMPT enables voluntary preemption points in the Linux kernel, but it further makes kernel code preemptible when not executing in a critical section. 

By adding non-critical-section preemption, PREEMPT enables kernel code to be preempted involuntarily. Higher-priority processes can interrupt execution threads in low-priority processes even when executing a system call in kernel mode and when not about to reach a natural preemption point. 

The reduced kernel latencies are suitable for desktop or embedded systems with latency requirements in the milliseconds range and they result in slightly lower throughput and runtime overhead to kernel code.

As per the screen capture, my Linux machine is running the 22.04 LTS release of Ubuntu Desktop with version 5.19 of the Linux kernel. PREEMPT_VOLUNTARY is the default preemption model selected, as the kernel is for a desktop system, whereas CONFIG_PREEMPT is not set.

From the above, PREEMPT is a configuration flavour of mainline, with the highest preemption level available in the kernel. Is that all we can do to achieve real-time in Linux?

PREEMPT_RT

PREEMPT_RT is the name of the patchset hosted at the Linux Foundation to implement a priority scheduler and other supporting real-time mechanisms. Whereas the patches are slowly being mainlined, they are not yet fully upstream in the mainline Linux kernel. The PREEMPT_RT locking code, representing a bulk of the outstanding real-time patches, was merged in Linux 5.15, but there is still more work to be upstreamed. PREEMPT_RT is nowadays the de-facto Linux real-time implementation, despite various mechanisms to bring real-time compute into the Linux kernel having been attempted through the years, like Xenomai and RTAI.

Other approaches to bringing real-time in Linux often implement a ‘co-kernel’ running concurrently with a soft Linux kernel, whereas PREEMPT_RT differs by affecting the kernel itself. The PREEMPT_RT patchset strives for full preemption in the Linux kernel, including in critical sections and except for very low-level and critical code paths like entry code and low-level interrupt handling.

The goal with PREEMPT_RT is to make all code running in kernel mode involuntarily preempted at any time by bringing most execution contexts under scheduler control.

PREEMPT_RT does so by replacing locking primitives like spinlocks with variants aware of priority inheritance, enforcing interrupt threading and introducing mechanisms to break up long non-preemptible sections. Other mechanisms the PREEMPT_RT patch adopts to make the kernel deterministic involve replacing mutexes in the mainline Linux kernel with rt_mutexes, to implement priority inheritance and avoid priority inversion [2].

Whereas PREEMPT_RT doesn’t strive for the lowest latencies possible, an unbounded latency would represent a bug, as PREEMPT_RT makes the Linux kernel preemptible by providing deterministic response times.

Real-time Ubuntu

Ubuntu 22.04 LTS with the out-of-tree PREEMPT_RT patches entered general availability last February. Real-time Ubuntu is more preemptive than mainline Linux. It reduces the kernel latencies as required by the most exacting workloads, and helps ensure a time-predictable task execution. Device manufacturers can focus on their business drivers, shortening time-to-market by relying on an enterprise-grade, real-time Linux kernel supported over 10+ years. 

Enterprise customers can access the real-time kernel via Ubuntu Core, the secure, application-centric IoT OS for embedded devices. Ubuntu Core is Ubuntu for IoT and embedded environments, delivering the resiliency and security that developers around the world expect from Ubuntu. It is an OS with a minimal footprint and a modular and simple architecture, built on snaps, the universal Linux packaging format. 

Alternatively, end-users can deploy Ubuntu Server 22.04 LTS with the real-time kernel, available via Ubuntu Pro, Canonical’s comprehensive enterprise security and compliance subscription, covering all aspects of open infrastructure. A free tier is available for personal and small-scale commercial use, in line with the company’s community commitment and mission to ease open-source access and consumption. 

Canonical’s enterprise-grade software support and long-term maintenance for the Real-time Ubuntu kernel ensure product longevity.

Further reading

As the processing of time-sensitive workloads depends upon the capacity of the Linux system to respond within a bounded interval of time, real-time at the kernel level denotes a deterministic response to an external event. The Real-time Ubuntu kernel will power the next generation of robotics, IoT, and telco innovations by providing a known response time to their stringent low-latency requirements. By minimising the non-preemptible critical sections in kernel code, Ubuntu with the PREEMPT_RT patches makes the kernel more preemptive than mainline.