Arrakis: The Operating System is the Control Plane

What kind of paper is this?

• Hardware/Computing landscape is changing (networks are really fast and some devices provide virtualization), so let's redesign system sotware to this new reality.

The Story

• Once upon a time system overheads were dwarfed by the relatively poor performance of IO. However, today's IO (networking in particular) are both significantly faster and offer virtualization support. Now, the software stack that sits on top of these devices is the bottleneck. If we can give applications direct access to the hardware for such data movement, they can bypass all that kernel overhead, making an important class of servers much faster. And now everyone who uses such services lives happily ever after.

Today's stacks are deeply layered and introduce overhead

• Networking
  • SW demultiplexing
  • Security checks
  • Scheduling
  • Context switching
  • Cache and lock contension
  • Queue management
• Storage (done with fsync) (Used to not be such a big deal when disks were the bottleneck; with faster persistence, CPU time is a greater concern.)
  • Data copies
  • Parameter checking
  • Allocation
  • VFS
  • Snapshots
  • Metadata updates
• Arakis solution: use SR-IOV to remove kernel mediation from the data plane; eliminates some overhead entirely and reduces other overheads. Eschewing POSIX produces true zero-copy.

Virtual IO

• SR-IOV is designed to support multiple VMs sharing an IO device
• Single device (physical function) can be dynamically turned into multiple PCI devices (virtual functions).
• Hypervisor creates the virtual functions and install filters to demultiplex operations to the right virtual function (guest OS).
• Arrakis uses SR-IOV and IOMMUs to give applications (instead of guest VMs) direct access to IO devices.
• Note: This is not so much a new idea as the modern embodiment of an idea that's been around for quite some time.

Goals

• Remove kernel from data plane operations
• Require no changes for application programming (but allow changes to produce even better performance).
• Provide abstractions useful across many devices.
• Enabling technology: While SR-IOV is designed for VMs, in Arrakis, each application can have its own networking stack and virtual device.

Arrakis Architecture

• Assumes virtualized devices (i.e., SR-IOV) -- and their interface presents the abstraction of idealized virtual devices, ideally provided entirely by the hardware.
• IO stacks provided as libraries (a partial libOS).
• Each device interacts with applications through a DMA send-receive queue. Optionally might include device specific operations, e.g., TCP checksum and segmentation.
• Transmit filters (predicates) prevent applications from sending bad packets or avoiding security checks.
• Receive filters (predicates) direct incoming packets to the right virtual device.
• Privileged software can install bandwidth limiters to implement policies among the virtual devices.

Implementation Details