OpenDataPlane (ODP) Users-Guide

As an evolving standard, the ODP API specification is released under an incrementing version number, and corresponding implementations of ODP, as well as the validation suite that verifies API conformance, are linked to this version number. ODP versions are specified using a standard three-level number (major.minor.fixlevel) that are incremented according to the degree of change the level represents. Increments to the fix level represent clarification of the specification or other minor changes that do not affect either the syntax or semantics of the specification. Such changes in the API specification are expected to be rare. Increments to the minor level represent the introduction of new APIs or functional capabilities, or changes to the specified syntax or functional behavior of APIs and thus may require application source code changes. Such changes are well documented in the release notes for each revision of the specification. Finally, increments to the major level represent significant structural changes that most likely require some level of application source code change, again as documented in the release notes for that version.

ODP implementations are free to use whatever release naming/numbering conventions they wish, as long as it is clear what level of the ODP API a given release implements. A recommended convention is to use the same three level numbering scheme where the major and minor numbers correspond to the ODP API level and the fix level represents an implementation-defined service level associated with that API level implementation. The LNG-supplied ODP reference implementations follow this convention.

The ODP validation test suite follows these same naming conventions. The major and minor release numbers correspond to the ODP API level that the suite validates and the fix level represents the service level of the validation suite itself for that API level.

ODP has three primary goals that follow from its component structure. The first is application portability across a wide range of platforms. These platforms differ in terms of processor instruction set architecture, number and types of application processing cores, memory organization, as well as the number and type of platform specific hardware acceleration and offload features that are available. ODP applications can move from one conforming implementation to another with at most a recompile.

Second, ODP is designed to permit data plane applications to avail themselves of platform-specific features, including specialized hardware accelerators, without specialized programming. This is achieved by separating the API specification from their implementation on individual platforms. Since each platform implements each ODP API in a manner optimal to that platform, applications automatically gain the benefit of such optimizations without the need for explicit programming.

Third, ODP is designed to allow applications to scale out automatically to support many core architectures. This is done using an event based programming model that permits applications to be written to be independent of the number of processing cores that are available to realize application function. The result is that an application written to this model does not require redesign as it scales from 4, to 40, to 400 cores.

The thread is the fundamental programming unit in ODP. ODP applications are organized into a collection of threads that perform the work that the application is designed to do. ODP threads may or may not share memory with other threads—​that is up to the implementation. Threads come in two "flavors": control and worker, that are represented by the abstract type odp_thread_type_t.

A control thread is a supervisory thread that organizes the operation of worker threads. Worker threads, by contrast, exist to perform the main processing logic of the application and employ a run to completion model. Worker threads, in particular, are intended to operate on dedicated processing cores, especially in many core processing environments, however a given implementation may multitask multiple threads on a single core if desired (typically on smaller and lower performance target environments).

In addition to thread types, threads have associated attributes such as thread mask and scheduler group that determine where they can run and the type of work that they can handle. These will be discussed in greater detail later.

Events are what threads process to perform their work. Events can represent new work, such as the arrival of a packet that needs to be processed, or they can represent the completion of requests that have executed asynchronously. Events can also represent notifications of the passage of time, or of status changes in various components of interest to the application. Events have an event type that describes what it represents. Threads can create new events or consume events processed by them, or they can perform some processing on an event and then pass it along to another component for further processing. References to events are via handles of abstract type odp_event_t. Cast functions are provided to convert these into specific handles of the appropriate type represented by the event.

A queue is a message passing channel that holds events. Events can be added to a queue via enqueue operations or removed from a queue via dequeue operations. The endpoints of a queue will vary depending on how it is used. Queues come in two major types: plain and scheduled, which will be discussed in more detail when the event model is introduced. Queues may also have an associated context, which represents a persistent state for all events that make use of it. These states are what permit threads to perform stateful processing on events as well as stateless processing.

Queues are represented by handles of abstract type odp_queue_t.

A pool is a shared memory area from which elements may be drawn. Pools represent the backing store for events, among other things. Pools are typically created and destroyed by the application during initialization and termination, respectively, and then used during processing. Pools may be used by ODP components exclusively, by applications exclusively, or their use may be shared between the two. Pools have an associated type that characterizes the elements that they contain. The two most important pool types are Buffer and Packet.

Pools are represented by handles of abstract type odp_pool_t.

Shared memory represents raw blocks of storage that are sharable between threads. They are the building blocks of pools but can be used directly by ODP applications if desired.

Shared memory is represented by handles of abstract type odp_shm_t.

A buffer is a fixed sized block of shared storage that is used by ODP components and/or applications to realize their function. Buffers contain zero or more bytes of application data as well as system maintained metadata that provide information about the buffer, such as its size or the pool it was allocated from. Metadata is an important ODP concept because it allows for arbitrary amounts of side information to be associated with an ODP object. Most ODP objects have associated metadata and this metadata is manipulated via accessor functions that act as getters and setters for this information. Getter access functions permit an application to read a metadata item, while setter access functions permit an application to write a metadata item. Note that some metadata is inherently read only and thus no setter is provided to manipulate it. When object have multiple metadata items, each has its own associated getter and/or setter access function to inspect or manipulate it.

Buffers are represented by handles of abstract type odp_buffer_t.

Packets are received and transmitted via I/O interfaces and represent the basic data that data plane applications manipulate. Packets are drawn from pools of type ODP_POOL_PACKET. Unlike buffers, which are simple objects, ODP packets have a rich set of semantics that permit their inspection and manipulation in complex ways to be described later. Packets also support a rich set of metadata as well as user metadata. User metadata permits applications to associate an application-determined amount of side information with each packet for its own use.

Packets are represented by handles of abstract type odp_packet_t.

PktIO is how ODP represents I/O interfaces. A pktio object is a logical port capable of receiving (RX) and/or transmitting (TX) packets. This may be directly supported by the underlying platform as an integrated feature, or may represent a device attached via a PCIE or other bus.

PktIOs are represented by handles of abstract type odp_pktio_t.

The time API is used to measure time intervals and track time flow of an application and presents a convenient way to get access to an implementation-defined time source. The time API consists of two main parts: local time API and global time API.

Timers are how ODP applications measure and respond to the passage of time. Timers are drawn from specialized pools called timer pools that have their own abstract type (odp_timer_pool_t). Applications may have many timers active at the same time and can set them to use either relative or absolute time. When timers expire they create events of type odp_timeout_t, which serve as notifications of timer expiration.

Multiple threads operating in parallel typically require various synchronization services to permit them to operate in a reliable and coordinated manner. ODP provides a rich set of locks, barriers, and similar synchronization primitives, as well as abstract types for representing various types of atomic variables. The ODP event model also makes use of queues to avoid the need for explicit locking in many cases. This will be discussed in the next section.

The Classifier provides a suite of APIs that control packet receive (RX) processing.

The classifier provides two logically related services:

Combined, these permit incoming packets to be sorted into flows, which are logically related sequences of packets that share common processing requirements. While many data plane applications perform stateless packet processing (e.g., for simple forwarding) others perform stateful packet processing. Flows anchor state information relating to these groups of packets.

A CoS determines two variables for packets belonging to a flow:

The PMRs supported by ODP permit flow determination based on combinations of packet field values (tuples). The main advantage of classification is that on many platforms these functions are performed in hardware, meaning that classification occurs at line rate as packets are being received without any explicit processing by the ODP application.

Note that the use of the classifier is optional. Applications may directly receive packets from a corresponding PktIO input queue via direct polling if they choose.

The Scheduler provides a suite of APIs that control scalable event processing.

The Scheduler is responsible for selecting and dispatching one or more events to a requesting thread. Event selection is based on several factors involving both the queues containing schedulable events and the thread making an odp_schedule() or odp_schedule_multi() call.

ODP queues have a scheduling priority that determines how urgently events on them should be processed relative to events contained in other queues. Queues also have a scheduler group id associated with them that must match the associated scheduler group thread mask of the thread calling the scheduler. This permits events to be grouped for processing into classes and have threads that are dedicated to processing events from specified classes. Threads can join and leave scheduler groups dynamically, permitting easy application response to increases in demand.

When a thread receives an event from the scheduler, it in turn can invoke other processing engines via ODP APIs (e.g., crypto processing) that can operate asynchronously. When such processing is complete, the result is that a completion event is added to a schedulable queue where it can be scheduled back to a thread to continue processing with the results of the requested asynchronous operation.

Threads themselves can enqueue events to queues for downstream processing by other threads, permitting flexibility in how applications structure themselves to maximize concurrency.

The Traffic Manager provides a suite of APIs that control traffic shaping and Quality of Service (QoS) processing for packet output.