OpenDataPlane (ODP) Users-Guide

As an evolving standard, the ODP API specification is released under an incrementing version number, and corresponding implementations of ODP, as well as the validation suite that verifies API conformance, are linked to this version number. ODP versions are specified using a standard three-level number (major.minor.fixlevel) that are incremented according to the degree of change the level represents. Increments to the fix level represent clarification of the specification or other minor changes that do not affect either the syntax or semantics of the specification. Such changes in the API specification are expected to be rare. Increments to the minor level represent the introduction of new APIs or functional capabilities, or changes to the specified syntax or functional behavior of APIs and thus may require application source code changes. Such changes are well documented in the release notes for each revision of the specification. Finally, increments to the major level represent significant structural changes that most likely require some level of application source code change, again as documented in the release notes for that version.

ODP implementations are free to use whatever release naming/numbering conventions they wish, as long as it is clear what level of the ODP API a given release implements. A recommended convention is to use the same three level numbering scheme where the major and minor numbers correspond to the ODP API level and the fix level represents an implementation-defined service level associated with that API level implementation. The LNG-supplied ODP reference implementations follow this convention.

The ODP validation test suite follows these same naming conventions. The major and minor release numbers correspond to the ODP API level that the suite validates and the fix level represents the service level of the validation suite itself for that API level.

ODP has three primary goals that follow from its component structure. The first is application portability across a wide range of platforms. These platforms differ in terms of processor instruction set architecture, number and types of application processing cores, memory organization, as well as the number and type of platform specific hardware acceleration and offload features that are available. ODP applications can move from one conforming implementation to another with at most a recompile.

Second, ODP is designed to permit data plane applications to avail themselves of platform-specific features, including specialized hardware accelerators, without specialized programming. This is achieved by separating the API specification from their implementation on individual platforms. Since each platform implements each ODP API in a manner optimal to that platform, applications automatically gain the benefit of such optimizations without the need for explicit programming.

Third, ODP is designed to allow applications to scale out automatically to support many core architectures. This is done using an event based programming model that permits applications to be written to be independent of the number of processing cores that are available to realize application function. The result is that an application written to this model does not require redesign as it scales from 4, to 40, to 400 cores.

The thread is the fundamental programming unit in ODP. ODP applications are organized into a collection of threads that perform the work that the application is designed to do. ODP threads may or may not share memory with other threads—​that is up to the implementation. Threads come in two "flavors": control and worker, that are represented by the abstract type odp_thread_type_t.

A control thread is a supervisory thread that organizes the operation of worker threads. Worker threads, by contrast, exist to perform the main processing logic of the application and employ a run to completion model. Worker threads, in particular, are intended to operate on dedicated processing cores, especially in many core processing environments, however a given implementation may multitask multiple threads on a single core if desired (typically on smaller and lower performance target environments).

In addition to thread types, threads have associated attributes such as thread mask and scheduler group that determine where they can run and the type of work that they can handle. These will be discussed in greater detail later.

Events are what threads process to perform their work. Events can represent new work, such as the arrival of a packet that needs to be processed, or they can represent the completion of requests that have executed asynchronously. Events can also represent notifications of the passage of time, or of status changes in various components of interest to the application. Events have an event type that describes what it represents. Threads can create new events or consume events processed by them, or they can perform some processing on an event and then pass it along to another component for further processing. References to events are via handles of abstract type odp_event_t. Cast functions are provided to convert these into specific handles of the appropriate type represented by the event.

A queue is a message passing channel that holds events. Events can be added to a queue via enqueue operations or removed from a queue via dequeue operations. The endpoints of a queue will vary depending on how it is used. Queues come in two major types: plain and scheduled, which will be discussed in more detail when the event model is introduced. Queues may also have an associated context, which represents a persistent state for all events that make use of it. These states are what permit threads to perform stateful processing on events as well as stateless processing.

Queues are represented by handles of abstract type odp_queue_t.

A pool is a shared memory area from which elements may be drawn. Pools represent the backing store for events, among other things. Pools are typically created and destroyed by the application during initialization and termination, respectively, and then used during processing. Pools may be used by ODP components exclusively, by applications exclusively, or their use may be shared between the two. Pools have an associated type that characterizes the elements that they contain. The two most important pool types are Buffer and Packet.

Pools are represented by handles of abstract type odp_pool_t.

Shared memory represents raw blocks of storage that are sharable between threads. They are the building blocks of pools but can be used directly by ODP applications if desired.

Shared memory is represented by handles of abstract type odp_shm_t.

A buffer is a fixed sized block of shared storage that is used by ODP components and/or applications to realize their function. Buffers contain zero or more bytes of application data as well as system maintained metadata that provide information about the buffer, such as its size or the pool it was allocated from. Metadata is an important ODP concept because it allows for arbitrary amounts of side information to be associated with an ODP object. Most ODP objects have associated metadata and this metadata is manipulated via accessor functions that act as getters and setters for this information. Getter access functions permit an application to read a metadata item, while setter access functions permit an application to write a metadata item. Note that some metadata is inherently read only and thus no setter is provided to manipulate it. When object have multiple metadata items, each has its own associated getter and/or setter access function to inspect or manipulate it.

Buffers are represented by handles of abstract type odp_buffer_t.

Packets are received and transmitted via I/O interfaces and represent the basic data that data plane applications manipulate. Packets are drawn from pools of type ODP_POOL_PACKET. Unlike buffers, which are simple objects, ODP packets have a rich set of semantics that permit their inspection and manipulation in complex ways to be described later. Packets also support a rich set of metadata as well as user metadata. User metadata permits applications to associate an application-determined amount of side information with each packet for its own use.

Packets are represented by handles of abstract type odp_packet_t.

PktIO is how ODP represents I/O interfaces. A pktio object is a logical port capable of receiving (RX) and/or transmitting (TX) packets. This may be directly supported by the underlying platform as an integrated feature, or may represent a device attached via a PCIE or other bus.

PktIOs are represented by handles of abstract type odp_pktio_t.

The time API is used to measure time intervals and track time flow of an application and presents a convenient way to get access to an implementation-defined time source. The time API consists of two main parts: local time API and global time API.

Timers are how ODP applications measure and respond to the passage of time. Timers are drawn from specialized pools called timer pools that have their own abstract type (odp_timer_pool_t). Applications may have many timers active at the same time and can set them to use either relative or absolute time. When timers expire they create events of type odp_timeout_t, which serve as notifications of timer expiration.

Multiple threads operating in parallel typically require various synchronization services to permit them to operate in a reliable and coordinated manner. ODP provides a rich set of locks, barriers, and similar synchronization primitives, as well as abstract types for representing various types of atomic variables. The ODP event model also makes use of queues to avoid the need for explicit locking in many cases. This will be discussed in the next section.

The Classifier provides a suite of APIs that control packet receive (RX) processing.

The classifier provides two logically related services:

Combined, these permit incoming packets to be sorted into flows, which are logically related sequences of packets that share common processing requirements. While many data plane applications perform stateless packet processing (e.g., for simple forwarding) others perform stateful packet processing. Flows anchor state information relating to these groups of packets.

A CoS determines two variables for packets belonging to a flow:

The PMRs supported by ODP permit flow determination based on combinations of packet field values (tuples). The main advantage of classification is that on many platforms these functions are performed in hardware, meaning that classification occurs at line rate as packets are being received without any explicit processing by the ODP application.

Note that the use of the classifier is optional. Applications may directly receive packets from a corresponding PktIO input queue via direct polling if they choose.

The Scheduler provides a suite of APIs that control scalable event processing.

The Scheduler is responsible for selecting and dispatching one or more events to a requesting thread. Event selection is based on several factors involving both the queues containing schedulable events and the thread making an odp_schedule() or odp_schedule_multi() call.

ODP queues have a scheduling priority that determines how urgently events on them should be processed relative to events contained in other queues. Queues also have a scheduler group id associated with them that must match the associated scheduler group thread mask of the thread calling the scheduler. This permits events to be grouped for processing into classes and have threads that are dedicated to processing events from specified classes. Threads can join and leave scheduler groups dynamically, permitting easy application response to increases in demand.

When a thread receives an event from the scheduler, it in turn can invoke other processing engines via ODP APIs (e.g., crypto processing) that can operate asynchronously. When such processing is complete, the result is that a completion event is added to a schedulable queue where it can be scheduled back to a thread to continue processing with the results of the requested asynchronous operation.

Threads themselves can enqueue events to queues for downstream processing by other threads, permitting flexibility in how applications structure themselves to maximize concurrency.

The Traffic Manager provides a suite of APIs that control traffic shaping and Quality of Service (QoS) processing for packet output.

The final stage of packet processing is to transmit it. Here, applications have several choices. As with RX processing, applications may send packets directly to PktIO TX queues for direct transmission. Often, however, applications need to perform traffic shaping and related Quality of Service (QoS) processing on the packets comprising a flow as part of transmit processing. To handle this need, ODP provides a suite of Traffic Manager APIs that permit programmatic establishment of arbiters, shapers, etc. that control output packet processing to achieve desired QoS goals. Again, the advantage here is that on many platforms traffic management functions are implemented in hardware, permitting transparent offload of this work.

Applications only include the 'include/odp_api.h' file, which includes the 'platform/<implementation name>/include/odp/api' files to provide a complete definition of the API on that platform. The doxygen documentation defining the behavior of the ODP API is all contained in the public API files, and the actual definitions for an implementation will be found in the per platform directories. Per-platform data that might normally be a #define can be recovered via the appropriate access function if the #define is not directly visible to the application.

The first API that must be called by an ODP application is odp_init_global():

This takes two pointers. The first, odp_init_t, contains ODP initialization data that is platform independent and portable, while the second, odp_platform_init_t, is passed unparsed to the implementation to be used for platform specific data that is not yet, or may never be suitable for the ODP API. Each of these parameters is optional and may be specified as NULL to accept the implementation-defined default initialization values.

Calling odp_init_global() establishes the ODP API framework and MUST be called before any other ODP API may be called. Note that it is only called once per application. A successful call to odp_init_global() returns rc = 0 and sets the instance variable supplied as input to the call to a handle representing this unique ODP instance.

The odp_init_t parameter is used to specify various customizations to the ODP environment being established by this call. For example, the caller can specify the maximum number of worker threads it will use, the thread masks associated with these threads, as well as whether the default logging or abort functions are to be overridden with an application-supplied handler.

The application may also provide optimization hints to the ODP implementation if it knows that it will never use specific ODP feature sets, such as the packet classifier or traffic manager. Implementations may use such hints to provide optimized behavior to applications that are known not to need these features.

Following global initialization, each thread in turn calls 'odp_init_local()'. This establishes the local ODP thread context for that thread and MUST be called before other ODP APIs may be called by that thread. The sole argument to this call is the instance variable returned by odp_init_global().

Shutdown is the logical reverse of the initialization procedure, with odp_term_local() called for each thread before odp_term_global() is called to terminate ODP.

ODP Applications follow the general structure flow shown below:

ODP resources are represented via handles that have abstract type odp_resource_t. So pools are represented by handles of type odp_pool_t, queues by handles of type odp_queue_t, etc. Each such type has a distinguished type ODP_RESOURCE_INVALID that is used to indicate a handle that does not refer to a valid resource of that type. Resources are typically created via an API named odp_resource_create() that returns a handle of type odp_resource_t that represents the created object. This returned handle is set to ODP_RESOURCE_INVALID if, for example, the resource could not be created due to resource exhaustion. Invalid resources do not necessarily represent error conditions. For example, ODP_EVENT_INVALID in response to an odp_queue_deq() call to get an event from a queue simply indicates that the queue is empty.

Unless specifically noted in the API, all ODP resources are global to the ODP application, whether it runs as a single process or multiple processes. ODP handles therefore have common meaning within an ODP application but have no meaning outside the scope of the application.

Many ODP resource objects, such as pools and queues, support an application-specified character string name that is associated with an ODP object at create time. This name serves two purposes: documentation, and lookup. The lookup function is particularly useful to allow an ODP application that is divided into multiple processes to obtain the handle for the common resource.

Because ODP offers a programming framework rather than a programming environment, it is designed to be able to work alongside APIs offered by other frameworks with minimal interference. Therefore when we speak of portability in an ODP context, we of necessity speak of portability of those portions of the application that make use of ODP APIs. If an application uses non-ODP APIs then those must be taken into consideration as well when assessing the portability of the entire application. For many applications, it suffices to isolate certain non-portable code to a few areas of the application with the result that the application is significantly more portable than it would be without using ODP. Especially when dealing with existing applications that run in production environments, ODP may well be introduced in an incremental manner with the result being that the application becomes more portable only over time.

ODP has been designed to support both source and binary portability. Source portability is intrinsic to the ODP API specification itself. Any application written to the ODP API specification will be source portable between any conforming ODP implementation with at most a recompile. This is because ODP APIs do not expose implementation details or internal structures that may vary from platform to platform.

For platforms that share a common Instruction Set Architecture (ISA), ODP can also offer binary portability via the specification of an Application Binary Interface (ABI). This is especially useful in a Network Function Virtualization (NFV) environment where a data plane application may be developed and compiled on one platform for distribution and then deployed on many different platforms by an NFV Orchestrator function.

To assist in meeting these needs, ODP offers two distinct application profiles that are designed to characterize the needs of different types of data plane applications: the Embedded Profile and the Cloud Profile.

An ABI consists of several conventions that ensure that a program compiled against one ODP implementation can run unchanged on another platform that has a possibly very different ODP implementation without requiring recompilation. These include:

Note that an ABI definition exists within a specific Instruction Set Architecture (ISA), such as x86-64 or AArch64. Binaries cannot directly port between ISAs—​that requires a recompilation.

Each ODP implementation will identify which ABI definition it supports, if any. When compiling against an ODP implementation in ABI compatibility mode, the resulting binary is automatically binary compatible with all other ODP implementations that share this ABI. For example, for the x86-64 ISA, both the odp-linux and odp-dpdk implementations are a common ABI.

Blocks of shared memory can be created using the odp_shm_reserve() API call. The call expects a shared memory block name, a block size, an alignment requirement, and optional flags as parameters. It returns a odp_shm_t handle. The size and alignment requirement are given in bytes. The provided name does not have to be unique, i.e. a given name can be used multiple times, when reserving different blocks.

The returned odp_shm_t handle can then be used to retrieve the actual address (in the caller’s ODP thread virtual address space) of the created shared memory block.

The address returned by odp_shm_addr() is normally valid only in the calling ODP thread space: odp_shm_t handles can be shared between ODP threads and remain valid within any threads, whereas the address returned by odp_shm_addr(shm) may differ from ODP threads to ODP threads (for the same 'shm' block), and should therefore not be shared between ODP threads. For instance, it would be correct to send a shm handle using IPC between two ODP threads and let each of these thread do their own odp_shm_addr() to get the block address. Directly sending the address returned by odp_shm_addr() from one ODP thread to another would however possibly fail (the address may make no sense in the receiver address space).

The address returned by odp_shm_addr() is nevertheless guaranteed to be aligned according to the alignment requirements provided at block creation time, even if the call to odp_shm_addr() is performed by a different ODP thread than the one which originally called odp_shm_reserve().

All shared memory blocks are contiguous in any ODP thread addressing space: 'address' to 'address'\+'size' (where 'size' is the shared memory block size, as provided in the odp_shm_reserve() call) is read and writeable and mapping the shared memory block. There is no fragmentation.

The exception to this rule is if the odp_shm_t is created with the ODP_SHM_SINGLE_VA flag. This requests that odp_shm_addr() return the same virtual address for all ODP threads in this instance. Note that there may be a performance cost or shm size limit associated with providing this function in some implementations.

By default ODP threads are assumed to behave as cache coherent systems: Any change performed on a shared memory block is guaranteed to eventually become visible to other ODP threads sharing this memory block. Nevertheless, there is no implicit memory barrier associated with any action on shared memories: When a change performed by an ODP thread becomes visible to another ODP thread is not known: An application using shared memory blocks has to use some memory barrier provided by ODP to guarantee shared data validity between ODP threads.

The virtual address at which a given memory block is mapped in different ODP threads may differ from ODP thread to ODP thread, if ODP threads have separate virtual spaces (for instance if ODP threads are implemented as processes). However, the ODP_SHM_SINGLE_VA flag can be used at odp_shm_reserve() time to guarantee address uniqueness in all ODP threads, regardless of their implementation or creation time.

As mentioned, shared memory handles can be sent from ODP threads to ODP threads using any IPC mechanism, and then the block address retrieved. A simpler approach to get the shared memory block handle of an already created block is to use the odp_shm_lookup() API function call. This nevertheless requires the calling ODP thread to provide the name of the shared memory block: odp_shm_lookup() will return ODP_SHM_INVALID if no shared memory block with the provided name is known by ODP. When multiple blocks were reserved using the same name, the lookup function will return the handle of any of these blocks.

Freeing shared memory is performed using the odp_shm_free() API call. odp_shm_free() takes one single argument, the shared memory block handle. Any ODP thread is allowed to perform a odp_shm_free() on a shared memory block (i.e. the thread performing the odp_shm_free() may be different from the thread which did the odp_shm_reserve()). Shared memory blocks should be freed only once, and once freed, a shared memory block should no longer be referenced by any ODP threads.

ODP provides ways of sharing memory with entities located outside ODP instances:

Sharing a block of memory with an external (non ODP) thread is achieved by setting the ODP_SHM_PROC flag at odp_shm_reserve() time. How the memory block is retrieved on the Operating System side is implementation and Operating System dependent.

Sharing a block of memory with an external ODP instance (running on the same Operating System) is achieved by setting the ODP_SHM_EXPORT flag at odp_shm_reserve() time. A block of memory created with this flag in an ODP instance A, can be "mapped" into a remote ODP instance B (on the same OS) by using the odp_shm_import(), on ODP instance B:

Note that the handles shmA and shmB are scoped by each ODP instance (you can not use them outside the ODP instance they belong to). Also note that both ODP instances have to call odp_shm_free() when done.

The last argument to odp_shm_reserve() is a set of ORed flags. The following flags are supported:

The scheduler’s dispatching job is to return the next event from the highest priority SCHED queue that the caller is eligible to receive events from. This latter consideration is determined by the queues scheduler group, which is set at queue create time, and by the caller’s scheduler group mask that indicates which scheduler group(s) it belongs to. Scheduler groups are represented by handles of type odp_scheduler_group_t and are created by the odp_scheduler_group_create() API. A number of scheduler groups are predefined by ODP. These include ODP_SCHED_GROUP_ALL (all threads), ODP_SCHED_GROUP_WORKER (all worker threads), and ODP_SCHED_GROUP_CONTROL (all control threads). The application is free to create additional scheduler groups for its own purpose and threads can join or leave scheduler groups using the odp_scheduler_group_join() and odp_scheduler_group_leave() APIs

The prio field of the odp_queue_param_t specifies the queue’s scheduling priority, which is how queues within eligible scheduler groups are selected for dispatch. Queues have a default scheduling priority of NORMAL but can be set to HIGHEST or LOWEST according to application needs.

In addition to its dispatching function, which provide automatic scalability to ODP applications in many core environments, the other main function of the scheduler is to provide event synchronization services that greatly simplify application programming in a parallel processing environment. A queue’s SYNC mode determines how the scheduler handles the synchronization processing of multiple events originating from the same queue.

Three types of queue scheduler synchronization area supported: Parallel, Atomic, and Ordered.

Atomic queues simplify event synchronization because only a single thread may process event(s) from a given atomic queue at a time. Events scheduled from atomic queues thus can be processed lock free because the locking is being done implicitly by the scheduler. Note that the caller may receive one or more events from the same atomic queue if odp_schedule_multi() is used. In this case these multiple events all share the same atomic scheduling context.

In this example, no matter how many events may be held in an atomic queue, only one calling thread can receive scheduled events from it at a time. Here two threads process events from two different atomic queues. Note that there is no synchronization between different atomic queues, only between events originating from the same atomic queue. The queue context associated with the atomic queue is held until the next call to the scheduler or until the application explicitly releases it via a call to odp_schedule_release_atomic().

Note that while atomic queues simplify programming, the serial nature of atomic queues may impair scaling.

Ordered queues provide the best of both worlds by providing the inherent scalability of parallel queues, with the easy synchronization of atomic queues.

When scheduling events from an ordered queue, the scheduler dispatches multiple events from the queue in parallel to different threads, however the scheduler also ensures that the relative sequence of these events on output queues is identical to their sequence from their originating ordered queue.

As with atomic queues, the ordering guarantees associated with ordered queues refer to events originating from the same queue, not for those originating on different queues. Thus in this figure three thread are processing events 5, 3, and 4, respectively from the first ordered queue. Regardless of how these threads complete processing, these events will appear in their original relative order on their output queue.

As with other ODP components, the ODP scheduler offers a range of capabilities and configuration options that are used by applications to control its behavior.

The sequence of API calls used by applications that make use of the scheduler is as follows:

The odp_schedule_capability() API returns an odp_schedule_capability_t struct that defines various limits and capabilities offered by this implementation of the ODP scheduler. Of note is the max_flow_id capability, which indicates whether this implementation is able to operate in flow aware mode.

A packet consists of a sequence of octets conforming to an architected format, such as Ethernet, that can be received and transmitted via the ODP pktio abstraction. Packets have a length, which is the number of bytes in the packet. Packet data in ODP is referenced via offsets since these reflect the logical contents and structure of a packet independent of how particular ODP implementations store that data.

These concepts are shown in the following diagram:

Packet data consists of zero or more headers followed by 0 or more bytes of payload, followed by zero or more trailers. Shown here are various APIs that permit applications to examine and navigate various parts of a packet and to manipulate its structure.

To support packet manipulation, predefined headroom and tailroom areas are logically associated with a packet. Packets can be adjusted by pulling and pushing these areas. Typical packet processing might consist of stripping headers from a packet via odp_packet_pull_head() calls as part of receive processing and then replacing them with new headers via odp_packet_push_head() calls as the packet is being prepared for transmit. Note that while headroom and tailroom represent reserved areas of memory, these areas are not addressable or directly usable by ODP applications until they are made part of the packet via associated push operations. Similarly, bytes removed via pull operations become part of a packet’s headroom or tailroom and are again no longer accessible to the application.

ODP platforms use various methods and techniques to store and process packets efficiently. These vary considerably from platform to platform, so to ensure portability across them ODP adopts certain conventions for referencing packets.

ODP APIs use a handle of type odp_packet_t to refer to packet objects. Associated with packets are various bits of system metadata that describe the packet. By referring to the metadata, ODP applications accelerate packet processing by minimizing the need to examine packet data. This is because the metadata is populated by parsing and classification functions that are coupled to ingress processing that occur prior to a packet being presented to the application via the ODP scheduler.

When an ODP application needs to examine the contents of a packet, it requests addressability to it via an API call that makes the packet (or a contiguously addressable segment of it) available for coherent access by the application. To ensure portability, ODP applications assume that the underlying implementation stores packets in segments of implementation-defined and managed size. These represent the contiguously addressable portions of a packet that the application may refer to via normal memory accesses. ODP provides APIs that allow applications to operate on packet segments in an efficient and portable manner as needed. By combining these with the metadata provided by packets, ODP applications can operate in a fully platform-independent manner while still achieving optimal performance across the range of platforms that support ODP.

The use of segments for packet addressing and their relationship to metadata is shown in this diagram:

The packet metadata is set during parsing and identifies the starting offsets of the various headers in the packet. The packet itself is physically stored as a sequence of segments that area managed by the ODP implementation. Segment 0 is the first segment of the packet and is where the packet’s headroom and headers typically reside. Depending on the length of the packet, additional segments may be part of the packet and contain the remaining packet payload and tailroom. The application need not concern itself with segments except that when the application requires addressability to a packet it understands that addressability is provided on a per-segment basis. So, for example, if the application makes a call like odp_packet_l4_ptr() to obtain addressability to the packet’s Layer 4 header, the returned length from that call is the number of bytes from the start of the Layer 4 header that are contiguously addressable to the application from the returned pointer address. This is because the following byte occupies a different segment and may be stored elsewhere. To obtain access to those bytes, the application simply requests addressability to that offset and it will be able to address the packet bytes that occupy the next segment, etc. Note that the returned length for any packet addressability call is always the lesser of the remaining packet length or size of its containing segment. So a mapping for segment 2 in the above figure, for example, would return a length that extends only to the end of the packet since the remaining bytes are part of the tailroom reserved for the packet and are not usable by the application until made available to it by an appropriate API call.

While the push/pull APIs permit applications to perform efficient manipulation of packets within the current segment structure, ODP also provides APIs that permit segments to be added or removed. The odp_packet_extend_head() and odp_packet_trunc_head() APIs permit segments to be added or removed from the beginning of a packet, while odp_packet_extend_tail() and odp_packet_trunc_tail() permit segments to be added or removed from the end of a packet. Extending a packet adds one or more segments to permit packets to grow up to implementation-defined limits. Truncating a packet removes one or more segments to shrink the size of a packet beyond its initial or final segment.

As noted, packet metadata is normally set by the parser as part of classification that occurs during packet receive processing. It is important to note that this metadata may be changed by the application to reflect changes in the packet contents and/or structure as part of its processing of the packet. While changing this metadata may effect some ODP APIs, changing metadata is designed to document application changes to the packet but does not in itself cause those changes to be made. For example, if an application changes the Layer 3 offset by using the odp_packet_l3_offset_set() API, the subsequent calls to odp_packet_l3_ptr() will return an address starting from that changed offset, changing an attribute like odp_packet_has_udp_set() will not, by itself, turn a non-UDP packet into a valid UDP packet. Applications are expected to exercise appropriate care when changing packet metadata to ensure that the resulting metadata changes reflect the actual changed packet structure that the application has made.

ODP Packet manipulation APIs can be divided into two categories: Those that do not change a packet’s segment structure, and those that potentially do change this structure. We’ve already seen one example of this. The push/pull APIs permit manipulation of packet headroom/tailroom that does not result in changes to packet segmentation, while the corresponding extend/trunc APIs provide the same functionality but with the potential that segments may be added to or removed from the packet as part of the operation.

The reason for having two different types of APIs that perform similar functions is that it is expected that on most implementations operations that do not change packet segment structure will be more efficient than those that do. To account for this, APIs that potentially involve a change in packet segmentation always take an output odp_packet_t parameter or return value. Applications are expected to use this new handle for the resulting packet instead of the old (input) handle as the implementation may have returned a new handle that now represents the transformed packet.

To enable applications that manipulate packets this way to operate most efficiently the return codes from these APIs follow a standard convention. As usual, return codes less than zero indicate error and result in no change to the input packet. A return code of zero indicates success, but also indicates that any cached addressability to the packet is still valid. Return codes greater than zero also indicate success but with a potential change to packet addressability. For example, if an application had previously obtained addressability to a packet’s Layer 3 header via the odp_packet_l3_ptr() API, a return code of zero would mean that the application may continue to use that pointer for access to the L3 header, while a return code greater than zero would mean that the application should reissue that call to re-obtain addressability as the packet segmentation may have changed and hence the old pointer may no longer be valid.

To support efficient multicast, retransmit, and related processing, ODP supports two additional types of packet manipulation: static and dynamic references. A reference is a lightweight mechanism for creating aliases to packets as well as to create packets that share data bytes with other packets to avoid unnecessary data copying.

Packet parsing is normally triggered automatically as part of packet RX processing. However, the application can trigger parsing explicitly via the API:

This is typically done following packet decapsulation or other preprocessing that would prevent RX parsing from "seeing" the relevant portion of the packet. The odp_packet_parse_param_t struct that is passed to control the depth of the desired parse, as well as whether checksum validation should be performed as part of the parse, and if so which checksums require this processing.

Packets containing Layer 3 (IPv4) and Layer 4 (TCP, UDP, SCTP) checksums can have these validated (on RX) and generated (on TX) automatically. This is normally controlled by the settings on the PktIOs that receive/transmit them, however they can also be controlled on an individual packet basis.

Packets have associated odp_packet_chksum_status_t metadata that indicates the state any checksums contained in that packet. These can be queried via the APIs odp_packet_l3_chksum_status() and odp_packet_l4_chksum_status(), respectively. Checksums can either be known good, known bad, or unknown, where unknown means that checksum validation processing has not occurred or the attempt to validate the checksum failed.

Similarly, the odp_packet_l3_chksum_insert() and odp_packet_l4_chksum_insert() APIs may be used to override default checksum processing for individual packets prior to transmission. If no explicit checksum processing is specified for a packet, then any checksum generation is controlled by the PktIO configuration of the interface used to transmit it.

PktIO objects begin life by being opened with odp_pktio_open() API:

The function has three arguments: a name, which is an implementation-defined string that identifies the logical interface to be opened, a pool that identifies the ODP pool that storage for received packets should be allocated from, and a param structure that specifies I/O options to be associated with this PktIO instance.

ODP defines "loop" as a reserved name to indicate that this PktIO represents a loopback interface. Loopback interfaces are useful as a means of recycling packets back for reclassification after decryption or decapsulation, as well as for diagnostic or testing purposes. For example, when receiving IPsec traffic, the classifier is able to recognize that the traffic is IPsec, however until the traffic is decrypted it is unable to say what that traffic contains. So following decryption, sending the decrypted packet back to a loopback interface allows the classifier to take a "second look" at the packet and properly classify the decrypted payload. Similar considerations apply to tunneled packets that must first be decapsulated to reveal the true payload.

The pool specifies the default pool to use for packet allocation if not overridden by the classifier due to a specific or default Class-of-Service (CoS) match on the packet. The param struct, in turn, specifies the input and output modes of the PktIO.

Associated with each PktIO is a set of capabilities that provide information such as the maximum number of input/output queues it supports, its configuration options, and the operations is supports. These are aggregated into odp_pktio_capability_t the struct, which is returned by odp_pktio_capability() API.

PktIO objects support four different Input and Output modes, that may be specified independently at open time.

The DISABLED modes indicate that either input or output is prohibited on this PktIO. Attempts to receive packets on a PktIO whose in_mode is DISABLED return no packets while packets sent to a PktIO whose out_mode is DISABLED are discarded.

To facilitate implementation of the ODP timer APIs, an additional timer API is provided. During initialization, applications are expected to create the timer pools they need and then call odp_timer_pool_start_multi(). Following start, applications may allocate, set, cancel, and free timers from their associated timer pools. During termination processing, after all timers allocated from a timer pool have been freed, the pool itself should be released via a call to odp_timer_pool_destroy().

The purpose of ODP timers is to schedule their associated timeout events, which are how applications actually react to the passage of time. To help with this, several additional APIs and conventions are provided.

Timer allocation is performed via the odp_timer_alloc() API:

Note that in addition to the timer pool and queue, a user pointer is provided. This is to allow context associated with the timeout to be communicated. Upon receiving a timeout event, the application can use the odp_timeout_user_ptr() API to retrieve the user pointer associated with the timer that triggered this event.

An worker thread receiving events that may include timeouts might be structured as follows:

To apply a cryptographic operation to a packet a session must be created. All packets processed by a session share the parameters that define the session.

ODP supports synchronous and asynchronous crypto sessions. For asynchronous sessions, the output of crypto operation is posted in a queue defined as the completion queue in its session parameters.

ODP crypto APIs support chained operation sessions in which hashing and ciphering both can be achieved using a single session and operation call. The order of cipher and hashing can be controlled by the auth_cipher_text session parameter.

Other Session parameters include algorithms, keys, initialization vector lengths, encode or decode, output queue for async mode and output packet pool for allocation of an output packet if required.

The parameters that describe the characteristics of a crypto session are encoded in the odp_crypto_session_param_t struct that is passed to the odp_crypto_session_create() API. A successful call returns an odp_crypto_session_t object that in turn is passed as an input parameter to crypto operation calls.

When an application is finished with a crypto session the odp_crypto_session_destroy() API is used to release the resources associated with an odp_crypto_session_t.

After session creation, a cryptographic operation can be applied to a packet synchronously or asynchronously. odp_crypto_op() is the synchronous API while odp_crypto_op_enq() is the asynchronous API. To check which of these are supported by the ODP implementation, examine the sync_mode and async_mode fields in the odp_crypto_capability_t struct returned by the odp_crypto_capability() API.

Both forms take an input array of packets, an optional output array of packets to receive the results, and an array of odp_crypto_packet_op_param_t structs that describe the operation to be performed on each input packet. The output array may be the same packets to request in-place operation, or may be specified as ODP_PACKET_INVALID to request that ODP allocate output packets from the pool associated with the odp_crypto_session_t being used.

The op_mode field of odp_crypto_session_t indicates whether asynchronous or synchronous operations are used with the session. If op_mode is set to ODP_CRYPTO_SYNC then the synchronous API must be used and if op_mode is set to ODP_CRYPTO_ASYNC then the asynchronous API must be used. It is an error to use a form of the API that does not match the mode of the crypto session.

The output of a crypto operation is an odp_packet_t (one for each input packet) that is returned either synchronously or asynchronously. Asynchronous return is in the form of ODP_EVENT_PACKET events that have event subtype ODP_EVENT_PACKET_CRYPTO. The packet associated with such events is obtained via the odp_crypto_packet_from_event() API. The odp_crypto_result() API, in turn, retrieves the odp_crypto_packet_result_t from this odp_packet_t that contains:

ODP provides two APIs to generate various kinds of random data bytes. Random data is characterized by kind, which specifies the "quality" of the randomness required. ODP support three kinds of random data:

These form a hierarchy with BASIC being the lowest kind of random and TRUE being the highest. The main API for accessing random data is:

The expectation is that lesser-quality random is easier and faster to generate while higher-quality random may take more time. Implementations are always free to substitute a higher kind of random than the one requested if they are able to do so more efficiently, however calls must return a failure indicator (rc < 0) if a higher kind of data is requested than the implementation can provide. This is most likely the case for ODP_RANDOM_TRUE since not all platforms have access to a true hardware random number generator.

The odp_random_max_kind() API returns the highest kind of random data available on this implementation.

For testing purposes it is often desirable to generate repeatable sequences of "random" data. To address this need ODP provides the additional API:

This operates the same as odp_random_data() except that it always returns data of kind ODP_RANDOM_BASIC and an additional thread-local seed parameter is provide that specifies a seed value to use in generating the data. This value is updated on each call, so repeated calls with the same variable will generate a sequence of random data starting from the initial specified seed. If another sequence of calls is made starting with the same initial seed value, then odp_random_test_data() will return the same sequence of data bytes.

ODP provides the API odp_crypto_capability() to inquire the implementation’s crypto capabilities. This interface returns a the maximum number of crypto sessions supported as well as bitmasks for supported algorithms and hardware backed algorithms.

As with other features, ODP provides APIs that permit applications to query platform-specific IPsec capabilities. The odp_ipsec_capability() API queries the general IPsec features available while the odp_ipsec_cipher_capability() and odp_ipsec_auth_capability() APIs provide detail on the range of cipher and authentication algorithms supported by IPsec on this platform.

General IPsec capabilities that are reported include:

In addition, capabilities also inform the application of the maximum number of destination queues and classification CoS targets supported. These will be discussed further later.

Prior to making use of IPsec services, the odp_ipsec_config() API is used to configure IPsec processing options. This API takes a pointer to an odp_ipsec_config_t struct as its argument.

The odp_ipsec_config_t struct specifies the inbound and outbound processing modes (SYNC, ASYNC, or INLINE) that the application plans to use, the maximum number of Security Associations it will use, and sets inbound and outbound processing options.

IPsec introduces one new event type and one new event subtype. These are:

IPsec-related events are thus part of normal and exception processing when working with IPsec.

The fundamental "building block" for IPsec processing is the Security Association (SA). Similar to a crypto session, the SA encapsulates the keying material and context needed to perform IPsec protocol processing for inbound or outbound packets on a given flow, as well as additional processing options that control how IPsec is to be used for packets processed under this SA. Security Associations are unidirectional (RX or TX) so a flow that requires both inbound (decrypt) and outbound (encrypt) IPsec functions will have two SAs associated with it. SAs in ODP are represented by the abstract type odp_ipsec_sa_t.

After ODP initialization, IPsec support is dormant until it is configured by a call to odp_ipsec_config() as described earlier. Once configured, SAs may be created by calling odp_ipsec_sa_create().

ODP compression services are session based and operate on input packets and deliver output packets. A compression session (odp_comp_session_t) provides the context for controlling the operations performed on packets. All of the packets processed by a session share the parameters that define the session.

ODP supports synchronous and asynchronous compression sessions. For asynchronous sessions, the output of a compression operation is posted to a queue defined as the completion queue in its session parameters.

Other session parameters include: the type of operation (compression or decompression), the operating mode (synchronous or asynchronous), the compression and hashing algorithms to be used, as well as any parameters needed by those algorithms to configure them. For asynchronous compression sessions, the application also specifies whether queue order must be maintained. Additional throughput may be achieved in some implementations if strict ordering is not required.

The parameters that describe the characteristics of a compression session are encoded in the odp_comp_session_param_t struct that is passed to the odp_comp_session_create() API. A successful call returns an odp_comp_session_t handle that is then used as an input parameter to compression operation calls.

When an application is finished with a compression session, the odp_comp_session_destroy() API is used to release the resources associated with an odp_comp_session_t.

After session creation, a compression operation can be applied to a packet in one of two ways: synchronous and asynchronous, depending on how the session was created.

The packet scheduling/dropping techniques that can be applied to input traffic include any mixture of the following:

Note that Bandwidth Shaping is the only feature that can cause packets to be "delayed", and Weighted Random Early Discard is the only feature (other than input queues becoming full) that can cause packets to be dropped.

This API supports the ability to do Hierarchical Scheduling whereby the final scheduling decision is controlled by equal priority schedulers, strict priority multiplexers, bandwidth shapers - at multiple levels - all forming a tree rooted at a single egress object. In other words, all tm_queues and tm_nodes have the property that their logical "output" feeds into one fan-in of a subsequent tm_node or egress object - forming a proper tree.

Multi-level/hierarchical scheduling adds both great control and significant complexity. Logically, despite the implication of the tm_node tree diagrams, there are no queues between the levels of hierarchy. Instead all packets are held in their input queue, until such time that the totality of all of the tm_nodes in the single path from input queue to output object agrees that this packet should be the next to be chosen to leave the TM system through the output object "portal". Hence what flows from level to level is the "local choice" of what packet/tm_queue should next be serviced.

It is important to recognize the difference between the "abstract" mathematical model of the prescribed behavior and real implementations. The model describes the Ideal, but theoretically desired behavior, but such an Ideal is generally not practical to implement. Instead, one understands that virtually all Real TM systems attempt to approximate the Ideal behavior as given by the TM configuration as best as they can - while still attaining high packet processing performance. The idea is that instead of trying too hard to be "perfect" at the granularity of say microseconds, it may be better to instead try to match the long term Ideal behavior over a much more reasonable period of time like a millisecond. It is generally better to have a stable implementation that when averaged over a period of several milliseconds matches the Ideal behavior very closely than to have an implementation that is perhaps more accurate over a period of microseconds, but whose millisecond averaged behavior drifts away from the Ideal case.

Following is the functionality that is required of the classification API, and its underlying implementation. The details and order of the following paragraph is informative, and is only intended to help convey the functional scope of a classifier and provide context for the API. In reality, implementations may execute many of these steps concurrently, or in different order while maintaining the evident dependencies:

The above is an abstract description of the classifier functionality, and may be applied to a variety of applications in many different ways. The ultimate meaning of how this functionality applies to an application also depends on other ODP modules, so the above may not complete a full depiction. For instance, the exact meaning of priority, which is a per-queue attribute is influenced by the ODP scheduler semantics, and the system behavior under stress depends on the ODP buffer pool module behavior.

For the sole purpose of illustrating the above abstract functionality, here is an example of a Layer-2 (IEEE 802.1D) bridge application: Such a forwarding application that also adheres to IEEE 802.1p/q priority, which has 8 traffic priority levels, might create 8 odp_buffer_pool instances, one for each PCP priority level, and 8 odp_queue instances one per priority level. Incoming packets will be inspected for a VLAN header; the PCP field will be extracted, and used to select both the pool and the queue. Because each queue will be assigned a priority value, the packets with highest PCP values will be scheduled before any packet with a lower PCP value. Also, in a case of congestion, buffer pools for lower priority packets will be depleted earlier than the pools containing packets of the high priority, and hence the lower priority packets will be dropped (assuming that is the only flow control method that is supported in the platform) while higher priority packets will continue to be received into buffers and processed.

To program the classifier, a class-of-service instance must be created, which will contain the packet filtering resources that it may require. All subsequent calls refer to one or more of these resources.

Each class of service instance must be associated with a single queue or queue group, which will be the destination of all packets matching that particular filter. The queue assignment is implemented as a separate function call such that the queue may be modified at any time, without tearing down the filters that define the class of service. In other words, it is possible to change the destination queue for a class of service defined by its filters quickly and dynamically.

Optionally, on platforms that support multiple packet buffer pools, each class of service may be assigned a different pool such that when buffers are exhausted for one class of service, other classes are not negatively impacted and continue to be processed.

There is a odp_cos_t assigned to each port with the odp_pktio_default_cos_set() function, which will function as the default class-of-service for all packets received from an ingress port, that do not match any of the filters defined subsequently. At minimum this default class-of-service must have a queue and a buffer pool assigned to it on platforms that support multiple packet buffer pools. Multiple odp_pktio instances (i.e., multiple ports) may each have their own default odp_cos, or may share a odp_cos with other ports, based on application requirements.

Error class of service is assigned to an ingress port using the function odp_pktio_error_cos_set(). All the packets received with error from this specific ingress port are assigned to this error class-of-service. At minimum this error class-of-service must have a queue and a buffer pool assigned to it. Multiple pktio instances (i.e., multiple ports) may each have their own error class of service, or may share an error CoS with other ports, based on application requirements.

Each class of service has a drop_policy configured during creation. The valid value are ODP_COS_DROP_POOL and ODP_COS_DROP_NEVER. If the drop_policy is set to ODP_COS_DROP_POOL then the packets assigned to the CoS follows the drop policy of the associated pool i.e., depending on the Random Early Discard or any other configuration of the pool the packet might get dropped. If the drop_policy is set to ODP_COS_DROP_NEVER then the Random Early Discard of the pool is ignored.

During creation of the class of service if the pool or queue is set as INVALID using ODP_POOL_INVALID or ODP_QUEUE_INVALID field then any packet assigned to the specific CoS are dropped.

For each odp_pktio port, the API allows the assignment of a class-of-service to a packet. Application can program a number of pattern matching rules that assign a class-of-service for packets with header fields matching specified values. Using these matching rules the application should be able for example to identify all packets containing VoIP traffic based on the protocol being UDP, and a specific destination or source port numbers, and appropriately assign these packets a class-of-service that maps to a higher priority queue, assuring voice packets a lower and bound latency.

Here are the specific information elements that are stored within the packet meta data structure:

The ODP packet API module provides accessors for retrieving the above meta data fields from the container buffer in an implementation-independent manner.

CoS configuration can be best illustrated by drawing a tree, where each CoS is the vertex, and each link between any two vertices is a PMR. The root node for the tree is the default CoS which is attached with the pktio interface. All of the CoS vertices can be final for some packets, if these packets do not match any of the link PMRs.

odp_pktio_default_cos_set(odp_pktio_t pktio, odp_cos_t default_cos);

pmr1 = odp_cls_pmr_create(pmr_match1, default_cos, cos1);
pmr2 = odp_cls_pmr_create(pmr_match2, default_cos, cos2);
pmr3 = odp_cls_pmr_create(pmr_match3, default_cos, cos3);

pmr11 = odp_cls_pmr_create(pmr_match11, cos1, cos11);
pmr12 = odp_cls_pmr_create(pmr_match12, cos1, cos12);

pmr21 = odp_cls_pmr_create(pmr_match11, cos2, cos21);
pmr31 = odp_cls_pmr_create(pmr_match11, cos3, cos31);

The above configuration DOES imply order - a packet that matches pmr_match1 will then be applied to pmr_match11 and pmr_match12, and as a result could terminate with either cost1, cos11, cos12. In this case the packet was subjected to two match attempts in total.

The remaining two lines illustrate how a packet that matches pmr_match11 could end up with either cos11, cos21 or cos31, depending on whether it matches pmr_march1, pmr_march2 or pmr_match3.

Let’s look at DNS packets, these are identified by using UDP port 53, but each UDP packet may run atop of IPv4 or IPv6, and in turn an IP packet might be received as either multicast or unicast,

PMR-L2 = match all multicast/broadcast packets based on DMAC address
PMR_L3_IP4 = match all IPv4 packets
PMR_L3_IP6 = match all IPv6 packets
PMR_L4_UDP = match all UDP packets
PMR_L4_53 = match all packets with dest port = 53

In this case, a packet may change CoS between 0 and 5 times, meaning that up to 5 PMRs may be applied in series, and the order

Another interesting point is that an implementation will probably impose on a limit of how many PMRs can be applied to a packet in series, so in the above example, if an implementation limit on the number of consecutive classification steps is 4, then all the DNS packets may only reach cos_udp?_?c set of vertices.

If compiled using --enable-pcapng-support ODP will offer packet capturing functionality in PcapNg format. If the /var/run/odp directory exists prior to launching the application ODP will create a fifo for each NIC queue. Queue naming will be of the following format: <odp global pid>-<NIC name>-flow-<queue number>. Linux dd application can be used for capturing a sample of the live stream from the fifo. Killing ether the application or dd will stop the capturing process.