Subject: Re: WG Review: Remote Direct Data Placement (rddp)

From: Timur Shemsedinov <[email-masked]>
Date: Mon, 17 Jun 2002 19:29:05 +0300

SC> Remote Direct Data Placement (rddp)
SC> -----------------------------------
SC> 1) A (transport independent) protocol core to support direct data
SC>    placement from the NIC into specified memory, usually application    
SC>    buffers.
SC> 2) A (transport independent) protocol core layered on top of the direct
SC>    data placement protocol that specifies control of RDMA.
SC> 3) A mapping of the direct data placement protocol and the RDMA control
SC>    protocol to SCTP.
I have some questions.
Whether it is supposed what server module will be inside the application
(or in separate service)? This question has arisen because processes in
the majority of operational systems have different address spaces.
So, application should use mechanisms of memory sharing and
cross-application interaction or RDMA module should be included into
each application.
And how to understand "transport independent" in this case?

thanks,
Best regards,
Timur                            mailto:[email-masked]

---

From: Caitlin Bestler <[email-masked]>
Date: Tue, 18 Jun 2002 09:59:45 -0500

On 6/17/02, Timur Shemsedinov wrote:

>SC>-----------------------------------
>SC> 1) A (transportindependent) protocol core to support
>SC> direct data placement from the NIC into specified 
>SC> memory, usually application buffers.
>SC> 2) A (transportindependent) protocol core layered on
>SC> top of the direct data placement protocol that 
>SC> specifies control of RDMA.
>SC> 3) A mapping of the direct data placement protocol
>SC> and the RDMA control protocol to SCTP.

>I have some questions.
>Whether it is supposed what server module will be inside
>the application (or in separate service)? This question
>has arisen because processes in the majority of
>operational systems have different address spaces.

Direct Data Placement protocols typically support anonymous
target buffers (essentially the same semantics as SCTP) as
well as tagged buffers.

Each placement specifies a tag and an address to be
interpreted in the context of that tag. How tags are
obtained is a local interface issue. How they are
distributed is a ULP issue.

However, the typical local interface is that the application
registers a portion of its own address space, and thereby
obtains tags that it can distribute to its peers.

You can find examples of such interfaces with InfiniBand
(http://www.infinibandta.org), the Virtual Interface
Archtiecture (http://www.vidf.org) and the Direct Access
Transport API (http://www.datcollaborative.org).

InfiniBand is an excellent example of a specific solution.
The DAT specifications present a technology neutral
definition of Direct Data Placement capable transports.

>So, application should use mechanisms of memory sharing
>and cross-application interaction or RDMA module should be
>included into each application.

Typically, the local interface is indeed a user mode
library. However, it is local interface, so it can be
implemented many ways and not affect interoperability.

The reason it will tend to be a user mode library is to
avoid unnecessary user/kernel mode transitions. Typically
kernel interactions will be done when memory is registered
and endpoints are created. The goal is to leave the kernel
out of the loop on a per data transfer basis to the greatest
extent possible. Current local bus architectures,
particularly how interrupts work, limit the degree to which
that can be done. However things can still be minimized, and
most features of the local interfaces are intended to keep
user/kernel interactions as minimal as possible without
creating security holes.

>And how to understand "transport independent" in this case?

The meaning of a sucessfully delivered DDP or RDMA message
is not dependent upon the underlying IP transport protocol
that was used to deliver it.

The goal is to define protocols that run over ubiquitous
transports, not to define a new transport. Applications
should not be restricted to SCTP, but free to choose other
IP transports that are suitable to their needs. One example
would be use of DCP for applications (such as streaming
media) that prefer unreliable delivery because real-time
delivery requirements severely limit the time window for
retries.

This is similar to many protocols above layer 4 that run
over multiple transports, including RPC and SNMP.

Caitlin Bestler
Asomi Network Technologies