• BulkTransfers

Bulk Transfers

Bulk transfer (bulk_transfer.{c,h}) is a mechanism to facilitate (relatively large) data exchanges between processes (user programs or barrelfish services). It is intended to be used on top of normal barrelfish messages to avoid the memory copying overhead when the architecture permits it (e.g., in shared memory architectures).

Mechanism overview

We assume two processes: M and S (the interface is not symmetric).

Initially, M allocates a single memory frame (continuous physical memory). This frame is described by a struct capref, and, in the common case, it is mapped to the virtual address space of both M and S.

M maps the frame to its virtual address space, and splits it to buffers of constant size that form a buffer pool. Each buffer is described by its id. M can send the frame (struct capref) to S, and S can map it into its own address space.

Buffers can now by exchanged using buffer ids. The buffers are exclusively managed by M. M needs to allocate buffers, as well as return them to the pool when the buffer is consumed.

Interface overview

S uses bulk_slave_* functions, while M uses the rest of bulk_* functions defined in the header.

data structures

struct bulk_transfer;        // a pool of buffers corresponding to a single frame
struct bulk_buf;             // a buffer
struct bulk_transfer_slave;  // a helper for S to access the buffers using the ids

initialization

Assuming we want to allocate 128 buffers of 1024 bytes each:

#define BLOCKS_NR  128
#define BLOCK_SIZE 1024
#define BULK_SIZE  (BLOCKS_NR*BLOCK_SIZE)

// This is M
struct capref        cap;
struct bulk_transfer bt;
// initialize buffer pool
bulk_create(BULK_SIZE, BLOCK_SIZE, &bt, &cap);
    |------------- send cap to S ------->
                                   // This is S
                                   void *pool;
                                   struct bulk_transfer_slave bts;
                                   // map cap to this virtual address space
                                   vspace_map_one_frame(&pool, BULK_SIZE, cap, ...);
                                   // initialize bulk_slave
                                   bulk_slave_init(pool, BULK_SIZE, &bts);

buffer exchange

 // this is M
 // allocate a buffer from the pool
 struct bulk_buf *bb = bulk_alloc(&bt);
 // handle failure
 if (bb == NULL) ...;
 // get the buffer's id
 uintptr_t bufid = bulk_buf_get_id(bb);
 // (possibly) do something with buffer
 char *buf = bulk_buf_get_mem(bb);
 ...
    |------- send bufid to S --------->
                                 // this is S
                                 // get memory from buffer id
                                 void *buf = bulk_slave_buf_get_mem(bts, bufid, NULL);
                                 // do something with buffer
                                 ...
    <------- send bufid to M --------|
 // (possibly) do something with buffer
 ...
 // return buffer to the pool
 bulk_free(&bt, bufid);

Depending on the data direction, M will need to call bulk_prepare_{send,recv}() and S bulk_slave_prepare_{send,recv}().

For example, if S needs to send data to M:

• S needs to call bulk_slave_prepare_send() before sending the buffer id

• M needs to call bulk_prepare_recv() before reading the data

Note that the above code is untested.

For more details see:

• usr/ramfsd/service.c and lib/vfs/vfs_ramfs.c
• usr/bench/bulk_bench/bulkbench.c

and other uses of bulk transfers in the barrelfish source.