Users of VMMC are encouraged to join the email list used for VMMC announcements. Join the list to keep up with the latest VMMC developments. To join the email list, send a message to majordomo@cs.princeton.edu with "subscribe vmmc-announce youremail@yourcompany.com" in the body of the message. Email traffic is minimal.

We welcome input about functionality, bugs, and possible extensions to the API. Please send your comments or questions to vmmc-support@cs.princeton.edu. For more information on The SHRIMP Project (including technical papers) please visit our web site at http://www.cs.princeton.edu/shrimp.

1. VMMC Cluster Service and Utilities

Introduction

1. Administrator’s Guide

Starting and stopping VMMC

Administering the VMMC cluster involves enabling and disabling VMMC system, checking log files, and session and process management. The active components of the VMMC runtime system on each cluster node include the cluster server VMMCSVR, the device driver VMMCDRV, and the firmware running on the Myrinet network interface.

VMMCSVR: The VMMCSVR is configured to start up at system boot time. The VMMCSVR runs under the local "vmmc" account. Its type registry setting (\\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VMMCSVR\type) is 0x110, which allows VMMCSVR to interact with a logged-on user. An administrator can manually stop and re-start VMMCSVR with Windows NT’s NET command as follows:

NET start VMMCSVR

NET stop VMMCSVR

VMMCDRV: The VMMCDRV are started and stopped manually, using the NET command.

Myrinet: An administrator can enable the Myrinet network interface by loading the MCP (Myrinet Control Program). This can be only after the VMMCDRV is properly started. The VMMC system software on each cluster node includes a Windows script (batch file) MCPSTART.BAT for loading the MCP and a script MCPSTOP.BAT for disabling the Myrinet network interface. They are located in %SystemRoot%\vmmc\bin.

System log files

On each VMMC cluster node, the VMMC system produces two log files, server.log and driver.log. Both reside in %SystemRoot%\vmmc\logs directory. The server.log file contains information and error logs from VMMCSVR, mostly regarding session and process management. The driver.log file contains logs for the VMMCDRV device driver and the Myrinet network interface.

Since the %SystemRoot%\vmmc directory is exported as VMMC share during the setup, an administrator can view the log files remotely accessing the VMMC share on a different Windows NT workstation. For example, the following command enable accesses to the log file on vmmc-node-1:

NET use \\vmmc-node-1\vmmc vmmc /user:vmmc

TYPE \\vmmc-node-1\vmmc\logs\server.log

VMMC system account and network share

During VMMC system installation, a local account vmmc is created on each cluster node. We have chosen "vmmc" as the password. An administrator can modify it using the User Manager program. However, the administrator must make sure that the VMMC service can still run under the local vmmc account. The Service Control Manager (invoked from the Control Panel) can be used for this purpose.

The home directory for the vmmc account is %SystemRoot%\vmmc. If UWIN is installed on the cluster node, the .rhosts file must be located in this directory in order for the rsh service to work properly.

User’s Guide

In this section we give an overview of the components involved in the Windows NT implementation of VMMC. We also described how user’s can run VMMC applications using our GUI interface.

1. Installing the VMMC SDK

 

Programming Model

This section describes the VMMC programming model.

1. VMMC Overview

 

Virtual memory-mapped communication is a mechanism for protected data transfer from the sender's virtual address space to the receiver's virtual address space. Communication is protected because it may take place only after the receiver gives the sender permission to transfer data to a given area of the receiver's address space. The receiving process expresses this permission by exporting areas of its address space as receive buffers where it is willing to accept incoming data. A sending process must import remote buffers which it will use as destinations for transferred data. An exporter can restrict possible importers of a buffer; VMMC enforces the restrictions when an import is attempted. After a successful import, the sender can transfer data from its virtual memory into the imported receive buffer. VMMC makes sure that transferred data do not overwrite receiver's addresses outside the destination receive buffer.

VMMC supports two data transfer modes: deliberate update and automatic update (NOTE: automatic update is only available on SHRIMP network interfaces and will not be discussed here.) Deliberate update is an explicitly initiated transfer of a contiguous block of data from any (readable) virtual address in the caller's process to a previously imported receive buffer. VMMC guarantees in-order, reliable delivery of deliberate update messages.

When a message arrives at its destination, it is transferred directly into the memory of the receiving process, without interrupting the receiver's CPU. Thus there is no explicit receive operation in VMMC.

VMMC supports user-level buffer management because transfer is performed between user-level memory locations. Buffer management is divorced from the data movement mechanism and becomes the responsibility of the communicating parties. Zero copy protocols are possible because a direct application-to-application transfer of data can occur.

The CPU overhead to send data is very small, only a few user-level instructions are needed for deliberate update. The model does not impose any CPU overhead to receive data, as there is no explicit receive operation. CPU involvement in receiving data can be as little as checking a flag; moreover program logic can be used to reason about what data has already arrived (since messages are delivered in order).

Our model as described in this section can be applied to communication between processes executing on one uniprocessor machine, on separate processors of a shared memory multiprocessor, or between processes executing on different nodes in a local area network. In the former two cases, VMMC is a special restricted case of shared memory communication with deliberate update added for bulk transfer. The LAN case is discussed in the next section.

2. Machine Organization

SHRIMP machine is a set of UNIX nodes. Each node is identified with a unique Internet address. VMMC uses these addresses for node identification. The vmmc_Hosts() call can be used to obtain node ids of a SHRIMP machine.

Processes are identified with squids. A squid (Shrimp quad-word ID) is a kind of process ID which is unique within a given node and is guaranteed not to be recycled in the lifetime of a given VMMC machine (UNIX process ids are recycled, so they cannot be used for identification across nodes). Note that full process identification is a (node, squid) pair. vmmc_GetSquid() returns squid of a calling process. vmmc_MyNode() returns the id for the node of the calling process.

User address space is divided in VMMC pages, which size is given by vmmc_PageSize() (currently 4096 bytes). Each such page is a multiple of virtual memory page. Each VMMC page contains integer number of VMMC words. The size of a VMMC word is given by vmmc_WordSize()(currently 4 bytes).

3. Receive Buffers

Communication in the VMMC model is based on receive buffers. A receive buffer is a contiguous region of process memory used for receiving data from other processes using VMMC. Each receive buffer is identified with user-selected buffer id (unsigned integer). A receiver process makes a receive buffer available for senders with vmmc_ExportRecvBuf() call which takes as arguments buffer id, buffer starting address and buffer length. Buffer id must be unique among all ids of receive buffers exported by a given process. Receive buffers cannot overlap.


Figure 2. Destination Address Space (dspace)

4. Destination Proxy Space

Destination proxy space (DestSpace} in short) is a logically separate special address space in each sender process; it is used for addressing imported receive buffers. This new address space is a subspace of the sender's virtual address space, but it is not backed by local memory and its addresses do not refer to code or local data. Instead, they are used only to specify destinations for data transfer. After sender imports a remote receive buffer, a representation of the imported buffer is mapped into the sender's destination proxy space. This representation is simply a range of addresses in destination proxy space. An address belonging to a given range can be translated by the VMMC into a destination machine, process, and virtual address. Figure 1 shows the mapping of a receive buffer allocated on node Receiver into destination proxy space of node Sender.

5. Establishing import-export mappings

A sender process has to import a given receive buffer before it can send any data. The import operation is implemented with vmmc_ImportRecvBuf() call. Import takes as arguments receive buffer id and full identification of a process (nodeId, pid) which exported this receive buffer. Import succeeds only after export call has been completed for this receive buffer. There is also an asynchronous version of the import call, vmmc_ImportRecvBufReq(), which issues only an import request and returns immediately.

For a given process, ids of exported and imported buffers belong to two disjoint name spaces. As a result, one buffer id can be used in both export and import calls. However, for a given process, ids of exported buffers have to be unique. The full identification of an imported buffer is not its buffer id, but a triple (buffid, nodeId, pid). As a result it is possible to import two buffers with the same buffer id, if two different processes exported them.

With one export call, a process can export exactly one receive buffer to potentially unlimited number of processes. With one import call, a process can import only one receive buffer.

Imported receive buffers are mapped into destination proxy space (Figure 1). The import call returns an address in local DestSpace, which corresponds to the imported buffer. If we import a buffer of size nwords (each word is vmmc_WordSize() bytes, currently four) and its address assigned by the import call is raddr, then the range raddr,...,raddr+nwords*4-1 in DestSpace corresponds to imported receive buffer. This address range represents a proxy buffer in sender's address space for this receive buffer. Given address in DestSpace can belong to no more than one proxy. The terms proxy and imported receive buffer denote the same thing: a local representation of a receive buffer which has been imported from remote node and mapped into local DestSpace.

We say that a successful import establishes an import-export link between a receive buffer and its local proxy. Establishing an import-export mapping requires a trusted third party (operating system kernel or daemon process) to verify protection. Therefore, creating mappings can be relatively expensive. However, it should occur infrequently, as import is required only once for a given receive buffer; afterward messages can be sent directly from user-level.

6. Protection Guarantees

Receive buffers need not begin or end on a page boundary. Although the SHRIMP library respects the true boundaries of a receive buffer, the SHRIMP hardware enforces protection on a page granularity.

Thus, if a process exports a receive buffer and a malicious process imports it, the importing process will be able, by bypassing the SHRIMP library, to send data to locations that are on the same page as the buffer but are not actually part of the buffer.

If you must communicate with a process you don't trust, you can assure absolute safety by aligning your buffer at the beginning of a page, and making its size a multiple of the page size. The size of VMMC page is returned by vmmc_PageSize().

7. Deliberate update

Deliberate update is an explicit request to transfer data from anywhere in virtual memory of a sender to a previously imported receive buffer. Deliberate update requires explicit operation, vmmc_SendData(), to initiate communication on the sender's side. vmmc_SendData() can transfer data from any memory in sender's address space (excluding DestSpace) to previously imported receive buffer on remote node. vmmc_SendData() takes as arguments address in DestSpace, which identifies receive buffer to be used, local "standard" (i.e. not in DestSpace) address which identifies data to be send, and nwords which gives the size of the message. If no error occurs, vmmc_SendData() returns after all data has been sent out to the network.

VMMC also provides a non-blocking variant of vmmc_SendData() called vmmc_SendDataAsync(). The non-blocking send is designed to minimize the CPU overhead required to start a data transfer.

8. Transfer Redirection

The basic virtual memory-mapped communication model provides protected, user-level communication to move data directly from a send buffer to a receive buffer without any copying. Since it requires the sender to know the receiver buffer address, it does not provide good support for connection-oriented high-level communication APIs.

The VMMC includes a mechanism called transfer redirection. The basic idea is to use a default, redirectable receive buffer when a sender does not know the final receive buffer addresses.

Redirection is a local operation affecting only the receiving process. The sender does not have to be aware of a redirection and always sends data to the default buffer. When the data arrives at the receive side, the redirection mechanism checks to see whether a redirection address has been posted. If no redirection address has been posted, the data will be moved to the default buffer. Later, when the receiver posts the receive buffer address, the data will be copied from the default buffer to the receive buffer, as shown in Figure 3(a).


Figure 3. Transfer redirection uses a default buffer to hold data in case receiver posts no buffer address or posts it too late, and moves data directly from the network to the user buffer if the receiver posts its buffer address before the data arrives.

If the receiver posts its buffer address before the message arrives, the message will be put into the user buffer directly from the network without any copying, as shown in Figure 3(b). If the receiver posts its buffer address during message arrival, the message will be partially placed in the default buffer and partially placed in the posted receive buffer. The redirection mechanism tells the receiver exactly how much and what part of a message is redirected. When partial redirection occurs, this information allows the receiver to copy the part of the message that is placed in the default buffer.

For redirectable receive buffers, VMMC provides additional functionality to help user processes figure out data arrival. When last chunk of a message arrives, a receive-buffer specific memory location in user space is updated with the receive buffer offset corresponding to the last word transferred.

There are two calls for a receiver to perform redirection on the redirectable buffer. The first is:
vmmc_PostRedir (defBuf, offset, size, userBuf)
where defBuf is the default buffer for redirection, offset is the offset within this buffer indicating the starting address of the redirection, size is the number of bytes to be redirected, and userBuf is the destination user buffer address for redirection. This call posts a transfer redirection at user level and currently at most one redirection can be posted for a receive buffer.

The second call is:
vmmc_EndRedir (defBuf, &redirSize, &redirOffset)
where defBuf is the default buffer for redirection, redirSize returns the number of bytes redirected and redirOffset returns the offset of the first byte directed in the default buffer. This call cancels a transfer redirection posted to the default buffer. If no data has been redirected, the returned number of redirected bytes is equal to zero. If a redirection did happen, the redirected data is in the contiguous user address space area starting at address userBuf + redirOffset - offset.

The transfer redirection mechanism naturally extends the basic communication model and it is very simple. It is controlled entirely by the receiver side; the sender does not need to know what the actual destination address is nor whether a transfer redirection takes place.

Our implementation allows redirection to happen at most once for each vmmc_PostRedir(). As soon as a message is redirected from a redirectable buffer to a receive buffer, it cancels the redirection automatically. We choose this design to simplify vmmc_EndRedir(). Otherwise, it would have to return multiple redirection regions.

The transfer redirection is designed for a single pair of sender and receiver. This design decision is based on the fact that most connection-based high-level APIs deal with a single pair of sender and receiver per connection.. To avoid interleaving messages from different senders, a redirectable receive buffer should be imported by just one sender which is easy to enforce when the buffer is exported. If one really needs to support multiple senders and the communicating processes trust each other, a high-level protocol can be used to ensure that only one sender can send a message to a redirected buffer at a time.

How much data is actually redirected depends on the relative timing of vmmc_PostRedir() call and data arrival time, assuming vmmc_EndRedir() does not interfere with data redirection. If a redirection is posted before message arrival, the data will be redirected. If a redirection is posted after data starts arriving but before its transfer finishes, the remainder of arriving message will be redirected.

The actual redirection registration and data structures reside on the network interface memory which is mapped to the user address space. Each user process has a set of redirection data structures allocated, one for each registered redirectable buffer. To post a transfer redirection, a user process invokes the translation library to translate the destination user buffer address into an index This process fills in the UTLB if necessary. Next, the user process fills the translation results as well as other arguments of the vmmc_PostRedir() call into the redirection structure including index, offset, a receive buffer offset where redirection should start, and the size to redirect.

On arrival of each chunk of data from the network, the LANai control program (in the firmware) checks if any bytes of this chunk should be redirected by consulting the redirection structure associated with the default receive buffer of the data. Data that are not redirected are moved into the default buffer. For redirection, the data chunk may be scattered into multiple subchunks because DMA from the network interface to the host memory cannot cross page boundaries or redirection boundaries. Upon the completion of the redirection, the LANai control program will update the redirection status by adding the number of bytes redirected. The LANai control program will also cancel the redirection after processing the last (as indicated by a special header flag) chunk of the message.

vmmc_EndRedir() cancels the posted redirection. If the redirection has been canceled already by the LANai, the call simply collects redirection status from this receive buffer redirection structure in the SRAM. If the redirection has not been canceled yet, this call will set the size in the redirection structure to zero. Since both the LANai and the host can cancel a redirection, race conditions are possible. Our implementation avoids the race conditions by using a redirectionActive flag in the redirection structure. This flag is set by the LANai and read by the host. The LANai sets the flag whenever a redirection is in progress and resets it when the transfer completes. The host can cancel the redirection only after the flag is reset. This method avoids race conditions and ensures that there will be no more transfer of data to the user buffer after the cancellation is completed.

Our implementation of transfer redirection is safe even in the presence of malicious users. Since each redirection structure is mapped by only one user, only its owner can write it. If a redirection structure owner does not follow the protocol described above, the worst thing which can happen is that vmmc_EndRedir() returns wrong size and location of the redirection data which has been deposited to some random physical memory pages taken from this user's redirection TLB. However, the invariant of this TLB is that all such pages are writable by this user, so no harm can be done to other users.

9. Notifications

When sending a message we have the choice of transferring data only or data and control. The mechanism we use to transfer control is called a notification. Notifications are similar to UNIX signals. When a message with a notification attached arrives at destination receive buffer a user-level notification handler is invoked after the message data in user memory.

Handlers can be associated with receive buffers during the export operation. Each receive buffer can have zero or one handler. If a message with a notification arrives at receive buffer with no handler attached, this notification has no effect. vmmc_SendDataNotify() sends message with notification in deliberate update mode. This call takes the same arguments as vmmc_SendData().

Each handler has the same function signature (i.e. number and type of arguments). The first argument is the address of the last word of data transferred by the message that generated this notification, the second argument is the value of this word. Since VMMC continues to receive incoming messages between the arrival of a message with notification and the call to the associated user handler, the data of a notification message can be overwritten by subsequent messages even before the handler is called. However, VMMC makes sure that the handler is called with the value of the last word, as delivered by the message with notification.

VMMC provides two calls: vmmc_BlockNotifications() and vmmc_UnblockNotifications() to control the delivery of notifications. Blocking notifications is useful to ensure consistency of data structures modified by both user-level handlers and main thread of execution. Blocking notifications is global, i.e. it affects all receive buffers of a given process. When notifications are blocked, they are queued by the system. After they are unblocked, they are delivered in-order to the appropriate user-level handlers. Since there is limited space to store queued notifications, they should not be blocked for too long.

For each vmmc_BlockNotifications() there should be a call to vmmc_UnblockNotifications(). Pairs of these calls can be nested. vmmc_UnblockNotifications() unblocks notifications only if it is called at the first nesting level. In this case, the call returns a positive integer, otherwise it returns zero. To make sure that notifications are unblocked unconditionally, one can call vmmc_UnblockNotifications() in the loop until it returns positive integer. If notifications are unblocked, further calls to vmmc_UnblockNotifications()have no effect.

While user-level notification handler is executing, notifications remain blocked. Notification handler should not block or wait spinning. Not all SBL calls can be used from within notification handler. Both vmmc_BlockNotifications() and vmmc_UnblockNotifications() can be called from within the handler, provided they are paired.

However, any attempt to unblock notifications unconditionally by repeated calls to vmmc_UnblockNotifications() will eventually return an error (as notifications must remain blocked within a handler).

10. Removal of Import-Export Link

There are two calls provided to undo export and import operations. The importer of a given buffer executes vmmc_UnimportRecvBuf(). This call undoes a previous vmmc_ImportRecvBuf() call by breaking the connection to the remote receive buffer, and de-allocating the local proxy memory.

vmmc_UnexportRecvBuf() undoes a previous call to vmmc_ExportRecvBuf(). All existing connections to the buffer are forcibly broken, and the buffer is made unavailable for further connections. In particular all importers of this buffer cannot send any messages to this buffer after this call completes.

Most VMMC calls return error status. Negative integer indicates an error, while zero means no error occurred. Errors can be reported with vmmc_Error().

• vmmcErr_BadAlignment: newly created buffer is not aligned to VMMC word boundary; or send or destination address is not aligned to such boundary.
• vmmcErr_BufAlreadyExists: receive buffer with a given buffer id already as been exported
• vmmcErr_BadIbufState: imported receive buffer is in bad state
• vmmcErr_BadRbufState: exported receive buffer is in bad state.
• vmmcErr_BadProxyAddr: proxy address does not belong to imported receive buffer (or first and last proxy addresses specified by vmmc_SendData() do not belong to the same imported receive buffer).
• vmmcErr_BadSize: number of words to be sent is not positive
• vmmcErr_BlockedNotifications: blocked notifications lead to deadlock. This happens when we want to unexport a buffer with pending notifications that are blocked.
• vmmcErr_NoDestSpace: no more destination space
• vmmcErr_NoMem: VMMC cannot allocate memory
• vmmcErr_NoSuchNode: bad node id
• vmmcErr_NoSuchProcess: bad process id
• vmmcErr_NotInHandler: this call cannot be used from within notification handler
• vmmcErr_OverlappedBufs: new receive buffer would overlap with existing buffer
• vmmcErr_Sending: error while sending a message

Returns the names of all hosts that are part of the multi-computer. This includes hosts that may not be part of the user’s current session (see vmmc_SessionHosts()).

1. 
   
   vmmc_AllHosts() returns all the nodes of the VMMC multicomputer. Once a user program is started, nodes should not be added or deleted. The hostIds array should not be freed or written by user program.

• vmmc_Success (the constant 0)
• negative error code on failure

1. 
   
   vmmc_AsyncStatus() returns a status of an asynchronous request initiated with either vmcc_SendDataAsync() or vmcc_SendDataAsyncNotify().

• vmmc_Success (the constant 0)
• vmmcErr_NoSuchError if the message is still in progress
• other negative error code on failure

1. 
   
   vmmc_BlockNotifications() blocks delivery of notifications to all receive buffers of a calling process. Blocked notifications are queued and will be delivered after notifications are unblocked by a call to vmmc_UnblockNotifications(). Each call to vmmc_BlockNotifications() should be paired with a call to vmmc_UnblockNotifications(). These pairs can be nested, in which case the last vmmc_UnblockNotifications() (at nesting level one) will unblock notifications.
   
   Notifications are automatically blocked when notification handler is called. vmmc_BlockNotifications() can be called from within a notification handler.

1. 
   
   VMMC keeps track of the index of the last word sent to redirectable buffers. This word index is relative to the start of a buffer. vmmc_ClearDataEnd() resets this index to vmmcErr_NoSuchError. The index can read with the function vmmc_DataEnd().

1. 
   
   VMMC keeps track of the index of the last word sent to redirectable buffers. This word index is relative to the start of a buffer. vmmc_DataEnd() returns this index for a specified buffer. If the buffer has received no data, vmmcErr_NoSuchError is returned. The user can reset the index to vmmcErr_NoSuchError by using vmmc_ClearDataEnd().

• vmmcErr_NoSuchError when no data has arrived in the buffer or if the user has reset the index with a call to vmmc_ClearDataEnd()
• a positive integer specifying the word index of the buffer where the last word was written to

1. 
   
   Terminates redirection on a specified buffer. The last two arguments return the status of the redirection. It could be that all, some, or none, of the incoming data was redirected. Use *numWordsPtr and *firstWordOffsetPtr to determine exactly how much data was redirected.

• vmmc_Success (the constant 0)
• negative error code on failure

1. 
   
   IN int32 *buf : address of the receive buffer
   
   IN uint32 nwords: size of the receive buffer in words
   
   IN uint32 rbufid: receive buffer id
   
   IN vmmc_callback_t proc: notification handler, user-level function | NULL for no handler
   
   IN uint32 flags: specify export characteristics, constants can be ORed
   BUF_WRITE_ONLY buffer can only be written
   BUF_READ_ONLY buffer can only be read
   BUF_READWRITE buffer can be read and written
   BUF_REDIRECTABLE buffer can redirected
   BUF_GLOBAL_SPACE buffer is allocated from the global buffer space
   
   IN vmmc_perm_t perm: export permissions (not used yet)
   
   OUT vmmc_exphandle_t *bufHandle: handle that will identify the exported buffer

1. 
   
   Exports receive buffer for importing by senders. Note that this call does not allocate any memory. If there is no memory allocated to back this buffer, page faults may occur.
   The newly created receive buffer is identified with rbufid, and occupies memory between buf and
   buf + vmmc_WordSize()*nwords - 1.
   buf should be word aligned. Receive buffers cannot overlap.

• vmmc_Success
• negative error code on failure

1. 
   
   vmmc_GetData() retrieves nwords from address remoteSrcProxyAddr and copies them to the local buffer specified by localDstBuf. vmmc_GetData() returns after the message has arrived in the user buffer.
   
   remoteSrcProxyAddr determines the source of the data. This is the receive buffer which proxy in local DestSpace contains localDstBuf address. Since each DestSpace address belongs to no more than one proxy, this identification is unique. The data will be put in the receive buffer on destination node starting from the offset equal to the offset of localDstBuf from the beginning of this buffer proxy in local DestSpace.
   
   Both remoteSrcProxyAddr and localDstBuf should be VMMC word aligned. Both remoteSrcProxyAddr (source address of the first word) and remoteSrcProxyAddr+4*nwords-1 (source address of the last word) should belong to the same imported receive buffer.
   
   vmmc_GetData() can be used from within notification handler.
   
   Note: Page faults are possible if remoteSrcProxyAddr or localDstBuf do not correspond to valid addresses.

• vmmc_Success
• negative error code on failure

1. 
   
   vmmc_GetData() retrieves nwords from address remoteSrcProxyAddr and copies them to the local buffer specified by localDstBuf. vmmc_GetData() returns after the message has arrived in the user buffer.
   
   remoteSrcProxyAddr determines the source of the data. This is the receive buffer which proxy in local DestSpace contains localDstBuf address. Since each DestSpace address belongs to no more than one proxy, this identification is unique. The data will be put in the receive buffer on destination node starting from the offset equal to the offset of localDstBuf from the beginning of this buffer proxy in local DestSpace.
   
   Both remoteSrcProxyAddr and localDstBuf should be VMMC word aligned. Both remoteSrcProxyAddr (source address of the first word) and remoteSrcProxyAddr+4*nwords-1 (source address of the last word) should belong to the same imported receive buffer.
   
   vmmc_GetData() can be used from within notification handler.
   
   Note: Page faults are possible if remoteSrcProxyAddr or localDstBuf do not correspond to valid addresses.

• vmmc_Success
• negative error code on failure.

1. 
   
   vmmc_ImportRecvBuf() imports receive buffer which has been exported by a process squid executing on node node.
   
   The imported receive buffer is allocated in a region of destination address space between proxyBuf and proxyBuf + vmmc_WordSize()*nwords -1.

1. 
   
   ************************** EDIT DESCRIPTION ************************
   
   vmmc_ImportRecvBuf () imports receive buffer which has been exported by a process squid executing on node node.
   
   The imported receive buffer is allocated in a region of destination address space between proxyBuf and proxyBuf + vmmc_WordSize()*nwords -1.

1. 
   
   ************************** EDIT DESCRIPTION ************************
   
   vmmc_ImportRecvBuf () imports receive buffer which has been exported by a process squid executing on node node.
   
   The imported receive buffer is allocated in a region of destination address space between proxyBuf and proxyBuf + vmmc_WordSize()*nwords -1.

1. 
   
   vmmc_Parent() returns node and process id of the parent process. This call is guaranteed to work only if the caller process has been started with vmmc_Spawn(). In such case, the parent process is a process which executed vmmc_Spawn().

1. 
   
   vmmc_SendData() sends message of nwords taken from address sendAddr. vmmc_SendData() returns after the message has been transferred to the network.
   
   destAddr determines the destination of the data. This is the receive buffer which proxy in local DestSpace contains destAddr address. Since each DestSpace address belongs to no more than one proxy, this identification is unique. The data will be put in the receive buffer on destination node starting from offset equal to offset of destAddr from the beginning of this buffer proxy in local DestSpace.
   
   Both sendAddr and destAddr should be VMMC word aligned. Both destAddr (destination address of the first word) and destAddr+4*nwords-1 (destination address of the last word) should belong to the same imported receive buffer.
   
   vmmc_SendData() can be used from within notification handler.

Note: Page faults are possible if sendAddr or destAddr do not correspond to valid addresses.

1. 
   
   vmmc_SendDataAsync() initiates a send of nwords taken from an address sendAddr. vmmc_SendDataAsync() returns immediately without waiting for message transmission to the network.
   
   destAddr determines the destination of the data. This is the receive buffer which proxy in local DestSpace contains destAddr address. Since each DestSpace address belongs to no more than one proxy, this identification is unique. The data will be put in the receive buffer on destination node starting from offset equal to offset of destAddr from the beginning of this buffer proxy in local DestSpace.
   
   Both sendAddr and destAddr should be VMMC word aligned. Both destAddr (destination address of the first word) and destAddr+4*nwords-1 (destination address of the last word) should belong to the same imported receive buffer.
   
   vmmc_SendDataAsync() can be used from within notification handler.

Note: The send buffer cannot be reused until this message completion status is checked with vmmc_AsyncSendStatus(). Page faults are possible if sendAddr or destAddr does not correspond to valid addresses.

1. 
   
   vmmc_SendDataAsync() initiates a send of nwords taken from an address sendAddr. vmmc_SendDataAsync() returns immediately without waiting for message transmission to the network.
   
   destAddr determines the destination of the data. This is the receive buffer which proxy in local DestSpace contains destAddr address. Since each DestSpace address belongs to no more than one proxy, this identification is unique. The data will be put in the receive buffer on destination node starting from offset equal to offset of destAddr from the beginning of this buffer proxy in local DestSpace.
   
   Both sendAddr and destAddr should be VMMC word aligned. Both destAddr (destination address of the first word) and destAddr+4*nwords-1 (destination address of the last word) should belong to the same imported receive buffer.
   
   vmmc_SendDataAsync() can be used from within notification handler.

Note: The send buffer cannot be reused until this message completion status is checked with vmmc_AsyncSendStatus(). Page faults are possible if sendAddr or destAddr does not correspond to valid addresses.

1. 
   
   vmmc_SendDataWithNotify() sends message with notification of nwords taken from address sendAddr. vmmc_SendDataWithNotify() returns after message has been transferred to the network.
   
   destAddr determines the destination of the data. This is the receive buffer which proxy in local DestSpace contains destAddr address. Since each DestSpace address belongs to no more than one proxy, this identification is unique. The data will be put in receive buffer on destination node starting from offset equal to offset of destAddr from the beginning of this buffer proxy in local DestSpace.
   
   Both sendAddr and destAddr should be VMMC word aligned. Both destAddr (destination address of the first word) and destAddr+4*nwords-1 (destination address of the last word) should belong to the same imported receive buffer.
   
   On receiving this message, a user-level handler associated with receive buffer will be called after the message data is transferred into memory. If no handler has been specified by vmmc_ExportRecvBuf(), this call is equivalent to vmmc_SendData() since notifications are not used.

Note: Page faults are possible if sendAddr or destAddr does not correspond to valid addresses.

1. 
   
   IN char *execfile: new process will execute file of this name. execfile given as full path on remote
   machine guarantees its successful location.
   
   IN char **argv: list of arguments for new process, starting from argv[1]. Last element of this list
   should be NULL.
   
   IN NodeId node: identifies machine on which new process will be started.
   
   OUT vmmc_Squid *squid: returns squid of a new process.

1. 
   
   vmmc_Spawn() starts a new process.
   You can use rexec to do that too (if you know how to get the process id).

1. 
   
   vmmc_UnblockNotifications() conditionally unblocks delivery of notifications to all receive buffers of a calling process. VMMC maintains internal counter which counts blocking level of notifications. This counter is incremented each time vmmc_BlockNotifications() is called. When vmmc_UnblockNotifications() is called, and this counter is positive, it is decremented. When this counter is zero, a call to vmmc_UnblockNotifications() has no effect. Notifications are unblocked only if this counter reaches zero.
   
   Notifications are automatically blocked when notification handler is called. vmmc_UnblockNotifications() can be called from within notification handler, but it cannot actually unblock notifications in such case. If the internal counter is one, and vmmc_UnblockNotifications() is called from the handler, this call returns the error vmmc_ENotInHandler and the counter remains unchanged.