vMMIO Introduction
On physical devices, software and hardware interact in a predefined way to yield predefined results. Namely, hardware is driven by software drivers according to its specification. Fundamentally, this interaction distills into a series of address and data bus read and write operations. On AVH virtual devices this behavior is identical, save for the distinction that software interacts with virtual software-defined models of the physical hardware device being virtualized.
AVH implements and extends virtual hardware model capabilities through what is referred to as virtual Memory Mapped IO (vMMIO). This allows user-supplied code to be executed in response to VM physical memory read or write bus transactions. The full suite of features available also include DMA for arbitrary memory read and write, as well as control of a dedicated virtual IRQ line.
The CoreIO Library API
The CoreIO library allows external programs to act as MMIO responders, send interrupt requests to the VM, and perform DMA.
Connecting to a target VM
To use the library in your C program, you need to connect to a VM:
int coreio_connect(const char *target);
Connect to a VM.
target string like "10.10.0.3:23456"
Returns error flag.
If the result was zero, you can continue using the CoreIO API.
Registering a vMMIO Range
The vMMIO ranges must be created either through the web API or through the UI. After VM creation, a vMMIO range may be created by using the relevant tab under Settings.
Start and Size represent the physical address space of the virtual MMIO range being created. The IRQ is used when vMMIO handlers implement a virtual IRQ line linked to the core interrupt controller. And port represents the network port on which AVH will listen for connections.
:::Note vMMIO ranges created without any associated client handlers (below) will not trigger bus faults within the VM. The default behavior is to return a 0xaa in each byte up to the width size of the access (e.g. 0xaaaa for two byte reads). :::
Registering a vMMIO Handler
After registering the range, the access handler must also be registered so that AVH knows where to route the access events to.
The API is based around a few concepts. First, there's the vMMIO range. Each range is a 4k-aligned block of address space that a CoreIO client can claim. Ranges are indexed from 0, in the order they were specified during VM creation. Up to 16 vMMIO ranges can be created per VM instance.
To claim a vMMIO range, create and fill out a callback structure:
//add defines here
typedef struct {
/* MMIO operations. */
int (*read)(void *priv, uint64_t addr, size_t len, void *buf, unsigned flags);
int (*write)(void *priv, uint64_t addr, size_t len, void *buf, unsigned flags);
/* MMIO pair-wise operations. If not supplied, read/write are used twice. */
int (*readp)(void *priv, uint64_t addr, size_t len, void *buf1, void *buf2, unsigned flags);
int (*writep)(void *priv, uint64_t addr, size_t len, void *buf1, void *buf2, unsigned flags);
} coreio_func_t;
The read/write functions are called with:
priv- your unmodified private pointer (funcpfrom register function)addr- VM physical address of accesslen- Byte length of access, for example 1, 2, 4, 8, 16 ...buf- Data buffer (contains VM write contents on write, else destinations buffer for reads)flags- Flags describing access mode, seeCOREIO_FLAGS_*
The functions can return 1 to signal failure, which will become a hardware-origin bus error when it hits the VM.
After you have your handlers and coreio_func_t filled out, register it:
int coreio_register(unsigned rid, const coreio_func_t *func, void *funcp);
Register a MMIO handler.
rid MMIO range ID to register
func io handler function table
funcp io handler function parameter (passed to callbacks)
Returns error flag.
Processing Read/Write Events
Then you must construct a main loop of your program. The main loop support is intended to work with select(). You call coreio_preparefds before select() to add CoreIO's sockets to select wait list, and coreio_processfds after select to process the socket events:
int coreio_preparefds(int nfds, fd_set *readfds, fd_set *writefds);
Prepare fd_sets for select(2).
nfds current index of maximum fd in sets + 1
readfds readfds to update
writefds writefds to update
Returns new nfds.
int coreio_processfds(fd_set *readfds, fd_set *writefds);
Process fd_sets after select(2).
readfds readfds to process
writefds writefds to process
Returns error flag (for instance, connection to VM lost).
A simple implementation of a main loop, without any other event processing, looks like this:
/* ... */
fd_set readfds, writefds;
int res;
while(1) {
FD_ZERO(&writefds);
FD_ZERO(&readfds);
int nfds = coreio_preparefds(0, &readfds, &writefds);
if(nfds < 0) /* error in prepare */
break;
select(nfds, &readfds, &writefds, NULL, NULL);
res = coreio_processfds(&readfds, &writefds);
if(res) /* error in process */
break;
}
/* ... */
This kind of main loop, with a timeout, is provided as a convenience by the library:
int coreio_mainloop(long long usec);
Simple implementation of a main loop.
usec time to spend in loop, in microseconds;
negative means forever
Returns error flag.
Interrupts
In addition to processing MMIO events, your program can also send interrupts. Note that in the VMs, all interrupt lines are level. The interrupt controller can interpret them as edge, but to trigger an edge interrupt you must send two updates: one from 0 to 1, one from 1 to 0. It is not sufficient to send 1 on each IRQ trigger as this does not create an edge event.
int coreio_irq_update(unsigned rid, unsigned iid, unsigned state);
Send an IRQ update.
rid MMIO range ID associated with IRQ
iid IRQ index
state IRQ line state (1 - active, 0 - inactive)
Direct Memory Access (DMA)
There are two ways to perform DMA access, blocking and non-blocking. Blocking access is simpler. You just call a read/write function, and it returns when done. Non-blocking access requires a completion function (and its opaque parameter pointer, which will be passed to it without change). The function will be called at a future time during coreio_processfds().
int coreio_dma_read(unsigned mid, unsigned sid, uint64_t addr,
size_t len,void *buf, unsigned flags,
void (*cpl)(void *, int), void *cplp);
Start a DMA read.
mid IOMMU ID (0 = direct)
sid IOMMU region ID
addr target address
size size of read to perform
buf memory buffer to fill with data
flags bus access flags
cpl completion function; if NULL, call is blocking
cplp completion function parameter
Returns error flag.
int coreio_dma_write(unsigned mid, unsigned sid, uint64_t addr,
size_t len, void *buf, unsigned flags,
void (*cpl)(void *, int), void *cplp);
Start a DMA write.
mid IOMMU ID (0 = direct)
sid IOMMU region ID
addr target address
size size of write to perform
buf memory buffer with data
flags bus access flags
cpl completion function; if NULL, call is blocking
cplp completion function parameter
Returns error flag.
On most small VMs, you can pass 0 as IOMMU ID and region ID. This specifies that the access should target VM physical memory directly.
Finally, to cleanly shut down the CoreIO library, use:
void coreio_disconnect(void);
Examples
The example sources, available as a tarball, include the following:
rpi-test
- Use an inside and outside VM agent for demonstrating end-to-end vMMIO cycle
slider-mmio
- Use Python bindings to send an IRQ to the VM depending on the state of a graphical slider
vsi-waveio
- Create a whole virtual device model confronting to Arm's VSI specification that streams a WAV file into the VM
Further Reading
For an additional example, please see our vMMIO Example on Raspberry Pi 4 article.