[vsock] better handle Lifecycle (#1104)

This PR extends the VsockPoller event loop so that it complies with the Lifecycle trait within propolis.

The virtio device now requests a pause to the event-loop by sending a VsockEvent::Pause via port_send(3C).
When the event loop receives this event it finishes processing any tasks and then waits on a mpsc channel. This channel is responsible for processing follow up events driven by the virtio device lifecycle such as resume, reset, and halt. The reset event takes care of cleaning up all state internal to VsockPoller in addition to removing the cached descriptor chain in VsockVq. It's important we drop this cached descriptor as using it across device resets would cause propolis to scribble data into a random location in guest memory depending on where the GuestAddr points.

Finally with this work in place it should make solving #1065 trivial.

Author: papertigers

lib/propolis/src/hw/virtio/vsock.rs

@@ -151,6 +151,15 @@ impl VsockVq {
 
         Some(packet)
     }
+
+    /// Drop any cached descriptor chain.
+    ///
+    /// This MUST be called when reseting the virtio-socket device so
+    /// that we don't use stale `GuestAddr`s across device resets.
+    #[cfg(target_os = "illumos")]
+    pub(crate) fn clear_rx_chain(&mut self) {
+        self.rx_chain = None;
+    }
 }
 
 pub struct PciVirtioSock {
@@ -253,8 +262,19 @@ impl Lifecycle for PciVirtioSock {
     fn type_name(&self) -> &'static str {
         "pci-virtio-socket"
     }
+    fn pause(&self) {
+        let _ = self.backend.pause();
+        self.backend.wait_stopped();
+    }
     fn reset(&self) {
         self.virtio_state.reset(self);
+        self.backend.reset();
+    }
+    fn resume(&self) {
+        self.backend.resume();
+    }
+    fn halt(&self) {
+        self.backend.halt();
     }
     fn migrate(&'_ self) -> Migrator<'_> {
         // TODO (MTZ):