nbdkit-streaming-plugin(1)

📦 version 1.24.1, in ubuntu jammy

💖 nbdkit-streaming-plugin 💖

okay bestie, so this is the manual for nbdkit-streaming-plugin, one of the lil plugins that ships with nbdkit (that's section 1 of the manual, iykyk). it streams stuff in or out of a local pipe or socket and honestly? it's giving main character energy. let's get into it 🫶

✨ NAME ✨

nbdkit-streaming-plugin — the nbdkit streaming plugin, no cap

🌈 SYNOPSIS 🌈

nbdkit streaming write=PIPE [size=SIZE]

nbdkit streaming read=PIPE [size=SIZE]

💅 DESCRIPTION 💅

so nbdkit-streaming-plugin is a plugin for nbdkit(1) and what it does is stream data in or out of a local pipe or socket. literally that's the whole vibe.

if you were looking for an alternate tool that does something similar, that would be nbdcopy(1). and if your goal is to turn an NBD export into a local file, you want nbdfuse(1) instead. different queens, different jobs 💅

okay so here's the tea: if the NBD client opens the NBD port and writes from the start to the end of the disk without seeking backwards, you can turn that into a stream of data on a local pipe or socket like this:

nbdkit streaming write=./pipe

and here's how that looks, visualized:

               ┌───────────┐
NBD           │ nbdkit    │        plugin streams
client ──────▶│ streaming │──────▶ data to ./pipe
writes        │ plugin    │
               └───────────┘

and the other way around — if the NBD client opens the NBD port and reads from the start to the end of the disk without seeking backwards, then you can turn a local pipe or socket into a stream of data for that client:

nbdkit streaming read=./pipe

visualized:

                  ┌───────────┐
plugin streams   │ nbdkit    │        NBD
data from ──────▶│ streaming │──────▶ client
./pipe           │ plugin    │        reads
                  └───────────┘
heads up bestie: ./pipe (or the local socket) sees raw data — it is not using the NBD protocol. if you actually want to forward NBD to a local socket connected to another NBD server, use nbdkit-nbd-plugin(1). don't get it twisted 🙈

🧚 Combining this plugin with qemu tools 🧚

typical usage is with qemu tools, fr. so the following command does not work, because the output is a pipe or socket (and qemu is dramatic about it):

$ mkfifo pipe
$ qemu-img convert -n input -O raw ./pipe
qemu-img: Could not open './pipe': A regular file was expected by
the 'file' driver, but something else was given

but this? this works:

nbdkit -U - streaming write=./pipe \
       --run ' qemu-img convert -n input -O raw $nbd '
                 ┌───────────┐       ┌───────────┐
reads            │ qemu-img  │       │ nbdkit    │        plugin streams
input ──────▶    │ convert   │──────▶│ streaming │──────▶ data to ./pipe
file             │ command   │       │ plugin    │
                 └───────────┘      ↑└───────────┘
                                 Unix domain socket (-U -)

this only works because the output format is raw and qemu-img can write that linearly — no seeking backwards, no writing blocks out of order, periodt. this approach cannot work for other formats such as qcow2, because those contain metadata that must be updated by seeking back to the start of the file, and that's just not possible when the output is a pipe. it's giving impossible, because it is 💀

the reverse situation is getting qemu-img to read from a pipe:

nbdkit -U - streaming read=./pipe \
       --run ' qemu-img convert -f raw $nbd -O qcow2 output.qcow2 '

for use of the --run and -U - options, go check out nbdkit-captive(1) ✨

🎀 PARAMETERS 🎀

either read or write is required, but not both. pick a lane bestie.

  • read=FILENAME — read the data stream from the named pipe or socket. if the pipe or socket does not exist, then it is created (as a named FIFO); otherwise the existing pipe or socket is opened and used. (nbdkit ≥ 1.22)
  • write=FILENAME — write the data stream to the named pipe or socket. if the pipe or socket does not exist, then it is created (as a named FIFO); otherwise the existing pipe or socket is opened and used.
  • pipe=FILENAME — for backwards compatibility with nbdkit ≤ 1.20, this is a synonym for write=FILENAME. it's the vintage edition, we love her but she's legacy 💜
  • size=SIZE — specify the virtual size of the stream.

this parameter is optional. if you don't specify it, then the virtual disk appears to the client to be very large (effectively infinite). whether you need to specify this parameter depends on the client — some clients don't check the size and just write/stream, and others do checks or calculations based on the apparent size. it is not possible for the plugin to work out the size itself, since it doesn't see the whole stream until it has all been streamed. like, it literally cannot know until it's over 😭

🌸 FILES 🌸

  • $plugindir/nbdkit-streaming-plugin.so — the plugin. use nbdkit --dump-config to find the location of $plugindir.

⭐ VERSION ⭐

nbdkit-streaming-plugin first appeared in nbdkit 1.2. a true icon since day one 👑

🦋 SEE ALSO 🦋

nbdkit(1), nbdkit-file-plugin(1), nbdkit-nbd-plugin(1), nbdkit-plugin(3), nbdkit-captive(1), nbdcopy(1), nbdfuse(1), qemu-img(1).

the whole friend group, we stan 🫶

💖 AUTHORS 💖

Richard W.M. Jones

Copyright (C) 2014-2020 Red Hat Inc.

👑 LICENSE 👑

redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

  • Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
  • Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
  • Neither the name of Red Hat nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

and that's the whole page bestie!! this one's from the nbdkit-1.24.1 era, dated 2021-01-20, and it slayed then and it slays now. go stream something, ily, bye 💖✨🎀

📦 where this page comes from