Skip to content

NODES.md

Latest commit

 

History

History
1386 lines (1090 loc) · 75.5 KB

NODES.md

File metadata and controls

1386 lines (1090 loc) · 75.5 KB
Raw

input

1 output: av::Packet

  • url (string of URL)
  • options (dictionary) - options for libavformat
  • timeout (float, seconds) - packet read timeout
  • initial_timeout (float, seconds) - URL open timeout

input_rec

1 output: av::Packet

This node is a special case on the input node with some changes required to plae & seek recoding files. It has the same parameters as the input node and some additional

  • url (string of URL)
  • options (dictionary) - options for libavformat
  • timeout (float, seconds) - packet read timeout
  • initial_timeout (float, seconds) - URL open timeout
  • live_delay (float, seconds) - time to delay between latest available packet and currently displayed in live mode (default 1)
  • start_ts (string) - start timestamp
  • stop_ts (string) - stop timestamp
  • loop (bool) - if set to true, video will be played in a loop
  • loop_continuous_ts (bool, default false) - with loop, shift each forward pass by the span of the previous one (earliest timestamp to latest timestamp + duration of the video stream, or of all streams in a file without video), so timestamps keep rising across loops instead of jumping back to the start. All streams get the same shift to stay in sync. Seek targets and stop_ts still refer to file timestamps, while output packets carry the shifted ones; intended for free-running loops.
  • stop_delay (int, miliseconds) - time to hold on last read frame before reporting end od processing
  • seek_table (string of URL) - file with fast-seek offsets (may be generated by the output node)
  • ts_offsets (string of URL) - file with timestamp offsets (may be generated by the sentinel node)
  • timestamp_source (string) - timestamp source used to synchronize multiple videos. Possible values are wallclock and input. Both values are calculated from the timestamps offsets file (ts_offsets)
  • preseek (float, seconds, default 0) - how many seconds to preseek back to increase chance of finding a keyframe, when seek to timestamp (seek command) is requested
  • team (string, name of instance-shared object) - if specified, seeks on the input nodes may be synchronized inside specified team
  • frame_metadata (list of strings, default: all of them) - which keys the node stamps on the frames its decoders hand back: video_ts, video_pts, input_ts, output_ts, wallclock_ts, wallclock, frame_no, frame_ts on video frames (frame_no, and frame_ts from the table, need seek_table), frame_ts and sample_rate on audio frames. realtime reads frame_no, frame_ts and wallclock for its team and playback info and speed_video reads frame_ts; a graph that reads none of them may pass [] and spare every frame the stamping and every hop the dictionary copy
  • routing (dictionary of string => string) and streams_filter (string) - as in demux: each routed stream goes straight to its own queue, given instead of dst. Packets of streams without a route are dropped and the EOF marker goes to every routed queue. Spares the demux hop for a file whose streams the node reads anyway; wait_for_keyframe stays with demux

Runtime control: node.object.set <name> request-eof true interrupts further input reads while keeping the producer alive to drain buffered demux/parser packets. It then sends one common packet EOF and finishes (respecting stop_delay), even if looping or ordinary EOF notification was disabled. This is irreversible and intended for a running forward input whose live publisher has finished but whose relay keeps the connection open. It preserves normal downstream backpressure; the caller must keep consumers running and enforce a shutdown deadline. node.stop remains an immediate abort.

realtime

Rate limit output packets/frames to wallclock. This way, DTS (in packets) or PTS (in frames) differences will equal wallclock differences at this node's sink.

Also, allows inter-stream synchronization (as long as timestamps in them are synchronized) and automatic flushing if too much data is buffered in queues before this node.

This node is non-blocking.

1 input, 1 output: anything

  • leak_after (float, seconds) - if specified, bypass rate limiting after having input packets available immediately (in other words, at least one packet was always enqueued) for specified time. Intended for segmented inputs for which realtime node is useful to prevent bursts, while we don't want clock drift problems and missed segments in long running streams.
  • speed (float) - default 1, implemented by scaling wallclock's timebase (millisecond precision) so values between ~0.9995 and ~1.0006 are treated as 1.
  • tick_period (string of rational, seconds) - if specified and tick_source is also specified, anti-jitter filter will be enabled, assuming that tick source emits a tick every tick_period. Generally should be set to 1/FPS, e.g. 1/60. The filter maintains its own clock independent of wallclock, but will resync to the wallclock if it drifts too much. Without tick_source, pacing still follows wallclock, but a specified tick_period selects an internal time base of one quarter of that period. To choose only the time base, use time_base.
  • time_base (string of rational, seconds, optional) - the unit the node counts in and set_pts stamps in. Default: the millisecond wallclock time base. 1/120000 represents 1/30, 1001/30000, 1001/60000 and 1001/24000 exactly. Nothing ticks at this rate: the node still wakes for a frame and for its release time. The clock it reads counts milliseconds, so a frame whose release time falls inside the current millisecond is released then, up to a millisecond early, and stamped with its exact time. Not together with tick_period. With a finer time base, set negative_time_tolerance and discontinuity_threshold explicitly, in seconds, to keep their durations; the built-in values assume milliseconds.
  • set_pts (bool, default false) - set PTS to wallclock timestamps corresponding to time when packets are outputted (or, more precisely, when they would be outputted if there was no jitter)
  • forward_eof (bool, default false) - pass the end-of-stream marker on to the output, in order behind the last frame and never stamped by set_pts. By default the marker is consumed here; a consumer that must tell the end of a file from a stalled producer (clip_cache) needs it. Do not enable it in front of a node that finishes on the marker unless that is intended.

Input tolerance parameters:

  • negative_time_tolerance (float, seconds) - default 0.25. Do not resync if newly arrived packet should have been emitted at most that much time in the past. 0 to disable and always resync in such situation - effectively increasing latency until sufficient buffering for smooth output is achieved.
  • negative_time_discard (float, seconds) - if specified, if newly arrived packet should have been emitted at least that much time in the past, discard this packet. Discarding has lower priority than resyncing (negative_time_tolerance), so the value must be less than negative_time_tolerance to make sense, equal or higher values disable discarding.
  • discontinuity_threshold (float, seconds) - default 1. If we need to wait for more than specified time, treat as discontinuity and resync. Default value may be unsuitable (too small) for multiple source synchronization in case more data is buffered.
  • jitter_margin (float, seconds) - default 0. When (re)syncing, add this value to the time to be waited. This prevents frequent resyncing and visible jitter at the cost of higher latency. It makes sense only with unbuffered output (e.g. display)
  • initial_jitter_margin (float, seconds) - default = jitter_margin. jitter_margin to use for the first frame received after node start, after discontinuity or after leak_after-triggered bypass, but not after "negative time to wait (...), resyncing"

Inter-stream synchronization parameters:

  • team (string, name of instance-shared object) - if specified, realtime nodes with the same team will cooperate to have their output synchronized. Use only if timestamps are synchronized.
  • master (bool) - default true. Only masters are allowed to resync in case of discontinuity. A team can have multiple masters.

Automatic flushing parameters (experimental):

  • input_ts_queue (string, name of queue) - which queue should be treated as the beginning of buffering chain. Usually should be set to the output of the demuxer. If unspecified, automatic flushing is disabled.
  • intermediate_queues (list of strings) - list of intermediate queues that will be examined whether they contain packets
  • max_buffered (float, seconds, default 5.5) - start flushing when the buffering chain has more than this amount buffered (calculated as difference between timestamp of last packet inserted into the input_ts_queue and timestamp of the frame coming to this node)
  • min_buffered (float, seconds, default 0.5) - stop flushing when the buffering chain has less than this amount buffered, or input_ts_queue and intermediate_queues are all empty

For each passing packet, time to wait is computed (how long should we sleep before outputting that packet, to maintain realtime output rate) and different actions are performed based on its value

  • if timeToWait <= 0:
    • if timeToWait < -negative_time_tolerance:
      • resync and emit packet
    • else if timeToWait < -negative_time_discard:
      • discard this packet
    • else: /* -negative_time_discard <= timeToWait <= 0 */
      • emit packet immediately
  • else: /* timeToWait > 0 */
    • if timeToWait < discontinuity_threshold:
      • normal behavior - wait timeToWait and emit packet
    • else: /* timeToWait >= discontinuity_threshold */
      • resync and emit packet

Note: This pseudocode omits leak_after logic.

Recommended options for encoding from segmented input (HLS, DASH):

  • speed: 1.01
  • leak_after: segment length * 2

Recommended options for displaying live video:

  • negative_time_tolerance: 0.001 - 0.02 depending on clock precision of your system
  • jitter_margin: 0.1 or some more

demux

1 input, many outputs: av::Packet

  • streams_filter (string): if present, filter input streams according to ffmpeg syntax (for example "p:1" to select Program 1 - useful for HLSes) before parsing routing keys
  • routing (dictionary of string => string): stream mapping, keys are streams in input file, values are queues, like { "v:0": "videoin", "a:0": "audioin" }, used instead of dst
    • keys may be prefixed with ? to indicate optional input (ignore the route if stream not found)
  • wait_for_keyframe (bool): default false, discard packets until a keyframe appears in any video stream

input_rec accepts the same routing and streams_filter and then needs no demux.

dec_video, dec_audio

1 input: av::Packet, 1 output: av::VideoFrame or av::AudioSamples respectively

  • codec (string) - optional, codec name, auto-detected by libavcodec if not specified
  • codec_map (dictionary of string => string) - optional, use specified decoder for matching stream's codec, for example to use cuvid for h264 and hevc streams: "codec_map": {"h264": "h264_cuvid", "hevc": "hevc_cuvid"}
  • pixel_format (string) - optional, if unspecified, libavcodec and/or codec will select best possible pixel format for given input stream. As seen in pix_fmt field of FFmpeg's AVPacket, so it doesn't have to be any real pixel format (e.g. yuv420p) but can also be hardware acceleration specification (e.g. cuda)
    • if starts with ?, prefer specified pixel format, e.g. ?cuda, but allow use of any
    • if doesn't start with ?, force specified pixel format, fail if it's incompatible with codec or stream
  • hwaccel (string, name of instance-shared object) - optional, name of hwaccel previously created with hwaccel.init
  • hwaccel_only_for_codecs (list of strings) - use hwaccel only for specified input stream codecs, useful because apparently setting hw_device_ctx in normally-software libavcodecs triggers frame corruption bugs
  • options (dictionary) - optional, options passed to libavcodec

extract_timestamps

Set PTS to timecode from video frame's side data.

1 input, 1 output: av::VideoFrame

  • team (string, name of instance-shared object) - if specified, multiple extract_timestamps and extract_timestamps_slave nodes within the same team will share the same offset, so streams not containing timecode side data (e.g. audio) will be synchronized to the same timecode, too (as long as their PTSes are synchronized to each other).

  • passthrough_before_available (bool) - default false, meaning that processing will be blocked (and input queue will grow) as long as timecode isn't available. If true, frames will be passed through with PTS unchanged in such case.

  • drop_before_available (bool) - discard incoming packets before timecode is available. Disabled by default. Has lower priority than passthrough_before_available.

  • timecodes (list of strings), default ["S12M"] - side data to get timecodes from. If specified timecode doesn't exist, next one in the list is tried. Possible items:

    • S12M.1 or S12M - SMPTE 12M = SEI
    • S12M.2
    • S12M.3
    • GOP

  • liveu (bool, default false) - workaround for LiveU-encoded SMPTE 12M. Treat drop bit as a part of frames field.

  • frame_rate_source (string) - timecodes have frame numbers, so to calculate PTS from them, number of frames per second must be known. Possible values:

    • fps - default. Use FPS from nearest node implementing IFrameRateSource (decoder, filter or force_fps)
    • timebase - use 1/timebase from nearest node implementing ITimeBaseSource (decoder, filter or force_fps)

    both values may yield the same or different results depending on input stream. Some video streams are generated by skipping every second frame from higher FPS stream, with S12M side-data preserved in remaining frames. In such cases fps is wrong and timebase is correct.

extract_timestamps_slave

Set PTS to timecode extracted by extract_timestamps node.

1 input, 1 output: av::VideoFrame or av::AudioSamples

Supports parameters working the same as in extract_timestamps node:

  • team - mandatory
  • passthrough_before_available
  • drop_before_available

filter_video, filter_audio

1 input, 1 output: av::VideoFrame or av::AudioSamples, respectively

  • graph (string) - FFmpeg filter graph
  • hwaccel (string, name of instance-shared object) - optional (mandatory for some filters), name of hwaccel previously created with hwaccel.init
  • threads (int >= 0, default 0) - slice threads of the FFmpeg filter graph (nb_threads); 0 starts one per CPU for every graph. Set 1 for graphs of GPU filters, setparams, settb and the like, which do no CPU slice work.

speed_video, speed_audio

Change timestamps so that playback speed changes in real-time.

1 input, 1 output: av::VideoFrame or av::AudioSamples, respectively

  • team (string, name of instance-shared object, default "default") - team name that will be used for changing the speed using speed.set command
  • discard_when_speed_changed (bool, default false) - discard frames when speed isn't equal to 1. Intended for audio streams.
  • timebase (string of rational) - optional, if specified, will scale incoming timestamps to this timebase. Otherwise original timebase will be preserved. To get as smooth output as possible, set it to your native output timebase (1/fps).
  • speed (float) - initial speed, default: 1
  • sync_team (string, name of instance-shared object) - realtime team name. Required by smooth change playback direction (eg. from forward to backward) to flush all the queues betweeen input and the realtime nodes.

pause

Stop passing through frames/packets and resume on request (pause, resume commands).

1 input, 1 output: anything

  • team (string, name of instance-shared object, default "default") - team name that will be used for control
  • paused (bool) - sets the team initially in paused state

force_fps

Duplicate and drop frames to achieve requested FPS

1 input, 1 output: av::VideoFrame

  • fps (string of rational) - target FPS as a string, e.g. 25 or 30000/1001
  • center_phase (bool, default false) - shift input timestamps by a constant, chosen on the first frame and again after a discontinuity, so that frame lands exactly on an output tick. Timestamp jitter below half a frame (e.g. from realtime with set_pts) then never duplicates or drops frames, whatever the source's phase. Output timestamps move by less than half a frame.

smooth_timestamps

Overwrite timestamps with a smoothed monotonic timeline (previous + duration), while keeping A/V sync by resyncing to input timestamps when averaged drift grows too large.

This node is non-blocking.

1 input, 1 output: av::VideoFrame or av::AudioSamples

For video:

  • duration (string of rational, seconds) - frame duration, typically 1/FPS (e.g. 1/25, 1001/30000), or
  • fps (string of rational) - frames per second (alternative to duration)

For both video and audio:

  • resync_threshold (float seconds, default 0.02) - resync output timeline to input PTS when averaged drift exceeds this
  • drift_window (int, default 300) - drift averaging window size (samples); larger = smoother, smaller = more reactive
  • min_samples_before_resync (int, default 150) - ignore drift until this many frames/samples have been observed
  • discontinuity_threshold (float seconds, default 2.0) - hard reset (resync) when input PTS jump exceeds this

assume_video_format / assume_audio_format

Set initial metadata to allow nodes that rely on them to start when real metadata aren't available yet.

1 input, 1 output: av::VideoFrame or av::AudioSamples

Shared parameter (also supported by fake_video_format / fake_audio_metadata):

  • ignore_eof (bool, default false) - discard upstream EOF markers and keep processing subsequent frames. Enable only at a live reconnect boundary before a sentinel. The default forwards EOF and finishes; explicit shutdown and errors are unaffected.

Parameters for video:

  • width (int) - default 1920
  • height (int) - default 1080
  • pixel_format (string) - default yuv420p
  • real_pixel_format (string) - specify only if pixel_format is hardware-accelerated (e.g. cuda)

Parameters for audio:

  • sample_rate (int) - default 48000
  • sample_format (string) - default s32p
  • channel_layout (string) - default stereo

sentinel_video / sentinel_audio

A sentinel

  • guards streams against wild timestamps - PTSes jumping forward or backward, or repeating timestamps
  • inserts backup frames when input signal is not available for specified time:
    • in audio stream: silence
    • in video stream, for maximum time of freeze parameter: repeated last frame
    • in video stream: custom slate, can be used for adding "we'll be back shortly" card

Sentinel's output has "ideal" timestamps with tolerance specified in sentinel's parameters. In other words, it ensures output stream continuity.

1 input, 1 output: av::VideoFrame/av::AudioSamples

  • timeout (float, seconds) - default 1, seconds to wait for input frame before inserting frozen or backup frame

  • correction_group (string, name of instance-shared object) - optional, defaults to "default", used for sharing the clock between streams

  • forward_start_shift (bool):

    • true: if input streams are present when starting the sentinels, forward relative shifts of their first packets (i.e. A-V offset) to output.
    • false (default): start all output streams at exact PTS = start_ts

  • max_streams_diff (float, seconds) - default 0.001, tolerance in seconds

  • start_ts (float, seconds) - default 10, first output timestamp

  • lock_timeshift (bool) - after receiving first PTS, maintain constant input-output PTS difference. Disabled by default. Enable only if you're sure that input timestamps are synchronized to a real-time clock.

  • eof_passthrough (bool) - default false. When true, an upstream EOF marker is forwarded downstream and this node finishes instead of treating EOF as a stall and generating backup frames. Use for VOD / faster-than-realtime graphs that need encoder flush and output trailers. Live graphs should leave this off.

  • reporting_url (optional, string of URL) - if specified, correction time shift changes will be reported to this URL as HTTP POST with JSON body:

    {"changed_at":128.1,"input_pts_offset":126.75999999999999,"output_pts_offset":10.0}

    • changed_at - output timestamp of the change relative to first output PTS (output_pts_offset)
    • input_pts_offset - what sentinel needs to add to input timestamp to achieve output PTS, minus output_pts_offset
    • output_pts_offset = first output PTS, constant through processing, changeable using start_ts parameter

For video only:

  • freeze (float) - default 5, seconds to duplicate last good frame before outputting backup frame
  • backup_frame (string of URL) - backup frame (slate) image
  • backup_picture_buffer (string, name of instance-shared object) - read backup frame (slate) from this buffer. Use picture_buffer_sink to write frame to the buffer. Sentinel will reload the slate from the picture buffer every 64 frames and on every input signal break triggering slate insertion.
  • initial_picture_buffer (string, name of instance-shared object) - initialize last frame buffer with this buffer, so that at the beginning of stream it will be used for at most freeze duration. Useful to insert black frame instead of slate at the beginning when forward_start_shift is set to false. If unspecified, regular backup_frame or backup_picture_buffer will be used.

For sentinel_video, either backup_frame or backup_picture_buffer must be provided.

rescale_video

Dynamic video scaler, maintains constant output dimensions and pixel format even if input stream changes parameters. Does not resample FPS (see force_fps node)

1 input, 1 output: av::VideoFrame

resample_audio

Dynamic audio resampler, maintains continuity of output stream even if input stream changes parameters.

1 input, 1 output: av::AudioSamples

  • dst_sample_rate (int)
  • dst_channel_layout (string)
  • dst_sample_format (string)
  • compensation (float)
    • 0 (default) means brutal compensation using built-in avplumber's sample dropping algorithm
    • any negative value means brutal compensation using libswresample, may not work correctly
    • positive value between 0 and 1 means soft compensation using libswresample, value means fraction of samples to compensate, may not work correctly

split

1 input, multi outputs: anything

  • drop (bool) - drop packets if output queue is full, disabled by default

one_to_many, one_to_many_nb

1 input, multi outputs (at most 32): anything. Each frame is handed to the outputs selected by a mask that can change while the node runs.

  • outputs (int or bit string, default 1) - the outputs that receive frames, bit n for the n-th name in dst; node.object.set <node> outputs <mask> changes it
  • drop (bool, default false) - drop a frame for an output whose queue is full instead of waiting for room
  • frame_color (object, optional) - colour fields written on every video frame before it is handed out: color_trc, color_primaries, colorspace, color_range
  • subscribed_outputs (object, optional) - output name => frame subscription name: that output receives frames while its subscription is enabled, whatever the mask says, and never waits

one_to_many runs on a thread of its own, woken by every frame. one_to_many_nb runs on an event loop (event_loop, as realtime does) when its input signals a frame, so a graph with many of them has no thread per fan-out; it cannot wait for room in an output, so it requires drop: true.

join_metadata

Join metadata from an auxiliary stream into a primary stream by exact timestamp match.

2 inputs, 1 output: av::VideoFrame or av::AudioSamples

  • src must contain exactly 2 queues in this order: [primary, auxiliary]
  • timestamps are compared exactly (primary.pts == auxiliary.pts)
  • empty queue is treated as "not ready yet" (the node waits), not as missing frame
  • when both heads are present:
    • if timestamps match: copy auxiliary metadata to primary (av_dict_copy) and emit primary
    • if primary.pts < auxiliary.pts: emit primary unchanged (auxiliary missing for that primary timestamp)
    • if auxiliary.pts < primary.pts: drop auxiliary frame (primary missing for that auxiliary timestamp)
  • output timestamp/timebase remains the same as primary input

Useful for running heavy processing (e.g. neural network inference) on downscaled frames and merge produced metadata back onto original-resolution frames.

crop_metadata_cuda

Metadata-driven CUDA crop. Reads a bounding-box from per-frame metadata (e.g. produced by a detection node) and crops the frame to that region using an FFmpeg CUDA filter graph. Intended for reframing workflows such as face-detection crop.

Requires HAVE_CUDA=1.

1 input, 1 output: av::VideoFrame (CUDA)

  • metadata_key (string, default "reframer_bbox") - frame metadata key containing the crop coordinates. Expected value format: {"x": 100, "y": 50, "w": 640, "h": 360} (pixel coordinates in input frame space).
  • dst_width (int) - fallback output width used when no valid metadata is present on a frame
  • dst_height (int) - fallback output height used when no valid metadata is present on a frame
  • offset_log_path (string, default "reframer.log") - path to write per-frame crop offset log (for debugging)
  • debug_log_every_n (int, default 0) - if nonzero, log crop parameters every N frames to the avplumber log

force_keyframe

Set keyframe flag in frame to make encoder output keyframe. Unlike -g encoder option in FFmpeg, works with non-integer FPS.

1 input, 1 output: av::VideoFrame

  • interval_sec (int / float / string of rational) - optional, keyframe interval, in seconds. If omitted, periodic forcing is disabled and the node acts as a pass-through until triggered externally.
  • min_interval_ms (non-negative int, default 0) - minimum media-PTS spacing between forced keyframes, shared by periodic and external requests. 0 preserves unrestricted forcing. Requires valid PTS when enabled; a backwards jump before the last forced PTS resets the limiter. The Janus mixer output defaults to 150 (9 frames at 60 fps), configurable with --keyframe-min-interval-ms or JanusVideoConfig.keyframe_min_interval_ms.

While rate-limited, ordinary frames pass through without waiting. Requests coalesce and remain pending until the first eligible frame, including requests from RTCP PLI/FIR; these may wait up to the minimum interval plus frame rounding. Periodic forcing cannot bypass the limit. This limits the node's forced frames, not keyframes independently inserted by an encoder, and is not a wall-clock RTP packet pacer.

Runtime control (via node.object.set / node.object.get):

  • node.object.set <name> trigger true — request one keyframe on the next eligible frame (one-shot, edge-triggered). Also accepted as key force or request.
  • node.object.get <name> status — returns a JSON object: 
    {
  "requested_generation": 3,
  "forced_generation": 3,
  "pending": false,
  "triggered_frames": 2,
  "periodic_frames": 60,
  "interval_enabled": true,
  "min_interval_ms": 150
}
  • node.object.get <name> pending — bool, true if a trigger has been requested but not yet applied to a frame.

enc_video, enc_audio

Encodes video or audio frames.

1 input: av::VideoFrame or av::AudioSamples, 1 output: av::Packet

  • codec (string) - mandatory
  • options (dictionary) - options passed to libavcodec
  • hwaccel (string, name of instance-shared object) - optional (mandatory for some encoders), name of hwaccel previously created with hwaccel.init
  • hdr_metadata (object, enc_video only; enc_audio rejects it) - static HDR metadata serialised verbatim as side data before the encoder opens: primaries (three [x, y] pairs), white_point ([x, y]), max_luminance / min_luminance and max_cll / max_fall in nits. hevc_nvenc and av1_nvenc emit the HDR10 SEIs from it when FFmpeg is built against nv-codec-headers 13 (driver >= 570); older headers silently emit none. The mixer fills it for PQ renditions (BT.2020 / D65) from pyplumber.mixer.color.hdr_metadata.
  • timestamps_passthrough (bool) - default false, intended for codecs that don't buffer data (otherwise bad things like repeated timestamps may happen), replace PTS & DTS in outgoing packet with incoming PTS

packet_relay

Insert it between demuxer and muxer to remux packets without transcoding.

1 input, 1 output: av::Packet

no parameters

bsf

BitStream filter

1 input, 1 output: av::Packet

  • bsf (string) - name of bsf to use

mux

multiple inputs, 1 output: av::Packet

  • fix_timestamps (bool) - shift PTSes and DTSes so that DTSes are always increasing and (PTS >= DTS). Disabled by default.
  • ts_sort_wait (float, seconds) - default 2.5, maximum time to wait for all streams to select the packet with least DTS. Set to 0 to emit packets as soon as they arrive.
  • stream_ids (list of positive integers) - if specified, will set custom stream ids in output container. Useful for mpegts format output.
  • metadata (list of dictionaries) - if specified, will set metadata in output container's streams.

output

1 input: av::Packet

  • format (string) - mandatory
  • url (string of URL) - mandatory
  • options (dictionary) - format options that will be passed to libavformat
  • seek_table (string of file name) - if specified, binary file with native endianness with seek table will be written.
  • seek_table_text (string of file name) - if specified, text file with seek table will be written.
  • hold (boolean, optional, default false) - start held: packets are taken from the input and not written, so everything before the output runs (an encoder opens on its first frame) while nothing leaves. node.object.set <name> hold false releases it; writing then starts at the next key packet, so ask the encoder for one. hold true holds it again.

Format of binary seek table: 16-byte records containing:

  • timestamp of the frame in milliseconds: int64_t
  • bytes offset in the output file: uint64_t

Format of text seek table: values as above separated by space, each record in one line

Pending seek-table entries are flushed and the canonical seek-table links are published when a finite input reaches EOF, including a final partial batch.

jack_sink

1 input: av::AudioSamples (sample format must be fltp, sample rate must be equal to JACK's)

Output audio to JACK Audio Connection Kit server. Can be used for both inter-app routing and outputting audio to the sound card. Note that JACK has its own clock (soundcard clock or in case of dummy device - OS monotonic clock) so in long running streams underruns or overruns may occur, unless the stream's clock and JACK clock are synchronized (for example, input is AES67 RTP and the hardware audio interface is synchronized to AES67 master clock using wordclock or S/PDIF).

Both sample format and sample rate must match configuration of the JACK server (sample format is always fltp in JACK). Use resample_audio node to convert.

This node has an internal buffer to ensure that JACK thread can run in real time. In case of bursty streams (e.g. coming from the Internet) this buffering may be insufficinent. The realtime node with a small negative_time_tolerance (below JACK period size) will help in such cases.

  • channels_count (int, default 2) - number of JACK ports to create. If incoming audio stream has less channels, the remaining ones will be filled with silence. If more, the excessive channels will be discarded.
  • port_prefix (string, default empty string) - if specified, will append channel index to this string and it'll become the JACK port name. If unspecified, JACK port name will be only the channel index.
  • connect_port_prefix (string) - if specified, will append port number to this string and try to connect to an input port with that name in the JACK graph
  • client_name (string) - mandatory, name of the JACK client. If multiple jack_sink nodes are created with the same client name, ports will belong to the same JACK client (make sure to set port_prefix in such cases). If they have different client names, multiple JACK clients will be created. What is better depends on the use case - multiple JACK clients allow parallel processing but put more overhead on the CPU.

raw_output

Write stream of raw packets or frames to file or pipe. Unlike output node, can be used anywhere in graph. Outputs whatever will be thrown on it. Use nodes rescale_video or resample_audio to convert to required format.

1 input: anything

  • path (string) - file/pipe name, not URL, must be reachable via Unix file structure, libavformat protocols like tcp:// aren't supported
  • output_group (string) - default "default". Name of output group for control (see output.start and output.stop commands) and synchronization. Since raw output doesn't have PTSes, avplumber will try to synchronize audio with video by cutting first audio frame to make it start together with first video frame.

reinterpret_planes_video / reinterpret_planes_audio

Reinterpret plane pointers (like reinterpret_cast) without copying data. Works like a controlled cast of frame layout:

  • For video, re-map data planes and change destination pixel format metadata.
  • For audio (not tested yet), re-map planar channels and change destination sample format metadata.

No memory copies are performed. Plane pointers and buffer references are reused. Dimensions and linesizes are validated using av_pix_fmt_ functions.

Intended as a more versatile alternative to shuffleplanes which does not support hardware frames.

1 input, 1 output: av::VideoFrame or av::AudioSamples respectively

  • plane_map (object) - optional, { "dst_plane": src_plane } mapping. If omitted, planes are mapped 1:1 up to the minimum plane count (e.g. yuva420p -> yuv420p drops alpha by default).

For video only:

  • dst_pixel_format (string, required) - destination software pixel format name (e.g. gray, yuv420p). When using hardware frames, this updates only hw_frames_ctx->sw_format; AVFrame::format (the hardware pixel format, e.g. cuda) stays unchanged.
  • hw_frames (bool, video) - default false. If true, the node expects a hardware input frame and keeps the hardware AVFrame::format unchanged. It clones hw_frames_ctx and sets its sw_format to dst_pixel_format.

For audio only:

  • dst_sample_format (string, required) - destination sample format name (e.g. fltp). Interleaved audio is supported only when source and destination sample formats match exactly (no copying is performed).

Constraints:

  • No conversions are performed. The node does not copy, up/download, or resample; it only reinterprets pointers and metadata.
  • For video, mapping between hardware and non-hardware must be consistent: hw_frames=true is required for hardware inputs; hw_frames=false rejects hardware inputs. Otherwise pointers would become invalid.
  • For audio, planar-to-planar remaps are supported; interleaved requires identical formats and is effectively pass-through.

Examples:

Map V component of yuv420p to grayscale:

{"type":"reinterpret_planes_video","name":"v_to_gray","group":"proc",
 "src":"v_in","dst":"v_out",
 "dst_pixel_format":"gray",
 "plane_map":{"0":2}}

Drop alpha from yuva420p to yuv420p (default plane map) in hardware frames:

{"type":"reinterpret_planes_video","name":"drop_alpha","group":"proc",
 "src":"v_in","dst":"v_out",
 "dst_pixel_format":"yuv420p",
 "hw_frames":true}

Audio: select channel 1 (second plane) as mono FLT planar:

{"type":"reinterpret_planes_audio","name":"pick_ch1","group":"proc",
 "src":"a_in","dst":"a_out",
 "dst_sample_format":"fltp",
 "plane_map":{"0":1}}

picture_buffer_sink

Take a frame and write it to picture buffer that can be later used by sentinel_video.

1 input: av::VideoFrame

  • buffer (string, name of instance-shared object) - mandatory, picture buffer name
  • once (bool) - default true, finish after getting a single frame

null_sink

Discards incoming packets just like /dev/null.

1 input: anything

no parameters

write_audio_envelope

Writes audio envelope data for waveform display (no images; data only). One binary file per granularity level (like mipmaps for different zoom), plus an index.json describing layout. Files are appended incrementally; recording length need not be known in advance. To align envelope samples to video frames, use the video frame duration as one of the granularities (e.g. "1/25" for 25 fps).

1 input: av::AudioSamples

  • path (string) - directory (or base path) for output files; must be a filesystem path
  • granularities (list of strings) - segment duration per envelope sample, as rationals in seconds (e.g. ["1/25", "1/2", "1"]). One file per entry

The node writes index.json (version, sample_rate, channels, levels map keyed by duration_sec with file name per level, metrics map keyed by id with offset_bytes and stride_bytes).

Binary layout

Each sample is stored as interleaved records with channel data, currently containing: positive peak, negative peak, RMS. To read one metric across channels, use offset_bytes and stride_bytes from index.json. For forward compatibility, do not assume that each channel's record will always contain 3 bytes.

Example for 2 channels (one envelope sample = currently 6 bytes):

  byte index:   0     1     2   |  3     4     5   |  6   ...
                ^     ^     ^   |  ^     ^     ^   |  ^
  metric:      pos   neg   rms  | pos   neg   rms  | pos
  channel:     ch0   ch0   ch0  | ch1   ch1   ch1  | ch0
  time range:  \________________________________/    \____...
                               0..1                     1...   (unit: segment duration)

  positive_peak: offset_bytes=0, stride_bytes=3  -->  read at 0, 3
  negative_peak: offset_bytes=1, stride_bytes=3  -->  read at 1, 4
  rms:           offset_bytes=2, stride_bytes=3  -->  read at 2, 5

Each envelope sample is 1 byte per value, 0.5 dB resolution, -127..0 dB range: 0 = -127dB or less, 254 = -0.5..=0dB, 255 = clip

Example index.json

{
  "channels": 2,
  "levels": {
    "1": { "file": "level_2.bin" },
    "1/60": { "file": "level_0.bin" },
    "1/8": { "file": "level_1.bin" }
  },
  "metrics": {
    "negative_peak": { "offset_bytes": 1, "stride_bytes": 3 },
    "positive_peak": { "offset_bytes": 0, "stride_bytes": 3 },
    "rms": { "offset_bytes": 2, "stride_bytes": 3 }
  },
  "sample_rate": 48000,
  "version": 1
}

Rendering envelope image recommendations

The renderer needs to:

  1. read index.json
  2. open binary file corresponding to the needed time resolution (recommended: greater or equal to waveform image resolution), specified in levels map
  3. for each needed metric, initially jump to the offset_bytes and then jump by stride_bytes to read subsequent interleaved samples (so the actual stride for a single channel is stride_bytes*channels)
  4. if waveform is to be rectified, compute max(positive_peak, negative_peak)
  5. if mono waveform is to be displayed, compute maximum (for peaks) or RMS (for RMS) of all channels
  6. if the read time resolution was greater than resolution of the waveform to be displayed, compute maximum or RMS of waveform values that correspond to the same “pixel” on the resulting image

ipc_cuda_source

Get video frames from CUDA IPC memory. Frame pointer and parameters are read from named pipe. See src/nodes/cuda/ipc_cuda_source.cpp for structure.

1 output: av::VideoFrame (frame's pixel format will always be cuda)

  • pipe (string) - mandatory, path to named pipe
  • hwaccel (string, name of instance-shared object) - mandatory, name of hwaccel previously created with hwaccel.init

raw_video_file_cuda

Play a headerless raw NV12 or P010 file as CUDA frames. Each picture is read from the file straight into page-locked host memory and the device takes it from there, so the samples are copied once on the CPU (page cache to that memory). The chain input_rec (rawvideo) → demux → dec_video → filter_video (hwupload_cuda=pinned=1) copies them twice and runs four threads; this node runs one. Nothing is kept between passes: every picture shown is read from the file and transferred for that showing.

1 output: av::VideoFrame (hardware "pixel format" cuda). The node blocks while its output queue is full, so the consumer (typically realtime) paces the reading; give the queue a capacity of 1 to hold one uploaded picture ahead. Timestamps count pictures in a time base of one frame period.

Parameters:

  • url (string, required) - path of the file. A trailing part of a picture is not shown.
  • pixel_format (string, required) - nv12 or p010le; rows carry no padding.
  • video_size (string, required) - WIDTHxHEIGHT
  • framerate (string, required) - rational, e.g. 30000/1001
  • hwaccel (string, required) - CUDA device created with hwaccel.init
  • loop (bool, default false) - start again after the last picture. Without it the node sends the end-of-file marker (unless send_eof is false) and finishes, which auto_restart acts on.
  • loop_continuous_ts (bool, default false) - timestamps count on across passes instead of starting from zero at each one
  • frame_color (object, optional) - colour fields written on every frame, as for one_to_many: color_trc, color_primaries, colorspace, color_range
  • write_combined (bool, default false) - allocate the page-locked memory write-combined. Whether that makes the read cheaper depends on the host; measure before using it.
  • pool_frames (int, default 0: no bound) - at most this many device frames. FFmpeg's pool grows to the most frames ever out at once and keeps them; bounded, the node waits (up to five seconds, as it does for device memory) until one comes back. stats reports frames_allocated.
  • resident (bool, default false) - with loop: keep the file's pictures in the page-locked memory. Each is read when it is first shown and only transferred after that, so a pass after the first reads nothing. Where a pass starts the node compares the file's modification time and reads the pictures again if it has changed. The memory is the size of the file and cannot be swapped out; the node fails to start if it cannot be had.

The read is a blocking local pread; there is no read timeout, and a file that ends inside a picture after it was opened stops the node with an error. node.object.get <node> stats returns frames, passes, the number of pictures read from the file (file_reads) and the duration of the last read and upload (read_us, upload_us).

The upload half is src/nodes/hwaccel/CudaPinnedUploader.hpp, which any source able to write its samples where it is told can use.

ipc_audio_source

Get audio frames from named pipe. See src/nodes/ipc_audio_source.cpp for header structure. Header must be followed by interleaved audio samples.

1 output: av::AudioSamples

  • pipe (string) - mandatory, path to named pipe

extract_cc_data

1 input: av::VideoFrame, 1 output: av::Packet

Extract ATSC A53 Part 4 Closed Captions data from video frame. Subtitle codec is usually EIA-708 or 608 in such side data. When outputted to UDP with libavformat's special data 'muxer' (see examples/extract_cc_data.avplumber), subtitles can be parsed using CCExtractor or GStreamer (YMMV).

no parameters

firewall

Drop data with invalid timestamps; pass through otherwise. Useful to prevent malformed inputs from propagating downstream.

1 input, 1 output: anything

no parameters

ipc_dmabuf_source

Receive GPU frames via a UNIX domain socket with FD passing (DMA-BUF). Produces DRM PRIME frames with metadata taken from the sender.

Frame references retain their release-ACK connection through node shutdown or reconnection. The matching dma-browser sender stops deliveries on Drain and reclaims outstanding frames on Drained, sent after the last reference is freed. An unclean disconnect quarantines unacknowledged browser textures; see DMA-BUF recovery and counters.

Each frame is a 48-byte TexInfo header with one DMA-BUF fd sent via SCM_RIGHTS. Stock Electron waits for its GPU work before delivering the paint event.

1 output: av::VideoFrame (hardware "pixel format" DRM_PRIME)

Parameters:

  • socket (string, required) - path to the UNIX domain socket
  • hwaccel (string, optional) - name of hwaccel object; when set, a matching hw_frames_ctx is attached for downstream filters/encoders
  • fps (rational string, optional) - publishes the sender's requested frame rate to downstream nodes; frame timestamps use a 1/1000000 time base
  • return_queue (integer 0 to 64, optional, default 0) - frames that wait to go back to the sender. 0 sends each release ACK when the graph lets its frame go. With N, a finished frame waits in the return queue and one goes back per frame that arrives; only beyond N waiting does the oldest go at once. Chromium's capture pool keeps two idle buffers and destroys the rest, so a consumer that lets frames go in bunches makes it destroy and allocate buffers all the time, and every new allocation is a new import (and a released one) downstream. One buffer back per buffer handed out keeps the pool on a fixed set. Queued frames count against the sender's ringSize: open the window with the frames in flight plus N.

node.object.get <node> frame_stats reports the received-frame count, plus released frames, sent ACK messages, pending ACK messages and frames in the return queue (returns_queued) on the current connection. ACK counters reset on reconnection; the received count does not. Lifecycle drain markers also count as sent messages. Pending ACKs distinguish frames already released by the graph from frames still referenced downstream.

ipc_socket_audio_source

Receive audio frames over a UNIX domain socket. Expects a simple header followed by interleaved float32 PCM.

1 output: av::AudioSamples

Parameters:

  • socket (string, required) - path to the UNIX domain socket
  • sample_rate (int, default 48000)
  • channels (int, default 2)
  • reconnect_delay_ms (int, default 50)

cuda_to_egl_image

Convert CUDA av::VideoFrame to EglImageFrame (RGBA) using CUDA kernels and an EGL-backed texture pool. Intended for zero-copy rendering paths (e.g. OBS).

1 input: av::VideoFrame (pixel format cuda), 1 output: EglImageFrame

Parameters:

  • pool_id (name of instance-shared object, default "default") - shared EGL image pool id
  • pool_size (int, default 8) - pool capacity

drm_prime_to_cuda

Import DRM PRIME frames into CUDA frames via EGL/GL interop, without copying: the output frame is the mapped buffer and keeps the input frame, and with it the producer's buffer, until the last consumer releases it. Non-DRM PRIME frames are passed through unchanged. A tiled (array) import is a texture-backed frame that only the CUDA compositor's RGB-to-YUV path samples; the compositor rejects other combinations before submitting draw work.

1 input: av::VideoFrame (expects DRM_PRIME hardware "pixel format", pass-through otherwise), 1 output: av::VideoFrame (hardware "pixel format" cuda)

Parameters:

  • hwaccel (string, required) - CUDA device created with hwaccel.init

Retired imports are released on a cleanup worker shared by the instance. Incoming frames needing a new import are dropped while cleanup is pending or their import budget is full; cached imports continue normally. Dropped inputs release their browser capture slots.

node.object.set <node> purge_idle <ms> releases, within a quarter of a second, every import that no frame holds and none has used for that many milliseconds, without waiting for import_ttl_ms or for the next frame: for a caller that knows a burst of allocations is over, such as a show that has finished starting. They count as expired.

node.object.get <node> import_stats includes an admission_dropped frame counter, a texture_failed frame counter (an array import whose texture object could not be created; the frame is dropped) and instance-wide cleanup counts. These counters distinguish admission drops from EGL/CUDA import work without per-frame logs.

object_tracker

Apply ByteTrack to object detections already attached to a video frame. The node is model- and domain-neutral: by default it tracks every detection in yolo_detections.

1 input: av::VideoFrame, 1 output: the same frame with track annotations added

Parameters:

  • metadata_key (string, optional, default yolo_detections) - detection payload to update
  • target_labels (array of strings, optional) - track only matching labels
  • target_class (int, optional) - track only this class ID
  • min_conf (number, optional, default 0.01) - minimum detection confidence
  • frame_rate, track_buffer, track_thresh, high_thresh, match_thresh, low_match_thresh - ByteTrack tuning
  • camera_shot_metadata_key (string, optional) - reset tracks on shot transitions
  • predict_on_empty, emit_lost_tracks (bool, optional) - lost-track output behavior

draw_bbox, draw_bbox_labels, draw_trail, draw_keypoints, draw_segmask

Draw detection results that are already attached to a video frame as JSON metadata: boxes, labels of 5x7 bitmap text (A-Z, 0-9, : _ - .), a trail of line segments, pose keypoints, segmentation masks. Built with HAVE_CUDA=1 and NEURAL_NET=1. The box and label parameter tables are in draw_bbox and draw_bbox_labels.

1 input: av::VideoFrame, hardware "pixel format" cuda or cuarray, software format nv12 (anything else is an error), 1 output: av::VideoFrame, always cuda, with the input's timestamps, metadata and side data

The node never draws on a frame another reader may hold. Which frame it draws on depends on its input:

  • cuda from the picture pool of a draw node above, when no other frame references it: the input frame itself, without a copy. This is checked for every frame. A split, a wiretap or a node that keeps frames between two draw nodes leaves a second reference, and the node then copies, into a frame of the pool the picture came from.
  • any other cuda frame: a copy in a frame of a pool the node owns. The pool has the device, the software format and the width and height of the input's frames context, so the copy has the layout and the size a frame of the input's own pool has (the size of the frames context, which for a decoder can be the coded size, 1088 lines for 1080p H.264). Both planes are copied with one call when the two frames hold them in a single buffer at one pitch, otherwise with one call each.
  • cuarray (an NVDEC surface, which is also the decoder's reference picture): a linear picture of it at the frame's size, read once through the rect compositor's array reader (cuda_rect_draw, so it needs HAVE_NVCC=1) into a frame of the same pool. The stream is synchronized and the surface released before the frame is put, also when the draw fails, so a draw node holds one decoder surface while it processes a frame and none while it waits.

So the first draw node of a chain makes the picture and the nodes below draw on it: one picture per frame and chain, whatever the input storage. This is the usual case, not a guarantee: a node releases its own reference to a frame only after it has put it, and a node below that takes the frame before that sees two references and copies. The pictures object counts both.

The output frames of a chain belong to the frames context of its first node's pool, not to the one the input came from. It is recreated when the input's device, width or height changes and when the node is restarted. A consumer that binds to one frames context, such as an hwdownload filter, has to be set up again then, as it has to when the context of its input changes.

All nodes of a chain run on the CUDA stream of the frame's device, for a CUarray decoder its own stream; the nodes take no hwaccel parameter. A node synchronizes that stream after its kernels. A node with nothing to draw on a linear frame launches nothing and does not synchronize: a copy it made is still queued on that stream when the frame is put.

node.object.get <node> pictures returns how many frames the node drew on a picture made of a CUarray (from_array), on a copy (copied) and on its input frame (in_place).

Coordinates: detections[].xyxy is in model pixels unless the payload has a coord_space other than model, in which case it is in pixels of the frame the node receives. Model pixels are mapped with model_content_width, model_content_height, model_content_offset_x, model_content_offset_y when set (a letterboxed model input), otherwise scaled by the frame size over the payload's model_width, model_height.

Parameters common to all:

  • metadata_key (string) - the frame metadata entry to draw; default reframer_bbox (draw_bbox), yolo_players (draw_bbox_labels), yolo_detections (draw_trail, draw_segmask), yolo_pose (draw_keypoints). draw_bbox and draw_bbox_labels also take metadata_keys (array).
  • model_content_width, model_content_height, model_content_offset_x, model_content_offset_y (number, default 0: unused)
  • debug_log_every_n (int, default 0)

ml_debug

Everything a model says about a frame, drawn in one pass: detection boxes, their labels, pose or face keypoints and a trail. layers (array, required) are painted in the order given. A layer has a kind and the parameters of the draw node of that kind, which reads the frame's metadata for it:

kind Reads and takes the parameters of
boxes draw_bbox
labels draw_bbox_labels
keypoints draw_keypoints
trail draw_trail 
{"type": "ml_debug", "src": "detected", "dst": "drawn", "layers": [
  {"kind": "boxes", "metadata_key": "yolo_players", "label_colors": {"Player": "green"}},
  {"kind": "labels", "metadata_key": "yolo_players", "label_template": "ID:{track_id}", "font_scale": 1},
  {"kind": "keypoints", "metadata_key": "yolo_pose", "radius": 3, "color": "green"},
  {"kind": "trail", "metadata_key": "ball_trail", "thickness": 1, "color": "light_blue"}
]}

The frame comes out as it would from a chain of those draw nodes in the same order, pixel for pixel, and the node handles frames as they do (input and output formats, the picture it draws on, the pictures object). What differs is the work: a metadata key that several layers read is parsed once; the items are sorted on the CPU into the 16x16-pixel tiles they touch, and one kernel launch visits only those tiles, each thread only the items of its tile; the stream is synchronized once. A draw node launches a thread for every pixel of the frame and every thread tests every item, so 118 face landmarks on a 1920x1080 frame are 306 million tests there and some tens of thousands here.

node.object.get <node> draw returns the items and tiles of the last frame and the frames that had anything drawn on them.

tests/cuda/nvdec/draw_arrays.py runs chains on linear and on CUarray decode on an NVIDIA GPU: an undrawn chain against the plain decode, three draw_bbox against one, one ml_debug against the chain of four draw nodes and against three draw_bbox, CUarray against linear, the pictures counters, and with --expect the linear cases against a report of another build. tests/test_draw_picture_frame.py checks without a GPU which frames count as private, which pool a copy comes from, when one call may copy both planes, and that mask side data keeps its buffer.

drm_prime_to_egl_image

Import DRM PRIME (DMA-BUF) frames into an EGLImageKHR via EGL_EXT_image_dma_buf_import and output them as EglImageFrame (no CUDA processing).

1 input: av::VideoFrame (expects DRM_PRIME hardware "pixel format"; non-DRM PRIME frames are dropped), 1 output: EglImageFrame

Supported DRM formats (layer0/plane0 only):

  • DRM_FORMAT_ABGR8888
  • DRM_FORMAT_ARGB8888

Cache behavior:

  • Identifies a physical allocation by st_dev, st_ino, dimensions, DRM fourcc, modifier, plane offset, and plane pitch. FD integers are not identity because they can be reused.
  • Keeps a duplicated DMA-BUF FD alive with every cached EGLImageKHR.
  • Reuses the same immutable EGLImageKHR when that allocation returns. The producer/consumer protocol must retain each delivered frame until downstream GPU reads finish; ipc_dmabuf_source provides that ordering through release acknowledgements.
  • Evicts idle entries after ttl and the least-recently-used entry at the configured limit.

Parameters:

  • cache_mode (string, optional, default "reuse") - reuse or off
  • ttl (float seconds, optional, default 3.0) - cache entry time-to-live
  • max_cache_entries (integer, optional, default 64) - maximum retained imports
  • debug_log_every_n (integer, optional, default 0) - periodically log aggregate cache counters

egl_image_cuda_overlay

Scale and compose multiple cached EglImageFrame inputs directly into one pitched CUDA RGB0 frame (or RGBA with output_alpha). Each stable input image is registered once with cuGraphicsEGLRegisterImage; its CUeglFrame and CUDA texture object are then kept for that physical allocation. CUDA EGL image resources do not require per-frame map/unmap. Every output tick samples the cached texture objects with bilinear scaling and writes the configured rectangles directly. There are no per-input CUDA frames, scale filters, or pixel copies.

The node owns its output clock. It keeps the latest frame independently for each input, so a stalled input repeats its last image without blocking other inputs or the program output. The frame lifetime holders are released only after a CUDA event confirms that the sampling kernels have completed.

N inputs: EglImageFrame, 1 output: CUDA av::VideoFrame with RGB0 or RGBA software format

Parameters:

  • src (array of edge names, required) - input EGL image edges
  • dst (string, required) - output CUDA video edge
  • hwaccel (string, required) - CUDA device created with hwaccel.init
  • width, height (integers, required) - output canvas dimensions
  • layers (array, required, one per input) - objects containing dst_x, dst_y, dst_w, and dst_h
  • fps (ratio string, optional, default "60/1") - independent compositor output rate
  • output_alpha (boolean, optional, default false) - advertise RGBA output, clear uncovered pixels to transparent black, and copy each layer's source alpha when its EGL format exposes one (otherwise opaque). The default keeps the existing opaque RGB0 output for encoder compatibility. Layers overwrite earlier rectangles; this option preserves alpha for downstream composition, it does not add source-over blending between overlapping layers.
  • cache_ttl (float seconds, optional, default 3.0) - idle interop-slot lifetime
  • max_cache_entries (integer, optional, default 440) - maximum registered slots
  • debug_log_every_n (integer, optional, default 0) - periodically log aggregate compositor counters

tests/cuda/test_egl_overlay_alpha.cu checks the production kernels on an NVIDIA GPU, including opaque compatibility, alpha lane layouts, transparent uncovered pixels, bilinear alpha sampling, and pitched output. It does not replace an end-to-end DMA-BUF/browser interop test.

cuda_rect_overlay

Compose CUDA frames into one canvas: each layer scales one input's frame, or a crop of it, into a rectangle; the background and every layer are drawn in one kernel launch. The node has no clock: each output frame draws the active inputs' frames of one timestamp, matched across inputs, and an input without a frame at that timestamp repeats its previous one. Upstream owns pacing. The mixer's clocked compositor is mixer_compositor, its downstream keyer mixer_keyer.

N inputs: CUDA av::VideoFrame (src, at most 193), 1 output: CUDA av::VideoFrame with software format sw_format

Parameters:

  • hwaccel (string, required) - CUDA device created with hwaccel.init
  • width, height (int, required) - canvas size
  • sw_format (string, default nv12) - canvas storage: semiplanar YUV (nv12, p010le, p210le) or packed 8-bit RGB (rgb0, bgr0, rgba, bgra). A semiplanar YUV input of another depth or chroma subsampling is converted as it is drawn: 8-bit codes are multiplied up to a 10-bit canvas, 10-bit codes divided down to an 8-bit one and rounded to nearest (min(255, (v + 2) >> 2)), without dithering. That is a depth change only, not tone mapping, so a deeper input tagged PQ or HLG is an error on an 8-bit canvas (the text carries hdr_source_transfer=pq or =hlg): tone map it before the node. With color set it is refused like any input whose colour tags differ from the canvas's.
  • layers (array of objects, required) - at most max_layers; each draws input (an index in src; omitted, the layer's position) at dst_x, dst_y, dst_w, dst_h (size omitted: the source's), optionally from crop ({"x", "y", "w", "h"}), with fit stretch (default) or contain (source_canvas {"w", "h"} letterboxes into a virtual source canvas), in z order (lower first), blend honouring the source's alpha. filter resamples a YUV source on a YUV canvas with bilinear (default), bicubic (4x4 cubic, no prefilter when shrinking), multisample (the mean of samples bilinear samples, 4 (default) or 8, spread over the source area of one output pixel) or auto: multisample when the layer shrinks by more than multisample_above (default 2.0), bicubic when it enlarges by more than bicubic_above (default 1.3), else bilinear; the axis with the larger factor decides, shrinking first. bicubic_param (default 0) is the cubic's coefficient as FFmpeg scale_cuda's param: 0 is that filter's default bicubic, which weighs only the two inner taps per axis (Hermite), 0.5 is Catmull-Rom. Packed RGB sources and canvases are always bilinear. Per-frame metadata keeps a layer's filter settings unless it names them. The format is parsed in src/mixer/primitives/compositor_layers.hpp.
  • max_layers (int, default 256) - layers per frame; sizes the GPU rect table
  • active_inputs (mask, default all) - inputs drawn: a JSON number, or a least-significant-bit-first bit string ("1011") beyond 64 inputs
  • metadata_key (string, default rect_overlay_v1) - frame metadata key with per-frame layer changes, {"layers": [...]} in src order or {"<index>": {...}}, read from the frame the output copies its properties from
  • color (sdr, hlg or pq, optional) - canvas colour contract: inputs' colour tags are checked against it and the output carries it; HDR needs 10-bit storage. Unset, the output takes its tags from that metadata frame.
  • sdr_white, hdr_peak (float nits, default 203, 1000) - graphics white and peak with color
  • warmup_timeout_ms (int, default 0: no bound) - how long to wait for every active input's first frame before drawing without the missing ones
  • debug_log_every_n (int, default 0) - log every n-th output frame

node.object.set <node> active_inputs <mask> and node.object.set <node> layers [...] change the mask and the layers. The parameters fps, aux_mode, clock_input, subscriptions and pgm_delay_frames fail, as do the objects prewarm_inputs, warm_reset, composition and fade_inputs: they belong to mixer_compositor and mixer_keyer.

mixer_compositor

The mixer's clocked compositor: its scene slots, the wipe overlay and the AUX buses. Parameters as for cuda_rect_overlay, but the node owns its output clock: inputs are live with monotonic timestamps, and for every tick of fps a shared playout (src/mixer/Playout.hpp) picks each input's frame stamped nearest that tick, drawn latency_ms after it. A late input repeats its previous frame rather than delaying the output. See mixer.md.

Parameters, in addition:

  • fps (ratio string, required) - output rate
  • latency_ms (float, default two frames, at most six) - how long after a tick its frame is drawn: the inputs' arrival budget
  • pgm_delay_frames (int, optional) - match the last input that many ticks back (a Program preview's finished program arrives one frame after its sources)
  • input_poll_ms (int, default 0) - 0: the output thread is woken by every frame an input gives. Above 0: it sleeps until the next deadline or a control change and looks for frames at most this often in between. A tick is drawn at its deadline in both cases; the second spares a compositor with many inputs thousands of wake-ups a second. Each input's queue must hold what arrives within one interval. node.object.get <node> status reports the setting and input_passes, how often the thread has looked.
  • aux_mode (bool, default false) - an AUX bus: a private CUDA stream, subscribed inputs, layers from composition only (no per-frame metadata layers), and frames dropped rather than waited for when the output edge is full
  • output_hwaccel (string, required with aux_mode) - name the bus's device with that stream is published under, for its conversion and encoder
  • subscriptions (array of strings, required with aux_mode) - one shared frame subscription per input, in src order; sources never wait for the bus
  • mixer (string, optional with aux_mode) - the mixer whose scene definitions are frozen from then on

clock_input fails: it is mixer_keyer's.

Objects (node.object.set):

  • active_inputs, layers - as for cuda_rect_overlay
  • prewarm_inputs (mask) - inputs whose frames advance through the playout without being drawn, so a switch to them is immediate
  • warm_reset - reject frames stamped over a tick before the one being drawn; prewarmed inputs keep their held pictures
  • composition (aux_mode only) - {"layers": [...], "active_inputs": <mask>, "revision": "<string>"}: applied on the tick its new inputs are ready; if they are not ready within max(250 ms, twice latency_ms), the previous layout stays and composition_error says so. revision (optional) names it in the status once it is drawn.

node.object.get <node> status: output_drops (frames dropped for a full output edge; the bus keeps playing and draws again once it has room), playout (frames, repeats, discarded, overflow, missed_deadlines: totals over the inputs; node.object.get <node> per_input gives repeats, discarded and overflow of each input, for finding the one that misbehaves), with aux_mode composition_pending, composition_error and composition_revision, the revision of the composition being drawn (set when one is applied at once or a staged one becomes ready, unchanged when a staged one is dropped; empty for one without a revision).

node.object.set <node> reset_counters true starts missed_deadlines, the per-input counters, output_drops and playout.timing again from zero, for a caller that has finished starting the show; frames keeps counting and the playout is not disturbed.

playout.timing says where each drawn tick's time went: period_ms, bounds_ms (the upper bounds of all buckets but the last: a quarter, a half, three quarters, one, one and a quarter, one and a half, two and three frame periods) and four histograms, each with counts per bucket, count, mean_ms and max_ms since the node started: draw (the whole draw), issue (queueing it: the node's own work and the CUDA library's locking), gpu_wait (waiting for the GPU to finish it and the frames it reads) and downstream (waiting for room on the output edge). A draw in the first four buckets fitted its slot.

mixer_keyer

The mixer's downstream keyer (DSK): blended key layers over the program. Parameters as for cuda_rect_overlay; the program input drives the output. Each program frame is drawn at once over every key above fade level 0, with each key's newest frame stamped at or before that program tick. Keys are never waited for; with no key visible the program frame passes through untouched.

With keys visible the whole canvas is composed in one pass. Keys named in bounded_keys may instead be composed alone: when every visible key is one of them, the output is a copy of the program frame on which only the rectangles those keys cover are composed, so a key on a third of the picture costs a third of the compositing. Keys that overlap share a rectangle. This needs the program layer to be the program frame unchanged (the canvas's format and size, drawn whole and unscaled) and the keys to lie in at most four separate rectangles covering at most 60 % of the canvas; otherwise the whole canvas is composed. The picture is the same either way.

Parameters, in addition:

  • clock_input (int, required) - the program's index in src
  • bounded_keys (array of ints or bool, default none) - the inputs of the keys whose rectangles may be composed alone; true names every key
  • fps (ratio string, required) - the program's rate; paces the key subscriptions and the fades
  • subscriptions (array of strings, optional) - one shared frame subscription per input, in src order; an empty name (the program) subscribes nothing
  • mixer (string, optional) - the mixer whose background transitions the Next Transition keys follow (see prepare_inputs)
  • ftb_input (int, optional) - an input nothing feeds: the keyer holds one opaque black frame there, which its layer stretches over the canvas above every key. Its key fade is the fade to black of the keyed program; the clean feed, taken before the keyer, is never faded.

aux_mode, latency_ms and pgm_delay_frames fail: the keyer has no playout.

Objects (node.object.set):

  • active_inputs <mask> cuts the keys to the mask; fade_inputs {"active_inputs": <mask>, "duration_ms": 0..10000, "curve": <name>} fades the keys the mask changes (0 cuts)
  • key_changes {"inputs": <mask>, "active_inputs": <mask>, "toggle": bool, "duration_ms": 0..10000, "curve": <name>} changes the keys in inputs together on the next program frame, to the mask or (toggle) to the opposite of their state; a cut also ends a fade they are running
  • prepare_inputs {"inputs": <mask>} selects the Next Transition keys. From the first program frame of each following background take of mixer, a key turning on is at the take's progress and one turning off at its complement: they mix with a mix, hold when it is held, follow the lever, cut with a cut and switch at a wipe clip's cut point, both ways with the lever's wipe. A completed take leaves them switched; a cancelled one returns them to the state they had. A command for such a key during the take ends the take for it first.

node.object.get <node> keys lists, per key input, what the last program frame showed: on (the key's state), on_air, phase (on, off, fading-in, fading-out, or unknown before the first frame), opacity, available (it has a frame), pending (the state is not on air yet) and next_take.

mixer_pvw_follow

The only writer of an AUX bus compositor's composition: each one is the layers of the mixer's preview scene followed by a base, so the preview changes on the bus frame that leaves it with the take's first program frame (see mixer.md). What the layers draw is the control side's: this node knows no cells. No edges.

Parameters:

  • mixer (string, required) - the mixer whose preview changes it follows
  • compositor (string, required) - the bus's mixer_compositor (aux_mode), resolved once at creation
  • fps (ratio string, required), latency_ms (float, required) - the bus compositor's rate and playout latency
  • main_latency_ms (float, default latency_ms) - the main mixer's playout latency: when a program frame leaves its compositor
  • pgm_delay_frames (int, default 0) - the bus's program input delay
  • align (string, default program) - program: change with the program output; pgm_tile: with the bus's program input, pgm_delay_frames later
  • layout (object, optional) - the initial layout object

Objects (node.object.set):

  • layout - {"revision": "<string>", "pvw": {"<scene>": {"layers": [...], "active_inputs": <mask>}, ...}, "base": {"layers": [...], "active_inputs": <mask>}}, replaced as a whole and applied at the next wake. Every composition carries its revision.

node.object.get <node> status: the last change's pvw_scene, pgm_scene, kind, align, target_tick and latencies (pvw_latency_ms, pgm_latency_ms, pvw_minus_pgm_ms, last_change_to_apply_ms, last_target_error_ticks, target_unreachable), the layout_revision it holds and error.

Objects (node.object.set):

  • active_inputs (mask) - the keys on air, switched with a cut
  • fade_inputs ({"active_inputs": <mask>, "duration_ms": 0..10000, "curve": "linear"}) - the keys the mask switches fade over duration_ms; curve is linear, ease-in, ease-out or ease-in-out
  • layers - as for cuda_rect_overlay

jittergen

Enabled only if avplumber is compiled with BUILD_TYPE=Debug. Delay packets or frames for a random time. Timestamps aren't modified. Delay will be gradually decreased down to 1ms if a congestion is detected.

1 input, 1 output: anything

delaygen

Enabled only if avplumber is compiled with BUILD_TYPE=Debug. Delay packets or frames for specified time. Timestamps aren't modified.

1 input, 1 output: anything

  • delay (float) - mandatory, delay in seconds