CODLAB / Blog · Software & development

CardinalHost on Linux Wayland: PipeWire, /dev/uinput, Hybrid H.264 Transport and Why Frame Intervals Do Not Measure Latency

15 Nov 2026 CODLAB Team 11 min

As of November 2026: CardinalHost (currently pre-launch and documented at cardinalhost.org) delivers Linux Wayland remote support without falling back to X11 by combining xdg-desktop-portal, PipeWire and /dev/uinput. Video starts as H.264 Annex-B inside SSH, optionally promotes to ChaCha20-Poly1305 authenticated UDP after three complete frames, and strictly separates passive inter-frame intervals on stationary desktops from active input-to-pixel latency, whose benchmark runs remain open.

Why Wayland Invalidates Legacy Remote Desktop Assumptions

For decades, remote desktop tools on Linux relied on X11’s flat security perimeter: any local process with access to the X socket and authorization cookie could capture the entire root window and inject synthetic keyboard or pointer events into arbitrary applications through the XTest extension. While convenient for quick remote access tools, that model offered no architectural boundary between an unprivileged utility and full desktop takeover.

Modern Wayland compositors invert that assumption by design. Client surfaces are isolated from one another, and neither screen capture nor global input synthesis is permitted without explicit mediation by the compositor and session portals. In practice, many existing remote support tools still ask users to log out and switch back to a legacy Xorg session, fail with a blank black frame, or terminate abruptly when the local user permits screen viewing without granting remote input.

In our applied research and systems engineering work on CardinalHost within our projects research overview, we chose to treat Wayland’s security boundaries as first-class protocol primitives rather than obstacles. As part of the Cardinal product family, CardinalHost negotiates permissions explicitly and adapts cleanly to whatever capability set the session grants.

Two Distinct Linux Trust Paths Instead of One Ambiguous Agent

A foundational architectural choice in CardinalHost is separating unattended administration of machines you own from interactive remote assistance on a machine operated by someone else. Combining both use cases into a single background daemon inevitably creates either excessive privileges during support sessions or unnecessary friction for infrastructure engineers.

Dimension Path A: ServerMount Agent (Owner-Managed Systems) Path B: Public Assistance Engine (Assisted Workstation Support)
Primary Scenario Administration of personal or organisational Linux hosts by their owner Time-bounded remote assistance on another person’s desktop session
Deployment & Launch Started on demand over an existing SSH connection from ~/.local/bin Enrolled system helper (pubblico-linux.sh) with a fixed launch path
SSH Key Scoping Standard administrative SSH credentials managed by the system owner Dedicated SSH key restricted via ForceCommand with no interactive shell
Visibility & Revocation No additional inbound ports beyond SSH; no cloud account or external relay Graphical consent dialog, persistent tray indicator and instant one-click revocation

On Path A, CardinalHost launches the remote helper directly through the operator’s existing SSH session: no permanently listening custom daemon is exposed to the public network, and pure SSH operation requires no external relay broker (when optional STUN is configured for direct UDP, the configured STUN server observes the reflexive endpoint address per RFC 5389). On Path B, the person receiving support retains continuous sovereignty over their workstation: the enrolled SSH key cannot spawn an arbitrary shell because ForceCommand locks execution to the assistance binary, a graphical consent prompt must be approved before capture begins, a desktop tray icon indicates active sharing, and revoking consent immediately tears down both capture and input injection.

Capture via PipeWire and Input via /dev/uinput

Under Linux Wayland, CardinalHost structures desktop access across three explicit layers: D-Bus portal negotiation, PipeWire media transport, and kernel/portal input injection.

  • Portal Negotiation: Using org.freedesktop.portal.ScreenCast and org.freedesktop.portal.RemoteDesktop, the Linux agent requests screen capture and input rights from the active desktop session. Monitor selection and permission prompts remain fully controlled by the compositor.
  • H.264 Annex-B Pipeline Handover and Colour-Conversion Status: Once granted, the portal provides a PipeWire node descriptor from which our GStreamer pipeline reads frames via pipewiresrc and emits an H.264 Annex-B stream with SPS/PPS placed ahead of stream entry points. In the state measured on 15 August 2026 on ServerMount Agent (Path A), frames arrived in system memory, underwent two sequential conversions (videoconvert in software followed by vapostproc) and consumed roughly 50% of one CPU core even on a motionless desktop; since 15 September 2026 (deliverable P3), the public assistance engine (Path B) negotiates a direct DMA-BUF-to-VA-API path with automatic system-memory fallback (verified for pipeline structure in GStreamer test benches, though not yet measured on real VA hardware), whereas ServerMount Agent still converts in software.
  • Deterministic View-Only Degradation: If the desktop portal declines remote input permissions—or if /dev/uinput device permissions are absent—CardinalHost does not drop the connection or stall in an inconsistent state. Instead, the handshake explicitly reports input unavailability and transitions into view-only mode, preserving live visual diagnostics while clearly informing the operator.

Transparent Platform Coverage: Full end-to-end Wayland operation—covering PipeWire capture, pointer and keyboard control via /dev/uinput, and deterministic fallback to view-only—is currently field-tested on Fedora 45 running GNOME 51. Support for KDE Plasma, wlroots compositors (such as Sway and Hyprland), and distribution-specific portal profiles on Ubuntu/Debian and Arch Linux is actively in development and documented as such rather than overstated.

Hybrid Video Transport: SSH Baseline with Authenticated Direct UDP Promotion

Remote desktop protocols face a well-known transport trade-off. Tunnelling video over SSH/TCP works reliably across strict corporate firewalls without extra port forwarding, yet packet loss on wide-area links can trigger TCP head-of-line blocking: a single lost segment delays delivery of subsequent video packets already sitting in the kernel receive buffer. Conversely, raw UDP avoids head-of-line stalls but frequently fails across restrictive NATs if used as the sole transport.

CardinalHost addresses this tension through a staged hybrid transport (currently implemented solely on the .agenteLinux path; the public assistance engine on Path B remains inside SSH):

  1. Immediate Startup Inside SSH: Every remote session begins immediately with H.264 Annex-B streaming inside the authenticated SSH tunnel, guaranteeing connectivity whenever SSH itself can reach the host.
  2. RFC 5389 STUN Discovery and SEI Signalling: When a STUN server is configured, endpoints discover reflexive public addresses in parallel with the live SSH stream via standard STUN Binding Requests (RFC 5389, MAGIC_COOKIE = 0x2112A442, parsing XOR-MAPPED-ADDRESS), exchanging parameters over the SSH process stdin channel and H.264 user_data_unregistered SEI messages.
  3. ChaCha20-Poly1305 Authenticated Probes: Before shifting media traffic, the client sends a random 32-byte key and session ID over the authenticated SSH channel, and both peers exchange UDP probes encrypted and authenticated with ChaCha20-Poly1305. Unauthenticated packets are dropped immediately.
  4. Three-Frame Gate and Confirmation-to-First-Frame Interval: A single small UDP probe succeeding does not prove that full-sized video frames will clear path MTU limits. CardinalHost switches to direct UDP only after three consecutive complete video frames arrive over UDP: the agent requests an IDR keyframe before sending promotion confirmation, and the client discards the three probe frames so the decoder resumes cleanly after confirmation. Across seven alternating runs on 15 August 2026 (prova-collegamento-diretto.py on Ubuntu 26.04 GNOME over .agenteLinux), the interval from promotion confirmation to first complete delivered frame measured 0.78 s to 1.50 s against a declared 2.0 s threshold—measuring keyframe resumption after confirmation, not total negotiation time or perceived input latency.
  5. 200 ms Heartbeats and 900 ms Fallback Threshold: Active UDP sessions exchange authenticated heartbeats every 200 ms. If authenticated UDP packets stop for 900 ms, the client holds the last frame on screen, re-selects the open SSH tunnel and requests a fresh IDR with SPS/PPS (900 ms is the documented transport fallback threshold without dropping the session, not a guarantee of zero visual pause).

To prevent keyframe storms on lossy links, the agent generates at most one forced IDR keyframe every 500 ms (at most 2 IDRs per second), while the client repeats an unfulfilled keyframe request over UDP every 200 ms until the IDR arrives. Sender byte pacing tracks receiver-observed throughput while maintaining a 10% safety headroom whenever loss is detected.

Applied Research in Practice: Two Falsified Hypotheses and a Rejected Filter on a Stationary Desktop

Rigorous systems engineering means letting measurements overturn attractive explanations—including our own. On 13 August 2026, we profiled the ServerMount Agent path on a real Linux machine running Ubuntu 26.04 LTS with KDE Plasma on Wayland (Intel VA-API encoder, 60 requested fps, quality 80) looking at a completely stationary desktop. On that motionless scene, the stream delivered 52.8 fps at 14.3 Mbit/s (SPS/PPS ahead of all 14 IDR frames, first frame after 1.8 s), a median inter-frame interval (p50) measured locally on the host of 16.67 ms, alongside p95 ≈ 150 ms and maximum gaps around 1.5 seconds.

Two textbook systems hypotheses initially seemed to explain those pauses—and both failed when tested directly:

  • Hypothesis 1 (“The pauses are TCP inside SSH”): Falsified because measuring the exact same stream locally on the host itself without a network in between still produced p95 = 155 ms and max = 1.5 s, while a raw transfer (dd) over the same SSH connection sustained 139 Mbit/s at p95 = 2.4 ms—roughly ten times the stream bitrate.
  • Hypothesis 2 (“The pauses come from large IDR keyframes”): An IDR keyframe weighed roughly 585 KB versus 23 KB for an ordinary frame; yet lengthening the GOP to cut IDR frames from 14 to 2 across 15 seconds worsened p95 to 78 ms instead of 67 ms, refuting that explanation too.

Follow-up measurements between 13 and 15 August 2026 on the same stationary desktop uncovered two concrete pipeline effects instead. First, videorate was repeating the previous frame during idle periods while the VA-API encoder operated in constant-bitrate mode, spending bandwidth encoding duplicates. Dropping videorate copies and switching to variable bitrate reduced stationary traffic (15-second runs) from 51.7 fps at 13.8 Mbit/s to 23.5 fps at 2.97 Mbit/s—paired with a lightweight SEI liveness marker every 2.5 seconds during quiet periods, while explicitly leaving behaviour under real desktop motion as an open measurement task. Second, roughly 47 of 50 CPU percentage points went into software colour conversion (videoconvert BGRx→NV12). A sparse byte sample (one byte per 4096) initially misread 92% of compositor frames as identical; when we built and measured a full byte-for-byte pre-conversion filter on 15 August, CPU usage rose from 47.8% to 58.4% and the filter was rejected, because only 4% to 9% of frames were strictly identical (the remainder changed a median 0.016% of bytes, or 1.6% of 64 KiB blocks, such as clock or panel pixels) and libgstpipewire.so (GStreamer 1.28.2) discarded PipeWire’s VideoDamage region metadata.

Methodological Takeaway: Passive inter-frame intervals on a stationary desktop do not measure user input-to-pixel latency. They conflate screen quiescence, tiny compositor updates and encoder cadence—a clean p50 on a motionless screen does not prove responsiveness under active motion.

Those corrections set two strict rules for our tooling and public communication. First, an automated active stimulus-to-pixel test bench (misura-tasto-pixel.py, driving full-screen black/white transitions and decoding luminance timestamps from the live H.264 stream) has been built and prepared, but has not yet been executed on a live desktop—so we report no input-to-pixel latency figure yet. Second, on cardinalhost.org we publish no comparative latency claims against Parsec, RustDesk or TeamViewer until reproducible end-to-end stimulus runs under real motion are completed. We apply the same verifiable gate discipline in our Engineering Note on automated Playwright and axe accessibility gates and across our AI and systems integration engineering services.

What Stands Today—and What Comes Next

Today, CardinalHost’s remote desktop layer rests on verifiable foundations: separated trust paths for owner-administered servers and consent-based workstation support, native Wayland capture and input with deterministic view-only degradation on Fedora 45 / GNOME 51, and a cryptographically authenticated SSH-to-UDP promotion state machine with a three-frame promotion rule and 900 ms fallback threshold. Next on our engineering roadmap are expanding automated Wayland conformance testing to KDE Plasma and wlroots compositors, unifying direct UDP promotion across all agent targets, and executing and publishing the prepared input-to-pixel latency runs under real desktop motion.

Frequently Asked Questions (FAQ)

Does CardinalHost require switching Linux desktops back to an X11 session?

No. CardinalHost works natively on Wayland by negotiating capture and input permissions through xdg-desktop-portal (ScreenCast and RemoteDesktop), capturing frames via PipeWire and injecting input through /dev/uinput. End-to-end validation is currently field-tested on Fedora 45 with GNOME 51, while additional compositors are in active development.

What happens when the Wayland desktop portal refuses remote keyboard or mouse control?

CardinalHost degrades deterministically to view-only mode without terminating the connection. Video streaming over PipeWire continues normally, and the client interface clearly indicates that remote input was not granted.

Under what conditions does CardinalHost switch video transport from SSH to direct UDP?

Every session starts inside the encrypted SSH channel. Exclusively on the .agenteLinux path (the public assistance engine on Path B stays on SSH), switching to ChaCha20-Poly1305 authenticated UDP occurs only after optional RFC 5389 STUN discovery, three consecutive complete UDP video frames, and an IDR keyframe requested prior to promotion confirmation (0.78–1.50 s from confirmation to first complete frame across seven test runs, threshold 2.0 s). If authenticated UDP traffic stops for 900 ms, the client holds the last frame and returns to the open SSH tunnel with a fresh IDR request.

Why does CodLab not publish comparative latency numbers against Parsec, RustDesk or TeamViewer yet?

Because our August 2026 measurements on Ubuntu 26.04 / KDE Wayland captured passive inter-frame intervals on a stationary screen. Those intervals reflect screen quiescence, duplicate filtering and tiny panel updates rather than the response time from user input to a changed pixel. Our active stimulus-to-pixel test bench is prepared, but comparative runs under real motion have not yet been executed.

Share this article

Comments

No comments yet. Be the first to comment!

Leave a comment

Will not be published

Related Articles

Project on your mind?

We develop tailored software and AI solutions. Let's start your next project together.

Free initial consultation