JellyfinSyncPlus/PLAN.md

JellyfinSyncPlus — Plan

Problem

Jellyfin's built-in SyncPlay drifts noticeably when watching with remote friends (tested with 2). Symptom: one participant is consistently ahead or behind the others, not just momentarily but persistently.

Root cause

Confirmed against known Jellyfin issues (jellyfin-web#6210, jellyfin#5557, jellyfin#7579): when group members end up on different playback paths (one Direct Plays, another transcodes -- due to codec/bitrate/device differences), the transcoding client's decode/render pipeline runs measurably behind, and pause/resume compounds the gap. SyncPlay's own resync mechanism (periodic hard seeks) is also fragile and doesn't fully compensate.

Immediate mitigation (no code, do this first / always): check Dashboard -> Active Sessions during a session and confirm all clients show Direct Play, not Transcode. Force "Maximum/Original" quality on every client. Fully leave/rejoin the SyncPlay group rather than trusting in-place resync.

Why build something custom

Jellyfin's own SyncPlay is the only real option today -- nothing has superseded it as of mid-2026. jellyfin-mpv-shim is actively maintained (now under the official jellyfin GitHub org) but rides the same underlying SyncPlay protocol; it only helps indirectly by avoiding transcode on capable devices. Standalone syncplay.pl + Jellyfin direct-stream URLs is undocumented/unproven for this use case. So: build a plugin that replaces SyncPlay's weak points rather than switching tools.

Architecture -- three pillars

Status note (2026-07-09): all three pillars below were built and then reverted after real bugs/unwanted behavior in live use (see "Rough phase breakdown" for the full account of each). Pillars 1/2: forcing quality/path alignment through the PlaybackInfo request froze clients on quality switches and left them permanently desynced. Pillar 3: a scoped-down active seek-correction on top of stock Jellyfin's own (already quite good) client correction caused its own feedback-loop problems and, even fixed, wasn't what the user wanted in practice. The original rationale is kept below for context, but as of this note the plugin has no active functionality -- it's back to the Phase 0 scaffold baseline. Read "Rough phase breakdown" for what's actually true today, and don't restart from these designs without addressing why each one broke.

1. Group quality negotiation (the easy piece)

On group formation, resolve each member's device capability/bitrate profile (Jellyfin already does this per-client via its existing GetPlaybackInfo / DeviceProfile negotiation -- reuse that logic, don't reinvent it) and pick the most restrictive result across the whole group. Every member streams at that one common profile.

2. Shared transcode (the hard, novel piece)

Problem with (1) alone: even at matched quality, N independently-running ffmpeg transcode jobs (one per client, Jellyfin's default) have their own segment boundaries and start offsets -- literally different encodes, which is its own source of drift.

Jellyfin's ITranscodeManager ties one transcode job to one PlaySessionId by design; there's no built-in dedup for identical (item, profile) requests. Patching that is real core surgery -- not the v1 approach.

Pragmatic v1: host-relay. One group member is the "host" -- their session transcodes/direct-plays completely normally, untouched. The plugin exposes a relay API route that proxies the host's already-generated HLS playlist/segments to the other group members, so everyone reads byte-identical media without touching Jellyfin's transcode internals at all. Gets ~90% of the benefit of true shared-session transcoding for a fraction of the engineering risk.

3. Active drift correction (still needed even with #1 + #2)

Identical bytes doesn't mean identical render timing -- per-client network fetch latency and decode speed still differ. Needed on top:

• Virtual playhead as the single authority. Server tracks {position, wall-clock timestamp of that position, playing/paused, speed} and computes expected_position = last_position + (now - last_command_time). All clients (including the host) sync against this abstraction, never against each other -- avoids the host always looking "correct" to itself.
• NTP-style clock calibration per client on join -- ping/pong to estimate RTT and clock offset. Needed both for scheduling (below) and for correctly interpreting heartbeat reports.
• Heartbeats + EMA-smoothed drift. Each client reports its real player-engine position (not a fetch/buffer estimate) every 1-2s. Server diffs against the virtual playhead (RTT/2-adjusted) and smooths with an EMA filter rather than reacting to every raw sample.
• Two-tier correction:
  • Small drift (~150-300ms): nudge local playback rate briefly (1.0 -> ~1.02-1.05x or 0.97x) until closed -- imperceptible, no visible jump. (Same trick syncplay.pl uses for local-file sync.)
  • Large drift (~1.5-2s+, typically post-stall): hard-seek that client back onto the playhead instead -- speed-nudging alone would take too long and be noticeable.
• Scheduled commands, not "do it now." Play/pause/seek carry an absolute UTC execution timestamp; each client fires locally at exactly T using its calibrated offset, so message-delivery latency itself isn't a desync source.
• Late joiners / reconnects both reduce to: query current virtual playhead, buffer ahead until healthy, join via a scheduled seek to the interpolated position.

Scope confirmed with user

• Disconnects, late joiners, seek/pause propagation: in scope for v1.
• Quality-profile picker menu (AVC-only override etc.): explicitly out of scope for the bot project (MR-Discord) but not discussed here -- this project only forces a common profile for sync purposes, doesn't add a UI picker.

Target environment (real, not hypothetical)

• Jellyfin runs in the entertainment namespace of the user's k8s cluster, image linuxserver/jellyfin:10.11.6 (pinned -- build the plugin against this exact server version's ABI).
• Config volume: hostPath PVC at /mnt/redundant/k8s/jellyfin/config (jellyfin-config-pvc, jellyfin-config StorageClass).
• Transcoding scratch: hostPath PVC at /mnt/nvme/k8s/jellyfin/transcoding (jellyfin-transcoding-pvc, currently on the local-retain StorageClass -- scheduled for consolidation into nvme-retain in the separate, paused k8s/entertainment hardening plan, not part of this project). See entertainment/base/jellyfin/deployment.yaml and entertainment/base/jellyfin/config-claim.yaml.
• Deployed via ArgoCD, app name entertainment. Does not auto-sync -- every deploy needs a manual kubectl patch application entertainment -n argocd --type merge -p '{"operation":{"sync":{"revision":"HEAD"}}}' (same pattern used for MR-Discord releases).
• Sibling project MR-Discord already has a working tag-push -> Gitea Actions -> build -> deploy pipeline against this same cluster; reuse that pattern for this project's releases once it's stable.

Dev / test / iterate workflow

Two different loops -- do not conflate them:

1. Fast dev loop (default, use this for almost everything): run a throwaway local Jellyfin container (docker run linuxserver/jellyfin:10.11.6, same tag as prod -- avoids ABI surprises) with the plugins folder bind-mounted. Loop: dotnet build -> copy DLL into the mounted folder -> restart the local container -> check logs -> test. Jellyfin has no plugin hot-reload, so a restart is unavoidable per change, but a local container restart is seconds, not a k8s rollout.
2. Milestone testing against the real pod (deliberate, ask first): copy the built plugin into the real hostPath config dir on the node, then kubectl rollout restart deployment/jellyfin -n entertainment. This briefly interrupts the live, shared Jellyfin instance -- do not do this routinely; only for actual multi-friend sync test sessions, and confirm with the user first.
3. Release distribution (later, once stable): package as a versioned zip
   • meta.json, host a plugin-repository manifest via Gitea (raw file URLs or release attachments both work, Jellyfin just needs HTTP(S) access to the manifest + zip), add the repo URL in Jellyfin's dashboard. Mirrors MR-Discord's tag -> Gitea Actions -> build pipeline, just producing a plugin zip instead of a Docker image. Slow iteration loop -- not for development, only for shipping tested versions.

Known friction / risks (flagged going in, not discovered later)

• ABI/version pinning: meta.json's targetAbi must match the running server version. Already fine since the deployment is pinned to 10.11.6, not floating -- build against that exact version's SDK.
• Reaches past the stable plugin API -- narrower than originally assumed. Verified during Phase 1: ISyncPlayManager, IGroupStateContext, GroupInfoDto, and all the IGroupPlaybackRequest DTOs are public in the Jellyfin.Controller NuGet package -- group internals aren't actually hidden. What is missing from the SDK is the concrete implementations: Emby.Server.Implementations.SyncPlay.SyncPlayManager (needed to decorate ISyncPlayManager without reimplementing group logic) and Jellyfin.Api.Controllers.MediaInfoController / Jellyfin.Api.Models.MediaInfoDtos.PlaybackInfoDto (needed to target the PlaybackInfo action with an MVC filter). Both are public classes but ship only inside the server image, not as NuGet packages. Working solution: extract the two DLLs straight from the pinned linuxserver/jellyfin:10.11.6 image (docker cp <container>:/usr/lib/jellyfin/bin/{Emby.Server.Implementations,Jellyfin.Api}.dll) into lib/jellyfin-10.11.6/ and reference them as <Reference Private="false"> -- compile-time only, since they're already loaded in the host process at runtime. No source checkout/submodule needed after all; re-extract these two files if the pinned server version ever changes. See src/JellyfinSyncPlus/SyncPlay/ for the resulting decorator (SyncPlusSyncPlayManager) and MVC filter (SyncPlusPlaybackInfoFilter).
• Host-relay approach (pillar 2) means the "host" member is a single point of failure for the group's stream -- needs a defined handoff/reselection behavior if the host disconnects mid-session (not just late-joiners).

Rough phase breakdown

• Phase 0 -- scaffold & dev loop. DONE. dotnet plugin project scaffold against Jellyfin.Controller/Jellyfin.Model 10.11.6 (public SDK was sufficient for this trivial plugin), local docker dev Jellyfin (linuxserver/jellyfin:10.11.6) with bind-mounted plugins folder, confirmed via container logs: Loaded plugin: JellyfinSyncPlus 0.1.0.0. Gotcha discovered: in linuxserver/jellyfin, the real plugins directory is /config/data/plugins, not /config/plugins (the latter exists but is unused/dead) -- dev/docker-compose.yml bind-mounts to the correct path now, don't regress this on a future compose edit.
• Phase 1 -- group quality negotiation. REVERTED 2026-07-09, superseded by Phase 3's approach below. Built and live-verified (bitrate cap negotiation via GroupQualityStore/SyncPlusPlaybackInfoFilter, working correctly against real multi-device sessions -- see git history for the full account if this needs resurrecting). Torn out after a real, serious bug surfaced in actual use: switching quality mid-playback forces Jellyfin to issue a new PlaySessionId and reinitialize the player; our filter mutating that request made the client freeze on the switch and then never resync afterward -- worse than the drift it was meant to prevent. Root problem wasn't the negotiation math, it was forcing anything through the PlaybackInfo request/response path at all, since Jellyfin has no seam there for a plugin to intervene without looking, to the client, exactly like a user-initiated quality change. Replaced by active, corrective drift-correction instead of preventive path-forcing (Phase 3).
• Phase 2 -- host-relay shared transcode. REVERTED 2026-07-09, same reason as Phase 1 (the relay redirect went through the same PlaybackInfo response mutation that caused the freeze/permanent-desync bug). The engineering was sound and fully live-verified end-to-end against real video (real bug found and fixed along the way: browsers fetch HLS playlist/segment URLs with no auth header, only Jellyfin's ApiKey= query-param convention, which the relay didn't originally include) -- but it's not worth resurrecting unless a future need specifically requires deduplicating transcode load across a group, since Phase 3's correction now handles the drift problem this was meant to solve, without the instability. See git history for the implementation if transcode load dedup becomes a real requirement later.
• Phase 3 -- active drift correction. TRIED, REVERTED 2026-07-09. Built a scoped-down version of this section's original plan (server watches standard PlaybackProgress reports, seeks back any member who drifts more than 3 seconds ahead of the group's laggard, one-directional only, plus a toast notification -- all via existing Jellyfin mechanisms, no custom client). Confirmed along the way that stock Jellyfin's web client already ships real NTP-style clock sync and two-tier correction (syncPlay-core-PlaybackCore.*.chunk.js, SpeedToSync/SkipToSync, with a real settings UI) and keeps steady-state drift bounded to ~70-340ms on its own -- that finding still stands and is worth remembering if this gets revisited. But the active correction on top of it didn't hold up in real use: a manual timeline seek triggered a feedback loop (fixed once -- needed a per-device cooldown and to stop trusting an optimistically- written position as the "laggard" reference -- see git history for the full bug writeup) and even after that fix the user's call was that the behavior still wasn't right in practice. Pulled entirely rather than chase it further: SyncPlusDriftCorrector, SyncPlusSyncPlayManager, IGroupMembershipStore/GroupMembershipStore, and the Emby.Server.Implementations.dll reference they needed are all gone. The plugin is back to the Phase 0 baseline: loads, does nothing. All three original pillars (quality negotiation, host-relay, active correction) have now been tried and reverted -- see git history for any of them if a future direction wants to resurrect pieces, but don't restart from these designs without addressing why each one broke in real use (see the Phase 1/2 entries above and this one).
• Phase 4 -- disconnects, late joiners, seek/pause polish.
• Phase 5 -- packaging & release pipeline. Gitea-hosted plugin repository manifest, versioned zip releases, mirroring MR-Discord's pipeline.

Stats for nerds (added 2026-07-09, outside the original three-pillar scope)

DONE, live-verified. After all three active-intervention pillars got reverted, the next thing built was purely observational instead: a YouTube-"stats for nerds"-style live view of every SyncPlay group's members, their drift relative to each other, and their playback method/bitrate/codec. Deliberately has zero effect on playback -- it only reads ISessionManager.Sessions live on each request, no in-memory state, no corrective action -- so it can't have the feedback-loop or freeze failure modes the reverted features did.

• SyncPlusStatsController (GET /SyncPlus/Stats, admin-only via MediaBrowser.Common.Api.Policies.RequiresElevation) returns a live JSON snapshot: for each SyncPlay group (from the passive IGroupMembershipStore/ SyncPlusSyncPlayManager tracking -- same decorator pattern as before, minus anything that mutates a request or issues a command), each member's device/user name, now-playing item, position, drift in ms versus the group's laggard, pause state, play method, and transcode bitrate/codec/container/reason.
• Real blocker found and worked around: originally built as an interactive plugin config page (configPage.html, reachable from Dashboard -> Plugins), but its inline <script> never ran -- confirmed live by inspecting the actual DOM after injection: 0 <script> elements survived. Jellyfin's web client runs plugin config page HTML through DOMPurify before injecting it into the SPA, which strips script tags entirely (also confirmed the older pageshow/pagehide custom events legacy plugin pages relied on aren't dispatched by the current React-based client either -- tried that first, same silent no-op result). No script tag survives that DOMPurify pass, regardless of what would have triggered it.
• Fix: serve the interactive page as a genuine standalone HTML response instead (GET /SyncPlus/Stats/Page, [AllowAnonymous] since the page shell itself has nothing sensitive in it) -- a real page navigation isn't run through the SPA's sanitizer, so inline <script> works normally there. The page reads the current user's token straight out of the same jellyfin_credentials localStorage entry the main web client itself uses (same origin, so no separate login needed as long as the admin is already logged into Jellyfin in that browser) and uses it to poll the real, auth-gated /SyncPlus/Stats endpoint every second. The plugin's actual Dashboard config page is now just a static link to this standalone page -- static HTML links work fine even with scripts stripped.
• Live-verified 2026-07-09: confirmed the DOMPurify script-stripping diagnosis directly (0 script tags in the post-injection DOM), confirmed the standalone page loads and polls correctly (timestamp advancing second-to-second, no console errors), and confirmed a real SyncPlay group join shows up correctly in the JSON response (device/user name populated immediately; position/bitrate fields are null until actual playback starts, which is correct -- there's nothing to report yet).
• Not yet tested: real multi-device drift numbers (only one browser session was available this round) and the config-page link's target="_blank" actually opening a new tab in a real browser (vs. this session's single-tab preview environment, where it was verified by direct navigation instead).
• Real bug found and fixed same day: the stats page's JS assumed camelCase JSON property names (group.members, m.deviceId), but Jellyfin's API serializes in PascalCase (Members, DeviceId) -- confirmed against the actual response body, which had been sitting right there in an earlier curl test output the whole time. Threw Cannot read properties of undefined (reading 'forEach') as soon as a real group existed to render. Fixed by matching the JS property access to the DTOs exactly; worth double-checking actual response shapes against assumptions before shipping JS that consumes them, not just assuming a convention.
• Manual "sync me" button added 2026-07-09. User-requested: a one-shot, human-triggered action to fix your own drift on demand, as an alternative to the automatic corrector that got reverted for looping. POST /SyncPlus/Stats/Sync -- any authenticated user (not admin-gated like GetStats, since it only ever touches the caller's own session) seeks their own device to match whoever else in their group is furthest behind, using the exact same single-session SendSyncPlayCommand seek mechanism the reverted auto-corrector used. The difference that matters: this only ever fires once, when a human clicks a button -- there's no "again" until someone clicks again, so it structurally can't produce the re-correct-faster-than-the-previous-seek-lands feedback loop that broke the automatic version. No-ops safely (with an explanatory message, not an error) if the caller isn't in a group, isn't playing anything, or is already at or behind the group's laggard -- never pulls anyone but the caller, and never moves the caller forward past the laggard's position (same spoiler-avoidance principle as before). Button added to the stats page (/SyncPlus/Stats/Page). Live-verified 2026-07-09: confirmed both no-op paths fire correctly and with the right message ("not in a group", "nothing playing on this device"). Not yet tested: the actual successful-correction path with a real drifted second device (only one browser session available this round, same recurring constraint as everything else that needs two real logins).

Open questions (revisit before/at the relevant phase, not now)

• Exact client compatibility target for v1 -- web client only, or also jellyfin-mpv-shim/mobile apps? (Web client is simplest to control fully; others may need companion-side changes.)
• Host reselection policy on disconnect -- pick another member automatically, or pause the group and prompt?