Initial plan: SyncPlay drift fix via quality negotiation, host-relay, active drift correction
This commit is contained in:
@@ -0,0 +1,189 @@
|
||||
# JellyfinSyncPlus — Plan
|
||||
|
||||
## Problem
|
||||
|
||||
Jellyfin's built-in SyncPlay drifts noticeably when watching with remote friends
|
||||
(tested with 2). Symptom: one participant is consistently ahead or behind the
|
||||
others, not just momentarily but persistently.
|
||||
|
||||
## Root cause
|
||||
|
||||
Confirmed against known Jellyfin issues (jellyfin-web#6210, jellyfin#5557,
|
||||
jellyfin#7579): when group members end up on different playback paths (one
|
||||
Direct Plays, another transcodes -- due to codec/bitrate/device differences),
|
||||
the transcoding client's decode/render pipeline runs measurably behind, and
|
||||
pause/resume compounds the gap. SyncPlay's own resync mechanism (periodic hard
|
||||
seeks) is also fragile and doesn't fully compensate.
|
||||
|
||||
Immediate mitigation (no code, do this first / always): check Dashboard ->
|
||||
Active Sessions during a session and confirm all clients show Direct Play, not
|
||||
Transcode. Force "Maximum/Original" quality on every client. Fully leave/rejoin
|
||||
the SyncPlay group rather than trusting in-place resync.
|
||||
|
||||
## Why build something custom
|
||||
|
||||
Jellyfin's own SyncPlay is the only real option today -- nothing has superseded
|
||||
it as of mid-2026. `jellyfin-mpv-shim` is actively maintained (now under the
|
||||
official `jellyfin` GitHub org) but rides the same underlying SyncPlay protocol;
|
||||
it only helps indirectly by avoiding transcode on capable devices. Standalone
|
||||
`syncplay.pl` + Jellyfin direct-stream URLs is undocumented/unproven for this
|
||||
use case. So: build a plugin that replaces SyncPlay's weak points rather than
|
||||
switching tools.
|
||||
|
||||
## Architecture -- three pillars
|
||||
|
||||
### 1. Group quality negotiation (the easy piece)
|
||||
|
||||
On group formation, resolve each member's device capability/bitrate profile
|
||||
(Jellyfin already does this per-client via its existing `GetPlaybackInfo` /
|
||||
`DeviceProfile` negotiation -- reuse that logic, don't reinvent it) and pick the
|
||||
most restrictive result across the whole group. Every member streams at that
|
||||
one common profile.
|
||||
|
||||
### 2. Shared transcode (the hard, novel piece)
|
||||
|
||||
Problem with (1) alone: even at matched quality, N independently-running
|
||||
ffmpeg transcode jobs (one per client, Jellyfin's default) have their own
|
||||
segment boundaries and start offsets -- literally different encodes, which is
|
||||
its own source of drift.
|
||||
|
||||
Jellyfin's `ITranscodeManager` ties one transcode job to one `PlaySessionId` by
|
||||
design; there's no built-in dedup for identical (item, profile) requests.
|
||||
Patching that is real core surgery -- **not** the v1 approach.
|
||||
|
||||
**Pragmatic v1: host-relay.** One group member is the "host" -- their session
|
||||
transcodes/direct-plays completely normally, untouched. The plugin exposes a
|
||||
relay API route that proxies the host's already-generated HLS
|
||||
playlist/segments to the other group members, so everyone reads byte-identical
|
||||
media without touching Jellyfin's transcode internals at all. Gets ~90% of the
|
||||
benefit of true shared-session transcoding for a fraction of the engineering
|
||||
risk.
|
||||
|
||||
### 3. Active drift correction (still needed even with #1 + #2)
|
||||
|
||||
Identical bytes doesn't mean identical render timing -- per-client network
|
||||
fetch latency and decode speed still differ. Needed on top:
|
||||
|
||||
- **Virtual playhead as the single authority.** Server tracks
|
||||
`{position, wall-clock timestamp of that position, playing/paused, speed}`
|
||||
and computes `expected_position = last_position + (now - last_command_time)`.
|
||||
All clients (including the host) sync against this abstraction, never
|
||||
against each other -- avoids the host always looking "correct" to itself.
|
||||
- **NTP-style clock calibration per client on join** -- ping/pong to estimate
|
||||
RTT and clock offset. Needed both for scheduling (below) and for correctly
|
||||
interpreting heartbeat reports.
|
||||
- **Heartbeats + EMA-smoothed drift.** Each client reports its real
|
||||
player-engine position (not a fetch/buffer estimate) every 1-2s. Server
|
||||
diffs against the virtual playhead (RTT/2-adjusted) and smooths with an EMA
|
||||
filter rather than reacting to every raw sample.
|
||||
- **Two-tier correction:**
|
||||
- Small drift (~150-300ms): nudge local playback rate briefly
|
||||
(1.0 -> ~1.02-1.05x or 0.97x) until closed -- imperceptible, no visible
|
||||
jump. (Same trick `syncplay.pl` uses for local-file sync.)
|
||||
- Large drift (~1.5-2s+, typically post-stall): hard-seek that client back
|
||||
onto the playhead instead -- speed-nudging alone would take too long and
|
||||
be noticeable.
|
||||
- **Scheduled commands, not "do it now."** Play/pause/seek carry an absolute
|
||||
UTC execution timestamp; each client fires locally at exactly `T` using its
|
||||
calibrated offset, so message-delivery latency itself isn't a desync source.
|
||||
- **Late joiners / reconnects** both reduce to: query current virtual
|
||||
playhead, buffer ahead until healthy, join via a scheduled seek to the
|
||||
interpolated position.
|
||||
|
||||
## Scope confirmed with user
|
||||
|
||||
- Disconnects, late joiners, seek/pause propagation: in scope for v1.
|
||||
- Quality-profile picker menu (AVC-only override etc.): explicitly out of
|
||||
scope for the *bot* project (MR-Discord) but not discussed here -- this
|
||||
project only forces a common profile for sync purposes, doesn't add a UI
|
||||
picker.
|
||||
|
||||
## Target environment (real, not hypothetical)
|
||||
|
||||
- Jellyfin runs in the `entertainment` namespace of the user's k8s cluster,
|
||||
image `linuxserver/jellyfin:10.11.6` (pinned -- build the plugin against
|
||||
this exact server version's ABI).
|
||||
- Config volume: hostPath PVC at `/mnt/redundant/k8s/jellyfin/config`
|
||||
(`jellyfin-config-pvc`, `jellyfin-config` StorageClass).
|
||||
- Transcoding scratch: hostPath PVC at `/mnt/nvme/k8s/jellyfin/transcoding`
|
||||
(`jellyfin-transcoding-pvc`, currently on the `local-retain` StorageClass --
|
||||
scheduled for consolidation into `nvme-retain` in the separate,
|
||||
paused k8s/entertainment hardening plan, not part of this project).
|
||||
See [entertainment/base/jellyfin/deployment.yaml](../entertainment/base/jellyfin/deployment.yaml)
|
||||
and [entertainment/base/jellyfin/config-claim.yaml](../entertainment/base/jellyfin/config-claim.yaml).
|
||||
- Deployed via ArgoCD, app name `entertainment`. **Does not auto-sync** --
|
||||
every deploy needs a manual
|
||||
`kubectl patch application entertainment -n argocd --type merge -p '{"operation":{"sync":{"revision":"HEAD"}}}'`
|
||||
(same pattern used for MR-Discord releases).
|
||||
- Sibling project [MR-Discord](../MR-Discord) already has a working
|
||||
tag-push -> Gitea Actions -> build -> deploy pipeline against this same
|
||||
cluster; reuse that pattern for this project's releases once it's stable.
|
||||
|
||||
## Dev / test / iterate workflow
|
||||
|
||||
Two different loops -- do not conflate them:
|
||||
|
||||
1. **Fast dev loop (default, use this for almost everything):** run a
|
||||
throwaway local Jellyfin container (`docker run linuxserver/jellyfin:10.11.6`,
|
||||
same tag as prod -- avoids ABI surprises) with the plugins folder
|
||||
bind-mounted. Loop: `dotnet build` -> copy DLL into the mounted folder ->
|
||||
restart the local container -> check logs -> test. Jellyfin has no plugin
|
||||
hot-reload, so a restart is unavoidable per change, but a local container
|
||||
restart is seconds, not a k8s rollout.
|
||||
2. **Milestone testing against the real pod (deliberate, ask first):** copy
|
||||
the built plugin into the real hostPath config dir on the node, then
|
||||
`kubectl rollout restart deployment/jellyfin -n entertainment`. This
|
||||
briefly interrupts the live, shared Jellyfin instance -- **do not do this
|
||||
routinely; only for actual multi-friend sync test sessions, and confirm
|
||||
with the user first.**
|
||||
3. **Release distribution (later, once stable):** package as a versioned zip
|
||||
+ `meta.json`, host a plugin-repository manifest via Gitea (raw file URLs
|
||||
or release attachments both work, Jellyfin just needs HTTP(S) access to
|
||||
the manifest + zip), add the repo URL in Jellyfin's dashboard. Mirrors
|
||||
MR-Discord's tag -> Gitea Actions -> build pipeline, just producing a
|
||||
plugin zip instead of a Docker image. Slow iteration loop -- not for
|
||||
development, only for shipping tested versions.
|
||||
|
||||
## Known friction / risks (flagged going in, not discovered later)
|
||||
|
||||
- **ABI/version pinning**: `meta.json`'s `targetAbi` must match the running
|
||||
server version. Already fine since the deployment is pinned to `10.11.6`,
|
||||
not floating -- build against that exact version's SDK.
|
||||
- **Reaches past the stable plugin API.** SyncPlay group internals and
|
||||
transcode-session state aren't part of Jellyfin's published plugin SDK
|
||||
surface. Realistically requires building against Jellyfin server source at
|
||||
the matching tag (`git` checkout/submodule of `jellyfin/jellyfin` @
|
||||
`v10.11.6`) rather than just NuGet packages. This is closer to
|
||||
"semi-fork with a plugin wrapper" than a typical plugin -- sizing decisions
|
||||
should account for that.
|
||||
- Host-relay approach (pillar 2) means the "host" member is a single point of
|
||||
failure for the group's stream -- needs a defined handoff/reselection
|
||||
behavior if the host disconnects mid-session (not just late-joiners).
|
||||
|
||||
## Rough phase breakdown
|
||||
|
||||
- **Phase 0 -- scaffold & dev loop.** `dotnet` plugin project scaffold against
|
||||
`Jellyfin.Controller` (or a source checkout of `jellyfin/jellyfin@v10.11.6`
|
||||
if the public SDK isn't sufficient), local docker dev Jellyfin with
|
||||
bind-mounted plugins folder, confirm a trivial "hello world" plugin loads
|
||||
and shows up in Dashboard -> Plugins.
|
||||
- **Phase 1 -- group quality negotiation.** Hook group-join, resolve each
|
||||
member's device profile, compute the common lowest profile, verify against
|
||||
real device/browser combos.
|
||||
- **Phase 2 -- host-relay shared transcode.** Custom API route serving the
|
||||
host session's HLS playlist/segments to other members; handle host
|
||||
disconnect/reselection.
|
||||
- **Phase 3 -- active drift correction.** Virtual playhead, clock calibration,
|
||||
heartbeat + EMA, two-tier nudge/reseek correction, scheduled absolute-time
|
||||
commands.
|
||||
- **Phase 4 -- disconnects, late joiners, seek/pause polish.**
|
||||
- **Phase 5 -- packaging & release pipeline.** Gitea-hosted plugin repository
|
||||
manifest, versioned zip releases, mirroring MR-Discord's pipeline.
|
||||
|
||||
## Open questions (revisit before/at the relevant phase, not now)
|
||||
|
||||
- Exact client compatibility target for v1 -- web client only, or also
|
||||
`jellyfin-mpv-shim`/mobile apps? (Web client is simplest to control fully;
|
||||
others may need companion-side changes.)
|
||||
- Host reselection policy on disconnect -- pick another member automatically,
|
||||
or pause the group and prompt?
|
||||
Reference in New Issue
Block a user