perf(video): move NV12 conversion to GPU - #14
Merged
programmersd21 merged 3 commits intoAug 11, 2026
Conversation
Reviewer's GuideIntroduces a GPU-based NV12→RGB conversion path and generation-aware video playback pipeline that preserves hardware-decoded NV12 frames, enforces backpressure and FPS caps, and reuses per-video conversion resources across daemon and preview flows while retaining RGBA fallback and validating new shaders and color handling utilities. Sequence diagram for generation-safe NV12 playback with GPU conversion and FPS capsequenceDiagram
actor User
participant Daemon
participant RenderState
participant VideoPlayback
participant VideoDecoder
participant Renderer
participant LivePacer
User->>Daemon: select video
Daemon->>RenderState: commit_video(path)
RenderState->>VideoPlayback: start(path, hw_accel, preload_frames, generation)
VideoPlayback->>VideoDecoder: with_preload(path, hw_accel, preload_frames)
VideoDecoder-->>VideoPlayback: VideoMetadata(NV12/RGBA)
RenderState->>Renderer: create_video_texture(metadata.width, metadata.height)
RenderState->>VideoPlayback: wait_first_frame(timeout)
VideoPlayback-->>RenderState: first VideoFrame{data: VideoFrameData}
RenderState->>Renderer: update_video_texture(video_texture, frame.data)
Note over VideoDecoder: decode_hw NV12
VideoDecoder->>VideoDecoder: copy_nv12_planes()
VideoDecoder-->>VideoPlayback: next_frame()
loop play_video
Daemon->>LivePacer: wait_until(previous + min_frame_interval)
Daemon->>VideoPlayback: next_frame_in_generation(generation)
VideoPlayback-->>Daemon: VideoFrame or None
alt frame available
Daemon->>Renderer: update_video_texture(video_texture, frame.data)
else no frame
Daemon->>VideoPlayback: time_until_next_frame_in_generation(generation)
VideoPlayback-->>Daemon: wait_duration
Daemon->>LivePacer: wait_until(now + wait_duration)
end
Daemon->>Renderer: present(FrameRequest{bg_bind, new_bind})
Renderer-->>Daemon: FrameStatus::Presented
Daemon->>LivePacer: record last_present
end
Daemon->>VideoPlayback: stop_generation(generation)
VideoPlayback->>VideoDecoder: drop()
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
|
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughThe video pipeline now preserves NV12 frames and YUV metadata, converts them to sRGB through a dedicated WGSL renderer pipeline, and uses generation-aware playback with configurable preloading and FPS pacing in daemon and preview paths. ChangesVideo pipeline
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant VideoDecoder
participant VideoPlayback
participant Renderer
participant DaemonOrPreview
VideoDecoder->>VideoPlayback: decoded VideoFrame
VideoPlayback->>DaemonOrPreview: generation-specific frame
DaemonOrPreview->>Renderer: update_video_texture
Renderer->>DaemonOrPreview: VideoTexture bind groups
DaemonOrPreview->>Renderer: render video texture
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches 💡 1🛠️ Fix failing CI checks 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (2)
wallr-core/src/video/playback.rs (1)
57-68: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueDocument the generation contract on the public generation-scoped methods.
stop_generation,next_frame_in_generation, andtime_until_next_frame_in_generationare public and silently no-op when the generation does not match.wallr-core/src/daemon/mod.rsrelies on that behavior inplay_videoto avoid killing a successor's playback. A short doc comment records the contract for future callers.The lock order is consistent with
next_frame_for_generationandseek(decoder, then scheduler, then pending), so no deadlock is introduced.📝 Proposed doc comments
+ /// Stops playback only if `generation` is still the active generation. + /// A newer `start` call makes this a no-op, so a superseded caller cannot + /// stop its successor's playback. pub fn stop_generation(&self, generation: u64) {🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@wallr-core/src/video/playback.rs` around lines 57 - 68, Add concise public documentation to stop_generation, next_frame_in_generation, and time_until_next_frame_in_generation describing the generation-match contract and that mismatched generations are silently ignored without affecting successor playback. Preserve the existing lock ordering and behavior.wallr-core/src/video/decoder.rs (1)
495-536: 🎯 Functional Correctness | 🔵 Trivial | 💤 Low valueBackpressure exit discards the decoded frame on pause.
If a
Pausecontrol arrives while the queue is full, the loop setsinterruptedand breaks out of thereceive_frameloop. The already-decoded frame is dropped, plus any other frames still buffered in the decoder. AfterResume, playback continues from the next packet read, so a short visual jump is possible.The stop and seek paths do not care about the dropped frame. Only the pause path loses a valid frame. Consider keeping the frame and re-checking the queue after resume, or accept the drop and document it.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@wallr-core/src/video/decoder.rs` around lines 495 - 536, The backpressure handling in the decoder’s receive_frame loop drops an already-decoded frame when Pause arrives. Preserve that frame across pause by distinguishing pause interruption from seek/stop, retaining the decoded frame and resuming queue-capacity checks after Resume; keep existing stop and seek behavior unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@wallr-core/shaders/nv12_to_rgb.wgsl`:
- Around line 28-34: Align the NV12 shader transfer-function convention with the
RGBA fallback and the effects shader’s expected input. Update sdr_to_linear and
the output texture/view configuration together so both decode paths produce
consistently encoded mid-tones, preserving the chosen sRGB-equivalent or
BT.709-linear convention across nv12_to_rgb and the fallback upload path.
In `@wallr-core/src/daemon/mod.rs`:
- Around line 738-749: Update the video texture setup in the first-frame commit
path to derive its dimensions from first_frame when present, then create the
texture using those dimensions before calling update_video_texture. Preserve the
metadata dimensions as the fallback when no first frame exists, including the
existing black-texture behavior.
- Around line 1249-1253: Update the dimension-mismatch branch in the playback
frame loop to log the mismatch once and wait before retrying, rather than
immediately continuing. Ensure the wait also applies when last_present is None,
preventing repeated mismatched frames from spinning on the playback mutex while
preserving the existing generation-change handling.
---
Nitpick comments:
In `@wallr-core/src/video/decoder.rs`:
- Around line 495-536: The backpressure handling in the decoder’s receive_frame
loop drops an already-decoded frame when Pause arrives. Preserve that frame
across pause by distinguishing pause interruption from seek/stop, retaining the
decoded frame and resuming queue-capacity checks after Resume; keep existing
stop and seek behavior unchanged.
In `@wallr-core/src/video/playback.rs`:
- Around line 57-68: Add concise public documentation to stop_generation,
next_frame_in_generation, and time_until_next_frame_in_generation describing the
generation-match contract and that mismatched generations are silently ignored
without affecting successor playback. Preserve the existing lock ordering and
behavior.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 910d7b30-0355-4f00-9da4-b699db041505
📒 Files selected for processing (9)
wallr-core/shaders/nv12_to_rgb.wgslwallr-core/src/daemon/mod.rswallr-core/src/preview/mod.rswallr-core/src/renderer/mod.rswallr-core/src/shader/mod.rswallr-core/src/video/decoder.rswallr-core/src/video/mod.rswallr-core/src/video/playback.rswallr-core/src/video/scheduler.rs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Performance
Measured with the same H.264 3840x2160@60 wallpaper using NVDEC on an NVIDIA RTX 5090:
The standalone NVDEC probe decoded and packed 300 frames at 353.5 FPS.
Validation
cargo fmt --all --checkcargo check --workspacecargo clippy --workspace --all-targets -- -D warningscargo test --workspace(50 passed)Scope
This optimizes SDR NV12 playback with BT.601, BT.709, and BT.2020 non-constant-luminance matrix/range handling. HDR and P010 output remain out of scope.
Summary by Sourcery
Optimize video playback by preserving NV12 frames through the decode pipeline and performing YUV-to-RGB conversion on the GPU while tightening scheduling, buffering, and seek behavior for live video.
New Features:
Enhancements:
Tests:
Summary by CodeRabbit