video_upscaler/context/README.md

🧠 AI Video Upscaler - Project Context & Reference

Welcome! This document acts as a memory reference folder for AI agents working on this project. It details the system architecture, code organization, pipeline stages, and custom feature implementations (Resumption & Queue Management).

---

🏗️ System Architecture

The AI Video Upscaler is a full-stack, single-user application designed to upscale videos locally using Vulkan GPU-accelerated AI models.

Technology Stack

1. FastAPI (Python Backend): Handles API endpoints, serving static assets, WebSockets, background threads, and orchestrates subprocess execution.
2. Real-ESRGAN ncnn-vulkan (AI Engine): A compiled C++ Vulkan binary located in realesrgan-bin/realesrgan-ncnn-vulkan.
3. FFmpeg & FFprobe (Multimedia Toolkit): Splitting videos into high-quality JPEG frames, querying metadata, extracting audio streams, and re-muxing audio/subtitles back into the upscaled output.
4. Vanilla HTML/CSS/JS (Frontend): Cyberpunk-themed web dashboard with real-time progress logging, side-by-side comparative preview, and queue management.

---

📁 Key File Map

• app/main.py: API route definitions, WebSocket orchestrator, custom thread-safe job queue, and JSON database state preservation (jobs.json).
• app/upscaler.py: Core pipeline orchestrator (run_upscale_pipeline). Executes external FFmpeg and Real-ESRGAN commands. Contains the resume logic.
• static/index.html, static/styles.css, static/app.js: Frontend interface, WebSocket handlers, interactive comparison slider, zoom/pan tool, and queue reordering actions.
• jobs.json: Persisted file storing details of all jobs (automatically generated at root).

---

🔄 Upscale Pipeline Lifecycle

A typical upscale job follows these sequential steps:

1. Queued (queued): Added to the FIFO queue.
2. Analyzing (analyzing): GPU lock acquired. Querying stream specs with ffprobe.
3. Extracting (extracting): FFmpeg extracts video frames into temp/<job_id>/input_frames/frame_%08d.jpg.
4. Upscaling (upscaling): Real-ESRGAN binary upscales images to temp/<job_id>/output_frames/frame_%08d.jpg.
5. Assembling (assembling): FFmpeg merges upscaled frames with original audio/subtitles.
6. Completed (completed) / Failed (failed) / Cancelled (cancelled) / Interrupted (interrupted).

---

⚡ Custom Enhancements

1. Job Resumption (interrupted status)

• Persistency: The status of all jobs is stored in jobs.json at the root. On application restart, any incomplete/active job is automatically loaded in the interrupted status.
• Skip Extraction: On resume, if temp/<job_id>/input_frames contains frames, the extraction step is bypassed.
• Incremental Upscaling: The upscaler inspects output_frames/ and checks for already upscaled frames. It removes corresponding files from input_frames/, meaning the AI model only processes the remaining un-upscaled frames.
• Fast Assembly: If all frames are already upscaled, it skips the upscaling step entirely and directly runs the FFmpeg reassembly.

2. Queue Management (Cancel & Reorder)

• Custom Queue (CustomJobQueue): A thread-safe, list-backed FIFO queue replacing the standard queue.Queue. It allows:
  • Querying the active queue list (GET /api/queue).
  • Swapping queued jobs and re-ordering (POST /api/queue/reorder).
  • Graceful removal upon cancellation.
• Interactive UI Arrows: Arrows are displayed next to queued items in the dashboard to move jobs up and down, triggering the API to swap their order on-the-fly.

3. Automatic Virtual Environment Setup (start.py)

• Self-Sufficiency: Running python3 start.py automatically checks for a local virtual environment (venv/ or .venv/).
• Auto-Provisioning: If no virtual environment is found, start.py will initialize one in venv/, upgrade pip, install all dependencies listed in requirements.txt, mark the Real-ESRGAN binary as executable (chmod +x), and create necessary folders (uploads/, outputs/, temp/).
• Seamless Launch: It then automatically launches the server process using the newly created environment interpreter.