# 🧠 AI Video Upscaler - Project Context & Reference

Welcome! This document acts as a memory reference folder for AI agents working on this project. It details the system architecture, code organization, pipeline stages, and custom feature implementations (Resumption & Queue Management).

---

## 🏗️ System Architecture

The AI Video Upscaler is a full-stack, single-user application designed to upscale videos locally using Vulkan GPU-accelerated AI models.

### Technology Stack
1. **FastAPI (Python Backend)**: Handles API endpoints, serving static assets, WebSockets, background threads, and orchestrates subprocess execution.
2. **Real-ESRGAN ncnn-vulkan (AI Engine)**: A compiled C++ Vulkan binary located in `realesrgan-bin/realesrgan-ncnn-vulkan`.
3. **FFmpeg & FFprobe (Multimedia Toolkit)**: Splitting videos into high-quality JPEG frames, querying metadata, extracting audio streams, and re-muxing audio/subtitles back into the upscaled output.
4. **Vanilla HTML/CSS/JS (Frontend)**: Cyberpunk-themed web dashboard with real-time progress logging, side-by-side comparative preview, and queue management.

---

## 📁 Key File Map

- **`app/main.py`**: API route definitions, WebSocket orchestrator, custom thread-safe job queue, and JSON database state preservation (`jobs.json`).
- **`app/upscaler.py`**: Core pipeline orchestrator (`run_upscale_pipeline`). Executes external FFmpeg and Real-ESRGAN commands. Contains the resume logic.
- **`static/index.html`**, **`static/styles.css`**, **`static/app.js`**: Frontend interface, WebSocket handlers, interactive comparison slider, zoom/pan tool, and queue reordering actions.
- **`jobs.json`**: Persisted file storing details of all jobs (automatically generated at root).

---

## 🔄 Upscale Pipeline Lifecycle

A typical upscale job follows these sequential steps:
1. **Queued (`queued`)**: Added to the FIFO queue.
2. **Analyzing (`analyzing`)**: GPU lock acquired. Querying stream specs with `ffprobe`.
3. **Extracting (`extracting`)**: FFmpeg extracts video frames into `temp/<job_id>/input_frames/frame_%08d.jpg`.
4. **Upscaling (`upscaling`)**: Real-ESRGAN binary upscales images to `temp/<job_id>/output_frames/frame_%08d.jpg`.
5. **Assembling (`assembling`)**: FFmpeg merges upscaled frames with original audio/subtitles.
6. **Completed (`completed`)** / **Failed (`failed`)** / **Cancelled (`cancelled`)** / **Interrupted (`interrupted`)**.

---

## ⚡ Custom Enhancements

### 1. Job Resumption (`interrupted` status)
- **Persistency**: The status of all jobs is stored in `jobs.json` at the root. On application restart, any incomplete/active job is automatically loaded in the `interrupted` status.
- **Skip Extraction**: On resume, if `temp/<job_id>/input_frames` contains frames, the extraction step is bypassed.
- **Incremental Upscaling**: The upscaler inspects `output_frames/` and checks for already upscaled frames. It removes corresponding files from `input_frames/`, meaning the AI model only processes the remaining un-upscaled frames.
- **Fast Assembly**: If all frames are already upscaled, it skips the upscaling step entirely and directly runs the FFmpeg reassembly.

### 2. Queue Management (Cancel & Reorder)
- **Custom Queue (`CustomJobQueue`)**: A thread-safe, list-backed FIFO queue replacing the standard `queue.Queue`. It allows:
  - Querying the active queue list (`GET /api/queue`).
  - Swapping queued jobs and re-ordering (`POST /api/queue/reorder`).
  - Graceful removal upon cancellation.
- **Interactive UI Arrows**: Arrows are displayed next to queued items in the dashboard to move jobs up and down, triggering the API to swap their order on-the-fly.

### 3. Automatic Virtual Environment Setup (`start.py`)
- **Self-Sufficiency**: Running `python3 start.py` automatically checks for a local virtual environment (`venv/` or `.venv/`).
- **Auto-Provisioning**: If no virtual environment is found, `start.py` will initialize one in `venv/`, upgrade `pip`, install all dependencies listed in `requirements.txt`, mark the Real-ESRGAN binary as executable (`chmod +x`), and create necessary folders (`uploads/`, `outputs/`, `temp/`).
- **Seamless Launch**: It then automatically launches the server process using the newly created environment interpreter.

### 4. Advanced AI Enhancements
- **AI Face Restoration (GFPGAN)**: Runs as a python subprocess invoking `gfpgan.inference_gfpgan` to restore and clear up human faces in low-resolution video frames. Results are copied directly back into the frame output folder before motion interpolation and final video assembly.
- **AI Frame Interpolation (RIFE)**: Runs using the `rife-ncnn-vulkan` binary (expected in `rife-bin/`). Smooths motion by generating and inserting intermediate frames, doubling the framerate. Falls back to FFmpeg's `minterpolate` optical flow filter if the Vulkan binary is not present.
- **AI Audio Denoising (RNNoise)**: Transports and filters audio using the deep-learning-based `arnnoise` FFmpeg filter, eliminating background noise from output tracks during assembly.