65 lines
4.9 KiB
Markdown
65 lines
4.9 KiB
Markdown
# 🧠 AI Video Upscaler - Project Context & Reference
|
|
|
|
Welcome! This document acts as a memory reference folder for AI agents working on this project. It details the system architecture, code organization, pipeline stages, and custom feature implementations (Resumption & Queue Management).
|
|
|
|
---
|
|
|
|
## 🏗️ System Architecture
|
|
|
|
The AI Video Upscaler is a full-stack, single-user application designed to upscale videos locally using Vulkan GPU-accelerated AI models.
|
|
|
|
### Technology Stack
|
|
1. **FastAPI (Python Backend)**: Handles API endpoints, serving static assets, WebSockets, background threads, and orchestrates subprocess execution.
|
|
2. **Real-ESRGAN ncnn-vulkan (AI Engine)**: A compiled C++ Vulkan binary located in `realesrgan-bin/realesrgan-ncnn-vulkan`.
|
|
3. **FFmpeg & FFprobe (Multimedia Toolkit)**: Splitting videos into high-quality JPEG frames, querying metadata, extracting audio streams, and re-muxing audio/subtitles back into the upscaled output.
|
|
4. **Vanilla HTML/CSS/JS (Frontend)**: Cyberpunk-themed web dashboard with real-time progress logging, side-by-side comparative preview, and queue management.
|
|
|
|
---
|
|
|
|
## 📁 Key File Map
|
|
|
|
- **`app/main.py`**: API route definitions, WebSocket orchestrator, custom thread-safe job queue, and JSON database state preservation (`jobs.json`).
|
|
- **`app/upscaler.py`**: Core pipeline orchestrator (`run_upscale_pipeline`). Executes external FFmpeg and Real-ESRGAN commands. Contains the resume logic.
|
|
- **`static/index.html`**, **`static/styles.css`**, **`static/app.js`**: Frontend interface, WebSocket handlers, interactive comparison slider, zoom/pan tool, and queue reordering actions.
|
|
- **`jobs.json`**: Persisted file storing details of all jobs (automatically generated at root).
|
|
|
|
---
|
|
|
|
## 🔄 Upscale Pipeline Lifecycle
|
|
|
|
A typical upscale job follows these sequential steps:
|
|
1. **Queued (`queued`)**: Added to the FIFO queue.
|
|
2. **Analyzing (`analyzing`)**: GPU lock acquired. Querying stream specs with `ffprobe`.
|
|
3. **Extracting (`extracting`)**: FFmpeg extracts video frames into `temp/<job_id>/input_frames/frame_%08d.jpg`.
|
|
4. **Upscaling (`upscaling`)**: Real-ESRGAN binary upscales images to `temp/<job_id>/output_frames/frame_%08d.jpg`.
|
|
5. **Assembling (`assembling`)**: FFmpeg merges upscaled frames with original audio/subtitles.
|
|
6. **Completed (`completed`)** / **Failed (`failed`)** / **Cancelled (`cancelled`)** / **Interrupted (`interrupted`)**.
|
|
|
|
---
|
|
|
|
## ⚡ Custom Enhancements
|
|
|
|
### 1. Job Resumption (`interrupted` status)
|
|
- **Persistency**: The status of all jobs is stored in `jobs.json` at the root. On application restart, any incomplete/active job is automatically loaded in the `interrupted` status.
|
|
- **Skip Extraction**: On resume, if `temp/<job_id>/input_frames` contains frames, the extraction step is bypassed.
|
|
- **Incremental Upscaling**: The upscaler inspects `output_frames/` and checks for already upscaled frames. It removes corresponding files from `input_frames/`, meaning the AI model only processes the remaining un-upscaled frames.
|
|
- **Fast Assembly**: If all frames are already upscaled, it skips the upscaling step entirely and directly runs the FFmpeg reassembly.
|
|
|
|
### 2. Queue Management (Cancel & Reorder)
|
|
- **Custom Queue (`CustomJobQueue`)**: A thread-safe, list-backed FIFO queue replacing the standard `queue.Queue`. It allows:
|
|
- Querying the active queue list (`GET /api/queue`).
|
|
- Swapping queued jobs and re-ordering (`POST /api/queue/reorder`).
|
|
- Graceful removal upon cancellation.
|
|
- **Interactive UI Arrows**: Arrows are displayed next to queued items in the dashboard to move jobs up and down, triggering the API to swap their order on-the-fly.
|
|
|
|
### 3. Automatic Virtual Environment Setup (`start.py`)
|
|
- **Self-Sufficiency**: Running `python3 start.py` automatically checks for a local virtual environment (`venv/` or `.venv/`).
|
|
- **Auto-Provisioning**: If no virtual environment is found, `start.py` will initialize one in `venv/`, upgrade `pip`, install all dependencies listed in `requirements.txt`, mark the Real-ESRGAN binary as executable (`chmod +x`), and create necessary folders (`uploads/`, `outputs/`, `temp/`).
|
|
- **Seamless Launch**: It then automatically launches the server process using the newly created environment interpreter.
|
|
|
|
### 4. Advanced AI Enhancements
|
|
- **AI Face Restoration (GFPGAN)**: Runs as a python subprocess invoking `gfpgan.inference_gfpgan` to restore and clear up human faces in low-resolution video frames. Results are copied directly back into the frame output folder before motion interpolation and final video assembly.
|
|
- **AI Frame Interpolation (RIFE)**: Runs using the `rife-ncnn-vulkan` binary (expected in `rife-bin/`). Smooths motion by generating and inserting intermediate frames, doubling the framerate. Falls back to FFmpeg's `minterpolate` optical flow filter if the Vulkan binary is not present.
|
|
- **AI Audio Denoising (RNNoise)**: Transports and filters audio using the deep-learning-based `arnnoise` FFmpeg filter, eliminating background noise from output tracks during assembly.
|
|
|