product_prd.md

Product Requirement Document – The Great Web Heist of Talha

Problem statement

Building search engines is usually out of reach for a single engineer, but AI-assisted tooling and modern developer workflows make it feasible to prototype a small-scale version in a day. For the BLG 483E AI Aided Computer Engineering course, we need a single-machine crawler and search engine that can be steered and explained by a human, not just generated by an opaque library. The system should recursively crawl from a starting URL, index discovered pages, and expose a simple search interface and dashboard that make the architecture and runtime behavior observable for teaching.

Goals

• Index web content starting from an origin URL up to depth k, never visiting the same page twice for a job.
• Search previously indexed pages using a simple text-based relevancy heuristic, returning triples (relevant_url, origin_url, depth).
• Semantic search support over crawled pages using embedding vectors (all-MiniLM-L6-v2) while preserving the existing classic search.
• Live behavior: allow search while indexing is running so new pages appear in results as soon as they are processed.
• Visibility: provide a React dashboard that exposes crawl progress, queue depth, backpressure state, and job list.
• Local persistence: store a lightweight snapshot of pages and jobs on disk so that search can resume after a restart.
• Embedding operations: provide a separate UI mode to control a single embedding engine (start/pause, speed, max-page scope) and reset embeddings when needed.
• AI-friendly design: keep the crawler and indexer logic explicit and understandable, avoiding “black box” scraping libraries.

Non-goals

• Internet-scale crawling, cross-machine distribution, or strict guarantees about completeness of the web graph.
• Production-grade search relevance (no BM25, no learning-to-rank; a simple token-based heuristic is sufficient).
• Full politeness policies (robots.txt, per-domain rate scheduling); the current implementation focuses on global queue limiting plus per-job rate controls.
• Strong security hardening (sandboxing, HTML sanitization) beyond what’s necessary for a localhost classroom demo.

User stories

• As a student, I can enter an origin URL and depth k in the UI and start an indexing job, so I can see how the crawler explores the web graph.
• As a student, I can run queries in the search panel while a crawl is in progress and see newly indexed pages appear without restarting the search service.
• As a student, I can compare classical and semantic results for the same query in side-by-side tables.
• As a student, I can monitor the dashboard to understand how queue depth, processed URL count, and backpressure state change over time as the crawler runs.
• As a student, I can restart the backend and immediately search previously indexed pages because they were persisted locally.
• As a student, I can open an Embeddings page, tune embedding speed/max pages, and monitor % embedded in real time.
• As an instructor, I can point at the dashboard and code to explain concurrency (worker pool, queue), backpressure (global queue limit), and uniqueness (visited set).

Functional requirements

• Index endpoint

  • POST /index accepts { origin: string, k: number }.
  • Spawns an asynchronous crawl job identified by job_id.
  • Enforces depth limit k and “never crawl the same URL twice per job” using a normalized URL visited set.
  • Applies global backpressure using a shared global queue limit and a fixed worker count; when the limit is reached, link discovery pauses and backpressure_state transitions to "queue_full".

• Search endpoint

  • GET /search?query=... returns a list of results, each containing (relevant_url, origin_url, depth, score, title).
  • Uses an in-memory inverted index built from page titles and body text tokens.
  • Scores pages by aggregate term frequency for query tokens and sorts descending.
  • Reads directly from the current in-memory index so queries reflect pages as soon as they are indexed.
  • GET /search/semantic?query=... returns semantic matches from persisted page embeddings with similarity scores.

• Embedding engine endpoints

  • GET /embeddings/status returns embedding progress and current engine state.
  • POST /embeddings/start starts (or continues) embedding missing pages with { rate_limit_per_sec, max_pages }.
  • POST /embeddings/pause pauses active embedding work.
  • POST /embeddings/rate-limit updates embedding speed during runtime.
  • POST /embeddings/clear deletes all stored embeddings after explicit user confirmation in UI.

• Status/metrics endpoints

  • GET /jobs/{job_id} returns job metadata and per-job statistics (status, processed URLs, queue depth estimate, backpressure state).
  • GET /metrics aggregates global stats across jobs: processed_urls, queued_urls, queue_max, backpressure_state, active_workers, and a compact jobs_summary list.

• React UI

  • Index control panel: origin+depth form, “Start Indexing” button, and active job id display.
  • Search panel: query input with side-by-side classical and semantic result tables.
  • Embeddings panel: manual embedding controls (start/pause, speed, max-page limit, clear-all reset) and live progress cards.
  • System dashboard: live metrics cards and jobs table, auto-refreshing every few seconds.

• Persistence

  • Store pages in a local SQLite database (via Python sqlite3 stdlib) with columns (url, origin_url, depth, title, body_snippet).
  • Store job metadata (id, origin_url, max_depth, created_at, status) in the same DB.
  • Store semantic vectors in page_embeddings; progress is derived from pages vs page_embeddings.
  • On startup, rebuild the in-memory index from stored pages and repopulate the job list from stored jobs.

Constraints

• Backend implemented in Python; crawling/indexing core remains on standard-library modules (urllib, html.parser, asyncio, sqlite3) and semantic search adds a model dependency (sentence-transformers).
• React frontend implemented with Vite + React + TypeScript, running on localhost, proxying API calls to the Python backend.
• Single-process, single-machine design: concurrency is implemented with asyncio tasks and a bounded queue, not multiple OS processes or machines.