embedding v1

Author: TalhaMudo

README.md

@@ -1,4 +1,4 @@
-# Brightwave-Style Crawler Project
+Crawler Project
 
 This project implements a single-machine web crawler and keyword search engine with a live dashboard. It is designed to satisfy the assignment requirements around:
 
@@ -12,7 +12,7 @@ This project implements a single-machine web crawler and keyword search engine w
 The system has two major parts:
 
 - **Crawler service**: creates and runs crawl jobs, supports pause/resume, tracks queue and worker state, and persists job state.
-- **Search service**: indexes crawled pages and answers queries with relevant URLs and metadata.
+- **Search service**: keeps classic keyword ranking and semantic vector ranking side by side, and answers queries with relevant URLs and metadata.
 
 ## What The System Does
 
@@ -26,11 +26,12 @@ When you submit an indexing request (`origin`, `k`), the backend creates a crawl
 
 ### 2) Search
 
-Search uses an in-memory inverted index built from page title and body text.
+Search exposes two modes over indexed pages: classic term search and semantic vector search.
 
-- Tokenization is regex-based and case-insensitive.
+- Classic tokenization is regex-based and case-insensitive.
+- Semantic search uses `all-MiniLM-L6-v2` embeddings generated from crawled page snapshots.
 - Results include `relevant_url`, `origin_url`, and `depth` (plus score/title for UI visibility).
-- Search is available while crawlers are still running, so newly indexed pages appear as they are discovered.
+- Both search modes are available while crawlers are still running, so newly indexed pages appear as they are discovered.
 - `I'm Feeling Lucky` picks a **random** result from matches and opens it.
 
 ### 3) Runtime Controls and Visibility
@@ -41,8 +42,10 @@ The dashboard allows:
 - pausing and resuming existing jobs,
 - changing **global queue limit** across all jobs,
 - changing **per-job request rate** live from each job card,
+- starting, pausing, and resuming **embedding jobs** with configurable embed speed and max-page limit,
 - inspecting detailed job state (frontier preview, counters, recent events),
-- viewing global metrics (processed/discovered/duplicates/failed, queue pressure, workers).
+- viewing global metrics (processed/discovered/duplicates/failed, queue pressure, workers),
+- viewing embedding progress metrics (embedded/remaining/failed and `% embedded`).
 
 ## Architecture Summary
 
@@ -52,7 +55,9 @@ The dashboard allows:
   - `pages` table for indexed page snapshots,
   - `jobs` table for job metadata and counters,
   - `job_visited` and `job_frontier` for resumable crawl state,
-  - `job_events` for job timeline details.
+  - `job_events` for job timeline details,
+  - `page_embeddings` for semantic vectors,
+  - `embedding_jobs` for resumable embedding job state.
 
 ## API Endpoints
 
@@ -64,6 +69,9 @@ The dashboard allows:
 - `GET /search?query=...&limit=...`  
   Returns relevant indexed URLs.
 
+- `GET /search/semantic?query=...&limit=...`  
+  Returns semantic nearest-neighbor matches over embedded pages.
+
 ### Job Control and Details
 
 - `GET /jobs/{job_id}`  
@@ -86,6 +94,26 @@ The dashboard allows:
 - `POST /settings/queue-limit`  
   Updates the **global** queue limit shared by all jobs.
 
+### Embeddings
+
+- `POST /embeddings/jobs/start`  
+  Starts a manual embedding job with configurable speed and max-page scope.
+
+- `GET /embeddings/jobs`  
+  Lists embedding jobs and progress.
+
+- `GET /embeddings/jobs/{job_id}`  
+  Returns full embedding job status and counters.
+
+- `POST /embeddings/jobs/{job_id}/pause`  
+  Pauses a running embedding job.
+
+- `POST /embeddings/jobs/{job_id}/resume`  
+  Resumes a paused embedding job.
+
+- `POST /embeddings/jobs/{job_id}/rate-limit`  
+  Updates embedding speed (`pages/s`) for a job.
+
 ## Backpressure and Load Management
 
 The crawler enforces load control using:
@@ -124,17 +152,18 @@ Open the printed Vite URL (usually `http://127.0.0.1:5173`).
 3. Start a second crawler to demonstrate concurrent jobs.
 4. Update global queue limit and per-job req/s live.
 5. Pause and resume one job.
-6. Run Search and `I'm Feeling Lucky`.
-7. Click a job card and inspect frontier preview + event timeline.
+6. Open Embeddings mode and start a manual embedding run with speed/limit.
+7. Run Search and compare Classical vs Semantic result tables.
+8. Click a crawler job card and inspect frontier preview + event timeline.
 
 ## Limitations (Current Scope)
 
-- Search is term-frequency based (no semantic embeddings).
+- Semantic quality depends on currently stored page snippets and embedding coverage.
 - Single-process deployment (no horizontal scaling).
 - No robots.txt policy enforcement yet.
 - No hard per-domain crawl budget yet.
 
-## Production Next Steps (1-2 Paragraphs)
+## Production Next Steps
 
 To productionize this system, the first step is to separate responsibilities into independent services: crawl scheduler, crawl workers, indexing pipeline, and query API. The current asyncio loop can evolve into a worker template that pulls tasks from a durable queue and writes normalized page documents to persistent storage. The in-memory inverted index should be replaced with a dedicated search layer (for example BM25-capable engine) to improve ranking quality, scalability, and operational resilience.
 

backend/__pycache__/app.cpython-314.pyc

@@ -11,6 +11,7 @@
 from .crawler import crawler_service
 from .indexer import index_service
 from .models import CrawlJob, JobStatus, summarize_jobs
+from .semantic_index import semantic_index_service
 from .storage import init_db, load_job_events, load_jobs, load_pages
 
 
@@ -37,6 +38,29 @@ class SearchResponse(BaseModel):
     results: List[SearchResult]
 
 
+class EmbeddingStartRequest(BaseModel):
+    rate_limit_per_sec: float = 1.0
+    max_pages: int | None = None
+
+
+class EmbeddingRateLimitRequest(BaseModel):
+    rate_limit_per_sec: float
+
+
+class EmbeddingStatusResponse(BaseModel):
+    updated_at: str
+    status: str
+    model_name: str
+    rate_limit_per_sec: float
+    max_pages: int | None = None
+    total_pages: int
+    embedded_pages: int
+    failed_pages: int
+    remaining_pages: int
+    progress_percent: float
+    error_message: str | None = None
+
+
 class JobEventResponse(BaseModel):
     created_at: str
     level: str
@@ -122,6 +146,71 @@ async def search(query: str, limit: int | None = None) -> SearchResponse:
     return SearchResponse(results=results)
 
 
+@app.get("/search/semantic", response_model=SearchResponse)
+async def semantic_search(query: str, limit: int | None = None) -> SearchResponse:
+    raw_results = await semantic_index_service.search(query, limit=limit)
+    results = [
+        SearchResult(
+            relevant_url=url,
+            origin_url=origin_url,
+            depth=depth,
+            score=score,
+            title=title or None,
+        )
+        for url, origin_url, depth, score, title in raw_results
+    ]
+    return SearchResponse(results=results)
+
+
+async def _serialize_embedding_status() -> EmbeddingStatusResponse:
+    status = await semantic_index_service.get_engine_status()
+    return EmbeddingStatusResponse(
+        updated_at=status.updated_at.isoformat(),
+        status=status.status,
+        model_name=status.model_name,
+        rate_limit_per_sec=status.rate_limit_per_sec,
+        max_pages=status.max_pages,
+        total_pages=status.total_pages,
+        embedded_pages=status.embedded_pages,
+        failed_pages=status.failed_pages,
+        remaining_pages=status.remaining_pages,
+        progress_percent=status.progress_percent,
+        error_message=status.error_message,
+    )
+
+
+@app.get("/embeddings/status", response_model=EmbeddingStatusResponse)
+async def embedding_status() -> EmbeddingStatusResponse:
+    return await _serialize_embedding_status()
+
+
+@app.post("/embeddings/start", response_model=EmbeddingStatusResponse)
+async def start_embedding_engine(request: EmbeddingStartRequest) -> EmbeddingStatusResponse:
+    if request.rate_limit_per_sec <= 0:
+        raise HTTPException(status_code=400, detail="rate_limit_per_sec must be positive")
+    if request.max_pages is not None and request.max_pages <= 0:
+        raise HTTPException(status_code=400, detail="max_pages must be positive")
+    await semantic_index_service.start_engine(
+        rate_limit_per_sec=request.rate_limit_per_sec,
+        max_pages=request.max_pages,
+    )
+    return await _serialize_embedding_status()
+
+
+@app.post("/embeddings/pause", response_model=EmbeddingStatusResponse)
+async def pause_embedding_engine() -> EmbeddingStatusResponse:
+    await semantic_index_service.pause_engine()
+    return await _serialize_embedding_status()
+
+
+@app.post("/embeddings/rate-limit", response_model=EmbeddingStatusResponse)
+async def update_embedding_engine_rate_limit(request: EmbeddingRateLimitRequest) -> EmbeddingStatusResponse:
+    if request.rate_limit_per_sec <= 0:
+        raise HTTPException(status_code=400, detail="rate_limit_per_sec must be positive")
+    await semantic_index_service.update_rate_limit(request.rate_limit_per_sec)
+    return await _serialize_embedding_status()
+
+
 def _serialize_job(job_id: str) -> JobDetailResponse:
     ctx = crawler_service.get_job_context(job_id)
     if ctx is None:
@@ -243,4 +332,5 @@ async def on_startup() -> None:
 
     for job in load_jobs():
         crawler_service.register_job(job)
+    await semantic_index_service.initialize()
 

backend/__pycache__/models.cpython-314.pyc

@@ -41,6 +41,22 @@ class CrawlJob:
     stats: CrawlStats = field(default_factory=CrawlStats)
 
 
+@dataclass
+class EmbeddingJobRecord:
+    id: str
+    created_at: datetime
+    updated_at: datetime
+    status: JobStatus
+    model_name: str
+    rate_limit_per_sec: float = 1.0
+    max_pages: int | None = None
+    total_pages: int = 0
+    embedded_pages: int = 0
+    failed_pages: int = 0
+    error_message: Optional[str] = None
+    pending_urls_json: str = "[]"
+
+
 PageRecord = Tuple[str, str, int]
 IndexEntry = Tuple[str, str, int, float]