readme and ui finalization

Author: TalhaMudo

README.md

@@ -1,5 +1,5 @@
 Crawler Project
-
+Github link: https://github.com/TalhaMudo/The-Great-Web-Heist-of-Talha
 This project implements a single-machine web crawler and keyword search engine with a live dashboard. It is designed to satisfy the assignment requirements around:
 
 - indexing from an origin URL up to depth `k`,
@@ -65,7 +65,7 @@ The dashboard allows:
 ### Index/Search
 
 - `POST /index`  
-  Starts a new crawl job with origin URL, depth `k`, and optional initial rate limit.
+  Starts a new crawl job with origin URL, depth `k`, optional `max_urls_to_visit`, and optional initial rate limit.
 
 - `GET /search?query=...&limit=...`  
   Returns relevant indexed URLs.

crawler.db

@@ -89,9 +89,9 @@ type EmbeddingStatus = {
 export const App: React.FC = () => {
   const [viewMode, setViewMode] = useState<"crawler" | "search" | "embeddings">("crawler");
   const [origin, setOrigin] = useState("");
-  const [depth, setDepth] = useState(2);
+  const [depthInput, setDepthInput] = useState("2");
   const [maxUrlsToVisit, setMaxUrlsToVisit] = useState("500");
-  const [rateLimit, setRateLimit] = useState(1.0);
+  const [rateLimitInput, setRateLimitInput] = useState("1.0");
   const [currentJobId, setCurrentJobId] = useState<string | null>(null);
   const [selectedJobId, setSelectedJobId] = useState<string | null>(null);
   const [selectedJob, setSelectedJob] = useState<JobDetail | null>(null);
@@ -184,15 +184,36 @@ export const App: React.FC = () => {
   const startIndex = async () => {
     setError(null);
     try {
+      const trimmedDepth = depthInput.trim();
+      const parsedDepth = Number(trimmedDepth);
+      if (!trimmedDepth || !Number.isInteger(parsedDepth) || parsedDepth < 0 || parsedDepth > 8) {
+        throw new Error("Max depth (k) must be an integer between 0 and 8.");
+      }
+
+      const trimmedRate = rateLimitInput.trim();
+      const parsedRateLimit = Number(trimmedRate);
+      if (!trimmedRate || !Number.isFinite(parsedRateLimit) || parsedRateLimit <= 0) {
+        throw new Error("Crawler speed (req/s) must be a positive number.");
+      }
+
+      const trimmedMaxUrls = maxUrlsToVisit.trim();
+      const parsedMaxUrls = trimmedMaxUrls ? Number(trimmedMaxUrls) : null;
+      if (
+        trimmedMaxUrls &&
+        (!Number.isInteger(parsedMaxUrls) || parsedMaxUrls === null || parsedMaxUrls <= 0)
+      ) {
+        throw new Error("Max URLs to Visit must be a positive integer.");
+      }
+
       setIsIndexing(true);
       const res = await fetch("/index", {
         method: "POST",
         headers: { "Content-Type": "application/json" },
         body: JSON.stringify({
           origin,
-          k: depth,
-          max_urls_to_visit: maxUrlsToVisit ? Number(maxUrlsToVisit) : null,
-          rate_limit_per_sec: rateLimit,
+          k: parsedDepth,
+          max_urls_to_visit: parsedMaxUrls,
+          rate_limit_per_sec: parsedRateLimit,
         }),
       });
       if (!res.ok) {
@@ -401,7 +422,7 @@ export const App: React.FC = () => {
       </div>
       {!query && (
         <p className="hint">
-          Enter a query to search. Your crawled and embedded pages are ready, but search needs at least one keyword.
+          Enter a query to search.
         </p>
       )}
       <div className="dual-results-grid">
@@ -444,7 +465,7 @@ export const App: React.FC = () => {
           <h3>Semantic Search</h3>
           <div className="results">
             {semanticResults.length === 0 ? (
-              <p className="hint">No semantic results yet. Run the embedding engine first.</p>
+              <p className="hint">No semantic results yet. Run the embedding engine first if not.</p>
             ) : (
               <table className="search-table">
                 <thead>
@@ -623,8 +644,8 @@ export const App: React.FC = () => {
                 type="number"
                 min={0}
                 max={8}
-                value={depth}
-                onChange={(e) => setDepth(Number(e.target.value))}
+                value={depthInput}
+                onChange={(e) => setDepthInput(e.target.value)}
               />
             </label>
             <label className="field">
@@ -644,8 +665,8 @@ export const App: React.FC = () => {
                 type="number"
                 min={0.1}
                 step={0.1}
-                value={rateLimit}
-                onChange={(e) => setRateLimit(Number(e.target.value))}
+                value={rateLimitInput}
+                onChange={(e) => setRateLimitInput(e.target.value)}
               />
             </label>
             <button onClick={startIndex} disabled={!origin || isIndexing}>

frontend/src/App.tsx

@@ -19,7 +19,7 @@ Building search engines is usually out of reach for a single engineer, but AI-as
 
 - Internet-scale crawling, cross-machine distribution, or strict guarantees about completeness of the web graph.
 - Production-grade search relevance (no BM25, no learning-to-rank; a simple token-based heuristic is sufficient).
-- Full politeness policies (robots.txt, per-domain rate scheduling); we only implement a global rate limiter.
+- Full politeness policies (robots.txt, per-domain rate scheduling); the current implementation focuses on global queue limiting plus per-job rate controls.
 - Strong security hardening (sandboxing, HTML sanitization) beyond what’s necessary for a localhost classroom demo.
 
 #### User stories
@@ -30,15 +30,15 @@ Building search engines is usually out of reach for a single engineer, but AI-as
 - **As a student**, I can monitor the dashboard to understand how queue depth, processed URL count, and backpressure state change over time as the crawler runs.
 - **As a student**, I can restart the backend and immediately search previously indexed pages because they were persisted locally.
 - **As a student**, I can open an Embeddings page, tune embedding speed/max pages, and monitor `% embedded` in real time.
-- **As an instructor**, I can point at the dashboard and code to explain concurrency (worker pool, queue), backpressure (bounded queue), and uniqueness (visited set).
+- **As an instructor**, I can point at the dashboard and code to explain concurrency (worker pool, queue), backpressure (global queue limit), and uniqueness (visited set).
 
 #### Functional requirements
 
 - **Index endpoint**
   - `POST /index` accepts `{ origin: string, k: number }`.
   - Spawns an asynchronous crawl job identified by `job_id`.
   - Enforces depth limit `k` and “never crawl the same URL twice per job” using a normalized URL visited set.
-  - Applies global backpressure using a bounded `asyncio.Queue` and a fixed worker count; when the queue is full, link discovery pauses and `backpressure_state` transitions to `"queue_full"`.
+  - Applies global backpressure using a shared global queue limit and a fixed worker count; when the limit is reached, link discovery pauses and `backpressure_state` transitions to `"queue_full"`.
 
 - **Search endpoint**
   - `GET /search?query=...` returns a list of results, each containing `(relevant_url, origin_url, depth, score, title)`.