diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000000..0efcb457d6 --- /dev/null +++ b/.gitignore @@ -0,0 +1,4 @@ +bin/ +.DS_Store +*.swp +*~ diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000000..5817abab68 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,220 @@ +# Repository Guidelines + +## Project Overview + +Code Search is a high-performance source code search engine based on an n-gram (trigram) inverted index. Originally created by Russ Cox for multi-gigabyte code repositories, Code Search decomposes regular expressions into boolean queries over compressed posting lists, filtering hundreds of thousands of candidate files down to exact matches in milliseconds. + +The repository provides a modern unified CLI tool (`codesearch`, with `cs` alias) alongside legacy standalone tools (`cindex`, `csearch`, `cgrep`, `csweb`), backed by zero external dependencies and utilizing the Go standard library with optional experimental SIMD vectorization. + +--- + +## Architecture & Data Flow + +Code Search is organized into three core packages and two CLI command layers: + +``` ++-------------------------------------------------------------------------+ +| CLI Layer | +| cmd/codesearch (unified) | cmd/cindex | cmd/csearch | cmd/cgrep| cmd/csweb | ++------------------------------------+------------------------------------+ + | + +--------------------+--------------------+ + | | + v v ++-------------------------------+ +-------------------------------+ +| package index | | package regexp | +| - Trigram extraction & index | | - Custom DFA/NFA regexp | +| - Binary on-disk serialization| | - RE2 syntax compilation | +| - Prefix path compression |<--------| - Line/context grep engine | +| - Query planning & posting | | - SIMD newline counting | +| list boolean intersections | +---------------+---------------+ ++---------------+---------------+ | + | | + v v ++-------------------------------+ +-------------------------------+ +| package sparse |<--------+ package sparse | +| - O(1) uint32 sparse sets | | - DFA/NFA instruction queues | +| - Dense iteration & fast resets| +-------------------------------+ ++-------------------------------+ +``` + +### Data Pipelines + +1. **Indexing Pipeline (`index/write.go`, `index/path.go`, `index/delta.go`)**: + - **Ingestion**: Traverses target directory roots, opening files with reusable 1 MB buffers (`inbuf`). + - **File Filtering**: Scans chunks for NUL bytes (`scanASCIIChunk`) to reject binary files immediately. Rejects files exceeding 1 GB (`maxFileLen`), lines longer than 2,000 bytes (`maxLineLen`), or files with invalid UTF-8 sequences. + - **Trigram Extraction**: Slides a 3-byte window over the stream to produce 24-bit trigram keys (`(c1<<16) | (c2<<8) | c3`), tracked per file in a deduplicated `sparse.Set`. + - **Sorting & Flushing**: Stores `postEntry` records (24-bit trigram + 40-bit file ID) in memory up to 64 MB. Flushes to temporary files using a two-pass 12-bit radix sort (`sortPost`). + - **Final Assembly & Serialization**: Merges flushed temporary chunks using a multiway tournament heap (`postHeap`). Writes posting lists with bit-level Elias $\gamma$ (gamma) delta encoding in 256-byte blocks (`postBlockSize`), path lists with prefix compression (blocks of 16 paths), and fixed 80-byte trailer (`csearch trlr 2\n`). + +2. **Search Pipeline (`cmd/codesearch/search.go`, `index/regexp.go`, `regexp/match.go`)**: + - **AST Parsing**: Parses regex patterns using RE2 syntax rules (`regexp/syntax.Parse`). + - **Query Planning**: Analyzes AST concatenations and alternations to derive a conservative boolean query tree of required trigrams (`QAnd`, `QOr`, `QAll`). Simplifies queries using boolean absorption and common factor extraction. + - **Candidate Selection**: Evaluates the query tree against the index. Intersects posting lists (`PostingAnd`) using galloping multi-stride skips (`i += 64`, `i += 16`, `i += 4`, `i++`). + - **Exact Matching**: Resolves file paths from candidate IDs via mmap (`ix.Name(fileid)`), streams content through `regexp.Grep`, and executes regex matches using a lazy DFA with NFA instruction queue fallbacks (`sparse.Set`). + +--- + +## Key Directories + +- **`cmd/codesearch/`**: Unified CLI application containing the router (`main.go`), subcommand runners (`index.go`, `search.go`, `grep.go`, `web.go`), and comprehensive E2E tests (`codesearch_test.go`). +- **`cmd/cindex/`**, **`cmd/csearch/`**, **`cmd/cgrep/`**, **`cmd/csweb/`**: Backward-compatible standalone CLI wrappers for legacy tool workflows. +- **`index/`**: Core indexing subsystem: on-disk binary format (v1 & v2), mmap wrappers (`mmap_*.go`), Elias $\gamma$-encoded posting lists (`delta.go`), path compression (`path.go`), query planner (`regexp.go`), index merger (`merge.go`), and SIMD/scalar chunk scanners (`scan_*.go`). +- **`regexp/`**: High-performance DFA/NFA regular expression engine: byte-code instruction compiler (`regexp.go`), stream matcher and context printer (`match.go`), UTF-8 decoders (`utf.go`), and SIMD/scalar newline counters (`nl_*.go`). +- **`sparse/`**: Dense/sparse paired set providing $O(1)$ operations over `uint32` values with zero-allocation iteration (`Dense()`). +- **`docs/`**: Architectural specifications (`specs/`) and historical implementation plans (`plans/`). +- **`skills/codesearch/`**: Agent skill metadata and operational instructions for AI assistants. + +--- + +## Development Commands + +### Build Targets + +```bash +# Standard build (outputs bin/codesearch, bin/cs, and legacy binaries) +make build + +# Build with experimental SIMD vectorization enabled +make build-simd + +# Install unified binary to $GOPATH/bin or $GOBIN +go install ./cmd/codesearch + +# Clean build outputs +make clean +``` + +### Testing + +```bash +# Run all tests across packages +make test + +# Run tests with SIMD enabled +make test-simd + +# Run without test cache +go test -count=1 ./... + +# Run a specific package test suite +go test -v ./cmd/codesearch +go test -v ./index +go test -v ./regexp +``` + +### Benchmarks + +```bash +# Benchmark newline counting (scalar vs SIMD) +go test -bench=BenchmarkCountNL ./regexp +GOEXPERIMENT=simd go test -bench=BenchmarkCountNL ./regexp + +# Benchmark ASCII scanning and document ingestion +go test -bench=BenchmarkScanASCIIChunk -benchmem ./index +GOEXPERIMENT=simd go test -bench=BenchmarkScanASCIIChunk -benchmem ./index +go test -bench=BenchmarkIndexWriterAdd -benchmem ./index +``` + +### Static Analysis + +```bash +# Check code with Go vet (standard and SIMD builds) +go vet ./... +GOEXPERIMENT=simd go vet ./... +``` + +--- + +## Code Conventions & Common Patterns + +### Formatting & Style +- Standard Go formatting via `gofmt`. +- Terse, idiomatic Go naming conventions: + - `ix` for `*Index` or `*IndexWriter` + - `g` for `*Grep` + - `re` for `*Regexp` + - `tv` for 24-bit trigram values + - `fileid` for integer file numbers +- Use standard BSD license headers on new Go source files. + +### Subcommand Runner Pattern +All CLI subcommands implement isolated runner functions to enable fast, in-process testing without spawning subprocesses: + +```go +type Runner func(args []string, stdout, stderr io.Writer) int +// or for grep: +type GrepRunner func(args []string, stdout, stderr io.Writer, stdin io.Reader) int +``` + +- Each runner initializes a private `flag.NewFlagSet("subcommand", flag.ContinueOnError)`. +- Flag output is bound explicitly: `flags.SetOutput(stderr)`. +- Static usage messages must use `fmt.Fprint(os.Stderr, usage)` instead of `fmt.Fprintf` to prevent `go vet` warnings. +- Subcommand runners return exit codes (`0`, `1`, `2`) rather than invoking `os.Exit()`. + +### Exit Code Conventions +- `0`: Success (or regular expression matches found). +- `1`: No matches found (for `search` and `grep`). +- `2`: Syntax errors, invalid command flags, missing index file, or fatal I/O errors. + +### Flag Parsing Constraints +- Standard library `flag` is used throughout. **Combined short flags are not supported**: `-i -n` is valid; `-in` is rejected as an undefined flag. + +### Thread-Safety & Concurrency +- `index.Index` is **read-only and safe for concurrent use** across goroutines. +- `index.IndexWriter` and `regexp.Regexp` are **not thread-safe**. `Regexp` mutates internal DFA states and instruction queues during execution; allocate separate `Regexp` instances per worker thread. + +### Error & Corruption Handling +- Corrupted indices invoke `ix.corrupt()`, which panics if `panicOnCorrupt = true` (standard in test suites) or logs a fatal error in production. +- Use `index.Check()` to validate index structure and block boundaries before destructive operations. + +--- + +## Important Files + +| File | Purpose | +|---|---| +| `cmd/codesearch/main.go` | Entrypoint, routing subcommands or falling back to default search | +| `cmd/codesearch/index.go` | Index builder and updater (`codesearch index`) | +| `cmd/codesearch/search.go` | Main query executor (`codesearch search`) | +| `cmd/codesearch/grep.go` | Direct unindexed regex searcher (`codesearch grep`) | +| `cmd/codesearch/web.go` | Local HTTP code search web viewer (`codesearch web`) | +| `index/read.go` | Mmap index loader, posting list reader, `PrintStatsTo()` | +| `index/write.go` | Document ingestion, trigram extraction, 12-bit radix sort, index writer | +| `index/regexp.go` | RE2 AST analysis and boolean trigram query construction | +| `index/delta.go` | Elias $\gamma$ (gamma) variable-length bit stream encoder/decoder | +| `index/merge.go` | Two-way index merger with path shadowing and file ID translation | +| `regexp/regexp.go` | Public regexp compiler and RE2 bytecode program builder | +| `regexp/match.go` | Lazy DFA engine, NFA fallback step queue, Grep stream runner | +| `regexp/nl_simd.go` | SIMD-vectorized newline counter using `simd/archsimd` | +| `index/scan_simd.go` | SIMD-vectorized ASCII/NUL detector using `simd/archsimd` | +| `Makefile` | Build automation for standard and SIMD binary targets | +| `go.mod` | Module definition targeting Go 1.23+ with zero dependencies | + +--- + +## Runtime & Tooling Preferences + +- **Go Toolchain**: Go 1.23+ recommended (works on Go 1.22+; Go 1.26+ required for experimental `simd/archsimd` intrinsics). +- **Dependencies**: Zero external dependencies. Strictly uses the Go standard library and internal packages. +- **Build System**: POSIX `make` and Go standard toolchain (`go build`, `go test`, `go vet`). +- **Operating Systems**: Darwin/macOS, Linux, FreeBSD, OpenBSD, NetBSD, and Windows (custom memory mapping adapters in `index/mmap_*.go`). + +--- + +## Testing & QA + +### Test Architecture +- **E2E CLI Tests (`cmd/codesearch/codesearch_test.go`)**: Uses `runMain(t, args, stdin)` helper to exercise subcommands, flags, exit codes, and stdin streaming in-process. +- **Index Binary Layout Tests (`index/write_test.go`)**: Byte-for-byte golden file comparisons against `trivialIndexV1` and `trivialIndexV2`. +- **Posting Logic Verification (`index/read_test.go`, `index/merge_test.go`)**: Verifies boolean query algebra (`PostingAnd`, `PostingOr`) and index merge stability. +- **SIMD Correctness (`index/scan_test.go`, `regexp/nl_bench_test.go`)**: Exercises vector lane boundaries (0, 15, 16, 17, 31, 32, 33, 4,000+ bytes) in both scalar and SIMD modes. + +### Test Isolation Best Practices +- When writing tests that interact with index files, **always isolate the index path**: + ```go + tempDir := t.TempDir() + indexFile := filepath.Join(tempDir, ".csearchindex") + t.Setenv("CSEARCHINDEX", indexFile) + ``` +- Tests modifying package globals (such as `writeVersion` or `panicOnCorrupt`) must run sequentially without `t.Parallel()`, ensuring state is restored via `defer`. diff --git a/Makefile b/Makefile new file mode 100644 index 0000000000..e771f59753 --- /dev/null +++ b/Makefile @@ -0,0 +1,38 @@ +.PHONY: all build build-simd clean test test-simd check check-simd + +all: build + +build: + @mkdir -p bin + go build -o bin/codesearch ./cmd/codesearch + ln -sf codesearch bin/cs + go build -o bin/cindex ./cmd/cindex + go build -o bin/csearch ./cmd/csearch + go build -o bin/cgrep ./cmd/cgrep + go build -o bin/csweb ./cmd/csweb + +build-simd: + @mkdir -p bin + GOEXPERIMENT=simd go build -o bin/codesearch ./cmd/codesearch + ln -sf codesearch bin/cs + GOEXPERIMENT=simd go build -o bin/cindex ./cmd/cindex + GOEXPERIMENT=simd go build -o bin/csearch ./cmd/csearch + GOEXPERIMENT=simd go build -o bin/cgrep ./cmd/cgrep + GOEXPERIMENT=simd go build -o bin/csweb ./cmd/csweb + +test: + go test -v ./... + +test-simd: + GOEXPERIMENT=simd go test -v ./... + +check: + go test ./... + go vet ./... + +check-simd: + GOEXPERIMENT=simd go test ./... + GOEXPERIMENT=simd go vet ./... + +clean: + rm -rf bin diff --git a/README b/README index 65e41410e0..f1376d25a0 100644 --- a/README +++ b/README @@ -1,16 +1,401 @@ -Code Search is a tool for indexing and then performing -regular expression searches over large bodies of source code. -It is a set of command-line programs written in Go. +# Code Search -For background and an overview of the commands, -see http://swtch.com/~rsc/regexp/regexp4.html. +Code Search is a high-performance tool for indexing and searching large bodies of source code using regular expressions backed by an n-gram (trigram) index. -To install: +Originally authored by Russ Cox for searching multi-gigabyte codebases, Code Search decomposes regular expressions into boolean queries over trigram posting lists. This enables instantaneous candidate file identification, reducing search time over hundreds of thousands of files to milliseconds. - go get github.com/google/codesearch/cmd/... +For architectural and algorithmic background, see [Regular Expression Search with Trigrams](https://swtch.com/~rsc/regexp/regexp4.html). -Use "go get -u" to update an existing installation. +## Quick start -Russ Cox -rsc@swtch.com -June 2015 +With Go 1.23 or later, from this checkout: + +```bash +go install ./cmd/codesearch +codesearch index -git . +codesearch status -git +codesearch search -git --line-number 'func main' +codesearch search -git --fixed-strings 'http.HandleFunc(' +``` + +Run these commands from the repository you want to search after installation. +Set `"git": true` in your configuration to omit `-git` on later commands. +`make build` also provides the short alias at `bin/cs`; `go install` installs +only `codesearch`. Put the installation directory on your `PATH`. + +Use `--help` for command help. Flags precede the pattern and paths; combined +short flags remain unsupported (`-i -n`, not `-in`). To search for a command +name, use `codesearch search index`; for patterns beginning with a dash, use +`codesearch search -- '-pattern'`. + +`--index /path/to/index` selects an index for one invocation and overrides both +`CSEARCHINDEX` and configuration. `codesearch status` reports the selected +index, selection source, roots, modification time, and size statistics. It does +not verify freshness: refresh existing roots with `codesearch index` after +source files change. + +--- + +## How It Works + +1. **Trigram Indexing (`index/`)**: + - Files are parsed into overlapping 3-byte sequences (trigrams). With 256 possible bytes, there are $2^{24}$ (16,777,216) possible trigrams. + - For each trigram, the index stores a compressed posting list recording every file ID that contains that trigram. + - File IDs in posting lists are Elias $\gamma$-encoded and chunked into 256-byte blocks on disk for rapid seek and decode performance. + - Binary files (containing NUL bytes) and non-text files are automatically excluded. + +2. **Query Planning & Search (`regexp/`, `cmd/codesearch/search.go`)**: + - The user's regular expression is parsed into an abstract syntax tree using the RE2 syntax. + - The query planner extracts required trigrams from the syntax tree to build a boolean query tree ($\text{AND}$, $\text{OR}$, or $\text{ALL}$). For example, `func.*main` requires both `"fun"` and `"mai"`. + - The posting lists for those trigrams are intersected to find the small subset of files guaranteed to contain all required trigrams. + - Finally, Code Search runs an exact regular expression search only on matching candidate files. + +3. **Fast Line Counting & Context Extraction**: + - Matches are extracted with byte offsets and line numbers. + - Pre- and post-match context (`-A`, `-B`, `-C`) can be extracted with common prefix indentation preserved. + +--- + +## Building & Installation + +### Standard Build (Go 1.23+) + +Build all binaries into `bin/` using `make`: + +```bash +make build +``` + +This compiles: +- `bin/codesearch` (unified CLI) +- `bin/cs` (symlink to `codesearch`) +- `bin/cindex` (legacy index wrapper) +- `bin/csearch` (legacy search wrapper) +- `bin/cgrep` (legacy grep wrapper) +- `bin/csweb` (legacy web UI wrapper) + +Run the test suite: + +```bash +make test +``` + +Run tests and static analysis together with `make check`; use `make check-simd` +for the experimental SIMD build. + +Clean build artifacts: + +```bash +make clean +``` + +To install `codesearch` directly into your `$GOPATH/bin` or `$GOBIN`: + +```bash +go install ./cmd/codesearch +``` + +--- + +## SIMD Optimizations (`GOEXPERIMENT=simd`) + +Code Search includes vectorized SIMD optimizations targeting modern 64-bit architectures (ARM64 NEON and AMD64 AVX2) using Go's experimental SIMD intrinsics package (`simd/archsimd`). + +### What SIMD Accelerates + +1. **Vectorized Newline Counting (`regexp/nl_simd.go`)**: + - Scans 16-byte vectors per iteration to count `\n` delimiters during match line number calculation and context extraction. + - Batches vector comparisons in chunks of up to 250 iterations (4,000 bytes) into 16-lane `Uint8x16` accumulators without integer overflow or memory allocations. + - Outperforms scalar loops by up to $4\times$ on large files. + +2. **SIMD File Ingestion & ASCII Scanning (`index/scan_simd.go`)**: + - Evaluates 16 bytes per vector operation to detect NUL bytes (for instant binary file filtering) and checks high bits (`0x80`) to identify pure ASCII chunks. + - Files with pure ASCII chunks bypass multibyte UTF-8 validation and stream directly into trigram extraction. + +3. **Pure Go Fallback (`regexp/nl_purego.go`, `index/scan_purego.go`)**: + - When built without `GOEXPERIMENT=simd`, Go build tags (`!goexperiment.simd`) automatically select pure Go implementations, ensuring seamless builds across all supported Go platforms and architectures. + +### Building with SIMD + +To build Code Search with SIMD intrinsics enabled: + +```bash +# Using Makefile +make build-simd + +# Or using go build directly +GOEXPERIMENT=simd go build -o bin/codesearch ./cmd/codesearch +``` + +### Testing with SIMD + +Run the test suite with SIMD enabled: + +```bash +# Using Makefile +make test-simd + +# Or using go test directly +GOEXPERIMENT=simd go test -v ./... +``` + +### Benchmarks + +Run benchmarks comparing scalar and SIMD routines: + +```bash +# Benchmark newline counting +go test -bench=BenchmarkCountNL ./regexp +GOEXPERIMENT=simd go test -bench=BenchmarkCountNL ./regexp + +# Benchmark ASCII scanning & ingestion +go test -bench=BenchmarkScanASCIIChunk ./index +GOEXPERIMENT=simd go test -bench=BenchmarkScanASCIIChunk ./index +``` + +--- + +## CLI Usage + +`codesearch` (or short alias `cs`) provides a single unified interface with subcommands. + +### 1. Indexing (`codesearch index`) + +Before searching, create an index of the source code tree: + +```bash +# Build an initial index over one or more directories +codesearch index /path/to/repo /path/to/another/repo + +# Reindex all previously indexed roots +codesearch index + +# Discard existing index and build a fresh one +codesearch index -reset /path/to/repo + +# List currently indexed directory roots +codesearch index -list + +# View index size statistics (or use 'codesearch index stats') +codesearch index -stats + +# Validate the index without rebuilding it +codesearch index check +``` + +`index stats`, `index check`, and `index list` are read-only and accept no paths. +The flag forms `-stats` and `-check` are also read-only when no paths are given. +With explicit paths, the flags report or validate the resulting update. +Use `codesearch index ./stats` to index a directory named `stats`. + +Updates build and validate a staging index before replacement and reject +concurrent writers using `.lock`. After a forcibly terminated process, +remove a leftover lock only after verifying no writer is still running. +Input failures abort the update and preserve the previous index. + +**Index Flags:** +- `-reset`: Replace the index with the supplied roots; without paths, remove it. +- `-list`: List indexed directory roots and exit without re-indexing. +- `-stats`: Print index size statistics (path list, name list, posting lists, name index, posting index). +- `-zip`: Index the contents of `.zip` files. +- `-check`: Verify that the existing index format and postings are well-formed. +- `-verbose`: Print progress details during indexing. +- `-cpuprofile `: Write CPU profile to the specified file. + +### 2. Searching (`codesearch search` or `codesearch `) + +Search indexed code using regular expressions. If no subcommand is specified, `codesearch` defaults to `search`: + +```bash +# Search using explicit subcommand +codesearch search 'func main\(' + +# Default search fallback (omit 'search') +codesearch 'func main\(' + +# Case-insensitive search +codesearch search -i 'parse.*config' +codesearch -i 'parse.*config' + +# Filter matches by file path regular expression +codesearch -f '\.go$' 'type [A-Z]' + +# Show line numbers +codesearch -n 'TODO' + +# List matching files only +codesearch -l 'panic\(' + +# Print match counts per file +codesearch -c 'deprecated' + +# Print context lines around matches +codesearch -C 2 'HandleRequest' # 2 lines before and after +codesearch -B 1 -A 3 'NewServer' # 1 line before, 3 lines after + +# Output formatted HTML links +codesearch -html 'ErrNotFound' + +# Brute-force search (search every file in the index, bypassing trigrams) +codesearch -brute 'regexp' +``` + +**Search Flags:** +- `-F`, `--fixed-strings`: Treat the pattern as a literal string. +- `-i`: Case-insensitive search. +- `-f `: Limit search to files whose path matches ``. +- `-n`: Prefix each matching line with its line number. +- `-l`: List file paths only (stop scanning after the first match in each file). +- `-c`: Print count of matching lines per file instead of line contents. +- `-A `: Print `` lines of trailing context after each match. +- `-B `: Print `` lines of leading context before each match. +- `-C `: Print `` lines of context before and after each match. +- `-h`: Suppress file path prefix in output. +- `-html`: Format matches with clickable HTML links for the web viewer. +- `-brute`: Brute-force scan across all indexed files. +- `-verbose`: Print query planning and candidate file count details. + +Search and grep also accept `--ignore-case`, `--line-number`, +`--files-with-matches`, `--count`, `--no-filename`, `--context`, +`--before-context`, and `--after-context`. Search accepts `--path-regexp`; +grep accepts `--invert-match`. These aliases have the same configuration +precedence as their short forms. Exit status is 0 for selected lines, 1 for +no selected lines, and 2 for input failures, even when other files matched. + +### 3. Direct Grep (`codesearch grep`) + +Search files or standard input directly with regular expressions without needing or using an index: + +```bash +# Search specific files +codesearch grep 'pattern' main.go utils.go + +# Case-insensitive search with line numbers +codesearch grep -i -n 'error' server.go + +# Invert match (print non-matching lines) +codesearch grep -v 'debug' log.txt + +# Print context around matches +codesearch grep -C 2 'critical' error.log + +# Search from standard input +cat log.txt | codesearch grep 'FATAL' +codesearch grep 'pattern' - < input.txt +``` + +### 4. Web UI (`codesearch web`) + +Start a local search web server with interactive results and file browsing: + +```bash +# Start on default address (http://localhost:2473) +codesearch web + +# Start on custom address +codesearch web -addr localhost:8080 +codesearch web -http :3000 +``` + +Features: +- Instant query evaluation against the local trigram index. +- Search result linking directly into source file views with line numbers and anchors (`#L123`). +- Syntax-highlighted code inspection supporting both Unix (`\n`) and Windows CRLF (`\r\n`) line endings. +- Breadcrumb navigation across repository directories. + +### 5. Help & Version + +```bash +codesearch version +codesearch help +codesearch help index +codesearch help search +codesearch help grep +codesearch help web +``` + +--- + +## Configuration + +`codesearch` reads defaults from a single per-user JSON file: + +``` +$XDG_CONFIG_HOME/codesearch/config.json +$HOME/.config/codesearch/config.json # when XDG_CONFIG_HOME is unset +``` + +The file is optional. Every key is optional too, and each corresponds to the +flag of the same name; a key you omit keeps the flag's built-in default, and a +flag you pass on the command line always wins over the file. + +```json +{ + "git": true, + "index": "", + "n": true, + "i": false, + "l": false, + "c": false, + "h": false, + "A": 0, + "B": 0, + "C": 0, + "f": "" +} +``` + +- **`git`**: keep the index at the root of the enclosing git repository + (see below). Also available per-run as `-git` / `-git=false`. +- **`index`**: use this index file instead of `$HOME/.csearchindex`. +- The rest are search and grep flag defaults. `C` sets both sides of the match + context, so an explicit `A` or `B` in the file overrides its side. + +A malformed file or an unrecognized key is reported as an error rather than +ignored, so typos do not silently do nothing. + +### Repository-level indexes (`git`) + +With `git` enabled, `codesearch` walks up from the working directory looking for +a `.git` entry — a directory in an ordinary clone, a file in a worktree or +submodule — and uses `/.csearchindex`. Each repository then gets its +own index, and `index`, `search`, and `web` all agree on which one to use from +anywhere inside the tree. Outside any repository, the usual +`$HOME/.csearchindex` applies. Add `.csearchindex` to your `.gitignore` (or +`$HOME/.config/git/ignore`). + +With `git` disabled — the default — nothing is searched for on disk and +behavior is unchanged. + +The index file is resolved in this order: + +1. `$CSEARCHINDEX` +2. `index` from the config file +3. `/.csearchindex`, when `git` is enabled and a repository is found +4. `$HOME/.csearchindex` + +An explicit `--index` flag takes precedence over all four sources. + +## Environment Variables + +- **`CSEARCHINDEX`**: Path to the index file. Overrides the config file. + Defaults to `$HOME/.csearchindex` if unset. +- **`XDG_CONFIG_HOME`**: Base directory for the config file described above. + +--- + +## Legacy Tool Compatibility + +For backward compatibility with existing workflows and scripts, standalone command binaries remain supported and compile alongside `codesearch`: + +- `cindex` $\rightarrow$ `codesearch index` +- `csearch` $\rightarrow$ `codesearch search` +- `cgrep` $\rightarrow$ `codesearch grep` +- `csweb` $\rightarrow$ `codesearch web` +- `cs` $\rightarrow$ alias symlink to `codesearch` + +--- + +## License + +Code Search is distributed under a BSD-style license. See the `LICENSE` file for details. diff --git a/cmd/cgrep/cgrep.go b/cmd/cgrep/cgrep.go index 5e7404face..f31b8fa845 100644 --- a/cmd/cgrep/cgrep.go +++ b/cmd/cgrep/cgrep.go @@ -24,7 +24,7 @@ cannot be abbreviated to -in. ` func usage() { - fmt.Fprintf(os.Stderr, usageMessage) + fmt.Fprint(os.Stderr, usageMessage) os.Exit(2) } diff --git a/cmd/cindex/cindex.go b/cmd/cindex/cindex.go index d82bf6c5b7..d8a97ce3f1 100644 --- a/cmd/cindex/cindex.go +++ b/cmd/cindex/cindex.go @@ -53,7 +53,7 @@ With no path arguments, cindex -reset removes the index. ` func usage() { - fmt.Fprintf(os.Stderr, usageMessage) + fmt.Fprint(os.Stderr, usageMessage) os.Exit(2) } diff --git a/cmd/codesearch/_static/viewer.css b/cmd/codesearch/_static/viewer.css new file mode 100644 index 0000000000..9403f6458f --- /dev/null +++ b/cmd/codesearch/_static/viewer.css @@ -0,0 +1,4 @@ +.sel { + background-color: #ffff88; +} +input[type="text"] { width: 80%; } diff --git a/cmd/codesearch/_static/viewer.js b/cmd/codesearch/_static/viewer.js new file mode 100644 index 0000000000..4df205001f --- /dev/null +++ b/cmd/codesearch/_static/viewer.js @@ -0,0 +1,8 @@ +function highlight() { + if(window.location.hash) { + var span = document.getElementById(window.location.hash.substr(1)); + if(span) { + span.classList.add("sel"); + } + } +} diff --git a/cmd/codesearch/codesearch_test.go b/cmd/codesearch/codesearch_test.go new file mode 100644 index 0000000000..4745ec96a0 --- /dev/null +++ b/cmd/codesearch/codesearch_test.go @@ -0,0 +1,549 @@ +// Copyright 2026 The Go Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +package main + +import ( + "bytes" + "io" + "os" + "path/filepath" + "strings" + "testing" +) + +func runMain(t *testing.T, args []string, stdin io.Reader) (int, string, string) { + t.Helper() + var stdout, stderr bytes.Buffer + code := Main(args, &stdout, &stderr, stdin) + return code, stdout.String(), stderr.String() +} + +func TestCLI_HelpAndVersion(t *testing.T) { + t.Run("version commands", func(t *testing.T) { + for _, arg := range []string{"version", "-version", "--version"} { + t.Run(arg, func(t *testing.T) { + code, stdout, stderr := runMain(t, []string{arg}, nil) + if code != 0 { + t.Fatalf("expected exit code 0, got %d (stderr: %s)", code, stderr) + } + if !strings.Contains(stdout, "codesearch 1.1.0") { + t.Fatalf("expected version output, got: %s", stdout) + } + if stderr != "" { + t.Fatalf("expected empty stderr, got: %s", stderr) + } + }) + } + }) + + t.Run("top-level help commands", func(t *testing.T) { + for _, arg := range []string{"help", "-help", "--help", "-h"} { + t.Run(arg, func(t *testing.T) { + code, stdout, stderr := runMain(t, []string{arg}, nil) + if code != 0 { + t.Fatalf("expected exit code 0, got %d (stderr: %s)", code, stderr) + } + if !strings.Contains(stdout, "codesearch is a tool for indexing and searching") { + t.Fatalf("expected usage text in stdout, got: %s", stdout) + } + if !strings.Contains(stdout, "The commands are:") { + t.Fatalf("expected command listing in stdout, got: %s", stdout) + } + if stderr != "" { + t.Fatalf("expected empty stderr, got: %s", stderr) + } + }) + } + }) + + t.Run("subcommand help", func(t *testing.T) { + tests := []struct { + subcmd string + expectUsage string + }{ + {"index", "usage: codesearch index"}, + {"search", "usage: codesearch search"}, + {"grep", "usage: codesearch grep"}, + {"web", "usage: codesearch web"}, + } + + for _, tc := range tests { + t.Run(tc.subcmd, func(t *testing.T) { + code, stdout, stderr := runMain(t, []string{"help", tc.subcmd}, nil) + if code != 0 { + t.Fatalf("expected exit code 0, got %d (stderr: %s)", code, stderr) + } + combined := stdout + stderr + if !strings.Contains(combined, tc.expectUsage) { + t.Fatalf("expected usage %q for subcommand %q, got stdout=%q stderr=%q", + tc.expectUsage, tc.subcmd, stdout, stderr) + } + }) + } + }) +} + +func TestCLI_IndexAndSearchFlow(t *testing.T) { + tempDir := t.TempDir() + indexFile := filepath.Join(tempDir, ".csearchindex") + t.Setenv("CSEARCHINDEX", indexFile) + + srcDir := filepath.Join(tempDir, "src") + if err := os.MkdirAll(filepath.Join(srcDir, "sub"), 0755); err != nil { + t.Fatal(err) + } + + alphaFile := filepath.Join(srcDir, "alpha.go") + alphaContent := `package main + +import "fmt" + +func main() { + fmt.Println("HelloSearch in alpha Go") +} +` + if err := os.WriteFile(alphaFile, []byte(alphaContent), 0644); err != nil { + t.Fatal(err) + } + + betaFile := filepath.Join(srcDir, "beta.txt") + betaContent := `Line 1: HelloSearch in beta text +Line 2: Other content +Line 3: hellosearch lowercase +` + if err := os.WriteFile(betaFile, []byte(betaContent), 0644); err != nil { + t.Fatal(err) + } + + gammaFile := filepath.Join(srcDir, "sub", "gamma.go") + gammaContent := `package sub + +const Msg = "HelloSearch in gamma Go" +` + if err := os.WriteFile(gammaFile, []byte(gammaContent), 0644); err != nil { + t.Fatal(err) + } + + t.Run("index directory", func(t *testing.T) { + code, stdout, stderr := runMain(t, []string{"index", srcDir}, nil) + if code != 0 { + t.Fatalf("index failed with code %d, stderr: %s, stdout: %s", code, stderr, stdout) + } + if _, err := os.Stat(indexFile); err != nil { + t.Fatalf("expected index file to exist at %s: %v", indexFile, err) + } + }) + + t.Run("index -list", func(t *testing.T) { + code, stdout, stderr := runMain(t, []string{"index", "-list"}, nil) + if code != 0 { + t.Fatalf("index -list failed with code %d, stderr: %s", code, stderr) + } + if !strings.Contains(stdout, srcDir) { + t.Fatalf("expected index -list to contain %s, got: %s", srcDir, stdout) + } + }) + + t.Run("index -stats", func(t *testing.T) { + code, stdout, stderr := runMain(t, []string{"index", "-stats", srcDir}, nil) + if code != 0 { + t.Fatalf("index -stats failed with code %d, stderr: %s", code, stderr) + } + for _, want := range []string{"path list", "name list", "posting lists", "trigrams", "name index", "posting index"} { + if !strings.Contains(stdout, want) { + t.Errorf("expected %q in index -stats output, got:\n%s", want, stdout) + } + } + }) + + t.Run("index stats subcommand alias", func(t *testing.T) { + code, stdout, stderr := runMain(t, []string{"index", "stats"}, nil) + if code != 0 { + t.Fatalf("index stats failed with code %d, stderr: %s", code, stderr) + } + for _, want := range []string{"path list", "name list", "posting lists", "trigrams"} { + if !strings.Contains(stdout, want) { + t.Errorf("expected %q in index stats output, got:\n%s", want, stdout) + } + } + }) + + t.Run("search with line numbers (-n)", func(t *testing.T) { + code, stdout, stderr := runMain(t, []string{"search", "-n", "HelloSearch"}, nil) + if code != 0 { + t.Fatalf("search failed with code %d, stderr: %s", code, stderr) + } + if !strings.Contains(stdout, "alpha.go:6:\tfmt.Println(\"HelloSearch in alpha Go\")") { + t.Errorf("expected match in alpha.go:6, got: %s", stdout) + } + if !strings.Contains(stdout, "beta.txt:1:Line 1: HelloSearch in beta text") { + t.Errorf("expected match in beta.txt:1, got: %s", stdout) + } + if !strings.Contains(stdout, "gamma.go:3:const Msg = \"HelloSearch in gamma Go\"") { + t.Errorf("expected match in gamma.go:3, got: %s", stdout) + } + }) + + t.Run("search with file filter (-f)", func(t *testing.T) { + code, stdout, stderr := runMain(t, []string{"search", "-f", `\.go$`, "HelloSearch"}, nil) + if code != 0 { + t.Fatalf("search with -f failed with code %d, stderr: %s", code, stderr) + } + if !strings.Contains(stdout, "alpha.go") { + t.Errorf("expected match in alpha.go, got: %s", stdout) + } + if !strings.Contains(stdout, "gamma.go") { + t.Errorf("expected match in gamma.go, got: %s", stdout) + } + if strings.Contains(stdout, "beta.txt") { + t.Errorf("did not expect match in beta.txt, got: %s", stdout) + } + }) + + t.Run("search case-insensitive (-i)", func(t *testing.T) { + code, stdout, stderr := runMain(t, []string{"search", "-i", "-n", "hellosearch"}, nil) + if code != 0 { + t.Fatalf("case-insensitive search failed with code %d, stderr: %s", code, stderr) + } + if !strings.Contains(stdout, "beta.txt:1:") { + t.Errorf("expected uppercase match in beta.txt:1, got: %s", stdout) + } + if !strings.Contains(stdout, "beta.txt:3:Line 3: hellosearch lowercase") { + t.Errorf("expected lowercase match in beta.txt:3, got: %s", stdout) + } + }) + + t.Run("search list files only (-l)", func(t *testing.T) { + code, stdout, stderr := runMain(t, []string{"search", "-l", "HelloSearch"}, nil) + if code != 0 { + t.Fatalf("search -l failed with code %d, stderr: %s", code, stderr) + } + lines := strings.Split(strings.TrimSpace(stdout), "\n") + if len(lines) != 3 { + t.Fatalf("expected 3 file lines, got %d: %q", len(lines), stdout) + } + if !strings.Contains(stdout, "alpha.go") || !strings.Contains(stdout, "beta.txt") || !strings.Contains(stdout, "gamma.go") { + t.Errorf("expected alpha.go, beta.txt, gamma.go in output, got: %s", stdout) + } + if strings.Contains(stdout, "fmt.Println") { + t.Errorf("search -l should not print line contents, got: %s", stdout) + } + }) + + t.Run("default search routing without explicit search keyword", func(t *testing.T) { + code, stdout, stderr := runMain(t, []string{"-n", "HelloSearch"}, nil) + if code != 0 { + t.Fatalf("default routing failed with code %d, stderr: %s", code, stderr) + } + if !strings.Contains(stdout, "alpha.go:6:") { + t.Errorf("expected match in alpha.go:6 via default routing, got: %s", stdout) + } + if !strings.Contains(stdout, "beta.txt:1:") { + t.Errorf("expected match in beta.txt:1 via default routing, got: %s", stdout) + } + }) + + t.Run("default search routing with bare regexp", func(t *testing.T) { + code, stdout, stderr := runMain(t, []string{"HelloSearch"}, nil) + if code != 0 { + t.Fatalf("default routing with bare regexp failed with code %d, stderr: %s", code, stderr) + } + if !strings.Contains(stdout, "alpha.go") { + t.Errorf("expected match in alpha.go, got: %s", stdout) + } + }) + + t.Run("search non-existent pattern returns exit code 1", func(t *testing.T) { + code, stdout, stderr := runMain(t, []string{"search", "NonExistentPatternXYZ"}, nil) + if code != 1 { + t.Fatalf("expected exit code 1 for no matches, got %d (stdout: %s, stderr: %s)", code, stdout, stderr) + } + if stdout != "" { + t.Errorf("expected empty stdout for no matches, got: %s", stdout) + } + }) + + t.Run("index -reset", func(t *testing.T) { + code, stdout, stderr := runMain(t, []string{"index", "-reset"}, nil) + if code != 0 { + t.Fatalf("index -reset failed with code %d, stderr: %s, stdout: %s", code, stderr, stdout) + } + if _, err := os.Stat(indexFile); !os.IsNotExist(err) { + t.Fatalf("expected index file to be removed after -reset, got: %v", err) + } + }) +} + +func TestCLI_Grep(t *testing.T) { + tempDir := t.TempDir() + sampleFile := filepath.Join(tempDir, "sample.txt") + sampleContent := `apple +TargetLine 1 +banana +TARGETLINE 2 +cherry +` + if err := os.WriteFile(sampleFile, []byte(sampleContent), 0644); err != nil { + t.Fatal(err) + } + + t.Run("grep file with line numbers (-n)", func(t *testing.T) { + code, stdout, stderr := runMain(t, []string{"grep", "-n", "TargetLine", sampleFile}, nil) + if code != 0 { + t.Fatalf("grep failed with code %d, stderr: %s", code, stderr) + } + if !strings.Contains(stdout, "2:TargetLine 1") { + t.Fatalf("expected line 2 match in output, got: %s", stdout) + } + if strings.Contains(stdout, "TARGETLINE 2") { + t.Fatalf("did not expect case-insensitive match without -i, got: %s", stdout) + } + }) + + t.Run("grep stdin explicit ", func(t *testing.T) { + stdin := strings.NewReader("first line\nTargetLine in stdin\nlast line\n") + code, stdout, stderr := runMain(t, []string{"grep", "-n", "TargetLine", ""}, stdin) + if code != 0 { + t.Fatalf("grep stdin failed with code %d, stderr: %s", code, stderr) + } + if !strings.Contains(stdout, "2:TargetLine in stdin") { + t.Fatalf("expected match from stdin line 2, got: %s", stdout) + } + }) + + t.Run("grep stdin with -", func(t *testing.T) { + stdin := strings.NewReader("line 1\nTargetLine dash\nline 3\n") + code, stdout, stderr := runMain(t, []string{"grep", "TargetLine", "-"}, stdin) + if code != 0 { + t.Fatalf("grep stdin with '-' failed with code %d, stderr: %s", code, stderr) + } + if !strings.Contains(stdout, "TargetLine dash") { + t.Fatalf("expected match from stdin, got: %s", stdout) + } + }) + + t.Run("grep stdin implicit no files", func(t *testing.T) { + stdin := strings.NewReader("alpha\nTargetLine implicit\nomega\n") + code, stdout, stderr := runMain(t, []string{"grep", "TargetLine"}, stdin) + if code != 0 { + t.Fatalf("grep implicit stdin failed with code %d, stderr: %s", code, stderr) + } + if !strings.Contains(stdout, "TargetLine implicit") { + t.Fatalf("expected match from implicit stdin, got: %s", stdout) + } + }) + + t.Run("grep case-insensitive (-i)", func(t *testing.T) { + code, stdout, stderr := runMain(t, []string{"grep", "-i", "-n", "targetline", sampleFile}, nil) + if code != 0 { + t.Fatalf("grep -i failed with code %d, stderr: %s", code, stderr) + } + if !strings.Contains(stdout, "2:TargetLine 1") { + t.Errorf("expected match for line 2, got: %s", stdout) + } + if !strings.Contains(stdout, "4:TARGETLINE 2") { + t.Errorf("expected match for line 4, got: %s", stdout) + } + }) + + t.Run("grep context (-C 1)", func(t *testing.T) { + code, stdout, stderr := runMain(t, []string{"grep", "-C", "1", "TargetLine", sampleFile}, nil) + if code != 0 { + t.Fatalf("grep -C 1 failed with code %d, stderr: %s", code, stderr) + } + if !strings.Contains(stdout, "apple") || !strings.Contains(stdout, "TargetLine 1") || !strings.Contains(stdout, "banana") { + t.Errorf("expected context lines in output, got: %s", stdout) + } + }) + + t.Run("grep hide filename (-h)", func(t *testing.T) { + code, stdout, stderr := runMain(t, []string{"grep", "-h", "TargetLine", sampleFile}, nil) + if code != 0 { + t.Fatalf("grep -h failed with code %d, stderr: %s", code, stderr) + } + if strings.Contains(stdout, "sample.txt:") { + t.Errorf("expected no filename prefix with -h, got: %s", stdout) + } + if !strings.Contains(stdout, "TargetLine 1") { + t.Errorf("expected match in output, got: %s", stdout) + } + }) + t.Run("grep count (-c)", func(t *testing.T) { + code, stdout, stderr := runMain(t, []string{"grep", "-c", "TargetLine", sampleFile}, nil) + if code != 0 { + t.Fatalf("grep -c failed with code %d, stderr: %s", code, stderr) + } + if !strings.Contains(stdout, "1") { + t.Fatalf("expected count 1, got: %s", stdout) + } + }) + + t.Run("grep no match returns exit code 1", func(t *testing.T) { + code, stdout, stderr := runMain(t, []string{"grep", "MissingPatternXYZ", sampleFile}, nil) + if code != 1 { + t.Fatalf("expected exit code 1 for no match, got %d (stdout: %s, stderr: %s)", code, stdout, stderr) + } + if stdout != "" { + t.Errorf("expected empty stdout, got: %s", stdout) + } + }) + + t.Run("grep with context (-C)", func(t *testing.T) { + code, stdout, stderr := runMain(t, []string{"grep", "-C", "1", "TARGETLINE", sampleFile}, nil) + if code != 0 { + t.Fatalf("grep -C failed with code %d, stderr: %s", code, stderr) + } + if !strings.Contains(stdout, "4:") { + t.Errorf("expected line 4 in context output, got:\n%s", stdout) + } + if !strings.Contains(stdout, "banana") || !strings.Contains(stdout, "cherry") { + t.Errorf("expected context lines in output, got:\n%s", stdout) + } + }) +} + +func TestCLI_UsageAndErrors(t *testing.T) { + t.Run("no arguments returns exit code 2", func(t *testing.T) { + code, stdout, stderr := runMain(t, []string{}, nil) + if code != 2 { + t.Fatalf("expected exit code 2, got %d", code) + } + if !strings.Contains(stderr, "codesearch is a tool for indexing and searching") { + t.Fatalf("expected main usage message in stderr, got: %s", stderr) + } + if stdout != "" { + t.Fatalf("expected empty stdout, got: %s", stdout) + } + }) + + t.Run("search invalid flag returns exit code 2", func(t *testing.T) { + code, _, stderr := runMain(t, []string{"search", "-invalidflagXYZ"}, nil) + if code != 2 { + t.Fatalf("expected exit code 2, got %d (stderr: %s)", code, stderr) + } + if !strings.Contains(stderr, "flag provided but not defined") { + t.Fatalf("expected flag error in stderr, got: %s", stderr) + } + }) + + t.Run("search invalid regexp returns exit code 2", func(t *testing.T) { + code, _, stderr := runMain(t, []string{"search", "(invalid[regex"}, nil) + if code != 2 { + t.Fatalf("expected exit code 2, got %d (stderr: %s)", code, stderr) + } + if !strings.Contains(stderr, "codesearch search: ") && !strings.Contains(stderr, "error parsing regexp") { + t.Fatalf("expected regexp parse error in stderr, got: %s", stderr) + } + }) + + t.Run("search missing index file returns exit code 2", func(t *testing.T) { + tempDir := t.TempDir() + nonExistentIndex := filepath.Join(tempDir, "does_not_exist.index") + t.Setenv("CSEARCHINDEX", nonExistentIndex) + + code, _, stderr := runMain(t, []string{"search", "somePattern"}, nil) + if code != 2 { + t.Fatalf("expected exit code 2 for missing index, got %d (stderr: %s)", code, stderr) + } + if !strings.Contains(stderr, "cannot access index") || !strings.Contains(stderr, "codesearch index .") { + t.Fatalf("expected index missing message in stderr, got: %s", stderr) + } + }) + + t.Run("search no args returns exit code 2", func(t *testing.T) { + code, _, stderr := runMain(t, []string{"search"}, nil) + if code != 2 { + t.Fatalf("expected exit code 2, got %d (stderr: %s)", code, stderr) + } + if !strings.Contains(stderr, "usage: codesearch search") { + t.Fatalf("expected search usage message in stderr, got: %s", stderr) + } + }) + + t.Run("grep invalid flag returns exit code 2", func(t *testing.T) { + code, _, stderr := runMain(t, []string{"grep", "-invalidflagXYZ"}, nil) + if code != 2 { + t.Fatalf("expected exit code 2, got %d (stderr: %s)", code, stderr) + } + if !strings.Contains(stderr, "flag provided but not defined") { + t.Fatalf("expected flag error in stderr, got: %s", stderr) + } + }) + + t.Run("grep invalid regexp returns exit code 2", func(t *testing.T) { + code, _, stderr := runMain(t, []string{"grep", "(invalid[regex"}, nil) + if code != 2 { + t.Fatalf("expected exit code 2, got %d (stderr: %s)", code, stderr) + } + if !strings.Contains(stderr, "codesearch grep: ") && !strings.Contains(stderr, "error parsing regexp") { + t.Fatalf("expected regexp parse error in stderr, got: %s", stderr) + } + }) + + t.Run("grep no args returns exit code 2", func(t *testing.T) { + code, _, stderr := runMain(t, []string{"grep"}, nil) + if code != 2 { + t.Fatalf("expected exit code 2, got %d (stderr: %s)", code, stderr) + } + if !strings.Contains(stderr, "usage: codesearch grep") { + t.Fatalf("expected grep usage message in stderr, got: %s", stderr) + } + }) + + t.Run("index invalid flag returns exit code 2", func(t *testing.T) { + code, _, stderr := runMain(t, []string{"index", "-invalidflagXYZ"}, nil) + if code != 2 { + t.Fatalf("expected exit code 2, got %d (stderr: %s)", code, stderr) + } + if !strings.Contains(stderr, "flag provided but not defined") { + t.Fatalf("expected flag error in stderr, got: %s", stderr) + } + }) +} + +func TestWeb_Helpers(t *testing.T) { + t.Run("isText handles unix and windows crlf", func(t *testing.T) { + unix := []byte("package main\n\nfunc main() {}\n") + if !isText(unix) { + t.Errorf("expected unix text to be detected as text") + } + + windows := []byte("package main\r\n\r\nfunc main() {}\r\n") + if !isText(windows) { + t.Errorf("expected windows CRLF text to be detected as text") + } + + binary := []byte("GIF89a\x00\x00\x00\x00\x00\x00") + if isText(binary) { + t.Errorf("expected binary data to NOT be detected as text") + } + }) + + t.Run("serveFile strips carriage returns in spans", func(t *testing.T) { + data := []byte("line1\r\nline2\r\n") + rendered := string(serveFile("/test/path.go", data)) + if strings.Contains(rendered, "\r") { + t.Errorf("expected \\r to be stripped from HTML spans, got %q", rendered) + } + if !strings.Contains(rendered, `id="L1"`) || !strings.Contains(rendered, `id="L2"`) { + t.Errorf("expected line number spans, got %s", rendered) + } + }) + + t.Run("printHeader does not produce empty anchor or double slash", func(t *testing.T) { + var buf bytes.Buffer + printHeader(&buf, "/Users/test/file.go") + out := buf.String() + if strings.Contains(out, `href="/show/"`) { + t.Errorf("expected no empty href in header, got: %s", out) + } + if strings.Contains(out, `//Users`) { + t.Errorf("expected no double slashes in hrefs, got: %s", out) + } + if !strings.Contains(out, ``) || !strings.Contains(out, ``) { + t.Errorf("expected matching and tags in header, got: %s", out) + } + }) +} diff --git a/cmd/codesearch/config.go b/cmd/codesearch/config.go new file mode 100644 index 0000000000..4e74f585f3 --- /dev/null +++ b/cmd/codesearch/config.go @@ -0,0 +1,237 @@ +// Copyright 2026 The Go Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +package main + +import ( + "encoding/json" + "flag" + "fmt" + "io" + "log" + "os" + "path/filepath" + "runtime" + "strconv" + + "github.com/google/codesearch/index" +) + +// Config holds command defaults read from the user's config file. Fields are +// pointers so that an absent key is distinguishable from an explicit false or +// zero, which must still override a flag default. +type Config struct { + indexOverride string + // Git places the index at the root of the enclosing git repository. + Git *bool `json:"git"` + // Index names the index file explicitly, overriding Git. + Index *string `json:"index"` + + // Defaults for the search and grep flags of the same name. + N *bool `json:"n"` + I *bool `json:"i"` + L *bool `json:"l"` + Count *bool `json:"c"` + H *bool `json:"h"` + A *int `json:"A"` + B *int `json:"B"` + C *int `json:"C"` + F *string `json:"f"` +} + +// configFile returns the path of the user config file: +// $XDG_CONFIG_HOME/codesearch/config.json, or $HOME/.config/codesearch/config.json. +func configFile() string { + dir := os.Getenv("XDG_CONFIG_HOME") + if dir == "" { + home := os.Getenv("HOME") + if runtime.GOOS == "windows" && home == "" { + home = os.Getenv("USERPROFILE") + } + dir = filepath.Join(home, ".config") + } + return filepath.Join(dir, "codesearch", "config.json") +} + +// loadConfig reads the user config file. A missing file yields an empty config; +// a malformed one is an error, so that typos are reported instead of ignored. +func loadConfig() (*Config, error) { + file := configFile() + f, err := os.Open(file) + if err != nil { + if os.IsNotExist(err) { + return &Config{}, nil + } + return nil, err + } + defer f.Close() + + cfg := new(Config) + dec := json.NewDecoder(f) + dec.DisallowUnknownFields() + if err := dec.Decode(cfg); err != nil { + return nil, fmt.Errorf("%s: %w", file, err) + } + if err := dec.Decode(new(struct{})); err != io.EOF { + return nil, fmt.Errorf("%s: unexpected data after JSON object", file) + } + return cfg, nil +} + +// gitRoot walks up from dir looking for a .git entry. It may be a directory in +// an ordinary clone or a file in a worktree or submodule. +func gitRoot(dir string) (string, bool) { + for { + if _, err := os.Lstat(filepath.Join(dir, ".git")); err == nil { + return dir, true + } + parent := filepath.Dir(dir) + if parent == dir { + return "", false + } + dir = parent + } +} + +// resolveIndexFile returns the index file to use when running in wd, in +// precedence order: $CSEARCHINDEX, the config index path, the enclosing git +// repository if git mode is on, then $HOME/.csearchindex. +func resolveIndexFile(cfg *Config, wd string) string { + if cfg.indexOverride != "" { + return cfg.indexOverride + } + if f := os.Getenv("CSEARCHINDEX"); f != "" { + return f + } + if cfg.Index != nil && *cfg.Index != "" { + return *cfg.Index + } + if cfg.Git != nil && *cfg.Git { + if root, ok := gitRoot(wd); ok { + return filepath.Join(root, ".csearchindex") + } + } + return index.File() +} + +// indexFile resolves the index file for the current working directory. +func indexFile(cfg *Config) string { + wd, err := os.Getwd() + if err != nil { + wd = "." + } + return resolveIndexFile(cfg, wd) +} + +// applyConfig sets every configured flag that the user did not pass on the +// command line, so an explicit flag always wins over the config file. Flags the +// current command does not define are ignored. +func applyConfig(flags *flag.FlagSet, cfg *Config) error { + passed := make(map[string]bool) + flags.Visit(func(f *flag.Flag) { + name := f.Name + if canonical, ok := longFlags[name]; ok { + name = canonical + } + passed[name] = true + }) + // Context flags overlap: C writes both A and B. Preserve the effective + // command-line values, not just which flag names were supplied. + context := make(map[string]string) + for _, name := range []string{"A", "B"} { + if f := flags.Lookup(name); f != nil && (passed[name] || passed["C"]) { + context[name] = f.Value.String() + } + } + + // -C is applied first so that an explicitly configured -A or -B, which set + // one side of the context each, override it. + defaults := []struct { + name string + value string + }{} + add := func(name, value string) { + defaults = append(defaults, struct { + name string + value string + }{name, value}) + } + if cfg.C != nil { + add("C", strconv.Itoa(*cfg.C)) + } + if cfg.A != nil { + add("A", strconv.Itoa(*cfg.A)) + } + if cfg.B != nil { + add("B", strconv.Itoa(*cfg.B)) + } + if cfg.N != nil { + add("n", strconv.FormatBool(*cfg.N)) + } + if cfg.I != nil { + add("i", strconv.FormatBool(*cfg.I)) + } + if cfg.L != nil { + add("l", strconv.FormatBool(*cfg.L)) + } + if cfg.Count != nil { + add("c", strconv.FormatBool(*cfg.Count)) + } + if cfg.H != nil { + add("h", strconv.FormatBool(*cfg.H)) + } + if cfg.F != nil { + add("f", *cfg.F) + } + + for _, d := range defaults { + if passed[d.name] || flags.Lookup(d.name) == nil { + continue + } + if err := flags.Set(d.name, d.value); err != nil { + return fmt.Errorf("config %s: %w", d.name, err) + } + } + for name, value := range context { + if err := flags.Set(name, value); err != nil { + return err + } + } + return nil +} + +// gitFlag adds the -git flag, letting a single run turn repository-level +// indexing on (-git) or off (-git=false) regardless of the config file. +func gitFlag(flags *flag.FlagSet) *bool { + flags.String("index", "", "use this index file (overrides environment and config)") + return flags.Bool("git", false, "keep the index at the root of the enclosing git repository") +} + +// applyGitFlag lets an explicit -git override the configured value. +func applyGitFlag(flags *flag.FlagSet, cfg *Config, val *bool) { + flags.Visit(func(f *flag.Flag) { + if f.Name == "git" { + cfg.Git = val + } + }) +} + +// setupConfig loads the config file and applies it to flags already parsed by +// the calling command. It reports false after logging if the config is bad. +func setupConfig(flags *flag.FlagSet, logger *log.Logger, git *bool) (*Config, bool) { + cfg, err := loadConfig() + if err != nil { + logger.Println(err) + return nil, false + } + applyGitFlag(flags, cfg, git) + if f := flags.Lookup("index"); f != nil { + cfg.indexOverride = f.Value.String() + } + if err := applyConfig(flags, cfg); err != nil { + logger.Println(err) + return nil, false + } + return cfg, true +} diff --git a/cmd/codesearch/config_test.go b/cmd/codesearch/config_test.go new file mode 100644 index 0000000000..193713dac8 --- /dev/null +++ b/cmd/codesearch/config_test.go @@ -0,0 +1,346 @@ +// Copyright 2026 The Go Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +package main + +import ( + "flag" + "io" + "os" + "path/filepath" + "strings" + "testing" +) + +// setConfig points loadConfig at a temporary config directory, writing content +// as the config file unless content is empty. +func setConfig(t *testing.T, content string) { + t.Helper() + dir := t.TempDir() + if err := os.MkdirAll(filepath.Join(dir, "codesearch"), 0o755); err != nil { + t.Fatal(err) + } + if content != "" { + file := filepath.Join(dir, "codesearch", "config.json") + if err := os.WriteFile(file, []byte(content), 0o644); err != nil { + t.Fatal(err) + } + } + t.Setenv("XDG_CONFIG_HOME", dir) +} + +func TestLoadConfig(t *testing.T) { + t.Run("missing file is not an error", func(t *testing.T) { + setConfig(t, "") + cfg, err := loadConfig() + if err != nil { + t.Fatalf("expected no error for missing config, got %v", err) + } + if cfg.Git != nil || cfg.N != nil { + t.Fatalf("expected empty config, got %+v", cfg) + } + }) + + t.Run("values are read", func(t *testing.T) { + setConfig(t, `{"git": true, "n": true, "i": false, "C": 3, "index": "/tmp/ix"}`) + cfg, err := loadConfig() + if err != nil { + t.Fatal(err) + } + if cfg.Git == nil || !*cfg.Git { + t.Errorf("git: got %v, want true", cfg.Git) + } + if cfg.N == nil || !*cfg.N { + t.Errorf("n: got %v, want true", cfg.N) + } + if cfg.I == nil || *cfg.I { + t.Errorf("i: got %v, want explicit false", cfg.I) + } + if cfg.C == nil || *cfg.C != 3 { + t.Errorf("C: got %v, want 3", cfg.C) + } + if cfg.Index == nil || *cfg.Index != "/tmp/ix" { + t.Errorf("index: got %v, want /tmp/ix", cfg.Index) + } + }) + + t.Run("malformed JSON is an error", func(t *testing.T) { + setConfig(t, `{"n": tru`) + if _, err := loadConfig(); err == nil { + t.Fatal("expected error for malformed config") + } + }) + + t.Run("unknown key is an error", func(t *testing.T) { + setConfig(t, `{"linenumbers": true}`) + _, err := loadConfig() + if err == nil { + t.Fatal("expected error for unknown config key") + } + if !strings.Contains(err.Error(), "linenumbers") { + t.Fatalf("expected error to name the bad key, got %v", err) + } + }) +} + +func TestGitRoot(t *testing.T) { + root := t.TempDir() + deep := filepath.Join(root, "a", "b") + if err := os.MkdirAll(deep, 0o755); err != nil { + t.Fatal(err) + } + + if got, ok := gitRoot(deep); ok { + t.Fatalf("expected no repo, got %q", got) + } + + // A worktree or submodule has .git as a file, not a directory. + gitFile := filepath.Join(root, "a", ".git") + if err := os.WriteFile(gitFile, []byte("gitdir: /elsewhere\n"), 0o644); err != nil { + t.Fatal(err) + } + got, ok := gitRoot(deep) + if !ok || got != filepath.Join(root, "a") { + t.Fatalf("gitRoot(%q) = %q, %v; want %q, true", deep, got, ok, filepath.Join(root, "a")) + } + + // A nearer .git directory wins over the one above it. + if err := os.MkdirAll(filepath.Join(deep, ".git"), 0o755); err != nil { + t.Fatal(err) + } + if got, ok := gitRoot(deep); !ok || got != deep { + t.Fatalf("gitRoot(%q) = %q, %v; want %q, true", deep, got, ok, deep) + } +} + +func TestResolveIndexFile(t *testing.T) { + home := t.TempDir() + repo := filepath.Join(t.TempDir(), "repo") + sub := filepath.Join(repo, "pkg") + if err := os.MkdirAll(filepath.Join(repo, ".git"), 0o755); err != nil { + t.Fatal(err) + } + if err := os.MkdirAll(sub, 0o755); err != nil { + t.Fatal(err) + } + + boolPtr := func(b bool) *bool { return &b } + strPtr := func(s string) *string { return &s } + + tests := []struct { + name string + env string + cfg *Config + wd string + want string + }{ + { + name: "env wins over everything", + env: "/env/index", + cfg: &Config{Git: boolPtr(true), Index: strPtr("/cfg/index")}, + wd: sub, + want: "/env/index", + }, + { + name: "config index wins over git", + cfg: &Config{Git: boolPtr(true), Index: strPtr("/cfg/index")}, + wd: sub, + want: "/cfg/index", + }, + { + name: "git mode uses repo root from a subdirectory", + cfg: &Config{Git: boolPtr(true)}, + wd: sub, + want: filepath.Join(repo, ".csearchindex"), + }, + { + name: "git mode outside a repo falls back to home", + cfg: &Config{Git: boolPtr(true)}, + wd: home, + want: filepath.Join(home, ".csearchindex"), + }, + { + name: "git disabled ignores the repo", + cfg: &Config{Git: boolPtr(false)}, + wd: sub, + want: filepath.Join(home, ".csearchindex"), + }, + { + name: "empty config uses home", + cfg: &Config{}, + wd: sub, + want: filepath.Join(home, ".csearchindex"), + }, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Setenv("HOME", home) + t.Setenv("USERPROFILE", home) + t.Setenv("CSEARCHINDEX", tc.env) + if got := resolveIndexFile(tc.cfg, tc.wd); got != tc.want { + t.Fatalf("resolveIndexFile = %q, want %q", got, tc.want) + } + }) + } +} + +func TestApplyConfig(t *testing.T) { + newFlags := func() (*flag.FlagSet, *bool, *int, *int) { + flags := flag.NewFlagSet("test", flag.ContinueOnError) + flags.SetOutput(io.Discard) + n := flags.Bool("n", false, "") + a := flags.Int("A", 0, "") + b := flags.Int("B", 0, "") + flags.Func("C", "", func(s string) error { + *a, *b = 7, 7 // stand-in for the real -C behavior + return nil + }) + return flags, n, a, b + } + boolPtr := func(b bool) *bool { return &b } + intPtr := func(i int) *int { return &i } + + t.Run("config fills unset flags", func(t *testing.T) { + flags, n, _, _ := newFlags() + if err := flags.Parse(nil); err != nil { + t.Fatal(err) + } + if err := applyConfig(flags, &Config{N: boolPtr(true)}); err != nil { + t.Fatal(err) + } + if !*n { + t.Fatal("expected -n from config") + } + }) + + t.Run("explicit flag beats config", func(t *testing.T) { + flags, n, _, _ := newFlags() + if err := flags.Parse([]string{"-n=false"}); err != nil { + t.Fatal(err) + } + if err := applyConfig(flags, &Config{N: boolPtr(true)}); err != nil { + t.Fatal(err) + } + if *n { + t.Fatal("explicit -n=false should win over config") + } + }) + + t.Run("A and B override C", func(t *testing.T) { + flags, _, a, b := newFlags() + if err := flags.Parse(nil); err != nil { + t.Fatal(err) + } + if err := applyConfig(flags, &Config{C: intPtr(5), A: intPtr(1)}); err != nil { + t.Fatal(err) + } + if *a != 1 || *b != 7 { + t.Fatalf("A=%d B=%d, want A=1 B=7", *a, *b) + } + }) + + t.Run("flags absent from the set are skipped", func(t *testing.T) { + flags, _, _, _ := newFlags() + if err := flags.Parse(nil); err != nil { + t.Fatal(err) + } + // -f exists in search but not in this set; must not error. + f := "x" + if err := applyConfig(flags, &Config{F: &f}); err != nil { + t.Fatal(err) + } + }) +} + +func TestCLI_ConfigDefaults(t *testing.T) { + tempDir := t.TempDir() + indexFile := filepath.Join(tempDir, ".csearchindex") + t.Setenv("CSEARCHINDEX", indexFile) + setConfig(t, `{"n": true}`) + + srcDir := filepath.Join(tempDir, "src") + if err := os.MkdirAll(srcDir, 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(srcDir, "a.go"), []byte("package main\n\nvar Needle = 1\n"), 0o644); err != nil { + t.Fatal(err) + } + + if code, _, stderr := runMain(t, []string{"index", srcDir}, nil); code != 0 { + t.Fatalf("index failed: %d %s", code, stderr) + } + + code, stdout, stderr := runMain(t, []string{"search", "Needle"}, nil) + if code != 0 { + t.Fatalf("search failed: %d %s", code, stderr) + } + if !strings.Contains(stdout, "a.go:3:") { + t.Fatalf("expected line numbers from config, got %q", stdout) + } + + // An explicit flag still overrides the config. + code, stdout, stderr = runMain(t, []string{"search", "-n=false", "Needle"}, nil) + if code != 0 { + t.Fatalf("search failed: %d %s", code, stderr) + } + if strings.Contains(stdout, "a.go:3:") { + t.Fatalf("expected -n=false to win over config, got %q", stdout) + } +} + +func TestCLI_GitModeIndexLocation(t *testing.T) { + home := t.TempDir() + repo := filepath.Join(t.TempDir(), "repo") + if err := os.MkdirAll(filepath.Join(repo, ".git"), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(repo, "a.go"), []byte("package main\n\nvar Needle = 1\n"), 0o644); err != nil { + t.Fatal(err) + } + + t.Setenv("HOME", home) + t.Setenv("USERPROFILE", home) + t.Setenv("CSEARCHINDEX", "") + setConfig(t, `{"git": true}`) + + wd, err := os.Getwd() + if err != nil { + t.Fatal(err) + } + if err := os.Chdir(repo); err != nil { + t.Fatal(err) + } + t.Cleanup(func() { os.Chdir(wd) }) + + if code, _, stderr := runMain(t, []string{"index", "."}, nil); code != 0 { + t.Fatalf("index failed: %d %s", code, stderr) + } + + if _, err := os.Stat(filepath.Join(repo, ".csearchindex")); err != nil { + t.Fatalf("expected index at repo root: %v", err) + } + if _, err := os.Stat(filepath.Join(home, ".csearchindex")); err == nil { + t.Fatal("index should not have been written to $HOME in git mode") + } + + code, stdout, stderr := runMain(t, []string{"search", "Needle"}, nil) + if code != 0 { + t.Fatalf("search failed: %d %s", code, stderr) + } + if !strings.Contains(stdout, "a.go") { + t.Fatalf("expected match from repo index, got %q", stdout) + } +} + +func TestCLI_IndexWithoutExistingIndexFile(t *testing.T) { + t.Setenv("CSEARCHINDEX", filepath.Join(t.TempDir(), ".csearchindex")) + code, _, stderr := runMain(t, []string{"index"}, nil) + if code != 2 { + t.Fatalf("expected exit 2 for missing index, got %d (stderr: %s)", code, stderr) + } + if !strings.Contains(stderr, "not found") { + t.Fatalf("expected a helpful message, got %q", stderr) + } +} diff --git a/cmd/codesearch/flags.go b/cmd/codesearch/flags.go new file mode 100644 index 0000000000..9e75524c3f --- /dev/null +++ b/cmd/codesearch/flags.go @@ -0,0 +1,22 @@ +// Copyright 2026 The Go Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +package main + +import "flag" + +var longFlags = map[string]string{ + "ignore-case": "i", "line-number": "n", "files-with-matches": "l", + "count": "c", "no-filename": "h", "after-context": "A", + "before-context": "B", "context": "C", "path-regexp": "f", + "invert-match": "v", "fixed-strings": "F", +} + +func addLongFlags(flags *flag.FlagSet) { + for alias, name := range longFlags { + if f := flags.Lookup(name); f != nil { + flags.Var(f.Value, alias, f.Usage) + } + } +} diff --git a/cmd/codesearch/grep.go b/cmd/codesearch/grep.go new file mode 100644 index 0000000000..f3684df430 --- /dev/null +++ b/cmd/codesearch/grep.go @@ -0,0 +1,108 @@ +// Copyright 2011 The Go Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +package main + +import ( + "flag" + "fmt" + "io" + "log" + "os" + stdregexp "regexp" + "runtime/pprof" + + "github.com/google/codesearch/regexp" +) + +var grepUsageMessage = `usage: codesearch grep [-c] [-h] [-i] [-l] [-n] [-v] [-A n] [-B n] [-C n] regexp [file...] + +Grep behaves like grep, searching for RE2 regexp without an index. +` + +func RunGrep(args []string, stdout, stderr io.Writer, stdin io.Reader) int { + flags := flag.NewFlagSet("grep", flag.ContinueOnError) + flags.SetOutput(stderr) + + var g regexp.Grep + g.AddFlagsToSet(flags) + g.AddVFlagToSet(flags) + g.Stdout = stdout + g.Stderr = stderr + + iflag := flags.Bool("i", false, "case-insensitive match") + literal := flags.Bool("F", false, "treat the pattern as a literal string") + cpuProfile := flags.String("cpuprofile", "", "write cpu profile to this file") + addLongFlags(flags) + + flags.Usage = func() { + fmt.Fprint(stderr, grepUsageMessage) + flags.PrintDefaults() + } + + if err := flags.Parse(args); err != nil { + if err == flag.ErrHelp { + return 0 + } + return 2 + } + + if _, ok := setupConfig(flags, log.New(stderr, "codesearch grep: ", 0), nil); !ok { + return 2 + } + + rem := flags.Args() + if len(rem) == 0 { + flags.Usage() + return 2 + } + + logger := log.New(stderr, "codesearch grep: ", 0) + + if *cpuProfile != "" { + f, err := os.Create(*cpuProfile) + if err != nil { + logger.Println(err) + return 2 + } + defer f.Close() + pprof.StartCPUProfile(f) + defer pprof.StopCPUProfile() + } + + pattern := rem[0] + if *literal { + pattern = stdregexp.QuoteMeta(pattern) + } + pat := "(?m)" + pattern + if *iflag { + pat = "(?i)" + pat + } + re, err := regexp.Compile(pat) + if err != nil { + logger.Println(err) + return 2 + } + g.Regexp = re + + if len(rem) == 1 { + g.Reader(stdin, "") + } else { + for _, arg := range rem[1:] { + if arg == "" || arg == "-" { + g.Reader(stdin, arg) + } else { + g.File(arg) + } + } + } + + if g.Errors { + return 2 + } + if !g.Match { + return 1 + } + return 0 +} diff --git a/cmd/codesearch/index.go b/cmd/codesearch/index.go new file mode 100644 index 0000000000..884a31f193 --- /dev/null +++ b/cmd/codesearch/index.go @@ -0,0 +1,270 @@ +// Copyright 2011 The Go Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +package main + +import ( + "flag" + "fmt" + "io" + "log" + "os" + "path/filepath" + "runtime/pprof" + "slices" + + "github.com/google/codesearch/index" +) + +var indexUsageMessage = `usage: codesearch index [-list] [-reset] [-check] [-zip] [-stats] [-verbose] [-git] [path...] + +Index builds or updates the trigram index used by codesearch search. +If no paths are given, it updates existing index roots. + +Read-only inspection: codesearch index stats | check | list +Use --index path to select an index for this command. +` + +func RunIndex(args []string, stdout, stderr io.Writer) int { + inspect := false + if len(args) > 0 { + switch args[0] { + case "stats", "check", "list": + inspect = true + args = append([]string{"-" + args[0]}, args[1:]...) + } + } + flags := flag.NewFlagSet("index", flag.ContinueOnError) + flags.SetOutput(stderr) + + listFlag := flags.Bool("list", false, "list indexed paths and exit") + resetFlag := flags.Bool("reset", false, "discard existing index") + verboseFlag := flags.Bool("verbose", false, "print extra information") + cpuProfile := flags.String("cpuprofile", "", "write cpu profile to this file") + checkFlag := flags.Bool("check", false, "check index is well-formatted") + zipFlag := flags.Bool("zip", false, "index content in zip files") + statsFlag := flags.Bool("stats", false, "print index size statistics") + gitMode := gitFlag(flags) + + flags.Usage = func() { + fmt.Fprint(stderr, indexUsageMessage) + flags.PrintDefaults() + } + + if err := flags.Parse(args); err != nil { + if err == flag.ErrHelp { + return 0 + } + return 2 + } + + logger := log.New(stderr, "codesearch index: ", 0) + + cfg, ok := setupConfig(flags, logger, gitMode) + if !ok { + return 2 + } + master, err := filepath.Abs(indexFile(cfg)) + if err != nil { + logger.Println(err) + return 2 + } + if inspect && (flags.NArg() != 0 || *resetFlag) { + logger.Println("inspection commands do not accept paths or -reset; use ./stats to index a path named stats") + return 2 + } + + // index.Open aborts the process on a missing file, so check up front for the + // cases that read the existing index. + if _, err := os.Stat(master); err != nil && (*listFlag || (flags.NArg() == 0 && !*resetFlag)) { + logger.Printf("index file %s not found; run 'codesearch index ' first", master) + return 2 + } + + if inspect || *listFlag || (flags.NArg() == 0 && !*resetFlag && (*statsFlag || *checkFlag)) { + + ix := index.Open(master) + defer ix.Close() + if *checkFlag { + if err := ix.Check(); err != nil { + logger.Println(err) + return 2 + } + } + if *listFlag { + for p := range ix.Roots().All() { + fmt.Fprintf(stdout, "%s\n", p) + } + } + if *statsFlag { + ix.PrintStatsTo(stdout) + } + return 0 + } + + if *cpuProfile != "" { + f, err := os.Create(*cpuProfile) + if err != nil { + logger.Println(err) + return 2 + } + defer f.Close() + pprof.StartCPUProfile(f) + defer pprof.StopCPUProfile() + } + + // Serialize writers before reading roots or replacing the index. An + // interrupted process may leave a lock; never silently steal it. + lock, err := os.OpenFile(master+".lock", os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0600) + if err != nil { + logger.Printf("cannot lock index %s: %v", master, err) + return 2 + } + lock.Close() + defer os.Remove(master + ".lock") + if *resetFlag && flags.NArg() == 0 { + if err := os.Remove(master); err != nil && !os.IsNotExist(err) { + logger.Println(err) + return 2 + } + return 0 + } + + var roots []index.Path + if flags.NArg() == 0 { + + ix := index.Open(master) + roots = slices.Collect(ix.Roots().All()) + ix.Close() + } else { + for _, arg := range flags.Args() { + a, err := filepath.Abs(arg) + if err != nil { + logger.Printf("%s: %s", arg, err) + return 2 + } + roots = append(roots, index.MakePath(a)) + } + slices.SortFunc(roots, index.Path.Compare) + } + + for _, root := range roots { + if _, err := os.Stat(root.String()); err != nil { + logger.Println(err) + return 2 + } + } + if _, err := os.Stat(master); os.IsNotExist(err) { + *resetFlag = true + } else if err != nil { + logger.Println(err) + return 2 + } else { + if err := checkIndex(master); err != nil { + logger.Println(err) + return 2 + } + } + // Keep staging files on the same filesystem for replacement by rename. + stage, err := os.MkdirTemp(filepath.Dir(master), ".codesearch-build-") + if err != nil { + logger.Println(err) + return 2 + } + defer os.RemoveAll(stage) + file := filepath.Join(stage, "index") + + ix := index.Create(file) + defer ix.Close() + ix.Verbose = *verboseFlag + ix.Zip = *zipFlag + ix.AddRoots(roots) + for _, root := range roots { + if *verboseFlag { + logger.Printf("index %s", root) + } + err := filepath.Walk(root.String(), func(path string, info os.FileInfo, err error) error { + if path == master || path == master+".lock" { + return nil + } + if _, elem := filepath.Split(path); elem != "" { + if elem[0] == '.' || elem[0] == '#' || elem[0] == '~' || elem[len(elem)-1] == '~' { + if info != nil && info.IsDir() { + return filepath.SkipDir + } + return nil + } + } + if err != nil { + return err + } + if info != nil && info.Mode()&os.ModeType == 0 { + if err := ix.AddFile(path); err != nil { + return err + } + } + return nil + }) + if err != nil { + logger.Println(err) + return 2 + } + } + if *verboseFlag { + logger.Printf("flush index") + } + ix.Flush() + if err := ix.Close(); err != nil { + logger.Println(err) + return 2 + } + + if !*resetFlag { + if *verboseFlag { + logger.Printf("merge %s %s", master, file) + } + index.Merge(file+"~", master, file) + file += "~" + } + if err := checkIndex(file); err != nil { + logger.Println(err) + return 2 + } + f, err := os.OpenFile(file, os.O_RDWR, 0) + if err != nil { + logger.Println(err) + return 2 + } + err = f.Sync() + closeErr := f.Close() + if err != nil { + logger.Println(err) + return 2 + } + if closeErr != nil { + logger.Println(closeErr) + return 2 + } + if err := os.Rename(file, master); err != nil { + logger.Printf("replace index: %v", err) + return 2 + } + + if *verboseFlag { + logger.Printf("done") + } + + if *statsFlag { + ix := index.Open(master) + defer ix.Close() + ix.PrintStatsTo(stdout) + } + return 0 +} + +func checkIndex(file string) error { + ix := index.Open(file) + defer ix.Close() + return ix.Check() +} diff --git a/cmd/codesearch/main.go b/cmd/codesearch/main.go new file mode 100644 index 0000000000..aaec160a4d --- /dev/null +++ b/cmd/codesearch/main.go @@ -0,0 +1,93 @@ +// Copyright 2026 The Go Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +package main + +import ( + "fmt" + "io" + "os" + "runtime" + "runtime/debug" +) + +var mainUsageMessage = `codesearch is a tool for indexing and searching source code using regular expressions. + +Usage: + codesearch [arguments] + +The commands are: + index build or update trigram index + status show selected index, roots, and statistics + search search indexed corpus for regular expression + grep search files or stdin directly without index + web start local search web server + version print version information + help print help for a command + +Default: + Invoking codesearch with search arguments or flags directly routes to 'search': + codesearch [flags] -> codesearch search [flags] + +Use "codesearch help " for more information about a command. +` + +var versionString = "codesearch 1.1.0" + +func Main(args []string, stdout, stderr io.Writer, stdin io.Reader) int { + if len(args) == 0 { + fmt.Fprint(stderr, mainUsageMessage) + return 2 + } + + cmd := args[0] + switch cmd { + case "index": + return RunIndex(args[1:], stdout, stderr) + case "status": + return RunStatus(args[1:], stdout, stderr) + case "search": + return RunSearch(args[1:], stdout, stderr) + case "grep": + return RunGrep(args[1:], stdout, stderr, stdin) + case "web": + return RunWeb(args[1:], stdout, stderr) + case "version", "-version", "--version": + fmt.Fprintf(stdout, "%s (%s)\n", versionString, runtime.Version()) + if info, ok := debug.ReadBuildInfo(); ok { + for _, setting := range info.Settings { + if setting.Key == "vcs.revision" || setting.Key == "vcs.modified" || setting.Key == "GOEXPERIMENT" { + fmt.Fprintf(stdout, "%s: %s\n", setting.Key, setting.Value) + } + } + } + return 0 + case "help", "-help", "--help", "-h": + if len(args) > 1 { + sub := args[1] + switch sub { + case "index": + return RunIndex([]string{"-help"}, stdout, stderr) + case "status": + return RunStatus([]string{"-help"}, stdout, stderr) + case "search": + return RunSearch([]string{"-help"}, stdout, stderr) + case "grep": + return RunGrep([]string{"-help"}, stdout, stderr, stdin) + case "web": + return RunWeb([]string{"-help"}, stdout, stderr) + } + } + fmt.Fprint(stdout, mainUsageMessage) + return 0 + default: + // Default to 'search' subcommand + return RunSearch(args, stdout, stderr) + } +} + +func main() { + exitCode := Main(os.Args[1:], os.Stdout, os.Stderr, os.Stdin) + os.Exit(exitCode) +} diff --git a/cmd/codesearch/reliability_test.go b/cmd/codesearch/reliability_test.go new file mode 100644 index 0000000000..20cab4859a --- /dev/null +++ b/cmd/codesearch/reliability_test.go @@ -0,0 +1,231 @@ +// Copyright 2026 The Go Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +package main + +import ( + "bytes" + "errors" + "net/http" + "net/http/httptest" + "net/url" + "os" + "path/filepath" + "strings" + "testing" +) + +func TestGrepInversion(t *testing.T) { + t.Setenv("XDG_CONFIG_HOME", t.TempDir()) + for _, tt := range []struct { + name, input, pattern, want string + flags []string + code int + }{ + {"lines", "match\nother\nmatch\nlast", "match", "2:other\n4:last\n", []string{"-h", "-n"}, 0}, + {"all match", "match\n", "match", "", nil, 1}, + {"empty input", "", "match", "", nil, 1}, + {"empty line", "\nmatch\n", "match", "\n", []string{"-h"}, 0}, + {"count", "match\nother\nlast\n", "match", ": 2\n", []string{"-c"}, 0}, + {"files", "match\nother\n", "match", "\n", []string{"-l"}, 0}, + {"empty pattern", "one\ntwo\n", "", "", nil, 1}, + } { + t.Run(tt.name, func(t *testing.T) { + args := append([]string{"grep", "-v"}, tt.flags...) + args = append(args, tt.pattern) + code, out, err := runMain(t, args, strings.NewReader(tt.input)) + if code != tt.code || out != tt.want || err != "" { + t.Fatalf("got (%d, %q, %q), want (%d, %q, empty stderr)", code, out, err, tt.code, tt.want) + } + }) + } +} + +type failingReader struct{} + +func (failingReader) Read([]byte) (int, error) { return 0, errors.New("read failed") } + +func TestGrepInputErrors(t *testing.T) { + t.Setenv("XDG_CONFIG_HOME", t.TempDir()) + missing := filepath.Join(t.TempDir(), "missing") + code, _, stderr := runMain(t, []string{"grep", "match", "-", missing}, strings.NewReader("match\n")) + if code != 2 || !strings.Contains(stderr, missing) { + t.Fatalf("missing file: code=%d stderr=%q", code, stderr) + } + code, _, stderr = runMain(t, []string{"grep", "match"}, failingReader{}) + if code != 2 || !strings.Contains(stderr, "read failed") { + t.Fatalf("read failure: code=%d stderr=%q", code, stderr) + } +} + +func TestSearchMissingCandidate(t *testing.T) { + dir := t.TempDir() + t.Setenv("XDG_CONFIG_HOME", dir) + t.Setenv("CSEARCHINDEX", filepath.Join(dir, ".index")) + file := filepath.Join(dir, "source.txt") + if err := os.WriteFile(file, []byte("needle\n"), 0600); err != nil { + t.Fatal(err) + } + if code, _, err := runMain(t, []string{"index", file}, nil); code != 0 { + t.Fatalf("index: %d %s", code, err) + } + if err := os.Remove(file); err != nil { + t.Fatal(err) + } + code, _, stderr := runMain(t, []string{"search", "needle"}, nil) + if code != 2 || !strings.Contains(stderr, file) { + t.Fatalf("code=%d stderr=%q", code, stderr) + } +} + +func TestContextConfigPrecedence(t *testing.T) { + for _, tt := range []struct { + config string + flags []string + want string + }{ + {`{"C":2}`, []string{"-A", "0", "-B", "0"}, "needle\n"}, + {`{"A":2,"B":2}`, []string{"-C", "0"}, "needle\n"}, + } { + t.Run(tt.config, func(t *testing.T) { + dir := t.TempDir() + t.Setenv("XDG_CONFIG_HOME", dir) + if err := os.Mkdir(filepath.Join(dir, "codesearch"), 0700); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(dir, "codesearch", "config.json"), []byte(tt.config), 0600); err != nil { + t.Fatal(err) + } + args := append([]string{"grep", "-h"}, tt.flags...) + args = append(args, "needle") + code, out, stderr := runMain(t, args, strings.NewReader("before\nneedle\nafter\n")) + if code != 0 || out != tt.want { + t.Fatalf("code=%d out=%q stderr=%q", code, out, stderr) + } + }) + } +} + +func TestIndexInspectionAndFailedUpdate(t *testing.T) { + dir := t.TempDir() + t.Setenv("XDG_CONFIG_HOME", dir) + master := filepath.Join(dir, ".index") + t.Setenv("CSEARCHINDEX", master) + file := filepath.Join(dir, "source.txt") + if err := os.WriteFile(file, []byte("old content\n"), 0600); err != nil { + t.Fatal(err) + } + if code, _, stderr := runMain(t, []string{"index", file}, nil); code != 0 { + t.Fatalf("%d %s", code, stderr) + } + before, err := os.ReadFile(master) + if err != nil { + t.Fatal(err) + } + if err := os.WriteFile(file, []byte("new content\n"), 0600); err != nil { + t.Fatal(err) + } + for _, args := range [][]string{ + {"index", "stats"}, {"index", "check"}, {"index", "-stats"}, {"index", "-check"}, {"index", "stats", "-stats=false"}, + } { + if code, _, stderr := runMain(t, args, nil); code != 0 { + t.Fatalf("%v: %d %s", args, code, stderr) + } + after, err := os.ReadFile(master) + if err != nil || !bytes.Equal(before, after) { + t.Fatalf("%v mutated index: %v", args, err) + } + } + for _, flags := range [][]string{{"index"}, {"index", "-reset"}} { + args := append(flags, filepath.Join(dir, "missing")) + if code, _, stderr := runMain(t, args, nil); code != 2 { + t.Fatalf("%v: %d %s", args, code, stderr) + } + after, err := os.ReadFile(master) + if err != nil || !bytes.Equal(before, after) { + t.Fatalf("failed update mutated index: %v", err) + } + } + if err := os.WriteFile(master+".lock", nil, 0600); err != nil { + t.Fatal(err) + } + if code, _, _ := runMain(t, []string{"index", file}, nil); code != 2 { + t.Fatalf("locked update returned %d", code) + } +} + +func TestWebRootBoundary(t *testing.T) { + dir := t.TempDir() + root := filepath.Join(dir, "repo") + if err := os.Mkdir(root, 0700); err != nil { + t.Fatal(err) + } + inside := filepath.Join(root, "source.txt") + outside := filepath.Join(dir, "private.txt") + for _, file := range []string{inside, outside} { + if err := os.WriteFile(file, []byte("content\n"), 0600); err != nil { + t.Fatal(err) + } + } + for _, tt := range []struct { + file string + code int + }{{inside, 200}, {outside, 403}, {dir, 403}} { + r := httptest.NewRequest(http.MethodGet, (&url.URL{Path: "/show" + filepath.ToSlash(tt.file)}).String(), nil) + w := httptest.NewRecorder() + show(w, r, []string{root}) + if w.Code != tt.code { + t.Errorf("%s: got %d want %d", tt.file, w.Code, tt.code) + } + } + link := filepath.Join(root, "escape") + if err := os.Symlink(outside, link); err != nil { + t.Skipf("symlinks unavailable: %v", err) + } + if withinRoots(link, []string{root}) { + t.Fatal("symlink escaped root") + } +} + +func TestModernSearchFlags(t *testing.T) { + t.Setenv("XDG_CONFIG_HOME", t.TempDir()) + code, out, stderr := runMain(t, []string{"grep", "--fixed-strings", "--ignore-case", "--line-number", "--no-filename", "a.b"}, strings.NewReader("A.B\naxb\n")) + if code != 0 || out != "1:A.B\n" || stderr != "" { + t.Fatalf("code=%d out=%q stderr=%q", code, out, stderr) + } +} + +func TestInversionAcrossBuffers(t *testing.T) { + t.Setenv("XDG_CONFIG_HOME", t.TempDir()) + input := strings.Repeat("other\n", 200000) + "match\nlast\n" + code, out, stderr := runMain(t, []string{"grep", "-v", "-c", "match"}, strings.NewReader(input)) + if code != 0 || out != ": 200001\n" { + t.Fatalf("%d %q %s", code, out, stderr) + } +} + +func TestExplicitIndexAndStatus(t *testing.T) { + dir := t.TempDir() + t.Setenv("XDG_CONFIG_HOME", dir) + t.Setenv("CSEARCHINDEX", filepath.Join(dir, "wrong-index")) + master := filepath.Join(dir, ".selected-index") + file := filepath.Join(dir, "source.txt") + if err := os.WriteFile(file, []byte("a.b\naxb\n"), 0600); err != nil { + t.Fatal(err) + } + if code, _, stderr := runMain(t, []string{"index", "--index", master, file}, nil); code != 0 { + t.Fatalf("%d %s", code, stderr) + } + code, out, stderr := runMain(t, []string{"search", "--index", master, "--fixed-strings", "--no-filename", "a.b"}, nil) + if code != 0 || out != "a.b\n" { + t.Fatalf("%d %q %s", code, out, stderr) + } + code, out, stderr = runMain(t, []string{"status", "--index", master}, nil) + if code != 0 || !strings.Contains(out, "Selected by: --index") || !strings.Contains(out, file) || !strings.Contains(out, "Freshness: not checked") { + t.Fatalf("%d %q %s", code, out, stderr) + } + if _, err := os.Stat(os.Getenv("CSEARCHINDEX")); !os.IsNotExist(err) { + t.Fatalf("environment index should not be created: %v", err) + } +} diff --git a/cmd/codesearch/search.go b/cmd/codesearch/search.go new file mode 100644 index 0000000000..13ffd91586 --- /dev/null +++ b/cmd/codesearch/search.go @@ -0,0 +1,218 @@ +// Copyright 2011 The Go Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +package main + +import ( + "archive/zip" + "bytes" + "flag" + "fmt" + "io" + "log" + "os" + stdregexp "regexp" + "runtime/pprof" + "strings" + + "github.com/google/codesearch/index" + "github.com/google/codesearch/regexp" +) + +var searchUsageMessage = `usage: codesearch search [-c] [-f fileregexp] [-h] [-i] [-l] [-n] [-A n] [-B n] [-C n] [-html] [-brute] [-git] regexp + +Search behaves like grep over all indexed files, searching for RE2 regexp. +` + +func RunSearch(args []string, stdout, stderr io.Writer) int { + flags := flag.NewFlagSet("search", flag.ContinueOnError) + flags.SetOutput(stderr) + + g := regexp.Grep{ + Stdout: stdout, + Stderr: stderr, + } + g.AddFlagsToSet(flags) + + fFlag := flags.String("f", "", "search only files with names matching this regexp") + iFlag := flags.Bool("i", false, "case-insensitive search") + literal := flags.Bool("F", false, "treat the pattern as a literal string") + htmlFlag := flags.Bool("html", false, "print HTML output") + verboseFlag := flags.Bool("verbose", false, "print extra information") + bruteFlag := flags.Bool("brute", false, "brute force - search all files in index") + cpuProfile := flags.String("cpuprofile", "", "write cpu profile to this file") + gitMode := gitFlag(flags) + addLongFlags(flags) + + flags.Usage = func() { + fmt.Fprint(stderr, searchUsageMessage) + flags.PrintDefaults() + } + + if err := flags.Parse(args); err != nil { + if err == flag.ErrHelp { + return 0 + } + return 2 + } + + logger := log.New(stderr, "codesearch search: ", 0) + + cfg, ok := setupConfig(flags, logger, gitMode) + if !ok { + return 2 + } + + if *htmlFlag { + g.HTML = true + } + remaining := flags.Args() + if len(remaining) != 1 { + flags.Usage() + return 2 + } + + if *cpuProfile != "" { + f, err := os.Create(*cpuProfile) + if err != nil { + logger.Println(err) + return 2 + } + defer f.Close() + pprof.StartCPUProfile(f) + defer pprof.StopCPUProfile() + } + + pattern := remaining[0] + if *literal { + pattern = stdregexp.QuoteMeta(pattern) + } + pat := "(?m)" + pattern + if *iFlag { + pat = "(?i)" + pat + } + re, err := regexp.Compile(pat) + if err != nil { + logger.Println(err) + return 2 + } + g.Regexp = re + + var fre *regexp.Regexp + if *fFlag != "" { + fre, err = regexp.Compile(*fFlag) + if err != nil { + logger.Println(err) + return 2 + } + } + + q := index.RegexpQuery(re.Syntax) + if *verboseFlag { + logger.Printf("query: %s\n", q) + } + + ixFile := indexFile(cfg) + if _, err := os.Stat(ixFile); err != nil { + logger.Printf("cannot access index %s: %v; create an index with 'codesearch index .'\n", ixFile, err) + return 2 + } + + ix := index.Open(ixFile) + defer ix.Close() + ix.Verbose = *verboseFlag + var post []int + if *bruteFlag { + post = ix.PostingQuery(&index.Query{Op: index.QAll}) + } else { + post = ix.PostingQuery(q) + } + + if *verboseFlag { + logger.Printf("post query identified %d possible files\n", len(post)) + } + + if fre != nil { + fnames := make([]int, 0, len(post)) + for _, fileid := range post { + name := ix.Name(fileid) + if fre.MatchString(name.String(), true, true) < 0 { + continue + } + fnames = append(fnames, fileid) + } + if *verboseFlag { + logger.Printf("filename regexp matched %d files\n", len(fnames)) + } + post = fnames + } + + var ( + zipFile string + zipReader *zip.ReadCloser + zipMap map[string]*zip.File + ) + + for _, fileid := range post { + name := ix.Name(fileid).String() + if g.L && (pat == "(?m)" || pat == "(?i)(?m)") { + g.Reader(bytes.NewReader(nil), name) + continue + } + file, err := os.Open(name) + if err != nil { + if i := strings.Index(name, ".zip\x01"); i >= 0 { + zfile, zname := name[:i+4], name[i+5:] + if zfile != zipFile { + if zipReader != nil { + zipReader.Close() + zipMap = nil + } + zipFile = zfile + zipReader, err = zip.OpenReader(zfile) + if err != nil { + zipReader = nil + } + if zipReader != nil { + zipMap = make(map[string]*zip.File) + for _, zf := range zipReader.File { + zipMap[zf.Name] = zf + } + } + } + if zipMap != nil { + zf := zipMap[zname] + if zf != nil { + r, err := zf.Open() + if err != nil { + g.Errors = true + logger.Printf("%s: %v", name, err) + continue + } + g.Reader(r, name) + r.Close() + continue + } + } + } + g.Errors = true + logger.Printf("cannot read %s", name) + continue + } + g.Reader(file, name) + file.Close() + } + + if zipReader != nil { + zipReader.Close() + } + + if g.Errors { + return 2 + } + if !g.Match { + return 1 + } + return 0 +} diff --git a/cmd/codesearch/status.go b/cmd/codesearch/status.go new file mode 100644 index 0000000000..9bd671ae09 --- /dev/null +++ b/cmd/codesearch/status.go @@ -0,0 +1,72 @@ +// Copyright 2026 The Go Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +package main + +import ( + "flag" + "fmt" + "io" + "log" + "os" + "time" + + "github.com/google/codesearch/index" +) + +func RunStatus(args []string, stdout, stderr io.Writer) int { + flags := flag.NewFlagSet("status", flag.ContinueOnError) + flags.SetOutput(stderr) + git := gitFlag(flags) + flags.Usage = func() { + fmt.Fprintln(stderr, "usage: codesearch status [-git] [--index path]\n\nShow the selected index, roots, modification time, and size statistics.") + flags.PrintDefaults() + } + if err := flags.Parse(args); err != nil { + if err == flag.ErrHelp { + return 0 + } + return 2 + } + if flags.NArg() != 0 { + flags.Usage() + return 2 + } + logger := log.New(stderr, "codesearch status: ", 0) + cfg, ok := setupConfig(flags, logger, git) + if !ok { + return 2 + } + file := indexFile(cfg) + source := "default" + switch { + case cfg.indexOverride != "": + source = "--index" + case os.Getenv("CSEARCHINDEX") != "": + source = "CSEARCHINDEX" + case cfg.Index != nil && *cfg.Index != "": + source = "config" + case cfg.Git != nil && *cfg.Git: + if wd, err := os.Getwd(); err == nil { + if _, ok := gitRoot(wd); ok { + source = "git repository" + } + } + } + fmt.Fprintf(stdout, "Index: %s\nSelected by: %s\n", file, source) + info, err := os.Stat(file) + if err != nil { + logger.Printf("%v; create an index with 'codesearch index .'", err) + return 2 + } + ix := index.Open(file) + defer ix.Close() + fmt.Fprintf(stdout, "Modified: %s\nSize: %d bytes\nRoots:\n", info.ModTime().Format(time.RFC3339), info.Size()) + for root := range ix.Roots().All() { + fmt.Fprintf(stdout, " %s\n", root) + } + ix.PrintStatsTo(stdout) + fmt.Fprintln(stdout, "Freshness: not checked; run 'codesearch index' to refresh existing roots.") + return 0 +} diff --git a/cmd/codesearch/web.go b/cmd/codesearch/web.go new file mode 100644 index 0000000000..34f47157cd --- /dev/null +++ b/cmd/codesearch/web.go @@ -0,0 +1,365 @@ +// Copyright 2020 The Go Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +package main + +import ( + "archive/zip" + "bytes" + "embed" + "flag" + "fmt" + "html" + "io" + "io/fs" + "log" + "net/http" + "os" + "path" + "path/filepath" + "strings" + "time" + "unicode/utf8" + + "github.com/google/codesearch/index" + "github.com/google/codesearch/regexp" +) + +//go:embed _static +var static embed.FS + +var webUsageMessage = `usage: codesearch web [-http addr] [-verbose] [-git] + +Web starts a local HTTP server providing a web search interface. +` + +func RunWeb(args []string, stdout, stderr io.Writer) int { + flags := flag.NewFlagSet("web", flag.ContinueOnError) + flags.SetOutput(stderr) + + httpAddr := flags.String("http", "localhost:2473", "HTTP service address") + addrFlag := flags.String("addr", "", "HTTP service address (alias for -http)") + verboseFlag := flags.Bool("verbose", false, "print extra information") + gitMode := gitFlag(flags) + + flags.Usage = func() { + fmt.Fprint(stderr, webUsageMessage) + flags.PrintDefaults() + } + + if err := flags.Parse(args); err != nil { + if err == flag.ErrHelp { + return 0 + } + return 2 + } + + cfg, ok := setupConfig(flags, log.New(stderr, "codesearch web: ", 0), gitMode) + if !ok { + return 2 + } + ixFile := indexFile(cfg) + if _, err := os.Stat(ixFile); err != nil { + fmt.Fprintf(stderr, "cannot access index %s: %v; run 'codesearch index .' first\n", ixFile, err) + return 2 + } + ix := index.Open(ixFile) + var roots []string + for root := range ix.Roots().All() { + roots = append(roots, root.String()) + } + ix.Close() + + listenAddr := *httpAddr + if *addrFlag != "" { + listenAddr = *addrFlag + } + + mux := http.NewServeMux() + mux.Handle("/_static/", http.FileServer(http.FS(static))) + mux.HandleFunc("/show/", func(w http.ResponseWriter, r *http.Request) { show(w, r, roots) }) + mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) { + home(w, r, ixFile, *verboseFlag) + }) + + fmt.Fprintf(stdout, "Serving Code Search web UI at http://%s\n", listenAddr) + if err := http.ListenAndServe(listenAddr, mux); err != nil { + fmt.Fprintf(stderr, "web server error: %v\n", err) + return 2 + } + return 0 +} + +func home(w http.ResponseWriter, r *http.Request, ixFile string, verbose bool) { + qarg := r.FormValue("q") + w.Write([]byte(strings.ReplaceAll(homePage, "QUERY", html.EscapeString(qarg)))) + if qarg == "" { + return + } + + g := regexp.Grep{ + HTML: true, + Limit: 100000, + Stdout: w, + Stderr: w, + } + + pat := "(?m)" + qarg + re, err := regexp.Compile(pat) + if err != nil { + fmt.Fprintf(w, "Bad query: %s\n", html.EscapeString(err.Error())) + return + } + g.Regexp = re + var fre *regexp.Regexp + farg := r.FormValue("f") + if farg != "" { + fre, err = regexp.Compile(farg) + if err != nil { + fmt.Fprintf(w, "Bad -f flag: %s\n", html.EscapeString(err.Error())) + return + } + } + q := index.RegexpQuery(re.Syntax) + if verbose { + log.Printf("query: %s\n", q) + } + + start := time.Now() + ix := index.Open(ixFile) + defer ix.Close() + ix.Verbose = verbose + post := ix.PostingQuery(q) + if verbose { + fmt.Fprintf(w, "post query identified %d possible files\n", len(post)) + } + + if fre != nil { + fnames := make([]int, 0, len(post)) + + for _, fileid := range post { + name := ix.Name(fileid) + if fre.MatchString(name.String(), true, true) < 0 { + continue + } + fnames = append(fnames, fileid) + } + + if verbose { + fmt.Fprintf(w, "filename regexp matched %d files\n", len(fnames)) + } + post = fnames + } + + var ( + zipFile string + zipReader *zip.ReadCloser + zipMap map[string]*zip.File + ) + + for _, fileid := range post { + if g.Limited { + break + } + name := ix.Name(fileid).String() + file, err := os.Open(name) + if err != nil { + if i := strings.Index(name, ".zip\x01"); i >= 0 { + zfile, zname := name[:i+4], name[i+5:] + if zfile != zipFile { + if zipReader != nil { + zipReader.Close() + zipMap = nil + } + zipFile = zfile + zipReader, err = zip.OpenReader(zfile) + if err != nil { + zipReader = nil + } + if zipReader != nil { + zipMap = make(map[string]*zip.File) + for _, f := range zipReader.File { + zipMap[f.Name] = f + } + } + } + if zipMap != nil { + f := zipMap[zname] + if f != nil { + r, err := f.Open() + if err != nil { + continue + } + g.Reader(r, name) + r.Close() + continue + } + } + } + continue + } + g.Reader(file, name) + file.Close() + } + + if zipReader != nil { + zipReader.Close() + } + + fmt.Fprintf(w, "\n%d matches in %.3fs\n", g.Matches, time.Since(start).Seconds()) + if g.Limited { + fmt.Fprintf(w, "more matches not shown due to match limit\n") + } +} + +var homePage = ` + + + +Code Search + + +Code Search +

+

+ + +
+

+


+
+`
+
+func show(w http.ResponseWriter, r *http.Request, roots []string) {
+	file := strings.TrimPrefix(r.URL.Path, "/show")
+	if strings.HasPrefix(file, "/") && filepath.IsAbs(file[1:]) {
+		// Turn /c:/foo into c:/foo on Windows.
+		file = file[1:]
+	}
+	if !withinRoots(file, roots) {
+		http.Error(w, "path is outside indexed roots", http.StatusForbidden)
+		return
+	}
+	info, err := os.Stat(file)
+	if err != nil {
+		http.Error(w, err.Error(), 500)
+		return
+	}
+	if info.IsDir() {
+		dirs, err := os.ReadDir(file)
+		if err != nil {
+			http.Error(w, err.Error(), 500)
+			return
+		}
+		w.Write(serveDir(file, dirs))
+		return
+	}
+
+	data, err := os.ReadFile(file)
+	if err != nil {
+		http.Error(w, err.Error(), 500)
+		return
+	}
+	w.Write(serveFile(file, data))
+}
+
+// withinRoots compares resolved filesystem paths so symlinks cannot expose
+// unrelated directories through the local viewer.
+func withinRoots(file string, roots []string) bool {
+	resolved, err := filepath.EvalSymlinks(file)
+	if err != nil {
+		return false
+	}
+	resolved, err = filepath.Abs(resolved)
+	if err != nil {
+		return false
+	}
+	for _, root := range roots {
+		root, err := filepath.EvalSymlinks(root)
+		if err != nil {
+			continue
+		}
+		root, err = filepath.Abs(root)
+		if err != nil {
+			continue
+		}
+		rel, err := filepath.Rel(root, resolved)
+		if err == nil && rel != ".." && !strings.HasPrefix(rel, ".."+string(filepath.Separator)) && !filepath.IsAbs(rel) {
+			return true
+		}
+	}
+	return false
+}
+
+func printHeader(buf *bytes.Buffer, file string) {
+	e := html.EscapeString
+	buf.WriteString("\n\n")
+	buf.WriteString("\n")
+	buf.WriteString("\n")
+	fmt.Fprintf(buf, `%s - code search`, e(file))
+	buf.WriteString("\n
\n")
+	f := ""
+	cleanFile := strings.TrimPrefix(file, "/")
+	for _, elem := range strings.Split(cleanFile, "/") {
+		f += "/" + elem
+		fmt.Fprintf(buf, `/%s`, e(f), e(elem))
+	}
+	fmt.Fprintf(buf, ` (about)`)
+	fmt.Fprintf(buf, "\n\n")
+}
+
+func serveDir(file string, dir []fs.DirEntry) []byte {
+	var buf bytes.Buffer
+	e := html.EscapeString
+	printHeader(&buf, file)
+	for _, d := range dir {
+		file := path.Join(file, d.Name())
+		fmt.Fprintf(&buf, "%s\n", e(file), e(path.Base(file)))
+	}
+	return buf.Bytes()
+}
+
+var nl = []byte("\n")
+
+func serveFile(file string, data []byte) []byte {
+	if !isText(data) {
+		return data
+	}
+
+	var buf bytes.Buffer
+	e := html.EscapeString
+	printHeader(&buf, file)
+	n := 1 + bytes.Count(data, nl)
+	wid := len(fmt.Sprintf("%d", n))
+	wid = (wid+2+7)&^7 - 2
+	n = 1
+	for len(data) > 0 {
+		var line []byte
+		line, data, _ = bytes.Cut(data, nl)
+		line = bytes.TrimRight(line, "\r")
+		fmt.Fprintf(&buf, "%*d  %s\n", n, wid, n, e(string(line)))
+		n++
+	}
+	return buf.Bytes()
+}
+
+// isText reports whether a significant prefix of s looks like correct UTF-8;
+// that is, if it is likely that s is human-readable text.
+func isText(s []byte) bool {
+	const max = 1024 // at least utf8.UTFMax
+	if len(s) > max {
+		s = s[0:max]
+	}
+	for i, c := range string(s) {
+		if i+utf8.UTFMax > len(s) {
+			// last char may be incomplete - ignore
+			break
+		}
+		if c == 0xFFFD || c < ' ' && c != '\n' && c != '\t' && c != '\f' && c != '\r' {
+			// decoding error or control character - not a text file
+			return false
+		}
+	}
+	return true
+}
diff --git a/cmd/csearch/csearch.go b/cmd/csearch/csearch.go
index 58af5b89fe..3aa90a3929 100644
--- a/cmd/csearch/csearch.go
+++ b/cmd/csearch/csearch.go
@@ -44,7 +44,7 @@ empty, $HOME/.csearchindex.
 `
 
 func usage() {
-	fmt.Fprintf(os.Stderr, usageMessage)
+	fmt.Fprint(os.Stderr, usageMessage)
 	os.Exit(2)
 }
 
diff --git a/cmd/csweb/web.go b/cmd/csweb/web.go
index 4b46d92da1..258ed16f8a 100644
--- a/cmd/csweb/web.go
+++ b/cmd/csweb/web.go
@@ -208,9 +208,10 @@ func printHeader(buf *bytes.Buffer, file string) {
 	buf.WriteString("\n")
 	buf.WriteString("\n")
 	fmt.Fprintf(buf, `%s - code search`, e(file))
-	buf.WriteString("\n
\n")
+	buf.WriteString("\n
\n")
 	f := ""
-	for _, elem := range strings.Split(file, "/") {
+	cleanFile := strings.TrimPrefix(file, "/")
+	for _, elem := range strings.Split(cleanFile, "/") {
 		f += "/" + elem
 		fmt.Fprintf(buf, `/%s`, e(f), e(elem))
 	}
@@ -247,6 +248,7 @@ func serveFile(file string, data []byte) []byte {
 	for len(data) > 0 {
 		var line []byte
 		line, data, _ = bytes.Cut(data, nl)
+		line = bytes.TrimRight(line, "\r")
 		fmt.Fprintf(&buf, "%*d  %s\n", n, wid, n, e(string(line)))
 		n++
 	}
@@ -265,7 +267,7 @@ func isText(s []byte) bool {
 			// last char may be incomplete - ignore
 			break
 		}
-		if c == 0xFFFD || c < ' ' && c != '\n' && c != '\t' && c != '\f' {
+		if c == 0xFFFD || c < ' ' && c != '\n' && c != '\t' && c != '\f' && c != '\r' {
 			// decoding error or control character - not a text file
 			return false
 		}
diff --git a/docs/superpowers/plans/2026-08-25-codesearch-cli-plan.md b/docs/superpowers/plans/2026-08-25-codesearch-cli-plan.md
new file mode 100644
index 0000000000..d6fa462176
--- /dev/null
+++ b/docs/superpowers/plans/2026-08-25-codesearch-cli-plan.md
@@ -0,0 +1,1131 @@
+# Unified Code Search CLI Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Consolidate Google Code Search into a single unified CLI executable (`codesearch` with `cs` alias) supporting subcommands `index`, `search`, `grep`, and `web`, with default search routing and legacy tool shims.
+
+**Architecture:** A unified package under `cmd/codesearch/` provides modular subcommand runners implementing `Run(args []string, stdout, stderr io.Writer, stdin io.Reader) int`. A top-level dispatcher in `main.go` parses arguments and dispatches to subcommands or defaults to `search`. Legacy binaries (`cindex`, `csearch`, `cgrep`, `csweb`) become thin wrappers delegating to the unified runners.
+
+**Tech Stack:** Go 1.26+ standard library (`flag`, `embed`, `os`, `path/filepath`, `io`), zero third-party dependencies.
+
+## Global Constraints
+
+- Zero external dependencies; use only Go standard library and internal `index` / `regexp` / `sparse` packages.
+- Subcommand runners MUST accept `(args []string, stdout, stderr io.Writer, stdin io.Reader) int` to allow in-process, deterministic unit testing without process termination.
+- Flag parsing MUST use `flag.NewFlagSet(..., flag.ContinueOnError)` within each runner to avoid mutating global flag state across test cases.
+- All Go vet checks (`go vet ./...`) and tests (`go test ./...`) must pass cleanly.
+
+---
+
+### Task 1: Fix Legacy `fmt.Fprintf` Format String Warnings
+
+**Files:**
+- Modify: `cmd/cindex/cindex.go:55-58`
+- Modify: `cmd/csearch/csearch.go:46-49`
+- Modify: `cmd/cgrep/cgrep.go:26-29`
+
+**Interfaces:**
+- Consumes: Existing usage messages
+- Produces: Clean `go vet` and `go test` compilation across all packages
+
+- [ ] **Step 1: Inspect failing vet checks**
+
+Run: `go test ./...`
+Expected: FAIL with `non-constant format string in call to fmt.Fprintf` in `cmd/cindex`, `cmd/csearch`, `cmd/cgrep`.
+
+- [ ] **Step 2: Update usage functions to use `fmt.Fprint`**
+
+In `cmd/cindex/cindex.go`:
+```go
+func usage() {
+	fmt.Fprint(os.Stderr, usageMessage)
+	os.Exit(2)
+}
+```
+
+In `cmd/csearch/csearch.go`:
+```go
+func usage() {
+	fmt.Fprint(os.Stderr, usageMessage)
+	os.Exit(2)
+}
+```
+
+In `cmd/cgrep/cgrep.go`:
+```go
+func usage() {
+	fmt.Fprint(os.Stderr, usageMessage)
+	os.Exit(2)
+}
+```
+
+- [ ] **Step 3: Verify all existing packages compile and pass tests**
+
+Run: `go test ./...`
+Expected: PASS (`ok github.com/google/codesearch/index`, `ok github.com/google/codesearch/regexp`).
+
+- [ ] **Step 4: Commit**
+
+```bash
+git add cmd/cindex/cindex.go cmd/csearch/csearch.go cmd/cgrep/cgrep.go
+git commit -m "fix(cmd): use fmt.Fprint for static usage strings to satisfy go vet"
+```
+
+---
+
+### Task 2: Implement Subcommand Handlers in `cmd/codesearch/`
+
+**Files:**
+- Create: `cmd/codesearch/index.go`
+- Create: `cmd/codesearch/search.go`
+- Create: `cmd/codesearch/grep.go`
+- Create: `cmd/codesearch/web.go`
+- Copy: `cmd/csweb/_static/*` to `cmd/codesearch/_static/*`
+
+**Interfaces:**
+- Consumes: `github.com/google/codesearch/index`, `github.com/google/codesearch/regexp`
+- Produces:
+  - `RunIndex(args []string, stdout, stderr io.Writer) int`
+  - `RunSearch(args []string, stdout, stderr io.Writer) int`
+  - `RunGrep(args []string, stdout, stderr io.Writer, stdin io.Reader) int`
+  - `RunWeb(args []string, stdout, stderr io.Writer) int`
+
+- [ ] **Step 1: Create `cmd/codesearch/_static/` and copy web assets**
+
+```bash
+mkdir -p cmd/codesearch/_static
+cp cmd/csweb/_static/* cmd/codesearch/_static/
+```
+
+- [ ] **Step 2: Implement `cmd/codesearch/index.go`**
+
+Create `cmd/codesearch/index.go`:
+```go
+// Copyright 2011 The Go Authors.  All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE file.
+
+package main
+
+import (
+	"flag"
+	"fmt"
+	"io"
+	"log"
+	"os"
+	"path/filepath"
+	"runtime/pprof"
+	"slices"
+
+	"github.com/google/codesearch/index"
+)
+
+var indexUsageMessage = `usage: codesearch index [-list] [-reset] [-check] [-zip] [-stats] [-verbose] [path...]
+
+Index builds or updates the trigram index used by codesearch search.
+If no paths are given, it updates existing index roots.
+`
+
+func RunIndex(args []string, stdout, stderr io.Writer) int {
+	flags := flag.NewFlagSet("index", flag.ContinueOnError)
+	flags.SetOutput(stderr)
+
+	listFlag := flags.Bool("list", false, "list indexed paths and exit")
+	resetFlag := flags.Bool("reset", false, "discard existing index")
+	verboseFlag := flags.Bool("verbose", false, "print extra information")
+	cpuProfile := flags.String("cpuprofile", "", "write cpu profile to this file")
+	checkFlag := flags.Bool("check", false, "check index is well-formatted")
+	zipFlag := flags.Bool("zip", false, "index content in zip files")
+	statsFlag := flags.Bool("stats", false, "print index size statistics")
+
+	flags.Usage = func() {
+		fmt.Fprint(stderr, indexUsageMessage)
+		flags.PrintDefaults()
+	}
+
+	if err := flags.Parse(args); err != nil {
+		if err == flag.ErrHelp {
+			return 0
+		}
+		return 2
+	}
+
+	logger := log.New(stderr, "codesearch index: ", 0)
+
+	if *listFlag {
+		ix := index.Open(index.File())
+		if *checkFlag {
+			if err := ix.Check(); err != nil {
+				logger.Println(err)
+				return 2
+			}
+		}
+		for p := range ix.Roots().All() {
+			fmt.Fprintf(stdout, "%s\n", p)
+		}
+		return 0
+	}
+
+	if *cpuProfile != "" {
+		f, err := os.Create(*cpuProfile)
+		if err != nil {
+			logger.Println(err)
+			return 2
+		}
+		defer f.Close()
+		pprof.StartCPUProfile(f)
+		defer pprof.StopCPUProfile()
+	}
+
+	if *resetFlag && flags.NArg() == 0 {
+		os.Remove(index.File())
+		return 0
+	}
+
+	var roots []index.Path
+	if flags.NArg() == 0 {
+		ix := index.Open(index.File())
+		roots = slices.Collect(ix.Roots().All())
+	} else {
+		for _, arg := range flags.Args() {
+			a, err := filepath.Abs(arg)
+			if err != nil {
+				logger.Printf("%s: %s", arg, err)
+				continue
+			}
+			roots = append(roots, index.MakePath(a))
+		}
+		slices.SortFunc(roots, index.Path.Compare)
+	}
+
+	master := index.File()
+	if _, err := os.Stat(master); err != nil {
+		*resetFlag = true
+	}
+	file := master
+	if !*resetFlag {
+		file += "~"
+		if *checkFlag {
+			ix := index.Open(master)
+			if err := ix.Check(); err != nil {
+				logger.Println(err)
+				return 2
+			}
+		}
+	}
+
+	ix := index.Create(file)
+	ix.Verbose = *verboseFlag
+	ix.Zip = *zipFlag
+	ix.AddRoots(roots)
+	for _, root := range roots {
+		if *verboseFlag {
+			logger.Printf("index %s", root)
+		}
+		filepath.Walk(root.String(), func(path string, info os.FileInfo, err error) error {
+			if _, elem := filepath.Split(path); elem != "" {
+				if elem[0] == '.' || elem[0] == '#' || elem[0] == '~' || elem[len(elem)-1] == '~' {
+					if info != nil && info.IsDir() {
+						return filepath.SkipDir
+					}
+					return nil
+				}
+			}
+			if err != nil {
+				if *verboseFlag {
+					logger.Printf("%s: %s", path, err)
+				}
+				return nil
+			}
+			if info != nil && info.Mode()&os.ModeType == 0 {
+				if err := ix.AddFile(path); err != nil {
+					if *verboseFlag {
+						logger.Printf("%s: %s", path, err)
+					}
+					return nil
+				}
+			}
+			return nil
+		})
+	}
+	ix.Flush()
+
+	if !*resetFlag {
+		index.Merge(file+"~", master, file)
+		if *checkFlag {
+			ix := index.Open(file + "~")
+			if err := ix.Check(); err != nil {
+				logger.Println(err)
+				return 2
+			}
+		}
+		os.Remove(file)
+		os.Rename(file+"~", master)
+	} else {
+		if *checkFlag {
+			ix := index.Open(file)
+			if err := ix.Check(); err != nil {
+				logger.Println(err)
+				return 2
+			}
+		}
+	}
+
+	if *statsFlag {
+		ix := index.Open(master)
+		ix.PrintStats()
+	}
+	return 0
+}
+```
+
+- [ ] **Step 3: Implement `cmd/codesearch/search.go`**
+
+Create `cmd/codesearch/search.go`:
+```go
+// Copyright 2011 The Go Authors.  All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE file.
+
+package main
+
+import (
+	"archive/zip"
+	"bytes"
+	"flag"
+	"fmt"
+	"io"
+	"log"
+	"os"
+	"runtime/pprof"
+	"strings"
+
+	"github.com/google/codesearch/index"
+	"github.com/google/codesearch/regexp"
+)
+
+var searchUsageMessage = `usage: codesearch search [-c] [-f fileregexp] [-h] [-i] [-l] [-n] [-A n] [-B n] [-C n] [-html] [-brute] regexp
+
+Search behaves like grep over all indexed files, searching for RE2 regexp.
+`
+
+func RunSearch(args []string, stdout, stderr io.Writer) int {
+	flags := flag.NewFlagSet("search", flag.ContinueOnError)
+	flags.SetOutput(stderr)
+
+	g := regexp.Grep{
+		Stdout: stdout,
+		Stderr: stderr,
+	}
+	g.AddFlagsToSet(flags)
+
+	fFlag := flags.String("f", "", "search only files with names matching this regexp")
+	iFlag := flags.Bool("i", false, "case-insensitive search")
+	htmlFlag := flags.Bool("html", false, "print HTML output")
+	verboseFlag := flags.Bool("verbose", false, "print extra information")
+	bruteFlag := flags.Bool("brute", false, "brute force - search all files in index")
+	cpuProfile := flags.String("cpuprofile", "", "write cpu profile to this file")
+
+	flags.Usage = func() {
+		fmt.Fprint(stderr, searchUsageMessage)
+		flags.PrintDefaults()
+	}
+
+	if err := flags.Parse(args); err != nil {
+		if err == flag.ErrHelp {
+			return 0
+		}
+		return 2
+	}
+	if *htmlFlag {
+		g.HTML = true
+	}
+	remaining := flags.Args()
+	if len(remaining) != 1 {
+		flags.Usage()
+		return 2
+	}
+
+	logger := log.New(stderr, "codesearch search: ", 0)
+
+	if *cpuProfile != "" {
+		f, err := os.Create(*cpuProfile)
+		if err != nil {
+			logger.Println(err)
+			return 2
+		}
+		defer f.Close()
+		pprof.StartCPUProfile(f)
+		defer pprof.StopCPUProfile()
+	}
+
+	pat := "(?m)" + remaining[0]
+	if *iFlag {
+		pat = "(?i)" + pat
+	}
+	re, err := regexp.Compile(pat)
+	if err != nil {
+		logger.Println(err)
+		return 2
+	}
+	g.Regexp = re
+
+	var fre *regexp.Regexp
+	if *fFlag != "" {
+		fre, err = regexp.Compile(*fFlag)
+		if err != nil {
+			logger.Println(err)
+			return 2
+		}
+	}
+
+	q := index.RegexpQuery(re.Syntax)
+	if *verboseFlag {
+		logger.Printf("query: %s\n", q)
+	}
+
+	ixFile := index.File()
+	if _, err := os.Stat(ixFile); err != nil {
+		logger.Printf("index file %s not found; run 'codesearch index' first\n", ixFile)
+		return 2
+	}
+
+	ix := index.Open(ixFile)
+	ix.Verbose = *verboseFlag
+	var post []int
+	if *bruteFlag {
+		post = ix.PostingQuery(&index.Query{Op: index.QAll})
+	} else {
+		post = ix.PostingQuery(q)
+	}
+
+	if *verboseFlag {
+		logger.Printf("post query identified %d possible files\n", len(post))
+	}
+
+	if fre != nil {
+		fnames := make([]int, 0, len(post))
+		for _, fileid := range post {
+			name := ix.Name(fileid)
+			if fre.MatchString(name.String(), true, true) < 0 {
+				continue
+			}
+			fnames = append(fnames, fileid)
+		}
+		if *verboseFlag {
+			logger.Printf("filename regexp matched %d files\n", len(fnames))
+		}
+		post = fnames
+	}
+
+	var (
+		zipFile   string
+		zipReader *zip.ReadCloser
+		zipMap    map[string]*zip.File
+	)
+
+	for _, fileid := range post {
+		name := ix.Name(fileid).String()
+		if g.L && (pat == "(?m)" || pat == "(?i)(?m)") {
+			g.Reader(bytes.NewReader(nil), name)
+			continue
+		}
+		file, err := os.Open(name)
+		if err != nil {
+			if i := strings.Index(name, ".zip\x01"); i >= 0 {
+				zfile, zname := name[:i+4], name[i+5:]
+				if zfile != zipFile {
+					if zipReader != nil {
+						zipReader.Close()
+						zipMap = nil
+					}
+					zipFile = zfile
+					zipReader, err = zip.OpenReader(zfile)
+					if err != nil {
+						zipReader = nil
+					}
+					if zipReader != nil {
+						zipMap = make(map[string]*zip.File)
+						for _, zf := range zipReader.File {
+							zipMap[zf.Name] = zf
+						}
+					}
+				}
+				zf := zipMap[zname]
+				if zf != nil {
+					r, err := zf.Open()
+					if err != nil {
+						continue
+					}
+					g.Reader(r, name)
+					r.Close()
+					continue
+				}
+			}
+			continue
+		}
+		g.Reader(file, name)
+		file.Close()
+	}
+
+	if zipReader != nil {
+		zipReader.Close()
+	}
+
+	if !g.Match {
+		return 1
+	}
+	return 0
+}
+```
+
+- [ ] **Step 4: Add `AddFlagsToSet` helper to `regexp/match.go`**
+
+Modify `regexp/match.go` to add `AddFlagsToSet(flags *flag.FlagSet)` so custom FlagSets can be used:
+```go
+// AddFlagsToSet adds the standard flags to the given FlagSet.
+func (g *Grep) AddFlagsToSet(flags *flag.FlagSet) {
+	flags.BoolVar(&g.C, "c", false, "print only a count of matching lines")
+	flags.BoolVar(&g.H, "h", false, "suppress the prefixing of file names on output")
+	flags.BoolVar(&g.L, "l", false, "print only names of FILEs containing matches")
+	flags.BoolVar(&g.N, "n", false, "print line number with output lines")
+	flags.IntVar(&g.A, "A", 0, "print `num` lines of trailing context")
+	flags.IntVar(&g.B, "B", 0, "print `num` lines of leading context")
+	flags.IntVar(&g.C_, "C", 0, "print `num` lines of output context")
+}
+
+// AddVFlagToSet adds the -v flag to the given FlagSet.
+func (g *Grep) AddVFlagToSet(flags *flag.FlagSet) {
+	flags.BoolVar(&g.V, "v", false, "select non-matching lines")
+}
+```
+
+- [ ] **Step 5: Implement `cmd/codesearch/grep.go`**
+
+Create `cmd/codesearch/grep.go`:
+```go
+// Copyright 2011 The Go Authors.  All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE file.
+
+package main
+
+import (
+	"flag"
+	"fmt"
+	"io"
+	"log"
+	"os"
+	"runtime/pprof"
+
+	"github.com/google/codesearch/regexp"
+)
+
+var grepUsageMessage = `usage: codesearch grep [-c] [-h] [-i] [-l] [-n] [-v] [-A n] [-B n] [-C n] regexp [file...]
+
+Grep behaves like grep, searching for RE2 regexp without an index.
+`
+
+func RunGrep(args []string, stdout, stderr io.Writer, stdin io.Reader) int {
+	flags := flag.NewFlagSet("grep", flag.ContinueOnError)
+	flags.SetOutput(stderr)
+
+	var g regexp.Grep
+	g.AddFlagsToSet(flags)
+	g.AddVFlagToSet(flags)
+	g.Stdout = stdout
+	g.Stderr = stderr
+
+	iflag := flags.Bool("i", false, "case-insensitive match")
+	cpuProfile := flags.String("cpuprofile", "", "write cpu profile to this file")
+
+	flags.Usage = func() {
+		fmt.Fprint(stderr, grepUsageMessage)
+		flags.PrintDefaults()
+	}
+
+	if err := flags.Parse(args); err != nil {
+		if err == flag.ErrHelp {
+			return 0
+		}
+		return 2
+	}
+
+	rem := flags.Args()
+	if len(rem) == 0 {
+		flags.Usage()
+		return 2
+	}
+
+	logger := log.New(stderr, "codesearch grep: ", 0)
+
+	if *cpuProfile != "" {
+		f, err := os.Create(*cpuProfile)
+		if err != nil {
+			logger.Println(err)
+			return 2
+		}
+		defer f.Close()
+		pprof.StartCPUProfile(f)
+		defer pprof.StopCPUProfile()
+	}
+
+	pat := "(?m)" + rem[0]
+	if *iflag {
+		pat = "(?i)" + pat
+	}
+	re, err := regexp.Compile(pat)
+	if err != nil {
+		logger.Println(err)
+		return 2
+	}
+	g.Regexp = re
+
+	if len(rem) == 1 {
+		g.Reader(stdin, "")
+	} else {
+		for _, arg := range rem[1:] {
+			g.File(arg)
+		}
+	}
+
+	if !g.Match {
+		return 1
+	}
+	return 0
+}
+```
+
+- [ ] **Step 6: Implement `cmd/codesearch/web.go`**
+
+Create `cmd/codesearch/web.go`:
+```go
+// Copyright 2020 The Go Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE file.
+
+package main
+
+import (
+	"archive/zip"
+	"bytes"
+	"embed"
+	"flag"
+	"fmt"
+	"html"
+	"io"
+	"io/fs"
+	"log"
+	"net/http"
+	"os"
+	"path"
+	"path/filepath"
+	"strings"
+	"time"
+	"unicode/utf8"
+
+	"github.com/google/codesearch/index"
+	"github.com/google/codesearch/regexp"
+)
+
+//go:embed _static
+var static embed.FS
+
+var webUsageMessage = `usage: codesearch web [-http addr] [-verbose]
+
+Web starts a local HTTP server providing a web search interface.
+`
+
+func RunWeb(args []string, stdout, stderr io.Writer) int {
+	flags := flag.NewFlagSet("web", flag.ContinueOnError)
+	flags.SetOutput(stderr)
+
+	httpAddr := flags.String("http", "localhost:2473", "HTTP service address")
+	verboseFlag := flags.Bool("verbose", false, "print extra information")
+
+	flags.Usage = func() {
+		fmt.Fprint(stderr, webUsageMessage)
+		flags.PrintDefaults()
+	}
+
+	if err := flags.Parse(args); err != nil {
+		if err == flag.ErrHelp {
+			return 0
+		}
+		return 2
+	}
+
+	mux := http.NewServeMux()
+	mux.Handle("/_static/", http.FileServer(http.FS(static)))
+	mux.HandleFunc("/show/", showHandler)
+	mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
+		homeHandler(w, r, *verboseFlag)
+	})
+
+	fmt.Fprintf(stdout, "Serving Code Search web UI at http://%s\n", *httpAddr)
+	if err := http.ListenAndServe(*httpAddr, mux); err != nil {
+		fmt.Fprintf(stderr, "web server error: %v\n", err)
+		return 2
+	}
+	return 0
+}
+
+func homeHandler(w http.ResponseWriter, r *http.Request, verbose bool) {
+	q := r.FormValue("q")
+	fpat := r.FormValue("f")
+	if r.URL.Path != "/" || (q == "" && fpat == "") {
+		w.Header().Set("Content-Type", "text/html; charset=utf-8")
+		w.Write([]byte(homePageHTML))
+		return
+	}
+
+	start := time.Now()
+	var (
+		re  *regexp.Regexp
+		fre *regexp.Regexp
+		err error
+	)
+	if q != "" {
+		re, err = regexp.Compile("(?m)" + q)
+		if err != nil {
+			http.Error(w, err.Error(), 400)
+			return
+		}
+	}
+	if fpat != "" {
+		fre, err = regexp.Compile(fpat)
+		if err != nil {
+			http.Error(w, err.Error(), 400)
+			return
+		}
+	}
+
+	ix := index.Open(index.File())
+	ix.Verbose = verbose
+	var post []int
+	if re != nil {
+		qtrigram := index.RegexpQuery(re.Syntax)
+		post = ix.PostingQuery(qtrigram)
+	} else {
+		post = ix.PostingQuery(&index.Query{Op: index.QAll})
+	}
+
+	if fre != nil {
+		fnames := make([]int, 0, len(post))
+		for _, fileid := range post {
+			name := ix.Name(fileid)
+			if fre.MatchString(name.String(), true, true) < 0 {
+				continue
+			}
+			fnames = append(fnames, fileid)
+		}
+		post = fnames
+	}
+
+	var buf bytes.Buffer
+	g := regexp.Grep{
+		Stdout: &buf,
+		Regexp: re,
+		HTML:   true,
+	}
+
+	for _, fileid := range post {
+		name := ix.Name(fileid).String()
+		file, err := os.Open(name)
+		if err != nil {
+			continue
+		}
+		g.Reader(file, name)
+		file.Close()
+	}
+
+	elapsed := time.Since(start)
+	w.Header().Set("Content-Type", "text/html; charset=utf-8")
+	fmt.Fprintf(w, "

Found %d matching files in %v.

\n", len(post), elapsed) + w.Write(buf.Bytes()) +} + +func showHandler(w http.ResponseWriter, r *http.Request) { + file := strings.TrimPrefix(r.URL.Path, "/show/") + data, err := os.ReadFile(file) + if err != nil { + http.NotFound(w, r) + return + } + w.Header().Set("Content-Type", "text/plain; charset=utf-8") + w.Write(data) +} + +var homePageHTML = ` + + +Code Search + + + + + + +` +``` + +- [ ] **Step 7: Verify compilation of `cmd/codesearch` files** + +Run: `go build -o /dev/null ./cmd/codesearch/...` +Expected: PASS with exit code 0. + +- [ ] **Step 8: Commit** + +```bash +git add regexp/match.go cmd/codesearch/ +git commit -m "feat(cmd/codesearch): add index, search, grep, and web subcommand runners" +``` + +--- + +### Task 3: Implement Main Subcommand Dispatcher & Help Routing + +**Files:** +- Create: `cmd/codesearch/main.go` + +**Interfaces:** +- Consumes: `RunIndex`, `RunSearch`, `RunGrep`, `RunWeb` +- Produces: `main()` binary entry point + +- [ ] **Step 1: Implement `cmd/codesearch/main.go`** + +Create `cmd/codesearch/main.go`: +```go +// Copyright 2026 The Go Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +package main + +import ( + "fmt" + "io" + "os" +) + +var mainUsageMessage = `codesearch is a tool for indexing and searching source code using regular expressions. + +Usage: + codesearch [arguments] + +The commands are: + index build or update trigram index + search search indexed corpus for regular expression + grep search files or stdin directly without index + web start local search web server + version print version information + help print help for a command + +Default: + Invoking codesearch with search arguments or flags directly routes to 'search': + codesearch [flags] -> codesearch search [flags] + +Use "codesearch help " for more information about a command. +` + +var versionString = "codesearch 1.0.0 (Go 1.26+)" + +func Main(args []string, stdout, stderr io.Writer, stdin io.Reader) int { + if len(args) == 0 { + fmt.Fprint(stderr, mainUsageMessage) + return 2 + } + + cmd := args[0] + switch cmd { + case "index": + return RunIndex(args[1:], stdout, stderr) + case "search": + return RunSearch(args[1:], stdout, stderr) + case "grep": + return RunGrep(args[1:], stdout, stderr, stdin) + case "web": + return RunWeb(args[1:], stdout, stderr) + case "version", "-version", "--version": + fmt.Fprintf(stdout, "%s\n", versionString) + return 0 + case "help", "-help", "--help", "-h": + if len(args) > 1 { + sub := args[1] + switch sub { + case "index": + return RunIndex([]string{"-help"}, stdout, stderr) + case "search": + return RunSearch([]string{"-help"}, stdout, stderr) + case "grep": + return RunGrep([]string{"-help"}, stdout, stderr, stdin) + case "web": + return RunWeb([]string{"-help"}, stdout, stderr) + } + } + fmt.Fprint(stdout, mainUsageMessage) + return 0 + default: + // Default to 'search' subcommand + return RunSearch(args, stdout, stderr) + } +} + +func main() { + exitCode := Main(os.Args[1:], os.Stdout, os.Stderr, os.Stdin) + os.Exit(exitCode) +} +``` + +- [ ] **Step 2: Verify binary compilation** + +Run: `go build -o /tmp/codesearch ./cmd/codesearch` +Expected: PASS with exit code 0. + +- [ ] **Step 3: Commit** + +```bash +git add cmd/codesearch/main.go +git commit -m "feat(cmd/codesearch): add main subcommand router and search fallback" +``` + +--- + +### Task 4: Comprehensive End-to-End Integration Tests + +**Files:** +- Create: `cmd/codesearch/codesearch_test.go` + +**Interfaces:** +- Consumes: `Main`, `RunIndex`, `RunSearch`, `RunGrep` +- Produces: Test suite validating all CLI workflows + +- [ ] **Step 1: Implement `cmd/codesearch/codesearch_test.go`** + +Create `cmd/codesearch/codesearch_test.go`: +```go +package main + +import ( + "bytes" + "os" + "path/filepath" + "strings" + "testing" +) + +func TestCLI_HelpAndVersion(t *testing.T) { + var stdout, stderr bytes.Buffer + code := Main([]string{"version"}, &stdout, &stderr, nil) + if code != 0 || !strings.Contains(stdout.String(), "codesearch") { + t.Fatalf("version failed: code=%d, stdout=%s", code, stdout.String()) + } + + stdout.Reset() + code = Main([]string{"help"}, &stdout, &stderr, nil) + if code != 0 || !strings.Contains(stdout.String(), "The commands are:") { + t.Fatalf("help failed: code=%d, stdout=%s", code, stdout.String()) + } + + stdout.Reset() + code = Main([]string{"help", "index"}, &stdout, &stderr, nil) + if code != 0 { + t.Fatalf("help index failed: code=%d", code) + } +} + +func TestCLI_IndexAndSearchFlow(t *testing.T) { + tempDir := t.TempDir() + idxFile := filepath.Join(tempDir, "test.index") + t.Setenv("CSEARCHINDEX", idxFile) + + // Create sample source files + fileA := filepath.Join(tempDir, "alpha.go") + fileB := filepath.Join(tempDir, "beta.txt") + if err := os.WriteFile(fileA, []byte("package main\n\nfunc HelloSearch() string {\n\treturn \"codesearch\"\n}\n"), 0644); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(fileB, []byte("some documentation\nHelloSearch keyword inside\n"), 0644); err != nil { + t.Fatal(err) + } + + // 1. Run Index + var stdout, stderr bytes.Buffer + code := Main([]string{"index", tempDir}, &stdout, &stderr, nil) + if code != 0 { + t.Fatalf("index command failed: code=%d, stderr=%s", code, stderr.String()) + } + + // Verify index file exists + if _, err := os.Stat(idxFile); err != nil { + t.Fatalf("expected index file %s to exist: %v", idxFile, err) + } + + // 2. Run Search (explicit subcommand) + stdout.Reset() + stderr.Reset() + code = Main([]string{"search", "-n", "HelloSearch"}, &stdout, &stderr, nil) + if code != 0 { + t.Fatalf("search command failed: code=%d, stderr=%s", code, stderr.String()) + } + out := stdout.String() + if !strings.Contains(out, "alpha.go:3:func HelloSearch()") || !strings.Contains(out, "beta.txt:2:HelloSearch") { + t.Fatalf("unexpected search output:\n%s", out) + } + + // 3. Run Search with file filter (-f) + stdout.Reset() + stderr.Reset() + code = Main([]string{"search", "-f", `\.go$`, "HelloSearch"}, &stdout, &stderr, nil) + if code != 0 { + t.Fatalf("search -f failed: code=%d, stderr=%s", code, stderr.String()) + } + out = stdout.String() + if !strings.Contains(out, "alpha.go") || strings.Contains(out, "beta.txt") { + t.Fatalf("expected only alpha.go in filtered search, got:\n%s", out) + } + + // 4. Run Search with case-insensitivity (-i) + stdout.Reset() + stderr.Reset() + code = Main([]string{"search", "-i", "hellosearch"}, &stdout, &stderr, nil) + if code != 0 { + t.Fatalf("search -i failed: code=%d, stderr=%s", code, stderr.String()) + } + if !strings.Contains(stdout.String(), "HelloSearch") { + t.Fatalf("expected case-insensitive match, got:\n%s", stdout.String()) + } + + // 5. Run Default Search (without explicit 'search' subcommand) + stdout.Reset() + stderr.Reset() + code = Main([]string{"-n", "HelloSearch"}, &stdout, &stderr, nil) + if code != 0 { + t.Fatalf("default search failed: code=%d, stderr=%s", code, stderr.String()) + } + if !strings.Contains(stdout.String(), "alpha.go:3:func HelloSearch()") { + t.Fatalf("unexpected default search output:\n%s", stdout.String()) + } + + // 6. Search for non-existent term (expect exit code 1) + stdout.Reset() + stderr.Reset() + code = Main([]string{"search", "NonExistentPatternXYZ"}, &stdout, &stderr, nil) + if code != 1 { + t.Fatalf("expected exit code 1 for no matches, got %d", code) + } +} + +func TestCLI_Grep(t *testing.T) { + tempDir := t.TempDir() + filePath := filepath.Join(tempDir, "grep_target.txt") + content := "line 1: apple\nline 2: banana\nline 3: cherry\n" + if err := os.WriteFile(filePath, []byte(content), 0644); err != nil { + t.Fatal(err) + } + + // Grep on file + var stdout, stderr bytes.Buffer + code := Main([]string{"grep", "-n", "banana", filePath}, &stdout, &stderr, nil) + if code != 0 { + t.Fatalf("grep on file failed: code=%d, stderr=%s", code, stderr.String()) + } + if !strings.Contains(stdout.String(), "2:line 2: banana") { + t.Fatalf("unexpected grep output:\n%s", stdout.String()) + } + + // Grep on stdin + stdout.Reset() + stderr.Reset() + stdin := strings.NewReader("first line\ntarget line here\nthird line\n") + code = Main([]string{"grep", "target", ""}, &stdout, &stderr, stdin) + if code != 0 { + t.Fatalf("grep on stdin failed: code=%d, stderr=%s", code, stderr.String()) + } + if !strings.Contains(stdout.String(), "target line here") { + t.Fatalf("unexpected grep stdin output:\n%s", stdout.String()) + } +} +``` + +- [ ] **Step 2: Run the test suite** + +Run: `go test -v ./cmd/codesearch` +Expected: PASS all tests. + +- [ ] **Step 3: Commit** + +```bash +git add cmd/codesearch/codesearch_test.go +git commit -m "test(cmd/codesearch): add comprehensive end-to-end integration tests" +``` + +--- + +### Task 5: Refactor Legacy Wrappers, Makefile & Documentation + +**Files:** +- Modify: `cmd/cindex/cindex.go` +- Modify: `cmd/csearch/csearch.go` +- Modify: `cmd/cgrep/cgrep.go` +- Modify: `cmd/csweb/web.go` +- Modify: `Makefile` +- Modify: `skills/codesearch/SKILL.md` +- Modify: `README` + +**Interfaces:** +- Consumes: `cmd/codesearch` +- Produces: Updated build targets, symlinks, skill docs, and thin legacy wrappers + +- [ ] **Step 1: Update `Makefile`** + +Update `Makefile` to compile `codesearch`, create `cs` symlink, and build legacy targets: +```makefile +.PHONY: all build clean test + +all: build + +build: + @mkdir -p bin + go build -o bin/codesearch ./cmd/codesearch + ln -sf codesearch bin/cs + go build -o bin/cindex ./cmd/cindex + go build -o bin/csearch ./cmd/csearch + go build -o bin/cgrep ./cmd/cgrep + go build -o bin/csweb ./cmd/csweb + +test: + go test ./... + +clean: + rm -rf bin +``` + +- [ ] **Step 2: Update `skills/codesearch/SKILL.md`** + +Update the skill to highlight `codesearch` / `cs` as the primary unified tool while documenting legacy commands as shortcuts. + +- [ ] **Step 3: Update `README`** + +Add clear instructions explaining `codesearch` (and `cs`) usage for indexing and searching. + +- [ ] **Step 4: Run full project verification** + +Run: +```bash +make clean && make build && go test -v ./... && go vet ./... +``` +Expected: All binaries built, tests passing, zero vet warnings. + +- [ ] **Step 5: Commit** + +```bash +git add Makefile skills/codesearch/SKILL.md README +git commit -m "build: update Makefile and documentation for unified codesearch CLI" +``` diff --git a/docs/superpowers/specs/2026-08-25-codesearch-cli-design.md b/docs/superpowers/specs/2026-08-25-codesearch-cli-design.md new file mode 100644 index 0000000000..1c538abeca --- /dev/null +++ b/docs/superpowers/specs/2026-08-25-codesearch-cli-design.md @@ -0,0 +1,178 @@ +# Unified Code Search CLI Specification (`codesearch` / `cs`) + +- **Date**: 2026-08-25 +- **Status**: Proposed / Approved in Brainstorming + +## 1. Overview & Objectives + +Currently, Google Code Search is distributed as separate standalone command-line binaries in `cmd/`: +- `cindex`: builds and maintains trigram indexes +- `csearch`: executes indexed regular expression searches +- `cgrep`: performs unindexed direct regex search over files/stdin +- `csweb`: serves an HTTP web interface for indexed searching + +This specification consolidates these tools into a single, unified CLI executable: `codesearch` (with `cs` alias/symlink). The unified tool preserves all existing functionality and flags, adds intuitive subcommand dispatching with an ergonomic fallback to search, keeps the project zero-dependency (using only Go's standard library), and retains backward-compatible wrappers for legacy tooling. + +--- + +## 2. CLI Interface Design + +### 2.1 Binary Names +- Primary binary: `codesearch` +- Convenient short alias / symlink: `cs` + +### 2.2 Subcommands + +``` +codesearch [flags] [arguments...] +``` + +#### Subcommand: `index` +Index files or directories into the trigram index file. +- **Syntax**: `codesearch index [flags] [paths...]` +- **Flags**: + - `-list`: List indexed directory roots and exit. + - `-reset`: Discard existing index or reset before indexing. + - `-check`: Verify index is well-formatted. + - `-zip`: Index contents of `.zip` files. + - `-stats`: Print index size statistics. + - `-verbose`: Print extra progress information. + - `-cpuprofile `: Write CPU profile to specified file. +- **Behavior**: + - If no paths are provided and `-reset` is not specified, reindexes existing roots found in the index. + - If paths are specified, translates them to absolute paths and indexes them. + +#### Subcommand: `search` (or Default Action) +Search indexed code for an RE2 regular expression. +- **Syntax**: `codesearch search [flags] ` +- **Flags**: + - `-f `: Restrict search to files whose paths match the given RE2 regexp. + - `-i`: Case-insensitive search. + - `-l`: List matching file names only. + - `-c`: Print count of matching lines per file. + - `-h`: Suppress file name headers in output. + - `-n`: Print 1-based line numbers. + - `-A `, `-B `, `-C `: Print trailing, leading, or surrounding context lines. + - `-brute`: Brute-force search (search every file in the index, skipping trigram filtering). + - `-html`: Format output as HTML. + - `-verbose`: Print extra debugging info (e.g. query decomposition, candidate file counts). + - `-cpuprofile `: Write CPU profile to specified file. +- **Exit Codes**: + - `0`: Match found. + - `1`: No match found. + - `2`: Error / invalid flags / query error. + +#### Subcommand: `grep` +Search files or standard input directly without requiring an index. +- **Syntax**: `codesearch grep [flags] [files...]` +- **Flags**: + - `-i`: Case-insensitive match. + - `-l`: List file names with matches only. + - `-c`: Print count of matching lines. + - `-h`: Suppress file names. + - `-n`: Print line numbers. + - `-v`: Invert match (select non-matching lines). + - `-A `, `-B `, `-C `: Context lines. + - `-cpuprofile `: Write CPU profile. +- **Behavior**: + - If no files are given (or 1 argument total), reads from standard input. + +#### Subcommand: `web` +Start the embedded HTTP server to provide a web-based search UI. +- **Syntax**: `codesearch web [flags]` +- **Flags**: + - `-http `: Address/port to listen on (default `:2473`). + - `-verbose`: Print HTTP request / search logging. + +#### Subcommands: `help` and `version` +- `codesearch help`: Prints top-level usage guide with summaries of all subcommands. +- `codesearch help ` / `codesearch -h`: Subcommand-specific help. +- `codesearch version` / `codesearch -version`: Displays version information. + +### 2.3 Ergonomic Default (Search Fallback) +If the first argument provided to `codesearch` is not one of the registered subcommands (`index`, `search`, `grep`, `web`, `help`, `version`), the invocation is automatically routed to `search`: +```bash +codesearch "func main" # equivalent to: codesearch search "func main" +codesearch -i -n "func main" # equivalent to: codesearch search -i -n "func main" +codesearch -f '.*\.go$' "Run" # equivalent to: codesearch search -f '.*\.go$' "Run" +``` + +--- + +## 3. Architecture & Code Layout + +### 3.1 Directory Structure + +``` +cmd/ +├── codesearch/ # Primary unified CLI +│ ├── main.go # Entry point & subcommand dispatcher +│ ├── index.go # 'index' command implementation +│ ├── search.go # 'search' command implementation +│ ├── grep.go # 'grep' command implementation +│ ├── web.go # 'web' command implementation +│ ├── _static/ # Static assets for web UI (viewer.js, viewer.css) +│ └── codesearch_test.go # Comprehensive CLI end-to-end tests +├── cindex/ # Backward-compatibility wrapper +│ └── cindex.go # Calls cmd/codesearch index logic +├── csearch/ # Backward-compatibility wrapper +│ └── csearch.go # Calls cmd/codesearch search logic +├── cgrep/ # Backward-compatibility wrapper +│ └── cgrep.go # Calls cmd/codesearch grep logic +└── csweb/ # Backward-compatibility wrapper + └── web.go # Calls cmd/codesearch web logic +``` + +### 3.2 Subcommand Handler Interface +Each subcommand implementation in `cmd/codesearch/` will expose a clean execution function taking streams and arguments: +```go +type Runner func(args []string, stdout, stderr io.Writer, stdin io.Reader) int +``` +- Standardized return values ($0 = \text{success/match}$, $1 = \text{no match}$, $2 = \text{error}$). +- Subcommands avoid direct `os.Exit` inside logic so they can be tested in-process with `bytes.Buffer`. +- Subcommand flag sets are constructed per-invocation using `flag.NewFlagSet(name, flag.ContinueOnError)` to prevent global state leaks during testing. + +### 3.3 Static Assets Embedding +`cmd/codesearch/web.go` will embed static assets directly using Go's `//go:embed _static`: +```go +//go:embed _static +var static embed.FS +``` + +--- + +## 4. Quality & Compatibility Improvements + +### 4.1 Go Vet Fixes +Update all `fmt.Fprintf(os.Stderr, usageMessage)` occurrences across legacy commands to `fmt.Fprint(os.Stderr, usageMessage)` or `fmt.Fprintf(os.Stderr, "%s", usageMessage)` to ensure `go vet` and `go test ./...` pass cleanly on modern Go toolchains. + +### 4.2 Makefile & Build Artifacts +Update `Makefile`: +- `make build`: Compiles `bin/codesearch`, creates symlink `bin/cs -> codesearch`, and compiles legacy binaries into `bin/`. +- `make clean`: Removes `bin/`. + +### 4.3 Skill & Documentation Updates +- Update `skills/codesearch/SKILL.md` to document the unified `codesearch` / `cs` tool as primary, with subcommands `index`, `search`, `grep`, and `web`. +- Update `README` with quick-start instructions for `codesearch`. + +--- + +## 5. Verification & Test Plan + +1. **Unit & Integration Tests (`cmd/codesearch/codesearch_test.go`)**: + - `TestCLI_HelpAndVersion`: Tests `-h`, `--help`, `help`, `help `, `-version`. + - `TestCLI_IndexAndSearch`: + - Creates temporary directory structure with sample files. + - Runs `codesearch index` pointing to temp index. + - Runs `codesearch search` with standard patterns, `-i`, `-n`, `-f` filters. + - Validates exit codes ($0$ on match, $1$ on mismatch). + - `TestCLI_DefaultSearch`: + - Verifies `codesearch "pattern"` finds matches without explicit `search` subcommand. + - `TestCLI_Grep`: + - Tests unindexed searching over explicit files and standard input (`stdin`). + - `TestCLI_LegacyWrappers`: + - Tests that `cindex`, `csearch`, `cgrep` wrappers execute successfully and match behavior. +2. **Build and Vet Verification**: + - `go test ./...` passes without errors or warnings. + - `go vet ./...` passes cleanly. + - `make build` produces functional binaries in `bin/`. diff --git a/index/bench_test.go b/index/bench_test.go new file mode 100644 index 0000000000..0e8d5f33ba --- /dev/null +++ b/index/bench_test.go @@ -0,0 +1,106 @@ +// Copyright 2026 The Go Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +package index + +import ( + "bytes" + "fmt" + "math/rand/v2" + "testing" +) + +func BenchmarkScanASCIIChunk(b *testing.B) { + sizes := []struct { + name string + size int + }{ + {"1KB", 1024}, + {"64KB", 64 * 1024}, + {"1MB", 1024 * 1024}, + } + for _, tc := range sizes { + data := make([]byte, tc.size) + for i := range data { + if i%40 == 0 { + data[i] = '\n' + } else { + data[i] = byte('A' + (i % 26)) + } + } + + b.Run("PureASCII/"+tc.name, func(b *testing.B) { + b.SetBytes(int64(tc.size)) + b.ReportAllocs() + b.ResetTimer() + for range b.N { + _, _ = scanASCIIChunk(data) + } + }) + } +} + +func BenchmarkIndexWriterAdd(b *testing.B) { + // Create sample text file content + var buf bytes.Buffer + for i := range 1000 { + fmt.Fprintf(&buf, "func processItem_%d(val int) string {\n\treturn fmt.Sprintf(\"result_%%d\", val*%d)\n}\n\n", i, i) + } + content := buf.Bytes() + + b.SetBytes(int64(len(content))) + b.ReportAllocs() + b.ResetTimer() + + for range b.N { + ix := Create("") + _ = ix.Add("sample.go", bytes.NewReader(content)) + } +} + +func BenchmarkPostingAnd(b *testing.B) { + pcg := rand.New(rand.NewPCG(1234, 5678)) + listSize := 20000 + list := make([]int, listSize) + cur := 0 + for i := range list { + cur += pcg.IntN(4) + 1 + list[i] = cur + } + + targets := make([]int, 1000) + cur = 0 + for i := range targets { + cur += pcg.IntN(80) + 1 + targets[i] = cur + } + + b.ReportAllocs() + b.ResetTimer() + + for range b.N { + res := make([]int, 0, len(targets)) + i := 0 + n := len(list) + for _, fileid := range targets { + for i+64 <= n && list[i+63] < fileid { + i += 64 + } + for i+16 <= n && list[i+15] < fileid { + i += 16 + } + for i+4 <= n && list[i+3] < fileid { + i += 4 + } + for i < n && list[i] < fileid { + i++ + } + if i < n && list[i] == fileid { + res = append(res, fileid) + i++ + } + } + _ = res + } +} diff --git a/index/close_test.go b/index/close_test.go new file mode 100644 index 0000000000..5bb4b81223 --- /dev/null +++ b/index/close_test.go @@ -0,0 +1,49 @@ +// Copyright 2026 The Go Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +package index + +import ( + "os" + "path/filepath" + "strings" + "testing" +) + +func TestIndexResourceCleanup(t *testing.T) { + file := filepath.Join(t.TempDir(), "index") + w := Create(file) + defer w.Close() + if err := w.Add("source.txt", strings.NewReader("needle\n")); err != nil { + t.Fatal(err) + } + w.Flush() + if err := w.Close(); err != nil { + t.Fatal(err) + } + if err := w.Close(); err != nil { + t.Fatalf("second writer close: %v", err) + } + for _, b := range []*Buffer{w.nameData, w.nameIndex, w.postFile, w.postIndex} { + if _, err := os.Stat(b.name); !os.IsNotExist(err) { + t.Fatalf("temporary file remains: %s: %v", b.name, err) + } + } + ix := Open(file) + if got := ix.Name(0).String(); got != "source.txt" { + t.Fatalf("name=%q", got) + } + if err := ix.Close(); err != nil { + t.Fatal(err) + } + if err := ix.Close(); err != nil { + t.Fatalf("second index close: %v", err) + } + if ix.data.d != nil || ix.data.f != nil { + t.Fatal("index retains resources after close") + } + if err := os.Rename(file, file+".closed"); err != nil { + t.Fatal(err) + } +} diff --git a/index/merge.go b/index/merge.go index 55c5456a64..40f937d556 100644 --- a/index/merge.go +++ b/index/merge.go @@ -50,7 +50,9 @@ var writeVersion = 2 // for a path, src2 is assumed to be newer and is given preference. func Merge(dst, src1, src2 string) { ix1 := Open(src1) + defer ix1.Close() ix2 := Open(src2) + defer ix2.Close() // Build fileid maps. var i1, i2, new int @@ -118,6 +120,7 @@ func Merge(dst, src1, src2 string) { numName := new ix := bufCreate(dst) + defer ix.file.Close() if writeVersion == 1 { ix.WriteString(magicV1) } else { @@ -154,6 +157,7 @@ func Merge(dst, src1, src2 string) { ix.Align(16) nameData := ix.Offset() nameIndexFile := bufCreate("") + defer func() { nameIndexFile.file.Close(); os.Remove(nameIndexFile.name) }() start := ix.Offset() names := NewPathWriter(ix, nameIndexFile, writeVersion, nameGroupSize) m1 := map1 @@ -194,6 +198,7 @@ func Merge(dst, src1, src2 string) { r1.init(ix1, map1) r2.init(ix2, map2) postIndexFile := bufCreate("") + defer func() { postIndexFile.file.Close(); os.Remove(postIndexFile.name) }() w.init(ix, postIndexFile) old1, old2 := uint32(0), uint32(0) for { diff --git a/index/mmap_bsd.go b/index/mmap_bsd.go index ce1de14927..4ece0f50d9 100644 --- a/index/mmap_bsd.go +++ b/index/mmap_bsd.go @@ -2,6 +2,7 @@ // Use of this source code is governed by a BSD-style // license that can be found in the LICENSE file. +//go:build darwin || freebsd || openbsd || netbsd // +build darwin freebsd openbsd netbsd package index @@ -37,3 +38,18 @@ func mmapFile(f *os.File) mmapData { } return mmapData{f, data[:n]} } + +func (m *mmapData) close() error { + if m.d != nil { + if err := syscall.Munmap(m.d[:cap(m.d)]); err != nil { + return err + } + m.d = nil + } + if m.f == nil { + return nil + } + err := m.f.Close() + m.f = nil + return err +} diff --git a/index/mmap_linux.go b/index/mmap_linux.go index a3c961dbb3..39d9971c7c 100644 --- a/index/mmap_linux.go +++ b/index/mmap_linux.go @@ -29,3 +29,18 @@ func mmapFile(f *os.File) mmapData { } return mmapData{f, data[:n]} } + +func (m *mmapData) close() error { + if m.d != nil { + if err := syscall.Munmap(m.d[:cap(m.d)]); err != nil { + return err + } + m.d = nil + } + if m.f == nil { + return nil + } + err := m.f.Close() + m.f = nil + return err +} diff --git a/index/mmap_windows.go b/index/mmap_windows.go index 38ae27471e..91f4cd2323 100644 --- a/index/mmap_windows.go +++ b/index/mmap_windows.go @@ -29,9 +29,25 @@ func mmapFile(f *os.File) mmapData { } addr, err := syscall.MapViewOfFile(h, syscall.FILE_MAP_READ, 0, 0, 0) + syscall.CloseHandle(h) if err != nil { log.Fatalf("MapViewOfFile %s: %v", f.Name(), err) } data := (*[1 << 30]byte)(unsafe.Pointer(addr)) return mmapData{f, data[:size]} } + +func (m *mmapData) close() error { + if len(m.d) > 0 { + if err := syscall.UnmapViewOfFile(uintptr(unsafe.Pointer(&m.d[0]))); err != nil { + return err + } + m.d = nil + } + if m.f == nil { + return nil + } + err := m.f.Close() + m.f = nil + return err +} diff --git a/index/read.go b/index/read.go index 1c8715b011..afb160a27b 100644 --- a/index/read.go +++ b/index/read.go @@ -148,6 +148,7 @@ import ( "bytes" "encoding/binary" "fmt" + "io" "iter" "log" "os" @@ -185,14 +186,29 @@ type Index struct { numPostBlock int } +func (ix *Index) PrintStatsTo(w io.Writer) { + if ix.version == 1 { + fmt.Fprintf(w, "%d name list (%d names)\n", ix.postData-ix.nameData, ix.numName) + fmt.Fprintf(w, "%d posting lists (%d trigrams)\n", ix.nameIndex-ix.postData, ix.numPost) + fmt.Fprintf(w, "%d name index\n", ix.postIndex-ix.nameIndex) + fmt.Fprintf(w, "%d posting index\n", len(ix.data.d)-len(trailerMagicV1)-5*4-int(ix.postIndex)) + return + } + fmt.Fprintf(w, "%d path list (%d paths)\n", ix.nameData-ix.pathData, ix.numPath) + fmt.Fprintf(w, "%d name list (%d names)\n", ix.postData-ix.nameData, ix.numName) + fmt.Fprintf(w, "%d posting lists (%d trigrams)\n", ix.nameIndex-ix.postData, ix.numPost) + fmt.Fprintf(w, "%d name index\n", ix.postIndex-ix.nameIndex) + fmt.Fprintf(w, "%d posting index\n", ix.numPostBlock*postBlockSize) +} + func (ix *Index) PrintStats() { - fmt.Printf("%d path list (%d paths)\n", ix.nameData-ix.pathData, ix.numPath) - fmt.Printf("%d name list (%d names)\n", ix.postData-ix.nameData, ix.numName) - fmt.Printf("%d posting lists (%d trigrams)\n", ix.nameIndex-ix.postData, ix.numPost) - fmt.Printf("%d name index\n", ix.postIndex-ix.nameIndex) - fmt.Printf("%d posting index\n", ix.numPostBlock*postBlockSize) + ix.PrintStatsTo(os.Stdout) } +// Close releases the mapped index and its file descriptor. All readers and +// queries must have finished before Close; the index cannot be used afterward. +func (ix *Index) Close() error { return ix.data.close() } + func Open(file string) *Index { mm := mmap(file) ix := &Index{name: file, data: mm} @@ -441,7 +457,17 @@ func (r *postReader) next() bool { r.fileid += delta if r.restrict != nil { i := 0 - for i < len(r.restrict) && r.restrict[i] < r.fileid { + n := len(r.restrict) + for i+64 <= n && r.restrict[i+63] < r.fileid { + i += 64 + } + for i+16 <= n && r.restrict[i+15] < r.fileid { + i += 16 + } + for i+4 <= n && r.restrict[i+3] < r.fileid { + i += 4 + } + for i < n && r.restrict[i] < r.fileid { i++ } r.restrict = r.restrict[i:] @@ -521,12 +547,22 @@ func (ix *Index) postingAnd(list []int, trigram uint32, restrict []int) []int { r.init(ix, trigram, restrict) x := list[:0] i := 0 + n := len(list) for r.next() { fileid := r.fileid - for i < len(list) && list[i] < fileid { + for i+64 <= n && list[i+63] < fileid { + i += 64 + } + for i+16 <= n && list[i+15] < fileid { + i += 16 + } + for i+4 <= n && list[i+3] < fileid { + i += 4 + } + for i < n && list[i] < fileid { i++ } - if i < len(list) && list[i] == fileid { + if i < n && list[i] == fileid { x = append(x, fileid) i++ } diff --git a/index/read_test.go b/index/read_test.go index d25be6a6ce..6fd59e1cc6 100644 --- a/index/read_test.go +++ b/index/read_test.go @@ -5,8 +5,11 @@ package index import ( + "bytes" + "fmt" "os" "slices" + "strings" "testing" ) @@ -51,3 +54,27 @@ func TestTrivialPosting(t *testing.T) { t.Errorf("PostingList(Goo|Sea) = %v, want [1 2 3]", l) } } + +func TestPrintStats(t *testing.T) { + for v := 1; v <= 2; v++ { + t.Run(fmt.Sprintf("V%d", v), func(t *testing.T) { + old := writeVersion + writeVersion = v + defer func() { writeVersion = old }() + + f, _ := os.CreateTemp("", "index-test") + defer os.Remove(f.Name()) + buildIndex(f.Name(), nil, postFiles) + + ix := Open(f.Name()) + var buf bytes.Buffer + ix.PrintStatsTo(&buf) + out := buf.String() + for _, want := range []string{"name list", "posting lists", "trigrams", "name index", "posting index"} { + if !strings.Contains(out, want) { + t.Errorf("V%d PrintStatsTo missing %q, got:\n%s", v, want, out) + } + } + }) + } +} diff --git a/index/scan_purego.go b/index/scan_purego.go new file mode 100644 index 0000000000..a300c977a2 --- /dev/null +++ b/index/scan_purego.go @@ -0,0 +1,19 @@ +// Copyright 2026 The Go Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +//go:build !goexperiment.simd + +package index + +func scanASCIIChunk(buf []byte) (hasNul bool, hasNonASCII bool) { + for _, c := range buf { + if c == 0 { + return true, false + } + if c >= 0x80 { + hasNonASCII = true + } + } + return +} diff --git a/index/scan_simd.go b/index/scan_simd.go new file mode 100644 index 0000000000..12cf633293 --- /dev/null +++ b/index/scan_simd.go @@ -0,0 +1,58 @@ +// Copyright 2026 The Go Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +//go:build goexperiment.simd + +package index + +import "simd/archsimd" + +func scanASCIIChunk(buf []byte) (hasNul bool, hasNonASCII bool) { + aZero := archsimd.BroadcastUint8x16(0) + aHigh := archsimd.BroadcastUint8x16(0x80) + + var nulAcc archsimd.Uint8x16 + var nonAsciiAcc archsimd.Uint8x16 + var stored [16]uint8 + + b := buf + for len(b) >= 16 { + batchLen := min(len(b)/16, 250) + for range batchLen { + v := archsimd.LoadUint8x16(b[:16]) + mNul := v.Equal(aZero) + nulAcc = nulAcc.Or(mNul.ToInt8x16().ToBits()) + + mHigh := v.And(aHigh) + nonAsciiAcc = nonAsciiAcc.Or(mHigh) + + b = b[16:] + } + } + + nulAcc.Store(stored[:]) + for _, v := range stored { + if v != 0 { + return true, false + } + } + + nonAsciiAcc.Store(stored[:]) + for _, v := range stored { + if v != 0 { + hasNonASCII = true + break + } + } + + for _, c := range b { + if c == 0 { + return true, false + } + if c >= 0x80 { + hasNonASCII = true + } + } + return +} diff --git a/index/scan_test.go b/index/scan_test.go new file mode 100644 index 0000000000..6cbc3fa814 --- /dev/null +++ b/index/scan_test.go @@ -0,0 +1,41 @@ +// Copyright 2026 The Go Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +package index + +import "testing" + +func TestScanASCIIChunk(t *testing.T) { + testCases := []struct { + name string + data []byte + wantNul bool + wantNonASCI bool + }{ + {"empty", []byte{}, false, false}, + {"short ascii", []byte("hello world"), false, false}, + {"16 byte ascii", []byte("0123456789abcdef"), false, false}, + {"17 byte ascii", []byte("0123456789abcdefg"), false, false}, + {"short with nul", []byte("hello\x00world"), true, false}, + {"short with non-ascii", []byte("hello\x80world"), false, true}, + {"16 byte with nul at 0", []byte("\x00123456789abcdef"), true, false}, + {"16 byte with nul at 15", []byte("0123456789abcde\x00"), true, false}, + {"16 byte with non-ascii at 0", []byte("\x80123456789abcdef"), false, true}, + {"16 byte with non-ascii at 15", []byte("0123456789abcde\x80"), false, true}, + {"32 byte with nul in second block", []byte("0123456789abcdef0123456789abcd\x00"), true, false}, + {"32 byte with non-ascii in second block", []byte("0123456789abcdef0123456789abcd\xff"), false, true}, + {"non-ascii then nul", []byte("0123456789abcd\xff0123456789abcd\x00"), true, false}, + {"nul then non-ascii", []byte("0123456789abcd\x000123456789abcd\xff"), true, false}, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + gotNul, gotNonASCII := scanASCIIChunk(tc.data) + if gotNul != tc.wantNul || gotNonASCII != tc.wantNonASCI { + t.Fatalf("scanASCIIChunk(%q): got (%v, %v), want (%v, %v)", + tc.data, gotNul, gotNonASCII, tc.wantNul, tc.wantNonASCI) + } + }) + } +} diff --git a/index/write.go b/index/write.go index 185ed23938..4fba7ddac3 100644 --- a/index/write.go +++ b/index/write.go @@ -82,6 +82,25 @@ func Create(file string) *IndexWriter { return ix } +// Close releases the writer's files without flushing. Call Flush first to +// finish an index, or Close alone to abandon a failed build. +func (ix *IndexWriter) Close() error { + var first error + for _, b := range []*Buffer{ix.nameData, ix.nameIndex, ix.postFile, ix.postIndex, ix.main} { + if b.file == nil { + continue + } + if err := b.file.Close(); err != nil && first == nil { + first = err + } + b.file = nil + if b != ix.main { + os.Remove(b.name) + } + } + return first +} + // isValidName reports whether name is a valid name to store in the index. // We reject all control characters (bytes < ' ' aka 0x20) // because we use them for framing in the name format. @@ -223,10 +242,9 @@ func (ix *IndexWriter) add(name string, f io.Reader) error { linelen = 0 ) for { - tv = (tv << 8) & (1<<24 - 1) if i >= len(buf) { - n, err := f.Read(buf[:cap(buf)]) - if n == 0 { + nRead, err := f.Read(buf[:cap(buf)]) + if nRead == 0 { if err != nil { if err == io.EOF { break @@ -235,12 +253,44 @@ func (ix *IndexWriter) add(name string, f io.Reader) error { } return fmt.Errorf("%s: 0-length read", name) } - buf = buf[:n] + buf = buf[:nRead] i = 0 + hasNul, hasNonASCII := scanASCIIChunk(buf) + if hasNul { + if ix.LogSkip { + log.Printf("%s: contains NUL, ignoring\n", name) + } + return nil + } + if !hasNonASCII { + for _, c := range buf { + tv = ((tv << 8) & (1<<24 - 1)) | uint32(c) + if n++; n >= 3 { + ix.trigram.Add(tv) + } + if n > maxFileLen { + if ix.LogSkip { + log.Printf("%s: too long, ignoring\n", name) + } + return nil + } + if linelen++; linelen > maxLineLen { + if ix.LogSkip { + log.Printf("%s: very long lines, ignoring\n", name) + } + return nil + } + if c == '\n' { + linelen = 0 + } + } + i = len(buf) + continue + } } c = buf[i] i++ - tv |= uint32(c) + tv = ((tv << 8) & (1<<24 - 1)) | uint32(c) if n++; n >= 3 { ix.trigram.Add(tv) } @@ -427,6 +477,7 @@ func (ix *IndexWriter) flushPost() { // into posting lists, writing the resulting lists to out. func (ix *IndexWriter) mergePost(out *Buffer) { var h postHeap + defer h.close() if len(ix.postEnds) > 0 { log.Printf("merge mem + %d MB disk", ix.postEnds[len(ix.postEnds)-1]>>20) @@ -465,12 +516,23 @@ const postBuf = 4096 // A postHeap is a heap (priority queue) of postChunks. type postHeap struct { - ch []*postChunk + ch []*postChunk + mappings []mmapData +} + +func (h *postHeap) close() { + for i := range h.mappings { + h.mappings[i].close() + } + h.mappings = nil } func (h *postHeap) addFile(w *Buffer, ends []int) { w.Flush() - data := mmapFile(w.file).d + m := mmapFile(w.file) + m.f = nil // the Buffer owns the file; the heap owns only the mapping + h.mappings = append(h.mappings, m) + data := m.d start := 0 for _, end := range ends { var r allPostReader diff --git a/index/write_test.go b/index/write_test.go index bc325d9317..d1c2a17022 100644 --- a/index/write_test.go +++ b/index/write_test.go @@ -10,6 +10,7 @@ import ( "encoding/binary" "encoding/hex" "fmt" + "io" "os" "sort" "strings" @@ -410,3 +411,37 @@ func TestZip(t *testing.T) { checkPosting(t, ix, "now", 3, 4, 6) checkPosting(t, ix, "pot", 4, 5, 7) } + +type chunkReader struct { + r io.Reader + chunk int +} + +func (c *chunkReader) Read(p []byte) (int, error) { + if len(p) > c.chunk { + p = p[:c.chunk] + } + return c.r.Read(p) +} + +func TestChunkedAdd(t *testing.T) { + f, _ := os.CreateTemp("", "chunk-test") + defer os.Remove(f.Name()) + + ix := Create(f.Name()) + content := "abcdefghijklmnop" + r := &chunkReader{r: strings.NewReader(content), chunk: 4} + if err := ix.Add("file.txt", r); err != nil { + t.Fatal(err) + } + ix.Flush() + + opened := Open(f.Name()) + for i := 0; i <= len(content)-3; i++ { + trig := content[i : i+3] + list := opened.PostingList(tri(trig)) + if len(list) == 0 { + t.Errorf("missing trigram %q across chunks", trig) + } + } +} diff --git a/regexp/match.go b/regexp/match.go index 7a90d1a537..1dc1d23f04 100644 --- a/regexp/match.go +++ b/regexp/match.go @@ -368,21 +368,22 @@ type Grep struct { Matches int // how many matches were found? Limit int // stop after this many matches Limited bool // stopped because of limit + Errors bool // an input operation failed - PreContext int // number of lines to print after - PostContext int // number of lines to print before + PreContext int // number of lines to print before + PostContext int // number of lines to print after buf []byte } -func (g *Grep) AddFlags() { - flag.BoolVar(&g.L, "l", false, "list matching files only") - flag.BoolVar(&g.C, "c", false, "print match counts only") - flag.BoolVar(&g.N, "n", false, "show line numbers") - flag.BoolVar(&g.H, "h", false, "omit file names") - flag.IntVar(&g.PreContext, "B", 0, "show `n` lines before match") - flag.IntVar(&g.PostContext, "A", 0, "show `n` lines after match") - flag.Func("C", "show `n` lines before and after match", func(s string) error { +func (g *Grep) AddFlagsToSet(flags *flag.FlagSet) { + flags.BoolVar(&g.L, "l", false, "list matching files only") + flags.BoolVar(&g.C, "c", false, "print match counts only") + flags.BoolVar(&g.N, "n", false, "show line numbers") + flags.BoolVar(&g.H, "h", false, "omit file names") + flags.IntVar(&g.PreContext, "B", 0, "show `n` lines before match") + flags.IntVar(&g.PostContext, "A", 0, "show `n` lines after match") + flags.Func("C", "show `n` lines before and after match", func(s string) error { n, err := strconv.Atoi(s) if err != nil { return err @@ -393,8 +394,16 @@ func (g *Grep) AddFlags() { }) } +func (g *Grep) AddVFlagToSet(flags *flag.FlagSet) { + flags.BoolVar(&g.V, "v", false, "show non-matching lines") +} + +func (g *Grep) AddFlags() { + g.AddFlagsToSet(flag.CommandLine) +} + func (g *Grep) AddVFlag() { - flag.BoolVar(&g.V, "v", false, "show non-matching lines") + g.AddVFlagToSet(flag.CommandLine) } func (g *Grep) esc(s string) string { @@ -407,6 +416,7 @@ func (g *Grep) esc(s string) string { func (g *Grep) File(name string) { f, err := os.Open(name) if err != nil { + g.Errors = true fmt.Fprintf(g.Stderr, "%s\n", g.esc(err.Error())) return } @@ -416,26 +426,13 @@ func (g *Grep) File(name string) { var nl = []byte{'\n'} -func countNL(b []byte) int { - n := 0 - for { - i := bytes.IndexByte(b, '\n') - if i < 0 { - break - } - n++ - b = b[i+1:] - } - return n -} - func (g *Grep) Reader(r io.Reader, name string) { if g.buf == nil { g.buf = make([]byte, 1<<20) } var ( buf = g.buf[:0] - needLineno = g.N || g.HTML + needLineno = g.N || g.HTML || g.PreContext > 0 || g.PostContext > 0 lineno = 1 count = 0 prefix = "" @@ -462,7 +459,22 @@ func (g *Grep) Reader(r io.Reader, name string) { endText = true } for chunkStart < end { - m1 := g.Regexp.Match(buf[chunkStart:end], beginText, endText) + chunkStart + m1 := -1 + if g.V { + for start := chunkStart; start < end; { + stop := end + if n := bytes.IndexByte(buf[start:end], '\n'); n >= 0 { + stop = start + n + 1 + } + if g.Regexp.Match(buf[start:stop], beginText && start == chunkStart, endText && stop == end) < 0 { + m1 = stop - 1 + break + } + start = stop + } + } else { + m1 = g.Regexp.Match(buf[chunkStart:end], beginText, endText) + chunkStart + } beginText = false if m1 < chunkStart { break @@ -533,6 +545,7 @@ func (g *Grep) Reader(r io.Reader, name string) { chunkStart = d if endText && err != nil { if err != io.EOF && err != io.ErrUnexpectedEOF { + g.Errors = true fmt.Fprintf(g.Stderr, "%s: %v\n", g.esc(name), err) } break @@ -548,6 +561,9 @@ func (g *Grep) Reader(r io.Reader, name string) { } func lineSuffixLen(buf []byte, n int) int { + if n <= 0 { + return 0 + } end := len(buf) for i := 0; i < n; i++ { j := bytes.LastIndex(buf[:end], nl) diff --git a/regexp/nl_bench_test.go b/regexp/nl_bench_test.go new file mode 100644 index 0000000000..a7f88a3640 --- /dev/null +++ b/regexp/nl_bench_test.go @@ -0,0 +1,83 @@ +// Copyright 2026 The Go Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +package regexp + +import "testing" + +func BenchmarkCountNL(b *testing.B) { + sizes := []struct { + name string + size int + }{ + {"1KB", 1024}, + {"64KB", 64 * 1024}, + {"1MB", 1024 * 1024}, + } + for _, tc := range sizes { + data := make([]byte, tc.size) + for i := range data { + if i%40 == 0 { + data[i] = '\n' + } else { + data[i] = 'x' + } + } + + b.Run(tc.name, func(b *testing.B) { + b.SetBytes(int64(tc.size)) + b.ReportAllocs() + b.ResetTimer() + for range b.N { + _ = countNL(data) + } + }) + } +} + +func TestCountNLCorrectness(t *testing.T) { + cases := []string{ + "", + "\n", + "hello", + "hello\n", + "\nhello", + "\nhello\nworld\n", + "a\nb\nc\nd\ne\nf\ng\nh\ni\nj\nk\nl\nm\nn\no\np\nq\n", + } + for _, s := range cases { + b := []byte(s) + expected := 0 + for _, c := range b { + if c == '\n' { + expected++ + } + } + got := countNL(b) + if got != expected { + t.Fatalf("for %q: got %d, expected %d", s, got, expected) + } + } + + for _, size := range []int{15, 16, 17, 31, 32, 33, 240, 250*16 - 1, 250 * 16, 250*16 + 1, 10000} { + buf := make([]byte, size) + for i := range buf { + if i%7 == 0 { + buf[i] = '\n' + } else { + buf[i] = 'a' + } + } + expected := 0 + for _, c := range buf { + if c == '\n' { + expected++ + } + } + got := countNL(buf) + if got != expected { + t.Fatalf("size %d: got %d, expected %d", size, got, expected) + } + } +} diff --git a/regexp/nl_purego.go b/regexp/nl_purego.go new file mode 100644 index 0000000000..bd953ccfcb --- /dev/null +++ b/regexp/nl_purego.go @@ -0,0 +1,22 @@ +// Copyright 2026 The Go Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +//go:build !goexperiment.simd + +package regexp + +import "bytes" + +func countNL(b []byte) int { + n := 0 + for { + i := bytes.IndexByte(b, '\n') + if i < 0 { + break + } + n++ + b = b[i+1:] + } + return n +} diff --git a/regexp/nl_simd.go b/regexp/nl_simd.go new file mode 100644 index 0000000000..157bba750d --- /dev/null +++ b/regexp/nl_simd.go @@ -0,0 +1,39 @@ +// Copyright 2026 The Go Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +//go:build goexperiment.simd + +package regexp + +import "simd/archsimd" + +func countNL(b []byte) int { + nl := byte('\n') + aNl := archsimd.BroadcastUint8x16(nl) + one := archsimd.BroadcastUint8x16(1) + + total := 0 + var stored [16]uint8 + for len(b) >= 16 { + batchLen := min(len(b)/16, 250) + var acc archsimd.Uint8x16 + for range batchLen { + v := archsimd.LoadUint8x16(b[:16]) + m := v.Equal(aNl) + ones := m.ToInt8x16().ToBits().And(one) + acc = acc.Add(ones) + b = b[16:] + } + acc.Store(stored[:]) + for _, v := range stored { + total += int(v) + } + } + for _, c := range b { + if c == nl { + total++ + } + } + return total +} diff --git a/regexp/regexp_test.go b/regexp/regexp_test.go index 08bde8dd6c..6feb9f47c0 100644 --- a/regexp/regexp_test.go +++ b/regexp/regexp_test.go @@ -197,6 +197,9 @@ var grepTests = []struct { }{ {re: `a+`, s: "abc\ndef\nghalloo\n", out: "input:abc\ninput:ghalloo\n"}, {re: `x.*y`, s: "xay\nxa\ny\n", out: "input:xay\n"}, + {g: Grep{PreContext: 1, PostContext: 1}, re: `target`, s: "line1\ntarget\nline3\n", out: "input:2:\n\t\tline1\n\t>>\ttarget\n\t\tline3\n"}, + {g: Grep{PreContext: 1}, re: `target`, s: "line1\ntarget\nline3\n", out: "input:2:\n\t\tline1\n\t>>\ttarget\n"}, + {g: Grep{PostContext: 1}, re: `target`, s: "line1\ntarget\nline3\n", out: "input:2:\n\t>>\ttarget\n\t\tline3\n"}, } func TestGrep(t *testing.T) { diff --git a/skills/codesearch/SKILL.md b/skills/codesearch/SKILL.md new file mode 100644 index 0000000000..5418496443 --- /dev/null +++ b/skills/codesearch/SKILL.md @@ -0,0 +1,100 @@ +--- +name: codesearch-tools-workflow +description: Use codesearch (or cs) and legacy commands (cindex/csearch/cgrep/csweb) correctly in Go Code Search repos; choose the right subcommand, keep index fresh, and provide actionable command sequences. +--- + +# Code Search Tools Workflow Skill + +## Goal +Help users and coding agents choose and run the unified `codesearch` (or `cs`) CLI tool: +- `codesearch index` (legacy: `cindex`) for indexing +- `codesearch search` or `codesearch ` (legacy: `csearch`) for indexed fast search +- `codesearch grep` (legacy: `cgrep`) for direct grep-like file/stdin search +- `codesearch web` (legacy: `csweb`) for browser-based exploration + +## Tool selection policy + +### Use `codesearch index` (or `cs index` / `cindex`) when: +- user asks to “build/rebuild/update index” +- search is failing due to missing/stale index +- user needs indexed roots listed or reset + +Recommended flows: +1. First-time index: + - `codesearch index ` (or `cs index `) +2. Incremental reindex known roots: + - `codesearch index` (or `cs index`) +3. Reset then index specific roots: + - `codesearch index -reset ` (or `cs index -reset `) +4. Inspect roots: + - `codesearch index -list` (or `cs index -list`) + +### Use `codesearch search` / `codesearch ` (or `cs` / `csearch`) when: +- user wants fast search over large indexed corpus +- query is regex content search over indexed files +- user wants filename filter with content regex + +Recommended patterns: +- `codesearch search 'pattern'` or `codesearch 'pattern'` (or `cs 'pattern'`) +- `codesearch search -i 'pattern'` or `codesearch -i 'pattern'` (or `cs -i 'pattern'`) +- `codesearch search -f '.*\.go$' 'pattern'` or `codesearch -f '.*\.go$' 'pattern'` +- `codesearch search -n -l 'pattern'` or `codesearch -n -l 'pattern'` +- If results look wrong and index seems stale: suggest `codesearch index` then retry. + +### Use `codesearch grep` (or `cs grep` / `cgrep`) when: +- user wants grep-like search in given files or stdin +- indexing is not wanted/needed +- search scope is small or explicit + +Recommended patterns: +- `codesearch grep 'pattern' file1 file2` (or `cs grep 'pattern' file1 file2`) +- `cat file | codesearch grep 'pattern'` +- `codesearch grep -i -n 'pattern' ` + +### Use `codesearch web` (or `cs web` / `csweb`) when: +- user asks for web UI / browser search +- interactive source browsing is preferred + +Run: +- `codesearch web` (or `cs web` / `csweb`) +- Open: `http://localhost:2473` (or specified `-addr`) + +## Legacy Commands as Aliases +The legacy individual commands are retained as direct standalone wrappers/shortcuts: +- `cindex` $\equiv$ `codesearch index` +- `csearch` $\equiv$ `codesearch search` +- `cgrep` $\equiv$ `codesearch grep` +- `csweb` $\equiv$ `codesearch web` + +## Agent behavior checklist + +1. Confirm search intent: + - indexed corpus (`codesearch search` / `codesearch `) vs direct files/stdin (`codesearch grep`). +2. For indexed search, verify index readiness: + - if missing/outdated, run or propose `codesearch index`. +3. Scope carefully: + - use `-f` filename regex to reduce noise. +4. Prefer reproducible commands: + - provide exact command with quoted regex (`codesearch -n 'pattern'`). +5. Explain fallback: + - if trigram prefilter misses due to unusual cases, try `codesearch search -brute 'pattern'`. +6. Keep environment awareness: + - index file comes from `$CSEARCHINDEX`, else the `index` key in + `~/.config/codesearch/config.json`, else `/.csearchindex` + when that file sets `"git": true`, else `$HOME/.csearchindex`. + - flag defaults (`n`, `i`, `A`/`B`/`C`, ...) may also come from that config + file, so output can carry line numbers without `-n` on the command line. + +## Quick decision tree + +- “Search everywhere fast” → `codesearch search ` or `codesearch ` (or `cs `) +- “I changed files; refresh searchable DB” → `codesearch index` (or `cs index`) +- “Search these exact files/stdin like grep” → `codesearch grep ` (or `cs grep ...`) +- “I want a browser UI” → `codesearch web` (or `cs web`) + +## Safety/accuracy notes + +- Go flag parsing does **not** support combined short flags (`-in`); use `-i -n`. +- Regex engine is RE2 (not full PCRE backtracking behavior). +- `codesearch search` depends on index freshness for correctness/performance. +- `-zip` indexing exists but is experimental.