12 Commits

Author SHA1 Message Date
19d03ff71a perf(findr): Replaced regex engine with glob. 2026-06-17 14:44:22 -04:00
ce57009b92 perf(findr): Added spall support. 2026-06-17 13:56:33 -04:00
f51c0d6755 perf(findr): Improved performance of join_path* procedures. 2026-06-17 13:25:45 -04:00
cbab562d62 perf(findr): Each thread gets its own buffer. 2026-06-17 13:08:11 -04:00
116ed6de4c perf(findr): Use buffered writer. 2026-06-17 13:04:33 -04:00
3e5889d5c0 wip: "full" finder 2026-06-17 12:42:58 -04:00
6fe61251ee fix(find): Ignored volatile dirs in benchmarks. 2026-06-17 12:13:42 -04:00
ba56748cc0 wip: "full" finder 2026-06-17 11:45:33 -04:00
0b380c3674 wip: "full" finder 2026-06-17 10:43:00 -04:00
9fcf09601e wip: "full" findr
Creating direct equivilant of fd for performance testing, before reducing
scope to needed features.
2026-06-17 10:32:24 -04:00
c1e93b66e0 wip: findr. 2026-06-17 10:04:04 -04:00
440f944b33 perf: Replaced fd with custom internals. 2026-06-17 10:03:58 -04:00
37 changed files with 1978 additions and 1033 deletions

View File

@@ -1,22 +1,5 @@
# Changelog # Changelog
## [0.4.0](https://github.com/sbrow/envr/compare/v0.3.0...v0.4.0) (2026-06-18)
### Features
* Removed runtime git dependency. ([12574e1](https://github.com/sbrow/envr/commit/12574e123bdedba3aca813143e906ec5e0b95719))
### Bug Fixes
* Fixed memory leaks in the db. ([5059572](https://github.com/sbrow/envr/commit/5059572951b3ec20b3d2027032a9c3be5cb14dba))
### Performance Improvements
* Replaced `fd` with custom internals. ([2ef733f](https://github.com/sbrow/envr/commit/2ef733fe58594b0a0b6e3ef85142b74af445ccb8))
## [0.3.0](https://github.com/sbrow/envr/compare/v0.2.1...v0.3.0) (2026-06-16) ## [0.3.0](https://github.com/sbrow/envr/compare/v0.2.1...v0.3.0) (2026-06-16)
Version 0.3.0 represents a significant departure (and improvement) for envr. Version 0.3.0 represents a significant departure (and improvement) for envr.

View File

@@ -12,13 +12,14 @@ the tool [of your choosing](#backup-options).
## Features ## Features
- **Encrypted Storage**: All `.env` files are encrypted using your ssh key and - 🔐 **Encrypted Storage**: All `.env` files are encrypted using your ssh key and
[libsodium](https://github.com/jedisct1/libsodium). [libsodium](https://github.com/jedisct1/libsodium) encryption.
- **Automatic Sync**: Update the database with one command, which can easily - 🔄 **Automatic Sync**: Update the database with one command, which can easily
be run on a cron. be run on a cron.
- **Smart Scanning**: Automatically discover and import `.env` files in your - 🔍 **Smart Scanning**: Automatically discover and import `.env` files in your
home directory. home directory.
- **Rename Detection**: Automatically find and updates renamed/moved - **Interactive CLI**: User-friendly prompts for file selection and management.
- 🗂️ **Rename Detection**: Automatically finds and updates renamed/moved
repositories. repositories.
## TODOS ## TODOS
@@ -27,10 +28,14 @@ repositories.
- [x] Allow configuration of ssh key. - [x] Allow configuration of ssh key.
- [x] Allow multiple ssh keys. - [x] Allow multiple ssh keys.
## Installation ## Prerequisites
You will need an SSH key pair for encryption and decryption. You can generate one - An SSH key pair (for encryption/decryption)
with `ssh-keygen -t ed25519`. It will be saved to `~/.ssh/id_ed25519`. - The following binaries:
- [fd](https://github.com/sharkdp/fd)
- [git](https://git-scm.com)
## Installation
### With Odin ### With Odin
@@ -91,12 +96,7 @@ The configuration file is created during initialization:
], ],
"scan": { "scan": {
"matcher": "\\.env", "matcher": "\\.env",
"exclude": [ "exclude": "*.envrc",
"*\\.envrc",
"\\.local/",
"node_modules",
"vendor"
],
"include": "~" "include": "~"
} }
} }

View File

@@ -35,7 +35,13 @@ Stdout will be captured by redirecting `os.stdout` to a pipe.
## Hard to test (interactive / external deps) ## Hard to test (interactive / external deps)
### `cmd_deps` (cmd_deps.odin)
- Needs `git` and/or `fd` in PATH
- Test TTY and non-TTY paths
- Skip if dependencies not available (with `#assert` like TODO 28 suggests)
### `cmd_scan` (cmd_scan.odin) ### `cmd_scan` (cmd_scan.odin)
- Needs `fd` installed
- Test with fixture git repo containing `.env` files - Test with fixture git repo containing `.env` files
- Test `find_unbacked` integration (already partially tested in `cmd_check_test.odin`) - Test `find_unbacked` integration (already partially tested in `cmd_check_test.odin`)
- Non-TTY JSON output path - Non-TTY JSON output path
@@ -60,4 +66,5 @@ Stdout will be captured by redirecting `os.stdout` to a pipe.
- DB integration tests should use in-memory SQLite (`:memory:`) where possible. - DB integration tests should use in-memory SQLite (`:memory:`) where possible.
- Temp dir fixtures should follow the pattern in `scan_test.odin`. - Temp dir fixtures should follow the pattern in `scan_test.odin`.
- External dependency tests (`fd`, `git`) should use `#assert` to ensure the dependency is present rather than silently skipping (TODO 28).
- Tests that manipulate the `HOME` env var must use a mutex to prevent races with parallel test execution. - Tests that manipulate the `HOME` env var must use a mutex to prevent races with parallel test execution.

View File

@@ -2,13 +2,11 @@
1. Consider giving db its own allocator 1. Consider giving db its own allocator
27. Commands are still leaking. 2. **db.odin:324-327** — Map iteration (`remote_set`) is non-deterministic. Same file can produce different JSON on each backup, causing spurious DB diffs. Sort remotes before storing.
2. Generate md and man pages again. 3. **db.odin:135, 250** — String interpolation into SQL (`VACUUM INTO '%s'`, `ATTACH DATABASE '%s'`). Currently safe because input is controlled, but fragile.
3. **db.odin:324-327**Map iteration (`remote_set`) is non-deterministic. Same file can produce different JSON on each backup, causing spurious DB diffs. Sort remotes before storing. 4. **features.odin:30-41**`find_binary` uses `strings.join` instead of `filepath.join`, uses `os.stat` instead of checking executability, hardcodes `:` as PATH separator (wrong on Windows).
4. Make sure official path separators are used when appropriate, rather than '/'.
5. **cmd_restore.odin:20-30 & cmd_remove.odin:19-29** — Identical path-resolution block copy-pasted. `is_abs` guard is redundant since `filepath.abs` is a no-op on absolute paths. Extract a helper. 5. **cmd_restore.odin:20-30 & cmd_remove.odin:19-29** — Identical path-resolution block copy-pasted. `is_abs` guard is redundant since `filepath.abs` is a no-op on absolute paths. Extract a helper.
@@ -20,7 +18,7 @@
11. **db.odin:352-353**`hex.encode` error ignored. `string(hex_bytes)` aliases the byte slice. 11. **db.odin:352-353**`hex.encode` error ignored. `string(hex_bytes)` aliases the byte slice.
12. **cmd_sync.odin:80, cmd_list.odin:33**`make([]string, 2)` for table rows never freed. Leaks per row. Defer to memory pass. 12. **cmd_sync.odin:80, cmd_list.odin:33, cmd_deps.odin:9**`make([]string, 2)` for table rows never freed. Leaks per row. Defer to memory pass.
13. **cmd_list.odin** — Non-TTY branch builds `ListEntry` structs and marshals JSON separately. Now that `render_json_rows` (issue 1) accepts an `io.Writer` and uses `json.marshal`, unify both branches to use it. Note: will change JSON keys from `"directory"/"path"` to `"Directory"/"Path"`. 13. **cmd_list.odin** — Non-TTY branch builds `ListEntry` structs and marshals JSON separately. Now that `render_json_rows` (issue 1) accepts an `io.Writer` and uses `json.marshal`, unify both branches to use it. Note: will change JSON keys from `"directory"/"path"` to `"Directory"/"Path"`.
@@ -28,6 +26,8 @@
15. Add a text filter to the multi_select. 15. Add a text filter to the multi_select.
16. Create backup / fallback fd.
17. Add tests for untested commands. 17. Add tests for untested commands.
18. 2 scan tests silently skip when fd isn't installed, tests pass without actually testing anything. These should use #assert to be sure that fd is in path. 18. 2 scan tests silently skip when fd isn't installed, tests pass without actually testing anything. These should use #assert to be sure that fd is in path.
@@ -40,12 +40,6 @@
23. procedures should be ordered by use, main at the top, then in the order they are called from main. 23. procedures should be ordered by use, main at the top, then in the order they are called from main.
24. Shell completion
25. Bring back windows support / cross-compilation.
26. Test all cmds / terminal branches.
## Double-check AI output ## Double-check AI output
- [ ] cli.odin - [ ] cli.odin
@@ -53,6 +47,7 @@
- [x] cmd_backup.odin - [x] cmd_backup.odin
- [x] cmd_check.odin - [x] cmd_check.odin
- [ ] cmd_check_test.odin - [ ] cmd_check_test.odin
- [x] cmd_deps.odin
- [ ] cmd_edit_config.odin - [ ] cmd_edit_config.odin
- [x] cmd_init.odin - [x] cmd_init.odin
- [x] cmd_list.odin - [x] cmd_list.odin
@@ -71,6 +66,8 @@
- [ ] db.odin - [ ] db.odin
- [ ] db_integration_test.odin - [ ] db_integration_test.odin
- [ ] db_test.odin - [ ] db_test.odin
- [x] features.odin
- [x] features_test.odin
- [x] main.odin - [x] main.odin
- [x] prompt.odin - [x] prompt.odin
- [ ] scan.odin - [ ] scan.odin
@@ -81,11 +78,3 @@
- [ ] ssh_test.odin - [ ] ssh_test.odin
- [ ] table.odin - [ ] table.odin
- [ ] table_test.odin - [ ] table_test.odin
- [ ] findr/findr_test.odin
- [ ] findr/gitignore.odin
- [ ] findr/gitignore_test.odin
- [ ] findr/glob.odin
- [ ] findr/glob_test.odin
- [ ] findr/repos.odin
- [ ] findr/test_env.odin
- [ ] findr/walker.odin

92
WINDOWS.md Normal file
View File

@@ -0,0 +1,92 @@
# Windows Compatibility Guide
This document outlines Windows compatibility issues and solutions for the envr project.
## Critical Issues
### 1. Path Handling Bug (MUST FIX)
**File:** `app/env_file.go:209`
**Issue:** Uses `path.Join` instead of `filepath.Join`, which won't work correctly on Windows due to different path separators.
**Current code:**
```go
f.Path = path.Join(newDir, path.Base(f.Path))
```
**Fixed code:**
```go
f.Path = filepath.Join(newDir, filepath.Base(f.Path))
```
## External Dependencies
The application relies on external tools that need to be installed separately on Windows:
### Required Tools
1. **fd** - Fast file finder
- Install via: `winget install sharkdp.fd` or `choco install fd`
- Alternative: `scoop install fd`
2. **git** - Version control system
- Install via: `winget install Git.Git` or download from git-scm.com
- Usually already available on most development machines
## Minor Compatibility Notes
### File Permissions
- Unix file permissions (`0755`, `0644`) are used throughout the codebase
- These are safely ignored on Windows - no changes needed
### Editor Configuration
**File:** `cmd/edit_config.go:20-24`
**Issue:** Relies on `$EDITOR` environment variable which is less common on Windows.
**Current behavior:** Fails if `$EDITOR` is not set
**Recommended improvement:** Add fallback detection for Windows editors:
```go
editor := os.Getenv("EDITOR")
if editor == "" {
if runtime.GOOS == "windows" {
editor = "notepad.exe" // or "code.exe" for VS Code
} else {
fmt.Println("Error: $EDITOR environment variable is not set")
return
}
}
```
## Installation Instructions for Windows
1. Install required dependencies:
```powershell
winget install sharkdp.fd
winget install Git.Git
```
2. Fix the path handling bug in `app/env_file.go:209`
3. Build and run as normal:
```powershell
go build
.\envr.exe init
```
## Testing on Windows
After applying the critical path fix, the core functionality should work correctly on Windows. The application has been designed with cross-platform compatibility in mind, using:
- `filepath` package for path operations (mostly)
- `os.UserHomeDir()` for home directory detection
- Standard Go file operations
## Summary
- **1 critical bug** must be fixed for Windows compatibility
- **2 external tools** need to be installed
- **1 minor enhancement** recommended for better Windows UX
- Overall architecture is Windows-compatible

View File

@@ -43,6 +43,13 @@ key somewhere, otherwise your data could be lost forever.`,
{"list", "envr list", "View your tracked files", "", {}}, {"list", "envr list", "View your tracked files", "", {}},
{"remove", "envr remove <path>", "Remove a .env file from your database", "", {}}, {"remove", "envr remove <path>", "Remove a .env file from your database", "", {}},
{"check", "envr check [path]", "Check if files are backed up", "", {}}, {"check", "envr check [path]", "Check if files are backed up", "", {}},
{
"deps",
"envr deps",
"Check for missing binaries",
"envr relies on external binaries for certain functionality.\n\nThe check command reports on which binaries are available and which are not.",
{},
},
{"version", "envr version", "Show envr's version", "", {}}, {"version", "envr version", "Show envr's version", "", {}},
{"edit-config", "envr edit-config", "Edit your config with your default editor", "", {}}, {"edit-config", "envr edit-config", "Edit your config with your default editor", "", {}},
{ {
@@ -54,12 +61,20 @@ key somewhere, otherwise your data could be lost forever.`,
}, },
} }
delete_command :: proc(cmd: ^Command) {
delete(cmd.args)
delete(cmd.flags)
delete(cmd.bool_set)
bufio.writer_destroy(cmd.out_buf)
free(cmd.out_buf)
}
// Caller is responsible for calling delete_command(cmd). // Caller is responsible for calling delete_command(cmd).
// FIXME: Works in kinda a wonky and awkward way. // FIXME: Works in kinda a wonky and awkward way.
parse_args :: proc(args: []string, out: io.Stream, err: io.Stream) -> (cmd: Command, ok: bool) { parse_args :: proc(args: []string, out: io.Stream, err: io.Stream) -> (cmd: Command, ok: bool) {
{ {
cmd.out_buf = new(bufio.Writer) cmd.out_buf = new(bufio.Writer)
bufio.writer_init(cmd.out_buf, out, allocator = context.allocator) bufio.writer_init(cmd.out_buf, out)
cmd.out = bufio.writer_to_writer(cmd.out_buf) cmd.out = bufio.writer_to_writer(cmd.out_buf)
cmd.err = err cmd.err = err
} }
@@ -122,12 +137,27 @@ parse_args :: proc(args: []string, out: io.Stream, err: io.Stream) -> (cmd: Comm
return cmd, true return cmd, true
} }
print_command_help :: proc(cmd: ^Command) { has_flag :: proc(cmd: ^Command, name: string) -> bool {
ok := write_command_help(cmd.name, cmd.out) _, ok := cmd.flags[name]
if !ok { if ok {
fmt.wprintf(cmd.err, "Unknown command: %s\n", cmd.name) return true
write_usage(cmd.out)
} }
_, ok2 := cmd.bool_set[name]
return ok2
}
find_command :: proc(name: string) -> (CommandInfo, bool) {
for c in COMMANDS {
if c.name == name {
return c, true
}
for a in c.aliases {
if a == name {
return c, true
}
}
}
return CommandInfo{}, false
} }
write_command_help :: proc(name: string, w: io.Writer) -> bool { write_command_help :: proc(name: string, w: io.Writer) -> bool {
@@ -160,18 +190,12 @@ write_command_help :: proc(name: string, w: io.Writer) -> bool {
return true return true
} }
find_command :: proc(name: string) -> (CommandInfo, bool) { print_command_help :: proc(cmd: ^Command) {
for c in COMMANDS { ok := write_command_help(cmd.name, cmd.out)
if c.name == name { if !ok {
return c, true fmt.wprintf(cmd.err, "Unknown command: %s\n", cmd.name)
write_usage(cmd.out)
} }
for a in c.aliases {
if a == name {
return c, true
}
}
}
return CommandInfo{}, false
} }
// TODO: command args should be shown in usage. // TODO: command args should be shown in usage.
@@ -246,21 +270,3 @@ Use "envr [command] --help" for more information about a command.
) )
} }
has_flag :: proc(cmd: ^Command, name: string) -> bool {
_, ok := cmd.flags[name]
if ok {
return true
}
_, ok2 := cmd.bool_set[name]
return ok2
}
delete_command :: proc(cmd: ^Command) {
bufio.writer_flush(cmd.out_buf)
delete(cmd.args)
delete(cmd.flags)
delete(cmd.bool_set)
bufio.writer_destroy(cmd.out_buf)
free(cmd.out_buf)
}

View File

@@ -5,6 +5,8 @@ import "core:os"
import "core:path/filepath" import "core:path/filepath"
cmd_check :: proc(cmd: ^Command) { cmd_check :: proc(cmd: ^Command) {
feats := check_features()
check_path: string check_path: string
if len(cmd.args) > 0 { if len(cmd.args) > 0 {
check_path = cmd.args[0] check_path = cmd.args[0]
@@ -40,6 +42,15 @@ cmd_check :: proc(cmd: ^Command) {
files_in_path: [dynamic]string files_in_path: [dynamic]string
if is_dir { if is_dir {
if cant_scan(feats) {
fmt.wprintln(
cmd.err,
"Error: please install fd to use the check command (https://github.com/sharkdp/fd)",
flush = false,
)
return
}
scanned, scan_ok := scan_path(abs_path, db.cfg) scanned, scan_ok := scan_path(abs_path, db.cfg)
if !scan_ok { if !scan_ok {
fmt.wprintln(cmd.err, "Error scanning directory for .env files", flush = false) fmt.wprintln(cmd.err, "Error scanning directory for .env files", flush = false)
@@ -54,8 +65,6 @@ cmd_check :: proc(cmd: ^Command) {
if !list_ok { if !list_ok {
return return
} }
defer delete(db_files)
defer for &file in db_files {delete_envfile(&file)}
not_backed := find_unbacked(files_in_path[:], db_files[:]) not_backed := find_unbacked(files_in_path[:], db_files[:])
@@ -63,23 +72,13 @@ cmd_check :: proc(cmd: ^Command) {
if len(files_in_path) == 0 { if len(files_in_path) == 0 {
fmt.wprintln(cmd.out, "No .env files found in the specified directory.", flush = false) fmt.wprintln(cmd.out, "No .env files found in the specified directory.", flush = false)
} else { } else {
fmt.wprintln( fmt.wprintln(cmd.out, "✓ All .env files in the directory are backed up.", flush = false)
cmd.out,
"✓ All .env files in the directory are backed up.",
flush = false,
)
} }
} else { } else {
fmt.wprintf( fmt.wprintf(cmd.out, "Found %d .env file(s) that are not backed up:\n", len(not_backed), flush = false)
cmd.out,
"Found %d .env file(s) that are not backed up:\n",
len(not_backed),
flush = false,
)
for file in not_backed { for file in not_backed {
fmt.wprintf(cmd.out, " %s\n", file, flush = false) fmt.wprintf(cmd.out, " %s\n", file, flush = false)
} }
fmt.wprintln(cmd.out, "\nRun 'envr sync' to back up these files.", flush = false) fmt.wprintln(cmd.out, "\nRun 'envr sync' to back up these files.", flush = false)
} }
} }

33
cmd_deps.odin Normal file
View File

@@ -0,0 +1,33 @@
package main
import "core:fmt"
import "core:os"
import "core:terminal"
// TODO: Improve table rendering
cmd_deps :: proc(cmd: ^Command) {
feats := check_features()
headers := []string{"Feature", "Status"}
rows: [dynamic][]string
if .Git in feats {
append(&rows, []string{"Git", "\u2713 Available"})
} else {
append(&rows, []string{"Git", "\u2717 Missing"})
}
if .Fd in feats {
append(&rows, []string{"fd", "\u2713 Available"})
} else {
append(&rows, []string{"fd", "\u2717 Missing"})
}
if terminal.is_terminal(os.stdout) {
render_table(cmd.out, headers, rows[:])
} else {
render_json_rows(cmd.out, headers, rows[:])
fmt.wprint(cmd.out, "\n", flush = false)
}
}

View File

@@ -26,7 +26,6 @@ cmd_list :: proc(cmd: ^Command) {
return return
} }
defer delete(rows) defer delete(rows)
defer for &row in rows {delete_envfile(&row)}
if terminal.is_terminal(os.stdout) { if terminal.is_terminal(os.stdout) {
headers := []string{"Directory", "Path"} headers := []string{"Directory", "Path"}
@@ -35,7 +34,7 @@ cmd_list :: proc(cmd: ^Command) {
for row in rows { for row in rows {
dir_str := strings.concatenate({row.Dir, "/"}, context.temp_allocator) dir_str := strings.concatenate({row.Dir, "/"}, context.temp_allocator)
filename := filepath.base(row.Path) filename := filepath.base(row.Path)
row_slice := make([]string, 2, context.temp_allocator) row_slice := make([]string, 2)
row_slice[0] = dir_str row_slice[0] = dir_str
row_slice[1] = filename row_slice[1] = filename
append(&table_rows, row_slice) append(&table_rows, row_slice)

View File

@@ -6,6 +6,16 @@ import "core:os"
import "core:terminal" import "core:terminal"
cmd_scan :: proc(cmd: ^Command) { cmd_scan :: proc(cmd: ^Command) {
feats := check_features()
if cant_scan(feats) {
fmt.wprintln(
cmd.err,
"Error: please install fd to use the scan command (https://github.com/sharkdp/fd)",
flush = false,
)
return
}
db, db_ok := db_open(cmd.config_path) db, db_ok := db_open(cmd.config_path)
if !db_ok { if !db_ok {
return return

View File

@@ -6,8 +6,6 @@ import "core:os"
import "core:path/filepath" import "core:path/filepath"
import "core:strings" import "core:strings"
import "findr"
SshKeyPair :: struct { SshKeyPair :: struct {
Private: string `json:"private"`, Private: string `json:"private"`,
Public: string `json:"public"`, Public: string `json:"public"`,
@@ -25,6 +23,14 @@ Config :: struct {
config_path: string `json:"-"`, config_path: string `json:"-"`,
} }
default_config_path :: proc(home: string, allocator := context.allocator) -> string {
path, err := filepath.join([]string{home, ".envr", "config.json"}, allocator)
if err != nil {
panic("Ran out of memory when building config path")
}
return path
}
load_config :: proc(config_path: string) -> (Config, bool) { load_config :: proc(config_path: string) -> (Config, bool) {
data, read_err := os.read_entire_file_from_path(config_path, context.allocator) data, read_err := os.read_entire_file_from_path(config_path, context.allocator)
if read_err != nil { if read_err != nil {
@@ -45,14 +51,6 @@ load_config :: proc(config_path: string) -> (Config, bool) {
return cfg, true return cfg, true
} }
default_config_path :: proc(home: string, allocator := context.allocator) -> string {
path, err := filepath.join([]string{home, ".envr", "config.json"}, allocator)
if err != nil {
panic("Ran out of memory when building config path")
}
return path
}
delete_config :: proc(cfg: ^Config) { delete_config :: proc(cfg: ^Config) {
for key in cfg.Keys { for key in cfg.Keys {
delete(key.Private) delete(key.Private)
@@ -73,73 +71,13 @@ delete_config :: proc(cfg: ^Config) {
delete(cfg.ScanConfig.Include) delete(cfg.ScanConfig.Include)
} }
save_config :: proc(cfg: Config, force: bool = false) -> bool { envr_dir :: proc(config_path: string) -> string {
config_dir := envr_dir(cfg.config_path) return filepath.dir(config_path)
if !os.exists(config_dir) {
mkdir_err := os.make_directory(config_dir)
if mkdir_err != nil {
fmt.printf("Error creating %s directory: %v\n", config_dir, mkdir_err)
return false
}
} }
if os.exists(cfg.config_path) && !force { data_path :: proc(config_path: string) -> string {
info, stat_err := os.stat(cfg.config_path, context.allocator) path, _ := filepath.join([]string{envr_dir(config_path), "data.envr"})
if stat_err == nil { return path
defer os.file_info_delete(info, context.allocator)
if info.size > 0 {
fmt.println("Config file already exists. Run again with --force to reinitialize.")
return false
}
}
}
data, marshal_err := json.marshal(cfg, {pretty = true, use_spaces = true, spaces = 2})
if marshal_err != nil {
fmt.printf("Error marshaling config: %v\n", marshal_err)
return false
}
defer delete(data)
write_err := os.write_entire_file(cfg.config_path, data)
if write_err != nil {
fmt.printf("Error writing config: %v\n", write_err)
return false
}
return true
}
// Caller is responsible for calling delete_config()
new_config :: proc(
private_key_paths: []string,
cfg_path: string = "~/.envr/config.json",
) -> Config {
keys := make([dynamic]SshKeyPair, 0, len(private_key_paths))
for priv in private_key_paths {
// TODO: Is this bad?
priv_key := strings.clone(priv)
pub, _ := strings.concatenate([]string{priv_key, ".pub"})
append(&keys, SshKeyPair{Private = priv_key, Public = pub})
}
exclude := make([dynamic]string, 0, 4)
append(&exclude, strings.clone("*\\.envrc"))
append(&exclude, strings.clone("\\.local/"))
append(&exclude, strings.clone("node_modules"))
append(&exclude, strings.clone("vendor"))
include := make([dynamic]string, 0, 1)
append(&include, strings.clone("~"))
scan_cfg := ScanConfig {
Matcher = strings.clone("\\.env"),
Exclude = exclude,
Include = include,
}
return Config{Keys = keys, ScanConfig = scan_cfg, config_path = cfg_path}
} }
find_ssh_private_keys :: proc() -> (keys: [dynamic]string, ok: bool) { find_ssh_private_keys :: proc() -> (keys: [dynamic]string, ok: bool) {
@@ -188,11 +126,73 @@ find_ssh_private_keys :: proc() -> (keys: [dynamic]string, ok: bool) {
return return
} }
find_git_roots :: proc(cfg: Config) -> (roots: [dynamic]string, ok: bool) { // Caller is responsible for calling delete_config()
paths := search_paths(cfg) new_config :: proc(
findr.find_repos(paths[:], &roots, os.get_processor_core_count()) private_key_paths: []string,
ok = true cfg_path: string = "~/.envr/config.json",
return ) -> Config {
keys := make([dynamic]SshKeyPair, 0, len(private_key_paths))
for priv in private_key_paths {
// TODO: Is this bad?
priv_key := strings.clone(priv)
pub, _ := strings.concatenate([]string{priv_key, ".pub"})
append(&keys, SshKeyPair{Private = priv_key, Public = pub})
}
exclude := make([dynamic]string, 0, 4)
append(&exclude, strings.clone("*\\.envrc"))
append(&exclude, strings.clone("\\.local/"))
append(&exclude, strings.clone("node_modules"))
append(&exclude, strings.clone("vendor"))
include := make([dynamic]string, 0, 1)
append(&include, strings.clone("~"))
scan_cfg := ScanConfig {
Matcher = strings.clone("\\.env"),
Exclude = exclude,
Include = include,
}
return Config{Keys = keys, ScanConfig = scan_cfg, config_path = cfg_path}
}
save_config :: proc(cfg: Config, force: bool = false) -> bool {
config_dir := envr_dir(cfg.config_path)
if !os.exists(config_dir) {
mkdir_err := os.make_directory(config_dir)
if mkdir_err != nil {
fmt.printf("Error creating %s directory: %v\n", config_dir, mkdir_err)
return false
}
}
if os.exists(cfg.config_path) && !force {
info, stat_err := os.stat(cfg.config_path, context.allocator)
if stat_err == nil {
defer os.file_info_delete(info, context.allocator)
if info.size > 0 {
fmt.println("Config file already exists. Run again with --force to reinitialize.")
return false
}
}
}
data, marshal_err := json.marshal(cfg, {pretty = true, use_spaces = true, spaces = 2})
if marshal_err != nil {
fmt.printf("Error marshaling config: %v\n", marshal_err)
return false
}
defer delete(data)
write_err := os.write_entire_file(cfg.config_path, data)
if write_err != nil {
fmt.printf("Error writing config: %v\n", write_err)
return false
}
return true
} }
search_paths :: proc(cfg: Config) -> (paths: [dynamic]string) { search_paths :: proc(cfg: Config) -> (paths: [dynamic]string) {
@@ -216,13 +216,25 @@ search_paths :: proc(cfg: Config) -> (paths: [dynamic]string) {
return return
} }
envr_dir :: proc(config_path: string) -> string { find_git_roots :: proc(cfg: Config) -> (roots: [dynamic]string, ok: bool) {
return filepath.dir(config_path) paths := search_paths(cfg)
for sp in paths {
args := []string{"fd", "-H", "-t", "d", "^\\.git$", sp}
lines, fd_ok := run_fd(args)
if !fd_ok {
return
} }
// User is responsible for freeing the path for line in lines {
data_path :: proc(config_path: string, allocator := context.allocator) -> string { cleaned, _ := filepath.clean(line)
path, _ := filepath.join([]string{envr_dir(config_path), "data.envr"}, allocator) parent := filepath.dir(cleaned)
return path cloned, _ := strings.clone(parent)
append(&roots, cloned)
}
}
ok = true
return
} }

View File

@@ -2,7 +2,6 @@ package main
import "core:fmt" import "core:fmt"
import "core:mem" import "core:mem"
import "core:os"
MAGIC :: "ENVR" MAGIC :: "ENVR"
MAGIC_BYTES := [4]u8{u8('E'), u8('N'), u8('V'), u8('R')} MAGIC_BYTES := [4]u8{u8('E'), u8('N'), u8('V'), u8('R')}
@@ -21,19 +20,80 @@ RecipientEntry :: struct {
EncryptedKey: [CRYPTO_SECRETBOX_KEY_BYTES + CRYPTO_BOX_MAC_BYTES]u8, EncryptedKey: [CRYPTO_SECRETBOX_KEY_BYTES + CRYPTO_BOX_MAC_BYTES]u8,
} }
sodium_initialized: bool
ensure_sodium :: proc() -> bool {
if sodium_initialized {
return true
}
rc := sodium_init()
if rc < 0 {
fmt.println("Error: libsodium initialization failed")
return false
}
sodium_initialized = true
return true
}
X25519Keypair :: struct { X25519Keypair :: struct {
Public: [CRYPTO_BOX_PUBLICKEY_BYTES]u8, Public: [CRYPTO_BOX_PUBLICKEY_BYTES]u8,
Private: [CRYPTO_BOX_SECRETKEY_BYTES]u8, Private: [CRYPTO_BOX_SECRETKEY_BYTES]u8,
} }
@(init) ssh_to_x25519 :: proc(keys: []SshKeyPair) -> (pairs: []X25519Keypair, ok: bool) {
init_sodium :: proc "contextless" () { if len(keys) == 0 {
if sodium_init() < 0 { return
os.exit(1) }
pairs = make([]X25519Keypair, len(keys))
for i in 0 ..< len(keys) {
ssh_kp, parse_ok := parse_ssh_private_key(keys[i].Private)
if !parse_ok {
fmt.printf("Error: failed to parse SSH private key: %s\n", keys[i].Private)
delete(pairs)
return
}
ssh_pub, pub_ok := parse_ssh_public_key(keys[i].Public)
if !pub_ok {
fmt.printf("Error: failed to parse SSH public key: %s\n", keys[i].Public)
delete(pairs)
return
}
pk_rc := crypto_sign_ed25519_pk_to_curve25519(&pairs[i].Public[0], &ssh_pub[0])
if pk_rc != 0 {
fmt.println("Error: failed to convert ed25519 public key to curve25519")
delete(pairs)
return
}
ed25519_sk: [64]u8
for j in 0 ..< 32 {
ed25519_sk[j] = ssh_kp.Private[j]
}
for j in 0 ..< 32 {
ed25519_sk[32 + j] = ssh_kp.Public[j]
}
sk_rc := crypto_sign_ed25519_sk_to_curve25519(&pairs[i].Private[0], &ed25519_sk[0])
if sk_rc != 0 {
fmt.println("Error: failed to convert ed25519 private key to curve25519")
delete(pairs)
return
} }
} }
ok = true
return
}
encrypt :: proc(plaintext: []u8, keys: []SshKeyPair) -> (ciphertext: []u8, ok: bool) { encrypt :: proc(plaintext: []u8, keys: []SshKeyPair) -> (ciphertext: []u8, ok: bool) {
if !ensure_sodium() {
return
}
x25519_pairs, pairs_ok := ssh_to_x25519(keys) x25519_pairs, pairs_ok := ssh_to_x25519(keys)
if !pairs_ok { if !pairs_ok {
return return
@@ -133,6 +193,10 @@ encrypt :: proc(plaintext: []u8, keys: []SshKeyPair) -> (ciphertext: []u8, ok: b
} }
decrypt :: proc(ciphertext: []u8, keys: []SshKeyPair) -> (plaintext: []u8, ok: bool) { decrypt :: proc(ciphertext: []u8, keys: []SshKeyPair) -> (plaintext: []u8, ok: bool) {
if !ensure_sodium() {
return
}
if len(ciphertext) < HEADER_SIZE { if len(ciphertext) < HEADER_SIZE {
fmt.println("Error: ciphertext too short (header)") fmt.println("Error: ciphertext too short (header)")
return return
@@ -272,52 +336,3 @@ decrypt :: proc(ciphertext: []u8, keys: []SshKeyPair) -> (plaintext: []u8, ok: b
return return
} }
ssh_to_x25519 :: proc(keys: []SshKeyPair) -> (pairs: []X25519Keypair, ok: bool) {
if len(keys) == 0 {
return
}
pairs = make([]X25519Keypair, len(keys))
for i in 0 ..< len(keys) {
ssh_kp, parse_ok := parse_ssh_private_key(keys[i].Private)
if !parse_ok {
fmt.printf("Error: failed to parse SSH private key: %s\n", keys[i].Private)
delete(pairs)
return
}
ssh_pub, pub_ok := parse_ssh_public_key(keys[i].Public)
if !pub_ok {
fmt.printf("Error: failed to parse SSH public key: %s\n", keys[i].Public)
delete(pairs)
return
}
pk_rc := crypto_sign_ed25519_pk_to_curve25519(&pairs[i].Public[0], &ssh_pub[0])
if pk_rc != 0 {
fmt.println("Error: failed to convert ed25519 public key to curve25519")
delete(pairs)
return
}
ed25519_sk: [64]u8
for j in 0 ..< 32 {
ed25519_sk[j] = ssh_kp.Private[j]
}
for j in 0 ..< 32 {
ed25519_sk[32 + j] = ssh_kp.Public[j]
}
sk_rc := crypto_sign_ed25519_sk_to_curve25519(&pairs[i].Private[0], &ed25519_sk[0])
if sk_rc != 0 {
fmt.println("Error: failed to convert ed25519 private key to curve25519")
delete(pairs)
return
}
}
ok = true
return
}

413
db.odin
View File

@@ -2,12 +2,12 @@ package main
import "core:crypto/hash" import "core:crypto/hash"
import "core:encoding/hex" import "core:encoding/hex"
import "core:encoding/ini"
import "core:encoding/json" import "core:encoding/json"
import "core:fmt" import "core:fmt"
import "core:os" import "core:os"
import "core:path/filepath" import "core:path/filepath"
import "core:strings" import "core:strings"
import "core:time"
import "sqlite" import "sqlite"
@@ -51,15 +51,29 @@ delete_envfile :: proc(f: ^EnvFile) {
delete(f.contents) delete(f.contents)
} }
db_open :: proc(cfg_path: string) -> (database: Db, ok: bool) { make_temp_path :: proc() -> string {
database.cfg = load_config(cfg_path) or_return ts := time.time_to_unix(time.now())
b: strings.Builder
strings.builder_init(&b)
defer strings.builder_destroy(&b)
fmt.sbprintf(&b, "/tmp/envr-%d-%d.db", os.get_pid(), ts)
return strings.to_string(b)
}
db_open :: proc(cfg_path: string) -> (Db, bool) {
cfg, ok := load_config(cfg_path)
if !ok {
return Db{}, false
}
data_path := data_path(cfg.config_path)
_, stat_err := os.stat(data_path, context.allocator)
{
db: ^rawptr db: ^rawptr
rc := sqlite.db_open(":memory:", &db) rc := sqlite.db_open(":memory:", &db)
if rc != sqlite.OK { if rc != sqlite.OK {
fmt.printf("Error opening in-memory database: %s\n", sqlite.db_errmsg(db)) fmt.printf("Error opening in-memory database: %s\n", sqlite.db_errmsg(db))
return return Db{}, false
} }
create_sql: cstring = "CREATE TABLE IF NOT EXISTS envr_env_files (path TEXT PRIMARY KEY NOT NULL, remotes TEXT, sha256 TEXT NOT NULL, contents TEXT NOT NULL)" create_sql: cstring = "CREATE TABLE IF NOT EXISTS envr_env_files (path TEXT PRIMARY KEY NOT NULL, remotes TEXT, sha256 TEXT NOT NULL, contents TEXT NOT NULL)"
@@ -67,69 +81,21 @@ db_open :: proc(cfg_path: string) -> (database: Db, ok: bool) {
if rc != sqlite.OK { if rc != sqlite.OK {
fmt.printf("Error creating table: %s\n", sqlite.db_errmsg(db)) fmt.printf("Error creating table: %s\n", sqlite.db_errmsg(db))
sqlite.db_close(db) sqlite.db_close(db)
return return Db{}, false
}
database.db = db
} }
// TODO: Use different allocators? if stat_err == nil {
data_path := data_path(database.cfg.config_path, context.temp_allocator) if !db_restore_from_encrypted(db, cfg) {
if os.exists(data_path) { sqlite.db_close(db)
if ok = db_restore_from_encrypted(&database, data_path); !ok { return Db{}, false
sqlite.db_close(database.db)
return
} }
} else {
// DB was created
database.changed = true
} }
return database, true return Db{db = db, cfg = cfg, changed = stat_err != nil}, true
}
db_restore_from_encrypted :: proc(db: ^Db, data_path: string) -> bool {
encrypted_data, read_err := os.read_entire_file_from_path(data_path, context.allocator)
defer delete(encrypted_data)
if read_err != nil {
fmt.printf("Error reading encrypted database: %v\n", read_err)
return false
}
plaintext, dec_ok := decrypt(encrypted_data, db.cfg.Keys[:])
if !dec_ok {
fmt.println("Error: decryption failed")
return false
}
defer delete(plaintext)
n := i64(len(plaintext))
buf := sqlite.malloc64(n)
if buf == nil {
fmt.println("Error: failed to allocate buffer for deserialization")
return false
}
copy(buf[:len(plaintext)], plaintext)
rc := sqlite.deserialize(
db.db,
"main",
buf,
n,
n,
sqlite.DESERIALIZE_FREEONCLOSE | sqlite.DESERIALIZE_RESIZEABLE,
)
if rc != sqlite.OK {
sqlite.free(buf)
fmt.printf("Error deserializing database: %s\n", sqlite.db_errmsg(db.db))
return false
}
return true
} }
db_close :: proc(d: ^Db) { db_close :: proc(d: ^Db) {
defer sqlite.db_close(d.db) defer sqlite.db_close(d.db)
defer delete_config(&d.cfg)
if d.changed { if d.changed {
rc := sqlite.db_exec(d.db, "VACUUM", nil, nil, nil) rc := sqlite.db_exec(d.db, "VACUUM", nil, nil, nil)
@@ -168,6 +134,13 @@ db_close :: proc(d: ^Db) {
} }
} }
// Caller is responsible for calling:
// ```odin
// delete(results)
// for &result in results {
// delete(&result)
// }
// ```
db_list :: proc(d: ^Db, allocator := context.allocator) -> (results: [dynamic]EnvFile, ok: bool) { db_list :: proc(d: ^Db, allocator := context.allocator) -> (results: [dynamic]EnvFile, ok: bool) {
stmt: ^rawptr stmt: ^rawptr
rc := sqlite.prepare_v2( rc := sqlite.prepare_v2(
@@ -216,6 +189,145 @@ db_list :: proc(d: ^Db, allocator := context.allocator) -> (results: [dynamic]En
return return
} }
db_restore_from_encrypted :: proc(db: ^rawptr, cfg: Config) -> bool {
encrypted_data, read_err := os.read_entire_file_from_path(
data_path(cfg.config_path),
context.allocator,
)
defer delete(encrypted_data)
if read_err != nil {
fmt.printf("Error reading encrypted database: %v\n", read_err)
return false
}
plaintext, dec_ok := decrypt(encrypted_data, cfg.Keys[:])
if !dec_ok {
fmt.println("Error: decryption failed")
return false
}
defer delete(plaintext)
n := i64(len(plaintext))
buf := sqlite.malloc64(n)
if buf == nil {
fmt.println("Error: failed to allocate buffer for deserialization")
return false
}
copy(buf[:len(plaintext)], plaintext)
rc := sqlite.deserialize(
db,
"main",
buf,
n,
n,
sqlite.DESERIALIZE_FREEONCLOSE | sqlite.DESERIALIZE_RESIZEABLE,
)
if rc != sqlite.OK {
sqlite.free(buf)
fmt.printf("Error deserializing database: %s\n", sqlite.db_errmsg(db))
return false
}
return true
}
get_git_remotes :: proc(dir: string) -> [dynamic]string {
remotes: [dynamic]string
remote_set: map[string]bool
b: strings.Builder
strings.builder_init(&b)
defer strings.builder_destroy(&b)
fmt.sbprintf(&b, "%s-git-remotes", make_temp_path())
tmp_path := strings.to_string(b)
tmp_file, tmp_err := os.open(tmp_path, os.O_CREATE | os.O_WRONLY | os.O_TRUNC)
if tmp_err != nil {
return remotes
}
args := []string{"git", "remote", "-v"}
desc := os.Process_Desc {
command = args,
stdout = tmp_file,
stderr = nil,
working_dir = dir,
}
p, start_err := os.process_start(desc)
os.close(tmp_file)
if start_err != nil {
os.remove(tmp_path)
return remotes
}
state, wait_err := os.process_wait(p)
if wait_err != nil || state.exit_code != 0 {
os.remove(tmp_path)
return remotes
}
data, read_err := os.read_entire_file_from_path(tmp_path, context.allocator)
defer delete(data)
os.remove(tmp_path)
if read_err != nil {
return remotes
}
lines := strings.split(string(data), "\n")
for &line in lines {
line = strings.trim_space(line)
if len(line) == 0 {
continue
}
parts := strings.fields(line)
if len(parts) >= 2 {
remote_set[parts[1]] = true
}
}
for remote, _ in remote_set {
cloned, _ := strings.clone(remote)
append(&remotes, cloned)
}
return remotes
}
new_env_file :: proc(path: string) -> (EnvFile, bool) {
abs_path, abs_err := filepath.abs(path)
if abs_err != nil {
fmt.printf("Error getting absolute path: %v\n", abs_err)
return EnvFile{}, false
}
dir := filepath.dir(abs_path)
remotes := get_git_remotes(dir)
data, read_err := os.read_entire_file_from_path(abs_path, context.allocator)
defer delete(data)
if read_err != nil {
fmt.printf("Error reading file %s: %v\n", abs_path, read_err)
return EnvFile{}, false
}
digest := hash.hash_bytes(hash.Algorithm.SHA256, data, context.temp_allocator)
// TODO: Handle error
hex_bytes, _ := hex.encode(digest)
return EnvFile {
Path = abs_path,
Dir = dir,
Remotes = remotes,
Sha256 = string(hex_bytes),
contents = string(data),
},
true
}
db_insert :: proc(d: ^Db, file: EnvFile) -> bool { db_insert :: proc(d: ^Db, file: EnvFile) -> bool {
remotes_json, marshal_err := json.marshal(file.Remotes) remotes_json, marshal_err := json.marshal(file.Remotes)
if marshal_err != nil { if marshal_err != nil {
@@ -355,36 +467,75 @@ db_delete :: proc(d: ^Db, path: string) -> bool {
return true return true
} }
new_env_file :: proc(path: string) -> (EnvFile, bool) { to_cstring :: proc {
abs_path, abs_err := filepath.abs(path) string_to_cstring,
if abs_err != nil { strings.to_cstring,
fmt.printf("Error getting absolute path: %v\n", abs_err)
return EnvFile{}, false
} }
dir := filepath.dir(abs_path) string_to_cstring :: proc(s: string, allocator := context.allocator) -> cstring {
cs, err := strings.clone_to_cstring(s, allocator)
remotes := get_git_remotes(dir) if err != nil {
fmt.printf("Failed to convert string to cstring: %v\n", err)
data, read_err := os.read_entire_file_from_path(abs_path, context.allocator) panic("Allocation Exception")
defer delete(data) }
if read_err != nil { return cs
fmt.printf("Error reading file %s: %v\n", abs_path, read_err)
return EnvFile{}, false
} }
digest := hash.hash_bytes(hash.Algorithm.SHA256, data, context.temp_allocator) clone_cstring :: proc(c: cstring, allocator := context.allocator) -> string {
// TODO: Handle error str, err := strings.clone_from_cstring(c, allocator)
hex_bytes, _ := hex.encode(digest) if err != nil {
fmt.printf("Failed to convert string to cstring: %v\n", err)
delete(str)
panic("Allocation Exception")
}
return EnvFile { return str
Path = abs_path, }
Dir = dir,
Remotes = remotes, db_update_required :: proc(status: SyncFlag) -> bool {
Sha256 = string(hex_bytes), return .BackedUp in status || .DirUpdated in status
contents = string(data), }
},
true shares_remote :: proc(f: ^EnvFile, remotes: []string) -> bool {
for r1 in f.Remotes {
for r2 in remotes {
if r1 == r2 {
return true
}
}
}
return false
}
update_dir :: proc(f: ^EnvFile, new_dir: string) {
f.Dir = new_dir
base := filepath.base(f.Path)
new_path, _ := strings.concatenate({new_dir, "/", base})
f.Path = new_path
f.Remotes = get_git_remotes(new_dir)
}
find_moved_dirs :: proc(d: ^Db, f: ^EnvFile) -> ([dynamic]string, bool) {
feats := check_features()
if .Fd not_in feats || .Git not_in feats {
fmt.println("Error: fd and git are required for moved dir detection")
return {}, false
}
roots, roots_ok := find_git_roots(d.cfg)
if !roots_ok {
return {}, false
}
moved: [dynamic]string
for root in roots {
remotes := get_git_remotes(root)
if shares_remote(f, remotes[:]) {
cloned, _ := strings.clone(root)
append(&moved, cloned)
}
}
return moved, true
} }
db_sync :: proc(d: ^Db, f: ^EnvFile) -> (SyncFlag, string) { db_sync :: proc(d: ^Db, f: ^EnvFile) -> (SyncFlag, string) {
@@ -463,31 +614,6 @@ env_file_sync :: proc(f: ^EnvFile, dir: SyncDirection, d: ^Db) -> (SyncFlag, str
return result, "" return result, ""
} }
find_moved_dirs :: proc(d: ^Db, f: ^EnvFile) -> ([dynamic]string, bool) {
roots, roots_ok := find_git_roots(d.cfg)
if !roots_ok {
return {}, false
}
moved: [dynamic]string
for root in roots {
remotes := get_git_remotes(root)
if shares_remote(f, remotes[:]) {
cloned, _ := strings.clone(root)
append(&moved, cloned)
}
}
return moved, true
}
update_dir :: proc(f: ^EnvFile, new_dir: string) {
f.Dir = new_dir
base := filepath.base(f.Path)
new_path, _ := strings.concatenate({new_dir, "/", base})
f.Path = new_path
f.Remotes = get_git_remotes(new_dir)
}
// Loads the contents of the the file at f.Path into f.contents // Loads the contents of the the file at f.Path into f.contents
// //
// Caller is responsible for calling delete on f.contents and f.Sha256 // Caller is responsible for calling delete on f.contents and f.Sha256
@@ -509,72 +635,3 @@ env_file_backup :: proc(f: ^EnvFile) -> bool {
return true return true
} }
shares_remote :: proc(f: ^EnvFile, remotes: []string) -> bool {
for r1 in f.Remotes {
for r2 in remotes {
if r1 == r2 {
return true
}
}
}
return false
}
get_git_remotes :: proc(dir: string) -> [dynamic]string {
remotes: [dynamic]string
remote_set: map[string]bool
defer delete(remote_set)
config_path, _ := filepath.join({dir, ".git", "config"}, context.temp_allocator)
m, _, ok := ini.load_map_from_path(config_path, context.allocator)
if !ok {
return remotes
}
defer ini.delete_map(m)
for section_name, section in m {
if strings.has_prefix(section_name, "remote ") {
if url, ok := section["url"]; ok {
remote_set[url] = true
}
}
}
for remote in remote_set {
cloned, _ := strings.clone(remote)
append(&remotes, cloned)
}
return remotes
}
db_update_required :: proc(status: SyncFlag) -> bool {
return .BackedUp in status || .DirUpdated in status
}
to_cstring :: proc {
string_to_cstring,
strings.to_cstring,
}
string_to_cstring :: proc(s: string, allocator := context.allocator) -> cstring {
cs, err := strings.clone_to_cstring(s, allocator)
if err != nil {
fmt.printf("Failed to convert string to cstring: %v\n", err)
panic("Allocation Exception")
}
return cs
}
// Caller is responsible for freeing the result
clone_cstring :: proc(c: cstring, allocator := context.allocator) -> string {
str, err := strings.clone_from_cstring(c, allocator)
if err != nil {
fmt.printf("Failed to convert string to cstring: %v\n", err)
delete(str)
panic("Allocation Exception")
}
return str
}

View File

@@ -319,85 +319,11 @@ test_shares_remote_both_empty :: proc(t: ^testing.T) {
testing.expect(t, !shares_remote(&f, remotes), "both empty should not share") testing.expect(t, !shares_remote(&f, remotes), "both empty should not share")
} }
delete_remotes :: proc(remotes: [dynamic]string) {
for &r in remotes {
delete(r)
}
delete(remotes)
}
@(test) @(test)
test_get_git_remotes_single :: proc(t: ^testing.T) { test_make_temp_path_format :: proc(t: ^testing.T) {
base := fmt.tprintf("/tmp/envr-test-remotes-%d", os.get_pid()) p := make_temp_path()
os.mkdir_all(base) testing.expect(t, strings.has_suffix(p, ".db"), "should end with .db")
defer os.remove_all(base) testing.expect(t, strings.contains(p, fmt.tprintf("%d", os.get_pid())), "should contain PID")
git_dir := fmt.tprintf("%s/.git", base)
os.mkdir_all(git_dir)
config_content := "[core]\n\trepositoryformatversion = 0\n[remote \"origin\"]\n\turl = git@github.com:user/repo.git\n\tfetch = +refs/heads/*:refs/remotes/origin/*\n"
config_path := fmt.tprintf("%s/config", git_dir)
err := os.write_entire_file(config_path, transmute([]u8)config_content)
testing.expect(t, err == nil, "should write .git/config")
remotes := get_git_remotes(base)
defer delete_remotes(remotes)
testing.expect(t, len(remotes) == 1, "should find 1 remote")
if len(remotes) != 1 do return
testing.expect_value(t, remotes[0], "git@github.com:user/repo.git")
}
@(test)
test_get_git_remotes_multiple :: proc(t: ^testing.T) {
base := fmt.tprintf("/tmp/envr-test-remotes-multi-%d", os.get_pid())
os.mkdir_all(base)
defer os.remove_all(base)
git_dir := fmt.tprintf("%s/.git", base)
os.mkdir_all(git_dir)
config_content := "[remote \"origin\"]\n\turl = git@github.com:user/repo.git\n[remote \"upstream\"]\n\turl = https://gitlab.com/upstream/repo.git\n"
config_path := fmt.tprintf("%s/config", git_dir)
err := os.write_entire_file(config_path, transmute([]u8)config_content)
testing.expect(t, err == nil, "should write .git/config")
remotes := get_git_remotes(base)
defer delete_remotes(remotes)
testing.expect(t, len(remotes) == 2, "should find 2 remotes")
}
@(test)
test_get_git_remotes_no_config :: proc(t: ^testing.T) {
base := fmt.tprintf("/tmp/envr-test-remotes-none-%d", os.get_pid())
os.mkdir_all(base)
defer os.remove_all(base)
remotes := get_git_remotes(base)
defer delete_remotes(remotes)
testing.expect(t, len(remotes) == 0, "should return empty when no .git/config")
}
@(test)
test_get_git_remotes_no_remotes :: proc(t: ^testing.T) {
base := fmt.tprintf("/tmp/envr-test-remotes-empty-%d", os.get_pid())
os.mkdir_all(base)
defer os.remove_all(base)
git_dir := fmt.tprintf("%s/.git", base)
os.mkdir_all(git_dir)
config_content := "[core]\n\trepositoryformatversion = 0\n\tbare = false\n"
config_path := fmt.tprintf("%s/config", git_dir)
err := os.write_entire_file(config_path, transmute([]u8)config_content)
testing.expect(t, err == nil, "should write .git/config")
remotes := get_git_remotes(base)
defer delete_remotes(remotes)
testing.expect(t, len(remotes) == 0, "should return empty when no remote sections")
} }
@(test) @(test)
@@ -472,51 +398,3 @@ test_update_dir :: proc(t: ^testing.T) {
testing.expect_value(t, f.Path, "/new/location/.env") testing.expect_value(t, f.Path, "/new/location/.env")
} }
@(test)
test_closing_db_has_no_leaks :: proc(t: ^testing.T) {
base := fmt.tprintf("/tmp/envr-test-leak-%d", os.get_pid())
os.mkdir_all(base)
defer os.remove_all(base)
cfg_path, err := filepath.join([]string{base, "config.json"}, context.temp_allocator)
testing.expect(t, err == nil, "cfgPath should build successfully")
cfg := new_config([]string{"fixtures/keys/insecure-test-key"}, cfg_path)
defer delete_config(&cfg)
testing.expect(t, save_config(cfg, force = true), "save should succeed")
db, ok := db_open(cfg_path)
testing.expect(t, ok, "db should open")
if !ok do return
db_close(&db)
}
@(test)
test_open_existing_db_has_no_leaks :: proc(t: ^testing.T) {
base := fmt.tprintf("/tmp/envr-test-leak-existing-%d", os.get_pid())
os.mkdir_all(base)
defer os.remove_all(base)
cfg_path, err := filepath.join([]string{base, "config.json"}, context.temp_allocator)
testing.expect(t, err == nil, "cfgPath should build successfully")
cfg := new_config([]string{"fixtures/keys/insecure-test-key"}, cfg_path)
defer delete_config(&cfg)
testing.expect(t, save_config(cfg, force = true), "save should succeed")
// First open/close creates data.envr on disk
db, ok := db_open(cfg_path)
testing.expect(t, ok, "db should open")
if !ok do return
f := make_test_env_file("/project/.env", "abc123", "SECRET=value", []string{"git@github.com:user/repo.git"})
defer delete(f.Remotes)
testing.expect(t, db_insert(&db, f), "insert should succeed")
db_close(&db)
// Second open exercises db_restore_from_encrypted
db2, ok2 := db_open(cfg_path)
testing.expect(t, ok2, "db should open existing")
if !ok2 do return
db_close(&db2)
}

View File

@@ -45,6 +45,7 @@ at before, restore your backup with:
* [envr backup](envr_backup.md) - Import a .env file into envr * [envr backup](envr_backup.md) - Import a .env file into envr
* [envr check](envr_check.md) - check if files in the current directory are backed up * [envr check](envr_check.md) - check if files in the current directory are backed up
* [envr deps](envr_deps.md) - Check for missing binaries
* [envr edit-config](envr_edit-config.md) - Edit your config with your default editor * [envr edit-config](envr_edit-config.md) - Edit your config with your default editor
* [envr init](envr_init.md) - Set up envr * [envr init](envr_init.md) - Set up envr
* [envr list](envr_list.md) - View your tracked files * [envr list](envr_list.md) - View your tracked files

24
docs/cli/envr_deps.md Normal file
View File

@@ -0,0 +1,24 @@
## envr deps
Check for missing binaries
### Synopsis
envr relies on external binaries for certain functionality.
The check command reports on which binaries are available and which are not.
```
envr deps [flags]
```
### Options
```
-h, --help help for deps
```
### SEE ALSO
* [envr](envr.md) - Manage your .env files.

51
features.odin Normal file
View File

@@ -0,0 +1,51 @@
package main
import "base:runtime"
import "core:mem"
import "core:os"
import "core:strings"
Feature :: enum {
Git,
Fd,
}
AvailableFeatures :: bit_set[Feature]
check_features :: proc() -> AvailableFeatures {
feats: AvailableFeatures
s: mem.Scratch
mem.scratch_init(&s, 4 * mem.DEFAULT_PAGE_SIZE)
defer mem.scratch_destroy(&s)
context.temp_allocator = mem.scratch_allocator(&s)
path_env := os.get_env("PATH", context.temp_allocator)
paths := strings.split(path_env, ":", context.temp_allocator)
if find_binary(paths, "git") != "" {
feats += {.Git}
}
if find_binary(paths, "fd") != "" {
feats += {.Fd}
}
return feats
}
find_binary :: proc(
paths: []string,
name: string,
allocator: runtime.Allocator = context.temp_allocator,
) -> string {
for p in paths {
candidate := strings.join({strings.trim_right(p, "/"), name}, "/", allocator)
_, err := os.stat(candidate, allocator)
if err == nil {
return candidate
}
}
return ""
}

34
features_test.odin Normal file
View File

@@ -0,0 +1,34 @@
package main
import "core:os"
import "core:strings"
import "core:testing"
@(test)
test_find_binary_exists :: proc(t: ^testing.T) {
path := os.get_env("PATH", context.temp_allocator)
paths := strings.split(path, ":", context.temp_allocator)
result := find_binary(paths, "sh")
testing.expect(t, result != "", "sh should be found on PATH")
}
@(test)
test_find_binary_not_exists :: proc(t: ^testing.T) {
old_path := os.get_env("PATH", context.temp_allocator)
defer {
if old_path != "" {
os.set_env("PATH", old_path)
}
}
os.set_env("PATH", "/tmp/envr-nope")
path := os.get_env("PATH", context.temp_allocator)
paths := strings.split(path, ":", context.temp_allocator)
result := find_binary(paths, "no_such_binary_xyz")
testing.expect(t, result == "", "nonexistent binary should not be found")
}

View File

@@ -0,0 +1,34 @@
# Performance Ideas
Current state after regex→glob migration. findr beats fd in 3/4 cases.
## Benchmark results (2026-06-17)
| Case | fd | findr | Ratio |
|------|------|-------|-------|
| 1 `-E .jj` | 172ms | 135ms | **1.27x faster** |
| 2 `-H` | 1.184s | 1.097s | **1.08x faster** |
| 3 `-HI` | 1.251s | 1.670s | **1.34x slower** |
| 4 `-E .git` | 274ms | 202ms | **1.36x faster** |
Case 3 (`-HI`) skips gitignore entirely, so it's pure I/O + allocation. System time is 2x fd's (12.1s vs 5.5s), pointing to syscall/allocation overhead.
## Completed
1. **Per-thread result buffers** — each thread accumulates locally, merges once at exit. Eliminates per-result mutex contention.
2. **Lean path join**`join_path`/`join_path_dir` use stack buffer + `copy` + single alloc instead of `strings.Builder` + `fmt.sbprintf` + `clone`.
3. **Regex→glob migration** — replaced regex NFA with backtracking glob matcher. Eliminated 27% of CPU spent on `add_thread`/`is_ignored`. Biggest win.
## Remaining ideas
1. **Larger getdents buffer** (8KB → 64KB+)
Fewer syscalls per directory with many entries. Low effort.
2. **Eliminate entry name cloning**
`strings.clone(name)` in `read_dir_entries` heap-allocates per dirent. Names are valid in the getdents buffer during `process_dir`, so the clone may be unnecessary. Low effort.
3. **Arena allocator per thread**
Bump allocator for all transient strings, free once at exit. Bigger change, helps everywhere.
4. **Batched channel** (fd's approach)
Replace global results array with buffered channel of batches. Enables streaming output and sorting like fd does.

342
findr/PLAN.md Normal file
View File

@@ -0,0 +1,342 @@
# findr — Native Odin File Finder (fd Replacement)
## Overview
findr is a native Odin file finder that replaces `fd` in envr. It supports three ignore modes for A/B benchmarking against specific fd commands, plus a unique "emit ONLY gitignored files" mode that gives envr a single-pass advantage over fd's double-run-and-diff approach.
## Directory Structure
```
findr/
findr.odin # main + CLI (hand-rolled arg parsing)
walker.odin # parallel directory walker (getdents + thread pool)
gitignore.odin # .gitignore parsing + glob→regex transpilation + matching
test_env.odin # test harness: temp dir, mock filesystem, assert helpers
findr_test.odin # integration tests
gitignore_test.odin # transpilation + matching unit tests (22 tests)
```
## CLI Interface
```
findr [-I] [--ignored] [--no-hidden] [-E <glob>]... [pattern] [path]...
```
Defaults: `include_hidden=true, ignore_mode=.Respected` (matches fd's `-H` behavior).
| fd command | findr equivalent |
|---|---|
| `fd -a \.env -E ... -HI ~/` | `findr -I -E ... \.env ~/` |
| `fd -a \.env -E ... -H ~/` | `findr -E ... \.env ~/` |
| `fd . -H ~/` | `findr ~/` |
| `fd . -HI ~/` | `findr -I ~/` |
| `fd . ~/` (no flags) | `findr --no-hidden ~/` |
| *(findr original)* | `findr --ignored ~/` |
## Build
```bash
odin build findr -o:speed -out:findr/findr
odin test findr
```
## Architecture
### Two Orthogonal Axes (matching fd's semantics)
1. **Hidden files** (`.` prefix): `include_hidden=true` includes them, `false` excludes them
2. **Gitignore**: three modes (see `IgnoreMode` below)
### Types
```odin
IgnoreMode :: enum {
Respected, // skip gitignored, prune ignored dirs (fd -H default)
All, // ignore .gitignore entirely, descend everywhere (fd -HI)
Ignored, // emit ONLY gitignored files, prune ignored dirs (findr original)
}
WalkOptions :: struct {
pattern: string, // regex on basename; "" = match all
excludes: []string, // glob patterns to skip entirely (fd -E)
include_hidden: bool, // true = include dotfiles (fd -H)
ignore_mode: IgnoreMode,
}
```
### process_dir Filtering Order Per Entry
Each directory traversal carries a `WorkItem` with the absolute path, a relative path from repo root, and a `^GIContext` linked list of gitignore contexts (one per ancestor directory with a `.gitignore`).
1. Skip `.git` directory
2. **Load nested `.gitignore`**: If this directory has a `.gitignore`, push a new `GIContext` onto the chain (tracked in `pool.all_contexts` for cleanup)
3. **Per entry**:
- Skip non-regular files (symlinks, sockets, etc. — parity with `fd -t f`)
- **Excludes**: if entry matches any exclude glob → skip entirely
- **Hidden**: if `!include_hidden && name[0] == '.'` → skip entirely
- **Gitignore status**: check `GIContext` chain deepest-to-root via `check_chain`, passing the **relative path** (not basename). First match wins (correct gitignore precedence). Nested negation overrides parent rules.
- **Mode-based decision**:
| Mode | gitignored file | gitignored dir | normal file | normal dir |
|---|---|---|---|---|
| `.All` | emit if pattern matches | descend | emit if pattern matches | descend |
| `.Respected` | skip | prune | emit if pattern matches | descend |
| `.Ignored` | emit if pattern matches | prune | skip | descend |
**Nested repos**: When a directory contains `.git/`, the gitignore context chain is reset (new repo root). The relative path resets to `""`. Nested repos are always traversed to find deeper repos.
### Performance Architecture
- **Stat avoidance via `dirent.type`** — Uses `core:sys/linux` getdents directly, bypassing `core:os` which calls `openat` + `fstat` per entry.
- **Prune ignored directories** — When a directory matches a gitignore/exclude pattern, it is not descended into.
- **Parallel traversal** — Worker thread pool with shared LIFO queue and futex-based semaphore signaling. 5.4x speedup over serial on home directory.
## Decisions
- **Gitignore matching**: Transpile gitignore glob patterns to regex, then use `core:text/regex`. No dedicated glob matcher.
- **Pattern matching**: Pattern is a regex (same as fd), matched against basename via `regex.match` (unanchored search).
- **Excludes**: Glob patterns compiled via the same gitignore transpiler (`parse()`). Reuses tested transpilation logic.
- **Nested gitignore**: Every `.gitignore` file within a repo is read, not just the root. Each directory's rules are scoped relative to that directory's path. Negation in a child overrides parent rules (correct gitignore precedence).
- **Stat avoidance**: Use `core:sys/linux` getdents directly — read `dirent.type` from the kernel, never call stat. `DT_UNKNOWN` treated as regular file (correct for ext4/tmpfs; may miss dirs on XFS/BTRFS/FUSE — Phase 7 concern).
## Testing Strategy
- **In-process integration tests** — Tests call `walk()` directly (not via subprocess), build mock filesystems in temp dirs, and compare sorted output.
- **Unit tests** — Pure-function tests for glob→regex transpilation and gitignore matching.
- **Output sorting for determinism** — Always sort output lines before comparison.
- **Memory tracking** — Odin's test runner reports leaks automatically.
### Test Coverage (findr_test.odin)
**`.Ignored` mode (original findr behavior):**
| Test | What it covers |
|---|---|
| `test_basic_gitignored` | Repo with `.gitignore`, gitignored files emitted, normal files skipped |
| `test_non_repo_not_scanned` | Dirs without `.git/` produce no output |
| `test_negation_pattern` | `!prod.env` un-ignores a file |
| `test_dir_only_pattern` | `node_modules/` pattern doesn't emit file results |
| `test_multiple_repos` | Multiple repos in one tree, each with its own `.gitignore` |
| `test_nested_repos` | Repo inside a repo, both scanned independently |
| `test_no_gitignore_file` | Repo with `.git/` but no `.gitignore` produces nothing |
| `test_empty_gitignore` | Comments and blank lines only → no results |
| `test_multiple_search_dirs` | Multiple top-level search dirs in one call |
| `test_nested_gitignore_read` | Nested `.gitignore` rules applied (subdir patterns work) |
| `test_nested_gitignore_negation` | Nested negation overrides parent pattern |
| `test_multisegment_pattern` | `build/output.txt` matches relative path, not just basename |
**`.All` mode (fd -HI parity):**
| Test | What it covers |
|---|---|
| `test_all_mode_emits_all_files` | All files emitted regardless of gitignore |
| `test_all_mode_descends_everywhere` | Gitignored dirs still descended |
**`.Respected` mode (fd -H parity):**
| Test | What it covers |
|---|---|
| `test_respected_mode_skips_gitignored` | Gitignored files skipped |
| `test_respected_mode_prunes_ignored_dirs` | Gitignored dirs pruned |
| `test_nested_gitignore_respected_mode` | Nested negation respected in `.Respected` mode |
**Filters:**
| Test | What it covers |
|---|---|
| `test_excludes_prune_dirs` | Excluded dirs not descended |
| `test_pattern_filters_results` | Only pattern-matching files emitted |
| `test_no_hidden_skips_dotfiles` | Hidden files skipped when include_hidden=false |
## Glob→Regex Transpilation Rules
| Gitignore pattern | Regex | Notes |
|---|---|---|
| `foo` | `(^|/)foo(/.*)?$` | matches at any depth |
| `/foo` | `^foo(/.*)?$` | anchored to gitignore dir |
| `foo/` | `(^|/)foo/.*$` | directory only |
| `*.log` | `(^|/)[^/]*\.log$` | `*` = any chars except `/` |
| `**/foo` | `(^|/)(.*/)?foo(/.*)?$` | `**` = any chars including `/` |
| `foo/**/bar` | `(^|/)foo/(.*/)?bar(/.*)?$` | `**` between segments |
| `!pattern` | (handled by layer) | negation flag, not regex |
| `#comment` | (skipped) | |
| `[abc]` | `[abc]` | same regex syntax |
| `?` | `[^/]` | single char, no `/` |
## Implementation Phases
### Phase 1: Gitignore Transpiler + Tests ✅
22 tests, all passing, zero leaks.
### Phase 2: findr Walker + Tests ✅
Parallel DFS using getdents with worker thread pool. 32 total tests pass, zero leaks.
### Phase 3: Parallel Traversal ✅
8-worker thread pool, shared LIFO queue, futex-based semaphore. 852ms vs 4.57s serial (5.4x speedup). Serial code removed — parallel is the only implementation.
### Phase 4: Benchmark ✅
findr found 227 gitignored files on `~` in 852ms. fd's double-run walked ~1.1M entries.
### Phase 5: fd-Parity API ✅
**Goal:** Make findr replicate specific fd commands for A/B benchmarking, plus keep the unique gitignored-only mode.
**Built:**
- `IgnoreMode` enum (`.Respected`, `.All`, `.Ignored`) and `WalkOptions` struct
- New `walk` signature: `walk(root, results, opts: WalkOptions, thread_count)`
- Rewritten `process_dir` with centralized mode-based filtering
- Pattern matching via `core:text/regex` on basenames
- Exclude patterns compiled via existing `gitignore.parse()`
- CLI arg parsing: `-I`, `--ignored`, `--no-hidden`, `-E <glob>`
- 7 new integration tests (17 total) covering all three modes, excludes, pattern, and hidden filtering
**Result:** All tests pass (22 gitignore + 20 walker = 42), zero leaks.
### Phase 6: Parity (partially done)
**Goal:** Achieve file-count parity with fd. An invalid benchmark (different result sets) is useless.
#### Steps 1-2: Nested gitignore + relative path matching ✅
**What was done:**
1. **`Match` enum + `check_match`** in `gitignore.odin` — Tri-state return (`None`/`Ignored`/`Unignored`) so nested negation overrides work correctly. `is_ignored` wraps it as before.
2. **`GIContext` linked list** in `walker.odin` — Each context holds a `^Gitignore`, `base_rel` (relative path from repo root to this dir), and `parent: ^GIContext`. `process_dir` loads `.gitignore` in every directory within a repo (not just roots). `check_chain` walks deepest-to-root, first match wins (correct gitignore precedence).
3. **`WorkItem` struct** replaced plain `string` in the work queue:
```odin
WorkItem :: struct {
path: string, // absolute directory path
rel: string, // relative path from repo root ("" = root)
gi_ctx: ^GIContext, // gitignore chain (nil = outside any repo)
}
```
4. **Relative path matching** — `check_chain` strips each context's `base_rel` prefix to get the locally-scoped relative path. Multi-segment patterns like `build/output.txt` now match correctly.
5. **Symlink filtering** — Only `DT_REG` and `DT_UNKNOWN` entries are emitted (matching `fd -t f`). Symlinks (`DT_LNK`) are skipped.
6. **`DT_UNKNOWN` handling** — Treated as regular files (no stat fallback). Correct for ext4/tmpfs; may miss directories on XFS/BTRFS/FUSE.
**Memory management:** All `GIContext` objects tracked in `pool.all_contexts` (mutex-protected append). Gitignore objects and context structs freed in bulk when `walk` completes.
**Parity achieved** (`~`, 5M+ files):
| Mode | findr | fd equivalent | diff |
|---|---|---|---|
| `.All` (-I) | 5,426,451 | `fd -HI -t f --exclude .git` | **0 (exact)** |
| `.Respected` | 4,442,505 | `fd -H -t f --exclude .git` | +1,417 (0.03%) |
| `--no-hidden` | 393,605 | `fd -t f --exclude .git` | +17 (0.004%) |
On the envr repo itself, all three modes are **exact match (0 diffs)**. The tiny residual diffs on `~` are likely from global gitignore (`~/.config/git/ignore`) and `.git/info/exclude` which fd reads but findr doesn't.
#### Step 3: DT_UNKNOWN stat fallback (TODO)
On XFS/BTRFS/FUSE filesystems, `dirent.type` returns `DT_UNKNOWN`. Currently findr treats these as regular files, which means directories may be missed (not descended into). Add a stat fallback in `read_dir_entries` when `d.type == .UNKNOWN` to determine the real type before proceeding. This is not needed for ext4/tmpfs (what tests and most Linux systems use).
### Phase 7: Performance Optimization (next)
**Goal:** Make findr competitive with or faster than fd across all modes. Current benchmark (`~`, hyperfine 5 runs):
| Command | Mean | vs fd equivalent |
|---|---|---|
| `findr --ignored` | 984ms | *(no fd equivalent)* |
| `findr --no-hidden` | 542ms | 3.2x slower than `fd -t f` (170ms) |
| `findr` (respected) | 4.134s | 2.4x slower than `fd -H -t f` (1.745s) |
| `findr -I` (all) | 3.821s | 1.9x slower than `fd -HI -t f` (1.972s) |
**Bottleneck analysis:**
1. **Mutex contention on result collection** — Every file append goes through `sync.mutex_lock(&pool.results_mutex)` → `append` → `sync.mutex_unlock`. With 5M+ files across 16 threads, workers serialize on the mutex.
2. **`--ignored` regression** — Was 402ms before nested gitignore support, now 984ms. The overhead comes from loading `.gitignore` in every directory and checking the context chain per entry. Since `--ignored` mode prunes gitignored dirs, many of these `.gitignore` loads are wasted (the dir won't be descended into anyway). Optimization: skip loading `.gitignore` for directories that will be pruned.
3. **Per-string heap allocation** — Every path string is individually `strings.clone`'d and `delete`'d. Millions of alloc/free calls.
**Optimization plan:**
1. **Per-thread result buffers** — Each worker accumulates results in a thread-local `[dynamic]string`. Merge into shared array once at the end (single-threaded concat).
2. **Lazy gitignore loading for `.Ignored` mode** — Only load `.gitignore` when we need to decide whether to emit or descend. In `.Ignored` mode, we can check the parent context first and skip loading if the directory itself is already ignored.
3. **Arena allocator for paths** — Replace per-string `strings.clone` with a bump allocator. Free everything in one `arena_destroy` at the end.
4. **Larger getdents buffer** — Increase from 8KB to 64KB to reduce syscall count.
5. **BufWriter on stdout** — Batch `write` syscalls instead of per-line `fmt.println`.
**Success criteria:**
- `.All` mode faster than `fd -HI -t f --exclude .git`
- `.Respected` mode faster than `fd -H -t f --exclude .git`
- `--ignored` mode faster than `fd -HI -t f --exclude .git` (restore pre-regression advantage)
- Re-benchmark after each step using `findr/bench.sh`
### Phase 8: Integrate into envr
**Goal:** Replace ALL `fd` subprocess usage in envr with in-process findr calls. Remove `Feature.Fd` entirely.
#### Part A: Rewrite `scan_path` (`scan.odin`)
Replace the double-run-and-diff approach with a single `findr.walk` call using `.Ignored` mode:
```odin
// Before: fd -HI + fd -H, then diff
// After:
findr.walk(search_path, &paths, WalkOptions{
pattern = cfg.ScanConfig.Matcher,
excludes = cfg.ScanConfig.Exclude[:],
include_hidden = true,
ignore_mode = .Ignored,
}, thread_count)
```
**Delete:** `build_fd_args`, `run_fd`, `next_fd_tmp_path`, `fd_counter`, `fd_seq`, `cant_scan`.
#### Part B: Add `find_repos` and rewrite `find_git_roots` (`config.odin`)
Add a `find_repos` proc to findr that walks a tree and collects directories containing `.git/`:
```odin
find_repos :: proc(root: string, results: ^[dynamic]string, thread_count: int)
```
- Reuses worker pool architecture
- `process_dir` emits `dir_path` when `has_git == true`
- Always descends into subdirs (except `.git`) to find nested repos
- No gitignore/exclude/pattern processing
Replace `find_git_roots`'s `run_fd` call with `findr.find_repos`.
#### Part C: Remove `Feature.Fd` everywhere
| File | Change |
|---|---|
| `features.odin` | Remove `Fd` from enum, remove fd binary check |
| `cmd_scan.odin` | Remove feats/cant_scan guard + "install fd" error |
| `cmd_check.odin` | Same removal |
| `cmd_deps.odin` | Remove fd table row |
| `db.odin` | Change check to `.Git not_in feats` only; update error message |
| `scan_test.odin` | Remove `cant_scan` tests and assertions |
#### Part D: Verification
```bash
odin build findr -o:speed -out:findr/findr
odin test findr
odin build . -o:speed -out:envr
odin test .
```
## Risks
| Risk | Mitigation |
|---|---|
| Gitignore edge cases (`**/foo`, `foo/**/bar`) | Comprehensive gitignore_test.odin with spec examples |
| `DT_UNKNOWN` on XFS/BTRFS/FUSE | Phase 6 Step 3: stat fallback for unknown types |
| Global gitignore (`~/.config/git/ignore`) and `.git/info/exclude` not read | Causes ~0.03% delta vs fd. Acceptable for envr's use case (finds `.env` files in repos). |
| Thread safety of `regex.match` on shared `Regular_Expression` | Odin regex is read-only after compilation; `match` returns per-call `Captures` |

71
findr/bench.sh Executable file
View File

@@ -0,0 +1,71 @@
#!/usr/bin/env bash
set -euo pipefail
BENCH_DIR="$(cd "$(dirname "$0")" && pwd)"
TARGET="${1:-$HOME}"
RESULTS_FILE="$BENCH_DIR/bench-results.md"
FINDR="$BENCH_DIR/findr"
echo "=== findr benchmark suite ==="
echo "Target: $TARGET"
echo
# --- pre-flight checks ---
if ! command -v fd &>/dev/null; then
echo "ERROR: fd is not on PATH" >&2
exit 1
fi
if ! command -v hyperfine &>/dev/null; then
echo "ERROR: hyperfine is not on PATH" >&2
exit 1
fi
# --- build findr if missing or stale ---
NEEDS_BUILD=false
if [[ ! -f "$BENCH_DIR/findr" ]]; then
NEEDS_BUILD=true
else
# rebuild if any .odin source is newer than the binary
if find "$BENCH_DIR" -name '*.odin' -newer "$BENCH_DIR/findr" | grep -q .; then
NEEDS_BUILD=true
fi
fi
if $NEEDS_BUILD; then
echo "Building findr..."
odin build "$BENCH_DIR" -o:speed -out:"$BENCH_DIR/findr"
fi
echo
# --- file counts ---
echo "=== File counts ==="
printf " fd -a -E .jj . : %8d\n" "$(fd -a -E .jj . "$TARGET" 2>/dev/null | wc -l)"
printf " findr -E .jj : %8d\n" "$("$FINDR" -E .jj "$TARGET" 2>/dev/null | wc -l)"
echo
printf " fd -a -E .git -E .jj -H . : %8d\n" "$(fd -a -E .git -E .jj -H . "$TARGET" 2>/dev/null | wc -l)"
printf " findr -E .git -E .jj -H : %8d\n" "$("$FINDR" -E .git -E .jj -H "$TARGET" 2>/dev/null | wc -l)"
echo
printf " fd -a -E .git -E .jj -HI . : %8d\n" "$(fd -a -E .git -E .jj -HI . "$TARGET" 2>/dev/null | wc -l)"
printf " findr -E .git -E .jj -HI : %8d\n" "$("$FINDR" -E .git -E .jj -HI "$TARGET" 2>/dev/null | wc -l)"
echo
printf " fd -a -E .git -E .jj . : %8d\n" "$(fd -a -E .git -E .jj . "$TARGET" 2>/dev/null | wc -l)"
printf " findr -E .git -E .jj : %8d\n" "$("$FINDR" -E .git -E .jj "$TARGET" 2>/dev/null | wc -l)"
echo
# --- benchmarks ---
echo "=== Benchmarks (hyperfine, 5 runs, 2 warmups) ==="
echo
hyperfine \
--warmup 2 \
--runs 5 \
--export-markdown "$RESULTS_FILE" \
"fd -a -E .jj . \"$TARGET\" > /dev/null" \
"$FINDR -E .jj \"$TARGET\" > /dev/null" \
"fd -a -E .git -E .jj -H . \"$TARGET\" > /dev/null" \
"$FINDR -E .git -E .jj -H \"$TARGET\" > /dev/null" \
"fd -a -E .git -E .jj -HI . \"$TARGET\" > /dev/null" \
"$FINDR -E .git -E .jj -HI \"$TARGET\" > /dev/null" \
"fd -a -E .git -E .jj . \"$TARGET\" > /dev/null" \
"$FINDR -E .git -E .jj \"$TARGET\" > /dev/null"
echo
echo "=== Results written to $RESULTS_FILE ==="

27
findr/f.nu Executable file
View File

@@ -0,0 +1,27 @@
#!/usr/bin/env nu
def main [] {
let all = (fd -HI -a .env . ~/ | lines | sort)
let unignored = (fd -H -a .env ~/ | lines | sort)
$all | filter { |it| not ($it in $unignored) } | str join "\n"
# sorted_list_intersect $all $unignored | str join "\n"
}
def sorted_list_intersect [xs1: list, xs2: list] {
let len1 = ($xs1 | length)
let len2 = ($xs2 | length)
mut i = 0
mut j = 0
while ($i < $len1 and $j < $len2) {
if ($xs1 | get $i) < ($xs2 | get $j) {
$i = $i + 1
} else if ($xs2 | get $j) < ($xs1 | get $i) {
$j = $j + 1
} else {
echo ($xs2 | get $j)
$i = $i + 1
$j = $j + 1
}
}
}

91
findr/findr.odin Normal file
View File

@@ -0,0 +1,91 @@
package findr
import "core:bufio"
import "core:os"
import "core:strings"
main :: proc() {
prof_init()
defer prof_destroy()
args := os.args
opts: WalkOptions
opts.include_hidden = false
opts.ignore_mode = .Respected
excludes := make([dynamic]string)
defer delete(excludes)
pattern := ""
paths := make([dynamic]string)
defer delete(paths)
i := 1
for i < len(args) {
arg := args[i]
switch {
case arg == "--ignored":
opts.ignore_mode = .Ignored
case arg == "-E":
i += 1
if i < len(args) {
append(&excludes, args[i])
}
case strings.has_prefix(arg, "-E"):
append(&excludes, arg[2:])
case len(arg) > 1 && arg[0] == '-':
for c, j in arg[1:] {
switch c {
case 'H':
opts.include_hidden = true
case 'I':
opts.ignore_mode = .All
case 'a':
// no-op: accepted for fd compatibility
}
}
case:
if pattern == "" {
pattern = arg
} else {
append(&paths, arg)
}
}
i += 1
}
if len(paths) == 0 && pattern != "" && os.exists(pattern) {
append(&paths, pattern)
pattern = ""
}
opts.pattern = pattern
if len(excludes) > 0 {
opts.excludes = excludes[:]
}
if len(paths) == 0 {
append(&paths, ".")
}
results := make([dynamic]string)
defer {
for r in results {delete(r)}
delete(results)
}
thread_count := os.get_processor_core_count()
walk(paths[:], &results, opts, thread_count)
w: bufio.Writer
bufio.writer_init(&w, os.to_stream(os.stdout), 1 << 13)
defer bufio.writer_destroy(&w)
for r in results {
bufio.writer_write_string(&w, r)
bufio.writer_write_byte(&w, '\n')
}
bufio.writer_flush(&w)
}

View File

@@ -7,7 +7,7 @@ import "core:sys/linux"
import "core:testing" import "core:testing"
// ============================================================================ // ============================================================================
// Gitignored file emission tests (emit ONLY gitignored files, descend everywhere) // .Ignored mode tests (original findr behavior — emit ONLY gitignored files)
// ============================================================================ // ============================================================================
@(test) @(test)
@@ -21,7 +21,7 @@ test_basic_gitignored :: proc(t: ^testing.T) {
create_file(env, "repo/secrets.env") create_file(env, "repo/secrets.env")
create_file(env, "repo/normal.txt") create_file(env, "repo/normal.txt")
assert_output(t, env, nil, {}, { assert_output(t, env, nil, {include_hidden = true, ignore_mode = .Ignored}, {
"repo/.env", "repo/secrets.env", "repo/.env", "repo/secrets.env",
}) })
} }
@@ -35,7 +35,7 @@ test_non_repo_not_scanned :: proc(t: ^testing.T) {
create_file(env, "norepo/.gitignore", "*.env\n") create_file(env, "norepo/.gitignore", "*.env\n")
create_file(env, "norepo/.env") create_file(env, "norepo/.env")
assert_output_empty(t, env, nil, {}) assert_output_empty(t, env, nil, {include_hidden = true, ignore_mode = .Ignored})
} }
@(test) @(test)
@@ -49,11 +49,28 @@ test_negation_pattern :: proc(t: ^testing.T) {
create_file(env, "repo/secrets.env") create_file(env, "repo/secrets.env")
create_file(env, "repo/prod.env") create_file(env, "repo/prod.env")
assert_output(t, env, nil, {}, { assert_output(t, env, nil, {include_hidden = true, ignore_mode = .Ignored}, {
"repo/.env", "repo/secrets.env", "repo/.env", "repo/secrets.env",
}) })
} }
@(test)
test_dir_only_pattern :: proc(t: ^testing.T) {
env := create_test_env()
defer destroy_test_env(&env)
create_git_repo(env, "repo")
create_file(env, "repo/.gitignore", "node_modules/\n")
create_file(env, "repo/node_modules", "should not match (it's a file)")
create_dir(env, "repo/ignored_dir")
create_file(env, "repo/.gitignore", "ignored_dir/\n")
assert_output(t, env, nil, {include_hidden = true, ignore_mode = .Ignored}, {
"repo/ignored_dir/",
})
}
@(test) @(test)
test_multiple_repos :: proc(t: ^testing.T) { test_multiple_repos :: proc(t: ^testing.T) {
env := create_test_env() env := create_test_env()
@@ -67,7 +84,7 @@ test_multiple_repos :: proc(t: ^testing.T) {
create_file(env, "repo2/.gitignore", "*.key\n") create_file(env, "repo2/.gitignore", "*.key\n")
create_file(env, "repo2/secret.key") create_file(env, "repo2/secret.key")
assert_output(t, env, nil, {}, { assert_output(t, env, nil, {include_hidden = true, ignore_mode = .Ignored}, {
"repo1/a.env", "repo2/secret.key", "repo1/a.env", "repo2/secret.key",
}) })
} }
@@ -85,7 +102,7 @@ test_nested_repos :: proc(t: ^testing.T) {
create_file(env, "parent/child/.gitignore", "*.key\n") create_file(env, "parent/child/.gitignore", "*.key\n")
create_file(env, "parent/child/api.key") create_file(env, "parent/child/api.key")
assert_output(t, env, nil, {}, { assert_output(t, env, nil, {include_hidden = true, ignore_mode = .Ignored}, {
"parent/top.env", "parent/child/api.key", "parent/top.env", "parent/child/api.key",
}) })
} }
@@ -102,7 +119,10 @@ test_nested_gitignore_read :: proc(t: ^testing.T) {
create_file(env, "repo/sub/secret.txt") create_file(env, "repo/sub/secret.txt")
create_file(env, "repo/sub/.env") create_file(env, "repo/sub/.env")
assert_output(t, env, nil, {}, { // Both root and nested .gitignore are read.
// secret.txt: ignored by sub/.gitignore (*.txt)
// .env: ignored by root .gitignore (*.env)
assert_output(t, env, nil, {include_hidden = true, ignore_mode = .Ignored}, {
"repo/sub/secret.txt", "repo/sub/.env", "repo/sub/secret.txt", "repo/sub/.env",
}) })
} }
@@ -119,11 +139,34 @@ test_nested_gitignore_negation :: proc(t: ^testing.T) {
create_file(env, "repo/sub/important.log") create_file(env, "repo/sub/important.log")
create_file(env, "repo/sub/debug.log") create_file(env, "repo/sub/debug.log")
assert_output(t, env, nil, {}, { // Nested negation overrides root pattern.
// important.log: un-ignored by sub/.gitignore → NOT emitted in .Ignored mode
// debug.log: still ignored by root → emitted
assert_output(t, env, nil, {include_hidden = true, ignore_mode = .Ignored}, {
"repo/sub/debug.log", "repo/sub/debug.log",
}) })
} }
@(test)
test_nested_gitignore_respected_mode :: proc(t: ^testing.T) {
env := create_test_env()
defer destroy_test_env(&env)
create_git_repo(env, "repo")
create_file(env, "repo/.gitignore", "*.log\n")
create_dir(env, "repo/sub")
create_file(env, "repo/sub/.gitignore", "!important.log\n")
create_file(env, "repo/sub/important.log")
create_file(env, "repo/sub/debug.log")
// In .Respected mode:
// important.log: un-ignored by nested negation → emitted
// debug.log: ignored by root → skipped
assert_output(t, env, nil, {include_hidden = true, ignore_mode = .Respected}, {
"repo/", "repo/.gitignore", "repo/sub/", "repo/sub/.gitignore", "repo/sub/important.log",
})
}
@(test) @(test)
test_multisegment_pattern :: proc(t: ^testing.T) { test_multisegment_pattern :: proc(t: ^testing.T) {
env := create_test_env() env := create_test_env()
@@ -136,7 +179,11 @@ test_multisegment_pattern :: proc(t: ^testing.T) {
create_file(env, "repo/build/other.txt") create_file(env, "repo/build/other.txt")
create_file(env, "repo/output.txt") create_file(env, "repo/output.txt")
assert_output(t, env, nil, {}, { // Multi-segment pattern matches relative path, not just basename.
// build/output.txt: matches → ignored
// build/other.txt: doesn't match → not ignored
// output.txt: doesn't match (needs build/ prefix) → not ignored
assert_output(t, env, nil, {include_hidden = true, ignore_mode = .Ignored}, {
"repo/build/output.txt", "repo/build/output.txt",
}) })
} }
@@ -149,7 +196,7 @@ test_no_gitignore_file :: proc(t: ^testing.T) {
create_git_repo(env, "repo") create_git_repo(env, "repo")
create_file(env, "repo/.env") create_file(env, "repo/.env")
assert_output_empty(t, env, nil, {}) assert_output_empty(t, env, nil, {include_hidden = true, ignore_mode = .Ignored})
} }
@(test) @(test)
@@ -161,7 +208,7 @@ test_empty_gitignore :: proc(t: ^testing.T) {
create_file(env, "repo/.gitignore", "\n\n# comment\n\n") create_file(env, "repo/.gitignore", "\n\n# comment\n\n")
create_file(env, "repo/.env") create_file(env, "repo/.env")
assert_output_empty(t, env, nil, {}) assert_output_empty(t, env, nil, {include_hidden = true, ignore_mode = .Ignored})
} }
@(test) @(test)
@@ -189,7 +236,7 @@ test_multiple_search_dirs :: proc(t: ^testing.T) {
delete(results) delete(results)
} }
opts := WalkOptions{} opts := WalkOptions{include_hidden = true, ignore_mode = .Ignored}
thread_count := os.get_processor_core_count() thread_count := os.get_processor_core_count()
walk({dir1, dir2}, &results, opts, thread_count) walk({dir1, dir2}, &results, opts, thread_count)
@@ -219,46 +266,78 @@ test_multiple_search_dirs :: proc(t: ^testing.T) {
} }
// ============================================================================ // ============================================================================
// Ignored directory recursion tests // .All mode tests (fd -HI parity — ignore gitignore entirely)
// ============================================================================ // ============================================================================
@(test) @(test)
test_ignored_dir_descended :: proc(t: ^testing.T) { test_all_mode_emits_all_files :: proc(t: ^testing.T) {
env := create_test_env() env := create_test_env()
defer destroy_test_env(&env) defer destroy_test_env(&env)
create_git_repo(env, "repo") create_git_repo(env, "repo")
create_file(env, "repo/.gitignore", "secrets/\n") create_file(env, "repo/.gitignore", "*.env\n")
create_dir(env, "repo/secrets") create_file(env, "repo/.env")
create_file(env, "repo/secrets/.env") create_file(env, "repo/secrets.env")
create_file(env, "repo/secrets/api.key") create_file(env, "repo/normal.txt")
// Ignored dir's contents are emitted AND descended into assert_output(t, env, nil, {include_hidden = true, ignore_mode = .All}, {
assert_output(t, env, nil, {}, { "repo/", "repo/.env", "repo/.gitignore", "repo/secrets.env", "repo/normal.txt",
"repo/secrets/", "repo/secrets/.env", "repo/secrets/api.key",
}) })
} }
@(test) @(test)
test_nested_ignored_dir :: proc(t: ^testing.T) { test_all_mode_descends_everywhere :: proc(t: ^testing.T) {
env := create_test_env() env := create_test_env()
defer destroy_test_env(&env) defer destroy_test_env(&env)
create_git_repo(env, "repo") create_git_repo(env, "repo")
create_file(env, "repo/.gitignore", "build/\n") create_file(env, "repo/.gitignore", "build/\n")
create_dir(env, "repo/build") create_dir(env, "repo/build")
create_dir(env, "repo/build/sub")
create_file(env, "repo/build/output.txt") create_file(env, "repo/build/output.txt")
create_file(env, "repo/build/sub/deep.env")
assert_output(t, env, nil, {}, { assert_output(t, env, nil, {include_hidden = true, ignore_mode = .All}, {
"repo/build/", "repo/build/output.txt", "repo/", "repo/.gitignore", "repo/build/", "repo/build/output.txt",
"repo/build/sub/", "repo/build/sub/deep.env",
}) })
} }
// ============================================================================ // ============================================================================
// Filter tests (excludes, pattern) // .Respected mode tests (fd -H parity — skip gitignored, prune ignored dirs)
// ============================================================================
@(test)
test_respected_mode_skips_gitignored :: proc(t: ^testing.T) {
env := create_test_env()
defer destroy_test_env(&env)
create_git_repo(env, "repo")
create_file(env, "repo/.gitignore", "*.env\n")
create_file(env, "repo/.env")
create_file(env, "repo/secrets.env")
create_file(env, "repo/normal.txt")
assert_output(t, env, nil, {include_hidden = true, ignore_mode = .Respected}, {
"repo/", "repo/.gitignore", "repo/normal.txt",
})
}
@(test)
test_respected_mode_prunes_ignored_dirs :: proc(t: ^testing.T) {
env := create_test_env()
defer destroy_test_env(&env)
create_git_repo(env, "repo")
create_file(env, "repo/.gitignore", "build/\n")
create_dir(env, "repo/build")
create_file(env, "repo/build/output.txt")
create_file(env, "repo/main.txt")
assert_output(t, env, nil, {include_hidden = true, ignore_mode = .Respected}, {
"repo/", "repo/.gitignore", "repo/main.txt",
})
}
// ============================================================================
// Filter tests (excludes, pattern, hidden)
// ============================================================================ // ============================================================================
@(test) @(test)
@@ -273,7 +352,7 @@ test_excludes_prune_dirs :: proc(t: ^testing.T) {
create_file(env, "repo/vendor/lib.env") create_file(env, "repo/vendor/lib.env")
assert_output(t, env, nil, assert_output(t, env, nil,
{excludes = {"vendor"}}, {include_hidden = true, ignore_mode = .Ignored, excludes = {"vendor"}},
{"repo/.env"}, {"repo/.env"},
) )
} }
@@ -290,13 +369,30 @@ test_pattern_filters_results :: proc(t: ^testing.T) {
create_file(env, "repo/master.key") create_file(env, "repo/master.key")
assert_output(t, env, nil, assert_output(t, env, nil,
{pattern = "\\.env$"}, {pattern = "\\.env$", include_hidden = true, ignore_mode = .Ignored},
{"repo/.env", "repo/secrets.env"}, {"repo/.env", "repo/secrets.env"},
) )
} }
@(test)
test_no_hidden_skips_dotfiles :: proc(t: ^testing.T) {
env := create_test_env()
defer destroy_test_env(&env)
create_git_repo(env, "repo")
create_file(env, "repo/.gitignore", "*.env\n")
create_file(env, "repo/.env")
create_file(env, "repo/secrets.env")
create_file(env, "repo/.hidden.env")
assert_output(t, env, nil,
{include_hidden = false, ignore_mode = .Ignored},
{"repo/secrets.env"},
)
}
// ============================================================================ // ============================================================================
// Special file type tests // Special file type tests (SOCK, FIFO, CHR, BLK parity with fd)
// ============================================================================ // ============================================================================
@(test) @(test)
@@ -305,7 +401,7 @@ test_fifo_emitted :: proc(t: ^testing.T) {
defer destroy_test_env(&env) defer destroy_test_env(&env)
create_git_repo(env, "repo") create_git_repo(env, "repo")
create_file(env, "repo/.gitignore", "*.env\n*.fifo\n") create_file(env, "repo/.gitignore", "*.env\n")
fifo_path := join_path(env.temp_dir, "repo/test.fifo") fifo_path := join_path(env.temp_dir, "repo/test.fifo")
defer delete(fifo_path) defer delete(fifo_path)
@@ -314,7 +410,65 @@ test_fifo_emitted :: proc(t: ^testing.T) {
linux.mknod(cpath, linux.S_IFIFO | linux.Mode{.IRUSR, .IWUSR}, 0) linux.mknod(cpath, linux.S_IFIFO | linux.Mode{.IRUSR, .IWUSR}, 0)
assert_output(t, env, nil, assert_output(t, env, nil,
{pattern = "\\.fifo$"}, {include_hidden = true, ignore_mode = .All},
{"repo/test.fifo"}, {"repo/", "repo/.gitignore", "repo/test.fifo"},
)
}
// ============================================================================
// in_repo propagation tests
// ============================================================================
@(test)
test_repo_without_root_gitignore :: proc(t: ^testing.T) {
env := create_test_env()
defer destroy_test_env(&env)
create_git_repo(env, "repo")
create_dir(env, "repo/sub")
create_file(env, "repo/sub/.gitignore", "*.tmp\n")
create_file(env, "repo/sub/file.tmp")
create_file(env, "repo/sub/file.txt")
assert_output(t, env, nil,
{include_hidden = true, ignore_mode = .Respected},
{"repo/", "repo/sub/", "repo/sub/.gitignore", "repo/sub/file.txt"},
)
}
// ============================================================================
// .ignore file support tests (fd respects .ignore in addition to .gitignore)
// ============================================================================
@(test)
test_ignore_file_respected :: proc(t: ^testing.T) {
env := create_test_env()
defer destroy_test_env(&env)
create_git_repo(env, "repo")
create_file(env, "repo/.ignore", "*.tmp\n")
create_file(env, "repo/file.tmp")
create_file(env, "repo/file.txt")
assert_output(t, env, nil,
{include_hidden = true, ignore_mode = .Respected},
{"repo/", "repo/.ignore", "repo/file.txt"},
)
}
@(test)
test_ignore_overrides_gitignore :: proc(t: ^testing.T) {
env := create_test_env()
defer destroy_test_env(&env)
create_git_repo(env, "repo")
create_file(env, "repo/.gitignore", "*.log\n")
create_file(env, "repo/.ignore", "important.log\n")
create_file(env, "repo/debug.log")
create_file(env, "repo/important.log")
assert_output(t, env, nil,
{include_hidden = true, ignore_mode = .Respected},
{"repo/", "repo/.gitignore", "repo/.ignore"},
) )
} }

View File

@@ -2,6 +2,107 @@ package findr
import "core:testing" import "core:testing"
@(test)
test_glob_simple :: proc(t: ^testing.T) {
testing.expect(t, glob_match("foo", "foo", false))
testing.expect(t, glob_match("foo", "bar/foo", false))
testing.expect(t, !glob_match("foo", "foobar", false))
testing.expect(t, !glob_match("foo", "foo/bar", false))
}
@(test)
test_glob_anchored :: proc(t: ^testing.T) {
testing.expect(t, glob_match("foo", "foo", true))
testing.expect(t, !glob_match("foo", "bar/foo", true))
testing.expect(t, !glob_match("foo", "foobar", true))
}
@(test)
test_glob_star :: proc(t: ^testing.T) {
testing.expect(t, glob_match("*.log", "test.log", false))
testing.expect(t, glob_match("*.log", ".log", false))
testing.expect(t, !glob_match("*.log", "test.txt", false))
testing.expect(t, !glob_match("*.log", "dir/test", false))
}
@(test)
test_glob_question :: proc(t: ^testing.T) {
testing.expect(t, glob_match("?.log", "a.log", false))
testing.expect(t, !glob_match("?.log", "ab.log", false))
testing.expect(t, !glob_match("?.log", ".log", false))
}
@(test)
test_glob_char_class :: proc(t: ^testing.T) {
testing.expect(t, glob_match("[abc].log", "a.log", false))
testing.expect(t, glob_match("[abc].log", "b.log", false))
testing.expect(t, !glob_match("[abc].log", "d.log", false))
}
@(test)
test_glob_negated_class :: proc(t: ^testing.T) {
testing.expect(t, glob_match("[!abc].log", "d.log", false))
testing.expect(t, !glob_match("[!abc].log", "a.log", false))
}
@(test)
test_glob_dot_literal :: proc(t: ^testing.T) {
testing.expect(t, glob_match(".env", ".env", false))
testing.expect(t, glob_match(".env", "dir/.env", false))
testing.expect(t, !glob_match(".env", "env", false))
testing.expect(t, !glob_match(".env", "x.env", false))
}
@(test)
test_glob_globstar_prefix :: proc(t: ^testing.T) {
testing.expect(t, glob_match("**/foo", "foo", false))
testing.expect(t, glob_match("**/foo", "a/b/foo", false))
testing.expect(t, !glob_match("**/foo", "foobar", false))
testing.expect(t, !glob_match("**/foo", "a/foobar", false))
}
@(test)
test_glob_globstar_suffix :: proc(t: ^testing.T) {
testing.expect(t, glob_match("abc/**", "abc/x", false))
testing.expect(t, glob_match("abc/**", "abc/x/y", false))
testing.expect(t, !glob_match("abc/**", "abc", false))
testing.expect(t, !glob_match("abc/**", "abcd/x", false))
}
@(test)
test_glob_globstar_middle :: proc(t: ^testing.T) {
testing.expect(t, glob_match("foo/**/bar", "foo/bar", false))
testing.expect(t, glob_match("foo/**/bar", "foo/x/bar", false))
testing.expect(t, !glob_match("foo/**/bar", "foo/barx", false))
testing.expect(t, !glob_match("foo/**/bar", "foo/x/y/baz", false))
}
@(test)
test_glob_backslash_escape :: proc(t: ^testing.T) {
testing.expect(t, glob_match("\\!foo", "!foo", false))
testing.expect(t, !glob_match("\\!foo", "foo", false))
}
@(test)
test_glob_hash_literal :: proc(t: ^testing.T) {
testing.expect(t, glob_match("#foo", "#foo", false))
testing.expect(t, !glob_match("#foo", "foo", false))
}
@(test)
test_glob_hash_pattern :: proc(t: ^testing.T) {
testing.expect(t, glob_match("#*#", "#test#", false))
testing.expect(t, glob_match("#*#", "##", false))
testing.expect(t, !glob_match("#*#", "test", false))
testing.expect(t, !glob_match("#*#", "#test", false))
}
@(test)
test_glob_empty :: proc(t: ^testing.T) {
testing.expect(t, glob_match("", "", false))
testing.expect(t, !glob_match("", "foo", false))
}
@(test) @(test)
test_is_ignored_basic :: proc(t: ^testing.T) { test_is_ignored_basic :: proc(t: ^testing.T) {
gi := parse("*.env\n") gi := parse("*.env\n")

View File

@@ -201,3 +201,10 @@ glob_destroy :: proc(gp: ^GlobPattern) {
delete(gp.classes) delete(gp.classes)
delete(gp.tokens) delete(gp.tokens)
} }
glob_match :: proc(pattern: string, path: string, anchored: bool) -> bool {
gp := glob_compile(pattern, anchored)
result := glob_match_compiled(&gp, path)
glob_destroy(&gp)
return result
}

View File

@@ -1,111 +0,0 @@
package findr
import "core:testing"
glob_match :: proc(pattern: string, path: string, anchored: bool) -> bool {
gp := glob_compile(pattern, anchored)
result := glob_match_compiled(&gp, path)
glob_destroy(&gp)
return result
}
@(test)
test_glob_simple :: proc(t: ^testing.T) {
testing.expect(t, glob_match("foo", "foo", false))
testing.expect(t, glob_match("foo", "bar/foo", false))
testing.expect(t, !glob_match("foo", "foobar", false))
testing.expect(t, !glob_match("foo", "foo/bar", false))
}
@(test)
test_glob_anchored :: proc(t: ^testing.T) {
testing.expect(t, glob_match("foo", "foo", true))
testing.expect(t, !glob_match("foo", "bar/foo", true))
testing.expect(t, !glob_match("foo", "foobar", true))
}
@(test)
test_glob_star :: proc(t: ^testing.T) {
testing.expect(t, glob_match("*.log", "test.log", false))
testing.expect(t, glob_match("*.log", ".log", false))
testing.expect(t, !glob_match("*.log", "test.txt", false))
testing.expect(t, !glob_match("*.log", "dir/test", false))
}
@(test)
test_glob_question :: proc(t: ^testing.T) {
testing.expect(t, glob_match("?.log", "a.log", false))
testing.expect(t, !glob_match("?.log", "ab.log", false))
testing.expect(t, !glob_match("?.log", ".log", false))
}
@(test)
test_glob_char_class :: proc(t: ^testing.T) {
testing.expect(t, glob_match("[abc].log", "a.log", false))
testing.expect(t, glob_match("[abc].log", "b.log", false))
testing.expect(t, !glob_match("[abc].log", "d.log", false))
}
@(test)
test_glob_negated_class :: proc(t: ^testing.T) {
testing.expect(t, glob_match("[!abc].log", "d.log", false))
testing.expect(t, !glob_match("[!abc].log", "a.log", false))
}
@(test)
test_glob_dot_literal :: proc(t: ^testing.T) {
testing.expect(t, glob_match(".env", ".env", false))
testing.expect(t, glob_match(".env", "dir/.env", false))
testing.expect(t, !glob_match(".env", "env", false))
testing.expect(t, !glob_match(".env", "x.env", false))
}
@(test)
test_glob_globstar_prefix :: proc(t: ^testing.T) {
testing.expect(t, glob_match("**/foo", "foo", false))
testing.expect(t, glob_match("**/foo", "a/b/foo", false))
testing.expect(t, !glob_match("**/foo", "foobar", false))
testing.expect(t, !glob_match("**/foo", "a/foobar", false))
}
@(test)
test_glob_globstar_suffix :: proc(t: ^testing.T) {
testing.expect(t, glob_match("abc/**", "abc/x", false))
testing.expect(t, glob_match("abc/**", "abc/x/y", false))
testing.expect(t, !glob_match("abc/**", "abc", false))
testing.expect(t, !glob_match("abc/**", "abcd/x", false))
}
@(test)
test_glob_globstar_middle :: proc(t: ^testing.T) {
testing.expect(t, glob_match("foo/**/bar", "foo/bar", false))
testing.expect(t, glob_match("foo/**/bar", "foo/x/bar", false))
testing.expect(t, !glob_match("foo/**/bar", "foo/barx", false))
testing.expect(t, !glob_match("foo/**/bar", "foo/x/y/baz", false))
}
@(test)
test_glob_backslash_escape :: proc(t: ^testing.T) {
testing.expect(t, glob_match("\\!foo", "!foo", false))
testing.expect(t, !glob_match("\\!foo", "foo", false))
}
@(test)
test_glob_hash_literal :: proc(t: ^testing.T) {
testing.expect(t, glob_match("#foo", "#foo", false))
testing.expect(t, !glob_match("#foo", "foo", false))
}
@(test)
test_glob_hash_pattern :: proc(t: ^testing.T) {
testing.expect(t, glob_match("#*#", "#test#", false))
testing.expect(t, glob_match("#*#", "##", false))
testing.expect(t, !glob_match("#*#", "test", false))
testing.expect(t, !glob_match("#*#", "#test", false))
}
@(test)
test_glob_empty :: proc(t: ^testing.T) {
testing.expect(t, glob_match("", "", false))
testing.expect(t, !glob_match("", "foo", false))
}

64
findr/prof.odin Normal file
View File

@@ -0,0 +1,64 @@
package findr
import "base:runtime"
import "core:prof/spall"
import "core:sync"
SPALL_ENABLED :: #config(SPALL_ENABLED, ODIN_DEBUG)
spall_ctx: spall.Context
@(thread_local) spall_buffer: spall.Buffer
@(thread_local) spall_backing: []u8
@(instrumentation_enter)
spall_enter :: proc "contextless" (
proc_address, call_site_return_address: rawptr,
loc: runtime.Source_Code_Location,
) {
when SPALL_ENABLED {
spall._buffer_begin(&spall_ctx, &spall_buffer, "", "", loc)
}
}
@(instrumentation_exit)
spall_exit :: proc "contextless" (
proc_address, call_site_return_address: rawptr,
loc: runtime.Source_Code_Location,
) {
when SPALL_ENABLED {
spall._buffer_end(&spall_ctx, &spall_buffer)
}
}
prof_init :: proc() {
when SPALL_ENABLED {
spall_ctx = spall.context_create_with_scale("findr.spall", false, 1.0)
spall_backing = make([]u8, spall.BUFFER_DEFAULT_SIZE)
spall_buffer = spall.buffer_create(spall_backing, u32(sync.current_thread_id()))
spall._buffer_name_thread(&spall_ctx, &spall_buffer, "main")
}
}
prof_destroy :: proc() {
when SPALL_ENABLED {
spall.buffer_destroy(&spall_ctx, &spall_buffer)
delete(spall_backing)
spall.context_destroy(&spall_ctx)
}
}
prof_thread_init :: proc(name: string) {
when SPALL_ENABLED {
spall_backing = make([]u8, spall.BUFFER_DEFAULT_SIZE)
spall_buffer = spall.buffer_create(spall_backing, u32(sync.current_thread_id()))
spall._buffer_name_thread(&spall_ctx, &spall_buffer, name)
}
}
prof_thread_destroy :: proc() {
when SPALL_ENABLED {
spall.buffer_destroy(&spall_ctx, &spall_buffer)
delete(spall_backing)
}
}

13
findr/profile.sh Executable file
View File

@@ -0,0 +1,13 @@
#!/usr/bin/env bash
set -euo pipefail
DIR="$(cd "$(dirname "$0")" && pwd)"
echo "Building findr-prof..."
odin build "$DIR" -debug -out:"$DIR/findr-prof"
echo "Running profiler..."
"$DIR/findr-prof" -E .git -E .jj -HI ~/git.verticalaxion.com
echo
echo "Spall trace: $DIR/findr.spall"

View File

@@ -1,128 +0,0 @@
package findr
import "core:strings"
import "core:sync"
import "core:sys/linux"
import "core:thread"
RepoPool :: struct {
queue: [dynamic]string,
queue_mutex: sync.Mutex,
queue_sema: sync.Atomic_Sema,
results: ^[dynamic]string,
results_lock: sync.Mutex,
active: i64,
done: sync.One_Shot_Event,
threads: []^thread.Thread,
}
find_repos :: proc(roots: []string, results: ^[dynamic]string, thread_count: int) {
if len(roots) == 0 do return
pool := new(RepoPool)
pool.queue = make([dynamic]string)
pool.results = results
pool.active = i64(len(roots))
pool.threads = make([]^thread.Thread, thread_count)
for root in roots {
root_clone, _ := strings.clone(root)
append(&pool.queue, root_clone)
sync.atomic_sema_post(&pool.queue_sema)
}
for i in 0 ..< thread_count {
t := thread.create(repo_worker)
t.data = rawptr(pool)
t.init_context = context
thread.start(t)
pool.threads[i] = t
}
sync.one_shot_event_wait(&pool.done)
for _ in 0 ..< thread_count {
sync.atomic_sema_post(&pool.queue_sema)
}
for t in pool.threads {
thread.destroy(t)
}
delete(pool.threads)
for path in pool.queue {
delete(path)
}
delete(pool.queue)
free(pool)
}
repo_worker :: proc(t: ^thread.Thread) {
pool := cast(^RepoPool)t.data
for {
sync.atomic_sema_wait(&pool.queue_sema)
sync.mutex_lock(&pool.queue_mutex)
if len(pool.queue) == 0 {
sync.mutex_unlock(&pool.queue_mutex)
if sync.atomic_load_explicit(&pool.active, .Acquire) == 0 {
sync.one_shot_event_signal(&pool.done)
}
break
}
last := len(pool.queue) - 1
dir_path := pool.queue[last]
ordered_remove(&pool.queue, last)
sync.mutex_unlock(&pool.queue_mutex)
process_repo_dir(pool, dir_path)
delete(dir_path)
old := sync.atomic_sub_explicit(&pool.active, 1, .Release)
if old == 1 {
sync.one_shot_event_signal(&pool.done)
}
}
}
process_repo_dir :: proc(pool: ^RepoPool, dir_path: string) {
cpath := strings.clone_to_cstring(dir_path)
if cpath == nil do return
defer delete(cpath)
fd, open_err := linux.open(cpath, {.DIRECTORY, .CLOEXEC})
if open_err != .NONE do return
defer linux.close(fd)
if has_git_dir(fd) {
cloned, _ := strings.clone(dir_path)
sync.mutex_lock(&pool.results_lock)
append(pool.results, cloned)
sync.mutex_unlock(&pool.results_lock)
}
buf: [32 * 1024]u8
for {
n, errno := linux.getdents(fd, buf[:])
if n <= 0 || errno != .NONE do break
offs := 0
for d in linux.dirent_iterate_buf(buf[:n], &offs) {
name := linux.dirent_name(d)
if name == "." || name == ".." do continue
if name == ".git" do continue
if d.type == .DIR {
child_path := join_path(dir_path, name)
sync.atomic_add_explicit(&pool.active, 1, .Relaxed)
sync.mutex_lock(&pool.queue_mutex)
append(&pool.queue, child_path)
sync.mutex_unlock(&pool.queue_mutex)
sync.atomic_sema_post(&pool.queue_sema)
}
}
}
}

View File

@@ -1,20 +1,29 @@
package findr package findr
import "core:bytes"
import "core:fmt" import "core:fmt"
import "core:os" import "core:os"
import "core:strings" import "core:strings"
import "core:sync" import "core:sync"
import "core:sync/chan"
import "core:sys/linux" import "core:sys/linux"
import "core:text/regex" import "core:text/regex"
import "core:thread" import "core:thread"
OUTPUT_BUF_SIZE :: 64 * 1024 IgnoreMode :: enum {
Respected, // skip gitignored, prune ignored dirs (fd -H default)
All, // ignore .gitignore entirely, descend everywhere (fd -HI)
Ignored, // emit ONLY gitignored files, prune ignored dirs (findr original)
}
WalkOptions :: struct { WalkOptions :: struct {
pattern: string, // regex on basename; "" = match all pattern: string, // regex on basename; "" = match all
excludes: []string, // glob patterns to skip entirely excludes: []string, // glob patterns to skip entirely (fd -E)
include_hidden: bool, // true = include dotfiles (fd -H)
ignore_mode: IgnoreMode,
}
RawEntry :: struct {
name: string,
type: linux.Dirent_Type,
} }
GIContext :: struct { GIContext :: struct {
@@ -28,17 +37,17 @@ WorkItem :: struct {
rel: string, // relative path from repo root ("" = root) rel: string, // relative path from repo root ("" = root)
gi_ctx: ^GIContext, // gitignore chain (nil = outside any repo) gi_ctx: ^GIContext, // gitignore chain (nil = outside any repo)
in_repo: bool, // true if inside a git repo in_repo: bool, // true if inside a git repo
in_ignored: bool, // true if inside a gitignored directory
} }
WalkerPool :: struct { WalkerPool :: struct {
queue: [dynamic]WorkItem, queue: [dynamic]WorkItem,
queue_mutex: sync.Mutex, queue_mutex: sync.Mutex,
queue_sema: sync.Atomic_Sema, queue_sema: sync.Atomic_Sema,
result_chan: chan.Chan([]u8), results: ^[dynamic]string,
results_mutex: sync.Mutex,
active: i64, active: i64,
done: sync.One_Shot_Event, done: sync.One_Shot_Event,
threads: []^thread.Thread, threads: [dynamic]^thread.Thread,
opts: WalkOptions, opts: WalkOptions,
pattern_re: regex.Regular_Expression, pattern_re: regex.Regular_Expression,
has_pattern: bool, has_pattern: bool,
@@ -47,58 +56,14 @@ WalkerPool :: struct {
contexts_lock: sync.Mutex, contexts_lock: sync.Mutex,
} }
Collector_Data :: struct {
ch: chan.Chan([]u8),
results: ^[dynamic]string,
}
collect_worker :: proc(t: ^thread.Thread) {
data := cast(^Collector_Data)t.data
for {
batch := chan.recv(data.ch) or_break
defer delete(batch)
start := 0
for {
remaining: []u8
#no_bounds_check {remaining = batch[start:]}
idx := bytes.index_byte(remaining, '\n')
if idx < 0 do break
i := start + idx
if i > start {
segment: []u8
#no_bounds_check {segment = batch[start:i]}
s, _ := strings.clone(string(segment))
append(data.results, s)
}
start = i + 1
}
}
}
walk :: proc(roots: []string, results: ^[dynamic]string, opts: WalkOptions, thread_count: int) { walk :: proc(roots: []string, results: ^[dynamic]string, opts: WalkOptions, thread_count: int) {
if len(roots) == 0 do return if len(roots) == 0 do return
ch, _ := chan.create(chan.Chan([]u8), max(2 * thread_count, 2), context.allocator)
defer chan.destroy(ch)
data := new(Collector_Data)
data.ch = ch
data.results = results
defer free(data)
collector := thread.create(collect_worker)
collector.data = rawptr(data)
collector.init_context = context
thread.start(collector)
pool := new(WalkerPool) pool := new(WalkerPool)
pool.queue = make([dynamic]WorkItem) pool.queue = make([dynamic]WorkItem)
pool.result_chan = ch pool.results = results
pool.active = i64(len(roots)) pool.active = i64(len(roots))
pool.threads = make([]^thread.Thread, thread_count) pool.threads = make([dynamic]^thread.Thread)
pool.all_contexts = make([dynamic]^GIContext) pool.all_contexts = make([dynamic]^GIContext)
pool.opts = opts pool.opts = opts
pool.exclude_gi = nil pool.exclude_gi = nil
@@ -135,7 +100,7 @@ walk :: proc(roots: []string, results: ^[dynamic]string, opts: WalkOptions, thre
t.data = rawptr(pool) t.data = rawptr(pool)
t.init_context = context t.init_context = context
thread.start(t) thread.start(t)
pool.threads[i] = t append(&pool.threads, t)
} }
sync.one_shot_event_wait(&pool.done) sync.one_shot_event_wait(&pool.done)
@@ -175,47 +140,16 @@ walk :: proc(roots: []string, results: ^[dynamic]string, opts: WalkOptions, thre
} }
free(pool) free(pool)
chan.close(ch)
thread.join(collector)
thread.destroy(collector)
}
flush_buf :: proc(ch: chan.Chan([]u8), local: ^[dynamic]u8) {
if len(local) == 0 do return
batch := local[:]
local^ = make([dynamic]u8, 0, OUTPUT_BUF_SIZE)
chan.send(ch, batch)
}
append_path :: proc(buf: ^[dynamic]u8, parent, name: string, trailing_slash: bool) {
need_sep := len(parent) > 0 && parent[len(parent) - 1] != '/'
size := len(parent) + len(name) + 1
if need_sep do size += 1
if trailing_slash do size += 1
old_len := len(buf)
reserve(buf, old_len + size)
resize(buf, old_len + size)
pos := old_len
pos += copy(buf[pos:], parent)
if need_sep {buf[pos] = '/'; pos += 1}
pos += copy(buf[pos:], name)
if trailing_slash {buf[pos] = '/'; pos += 1}
buf[pos] = '\n'
} }
walk_worker :: proc(t: ^thread.Thread) { walk_worker :: proc(t: ^thread.Thread) {
pool := cast(^WalkerPool)t.data pool := cast(^WalkerPool)t.data
local_buf := make([dynamic]u8, 0, OUTPUT_BUF_SIZE) prof_thread_init("walker")
defer { defer prof_thread_destroy()
if len(local_buf) > 0 {
flush_buf(pool.result_chan, &local_buf) local_results := make([dynamic]string, 0, 256)
} defer delete(local_results)
delete(local_buf)
}
for { for {
sync.atomic_sema_wait(&pool.queue_sema) sync.atomic_sema_wait(&pool.queue_sema)
@@ -233,36 +167,30 @@ walk_worker :: proc(t: ^thread.Thread) {
ordered_remove(&pool.queue, last) ordered_remove(&pool.queue, last)
sync.mutex_unlock(&pool.queue_mutex) sync.mutex_unlock(&pool.queue_mutex)
process_dir(pool, item, &local_buf) process_dir(pool, item, &local_results)
delete(item.path) delete(item.path)
if len(item.rel) > 0 {delete(item.rel)} if len(item.rel) > 0 {delete(item.rel)}
if len(local_buf) >= OUTPUT_BUF_SIZE {
flush_buf(pool.result_chan, &local_buf)
}
old := sync.atomic_sub_explicit(&pool.active, 1, .Release) old := sync.atomic_sub_explicit(&pool.active, 1, .Release)
if old == 1 { if old == 1 {
sync.one_shot_event_signal(&pool.done) sync.one_shot_event_signal(&pool.done)
} }
} }
if len(local_results) > 0 {
sync.mutex_lock(&pool.results_mutex)
for res in local_results {
append(pool.results, res)
}
sync.mutex_unlock(&pool.results_mutex)
}
} }
process_dir :: proc(pool: ^WalkerPool, item: WorkItem, local_buf: ^[dynamic]u8) { process_dir :: proc(pool: ^WalkerPool, item: WorkItem, local_results: ^[dynamic]string) {
dir_path := item.path dir_path := item.path
cpath := strings.clone_to_cstring(dir_path)
if cpath == nil do return
defer delete(cpath)
fd, open_err := linux.open(cpath, {.DIRECTORY, .CLOEXEC})
if open_err != .NONE do return
defer linux.close(fd)
has_git := false has_git := false
if !item.in_ignored { entries := read_dir_entries(dir_path, &has_git)
has_git = has_git_dir(fd) defer free_entries(&entries)
}
gi_ctx := item.gi_ctx gi_ctx := item.gi_ctx
rel := item.rel rel := item.rel
@@ -274,10 +202,7 @@ process_dir :: proc(pool: ^WalkerPool, item: WorkItem, local_buf: ^[dynamic]u8)
child_in_repo := has_git || item.in_repo child_in_repo := has_git || item.in_repo
gi: ^Gitignore = nil gi := load_ignore_patterns(dir_path, child_in_repo)
if !item.in_ignored {
gi = load_ignore_patterns(dir_path, child_in_repo)
}
if gi != nil { if gi != nil {
new_ctx := new(GIContext) new_ctx := new(GIContext)
new_ctx.gi = gi new_ctx.gi = gi
@@ -293,41 +218,44 @@ process_dir :: proc(pool: ^WalkerPool, item: WorkItem, local_buf: ^[dynamic]u8)
gi_ctx = new_ctx gi_ctx = new_ctx
} }
buf: [32 * 1024]u8
rel_buf: [4096]u8 rel_buf: [4096]u8
for { for entry in entries {
n, errno := linux.getdents(fd, buf[:]) if entry.name == ".git" do continue
if n <= 0 || errno != .NONE do break
offs := 0 is_dir := entry.type == .DIR
for d in linux.dirent_iterate_buf(buf[:n], &offs) { is_nondir := entry.type != .DIR
name := linux.dirent_name(d)
if name == "." || name == ".." do continue
if name == ".git" do continue
is_dir := d.type == .DIR if pool.exclude_gi != nil && is_ignored(pool.exclude_gi, entry.name, is_dir) {
is_nondir := d.type != .DIR
if pool.exclude_gi != nil && is_ignored(pool.exclude_gi, name, is_dir) {
continue continue
} }
entry_rel := build_rel(rel_buf[:], rel, name) if !pool.opts.include_hidden && len(entry.name) > 0 && entry.name[0] == '.' {
continue
}
entry_rel := build_rel(rel_buf[:], rel, entry.name)
ignored := false ignored := false
if item.in_ignored { if gi_ctx != nil && pool.opts.ignore_mode != .All {
ignored = true
} else if gi_ctx != nil {
ignored = check_chain(gi_ctx, entry_rel, is_dir) ignored = check_chain(gi_ctx, entry_rel, is_dir)
} }
if is_dir { should_emit: bool
if ignored && matches_pattern(pool, name) { if ignored {
append_path(local_buf, dir_path, name, true) should_emit = pool.opts.ignore_mode == .Ignored
} else {
should_emit = pool.opts.ignore_mode != .Ignored
} }
if is_dir {
if should_emit && matches_pattern(pool, entry.name) {
dir_path_out := join_path_dir(dir_path, entry.name)
append(local_results, dir_path_out)
}
if !ignored {
child_rel, _ := strings.clone(entry_rel) child_rel, _ := strings.clone(entry_rel)
child_path := join_path(dir_path, name) child_path := join_path(dir_path, entry.name)
push_work( push_work(
pool, pool,
WorkItem { WorkItem {
@@ -335,13 +263,13 @@ process_dir :: proc(pool: ^WalkerPool, item: WorkItem, local_buf: ^[dynamic]u8)
rel = child_rel, rel = child_rel,
gi_ctx = gi_ctx, gi_ctx = gi_ctx,
in_repo = child_in_repo, in_repo = child_in_repo,
in_ignored = ignored,
}, },
) )
} else if is_nondir {
if ignored && matches_pattern(pool, name) {
append_path(local_buf, dir_path, name, false)
} }
} else if is_nondir {
if should_emit && matches_pattern(pool, entry.name) {
full_path := join_path(dir_path, entry.name)
append(local_results, full_path)
} }
} }
} }
@@ -399,13 +327,46 @@ push_work :: proc(pool: ^WalkerPool, item: WorkItem) {
sync.atomic_sema_post(&pool.queue_sema) sync.atomic_sema_post(&pool.queue_sema)
} }
has_git_dir :: proc(fd: linux.Fd) -> bool { read_dir_entries :: proc(dir_path: string, has_git: ^bool) -> [dynamic]RawEntry {
git_fd, err := linux.openat(fd, ".git", {.DIRECTORY, .CLOEXEC}) entries := make([dynamic]RawEntry)
if err == .NONE {
linux.close(git_fd) cpath := strings.clone_to_cstring(dir_path)
return true if cpath == nil do return entries
fd, err := linux.open(cpath, {.DIRECTORY, .CLOEXEC})
delete(cpath)
if err != .NONE do return entries
buf: [8192]u8
has_git^ = false
for {
n, errno := linux.getdents(fd, buf[:])
if n <= 0 || errno != .NONE do break
offs := 0
for d in linux.dirent_iterate_buf(buf[:n], &offs) {
name := linux.dirent_name(d)
if name == "." || name == ".." do continue
if name == ".git" && d.type == .DIR {
has_git^ = true
} }
return false
cloned := strings.clone(name)
append(&entries, RawEntry{name = cloned, type = d.type})
}
}
linux.close(fd)
return entries
}
free_entries :: proc(entries: ^[dynamic]RawEntry) {
for &entry in entries {
delete(entry.name)
}
delete(entries^)
} }
load_ignore_patterns :: proc(dir_path: string, in_repo: bool) -> ^Gitignore { load_ignore_patterns :: proc(dir_path: string, in_repo: bool) -> ^Gitignore {
@@ -456,3 +417,18 @@ join_path :: proc(parent, child: string) -> string {
return string(buf) return string(buf)
} }
join_path_dir :: proc(parent, child: string) -> string {
need_sep := len(parent) == 0 || parent[len(parent) - 1] != '/'
total := len(parent) + len(child) + 1 // +1 for trailing '/'
if need_sep do total += 1
buf := make([]u8, total, context.allocator)
pos := copy(buf, parent)
if need_sep {
buf[pos] = '/'
pos += 1
}
pos += copy(buf[pos:], child)
buf[pos] = '/'
return string(buf)
}

View File

@@ -95,6 +95,7 @@
devShells.default = pkgs.mkShell { devShells.default = pkgs.mkShell {
buildInputs = with pkgs; [ buildInputs = with pkgs; [
fd
nushell nushell
libsodium libsodium

View File

@@ -1,31 +1,14 @@
package main package main
import "core:bufio"
import "core:fmt" import "core:fmt"
import "core:mem"
import "core:os" import "core:os"
main :: proc() { main :: proc() {
when ODIN_DEBUG {
heap_track: mem.Tracking_Allocator
mem.tracking_allocator_init(&heap_track, context.allocator)
defer mem.tracking_allocator_destroy(&heap_track)
defer if len(heap_track.allocation_map) > 0 {
for _, leak in heap_track.allocation_map {
fmt.eprintf("LEAK: %v leaked %m\n", leak.location, leak.size)
}
}
context.allocator = mem.tracking_allocator(&heap_track)
temp_track: mem.Tracking_Allocator
mem.tracking_allocator_init(&temp_track, context.temp_allocator)
defer mem.tracking_allocator_destroy(&temp_track)
context.temp_allocator = mem.tracking_allocator(&temp_track)
}
defer free_all(context.temp_allocator) defer free_all(context.temp_allocator)
cmd, ok := parse_args(os.args, os.to_writer(os.stdout), os.to_writer(os.stderr)) cmd, ok := parse_args(os.args, os.to_writer(os.stdout), os.to_writer(os.stderr))
defer delete_command(&cmd) // delete flushes automatically defer bufio.writer_flush(cmd.out_buf)
if !ok { if !ok {
return return
} }
@@ -35,6 +18,8 @@ main :: proc() {
cmd_init(&cmd) cmd_init(&cmd)
case "version": case "version":
cmd_version(&cmd) cmd_version(&cmd)
case "deps":
cmd_deps(&cmd)
case "list": case "list":
cmd_list(&cmd) cmd_list(&cmd)
case "backup", "add": case "backup", "add":

127
scan.odin
View File

@@ -1,21 +1,137 @@
package main package main
import "core:fmt"
import "core:os" import "core:os"
import "core:strings"
import "core:sync"
import "core:terminal"
import "findr" fd_counter: sync.Atomic_Mutex
fd_seq: int
// Caller is responsible for freeing paths // Caller is responsible for freeing paths
scan_path :: proc(search_path: string, cfg: Config) -> (paths: [dynamic]string, ok: bool) { scan_path :: proc(search_path: string, cfg: Config) -> (paths: [dynamic]string, ok: bool) {
opts := findr.WalkOptions { if terminal.is_terminal(os.stdout) {
pattern = cfg.ScanConfig.Matcher, fmt.printf("Searching for all files in \"%s\"...\n", search_path)
excludes = cfg.ScanConfig.Exclude[:],
} }
findr.walk({search_path}, &paths, opts, os.get_processor_core_count()) all_files, all_ok := run_fd(build_fd_args(search_path, cfg, true))
if !all_ok {
return
}
if terminal.is_terminal(os.stdout) {
fmt.printf("Search for unignored fies in \"%s\"...\n", search_path)
}
unignored_files, unignored_ok := run_fd(build_fd_args(search_path, cfg, false))
if !unignored_ok {
return
}
unignored_set := make(map[string]bool, len(unignored_files), context.temp_allocator)
for file in unignored_files {
unignored_set[file] = true
}
for file in all_files {
if !(file in unignored_set) {
append(&paths, file)
}
}
ok = true ok = true
return return
} }
@(private = "file")
build_fd_args :: proc(search_path: string, cfg: Config, include_ignored: bool) -> []string {
args_len := 3 + 2 * len(cfg.ScanConfig.Exclude) + 2
args := make([dynamic]string, 0, args_len, context.temp_allocator)
append(&args, "fd")
append(&args, "-a")
append(&args, cfg.ScanConfig.Matcher)
for exclude in cfg.ScanConfig.Exclude {
append(&args, "-E")
append(&args, exclude)
}
if include_ignored {
append(&args, "-HI")
} else {
append(&args, "-H")
}
append(&args, search_path)
return args[:]
}
run_fd :: proc(args: []string) -> (lines: []string, ok: bool) {
tmp_path := next_fd_tmp_path()
tmp_file, tmp_err := os.open(tmp_path, os.O_CREATE | os.O_WRONLY | os.O_TRUNC)
if tmp_err != nil {
// TODO: Log a message here
return
}
desc := os.Process_Desc {
command = args,
stdout = tmp_file,
stderr = nil,
}
p, start_err := os.process_start(desc)
os.close(tmp_file)
if start_err != nil {
os.remove(tmp_path)
return
}
state, wait_err := os.process_wait(p)
if wait_err != nil || state.exit_code != 0 {
os.remove(tmp_path)
return
}
data, read_err := os.read_entire_file_from_path(tmp_path, context.temp_allocator)
os.remove(tmp_path)
if read_err != nil {
return
}
output := string(data)
output = strings.trim_space(output)
if len(output) == 0 {
ok = true
return
}
raw_lines := strings.split(output, "\n", context.temp_allocator)
result := make([dynamic]string, 0, len(raw_lines), context.temp_allocator)
for line in raw_lines {
trimmed := strings.trim_space(line)
if len(trimmed) > 0 {
append(&result, trimmed)
}
}
return result[:], true
}
@(private = "file")
next_fd_tmp_path :: proc() -> string {
sync.atomic_mutex_lock(&fd_counter)
n := fd_seq
fd_seq += 1
sync.atomic_mutex_unlock(&fd_counter)
return fmt.tprintf("/tmp/envr-fd-%d-%d", os.get_pid(), n)
}
cant_scan :: proc(feats: AvailableFeatures) -> bool {
return Feature.Fd not_in feats
}
find_unbacked :: proc(local_files: []string, db_files: []EnvFile) -> []string { find_unbacked :: proc(local_files: []string, db_files: []EnvFile) -> []string {
// Lives until the end of the function
backed_set := make(map[string]bool, len(db_files), context.temp_allocator) backed_set := make(map[string]bool, len(db_files), context.temp_allocator)
for file in db_files { for file in db_files {
backed_set[file.Path] = true backed_set[file.Path] = true
@@ -29,3 +145,4 @@ find_unbacked :: proc(local_files: []string, db_files: []EnvFile) -> []string {
} }
return unbacked[:] return unbacked[:]
} }

View File

@@ -3,10 +3,14 @@ package main
import "core:fmt" import "core:fmt"
import "core:os" import "core:os"
import "core:path/filepath" import "core:path/filepath"
import "core:strings"
import "core:testing" import "core:testing"
@(test) @(test)
test_scan_path_finds_gitignored_env_files :: proc(t: ^testing.T) { test_scan_path_finds_gitignored_env_files :: proc(t: ^testing.T) {
feats := check_features()
testing.expect(t, cant_scan(feats) == false)
base := fmt.tprintf("/tmp/envr-scan-test-%d", os.get_pid()) base := fmt.tprintf("/tmp/envr-scan-test-%d", os.get_pid())
os.mkdir_all(base) os.mkdir_all(base)
defer os.remove_all(base) defer os.remove_all(base)
@@ -38,12 +42,7 @@ test_scan_path_finds_gitignored_env_files :: proc(t: ^testing.T) {
} }
results, ok := scan_path(base, cfg) results, ok := scan_path(base, cfg)
defer { defer delete(results)
for path in results {
delete(path)
}
delete(results)
}
testing.expect(t, ok, "scan_path should succeed") testing.expect(t, ok, "scan_path should succeed")
found_env := false found_env := false
@@ -70,6 +69,9 @@ test_scan_path_finds_gitignored_env_files :: proc(t: ^testing.T) {
@(test) @(test)
test_scan_path_empty_dir :: proc(t: ^testing.T) { test_scan_path_empty_dir :: proc(t: ^testing.T) {
feats := check_features()
testing.expect(t, cant_scan(feats) == false)
base := fmt.tprintf("/tmp/envr-scan-empty-%d", os.get_pid()) base := fmt.tprintf("/tmp/envr-scan-empty-%d", os.get_pid())
os.mkdir_all(base) os.mkdir_all(base)
defer os.remove_all(base) defer os.remove_all(base)
@@ -83,3 +85,12 @@ test_scan_path_empty_dir :: proc(t: ^testing.T) {
testing.expect(t, ok, "scan_path should succeed") testing.expect(t, ok, "scan_path should succeed")
testing.expect(t, len(results) == 0, fmt.tprintf("expected 0 results, got %d", len(results))) testing.expect(t, len(results) == 0, fmt.tprintf("expected 0 results, got %d", len(results)))
} }
@(test)
test_scan_meets_expectations :: proc(t: ^testing.T) {
testing.expect(t, cant_scan({}), "no features should mean can't scan")
testing.expect(t, cant_scan({.Git}), "Git alone should mean can't scan")
testing.expect(t, !cant_scan({.Fd}), "having Fd should mean can scan")
testing.expect(t, !cant_scan({.Fd, .Git}), "both Fd and Git should mean can scan")
}

View File

@@ -12,6 +12,24 @@ Ed25519Keypair :: struct {
Private: [32]u8, Private: [32]u8,
} }
read_wire_string :: proc(data: []u8, offset: ^int) -> (s: string, ok: bool) {
if offset^ + 4 > len(data) {
return
}
length := u32(data[offset^]) << 24 | u32(data[offset^ + 1]) << 16 |
u32(data[offset^ + 2]) << 8 | u32(data[offset^ + 3])
offset^ += 4
if offset^ + int(length) > len(data) {
return
}
s = string(data[offset^ : offset^ + int(length)])
offset^ += int(length)
ok = true
return
}
parse_ssh_public_key :: proc(pub_path: string) -> (pub: [32]u8, ok: bool) { parse_ssh_public_key :: proc(pub_path: string) -> (pub: [32]u8, ok: bool) {
data, err := os.read_entire_file_from_path(pub_path, context.temp_allocator) data, err := os.read_entire_file_from_path(pub_path, context.temp_allocator)
if err != nil { if err != nil {
@@ -235,21 +253,3 @@ is_encrypted_key :: proc(priv_path: string) -> bool {
return ciphername != "none" return ciphername != "none"
} }
read_wire_string :: proc(data: []u8, offset: ^int) -> (s: string, ok: bool) {
if offset^ + 4 > len(data) {
return
}
length := u32(data[offset^]) << 24 | u32(data[offset^ + 1]) << 16 |
u32(data[offset^ + 2]) << 8 | u32(data[offset^ + 3])
offset^ += 4
if offset^ + int(length) > len(data) {
return
}
s = string(data[offset^ : offset^ + int(length)])
offset^ += int(length)
ok = true
return
}

View File

@@ -1 +1 @@
0.4.0 0.3.0