6 Commits

5 changed files with 119 additions and 319 deletions

View File

@@ -1,268 +0,0 @@
# Table Rendering Memory Optimization Plan
## Executive Summary
This plan outlines improvements to eliminate excessive memory allocations and copies in the Odin table rendering system. The current implementation makes 10+ allocations per row, while the Zig equivalent makes zero allocations for rendering. This optimization will reduce memory usage, improve performance, and align with the project's efficiency goals.
## Current State Analysis
### Zig Version (Reference Implementation)
- **Allocations**: 1 (data only)
- **Data copies**: 0
- **String allocation**: 0
- **Column widths**: Stack array
- **Output**: Direct to writer
### Odin Version (Current Implementation)
- **Allocations**: 10+ per row
- **Data copies**: Multiple per row
- **String allocation**: 2+ per row (concatenate + slice)
- **Column widths**: Heap allocated
- **Output**: Builder → stdout
### Current Issues Identified
1. **Table Infrastructure** (`table.odin`)
- Uses `strings.Builder` which allocates per-line memory
- Heap-allocated `[dynamic]int` for column widths
- Multiple `strings.concatenate()` calls creating new strings
2. **Command Implementations**
- `cmd_list`: Creates intermediate `[]string` slices per row, allocates new strings via `strings.concatenate()`
- `cmd_sync`: Creates `SyncEntry` structs with cloned strings, allocates dynamic arrays
- `cmd_deps`: Allocates dynamic rows array unnecessarily
3. **Memory Pattern**
- Each command allocates `[][]string` for table data
- Manual struct-to-row transformation creates copies
- Duplicate code across all table-using commands
## Proposed Solutions
### Phase 1: Core Table Infrastructure Overhaul
#### 1.1 Direct Writer-Based Rendering
**Current:**
```odin
b: strings.Builder
strings.builder_init(&b)
// ... build table in builder
fmt.println(strings.to_string(b))
```
**Proposed:**
```odin
render_table :: proc(writer: io.Writer, headers: []string, rows: [][]string)
```
- Replace `strings.Builder` with `io.Writer` output
- Eliminate intermediate string allocations
- Write table components directly to output stream
#### 1.2 Stack-Based Column Widths
**Current:**
```odin
col_widths := make([dynamic]int, 0, len(headers))
```
**Proposed:**
- Use fixed stack arrays for reasonable column counts
- Implement small buffer optimization (SBO) for variable column counts
- Only allocate for tables exceeding threshold (e.g., 16 columns)
#### 1.3 Zero-Copy String Handling
**Current:**
```odin
dir_str := strings.concatenate({row.Dir, "/"}, context.temp_allocator)
```
**Proposed:**
- Replace `strings.concatenate()` with string slicing
- Work directly with `EnvFile.Path` and `EnvFile.Dir` fields
- Use `filepath.base()` and `filepath.dir()` without allocation where possible
### Phase 2: Generic Table Interface
#### 2.1 Field-Based Table Renderer
```odin
Table_Field :: struct {
name: string,
value: string, // String view, no allocation
alignment: Alignment,
}
Table_Config :: struct {
writer: io.Writer,
fields: []Table_Field,
col_widths: []int,
}
render_row :: proc(cfg: Table_Config, row_data: any)
```
- Accept struct fields directly without intermediate arrays
- Support field selection (show only specific fields)
- Alignment options (left/center/right)
#### 2.2 Field Extraction Procs
- Generate field extraction helpers for each struct type
- Avoid string allocation by returning string views
- Cache computed values (like formatted status strings)
#### 2.3 Streaming Table Processing
- Process rows one at a time without collecting all rows
- Reduce peak memory usage from O(N × strings) to O(table_structure)
- Enable early termination if needed
### Phase 3: Command-Specific Optimizations
#### 3.1 Eliminate Intermediate Structs
**Current (cmd_sync):**
```odin
for &file in files {
// ... processing
path_str, _ := strings.clone(file.Path)
status_str, _ := strings.clone(status)
append(&results, SyncEntry{Path = path_str, Status = status_str})
}
```
**Proposed:**
```odin
for &file in files {
result, err_msg := db_sync(&db, &file)
// Direct rendering with zero-copy
render_sync_row(writer, file, result, err_msg)
}
```
- `cmd_sync`: Work directly with `EnvFile` + `SyncFlagEnum`
- `cmd_list`: Use `EnvFile` fields directly, no `ListEntry`
- Generate table content on-the-fly
#### 3.2 In-Place Status Computation
```odin
get_sync_status :: proc(result: SyncFlag, err_msg: string) -> string {
switch {
case .Error in result: return if len(err_msg) > 0 then err_msg else "error"
case .BackedUp in result: return "Backed Up"
case .Restored in result: return "Restored"
case .DirUpdated in result: return "Moved"
case: return "OK"
}
}
```
- Compute status strings without allocation (use static lookup)
- Cache formatted status values if needed
- Reduce allocation count from N to 0 or 1
#### 3.3 Batch Processing
- Reduce allocation count by pooling small allocations
- Use `context.temp_allocator` more effectively
- Pre-allocate buffers for expected sizes
### Phase 4: JSON Output Separation
#### 4.1 Unified JSON Rendering
```odin
render_json_rows :: proc(writer: io.Writer, rows: any, field_names: []string)
```
- Create centralized JSON rendering helper
- Work with same structs as table rendering
- Use reflection or explicit field marshaling
#### 4.2 Format-Agnostic Interface
- Commands generate data → renderers handle format
- Table renderer focuses only on ASCII/Unicode output
- Keep terminal detection in command layer
## Expected Improvements
| Metric | Current | Target | Improvement |
|--------|---------|--------|-------------|
| **Allocations** | 10+ per row | 0-1 per table | 10x+ reduction |
| **Memory copies** | 2-3 per row | 0 | 100% reduction |
| **Peak memory** | O(N × strings) | O(table_structure) | Constant factor |
| **Throughput** | Baseline | 2-3x faster | Performance boost |
## Implementation Strategy
### High-Priority Changes
1. Replace `strings.Builder` with direct `io.Writer` output
2. Convert column widths to stack-based allocation
3. Eliminate intermediate struct allocations in commands
### Medium-Priority Changes
1. Create generic field-based table interface
2. Implement streaming table processing
3. Centralize JSON rendering logic
### Low-Priority Changes
1. Add alignment options beyond left-aligned
2. Implement comprehensive field introspection
3. Add advanced table formatting features
## Tradeoff Questions
Before implementation begins, we need to resolve these architectural questions:
### 1. Generality vs. Performance
**Question:** Should we create a fully generic table renderer (similar to Zig's `Table(T)`) or focus on optimizing the current 3 use cases first?
**Options:**
- **Generic approach**: Higher development cost, future-proof, may have some overhead
- **Specific optimization**: Faster implementation, maximum performance for current use cases, less flexible
**Recommendation:** Start with specific optimizations for current use cases, then generalize patterns that emerge.
### 2. Alignment Support
**Question:** Does the project need left/center/right alignment support, or is left-alignment sufficient?
**Context:** Zig supports alignment options, but current Odin implementation only left-aligns. Most CLI tables work fine with left alignment.
**Recommendation:** Start with left-alignment only, add alignment if specific use cases demand it.
### 3. API Compatibility
**Question:** Should we maintain the current `render_table()` API signature, or are breaking changes acceptable?
**Current API:**
```odin
render_table :: proc(headers: []string, rows: [][]string)
```
**Options:**
- **Maintain API**: Slower to implement, backward compatible, may need adapter layers
- **Break API**: Faster implementation, cleaner code, requires updates to all callers
**Recommendation:** Breaking changes are acceptable since this is an optimization-focused effort and callers are limited to 3 commands.
### 4. Odin Capabilities
**Question:** What runtime reflection or field introspection capabilities does Odin provide?
**Context:** Zig uses `@typeInfo()` and comptime field iteration. We need to understand Odin's equivalent capabilities to design the optimal solution.
**Recommendation:** Investigate Odin's runtime type information capabilities before finalizing the generic table interface design.
### 5. Testing Strategy
**Question:** Should we add comprehensive tests for new table rendering before optimizing commands, or optimize incrementally with tests added afterwards?
**Options:**
- **Test-first**: More robust, catches regressions early, slower initial development
- **Optimize-first**: Faster development, may miss edge cases, requires retroactive testing
**Recommendation:** Hybrid approach - add basic tests for core infrastructure, then optimize incrementally with additional tests for each command.
## Next Steps
1. **Research Phase**: Investigate Odin's type system and reflection capabilities
2. **Prototype Phase**: Create minimal working prototype of zero-allocation table renderer
3. **Refactor Phase**: Incrementally update commands to use new infrastructure
4. **Test Phase**: Add comprehensive tests and verify memory improvements
5. **Benchmark Phase**: Measure performance improvements and memory usage
## Success Criteria
- [ ] Zero allocations for table rendering (excluding initial data)
- [ ] Zero string copies in the happy path
- [ ] All 3 commands (`list`, `sync`, `deps`) use new infrastructure
- [ ] Performance improvement of 2x or more
- [ ] Memory usage reduction of 50% or more
- [ ] No regression in table formatting quality
- [ ] Backward compatibility with JSON output format

View File

@@ -63,42 +63,3 @@ Note: These todos can wait until all the subcommands have been ported.
28. 2 scan tests silently skip Low When fd isn't installed, tests pass without actually testing anything. These should use #assert to be sure that fd is in path. 28. 2 scan tests silently skip Low When fd isn't installed, tests pass without actually testing anything. These should use #assert to be sure that fd is in path.
38. Try to do all encryption / decryption in memory - only read / write encrypted data to disk. 38. Try to do all encryption / decryption in memory - only read / write encrypted data to disk.
## Double-check AI output
- [ ] cli.odin
- [ ] config.odin
- [ ] crypto.odin
- [ ] db.odin
- [ ] features.odin
- [ ] main.odin
- [ ] prompt.odin
- [ ] scan.odin
- [ ] sodium.odin
- [ ] ssh.odin
- [ ] table.odin
- [ ] cmd_backup.odin
- [ ] cmd_check.odin
- [ ] cmd_deps.odin
- [ ] cmd_edit_config.odin
- [ ] cmd_init.odin
- [ ] cmd_list.odin
- [ ] cmd_nushell_completion.odin
- [ ] cmd_remove.odin
- [ ] cmd_restore.odin
- [ ] cmd_scan.odin
- [ ] cmd_sync.odin
- [ ] cmd_version.odin
- [ ] sqlite/sqlite.odin
- [ ] cli_test.odin
- [ ] cmd_check_test.odin
- [ ] cmd_list_test.odin
- [ ] cmd_nushell_completion_test.odin
- [ ] config_test.odin
- [ ] crypto_test.odin
- [ ] db_integration_test.odin
- [ ] db_test.odin
- [ ] features_test.odin
- [ ] scan_test.odin
- [ ] ssh_test.odin
- [ ] table_test.odin

View File

@@ -1,10 +1,5 @@
package main package main
import "core:fmt"
import "core:io"
import "core:os"
import "core:terminal"
cmd_deps :: proc(cmd: ^Command) { cmd_deps :: proc(cmd: ^Command) {
feats := check_features() feats := check_features()
@@ -23,12 +18,6 @@ cmd_deps :: proc(cmd: ^Command) {
append(&rows, []string{"fd", "\u2717 Missing"}) append(&rows, []string{"fd", "\u2717 Missing"})
} }
if terminal.is_terminal(os.stdout) {
render_table(headers, rows[:]) render_table(headers, rows[:])
} else {
w := io.to_writer(os.to_writer(os.stdout))
render_json_rows(w, headers, rows[:])
io.write_string(w, "\n")
}
} }

109
quash Normal file
View File

@@ -0,0 +1,109 @@
@ ukspssxz spencer.brower@proton.me 2026-06-12 16:45:22 default@ 548fe7ec
(no description set)
suwmwvkl spencer.brower@proton.me 2026-06-12 16:40:25 odin a1e93345
│ ci: Updated github action.
tqpkpmus spencer.brower@proton.me 2026-06-12 16:35:39 eed36089
│ feat: Removed go code.
yzzzmznw spencer.brower@proton.me 2026-06-12 16:35:34 75b77845
│ build: Converted Makefile and flake package.
kvtmxpyn spencer.brower@proton.me 2026-06-12 15:54:44 4ec2b22b
│ refactor: removed `is_tty`.
pouwppuo spencer.brower@proton.me 2026-06-12 15:48:12 0276db76
│ refactor: Switched from age to libsodium.
txoxnuzl spencer.brower@proton.me 2026-06-12 15:36:10 a0e2c995
│ docs: Updated TODOs.
zvrkmqpk spencer.brower@proton.me 2026-06-12 15:01:50 d0dc93ab
│ feat(odin): Migrated nushell-completion command to go.
zpmvtmzx spencer.brower@proton.me 2026-06-12 14:50:42 91ada61c
│ feat: Added tests.
vsqmlvlq spencer.brower@proton.me 2026-06-12 14:17:56 9b395677
│ fix: Fixed the rest of the (tested) leaks.
rwzttsll spencer.brower@proton.me 2026-06-12 13:37:09 43dd8aca
│ perf: Improved writer performance.
rovqumvz spencer.brower@proton.me 2026-06-12 13:25:50 db1b863e
│ fix: fixing leaks.
quqsmwmx spencer.brower@proton.me 2026-06-12 10:45:43 e9660501
│ fix: Added proper help text to all commands.
uupootzn spencer.brower@proton.me 2026-06-12 10:28:41 7629dd2c
│ fix: Got rid of go fallback code.
svkzoqxq spencer.brower@proton.me 2026-06-12 10:22:21 7c7ddf46
│ fix: Fixed memory leaks in `find_binary`.
yzvwlzvq spencer.brower@proton.me 2026-06-12 10:22:21 a1e945a6
│ feat(odin): Ported init command.
yklwuqrm spencer.brower@proton.me 2026-06-12 09:12:55 0a332adf
│ feat(odin): Ported scan command.
unktymmr spencer.brower@proton.me 2026-06-12 08:27:14 4e1e3590
│ feat(odin): port check command to odin.
oyllntvp spencer.brower@proton.me 2026-06-12 08:02:08 82bec68b
│ fix: Fixing AI oopsies.
lowokuok spencer.brower@proton.me 2026-06-11 21:26:59 2cb6067a
│ feat(odin): ported edit-config command to odin.
vlssoopk spencer.brower@proton.me 2026-06-11 21:25:11 3668df57
│ feat(odin): ported restore command to odin.
tunwtypr spencer.brower@proton.me 2026-06-11 21:21:59 d2127e47
│ feat(odin): Ported remove command.
nrnpskps spencer.brower@proton.me 2026-06-11 21:17:52 cb7db967
│ feat(odin): Added long text and --help flags.
swwzkunx spencer.brower@proton.me 2026-06-11 21:14:11 c92155a1
│ feat(odin): ported backup command.
tsnurnzr spencer.brower@proton.me 2026-06-11 21:05:39 b1d24161
│ feat(odin): ported list command.
vwolkxsl spencer.brower@proton.me 2026-06-11 21:05:33 40f0b3c3
│ feat(odin): ported deps command, added utilities (features, tty, table).
rqrrlqlk spencer.brower@proton.me 2026-06-11 20:34:53 d84e43d0
│ odin: scaffold project with CLI parser, version command, Go fallback
znnskorn spencer.brower@proton.me 2026-06-11 20:08:27 28f96df4
│ feat: Started odin setup.
│ ○ rykmnnwl spencer.brower@proton.me 2026-06-11 20:00:08 zig 42c01a08
│ │ feat: init command.
│ ○ ztntvnnw spencer.brower@proton.me 2026-06-09 11:01:15 d3eb4e84
│ │ fix: Fixed issue with buffer size.
│ ○ pqzlpytk spencer.brower@proton.me 2026-06-09 09:50:38 6acd1f9d
│ │ refactor: Moved deps into `root.zig`.
│ ○ slkwsoqy spencer.brower@proton.me 2026-06-09 09:41:13 681931fb
│ │ feat: Added table viewer.
│ ○ qkmlntsm spencer.brower@proton.me 2026-05-27 19:30:19 acbda090
│ │ feat: list cmd.
│ ○ vxnsyxqp spencer.brower@proton.me 2026-05-27 18:27:21 fc8474d7
│ │ feat: Restore db from file.
│ ○ uoowvkxx spencer.brower@proton.me 2026-05-03 12:45:43 8f2c2419
│ │ feat(config): Added data path.
│ ○ qrkuztko spencer.brower@proton.me 2026-05-01 10:30:12 3e6c1752
│ │ feat: accept config in Db
│ ○ vrxoyzlo spencer.brower@proton.me 2026-04-30 22:37:31 fd0f8bba
│ │ feat(age): accept multiple recipients.
│ ○ rquvonut spencer.brower@proton.me 2026-04-30 21:03:38 65571393
│ │ feat: Implemented basic db operation.
│ ○ nwzoqvoq spencer.brower@proton.me 2026-04-29 16:35:38 e5286527
│ │ feat: Created own age wrapper.
│ ○ rltyxtqr spencer.brower@proton.me 2026-04-28 17:49:04 02ce5e46
│ │ feat: Added age-ffi.
│ ○ krzuylpu spencer.brower@proton.me 2026-04-26 17:29:37 a13264c8
│ │ feat: zig-sqlite.
│ ○ nqlotzkk spencer.brower@proton.me 2026-04-24 11:19:31 799d95a4
│ │ feat: added Config parsing.
│ ○ npvzptmw spencer.brower@proton.me 2026-04-23 16:53:47 217bb413
│ │ feat(comma): Added help method.
│ ○ rrlywnkm spencer.brower@proton.me 2026-04-21 19:42:02 a547409e
│ │ docs: Added AI Disclaimer to README.md.
│ ○ plqqwlws spencer.brower@proton.me 2026-04-21 19:34:09 53cf22bc
│ │ feat: Added help output for commands.
│ ○ znpvknpm spencer.brower@proton.me 2026-04-21 18:13:35 ae445459
│ │ feat(comma): Added enum value for unknown commands.
│ ○ zqpvlvms spencer.brower@proton.me 2026-04-21 18:02:58 bd2a5455
│ │ feat: Migrated `deps` command.
│ ○ wqslwyqo spencer.brower@proton.me 2026-04-20 17:08:26 8a503ced
│ │ refactor: Broke comma into a separate package.
│ ○ trqurnkq spencer.brower@proton.me 2026-04-20 16:14:43 33b0063c
│ │ feat: Added command structure.
│ │ ○ spllvvwm spencer.brower@proton.me 2026-04-20 10:15:48 envr-zig@ ac94b33e
│ ├─╯ (empty) (no description set)
│ ○ olwurpsw spencer.brower@proton.me 2026-04-18 16:28:30 43b03e0a
│ │ wip: feat: Migrated version command to zig.
│ ○ mnqunpro spencer.brower@proton.me 2026-04-17 16:41:45 ce135e9c
│ │ feat: Created zig wrapper.
│ ○ unkrrvon spencer.brower@proton.me 2026-04-17 15:49:00 6a611150
├─╯ feat: Added zig config.
◆ psmotwus 6729162+sbrow@users.noreply.github.com 2026-01-12 15:42:05 go main v0.2.1 c6d03088
│ chore(main): release 0.2.1
~

View File

@@ -3,9 +3,18 @@ package main
import "core:encoding/json" import "core:encoding/json"
import "core:fmt" import "core:fmt"
import "core:io" import "core:io"
import "core:os"
import "core:strings" import "core:strings"
import "core:terminal"
render_table :: proc(headers: []string, rows: [][]string) { render_table :: proc(headers: []string, rows: [][]string) {
if !terminal.is_terminal(os.stdout) {
w := io.to_writer(os.to_writer(os.stdout))
render_json_rows(w, headers, rows)
io.write_string(w, "\n")
return
}
col_widths := make([dynamic]int, 0, len(headers)) col_widths := make([dynamic]int, 0, len(headers))
for i in 0 ..< len(headers) { for i in 0 ..< len(headers) {
append(&col_widths, strings.rune_count(headers[i])) append(&col_widths, strings.rune_count(headers[i]))