diff --git a/AGENTS.md b/AGENTS.md index 6e17f71..b852169 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -32,6 +32,7 @@ thor/ ├── frontmatter.odin # JSON frontmatter parser (supports nested og + lastmod) ├── defaults.odin # DEFAULTS_PATH constant (#directory) ├── main.odin # Entry point +├── bench/ # Template rendering benchmark └── defaults/layouts/ # Bundled default templates ``` @@ -64,6 +65,7 @@ thor/ | | `sectionate.odin` | `wrap_sections` — splits HTML at `

` into `
` wrappers | | | `highlight.odin` | Syntax highlighting via tree-sitter. Imports `../treesitter`. | | `mustache/` | See [Mustache engine](#mustache-engine) below | Template engine | +| `bench/` | `bench.odin` + `templates/` | Standalone template rendering benchmark. Generates 500 posts + 100 comments, renders with indented partials + inheritance + pipes. `--dump ` for output validation, positional arg for iteration count (default 250). | Icon SVGs live as HTML partials in `layouts/partials/icons/` (home, github, rss, chevron_up, star). @@ -333,6 +335,16 @@ odin test . # main package tests (site, frontmatter) odin test . -all-packages # includes mustache specs, lambdas, pipes, diagnostics, markdown tests ``` +### Benchmark + +```bash +cd thor +odin build bench -o:speed +./bench.bin # 250 iterations, prints timing +./bench.bin --dump output.html # render once, write to file for diff validation +./bench.bin 1000 # custom iteration count +``` + ## Mustache engine Spec-compliant implementation at `mustache/`. See `mustache/SPEC.md` for the implementation specification, `mustache/EXTENSIONS.md` for non-standard extensions (pipes), and `mustache/diagnostic.odin` for the rust-style error formatter. @@ -341,11 +353,11 @@ Spec-compliant implementation at `mustache/`. See `mustache/SPEC.md` for the imp | File | Responsibility | |---|---| -| `mustache.odin` | Public API (`parse`, `render`, `Template`), parser (`parse_section`), renderer (`render_nodes`, takes `Template` by value), template inheritance (`merge_block_overrides`), `delete_template`/`delete_partials`. Pipe support in Variable/Unescaped/Section/Inverted tags. | +| `mustache.odin` | Public API (`parse`, `render`, `Template`), parser (`parse_section`), renderer (`render_nodes` with `Indent_State` for partial indentation), template inheritance (`merge_block_overrides`), `delete_template`/`delete_partials`. Pipe support in Variable/Unescaped/Section/Inverted tags. | | `tokenizer.odin` | Tokenizer (template string → `[]Token`), standalone whitespace detection | | `data.odin` | Reflection-based data model: `base_value` (peels union/any/nested-any layers), `lookup_in` (structs + maps, handles `Type_Info_Any` value kind in maps), `resolve_name`, `is_truthy`, `any_to_string`, `list_info`, `extract_list_element`, `call_interp_lambda`/`call_section_lambda` | | `pipes.odin` | Pipes extension: `Pipe_Filter` AST, `parse_pipeline` (takes `pos`), `apply_pipeline`, `apply_filter` (switch dispatch: `group_by` + `format`), `apply_group_by`, `apply_format`. Stored on `Node.filters`; render-scoped results in temp allocator. | -| `diagnostic.odin` | Rust-style error formatter: `format_error` (multi-line context, ANSI colors via `core:terminal/ansi`, `colorize` param), `format_render_error` (dispatch on `Render_Error`), `line_col`, `line_text`, `context_extent`, `count_lines`, `digit_count`, `should_colorize`. | +| `diagnostic.odin` | Rust-style error formatter: `format_error` (multi-line context, ANSI colors via `core:terminal/ansi`, `colorize` param), `format_render_error` (formats `Error`), `line_col`, `line_text`, `context_extent`, `count_lines`, `digit_count`, `should_colorize`. | | `suggest.odin` | Strict-warning helpers: `validate_key_path` (walks dotted path, crosses maps silently), `suggest_correction` (Levenshtein via `core:strings/levenshtein_distance`), `collect_struct_keys` (via reflection, recurses into `using`), `struct_has_field` (distinguishes missing field from nil value — needed for `Maybe(bool)`), `collect_partial_names`, `collect_block_names`. | | `spec_test.odin` | JSON spec test runner — loads `spec/specs/*.json`, runs each test case. Uses `log.nil_logger()` to suppress expected warnings. | | `lambda_test.odin` | Spec lambda tests | @@ -361,7 +373,7 @@ render(tmpl, data, partials) → render_nodes (walks flat node array against con ``` - **Two-phase API**: `parse()` produces a reusable `Template`, `render()` walks it against data. Templates parsed once, rendered many times. -- **Flat `[dynamic]Node` array** with `first_child`/`child_count` indices — pre-order layout. Each `Node` carries `pos: int` (byte offset into source) for diagnostics. +- **Flat `[dynamic]Node` array** with `children: []Node` slices (pre-order layout; slices point into the backing array). Each `Node` carries `pos: int` (byte offset into source) for diagnostics. - **`Template`** carries `source` and `path` — used by diagnostics to show file location and source context. - **Context stack**: `^[dynamic]any` with `append`/`pop` for section push/pop. - **`render_nodes` takes `Template` by value** (not `^Template`) — Odin's calling convention promotes to pointer when efficient. Eliminates "local copy" patterns at call sites. @@ -370,12 +382,13 @@ render(tmpl, data, partials) → render_nodes (walks flat node array against con - **`lookup_in`** resolves keys on structs (via `reflect.struct_field_value_by_name` with `allow_using = true`) and maps. Detects `Type_Info_Any` value kind in maps and reads the inner any directly to avoid double-wrap. - **Template inheritance**: `{{*key}}` resolves partial name from data context at render time. +- **Render-time partial indentation**: `Indent_State` threads `at_line_start` through `render_nodes` so partial indent is applied at render time (via `write_indented` on Text nodes) instead of reparsing the partial's source. `render_template` writes initial indent, creates state, calls `render_nodes`. Data-injected newlines don't pick up indent (Variables don't update `at_line_start`). ### Diagnostics Rust-style error messages with multi-line source context, caret underlines, and Levenshtein suggestions. ANSI colors via `core:terminal/ansi`, gated on `should_colorize()` (TTY detection on stderr). -**Error types**: `Syntax_Error{msg, pos}` and `Data_Error{msg, pos}` — both carry byte offset into template source. (`Partial_Error` was removed — dead code.) +**Error types**: `Error_Body{msg, pos, kind}` where `kind` is `Error_Kind.Syntax` (parse-time) or `Error_Kind.Data` (render-time). `Error` is a single-variant union wrapping `Error_Body` (nilable for `!= nil` / `or_return`). **Strict-by-default warnings** — `render_nodes` emits `log.warnf` diagnostics for: - Unknown keys in `{{k}}`, `{{{k}}}`, `{{#k}}`, `{{^k}}` (via `validate_key_path` + `suggest_correction`) @@ -424,7 +437,6 @@ See `mustache/EXTENSIONS.md`. You may never, *ever* remove `TODO:` or `FIXME:` comments. Those are for humans, not machines. See `HUGO.md` for analysis of why thor doesn't need Hugo's shortcode context isolation. -See `mustache/PARTIAL_INDENT.md` for whitespace handling analysis. See `mustache/SPEC.md` for the original implementation specification. See `mustache/EXTENSIONS.md` for non-standard extensions (pipes). diff --git a/bench/bench-better-redinent b/bench/bench-better-redinent new file mode 100755 index 0000000..6346d63 Binary files /dev/null and b/bench/bench-better-redinent differ diff --git a/bench/bench-perf b/bench/bench-perf new file mode 100755 index 0000000..7c2a270 Binary files /dev/null and b/bench/bench-perf differ diff --git a/mustache/mustache.odin b/mustache/mustache.odin index 85d9005..953b669 100644 --- a/mustache/mustache.odin +++ b/mustache/mustache.odin @@ -81,6 +81,14 @@ Block_Override :: struct { source: Template, } +// Indent_State threads partial-indent tracking through render_nodes so +// the renderer can apply indentation at render time instead of reparsing +// the partial's source with indentation baked in. +Indent_State :: struct { + indent: string, + at_line_start: bool, +} + delete_template :: proc(tmpl: ^Template) { if tmpl != nil && len(tmpl.nodes) > 0 { delete(tmpl.nodes) @@ -393,20 +401,29 @@ find_common_indent :: proc(children: []Node) -> string { text := children[i].text if len(text) > 0 { line_start := 0 - for j in 0 ..= len(text) { - if j == len(text) || text[j] == '\n' { - line := text[line_start:j] - if len(strings.trim_space(line)) > 0 { - ws := leading_whitespace(line) - if !found { - common = ws - found = true - } else if len(ws) < len(common) { - common = ws - } - } - line_start = j + 1 + for { + // Bulk-scan to next newline (AVX2-backed) instead of byte-by-byte. + rel := strings.index_byte(text[line_start:], '\n') + line_end: int + if rel < 0 { + line_end = len(text) + } else { + line_end = line_start + rel } + line := text[line_start:line_end] + if len(strings.trim_space(line)) > 0 { + ws := leading_whitespace(line) + if !found { + common = ws + found = true + } else if len(ws) < len(common) { + common = ws + } + } + if rel < 0 { + break + } + line_start = line_end + 1 } } } @@ -442,16 +459,19 @@ remove_line_indent :: proc(s: string, indent: string, allocator := context.alloc if at_line_start { if i + len(indent) <= len(s) && s[i:i + len(indent)] == indent { i += len(indent) - at_line_start = false - continue } at_line_start = false } - append(&buf, s[i]) - if s[i] == '\n' { - at_line_start = true + // Bulk-append up to and including the next newline (AVX2-backed). + rel := strings.index_byte(s[i:], '\n') + if rel < 0 { + append(&buf, s[i:]) + break } - i += 1 + next := i + rel + append(&buf, s[i:next + 1]) + i = next + 1 + at_line_start = true } return string(buf[:]) @@ -461,34 +481,6 @@ remove_line_indent :: proc(s: string, indent: string, allocator := context.alloc // Indentation helpers // --------------------------------------------------------------------------- -indent_lines :: proc(source: string, indent: string) -> string { - if len(indent) == 0 || len(source) == 0 { - return source - } - b: strings.Builder - strings.builder_init(&b, context.temp_allocator) - write_indented(&b, indent, source) - return strings.to_string(b) -} - -write_indented :: proc(b: ^strings.Builder, indent: string, content: string) { - if len(indent) == 0 || len(content) == 0 { - strings.write_string(b, content) - return - } - at_line_start := true - for i in 0 ..< len(content) { - if at_line_start { - strings.write_string(b, indent) - at_line_start = false - } - strings.write_byte(b, content[i]) - if content[i] == '\n' { - at_line_start = true - } - } -} - render_template :: proc( pt: Template, ctx: ^[dynamic]any, @@ -497,21 +489,36 @@ render_template :: proc( blocks: map[string]Block_Override, indent: string, ) -> Error { - if len(indent) > 0 && len(pt.source) > 0 { - // Per Mustache spec: the partial's source is indented before rendering, - // not its output. This is necessary so that data-injected newlines - // (e.g. from `{{{content}}}` where content contains `\n`) do NOT pick - // up the indent — only source-level line breaks do. - indented := indent_lines(pt.source, indent) - reparse := parse( - indented, - pt.path, - context.temp_allocator, - context.temp_allocator, - ) or_return - return render_nodes(reparse, reparse.nodes[:], ctx, partials, b, blocks) + if len(indent) > 0 { + state := Indent_State{indent = indent, at_line_start = false} + strings.write_string(b, indent) // first line always gets indent + return render_nodes(pt, pt.nodes[:], ctx, partials, b, blocks, &state) + } + return render_nodes(pt, pt.nodes[:], ctx, partials, b, blocks, nil) +} + +write_indented :: proc(b: ^strings.Builder, indent: string, content: string, at_line_start: ^bool) { + if len(indent) == 0 || len(content) == 0 { + strings.write_string(b, content) + return + } + i := 0 + for i < len(content) { + if at_line_start^ { + strings.write_string(b, indent) + at_line_start^ = false + } + // Bulk-write up to and including the next newline (AVX2-backed). + rel := strings.index_byte(content[i:], '\n') + if rel < 0 { + strings.write_string(b, content[i:]) + return + } + next := i + rel + strings.write_string(b, content[i:next + 1]) + i = next + 1 + at_line_start^ = true } - return render_nodes(pt, pt.nodes[:], ctx, partials, b, blocks) } // --------------------------------------------------------------------------- @@ -525,16 +532,25 @@ render_nodes :: proc( partials: map[string]Template, b: ^strings.Builder, blocks: map[string]Block_Override = nil, + indent_state: ^Indent_State = nil, ) -> Error { i := 0 for i < len(nodes) { node := nodes[i] switch node.kind { case .Text: - strings.write_string(b, node.text) + if indent_state != nil { + write_indented(b, indent_state.indent, node.text, &indent_state.at_line_start) + } else { + strings.write_string(b, node.text) + } i += 1 case .Variable: + if indent_state != nil && indent_state.at_line_start { + strings.write_string(b, indent_state.indent) + indent_state.at_line_start = false + } val := resolve_name(node.key, ctx[:]) if val == nil { warn_unknown_key(current, ctx[:], node) @@ -563,6 +579,7 @@ render_nodes :: proc( partials, &temp, blocks, + nil, ) or_return write_value(b, strings.to_string(temp), escape = true) } @@ -572,6 +589,10 @@ render_nodes :: proc( i += 1 case .Unescaped: + if indent_state != nil && indent_state.at_line_start { + strings.write_string(b, indent_state.indent) + indent_state.at_line_start = false + } val := resolve_name(node.key, ctx[:]) if val == nil { warn_unknown_key(current, ctx[:], node) @@ -600,6 +621,7 @@ render_nodes :: proc( partials, &temp, blocks, + nil, ) or_return write_value(b, strings.to_string(temp), escape = false) } @@ -635,6 +657,7 @@ render_nodes :: proc( partials, b, blocks, + nil, ) or_return } } else if is_truthy(val) { @@ -652,12 +675,13 @@ render_nodes :: proc( partials, b, blocks, + indent_state, ) or_return } } else { append(ctx, val) defer pop(ctx) - render_nodes(current, children, ctx, partials, b, blocks) or_return + render_nodes(current, children, ctx, partials, b, blocks, indent_state) or_return } } i += 1 + len(node.children) @@ -675,7 +699,7 @@ render_nodes :: proc( val = transformed } if !is_truthy(val) { - render_nodes(current, node.children, ctx, partials, b, blocks) or_return + render_nodes(current, node.children, ctx, partials, b, blocks, indent_state) or_return } i += 1 + len(node.children) @@ -690,6 +714,9 @@ render_nodes :: proc( warn_missing_partial(current, partials, node, name) } else { render_template(pt, ctx, partials, b, nil, node.indent) or_return + if indent_state != nil { + indent_state.at_line_start = false + } } i += 1 @@ -720,8 +747,10 @@ render_nodes :: proc( partials, &temp, content_blocks, + nil, ) or_return - write_indented(b, node.indent, strings.to_string(temp)) + at_ls := true + write_indented(b, node.indent, strings.to_string(temp), &at_ls) } else { render_nodes( render_current, @@ -730,6 +759,7 @@ render_nodes :: proc( partials, b, content_blocks, + indent_state, ) or_return } i += 1 + len(node.children) @@ -743,6 +773,9 @@ render_nodes :: proc( } else { warn_unmatched_block_overrides(current, pt, parent_children) render_template(pt, ctx, partials, b, merged, node.indent) or_return + if indent_state != nil { + indent_state.at_line_start = false + } } i += 1 + len(node.children) } diff --git a/mustache/tokenizer.odin b/mustache/tokenizer.odin index 500ed7f..59a43bf 100644 --- a/mustache/tokenizer.odin +++ b/mustache/tokenizer.odin @@ -36,60 +36,73 @@ tokenize :: proc( text_start := 0 for i < len(src) { - if src[i] == '{' && i + 1 < len(src) && src[i + 1] == '{' { - if i > text_start { - append(&tokens, Token{kind = .Text, value = src[text_start:i], pos = text_start}) + // Find next '{' via AVX2/SSE-backed memchr instead of byte-by-byte scan. + rel := strings.index_byte(src[i:], '{') + if rel < 0 { + i = len(src) + break + } + next := i + rel + // Single '{' (not '{{') — advance past it and keep scanning. + if next + 1 >= len(src) || src[next + 1] != '{' { + i = next + 1 + continue + } + i = next + + if i > text_start { + append(&tokens, Token{kind = .Text, value = src[text_start:i], pos = text_start}) + } + + tag_pos := i + + if i + 2 < len(src) && src[i + 2] == '{' { + content_start := i + 3 + idx := strings.index(src[content_start:], "}}}") + if idx < 0 { + return tokens, Error_Body { + msg = "unclosed triple mustache '{{{'", + pos = tag_pos, + kind = .Syntax, + } + } + close := content_start + idx + key := strings.trim_space(src[content_start:close]) + append(&tokens, Token{kind = .Unescaped, value = key, pos = tag_pos}) + i = close + 3 + text_start = i + } else { + content_start := i + 2 + sigil: byte = 0 + if content_start < len(src) { + sigil = src[content_start] } - tag_pos := i + kind: Token_Kind + key_start := content_start - if i + 2 < len(src) && src[i + 2] == '{' { - content_start := i + 3 - idx := strings.index(src[content_start:], "}}}") - if idx < 0 { - return tokens, Error_Body { - msg = "unclosed triple mustache '{{{'", - pos = tag_pos, - kind = .Syntax, - } - } - close := content_start + idx - key := strings.trim_space(src[content_start:close]) - append(&tokens, Token{kind = .Unescaped, value = key, pos = tag_pos}) - i = close + 3 - text_start = i - } else { - content_start := i + 2 - sigil: byte = 0 - if content_start < len(src) { - sigil = src[content_start] - } + switch sigil { + case '&': + kind = .Unescaped; key_start = content_start + 1 + case '#': + kind = .Section_Open; key_start = content_start + 1 + case '^': + kind = .Inverted_Open; key_start = content_start + 1 + case '/': + kind = .Section_Close; key_start = content_start + 1 + case '!': + kind = .Comment; key_start = content_start + 1 + case '>': + kind = .Partial; key_start = content_start + 1 + case '<': + kind = .Parent; key_start = content_start + 1 + case '$': + kind = .Block_Open; key_start = content_start + 1 + case: + kind = .Variable + } - kind: Token_Kind - key_start := content_start - - switch sigil { - case '&': - kind = .Unescaped; key_start = content_start + 1 - case '#': - kind = .Section_Open; key_start = content_start + 1 - case '^': - kind = .Inverted_Open; key_start = content_start + 1 - case '/': - kind = .Section_Close; key_start = content_start + 1 - case '!': - kind = .Comment; key_start = content_start + 1 - case '>': - kind = .Partial; key_start = content_start + 1 - case '<': - kind = .Parent; key_start = content_start + 1 - case '$': - kind = .Block_Open; key_start = content_start + 1 - case: - kind = .Variable - } - - close_idx := strings.index(src[key_start:], "}}") + close_idx := strings.index(src[key_start:], "}}") if close_idx < 0 { return tokens, Error_Body { msg = "unclosed tag '{{'", @@ -97,40 +110,37 @@ tokenize :: proc( kind = .Syntax, } } - close := key_start + close_idx + close := key_start + close_idx - content := src[key_start:close] + content := src[key_start:close] - if kind == .Comment { - append(&tokens, Token{kind = .Comment, value = content, pos = tag_pos}) - } else if kind == .Partial { - trimmed := strings.trim_space(content) - is_dyn := false - if len(trimmed) > 0 && trimmed[0] == '*' { - is_dyn = true - trimmed = strings.trim_space(trimmed[1:]) - } - append( - &tokens, - Token { - kind = .Partial, - value = trimmed, - is_dynamic = is_dyn, - pos = tag_pos, - }, - ) - } else { - append( - &tokens, - Token{kind = kind, value = strings.trim_space(content), pos = tag_pos}, - ) + if kind == .Comment { + append(&tokens, Token{kind = .Comment, value = content, pos = tag_pos}) + } else if kind == .Partial { + trimmed := strings.trim_space(content) + is_dyn := false + if len(trimmed) > 0 && trimmed[0] == '*' { + is_dyn = true + trimmed = strings.trim_space(trimmed[1:]) } - - i = close + 2 - text_start = i + append( + &tokens, + Token { + kind = .Partial, + value = trimmed, + is_dynamic = is_dyn, + pos = tag_pos, + }, + ) + } else { + append( + &tokens, + Token{kind = kind, value = strings.trim_space(content), pos = tag_pos}, + ) } - } else { - i += 1 + + i = close + 2 + text_start = i } }