perf: Improved performance of highlighter code.

LOAD_GRAMMARS_IDEAS.md

@@ -0,0 +1,88 @@
+# load_grammar Optimization Ideas
+
+## Current bottleneck
+
+`load_grammar` is called once per unique language on first use. Each call costs ~30-40ms:
+- `dlopen` — loads `.so` from disk (dominant)
+- `read_entire_file_from_path` — reads `.scm` from disk
+- `query_new` — compiles query
+- `query_cursor_new` — allocates cursor (cheap)
+
+With ~7 languages, first-load cost is ~210-280ms. Subsequent calls hit cache (~0ms).
+
+## Code-level optimizations
+
+### Avoid double-cloning query source
+
+Currently: `read_entire_file_from_path` → `[]byte` → `string(raw)` → `strings.clone_to_cstring(query_src)`. Two copies of the entire `.scm` file.
+
+Fix: read into a buffer with a trailing null byte, use directly as cstring. One read, zero copies.
+
+### Reuse the grammar path string
+
+`fmt.tprintf("%s/%s.so", grammar_dir, lang)` is computed in `ensure_parser` AND again in error diagnostics (line 284). Compute once, store in `Grammar_Cache`.
+
+### Avoid `fmt.tprintf` for symbol name
+
+`"tree_sitter_" + lang` can be simple concatenation instead of format string parsing.
+
+### Extract error diagnostics to a separate proc
+
+The query error diagnostics (lines 250-300) make up more than half of `load_grammar`. Extract to `report_query_error(lang, query_src, err_offset, err_type, query_path)`. Makes `load_grammar` focused on the happy path.
+
+## Architectural optimizations
+
+### Pre-scan content for languages, then batch-load
+
+Before rendering, scan all markdown for `language-X` code blocks. Collect the unique set. Load all grammars in one pass.
+
+Benefits:
+- Makes cost visible ("loading grammars for: bash, odin, nu...")
+- Enables parallel loading
+- Moves cost to a predictable point in the build
+
+### Parallel grammar loading
+
+`dlopen` is thread-safe (POSIX). Load N grammars on N threads simultaneously.
+
+- Sequential: ~210ms (7 languages × ~30ms each)
+- Parallel (4 cores): ~60ms
+- Parallel (7 threads): ~30ms
+
+Biggest single win for first-load time. Odin's `core:thread` or `core:sync` can manage the thread pool.
+
+### Static-link more grammars
+
+Extend the `mkGrammarStaticLib` pattern (already used for HTML/CSS in `flake.nix`) to bash, odin, nu, etc.
+
+- Zero `dlopen` — grammars baked into binary
+- Zero `read_entire_file_from_path` — queries embedded via `#load` or `#directory`
+- Zero `query_new` compilation — could pre-compile or use compile-time embedding
+- Downside: binary size grows (~500KB per language), adding languages requires recompiling
+
+### Embed queries at compile time
+
+Even without static-linking grammars, the `.scm` query files could be embedded via `#load` or `#directory`.
+
+- Eliminates `read_entire_file_from_path` per language
+- Smaller win than static linking but simpler
+- Queries are small text files (~2-10KB each)
+
+### Cache compiled queries across runs
+
+`query_new` compiles the `.scm` source into an internal representation. If this could be serialized and cached to disk (like a `.thorcache`), subsequent builds could skip compilation.
+
+- Tree-sitter's query format isn't documented as serializable — would need investigation
+- Even if not serializable, caching the raw `.scm` source in memory avoids re-reading from disk in watch mode (already handled by `grammar_cache` persistence)
+
+### Pre-link common grammars in the flake
+
+The flake could compile grammar `.so` files as Nix build inputs and pass their paths to thor at runtime via the `grammars` config field. This doesn't eliminate `dlopen` but ensures the files are always available in the Nix store.
+
+## Priority ranking
+
+1. **Parallel grammar loading** — immediate ~4-7x speedup on first load, no binary changes
+2. **Static-link common grammars** — eliminates the problem entirely, but requires flake work
+3. **Pre-scan + batch-load** — enables parallel loading and better UX logging
+4. **Avoid double-clone of query source** — quick code cleanup, small win
+5. **Embed queries at compile time** — eliminates file reads, moderate effort

PARALLELIZATION.md

@@ -0,0 +1,115 @@
+# Parallelization Plan
+
+## Current bottleneck
+
+`compile_query` (which contains `query_new`) is ~100% of the grammar loading cost. With ~7 languages, sequential loading costs ~200-280ms. The rest of the build is ~12ms.
+
+## Thread safety concerns
+
+Current code is single-threaded and assumes it:
+- `grammar_cache` map — concurrent writes unsafe
+- `context.temp_allocator` — shared, not thread-safe
+- `Site.pages` dynamic array — concurrent appends unsafe
+- Site arena — bump pointer, not atomic
+
+Any parallelization must address these.
+
+## Dependency graph
+
+```
+init_site → build_vfs → scan_content (discover files)
+  → grammar loading (must finish before any highlight_code)
+  → load_page × N (markdown pipeline + highlight_code)
+  → render_site (template rendering × N + minify × N)
+  → RSS + sitemap (needs all pages loaded)
+  → asset copying (independent)
+```
+
+Key barriers:
+- Grammar loading must finish before any highlighting
+- Pages must be loaded before rendering
+- All pages must be rendered before RSS/sitemap
+- Minify + write are independent per page after render
+
+## Three approaches
+
+### Option 1: Per-subsystem parallelism
+
+```
+Phase 1: Parallel grammar loading (7 threads)
+Phase 2: Parallel content loading (N pages on M threads)
+Phase 3: Parallel template rendering (N pages on M threads)
+Phase 4: Parallel minification (N pages on M threads)
+Phase 5: Sequential: RSS, sitemap, assets
+```
+
+Pros:
+- Simple barriers — each phase completes before the next starts
+- Easy to reason about
+
+Cons:
+- Thread pool setup/teardown per phase (or reuse a pool with barriers)
+- Memory pressure: all pages loaded before any rendered
+- Load imbalance within phases (complex pages vs simple)
+
+### Option 2: Per-page parallelism
+
+```
+Pre-load all grammars (sequential or parallel)
+Then for each page (in parallel):
+    load_page → md.process → render → minify → write
+Then sequential: RSS, sitemap, assets
+```
+
+Pros:
+- Natural work unit — each page flows through the full pipeline independently
+- No barriers between phases
+
+Cons:
+- Complex pages clog workers while simple pages finish fast
+- Grammar loading must happen first (barrier)
+- Needs thread-safe shared state (grammar cache, template cache, allocator)
+
+### Option 3: Generic worker queue
+
+```
+Single thread pool with work stealing.
+Jobs: grammar_load(lang), load_page(file), render_page(page), minify(html), write_output(path)
+Dependencies tracked via futures or callbacks.
+```
+
+Pros:
+- Most flexible — handles all work types
+- Best load balancing (work stealing across types)
+- No wasted thread setup between phases
+
+Cons:
+- Most complex to implement
+- Need dependency tracking (can't render before load completes)
+- Careful shared-state management needed
+- Memory management with arena allocator (thread safety)
+
+## Recommended starting point
+
+**Pre-load grammars in parallel, keep everything else sequential.**
+
+```
+Phase 0: Pre-scan content for unique languages (fast, sequential)
+Phase 1: Parallel grammar loading (one thread per language, ~30ms)
+         → each thread writes to a pre-assigned slot (no map contention)
+         → each thread uses its own temp allocator
+         → main thread merges results into grammar_cache after join
+Phase 2: Sequential build as today (~12ms)
+```
+
+Expected: ~42ms total instead of ~210ms. Minimal architecture change — no thread-safe allocators needed for the rest of the pipeline.
+
+This can later evolve toward option 3 (generic queue) if page count grows or per-page processing becomes a bottleneck.
+
+## Implementation notes
+
+- Odin's `core:thread` or `core:sync` can manage the thread pool
+- Each grammar-loading thread needs its own `context.temp_allocator` for path strings
+- The `grammar_cache` map write happens on the main thread after all threads join
+- `dlopen` is thread-safe (POSIX guarantee)
+- `query_new` is likely thread-safe (independent computation per language, no shared state in tree-sitter)

TODOS.md

@@ -17,9 +17,15 @@
 - [ ] Split `load_page` into frontmatter-parse + body-process phases so draft pages can skip the markdown pipeline entirely
 - [ ] Use spall to find ways to reduce run time.
 - [ ] Consider using `#soa` for Page lists.
+- [ ] too many `write_string` calls in `highlight_block`
+- [ ] return `src: cstring` from `load_query`.
+- [ ] Improve `unescape_html` with simd.
+- [ ] generate summary before syntax highlighting.
+
 
 ## Memory Management
 
+- [ ] Leaks in highlighter code.
 - [ ] Not sure whether to use temp allocator or site_allocator in opengraph.odin.
 - [ ] Not sure whether to use temp allocator or site_allocator in `site_load_content`.
 - [ ] Might not need to allocate in `strip_html_tags`

flake.nix

@@ -133,7 +133,7 @@
 
             buildPhase = ''
               runHook preBuild
-              odin build . -o:speed -out:${pname}-keep
+              odin build . -o:speed -no-bounds-check -out:${pname}-keep
               runHook postBuild
             '';
 

html.odin

@@ -29,7 +29,7 @@ strip_html_tags :: proc(s: string, allocator := context.allocator) -> string {
 }
 
 unescape_html :: proc(s: string) -> string {
-	sb := strings.builder_make()
+	sb := strings.builder_make_len(len(s))
 	defer strings.builder_destroy(&sb)
 
 	start := 0
@@ -41,15 +41,21 @@ unescape_html :: proc(s: string) -> string {
 		if semi < 0 {
 			break
 		}
-		entity := s[i : i + semi + 1]
+		entity := s[i:i + semi + 1]
 		replacement := ""
 		switch entity {
-		case "&amp;":  replacement = "&"
-		case "&lt;":   replacement = "<"
-		case "&gt;":   replacement = ">"
-		case "&quot;": replacement = "\""
-		case "&#39;", "&apos;": replacement = "'"
-		case: continue
+		case "&amp;":
+			replacement = "&"
+		case "&lt;":
+			replacement = "<"
+		case "&gt;":
+			replacement = ">"
+		case "&quot;":
+			replacement = "\""
+		case "&#39;", "&apos;":
+			replacement = "'"
+		case:
+			continue
 		}
 		if i > start {
 			strings.write_string(&sb, s[start:i])
@@ -77,7 +83,7 @@ generate_summary :: proc(html: string, max_words: int = 70) -> string {
 	separated, _ = strings.replace_all(separated, "</h1>", "\n\n")
 	separated, _ = strings.replace_all(separated, "</h2>", "\n\n")
 	separated, _ = strings.replace_all(separated, "</h3>", "\n\n")
-	separated,_ = strings.replace_all(separated, "</h4>", "\n\n")
+	separated, _ = strings.replace_all(separated, "</h4>", "\n\n")
 	separated, _ = strings.replace_all(separated, "</h5>", "\n\n")
 	separated, _ = strings.replace_all(separated, "</h6>", "\n\n")
 	separated, _ = strings.replace_all(separated, "</li>", "\n\n")
@@ -135,3 +141,4 @@ generate_summary :: proc(html: string, max_words: int = 70) -> string {
 
 	return strings.to_string(sb)
 }
+

markdown/highlight.odin

@@ -16,7 +16,7 @@ find_first_error_line :: proc(root: ts.Node) -> int {
 	if ts.node_is_error(root) {
 		return int(ts.node_start_point(root).row) + 1
 	}
-	for i in 0..<ts.node_child_count(root) {
+	for i in 0 ..< ts.node_child_count(root) {
 		child := ts.node_child(root, u32(i))
 		if ts.node_has_error(child) {
 			line := find_first_error_line(child)
@@ -28,55 +28,66 @@ find_first_error_line :: proc(root: ts.Node) -> int {
 	return 0
 }
 
-capture_name_to_css :: proc(name: string) -> string {
-	sb := strings.builder_make()
-	seg := strings.builder_make()
+write_span_open :: proc(b: ^strings.Builder, buf: ^[128]u8, name: string) {
+	pos := 0
+
+	prefix := "<span class=\""
+	for i in 0 ..< len(prefix) {
+		buf[pos] = prefix[i]
+		pos += 1
+	}
+
 	first := true
-	for i in 0..<len(name) {
+	for i in 0 ..< len(name) {
 		if name[i] == '.' {
-			if !first do strings.write_byte(&sb, ' ')
+			if !first {buf[pos] = ' '; pos += 1}
 			first = false
-			strings.write_string(&sb, "hl-")
-			strings.write_string(&sb, strings.to_string(seg))
-			strings.write_byte(&seg, '-')
-		} else {
-			strings.write_byte(&seg, name[i])
+			buf[pos] = 'h'; buf[pos + 1] = 'l'; buf[pos + 2] = '-'; pos += 3
+			for j in 0 ..< i {
+				buf[pos] = '-' if name[j] == '.' else name[j]
+				pos += 1
 			}
 		}
-	if !first do strings.write_byte(&sb, ' ')
-	strings.write_string(&sb, "hl-")
-	strings.write_string(&sb, strings.to_string(seg))
-	return strings.to_string(sb)
+	}
+	if !first {buf[pos] = ' '; pos += 1}
+	buf[pos] = 'h'; buf[pos + 1] = 'l'; buf[pos + 2] = '-'; pos += 3
+	for j in 0 ..< len(name) {
+		buf[pos] = '-' if name[j] == '.' else name[j]
+		pos += 1
+	}
+	buf[pos] = '"'; pos += 1
+	buf[pos] = '>'; pos += 1
+
+	strings.write_string(b, string(buf[:pos]))
 }
 
-escape_html :: proc(s: string) -> string {
-	sb := strings.builder_make()
-	defer strings.builder_destroy(&sb)
-
+write_escaped :: proc(b: ^strings.Builder, s: string) {
 	start := 0
-	for i in 0..<len(s) {
+	for i in 0 ..< len(s) {
 		switch s[i] {
 		case '&':
-			if i > start do strings.write_string(&sb, s[start:i])
-			strings.write_string(&sb, "&amp;")
+			if i > start do strings.write_string(b, s[start:i])
+			strings.write_string(b, "&amp;")
 			start = i + 1
 		case '<':
-			if i > start do strings.write_string(&sb, s[start:i])
-			strings.write_string(&sb, "&lt;")
+			if i > start do strings.write_string(b, s[start:i])
+			strings.write_string(b, "&lt;")
 			start = i + 1
 		case '>':
-			if i > start do strings.write_string(&sb, s[start:i])
-			strings.write_string(&sb, "&gt;")
+			if i > start do strings.write_string(b, s[start:i])
+			strings.write_string(b, "&gt;")
 			start = i + 1
 		case '"':
-			if i > start do strings.write_string(&sb, s[start:i])
-			strings.write_string(&sb, "&quot;")
+			if i > start do strings.write_string(b, s[start:i])
+			strings.write_string(b, "&quot;")
 			start = i + 1
 		}
 	}
-	if start == 0 do return s
-	if start < len(s) do strings.write_string(&sb, s[start:])
-	return strings.to_string(sb)
+	if start == 0 {
+		strings.write_string(b, s)
+	} else if start < len(s) {
+		strings.write_string(b, s[start:])
+	}
 }
 
 unescape_html :: proc(s: string) -> string {
@@ -84,19 +95,25 @@ unescape_html :: proc(s: string) -> string {
 	defer strings.builder_destroy(&sb)
 
 	start := 0
-	for i in 0..<len(s) {
+	for i in 0 ..< len(s) {
 		if s[i] != '&' do continue
 		semi := strings.index(s[i:], ";")
 		if semi < 0 do break
-		entity := s[i : i + semi + 1]
+		entity := s[i:i + semi + 1]
 		replacement := ""
 		switch entity {
-		case "&amp;":  replacement = "&"
-		case "&lt;":   replacement = "<"
-		case "&gt;":   replacement = ">"
-		case "&quot;": replacement = "\""
-		case "&#39;", "&apos;": replacement = "'"
-		case: continue
+		case "&amp;":
+			replacement = "&"
+		case "&lt;":
+			replacement = "<"
+		case "&gt;":
+			replacement = ">"
+		case "&quot;":
+			replacement = "\""
+		case "&#39;", "&apos;":
+			replacement = "'"
+		case:
+			continue
 		}
 		if i > start do strings.write_string(&sb, s[start:i])
 		strings.write_string(&sb, replacement)
@@ -128,21 +145,25 @@ highlight_block :: proc(code: string, lang: string, file_path: string) -> string
 	if ts.node_has_error(root) {
 		line := find_first_error_line(root)
 		if line > 0 {
-			log.warnf("highlight: syntax errors in %s code block at line %d (%s)", lang, line, file_path)
+			log.warnf(
+				"highlight: syntax errors in %s code block at line %d (%s)",
+				lang,
+				line,
+				file_path,
+			)
 		} else {
 			log.warnf("highlight: syntax errors in %s code block (%s)", lang, file_path)
 		}
 	}
 
-	cursor := ts.query_cursor_new()
+	cursor := gc.cursor
 	if cursor == nil {
 		return code
 	}
-	defer ts.query_cursor_delete(cursor)
 
 	ts.query_cursor_exec(cursor, gc.query, root)
 
-	captures: [dynamic]Capture
+	captures := make([dynamic]Capture, 0, 64, context.temp_allocator)
 	defer delete(captures)
 
 	match: ts.Query_Match
@@ -162,29 +183,34 @@ highlight_block :: proc(code: string, lang: string, file_path: string) -> string
 		if len(name_full) > int(name_len) {
 			name = name_full[:int(name_len)]
 		}
-		append(&captures, Capture{
+		append(
+			&captures,
+			Capture {
 				start = ts.node_start_byte(cap.node),
 				end = ts.node_end_byte(cap.node),
 				name = name,
-		})
+			},
+		)
 	}
 
 	if len(captures) == 0 {
 		return code
 	}
 
-	sb := strings.builder_make()
+	sb := strings.builder_make_len(len(code) * 2)
 
 	last_pos: u32 = 0
-	stack: [dynamic]Capture
+	stack := make([dynamic]Capture, 0, 16, context.temp_allocator)
 	defer delete(stack)
 
+	buf: [128]u8
+
 	for cap in captures {
 		for len(stack) > 0 {
 			top := stack[len(stack) - 1]
 			if top.end <= cap.start {
 				if top.end > last_pos {
-					strings.write_string(&sb, escape_html(raw_code[last_pos:top.end]))
+					write_escaped(&sb, raw_code[last_pos:top.end])
 				}
 				strings.write_string(&sb, "</span>")
 				last_pos = top.end
@@ -195,26 +221,25 @@ highlight_block :: proc(code: string, lang: string, file_path: string) -> string
 		}
 
 		if cap.start > last_pos {
-			strings.write_string(&sb, escape_html(raw_code[last_pos:cap.start]))
+			write_escaped(&sb, raw_code[last_pos:cap.start])
 			last_pos = cap.start
 		}
 
-		css_class := capture_name_to_css(cap.name)
-		strings.write_string(&sb, fmt.tprintf("<span class=\"%s\">", css_class))
+		write_span_open(&sb, &buf, cap.name)
 		append(&stack, cap)
 	}
 
 	for len(stack) > 0 {
 		top := pop(&stack)
 		if top.end > last_pos {
-			strings.write_string(&sb, escape_html(raw_code[last_pos:top.end]))
+			write_escaped(&sb, raw_code[last_pos:top.end])
 		}
 		strings.write_string(&sb, "</span>")
 		last_pos = top.end
 	}
 
 	if int(last_pos) < len(raw_code) {
-		strings.write_string(&sb, escape_html(raw_code[last_pos:]))
+		write_escaped(&sb, raw_code[last_pos:])
 	}
 
 	return strings.to_string(sb)
@@ -266,7 +291,7 @@ highlight_code :: proc(html: string, file_path: string) -> string {
 
 		code := html[code_start:end_idx]
 		highlighted := highlight_block(code, lang, file_path)
-		strings.write_string(&sb, fmt.tprintf(`<pre><code class="language-%s">%s</code></pre>`, lang, highlighted))
+		fmt.sbprintf(&sb, `<pre><code class="language-%s">%s</code></pre>`, lang, highlighted)
 
 		pos = end_idx + len(CODE_END)
 	}
@@ -280,3 +305,4 @@ highlight_code :: proc(html: string, file_path: string) -> string {
 	}
 	return strings.to_string(sb)
 }
+

site_test.odin

@@ -179,3 +179,24 @@ test_init_site_md_enable_disable :: proc(t: ^testing.T) {
 	testing.expect(t, .Sidenotes in site.markdown_extensions)
 }
 
+@(test)
+test_init_site_config_paths :: proc(t: ^testing.T) {
+	path := write_temp_config("paths", `{
+		"content_dir": "/custom/content",
+		"assets_dir": "/custom/assets",
+		"output_dir": "/custom/output",
+		"layouts_dir": "/custom/layouts"
+	}`)
+	defer os.remove(path)
+
+	site: Site
+	args := []string{"thor", fmt.tprintf("-config:%s", path)}
+	init_site(&site, args)
+	defer destroy_site(&site)
+
+	testing.expect_value(t, site.content_dir, "/custom/content")
+	testing.expect_value(t, site.assets_dir, "/custom/assets")
+	testing.expect_value(t, site.output_dir, "/custom/output")
+	testing.expect_value(t, site.layouts_dir, "/custom/layouts")
+}
+

treesitter/treesitter.odin

@@ -121,6 +121,7 @@ Grammar_Cache :: struct {
 	language:     Language,
 	parser:       Parser,
 	query:        Query,
+	cursor:       Query_Cursor,
 	query_failed: bool,
 }
 
@@ -184,19 +185,15 @@ ensure_parser :: proc(lang: string) -> ^Grammar_Cache {
 			return nil
 		}
 
-		so_path := fmt.tprintf("%s/%s.so", grammar_dir, lang)
-		so_c := strings.clone_to_cstring(so_path)
-		defer delete(so_c)
-		handle := dlopen(so_c, RTLD_LAZY)
+		so_path := fmt.caprintf("%s/%s.so", grammar_dir, lang, allocator = context.temp_allocator)
+		handle := dlopen(so_path, RTLD_LAZY)
 		if handle == nil {
 			log.warnf("treesitter: cannot load grammar %s (%s)", lang, so_path)
 			return nil
 		}
 
-		sym_name := fmt.tprintf("tree_sitter_%s", lang)
-		sym_c := strings.clone_to_cstring(sym_name)
-		defer delete(sym_c)
-		sym := dlsym(handle, sym_c)
+		sym_name := fmt.caprintf("tree_sitter_%s", lang, allocator = context.temp_allocator)
+		sym := dlsym(handle, sym_name)
 		if sym == nil {
 			log.errorf("treesitter: cannot find symbol %s in %s", sym_name, so_path)
 			return nil
@@ -240,8 +237,7 @@ load_grammar :: proc(lang: string) -> ^Grammar_Cache {
 		gc.query_failed = true
 		return nil
 	}
-	query_c := strings.clone_to_cstring(query_src)
-	defer delete(query_c)
+	query_c := strings.clone_to_cstring(query_src, context.temp_allocator)
 
 	err_offset: u32
 	err_type: Query_Error
@@ -299,6 +295,7 @@ load_grammar :: proc(lang: string) -> ^Grammar_Cache {
 	}
 
 	gc.query = query
+	gc.cursor = query_cursor_new()
 	return gc
 }