mirror of
https://github.com/sbrow/thor.git
synced 2026-10-25 10:11:55 -04:00
Compare commits
25
Commits
7cb1f5be32
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
b4f5e06ec8 | ||
|
|
122627c51c | ||
|
|
2f997e89bb | ||
|
|
90ad86c914 | ||
|
|
0612ba4453 | ||
|
|
565be86ac9 | ||
|
|
ed4019ff39 | ||
|
|
d334a6ed9f | ||
|
|
c570b29f9a | ||
|
|
83e488965e | ||
|
|
22dfa4c448 | ||
|
|
d02b22e94e | ||
|
|
489577b220 | ||
|
|
9f7db3a916 | ||
|
|
0392bfe587 | ||
|
|
0ac03c1c39 | ||
|
|
2f180f0e4c | ||
|
|
b932b22f15 | ||
|
|
4251efc173 | ||
|
|
5809eac195 | ||
|
|
72ef6111cc | ||
|
|
f6ce63dd4c | ||
|
|
317264b45b | ||
|
|
814b1c9996 | ||
|
|
7a8173832e |
@@ -59,7 +59,7 @@ jobs:
|
||||
|
||||
- name: Generate site
|
||||
working-directory: site
|
||||
run: ../result/bin/thor -base-url=https://sbrow.github.io/thor -minify
|
||||
run: ../result/bin/thor -base-url=https://oneidiotdeveloper.com/thor -minify
|
||||
|
||||
- name: Upload Pages artifact
|
||||
uses: actions/upload-pages-artifact@v3
|
||||
|
||||
@@ -0,0 +1,135 @@
|
||||
# Asset Pipeline Plan
|
||||
|
||||
Design for letting thor users bring their own CSS toolchain (Sass, PostCSS,
|
||||
Lightning CSS, Tailwind, …).
|
||||
|
||||
## Thesis
|
||||
|
||||
Let thor users bring their own CSS toolchain by **orchestrating the tool they
|
||||
already installed** — never bundling or reimplementing one. This is expressed
|
||||
through thor's existing mustache pipe grammar with two new ops: `tool` and
|
||||
`fingerprint`.
|
||||
|
||||
The earlier research design (content-addressed registry + deferred token
|
||||
resolution + a separate post-render path for "whole-output scanners") is
|
||||
**discarded**. It was overbuilt. Every tool — Tailwind included — scans
|
||||
*source*, which exists at render time, so every transform runs **eagerly,
|
||||
synchronously, and memoized**. No placeholder tokens, no post-render
|
||||
string-replace pass, no second integration surface.
|
||||
|
||||
## The two ops
|
||||
|
||||
```html
|
||||
<link href="{{ "css/main.scss" | tool sass | fingerprint | rel_url }}">
|
||||
```
|
||||
|
||||
```json5
|
||||
// thor.json
|
||||
{
|
||||
"tools": {
|
||||
"sass": "sass $in $out",
|
||||
"postcss": "postcss $in -o $out",
|
||||
"lightningcss": "lightningcss --minify $in -o $out",
|
||||
"tailwind": "tailwindcss -i $in -o $out"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Core model
|
||||
|
||||
The value flowing through the pipe is **bytes held by thor**, with a logical
|
||||
name attached (`css/main.css`). Each stage transforms the bytes; the **final
|
||||
stage publishes** one file to the output dir and returns its URL.
|
||||
|
||||
- A literal (`"css/main.scss"`) or context key names the **source** file on disk.
|
||||
- `| tool <name>` runs a subprocess, replacing the carried bytes with its output.
|
||||
- `| fingerprint` writes the content-addressed file and returns its hashed URL.
|
||||
- The chain composes with the existing `rel_url` for base_url subpaths.
|
||||
|
||||
**Memoization:** results are cached per build, keyed by (op-chain, source). The
|
||||
first page to reference an asset runs the tool; every other page hits the cache.
|
||||
This is what stops N pages from shelling out N times.
|
||||
|
||||
## `| tool <name>`
|
||||
|
||||
Looks up `<name>` in the `thor.json` `tools` table and runs its command
|
||||
template. `$in` and `$out` are **optional per-tool I/O adapters**:
|
||||
|
||||
| Template has | Behavior |
|
||||
|---|---|
|
||||
| `$in` | thor writes current bytes to a scratch path, substitutes it |
|
||||
| no `$in` | thor pipes current bytes to the tool's **stdin** |
|
||||
| `$out` | thor substitutes a scratch path, reads it back after |
|
||||
| no `$out` | thor captures the tool's **stdout** |
|
||||
|
||||
Defaulting to stdin/stdout avoids scratch-file overhead when the tool supports
|
||||
it. Tools that can't stream (Lightning CSS wants a path; Tailwind v3 needs an
|
||||
input file) just get `$in`/`$out` in their template. Either way the bytes are
|
||||
thor's, so chaining works regardless of each tool's preference:
|
||||
`| tool sass | tool postcss` composes even if sass streams and postcss wants
|
||||
files.
|
||||
|
||||
- **`$out` is always thor-controlled scratch** — the tool never picks the
|
||||
published filename.
|
||||
- **Extension derivation:** thor owns the output name; `main.scss` through
|
||||
`tool sass` becomes `main.css`.
|
||||
- **Errors:** thor gates on exit code and surfaces the tool's stderr through one
|
||||
code path. If the tool is missing from PATH, fail with its stderr — never
|
||||
bundle, never assume a runtime.
|
||||
- **A bare `tool` (no `fingerprint`) still publishes** — to the logical unhashed
|
||||
path (`output/css/main.css`), returning `/css/main.css`.
|
||||
|
||||
## `| fingerprint`
|
||||
|
||||
Content-addresses whatever bytes it currently holds. It has **two outputs**:
|
||||
|
||||
- **Side effect:** writes `css/main.<hash>.css` to the output dir.
|
||||
- **Return value:** the hashed URL string `/css/main.<hash>.css` (feeds
|
||||
`rel_url`).
|
||||
|
||||
Properties:
|
||||
|
||||
- **Works standalone**, no `tool` needed: `{{ "css/main.css" | fingerprint }}`
|
||||
is plain cache-busting of a hand-written file — probably the most common use.
|
||||
- **Position decides what's hashed** — it hashes current bytes, so it belongs
|
||||
**last** in the chain. No terminal-only special-casing; the rule is just "put
|
||||
it last."
|
||||
- **Hash format:** `name.<hash>.ext` (CDN-friendly), 8 hex chars of SHA-256.
|
||||
|
||||
## Tailwind is not special
|
||||
|
||||
It scans your **source** (`layouts/`, `content/`, partials) — which exists at
|
||||
render time — so it runs synchronously like sass. Its `thor.json` entry just
|
||||
needs content globs pointing at the source tree (or v4 auto-detection). The
|
||||
**one** asymmetry is watch invalidation: sass re-runs when `main.scss` changes;
|
||||
Tailwind must re-run when *any* scanned source changes. That's a cache-key
|
||||
breadth difference, not an architectural one. The cost of scanning source vs.
|
||||
output is losing dynamically-assembled class tokens (`col-{{n}}`) — a Tailwind
|
||||
anti-pattern whose sanctioned fix is a safelist, not warping the build.
|
||||
|
||||
## Interaction: `copy_assets_dir`
|
||||
|
||||
Today thor copies everything under `assets/` to the output dir. Any source
|
||||
consumed by a pipe must be **excluded from that plain copy** — otherwise you'd
|
||||
ship the raw `main.scss` (or a dead unhashed `main.css` alongside the hashed
|
||||
one). The pipe owns the write for assets it touches.
|
||||
|
||||
## Implementation steps (separable)
|
||||
|
||||
1. **`| fingerprint`** — no subprocess, no config. Hash + write + return URL,
|
||||
plus the `copy_assets_dir` exclusion. Smallest blast radius, independently
|
||||
useful, ships first.
|
||||
2. **`| tool <name>`** — `thor.json` tools table, the `$in`/`$out`/stdin/stdout
|
||||
adapter, subprocess + exit-code/stderr handling, extension derivation,
|
||||
memoization. Composes with step 1.
|
||||
|
||||
## Open questions for implementation
|
||||
|
||||
- **Watch invalidation:** simplest correct v1 recomputes the memo each build
|
||||
(cheap for the common single-asset case). A cross-build skip keyed on source
|
||||
content/mtime is a later optimization — and Tailwind's broad content-glob key
|
||||
is where it matters most.
|
||||
- **Source-map sidecars:** sass/lightning emit `.map` files next to `$out`;
|
||||
disable in the template or ignore the sidecar. Minor.
|
||||
- **Error policy default:** hard-fail the build vs. warn-and-skip on tool
|
||||
failure/absence. Lean fail; possibly configurable later.
|
||||
@@ -0,0 +1,96 @@
|
||||
# Visualizing & Managing `thor` Binary Size
|
||||
|
||||
How to inspect what makes up the compiled `thor` binary, and how to keep that
|
||||
inspection useful once we bundle layout/asset files into the binary.
|
||||
|
||||
## Quick inspection (tools already available)
|
||||
|
||||
The binary is a not-stripped ELF, so it ships symbols we can break down.
|
||||
|
||||
```bash
|
||||
# Section totals (text / data / bss)
|
||||
size ./thor
|
||||
|
||||
# Top symbols by size
|
||||
nm --print-size --size-sort --radix=d ./thor \
|
||||
| awk '{sz=$2+0; sub(/^[0-9a-fA-F]+ +[0-9]+ +[A-Za-z] +/,""); printf "%8d %s\n", sz, $0}' \
|
||||
| sort -rn | head -25
|
||||
|
||||
# Aggregate symbol sizes by Odin module namespace (foo::bar -> "foo")
|
||||
nm --print-size --size-sort --radix=d ./thor \
|
||||
| awk '{sz=$2+0; rest=$0; sub(/^[0-9a-fA-F]+ +[0-9]+ +[A-Za-z] +/,"",rest);
|
||||
ns=rest; if (rest ~ /::/){sub(/::.*/,"",ns)}
|
||||
else if (rest ~ /^ts_/){ns="treesitter (ts_*)"} else {ns="<other/runtime>"};
|
||||
s[ns]+=sz} END{for (k in s) printf "%9d %s\n", s[k], k}' \
|
||||
| sort -rn | head -25
|
||||
```
|
||||
|
||||
As of this writing the top contributors were: tree-sitter tables (~118 KB),
|
||||
`main` (~96 KB), runtime/other (~93 KB), `mustache` (~67 KB), then `fmt` /
|
||||
`encoding_json` from the stdlib.
|
||||
|
||||
## Better visualization: `bloaty`
|
||||
|
||||
`bloaty` (Google's binary-size profiler) is the right tool. Not installed, but
|
||||
runnable via Nix:
|
||||
|
||||
```bash
|
||||
bloaty ./thor # by section
|
||||
bloaty ./thor -d symbols -n 30 # by symbol, hierarchical
|
||||
bloaty ./thor -d sections,symbols # section then symbols within each
|
||||
# via nix (no install):
|
||||
nix run nixpkgs#bloaty -- ./thor -d symbols -n 30
|
||||
```
|
||||
|
||||
For a stripped-size estimate:
|
||||
|
||||
```bash
|
||||
readelf -S ./thor | grep -E 'debug|symtab|strtab' # strippable overhead
|
||||
strip -s ./thor -o thor.stripped && size thor.stripped
|
||||
```
|
||||
|
||||
Odin also supports smaller builds: `odin build . -o:size -no-bounds-check`.
|
||||
|
||||
## IMPORTANT: keeping bloaty useful once we bundle layouts into the binary
|
||||
|
||||
Today the VFS (`vfs.odin`) reads layout/asset files lazily from disk at runtime
|
||||
(`VFS_Entry.fs_path` + `os.read_entire_file_from_path`); `VFS_Entry.data` is
|
||||
never populated at build time. So embedded content is **not** in `./thor` yet.
|
||||
|
||||
When we switch to embedding via Odin `#load(...)`, the file bytes land in the
|
||||
binary's `.rodata` section. Whether bloaty can show *per-file* sizes depends on
|
||||
how we emit the embed:
|
||||
|
||||
- **Section total — always visible.** `.rodata` grows by ~the total bundled
|
||||
size. `bloaty ./thor -d sections` (or `size`) shows the aggregate, and we can
|
||||
diff before/after. This always works.
|
||||
|
||||
- **Per-file breakdown — only if each file is a NAMED symbol.**
|
||||
- A generated `map[string]VFS_Entry` literal full of inline `#load(...)`
|
||||
expressions makes each blob an **anonymous** `.rodata` constant. Bloaty
|
||||
lumps them together — no per-file breakdown.
|
||||
- Assigning each file to its own **package-level variable** makes each a named
|
||||
global symbol, so `bloaty ./thor -d symbols` (and the `nm` recipe above)
|
||||
show each file individually:
|
||||
|
||||
```odin
|
||||
layout_base_html := #load("defaults/layouts/base.html")
|
||||
layout_single_html := #load("defaults/layouts/single.html")
|
||||
// ...then reference these in the VFS map instead of inlining #load.
|
||||
```
|
||||
|
||||
### Decision to bake into the bundling codegen
|
||||
|
||||
When we build the codegen that replaces the runtime `mount_dir` disk walk, have
|
||||
it emit **one named package-level variable per embedded file** (not an anonymous
|
||||
map literal). That single choice is what makes bloaty / `nm` per-file analysis
|
||||
possible — cheap to do up front, painful to retrofit.
|
||||
|
||||
Additional options that help regardless of symbolization:
|
||||
|
||||
1. Build with Odin `-debug` (DWARF) and use `bloaty ./thor -d compileunits` to
|
||||
attribute embedded bytes to the generated source file.
|
||||
2. Since the generator already knows every file and its size, emit a size
|
||||
manifest at bundle time (`path -> bytes`, sorted / treemap). This is the most
|
||||
reliable per-file view and can't drift from reality; bloaty then just
|
||||
confirms the `.rodata` total matches.
|
||||
@@ -0,0 +1,171 @@
|
||||
# Grammar cache thread-safety & allocator lifetime
|
||||
|
||||
Status: **resolved.** All three issues below are fixed. The cache was redesigned
|
||||
so the library caches only the immutable, shareable objects and callers own the
|
||||
per-thread ones. The rest of this document is kept as the rationale.
|
||||
|
||||
## Resolution (what shipped)
|
||||
|
||||
The `Grammar_Cache` (which mixed shareable `language`/`query` with per-thread
|
||||
`parser`/`cursor`) was split:
|
||||
|
||||
- **`Grammar`** holds only the immutable `language` + `query`. It lives in a
|
||||
process-lifetime **`registry`** (`map[string]^Grammar`) guarded by one mutex.
|
||||
- **`grammar(lang)`** is the thread-safe, self-initializing, lazy loader — fixes
|
||||
**#1a** (all map access under the lock) and lets tests run with no setup.
|
||||
- The registry allocator is pinned with `os.heap_allocator()` — fixes **#2**; the
|
||||
cache no longer captures a transient `context.allocator`.
|
||||
- Parsers/cursors are **caller-owned**: `open_parser(g)` mints one, the caller
|
||||
`defer`s `parser_delete` / `query_cursor_delete`. No parser is shared, so
|
||||
**#1b** is gone. `open_parser` is the seam where a future pool / thread-local
|
||||
strategy plugs in without touching call sites.
|
||||
- Deleted: `ensure_parser`, `load_grammar`, `preload_grammar`, `grammar_worker`,
|
||||
`cache_mutex`; `preload_grammars` is now a serial registry warm-up. The minify
|
||||
tests dropped their `parse_mu` serialization and preload scaffolding entirely.
|
||||
|
||||
The **later** decision — pool vs per-thread parser reuse for genuine parallel
|
||||
parsing — is still open, but intentionally so: it only matters once we parse in
|
||||
parallel, and it now changes only `open_parser`, not the rest of the library.
|
||||
The original investigation follows.
|
||||
|
||||
## How we got here
|
||||
|
||||
While adding a feature to minify inline `<style>` blocks (route the CSS body
|
||||
through `minify_css`), the new unit tests exercised `minify_html` / `minify_css`
|
||||
for the first time. Those are the first tests to touch the tree-sitter parser
|
||||
cache, and under the multi-threaded test runner (default 16 threads) they failed
|
||||
**intermittently** with a rotating cast of symptoms:
|
||||
|
||||
- `panic: unable to insert into a map`
|
||||
- tree-sitter internal aborts: `ts_subtree_retain: Assertion 'self.ptr->ref_count > 0' failed`, `_array__erase: Assertion 'index < *size' failed`
|
||||
- `malloc(): unaligned tcache chunk detected`
|
||||
- assertions passing on the wrong data (e.g. minified output missing `<style>`)
|
||||
|
||||
The flakiness comes from **three distinct problems**, two thread-safety and one
|
||||
allocator-lifetime. They are orthogonal — fixing one does not fix the others.
|
||||
|
||||
## The three issues
|
||||
|
||||
### #1a — concurrent cache **writes** (data race)
|
||||
|
||||
`ensure_parser` (`treesitter.odin`) reads and writes `grammar_store.cache`
|
||||
with **no lock**. `cache_mutex` exists but is only taken on the parallel preload
|
||||
path (`grammar_worker`), never in the lazy `ensure_parser` / `load_grammar`
|
||||
paths. Two threads first-touching `"html"`/`"css"` at once corrupt the map.
|
||||
|
||||
### #1b — concurrent parser **use** (data race)
|
||||
|
||||
There is exactly **one cached `Grammar_Cache` (and one `parser`) per language**.
|
||||
When two threads both call `minify_html`, they fetch the *same* `gc.parser` and
|
||||
call `parser_parse_string` on it simultaneously. A tree-sitter `TSParser`
|
||||
mutates its internal subtree pool during a parse, so concurrent parses corrupt
|
||||
it — that's the `ts_subtree_retain` / `_array__erase` aborts.
|
||||
|
||||
This is **not fixed by preloading/"loading html+css statically."** Loading gives
|
||||
you *one shared* parser; the race is about *using* that one parser from two
|
||||
threads. It is fixed only by serializing the parse or giving each thread its own
|
||||
parser.
|
||||
|
||||
### #2 — allocator lifetime (not a race)
|
||||
|
||||
`init_persistent` captures `context.allocator` into the process-lifetime
|
||||
`grammar_store`. In `main` that happens to be the heap (it runs before the site
|
||||
arena is installed at `main.odin`), so production is fine. But whatever
|
||||
allocator is live at that instant becomes the cache's owner forever:
|
||||
|
||||
- Under the **test runner**, `context.allocator` is a **per-test tracking
|
||||
allocator** that is torn down when that test ends → the global cache is left
|
||||
pointing at dead memory (→ `unable to insert into a map`, garbage reads).
|
||||
- A **lazy** first call during rendering would capture the **site arena**
|
||||
(`context.allocator` after `main.odin` swaps it in), which is destroyed every
|
||||
watch-mode rebuild → dangling parsers across builds.
|
||||
|
||||
This would break even single-threaded; threading just decides *which* test wins
|
||||
the init race.
|
||||
|
||||
## tree-sitter's threading model (the root fact)
|
||||
|
||||
tree-sitter objects have specific, documented threading rules:
|
||||
|
||||
| Object | Shareable across threads? | Rule |
|
||||
|---|---|---|
|
||||
| `TSLanguage` (`tree_sitter_html()` etc.) | Yes | Immutable; share freely. |
|
||||
| `TSParser` (`gc.parser`) | **No** | One per thread; never parse on one parser from two threads. |
|
||||
| `TSTree` | Only via `ts_tree_copy` | Copy before using on another thread. |
|
||||
| `TSQuery` (`gc.query`) | Yes | Immutable after compile. |
|
||||
| `TSQueryCursor` (`gc.cursor`) | **No** | One per thread, like the parser. |
|
||||
|
||||
This explains the existing design:
|
||||
|
||||
- The preload fan-out (`preload_grammar` / `grammar_worker`) creates a parser
|
||||
**per worker** and only publishes the finished `Grammar_Cache` under
|
||||
`cache_mutex` — because parsers can't be shared during construction/use.
|
||||
- `ensure_parser` + the shared cached parser is fine in production **only
|
||||
because rendering is single-threaded**.
|
||||
|
||||
A cached `Grammar_Cache` mixes shareable (`language`, `query`) and per-thread
|
||||
(`parser`, `cursor`) objects in one shared struct, which makes it
|
||||
single-threaded-use by construction — a fact that was previously undocumented.
|
||||
|
||||
## Where this actually bites
|
||||
|
||||
**Only in tests.** Confirmed by grep:
|
||||
|
||||
- Production parses on one thread: every real caller — `minify.odin` (html/css)
|
||||
and `markdown/highlight.odin` — runs inside `main`'s single-threaded render
|
||||
loop. No concurrent `ensure_parser` / `parser_parse_string`.
|
||||
- The one threaded path in production, `preload_grammars`, is already safe (own
|
||||
parser per worker + `cache_mutex` on the write) and only handles non-builtin
|
||||
grammars.
|
||||
- Only `minify_test.odin` touches tree-sitter from tests, and the test runner
|
||||
runs those on many threads.
|
||||
|
||||
So production is correct **by convention** (single-threaded-from-heap usage),
|
||||
not **by construction** — nothing stops a future parallel render from breaking
|
||||
it, and until now nothing documented the constraint.
|
||||
|
||||
## Chosen direction
|
||||
|
||||
**For now:** keep the single global grammar cache and make it safe by
|
||||
**serializing the non-thread-safe behavior with a mutex** — only one thread uses
|
||||
a parser at a time. This is simple, matches how production already behaves, and
|
||||
unblocks the tests.
|
||||
|
||||
**Later (TODO):** decide whether true multi-threaded parsing is worth it for
|
||||
performance. If yes, give each thread its own parser + cursor (as the preload
|
||||
workers already do) rather than sharing one behind a lock.
|
||||
|
||||
## Implementation options (deferred — pick when we resume)
|
||||
|
||||
All assume the allocator fix (#2) below, which is independent and unambiguous.
|
||||
|
||||
- **#2 allocator (do regardless):** pin the heap allocator explicitly —
|
||||
`grammar_store.allocator = runtime.heap_allocator()` (or `os.heap_allocator()`)
|
||||
instead of `context.allocator`. Makes the cache persistent from any context;
|
||||
behavior-identical in production. Optionally make `init_persistent` idempotent
|
||||
and/or an `@(init)` so callers need no ordering.
|
||||
|
||||
Then, for the races:
|
||||
|
||||
- **Option A — library mutex (sustainable):** take `cache_mutex` in
|
||||
`ensure_parser` (fixes #1a) and add a locking `parse` helper that wraps
|
||||
`parser_parse_string` (fixes #1b), wired into the three call sites
|
||||
(`minify.odin` ×2, `markdown/highlight.odin` ×1). Tests then need no
|
||||
scaffolding. Serializes all parsing.
|
||||
|
||||
- **Option B — test-side only (minimal):** leave the library single-threaded by
|
||||
contract; make the tests not violate it. Either fold the minify tests into one
|
||||
`@(test)` (no concurrency at all → no locks), or wrap the `minify_*` calls in a
|
||||
test-local mutex. Production stays lock-free.
|
||||
|
||||
Recommendation captured during the discussion: the allocator one-liner is
|
||||
worth doing immediately; for the races, Option A is the "correct" long-term
|
||||
answer but only necessary if/when we want concurrent parsing — otherwise
|
||||
Option B is enough.
|
||||
|
||||
## Pointers
|
||||
|
||||
- `treesitter/treesitter.odin`: doc comments on `Grammar_Cache`, `cache_mutex`,
|
||||
`init_persistent`, `ensure_parser` reference this file.
|
||||
- Original feature that surfaced this: minify inline `<style>` bodies via
|
||||
`minify_css` (reverted; to be re-landed alongside whichever fix we choose).
|
||||
@@ -87,6 +87,7 @@
|
||||
- [ ] don't use bullshit "sub-tokens", add filters and pipes as proper tokens.
|
||||
- [ ] Create a Location struct that somewhat matches Odin's [Source_Code_Location](https://pkg.odin-lang.org/base/runtime/#Source_Code_Location)?
|
||||
- Note that odin's version doesn't contain the stack trace.
|
||||
- [ ] Link checker / linter that warns if links don't resolve. ("please use {{whatever | rel_url }}").
|
||||
|
||||
### Memory Management
|
||||
|
||||
|
||||
@@ -2,8 +2,48 @@
|
||||
|
||||
## Priority Unclear (Evaluate Later)
|
||||
|
||||
- [ ] Decide the parallel-parsing strategy for the grammar cache (pool vs
|
||||
thread-local parsers) — deferred until the parallel usage shape is clear.
|
||||
- Resolved already: the registry (`grammar()`) is thread-safe and self-
|
||||
initializing, its allocator is pinned to the heap, and parsers/cursors are now
|
||||
caller-owned (`open_parser`) instead of cached — so the library holds no
|
||||
single-thread-use state and the multi-threaded tests pass without locks.
|
||||
- Remaining choice only matters once we actually parse in parallel: reuse
|
||||
parsers via a pool or per-thread storage, plugged in behind `open_parser`
|
||||
without touching call sites. A `TSParser`/`TSQueryCursor` is one-per-thread.
|
||||
- Full write-up: `GRAMMAR_CACHE_THREADING.md`.
|
||||
- [x] Restore parallel grammar preload (serialized in `edb2e64`); `preload_grammars`
|
||||
now fans `build_grammar` across a bounded `core:thread.Pool`.
|
||||
- [ ] Add tests: pool fan-out correctness + `-sanitize:thread` for race-freedom.
|
||||
- [ ] Concurrent-read path: an `RW_Mutex` on `grammar()` only helps a future
|
||||
multi-threaded render, not the load side. Revisit if render goes parallel.
|
||||
- [ ] instead of `warnings: [dynamic]Error` we should use `warnings: [dynamic; 8]Error`
|
||||
- when limit reached, the template fails and stops rendering.
|
||||
- [ ] Brainstorm plugin/tool/(postccs/less/tailwind/sass) support.
|
||||
- [ ] `fingerprint` pipe + contents-based composition. The `tool` pipe currently
|
||||
carries a URL string and each stage publishes a file to the output dir, so
|
||||
`| tool sass | fingerprint` would leave a dead unhashed intermediate. When
|
||||
fingerprint lands, move the pipe value from a URL string to a contents/
|
||||
Resource (bytes + logical name) where only the terminal stage publishes;
|
||||
a Resource must stringify to its URL so existing bare-`tool` templates keep
|
||||
working. stdin/stdout tools compose best there (bytes stay in memory), which
|
||||
is why stdin/stdout is the documented preferred form over `{{in}}`/`{{out}}`.
|
||||
- [ ] Reconsider `"menus": {}` opt-out: it currently discards all page frontmatter
|
||||
menus (with only a warning). Decide whether empty config should instead just
|
||||
suppress the auto-generated menu while still honoring frontmatter.
|
||||
- [ ] asset css files not automatically included in site.
|
||||
- should we at least warn about them?
|
||||
- [ ] Review the whole `copy_assets_dir` pipeline. It copies every file under
|
||||
`assets/` to the output dir blindly, whether or not anything references it,
|
||||
and now also skips dests that already exist (a blunt collision guard so the
|
||||
raw copy doesn't clobber `tool`-pipe output written earlier in the build).
|
||||
Reconsider tracking which assets are actually used/published instead.
|
||||
- [ ] warn if stylesheet url starts with /assets (and is therefore invalid).
|
||||
- [ ] warn when heading levels are skipped in a toc.
|
||||
- [ ] extract `[TOC]` from `.md` body (or at least from the start.)
|
||||
- [ ] add `--production` flag
|
||||
- set `minify` to true
|
||||
- warn when `base_url` unset
|
||||
|
||||
## General
|
||||
|
||||
@@ -43,6 +83,9 @@ Remember to polish existing features before moving on to new ones.
|
||||
- do we want to do this or just use DIAGNOSTIC_TESTS.md as the tests?
|
||||
- [ ] also update to use file logger so they don't clog up the output
|
||||
- [ ] narrow highlight in diagnostic to relevent part of line.
|
||||
- [ ] page.url and page.permalink are confusing names. Switch to url + page_url
|
||||
- There is confusion over how to do this and whethor or not `| rel_url` should be needed to render these.
|
||||
Requires **careful thought**.
|
||||
|
||||
## Documentation
|
||||
|
||||
|
||||
+24
-1
@@ -5,7 +5,20 @@ import "core:log"
|
||||
import "core:os"
|
||||
import "core:strings"
|
||||
|
||||
import ts "treesitter"
|
||||
|
||||
copy_assets_dir :: proc(vfs: ^VFS, output_dir: string, features: bit_set[Feature]) {
|
||||
// One CSS parser, reused for every .css asset (single-threaded). Owned here.
|
||||
css_parser: ts.Parser
|
||||
if .Minify in features {
|
||||
if g, ok := ts.grammar("css"); ok {
|
||||
css_parser = ts.open_parser(g)
|
||||
}
|
||||
}
|
||||
defer if css_parser != nil {
|
||||
ts.parser_delete(css_parser)
|
||||
}
|
||||
|
||||
for virtual_path, entry in vfs.files {
|
||||
if !strings.has_prefix(virtual_path, "assets/") {
|
||||
continue
|
||||
@@ -14,6 +27,12 @@ copy_assets_dir :: proc(vfs: ^VFS, output_dir: string, features: bit_set[Feature
|
||||
rel := virtual_path[len("assets/"):]
|
||||
dest := fmt.tprintf("%s/%s", output_dir, rel)
|
||||
|
||||
// Don't clobber a file already written this build (e.g. the output of a
|
||||
// `tool` pipe, which publishes to the same dest path as its source).
|
||||
if os.exists(dest) {
|
||||
continue
|
||||
}
|
||||
|
||||
if idx := strings.last_index(dest, "/"); idx >= 0 {
|
||||
if err := os.make_directory_all(dest[:idx]); err != nil && err != .Exist {
|
||||
log.warnf("cannot create %s: %v", dest[:idx], err)
|
||||
@@ -24,7 +43,11 @@ copy_assets_dir :: proc(vfs: ^VFS, output_dir: string, features: bit_set[Feature
|
||||
if .Minify in features && strings.has_suffix(rel, ".css") {
|
||||
data, ok := vfs_get(vfs, virtual_path)
|
||||
if ok {
|
||||
write_file(dest, minify_css(string(data)))
|
||||
if m, mok := minify_css(css_parser, string(data)); mok {
|
||||
write_file(dest, m)
|
||||
} else {
|
||||
write_file(dest, string(data))
|
||||
}
|
||||
}
|
||||
} else if entry.data != nil {
|
||||
if err := os.write_entire_file(dest, entry.data); err != nil {
|
||||
|
||||
+54
-1
@@ -71,6 +71,9 @@ site_load_content :: proc(site: ^Site) {
|
||||
|
||||
for &page in site.pages {
|
||||
page.url = fmt.tprintf("%s%s", site.base_url, page.permalink)
|
||||
if site.base_path != "" {
|
||||
page.content = rewrite_root_urls(page.content, site.base_path, site_allocator(site))
|
||||
}
|
||||
}
|
||||
|
||||
build_menus(site)
|
||||
@@ -289,6 +292,57 @@ load_page :: proc(
|
||||
return
|
||||
}
|
||||
|
||||
// rewrite_root_urls prefixes root-relative href/src attribute values with
|
||||
// base_path so links authored in content (e.g. [x](/ideas)) resolve under a
|
||||
// subpath deployment. Only values beginning with a single '/' are rewritten;
|
||||
// absolute ("http://…"), protocol-relative ("//cdn…"), fragment ("#x"),
|
||||
// mailto:, and document-relative ("foo/bar", "./foo") values all fail that
|
||||
// test and pass through untouched. Assumes cmark's double-quoted output.
|
||||
rewrite_root_urls :: proc(html, base_path: string, allocator := context.allocator) -> string {
|
||||
b := strings.builder_make(allocator)
|
||||
strings.builder_grow(&b, len(html))
|
||||
|
||||
i := 0
|
||||
for i < len(html) {
|
||||
n := attr_opener_at(html, i) // ` href="` or ` src="`
|
||||
if n == 0 {
|
||||
strings.write_byte(&b, html[i])
|
||||
i += 1
|
||||
continue
|
||||
}
|
||||
strings.write_string(&b, html[i:i + n]) // copy the opener verbatim
|
||||
i += n
|
||||
|
||||
end := strings.index_byte(html[i:], '"')
|
||||
if end < 0 { // malformed; bail cleanly
|
||||
strings.write_string(&b, html[i:])
|
||||
break
|
||||
}
|
||||
value := html[i:i + end]
|
||||
if url_is_root_relative(value) {
|
||||
strings.write_string(&b, base_path)
|
||||
}
|
||||
strings.write_string(&b, value) // closing quote copied next loop
|
||||
i += end
|
||||
}
|
||||
return strings.to_string(b)
|
||||
}
|
||||
|
||||
// url_is_root_relative reports whether value is a site-root path ("/faq/") as
|
||||
// opposed to protocol-relative ("//cdn") or anything not starting with '/'.
|
||||
url_is_root_relative :: proc(value: string) -> bool {
|
||||
return len(value) >= 1 && value[0] == '/' && !strings.has_prefix(value, "//")
|
||||
}
|
||||
|
||||
// attr_opener_at returns the length of a ` href="` / ` src="` attribute opener
|
||||
// at s[i:], or 0. The leading space anchors the match to a real HTML attribute
|
||||
// so we don't rewrite the strings inside link text or code blocks.
|
||||
attr_opener_at :: proc(s: string, i: int) -> int {
|
||||
if strings.has_prefix(s[i:], " href=\"") do return 7
|
||||
if strings.has_prefix(s[i:], " src=\"") do return 6
|
||||
return 0
|
||||
}
|
||||
|
||||
is_content_file :: proc(name: string) -> bool {
|
||||
return strings.has_suffix(name, ".md") || strings.has_suffix(name, ".html")
|
||||
}
|
||||
@@ -300,4 +354,3 @@ strip_extension :: proc(name: string) -> string {
|
||||
}
|
||||
return name[:dot]
|
||||
}
|
||||
|
||||
|
||||
@@ -5,7 +5,7 @@
|
||||
{{&content}}
|
||||
</header>
|
||||
<ul>
|
||||
{{#pages}} <li><a href="{{permalink}}">{{&title}}</a><span>{{#date}}<time
|
||||
{{#pages}} <li><a href="{{permalink | rel_url}}">{{&title}}</a><span>{{#date}}<time
|
||||
datetime="{{date}}">{{date | format}}</time>{{/date}}</span>
|
||||
</li>
|
||||
{{/pages}}
|
||||
|
||||
@@ -1,9 +1,9 @@
|
||||
<header>
|
||||
<nav>
|
||||
<ul>
|
||||
<li><a href="/">{{> home-link}}</a></li>
|
||||
<li><a href="{{site.base_path}}/">{{> home-link}}</a></li>
|
||||
{{#menus.main}}
|
||||
<li><a href="{{url}}">{{name}}</a></li>
|
||||
<li><a href="{{url | rel_url}}">{{name}}</a></li>
|
||||
{{/menus.main}}
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
@@ -1,4 +1,13 @@
|
||||
{{^params.stylesheets}}
|
||||
<style>
|
||||
/* Nav */
|
||||
header nav ul {
|
||||
list-style: none;
|
||||
display: flex;
|
||||
gap: 1rem;
|
||||
padding: 0;
|
||||
}
|
||||
|
||||
/* Base layout */
|
||||
body {
|
||||
margin-left: auto;
|
||||
@@ -220,4 +229,66 @@ label.margin-toggle:not(.sidenote-number) {
|
||||
}
|
||||
}
|
||||
</style>
|
||||
{{#params.stylesheets}}<link rel="stylesheet" href="{{.}}">{{/params.stylesheets}}
|
||||
{{/params.stylesheets}}
|
||||
<style>
|
||||
/* Syntax highlighting. Build-time <span class="hl-*"> wrappers only appear
|
||||
inside fenced code blocks. The base class (e.g. hl-keyword) is always
|
||||
emitted alongside any dotted variant (hl-keyword-control), so styling the
|
||||
base names covers every capture. Re-theme by overriding the --hl-* vars. */
|
||||
:root {
|
||||
--hl-bg: #282c34;
|
||||
--hl-fg: #abb2bf;
|
||||
--hl-comment: #5c6370;
|
||||
--hl-keyword: #c678dd;
|
||||
--hl-string: #98c379;
|
||||
--hl-escape: #56b6c2;
|
||||
--hl-number: #d19a66;
|
||||
--hl-function: #61afef;
|
||||
--hl-type: #e5c07b;
|
||||
--hl-variable: #e06c75;
|
||||
--hl-attribute: #d19a66;
|
||||
}
|
||||
|
||||
pre:has(> code [class^="hl-"]) {
|
||||
background: var(--hl-bg);
|
||||
color: var(--hl-fg);
|
||||
padding: 0.75rem 1rem;
|
||||
border-radius: 4px;
|
||||
}
|
||||
|
||||
.hl-comment { color: var(--hl-comment); font-style: italic; }
|
||||
.hl-keyword,
|
||||
.hl-conditional,
|
||||
.hl-repeat,
|
||||
.hl-exception,
|
||||
.hl-include,
|
||||
.hl-storageclass { color: var(--hl-keyword); }
|
||||
.hl-operator,
|
||||
.hl-punctuation { color: var(--hl-fg); }
|
||||
.hl-string,
|
||||
.hl-character { color: var(--hl-string); }
|
||||
.hl-escape,
|
||||
.hl-string-escape,
|
||||
.hl-string-special { color: var(--hl-escape); }
|
||||
.hl-number,
|
||||
.hl-float,
|
||||
.hl-boolean,
|
||||
.hl-constant { color: var(--hl-number); }
|
||||
.hl-function,
|
||||
.hl-method,
|
||||
.hl-constructor { color: var(--hl-function); }
|
||||
.hl-type,
|
||||
.hl-namespace,
|
||||
.hl-module { color: var(--hl-type); }
|
||||
.hl-variable,
|
||||
.hl-property,
|
||||
.hl-field,
|
||||
.hl-parameter,
|
||||
.hl-tag,
|
||||
.hl-symbol { color: var(--hl-variable); }
|
||||
.hl-attribute,
|
||||
.hl-label,
|
||||
.hl-preproc { color: var(--hl-attribute); }
|
||||
</style>
|
||||
{{#params.stylesheets}}<link rel="stylesheet" href="{{.|rel_url}}">
|
||||
{{/params.stylesheets}}
|
||||
|
||||
@@ -7,7 +7,7 @@
|
||||
<section>
|
||||
<h2>{{key}}</h2>
|
||||
<ul>
|
||||
{{#items}} <li><a href="{{permalink}}">{{&title}}</a><span>{{#date}}<time
|
||||
{{#items}} <li><a href="{{permalink | rel_url}}">{{&title}}</a><span>{{#date}}<time
|
||||
datetime="{{date}}">{{date | format}}</time>{{/date}}</span>
|
||||
</li>
|
||||
{{/items}}
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
package main
|
||||
|
||||
import "base:runtime"
|
||||
import "core:fmt"
|
||||
import "core:strings"
|
||||
import "core:time"
|
||||
@@ -110,8 +111,7 @@ generate_sitemap :: proc(site: ^Site) -> string {
|
||||
return strings.to_string(sb)
|
||||
}
|
||||
|
||||
// TODO: Leaks
|
||||
format_rfc822 :: proc(iso: string) -> string {
|
||||
format_rfc822 :: proc(iso: string, allocator := context.temp_allocator) -> string {
|
||||
if len(iso) < 19 {
|
||||
// TODO: should indicate error somehow
|
||||
return iso
|
||||
@@ -124,21 +124,66 @@ format_rfc822 :: proc(iso: string) -> string {
|
||||
buf: [8]byte
|
||||
t := time.to_string_hms(date, buf[:])
|
||||
|
||||
// RFC-822 zone: sign followed by two-digit hours and minutes (e.g. +0000,
|
||||
// +0530, -0800). offset is in minutes.
|
||||
sign := '+'
|
||||
off := offset
|
||||
if off < 0 {
|
||||
sign = '-'
|
||||
off = -off
|
||||
}
|
||||
|
||||
return fmt.aprintf(
|
||||
"%s, %02d %s %d %s %3d%2d",
|
||||
"%s, %02d %s %d %s %c%02d%02d",
|
||||
weekday[:3],
|
||||
time.day(date),
|
||||
month[:3],
|
||||
time.year(date),
|
||||
t,
|
||||
offset / 60,
|
||||
offset % 60,
|
||||
sign,
|
||||
off / 60,
|
||||
off % 60,
|
||||
allocator = allocator,
|
||||
)
|
||||
}
|
||||
|
||||
xml_escape :: proc(s: string) -> string {
|
||||
r, _ := strings.replace_all(s, "&", "&")
|
||||
r, _ = strings.replace_all(r, "<", "<")
|
||||
r, _ = strings.replace_all(r, ">", ">")
|
||||
return r
|
||||
// xml_escape escapes the XML metacharacters '&', '<' and '>' in s. Only these
|
||||
// ASCII characters are touched; all other bytes (including UTF-8 multibyte
|
||||
// sequences) pass through verbatim, so s must already be valid UTF-8.
|
||||
//
|
||||
// The result is allocated in `allocator`, which defaults to
|
||||
// context.temp_allocator — the returned string is only valid until the next
|
||||
// temp allocator reset, so callers must not retain it across frames.
|
||||
xml_escape :: proc(
|
||||
s: string,
|
||||
allocator := context.temp_allocator,
|
||||
) -> (
|
||||
escaped: string,
|
||||
err: runtime.Allocator_Error,
|
||||
) #optional_allocator_error {
|
||||
sb := strings.builder_make_len_cap(0, len(s), allocator) or_return
|
||||
|
||||
// Single pass over the bytes, copying runs of ordinary text in bulk and
|
||||
// emitting an entity for each metacharacter. The switch is both the
|
||||
// membership test and the replacement lookup, so no per-match scan.
|
||||
start := 0
|
||||
for i := 0; i < len(s); i += 1 {
|
||||
esc: string
|
||||
switch s[i] {
|
||||
case '&':
|
||||
esc = "&"
|
||||
case '<':
|
||||
esc = "<"
|
||||
case '>':
|
||||
esc = ">"
|
||||
case:
|
||||
continue // ordinary byte: copied as part of the next run
|
||||
}
|
||||
strings.write_string(&sb, s[start:i]) // run before the metacharacter
|
||||
strings.write_string(&sb, esc)
|
||||
start = i + 1
|
||||
}
|
||||
strings.write_string(&sb, s[start:]) // tail
|
||||
return strings.to_string(sb), nil
|
||||
}
|
||||
|
||||
|
||||
+166
@@ -0,0 +1,166 @@
|
||||
#+test
|
||||
package main
|
||||
|
||||
import "core:strings"
|
||||
import "core:testing"
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// xml_escape
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
@(test)
|
||||
test_xml_escape_passthrough :: proc(t: ^testing.T) {
|
||||
testing.expect_value(t, xml_escape(""), "")
|
||||
testing.expect_value(t, xml_escape("hello world"), "hello world")
|
||||
// UTF-8 multibyte bytes are not metacharacters and pass through untouched.
|
||||
testing.expect_value(t, xml_escape("héllo — 世界"), "héllo — 世界")
|
||||
}
|
||||
|
||||
@(test)
|
||||
test_xml_escape_entities :: proc(t: ^testing.T) {
|
||||
testing.expect_value(t, xml_escape("&"), "&")
|
||||
testing.expect_value(t, xml_escape("<"), "<")
|
||||
testing.expect_value(t, xml_escape(">"), ">")
|
||||
// Adjacent specials, and specials at the very start/end of the string.
|
||||
testing.expect_value(t, xml_escape("&<>"), "&<>")
|
||||
testing.expect_value(t, xml_escape("<a>"), "<a>")
|
||||
}
|
||||
|
||||
@(test)
|
||||
test_xml_escape_mixed :: proc(t: ^testing.T) {
|
||||
testing.expect_value(t, xml_escape("a<b>c&d"), "a<b>c&d")
|
||||
testing.expect_value(
|
||||
t,
|
||||
xml_escape("<p>Tom & Jerry</p>"),
|
||||
"<p>Tom & Jerry</p>",
|
||||
)
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// format_rfc822
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
@(test)
|
||||
test_format_rfc822_short_passthrough :: proc(t: ^testing.T) {
|
||||
// Inputs shorter than a full ISO-8601 timestamp are returned unchanged
|
||||
// (and are the input slice itself, so they must not be deleted).
|
||||
testing.expect_value(t, format_rfc822("2024"), "2024")
|
||||
testing.expect_value(t, format_rfc822(""), "")
|
||||
}
|
||||
|
||||
@(test)
|
||||
test_format_rfc822_full_date :: proc(t: ^testing.T) {
|
||||
// Results are temp-allocated (reset by the test runner), so no delete.
|
||||
// 2024-01-15 is a Monday; wall-clock time is preserved and the zone is
|
||||
// rendered as an RFC-822 ±HHMM offset.
|
||||
testing.expect_value(
|
||||
t,
|
||||
format_rfc822("2024-01-15T10:30:00Z"),
|
||||
"Mon, 15 Jan 2024 10:30:00 +0000",
|
||||
)
|
||||
testing.expect_value(
|
||||
t,
|
||||
format_rfc822("2024-01-15T10:30:00+05:30"),
|
||||
"Mon, 15 Jan 2024 10:30:00 +0530",
|
||||
)
|
||||
testing.expect_value(
|
||||
t,
|
||||
format_rfc822("2024-01-15T10:30:00-08:00"),
|
||||
"Mon, 15 Jan 2024 10:30:00 -0800",
|
||||
)
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// generate_rss
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
@(test)
|
||||
test_generate_rss_structure :: proc(t: ^testing.T) {
|
||||
site: Site
|
||||
site.title = "My & Site"
|
||||
site.description = "Desc <here>"
|
||||
site.base_url = "https://example.com"
|
||||
site.pages = make(#soa[dynamic]Page)
|
||||
defer delete_soa(site.pages)
|
||||
|
||||
// A real content page.
|
||||
append(
|
||||
&site.pages,
|
||||
Page {
|
||||
section = "posts",
|
||||
title = "First <Post>",
|
||||
url = "https://example.com/posts/first/",
|
||||
content = "<p>Hello & welcome</p>",
|
||||
},
|
||||
)
|
||||
// The home index page (section == "" && _is_index) must be skipped.
|
||||
append(
|
||||
&site.pages,
|
||||
Page {
|
||||
section = "",
|
||||
_is_index = true,
|
||||
title = "HOME_SHOULD_BE_SKIPPED",
|
||||
url = "https://example.com/",
|
||||
content = "home",
|
||||
},
|
||||
)
|
||||
// A second content page.
|
||||
append(
|
||||
&site.pages,
|
||||
Page {
|
||||
section = "posts",
|
||||
title = "Second Post",
|
||||
url = "https://example.com/posts/second/",
|
||||
content = "plain body",
|
||||
},
|
||||
)
|
||||
|
||||
rss := generate_rss(&site)
|
||||
defer delete(rss)
|
||||
|
||||
// Channel header: site title/description are XML-escaped, link gets a
|
||||
// trailing slash, and the atom self-link points at index.xml.
|
||||
testing.expect(t, strings.contains(rss, "<title>My & Site</title>"), rss)
|
||||
testing.expect(
|
||||
t,
|
||||
strings.contains(rss, "<description>Desc <here></description>"),
|
||||
rss,
|
||||
)
|
||||
testing.expect(t, strings.contains(rss, "<link>https://example.com/</link>"), rss)
|
||||
testing.expect(
|
||||
t,
|
||||
strings.contains(rss, `<atom:link href="https://example.com/index.xml"`),
|
||||
rss,
|
||||
)
|
||||
|
||||
// Exactly two <item>s — the home index page is skipped.
|
||||
testing.expect_value(t, strings.count(rss, "<item>"), 2)
|
||||
testing.expect(t, !strings.contains(rss, "HOME_SHOULD_BE_SKIPPED"), rss)
|
||||
|
||||
// First item: title and body are XML-escaped, link is verbatim.
|
||||
testing.expect(t, strings.contains(rss, "<title>First <Post></title>"), rss)
|
||||
testing.expect(
|
||||
t,
|
||||
strings.contains(
|
||||
rss,
|
||||
"<description><p>Hello & welcome</p></description>",
|
||||
),
|
||||
rss,
|
||||
)
|
||||
testing.expect(
|
||||
t,
|
||||
strings.contains(rss, "<link>https://example.com/posts/first/</link>"),
|
||||
rss,
|
||||
)
|
||||
|
||||
// Undated pages get the sentinel pubDate (no date formatting/allocation).
|
||||
testing.expect(
|
||||
t,
|
||||
strings.contains(rss, "<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>"),
|
||||
rss,
|
||||
)
|
||||
|
||||
// Well-formed at the boundaries.
|
||||
testing.expect(t, strings.has_prefix(rss, `<?xml version="1.0"`), rss)
|
||||
testing.expect(t, strings.has_suffix(rss, "</channel>\n</rss>"), rss)
|
||||
}
|
||||
@@ -167,6 +167,12 @@
|
||||
gdb
|
||||
perf
|
||||
|
||||
bloaty # du for binaries?
|
||||
|
||||
# temp
|
||||
hugo
|
||||
pandoc
|
||||
|
||||
# IDE
|
||||
unstable.helix
|
||||
claude-code
|
||||
|
||||
+261
-134
@@ -4,140 +4,288 @@ import cm "vendor:commonmark"
|
||||
|
||||
import "core:strings"
|
||||
|
||||
// DefList_Entry represents a single term-definition pair in a definition list.
|
||||
DefList_Entry :: struct {
|
||||
term: string,
|
||||
definition: string,
|
||||
}
|
||||
|
||||
// convert_deflists scans markdown text for definition list patterns and
|
||||
// converts them to <dl><dt><dd> HTML blocks before cmark processing.
|
||||
//
|
||||
// A definition line starts with optional whitespace followed by a colon and
|
||||
// a space. The term is the nearest preceding non-blank line (immediately or
|
||||
// within one blank line). Consecutive term+definition pairs are grouped into
|
||||
// a single <dl> block.
|
||||
// A definition line starts with fewer than four columns of leading whitespace
|
||||
// followed by a colon and a space (or end-of-line). The term is the nearest
|
||||
// preceding non-blank line (immediately or within one blank line). Consecutive
|
||||
// term+definition pairs are grouped into a single <dl> block, and consecutive
|
||||
// definition lines under one term become multiple <dd> elements.
|
||||
//
|
||||
// Terms and definitions are rendered through cmark individually so that
|
||||
// inline markdown (code, links, emphasis) is processed.
|
||||
// The scanner walks the body one line at a time using byte offsets: it never
|
||||
// splits the input, allocates nothing per line, and copies non-deflist text
|
||||
// through verbatim. Matching is disabled inside fenced code blocks. Terms and
|
||||
// definitions are rendered through cmark individually so that inline markdown
|
||||
// (code, links, emphasis) is processed.
|
||||
// Line_Kind classifies a line for convert_deflists's group-finding scan. Other
|
||||
// covers fenced-code delimiters and their contents, where definition syntax is
|
||||
// inert.
|
||||
Line_Kind :: enum {
|
||||
None,
|
||||
Blank,
|
||||
Term,
|
||||
Def,
|
||||
Other,
|
||||
}
|
||||
|
||||
convert_deflists :: proc(body: string, allocator := context.allocator) -> string {
|
||||
lines := strings.split(body, "\n", allocator = context.temp_allocator)
|
||||
// Fast path: a body with no definition line at all cannot contain a
|
||||
// definition list, so return it untouched — skipping the builder allocation
|
||||
// and the classifying scan below. This is the common case (most pages have
|
||||
// no definition lists), and the guard scan is leaner than the full loop.
|
||||
if !has_def_line(body) {
|
||||
return body
|
||||
}
|
||||
|
||||
sb := strings.builder_make(context.temp_allocator)
|
||||
|
||||
first := true
|
||||
need_blank := false
|
||||
// emitted_up_to is the byte offset up to which `body` has been copied into
|
||||
// `sb`. It stays at 0 until the first deflist is emitted; if it is still 0
|
||||
// at the end, nothing matched and we can return the body unchanged.
|
||||
emitted_up_to := 0
|
||||
|
||||
i := 0
|
||||
for i < len(lines) {
|
||||
entries, matched, next := try_match_deflist(lines, i)
|
||||
if matched {
|
||||
html := render_deflist(entries)
|
||||
if !first {
|
||||
strings.write_string(&sb, "\n\n")
|
||||
in_fence := false
|
||||
fence_marker: u8 = 0
|
||||
|
||||
// The scan is definition-driven: it walks line by line, remembering the two
|
||||
// previous lines' kinds and offsets, and only does group-finding work when it
|
||||
// reaches a (rare) definition line — whose term is the immediately preceding
|
||||
// line, or the one before a single blank line. This avoids a per-line forward
|
||||
// lookahead: each line is classified once, with no rescanning.
|
||||
prev1_kind, prev2_kind := Line_Kind.None, Line_Kind.None
|
||||
prev1_pos, prev2_pos := 0, 0
|
||||
|
||||
pos := 0
|
||||
for {
|
||||
line, next, ok := line_at(body, pos)
|
||||
if !ok {
|
||||
break
|
||||
}
|
||||
strings.write_string(&sb, html)
|
||||
first = false
|
||||
need_blank = true
|
||||
i = next
|
||||
|
||||
// Classify the line. Definition-list syntax is inert inside fenced code
|
||||
// blocks, so fence delimiters and their contents are classified Other.
|
||||
kind: Line_Kind
|
||||
if in_fence {
|
||||
if is_fence_line(line, fence_marker) {
|
||||
in_fence = false
|
||||
}
|
||||
kind = .Other
|
||||
} else if marker, opened := fence_open(line); opened {
|
||||
in_fence = true
|
||||
fence_marker = marker
|
||||
kind = .Other
|
||||
} else if is_def_line(line) {
|
||||
kind = .Def
|
||||
} else if is_blank_line(line) {
|
||||
kind = .Blank
|
||||
} else {
|
||||
kind = .Term
|
||||
}
|
||||
|
||||
// A definition-list group begins at the term preceding this def line
|
||||
// (immediately, or across a single blank line). Flush pending passthrough
|
||||
// text verbatim, then render the whole group directly into the builder.
|
||||
if kind == .Def {
|
||||
term_pos := -1
|
||||
if prev1_kind == .Term {
|
||||
term_pos = prev1_pos
|
||||
} else if prev1_kind == .Blank && prev2_kind == .Term {
|
||||
term_pos = prev2_pos
|
||||
}
|
||||
if term_pos >= 0 {
|
||||
strings.write_string(&sb, body[emitted_up_to:term_pos])
|
||||
group_end := write_deflist_group(&sb, body, term_pos)
|
||||
// write_deflist_group emits a raw <dl> HTML block with no
|
||||
// trailing newline, so the blank line the author left after the
|
||||
// list is consumed as the </dl> line's terminator rather than a
|
||||
// separating blank line. CommonMark (type-6 HTML block) then
|
||||
// swallows the following block — e.g. a fenced code example —
|
||||
// as raw, unescaped HTML, leaking its markup (a template's
|
||||
// <h1>) into the document. Re-emit a newline so </dl> stands on
|
||||
// its own line and the author's blank line still separates the
|
||||
// block. Only when a blank line actually follows: fused
|
||||
// non-blank fall-through content is left as-is.
|
||||
if line, _, ok := line_at(body, group_end); ok && is_blank_line(line) {
|
||||
strings.write_byte(&sb, '\n')
|
||||
}
|
||||
emitted_up_to = group_end
|
||||
pos = group_end
|
||||
prev1_kind, prev2_kind = .None, .None
|
||||
in_fence = false
|
||||
continue
|
||||
}
|
||||
|
||||
if need_blank {
|
||||
strings.write_string(&sb, "\n\n")
|
||||
need_blank = false
|
||||
} else if !first {
|
||||
strings.write_string(&sb, "\n")
|
||||
}
|
||||
strings.write_string(&sb, lines[i])
|
||||
first = false
|
||||
i += 1
|
||||
}
|
||||
|
||||
prev2_kind, prev2_pos = prev1_kind, prev1_pos
|
||||
prev1_kind, prev1_pos = kind, pos
|
||||
pos = next
|
||||
}
|
||||
|
||||
if emitted_up_to == 0 {
|
||||
// No definition list anywhere; the body passes through unchanged.
|
||||
return body
|
||||
}
|
||||
strings.write_string(&sb, body[emitted_up_to:])
|
||||
return strings.clone(strings.to_string(sb), allocator)
|
||||
}
|
||||
|
||||
// try_match_deflist attempts to match a definition list group starting at
|
||||
// lines[start]. A group is one or more term+definition pairs. Returns the
|
||||
// matched entries, whether a match was found, and the index past the group.
|
||||
try_match_deflist :: proc(
|
||||
lines: []string,
|
||||
start: int,
|
||||
) -> (
|
||||
entries: [dynamic]DefList_Entry,
|
||||
ok: bool,
|
||||
end: int,
|
||||
) {
|
||||
entries = make([dynamic]DefList_Entry, 0, allocator = context.temp_allocator)
|
||||
end = start
|
||||
|
||||
i := start
|
||||
for i < len(lines) {
|
||||
// A term must be non-blank and not itself a def line
|
||||
if is_blank_line(lines[i]) || is_def_line(lines[i]) {
|
||||
break
|
||||
// line_at returns the line beginning at byte offset `pos` (excluding its
|
||||
// trailing newline) along with the offset of the following line. `pos` must be
|
||||
// 0 or one past a '\n'. ok is false once `pos` reaches the end of `body`.
|
||||
line_at :: proc(body: string, pos: int) -> (line: string, next: int, ok: bool) {
|
||||
if pos >= len(body) {
|
||||
return "", pos, false
|
||||
}
|
||||
nl := strings.index_byte(body[pos:], '\n')
|
||||
if nl < 0 {
|
||||
return body[pos:], len(body), true
|
||||
}
|
||||
return body[pos:pos + nl], pos + nl + 1, true
|
||||
}
|
||||
|
||||
// Look for a def line: immediately after or with one blank line
|
||||
def_idx := i + 1
|
||||
if def_idx < len(lines) && is_blank_line(lines[def_idx]) {
|
||||
def_idx += 1
|
||||
}
|
||||
|
||||
if def_idx >= len(lines) || !is_def_line(lines[def_idx]) {
|
||||
break
|
||||
}
|
||||
|
||||
// Found a term + def pair
|
||||
append(
|
||||
&entries,
|
||||
DefList_Entry {
|
||||
term = strings.trim_space(lines[i]),
|
||||
definition = def_content(lines[def_idx]),
|
||||
},
|
||||
)
|
||||
i = def_idx + 1
|
||||
|
||||
// After a pair, check if another pair follows (optionally
|
||||
// separated by one blank line). If so, continue the group.
|
||||
// If not, break without consuming the blank line.
|
||||
if i < len(lines) && is_blank_line(lines[i]) {
|
||||
after_blank := i + 1
|
||||
if after_blank < len(lines) &&
|
||||
!is_blank_line(lines[after_blank]) &&
|
||||
!is_def_line(lines[after_blank]) {
|
||||
// Check whether a def follows this potential term
|
||||
check_def := after_blank + 1
|
||||
if check_def < len(lines) && is_blank_line(lines[check_def]) {
|
||||
check_def += 1
|
||||
}
|
||||
if check_def < len(lines) && is_def_line(lines[check_def]) {
|
||||
i = after_blank
|
||||
continue
|
||||
}
|
||||
}
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
if len(entries) > 0 {
|
||||
ok = true
|
||||
end = i
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
// is_def_line returns true if the line is a definition line:
|
||||
// optional leading whitespace, a colon, then whitespace or end-of-line.
|
||||
is_def_line :: proc(line: string) -> bool {
|
||||
trimmed := strings.trim_left(line, " \t")
|
||||
if len(trimmed) < 1 || trimmed[0] != ':' {
|
||||
return false
|
||||
}
|
||||
if len(trimmed) == 1 {
|
||||
// has_def_line reports whether any line in `body` is a definition line (see
|
||||
// is_def_line). It is a cheap guard for convert_deflists: a body with no
|
||||
// definition line cannot contain a definition list. There are no false
|
||||
// negatives (every definition list requires a definition line); a false
|
||||
// positive — e.g. a colon line that only appears inside a code fence — simply
|
||||
// falls through to the full scan, which then finds no match and returns `body`.
|
||||
has_def_line :: proc(body: string) -> bool {
|
||||
pos := 0
|
||||
for {
|
||||
line, next := line_at(body, pos) or_break
|
||||
if is_def_line(line) {
|
||||
return true
|
||||
}
|
||||
return trimmed[1] == ' ' || trimmed[1] == '\t'
|
||||
pos = next
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// write_deflist_group renders the definition-list group beginning at `start`
|
||||
// directly into `sb`, wrapped in <dl>...</dl>, and returns the byte offset one
|
||||
// past the group (the start of the first line not consumed). The caller must
|
||||
// have confirmed via term_has_def that a group starts at `start`, so the group
|
||||
// always contains at least one term+definition pair. A trailing blank line that
|
||||
// separates the group from following content is left unconsumed.
|
||||
write_deflist_group :: proc(sb: ^strings.Builder, body: string, start: int) -> (end: int) {
|
||||
strings.write_string(sb, "<dl>")
|
||||
|
||||
pos := start
|
||||
for {
|
||||
term_line, term_next, term_ok := line_at(body, pos)
|
||||
if !term_ok || is_blank_line(term_line) || is_def_line(term_line) {
|
||||
break
|
||||
}
|
||||
|
||||
// The definition follows immediately or across a single blank line.
|
||||
def_pos := term_next
|
||||
def_line, def_next, def_ok := line_at(body, def_pos)
|
||||
if def_ok && is_blank_line(def_line) {
|
||||
def_pos = def_next
|
||||
def_line, def_next, def_ok = line_at(body, def_pos)
|
||||
}
|
||||
if !def_ok || !is_def_line(def_line) {
|
||||
break
|
||||
}
|
||||
|
||||
// Emit the term with its first definition, then any consecutive
|
||||
// definition lines as additional <dd>s under the same term.
|
||||
strings.write_string(sb, "<dt>")
|
||||
strings.write_string(sb, render_inline_md(strings.trim_space(term_line)))
|
||||
strings.write_string(sb, "</dt><dd>")
|
||||
strings.write_string(sb, render_inline_md(def_content(def_line)))
|
||||
strings.write_string(sb, "</dd>")
|
||||
pos = def_next
|
||||
for {
|
||||
cont, cont_next, cont_ok := line_at(body, pos)
|
||||
if !cont_ok || !is_def_line(cont) {
|
||||
break
|
||||
}
|
||||
strings.write_string(sb, "<dd>")
|
||||
strings.write_string(sb, render_inline_md(def_content(cont)))
|
||||
strings.write_string(sb, "</dd>")
|
||||
pos = cont_next
|
||||
}
|
||||
|
||||
// Continue the group if another term+definition pair follows,
|
||||
// optionally separated by one blank line. Otherwise stop here without
|
||||
// consuming a trailing blank line.
|
||||
blank_line, after_blank, blank_ok := line_at(body, pos)
|
||||
if blank_ok && is_blank_line(blank_line) {
|
||||
if term_has_def(body, after_blank) {
|
||||
pos = after_blank
|
||||
continue
|
||||
}
|
||||
break
|
||||
}
|
||||
// No blank line: loop and let the top re-test this line as a term.
|
||||
}
|
||||
|
||||
strings.write_string(sb, "</dl>")
|
||||
return pos
|
||||
}
|
||||
|
||||
// term_has_def reports whether the line at `pos` is a term that is immediately
|
||||
// (or across one blank line) followed by a definition line.
|
||||
term_has_def :: proc(body: string, pos: int) -> bool {
|
||||
line, next, ok := line_at(body, pos)
|
||||
if !ok || is_blank_line(line) || is_def_line(line) {
|
||||
return false
|
||||
}
|
||||
def_pos := next
|
||||
def_line, def_next, def_ok := line_at(body, def_pos)
|
||||
if def_ok && is_blank_line(def_line) {
|
||||
def_pos = def_next
|
||||
def_line, def_next, def_ok = line_at(body, def_pos)
|
||||
}
|
||||
return def_ok && is_def_line(def_line)
|
||||
}
|
||||
|
||||
// fence_open reports whether `line` opens a fenced code block and, if so, which
|
||||
// marker character it uses. Leading spaces are ignored.
|
||||
fence_open :: proc(line: string) -> (marker: u8, ok: bool) {
|
||||
trimmed := strings.trim_left(line, " ")
|
||||
if strings.has_prefix(trimmed, "```") {
|
||||
return '`', true
|
||||
}
|
||||
if strings.has_prefix(trimmed, "~~~") {
|
||||
return '~', true
|
||||
}
|
||||
return 0, false
|
||||
}
|
||||
|
||||
// is_fence_line reports whether `line` is a fence using the given marker
|
||||
// character (used to detect the closing fence of an open block).
|
||||
is_fence_line :: proc(line: string, marker: u8) -> bool {
|
||||
trimmed := strings.trim_left(line, " ")
|
||||
if marker == '~' {
|
||||
return strings.has_prefix(trimmed, "~~~")
|
||||
}
|
||||
return strings.has_prefix(trimmed, "```")
|
||||
}
|
||||
|
||||
// is_def_line returns true if the line is a definition line: fewer than four
|
||||
// columns of leading whitespace, a colon, then whitespace or end-of-line. Four
|
||||
// or more columns of indentation is a code block, not a definition.
|
||||
is_def_line :: proc(line: string) -> bool {
|
||||
indent := 0
|
||||
n := 0
|
||||
for n < len(line) && (line[n] == ' ' || line[n] == '\t') {
|
||||
indent += line[n] == '\t' ? 4 : 1
|
||||
n += 1
|
||||
}
|
||||
if indent >= 4 {
|
||||
return false
|
||||
}
|
||||
rest := line[n:]
|
||||
if len(rest) < 1 || rest[0] != ':' {
|
||||
return false
|
||||
}
|
||||
if len(rest) == 1 {
|
||||
return true
|
||||
}
|
||||
return rest[1] == ' ' || rest[1] == '\t'
|
||||
}
|
||||
|
||||
// def_content extracts the definition text from a definition line,
|
||||
@@ -154,27 +302,6 @@ is_blank_line :: proc(line: string) -> bool {
|
||||
return strings.trim_space(line) == ""
|
||||
}
|
||||
|
||||
// render_deflist builds the <dl> HTML block from a list of entries.
|
||||
// Each term and definition is rendered through cmark to process inline
|
||||
// markdown. Result lives in context.temp_allocator.
|
||||
render_deflist :: proc(entries: [dynamic]DefList_Entry) -> string {
|
||||
sb := strings.builder_make(context.temp_allocator)
|
||||
|
||||
strings.write_string(&sb, "<dl>")
|
||||
for entry in entries {
|
||||
term_html := render_inline_md(entry.term)
|
||||
def_html := render_inline_md(entry.definition)
|
||||
strings.write_string(&sb, "<dt>")
|
||||
strings.write_string(&sb, term_html)
|
||||
strings.write_string(&sb, "</dt><dd>")
|
||||
strings.write_string(&sb, def_html)
|
||||
strings.write_string(&sb, "</dd>")
|
||||
}
|
||||
strings.write_string(&sb, "</dl>")
|
||||
|
||||
return strings.to_string(sb)
|
||||
}
|
||||
|
||||
// render_inline_md renders a snippet of markdown through cmark and strips
|
||||
// the surrounding <p> tags. Result lives in context.temp_allocator.
|
||||
render_inline_md :: proc(text: string) -> string {
|
||||
|
||||
@@ -0,0 +1,92 @@
|
||||
#+test
|
||||
package markdown
|
||||
|
||||
import "core:testing"
|
||||
|
||||
// Fixture-driven tests for convert_deflists. Every fixture is exact-matched
|
||||
// against its oracle in testdata/deflists/expected/. See
|
||||
// testdata/deflists/reference/README.md for how each expected file was decided
|
||||
// (cross-checked against Pandoc and Hugo/Goldmark).
|
||||
//
|
||||
// The fixture inputs are the definition-list examples from
|
||||
// https://www.markdownlang.com/extended/definition-lists.html
|
||||
//
|
||||
// Provenance of each expected file:
|
||||
// 01,02,08,09 Pandoc and Hugo agree; oracle = that agreed HTML.
|
||||
// 06 Both agree it is NOT a deflist; convert_deflists passes the
|
||||
// markdown through unchanged, so expected = the raw input.
|
||||
// 07 CommonMark's <4-space marker rule: we accept the 2-space colon
|
||||
// (Pandoc does; Hugo does not).
|
||||
// 05 Block nesting is unsupported; our flatten happens to match
|
||||
// Pandoc's output exactly.
|
||||
// 03,04 Block content in definitions is unsupported; expected captures
|
||||
// our current degraded behavior (a <dl> for the term/first line,
|
||||
// the rest falling through as markdown) as a regression guard.
|
||||
Deflist_Fixture :: struct {
|
||||
name: string,
|
||||
input: string,
|
||||
expected: string,
|
||||
}
|
||||
|
||||
@(test)
|
||||
test_deflist_fixtures :: proc(t: ^testing.T) {
|
||||
cases := []Deflist_Fixture {
|
||||
{
|
||||
"01-basic",
|
||||
#load(#directory + "testdata/deflists/01-basic.md", string),
|
||||
#load(#directory + "testdata/deflists/expected/01-basic.html", string),
|
||||
},
|
||||
{
|
||||
"02-multiple-defs",
|
||||
#load(#directory + "testdata/deflists/02-multiple-defs.md", string),
|
||||
#load(#directory + "testdata/deflists/expected/02-multiple-defs.html", string),
|
||||
},
|
||||
{
|
||||
"03-multiline-def",
|
||||
#load(#directory + "testdata/deflists/03-multiline-def.md", string),
|
||||
#load(#directory + "testdata/deflists/expected/03-multiline-def.html", string),
|
||||
},
|
||||
{
|
||||
"04-markdown-in-def",
|
||||
#load(#directory + "testdata/deflists/04-markdown-in-def.md", string),
|
||||
#load(#directory + "testdata/deflists/expected/04-markdown-in-def.html", string),
|
||||
},
|
||||
{
|
||||
"05-nested",
|
||||
#load(#directory + "testdata/deflists/05-nested.md", string),
|
||||
#load(#directory + "testdata/deflists/expected/05-nested.html", string),
|
||||
},
|
||||
{
|
||||
"06-compact",
|
||||
#load(#directory + "testdata/deflists/06-compact.md", string),
|
||||
#load(#directory + "testdata/deflists/expected/06-compact.html", string),
|
||||
},
|
||||
{
|
||||
"07-extra-space",
|
||||
#load(#directory + "testdata/deflists/07-extra-space.md", string),
|
||||
#load(#directory + "testdata/deflists/expected/07-extra-space.html", string),
|
||||
},
|
||||
{
|
||||
"08-glossary",
|
||||
#load(#directory + "testdata/deflists/08-glossary.md", string),
|
||||
#load(#directory + "testdata/deflists/expected/08-glossary.html", string),
|
||||
},
|
||||
{
|
||||
"09-api-params",
|
||||
#load(#directory + "testdata/deflists/09-api-params.md", string),
|
||||
#load(#directory + "testdata/deflists/expected/09-api-params.html", string),
|
||||
},
|
||||
}
|
||||
|
||||
for c in cases {
|
||||
result := convert_deflists(c.input, context.temp_allocator)
|
||||
testing.expectf(
|
||||
t,
|
||||
result == c.expected,
|
||||
"[%s]\n expected: %q\n got: %q",
|
||||
c.name,
|
||||
c.expected,
|
||||
result,
|
||||
)
|
||||
}
|
||||
}
|
||||
@@ -90,6 +90,17 @@ test_empty_body :: proc(t: ^testing.T) {
|
||||
testing.expect_value(t, result, "")
|
||||
}
|
||||
|
||||
// Regression: a fenced code block following a definition list must stay a
|
||||
// separate block. Without a blank line after the emitted </dl>, CommonMark
|
||||
// treats the fence as raw HTML continuation of the <dl> block and passes its
|
||||
// contents through unescaped (leaking e.g. an example <h1> into the document).
|
||||
@(test)
|
||||
test_deflist_before_code_fence :: proc(t: ^testing.T) {
|
||||
input := "term\n\n: def\n\n```\n<h1>example</h1>\n```"
|
||||
result := convert_deflists(input, context.temp_allocator)
|
||||
testing.expect(t, strings.contains(result, "</dl>\n\n```"))
|
||||
}
|
||||
|
||||
@(test)
|
||||
test_docs_md_pattern :: proc(t: ^testing.T) {
|
||||
input := "content\n\n: `content` holds your pages.\n\n assets\n : `assets` contains files."
|
||||
|
||||
+19
-9
@@ -90,9 +90,12 @@ write_escaped :: proc(b: ^strings.Builder, s: string) {
|
||||
}
|
||||
}
|
||||
|
||||
// unescape_html returns a caller-owned string (free it with delete). When the
|
||||
// input has no entities it returns a clone; otherwise it returns the builder's
|
||||
// buffer. It must NOT destroy the builder — to_string aliases that buffer, so the
|
||||
// returned string owns it.
|
||||
unescape_html :: proc(s: string) -> string {
|
||||
sb := strings.builder_make()
|
||||
defer strings.builder_destroy(&sb)
|
||||
|
||||
start := 0
|
||||
for i in 0 ..< len(s) {
|
||||
@@ -119,22 +122,28 @@ unescape_html :: proc(s: string) -> string {
|
||||
strings.write_string(&sb, replacement)
|
||||
start = i + semi + 1
|
||||
}
|
||||
if start == 0 do return s
|
||||
if start == 0 do return strings.clone(s)
|
||||
if start < len(s) do strings.write_string(&sb, s[start:])
|
||||
return strings.to_string(sb)
|
||||
}
|
||||
|
||||
highlight_block :: proc(code: string, lang: string, file_path: string) -> string {
|
||||
gc := ts.load_grammar(lang)
|
||||
if gc == nil {
|
||||
g, ok := ts.grammar(lang)
|
||||
if !ok {
|
||||
return code
|
||||
}
|
||||
parser := ts.open_parser(g)
|
||||
if parser == nil {
|
||||
return code
|
||||
}
|
||||
defer ts.parser_delete(parser)
|
||||
|
||||
raw_code := unescape_html(code)
|
||||
defer delete(raw_code)
|
||||
raw_c := strings.clone_to_cstring(raw_code)
|
||||
defer delete(raw_c)
|
||||
|
||||
tree := ts.parser_parse_string(gc.parser, nil, raw_c, u32(len(raw_code)))
|
||||
tree := ts.parser_parse_string(parser, nil, raw_c, u32(len(raw_code)))
|
||||
if tree == nil {
|
||||
return code
|
||||
}
|
||||
@@ -156,12 +165,13 @@ highlight_block :: proc(code: string, lang: string, file_path: string) -> string
|
||||
}
|
||||
}
|
||||
|
||||
cursor := gc.cursor
|
||||
if cursor == nil {
|
||||
if g.query == nil {
|
||||
return code
|
||||
}
|
||||
cursor := ts.query_cursor_new()
|
||||
defer ts.query_cursor_delete(cursor)
|
||||
|
||||
ts.query_cursor_exec(cursor, gc.query, root)
|
||||
ts.query_cursor_exec(cursor, g.query, root)
|
||||
|
||||
captures := make([dynamic]Capture, 0, 64, context.temp_allocator)
|
||||
defer delete(captures)
|
||||
@@ -174,7 +184,7 @@ highlight_block :: proc(code: string, lang: string, file_path: string) -> string
|
||||
}
|
||||
cap := match.captures[capture_idx]
|
||||
name_len: u32
|
||||
name_c := ts.query_capture_name_for_id(gc.query, cap.index, &name_len)
|
||||
name_c := ts.query_capture_name_for_id(g.query, cap.index, &name_len)
|
||||
if name_c == nil {
|
||||
continue
|
||||
}
|
||||
|
||||
@@ -0,0 +1,103 @@
|
||||
#+test
|
||||
package markdown
|
||||
|
||||
import ts "../treesitter"
|
||||
|
||||
import "core:mem"
|
||||
import "core:strings"
|
||||
import "core:testing"
|
||||
|
||||
// highlight_block returns the input string unchanged (borrowed) when it can't
|
||||
// highlight — no grammar, no query, or zero captures — and a freshly-allocated
|
||||
// string when it does. Free only the latter; identity (raw_data) tells them
|
||||
// apart regardless of content.
|
||||
@(private = "file")
|
||||
free_highlight :: proc(out, code: string) {
|
||||
if raw_data(out) != raw_data(code) {
|
||||
delete(out)
|
||||
}
|
||||
}
|
||||
|
||||
// A builtin grammar (css) highlights into hl- spans, preserving the source text.
|
||||
@(test)
|
||||
test_highlight_block_css :: proc(t: ^testing.T) {
|
||||
code := "a{color:red}"
|
||||
out := highlight_block(code, "css", "test")
|
||||
defer free_highlight(out, code)
|
||||
|
||||
testing.expect(t, strings.contains(out, "<span class=\"hl-"), "css should be highlighted")
|
||||
testing.expect(t, strings.contains(out, "</span>"))
|
||||
testing.expect(t, strings.contains(out, "color"))
|
||||
testing.expect(t, strings.contains(out, "red"))
|
||||
}
|
||||
|
||||
// The other builtin grammar (html) highlights, and the source's own '<'/'>' are
|
||||
// HTML-escaped in the output (so they can't be confused with the wrapping spans).
|
||||
@(test)
|
||||
test_highlight_block_html_escapes_source :: proc(t: ^testing.T) {
|
||||
code := "<p>hi</p>"
|
||||
out := highlight_block(code, "html", "test")
|
||||
defer free_highlight(out, code)
|
||||
|
||||
testing.expect(t, strings.contains(out, "<span class=\"hl-"), "html should be highlighted")
|
||||
testing.expect(t, strings.contains(out, "hi"))
|
||||
// The literal angle brackets of the source are escaped in the output.
|
||||
testing.expect(t, strings.contains(out, "<"))
|
||||
testing.expect(t, strings.contains(out, ">"))
|
||||
}
|
||||
|
||||
// A language with no available grammar returns the input verbatim (best-effort:
|
||||
// highlighting is optional). The returned string must be the same buffer, so
|
||||
// free_highlight leaves it alone.
|
||||
@(test)
|
||||
test_highlight_block_unknown_lang_passthrough :: proc(t: ^testing.T) {
|
||||
code := "fn main() {}"
|
||||
out := highlight_block(code, "no-such-language", "test")
|
||||
defer free_highlight(out, code)
|
||||
|
||||
testing.expect_value(t, out, code)
|
||||
testing.expect(t, raw_data(out) == raw_data(code), "passthrough must return the input buffer")
|
||||
}
|
||||
|
||||
// End-to-end: highlight_code finds a fenced code block in rendered HTML and
|
||||
// rewrites its body with highlight spans while preserving the <pre><code> shell.
|
||||
// Run under a scratch arena (as production does) so the intermediate allocations
|
||||
// highlight_code doesn't individually free are reclaimed in one shot.
|
||||
@(test)
|
||||
test_highlight_code_wraps_css_block :: proc(t: ^testing.T) {
|
||||
arena: mem.Dynamic_Arena
|
||||
mem.dynamic_arena_init(&arena)
|
||||
defer mem.dynamic_arena_destroy(&arena)
|
||||
context.allocator = mem.dynamic_arena_allocator(&arena)
|
||||
|
||||
html := `<pre><code class="language-css">a{color:red}</code></pre>`
|
||||
out := highlight_code(html, "test")
|
||||
|
||||
testing.expect(t, strings.contains(out, `<pre><code class="language-css">`))
|
||||
testing.expect(t, strings.contains(out, "</code></pre>"))
|
||||
testing.expect(
|
||||
t,
|
||||
strings.contains(out, "<span class=\"hl-"),
|
||||
"block body should be highlighted",
|
||||
)
|
||||
}
|
||||
|
||||
// unescape_html must return a caller-owned allocation the caller frees exactly
|
||||
// once. The buggy version frees its builder internally (defer builder_destroy)
|
||||
// and returns the freed alias, so the caller's delete becomes a double-free that
|
||||
// the test runner's tracking allocator flags.
|
||||
@(test)
|
||||
test_unescape_html_entities_owned :: proc(t: ^testing.T) {
|
||||
got := unescape_html("<a>")
|
||||
defer delete(got)
|
||||
testing.expect_value(t, got, "<a>")
|
||||
}
|
||||
|
||||
// Same contract on the no-entity path: the buggy version returns the borrowed
|
||||
// input, so freeing it is a bad free (pointer not from the allocator).
|
||||
@(test)
|
||||
test_unescape_html_passthrough_owned :: proc(t: ^testing.T) {
|
||||
got := unescape_html("plain text")
|
||||
defer delete(got)
|
||||
testing.expect_value(t, got, "plain text")
|
||||
}
|
||||
+2
@@ -0,0 +1,2 @@
|
||||
Term
|
||||
: Definition content
|
||||
@@ -0,0 +1,7 @@
|
||||
Markdown
|
||||
: A lightweight markup language
|
||||
: Created by John Gruber in 2004
|
||||
|
||||
HTML
|
||||
: A standard markup language used to create web pages
|
||||
: Used to create web pages
|
||||
@@ -0,0 +1,5 @@
|
||||
Term
|
||||
: This is the first line of definition content
|
||||
This is the second line, needs at least one space indentation
|
||||
|
||||
This is a new paragraph, needs at least one space indentation and a preceding blank line
|
||||
@@ -0,0 +1,7 @@
|
||||
Markdown Syntax
|
||||
: **Markdown** supports multiple text formatting:
|
||||
- *Italic* and **Bold**
|
||||
- [Link](https://www.markdownlang.com)
|
||||
- `Inline code`
|
||||
- > Block quote
|
||||
- Lists and other elements
|
||||
+6
@@ -0,0 +1,6 @@
|
||||
Outer Term
|
||||
: Outer Definition
|
||||
|
||||
Inner Term
|
||||
: Inner Definition
|
||||
: Another Inner Definition
|
||||
+1
@@ -0,0 +1 @@
|
||||
Term: Definition
|
||||
@@ -0,0 +1,2 @@
|
||||
Term
|
||||
: Definition
|
||||
+11
@@ -0,0 +1,11 @@
|
||||
API
|
||||
: An application programming interface, a set of rules that allow different software applications to communicate with each other.
|
||||
|
||||
Framework
|
||||
: A software library that provides a standardized structure for application development.
|
||||
|
||||
Git
|
||||
: A distributed version control system used to track changes in a project development process.
|
||||
|
||||
IDE
|
||||
: An integrated development environment, a software application that includes a code editor and various development tools.
|
||||
+14
@@ -0,0 +1,14 @@
|
||||
user_id
|
||||
: **Required** - The unique identifier for the user.
|
||||
: Type: `integer`
|
||||
|
||||
name
|
||||
: **Optional** - The display name for the user.
|
||||
: Type: `string`
|
||||
: Default: `null`
|
||||
|
||||
status
|
||||
: **Optional** - The user status.
|
||||
: Type: `string`
|
||||
: Allowed values: `active`, `inactive`, `suspended`
|
||||
: Default: `active`
|
||||
@@ -0,0 +1 @@
|
||||
<dl><dt>Term</dt><dd>Definition content</dd></dl>
|
||||
@@ -0,0 +1 @@
|
||||
<dl><dt>Markdown</dt><dd>A lightweight markup language</dd><dd>Created by John Gruber in 2004</dd><dt>HTML</dt><dd>A standard markup language used to create web pages</dd><dd>Used to create web pages</dd></dl>
|
||||
@@ -0,0 +1,3 @@
|
||||
<dl><dt>Term</dt><dd>This is the first line of definition content</dd></dl> This is the second line, needs at least one space indentation
|
||||
|
||||
This is a new paragraph, needs at least one space indentation and a preceding blank line
|
||||
@@ -0,0 +1,5 @@
|
||||
<dl><dt>Markdown Syntax</dt><dd><strong>Markdown</strong> supports multiple text formatting:</dd></dl> - *Italic* and **Bold**
|
||||
- [Link](https://www.markdownlang.com)
|
||||
- `Inline code`
|
||||
- > Block quote
|
||||
- Lists and other elements
|
||||
@@ -0,0 +1 @@
|
||||
<dl><dt>Outer Term</dt><dd>Outer Definition</dd><dt>Inner Term</dt><dd>Inner Definition</dd><dd>Another Inner Definition</dd></dl>
|
||||
@@ -0,0 +1 @@
|
||||
Term: Definition
|
||||
@@ -0,0 +1 @@
|
||||
<dl><dt>Term</dt><dd>Definition</dd></dl>
|
||||
@@ -0,0 +1 @@
|
||||
<dl><dt>API</dt><dd>An application programming interface, a set of rules that allow different software applications to communicate with each other.</dd><dt>Framework</dt><dd>A software library that provides a standardized structure for application development.</dd><dt>Git</dt><dd>A distributed version control system used to track changes in a project development process.</dd><dt>IDE</dt><dd>An integrated development environment, a software application that includes a code editor and various development tools.</dd></dl>
|
||||
@@ -0,0 +1 @@
|
||||
<dl><dt>user_id</dt><dd><strong>Required</strong> - The unique identifier for the user.</dd><dd>Type: <code>integer</code></dd><dt>name</dt><dd><strong>Optional</strong> - The display name for the user.</dd><dd>Type: <code>string</code></dd><dd>Default: <code>null</code></dd><dt>status</dt><dd><strong>Optional</strong> - The user status.</dd><dd>Type: <code>string</code></dd><dd>Allowed values: <code>active</code>, <code>inactive</code>, <code>suspended</code></dd><dd>Default: <code>active</code></dd></dl>
|
||||
@@ -0,0 +1,60 @@
|
||||
# 3-way behavior comparison: ours vs Pandoc vs Hugo
|
||||
|
||||
Captured for later review (e.g. if we ever pursue "Option B" — full CommonMark
|
||||
block semantics inside definitions). Compares `convert_deflists` output (Option A,
|
||||
inline/flat) against Pandoc (`-f markdown -t html5`) and Hugo/Goldmark for the
|
||||
four fixtures where behavior is interesting. Raw renderings live in `pandoc/` and
|
||||
`hugo/`; ours is in `../expected/`. Whitespace/formatting differences ignored
|
||||
below — only structural differences are described.
|
||||
|
||||
## 03-multiline-def — definition with a continuation line + a second paragraph
|
||||
*How much multi-line content belongs to the `<dd>`?*
|
||||
- **Ours:** only the **first line** goes in the `<dd>`. The continuation line and
|
||||
the second paragraph fall out of the `<dl>` as **raw, unrendered markdown**.
|
||||
- **Pandoc:** folds the continuation line **into** the `<dd>` (lazy continuation),
|
||||
then **ejects** the second paragraph as a sibling `<p>` after `</dl>`.
|
||||
- **Hugo:** keeps **everything inside** the `<dd>` — continuation line as text,
|
||||
second paragraph as a nested `<p>`.
|
||||
- All three differ. We capture the least, Pandoc the middle, Hugo the most.
|
||||
|
||||
## 04-markdown-in-def — definition containing a bullet list (with a blockquote)
|
||||
*What happens to the nested list?*
|
||||
- **Ours:** intro line becomes the `<dd>`; the **entire bullet list falls out as
|
||||
raw markdown** (unrendered `- …` lines after `</dl>`).
|
||||
- **Pandoc:** bullets **flattened into the `<dd>`** as inline text — no `<ul>`, the
|
||||
`-` become literal dashes.
|
||||
- **Hugo:** bullets become a **real nested `<ul>`/`<li>`** (blockquote and all)
|
||||
inside the `<dd>`.
|
||||
- All three differ; same shape as 03. We drop it, Pandoc absorbs-but-flattens,
|
||||
Hugo builds true structure.
|
||||
|
||||
## 05-nested — a definition list nested under another definition
|
||||
*Nest or flatten?*
|
||||
- **Ours:** **flatten** — one flat `<dl>`; "Inner Term" is just another `<dt>`.
|
||||
- **Pandoc:** **flatten** — structurally **identical to ours**.
|
||||
- **Hugo:** **nests** a real inner `<dl>` inside the outer `<dd>`.
|
||||
- Ours == Pandoc; Hugo is the odd one out.
|
||||
|
||||
## 07-extra-space — term then a 2-space-indented `: Definition`
|
||||
*Is a 2-space colon a definition marker?*
|
||||
- **Ours:** **yes** → clean `<dl>`.
|
||||
- **Pandoc:** **yes** → structurally **identical to ours**.
|
||||
- **Hugo:** **no** → rejects it, emits a plain `<p>`.
|
||||
- Ours == Pandoc; Hugo is the odd one out.
|
||||
|
||||
## Pattern
|
||||
|
||||
Two clean groups:
|
||||
|
||||
- **Marker / flattening decisions (05, 07):** we land **exactly on Pandoc**. These
|
||||
are the calls our CommonMark-marker + flat-inline design makes cleanly; Pandoc
|
||||
agrees, Hugo diverges (nests / stricter marker).
|
||||
- **Block-content-in-a-definition (03, 04):** we diverge from **both** tools, doing
|
||||
**less** — we emit the `<dl>` for the term/first line and let the rest fall
|
||||
through as raw markdown, rather than absorbing it into the `<dd>` (flattened like
|
||||
Pandoc, or structured like Hugo).
|
||||
|
||||
In short: our behavior is **"Pandoc, minus the ability to pull block content into
|
||||
a `<dd>`."** Single inline-line definitions (01, 02, 05, 07, 08, 09) match a real
|
||||
engine; genuinely multi-block definitions (03, 04) stop at the first line and pass
|
||||
the remainder through. Closing that gap is what Option B would entail.
|
||||
+44
@@ -0,0 +1,44 @@
|
||||
# Definition-list test fixtures
|
||||
|
||||
Inputs (`../NN-name.md`) are the definition-list examples from
|
||||
<https://www.markdownlang.com/extended/definition-lists.html>.
|
||||
|
||||
`pandoc/` and `hugo/` hold each input rendered by Pandoc (`-f markdown -t html5`)
|
||||
and Hugo (Goldmark with `markup.goldmark.extensions.definitionList = true`).
|
||||
They are retained as reference for a possible future "Option B" (full CommonMark
|
||||
block semantics inside definitions). They are NOT the test oracle for the current
|
||||
implementation.
|
||||
|
||||
`../expected/NN.html` is the oracle for `convert_deflists`: our inline fragment
|
||||
format (no inter-tag newlines, no trailing newline), derived from the trusted
|
||||
tool per the decisions below.
|
||||
|
||||
## Decisions (Option A — simple / inline)
|
||||
|
||||
Marker whitespace follows CommonMark: `:` with 0–3 spaces of indent is a
|
||||
definition; 4+ (tab = 4) is a code block. Definition bodies are inline/flat.
|
||||
|
||||
Every fixture has a concrete exact-match oracle in `../expected/NN.html`.
|
||||
|
||||
| Fixture | Pandoc | Hugo | Decision | expected/ source |
|
||||
|---------|--------|------|----------|------------------|
|
||||
| 01-basic | `<dl>` | `<dl>` | support | hugo (agreed) |
|
||||
| 02-multiple-defs | multi `<dd>` | multi `<dd>` | support | hugo (agreed) |
|
||||
| 06-compact (`Term: Definition`) | `<p>` | `<p>` | NOT a deflist → passthrough | raw input |
|
||||
| 07-extra-space (2-space `:`) | `<dl>` | `<p>` | support (accept, CommonMark ≤3-space rule) | hand-written |
|
||||
| 08-glossary | `<dl>` | `<dl>` | support | hugo (agreed) |
|
||||
| 09-api-params | multi `<dd>` | multi `<dd>` | support | hugo (agreed) |
|
||||
| 05-nested | flattens | nested `<dl>` | flatten (matches Pandoc) | snapshot (== pandoc) |
|
||||
| 03-multiline-def | ejects 2nd para | nests 2nd para in `<dd>` | degrade | snapshot (current output) |
|
||||
| 04-markdown-in-def | flattens list | nested `<ul>`/`<blockquote>` | degrade | snapshot (current output) |
|
||||
|
||||
A detailed 3-way behavior comparison (ours vs Pandoc vs Hugo) for the diverging
|
||||
fixtures is in `COMPARISON.md`, kept for Option-B review.
|
||||
|
||||
03/04/05 require block-level content inside `<dd>` (multi-paragraph, nested
|
||||
lists/blockquotes, nested `<dl>`), which the inline preprocessor does not
|
||||
produce. 05 flattens exactly as Pandoc does, so its snapshot equals Pandoc's
|
||||
output. 03/04 emit a `<dl>` for the term and first line, then let the remaining
|
||||
block content fall through as markdown; their `expected/` files snapshot that
|
||||
current degraded behavior as a regression guard, not an endorsed rendering.
|
||||
Revisit via Option B (full CommonMark block semantics) if real content needs it.
|
||||
@@ -0,0 +1,4 @@
|
||||
<dl>
|
||||
<dt>Term</dt>
|
||||
<dd>Definition content</dd>
|
||||
</dl>
|
||||
@@ -0,0 +1,8 @@
|
||||
<dl>
|
||||
<dt>Markdown</dt>
|
||||
<dd>A lightweight markup language</dd>
|
||||
<dd>Created by John Gruber in 2004</dd>
|
||||
<dt>HTML</dt>
|
||||
<dd>A standard markup language used to create web pages</dd>
|
||||
<dd>Used to create web pages</dd>
|
||||
</dl>
|
||||
@@ -0,0 +1,7 @@
|
||||
<dl>
|
||||
<dt>Term</dt>
|
||||
<dd>This is the first line of definition content
|
||||
This is the second line, needs at least one space indentation
|
||||
<p>This is a new paragraph, needs at least one space indentation and a preceding blank line</p>
|
||||
</dd>
|
||||
</dl>
|
||||
@@ -0,0 +1,16 @@
|
||||
<dl>
|
||||
<dt>Markdown Syntax</dt>
|
||||
<dd><strong>Markdown</strong> supports multiple text formatting:
|
||||
<ul>
|
||||
<li><em>Italic</em> and <strong>Bold</strong></li>
|
||||
<li><a href="https://www.markdownlang.com">Link</a></li>
|
||||
<li><code>Inline code</code></li>
|
||||
<li>
|
||||
<blockquote>
|
||||
<p>Block quote</p>
|
||||
</blockquote>
|
||||
</li>
|
||||
<li>Lists and other elements</li>
|
||||
</ul>
|
||||
</dd>
|
||||
</dl>
|
||||
@@ -0,0 +1,10 @@
|
||||
<dl>
|
||||
<dt>Outer Term</dt>
|
||||
<dd>Outer Definition
|
||||
<dl>
|
||||
<dt>Inner Term</dt>
|
||||
<dd>Inner Definition</dd>
|
||||
<dd>Another Inner Definition</dd>
|
||||
</dl>
|
||||
</dd>
|
||||
</dl>
|
||||
@@ -0,0 +1 @@
|
||||
<p>Term: Definition</p>
|
||||
@@ -0,0 +1,2 @@
|
||||
<p>Term
|
||||
: Definition</p>
|
||||
@@ -0,0 +1,10 @@
|
||||
<dl>
|
||||
<dt>API</dt>
|
||||
<dd>An application programming interface, a set of rules that allow different software applications to communicate with each other.</dd>
|
||||
<dt>Framework</dt>
|
||||
<dd>A software library that provides a standardized structure for application development.</dd>
|
||||
<dt>Git</dt>
|
||||
<dd>A distributed version control system used to track changes in a project development process.</dd>
|
||||
<dt>IDE</dt>
|
||||
<dd>An integrated development environment, a software application that includes a code editor and various development tools.</dd>
|
||||
</dl>
|
||||
@@ -0,0 +1,14 @@
|
||||
<dl>
|
||||
<dt>user_id</dt>
|
||||
<dd><strong>Required</strong> - The unique identifier for the user.</dd>
|
||||
<dd>Type: <code>integer</code></dd>
|
||||
<dt>name</dt>
|
||||
<dd><strong>Optional</strong> - The display name for the user.</dd>
|
||||
<dd>Type: <code>string</code></dd>
|
||||
<dd>Default: <code>null</code></dd>
|
||||
<dt>status</dt>
|
||||
<dd><strong>Optional</strong> - The user status.</dd>
|
||||
<dd>Type: <code>string</code></dd>
|
||||
<dd>Allowed values: <code>active</code>, <code>inactive</code>, <code>suspended</code></dd>
|
||||
<dd>Default: <code>active</code></dd>
|
||||
</dl>
|
||||
@@ -0,0 +1,6 @@
|
||||
<dl>
|
||||
<dt>Term</dt>
|
||||
<dd>
|
||||
Definition content
|
||||
</dd>
|
||||
</dl>
|
||||
@@ -0,0 +1,16 @@
|
||||
<dl>
|
||||
<dt>Markdown</dt>
|
||||
<dd>
|
||||
A lightweight markup language
|
||||
</dd>
|
||||
<dd>
|
||||
Created by John Gruber in 2004
|
||||
</dd>
|
||||
<dt>HTML</dt>
|
||||
<dd>
|
||||
A standard markup language used to create web pages
|
||||
</dd>
|
||||
<dd>
|
||||
Used to create web pages
|
||||
</dd>
|
||||
</dl>
|
||||
@@ -0,0 +1,9 @@
|
||||
<dl>
|
||||
<dt>Term</dt>
|
||||
<dd>
|
||||
This is the first line of definition content This is the second line,
|
||||
needs at least one space indentation
|
||||
</dd>
|
||||
</dl>
|
||||
<p>This is a new paragraph, needs at least one space indentation and a
|
||||
preceding blank line</p>
|
||||
@@ -0,0 +1,9 @@
|
||||
<dl>
|
||||
<dt>Markdown Syntax</dt>
|
||||
<dd>
|
||||
<strong>Markdown</strong> supports multiple text formatting: -
|
||||
<em>Italic</em> and <strong>Bold</strong> - <a
|
||||
href="https://www.markdownlang.com">Link</a> - <code>Inline code</code>
|
||||
- > Block quote - Lists and other elements
|
||||
</dd>
|
||||
</dl>
|
||||
@@ -0,0 +1,13 @@
|
||||
<dl>
|
||||
<dt>Outer Term</dt>
|
||||
<dd>
|
||||
Outer Definition
|
||||
</dd>
|
||||
<dt>Inner Term</dt>
|
||||
<dd>
|
||||
Inner Definition
|
||||
</dd>
|
||||
<dd>
|
||||
Another Inner Definition
|
||||
</dd>
|
||||
</dl>
|
||||
@@ -0,0 +1 @@
|
||||
<p>Term: Definition</p>
|
||||
@@ -0,0 +1,6 @@
|
||||
<dl>
|
||||
<dt>Term</dt>
|
||||
<dd>
|
||||
Definition
|
||||
</dd>
|
||||
</dl>
|
||||
@@ -0,0 +1,22 @@
|
||||
<dl>
|
||||
<dt>API</dt>
|
||||
<dd>
|
||||
An application programming interface, a set of rules that allow
|
||||
different software applications to communicate with each other.
|
||||
</dd>
|
||||
<dt>Framework</dt>
|
||||
<dd>
|
||||
A software library that provides a standardized structure for
|
||||
application development.
|
||||
</dd>
|
||||
<dt>Git</dt>
|
||||
<dd>
|
||||
A distributed version control system used to track changes in a project
|
||||
development process.
|
||||
</dd>
|
||||
<dt>IDE</dt>
|
||||
<dd>
|
||||
An integrated development environment, a software application that
|
||||
includes a code editor and various development tools.
|
||||
</dd>
|
||||
</dl>
|
||||
@@ -0,0 +1,33 @@
|
||||
<dl>
|
||||
<dt>user_id</dt>
|
||||
<dd>
|
||||
<strong>Required</strong> - The unique identifier for the user.
|
||||
</dd>
|
||||
<dd>
|
||||
Type: <code>integer</code>
|
||||
</dd>
|
||||
<dt>name</dt>
|
||||
<dd>
|
||||
<strong>Optional</strong> - The display name for the user.
|
||||
</dd>
|
||||
<dd>
|
||||
Type: <code>string</code>
|
||||
</dd>
|
||||
<dd>
|
||||
Default: <code>null</code>
|
||||
</dd>
|
||||
<dt>status</dt>
|
||||
<dd>
|
||||
<strong>Optional</strong> - The user status.
|
||||
</dd>
|
||||
<dd>
|
||||
Type: <code>string</code>
|
||||
</dd>
|
||||
<dd>
|
||||
Allowed values: <code>active</code>, <code>inactive</code>,
|
||||
<code>suspended</code>
|
||||
</dd>
|
||||
<dd>
|
||||
Default: <code>active</code>
|
||||
</dd>
|
||||
</dl>
|
||||
+17
-1
@@ -1,5 +1,6 @@
|
||||
package markdown
|
||||
|
||||
import "core:log"
|
||||
import "core:strings"
|
||||
|
||||
// generate_toc scans rendered HTML for <h1>-<h6> tags with id attributes and
|
||||
@@ -28,9 +29,24 @@ generate_toc :: proc(html: string, allocator := context.allocator) -> string {
|
||||
current_level = level
|
||||
strings.write_string(&b, "<ul>\n")
|
||||
} else if level > current_level {
|
||||
for current_level < level {
|
||||
if level - current_level > 1 {
|
||||
log.warnf(
|
||||
"heading level skips from h%d to h%d (\"%s\"); " +
|
||||
"table of contents inserts empty levels to keep valid nesting",
|
||||
current_level,
|
||||
level,
|
||||
text,
|
||||
)
|
||||
}
|
||||
// The first step down nests inside the current, still-open <li>.
|
||||
// Each further skipped level needs its own <li> to host the deeper
|
||||
// <ul>, since a <ul> may only contain <li> — otherwise a jump of
|
||||
// more than one level emits stacked, parentless <ul> tags.
|
||||
strings.write_string(&b, "<ul>\n")
|
||||
current_level += 1
|
||||
for current_level < level {
|
||||
strings.write_string(&b, "<li>\n<ul>\n")
|
||||
current_level += 1
|
||||
}
|
||||
} else if level < current_level {
|
||||
strings.write_string(&b, "</li>\n")
|
||||
|
||||
@@ -0,0 +1,35 @@
|
||||
#+test
|
||||
package markdown
|
||||
|
||||
import "core:strings"
|
||||
import "core:testing"
|
||||
|
||||
@(test)
|
||||
test_toc_sequential_nesting :: proc(t: ^testing.T) {
|
||||
html := `<h1 id="a">A</h1><h2 id="b">B</h2><h1 id="c">C</h1>`
|
||||
result := generate_toc(html, context.temp_allocator)
|
||||
testing.expect(t, strings.contains(result, `<a href="#a">A</a>`))
|
||||
testing.expect(t, strings.contains(result, `<a href="#b">B</a>`))
|
||||
testing.expect(t, strings.contains(result, `<a href="#c">C</a>`))
|
||||
// A well-formed list never places a <ul> directly inside a <ul>.
|
||||
testing.expect(t, !strings.contains(result, "<ul>\n<ul>"))
|
||||
}
|
||||
|
||||
// Regression: a heading level that jumps by more than one (h1 -> h3) must not
|
||||
// emit stacked, parentless <ul> tags. Each skipped level is wrapped in an <li>
|
||||
// so the list stays valid.
|
||||
@(test)
|
||||
test_toc_level_jump_stays_valid :: proc(t: ^testing.T) {
|
||||
html := `<h1 id="a">A</h1><h3 id="b">B</h3>`
|
||||
result := generate_toc(html, context.temp_allocator)
|
||||
testing.expect(t, !strings.contains(result, "<ul>\n<ul>"))
|
||||
// The intermediate level is hosted by an <li>.
|
||||
testing.expect(t, strings.contains(result, "<li>\n<ul>"))
|
||||
testing.expect(t, strings.contains(result, `<a href="#b">B</a>`))
|
||||
}
|
||||
|
||||
@(test)
|
||||
test_toc_empty_when_no_headings :: proc(t: ^testing.T) {
|
||||
result := generate_toc("<p>no headings here</p>", context.temp_allocator)
|
||||
testing.expect_value(t, result, "")
|
||||
}
|
||||
+24
-26
@@ -4,7 +4,6 @@ import "core:encoding/json"
|
||||
import "core:fmt"
|
||||
import "core:log"
|
||||
import "core:mem"
|
||||
import "core:os"
|
||||
import "core:strings"
|
||||
|
||||
DEFAULT_WEIGHT :: 10
|
||||
@@ -112,42 +111,32 @@ parse_page_menus :: proc(
|
||||
return result
|
||||
}
|
||||
|
||||
// build_menus populates site.menus:
|
||||
// 1. Config menus (thor.json "menus" key present) — exclusive, preserves array order
|
||||
// 2. Auto-menus (sections + root-level pages) + page frontmatter menus — merged, sorted
|
||||
// build_menus populates site.menus by merging every source into one menu set:
|
||||
// 1. Auto-menus (sections + root-level pages)
|
||||
// 2. Config menus (thor.json "menus" key), already parsed into site.menus
|
||||
// 3. Page frontmatter menus
|
||||
// All three are combined per menu name and sorted by weight, then name.
|
||||
//
|
||||
// If "menus" is present but empty ({}) it means explicit opt-out: no menus.
|
||||
// Config menus cannot be mixed with page frontmatter menus (error).
|
||||
// If "menus" is present but empty ({}) it means explicit opt-out: no menus at
|
||||
// all, and page frontmatter menus are ignored.
|
||||
build_menus :: proc(site: ^Site) {
|
||||
if site.menus != nil {
|
||||
has_menus := false
|
||||
// Explicit opt-out: "menus": {} present but empty → no menus; page
|
||||
// frontmatter menus are ignored (with a warning).
|
||||
// TODO(TODOS.md): silently discarding frontmatter here is harsh; revisit.
|
||||
if site.menus != nil && len(site.menus) == 0 {
|
||||
for page in site.pages {
|
||||
if len(page.menus) > 0 {
|
||||
has_menus = true
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
// Already populated from config in site_apply_config
|
||||
if len(site.menus) == 0 {
|
||||
// Explicit opt-out ("menus": {})
|
||||
if has_menus {
|
||||
log.warnf(
|
||||
"menus: config has empty menus but pages have frontmatter menu entries; ignoring page menus",
|
||||
)
|
||||
break
|
||||
}
|
||||
return
|
||||
}
|
||||
// Config menus active
|
||||
if has_menus {
|
||||
log.fatalf("menus: cannot mix config menus with frontmatter menus")
|
||||
os.exit(1)
|
||||
}
|
||||
warn_all_duplicate_weights(site)
|
||||
return
|
||||
}
|
||||
|
||||
// No config menus — auto-generate, then merge page menus on top
|
||||
// Merge all sources: auto section/root menus, config menus (already in
|
||||
// site.menus if present), then page frontmatter menus.
|
||||
collect_auto_menus(site)
|
||||
merge_page_menus(site)
|
||||
warn_all_duplicate_weights(site)
|
||||
@@ -253,8 +242,17 @@ collect_auto_menus :: proc(site: ^Site) {
|
||||
return
|
||||
}
|
||||
|
||||
sort_menu_entries(entries[:])
|
||||
if site.menus == nil {
|
||||
site.menus = make(map[string][]Menu_Entry, alloc)
|
||||
}
|
||||
// Merge into any existing "main" (e.g. config menus) rather than clobbering.
|
||||
if existing, ok := site.menus["main"]; ok {
|
||||
merged := make([dynamic]Menu_Entry, 0, len(existing) + len(entries), alloc)
|
||||
append(&merged, ..existing)
|
||||
append(&merged, ..entries[:])
|
||||
entries = merged
|
||||
}
|
||||
sort_menu_entries(entries[:])
|
||||
site.menus["main"] = entries[:]
|
||||
}
|
||||
|
||||
|
||||
@@ -421,6 +421,71 @@ test_auto_menus_no_duplicate_with_frontmatter :: proc(t: ^testing.T) {
|
||||
testing.expect_value(t, main[0].name, "Ideas")
|
||||
}
|
||||
|
||||
// --- config + frontmatter + auto merge tests ---
|
||||
|
||||
@(test)
|
||||
test_build_menus_config_merges_frontmatter :: proc(t: ^testing.T) {
|
||||
site: Site
|
||||
mem.dynamic_arena_init(&site.arena)
|
||||
defer mem.dynamic_arena_destroy(&site.arena)
|
||||
context.allocator = site_allocator(&site)
|
||||
|
||||
// Simulate config menus as site_apply_config would populate them.
|
||||
site.menus = make(map[string][]Menu_Entry, site_allocator(&site))
|
||||
cfg := make([dynamic]Menu_Entry, 0, 1, site_allocator(&site))
|
||||
append(&cfg, Menu_Entry{name = "GitHub", url = "https://example.com", weight = 20})
|
||||
site.menus["main"] = cfg[:]
|
||||
|
||||
// A root-level page opting into "main" via frontmatter.
|
||||
page := make_page("Ideas", "/ideas/")
|
||||
page.menus = parse_page_menus(parse_raw(`"main"`), page, site_allocator(&site))
|
||||
|
||||
site.pages = make(#soa[dynamic]Page, 0, 1, site_allocator(&site))
|
||||
append(&site.pages, page)
|
||||
|
||||
build_menus(&site)
|
||||
|
||||
main, ok := site.menus["main"]
|
||||
testing.expect(t, ok)
|
||||
// GitHub (config, weight 20) + Ideas (frontmatter, nil → DEFAULT_WEIGHT 10).
|
||||
testing.expect(t, len(main) == 2, "expected config + frontmatter merged")
|
||||
testing.expect_value(t, main[0].name, "Ideas")
|
||||
testing.expect_value(t, main[1].name, "GitHub")
|
||||
}
|
||||
|
||||
@(test)
|
||||
test_build_menus_config_merges_auto :: proc(t: ^testing.T) {
|
||||
site: Site
|
||||
mem.dynamic_arena_init(&site.arena)
|
||||
defer mem.dynamic_arena_destroy(&site.arena)
|
||||
context.allocator = site_allocator(&site)
|
||||
|
||||
// Config menu in a different bucket ("footer"); auto-gen owns "main".
|
||||
site.menus = make(map[string][]Menu_Entry, site_allocator(&site))
|
||||
cfg := make([dynamic]Menu_Entry, 0, 1, site_allocator(&site))
|
||||
append(&cfg, Menu_Entry{name = "GitHub", url = "https://example.com", weight = 20})
|
||||
site.menus["footer"] = cfg[:]
|
||||
|
||||
// A section (non-index) page triggers an auto "main" section entry.
|
||||
page := make_page("Post", "/blog/post/")
|
||||
page.section = "blog"
|
||||
|
||||
site.pages = make(#soa[dynamic]Page, 0, 1, site_allocator(&site))
|
||||
append(&site.pages, page)
|
||||
|
||||
build_menus(&site)
|
||||
|
||||
footer, ok_f := site.menus["footer"]
|
||||
testing.expect(t, ok_f, "config footer menu should be preserved")
|
||||
testing.expect(t, len(footer) == 1)
|
||||
testing.expect_value(t, footer[0].name, "GitHub")
|
||||
|
||||
main, ok_m := site.menus["main"]
|
||||
testing.expect(t, ok_m, "auto-generated main menu should exist alongside config")
|
||||
testing.expect(t, len(main) == 1)
|
||||
testing.expect_value(t, main[0].name, "Blog")
|
||||
}
|
||||
|
||||
// --- warn_duplicate_weights tests ---
|
||||
|
||||
@(test)
|
||||
|
||||
+97
-21
@@ -12,18 +12,31 @@ Range :: struct {
|
||||
end: u32,
|
||||
}
|
||||
|
||||
minify_html :: proc(source: string) -> string {
|
||||
gc := ts.ensure_parser("html")
|
||||
if gc == nil {
|
||||
return source
|
||||
// minify_html collapses insignificant whitespace, strips comments, and minifies
|
||||
// inline <style> bodies (via minify_css). The caller owns html_parser and
|
||||
// css_parser and passes them in; both may be reused across calls. On success it
|
||||
// returns (owned string, true) and the caller must free the string. On failure
|
||||
// (nil html_parser, parse failure, or syntax errors) it returns ("", false) and
|
||||
// the caller should keep the original source unchanged.
|
||||
minify_html :: proc(
|
||||
html_parser: ts.Parser,
|
||||
css_parser: ts.Parser,
|
||||
source: string,
|
||||
) -> (
|
||||
minified: string,
|
||||
ok: bool,
|
||||
) {
|
||||
if html_parser == nil {
|
||||
return minified, ok
|
||||
}
|
||||
|
||||
source_c := strings.clone_to_cstring(source)
|
||||
defer delete(source_c)
|
||||
|
||||
tree := ts.parser_parse_string(gc.parser, nil, source_c, u32(len(source)))
|
||||
tree := ts.parser_parse_string(html_parser, nil, source_c, u32(len(source)))
|
||||
if tree == nil {
|
||||
return source
|
||||
ok = false
|
||||
return minified, ok
|
||||
}
|
||||
defer ts.tree_delete(tree)
|
||||
|
||||
@@ -31,20 +44,24 @@ minify_html :: proc(source: string) -> string {
|
||||
|
||||
if ts.node_has_error(root) {
|
||||
log.warnf("minify: HTML parse errors, skipping minification")
|
||||
return source
|
||||
ok = false
|
||||
return minified, ok
|
||||
}
|
||||
|
||||
comments: [dynamic]Range
|
||||
defer delete(comments)
|
||||
preserves: [dynamic]Range
|
||||
defer delete(preserves)
|
||||
styles: [dynamic]Range
|
||||
defer delete(styles)
|
||||
|
||||
collect_html_ranges(root, source, &comments, &preserves)
|
||||
collect_html_ranges(root, source, &comments, &preserves, &styles)
|
||||
|
||||
sb := strings.builder_make()
|
||||
|
||||
ci := 0
|
||||
pi := 0
|
||||
si := 0
|
||||
i := 0
|
||||
last_written: u8 = 0
|
||||
|
||||
@@ -61,6 +78,27 @@ minify_html :: proc(source: string) -> string {
|
||||
continue
|
||||
}
|
||||
|
||||
if si < len(styles) && u32(i) >= styles[si].start {
|
||||
s := styles[si]
|
||||
seg := source[i:s.end]
|
||||
if minified, mok := minify_css(css_parser, seg); mok {
|
||||
strings.write_string(&sb, minified)
|
||||
if len(minified) > 0 {
|
||||
last_written = minified[len(minified) - 1]
|
||||
}
|
||||
delete(minified)
|
||||
} else {
|
||||
// CSS minify skipped/failed: keep the style body verbatim.
|
||||
strings.write_string(&sb, seg)
|
||||
if len(seg) > 0 {
|
||||
last_written = seg[len(seg) - 1]
|
||||
}
|
||||
}
|
||||
i = int(s.end)
|
||||
si += 1
|
||||
continue
|
||||
}
|
||||
|
||||
if ci < len(comments) && u32(i) >= comments[ci].start {
|
||||
i = int(comments[ci].end)
|
||||
ci += 1
|
||||
@@ -93,7 +131,10 @@ minify_html :: proc(source: string) -> string {
|
||||
}
|
||||
}
|
||||
|
||||
return strings.to_string(sb)
|
||||
|
||||
minified, ok = strings.to_string(sb), true
|
||||
|
||||
return minified, ok
|
||||
}
|
||||
|
||||
collect_html_ranges :: proc(
|
||||
@@ -101,6 +142,7 @@ collect_html_ranges :: proc(
|
||||
source: string,
|
||||
comments: ^[dynamic]Range,
|
||||
preserves: ^[dynamic]Range,
|
||||
styles: ^[dynamic]Range,
|
||||
) {
|
||||
child_count := ts.node_named_child_count(node)
|
||||
for i in 0 ..< child_count {
|
||||
@@ -112,7 +154,14 @@ collect_html_ranges :: proc(
|
||||
comments,
|
||||
Range{start = ts.node_start_byte(child), end = ts.node_end_byte(child)},
|
||||
)
|
||||
} else if type_str == "script_element" || type_str == "style_element" {
|
||||
} else if type_str == "style_element" {
|
||||
// Preserve the <style> tags but minify the CSS body (the raw_text
|
||||
// child) via the CSS-aware pass.
|
||||
raw, ok := style_raw_text(child)
|
||||
if ok {
|
||||
append(styles, Range{start = ts.node_start_byte(raw), end = ts.node_end_byte(raw)})
|
||||
}
|
||||
} else if type_str == "script_element" {
|
||||
append(
|
||||
preserves,
|
||||
Range{start = ts.node_start_byte(child), end = ts.node_end_byte(child)},
|
||||
@@ -125,14 +174,27 @@ collect_html_ranges :: proc(
|
||||
Range{start = ts.node_start_byte(child), end = ts.node_end_byte(child)},
|
||||
)
|
||||
} else {
|
||||
collect_html_ranges(child, source, comments, preserves)
|
||||
collect_html_ranges(child, source, comments, preserves, styles)
|
||||
}
|
||||
} else {
|
||||
collect_html_ranges(child, source, comments, preserves)
|
||||
collect_html_ranges(child, source, comments, preserves, styles)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// style_raw_text returns the raw_text (CSS body) child of a style_element.
|
||||
// ok is false for an empty <style></style>, which has no raw_text child.
|
||||
style_raw_text :: proc(style_element: ts.Node) -> (raw: ts.Node, ok: bool) {
|
||||
child_count := ts.node_named_child_count(style_element)
|
||||
for i in 0 ..< child_count {
|
||||
child := ts.node_named_child(style_element, u32(i))
|
||||
if string(ts.node_type(child)) == "raw_text" {
|
||||
return child, true
|
||||
}
|
||||
}
|
||||
return {}, false
|
||||
}
|
||||
|
||||
html_tag_name :: proc(element: ts.Node, source: string) -> string {
|
||||
child_count := ts.node_named_child_count(element)
|
||||
for i in 0 ..< child_count {
|
||||
@@ -168,18 +230,23 @@ is_css_delim :: proc(c: u8) -> bool {
|
||||
return false
|
||||
}
|
||||
|
||||
minify_css :: proc(source: string) -> string {
|
||||
gc := ts.ensure_parser("css")
|
||||
if gc == nil {
|
||||
return source
|
||||
// minify_css collapses insignificant whitespace and strips comments. The caller
|
||||
// owns `parser` and passes it in; it may be reused across calls. On success it
|
||||
// returns (owned string, true) and the caller must free the string. On failure
|
||||
// (nil parser, parse failure, or syntax errors) it returns ("", false) and the
|
||||
// caller should keep the original source unchanged.
|
||||
minify_css :: proc(parser: ts.Parser, source: string) -> (minified: string, ok: bool) {
|
||||
if parser == nil {
|
||||
return minified, ok
|
||||
}
|
||||
|
||||
source_c := strings.clone_to_cstring(source)
|
||||
defer delete(source_c)
|
||||
|
||||
tree := ts.parser_parse_string(gc.parser, nil, source_c, u32(len(source)))
|
||||
tree := ts.parser_parse_string(parser, nil, source_c, u32(len(source)))
|
||||
if tree == nil {
|
||||
return source
|
||||
ok = false
|
||||
return minified, ok
|
||||
}
|
||||
defer ts.tree_delete(tree)
|
||||
|
||||
@@ -187,7 +254,8 @@ minify_css :: proc(source: string) -> string {
|
||||
|
||||
if ts.node_has_error(root) {
|
||||
log.warnf("minify: CSS parse errors, skipping minification")
|
||||
return source
|
||||
ok = false
|
||||
return minified, ok
|
||||
}
|
||||
|
||||
comments: [dynamic]Range
|
||||
@@ -224,7 +292,14 @@ minify_css :: proc(source: string) -> string {
|
||||
next = source[j]
|
||||
}
|
||||
|
||||
if !is_css_delim(last_written) && !is_css_delim(next) {
|
||||
// Emit a single separating space, but never a leading space
|
||||
// (last_written == 0), a doubled space (last_written == ' ',
|
||||
// which happens when a stripped comment sits between two
|
||||
// whitespace runs), or one adjacent to a delimiter.
|
||||
if last_written != 0 &&
|
||||
last_written != ' ' &&
|
||||
!is_css_delim(last_written) &&
|
||||
!is_css_delim(next) {
|
||||
strings.write_byte(&sb, ' ')
|
||||
last_written = ' '
|
||||
}
|
||||
@@ -236,7 +311,8 @@ minify_css :: proc(source: string) -> string {
|
||||
}
|
||||
}
|
||||
|
||||
return strings.to_string(sb)
|
||||
minified, ok = strings.to_string(sb), true
|
||||
return minified, ok
|
||||
}
|
||||
|
||||
collect_css_comments :: proc(node: ts.Node, comments: ^[dynamic]Range) {
|
||||
|
||||
@@ -0,0 +1,98 @@
|
||||
#+test
|
||||
package main
|
||||
|
||||
import ts "treesitter"
|
||||
|
||||
import "core:strings"
|
||||
import "core:testing"
|
||||
|
||||
// The treesitter registry is thread-safe and self-initializing (grammar() lazily
|
||||
// binds it to the heap), and parsers are caller-owned. Each test opens its own
|
||||
// parser(s) via ts.open_parser, passes them to minify_*, and frees the returned
|
||||
// string — so the multi-threaded test runner needs no shared state or locking.
|
||||
|
||||
@(private = "file")
|
||||
open_parsers :: proc(t: ^testing.T) -> (html_parser, css_parser: ts.Parser) {
|
||||
hg, hok := ts.grammar("html")
|
||||
cg, cok := ts.grammar("css")
|
||||
testing.expect(t, hok, "html grammar should load")
|
||||
testing.expect(t, cok, "css grammar should load")
|
||||
return ts.open_parser(hg), ts.open_parser(cg)
|
||||
}
|
||||
|
||||
@(private = "file")
|
||||
expect_css :: proc(t: ^testing.T, parser: ts.Parser, input, want: string) {
|
||||
got, ok := minify_css(parser, input)
|
||||
testing.expect(t, ok)
|
||||
defer delete(got)
|
||||
testing.expect_value(t, got, want)
|
||||
}
|
||||
|
||||
// --- minify_html: inline <style> bodies get CSS-minified ---
|
||||
|
||||
@(test)
|
||||
test_minify_html_inline_style :: proc(t: ^testing.T) {
|
||||
hp, cp := open_parsers(t)
|
||||
defer ts.parser_delete(hp)
|
||||
defer ts.parser_delete(cp)
|
||||
|
||||
input := `<html><head><style>
|
||||
body {
|
||||
color: red;
|
||||
}
|
||||
</style></head><body><p>hi</p></body></html>`
|
||||
result, ok := minify_html(hp, cp, input)
|
||||
testing.expect(t, ok)
|
||||
defer delete(result)
|
||||
|
||||
// The CSS body is collapsed by minify_css (delimiters lose surrounding
|
||||
// whitespace)...
|
||||
testing.expect(t, strings.contains(result, "body{color:red;}"))
|
||||
// ...while the <style> tags themselves survive.
|
||||
testing.expect(t, strings.contains(result, "<style>"))
|
||||
testing.expect(t, strings.contains(result, "</style>"))
|
||||
}
|
||||
|
||||
@(test)
|
||||
test_minify_html_empty_style :: proc(t: ^testing.T) {
|
||||
hp, cp := open_parsers(t)
|
||||
defer ts.parser_delete(hp)
|
||||
defer ts.parser_delete(cp)
|
||||
|
||||
// An empty <style></style> has no raw_text child; must not crash.
|
||||
input := `<html><head><style></style></head><body></body></html>`
|
||||
result, ok := minify_html(hp, cp, input)
|
||||
testing.expect(t, ok)
|
||||
defer delete(result)
|
||||
testing.expect(t, strings.contains(result, "<style></style>"))
|
||||
}
|
||||
|
||||
// --- minify_css: whitespace around stripped comments / leading edge ---
|
||||
|
||||
@(test)
|
||||
test_minify_css_leading_and_comment :: proc(t: ^testing.T) {
|
||||
cg, cok := ts.grammar("css")
|
||||
testing.expect(t, cok, "css grammar should load")
|
||||
cp := ts.open_parser(cg)
|
||||
defer ts.parser_delete(cp)
|
||||
|
||||
// Leading whitespace and a comment between two whitespace runs must not
|
||||
// leave a leading space or a doubled space.
|
||||
expect_css(t, cp, "\n/* c */\na {}", "a{}")
|
||||
expect_css(t, cp, " a {}", "a{}")
|
||||
expect_css(t, cp, "a {}\n/* trailing */", "a{}")
|
||||
}
|
||||
|
||||
@(test)
|
||||
test_minify_html_script_preserved :: proc(t: ^testing.T) {
|
||||
hp, cp := open_parsers(t)
|
||||
defer ts.parser_delete(hp)
|
||||
defer ts.parser_delete(cp)
|
||||
|
||||
// <script> stays byte-for-byte (whitespace-significant JS).
|
||||
input := "<html><body><script>\nconst s = \"a b\";\n</script></body></html>"
|
||||
result, ok := minify_html(hp, cp, input)
|
||||
testing.expect(t, ok)
|
||||
defer delete(result)
|
||||
testing.expect(t, strings.contains(result, "const s = \"a b\";"))
|
||||
}
|
||||
@@ -4,3 +4,4 @@
|
||||
- [ ] Simplify `trim_standalone_whitespace` — four temp arrays (`li_buf`, `ri_buf`, `left_done`, `right_done`) could potentially be reduced to two by merging the done-tracking into the index buffers using sentinel values (e.g. `-2` = already trimmed).
|
||||
- [ ] Adjacent standalone tags don't capture indent for the second tag (e.g. ` {{>a}}\n {{>b}}` — `{{>b}}` gets `indent=""` because its left text was consumed by `{{>a}}`'s right-trim). Blocks recover via `deindent_blocks`, but partials/parents have no fallback.
|
||||
- [ ] lazy load partials ? (with mutex)
|
||||
- [ ] Support changing delimiters (Set Delimiter tag `{{=<% %>=}}`). Delimiters `{{`/`}}`/`{{{`/`}}}` are hardcoded in [tokenizer.odin](./tokenizer.odin); needs an `=` sigil case, mutable open/close delimiters threaded through `tokenize`, and wiring up `spec/specs/delimiters.json` in `spec_test.odin` (fixture present but not run).
|
||||
|
||||
+33
-6
@@ -182,6 +182,7 @@ render :: proc(
|
||||
partials: map[string]Template = nil,
|
||||
allocator := context.allocator,
|
||||
warnings: ^[dynamic]Error = nil,
|
||||
tools: ^Tool_Registry = nil,
|
||||
) -> (
|
||||
result: string,
|
||||
err: Error,
|
||||
@@ -207,7 +208,15 @@ render :: proc(
|
||||
}
|
||||
|
||||
all_nodes := tmpl.nodes[:]
|
||||
err = render_nodes(tmpl, all_nodes, &ctx, partials, &builder, warnings = warnings)
|
||||
err = render_nodes(
|
||||
tmpl,
|
||||
all_nodes,
|
||||
&ctx,
|
||||
partials,
|
||||
&builder,
|
||||
warnings = warnings,
|
||||
tools = tools,
|
||||
)
|
||||
if err != nil {
|
||||
return result, err
|
||||
}
|
||||
@@ -588,6 +597,7 @@ render_template :: proc(
|
||||
blocks: map[string]Block_Override,
|
||||
indent: string,
|
||||
warnings: ^[dynamic]Error = nil,
|
||||
tools: ^Tool_Registry = nil,
|
||||
) -> Error {
|
||||
if len(indent) > 0 {
|
||||
state := Indent_State {
|
||||
@@ -595,9 +605,9 @@ render_template :: proc(
|
||||
at_line_start = false,
|
||||
}
|
||||
strings.write_string(b, indent) // first line always gets indent
|
||||
return render_nodes(pt, pt.nodes[:], ctx, partials, b, blocks, &state, warnings)
|
||||
return render_nodes(pt, pt.nodes[:], ctx, partials, b, blocks, &state, warnings, tools)
|
||||
}
|
||||
return render_nodes(pt, pt.nodes[:], ctx, partials, b, blocks, nil, warnings)
|
||||
return render_nodes(pt, pt.nodes[:], ctx, partials, b, blocks, nil, warnings, tools)
|
||||
}
|
||||
|
||||
write_indented :: proc(
|
||||
@@ -642,6 +652,7 @@ render_nodes :: proc(
|
||||
blocks: map[string]Block_Override = nil,
|
||||
indent_state: ^Indent_State = nil,
|
||||
warnings: ^[dynamic]Error = nil,
|
||||
tools: ^Tool_Registry = nil,
|
||||
) -> Error {
|
||||
i := 0
|
||||
for i < len(nodes) {
|
||||
@@ -672,6 +683,7 @@ render_nodes :: proc(
|
||||
ctx[:],
|
||||
current,
|
||||
warnings,
|
||||
tools,
|
||||
)
|
||||
if perr != nil {
|
||||
return tag_error(perr, current)
|
||||
@@ -698,6 +710,7 @@ render_nodes :: proc(
|
||||
ctx[:],
|
||||
current,
|
||||
warnings,
|
||||
tools,
|
||||
)
|
||||
if perr != nil {
|
||||
return tag_error(perr, current)
|
||||
@@ -720,6 +733,7 @@ render_nodes :: proc(
|
||||
ctx[:],
|
||||
current,
|
||||
warnings,
|
||||
tools,
|
||||
)
|
||||
if perr != nil {
|
||||
return tag_error(perr, current)
|
||||
@@ -743,6 +757,7 @@ render_nodes :: proc(
|
||||
blocks,
|
||||
indent_state,
|
||||
warnings,
|
||||
tools,
|
||||
) or_return
|
||||
}
|
||||
} else {
|
||||
@@ -775,6 +790,7 @@ render_nodes :: proc(
|
||||
ctx[:],
|
||||
current,
|
||||
warnings,
|
||||
tools,
|
||||
)
|
||||
if perr != nil {
|
||||
return tag_error(perr, current)
|
||||
@@ -791,6 +807,7 @@ render_nodes :: proc(
|
||||
blocks,
|
||||
indent_state,
|
||||
warnings,
|
||||
tools,
|
||||
) or_return
|
||||
}
|
||||
i += 1 + len(node.children)
|
||||
@@ -805,7 +822,7 @@ render_nodes :: proc(
|
||||
if !found {
|
||||
warn_missing_partial(current, partials, node, name)
|
||||
} else {
|
||||
render_template(pt, ctx, partials, b, nil, node.indent, warnings) or_return
|
||||
render_template(pt, ctx, partials, b, nil, node.indent, warnings, tools) or_return
|
||||
if indent_state != nil {
|
||||
indent_state.at_line_start = false
|
||||
}
|
||||
@@ -841,6 +858,7 @@ render_nodes :: proc(
|
||||
content_blocks,
|
||||
nil,
|
||||
warnings,
|
||||
tools,
|
||||
) or_return
|
||||
at_ls := true
|
||||
write_indented(b, node.indent, strings.to_string(temp), &at_ls)
|
||||
@@ -854,6 +872,7 @@ render_nodes :: proc(
|
||||
content_blocks,
|
||||
indent_state,
|
||||
warnings,
|
||||
tools,
|
||||
) or_return
|
||||
}
|
||||
i += 1 + len(node.children)
|
||||
@@ -866,7 +885,16 @@ render_nodes :: proc(
|
||||
warn_missing_partial(current, partials, node, node.key)
|
||||
} else {
|
||||
warn_unmatched_block_overrides(current, pt, parent_children)
|
||||
render_template(pt, ctx, partials, b, merged, node.indent, warnings) or_return
|
||||
render_template(
|
||||
pt,
|
||||
ctx,
|
||||
partials,
|
||||
b,
|
||||
merged,
|
||||
node.indent,
|
||||
warnings,
|
||||
tools,
|
||||
) or_return
|
||||
if indent_state != nil {
|
||||
indent_state.at_line_start = false
|
||||
}
|
||||
@@ -1057,4 +1085,3 @@ warn_context_depth :: proc(current: Template, node: Node) {
|
||||
)
|
||||
log.warnf("%s", diag)
|
||||
}
|
||||
|
||||
|
||||
+93
-4
@@ -24,6 +24,8 @@ Pipe_Op :: enum {
|
||||
Group_By,
|
||||
First,
|
||||
Last,
|
||||
Rel_Url,
|
||||
Tool,
|
||||
}
|
||||
|
||||
// Pipe op names are derived from the enum via reflection (lowercased).
|
||||
@@ -224,13 +226,14 @@ apply_pipeline :: proc(
|
||||
ctx: []any,
|
||||
tmpl: Template,
|
||||
warnings: ^[dynamic]Error = nil,
|
||||
tools: ^Tool_Registry = nil,
|
||||
) -> (
|
||||
current: any,
|
||||
err: Error,
|
||||
) {
|
||||
current = value
|
||||
for &filter in filters {
|
||||
result, ferr := apply_filter(current, &filter, pos, ctx, tmpl)
|
||||
result, ferr := apply_filter(current, &filter, pos, ctx, tmpl, tools)
|
||||
if ferr != nil {
|
||||
b := body(ferr)
|
||||
if b.kind != .Warning {
|
||||
@@ -277,6 +280,7 @@ apply_filter :: proc(
|
||||
pos: int,
|
||||
ctx: []any,
|
||||
tmpl: Template,
|
||||
tools: ^Tool_Registry = nil,
|
||||
) -> (
|
||||
any,
|
||||
Error,
|
||||
@@ -345,10 +349,83 @@ apply_filter :: proc(
|
||||
case .Last:
|
||||
span := filter.end_pos - filter.op_pos
|
||||
return take(value, filter.args[:], false, tmpl, filter.op_pos, span)
|
||||
case .Rel_Url:
|
||||
bv, _ := base_value(value)
|
||||
str, ok := reflect.as_string(bv)
|
||||
if !ok {
|
||||
return value, Error_Body {
|
||||
msg = "rel_url may only be used on strings",
|
||||
pos = pos,
|
||||
kind = .Data,
|
||||
}
|
||||
}
|
||||
result := rel_url(str, ctx)
|
||||
return any{new_clone(result, context.temp_allocator), typeid_of(string)}, nil
|
||||
case .Tool:
|
||||
if len(filter.args) != 1 {
|
||||
return value, Error_Body {
|
||||
msg = fmt.tprintf(
|
||||
"tool expects 1 argument (the tool name), got %d",
|
||||
len(filter.args),
|
||||
),
|
||||
pos = filter.op_pos,
|
||||
span = filter.end_pos - filter.op_pos,
|
||||
kind = .Data,
|
||||
}
|
||||
}
|
||||
bv, _ := base_value(value)
|
||||
str, ok := reflect.as_string(bv)
|
||||
if !ok {
|
||||
return value, Error_Body {
|
||||
msg = "tool may only be used on strings",
|
||||
pos = pos,
|
||||
kind = .Data,
|
||||
}
|
||||
}
|
||||
result, terr := run_tool(tools, filter.args[0], str, filter.op_pos)
|
||||
if terr != nil {
|
||||
return value, terr
|
||||
}
|
||||
return any{new_clone(result, context.temp_allocator), typeid_of(string)}, nil
|
||||
}
|
||||
return {}, nil
|
||||
}
|
||||
|
||||
// rel_url prefixes a root-relative path (e.g. "/faq/") with the site's
|
||||
// base_path so links resolve correctly under a subpath deployment. Values that
|
||||
// are already absolute — a scheme ("http://"), protocol-relative ("//"), an
|
||||
// in-page fragment ("#"), or a "mailto:" — are returned unchanged, which keeps
|
||||
// external menu and footer links intact. base_path is read from the context
|
||||
// stack the same way `format` reads date_format/timezone.
|
||||
rel_url :: proc(path: string, ctx: []any) -> string {
|
||||
if path == "" {
|
||||
return path
|
||||
}
|
||||
if strings.has_prefix(path, "http://") ||
|
||||
strings.has_prefix(path, "https://") ||
|
||||
strings.has_prefix(path, "//") ||
|
||||
strings.has_prefix(path, "#") ||
|
||||
strings.has_prefix(path, "mailto:") {
|
||||
return path
|
||||
}
|
||||
|
||||
base := ""
|
||||
if raw := resolve_name("base_path", ctx); raw != nil {
|
||||
if s, ok := reflect.as_string(raw); ok {
|
||||
base = s
|
||||
}
|
||||
}
|
||||
if base == "" {
|
||||
return path
|
||||
}
|
||||
|
||||
// Only prefix root-relative paths; leave document-relative ones alone.
|
||||
if path[0] == '/' {
|
||||
return fmt.tprintf("%s%s", base, path)
|
||||
}
|
||||
return path
|
||||
}
|
||||
|
||||
apply_format :: proc(
|
||||
iso: string,
|
||||
args: []string,
|
||||
@@ -386,7 +463,15 @@ apply_format :: proc(
|
||||
}
|
||||
|
||||
// Groups preserve first-appearance order from the input list.
|
||||
apply_group_by :: proc(value: any, args: []string, pos: int, span: int) -> (result: any, err: Error) {
|
||||
apply_group_by :: proc(
|
||||
value: any,
|
||||
args: []string,
|
||||
pos: int,
|
||||
span: int,
|
||||
) -> (
|
||||
result: any,
|
||||
err: Error,
|
||||
) {
|
||||
if len(args) != 1 {
|
||||
hint := ""
|
||||
if len(args) == 0 {
|
||||
@@ -406,7 +491,12 @@ apply_group_by :: proc(value: any, args: []string, pos: int, span: int) -> (resu
|
||||
|
||||
elem_info, count, data := list_info(value)
|
||||
if elem_info == nil {
|
||||
return nil, Error_Body{msg = "group_by expects a list", pos = pos, span = span, kind = .Data}
|
||||
return nil, Error_Body {
|
||||
msg = "group_by expects a list",
|
||||
pos = pos,
|
||||
span = span,
|
||||
kind = .Data,
|
||||
}
|
||||
}
|
||||
|
||||
groups := make([dynamic]Group, 0, 8, context.temp_allocator)
|
||||
@@ -557,4 +647,3 @@ take :: proc(
|
||||
return any{new_clone(str[start:end], context.temp_allocator), typeid_of(string)}, warning
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,319 @@
|
||||
package mustache
|
||||
|
||||
import "core:fmt"
|
||||
import "core:os"
|
||||
import "core:strings"
|
||||
import "core:sync"
|
||||
|
||||
// The `tool` pipe runs a user-configured external program over a site asset.
|
||||
// A tool is a command string that is itself a mustache template; rendering it
|
||||
// yields the command line, which is tokenized into argv and run via
|
||||
// os.process_exec (no shell). The command template has three variables:
|
||||
//
|
||||
// {{in}} absolute path to the source file the tool should read
|
||||
// {{out}} absolute path the tool should write its result to
|
||||
// {{#minify}} section, non-empty when the site is building with minify on
|
||||
//
|
||||
// {{in}}/{{out}} are optional per tool. If a command omits {{in}}, thor feeds
|
||||
// the source file on the tool's stdin; if it omits {{out}}, thor captures the
|
||||
// tool's stdout. Stdin/stdout is the preferred form (it will compose best with
|
||||
// the future `fingerprint` pipe); {{in}}/{{out}} is the fallback for tools that
|
||||
// can't stream. The pipe takes an asset path (e.g. "/css/main.css"), runs the
|
||||
// tool, and returns the root-relative URL of the written output.
|
||||
|
||||
// Tool_Registry is the host-supplied configuration for the `tool` pipe. The
|
||||
// host (thor) owns one per build and threads a pointer to it through render, so
|
||||
// it never enters the template context stack and can't be resolved as a key.
|
||||
// The config fields are written once before rendering and only read thereafter;
|
||||
// `manifest` is the sole mutable state and is guarded by `mu` — needed because
|
||||
// render may go parallel and because os.process_start is not thread-safe.
|
||||
Tool_Registry :: struct {
|
||||
commands: map[string]string, // tool name -> command template
|
||||
input_dir: string, // base dir for resolving {{in}} (the assets dir)
|
||||
output_dir: string, // base dir for the written output (the output dir)
|
||||
minify: bool,
|
||||
manifest: map[string]string, // (name\x00input) -> output URL, per build
|
||||
mu: sync.Mutex,
|
||||
}
|
||||
|
||||
// tool_registry_init prepares `reg` for one build: the input/output base dirs,
|
||||
// the minify flag, and a fresh (empty) manifest. `commands` is configured
|
||||
// separately (from thor.json) and left untouched. The manifest is temp-allocated
|
||||
// — build-scoped, discarded when the build's temp allocator is reset. Nothing
|
||||
// needs to be cleaned up between builds: the host recreates the whole registry.
|
||||
tool_registry_init :: proc(reg: ^Tool_Registry, input_dir, output_dir: string, minify: bool) {
|
||||
reg.input_dir = input_dir
|
||||
reg.output_dir = output_dir
|
||||
reg.minify = minify
|
||||
reg.manifest = make(map[string]string, context.temp_allocator)
|
||||
}
|
||||
|
||||
// run_tool executes the named tool over `input` (an asset path) and returns the
|
||||
// root-relative URL of the produced file. Any failure is a hard error (.Data).
|
||||
run_tool :: proc(
|
||||
reg: ^Tool_Registry,
|
||||
name: string,
|
||||
input: string,
|
||||
pos: int,
|
||||
) -> (
|
||||
url: string,
|
||||
err: Error,
|
||||
) {
|
||||
if reg == nil {
|
||||
return "", Error_Body{msg = "tool pipe is not available here", pos = pos, kind = .Data}
|
||||
}
|
||||
|
||||
// Absolute URLs (CDN links, protocol-relative) have no local source; pass
|
||||
// them through so a mixed stylesheet list stays safe.
|
||||
if is_absolute_url(input) {
|
||||
return input, nil
|
||||
}
|
||||
|
||||
cmd_tmpl, ok := reg.commands[name]
|
||||
if !ok {
|
||||
return "", Error_Body {
|
||||
msg = fmt.tprintf("unknown tool '%s' (add it to thor.json \"tools\")", name),
|
||||
pos = pos,
|
||||
kind = .Data,
|
||||
}
|
||||
}
|
||||
|
||||
// Logical path -> in/out/url. Strip a single leading '/'.
|
||||
rel := input
|
||||
if len(rel) > 0 && rel[0] == '/' {
|
||||
rel = rel[1:]
|
||||
}
|
||||
if rel == "" {
|
||||
return "", Error_Body{msg = "tool: empty asset path", pos = pos, kind = .Data}
|
||||
}
|
||||
in_path := fmt.tprintf("%s/%s", reg.input_dir, rel)
|
||||
out_path := fmt.tprintf("%s/%s", reg.output_dir, rel)
|
||||
out_url := fmt.tprintf("/%s", rel)
|
||||
|
||||
if !os.exists(in_path) {
|
||||
return "", Error_Body {
|
||||
msg = fmt.tprintf("tool '%s': source file not found: %s", name, in_path),
|
||||
pos = pos,
|
||||
kind = .Data,
|
||||
}
|
||||
}
|
||||
|
||||
// Render the command template. Done OUTSIDE the lock so a command template
|
||||
// that (mistakenly) uses a `tool` pipe can't deadlock on reg.mu. The
|
||||
// registry fields it reads are immutable during rendering.
|
||||
cmd_str, rerr := render_command(cmd_tmpl, in_path, out_path, reg.minify, pos)
|
||||
if rerr != nil {
|
||||
return "", rerr
|
||||
}
|
||||
|
||||
argv := tokenize_command(cmd_str, context.temp_allocator)
|
||||
if len(argv) == 0 {
|
||||
return "", Error_Body {
|
||||
msg = fmt.tprintf("tool '%s': command is empty", name),
|
||||
pos = pos,
|
||||
kind = .Data,
|
||||
}
|
||||
}
|
||||
if !find_on_path(argv[0]) {
|
||||
return "", Error_Body {
|
||||
msg = fmt.tprintf("tool '%s': program '%s' not found on PATH", name, argv[0]),
|
||||
pos = pos,
|
||||
kind = .Data,
|
||||
}
|
||||
}
|
||||
|
||||
sync.lock(®.mu)
|
||||
defer sync.unlock(®.mu)
|
||||
|
||||
key := fmt.tprintf("%s\x00%s", name, input)
|
||||
if cached, hit := reg.manifest[key]; hit {
|
||||
return cached, nil
|
||||
}
|
||||
|
||||
// How the tool speaks I/O is inferred from whether the command references
|
||||
// the paths thor substituted:
|
||||
// {{in}} present -> the tool opens the source file itself
|
||||
// {{in}} absent -> thor feeds the source file on stdin
|
||||
// {{out}} present -> the tool writes the output file itself
|
||||
// {{out}} absent -> thor captures stdout and writes it to out_path
|
||||
// Either way the run ends with a real file at out_path.
|
||||
uses_in := strings.contains(cmd_str, in_path)
|
||||
uses_out := strings.contains(cmd_str, out_path)
|
||||
|
||||
// Ensure the output's parent dir exists (for the tool's {{out}} write or
|
||||
// thor's stdout write).
|
||||
if idx := strings.last_index(out_path, "/"); idx >= 0 {
|
||||
if merr := os.make_directory_all(out_path[:idx]); merr != nil && merr != .Exist {
|
||||
return "", Error_Body {
|
||||
msg = fmt.tprintf("tool '%s': cannot create %s: %v", name, out_path[:idx], merr),
|
||||
pos = pos,
|
||||
kind = .Data,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
desc := os.Process_Desc {
|
||||
command = argv,
|
||||
}
|
||||
// Feed the source on stdin unless the tool takes it as a {{in}} path.
|
||||
stdin_file: ^os.File
|
||||
if !uses_in {
|
||||
f, oerr := os.open(in_path)
|
||||
if oerr != nil {
|
||||
return "", Error_Body {
|
||||
msg = fmt.tprintf("tool '%s': cannot open %s: %v", name, in_path, oerr),
|
||||
pos = pos,
|
||||
kind = .Data,
|
||||
}
|
||||
}
|
||||
stdin_file = f
|
||||
desc.stdin = f
|
||||
}
|
||||
defer if stdin_file != nil {
|
||||
os.close(stdin_file)
|
||||
}
|
||||
|
||||
state, stdout, stderr, xerr := os.process_exec(desc, context.temp_allocator)
|
||||
if xerr != nil {
|
||||
return "", Error_Body {
|
||||
msg = fmt.tprintf("tool '%s': failed to run '%s': %v", name, argv[0], xerr),
|
||||
pos = pos,
|
||||
kind = .Data,
|
||||
}
|
||||
}
|
||||
if state.exit_code != 0 {
|
||||
return "", Error_Body {
|
||||
msg = fmt.tprintf(
|
||||
"tool '%s': '%s' exited %d\n%s",
|
||||
name,
|
||||
argv[0],
|
||||
state.exit_code,
|
||||
strings.trim_space(string(stderr)),
|
||||
),
|
||||
pos = pos,
|
||||
kind = .Data,
|
||||
}
|
||||
}
|
||||
|
||||
// If the tool streamed to stdout, persist it as the output file.
|
||||
if !uses_out {
|
||||
if werr := os.write_entire_file(out_path, stdout); werr != nil {
|
||||
return "", Error_Body {
|
||||
msg = fmt.tprintf("tool '%s': cannot write %s: %v", name, out_path, werr),
|
||||
pos = pos,
|
||||
kind = .Data,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Memoize for the rest of this build. `key` and `out_url` are already on the
|
||||
// temp allocator — the same build-scoped lifetime as the manifest — so they
|
||||
// are stored directly, no clone needed.
|
||||
if reg.manifest == nil {
|
||||
reg.manifest = make(map[string]string, context.temp_allocator)
|
||||
}
|
||||
reg.manifest[key] = out_url
|
||||
return out_url, nil
|
||||
}
|
||||
|
||||
// render_command renders a tool's command template with the in/out/minify
|
||||
// context. Paths render through mustache's default HTML escaping; the paths we
|
||||
// generate are controlled and contain no HTML-special characters, so this is
|
||||
// harmless in practice.
|
||||
render_command :: proc(
|
||||
tmpl_src, in_path, out_path: string,
|
||||
minify: bool,
|
||||
pos: int,
|
||||
) -> (
|
||||
string,
|
||||
Error,
|
||||
) {
|
||||
tmpl, perr := parse(tmpl_src, "", context.temp_allocator, context.temp_allocator)
|
||||
if perr != nil {
|
||||
b := body(perr)
|
||||
return "", Error_Body {
|
||||
msg = fmt.tprintf("invalid tool command template: %s", b.msg),
|
||||
pos = pos,
|
||||
kind = .Data,
|
||||
}
|
||||
}
|
||||
// `in`/`out` are Odin keywords, so the context is a map rather than a struct.
|
||||
data := make(map[string]any, 3, context.temp_allocator)
|
||||
data["in"] = in_path
|
||||
data["out"] = out_path
|
||||
data["minify"] = minify
|
||||
out, rerr := render(tmpl, data, allocator = context.temp_allocator)
|
||||
if rerr != nil {
|
||||
b := body(rerr)
|
||||
return "", Error_Body {
|
||||
msg = fmt.tprintf("failed to render tool command: %s", b.msg),
|
||||
pos = pos,
|
||||
kind = .Data,
|
||||
}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// is_absolute_url reports whether a path points outside the local site.
|
||||
is_absolute_url :: proc(s: string) -> bool {
|
||||
return(
|
||||
strings.has_prefix(s, "http://") ||
|
||||
strings.has_prefix(s, "https://") ||
|
||||
strings.has_prefix(s, "//") \
|
||||
)
|
||||
}
|
||||
|
||||
// tokenize_command splits a command line into argv on whitespace, treating a
|
||||
// double-quoted span as a single token with the quotes stripped. No escape
|
||||
// sequences or single quotes (kept deliberately minimal — commands come from a
|
||||
// trusted config, not untrusted input).
|
||||
tokenize_command :: proc(cmd: string, allocator := context.temp_allocator) -> []string {
|
||||
out := make([dynamic]string, 0, 8, allocator)
|
||||
i := 0
|
||||
for i < len(cmd) {
|
||||
for i < len(cmd) && is_pipe_space(cmd[i]) {
|
||||
i += 1
|
||||
}
|
||||
if i >= len(cmd) {
|
||||
break
|
||||
}
|
||||
if cmd[i] == '"' {
|
||||
j := i + 1
|
||||
for j < len(cmd) && cmd[j] != '"' {
|
||||
j += 1
|
||||
}
|
||||
append(&out, strings.clone(cmd[i + 1:j], allocator))
|
||||
i = j + 1 if j < len(cmd) else j
|
||||
} else {
|
||||
start := i
|
||||
for i < len(cmd) && !is_pipe_space(cmd[i]) {
|
||||
i += 1
|
||||
}
|
||||
append(&out, strings.clone(cmd[start:i], allocator))
|
||||
}
|
||||
}
|
||||
return out[:]
|
||||
}
|
||||
|
||||
// find_on_path reports whether `prog` is executable: an explicit path is checked
|
||||
// directly, otherwise each PATH entry is probed.
|
||||
find_on_path :: proc(prog: string) -> bool {
|
||||
if strings.contains(prog, "/") {
|
||||
return os.exists(prog)
|
||||
}
|
||||
path := os.get_env_alloc("PATH", context.temp_allocator)
|
||||
if path == "" {
|
||||
return false
|
||||
}
|
||||
sep := ";" when ODIN_OS == .Windows else ":"
|
||||
for dir in strings.split_iterator(&path, sep) {
|
||||
if dir == "" {
|
||||
continue
|
||||
}
|
||||
candidate := fmt.tprintf("%s/%s", dir, prog)
|
||||
if os.exists(candidate) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
@@ -0,0 +1,208 @@
|
||||
#+test
|
||||
package mustache
|
||||
|
||||
import "core:fmt"
|
||||
import "core:os"
|
||||
import "core:strings"
|
||||
import "core:testing"
|
||||
|
||||
// tool_test_dirs makes a fresh in/out dir pair under the system temp dir and
|
||||
// returns their paths (unique per name).
|
||||
tool_test_dirs :: proc(name: string) -> (in_dir, out_dir: string) {
|
||||
base, _ := os.temp_directory(context.temp_allocator)
|
||||
in_dir = fmt.tprintf("%s/thor_tool_test/%s/in", base, name)
|
||||
out_dir = fmt.tprintf("%s/thor_tool_test/%s/out", base, name)
|
||||
os.make_directory_all(in_dir)
|
||||
os.make_directory_all(out_dir)
|
||||
return
|
||||
}
|
||||
|
||||
// Tool_Test_Ctx carries the asset path as a context value (mustache resolves
|
||||
// tag keys from the context, not from quoted literals).
|
||||
Tool_Test_Ctx :: struct {
|
||||
p: string,
|
||||
}
|
||||
|
||||
@(test)
|
||||
test_tool_unknown_tool :: proc(t: ^testing.T) {
|
||||
reg: Tool_Registry // no commands configured
|
||||
tpl, _ := parse(`{{ p | tool nope }}`)
|
||||
defer delete_template(&tpl)
|
||||
_, err := render(tpl, Tool_Test_Ctx{p = "/a.css"}, tools = ®)
|
||||
testing.expect(t, err != nil, "unknown tool should be a hard error")
|
||||
}
|
||||
|
||||
@(test)
|
||||
test_tool_unavailable_without_registry :: proc(t: ^testing.T) {
|
||||
// No registry threaded through (tools == nil): the pipe must fail rather
|
||||
// than silently do nothing.
|
||||
tpl, _ := parse(`{{ p | tool anything }}`)
|
||||
defer delete_template(&tpl)
|
||||
_, err := render(tpl, Tool_Test_Ctx{p = "/a.css"})
|
||||
testing.expect(t, err != nil, "tool pipe without a registry should error")
|
||||
}
|
||||
|
||||
@(test)
|
||||
test_tool_arg_count :: proc(t: ^testing.T) {
|
||||
reg: Tool_Registry
|
||||
|
||||
// Zero args.
|
||||
tpl0, _ := parse(`{{ p | tool }}`)
|
||||
defer delete_template(&tpl0)
|
||||
_, err0 := render(tpl0, Tool_Test_Ctx{p = "/a.css"}, tools = ®)
|
||||
testing.expect(t, err0 != nil, "tool with no name should error")
|
||||
|
||||
// Two args.
|
||||
tpl2, _ := parse(`{{ p | tool one two }}`)
|
||||
defer delete_template(&tpl2)
|
||||
_, err2 := render(tpl2, Tool_Test_Ctx{p = "/a.css"}, tools = ®)
|
||||
testing.expect(t, err2 != nil, "tool with two args should error")
|
||||
}
|
||||
|
||||
@(test)
|
||||
test_tool_absolute_url_passthrough :: proc(t: ^testing.T) {
|
||||
// A CDN URL has no local source; it should pass through untouched.
|
||||
reg: Tool_Registry
|
||||
url, err := run_tool(®, "anything", "https://cdn.example.com/x.css", 0)
|
||||
testing.expect(t, err == nil, "absolute URL should not error")
|
||||
testing.expect_value(t, url, "https://cdn.example.com/x.css")
|
||||
}
|
||||
|
||||
@(test)
|
||||
test_tool_source_not_found :: proc(t: ^testing.T) {
|
||||
in_dir, out_dir := tool_test_dirs("missing_src")
|
||||
commands := make(map[string]string, context.temp_allocator)
|
||||
commands["cp"] = "cp {{in}} {{out}}"
|
||||
reg := Tool_Registry {
|
||||
commands = commands,
|
||||
input_dir = in_dir,
|
||||
output_dir = out_dir,
|
||||
}
|
||||
|
||||
_, err := run_tool(®, "cp", "/nope.css", 0)
|
||||
testing.expect(t, err != nil, "missing source file should be a hard error")
|
||||
}
|
||||
|
||||
@(test)
|
||||
test_tool_missing_binary :: proc(t: ^testing.T) {
|
||||
in_dir, out_dir := tool_test_dirs("missing_bin")
|
||||
_ = os.write_entire_file_from_string(fmt.tprintf("%s/a.css", in_dir), "body{}")
|
||||
|
||||
commands := make(map[string]string, context.temp_allocator)
|
||||
commands["ghost"] = "thor-tool-does-not-exist-xyz {{in}} {{out}}"
|
||||
reg := Tool_Registry {
|
||||
commands = commands,
|
||||
input_dir = in_dir,
|
||||
output_dir = out_dir,
|
||||
}
|
||||
|
||||
_, err := run_tool(®, "ghost", "/a.css", 0)
|
||||
testing.expect(t, err != nil, "missing binary should be caught by PATH check")
|
||||
}
|
||||
|
||||
@(test)
|
||||
test_tool_runs :: proc(t: ^testing.T) {
|
||||
when ODIN_OS != .Windows {
|
||||
in_dir, out_dir := tool_test_dirs("runs")
|
||||
os.make_directory_all(fmt.tprintf("%s/css", in_dir))
|
||||
_ = os.write_entire_file_from_string(
|
||||
fmt.tprintf("%s/css/main.css", in_dir),
|
||||
"body { color: red }",
|
||||
)
|
||||
|
||||
commands := make(map[string]string, context.temp_allocator)
|
||||
commands["copyit"] = "cp {{in}} {{out}}"
|
||||
reg := Tool_Registry {
|
||||
commands = commands,
|
||||
input_dir = in_dir,
|
||||
output_dir = out_dir,
|
||||
}
|
||||
|
||||
tpl, _ := parse(`{{ p | tool copyit }}`)
|
||||
defer delete_template(&tpl)
|
||||
result, err := render(tpl, Tool_Test_Ctx{p = "/css/main.css"}, tools = ®)
|
||||
defer delete(result)
|
||||
testing.expect(t, err == nil, "cp tool should succeed")
|
||||
testing.expect_value(t, result, "/css/main.css")
|
||||
|
||||
// The tool wrote the output file.
|
||||
written := fmt.tprintf("%s/css/main.css", out_dir)
|
||||
data, rerr := os.read_entire_file_from_path(written, context.temp_allocator)
|
||||
testing.expect(t, rerr == nil, "output file should exist")
|
||||
testing.expect_value(t, string(data), "body { color: red }")
|
||||
|
||||
// The result was memoized; a second call for the same asset returns the
|
||||
// same URL from the registry (mutated through the pointer).
|
||||
url2, err2 := run_tool(®, "copyit", "/css/main.css", 0)
|
||||
testing.expect(t, err2 == nil, "memoized call should not error")
|
||||
testing.expect_value(t, url2, "/css/main.css")
|
||||
testing.expect(t, len(reg.manifest) == 1, "result should be memoized")
|
||||
}
|
||||
}
|
||||
|
||||
// tool_io_case runs `cmd` over a source file containing `content` and returns
|
||||
// the resulting output-file contents. Exercises the in/out ↔ stdin/stdout 2×2.
|
||||
tool_io_case :: proc(name, cmd, content: string) -> (out: string, ok: bool) {
|
||||
in_dir, out_dir := tool_test_dirs(name)
|
||||
_ = os.write_entire_file_from_string(fmt.tprintf("%s/a.css", in_dir), content)
|
||||
commands := make(map[string]string, context.temp_allocator)
|
||||
commands[name] = cmd
|
||||
reg := Tool_Registry {
|
||||
commands = commands,
|
||||
input_dir = in_dir,
|
||||
output_dir = out_dir,
|
||||
}
|
||||
_, err := run_tool(®, name, "/a.css", 0)
|
||||
if err != nil {
|
||||
return "", false
|
||||
}
|
||||
data, rerr := os.read_entire_file_from_path(
|
||||
fmt.tprintf("%s/a.css", out_dir),
|
||||
context.temp_allocator,
|
||||
)
|
||||
return string(data), rerr == nil
|
||||
}
|
||||
|
||||
@(test)
|
||||
test_tool_stdin_stdout :: proc(t: ^testing.T) {
|
||||
when ODIN_OS != .Windows {
|
||||
// `cat` with no args: reads stdin (thor feeds the source), writes stdout
|
||||
// (thor captures and persists it).
|
||||
out, ok := tool_io_case("stream", "cat", "body{a:1}")
|
||||
testing.expect(t, ok, "cat (stdin→stdout) should produce output")
|
||||
testing.expect_value(t, out, "body{a:1}")
|
||||
}
|
||||
}
|
||||
|
||||
@(test)
|
||||
test_tool_in_path_stdout :: proc(t: ^testing.T) {
|
||||
when ODIN_OS != .Windows {
|
||||
// `cat {{in}}`: tool opens the file by path, streams to stdout.
|
||||
out, ok := tool_io_case("in_stream", "cat {{in}}", "body{b:2}")
|
||||
testing.expect(t, ok, "cat {{in}} (file→stdout) should produce output")
|
||||
testing.expect_value(t, out, "body{b:2}")
|
||||
}
|
||||
}
|
||||
|
||||
@(test)
|
||||
test_tool_stdin_out_path :: proc(t: ^testing.T) {
|
||||
when ODIN_OS != .Windows {
|
||||
// `tee {{out}}`: thor feeds stdin, tool writes the file itself.
|
||||
out, ok := tool_io_case("stream_out", "tee {{out}}", "body{c:3}")
|
||||
testing.expect(t, ok, "tee {{out}} (stdin→file) should produce output")
|
||||
testing.expect_value(t, out, "body{c:3}")
|
||||
}
|
||||
}
|
||||
|
||||
@(test)
|
||||
test_tool_minify_section :: proc(t: ^testing.T) {
|
||||
// The command template is itself mustache: {{#minify}} should expand only
|
||||
// when the registry has minify on. Verify via the rendered command string.
|
||||
cmd, err := render_command("prog {{#minify}}--min{{/minify}} {{in}}", "/a", "/b", true, 0)
|
||||
testing.expect(t, err == nil, "render_command should succeed (minify on)")
|
||||
testing.expect(t, strings.contains(cmd, "--min"), "minify section should render when on")
|
||||
|
||||
cmd2, err2 := render_command("prog {{#minify}}--min{{/minify}} {{in}}", "/a", "/b", false, 0)
|
||||
testing.expect(t, err2 == nil, "render_command should succeed (minify off)")
|
||||
testing.expect(t, !strings.contains(cmd2, "--min"), "minify section should be empty when off")
|
||||
}
|
||||
+48
-8
@@ -3,6 +3,7 @@ package main
|
||||
import "mustache"
|
||||
|
||||
import diags "diagnostics"
|
||||
import ts "treesitter"
|
||||
|
||||
import "core:encoding/json"
|
||||
import "core:fmt"
|
||||
@@ -138,8 +139,14 @@ render_template :: proc(
|
||||
ctx: Template_Context,
|
||||
partials: map[string]mustache.Template,
|
||||
reported_errors: ^map[string]bool,
|
||||
tools: ^mustache.Tool_Registry,
|
||||
) -> string {
|
||||
result, err := mustache.render(content_tpl, []any{ctx.site, ctx.page, ctx}, partials)
|
||||
result, err := mustache.render(
|
||||
content_tpl,
|
||||
[]any{ctx.site, ctx.page, ctx},
|
||||
partials,
|
||||
tools = tools,
|
||||
)
|
||||
if err != nil {
|
||||
formatted := mustache.format_render_error(
|
||||
err,
|
||||
@@ -184,6 +191,16 @@ render_site :: proc(site: ^Site) {
|
||||
timezone = site.tz,
|
||||
}
|
||||
|
||||
// Prepare the `tool` pipe registry for this build (commands were filled from
|
||||
// thor.json at config time). A pointer to it is threaded into every
|
||||
// mustache.render call.
|
||||
mustache.tool_registry_init(
|
||||
&site.tool_registry,
|
||||
site.assets_dir,
|
||||
site.output_dir,
|
||||
.Minify in site.features,
|
||||
)
|
||||
|
||||
// Find home page
|
||||
home: Page
|
||||
has_home := false
|
||||
@@ -205,6 +222,24 @@ render_site :: proc(site: ^Site) {
|
||||
}
|
||||
}
|
||||
|
||||
// Parsers for minification, created once and reused across every page
|
||||
// (render is single-threaded). Owned here; freed when render_site returns.
|
||||
html_parser, css_parser: ts.Parser
|
||||
if .Minify in site.features {
|
||||
if g, gok := ts.grammar("html"); gok {
|
||||
html_parser = ts.open_parser(g)
|
||||
}
|
||||
if g, gok := ts.grammar("css"); gok {
|
||||
css_parser = ts.open_parser(g)
|
||||
}
|
||||
}
|
||||
defer if html_parser != nil {
|
||||
ts.parser_delete(html_parser)
|
||||
}
|
||||
defer if css_parser != nil {
|
||||
ts.parser_delete(css_parser)
|
||||
}
|
||||
|
||||
// Render individual content pages (skip all index pages)
|
||||
for page in pages {
|
||||
if page._is_index {
|
||||
@@ -213,7 +248,9 @@ render_site :: proc(site: ^Site) {
|
||||
tpl := get_template(&site.vfs, page.layout, &template_cache)
|
||||
html := render_page_html(page, site, tpl, partials, ctx, &errors)
|
||||
if .Minify in site.features {
|
||||
html = minify_html(html)
|
||||
if m, mok := minify_html(html_parser, css_parser, html); mok {
|
||||
html = m
|
||||
}
|
||||
}
|
||||
write_page(site.output_dir, page.permalink, html)
|
||||
}
|
||||
@@ -243,7 +280,9 @@ render_site :: proc(site: ^Site) {
|
||||
&errors,
|
||||
)
|
||||
if .Minify in site.features {
|
||||
html = minify_html(html)
|
||||
if m, mok := minify_html(html_parser, css_parser, html); mok {
|
||||
html = m
|
||||
}
|
||||
}
|
||||
write_page(site.output_dir, fmt.aprintf("/%s/", section), html)
|
||||
}
|
||||
@@ -253,7 +292,9 @@ render_site :: proc(site: ^Site) {
|
||||
home_tpl := get_template(&site.vfs, "home", &template_cache)
|
||||
home_html := render_home_html(home, site, home_tpl, partials, ctx, &errors)
|
||||
if .Minify in site.features {
|
||||
home_html = minify_html(home_html)
|
||||
if m, mok := minify_html(html_parser, css_parser, home_html); mok {
|
||||
home_html = m
|
||||
}
|
||||
}
|
||||
write_file(fmt.tprintf("%s/index.html", site.output_dir), home_html)
|
||||
}
|
||||
@@ -292,7 +333,7 @@ render_page_html :: proc(
|
||||
ctx.page = page
|
||||
ctx.og = og_for_page(site.og, page)
|
||||
ctx.params = merge_params(site.params, page.params)
|
||||
return render_template(content_tpl, ctx, partials, seen)
|
||||
return render_template(content_tpl, ctx, partials, seen, &site.tool_registry)
|
||||
}
|
||||
|
||||
render_home_html :: proc(
|
||||
@@ -316,7 +357,7 @@ render_home_html :: proc(
|
||||
ctx.og = og_for_page(site.og, home)
|
||||
ctx.params = merge_params(site.params, home.params)
|
||||
|
||||
return render_template(content_tpl, ctx, partials, seen)
|
||||
return render_template(content_tpl, ctx, partials, seen, &site.tool_registry)
|
||||
}
|
||||
|
||||
render_section :: proc(
|
||||
@@ -355,7 +396,7 @@ render_section :: proc(
|
||||
}
|
||||
ctx.posts = posts
|
||||
ctx.params = merge_params(site.params, ctx.page.params)
|
||||
return render_template(content_tpl, ctx, partials, seen)
|
||||
return render_template(content_tpl, ctx, partials, seen, &site.tool_registry)
|
||||
}
|
||||
|
||||
load_partials :: proc(vfs: ^VFS) -> map[string]mustache.Template {
|
||||
@@ -445,4 +486,3 @@ write_file :: proc(path: string, html: string) {
|
||||
log.errorf("cannot write %s: %v", path, err)
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -10,6 +10,7 @@ import "core:time/datetime"
|
||||
import "core:time/timezone"
|
||||
|
||||
import md "markdown"
|
||||
import "mustache"
|
||||
|
||||
|
||||
// Site_Context holds the site date that is accessible in templates.
|
||||
@@ -17,6 +18,12 @@ Site_Context :: struct {
|
||||
title: string,
|
||||
description: string,
|
||||
base_url: string,
|
||||
// base_path is the path component of base_url, normalized to a leading slash
|
||||
// with no trailing slash ("" for root-hosted sites). Templates prefix internal
|
||||
// root-relative links with it (directly, or via the `rel_url` pipe) so that
|
||||
// subpath deployments — e.g. project pages at https://user.github.io/proj —
|
||||
// resolve correctly.
|
||||
base_path: string,
|
||||
params: json.Object,
|
||||
og: Open_Graph,
|
||||
menus: map[string][]Menu_Entry,
|
||||
@@ -42,6 +49,11 @@ Site :: struct {
|
||||
tz: ^datetime.TZ_Region,
|
||||
grammars: string,
|
||||
queries: string,
|
||||
// tool_registry is the `tool` pipe's state: `commands` is filled from
|
||||
// thor.json at config time; the per-build I/O dirs, minify flag, and memo
|
||||
// are set by mustache.tool_registry_init in render_site. A pointer is
|
||||
// threaded into mustache.render.
|
||||
tool_registry: mustache.Tool_Registry,
|
||||
}
|
||||
|
||||
Date_Preferences :: struct {
|
||||
@@ -73,6 +85,7 @@ Config_File :: struct {
|
||||
date: Date_Preferences,
|
||||
grammars: string,
|
||||
queries: string,
|
||||
tools: map[string]string,
|
||||
}
|
||||
|
||||
// Configuration loaded from command line arguments. Gets folded in to Site
|
||||
@@ -127,6 +140,7 @@ init_site :: proc(site: ^Site, flags: Flags) {
|
||||
}
|
||||
|
||||
site_apply_cli_flags(site, flags)
|
||||
site.base_path = site_base_path(site.base_url, site_allocator(site))
|
||||
site.config_path = path
|
||||
|
||||
site.og = og_for_site(site)
|
||||
@@ -211,6 +225,52 @@ site_apply_config :: proc(site: ^Site, config: Config_File, config_dir: string)
|
||||
|
||||
site.grammars = expand_path(config.grammars, site_allocator(site))
|
||||
site.queries = expand_path(config.queries, site_allocator(site))
|
||||
|
||||
if config.tools != nil {
|
||||
site.tool_registry.commands = config.tools
|
||||
}
|
||||
}
|
||||
|
||||
// site_base_path extracts the path component of base_url, normalized to a
|
||||
// leading slash with no trailing slash. Returns "" for root-hosted sites.
|
||||
//
|
||||
// "https://ex.com/thor/" → "/thor"
|
||||
// "https://ex.com/a/b" → "/a/b"
|
||||
// "https://ex.com" → ""
|
||||
// "http://localhost:8080/" → ""
|
||||
site_base_path :: proc(base_url: string, allocator := context.allocator) -> string {
|
||||
url := base_url
|
||||
|
||||
// Strip scheme so the first '/' we find is the start of the path, not the
|
||||
// "//" separating scheme and host.
|
||||
if idx := strings.index(url, "://"); idx >= 0 {
|
||||
url = url[idx + 3:]
|
||||
}
|
||||
|
||||
// The path begins at the first '/' after the host.
|
||||
slash := strings.index_byte(url, '/')
|
||||
if slash < 0 {
|
||||
return ""
|
||||
}
|
||||
path := url[slash:]
|
||||
|
||||
// Drop any query or fragment.
|
||||
if idx := strings.index_byte(path, '?'); idx >= 0 {
|
||||
path = path[:idx]
|
||||
}
|
||||
if idx := strings.index_byte(path, '#'); idx >= 0 {
|
||||
path = path[:idx]
|
||||
}
|
||||
|
||||
// Trim trailing slashes; a bare "/" normalizes to "".
|
||||
for len(path) > 0 && path[len(path) - 1] == '/' {
|
||||
path = path[:len(path) - 1]
|
||||
}
|
||||
if path == "" {
|
||||
return ""
|
||||
}
|
||||
|
||||
return strings.clone(path, allocator)
|
||||
}
|
||||
|
||||
// expand_path replaces a leading ~/ with $HOME/.
|
||||
|
||||
@@ -0,0 +1,34 @@
|
||||
body>header {
|
||||
background: var(--pico-primary-background);
|
||||
position: sticky;
|
||||
top: 0;
|
||||
|
||||
nav>ul li a {
|
||||
color: var(--pico-primary-inverse);
|
||||
}
|
||||
}
|
||||
|
||||
/* TODO: Should probaby use a container for the page content like Odin does */
|
||||
.toc {
|
||||
position: fixed;
|
||||
height: calc(100vh - 116px);
|
||||
width: 280px;
|
||||
top: 116px;
|
||||
left: 0;
|
||||
overflow-y: auto;
|
||||
--pico-nav-element-spacing-vertical: 0.25rem;
|
||||
|
||||
& ul {
|
||||
display: block;
|
||||
|
||||
&>li {
|
||||
margin-left: 1rem;
|
||||
width: 100%;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
main>article {
|
||||
width: 60%;
|
||||
margin: auto;
|
||||
}
|
||||
+108
-4
@@ -17,6 +17,8 @@ etc.
|
||||
TODO: Use consistent spacing with mustache tags
|
||||
-->
|
||||
|
||||
[TOC]
|
||||
|
||||
## Introduction
|
||||
|
||||
This guide assumes you have either read [The Guide](../guide), or have built a [Hugo](https://gohugo.io) site before. It also assumes you have a basic knowledge of HTML and CSS.
|
||||
@@ -48,12 +50,36 @@ The highlight extension enables server-side syntax highlighting, powered by Tree
|
||||
|
||||
If enabled, `minify` will perform simple whitespace removal on all your output `.html` files, and any `.css` files in your `assets` directories. Minifying JavaScript is not supported (yet).
|
||||
|
||||
TODO: Do we minify inline css?
|
||||
Inline CSS will be minified.
|
||||
|
||||
#### Sections
|
||||
|
||||
The `sections` extension wraps each of your content sections in a `<section>` block. Sections are opened just before each heading, and closed before the next heading, or the end the document, whichever comes first.
|
||||
|
||||
```markdown
|
||||
<!-- Example -->
|
||||
## First Section
|
||||
|
||||
content
|
||||
|
||||
## Second Section
|
||||
|
||||
content
|
||||
```
|
||||
|
||||
Becomes:
|
||||
|
||||
```html
|
||||
<section>
|
||||
<h2>First Section</h2>
|
||||
<p>content</p>
|
||||
</section>
|
||||
<section>
|
||||
<h2>Second Section</h2>
|
||||
<p>content</p>
|
||||
</section>
|
||||
```
|
||||
|
||||
### Opt-Out Features
|
||||
|
||||
- emoji
|
||||
@@ -272,6 +298,14 @@ When building your page(s), the following keys are accessible to your template f
|
||||
|
||||
: The HTML rendered page content
|
||||
|
||||
`page.url`
|
||||
|
||||
: The full url to the page. .e.g. `https://example.com/about`
|
||||
|
||||
`page.rel_url`
|
||||
|
||||
: The url to the page, minus the `site.base_url`, e.g. `/about`.
|
||||
|
||||
TODO: write
|
||||
|
||||
#### Site
|
||||
@@ -366,6 +400,15 @@ To get around this, Thor extends mustache to include a handful of data filters i
|
||||
|
||||
: Used to display a date in a particular format. See [DateTimes](#datetimes). If no format is given, the default will be used.
|
||||
|
||||
`rel_url`
|
||||
|
||||
: Should only be used by theme designers or when building a site where the homepage is not the website root (i.e. https://example.com/my-cool-site/).
|
||||
Will take a url like `/about` and convert it to `https://example.com/my-cool-site/about`
|
||||
|
||||
`tool`
|
||||
|
||||
: Allows you to run your assets / content through any external tools you've configured. Most commonly used for css processors like Tailwind or Sass. See [Tools](#tools) for more info.
|
||||
|
||||
#### Chaining Pipes
|
||||
|
||||
Thor allows you to chain two or more pipes, to allow complex data manipulation. However you are limited to no more than **8 pipes**[^pipes] for any given tag. If you believe you need more than 8 pipes, please [open an issue](../issues) with a **clear** and **concrete** example of the problem you are facing.
|
||||
@@ -552,14 +595,73 @@ The following keys are available:
|
||||
|
||||
Menu configuration for the site. Note that (unlike Hugo) if your configuration contains this key, frontmatter `menus` will be ignored. See [menus](#menus) for more info.
|
||||
|
||||
`tools`
|
||||
|
||||
Allows you to configure external content/asset processing tools. Too... See [Tools](#tools) for more info.
|
||||
|
||||
## Tools
|
||||
|
||||
Configuring tools will allow you to create custom pipelines for your content and assets. For example, you could create a tailwindcss tool to pre-process your css files. You could also create a conversion tool to make all your images be a certain format or maximum size.
|
||||
|
||||
Tools are configured in your config file and utilized in your template files:
|
||||
|
||||
> [!DANGER]: You should never run a tool you have not configured yourself, as running arbetrary commands can be extremely dangerous.
|
||||
|
||||
```mustache
|
||||
{{!layouts/partials/styles.css}}
|
||||
{{#params.stylesheets}}<link rel="stylesheet" href="{{. | tool tailwind}}">
|
||||
{{/params.stylesheets}}
|
||||
```
|
||||
|
||||
Essentially, each tool is just a bash-style commandline string with a few special variables:
|
||||
|
||||
<!-- TODO: It might be a terrible idea to combine mustache templates with shell scripts :/ -->
|
||||
|
||||
```json5
|
||||
// thor.json
|
||||
{
|
||||
"tools": {
|
||||
// For demonstration purposes only, not recommended (see below).
|
||||
"tailwind": "tailwindcss -i {{in}} -o {{out}}",
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Both `{{in}}` and `{{out}}` exist to support tools that can't read from `stdin` and/or can't write to `stdout`. If your tool can do those things, it is ***highly recommended*** to avoid `{{in}}` + `{{out}}`, as each requires thor to make **a full round-trip** to the file system, rather than keeping everything in memory.
|
||||
|
||||
```json5
|
||||
// thor.json
|
||||
{
|
||||
"tools": {
|
||||
// The recommended way to configure Tailwind 4.3.3
|
||||
"tailwind": "tailwindcss {{#minify}}--minify{{/minify}} -i -",
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Special Variables
|
||||
|
||||
`in`
|
||||
|
||||
: The path to a file created by thor, containing the content that will be processed by your tool.
|
||||
|
||||
`out`
|
||||
|
||||
: The path to a file, created by your tool, that will be represents the file output by the tool, that will be ingested by thor.
|
||||
|
||||
`minify`
|
||||
|
||||
: True if the `minify` config option has been set, or `--production` flag has been passed.
|
||||
|
||||
- it is up to you to have the necessary binaries installed - thor won't do it for you.
|
||||
- every tool you want to use must be configured - there is no default <!-- TODO: Should there be? Perhaps we could include support for them in a future, interactive site builder setup TUI. -->
|
||||
|
||||
## Open Graph
|
||||
|
||||
Thor supports [Open Graph](https://ogp.me/) tags, allowing you to customize how your site looks when linked to on social media or in apps like Discord. It is accessible through the `{{>opengraph}}` template, which should be automatically included in your header by your theme.
|
||||
|
||||
```html
|
||||
TODO: {{>opengraph shorcode here}}
|
||||
TODO: {{opengraph shorcode here}}
|
||||
```
|
||||
|
||||
The options are sourced [using the context stack], first from your page frontmatter, and then from your site config. Thor will attempt to infer as much information as possible, to keep you from needing to set them explictly.
|
||||
@@ -591,9 +693,11 @@ TODO: Modules
|
||||
|
||||
TODO: Themes
|
||||
|
||||
## Zen
|
||||
|
||||
> The guding principals behind Thor.
|
||||
<hgroup>
|
||||
<h2 id="zen">Zen</h2>
|
||||
<p>The guding principals behind Thor.</p>
|
||||
</hgroup>
|
||||
|
||||
Simpler is Better
|
||||
|
||||
|
||||
@@ -17,7 +17,7 @@ At the heart of any good SSG is the content files. Thor supports markdown (`.md`
|
||||
|
||||
Let's make a home page - copy the following text and place it into `./index.md`
|
||||
|
||||
```md
|
||||
```markdown
|
||||
# Hello, World!
|
||||
|
||||
Welcome to your new site.
|
||||
|
||||
+21
-4
@@ -1,13 +1,30 @@
|
||||
{
|
||||
"date": {
|
||||
"timezone": "America/New_York"
|
||||
"timezone": "America/New_York",
|
||||
"format": "January 2, 2006"
|
||||
},
|
||||
"markdown_extensions": {
|
||||
"footnotes": true
|
||||
"footnotes": true,
|
||||
"highlight": true
|
||||
},
|
||||
"menus": {
|
||||
"main": [
|
||||
{
|
||||
"name": "GitHub",
|
||||
"url": "https://github.com/sbrow/thor",
|
||||
"weight": 20
|
||||
}
|
||||
]
|
||||
},
|
||||
"params": {
|
||||
"author": {
|
||||
"name": "Spencer Brower"
|
||||
}
|
||||
}
|
||||
},
|
||||
"stylesheets": [
|
||||
"https://cdn.jsdelivr.net/npm/@picocss/pico@2/css/pico.blue.min.css",
|
||||
"/css/main.css"
|
||||
]
|
||||
},
|
||||
"grammars": "~/.config/helix/runtime/grammars/",
|
||||
"queries": "/nix/store/n9da8d007ygbgsx983jr3ar3wb1fsh6q-helix-25.07.1/lib/runtime/queries"
|
||||
}
|
||||
+182
-128
@@ -7,6 +7,7 @@ import "core:mem"
|
||||
import "core:os"
|
||||
import "core:strings"
|
||||
import "core:sync"
|
||||
import si "core:sys/info"
|
||||
import "core:thread"
|
||||
|
||||
grammar_dir: string
|
||||
@@ -120,11 +121,15 @@ foreign css_grammar {
|
||||
tree_sitter_css :: proc() -> Language ---
|
||||
}
|
||||
|
||||
Grammar_Cache :: struct {
|
||||
// A Grammar bundles only the immutable, shareable tree-sitter objects for a
|
||||
// language: `language` (from tree_sitter_x() or a dlopen'd .so) and `query` (the
|
||||
// compiled highlight query). Both are immutable after construction and safe to
|
||||
// read from any number of threads. The stateful, single-thread-use objects —
|
||||
// parser and query cursor — are deliberately NOT stored here; the caller creates
|
||||
// and owns those (see `open_parser`). See GRAMMAR_CACHE_THREADING.md.
|
||||
Grammar :: struct {
|
||||
language: Language,
|
||||
parser: Parser,
|
||||
query: Query,
|
||||
cursor: Query_Cursor,
|
||||
query_failed: bool,
|
||||
}
|
||||
|
||||
@@ -132,17 +137,98 @@ Get_Language_Proc :: #type proc() -> Language
|
||||
|
||||
SPALL :: #config(SPALL, false)
|
||||
|
||||
grammar_store: Grammar_Store
|
||||
cache_mutex: sync.Mutex
|
||||
|
||||
Grammar_Store :: struct {
|
||||
cache: map[string]^Grammar_Cache,
|
||||
// registry is the process-lifetime cache of immutable Grammars, keyed by language
|
||||
// name. It is the library's ONLY shared mutable state; `mu` serializes all access.
|
||||
// A nil value memoizes "tried and unavailable" so misses don't re-dlopen. Parsers
|
||||
// live in the caller, so the registry holds nothing single-thread-use.
|
||||
Grammar_Registry :: struct {
|
||||
mu: sync.Mutex,
|
||||
grammars: map[string]^Grammar,
|
||||
allocator: mem.Allocator,
|
||||
}
|
||||
|
||||
registry: Grammar_Registry
|
||||
|
||||
// init_persistent is optional — grammar() lazily initializes the registry on first
|
||||
// use — but callers (main) may call it to bind the cache early. The allocator is
|
||||
// pinned to the OS heap, NOT context.allocator, so the cache survives any transient
|
||||
// allocator (per-build arena, per-test tracking) that happens to be live.
|
||||
init_persistent :: proc() {
|
||||
grammar_store.allocator = context.allocator
|
||||
grammar_store.cache = make(map[string]^Grammar_Cache, grammar_store.allocator)
|
||||
sync.mutex_lock(®istry.mu)
|
||||
defer sync.mutex_unlock(®istry.mu)
|
||||
ensure_registry()
|
||||
}
|
||||
|
||||
// ensure_registry binds the registry to the heap on first use. Caller must hold mu.
|
||||
@(private)
|
||||
ensure_registry :: proc() {
|
||||
if registry.grammars == nil {
|
||||
registry.allocator = os.heap_allocator()
|
||||
registry.grammars = make(map[string]^Grammar, registry.allocator)
|
||||
}
|
||||
}
|
||||
|
||||
// grammar returns the immutable Grammar for `lang`, loading it on first use.
|
||||
// Thread-safe: concurrent callers serialize on `mu`. Returns (nil, false) when the
|
||||
// grammar cannot be loaded. The returned pointer is immutable — callers may cache
|
||||
// and read it (its `language`/`query`) without any further locking.
|
||||
grammar :: proc(lang: string) -> (^Grammar, bool) {
|
||||
sync.mutex_lock(®istry.mu)
|
||||
defer sync.mutex_unlock(®istry.mu)
|
||||
ensure_registry()
|
||||
if g, seen := registry.grammars[lang]; seen {
|
||||
return g, g != nil
|
||||
}
|
||||
g := build_grammar(lang)
|
||||
registry.grammars[lang] = g
|
||||
return g, g != nil
|
||||
}
|
||||
|
||||
// build_grammar loads a language and compiles its highlight query. It touches no
|
||||
// shared mutable state — only the set-once `registry.allocator` (for the heap
|
||||
// `new`), read-only globals, and the per-thread temp allocator — so it is safe to
|
||||
// run OFF the lock and concurrently, provided `ensure_registry` has already run
|
||||
// (see preload_grammars). Returns nil if the language itself cannot be loaded; a
|
||||
// grammar whose query is absent/failed is still returned (parsing works,
|
||||
// highlighting does not).
|
||||
@(private)
|
||||
build_grammar :: proc(lang: string) -> ^Grammar {
|
||||
language, ok := load_language(lang)
|
||||
if !ok {
|
||||
return nil
|
||||
}
|
||||
g := new(Grammar, registry.allocator)
|
||||
g.language = language
|
||||
if query, qok := compile_query(lang, language); qok {
|
||||
g.query = query
|
||||
} else {
|
||||
g.query_failed = true
|
||||
}
|
||||
return g
|
||||
}
|
||||
|
||||
// open_parser creates a fresh parser bound to the grammar's language. The CALLER
|
||||
// owns the returned parser and must free it with parser_delete. This is the seam
|
||||
// for parser lifetime: today callers open one and reuse it across a batch of
|
||||
// parses on a single thread (render opens an html+css pair per render_site;
|
||||
// assets opens one per copy_assets_dir), which is safe because nothing is shared.
|
||||
// A future pooling or thread-local strategy can be dropped in here without
|
||||
// touching any call site. Returns nil on failure.
|
||||
open_parser :: proc(g: ^Grammar) -> Parser {
|
||||
if g == nil {
|
||||
return nil
|
||||
}
|
||||
parser := parser_new()
|
||||
if parser == nil {
|
||||
log.errorf("treesitter: cannot create parser")
|
||||
return nil
|
||||
}
|
||||
if !parser_set_language(parser, g.language) {
|
||||
log.errorf("treesitter: ABI mismatch setting parser language")
|
||||
parser_delete(parser)
|
||||
return nil
|
||||
}
|
||||
return parser
|
||||
}
|
||||
|
||||
when SPALL {
|
||||
@@ -153,6 +239,19 @@ when SPALL {
|
||||
_thread_init = init
|
||||
_thread_cleanup = cleanup
|
||||
}
|
||||
|
||||
// Pool-shaped adapters: the thread pool calls these once per worker thread at
|
||||
// startup/shutdown, so each worker's parses land in their own SPALL thread.
|
||||
spall_pool_thread_init :: proc(t: ^thread.Thread, data: rawptr) {
|
||||
if _thread_init != nil {
|
||||
_thread_init()
|
||||
}
|
||||
}
|
||||
spall_pool_thread_fini :: proc(t: ^thread.Thread, data: rawptr) {
|
||||
if _thread_cleanup != nil {
|
||||
_thread_cleanup()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
builtin_language :: proc(lang: string) -> (language: Language, ok: bool) {
|
||||
@@ -170,7 +269,7 @@ builtin_language :: proc(lang: string) -> (language: Language, ok: bool) {
|
||||
// load_query returns the highlight query source for a language. Builtin
|
||||
// languages (html/css) are baked into the binary via `#load`; all others are
|
||||
// read from the runtime `query_dir`. `path` is the on-disk location for
|
||||
// diagnostics ("(builtin)" for embedded queries). Mirrors `ensure_parser`.
|
||||
// diagnostics ("(builtin)" for embedded queries). Mirrors `load_language`.
|
||||
load_query :: proc(lang: string) -> (src: string, path: string, ok: bool) {
|
||||
switch lang {
|
||||
case "html":
|
||||
@@ -219,44 +318,7 @@ load_language :: proc(lang: string) -> (language: Language, ok: bool) {
|
||||
return
|
||||
}
|
||||
|
||||
ensure_parser :: proc(lang: string) -> ^Grammar_Cache {
|
||||
if cached, ok := grammar_store.cache[lang]; ok {
|
||||
return cached
|
||||
}
|
||||
|
||||
grammar_store.cache[lang] = nil
|
||||
|
||||
language, ok := load_language(lang)
|
||||
if !ok {
|
||||
return nil
|
||||
}
|
||||
|
||||
parser := parser_new()
|
||||
if parser == nil {
|
||||
log.errorf("treesitter: cannot create parser for %s", lang)
|
||||
return nil
|
||||
}
|
||||
if !parser_set_language(parser, language) {
|
||||
log.errorf("treesitter: ABI mismatch for %s grammar", lang)
|
||||
parser_delete(parser)
|
||||
return nil
|
||||
}
|
||||
|
||||
gc := new(Grammar_Cache, grammar_store.allocator)
|
||||
gc.language = language
|
||||
gc.parser = parser
|
||||
grammar_store.cache[lang] = gc
|
||||
return gc
|
||||
}
|
||||
|
||||
compile_query :: proc(
|
||||
lang: string,
|
||||
language: Language,
|
||||
) -> (
|
||||
query: Query,
|
||||
cursor: Query_Cursor,
|
||||
ok: bool,
|
||||
) {
|
||||
compile_query :: proc(lang: string, language: Language) -> (query: Query, ok: bool) {
|
||||
query_src, query_path, qok := load_query(lang)
|
||||
if !qok {
|
||||
return
|
||||
@@ -318,113 +380,104 @@ compile_query :: proc(
|
||||
return
|
||||
}
|
||||
|
||||
cursor = query_cursor_new()
|
||||
ok = true
|
||||
return
|
||||
}
|
||||
|
||||
load_grammar :: proc(lang: string) -> ^Grammar_Cache {
|
||||
gc := ensure_parser(lang)
|
||||
if gc == nil {
|
||||
return nil
|
||||
}
|
||||
if gc.query != nil {
|
||||
return gc
|
||||
}
|
||||
if gc.query_failed {
|
||||
return nil
|
||||
// PRELOAD_MAX_WORKERS caps the pool when the CPU count is unknown or absurd.
|
||||
PRELOAD_MAX_WORKERS :: 4
|
||||
|
||||
// Preload_Task is the per-language work item handed to a pool worker.
|
||||
Preload_Task :: struct {
|
||||
lang: string,
|
||||
}
|
||||
|
||||
query, cursor, ok := compile_query(lang, gc.language)
|
||||
if !ok {
|
||||
gc.query_failed = true
|
||||
return nil
|
||||
}
|
||||
|
||||
gc.query = query
|
||||
gc.cursor = cursor
|
||||
return gc
|
||||
}
|
||||
|
||||
preload_grammar :: proc(lang: string) -> ^Grammar_Cache {
|
||||
language, ok := load_language(lang)
|
||||
if !ok {
|
||||
return nil
|
||||
}
|
||||
|
||||
parser := parser_new()
|
||||
if parser == nil {
|
||||
log.errorf("treesitter: cannot create parser for %s", lang)
|
||||
return nil
|
||||
}
|
||||
if !parser_set_language(parser, language) {
|
||||
log.errorf("treesitter: ABI mismatch for %s grammar", lang)
|
||||
parser_delete(parser)
|
||||
return nil
|
||||
}
|
||||
|
||||
gc := new(Grammar_Cache, grammar_store.allocator)
|
||||
gc.language = language
|
||||
gc.parser = parser
|
||||
|
||||
query, cursor, qok := compile_query(lang, language)
|
||||
if !qok {
|
||||
gc.query_failed = true
|
||||
return gc
|
||||
}
|
||||
|
||||
gc.query = query
|
||||
gc.cursor = cursor
|
||||
return gc
|
||||
// preload_worker builds one grammar off the lock (dlopen + query compile — the
|
||||
// expensive, parallelizable part), then takes `mu` only to publish the finished
|
||||
// pointer. Publishing nil memoizes a failed load, matching grammar()/build_grammar.
|
||||
@(private)
|
||||
preload_worker :: proc(task: thread.Task) {
|
||||
pt := cast(^Preload_Task)task.data
|
||||
g := build_grammar(pt.lang)
|
||||
sync.mutex_lock(®istry.mu)
|
||||
registry.grammars[pt.lang] = g
|
||||
sync.mutex_unlock(®istry.mu)
|
||||
}
|
||||
|
||||
// preload_grammars warms the registry for the given languages ahead of the render
|
||||
// loop, so the first page that needs one doesn't pay the load cost inline. The
|
||||
// expensive per-language work (dlopen + compile_query) runs concurrently on a
|
||||
// bounded thread pool. Builtins are always available lazily and are skipped.
|
||||
//
|
||||
// INVARIANT: preload runs to completion before any concurrent lazy grammar() use
|
||||
// (render is single-threaded and runs afterward), so the registry is never read
|
||||
// while workers publish. Workers still take `mu` for the map insert, keeping the
|
||||
// "registry is only ever mutated under mu" discipline uniform with grammar().
|
||||
preload_grammars :: proc(languages: []string) {
|
||||
if len(languages) == 0 {
|
||||
return
|
||||
}
|
||||
|
||||
// Filter out already-loaded languages (watch mode reuse)
|
||||
// Snapshot (main thread, under mu): pin the registry allocator before any
|
||||
// worker reads it, and collect the languages still needing a load. Dedupes
|
||||
// against builtins and already-cached grammars (watch-mode reuse).
|
||||
sync.mutex_lock(®istry.mu)
|
||||
ensure_registry()
|
||||
to_load := make([dynamic]string, 0, len(languages), context.temp_allocator)
|
||||
for lang in languages {
|
||||
if cached, ok := grammar_store.cache[lang]; ok && cached != nil {
|
||||
if _, bok := builtin_language(lang); bok {
|
||||
continue
|
||||
}
|
||||
if _, bok := builtin_language(lang); bok {
|
||||
if _, seen := registry.grammars[lang]; seen {
|
||||
continue
|
||||
}
|
||||
append(&to_load, lang)
|
||||
}
|
||||
sync.mutex_unlock(®istry.mu)
|
||||
|
||||
if len(to_load) == 0 {
|
||||
return
|
||||
}
|
||||
|
||||
threads := make([]^thread.Thread, len(to_load), context.temp_allocator)
|
||||
for i in 0 ..< len(to_load) {
|
||||
threads[i] = thread.create_and_start_with_poly_data(to_load[i], grammar_worker)
|
||||
}
|
||||
for t in threads {
|
||||
thread.join(t)
|
||||
thread.destroy(t)
|
||||
}
|
||||
// Bind the pool to the logical core count (load is dlopen/CPU-bound), but
|
||||
// never more workers than tasks, never fewer than one.
|
||||
worker_count: int = ---
|
||||
if _, logical, ok := si.cpu_core_count(); ok {
|
||||
assert(logical > 0)
|
||||
worker_count = min(logical, len(to_load))
|
||||
} else {
|
||||
worker_count = min(PRELOAD_MAX_WORKERS, len(to_load))
|
||||
}
|
||||
|
||||
grammar_worker :: proc(lang: string) {
|
||||
// Serial fast path: a single worker gains nothing from the pool machinery.
|
||||
if worker_count == 1 {
|
||||
for lang in to_load {
|
||||
g := build_grammar(lang)
|
||||
sync.mutex_lock(®istry.mu)
|
||||
registry.grammars[lang] = g
|
||||
sync.mutex_unlock(®istry.mu)
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
// The pool appends to its own `tasks_done` from worker threads, so its
|
||||
// bookkeeping allocator MUST be thread-safe — use the heap, not the (possibly
|
||||
// arena) context allocator that is live during a build.
|
||||
pool: thread.Pool
|
||||
init_proc: thread.Thread_Init_Proc = nil
|
||||
fini_proc: thread.Thread_Init_Proc = nil
|
||||
when SPALL {
|
||||
if _thread_init != nil {
|
||||
_thread_init()
|
||||
}
|
||||
defer if _thread_cleanup != nil {
|
||||
_thread_cleanup()
|
||||
}
|
||||
init_proc = spall_pool_thread_init
|
||||
fini_proc = spall_pool_thread_fini
|
||||
}
|
||||
thread.pool_init(&pool, os.heap_allocator(), worker_count, init_proc, nil, fini_proc, nil)
|
||||
defer thread.pool_destroy(&pool)
|
||||
|
||||
gc := preload_grammar(lang)
|
||||
if gc != nil {
|
||||
sync.mutex_lock(&cache_mutex)
|
||||
grammar_store.cache[lang] = gc
|
||||
sync.mutex_unlock(&cache_mutex)
|
||||
tasks := make([]Preload_Task, len(to_load), context.temp_allocator)
|
||||
thread.pool_start(&pool)
|
||||
for lang, i in to_load {
|
||||
tasks[i].lang = lang
|
||||
// Per-task context allocator = heap, so any stray context.allocator use
|
||||
// inside a worker is thread-safe.
|
||||
thread.pool_add_task(&pool, os.heap_allocator(), preload_worker, &tasks[i], i)
|
||||
}
|
||||
thread.pool_finish(&pool) // barrier: all workers joined before we return
|
||||
}
|
||||
|
||||
extract_query_token :: proc(src: []byte, offset: u32) -> string {
|
||||
@@ -459,3 +512,4 @@ helix_version_from_path :: proc(path: string) -> string {
|
||||
if end <= start do return ""
|
||||
return path[start:end]
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user