spencer/thor
1
0
Fork 0
mirror of https://github.com/sbrow/thor.git synced 2026-10-25 10:11:55 -04:00
Code Issues Packages Projects Releases Wiki Activity

Compare commits

base: spencer/thor:main
Branches Tags
spencer/thor:main
..
compare: spencer/thor:0dc6cc117f88c1af3249b9a145c346658eac3aa4
Branches Tags
spencer/thor:main
1 Commits
main .. 0dc6cc117f
Author SHA1 Message Date
Spencer Brower 0dc6cc117f feat: Added robots.txt 2026-07-11 10:45:22 -04:00
186 changed files with 736 additions and 23707 deletions
Expand all files Collapse all files
.envrc
+1 -1
View File
@@ -1,3 +1,3 @@
use flake use flake
export PATH="$PATH:./" export PATH="$PATH:./vendor/bin"
.github/workflows/pages.yml
-78
View File
@@ -1,78 +0,0 @@
# Custom Pages build: compiles Thor from source via the Nix flake, uses it to
# generate thor/site/, and publishes the result. Replaces the default Jekyll build.
name: Deploy site to GitHub Pages
on:
push:
branches: [main]
# Only rebuild when something that affects the output changes.
paths:
- "site/**"
- "**/*.odin"
- "defaults/**"
- "flake.nix"
- "flake.lock"
- ".github/workflows/pages.yml"
workflow_dispatch:
permissions:
contents: read
pages: write
id-token: write
# Allow one concurrent deployment; don't cancel an in-progress publish.
concurrency:
group: pages
cancel-in-progress: false
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Install Nix
uses: cachix/install-nix-action@v30
with:
extra_nix_config: |
experimental-features = nix-command flakes
# Free cache: snapshots /nix/store into GitHub's Actions cache. The key is
# hashed over the inputs that change Thor's output, so a site-content-only
# change is a full hit (Thor not recompiled), while an Odin source change
# restores the nearest store and rebuilds just the delta.
- name: Cache /nix/store
uses: nix-community/cache-nix-action@v6
with:
primary-key: nix-${{ runner.os }}-${{ hashFiles('flake.lock', '**/*.nix', '**/*.odin') }}
restore-prefixes-first-match: nix-${{ runner.os }}-
gc-max-store-size-linux: 2G
purge: true
purge-prefixes: nix-${{ runner.os }}-
purge-created: 604800 # drop entries older than 7 days
# First run compiles Thor + the tree-sitter grammar libs; later runs
# restore them from the cache above and skip the compile.
- name: Build Thor
run: nix build .#default --print-build-logs
- name: Generate site
working-directory: site
run: ../result/bin/thor -base-url=https://oneidiotdeveloper.com/thor -minify
- name: Upload Pages artifact
uses: actions/upload-pages-artifact@v3
with:
path: site/public
deploy:
needs: build
runs-on: ubuntu-latest
environment:
name: github-pages
url: ${{ steps.deployment.outputs.page_url }}
steps:
- name: Deploy to GitHub Pages
id: deployment
uses: actions/deploy-pages@v4
.gitignore
-2
View File
@@ -1,5 +1,3 @@
.direnv .direnv
result result
thor thor
*.spall
thor-prof
AGENTS.md
-524
View File
@@ -1,524 +0,0 @@
# Thor — Odin Static Site Generator
Thor is a static site generator written in [Odin](https://odin-lang.org), replacing Hugo for the `sbrow.github.io` blog. It lives at `./thor/` as a git subtree with its own `flake.nix`.
## 🚫 DO NOT EDIT THE DOCS — BY HUMANS, FOR HUMANS
> **THE DOCUMENTATION UNDER `thor/site/` IS HANDWRITTEN, BY HUMANS, FOR HUMANS.**
>
> **NO AI, AGENT, BOT, ASSISTANT, OR OTHER NON-HUMAN MAY EDIT, REWRITE,
> REPHRASE, REFORMAT, "IMPROVE," SUMMARIZE, OR GENERATE ANY FILE UNDER
> `thor/site/` — EVER.**
>
> These are not machine artifacts. A human wrote every word. AI may be
> consulted as a sanity check, but the prose stays human. If you are not a
> human, do not touch these files. See `thor/site/content/ai.md`.
## Architecture
```
thor.json ← site config (title, base_url, params, modules, og)
content/ ← markdown and HTML content files
layouts/ ← Mustache templates + partials (user overrides)
assets/ ← CSS (Tufte-based), JS, fonts, images
thor/defaults/ ← bundled default templates (embedded via #directory)
public/ ← build output (generated)
```
### Package structure
```
thor/
├── treesitter/ # FFI types + grammar management (standalone package)
├── markdown/ # Content transformation pipeline (imports ../treesitter)
├── mustache/ # Template engine with lambdas + pipe filters + diagnostics
├── diagnostics/ # Shared diagnostic utilities (suggest_correction)
├── content.odin # Page struct, Pending_File, scan_content_files, collect_languages, load_page
├── render.odin # Template rendering, Template_Context, sort_pages, RSS, sitemap
├── menus.odin # Menu_Entry, DEFAULT_WEIGHT, build_menus, collect_auto_menus, merge_page_menus, parse_page_menus, parse_config_menus
├── site.odin # Config (Flags, Config_File, Site, Site_Context), init_site
├── minify.odin # HTML/CSS minification (imports treesitter)
├── feed.odin # RSS + sitemap generation
├── vfs.odin # Union file system (defaults → modules → site)
├── assets.odin # VFS-based asset copying
├── html.odin # HTML helpers: strip_html_tags, unescape_html, generate_summary, generate_description
├── opengraph.odin # Open_Graph struct + og_for_site/og_for_page
├── frontmatter.odin # JSON frontmatter parser (supports nested og + lastmod + weight + menus)
├── defaults.odin # DEFAULTS_PATH constant (#directory)
├── main.odin # Entry point
├── bench/ # Template rendering benchmark
└── defaults/layouts/ # Bundled default templates
```
### Source files (main package)
| File | Responsibility |
|---|---|
| `main.odin` | Entry point. Parses CLI flags via `core:flags`, sets logger level from `-verbose`/`-quiet`, calls `init_site`, `build_vfs`, wires `treesitter.grammar_dir`/`query_dir` from config, `site_load_content`, `render_site`. Optional Spall profiling via `SPALL` config flag. |
| `site.odin` | `Flags` (CLI, includes `-verbose`/`-quiet`), `Config_File` (thor.json), `Site_Context` (template-facing: `title`, `description`, `base_url`, `params`, `og`, `menus`), `Site` (runtime state + arena + VFS + pages + `og`). `Feature` enum. `init_site(site, flags)` — takes pre-parsed `Flags`. Config menu parsing in `site_apply_config`. |
| `content.odin` | `Page` struct (includes `weight`, `menus`, `params: json.Value`, `toc: string`, `og`), `Pending_File` struct, `scan_content_files` (section-aware walk that handles leaf bundles), `collect_languages` (pre-scan for code fence languages), `load_page` (falls back to file mtime, generates TOC via `md.generate_toc` when frontmatter `"toc": true`), `infer_layout`. Calls `md.process()` for the markdown pipeline. |
| `render.odin` | Template rendering: `render_site`, `render_page_html`, `render_home_html`, `render_section`. `Template_Context` (unified render struct with `site: Site_Context`, `page: Page`, `menus`, `params`, `posts`, `pages`). 3-frame context stack via `[]any{ctx.site, ctx.page, ctx}`. `merge_params(site, page)` — shallow merge of site + page params. Error deduplication via `seen: ^map[string]bool` passed through render chain. `sort_pages` (weight primary, date secondary). `to_title_case` for section display names. VFS-based template loading with fallback chain (`get_template`). |
| `menus.odin` | Menu system: `Menu_Entry {name, url, weight: Maybe(int)}`, `DEFAULT_WEIGHT = 10`. `build_menus` (priority chain: config → auto + page frontmatter, then `warn_all_duplicate_weights`). `collect_auto_menus` (sections + root-level pages, skips pages with explicit `"menus": "main"` frontmatter). `merge_page_menus` (frontmatter entries with effective weight fallback via nil check). `parse_page_menus` (string/array/object forms). `parse_config_menus` (from thor.json). `sort_menu_entries` / `compare_menu_entries` (weight primary via `.? or_else DEFAULT_WEIGHT`, name secondary). `warn_duplicate_weights` / `warn_all_duplicate_weights` (log when two entries in same menu have same explicitly-set weight). |
| `minify.odin` | HTML/CSS minification via tree-sitter. Imports `ts "treesitter"`. |
| `feed.odin` | RSS feed + sitemap XML. Uses `page.url` for canonical URLs. |
| `vfs.odin` | Union file system: `VFS`, `build_vfs`, `mount_dir`, `mount_subdir`, `mount_recursive`, `vfs_get`, `vfs_get_entry`, `vfs_entry_data`. Layers defaults → modules → site. |
| `assets.odin` | `copy_assets_dir` — iterates VFS entries with `assets/` prefix, minifies CSS, copies verbatim or via `os.copy_file`. |
| `html.odin` | `strip_html_tags`, `unescape_html`, `generate_summary` (word-count truncation, zero-alloc), `generate_description` (HTML→plain text: strip tags, decode entities, collapse whitespace). |
| `opengraph.odin` | `Open_Graph` struct (fields ordered per OGP spec, `is_article: Maybe(bool)`). `og_for_site(site)` for site defaults (from config + derived), `og_for_page(site_og, page)` for page-specific (overlay page.og + derive from page data). Description falls back to `generate_description(generate_summary(body_html))`. |
| `frontmatter.odin` | JSON frontmatter parser (`{ }` delimited). Supports `layout`, `lastmod`, `weight: Maybe(int)`, `menus`, `params: json.Value`, `toc: bool`, and nested `og` object (via `json_get_open_graph`). Helpers: `json_get_string`, `json_get_bool`, `json_get_int` (returns `Maybe(int)`, nil for absent/invalid). |
| `defaults.odin` | `DEFAULTS_PATH` constant, resolved at compile time via `#directory` so bundled templates ship in the binary. |
### Subpackages
| Package | Files | Responsibility |
|---|---|---|
| `treesitter/` | `treesitter.odin` | FFI types (`Parser`, `Node`, `Query`, etc.), `@(link_prefix="ts_")` foreign bindings, grammar management (`Grammar_Store` with persistent allocator, `load_language`/`compile_query` building blocks, `ensure_parser`/`load_grammar` lazy loading, `preload_grammar`/`preload_grammars` for parallel loading with `sync.Mutex` cache protection), statically-linked HTML/CSS grammars |
| `markdown/` | `markdown.odin` | `Extension` enum, `DEFAULT_EXTENSIONS`, `EXTENSION_NAMES`, `process(body, ext, file_path, allocator)` — full pipeline (clones cmark output, frees original), `parse_extension_list` (with "did you mean?" hints via `diagnostics.suggest_correction`), `apply_extension_config` (same), `resolve_extension_conflicts` (Footnotes wins over Sidenotes) |
| | `footnotes.odin` | `strip_definitions` (pre-cmark, shared by `.Sidenotes` + `.Footnotes`), `inject_notes` (post-cmark sidenote rendering), `inject_footnotes` (post-cmark standard footnote rendering — numbered `<sup>` links + `<section class="footnotes"><ol>` at bottom). `.Sidenotes` and `.Footnotes` are mutually exclusive; `resolve_extension_conflicts` in `markdown.odin` picks `.Footnotes` if both are set. |
| | `alerts.odin` | `inject_alerts` — GitHub alert blocks (`> [!NOTE]`) → styled blockquotes with semantic class names (`alert-note` etc.) |
| | `emoji.odin` | `expand_emoji` — `:shortcode:` → unicode emoji |
| | `sectionate.odin` | `wrap_sections` — splits HTML at `<h2>` into `<section>` wrappers |
| | `highlight.odin` | Syntax highlighting via tree-sitter. Imports `../treesitter`. |
| | `heading_ids.odin` | `inject_heading_ids` — adds `id` attributes to `<h1>`-`<h6>` from heading text. Slug-based, deduplicated. |
| | `deflists.odin` | `convert_deflists` — pre-cmark pass. Scans for definition list patterns (`term\n\n: definition`) and converts to `<dl><dt><dd>` HTML blocks. Terms and definitions rendered through cmark individually for inline markdown. Consecutive pairs grouped into single `<dl>`. |
| | `toc.odin` | `generate_toc(html, allocator)` — page-level feature (not a pipeline extension). Scans `<h1>`-`<h6>` for IDs (after `inject_heading_ids`), builds nested `<ul>` with `<a href="#id">` links. Called from `load_page` when frontmatter `"toc": true`. Depends on `.HeadingIDs` being enabled. |
| `mustache/` | See [Mustache engine](#mustache-engine) below | Template engine |
| `diagnostics/` | `diagnostics.odin` | Shared diagnostic utilities: `suggest_correction` (Levenshtein "did you mean?"), `format_error` (rust-style multi-line error formatter with `caret_start`/`caret_end` params), `line_col`, `line_text`, `count_lines`, `digit_count`, `display_width`, `should_colorize`, `write_gutter`. Used by `mustache` (tag-aware errors via `format_tag_error` wrapper), `markdown` (extension name suggestions), and `render.odin` (parse/render error display). |
| `bench/` | `bench.odin` + `templates/` | Standalone template rendering benchmark. Generates 500 posts + 100 comments, renders with indented partials + inheritance + pipes. `--dump <path>` for output validation, positional arg for iteration count (default 250). |
Icon SVGs live as HTML partials in `layouts/partials/icons/` (home, github, rss, chevron_up, star).
### Data flow
```
thor.json → find_config → init_site (5-step)
→ build_vfs (defaults/layouts → modules → site/layouts, site/assets)
→ site_load_content (scan_content_files + collect_languages + preload_grammars + load_page + url computation + build_menus + warn_all_duplicate_weights)
→ render_site
→ load_partials + get_template (VFS + fallback chain)
→ render_page_html / render_home_html / render_section (3-frame context stack: site, page, ctx)
→ optional minify_html
→ public/
```
## Page struct
```odin
Page :: struct {
section: string, // "" for root, "posts", etc.
slug: string,
layout: string, // inferred or frontmatter override
permalink: string, // relative URL path
url: string, // full canonical URL (base_url + permalink)
title: string,
description: string,
date: string,
year: string,
weight: Maybe(int), // page ordering (nil = unset, defaults to DEFAULT_WEIGHT at comparison time)
lastmod: string,
menus: map[string]Menu_Entry, // frontmatter menu assignments
params: json.Value, // per-page params (merged with site params at render time)
content: string, // rendered HTML body
og: Open_Graph,
draft: bool,
toc: string, // generated table of contents HTML (empty if not requested)
_is_index: bool `private`,
}
```
```
No `Page_Type` enum — page type is inferred from section + `_is_index`. Layout is inferred via `infer_layout(section, is_index)`:
- Home (root index): `"home"`
- Section index: `"<section>_index"` (e.g. `"posts_index"`)
- Section page: singularized section (e.g. `"post"`)
- Root page: `"page"`
**Template fallback chain** (in `get_template`): for content pages, `post → page → base`; for section indexes, `posts_index → section_index → page → base`. Fallbacks logged at debug level. Frontmatter `layout` field overrides the inferred value.
## Menus
Menu system in `menus.odin`. `Menu_Entry :: struct {name: string, url: string, weight: int}`. `DEFAULT_WEIGHT = 10`.
### Sources (priority chain, no mixing)
1. **Config menus** (`"menus"` key in `thor.json`) — exclusive. `"menus": {}` = explicit opt-out (no menus). Config entries sorted by weight.
2. **Auto-menus + page frontmatter** — always run together when no config menus:
- Auto: one entry per section directory + one per root-level non-index page. Alphabetical.
- Page frontmatter: `"menus": "main"` (string), `["main", "footer"]` (array), or `{"main": {"weight": 30}}` (object with per-menu weight). Merged with auto entries, sorted by weight.
### Weight
All weight fields use `Maybe(int)` — nil means "unset," `some(v)` means explicitly set. This distinguishes `"weight": 10` (explicit) from no weight key (defaults to `DEFAULT_WEIGHT` at comparison time via `.? or_else DEFAULT_WEIGHT`). Eliminates the old `0`-as-sentinel pattern from `json_get_int`.
- `Page.weight: Maybe(int)` — page-level ordering. nil = unset. Affects `sort_pages` (weight primary, date secondary).
- `Menu_Entry.weight: Maybe(int)` — per-menu ordering. nil for auto-generated entries and string/array frontmatter forms. Explicit value from object frontmatter form `{"weight": N}`.
- Effective weight in `merge_page_menus`: per-menu weight if set, else falls back to `page.weight`. Both `Maybe(int)`, so nil propagates naturally — no value-based sentinel check.
- Sorted ascending via `.? or_else DEFAULT_WEIGHT`, name alphabetical for ties.
### Templates
```html
{{#menus.main}}
<li><a href="{{url}}">{{name}}</a></li>
{{/menus.main}}
```
`Template_Context.menus` resolves above `Page.menus` (frontmatter assignments) on the 3-frame context stack. Accessible as `{{#menus.main}}` or `{{#site.menus.main}}`.
### Duplicate weight warnings
`warn_duplicate_weights` (called from `build_menus` after all menus are sorted) logs a warning when two entries in the same menu have the same explicitly-set weight. Only non-nil weights are checked — nil (unset/default) entries are never flagged, so auto-generated entries don't produce noise. The warning includes the menu name, weight value, and both entry names.
## Config system
Config is split into three structs with a clear 5-step initialization flow:
- **`Flags`** — CLI args only. Parsed once in `main.odin` via `core:flags`, passed to `init_site`. Includes path overrides (`--content`, `--assets`, `--output`, `--layouts`), build-mode toggles (`-drafts`, `-watch`, `-minify`), log level (`-verbose` → Debug, `-quiet` → Warning), and `-ext`/`-no-ext` for markdown extension overrides.
- **`Config_File`** — parsed from `thor.json` via `json.unmarshal_string`. Holds title, paths, `markdown_extensions` (JSON), `params` (JSON), `modules` (JSON array of relative paths), `og` (`Open_Graph` struct for site-level OG defaults).
- **`Site`** — runtime state: arena, pages, modules, VFS, `features: bit_set[Feature]`, `markdown_extensions: bit_set[md.Extension]`, `og: Open_Graph` (resolved site-level OG).
**`Feature` enum** — `Drafts`, `Minify`, `Watch`. Checked with `.Minify in site.features`.
**`markdown.Extension` enum** (in the `markdown` package, not main) — `Emoji`, `Sidenotes`, `Alerts`, `Highlight`, `Sections`, `HeadingIDs`, `DefLists`, `Footnotes`. Default is `md.DEFAULT_EXTENSIONS` (currently `.Emoji, .Sidenotes, .Alerts, .HeadingIDs, .DefLists`). Configurable via:
- `thor.json`: `"markdown_extensions": { "emoji": true, "highlight": false, ... }`
- CLI: `-ext:highlight,sections` (enable) / `-no-ext:emoji` (disable). Comma-separated, case-insensitive.
**`find_config`** — walks up from CWD looking for `thor.json`. Falls back to `./thor.json`.
Config precedence: `CLI flags > thor.json values > hardcoded defaults`.
```json
{
"title": "...",
"base_url": "...",
"modules": ["../path/to/module"],
"og": {
"image": "https://example.com/og.png"
},
"date": {
"format": "2 Jan 2006",
"timezone": "America/New_York"
},
"grammars": "~/.config/helix/runtime/grammars/",
"queries": "/path/to/tree-sitter/queries",
"markdown_extensions": { "emoji": true, "highlight": false },
"menus": {
"main": [
{"name": "Home", "url": "/", "weight": 1},
{"name": "About", "url": "/about/"}
]
},
"params": {
"social": [
{ "name": "github", "url": "...", "icon": "icons/github" }
]
}
}
```
## VFS (Union File System)
Layered directory resolution for templates and assets: `site layouts/ → module layouts/ → defaults/layouts/`.
```odin
VFS :: struct { files: map[string]VFS_Entry }
VFS_Entry :: struct { fs_path: string, data: []byte }
```
`build_vfs` mounts in reverse precedence (defaults first, site last overwrites). `DEFAULTS_PATH` resolved at compile time via `#directory`, so bundled templates ship inside the binary. Modules configured via `"modules": ["../path"]` in `thor.json` — each module contributes `layouts/` and `assets/` subdirectories.
Three access patterns:
- `vfs_get(vfs, path) -> ([]byte, bool)` — data only (lazy-loaded from disk)
- `vfs_get_entry(vfs, path) -> (VFS_Entry, []byte, bool)` — entry + data (for callers that need `fs_path` for diagnostics)
- `vfs_entry_data(entry) -> ([]byte, bool)` — data from an entry already in hand (avoids redundant map lookup when iterating `vfs.files`)
Content is **not yet in the VFS** — `scan_content_files` still uses direct filesystem reads. (See `TODOS.md`.)
## Open Graph
`Open_Graph` struct in `opengraph.odin` with fields ordered per [ogp.me](https://ogp.me/) spec. `is_article` is `Maybe(bool)` — nil means "unset" (distinguished from explicitly `false`).
**Site-level** (`og_for_site`): starts from `Config_File.og` (user-supplied defaults from `thor.json`), then fills empty fields derivable from `Site`:
- `site_name ← site.title`
- `locale ← "en_US"` (default if unset)
**Page-level** (`og_for_page`): copies site OG, derives page-specific fields, then overlays `Page.og` (from frontmatter):
- `url ← page.url`
- `title ← page.title` (falls back to `site_name` if empty)
- `type ← "article" if !page._is_index else "website"`
- `is_article ← !page._is_index`
- `section ← page.section`
- `published_time / modified_time ← page.date / page.lastmod`
- `description ← page.description`, else `generate_description(generate_summary(body_html))` (scrubbed plain text)
Paths through maps (e.g. `params.*`) are silently allowed — not validated. Templates access via `{{og.url}}`, `{{og.title}}`, `{{#og.is_article}}`, etc.
## Markdown pipeline
Lives in the `markdown` package. Entry point: `md.process(body, ext, file_path)`. All `.html` content files skip the pipeline entirely — body is used as-is.
```
raw markdown
→ md.strip_definitions (if .Sidenotes || .Footnotes — pre-cmark)
→ md.convert_deflists (if .DefLists — pre-cmark)
→ cmark markdown_to_html (Unsafe mode for HTML passthrough)
→ md.expand_emoji (if .Emoji — post-cmark)
→ md.inject_notes (if .Sidenotes — post-cmark, sidenote rendering)
→ md.inject_footnotes (if .Footnotes — post-cmark, standard footnote rendering)
→ md.inject_alerts (if .Alerts — post-cmark)
→ md.highlight_code (if .Highlight — post-cmark)
→ md.inject_heading_ids (if .HeadingIDs — post-cmark, pre-sections)
→ md.wrap_sections (if .Sections — post-cmark)
```
Each step is gated by `bit_set[md.Extension]`.
## Template system
Templates use Mustache with template inheritance (`{{<base}}` / `{{$block}}`):
```html
<!-- base.html -->
<body>{{> nav}}{{$main}}{{/main}}{{> footer}}</body>
<!-- page.html (content layout) -->
{{<base}}
{{$main}}
<main><article><h1>{{page.title}}</h1>{{&content}}</article></main>
{{/main}}
{{/base}}
```
Data is passed as a single `Template_Context` struct. `render_template` passes a 3-frame context stack `[]any{ctx.site, ctx.page, ctx}` to `mustache.render`, which auto-detects `[]any` and expands each element into a stack frame. Name resolution walks top-to-bottom: `Template_Context` → `Page` → `Site_Context`. Fields not found on the top frame fall through to lower frames.
```odin
Template_Context :: struct {
site: Site_Context, // site-level data (title, description, base_url, params, og)
menus: map[string][]Menu_Entry, // generated menu data (copied from site, resolves above Page.menus)
now: string, // UTC ISO 8601 build timestamp
date_format: string, // from site.date.format (thor.json)
timezone: ^datetime.TZ_Region, // for format pipe
og: Open_Graph, // computed per-page OG
params: json.Value, // merged site + page params (resolves above Page.params)
page: Page, // current page
pages: [dynamic]Page, // home page list
posts: [dynamic]Page, // section post list
}
```
`Site_Context` is embedded in `Site` via `using site_context`. Fields like `site.title`, `site.menus`, `site.params` are accessed directly on `Site` through promotion. `Template_Context.menus` is copied from `site.menus` to resolve above `Page.menus` (frontmatter assignments) on the context stack. `Template_Context.params` is set per-page via `merge_params(site.params, page.params)` — site params overlaid with page params. Browser title is handled by the `{{> title}}` partial (not a computed field).
`render_site` pre-parses all partials and the base layout once (via `mustache.parse`), then per-layout templates are cached in `get_template`. Year-based grouping on section index pages is done in the template via `{{#posts | group_by year}}` (see Pipes extension below).
### Pipes extension
Section tags and interpolation tags may transform the resolved value before rendering:
```handlebars
{{#posts | group_by year}}
{{key}}: {{#items}}{{title}}, {{/items}}
{{/posts}}
<!-- Interpolation pipe: format a date for display -->
<time datetime="{{date}}">{{date | format}}</time>
```
Currently implemented: `group_by <field>` (list → list-of-groups) and `format` (ISO date string → display string like "15 Mar 2026"). The `format` pipe resolves `date_format` (string) and `timezone` (`^datetime.TZ_Region`) from the data context. When `timezone` is non-nil, dates are DST-aware converted before formatting. The `MST` token reflects the active timezone abbreviation (e.g. `"EST"`/`"EDT"`) or the source offset (e.g. `"UTC-04:00"`) when no target tz is configured. TZ data is loaded once by `init_site` via `timezone.region_load` using the site arena allocator, stored on `Site.tz`, and freed when the arena is destroyed. Filter results live in `context.temp_allocator` (render-scoped). See `mustache/EXTENSIONS.md` for syntax details, caps (`MAX_PIPES`, `MAX_PIPE_ARGS`), and the `Group` struct shape.
### Comments
`page.html` includes `{{> comments}}`. The `comments.html` partial self-guards with `{{#og.is_article}}` so it only renders on article pages — no separate `is_post` flag.
## Syntax highlighting
Build-time highlighting via Tree-sitter C FFI. No client-side JavaScript.
- **HTML and CSS grammars** statically linked via Nix (`mkGrammarStaticLib` in `thor/flake.nix`). Always available, no `dlopen`.
- **Other grammars** (bash, odin, nu, etc.) loaded via `dlopen` from `.so` files. Pre-scanned from content code fences and loaded in parallel via `preload_grammars` (one thread per language, `sync.Mutex` on `Grammar_Store.cache`). `Grammar_Store.allocator` is the OS heap (set by `init_persistent` before arena override) so grammars persist across watch-mode rebuilds.
- Grammar and query paths configured via `thor.json` (`grammars`, `queries`). Flow: `thor.json` → `Config_File` → `Site` → `main.odin` sets `treesitter.grammar_dir`/`treesitter.query_dir`. Tilde (`~/`) expanded by `expand_path` in `site.odin`. Paths logged at startup.
- Grammar loading split: `ensure_parser` (parser only, used by minify) vs `load_grammar` (parser + query, used by highlight).
- Capture names mapped to CSS classes: `keyword` → `.hl-keyword`, etc.
- Atom-one-dark color theme in `main.css`.
## Minification
Optional, enabled with `-minify` flag (`.Minify` in `Feature` bit_set).
- **HTML** — tree-sitter parses output, strips comments, removes inter-tag whitespace, preserves `<pre>`/`<code>`/`<textarea>`/`<script>`/`<style>` content. Applied after template rendering.
- **CSS** — tree-sitter parses `.css` files in `assets/`, strips comments, collapses whitespace, trims around `{};:,`. Applied during `copy_assets_dir`.
- Non-CSS files in `assets/` copied verbatim.
## Memory management
- `Site` owns a `mem.Dynamic_Arena`
- `init_site` calls `mem.dynamic_arena_init(&site.arena)` (Odin's default alignment suffices)
- Config loading (flags + JSON) uses the arena allocator explicitly
- `site_allocator(site)` returns the arena allocator for callers
- `destroy_site` frees the arena
- `main.odin` sets `context.logger = log.create_console_logger()` — without this, all `log.*` calls are silently dropped
- `context.allocator` is set to `site_allocator(&site)` in the main loop
- `context.temp_allocator` freed per watch-loop iteration via `defer free_all`
## Spall profiling
Optional, compiled out by default. Enabled with `-define:SPALL=true`:
```bash
odin build . -define:SPALL=true -o:speed -out:thor-prof
./thor-prof -drafts # generates thor.spall
```
Uses `core:prof/spall` with `@(instrumentation_enter)`/`@(instrumentation_exit)` hooks — every function auto-instrumented, no manual annotation needed.
## Building
### Local development
```bash
nix develop
# From blog root:
odin run ./thor -- -drafts
# Assets (CSS/JS/fonts) are copied/minified automatically by thor
caddy run # serves public/ on blog.localhost
```
No CSS build step — `main.css` is static Tufte-based CSS, no preprocessor or compiler needed.
### Production build
```bash
nix build # runs thor, outputs to ./result/
```
### Tests
```bash
cd thor
odin test . # main package tests (site, frontmatter)
odin test . -all-packages # includes mustache specs, lambdas, pipes, diagnostics, markdown tests
```
### Benchmark
```bash
cd thor
odin build bench -o:speed
./bench.bin # 250 iterations, prints timing
./bench.bin --dump output.html # render once, write to file for diff validation
./bench.bin 1000 # custom iteration count
```
## Mustache engine
Spec-compliant implementation at `mustache/`. See `mustache/SPEC.md` for the implementation specification, `mustache/EXTENSIONS.md` for non-standard extensions (pipes), and `mustache/diagnostic.odin` for the rust-style error formatter.
### Files
| File | Responsibility |
|---|---|
| `mustache.odin` | Public API (`parse`, `render`, `Template`), parser (`parse_section`), renderer (`render_nodes` with `Indent_State` for partial indentation), template inheritance (`merge_block_overrides`), `delete_template`/`delete_partials`. Pipe support in Variable/Unescaped/Section/Inverted tags. |
| `tokenizer.odin` | Tokenizer (template string → `[]Token`), standalone whitespace detection |
| `data.odin` | Reflection-based data model: `base_value` (peels union/any/nested-any layers), `lookup_in` (structs + maps, handles `Type_Info_Any` value kind in maps), `resolve_name`, `is_truthy`, `any_to_string`, `write_value`, `list_info`, `extract_list_element`, `collect_map_keys` |
| `pipes.odin` | Pipes extension: `Pipe_Op` enum (`.Format`, `.Group_By`), `pipe_op_from_string`/`pipe_op_candidates` (reflection-based enum name lookup), `Pipe_Filter` AST (with `op_pos` for diagnostics), `parse_pipeline` (tracks byte offsets via `strings.index`), `apply_pipeline`, `apply_filter` (exhaustive enum switch), `apply_group_by`, `apply_format`. Misspelled pipe op suggestions via `suggest_correction`. Stored on `Node.filters`; render-scoped results in temp allocator. |
| `diagnostic.odin` | Mustache-specific diagnostics: `context_extent` (scans `{{ }}` tags for caret underline), `format_tag_error` (wraps `diags.format_error` + `context_extent`), `format_render_error` (formats `Error` using `Template` source/path). Generic error formatting lives in `diagnostics/` package. |
| `suggest.odin` | Strict-warning helpers: `validate_key_path` (walks dotted path, crosses maps silently), `suggest_correction` (Levenshtein via `core:strings/levenshtein_distance`), `collect_struct_keys` (via reflection, recurses into `using`), `struct_has_field` (distinguishes missing field from nil value — needed for `Maybe(bool)`), `collect_partial_names`, `collect_block_names`. |
| `spec_test.odin` | JSON spec test runner — loads `spec/specs/*.json`, runs each test case. Uses `log.nil_logger()` to suppress expected warnings. |
| `pipes_test.odin` | Pipe filter tests (`group_by` + `format`) |
| `diagnostic_test.odin` | Golden-output tests for `format_error` (multi-line context, edge cases, alignment, caret position, hint) + parser error message brace-escaping |
| `suggest_test.odin` | Tests for `validate_key_path`, `suggest_correction`, `struct_has_field` with `Maybe(bool)` and `using`-promoted fields |
### Architecture
```
parse(source, path) → tokenize → trim_standalone_whitespace → parse_section → Template
render(tmpl, data, partials) → render_nodes (walks flat node array against context stack) → string
```
- **Two-phase API**: `parse()` produces a reusable `Template`, `render()` walks it against data. Templates parsed once, rendered many times.
- **Flat `[dynamic]Node` array** with `children: []Node` slices (pre-order layout; slices point into the backing array). Each `Node` carries `pos: int` (byte offset into source) for diagnostics.
- **`Template`** carries `source` and `path` — used by diagnostics to show file location and source context.
- **Context stack**: `^[dynamic]any` with `append`/`pop` for section push/pop.
- **`render_nodes` takes `Template` by value** (not `^Template`) — Odin's calling convention promotes to pointer when efficient. Eliminates "local copy" patterns at call sites.
- **`Block_Override.source: Template`** — carries the template that defined the override, so warnings inside block overrides point at the correct file.
- **`base_value`** peels Named/Distinct/Union layers (including `json.Value`). Also unwraps nested `any`-of-`any` (which occurs when `map[string]any` values are read via runtime map internals).
- **`lookup_in`** resolves keys on structs (via `reflect.struct_field_value_by_name` with `allow_using = true`) and maps. Detects `Type_Info_Any` value kind in maps and reads the inner any directly to avoid double-wrap.
- **Template inheritance**: `{{<parent}}` loads parent from partials, `{{$block}}` defines overridable sections. `merge_block_overrides` propagates overrides through multi-level chains.
- **Dynamic partial names**: `{{>*key}}` resolves partial name from data context at render time.
- **Render-time partial indentation**: `Indent_State` threads `at_line_start` through `render_nodes` so partial indent is applied at render time (via `write_indented` on Text nodes) instead of reparsing the partial's source. `render_template` writes initial indent, creates state, calls `render_nodes`. Data-injected newlines don't pick up indent (Variables don't update `at_line_start`).
### Diagnostics
Rust-style error messages with multi-line source context, caret underlines, and Levenshtein suggestions. ANSI colors via `core:terminal/ansi`, gated on `should_colorize()` (TTY detection on stderr).
**Error types**: `Error_Body{msg, pos, kind, source, path, span, hint}` where `kind` is `Error_Kind.Syntax` (parse-time) or `Error_Kind.Data` (render-time). `source`/`path` carry the template the error originated in (set by `tag_error` — enables correct file/line for errors inside partials). `span` controls caret underline width (used by pipe op diagnostics). `hint` carries "did you mean?" suggestions. `Error` is a single-variant union wrapping `Error_Body` (nilable for `!= nil` / `or_return`).
**Error deduplication**: `render_template` takes `seen: ^map[string]bool`. Duplicate errors (same formatted diagnostic) are suppressed across page renders within a single build.
**Partial source tracking**: `tag_error(err, current)` stamps render-time errors with `current.source`/`current.path` at the 4 `apply_pipeline` call sites in `render_nodes`. Ensures errors inside partials point at the partial file, not the top-level template.
**Strict-by-default warnings** — `render_nodes` emits `log.warnf` diagnostics for:
- Unknown keys in `{{k}}`, `{{{k}}}`, `{{#k}}`, `{{^k}}` (via `validate_key_path` + `suggest_correction`)
- Missing partials (`{{> name}}` not in partials map)
- Missing parent templates (`{{<name}}` not in partials map)
- Unmatched block overrides (`{{$name}}` doesn't match any block in parent template)
**Exceptions** (no warning):
- `{{.}}` and dot-prefixed names (current context)
- Paths that cross a map (e.g., `params.*`) — validated for typos via Levenshtein: close matches warn with a suggestion, genuinely absent keys are suppressed silently
- `Maybe(bool)` fields with nil value (field exists, value is nil — distinguished via `struct_has_field`)
- Found fields whose value is nil/empty (e.g., nil `json.Value` union) — the field exists, looking up sub-keys is valid "not found" behavior
**Block override source tracking**: `Block_Override.source: Template` ensures warnings inside block overrides point at the override's source file (e.g., `page.html`), not the parent template (`base.html`).
### Pipes
`{{key | op args…}}` for interpolation, `{{#key | op args…}}…{{/key}}` for sections. Stored as `[dynamic; MAX_PIPES]Pipe_Filter` on each `Node`. Applied in the renderer via `apply_pipeline` before truthiness/interpolation. Implemented filters:
- `group_by <field>` — list → `[dynamic]Group` where `Group{key, items}`
- `format` — ISO 8601 date string → display string (e.g., "15 Mar 2026")
See `mustache/EXTENSIONS.md`.
### Not implemented
- Set delimiters (`{{= =}}`, `delimiters.json`)
- Partial invocation stack in diagnostics (warnings inside partials point at the partial file but don't show the `{{> name}}` invocation site — see TODOS.md)
## Known limitations
- CSS/JS cache busting uses manual `?v=N` query params instead of content hashing.
- Tree-sitter grammar/query paths must be configured manually via `thor.json` (`grammars`, `queries`) — no auto-discovery. HTML/CSS are statically linked.
- `map[string]any` only works through `lookup_in`'s special-case handling; thor otherwise uses structs.
- `format_f64` in mustache brute-forces shortest float representation.
- Content directory not mounted in VFS (modules can ship templates/assets but not content packs yet).
- Per-page params rendering is incomplete — `merge_params` produces correct data, but `base_value` may return nil for `json.Value` fields accessed through the `[]any` context stack via reflection in some cases. Warning suppression masks this; actual rendering may not work for all param values.
- Lambda support removed. Mustache lambdas (`proc() -> string` in data context) are not supported. Pipes cover data transformation.
## Design decisions
You may never, *ever* remove `TODO:` or `FIXME:` comments. Those are for humans, not machines.
See `HUGO.md` for analysis of why thor doesn't need Hugo's shortcode context isolation.
See `mustache/SPEC.md` for the original implementation specification.
See `mustache/EXTENSIONS.md` for non-standard extensions (pipes).
See `DIAGNOSTICS.md` for the two-tier diagnostic system design.
See `mustache/TOKENIZERS.md` for tokenizer architecture comparison (Go, Liquid, Thor).
## Odin language facts
These are things that are easy to get wrong:
- **Proc arguments are immutable.** You cannot assign to a parameter directly. To get a mutable copy, shadow it: `x := x`. If you need to modify the source, pass a pointer `^x`.
- **`for` each loops use `item, idx` order**, not `idx, item`. Correct: `for item, idx in arr`. Wrong: `for idx, item in arr`.
- **`make([dynamic]T, n, allocator)` sets capacity, not length.** To get length=0 with capacity=n, use `make([dynamic]T, 0, n, allocator)`. Using `make([dynamic]T, n, allocator)` creates `len=n` with `n` zero-initialized elements.
- `#partial switch` is usually a code smell. prefer a `case all, extra, types:` branch.
- you don't usually need to create arena allocators in tests, instead use context.temp_allocator if you want to simplify cleanup.
- you don't need to manually set up a tracking allocator in tests. the context.allocator will warn you about leaks.
- **`Maybe(T)` unwrap syntax:** `value.? or_else default`. Not `value or_else default` — `or_else` works on the `?T` returned by `.?`, not on `Maybe(T)` directly.
- **`Maybe(T)` equality:** `a == b` works directly between two `Maybe(T)` values (nil == nil → true, some(5) == some(5) → true, nil == some(5) → false). Also `a == 5` works (int coerces to `Maybe(int)`).
- **File logger in tests:** `log.create_file_logger(&f)` + `context.logger = logger` captures log output. Must be set inline in the test proc (not via a helper proc) for context propagation. Clean up with `log.destroy_file_logger(logger)` then `os.read_entire_file_from_path` to verify output.
- **`fmt.sbprintf` writes directly to a `strings.Builder`.** Prefer `fmt.sbprintf(&sb, fmt, args...)` over `fmt.aprintf(fmt, args...)` + `defer delete` + `strings.write_string`. The `aprintf` pattern allocates an intermediate string, requires manual cleanup, and queues a `defer delete` per loop iteration. `sbprintf` avoids all of this.
## TODO
See `TODOS.md` for the full list.
ASSET_PIPELINE_PLAN.md
-135
View File
@@ -1,135 +0,0 @@
# Asset Pipeline Plan
Design for letting thor users bring their own CSS toolchain (Sass, PostCSS,
Lightning CSS, Tailwind, …).
## Thesis
Let thor users bring their own CSS toolchain by **orchestrating the tool they
already installed** — never bundling or reimplementing one. This is expressed
through thor's existing mustache pipe grammar with two new ops: `tool` and
`fingerprint`.
The earlier research design (content-addressed registry + deferred token
resolution + a separate post-render path for "whole-output scanners") is
**discarded**. It was overbuilt. Every tool — Tailwind included — scans
*source*, which exists at render time, so every transform runs **eagerly,
synchronously, and memoized**. No placeholder tokens, no post-render
string-replace pass, no second integration surface.
## The two ops
```html
<link href="{{ "css/main.scss" | tool sass | fingerprint | rel_url }}">
```
```json5
// thor.json
{
"tools": {
"sass": "sass $in $out",
"postcss": "postcss $in -o $out",
"lightningcss": "lightningcss --minify $in -o $out",
"tailwind": "tailwindcss -i $in -o $out"
}
}
```
## Core model
The value flowing through the pipe is **bytes held by thor**, with a logical
name attached (`css/main.css`). Each stage transforms the bytes; the **final
stage publishes** one file to the output dir and returns its URL.
- A literal (`"css/main.scss"`) or context key names the **source** file on disk.
- `| tool <name>` runs a subprocess, replacing the carried bytes with its output.
- `| fingerprint` writes the content-addressed file and returns its hashed URL.
- The chain composes with the existing `rel_url` for base_url subpaths.
**Memoization:** results are cached per build, keyed by (op-chain, source). The
first page to reference an asset runs the tool; every other page hits the cache.
This is what stops N pages from shelling out N times.
## `| tool <name>`
Looks up `<name>` in the `thor.json` `tools` table and runs its command
template. `$in` and `$out` are **optional per-tool I/O adapters**:
| Template has | Behavior |
|---|---|
| `$in` | thor writes current bytes to a scratch path, substitutes it |
| no `$in` | thor pipes current bytes to the tool's **stdin** |
| `$out` | thor substitutes a scratch path, reads it back after |
| no `$out` | thor captures the tool's **stdout** |
Defaulting to stdin/stdout avoids scratch-file overhead when the tool supports
it. Tools that can't stream (Lightning CSS wants a path; Tailwind v3 needs an
input file) just get `$in`/`$out` in their template. Either way the bytes are
thor's, so chaining works regardless of each tool's preference:
`| tool sass | tool postcss` composes even if sass streams and postcss wants
files.
- **`$out` is always thor-controlled scratch** — the tool never picks the
published filename.
- **Extension derivation:** thor owns the output name; `main.scss` through
`tool sass` becomes `main.css`.
- **Errors:** thor gates on exit code and surfaces the tool's stderr through one
code path. If the tool is missing from PATH, fail with its stderr — never
bundle, never assume a runtime.
- **A bare `tool` (no `fingerprint`) still publishes** — to the logical unhashed
path (`output/css/main.css`), returning `/css/main.css`.
## `| fingerprint`
Content-addresses whatever bytes it currently holds. It has **two outputs**:
- **Side effect:** writes `css/main.<hash>.css` to the output dir.
- **Return value:** the hashed URL string `/css/main.<hash>.css` (feeds
`rel_url`).
Properties:
- **Works standalone**, no `tool` needed: `{{ "css/main.css" | fingerprint }}`
is plain cache-busting of a hand-written file — probably the most common use.
- **Position decides what's hashed** — it hashes current bytes, so it belongs
**last** in the chain. No terminal-only special-casing; the rule is just "put
it last."
- **Hash format:** `name.<hash>.ext` (CDN-friendly), 8 hex chars of SHA-256.
## Tailwind is not special
It scans your **source** (`layouts/`, `content/`, partials) — which exists at
render time — so it runs synchronously like sass. Its `thor.json` entry just
needs content globs pointing at the source tree (or v4 auto-detection). The
**one** asymmetry is watch invalidation: sass re-runs when `main.scss` changes;
Tailwind must re-run when *any* scanned source changes. That's a cache-key
breadth difference, not an architectural one. The cost of scanning source vs.
output is losing dynamically-assembled class tokens (`col-{{n}}`) — a Tailwind
anti-pattern whose sanctioned fix is a safelist, not warping the build.
## Interaction: `copy_assets_dir`
Today thor copies everything under `assets/` to the output dir. Any source
consumed by a pipe must be **excluded from that plain copy** — otherwise you'd
ship the raw `main.scss` (or a dead unhashed `main.css` alongside the hashed
one). The pipe owns the write for assets it touches.
## Implementation steps (separable)
1. **`| fingerprint`** — no subprocess, no config. Hash + write + return URL,
plus the `copy_assets_dir` exclusion. Smallest blast radius, independently
useful, ships first.
2. **`| tool <name>`** — `thor.json` tools table, the `$in`/`$out`/stdin/stdout
adapter, subprocess + exit-code/stderr handling, extension derivation,
memoization. Composes with step 1.
## Open questions for implementation
- **Watch invalidation:** simplest correct v1 recomputes the memo each build
(cheap for the common single-asset case). A cross-build skip keyed on source
content/mtime is a later optimization — and Tailwind's broad content-glob key
is where it matters most.
- **Source-map sidecars:** sass/lightning emit `.map` files next to `$out`;
disable in the template or ignore the sidecar. Minor.
- **Error policy default:** hard-fail the build vs. warn-and-skip on tool
failure/absence. Lean fail; possibly configurable later.
AUDITING_BINARY_SIZE.md
-96
View File
@@ -1,96 +0,0 @@
# Visualizing & Managing `thor` Binary Size
How to inspect what makes up the compiled `thor` binary, and how to keep that
inspection useful once we bundle layout/asset files into the binary.
## Quick inspection (tools already available)
The binary is a not-stripped ELF, so it ships symbols we can break down.
```bash
# Section totals (text / data / bss)
size ./thor
# Top symbols by size
nm --print-size --size-sort --radix=d ./thor \
| awk '{sz=$2+0; sub(/^[0-9a-fA-F]+ +[0-9]+ +[A-Za-z] +/,""); printf "%8d %s\n", sz, $0}' \
| sort -rn | head -25
# Aggregate symbol sizes by Odin module namespace (foo::bar -> "foo")
nm --print-size --size-sort --radix=d ./thor \
| awk '{sz=$2+0; rest=$0; sub(/^[0-9a-fA-F]+ +[0-9]+ +[A-Za-z] +/,"",rest);
ns=rest; if (rest ~ /::/){sub(/::.*/,"",ns)}
else if (rest ~ /^ts_/){ns="treesitter (ts_*)"} else {ns="<other/runtime>"};
s[ns]+=sz} END{for (k in s) printf "%9d %s\n", s[k], k}' \
| sort -rn | head -25
```
As of this writing the top contributors were: tree-sitter tables (~118 KB),
`main` (~96 KB), runtime/other (~93 KB), `mustache` (~67 KB), then `fmt` /
`encoding_json` from the stdlib.
## Better visualization: `bloaty`
`bloaty` (Google's binary-size profiler) is the right tool. Not installed, but
runnable via Nix:
```bash
bloaty ./thor # by section
bloaty ./thor -d symbols -n 30 # by symbol, hierarchical
bloaty ./thor -d sections,symbols # section then symbols within each
# via nix (no install):
nix run nixpkgs#bloaty -- ./thor -d symbols -n 30
```
For a stripped-size estimate:
```bash
readelf -S ./thor | grep -E 'debug|symtab|strtab' # strippable overhead
strip -s ./thor -o thor.stripped && size thor.stripped
```
Odin also supports smaller builds: `odin build . -o:size -no-bounds-check`.
## IMPORTANT: keeping bloaty useful once we bundle layouts into the binary
Today the VFS (`vfs.odin`) reads layout/asset files lazily from disk at runtime
(`VFS_Entry.fs_path` + `os.read_entire_file_from_path`); `VFS_Entry.data` is
never populated at build time. So embedded content is **not** in `./thor` yet.
When we switch to embedding via Odin `#load(...)`, the file bytes land in the
binary's `.rodata` section. Whether bloaty can show *per-file* sizes depends on
how we emit the embed:
- **Section total — always visible.** `.rodata` grows by ~the total bundled
size. `bloaty ./thor -d sections` (or `size`) shows the aggregate, and we can
diff before/after. This always works.
- **Per-file breakdown — only if each file is a NAMED symbol.**
- A generated `map[string]VFS_Entry` literal full of inline `#load(...)`
expressions makes each blob an **anonymous** `.rodata` constant. Bloaty
lumps them together — no per-file breakdown.
- Assigning each file to its own **package-level variable** makes each a named
global symbol, so `bloaty ./thor -d symbols` (and the `nm` recipe above)
show each file individually:
```odin
layout_base_html := #load("defaults/layouts/base.html")
layout_single_html := #load("defaults/layouts/single.html")
// ...then reference these in the VFS map instead of inlining #load.
```
### Decision to bake into the bundling codegen
When we build the codegen that replaces the runtime `mount_dir` disk walk, have
it emit **one named package-level variable per embedded file** (not an anonymous
map literal). That single choice is what makes bloaty / `nm` per-file analysis
possible — cheap to do up front, painful to retrofit.
Additional options that help regardless of symbolization:
1. Build with Odin `-debug` (DWARF) and use `bloaty ./thor -d compileunits` to
attribute embedded bytes to the generated source file.
2. Since the generator already knows every file and its size, emit a size
manifest at bundle time (`path -> bytes`, sorted / treemap). This is the most
reliable per-file view and can't drift from reality; bloaty then just
confirms the `.rodata` total matches.
CODE_REVIEW.md
-38
View File
@@ -1,38 +0,0 @@
## Code Review
A human should manually review every file in the project. AI cannot complete
these tasks.
- [ ] Review every file in thor
- [ ] Review assets.odin
- [ ] Review content.odin
- [ ] Review defaults.odin
- [ ] Review feed.odin
- [ ] Review frontmatter.odin
- [ ] Review main.odin
- [ ] Review minify.odin
- [ ] Review `markdown/`
- [x] Review alerts.odin
- [x] Review alerts_test.odin
- [x] Review emoji.odin
- [x] Review emoji_test.odin
- [ ] Review footnotes.odin
- [ ] Review footnotes_test.odin
- [ ] Review deflists.odin
- [ ] Review deflists_test.odin
- [ ] Review highlight.odin
- [ ] Review markdown.odin
- [x] Review sectionate.odin
- [x] Review sectionate_test.odin
- [ ] Review suggest.odin
- [ ] Review suggest_test.odin
- [ ] Review `diagnostics/`
- [ ] Review diagnostics.odin
- [ ] Review diagnostics_test.odin
- [x] Review opengraph.odin
- [ ] Review render.odin
- [x] Review site.odin
- [ ] Review treesitter/treesitter.odin
- [ ] Review vfs.odin
- [ ] Review procs
- [ ] markdown.transform_alert
DIAGNOSTICS.md
-61
View File
@@ -1,61 +0,0 @@
# Diagnostics
Thor has two diagnostic tiers. This document explains why, and when to use each.
## Tier 1: Rust-style rich diagnostics (mustache engine)
`mustache/diagnostic.odin` implements multi-line source context, caret underlines, ANSI colors (gated on TTY detection), and Levenshtein suggestions. Used exclusively by the mustache engine for template errors:
- Unknown keys in `{{k}}`, `{{{k}}}`, `{{#k}}`, `{{^k}}`
- Missing partials (`{{> name}}`)
- Missing parent templates (`{{<name}}`)
- Unmatched block overrides (`{{$name}}`)
- Parse-time syntax errors
These benefit from rich diagnostics because **exact source location matters** — templates have complex syntax, and the user often doesn't know *where* the problem is. The diagnostic system operates on `Template.source` with byte offsets, producing output like:
```
error: unknown key 'titel' in {{page.titel}}
--> layouts/page.html:12:22
|
12 | <h1>{{page.titel}}</h1>
| ^^^^^
|
= hint: did you mean 'title'?
```
## Tier 2: Simple log warnings (content and runtime)
Everything outside the mustache engine uses `log.warnf` — flat one-line messages via `core:log`:
- Missing frontmatter dates (fallback to file mtime)
- Duplicate menu weights
- Non-numeric weight values in frontmatter
- Tree-sitter highlight errors
- Menu system issues (mixing config/frontmatter menus, etc.)
These are simple, actionable, and cross-file. The problem isn't *location* — it's that two files disagree, or a value is missing. A caret pointing at one file doesn't help; the message already communicates what to fix:
```
[WARN] --- [menus.odin:290:warn_duplicate_weights()] menus('main'):'Ideas' and 'Stuff' share the same weight (11).
```
## Why not use rich diagnostics everywhere?
Rust's diagnostic model is built for a single compilation unit with full AST/IR data. Three obstacles prevent reusing it for content warnings:
1. **Source tracking**: The mustache diagnostic system operates on `Template.source` (byte offsets into template strings). Content warnings come from frontmatter in markdown files — different source, different parser, no position tracking. Reusing the system would require building a parallel position-tracking infrastructure for frontmatter.
2. **Cross-file context**: Rust diagnostics point at one location. Weight duplicates are a relationship between two files. Rich diagnostics would need to show *both* file locations, which is more infrastructure for marginal value.
3. **Diminishing returns**: Rust diagnostics shine for syntax/type errors where the user doesn't understand the failure. Content warnings are already self-explanatory — "these two pages share weight 11" doesn't need a caret to be actionable.
## When to upgrade a Tier 2 warning to Tier 1
If a warning's usefulness would significantly improve from showing exact source location (e.g., a frontmatter syntax error where the user needs to see *which line* is malformed), consider extending the diagnostic system to frontmatter. This would require:
1. Position tracking in `frontmatter.odin` (store byte offsets for each parsed field)
2. A `format_frontmatter_error` proc modeled on `format_render_error`
3. File path propagation through the page loading pipeline
This is not currently planned — see `TODOS.md`.
DIAGNOSTIC_TESTS.yaml
-379
View File
@@ -1,379 +0,0 @@
# This file contains tests for all known mustache syntax errors and their
# desired error messages. Each one is converted into an Odin test.
name: Missing 2nd closing brace
enable: false # TODO: Re-enable when passing
should: error
input: |
<menu>
<a class="goto-top opacity-0" href="#">{{> icons/chevron_up}</a>
{{#params.social}}
<a href="{{url}}" target="_blank" rel="noopener noreferrer me" title="{{name}}">{{>* icon}}</a>
{{/params.social}}
</menu>
<p>Proudly built with <a href="https://github.com/sbrow/thor/">Thor</a> and <a
expected: |
Missing closing brace
--> <test>:1:1
1 | <menu>
2 | <a class="goto-top opacity-0" href="#">{{> icons/chevron_up}</a>
| ^^ opened ^ never closed
3 | {{#params.social}}
4 | <a href="{{url}}" target="_blank" rel="noopener noreferrer me" title="{{name}}">{{>* icon}}</a>
5 | {{/params.social}}
|
---
name: Missing 2nd opening brace
enable: false # TODO: Re-enable when passing
should: error
input: |
<menu>
<a class="goto-top opacity-0" href="#">{> icons/chevron_up}}</a>
{{#params.social}}
<a href="{{url}}" target="_blank" rel="noopener noreferrer me" title="{{name}}">{{>* icon}}</a>
{{/params.social}}
</menu>
<p>Proudly built with <a href="https://github.com/sbrow/thor/">Thor</a> and <a
expected: |
Missing opening brace
--> <test>:1:1
1 | <menu>
2 | <a class="goto-top opacity-0" href="#">{> icons/chevron_up}}</a>
| ^ opened ^^ closed
3 | {{#params.social}}
4 | <a href="{{url}}" target="_blank" rel="noopener noreferrer me" title="{{name}}">{{>* icon}}</a>
5 | {{/params.social}}
|
---
# TODO: make it work
name: Invalid timezone
enable: false
should: warn
input: |
{
"date": {
"timezone": "America/New_Yorkside"
}
}
expected: "Did you mean 'America/New_York'? pointing to line+col no in the config"
actual: |
[WARN ] --- [148:init_site()] unable to load timezone 'America/New_Yorkside'
---
name: Mismatched section tags
enable: false # TODO: Re-enable when passing
should: error
input: |
<footer>
<menu>
<a class="goto-top opacity-0" href="#">{{> icons/chevron_up}}</a>
{{#params.ocial}}
<a href="{{url}}" target="_blank" rel="noopener noreferrer me" title="{{name}}">{{>* icon}}</a>
{{/params.social}}
</menu>
<p>Proudly built with <a href="https://github.com/sbrow/thor/">Thor</a> and <a
href="https://edwardtufte.github.io/tufte-css/">Tufte CSS</a>
</p>
<p><small>&copy;</small> {{now | format "2006" }} {{params.author.name}}</p>
</footer>
expected: |
Mismatched section: 'ocial' opened, 'social' closed
--> <test>:4:5
|
4 | {{#params.ocial}}
| ^^^^^ opened — did you mean 'social'?
5 | <a href="{{url}}" target="_blank" rel="noopener noreferrer me" title="{{name}}">{{>* icon}}</a>
6 | {{/params.social}}
| ^^^^^^ expected {{/params.ocial}}, got {{/params.social}}
|
---
name: Missing closing section
enable: false # TODO: Re-enable when passing
should: error
input: |
<footer>
<menu>
<a class="goto-top opacity-0" href="#">{{> icons/chevron_up}}</a>
{{#params.social}}
<a href="{{url}}" target="_blank" rel="noopener noreferrer me" title="{{name}}">{{>* icon}}</a>
<!-- {{/params.social}} -->
</menu>
<p>Proudly built with <a href="https://github.com/sbrow/thor/">Thor</a> and <a
href="https://edwardtufte.github.io/tufte-css/">Tufte CSS</a>
</p>
<p><small>&copy;</small> {{now | format "2006" }} {{params.author.name}}</p>
</footer>
expected: |
Missing closing tag '{{/params.social}}'
--> <test>:4:5
|
4 | {{#params.social}}
| ^^^^^^^^^^^^^^^^^ opened
5 | <a href="{{url}}" target="_blank" rel="noopener noreferrer me" title="{{name}}">{{>* icon}}</a>
6 | <!-- {{/params.social}} -->
| ^^^^^^^^^^^^^^^^^^ closing tag is commented out
|
---
name: empty tag
enable: true
should: error
input: |
<footer>
<menu>
<a class="goto-top opacity-0" href="#">{{> icons/chevron_up}}</a>
{{#params.social}}
<a href="{{url}}" target="_blank" rel="noopener noreferrer me" title="{{name}}">{{>* icon}}</a>
{{/params.social}}
</menu>
<p>Proudly built with <a href="https://github.com/sbrow/thor/">Thor</a> and <a
href="https://edwardtufte.github.io/tufte-css/">Tufte CSS</a>
</p>
<p><small>&copy;</small> {{}} {{params.author.name}}</p>
</footer>
expected: |
Empty tag
--> <test>:11:28
|
9 | href="https://edwardtufte.github.io/tufte-css/">Tufte CSS</a>
10 | </p>
11 | <p><small>&copy;</small> {{}} {{params.author.name}}</p>
| ^^^^ tag must contain a key
12 | </footer>
|
---
name: empty raw tag
enable: true
should: error
input: |
Hello, {{&}}
expected: |
Empty tag
--> <test>:1:8
|
1 | Hello, {{&}}
| ^^^^^ tag must contain a key
|
---
name: Missing pipe arguement
enable: true
should: error
input: |
<footer>
<menu>
<a class="goto-top opacity-0" href="#">{{> icons/chevron_up}}</a>
{{#params.social}}
<a href="{{url}}" target="_blank" rel="noopener noreferrer me" title="{{name}}">{{>* icon}}</a>
{{/params.social}}
</menu>
<p>Proudly built with <a href="https://github.com/sbrow/thor/">Thor</a> and <a
href="https://edwardtufte.github.io/tufte-css/">Tufte CSS</a>
</p>
<p><small>&copy;</small> {{now | group_by }} {{params.author.name}}</p>
</footer>
expected: |
group_by expects 1 argument, got 0
--> <test>:11:36
|
9 | href="https://edwardtufte.github.io/tufte-css/">Tufte CSS</a>
10 | </p>
11 | <p><small>&copy;</small> {{now | group_by }} {{params.author.name}}</p>
| ^^^^^^^^ missing argument
12 | </footer>
|
---
name: Double dot access
enable: false # TODO: Re-enable when passing
should: error
input: |
<footer>
<menu>
<a class="goto-top opacity-0" href="#">{{> icons/chevron_up}}</a>
{{#params.social}}
<a href="{{url}}" target="_blank" rel="noopener noreferrer me" title="{{name}}">{{>* icon}}</a>
{{/params.social}}
</menu>
<p>Proudly built with <a href="https://github.com/sbrow/thor/">Thor</a> and <a
href="https://edwardtufte.github.io/tufte-css/">Tufte CSS</a>
</p>
<p><small>&copy;</small> {{now | format "2006" }} {{params..name}}</p>
</footer>
expected: |
Invalid key path: consecutive dots
--> <test>:11:60
|
9 | href="https://edwardtufte.github.io/tufte-css/">Tufte CSS</a>
10 | </p>
11 | <p><small>&copy;</small> {{now | format "2006" }} {{params..name}}</p>
| ^^ missing path name
12 | </footer>
|
---
name: Too many opening braces
enable: false # TODO: Re-enable when passing
should: error
input: |
<footer>
<menu>
<a class="goto-top opacity-0" href="#">{{> icons/chevron_up}}</a>
{{#params.social}}
<a href="{{url}}" target="_blank" rel="noopener noreferrer me" title="{{name}}">{{>* icon}}</a>
{{/params.social}}
</menu>
<p>Proudly built with <a href="https://github.com/sbrow/thor/">Thor</a> and <a
href="https://edwardtufte.github.io/tufte-css/">Tufte CSS</a>
</p>
<p><small>&copy;</small> {{{{now | format "2006" }}} {{params.author.name}}</p>
</footer>
expected: |
Too many opening braces
--> <test>:11:28
|
9 | href="https://edwardtufte.github.io/tufte-css/">Tufte CSS</a>
10 | </p>
11 | <p><small>&copy;</small> {{{{now | format "2006" }}} {{params.author.name}}</p>
| ^^^^ expected {{{, got {{{{
12 | </footer>
|
---
name: Too many closing braces
enable: false # TODO: Re-enable when passing
should: error
input: |
<footer>
<menu>
<a class="goto-top opacity-0" href="#">{{> icons/chevron_up}}</a>
{{#params.social}}
<a href="{{url}}" target="_blank" rel="noopener noreferrer me" title="{{name}}">{{>* icon}}</a>
{{/params.social}}
</menu>
<p>Proudly built with <a href="https://github.com/sbrow/thor/">Thor</a> and <a
href="https://edwardtufte.github.io/tufte-css/">Tufte CSS</a>
</p>
<p><small>&copy;</small> {{now | format "2006" }}}} {{params.author.name}}</p>
</footer>
expected: |
Too many closing braces
--> <test>:11:50
|
9 | href="https://edwardtufte.github.io/tufte-css/">Tufte CSS</a>
10 | </p>
11 | <p><small>&copy;</small> {{now | format "2006" }}}} {{params.author.name}}</p>
| ^^^^ expected }}, got }}}
12 | </footer>
|
---
# TODO: Implement
name: Frontmatter missing closing quote on key
enable: false
should: warn
input: |
{
"title: Docs"
}
[TOC]
## Introduction
expected: 'TODO:'
actual: |
[ERROR] --- [42:parse_frontmatter()] failed to parse frontmatter JSON: Expected_Colon_After_Key
---
# TODO: Implement
name: Fronmatter missing opening quote on value
should: warn
enable: false
input: |
{
"title": Docs"
}
[TOC]
## Introduction
expected: 'detailed diagnostic pointing to the point of failure and suggesting the addition of a quote'
actual: Nothing. The page compiles with no warnings or errors
---
name: No error when missing index page
enable: false # TODO: Re-enable when passing
should: error
expected: 'Your site has no index page! Please create an index.md or index.html file in ./content/'
---
# FIXME: Doesn't work properly
name: format pipe with no date format set
enable: false # TODO: Re-enable when passing
should: warn
input: '{{ now | format }}'
expected: |
no date format configured — set date.format in thor.json or specify a format, e.g. {{key | format "2 Jan 2006"}}
--> <test>:1:10
|
1 | {{now | format}}
| ^^^^^^ defaulting to "2 Jan 2006"
---
# TODO: called First/last, but only tests first
name: first/last with zero argument
enable: true
should: error
input: |
<ul>{{#items | first 0}}<li>{{.}}</li>{{/items}}</ul>
expected: |
argument must not be zero
--> <test>:1:16
|
1 | <ul>{{#items | first 0}}<li>{{.}}</li>{{/items}}</ul>
| ^^^^^^^
|
---
# TODO: called First/last, but only tests first
name: first/last with negative argument
enable: true
should: warn
input: |
<ul>{{#items | first -3}}<li>{{.}}</li>{{/items}}</ul>
expected: |
first -3 is equivalent to last 3
--> <test>:1:16
|
1 | <ul>{{#items | first -3}}<li>{{.}}</li>{{/items}}</ul>
| ^^^^^^^^ consider using the clearer form
|
---
# TODO: called First/last, but only tests first
name: first/last on string with no argument
# enable: true
input: '{{word | first}}'
should: warn
expected: |
defaulting to n=1 for string
--> <test>:1:1
|
1 | {{word | first}}
| ^^^^^ consider specifying n explicitly
|
---
name: "first succeeds on params.author.name"
enable: true
input: '{{ params.author.name | first 3 }}'
should: ok
expected: "foo"
---
# TODO: called First/last, but only tests first
# TODO: Include | last 3 }} in the suggestion.
name: "first/last in partial"
enable: true
input: "{{> footer }}"
should: warn
expected: |
first -3 is equivalent to last 3
--> fixtures/layouts/partials/footer.html:11:78
|
9 | href="https://edwardtufte.github.io/tufte-css/">Tufte CSS</a>
10 | </p>
11 | <p><small>&copy;</small> {{! now | format "2006" }} {{params.author.name | first -3 }}</p>
| ^^^^^^^^ consider using the clearer form
12 | </footer>
|
---
name: "Pipes are ignored in comments"
enable: true
should: ok
input: 'Hello, {{! now | format "2006" }}'
expected: 'Hello, '
GRAMMAR_CACHE_THREADING.md
-171
View File
@@ -1,171 +0,0 @@
# Grammar cache thread-safety & allocator lifetime
Status: **resolved.** All three issues below are fixed. The cache was redesigned
so the library caches only the immutable, shareable objects and callers own the
per-thread ones. The rest of this document is kept as the rationale.
## Resolution (what shipped)
The `Grammar_Cache` (which mixed shareable `language`/`query` with per-thread
`parser`/`cursor`) was split:
- **`Grammar`** holds only the immutable `language` + `query`. It lives in a
process-lifetime **`registry`** (`map[string]^Grammar`) guarded by one mutex.
- **`grammar(lang)`** is the thread-safe, self-initializing, lazy loader — fixes
**#1a** (all map access under the lock) and lets tests run with no setup.
- The registry allocator is pinned with `os.heap_allocator()` — fixes **#2**; the
cache no longer captures a transient `context.allocator`.
- Parsers/cursors are **caller-owned**: `open_parser(g)` mints one, the caller
`defer`s `parser_delete` / `query_cursor_delete`. No parser is shared, so
**#1b** is gone. `open_parser` is the seam where a future pool / thread-local
strategy plugs in without touching call sites.
- Deleted: `ensure_parser`, `load_grammar`, `preload_grammar`, `grammar_worker`,
`cache_mutex`; `preload_grammars` is now a serial registry warm-up. The minify
tests dropped their `parse_mu` serialization and preload scaffolding entirely.
The **later** decision — pool vs per-thread parser reuse for genuine parallel
parsing — is still open, but intentionally so: it only matters once we parse in
parallel, and it now changes only `open_parser`, not the rest of the library.
The original investigation follows.
## How we got here
While adding a feature to minify inline `<style>` blocks (route the CSS body
through `minify_css`), the new unit tests exercised `minify_html` / `minify_css`
for the first time. Those are the first tests to touch the tree-sitter parser
cache, and under the multi-threaded test runner (default 16 threads) they failed
**intermittently** with a rotating cast of symptoms:
- `panic: unable to insert into a map`
- tree-sitter internal aborts: `ts_subtree_retain: Assertion 'self.ptr->ref_count > 0' failed`, `_array__erase: Assertion 'index < *size' failed`
- `malloc(): unaligned tcache chunk detected`
- assertions passing on the wrong data (e.g. minified output missing `<style>`)
The flakiness comes from **three distinct problems**, two thread-safety and one
allocator-lifetime. They are orthogonal — fixing one does not fix the others.
## The three issues
### #1a — concurrent cache **writes** (data race)
`ensure_parser` (`treesitter.odin`) reads and writes `grammar_store.cache`
with **no lock**. `cache_mutex` exists but is only taken on the parallel preload
path (`grammar_worker`), never in the lazy `ensure_parser` / `load_grammar`
paths. Two threads first-touching `"html"`/`"css"` at once corrupt the map.
### #1b — concurrent parser **use** (data race)
There is exactly **one cached `Grammar_Cache` (and one `parser`) per language**.
When two threads both call `minify_html`, they fetch the *same* `gc.parser` and
call `parser_parse_string` on it simultaneously. A tree-sitter `TSParser`
mutates its internal subtree pool during a parse, so concurrent parses corrupt
it — that's the `ts_subtree_retain` / `_array__erase` aborts.
This is **not fixed by preloading/"loading html+css statically."** Loading gives
you *one shared* parser; the race is about *using* that one parser from two
threads. It is fixed only by serializing the parse or giving each thread its own
parser.
### #2 — allocator lifetime (not a race)
`init_persistent` captures `context.allocator` into the process-lifetime
`grammar_store`. In `main` that happens to be the heap (it runs before the site
arena is installed at `main.odin`), so production is fine. But whatever
allocator is live at that instant becomes the cache's owner forever:
- Under the **test runner**, `context.allocator` is a **per-test tracking
allocator** that is torn down when that test ends → the global cache is left
pointing at dead memory (→ `unable to insert into a map`, garbage reads).
- A **lazy** first call during rendering would capture the **site arena**
(`context.allocator` after `main.odin` swaps it in), which is destroyed every
watch-mode rebuild → dangling parsers across builds.
This would break even single-threaded; threading just decides *which* test wins
the init race.
## tree-sitter's threading model (the root fact)
tree-sitter objects have specific, documented threading rules:
| Object | Shareable across threads? | Rule |
|---|---|---|
| `TSLanguage` (`tree_sitter_html()` etc.) | Yes | Immutable; share freely. |
| `TSParser` (`gc.parser`) | **No** | One per thread; never parse on one parser from two threads. |
| `TSTree` | Only via `ts_tree_copy` | Copy before using on another thread. |
| `TSQuery` (`gc.query`) | Yes | Immutable after compile. |
| `TSQueryCursor` (`gc.cursor`) | **No** | One per thread, like the parser. |
This explains the existing design:
- The preload fan-out (`preload_grammar` / `grammar_worker`) creates a parser
**per worker** and only publishes the finished `Grammar_Cache` under
`cache_mutex` — because parsers can't be shared during construction/use.
- `ensure_parser` + the shared cached parser is fine in production **only
because rendering is single-threaded**.
A cached `Grammar_Cache` mixes shareable (`language`, `query`) and per-thread
(`parser`, `cursor`) objects in one shared struct, which makes it
single-threaded-use by construction — a fact that was previously undocumented.
## Where this actually bites
**Only in tests.** Confirmed by grep:
- Production parses on one thread: every real caller — `minify.odin` (html/css)
and `markdown/highlight.odin` — runs inside `main`'s single-threaded render
loop. No concurrent `ensure_parser` / `parser_parse_string`.
- The one threaded path in production, `preload_grammars`, is already safe (own
parser per worker + `cache_mutex` on the write) and only handles non-builtin
grammars.
- Only `minify_test.odin` touches tree-sitter from tests, and the test runner
runs those on many threads.
So production is correct **by convention** (single-threaded-from-heap usage),
not **by construction** — nothing stops a future parallel render from breaking
it, and until now nothing documented the constraint.
## Chosen direction
**For now:** keep the single global grammar cache and make it safe by
**serializing the non-thread-safe behavior with a mutex** — only one thread uses
a parser at a time. This is simple, matches how production already behaves, and
unblocks the tests.
**Later (TODO):** decide whether true multi-threaded parsing is worth it for
performance. If yes, give each thread its own parser + cursor (as the preload
workers already do) rather than sharing one behind a lock.
## Implementation options (deferred — pick when we resume)
All assume the allocator fix (#2) below, which is independent and unambiguous.
- **#2 allocator (do regardless):** pin the heap allocator explicitly —
`grammar_store.allocator = runtime.heap_allocator()` (or `os.heap_allocator()`)
instead of `context.allocator`. Makes the cache persistent from any context;
behavior-identical in production. Optionally make `init_persistent` idempotent
and/or an `@(init)` so callers need no ordering.
Then, for the races:
- **Option A — library mutex (sustainable):** take `cache_mutex` in
`ensure_parser` (fixes #1a) and add a locking `parse` helper that wraps
`parser_parse_string` (fixes #1b), wired into the three call sites
(`minify.odin` ×2, `markdown/highlight.odin` ×1). Tests then need no
scaffolding. Serializes all parsing.
- **Option B — test-side only (minimal):** leave the library single-threaded by
contract; make the tests not violate it. Either fold the minify tests into one
`@(test)` (no concurrency at all → no locks), or wrap the `minify_*` calls in a
test-local mutex. Production stays lock-free.
Recommendation captured during the discussion: the allocator one-liner is
worth doing immediately; for the races, Option A is the "correct" long-term
answer but only necessary if/when we want concurrent parsing — otherwise
Option B is enough.
## Pointers
- `treesitter/treesitter.odin`: doc comments on `Grammar_Cache`, `cache_mutex`,
`init_persistent`, `ensure_parser` reference this file.
- Original feature that surfaced this: minify inline `<style>` bodies via
`minify_css` (reverted; to be re-landed alongside whichever fix we choose).
HUGO.md
-11
View File
@@ -1,11 +0,0 @@
# Hugo Notes
from the [Hugo docs](https://gohugo.io/quick-reference/glossary/#default-sort-order)
default sort order
: The default sort order for page collections, used when no other criteria are set, follows this priority:
1. weight (ascending)
2. date (descending)
3. linkTitle falling back to title (ascending)
4. logical path (ascending)
PROBLEMS.md
-317
View File
@@ -1,317 +0,0 @@
# Thor — UX Problems Catalog
Adversarial review of error messages, behavioral inconsistencies, and user
frustration points. Established as a baseline on commit `314cab2`.
Each entry cites the source location so it can be tracked to a fix.
---
## Severity legend
- **Critical** — user mistake produces silent wrong output or an unhelpful
fatal error with no path forward.
- **High** — error or warning is emitted but missing "where" or "how to fix."
- **Medium** — inconsistency or gotcha that causes confusion or rework.
- **Low** — polish / minor frustration.
---
## A. Silent wrong output (no error, wrong result)
These are the most dangerous — the user gets *no signal* that something is wrong.
### A1. Non-JSON frontmatter silently treated as body content — Critical
`frontmatter.odin:26`
Thor expects JSON frontmatter delimited by bare `{` / `}` lines. A user
coming from Hugo/Jekyll writes YAML (`---`) or TOML (`+++`) frontmatter. It is
silently swallowed into the markdown body. No title, no date, no draft flag —
and no error. Likely the #1 onboarding trap.
### A2. Unknown `thor.json` keys silently ignored — Critical
`site.odin:162`
`json.unmarshal_string` skips unknown fields. A typo like `"tittle"` instead
of `"title"` produces a silently-empty title. No warning. (`TODOS.md` already
wants a JSON schema.)
### A3. Draft pages silently excluded — High
`content.odin:64`
When `-drafts` isn't passed, draft pages vanish with no log. User adds a
page, forgets the flag, page doesn't appear — zero feedback.
### A4. Naive singularization for layout inference — High
`content.odin:202`
`posts` → `post` (correct), but `series` → `serie`, `news` → `new`. The
layout silently falls through the fallback chain to `page`/`base`. No
"layout 'serie' not found for section 'series'" message — only a debug log
that's off by default.
### A5. `base_url` defaults to `localhost:8080` — Critical
`site.odin:100`
Forgetting to set it means every canonical URL, OG tag, and RSS link points
to localhost. No warning. Devastating in production builds.
### A6. Missing `content/` produces empty build — High
`content.odin:82`
`scan_content_files` logs a `warnf`, the build proceeds with zero pages, then
`log.infof("Rendered 0 pages")`. No fatal error, no "did you create
content/?" guidance.
### A7. RSS emits sentinel epoch date silently — Medium
`feed.odin:33`
Pages without a date get `"Mon, 01 Jan 0001 00:00:00 +0000"` in `<pubDate>`.
No warning that a page is dateless in the feed.
### A8. `format_rfc822` returns raw ISO on parse failure — Medium
`feed.odin:122-125`
`// TODO: should indicate error somehow` — short/malformed dates get embedded
verbatim in `<pubDate>`, producing invalid RSS with no warning.
---
## B. Error messages missing "where" or "how to fix"
### B1. `render_template` blanks the entire page on error — Critical
`render.odin:119-133`
A single bad tag/pipe anywhere produces `log.errorf` + `return ""`. The
output file is silently written empty. In a `nix build` (no visible
terminal), the user sees a blank page with zero clue why. Already noted in
`TODOS.md`.
### B2. Malformed `thor.json` degrades to defaults — Critical
`site.odin:162-168`
A JSON syntax error is a `warnf`, then `site_apply_path_defaults` kicks in.
The site builds with wrong paths and produces a confusing empty result — the
cause is two hops removed from the symptom.
### B3. Frontmatter parse error has no file location — Critical
`frontmatter.odin:41`
`"failed to parse frontmatter JSON: %v"` — no filename. On a 100-post site
the user can't find the bad file. Worse: `ok=false` silently drops the page
entirely.
### B4. `get_template` returns empty `Template{}` on missing base — High
`render.odin:88-89`
`"base.html not found in VFS"` — no guidance on how to fix (create the file,
check modules, etc.).
### B5. `dlopen` failures lack the OS reason and fix guidance — High
`treesitter/treesitter.odin:200-219`
"cannot load grammar %s (%s)" shows the path but not *why* (no `dlerror()`).
No guidance: "set the 'grammars' key in thor.json" or "this .so may be for a
different tree-sitter ABI."
### B6. Menu-mix fatal lacks location — High
`menus.odin:42`
`"cannot mix config menus with frontmatter menus"` — doesn't name which pages
have frontmatter menus.
### B7. Minify error doesn't name the page — Medium
`minify.odin:33`
"minify: HTML parse errors, skipping minification" — across 50 pages, which
one?
### B8. Timezone load failure is a warning with no guidance — Medium
`site.odin:148`
Doesn't state impact (dates render in UTC) or suggest valid names. Already
in `TODOS.md`.
### B9. No "config not found" message — Medium
`site.odin:113`
Silently falls back to `./thor.json`. Wrong-directory runs produce a
confusing default build.
---
## C. Silent skip of invalid user input
### C1. Unknown markdown extensions silently ignored (CLI) — High
`markdown/markdown.odin:49-68`
`parse_extension_list` has a switch with no default case. `-ext:higlight`
(typo for `highlight`) is silently a no-op.
### C2. Unknown markdown extensions silently ignored (config) — High
`markdown/markdown.odin:71-90`
`apply_extension_config` has a `// TODO: Silently discards invalid values.`
Unknown keys in `thor.json`'s `markdown_extensions` are silently dropped.
Non-boolean values are `or_continue`d.
---
## D. Naming inconsistencies
### D1. Markdown extensions have 3+ names — Medium
| Context | Name |
|---|---|
| `thor.json` key | `markdown_extensions` |
| CLI flag | `-ext` / `-no-ext` |
| Struct fields | `md_enable` / `md_disable` |
| JSON/CLI values | `emoji`, `sidenotes` (lowercase) |
| Enum members | `.Emoji`, `.Sidenotes` (PascalCase) |
### D2. `-ext` usage string omits `heading_ids` — Medium
`site.odin:91`
The help text lists `emoji,sidenotes,alerts,highlight,sections` but the enum
also has `HeadingIDs`. Users can't discover it from `--help`.
### D3. Starred field has three names — Low
- `Page.starred` (`content.odin:28`)
- `Frontmatter.isStarred` (`frontmatter.odin:17`) — so the JSON key is `isStarred`
- `AGENTS.md:101` says `is_starred` (stale)
### D4. Inconsistent error severity for similar failures — Medium
- Template **parse** error → `log.errorf` + `os.exit(1)` (fatal) — `render.odin:37-49`
- Template **render** error → `log.errorf` + return `""` (non-fatal, blank page) — `render.odin:126-131`
- Config parse error → `warnf` + fallback to defaults — `site.odin:163-165`
Same category of failure (user wrote something wrong) with wildly different
consequences.
### D5. Dead `os.exit(1)` after `log.fatalf` — Low
`render.odin:33`, `menus.odin:43`
`fatalf` already exits; the following line is dead code.
---
## E. Configuration gotchas
### E1. `"menus": {}` is a stealth opt-out — Medium
`menus.odin:31-38`
An empty object silently disables *all* auto-menus. A user who adds the key
intending to configure later quietly loses their nav. The semantics (absent
≠ empty) are undocumented outside code comments.
### E2. Config precedence is invisible — Medium
`site.odin`
CLI > JSON > defaults, but there's no "resolved config" log. Debugging "why
is my base_url wrong?" requires reading source.
### E3. `format` pipe logs ERROR but still renders — Medium
`pipes.odin:261-265`
Missing `date.format` produces `log.errorf` but falls back to
`DEFAULT_DATE_FORMAT`. The severity says "error" but the behavior says
"warning."
---
## F. File/directory behavior surprises
### F1. Root dirs = sections, nested dirs = leaf bundles — Medium
`content.odin:108-127`
This meaningful semantic distinction is entirely implicit.
`content/about/team.md` is a leaf bundle (page "about" with body from
team.md), not a section "about" with page "team". No error or guidance when
the user's mental model differs.
### F2. Missing `layouts/` silently uses defaults — High
`vfs.odin:38-40`
`mount_dir` returns silently if the directory doesn't exist. Wrong path →
all user templates missing → defaults used. No "layouts directory X not
found" message.
### F3. Missing section index silently synthesized — Medium
`render.odin:316-326`
A section with pages but no `index.md` gets a synthetic `Page` with only a
title. No warning. User expecting an error gets a mostly-blank page.
### F4. Windows line endings silently break frontmatter — Medium
`frontmatter.odin:26`
`has_prefix(content, "{\n")` fails on `\r\n`; the JSON is treated as body.
Zero feedback.
---
## G. Template authoring frustrations
### G1. Template fallback chain is silent at Info level — Medium
`render.odin:84`
Only `log.debugf`, which is off by default (`main.odin:48` sets `.Info`).
User's custom layout silently ignored, defaults used.
### G2. Render error blanks entire page — Critical
`render.odin:126-131`
(Same as B1 — restated here for the template-authoring perspective.) One bad
tag → whole page `""`. The most impactful silent failure in the system.
### G3. Pipe errors lack "did you mean?" suggestions — High
`mustache/pipes.odin:241`
Unknown key errors (`{{tittle}}`), missing partials, and unmatched block
overrides all get Levenshtein "did you mean?" hints via `suggest_correction`.
But unknown pipe operations (`{{date | formats}}`) get only `"unknown pipe op
'formats'"` with no suggestion. The known filter names (`"format"`,
`"group_by"`) are a small fixed set — perfect for suggestions.
Structural gap: `Error_Body` has no `hint` field, and
`format_render_error` doesn't pass `hint` to `format_error` (defaults to
`""`). So even if a suggestion were computed, there's nowhere to put it
without either appending to `msg` or adding `hint` to `Error_Body`.
### G4. Triple-mustache `{{{` mishandled by `tag_content_base` — Medium
`mustache/mustache.odin:230`
`tag_content_base` skips `{{` and sigils (`#^/&><$!`) to find where tag
content begins. But triple-mustache `{{{key}}}` is common — after `{{`,
the next char is `{`, which is not in the sigil list, so `base` points at
`{` instead of the actual key content. Any pipe position calculation for
`{{{key | format}}}` will be off by one byte.
---
## Cross-cutting themes
1. **Debug-level logging masks important fallbacks.** Layout fallbacks,
template misses, and grammar skips are all `debugf` — invisible at the
default Info level. Users never learn their customizations were ignored.
2. **The system fails open, not closed.** Missing files, missing
directories, missing config — all silently fall back to defaults rather
than surfacing the problem. Friendly until it isn't.
3. **No "resolved state" visibility.** There's no way for a user to see what
thor actually loaded: which layouts, which config values, which pages
were skipped as drafts. The build is a black box.
---
## Gold-standard examples to emulate
These are the parts of the codebase that already do it right:
- **Mustache diagnostics** (`mustache/diagnostic.odin` + `suggest.odin`):
rust-style multi-line context, caret underlines, Levenshtein "did you
mean?" hints, file:line:col.
- **Treesitter query version-mismatch** (`treesitter/treesitter.odin:299-315`):
explains the likely cause, shows both grammar/query versions, flags
mismatches explicitly.
README.md
-39
View File
@@ -1,39 +0,0 @@
# Thor
Thor is a simple Static Site Generator designed for personal blogs and other small websites.
Its core principals are simplicity and minimal configuration, so you can get started as quickly as possible.
It is based on Hugo, and gingerbill's SSG. Templating is done with (extended?) Mustache templates.
## What it does
- Syntax Highlighting server-side (with Tree Sitter) or client-side with highlight.js
- Mustache Templating
- OpenGraph Tags
- Menus (WIP)
- Extended Markdown ([See below](#extended-markdown))
- Basic (whitespace) minification.
- Union File System (Modules)
## What it doesn't do
- Internationalization
- Pagination (Yet)
- Themes
- Image Manipulation
- TailwindCSS integration
## Getting Started
Run `thor`
Check out `public/index.html`
Then follow [The Guide]()
For a more complete setup, run `thor new site`.
## Extended Markdown
- Emoji expansion
- margin style footnotes
- Github style alerts
- [and more]
ROADMAP.md
-131
View File
@@ -1,131 +0,0 @@
## v0.2.0
- [ ] [aliases](https://gohugo.io/methods/page/aliases/#redirects)?
- [ ] consider sites with date based urls.
- [ ] Create a json schema file for `thor.json`.
- [ ] Content-hash fingerprinting for CSS and JS cache busting
- [ ] Mount content in VFS
- [ ] `-watch` flag
- [x] basic poll loop
- [ ] filesystem poll loop
- [ ] event based
- [ ] if no `html` tag detected in output, re-render output with base template
(or whatever template is next in the chain). (Hugo seems to do this)
- [ ] Markdown
- [ ] Block attributes extension — hello-world.md
- [ ] add tables extension
- [ ] enable template level rendering of TOCs
- [ ] Write css for toc sidebar and figure out where to put it.
- [ ] Add [hugo style configuration](https://gohugo.io/configuration/markup/#table-of-contents)
- [ ] implicit page titles (Set when missing?)
- guess from h1 tag or first h2 tag?
- [ ] Syntax highlighting
- [ ] grammars
- [ ] load grammars dynamically
- [ ] Search in multiple places
- [ ] Download missing grammars.
- [ ] Durable highlight paths: read `GRAPHS_PATH`/`QUERIES_PATH` from env vars set by the flake instead of hardcoded nix store hashes, so they survive `nix flake update` and let the grammar/query version-mismatch detector fire automatically.
- [ ] Theme selector
- use http://github.com/helix-editor/helix/tree/master/runtime/themes) as a
guide
- [ ] Unit tests for highlighting helpers: `capture_name_to_css`, `escape_html`, `unescape_html`, `extract_query_token`, `helix_version_from_path`.
- [ ] `<pre><code>` blocks need to set background to theme background,
regardless of prefers-dark. (or use a different theme)
- [ ] Avoid `json.Value` / `json.Object` where possible.
- [ ] come up with scrapers / scrape sources to harvest site data
- we'll use this to help us sculpt defaults.
- [ ] merge `render_{section,home_html,page_html}` procs.
- [ ] try to combine render_page_html and render_home_html?
- [ ] make `parse` an overload of `parse_text/parse_inline` and `parse_file`, or something.
- [ ] refactor `mustache.load_partials` to accept an iterator so we don't copy `main.load_partials`
### Menus
- [ ] Nav items should be active when the current page is selected.
- [ ] configure opt-out of automatic sections being added to menu.
- [ ] nested menus (i.e. `parent` support)
### Strict Mode
- [ ] opt-in "strict_keys" mode. in this mode, key lookups may not view parent objects.
- [ ] warn/error when unknown key used in mustache.
- [ ] enforce heading structure.
- [ ] Either frontmatter.title set, or 1 h1 tag at top, not both
- [ ] No skipping.
### Dates
- [ ] display an error when no part of the date appears in the output.
- [ ] Handle 0 and whitespace padding i.e. "_2" -> " 2"
- [ ] Do we *need* mustache.Date_Components, or can we use core:time/datetime.DateTime?
## Beyond
- [ ] consider logging the number of times an error occurred. (diagnostics)
- [ ] Need to be careful about diagnostics across module boundaries.
- we don't necessarily want to warn users about theme designers mistakes. (though perhaps we do)
- [ ] consider reporting duplicate weights outside of menus (diagnostic)
- [ ] Do mustache's whitespace rules actually suit us, or should we make our own?
- [ ] create a default `head.html`?
- [ ] improve home link customization.
- [ ] currently an accessibility issue.
- [ ] get rid of the global variables in the `treesitter` package.
- [ ] site-wide Integrity hash
- Allows users to verify their output didn't change after upgrading to a new version
- [ ] Add `-production` flag
- sets `-minify`
- [ ] manually pass `site_allocator` to `load_page`
- [ ] Import/export packages. Hugo, jekyll, WordPress, etc.
- [ ] shortcodes?
- include-code shortcode (`{{< include-code ... >}}`) — i-ported-fd-to-odin
- [ ] follow symlinks in `scan_content`?
- [ ] Add overloads for every md extension? i.e. accept ^strings.Builder.
- [ ] content link checker?
- Checks all links on each page to make sure they are valid.
- [ ] Peruse [GitHub's](https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax#alerts)
docs for any juicy nuggets we may have missed.
- [ ] Avoid using `render_inline_md` if possible.
- [ ] don't use bullshit "sub-tokens", add filters and pipes as proper tokens.
- [ ] Create a Location struct that somewhat matches Odin's [Source_Code_Location](https://pkg.odin-lang.org/base/runtime/#Source_Code_Location)?
- Note that odin's version doesn't contain the stack trace.
- [ ] Link checker / linter that warns if links don't resolve. ("please use {{whatever | rel_url }}").
### Memory Management
- [ ] Leaks in highlighter code.
- [ ] Not sure whether to use temp allocator or site_allocator in opengraph.odin.
- [ ] Not sure whether to use temp allocator or site_allocator in `site_load_content`.
- [ ] Might not need to allocate in `strip_html_tags`
- [ ] Fix `apply_filter`'s `format` case (`mustache/pipes.odin`) boxing `apply_format`'s
`string` result into `any` via bare `return`, which materializes a hidden
header temp in `apply_filter`'s own stack frame. Dangling once the frame
returns; caused the `-o:speed` segfault in `write_value`. Fix: box explicitly
with `any{new_clone(formatted, context.temp_allocator), typeid_of(string)}`.
- [ ] Same pattern in `apply_group_by` (`mustache/pipes.odin`): `return groups, nil`
boxes a freshly-built `[dynamic]Group` as bare `any` — same latent
stack-temp UB, hasn't crashed yet but should get the same treatment.
### Performance
- [ ] See if we can disable bounds checks in `write_indented` and elsewhere.
- [ ] Instead of loading the site fresh each time in watch mode, create a
`reload_site` proc, that just updates changed resources.
- [ ] Only publish referenced assets.
- [ ] Split `load_page` into frontmatter-parse + body-process phases so draft pages can skip the markdown pipeline entirely
- [ ] Use spall to find ways to reduce run time.
- [ ] too many `write_string` calls in `highlight_block`
- [ ] return `src: cstring` from `load_query`.
- [ ] Improve `unescape_html` with simd.
- [ ] generate summary before syntax highlighting.
- [ ] generate summary before markdown to html conversion.
- [ ] mount_recursive is pretty significant
- [ ] thread pool for grammar loading is unbounded.
- [ ] load grammars async.
- [ ] during `load_page`:
- pass each code block to the treesitter queue
- continue working on the page,
- `await` the highlighted code.
- [ ] can markdown extensions run in parallel?
- [ ] enforce MAX_SLUG_LENGTH
- [ ] enforce MAX_CONTEXT_DEPTH
- [ ] ensure struct fields are ordered correctly
TODOS.md
+17 -143
View File
@@ -1,143 +1,17 @@
# TODOs - [ ] add content-hash fingerprinting for tailwind cache busting.
- [ ] create template language
## Priority Unclear (Evaluate Later) - Look at the source for 3 template languages that support mustache syntax,
and see how they implement the tempaltes, then pick the best one. One of them
- [ ] Decide the parallel-parsing strategy for the grammar cache (pool vs should be Hugo / go.
thread-local parsers) — deferred until the parallel usage shape is clear. - [ ] robots.txt?
- Resolved already: the registry (`grammar()`) is thread-safe and self- - [ ] Evaluate Tufte CSS — borrow sidenote CSS or replace TailwindCSS entirely
initializing, its allocator is pinned to the heap, and parsers/cursors are now - Option B: Steal Tufte's sidenote/margin-note CSS (adapt for dark theme), keep TailwindCSS
caller-owned (`open_parser`) instead of cached — so the library holds no - Option C: Full Tufte CSS — drop TailwindCSS, no build step, semantic HTML, customize for dark theme + Roboto
single-thread-use state and the multi-threaded tests pass without locks. - Our sidenote HTML pattern already matches Tufte's exactly
- Remaining choice only matters once we actually parse in parallel: reuse - [ ] GitHub alerts (`> [!CAUTION]` → styled blockquote) — 2 posts
parsers via a pool or per-thread storage, plugged in behind `open_parser` - [ ] Block attributes on code fences (`{ #ex-1 }`) — hello-world.md
without touching call sites. A `TSParser`/`TSQueryCursor` is one-per-thread. - [ ] Emoji shortcodes (`:shrug:` etc.) — 2 instances
- Full write-up: `GRAMMAR_CACHE_THREADING.md`. - [ ] include-code shortcode (`{{< include-code ... >}}`) — i-ported-fd-to-odin
- [x] Restore parallel grammar preload (serialized in `edb2e64`); `preload_grammars` - [ ] Nix build integration — main flake runs thor + tailwindcss instead of Hugo
now fans `build_grammar` across a bounded `core:thread.Pool`. - [ ] Copy-to-clipboard button on code blocks (was in Hugo's custom-head.html)
- [ ] Add tests: pool fan-out correctness + `-sanitize:thread` for race-freedom. - [ ] OpenGraph meta tags
- [ ] Concurrent-read path: an `RW_Mutex` on `grammar()` only helps a future
multi-threaded render, not the load side. Revisit if render goes parallel.
- [ ] instead of `warnings: [dynamic]Error` we should use `warnings: [dynamic; 8]Error`
- when limit reached, the template fails and stops rendering.
- [ ] Brainstorm plugin/tool/(postccs/less/tailwind/sass) support.
- [ ] `fingerprint` pipe + contents-based composition. The `tool` pipe currently
carries a URL string and each stage publishes a file to the output dir, so
`| tool sass | fingerprint` would leave a dead unhashed intermediate. When
fingerprint lands, move the pipe value from a URL string to a contents/
Resource (bytes + logical name) where only the terminal stage publishes;
a Resource must stringify to its URL so existing bare-`tool` templates keep
working. stdin/stdout tools compose best there (bytes stay in memory), which
is why stdin/stdout is the documented preferred form over `{{in}}`/`{{out}}`.
- [ ] Reconsider `"menus": {}` opt-out: it currently discards all page frontmatter
menus (with only a warning). Decide whether empty config should instead just
suppress the auto-generated menu while still honoring frontmatter.
- [ ] asset css files not automatically included in site.
- should we at least warn about them?
- [ ] Review the whole `copy_assets_dir` pipeline. It copies every file under
`assets/` to the output dir blindly, whether or not anything references it,
and now also skips dests that already exist (a blunt collision guard so the
raw copy doesn't clobber `tool`-pipe output written earlier in the build).
Reconsider tracking which assets are actually used/published instead.
- [ ] warn if stylesheet url starts with /assets (and is therefore invalid).
- [ ] warn when heading levels are skipped in a toc.
- [ ] extract `[TOC]` from `.md` body (or at least from the start.)
- [ ] add `--production` flag
- set `minify` to true
- warn when `base_url` unset
## General
Remember to polish existing features before moving on to new ones.
- [ ] Bake the default `layouts` into the binary at compile time via
`#load_directory` so the binary is self-contained and doesn't read
`DEFAULTS_PATH` from disk at runtime.
- `#load_directory` is NOT recursive and its `name` is the base filename
only, so `defaults/layouts` and `defaults/layouts/partials` must be loaded
separately and mounted into the VFS with `data` set (not `fs_path`).
- Removes the runtime dependency on `THOR_DEFAULTS_PATH` / `#directory` and
the need to ship `defaults/` alongside the binary (see `flake.nix`).
- [ ] `build` command alias of default
- [ ] `new site` command to set up new project
- [ ] Some render errors blank the entire page instead of scoping to the failing tag.
- [ ] We must remove all mention of `posts` from the odin code.
At present, "posts" are a user-level construct defined as pages in a
particular collection.
- [ ] ? Clean up the default layouts (no idea what this means)
- [ ] `group_by` currently requires a computed `year` field on the page.
- We should replace this with `{{ pages | group_by (date | "2006") }}` or similar
- [ ] does it make sense for partials to be inside layouts?
- current: `layouts/partials`
- alt1: `layouts`, `partials`
- alt2: `templates/layouts`, `templates/partials`
- [ ] add `sort_by` pipe
- [ ] pipes
- [x] first/last
- [ ] write test for error when n == 0
- [x] warn when using negative numbers
- [x] warn when using default arg on string
- [x] add the above warnings to DIAGNOSTIC_TESTS.md
- [ ] Instead of "clearer form", be exact: "last 5"
- [ ] write tests for `first 0` & `last 0`
- [ ] update tests to assert diagnostics when appropriate
- do we want to do this or just use DIAGNOSTIC_TESTS.md as the tests?
- [ ] also update to use file logger so they don't clog up the output
- [ ] narrow highlight in diagnostic to relevent part of line.
- [ ] page.url and page.permalink are confusing names. Switch to url + page_url
- There is confusion over how to do this and whethor or not `| rel_url` should be needed to render these.
Requires **careful thought**.
## Documentation
- [ ] talk about the context stack (and its limit).
- [ ] highlight the differences in the way menus are handled.
- [ ] explain that html/css/js comments are rendered, except in minify mode.
## Investigate
- [ ] running ./thor/thor still logs the debug message: using config /home/spencer/github.com/sbrow.github.io/thor.json
- wrong cwd?
## Diagnostics
All diagnostics (errors) should show
- *What* went wrong
- *where* (in the file)
- *where* (in the stack trace)
- *how* you can fix it (if applicable)
- [ ] try to make file paths clickable links.
- [ ] make sure the frontmatter parser has good diagnostics.
- [ ] add all known diagnostics to [DIAGNOSTIC TODOS](./DIAGNOSTIC_TESTS.yaml).
- [ ] fix the broken diagnostics in [DIAGNOSTIC TODOS](./DIAGNOSTIC_TESTS.yaml)
- [ ] show *stack trace* for template partials?
- [ ] show where the typo occurred (config file path for `apply_extension_config`, full `-ext:value` for `parse_extension_list`)
- [ ] **all** diagnostics in [DIAGNOSTIC TODOS](./DIAGNOSTIC_TESTS.yaml) must be tested.
- [ ] only report format errors once.
- [ ] only report missing partial errors once.
- [ ] better diagnostics for syntax errors in treesitter.
- [ ] Ensure diagnostics for MAX_CONTEXT_DEPTH are good.
- [ ] improve matching weights message.
- [ ] Test menu diagnostics
- [ ] Extend `tag_error` to all render-time errors, not just pipe errors.
Currently only pipe errors (4 sites in `render_nodes`) get stamped with
the correct template source/path. Other render errors still use the
content template's source/path, which can point at the wrong file.
## Analytics
Debug log stats:
- [ ] final Context_Stack cap
- [ ] highest PIPE args used
- [ ] longest slug length + name that generated it
- [ ] number of pages
- [ ] number of blocks
- [ ] enabled features / extensions
- [ ] etc
## sbrow.github.io
- [ ] The home file was accidentally deleted in commit 9a9f731
- [ ] Ideas is now in the wrong spot. Date is wrong, and it is showing date
when it shouldn't be.
WARNING_COUNTER.md
-109
View File
@@ -1,109 +0,0 @@
# Warning Counter Plan
## Goal
When the same warning fires multiple times across a single page render, show it once with a `[×N]` count prefix instead of N identical diagnostic blocks.
```
[×5] first -3 is equivalent to last 3 — consider using the clearer form
--> layouts/home.html:12:1
|
12 | <ul>{{#items | first -3}}<li>{{.}}</li>{{/items}}</ul>
| ^^^^^^^^^^^^^^^^^^^^^ consider using the clearer form
|
```
## Approach
Module-level buffer in `pipes.odin`. Warnings are collected during render, deduplicated + counted + logged after the page completes. No parameter threading through `render_nodes`.
## Changes
### `mustache/pipes.odin`
**New module-level state:**
```odin
_pending_warnings: [dynamic]Error
```
**New procs:**
```odin
reset_warnings :: proc() {
clear(&_pending_warnings)
}
flush_warnings :: proc(colorize: bool = false) {
if len(_pending_warnings) == 0 do return
counts := make(map[string]int, context.temp_allocator)
first: map[string]Error = make(map[string]Error, context.temp_allocator)
defer delete(counts)
defer delete(first)
for w in _pending_warnings {
b := body(w)
if _, seen := first[b.msg]; !seen {
first[b.msg] = w
}
counts[b.msg] += 1
}
for msg, w in first {
c := counts[msg]
b := body(w)
path := b.path != "" ? b.path : "<input>"
prefix := c > 1 ? fmt.tprintf("[×%d] ", c) : ""
diag := diags.format_error(
path, b.source, b.pos, b.msg,
hint = b.hint,
colorize = colorize,
)
log.warnf("%s%s", prefix, diag)
}
clear(&_pending_warnings)
}
```
**`apply_pipeline` change:**
Replace the current warning handling (which logs immediately):
```odin
if warnings != nil {
append(warnings, ferr)
} else {
log.warnf("%s", format_render_error(ferr, tmpl))
}
```
With:
```odin
append(&_pending_warnings, ferr)
```
Drop the `warnings: ^[dynamic]Error` parameter — no longer needed.
### `render.odin`
In `render_template`, wrap the render call:
```odin
mustache.reset_warnings()
result, err := mustache.render(content_tpl, []any{ctx.site, ctx.page, ctx}, partials)
mustache.flush_warnings(colorize = diags.should_colorize())
```
### `mustache/pipes.odin` cleanup
- Remove `warnings` parameter from `apply_pipeline` signature
- Remove `format_render_error` import/call from `apply_pipeline` (formatting moves to `flush_warnings`)
- Remove the `log` import if no longer used directly in `apply_pipeline`
## Scope
~25 lines new code (`reset_warnings` + `flush_warnings`), ~3 lines changed in `apply_pipeline`, ~2 lines added in `render.odin`. Total ~30 lines.
## Limitations
- Module-level state means warnings don't survive across page renders (by design — `reset_warnings` is called per page)
- Not thread-safe (fine for sequential rendering)
- Counting is by exact `msg` string — warnings with different messages but same root cause are counted separately
assets.odin
-62
View File
@@ -1,62 +0,0 @@
package main
import "core:fmt"
import "core:log"
import "core:os"
import "core:strings"
import ts "treesitter"
copy_assets_dir :: proc(vfs: ^VFS, output_dir: string, features: bit_set[Feature]) {
// One CSS parser, reused for every .css asset (single-threaded). Owned here.
css_parser: ts.Parser
if .Minify in features {
if g, ok := ts.grammar("css"); ok {
css_parser = ts.open_parser(g)
}
}
defer if css_parser != nil {
ts.parser_delete(css_parser)
}
for virtual_path, entry in vfs.files {
if !strings.has_prefix(virtual_path, "assets/") {
continue
}
rel := virtual_path[len("assets/"):]
dest := fmt.tprintf("%s/%s", output_dir, rel)
// Don't clobber a file already written this build (e.g. the output of a
// `tool` pipe, which publishes to the same dest path as its source).
if os.exists(dest) {
continue
}
if idx := strings.last_index(dest, "/"); idx >= 0 {
if err := os.make_directory_all(dest[:idx]); err != nil && err != .Exist {
log.warnf("cannot create %s: %v", dest[:idx], err)
continue
}
}
if .Minify in features && strings.has_suffix(rel, ".css") {
data, ok := vfs_get(vfs, virtual_path)
if ok {
if m, mok := minify_css(css_parser, string(data)); mok {
write_file(dest, m)
} else {
write_file(dest, string(data))
}
}
} else if entry.data != nil {
if err := os.write_entire_file(dest, entry.data); err != nil {
log.warnf("cannot write %s: %v", dest, err)
}
} else {
if err := os.copy_file(dest, entry.fs_path); err != nil {
log.warnf("cannot copy %s: %v", entry.fs_path, err)
}
}
}
}
bench/bench-after
BIN
View File
Binary file not shown.
bench/bench-before
BIN
View File
Binary file not shown.
bench/bench-better-redinent
BIN
View File
Binary file not shown.
bench/bench-perf
BIN
View File
Binary file not shown.
bench/bench.odin
-224
View File
@@ -1,224 +0,0 @@
package bench
import "../mustache"
import "core:fmt"
import "core:mem"
import "core:os"
import "core:strconv"
import "core:time"
Tag :: struct {
name: string,
slug: string,
}
Nav_Item :: struct {
url: string,
label: string,
}
Post :: struct {
title: string,
url: string,
date: string,
year: string,
excerpt: string,
author: string,
tags: [dynamic]Tag,
}
Comment :: struct {
author: string,
date: string,
body: string,
}
Page_Data :: struct {
title: string,
now: string,
posts: [dynamic]Post,
comments: [dynamic]Comment,
nav_items: [dynamic]Nav_Item,
}
TEMPLATE_DIR :: #directory
main :: proc() {
iterations := 250
dump_path := ""
i := 1
for i < len(os.args) {
if os.args[i] == "--dump" && i + 1 < len(os.args) {
dump_path = os.args[i + 1]
i += 2
} else {
n, ok := strconv.parse_int(os.args[i])
if ok && n > 0 {
iterations = n
}
i += 1
}
}
data_arena: mem.Dynamic_Arena
mem.dynamic_arena_init(&data_arena)
defer mem.dynamic_arena_destroy(&data_arena)
temp_arena: mem.Dynamic_Arena
mem.dynamic_arena_init(&temp_arena)
defer mem.dynamic_arena_destroy(&temp_arena)
context.allocator = mem.dynamic_arena_allocator(&data_arena)
context.temp_allocator = mem.dynamic_arena_allocator(&temp_arena)
base := parse_file("base.html")
page := parse_file("page.html")
defer mustache.delete_template(&base)
defer mustache.delete_template(&page)
partials := make(map[string]mustache.Template)
partials["base"] = base
partials["post"] = parse_file("partials/post.html")
partials["comment"] = parse_file("partials/comment.html")
defer mustache.delete_partials(partials)
data := generate_data()
if dump_path != "" {
result, err := mustache.render(page, data, partials, allocator = context.temp_allocator)
if err != nil {
b := mustache.body(err)
fmt.eprintln("render error:", b.msg)
os.exit(1)
}
werr := os.write_entire_file_from_string(dump_path, result)
if werr != nil {
fmt.eprintln("failed to write", dump_path, ":", werr)
os.exit(1)
}
fmt.println("wrote", len(result), "bytes to", dump_path)
return
}
for _ in 0 ..< 3 {
_, _ = mustache.render(page, data, partials, allocator = context.temp_allocator)
mem.dynamic_arena_free_all(&temp_arena)
}
start := time.now()
for _ in 0 ..< iterations {
_, _ = mustache.render(page, data, partials, allocator = context.temp_allocator)
mem.dynamic_arena_free_all(&temp_arena)
}
elapsed := time.since(start)
seconds := time.duration_seconds(elapsed)
per_render_ms := seconds * 1000 / f64(iterations)
fmt.printfln(
"iterations=%d total=%.3fs per_render=%.3fms",
iterations,
seconds,
per_render_ms,
)
}
parse_file :: proc(name: string) -> mustache.Template {
path := fmt.aprintf("%s/templates/%s", TEMPLATE_DIR, name)
data, err := os.read_entire_file_from_path(path, context.allocator)
if err != nil {
fmt.eprintln("failed to read", path, ":", err)
os.exit(1)
}
source := string(data)
tmpl, perr := mustache.parse(source, path)
if perr != nil {
b := mustache.body(perr)
fmt.eprintln("parse error in", path, ":", b.msg)
os.exit(1)
}
return tmpl
}
generate_data :: proc() -> Page_Data {
years := []string {
"2025",
"2024",
"2023",
"2022",
"2021",
"2020",
"2019",
"2018",
"2017",
"2016",
}
posts := make([dynamic]Post, 0, 500)
for year in years {
for i in 0 ..< 50 {
tags := make([dynamic]Tag, 0, 3)
append(
&tags,
Tag{name = fmt.aprintf("%s-notes", year), slug = fmt.aprintf("%s-notes", year)},
)
append(&tags, Tag{name = "writing", slug = "writing"})
append(&tags, Tag{name = "archive", slug = "archive"})
month := (i % 12) + 1
day := (i % 28) + 1
author := ""
if i % 3 != 0 {
author = fmt.aprintf("Author %d", i % 5)
}
append(
&posts,
Post {
title = fmt.aprintf("Post %d from %s", i, year),
url = fmt.aprintf("/%s/post-%d", year, i),
date = fmt.aprintf("%s-%02d-%02dT10:00:00Z", year, month, day),
year = year,
excerpt = "Lorem ipsum dolor sit amet, consectetur adipiscing elit.",
author = author,
tags = tags,
},
)
}
}
comments := make([dynamic]Comment, 0, 100)
for i in 0 ..< 100 {
year := years[i % len(years)]
month := (i % 12) + 1
day := (i % 28) + 1
append(
&comments,
Comment {
author = fmt.aprintf("Commenter %d", i),
date = fmt.aprintf("%s-%02d-%02dT12:00:00Z", year, month, day),
body = fmt.aprintf("Great post! This is comment number %d.", i),
},
)
}
nav_items := make([dynamic]Nav_Item, 0, 8)
append(&nav_items, Nav_Item{url = "/", label = "Home"})
append(&nav_items, Nav_Item{url = "/archive", label = "Archive"})
append(&nav_items, Nav_Item{url = "/about", label = "About"})
append(&nav_items, Nav_Item{url = "/tags", label = "Tags"})
append(&nav_items, Nav_Item{url = "/feed.xml", label = "RSS"})
append(&nav_items, Nav_Item{url = "https://github.com/example", label = "GitHub"})
append(&nav_items, Nav_Item{url = "https://twitter.com/example", label = "Twitter"})
append(&nav_items, Nav_Item{url = "mailto:nobody@example.com", label = "Email"})
return Page_Data {
title = "Post Archive",
now = "2025-07-21T12:00:00Z",
posts = posts,
comments = comments,
nav_items = nav_items,
}
}
bench/date/check.odin
-129
View File
@@ -1,129 +0,0 @@
package main
import "base:runtime"
import "common"
import "core:fmt"
import "core:log"
import "core:os"
import "core:strings"
import "core:time"
import "inlined"
import "original"
DATES :: #load(#directory + os.Path_Separator_String + "dates.txt")
FORMATS :: #load(#directory + os.Path_Separator_String + "formats.txt")
ITERATIONS :: 1_000
dates: []common.Date_Components
formats: []string
formatter :: #type proc(
date: common.Date_Components,
format: string,
allocator: runtime.Allocator,
) -> string
benchmark :: #type proc(
options: ^time.Benchmark_Options,
allocator: runtime.Allocator,
) -> (
err: time.Benchmark_Error,
)
Version :: struct {
name: string,
bench: benchmark,
}
init :: proc() {
raw_dates := strings.split_lines(string(DATES))
raw_dates = raw_dates[:len(raw_dates) - 1]
formats = strings.split_lines(string(FORMATS))
dates = make([]common.Date_Components, len(raw_dates))
assert(to_parsed(raw_dates, &dates))
}
to_parsed :: proc(raw_dates: []string, dates: ^[]common.Date_Components) -> bool {
for date, i in raw_dates {
// log.debugf("parsing '%s'", date)
dates[i] = common.parse_iso_date(date) or_return
}
return true
}
main :: proc() {
logger_opts: log.Options =
(log.Default_Console_Logger_Opts - log.Full_Timestamp_Opts - {.Short_File_Path})
console_logger := log.create_console_logger(.Info, logger_opts)
context.logger = console_logger
defer log.destroy_console_logger(console_logger)
init()
versions := [?]Version {
{"original", to_benchmark(original.format_date)},
{"inlined", to_benchmark(inlined.format_date)},
}
fmt.printfln(
"%-10s %14s %10s %14s %12s %8s",
"version",
"total",
"calls",
"calls/s",
"time/call",
"MB/s",
)
for version in versions {
defer free_all(context.temp_allocator)
b: time.Benchmark_Options
b.bench = version.bench
if err := time.benchmark(&b, context.temp_allocator); err != nil {
fmt.panicf("%v", err)
}
// fmt.printfln("%v", b)
per_call := time.Duration(i64(b.duration) / i64(b.count))
fmt.printfln(
"%-10s %14v % 10d % 14.0f %12v % 8.2f",
version.name,
b.duration,
b.count,
b.rounds_per_second,
per_call,
b.megabytes_per_second,
)
}
}
to_benchmark :: proc($f: formatter) -> benchmark {
return(
proc(
opts: ^time.Benchmark_Options,
allocator: runtime.Allocator,
) -> (
err: time.Benchmark_Error,
) {
for _ in 0 ..< ITERATIONS {
for date in dates {
for format in formats {
f(date, format, allocator)
opts.count += 1
opts.processed += size_of(date) + size_of(format)
}
}
opts.rounds += 1
}
return err
} \
)
}
bench/date/common/types.odin
-43
View File
@@ -1,43 +0,0 @@
package common
Date_Components :: struct {
year: int,
month: int,
day: int,
hour: int,
minute: int,
second: int,
}
// TODO: Use some kind of scanner interface
parse_iso_date :: proc(iso: string) -> (c: Date_Components, ok: bool) {
if len(iso) < 10 {
return {}, false
}
c.year = parse_2_digits(iso, 0) * 100 + parse_2_digits(iso, 2)
c.month = parse_2_digits(iso, 5)
c.day = parse_2_digits(iso, 8)
if c.month < 1 || c.month > 12 {
return {}, false
}
if c.day < 1 || c.day > 31 {
return {}, false
}
if len(iso) >= 19 && (iso[10] == 'T' || iso[10] == 't') {
c.hour = parse_2_digits(iso, 11)
c.minute = parse_2_digits(iso, 14)
c.second = parse_2_digits(iso, 17)
}
return c, true
}
parse_2_digits :: proc(s: string, offset: int) -> int {
if offset + 1 >= len(s) {
return 0
}
return (int(s[offset]) - 0x30) * 10 + (int(s[offset + 1]) - 0x30)
}
bench/date/dates.txt
-59
View File
@@ -1,59 +0,0 @@
2024-01-05
2024-02-14
2024-03-08
2024-04-22
2024-05-01
2024-06-19
2024-07-04
2024-08-30
2024-09-11
2024-10-31
2024-11-27
2024-12-25
2023-01-15T00:00:00-05:00
2023-02-14T09:30:15+01:00
2023-03-08T14:45:22-08:00
2023-04-22T23:59:59+05:30
2023-05-01T06:05:09-04:00
2023-06-19T18:20:33+09:00
2023-07-04T03:04:05-07:00
2023-08-30T11:11:11+00:00
2023-09-11T15:04:05-04:00
2023-10-31T20:15:30+02:00
2023-11-27T07:07:07-06:00
2023-12-25T12:30:45+03:00
2025-01-15T00:15:00-0500
2025-02-14T09:30:15+0100
2025-03-08T14:45:22-0800
2025-04-22T23:59:59+0530
2025-05-01T06:05:09-0400
2025-06-19T18:20:33+0900
2025-07-04T03:04:05-0700
2025-08-30T11:11:11+0000
2025-09-11T15:04:05-0400
2025-10-31T20:15:30+0200
2025-11-27T07:07:07-0600
2025-12-25T12:30:45+0300
2022-01-01T00:00:00Z
2022-02-14T01:30:00Z
2022-03-08T11:59:59Z
2022-04-22T12:00:00Z
2022-05-01T12:00:01Z
2022-06-19T13:15:00Z
2022-07-04T15:04:05Z
2022-08-30T18:45:30Z
2022-09-11T20:20:20Z
2022-10-31T22:10:10Z
2022-11-27T23:59:59Z
2022-12-25T09:09:09Z
2021-03-14t09:26:53
2021-07-20t23:00:00
2021-11-05t00:00:00
2020-06-15T15:04:05
2020-12-31T23:59:59
2020-01-01T00:00:00
2024-02-29T12:00:00Z
2000-02-29T00:00:00Z
1999-12-31T23:59:59Z
2026-01-01T00:00:00Z
2026-07-23T14:56:07-04:00
bench/date/formats.txt
-46
View File
@@ -1,46 +0,0 @@
2006
06
January
Jan
Monday
Mon
01
02
03
04
05
15
1
2
3
4
5
PM
pm
MST
2006-01-02
2006-01-02 15:04:05
2 Jan 2006
Jan 2, 2006
January 2, 2006
Monday, January 2, 2006
Mon Jan 2 2006
Mon Jan 02 2006
Mon, 02 Jan 2006 15:04:05 MST
01/02/2006
02/01/2006
2006/01/02
1/2/06
1/2/06 3:04PM
1/2/2006
3:04:05 PM
3:04 pm
15:04:05
15:04
15:04:05 MST
January 2006
Jan 06
2 January 2006
Monday 2 Jan 2006 at 15:04
06-01-02
3:4:5
bench/date/inlined/format.odin
-125
View File
@@ -1,125 +0,0 @@
package inlined
import "../common"
import "core:fmt"
import "core:log"
import "core:strings"
import "core:time"
import "core:time/datetime"
format_date :: proc(
dt: common.Date_Components,
fmt: string,
allocator := context.temp_allocator,
) -> string {
b: strings.Builder
strings.builder_init_len(&b, len(fmt), allocator)
for i := 0; i < len(fmt); {
matched := match_token(&b, dt, fmt[i:])
if matched > 0 {
i += matched
} else {
strings.write_byte(&b, fmt[i])
i += 1
}
}
log.debugf("formatted date: '%s'", b.buf)
return strings.to_string(b)
}
match_token :: proc(b: ^strings.Builder, dt: common.Date_Components, s: string) -> int {
if strings.has_prefix(
s,
"January",
) {strings.write_string(b, fmt.tprintf("%s", time.Month(dt.month))); return 7}
if strings.has_prefix(s, "Monday") {emit_weekday(b, dt, full = true); return 6}
if strings.has_prefix(
s,
"2006",
) {strings.write_string(b, fmt.tprintf("%04d", dt.year)); return 4}
if strings.has_prefix(s, "MST") {strings.write_string(b, "UTC"); return 3}
if strings.has_prefix(s, "Jan") {emit_month_abbr(b, dt); return 3}
if strings.has_prefix(s, "Mon") {emit_weekday(b, dt, full = false); return 3}
if strings.has_prefix(
s,
"06",
) {strings.write_string(b, fmt.tprintf("%02d", dt.year % 100)); return 2}
if strings.has_prefix(s, "02") {strings.write_string(b, fmt.tprintf("%02d", dt.day)); return 2}
if strings.has_prefix(
s,
"15",
) {strings.write_string(b, fmt.tprintf("%02d", dt.hour)); return 2}
if strings.has_prefix(
s,
"04",
) {strings.write_string(b, fmt.tprintf("%02d", dt.minute)); return 2}
if strings.has_prefix(
s,
"05",
) {strings.write_string(b, fmt.tprintf("%02d", dt.second)); return 2}
if strings.has_prefix(
s,
"01",
) {strings.write_string(b, fmt.tprintf("%02d", dt.month)); return 2}
if strings.has_prefix(s, "03") {emit_hour_12(b, dt, pad = true); return 2}
if strings.has_prefix(s, "PM") {
strings.write_string(b, "PM" if dt.hour >= 12 else "AM")
return 2
}
if strings.has_prefix(s, "pm") {
strings.write_string(b, "pm" if dt.hour >= 12 else "am")
return 2
}
if len(s) >= 1 {
switch s[0] {
case '2':
strings.write_string(b, fmt.tprintf("%d", dt.day)); return 1
case '1':
strings.write_string(b, fmt.tprintf("%d", dt.month)); return 1
case '4':
strings.write_string(b, fmt.tprintf("%d", dt.minute)); return 1
case '5':
strings.write_string(b, fmt.tprintf("%d", dt.second)); return 1
case '3':
emit_hour_12(b, dt, pad = false); return 1
case:
return 0
}
}
return 0
}
emit_month_abbr :: proc(b: ^strings.Builder, dt: common.Date_Components) {
name := fmt.tprintf("%s", time.Month(dt.month))
strings.write_string(b, name[:3 if len(name) >= 3 else len(name)])
}
emit_weekday :: proc(b: ^strings.Builder, dt: common.Date_Components, full: bool) {
date := datetime.Date {
year = i64(dt.year),
month = i8(dt.month),
day = i8(dt.day),
}
ordinal, err := datetime.date_to_ordinal(date)
if err != .None {
strings.write_string(b, "???")
return
}
weekday := datetime.day_of_week(ordinal)
name := fmt.tprintf("%s", weekday)
if full {
strings.write_string(b, name)
} else {
strings.write_string(b, name[:3 if len(name) >= 3 else len(name)])
}
}
emit_hour_12 :: proc(b: ^strings.Builder, dt: common.Date_Components, pad: bool) {
h12 := dt.hour % 12
if h12 == 0 {h12 = 12}
format := "%02d" if pad else "%d"
fmt.sbprintf(b, format, h12)
}
bench/date/inplace/format.odin
-132
View File
@@ -1,132 +0,0 @@
package inplace
import "../common"
import "core:bytes"
import "core:fmt"
import "core:mem"
import "core:strings"
import "core:time"
import "core:time/datetime"
format_date :: proc(
dt: common.Date_Components,
fmt: string,
allocator := context.temp_allocator,
) -> string {
b := make([]byte, len(fmt), context.temp_allocator)
x := transmute([]byte)(fmt)
mem.copy_non_overlapping(&b[0], &x, len(fmt))
return string(b)
// for i := 0; i < len(fmt); {
// matched := match_token(&b, dt, fmt[i:])
// if matched > 0 {
// i += matched
// } else {
// strings.write_byte(&b, fmt[i])
// i += 1
// }
// }
// log.debugf("formatted date: '%s'", b.buf)
// return strings.to_string(b)
}
FULL_MONTH: string : "January"
match_token :: proc(s: []byte, dt: common.Date_Components) {
if bytes.equal(s[:len(FULL_MONTH)], transmute([]u8)FULL_MONTH) {
mo := fmt.tprintf("%s", time.Month(dt.month))
mem.copy_non_overlapping(&s[0], &(transmute([]u8)mo)[0], len(mo))
}
/*
if strings.has_prefix(s, "Monday") {emit_weekday(b, dt, full = true); return 6}
if strings.has_prefix(
s,
"2006",
) {strings.write_string(b, fmt.tprintf("%04d", dt.year)); return 4}
if strings.has_prefix(s, "MST") {strings.write_string(b, "UTC"); return 3}
if strings.has_prefix(s, "Jan") {emit_month_abbr(b, dt); return 3}
if strings.has_prefix(s, "Mon") {emit_weekday(b, dt, full = false); return 3}
if strings.has_prefix(
s,
"06",
) {strings.write_string(b, fmt.tprintf("%02d", dt.year % 100)); return 2}
if strings.has_prefix(s, "02") {strings.write_string(b, fmt.tprintf("%02d", dt.day)); return 2}
if strings.has_prefix(
s,
"15",
) {strings.write_string(b, fmt.tprintf("%02d", dt.hour)); return 2}
if strings.has_prefix(
s,
"04",
) {strings.write_string(b, fmt.tprintf("%02d", dt.minute)); return 2}
if strings.has_prefix(
s,
"05",
) {strings.write_string(b, fmt.tprintf("%02d", dt.second)); return 2}
if strings.has_prefix(
s,
"01",
) {strings.write_string(b, fmt.tprintf("%02d", dt.month)); return 2}
if strings.has_prefix(s, "03") {emit_hour_12(b, dt, pad = true); return 2}
if strings.has_prefix(s, "PM") {
strings.write_string(b, "PM" if dt.hour >= 12 else "AM")
return 2
}
if strings.has_prefix(s, "pm") {
strings.write_string(b, "pm" if dt.hour >= 12 else "am")
return 2
}
if len(s) >= 1 {
switch s[0] {
case '2':
strings.write_string(b, fmt.tprintf("%d", dt.day)); return 1
case '1':
strings.write_string(b, fmt.tprintf("%d", dt.month)); return 1
case '4':
strings.write_string(b, fmt.tprintf("%d", dt.minute)); return 1
case '5':
strings.write_string(b, fmt.tprintf("%d", dt.second)); return 1
case '3':
emit_hour_12(b, dt, pad = false); return 1
case:
return 0
}
}
*/
}
emit_month_abbr :: proc(b: ^strings.Builder, dt: common.Date_Components) {
name := fmt.tprintf("%s", time.Month(dt.month))
strings.write_string(b, name[:3 if len(name) >= 3 else len(name)])
}
emit_weekday :: proc(b: ^strings.Builder, dt: common.Date_Components, full: bool) {
date := datetime.Date {
year = i64(dt.year),
month = i8(dt.month),
day = i8(dt.day),
}
ordinal, err := datetime.date_to_ordinal(date)
if err != .None {
strings.write_string(b, "???")
return
}
weekday := datetime.day_of_week(ordinal)
name := fmt.tprintf("%s", weekday)
if full {
strings.write_string(b, name)
} else {
strings.write_string(b, name[:3 if len(name) >= 3 else len(name)])
}
}
emit_hour_12 :: proc(b: ^strings.Builder, dt: common.Date_Components, pad: bool) {
h12 := dt.hour % 12
if h12 == 0 {h12 = 12}
format := "%02d" if pad else "%d"
fmt.sbprintf(b, format, h12)
}
bench/date/original/format.odin
-127
View File
@@ -1,127 +0,0 @@
package original
import "../common"
import "core:fmt"
import "core:log"
import "core:strings"
import "core:time"
import "core:time/datetime"
format_date :: proc(
dt: common.Date_Components,
fmt: string,
allocator := context.temp_allocator,
) -> string {
b: strings.Builder
strings.builder_init(&b, allocator)
for i := 0; i < len(fmt); {
matched := match_token(&b, dt, fmt[i:])
if matched > 0 {
i += matched
} else {
strings.write_byte(&b, fmt[i])
i += 1
}
}
log.debugf("formatted date: '%s'", b.buf)
return strings.to_string(b)
}
match_token :: proc(b: ^strings.Builder, dt: common.Date_Components, s: string) -> int {
if strings.has_prefix(
s,
"January",
) {strings.write_string(b, fmt.tprintf("%s", time.Month(dt.month))); return 7}
if strings.has_prefix(s, "Monday") {emit_weekday(b, dt, full = true); return 6}
if strings.has_prefix(
s,
"2006",
) {strings.write_string(b, fmt.tprintf("%04d", dt.year)); return 4}
if strings.has_prefix(s, "MST") {strings.write_string(b, "UTC"); return 3}
if strings.has_prefix(s, "Jan") {emit_month_abbr(b, dt); return 3}
if strings.has_prefix(s, "Mon") {emit_weekday(b, dt, full = false); return 3}
if strings.has_prefix(
s,
"06",
) {strings.write_string(b, fmt.tprintf("%02d", dt.year % 100)); return 2}
if strings.has_prefix(s, "02") {strings.write_string(b, fmt.tprintf("%02d", dt.day)); return 2}
if strings.has_prefix(
s,
"15",
) {strings.write_string(b, fmt.tprintf("%02d", dt.hour)); return 2}
if strings.has_prefix(
s,
"04",
) {strings.write_string(b, fmt.tprintf("%02d", dt.minute)); return 2}
if strings.has_prefix(
s,
"05",
) {strings.write_string(b, fmt.tprintf("%02d", dt.second)); return 2}
if strings.has_prefix(
s,
"01",
) {strings.write_string(b, fmt.tprintf("%02d", dt.month)); return 2}
if strings.has_prefix(s, "03") {emit_hour_12(b, dt, pad = true); return 2}
if strings.has_prefix(s, "PM") {emit_am_pm(b, dt); return 2}
if strings.has_prefix(s, "pm") {emit_am_pm_lower(b, dt); return 2}
if len(s) >= 1 {
switch s[0] {
case '2':
strings.write_string(b, fmt.tprintf("%d", dt.day)); return 1
case '1':
strings.write_string(b, fmt.tprintf("%d", dt.month)); return 1
case '4':
strings.write_string(b, fmt.tprintf("%d", dt.minute)); return 1
case '5':
strings.write_string(b, fmt.tprintf("%d", dt.second)); return 1
case '3':
emit_hour_12(b, dt, pad = false); return 1
case:
return 0
}
}
return 0
}
emit_month_abbr :: proc(b: ^strings.Builder, dt: common.Date_Components) {
name := fmt.tprintf("%s", time.Month(dt.month))
strings.write_string(b, name[:3 if len(name) >= 3 else len(name)])
}
emit_weekday :: proc(b: ^strings.Builder, dt: common.Date_Components, full: bool) {
date := datetime.Date {
year = i64(dt.year),
month = i8(dt.month),
day = i8(dt.day),
}
ordinal, err := datetime.date_to_ordinal(date)
if err != .None {
strings.write_string(b, "???")
return
}
weekday := datetime.day_of_week(ordinal)
name := fmt.tprintf("%s", weekday)
if full {
strings.write_string(b, name)
} else {
strings.write_string(b, name[:3 if len(name) >= 3 else len(name)])
}
}
emit_hour_12 :: proc(b: ^strings.Builder, dt: common.Date_Components, pad: bool) {
h12 := dt.hour % 12
if h12 == 0 {h12 = 12}
format := "%02d" if pad else "%d"
fmt.sbprintf(b, format, h12)
}
emit_am_pm :: proc(b: ^strings.Builder, dt: common.Date_Components) {
strings.write_string(b, "PM" if dt.hour >= 12 else "AM")
}
emit_am_pm_lower :: proc(b: ^strings.Builder, dt: common.Date_Components) {
strings.write_string(b, "pm" if dt.hour >= 12 else "am")
}
bench/date/parse.odin
-43
View File
@@ -1,43 +0,0 @@
package main
Date_Components :: struct {
year: int,
month: int,
day: int,
hour: int,
minute: int,
second: int,
}
// TODO: Use some kind of scanner interface
parse_iso_date :: proc(iso: string) -> (c: Date_Components, ok: bool) {
if len(iso) < 10 {
return {}, false
}
c.year = parse_2_digits(iso, 0) * 100 + parse_2_digits(iso, 2)
c.month = parse_2_digits(iso, 5)
c.day = parse_2_digits(iso, 8)
if c.month < 1 || c.month > 12 {
return {}, false
}
if c.day < 1 || c.day > 31 {
return {}, false
}
if len(iso) >= 19 && (iso[10] == 'T' || iso[10] == 't') {
c.hour = parse_2_digits(iso, 11)
c.minute = parse_2_digits(iso, 14)
c.second = parse_2_digits(iso, 17)
}
return c, true
}
parse_2_digits :: proc(s: string, offset: int) -> int {
if offset + 1 >= len(s) {
return 0
}
return (int(s[offset]) - 0x30) * 10 + (int(s[offset + 1]) - 0x30)
}
bench/date/returned/format.odin
-126
View File
@@ -1,126 +0,0 @@
package returned
import "../common"
import "core:fmt"
import "core:log"
import "core:strings"
import "core:time"
import "core:time/datetime"
format_date :: proc(
dt: common.Date_Components,
fmt: string,
allocator := context.temp_allocator,
) -> string {
b: strings.Builder
// strings.builder_init_len(&b, len(fmt), allocator)
strings.builder_init(&b, allocator)
for i := 0; i < len(fmt); {
matched := match_token(&b, dt, fmt[i:])
if matched > 0 {
i += matched
} else {
strings.write_byte(&b, fmt[i])
i += 1
}
}
log.debugf("formatted date: '%s'", b.buf)
return strings.to_string(b)
}
match_token :: proc(b: ^strings.Builder, dt: common.Date_Components, s: string) -> int {
if strings.has_prefix(
s,
"January",
) {strings.write_string(b, fmt.tprintf("%s", time.Month(dt.month))); return 7}
if strings.has_prefix(s, "Monday") {emit_weekday(b, dt, full = true); return 6}
if strings.has_prefix(
s,
"2006",
) {strings.write_string(b, fmt.tprintf("%04d", dt.year)); return 4}
if strings.has_prefix(s, "MST") {strings.write_string(b, "UTC"); return 3}
if strings.has_prefix(s, "Jan") {emit_month_abbr(b, dt); return 3}
if strings.has_prefix(s, "Mon") {emit_weekday(b, dt, full = false); return 3}
if strings.has_prefix(
s,
"06",
) {strings.write_string(b, fmt.tprintf("%02d", dt.year % 100)); return 2}
if strings.has_prefix(s, "02") {strings.write_string(b, fmt.tprintf("%02d", dt.day)); return 2}
if strings.has_prefix(
s,
"15",
) {strings.write_string(b, fmt.tprintf("%02d", dt.hour)); return 2}
if strings.has_prefix(
s,
"04",
) {strings.write_string(b, fmt.tprintf("%02d", dt.minute)); return 2}
if strings.has_prefix(
s,
"05",
) {strings.write_string(b, fmt.tprintf("%02d", dt.second)); return 2}
if strings.has_prefix(
s,
"01",
) {strings.write_string(b, fmt.tprintf("%02d", dt.month)); return 2}
if strings.has_prefix(s, "03") {emit_hour_12(b, dt, pad = true); return 2}
if strings.has_prefix(s, "PM") {
strings.write_string(b, "PM" if dt.hour >= 12 else "AM")
return 2
}
if strings.has_prefix(s, "pm") {
strings.write_string(b, "pm" if dt.hour >= 12 else "am")
return 2
}
if len(s) >= 1 {
switch s[0] {
case '2':
strings.write_string(b, fmt.tprintf("%d", dt.day)); return 1
case '1':
strings.write_string(b, fmt.tprintf("%d", dt.month)); return 1
case '4':
strings.write_string(b, fmt.tprintf("%d", dt.minute)); return 1
case '5':
strings.write_string(b, fmt.tprintf("%d", dt.second)); return 1
case '3':
emit_hour_12(b, dt, pad = false); return 1
case:
return 0
}
}
return 0
}
emit_month_abbr :: proc(b: ^strings.Builder, dt: common.Date_Components) {
name := fmt.tprintf("%s", time.Month(dt.month))
strings.write_string(b, name[:3 if len(name) >= 3 else len(name)])
}
emit_weekday :: proc(b: ^strings.Builder, dt: common.Date_Components, full: bool) {
date := datetime.Date {
year = i64(dt.year),
month = i8(dt.month),
day = i8(dt.day),
}
ordinal, err := datetime.date_to_ordinal(date)
if err != .None {
strings.write_string(b, "???")
return
}
weekday := datetime.day_of_week(ordinal)
name := fmt.tprintf("%s", weekday)
if full {
strings.write_string(b, name)
} else {
strings.write_string(b, name[:3 if len(name) >= 3 else len(name)])
}
}
emit_hour_12 :: proc(b: ^strings.Builder, dt: common.Date_Components, pad: bool) {
h12 := dt.hour % 12
if h12 == 0 {h12 = 12}
format := "%02d" if pad else "%d"
fmt.sbprintf(b, format, h12)
}
bench/templates/base.html
-13
View File
@@ -1,13 +0,0 @@
<!DOCTYPE html>
<html>
<head>
<title>{{$title}}{{/title}}</title>
{{$head}}{{/head}}
</head>
<body>
{{$nav}}{{/nav}}
{{$main}}{{/main}}
{{$sidebar}}{{/sidebar}}
{{$footer}}{{/footer}}
</body>
</html>
bench/templates/page.html
-45
View File
@@ -1,45 +0,0 @@
{{<base}}
{{$title}}Archive{{/title}}
{{$head}}
<meta name="description" content="Post archive">
{{/head}}
{{$nav}}
<nav>
<ul>
{{#nav_items}}
<li><a href="{{url}}">{{label}}</a></li>
{{/nav_items}}
</ul>
</nav>
{{/nav}}
{{$main}}
<main>
<h1>Archive</h1>
{{#posts | group_by year}}
<section class="year">
<h2>{{key}}</h2>
<ul>
{{#items}}
{{> post}}
{{/items}}
</ul>
</section>
{{/posts}}
</main>
{{/main}}
{{$sidebar}}
<aside>
<h3>Recent Comments</h3>
<ul>
{{#comments}}
{{> comment}}
{{/comments}}
</ul>
</aside>
{{/sidebar}}
{{$footer}}
<footer>
<p>Generated {{now}}</p>
</footer>
{{/footer}}
{{/base}}
bench/templates/partials/comment.html
-5
View File
@@ -1,5 +0,0 @@
<li class="comment">
<strong>{{author}}</strong>
<time datetime="{{date}}">{{date | format}}</time>
<p>{{body}}</p>
</li>
bench/templates/partials/post.html
-9
View File
@@ -1,9 +0,0 @@
<li class="post">
<article>
<h3><a href="{{url}}">{{title}}</a></h3>
<time datetime="{{date}}">{{date | format}}</time>
<p class="excerpt">{{excerpt}}</p>
{{#author}}<address>{{author}}</address>{{/author}}
{{#tags}}<a href="/tag/{{slug}}" class="tag">{{name}}</a>{{/tags}}
</article>
</li>
content.odin
+145 -274
View File
@@ -1,91 +1,111 @@
package main package main
import md "markdown" import cm "vendor:commonmark"
import ts "treesitter"
import "core:encoding/json"
import "core:fmt" import "core:fmt"
import "core:log"
import "core:os" import "core:os"
import "core:strings" import "core:strings"
import "core:time"
// Fields with underscores should never be set by the user. Page_Type :: enum {
Home,
Post,
Standalone,
}
Page :: struct { Page :: struct {
section: string, type: Page_Type,
slug: string, slug: string,
layout: string,
permalink: string, permalink: string,
url: string,
title: string, title: string,
description: string, description: string,
date: string, date: string,
year: string,
weight: Maybe(int),
lastmod: string,
menus: map[string]Menu_Entry,
params: json.Value,
content: string,
og: Open_Graph,
draft: bool, draft: bool,
toc: string, is_starred: bool,
_is_index: bool `private`, menu: string,
body: string,
body_html: string,
bundle_dir: string,
} }
Pending_File :: struct { // walk_content reads the content directory and returns all non-draft pages
path: string, // (or all pages if include_drafts is true).
section: string, walk_content :: proc(content_path: string, include_drafts: bool) -> []Page {
slug: string, pages: [dynamic]Page
is_index: bool,
}
// site_load_content reads the content directory and populates site.pages. collect_home(&pages, content_path)
// Drafts are excluded unless .Drafts is enabled. collect_standalone(&pages, content_path)
site_load_content :: proc(site: ^Site) {
site.pages = make(#soa[dynamic]Page, 0, 8, site_allocator(site))
// Phase 0: Enumerate content files posts_path := fmt.tprintf("%s/posts", content_path)
pending := make([dynamic]Pending_File, 0, 16, context.temp_allocator) if os.exists(posts_path) {
scan_content_files(site.content_dir, "", &pending) collect_posts(&pages, posts_path)
// Phase 1: Pre-scan for code fence languages
// Phase 2: Parallel grammar preload
if .Highlight in site.markdown_extensions {
languages := collect_languages(pending[:])
ts.preload_grammars(languages)
} }
// Phase 3: Load pages (grammars already cached) if !include_drafts {
for file in pending { filtered: [dynamic]Page
page, ok := load_page( for &page in pages {
file.path, if !page.draft {
file.section, append(&filtered, page)
file.slug, }
file.is_index, }
site.markdown_extensions, delete(pages)
) return filtered[:]
if ok && (!page.draft || .Drafts in site.features) { }
append(&site.pages, page)
return pages[:]
}
collect_home :: proc(pages: ^[dynamic]Page, content_path: string) {
html_path := fmt.tprintf("%s/index.html", content_path)
if os.exists(html_path) {
page, ok := load_page(html_path, .Home, "")
if ok {
page.permalink = "/"
append(pages, page)
}
return
}
md_path := fmt.tprintf("%s/index.md", content_path)
if os.exists(md_path) {
page, ok := load_page(md_path, .Home, "")
if ok {
page.permalink = "/"
append(pages, page)
} }
} }
for &page in site.pages {
page.url = fmt.tprintf("%s%s", site.base_url, page.permalink)
if site.base_path != "" {
page.content = rewrite_root_urls(page.content, site.base_path, site_allocator(site))
}
}
build_menus(site)
} }
// scan_content_files walks the content directory and collects Pending_File collect_standalone :: proc(pages: ^[dynamic]Page, content_path: string) {
// entries. At the root level (section=""), directories are treated as entries, err := os.read_all_directory_by_path(content_path, context.allocator)
// sections. Within a section, directories are treated as leaf bundles.
scan_content_files :: proc(dir: string, section: string, pending: ^[dynamic]Pending_File) {
entries, err := os.read_all_directory_by_path(dir, context.allocator)
if err != nil { if err != nil {
log.warnf("cannot read %s: %v", dir, err) fmt.eprintfln("thor: cannot read %s: %v", content_path, err)
return
}
defer os.file_info_slice_delete(entries, context.allocator)
for entry in entries {
if !is_content_file(entry.name) {
continue
}
if entry.name == "index.md" || entry.name == "index.html" {
continue
}
if entry.type != .Regular {
continue
}
slug := strip_extension(entry.name)
page, ok := load_page(entry.fullpath, .Standalone, slug)
if ok {
append(pages, page)
}
}
}
collect_posts :: proc(pages: ^[dynamic]Page, posts_path: string) {
entries, err := os.read_all_directory_by_path(posts_path, context.allocator)
if err != nil {
fmt.eprintfln("thor: cannot read %s: %v", posts_path, err)
return return
} }
defer os.file_info_slice_delete(entries, context.allocator) defer os.file_info_slice_delete(entries, context.allocator)
@@ -96,253 +116,81 @@ scan_content_files :: proc(dir: string, section: string, pending: ^[dynamic]Pend
if !is_content_file(entry.name) { if !is_content_file(entry.name) {
continue continue
} }
slug := strip_extension(entry.name)
filename := strip_extension(entry.name) page, ok := load_page(entry.fullpath, .Post, slug)
is_idx := filename == "index" if ok {
slug := is_idx ? "" : filename append(pages, page)
}
append(
pending,
Pending_File {
path = strings.clone(entry.fullpath, context.temp_allocator),
section = section,
slug = slug,
is_index = is_idx,
},
)
case .Directory: case .Directory:
if section == "" { index_path := fmt.tprintf("%s/index.html", entry.fullpath)
scan_content_files(entry.fullpath, entry.name, pending) if !os.exists(index_path) {
} else { index_path = fmt.tprintf("%s/index.md", entry.fullpath)
index_path := fmt.tprintf("%s/index.html", entry.fullpath) }
if !os.exists(index_path) { if !os.exists(index_path) {
index_path = fmt.tprintf("%s/index.md", entry.fullpath) continue
} }
if os.exists(index_path) { page, ok := load_page(index_path, .Post, entry.name)
append( if ok {
pending, page.bundle_dir = entry.fullpath
Pending_File { append(pages, page)
path = strings.clone(index_path, context.temp_allocator),
section = section,
slug = entry.name,
is_index = false,
},
)
}
} }
case .Undetermined, .Symlink, .Named_Pipe, .Socket, .Block_Device, .Character_Device: case .Undetermined, .Symlink, .Named_Pipe, .Socket, .Block_Device, .Character_Device:
} }
} }
} }
// collect_languages scans .md files for code fence language identifiers
// (```lang or ~~~lang) and returns the unique set.
collect_languages :: proc(files: []Pending_File) -> []string {
set := make(map[string]bool, context.temp_allocator)
for file in files {
if !strings.has_suffix(file.path, ".md") {
continue
}
data, err := os.read_entire_file_from_path(file.path, context.temp_allocator)
if err != nil {
continue
}
content := string(data)
pos := 0
for pos < len(content) {
newline := strings.index_byte(content[pos:], '\n')
line_end := pos + newline if newline >= 0 else len(content)
line := content[pos:line_end]
i := 0
for i < len(line) && (line[i] == ' ' || line[i] == '\t') {
i += 1
}
if i + 3 <= len(line) &&
(line[i] == '`' && line[i + 1] == '`' && line[i + 2] == '`') ||
(i + 3 <= len(line) && line[i] == '~' && line[i + 1] == '~' && line[i + 2] == '~') {
fence_char := line[i]
j := i + 3
for j < len(line) && line[j] == fence_char {
j += 1
}
for j < len(line) && (line[j] == ' ' || line[j] == '\t') {
j += 1
}
lang_start := j
for j < len(line) {
c := line[j]
if c == ' ' || c == '\t' || c == '\r' || c == '\n' {
break
}
j += 1
}
if j > lang_start {
set[line[lang_start:j]] = true
}
}
pos = line_end + 1
}
}
result := make([dynamic]string, 0, len(set), context.temp_allocator)
for lang in set {
append(&result, lang)
}
return result[:]
}
infer_layout :: proc(section: string, is_index: bool) -> string {
if section == "" && is_index {
return "home"
}
if is_index {
return fmt.tprintf("%s_index", section)
}
if section != "" {
if len(section) > 1 && section[len(section) - 1] == 's' {
return section[:len(section) - 1]
}
return section
}
return "page"
}
load_page :: proc( load_page :: proc(
file_path: string, file_path: string,
section: string, page_type: Page_Type,
slug: string, slug: string,
is_index: bool,
ext: bit_set[md.Extension],
) -> ( ) -> (
page: Page, page: Page,
ok: bool, ok: bool,
) { ) {
data, err := os.read_entire_file_from_path(file_path, context.allocator) data, err := os.read_entire_file_from_path(file_path, context.allocator)
if err != nil { if err != nil {
log.warnf("cannot read %s: %v", file_path, err) fmt.eprintfln("thor: cannot read %s: %v", file_path, err)
return return
} }
content := string(data) content := string(data)
fm, body, parsed := parse_frontmatter(content) fm, body, parsed := parse_frontmatter(content)
if !parsed { if !parsed {
body = strings.trim_left(content, " \t\r\n") fmt.eprintfln("thor: no frontmatter in %s", file_path)
return
} }
page.section = section page.type = page_type
page.slug = slug page.slug = slug
page._is_index = is_index page.title = fm.title
page.title = fm.title
page.description = fm.description page.description = fm.description
page.date = fm.date page.date = fm.date
if page.date == "" { page.draft = fm.draft
info, stat_err := os.stat(file_path, context.allocator) page.is_starred = fm.isStarred
if stat_err == nil { page.menu = fm.menu
page.date, _ = time.time_to_rfc3339( page.body = strings.clone(body)
info.modification_time,
0,
false,
context.allocator,
)
os.file_info_delete(info, context.allocator)
log.warnf(
"no date in frontmatter for %s, using file modification time: %s",
file_path,
page.date,
)
}
}
page.year = get_year(page.date)
page.weight = fm.weight
page.lastmod = fm.lastmod
page.draft = fm.draft
page.params = fm.params
if section == "" && is_index {
page.permalink = "/"
} else if is_index {
page.permalink = fmt.aprintf("/%s/", section)
} else if section == "" {
page.permalink = fmt.aprintf("/%s/", slug)
} else {
page.permalink = fmt.aprintf("/%s/%s/", section, slug)
}
page.menus = parse_page_menus(fm.menus, page, context.allocator)
page.layout = fm.layout if fm.layout != "" else infer_layout(section, is_index)
page.og = fm.og
if strings.has_suffix(file_path, ".html") { if strings.has_suffix(file_path, ".html") {
page.content = strings.clone(body, context.allocator) page.body_html = strings.clone(body)
} else { } else {
page.content = md.process(body, ext, file_path, context.allocator) clean_body, defs := strip_definitions(body)
html := cm.markdown_to_html_from_string(clean_body, {.Unsafe})
page.body_html = inject_sidenotes(html, defs)
} }
if fm.toc { switch page_type {
page.toc = md.generate_toc(page.content, context.allocator) case .Home:
page.permalink = "/"
case .Post:
page.permalink = fmt.aprintf("/posts/%s/", slug)
case .Standalone:
page.permalink = fmt.aprintf("/%s/", slug)
} }
ok = true ok = true
return return
} }
// rewrite_root_urls prefixes root-relative href/src attribute values with
// base_path so links authored in content (e.g. [x](/ideas)) resolve under a
// subpath deployment. Only values beginning with a single '/' are rewritten;
// absolute ("http://…"), protocol-relative ("//cdn…"), fragment ("#x"),
// mailto:, and document-relative ("foo/bar", "./foo") values all fail that
// test and pass through untouched. Assumes cmark's double-quoted output.
rewrite_root_urls :: proc(html, base_path: string, allocator := context.allocator) -> string {
b := strings.builder_make(allocator)
strings.builder_grow(&b, len(html))
i := 0
for i < len(html) {
n := attr_opener_at(html, i) // ` href="` or ` src="`
if n == 0 {
strings.write_byte(&b, html[i])
i += 1
continue
}
strings.write_string(&b, html[i:i + n]) // copy the opener verbatim
i += n
end := strings.index_byte(html[i:], '"')
if end < 0 { // malformed; bail cleanly
strings.write_string(&b, html[i:])
break
}
value := html[i:i + end]
if url_is_root_relative(value) {
strings.write_string(&b, base_path)
}
strings.write_string(&b, value) // closing quote copied next loop
i += end
}
return strings.to_string(b)
}
// url_is_root_relative reports whether value is a site-root path ("/faq/") as
// opposed to protocol-relative ("//cdn") or anything not starting with '/'.
url_is_root_relative :: proc(value: string) -> bool {
return len(value) >= 1 && value[0] == '/' && !strings.has_prefix(value, "//")
}
// attr_opener_at returns the length of a ` href="` / ` src="` attribute opener
// at s[i:], or 0. The leading space anchors the match to a real HTML attribute
// so we don't rewrite the strings inside link text or code blocks.
attr_opener_at :: proc(s: string, i: int) -> int {
if strings.has_prefix(s[i:], " href=\"") do return 7
if strings.has_prefix(s[i:], " src=\"") do return 6
return 0
}
is_content_file :: proc(name: string) -> bool { is_content_file :: proc(name: string) -> bool {
return strings.has_suffix(name, ".md") || strings.has_suffix(name, ".html") return strings.has_suffix(name, ".md") || strings.has_suffix(name, ".html")
} }
@@ -354,3 +202,26 @@ strip_extension :: proc(name: string) -> string {
} }
return name[:dot] return name[:dot]
} }
// copy_static_assets copies non-content files from the content root to the output directory.
copy_static_assets :: proc(content_path: string, output_dir: string) {
entries, err := os.read_all_directory_by_path(content_path, context.allocator)
if err != nil {
return
}
defer os.file_info_slice_delete(entries, context.allocator)
for entry in entries {
if entry.type != .Regular {
continue
}
if is_content_file(entry.name) {
continue
}
dest := fmt.tprintf("%s/%s", output_dir, entry.name)
if err := os.copy_file(dest, entry.fullpath); err != nil {
fmt.eprintfln("thor: cannot copy %s: %v", entry.name, err)
}
}
}
defaults.odin
-18
View File
@@ -1,18 +0,0 @@
package main
import "core:os"
// Path to the built-in default layouts, resolved at runtime.
//
// The default points at the `defaults` directory next to the source tree (via
// `#directory`), which works for local `odin run`/`odin build` from the repo.
// Packaged builds install `defaults/` to a fixed location and override this
// with `-define:THOR_DEFAULTS_PATH=<path>` (see flake.nix).
//
// TODO: bake these into the binary at compile time via `#load_directory` so the
// binary is self-contained and this runtime path goes away entirely. See
// TODOS.md.
DEFAULTS_PATH :: #config(
THOR_DEFAULTS_PATH,
#directory + os.Path_Separator_String + "defaults",
)
defaults/layouts/base.html
-18
View File
@@ -1,18 +0,0 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>{{> title}}</title>
{{> opengraph}}
{{> styles}}
{{> scripts}}
</head>
<body>
{{> nav}}{{$main}}{{/main}}
{{> footer}}
</body>
</html>
defaults/layouts/home.html
-15
View File
@@ -1,15 +0,0 @@
{{<base}}
{{$main}}
<main>
<header>
{{&content}}
</header>
<ul>
{{#pages}} <li><a href="{{permalink | rel_url}}">{{&title}}</a><span>{{#date}}<time
datetime="{{date}}">{{date | format}}</time>{{/date}}</span>
</li>
{{/pages}}
</ul>
</main>
{{/main}}
{{/base}}
defaults/layouts/page.html
-16
View File
@@ -1,16 +0,0 @@
{{<base}}
{{$main}}
<main>
<article>
<h1>{{page.title}}</h1>
{{#date}}<time class="subtitle" datetime="{{date}}">{{ date | format}}</time>{{/date}}
{{#page.toc}}
<nav class="toc">
{{&page.toc}}
</nav>
{{/page.toc}}
{{&content}}
</article>
</main>
{{/main}}
{{/base}}
defaults/layouts/partials/footer.html
-3
View File
@@ -1,3 +0,0 @@
<footer>
<p>&copy; {{now | format "2006"}} {{params.author.name}}</p>
</footer>
defaults/layouts/partials/home-link.html
-1
View File
@@ -1 +0,0 @@
{{site.title}}{{^site.title}}Home{{/site.title}}
defaults/layouts/partials/nav.html
-10
View File
@@ -1,10 +0,0 @@
<header>
<nav>
<ul>
<li><a href="{{site.base_path}}/">{{> home-link}}</a></li>
{{#menus.main}}
<li><a href="{{url | rel_url}}">{{name}}</a></li>
{{/menus.main}}
</ul>
</nav>
</header>
defaults/layouts/partials/opengraph.html
-9
View File
@@ -1,9 +0,0 @@
<meta property="og:url" content="{{og.url}}">
<meta property="og:site_name" content="{{og.site_name}}">
<meta property="og:title" content="{{og.title}}">
<meta property="og:description" content="{{og.description}}">
<meta property="og:locale" content="{{og.locale}}">
<meta property="og:type" content="{{og.type}}">
{{#og.is_article}}<meta property="article:section" content="{{og.section}}">
<meta property="article:published_time" content="{{og.published_time}}">
{{/og.is_article}}<meta property="og:image" content="{{og.image}}">
defaults/layouts/partials/scripts.html
-1
View File
@@ -1 +0,0 @@
{{params.scripts}}
defaults/layouts/partials/styles.html
-294
View File
@@ -1,294 +0,0 @@
{{^params.stylesheets}}
<style>
/* Nav */
header nav ul {
list-style: none;
display: flex;
gap: 1rem;
padding: 0;
}
/* Base layout */
body {
margin-left: auto;
margin-right: auto;
}
main {
max-width: 760px;
margin: 0 auto;
}
/* Sidenote layout — activates only when sidenote elements are present */
body:has(.sidenote, .marginnote) {
counter-reset: sidenote-counter;
padding-left: 12.5%;
width: 87.5%;
}
body:has(.sidenote, .marginnote) main {
max-width: none;
width: 60%;
margin: 0;
}
/* Images */
img {
max-width: 100%;
height: auto;
}
/* Superscript (footnote refs) */
sup {
line-height: 0;
}
/* Code blocks */
code,
pre>code {
font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace;
font-size: 1.0rem;
line-height: 1.42;
-webkit-text-size-adjust: 100%;
}
pre>code {
font-size: 0.9rem;
overflow-x: auto;
display: block;
}
/* Alerts (GitHub-style) */
.alert {
border-left: 4px solid;
border-radius: 0 4px 4px 0;
padding: 0.5rem 1rem;
margin-inline-start: 0;
}
.alert-title {
font-weight: bold;
margin-bottom: 0.25rem;
margin-top: 0.25rem;
}
.alert-note {
border-left-color: #3b82f6;
}
.alert-tip {
border-left-color: #22c55e;
}
.alert-important {
border-left-color: #a855f7;
}
.alert-warning {
border-left-color: #eab308;
}
.alert-caution {
border-left-color: #ef4444;
}
/* Definition lists */
dl,
ol,
ul {
font-size: 1.4rem;
line-height: 2rem;
}
dt:not(:first-child) {
margin-top: 0.25rem;
}
dd {
margin-left: 0;
}
/* Sidenotes, margin notes */
.sidenote,
.marginnote {
float: right;
clear: right;
margin-right: -45%;
width: 40%;
margin-top: 0.3rem;
margin-bottom: 0;
font-size: 0.85em;
line-height: 1.5;
vertical-align: baseline;
position: relative;
}
.sidenote-number {
counter-increment: sidenote-counter;
}
.sidenote-number:after,
.sidenote:before {
position: relative;
vertical-align: baseline;
color: var(--color-accent, currentColor);
}
.sidenote-number:after {
content: counter(sidenote-counter);
font-size: 0.8rem;
top: -0.5rem;
left: 0.1rem;
}
.sidenote:before {
content: counter(sidenote-counter) " ";
font-size: 0.8rem;
top: -0.5rem;
}
blockquote .sidenote,
blockquote .marginnote {
margin-right: -82%;
min-width: 59%;
text-align: left;
}
.marginnote>code,
.sidenote>code {
font-size: 1rem;
}
input.margin-toggle {
display: none;
}
label.sidenote-number {
display: inline-block;
max-height: 2rem;
}
label.margin-toggle:not(.sidenote-number) {
display: none;
}
/* Responsive */
@media (max-width: 1200px) {
body {
padding-left: 1rem;
padding-right: 1rem;
}
body:has(.sidenote, .marginnote) {
padding-left: 8%;
padding-right: 8%;
width: 84%;
}
body:has(.sidenote, .marginnote) main {
width: 100%;
max-width: 760px;
margin: 0 auto;
}
pre>code {
width: 97%;
}
img {
width: 100%;
}
label.margin-toggle:not(.sidenote-number) {
display: inline;
}
label.margin-toggle:not(.sidenote-number)::after {
content: "\2295";
}
.sidenote,
.marginnote {
display: none;
}
.margin-toggle:checked+.sidenote,
.margin-toggle:checked+.marginnote {
display: block;
float: left;
left: 1rem;
clear: both;
width: 95%;
margin: 1rem 2.5%;
vertical-align: baseline;
position: relative;
}
label {
cursor: pointer;
}
}
</style>
{{/params.stylesheets}}
<style>
/* Syntax highlighting. Build-time <span class="hl-*"> wrappers only appear
inside fenced code blocks. The base class (e.g. hl-keyword) is always
emitted alongside any dotted variant (hl-keyword-control), so styling the
base names covers every capture. Re-theme by overriding the --hl-* vars. */
:root {
--hl-bg: #282c34;
--hl-fg: #abb2bf;
--hl-comment: #5c6370;
--hl-keyword: #c678dd;
--hl-string: #98c379;
--hl-escape: #56b6c2;
--hl-number: #d19a66;
--hl-function: #61afef;
--hl-type: #e5c07b;
--hl-variable: #e06c75;
--hl-attribute: #d19a66;
}
pre:has(> code [class^="hl-"]) {
background: var(--hl-bg);
color: var(--hl-fg);
padding: 0.75rem 1rem;
border-radius: 4px;
}
.hl-comment { color: var(--hl-comment); font-style: italic; }
.hl-keyword,
.hl-conditional,
.hl-repeat,
.hl-exception,
.hl-include,
.hl-storageclass { color: var(--hl-keyword); }
.hl-operator,
.hl-punctuation { color: var(--hl-fg); }
.hl-string,
.hl-character { color: var(--hl-string); }
.hl-escape,
.hl-string-escape,
.hl-string-special { color: var(--hl-escape); }
.hl-number,
.hl-float,
.hl-boolean,
.hl-constant { color: var(--hl-number); }
.hl-function,
.hl-method,
.hl-constructor { color: var(--hl-function); }
.hl-type,
.hl-namespace,
.hl-module { color: var(--hl-type); }
.hl-variable,
.hl-property,
.hl-field,
.hl-parameter,
.hl-tag,
.hl-symbol { color: var(--hl-variable); }
.hl-attribute,
.hl-label,
.hl-preproc { color: var(--hl-attribute); }
</style>
{{#params.stylesheets}}<link rel="stylesheet" href="{{.|rel_url}}">
{{/params.stylesheets}}
defaults/layouts/partials/title.html
-1
View File
@@ -1 +0,0 @@
{{#site.title}}{{.}}{{#page.title}} | {{/page.title}}{{/site.title}}{{page.title}}
defaults/layouts/section_index.html
-19
View File
@@ -1,19 +0,0 @@
{{<base}}
{{$main}}
<main>
<h1>{{page.title}}</h1>
{{&content}}
{{#posts | group_by year}}
<section>
<h2>{{key}}</h2>
<ul>
{{#items}} <li><a href="{{permalink | rel_url}}">{{&title}}</a><span>{{#date}}<time
datetime="{{date}}">{{date | format}}</time>{{/date}}</span>
</li>
{{/items}}
</ul>
</section>
{{/posts}}
</main>
{{/main}}
{{/base}}
diagnostics/diagnostics.odin
-279
View File
@@ -1,279 +0,0 @@
package diagnostics
import "core:fmt"
import "core:os"
import "core:strings"
import "core:terminal/ansi"
import "core:unicode/utf8"
// suggest_correction returns the closest match from `available` to `missing`
// using Levenshtein distance, or "" if no good match exists. The threshold
// scales with the length of the missing key.
suggest_correction :: proc(available: []string, missing: string) -> string {
if len(available) == 0 || len(missing) == 0 {
return ""
}
threshold := max(2, len(missing) / 3)
best: string
best_dist := threshold + 1
for candidate in available {
if abs(len(candidate) - len(missing)) > threshold {
continue
}
d := strings.levenshtein_distance(missing, candidate)
if d <= threshold && d < best_dist {
best = candidate
best_dist = d
}
}
return best
}
// line_col returns the 1-indexed line and column for a byte offset in source.
line_col :: proc(source: string, pos_in: int) -> (line: int, col: int) {
pos := pos_in
if pos < 0 {
return 1, 1
}
if pos > len(source) {
pos = len(source)
}
line = 1
col = 1
for i := 0; i < pos; i += 1 {
if source[i] == '\n' {
line += 1
col = 1
} else {
col += 1
}
}
return
}
// line_text returns the Nth (1-indexed) line of source, without the trailing
// newline. Returns "" if line is out of range.
line_text :: proc(source: string, line: int) -> string {
if line < 1 {
return ""
}
current := 1
start := 0
for i := 0; i < len(source); i += 1 {
if current == line {
end := i
for end < len(source) && source[end] != '\n' {
end += 1
}
return source[start:end]
}
if source[i] == '\n' {
current += 1
start = i + 1
}
}
if current == line {
return source[start:]
}
return ""
}
// should_colorize returns true if stderr is a TTY and color output is wanted.
should_colorize :: proc() -> bool {
return os.is_tty(os.stderr)
}
// count_lines returns the number of '\n'-separated lines in source.
count_lines :: proc(source: string) -> int {
if len(source) == 0 {
return 1
}
n := 1
for c in source {
if c == '\n' {
n += 1
}
}
if len(source) > 0 && source[len(source) - 1] == '\n' {
n -= 1
}
return n
}
// digit_count returns the number of decimal digits in n (min 1).
digit_count :: proc(n: int) -> int {
if n <= 0 {
return 1
}
c := 0
x := n
for x > 0 {
c += 1
x /= 10
}
return c
}
// display_width returns the number of terminal cells `s` occupies.
display_width :: proc(s: string) -> int {
return utf8.rune_count_in_string(s)
}
// line_start_at returns the byte offset of the start of the line containing pos.
line_start_at :: proc(source: string, pos_in: int) -> int {
pos := pos_in
if pos < 0 {
return 0
}
if pos >= len(source) {
pos = len(source) - 1
}
for pos > 0 && source[pos - 1] != '\n' {
pos -= 1
}
return pos
}
// format_error produces a rust-style multi-line diagnostic string.
//
// <msg>
// --> <path>:<line>:<col>
// |
// N | <source line N-2>
// N | <source line N-1>
// N | <source line N — the error line>
// | ^^^^^^^^^^^ <hint>
// N | <source line N+1>
// N | <source line N+2>
// |
//
// `caret_start`/`caret_end` control the byte range of the caret underline.
// If not provided (or negative), defaults to a single-rune underline at `pos`.
format_error :: proc(
path: string,
source: string,
pos: int,
msg: string,
hint: string = "",
context_before: int = 2,
context_after: int = 2,
colorize: bool = false,
caret_start: int = -1,
caret_end: int = -1,
) -> string {
line, col := line_col(source, pos)
total_lines := count_lines(source)
start_line := line - context_before
if start_line < 1 {
start_line = 1
}
end_line := line + context_after
if end_line > total_lines {
end_line = total_lines
}
width := digit_count(end_line)
if width < 1 {
width = 1
}
cs := caret_start >= 0 ? caret_start : pos
ce := caret_end > cs ? caret_end : cs + 1
ls := line_start_at(source, pos)
caret_start_col := cs - ls + 1
caret_end_col := ce - ls + 1
sb := strings.builder_make(context.temp_allocator)
defer strings.builder_destroy(&sb)
red, faint, reset := "", "", ""
if colorize {
red = ansi.CSI + ansi.FG_RED + ansi.SGR
faint = ansi.CSI + ansi.FAINT + ansi.SGR
reset = ansi.CSI + ansi.RESET + ansi.SGR
}
// Header
strings.write_string(&sb, msg)
strings.write_byte(&sb, '\n')
// Location line
strings.write_string(&sb, faint)
for _ in 0 ..< width {
strings.write_byte(&sb, ' ')
}
strings.write_string(&sb, "--> ")
strings.write_string(&sb, reset)
fmt.sbprintf(&sb, "%s:%d:%d\n", path, line, col)
// Top gutter
write_gutter(&sb, width, faint, reset)
// Context lines
for n in start_line ..= end_line {
num_str := fmt.tprintf("%d", n)
strings.write_string(&sb, faint)
for _ in 0 ..< width - len(num_str) {
strings.write_byte(&sb, ' ')
}
strings.write_string(&sb, num_str)
strings.write_string(&sb, " | ")
strings.write_string(&sb, reset)
strings.write_string(&sb, line_text(source, n))
strings.write_byte(&sb, '\n')
// Caret row after error line
if n == line {
strings.write_string(&sb, faint)
for _ in 0 ..< width + 1 {
strings.write_byte(&sb, ' ')
}
strings.write_string(&sb, "| ")
strings.write_string(&sb, reset)
for _ in 1 ..< caret_start_col {
strings.write_byte(&sb, ' ')
}
if colorize {
strings.write_string(&sb, red)
}
for _ in 0 ..< caret_end_col - caret_start_col {
strings.write_byte(&sb, '^')
}
if colorize {
strings.write_string(&sb, reset)
}
if hint != "" {
strings.write_byte(&sb, ' ')
if colorize {
strings.write_string(&sb, faint)
}
strings.write_string(&sb, hint)
if colorize {
strings.write_string(&sb, reset)
}
}
strings.write_byte(&sb, '\n')
}
}
// Trailing gutter
write_gutter(&sb, width, faint, reset)
return strings.to_string(sb)
}
// write_gutter emits a faint pipe-only gutter line: `<width+1 spaces> |`.
write_gutter :: proc(sb: ^strings.Builder, width: int, faint: string, reset: string) {
strings.write_string(sb, faint)
for _ in 0 ..< width + 1 {
strings.write_byte(sb, ' ')
}
strings.write_string(sb, "|")
strings.write_string(sb, reset)
strings.write_byte(sb, '\n')
}
diagnostics/diagnostics_test.odin
-111
View File
@@ -1,111 +0,0 @@
#+test
package diagnostics
import "core:fmt"
import "core:testing"
@(test)
test_suggest_correction_exact :: proc(t: ^testing.T) {
available := []string{"title", "page.title", "body"}
testing.expect_value(t, suggest_correction(available, "page_titel"), "page.title")
}
@(test)
test_suggest_correction_close :: proc(t: ^testing.T) {
available := []string{"title", "body", "now"}
testing.expect_value(t, suggest_correction(available, "titel"), "title")
}
@(test)
test_suggest_correction_no_match :: proc(t: ^testing.T) {
available := []string{"completely_different", "unrelated"}
testing.expect_value(t, suggest_correction(available, "page_titel"), "")
}
@(test)
test_suggest_correction_empty :: proc(t: ^testing.T) {
testing.expect_value(t, suggest_correction([]string{}, "anything"), "")
testing.expect_value(t, suggest_correction([]string{"a"}, ""), "")
}
// ---------------------------------------------------------------------------
// line_col / line_text / count_lines / digit_count — primitive helpers
// ---------------------------------------------------------------------------
@(test)
test_line_col_basic :: proc(t: ^testing.T) {
src := "abc\ndef\nghi"
cases := [?]struct {
pos: int,
line: int,
col: int,
}{{0, 1, 1}, {2, 1, 3}, {3, 1, 4}, {4, 2, 1}, {6, 2, 3}, {7, 2, 4}, {8, 3, 1}}
for c in cases {
l, col := line_col(src, c.pos)
testing.expect(t, l == c.line, fmt.tprintf("pos %d: line %d, want %d", c.pos, l, c.line))
testing.expect(t, col == c.col, fmt.tprintf("pos %d: col %d, want %d", c.pos, col, c.col))
}
}
@(test)
test_line_col_empty :: proc(t: ^testing.T) {
l, col := line_col("", 0)
testing.expect_value(t, l, 1)
testing.expect_value(t, col, 1)
}
@(test)
test_line_col_negative :: proc(t: ^testing.T) {
l, col := line_col("abc", -1)
testing.expect_value(t, l, 1)
testing.expect_value(t, col, 1)
}
@(test)
test_line_col_past_end :: proc(t: ^testing.T) {
l, col := line_col("abc", 100)
testing.expect_value(t, l, 1)
testing.expect_value(t, col, 4)
}
@(test)
test_line_text_first :: proc(t: ^testing.T) {
src := "first\nsecond\nthird"
testing.expect_value(t, line_text(src, 1), "first")
testing.expect_value(t, line_text(src, 2), "second")
testing.expect_value(t, line_text(src, 3), "third")
}
@(test)
test_line_text_trailing_newline :: proc(t: ^testing.T) {
src := "first\nsecond\n"
testing.expect_value(t, line_text(src, 1), "first")
testing.expect_value(t, line_text(src, 2), "second")
testing.expect_value(t, line_text(src, 3), "")
}
@(test)
test_line_text_out_of_range :: proc(t: ^testing.T) {
testing.expect_value(t, line_text("abc", 5), "")
testing.expect_value(t, line_text("abc", 0), "")
}
@(test)
test_count_lines :: proc(t: ^testing.T) {
testing.expect_value(t, count_lines(""), 1)
testing.expect_value(t, count_lines("abc"), 1)
testing.expect_value(t, count_lines("a\nb"), 2)
testing.expect_value(t, count_lines("a\nb\n"), 2)
testing.expect_value(t, count_lines("a\nb\nc"), 3)
}
@(test)
test_digit_count :: proc(t: ^testing.T) {
testing.expect_value(t, digit_count(0), 1)
testing.expect_value(t, digit_count(1), 1)
testing.expect_value(t, digit_count(9), 1)
testing.expect_value(t, digit_count(10), 2)
testing.expect_value(t, digit_count(99), 2)
testing.expect_value(t, digit_count(100), 3)
testing.expect_value(t, digit_count(-5), 1)
}
feed.odin
+86 -126
View File
@@ -1,15 +1,20 @@
package main package main
import "base:runtime"
import "core:fmt" import "core:fmt"
import "core:strings" import "core:strings"
import "core:time"
generate_rss :: proc(site: ^Site) -> string { WEEKDAYS: [7]string = {"Sun", "Mon", "Tue", "Wed", "Thu", "Fri", "Sat"}
sb := strings.builder_make()
fmt.sbprintf( generate_rss :: proc(
&sb, pages: []Page,
site_title: string,
site_desc: string,
base_url: string,
) -> string {
parts: [dynamic]string
defer delete(parts)
append(&parts, fmt.aprintf(
`<?xml version="1.0" encoding="utf-8" standalone="yes"?> `<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"> <rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
<channel> <channel>
@@ -18,14 +23,14 @@ generate_rss :: proc(site: ^Site) -> string {
<description>%s</description> <description>%s</description>
<language>en-us</language> <language>en-us</language>
<atom:link href="%s/index.xml" rel="self" type="application/rss+xml"/>`, <atom:link href="%s/index.xml" rel="self" type="application/rss+xml"/>`,
xml_escape(site.title), xml_escape(site_title),
site.base_url, base_url,
xml_escape(site.description), xml_escape(site_desc),
site.base_url, base_url,
) ))
for page in site.pages { for page in pages {
if page.section == "" && page._is_index { if page.type == .Home {
continue continue
} }
@@ -34,156 +39,111 @@ generate_rss :: proc(site: ^Site) -> string {
pub_date = format_rfc822(page.date) pub_date = format_rfc822(page.date)
} }
fmt.sbprintf( append(&parts, fmt.aprintf(
&sb,
`<item> `<item>
<title>%s</title> <title>%s</title>
<link>%s</link> <link>%s%s</link>
<pubDate>%s</pubDate> <pubDate>%s</pubDate>
<guid>%s</guid> <guid>%s%s</guid>
<description>%s</description> <description>%s</description>
</item> </item>
`, `,
xml_escape(page.title), xml_escape(page.title),
page.url, base_url,
page.permalink,
pub_date, pub_date,
page.url, base_url,
xml_escape(page.content), page.permalink,
) xml_escape(page.body_html),
))
} }
strings.write_string(&sb, "</channel>\n</rss>") append(&parts, "</channel>\n</rss>")
return strings.to_string(sb)
return strings.join(parts[:], "")
} }
generate_sitemap :: proc(site: ^Site) -> string { generate_sitemap :: proc(pages: []Page, base_url: string) -> string {
sb := strings.builder_make() parts: [dynamic]string
defer delete(parts)
strings.write_string( append(
&sb, &parts,
`<?xml version="1.0" encoding="utf-8" standalone="yes"?> `<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml"> <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml">
`, `,
) )
for page in site.pages { for page in pages {
fmt.sbprintf(&sb, "<url><loc>%s</loc>", page.url) lastmod := ""
if page.date != "" { if page.date != "" {
fmt.sbprintf(&sb, "<lastmod>%s</lastmod>", page.date) lastmod = fmt.aprintf("<lastmod>%s</lastmod>", page.date)
} }
fmt.sbprintf(&sb, "</url>\n") append(&parts, fmt.aprintf(
"<url><loc>%s%s</loc>%s</url>\n",
base_url,
page.permalink,
lastmod,
))
} }
// Section index pages (for sections without an index in content) // Posts list page
sections := make(map[string]bool) posts_lastmod := ""
defer delete(sections) for page in pages {
for page in site.pages { if page.type == .Post && page.date > posts_lastmod {
if page.section != "" && !page._is_index { posts_lastmod = page.date
sections[page.section] = true
} }
} }
for section in sections { posts_lm := ""
has_index := false if posts_lastmod != "" {
for page in site.pages { posts_lm = fmt.aprintf("<lastmod>%s</lastmod>", posts_lastmod)
if page.section == section && page._is_index {
has_index = true
break
}
}
if has_index {
continue
}
section_lastmod := ""
for page in site.pages {
if page.section == section && !page._is_index && page.date > section_lastmod {
section_lastmod = page.date
}
}
fmt.sbprintf(&sb, "<url><loc>%s/%s/</loc>", site.base_url, section)
if section_lastmod != "" {
fmt.sbprintf(&sb, "<lastmod>%s</lastmod>", section_lastmod)
}
fmt.sbprintf(&sb, "</url>\n")
} }
append(&parts, fmt.aprintf("<url><loc>%s/posts/</loc>%s</url>\n", base_url, posts_lm))
strings.write_string(&sb, "</urlset>") append(&parts, "</urlset>")
return strings.to_string(sb)
return strings.join(parts[:], "")
} }
format_rfc822 :: proc(iso: string, allocator := context.temp_allocator) -> string { format_rfc822 :: proc(iso: string) -> string {
if len(iso) < 19 { if len(iso) < 19 {
// TODO: should indicate error somehow
return iso return iso
} }
date, offset, _ := time.iso8601_to_time_and_offset(iso) year := (int(iso[0]) - 0x30) * 1000 + (int(iso[1]) - 0x30) * 100 +
(int(iso[2]) - 0x30) * 10 + (int(iso[3]) - 0x30)
month := (int(iso[5]) - 0x30) * 10 + (int(iso[6]) - 0x30)
day := (int(iso[8]) - 0x30) * 10 + (int(iso[9]) - 0x30)
time := iso[11:19]
weekday := fmt.tprintf("%s", time.weekday(date)) // Timezone: -04:00 → -0400
month := fmt.tprintf("%s", time.month(date)) tz := "+0000"
buf: [8]byte if len(iso) >= 25 && (iso[19] == '+' || iso[19] == '-') {
t := time.to_string_hms(date, buf[:]) tz = fmt.tprintf("%c%s%s", iso[19], iso[20:22], iso[23:25])
// RFC-822 zone: sign followed by two-digit hours and minutes (e.g. +0000,
// +0530, -0800). offset is in minutes.
sign := '+'
off := offset
if off < 0 {
sign = '-'
off = -off
} }
// Sakamoto's method for day of week (0=Sunday)
t := [12]int{0, 3, 2, 5, 0, 3, 5, 1, 4, 6, 2, 4}
y := year
if month < 3 {
y -= 1
}
dow := (y + y / 4 - y / 100 + y / 400 + t[month - 1] + day) % 7
return fmt.aprintf( return fmt.aprintf(
"%s, %02d %s %d %s %c%02d%02d", "%s, %d %s %d %s %s",
weekday[:3], WEEKDAYS[dow],
time.day(date), day,
month[:3], MONTHS[month - 1],
time.year(date), year,
t, time,
sign, tz,
off / 60,
off % 60,
allocator = allocator,
) )
} }
// xml_escape escapes the XML metacharacters '&', '<' and '>' in s. Only these xml_escape :: proc(s: string) -> string {
// ASCII characters are touched; all other bytes (including UTF-8 multibyte r, _ := strings.replace_all(s, "&", "&amp;")
// sequences) pass through verbatim, so s must already be valid UTF-8. r, _ = strings.replace_all(r, "<", "&lt;")
// r, _ = strings.replace_all(r, ">", "&gt;")
// The result is allocated in `allocator`, which defaults to return r
// context.temp_allocator — the returned string is only valid until the next
// temp allocator reset, so callers must not retain it across frames.
xml_escape :: proc(
s: string,
allocator := context.temp_allocator,
) -> (
escaped: string,
err: runtime.Allocator_Error,
) #optional_allocator_error {
sb := strings.builder_make_len_cap(0, len(s), allocator) or_return
// Single pass over the bytes, copying runs of ordinary text in bulk and
// emitting an entity for each metacharacter. The switch is both the
// membership test and the replacement lookup, so no per-match scan.
start := 0
for i := 0; i < len(s); i += 1 {
esc: string
switch s[i] {
case '&':
esc = "&amp;"
case '<':
esc = "&lt;"
case '>':
esc = "&gt;"
case:
continue // ordinary byte: copied as part of the next run
}
strings.write_string(&sb, s[start:i]) // run before the metacharacter
strings.write_string(&sb, esc)
start = i + 1
}
strings.write_string(&sb, s[start:]) // tail
return strings.to_string(sb), nil
} }
feed_test.odin
-166
View File
@@ -1,166 +0,0 @@
#+test
package main
import "core:strings"
import "core:testing"
// ---------------------------------------------------------------------------
// xml_escape
// ---------------------------------------------------------------------------
@(test)
test_xml_escape_passthrough :: proc(t: ^testing.T) {
testing.expect_value(t, xml_escape(""), "")
testing.expect_value(t, xml_escape("hello world"), "hello world")
// UTF-8 multibyte bytes are not metacharacters and pass through untouched.
testing.expect_value(t, xml_escape("héllo — 世界"), "héllo — 世界")
}
@(test)
test_xml_escape_entities :: proc(t: ^testing.T) {
testing.expect_value(t, xml_escape("&"), "&amp;")
testing.expect_value(t, xml_escape("<"), "&lt;")
testing.expect_value(t, xml_escape(">"), "&gt;")
// Adjacent specials, and specials at the very start/end of the string.
testing.expect_value(t, xml_escape("&<>"), "&amp;&lt;&gt;")
testing.expect_value(t, xml_escape("<a>"), "&lt;a&gt;")
}
@(test)
test_xml_escape_mixed :: proc(t: ^testing.T) {
testing.expect_value(t, xml_escape("a<b>c&d"), "a&lt;b&gt;c&amp;d")
testing.expect_value(
t,
xml_escape("<p>Tom & Jerry</p>"),
"&lt;p&gt;Tom &amp; Jerry&lt;/p&gt;",
)
}
// ---------------------------------------------------------------------------
// format_rfc822
// ---------------------------------------------------------------------------
@(test)
test_format_rfc822_short_passthrough :: proc(t: ^testing.T) {
// Inputs shorter than a full ISO-8601 timestamp are returned unchanged
// (and are the input slice itself, so they must not be deleted).
testing.expect_value(t, format_rfc822("2024"), "2024")
testing.expect_value(t, format_rfc822(""), "")
}
@(test)
test_format_rfc822_full_date :: proc(t: ^testing.T) {
// Results are temp-allocated (reset by the test runner), so no delete.
// 2024-01-15 is a Monday; wall-clock time is preserved and the zone is
// rendered as an RFC-822 ±HHMM offset.
testing.expect_value(
t,
format_rfc822("2024-01-15T10:30:00Z"),
"Mon, 15 Jan 2024 10:30:00 +0000",
)
testing.expect_value(
t,
format_rfc822("2024-01-15T10:30:00+05:30"),
"Mon, 15 Jan 2024 10:30:00 +0530",
)
testing.expect_value(
t,
format_rfc822("2024-01-15T10:30:00-08:00"),
"Mon, 15 Jan 2024 10:30:00 -0800",
)
}
// ---------------------------------------------------------------------------
// generate_rss
// ---------------------------------------------------------------------------
@(test)
test_generate_rss_structure :: proc(t: ^testing.T) {
site: Site
site.title = "My & Site"
site.description = "Desc <here>"
site.base_url = "https://example.com"
site.pages = make(#soa[dynamic]Page)
defer delete_soa(site.pages)
// A real content page.
append(
&site.pages,
Page {
section = "posts",
title = "First <Post>",
url = "https://example.com/posts/first/",
content = "<p>Hello & welcome</p>",
},
)
// The home index page (section == "" && _is_index) must be skipped.
append(
&site.pages,
Page {
section = "",
_is_index = true,
title = "HOME_SHOULD_BE_SKIPPED",
url = "https://example.com/",
content = "home",
},
)
// A second content page.
append(
&site.pages,
Page {
section = "posts",
title = "Second Post",
url = "https://example.com/posts/second/",
content = "plain body",
},
)
rss := generate_rss(&site)
defer delete(rss)
// Channel header: site title/description are XML-escaped, link gets a
// trailing slash, and the atom self-link points at index.xml.
testing.expect(t, strings.contains(rss, "<title>My &amp; Site</title>"), rss)
testing.expect(
t,
strings.contains(rss, "<description>Desc &lt;here&gt;</description>"),
rss,
)
testing.expect(t, strings.contains(rss, "<link>https://example.com/</link>"), rss)
testing.expect(
t,
strings.contains(rss, `<atom:link href="https://example.com/index.xml"`),
rss,
)
// Exactly two <item>s — the home index page is skipped.
testing.expect_value(t, strings.count(rss, "<item>"), 2)
testing.expect(t, !strings.contains(rss, "HOME_SHOULD_BE_SKIPPED"), rss)
// First item: title and body are XML-escaped, link is verbatim.
testing.expect(t, strings.contains(rss, "<title>First &lt;Post&gt;</title>"), rss)
testing.expect(
t,
strings.contains(
rss,
"<description>&lt;p&gt;Hello &amp; welcome&lt;/p&gt;</description>",
),
rss,
)
testing.expect(
t,
strings.contains(rss, "<link>https://example.com/posts/first/</link>"),
rss,
)
// Undated pages get the sentinel pubDate (no date formatting/allocation).
testing.expect(
t,
strings.contains(rss, "<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>"),
rss,
)
// Well-formed at the boundaries.
testing.expect(t, strings.has_prefix(rss, `<?xml version="1.0"`), rss)
testing.expect(t, strings.has_suffix(rss, "</channel>\n</rss>"), rss)
}
flake.lock
Generated
+3 -3
View File
@@ -51,11 +51,11 @@
}, },
"nixpkgs-unstable": { "nixpkgs-unstable": {
"locked": { "locked": {
"lastModified": 1784525419, "lastModified": 1783475452,
"narHash": "sha256-yocJ4I4Kd4as0UPMFs9P7laPvvA2x/Bj8EW46iW3VVM=", "narHash": "sha256-3S92fSuv32mjJz2Qb2F4405c21J55F4NIphz5K6hxco=",
"owner": "NixOS", "owner": "NixOS",
"repo": "nixpkgs", "repo": "nixpkgs",
"rev": "a16c3fde2ffeab7f6326f50f460aaffde7ae066d", "rev": "05988b07fb05cbcb50be6bce197b4b5f75b5e61b",
"type": "github" "type": "github"
}, },
"original": { "original": {
flake.nix
+33 -99
View File
@@ -12,10 +12,12 @@
}; };
outputs = outputs =
inputs@{ flake-parts inputs@{ self
, flake-parts
, nixpkgs , nixpkgs
, ... , nixpkgs-unstable
# , process-compose-flake # , process-compose-flake
, treefmt-nix
}: }:
flake-parts.lib.mkFlake { inherit inputs; } { flake-parts.lib.mkFlake { inherit inputs; } {
imports = [ imports = [
@@ -25,49 +27,13 @@
systems = [ "x86_64-linux" ]; systems = [ "x86_64-linux" ];
perSystem = perSystem =
{ pkgs { pkgs, system, inputs', ... }: {
, system
, inputs'
, ...
}:
let
mkGrammarStaticLib = name: src: pkgs.stdenv.mkDerivation {
inherit name src;
dontConfigure = true;
buildPhase = ''
runHook preBuild
if [ -f src/scanner.cc ]; then
$CXX -fPIC -c src/scanner.cc -o scanner.o
elif [ -f src/scanner.c ]; then
$CC -fPIC -c src/scanner.c -o scanner.o
fi
$CC -fPIC -c src/parser.c -o parser.o
runHook postBuild
'';
installPhase = ''
runHook preInstall
mkdir -p $out/lib
ar rcs $out/lib/lib${name}.a *.o
runHook postInstall
'';
};
html-grammar = mkGrammarStaticLib "tree-sitter-html"
pkgs.tree-sitter-grammars.tree-sitter-html.src;
css-grammar = mkGrammarStaticLib "tree-sitter-css"
pkgs.tree-sitter-grammars.tree-sitter-css.src;
odin = pkgs.unstable.odin;
ols = pkgs.unstable.ols;
in
{
_module.args.pkgs = import nixpkgs { _module.args.pkgs = import nixpkgs {
inherit system; inherit system;
config.allowUnfree = true; config.allowUnfree = true;
overlays = [ overlays = [
(_final: _prev: { unstable = inputs'.nixpkgs-unstable.legacyPackages; }) (final: prev: { unstable = inputs'.nixpkgs-unstable.legacyPackages; })
]; ];
}; };
@@ -82,31 +48,22 @@
".env.local" ".env.local"
]; ];
# Format nix files # Format nix files
programs.nixpkgs-fmt.enable = true; programs.nixpkgs-fmt.enable = true;
programs.deadnix.enable = true; programs.deadnix.enable = true;
# Format js, json, and yaml files # Format js, json, and yaml files
programs.prettier.enable = true; programs.prettier.enable = true;
settings.formatter.prettier = { settings.formatter.prettier =
excludes = [ {
"public/**" excludes = [
"mustache/spec/specs/**" "public/**"
"*.html" "resources/js/modernizr.js"
"*.md" "storage/app/caniuse.json"
]; "*.md"
}; ];
};
settings.formatter.ols = {
command = "${pkgs.bash}/bin/bash";
options = [
"-euc"
''
${pkgs.ols}/bin/odinfmt -w .
''
];
includes = [ "*.odin" ];
};
}; };
#process-compose.default.settings.processes = { }; #process-compose.default.settings.processes = { };
@@ -117,21 +74,15 @@
src = ./.; src = ./.;
nativeBuildInputs = [ nativeBuildInputs = [
odin pkgs.odin
pkgs.pkg-config pkgs.pkg-config
]; ];
buildInputs = [ buildInputs = [
pkgs.git pkgs.git
pkgs.cmark pkgs.cmark
pkgs.tree-sitter
pkgs.tzdata
html-grammar
css-grammar
]; ];
LIBRARY_PATH = "${html-grammar}/lib:${css-grammar}/lib";
doCheck = true; doCheck = true;
checkPhase = '' checkPhase = ''
runHook preCheck runHook preCheck
@@ -141,49 +92,32 @@
buildPhase = '' buildPhase = ''
runHook preBuild runHook preBuild
odin build . -o:speed -no-bounds-check \ odin build . -o:speed -out:${pname}-keep
-define:THOR_DEFAULTS_PATH=$out/share/thor/defaults \ echo "Listing filles..."
-out:${pname}-keep ls .
runHook postBuild runHook postBuild
''; '';
installPhase = '' installPhase = ''
runHook preInstall runHook preInstall
install -Dm755 ${pname}-keep $out/bin/${pname} install -Dm755 ${pname}-keep $out/bin/${pname}
# Ship the built-in default layouts alongside the binary. The
# binary reads these at runtime from THOR_DEFAULTS_PATH (baked in
# above). Keep in sync with defaults.odin.
mkdir -p $out/share/thor
cp -r defaults $out/share/thor/defaults
runHook postInstall runHook postInstall
''; '';
}; };
devShells.default = pkgs.mkShell { devShells.default = pkgs.mkShell
buildInputs = [ odin ols ] ++ (with pkgs; [ {
cmark buildInputs = with pkgs; [
tree-sitter odin
ols
cmark
gdb # IDE
perf unstable.helix
typescript-language-server
bloaty # du for binaries? vscode-langservers-extracted
];
# temp };
hugo
pandoc
# IDE
unstable.helix
claude-code
typescript-language-server
vscode-langservers-extracted
]);
shellHook = ''
export LIBRARY_PATH="${html-grammar}/lib:${css-grammar}/lib:$LIBRARY_PATH"
'';
};
}; };
}; };
} }
footnotes.odin
+160
View File
@@ -0,0 +1,160 @@
package main
import cm "vendor:commonmark"
import "core:fmt"
import "core:strings"
// strip_definitions scans markdown text for footnote definitions ([^id]: text),
// removes them, and returns the cleaned text plus a map of id→definition.
// Handles multi-line definitions with indented continuation lines.
strip_definitions :: proc(body: string) -> (clean_body: string, defs: map[string]string) {
defs = make(map[string]string)
lines := strings.split(body, "\n")
output_lines: [dynamic]string
defer delete(output_lines)
i := 0
for i < len(lines) {
line := lines[i]
id, def_text, is_def := parse_def_line(line)
if !is_def {
append(&output_lines, line)
i += 1
continue
}
// Collect definition text (initial line + multi-line continuations)
def_parts: [dynamic]string
if def_text != "" {
append(&def_parts, def_text)
}
i += 1
for i < len(lines) {
next := lines[i]
// Stop at blank lines
if len(next) == 0 {
break
}
// Stop at new footnote definitions
_, _, is_new_def := parse_def_line(next)
if is_new_def {
break
}
// Include as continuation (trim indented lines)
if is_indented(next) {
append(&def_parts, strings.trim_left(next, " \t"))
} else {
append(&def_parts, next)
}
i += 1
}
defs[id] = strings.join(def_parts[:], "\n")
delete(def_parts)
}
clean_body = strings.join(output_lines[:], "\n")
return
}
// parse_def_line checks if a line is a footnote definition: [^id]: text
parse_def_line :: proc(line: string) -> (id: string, text: string, ok: bool) {
if len(line) < 5 || line[0] != '[' || line[1] != '^' {
return
}
close := strings.index(line[2:], "]")
if close < 0 {
return
}
close += 2
if close + 1 >= len(line) || line[close + 1] != ':' {
return
}
id = line[2:close]
text = strings.trim_left(line[close + 2:], " \t")
ok = true
return
}
is_indented :: proc(line: string) -> bool {
if len(line) == 0 {
return false
}
return line[0] == ' ' || line[0] == '\t'
}
// inject_sidenotes finds [^id] references in rendered HTML and replaces them
// with sidenote markup. Each definition is rendered through cmark separately.
inject_sidenotes :: proc(html: string, defs: map[string]string) -> string {
if len(defs) == 0 {
return html
}
parts: [dynamic]string
defer delete(parts)
remaining := html
for {
pos := strings.index(remaining, "[^")
if pos < 0 {
append(&parts, remaining)
break
}
// Append text before [^
append(&parts, remaining[:pos])
close := strings.index(remaining[pos + 2:], "]")
if close < 0 {
append(&parts, remaining[pos:])
break
}
id := remaining[pos + 2 : pos + 2 + close]
ref_end := pos + 2 + close + 1
def_text, found := defs[id]
if !found {
// No definition found, leave as literal text
append(&parts, remaining[pos:ref_end])
remaining = remaining[ref_end:]
continue
}
// Render definition through cmark for markdown support
def_html := cm.markdown_to_html_from_string(def_text, {.Unsafe})
def_html = strip_p_tags(def_html)
sidenote := fmt.aprintf(
`<label for="fn-%s" class="margin-toggle sidenote-number"></label><input type="checkbox" id="fn-%s" class="margin-toggle"><span class="sidenote">%s</span>`,
id,
id,
def_html,
)
append(&parts, sidenote)
remaining = remaining[ref_end:]
}
return strings.join(parts[:], "")
}
// strip_p_tags removes surrounding <p></p> if the HTML is a single paragraph.
strip_p_tags :: proc(html: string) -> string {
s := html
if len(s) > 0 && s[len(s) - 1] == '\n' {
s = s[:len(s) - 1]
}
if strings.has_prefix(s, "<p>") && strings.has_suffix(s, "</p>") {
return s[3:len(s) - 4]
}
return s
}
frontmatter.odin
+9 -54
View File
@@ -1,22 +1,17 @@
package main package main
import "core:encoding/json" import "core:encoding/json"
import "core:log" import "core:fmt"
import "core:strings" import "core:strings"
Frontmatter :: struct { Frontmatter :: struct {
title: string, title: string,
description: string, description: string,
date: string, date: string,
lastmod: string,
publishDate: string, publishDate: string,
weight: Maybe(int),
menus: json.Value,
params: json.Value,
layout: string,
og: Open_Graph,
draft: bool, draft: bool,
toc: bool, isStarred: bool,
menu: string,
} }
// parse_frontmatter splits raw file content into a Frontmatter struct and the // parse_frontmatter splits raw file content into a Frontmatter struct and the
@@ -38,9 +33,9 @@ parse_frontmatter :: proc(content: string) -> (fm: Frontmatter, body: string, ok
json_str := content[:end + 1] json_str := content[:end + 1]
body = strings.trim_left(content[end + 1:], " \t\r\n") body = strings.trim_left(content[end + 1:], " \t\r\n")
value, err := json.parse_string(json_str, spec = .JSON5) value, err := json.parse_string(json_str, spec = .JSON)
if err != nil { if err != nil {
log.errorf("failed to parse frontmatter JSON: %v", err) fmt.eprintfln("thor: failed to parse frontmatter JSON: %v", err)
return return
} }
defer json.destroy_value(value) defer json.destroy_value(value)
@@ -50,20 +45,13 @@ parse_frontmatter :: proc(content: string) -> (fm: Frontmatter, body: string, ok
return return
} }
fm.title = json_get_string(obj, "title") fm.title = json_get_string(obj, "title")
fm.description = json_get_string(obj, "description") fm.description = json_get_string(obj, "description")
fm.date = json_get_string(obj, "date") fm.date = json_get_string(obj, "date")
fm.lastmod = json_get_string(obj, "lastmod")
fm.publishDate = json_get_string(obj, "publishDate") fm.publishDate = json_get_string(obj, "publishDate")
fm.weight = json_get_int(obj, "weight")
fm.draft = json_get_bool(obj, "draft") fm.draft = json_get_bool(obj, "draft")
fm.toc = json_get_bool(obj, "toc") fm.isStarred = json_get_bool(obj, "isStarred")
if v, ok := obj["menus"]; ok { fm.menu = json_get_string(obj, "menu")
fm.menus = v
}
fm.layout = json_get_string(obj, "layout")
fm.og = json_get_open_graph(obj, "og")
if v, ok := obj["params"]; ok {fm.params = v}
ok = true ok = true
return return
@@ -87,36 +75,3 @@ json_get_bool :: proc(obj: json.Object, key: string) -> bool {
return false return false
} }
json_get_int :: proc(obj: json.Object, key: string) -> Maybe(int) {
if v, ok := obj[key]; ok {
switch val in v {
case json.Integer:
return int(val)
case json.Float:
return int(val)
case json.Boolean, json.String, json.Array, json.Object, json.Null:
return nil
}
}
return nil
}
json_get_open_graph :: proc(obj: json.Object, key: string) -> Open_Graph {
og: Open_Graph
if v, ok := obj[key]; ok {
if inner, ok2 := v.(json.Object); ok2 {
og.title = json_get_string(inner, "title")
og.type = json_get_string(inner, "type")
og.image = json_get_string(inner, "image")
og.url = json_get_string(inner, "url")
og.description = json_get_string(inner, "description")
og.locale = json_get_string(inner, "locale")
og.site_name = json_get_string(inner, "site_name")
og.published_time = json_get_string(inner, "published_time")
og.modified_time = json_get_string(inner, "modified_time")
og.section = json_get_string(inner, "section")
}
}
return og
}
html.odin
-217
View File
@@ -1,217 +0,0 @@
package main
import "core:strings"
strip_html_tags :: proc(s: string, allocator := context.allocator) -> string {
sb := strings.builder_make(allocator)
defer strings.builder_destroy(&sb)
in_tag := false
start := 0
for i in 0 ..< len(s) {
if s[i] == '<' && !in_tag {
if i > start {
strings.write_string(&sb, s[start:i])
}
in_tag = true
} else if s[i] == '>' && in_tag {
in_tag = false
start = i + 1
}
}
if start == 0 {
return s
}
if !in_tag && start < len(s) {
strings.write_string(&sb, s[start:])
}
return strings.to_string(sb)
}
unescape_html :: proc(s: string) -> string {
sb := strings.builder_make_len_cap(0, len(s))
defer strings.builder_destroy(&sb)
start := 0
for i in 0 ..< len(s) {
if s[i] != '&' {
continue
}
semi := strings.index(s[i:], ";")
if semi < 0 {
break
}
entity := s[i:i + semi + 1]
replacement := ""
switch entity {
case "&amp;":
replacement = "&"
case "&lt;":
replacement = "<"
case "&gt;":
replacement = ">"
case "&quot;":
replacement = "\""
case "&#39;", "&apos;":
replacement = "'"
case:
continue
}
if i > start {
strings.write_string(&sb, s[start:i])
}
strings.write_string(&sb, replacement)
start = i + semi + 1
}
if start == 0 {
return s
}
if start < len(s) {
strings.write_string(&sb, s[start:])
}
return strings.to_string(sb)
}
// generate_summary truncates an HTML string to the first max_words words.
// Walks forward counting whitespace→text transitions, skipping tag interiors
// so spaces inside attributes don't count. Returns a substring of the
// original — zero allocation. Mirrors Hugo's default (70 words).
generate_summary :: proc(html: string, max_words: int = 70) -> string {
if max_words <= 0 {
return ""
}
word_count := 0
in_word := false
in_tag := false
for i in 0 ..< len(html) {
c := html[i]
if in_tag {
if c == '>' {
in_tag = false
}
continue
}
if c == '<' {
in_tag = true
if in_word {
word_count += 1
if word_count >= max_words {
return html[:i]
}
in_word = false
}
continue
}
is_space := c == ' ' || c == '\n' || c == '\t' || c == '\r'
if is_space {
if in_word {
word_count += 1
if word_count >= max_words {
return html[:i]
}
in_word = false
}
} else {
in_word = true
}
}
return html
}
// generate_description converts an HTML fragment to plain text by stripping
// tags, decoding entities, and collapsing whitespace. Emits a space when
// exiting any tag so block-level boundaries aren't lost. Intended for OG
// descriptions — operate on the output of generate_summary for bounded input.
generate_description :: proc(html: string, allocator := context.temp_allocator) -> string {
sb := strings.builder_make_len_cap(0, len(html), allocator)
defer strings.builder_destroy(&sb)
in_tag := false
prev_was_space := true
run_start := 0
i := 0
for i < len(html) {
c := html[i]
if in_tag {
if c == '>' {
in_tag = false
if !prev_was_space {
strings.write_byte(&sb, ' ')
prev_was_space = true
}
}
i += 1
run_start = i
continue
}
if c == '<' {
if i > run_start {
strings.write_string(&sb, html[run_start:i])
prev_was_space = false
}
in_tag = true
i += 1
continue
}
if c == '&' {
if i > run_start {
strings.write_string(&sb, html[run_start:i])
prev_was_space = false
}
semi := strings.index(html[i:], ";")
if semi > 0 && semi <= 5 {
entity := html[i:i + semi + 1]
replacement := ""
switch entity {
case "&amp;":
replacement = "&"
case "&lt;":
replacement = "<"
case "&gt;":
replacement = ">"
case "&quot;":
replacement = "\""
case "&#39;", "&apos;":
replacement = "'"
case:
replacement = ""
}
if replacement != "" {
strings.write_string(&sb, replacement)
prev_was_space = false
i += semi + 1
run_start = i
continue
}
}
strings.write_byte(&sb, '&')
prev_was_space = false
i += 1
run_start = i
continue
}
if c == ' ' || c == '\n' || c == '\t' || c == '\r' {
if i > run_start {
strings.write_string(&sb, html[run_start:i])
prev_was_space = false
}
if !prev_was_space {
strings.write_byte(&sb, ' ')
prev_was_space = true
}
i += 1
run_start = i
continue
}
i += 1
}
if i > run_start && !in_tag {
strings.write_string(&sb, html[run_start:i])
}
result := strings.to_string(sb)
if len(result) > 0 && result[len(result) - 1] == ' ' {
result = result[:len(result) - 1]
}
return result
}
html_test.odin
-105
View File
@@ -1,105 +0,0 @@
#+test
package main
import "core:testing"
// --- generate_summary ---
@(test)
test_summary_short :: proc(t: ^testing.T) {
result := generate_summary("<p>Hello world</p>")
testing.expect_value(t, result, "<p>Hello world</p>")
}
@(test)
test_summary_word_limit :: proc(t: ^testing.T) {
result := generate_summary("<p>one two three four five</p>", max_words = 3)
testing.expect_value(t, result, "<p>one two three")
}
@(test)
test_summary_empty :: proc(t: ^testing.T) {
result := generate_summary("")
testing.expect_value(t, result, "")
}
@(test)
test_summary_no_words :: proc(t: ^testing.T) {
result := generate_summary("<p></p>")
testing.expect_value(t, result, "<p></p>")
}
@(test)
test_summary_tags_not_counted :: proc(t: ^testing.T) {
html := `<pre><code><span class="hl-keyword">if</span> x</code></pre>`
result := generate_summary(html, max_words = 1)
testing.expect_value(t, result, `<pre><code><span class="hl-keyword">if`)
}
// --- generate_description ---
@(test)
test_description_simple :: proc(t: ^testing.T) {
result := generate_description("<p>Hello world</p>")
testing.expect_value(t, result, "Hello world")
}
@(test)
test_description_entities :: proc(t: ^testing.T) {
result := generate_description("<p>Cats &amp; dogs &lt;3</p>")
testing.expect_value(t, result, "Cats & dogs <3")
}
@(test)
test_description_nested_tags :: proc(t: ^testing.T) {
result := generate_description("<p><strong>Bold</strong> text</p>")
testing.expect_value(t, result, "Bold text")
}
@(test)
test_description_block_boundary :: proc(t: ^testing.T) {
result := generate_description("<p>First</p><p>Second</p>")
testing.expect_value(t, result, "First Second")
}
@(test)
test_description_whitespace_collapse :: proc(t: ^testing.T) {
result := generate_description("<p> Multiple spaces </p>")
testing.expect_value(t, result, "Multiple spaces")
}
@(test)
test_description_empty :: proc(t: ^testing.T) {
result := generate_description("")
testing.expect_value(t, result, "")
}
@(test)
test_description_plain_text :: proc(t: ^testing.T) {
result := generate_description("Just plain text")
testing.expect_value(t, result, "Just plain text")
}
@(test)
test_description_highlighted_code :: proc(t: ^testing.T) {
result := generate_description(`<pre><code><span class="hl-keyword">if</span> x</code></pre>`)
testing.expect_value(t, result, "if x")
}
@(test)
test_description_list_items :: proc(t: ^testing.T) {
result := generate_description("<ul><li>One</li><li>Two</li></ul>")
testing.expect_value(t, result, "One Two")
}
@(test)
test_description_headings :: proc(t: ^testing.T) {
result := generate_description("<h1>Title</h1><p>Body</p>")
testing.expect_value(t, result, "Title Body")
}
@(test)
test_description_blockquote :: proc(t: ^testing.T) {
result := generate_description("<blockquote>Quote</blockquote>")
testing.expect_value(t, result, "Quote")
}
icons.odin
+11
View File
@@ -0,0 +1,11 @@
package main
ICON_HOME :: `<svg aria-hidden="true" class="hi-svg-inline h-6 w-6" fill="none" height="1em" stroke="currentColor" stroke-linecap="round" stroke-linejoin="round" stroke-width="2" viewBox="0 0 24 24" width="1em"><path d="m3 9 9-7 9 7v11a2 2 0 01-2 2H5a2 2 0 01-2-2z"/><polyline points="9 22 9 12 15 12 15 22"/></svg>`
ICON_GITHUB :: `<svg aria-hidden="true" class="hi-svg-inline h-7 w-7" fill="none" height="1em" stroke="currentColor" stroke-linecap="round" stroke-linejoin="round" stroke-width="2" viewBox="0 0 24 24" width="1em"><path d="M15 22v-4a4.8 4.8.0 00-1-3.5c3 0 6-2 6-5.5.08-1.25-.27-2.48-1-3.5.28-1.15.28-2.35.0-3.5.0.0-1 0-3 1.5-2.64-.5-5.36-.5-8 0C6 2 5 2 5 2c-.3 1.15-.3 2.35.0 3.5A5.403 5.403.0 004 9c0 3.5 3 5.5 6 5.5-.39.49-.68 1.05-.85 1.65-.17.6-.22 1.23-.15 1.85v4"/><path d="M9 18c-4.51 2-5-2-7-2"/></svg>`
ICON_RSS :: `<svg aria-hidden="true" class="hi-svg-inline h-7 w-7" fill="none" height="1em" stroke="currentColor" stroke-linecap="round" stroke-linejoin="round" stroke-width="2" viewBox="0 0 24 24" width="1em"><path d="M4 11a9 9 0 019 9"/><path d="M4 4a16 16 0 0116 16"/><circle cx="5" cy="19" r="1"/></svg>`
ICON_CHEVRON_UP :: `<svg aria-hidden="true" class="hi-svg-inline h-10 w-10" fill="none" height="1em" stroke="currentColor" stroke-linecap="round" stroke-linejoin="round" stroke-width="2" viewBox="0 0 24 24" width="1em"><path d="m18 15-6-6-6 6"/></svg>`
ICON_STAR :: `<svg aria-hidden="true" class="hi-svg-inline h-3.5 w-3.5 text-yellow-500 align-baseline mr-2" fill="currentColor" height="1em" viewBox="0 0 16 16" width="1em"><path d="M3.612 15.443c-.386.198-.824-.149-.746-.592l.83-4.73L.173 6.765c-.329-.314-.158-.888.283-.95l4.898-.696L7.538.792c.197-.39.73-.39.927 0l2.184 4.327 4.898.696c.441.062.612.636.282.95l-3.522 3.356.83 4.73c.078.443-.36.79-.746.592L8 13.187l-4.389 2.256z"/></svg>`
main.odin
+56 -104
View File
@@ -1,120 +1,72 @@
package main package main
import "base:runtime"
import "core:flags" import "core:flags"
import "core:log" import "core:fmt"
import "core:mem"
import "core:os" import "core:os"
import "core:prof/spall"
import "core:sync"
import "core:time"
import "treesitter" Options :: struct {
base_url: string `args:"name=base-url" usage:"hostname (and path) to the root, e.g. https://example.com/"`,
SPALL :: #config(SPALL, false) content_dir: string `args:"name=content" usage:"where to look for content files"`,
output_dir: string `args:"name=output" usage:"where to export the completed site"`,
when SPALL { drafts: bool `args:"name=drafts" usage:"Whether to render draft posts"`,
spall_ctx: spall.Context
@(thread_local)
spall_buffer: spall.Buffer
init_spall_for_thread :: proc() {
backing := make([]u8, spall.BUFFER_DEFAULT_SIZE, context.temp_allocator)
spall_buffer = spall.buffer_create(backing, u32(sync.current_thread_id()))
}
cleanup_spall_for_thread :: proc() {
spall.buffer_destroy(&spall_ctx, &spall_buffer)
}
} }
main :: proc() { main :: proc() {
when SPALL { opt: Options
ctx, ok := spall.context_create_with_scale("thor.spall", false, 1.0)
if !ok { flags.parse_or_exit(&opt, os.args, .Odin)
log.fatal("Failed to create spall trace file")
os.exit(1) load_default_options(&opt)
pages := walk_content(opt.content_dir, opt.drafts)
render_site(pages, opt.content_dir, opt.output_dir, opt.base_url)
}
load_default_options :: proc(opt: ^Options) {
if opt.content_dir == "" {
opt.content_dir = "./content"
}
if opt.output_dir == "" {
opt.output_dir = "./public"
}
if opt.base_url == "" {
opt.base_url = "http://localhost:8080"
}
}
print_summary :: proc(pages: []Page) {
fmt.printfln("Pages: %d\n", len(pages))
for page in pages {
type_label := "post"
if page.type == .Standalone {
type_label = "standalone"
} }
spall_ctx = ctx
defer spall.context_destroy(&spall_ctx)
spall_backing := make([]u8, spall.BUFFER_DEFAULT_SIZE) date_short := page.date
defer delete(spall_backing) if len(date_short) > 10 {
date_short = date_short[:10]
}
spall_buffer = spall.buffer_create(spall_backing, u32(sync.current_thread_id())) badge := ""
defer spall.buffer_destroy(&spall_ctx, &spall_buffer) if page.draft {
} badge = " (draft)"
} else if page.is_starred {
badge = " *"
}
cli_flags: Flags fmt.printfln(
flags.parse_or_exit(&cli_flags, os.args, .Odin) " [%-11s] %-30s %s %s%s (%d bytes html)",
type_label,
level: log.Level page.title,
switch { date_short,
case cli_flags.quiet: page.permalink,
level = .Warning badge,
case cli_flags.verbose: len(page.body_html),
level = .Debug
case:
level = .Info
}
logger_opts: log.Options =
(log.Default_Console_Logger_Opts - log.Full_Timestamp_Opts - {.Short_File_Path})
console_logger := log.create_console_logger(level, logger_opts)
context.logger = console_logger
defer log.destroy_console_logger(console_logger)
treesitter.init_persistent()
when SPALL {
treesitter.set_thread_callbacks(init_spall_for_thread, cleanup_spall_for_thread)
}
for {
defer free_all(context.temp_allocator)
defer log.debugf(
"max_temp_allocator_size=%M",
(cast(^runtime.Default_Temp_Allocator)context.temp_allocator.data)^.arena.total_used,
) )
tick := time.tick_now()
site: Site
init_site(&site, cli_flags)
defer destroy_site(&site)
// TODO: Make it so this isn't necessary
context.allocator = site_allocator(&site)
defer log.debugf(
"current_site_allocator_size=%M",
site.arena.block_size * (len(site.arena.used_blocks) + len(site.arena.unused_blocks)) -
site.arena.bytes_left,
)
build_vfs(&site)
treesitter.grammar_dir = site.grammars
treesitter.query_dir = site.queries
site_load_content(&site)
render_site(&site)
log.infof("Built site in %M", time.tick_since(tick))
(.Watch in site.features) or_break
time.sleep(5 * time.Second)
}
}
when SPALL {
@(instrumentation_enter)
spall_enter :: proc "contextless" (
proc_address, call_site_return_address: rawptr,
loc: runtime.Source_Code_Location,
) {
spall._buffer_begin(&spall_ctx, &spall_buffer, "", "", loc)
}
@(instrumentation_exit)
spall_exit :: proc "contextless" (
proc_address, call_site_return_address: rawptr,
loc: runtime.Source_Code_Location,
) {
spall._buffer_end(&spall_ctx, &spall_buffer)
} }
} }
main_test.odin
+1
View File
@@ -7,3 +7,4 @@ import "core:testing"
test_true :: proc(t: ^testing.T) { test_true :: proc(t: ^testing.T) {
testing.expect(t, true) testing.expect(t, true)
} }
markdown/alerts.odin
-97
View File
@@ -1,97 +0,0 @@
#+feature dynamic-literals
package markdown
import "core:strings"
ALERT_EMOJIS: map[string]string = {
"note" = "ℹ️",
"tip" = "💡",
"important" = "❗",
"warning" = "⚠️",
"caution" = "❗",
}
inject_alerts :: proc(html: string) -> string {
sb := strings.builder_make()
defer strings.builder_destroy(&sb)
remaining := html
for {
bq_start := strings.index(remaining, "<blockquote>")
if bq_start < 0 {
strings.write_string(&sb, remaining)
break
}
bq_close := strings.index(remaining, "</blockquote>")
if bq_close < 0 {
strings.write_string(&sb, remaining)
break
}
bq_end := bq_close + len("</blockquote>")
bq := remaining[bq_start:bq_end]
strings.write_string(&sb, remaining[:bq_start])
transform_alert(&sb, bq)
remaining = remaining[bq_end:]
}
return strings.to_string(sb)
}
transform_alert :: proc(sb: ^strings.Builder, bq: string) {
pos := len("<blockquote>")
for pos < len(bq) &&
(bq[pos] == '\n' || bq[pos] == '\r' || bq[pos] == ' ' || bq[pos] == '\t') {
pos += 1
}
if pos + 4 >= len(bq) ||
bq[pos] != '<' ||
bq[pos + 1] != 'p' ||
bq[pos + 2] != '>' ||
bq[pos + 3] != '[' ||
bq[pos + 4] != '!' {
strings.write_string(sb, bq)
return
}
close := strings.index(bq[pos + 5:], "]")
if close < 0 {
strings.write_string(sb, bq)
return
}
type_raw := bq[pos + 5:pos + 5 + close]
type_lower := strings.to_lower(type_raw, context.temp_allocator)
emoji, found := ALERT_EMOJIS[type_lower]
if !found {
strings.write_string(sb, bq)
return
}
after_type := pos + 5 + close + 1
content_start := after_type
if content_start < len(bq) && (bq[content_start] == '+' || bq[content_start] == '-') {
content_start += 1
}
if content_start < len(bq) && bq[content_start] == ' ' {
content_start += 1
}
rest := bq[content_start:]
strings.write_string(sb, `<blockquote class="alert alert-`)
strings.write_string(sb, type_lower)
strings.write_string(sb, `">`)
strings.write_string(sb, "\n")
strings.write_string(sb, `<p class="alert-title">`)
strings.write_string(sb, emoji)
strings.write_string(sb, " ")
strings.write_string(sb, rest)
}
markdown/alerts_test.odin
-103
View File
@@ -1,103 +0,0 @@
#+feature dynamic-literals
#+test
package markdown
import "core:testing"
@(test)
test_alerts_render_properly :: proc(t: ^testing.T) {
testing.expect_value(
t,
inject_alerts("<blockquote>\n<p>[!NOTE] Hello.</p>\n</blockquote>"),
`<blockquote class="alert alert-note">
<p class="alert-title">ℹ️ Hello.</p>
</blockquote>`,
)
testing.expect_value(
t,
inject_alerts("<blockquote>\n<p>[!TIP] Be smart.</p>\n</blockquote>"),
`<blockquote class="alert alert-tip">
<p class="alert-title">💡 Be smart.</p>
</blockquote>`,
)
testing.expect_value(
t,
inject_alerts("<blockquote>\n<p>[!CAUTION] Danger!</p>\n</blockquote>"),
`<blockquote class="alert alert-caution">
<p class="alert-title">❗ Danger!</p>
</blockquote>`,
)
}
@(test)
test_non_alerts_pass_through_inject_alerts :: proc(t: ^testing.T) {
testing.expect_value(
t,
inject_alerts("<blockquote>\n<p>Just a quote.</p>\n</blockquote>"),
"<blockquote>\n<p>Just a quote.</p>\n</blockquote>",
)
testing.expect_value(
t,
inject_alerts("<blockquote>\n<p>[!UNKNOWN] Nope.</p>\n</blockquote>"),
"<blockquote>\n<p>[!UNKNOWN] Nope.</p>\n</blockquote>",
)
testing.expect_value(
t,
inject_alerts("<p>No blockquote here.</p>"),
"<p>No blockquote here.</p>",
)
testing.expect_value(t, inject_alerts(""), "")
}
@(test)
test_case_insensitive_alerts_render :: proc(t: ^testing.T) {
testing.expect_value(
t,
inject_alerts("<blockquote>\n<p>[!Note] Mixed case.</p>\n</blockquote>"),
`<blockquote class="alert alert-note">
<p class="alert-title">ℹ️ Mixed case.</p>
</blockquote>`,
)
testing.expect_value(
t,
inject_alerts("<blockquote>\n<p>[!NOTE] Upper case.</p>\n</blockquote>"),
`<blockquote class="alert alert-note">
<p class="alert-title">ℹ️ Upper case.</p>
</blockquote>`,
)
}
@(test)
test_multiple_alerts_render_together :: proc(t: ^testing.T) {
testing.expect_value(
t,
inject_alerts(
"<blockquote>\n<p>[!NOTE] First.</p>\n</blockquote>\n<blockquote>\n<p>[!TIP] Second.</p>\n</blockquote>",
),
`<blockquote class="alert alert-note">
<p class="alert-title">ℹ️ First.</p>
</blockquote>
<blockquote class="alert alert-tip">
<p class="alert-title">💡 Second.</p>
</blockquote>`,
)
testing.expect_value(
t,
inject_alerts(
"<blockquote>\n<p>Regular.</p>\n</blockquote>\n<blockquote>\n<p>[!WARNING] Alert!</p>\n</blockquote>",
),
`<blockquote>
<p>Regular.</p>
</blockquote>
<blockquote class="alert alert-warning">
<p class="alert-title">⚠️ Alert!</p>
</blockquote>`,
)
}
markdown/deflists.odin
-324
View File
@@ -1,324 +0,0 @@
package markdown
import cm "vendor:commonmark"
import "core:strings"
// convert_deflists scans markdown text for definition list patterns and
// converts them to <dl><dt><dd> HTML blocks before cmark processing.
//
// A definition line starts with fewer than four columns of leading whitespace
// followed by a colon and a space (or end-of-line). The term is the nearest
// preceding non-blank line (immediately or within one blank line). Consecutive
// term+definition pairs are grouped into a single <dl> block, and consecutive
// definition lines under one term become multiple <dd> elements.
//
// The scanner walks the body one line at a time using byte offsets: it never
// splits the input, allocates nothing per line, and copies non-deflist text
// through verbatim. Matching is disabled inside fenced code blocks. Terms and
// definitions are rendered through cmark individually so that inline markdown
// (code, links, emphasis) is processed.
// Line_Kind classifies a line for convert_deflists's group-finding scan. Other
// covers fenced-code delimiters and their contents, where definition syntax is
// inert.
Line_Kind :: enum {
None,
Blank,
Term,
Def,
Other,
}
convert_deflists :: proc(body: string, allocator := context.allocator) -> string {
// Fast path: a body with no definition line at all cannot contain a
// definition list, so return it untouched — skipping the builder allocation
// and the classifying scan below. This is the common case (most pages have
// no definition lists), and the guard scan is leaner than the full loop.
if !has_def_line(body) {
return body
}
sb := strings.builder_make(context.temp_allocator)
// emitted_up_to is the byte offset up to which `body` has been copied into
// `sb`. It stays at 0 until the first deflist is emitted; if it is still 0
// at the end, nothing matched and we can return the body unchanged.
emitted_up_to := 0
in_fence := false
fence_marker: u8 = 0
// The scan is definition-driven: it walks line by line, remembering the two
// previous lines' kinds and offsets, and only does group-finding work when it
// reaches a (rare) definition line — whose term is the immediately preceding
// line, or the one before a single blank line. This avoids a per-line forward
// lookahead: each line is classified once, with no rescanning.
prev1_kind, prev2_kind := Line_Kind.None, Line_Kind.None
prev1_pos, prev2_pos := 0, 0
pos := 0
for {
line, next, ok := line_at(body, pos)
if !ok {
break
}
// Classify the line. Definition-list syntax is inert inside fenced code
// blocks, so fence delimiters and their contents are classified Other.
kind: Line_Kind
if in_fence {
if is_fence_line(line, fence_marker) {
in_fence = false
}
kind = .Other
} else if marker, opened := fence_open(line); opened {
in_fence = true
fence_marker = marker
kind = .Other
} else if is_def_line(line) {
kind = .Def
} else if is_blank_line(line) {
kind = .Blank
} else {
kind = .Term
}
// A definition-list group begins at the term preceding this def line
// (immediately, or across a single blank line). Flush pending passthrough
// text verbatim, then render the whole group directly into the builder.
if kind == .Def {
term_pos := -1
if prev1_kind == .Term {
term_pos = prev1_pos
} else if prev1_kind == .Blank && prev2_kind == .Term {
term_pos = prev2_pos
}
if term_pos >= 0 {
strings.write_string(&sb, body[emitted_up_to:term_pos])
group_end := write_deflist_group(&sb, body, term_pos)
// write_deflist_group emits a raw <dl> HTML block with no
// trailing newline, so the blank line the author left after the
// list is consumed as the </dl> line's terminator rather than a
// separating blank line. CommonMark (type-6 HTML block) then
// swallows the following block — e.g. a fenced code example —
// as raw, unescaped HTML, leaking its markup (a template's
// <h1>) into the document. Re-emit a newline so </dl> stands on
// its own line and the author's blank line still separates the
// block. Only when a blank line actually follows: fused
// non-blank fall-through content is left as-is.
if line, _, ok := line_at(body, group_end); ok && is_blank_line(line) {
strings.write_byte(&sb, '\n')
}
emitted_up_to = group_end
pos = group_end
prev1_kind, prev2_kind = .None, .None
in_fence = false
continue
}
}
prev2_kind, prev2_pos = prev1_kind, prev1_pos
prev1_kind, prev1_pos = kind, pos
pos = next
}
if emitted_up_to == 0 {
// No definition list anywhere; the body passes through unchanged.
return body
}
strings.write_string(&sb, body[emitted_up_to:])
return strings.clone(strings.to_string(sb), allocator)
}
// line_at returns the line beginning at byte offset `pos` (excluding its
// trailing newline) along with the offset of the following line. `pos` must be
// 0 or one past a '\n'. ok is false once `pos` reaches the end of `body`.
line_at :: proc(body: string, pos: int) -> (line: string, next: int, ok: bool) {
if pos >= len(body) {
return "", pos, false
}
nl := strings.index_byte(body[pos:], '\n')
if nl < 0 {
return body[pos:], len(body), true
}
return body[pos:pos + nl], pos + nl + 1, true
}
// has_def_line reports whether any line in `body` is a definition line (see
// is_def_line). It is a cheap guard for convert_deflists: a body with no
// definition line cannot contain a definition list. There are no false
// negatives (every definition list requires a definition line); a false
// positive — e.g. a colon line that only appears inside a code fence — simply
// falls through to the full scan, which then finds no match and returns `body`.
has_def_line :: proc(body: string) -> bool {
pos := 0
for {
line, next := line_at(body, pos) or_break
if is_def_line(line) {
return true
}
pos = next
}
return false
}
// write_deflist_group renders the definition-list group beginning at `start`
// directly into `sb`, wrapped in <dl>...</dl>, and returns the byte offset one
// past the group (the start of the first line not consumed). The caller must
// have confirmed via term_has_def that a group starts at `start`, so the group
// always contains at least one term+definition pair. A trailing blank line that
// separates the group from following content is left unconsumed.
write_deflist_group :: proc(sb: ^strings.Builder, body: string, start: int) -> (end: int) {
strings.write_string(sb, "<dl>")
pos := start
for {
term_line, term_next, term_ok := line_at(body, pos)
if !term_ok || is_blank_line(term_line) || is_def_line(term_line) {
break
}
// The definition follows immediately or across a single blank line.
def_pos := term_next
def_line, def_next, def_ok := line_at(body, def_pos)
if def_ok && is_blank_line(def_line) {
def_pos = def_next
def_line, def_next, def_ok = line_at(body, def_pos)
}
if !def_ok || !is_def_line(def_line) {
break
}
// Emit the term with its first definition, then any consecutive
// definition lines as additional <dd>s under the same term.
strings.write_string(sb, "<dt>")
strings.write_string(sb, render_inline_md(strings.trim_space(term_line)))
strings.write_string(sb, "</dt><dd>")
strings.write_string(sb, render_inline_md(def_content(def_line)))
strings.write_string(sb, "</dd>")
pos = def_next
for {
cont, cont_next, cont_ok := line_at(body, pos)
if !cont_ok || !is_def_line(cont) {
break
}
strings.write_string(sb, "<dd>")
strings.write_string(sb, render_inline_md(def_content(cont)))
strings.write_string(sb, "</dd>")
pos = cont_next
}
// Continue the group if another term+definition pair follows,
// optionally separated by one blank line. Otherwise stop here without
// consuming a trailing blank line.
blank_line, after_blank, blank_ok := line_at(body, pos)
if blank_ok && is_blank_line(blank_line) {
if term_has_def(body, after_blank) {
pos = after_blank
continue
}
break
}
// No blank line: loop and let the top re-test this line as a term.
}
strings.write_string(sb, "</dl>")
return pos
}
// term_has_def reports whether the line at `pos` is a term that is immediately
// (or across one blank line) followed by a definition line.
term_has_def :: proc(body: string, pos: int) -> bool {
line, next, ok := line_at(body, pos)
if !ok || is_blank_line(line) || is_def_line(line) {
return false
}
def_pos := next
def_line, def_next, def_ok := line_at(body, def_pos)
if def_ok && is_blank_line(def_line) {
def_pos = def_next
def_line, def_next, def_ok = line_at(body, def_pos)
}
return def_ok && is_def_line(def_line)
}
// fence_open reports whether `line` opens a fenced code block and, if so, which
// marker character it uses. Leading spaces are ignored.
fence_open :: proc(line: string) -> (marker: u8, ok: bool) {
trimmed := strings.trim_left(line, " ")
if strings.has_prefix(trimmed, "```") {
return '`', true
}
if strings.has_prefix(trimmed, "~~~") {
return '~', true
}
return 0, false
}
// is_fence_line reports whether `line` is a fence using the given marker
// character (used to detect the closing fence of an open block).
is_fence_line :: proc(line: string, marker: u8) -> bool {
trimmed := strings.trim_left(line, " ")
if marker == '~' {
return strings.has_prefix(trimmed, "~~~")
}
return strings.has_prefix(trimmed, "```")
}
// is_def_line returns true if the line is a definition line: fewer than four
// columns of leading whitespace, a colon, then whitespace or end-of-line. Four
// or more columns of indentation is a code block, not a definition.
is_def_line :: proc(line: string) -> bool {
indent := 0
n := 0
for n < len(line) && (line[n] == ' ' || line[n] == '\t') {
indent += line[n] == '\t' ? 4 : 1
n += 1
}
if indent >= 4 {
return false
}
rest := line[n:]
if len(rest) < 1 || rest[0] != ':' {
return false
}
if len(rest) == 1 {
return true
}
return rest[1] == ' ' || rest[1] == '\t'
}
// def_content extracts the definition text from a definition line,
// stripping the leading colon and surrounding whitespace.
def_content :: proc(line: string) -> string {
trimmed := strings.trim_left(line, " \t")
content := trimmed[1:]
content = strings.trim_left(content, " \t")
return content
}
// is_blank_line returns true for empty or whitespace-only lines.
is_blank_line :: proc(line: string) -> bool {
return strings.trim_space(line) == ""
}
// render_inline_md renders a snippet of markdown through cmark and strips
// the surrounding <p> tags. Result lives in context.temp_allocator.
render_inline_md :: proc(text: string) -> string {
raw := cm.markdown_to_html_from_string(text, {.Unsafe})
defer cm.free_string(raw)
return strings.clone(strip_p_tags(raw), context.temp_allocator)
}
// strip_p_tags removes surrounding <p></p> if the HTML is a single paragraph.
strip_p_tags :: proc(html: string) -> string {
s := html
if len(s) > 0 && s[len(s) - 1] == '\n' {
s = s[:len(s) - 1]
}
if strings.has_prefix(s, "<p>") && strings.has_suffix(s, "</p>") {
return s[3:len(s) - 4]
}
return s
}
markdown/deflists_fixtures_test.odin
-92
View File
@@ -1,92 +0,0 @@
#+test
package markdown
import "core:testing"
// Fixture-driven tests for convert_deflists. Every fixture is exact-matched
// against its oracle in testdata/deflists/expected/. See
// testdata/deflists/reference/README.md for how each expected file was decided
// (cross-checked against Pandoc and Hugo/Goldmark).
//
// The fixture inputs are the definition-list examples from
// https://www.markdownlang.com/extended/definition-lists.html
//
// Provenance of each expected file:
// 01,02,08,09 Pandoc and Hugo agree; oracle = that agreed HTML.
// 06 Both agree it is NOT a deflist; convert_deflists passes the
// markdown through unchanged, so expected = the raw input.
// 07 CommonMark's <4-space marker rule: we accept the 2-space colon
// (Pandoc does; Hugo does not).
// 05 Block nesting is unsupported; our flatten happens to match
// Pandoc's output exactly.
// 03,04 Block content in definitions is unsupported; expected captures
// our current degraded behavior (a <dl> for the term/first line,
// the rest falling through as markdown) as a regression guard.
Deflist_Fixture :: struct {
name: string,
input: string,
expected: string,
}
@(test)
test_deflist_fixtures :: proc(t: ^testing.T) {
cases := []Deflist_Fixture {
{
"01-basic",
#load(#directory + "testdata/deflists/01-basic.md", string),
#load(#directory + "testdata/deflists/expected/01-basic.html", string),
},
{
"02-multiple-defs",
#load(#directory + "testdata/deflists/02-multiple-defs.md", string),
#load(#directory + "testdata/deflists/expected/02-multiple-defs.html", string),
},
{
"03-multiline-def",
#load(#directory + "testdata/deflists/03-multiline-def.md", string),
#load(#directory + "testdata/deflists/expected/03-multiline-def.html", string),
},
{
"04-markdown-in-def",
#load(#directory + "testdata/deflists/04-markdown-in-def.md", string),
#load(#directory + "testdata/deflists/expected/04-markdown-in-def.html", string),
},
{
"05-nested",
#load(#directory + "testdata/deflists/05-nested.md", string),
#load(#directory + "testdata/deflists/expected/05-nested.html", string),
},
{
"06-compact",
#load(#directory + "testdata/deflists/06-compact.md", string),
#load(#directory + "testdata/deflists/expected/06-compact.html", string),
},
{
"07-extra-space",
#load(#directory + "testdata/deflists/07-extra-space.md", string),
#load(#directory + "testdata/deflists/expected/07-extra-space.html", string),
},
{
"08-glossary",
#load(#directory + "testdata/deflists/08-glossary.md", string),
#load(#directory + "testdata/deflists/expected/08-glossary.html", string),
},
{
"09-api-params",
#load(#directory + "testdata/deflists/09-api-params.md", string),
#load(#directory + "testdata/deflists/expected/09-api-params.html", string),
},
}
for c in cases {
result := convert_deflists(c.input, context.temp_allocator)
testing.expectf(
t,
result == c.expected,
"[%s]\n expected: %q\n got: %q",
c.name,
c.expected,
result,
)
}
}
markdown/deflists_test.odin
-115
View File
@@ -1,115 +0,0 @@
#+test
package markdown
import "core:strings"
import "core:testing"
@(test)
test_single_entry :: proc(t: ^testing.T) {
input := "term\n\n: definition"
result := convert_deflists(input, context.temp_allocator)
expected := "<dl><dt>term</dt><dd>definition</dd></dl>"
testing.expect_value(t, result, expected)
}
@(test)
test_single_entry_no_blank :: proc(t: ^testing.T) {
input := "term\n: definition"
result := convert_deflists(input, context.temp_allocator)
expected := "<dl><dt>term</dt><dd>definition</dd></dl>"
testing.expect_value(t, result, expected)
}
@(test)
test_indented_variant :: proc(t: ^testing.T) {
input := " term\n : definition"
result := convert_deflists(input, context.temp_allocator)
expected := "<dl><dt>term</dt><dd>definition</dd></dl>"
testing.expect_value(t, result, expected)
}
@(test)
test_multiple_entries :: proc(t: ^testing.T) {
input := "t1\n\n: d1\n\nt2\n\n: d2"
result := convert_deflists(input, context.temp_allocator)
expected := "<dl><dt>t1</dt><dd>d1</dd><dt>t2</dt><dd>d2</dd></dl>"
testing.expect_value(t, result, expected)
}
@(test)
test_mixed_indented_and_non_indented :: proc(t: ^testing.T) {
input := "t1\n\n: d1\n\n t2\n : d2\n\nt3\n\n: d3"
result := convert_deflists(input, context.temp_allocator)
expected := "<dl><dt>t1</dt><dd>d1</dd><dt>t2</dt><dd>d2</dd><dt>t3</dt><dd>d3</dd></dl>"
testing.expect_value(t, result, expected)
}
@(test)
test_inline_markdown_in_term :: proc(t: ^testing.T) {
input := "`code`\n\n: def"
result := convert_deflists(input, context.temp_allocator)
expected := "<dl><dt><code>code</code></dt><dd>def</dd></dl>"
testing.expect_value(t, result, expected)
}
@(test)
test_inline_markdown_in_definition :: proc(t: ^testing.T) {
input := "term\n\n: see [Content](#content) here"
result := convert_deflists(input, context.temp_allocator)
testing.expect(t, strings.contains(result, `<a href="#content">Content</a>`))
testing.expect(t, strings.contains(result, "<dd>see "))
testing.expect(t, strings.contains(result, "</dd>"))
}
@(test)
test_regular_text_passes_through :: proc(t: ^testing.T) {
input := "This is: not a deflist"
result := convert_deflists(input, context.temp_allocator)
testing.expect_value(t, result, input)
}
@(test)
test_colon_inside_paragraph_no_false_positive :: proc(t: ^testing.T) {
input := "First paragraph.\n\nSecond paragraph."
result := convert_deflists(input, context.temp_allocator)
testing.expect_value(t, result, input)
}
@(test)
test_deflist_between_paragraphs :: proc(t: ^testing.T) {
input := "Before.\n\nterm\n\n: def\n\nAfter."
result := convert_deflists(input, context.temp_allocator)
testing.expect(t, strings.has_prefix(result, "Before."))
testing.expect(t, strings.contains(result, "<dl><dt>term</dt><dd>def</dd></dl>"))
testing.expect(t, strings.has_suffix(result, "After."))
}
@(test)
test_empty_body :: proc(t: ^testing.T) {
result := convert_deflists("", context.temp_allocator)
testing.expect_value(t, result, "")
}
// Regression: a fenced code block following a definition list must stay a
// separate block. Without a blank line after the emitted </dl>, CommonMark
// treats the fence as raw HTML continuation of the <dl> block and passes its
// contents through unescaped (leaking e.g. an example <h1> into the document).
@(test)
test_deflist_before_code_fence :: proc(t: ^testing.T) {
input := "term\n\n: def\n\n```\n<h1>example</h1>\n```"
result := convert_deflists(input, context.temp_allocator)
testing.expect(t, strings.contains(result, "</dl>\n\n```"))
}
@(test)
test_docs_md_pattern :: proc(t: ^testing.T) {
input := "content\n\n: `content` holds your pages.\n\n assets\n : `assets` contains files."
result := convert_deflists(input, context.temp_allocator)
testing.expect(t, strings.contains(result, "<dl>"))
testing.expect(t, strings.contains(result, "<dt>content</dt>"))
testing.expect(t, strings.contains(result, "<dt>assets</dt>"))
testing.expect(t, strings.contains(result, "<code>content</code>"))
testing.expect(t, strings.contains(result, "<code>assets</code>"))
testing.expect(t, strings.contains(result, "</dl>"))
}
markdown/emoji.odin
-436
View File
@@ -1,436 +0,0 @@
#+feature dynamic-literals
package markdown
import "core:strings"
EMOJIS: map[string]string = {
// People
"smile" = "😄",
"laughing" = "😆",
"blush" = "😊",
"smiley" = "😃",
"wink" = "😉",
"joy" = "😂",
"rofl" = "🤣",
"relaxed" = "☺️",
"thinking" = "🤔",
"neutral_face" = "😐",
"expressionless" = "😑",
"no_mouth" = "😶",
"rolling_eyes" = "🙄",
"smirk" = "😏",
"persevere" = "😣",
"disappointed_relieved" = "😥",
"open_mouth" = "😮",
"zipper_mouth_face" = "🤐",
"hushed" = "😯",
"sleepy" = "😪",
"tired_face" = "😫",
"sleeping" = "😴",
"relieved" = "😌",
"stuck_out_tongue" = "😛",
"stuck_out_tongue_winking_eye" = "😜",
"stuck_out_tongue_closed_eyes" = "😝",
"drooling_face" = "🤤",
"unamused" = "😒",
"sweat" = "😅",
"pensive" = "😔",
"confused" = "😕",
"upside_down_face" = "🙃",
"money_mouth_face" = "🤑",
"astonished" = "😲",
"white_frowning_face" = "☹️",
"slightly_sad_face" = "☹️",
"confounded" = "😖",
"disappointed" = "😞",
"worried" = "😟",
"triumph" = "😤",
"cry" = "😢",
"sob" = "😭",
"frowning" = "😦",
"frowning_face" = "🙁",
"anguished" = "😧",
"fearful" = "😨",
"weary" = "😩",
"grimacing" = "😬",
"cold_sweat" = "😰",
"scream" = "😱",
"flushed" = "😳",
"dizzy_face" = "😵",
"rage" = "😡",
"angry" = "😠",
"innocent" = "😇",
"cowboy_hat_face" = "🤠",
"clown_face" = "🤡",
"mask" = "😷",
"thermometer_face" = "🤒",
"head_bandage" = "🤕",
"nauseated_face" = "🤢",
"sneezing_face" = "🤧",
"smiling_imp" = "😈",
"imp" = "👿",
"shrug" = "¯\\_(ツ)_/¯",
"facepalm" = "🤦",
"facepunch" = "👊",
"wave" = "👋",
"ok_hand" = "👌",
"thumbsup" = "👍",
"thumbsdown" = "👎",
"clap" = "👏",
"pray" = "🙏",
"point_up" = "☝️",
"point_down" = "👇",
"point_left" = "👈",
"point_right" = "👉",
"v" = "✌️",
"raised_hands" = "🙌",
"muscle" = "💪",
"fist" = "✊",
"hand" = "✋",
// Hearts & symbols
"heart" = "❤️",
"orange_heart" = "🧡",
"yellow_heart" = "💛",
"green_heart" = "💚",
"blue_heart" = "💙",
"purple_heart" = "💜",
"broken_heart" = "💔",
"sparkling_heart" = "💖",
"100" = "💯",
"anger" = "💢",
"checkered_flag" = "🏁",
"crossed_flags" = "🚩",
"rocket" = "🚀",
"star" = "⭐",
"star2" = "🌟",
"sparkles" = "✨",
"boom" = "💥",
"exclamation" = "❗",
"question" = "❓",
"grey_exclamation" = "❕",
"grey_question" = "❔",
"zzz" = "💤",
"warning" = "⚠️",
"no_entry_sign" = "🚫",
"no_entry" = "⛔",
"white_check_mark" = "✅",
"negative_squared_cross_mark" = "❎",
"x" = "❌",
"o" = "⭕",
"heavy_plus_sign" = "➕",
"heavy_minus_sign" = "➖",
"heavy_division_sign" = "➗",
"copyright" = "©️",
"registered" = "®️",
"tm" = "™️",
// Nature & weather
"fire" = "🔥",
"zap" = "⚡",
"sunny" = "☀️",
"cloud" = "☁️",
"rainbow" = "🌈",
"snowflake" = "❄️",
"dash" = "💨",
"tornado" = "🌪️",
"deciduous_tree" = "🌳",
"evergreen_tree" = "🌲",
"palm_tree" = "🌴",
"cactus" = "🌵",
"tulip" = "🌷",
"rose" = "🌹",
"sunflower" = "🌻",
"hibiscus" = "🌺",
"earth_africa" = "🌍",
"earth_americas" = "🌎",
"earth_asia" = "🌏",
"new_moon" = "🌑",
"full_moon" = "🌕",
// Food & drink
"coffee" = "☕",
"tea" = "🍵",
"beer" = "🍺",
"beers" = "🍻",
"wine_glass" = "🍷",
"tropical_drink" = "🍹",
"apple" = "🍎",
"green_apple" = "🍏",
"orange" = "🍊",
"lemon" = "🍋",
"banana" = "🍌",
"watermelon" = "🍉",
"grapes" = "🍇",
"strawberry" = "🍓",
"melon" = "🍈",
"cherries" = "🍒",
"peach" = "🍑",
"pineapple" = "🍍",
"pizza" = "🍕",
"hamburger" = "🍔",
"hotdog" = "🌭",
"taco" = "🌮",
"burrito" = "🌯",
"cake" = "🍰",
"cookie" = "🍪",
"chocolate_bar" = "🍫",
"candy" = "🍬",
"popcorn" = "🍿",
// Animals
"dog" = "🐶",
"cat" = "🐱",
"mouse" = "🐭",
"hamster" = "🐹",
"rabbit" = "🐰",
"bear" = "🐻",
"panda_face" = "🐼",
"koala" = "🐨",
"tiger" = "🐯",
"lion_face" = "🦁",
"cow" = "🐮",
"pig" = "🐷",
"frog" = "🐸",
"monkey_face" = "🐵",
"see_no_evil" = "🙈",
"hear_no_evil" = "🙉",
"speak_no_evil" = "🙊",
"owl" = "🦉",
"bat" = "🦇",
"wolf" = "🐺",
"boar" = "🐗",
"horse" = "🐴",
"unicorn" = "🦄",
"honeybee" = "🐝",
"bug" = "🐛",
"snail" = "🐌",
"beetle" = "🐞",
"ant" = "🐜",
"spider" = "🕷",
"scorpion" = "🦂",
"turtle" = "🐢",
"snake" = "🐍",
"octopus" = "🐙",
"shell" = "🐚",
"whale" = "🐳",
"dolphin" = "🐬",
"fish" = "🐟",
"tropical_fish" = "🐠",
"blowfish" = "🐡",
"penguin" = "🐧",
"chicken" = "🐔",
"bird" = "🐦",
"eagle" = "🦅",
// Objects
"computer" = "💻",
"iphone" = "📱",
"keyboard" = "⌨️",
"desktop_computer" = "🖥",
"printer" = "🖨",
"mouse_three_button" = "🖱",
"joystick" = "🕹",
"stash" = "💾",
"floppy_disk" = "💾",
"cd" = "💿",
"dvd" = "📀",
"vhs" = "📼",
"camera" = "📷",
"video_camera" = "📹",
"tv" = "📺",
"radio" = "📻",
"pager" = "📟",
"telephone" = "☎️",
"fax" = "📠",
"bulb" = "💡",
"candle" = "🕯",
"bathtub" = "🛁",
"shower" = "🚿",
"toilet" = "🚽",
"wrench" = "🔧",
"hammer" = "🔨",
"nut_and_bolt" = "🔩",
"gear" = "⚙️",
"link" = "🔗",
"lock" = "🔒",
"unlock" = "🔓",
"key" = "🔑",
"bell" = "🔔",
"bookmark" = "🔖",
"pushpin" = "📌",
"paperclip" = "📎",
"memo" = "📝",
"pencil2" = "✏️",
"black_nib" = "✒️",
"pen" = "🖊",
"paintbrush" = "🖌",
"crayon" = "🖍",
"book" = "📖",
"books" = "📚",
"ledger" = "📒",
"notebook" = "📓",
"scroll" = "📜",
"page_with_curl" = "📄",
"newspaper" = "📰",
"chart_with_upwards_trend" = "📈",
"chart_with_downwards_trend" = "📉",
"bar_chart" = "📊",
"calendar" = "📅",
"date" = "📅",
"hourglass" = "⌛",
"hourglass_flowing_sand" = "⏳",
"clock" = "🕐",
"alarm_clock" = "⏰",
"stopwatch" = "⏱",
"watch" = "⌚",
"moneybag" = "💰",
"yen" = "💴",
"dollar" = "💵",
"euro" = "💶",
"pound" = "💷",
"money_with_wings" = "💸",
"credit_card" = "💳",
"gem" = "💎",
"bomb" = "💣",
"gift" = "🎁",
"balloon" = "🎈",
"tada" = "🎉",
"confetti_ball" = "🎊",
"package" = "📦",
"mailbox" = "📫",
"inbox_tray" = "📥",
"outbox_tray" = "📤",
"email" = "✉️",
"envelope" = "✉️",
"incoming_envelope" = "📨",
// Activities
"soccer" = "⚽",
"basketball" = "🏀",
"football" = "🏈",
"baseball" = "⚾",
"tennis" = "🎾",
"8ball" = "🎱",
"bowling" = "🎳",
"video_game" = "🎮",
"dart" = "🎯",
"game_die" = "🎲",
"slot_machine" = "🎰",
"cards" = "🃏",
"black_joker" = "🃟",
"mahjong" = "🀄",
"musical_note" = "🎵",
"notes" = "🎶",
"saxophone" = "🎷",
"guitar" = "🎸",
"musical_keyboard" = "🎹",
"trumpet" = "🎺",
"violin" = "🎻",
"drum" = "🥁",
"headphones" = "🎧",
"microphone" = "🎤",
"level_slider" = "🎚",
"control_knobs" = "🎛",
"ticket" = "🎫",
"art" = "🎨",
"circus_tent" = "🎪",
"theater_masks" = "🎭",
"clapper" = "🎬",
// Travel
"car" = "🚗",
"taxi" = "🚕",
"bus" = "🚍",
"train" = "🚆",
"metro" = "🚇",
"light_rail" = "🚈",
"tram" = "🚊",
"bike" = "🚲",
"motorcycle" = "🏍",
"airplane" = "✈️",
"helicopter" = "🚁",
"boat" = "⛵",
"sailboat" = "⛵",
"ship" = "🚢",
"fuelpump" = "⛽",
"construction" = "🚧",
"house" = "🏠",
"house_with_garden" = "🏡",
"office" = "🏢",
"post_office" = "🏣",
"hospital" = "🏥",
"bank" = "🏦",
"hotel" = "🏨",
"school" = "🏫",
"department_store" = "🏬",
"church" = "⛪",
"castle" = "🏰",
"factory" = "🏭",
"tokyo_tower" = "🗼",
"statue_of_liberty" = "🗽",
"fountain" = "⛲",
"tent" = "⛺",
"mountain" = "⛰️",
"snow_capped_mountain" = "🏔",
"beach" = "🏖",
"camping" = "🏕",
"world_map" = "🗺",
"japan" = "🗾",
}
expand_emoji :: proc(text: string) -> string {
assert(len(EMOJIS) > 0)
sb := strings.builder_make()
defer strings.builder_destroy(&sb)
remaining := text
for {
colon := strings.index(remaining, ":")
if colon < 0 {
strings.write_string(&sb, remaining)
break
}
after := remaining[colon + 1:]
end := strings.index(after, ":")
if end < 0 {
strings.write_string(&sb, remaining)
break
}
shortcode := remaining[colon + 1:colon + 1 + end]
valid := true
for c in shortcode {
if !(c >= 'a' && c <= 'z') &&
!(c >= '0' && c <= '9') &&
c != '_' &&
c != '+' &&
c != '-' {
valid = false
break
}
}
if !valid || len(shortcode) == 0 {
strings.write_string(&sb, remaining[:colon + 1])
remaining = remaining[colon + 1:]
continue
}
emoji, found := EMOJIS[shortcode]
if !found {
strings.write_string(&sb, remaining[:colon + 1])
remaining = remaining[colon + 1:]
continue
}
strings.write_string(&sb, remaining[:colon])
strings.write_string(&sb, emoji)
remaining = remaining[colon + 1 + end + 1:]
}
return strings.to_string(sb)
}
markdown/emoji_test.odin
-33
View File
@@ -1,33 +0,0 @@
#+test
package markdown
import "core:testing"
@(test)
test_basic_emoji_expansion_works :: proc(t: ^testing.T) {
testing.expect_value(t, expand_emoji(":smile:"), "😄")
testing.expect_value(t, expand_emoji(":smile: :heart:"), "😄 ❤️")
testing.expect_value(t, expand_emoji(":smile::heart:"), "😄❤️")
testing.expect_value(t, expand_emoji("Hello :wave:!"), "Hello 👋!")
testing.expect_value(t, expand_emoji(":smile: rest"), "😄 rest")
testing.expect_value(t, expand_emoji("rest :smile:"), "rest 😄")
testing.expect_value(t, expand_emoji(":100:"), "💯")
testing.expect_value(t, expand_emoji(":stuck_out_tongue:"), "😛")
}
@(test)
test_emoji_skips_non_matches :: proc(t: ^testing.T) {
testing.expect_value(t, expand_emoji("Hello world"), "Hello world")
testing.expect_value(t, expand_emoji("Hello:"), "Hello:")
testing.expect_value(t, expand_emoji(":notreal:"), ":notreal:")
testing.expect_value(t, expand_emoji("http://example.com"), "http://example.com")
testing.expect_value(t, expand_emoji(""), "")
}
@(test)
test_emoji_skips_invalid_shortcodes :: proc(t: ^testing.T) {
testing.expect_value(t, expand_emoji("::"), "::")
testing.expect_value(t, expand_emoji(":Smile:"), ":Smile:")
testing.expect_value(t, expand_emoji(": not real :"), ": not real :")
}
markdown/footnotes.odin
-294
View File
@@ -1,294 +0,0 @@
package markdown
import "core:fmt"
import "core:strings"
Note_Kind :: enum {
Sidenote,
Marginnote,
}
// strip_definitions scans markdown text for note definitions ([^id]: text for
// sidenotes, [*id]: text for marginnotes), removes them, and returns the cleaned
// text plus separate maps of id->definition for each kind.
// Handles multi-line definitions with indented continuation lines.
strip_definitions :: proc(
body: string,
) -> (
clean_body: string,
sn_defs, mn_defs: map[string]string,
) {
lines := strings.split(body, "\n")
defer delete(lines)
out_sb := strings.builder_make()
defer strings.builder_destroy(&out_sb)
i := 0
for i < len(lines) {
line := lines[i]
id, def_text, kind, is_def := parse_def_line(line)
if !is_def {
if strings.builder_len(out_sb) > 0 {
strings.write_string(&out_sb, "\n")
}
strings.write_string(&out_sb, line)
i += 1
continue
}
def_sb := strings.builder_make()
if def_text != "" {
strings.write_string(&def_sb, def_text)
}
i += 1
for i < len(lines) {
next := lines[i]
if len(next) == 0 {
break
}
_, _, _, is_new_def := parse_def_line(next)
if is_new_def {
break
}
if strings.builder_len(def_sb) > 0 {
strings.write_string(&def_sb, "\n")
}
if is_indented(next) {
strings.write_string(&def_sb, strings.trim_left(next, " \t"))
} else {
strings.write_string(&def_sb, next)
}
i += 1
}
joined := strings.clone(strings.to_string(def_sb))
strings.builder_destroy(&def_sb)
if kind == .Marginnote {
mn_defs[id] = joined
} else {
sn_defs[id] = joined
}
}
clean_body = strings.clone(strings.to_string(out_sb))
return
}
// parse_def_line checks if a line is a note definition: [^id]: text (sidenote)
// or [*id]: text (marginnote).
parse_def_line :: proc(line: string) -> (id: string, text: string, kind: Note_Kind, ok: bool) {
if len(line) < 5 || line[0] != '[' {
return
}
if line[1] == '^' {
kind = .Sidenote
} else if line[1] == '*' {
kind = .Marginnote
} else {
return
}
close := strings.index(line[2:], "]")
if close < 0 {
return
}
close += 2
if close + 1 >= len(line) || line[close + 1] != ':' {
return
}
id = line[2:close]
text = strings.trim_left(line[close + 2:], " \t")
ok = true
return
}
is_indented :: proc(line: string) -> bool {
if len(line) == 0 {
return false
}
return line[0] == ' ' || line[0] == '\t'
}
// inject_notes finds [^id] (sidenote) and [*id] (marginnote) references in
// rendered HTML and replaces them with the appropriate margin markup. Each
// definition is rendered through cmark separately.
inject_notes :: proc(html: string, sn_defs, mn_defs: map[string]string) -> string {
if len(sn_defs) == 0 && len(mn_defs) == 0 {
return html
}
parts: strings.Builder
strings.builder_init_len(&parts, 0) // TODO: Set a reasonable default
defer strings.builder_destroy(&parts)
remaining := html
for {
sn_pos := strings.index(remaining, "[^")
mn_pos := strings.index(remaining, "[*")
is_margin := mn_pos >= 0 && (sn_pos < 0 || mn_pos < sn_pos)
pos := sn_pos
if is_margin {
pos = mn_pos
}
if pos < 0 {
strings.write_string(&parts, remaining)
break
}
// Append text before the reference
strings.write_string(&parts, remaining[:pos])
close := strings.index(remaining[pos + 2:], "]")
if close < 0 {
strings.write_string(&parts, remaining[pos:])
break
}
id := remaining[pos + 2:pos + 2 + close]
ref_end := pos + 2 + close + 1
defs := sn_defs
if is_margin {
defs = mn_defs
}
def_text, found := defs[id]
if !found {
// No definition found, leave as literal text
strings.write_string(&parts, remaining[pos:ref_end])
remaining = remaining[ref_end:]
continue
}
// Render definition through cmark for markdown support
def_html := render_inline_md(def_text)
note: string
defer delete(note)
if is_margin {
fmt.sbprintf(
&parts,
`<label for="mn-%s" class="margin-toggle"></label><input type="checkbox" id="mn-%s" class="margin-toggle"><span class="marginnote">%s</span>`,
id,
id,
def_html,
)
} else {
fmt.sbprintf(
&parts,
`<label for="fn-%s" class="margin-toggle sidenote-number"></label><input type="checkbox" id="fn-%s" class="margin-toggle"><span class="sidenote">%s</span>`,
id,
id,
def_html,
)
}
remaining = remaining[ref_end:]
}
return strings.to_string(parts)
}
// inject_footnotes finds [^id] and [*id] references in rendered HTML, numbers
// them sequentially by order of appearance, and replaces them with <sup> links.
// Appends a <section class="footnotes"><ol> at the end with definitions.
// Both sidenote ([^id]) and marginnote ([*id]) references are treated equally.
inject_footnotes :: proc(html: string, sn_defs, mn_defs: map[string]string) -> string {
if len(sn_defs) == 0 && len(mn_defs) == 0 {
return html
}
parts: strings.Builder
strings.builder_init_len(&parts, 0)
defer strings.builder_destroy(&parts)
number_of: map[string]int = make(map[string]int, allocator = context.temp_allocator)
ordered_ids: [dynamic]string = make([dynamic]string, 0, allocator = context.temp_allocator)
next_num := 1
remaining := html
for {
sn_pos := strings.index(remaining, "[^")
mn_pos := strings.index(remaining, "[*")
is_margin := mn_pos >= 0 && (sn_pos < 0 || mn_pos < sn_pos)
pos := sn_pos
if is_margin {
pos = mn_pos
}
if pos < 0 {
strings.write_string(&parts, remaining)
break
}
strings.write_string(&parts, remaining[:pos])
close := strings.index(remaining[pos + 2:], "]")
if close < 0 {
strings.write_string(&parts, remaining[pos:])
break
}
id := remaining[pos + 2:pos + 2 + close]
ref_end := pos + 2 + close + 1
defs := sn_defs
if is_margin {
defs = mn_defs
}
def_text, found := defs[id]
if !found {
strings.write_string(&parts, remaining[pos:ref_end])
remaining = remaining[ref_end:]
continue
}
num, seen := number_of[id]
if !seen {
num = next_num
number_of[id] = num
next_num += 1
append(&ordered_ids, id)
}
fmt.sbprintf(&parts, `<sup><a href="#fn-%d" id="fnref-%d">%d</a></sup>`, num, num, num)
remaining = remaining[ref_end:]
}
if len(ordered_ids) > 0 {
strings.write_string(&parts, "\n<section class=\"footnotes\">\n<hr>\n<ol>\n")
for id in ordered_ids {
def_text, ok := sn_defs[id]
if !ok {
def_text, ok = mn_defs[id]
}
if !ok do continue
def_html := render_inline_md(def_text)
num := number_of[id]
fmt.sbprintf(
&parts,
`<li id="fn-%d">%s <a href="#fnref-%d" class="footnote-backref">↩︎</a></li>` +
"\n",
num,
def_html,
num,
)
}
strings.write_string(&parts, "</ol>\n</section>")
}
return strings.to_string(parts)
}
markdown/footnotes_test.odin
-220
View File
@@ -1,220 +0,0 @@
#+feature dynamic-literals
#+test
package markdown
import "core:strings"
import "core:testing"
@(test)
test_parse_def_line :: proc(t: ^testing.T) {
// sidenote definition
id, text, kind, ok := parse_def_line("[^foo]: bar baz")
testing.expect(t, ok)
testing.expect(t, id == "foo")
testing.expect(t, text == "bar baz")
testing.expect(t, kind == .Sidenote)
// marginnote definition
id, text, kind, ok = parse_def_line("[*baz]: qux")
testing.expect(t, ok)
testing.expect(t, id == "baz")
testing.expect(t, text == "qux")
testing.expect(t, kind == .Marginnote)
// plain text is not a definition
_, _, _, ok = parse_def_line("regular text")
testing.expect(t, !ok)
// a bracket that is not a note (e.g. reference-link style) is ignored
_, _, _, ok = parse_def_line("[link]: url")
testing.expect(t, !ok)
}
@(test)
test_strip_definitions :: proc(t: ^testing.T) {
body := "Intro[^a] and [*b].\n\n[^a]: a side\n\n[*b]: a margin\n"
clean, sn_defs, mn_defs := strip_definitions(body)
defer {
delete(clean)
delete(sn_defs["a"])
delete(sn_defs)
delete(mn_defs["b"])
delete(mn_defs)
}
// references stay; definitions are removed
testing.expect(t, strings.contains(clean, "Intro[^a]"))
testing.expect(t, strings.contains(clean, "[*b]"))
testing.expect(t, !strings.contains(clean, "[^a]:"))
testing.expect(t, !strings.contains(clean, "[*b]:"))
// routed to the correct map by sigil
sa, sa_ok := sn_defs["a"]
testing.expect(t, sa_ok)
testing.expect(t, sa == "a side")
mb, mb_ok := mn_defs["b"]
testing.expect(t, mb_ok)
testing.expect(t, mb == "a margin")
// ids do not leak across maps
_, leaked := sn_defs["b"]
testing.expect(t, !leaked)
_, leaked2 := mn_defs["a"]
testing.expect(t, !leaked2)
}
@(test)
test_inject_notes :: proc(t: ^testing.T) {
html := "Text[^a] more [*b] end."
sn_defs := map[string]string {
"a" = "side note",
}
defer delete(sn_defs)
mn_defs := map[string]string {
"b" = "margin note",
}
defer delete(mn_defs)
out := inject_notes(html, sn_defs, mn_defs)
// sidenote: numbered, fn- prefix, .sidenote span, rendered text
testing.expect(
t,
strings.contains(out, `for="fn-a" class="margin-toggle sidenote-number"></label>`),
)
testing.expect(t, strings.contains(out, `class="sidenote"`))
testing.expect(t, strings.contains(out, "side note"))
// marginnote: unnumbered, mn- prefix, .marginnote span, rendered text
testing.expect(t, strings.contains(out, `for="mn-b" class="margin-toggle"></label>`))
testing.expect(t, strings.contains(out, `class="marginnote"`))
testing.expect(t, strings.contains(out, "margin note"))
}
@(test)
test_inject_notes_no_defs :: proc(t: ^testing.T) {
html := "No notes here."
sn := make(map[string]string)
mn := make(map[string]string)
testing.expect(t, inject_notes(html, sn, mn) == html)
}
@(test)
test_inject_notes_missing_ref :: proc(t: ^testing.T) {
html := "Ref[^missing] and [*missing] end."
sn := map[string]string {
"other" = "x",
}
defer delete_map(sn)
mn := map[string]string {
"other2" = "y",
}
defer delete_map(mn)
out := inject_notes(html, sn, mn)
// references with no matching definition are left as literal text
testing.expect(t, len(out) > 2)
testing.expect(t, strings.contains(out, "[^missing]"))
testing.expect(t, strings.contains(out, "[*missing]"))
}
@(test)
test_inject_footnotes_basic :: proc(t: ^testing.T) {
html := "Text[^a] end."
sn := map[string]string {
"a" = "a footnote",
}
defer delete_map(sn)
mn := make(map[string]string)
defer delete_map(mn)
out := inject_footnotes(html, sn, mn)
testing.expect(t, strings.contains(out, `<sup><a href="#fn-1" id="fnref-1">1</a></sup>`))
testing.expect(t, strings.contains(out, `<li id="fn-1">a footnote`))
testing.expect(t, strings.contains(out, `class="footnote-backref"`))
testing.expect(t, strings.contains(out, `<section class="footnotes">`))
testing.expect(t, strings.contains(out, "</section>"))
}
@(test)
test_inject_footnotes_numbered_by_appearance :: proc(t: ^testing.T) {
html := "Second[^b] then first[^a]."
sn := map[string]string {
"a" = "def a",
"b" = "def b",
}
defer delete_map(sn)
mn := make(map[string]string)
defer delete_map(mn)
out := inject_footnotes(html, sn, mn)
// b appears first in the text → 1, a → 2
testing.expect(t, strings.contains(out, `id="fnref-1">1</a></sup>`))
testing.expect(t, strings.contains(out, `id="fnref-2">2</a></sup>`))
testing.expect(t, strings.contains(out, `<li id="fn-1">def b`))
testing.expect(t, strings.contains(out, `<li id="fn-2">def a`))
}
@(test)
test_inject_footnotes_marginnote_treated_same :: proc(t: ^testing.T) {
html := "Sidenote[^a] and marginnote[*b]."
sn := map[string]string {
"a" = "sn def",
}
defer delete_map(sn)
mn := map[string]string {
"b" = "mn def",
}
defer delete_map(mn)
out := inject_footnotes(html, sn, mn)
// Both get numbered as regular footnotes
testing.expect(t, strings.contains(out, `id="fnref-1">1</a></sup>`))
testing.expect(t, strings.contains(out, `id="fnref-2">2</a></sup>`))
testing.expect(t, strings.contains(out, `<li id="fn-1">sn def`))
testing.expect(t, strings.contains(out, `<li id="fn-2">mn def`))
}
@(test)
test_inject_footnotes_no_defs :: proc(t: ^testing.T) {
html := "No notes here."
sn := make(map[string]string)
mn := make(map[string]string)
testing.expect(t, inject_footnotes(html, sn, mn) == html)
}
@(test)
test_inject_footnotes_missing_def :: proc(t: ^testing.T) {
html := "Ref[^missing] end."
sn := map[string]string {
"other" = "x",
}
defer delete_map(sn)
mn := make(map[string]string)
defer delete_map(mn)
out := inject_footnotes(html, sn, mn)
testing.expect(t, strings.contains(out, "[^missing]"))
testing.expect(t, !strings.contains(out, "<section"))
}
@(test)
test_inject_footnotes_inline_markdown :: proc(t: ^testing.T) {
html := "Text[^a] end."
sn := map[string]string {
"a" = "see [link](http://example.com) here",
}
defer delete_map(sn)
mn := make(map[string]string)
defer delete_map(mn)
out := inject_footnotes(html, sn, mn)
testing.expect(t, strings.contains(out, `<a href="http://example.com">link</a>`))
}
markdown/heading_ids.odin
-195
View File
@@ -1,195 +0,0 @@
package markdown
import "core:fmt"
import "core:log"
import "core:strings"
// A hypothetical maximum slug length.
// May be enforced in a later version (for performance)
MAX_SLUG_LENGTH :: #config(MAX_SLUG_LENGTH, 255)
inject_heading_ids :: proc(html: string, allocator := context.allocator) -> string {
sb := strings.builder_make_len_cap(0, len(html) + 256, allocator)
defer strings.builder_destroy(&sb)
seen := make(map[string]bool, 8, context.temp_allocator)
empty_count := 0
pos := 0
for {
h_start := find_heading_open(html, pos)
if h_start < 0 {
strings.write_string(&sb, html[pos:])
break
}
if h_start > pos {
strings.write_string(&sb, html[pos:h_start])
}
level := int(html[h_start + 2] - '0')
close_buf: [5]u8
close_buf[0] = '<'; close_buf[1] = '/'; close_buf[2] = 'h'
close_buf[3] = html[h_start + 2]
close_buf[4] = '>'
close_tag := string(close_buf[:])
close_rel := strings.index(html[h_start:], close_tag)
if close_rel < 0 {
strings.write_string(&sb, html[h_start:])
break
}
open_tag_end := h_start + 4
close_start := h_start + close_rel
close_end := close_start + 5
inner_html := html[open_tag_end:close_start]
text := extract_plain_text(inner_html, context.temp_allocator)
slug := slugify(text)
if len(slug) == 0 {
empty_count += 1
slug = fmt.tprintf("section-%d", empty_count)
}
slug = make_unique(slug, &seen)
fmt.sbprintf(&sb, `<h%d id="%s">`, level, slug)
strings.write_string(&sb, inner_html)
strings.write_string(&sb, close_tag)
pos = close_end
}
return strings.to_string(sb)
}
find_heading_open :: proc(html: string, start: int) -> int {
pos := start
for pos < len(html) - 3 {
if html[pos] == '<' &&
html[pos + 1] == 'h' &&
html[pos + 2] >= '1' &&
html[pos + 2] <= '6' &&
html[pos + 3] == '>' {
return pos
}
pos += 1
}
return -1
}
extract_plain_text :: proc(html: string, allocator := context.temp_allocator) -> string {
sb := strings.builder_make(allocator)
defer strings.builder_destroy(&sb)
in_tag := false
i := 0
for i < len(html) {
c := html[i]
if in_tag {
if c == '>' {
in_tag = false
}
i += 1
continue
}
if c == '<' {
in_tag = true
i += 1
continue
}
if c == '&' {
semi := strings.index(html[i:], ";")
if semi > 0 && semi <= 5 {
entity := html[i:i + semi + 1]
replacement := ""
switch entity {
case "&amp;":
replacement = "&"
case "&lt;":
replacement = "<"
case "&gt;":
replacement = ">"
case "&quot;":
replacement = "\""
case "&#39;", "&apos;":
replacement = "'"
case:
replacement = ""
}
if replacement != "" {
strings.write_string(&sb, replacement)
i += semi + 1
continue
}
}
strings.write_byte(&sb, '&')
i += 1
continue
}
strings.write_byte(&sb, c)
i += 1
}
return strings.to_string(sb)
}
slugify :: proc(text: string, allocator := context.temp_allocator) -> string {
sb := strings.builder_make_len_cap(0, 255, allocator)
defer strings.builder_destroy(&sb)
has_hyphen := false
for i in 0 ..< len(text) {
c := text[i]
if c >= 'A' && c <= 'Z' {
strings.write_byte(&sb, c + 32)
has_hyphen = false
} else if (c >= 'a' && c <= 'z') || (c >= '0' && c <= '9') {
strings.write_byte(&sb, c)
has_hyphen = false
} else {
if !has_hyphen {
strings.write_byte(&sb, '-')
has_hyphen = true
}
}
}
result := strings.to_string(sb)
if len(result) > 0 && result[len(result) - 1] == '-' {
result = result[:len(result) - 1]
}
if len(result) > MAX_SLUG_LENGTH {
log.warnf(
"Long slug detected (%d > %d). " +
"This may break in later versions of thor. " +
"slug=%s input=\"%s\"",
len(result),
MAX_SLUG_LENGTH,
result,
text,
)
}
return result
}
make_unique :: proc(slug: string, seen: ^map[string]bool) -> string {
if _, ok := seen^[slug]; !ok {
seen^[slug] = true
return slug
}
n := 1
for {
candidate := fmt.tprintf("%s-%d", slug, n)
if _, ok := seen^[candidate]; !ok {
seen^[candidate] = true
return candidate
}
n += 1
}
return ""
}
markdown/heading_ids_test.odin
-102
View File
@@ -1,102 +0,0 @@
#+test
package markdown
import "core:testing"
@(test)
test_heading_simple :: proc(t: ^testing.T) {
result := inject_heading_ids("<h2>Hello World</h2>")
testing.expect_value(t, result, `<h2 id="hello-world">Hello World</h2>`)
}
@(test)
test_heading_dedup :: proc(t: ^testing.T) {
result := inject_heading_ids("<h2>Intro</h2><p>text</p><h2>Intro</h2>")
testing.expect_value(
t,
result,
`<h2 id="intro">Intro</h2><p>text</p><h2 id="intro-1">Intro</h2>`,
)
}
@(test)
test_heading_nested_html :: proc(t: ^testing.T) {
result := inject_heading_ids("<h3>With <code>code</code></h3>")
testing.expect_value(t, result, `<h3 id="with-code">With <code>code</code></h3>`)
}
@(test)
test_heading_entities :: proc(t: ^testing.T) {
result := inject_heading_ids("<h2>Cats &amp; Dogs</h2>")
testing.expect_value(t, result, `<h2 id="cats-dogs">Cats &amp; Dogs</h2>`)
}
@(test)
test_heading_punctuation :: proc(t: ^testing.T) {
result := inject_heading_ids("<h2>Hello, World!</h2>")
testing.expect_value(t, result, `<h2 id="hello-world">Hello, World!</h2>`)
}
@(test)
test_heading_all_levels :: proc(t: ^testing.T) {
result := inject_heading_ids("<h1>A</h1><h2>B</h2><h3>C</h3><h4>D</h4><h5>E</h5><h6>F</h6>")
testing.expect_value(
t,
result,
`<h1 id="a">A</h1>` +
`<h2 id="b">B</h2>` +
`<h3 id="c">C</h3>` +
`<h4 id="d">D</h4>` +
`<h5 id="e">E</h5>` +
`<h6 id="f">F</h6>`,
)
}
@(test)
test_heading_preserves_text :: proc(t: ^testing.T) {
result := inject_heading_ids("<h2>It is <em>bold</em></h2>")
testing.expect_value(t, result, `<h2 id="it-is-bold">It is <em>bold</em></h2>`)
}
@(test)
test_heading_non_heading_tags :: proc(t: ^testing.T) {
input := "<header>Nav</header><h2>Title</h2><hr>"
result := inject_heading_ids(input)
testing.expect_value(t, result, `<header>Nav</header><h2 id="title">Title</h2><hr>`)
}
@(test)
test_heading_existing_attrs_skipped :: proc(t: ^testing.T) {
input := `<h2 class="foo">Title</h2>`
result := inject_heading_ids(input)
testing.expect_value(t, result, input)
}
@(test)
test_heading_empty :: proc(t: ^testing.T) {
result := inject_heading_ids("<h2></h2><h3></h3>")
testing.expect_value(t, result, `<h2 id="section-1"></h2><h3 id="section-2"></h3>`)
}
@(test)
test_heading_with_surrounding_content :: proc(t: ^testing.T) {
input := "<p>Before</p><h2>Title</h2><p>After</p>"
result := inject_heading_ids(input)
testing.expect_value(t, result, `<p>Before</p><h2 id="title">Title</h2><p>After</p>`)
}
@(test)
test_heading_numbers :: proc(t: ^testing.T) {
result := inject_heading_ids("<h2>Chapter 12</h2>")
testing.expect_value(t, result, `<h2 id="chapter-12">Chapter 12</h2>`)
}
@(test)
test_heading_triple_dedup :: proc(t: ^testing.T) {
result := inject_heading_ids("<h2>Foo</h2><h2>Foo</h2><h2>Foo</h2>")
testing.expect_value(
t,
result,
`<h2 id="foo">Foo</h2>` + `<h2 id="foo-1">Foo</h2>` + `<h2 id="foo-2">Foo</h2>`,
)
}
markdown/highlight.odin
-317
View File
@@ -1,317 +0,0 @@
package markdown
import ts "../treesitter"
import "core:fmt"
import "core:log"
import "core:strings"
Capture :: struct {
start: u32,
end: u32,
name: string,
}
find_first_error_line :: proc(root: ts.Node) -> int {
if ts.node_is_error(root) {
return int(ts.node_start_point(root).row) + 1
}
for i in 0 ..< ts.node_child_count(root) {
child := ts.node_child(root, u32(i))
if ts.node_has_error(child) {
line := find_first_error_line(child)
if line > 0 {
return line
}
}
}
return 0
}
write_span_open :: proc(b: ^strings.Builder, buf: ^[128]u8, name: string) {
pos := 0
prefix := "<span class=\""
for i in 0 ..< len(prefix) {
buf[pos] = prefix[i]
pos += 1
}
first := true
for i in 0 ..< len(name) {
if name[i] == '.' {
if !first {buf[pos] = ' '; pos += 1}
first = false
buf[pos] = 'h'; buf[pos + 1] = 'l'; buf[pos + 2] = '-'; pos += 3
for j in 0 ..< i {
buf[pos] = '-' if name[j] == '.' else name[j]
pos += 1
}
}
}
if !first {buf[pos] = ' '; pos += 1}
buf[pos] = 'h'; buf[pos + 1] = 'l'; buf[pos + 2] = '-'; pos += 3
for j in 0 ..< len(name) {
buf[pos] = '-' if name[j] == '.' else name[j]
pos += 1
}
buf[pos] = '"'; pos += 1
buf[pos] = '>'; pos += 1
strings.write_string(b, string(buf[:pos]))
}
write_escaped :: proc(b: ^strings.Builder, s: string) {
start := 0
for i in 0 ..< len(s) {
switch s[i] {
case '&':
if i > start do strings.write_string(b, s[start:i])
strings.write_string(b, "&amp;")
start = i + 1
case '<':
if i > start do strings.write_string(b, s[start:i])
strings.write_string(b, "&lt;")
start = i + 1
case '>':
if i > start do strings.write_string(b, s[start:i])
strings.write_string(b, "&gt;")
start = i + 1
case '"':
if i > start do strings.write_string(b, s[start:i])
strings.write_string(b, "&quot;")
start = i + 1
}
}
if start == 0 {
strings.write_string(b, s)
} else if start < len(s) {
strings.write_string(b, s[start:])
}
}
// unescape_html returns a caller-owned string (free it with delete). When the
// input has no entities it returns a clone; otherwise it returns the builder's
// buffer. It must NOT destroy the builder — to_string aliases that buffer, so the
// returned string owns it.
unescape_html :: proc(s: string) -> string {
sb := strings.builder_make()
start := 0
for i in 0 ..< len(s) {
if s[i] != '&' do continue
semi := strings.index(s[i:], ";")
if semi < 0 do break
entity := s[i:i + semi + 1]
replacement := ""
switch entity {
case "&amp;":
replacement = "&"
case "&lt;":
replacement = "<"
case "&gt;":
replacement = ">"
case "&quot;":
replacement = "\""
case "&#39;", "&apos;":
replacement = "'"
case:
continue
}
if i > start do strings.write_string(&sb, s[start:i])
strings.write_string(&sb, replacement)
start = i + semi + 1
}
if start == 0 do return strings.clone(s)
if start < len(s) do strings.write_string(&sb, s[start:])
return strings.to_string(sb)
}
highlight_block :: proc(code: string, lang: string, file_path: string) -> string {
g, ok := ts.grammar(lang)
if !ok {
return code
}
parser := ts.open_parser(g)
if parser == nil {
return code
}
defer ts.parser_delete(parser)
raw_code := unescape_html(code)
defer delete(raw_code)
raw_c := strings.clone_to_cstring(raw_code)
defer delete(raw_c)
tree := ts.parser_parse_string(parser, nil, raw_c, u32(len(raw_code)))
if tree == nil {
return code
}
defer ts.tree_delete(tree)
root := ts.tree_root_node(tree)
if ts.node_has_error(root) {
line := find_first_error_line(root)
if line > 0 {
log.warnf(
"highlight: syntax errors in %s code block at line %d (%s)",
lang,
line,
file_path,
)
} else {
log.warnf("highlight: syntax errors in %s code block (%s)", lang, file_path)
}
}
if g.query == nil {
return code
}
cursor := ts.query_cursor_new()
defer ts.query_cursor_delete(cursor)
ts.query_cursor_exec(cursor, g.query, root)
captures := make([dynamic]Capture, 0, 64, context.temp_allocator)
defer delete(captures)
match: ts.Query_Match
capture_idx: u32
for ts.query_cursor_next_capture(cursor, &match, &capture_idx) {
if capture_idx >= u32(match.capture_count) {
continue
}
cap := match.captures[capture_idx]
name_len: u32
name_c := ts.query_capture_name_for_id(g.query, cap.index, &name_len)
if name_c == nil {
continue
}
name_full := string(name_c)
name := name_full
if len(name_full) > int(name_len) {
name = name_full[:int(name_len)]
}
append(
&captures,
Capture {
start = ts.node_start_byte(cap.node),
end = ts.node_end_byte(cap.node),
name = name,
},
)
}
if len(captures) == 0 {
return code
}
sb := strings.builder_make_len(len(code) * 2)
last_pos: u32 = 0
stack := make([dynamic]Capture, 0, 16, context.temp_allocator)
defer delete(stack)
buf: [128]u8
for cap in captures {
for len(stack) > 0 {
top := stack[len(stack) - 1]
if top.end <= cap.start {
if top.end > last_pos {
write_escaped(&sb, raw_code[last_pos:top.end])
}
strings.write_string(&sb, "</span>")
last_pos = top.end
pop(&stack)
} else {
break
}
}
if cap.start > last_pos {
write_escaped(&sb, raw_code[last_pos:cap.start])
last_pos = cap.start
}
write_span_open(&sb, &buf, cap.name)
append(&stack, cap)
}
for len(stack) > 0 {
top := pop(&stack)
if top.end > last_pos {
write_escaped(&sb, raw_code[last_pos:top.end])
}
strings.write_string(&sb, "</span>")
last_pos = top.end
}
if int(last_pos) < len(raw_code) {
write_escaped(&sb, raw_code[last_pos:])
}
return strings.to_string(sb)
}
highlight_code :: proc(html: string, file_path: string) -> string {
PREFIX :: `<pre><code class="language-`
CODE_END :: `</code></pre>`
sb := strings.builder_make()
defer strings.builder_destroy(&sb)
pos := 0
found := false
for {
rel := strings.index(html[pos:], PREFIX)
if rel < 0 {
break
}
found = true
idx := pos + rel
if idx > pos {
strings.write_string(&sb, html[pos:idx])
}
lang_start := idx + len(PREFIX)
lang_end_rel := strings.index(html[lang_start:], `"`)
if lang_end_rel < 0 {
break
}
lang_end := lang_start + lang_end_rel
lang := html[lang_start:lang_end]
code_start := lang_end + 1
if code_start < len(html) && html[code_start] == '>' {
code_start += 1
} else {
pos = lang_end
continue
}
end_rel := strings.index(html[code_start:], CODE_END)
if end_rel < 0 {
break
}
end_idx := code_start + end_rel
code := html[code_start:end_idx]
highlighted := highlight_block(code, lang, file_path)
fmt.sbprintf(&sb, `<pre><code class="language-%s">%s</code></pre>`, lang, highlighted)
pos = end_idx + len(CODE_END)
}
if pos < len(html) && found {
strings.write_string(&sb, html[pos:])
}
if !found {
return html
}
return strings.to_string(sb)
}
markdown/highlight_test.odin
-103
View File
@@ -1,103 +0,0 @@
#+test
package markdown
import ts "../treesitter"
import "core:mem"
import "core:strings"
import "core:testing"
// highlight_block returns the input string unchanged (borrowed) when it can't
// highlight — no grammar, no query, or zero captures — and a freshly-allocated
// string when it does. Free only the latter; identity (raw_data) tells them
// apart regardless of content.
@(private = "file")
free_highlight :: proc(out, code: string) {
if raw_data(out) != raw_data(code) {
delete(out)
}
}
// A builtin grammar (css) highlights into hl- spans, preserving the source text.
@(test)
test_highlight_block_css :: proc(t: ^testing.T) {
code := "a{color:red}"
out := highlight_block(code, "css", "test")
defer free_highlight(out, code)
testing.expect(t, strings.contains(out, "<span class=\"hl-"), "css should be highlighted")
testing.expect(t, strings.contains(out, "</span>"))
testing.expect(t, strings.contains(out, "color"))
testing.expect(t, strings.contains(out, "red"))
}
// The other builtin grammar (html) highlights, and the source's own '<'/'>' are
// HTML-escaped in the output (so they can't be confused with the wrapping spans).
@(test)
test_highlight_block_html_escapes_source :: proc(t: ^testing.T) {
code := "<p>hi</p>"
out := highlight_block(code, "html", "test")
defer free_highlight(out, code)
testing.expect(t, strings.contains(out, "<span class=\"hl-"), "html should be highlighted")
testing.expect(t, strings.contains(out, "hi"))
// The literal angle brackets of the source are escaped in the output.
testing.expect(t, strings.contains(out, "&lt;"))
testing.expect(t, strings.contains(out, "&gt;"))
}
// A language with no available grammar returns the input verbatim (best-effort:
// highlighting is optional). The returned string must be the same buffer, so
// free_highlight leaves it alone.
@(test)
test_highlight_block_unknown_lang_passthrough :: proc(t: ^testing.T) {
code := "fn main() {}"
out := highlight_block(code, "no-such-language", "test")
defer free_highlight(out, code)
testing.expect_value(t, out, code)
testing.expect(t, raw_data(out) == raw_data(code), "passthrough must return the input buffer")
}
// End-to-end: highlight_code finds a fenced code block in rendered HTML and
// rewrites its body with highlight spans while preserving the <pre><code> shell.
// Run under a scratch arena (as production does) so the intermediate allocations
// highlight_code doesn't individually free are reclaimed in one shot.
@(test)
test_highlight_code_wraps_css_block :: proc(t: ^testing.T) {
arena: mem.Dynamic_Arena
mem.dynamic_arena_init(&arena)
defer mem.dynamic_arena_destroy(&arena)
context.allocator = mem.dynamic_arena_allocator(&arena)
html := `<pre><code class="language-css">a{color:red}</code></pre>`
out := highlight_code(html, "test")
testing.expect(t, strings.contains(out, `<pre><code class="language-css">`))
testing.expect(t, strings.contains(out, "</code></pre>"))
testing.expect(
t,
strings.contains(out, "<span class=\"hl-"),
"block body should be highlighted",
)
}
// unescape_html must return a caller-owned allocation the caller frees exactly
// once. The buggy version frees its builder internally (defer builder_destroy)
// and returns the freed alias, so the caller's delete becomes a double-free that
// the test runner's tracking allocator flags.
@(test)
test_unescape_html_entities_owned :: proc(t: ^testing.T) {
got := unescape_html("&lt;a&gt;")
defer delete(got)
testing.expect_value(t, got, "<a>")
}
// Same contract on the no-entity path: the buggy version returns the borrowed
// input, so freeing it is a bad free (pointer not from the allocator).
@(test)
test_unescape_html_passthrough_owned :: proc(t: ^testing.T) {
got := unescape_html("plain text")
defer delete(got)
testing.expect_value(t, got, "plain text")
}
markdown/markdown.odin
-152
View File
@@ -1,152 +0,0 @@
package markdown
import cm "vendor:commonmark"
import "core:encoding/json"
import "core:log"
import "core:strings"
import diags "../diagnostics"
Extension :: enum {
Emoji,
Sidenotes,
Alerts,
Highlight,
Sections,
HeadingIDs,
DefLists,
Footnotes,
}
DEFAULT_EXTENSIONS :: bit_set[Extension]{.Emoji, .Sidenotes, .Alerts, .HeadingIDs, .DefLists}
EXTENSION_NAMES :: []string{
"emoji", "sidenotes", "alerts", "highlight", "sections",
"heading_ids", "deflists", "footnotes",
}
// Caller is responsible for freeing string
process :: proc(
body: string,
ext: bit_set[Extension],
file_path: string,
allocator := context.allocator,
) -> string {
side_notes := make(map[string]string)
margin_notes := make(map[string]string)
clean_body := body
if .Sidenotes in ext || .Footnotes in ext {
clean_body, side_notes, margin_notes = strip_definitions(body)
}
if .DefLists in ext {
clean_body = convert_deflists(clean_body, allocator)
}
original_html := cm.markdown_to_html_from_string(clean_body, {.Unsafe})
html := strings.clone(original_html, allocator)
cm.free_string(original_html)
if .Emoji in ext {
html = expand_emoji(html)
}
if .Footnotes in ext {
html = inject_footnotes(html, side_notes, margin_notes)
} else if .Sidenotes in ext {
html = inject_notes(html, side_notes, margin_notes)
}
if .Alerts in ext {
html = inject_alerts(html)
}
if .Highlight in ext {
html = highlight_code(html, file_path)
}
if .HeadingIDs in ext {
html = inject_heading_ids(html)
}
if .Sections in ext {
html = wrap_sections(html)
}
return html
}
// Convert a ',' separated list of case-insensitive extension names to a bit set.
parse_extension_list :: proc(s: string) -> (result: bit_set[Extension]) {
for part in strings.split(s, ",", allocator = context.temp_allocator) {
name := strings.to_lower(strings.trim_space(part), allocator = context.temp_allocator)
e, ok := extension_from_name(name)
if !ok {
if name != "" {
suggestion := diags.suggest_correction(EXTENSION_NAMES, name)
if suggestion != "" {
log.warnf("unknown extension '%s' (did you mean '%s'?)", name, suggestion)
} else {
log.warnf("unknown extension '%s'", name)
}
}
continue
}
result += {e}
}
return result
}
// Given a map[Extension]bool, apply it to ext.
apply_extension_config :: proc(ext: ^bit_set[Extension], config: json.Object) {
for name, val in config {
enabled := val.(json.Boolean) or_continue
e, ok := extension_from_name(name)
if !ok {
lower := strings.to_lower(name, context.temp_allocator)
suggestion := diags.suggest_correction(EXTENSION_NAMES, lower)
if suggestion != "" {
log.warnf("unknown extension '%s' (did you mean '%s'?)", name, suggestion)
} else {
log.warnf("unknown extension '%s'", name)
}
continue
}
if enabled {
ext^ += {e}
} else {
ext^ -= {e}
}
}
}
extension_from_name :: proc(name: string) -> (e: Extension, ok: bool) {
switch name {
case "emoji":
e = .Emoji
ok = true
case "sidenotes":
e = .Sidenotes
case "alerts":
e = .Alerts
case "highlight":
e = .Highlight
case "sections":
e = .Sections
case "heading_ids":
e = .HeadingIDs
case "deflists":
e = .DefLists
case "footnotes":
e = .Footnotes
case:
// Do nothing
}
return e, ok || e != .Emoji
}
// resolve_extension_conflicts resolves mutually exclusive extensions.
// Footnotes and Sidenotes share the same [^id] syntax but render differently;
// if both are enabled (e.g. from defaults + CLI), Footnotes wins.
resolve_extension_conflicts :: proc(ext: ^bit_set[Extension]) {
if .Footnotes in ext^ && .Sidenotes in ext^ {
ext^ -= {.Sidenotes}
}
}
markdown/sectionate.odin
-45
View File
@@ -1,45 +0,0 @@
package markdown
import "core:strings"
wrap_sections :: proc(html: string) -> string {
H2 :: "<h2"
sb := strings.builder_make()
defer strings.builder_destroy(&sb)
pos := 0
search_pos := 0
found := false
for {
rel := strings.index(html[search_pos:], H2)
if rel < 0 {
break
}
idx := search_pos + rel
found = true
if idx > pos {
strings.write_string(&sb, "<section>")
strings.write_string(&sb, html[pos:idx])
strings.write_string(&sb, "</section>")
}
pos = idx
search_pos = idx + len(H2)
}
if pos < len(html) {
strings.write_string(&sb, "<section>")
strings.write_string(&sb, html[pos:])
strings.write_string(&sb, "</section>")
found = true
}
if found {
return strings.to_string(sb)
} else {
return html
}
}
markdown/sectionate_test.odin
-48
View File
@@ -1,48 +0,0 @@
#+test
package markdown
import "core:testing"
@(test)
test_wrap_sections_works :: proc(t: ^testing.T) {
testing.expect_value(
t,
wrap_sections("<p>intro</p><h2>Title</h2><p>body</p>"),
"<section><p>intro</p></section><section><h2>Title</h2><p>body</p></section>",
)
testing.expect_value(
t,
wrap_sections("<p>a</p><h2>A</h2><p>b</p><h2>B</h2><p>c</p>"),
"<section><p>a</p></section><section><h2>A</h2><p>b</p></section><section><h2>B</h2><p>c</p></section>",
)
testing.expect_value(
t,
wrap_sections("<h2>Title</h2><p>body</p>"),
"<section><h2>Title</h2><p>body</p></section>",
)
testing.expect_value(
t,
wrap_sections("<p>intro</p><h2>Title</h2>"),
"<section><p>intro</p></section><section><h2>Title</h2></section>",
)
}
@(test)
test_wrap_sections_doesnt_split_content :: proc(t: ^testing.T) {
testing.expect_value(
t,
wrap_sections("<p>just text</p>"),
"<section><p>just text</p></section>",
)
testing.expect_value(t, wrap_sections(""), "")
testing.expect_value(
t,
wrap_sections("<h1>Big</h1><h3>Small</h3>"),
"<section><h1>Big</h1><h3>Small</h3></section>",
)
}
markdown/testdata/deflists/01-basic.md
Vendored
-2
View File
@@ -1,2 +0,0 @@
Term
: Definition content
markdown/testdata/deflists/02-multiple-defs.md
Vendored
-7
View File
@@ -1,7 +0,0 @@
Markdown
: A lightweight markup language
: Created by John Gruber in 2004
HTML
: A standard markup language used to create web pages
: Used to create web pages
markdown/testdata/deflists/03-multiline-def.md
Vendored
-5
View File
@@ -1,5 +0,0 @@
Term
: This is the first line of definition content
This is the second line, needs at least one space indentation
This is a new paragraph, needs at least one space indentation and a preceding blank line
markdown/testdata/deflists/04-markdown-in-def.md
Vendored
-7
View File
@@ -1,7 +0,0 @@
Markdown Syntax
: **Markdown** supports multiple text formatting:
- *Italic* and **Bold**
- [Link](https://www.markdownlang.com)
- `Inline code`
- > Block quote
- Lists and other elements
markdown/testdata/deflists/05-nested.md
Vendored
-6
View File
@@ -1,6 +0,0 @@
Outer Term
: Outer Definition
Inner Term
: Inner Definition
: Another Inner Definition
markdown/testdata/deflists/06-compact.md
Vendored
-1
View File
@@ -1 +0,0 @@
Term: Definition
markdown/testdata/deflists/07-extra-space.md
Vendored
-2
View File
@@ -1,2 +0,0 @@
Term
: Definition
markdown/testdata/deflists/08-glossary.md
Vendored
-11
View File
@@ -1,11 +0,0 @@
API
: An application programming interface, a set of rules that allow different software applications to communicate with each other.
Framework
: A software library that provides a standardized structure for application development.
Git
: A distributed version control system used to track changes in a project development process.
IDE
: An integrated development environment, a software application that includes a code editor and various development tools.
markdown/testdata/deflists/09-api-params.md
Vendored
-14
View File
@@ -1,14 +0,0 @@
user_id
: **Required** - The unique identifier for the user.
: Type: `integer`
name
: **Optional** - The display name for the user.
: Type: `string`
: Default: `null`
status
: **Optional** - The user status.
: Type: `string`
: Allowed values: `active`, `inactive`, `suspended`
: Default: `active`
markdown/testdata/deflists/expected/01-basic.html
Vendored
-1
View File
@@ -1 +0,0 @@
<dl><dt>Term</dt><dd>Definition content</dd></dl>
markdown/testdata/deflists/expected/02-multiple-defs.html
Vendored
-1
View File
@@ -1 +0,0 @@
<dl><dt>Markdown</dt><dd>A lightweight markup language</dd><dd>Created by John Gruber in 2004</dd><dt>HTML</dt><dd>A standard markup language used to create web pages</dd><dd>Used to create web pages</dd></dl>
markdown/testdata/deflists/expected/03-multiline-def.html
Vendored
-3
View File
@@ -1,3 +0,0 @@
<dl><dt>Term</dt><dd>This is the first line of definition content</dd></dl> This is the second line, needs at least one space indentation
This is a new paragraph, needs at least one space indentation and a preceding blank line
markdown/testdata/deflists/expected/04-markdown-in-def.html
Vendored
-5
View File
@@ -1,5 +0,0 @@
<dl><dt>Markdown Syntax</dt><dd><strong>Markdown</strong> supports multiple text formatting:</dd></dl> - *Italic* and **Bold**
- [Link](https://www.markdownlang.com)
- `Inline code`
- > Block quote
- Lists and other elements
markdown/testdata/deflists/expected/05-nested.html
Vendored
-1
View File
@@ -1 +0,0 @@
<dl><dt>Outer Term</dt><dd>Outer Definition</dd><dt>Inner Term</dt><dd>Inner Definition</dd><dd>Another Inner Definition</dd></dl>
markdown/testdata/deflists/expected/06-compact.html
Vendored
-1
View File
@@ -1 +0,0 @@
Term: Definition
markdown/testdata/deflists/expected/07-extra-space.html
Vendored
-1
View File
@@ -1 +0,0 @@
<dl><dt>Term</dt><dd>Definition</dd></dl>
markdown/testdata/deflists/expected/08-glossary.html
Vendored
-1
View File
@@ -1 +0,0 @@
<dl><dt>API</dt><dd>An application programming interface, a set of rules that allow different software applications to communicate with each other.</dd><dt>Framework</dt><dd>A software library that provides a standardized structure for application development.</dd><dt>Git</dt><dd>A distributed version control system used to track changes in a project development process.</dd><dt>IDE</dt><dd>An integrated development environment, a software application that includes a code editor and various development tools.</dd></dl>
markdown/testdata/deflists/expected/09-api-params.html
Vendored
-1
View File
@@ -1 +0,0 @@
<dl><dt>user_id</dt><dd><strong>Required</strong> - The unique identifier for the user.</dd><dd>Type: <code>integer</code></dd><dt>name</dt><dd><strong>Optional</strong> - The display name for the user.</dd><dd>Type: <code>string</code></dd><dd>Default: <code>null</code></dd><dt>status</dt><dd><strong>Optional</strong> - The user status.</dd><dd>Type: <code>string</code></dd><dd>Allowed values: <code>active</code>, <code>inactive</code>, <code>suspended</code></dd><dd>Default: <code>active</code></dd></dl>
markdown/testdata/deflists/reference/COMPARISON.md
Vendored
-60
View File
@@ -1,60 +0,0 @@
# 3-way behavior comparison: ours vs Pandoc vs Hugo
Captured for later review (e.g. if we ever pursue "Option B" — full CommonMark
block semantics inside definitions). Compares `convert_deflists` output (Option A,
inline/flat) against Pandoc (`-f markdown -t html5`) and Hugo/Goldmark for the
four fixtures where behavior is interesting. Raw renderings live in `pandoc/` and
`hugo/`; ours is in `../expected/`. Whitespace/formatting differences ignored
below — only structural differences are described.
## 03-multiline-def — definition with a continuation line + a second paragraph
*How much multi-line content belongs to the `<dd>`?*
- **Ours:** only the **first line** goes in the `<dd>`. The continuation line and
the second paragraph fall out of the `<dl>` as **raw, unrendered markdown**.
- **Pandoc:** folds the continuation line **into** the `<dd>` (lazy continuation),
then **ejects** the second paragraph as a sibling `<p>` after `</dl>`.
- **Hugo:** keeps **everything inside** the `<dd>` — continuation line as text,
second paragraph as a nested `<p>`.
- All three differ. We capture the least, Pandoc the middle, Hugo the most.
## 04-markdown-in-def — definition containing a bullet list (with a blockquote)
*What happens to the nested list?*
- **Ours:** intro line becomes the `<dd>`; the **entire bullet list falls out as
raw markdown** (unrendered `- …` lines after `</dl>`).
- **Pandoc:** bullets **flattened into the `<dd>`** as inline text — no `<ul>`, the
`-` become literal dashes.
- **Hugo:** bullets become a **real nested `<ul>`/`<li>`** (blockquote and all)
inside the `<dd>`.
- All three differ; same shape as 03. We drop it, Pandoc absorbs-but-flattens,
Hugo builds true structure.
## 05-nested — a definition list nested under another definition
*Nest or flatten?*
- **Ours:** **flatten** — one flat `<dl>`; "Inner Term" is just another `<dt>`.
- **Pandoc:** **flatten** — structurally **identical to ours**.
- **Hugo:** **nests** a real inner `<dl>` inside the outer `<dd>`.
- Ours == Pandoc; Hugo is the odd one out.
## 07-extra-space — term then a 2-space-indented `: Definition`
*Is a 2-space colon a definition marker?*
- **Ours:** **yes** → clean `<dl>`.
- **Pandoc:** **yes** → structurally **identical to ours**.
- **Hugo:** **no** → rejects it, emits a plain `<p>`.
- Ours == Pandoc; Hugo is the odd one out.
## Pattern
Two clean groups:
- **Marker / flattening decisions (05, 07):** we land **exactly on Pandoc**. These
are the calls our CommonMark-marker + flat-inline design makes cleanly; Pandoc
agrees, Hugo diverges (nests / stricter marker).
- **Block-content-in-a-definition (03, 04):** we diverge from **both** tools, doing
**less** — we emit the `<dl>` for the term/first line and let the rest fall
through as raw markdown, rather than absorbing it into the `<dd>` (flattened like
Pandoc, or structured like Hugo).
In short: our behavior is **"Pandoc, minus the ability to pull block content into
a `<dd>`."** Single inline-line definitions (01, 02, 05, 07, 08, 09) match a real
engine; genuinely multi-block definitions (03, 04) stop at the first line and pass
the remainder through. Closing that gap is what Option B would entail.
markdown/testdata/deflists/reference/README.md
Vendored
-44
View File
@@ -1,44 +0,0 @@
# Definition-list test fixtures
Inputs (`../NN-name.md`) are the definition-list examples from
<https://www.markdownlang.com/extended/definition-lists.html>.
`pandoc/` and `hugo/` hold each input rendered by Pandoc (`-f markdown -t html5`)
and Hugo (Goldmark with `markup.goldmark.extensions.definitionList = true`).
They are retained as reference for a possible future "Option B" (full CommonMark
block semantics inside definitions). They are NOT the test oracle for the current
implementation.
`../expected/NN.html` is the oracle for `convert_deflists`: our inline fragment
format (no inter-tag newlines, no trailing newline), derived from the trusted
tool per the decisions below.
## Decisions (Option A — simple / inline)
Marker whitespace follows CommonMark: `:` with 0–3 spaces of indent is a
definition; 4+ (tab = 4) is a code block. Definition bodies are inline/flat.
Every fixture has a concrete exact-match oracle in `../expected/NN.html`.
| Fixture | Pandoc | Hugo | Decision | expected/ source |
|---------|--------|------|----------|------------------|
| 01-basic | `<dl>` | `<dl>` | support | hugo (agreed) |
| 02-multiple-defs | multi `<dd>` | multi `<dd>` | support | hugo (agreed) |
| 06-compact (`Term: Definition`) | `<p>` | `<p>` | NOT a deflist → passthrough | raw input |
| 07-extra-space (2-space `:`) | `<dl>` | `<p>` | support (accept, CommonMark ≤3-space rule) | hand-written |
| 08-glossary | `<dl>` | `<dl>` | support | hugo (agreed) |
| 09-api-params | multi `<dd>` | multi `<dd>` | support | hugo (agreed) |
| 05-nested | flattens | nested `<dl>` | flatten (matches Pandoc) | snapshot (== pandoc) |
| 03-multiline-def | ejects 2nd para | nests 2nd para in `<dd>` | degrade | snapshot (current output) |
| 04-markdown-in-def | flattens list | nested `<ul>`/`<blockquote>` | degrade | snapshot (current output) |
A detailed 3-way behavior comparison (ours vs Pandoc vs Hugo) for the diverging
fixtures is in `COMPARISON.md`, kept for Option-B review.
03/04/05 require block-level content inside `<dd>` (multi-paragraph, nested
lists/blockquotes, nested `<dl>`), which the inline preprocessor does not
produce. 05 flattens exactly as Pandoc does, so its snapshot equals Pandoc's
output. 03/04 emit a `<dl>` for the term and first line, then let the remaining
block content fall through as markdown; their `expected/` files snapshot that
current degraded behavior as a regression guard, not an endorsed rendering.
Revisit via Option B (full CommonMark block semantics) if real content needs it.
markdown/testdata/deflists/reference/hugo/01-basic.html
Vendored
-4
View File
@@ -1,4 +0,0 @@
<dl>
<dt>Term</dt>
<dd>Definition content</dd>
</dl>
markdown/testdata/deflists/reference/hugo/02-multiple-defs.html
Vendored
-8
View File
@@ -1,8 +0,0 @@
<dl>
<dt>Markdown</dt>
<dd>A lightweight markup language</dd>
<dd>Created by John Gruber in 2004</dd>
<dt>HTML</dt>
<dd>A standard markup language used to create web pages</dd>
<dd>Used to create web pages</dd>
</dl>
markdown/testdata/deflists/reference/hugo/03-multiline-def.html
Vendored
-7
View File
@@ -1,7 +0,0 @@
<dl>
<dt>Term</dt>
<dd>This is the first line of definition content
This is the second line, needs at least one space indentation
<p>This is a new paragraph, needs at least one space indentation and a preceding blank line</p>
</dd>
</dl>

Some files were not shown because too many files have changed in this diff Show More