spencer/thor
1
0
Fork 0
mirror of https://github.com/sbrow/thor.git synced 2026-10-25 10:11:55 -04:00
Code Issues Packages Projects Releases Wiki Activity
Files
83e488965e564534a43eb3f2e3fde7c698d4e35a
thor/ASSET_PIPELINE_RESEARCH.md
T
Spencer Brower 5809eac195 chore: Added notes.
2026-08-30 15:25:41 -04:00

16 KiB
Raw Blame History

Asset Pipeline Research: Bring-Your-Own CSS Preprocessor

Research and design notes for letting thor users bring their own CSS toolchain (Sass, PostCSS/Tailwind, Less, Lightning CSS, …) instead of thor embedding or tracking any of them.

Problem statement

The CSS pre-processor landscape is large and moving (Tailwind, PostCSS, Less, Sass, Lightning CSS, …). thor is a ~1.9 MB self-contained binary that tracks its own size (see AUDITING_BINARY_SIZE.md); bundling any one preprocessor — let alone keeping up with all of them — is a non-starter on both size and maintenance grounds. So "bring your own" has to mean thor orchestrates the tool the user already installed, never reimplements it.

Today copy_assets_dir (assets.odin) copies everything under assets/ to the output dir and optionally minifies .css via the tree-sitter pass in minify.odin. There is no preprocessing seam.

What we are NOT doing, and why

  • Bundle a specific preprocessor — rejected: binary-size-hostile, and picks winners in a space we said we can't keep up with.
  • Hugo Pipes-style template resource pipeline (resources.Get | css.Sass | minify | fingerprint) — powerful, but needs a resource abstraction and a non-logic-less template language to host the chained syntax. thor's engine is logic-less mustache; wrong fit.
  • Curated shell-out for named tools only (what Hugo actually ships — only css.Sass and css.PostCSS shell out) — this just re-picks winners.

The chosen direction reuses thor's existing mustache pipe grammar (the same mechanism as the rel_url pipe) to express asset transforms declaratively, where the asset is referenced.

Correcting the record on Hugo

An earlier draft attributed a generic "pre/post-build shell hook" model to Hugo. That is wrong. Hugo has no generic build hooks. Its CSS handling is Hugo Pipes — a template-invoked resource pipeline of fixed, named functions (css.Sass, css.PostCSS, minify, fingerprint, resources.Concat, …). Only two of those shell out: css.PostCSS (to a local postcss-cli under Node) and css.Sass (embedded LibSass, or a dart-sass binary). Generic "run any command pre/post build" is closer to npm-scripts / Makefile glue, or Eleventy's before/after events (which run JavaScript, not shell commands). No mainstream SSG markets a generic command hook as its asset story.

Tool CLI research

Five representative tools were researched for their exact shell-out contract. The questions that determine thor's interface are the same for each: stdin/stdout vs. file-path args, whether a config file is mandatory, whether it needs to scan rendered HTML, watch support, minify overlap, and runtime dependencies.

Mechanical matrix

Tool stdin→stdout Config file Self-contained (no Node) Self-minifies Errors
Dart Sass yes (sass -) none yes — standalone binary --style=compressed stderr, exit 65
PostCSS (postcss-cli) yes (< in > out) optional (--use) no — needs Node + node_modules via cssnano plugin stderr, exit 1
lessc yes (- / positional) none no — needs Node weak/deprecated (-x) stderr, nonzero
Lightning CSS file-in, stdout-out none (flags) yes — Rust binary (npm-distributed) --minify (its core purpose) stderr, nonzero
Tailwind v4 yes (-i - -o -) none (CSS-first) yes — standalone binary (Bun) --minify stderr, nonzero
Tailwind v3 no (needs -i file) tailwind.config.js yes — standalone (pkg/Node embed) --minify stderr, nonzero

Two properties are uniform and load-bearing:

  • Every tool writes errors to stderr and returns nonzero on failure. thor can gate on exit code and surface stderr through a single code path for all tools.
  • Every tool can stream to stdout. Differences (Lightning CSS wants a path not stdin; lessc's output is positional; Tailwind v3 needs an input file) are all absorbable by a per-tool command template with placeholders.

Per-tool detail

Dart Sass — sass input.scss output.css, or sass - for stdin, prints to stdout when no output arg. No config file exists or is supported; configured by CLI flags + @use/@import in source. Load paths via --load-path/-I. --style=compressed minifies (overlaps thor's minifier). Source maps on by default (--no-source-map to disable). Exit 64 = usage error, 65 = compile error. Distribution: standalone release bundles the Dart VM + snapshot, no Node; the npm sass package is dart2js and needs Node. Prefer the standalone.

PostCSS (postcss-cli, binary postcss) — positional input files or stdin; stdout is the default when no -o/--dir/--replace. Config (postcss.config.js / .postcssrc, cosmiconfig, walks up) is not required — plugins can be given with -u/--use (e.g. postcss in.css -u autoprefixer -u cssnano -o out.css). Zero plugins = near pass-through. postcss is only a plugin host — autoprefixer, tailwind, cssnano are separate npm packages resolved from node_modules. -w/--watch tracks the full @import dependency graph via PostCSS dependency messages. Exit 1 on error, errors to stderr, stdout stays clean for piping. Hard Node + node_modules dependency. Invoke via npx postcss to resolve local .bin.

lessc — lessc [opts] <source> [destination]; - reads stdin; stdout when destination omitted (no -o flag — output is positional). No config file; .less source + flags. --include-path for import search. -x/--compress is deprecated/weak — let thor's minifier compress. No CLI watch mode (watch only exists in the browser runtime). Errors to stderr, nonzero on compile error. Hard Node dependency (bin/lessc is a Node script).

Lightning CSS (lightningcss) — input file positional, -o/--output or stdout when omitted. No config, flag-driven. --minify is the tool's core purpose (directly overlaps thor's minifier — pick one). --bundle inlines @imports; --targets "<browserslist>" downlevels modern CSS. No built-in watch. Errors to stderr, nonzero. Runtime: self-contained Rust binary, no Node to run — but distribution is npm-only (lightningcss-cli pulls a platform package like lightningcss-cli-linux-x64-gnu); no official GitHub/Homebrew/Winget binaries. Can extract the binary from the npm registry tarball and run it standalone.

Tailwind CSS standalone CLI (v3 and v4) — both ship a self-contained per-platform binary (./tailwindcss), no Node required (v3 via Vercel pkg, v4 via Bun). -i/--input, -o/--output (defaults to stdout in v4). v4 reads stdin with -i -; v3 effectively needs an input file. Input is a CSS entry file: v3 uses @tailwind base/components/utilities;, v4 uses @import "tailwindcss";. Config: v3 uses tailwind.config.js (auto-discovered, -c to override) and in practice needs it (or --content) to know what to scan; v4 is CSS-first and genuinely zero-config. --minify in both. Errors to stderr, nonzero (codes not formally enumerated — check exit != 0).

Content scanning (the critical Tailwind property): Tailwind scans source/template files as plain text for complete, static class-name tokens. It does not parse code and does not strictly require rendered HTML — it scans whatever files you point it at. But for an SSG the safe strategy is to scan the generated output after render, so classes that only appear post-render aren't missed. v3: content: ["./public/**/*.html"] or --content. v4: automatic detection, but it ignores .gitignored paths — so a git-ignored public/ must be added explicitly with @source "../public".

Design conclusions

The one split that matters: two tool shapes

The research draws a hard line between two categories that need different integration surfaces:

  1. Per-asset transformers — Sass, Less, Lightning CSS, PostCSS-as-compiler. Input = one source file → output = one CSS file. These fit a mustache pipe: {{ "css/main.scss" | tool sass }}. Each tool's CLI quirk (stdin vs. path, positional output, etc.) is absorbed by a per-tool command template.

  2. Whole-output scanners — Tailwind, alone. Its "input" is not one asset; it is an entry CSS file plus every rendered HTML file it scans. At the moment a per-page pipe would fire, the full output does not exist yet. Tailwind must run once, after all HTML is written, pointed at output_dir. That is a post-render step, categorically not a pipe.

The chicken-and-egg, and the fix

A <link> wants a fingerprinted (cache-busted) URL, but for Tailwind the CSS isn't produced until after render — so at render time the template cannot know the hash. Hugo dodges this with a lazy multi-pass resource graph; thor is single-pass.

Resolution — emit a placeholder at render, resolve at the end. A pipe like {{ "css/main.scss" | tool sass | fingerprint }} writes a stable token (e.g. ⟦asset:css/main.scss⟧) into the HTML. After render, thor runs the tools, populates a content-addressed asset registry (logical name → final hashed URL), then does one cheap string-replace pass over the output swapping tokens → real URLs.

This single mechanism dissolves every constraint encountered:

  • Cache-busting: the hash is known at resolve time, so the token becomes /css/main.a1b2c3.css uniformly, even for tools that rename output themselves.
  • Hash-after-minify: resolution is the last step, so the hash is always of the exact final served bytes.
  • Same asset → same hash across N pages: the registry is computed once; every token resolves identically. (This is also the memoization that keeps a per-page pipe from shelling out N times for the same file.)
  • Tailwind: its post-render output lands in the registry like any other asset; tokens that referenced it resolve correctly despite the CSS not existing at render time.

Cache-busting: two regimes

  • Regime A — thor owns the hash (recommended, and what every researched compiler needs). Tool → stdout → thor minifies → thor hashes the final bytes → writes main.<hash>.css → registers the URL. None of Sass/PostCSS/Less/ Lightning CSS/Tailwind fingerprint on their own; fingerprinting is a bundler/plugin behavior. thor stays in full control of the name because it writes the file. Mirrors Hugo's | fingerprint → .RelPermalink.
  • Regime B — the tool owns the hash (bundlers: esbuild/vite/parcel, or PostCSS + postcss-hash). The tool renames output to a name thor can't predict, and stdout capture doesn't reveal it. The standard answer these tools already provide is a manifest (manifest.json mapping "main.css" → "main.a1b2c3.css"). Supporting this class means thor ingests that manifest into the same registry rather than scanning for files.

Both regimes converge on the same abstraction: a build-scoped, content-addressed asset registry (logical name → final emitted URL), populated by thor's own transform+fingerprint or by an external tool's manifest, and resolved through the placeholder pass.

Proposed interface shape

  • thor.json tools table — one command template per tool. Keeps shell commands out of templates (a single trust surface), and absorbs each CLI's quirks. $in = source path; stdout is captured:

    "tools": {
  "sass":         "sass --no-source-map $in",
  "postcss":      "postcss -u autoprefixer $in",
  "lightningcss": "lightningcss --minify $in",
  "tailwind":     { "run": "tailwindcss -i $in -o -", "scan": "output" }
}

    The scan: output marker flips a tool from per-asset (pipe) to post-render (whole-output) mode.

  • Pipe ops — tool <name> (transform → asset handle), minify, fingerprint — all resolving through the registry, chainable with the existing rel_url:

    <link href="{{ "css/main.scss" | tool sass | minify | fingerprint | rel_url }}">

    A lookup-direction op resolves a logical name to its final URL: {{ "css/main.css" | asset }} → /css/main.a1b2c3.css.

  • Final resolve pass over output_dir replacing asset tokens with registry URLs. This is also where post-render scanners (Tailwind) have already run.

Fit with the existing mustache pipe grammar

{{ "css/main.css" | tool postcss }} already parses under mustache/pipes.odin: the key is a quoted literal (tokenize_fields keeps the quotes so a literal is distinguishable from a context key), tool is the op, postcss is one arg (under MAX_PIPE_ARGS = 2). Adding an op is the documented one-liner path — a variant on Pipe_Op plus a case in apply_filter. The only architectural departure from existing ops (all pure value→value string transforms) is that tool/fingerprint have side effects (subprocess, file write) and must be backed by the registry — which is why the placeholder + deferred-resolve model matters: it keeps render itself side-effect-light and pushes the real work to a defined post-render phase.

Cross-cutting decisions

  1. Node dependency is not thor's problem. PostCSS and lessc require Node + node_modules; Sass/Lightning CSS/Tailwind ship self-contained binaries. thor's job is to exec whatever is on PATH and fail gracefully with the tool's stderr if it's absent — never bundle, never assume a runtime. The Nix flake makes pinning the toolchain pleasant for users who want reproducibility.
  2. Minification must be explicit, never doubled. Every one of these tools can self-minify, and thor has its own minifier. Make minify an explicit pipe step and do not auto-apply thor's global minify to tool-produced assets — otherwise | tool lightningcss (already minified) gets re-minified. The pipe chain is the single source of truth for whether/when an asset is minified.
  3. thor drives the watch; ignore the tools' own --watch. thor already polls (5 s). Invoke each tool once per build as a pipeline step. Sass/PostCSS/ Tailwind have watch modes, but running them would put a second watcher in competition with thor's loop — avoid it.

Headline

The feature is not "a tool pipe." It is a content-addressed asset registry with deferred token resolution, exposed through two pipe ops (tool, fingerprint/asset) plus one post-render hook (for whole-output scanners like Tailwind). That is the minimum that handles both tool shapes and cache-busting without introducing a multi-pass render engine.

Open questions for the design phase

  • Placeholder token format and collision-safety (must survive HTML minification untouched — likely treat like the pre/code preserve ranges in minify.odin).
  • Registry lifetime and keying (content hash vs. path+mtime) and how it threads through the watch loop for incremental rebuilds.
  • Whether fingerprint is terminal-only, or the resolve pass always hashes post-minify regardless of chain position.
  • Manifest ingestion format for Regime B (adopt Vite/webpack manifest.json shape?).
  • Error policy: hard-fail the build vs. warn-and-skip when a tool exits nonzero or is missing from PATH (probably configurable, defaulting to fail).

Sources

Reference in New Issue View Git Blame Copy Permalink