Deep dives on Katalyst Documentation

Vision and scope

Mon, 01 Jan 0001 00:00:00 +0000

Vision and scope#

Traditional data management often forces teams into binary choices: structured or unstructured, rigid or chaotic. Katalyst is an experimental framework aimed at enabling fast, low-risk evolution through progressive typing in the storage layer.

Database management is risky and rigid#

Backend architecture and database changes are frequently high-risk operations. Teams therefore wrap data systems in heavy governance for access control, schema changes, and migrations. Those controls are necessary, but the cost can make change rare. Rare change creates rigidity.

Progressive operations

Mon, 01 Jan 0001 00:00:00 +0000

Progressive Operations#

How storage backends evolve as query complexity grows. Each tier unlocks new operations, but requires structural commitments the previous tier doesn’t.

Structuredness comes down to which operations a backend supports, and schemas and checks are the means: enforcing checks is what makes new operations available. The core thesis follows: many knowledge systems start as filesystems and progressively acquire database-like structure. The progression isn’t arbitrary, each tier is driven by a class of operations that can’t be satisfied at the previous level.

Core concepts

Mon, 01 Jan 0001 00:00:00 +0000

Core concepts#

Status: work in progress. A deliberately abstract sketch. These concepts are not about katalyst specifically; katalyst is one instantiation among many. Expect revisions until they settle.

The vocabulary katalyst reasons in, general enough to describe a Postgres table, a directory of markdown files, and a MongoDB collection the same way, so the abstractions built on top bridge them too. Each term’s canonical definition lives in the glossary; this page introduces the concepts and how they fit. For the katalyst-specific instantiation, see the domain model.

Domain model

Mon, 01 Jan 0001 00:00:00 +0000

Domain model#

What katalyst is about: the concepts it manipulates and how they relate. This is the conceptual map and the entry point to the subsystem deep-dives - each piece is summarized here and documented in full on its own page.

This page is the katalyst-specific map; core concepts is the same map at the general, tool-agnostic altitude (the same vocabulary applied to a Postgres table or a MongoDB collection). For what the commands do, see the getting-started tutorial and the configuration reference.

Storage layer

Mon, 01 Jan 0001 00:00:00 +0000

Storage layer#

Status: partly shipped. The seam and the config model exist (internal/storage, storage instances under .katalyst/storage/); the richer mapping (multi-coordinate templates, inferred mode, non-filesystem backends) is still ahead. This page describes the whole arc, and notes what is built versus planned.

What the storage layer is#

The storage layer is the two-way mapping between a backend store and the Katalyst domain model. It answers: what collections and items does this store contain, and where does each one live?, in both directions. It is Katalyst’s realization of the general storage concept from core concepts: the filesystem is one backend; SQLite, directories of CSVs, S3 buckets, and hosted APIs are others. The first real stress test is SQLite, because it is the first backend that forces the granularity question below.

Collections

Mon, 01 Jan 0001 00:00:00 +0000

Collections#

The internal/project loader (loader.go) is the orchestration hub: it loads a project’s .katalyst/ directory, resolves named schemas, and assembles storage instances and their collections (each object type parses its own config — the storage registry validates a declared type, and a collection parses its own block in storage/collection). It decides which schema applies to a given item, and the check lifecycle is driven from here. This page is the model and the why; for the key-by-key surface see the configuration reference.

Checks

Mon, 01 Jan 0001 00:00:00 +0000

Checks#

A check asserts one condition on an item and reports a violation when the item fails it. The check engine turns a collection’s configuration into a verdict on each item: it resolves the checks that apply, runs them, and collects their violations. This page explains the model the engine is built on, how check libraries supply and run checks, and why the pieces are shaped the way they are. For the per-type catalog see the check types reference; for the end-to-end data flow of one check invocation see the domain model.

Inspectors

Mon, 01 Jan 0001 00:00:00 +0000

Inspectors#

An inspector profiles content and returns evidence: counts and distributions, never recommendations. Inspectors are the descriptive dual of checks - a check asserts a predicate and reports violations; an inspector reports the distribution that predicate would be tested against. They drive the inspect command. For the per-inspector catalog see the inspectors reference; this page is the model and the rationale behind it.

Two layers#

Inspectors come in two layers, distinguished by how they reference the data:

Frontmatter and fix

Mon, 01 Jan 0001 00:00:00 +0000

Frontmatter and fix#

How Katalyst parses a markdown file’s frontmatter, the in-memory document that produces, and why fix rewrites that frontmatter the opinionated way it does. The codec (parse and encode) lives in internal/codec/markdownbodytext; the fix transform that drives the canonical form, and the backend write that persists it, live in internal/fix and internal/storage/collection/filesystem respectively.

The markdown document#

The unit of work is a file on disk with two optional regions: