tCMS Design =========== This document describes how tCMS is put together and, more importantly, *why* -- so that you can tell at a glance where your change belongs. Read the [Readme](Readme.md) first for installation and the feature-by-feature tour. This is the architecture behind it. What tCMS is for ================ **Disintermediation from aggregators.** Publishing something today usually means publishing it on somebody else's property. The aggregator owns the URL, the audience, the ranking, the terms, and the right to change any of them without telling you. Your content is the product being sold, and you are renting access to people who already agreed to hear from you. tCMS exists so that the thing you publish lives at *your* domain, in *your* database, under *your* rules, and so that standing that up is a day's work rather than a project. The obstacle has never been "can I host a blog". It is that real sites are not blogs. A client wants invoices, or a photo gallery with EXIF, or a list of virtual machines with a start button, or a recipe index with cook times, or a paywalled newsletter, or all of those on one site. Every CMS handles the first one, and every CMS makes the second one a plugin project. So the design goal is narrower and more useful than "a CMS": > **Make a new content type cheap enough that building the client the thing they > actually asked for is faster than talking them out of it.** That is the bet the whole architecture is placed on. If you are a Perl contractor holding a statement of work, the question you should be able to answer in minutes is "what does a `Widget` post look like, and where does the Widget page come from" -- and the answer should be a template, a small JSON file, and nothing else. The one idea ============ **Everything is a post.** Not "everything is a post, plus pages, plus users, plus categories, plus settings". Everything: | The thing | Is a post | |---|---| | A blog entry | tagged with its series, `form` is `blog.tx` | | A page | a post whose `local_href` is where you want it | | A user profile | a post tagged `about`; the user *is* their profile page | | A category / section | a "series" post, which owns an ACL and names its children's type | | A virtual machine | a post, built on demand by a datasource, never stored | A post is a JSON object. It has a handful of fields every post has -- `id`, `title`, `data`, `tags`, `created`, `version`, `visibility`, `local_href`, `callback` -- and then whatever else its type says it has. Two consequences fall out of that, and most of the system is downstream of them. **The routing table is data, not code.** Every post carries a `local_href` and a `callback`. At startup, `_routes` in TCMS.pm asks the data model for every post and builds the routing table out of them, then merges the static routes from `Trog::Routes::HTML`, `Trog::Routes::JSON` and the active theme over the top. Publishing a post creates a URL because the URL *is* a field on the post. There is no separate router config to keep in sync, and no rewrite rules. A key of the table is a path, or an OpenAPI path template such as `/sitemap/{map}.xml` or `/lib/{module+}`. `TCMS::build_routes` turns the table into the OpenAPI 3 document that tPSGI routes by. Its flags, such as `auth` and `totp_exempt`, reach the callback as `x-tcms-auth` and so on in `$query->{dispatcher}`. The path of a post cannot hold a `{` or a `}`, because OpenAPI has no way to escape them. What a request may send is declared, and tPSGI refuses anything else with a 400. `%inputs` in `Trog::Routes::HTML` names what each callback takes: the query of a GET and the form of a POST. `build_routes` declares that for every route of the callback, so a post that names its callback by string gets the same, less what the route fixes in its `data` or holds in its path. A route can declare its own `parameters` instead. The callback finds the input of the request in `$query->{input}`: the query, the form, each variable of the template without the `+`, and the route's `data`. The rest of `$query` is what tPSGI and the dispatcher set, which no request can replace. So a page can hand `%$query` to its template, and the request still chooses none of the template's variables. **Adding a content type does not require touching Perl.** A post type is a template that says how to display and edit it, plus a JSON sidecar that says what fields it has. Both are files in a directory. See [Post types](#post-types). The request lifecycle ===================== tCMS is a routing module, not a server. It is run by **tPSGI**, which owns the socket, the process model, the access log and the static file cache. tPSGI hands each request to the callback tCMS registered for that route, and hands the callback a `tpsgi` object for the things only the server can do -- redirect, forbid, invalidate cached renders, run work after the response is closed. ``` tPSGI │ route lookup against the table TCMS built at startup ▼ TCMS::build_routes' wrapper ← identity, ACLs, logging context │ ▼ Trog::Routes::HTML:: ← e.g. series() → posts() → index() │ ├──▶ Trog::Data->new(config) ← the data model: get() the posts │ └──▶ Trog::DataSource ← unless this type's posts are built, not stored │ ├──▶ enrich_post ← resolve declared relations, run enrich subs │ ▼ Trog::Renderer->render(...) ← picks a renderer by content type │ └──▶ Text::Xslate ← templates, with component() and render_it() ▼ [ code, headers, body ] ← a PSGI triplet │ └──▶ tpsgi->save_render(...) ← if this page is cacheable, write a static ``` The wrapper installed by `TCMS::build_routes` is where a request stops being anonymous bytes and becomes a request by somebody: 1. Initialise logging into tPSGI's log directory. 2. If `config/setup` does not exist, this is a fresh install -- everything goes to the setup route. 3. Pull the `tcmslogin` cookie, resolve it to a user, load that user's ACLs into `$query->{user_acls}`. 4. Strip an `admin` ACL from anything the *request* asked for unless the user actually holds it. 5. Refuse the request if the route requires auth and nobody is logged in. Everything downstream reads `$query->{user_acls}` and never asks who the user is again. The static render cache ----------------------- `Trog::Renderer::render` hands finished pages back to tPSGI to save as static files, but only when caching one is safe. It skips when the render is a component, when the route says `nocache`, when there is no route to key on, when the request had a query string, **when a user is logged in**, when the status is not 200, or when running at debug level. That last-but-two is the important one: a logged-in reader sees `/secure/...` variants and editing chrome, and caching those would serve one user's view to another. Anonymous readers get files off disk; editors get live renders. Saving a post invalidates the cache. `nocache` is read off both the render options and the page data, so a route's `nocache` flag -- which `TCMS::build_routes` puts on the query -- reaches it whether or not the route thought to forward it. That is also how a datasource which cannot say when its posts go stale keeps its pages out of the cache. Post types ========== This is the extension point that matters, so it gets the most detail. A post type is two files in `www/templates/html/components/forms/`: blog.tx the template: how a post of this type displays, and how it edits blog.json the sidecar: an OpenAPIv3 object schema for its fields `Trog::DataModule::schema_for` merges the sidecar with the base post schema and uses the result to validate every save. The base schema is merged **last**, on purpose: a sidecar can add fields but can never redefine a core one. A sidecar retyping `callback` as a plain string would defeat the check that the sub it names actually exists, which is a privilege problem rather than a cosmetic one. The sidecar carries the wizard's own UI metadata on `x-` extension keys, so there is one file to keep in sync rather than two: | Key | Means | |---|---| | `x-tcms-label`, `x-tcms-placeholder` | what the editor shows for this field | | `x-tcms-input` | which widget to render | | `x-tcms-private` | editors only -- the value is dropped before rendering, so no template mistake can leak it | | `x-tcms-indexed` | give this field its own indexed column, where the data model can | | `x-tcms-relation-form` | this field holds another post's UUID | | `x-tcms-relations` | what to resolve onto the post at render time | | `x-tcms-datasource` | these posts are built by a module, not stored | | `x-tcms-post-type` | the type's own name, description, and everything the wizard was told to build it | You do not have to write either file by hand. **The Post Type Wizard** (`/admin/wyzzerdd`) generates both from a form: name the fields, pick their types, tick whether each is required, editors-only, or indexed, and write the display snippet. It writes the template and the sidecar into whichever forms directory is in use -- the theme's if it has one, the stock one otherwise -- and the type is live on the next request. No restart: `themed_templates_in_dir` re-reads the directory every call and Xslate resolves includes lazily. The generated template is ordinary Kolon, so hand-editing it afterwards is expected and fine. Regenerating the type overwrites it, which is why the wizard says so at the top of what it writes. The wizard's "Start from" dropdown reopens a type that already exists: it fills the form back in from the sidecar, shows the type's description, and ticks Overwrite, so changing one thing about a type is a two-minute job rather than filling the form in again from memory. That works because `x-tcms-post-type` records everything the wizard was told -- the description, the display template, the checkboxes, the placeholder -- and `schema_for` strips that whole key before validation, so none of it can affect whether a post saves. Canned blocks ------------- Most of an editor is not the type's own: `preview.tx`, `tags.tx`, `aliases.tx`, `visibility.tx`, `acls.tx` and `attachments.tx` are blocks every form splices in, and the wizard's "Canned Fields" checkboxes are which of them to include. Each has a sidecar of its own beside it, in the components dir rather than in `forms/`: preview.tx the block: the markup an editor splices in preview.json the sidecar: an OpenAPIv3 fragment for what it collects `Trog::DataModule::includes_for` reads a type's `: include` lines back out of its template -- the only account there is for a hand-written type -- and `include_schema_for` turns each one into the fields it collects. Two things follow from one declaration: - `schema_for` merges them in, so a generated type keeps the preview image its own editor collected. `validate` drops every field the schema does not describe, and nothing described those. - The wizard tells a canned field apart from one somebody typed in, so ticking "Preview image upload" and listing `preview` as a custom text field are no longer the same thing said twice. A block never overrules the type that borrows it, and the base schema still overrules both: the merge order is base, then the type's own sidecar, then the blocks it includes. A block with no sidecar collects nothing, which is right for `edit_head.tx` and the rest of the furniture. Series, and how content is organised ------------------------------------ A **series** is a post of type `series.tx` that does two jobs: - It **owns an ACL**. Its `aclname` is the tag its children carry, so "who may read this section" is one field on one post. - It **names its children's type** via `child_form`. A series of `invoice.tx` lists invoices; the same site's series of `blog.tx` lists blog entries. Tag a series `topbar` and it appears in the site navigation. That is the whole of "site structure" -- there is no menu editor, because the menu is a query. Visibility and ACLs ------------------- `visibility` is one of `public`, `unlisted`, `private`. `Trog::DataModule::process` folds it into the post's `tags`, along with the post's ACLs when it is private, and then deletes the separate `acls` field. **Tags are the whole truth about who may see a post**, which is why `get()` answers the `acls` filter by looking at tags and nothing else. Know this before you touch anything that filters. A change that makes tags and ACLs diverge is a disclosure bug, not a refactor. Storage: data models ==================== A data model is where posts live. `Trog::Data` is a factory; the implementation is chosen by `general.data_model` in the config. `Trog::DataModule` is the base class and provides the query semantics -- `get()`, `filter()`, `paginate()`, version rollup, validation, and the derived-field plumbing in `process`. A subclass must implement `read`, `write`, `count`, `tags`, `lang` and `help`, and may override `get` if it can answer a query more directly than by filtering in Perl. | Model | What it is | Use it when | |---|---|---| | `DUMMY` | one JSON blob | never in production; it is not race-safe. It exists as the smallest possible worked example of the interface | | `FlatFile` | one file per post, SQLite alongside as a tag index | you want the posts to be greppable files on disk, and the site is small | | `SQLite` | posts in SQLite, blob and all | anything else | Posts are **versioned**. A save appends a version rather than overwriting, and `get()` hands back the newest one, carrying `version_max` plus the original `created` and `author` from the first version. Asking for a specific `version` gets that one. The SQLite model is worth understanding as a pattern, because it is the shape a new model should copy: it stores the same JSON blob the flat file model would have written and projects everything worth querying back out of it as `GENERATED ALWAYS ... VIRTUAL` columns. The blob stays the single source of truth, the projections cost no storage and cannot drift from it, and adding a queryable field is an `ALTER TABLE` plus a `CREATE INDEX` with no migration -- because the data never moves. Tag membership needs one row per tag, so triggers unpack `json_each` into an index table; search is FTS5 with the trigram tokenizer, which is the one tokenizer that answers the substring queries the search box has always meant. Foreign authority: data sources =============================== Sometimes the authoritative copy of something is not yours and never will be. A hypervisor knows what guests exist; a payment processor knows what was paid; an LDAP directory knows who works here. Copying that into the datastore means it is wrong the moment you copy it. A **datasource** lets a post type say "my posts are built, not stored". The series is still an ordinary post; only its children are synthesized, and they are rebuilt on every view. "x-tcms-datasource": "Trog::DataSource::Virt" The module provides `posts($series, $query)` and may provide `filter`, `order`, `lang`, `help` and `EDITABLE`. `Trog::DataSource` documents the whole contract and supplies the defaults. `Trog::DataSource::DirIndex` is the shortest complete example: a directory becomes a listing, with a hard constraint on which directories it will look at and an inotify watch so the cached page is thrown away when the directory changes. Two things about the contract are worth knowing before you write one, because both are places the obvious implementation is wrong: - **Your posts have never been near the datastore**, so nothing has filtered them. The reader's search was answered against the datastore and thrown away along with the posts it filtered. `filter()` is how the search reaches your posts, and you should implement it if your posts carry anything worth searching beyond a title. - **Nothing else will invalidate your page's static render.** A post being saved does not invalidate it, because no post was saved -- the thing you depend on changed instead. So you must answer one of two ways. Either you can name what you depend on, in which case register a `$tpsgi->add_watch` for it with a stable `key`, invalidate through the tPSGI object your callback is *handed* rather than the one it closed over, and declare `CACHEABLE` (`DirIndex::watch` is four lines of this). Or you cannot, in which case say nothing and your pages are never cached -- which is the default, because a page that is merely slow beats one that is wrong with no way of becoming right. - **Your posts probably all share a timestamp**, because you stamped them with the time you built them. Stored posts paginate by a `created` cursor, which on such a page cannot move -- every cursor is the same instant. Datasource pages therefore paginate by page number, which is why `order()` has to be total: page 2 of a shuffled list is a lottery. Rendering ========= `Trog::Renderer->render(%options)` dispatches on content type to a renderer under `Trog::Renderer::`, all of which subclass `Trog::Renderer::Base`. The known types are in `Trog::Vars::%content_types`: text, html, json, blob, xml, xsl, css, rss, email. Base sets up the Xslate instance, resolves templates against the theme's directory and then the stock one, and injects two functions every template can call: - `render_it($string)` -- render a template string that came out of a post, which is what lets a post body contain `<: embed(12345, 'embed') :>`. - `component('Name', {...})` -- render a `Trog::Component`. Components ---------- A component is a module in `Trog::Component` with a `render(%args)` returning a string. Templates ask for what they want by name rather than having a route render it and thread the string through: <: component('EmojiPicker') :> <: component('Gallery', { album => $post.id, cols => 3 }) :> The page furniture is built this way -- `Header`, `Footer`, `HtmlTitle`, `MidTitle`, `TopBar`, `LeftBar`, `RightBar`, `FootBar`, `CategoryBar` are what `index.tx` is made of. Each module owns the name of the template it renders, so adding a piece of furniture does not mean editing a route. One caution if you write one: Xslate catches an exception thrown out of a template function, warns, and renders on with an empty string where the call was. A component that merely dies would vanish from the page with a line on stderr to show for it, so `Trog::Component` also records the failure for `Renderer::Base` to turn into a 500. Themes ------ A theme is a directory under `www/themes/` mirroring the structure of `www/`. Templates resolve against the theme's directory first and the stock one second, so a theme overrides individual templates without forking the ones next to them. Stylesheets are the exception and are loaded both, theme last, so a theme can override a handful of rules without restating the sheet. A theme may ship a `routes.pm` to add routes of its own. A script resolves as a template does. A route names the scripts of its page in its definition: `scripts` for every reader, and `authenticated_scripts` for a reader who is logged in as well. A theme names the scripts of every page in `@Theme::scripts`, which go before those of the route. A template does not write a `