tCMS

tCMS

A flexible perl CMS which supports multiple data models and content types. Should be readily portable/hostable between any other system that runs tCMS due to being largely self-contained.

tCMS exists for disintermediation from aggregators: your content, at your domain, in your database, under your rules. It is built so that standing up a site hosting content of any imaginable type is a day's work rather than a project -- which is to say, it is built for people who get paid to deliver the thing a client actually asked for.

See design.md for the architecture: what a post is, why the routing table is data, how to add a content type without writing Perl, and where any given change belongs. Start there if you are going to work on tCMS or build something on top of it.

tCMS is built to be run by tPSGI.

Simple deployment is currently:

• make -f Installer.mk depend
• make -f Installer.mk install
• Setup tPSGI (supposing this clone is in a subdir tCMS of the tPSGI clone):

.tpsgi.conf :

http_user=www-data
user=INSERT_MY_USERNAME_HERE
domain=INSERT_MY_DOMAIN_NAME_HERE
routers=tCMS/lib/TCMS.pm
indices=
custom_log="/var/log/www/tpsgi.log"
basedir="tCMS"

Then:

• open tmux or screen
• HOME=. bin/tpsgi -p 5001

You won't want to run like this in production, but this is probably how you want to develop your themes or hack on tCMS itself.

Nothing hands a vault key to a tPSGI you started yourself: the systemd credential and the unit that imports it are not in the picture, and service/tpsgi.sh, which is what would have put it in the environment, never ran. So if you want /secrets to work while developing, mint one into the checkout:

bin/tcms-vault-key --file

That writes config/secrets.key, which is the last place Trog::Vault looks. It is the right thing for development and testing and the wrong thing for a production host, for the reason under Stored Secrets: it sits next to the database it protects, so anything that backs up one backs up the other. .gitignore keeps it out of the repository; keeping it out of your backups is on you.

Production Deployment

See the trog-provisioner repository.

You will be interested particularly in:

• Provisioner::Recipe::tcms
• Provisioner::Recipe::tpsgi
• Provisioner::Recipe::nginxproxy

Many of the advanced features of tCMS won't work quite right without the configurations encoded therein.

Content Types

See design.md for how post types work and what the sidecar can declare. In brief:

Content templates are modular. Add in a template to www/templates/html/components/forms which describe the content and how to edit it. Our post data storage being JSON allows us the flexibility to have any kind of meta associated with posts, so go hog wild.

You describe the type of data provided in a sidecar JSON file alongside the form's template. There's a wizard you can use to build these!

The wizard's "Start from" dropdown reopens any type that already exists, filling the form in from its sidecar so you can change one thing and save it back. Each type carries a description saying what it is for, which the dropdown shows and a collapsed textarea lets you edit.

The canned blocks an editor reuses -- preview.tx, tags.tx, attachments.tx and the rest -- have sidecars of their own beside them in components/, saying which fields each one collects. That is how the wizard knows a preview image is a box to tick rather than a custom field to list, and how a type built by the wizard keeps the preview image its editor uploaded.

From there you make Series of the content type you want; tag the series with 'topbar' if you want it to show up in the top links.

Currently supported:

• Microblogs
• Blogs
• Files (Video/Audio/Images/Other)
• About Pages
• Presentations Virtualization Data:
• Hypervisors
• Guests

Embedding Posts within other Posts

If you know a Post's ID (see the numbers at the end of it's URI when viewing it's permalink denoted by the chain emoji) You can embed template logic into your posts like so:

<: embed(12345, 'embed') :>

The first parameter is the ID number of the post. The second parameter is the formatting style:

• embed : default, shows the post with a recessed border as an excerpt.
• media : only show media portion of the post, if any.
• inline : show everything about the post, save for the title.

These will be added as classes to the embedded post, so you can theme this appropriately.

Data Models

Posts are normally stored as a file somewhere.

• DUMMY - A JSON blob. Used for testing mostly, but could be handy for very small sites.
• Flat File - Pretty much the tCMS1 data model, but with an SQLite index bolted on.
• SQLite - The posts themselves in SQLite, blob and all.

Pick one with general.data_model in your config.

The SQLite model stores each version of each post as the same JSON blob the flat file model would have written, and projects everything worth querying back out of it as GENERATED ALWAYS ... VIRTUAL columns, which cost no storage and cannot drift from the post they came from. Filtering, paging and search are then the database's job rather than a grep over every post on disk -- on a 20,000 post site, a search goes from about 2.5 seconds to about 3 milliseconds. Search is an FTS5 index built with the trigram tokenizer, so it keeps the case insensitive substring matching the search box always had.

Adding a queryable field is one ALTER TABLE and one CREATE INDEX in the next update file beside the schema, such as schema/sqlite.schema.v1.update.sql. Trog::SQLite runs each update file once per database, in order. No migration of the data, because the data itself never moves:

ALTER TABLE posts ADD COLUMN subhead TEXT
    GENERATED ALWAYS AS (json_extract(post_data, '$.subhead')) VIRTUAL;
CREATE INDEX posts_subhead ON posts(subhead);

For a post type's own custom fields you don't have to write that yourself: the Post Type Wizard offers an "Index this field" checkbox per field, off by default, and saving the type does the above for every field that has it ticked. Untick it and the index goes away again on the next save. It is worth ticking for a field you actually search or filter on, and not otherwise -- an index costs a little on every save and some disk.

On a data model with no notion of an index the checkbox simply does nothing, so the wizard doesn't have to know which model the site is running.

To move an existing flat file site over, run bin/migrate.pl from the tCMS root and then set data_model=SQLite and restart. It copies rather than moves, so data/files is left alone and the way back is to set data_model back. It is safe to re-run -- run it once against the live site, then again after a final quiet period to pick up whatever was written in between. --dry-run tells you what it would do.

Data Sources

Sometimes you want to consider something else authoritative that isn't a datamodel under our control.

• Virt - Talk to a libvirt HV to list guests.
• ProvisionedVirt - Virt, plus the recipe each guest was built from and a button to build it again.
• DirIndex - List the contents of a directory.

ProvisionedVirt is a subclass of Virt, and shows what subclassing one is for: it inherits the libvirt half unchanged and adds the other half of the story, which is that a guest on a Troglodyne hypervisor was provisioned from a recipe on disk. Tell it where trog-provisioner is checked out and it shows each guest's recipes.d/$domain.yaml and offers a Reprovision button running bin/provision, which generates the guest's configuration from that recipe and then builds it:

[provisioner]
    trog_provisioner = /home/you/Code/trog-provisioner

Leave that out and it degrades to a plain guest listing. The recipes themselves are not in the checkout -- they describe an installation rather than the software, so trog-provisioner keeps them in /etc/trog-provisioner, or wherever TROG_PROVISIONER_CONFIG says, and we read them from the same place.

Reprovisioning is not recoverable: it destroys the guest and builds it again, and whatever the recipe does not describe does not come back. It is POST and admin only, it refuses outright for the guest tCMS is itself running on (as Virt already refuses to power that one off), and nothing it runs goes near a shell.

A provision takes minutes and an HTTP worker does not have minutes, so the work is handed to a detached process. Nothing waits on it, which means it has to report on itself: it writes logs/reprovision/$domain.log and a status beside it, and the guest's entry shows which of

reprovisioning / provisioned / reprovision failed / reprovision interrupted

it is, who asked for it, and a link to read the whole log. While one is running the button is not offered, so a second cannot be started by double clicking. A run whose status still says 'running' but whose process is gone reads as interrupted rather than as going on forever -- that is a guess from the only evidence there is, and it is the honest one.

The button asks for two passwords, and neither is written to the config, the log, or the process table -- they go down the provisioner's stdin as a credentials block and nowhere else.

The first is your KeePass passphrase, because every recipe inherits secret: values from _base. The alternative would be keeping the master password for every secret you hold in a file the webserver can read.

The second is your sudo password on the hypervisor, and it is optional because it is usually unnecessary: give that login passwordless sudo and there is nothing to ask for. Where you cannot, sudo over ssh has no terminal to ask at, so a run that needs one and was not given one dies several minutes in on a prompt nobody will ever see. Leave the field empty if this does not apply to you.

Typing them every time has its own cost -- a passphrase you have to have to hand is one you write down somewhere -- so you can tick "remember these" and have them stored in your vault instead. After that the button asks for a TOTP code, which is spent on that one reprovision and exchanged for both. See Stored Secrets, below.

Stored Secrets

Some of what tCMS does for you needs a password that is not tCMS's. /secrets is where each user keeps those, and what is kept there is sealed with a key the database does not contain, so a stolen config/auth.db is ciphertext.

Reprovisioning uses two of them, provisioner and provisioner_sudo, and the reprovision dialog is the easy way to put them there.

AES-256-GCM per row, under a key derived for that row from a fresh random salt, with the username and the secret's name bound in as associated data -- so a row cannot be lifted into somebody else's account or renamed into being the answer to a different question. Nothing renders a value back; if you have forgotten one, replace it.

The master key is looked for in TPSGI_VAULT_KEY, then $CREDENTIALS_DIRECTORY/tpsgi-vault, then config/secrets.key. Use the first: tPSGI's bin/build_service mints it and puts it in systemd's credential store, sealed to this machine's TPM if it has one, and its unit hands it over before it chroots -- so it never touches a disk here. It is named for tPSGI rather than tCMS because tPSGI owns the unit and runs whatever is in the directory. bin/tcms-vault-key --file writes one to config/secrets.key instead. That is what to use when you are running tPSGI by hand rather than under systemd, which is the ordinary way to develop against this and to debug it -- see Development above. It is also the answer on a machine with no systemd at all, and there it is the weaker choice: a key beside the database it protects is a key in the same backup as the database it protects.

tCMS deletes the key from its environment as it loads, because one of the things it forks is a provisioner it hands a decrypted password to. With no key there is no vault: nothing is stored, nothing already stored is offered, and everything that would have used one goes on asking for the password directly.

The key lives and dies with the machine. It is deliberately not in any backup, not in the checkout, and not something trog-provisioner carries onto a rebuilt guest -- so rebuilding the host means every stored secret is gone and each user stores theirs again. That is the intended cost. Nothing breaks in the meantime: a row that will not open says so on /secrets, the reprovision button goes back to asking for the passphrase, and no code is spent finding out.

What unlocks a secret is a TOTP code, and a code is spent when it is used -- a code authorizes one thing rather than everything you can do in its window. That includes logging in, so if you have just logged in you will be waiting for the next code. That is the cost, and it is thirty seconds.

DirIndex is the one to copy if you are writing your own. Point a series at a directory with its "Directory to index" field, build a child type in the wizard with DirIndex as its datasource, and every file in there becomes a post carrying its name, path, size, mtime, extension and content type -- so a downloads page, a gallery or a release listing is a template and nothing else. It will only index directories under www/ -- which the webserver already serves, so a listing discloses nothing a guessed URL would not -- and never www/assets/private.

That last is a guard rail rather than a protection: nginx gates those files by auth_requesting the /authenticated route, so a listing of them would not hand the files out, only the names. Sharing that directory is operator error, and refusing it turns a mistake in a text field into a line in the log. See the POD in lib/Trog/DataSource/DirIndex.pm.

A post type names one in its sidecar with x-tcms-datasource, and a series of that type then lists whatever the source builds instead of posts somebody wrote. The series is still an ordinary post; only its children are synthesized, and they are rebuilt on every view rather than stored.

A source has to provide posts($series, $query). It may also provide:

• filter($query, @posts) - apply the reader's search to the posts it built. Those posts have never been near the datastore, so the search the reader typed would otherwise be answered against the datastore and then thrown away along with the posts it filtered. Trog::DataSource::filter is the default, and searches a post's title and body; implement your own to search the fields your posts actually carry, as Virt does for a guest's name, state and hypervisor.
• order($query, @posts) - the order they belong in. Defaults to newest first, then by title, then by id. Pagination hands out page 2 of an order, so there has to be one and it has to be the same one next time somebody asks.
• lang() and help() - what the search box on such a page is searching, and where to read about it. The data model's answer describes the datastore, which is not what these pages are serving.
• EDITABLE - whether the wizard should generate an editor for the type. A source building its posts from somewhere else has nothing to edit, and saying nothing means no.
• CACHEABLE - whether its pages may be saved as static renders. Saying nothing means no; see below.

Datasource pages would be cached as statics exactly like any other page, which raises the question of what throws that cache away -- saving a post will not, because no post was saved when somebody dropped a file in a directory. So a source has to say which of the two it is:

• CACHEABLE - it knows what it depends on and says when that changes, so its pages may be cached. Saying nothing means no, and that is the safe way round: a source whose author has not thought about invalidation gets the answer that is merely slow rather than the one that is wrong. DirIndex says yes because it watches the directory; Virt says no, because a guest's state is the point of the page and nothing could ever tell us it changed.

A source declaring CACHEABLE should say when it changes, using tPSGI's watch facility, as DirIndex does:

$tpsgi->add_watch( $directory, sub {
    my ( $watcher, $change ) = @_;
    $watcher->invalidate_renders('html');
}, key => "something stable" );

Pass an explicit key. posts() runs per request, so a closure built there has a new address every time and the default key would stack another copy of the same callback on every view. Use the tPSGI object the callback is handed rather than the one from the request that registered it: watches are shared between workers, and whichever worker notices the change is the one that has to act on it.

Such pages paginate by page number rather than by the created cursor stored posts use. A datasource has already built every post by the time the paginator sees them, so there is nothing to save by paging with a cursor -- and it would not work anyway for a source that stamps everything with the time it built them, as libvirt guests are: every cursor on the page is the same instant, so Prev asks for everything older than now and gets nothing.

See lib/Trog/DataSource.pm for the whole contract.

Components

Sometimes you re-use a template a lot. Sometimes it also needs special handling. This is when you write a 'component'. Example:

Trog::Component::EmojiPicker -> renders www/templates/html/components/emojis.tx

A component is a module in the Trog::Component namespace with a render(%args) method returning a string. Templates call it by name:

<: component('EmojiPicker') :>
<: component('Gallery', { album => $post.id, cols => 3 }) :>

The optional hash is flattened into the component's render(); nullary components ignore it. Output comes back marked raw, so no | mark_raw is needed. Routes don't have to know a component exists -- the template asks for what it wants.

The page furniture is built this way: Header, Footer, HtmlTitle, MidTitle, TopBar, LeftBar, RightBar, FootBar and CategoryBar are what index.tx is made of, and PostHeader/PostFooter are the per-series header and footer posts.tx pulls in.

Ideas to come:

domain picker at top -- manage all your web properties from one place

login and registration (forces email for a domain to allow posting on said domain) User data also stored in ES -- it's their profile page!

Error and Access logs immediately dumped into ES for EZ viewing in grafana

Automatic analytics!

Multiple auth models (ldap, oauth etc)

Builtin paywall -- add in LDAP users not on primary domain, give differing privs Have all content able to assign to paywall packages

One click share to social via oauth Mailing list blasts for paywall content