Serve the scripts of a page as one versioned bundle

A page loaded each of its scripts as a separate request. Some came from
the list that a route names, and the templates wrote the others as
script tags in the body. A post page made four requests for scripts.

Now the route names every script of its page, and finish_render links
one bundle of them at /bundle/{version}/{names}.js. Trog::Bundle builds
the bundle. It takes the theme's copy of a script in place of the stock
one, as themed_script() did. It minifies each file unless the name of
the file ends in .min.js. It puts a newline and a semicolon between the
files, because a file can end in a line comment. The version is a digest
of the body, so the response has the immutable Cache-Control. A process
builds each bundle once and builds it again when a file in it changes.
At an old version, the route redirects to the current one, so a page that
was cached before the change still gets its scripts.

The names come from the URL, so anyone can ask for a list that no page
uses. The cache is emptied when it holds 16 bundles, which keeps the
memory of a worker bounded. The bundle is not saved through save_render,
because each list that somebody asked for would then stay on disk.

The vendored chart.js and reveal.js are already minified, and the
minifier drops a comment unless it contains the word "copyright". So a
second pass removed the MIT notice from chart.js. Installer.mk now saves
the two files as chart.min.js and reveal.min.js, and the bundle leaves
them as they are. An install that has the old names must run make
prereq-frontend again. Until then, the bundle leaves out each script that
is missing and logs a warning, and the rest of the page still works.

config() and resetpass() named post.js in the hash that they pass to
index(). index() replaced that key with its own list of scripts, so the
pages never loaded post.js, and neither page calls a function from it.
Those keys are gone, so that a reader does not think that they work.

The bundle goes in the head, as the listed scripts did. Each moved script
only defines functions, or waits for DOMContentLoaded, so it works there.
The bundle is not deferred, because presentation.tx calls
Reveal.initialize() inline as soon as the page parses.

I searched for an existing bundler or minifier of scripts and found
none. Trog::Renderer::css minifies templates, not files.

Closes #372

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

.gitignore

@@ -11,8 +11,8 @@ pm_to_blib
 pod2htmd.tmp
 www/scripts/list.min.json
 www/scripts/highlight.min.js
-www/scripts/chart.js
-www/scripts/reveal.js
+www/scripts/chart.min.js
+www/scripts/reveal.min.js
 www/styles/obsidian.min.css
 www/styles/reveal.css
 www/styles/reveal-white.css

Installer.mk

@@ -22,6 +22,8 @@ prereq-frontend:
 		"https://raw.githubusercontent.com/highlightjs/cdn-release/main/build/highlight.min.js" \
 		"https://cdn.jsdelivr.net/npm/chart.js" \
 		"https://raw.githubusercontent.com/hakimel/reveal.js/master/dist/reveal.js"; popd
+	mv www/scripts/chart.js www/scripts/chart.min.js
+	mv www/scripts/reveal.js www/scripts/reveal.min.js
 	mkdir -p www/styles; pushd www/styles && curl -L --remote-name-all \
 		"https://raw.githubusercontent.com/highlightjs/cdn-release/main/build/styles/obsidian.min.css" \
 	    "https://raw.githubusercontent.com/hakimel/reveal.js/master/dist/reveal.css" \

Makefile.PL

@@ -133,6 +133,7 @@ WriteMakefile(
         'WWW::Sitemap::XML'                   => '0',
         'WWW::SitemapIndex::XML'              => '0',
         'CSS::Minifier::XS'                   => '0',
+        'JavaScript::Minifier::XS'            => '0',
         'Path::Tiny'                          => '0',
         'Log::Dispatch'                       => '0',
         'Log::Dispatch::FileRotate'           => '0',

design.md

@@ -378,6 +378,13 @@ Stylesheets are the exception and are loaded both, theme last, so a theme can
 override a handful of rules without restating the sheet. A theme may ship a
 `routes.pm` to add routes of its own.
 
+A script resolves as a template does. A route names the scripts of its page in
+a list, and a template does not write a `<script src>` tag of its own.
+`finish_render` links the list as one bundle, which `Trog::Bundle` builds and
+`/bundle/{version}/{names}.js` serves. The version is a digest of the bundle,
+so a browser keeps the bundle until a script in it changes. A script whose name
+ends in `.min.js` goes into the bundle as it is, and the others are minified.
+
 Composition: relations and enrichment
 =====================================
 

lib/Trog/Bundle.pm

@@ -0,0 +1,87 @@
+package Trog::Bundle;
+
+use 5.040;
+use re '/aasx';
+use warnings FATAL => 'all';
+
+use Digest::SHA              ();
+use File::Slurper            ();
+use JavaScript::Minifier::XS ();
+use List::Util               ();
+
+use Trog::Log    qw{:all};
+use Trog::Themes ();
+
+=head1 Trog::Bundle
+
+The scripts that a page names, served as one response.
+
+A route names its scripts in a list, and finish_render() in Trog::Routes::HTML
+links the bundle of them at the href that scripts() returns.  The route
+C</bundle/{version}/{names}.js> serves that bundle.
+
+=cut
+
+# The names of a bundle, as its href carries them, to the bundle.
+my %bundles;
+
+# The number of bundles kept.  A site has a few lists of scripts, and anyone can
+# ask for another one, so the cache is emptied when it grows past this.
+my $max_bundles = 16;
+
+=head2 scripts(@names) = HASHREF or undef
+
+The bundle of the scripts that @names names, each of them a file name under
+F<www/scripts>, such as C<post.js>.  The theme's copy of a script is used in
+place of the stock one.  The order is the order of @names, and a name that is
+given twice is used once.
+
+Returns undef if no name is a script.  A name that is not a script is left out,
+with a warning.
+
+The hash has these keys:
+
+    href     where the bundle is served, which changes when its body does
+    version  a digest of the body, which is in the href
+    body     the scripts, each one minified unless its name ends in .min.js
+
+A process builds each bundle once, and builds it again when a file in it
+changes.
+
+=cut
+
+sub scripts (@names) {
+    my ( @kept, @files );
+    foreach my $name ( List::Util::uniq(@names) ) {
+        my $file = $name =~ m/\A\w[\w.-]*[.]js\z/ ? Trog::Themes::themed_script($name) =~ s{\A/}{www/}r : undef;
+        if ( !$file || !-f $file ) {    ## no critic (ProhibitFiletest_f) -- a name that is not a file is left out
+            WARN("There is no script $name to bundle");
+            next;
+        }
+        push( @kept,  $name );
+        push( @files, $file );
+    }
+    return undef unless @files;
+
+    my $names     = join( '+',  map { s/[.]js\z//r } @kept );
+    my $signature = join( "\0", map { join( ':', $_, ( stat $_ )[ 7, 9 ] ) } @files );
+    return $bundles{$names} if $bundles{$names} && $bundles{$names}{signature} eq $signature;
+
+    # A newline before the semicolon, because a file can end in a line comment.
+    my $body = join(
+        "\n;\n",
+        map {
+            my $js = File::Slurper::read_binary($_);
+            m/[.]min[.]js\z/ ? $js : JavaScript::Minifier::XS::minify($js)
+        } @files
+    );
+    my $version = substr( Digest::SHA::sha256_hex($body), 0, 16 );
+
+    %bundles = () if keys %bundles >= $max_bundles;
+    return $bundles{$names} = {
+        href      => "/bundle/$version/$names.js",
+        version   => $version,
+        body      => $body,
+        signature => $signature,
+    };
+}

lib/Trog/Routes/HTML.pm

@@ -33,6 +33,7 @@ use Trog::Config;
 use Trog::Auth;
 use Trog::Data;
 use Trog::Themes;
+use Trog::Bundle;
 use Trog::Renderer;
 use Trog::Email;
 use Trog::DataSource;
@@ -295,6 +296,10 @@ our %routes = (
         method   => 'GET',
         callback => \&Trog::Routes::HTML::rss_style,
     },
+    '/bundle/{version}/{names}.js' => {
+        method   => 'GET',
+        callback => \&Trog::Routes::HTML::bundle,
+    },
 );
 
 # Grab theme routes
@@ -838,7 +843,6 @@ sub config ( $query = {} ) {
             title           => 'Configure tCMS',
             theme_dir       => Trog::Themes::td(),
             stylesheets     => [qw{config.css}],
-            scripts         => [qw{post.js}],
             themes          => get_themes() || [],
             config_sections => config_sections(),
             message         => $query->{message},
@@ -1043,7 +1047,6 @@ sub resetpass ($query) {
             title       => 'Request Authentication Resets',
             theme_dir   => Trog::Themes::td(),
             stylesheets => [qw{config.css}],
-            scripts     => [qw{post.js}],
             message     => $query->{message},
             failure     => $query->{failure},
             scheme      => $query->{scheme},
@@ -1883,7 +1886,8 @@ sub posts ( $query, $direct = 0 ) {
     return $content if ref $content eq "ARRAY";
 
     return $content if $direct;
-    return Trog::Routes::HTML::index( $query, $content, $styles );
+    my @scripts = ( ( $is_admin ? qw{post.js post_relations.js} : () ), qw{highlight.min.js reveal.min.js} );
+    return Trog::Routes::HTML::index( $query, $content, $styles, \@scripts );
 }
 
 =head2 redact_private($post, $seen)
@@ -2350,7 +2354,7 @@ sub metrics ($query) {
         },
         undef,
         ['post.css'],
-        ['chart.js'],
+        ['chart.min.js'],
     );
 }
 
@@ -2667,6 +2671,7 @@ sub post_wizard ($query) {
         },
         undef,
         ['post.css'],
+        ['post_wizard.js'],
     );
 }
 
@@ -3379,6 +3384,31 @@ sub icon ($query) {
     return $query->{tpsgi}->serve( $path, $tpath, $query->{start}, $query->{streaming}, $query->{ranges}, $query->{last_fetched}, $query->{deflate} );
 }
 
+=head2 bundle
+
+Implements /bundle/{version}/{names}.js, the scripts of a page as one response.
+See Trog::Bundle.
+
+The body of a bundle is the same at a given href, so the browser keeps it.  An
+href with an old version is redirected to the current one, so a page that was
+cached before a script changed still gets its scripts.
+
+=cut
+
+sub bundle ($query) {
+    my $bundle = Trog::Bundle::scripts( map { "$_.js" } split( /[+]/, $query->{names} ) );
+    return $query->{tpsgi}->notfound($query) unless $bundle;
+    return $query->{tpsgi}->redirect( $bundle->{href} ) if $query->{route} ne $bundle->{href};
+    return [
+        200,
+        [
+            'Content-type'  => 'text/javascript',
+            'Cache-Control' => $Trog::Vars::cache_control{static},
+        ],
+        [ $bundle->{body} ],
+    ];
+}
+
 =head2 totp_qr
 
 Implements the /totp/* routes.
@@ -3459,15 +3489,15 @@ sub finish_render ( $template, $vars, %headers ) {
     # Theme-ize the paths
     $vars->{stylesheets}  = [ @{ build_themed_styles( $vars->{stylesheets} ) } ];
     $vars->{print_styles} = [ @{ build_themed_styles( $vars->{print_styles} ) } ];
-    $vars->{scripts}      = [ map { Trog::Themes::themed_script("$_") =~ s/^www\///r } @{ Trog::Utils::coerce_array( $vars->{scripts} ) } ];
+    my $bundle = Trog::Bundle::scripts( @{ Trog::Utils::coerce_array( $vars->{scripts} ) } );
+    $vars->{scripts} = $bundle ? [ $bundle->{href} ] : [];
 
     # Add in avatars.css, it's special
     push( @{ $vars->{stylesheets} }, "/styles/avatars.css" );
 
-    # Absolute-ize the paths for scripts & stylesheets
+    # Absolute-ize the paths for stylesheets
     @{ $vars->{stylesheets} }  = map { CORE::index( $_, '/' ) == 0 ? $_ : "/$_" } @{ $vars->{stylesheets} };
     @{ $vars->{print_styles} } = map { CORE::index( $_, '/' ) == 0 ? $_ : "/$_" } @{ $vars->{print_styles} };
-    @{ $vars->{scripts} }      = map { CORE::index( $_, '/' ) == 0 ? $_ : "/$_" } @{ $vars->{scripts} };
 
     # TODO Smash together the stylesheets and minify
 

t/Trog-Bundle.t

@@ -0,0 +1,96 @@
+#!/usr/bin/env perl
+use 5.040;
+use re '/aasx';
+use warnings FATAL => 'all';
+
+use Test::More;
+use Test::MockModule qw{strict};
+use Cwd();
+use File::Path();
+use File::Temp();
+use Path::Tiny();
+
+use FindBin::libs;
+
+# Every path in tCMS is relative to the instance root, so this stands in one.
+my $root   = File::Temp::tempdir( 'tcms-bundle-XXXXXX', TMPDIR => 1, CLEANUP => 0 );
+my $oldcwd = Cwd::getcwd();
+chdir($root) or die "could not chdir to the sandbox: $!";
+
+END {
+    chdir($oldcwd)                 if $oldcwd;
+    File::Path::remove_tree($root) if $root && -d $root;
+}
+
+require_ok('Trog::Bundle') or BAIL_OUT("Can't find SUT");
+
+my $stock = Path::Tiny->new('www/scripts');
+my $theme = Path::Tiny->new('www/themes/spec/scripts');
+$_->mkpath foreach ( $stock, $theme );
+
+$stock->child('one.js')->spew_raw("function one () {\n    return 1;\n}\n");
+$stock->child('two.js')->spew_raw("// two\nvar two = 2;\n");
+$stock->child('vendor.min.js')->spew_raw("var   vendor = 'kept as it is'; // a trailing comment");
+$stock->child('themed.js')->spew_raw("var themed = 'stock';\n");
+$theme->child('themed.js')->spew_raw("var themed = 'theme';\n");
+
+my $themedir = '';
+my $mock     = Test::MockModule->new('Trog::Themes');
+$mock->redefine( 'get_dir', sub { $themedir } );
+
+my @warned;
+local $SIG{__WARN__} = sub { push @warned, $_[0] };
+
+subtest 'the scripts, in the order named' => sub {
+    my $bundle = Trog::Bundle::scripts(qw{two.js vendor.min.js one.js two.js});
+
+    is( $bundle->{body}, "var two=2;\n;\nvar   vendor = 'kept as it is'; // a trailing comment\n;\nfunction one(){return 1;}", 'minified, except the one that already is, and a name given twice is used once' );
+    like( $bundle->{version}, qr/\A[[:xdigit:]]{16}\z/, 'the version is a digest' );
+    is( $bundle->{href}, "/bundle/$bundle->{version}/two+vendor.min+one.js", 'and the href carries it, and the names' );
+};
+
+subtest 'a name that is not a script' => sub {
+    @warned = ();
+    my $bundle = Trog::Bundle::scripts( qw{one.js nothere.js ../one.js .hidden.js sub/one.js one.css}, 'one.js+two' );
+    is( $bundle->{href}, "/bundle/$bundle->{version}/one.js", 'is left out' );
+    is( scalar(@warned), 6,                                   'with a warning for each' ) or diag( explain( \@warned ) );
+    like( $warned[0], qr/no[ ]script[ ]nothere[.]js/, 'that names it' );
+
+    @warned = ();
+    is( Trog::Bundle::scripts(qw{nothere.js}), undef, 'and a list of nothing else is no bundle' );
+    is( Trog::Bundle::scripts(),               undef, 'nor is an empty one' );
+};
+
+subtest "the theme's copy" => sub {
+    is( Trog::Bundle::scripts(qw{themed.js})->{body}, "var themed='stock';", 'is not used without the theme' );
+
+    $themedir = 'www/themes/spec';
+    is( Trog::Bundle::scripts(qw{themed.js})->{body}, "var themed='theme';",       'and is used in place of the stock one with it' );
+    is( Trog::Bundle::scripts(qw{one.js})->{body},    'function one(){return 1;}', 'while a script the theme lacks is the stock one' );
+    $themedir = '';
+};
+
+subtest 'a script that changes' => sub {
+    my $before = Trog::Bundle::scripts(qw{one.js});
+    is( Trog::Bundle::scripts(qw{one.js}), $before, 'the bundle is built once' );
+
+    $stock->child('one.js')->spew_raw("function one () {\n    return 'one';\n}\n");
+    utime( time + 10, time + 10, $stock->child('one.js') ) or die "could not touch one.js: $!";
+
+    my $after = Trog::Bundle::scripts(qw{one.js});
+    is( $after->{body}, "function one(){return'one';}", 'and built again when a file in it changes' );
+    isnt( $after->{href}, $before->{href}, 'at a new href' );
+};
+
+subtest 'many lists' => sub {
+    $stock->child("s$_.js")->spew_raw("var s$_;") for 1 .. 20;
+
+    @warned = ();
+    my %bundle = map { $_ => Trog::Bundle::scripts( 'one.js', "s$_.js" ) } 1 .. 20;
+    my %body   = map { $_ => $bundle{$_}{body} } keys %bundle;
+    is( scalar(@warned), 0, 'twenty lists' );
+    isnt( Trog::Bundle::scripts(qw{one.js s1.js}), $bundle{1}, 'are more than the cache keeps, so it was emptied' );
+    is( Trog::Bundle::scripts( 'one.js', "s$_.js" )->{body}, $body{$_}, "list $_ still has its body after the cache is emptied" ) for 1, 20;
+};
+
+done_testing();

t/Trog-Routes-HTML-series-lifecycle.t

@@ -134,6 +134,12 @@ BEGIN {
     print {$fh} '{"emojis":[{"category":"Smileys","emoji":"X"}]}';
     close($fh) or die $!;
 
+    # The vendored scripts are downloaded at install, so they are not in the
+    # checkout; these stand in for them.
+    File::Slurper::Temp::write_text( "$ROOT/www/scripts/reveal.min.js",    "var Reveal = 'spec reveal';" );
+    File::Slurper::Temp::write_text( "$ROOT/www/scripts/highlight.min.js", "var hljs = 'spec highlight';" );
+    File::Copy::copy( "$REPO/www/scripts/$_", "$ROOT/www/scripts/$_" ) or die $! for qw{post.js post_relations.js post_wizard.js};
+
     # Text::Xslate computes its default cache dir from $HOME once, at load time.
     # Without this the sandbox writes compiled templates into the real ~/.xslate_cache.
     $ENV{HOME} = $ROOT;
@@ -1225,6 +1231,34 @@ subtest 'presentation' => sub {
     my $deck = qq{<a href="/posts/$child->{id}?embed=1&nochrome=1">Click Here to view</a>};
     like( $body, qr{\Q$deck\E}, 'deck link rendered' );
     unlike( $body, qr/Slide[ ]one[ ]body\./, 'slides themselves are not rendered inline' );
+
+    my $deck_query = sub (@who) { return ( route => "/posts/$child->{id}", id => $child->{id}, embed => 1, nochrome => 1, @who ) };
+    my ( $code, $page, $err ) = render( anon( $deck_query->() ), \&Trog::Routes::HTML::posts );
+    is( $code, 200, 'the deck renders' ) or return diag($err);
+    like( $page, qr/Reveal[.]initialize/, 'and starts reveal' );
+
+    my @srcs = $page =~ m{<script\b[^>]*\bsrc="([^"]+)"}g;
+    is( scalar(@srcs), 1, 'from one script' ) or return diag( explain( \@srcs ) );
+    my ( $version, $names ) = $srcs[0] =~ m{\A/bundle/(\w+)/([^/]+)[.]js\z} or return fail("$srcs[0] is a bundle");
+    is( $names, 'highlight.min+reveal.min', 'which bundles what the deck needs, and no editing scripts for a visitor' );
+
+    my ( $bcode, $bundle, $berr, $headers ) = render( anon( route => $srcs[0], version => $version, names => $names ), \&Trog::Routes::HTML::bundle );
+    is( $bcode, 200, 'the bundle is served' ) or return diag($berr);
+    like( $bundle, qr/spec[ ]highlight.*spec[ ]reveal/s, 'with both scripts in it, in order' );
+    my %h = @$headers;
+    is( $h{'Content-type'}, 'text/javascript', 'as javascript' );
+    like( $h{'Cache-Control'}, qr/immutable/, 'that the browser keeps, since the href changes with the body' );
+
+    ($bcode) = render( anon( route => "/bundle/0000/$names.js", version => '0000', names => $names ), \&Trog::Routes::HTML::bundle );
+    is( $bcode, 302, 'an old version is sent to the current one' );
+    ($bcode) = render( anon( route => '/bundle/0000/nothere.js', version => '0000', names => 'nothere' ), \&Trog::Routes::HTML::bundle );
+    is( $bcode, 404, 'and a bundle of nothing is not found' );
+
+    ( $code, $page, $err ) = render( admin( $deck_query->() ), \&Trog::Routes::HTML::posts );
+    is( $code, 200, 'the deck renders for an editor' ) or return diag($err);
+    my $editing = qr{post[+]post_relations};
+    my $viewing = qr{highlight[.]min[+]reveal[.]min};
+    like( $page, qr{<script\b[^>]*\bsrc="/bundle/\w+/$editing[+]$viewing[.]js"}, 'with the editing scripts first' );
 };
 
 subtest 'invoice' => sub {

www/templates/html/components/forms/presentation.tx

@@ -30,7 +30,6 @@
       			<: } :>
       		</div>
     	</div>
-    	<script src="/scripts/reveal.js"></script>
     	<script>
       		Reveal.initialize({
                 hash:true,

www/templates/html/components/post_wizard.tx

@@ -1,4 +1,3 @@
-<script type="text/javascript" src="/scripts/post_wizard.js"></script>
 <script type="application/json" id="wizard-types"><: $types | mark_raw :></script>
 <div id="backoffice">
 <p class="title">

www/templates/html/components/posts.tx

@@ -1,6 +1,4 @@
 : if ( $can_edit ) {
-    <script type="text/javascript" src="/scripts/post.js"></script>
-    <script type="text/javascript" src="/scripts/post_relations.js"></script>
     <div id="emoji_picker_modal" class="modal" style="display:none;">
     <: component('EmojiPicker') :>
     </div>
@@ -36,8 +34,6 @@
     : }
     </div>
 : }
-<script type="text/javascript" src="/scripts/highlight.min.js"></script>
-<script type="text/javascript" src="/scripts/reveal.js"></script>
 <script>
 document.addEventListener("DOMContentLoaded", function(){
     hljs.highlightAll();