Serving a web site from a shell script is fun and easy

2026-01-10

This web site is served from an es script. That’s a pretty unique choice among websites, so it might be worth a little explanation as to why I did that, and why I haven’t gotten annoyed about the choice. We’ll also go over some of the details of the script itself, at least in the form it takes as of this time of writing. The server code is here, and the git repository, including all the page sources and other content, is hosted here.

A tour of the server

The server loop

Right at the beginning of the server is the most important part. It’s what makes the whole thing go. After a bit of messing about to get a $server-port depending on if the shell was started in a Docker container, we have:

if {~ $NCAT_LOCAL_PORT ()} {
	forever {ncat -k -l -p $server-port -e $0 || exit 3}
}

This bit of code is controlled by the $NCAT_LOCAL_PORT variable, which is unset when the script is originally invoked, but set later by ncat. Because of this, the top-level script just runs an infinite loop of ncat calls.

This ncat is the part that handles the actual TCP networking. Sorry if you thought the shell would natively handle that—es isn’t quite capable of something like that yet. In particular, this command is written for the ncat distributed as part of the nmap project; certain other versions of netcat lack the -e argument this server depends on.

The exact invocation looks like this:

ncat -k -l -p $server-port -e $0

The -k and -l flags are what make ncat run as a TCP server. The -p $server-port configures the port on which to listen. This is set to 8080 when running within Docker, since that’s the standard, and 8181 when running outside a Docker container, since that’s more likely to be free on an arbitrary host.

The last flag, -e, configures ncat to execute a command when a request is received. The command can read its standard input to look at the request, and can write to standard output to specify the response. ncat is actually pretty clever about this—instead of buffering the response in memory, it sends it as the child process writes to stdout; this helps pages start rendering in browsers even if the server gets hung up while producing them.

The argument we give to -e is the script itself, stored in $0. When we invoke the subcommand, because ncat has set $NCAT_LOCAL_PORT, we skip running the server loop and instead move on to the rest of the script which handles the individual requests.

Request handling

While ncat does all the hard work of handling TCP networking, it doesn’t actually do anything about HTTP, so that has to be implemented in the script. So the first thing we do is define a respond function which takes a numeric status code code, a MIME type type, and optional flags to control things like caching or compression, and uses those arguments to print the headers of the reply.

Then we define a couple helper functions to serve a “page”, which is what we call our custom templated HTML files with smatterings of es code thrown in. As an example, we can look at the actual source of the server source code page:

<; cat tmpl/header.html >

<title>jpco.io | The script that served this title</title>
<meta name=description content="The script that served this description">

<; build-nav /server.html >

<main>
<p>
This is the script that served this request, written in <a href=/es><i>es</i></a>.

<pre id=main-block>
<; sed -e 's/&/\&amp;/g' -e 's/</\&lt;/g' < serve.es >
</pre>
</main>

Using this extremely basic templating system we give ourselves access to the shell within the page, and we use that to add the page header, the navigation bar, and print the server script. The templates, such as they are, are the most obviously deficient part of this whole setup, but are good enough to serve what this site actually needs, which isn’t much.

The build-page function reads these templatized files line by line and prints the output or runs the command for each line as appropriate. The function build-nav, which is called from the page, is defined in the server script as it is nearly universal, and simply prints some HTML which formats the path argument given to it. Then the serve-page function simply wraps up respond and build-page into a single convenient function call.

After these helper functions are defined, we get to the business of handling the request. We read the method, the path (which we call reqpath to avoid colliding with the normal $path), and the HTTP version:

(method reqpath version) = <={%split ' ' <={~~ <=%read *\r}}

We have to handle the \r\ns in the request explicitly, which is annoying, but not too much of a problem. Fortunately, for responses, ncat inserts \rs as necessary so we don’t have to think about them in our echo calls. After the first line, and a bit of handling for query strings, we move on to reading the headers.

# this whole bit is structured to try to minimize the number of fork/execs
let (header-names = (); header-values = ()) {
	# read in headers
	while {!~ <={header = <=%read} \r} {
		let ((n v) = <={~~ $header *': '*\r})
		if {!~ $#n 0} {
			header-names = $header-names $^n
			header-values = $header-values $^v
		}
	}
	# convert to lowercase, if necessary
	if {~ $header-names *[A-Z]*} {
		local (lhns = $header-names) {
			eval `` '' {var lhns | awk '{print tolower($0)}'}
			header-names = $lhns
		}
	}
	# set the header variables
	for (n = $header-names; v = $header-values)
		head-$n = $v
}

Here we read in headers and save the header values within variables of the form head-$name, so that $head-host contains something like jpco.io. There is some extra handling to enforce that case-insensitive headers (host, Host, hOsT) are all made into lower-case variables ($head-host); this uses awk, which incurs some overhead due to the extra fork/execs, but is structured to limit the number of necessary calls to either zero or one.

Routing

After reading in the request, we have everything we need to serve the response. The whole router is just one big if statement.

if (
	# redirect www.jpco.io to jpco.io
	{~ $head-host www.jpco.io} {
		destination = https://jpco.io$reqpath
		respond 301 text/plain
		echo Redirecting to $destination ...
	}

	# draft built pages; only serve these locally
	# before "real" pages so we can draft changes too
	{!$IN_DOCKER && access -f draft/$reqpath^.es} {
		serve-page draft/$reqpath^.es
	}
	{!$IN_DOCKER && access -f draft/$reqpath/index.html.es} {
		serve-page draft/$reqpath/index.html.es
	}

	# built pages. don't cache these
	{access -f page/$reqpath^.es} {
		serve-page page/$reqpath^.es
	}
	{access -f page/$reqpath/index.html.es} {
		serve-page page/$reqpath/index.html.es
	}

	# static files
	{access -f static/$reqpath} {
		serve static/$reqpath cache
	}

	# 404
	{
		respond 404 text/html
		build-page &lt; page/404.html.es $reqpath
	}
)

Here’s where it all comes together.

If the request is coming to www.jpco.io, redirect it to jpco.io.
If we’re in “dev mode” and not in a Docker container, serve the request as a page if it matches a file in the draft directory.
If the request matches a file in the page/ directory, serve it as a dynamically-built page.
If the request matches a static resource, serve it verbatim.
Otherwise, serve the 404 page with a 404 code, since we didn’t find anything.

How it’s run

When I’m working on changes to the site, I can run this script as

; ./serve.es

and it works great. Pages are always served live, so all I have to do is save the page I’m working on and reload. The server is also always served live thanks to ncat -e $0, so unless I’m making a change to the small server-loop section at the very top, I don’t even need to re-run the server script after making a change.

In “prod”, I package up the contents of the repo from HEAD as well as a fresh es built from HEAD and the couple of binary dependencies (ncat, man, file) into a Docker container and serve it from Google Cloud Run. Cloud Run takes care of details around HTTPS for me, which is nice. Building and deploying a new version is done with the deploy.es script, which runs something like:

gcloud builds submit --tag gcr.io/jpco-cloud/web:0.76 .
gcloud run deploy --platform managed --image=gcr.io/jpco-cloud/web:0.76

I won’t go into the Dockerfile these commands use since it’s pretty extremely basic, but it’s in the repository for this site if anybody really wants to take a look.

Okay… but why?

This is obviously not a very good general web server. It’s relatively slow in the first place compared to something in a so-called blazingly-fast language, and I imagine it scales pretty poorly.

But none of that actually matters. I didn’t write this server to serve any web site, I wrote it to serve this web site, and this web site is really pretty dead simple, and it doesn’t get very much traffic at all, so I don’t care about complicated server-side logic, templating, or the degree to which the fast is blazing.

What I really want is exactly what this server gives me. I want a really convenient environment to write new pages in without bothering with any sort of recompilation flow. I want a router that is extremely simple but more flexible than a pure directory structure-based setup. And I want all of it without some kind of goofy toolchain, framework, or runtime dependencies that do more to get in my way than help me serve this extremely simple site. I’m not a web developer so whenever I’m not actively working on this site, I’m not really thinking about any web technologies, so using fancy special-purpose tools is a net increase to my cognitive load, not the other way around.

Admittedly, there’s also some aesthetic joy to it. I prefer a website that’s pretty bare, and I like to stay “close to the metal” of HTTP. I like to have that little bit of extra control, since I’m not doing anything particularly fancy or high-stakes. And, honestly, I also just like to be able to say that I’m serving my personal web site from a shell script.

Related work

I am, of course, far from the first person to want to use a shell script to serve web pages. Following are a few relevant example projects which try to be general web frameworks or servers written using different shells. This site’s server is slightly different than these general libraries, being intended as a special-purpose single-site server, but the other projects have interesting lessons to teach besides.

Bash on Balls

When I first mentioned this site to a buddy, he said “oh, like balls?” and then after a brief second of confusion linked to balls.

Balls (Bash on Balls) is a whole web “framework” written in Bash, including CGI and other support, but at its core, it has a similar loop as this script, using a (more traditional) nc pipeline to manage networking:

http_sock=$BALLS_TMP/balls.http.$$.sock
[ -p $http_sock ] || mkfifo $http_sock

while true; do
	cat $http_sock | nc -l -p $BALLS_PORT | (
		http::parse_request
		balls::route > $http_sock
	)
done

The core server loop in balls, using nc.

Requests that nc receives it prints to its standard output, and anything it receives on standard input it sends as a reply. This requres the $http_sock file exist so that the balls::route function can communicate with the nc command earlier in the pipeline. I do think that this could be done in a more “advanced bash” way these days using a coprocess, but Bash’s big fancy networking feature, its socket programming /dev/tcp paths, apparently can’t be used to listen for connections.

ZWS

An even older project exists called ZWS which does something very similar in zsh. This project is mostly notable because zsh’s kitchen-sink nature means that it has a module, zsh/net/tcp, which contains a large number of functions related to TCP, including one, tcp_proxy, which performs logic like ncat -e entirely as a shell built-in.

zmodload zsh/net/tcp
autoload -U tcp_proxy

tcp_proxy $opts[-p] serve

The core server loop in ZWS, using tcp_proxy and the serve function.

If es were to add loadable primitives or a module system of some kind, networking would be an early use case to explore, and zsh’s precedent seems especially relevant. Notably, the actual built-in part of zsh’s TCP handling only seems to define the ztcp command, and everything else is a function adding sugar on top.

werc and rc-httpd

The last, and most relevant, work in this space is the werc web “anti-framework” used heavily in the Plan 9 space these days. Werc is not itself a web server, but it does come packaged with the rc-httpd server, which is very similar to the one used for this site.

A particular distinction of rc-httpd is that it does not do (or control) the networking or main loop at all, but instead simply expects to be called with files already managed for each request; typically, presumably, because this would be done with one of the listen(8) utilities in Plan 9. Like a lot of the design choices made in Plan 9, the outcome seems to be a net simplification of each program.

If there were a “web framework” written for es, then werc would be the obvious model to follow first.