A (re-)introduction to the extensible shell

I am excited to announce that version 0.10.0 of es, the extensible shell, has been released.

This is the first release of es in almost four years, and the first time the release hasn’t started with 0.9 in over three decades. So, while this release doesn’t contain any profound changes to the language or interpreter, it should be taken as a signal that something has changed in es, and that right now is a great time to try picking it up.

For those not already familiar with it, the best introductions to es are the original paper presented at Usenix 1993 or the man page, but I’ll provide a shorter, incomplete introduction to the shell here, the state of the shell after its over three decades of existence, and some ideas on what interesting work there still is to be done.

The current version of es is hosted on the GitHub repository. It is written in ANSI C, though a somewhat quirky form of C due to some extensive preprocessor macro use to implement some of its features. It should successfully build with any ANSI C compiler and it should successfully run on any OS which implements a reasonable portion of the POSIX.1-2001 standard.

What is es?

Es is a Unix shell first developed in the early 1990s by Byron Rakitzis and Paul Haahr, based directly on Rakitzis’ earlier port of the shell rc from Plan 9 to Unix. To quote the 1993 es Usenix paper:

[w]hile rc was an experiment in adding modern syntax to Bourne shell semantics, es is an exploration of new semantics combined with rc-influenced syntax: es has lexically scoped variables, first-class functions, and an exception mechanism, which are concepts borrowed from modern programming languages such as Scheme and ML.

Simple es commands closely resemble other shells, with familiar syntax for things such as pipes, redirections, $variables, wildcards, and backgrounding. These things should be especially familiar for those experienced with rc. 

make -p >[2] /dev/null | grep -o '^[^#].*:'

Also like in rc, all es variables are flat lists, and a variable’s value is never split or otherwise mangled unless a specific command like eval is invoked to do just that. This makes the common experience of using variables much more natural than in Bourne-derived shells. 

; args = -l 'Long Document.pdf'
; ls $args
-rw-r--r-- 1 jpco jpco 12345 Aug 31 15:44 'Long Document.pdf'

Where es differs from rc is in its influence from functional programming. In particular, from Scheme, es draws features like higher-order functions and lexical scope. 

fn map cmd args {
	for (i = $args)
		$cmd $i
}
map @ i {cd $i; rm -f *} /tmp /var/tmp

In this example, @ i {cd $i; rm -f *} is a lambda expression—an anonymous, inline function—which takes an argument i, cds to the directory named in $i, and then rm -fs everything in the directory. Blocks of code (“code fragments” in es-speak), represented as {commands in curly braces}, can also be passed around in variables or passed as arguments to functions. These blocks of code function like lambda expressions, except that they bind no arguments. 

; var = {echo first} {echo second}
; $var(1)  # invoke the first element of $var
first
; $var(2)  # invoke the second element of $var
second

Many constructs within es are implemented as functions which take lambda expressions and/or code fragments as arguments. For example, the builtin command catch, which is core to es’ exception mechanism, takes a lambda expression for its catcher, and a code fragment for its body. 

catch @ e {        # this is the start of the catcher lambda argument
	echo caught $e
} {                # this is the start of the body code fragment argument
	if {!rm $file} {
		throw error rm could not remove file
	}
}

Because so many behaviors in es are modeled internally as function calls, the internals of the shell are easily visible and changeable. The following shows how pipes are implemented: when parsed, the | syntax is desugared into a call to the %pipe function, and the %pipe function is itself just a call to the $&pipe primitive, the built-in function which performs the behavior. 

; echo {ls | wc -l}
{%pipe {ls} 1 0 {wc -l}}
; echo $fn-%pipe
$&pipe

With this knowledge, the definition of %pipe can be changed to add custom, extended behaviors, a practice refered to in es parlance as spoofing. A classic example of spoofing, adapted from the Usenix paper, is extending %pipe to time each pipeline command individually. 

; cat pipehook.es
fn %pipe {
	let (args = ()) {
		for ((c i o) = $*)
			args = $args {time $c} $i $o
		$&pipe $args
	}
}
$*
; es pipehook.es {cat page/es/paper.html.es | tr -cs a-zA-Z0-9 '\012' | sort | uniq -c | sort -nr | sed 6q}
 0.001r   0.000u   0.000s	{cat page/es/paper.html.es}
 0.001r   0.000u   0.000s	{tr -cs a-zA-Z0-9 '\012'}
 0.005r   0.004u   0.001s	{sort}
 0.006r   0.001u   0.000s	{uniq -c}
    367 a
    302 the
    286 i
    266 code
    165 to
    162 of
 0.007r   0.001u   0.000s	{sed 6q}
 0.007r   0.002u   0.000s	{sort -nr}

Spoofing lets users redefine large swaths of the shell’s internal behavior. For example, the %write-history function is called by the shell to write a command to the shell history after reading it. To make the shell avoid writing duplicate commands to history, one can simply redefine the function as follows: 

let (write = $fn-%write-history; last-cmd = ())
fn %write-history cmd {
	if {!~ $cmd $last-cmd} {
		$write $cmd
		last-cmd = $cmd
	}
}

We can go through this example line-by-line.

  1. let (write = $fn-%write-history; last-cmd = ())

    This creates a (lexical) binding of the current definition of %write-history to the variable write, and of ()—the empty list—to the variable last-cmd.

  2. fn %write-history cmd {

    This creates a new definition of %write-history with one parameter, $cmd. Thanks to the write variable bound with the let in the previous line, the old definition of %write-history is accessible within the body of this function. This is a very common idiom in es, because it allows multiple spoofs of a single function to “stack” with one another with little difficulty.

    Note also that when the last-cmd variable was bound, the binding was created outside of this function. That means the binding, and therefore the value of last-cmd, persists across function calls.

  3. if {!~ $cmd $last-cmd} {

    This compares $cmd against $last-cmd. If they differ, then...

  4. $write $cmd

    We call the previous definition of %write-history on our new $cmd.

  5. last-cmd = $cmd

    Then, we set last-cmd to $cmd. Because last-cmd persists across calls to %write-history, this saves this value to compare against future values of $cmd.

This bit of code implements what in Bash would be achieved using HISTCONTROL=ignoredups, and it’s reasonable to note that for this specific case, the Bash version is quite a bit more concise and easier to configure than the es version is to script up.

But that’s because this is a specific feature, one of a great many, that has been pre-implemented in Bash. HISTCONTROL is a colon-separated ($PATH-style) list which can contain one of a few special, hardcoded tokens, which correspond with particular behaviors around writing shell history (an exercise for the reader: without looking it up, what are all the possible tokens?) It is semi-redundant with another special variable, HISTIGNORE, which controls yet further behaviors, configured using a different kind of colon-separated ($PATH-style) list, with syntax unique to the HISTIGNORE variable (More exercises for the reader: without looking it up, what is that syntax? In what cases are HISTCONTROL and HISTIGNORE redundant? In those cases, which is preferable to use?) There is a lot to know about these two variables.

Es takes a different approach instead. There is more knowledge required to hand-write the equivalent of HISTCONTROL=ignoredups, but most of that knowledge is general: spoofing functions, saving state across function calls, comparing values. These are some of the tools of es scripting; once somebody is proficient in writing es scripts, they’re proficient in customizing es.

The es approach also enables a fundamentally greater degree of flexibility, because an es function can do anything es can do. When Paul and Byron exposed the shell’s interactive REPL as the function %interactive-loop, they also exposed the non-interactive REPL as %batch-loop, despite having reservations that actually modifying the non-interactive REPL would ever be a good idea. Doing so, however, made it possible to write an alternate version of %batch-loop that parses an entire script before running any of its commands, which is a convenient way to “sanity-check” code and avoid running half of a buggy script.

This degree of flexibility is remarkable: this kind of pre-parsing was never implemented or even considered by the authors of es, but it’s just as possible as it is under other shells which officially advertise the behavior.

What’s been happening with es?

Es was mostly developed over the course of 1992-1995. The bulk of development went through the release of version 0.84; 0.88 was released after the authors had taken a break, and then after that release both of them got too busy with life and jobs to continue work on the shell.

After that, maintainership passed through a couple hands, leading eventually to the current maintainer James Haggerty, but that maintainership was largely focused on keeping es basically functional over the decades as OSes, build systems, and code-hosting practices have evolved.

This left es as an incomplete experiment: Paul and Byron didn’t have time to achieve a good amount of what they planned on, and even if they had, their near- to medium-term plans certainly didn’t sum up to everything the shell could be made to do.

Recently, however, there has been more activity, which has just been bundled up in version 0.10.0. This new version of es is not significantly different as a language from prior versions, but it contains a good number of bug fixes. Something like 20 PRs have been merged to fix different ways to crash the shell, and each way now has automated regression testing run on every new PR written. Portability has also been significantly improved, as obsolete portability-oriented code has been removed and the es runtime has been moved onto the very widely-supported functions in the POSIX.1-2001 standard.

There has also been a large collection of small improvements: readline integration is better, supporting variables and primitives, and writing to history has been tweaked. Most left-over implications of assignments returning the assigned value have been tidied up. Waiting, process group handling, and terminal assignment have all been fixed up to be made more predictable, as has signal handling.

One other change is the addition of “canonical extension” scripts. These are scripts distributed (and installed) with es, not built into the shell itself, but available as officially supported implementations of certain spoofs. The initial canonical extensions are

In addition, there has been some refactoring of the internals in order to support larger near-term changes…

Es futures

So, what’s next for es? Well, there are a couple of near-term goals I would like to achieve.

I would like to improve how shell input is read and parsed. Es has long had support for readline, but that support is limited, because while the parser is running, the shell can’t run commands written in es script—only hardcoded behavior. Some work that has been recently done with how parser memory is managed changes this, and will enable things like programmable tab completion, or even swapping out readline for other libraries entirely. Given that there are multiple es forks featuring custom, hand-rolled line editing libraries, making this easier to swap out seems like meeting an active desire of users.

I would also like to add some form of support for job control to the shell. There is a long history of religious arguments about job control in both es and rc, and I admit that I find many shells’ abstraction of a “job” to be more obnoxious than useful. However, managing process groups, which is the core of job control, is something that should be possible for any competent Unix shell, and I believe that it can be possible for es with small extensions to existing shell mechanisms; particularly, the $&newpgrp and $&wait primitives. Once that support is added, then a simple job-control.es script can be easily added as a canonical extension.

Both of these ideas, programmable input and job control, are in large part in service of a larger goal, which is to grow the es community. Es, I think, has real design strengths which have appealed to people like myself even despite other limitations of the shell, and during periods when development was stalled. Removing those limitations and allowing people to interact with their shell in ways that are familiar to them (that is, job control and fancy programmable input), and managing to do so in ways that are consistent with the shell’s existing design, serves to both make the shell more practically useful and demonstrate that its design works.

Ideally though, I don’t want to add too much to what’s built into the upstream es. The current feature set is pretty good, and I think it’s right to have a shell that starts small and lets users build on that, rather than the other way around. Whatever is added to the core, upstream shell should function as a sort of meta feature, enabling not just this or that particular use but a whole kind of extensibility or programmability.

Outside of actual development work, I intend to write more posts to document aspects of es, making it easier to get a strong grasp of the shell without having to dive into the codebase or trawl the old mailing list just to have an idea of how certain things work or why they’re implemented the way they are.

Some pages I ought to get around to writing include:

All in all, I’d like to help build a solid enough foundation for es, along with documentation and tooling support, that it lowers the barrier to entry for hacking on the shell significantly. Over the years, while the upstream shell has been quiet, multiple individual forks have spun up, proving that motivation to do things with es has never really gone away, even if it has been disorganized. If upstream es can better avoid all that effort hitting dead-ends in defunct personal forks, that would be fantastic for the shell and its community.

I believe that the continuing endurance of es is directly due to the fact that, even as an old and incomplete experiment, it is still a shining example of software design. It is simple, powerful, and predictable; it can be used to host a web site or function at the core of a desktop. Es is an elegant piece of software that I’m happy to use every day.