A (re-)introduction to the extensible shell

I am excited to announce that version 0.10.0 of es, the extensible shell, has been released.

This is the first release of es in almost four years, and the first time the release hasn’t started with 0.9 in over three decades. So, while this release doesn’t contain any profound changes to the language or interpreter, it should be taken as a signal that activity in es has picked up once again.

For those not already familiar with it, the best introductions to es are the original paper presented at Usenix 1993 or the man page, but I’ll provide a shorter, incomplete introduction to the shell here, the state of the shell after its over three decades of existence, and some ideas on what interesting work there still is to be done.

The current version of es is hosted on the GitHub repository. It is written in a somewhat quirky form of ANSI C, making somewhat heavy use of a preprocessor macros to implement a few core behaviors. It should successfully build using any ANSI C compiler and it should successfully run on any OS which implements a reasonable portion of the POSIX.1-2001 standard.

What is es?

Es is a Unix shell first developed in the early 1990s by Byron Rakitzis and Paul Haahr, based directly on Rakitzis’ earlier port of the shell rc from Plan 9 to Unix. To quote the 1993 es Usenix paper:

[w]hile rc was an experiment in adding modern syntax to Bourne shell semantics, es is an exploration of new semantics combined with rc-influenced syntax: es has lexically scoped variables, first-class functions, and an exception mechanism, which are concepts borrowed from modern programming languages such as Scheme and ML.

Simple es commands closely resemble other shells, with familiar syntax for things such as pipes, redirections, $variables, wildcards, and backgrounding. These things should be especially familiar for those experienced with rc. 

make -p >[2] /dev/null | grep -o '^[^#].*:'

Also like in rc, all es variables are flat lists, and a variable’s value is never split or otherwise mangled unless a specific command like eval is invoked to do just that. This makes the common experience of using variables much more natural than in Bourne-derived shells. 

; args = -l 'Long Document.pdf'
; ls $args
-rw-r--r-- 1 jpco jpco 12345 Aug 31 15:44 'Long Document.pdf'

Where es differs from rc is in its influence from functional programming. In particular, from Scheme, es draws features like higher-order functions and lexical scope. 

fn map cmd args {
	for (i = $args)
		$cmd $i
}
map @ i {cd $i; rm -f *} /tmp /var/tmp

In this example, @ i {cd $i; rm -f *} is a lambda expression—an anonymous, inline function—which takes an argument i, cds to the directory named in $i, and then rm -fs everything in the directory. In this example, it is being used as the first argument (everything between the @ and closing } is parsed as a single term) passed to the map function.

Blocks of code (“code fragments” in es-speak), represented as {commands in curly braces}, can also be stored as terms within variables, or passed as arguments to functions. These blocks of code function like lambda expressions, except that they bind no arguments. 

; var = {echo first} {echo second}
; $var(1)  # take the first term of $var and run it
first
; $var(2)  # take the second term of $var and run it
second

Many constructs within es are implemented as functions which take lambda expressions and/or code fragments as arguments. For example, the builtin command catch, which is core to es’ exception mechanism, takes a lambda expression for its catcher, and a code fragment for its body. 

catch @ e {        # this is the start of the catcher lambda argument
	echo caught $e
} {                # this is the start of the body code fragment argument
	if {!rm $file} {
		throw error rm could not remove file
	}
}

Because so many behaviors in es are modeled internally as function calls, and function definitions are visible to users and changeable at will, the internals of the shell are highly malleable. The following shows how pipes are implemented: when parsed, the | syntax is desugared into a call to the %pipe function, and the %pipe function is itself just a call to the $&pipe primitive, the built-in function which performs the behavior. 

; echo {ls | wc -l}
{%pipe {ls} 1 0 {wc -l}}
; echo $fn-%pipe
$&pipe

With this knowledge, the definition of %pipe can be changed to add custom, extended behaviors, a practice refered to in es parlance as spoofing. A classic example of spoofing, adapted from the Usenix paper, is extending %pipe to time each pipeline command individually. 

; cat pipe-spoof.es
fn %pipe {
	let (args = ()) {
		for ((c i o) = $*)
			args = $args {time $c} $i $o
		$&pipe $args
	}
}
; . pipe-spoof.es  # "load" the spoof into the current shell process
; cat page/es/paper.html.es | tr -cs a-zA-Z0-9 '\012' | sort | uniq -c | sort -nr | sed 6q
 0.001r   0.000u   0.000s	{cat page/es/paper.html.es}
 0.001r   0.000u   0.000s	{tr -cs a-zA-Z0-9 '\012'}
 0.005r   0.004u   0.001s	{sort}
 0.006r   0.001u   0.000s	{uniq -c}
    367 a
    302 the
    286 i
    266 code
    165 to
    162 of
 0.007r   0.001u   0.000s	{sed 6q}
 0.007r   0.002u   0.000s	{sort -nr}

Spoofing is the major mechanism by which users can redefine large swaths of the shell’s internal behavior. For example, the %write-history function is called by the shell to write a command to the shell history after reading it. To make the shell avoid writing duplicate commands to history, one can simply redefine the function as follows: 

let (write = $fn-%write-history; last-cmd = ())
fn %write-history cmd {
	if {!~ $cmd $last-cmd} {
		$write $cmd
		last-cmd = $cmd
	}
}

We can go through this example line-by-line.

  1. let (write = $fn-%write-history; last-cmd = ())

    This creates a (lexical) binding of the current definition of %write-history to the variable write, and of ()—the empty list—to the variable last-cmd.

  2. fn %write-history cmd {

    This creates a new definition of %write-history with one parameter, $cmd. Thanks to the write variable bound with the let in the previous line, the old definition of %write-history is accessible within the body of this function. This is a very common idiom in es, because it allows multiple spoofs of a single function to “stack” with one another with little difficulty.

    Note also that when the last-cmd variable was bound, the binding was created outside of this function. That means the binding, and therefore the value of last-cmd, persists across function calls.

  3. if {!~ $cmd $last-cmd} {

    This compares $cmd against $last-cmd. If they differ, then...

  4. $write $cmd

    We call the previous definition of %write-history on our new $cmd.

  5. last-cmd = $cmd

    Then, we set last-cmd to $cmd. Because last-cmd persists across calls to %write-history, this saves this value to compare against future values of $cmd.

This bit of code implements what in Bash would be achieved using HISTCONTROL=ignoredups, and it’s reasonable to note that for this specific case, the Bash version is quite a bit more concise and easier to configure than the es version is to script up.

But that’s because this is a specific feature, one of a great many, that has been pre-implemented in Bash. HISTCONTROL is a colon-separated ($PATH-style) list which can contain one of a few special, hardcoded tokens, which correspond with particular behaviors around writing shell history (an exercise for the reader: without looking it up, what are all the possible tokens?) It is semi-redundant with another special variable, HISTIGNORE, which controls yet further behaviors, configured using a different kind of colon-separated ($PATH-style) list, with syntax unique to the HISTIGNORE variable (More exercises for the reader: without looking it up, what is that syntax? In what cases are HISTCONTROL and HISTIGNORE redundant? In those cases, which is preferable to use?) There is a lot to know about these two variables.

Es takes a different approach instead. There is more knowledge required to hand-write the equivalent of HISTCONTROL=ignoredups, but most of that knowledge is general: spoofing functions, saving state across function calls, comparing values. These are some of the tools of es scripting; once somebody is proficient in writing es scripts, they’re proficient in customizing es.

The es approach also enables a fundamentally greater degree of flexibility, because an es function can do anything es can do. When Paul and Byron exposed the shell’s interactive REPL as the function %interactive-loop, they also exposed the non-interactive REPL as %batch-loop, despite having reservations that actually modifying the non-interactive REPL would ever be a good idea. Doing so, however, made it possible to write an alternate version of %batch-loop that parses an entire script before running any of its commands, which is a convenient way to “sanity-check” code and avoid running half of a buggy script.

This degree of flexibility is remarkable. The kind of pre-parsing performed here was never implemented or even considered by the authors of es, but it’s just as possible as it is under other shells which officially advertise the behavior.

What’s been happening with es?

Es was mostly developed over the course of 1992-1995. The bulk of development went through the release of version 0.84; 0.88 was released after the authors had taken a break, and then after that release both of them got too busy with life and jobs to continue work on the shell.

After that, maintainership passed through a couple hands, leading eventually to the current maintainer James Haggerty, but that maintainership was largely focused on keeping es basically functional over the decades as OSes, build systems, and code-hosting practices have evolved.

This left es as an incomplete experiment: Paul and Byron didn’t have time to achieve a good amount of what they planned on, and even if they had, their near- to medium-term plans certainly didn’t sum up to everything the shell could be made to do.

Recently, however, there has been more activity, which has just been bundled up in version 0.10.0. This new version of es is not significantly different as a language from prior versions, but it contains a good number of bug fixes. Something like 20 PRs have been merged to fix different ways to crash the shell, and each way now has automated regression testing run on every new PR written. Portability has also been significantly improved, as obsolete portability-oriented code has been removed and the es runtime has been moved onto the very widely-supported functions in the POSIX.1-2001 standard.

There has also been a large collection of small improvements: readline integration is better, supporting variables and primitives, and writing to history has been tweaked. Most of the left-over implications of assignments returning the assigned value have been tidied up. Waiting, process group handling, and terminal assignment have all been fixed up to be made more predictable, as has signal handling.

One other change is the addition of “canonical extension” scripts. These are scripts distributed (and installed) with es, not built into the shell itself, but available as officially supported implementations of certain spoofs. The initial canonical extensions distributed with the shell are

In addition, there has been some refactoring of the internals in order to support larger near-term changes…

Es futures

So, what’s next for es? Well, there are a couple of near-term goals I would like to achieve.

Shell input has recently been refactored so that it can be meaningfully extended like other parts of the shell, but this hasn’t yet been exploited. For example, there’s no mechanism for programmable tab completion, and there’s no alternative to readline integration, such as editline, linenoise, replxx, linecook, or even something like an es version of ble.sh. Essentially, it is finally possible to make es input extensible, but I want to actually follow through.

I would also like to add some form of support for job control to the shell. There is a long history of religious arguments about job control in both es and rc, and I admit that I find many shells’ abstraction of a “job” to be more obnoxious than useful. Despite that, the ability to manage process groups—which is the real, core functionality of job control—is something that should be possible for any competent Unix shell, and I believe that it can be possible for es with only small extensions to existing shell mechanisms. Once that support is added, then a simple job-control.es script can be easily added as a canonical extension.

Both of these ideas are in service of a larger goal, which is to grow the es community. Es, I think, has real design strengths which have appealed to people such as myself despite the shell’s current limitations. Removing those limitations and allowing people to interact with their shell in ways that are familiar to them (that is, using job control and fancy programmable input), and managing to do so in ways that are consistent with the shell’s design objectives, serves to both make the shell more practically useful and demonstrate that the design works.

Ideally though, I don’t want to add too much to what’s built into the upstream es. The current feature set is pretty good, and I think it’s right to have a shell that starts small and lets users build on that. Whatever is added to the es that everybody uses should exist to enable people to add the features they need.

Outside of actual development work, I intend to write more posts to document various aspects of es, making it easier to get a strong grasp of the shell without having to dive into the codebase or trawl the old mailing list just to have an idea of how certain things work or why they’re implemented the way they are. (It is an opinion of mine that somebody shouldn’t have to learn C just to learn es.)

Some pages I ought to get around to writing include:

All in all, I’d like to help build a solid enough foundation for es, along with documentation and tooling support, that it significantly lowers the barrier to entry for hacking on the shell. Over the years, while the upstream shell has been quiet, multiple individual forks have spun up, proving that motivation to do things with es has never really gone away, even if it has been scattershot. If upstream es can better organize these contributions to enable people to more effectively build on each others’ work, I believe it would be a huge benefit to the momentum of the shell and its community.

I believe that the continuing endurance of es is directly due to the fact that, even as an old and incomplete experiment, it is still a shining example of software design. It is simple, powerful, and predictable; it can be used to host a web site or function at the core of a desktop. All I want to do is continue to develop upon its strengths, and enable others to do so as well.