A (re-)introduction to the extensible shell

Es is the extensible shell.

The best (if slightly out-of-date) introductions to the shell are the original es paper presented at Usenix 1993 or the es man page, but I'll provide a shorter, incomplete introduction to the shell here, the state of the shell after three decades, and some ideas on what interesting work there is left to be done.

The current version of es is hosted on the GitHub repository.

What is es?

Es is a Unix shell first developed in the early 1990s by Byron Rakitzis and Paul Haahr, based directly on Rakitzis' earlier port of the shell rc from Plan 9 to Unix. As the paper puts it,

[w]hile rc was an experiment in adding modern syntax to Bourne shell semantics, es is an exploration of new semantics combined with rc-influenced syntax: es has lexically scoped variables, first-class functions, and an exception mechanism, which are concepts borrowed from modern programming languages such as Scheme and ML.

Simple es commands closely resemble other shells, with pipes, redirections, $variables, wildcards, backgrounding, and more. Redirection syntax particularly resembles rc. 

make -npq >[2] /dev/null | grep '.*:'

Also like rc, es has list-typed variables, no implicit rescanning, and no double quotes. These together make variables significantly more straightforward to use than in POSIX-compatible shells. 

; args = -l 'Long Document.pdf'
; ls $args
-rw-r--r-- 1 jpco jpco 12345 Aug 31 15:44 'Long Document.pdf'

From Scheme, es draws features like first-class functions and lexical scope. 

fn map cmd args {
  for (i = $args)
    $cmd $i
}
map @ i { cd $i; rm -f * } /tmp /var/tmp

In this example, @ i { cd $i; rm -f * } is a lambda expression—an inline function—which takes a argument i, cds to it, and then rm -fs everything in the directory.

Nearly everything in es is a function under the hood, and functions are just variables whose names start with fn-. Like other variables, functions can be redefined. 

; echo {command > file}
{%create <={%one file} {command}}
; echo $fn-%create
%openfile w
; # this is not very useful
; fn-%create = echo
; command > file
1 file {command}

This lets users redefine large swaths of the shell's behavior. For example, the %write-history function is called by the shell to write a command to history after reading it. To make the shell avoid writing duplicate commands to history, one can write: 

let (write = $fn-%write-history; last-cmd = ())
fn %write-history cmd {
  if {!~ $cmd $last-cmd} {
    $write $cmd
    last-cmd = $cmd
  }
}

We can go through this example line-by-line.

  1. let (write = $fn-%write-history; last-cmd = ())

    This creates a lexical binding of the variable write to the current definition of %write-history, and of the variable last-cmd to () (the empty list).

  2. fn %write-history cmd {

    This creates a new definition of %write-history. Thanks to the write variable bound with the let in the previous line, the old definition of %write-history is accessible within this function. This is a very common idiom in es, used for “spoofing” functions, or creating new definitions to suit preferences or create situational benefit.

    That let also bound the last-cmd variable; it's only bound to the empty list initially, but because that binding is created outside the body of the function, its value will persist across calls.

  3. if {!~ $cmd $last-cmd} {

    This compares $cmd against $last-cmd. If they differ, then...

  4. $write $cmd

    We call the old %write-history on the new $cmd.

  5. last-cmd = $cmd

    Then, we set last-cmd to $cmd. Because last-cmd persists across function calls, this effectively saves this command for future calls.

This bit of code implements what in Bash would be achieved using HISTCONTROL=ignoredups, and it's reasonable to note that the Bash version is quite a bit more concise than the es one. However, the es method has its own major benefits “at scale”, when considering shell features in aggregate.

Bash, and many other shells, add behaviors and customization via special variables and options, with their own special values and little languages, requiring a good deal of memorization of a large “menu” of tweaks and tricks to make the shell work as desired.

Es takes a different approach instead, exposing the core behaviors of the shell in a way that allows customization using the same scripting techniques that would, largely, be used for anything else in the shell. Once somebody is proficient in writing es scripts, they're proficient in customizing es.

The es approach also enables a fundamentally greater degree of flexibility. When Paul and Byron exposed the shell's interactive REPL as the function %interactive-loop, they also exposed the non-interactive REPL as %batch-loop, despite having reservations that actually modifying the non-interactive REPL would ever be a good idea. Doing so, however, makes it possible to write an alternate version of %batch-loop that parses an entire script before running any of its commands, which is a convenient way to “sanity-check” code and avoid running just half of a buggy script.

This degree of flexibility is remarkable: this kind of pre-parsing was never implemented or even considered by the authors of es, but it's just as possible as it is under other shells which officially advertise the behavior.

What's been happening with es?

Es was mostly developed over the course of 1992-1995. The major development went through the release of version 0.84; 0.88 was released after a hiatus from the original authors, and then both of them got too busy with life and “real” work to continue on the shell.

After that, maintainership passed through a couple hands, leading eventually to the current maintainer James Haggerty, but development was largely focused on keeping es functional and available over the decades as OSes, build systems, and code-hosting practices have evolved.

This has left the shell as an incomplete experiment: Paul and Byron didn't have time to achieve a good amount of what they planned on, and even if they had, their near- to medium-term plans certainly didn't sum up to everything the shell could be made to do.

However: at its core es has a simple and powerful design which removes a huge amount of the friction of shell scripting. Its ethos of providing fewer and more powerful language and runtime mechanisms makes it relatively easy to know top to bottom, and surprisingly easy to modify its internals. It is, genuinely, an extremely elegant piece of software that I am very glad to use every day.

Es futures

So what would best be done with es now?

There are a few major themes where I would like to see improvement, and would be willing to dedicate effort to make that happen.

First of all, I would love to get more usage in the shell. As more people use es, more creativity is applied to using and customizing the shell, and benefits of its flexibility compound. Packaging es for more OSes and Linux distros will help, as would more writing about the shell and more documentation online.

Quite a bit of existing knowledge about es is wrapped up inside the old mailing list or the source code, and users shouldn't be reasonably expected to dig around GitHub or years worth of old mail to understand a piece of software enough to use it effectively.

Tooling support would be helpful as well; syntax highlighting for editors, maybe even some kind of LSP integration, as well as reviving (and documenting) the esdebug script.

I would also like to close the gaps where es is unable to perform common shell behaviors today. It's not wrong for es to be small and minimal by default, but a shell that's supposedly extensible should be able to support, say, job control, or customizable interactive behaviors.

I am also interested in pushing es' extensibility even further. While the shell is already extensible, some major chunks of the shell are hard-coded in ways that they don't have to be; for example, most of the existing main() function could be scripted within the shell.

The primitives which back most es commands can also be made extensible through dynamic library loading, which has been well standardized and is supported across Unices. This would allow the shell to perform novel and OS-specific behaviors, like interacting with networks, performing GUI scripting, or sandboxing chunks of scripts. Careful design around versioning could also enable a good backwards-compatibility story without hamstringing the shell's ability to change over time.

Lastly, there is a lot of opportunity to make improvements to the runtime to support all of the above, as well as to make the shell faster to run. Es was never optimized in either runtime or memory to a meaningful degree, so there is significant low-hanging fruit there.

In particular, drawing from the rich tradition of Scheme interpretation methods could enable powerful things like tail-call optimization, better exception support without setjmp(3)/longjmp(3) (enabling better cross-language interaction), more efficient memory use, improved speed, and even features like continuations or lightweight threading.

To write