A (re-)introduction to the extensible shell

Es is the extensible shell.

The best (if slightly out-of-date) introductions to the shell are the original es paper presented at Usenix 1993 or the es man page, but I'll provide a shorter, incomplete introduction to the shell here, the state of the shell after three decades of existence, and some ideas on what interesting work there still is to be done.

The current version of es is hosted on the GitHub repository. It is written in ANSI C, though a somewhat quirky form due to some extensive preprocessor macro use implementing some of its features. It should be portable to any OS which implements a reasonable portion of the POSIX.1-2001 standard.

What is es?

Es is a Unix shell first developed in the early 1990s by Byron Rakitzis and Paul Haahr, based directly on Rakitzis' earlier port of the shell rc from Plan 9 to Unix. As the Usenix paper puts it,

[w]hile rc was an experiment in adding modern syntax to Bourne shell semantics, es is an exploration of new semantics combined with rc-influenced syntax: es has lexically scoped variables, first-class functions, and an exception mechanism, which are concepts borrowed from modern programming languages such as Scheme and ML.

Simple es commands closely resemble other shells, with the typical syntax for things such as pipes, redirections, $variables, wildcards, and backgrounding. The redirection syntax particularly resembles rc. 

make -p >[2] /dev/null | grep -o '^[^#].*:'

Also like rc, es has list-typed variables, no automatic rescanning, and no double quotes. These together make variables significantly more straightforward to use than in POSIX-compatible shells. 

; args = -l 'Long Document.pdf'
; ls $args
-rw-r--r-- 1 jpco jpco 12345 Aug 31 15:44 'Long Document.pdf'

Where es differs from rc is in its influence from functional languages. In particular, from Scheme, es draws features like higher-order functions and lexical scope. 

fn map cmd args {
  for (i = $args)
    $cmd $i
}
map @ i {cd $i; rm -f *} /tmp /var/tmp

In this example, @ i {cd $i; rm -f *} is a lambda expression—an inline function—which takes an argument i, cds to the directory named in $i, and then rm -fs everything in the directory. Blocks of code (“code fragments”) can also be passed around in variables, represented as {commands in curly braces}. These blocks of code function like lambda expressions, except that they bind no arguments. 

; var = {echo first} {echo second}
; $var(1)  # invoke the first element of $var
first
; $var(2)  # invoke the second element of $var
second

Nearly everything in es is a function under the hood, and functions are just variables whose names start with the prefix fn-. 

; echo {ls | wc -l}
{%pipe {ls} 1 0 {wc -l}}
; echo $fn-%pipe
$&pipe

This lets users redefine large swaths of the shell's behavior. For example, the %write-history function is called by the shell to write a command to the shell history after reading it. To make the shell avoid writing duplicate commands to history, one can write: 

let (write = $fn-%write-history; last-cmd = ())
fn %write-history cmd {
  if {!~ $cmd $last-cmd} {
    $write $cmd
    last-cmd = $cmd
  }
}

We can go through this example line-by-line.

  1. let (write = $fn-%write-history; last-cmd = ())

    This creates a (lexical) binding of the current definition of %write-history to the variable write, and of ()—the empty list—to the variable last-cmd.

  2. fn %write-history cmd {

    This creates a new definition of %write-history. Thanks to the write variable bound with the let in the previous line, the old definition of %write-history is accessible within this function. This is a very common idiom in es, used for “spoofing” functions, or creating new definitions to suit preferences or create situational benefit.

    Note also that when the last-cmd variable was bound, the binding was created outside of this function. That means the binding, and therefore the definition of last-cmd, exists across function calls.

  3. if {!~ $cmd $last-cmd} {

    This compares $cmd against $last-cmd. If they differ, then...

  4. $write $cmd

    We call the old %write-history on the new $cmd.

  5. last-cmd = $cmd

    Then, we set last-cmd to $cmd. Because last-cmd persists across function calls, this effectively saves this command for future calls.

This bit of code implements what in Bash would be achieved using HISTCONTROL=ignoredups, and it's reasonable to note that the Bash version is quite a bit more concise than the es one. However, the es method has its own major benefits “at scale”, when considering shell features in aggregate.

Bash, and many other shells, add behaviors and customization via special variables and options, with their own special values and little languages, requiring a good deal of memorization of a large “menu” of tweaks and tricks to make the shell work as desired.

Es takes a different approach instead, exposing the core behaviors of the shell in a way that allows customization using the same scripting techniques that would, largely, be used for anything else in the shell. Once somebody is proficient in writing es scripts, they're proficient in customizing es.

The es approach also enables a fundamentally greater degree of flexibility. When Paul and Byron exposed the shell's interactive REPL as the function %interactive-loop, they also exposed the non-interactive REPL as %batch-loop, despite having reservations that actually modifying the non-interactive REPL would ever be a good idea. Doing so, however, makes it possible to write an alternate version of %batch-loop that parses an entire script before running any of its commands, which is a convenient way to “sanity-check” code and avoid running just half of a buggy script.

This degree of flexibility is remarkable: this kind of pre-parsing was never implemented or even considered by the authors of es, but it's just as possible as it is under other shells which officially advertise the behavior.

What's been happening with es?

Es was mostly developed over the course of 1992-1995. The bulk of development went through the release of version 0.84; 0.88 was released after the authors had taken a break, and then after that release both of them got too busy with life and jobs to continue work on the shell.

After that, maintainership passed through a couple hands, leading eventually to the current maintainer James Haggerty, but development was largely focused on keeping es functional over the decades as OSes, build systems, and code-hosting practices have evolved.

This left es as an incomplete experiment: Paul and Byron didn't have time to achieve a good amount of what they planned on, and even if they had, their near- to medium-term plans certainly didn't sum up to everything the shell could be made to do.

Recently, however, there has been more activity. A few of us have worked over the last year or two largely to shore up es' portability and reliability, including adding a test suite and supporting stricter compiler flags as well as more static analysis tools.

Es futures

So what's next for es? Well, there are a couple active projects I am pursuing.

Near-term, I would like to improve how es reads commands from its input. Es currently can be built with support for readline, but that support is somewhat limited—things like programmable tab completion simply can't be achieved. Some work in how memory is managed should enable things like programmable tab completion or even swapping out readline for other libraries entirely. Given there are multiple es forks featuring custom, hand-rolled line editing capabilities, making this easier to swap out seems ideal.

I would also like to add some form of job control to the shell. There is a long history of fighting job control in both es and rc, but I believe that it can be done in a way where, with a little more flexibility added to existing behaviors, users can handle process groups effectively and build a job-control system of their own.

Both of these projects are at least in part in the service of a somewhat larger goal, which is to grow the es community. Es, I think, has real design strengths which have appealed to people (like myself) even during periods when development on the shell was stalled. The several individual forks of es demonstrate this—but, in my opinion, those forks are also unfortunate, as they don't contribute to the es project as a whole.

Allowing people to interact with their shell in ways that are familiar to them (that is, job control and fancy programmable input), and doing so in ways that are consistent with or even extend the shell's existing design, serves to both make the shell more practically useful and demonstrate its design works.

Ideally though, I don't want to add too much to upstream es. The current feature set is pretty good. Whatever is added should function as a sort of meta feature, enabling not only some particular use but a whole new category of extensibility.

This is why, for example, I'm not just interested in adding programmable readline completion, but making it so that input to the shell is completely programmable. This extends the existing es tendency of making internal shell behaviors external, complementing %interactive-loop and %batch-loop, but also makes it more feasible to do things like call readline in other contexts, or write other line-editing libraries which can read input in other ways.

In addition to extensibility, I would like to try to follow through a bit on es' programmability. A couple examples here include:

Lastly, I would like to write more things to document aspects of es, making it easier to get a strong grasp of the shell without having to dive into the code itself or trawl the old mailing list just to have an idea of how certain things work or why they were implemented the way they were.

Some pages I ought to get around to writing include:

All in all, I'd like to have a good enough foundation for es, along with documentation and tooling support, that people can really get to hacking on it. Over the years, while the upstream shell has been quiet, multiple forks have spun up, proving that motivation to do something with es has never gone away.

And, to me, it makes a ton of sense why. At its core, es has a simple and powerful design which removes a huge amount of the friction of shell scripting, which is otherwise one of the most powerful ways to use a computer. Es' ethos of providing a few powerful and orthogonal language and runtime mechanisms makes it relatively easy to know top to bottom, and surprisingly easy to modify its internals. It is, genuinely, an extremely elegant piece of software that I am very glad to use every day.