Thoughts for extensible es primitives

The extensible shell es can be thought of has having three layers, each with different degrees of malleability.

The top layer contains everything defined in the shell: variables and functions. Due to the nice qualities of es, this layer includes things like the definition of the interactive and non-interactive REPLs, control flow constructs like while, path-searching behavior, the cd builtin, and so on. This is the “extensible” part of the extensible shell.

The bottom layer is the core shell runtime implementing the GC, data structures, variables, closures, glomming, and so on. In general, nothing in this layer is modifiable without hacking on the runtime, since this is the foundation of the shell and needs to be kept coherent to support everything else. This layer also tends to work with data outside of the abstractions available to es data structures, making it difficult to express the desired behavior of these mechanisms using the shell language at all. There are a few hooks into specific behaviors at this layer, such as the %home hook function, but for the most part this layer is implicit or even invisible.

Betwee the core shell runtime and the user-definable functions and variables sits the middle layer, which is the topic of this page: it is the primitives layer. Primitives are the callable objects in es script which are implemented using C, and which, in some sense, make up the “standard library” of the shell. A large proportion of the shell’s behaviors, especially those that are relevant to its nature as a shell, are done by primitives such as $&pipe, $&fork, $&read, and $&echo. While the internals of a primitive are opaque, a well-designed and well-documented primitive has behavior that is understandable, predictable, and orthogonal to other primitives.

As a broad pattern, improving the extensibility of es is a process of “raising” pieces of the shell from lower layers to higher ones: exposing built-in behaviors of the shell as primitives which are connected via functions, reducing the scope of those primitives, or even replacing them entirely with pure-es functions.

However, there is an entire angle of extensibility which could also be in es but isn’t, and that is in extending the set of primitives present in the shell. Today, es has a fairly fixed, rigid set of primitives; for a few primitives, it is possible to tweak whether they are included in the shell using build flags, but that control is limited to just those special primitives, and its implementation is messy and ad-hoc in the same way that any other preprocessor-based code inclusion based on #ifdef is.

This page is an exploration of what it might look like to make es primitives more properly extensible, and how far that idea could be taken. As it turns out, there seems to be a path forward all the way to a Tcl-style “librarified” es, which can be used as a command language in ways unrelated to its current role as a Unix shell, which has long been a goal of the original authors. And, while this distant goal is enticing, there is also utility in more modest changes required to get there.

Motivation

What follows are a few general categories of potential use cases for extensible primitives. These are in order roughly from the least ambitious to most ambitious.

Extended behaviors

There are a number of behaviors which es could have and doesn’t, out of a desire to keep the shell small. As a design target, es by default only exposes a set of features comparable to that of rc, which could be considered the optimal minimal set. However, many people expect more features, and, excitingly, different people expect different combinations of features. While the extensibility of es’ scripting layer can cover much of this diversity, some of it can only be implemented with support from the C runtime. This is where extensible primitives can come in.

Some particular ideas for this include small additions such as $&getpid or $&getcwd (the absence of the latter in es today seems to be a direct cause of an additional 20-50 lines of every single es user’s .esrc). Similarly, marginal features such as arithmetic or regular expressions could be added to the shell relatively cheaply as primitives (although, in my opinion, arithmetic would ideally be performed as part of glomming, rather than as a set of commands). Shell input libraries are another likely candidate—es’ input logic has recently been refactored so that readline is exposed entirely as a set of primitives, and other input libraries could be added to the shell in a similar way.

Extensible primitives could also be used to do things not traditionally performed by shells. Both bash and zsh have some facilities for networking, for example. Designs for coprocesses, too, could be developed. More radically, many recent shells have focused on working with structured data in certain constructs where traditional shells would work on simple strings, and es could do the same, with the right primitive support.

Experimental contributions

As I see it, es developers today face a dilemma. Either they keep their contributions in a personal fork, limiting its reach and making it harder for good ideas to compound, or they attempt to upstream their changes into a very conservative codebase, probably resulting in a response of “thanks for the attempt, but no thanks.”

I believe this has a chilling effect on contribution, and, given es doesn’t exactly suffer from an overabundance of contributors, I believe this has a chilling effect on the shell as a whole.

A mechanism for extensible primitives, by providing es a way to more flexibly include code, could offer a way to resolve this dilemma. Experimental changes could be added to the upstream es repository as a set of extensible primitives, and only be included in the shell on an opt-in basis. This would improve the visibility of these changes compared to stuffing them in a personal repo, without immediately needing to be fully, backwards-compatibly “production grade”.

This is, of course, closely related to the extended-behaviors use case, as many extended behaviors would likely be (or at least start as) experimental changes. However, it is also likely that people may want to create alternate implementations of existing primitives. This suggests that es’ extensible-primitives mechanism should provide a facility for this, and potentially a facility to express that a primitive expression like $&parse should refer to a particular version of a parse primitive.

OS specialization

Es is highly portable, and that’s a very good thing. However, erring towards portability is often in practice merely implementing the lowest common denominator of behavior, as the shell is limited to the feature set of the POSIX.1-2001 specification.

Extensible primitives could provide a structured way to keep es portable in its core, but still make use of non-standard and OS-specific APIs as desired.

For example: With the right OS support, such as jails or capsicum or seccomp, could a new sort of restricted shell be implemented that isn’t compromised in the same ways as traditional restricted shells? What kind of Haiku GUI scripting could be made possible, or at least more ergonomic, with built-in shell support? Could one of Linux’s many fancy process-handling mechanisms (clone() with all its many flags, pidfds, etc.) be exploited to good advantage?

This could take the form of additional primitives for additional behavior, or alternatives to existing primitives for specialization. A related point here is that this may involve primitives implemented across languages. Fortunately, C is already the lingua franca of FFI, but attempting to work in other languages has implications for some of the more pathological aspects of es’ runtime, like its exception mechanism.

Versioned primitives

If designed properly, a system for extensible primitives could be used to support a better, more flexible backwards compatibility story, by allowing newer versions of primitives to be made available without (immediately) removing the older behavior on which users depend.

The descriptions of other use-cases have already mentioned supporting multiple versions of a given primitive, and this case would follow through on that idea. It would require most of the shell’s built-in primitives to be moved into an extensible-primitive library, but assuming that is done, then already-mentioned methods to specify “$&parse refers to this person’s implementation of $&parse” would work just as well to specify, say, “the es-0.10.0 version, rather than the es-0.9.3 version”.

This could allow the future to become just another opt-in feature. Scripts (and users) which have no need of the future can continue to use older designs for primitives without trouble.

A specific example here would be the rewrite of $&time that was recently done. While the new version is a strict improvement on the previous one (being capable of everything the old primitive could do and more), it is a backwards-incompatible update, because it changes the parameters of the $&time primitive. This could be side-stepped with a mechanism that lets users select the newer or older $&time specifically.

Build-time functionality and alternative shell binaries

Extensible primitives could become part of the es build process, and even be used to build alternate es-based binaries.

The way es is currently built involves:

  1. building the esdump binary, containing the core es runtime as well as dump.c
  2. running esdump so that it runs initial.es and then dumps the shell’s memory state to initial.c
  3. building the real es binary with the generated initial.c, which starts the real shell

This is all specialized, hard-coded behavior, which could be generalized by refactoring the build (and startup) process to make use of extensible primitives. Instead of a distinct esdump binary, dumping behavior could be performed by an es binary/script which loads dumping-specific primitives, reads initial.es, and then calls $&dump. The $&dump primitive could produce a C file which itself defines a $&loadinitial primitive, and then the final es binary could run that $&loadinitial on startup.

If properly designed, this process can be generalized to produce alternative es-based programs, which don’t necessarily even act as shells. This could even be a method to produce something like a librarified es, though the binary’s main() would be in the es code. This would, I think, represent the full development of the concept of extensible primitives.

Prior art

zsh

Zsh is probably the shell most famous for its linkable modules, which give it even more features on top of its already prodigious feature set. These largely fall into the “extended behaviors” case. Es and zsh are not very similar in their design goals, but the amount of time and attention paid to zsh means that it is worth looking at for inspiration as far as uses of extensible primitives, if not their design or implementation.

Modules documented in zshmodules(1) include:

Zsh also seems to have namespacing that it uses for its module system.

Inferno sh

The shell from the Inferno OS, simply called sh, has a concept of modules deeply integrated into its functionality. This shell, which incidentally directly names es as an inspiration, moves most of the typical behaviors of a shell out of its core runtime, leaving (as documented in its paper) only builtin, exit, load, loaded, run, unload, whatis, and a couple quoting utilities.

This is just enough capability to load other modules. The behaviors to actually implement a shell are then placed in the std module, which defines functions like fn; !, and, and or; if, for, and while; whatis; and raise and rescue, its versions of throw and catch. Whereas zsh modules are oriented toward adding features on top of an already capable shell, sh’s modules are used to build a shell from a minimal language runtime. (Some behaviors which are functions in sh, like fn and for, cannot be functions in es due to their interaction with lexical scope; if they were not in the core es, these would need to be implemented using an alternative parser or some kind of extensible syntax instead.)

Unfortunately, sh and the inferno OS on which it depends have become museum-piece software over the years, so the design hasn’t been exercised especially intensely. However, it serves as probably the most relevant prior art for a potential concept of es which is built on top of extensible primitives.

“Real” programming languages

It is possible to extend the Python interpreter with modules, so let’s look at how that works, and, potentially, how it is used. This includes the ability to create opaque, module-specific objects in Python; es has no facility for this at the moment, but it is worth considering a mechanism for opaque handles, as some built-in behavior (file handling) would benefit from the behaior, and it is likely to be useful in es “modules” as well.

Python also explicitly enables the ability to import the __future__ using future statements. Lessons from future statements and how they help with backwards compatibility, as well as the general difficulties of Python 2-to-3, should be informative for the design of extensible primitives as a change-management feature.

Limitations

For this kind of primitive extensibility to be as effective as possible—particularly with longer-term goals such as alternative (non-shell) es binaries, a number of behaviors would need to be “lifted” out of the core shell runtime and into primitives and scripted functions.

Running external binaries, and how that would be done, is one such case. See my %whatis- and %run-changing PR, also useful for my job control design. This change is sufficient to remove the special ability to run external binaries from the core shell runtime, instead making it a possible shell-related extension.

Interaction with the environment is a big one. This would look like some way to explicitly go about importing variables from the environment on the one hand (with some degree of control over which things are imported, for the sake of the -p flag), and a way to explicitly manage the exporting of variables to the environment. The noexport variable already exists, but is almost certainly insufficient for the job; as a starting point for this extensibility, it should be possible for people to forsake the rc-inspired method of interacting with the environment in favor of a more bash- and other POSIX-shell-based method. I could see this being covered by:

Parsing is another big one—plenty of people might not want a non-shell command language with redirection or pipes or I/O substitution. Extensibility in the parser was a goal of the original authors, but didn’t go anywhere. This could still potentially be done, but a simpler story would be to use extensible primitives to “swap out” which parser is used in different settings. It is already the case that environment-parsing is almost entirely performed on desugared es commands, so a parser that only understands the desugared language should be highly effective in that case. Automatic conversion from strings more frequently needs to parse “sugary” syntax, but a hook could be exposed to allow the right parser to be selected. All of this would be improved by a more-orthogonal parser which does not perform any implicit redirection (see my discussion of recent input changes for details).

Globbing is yet another case: outside of a shell, a command like echo * may not be most useful as a way to list files in the current working directory, but instead any other set of currently-relevant objects (an example given long ago was program symbols in an es-powered debugger.) Unfortunately, despite the fact that exposing the shell in a meaningful way has been a desire for almost as long as the shell has existed, there has hardly even been a design proposed to actually do so. In large part, this is because it’s difficult to expose to users the difference between * and '*': glomming, and the subset that is globbing, directly works with data structures that are hard to work with in es script.