Job control and the extensible shell

if someone comes to me and Byron and says, ``i need these two extra primitives and this one extra line of conditionally compiled code in signal.c and i'll then be able to write full csh-style job control in es,'' we'd probably put it in, albeit with some serious grumbling. (i don't think it will happen, though. it just don't look that easy to me.)

— Paul Haahr, 21 Oct 1992 mailing list post

Not a lot of people understand job control very well, but a lot of people do seem to rely on it in one way or another. This has been an unfortunate combination for es: many, many requests for job control have been made, and a few brave souls have implemented versions of it in private forks over the years, but none of those versions have been upstreamed.

Part of the challenge is cultural. Es, due to its rc heritage, comes from a community which values minimalism and good design, and hates job control. After all, Plan 9 lacks job control; instead, as the argument goes, the windowing system provides the same sort of abilities in a better way.

Part of the challenge with job control and es is, however, technical; the design of es is sufficiently unique that job control can’t simply be bolted onto it without some original thought, and none of the attempts at job control thus far have done a good job of integrating it into the way that es does things. In particular, the following good design practices for es should be followed:

• New behavior should be added by generalizing existing features, rather than adding new ones
• Extensibility should happen by overloading hook functions, rather than by toggling pre-defined options and magic variables
• Features should be generally useful, including to those who don’t opt into job control
• Features should be orthogonal to each other: if job control requires several additions, each of those additions should be understandable and useful on its own

I believe I have a design for job control which achieves all of these reasonably well and deserves to be upstreamed into es.

Background: What is job control?

What is job control? What specific behaviors are necessary to say that job control is supported?

There are two answers to this question, in a sense. The term “job” won’t show up anywhere in the strace of a job-controlling shell. This is because job control is a feature, which predates the contemporary operating system mechanisms which implement it.

Job control was first introduced with csh way back in the late ’70s. The purpose of the feature is to allow users to control commands while they’re running. This includes, as advertised in the csh paper:

• Running a command in the background with &
• Stopping the current foreground command by typing the ^Z character
• Restarting a stopped command in the foreground with the fg command
• Restarting a stopped command in the background with the bg command

The csh paper also discusses how notifications are printed to the terminal when a command is stopped or exits, and how the shell maintains a table of numbered commands which can be used instead of processs IDs in certain commands. These aren’t strictly “ways users can control running commands”, but they are useful, so they might be considered optional to add in es.

Now, in csh, and traditionally in job controlling shells, a single command is a job, but in POSIX system calls, that term doesn’t exist. Instead, the term to use is a process group. In fact, “job control” can be more accurately described as “process group management”.

In a typical shell session, the user starts a terminal process, and the terminal then spawns a shell process. That might create a process tree like:

PID     COMMAND
1024    xterm
1025     \_ es

; grep '~' .esrc | wc -l

then while that command is running, the process tree would become

PID     COMMAND
1024    xterm
1025     \_ es
1213         \_ grep
1214         \_ wc

We can see that for this single command, the shell spawns two processes with different process IDs.

Now, suppose the user types control-C (^C) into the terminal while these are running. When this happens, it’s the terminal’s job to send a SIGINT to stop what’s running. But which of the three processes (es, grep, and/or wc) should the terminal send that signal to?

In the non-job-control world, the signal must go to all of them. It’s up to the shell to ignore or handle a SIGINT while it’s waiting for child processes to finish. Similar questions have the same answer: Which processes can read from the terminal? All of them—and if multiple try to read from the terminal at once, things can get messy.

For the sake of job control, two new OS-level concepts are introduced: the session and the process group. Every process is part of a process group, and every process group is part of a session; a session can have more than one process group, and a process group can have more than one process.

A session is the set of all the processes associated with a terminal, called the “controlling terminal”.

A process group is the set of processes which can be in the foreground of that terminal at any one time.

So for our example command in a job-controlling shell, this would look something like:

SID    PGID    PID    COMMAND
1024   1024   1024    xterm
1025   1025   1025     \_ es
1025   1213   1213         \_ grep
1025   1213   1214         \_ wc

The session with ID 1025 has the running xterm as its controlling terminal, and contains the shell and both processes in the command. The shell has its own process group, 1025, corresponding with its own PID; while the two processes from the command are running, they have their own process group ID. This process group 1213 is the foreground process group in the terminal; the shell will be in the background until the command finishes or stops.

The foreground process group contains the set of processes the terminal interacts with. In particular, signals generated due to characters like ^C are sent to all the processes in the foreground, and foreground processes can read from the terminal, while background processes that try to do so are sent a SIGTTIN signal which (by default) stops them.

Speaking of stopping: Job control requires the shell to be aware not only of processes which have exited or been killed by a signal, but also those which are stopped—that is, paused by a specific signal such as SIGTSTP, SIGTTIN, or SIGTTOU. These processes are not running but can be continued by sending them a SIGCONT signal. Both the fg and bg commands in job control shells use this signal.

These are, in general, the OS-level mechanisms which back job control. The GNU project have a very usefully detailed guide on everything required to implement it, which goes into basically every system call required (many of which es already performs in order to avoid cases where other processes misbehave and cause the terminal to exit!) So, how should es expose these behaviors to users?

Extensible job control

The central requirement of job control is the ability to spawn child processes in their own process group. It turns out that es already has a function which places processes in a new process group, called newpgrp. This function takes no arguments and places the current es process into its own process group. It was added to work around a bug in NeXTSTEP where terminals created every top-level shell in a single process group, presumably under the assumption that each shell would manage its own process group correctness.

I propose extending newpgrp so that, when given a command as its arguments, that command will be invoked such that every new process created in the process will be placed into a single, shared process group. In the following example, both cat and base64 would run in a child process group.

; newpgrp {grep '~' .esrc | wc -l}

A couple notes about this newpgrp:

• While every child process can run in a different process group from the shell itself, if the child processes don’t run at the same time, they cannot run in the same process group. In a command like newpgrp {cat file1; cat file2}, newpgrp would place each cat into a separate process group.
• In cases where newpgrp calls nest, the outer call “wins”, such that every concurrent child process within the context of any newpgrp call will be placed into a common process group. This is less intuitive than the alternative but works more naturally with how newpgrp is hooked into a job-control shell.
• The scope of a newpgrp call is dynamic; newpgrp my-func, where my-func creates some child processes, would place those processes into a process group.

Now we need a way to call newpgrp automatically in the right situations. Most of the ways es has to spawn a subprocess, like %pipe and %background, already have hook functions, but the simplest and most common way, a simple foreground binary invocation like ls, does not.

To fix this, I propose taking the existing %run function and making it a hook function used by the shell to invoke other binaries. This would add no new features to es, but like with newpgrp, it would take a function that is currently obscure and make it much more actively useful. To my mind, using %run as a hook is the Right Thing, with or without job control motivating it.

With these two changes, it would take something like the following to start spawning child processes in their own process groups.

let (r = $fn-%run)
fn %run {
	$fn-%make-job $r $*
}

let (p = $fn-%pipe)
fn %pipe {
	$fn-%make-job $p $*
}

let (b = $fn-%background)
fn %background {
	$fn-%make-job $b $*
}

In this snippet we use %make-job instead of calling newpgrp directly in order that process group creation can be toggled on and off as necessary. This is immediately useful for the other functions which create subprocesses, which we have to hook like

let (b = $fn-%backquote)
fn %backquote {
	local (fn-%make-job = ()) $b $*
}

let (r = $fn-%readfrom)
fn %readfrom f in cmd {
	$r $f {local (fn-%make-job = ()) $in} $cmd
}

let (w = $fn-%writeto)
fn %writeto f out cmd {
	$w $f {local (fn-%make-job = ()) $out} $cmd
}

Now that process groups can be created, we need mechanisms to refer to them and to manage them. Es already has a mechanism to refer to child processes, used by %background and consistent with conventions in the rest of the OS: the pid. The natural extension of this convention for process group IDs is to refer to the pgid as a negative number. This is used both within the POSIX specification for libc (see wait(3p)) and command-line utilities (see kill(1p)). Using negative numbers for process groups is the most promising way to avoid needing to wrap a number of commands such as kill in functions whose purpose is just to convert from some version of a “job name” to one or more pids.

In order to facilitate getting the pgid for a particular process group, I propose a new hook function called %reap. This function would be called by the shell’s internal wait() logic any time a subprocess has stopped or exited. The current behavior for %reap to imitate would be the internal printstatus() function, which prints a message to standard out when a process is killed by a signal; with job control, %reap can be extended to manage $apid in a more job-control-useful manner, and to print appropriate messages when processes are stopped.

fn %reap pid did status {
	if {~ $did stopped} {
		apid = $pid
	} {~ $did signaled && !~ $status (sigint sigpipe)} {
		echo >[1=2] $pid^: $status
	}
}

The parameters to %reap are: pid, representing the process ID (or process group ID) of the process which has exited; did, one of signaled, exited, or stopped (note the correspondence with the WIF.*() macros documented in wait(3p)); and status, which has either the exit status or signal of the process.

Broadly speaking, %run makes it possible to communicate “down” to the child process in new ways, and %reap makes it possible to communicate “up” to the parent shell. This is a little unintuitive, but it’s largely determined by the call site of the two functions: a %run call may be made in a subshell already created by %pipe, for example, where it can’t modify the parent shell’s state, but %reap is called from the parent shell before the information it exposes is needed for something like job control.

To be able to foreground, background, and continue process groups in es, I propose two new flags on the $&wait primitive. The first, -n, corresponds with the WNOHANG flag of the waitpid() function (see wait(3p)): if no process is ready to be reaped at the time wait is called, it will not hang and will instead return immediately. The second flag, -c, causes $&wait to foreground the process if necessary and send it a SIGCONT signal (it must be done in this order to avoid a race where the process gets itself stopped by reading from the terminal after the SIGCONT and before the foregrounding; this is why we build it into $&wait).

With these two flags, basic versions of fg and bg can be implemented as

fn-fg = $&wait -c
fn-bg = $&wait -nc

(real versions of these two functions should be written to default to using $apid if no pid is given.)

One last, small addition would be new syntax for the signals variable, which would behave like the /signame syntax which ignores a signal only in the shell process but leaves its handling default in child processes, but applies that logic to the shell’s process group instead. I don’t have a strong opinion about what symbol should be used, but +signame is the syntax I have used thus far. This would allow the shell, and processes started within the shell’s process group as with %backquote or %readfrom and %writeto, to be protected from job-control signals while other subprocesses are appropriately exposed to them.

So, these are all the components that I think are necessary to implement a job control shell. The last component of note is a spoof of %interactive-loop, in order to ignore the necessary signals, define %make-job, and place the terminal in its own process group:

let (loop = $fn-%interactive-loop)
fn %interactive-loop {
	signals = $signals +sigtstp +sigttin +sigttou
	local (
		fn-%make-job = newpgrp
		noexport = $noexport fn-%make-job
	) {
		newpgrp
		$loop $*
	}
}

and there you have it. Along with some invisible handling of the foreground terminal done automatically by the shell, this is everything required to manage process groups in es—the core feature set that job control needs.

Discussion

I believe that what is described here is a reasonable, roughly minimal set of features which make it possible to implement job control in es. It is largely backwards-compatible, it works primarily by extending existing mechanisms (newpgrp, %run, wait, $signals) in backward-compatible ways, and for people who don’t use job control, it ought to be occasionally useful and otherwise unobtrusive. A %run hook can be used to customize how binaries are invoked, for debugging purposes or to imitate something like what bash or ksh sets $_ to for child processes. wait -n creates the ability to check for process completion without blocking, and %reap should enable a “wait-any” mechanism to be written.

A side-effect of the %run change is that it removes shell-specific behavior (the fork/exec call pair) from es’ core evaluation logic. This helps generalize the shell; if primitives are made customizable, then it becomes (in theory) possible to swap out $&run with, for example, an alternative using posix_spawn. With some small adjustments to es’ evaluation logic (namely, using %whatis as a hook function), then the shell moves meaningfully closer to the “librarified” version that was once envisioned for it.

Of course, this isn’t the only way that job control or process group management could be implemented. There is a notable previous implementation by Olaf Rogalsky which exists in a branch in the es repository on Github. In my mind, that version is faulty for a couple of reasons. For one thing, it causes the shell to maintain a hard-coded table of jobs, necessitating translation functions for binaries such as kill and renice—something this version manages to sidestep by using -pgid identifiers instead. Admittedly, this may be a benefit of implementing this feature much later, after operating systems have all standardized onto POSIX. A more egregious fault of Olaf’s version is the presence of the $&procchange and $&procfree primitives, which modify (and remove) internally-tracked information that should be managed wholly by the shell runtime. I consider these two primitives similar to requiring a $&free primitive to free up memory.

Hypothetical setups have been dreamt up in the past where the relevant process-management primitives ($&fork, $&background, $&wait, $&pipe, $&readfrom, $&writeto, etc.) are decomposed into smaller units of work, such that the existing hook functions could be defined something like, for example:

fn %run bin argv {
	$&wait <={$&fork {$&exec $bin $argv}}
}

fn %background cmd {
	$&fork {
		$&newpgrp
		$cmd
	}
}

This is an interesting direction, and, if done correctly, could be a useful generalization, requiring fewer distinct primitives to perform the same (or more) actions.

However, achieving job control with these “smaller” primitives would still require changes like making %run a hook function, adding the ability to create process groups from multiple processes and wait for the whole group as a unit, and performing non-blocking waits and (correctly) foregrounding/continuing processes. Because of this, I believe that this kind of primitive-decomposition work is not an alternative to what is proposed here so much as it would be a future, additional area to investigate.

There are some limitations to what is proposed here; for example, there is no clear path to implement the notify/NOTIFY options in csh and zsh. Presumably this could be performed using a sigchld exception handler, but I expect some novel mechanism of cooperation with the shell’s input routines would be required.

An even larger limitation is the shell’s inability to properly control internal and compound commands. The promise of job control is that anything typed into the terminal should be controllable as a unit, but none of the following would be managed correctly with this proposed setup:

; while {true} {sleep 1}
; touch foo; sleep 100; rm foo
; x = `{date; sleep 100; date}

The ideal behavior for these commands is that job control works just the same for them as it would if they were placed in a child process. This isn’t possible with es as it is today; it would require some way to save the continuation of a command, and to restart an internal command in the background (which, to be most correct, would require threading such that a backgrounded command could change a variable’s value).

However, in addition to being difficult to implement, I believe this isn’t particularly necessary for practical use. As far as I am aware, no other shells handle these cases correctly:

Compound commands and command lists of the form “a ; b ; c” are not handled gracefully when combined with process suspension. When a process is stopped, the shell immediately executes the next command in the list or breaks out of any existing loops.

— Bash 5.3 man page BUGS section

Shell builtin functions are not stoppable/restartable. Command sequences of the form

a ; b ; c

are also not handled gracefully when stopping is attempted. If you suspend ‘b’, the shell will then immediately execute ‘c’.

— tcsh 6.24.16 man page BUGS section

Moreover, I believe that even if es were to gain the kind of continuation and threading mechanisms required to do this, they would still need to be integrated with the OS’ existing conventions for pausing and restarting work, which leads me to suspect it would not obviate this page’s proposed process group management mechanisms, but instead extend them.

Maybe someday I will be able to write a page about advancing the state of the art of job control with es, using mechanisms for delimited first-class continuations, threading, and smaller, simpler primitives to do it. Until then, I think adding these described mechanisms does a sufficient job giving users the utility they expect with minimal changes.