Erlang slave nodes and ssh login shells

The Erlang runtime environment is more similar to an operating system than to a traditional language runtime library. An Erlang "node" is an Erlang instance started with the flag -name (or -sname, for "short names" if your network does not rely on DNS). For example:

$ erl -sname foo

will give you an interactive Erlang shell with a prompt like this:

Eshell V9.2.1  (abort with ^G)
(foo@rocka)1> 

and you can see from the prompt that this is running as a "node" with the node name foo@rocka. In practice, this means that networking is enabled, allowing Erlang processes on this node to communicate with processes on other nodes, either on the same host machine or on other machines, through the ordinary Erlang message passing mechanism.

Working with multiple nodes

If I open a separate console and start another node bar@rocka in the same way, I can then connect these two and start doing interesting multi-node stuff:

(foo@rocka)1> net_adm:ping(bar@rocka).
pong
(foo@rocka)2> nodes().                                         
[bar@rocka]

For instance, if I have implemented a module mysrv that runs a simple counter server, I can load the code onto the other node like this - even if the nodes do not share a file system - and start it on that node.

(foo@rocka)3> BeamFile = code:which(mysrv).
(foo@rocka)4> {ok, BeamBin} = file:read_file(BeamFile).
(foo@rocka)5> rpc:call(bar@rocka, code, load_binary, [mysrv, "", BeamBin]).
(foo@rocka)6> Pid = spawn(bar@rocka, mysrv, run, []).

The last command should print something like <8112.88.0> where the non-zero leftmost number in the Pid indicates that it runs on another node than the local one. We can try to request numbers from the server to check that it is running:

(foo@rocka)7> mysrv:next(Pid).
0
(foo@rocka)8> mysrv:next(Pid).
1
(foo@rocka)9> mysrv:next(Pid).
2

That's cool and all, but it requires that you first do the orchestration of starting the nodes on the various machines (and making sure they get restarted if need be). In some situations, it would be much nicer if one Erlang node could simply start another one, on demand. For example:

• Having a node on one machine coordinate a number of worker nodes on a cluster of machines
• Writing test suites for code that cooperates across two or more nodes
• Protecting the main node from failure when running less dependable native code (NIFs or C drivers, or a WX-based GUI)
• Not running certain operations in the same OS process context as the main Erlang node

And of course, you want to get notified if one of these nodes crashes, and you want them to disappear automatically if the main node dies.

Running slave nodes

Erlang has standard library support for starting and supervising such slave nodes, and it looks straightforward enough:

  slave:start(Hostname, Nodeame, ErlangArgs)

for example:

  slave:start("rocka", "baz", "-s io write hello")

For this to work, you need help from your operating system to actually get the new nodes running, and by default it is assumed that the command for doing this is rsh. (The original Unix rsh command is deprecated these days, but on many machines nowadays, the rsh name is simply a link to ssh.) Edit: the patch to switch to using ssh by default was merged upstream in February 2019 and should be included in the following major release, Erlang/OTP 22.

You can however pass a flag to Erlang to tell it to use any other command, like this:

$ erl -sname foo -rsh /usr/bin/ssh

Easy! But there is a snag: the way this is implemented, you cannot pass additional flags to the command, and the default behaviour for ssh is to create a non-interactive non-login shell, if a command is specified.

This means that your attempt to start a node on a remote machine will probably fail because the command erl could not be found in the path, for a non-login shell (at least if you have standard .profile/.bashrc files). And -rsh "/usr/bin/ssh -l" does not work either, because the given string is treated by Erlang as a single path (which is allowed to contain spaces).

You may also be mystified by why the following might work:

(foo@rocka)1> slave:start("rocka","baz", "").
{ok,baz@rocka}

while this might not:

(foo@rocka)1> slave:start("localhost","baz", "").
bash: erl: command not found

and the reason is that if the given hostname matches the hostname of the local node, the use of the rsh command is bypassed, and the new node is simply launched as erl ... <UserArgs> instead of "<Rsh>" <Hostname> erl ... <UserArgs> - and you probably had erl in your current path already, so it just worked. Also note that there is no way of specifying an absolute path to the erl command - it has to be in the path.

To get around this, you could set up a specific environment on all the remote machines, either by customizing their ssh configurations, or by creating custom user accounts for running the nodes. But if you want it to "just work", using a regular user account with no special modifications, how can you get around these limitations?

Starting the nodes in a login shell

The answer is to use a simple wrapper script like this one, for rearranging the arguments and passing them on to ssh along with the -l flag:

    #!/bin/bash
    host="$1"
    shift
    exec /usr/bin/ssh "$host" "/bin/bash -l -c '$@'"

Put the above in a file myrsh, make it executable with chmod a+x myrsh, and then start Erlang as follows:

$ erl -sname foo -rsh myrsh

That's all! Now it should work, both for "localhost" and for any remote machine that you can log in to (only assuming you have Erlang installed):

(foo@rocka)1> slave:start("localhost","baz", "").
{ok,baz@localhost}

You could augment the script as you please - for example, adding logging of the arguments being passed to the command, which was something I used quite a lot while debugging this stuff. The only tricky part of all of this was to figure out what was actually happening in the different cases, how the actual command line was constructed, whether there are any straightforward alternatives for getting ssh to behave like you want (not really), and whether your shell really should set up your user path only if it is a login shell (as far as I can tell, yes).

I hope this can be of help to anyone else who wants to try out the slave node feature, which understandably has been rather underused.