remote_startup(5)

📦 version 8.1.9, in debian bookworm, debian trixie, ubuntu jammy, ubuntu noble, ubuntu questing, ubuntu resolute, ubuntu stonking

✨ remote_startup(5) ✨

the Grid Engine remote startup mechanism — welcome to the page bestie, we're in remote_startup(5) from the Grid Engine User Commands era and it's a whole saga 🎀

💖 NAME 💖

remote_startup — the Grid Engine remote startup mechanism 🦋

🌸 DESCRIPTION 🌸

okay so here's the tea 🍓 Grid Engine ships several commands that let you do interactive stuff, or remote-start a tightly integrated parallel job. And each of those commands can be configured with qconf(1) (that's the -sconf option, bestie, don't forget it 💅) to use different daemons and commands to kick off the final session.

Which means: different startup methods can totally contain different daemons and commands, and they have zero relationship to the other startup methods. No cap. Although like… ngl, it's usually desirable to have the same communication method for all of them. Consistency is iconic ✨

Every single method needs its own separate instance of the communication daemon, started by sge_shepherd(8) for each job. Which means that daemon has to use a randomly-chosen port, and the client will try to connect to it. Is this extra? Yes. Is it necessary? Also yes — it's what makes tight integration work, so everything for a given job stays in the same process tree and Grid Engine can properly control and account for it. 👑

🧚 QLOGIN 🧚

An interactive qlogin session, invoked by qlogin(1), is gonna show up in qstat under the default name QLOGIN — unless you changed it with the -N name option. Two entries do the heavy lifting for establishing the communication to start a session like this:

  • qlogin_daemon
  • qlogin_command

the default vibe

The default is the value builtin, which triggers an SGE internal communication method:

qlogin_command      builtin
qlogin_daemon       builtin

if you want telnet instead (yes, telnet, hear me out)

In cases where you want a different communication method, you can also set it up for the formerly-used communication method based on telnet. And before you come for me — despite the fact that telnet is used, there's no need to have telnetd running all the time. SGE will start a unique one for each job, even when it's from the same user 😭 and telnet itself can stay disabled as a system service (whether under init or inetd). The only thing you need is for the file /etc/hosts.equiv to contain the name of the machines from where a qlogin(1) should be allowed. Usually that's the head node of a cluster, or particular submit machines. Using telnet the traffic is not encrypted — but lowkey that may be reasonable on a private subnet for the cluster, especially if SGE's CSP security isn't being used to secure the system generally. 🫶

To achieve this:

qlogin_command      /usr/bin/telnet
qlogin_daemon       /usr/sbin/in.telnetd

must be defined in sge_conf(5). The defined qlogin_command will then be called with two additional parameters: HOST, and PORT — in exactly that order 💅 — which refer to the machine the qlogin_command should address and the port to be used.

the SSH glow-up

You can also use this to set up secure communication using SSH (which can additionally provide X and credential forwarding, as well as compression — main character energy fr fr). In this case a small wrapper must be implemented, whose sole purpose is to swap the two given arguments and prepend -p to the port argument. A suitable one is installed as $SGE_ROOT/util/resources/wrappers/qlogin_wrapper:

qlogin_command      /opt/sge/util/resources/wrappers/qlogin_wrapper
qlogin_daemon       /usr/sbin/sshd -i

🎀 QRLOGIN 🎀

An interactive qrlogin session, invoked by qrsh(1) without a command, will show up in qstat having the default name QRLOGIN — unless changed by the -N name option. The two entries responsible for establishing the command to start such a session are:

  • rlogin_daemon
  • rlogin_command

the default vibe, again

The default is the value builtin, which will trigger an SGE internal communication method:

rlogin_command      builtin
rlogin_daemon       builtin

the rlogin route

In cases where you want a different communication method, it can follow the formerly-used communication method based on rlogin. And just like for telnet, rlogin can stay disabled as a system service (whether under init or inetd). The only thing needed is for /etc/hosts.equiv to contain the name of the machines from where a qrsh(1) should be allowed. Often that's the head node of a cluster, or particular submit machines. Using rlogin, the traffic is not encrypted — but again, that may be reasonable on a private subnet for the cluster, especially if SGE's CSP security isn't used to secure the system generally.

To achieve this:

rlogin_command      $SGE_ROOT/utilbin/$ARC/rlogin
rlogin_daemon       /usr/sbin/in.rlogind

must be defined in sge_conf(5). The value of $SGE_ROOT must be replaced by the root of the installation, and $ARC must be replaced by the particular platform architecture — because use of environment variables is not implemented for these entries, bestie, don't try it 🙈 When the cluster is homogeneous, it can be set to e.g. lx-amd64 or lx-x86. In a heterogeneous cluster, local configurations need to be defined, where preferably the minority of machines will get local configurations.

The defined rlogin_command will then be called with three additional parameters: -p, PORT, and HOST — in exactly that order — which refer to the machine the rlogin_command should address, and the port to be used.

This can also be used to set up a secure communication using SSH:

rlogin_command      /usr/bin/ssh
rlogin_daemon       /usr/sbin/sshd -i

🦋 QRSH 🦋

An interactive session for a remote command, invoked by qrsh(1) with a command, will show up in qstat by default with the name of the command issued — unless changed by the -N name option. The two entries responsible for establishing the communication to start such a session are:

  • rsh_daemon
  • rsh_command

oh and heads up: this startup method will also be used by the master task of a tightly integrated parallel job to start slave processes on other granted exechosts. so it's kind of a big deal 💅

the default vibe, the trilogy

The default is the value builtin, which will trigger an SGE internal communication method:

rsh_command         builtin
rsh_daemon          builtin

the rsh route

In cases where you want a different communication method, it can also be set up for the formerly-used communication method based on rsh. And as for telnet, rsh can stay disabled as a system service (whether under init or inetd). Only the file /etc/hosts.equiv needs to contain the name of the machines from where a qrsh(1) should be allowed. This is often the head node of a cluster, or particular submit machines. Using rsh, the traffic is not encrypted — but that may be reasonable on a private subnet for the cluster, especially if SGE's CSP security isn't used to secure the system generally.

To achieve this:

rsh_command         $SGE_ROOT/utilbin/$ARC/rsh
rsh_daemon          $SGE_ROOT/utilbin/$ARC/rshd -l

must be defined in sge_conf(5). The value of $SGE_ROOT must be replaced by the root of the installation, and $ARC must be replaced by the particular platform architecture, as use of environment variables is not implemented for these entries. When the cluster is homogeneous, it can be set to e.g. lx-amd64 or lx-x86. In a heterogeneous cluster local configurations need to be defined, where preferably the minority of machines will get local configurations.

The defined rsh_command will then be called with four additional parameters: -n, -p, PORT, and HOST — in exactly that order — which refer to the machine the rsh_command should address and the port to be used.

This can also be used to set up a secure communication using SSH:

rsh_command         /usr/bin/ssh
rsh_daemon          /usr/sbin/sshd -i

Again, this is independent of SSH as a system service, which can remain disabled. ✨

🌈 LOCAL CONFIGURATIONS OF EXECHOSTS 🌈

okay this part is important so pay attention bestie 👑

It is important to note that the communication method set up for one particular startup method must match at each end. You can achieve that either by using only a global configuration, or by carefully setting up local configurations for the exechosts involved. Whether or not local configurations exist (which you must take care of!) can be checked with qconf -sconfl.

As a general rule, for setting up a communication method between a machine A (where the command is issued) and a machine B (where the daemon is started), it must be guaranteed that the:

setup communication method for the command on machine A
(either global configuration from sge_conf(5) or local
configuration qconf -sconf A of machine A)

matches

setup communication method for the daemon for machine B
(either global configuration from sge_conf(5) or local
configuration qconf -sconf B of machine B)

This way it is also possible to use different communication methods, depending whether a connection from A to B is invoked, or from B to A. slay. 💅

💀 RESTRICTIONS 💀

For all three communication methods, a direct connection between the target and the source machine where the particular command was issued must exist. This can also be implemented using TCP/IP forwarding, but will usually fail if one machine is behind NAT, which will mangle the machines' addresses. 😭

The communication methods won't work with simple firewalling of the exec hosts, since the methods use a random port. It may be possible to set up application-specific firewalling, if necessary, or to wrap the methods and start an SSH tunnel on the port specified for each communication instance.

⚠️ also: the builtin method does not support forwarding of X graphics from the compute nodes, or GSSAPI tokens to them. If you need that for any of the remote methods, you will want to set up SSH communication instead.

🔥 SSH AUTHENTICATION 🔥

To allow the SSH setup explained above to work, the user must be authenticated without the use of a passphrase. Entering a passphrase would still work for interactive commands, but it will fail in the case of a tightly integrated parallel job, where the master process tries to start a slave process on another exechost. 😬

You can set up passphraseless SSH keys, although this is discouraged. A simpler and global working setup is to use host-based authentication at http://arc.liv.ac.uk/SGE/howto/hostbased-ssh.html for the machines inside the cluster. 🫶

✨ SSH TIGHT INTEGRATION ✨

Okay so bestie, this is the part where ssh and SGE become actual besties instead of just polite acquaintances. For a tight integration of SSH into SGE, the sshd that gets started needs an extra group ID attached to it. With that extra group ID, SGE can record the resource consumption and the computing time correctly, which is huge. And also — a qdel of such a job will actually be able to succeed. 💅

So there are two ways to get that tight SSH integration:

  • Use of PAM — literally the easiest route on supported platforms (at least GNU/Linux): there's a pam(7) module, pam_sge-qrsh-setup(8), made for use with the system ssh. It attaches that necessary additional group ID to the started process so you get the tight integration. Also go read the workshop paper at ⟨http://gridengine.org/assets/static/ws2007/K5SGE.pdf⟩. 🎀
  • Recompile Grid Engine with ./aimk -tight-ssh ... — the source of Grid Engine has all the necessary additions to compile a modified sshd, and that modified one will honor the additional group ID and attach it to the started process too. For this you gotta provide the source of OpenSSH in the directory 3rd_party inside $SGE_ROOT, and it has to have the plain name 'openssh'. Inside that directory, the file sshd.c needs to be patched:

in main():

init_rng();
#ifdef SGESSH_INTEGRATION
sgessh_readconfig();
#endif

in privsep_postauth():

/* Drop privileges */
#ifdef SGESSH_INTEGRATION
sgessh_do_setusercontext(authctxt->pw);
#else
do_setusercontext(authctxt->pw);
#endif

And see the original documentation at ⟨http://gridengine.org/assets/static/ws2007/SGE-openSSHTightIntegration.RonChen.pdf⟩. 📄

🔒 RESTRICTING ACCESS 🔒

So with the builtin method in use, there's no need to allow direct access for normal users to the compute nodes with ssh etc. But like… you might still want to let users get onto the nodes for debugging, and that's valid. If you don't want to over-subscribe the nodes — so that qrsh etc. can still be used for access — you can use PAM to restrict a user's access to only the nodes where they have a running job, which keeps the interference with everyone else to a minimum. ✨

There are two possible ways to do this:

🛡️ SECURITY 🛡️

See the notes above concerning security of the communication channel. That's it, that's the section. 😌

🌈 EXAMPLES 🌈

Using SSH with the PAM module, forcing tty allocation, and preventing the delegation of GSSAPI credentials to the compute nodes:

rsh_daemon     /opt/sge/util/rshdwrapper
rsh_command    ssh -tt -o GSSAPIDelegateCredentials=no
qlogin_daemon  /opt/sge/util/rshdwrapper
qlogin_command ssh -tt -o GSSAPIDelegateCredentials=no
rlogin_daemon  /opt/sge/util/rshdwrapper
rlogin_command ssh -tt -o GSSAPIDelegateCredentials=no

Old-style method, using telnet and rlogin (yes, we're going vintage, it's giving throwback 🦋):

qlogin_command /usr/bin/telnet
qlogin_daemon  /usr/sbin/in.telnetd
rlogin_command /opt/sge/utilbin/lx-x86/rlogin
rlogin_daemon  /usr/sbin/in.rlogind
rsh_command    /opt/sge/utilbin/lx-x86/rsh
rsh_daemon     /opt/sge/utilbin/lx-x86/rshd -l

📁 FILES 📁

  • $SGE_ROOT/util/resources/wrappers/qlogin_wrapper — SSH-based wrapper for qlogin (see above)

👀 SEE ALSO 👀

qconf(1), qlogin(1), qrsh(1), sge_conf(5), pam_sge_authorize(8), pam_sge-qrsh-setup(8), Grid Engine-specific remote programs ⟨http://arc.liv.ac.uk/repos/darcs/sge/source/3rdparty/remote/remote.html⟩.

✍️ AUTHOR ✍️

Man page written by Reuti, partly based on Sun material. Some additions by Dave Love. Absolute icons for documenting all this, no cap. 🫶

See sge_intro(1) for a full statement of rights and permissions.

And that's a wrap on remote_startup(5) from SGE 8.1.3pre, dated 2010/11/22 20:58:24. Thank you for coming to my TED talk bestie, now go configure your ssh and slay 💖✨🎀

📦 where this page comes from