SGE_CKPT(5)

💖 SGE_CKPT 💖

okay bestie, so this is the Grid Engine User Commands manual page for sge_ckpt — aka the whole checkpointing mechanism and the checkpointing support that comes with it. it's SGE_CKPT(5), fifth section, and it's genuinely so important. sit down, hydrate, let's go 🫶

---

✨ NAME ✨

sge_ckpt — the Grid Engine checkpointing mechanism and checkpointing support 🎀

---

🌸 DESCRIPTION 🌸

so Grid Engine supports two levels of checkpointing, and you're gonna want to know both:

• the user level, and
• an operating system-provided transparent level.

User level checkpointing is when the applications do their own checkpointing — like, they write restart files at certain times or at certain algorithmic steps, and then they properly process those restart files when they get restarted. very self-sufficient of them, honestly 💅

Transparent checkpointing though? that has to be provided by the operating system, and it's usually integrated right into the operating system kernel. an example of a kernel-integrated checkpointing facility is the Hibernator package from Softway for SGI IRIX platforms. iconic behavior ngl 🧚

now, checkpointing jobs need to be identified to the Grid Engine system by using the -ckpt option of the qsub(1) command. the argument to this flag refers to a so-called checkpointing environment, which defines the attributes of the checkpointing method you're gonna use — see checkpoint(5) for the details, iykyk. checkpointing environments get set up by the qconf(1) options -ackpt, -dckpt, -mckpt and -sckpt. and the qsub(1) option -c can be used to overwrite the when attribute for the checkpointing environment you referenced. slay 🌈

---

🚨 checkpointing jobs hit DIFFERENT 🚨

so here's the thing, and it's lowkey the most important part: as opposed to the behavior for regular batch jobs, checkpointing jobs (see the -ckpt option to qsub(1)) are aborted under conditions where batch or interactive jobs are suspended or even just stay unaffected. 💀

these conditions are:

• Explicit suspension of the queue or job via qmod(1), done by the cluster administration or a queue owner — but only if the x occasion specifier (see qsub(1) -c and checkpoint(5)) was assigned to the job.
• A load average value exceeding the suspend threshold as configured for the corresponding queues (see queue_conf(5)).
• Shutdown of the Grid Engine execution daemon sge_execd(8) that's responsible for the checkpointing job.

after they're aborted, jobs will migrate to other hosts, and possibly to other cluster queues too — unless they were submitted to one specific queue by an explicit user request. and the migration of jobs leads to dynamic load balancing. ✨

Note: aborting checkpointed jobs will free all resources (memory, swap space) which the job occupies at that time. this is opposed to the situation for suspended regular jobs, which still use virtual memory and other consumable resources. so like, it's giving efficient but also it's giving "your job is gone bestie" 😭

---

🦋 RESTRICTIONS 🦋

okay this part is the reality check, and we don't gatekeep:

When a job migrates to another machine, at present no files are transferred automatically to that machine. which means all files used throughout the entire job — including restart files, executables, and scratch files — must be visible or transferred explicitly (e.g. at the beginning of the job script). no cap, plan for it.

---

There are also some practical limitations regarding use of disk space for transparently checkpointing jobs. checkpoints of a transparently checkpointed application are usually stored in a checkpoint file or directory by the operating system. that file or directory contains all the text, data, and stack space for the process, along with some additional control information. which means jobs that use a very large virtual address space will generate very large checkpoint files. 😭

and also — the workstations on which the jobs will actually execute may have little free disk space. so it's not always possible to transfer a transparent checkpointing job to a machine, even though that machine is idle. and since large virtual memory jobs must wait for a machine that is both idle and has a sufficient amount of free disk space, such jobs may suffer long turnaround times. the vibes are, unfortunately, waiting-room vibes 🙈

---

There is currently no mechanism for restarting jobs with the same resources they were granted originally. that might be important if they were submitted with a choice or range of resources and start running in a particular way with whatever they're given. so yeah, they get what they get 💅

---

Similarly, with heterogeneous execution hosts, jobs may need to restart on a host which supports a superset of the instruction set where the job originally ran — that is, if the checkpoint mechanism (e.g. BLCR or DMTCP) dumps an image of the running process. runtime libraries, in particular, may initialize themselves depending on details of the architecture they start up on — say, to use a specific type of vector unit. then they may fail if moved to an older host of similar architecture which lacks that feature, even if they were compiled for a common instruction set. it's giving "same family, different era" and it does NOT always end well 😭

---

👑 SEE ALSO 👑

• sge_intro(1)
• qconf(1)
• qmod(1)
• qsub(1)
• checkpoint(5)

---

💜 COPYRIGHT 💜

See sge_intro(1) for a full statement of rights and permissions.

---

and that's the whole page bestie, we ate and left absolutely no crumbs 🍓

this was SGE 8.1.3pre, dated 2012-09-18, SGE_CKPT(5) periodt 💅✨