NAME
     Sun Grid Engine Checkpointing - the Sun Grid  Engine  check-
     pointing mechanism and checkpointing support

DESCRIPTION
     Sun Grid Engine supports two levels  of  checkpointing:  the
     user  level  and  a  operating  system  provided transparent
     level. User  level  checkpointing  refers  to  applications,
     which do their own checkpointing by writing restart files at
     certain times or algorithmic steps and by properly  process-
     ing these restart files when restarted.

     Transparent checkpointing has to be provided by the  operat-
     ing system and is usually integrated in the operating system
     kernel. An example for  a  kernel  integrated  checkpointing
     facility is the Hibernator package from Softway for SGI IRIX
     platforms.

     Checkpointing jobs need to be identified  to  the  Sun  Grid
     Engine  system by using the -ckpt option of the qsub1() com-
     mand. The argument to this flag refers to a so called check-
     pointing  environment,  which  defines the attributes of the
     checkpointing method  to  be  used  (see  checkpoint5()  for
     details).   Checkpointing  environments  are  setup  by  the
     qconf1() options -ackpt,  -dckpt,  -mckpt  and  -sckpt.  The
     qsub1()  option  -c can be used to overwrite the when attri-
     bute for the referenced checkpointing environment.

     If a queue is of the type CHECKPOINTING, jobs need  to  have
     the checkpointing attribute flagged (see the -ckpt option to
     qsub1()) to be permitted to run in such a queue. As  opposed
     to  the  behavior for regular batch jobs, checkpointing jobs
     are aborted under conditions, for which batch or interactive
     jobs are suspended or even stay unaffected. These conditions
     are:

     o  Explicit suspension of the queue or job  via  qmod1()  by
        the  cluster  administration  or  a  queue owner if the x
        occasion specifier (see qsub1() -c and checkpoint5()) was
        assigned to the job.

     o  A load average value exceeding the suspend  threshold  as
        configured    for    the    corresponding   queues   (see
        queue_conf5().)

     o  Shutdown  of  the  Sun  Grid  Engine   execution   daemon
        sge_execd8() being responsible for the checkpointing job.

     After abortion, the jobs will migrate to other queues unless
     they  were  submitted  to  one specific queue by an explicit
     user request.  The migration of jobs leads to a dynamic load
     balancing.   Note:  The  abortion  of checkpointed jobs will
     free all resources (memory, swap space) which the job  occu-
     pies  at  that  time.  This  is opposed to the situation for
     suspended regular jobs, which still cover swap space.

RESTRICTIONS
     When a job migrates to a queue on another machine at present
     no files are transferred automatically to that machine. This
     means that all files which are used  throughout  the  entire
     job  including  restart files, executables and scratch files
     must be visible  or  transferred  explicitly  (e.g.  at  the
     beginning of the job script).

     There are also some practical limitations regarding  use  of
     disk space for transparently checkpointing jobs. Checkpoints
     of a  transparently  checkpointed  application  are  usually
     stored  in  a  checkpoint file or directory by the operating
     system. The file or directory contains all the  text,  data,
     and  stack space for the process, along with some additional
     control information. This means jobs which use a very  large
     virtual  address  space  will generate very large checkpoint
     files. Also the workstations on which the jobs will actually
     execute  may  have  little  free  disk space. Thus it is not
     always possible to transfer a transparent checkpointing  job
     to  a machine, even though that machine is idle. Since large
     virtual memory jobs must wait for a  machine  that  is  both
     idle,  and  has a sufficient amount of free disk space, such
     jobs may suffer long turnaround times.

SEE ALSO
     sge_intro1(,) qconf1(,) qmod1(,) qsub1(,) checkpoint5(,) Sun
     Grid  Engine Installation and Administration Sun Grid Engine
     User's Guide

COPYRIGHT
     See sge_intro1() for a full statement of rights and  permis-
     sions.