This means one can conduct performance tuning and troubleshooting of the qmaster/scheduler of a production SGE cluster, without compiling a special version of the binaries, and without restarting daemons. Currently, Solaris 10, FreeBSD, NetBSD, and Mac OS X are the only OSes with DTrace support.
A wrapper shell script (monitor.sh) is also included to invoke the DTrace script from the command line with all the parameters needed:
% monitor.sh -help monitor.sh [options] options: -cell use $SGE_CELL other than "default" -interval use statistics interval other than "15sec" -spooling show qmaster spooling probes -requests show incoming qmaster request probes
Monitoring Grid Engine Masters with dtrace ------------------------------------------ Content ------- 1. Introduction 2. Master bottleneck analyis with dtrace 3. Copyright 1. Introduction --------------- Dtrace is a comprehensive framework for tracing dynamic events in Solaris 10. Please see under http://www.sun.com/bigadmin/content/dtrace/ for more detailed information about dtrace. 2. Master bottleneck analyis with dtrace ---------------------------------------- Understanding the bottlenecks of distributed systems is crucial for performance tuning. The script $SGE_ROOT/util/dtrace/monitor.sh allows a Grid Engine master be monitored, if Solaris 10 dtrace(1) can be used. Monitor.sh measures throughput-relevant data of your running Grid Engine master and compiles this data into few indices that are printed in a single-line view per interval with columns below. Spooling: #wrt Number of qmaster write operations via spool_write_object() and spool_delete_object(). Almost any significant write operation goes through this function both in bdb/classic spooling. wrt/ms Total time all threads spend in spool_write_object() in micro seconds. Message processing: #rep Number of reports qmaster processed through sge_c_report(). Most data sent by execd's to qmaster comes as such a report (job/load/config report). #gdi Number of GDI requests qmaster processed through do_gdi_request(). Almost anything sent from client commands arrives qmaster as a GDI request, but also execd's and scheduler use GDI requests. #ack Number of ACK messages qmaster processed through do_c_ack(). High numbers of ACK messages can be an indication of job signalling, but they are used also for other purposes. Scheduling: #dsp Number of calls to dispatch_jobs() in schedd. Each call to dispatch_jobs() can seen as a scheduling run. dsp/ms Total time scheduler spent in all calls to dispatch_jobs(). #sad Number of calls to select_assign_debit(). Each call to select_assign_debit() can be seen as a try of the scheduler to find an assignement or a reservation for a job. Qmaster/Schedd synchronization: #snd Number of event packages sent by qmaster to schedd. If that number goes down to zero over longer time there is something wrong and qmaster/schedd get out of sync. #rcv Number of event packages received by schedd from qmaster. If that number goes down to zero over longer time there is something wrong and qmaster/schedd get out of sync. Qmaster communication: #in++ Number of messages added into qmaster received messages buffer. #in-- Number of messages removed from qmaster received messages buffer. If more messages are added than removed during an interval, the total of messages not yet processed is about to grow. #out++ Number of messages added into qmaster send messages buffer. #out-- Number of messages removed from qmaster send messages buffer. If more messages are added than removed during an interval, the total of not yet messages not yet delivered is about to grow. Qmaster locks: #lck0/#ulck0 Number of calls to sge_lock()/sge_unlock() for qmasters "global" lock. This lock must always be obtained, when qmaster-internal lists (job list, queue list, etc.) are accessed. #lck1/#ulck1 Number of calls to sge_lock()/sge_unlock() for qmasters "master_config" lock. This lock is a secondary lock, but also plays it's role. note, currently the following options are supported: -interval