Functional Specification: Job Submission Verifier
=================================================
Version Comment Date Author
------- ------------------------------------- -------- -------------
0.1 Initial version ? Andreas Haas
0.5 Describe changes so that enhancement 17-09-08 Ernst Bablick
can be implemented for Urubu with
less performance loss
0.6 added missing parts according to 22-09-08 Ernst Bablick
discussion with RD and AS
0.7 Changes according to discussion on 23-09-08 Ernst Bablick
users mailing list
1 INTRODUCTION
============
In the past some of our users expressed their need for some kind of
presubmission procedure which is executed whenever a job enters the GE
system. (see also issue #2621). Here are some examples what should be
done in such a procedure:
- Check accounting DB to make sure the user has enough wall clock
hours in their account to run the requested job on the requested
slots for the requested time.
- Guarantee that the number of slots requested is a multiple of 16 for
parallel jobs.
- Verify that the user can write to various shared filesystems.
- Make sure that the user does not request certain -l resources that
might not behave the way the user expects them to (h_vmem, h_data,
etc).
- Add required resource requests that users don't now are mandatory.
- Add a project request of the form -P queue_name where queue_name is
the queue used with the -q option.
- Make sure that the user hasn't messed up their ssh keys so badly
that they cannot ssh into compute nodes w/o a passphrase.
- Print out status messages and errors about the above as well as
printing out the queue, allocation account name, PE,
total number of tasks requested, and number of tasks per node
requested.
- Print out an motd-like message at the top of qsub output
> qsub job.sge
Welcome
-------
Please note that we strongly advise using the mvapich-devel MPI
stack for running jobs with more than 2048 MPI tasks.
---------------------------------------------------------------
--> Submitting 16 tasks...
--> Submitting 16 tasks/host...
--> Submitting exclusive job to 1 hosts...
...
2 PROJECT OVERVIEW
================
2.1 Project Aim
Aim of the project is it to provide a interface enhancement for GE that
allows it to define job verification/modification routines which will
either be executed on client side or within qmaster process when a
job enters the system or both.
2.2 Project Benefit
The administrator of a GE cluster can define additional policies needed.
The GE cluster will not be loaded with jobs which would break a defined
policy if a job verification/modification routine is defined.
2.3 Project Duration
2.4 Project Dependencies
There are no known dependencies with other projects
3 SYSTEM ARCHITECTURE
===================
3.1 Enhancement Functions
Here is the summary of the customer needs:
(N1) The administrator gets the possibility to define job verification
procedure which will be executed in qsub, qrsh, qsh, qlogin, qmon
and applications using DRMAA, to evaluate a job before it is send
to qmaster
(N2) The administrator gets the possibility to define a job verification
procedure which will be executed on qmaster side before a job
is finally added to the qmaster data store or before the
modification of a job is finally accepted.
(N3) It will be possible to define under which user account the
verification procedure within the master is executed. By default
the script is executed as sgeadmin user.
(N4) Data defining the job will be provided to the verification
procedure.
(N5) After evaluating a job the verification result might either be:
* accept job
* correct parameters part of the job specification
* reject job
* temporarily reject job (it might be accepted later)
(N6) Nearly all parameters which define a job can be changed by the
verification procedure but there are some exceptions. Following
things are only available as read only parameter:
* type (qsub job => qlogin ...)
* script file to be executed
* arguments passed to the job
* user who submitted the job
The job script contend itself is not available in the job
submission verification script.
(N7) As a minimum requirement at least following parameters have to be
changeable by the job verification procedure in a first
implementation
* pe request
* resource requests (hard and soft)
* queue and host requests
* project request
Implementation notes and necessary steps:
(I1) (N1) and (N2) will be realized as script. The script language can
be chosen by the administrator.
(I2) The script has to be written in a way so that it can be executed
like a loadsensor script. It has to accept commands and
parameters from stdin and return results via stdout.
It should not terminate until it gets a corresponding command.
(I3) qsub, qrsh, qsh, qlogin and qmon and the DRMAA library (N1)
will start a presubmission script which can be configured by using
"-jsv <jsi_url>" in the cluster wide sge_request file. The script
specified with <jsi_url> will be started under the user account
of the user which tries to submit a job.
(I4) The script to be evaluated in qmaster (N2) has to be configured
in the cluster configuration. The parameter will be named
"server_jsv". The value is also a <jsi_url>
(I4.1) <jsi_url> will have following format
<jsi_url> := [ <type> ":" ] [ <user> "@" ] <path>
<user> is the username under which the JSV code will be executed.
<type> might be the string "script". (In future we might support
shared libraries or master plugins which might be written in java.)
(I5) One instance of server_jsv will be started during startup of
qmaster for each worker thread or whenever the cluster
configuration parameter changes or whenever the timestamp of the
script file changes.
(I6) The server side instances of the verification scripts are connected
to the worker threads via pipes. Parameters and commands will
be send to the scripts and the response is read from the script
output.
(I7) After the script has been started it has to be responsive to
execute following commands. Please note that each command
might print ERROR=<message> to stdout to indicate an error.
command action
------- ---------------------------------------------------------
START Trashes cached data and starts a verification for a
new job.
Prints "STARTED" to stdout
After that the script accepts only a "BEGIN" or one or
multiple "PARAM <name> <value>" commands
BEGIN This command triggers the verification of provided
parameters set by "PARAM <name> <value>"
Prints "RESULT STATE <result>" as last line in the outout
and optionally "RESULT MSG <message>" and/or
"RESULT LOGMSG <message>" before that.
<result> might be:
ACCEPT
job is accepted without changes
CORRECT
job is accepted but all PARAM... which have
been sent between the initial BEGIN and the final
RESULT have to be evaluated and applied to the job
before it is accepted.
REJECT
job is rejected
REJECT_WAIT
job is rejected but might be accepted later
<message> is a user readable message
which will be sent to the client to be printed as
GDI answer (RESULT MSG) or it will be printed to
stdout of the client command (RESULT LOGMSG on
client side) or it will be printed to the master
messages file (RESULT LOGMSG on master side)
On client side the RESULT MSG and RESULT LOGMSG
messages will be ignored if the -terse option is used
with qsub.
PARAM <name> <value>
<name> and <value> are parameter names
and corresponding values as documented in submit(1) e.g.
<name> <value>
----------- ---------------------
a <date_time> in the format CCYYMMDDhhmmSS
ac <variable>[=<value>],...
ar <ar_id>
A <account_string>
b "y" | "n"
...
switches without additional arguments like -cwd or -notify
will be handled like arguments with yes/no argument
cwd "y" | "n"
notify "y" | "n"
...
Client parameter which were not specified by the
submitter of a job and which have a default value
during the submission won't be passed to the JSV script.
Additionally to the client switches following names
are supported
VERSION <major>.<minor>
e.g "1.0"
CLIENT "qsub" | "qsh" | "qlogin" | "qmon" | "qalter"
CONTEXT "client" | "server"
explains if the script is executed in a
client (N1) or in the master (N2)
JOB_ID <job_id> (only within master available)
SCRIPT <path_of_job_script>
SCRIPT_ARGS <arguments_for_job_script>
USER <submit_user_name>
QUIT Terminates the job submission verification script
Exampe: Find below the data which is sent to the job submission
verification script, when following job is submitted:
> qsub -pe p 3 -hard -l a=1,b=5 -soft -l q=all.q troete.sh
Please note that parameters that are not explicitely
requested by the submitter of a job are not passed
to the script. This means that e.g "-b n" of qsub won't be
passed to the script because this is the default
when nothing else is specified.
Input Output
01) "START"
02) "STARTED"
03) "PARAM CLIENT qsub"
04) "PARAM USER ernst"
05) "PARAM pe p 3"
06) "PARAM hard"
07) "PARAM l a=1,b=5"
08) "PARAM soft"
09) "PARAM l q=all.q"
10) "PARAM SCRIPT troete.sh"
11) "BEGIN"
12) "PARAM pe p 4"
13) "RESULT MSG no multiple of 4"
14) "RESULT STATE CORRECT"
15) "START"
16) "STARTED"
17) ...
99) "QUIT"
(I8) A bourne shell jsv script will be part of Urubu which can be used
as template for a GE administrator.
3.2 Overall Block Diagram
4 FUNCTIONAL DEFINITION
=====================
4.1 Performance
4.2 Reliability, Availability, Serviceability
4.3 Diagnostics
4.4 User Experience
4.5 Manufacturing
4.6 Quality Assurance
4.7 Security & Privacy
4.8 Mitigation Path
4.9 Documentation
4.10 Installation
4.11 Packing
4.12 Issues/Risks and Purposed Mitigation
5 COMPONENT DESCRIPTION
=====================
5.1 Component: Commandline
5.1.1 Overview
5.1.2 Functionality
5.1.3 Interfaces
5.1.4 Other Requirements