In the Sali lab¶
There is no need for most users to install the web framework. It is already installed for you by our sysadmins on the ‘modbase’ server machine.
Sysadmins: if you do need to make modifications to the framework itself, this can be done by cloning the git repository to a directory on modbase, making your changes, writing test cases to exercise those changes, running ‘scons test’ to make sure the changes didn’t break anything, ‘git commit’ and ‘git push’ to commit the changes, then running ‘sudo scons install’ to update the installed version of the framework.
Outside of the Sali lab¶
The framework is designed to work in the Sali lab environment, but can be used in other environments with some modification.
Python 3. The backend of the framework, which takes care of running jobs, is implemented in Python. (It does not work with Python 2.)
Flask. The frontend, which handles user submission of jobs and the display of results, is implemented in Python 3, and uses the Flask microframework.
SGE. The framework expects to be run on a machine that is a submit host to a Sun GridEngine compute cluster (or a compatible product, such as OGE or OGS). The framework talks to SGE via DRMAA, so you will also need the DRMAA Python bindings. The framework contains a class to talk to the SGE installation available in the Sali lab -
saliweb/python/saliweb/backend/__init__.py. To work in your environment, add a new
SGERunnersubclass to that file (see the implementation of
WyntonSGERunnerfor an example) setting the SGE_CELL and SGE_ROOT environment variables appropriately, setting DRMAA_LIBRARY_PATH to the location of your
libdrmaa.sofile, setting _runner_name to a unique value, and setting _qstat to the full path to your
MODELLER. The framework needs your academic MODELLER license key (in order for the
saliweb.frontend.check_modeller_key()function to work). Provide this with the scons modeller_key variable when running scons install.
Web server. The frontend consists of Python Flask scripts which need to be hosted by a web server. Apache with mod_wsgi is used in the Sali lab, but other web servers would probably work too (the only assumption made in the code is that files uploaded to the web server end up owned by the apache user, but this would be easy to change). The web server does not have to run on the same machine as the framework, but it does need access to the same filesystems (see below).
MySQL database. Both the frontend and backend need access to a MySQL database in which the jobs are stored. This requires the Python MySQLdb package. The framework itself needs no special access to the database, but each web service that uses the framework needs its own database. (The MySQL server can reside on a different machine to the framework if desired.)
Unix user. Each web service that uses the framework runs under its own user ID, and the jobs are run on the SGE cluster using the same ID. Thus, generally admin access to the cluster is required to add these users.
Filesystems. Each web service that uses the framework needs access to a local filesystem on the SGE submit host. This will be used by the web server to deposit newly-submitted jobs (the ‘incoming’ directory). The filesystem needs to support POSIX ACLs (this generally rules out NFS) since the directory will be owned by a Unix user unique to that web service, but will have a POSIX ACL applied to allow the Apache user to write files into it. Each web service will also need a directory on a shared volume, visible to the cluster nodes, where the files for jobs that run on the cluster are placed (the ‘running’ directory). (If space is limited on the network storage, web services can also be configured to move the files back to the cheaper local filesystem - the ‘completed’ directory - when the job is done.)