FCI - Folding@Home Client Info
Home
Documentation
Install
Upgrade
Downloads
Subversion Repository
Project Management
Links
About

Get Firefox!
Documentation
General
=======

Folding@Home Client Info, FCI in short, is a set of Perl scripts to
monitor Folding@Home clients over a network.

FCI consists of 2 parts, a client and a server. 
The clients send their information to the server where this information is
processed into XML files and displayed on a web page.
The server runs on Linux, FreeBSD & OpenBSD, this because of the applications
which run on the serverside, and the client runs on Linux, FreeBSD, OpenBSD,
Mac OS X and Windows.

Below is a semi-graphic representation of the servers and clients involved,
and describes the traffic between them. 

       +-------------------------+         +-------------------------+
       | Stanford Website        |         | qd Website              |
       |                         |         |                         |
       | Hosts:                  |         | Hosts:                  |
       |  daily_team_summary.txt |         |  qdinfo.dat             |
       |  daily_user_summary.txt |         |  qd                     |
       |  psummary.html          |         |  qd.exe                 |
       |  team0.txt              |         |  qd.openbsd             |
       |                         |         |  qd.freebsd             |
       |                         |         |  qd.darwin              |
       +-------------------------+         +-------------------------+
                     HTTP GET ||    HTTP GET || 
                              ||             || 
 The server downloads         ||             || The server downloads 
 the files hosted on          ||             || the latest qdinfo.dat
 the Stanford website,        ||             || and qd binaries from the
 either via the script        ||             || website of the qd author,
 fci-update-stanford-files.pl ||             || either via the script
 or via the web interface     ||             || fci-update-qd-files.pl 
                              ||             || or via the web interface
                             \||/           \||/
                              \/             \/
                      +--------------------------------+
                      | fci-server                     | STEP 3:
                      |                                | 
		      | Runs:                          | The server runs 
                      |  index.pl (Apache mod_perl)    | fci-update-stanford-files.pl,
                      |  fci-update-stanford-files.pl  | fci-update-qd-files.pl,
                      |  fci-update-qd-files.pl        | fci-update-xml-files.pl,
		      |  fci-update-project-images.pl  | fci-update-project-images.pl,
                      |  fci-update-xml-files.pl       | and optionally
		      |  fci-jmol-missing-projects.pl  | fci-jmol-missing-projects.pl
                      +--------------------------------+
                     HTTP GET ||   HTTP POST /\ 
                              ||            /||\
     STEP 1:                  ||             || STEP 2:
                              ||             || 
     The client downloads     ||             || The client uploads the
     the latest qdinfo.dat    ||             || information it gathered
     and optionally the qd    ||             || to the server
     binary from the server   ||             ||
     if newer versions are    ||             || 
     available before it      ||             || 
     submits its information  ||             || 
     to the server            ||             || 
                             \||/            || 
                              \/             || 
  		      +--------------------------------+
                      | fci-client                     |
                      |                                |
                      | Runs:                          |
                      |  fci-client.pl (which uses qd) |
                      |  Folding@Home client           |
                      +--------------------------------+


Client
======

The client is a Perl script, which invokes the qd binary to parse the queue.dat
file and to send its output and some other files to the server over HTTP.

Besides some files of the Folding@Home client it also tries to find the CPU 
speed and amount of installed RAM. Depending on which Operating System FCI is
being run on, it determines this via different methods:

on Linux it uses the files /proc/cpuinfo & /proc/meminfo (if readable)
on Windows it uses the commandline tools systeminfo
on FreeBSD and OpenBSD variants it will try to get the information from the
file /var/run/dmesg.boot (if readable).
on MacOS X v10.2 - v10.x it uses /usr/sbin/system_profiler, 
and on MacOS X v10.0 - v10.1 it uses AppleSystemProfiler

This is not a fullproof method, therefor it is not mandated by the server to
supply this information.

In previous versions of FCI, the unitinfo.txt file was used as the main source 
of information about the current Work Unit of the Folding@Home-client, but it
turned out not to be a reliable source, some cores tend to generate incomplete
unitinfo.txt files or because of the maximum length of the project name only
the first part of the project name was shown.
Since v1.0 qd is used to parse the queue.dat file which is a far more accurate
source of information, but as with the project summary on the server, it needs
to be frequently updated to be able to recognise new projects.

When the client is started, it first checks the ~/.fci/ directory and its 
content. ~/.fci/ is not used for the qd files on Windows, there the same 
directory which contains the fci-client.pl script is used for the qd files.

If ~/.fci/ does not exists, it is created and the qd binary and qdinfo.dat
files are copied from the paths where they are found. 
By default /usr/local/bin/ on Linux, FreeBSD and OpenBSD, but it will also try
to find the qd files in the same directory as the fci-client.pl script. This to 
make sure upgrades from FCI version prior to v1.0 will still work.
If it does exists, the required files are checked and the permissions of the 
residing files are verified. 
After the ~/.fci/ directory has been verified, the clients checks for newer 
versions of qdinfo.dat. If the qdinfo.dat file is newer it is downloaded and 
saved in the ~/.fci/ directory.
For security reasons fci-client.pl v1.0 does not automatically download the qd 
binary anymore. If a more recent functional revision is available, it will only
be downloaded if the --update-qd paramenter is given.
By automatically downloading the qdinfo.dat file, the most recent project 
information is available to qd when it parses the Folding@Home-client data.

After the qd files are updated, the client gathers the files to submit to the 
server and tries to discover the CPU speed and installed RAM. All this 
information is than submitted to the server via an HTTP POST where it will be
processed further.


Uploaded data
-------------
The client can upload serveral files and other information to the server. 
By default it only requires to upload:
- a client name
- the output of the qd command
- the fci-client version 

The client will upload additional data if present on the system. 
The additional data included by default: 
- the unitinfo.txt file
- the current.xyz file
- the client.cfg file
- the FAHlog.txt file
- the FAHlog-Prev.txt file
- the MyFolding.html file

You can skip the uploading of these additional files with the respective 
--skip-* commandline parameter. Use e.g. --skip-client-cfg to skip the upload
of the client.cfg file. Run fci-client.pl --help to see all available 
commandline parameters.


Data sources
------------
The client uses several files and applications to gather information of the 
Folding@Home-client, which it will submit to the server. On the server this 
information will be processed and presented on a web page. Below is a list of
data sources which the client uses and a description of what information it 
provides.
The directory in which the Folding@Home-client runs is: $folding_dir

- fci-client.conf		(optional)
  fci-client.pl configuration file
  provides configuration for the client,
  The main hash with configuration parameters in fci-client.pl is filled with
  hardcoded defaults first, those are overridden by parameters from 
  fci-client.conf, which ultimately can be overridden by commandline 
  parameters.

- qd
  Application to parse the queue.dat file
  main source of information, 
  provides information of the work units in the queue and 
  Folding@Home-client configuration
  
- $folding_dir/client.cfg       (optional)
  Folding@Home client configuration
  provides additional information of the Folding@Home-clients configuration
  
- $folding_dir/FAHlog.txt       (optional)
  Folding@Home client log file
  provides information of the Folding@Home-clients environment and progress
  
- $folding_dir/FAHlog-Prev.txt 	(optional)
  Folding@Home client log file of previous work units
  optional file, might not exist
  provides information of the Folding@Home-clients environment and progress of
  previous work units
  
- $folding_dir/MyFolding.html   (optional)
  Folding@Home client web page
  provides URLs of the users web pages at the Stanford website

- $folding_dir/unitinfo.txt 	(optional)
  Folding@Home clients work unit summary
  optional file, might not exist 
  (some cores don't generate (a proper) unitinfo.txt)
  provides some information of the current work unit
  
- $folding_dir/work/current.xyz (optional)
  Molecule display file of the current work unit
  provides data to generate an image of the molecule


Install Paths
-------------
On Linux, FreeBSD and OpenBSD all the files requered by the client are stored 
in ~/.fci/, the client itself can be located anywhere. If the client cannot find
the files it needs in ~/.fci/ it will try to find them in the directory in 
which itself is stored. If on startup the ~/.fci/ directory is not found, it is
created and the required files will be copied from /usr/local/bin/ and 
/usr/local/share/fci/ on Linux, FreeBSD and OpenBSD. On Windows all the files 
are stored in the same folder, the folding in which the client was installed, 
C:\Program Files\FCI\ by default.

- fci-client.pl
  On Linux, FreeBSD and OpenBSD this file should be installed in a directory in
  $PATH, /usr/local/bin/ by default on Linux, FreeBSD and OpenBSD.
  On Windows this files is installed in C:\Program Files\FCI\ by default, the
  user can specify the installetion directory when using the install.pl script.

  Default:
   Linux   : /usr/local/bin/fci-client.pl
   FreeBSD : /usr/local/bin/fci-client.pl
   OpenBSD : /usr/local/bin/fci-client.pl
   Windows : C:\Program Files\FCI\fci-client.pl
  
- qd
  On Linux, FreeBSD and OpenBSD this file is located in the ".fci" subdirectory
  of the home directory of the user executing fci-client.pl.
  On Windows fci-client.pl assumes this file to be in the same place as itself.
  
  The master binary which will be copied to the ".fci" directory on the first
  run, it is located in /usr/local/bin/ on Linux, FreeBSD and OpenBSD.
  
  Default:
   Linux   : ~/.fci/qd
   FreeBSD : ~/.fci/qd
   OpenBSD : ~/.fci/qd
   Windows : C:\Program Files\FCI\qd.exe
  
- qdinfo.dat
  On Linux, FreeBSD and OpenBSD this file is located in the ".fci" subdirectory 
  of the home directory of the user executing fci-client.pl.
  On Windows fci-client.pl assumes this file to be in the same place as itself.
  
  The master file which will be copied to the ".fci" directory on the first
  run is located in /usr/local/share/fci/ on Linux, FreeBSD and OpenBSD.
  
  Default:
   Linux   : ~/.fci/qdinfo.dat
   FreeBSD : ~/.fci/qdinfo.dat
   OpenBSD : ~/.fci/qdinfo.dat
   Windows : C:\Program Files\fci\qdinfo.dat

- fci-client.conf
  On Linux, FreeBSD and OpenBSD this file is located in the ".fci" subdirectory 
  of the home directory of the user executing fci-client.pl.
  On Windows fci-client.pl assumes this file to be in the same place as itself.
  
  Default:
   Linux   : ~/.fci/fci-client.conf
   FreeBSD : ~/.fci/fci-client.conf
   OpenBSD : ~/.fci/fci-client.conf
   Windows : C:\Program Files\fci\fci-client.conf


Server
======

The server side of FCI consists of serveral parts. Most important is the Apache 
mod_perl application which handles the client data uploads, and the display of 
the processed data. The actual processing of the uploaded data is done by 
separate scripts.

Older versions of FCI processed the uploaded data every time a webpage was 
requested, this worked fine on >1 Ghz machines, but slower machines had quite
a rough time processing the data. Also with the pre-1.0 releases of fci, the
generation of the protein image from the uploaded current.xyz file was done
at the moment of upload. This made uploads quite slow, the client had to wait
for a long time to get its response on his upload.

Now the server only stores the data the client uploads, making the handling of
the uploads much faster. The server will also be able to handle serveral 
concurrent uploads much better than before. As it was not uncommon to see some
10 clients uploading their data all at once, thanks to the nature of scheduling 
uploads.

When clients now upload their data, everything is saved as-is. The only file 
which is processed is the current.xyz file if it was uploaded. The project 
number and work unit name are extracted from the file, and the project number
is checked against the list of known projects. If the project is listed in the
known projects file no further processing is done. If the project is not listed
the xyz file will be copied to the project-data directory and a new record will
be created for this project in the known-projects list.

The script fci-update-project-images.pl is run to generate an image for this
new project. By default the image type for uploads is a single still image,
because these take the least amount of time to generate. The idea is to 
schedule fci-update-project-images.pl to run once a day or once an hour to 
generate the images for the missing image types for all known projects.

This way every image is generated only once, instead of at very upload. The 
more load intensive image types as rotate and xyz180 can be scheduled to be
generated at a time when the system has least load.

The uploaded data will only be used by the display mode of the webapplication
after it has been processed into the XML files used by that same display mode.
These XML files are generated by fci-update-xml-files.pl. This script will 
process the data uploaded by all the clients, the most important client data 
being the data output by qd as stored in qd-data.xml.new on upload. If other 
data was also uploaded by a client, these files will also be parsed by 
fci-update-xml-files.pl and used to update the XML files.

Besides client data, fci-update-xml-files.pl will also use some of the stats
files provides by Pandegroup on the Stanford webservers. These files are 
downloaded and processed by fci-update-stanford-files.pl. The files in 
question are the teamstats files (team0.txt), project summary (psummary.html),
the user stats file (daily_user_summary.txt) and the team stats file 
(daily_team_summary.txt). The psummary is updated every hour, the teamstats 
files are updated every 3 hours and the daily summaries are updated every 4 
hours. The files will only be downloaded and processed if their Last-Modified
date has changed since it was last downloaded. It is therefore advised to 
schedule fci-update-stanford-files.pl to run every, so your project summary 
will be updated as soon as possible, whereas the other files are only also 
downloaded if they have actually changed.

Because the Stanford files are used by fci-update-xml-files.pl, 
fci-update-stanford-files.pl should have been run before 
fci-update-xml-files.pl. I personally schedules my clients to upload every
hour on the 0th minute. On the server I schedule 
fci-update-stanford-files.pl to run every hour on the 0th minute. Every hour
on the 30th minute, I then run fci-update-xml-files.pl. Scheduling the 
different scripts in this way will make sure the fci-update-xml-files.pl is run
after the data sources it uses have been updated.

Independent of the data required by fci-update-xml-files.pl, which is mostly
focussed on client, user and team data, there is the data of the projects
which are being worked on by the clients. On client uploads an image is already
generated for each project, and so an xyz and pdb file are available for it 
too. To generate the images for the remaining image types run 
fci-update-project-images.pl, whose default behaviour is to generate all
missing images and their accompanying rasmol scripts and pdb files.

I have scheduled fci-update-project-images.pl to run every hour on the 45th 
minute, after fci-update-xml-files has run (on the 30th minute). 


Data sources
------------
The server connects all the data source together, the client supplied data and
that provided by Stanford and third parties. Below is a list of data sources 
which the server uses and a description of what information it provides.

- fci-client.pl
  The client application of fci
  main source of information,
  provides information from the clients in the form of several files 
  file upload is done via HTTP POST from the client machine
  
- http://fah-web.stanford.edu/daily_user_summary.txt
  Folding@Home project user stats
  provides information of the users in the project
  previously at: http://vspx27.stanford.edu/daily_user_summary.txt
  
- http://fah-web.stanford.edu/daily_team_summary.txt 
  Folding@Home project user stats
  provides information of the teams in the project
  previously at: http://vspx27.stanford.edu/daily_team_summary.txt
  
- http://fah-web.stanford.edu/psummary.html
  Folding@Home projects list
  provides information of the different projects running on Folding@Home
  previously at: http://vspx27.stanford.edu/psummary.html


Applications:
-------------
The server part of FCI uses and hosts some applications for use by either the 
server or the clients. The qd application is used by the clients to gather 
information of their Folding@Home client. Updated versions of qd and its 
data file, qdinfo.dat, are hosted on the server, before each upload, the client
downloads the updated files from the server if a newer version are available.
Some other applications are only used by the server, for processing of the 
uploaded information. 

- http://www.boston.quik.com/rph/qd
  Linux application to parse the queue.dat file of the Folding@Home client
  Runs on the Client, updates are stored on the server for automatic update
  
- http://www.boston.quik.com/rph/qd.exe
  Windows version of qd
  
- http://www.boston.quik.com/rph/qdinfo.dat
  Data file containing the information of the running projects for use in qd

- http://www.boston.quik.com/rph/xyz2pdb
  Linux application to convert an xyz molecule display file to a pdb file
  Used in combination with rasmol to generate an image of the molecule

- http://www.bernstein-plus-sons.com/cgi-bin/yaya/decomp.cgi/software/ \
  RasMol_2.7.2.1.1/RasMol.LINUX/RedHat_7/i386/rasmol_32BIT
  Linux application to manipulate molecule display files
  Used to generate images from display files
  
- http://www.bernstein-plus-sons.com/cgi-bin/yaya/decomp.cgi/software/ \
  RasMol_2.7.2.1.1/doc/rasmol.hlp
  Help document for rasmol binary

- fci-update-qd-files.pl
  Application to download the latest qd files to update the 
  client application pool
  Downloads the files:
   http://www.boston.quik.com/rph/qd
   http://www.boston.quik.com/rph/qd.exe
   http://www.boston.quik.com/rph/qdinfo.dat

- fci-update-stanford-files.pl
  Application to download the latest stanford files to update the 
  client application pool
  Downloads the files:
   http://vspx27.stanford.edu/daily_team_summary.txt
   http://vspx27.stanford.edu/daily_user_summary.txt
   http://vspx27.stanford.edu/psummary.html