Case Study - Monte Carlo Cellular Microphysiology on the Grid

Project Leaders: Terrence Sejnowski, Salk Institute;Henri Casanova, UCSD
Project Manager:Tom Bartol, Salk Institute
Project Web Sites:

http://www.mcell.cnl.salk.edu/
http://grail.sdsc.edu/projects/apst/
http://www.cs.ucsd.edu/groups/hpcl/scg/kelp.html

A piece of cerebral cortex the size of a large grain of sand may contain 5 billion interdigitated synapses of different shapes and sizes.  This microscopic intricacy presents a major challenge to researchers trying to gain a fundamental understanding of the brain and other biological structures that exhibit such diversity and complexity at the subcellular level.

In order to analyze the microscopic structure of brain tissue, researchers are using Monte Carlo simulations.  One of the most successful is MCell, a general Monte Carlo simulator of cellular microphysiology developed by Tom Bartol and Joel Stiles when they were at Cornell University (with Ed Salpeter, and the late Miriam Salpeter).  Bartol and Stiles are now at the Salk Institute and the Pittsburgh Supercomputing Center, respectively.

MCell produces highly realistic 3-D simulations of subcellular architecture and physiology, allowing unexplored aspects of neural signaling to be quantitatively modeled.  Together with partners at UC San Diego (UCSD) and the University of Tennessee, participants in this
alpha project have made significant progress toward providing a grid-enabled version of MCell that can handle much larger, more realistic data sets than previously possible.

MCell researchers have been limited by the lack of adequate computational infrastructure and the ability to use it to accommodate large-scale simulations as well as to navigate and map large parameter spaces.  Key limiting factors include the ability to efficiently access
remote computational, storage, and federated database resources, the ability to schedule the application so as to exploit fluctuating deliverable resource performance, and the ability to manage distributed and heterogeneous resources within a unified, service-oriented framework.  MCell is representative of a large class of NPACI metasystem applications which, through NPACKage, can now leverage NPACI and other Grid resources for large runs -- a feat which previously could only be accomplished with considerable difficulty.

There are three typical scenarios for use of MCell:

1. Small scale usage on typical workstations or small clusters of workstations.
2. Large-scale parameter sweep usage in a metacomputing environment.  This is accomplished using MCell with APST.
3. Single large-scale simulations on massively parallel supercomputers such as BlueHorizon.  This is accomplished using MCell with KeLP, called MCell-K.

APST is a Grid application execution environment that performs automatic scheduling and deployment of "parameter sweep" applications, that is applications consisting of large sets of independent computational tasks with potentially large datasets.  MCell belongs to this class of applications.  APST interfaces to several middleware infrastructures to launch and monitor computation on a wide variety of Grid resources, to move data among Grid storage resources, and to gather information about the status of the Grid platform.  APST uses sophisticated scheduling algorithms that take into account the cost of data movements when scheduling computation.

APST provides MCell users with transparent access to local and NPACI resources, shielding the user, to a large degree, from the vast array of authentication, storage, and queuing systems to be encountered.

Furthermore, APST provides a convenient XML-based interface by which users can specify their applications.   Consequently, APST is an ideal environment for MCell users as it provides them with transparent access to local and NPACI resources, shielding the user, to a large degree, from the vast array of authentication, storage, and queuing systems to be encountered.

KeLP brings to MCell-K a powerful abstraction layer to the MPI distributed parallel message passing system, giving an elegant, high-level interface and thus hiding the low-level intricacies of handling messages between parallel processors.  MCell-K is still being tested and validated but preliminary tests have demonstrated 70% parallel efficiency on runs scaled to 256 processors.

Due to the integrated design of APST and KeLP, MCell makes effective use of the following NPACKage components:

• APST
• NWS
• Globus
• GSI-OpenSSH
• Condor-G
• MPICH-G2

Using APST, we have successfully deployed large-scale MCell parameter sweeps on BlueHorizon to map part of a 4-D parameter space representing the transmission behavior at a nerve-muscle synapse.  This required 47040 runs, which completed in 48 hours running on a combination of 512 and 1024 processors and generated 50 gigabytes of output data representing new disciplinary results.  An additional set of 34560 runs was performed on 256 processors to form a preliminary sparse mapping of a 7-D parameter space.

We are presently performing an even larger parameter sweep to study the behavior of a synapse involved in controlling the pupil of the eye.  This study will require 300000 runs which be performed simultaneously on BlueHorizon at SDSC, LeMieux at PSC, Longhorn at University of Texas, Morpheus at University of Michigan, and on local cluster resources at Salk.  Thanks to APST and NPACKage, it is finally feasible to perform such multi-center parameter sweep studies.