User Guide - Resources

This document describes the resources at SDSC, Texas, and Michigan that make up the NPACI Grid.  Following this summary, a table of NPACI Grid login hosts is provided, followed by a a Grid Services Matrix specifying the services available on the NPACI Grid.

If you have an NPACI account but do not have access to one or more of the NPACI Grid resources, submit an account extension request at:

http://npacigrid.npaci.edu/account_extension_request.html

SDSC

Blue Horizon

Blue Horizon is a teraflop-scale Power3 based clustered SMP system from IBM.  The machine contains 1,152 processors and 576 GBytes of main memory, arranged as 144 Symmetric Multiprocessing (SMP) compute nodes.  Each node is equipped with 4 GBytes of memory shared among its 8 - 375 MHz Power3 processors.  Each node also has several GBytes of local disk space.  Nodes are connected by the Colony switch, a proprietary IBM interconnect.

Although most of the Blue Horizon resources are batch systems, 15 nodes are available for interactive use.  These "b80 nodes" are for developing and debugging parallel codes and are not intended for making production runs.  For more information:

Blue Horizon Home Page
System Access
Running Batch Jobs
Running Interactive Jobs

Griddle

Griddle is the host where Condor-G jobs must be submitted on the NPACI Grid. From griddle. jobs may be launched on any NPACI Grid resource.  Griddle has an Intel Pentium 4 IA32 processor and runs Redhat Linux version 7.2. 

Texas

Longhorn

The TACC IBM Power4 System consists of three IBM p690-HPC shared-memory server nodes, one IBM p690-Turbo shared memory node, and 32 IBM p655 shared-memory server nodes along with an IBM p690-HPC node as a login front-end.  Each of the p690-HPC server nodes contains 16 Power4 processors running at 1.3 GHz, while the p690-Turbo contains 32 Power4 processors at the same speed.  Each of the p655 server node contains 4 Power4 processors running at 1.3 GHz.  In total, the 224 total processor system has a peak performance 1.16 Tflops and an aggregate memory of 512 GB.  Each node is supported by 36/18 GB of local disks with an aggregate of 3/4 TB, and the faster storage GPFS is connected through the IBM Switch2 for a total of 7.1 TB of storage disk space.  The Power4 Systems run AIX, a scalable UNIX operating system with High Availability Cluster Multi-Processing (HACMP) capabilities.  For more information:

Longhorn Home Page
System Access
Running Batch Jobs
Running Interactive Jobs

Michigan

Hypnos and Morpheus

The hypnos cluster has 128 nodes of dual Athlon 2000MP CPUs and is reserved for NPACI users.  The morpheus cluster is for general use and consists of 50 nodes of dual Athlon 1600MP CPUs plus 17 nodes of dual Athlon 2600MP CPUs.  Each SMP node consists of two CPUs with one gigabyte of memory available per processor.  These AMD clusters run Red Hat Linux 7.2 and 7.3 with the typical GNU and Linux tools and utilities installed.  For more information:

AMD Clusters Home Page
System Access
Running batch and interactive jobs

Login Hosts

The following table specifies the hosts where you can login and access NPACI Grid services

NPACI Grid Login Nodes
Resource Site	Host	Notes
Blue Horizon	tf004i.sdsc.edu tf005i.sdsc.edu	Batch job submission only
	b80n01.sdsc.edu to b80n13.sdsc.edu	Interactive job submission only
Griddle	griddle.sdsc.edu	Condor-G job submission and monitoring
Longhorn	longhorn.tacc.utexas.edu	Batch & interactive job submission
	archive.tacc.utexas.edu	Long term storage
Michigan	morpheus.engin.umich.edu hypnos.engin.umich.edu	Batch & interactive job submission

Grid Services Matrix

This table below specifies the services and parameters necessary for using the NPACI Grid A key is provided at the end of the table.

NPACI Grid Services & Parameters
General (not site specific)
GIIS	Server	giis.npaci.edu
	Port	2135
NWS	Name Server Host	nws.npaci.edu
	Name Server Port	8090 (default)
	Memory Server Host	nws.npaci.edu
	Memory Server Port	8070 (default)
Condor-G	Job Submission & Monitoring Host	griddle.sdsc.edu
SDSC - Blue Horizon
Globus Job Submission	Gatekeepers	tf004i.sdsc.edu and tf005i.sdsc.edu for batch and interactive jobs b80n01.sdsc.edu through b80n13.sdsc.edu for interactive jobs
	Jobmanagers	jobmanager-fork (default) jobmanager-loadleveler (batch)
	Required RSL params for loadleveler on the b80 nodes	(queue=interactive) (max_wall_time=45) (environment=(MP_EUIDEVICE en0))
	Required RSL params for loadleveler on the tf004i and tf005i nodes	(queue=normal) (max_wall_time=45)
GridFTP	Server	tf004i.sdsc.edu
GSI SSH	Server	tf004i.sdsc.edu
	Port	1022
TACC - Longhorn
Globus Job Submission	Gatekeeper	longhorn.tacc.utexas.edu
	Jobmanagers	jobmanager-fork (default) jobmanager-loadleveler (batch)
	Required RSL params for loadleveler	(queue=normal) (max_wall_time=45) (max_memory=10) max_wall_time is in minutes; max_memory is in megabytes; set
GridFTP	Server	longhorn.tacc.utexas.edu
GSI SSH	Server	longhorn.tacc.utexas.edu
	Port	1022
UMich - Hypnos
Globus Job Submission	Gatekeeper	hypnos.engin.umich.edu
	Jobmanagers	jobmanager-fork (default) jobmanager-pbs (batch)
	Required RSL params for pbs	(queue=route) (max_wall_time=45) (email_address=your@email) max_wall_time is in minutes; the above value of 45 is an example
GridFTP	Server	hypnos.engin.umich.edu
GSI SSH	Server	hypnos.engin.umich.edu
	Port	1022
UMich - Morpheus
GRAM Job Submission	Gatekeeper	morpheus.engin.umich.edu
	Jobmanagers	jobmanager-fork (default) jobmanager-pbs (batch)
	Required RSL params for pbs	(queue=npaci) (max_wall_time=45) (email_address=your@email)
GridFTP	Server	morpheus.engin.umich.edu
GSI SSH	Server	morpheus.engin.umich.edu
	Port	1022

Key

• GRAM Job Submission: Services and parameters for submitting Globus jobs
  
  • Gatekeeper: The node where the globus GRAM server is running
  • Jobmanagers: Specifies the default job manager (usually fork) and any other jobmanagers available, such as those for submitting batch jobs
  • Required RSL Parameters: Specific parameters required for submitting jobs via the given job manager.  Job submission will fail if these parameters are not set.
    
• GridFTP: Host where the GridFTP server is running, specified for grid ftp transfer commands
  
• GSH SSH: Host where the GSI SSH server is running.  Note that this server uses a different port than SSH, which must be explicitly specified when running GSI SSH commands.