SI2006 track 1 Gfarm Installation and Configuration

From NBCRwiki

Jump to: navigation, search

Back to Track 1


Related Links

Gfarm testbed at PRAMGA
Gfarm official site

List of Hosts

metadata server

metadata cache server

file system nodes Public IP address required

Required software

  • PostgreSQL (8.1 or later is recommended)
  • Globus GSI Lib
  • Kerberos Lib

Optional software

  • OpenLDAP (2.2 or later is recommended)


  1. postgresql PostgreSQL download page
    • required for Gfarm metadata server
  2. openldap OpenLDAP download page
    • if used for metadata server (not required if using PostgreSQL)
  3. Gfarm1.3.1 Gfarm 1.3.1 download page
  4. Globus Toolkit GT4 download page
    • required for CSF4 machine, could be every frontend of cluster
  5. Kerberos Lib a available link
    • Note: Rocks4.1 needn't install Kerberos Lib; required by Gfarm meta data server.


Metadata Server

  1. Install PostgreSQL 8.1
    • official installation guide
    • Note: Please run initdb with -A argument, eg. $ /usr/local/pgsql/bin/initdb -D /usr/local/pgsql/data -A trust
  2. Install Globus GSI Lib and Kerberos Lib
  3. Install OpenLDAP if you want to take openldap as metadata database

Globus Toolkit installation

  1. Build and install Globus binary package (not a full installation)
    • [~$]# wget
    • [~$]# rpm -ivh globus-4.0.2-1.src.rpm
    • [~$]# rpmbuild -bb /usr/src/redhat/SPEC/globus.spec
    • [~$]# cd /usr/src/redhat/RPMS/i386
    • [~$]# rpm -ivh globus-gpt-4.0.2-1.i386.rpm globus-gssapi-gsi-gcc32-4.0.2-1.i386.rpm globus-proxy-utils-gcc32-4.0.2-1.i386.rpm

Gfarm Installation

Build Gfarm Binaries

  1. Build Gfarm binary package

Gfarm MetaData Server

  1. Install gfarm metaserver,
    • [root@gfarm gfarm1.3]# rpm –Uvh gfarm-gsi-libs-1.3-0.i386.rpm
    • [root@gfarm gfarm1.3]# rpm –Uvh gfarm-gsi-server-1.3-0.i386.rpm
  2. Metaserver configuration
    • [root@gfarm gfarm1.3]# export LD_LIBRARY_PATH=/usr/lib:$LD_LIBRARY_PATH,
    • [root@gfarm gfarm1.3]# config-gfarm -b postgresql -P /usr/local/pgsql
    • Note 1: Don't startup pgsql manually
    • Note 2: For the firewall configure, Open tcp port of postmaster and gfmd(10602 and 600 by default)
      • Copy /etc/gfarm.conf from metadata server to gfarm cluster frontend
      • specify metadb_serverhost in /etc/gfarm.conf
    • [root@~$]# rpm -Uvh gfarm-gsi-{libs,server}-1.3.1-0.i386.rpm
    • [root@~$]# chkconfig gfmd --add
    • [root@~$]# service gfmd start

Gfarm Cache Server

  1. setup gfarm cache server and config-agent
    • [root@~$]# rpm -Uvh gfarm-gsi-agent-1.3.1-0.i386.rpm
    • [root@~$]# config-agent
    • Note 1: For the firewall configure, Open tcp port of gfarm_agent(603 by default).
    • Note 2: the Cache server may be set up as necessary for distinct physical locations. In our case, the two clusters are very close, and the number of nodes are small, only one is necessary.

File System Nodes

  1. setup gfarm file node and config-gfsd
    • # rpm -Uvh postgresql-lib-7.4.13-2.RHEL4.1.i386.rpm
    • # rpm -ivh globus-gssapi-gsi-libs-gcc32-4.0.2-1.i386.rpm
    • Note 0: if globus-gssapi-gsi-gcc32 is already installed, this is not needed.
    • # rpm -Uvh gfarm-gsi-{libs,fsnode,client}-1.3.1-0.i386.rpm
    • Copy /etc/gfarm.conf from rocks-32 (metadata cache server)
    • # config-gfsd -h <> /var/spool/gfarm
      • for cluster frontends, /export2/spool/gfarm is recommended. Tyical disk space recommended is 20 GB
    • Note 1: Make sure CA certificates in /etc/grid-security/certificates
    • Note 2: Make sure hostcert.pem and hostkey.pem in /etc/grid-security, and mode of hostkey.pem is 400
    • Note 3: For the firewall configure, Open tcp port of gfsd(601 by default)
  2. Gfarm-Fuse installation
    1. Build FUSE binary package
      • [root@~$]# rpm -ivh /home/osamu/srpm/fuse-2.5.3-1.src.rpm
      • [root@~$]# rpmbuild -bb /usr/src/redhat/SPECS/fuse.spec
    2. Install FUSE on every file system node
      • # rpm -Uvh fuse-2.5.3-1.i386.rpm
    3. Build GfarmFS-FUSE binary package
      • [root@~$]# rpm -Uvh gfarm-gsi-devel-1.3.1-0.i386.rpm
      • [root@~$]# rpm -ivh /home/osamu/srpm/gfarmfs-fuse-1.3.0-0.src.rpm
      • [root@~$]# rpmbuild -bb /usr/src/redhat/SPECS/gfarmfs.spec
    4. Install GfarmFS-FUSE on every file system node
      • # rpm -Uvh gfarmfs-fuse-1.3.0-0.i386.rpm
      • Note: Install Gfarm-fuse on all the file nodes

Repeat for all other nodes

during config-gfsd, use

config-gfsd -h /state/partition1/spool/gfarm
  • Rocks has a mechanism to install all files automatically in case of a compute node reinstall due to power outages
    • use extend-compute.xml to add the configurations during reinstall
total 2028
-rw-r--r--  1 root root 312147 Aug 26 17:04 fuse-2.5.3-1.i386.rpm
-rw-r--r--  1 root root    391 Aug 26 17:03 gfarm.conf
-rw-r--r--  1 root root  45623 Aug 26 17:04 gfarmfs-fuse-1.3.0-0.i386.rpm
-rw-r--r--  1 root root 100339 Aug 26 17:02 gfarm-gsi-client-1.3.1-0.i386.rpm
-rw-r--r--  1 root root  53079 Aug 26 17:02 gfarm-gsi-fsnode-1.3.1-0.i386.rpm
-rw-r--r--  1 root root 687060 Aug 26 17:02 gfarm-gsi-libs-1.3.1-0.i386.rpm
-rw-r--r--  1 root root 837572 Aug 26 17:02 globus-gssapi-gsi-libs-gcc32-4.0.2-1.i386.rpm
drwxr-xr-x  2 root root   4096 Aug 26 17:00 postgres

total 3216
-rw-r--r--  1 root root  149597 Aug 26 17:00 postgresql-libs-7.4.13-2.RHEL4.1.i386.rpm
  • Please check for host keys, and grid-mapfile and trusted CA information in /etc/grid-security/

Firewall and Security

/etc/sysconfig/iptables /etc/grid-secruity/grid-mapfile

PRAGMA CA certificates


/etc/sysconfig/iptables needs to open firewalls.
-A INPUT -i eth1 -p tcp -m tcp --dport 600 -j ACCEPT 
-A INPUT -i eth1 -p tcp -m tcp --dport 601 -j ACCEPT 
-A INPUT -i eth1 -p tcp -m tcp --dport 603 -j ACCEPT 
-A INPUT -i eth1 -p udp -m udp --dport 600 -j ACCEPT 
-A INPUT -i eth1 -p tcp -m tcp --dport 10602 -j ACCEPT 

Mount gfarm-fuse

Each user needs to do this

  • create /gfs/$USER directory on every file system node
  • $ gfarmfs -lsfu /gfs/$USER
  • Note 1: Mount gfarm-fuse as user who prefer Gfarm on all the file nodes
  • Note 2: Mount gfarm-fuse at a directory of local file system. Do not use NFS.
  • Note 3: Do not specify '-o kernel_cache' when Gfarm file system is mounted from more than two clients.

Unmount gfarm-fuse

  • $ fusermount -u /gfs/$USER
  • Note: Only mounter can unmount

Metadata database backup and restore

  1. dump
    •  % gfdump -d > file.dump
    •  % gfhost -M > host.dump
    • Note: dumping metadata is possible on metadata server or metadata cache servers.
  2. restore
    • Create metadata server from the scratch by config-gfarm
    •  % gfdump -r < file.dump
    •  % gfhost -R < host.dump
    • Note: restoring metadata is possible on metadata server or metadata cache servers.

Testing jobmanager-sge

gfhost -lv
globusrun -a -r
globus-job-run /bin/hostname

Back to Track 1

Personal tools