CFDDFC on AWS: Launch a Cluster of Instances with the CLI

A Cluster of Instances

Once an instance is running, the user can launch further instances with the cluster subcommand.  A networked cluster is created with the original instance as the main node and additional instances as the secondary nodes.  The user accesses the cluster via the main instance, via SSH login or remote desktop.  Data is written to the elastic block storage (EBS) of the main instance.  The storage is accessible to the secondary instances across a virtual private network using a network file system (NFS).

Creating a Cluster of Instances

When creating a cluster, the user should first launch a main instance.  We recommend using the C5n series of instances which provide significantly higher network performance across all instance sizes.  Since the purpose of clustering is to access more cores than is available on a single instance, it makes sense to launch the largest instance in the series, so c5n.18xlarge is the natural choice.

The user needs to take care to provision sufficient storage on the main instance, since it stores all the data for the cluster.  The required storage depends on the type of simulation being performed and the configuration of data processing in OpenFOAM.  1 GB per core provides a reasonable starting point for determining the volume size.  EBS pricing is approximately $0.10 per GB-month (think: $1 per day for 300 GB).  Therefore, to launch the main instance with 300 GB of storage, the user would run the launch command with the following options:

cfddfc launch -instance c5n.18xlarge -volume 300

Once the main instance is running, a cluster of instances can be created using the cluster subcommand, specifying the number of secondary instances.  The number of secondary instances is specified by the -secondary option.  If the user wanted to created a cluster of 3 instances, i.e. of 3 × 36 = 108 cores, the user would add 2 secondary instances to the main instance by the following command:

cfddfc cluster -secondary 2

In the above command the instance ID of the main instance can be specified , e.g. aB1cD2eF, as an argument with the -key option:

cfddfc cluster -secondary 2 -key aB1cD2eF

Running in Parallel across the Cluster

The cluster subcommand creates a file named machines containing the local IP addresses of the main and secondary instances on the private network within AWS EC2.  The file is copied automatically into the $HOME/OpenFOAM directory.  The user can then run an OpenFOAM application, e.g. simpleFoam across all nodes on the cluster by launching the mpirun command with the machines file as an argument to the -hostfile option. 

For example, the following command runs simpleFoam over 108 cores on the 3 instances whose IP addresses are specified in the machines file, redirecting output to a file named log.simpleFoam:

mpirun -np 108 -hostfile $HOME/OpenFOAM/machines simpleFoam -parallel > log.simpleFoam &

Next Step → Terminate an Instance