Running CFDDFC with the AWS Console
- Launch an Instance
- Connect to an Instance
- Using OpenFOAM on an Instance
- Data Transfer to an Instance
- Connect with the Remote Desktop
- Data Storage
- Creating a Cluster of Instances
Secure copy protocol
Files can be transferred to and from an instance using the secure copy protocol (SCP), based on SSH described previously. SSH with a key pair provides both encryption for data transfer and authentication of the user. The OpenSSH secure copy application “
scp” enables file transfer from the command line on Linux, MacOS and Windows 10.
Copying files with scp
The basic command copies file(s) from a source location to a target location:
scp <source> <target>
A source or target is on the local machine is simply specified by the directory path and file name, e.g. “
/home/john/foo.txt”. This example uses the full directory path, but a relative path can be used and the path can be omitted if the file is in the current directory.
The source or target on the remote instance is specified by
<user>@<host>:<file/dir>. For example, for user
ubuntu, at IP address
126.96.36.199, a target home directory is specified as
email@example.com:~/. The command to copy a
foo.txt file to this location is:
scp foo.txt firstname.lastname@example.org:~/
This command will work if the relevant private key is added to the the SSH agent. Otherwise, the key file would need to be supplied with the command using the
-i option, e.g.
scp -i ~/.ssh/awskey.pem foo.txt email@example.com:~/
Copying a file named
foo.txt from the home directory on the instance back to the local machine would be (note the target “dot”):
scp -i ~/.ssh/awskey.pem firstname.lastname@example.org:~/foo.txt .
A directory can be copied with
scp using the
-r option. For example, imagine we want to copy the
pitzDaily case directory in our current directory to the instance. The case can be copied with the following command — assuming the key is added to the agent, otherwise include the
-i <keyfile> option.
scp -r pitzDaily email@example.com:~/
scp can be used for copying directories is likely that users will want to synchronize files on local and remote machines. The
rsync application is better suited to synchronization of files and uses
ssh as the default shell for connecting to the remote instance. Assuming the key is added to the agent, the command to copy the
pitzDaily case is:
rsync -av pitzDaily firstname.lastname@example.org:~/
-a option means “archive mode” which allows copying of directories while maintaining file timestamps, permissions and links. The
-v option writes more output, reporting on the copied files. Without the agent, the key file can be supplied as follows:
rsync -av -e "ssh -i ~/.ssh/awskey.pem" pitzDaily email@example.com:~/
Note: be careful about trailing slashes / on the ends of directory names which changes the meaning of the rsync command. The files can be synchronized from the remote instance to the local machine. It can be tested by creating an empty file
foo.txt in the
pitzDaily case on the remote machine by running the
touch command remotely:
ssh firstname.lastname@example.org "touch pitzDaily/foo.txt"
pitzDaily case is then synchronized back to the local machine, noting that only the
foo.txt file is copied.
rsync -av email@example.com:~/pitzDaily .
The user can also ensure that files are deleted at the target which do not exist at the source using the
--delete option. This can be tested by deleting the
foo.txt file created previously on the remote VM.
ssh firstname.lastname@example.org "rm pitzDaily/foo.txt"
The file is then deleted on the local machine by running the following command:
rsync -av --delete email@example.com:~/pitzDaily .
-z option can be included “
rsync -avz…” which compresses file data for transfer, which helps to reduce file transfer costs. Alternatively users can reduce the need for compressing data for transfer by writing data in binary format using the following option in the case
or compress the ASCII data with the options:
writeFormat ascii; writeCompression on;
Users may wish to synchronize case directories into the “
run” directory on the remote instance, denoted by the OpenFOAM environment variable
$FOAM_RUN. To do this, the environment variable should be “backquoted” to prevent its expansion on the local machine, ensuring it is correctly expanded on the remote instance, e.g.:
rsync -av pitzDaily firstname.lastname@example.org:\$FOAM_RUN/
run directory must be created on the remote instance, which occurs with the first SSH login into it.