CFDDFC on AWS: Data Transfer to an Instance

Secure copy protocol

Files can be transferred to and from an instance using the secure copy protocol (SCP), based on SSH described previously. SSH with a key pair provides both encryption for data transfer and authentication of the user. The OpenSSH secure copy application “scp” enables file transfer from the command line on Linux, MacOS and Windows 10.

Copying files with scp

The basic command copies file(s) from a source location to a target location:

scp <source> <target>

A source or target is on the local machine is simply specified by the directory path and file name, e.g. “/home/john/foo.txt”. This example uses the full directory path, but a relative path can be used and the path can be omitted if the file is in the current directory.

The source or target on the remote instance is specified by <user>@<host>:<file/dir>. For example, for user ubuntu, at IP address 123.45.67.89, a target home directory is specified as ubuntu@123.45.67.89:~/. The command to copy a foo.txt file to this location is:

scp foo.txt ubuntu@123.45.67.89:~/

This command will work if the relevant private key is added to the the SSH agent. Otherwise, the key file would need to be supplied with the command using the -i option, e.g.

scp -i ~/.ssh/awskey.pem foo.txt ubuntu@123.45.67.89:~/

Copying a file named foo.txt from the home directory on the instance back to the local machine would be (note the target “dot”):

scp -i ~/.ssh/awskey.pem ubuntu@123.45.67.89:~/foo.txt .

Copying directories

A directory can be copied with scp using the -r option. For example, imagine we want to copy the pitzDaily case directory in our current directory to the instance. The case can be copied with the following command — assuming the key is added to the agent, otherwise include the -i <keyfile> option.

scp -r pitzDaily ubuntu@123.45.67.89:~/

While scp can be used for copying directories is likely that users will want to synchronize files on local and remote machines. The rsync application is better suited to synchronization of files and uses ssh as the default shell for connecting to the remote instance. Assuming the key is added to the agent, the command to copy the pitzDaily case is:

rsync -av pitzDaily ubuntu@123.45.67.89:~/

The -a option means “archive mode” which allows copying of directories while maintaining file timestamps, permissions and links. The -v option writes more output, reporting on the copied files. Without the agent, the key file can be supplied as follows:

rsync -av -e "ssh -i ~/.ssh/awskey.pem" pitzDaily ubuntu@123.45.67.89:~/

Note: be careful about trailing slashes / on the ends of directory names which changes the meaning of the rsync command. The files can be synchronized from the remote instance to the local machine. It can be tested by creating an empty file foo.txt in the pitzDaily case on the remote machine by running the touch command remotely:

ssh ubuntu@123.45.67.89 "touch pitzDaily/foo.txt"

The pitzDaily case is then synchronized back to the local machine, noting that only the foo.txt file is copied.

rsync -av ubuntu@123.45.67.89:~/pitzDaily .

The user can also ensure that files are deleted at the target which do not exist at the source using the --delete option. This can be tested by deleting the foo.txt file created previously on the remote VM.

ssh ubuntu@123.45.67.89 "rm pitzDaily/foo.txt"

The file is then deleted on the local machine by running the following command:

rsync -av --delete ubuntu@123.45.67.89:~/pitzDaily .

Data compression

The -z option can be included “rsync -avz…” which compresses file data for transfer, which helps to reduce file transfer costs. Alternatively users can reduce the need for compressing data for transfer by writing data in binary format using the following option in the case controlDict file:

writeFormat      binary;

or compress the ASCII data with the options:

writeFormat      ascii;
writeCompression on;

Users may wish to synchronize case directories into the “run” directory on the remote instance, denoted by the OpenFOAM environment variable $FOAM_RUN. To do this, the environment variable should be “backquoted” to prevent its expansion on the local machine, ensuring it is correctly expanded on the remote instance, e.g.:

rsync -av pitzDaily ubuntu@123.45.67.89:\$FOAM_RUN/

Note: the run directory must be created on the remote instance, which occurs with the first SSH login into it.

Next Step → Connecting with the Remote Desktop