Skip to content

Data transfer

Transfer protocols#

A number of methods allow transferring data in/out of Sherlock. For most cases, we recommend using SSH-based file transfer commands, such as scp, sftp, or rsync. They will provide the best performance for data transfers from and to campus.

Most casual data transfers could be done through the login nodes, by pointing your transfer tool to login.sherlock.stanford.edu. But because of resource limits on the login nodes, larger transfer may not work as expected.

For transferring large amounts of data, Sherlock features a specific Data Transfer Node, with dedicated bandwidth, as well as a managed Globus Online endpoint, that can be used for scheduled, unattended data transfers.

We also provide tools on Sherlock to transfer data to various Cloud providers, such as AWS, Google Drive, Dropbox, Box, etc.

Prerequisites#

Most of the commands detailed below require a terminal and an SSH client1 on your local machine to launch commands.

You'll need to start a terminal and type the given example commands at the prompt, omitting the initial $ character (it just indicates a command prompt, and then should not be typed in).

Host keys#

Upon your very first connection to Sherlock, you will be greeted by a warning such as :

The authenticity of host 'login.sherlock.stanford.edu' can't be established.
ECDSA key fingerprint is SHA256:eB0bODKdaCWtPgv0pYozsdC5ckfcBFVOxeMwrNKdkmg.
Are you sure you want to continue connecting (yes/no)?

The same warning will be displayed if your try to connect to one of the Data Transfer Node (DTN):

The authenticity of host 'dtn.sherlock.stanford.edu' can't be established.
ECDSA key fingerprint is SHA256:eB0bODKdaCWtPgv0pYozsdC5ckfcBFVOxeMwrNKdkmg.
Are you sure you want to continue connecting (yes/no)?

This warning is normal: your SSH client warns you that it is the first time it sees that new computer. To make sure you are actually connecting to the right machine, you should compare the ECDSA key fingerprint shown in the message with one of the fingerprints below:

Key type Key Fingerprint
RSA SHA256:T1q1Tbq8k5XBD5PIxvlCfTxNMi1ORWwKNRPeZPXUfJA
legacy format: f5:8f:01:46:d1:f9:66:5d:33:58:b4:82:d8:4a:34:41
ECDSA SHA256:eB0bODKdaCWtPgv0pYozsdC5ckfcBFVOxeMwrNKdkmg
legacy format: 70:4c:76:ea:ae:b2:0f:81:4b:9c:c6:5a:52:4c:7f:64

If they match, you can proceed and type ‘yes’. Your SSH program will then store that key and will verify it for every subsequent SSH connection, to make sure that the server you're connecting to is indeed Sherlock.

Host keys warning#

If you've connected to Sherlock 1.0 before, there's a good chance the Sherlock 1.0 keys were stored by your local SSH client. In that case, when connecting to Sherlock 2.0 using the sherlock.stanford.edu alias, you will be presented with the following message:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: POSSIBLE DNS SPOOFING DETECTED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
The RSA host key for sherlock.stanford.edu has changed, and the key for
the corresponding IP address 171.66.97.101 is unknown. This could
either mean that DNS SPOOFING is happening or the IP address for the
host and its host key have changed at the same time.
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle
attack)!  It is also possible that a host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
SHA256:T1q1Tbq8k5XBD5PIxvlCfTxNMi1ORWwKNRPeZPXUfJA.
Please contact your system administrator.

You can just check that the SHA256 key listed in that warning message correctly matches the one listed in the table above, and if that's the case, you can safely remove the sherlock.stanford.edu entry from your ~/.ssh/known_hosts file with the following command on your local machine:

$ ssh-keygen -R sherlock.stanford.edu

and then connect again. You'll see the first-connection prompt mentioned above, and your SSH client will store the new keys for future connections.

SSH-based protocols#

Login name

In all the examples below, you'll need to replace <sunetid> by your actual SUNet ID. If you happen to use the same login name on your local machine, you can omit it..

SCP (Secure Copy)#

The easiest command to use to transfer files to/from Sherlock is scp. It works like the cp command, except it can work over the network to copy files from one computer to another, using the secure SSH protocol.

The general syntax to copy a file to a remote server is:

$ scp <source_file_path> <username>@<remote_host>:<destination_path>'

For instance, the following command will copy the file named foo from your local machine to your home directory on Sherlock:

$ scp foo <sunetid>@login.sherlock.stanford.edu:
Note the : character, that separates the hostname from the destination path. Here, the destination path is empty, which will instruct scp to copy the file in your home directory.

You can copy foo under a different name, or to another directory, with the following commands:

$ scp foo <sunetid>@login.sherlock.stanford.edu:bar
$ scp foo <sunetid>@login.sherlock.stanford.edu:~/subdir/baz

To copy back files from Sherlock to your local machine, you just need to reverse the order of the arguments:

$ scp <sunetid>@login.sherlock.stanford.edu:foo local_foo

And finally, scp also support recursive copying of directories, with the -r option:

$ scp -r dir/ <sunetid>@login.sherlock.stanford.edu:dir/
This will copy the dir/ directory and all of its contents in your home directory on Sherlock.

SFTP (Secure File Transfer Protocol)#

SFTP clients are interactive file transfer programs, similar to FTP, which perform all operations over an encrypted transport.

A variety of graphical SFTP clients are available for different OSes:

When setting up your connection to Sherlock in the above programs, use the following information:

Hostname: login.sherlock.stanford.edu
Port:     22
Username: SUNet ID
Password: SUNet ID password

OpenSSH also provides a command-line SFTP client, originally named sftp.

To log in to Sherlock:

$ sftp <sunetid>@login.sherlock.stanford.edu
Connected to login.sherlock.stanford.edu.
sftp>
For more information about using the command-line SFTP client, you can refer to this tutorial for more details and examples.

rsync#

If you have complex hierarchies of files to transfer, or if you need to synchronize a set of files and directories between your local machine and Sherlock, rsync will be the best tool for the job. It will efficiently transfer and synchronize files across systems, by checking the timestamp and size of files. Which means that it won't re-transfer files that have not changed since the last transfer, and will complete faster.

For instance, to transfer the whole ~/data/ folder tree from your local machine to your home directory on Sherlock, you can use the following command:

$ rsync -a ~/data/ <sunetid>@login.sherlock.stanford.edu:data/
Note the slash (/) at the end of the directories name, which is important to instruct rsync to synchronize the whole directories.

To get more information about the transfer rate and follow its progress, you can use additional options:

$ rsync -avP ~/data/ <sunetid>@login.sherlock.stanford.edu:data/
sending incremental file list
./
file1
      1,755,049 100%    2.01MB/s    0:00:00 (xfr#2, to-chk=226/240)
file2
      2,543,699 100%    2.48MB/s    0:00:00 (xfr#3, to-chk=225/240)
file3
     34,930,688  19%   72.62MB/s    0:00:08

[...]
For more information about using the rsync, you can refer to this tutorial for more details and examples.

SSHFS#

Sometimes, moving files in and out of the cluster, and maintaining two copies of each of the files you work on, both on your local machine and on Sherlock, may be painful. Fortunately, Sherlock offers the ability to mount any of its filesystems to your local machine, using a secure and encrypted connection.

With SSHFS, a FUSE-based filesystem implementation used to mount remote SSH-accessible filesystems, you can access your files on Sherlock as if they were locally stored on your own computer.

This comes particularly handy when you need to access those files from an application that is not available on Sherlock, but that you already use or can install on your local machine. Like a data processing program that you have licensed for your own computer but can't use on Sherlock, a specific text editor that only runs on MacOS, or any data-intensive 3D rendering software that wouldn't work comfortably enough over a forwarded X11 connection.

SSHFS is available for Linux , MacOS , and Windows .

SSHFS on macOS

SSHFS on macOS is known to try to automatically reconnect filesystem mounts after resuming from sleep or suspend, even without any valid credentials. As a result, it will generate a lot of failed connection attempts and likely make your IP address blacklisted on login nodes.

Make sure to unmount your SSHFS drives before putting your macOS system to sleep to avoid this situation.

SSHFS on Windows

Most of the open and free implementations of SSHFS on Windows don't support 2FA and will fail to mount Sherlock's filesystems. Until Microsoft implements a working FUSE driver in WSL, Windows users will have to rely on commercial solutions, or workarounds.

For instance, on a Linux machine with SSHFS installed, you could mount your Sherlock home directory with the following commands:

$ mkdir ~/sherlock_home
$ sshfs <sunetid>@login.sherlock.stanford.edu:./ ~/sherlock_home

And to unmount it:

$ fusermount -u ~/sherlock_home

For more information about using SSHFS on your local machine, you can refer to this tutorial for more details and examples.

Globus#

Globus improves SSH-based file transfer protocols by providing the following features:

  • automates large data transfers,
  • can resume failed transfers,
  • simplifies the implementation of high-performance transfers between computing centers.

Globus Online is a Software as a Service (SaaS) deployment of the Globus Toolkit which provides end-users with a browser interface to initiate data transfers between endpoints registered with the Globus Alliance. Globus Online allows registered users to "drag and drop" files from one endpoint to another. Endpoints are terminals for data; they can be laptops or supercomputers, and anything in between. The servers at Globus.org act as intermediaries-negotiating, monitoring and optimizing transfers through firewalls and across network address translation (NAT). Under certain circumstances with high performance hardware transfer rates exceeding 1 GB/s are possible. For more information about Globus, please see the Globus.org's documentation.

Authentication#

To use Globus Online, you will first need to authenticate at Globus.org. You can either sign up for a Globus.org account, or use your SUNet ID account for authentication to Globus Online (which will be required to authenticate to the Sherlock endpoint).

To use your SUNet ID, choose "Stanford University" from the drop down menu at the Login page and follow the instructions from there.

Transfer#

Endpoint name

The Globus endpoint name for Sherlock is SRCC Sherlock (aka srcc#sherlock)

You can use Globus Online to transfer data between your local workstation (e.g., your laptop or desktop) and Sherlock. In this workflow, you configure your local workstation as a Globus endpoint using Globus Connect.

  1. Log in to Globus.org
  2. Use the Manage Endpoints interface to "add Globus Connect Personal" as an endpoint (you'll need to install Globus Connect Personal on your local machine)
  3. Transfer Files, using your new workstation endpoint for one side of the transfer, and the Sherlock endpoint (SRCC Sherlock) on the other side.

You can also transfer data between two remote endpoints, by choosing another endpoint you have access to instead of your local machine.

CLI#

Globus also provides a command-line interface as an alternative to its web interface. This command-line interface is provided over an SSH connection to a Globus.org server. Please see the Globus CLI documentation for more details.

Data Transfer Node (DTN)#

A dedicated Data Transfer Node is available on Sherlock, to provide exclusive resources for large-scale data transfers.

The main benefit of using it is that transfer tasks can't be disrupted by other users interactive tasks or filesystem access and I/O-related workloads on the login nodes.

By using the Sherlock DTN, you'll make sure that your data flows will go through a computer whose sole purpose is to move data around.

It supports:

To transfer files via the DTN, simply use dtn.sherlock.stanford.edu as a remote server hostname. For instance:

$ scp foo <sunetid>@dtn.sherlock.stanford.edu:~/foo

No shell

The DTN doesn't provide any interactive shell, so connecting via SSH directly won't work. It will only accept scp, sftp, rsync of bbcp connections.

One important difference to keep in mind when transferring files through the Sherlock DTN is that the default destination path for files, unless specified, is the user $SCRATCH directory, not $HOME.

That means that the following command:

$ scp foo <sunetid>@dtn.sherlock.stanford.edu:
will create the foo file in $SCRATCH/foo, and not in $HOME/foo.

You can transfer file to your $HOME directory via the DTN by specifying the full path as the destination:

$ scp foo <sunetid>@dtn.sherlock.stanford.edu:$HOME/foo

Cloud storage#

If you need to backup some of your Sherlock files to cloud-based storage services, we also provide a set of utilities that can help.

Google Drive#

Google Drive storage for Stanford users

Google Drive is free for educational institutions. Meaning you can get free and unlimited storage on Google Drive using your @stanford.edu account. See the University IT Google Drive page for more details.

We provide the gdrive tool on Sherlock to interact with Google Drive. You'll just need to load the gdrive module to be able to use it to move your files from/to Google Drive:

$ module load system gdrive
$ gdrive help

For more details about how to use gdrive, please see the official documentation.

Other services#

If you need to access other cloud storage services, you can use rclone: it can be used to sync files and directories to and from Google Drive, Amazon S3, Dropbox, Google Cloud Storage, Amazon Drive, Microsoft OneDrive and many more.

$ ml load system rclone
$ rclone -h

For more details about how to use rclone, please see the official documentation.


  1. For more details, see the SSH clients page

  2. Fetch is a commercial program, and is available as part of the Essential Stanford Software bundle.