A number of methods allow transferring data in/out of Sherlock. For most cases, we recommend using SSH-based file transfer commands, such as
rsync. They will provide the best performance for data transfers from and to campus.
For large transfers, using DTNs is recommended
Most casual data transfers could be done through the login nodes, by pointing your transfer tool to
login.sherlock.stanford.edu. But because of resource limits on the login nodes, larger transfer may not work as expected.
For transferring large amounts of data, Sherlock features a specific Data Transfer Node, with dedicated bandwidth, as well as a managed Globus endpoint, that can be used for scheduled, unattended data transfers.
We also provide tools on Sherlock to transfer data to various Cloud providers, such as AWS, Google Drive, Dropbox, Box, etc.
Most of the commands detailed below require a terminal and an SSH client1 on your local machine to launch commands.
You'll need to start a terminal and type the given example commands at the prompt, omitting the initial
$ character (it just indicates a command prompt, and then should not be typed in).
Upon your very first connection to Sherlock, you will be greeted by a warning such as :
The authenticity of host 'login.sherlock.stanford.edu' can't be established. ECDSA key fingerprint is SHA256:eB0bODKdaCWtPgv0pYozsdC5ckfcBFVOxeMwrNKdkmg. Are you sure you want to continue connecting (yes/no)?
The same warning will be displayed if your try to connect to one of the Data Transfer Node (DTN):
The authenticity of host 'dtn.sherlock.stanford.edu' can't be established. ECDSA key fingerprint is SHA256:eB0bODKdaCWtPgv0pYozsdC5ckfcBFVOxeMwrNKdkmg. Are you sure you want to continue connecting (yes/no)?
This warning is normal: your SSH client warns you that it is the first time it sees that new computer. To make sure you are actually connecting to the right machine, you should compare the ECDSA key fingerprint shown in the message with one of the fingerprints below:
|Key type||Key Fingerprint|
If they match, you can proceed and type ‘yes’. Your SSH program will then store that key and will verify it for every subsequent SSH connection, to make sure that the server you're connecting to is indeed Sherlock.
Host keys warning#
If you've connected to Sherlock 1.0 before, there's a good chance the Sherlock 1.0 keys were stored by your local SSH client. In that case, when connecting to Sherlock 2.0 using the
sherlock.stanford.edu alias, you will be presented with the following message:
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: POSSIBLE DNS SPOOFING DETECTED! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ The RSA host key for sherlock.stanford.edu has changed, and the key for the corresponding IP address 184.108.40.206 is unknown. This could either mean that DNS SPOOFING is happening or the IP address for the host and its host key have changed at the same time. @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! Someone could be eavesdropping on you right now (man-in-the-middle attack)! It is also possible that a host key has just been changed. The fingerprint for the RSA key sent by the remote host is SHA256:T1q1Tbq8k5XBD5PIxvlCfTxNMi1ORWwKNRPeZPXUfJA. Please contact your system administrator.
You can just check that the SHA256 key listed in that warning message correctly matches the one listed in the table above, and if that's the case, you can safely remove the
sherlock.stanford.edu entry from your
~/.ssh/known_hosts file with the following command on your local machine:
$ ssh-keygen -R sherlock.stanford.edu
and then connect again. You'll see the first-connection prompt mentioned above, and your SSH client will store the new keys for future connections.
In all the examples below, you'll need to replace
<sunetid> by your actual SUNet ID. If you happen to use the same login name on your local machine, you can omit it.
SCP (Secure Copy)#
The easiest command to use to transfer files to/from Sherlock is
scp. It works like the
cpcommand, except it can work over the network to copy files from one computer to another, using the secure SSH protocol.
The general syntax to copy a file to a remote server is:
$ scp <source_file_path> <username>@<remote_host>:<destination_path>'
For instance, the following command will copy the file named
foofrom your local machine to your home directory on Sherlock:Note the
$ scp foo <sunetid>@login.sherlock.stanford.edu:
:character, that separates the hostname from the destination path. Here, the destination path is empty, which will instruct scp to copy the file in your home directory.
You can copy
foounder a different name, or to another directory, with the following commands:
$ scp foo <sunetid>@login.sherlock.stanford.edu:bar $ scp foo <sunetid>@login.sherlock.stanford.edu:~/subdir/baz
To copy back files from Sherlock to your local machine, you just need to reverse the order of the arguments:
$ scp <sunetid>@login.sherlock.stanford.edu:foo local_foo
scpalso support recursive copying of directories, with the
-roption:This will copy the
$ scp -r dir/ <sunetid>@login.sherlock.stanford.edu:dir/
dir/directory and all of its contents in your home directory on Sherlock.
SFTP (Secure File Transfer Protocol)#
SFTP clients are interactive file transfer programs, similar to FTP, which perform all operations over an encrypted transport.
A variety of graphical SFTP clients are available for different OSes:
When setting up your connection to Sherlock in the above programs, use the following information:
Hostname: login.sherlock.stanford.edu Port: 22 Username: SUNet ID Password: SUNet ID password
OpenSSH also provides a command-line SFTP client, originally named
To log in to Sherlock:For more information about using the command-line SFTP client, you can refer to this tutorial for more details and examples.
$ sftp <sunetid>@login.sherlock.stanford.edu Connected to login.sherlock.stanford.edu. sftp>
If you have complex hierarchies of files to transfer, or if you need to synchronize a set of files and directories between your local machine and Sherlock,
rsyncwill be the best tool for the job. It will efficiently transfer and synchronize files across systems, by checking the timestamp and size of files. Which means that it won't re-transfer files that have not changed since the last transfer, and will complete faster.
For instance, to transfer the whole
~/data/folder tree from your local machine to your home directory on Sherlock, you can use the following command:Note the slash (
$ rsync -a ~/data/ <sunetid>@login.sherlock.stanford.edu:data/
/) at the end of the directories name, which is important to instruct
rsyncto synchronize the whole directories.
To get more information about the transfer rate and follow its progress, you can use additional options:For more information about using the
$ rsync -avP ~/data/ <sunetid>@login.sherlock.stanford.edu:data/ sending incremental file list ./ file1 1,755,049 100% 2.01MB/s 0:00:00 (xfr#2, to-chk=226/240) file2 2,543,699 100% 2.48MB/s 0:00:00 (xfr#3, to-chk=225/240) file3 34,930,688 19% 72.62MB/s 0:00:08 [...]
rsync, you can refer to this tutorial for more details and examples.
Sometimes, moving files in and out of the cluster, and maintaining two copies of each of the files you work on, both on your local machine and on Sherlock, may be painful. Fortunately, Sherlock offers the ability to mount any of its filesystems to your local machine, using a secure and encrypted connection.
With SSHFS, a FUSE-based filesystem implementation used to mount remote SSH-accessible filesystems, you can access your files on Sherlock as if they were locally stored on your own computer.
This comes particularly handy when you need to access those files from an application that is not available on Sherlock, but that you already use or can install on your local machine. Like a data processing program that you have licensed for your own computer but can't use on Sherlock, a specific text editor that only runs on MacOS, or any data-intensive 3D rendering software that wouldn't work comfortably enough over a forwarded X11 connection.
SSHFS on macOS
SSHFS on macOS is known to try to automatically reconnect filesystem mounts after resuming from sleep or suspend, even without any valid credentials. As a result, it will generate a lot of failed connection attempts and likely make your IP address blacklisted on login nodes.
Make sure to unmount your SSHFS drives before putting your macOS system to sleep to avoid this situation.
For instance, on a Linux machine with SSHFS installed, you could mount your Sherlock home directory with the following commands:
$ mkdir ~/sherlock_home $ sshfs <sunetid>@login.sherlock.stanford.edu:./ ~/sherlock_home
And to unmount it:
$ umount ~/sherlock_home
For more information about using SSHFS on your local machine, you can refer to this tutorial for more details and examples.
Globus improves SSH-based file transfer protocols by providing the following features:
- automates large data transfers,
- handles transient errors, and can resume failed transfers,
- simplifies the implementation of high-performance transfers between computing centers.
Globus is a Software as a Service (SaaS) system that provides end-users with a browser interface to initiate data transfers between endpoints. Globus allows users to "drag and drop" files from one endpoint to another. Endpoints are terminals for data; they can be laptops or supercomputers, and anything in between. The Globus web service negotiates, monitors, and optimizes transfers through firewalls and across network address translation (NAT). Under certain circumstances, with high performance hardware transfer rates exceeding 1 GB/s are possible. For more information about Globus, please see the Globus documentation.
To use Globus, you will first need to authenticate at Globus.org. You can either sign up for a Globus account, or use your SUNet ID account for authentication to Globus (which will be required to authenticate to the Sherlock endpoint).
To use your SUNet ID, choose "Stanford University" from the drop down menu at the Login page and follow the instructions from there.
The Globus endpoint name for Sherlock is
You can use Globus to transfer data between your local workstation (e.g., your laptop or desktop) and Sherlock. In this workflow, you configure your local workstation as a Globus endpoint by installing the Globus Connect software.
- Log in to Globus.org
- Use the Manage Endpoints interface to "add Globus Connect Personal" as an endpoint (you'll need to install Globus Connect Personal on your local machine)
- Transfer Files, using your new workstation endpoint for one side of the transfer, and the Sherlock endpoint (
SRCC Sherlock) on the other side.
You can also transfer data between two remote endpoints, by choosing another endpoint you have access to instead of your local machine.
CLI and API#
Globus also provides a command-line interface (CLI) and application programming interface (API) as alternatives to its web interface.
For more information about the API, please see the Globus API documentation for more details.
$ module load system py-globus-cli $ globus login # follow instructions to get set up
Once you've authorized the application, you can use the
globus CLI to copy files in between endpoints and collections that you have access to. Endpoints and collections are identified by their unique UUID4 identifiers, which are viewable through the Globus web app. The CLI will step you through any additional authorizations required for you to access the endpoints or collections.
For example, to asynchronously copy files between Sherlock and Oak (if that you have already been allocated Oak storage):
$ GLOBUS_SHERLOCK_UUID="6881ae2e-db26-11e5-9772-22000b9da45e" $ GLOBUS_OAK_UUID="8b3a8b64-d4ab-4551-b37e-ca0092f769a7" $ globus transfer --recursive \ "$GLOBUS_SHERLOCK_UUID:$SCRATCH/my-interesting-project" \ "$GLOBUS_OAK_UUID:$OAK/my-interesting-project-copy"
Data Transfer Nodes (DTNs)#
The DTNs don't provide any interactive shell, so connecting via SSH directly won't work. It will only accept
A pool of dedicated Data Transfer Nodes is available on Sherlock, to provide exclusive resources for large-scale data transfers.
The main benefit of using it is that transfer tasks can't be disrupted by other users interactive tasks or filesystem access and I/O-related workloads on the login nodes.
By using the Sherlock DTNs, you'll make sure that your data flows will go through a computer whose sole purpose is to move data around.
To transfer files via the DTNs, simply use
dtn.sherlock.stanford.edu as a remote server host name. For instance:
$ scp foo <sunetid>@dtn.sherlock.stanford.edu:~/foo
$HOME on DTNs
One important difference to keep in mind when transferring files through the Sherlock DTNs is that the default destination path for files, unless specified, is the user
$SCRATCH directory, not
That means that the following command:
$ scp foo <sunetid>@dtn.sherlock.stanford.edu:
$SCRATCH/foo, and not in
You can transfer file to your
$HOME directory via the DTNs by specifying the full path as the destination:
$ scp foo <sunetid>@dtn.sherlock.stanford.edu:$HOME/foo
If you need to backup some of your Sherlock files to cloud-based storage services, we also provide a set of utilities that can help.
Google Drive storage for Stanford users
Google Drive is free for educational institutions. Meaning you can get free and unlimited storage on Google Drive using your @stanford.edu account. See the University IT Google Drive page for more details.
We provide the
rclone tool on Sherlock to interact with Google Drive. You'll just need to load the
rclone module to be able to use it to move your files from/to Google Drive:
$ module load system rclone $ rclone --help
This tutorial provides an example of tranferring files between Google Drive and Oak storage.
The Globus CLI (see above) can also be used to copy files from Sherlock to Stanford's Google Drive.
You can also access AWS storage from the Sherlock command line with the AWS Command Line Interface:
$ module load system aws-cli $ aws help
If you need to access other cloud storage services, you can use
rclone: it can be used to sync files and directories to and from Google Drive, Amazon S3, Box, Dropbox, Google Cloud Storage, Amazon Drive, Microsoft OneDrive and many more.
$ ml load system rclone $ rclone -h
For more details about how to use
rclone, please see the official documentation.