Dedicated Data Transfer Node (DTN)
To improve the data transfer experience of our supercluster users a separate dedicated data transfer server is available.
lrc-xfer.lbl.gov mounts all the cluster file systems such that users can transfer data into/from any cluster filesystem. Also NERSC HPSS data transfer utilities like hsi and htar are configured to work on this server.
Data Transfer Examples
- On Linux: scp/rsync
# Transfer data from a local machine to Lawrencium
scp file-xxx $USER@lrc-xfer.lbl.gov:/global/home/users/$USER
scp -r dir-xxx $USER@lrc-xfer.lbl.gov:/global/scratch/$USER
# Transfer from Lawrencium to a local machine
scp $USER@lrc-xfer.lbl.gov:/global/scratch/$USER/file-xxx ~/Desktop
# Transfer from Lawrencium to Another Institute
ssh $USER@lrc-xfer.lbl.gov # DTN
scp -r $USER@lrc-xfer.lbl.gov:/file-on-lawrencium $USER@other-institute:/destination/path/$USER
rsync: data transfer tool and a backup tool
rsync -avpz file-at-local $USER@lrc-xfer.lbl.gov:/global/home/user/$USER - On Window
WinSCP: SFTP client and FTP client for Microsoft Windows
FileZella: multi-platform program via SFTPGlobus
Globus for data transfers is highly recommended. Globus is available as a free service for any user to sign up. Please follow the instructions for access setup.
Additional Globus Features (ONLY for Berkeley Lab Users)
All Berkeley Lab users can use Globus to access and transfer files in/out of their Berkeley Lab Google drive also. Details on how to access LBL Google drive via Globus please refer to this instructions (LBL Only)
The lab now features the following connectors:
- For AWS buckets, browse the available collections in the Globus UI using the left hand side menu item “Collections” and look for “LBNL AWS S3 Collection” (UUID 9c6d5242-306a-4997-a69f-e79345086d68)
- For Google Cloud Platform, browse for “LBNL Google Cloud Storage Collection” (UUID 54047297-0b17-4dd9-ba50-ba1dc2063468)
- For Google Drive, browse for “LBNL Gdrive Access” (UUID 37286b85-fa2d-41bd-8110-f3ed7df32d62)
To facilitate browsing, please be sure the “Used Recently” option is not selected.
If you are interested in AWS buckets, please refer to this documentation.
With Globus Plus you can use Globus Connect Personal between personal machines for transfers to any of the currently supported connectors and collections. To request membership in Globus Plus, log into https://globus.lbl.gov and navigate to settings using the left side menu bar. Select the Globus Plus tab and “LBNL IT Plus Sponsor” and follow the directions.
If you see a connector you would like us to support, please send email to hpcshelp@lbl.gov.
in the bottom right or navigate to