Dedicated Data Transfer Node (DTN)
To improve the data transfer experience of our supercluster users a separate dedicated data transfer server is available.
lrc-xfer.lbl.gov mounts all the cluster file systems such that users can transfer data into/from any cluster filesystem. Also NERSC HPSS data transfer utilities like hsi and htar are configured to work on this server.
Data Transfer Examples
- On Linux: scp/rsync
# Transfer data from a local machine to Lawrencium scp file-xxx $USER@lrc-xfer.lbl.gov:/global/home/users/$USER scp -r dir-xxx $USER@lrc-xfer.lbl.gov:/global/scratch/$USER # Transfer from Lawrencium to a local machine scp $USER@lrc-xfer.lbl.gov:/global/scratch/$USER/file-xxx ~/Desktop # Transfer from Lawrencium to Another Institute ssh $USER@lrc-xfer.lbl.gov # DTN scp -r $USER@lrc-xfer.lbl.gov:/file-on-lawrencium $USER@other-institute:/destination/path/$USER rsync: data transfer tool and a backup tool rsync -avpz file-at-local $USER@lrc-xfer.lbl.gov:/global/home/user/$USER
- On Window
Additional Globus Features (ONLY for Berkeley Lab Users)
All Berkeley Lab users can use Globus to access and transfer files in/out of their Berkeley Lab Google drive also. Details on how to access LBL Google drive via Globus please refer to this instructions (LBL Only)
The lab now features the following connectors:
- For AWS buckets, browse the available collections in the Globus UI using the left hand side menu item “Collections” and look for “LBNL AWS S3 Collection” (UUID 9c6d5242-306a-4997-a69f-e79345086d68)
- For Google Cloud Platform, browse for “LBNL Google Cloud Storage Collection” (UUID 54047297-0b17-4dd9-ba50-ba1dc2063468)
- For Google Drive, browse for “LBNL Gdrive Access” (UUID 37286b85-fa2d-41bd-8110-f3ed7df32d62)
To facilitate browsing, please be sure the “Used Recently” option is not selected.
If you are interested in AWS buckets, please refer to this documentation.
With Globus Plus you can use Globus Connect Personal between personal machines for transfers to any of the currently supported connectors and collections. To request membership in Globus Plus, log into https://globus.lbl.gov and navigate to settings using the left side menu bar. Select the Globus Plus tab and “LBNL IT Plus Sponsor” and follow the directions.
If you see a connector you would like us to support, please send email to email@example.com.