rsync

On Linux filesystems we strongly recommend to use rsync to copy data. Here are some useful examples:

Example

Command

Note

Basic usage

rsync -Pauv $SDIR $SERV:/$DDIR/

  • Copies a source directory from the local server to the destination directory on a specified server.

  • Only newer files are copied.

  • Interrupted transmissions are continued

Copy to project directories

rsync -Pauv --no-group $SDIR $SERV:/$DDIR/
  • Everything beneath a project directory should belong to the project, hence chgrp -R $PROJECT $PROJ_DIR is executed automatically on several AWI systems.
  • A later/successive rsync into a project directory will reset (copy again) the file, if the group does not match. Use --no-group to prevent this. 

Set group/user and permissions

rsync -Pvrupog --chown=$USR:$GRP --chmod=D770,F660,Dg+s  $SDIR  $SERV:$/DDIR/

  • Sets user, group and sgid at the destination

  • please google yourself for more information on the specified options (wink)

Parallel rsync

cd $SDIR
ls | parallel -j8 rsync -Pauv {} $SERV:$DDIR/
ls | xargs -n1 -P8 -I% rsync -Pauv % $SERV:$DDIR/

  • Copy several (e.g. 8) directories at once.

  • Use with care, with a huge number you can easily jam the network (and you do surly (warning) not want to do that (wink))

On Windows you can install rsync via Cygwin or you can use https://en.wikipedia.org/wiki/Robocopy

sftp/filezilla

On WIndows/Linux you might also want to use filezilla. However, you should avoid (old fashined) ftp and should enforce sftp, by useing port 22 for security reasons.