scpi - scp improved with restartable downloads and advanced compression.
scpi.pl [options] [[user@]host:]src [...] [[user@]host:]dest
scpi (scp improved) is intended to be a drop-in replacement for the
scp utility that comes together with any Secure Shell (ssh) implementation.
scpi implements the much-needed restartable downloads that scp famously
doesn't support. As a couple of perks, scpi implements a few other
niceties such as optional use of advanced compression (gzip/bzip2)
and copying multiple files to one destination directory in one shot.
Options:
-h, -help brief help message
-1 passed to ssh
-1 passed to ssh
-4 passed to ssh
-6 passed to ssh
-B batch usage (not implemented)
-C passed to ssh
-c str passed to ssh
-F str passed to ssh
-i str passed to ssh
-l limit bandwidth used
-o str passed to ssh
-P int passed to ssh
-p preserve file times (not implemented)
-q don't display progress
-r recurse
-S str choose the ssh to use
-v passed to ssh
-R restart mode (resume broken downloads)
-n use no compression (default)
-b use bzip2 for compressing the data stream
(%, speed, and ETA will be inaccurate)
-g use gzip for compressing the data stream
(%, speed, and ETA will be inaccurate)
-d debug (extra output)
scpi is implemented in Perl, and requires Perl on the machine it's running,
as well as on the source and/or destination hosts. Also, scpi uses the ssh
executable on the client machine (but not scp).
The main target audience for scpi consists of people who enjoy scp's
security but are unhappy with scp's lack of a ``restart'' capability---if
the connection drops in the middle of a copy, scp forces redoing the
transfer all over again. scpi implements robust restarts. When invoked with
the -R flag, scpi assumes that the existing target file, if any, is a
valid fragment left from a previous scpi (or scp for that matter)
execution, and that the source file hasn't changed in the meantime.
It follows that scpi is best at transferring large files. To better support
that ability, scp supports gzip or bzip2 compression of the data
stream, which can be a big win in transmission speed at a low computational
cost, particularly over an agglomerated LAN. (You can also pass scpi the
-C flag, which scpi passes along to ssh instructing it to compress the
stream itself (but not the source and destination files). Generally, the
newer bzip2 algorithm is expected to be better than either gzip or
ssh's native compression.)
scpi.pl is available from http://erdani.org/code/scpi.pl . Save the file to a
local Unix (including Cygwin) directory and use it as you'd use scp.
You don't need to read this unless you plan on changing scpi yourself, or you enjoy cool hacks.
In essence, scpi starts ssh through a bidirectional pipe and implements its own little data transfer protocol on top of that pipe. The protocol simply prescribes that data transfers take place in ``chunks''. Each chunk consists of a number N (in decimal ASCII) followed by a newline, followed by N bytes of data. By convention, if N is zero, that means an entire transfer---in our case, a file of arbitrary size---has ended. Transmitting a large file in chunks has the advantage that the size of the file mustn't be known in advance, advantage exploited in scpi's implementation.
The need for such a protocol arises from the requirement of reusing one bidirectional stream (namely ssh's stdin and stdout) for multiple binary transfers. If only one blob of data were to be transferred, closing the stream would naturally signify the end of the data. However, a more sophisticated communication demands a reliable way to detect the size of a data block.
With this little protocol in place, scpi implements and uses internally a
very simple ``command line'' interface---a way to execute commands on a
remote system and either feed data
to their standard input or read their standard output. Simplicity and reuse of
good old utilities (such as tail and cat) make scpi robust and
easy to debug.
Written by Andrei Alexandrescu. Mosey to http://erdani.org/email to contact the author.
As of this time, the -p (preserve file time) and -B (batch usage) are ignored, the first because the author didn't need it yet, and the second because the author didn't have the time to look into what it really means.
scpi has not been tested with file names containing a newline, and it
almost sure won't work with them. Quotes and other special characters inside
file names are likely to cause trouble as well.
The implementation of bandwidth limitation hasn't been really thought through. When the option -l is in effect, there is some bandwith throttling, but perhaps the regularization algorithm could be vastly enhanced.
When using -b or -g (compression with bzip2 or gzip), the percent
transferred, speed, and estimated time of arrival (ETA) will likely be
inaccurate. This is because they reflect the amount of compressed, not ``real'',
data transferred. On the bright side, at the end of each file's transfer, the
percent complete will show the effective stream compression achieved on that
file.
The documentation for scp shows more details on the options. scpi emulates scp's options (and, just like scp, passes some of them to ssh).
Copyright (c) 2006 Andrei Alexandrescu.
This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.