Confusing output with --info=progress2

[Swire42]

The problem

Unlike --info=progress, --info=progress2 does not insert newlines after each transferred file.

This is a problem because progress2 displays 2 different kinds of information:

• During file transfer it seems to display instantaneous transfer rate and ETA (+ other things)
• Between 2 file transfers it seems to display average transfer rate since the beginnning and time elapsed (+ other things).

Without newlines, these 2 overwrite each other, making it hard to read, and confusing people (see comments on https://unix.stackexchange.com/questions/215271/understanding-the-output-of-info-progress2-from-rsync).

A quick fix is to use something like --info=name,progress2, which causes newlines to be inserted, thus separating the 2 kinds of output.

Possible solutions

Newlines

Adding newlines to --info=progress2.
Is it really meaningful to have one line per file when asking for total progress though ?

Simpler output

Only display instantaneous transfer rate and ETA (+ other things), as it seems to be the most relevant information.
This would have the advantage of keeping the one line output, while also making it readable.

Other solutions

• Adding more output control by extending --out-format to such information.
• Always displaying these 4 metrics instead of alternating.
• ...

[danie-dejager]

I'm experiencing the same problem with progress2. The percentage climbs up to 99% and from when it starts to copy files it stays there. The time and duration gets overwritten. Here are the output (I've pressed enter to get each line). None of those lines are telling me how far along I am except that I can see I'm transferring files 6782114.
CMD:
/bin/rsync -ah --info=progress2 /local1/ /local2/

          1.08T  99%   76.11MB/s    0:00:21  xfr#6782114, ir-chk=1006/8804021)
          1.11T  99%   32.06MB/s    0:00:34  xfr#6933062, ir-chk=1002/8999888)
          1.12T  99%   47.35MB/s    0:00:08  xfr#6933158, ir-chk=1004/9000004)
          1.12T  99%   49.32MB/s    5:59:52 (xfr#6940259, ir-chk=1002/9009511)
          1.12T  99%   24.17MB/s    0:00:21  xfr#6940291, ir-chk=1002/9009549)
          1.16T  99%   45.66MB/s    0:00:54  xfr#7195344, ir-chk=1004/9340055)
          1.19T  99%   48.66MB/s    6:28:07 (xfr#7232130, ir-chk=1003/9388134)

VERSION:

rsync  version 3.2.7  protocol version 31
Copyright (C) 1996-2022 by Andrew Tridgell, Wayne Davison, and others.
Web site: https://rsync.samba.org/
Capabilities:
    64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
    socketpairs, symlinks, symtimes, hardlinks, hardlink-specials,
    hardlink-symlinks, IPv6, atimes, batchfiles, inplace, append, ACLs,
    xattrs, optional secluded-args, iconv, prealloc, stop-at, no crtimes
Optimizations:
    SIMD-roll, no asm-roll, openssl-crypto, no asm-MD5
Checksum list:
    xxh128 xxh3 xxh64 (xxhash) md5 md4 none
Compress list:
    zstd lz4 zlibx zlib none
Daemon auth list:
    md5 md4

[WayneD]

In response to the reason this issue was opened, yes the output time is inconsistent and I have sadly forgotten to fix this. I plan to make --info=progress2 switch to a consistent time-remaining format for all the output lines since it is overwriting the line. If you want this output to be multi-line (which is definitely not the point of progress2 -- that's what --progress is for), you could run rsync with --outbuf=L and pipe it into tr '\r' '\n'.

As for the other comment, if you want an accurate percent done value, specify --no-inc-recursive (--no-i-r) so that the whole file-list gets scanned before starting the transfers. The cited output it is telling you that there are only 1000 more known files to check, but because it is "ir-chk" (not "to-chk") it is also telling you that the number of remaining files isn't really known (since they're still being scanned), so the percent done-ness is based on the currently known amount of remaining file data vs the amount of finished file data.

[danie-dejager]

@WayneD Thanks for looking into this. What I like about progress2 is that I can see rsync is busy and that the length of the output is not growing too much. With progress there could be a stretch of time with no change in output while in general the output could become a bit too busy. (I think I'd only want to see errors in a new line so I can review any errors afterwards.)