wget

delay wget progress bar display interval


How to delay the wget progress bar updating/refreshing interval?

By default it updates every milliseconds and each update will be captured as a single line to a STDOUT file when run in a job script that causes the job script STDOUT file large. I don't want to disable the progress bar completely as I still want to know the progress at different time.


Solution

  • wget knows the --progress option, that allows you to tweak the output settings a bit. An excerpt from the manpage:

       --progress=type
           Select the type of the progress indicator you wish to use.  Legal indicators are
           "dot" and "bar".
    
           [...]The "bar" indicator is used by default.  It draws an ASCII progress bar graphics
           (a.k.a "thermometer" display) indicating the status of retrieval.  If the output is
           not a TTY, the "dot" bar will be used by default.
    
           Use --progress=dot to switch to the "dot" display.  It traces the retrieval by
           printing dots on the screen, each dot representing a fixed amount of downloaded data.
    
           The progress type can also take one or more parameters.  The parameters vary based on
           the type selected.  Parameters to type are passed by appending them to the type
           sperated by a colon (:) like this: --progress=type:parameter1:parameter2.
    
           When using the dotted retrieval, you may set the style by specifying the type as
           dot:style.  Different styles assign different meaning to one dot.  With the "default"
           style each dot represents 1K, there are ten dots in a cluster and 50 dots in a line.
           The "binary" style has a more "computer"-like orientation---8K dots, 16-dots clusters
           and 48 dots per line (which makes for 384K lines).  The "mega" style is suitable for
           downloading large files---each dot represents 64K retrieved, there are eight dots in
           a cluster, and 48 dots on each line (so each line contains 3M).  If "mega" is not
           enough then you can use the "giga" style---each dot represents 1M retrieved, there
           are eight dots in a cluster, and 32 dots on each line (so each line contains 32M).
    
           With --progress=bar, there are currently two possible parameters, force and noscroll.
    
           When the output is not a TTY, the progress bar always falls back to "dot", even if
           --progress=bar was passed to Wget during invokation. This behaviour can be overridden
           and the "bar" output forced by using the "force" parameter as --progress=bar:force.
    
           By default, the bar style progress bar scroll the name of the file from left to right
           for the file being downloaded if the filename exceeds the maximum length allotted for
           its display.  In certain cases, such as with --progress=bar:force, one may not want
           the scrolling filename in the progress bar.  By passing the "noscroll" parameter,
           Wget can be forced to display as much of the filename as possible without scrolling
           through it.
    
           Note that you can set the default style using the "progress" command in .wgetrc.
           That setting may be overridden from the command line.  For example, to force the bar
           output without scrolling, use --progress=bar:force:noscroll
    

    I recommend to read the whole thing to understand how wgets progress meter works, but these are the important points in a nutshell:

    To summarize, in order to reduce the output when downloading a large file, you would invoke wget as follows:

    $> wget --progress=dot:mega <url>