I would like to know the total size of an open directory online. How can I do this?
An example use-case might be to find out how large a repository mirror would be.
How can I find the total size of open directories such as: http://old-releases.ubuntu.com/, or http://download.kiwix.org/, or http://apollo.sese.asu.edu/data/?
This is a fairly quick and simple way to find the size of any open directory online.
Install rclone and replace the URL with whatever you want.
Install Rclone (Binaries available here)
curl https://rclone.org/install.sh | sudo bash
Get Directory Size (Replace URL with any open directory, make sure not to remove :http:)
rclone size --http-url http://old-releases.ubuntu.com/ :http:
Using rclone + http with optional mount will do the trick.
This gives you the freedom to check the size with all sorts of methods.
rclone size http:
or
rclone mount http: directory/ then cd directory/ and du -sh or du -hd1 or ncdu (from here) or (NOT recommended) ls -shR
Be Kind
You might want to avoid hammering the server by adjusting the values and optionally adding/removing --fast-list in this command:
rclone size http: -v --tpslimit 5 --bwlimit 500K --checkers 5 --fast-list
Adjust up or down according to your needs and what you think the server can handle. For example, in just a couple minutes, I was able to use rclone size on a server that I thought would be fine with it, and got these results returned.
Examples
$ rclone size --http-url http://apollo.sese.asu.edu/data/ :http: --checkers 100
Total objects: 195669
Total size: 123.619 TBytes (135920738673216 Bytes)
$ rclone size --http-url http://download.kiwix.org :http: --checkers 100
Total objects: 24624
Total size: 5.711 TBytes (6279328743124 Bytes)