javadownloadniohttp-status-code-503

Only download a certain amount of files per second - "decrease" the download speed

It appears to me that my server only allows 60 files to be downloaded per second, but I have 63 of them - all tiny, YAML files. As a result, the last 3 files don't get downloaded and throw error 503. I am using Baeldung's NIO example:

public static void downloadWithJavaNIO(String fileURL, String localFilename) throws MalformedURLException {
        String credit = "github.com/eugenp/tutorials/blob/master/core-java-modules/core-java-networking-2/src/main/java/com/baeldung/download/FileDownload.java";

        URL url = new URL(fileURL);
        try (
            ReadableByteChannel readableByteChannel = Channels.newChannel(url.openStream());
            FileOutputStream fileOutputStream = new FileOutputStream(localFilename);
            FileChannel fileChannel = fileOutputStream.getChannel()
        ) {
            fileChannel.transferFrom(readableByteChannel, 0, Long.MAX_VALUE);
            fileOutputStream.close();
        } catch (IOException e) {
            e.printStackTrace();
        }
        
    }

I was thinking of saving currentTimeMillis somewhere and checking if a second had passed by the time the 61th file is pending. But are there any other good ideas?

Solution

  • FileChannel.transferFrom: An invocation of this method may or may not transfer all of the requested bytes.

    So I am not sure whether it works in all cases. Maybe with those small files.

    I would first try a correct (non-optimized) version:

    public static void downloadWithJavaNIO(String fileURL, String localFilename)
        throws MalformedURLException {
    URL url = new URL(fileURL);
    Path targetPath = Paths.get(localFilename);
    try (InputStream in = url.openStream()) {
        Files.copy(in, targetPath);
    } catch (IOException e) {
        System.getLogger(getClass().getName()).log(Level.ERROR, fileURL, e);
    }        
}