Calculating page document size for a URL in bytes

I wanted to create an function/API in node which will help me calculate the desktop and mobile page sizes for a given URL and also provide the compression/encoding strategy which was used by the server.

I found a way of doing this using axios so I'm sharing my solution here. Any further suggestions are appreciated.

We can use axios for this purpose as below

1. First we need to define User-Agent request header for desktop and mobile devices. Here I'm using a isMobile as a dummy flag which you can pass in your function:

   const headers = {
       'User-Agent': isMobile
           ? 'Mozilla/5.0 (Windows NT 10.0; Android 13; SM-G981B) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Mobile Safari/537.36'
           : 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36' ,
       'Accept-Encoding': 'gzip, deflate, br'
   }
   

2. Then we can use axios to get the page size as below:

   const response = await axios.get(url, {
       headers, 
       responseType: 'arraybuffer', 
       validateStatus: () => true,
       decompress: false,
   }); 
   const sizeInBytes = response.data.length; 
   const encoding = response.headers['content-encoding'] || 'none';
   

Note:

1. You can use the response header content-length to get the page size but there are many cases when this header is not provided by the server, hence I have used a response type of arraybuffer and calculated the length of the buffer.
2. if you don't use the decompress: false flag then axios/node will by default decompress the response and also you content-encoding header will not be present due to which you won't be able to identify the encoding type.