I need to examine a list of websites to check if they support the HSTS policy or not.
I grabbed their response headers. However, I am confused now because it appears that HSTS policy subscription can be done through preloaded lists and not only headers (I might be wrong but I am not getting this point).
Does checking the websites response headers (i.e. look for the Strict-Transport-Security header) gives me a correct answer to the questions: does the website support the HSTS policy?
Because there is a preloaded list. I am not sure: can a website subscribe to the list without sending the HSTS header? i.e. can a website support the HSTS policy without sending the Strict-Transport-Security header?
I checked Chrome's page for HSTS preloaded list subscription here. It says:
If your site is committed to HTTPS and you want to preload HSTS, we suggest the following steps: [...] Add the Strict-Transport-Security header to all HTTPS responses and ramp up the max-age in stages, using the following header values: [...]
Can anyone clarify to me. First, the word suggest in Chrome's page is not clear. Is it a must? or optional? Second, if it is a must, then, if the website that want to subscribe to the list needs to send the HSTS header anyways, why does it subscribe to the list? Is this list just to provide a mean to protect the first connection (which can not be protected using the HSTS header method)? or is it a mean of double-check or something like that? Please, clarify to me.
The bottom line question: Is it enough for me to check the headers to say if a given website support the HSTS policy or not, without checking the website against the Chrome preloaded list?
If checking the website against the Chrome's HSTS preloaded list is required, can you please point to me how to automate this (I can not perform this manually as I have a list of websites not one or two). Also, how to check the website against the list of a specific date in the past (few months ago).
Does checking the websites response headers (i.e. look for the Strict-Transport-Security header) gives me a correct answer to the questions: does the website support the HSTS policy?
Yes.
Because there is a preloaded list. I am not sure: can a website subscribe to the list without sending the HSTS header? i.e. can a website support the HSTS policy without sending the Strict-Transport-Security header?
Technically they can. However then they are subject to removal from the preload list. Also not all browsers support preload (though the mainstream ones do) and they don't all use the same preload list. So, the header is a must and the list is an optional extra.
Can anyone clarify to me. First, the word suggest in Chrome's page is not clear. Is it a must? or optional?
In general the rules are a MUST and they will not be automatically accepted to the list if the rules are not followed, and they may be deleted from the preload list. However rules are made to be broken and you can request addition manually if you want. For example https://gov.uk (the UK government) is on the preload list but without the includesSubDomain attribute on the HSTS header. This is presumably because they have not converted all their subdomains to HTTPS yet but still want the protection of preload for this important, top level site. Not all of us have the sway of the UK government so for the rest of, it's best to follow all the rules and submit automatically.
Second, if it is a must, then, if the website that want to subscribe to the list needs to send the HSTS header anyways, why does it subscribe to the list? Is this list just to provide a mean to protect the first connection (which can not be protected using the HSTS header method)? or is it a mean of double-check or something like that? Please, clarify to me.
Correct. Preload is to protect the first load. Without this, the browser has not seen the HSTS header and so doesn't know the site supports HSTS.
To be honest I think preload is overkill for most sites and I really dislike the concept of hardcoding this in a place out of your control, and this can be a foot gun, as I discuss in this blog post. In general (but not always!) the first request is relatively safe (as you have no cookies) and as long as you use secure website techniques (Secure, HTTPOnly cookies, HSTS and redirect to HTTPS) the risks are relatively low. However if you are a global, well-known site (e.g. UK Government) then preload does offer the best protection.
The bottom line question: Is it enough for me to check the headers to say if a given website support the HSTS policy or not, without checking the website against the Chrome preloaded list?
Pretty much, as discussed above. Though you might have some edge cases where it was (is?) still preloaded but has stopped publishing the header. It depends on why exactly you NEED to know whether HSTS is used or not.
If checking the website against the Chrome's HSTS preloaded list is required, can you please point to me how to automate this (I can not perform this manually as I have a list of websites not one or two). Also, how to check the website against the list of a specific date in the past (few months ago).
You would need to check this against the HSTS source code, and look at the historical versions of this list. Not that other browsers may not use this same list. For manually checking one or two sites current status, tools like SSLLabs or Hardenize basically parse this list for you.