I am trying to parse just the index page of every domain on the web. The only alternative that I can think of this that would produce somewhat acceptable results would be a general web crawl to identify every unique domain on the web.
Instead, I would like to be able to just iterate through a list of all domain names on the web. Does a public (or private but not incredibly expensive) database dump like this exist?
Related question that may answer this question: is it possible to host your own DNS server without querying any other dns servers?
I suppose that looping through every possible ipv4 combination is also an option but that would be less targeted than the set of all registered domains.
I think I may have found a solution: czds.icann.org/