javascriptajaxaxiosweb-crawler

Is there any way to crawl dynamic web site without library?

I'm using 'axios' to crawl data from web sites(https://movie.daum.net/premovie/released)

I need data of li tags under ol tag, but li tags aren't shown in source.

element tab in dev tool

source tab in dev tool

I guess the site renders li tags dynamically using JavaScript or etc., but I don't know how to crawl them after all tags are loaded.

The point is, I wanna do it without any library related to crawling like puppeteer.

Please let me know how to do it if you know it.

Found the way to solve it!

[웹 크롤링] JavaScript로 된 html 불러오기

It is written in Korean, but was really helpful.

Solution

  • Found the way to solve it!

    For my case, I found the XHR exists in network tab of dev tools. So I requested the 'get' axios method to XHR url, not the url of web page.

    XHR tab in dev tools

    [웹 크롤링] JavaScript로 된 html 불러오기

    I found it in here! It is written in Korean, but was really helpful.