Is there any way to crawl dynamic web site without library?
I'm using 'axios' to crawl data from web sites(https://movie.daum.net/premovie/released)
I need data of li tags under ol tag, but li tags aren't shown in source.
I guess the site renders li tags dynamically using JavaScript or etc., but I don't know how to crawl them after all tags are loaded.
The point is, I wanna do it without any library related to crawling like puppeteer.
Please let me know how to do it if you know it.
Found the way to solve it!
[웹 크롤링] JavaScript로 된 html 불러오기
It is written in Korean, but was really helpful.
Solution
Found the way to solve it!
For my case, I found the XHR exists in network tab of dev tools. So I requested the 'get' axios method to XHR url, not the url of web page.
[웹 크롤링] JavaScript로 된 html 불러오기
I found it in here! It is written in Korean, but was really helpful.
- How do I add last modified in the footer of my web page?
- board game win situation - searching algorithm
- What is the difference between JavaScript promises and async await?
- Puppeteer "TimeoutError is not defined"
- Can a promise resolve statement await a return from the function awaiting the promise?
- How to implement C# access modifiers in javascript?
- Implementing show/hide popup box on mouse click
- Random color generator
- Algorithm for drawing line with thickness / width
- How do you get a list of the names of all files present in a directory in Node.js?
- Text with multiple underlines
- Can the Google Maps API javascript be cached?
- css line-height: number to px
- how to add currency pattern to input element while typing in javascript?
- Unable to append javascript with document.body.appendChild
- What difference between tag <template> and script type = 'text/template'?
- Debounce in ReactJS
- Failed to load resource: the server responded with a status of 406 (Not Acceptable) ==>promptengine-compressed.js
- Vanilla JavaScript: Is there a way to toggle multiple CSS-classes in one statement?
- I am getting an error from the javascript code: var context = document.getElementById("gameCanvas").getContext("2d");
- Wordpress - enqueue script only if specific class is on page
- Error "EBUSY: resource busy or locked, open errno: -4082" when build storybook 7 in next.js 13
- Changing search value via JavaScript
- JavaScript: fields or properties
- save to file from firefox extension
- angular 13 SSR, assign a value to a property in onInit, causes ngOnInit() to be called multiple times
- What is the regex to match this string?
- secretOrPrivateKey must be an asymmetric key when using RS256
- Check if a node.js module is available
- JavaScript - how to make it possible to inherit