pythonweb-scrapingurlxpathrequest

How to find the correct URL when you made some choices on the web page?


I'm very new to learn about web scraping. By using xpath selector i am trying to get the knowledge on that webpage : https://seffaflik.epias.com.tr/transparency/uretim/planlama/kgup.xhtml

But the point is, whenever you change the date or the powerplant name, URL does not change therefore when you fetch the response, you are getting always the same and wrong answer. Is there a way to find the correct URL or anything else related to HTML Markup etc ?


Solution

  • For a scraping operation like this, you'll need to do a bit more than just load the document and then grab the content. The document in-question relies on JavaScript to load new information from some other resource after the user has defined a particular set of parameters and updated the form.

    After loading the document, you'll need to define your search parameters. You can do this via JavaScript injection or via your browser's console. For example, if you were trying to define the value for the first date field, you could use

    document.querySelectorAll('#j_idt199 input')[1].value = "Some/New/Date";
    

    Repeat this process for the other fields you wish to define in your search, and then run the following code to programmatically execute your search:

    document.querySelector('#j_idt199 button').click();
    

    After that, you can either grab the information you want using plain JS query selectors, or you can implement a scraping library like artoo.js to help you interpret the data and export it.