web-scrapinglimesurveydata-collectionparsehub

How to scrape changing survey data of an ajax web survey


I have a site where I want to scrape out the contents of the survey. The survey can not be extracted it is a limesurvey service with a PHP backend.

I tried using ParseHub but I am stuck. The survey uses an ajax next button which I can happily loop through and click as many times as I want but I don't know how to extract data per button clicks. Also the challenging factor is that the structure of data and questions are changing. Like one time you have a div with radio buttons, and one time you have a div of checkboxes. Can you guys help me sort this out? Let me know if you need the exported ParseHub project.

Thanks in advance!


Solution

  • All questions are defined in a twig file, so you can see the structure. FYI, there are a lot of them. You should be less concerned with the structure and more focused on the Ids. All of the option ids follow a format of "{some text}{survey id}X{question group id}X{question id}{subquestion title}". This would be easier to search for and parse information out of.

    There is an api too: https://api.limesurvey.org/classes/remotecontrol_handle.html#method_export_responses However it gives you all users not the current one you are on.

    There are options to create a plugin and this gives you access to events. From here you can push out messages, but I don't know what your end game is to know if this is what you are looking for.