pythonmediawikipywikibot

How to speed up Pywikibot?


I've built some report tools using Pywikibot. As things are growing it now takes up to 2 hours to finish the reports so I'm looking to speed things up. Main ideas:

Unfortunately I can't find much documentation about caching and db access. Only way seems to dive into the code, and well, there's limited information about database access in user-config.py. If there is any, where can I find good documentation about pywikibot caching and direct db access?

And, are there other ways to speed things up?


Solution

  • Use PreloadingGenerator so that pages are loaded in batches. Or MySQLPageGenerator if you use direct DB access.

    See examples here.