I've built some report tools using Pywikibot. As things are growing it now takes up to 2 hours to finish the reports so I'm looking to speed things up. Main ideas:
page.get(throttle=False)
handles thisUnfortunately I can't find much documentation about caching and db access. Only way seems to dive into the code, and well, there's limited information about database access in user-config.py
. If there is any, where can I find good documentation about pywikibot caching and direct db access?
And, are there other ways to speed things up?
Use PreloadingGenerator
so that pages are loaded in batches. Or MySQLPageGenerator
if you use direct DB access.
See examples here.