New to marketo so forgive me if there's an easy answer here.
For my use case, I need to pull all marketo leads from the database as well as all activities. From a daily quota perspective, leads aren't a huge issue but file size on activities is a problem. On first run, I'd need to retrieve all activities and leads from inception of Marketo db but one month of activities can be in excess of the 500mb limit. Is there a better/more efficient way to pull all the data I need or is the answer here just creating a process that pulls activities in 500mb increments and waits until the quota reset time to go again after exceeding 500mb?
Short answer: yes, just wait and go through as many days as you need.
Long answer: depends. You can contact Marketo about increasing the 500MB API export limit if needed—I believe this comes with an additional charge if needed on an ongoing basis, but I'm less clear on temporary increases to get historical data. Additionally, if you already have 500MB' worth of data in a month, the other options I'd recommend, such as specifically targeting activities and writing them on a one-off basis, probably aren't going to make sense.
You could also tweak what you're downloading in your bulk export—there's often repeated data that doesn't make sense to capture twice. Two examples I can think of immediately are around score changes (which produce both a unique score activity and a unique data value change activity) and lifecycle model changes (which also both produce a unique stage change activity and a data value change activity). Trimming these down may get you further along faster; in both cases, I'd recommend discarding the data value change.