Can't login with curl to GameAnalytics site

I use Game Analytics (GameAnalytics.com) in my app. For those who unfamiliar with GA it's a free mobile app analytics service, like flurry. I need some analysis which GameAnalytics will not do, that's why I download raw json event data, parse and store it in local DB for further processing. First I thought it's ok to download data manually each time, but then I realized I's annoying and it's better to automate this action.

I don't have deep understanding of webdev, php and js, though I use php for years mostly for utility programs and software backends.

So I used sniffer and was able to reproduce GET request with cookies on php:

$rollingCurl = new RollingCurl();
$rollingCurl->get("https://query-2.gameanalytics.com/v1/games/$game_id/export?start=$from&end=$to&_=0521345345333644",
  array(
      "Host: query-2.gameanalytics.com",
      "Connection: keep-alive",
      "Accept: application/json, text/plain, */*",
      "Origin: https://go.gameanalytics.com",
      "User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.122 YaBrowser/14.12.2125.10034 Safari/537.36",
      "Authorization: priv-eyJhbGciOiAiSFMyNTYiLCAidHlwIjogIkpXVCJ9.eyJ0bblahblah....",
      "Referer: https://go.gameanalytics.com/game/$game_id/settings/export-data",
      "Accept-Encoding: gzip,deflate",
      "Accept-Language: en;q=0.8"
  )
);

But this request uses cookies that will expire soon. So the only method is to copy cookies of existing session and paste them in php code, which is inconvenient.

I know that I must open curl session, login to gameanalytics.com, and then use this session to download data I need. The similar approach is successfully used in: Download flurry event log using curl

But I can't login for some reason. It look like some protection. Main login form generates target url with some number which I can't find in the page source.

go.gameanalytics.com/api/v1/public/login/basic?_=0600650333334639

Maybe there are some known schemes or workarounds? Or it is somehow connected with cookies page generates on first load?

When I use this number to login via curl, it fails:

$ch = curl_init('go.gameanalytics.com/api/v1/public/login/basic?_=0600650333334639');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.122 YaBrowser/14.12.2125.10034 Safari/537.36");
curl_setopt($ch, CURLOPT_COOKIEFILE, null);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, '{"email":"$EMAIL","password":"$PASSWORD","remember":false}');
$data = curl_exec($ch);
//returns: {"errors": [{"msg": null, "type": null, "id": "405"}], "results": []}

Just in case here's the log of the successful login query from browser (all sensitive data was changed of course):

POST https://go.gameanalytics.com/api/v1/public/login/basic?_=0665297234573059 

***Headers***
HTTP/1.1
Host: go.gameanalytics.com
Connection: keep-alive
Content-Length: 74
Accept: application/json, text/plain, */ *
Origin: https://go.gameanalytics.com
User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.122 YaBrowser/14.12.2125.10034 Safari/537.36
Content-Type: application/json;charset=UTF-8
Referer: https://go.gameanalytics.com/login
Accept-Encoding: gzip,deflate
Accept-Language: ru,en;q=0.8
Cookie: __utmt=1; AWSELB=F7D56DAB06335D820C6DA102697F76E63C69B5593D8E1CA61696696A520EC8B406E1ACF7E4CBD4284F123BC02162F0E8265430E673426B0EE263D10D29A3D5A; ga_user_id=ga_user_048304342233359914; ga_session_id=ga_session_0967241199979347; gaUserTokenOld={"token":"9.eyJsaW5rZWQiOiBudWxsLCAiaWQiOiAxNTU2OCwgImV4cCI6IDE0MjQzNDI2NTAsICJyZW1lbWJlciI6IGZhbHNlfQ.8ZYJ9yq2xETeAmY6rfiYAtxJybxDbGyaAE","exp":1424342650000}; __utma=153348830.970035277.1424344521.1424345621.1424355621.1; __utmb=153878340.5.10.1424451621; __utmc=156578830; __utmz=153854830.1424354621.1.1.utmcsr=yandex.ru|utmccn=(referral)|utmcmd=referral|utmcct=/; visitor_id20732=101885788

***Body***
{"email":"HERE_IS_MY_EMAIL","password":"HERE_IS_MY_PASSWORD","remember":false}

And response: 

***Headers***
HTTP/1.1 200 OK
Cache-Control: no-store, no-cache, private, must-revalidate, proxy-revalidate
Content-Type: application/json
Date: Thu, 19 Feb 2015 10:31:22 GMT
Server: nginx/1.6.2
Content-Length: 284
Connection: keep-alive

***Body***
{"errors": [], "results": [{"linked": null, "token": "9.eyJsaW5rZWQiOiBudWxsLCAiaWQiOiAxNTU2OCwgImV4cCI6IDE0MjQzNDI2NTAsICJyZW1lbWJlciI6IGZhbHNlfQ.8ZYJ9yq2xETeAmY6rfiYAtxJybxDbGyaAE", "id": 99568, "exp": 1424342781, "remember": false}]}

Solution

I work @GameAnalytics and designed/coded this system so I can help you out.

Please know:

we do not officially support manual usage of the web-tool api
your code can therefore break at any time if we change stuff

When automating using this approach do so responsibly.

do not download the same files over and over
stop the script in the future when you no longer need the data
keep the frequency of the script running to a minimum (like once each day)

Basically optimize your script to not stress our systems. We monitor the usage and reserve the right to block accounts that use a bad implementation.

Also you should remove the token in the post above. :)

Solution

Ok.. now that the disclaimer is over we can get to the fun stuff! You are basically on the right path.

get a token

Like you do above make a request to the public login route.

https://go.gameanalytics.com/api/v1/public/login/basic

You then retrieve a token in JSON data. This token will expire within 15 minutes or so (look at the exp timestamp value) and is also the value being placed in cookie by AngularJS later (when doing proper tool login).

use the token

Get the token from the data and use it to access all other web-tool routes by placing it in the "X-Authorization" header of a request. To test out the different routes I would recommend installing the useful "postman" Chrome extension.

getting the list of games

Use the token to get the data for your account.

https://go.gameanalytics.com/api/v1/user/data

This will return JSON data with studios, games etc. Locate the "studiosGames" field in the data and find the game you need. Inside the game dictionary you need to locate a token stored at "dataApiToken".

the downloads

The "dataApiToken" is also a token and it will expire. Use it to contact our metrics API for getting data for a specific game. This API uses the "Authorization" header for placing the token.

Call this url:

https://query-0.gameanalytics.com/v1/games/[game_id]/export?start=[start_timestamp]&end=[end_timestamp]

I recommend inspecting the tool using Chrome (network tab) to see what routes / parameters are used for your game.

From the request you should then get JSON data with urls to download. Use those to fetch the files. These urls will expire after some hours.

Let us know if this helps!

Please remember to not share your keys/token on websites (like this) as they grant other people access through our API's. And .. as mentioned - behave responsibly!