iosjsonparse-platform

Parse.com how to sync Data from Web in JSON

I have created some crawlers that gather data from some web sites with kimonolabs.com and parsehub.com

I want to create an iPhone application based on the data the crawlers get and synchronise the data that the crawler's provide me through their API.

So in case the crawlers find any new data from the website's I want to be synchronised in these data changes.

I have tried kimonolabs.com and parsehub.com to crawl some data from some websites. both crawlers are pretty much the same.

The JSON files could be at least 10mb so I think is better to try and synchronise all the data with some other service (like parse.com) and query the specific data to minimise the bandwidth for the iPhone application.

This is a data sample from the JSON API from kimonolabs: https://www.kimonolabs.com/api/5khb4j90?apikey=8OBDXxQPcoAcW9AWqHzAzh1J9rlWHwIM&kimbypage=0

I was thinking to use parse.com as a backend as well as to try and import this JSON data to parse.com which provides some hosting service as well.

After importing the data to parse.com, I need to analyse these data and serve some of these data (specific queried data) to an iPhone app that I will create based on the data from the crawlers.

How do I create a background job or cloud code in parse.com to import these data from a JSON API? I am new to parse.com

I tried this:

curl -X POST \
  -H "X-Parse-Application-Id: **APPID**" \
  -H "X-Parse-REST-API-Key: **RESTAPIKEY**" \
  -H "Content-Type: application/json" \
  -d 'https://www.kimonolabs.com/api/5khb4j90?apikey=8OBDXxQPcoAcW9AWqHzAzh1J9rlWHwIM&kimbypage=0' \
  https://api.parse.com/1/classes/CrawledData

but it's not working, I was thinking of creating a background job to with something similar like the code above.

Solution

  • In Parse Cloud Code..

    main.js

    Parse.Cloud.define("crawledData", function(request, response) {
  // this is the passed parameter (https://www.kimonolabs.com/api/...)
  var url = request.params.value;
  // do something with url
});

    Then you can send curl request like following

    curl -X POST \
  -H "X-Parse-Application-Id: **APPID**" \
  -H "X-Parse-REST-API-Key: **RESTAPIKEY**" \
  -H "Content-Type: application/json" \
  -d '{ "value": "https://www.kimonolabs.com/api/..." }' \
  https://api.parse.com/1/function/crawledData

    More reading on Cloud Code: https://parse.com/docs/cloud_code_guide

    More reading on Cloud Code Background jobs: https://parse.com/docs/cloud_code_guide#jobs