I'm using the bulkloader to upload data into my App Engine data storage.
I can't seem to store a dictionary into a JsonProperty and I am getting the following error:
BadValueError: Unsupported type for property nearby_countries: <type 'dict'>
My model defines this property as a JsonProperty:
nearby_countries = ndb.JsonProperty()
The only workaround I found seems to store a json.dumps() of my value instead but I guess this basically storing the string representation of the dictionary rather than the dictionary itself.
My understanding of the JsonProperty is that it takes a python object as the value and that I should not be bothered about the JSON serialization which ndb will take care of. Am I correct?
Value is a Python object (such as a list or a dict or a string) that is serializable using Python's json module; the Datastore stores the JSON serialization as a blob.
After lots of try & error as well as googling around for similar posts, I manage to find the following post which lead me the below solution:
http://blog.thekensta.com/2012/06/google-app-engine-bulk-loader-and-ndb.html
In short, JsonProperties are stored as blobs and we need to pass the bulkloader the correct transform method to generate a blob from the json string. We can use transform.blobproperty_from_base64 (from the google.appengine.ext.bulkload.transform module)
So I convert my list or dict to a string JSON string representation which then gets converted to a blob so that the bulkloader can store it:
import_transform: "lambda x: transform.blobproperty_from_base64(base64.b64encode(bytes(json.dumps(x.strip(' ,').split(',')))))"
The same reasoning fixes the TextProperty saved as Strings (mentioned in my comment above). You need to use db.Text as the transform function:
import_transform: db.Text
And to save a repeated=True TextProperty, I actually had to transform it to a blob as well:
import_transform: "lambda x: transform.blobproperty_from_base64(base64.b64encode(bytes(json.dumps(x.strip(' ,').split(',')))))"
(in the exemple above I actually turn a coma separated string into a list of Text objects to be stored in a TextProperty(repeated=True)