python google-app-engine foreign-keys bulkloader

In Google App Engine, how do I configure bulkloader.yaml to handle an optional foreign key on a model?

I have a model in App Engine that I am exporting in CSV format and then importing to my local dev environment using appcfg.py and bulkloader.yaml.

I can import and export most models, but I am having issues with models that have a foreign key which is not always present. I can use the lambda import below to always import the foreign key as None or use the create_foreign_key() transform to import the foreign key when every line in my csv file has the foreign key.

How do I configure bulkloader.py to import the foreign key when it is present and ignore it when it is not present?

- kind: MyModel
  connector: csv
  connector_options:
  property_map:
    - property: myOtherModel
      external_name: myOtherModel
      import_transform: "lambda x: x is None and None or None"
      #import_transform: transform.create_foreign_key('MyOtherModel', key_is_id=True)
      export_transform: transform.key_id_or_name_as_string

Just uncommenting out the second import_transform in place of the lambda transform will produce this error.

File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/bulkload/transform.py", line 127, in generate_foreign_key_lambda
    value = int(value)
ValueError: invalid literal for int() with base 10: ''

This error occurs when I run appcfg.py. All other imports without foreign keys or where foreign keys are always present are working properly.

appcfg.py upload_data --config_file=bulkloader.yaml --num_threads=1 --batch_size=50 --url=http://localhost:8080/remote_api --email=Chris --passin --kind=MyModel --filename=MyModel.csv

The myOtherModel column in the csv file sometimes contains the MyOtherModel.key().id() and sometimes not.

Eg.

myOtherModel
1234
4567

2345

5678

Solution

Default behivour when foreign key value does not exist, reference property takes value: None and code for this:

import_transform: transform.create_foreign_key('MyOtherModel')

As in your error message: value does not exist shows that you uses 'othercolumn','','anohtercolumn' instead of 'othercolumn',,'anohtercolumn'

So if the source '' is then to handle this:

from this link : http://eikke.com/python-ifelse-in-lambda/ as Thomas Thurman comments, lambda expression should look like:

import-transform: "lambda x: [x, None][x=='']"

I hope it works for you