pythonhbaseapache-pigjythonhappybase

'No module named happybase' when running from PIG

I have a Python UDF which is connecting to HBase using Happybase. If I run the code from Python 2.7 it works perfectly.

However when I call the Python UDF from Pig 0.15.0 I am getting the following error:

ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1121: Python Error. Traceback (most recent call last): import happybase ImportError: No module named happybase

In my Pig script I am registering my Python script (pigtest.py) like this:

REGISTER 'pigtest.py' using jython as myfuncs;

I tried to set the Happybase path in my Python script as follows but that didn't make a difference:

import sys
sys.path.append('/usr/local/lib/python2.7/dist-packages/happybase')
import happybase

I also tried adding "/usr/local/lib/python2.7/dist-packages/happybase" to the JYTHON_PATH in the .bashrc file (I'm on Ubuntu) but same error comes up.

It seems to me like I need to set the Happybase path somewhere, but I can't figure out where.

Solution

  • I was able to solve this with the help of the Jython user mailing list. You need to specify the parent directory of the Happybase folder, not the path to the Happybase dir itself like I was doing, by doing one of the following:

    Append the location to the sys.path in the Python script: 

    import sys
sys.path.append('/usr/local/lib/python2.7/dist-packages')
import happybase

    OR

    Add the location as an environment variable to JYTHONPATH (not JYTHON_PATH!):

    export JYTHONPATH = $JYTHONPATH:/usr/local/lib/python2.7/dist-packages