pythonhivesqlalchemyimpalablaze

How to load data into blaze from hive2


All,

I am attempting to load data into blaze from a hive2 thrift server. I would like to do some analysis similar to what is posted here. Here is my current process.

import blaze as bz
import sqlalchemy
import impala

conn = connect(host='myhost.url.com', port=10000, database='mydb', user='hive', auth_mechanism='PLAIN')
engine = sqlalchemy.create_engine('hive://', creator=conn) 
data = bz.data(engine)

I am able to make the connection and generate the engine, but when I run bz.data it fails with the error

 TypeError: 'HiveServer2Connection' object is not callable

Any help is appreciated.

Answer

from pyhive import import hive
import sqlalchemy
from impala.dbapi import import connect


def conn():                                               
    return connect(host='myhost.com', port=10000, database='database',        user='username', auth_mechanism='PLAIN')

engine = sqlalchemy.create_engine('hive://', creator=conn)


#Workaround
import blaze as bz


data = bz.data(engine)

Solution

  • from pyhive import import hive
    import sqlalchemy
    from impala.dbapi import import connect
    
    
    def conn():                                               
        return connect(host='myhost.com', port=10000, database='database', user='username', auth_mechanism='PLAIN')
    
    engine = sqlalchemy.create_engine('hive://', creator=conn)
    
    
    #Workaround
    import blaze as bz
    
    
    data = bz.data(engine)