hivehcatalog

How to get row and field delimiter using HCatalog Java API


POM dependency:

    <dependency>
        <groupId>org.apache.hive.hcatalog</groupId>
        <artifactId>hive-webhcat-java-client</artifactId>
        <version>1.2.1</version>
    </dependency>

I am able to get columns, partition columns, input file format, etc.

Useful code:

    HiveConf hcatConf = new HiveConf();

    hcatConf.setVar(HiveConf.ConfVars.METASTOREURIS, connectionUri);
    hcatConf.set("hive.metastore.local", "false");
    hcatConf.setIntVar(HiveConf.ConfVars.METASTORETHRIFTCONNECTIONRETRIES, THRIFT_CONNECTION_RETRY);
    hcatConf.set(HiveConf.ConfVars.HIVE_SUPPORT_CONCURRENCY.varname, "true");
    hcatConf.set(HiveConf.ConfVars.SEMANTIC_ANALYZER_HOOK.varname, HCatSemanticAnalyzer.class.getName());
    hcatConf.set(HiveConf.ConfVars.PREEXECHOOKS.varname, "");
    hcatConf.set(HiveConf.ConfVars.POSTEXECHOOKS.varname, "");

    hcatConf.setTimeVar(HiveConf.ConfVars.METASTORE_CLIENT_SOCKET_TIMEOUT, TIME_OUT, TimeUnit.MILLISECONDS);

    HCatClient client = null;
    HCatTable hTable = null;

    try {
        client = HCatClient.create(hcatConf);
        hTable = client.getTable(databaseName, tableName);
        System.out.println(hTable.getInputFileFormat());
        System.out.println(hTable.getOutputFileFormat());
        System.out.println(hTable.getSerdeLib());

    } catch (HCatException hCatEx) {
        LOG.error("Not able to connect to hive. Caused By;", hCatEx);
    }

How to get row and field delimiter for Text tables?

As per Javadoc of getSerdeParams(),

public Map<String,String> getSerdeParams()
- Returns parameters such as field delimiter,etc.

but in my case I am getting only 1 entry in this map

{serialization.format=1}

Solution

  • If I create a table:

    create table tbl1 (c1 int)  stored as textfile
    

    And when I run show create table tbl1:

    CREATE TABLE `tbl1`(
      `c1` int)
    ROW FORMAT SERDE 
      'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' 
    STORED AS INPUTFORMAT 
      'org.apache.hadoop.mapred.TextInputFormat' 
    OUTPUTFORMAT 
      'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
    LOCATION
      'hdfs://localhost:8020/apps/hive/warehouse/dev.db/tbl1'
    TBLPROPERTIES (
      'transient_lastDdlTime'='1477067078')
    

    No default delimiters are shown.

    When I create a table with delimiters:

    create table tbl2 (c1 int)  ROW FORMAT DELIMITED FIELDS TERMINATED BY "\," LINES TERMINATED BY "\n" stored as textfile;
    

    And when I run show create table tbl2:

    CREATE TABLE `tbl2`(
      `c1` int)
    ROW FORMAT DELIMITED 
      FIELDS TERMINATED BY ',' 
      LINES TERMINATED BY '\n' 
    STORED AS INPUTFORMAT 
      'org.apache.hadoop.mapred.TextInputFormat' 
    OUTPUTFORMAT 
      'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
    LOCATION
      'hdfs://localhost:8020/apps/hive/warehouse/dev.db/tbl2'
    TBLPROPERTIES (
      'transient_lastDdlTime'='1477067160')
    

    In 2nd case I explicitly mentioned delimiters. So, getSerdeParams() returned desired values.