hadoophiveapache-tezzstd

Unknown frame descriptor while reading zst file on hive?

I have a file a.csv.zst and i ve created textfile table.

When i select on hive i see first 100 rows without an error but when i select count(*) it gives an error unknown frame descriptor

I can decompress my file on linux with zstd -dc a.csv.zstd

How can i read it on hive?

It works on mr but not with tez

I didnt give an option to show compression level to hive table as i know the default value is 3 and i m not sure the file zstd compression level

Solution

  • My tez version is 0.10.2

    So it has hadoop-common 3.1 jar which has zstd decompressor java class and that jar has a bug and fixed after 3.2 version

    When i changed hadoop common har it works