I have an HBase table in which I have two column families, 'i:*' for info, and 'f:b' for file:blob. I am storing images in the value, some images are almost 12MB.
I can load/insert the files no problem in java, but as soon as I try to retrieve them via a scan for the f:b family value (the blobs), my scanner sits until it times out and every region server on my cluster dies in sequence (I have a 20 node cluster). The only way to stop this quasi virus that my scanner somehow inflicts upon my helpless nodes is to drop the table altogether (or so it seems).
I am using Cloudera EDH '0.98.6-cdh5.2.0'
unfortunately my client just times out, so no valuable exceptions there, and all I could get from my node logs are below
2014-10-27 21:47:36,106 WARN org.apache.hadoop.hbase.backup.HFileArchiver: Failed to archive class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://nameservice1/hbase/data/default/RASTER/92ceb2d86662ad6d959f4cc384229e0f/recovered.edits/0000000000000000029.temp
java.io.FileNotFoundException: File hdfs://nameservice1/hbase/data/default/RASTER/92ceb2d86662ad6d959f4cc384229e0f/recovered.edits/0000000000000000029.temp does not exist.
at org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:658)
at org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:104)
at org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:716)
at org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:712)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:712)
at org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath.getChildren(HFileArchiver.java:628)
at org.apache.hadoop.hbase.backup.HFileArchiver.resolveAndArchive(HFileArchiver.java:346)
at org.apache.hadoop.hbase.backup.HFileArchiver.resolveAndArchive(HFileArchiver.java:347)
at org.apache.hadoop.hbase.backup.HFileArchiver.resolveAndArchive(HFileArchiver.java:284)
at org.apache.hadoop.hbase.backup.HFileArchiver.archiveRegion(HFileArchiver.java:137)
at org.apache.hadoop.hbase.backup.HFileArchiver.archiveRegion(HFileArchiver.java:75)
at org.apache.hadoop.hbase.master.CatalogJanitor.cleanParent(CatalogJanitor.java:333)
at org.apache.hadoop.hbase.master.CatalogJanitor.scan(CatalogJanitor.java:254)
at org.apache.hadoop.hbase.master.CatalogJanitor.chore(CatalogJanitor.java:101)
at org.apache.hadoop.hbase.Chore.run(Chore.java:87)
at java.lang.Thread.run(Thread.java:745)
2014-10-27 21:47:36,129 WARN org.apache.hadoop.hbase.backup.HFileArchiver: Failed to complete archive of: [class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://nameservice1/hbase/data/default/RASTER/92ceb2d86662ad6d959f4cc384229e0f/recovered.edits/0000000000000000029.temp]. Those files are still in the original location, and they may slow down reads.
2014-10-27 21:47:36,129 WARN org.apache.hadoop.hbase.master.CatalogJanitor: Failed scan of catalog table
java.io.IOException: Received error when attempting to archive files ([class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://nameservice1/hbase/data/default/RASTER/92ceb2d86662ad6d959f4cc384229e0f/f, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://nameservice1/hbase/data/default/RASTER/92ceb2d86662ad6d959f4cc384229e0f/i, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://nameservice1/hbase/data/default/RASTER/92ceb2d86662ad6d959f4cc384229e0f/recovered.edits]), cannot delete region directory.
at org.apache.hadoop.hbase.backup.HFileArchiver.archiveRegion(HFileArchiver.java:148)
at org.apache.hadoop.hbase.backup.HFileArchiver.archiveRegion(HFileArchiver.java:75)
at org.apache.hadoop.hbase.master.CatalogJanitor.cleanParent(CatalogJanitor.java:333)
at org.apache.hadoop.hbase.master.CatalogJanitor.scan(CatalogJanitor.java:254)
at org.apache.hadoop.hbase.master.CatalogJanitor.chore(CatalogJanitor.java:101)
at org.apache.hadoop.hbase.Chore.run(Chore.java:87)
at java.lang.Thread.run(Thread.java:745)
2014-10-27 21:47:36,146 INFO org.apache.hadoop.hbase.master.SplitLogManager: Done splitting /hbase/splitWAL/WALs%2Finsight-staging-slave019.spadac.com%2C60020%2C1414446135179-splitting%2Finsight-staging-slave019.spadac.com%252C60020%252C1414446135179.1414446317771
here is my code for scanning the table
try {
if (hBaseConfig == null) {
hBaseConfig = HBaseConfiguration.create();
hBaseConfig.setInt("hbase.client.scanner.timeout.period", 1200000);
hBaseConfig.set("hbase.client.keyvalue.maxsize", "0");
hBaseConfig.set("hbase.master", PROPS.get().getProperty("hbase.master"));
hBaseConfig.set("hbase.zookeeper.quorum", PROPS.get().getProperty("zks"));
hBaseConfig.set("zks.port", "2181");
table = new HTable(hBaseConfig, "RASTER");
}
Scan scan = new Scan();
scan.addColumn("f".getBytes(), "b".getBytes());
scan.addColumn("i".getBytes(), "name".getBytes());
ResultScanner scanner = table.getScanner(scan);
for (Result rr = scanner.next(); rr != null; rr = scanner.next()) {
/*I NEVER EVEN GET HERE IF I SCAN FOR 'f:b'*/
CellScanner cs = rr.cellScanner();
String name = "";
byte[] fileBs = null;
while (cs.advance()) {
Cell current = cs.current();
byte[] cloneValue = CellUtil.cloneValue(current);
byte[] cloneFamily = CellUtil.cloneFamily(current);
byte[] qualBytes = CellUtil.cloneQualifier(current);
String fam = Bytes.toString(cloneFamily);
String qual = Bytes.toString(qualBytes);
if (fam.equals("i")) {
if (qual.equals("name")) {
name = Bytes.toString(cloneValue);
}
} else if (fam.equals("f") && qual.equals("b")) {
fileBs = cloneValue;
}
}
OutputStream bof = new FileOutputStream("c:\\temp\\" + name);
bof.write(fileBs);
break;
}
} catch (IOException ex) {
//removed
}
thanks Does anyone know why a scan for a large blob might annihilate my cluster? I am sure it's silly just can't figure it out.
Looks like this was the problem
hBaseConfig.set("hbase.client.keyvalue.maxsize", "0");
I changed it to "50" and it works now.