Posted by & filed under KijiSchema.

When building an HBase application, you need to be aware of the intricacies and quirks of HBase. For example, your choice of names for column families, or columns themselves can have a drastic effect on the amount of disk space necessary to store your data. In this article, we’ll see how building HBase applications with KijiSchema can help you avoid inefficient disk utilization.

A Sparse Datastore

Many customer tables have columns for two lines of an address, which we’ll call info:address1 and info:address2. However, not all customers use two lines for their address. Due to the HBase architecture, when users only fill in info:address1, the info:address2 column does not take up any space on disk. This is in stark contrast to most relational databases, where a row typically has space pre-allocated for each column in the table, even if the column is null. To understand how HBase stores null values for free, we need to look closer at how an HBase cell is stored on disk:

hbase_cell

In order to store nulls for free, each column value is stored with information about the column family and column qualifier that it is associated with. However, this flexibility also results in a potentially insidious issue: long names for a column family or column qualifier can lead to severe data bloat.

Shorter Family and Qualifier Names

The simplest solution to avoid this data bloat is to use shorter column family and column qualifier names. A single character is the ideal choice, since that will minimize the number of extra bytes per row. However, in an HBase table, having a meaningless column family name or column qualifier name can make application logic extremely confusing. Instead, Kiji uses a metadata table to map from meaningful names in Kiji to meaningless, but space-efficient names in HBase.

To illustrate some of these concepts, I’ve created a Kiji table:

schema> 
     -> CREATE TABLE kiji_test 
     ->   ROW KEY FORMAT RAW 
     ->   WITH LOCALITY GROUP 'default' (
     ->     GROUP TYPE FAMILY really_long_column_family_names_are_ok_with_kiji (
     ->     even_longer_column_qualifiers_are_no_problem_with_kiji_schema "string")
     -> );

Once the table is created, we can go to an HBase shell, and scan the Kiji metadata table to see what kind of information is stored:

hbase(main):002:0> scan 'kiji.default.meta'
ROW                   COLUMN+CELL                                                                                                                           
 kiji_test            column=layout:layout, timestamp=1361861432634, value=,\x830\x9BG\xFB\x8C\xF0#\xD7\xD0ue\xCE\x8E\x80\x12kiji_test\x00\x00\x00\x00\x02\x
                      02\x0Edefault\x00\x01\x00\x00\xFE\xFF\xFF\xFF\x0F\xFE\xFF\xFF\xFF\x0F\x00\x02\x02`really_long_column_family_names_are_ok_with_kiji\x00
                      \x01\x00\x00\x02\x02zeven_longer_column_qualifiers_are_no_problem_with_kiji_schema\x00\x01\x00\x02\x00\x02\x10"string"\x00\x00\x00\x00
                      \x00\x00\x00\x00\x00\x10kiji-1.0\x02\x021\x00                                                                                         
 kiji_test            column=layout:layout_id, timestamp=1361861432634, value=1                                                                            
 kiji_test            column=layout:update, timestamp=1361861432634, value=,\x830\x9BG\xFB\x8C\xF0#\xD7\xD0ue\xCE\x8E\x80\x12kiji_test\x00\x00\x00\x00\x02\x
                      02\x0Edefault\x00\x01\x00\x00\xFE\xFF\xFF\xFF\x0F\xFE\xFF\xFF\xFF\x0F\x00\x02\x02`really_long_column_family_names_are_ok_with_kiji\x00
                      \x01\x00\x00\x02\x02zeven_longer_column_qualifiers_are_no_problem_with_kiji_schema\x00\x01\x00\x02\x00\x02\x10"string"\x00\x00\x00\x00
                      \x00\x00\x00\x00\x00\x10kiji-1.0\x02\x021\x00

The data here is mostly unintelligible, but we can see that the column family names and the column qualifier names are both stored in the metadata table. Kiji uses that table to map from human-readable column names to HBase column names. If we look at the HBase table underlying the Kiji table, we find that, as suggested before, the column families have single character names:

hbase(main):003:0> describe 'kiji.default.table.kiji_test'
DESCRIPTION                                                                                                          ENABLED                                                        
 {NAME => 'kiji.default.table.kiji_test', FAMILIES => [{NAME => 'B', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0' true                                                           
 , VERSIONS => '2147483647', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => '2147483647', BLOCKSIZE => '65536',                                                                  
 IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}

Now that we understand how Kiji is solving the problem, we can test Kiji’s efficacy for ourselves. Let’s create a native HBase table which a schema similar to the Kiji table:

hbase(main):003:0> create 'hbase_test', {NAME => ‘really_long_column_family_names_are_ok_with_kiji’}

Now that we’ve got both of our tables in place, we run data loader, available on GitHub, which loads both tables with an identical dataset. To verify, we can take a peek at the data in each table:

hbase(main):002:0> get 'hbase_test', "key1"
COLUMN                                         CELL                                                                                                                                 
 really_long_column_family_names_are_ok_with_k timestamp=1361862070816, value=1234                                                                                                   
 iji:even_longer_column_qualifiers_are_no_prob                                                                                                                                       
 lem_with_kiji_schema                                                                                                                                                                

hbase(main):003:0> get 'kiji.default.table.kiji_test', "key1"
COLUMN                                         CELL                                                                                                                                 
 B:B:B                                         timestamp=1361862063712, value=\x00\x081234

Note the difference in the length of the column names. To make sure that all the data has been written to disk, we can also force a flush of the MemStore, and a major compaction, for completion’s sake.

hbase(main):004:0> flush 'hbase_test'; major_compact 'hbase_test'

hbase(main):005:0> flush 'kiji.default.table.kiji_test'; major_compact 'kiji.default.table.kiji_test'

With all the data written to disk, we can see the difference in disk space used by each table, despite having identical column family names, column qualifier names, and datasets.

jubjubbird:kiji-bento-1.0.0-rc3 natty$ hadoop fs -du -h /user/natty/hbase/kiji.default.table.kiji_test
573     /user/natty/hbase/kiji.default.table.kiji_test/.tableinfo.0000000001
0       /user/natty/hbase/kiji.default.table.kiji_test/.tmp
185.6k  /user/natty/hbase/kiji.default.table.kiji_test/f398286d1cee321dd2b6665d2dc1c601

jubjubbird:kiji-bento-1.0.0-rc3 natty$ hadoop fs -du -h /user/natty/hbase/hbase_test
613     /user/natty/hbase/hbase_test/.tableinfo.0000000001
0       /user/natty/hbase/hbase_test/.tmp
690.1k  /user/natty/hbase/hbase_test/f82b68ce259f41fb138efa69fcc5d8f3

The native HBase table requires more than three times the disk space as the Kiji table does.

Conclusion

The ability to sparsely store data is a major strength of HBase, since it allows for arbitrary sets of columns in each row. However, with great power comes great responsibility. The underlying implementation of the sparse datastore forces HBase schema designers to be careful about how they define their column families, and what they call their column qualifiers. By using KijiSchema to manage HBase tables, developers can put that issue to rest, and use meaningful column definitions without worrying about additional disk overhead.

Jon Natkins (@nattyice) is a Field Engineer at WibiData. Formerly, he was a software engineer at Cloudera, and has contributed to a variety of projects in the Apache Hadoop ecosystem. He holds an Sc.B in Computer Science from Brown University.

Leave a Reply

  • (will not be published)