Introduction

REST APIs allow client applications to interact with server applications over HTTP by issuing the four actions GET, POST, PUT and DELETE, which operate on server-side “resources.” In this post, we will introduce the basic KijiREST resources and their identifiers.

Every REST request is parametrized by a resource path that uniquely identifies a server-side resource. Other optional parameters relevant to the action are specified in the query and body sections of the request. The following call is an example row GET:

GET /v1/instances/dev_instance/tables/users/rows/c51ce410c124a10e0db5e4b97fc2af39[?...]

Entry-point

The KijiREST API version is placed at the root of the resource path. Mandatorily prefixing the resource path with the version in this manner results in graceful upgrades (see Apigee’s RESTful API Design and Lexical Scope’s How are REST APIs versioned? for case studies).

/v1/[...]

Every KijiREST service is associated with a single Kiji cluster and the cluster may be considered the “root” resource (after the REST API version, of course). The Kiji cluster’s version (distinct from the KijiREST API version) is a conceivable endpoint within the cluster:

/v1/version

Instances

A Kiji cluster contains Kiji instances. In order to avoid name clashes with other direct child resources of the Kiji cluster resource (e.g. version), all instances are grouped into a sub-collection named instances — this serves as a namespace for all instances on this cluster.

/v1/instances/

/v1/instances/<instance>

Tables

Every instance contains a subcollection named tables — this serves as a namespace for all the tables in this instance.

/v1/instances/<instance>/tables/

/v1/instances/<instance>/tables/<table>

Rows

Every table contains a subcollection named rows. Each <row> is identified by the ASCII encoding of its hexadecimal entity-id. Rows are the deepest identifiable resources.

/v1/instances/<instance>/tables/<table>/rows/

/v1/instances/<instance>/tables/<table>/rows/<row>

Instead of the hexadecimal entity-id, only the formatted entity-id for a row is available to provide advanced query features which accommodate resource identification — these concepts will be explained in greater detail in future posts.

Observation. The collections instances, tables, and rows are all “homogeneous” sets in that they each only contain one respective resource type.

The filesystem directory tree analogy

The above scheme induces a simple directory tree analogy where directories represent collections of resources.

For example, consider a cluster where there are two instances named dev_instance and prod_instance. The prod_instance contains two tables named customers and songs. This hierarchy is best described in a diagram.

Caveats

Though they serve the similar purpose of identifying Kiji resources, KijiURIs and KijiREST resource paths must to be distinguished. Compare the following KijiURI and the KijiREST resource path.

KijiURI: kiji://.env/prod/customers

REST resource path: /v1/instances/prod/tables/customers/rows/[...]

The KijiURI can identify the Kiji cluster, but the KijiREST resource path can’t. The KijiREST resource path can identify a row within a table, but the KijiURI can’t. Moreover, the KijiREST resource path has explicit boxing of instances, tables, and rows in order to leave room for additional cluster-level, instance-level, table-level endpoints in the future.

Finally

Reasonably designing the REST identifiers allow clients to intuitively conceive server-side resources. Stay tuned for upcoming articles on KijiREST usage patterns.

Resources

https://blog.apigee.com/detail/restful_api_design

https://restful-api-design.readthedocs.org/en/latest/resources.html

http://thereisnorightway.blogspot.com/2011/02/versioning-and-types-in-resthttp-api.html

We are pleased to announce an updated version of the BentoBox SDK for Kiji. This system includes several upgraded components and will make it easier to deliver new releases of Kiji software to you as well. See the previous announcement here.

Albacore v1.0.4 includes the following software:

• KijiSchema 1.0.3
• KijiSchema shell 1.0.0
• KijiMR 1.0.0-rc62
• KijiMR Library 1.0.0-rc61
• Kiji Hive Adapter 0.3.0 *UPDATED*
• KijiExpress 0.3.0 *UPDATED*
• KijiREST 0.1.0 *NEW*
• Example code: phonebook and music recommendation tutorials

This BentoBox is powered by Hadoop and HBase via CDH 4.1. It is built around the latest version of KijiSchema.

New: KijiREST

This version of the BentoBox is the first to contain KijiREST, a REST interface for interacting with KijiSchema.
KijiREST is in the rest/ directory of the BentoBox. Its README includes instructions on configuring and running a REST service..

Notable updates to Kiji Hive Adapter and KijiExpress

A new release of Kiji Hive Adapter and KijiExpress have been provided.

• There is now a KijiExpress Shell. A Scala shell preloaded for KijiExpress can now be run with the command “express shell --local” or “express shell --hdfs“. Once a pipe is fully specified from input to output, it can be run with “pipe.run”.
• The Kiji Hive Adapter now allows the ability to read EntityId in Hive. Thanks to Jeff Kolesky for the patch!
• Various additional bug fixes.

Conclusions

We’re excited about the momentum building behind the BentoBox. New components, improved software stability, and a smoother upgrade process will help enable more powerful big data applications to be built with the BentoBox SDK.

Ready to get started? Download the BentoBox today!

We are pleased to announce an updated version of the BentoBox SDK for Kiji. This system includes several upgraded components and will make it easier to deliver new releases of Kiji software to you as well. See the previous announcement here

Albacore v1.0.3 includes the following software:

• KijiSchema 1.0.3
• KijiSchema shell 1.0.0
• KijiMR 1.0.0-rc62
• KijiMR Library 1.0.0-rc61
• Kiji Hive adapter 0.2.0
• Kiji Express 0.2.0 *NEW*
• Example code: phonebook and music recommendation tutorials

This BentoBox is powered by Hadoop and HBase via CDH 4.1. It is built around the latest version of KijiSchema.

New: KijiExpress

This version of the BentoBox is the first to contain KijiExpress, a Scala domain-specific language (DSL) for analyzing and modeling data stored in tables managed by KijiSchema. KijiExpress can be used to author complex flows of MapReduce jobs using a concise and expressive API.

KijiExpress is in the express/ directory of the BentoBox. Its README includes instructions on running express jobs and scripts. A KijiExpress version of the music recommendation tutorial is also available in the examples/express-music directory.

Notable updates to KijiSchema, KijiMR, the KijiMR Library, and the Kiji Hive Adapter

A new release of KijiSchema, KijiMR, and the KijiMR library has been provided. This includes the following updates and changes:

• bin/kiji supports loading Hadoop distro-specific dependencies by inferring from "bin/hadoop version" or $KIJI_HADOOP_DISTRO_VER.
• Added the ability to retrieve entity id from a KijiTableKeyValueStore reader. Call KijiTableKeyValueStore.getTableForReader(reader) to get the underlying KijiTable object.
• Reorganize KijiMR project structure for multiple Hadoop distributions. KijiMR jars are now placed in $KIJI_MR_HOME/lib/distribution/hadoop2/. You can now optionally specify hadoop2 on the KijiMR artifact to make your distribution requirement explicit.
• Added a graph library to the KijiMR Library for developing item-category association mining models within Kiji
• Kiji Hive Adapter now properly decodes complex Avro types, Avro unions, and map type families.

Conclusions

We’re excited about the momentum building behind the BentoBox. New components, improved software stability, and a smoother upgrade process will help enable more powerful big data applications to be built with the BentoBox SDK.

Ready to get started? Download the BentoBox today!

Systems like Kiji can help you maintain backward compatibility with your own data, but there are still some challenges involved in using all the features of Apache Avro while maintaining backward compatibility. This blog post will describe a common problem associated with storage and retrieval of Avro data, how we get around this problem in Kiji’s own data structures, and how you can use this technique as well.

A Subtle Problem with SpecificRecords

Like many applications that depend on Avro, Kiji uses its “SpecificRecord” API to generate custom classes representing data structures based on Avro IDL files. A custom SpecificRecord class would contain Java member fields for each field declared in the IDL file, getters, setters, as well as the Avro schema associated with the record.

For example, the following Avro IDL defines a record that holds a point:

record Point {
  float x;
  float y;
  string label;
}

This will be compiled to a Java class named Point that can deserialize, manipulate, and serialize Point instances.

Suppose you’ve used a system like Kiji to save some Point instances that you can manipulate later, and that you had the following method in your code:

/** Create a new point with x and y fields doubled. */
public Point doubleXYDistance(Point input) {
  // Initialize with all fields of the input Point
  Point.Builder builder = Point.newBuilder(input); 
  builder.setX(builder.getX() * 2.0);
  builder.setY(builder.getY() * 2.0);
  return builder.build();  
}

The call to Point.newBuilder() is initialized with fields from the input Point, so fields such as label are set to whatever the input record had.

Then suppose that in the next version of your program, you change the Avro IDL like so:

record Point {
  float x;
  float y;
  float z = 0.0;
  string label;
}

When this is recompiled you’ll have a “z” field, which will be gracefully added to existing records as you read them (when resolving the writer schema with this new reader schema). No problem. The doubleXYDistance() function will do the correct thing with z values as well; initializing the output Point instance with a z-value of 0 if the input Point lacks a z value, and using the input Point’s z value otherwise.

The main compatibility challenge occurs when you write out new records (with x, y, and z) and then an older application (compiled with the “2-d” version of Point) wants to manipulate them.

The older application can gracefully read 3-d points as 2-d points, discarding the z value. But consider this function:

void modifyPoint() {
  Point in = loadPointCellFromKiji();
  Point out = doubleXYDistance(in);
  persistPointToKijiCell(out);
}

When this writes a new Point object back to Kiji, it will write out a 2-d point! While the best thing would be for the system to accommodate additional newer fields gracefully in this process, the next best thing is to at least detect that you are operating on data that is too new, and refuse to do so before you corrupt the persistent information store.

Protocol Versioning

Kiji has this problem with TableLayoutDesc objects that represent Kiji table layouts as an Avro data structure. The KijiSchema shell modifies existing layouts by reading the TableLayoutDesc from the Kiji meta table, creating a modified copy, and persisting it back. We use Avro’s SpecificRecord interface to capture this.

One option to deal with this would be to use Avro’s GenericRecord API; it would allow us to faithfully copy all the fields of the input record into the output. If the new schema is backward compatible with the old schema (as is the case in those two versions of the Point record), we could work with the writer schema interpretation of the data directly.

But by using GenericRecord, we’d lose type-safe functions like TableLayoutDesc.setName(String tableName) (or Point.setX(float x)); we’d have to use the GenericRecord.set(String fieldName, Object value) method, which is a much more error-prone API to operate.

Instead, Kiji uses something we call protocol versioning to get around this issue. Each record type that we intend to operate on has a field named version which is a string. The Kiji TableLayoutDesc Avro record (what your JSON table layout files interact with) includes a version field which today you should set to "layout-1.1".

In Kiji, a protocol version includes a protocol name and a version number in major.minor.revision format. The protocol name is a sanity check on what kind of record this version pertains to. For example, Kiji table layouts all contain a protocol name of “layout”; this prevents the most basic error of trying to parse a similar but unrelated JSON record object in a place where it sholdn’t be.

The version number is a standard version number that follows semantic versioning: the major version changes when an incompatible change is introduced; the minor version for a compatible new feature; and the revision for a bug fix.

When we parse a JSON-formatted Avro record from a file, or load a serialized object into a SpecificRecord, we use the ProtocolVersion class to parse the version field of the record. We can then use this to compare the minimum and maximum protocol versions tolerated by this version of Kiji itself to the record.

For example, the KijiSchema shell version 1.0.0 supports a maximum ProtocolVersion of layout-1.1.0 for table layouts. It would refuse to operate on a Kiji TableLayoutDesc with version = "layout-1.5.0". This TableLayoutDesc object might contain new fields that the SpecificRecord subclass compiled into KijiSchema Shell 1.0.0 would discard. The user would be directed to use a newer version of the shell to operate on the newer TableLayoutDesc record.

Data Interpretation Semantics

The notion of a protocol version goes beyond controlling the literal names and types of fields read and written by a given object. A protocol version might also govern the semantics of how an object is used.

For example, in Kiji, table layouts represented by TableLayoutDesc SpecificRecord objects are parsed by the KijiTableLayout class. This class performs a number of validity checks on the underlying TableLayoutDesc. As we want to guide users to stronger validity constraints, we need to make sure that newer versions of Kiji don’t refuse to operate with older persisted table layouts that fail to meet the new validity standards; a backwards-compatible mode must be employed to “grandfather in” any existing table layouts.

KijiTableLayout parses the ProtocolVersion from TableLayoutDesc.version and uses this to guide what semantics it imposes on table layouts. New table layouts that take advantage of the latest features might declare a higher-numbered protocol version that requires the new validity constraints. Old table layouts that use an older protocol version (like the current standard layout-1.1) would not be affected by these new rules. Of course, if you want to upgrade your table layout in a way that takes advantage of new features in the future, you’ll need to increment your protocol version — and possibly make other necessary adjustments to continue to conform to the current layout validation rules.

In the example above, you could imagine changing a protocol version in the Point example to denote that you now regard x and y values as being in meters instead of inches.

Protocol versions elsewhere in Kiji

We use the ProtocolVersion class in a number of places. For instance, if you write a JSON file that controls how bulk imports map fields of an input CSV file into columns of a Kiji table, this JSON file also contains the protocol version "import-1.0".

You can use this same JSON file to perform recurring imports from a data source indefinitely, while you periodically upgrade to new versions of KijiMR and the KijiMR library. While we might add new features that improve the bulk import process, an existing JSON import control file that declares a protocol version of import-1.0 will be parsed and interpreted consistently.

A Versioned Point

How do you prevent yourself from falling into a similar trap yourself? Let’s return to the Point example from earlier. If from the start you had declared:

record Point {
  float x;
  float y;
  string label;
  string version; //  must be point-1.0
}

Then your application could rule out a large class of errors if you write:

private static final ProtocolVersion MAX_VER = ProtocolVersion.parse("point-1.0");

void modifyPoint() {
  Point in = loadPointCellFromKiji();
  if (MAX_VER.compareTo(ProtocolVersion.parse(in.getVersion()) < 0) {
    throw new RuntimeException("Input point is too new! Please upgrade the point modifier.");
  }
  Point out = doubleXYDistance(in);
  persistPointToKijiCell(out);
}

The ProtocolVersion class implements Comparable<ProtocolVersion>, so you can compare them easily without needing to worry about string parsing or comparison yourself.

Then when you declare your 3-d point:

record Point {
  float x;
  float y;
  float z = 0.0;
  string label;
  string version; //  must be point-1.1
}

... you would also change MAX_VER to "point-1.1". The newer version of your software would work correctly with 3-d points, and the older version of your software would prevent itself from modifying them.

Note that in the Avro schema above, we don't declare the version field with a default value. We don't want Avro to automatically use the reader's default value here: we must use whatever version was persisted.

If you're retrofitting this behavior onto an older Avro data structure, you may need to gracefully handle a "null" default value for a new version field and treat it as the same as your "1.0.0" version.

Conclusions

Data compatibility is a difficult challenge to manage, and systems that make use of convenient and type-safe precompiled classes may be vulnerable to accidentally discarding or misinterpreting if they are not written carefully.

With Kiji, data compatibility is one of our primary concerns. We believe Kiji makes it easier to safely store large volumes of data in an evolvable fashion. But how applications manipulate and interpret that data can have a big impact on the final result. Avro's Generic and SpecificRecord interfaces offer different tradeoffs in the same flexibility/safety space, with GenericRecords offering you greater flexibility, while SpecificRecords offer a type-safe Java-friendly API.

Kiji uses SpecificRecord instances combined with this versioned protocol pattern to address this concern with its own data structures.

Curious to learn more about ProtocolVersion?

• Check out the javadoc and consider using it in your software.
• See its use in KijiTableImportDescriptor.
• Get started hacking with Kiji!

This release is primarily incremental improvements and bug fixes to the APIs of both modules. In the next few weeks, users can expect another version of BentoBox incorporating both artifacts along with updated tutorials. Users who want to use them right now can update the versions in their pom.xml files (ask on the kiji user mailing list for help).

Notable changes

• Row filter improvements have been made in both modules. RowFilters will now be correctly serialized into KijiMR jobs (Thanks to Jeff Kolesky for contributing to this).
• KijiTableKeyValueStore reader now uses EntityId keys instead of strings. This is an incompatible change from previous versions of the class in KijiMR.
• Bug fixes for KijiPager and KijiMR job set-up.
• The deprecated getEntityId() method has been removed from ProducerContext. Producers needing the entity id should retrieve it with KijiRowData.getEntityId() in their produce() methods instead.

As always, full release notes can be found in RELEASE_NOTES.txt at the top level of both modules.

We are pleased to announce that you can now download a new version of the BentoBox SDK for Kiji. This system includes several upgraded components and will make it easier to deliver new releases of Kiji software to you as well.

We are shifting how we version and release BentoBoxes: approximately each quarter, a new minor version (1.0 -> 1.1 -> 1.2) of the BentoBox will be released. Each such version has a name — in keeping with our overall theme, they’ll be named after sushi. This BentoBox release is called “Albacore.”

Albacore includes the following software:

• KijiSchema 1.0.1
• KijiSchema shell 1.0.0
• KijiMR 1.0.0-rc6
• KijiMR Library 1.0.0-rc6
• Kiji Hive adapter 0.1.1
• Example code: phonebook and music recommendation tutorials

This BentoBox is powered by Hadoop and HBase via CDH 4.1. It is built around the latest version of KijiSchema, which is effectively the same as the KijiSchema 1.0.0 release made last week.

BentoBox improvements

The BentoBox system has been revamped with a new bento upgrade command. When a new version of the BentoBox is available, just type bento upgrade and the new SDK will be downloaded and installed in place (don’t worry, we make a backup of your old BentoBox and data first).

BentoBox instances will periodically check for available updates and notify you when they’re available. You can then do an in-place upgrade with a single command.

The BentoBox now contains several components. To help you keep track of which version of each you’re running, the bento-version.txt file will provide you with a manifest.

Component upgrades

Going forward, each minor version of the BentoBox may include new minor versions of stable software we release (e.g., BentoBox 1.1 may contain KijiSchema 1.1 or 1.2, not 1.0).

Periodic revisions of the BentoBox (1.0.1, 1.0.2, etc) will include only revisions of stable components like KijiSchema (1.0.1, 1.0.2, 1.0.3, etc.). Experimental components like the Kiji Hive adapter, or “release candidate” components like KijiMR may be upgraded more aggressively.

New: A Hive Adapter for Kiji

This version of the BentoBox is the first to contain the new Kiji Hive adapter. This plugin for Hive allows you to run HiveQL queries against tables managed by KijiSchema.

The Kiji Hive adapter is in the hive-adapter/ directory of the BentoBox. Its README includes some example queries you can run against the table generated by the music recommendation tutorial.

Updates to KijiMR and KijiMR Library

A new release of KijiMR and the KijiMR library has been provided. This includes the following updates and changes:

• A new XML bulk importer is included (XMLBulkImporter).
• You can run bulk imports from the KijiSchema shell now. See the KijiMR bulk importers user guide for more information.
• The MapReduceJob class has been renamed to KijiMapReduceJob.
• Your implementations of KijiProducer should now use KijiTableContext.getEntityIdFactory() to create entity ids; ProducerContext.getEntityId() is now deprecated and will be removed soon.
• Some other minor improvements or changes to the API have been made; see the release notes for full details.

Conclusions

We’re excited about the momentum building behind the BentoBox. New components, improved software stability, and a smoother upgrade process will help enable more powerful big data applications to be built with the BentoBox SDK.

Ready to get started? Download the BentoBox today!

KijiSchema is the first such module to be subject to these requirements; over the coming months, we expect to go through a similar process of more volatile changes followed by a stabilization period for KijiMR. The result of that process will be KijiMR 1.0.0. The next version of KijiMR to be released is 1.0.0-rc6.

There are several important considerations for compatibility. This document will discuss:

• API version compatibility within KijiSchema and other post-1.0.0 modules
• Version compatibility between modules (e.g., KijiSchema and KijiMR)
• Our plan for new module release versioning
• The future of Kiji Bento Box releases

This document is somewhat long, but forms a contract of sorts, so you should read it. We want to be upfront about our expectations for everyone. This document is written both for the developers of Kiji, and for its users; by adhering to the standards and requirements stated here, we hope to minimize disruption in existing projects within and depending on Kiji while continuing to enable new features, and empower users to determine for themselves when they want to upgrade between versions of individual Kiji modules.

Read on to learn more.

API Compatibility Within a Module

Our goal is to release stable APIs while continuing to provide new features to users. What it means for an API to remain “stable” is set forth on our wiki, along with some guidelines for how to achieve this in Java code: https://github.com/kijiproject/wiki/wiki/Api-Compatibility

When releasing new versions of KijiSchema (or any future stable module), we will reason about how changes impact the version number as follows:

• An increase in the revision number (1.0.0 to 1.0.1) contains only bug fixes, or very safe, well-tested and noninvasive compatible improvements or new features.
• An increase in the minor version (1.0.x to 1.1.0) contains general bug fixes and compatible new features and improvements.
• An increase in the major version (1.x.y to 2.0.0) contains incompatible API changes and may require upgrades to lower-level aspects of the system like recorded metadata formats too.

What it means to be a “compatible” API change is defined below.

Each module like KijiSchema contains a large number of APIs that are intended for use by different audiences. The flexibility we have in enhancing KijiSchema is at odds with our requirements that some elements stay the same. At the same time, while we want to ensure that compatibility remains between versions, to do so too rigidly would rule out the gradual introduction of experimental features, which might need to change significantly (or be deprecated altogether) within a few versions. While through visibility modifiers like “public” and “private” Java enables us to enforce some of the boundaries of our API, there are other cases where this is not possible (e.g., a “private” class that needs to be used in two internal packages must be declared “public”). And Java does not allow us to express the stable-vs-experimental distinction directly at all.

We intend to broadly follow the contract set forth in semantic versioning (semver.org), but to do so, must define our APIs more specifically than the Java compiler would naturally enforce.

Our compromise is to use a set of Java annotations on all our classes. These are declared in the annotations module here: http://github.com/kijiproject/annotations. When using classes declared in Kiji modules, please refer to the Javadoc to check for the associated annotations:

For example, the javadoc for the interface org.kiji.schema.Kiji starts as follows:

org.kiji.schema

Interface Kiji

All Superinterfaces:
KijiTableFactory, ReferenceCountable<Kiji>
All Known Implementing Classes:
HBaseKiji

This interface uses all three of our annotation types: ApiAudience, ApiStability, and Inheritance.

One set of annotations describes the intended audience of a piece of code. The audiences are as follows:

• @ApiAudience.Public – This class is intended for use by everyone. It is held to our highest stability requirement.
• @ApiAudience.Framework – This class is intended for use only by other Kiji framework modules (like KijiMR). Framework-level classes may change in incompatible ways between minor versions (1.1.x to 1.2.0). Developers using framework-level classes should monitor dev@kiji.org and the issue tracker at jira.kiji.org for changes. End-users may find these APIs lower level than are expected for their consumption.
• @ApiAudience.Private – Only other classes in the same module (KijiSchema) are expected to access this implementation detail. No guarantees are made about stability.

Within our source code, several classes are declared in packages named “impl” (e.g., org.kiji.schema.impl). Any class in an “impl” package, or whose name ends in Impl (org.kiji.schema.impl.Foo, or org.kiji.schema.BarImpl) is an implementation detail; it is implicitly the same as ApiAudience.Private, though this may not be explicitly labeled.

Several classes are declared in packages named “org.kiji.*.framework” and are implicitly @ApiAudience.Framework-level classes, though they are typically labeled as well.

Each class also declares the stability level we believe we can provide. Over time, classes are expected to “graduate” to the most stable level. New experimental code is described as such as well. Our stability levels are described as follows:

• @ApiStability.Experimental – This is brand new and, even if its audience is intended to be public, it may change in incompatible ways between any versions (like 1.0.2 and 1.0.3). You should pay close attention to development mailing lists if you use any of these features, and provide feedback.
• @ApiStability.Evolving – APIs that are no longer experimental, and we believe will be stable “soon.” These APIs may evolve incompatibly between minor versions (1.1.x and 1.2.0). Classes marked ApiAudience.Public have a high burden of proof to justify an incompatible change in this state, but sometimes a change may be required.
• @ApiStability.Stable – The most mature, trusted, and heavily-used APIs have this annotation. If the class is marked ApiAudience.Public, an incompatible change to such a class would only come at the introduction of a new major version (2.0.0), or would be seen as a regression to fix. Framework-level classes may see changes between minor versions, but the burden to justify doing so is very high.
• @Deprecated – This Java annotation describes a class or method whose use is no longer preferred, and which may not be maintained over time. Classes and methods marked as @Deprecated during a release candidate may be removed in a later minor version. After 1.0.0, classes/methods marked as @ApiAudience.Public and subsequently marked @Deprecated will be removed only in 2.0.0. They may cease to interoperate with new features before this time.

Neither Java’s “interface” keyword nor abstract classes distinguish between interfaces that hide library functionality from its clients (APIs), and interfaces that clients are expected to implement and provide as “callbacks” into the library itself (SPIs). Depending on how an interface is intended to be used, what it means to evolve the interface in a stable fashion changes.

We declare a final set of annotations that describes how we expect clients to reason about inheritance of a class:

• @Inheritance.Sealed – We expect that the module itself provides all implementations of this interface (or extensions of this abstract class). For example, we do not expect users to implement org.kiji.schema.KijiTableWriter. If you implement this interface yourself, you risk an incompatible change.
• @Inheritance.Extensible – Users are permitted to extend this class or interface. For example, we expect that you will provide implementations of org.kiji.mapreduce.gather.KijiGatherer.

Sealed interfaces may have new methods added to them between versions. We expect that you will call methods of these interfaces, not implement them yourself.

Since you provide implementations of extensible interfaces, you may expect that we will not add a new abstract method definition to the interface–but we may deprecate a method we do not intend to call in the future.

Classes not otherwise marked should be implicitly considered @ApiAudience.Private and @Inheritance.Sealed. Please point out any such non-impl classes; we will mark them explicitly.

For many classes and methods, our Javadoc defines our API contracts more precisely. Areas where the Javadoc is ambiguous are subject to change, and we appreciate questions regarding their assumptions.

The bottom line:

• If you use a class declared @ApiAudience.Public and @ApiStability.Stable, your usage is guaranteed up to version 2.0.0.
• If you use a class declared @ApiAudience.Public and @ApiStability.Evolving (or @ApiAudience.Framework), there may be incompatible changes (though we endeavor not to).
• If it is marked @ApiStability.Experimental, all bets are off.
• The publicity and stability level of classes can increase over time. If you feel that a class advertised as framework-only is useful to end users, please discuss on dev@kiji.org. We can always make things “more public” if there is a justification for doing so, or create a safe public wrapper around the API in question.

Version Compatibility Between Modules

Different modules (e.g., KijiSchema and KijiMR) may rely on one another. This section describes how you may reason about which versions can interoperate with one another.

These modules may rely on @ApiAudience.Framework-level classes between one another to communicate. These classes may change in incompatible ways between minor versions (1.2 to 1.3).

Prior to 4/1/2013, all available modules had synchronized “rc” version numbers. Going forward, the version numbers of one module will have little or nothing to do with version numbers of another. KijiSchema may reach 1.3.0 much more quickly than KijiMR. The reverse may also be true.

KijiMR depends on KijiSchema. A given release of KijiMR will depend on some minor version of KijiSchema. It will interoperate correctly with all revisions in that minor version, but may not work correctly with the next minor version of KijiSchema. Cross-minor-version compatibility between modules is a goal of ours, but not necessarily a blocker to release.

For example:

Suppose KijiMR 1.2.0 depends on KijiSchema 1.3.0.

Subsequent revisions of KijiSchema (1.3.1, 1.3.2) will interoperate correctly. No guarantee is made about the next minor version (KijiSchema 1.4.0).

When the next minor version of a module is released, we will announce which versions of related modules it can interoperate with. For example, if KijiSchema 1.4.0 turned out to be compatible with KijiMR 1.2.0, we would announce that fact. Otherwise, expect KijiMR 1.3.0 to be released shortly. At that time, we would specify whether KijiMR 1.3.0 would interoperate with both KijiSchema 1.4.0 and 1.3.x, or only with 1.4.0.

New module release versioning

Throughout 2013, we expect to continue adding to the Kiji ecosystem. Additional components will provide new functionality. KijiSchema and KijiMR were based heavily on existing code from the WibiData codebase. When we released these components under the Kiji flag, we took the opportunity to refactor what we could, but wanted to reflect that the majority of the functionality was known, already implemented, and formed a relatively “complete” offering: thus the “1.0.0-rc” versioning. Additional rearchitecting was required to react to our new open source home, a process that we thank you for your patience in. These modules moved through a series of “release candidates” which reflected points in time during this refactoring.

New components under development will not be released with the “1.0.0-rc1″ designation. Such new components will be released with version numbers starting from 0.1.0 as designated by Semantic Versioning (semver.org). For such projects, stability will not be defined until version 1.0.0.

New features may introduce incompatible changes between 0.1.0 and 0.2.0. Where possible, we will endeavor to leave a “leapfrog” path open to you for migration.

For example:

KijiRocketship is the highest performance realest-real-time way to get to the moon and back.

• The first version is 0.1.0.
• An API (org.kiji.rocket.Engine.fireJets()) may be introduced in version 0.2.0.
• A preferred API () may be introduced in version 0.3.0. The original API (fireJets()) will be marked @Deprecated. The Javadoc will note the version number when deprecation happens, and what you should use instead.
• The original API will be removed at 0.4.0 or afterward.

This will allow clients of KijiRocketship to test changes with an existing release, rather than need to make all new changes immediately to evaluate the next release. We cannot guarantee that we can do this in all cases between 0.x and 0.(x+1), but will make a reasonable attempt to do so. Cases where incompatible changes are introduced will be noted in the release notes.

Future Kiji BentoBox Releases

The Kiji BentoBox is our SDK for developing Big Data Applications on top of Kiji. We have thus far created a new "rc" release for the bento box with each release of KijiSchema. In the future, bento box versions will be decoupled from an association with any particular Kiji module.

Individual modules will be released when it makes sense to do so for that given module, on a rolling basis. The BentoBox will see a new minor release (1.1, 1.2, etc) approximately every quarter. Along with a minor version number, we will also apply an alphabetical name to each release; like the versions of Android, Ubuntu, and other projects, the names will increase alphabetically (a, b, c...) each time. In keeping with our sushi theme, we'll name these after fish. Kiji BentoBox 1.0.0 "albacore" will be released shortly.

Each new minor version of the BentoBox will include a complete set of compatible Kiji modules; the most recent stable versions of each module that are compatible with one another will be included.

Revisions of the BentoBox (1.1.1, 1.1.2, etc) may be produced more frequently. These revisions will correct bugs in the Bento packaging or scripts itself. Stable modules (e.g., KijiSchema) will be upgraded only very conservatively (new revisions only) in such releases of the BentoBox. Unstable modules (version numbers like 1.0.0-rcX, or 0.x.y) will be upgraded more aggressively in such point releases; if you are developing against unstable modules, it is important that you keep up to date with your dependencies.

Conclusions

Thank you for reading this far. This "contract" clarifies how we intend to balance the desire for new features, experimentation, and agile open source development with the requirement for stability and predictability in the interfaces you depend on. In this and all Kiji development matters, we welcome your participation in this process and our discussions. We hope to continue to serve you in your Big Data needs and thank you for making Kiji a part of your system. Please direct any questions or oversights to user@kiji.org. It's likely that others would benefit from hearing the answers to any questions you have. And look forward to Bento Albacore soon!

Ready to try out Kiji now? Download the BentoBox today!

You can use this in your Maven projects, or download it from http://www.kiji.org/getstarted/.

A BentoBox containing KijiSchema 1.0.0 (and the next release of KijiMR and some other goodies) will be available within a couple weeks.

KijiSchema 1.0.0 represents the first stable API for this component; subsequent versions of KijiSchema will contain new features or improvements but compatibility with existing releases will be preserved. A longer message about this subject and the precise specifications of “compatible” and “stable” will be released tomorrow.

KijiSchema 1.0.0 includes a few important new features and changes since rc5. The degree of incompatible changes should be very modest.

• The close() method (deprecated in rc5) has been removed from KijiTable. You must now use KijiTable.release(). This is the biggest incompatible change in this release that most users will see.
• A new KijiBufferedWriter class allows you to buffer and transmit a number of writes to a Kiji table in batch for improved performance.
• Documentation (especially Javadoc) has been improved in a number of cases. We have attempted to specify more precisely how most of our core classes work.
• Stability annotations (Experimental, Evolving, or Stable) have been added to all classes.
• As a bug fix, get() and scan() operations on empty tables with no column families now return empty responses/scanners, rather than null.
• A few other bug fixes and tune-ups throughout.
• Note that many classes marked @Deprecated in the KijiSchema release candidates have been removed, but others persist. These classes may be removed soon–in particular, the org.kiji.schema.mapreduce package. Please use KijiMR for your MapReduce needs. (If you upgrade to KijiSchema 1.0.0, your code should compile without any deprecation warnings.)

All users are encouraged to promptly upgrade to KijiSchema 1.0.0. Usage of any “release candidate” editions of KijiSchema is heavily discouraged, as they are not compatible with the ongoing world.

While we have a variety of canned bulk importers described in the bulk importer userguide for parsing and importing various common file formats, there are other important job configuration parameters for how the imports are executed.

Option 1: Using the KijiSchema PUT API

In direct writing mode, KijiMR will create a MapReduce job that reads over the input and generates the desired put(s) associated with each entry in the data. These puts are directly applied to a Kiji table as the job progresses.

Benefits:

Drawbacks:

Option 2: Bulk Loading using HFiles

For bulk loading, Kiji can create a MapReduce job that reads over the input and generates HFiles containing the rows that can be loaded directly into HBase using their bulk load functionality. Once the job completes, the HFiles can be loaded into the Kiji table to allow them to be accessed.

Benefits:

Drawbacks:

My Recommendation

For a new cluster, the recommended practice is to bulk load via HFiles. This allows for the initial backlog of data to be imported quickly, while avoiding the compaction issues above(since there’s no data in the cluster). If this bulk import job is for an existing Kiji instance, using the direct puts will allow the existing cluster to continue to respond to requests while still accepting the new data.

The Test Drive

As part of my testing of this new functionality, I tried bulk importing some of the sample data from the Kaggle Blue Book for Bulldozers challenge. This data came in a CSV file with a header that contained 401,126 lines including the header. The header line contains the names of the 53 fields. The rest of this example assumes that you have a bento box and a Kiji instance installed.

Specifying the layout to create a Kiji table with this many columns would have required a rather long-winded DDL. Since we need at least one row for each column family:qualifier, this would involve a lot of tedious copy and pasting. To accelerate this process, I wrote a little script: generate-ddl.sh that takes in said CSV file, parses the header, and auto-generates a default DDL assuming that every field is a string. Then the user could simply modify this generated DDL and produce the layout that they are looking for. Once we are happy with this, we can create the table using the kiji-schema-shell:

./generate-ddl.sh Train.csv > Train.ddl
kiji-schema-shell --file=Train.ddl

The CSVBulkImporter requires an import descriptor that defines the mapping from the source fields in the CSV to the destination Kiji columns. Being that this is also dependent on the fields, I’ve written another little script generate-import-descriptor.sh that takes in said CSV file, parses the header, and autogenerates a default import descriptor JSON file.

./generate-import-descriptor.sh Train.csv > Train.json
hadoop fs -copyFromLocal Train.json /

Finally we need to trim the header from the data file that we wish to bulk-import, and copy it over to HDFS so that the MapReduce job can get at it.

tail -n +2 Train.csv > Train-no-header.csv
hadoop fs -copyFromLocal Train-no-header.csv /

With Option 1: Using the KijiSchema PUT API

We can use Kiji to create a bulk importer job whose output is a kiji table(note the –output parameter).

kiji bulk-import \
  -Dkiji.import.text.column.header_row=`head -1 Train.csv` \
  -Dkiji.import.text.input.descriptor.path=/Train.json \
  --importer=org.kiji.mapreduce.lib.bulkimport.CSVBulkImporter \
  --output="format=kiji table=kiji://.env/default/train nsplits=1" \
  --input="format=text file=/Train-no-header.csv"

This took 78 minutes on a bento cluster running on my local machine, and consumed 3.2g disk space.

With Option 2: Bulk loading via HFiles

Alternately, we can use Kiji to create a bulk importer whose output is a kiji table. Note that the main difference here is that the –output parameter has changed to using the hfile format and that we specify the destination HFile. Note: we still need a table to know what layout the HFile should use.

kiji bulk-import \
  -Dkiji.import.text.column.header_row=`head -1 data/Train.csv` \
  -Dkiji.import.text.input.descriptor.path=/Train.json \
  --importer=org.kiji.mapreduce.lib.bulkimport.CSVBulkImporter \
  --output="format=hfile table=kiji://.env/default/train nsplits=1 file=hdfs://localhost:8020/train.bulkload" \
  --input="format=text file=/Train-no-header.csv"

Finally once these files get created, they can be bulk loaded with the bulk-load tool:

kiji bulk-load \
  --hfile=hdfs://localhost:8020/train.bulkload/part-r-00000.hfile \
  --table=kiji://.env/default/train

This took 2 minutes on a bento cluster running on my local machine, and consumed 483.4m disk space. This is vastly (50x) faster than the individual PUTs on my little laptop.

Now all of this data has been loaded into Kiji to do whatever you might like for post processing!

Performance Stats

Above are the results for bulk importing of data into uncompressed Kiji tables. As you can see there’s nearly a 40x difference between bulk import time, and a 6x difference in the disk utilization. Using HFiles to bulk import data is more performant than doing direct writes, but there are potential operational difficulties on a running cluster with the possibly of an (expensive) compaction looming when the data files are loaded.

Addendum: Timestamps

In Common Pitfallfalls of Timestamps in HBase, we discuss many of the potential pain points of dealing with timestamps within HBase. While in general we don’t recommend manually setting timestamps, in the case of the initial bulk importing of data, it’s often beneficial to backfill the timestamps based on the data to be imported. This way initial imported data can have the same behavior as newly added data in your application.

This version of KijiSchema continues the maturation process of the KijiSchema framework’s API. The improvements from the previous version are primarily internal and incremental. Wherever possible, previous functionality has been kept and marked as deprecated rather than removed. Users should see fewer dramatic user-facing alterations than the rc4 release. Further, this will be the final release candidate before the stable 1.0.0 release. Between rc5 and 1.0.0 changes should be limited to critical changes, improving functionality of the DDL shell and CLI, and tidying up the code base.

This release is also synchronized with our first official release of the KijiMR module, which lets users write MapReduce jobs to bulk load and analyze data stored in KijiSchema tables. See the associated blogpost and userguide for more information about KijiMR. KijiMR is a younger module and can be expected to undergo a few iterations before its own 1.0.0 release.

Major new features

We added several new features to KijiSchema in this version:

• Support for formatted EntityIds (i.e. composite keys of multiple components) in all command line tools has had its coverage improved. Formatted EntityIds were added programmatically in rc4. This change makes such EntityIds usable for all KijiSchema command line tools.
• Support for formatted EntityIds in the DDL shell has also been added.
• kiji get and kiji scan have been added, more logically handling functionality which previously existed as part of the kiji ls command.
• kiji ls has changed including gaining the ability to list columns in a table, while the ability to extract data stored in rows has been moved into the get and scan tools mentioned above.
• A CLI tool for manipulating the KijiSchema SystemTable has been added. The SystemTable stores instance-wide settings and is now included in metadata backups.
• CLI tools now accept relative KijiURIs if the user is using kiji://.env as their zookeeper quorum. Instance, table, and column names must still be specified. For example, default/table_foo is shorthand for kiji://.env/default/table_foo.
• AtomicKijiPutter, providing the ability to apply multiple puts atomically to a single EntityId has been added. Use table.getWriterFactory().openAtomicWriter() to obtain one.

We made a few API changes that application developers should be aware of:

• New backup format — The API remains the same, but Metadata backup and its associated commandline tool now also saves information from the SystemTable (containing instance-wide settings) and uses a new format. You should re-backup your tables using the new version of the API.
• The kiji ls command can no longer perform get and scan — As mentioned above, new tools kiji get and kiji scan can be used for reading data stored in KijiSchema tables from scripts and the command
  line.
• KijiDataRequestValidator’s constructor has been hidden and validate() changed — Framework developers should use the factory method validatorForLayout() to get a validator which can validate KijiDataRequests against a table layout.
• KijiRowKeySplitter is now a non-static class — Use KijiRowKeysplitter.get() to obtain an instance.
• KijiRowData.getValues(“family”) and .getMostValues(“family”) now
  only work on map-type column families — This change is important to be able to return something more expressive than a map of Objects.
• KijiTable.close() has been deprecated — Applications should use `KijiTable.release()` instead.
• Kiji.modifyTableLayout() no longer takes a table name — Instead, the name is read from the TableLayoutDesc parameter. The older version of modifyTableLayout() still exists but has been
  deprecated.
• Kiji.createTable() now takes a TableLayoutDesc object — The older version of createTable() that takes a KijiTableLayout still exists but has been deprecated.
  

  Using new Kiji CLI features

  As mentioned above, the functionality of listing row data has been migrated out of kiji ls and into the get and scan tools. kiji ls has also gained the ability to list columns. Here are some example commands using these tools:

  List all kiji instances:

  kiji ls kiji://.env
    kiji ls kiji://localhost:2181
    kiji ls kiji://{host1,host2}:2181
    kiji ls # Relative KijiURI
  

  List all kiji tables in kiji://.env/default:

  kiji ls kiji://.env/default
    kiji ls default # Relative KijiURI
  

  List all columns in a kiji table ‘table_foo’:

  kiji ls default/table_foo
    kiji ls kiji://.env/default/table_foo
  

  For retrieving data from a single row where one knows the EntityId, use the kiji get tool. This commend retrieves displays the columns info:email and derived:domain from row identified by EntityId “bar” in table table_foo:

  kiji get default/table_foo/info:email,derived:domain \
      --entity-id=bar
  

  For getting more than one row, you can use the scan tool. This command scans through up to 10 rows, starting from the first row, and print columns info:email and derived:domain of table table_foo:

  kiji scan \
      kiji://.env/default/table_foo/info:email,derived:domain \
      --max-rows=10
  

  Scans also support start and limit ranges. This command scans through table table_foo form row start-row 0×50 (included) to limit-row 0xe0 (excluded):

  kiji scan \
      kiji://.env/default/table_foo/info:email,derived:domain \
      --start-row=hex:50 \
      --limit-row=hex:e0
  

  Command line tools will now accept formatted EntityIds for tables whose layout uses them. For example, the following command inserts a value into the users table, assuming that the table uses formatted EntityIds consisting of two strings (first and last name in this case):

  kiji put --target=default/users/info:state --entity-id="[ 'Kimball', 'Aaron' ]" --value='"CA"'
  

  In summary…

  KijiSchema’s rc5 release is an important step in the continued maturation of its functionality and API stabilization. Check back in a few weeks for the stable 1.0.0 release.

  Ready to try Kiji? Go download the BentoBox today!

  ]]> http://www.kiji.org/2013/03/15/announcing-kijischema-1-0-0-rc5/feed/ 0