A tool solves one single problem with a set of constraints. E.g. use a hammer to apply force on another flat surface. Constraint is that the effectiveness of hammer i.e. pressure exerted is inversely proportional to the area of application. Tools are everywhere in our abstract world of Software too.

We’ll explore a few ideas on the evolution or journey of a tool with a Key Value (KV) store as an example. Let’s start with the Why, Where and Who questions.

The purpose

KV stores persist blobs of data in a durable store. Their primary value prop stems from simple query API. Fewer upfront decisions (e.g. decide a key schema) lead to a low bar to adopt. Due to lack of support for transactions etc. they are fast and scale pretty well at a marginal cost to serve.

There are two cases of persistence. First, as a hot ephemeral cache to front load the traffic on another durable store. Second, use the KV store as the primary durable persistence. Both of these have different trade offs around consistency, latency etc. Usually the application tunes the store as needed.

Usually these stores provide more features like range queries, garbage collection, record level transaction, expiration etc. RocksDB is a fantastic example of one KV store.

Like the physical world, there are craftsmen who know how to wield a software tool. E.g. DB Admins. They understand the trade-offs that make a tool and can tune it to achieve the necessary objective. They are specialists.

However, as the tools become popular, they graduate to the repertoire of generalists. This is most of us. We know a little of everything, just enough to get by. And if need be we dive deep to do a job. So, unlike the specialized folks, we have a different motive. A job to be done.

This is the second step in the ladder of evolution.

Blend it into an Experience

People don’t simply buy products or services; they pull them into their lives to make progress. We call this progress the “job” they are trying to get done, and understanding this opens a world of innovation possibilities.

• Clayton Christensen¹

This generalist species (also called as Developers or Engineers) live in two core loops².

Inner loop is a circle of activities done in a single developer machine. Write some code using tools and libraries (reusable tools). The individual assembles several blocks of functionality together to reach a desired outcome for their application. Edit. Build. Debug. Test. Repeat.

A KV store embeds itself in the inner loop through an SDK. Edit phase is enhanced by the ambient code snippet suggestions on how to use the APIs, or Jupyter/Postman bundles. Build involves using tasks from the SDK to auto generate schema (e.g. from protobuf to cpp), or generate wrappers to make the API usage simpler. In Testing, the SDK provides a local or in-memory simulator to validate API calls along with fixtures and assertions for validation. Finally, the Debug phase provides hooks to intercept and trace API calls locally to diagnose the functional issues. Profiling or monitoring KV specific counters provide capability to run local load tests and observe performance.

Outer loop takes the output of the inner loop and circles it with the customer. Configure. Deploy. Observe. Repeat.

Configure is all about the management plane, i.e., create or modify the stores and tune them with config as code or similar. Deploy takes a container image with baked in necessities and pushes it out to a cloud environment. This makes the endpoint live. Observe exists as a separate set of endpoints to monitor the behavior in production and alert on a failure.

In essence, the KV tool now graduates to a large scope and blends itself with it’s customer workflows.

• Data contracts provide APIs to operate on the data persisted in the store. SDKs used by the app build on top of these. This provides the functional requirements for the app scenario.
• Management contracts provide APIs to operate on the store. E.g. handle the Deployment and Configuration aspects in the outer loop.
• Observation contracts provide Diagnostics and Monitoring capabilities. These provide the metrics etc. necessary for outer loop.

While this stage of evolution was about integration, the next is about extension.

Create an Ecosystem

An ecosystem brings in third parties into the periphery of what used to be the tool’s boundary. This happens in multiple interesting ways.

First, there are tools which build on top of our KV store. E.g. someone builds cache as a service on top of RocksDB. Another builds distribution and replication on top of sqlite (e.g. rqlite).

Second, there are tools which build within the KV store. E.g. extensions to RocksDB to store large blobs. Or extensions to sqlite that provide compression capabilities. This requires exposing the internals of KV store in the same modular approach as the external APIs.

The beauty of the ecosystem is symbiosis. KV store helps the participants in the ecosystem solve higher purposes than it could have done on its own. Everybody who builds on top of the KV store creates way more business impact. In return, the participants give back excellent feedback which refines the KV store’s architecture even further.

Tools are commodities. Experiences are premium. Ecosystems are the holy grail.

It is hard to build an ecosystem without a great experience. Similarly beautiful experiences are futile if the underlying core purpose of the tool is not solved.

It goes without saying that evolution is a function of feedback from participants in the ecosystem. A moment of lapse can shake the foundations with an incorrect abstraction. These are extremely hard trade-offs when the business priorities pull us in one side and the architectural purity on the other.