Key-Value Store tutorial

Building a Key-Value Store with Calimero SDK

The Calimero SDK for Rust empowers developers to build applications that compile to WebAssembly (Wasm) and run securely within the Calimero virtual machine. This guide will walk you through creating a complete key-value store application and preparing it for deployment.

Prerequisites

Before you begin, ensure you have Rust installed on your system. If not, follow the official Rust installation guide for your platform: Rust Installation Guide.

You should ensure you have the wasm32-unknown-unknown target installed. Run the following command in your terminal to install the target:

Terminal

rustup target add wasm32-unknown-unknown

Setting Up Your Project

To create a new project, initialize a Rust library project using Cargo. Run the following command in your terminal:

Terminal

cargo new --lib kv-store

You should have a tree that looks like this:

Terminal

$ tree kv-store
kv-store
├── Cargo.toml
└── src
    └── lib.rs

2 directories, 2 files

At this point, we can cd into the kv-store directory:

Terminal

cd kv-store

Next, you need to specify the crate-type as cdylib in your Cargo.toml file to generate a dynamic library that can be compiled to Wasm:

File: Cargo.toml

[lib]
crate-type = ["cdylib"]

You can now configure your project to use the Calimero SDK by adding it as a dependency in your Cargo.toml file:

File: Cargo.toml

[dependencies]
calimero-sdk = { git = "https://github.com/calimero-network/core" }
calimero-storage = { git = "https://github.com/calimero-network/core" }

Then, we need to specify a custom build profile for the most compact Wasm output:

File: Cargo.toml

[profile.app-release]
inherits = "release"
codegen-units = 1
opt-level = "z"
lto = true
debug = false
panic = "abort"
overflow-checks = true

Your Cargo.toml file should now look like this

File: Cargo.toml

[package]
name = "kv-store"
version = "0.1.0"
edition = "2021"

[lib]
crate-type = ["cdylib"]

[dependencies]
calimero-sdk = { git = "https://github.com/calimero-network/core" }
calimero-storage = { git = "https://github.com/calimero-network/core" }

[profile.app-release]
inherits = "release"
codegen-units = 1
opt-level = "z"
lto = true
debug = false
panic = "abort"
overflow-checks = true

And finally, create a build.sh script to compile your application into Wasm format, for example:

File: build.sh

#!/bin/bash
set -e

cd "$(dirname $0)"

TARGET="${CARGO_TARGET_DIR:-../../target}"

rustup target add wasm32-unknown-unknown

cargo build --target wasm32-unknown-unknown --profile app-release

mkdir -p res

cp $TARGET/wasm32-unknown-unknown/app-release/kv_store.wasm ./res/

You can optionally choose to install and use wasm-opt, for an additional optimization step in the build script. This step is not required but can help reduce the size of the generated Wasm file:

File: build.sh

if command -v wasm-opt > /dev/null; then
  wasm-opt -Oz ./res/kv_store.wasm -o ./res/kv_store.wasm
fi

Don't forget to make the build.sh script executable:

Terminal

chmod +x build.sh

At this point, your project structure should look like this:

Terminal

$ tree
.
├── Cargo.toml
├── build.sh
└── src
    └── lib.rs

2 directories, 3 files

Writing Your Application

Now, let's create a simple key-value store application using the Calimero SDK. Start by defining your application logic in lib.rs:

File: src/lib.rs

use calimero_sdk::borsh::{BorshDeserialize, BorshSerialize};
use calimero_sdk::app;

#[app::state]
#[derive(Default, BorshSerialize, BorshDeserialize)]
#[borsh(crate = "calimero_sdk::borsh")]
struct KvStore {}

#[app::logic]
impl KvStore {
    #[app::init]
    pub fn init() -> KvStore {
        KvStore {}
    }
}

The KvStore struct represents the state of your application, which will be borsh-encoded in the app-scoped state partition on the node's storage. The #[app::state] attribute macro marks the struct as the application state, permitting its use by Calimero SDK.

The #[app::logic] attribute macro marks the implementation block as the application logic, allowing you to define the methods that interact with the application state. An initializer method (named init) is denoted by the #[app::init] attribute macro, which is called when the application is executed against a freshly created context.

Consider a method like get that retrieves a value from the key-value store:

pub fn get(&self, key: &str) -> Result<Option<String>, Error> {
    // Snip...
}

The inputs must be deserializable from the transaction data, and the output must be serializable to the response data. The Option type is used to represent the possibility of the key not being present in the store. The Error type is used to represent the possible error conditions that may occur during the execution of the method.

And now, here's a complete example of a key-value store application:

File: src/lib.rs

use calimero_sdk::borsh::{BorshDeserialize, BorshSerialize};
use calimero_sdk::types::Error;
use calimero_sdk::{app, env};
use calimero_storage::collections::UnorderedMap;

#[app::state]
#[derive(Default, BorshSerialize, BorshDeserialize)]
#[borsh(crate = "calimero_sdk::borsh")]
struct KvStore {
    entries: UnorderedMap<String, String>,
}

#[app::logic]
impl KvStore {
    #[app::init]
    pub fn init() -> KvStore {
        KvStore {
            items: UnorderedMap::new(),
        }
    }

    pub fn set(&mut self, key: String, value: String) -> Result<(), Error> {
        env::log(&format!("Setting key: {:?} to value: {:?}", key, value));

        self.entries.insert(key, value)?;

        Ok(())
    }

    pub fn entries(&self) -> Result<BTreeMap<String, String>, Error> {
        env::log("Getting all entries");

        Ok(self.items.entries()?.collect())
    }

    pub fn get(&self, key: &str) -> Result<Option<String>, Error> {
        env::log(&format!("Getting key: {:?}", key));

        self.items.get(key).map_err(Into::into)
    }

    pub fn remove(&mut self, key: &str) -> Result<Option<String>, Error> {
        env::log(&format!("Removing key: {:?}", key));

        self.items.remove(key).map_err(Into::into)
    }
}

Building Your Application

Once your application logic is defined, run the ./build.sh script to compile your application into Wasm format. This script will generate kv_store.wasm in the res folder of your application.

Terminal

$ ./build.sh
info: component 'rust-std' for target 'wasm32-unknown-unknown' is up to date
   # Snip...
   Compiling calimero-sdk v0.1.0
   Compiling calimero-storage v0.1.0
   Compiling kv-store v0.1.0 (/apps/kv-store)
    Finished `app-release` profile [optimized] target(s) in 1.20s

$ tree
.
├── Cargo.toml
├── build.sh
├── res
│   └── kv_store.wasm
└── src
    └── lib.rs

3 directories, 4 files

Deploying Your Application

After successfully building your application, you can upload the compiled kv_store.wasm to the registry for use by a live Calimero node.

Writing Efficient Code with Calimero SDK

In the following code snippet:

File: src/lib.rs

pub fn get(&self, key: &str) -> Result<Option<String>, Error> {
    // Snip...
}

you'll notice that we prioritize using references instead of owned values. This approach optimizes performance and memory usage by minimizing unnecessary data copying.

For input parameters, such as &str and &[u8], utilizing references allows you to avoid unnecessary copying of data. Similarly, for output values, you can return references to data that live as long as &self or any of the input parameters. By doing so, you reduce memory overhead and improve the overall efficiency of your application.

Handling Errors with Calimero SDK

When designing methods that may potentially fail, it's recommended to return a Result type with an error variant representing the possible failure cases. This enables you to handle errors more effectively and communicate error conditions to users of your application. This is recommended over using the Error type exported from calimero_sdk and over panicking. Both of which only return a string message.

Error Report Comparison

Let's take the following cases (all of which fail when the key does not exist);

1. Using calimero_sdk::types::Error:

   This is provided for convenience, since most errors already don't implement Serialize, and so they cannot be immediately returned. This first converts the error to a string and then returns it. Which then JSON-encodes the string representation.

   File: src/lib.rs

   use calimero_sdk::types::Error;
   
   pub fn get(&self, key: &str) -> Result<String, Error> {
       self.items.get(key)?.ok_or_else(|| Error::msg("key not found"))
   }

   This failure will result in this outcome:

   ExecutionError([ 107, 75, 101, 121, 32, 110, 111, 116, 32, 102, 111, 117, 110, 100, 34 ])

   which can be decoded to

   "key not found"

   This Error can be constructed with ? so long as the source error implements std::error::Error.

   Behaviourally similar to anyhow::Error or eyre::Report.

2. Using a custom error type (recommended):

   For structured error handling, we recommend defining a custom error type that encodes all the possible error variants for that method. This allows you to provide more context about the error condition and handle different error scenarios more effectively. As opposed to string parsing.

   File: src/lib.rs

   use calimero_sdk::serde::Serialize;
   
   #[derive(Debug, Serialize)]
   #[serde(crate = "calimero_sdk::serde")]
   #[serde(tag = "kind", content = "data")]
   pub enum Error<'a> {
       NotFound(&'a str),
   }
   File: src/lib.rs

   pub fn get<'a>(&self, key: &'a str) -> Result<String, Error<'a>> {
       // Snip...
       Err(Error::NotFound(key))
   }

   This failure will result in this outcome:

   ExecutionError([ 123, 34, 107, 105, 110, 100, 34, 58, 34, 78, 111, 116, 70, 111, 117, 110, 100, 34, 44, 34, 100, 97, 116, 97, 34, 58, 34, 116, 104, 105, 110, 103, 34, 125 ])

   which can be decoded to

   { "kind": "NotFound", "data": "thing" }

   As will most likely be the case, you may need to work with storage errors while you've defined a custom error type.

   In this case, you can pull in thiserror to help.

   File: src/lib.rs

   use thiserror::Error;
   
   #[derive(Debug, Error, Serialize)]
   #[serde(crate = "calimero_sdk::serde")]
   #[serde(tag = "kind", content = "data")]
   pub enum Error<'a> {
       #[error("key not found: {0}")]
       NotFound(&'a str),
       #[error("store error: {0}")]
       StoreError(#[from] StoreError),
   }
   File: src/lib.rs

   pub fn get<'a>(&self, key: &'a str) -> Result<String, Error<'a>> {
       // Snip...
       self.items.get(key)?.ok_or_else(|| Error::NotFound(key))
   }

   An example store error would then be represented as:

   ExecutionError(
       [
           123, 34, 107, 105, 110, 100, 34, 58, 34, 83, 116, 111, 114, 101, 69, 114, 114, 111, 114, 34, 44, 34, 100, 97, 116, 97, 34, 58,
           34, 73, 110, 100, 101, 120, 32, 110, 111, 116, 32, 102, 111, 117, 110, 100, 32, 102, 111, 114, 32, 73, 68, 58, 32, 57, 51, 49, 53,
           97, 98, 101, 49, 101, 97, 101, 48, 102, 102, 53, 98, 48, 48, 52, 53, 51, 97, 100, 97, 102, 99, 99, 53, 102, 101, 102, 50, 49, 100,
           55, 52, 49, 51, 57, 55, 101, 50, 49, 99, 53, 49, 53, 51, 55, 99, 51, 54, 52, 52, 50, 52, 50, 48, 56, 55, 52, 57, 99, 57, 34, 125,
       ],
   )

   which can be decoded to

   {
     "kind": "StoreError",
     "data": "Index not found for ID: 9315abe1eae0ff5b00453adafcc5fef21d741397e21c51537c364424208749c9"
   }

3. Panic (ideally, development only)

   File: src/lib.rs

   pub fn get(&self, key: &str) -> String {
       self.items.get(key).expect("store error").expect("key not found")
   }

   A non-existent key would then lead to this outcome:

   HostError(
       Panic {
           context: Guest,
           message: "key not found",
           location: At {
               file: "apps/kv-store/src/lib.rs",
               line: 98,
               column: 14,
           },
       },
   )

   And a storage error, would produce this:

   HostError(
       Panic {
           context: Guest,
           message: "store error: StorageError(IndexNotFound(Id { bytes: [123, 240, 135, 21, 77, 143, 81, 169, 15, 202, 99, 210, 167, 165, 188, 156, 87, 146, 7, 211, 100, 92, 169, 189, 124, 115, 200, 242, 240, 73, 68, 123] }))",
           location: At {
               file: "apps/kv-store/src/lib.rs",
               line: 98,
               column: 14,
           },
       },
   )

By following the second (recommended) approach, you can handle errors more gracefully and provide meaningful feedback to users of your Calimero application.

And the first approach, if you want a hassle-free method of dealing with errors.

Emitting Events with Calimero SDK

To facilitate real-time monitoring of state transitions within your Calimero application, you can emit events using the app::emit! macro provided by the Calimero SDK. Event emission is particularly useful for handling live state transitions triggered by other actors, allowing subscribed clients to receive immediate updates about relevant actions.

Let's focus on emitting events for mutating calls, specifically set and remove methods:

First, define your custom events using the #[app::event] proc macro. In this example, we'll define events for setting a new key-value pair (Inserted), updating an existing value (Updated), and removing a key-value pair (Removed):

File: src/lib.rs

use calimero_sdk::app;

#[app::event]
pub enum Event<'a> {
    Inserted { key: &'a str, value: &'a str },
    Updated { key: &'a str, value: &'a str },
    Removed { key: &'a str },
}

Each event variant can carry additional data to provide context about the event.

Now, you need to associate the event with the application logic by annotating the application state.

File: src/lib.rs

#[app::state(emits = for<'a> Event<'a>)]
#[derive(Default, BorshSerialize, BorshDeserialize)]
#[borsh(crate = "calimero_sdk::borsh")]
struct KvStore {
    // Snip...
}

And finally, within your application logic methods, emit events using the app::emit! macro:

File: src/lib.rs

pub fn set(&mut self, key: String, value: String) -> Result<(), Error> {
    if self.items.contains(&key)? {
        app::emit!(Event::Updated {
            key: &key,
            value: &value,
        });
    } else {
        app::emit!(Event::Inserted {
            key: &key,
            value: &value,
        });
    }

    self.items.insert(key, value)?;

    Ok(())
}

pub fn remove(&mut self, key: &str) -> Result<String, Error> {
    app::emit!(Event::Removed { key });

    self.entries.remove(key)?.ok_or_else(|| Error::msg("key not found"))
}

In each method, we emit the corresponding event with relevant data. This allows external observers to react to these events and take appropriate actions.

By emitting events, you can ensure connected clients receive real-time updates about state transitions within your Calimero application, enabling them to respond to changes as they occur.

Ensuring Atomicity and Event Reliability in Calimero Applications

In Calimero applications, ensuring atomicity of state changes and reliability of event emission is crucial for maintaining data consistency and facilitating reliable communication between actors. Here's how atomicity and event reliability are enforced:

Atomic State Changes

When a method call fails, whether due to panics or returning an Err, all state changes made up to that point are discarded. This ensures that if an operation cannot be completed successfully, the application's state remains consistent and unaffected by partial updates. By enforcing atomicity, Calimero promotes data integrity and prevents inconsistencies that may arise from incomplete transactions.

Reliable Event Emission

Similarly, event emission in Calimero applications is tied to the successful execution of transactions. Events are only relayed when a transaction has been successfully executed, ensuring that external observers receive updates about state changes reliably. By linking event emission to transaction execution, Calimero guarantees that event notifications accurately reflect the application's current state, enhancing the reliability and consistency of communication between actors.

This also means it doesn't matter if the event emission is done before or after the state change, as the event will only be emitted if the state change is successful.

By adhering to these principles of atomicity and event reliability, Calimero applications maintain data integrity and enable robust interaction between different components, facilitating the development of secure and dependable decentralized systems.

Local-First Efficiency

Read-only operations (like get) have no network overhead, as they don't modify state and can be executed locally.

Conclusion

You've now learned how to set up a Rust project using the Calimero SDK, write a simple application, build it into Wasm, and prepare it for deployment. Experiment with different features and functionalities to create powerful and secure applications with Calimero.

Happy coding! 🚀