Tuesday 23 January 2024

How use Redis in a Rust web service

Redis has been a fundamental component of the web ecosystem for an extended period. Its versatility has led to its use in various capacities, including caching, message brokering, and database management.

In this guide, we’ll demonstrate how to use Redis inside a Rust web application.

We’ll build the application using the fantastic warp web framework. The techniques in this tutorial will work very similarly with other Rust web stacks.

We’ll explore three approaches to using Redis:

As a client library for Redis, redis-rs is the most stable and widely used crate, so all variants use it as their basis.

For the synchronous pool, we’ll use the r2d2-based r2d2-redis. We’ll use mobc for the asynchronous solution. There are plenty of other async connection pools, such as deadpool and bb8, that all work in a similar way.

The application itself doesn’t do much. There is a handler for each of the different Redis connection approaches, which puts a hardcoded string into Redis with an expiration time of 60 seconds and then reads it out again.

Without further ado, let’s get started!

Setup

First, let’s set up some shared types and define a module for each Redis connection approach:

mod direct;
mod mobc_pool;
mod r2d2_pool;

type WebResult<T> = std::result::Result<T, Rejection>;
type Result<T> = std::result::Result<T, Error>;

const REDIS_CON_STRING: &str = "redis://127.0.0.1/";

The two Result types are defined to save some typing and to represent internal Errors (Result) and external Errors (WebResult).

Next, define this internal Error-type and implement Reject for it so it can be used to return HTTP errors from handlers.

#[derive(Error, Debug)]
pub enum Error {
    #[error("mobc error: {0}")]
    MobcError(#[from] MobcError),
    #[error("direct redis error: {0}")]
    DirectError(#[from] DirectError),
    #[error("r2d2 error: {0}")]
    R2D2Error(#[from] R2D2Error),
}

#[derive(Error, Debug)]
pub enum MobcError {
    #[error("could not get redis connection from pool : {0}")]
    RedisPoolError(mobc::Error<mobc_redis::redis::RedisError>),
    #[error("error parsing string from redis result: {0}")]
    RedisTypeError(mobc_redis::redis::RedisError),
    #[error("error executing redis command: {0}")]
    RedisCMDError(mobc_redis::redis::RedisError),
    #[error("error creating Redis client: {0}")]
    RedisClientError(mobc_redis::redis::RedisError),
}

#[derive(Error, Debug)]
pub enum R2D2Error {
    #[error("could not get redis connection from pool : {0}")]
    RedisPoolError(r2d2_redis::r2d2::Error),
    #[error("error parsing string from redis result: {0}")]
    RedisTypeError(r2d2_redis::redis::RedisError),
    #[error("error executing redis command: {0}")]
    RedisCMDError(r2d2_redis::redis::RedisError),
    #[error("error creating Redis client: {0}")]
    RedisClientError(r2d2_redis::redis::RedisError),
}

#[derive(Error, Debug)]
pub enum DirectError {
    #[error("error parsing string from redis result: {0}")]
    RedisTypeError(redis::RedisError),
    #[error("error executing redis command: {0}")]
    RedisCMDError(redis::RedisError),
    #[error("error creating Redis client: {0}")]
    RedisClientError(redis::RedisError),
}

impl warp::reject::Reject for Error {}

This boilerplate is extensive, but it's necessary to implement three ways to achieve the same outcome with minimal code duplication. The error types are predefined, and Error types are specified for each Redis approach we'll implement. These errors handle connection, pool creation, and command execution issues. You'll see them in action later.

The first method we'll implement is using redis-rs directly, utilizing one connection per request. While suitable for low traffic, this approach doesn't scale well with numerous concurrent requests. Both synchronous and asynchronous APIs are available in redis-rs, and we'll focus on the latter.

Using `redis-rs` directly (async)

Establishing a new connection is our first step.

use crate::{DirectError::*, Result};
use redis::{aio::Connection, AsyncCommands, FromRedisValue};

pub async fn get_con(client: redis::Client) -> Result<Connection> {
    client
        .get_async_connection()
        .await
        .map_err(|e| RedisClientError(e).into())
}

In the above snippet, a redis::Client is passed in and an async connection comes out, handling the error. We’ll examine the creation of the redis::Client later.

Now that it’s possible to open a connection, the next step is to create some helpers to enable setting values in Redis and retrieve them again. In this basic case, we’ll only focus on String values.

pub async fn set_str(
    con: &mut Connection,
    key: &str,
    value: &str,
    ttl_seconds: usize,
) -> Result<()> {
    con.set(key, value).await.map_err(RedisCMDError)?;
    if ttl_seconds > 0 {
        con.expire(key, ttl_seconds).await.map_err(RedisCMDError)?;
    }
    Ok(())
}

pub async fn get_str(con: &mut Connection, key: &str) -> Result<String> {
    let value = con.get(key).await.map_err(RedisCMDError)?;
    FromRedisValue::from_redis_value(&value).map_err(|e| RedisTypeError(e).into())
}

This part will be incredibly similar between all three implementations, since this is just the redis-rs API in action. The API closely mirrors Redis commands, and the FromRedisValue trait is a convenient way to convert values into the expected data types.

So far, so good. Next up is a synchronous pool based on the widely used r2d2 crate.

Using r2d2 (sync)

The r2d2 crate was, to my knowledge, the first widely used connection pool, and it still enjoys widespread use. The Redis flavor for this base crate is r2d2-redis.

Since we’re using a connection pool now, the creation of this pool needs to be handled in the r2d2 module as well. Here’s the starting point:

pub type R2D2Pool = r2d2::Pool<RedisConnectionManager>;
pub type R2D2Con = r2d2::PooledConnection<RedisConnectionManager>;

const CACHE_POOL_MAX_OPEN: u32 = 16;
const CACHE_POOL_MIN_IDLE: u32 = 8;
const CACHE_POOL_TIMEOUT_SECONDS: u64 = 1;
const CACHE_POOL_EXPIRE_SECONDS: u64 = 60;

pub fn connect() -> Result<r2d2::Pool<RedisConnectionManager>> {
    let manager = RedisConnectionManager::new(REDIS_CON_STRING).map_err(RedisClientError)?;
    r2d2::Pool::builder()
        .max_size(CACHE_POOL_MAX_OPEN)
        .max_lifetime(Some(Duration::from_secs(CACHE_POOL_EXPIRE_SECONDS)))
        .min_idle(Some(CACHE_POOL_MIN_IDLE))
        .build(manager)
        .map_err(|e| RedisPoolError(e).into())
}

Following the definition of some constants to configure the pool, such as open and idle connections, connection timeout, and the lifetime of a connection, the pool is created using the RedisConnectionManager, which takes the redis connection string as an argument.

Don’t worry too much about the configuration values; they’re just shown here to demonstrate their availability. Properly setting them will depend on your specific use case, and most connection pools have reasonable default values for basic applications.

We require a method to obtain a connection from the pool and utilize the helpers to set and retrieve data from Redis.

pub fn get_con(pool: &R2D2Pool) -> Result<R2D2Con> {
    pool.get_timeout(Duration::from_secs(CACHE_POOL_TIMEOUT_SECONDS))
        .map_err(|e| {
            eprintln!("error connecting to redis: {}", e);
            RedisPoolError(e).into()
        })
}

pub fn set_str(pool: &R2D2Pool, key: &str, value: &str, ttl_seconds: usize) -> Result<()> {
    let mut con = get_con(&pool)?;
    con.set(key, value).map_err(RedisCMDError)?;
    if ttl_seconds > 0 {
        con.expire(key, ttl_seconds).map_err(RedisCMDError)?;
    }
    Ok(())
}

pub fn get_str(pool: &R2D2Pool, key: &str) -> Result<String> {
    let mut con = get_con(&pool)?;
    let value = con.get(key).map_err(RedisCMDError)?;
    FromRedisValue::from_redis_value(&value).map_err(|e| RedisTypeError(e).into())
}

As you can see, this is very similar to just using the redis-rs crate directly. Instead of a redis::Client, the pool is passed in and we try to get a connection from the pool with a configured timeout.

In set_str and get_str, the only thing that changes is that get_con is called on each invocation of these functions. In the previous case, this would have meant creating a Redis connection for every single command, which would have been even more inefficient.

But since we have a pool with precreated connections, we can get them here and they can be shared between requests freely while no Redis action is going on in our current request.

Also, notice how the API is almost exactly the same, except for the async and await parts, so moving from one to the other is not too painful if that is something you have to do at some point.

Using mobc (async)

Speaking of async, let’s look at the last of our three approaches: an async connection Pool.

As mentioned above, there are several crates for this. We’ll use mobc, mostly because I’ve used it successfully in a production setting already. Again, the other options work similarly and all seem like great options as well.

Let’s define our configurations and create the pool.

pub type MobcPool = Pool<RedisConnectionManager>;
pub type MobcCon = Connection<RedisConnectionManager>;

const CACHE_POOL_MAX_OPEN: u64 = 16;
const CACHE_POOL_MAX_IDLE: u64 = 8;
const CACHE_POOL_TIMEOUT_SECONDS: u64 = 1;
const CACHE_POOL_EXPIRE_SECONDS: u64 = 60;

pub async fn connect() -> Result<MobcPool> {
    let client = redis::Client::open(REDIS_CON_STRING).map_err(RedisClientError)?;
    let manager = RedisConnectionManager::new(client);
    Ok(Pool::builder()
        .get_timeout(Some(Duration::from_secs(CACHE_POOL_TIMEOUT_SECONDS)))
        .max_open(CACHE_POOL_MAX_OPEN)
        .max_idle(CACHE_POOL_MAX_IDLE)
        .max_lifetime(Some(Duration::from_secs(CACHE_POOL_EXPIRE_SECONDS)))
        .build(manager))
}

This bears a striking resemblance to r2d2, which is no surprise; several connection pool crates have drawn inspiration from r2d2's exceptional API.

To complete the async pool implementation, we have connections, getting, and setting:

async fn get_con(pool: &MobcPool) -> Result<MobcCon> {
    pool.get().await.map_err(|e| {
        eprintln!("error connecting to redis: {}", e);
        RedisPoolError(e).into()
    })
}

pub async fn set_str(pool: &MobcPool, key: &str, value: &str, ttl_seconds: usize) -> Result<()> {
    let mut con = get_con(&pool).await?;
    con.set(key, value).await.map_err(RedisCMDError)?;
    if ttl_seconds > 0 {
        con.expire(key, ttl_seconds).await.map_err(RedisCMDError)?;
    }
    Ok(())
}

pub async fn get_str(pool: &MobcPool, key: &str) -> Result<String> {
    let mut con = get_con(&pool).await?;
    let value = con.get(key).await.map_err(RedisCMDError)?;
    FromRedisValue::from_redis_value(&value).map_err(|e| RedisTypeError(e).into())
}

This should look quite familiar by now. The solution is a bit of a hybrid between the first and second options: The pool is passed in and fetches a connection at the start, but this time in an asynchronous way with async and await.

Bringing it all together

Now that we’ve covered the basics and we have three isolated ways to connect to and interact with Redis, the next step is to bring all of them together in a warp web application.

Let’s start with the main function to get an overview of how the application is structured and what is needed to get this to run.

#[tokio::main]
async fn main() {
    let redis_client = redis::Client::open(REDIS_CON_STRING).expect("can create redis client");
    let mobc_pool = mobc_pool::connect().await.expect("can create mobc pool");
    let r2d2_pool = r2d2_pool::connect().expect("can create r2d2 pool");

    let direct_route = warp::path!("direct")
        .and(with_redis_client(redis_client.clone()))
        .and_then(direct_handler);

    let r2d2_route = warp::path!("r2d2")
        .and(with_r2d2_pool(r2d2_pool.clone()))
        .and_then(r2d2_handler);

    let mobc_route = warp::path!("mobc")
        .and(with_mobc_pool(mobc_pool.clone()))
        .and_then(mobc_handler);

    let routes = mobc_route.or(direct_route).or(r2d2_route);
    warp::serve(routes).run(([0, 0, 0, 0], 8080)).await;
}

Not much is needed, it's a small amount of code!

When the application starts, we'll attempt to create a Redis client and two connection pools. If any of them fail, we'll fail too.

Next, we'll define the route definitions for each approach. They look very similar, the only difference being the with_xxx filters.

fn with_redis_client(
    client: redis::Client,
) -> impl Filter<Extract = (redis::Client,), Error = Infallible> + Clone {
    warp::any().map(move || client.clone())
}

fn with_mobc_pool(
    pool: MobcPool,
) -> impl Filter<Extract = (MobcPool,), Error = Infallible> + Clone {
    warp::any().map(move || pool.clone())
}

fn with_r2d2_pool(
    pool: R2D2Pool,
) -> impl Filter<Extract = (R2D2Pool,), Error = Infallible> + Clone {
    warp::any().map(move || pool.clone())
}

The above filters are merely simple extraction filters that make the passed-in entities available to the handlers with which they are used. In each case, the client and pools are cloned and moved into the handlers — no magic here.

Last, but not least, let’s examine the actual handlers to see how the API we defined above can be utilized.

First, using redis-rs directly:

async fn direct_handler(client: redis::Client) -> WebResult<impl Reply> {
    let mut con = direct::get_con()
        .await
        .map_err(|e| warp::reject::custom(e))?;
    direct::set_str(&mut con, "hello", "direct_world", 60)
        .await
        .map_err(|e| warp::reject::custom(e))?;
    let value = direct::get_str(&mut con, "hello")
        .await
        .map_err(|e| warp::reject::custom(e))?;
    Ok(value)
}

According to the with_direct filter, we obtain the redis::Client object and establish a Redis connection with it right away.

The handler then writes the string direct world into the hello key in Redis and immediately reads it back.

The r2d2 handler has a similar structure, but with an R2D2Pool object being passed in and without any async or await keywords.

async fn r2d2_handler(pool: R2D2Pool) -> WebResult<impl Reply> {
    r2d2_pool::set_str(&pool, "r2d2_hello", "r2d2_world", 60)
        .map_err(|e| warp::reject::custom(e))?;
    let value = r2d2_pool::get_str(&pool, "r2d2_hello").map_err(|e| warp::reject::custom(e))?;
    Ok(value)
}

While utilizing a synchronous pool with an asynchronous web framework is not usually recommended, there are numerous frameworks that do not rely on async/await, and for those, the synchronous pool will be ideal.

To conclude, here's the asynchronous pool implemented with mobc:

async fn mobc_handler(pool: MobcPool) -> WebResult<impl Reply> {
    mobc_pool::set_str(&pool, "mobc_hello", "mobc_world", 60)
        .await
        .map_err(|e| warp::reject::custom(e))?;
    let value = mobc_pool::get_str(&pool, "mobc_hello")
        .await
        .map_err(|e| warp::reject::custom(e))?;
    Ok(value)
}

The three Redis client implementations - filters, routes, and handlers - share striking similarities, making it easy to adapt to changing needs without requiring a complete overhaul of your codebase.

Now, it's time to launch the application and put it through its paces.

cargo run

Next, start a local Redis instance with Docker.

docker run -p 6379:6379 redis:5.0

If you employ curl to invoke the above-defined endpoints, you should witness the appropriate responses.

curl http://localhost:8080/direct
curl http://localhost:8080/r2d2
curl http://localhost:8080/mobc

For exceedingly simple applications where you only occasionally call into Redis, obtaining a new connection each time is perfectly acceptable. However, if you have multiple Redis calls per request with numerous requests occurring simultaneously, a connection pool will significantly enhance performance.

When in doubt, test, profile, and make an informed decision based on the data you gather.

Conclusion

In this post, we explored three approaches to utilizing Redis within a Rust web application. The most suitable method for your application will depend on your use case and the way you structure the app. At any rate, the ability to switch between crates and approaches without having to change too much is a significant advantage.

The redis-rs crate, together with the rich ecosystem of both synchronous and asynchronous connection pools, is ready for production use and strikes a great balance between usability and performance.

Hashtags:

rust redis programming web development

Created by:

Jorge García

Fullstack developer