Using Redis Modules For Advanced Use-Cases

Redis is an open-source, in-memory datastore with powerful native data structures. There are, however, use cases where you might need additional features or completely different capabilities altogether. Redis modules are advanced features that allow us to implement custom data types specific. Essentially, a module is a dynamic library that can be loaded into Redis at startup or on demand with the MODULE LOAD command. Modules can be written in various languages including C and Rust.

Implementing a new data type by ourselves with Redis modules is a nontrivial effort. Thankfully, there are many popular and widely used modules to solve problems such as full-text search (RediSearch), time series processing (RedisTimeSeries), and native JSON support (RedisJSON). Let’s get an overview of some of the more widely used Redis modules.

Overview of popular Redis Modules

This section provides a high-level overview of RediSearch, RedisJSON and RedisTimeSeries modules, along with the important commands associated with them.

RedisJSON

When it comes to handling JSON data in Redis, traditional approaches have their limitations. Redis, by default, treats values as simple strings, lacking built-in support for querying or manipulating structured data like JSON. As a result, developers often resort to storing JSON as stringified values, sacrificing the ability to perform efficient operations on individual fields or take advantage of nested structures. This approach requires parsing the entire JSON document even for minor updates, resulting in inefficiency and increased processing overhead.

The RedisJSON module addresses these challenges by introducing native JSON handling capabilities to Redis. It lets you store, update, and retrieve JSON like any other native data type.

RedisJSON offers a rich set of operations on JSON documents. Here are some of the most commonly used commands:

1. JSON.SET: Sets a JSON value in Redis.
2. JSON.GET: Retrieves JSON values associated with a given key.
3. JSON.DEL: Deletes a JSON value from Redis.
4. JSON.TYPE: Returns the type of a JSON value, indicating whether it is an object, array, string, number, boolean, or null.
5. JSON.ARRAPPEND: Appends one or more values to the end of a JSON array.
6. JSON.ARRLEN: Returns the length of a JSON array, providing the number of elements it contains.
7. JSON.OBJKEYS: Retrieves the keys of a JSON object, returning an array of all the keys present.
8. JSON.OBJLEN: Returns the size of JSON object.
9. JSON.NUMINCRBY: Increments a numeric value within a JSON document by a specified amount.
10. JSON.STRAPPEND: Appends a strings within a JSON document.

RediSearch

The RediSearch module enhances Redis with powerful full-text search capabilities. It introduces an inverted index-based search engine, allowing efficient and scalable text search. With RediSearch, you can index textual data, perform complex search queries, and retrieve relevant search results. It provides features like keyword search, exact phrase matching, boolean operations, pagination, sorting, and filtering. RediSearch scores search results based on how well they match the query, enabling you to rank and prioritize search results.

Here are some of the most commonly used RediSearch commands:

1. FT.CREATE: Creates a new search index with specified schema and configuration options.
2. FT.SEARCH: Performs a search query on the specified index, returning matching documents based on the given search criteria.
3. FT.AGGREGATE: Performs an aggregation query on the specified index, returning aggregated results based on the given aggregation criteria.
4. FT.INFO: Retrieves information about a search index, including statistics, configuration settings, and schema details.
5. FT.DROPINDEX: Deletes a search index, removing all the documents and associated data.

RedisTimeSeries

When it comes to handling high-volume time series data in Redis, there are a few techniques commonly used. These involve storing data in Redis Sorted Sets, where the score represents the timestamp, and the member represents the data value. Another approach is to use string keys with a timestamp as part of the key name. Each key-value pair represents a specific data point in the time series.

However, these techniques have some limitations when dealing with high-volume time series data. Traditional Redis data structures like String keys or Sorted Sets have limited querying capabilities, and it’s difficult to perform complex aggregations. Also, storing each data point as a separate key-value pair can consume a significant amount of memory when dealing with large datasets, which is often the case with time series.

The RedisTimeSeries module is a native time series data structure that provides efficient storage and querying capabilities for time series data. It adds querying capabilities such as retrieving data within a time range, performing aggregations (e.g., sum, count, average), down sampling, and interpolation. It also allows us to define retention policies that automatically expire or downsample data, enabling efficient data retention and reducing storage requirements. RedisTimeSeries also integrates with other tools in the ecosystem such as Prometheus and Grafana.

RedisTimeSeries offers a rich set of commands. Here are some of the most commonly used ones:

1. TS.CREATE: Creates a new time series with the specified key, labels, and retention policy.
2. TS.ADD: Adds a new data point to a time series, associating it with a timestamp and value.
3. TS.RANGE: Retrieves a range of data points from a time series within a specified time range.
4. TS.MRANGE: Retrieves data points from multiple time series within a specified time range.
5. TS.GET: Retrieves the latest data point from a time series.
6. TS.MGET: Retrieves the latest data points from multiple time series.
7. TS.INCRBY: Increments the value of a data point in a time series at a specific timestamp.
8. TS.DECRBY: Decrements the value of a data point in a time series at a specific timestamp.
9. TS.INFO: Retrieves metadata information about a time series, such as the number of samples, memory usage, and retention policy.
10. TS.QUERYINDEX: Performs an index-based query on a time series, retrieving data points based on specified filters and aggregations.

Now that we have a basic understanding of these Modules, let’s learn how to use them in practice.

Using Redis modules

To follow this part of the tutorial you will need to install a recent version of Go and Docker. All the aforementioned modules are available as part of Redis Stack, which bundles Redis with various related packages and services. You can start a Redis Stack local instance using Docker:

docker run -d --name redis-stack -p 6379:6379 redis/redis-stack:latest

Before starting, let’s create a Go module to host the examples:

go mod init go-redis-modules

RedisJSON

The below example demonstrates how to use various RedisJSON commands in a Go application. Note that we are using the go-redis client.

package main

import (
	"context"
	"errors"
	"fmt"
	"log"

	"github.com/redis/go-redis/v9"
)

func main() {
	rdb := redis.NewClient(&redis.Options{
		Addr: "localhost:6379",
	})

	ctx := context.Background()

	// JSON.SET
	err := rdb.Do(ctx, "JSON.SET", "mydoc", ".", `{"name":"John","credits":30,"cars":["honda","toyota"]}`).Err()
	if err != nil {
		log.Fatal(err)
	}

	// JSON.GET
	val, err := rdb.Do(ctx, "JSON.GET", "mydoc").Result()
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println("JSON.GET result -", val)

	// JSON.ARRAPPEND
	_, err = rdb.Do(ctx, "JSON.ARRAPPEND", "mydoc", ".cars", `"audi"`).Int()
	if err != nil {
		log.Fatal(err)
	}

	fmt.Println("added audi to list of cars using JSON.ARRAPPEND")

	// JSON.GET
	val, err = rdb.Do(ctx, "JSON.GET", "mydoc").Result()
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println("lastest document", val)

	// JSON.NUMINCRBY
	_, err = rdb.Do(ctx, "JSON.NUMINCRBY", "mydoc", ".credits", 5).Result()
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println("incremented credits by 5 using JSON.NUMINCRBY")

	// JSON.STRAPPEND
	_, err = rdb.Do(ctx, "JSON.STRAPPEND", "mydoc", ".name", `", Jr."`).Int()
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println("updated name using JSON.STRAPPEND")

	// JSON.GET
	val, err = rdb.Do(ctx, "JSON.GET", "mydoc").Result()
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println("latest document", val)

	// JSON.DEL
	_, err = rdb.Do(ctx, "JSON.DEL", "mydoc").Int()
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println("deleted document using JSON.DEL")

	err = rdb.Do(ctx, "JSON.GET", "mydoc").Err()
	if errors.Is(err, redis.Nil) {
		fmt.Println("document 'mydoc' could not be found")
	}
}

To run the above code, copy it to a file named main.go and execute the following commands:

# get dependencies
go get github.com/redis/go-redis/v9

# run program
go run main.go

You should see the following output:

JSON.GET result - {"name":"John","credits":30,"cars":["honda","toyota"]}
added audi to list of cars using JSON.ARRAPPEND
latest document {"name":"John","credits":30,"cars":["honda","toyota","audi"]}
incremented credits by 5 using JSON.NUMINCRBY
updated name using JSON.STRAPPEND
latest document {"name":"John, Jr.","credits":35,"cars":["honda","toyota","audi"]}
deleted document using JSON.DEL
document 'mydoc' could not be found

Notice that we did not need to fetch the entire JSON document to update it. RedisJSON provides granular operations to work with JSON data, thereby resulting in increased application performance (reduced latency) and reduced costs (reduced network bandwidth).

RediSearch

The below example demonstrates how to use various RediSearch commands in a Go application. Note the usage redisearch-goclient.

package main

import (
	"fmt"
	"math/rand"
	"strconv"

	"github.com/RediSearch/redisearch-go/redisearch"
	"github.com/gomodule/redigo/redis"
)

var pool *redis.Pool
var client *redisearch.Client

var cities = []string{"new york", "london", "paris"}

func init() {
	pool = &redis.Pool{Dial: func() (redis.Conn, error) {
		return redis.Dial("tcp", "localhost:6379")
	}}

	client = redisearch.NewClientFromPool(pool, "user-index")
}

func main() {

	schema := redisearch.NewSchema(redisearch.DefaultOptions).
		AddField(redisearch.NewTextFieldOptions("name", redisearch.TextFieldOptions{})).
		AddField(redisearch.NewTextFieldOptions("city", redisearch.TextFieldOptions{}))

	indexDefinition := redisearch.NewIndexDefinition().AddPrefix("user:")

	client.CreateIndexWithIndexDefinition(schema, indexDefinition)

	fmt.Println("redisearch index created")

	conn := pool.Get()

	for i := 1; i <= 10; i++ {

		hashName := "user:" + strconv.Itoa(i)
		val := redis.Args{hashName}.AddFlat(map[string]string{"user_id": strconv.Itoa(i), "city": cities[rand.Intn(len(cities))]})

		conn.Do("HSET", val...)

		fmt.Println("created hash -", hashName)
	}

	docs, total, _ := client.Search(redisearch.NewQuery("*"))

	fmt.Println("no. of indexed documents =", total)

	docs, total, _ = client.Search(redisearch.NewQuery("@city:(paris|london)"))

	fmt.Println("found", total, "users in london or paris")
	for _, doc := range docs {
		fmt.Println("document ID -", doc.Id)
		fmt.Println("document attributes -", doc.Properties)
	}

	err := client.DeleteDocument("user:1")
	if err != nil {
		fmt.Println("failed to delete document (hash) user:1", err)
	}
	fmt.Println("deleted document (hash) user:1")

	_, total, _ = client.Search(redisearch.NewQuery("*"))

	fmt.Println("no. of indexed documents =", total)

	err = client.DropIndex(true)
	if err != nil {
		fmt.Println("failed to drop index", err)
	}
	fmt.Println("dropped index and documents")

	_, total, _ = client.Search(redisearch.NewQuery("*"))

	fmt.Println("no. of indexed documents =", total)
}

To run the above code, copy it to a file named main.go and execute the following command:

go get github.com/RediSearch/redisearch-go/redisearch
go get github.com/gomodule/redigo/redis

go run main.go

You should see the following output:

redisearch index created
created hash - user:1
created hash - user:2
created hash - user:3
created hash - user:4
created hash - user:5
created hash - user:6
created hash - user:7
created hash - user:8
created hash - user:9
created hash - user:10
no. of indexed documents = 10
found 7 users in london or paris
document ID - user:1
document attributes - map[city:paris user_id:1]
document ID - user:3
document attributes - map[city:paris user_id:3]
document ID - user:4
document attributes - map[city:paris user_id:4]
document ID - user:5
document attributes - map[city:london user_id:5]
document ID - user:7
document attributes - map[city:london user_id:7]
document ID - user:8
document attributes - map[city:paris user_id:8]
document ID - user:9
document attributes - map[city:london user_id:9]
deleted document (hash) user:1
no. of indexed documents = 9
dropped index and documents
no. of indexed documents = 0

We used a Hash as the document, but RediSearch also works with RedisJSON. The index is automatically updated when documents are added or deleted. This example demonstrates simple ones including fetching all documents and filtering by ORclause on the city attribute to get users who live in “london” or “paris”.

RedisTimeSeries

The below example demonstrates how to use various RedisTimeSeries commands in a Go application. Note that we are using the redistimeseries-go client.

package main

import (
	"fmt"
	"log"
	"math/rand"
	"time"

	redistimeseries "github.com/RedisTimeSeries/redistimeseries-go"
)

const tsName = "temperature:living_room"

func main() {
	client := redistimeseries.NewClient("localhost:6379", "", nil)

	// TS.CREATE
	err := client.CreateKeyWithOptions(tsName, redistimeseries.DefaultCreateOptions)
	if err != nil {
		log.Fatal(err)
	}

	fmt.Println("created time series key", tsName)

	tempOptions := []float64{24, 24.5, 25}
	var storedTS []int64

	for i := 1; i <= 10; i++ {
		// TS.ADD
		res, err := client.AddAutoTs(tsName, tempOptions[(rand.Intn(len(tempOptions)))])
		storedTS = append(storedTS, res)
		if err != nil {
			log.Fatal(err)
		}
		
		time.Sleep(1 * time.Millisecond)
	}

	fmt.Println("added time series data")

	// TS.GET
	lastDataPoint, err := client.Get(tsName)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Printf("lastest data point - timestamp: %v, temp: %v\n", lastDataPoint.Timestamp, lastDataPoint.Value)

	// TS.RANGE
	dataPoints, err := client.RangeWithOptions(tsName, storedTS[0], storedTS[len(storedTS)-1], redistimeseries.DefaultRangeOptions)

	if err != nil {
		log.Fatal(err)
	}

	fmt.Println("TS.RANGE result")

	for _, dp := range dataPoints {
		fmt.Printf("timestamp: %v, temp: %v\n", dp.Timestamp, dp.Value)
	}
}

To run the above code, copy it to a file named main.go and execute the following command:

go get github.com/RedisTimeSeries/redistimeseries-go
go run main.go

You should see the following output:

created time series key temperature:living_room
added time series data
latest data point - timestamp: 1687671178115, temp: 24
TS.RANGE result
timestamp: 1687671178097, temp: 25
timestamp: 1687671178099, temp: 24
timestamp: 1687671178100, temp: 25
timestamp: 1687671178103, temp: 25
timestamp: 1687671178106, temp: 24.5
timestamp: 1687671178108, temp: 24
timestamp: 1687671178111, temp: 24.5
timestamp: 1687671178112, temp: 25
timestamp: 1687671178114, temp: 24.5
timestamp: 1687671178115, temp: 24

Redis Modules: Best practices

This section covers few best practices to optimize for scalability and performance in real-world deployments of these modules.

RedisJSON

• Optimize JSON structure for efficient access: When designing your JSON structure, consider the access patterns of your application. Flatten the JSON hierarchy whenever possible to avoid deep nesting. This allows you to access specific fields efficiently without fetching the entire JSON document. Use RedisJSON’s path expressions to retrieve or modify specific JSON elements without parsing the entire document.
• Consider memory management and data size: RedisJSON stores JSON documents as Redis strings. Keep in mind that Redis has a maximum string size limit of 512 MB. If your JSON documents are large, consider chunking them into smaller pieces or using compression.
• You can use RedisJSON as an index for RedisSearch: This integration enables you to perform complex search operations efficiently and retrieve JSON documents based on search criteria.

RediSearch

• Carefully choose and optimize search schema: When designing your search schema, consider the specific search requirements of your application. Define the fields that need to be searchable and choose appropriate field types (e.g., text, numeric, tag). Carefully analyze the data and select the most relevant fields to include in the search index. Avoid including unnecessary fields to minimize the index size and optimize search performance.
• Leverage query optimization techniques: RedisSearch provides various query optimization techniques to improve search performance. Utilize query filters to reduce the search space and narrow down the results based on specific criteria.
• Additionally, consider using stemming, synonyms, and other language-specific analyzers to enhance search accuracy.

RedisTimeSeries

• Choose the optimal time buckets: Select the appropriate time intervals based on the frequency and granularity of your data. Consider the trade-off between storage space and query performance. Smaller buckets provide more detailed data but result in larger memory usage.
• Leverage query optimization techniques: RedisSearch provides various query optimization techniques to improve search performance. Use query filters to reduce the search space and narrow down the results.
• Use compression and down-sampling: If you have high-frequency data and limited memory resources, consider enabling compression and down-sampling configurations. This helps reduce memory usage without compromising data accuracy.

Conclusion

This article provided an introduction to some popular Redis Modules and how they are better compared to the previous solutions using core Redis data structures. It covered their commands along with demonstration of how to use them in your Go applications.

Redis modules are highly configurable, so make sure to tune them to your specific requirements and be mindful of the best practices when utilizing them in production.