---
title: "Using Redis with redux"
author: "Rich FitzJohn"
date: "2025-08-01"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Using Redis with redux}
  %\VignetteEngine{knitr::rmarkdown}
  \usepackage[utf8]{inputenc}
---



`redux` provides a full interface to the Redis API; it provides a
hiredis driver wrapped in R and uses this to expose all
199 Redis commands as a set of
user-friendly R functions that do basic error checking.

It is possible to build user-friendly applications on top of this,
for example [`storr`](https://github.com/richfitz/storr) which
provides a content-addressable object store, and
[`rrqueue`](https://github.com/traitecoevo/rrqueue) /
[`rrq`](https://github.com/mrc-ide/rrq) which implement a scalable
queuing system.

The main entry point for creating a `redis_api` object is the
`hiredis` function:

``` r
r <- redux::hiredis()
```

By default, it will connect to a database running on the local
machine (`127.0.0.1`) and port 6379.  To connect to a different
host, or to specify a password, initial database, or to use a
socket connection, use the command \code{redis_config}.

The `redis_api` object is an [`R6`](https://github.com/r-lib/R6)
class with _many_ methods, each corresponding to a different Redis
command.

``` r
r
```

```
## <redis_api>
##   Redis commands:
##     APPEND: function
##     AUTH: function
##     BGREWRITEAOF: function
##     BGSAVE: function
##     ...
##     ZSCORE: function
##     ZUNIONSTORE: function
##   Other public methods:
##     clone: function
##     command: function
##     config: function
##     initialize: function
##     pipeline: function
##     reconnect: function
##     subscribe: function
##     type: function
##     version: function
```

For example, `SET` and `GET`:

``` r
r$SET("mykey", "mydata") # set the key "mykey" to the value "mydata"
```

```
## [Redis: OK]
```

``` r
r$GET("mykey")
```

```
## [1] "mydata"
```

# Serialisation

The value for most arguments must be a string or will be coerced
into one; clearly this is not going to be suitable for most R
objects.  The solution is to *serialise* the R object.  `redux` can
accept objects serialised to strings or to byte streams, and the
functions the `object_to_bin` and `object_to_string` functions can
help here, serialising the objects to binary and string
representations.  (Alternatively you can do this yourself using
`serialize`.)

``` r
obj <- redux::object_to_bin(1:10)
obj
```

```
##   [1] 42 0a 03 00 00 00 02 04 04 00 00 05 03 00 05 00 00 00 55 54 46 2d 38 ee 00
##  [26] 00 00 02 00 00 00 01 00 00 00 09 00 04 00 0e 00 00 00 63 6f 6d 70 61 63 74
##  [51] 5f 69 6e 74 73 65 71 02 00 00 00 01 00 00 00 09 00 04 00 04 00 00 00 62 61
##  [76] 73 65 02 00 00 00 0d 00 00 00 01 00 00 00 0d 00 00 00 fe 00 00 00 0e 00 00
## [101] 00 03 00 00 00 00 00 00 00 00 00 24 40 00 00 00 00 00 00 f0 3f 00 00 00 00
## [126] 00 00 f0 3f fe 00 00 00
```

or

``` r
str <- redux::object_to_string(1:10)
str
```

```
## [1] "A\n3\n263170\n197888\n5\nUTF-8\n238\n2\n1\n262153\n14\ncompact_intseq\n2\n1\n262153\n4\nbase\n2\n13\n1\n13\n254\n14\n3\n0x1.4p+3\n0x1p+0\n0x1p+0\n254\n"
```

The binary serialisation is faster, smaller, and preserves all the
bits of floating point numbers.  The string version might be
preferable where having only strings in the database is wanted.
The binary serialisation is compatible with the same approach used
in `RcppRedis`, though it is never done automatically.

These values can be deserialised:

``` r
redux::bin_to_object(obj)
```

```
##  [1]  1  2  3  4  5  6  7  8  9 10
```

``` r
redux::string_to_object(str)
```

```
##  [1]  1  2  3  4  5  6  7  8  9 10
```

So:

``` r
r$SET("mylist", redux::object_to_bin(1:10))
```

```
## [Redis: OK]
```

``` r
r$GET("mylist")
```

```
##   [1] 42 0a 03 00 00 00 02 04 04 00 00 05 03 00 05 00 00 00 55 54 46 2d 38 ee 00
##  [26] 00 00 02 00 00 00 01 00 00 00 09 00 04 00 0e 00 00 00 63 6f 6d 70 61 63 74
##  [51] 5f 69 6e 74 73 65 71 02 00 00 00 01 00 00 00 09 00 04 00 04 00 00 00 62 61
##  [76] 73 65 02 00 00 00 0d 00 00 00 01 00 00 00 0d 00 00 00 fe 00 00 00 0e 00 00
## [101] 00 03 00 00 00 00 00 00 00 00 00 24 40 00 00 00 00 00 00 f0 3f 00 00 00 00
## [126] 00 00 f0 3f fe 00 00 00
```

``` r
redux::bin_to_object(r$GET("mylist"))
```

```
##  [1]  1  2  3  4  5  6  7  8  9 10
```

Using string serialisation is similar:

``` r
r$SET("mylist", redux::object_to_string(1:10))
```

```
## [Redis: OK]
```

``` r
r$GET("mylist")
```

```
## [1] "A\n3\n263170\n197888\n5\nUTF-8\n238\n2\n1\n262153\n14\ncompact_intseq\n2\n1\n262153\n4\nbase\n2\n13\n1\n13\n254\n14\n3\n0x1.4p+3\n0x1p+0\n0x1p+0\n254\n"
```

``` r
redux::string_to_object(r$GET("mylist"))
```

```
##  [1]  1  2  3  4  5  6  7  8  9 10
```

This gives you all the power of Redis, but you will have to
manually serialise/deserialise all complicated R objects (i.e.,
everything other than logicals, numbers or strings).  Similarly,
you are responsible for type coercion/deserialisation when
retrieving data at the other end.

Note that you are not restricted to using serialised R objects as
values; you can use them as keys; this is perfectly valid:

``` r
r$SET(redux::object_to_bin(1:10), "mydata")
```

```
## [Redis: OK]
```

``` r
r$GET(redux::object_to_bin(1:10))
```

```
## [1] "mydata"
```


Beyond `GET` / `SET` / `DEL`, Redis offers potentially better ways
of holding things like lists using its native data types.  For
example;


``` r
r$RPUSH("mylist2", 1:10)
```

```
## [1] 10
```

(the returned value `10` indicates that the list "mylist2" is 10
elements long).  There are lots of commands for operating on lists.
For example, you can do things like;

* get an element by its index (note that this uses C-style base-0
indexing for consistency with the `Redis` documentation rather than
R's semantics)

``` r
r$LINDEX("mylist2", 1)
```

```
## [1] "2"
```

* set an element by its index

``` r
r$LSET("mylist2", 1, "carrot")
```

```
## [Redis: OK]
```

* get all of a list:

``` r
r$LRANGE("mylist2", 0, -1)
```

```
## [[1]]
## [1] "1"
##
## [[2]]
## [1] "carrot"
##
## [[3]]
## [1] "3"
##
## [[4]]
## [1] "4"
##
## [[5]]
## [1] "5"
##
## [[6]]
## [1] "6"
##
## [[7]]
## [1] "7"
##
## [[8]]
## [1] "8"
##
## [[9]]
## [1] "9"
##
## [[10]]
## [1] "10"
```

* or part of it:

``` r
r$LRANGE("mylist2", 0, 2)
```

```
## [[1]]
## [1] "1"
##
## [[2]]
## [1] "carrot"
##
## [[3]]
## [1] "3"
```

* pop elements off the front or back

``` r
r$LLEN("mylist2")
```

```
## [1] 10
```

``` r
r$LPOP("mylist2")
```

```
## [1] "1"
```

``` r
r$RPOP("mylist2")
```

```
## [1] "10"
```

``` r
r$LLEN("mylist2")
```

```
## [1] 8
```

Of course, each element of the list can be an R object if you run
it through `object_to_string`:

``` r
r$LPUSH("mylist2", redux::object_to_string(1:10))
```

```
## [1] 9
```

but you'll be responsible for converting that back (and detecting
/ knowing that this needs doing)

``` r
dat <- r$LRANGE("mylist2", 0, 2)
dat
```

```
## [[1]]
## [1] "A\n3\n263170\n197888\n5\nUTF-8\n238\n2\n1\n262153\n14\ncompact_intseq\n2\n1\n262153\n4\nbase\n2\n13\n1\n13\n254\n14\n3\n0x1.4p+3\n0x1p+0\n0x1p+0\n254\n"
##
## [[2]]
## [1] "carrot"
##
## [[3]]
## [1] "3"
```

``` r
dat[[1]] <- redux::string_to_object(dat[[1]])
dat
```

```
## [[1]]
##  [1]  1  2  3  4  5  6  7  8  9 10
##
## [[2]]
## [1] "carrot"
##
## [[3]]
## [1] "3"
```

As with all functions in the `redis_api` object, all functions and
their arguments are described in the Redis documentation.

# Pipelining

Every command set to Redis costs a round trip; even over the
loopback interface this can be expensive if done a very large
number of times.  Redis offers two ways of minimising this problem;
pipelining and lua scripting.  redux supports both.

To pipeline, use the `pipeline` method of the `hiredis` object:

``` r
redis <- redux::redis
r$pipeline(
  redis$PING(),
  redis$PING())
```

```
## [[1]]
## [Redis: PONG]
##
## [[2]]
## [Redis: PONG]
```

Here, `redis` is a special object within the package that
implements all the Redis commands but only formats them for use
rather than sends them.  The `pipeline` method collects these all
up and sends them to the server in a single batch, with the result
returned as a list.

If arguments are named, then the return value is named:

``` r
r$pipeline(
  a = redis$INCR("x"),
  b = redis$INCR("x"),
  c = redis$DEL("x"))
```

```
## $a
## [1] 1
##
## $b
## [1] 2
##
## $c
## [1] 1
```

here a variable "x" was incremented twice and then deleted.

If you use pipelining you should read the Redis page on
it (`http://redis.io/topics/pipelining`) because there are a few
restrictions and cautions.

Generating very large numbers (or variable numbers) of commands
with the above interface will be difficult because `pipeline` uses
the dots argument.  Instead, you can pass a list of commands to the
`.commands` argument of `pipeline`:

``` r
cmds <- lapply(seq_len(4), function(.) redis$PING())
r$pipeline(.commands = cmds)
```

```
## [[1]]
## [Redis: PONG]
##
## [[2]]
## [Redis: PONG]
##
## [[3]]
## [Redis: PONG]
##
## [[4]]
## [Redis: PONG]
```

# Subscriptions

On top of the key/value store aspect of Redis, it also offers a
publisher/subscriber model.  Publishing with `redux` is
straightforward; use the `PUBLISH` method:

``` r
r$PUBLISH("mychannel", "hello")
```

```
## [1] 0
```

The return value here is the number of subscribers to that channel;
in our case zero!

The `SUBSCRIBE` method should not be used as the client cannot deal
with messages directly (it is disabled in the interface to prevent
this).

Instead, use the `subscribe` (lower case) method.  This takes arguments:

* `channel`: name or pattern of the channel/s to subscribe to
  (scalar or vector).

* `transform`: A function that takes each message and processes it.
  Messages are R lists with elements: `type`, `pattern` (if a
  pattern was used), `channel` and `value` (see the Redis docs).
  Your transform function can turn this into anything it wants, and
  may have side effects such as printing to the screen, writing to
  a file, etc.

* `terminate`: A termination criterion.  given a *transformed*
  message (i.e., the result of `transform(x)`) return `TRUE` if
  we're processing messages.  Optional, but if not used set `n` to
  a finite number.

* collect: logical indicating if *transformed* messages should be
  collected and returned on exit.

* n: maximum number of messages to collect; once `n` messages have
  been collected we will terminate regardless of `terminate`.

* pattern: logical indicating if `channel` should be interpreted as
 a pattern.

* envir: environment in which to evaluate `transform` and `terminate`.

That all sounds a lot more complicated it really is.  To collect
all messages on the `"mychannel"` channel, stopping after 100
messages or a message reading exactly "goodbye" you would write:

``` r
res <- r$subscribe("mychannel",
                   transform = function(x) x$value,
                   terminate = function(x) identical(x, "goodbye"),
                   n = 100)
```

*NOTE*: you need to be careful here - `hiredis` internally uses a
blocking read which cannot be interrupted with Ctrl-C once started
unless a message is received on the channels being listened to!

To test this out, we need a second process that will publish to the
channel (or we'll wait forever).  This function will publish the
first 20 values out of the Nile data set.


```r
r <- redux::hiredis()
for (i in Nile[1:20]) {
  Sys.sleep(.05)
  r$PUBLISH("mychannel", i)
}
r$PUBLISH("mychannel", "goodbye")
```

This file is at `path_to_publisher` (in R's temporary directory)
and can be run with:

``` r
system2(file.path(R.home("bin"), "Rscript"), path_to_publisher,
        wait = FALSE, stdout = FALSE, stderr = FALSE)
```

to start the publisher.

Let's add a little debugging information to the transform function,
and set the subscriber off:

``` r
transform <- function(x) {
  message(format(Sys.time(), "%Y-%m-%d %H:%M:%OS3"),
          ": got message: ",
          x$value)
  x$value
}
```

``` r
res <- r$subscribe("mychannel",
                   transform = transform,
                   terminate = function(x) identical(x, "goodbye"),
                   n = 100)
```

```
## 2025-08-01 11:10:27.861: got message: 1120
```

```
## 2025-08-01 11:10:27.921: got message: 1160
```

```
## 2025-08-01 11:10:27.963: got message: 963
```

```
## 2025-08-01 11:10:28.014: got message: 1210
```

```
## 2025-08-01 11:10:28.065: got message: 1160
```

```
## 2025-08-01 11:10:28.116: got message: 1160
```

```
## 2025-08-01 11:10:28.167: got message: 813
```

```
## 2025-08-01 11:10:28.218: got message: 1230
```

```
## 2025-08-01 11:10:28.269: got message: 1370
```

```
## 2025-08-01 11:10:28.320: got message: 1140
```

```
## 2025-08-01 11:10:28.371: got message: 995
```

```
## 2025-08-01 11:10:28.422: got message: 935
```

```
## 2025-08-01 11:10:28.473: got message: 1110
```

```
## 2025-08-01 11:10:28.524: got message: 994
```

```
## 2025-08-01 11:10:28.574: got message: 1020
```

```
## 2025-08-01 11:10:28.624: got message: 960
```

```
## 2025-08-01 11:10:28.675: got message: 1180
```

```
## 2025-08-01 11:10:28.726: got message: 799
```

```
## 2025-08-01 11:10:28.777: got message: 958
```

```
## 2025-08-01 11:10:28.828: got message: 1140
```

```
## 2025-08-01 11:10:28.828: got message: goodbye
```



The timestamps in the printed output show when the message was
received (with fractional seconds so that this is more obvious
since this only takes ~1s to complete).

The `res` object contains all the values, including the "goodbye"
that was our end-of-stream message:

``` r
unlist(res)
```

```
##  [1] "1120"    "1160"    "963"     "1210"    "1160"    "1160"    "813"
##  [8] "1230"    "1370"    "1140"    "995"     "935"     "1110"    "994"
## [15] "1020"    "960"     "1180"    "799"     "958"     "1140"    "goodbye"
```

# Using commands that are not yet supported

The command set of `redux` is complete as of version 4.0 of redis, which is now quite old.  We will expand this soon, but you can use any new command by writing a simple wrapper in R.

To demonstrate this, we look at the command for `HSET`, using the `redis` object that can be used for pipelining, because this does all the actual work of assembling commands


``` r
redis$HSET
```

```
## function (key, field, value)
## {
##     assert_scalar2(key)
##     assert_scalar2(field)
##     assert_scalar2(value)
##     command(list("HSET", key, field, value))
## }
## <bytecode: 0x55eea49039e0>
## <environment: 0x55eea48423d0>
```

The definition here

* checks that `key`, `field` and `value` are scalars (counting a `raw` vector as length1)
* builds a command by constructing a list of strings (the literal `HSET` followed by `key`, `field` and `value`)
* passes that to redis via `command()`


``` r
redis$HSET("myhash", "myfield", "myvalue")
```

```
## [[1]]
## [1] "HSET"
##
## [[2]]
## [1] "myhash"
##
## [[3]]
## [1] "myfield"
##
## [[4]]
## [1] "myvalue"
```

On a real redis server then:


``` r
r <- redux::hiredis()
r$HSET("myhash", "myfield", "myvalue")
```

```
## [1] 1
```

``` r
r$HGET("myhash", "myfield")
```

```
## [1] "myvalue"
```

We can do this manually using `r$command()` and passing a list:


``` r
r$command(list("HSET", "myhash", "myfield", "myvalue2"))
```

```
## [1] 0
```

We can see this has worked by fetching our value


``` r
r$HGET("myhash", "myfield")
```

```
## [1] "myvalue2"
```

or, by analogy to the above



``` r
r$command(list("HGET", "myhash", "myfield"))
```

```
## [1] "myvalue2"
```

With this approach, you can use any new command, even before we release a new version.

# Potential applications

Because `redux` exposes all of Redis, you can roll your own data
structures.

First, a generator object that sets up a new list at `key` within
the database `r`.

``` r
rlist <- function(..., key = "rlist", r = redux::hiredis()) {
  dat <- vapply(c(...), redux::object_to_string, character(1))
  r$RPUSH(key, dat)
  ret <- list(r = r, key = key)
  class(ret) <- "rlist"
  ret
}
```

Then some S3 methods that work with this object.  I've only
implemented `length` and `[[`, but `[` would be useful here too as
would `print`.

``` r
length.rlist <- function(x) {
  x$r$LLEN(x$key)
}

`[[.rlist` <- function(x, i, ...) {
  redux::string_to_object(x$r$LINDEX(x$key, i - 1L))
}

`[[<-.rlist` <- function(x, i, value, ...) {
  x$r$LSET(x$key, i - 1L, redux::object_to_string(value))
  x
}
```

Then we have this weird object we can add things to.

``` r
obj <- rlist(1:10)
length(obj) # 10
```

```
## [1] 10
```

``` r
obj[[3]]
```

```
## [1] 3
```

``` r
obj[[3]] <- "an element"
obj[[3]]
```

```
## [1] "an element"
```

The object has reference semantics so that assignment does *not* make a copy:

``` r
obj2 <- obj
obj2[[2]] <- obj2[[2]] * 2
obj[[2]] == obj2[[2]]
```

```
## [1] TRUE
```

For a better version of this, see
[storr](https://github.com/richfitz/storr) which does similar things to implement "[indexable serialisation](http://htmlpreview.github.io/?https://raw.githubusercontent.com/richfitz/storr/master/inst/doc/storr.html#lists-and-indexable-serialisation)"

# Scripts

Redis allows storing and evaluating Lua scripts on the redis server.
At this point it's all getting a bit meta (using R to tell Redis to
call another dynamic language that drives Redis) but this can be
very useful - especially in avoiding race conditions (because a
script is atomic) and avoiding roundtrips.

Describing how to write Lua scripts is out of scope for this
document but is a bit fiddly.  Here is a trivial one that returns
the value of a key:

``` r
r$SET("key", "a")
```

```
## [Redis: OK]
```

``` r
res <- r$EVAL("return redis.call('get', 'key')", 1L, "key", NULL)
```

This can also be run by pushing the script into Redis and referring
to it by SHA:

``` r
sha <- r$SCRIPT_LOAD("return redis.call('get', 'key')")
r$SCRIPT_EXISTS(sha)
```

```
## [[1]]
## [1] 1
```

and calling it like so:

``` r
r$EVALSHA(sha, 1, "key", NULL)
```

```
## [1] "a"
```

A more interesting example, setting, incrementing and getting a key
(this is all do-able with redis commands)

``` r
lua <- '
  local keyname = KEYS[1]
  local value = ARGV[1]
  redis.call("SET", keyname, value)
  redis.call("INCR", keyname)
  return redis.call("GET", keyname)'
```

With the `redis_scripts` wrapper you can give friendly names to a script:

``` r
obj <- redux::redis_scripts(r, set_and_incr = lua)
```

And then call them by name:

``` r
res <- obj("set_and_incr", "foo", "10")
res
```

```
## [1] "11"
```

# Getting help

Because the interface `redux` uses is simply a wrapper around the
Redis API, the main source of documentation is the Redis help
itself at `http://redis.io`
