One place for hosting & domains

      Manage

      How To Manage Replicas and Clients in Redis


      Introduction

      Redis is an open-source, in-memory key-value data store. One of its most sought-after features is its support for replication: any Redis server can replicate its data to any number of replicas, allowing for high read scalability and strong data redundancy. Additionally, Redis was designed to allow many clients (up to 10000, by default) to connect and interact with data, making it a good choice for cases where many users need access to the same dataset.

      This tutorial goes over the commands used to manage Redis clients and replicas.

      How To Use This Guide
      This guide is written as a cheat sheet with self-contained examples. We encourage you to jump to any section that is relevant to the task you’re trying to complete.

      The commands shown in this guide were tested on an Ubuntu 18.04 server running Redis version 4.0.9. To set up a similar environment, you can follow Step 1 of our guide on How To Install and Secure Redis on Ubuntu 18.04. We will demonstrate how these commands behave by running them with redis-cli, the Redis command line interface. Note that if you’re using a different Redis interface — Redli, for example — the exact output of certain commands may differ.

      Alternatively, you could provision a managed Redis database instance to test these commands, but note that depending on the level of control allowed by your database provider, some commands in this guide may not work as described. To provision a DigitalOcean Managed Database, follow our Managed Databases product documentation. Then, you must either install Redli or set up a TLS tunnel in order to connect to the Managed Database over TLS.

      Note: The Redis project uses the terms “master” and “slave” in its documentation and in various commands to identify different roles in replication, though the project’s contributors are taking steps to change this language in cases where it doesn’t cause compatibility issues. DigitalOcean generally prefers to use the alternative terms “primary” and “replica”.

      This guide will default to “primary” and “replica” whenever possible, but note that there are a few instances where the terms “master” and “slave” unavoidably come up.

      Managing Replicas

      One of Redis’s most distinguishing features is its built-in replication. When using replication, Redis creates exact copies of the primary instance. These secondary instances reconnect to the primary any time their connections break and will always aim to remain an exact copy of the primary.

      If you’re not sure whether the Redis instance you’re currently connected to is a primary instance or a replica, you can check by running the role command:

      This command will return either master or slave, or potentially sentinel if you’re using Redis Sentinel.

      To designate a Redis instance as a replica of another instance on the fly, run the replicaof command. This command takes the intended primary server’s hostname or IP address and port as arguments:

      • replicaof hostname_or_IP port

      If the server was already a replica of another primary, it will stop replicating the old server and immediately start synchronizing with the new one. It will also discard the old dataset.

      To promote a replica back to being a primary, run the following replicaof command:

      This will stop the instance from replicating the primary server, but will not discard the dataset it has already replicated. This syntax is useful in cases where the original primary fails. After running replicaof no one on a replica of the failed primary, the former replica can be used as the new primary and have its own replicas as a failsafe.

      Note: Prior to version 5.0.0, Redis instead included a version of this command named slaveof.

      Managing Clients

      A client is any machine or software that connects to a server in order to access a service. Redis comes with several commands that help with tracking and managing client connections.

      The client list command returns a set of human-readable information about current client connections:

      Output

      "id=18165 addr=[2001:db8:0:0::12]:47460 fd=7 name=jerry age=72756 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping id=18166 addr=[2001:db8:0:1::12]:47466 fd=8 name= age=72755 idle=5 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info id=19381 addr=[2001:db8:0:2::12]:54910 fd=9 name= age=9 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=26 qbuf-free=32742 obl=0 oll=0 omem=0 events=r cmd=client "

      Here is what each of these fields mean:

      • id: a unique 64-bit client ID
      • name: the name of the client connection, as defined by a prior client setname command
      • addr: the address and port from which the client is connecting
      • fd: the file descriptor that corresponds to the socket over which the client is connecting
      • age: the total duration of the client connection, in seconds
      • flags: a set of one or more single-character flags that provide more granular detail about the clients; see the client list command documentation for more details
      • db: the current database ID number that the client is connected to (can be from 0 to 15)
      • sub: the number of channels the client is subscribed to
      • psub: the number of the client’s pattern-matching subscriptions
      • mutli: the number of commands the client has queued in a transaction (will show -1 if the client hasn’t begun a transaction or 0 if it has only started a transaction and not queued any commands)
      • qbuf: the client’s query buffer length, with 0 meaning it has no pending queries
      • qbuf-free: the amount of free space in the client’s query buffer, with 0 meaning that the query buffer is full
      • obl: the client’s output buffer length
      • oll: the length of the client’s output list, where replies are queued when its buffer is full
      • omem: the memory used by the client’s output buffer
      • events: the client’s file descriptor events, these can be r for “readable”, w for “writable,” or both
      • cmd: the last command run by the client

      Setting client names is useful for debugging connection leaks in whatever application is using Redis. Every new connection starts without an assigned name, but client setname can be used to create one for the current client connection. There’s no limit to how long client names can be, although Redis usually limits string lengths to 512 MB. Note, though, that client names cannot include spaces:

      To retrieve the name of a client connection, use the client getname command:

      Output

      "elaine"

      To retrieve a client’s connection ID, use the client id command:

      Output

      (integer) "19492"

      Redis client IDs are never repeated and are monotonically incremental. This means that if one client has an ID greater than another, then it was established at a later time.

      Blocking Clients and Closing Client Connections

      Replication systems are typically described as being either synchronous or asynchronous. In synchronous replication, whenever a client adds or changes data it must receive some kind of acknowledgement from a certain number of replicas for the change to register as having been committed. This helps to prevent nodes from having data conflicts but it comes at a cost of latency, since the client must wait to perform another operation until it has heard back from a certain number of replicas.

      In asynchronous replication, on the other hand, the client sees a confirmation that the operation is finished as soon as the data is written to local storage. There can, however, be a lag between this and when the replicas actually write the data. If one of the replicas fails before it can write the change, that write will be lost forever. So while asynchronous replication allows clients to continue performing operations without the latency caused by waiting for replicas, it can lead to data conflicts between nodes and may require extra work on the part of the database administrator to resolve those conflicts.

      Because of its focus on performance and low latency, Redis implements asynchronous replication by default. However, you can simulate synchronous replication with the wait command. wait blocks the current client connection for a specified amount of time (in milliseconds) until all the previous write commands are successfully transferred and accepted by a specified number of replicas. This command uses the following syntax:

      • wait number_of_replicas number_of_milliseconds

      For example, if you want to block your client connection until all the previous writes are registered by at least 3 replicas within a 30 millisecond timeout, your wait syntax would look like this:

      The wait command returns an integer representing the number of replicas that acknowledged the write commands, even if not every replica does so:

      Output

      2

      To unblock a client connection that has been previously blocked, whether from a wait, brpop, or xread command, you can run a client unblock command with the following syntax:

      To temporarily suspend every client currently connected to the Redis server, you can use the client pause command. This is useful in cases where you need to make changes to your Redis setup in a controlled way. For example, if you’re promoting one of your replicas to be the primary instance, you might pause every client beforehand so you can promote the replica and have the clients connect to it as the new primary without losing any write operations in the process.

      The client pause command requires you to specify the amount of time (in milliseconds) you’d like to suspend the clients. The following example will suspend all clients for one second:

      The client kill syntax allows you to close a single connection or a set of specific connections based on a number of different filters. The syntax looks like this:

      • client kill filter_1 value_1 ... filter_n value_n

      In Redis versions 2.8.12 and later, the following filters are available:

      • addr: allows you to close a client connection from a specified IP address and port
      • client-id: allows you to close a client connection based on its unique ID field
      • type: closes every client of a given type, which can be either normal, master, slave, or pubsub
      • skipme: the value options for this filter are yes and no:
        • if no is specified, the client calling the client kill command will not get skipped, and will be killed if the other filters apply to it
        • if yes is specified, the client running the command will be skipped and the kill command will have no effect on the client. skipme is always yes by default

      Conclusion

      This guide details a number of commands used to manage Redis clients and replicas. If there are other related commands, arguments, or procedures you’d like to see outlined in this guide, please ask or make suggestions in the comments below.

      For more information on Redis commands, see our tutorial series on How to Manage a Redis Database.



      Source link

      How To Manage Sets in Redis


      Introduction

      Redis is an open-source, in-memory key-value data store. Sets in Redis are collections of strings stored at a given key. When held in a set, an individual record value is referred to as a member. Unlike lists, sets are unordered and do not allow repeated values.

      This tutorial explains how to create sets, retrieve and remove members, and compare the members of different sets.

      How To Use This Guide
      This guide is written as a cheat sheet with self-contained examples. We encourage you to jump to any section that is relevant to the task you’re trying to complete.

      The commands and outputs shown in this guide were tested on an Ubuntu 18.04 server running Redis version 4.0.9. To obtain a similar setup, you can follow Step 1 of our guide on How To Install and Secure Redis on Ubuntu 18.04. We will demonstrate how these commands behave by running them with redis-cli, the Redis command line interface. Note that if you’re using a different Redis interface — Redli, for example — the exact outputs of certain commands may differ.

      Alternatively, you could provision a managed Redis database instance to test these commands, but note that depending on the level of control allowed by your database provider, some commands in this guide may not work as described. To provision a DigitalOcean Managed Database, follow our Managed Databases product documentation. Then, you must either install Redli or set up a TLS tunnel in order to connect to the Managed Database over TLS.

      Creating Sets

      The sadd command allows you to create a set and add one or more members to it. The following example will create a set at a key named key_horror with the members "Frankenstein" and "Godzilla":

      • sadd key_horror "Frankenstein" "Godzilla"

      If successful, sadd will return an integer showing how many members it added to the set:

      Output

      (integer) 2

      If you try adding members of a set to a key that’s already holding a non-set value, it will return an error. The first command in this block creates a list named key_action with one element, "Shaft". The next command tries to add a set member, "Shane", to the list, but this produces an error because of the clashing data types:

      • rpush key_action "Shaft"
      • sadd key_action "Shane"

      Output

      (error) WRONGTYPE Operation against a key holding the wrong kind of value

      Note that sets don’t allow more than one occurrence of the same member:

      • sadd key_comedy "It's" "A" "Mad" "Mad" "Mad" "Mad" "Mad" "World"

      Output

      (integer) 4

      Even though this saddcommand specifies eight members, it discards four of the duplicate "Mad" members resulting in a set size of 4.

      Retrieving Members from Sets

      In this section, we’ll go over a number of Redis commands that return information about the members held in a set. To practice the commands outlined here, run the following command, which will create a set with six members at a key called key_stooges:

      • sadd key_stooges "Moe" "Larry" "Curly" "Shemp" "Joe" "Curly Joe"

      To return every member from a set, run the smembers command followed by the key you want to inspect:

      Output

      1) "Curly" 2) "Moe" 3) "Larry" 4) "Shemp" 5) "Curly Joe" 6) "Joe"

      To check if a specific value is a member of a set, use the sismember command:

      • sismember key_stooges "Harpo"

      If the element "Harpo" is a member of the key_stooges set, sismember will return 1. Otherwise, it will return 0:

      Output

      (integer) 0

      To see how many members are in a given set (in other words, to find the cardinality of a given set), run scard:

      Output

      (integer) 6

      To return a random element from a set, run srandmember:

      Output

      "Larry"

      To return multiple random, distinct elements from a set, you can follow the srandmember command with the number of elements you want to retrieve:

      • srandmember key_stooges 3

      Output

      1) "Larry" 2) "Moe" 3) "Curly Joe"

      If you pass a negative number to srandmember, the command is allowed to return the same element multiple times:

      • srandmember key_stooges -3

      Output

      1) "Shemp" 2) "Curly Joe" 3) "Curly Joe"

      The random element function used in srandmember is not perfectly random, although its performance improves in larger data sets. See the command’s official documentation for more details.

      Removing Members from Sets

      Redis comes with three commands used to remove members from a set: spop, srem, and smove.

      spop randomly selects a specified number of members from a set and returns them, similar to srandmember, but then deletes them from the set. It accepts the name of the key containing a set and the number of members to remove from the set as arguments. If you don’t specify a number, spop will default to returning and removing a single value.

      The following example command will remove and return two randomly-selected elements from the key_stooges set created in the previous section:

      Output

      1) "Shemp" 2) "Larry"

      srem allows you to remove one or more specific members from a set, rather than random ones:

      • srem key_stooges "Joe" "Curly Joe"

      Instead of returning the members removed from the set, srem returns an integer showing how many members were removed:

      Output

      (integer) 2

      Use smove to move a member from one set to another. This command accepts as arguments the source set, the destination set, and the member to move, in that order. Note that smove only allows you to move one member at a time:

      • smove key_stooges key_jambands "Moe"

      If the command moves the member successfully, it will return (integer) 1:

      Output

      (integer) 1

      If smove fails, it will instead return (integer) 0. Note that if the destination key does not already exist, smove will create it before moving the member into it.

      Comparing Sets

      Redis also provides a number of commands that find the differences and similarities between sets. To demonstrate how these work, this section will reference three sets named presidents, kings, and beatles. If you’d like to try out the commands in this section yourself, create these sets and populate them using the following sadd commands:

      • sadd presidents "George" "John" "Thomas" "James"
      • sadd kings "Edward" "Henry" "John" "James" "George"
      • sadd beatles "John" "George" "Paul" "Ringo"

      sinter compares different sets and returns the set intersection, or values that appear in every set:

      • sinter presidents kings beatles

      Output

      1) "John" 2) "George"

      sinterstore performs a similar function, but instead of returning the intersecting members it creates a new set at the specified destination containing these intersecting members. Note that if the destination already exists, sinterstore will overwrite its contents:

      • sinterstore new_set presidents kings beatles
      • smembers new_set

      Output

      1) "John" 2) "George"

      sdiff returns the set difference — members resulting from the difference of the first specified set from each of the following sets:

      • sdiff presidents kings beatles

      Output

      1) "Thomas"

      In other words, sdiff looks at each member in the first given set and then compares those to members in each successive set. Any member in the first set that also appears in the following sets is removed, and sdiff returns the remaining members. Think of it as removing members of subsequent sets from the first set.

      sdiffstore performs a function similar to sdiff, but instead of returning the set difference it creates a new set at a given destination, containing the set difference:

      • sdiffstore new_set beatles kings presidents
      • smembers new_set

      Output

      1) "Paul" 2) "Ringo"

      Like sinterstore, sdiffstore will overwrite the destination key if it already exists.

      sunion returns the set union, or a set containing every member of every set you specify:

      • sunion presidents kings beatles

      Output

      1) "Thomas" 2) "George" 3) "Paul" 4) "Henry" 5) "James" 6) "Edward" 7) "John" 8) "Ringo"

      sunion treats the results like a new set in that it only allows one occurrence of any given member.

      sunionstore performs a similar function, but creates a new set containing the set union at a given destination instead of just returning the results:

      • sunionstore new_set presidents kings beatles

      Output

      (integer) 8

      As with sinterstore and sdiffstore, sunionstore will overwrite the destination key if it already exists.

      Conclusion

      This guide details a number of commands used to create and manage sets in Redis. If there are other related commands, arguments, or procedures you’d like to see outlined in this guide, please ask or make suggestions in the comments below.

      For more information on Redis commands, see our tutorial series on How to Manage a Redis Database.



      Source link

      How To Manage Redis Databases and Keys


      Introduction

      Redis is an open-source, in-memory key-value data store. A key-value data store is a type of NoSQL database in which keys serve as unique identifiers for their associated values. Any given Redis instance includes a number of databases, each of which can hold many different keys of a variety of data types. In this tutorial, we will go over how to select a database, move keys between databases, and manage and delete keys.

      How To Use This Guide
      This guide is written as a cheat sheet with self-contained examples. We encourage you to jump to any section that is relevant to the task you’re trying to complete.

      The commands and outputs shown in this guide were tested on an Ubuntu 18.04 server running Redis version 4.0.9. To obtain a similar setup, you can follow Step 1 of our guide on How To Install and Secure Redis on Ubuntu 18.04. We will demonstrate how these commands behave by running them with redis-cli, the Redis command line interface. Note that if you’re using a different Redis interface — Redli, for example — the exact outputs of certain commands may differ.

      Alternatively, you could provision a managed Redis database instance to test these commands, but note that depending on the level of control allowed by your database provider, some commands in this guide may not work as described. To provision a DigitalOcean Managed Database, follow our Managed Databases product documentation. Then, you must either install Redli or set up a TLS tunnel in order to connect to the Managed Database over TLS.

      Managing Databases

      Out of the box, a Redis instance supports 16 logical databases. These databases are effectively siloed off from one another, and when you run a command in one database it doesn’t affect any of the data stored in other databases in your Redis instance.

      Redis databases are numbered from 0 to 15 and, by default, you connect to database 0 when you connect to your Redis instance. However, you can change the database you’re using with the select command after you connect:

      If you’ve selected a database other than 0, it will be reflected in the redis-cli prompt:

      To swap all the data held in one database with the data held in another, use the swapdb command. The following example will swap the data held in database 6 with that in database 8, and any clients connected to either database will be able to see changes immediately:

      swapdb will return OK if the swap is successful.

      If you want to move a key to a different Redis instance, you can run migrate. This command ensures the key exists on the target instance before deleting it from the source instance. When you run migrate, the command must include the following elements in this order:

      • The hostname or IP address of the destination database
      • The target database’s port number
      • The name of the key you want to migrate
      • The database number where you want to store the key on the destination instance
      • A timeout, in milliseconds, which defines the maximum amount of idle communication time between the two machines. Note that this isn’t a time limit for the operation, just that the operation should always make some level of progress within the defined length of time

      To illustrate:

      • migrate 203.0.113.0 6379 key_1 7 8000

      Additionally, migrate allows the following options which you can add after the timeout argument:

      • COPY: Specifies that the key should not be deleted from the source instance
      • REPLACE: Specifies that if the key already exists on the destination, the migrate operation should delete and replace it
      • KEYS: Instead of providing a specific key to migrate, you can enter an empty string ("") and then use the syntax from the keys command to migrate any key that matches a pattern. For more information on how keys works, see our tutorial on How To Troubleshoot Issues in Redis.

      Managing Keys

      There are a number of Redis commands that are useful for managing keys regardless of what type of data they hold. We’ll go over a few of these in this section.

      rename will rename the specified key. If it’s successful, it will return OK:

      You can use randomkey to return a random key from the currently selected database:

      Output

      "any_key"

      Use type to determine what type of data the given key holds. This command’s output can be either string, list, hash, set, zset, or stream:

      Output

      "string"

      If the specified key doesn’t exist, type will return none instead.

      You can move an individual key to another database in your Redis instance with the move command. move takes the name of a key and the database where you want to move the key as arguments. For example, to move the key key_1 to database 8, you would run the following:

      move will return OK if moving the key was successful.

      Deleting Keys

      To delete one or more keys of any data type, use the del command followed by one or more keys that you want to delete:

      If this command deletes the key(s) successfully it will return (integer) 1. Otherwise, it will return (integer) 0.

      The unlink command performs a similar function as del, with the difference being that del blocks the client as the server reclaims the memory taken up by the key. If the key being deleted is associated with a small object, the amount of time it takes for del to reclaim the memory is very small and the blocking time may not even be noticeable.

      However, it can become inconvenient if, for example, the key you’re deleting is associated with many objects, such as a hash with thousands or millions of fields. Deleting such a key can take a noticeably long time, and you’ll be blocked from performing any other operations until it’s fully removed from the server’s memory.

      unlink, however, first determines the cost of deallocating the memory taken up by the key. If it’s small then unlink functions the same way as del by the key immediately while also blocking the client. However, if there’s a high cost to deallocate memory for a key, unlink will delete the key asynchronously by creating another thread and incrementally reclaim memory in the background without blocking the client:

      Since it runs in the background, it’s generally recommended that you use unlink to remove keys from your server to reduce errors on your clients, though del will also suffice in many cases.

      Warning: The following two commands are considered dangerous. The flushdb and flushall commands will irreversibly delete all the keys in a single database and all the keys in every database on the Redis server, respectively. We recommend that you only run these commands if you are absolutely certain that you want to delete all the keys in your database or server.

      It may be in your interest to rename these commands to something with a lower likelihood of being run accidentally.

      To delete all the keys in the selected database, use the flushdb command:

      To delete all the keys in every database on a Redis server (including the currently selected database), run flushall:

      Both flushdb and flushall accept the async option, which allows you to delete all the keys on a single database or every database in the cluster asynchronously. This allows them to function similarly to the unlink command, and they will create a new thread to incrementally free up memory in the background.

      Backing Up Your Database

      To create a backup of the currently selected database, you can use the save command:

      This will export a snapshot of the current dataset as an .rdb file, which is a database dump file that holds the data in an internal, compressed serialization format.

      save runs synchronously and will block any other clients connected to the database. Hence, the save command documentation recommends that this command should almost never be run in a production environment. Instead, it suggests using the bgsave command. This tells Redis to fork the database: the parent will continue to serve clients while the child process saves the database before exiting:

      Note that if clients add or modify data while the bgsave operation is occurring, these changes won’t be captured in the snapshot.

      You can also edit the Redis configuration file to have Redis save a snapshot automatically (known as snapshotting or RDB mode) after a certain amount of time if a minimum number of changes were made to the database. This is known as a save point. The following save point settings are enabled by default in the redis.conf file:

      /etc/redis/redis.conf

      . . .
      save 900 1
      save 300 10
      save 60 10000
      . . .
      dbfilename "nextfile.rdb"
      . . .
      

      With these settings, Redis will export a snapshot of the database to the file defined by the dbfilename parameter every 900 seconds if at least 1 key is changed, every 300 seconds if at least 10 keys are changed, and every 60 seconds if at least 10000 keys are changed.

      You can use the shutdown command to back up your Redis data and then close your connection. This command will block every client connected to the database and then perform a save operation if at least one save point is configured, meaning that it will export the database in its current state to an .rdb file while preventing clients from making any changes.

      Additionally, the shutdown command will flush changes to Redis’s append-only file before quitting if append-only mode is enabled. The append-only file mode (AOF) involves creating a log of every write operation on the server in a file ending in .aof after every snapshot. AOF and RDB modes can be enabled on the same server, and using both persistence methods is an effective way to back up your data.

      In short, the shutdown command is essentially a blocking save command that also flushes all recent changes to the append-only file and closes the connection to the Redis instance:

      Warning: The shutdown command is considered dangerous. By blocking your Redis server’s clients, you can make your data unavailable to users and applications that depend on it. We recommend that you only run this command if you are testing out Redis’s behavior or you are absolutely certain that you want to block all your Redis server’s clients.

      In fact, it may be in your interest to rename this command to something with a lower likelihood of being run accidentally.

      If you’ve not configured any save points but still want Redis to perform a save operation, append the save option to the `shutdown command:

      If you have configured at least one save point but you want to shut down the Redis server without performing a save, you can add the nosave argument to the command:

      Note that the append-only file can grow to be very long over time, but you can configure Redis to rewrite the file based on certain variables by editing the redis.conf file. You can also instruct Redis to rewrite the append-only file by running the bgrewriteaof command:

      bgrewriteaof will create the shortest set of commands needed to bring the database back to its current state. As this command’s name implies, it will run in the background. However, if another persistence command is running in a background process already, that command must finish before Redis will execute bgrewriteaof.

      Conclusion

      This guide details a number of commands used to manage databases and keys. If there are other related commands, arguments, or procedures you’d like to see in this guide, please ask or make suggestions in the comments below.

      For more information on Redis commands, see our tutorial series on How to Manage a Redis Database.



      Source link