A game studio stored session tokens only in ElastiCache for Memcached. A node replacement emptied that node and every player on it was signed out. The sessions must survive a node replacement. What should hold them?
Select an answer to reveal the explanation.
Short Explanation
Memcached is a whiteboard: wipe the board and the notes are gone, which is what you saw when the node was replaced. Sessions that have to outlive a node belong on ElastiCache for Redis with a replica. Another Memcached node does not receive a copy of the keys.
Full Explanation
ElastiCache for Memcached does not replicate and does not persist keys. Replacing a node starts that node empty, so every session that hashed to it disappears. ElastiCache for Redis can run as a replication group: the primary holds the sessions and a replica keeps a copy, so a node replacement or a failover still has the data. A larger Memcached node has more room and the same behavior. Replacement still drops its keys. Adding a Memcached node spreads keys across nodes by the client's hashing. It does not copy existing keys onto the new node, and loss of either node still drops that node's share. An RDS read replica would be a different design, and in this incident the sessions were not in RDS at all, so a replica of the database has nothing to serve. Caveat: Redis replication is asynchronous, so a failure in the tiny window before a write reaches the replica can still lose that write. Operational check: after the cutover, replace one replica in a test window and confirm a session written just before the replacement still reads back.