🧠 PyRedis

A Redis-like in-memory database server written in pure Python — no dependencies.

PyRedis is a from-scratch implementation of a Redis-compatible server that speaks the real Redis wire protocol (RESP) over TCP. It exists to show that a surprisingly complete in-memory database — with multiple data types, key expiry, AOF persistence, and concurrent clients — fits comfortably in a single Python file using nothing beyond the standard library. If you want to understand how Redis actually works under the hood, reading this code is a faster path than diving straight into the C source.

---

✨ Feature Highlights

Feature	What it does	Highlight
RESP Wire Protocol	Binary-safe serialisation identical to real Redis	Strings, ints, arrays, dicts, errors, NULL
Rich Data Structures	String, List, Set, Hash	Type-safe — WRONGTYPE errors enforced
Key Expiry (TTL)	Per-key deadlines with lazy + active eviction	SET key val EX 10 or EXPIRE key 30
AOF Persistence	Append-only file survives process restarts	Replay on startup; EXAT stores absolute deadlines
Thread-per-client	Each TCP connection runs in a daemon thread	threading.Lock guards all KV mutations
INFO command	Live server statistics	uptime, commands run, connected clients, key count

---

🏗️ Architecture

Each incoming TCP connection is dispatched to its own daemon thread. That thread owns a ProtocolHandler instance that reads RESP frames from the socket file-object, forwards the parsed command list to Server.get_response(), and writes the serialised response back. All mutations to the shared key-value store are protected by a single threading.Lock.

  Client A ──┐
  Client B ──┤──► TCP Socket ──► _connection_handler() [Thread]
  Client C ──┘                           │
                                          ▼
                                 ProtocolHandler
                               (RESP encode/decode)
                                          │
                                          ▼
                                 Server.get_response()
                               ┌──────────┴──────────┐
                          _kv store             _expires store
                     {key: (type, val)}     {key: deadline float}
                               │
                               ▼
                         AOF Writer (_aof_write)
                      appendonly.aof ── replay on boot

Concepts & Tools

Concept	Python Tool Used	Why
TCP Server	socket.socket + SO_REUSEADDR	Raw socket gives full protocol control
Concurrency	threading.Thread (daemon)	One thread per client, simple and portable
Thread Safety	threading.Lock	Guards _kv and _expires from race conditions
Active TTL Reaper	threading.Thread (daemon, 1 s loop)	Evicts expired keys proactively
Binary Protocol	io.BytesIO + socket.makefile("rwb")	Buffer writes before flush; file-like API on socket
Ordered Collection	collections.deque	O(1) push/pop from both ends for the List type
Serialisation	Custom RESP via BytesIO	Self-referential: same serialiser used for AOF and wire

---

🚀 Quick Start

Run the server

python redis_server.py
# [server] Listening on 127.0.0.1:31337

Connect a client (Python REPL)

from redis_server import Client

with Client() as c:
    c.set("hello", "world")
    print(c.get("hello"))        # b'world'

The Client class is a thin RESP wrapper around a plain TCP socket. Every method maps 1-to-1 to a server command, and Client works as a context manager — the socket is closed cleanly on exit.

---

📖 Command Reference

Strings

Command	Syntax	Returns	Example
GET	GET key	value or None	c.get("name")
SET	SET key value [EX seconds]	1	c.set("name", "ada", ex=60)
DELETE	DELETE key [key …]	number deleted	c.delete("name")
MGET	MGET key [key …]	list of values	c.mget("k1", "k2")
MSET	MSET key val [key val …]	number of pairs set	c.mset("k1", "v1", "k2", "v2")
FLUSH	FLUSH	number of keys cleared	c.flush()

Lists

Lists are backed by collections.deque for O(1) operations at both ends. stop in LRANGE is inclusive, matching Redis semantics. Empty lists are automatically removed from the store.

Command	Syntax	Returns	Example
LPUSH	LPUSH key value [value …]	new list length	c.lpush("q", "a", "b")
RPUSH	RPUSH key value [value …]	new list length	c.rpush("q", "c")
LPOP	LPOP key	popped value or None	c.lpop("q")
RPOP	RPOP key	popped value or None	c.rpop("q")
LRANGE	LRANGE key start stop	list slice (stop inclusive, negatives wrap)	c.lrange("q", 0, -1)
LLEN	LLEN key	list length	c.llen("q")

Sets

Command	Syntax	Returns	Example
SADD	SADD key member [member …]	count of new members added	c.sadd("tags", "python", "redis")
SREM	SREM key member [member …]	count removed	c.srem("tags", "redis")
SMEMBERS	SMEMBERS key	list of all members	c.smembers("tags")
SISMEMBER	SISMEMBER key member	1 or 0	c.sismember("tags", "python")
SINTER	SINTER key [key …]	intersection as list	c.sinter("s1", "s2")
SUNION	SUNION key [key …]	union as list	c.sunion("s1", "s2")

Hashes

HGETALL returns a flat interleaved list [field, value, field, value, …], just like real Redis. HSET returns the count of newly created fields, not total fields updated.

Command	Syntax	Returns	Example
HSET	HSET key field value [field value …]	count of new fields added	c.hset("user:1", "name", "ada", "age", "36")
HGET	HGET key field	value or None	c.hget("user:1", "name")
HDEL	HDEL key field [field …]	count deleted	c.hdel("user:1", "age")
HGETALL	HGETALL key	flat [field, val, …] list	c.hgetall("user:1")
HEXISTS	HEXISTS key field	1 or 0	c.hexists("user:1", "name")
HLEN	HLEN key	number of fields	c.hlen("user:1")

TTL / Expiry

Command	Syntax	Returns	Notes
EXPIRE	EXPIRE key seconds	1 if key exists, 0 otherwise	Sets a relative TTL in seconds
TTL	TTL key	seconds remaining, -1, or -2	-1 = no expiry set; -2 = key does not exist
PERSIST	PERSIST key	1 if expiry removed, 0 otherwise	Makes the key permanent again

TTL is stored internally as an absolute time.monotonic() deadline, so the value is immune to wall-clock adjustments. The SET command accepts EX <seconds> directly:

c.set("token", "abc123", ex=3600)   # expires in 1 hour

Server

Command	Syntax	Returns
INFO	INFO	Flat [field, value, …] list containing: uptime_seconds, total_commands, connected_clients, total_keys, aof_enabled, aof_path

pairs = c.info()
info  = dict(zip(pairs[::2], pairs[1::2]))
print(info["uptime_seconds"])   # e.g. 42
print(info["total_keys"])       # e.g. 7

---

💡 Feature Deep Dives

TTL & Expiry — lazy eviction + active reaper

PyRedis enforces TTLs through two complementary mechanisms.

Lazy eviction — every read command (GET, LPOP, HGET, …) calls _is_expired(key) before touching the store. If the key's monotonic deadline has passed it is deleted on the spot and the command behaves as if the key never existed.

Active eviction — a daemon thread (_expiry_reaper) wakes up once per second, scans _expires under the lock, and deletes every key whose deadline has passed. This keeps memory clean even for keys that are never read again.

import time
from redis_server import Client

with Client() as c:
    c.set("session", "abc123", ex=30)
    print(c.ttl("session"))     # → 30

    time.sleep(1)
    print(c.ttl("session"))     # → 29

    c.persist("session")
    print(c.ttl("session"))     # → -1  (no expiry)

AOF Persistence — every write survives a restart

Every mutating command serialises its arguments as a RESP array and appends it to appendonly.aof. On the next Server() construction, _replay_aof() re-reads the file and feeds each command back through get_response() — reconstructing the exact same in-memory state.

s1 = Server(aof_path="mydb.aof")
c1 = Client()
c1.set("city", "London")
c1.lpush("primes", "2", "3", "5")
c1.close()
# ... stop s1 ...

s2 = Server(aof_path="mydb.aof")   # replays automatically
c2 = Client()
print(c2.get("city"))              # b'London'
print(c2.lrange("primes", 0, -1))  # [b'5', b'3', b'2']
c2.close()

Why EXAT instead of EX? When the AOF is written, relative EX <seconds> TTLs are immediately converted to absolute monotonic deadlines using EXAT <float>. This means that if the server is restarted 10 seconds after a SET key val EX 30 was issued, the replayed key will correctly have only ~20 seconds of TTL remaining — not a fresh 30.

RESP Wire Protocol — what actually crosses the socket

The Redis Serialisation Protocol (RESP) is a plain-text, line-oriented binary format. Each value is prefixed with a single byte that identifies its type:

Prefix	Type	Example
+	Simple string	+OK\r\n
-	Error	-ERR unknown command\r\n
:	Integer	:42\r\n
$	Bulk string	$5\r\nhello\r\n
*	Array	*2\r\n$3\r\nfoo\r\n$3\r\nbar\r\n
%	Dictionary	%1\r\n$3\r\nkey\r\n$3\r\nval\r\n
$-1	NULL	$-1\r\n

A SET foo bar command sent by the client looks like this on the wire:

*3\r\n
$3\r\nSET\r\n
$3\r\nfoo\r\n
$3\r\nbar\r\n

ProtocolHandler.handle_request() decodes it step by step:

• Reads one byte → * → this is an array
• Reads the length line → 3 → expect 3 elements
• For each element, reads one byte → $ → bulk string
• Reads the length line → 3 → read exactly 3 bytes → SET / foo / bar
• Returns the Python list [b'SET', b'foo', b'bar']

The same ProtocolHandler._write() method is used for both the TCP wire response and the AOF file — the serialiser is completely self-contained.

---

🧪 Running Tests

python test_redis_server.py

Expected output:

..................................................
----------------------------------------------------------------------
Ran 48 tests in 2.3s

OK

The suite starts real server instances on port 31339 over genuine TCP sockets — no mocking. Each test class exercises an isolated feature group:

Class	What it tests
TestStringCommands	GET, SET, DELETE, MGET, MSET, FLUSH — including overwrites and missing-key behaviour
TestTTLExpiry	EX option, EXPIRE, TTL return values (-1/-2), PERSIST, lazy expiry confirmation
TestListCommands	LPUSH/RPUSH ordering, LPOP/RPOP, LRANGE with negative indices, type-mismatch errors
TestSetCommands	SADD deduplication, SREM, SMEMBERS, SISMEMBER, SINTER, SUNION, WRONGTYPE errors
TestHashCommands	HSET new-field count, HGET, HDEL, HGETALL flat list, HEXISTS, HLEN, WRONGTYPE errors
TestAOFPersistence	Full server restart cycle — strings, lists, sets, hashes, and TTL all survive replay
TestInfoCommand	INFO field presence, total_commands counter, total_keys reflection
TestMultipleClients	Two concurrent clients writing and reading independent keys; FLUSH visibility

---

🔭 What's Next (Extension Ideas)

• Pub/Sub — SUBSCRIBE / PUBLISH using per-channel queue.Queue; subscriber threads block on get() while the publisher does put()
• Pipelining — batch multiple commands in a single TCP round-trip by buffering write_response calls and flushing once at the end
• Sorted Sets — ZADD / ZRANGE / ZRANK using bisect for O(log n) inserts into a score-ordered list
• RDB snapshot — periodic full serialisation (e.g. with pickle or msgpack) to a binary file for faster cold-start than AOF replay
• ACL / AUTH — password-based command gating checked at the start of _connection_handler before any command is dispatched
• Cluster mode — consistent hashing across multiple server instances with a thin proxy layer routing keys by hash slot

---

📚 References

• Building a simple Redis server with Python — Charles Leifer's article that inspired this project
• Redis RESP Protocol Specification — the official wire-format reference
• Redis Command Documentation — full semantics for every command implemented here