Engineering behind r/place
3 min readApr 10, 2022
r/place — a collaborative pixel war
r/place is a popular collaborative pixel war that happens at reddit on April fools day. Redditor from different communities wage war among each other in conquering pixels in a 1000 x 1000 pixel canvas. A single user could only colour a single pixel (tile) every five minutes on the collaborative canvas. They collectively attempt to draw memes, flags, mascots, amongst other things, and defend their creations from other Redditors from overwriting. Finally we are left with a collaborative artwork that lapses as below.
Source : https://www.redditinc.com/blog/how-we-built-rplace/
Considerations:
- Shared global state of Canvas comprising of 1000 x 1000 pixels (1 Million).
- Globally distributed heavy read traffic.
- Real-time and Concurrent nature of application.
- Consistency : All clients must be kept in sync with the same view of the current board state, otherwise users with different versions of the board will have difficulty collaborating — bad user experience.
- Tile cooldown : Rate limiting writes from users (1 request /user /5 min).
- Prioritising Low Latency over Strong consistency. (Opting LWW strategy to reach consistency/resolve conflicts)
Canvas State Management:
- Cassandra : Single row with million columns, one for each pixel.
- Redis : 1 Million sized bitfield of 4-bit-integer ( representing colour of a pixel). (x,y coordinate of pixel translated as offset of bitfield).
4 bit x 1 million ~ 4Mb. - CDN : Caching the state of canvas closer to user.
- Canvas updates: Writes from clients are broadcasted using websockets.
Observations:
- Write load is limited due to Tile cooldown (rate-limiting).
(1 req/user/5 min ~ for 100k concurrent users, its 333 req/sec write traffic). - Read load is offloaded to CDN. Requests from CDN to app service is limited.
- Real time updates are served via web-socket service.
- Conflict due to Concurrent writes on a pixel is resolved via LWW strategy. (Eventual consistency)
- Significantly low latency achieved due to caching at CDN as well as at Redis.