How we made leaderboards at Buff.game

Or how to make and maintain leaderboards for millions of users

About Buff

Buff is the platform for gamers where they can play their favourite games and earn rewards, which can be redeemed to gift cards, gaming accessories, etc. The app is available on both Windows and mobile (iOS and Android). Here’s the link to the Buff.

Previous implementation

Leaderboards in desktop app

Before the refactoring and rewriting the leaderboards from scratch we had a previous version of the leaderboards. It consists in 2 leaderboards for each game: weekly and biweekly. The core concept it that implementation was to scan all users’ earnings for a specific type of earning (further transaction), aggregate them by game and write it to the Redis.

The problem here was that the table, which contains the transactions has ~700 millions of records, and the average number of daily transactions is ~3 millions. Also, the average time to generate a leaderboard was ~15–30 minutes, which is not acceptable.

As you can see, it is very hard for the PostgreSQL to process this huge amount of data. Thus, we had a lot of problems when some leaderboards for our most popular games were failing.

Current implementation

From this point we started to think how we can reduce the load on the database, and to speed up the leaderboards generation.

First thing that we came up with was to store leaderboards in Redis (in sorted sets). But the problem here is that the amount of users is huge, and the storage we’ll need for it will be tremendous.

We thought how we can combine the Redis and PostgreSQL, so that in Redis we’ll store only the portion of the leaderboards and thus it wouldn’t take a lot of space.

Daily leaderboards

To reduce the size of leaderboards in Redis we store only daily leaderboards in sorted sets. These leaderboards are hidden from users, and they are dynamic. For example, when the user finishes a game, the transactions is being created, and the event TransactionCreated is being published to the NATS. Leaderboard service will listen for all the TransactionCreated events and count transactions amounts for whitelisted transaction types (e.g. we will count only transaction for end game, premium bonuses, challenges, etc.).

At midnight we are starting to dump all of these daily leaderboards for each game to our database. As you can see we have a pretty straightforward structure of the database.

ERD for leaderboard service

So, here’s the basic algorithm for dumping daily leaderboards and generating weekly and biweekly leaderboards for each game:

1. Create a new day in the database for a specific game.
2. Start to scanning the daily leaderboard for a game. As you remember, we use sorted sets, so we’ll use the RedisZSCAN command. What it does is iteratively returns all the elements from the sorted set using cursors. This will help us to not to load the Redis and our server, as the amount of elements can be hundreds of thousands.
3. For each retrieved portion of elements we’ll save them into leaderboard_days_progresses table.
4. When the dumping has been done, we’ll make a queries for calculating the top 20 users (for both weekly and biweekly), which we will show on the leaderboard.
5. Save these top 20 users in the Redis.

Also, we need to store the amount of coins which user has earned during one week and two weeks. In order to do that we need to make the same queries which we’ve executed for top 20 users, but not to limit the returning records. We use the pg-query-stream package to iterate the every record which the database returns, and save it to the Redis hash, where the property will be the user’s id, and the value is the amount of coins he earned during the period.

Conclusion

And this is it. We’ve successfully migrated to the new leaderboards, and the amount of time for generating leaderboards for all the games now takes a couple of minutes instead of ~1 hour.