Activity Feeds, Message Queues, and Distributed Tasks

Bogdana, a fellow GSoC student, has started developing an activity feed for the dashboard. This will give users a better overview of recent activity relating to the review requests relevant to them.

Developing this feature in a scalable manner is important. After a short design discussion in today’s meeting, it became apparent that simply using the database for storage and retrieval of activity would be insufficient. As many social networks have discovered, a RDBMS quickly becomes too slow as the number of users and activities grows. Message queues, distributed processing, and caching can play a big role in keeping things snappy.

Review Board has many uses for distributed processing and message queues. Developing the activity feed in such a way that the architecture will be reusable for other purposes is highly desirable. Distributing the processing of diffs is one change which could greatly increase Review Boards scalability.

Recently I’ve been developing Review Bot, a distributed tool for automating static analysis with Review Board. Review Bot uses Celery, a distributed task queue, with RabbitMQ as its message queue backend. I wanted to share some articles I’ve discovered while developing Review Bot, and researching queues and activity feeds. Hopefully they will be useful.

Specific Software:

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s