About a year ago we open sourced the feed component of Fashiolista called Feedly. At Fashiolista it powers the flat feed, aggregated feed and the notification system. In general it allows you to build newsfeed and notification systems using Cassandra and/or Redis. Examples of what you can build are applications like the Facebook newsfeed, your Twitter stream or your Pinterest following page.
The project has come a long way over the past year and we’re really happy with the 0.9 release. In this blogpost I’ll show you how to use Feedly, Django, Celery and Redis to build a scalable Pinterest like newsfeed. (Feedly isn’t tied to Django so feel free to try setting this up using another framework)
What we’re building
You can play around with the end result on Heroku. In case it goes down here is a screenshot:
Now most parts of this app aren’t very polished or scalable. The part we are focusing on is the flat feed. (The page showing all the pins by people you follow). The system works by storing a feed for every user. As soon as someone pins an item it gets stored in the feeds of all their followers. Using this approach it becomes really easy to shard a newsfeed system. If you want to know more about the reasoning behind this solution you’ll find the background articles very interesting.
First step, setting up the feeds
Now we need to store those pins somewhere, so let’s create two feeds. One for your own pins and one for the pins of all the people you follow. (example: pin_feed.py)
# the feed containing pins by people you follow class PinFeed(RedisFeed): key_format = 'feed:normal:%(user_id)s' # the feed containing only your pins class UserPinFeed(PinFeed): key_format = 'feed:user:%(user_id)s'
Second step, connecting the followers
With the feeds setup we need a way to know who follows who. You can hook this up by creating a custom feedly class and implementing get_user_follower_ids as shown below. (example: pin_feedly.py)
class PinFeedly(Feedly): feed_classes = dict( normal=PinFeed, ) user_feed_class = UserPinFeed def add_pin(self, pin): # see example core.models.py activity = pin.create_activity() # add user activity adds it to the user feed, and starts the fanout self.add_user_activity(pin.user_id, activity) def get_user_follower_ids(self, user_id): return Follow.objects.filter(target=user_id).values_list('user_id', flat=True) feedly = PinFeedly()
Now when you call feedly.add_pin(pin) it will fetch the ids of all your followers and spawn many small tasks using Celery. Each of those tasks will push the pin to a few of your followers. This process is called the fanout.
Third step, the view
In the view you retrieve the feed instance by calling feedly.get_feeds(user_id)['normal']. The feed is slicable so doing feed[:25] gives you the top 25 activities in the feed.
# django example, this part would be different in another framework @login_required def feed(request): context = RequestContext(request) feed = feedly.get_feeds(request.user.id)['normal'] activities = list(feed[:25]) context['activities'] = activities response = render_to_response('core/feed.html', context) return response
That’s pretty much it. These 3 steps together with a bit of boilerplate for setting things up allow you to build the example app.
The full docs for Feedly can be found on Github and Read the docs. In addition to the Pinterest newsfeed example you can use Feedly to build:
- A Twitter like newsfeed
- A notification system like Facebook’s or Fashiolista’s
- An activity stream like Wanelo’s stories or Etsy’s activity stream
We’re eager to see what applications you’ll build with this. Be sure to let us know on Github if you run into any issues.