Redis, Sinatra, TwitterAPI

TweetGrabber – a Live Tweet Capturer using REDIS+SINATRA+TwitterAPI

Twitter has become one of the powerfull social networking site. And people do share a lot of good info’s here. Even people reports outages of various sites and services in Twitter. Many of the tech companies do have a valid accounts to keep track of user’s comments on their services/products and they even interacts with user’s. Website’s like “downrightnow.com”, uses Twitter as an information source to identify the outage of many famous websites and services. Even in our personal site’s we use Javascripts to display our tweets and other statuses. But since the Twitter API version 1.1, which needs ouath, many of the jquery plugins became obsolete. This time i got a requirement to built a system that keeps track of tweets based on custom keywords. But i wanted to shaw all the tweets in a category basis, but on a sinlge page, with some scrolling effects, at the same time the sroller should keeps on updating with tweets on a regular interval basis. Which means i’m more concerned about the new trends going on in the Twitter.

Since i’m a Rubyist, i decided to build a Ruby app with Sinatra web Frontend, one my favourite web frameworks. I’m hard lover of Redis, faster since it runs on ram, and i dont want to keep my old tweets. So, My app is pretty simple, there will a Feeder which will talk to Twitter’s API and get the Tweet’s these tweet’s will be then stored in Redis, The Sinatra Fronend will fetch the tweets, and will display it in a scrolling fashion on the front end. Since i’m a CLI junkie, i’m not familiar with HTMLs and UI’s, so i decided to go with Twitter Bootstrap to build the HTML pages.

There is a Ruby gem called ”TweetStream” which works very well with Twitter API v1.1. So i’m going to use this gem to talk to Twitter. Below is the simple architecture diagram for my app.

Let’s see each components in detail.

Feeder

Feeder is a ruby script which constantly talks to Twitter API and grabs the latest streaming tweets based on the search keywords. Add all the grabbed tweets are then stored into the Redis database. Since i’ve to display teets corresponding to each keyword in separate scrolling fashions, i’m using separate redis database for each. So the Feeder has multiple ruby methods, where each method will be used for each keyword and writes into the corresponding Redis DB. Below is one of the Ruby Method in the Feeder script.

                        #######  FEEDER ####### 

TweetStream.configure do |config|
  config.consumer_key       = 'xxxxxxxxxxxxxx'          => All these cosnumerkey,secret and oauth tokens have to be generated from
  config.consumer_secret    = 'xxxxxxxxxxxxxx'             the Twitter's API site, dev.twitter.com
  config.oauth_token        = 'xxxxxxxxxxxxxx'
  config.oauth_token_secret = 'xxxxxxxxxxxxxx'
end

def tweet_general
  TweetStream::Client.new.track('opensource') do |status|   =>  Thiss Tweatstream client will keep tracking the keyword "opensource"
    if ( status.lang == 'en' )
      push(
                'id' => status[:id],
               'text' => status.text,
               'username' => status.user.screen_name,
               'userid' => status.user[:id],
               'name' => status.user.name,
               'profile_image_url' => status.user.profile_image_url,
               'received_at' => Time.new.to_i,
               'user_link' => "http://twitter.com/"
             )
        end
  end
end
def push(data)
      @db = Redis.new
      @db.lpush('tweets_general', data.to_json)         => LPUSHing Tweets into the RedisDB
    end

Redis DB

In Redis, im going to use the LIST data type. LIST are simply list of strings, sorted by insertion order. It is possible to add elements to a Redis List pushing new elements on the head (on the left) or on the tail (on the right) of the list.

So the new tweets i will be pushing in from the head, and prevent the over population, i will be calling LTRIM operation preiodically to clear of the old tweets from the database. All these operations are done from the Feeder by corresponding ruby methods/functions.

Sinatra Frontend

I’m using SINATRA for building the frontend. SInce i’m not much familiar with HTML’s and UI’s, i decided to use Twitter Bootstrap for building the layout. And for each category, i’ve created a collapsible table, so that we can expand and collapse the required tables. Now, the next task is scrolling the tweets, for that i found a jquery plugin called Totem Ticker.ANd i enabled, refreshing for the div element which contains this scroller, so that after each refresh, the variable which supplies tweets to the scroller, will get updated with the newer tweets from the corresponding Redis DB.

As of now the app is working fine. But i’m planning to extend this to add more features, like adding the keywords dynamically from the Web Frontend, and displaying only those tables with the keywords. I will be digging more take it more powerfull :-). I’m going to push the working code soon into my GitHub account, so that others can also paly around on this, and can extend it for their own requirements.

Standard