Since i’m a Rubyist, i decided to build a Ruby app with Sinatra web Frontend, one my favourite web frameworks. I’m hard lover of Redis, faster since it runs on ram, and i dont want to keep my old tweets. So, My app is pretty simple, there will a Feeder which will talk to Twitter’s API and get the Tweet’s these tweet’s will be then stored in Redis, The Sinatra Fronend will fetch the tweets, and will display it in a scrolling fashion on the front end. Since i’m a CLI junkie, i’m not familiar with HTMLs and UI’s, so i decided to go with Twitter Bootstrap to build the HTML pages.
There is a Ruby gem called ”TweetStream” which works very well with Twitter API v1.1. So i’m going to use this gem to talk to Twitter. Below is the simple architecture diagram for my app.
Let’s see each components in detail.
Feeder is a ruby script which constantly talks to Twitter API and grabs the latest streaming tweets based on the search keywords. Add all the grabbed tweets are then stored into the Redis database. Since i’ve to display teets corresponding to each keyword in separate scrolling fashions, i’m using separate redis database for each. So the Feeder has multiple ruby methods, where each method will be used for each keyword and writes into the corresponding Redis DB. Below is one of the Ruby Method in the Feeder script.
####### FEEDER ####### TweetStream.configure do |config| config.consumer_key = 'xxxxxxxxxxxxxx' => All these cosnumerkey,secret and oauth tokens have to be generated from config.consumer_secret = 'xxxxxxxxxxxxxx' the Twitter's API site, dev.twitter.com config.oauth_token = 'xxxxxxxxxxxxxx' config.oauth_token_secret = 'xxxxxxxxxxxxxx' end def tweet_general TweetStream::Client.new.track('opensource') do |status| => Thiss Tweatstream client will keep tracking the keyword "opensource" if ( status.lang == 'en' ) push( 'id' => status[:id], 'text' => status.text, 'username' => status.user.screen_name, 'userid' => status.user[:id], 'name' => status.user.name, 'profile_image_url' => status.user.profile_image_url, 'received_at' => Time.new.to_i, 'user_link' => "http://twitter.com/" ) end end end def push(data) @db = Redis.new @db.lpush('tweets_general', data.to_json) => LPUSHing Tweets into the RedisDB end
In Redis, im going to use the LIST data type. LIST are simply list of strings, sorted by insertion order. It is possible to add elements to a Redis List pushing new elements on the head (on the left) or on the tail (on the right) of the list.
So the new tweets i will be pushing in from the head, and prevent the over population, i will be calling LTRIM operation preiodically to clear of the old tweets from the database. All these operations are done from the Feeder by corresponding ruby methods/functions.
I’m using SINATRA for building the frontend. SInce i’m not much familiar with HTML’s and UI’s, i decided to use Twitter Bootstrap for building the layout. And for each category, i’ve created a collapsible table, so that we can expand and collapse the required tables. Now, the next task is scrolling the tweets, for that i found a jquery plugin called Totem Ticker.ANd i enabled, refreshing for the div element which contains this scroller, so that after each refresh, the variable which supplies tweets to the scroller, will get updated with the newer tweets from the corresponding Redis DB.
As of now the app is working fine. But i’m planning to extend this to add more features, like adding the keywords dynamically from the Web Frontend, and displaying only those tables with the keywords. I will be digging more take it more powerfull :-). I’m going to push the working code soon into my GitHub account, so that others can also paly around on this, and can extend it for their own requirements.