Debian, logstash

Using Riak with Logstash

For the last few days i was playing around with Riak, a distributed database. It’s very simple to configure and use and offcourse it supports MapReduce. I wanted to try out the map reduce, and since logstash has a plugin to write data into riak,  i decided to use it with logstash on an Ubuntu 12.04 machine.

Configuring Riak

Installing Riak is very simple, it has only a few dependencies.

apt-get install  libssl0.9.8  erlang”

Once the dependencies are being installed, we have to download and install the deb package of Riak from its website.

wget http://downloads.basho.com.s3-website-us-east-1.amazonaws.com/riak/CURRENT/ubuntu/lucid/riak_1.2.1-1_i386.deb

dpkg -i  riak_1.2.1-1_i386.deb”.

Once Riak is installed, go to “/etc/riak”, where the config files are available. We can change the name of the riak node by editing the “vm.args” file. By default Riak will listen to “127.0.0.1”, but we can change this by editing “app.config” file.  In order to use enable https enable, we need to uncomment the https section in Riak core config. We also have to mention the path of the server key and certificate. Riak comes with a build in Admin console, which currently has very minimal functions. It shows the status of the riak nodes as well as the members in the riak ring. To enable this, open the “app.config”  go to “riak_control_config” and change the “enabled,false” to “enabled,true”. The user name and password can be mentioned in the userlist option.

If we have multiple machine we can create a riak cluster using riak-admin tool. Currently i’ve only one machine with Riak installed.

In Riak, data’s are stored in “Buckets”. A Bucket is a container and keyspace for data stored in Riak, with a set of common properties for its contents (the number of replicas, or n_val, for instance). Buckets are accessed at the top of the URL hierarchy under “riak”, e.g. /riak/bucket.

Configuring Logstash

Now we have Riak machine, listening on port port “8098”. Now we need to configure logstash to sendthe data to the riak. This is very simple because logstash has an output plugin which can directly write to riak.

In the output section of logstash config file, add the riak output plugin.  It should be like this,

” riak {

bucket => bucketname

type => typename

nodes => [“riakserverip”,”8098″,”riakserverip”,”8098″]

}”

In the nodes section we have to mention the riak node ip’s. Since i’ve only one riak node, i’m mentioning the same ip twice.

That’s it, now we need to start logstash, then logstash will start writing data into the bucket which we mentioned in the conf file.

There is one good GUI for Riak called “rekon“. Just get  the source code from github and edit the “install.sh” and change the ip mentioned in to the ip which riak listens to and execute it. Now we can access the GUI using the below url

http://riakserverip:8098/riak/rekon/go”

Using this we can see the buckets inside the riak and also the corresponding key values.

Now testing the “Map Reduce Function”

This is one of the main features of Riak. For example i’m going to write a map reduce function that will display all the keys in my bucket that has the keyword “mylinux”, which is the hostname of my machine. This function will return the key as well as the number of occurrences. Below is a simple MapReduce function.

“inputs”:”mybucket”,
“query”:[{“map”:{“language”:”javascript”,
“source”:”function(riakObject) {
var m = riakObject.values[0].data.match(\”mylinux\”);
return [[riakObject.key, (m ? m.length : 0 )]];
}”

To execute the map reduce function, execute the following command,

curl -X POST http://192.168.1.173:8098/mapred -H ‘Content-Type: application/json’ -d ‘{
“inputs”:”mybucket”,
“query”:[{“map”:{“language”:”javascript”,
“source”:”function(riakObject) {
var m = riakObject.values[0].data.match(\”mylinux\”);
return [[riakObject.key, (m ? m.length : 0 )]];
}”}}]}’

The above command will return all the keys which has the keyword “mylinux” along with number of occurrences.

Standard
Uncategorized

Hubot @ your service

A few months back, i got a chance to attend the rootconf2012, where i first came to know about “Hubot” , developed by github. I was very much interested, especially the gtalk plugin, where we can integrate “hubot” with a gmail account. We can make Hubot listen to every words and make it to respond back. There are so many default hubot-scripts which we can use to play around with it.

Configuring Hubot is very simple.

First, we’ll install all of the dependencies necessary to get Hubot up and running.

apt-get install build-essential  libssl-dev  git-core  redis-server  libexpat1-dev curl libcurl4-nss-dev libcurl4-openssl-dev”

 

Now, Download and extract Node.js

wget http://nodejs.org/dist/v0.9.2/node-v0.9.2.tar.gz”

tar xf  node-v0.9.2.tar.gz -C  /opt  &&  cd /opt/node-v0.9.2″

“./configure && make && make install”

For Gtalk Plugin we need “node-xmpp“. So we can use “npm” to install it. We also need CoffeeScipt.

“npm install node-xmpp”

“npm install  -g coffee-script”

Now clone the Hubot repository from GitHub.

“git clone git://github.com/github/hubot.git”

Now go inside the hubot folder and using the “hubot” binary inside the bin folder create a deployable hubot.

“./bin/hubot -c /opt/hubot”

Now go inside the new hubot folder and open the “package.json” using a text editor and add the hubot-gtalk dependency package.

“dependencies”: {
   “hubot”: “2.3.2”,
   “hubot-gtalk”: “>= 0.0.1”,
    “hubot-scripts”: “>= 2.1.0”,
   “coffee-script”: “1.3.3”,
    “optparse”: “1.0.3”,
    “scoped-http-client”: “0.9.7”,
    “log”: “1.3.0”,
    “connect”: “2.3.4”,
    “connect_router”: “1.8.6”,

Now use “npm install”  to install the dependencies.

Before starting the hubot, we need to configure the below parameters for Gtalk Adapter.

The GTalk adapter requires only the following environment variables.

  • HUBOT_GTALK_USERNAME (Should be full email address, e. g. username@gmail.com)
  • HUBOT_GTALK_PASSWORD

And the following are optional.

  • HUBOT_GTALK_WHITELIST_DOMAINS
  • HUBOT_GTALK_WHITELIST_USERS
  • HUBOT_GTALK_REGEXP_TRANSFORMATIONS

Once all the parameters are set, we can start the Hubot with Gtalk adapter.

“./bin/hubot -a gtalk”

Now Hubot is online with Gtalk. No we can add the Hubot gmail account to our Gtalk Account and start playing around with it. Hubot comes with a bunch of default scripts. If we type “help”, we will get a a bunch of options for each of these scripts.

Today I was able to execute some Bash commands, using my custom coffee scripts, which gave me some weird ideas, to use “Hubot” for “ChatOPS“. Let’s see how it works. Once it’s done i’ll update it in my blog. Wait for more……………….

Standard