Uncategorized

Using Logstash + Statsd + graphite – Part2 (Graphite & Statsd)

Now we can setup Graphite and Statsd, so logstash can start input to the statsd. I found a method which uses pip for installing the python packages, which makes steps easier.

# installing nodejs

sudo apt-get install python-software-properties
sudo apt-add-repository ppa:chris-lea/node.js
sudo apt-get update
sudo apt-get install nodejs npm
# Now install dependencies
sudo apt-get install memcached python-dev python-pip sqlite3 libcairo2 libcairo2-dev python-cairo pkg-config
# Install latest pip
pip install –upgrade pip
# Install carbon and graphite deps
cat >> /tmp/graphite_reqs.txt << EOF
django==1.3
python-memcached
django-tagging
twisted
whisper==0.9.9
carbon==0.9.9
graphite-web==0.9.9
EOF
sudo pip install -r /tmp/graphite_reqs.txt

# Configure carbon
cd /opt/graphite/conf/
sudo cp carbon.conf.example carbon.conf

# Create storage schema and copy it over
# Using the sample as provided in the statsd README
cat >> /tmp/storage-schemas.conf << EOF
# Schema definitions for Whisper files. Entries are scanned in order,
# and first match wins. This file is scanned for changes every 60 seconds.
#
# [name]
# pattern = regex
# retentions = timePerPoint:timeToStore, timePerPoint:timeToStore, …
[stats]
priority = 110
pattern = ^stats\..*
retentions = 10s:6h,1m:7d,10m:1y
EOF
sudo cp /tmp/storage-schemas.conf storage-schemas.conf
# Make sure log dir exists for webapp
sudo mkdir -p /opt/graphite/storage/log/webapp
# Copy over the local settings file and initialize database
cd /opt/graphite/webapp/graphite/
sudo cp local_settings.py.example local_settings.py
sudo python manage.py syncdb # Follow the prompts, creating a superuser is optional
# statsd
cd /opt && sudo git clone git://github.com/etsy/statsd.git
# StatsD configuration
cat >> /tmp/localConfig.js << EOF
{
graphitePort: 2003
, graphiteHost: “127.0.0.1”
, port: 8125
}
EOF
sudo cp /tmp/localConfig.js /opt/statsd/localConfig.js
# Starting statsd
node /path/to/statsd-folder/stats.js /path/to/statsd-folder/localConfig.js
# Starting Carbon cache
Go to Graphite bin directory,
“./carbon-cache.py start”
#Starting Graphite Dashboard
Go to Graphite bin directory,
“./run-graphite-devel-server.py /opt/graphite”, Now we can access the graphite dashboard by,
Image
Advertisements
Standard
Uncategorized

Using Logstash + Statsd + graphite – Part1 (Logstash)

This blog helps to set up a simple logstash+statsd+graphite setup, which we’ve currently deployed in our company.  Thanks to @jordansissel for building such a simple and powerful tool “Logstash”. We just need the logstash jar file and  a simple config to run it.

Setting up LOGSTASH

First,Download the latest logstash jar file from the logstash.net

Next we need to create a config file, ex:- logstash.conf, which should contains two mandatory parts “input” and “output“, and an optional “filter” part, where we can mention filter rules.

Image

Starting logstash is very easy, just execute the below command.

“java -jar logstash-1.1.1-monolithic.jar agent -f logstash.conf — web –backend elasticsearch:///?local”

Elastic search UI can be accessed by “htttp://ip-address:9292”

Standard