Install ELK stack without sudo access

Recently, on my blog post on installing ELK stack on Ubuntu/Debian I was asked if it was possible to have ELK stack running on a server without sudo access. I admit it's a question that got me curious. The answer is yes and no. If you do not have sudo access and do not have the applications listed below installed on your server then no, it's not possible to run ELK stack without sudo access. The expected applications are:

tar (needed for untarring elasticsearch, logstash and kibana)
make (making the redis server)
java (needed for logstash, and elasticsearch to run)

If you have all of the above available on your machine to be run as you then follow on!

Note: For all the instructions below I am assuming you are in your home directory i.e. cd ~. I have tested these instructions on a vagrant/virtualbox machine in /home/vagrant.

Install Elasticsearch

wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.4.2.zip
tar -xvf elasticsearch-1.4.2.tar.gz

Install Logstash

wget https://download.elasticsearch.org/logstash/logstash/logstash-1.4.2.tar.gz
tar -xvf logstash-1.4.2.tar.gz

Create Logstash config file

vi logstash.conf in your home directory and copy the contents below to it:

input {
  redis {
    host => "127.0.0.1"
    type => "redis"
    data_type => "list"
    key => "logstash"
  }
}
output {
stdout { }
  elasticsearch {
    cluster => "elasticsearch"
  }
}

Install Kibana

wget https://download.elasticsearch.org/kibana/kibana/kibana-4.0.0-beta3.tar.gz
tar -xvf kibana-4.0.0-beta3.tar.gz

Install Redis server

wget http://download.redis.io/releases/redis-2.8.19.tar.gz
tar -xvf redis-2.8.19.tar.gz
cd redis-2.8.19
make
cd ~

Start the ELK stack

nohup ~/elasticsearch-1.4.2/bin/elasticsearch > elasticsearch.log 2>&1 &
nohup ~/logstash-1.4.2/bin/logstash -f logstash.conf > logstash.log 2>&1 &
nohup ~/kibana-4.0.0-beta3/bin/kibana > kibana.log 2>&1 &
nohup ~/redis-2.8.19/src/redis-server > redis.log 2>&1 &

What we have done above is started elasticsearch, logstash, kibana, redis server in the background and redirected each of there output and error logs to their respective log files.

Done! Now you can access the kibana via your favorite browser at: http://yourserverip:5601

P.S: There is a bug in Kibana that requires a simple hack. Do this:

cd /tmp
wget 
https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.4.2.tar.gz && tar -xvf elasticsearch-1.4.2.tar.gz
mv ~/logstash-1.4.2/vendor/jar/elasticsearch-1.1.1/ /tmp
mv /tmp/elasticsearch-1.4.0.Beta1 ~/logstash-1.4.2/vendor/jar/

Running ELK stack via Supervisord

This is the bonus section!!!

Running processes in the background is great but I like more control over the processes. Hence I recommend using supervisord. To use supervisord you need to have pip or easy_install installed on your machine.

Install supervisord: easy_install supervisor or pip install supervisor --pre

Setup a config file for supervisor:
echo_supervisord_conf > ~/supervisord.conf

Now add the following lines to ~/supervisord.conf:

[program:elasticsearch]
command=elasticsearch-1.4.2/bin/elasticsearch
directory=/home/vagrant
autostart=true
startretries=3
stdout_logfile=/home/vagrant/elasticsearch.log
stderr_logfile=/home/vagrant/elasticsearch.log

[program:logstash]
command=logstash-1.4.2/bin/logstash -f logstash.conf
directory=/home/vagrant
autostart=true
startretries=3
stdout_logfile=/home/vagrant/logstash.log
stderr_logfile=/home/vagrant/logstash.log

[program:kibana]
command=kibana-4.0.0-beta3/bin/kibana
directory=/home/vagrant
autostart=true
startretries=3
stdout_logfile=/home/vagrant/kibana.log
stderr_logfile=/home/vagrant/kibana.log

[program:redis]
command=redis-2.8.19/src/redis-server
directory=/home/vagrant
autostart=true
startretries=3
stdout_logfile=/home/vagrant/redis.log
stderr_logfile=/home/vagrant/redis.log

Finally, start the supervisord: supervisord -c supervisord.conf -n. This will start supervisord in the foreground. Once you have verified that all your applications are running as expected, exit out of supervisord by pressing Ctrl+C and start supervisord as a daemon: supervisord -c supervisord.conf

This will start the supervisord as a daemon. You are all set.

You can start/stop elasticsearch, logstash, kibana, redis by: supervisorctl start <program_name_from_above> and supervisorctl stop <program_name_from_above>.

UPDATE (12/24/2015)

Install Logstash shipper without sudo access:

Log onto each machine from where you would like to ship the logs to ELK stack. And install logstash:

cd ~
wget https://download.elasticsearch.org/logstash/logstash/logstash-1.4.2.tar.gz
tar -xvf logstash-1.4.2.tar.gz

Once you have logstash installed, then create logstash.conf in your home directory (or a directory where you have write permissions) and copy the contents below to it:

input {  
  file {
    path => "/var/logs/your_app/your_app.log"
    start_position => beginning
    codec =>  multiline {
      'negate' => true
      'pattern' => '^\d'
      'what' => 'previous'
    }
  }
}

filter {  
  grok {
    # You probably will need to change the grok pattern to match your log pattern
    match => {"message" => "%{DATESTAMP:logDateTime} \[%{LOGLEVEL:logLevel}\] %{GREEDYDATA:logMessage}" }
}

output {  
  redis { host => "10.11.14.15" data_type => "list" key => "logstash" }
}

NOTE: On line 16: You probably would need to change the pattern to match your log statements if you want to tokenize the logs in a specific way. If you do not care about specific tokens then you can remove code block from line: 13-18.

Here's the simplified version:

input {  
  file {
    path => "/var/logs/your_app/your_app.log"
    start_position => beginning
    codec =>  multiline {
      'negate' => true
      'pattern' => '^\d'
      'what' => 'previous'
    }
  }
}

output {  
  redis { host => "10.11.14.15" data_type => "list" key => "logstash" }
}

Modify line 4 to provide the location of your log file. And, line 15 to change the ip address to your redis server ip address.

Finally start the logstash shipper (on each client machine):
nohup ~/logstash-1.4.2/bin/logstash -f ~/logstash.conf > ~/logstash.log 2>&1 &