Installing Graylog2 on Ubuntu 12.04 from source

Install MongoDB -10gen

The steps to install and run MongoDB in your Ubuntu 12.04 Precise Pangolin.

1. Add GPG Key

2. Edit your source.list

sudo nano /etc/apt/sources.list

and add this into bottom line :

3. Update and install

Install Java

sudo apt-get update

sudo apt-get install openjdk-7-jdk

Elasticsearch

wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.20.5.deb

sudo dpkg -i elasticsearch-0.20.5.deb
sudo service elasticsearch start

At this time If you get the message:

Starting ElasticSearch…
Waiting for ElasticSearch………………………..
WARNING: ElasticSearch may have failed to start.

The server instance didn’t have enough memory for the default ElasticSearch service wrapper settings. To fix this

changed ES_HEAP_SIZE from 1024 to 512

GUI elasticsearch-head. A web front end for an ElasticSearch cluster (http://mobz.github.com/elasticsearch-head)

Installing and Running as a plugin of ElasticSearch (this is the preferred method)

The plugin exectuable is available at /usr/share/elasticsearch/bin/plugin.

sudo /usr/local/share/elasticsearch/bin/plugin -install mobz/elasticsearch-head
open http://localhost:9200/_plugin/head/

This will automatically download the latest version of elasticsearch-head from github and run it as a plugin within the elasticsearch cluster. In this mode;

elasticsearch provides a simple webserver to run head
elasticsearch-head automatically connects to the node that is running it
is available at http://localhost:9200/_plugin/head/ (or whatever the address of your cluster is)
Will not work with elasticsearch prior to version 0.17

A. Graylog2-server Installation from source (downloads: https://github.com/Graylog2/graylog2-server/downloads)

cd /opt/
sudo wget http://download.graylog2.org/graylog2-server/graylog2-server-0.11.0.tar.gz -O graylog2-server.tar.gz
sudo tar -xf graylog2-server.tar.gz
sudo mv /opt/graylog2-server-0.11.0 /opt/graylog2-server
sudo rm graylog2-server.tar.gz

Configuration

Now copy the example configuration files:

sudo cp graylog2.conf.example /etc/graylog2.conf
sudo cp elasticsearch.yml.example /etc/graylog2-elasticsearch.yml

You can leave most variables as they are for a first start. All of them should be well documented.

Configure at least these variables in /etc/graylog2.conf:

is_master = true

Set only one graylog2-server node as the master. This node will perform periodical and maintenance actions that slave nodes won’t. Every slave node will accept messages just as the master nodes. Nodes will fall back to slave mode if there already is a master in the cluster.

elasticsearch_config_file = /etc/graylog2-elasticsearch.yml

This is the path to the ElasticSearch configuration file for the built-in ElasticSearch node of graylog2-server. Your graylog2-server node will act as a node

in your ElasticSearch cluster, but not store any data itself. It will distribute the writes to other nodes in the ElasticSearch cluster.

elasticsearch_max_docs_per_index = 20000000

How many log messages to keep per index. This setting multiplied with elasticsearch_max_number_of_indices results in the maximum number of messages in your Graylog2 setup. It is always better to have several more smaller indices than just a few larger ones.

elasticsearch_max_number_of_indices = 20

How many indices to have in total. If this number is reached, the oldest index will be deleted.

elasticsearch_shards = 4

The number of shards for your indices. A good setting here highly depends on the number of nodes in your ElasticSearch cluster. If you have one node, set it to 1. Read more about this in the knowledge base article about configuring and tuning ElasticSearch.

elasticsearch_replicas = 0

The number of replicas for your indices. A good setting here highly depends on the number of nodes in your ElasticSearch cluster. If you have one node, set it to 0. Read more about this in the knowledge base article about configuring and tuning ElasticSearch.

recent_index_ttl_minutes = 60

Graylog2 keeps a so called recent index that includes only the newest log messages. This allows fast overview pages in the web interface. The messages you see in the “show recent messages” view are from this index. If you have thousands of messages per minute, set it to 1 minute because there are so many new messages coming in. If you have just a few messages per minute, set it to a higher values to still have a good overview without having to click on “show all messages”.

mongodb_*

Enter your MongoDB connection and authentication information here. Make sure that you connect the web interface to the same database. You don’t need to configure mongodb_user and mongodb_password if mongodb_useauth is set to false.

…and at least these in /etc/graylog2-elasticsearch.yml:

cluster.name: graylog2

The cluster name of your ElasticSearch cluster. All nodes that are discovered will join the cluster if they have the same cluster name. This must be the same cluster name your ElasticSearch nodes have configured.

Multicast/Unicast
The default setting of ElasticSearch is to use Multicast to discover other nodes. This can be useful but a bit hard to configure depending on your network architecture. Also think about your broadcast domains: If a developer starts up an ElasticSearch node and is in the same multicast broadcast domain, he will join your production cluster (if the cluster.name is the same). If you don’t plan to change or add ElasticSearch nodes regulary, I would recommend to disable multicast and enable unicast. Do this by setting discovery.zen.ping.multicast.enabled: false to true and add your ElasticSearch node hosts to discovery.zen.ping.unicast.hosts. Multicast should be fine for a first quick start though and have no problems discovering a node on localhost.

Example for unicast discovery of a standard ElasticSearch server on the same host:

discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: [“127.0.0.1:9300”]

You might have to define different ports for your ElasticSearch node and the embedded graylog2-server ElasticSearch node if you are running them on the same host if you get port binding errors. It is recommended to have at least ElasticSearch running on a different host than graylog2-server.

A detailled documentation of all important configuration variables is available in this knowledge base and should be read after you have a first setup of Graylog2 running.

Starting the server

You need to have Java installed. Running the OpenJDK is totally fine and should be available on all platforms:

The first start should be performed without the ‘bin/graylog2ctl script to easily see warnings, errors or problems:

sudo java -jar graylog2-server.jar –debug

See the startup parameters page to learn more about available startup parameters. Note that you might have to be root to bind to port 514 for syslog.

You should see a line like this in the debug output if graylog2-server successfully connected to your ElasticSearch cluster:

2012-12-03 00:26:00,080 DEBUG: org.elasticsearch.transport.netty – [graylog2-server] connected to node [[Cyber][APKeeyD_T2uULknphyKlBg][inet[/192.168.1.6:9300]]]

This line indicates that your graylog2-server instance is up and ready to accept messages:

2012-12-03 00:26:01,684 INFO : org.graylog2.Core – Graylog2 up and running.

Let’s try that out! Send in a simple (not standard compliant) syslog message using netcat or your preferred socket communication tool:

~$ echo “<34> Hello Graylog2. Let’s be friends.” | nc -w 1 -u localhost 514

You should see this message being handled in your graylog2-server debug output:

2012-12-03 00:47:37,128 INFO : org.graylog2.inputs.syslog.SyslogProcessor – Date could not be parsed. Was set to NOW because allow_override_syslog_date is true.
2012-12-03 00:47:37,128 DEBUG: org.graylog2.inputs.syslog.SyslogProcessor – Adding received syslog message to process buffer: level: 2 | host: localhost | facility: security/authorization | add.: 0 | shortMessage: Hello Graylog2. Let’s be friends.
2012-12-03 00:47:37,128 DEBUG: org.graylog2.buffers.processors.ProcessBufferProcessor – Starting to process message .
2012-12-03 00:47:37,128 DEBUG: org.graylog2.buffers.processors.ProcessBufferProcessor – Applying filter [Rewriter] on message .
2012-12-03 00:47:37,129 DEBUG: org.graylog2.buffers.processors.ProcessBufferProcessor – Applying filter [Blacklister] on message .
2012-12-03 00:47:37,129 DEBUG: org.graylog2.buffers.processors.ProcessBufferProcessor – Applying filter [Tokenizer] on message .
2012-12-03 00:47:37,129 DEBUG: org.graylog2.filters.TokenizerFilter – Extracted <0> additional fields from message k=v pairs.
2012-12-03 00:47:37,129 DEBUG: org.graylog2.buffers.processors.ProcessBufferProcessor – Applying filter [StreamMatcher] on message .
2012-12-03 00:47:37,130 DEBUG: org.graylog2.filters.StreamMatcherFilter – Routed message to 1 streams.
2012-12-03 00:47:37,131 DEBUG: org.graylog2.buffers.processors.ProcessBufferProcessor – Applying filter [CounterUpdater] on message .
2012-12-03 00:47:37,131 DEBUG: org.graylog2.buffers.processors.ProcessBufferProcessor – Finished processing message. Writing to output buffer.
2012-12-03 00:47:37,131 DEBUG: org.graylog2.buffers.processors.OutputBufferProcessor – Processing message from OutputBuffer.
2012-12-03 00:47:37,131 DEBUG: org.graylog2.buffers.processors.OutputBufferProcessor – Writing message batch to [ElasticSearch Output]. Size <1>
2012-12-03 00:47:37,131 DEBUG: org.graylog2.outputs.ElasticSearchOutput – Writing <1> messages.
2012-12-03 00:47:37,141 DEBUG: org.graylog2.indexer.EmbeddedElasticSearchClient – Deflector index: Bulk indexed 1 messages, took 9 ms, failures: false
2012-12-03 00:47:37,141 DEBUG: org.graylog2.indexer.EmbeddedElasticSearchClient – Recent index: Bulk indexed 1 messages, took 9 ms, failures: false
2012-12-03 00:47:37,142 DEBUG: org.graylog2.buffers.processors.OutputBufferProcessor – Wrote message to all outputs. Finished handling.

Now exit and start the server using the control script:

cd bin/
sudo ./graylog2ctl start | stop | restart

This will start your graylog2-server in the background. Find a log in nohup.out in the same directory.

# Configure the local rsyslog server to talk to graylog2.

echo "*.* @localhost" | sudo tee /etc/rsyslog.d/32-graylog2.conf

sudo service rsyslog restart

Check that logs are making it into elasticsearch

   curl -XGET 'http://localhost:9200/graylog2/_search?pretty=true' -d '
{ 
    "query" : { 
        "matchAll" : {}  
    }   
}'

Graylog2-web-interface install (download: https://github.com/Graylog2/graylog2-web-interface/downloads)

cd /opt/
sudo wget http://download.graylog2.org/graylog2-web-interface/graylog2-web-interface-0.11.0.tar.gz -O graylog2-web-interface.tar.gz
sudo tar -xf graylog2-web-interface.tar.gz
sudo mv graylog2-web-interface-0.11.0 graylog2-web-interface
sudo rm graylog2-web-interface.tar.gz

# Setup mongoid.yml

sudo nano /opt/graylog2-web-interface/config/mongoid.yml

 production:
   host: localhost
   port: 27017
   username:
   password:
   database: graylog2

# Ruby 1.9.3 install

sudo apt-get install libcurl4-openssl-dev libssl-dev zlib1g-dev

sudo apt-get install ruby1.9.3

OR from source

wget ftp://ftp.ruby-lang.org/pub/ruby/1.9/ruby-1.9.3-p194.tar.gz
tar -xvzf ruby-1.9.3-p194.tar.gz
cd ruby-1.9.3-p194
./configure –prefix=/usr –disable-install-doc
sudo make
sudo make install

# Choose the default ruby version

 sudo update-alternatives –config ruby

# Install gems necessary to run graylog2

cd /opt/graylog2-web-interface
sudo gem install bundler

sudo bundle install
sudo gem install rake

sudo gem install passenger

# Passenger, Nginx install

sudo passenger-install-nginx-module –auto –auto-download –prefix=/opt/nginx

sudo nano /opt/nginx/conf/nginx.conf

http {
…
passenger_root /var/lib/gems/1.9.1/gems/passenger-3.0.19;
passenger_ruby /usr/bin/ruby1.9.1;
…
}

server {
listen 80;
server_name localhost;
root /opt/graylog2-web-interface/public;
passenger_enabled on;
}

Check the Users Guide for security and
optimization tips and other useful information:

/var/lib/gems/1.8/gems/passenger-3.0.19/doc/Users guide Nginx.html

Enjoy Phusion Passenger, a product of Phusion (www.phusion.nl) 🙂
https://www.phusionpassenger.com

Setup a script to allow us to control Nginx. We’re going to grab this from Linode:

wget -O init-deb.sh http://library.linode.com/assets/660-init-deb.sh

sudo mv init-deb.sh /etc/init.d/nginx

sudo chmod +x /etc/init.d/nginx

sudo /usr/sbin/update-rc.d -f nginx defaults

You can now control Nginx with this script. To start and stop the server manually, you run:

sudo /etc/init.d/nginx stop

sudo /etc/init.d/nginx start

The script *************

#! /bin/sh

### BEGIN INIT INFO
# Provides: nginx
# Required-Start: $all
# Required-Stop: $all
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: starts the nginx web server
# Description: starts nginx using start-stop-daemon
### END INIT INFO

PATH=/opt/nginx/sbin:/sbin:/bin:/usr/sbin:/usr/bin
DAEMON=/opt/nginx/sbin/nginx
NAME=nginx
DESC=nginx

test -x $DAEMON || exit 0

# Include nginx defaults if available
if [ -f /etc/default/nginx ] ; then
. /etc/default/nginx
fi

set -e

case “$1” in
start)
echo -n “Starting $DESC: “
start-stop-daemon –start –quiet –pidfile /opt/nginx/logs/$NAME.pid \
–exec $DAEMON — $DAEMON_OPTS
echo “$NAME.”
;;
stop)
echo -n “Stopping $DESC: “
start-stop-daemon –stop –quiet –pidfile /opt/nginx/logs/$NAME.pid \
–exec $DAEMON
echo “$NAME.”
;;
restart|force-reload)
echo -n “Restarting $DESC: “
start-stop-daemon –stop –quiet –pidfile \
/opt/nginx/logs/$NAME.pid –exec $DAEMON
sleep 1
start-stop-daemon –start –quiet –pidfile \
/opt/nginx/logs/$NAME.pid –exec $DAEMON — $DAEMON_OPTS
echo “$NAME.”
;;
reload)
echo -n “Reloading $DESC configuration: “
start-stop-daemon –stop –signal HUP –quiet –pidfile /opt/nginx/logs/$NAME.pid \
–exec $DAEMON
echo “$NAME.”
;;
*)
N=/etc/init.d/$NAME
echo “Usage: $N {start|stop|restart|reload|force-reload}” >&2
exit 1
;;
esac

exit 0

****************************

sudo service nginx start

nginx.conf (https://raw.github.com/matthieudolci/Scripts-GrayLog2/master/nginx/nginx.conf)

#user nobody;
worker_processes 4;

#error_log logs/error.log;
#error_log logs/error.log notice;
#error_log logs/error.log info;

#pid logs/nginx.pid;

events {
worker_connections 1024;
}

http {
passenger_root /var/lib/gems/1.9.1/gems/passenger-3.0.19;
passenger_ruby /usr/bin/ruby1.9.1;

include mime.types;
default_type application/octet-stream;

access_log logs/access.log;

sendfile on;
keepalive_timeout 65;
tcp_nodelay on;

gzip on;
gzip_comp_level 2;
gzip_proxied any;

server {
listen 80;
server_name graylog; # or localhost
root /opt/graylog2-web-interface/public;
passenger_enabled on;
rails_spawn_method smart;

location ~* \.(ico|css|js|gif|jpe?g|png)$ {
expires max;
add_header Pragma public;
add_header Cache-Control “public, must-revalidate, proxy-revalidate”;
}
}

}

Installing Bundler for Ruby in Ubuntu 10.04

When attempting to install Bundler for Ruby on Ubuntu 10.04, I got the following error:

shell> sudo gem install bundler
ERROR: Error installing bundler:
bundler requires RubyGems version >= 1.3.6

Running sudo gem -v I saw that I had 1.3.5. To get around this, simply install the available updater gem, then run it:

shell> sudo gem install rubygems-update
shell> sudo /var/lib/gems/1.8/bin/update_rubygems

Now running gem -v I see that I have 1.8.15 and I am able to install bundler:

shell> gem install bundler
Fetching: bundler-1.0.21.gem (100%)
Successfully installed bundler-1.0.21
1 gem installed
Installing ri documentation for bundler-1.0.21…
Installing RDoc documentation for bundler-1.0.21…

After searching for a solution on how to use graylog2 to receive syslog messages from Cisco switching equipment found a solution. Looks like the problem was related with the counter/sequence number that the equipment was sending by default.

If you see in graylog2 messages without a correct Host coming from a Cisco equipment, try the following config in the same equipment:

conf t

no logging message-counter syslog

no service sequence-numbers

logging trap “level”

logging “syslog server ip”

Install MongoDB

If you feel the need to mess around with MongoDB settings, they can be found in:
/etc/mongodb.conf – although they default settings should be fine as they are.

We will proceed by creating a user for greylog in the MongoDB console:
In the shell enter:

Remember to note down the user and password you chose.

Install Java

Download and install elastic search to /opt

Start it

Graylog2-server Installation

1. Download the latest graylog2-server tarball into /opt: https://github.com/Graylog2/graylog2-server/downloads
2. Install into /opt/graylog2-server

1. Configure /etc/graylog2.conf

1. Create /etc/init.d/graylog2-server

1. Create /etc/logrotate.d/graylog2-server

1. Start graylog2-server

Graylog2-web-interface install

1. Install pre-reqs

1. Create graylog user

1. Download the latest graylog2-web-interface tarball: https://github.com/Graylog2/graylog2-web-interface/downloads

1. Install gems necessary to run graylog2 using Bundler in the ruby 1.9.2 environment

1. Configure the web interface’s Mongo settings

The production: settings should look like this:

NOTE: MongoDB creates databases on the fly, so you do not need to create the database before specifying it within the config! (Unlike MYSQL or MSSQL where you are required to have created the database beforehand)

1. We will now configure the database to our needs:

1. Configure the graylog2-web-interface by editing email.yml, general.yml
2. Start the graylog2-web-interface

It will output on the lines of:

Connect to http://:3000. If everything is working, graylog2 will ask you to create the first user. Shutdown graylog2 (ctrl+c) after you create the first user

1. Create /etc/logrotate.d/graylog2-web-interface

1. Setup the webinterface to be served with Apache Mod Passenger

1. Enable the newly created site (Basically symlinking it to /etc/apache2/sites-enabled/…)

1. Restart apache

1. You should now be able to view the webinterface at http://server.domain.com:8080 or at the other configured address