Install MongoDB -10gen
The steps to install and run MongoDB in your Ubuntu 12.04 Precise Pangolin.
1. Add GPG Key
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv 7F0CEB10
2. Edit your source.list
sudo nano /etc/apt/sources.list
and add this into bottom line :
deb http://downloads-distro.mongodb.org/repo/ubuntu-upstart dist 10gen
3. Update and install
sudo apt-get update
sudo apt-get install mongodb-10gen
Install Java
sudo apt-get update
sudo apt-get install openjdk-7-jdk
Elasticsearch
wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.20.5.deb
sudo dpkg -i elasticsearch-0.20.5.deb
sudo service elasticsearch start
At this time If you get the message:
Starting ElasticSearch…
Waiting for ElasticSearch………………………..
WARNING: ElasticSearch may have failed to start.
The server instance didn’t have enough memory for the default ElasticSearch service wrapper settings. To fix this
changed ES_HEAP_SIZE from 1024 to 512
GUI elasticsearch-head. A web front end for an ElasticSearch cluster (http://mobz.github.com/elasticsearch-head)
Installing and Running as a plugin of ElasticSearch (this is the preferred method)
The plugin exectuable is available at /usr/share/elasticsearch/bin/plugin.
sudo /usr/local/share/elasticsearch/bin/plugin -install mobz/elasticsearch-head
open http://localhost:9200/_plugin/head/
This will automatically download the latest version of elasticsearch-head from github and run it as a plugin within the elasticsearch cluster. In this mode;
elasticsearch provides a simple webserver to run head
elasticsearch-head automatically connects to the node that is running it
is available at http://localhost:9200/_plugin/head/ (or whatever the address of your cluster is)
Will not work with elasticsearch prior to version 0.17
A. Graylog2-server Installation from source (downloads: https://github.com/Graylog2/graylog2-server/downloads)
cd /opt/
sudo wget http://download.graylog2.org/graylog2-server/graylog2-server-0.11.0.tar.gz -O graylog2-server.tar.gz
sudo tar -xf graylog2-server.tar.gz
sudo mv /opt/graylog2-server-0.11.0 /opt/graylog2-server
sudo rm graylog2-server.tar.gz
Configuration
Now copy the example configuration files:
sudo cp graylog2.conf.example /etc/graylog2.conf
sudo cp elasticsearch.yml.example /etc/graylog2-elasticsearch.yml
You can leave most variables as they are for a first start. All of them should be well documented.
Configure at least these variables in /etc/graylog2.conf:
is_master = true
Set only one graylog2-server node as the master. This node will perform periodical and maintenance actions that slave nodes won’t. Every slave node will accept messages just as the master nodes. Nodes will fall back to slave mode if there already is a master in the cluster.
elasticsearch_config_file = /etc/graylog2-elasticsearch.yml
This is the path to the ElasticSearch configuration file for the built-in ElasticSearch node of graylog2-server. Your graylog2-server node will act as a node
in your ElasticSearch cluster, but not store any data itself. It will distribute the writes to other nodes in the ElasticSearch cluster.
elasticsearch_max_docs_per_index = 20000000
How many log messages to keep per index. This setting multiplied with elasticsearch_max_number_of_indices results in the maximum number of messages in your Graylog2 setup. It is always better to have several more smaller indices than just a few larger ones.
elasticsearch_max_number_of_indices = 20
How many indices to have in total. If this number is reached, the oldest index will be deleted.
elasticsearch_shards = 4
The number of shards for your indices. A good setting here highly depends on the number of nodes in your ElasticSearch cluster. If you have one node, set it to 1. Read more about this in the knowledge base article about configuring and tuning ElasticSearch.
elasticsearch_replicas = 0
The number of replicas for your indices. A good setting here highly depends on the number of nodes in your ElasticSearch cluster. If you have one node, set it to 0. Read more about this in the knowledge base article about configuring and tuning ElasticSearch.
recent_index_ttl_minutes = 60
Graylog2 keeps a so called recent index that includes only the newest log messages. This allows fast overview pages in the web interface. The messages you see in the “show recent messages” view are from this index. If you have thousands of messages per minute, set it to 1 minute because there are so many new messages coming in. If you have just a few messages per minute, set it to a higher values to still have a good overview without having to click on “show all messages”.
mongodb_*
Enter your MongoDB connection and authentication information here. Make sure that you connect the web interface to the same database. You don’t need to configure mongodb_user and mongodb_password if mongodb_useauth is set to false.
…and at least these in /etc/graylog2-elasticsearch.yml:
cluster.name: graylog2
The cluster name of your ElasticSearch cluster. All nodes that are discovered will join the cluster if they have the same cluster name. This must be the same cluster name your ElasticSearch nodes have configured.
Multicast/Unicast
The default setting of ElasticSearch is to use Multicast to discover other nodes. This can be useful but a bit hard to configure depending on your network architecture. Also think about your broadcast domains: If a developer starts up an ElasticSearch node and is in the same multicast broadcast domain, he will join your production cluster (if the cluster.name is the same). If you don’t plan to change or add ElasticSearch nodes regulary, I would recommend to disable multicast and enable unicast. Do this by setting discovery.zen.ping.multicast.enabled: false to true and add your ElasticSearch node hosts to discovery.zen.ping.unicast.hosts. Multicast should be fine for a first quick start though and have no problems discovering a node on localhost.
Example for unicast discovery of a standard ElasticSearch server on the same host:
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: [“127.0.0.1:9300”]
You might have to define different ports for your ElasticSearch node and the embedded graylog2-server ElasticSearch node if you are running them on the same host if you get port binding errors. It is recommended to have at least ElasticSearch running on a different host than graylog2-server.
A detailled documentation of all important configuration variables is available in this knowledge base and should be read after you have a first setup of Graylog2 running.
Starting the server
You need to have Java installed. Running the OpenJDK is totally fine and should be available on all platforms:
The first start should be performed without the ‘bin/graylog2ctl script to easily see warnings, errors or problems:
sudo java -jar graylog2-server.jar –debug
See the startup parameters page to learn more about available startup parameters. Note that you might have to be root to bind to port 514 for syslog.
You should see a line like this in the debug output if graylog2-server successfully connected to your ElasticSearch cluster:
2012-12-03 00:26:00,080 DEBUG: org.elasticsearch.transport.netty – [graylog2-server] connected to node [[Cyber][APKeeyD_T2uULknphyKlBg][inet[/192.168.1.6:9300]]]
This line indicates that your graylog2-server instance is up and ready to accept messages:
2012-12-03 00:26:01,684 INFO : org.graylog2.Core – Graylog2 up and running.
Let’s try that out! Send in a simple (not standard compliant) syslog message using netcat or your preferred socket communication tool:
~$ echo “<34> Hello Graylog2. Let’s be friends.” | nc -w 1 -u localhost 514
You should see this message being handled in your graylog2-server debug output:
2012-12-03 00:47:37,128 INFO : org.graylog2.inputs.syslog.SyslogProcessor – Date could not be parsed. Was set to NOW because allow_override_syslog_date is true.
2012-12-03 00:47:37,128 DEBUG: org.graylog2.inputs.syslog.SyslogProcessor – Adding received syslog message to process buffer: level: 2 | host: localhost | facility: security/authorization | add.: 0 | shortMessage: Hello Graylog2. Let’s be friends.
2012-12-03 00:47:37,128 DEBUG: org.graylog2.buffers.processors.ProcessBufferProcessor – Starting to process message .
2012-12-03 00:47:37,128 DEBUG: org.graylog2.buffers.processors.ProcessBufferProcessor – Applying filter [Rewriter] on message .
2012-12-03 00:47:37,129 DEBUG: org.graylog2.buffers.processors.ProcessBufferProcessor – Applying filter [Blacklister] on message .
2012-12-03 00:47:37,129 DEBUG: org.graylog2.buffers.processors.ProcessBufferProcessor – Applying filter [Tokenizer] on message .
2012-12-03 00:47:37,129 DEBUG: org.graylog2.filters.TokenizerFilter – Extracted <0> additional fields from message k=v pairs.
2012-12-03 00:47:37,129 DEBUG: org.graylog2.buffers.processors.ProcessBufferProcessor – Applying filter [StreamMatcher] on message .
2012-12-03 00:47:37,130 DEBUG: org.graylog2.filters.StreamMatcherFilter – Routed message to 1 streams.
2012-12-03 00:47:37,131 DEBUG: org.graylog2.buffers.processors.ProcessBufferProcessor – Applying filter [CounterUpdater] on message .
2012-12-03 00:47:37,131 DEBUG: org.graylog2.buffers.processors.ProcessBufferProcessor – Finished processing message. Writing to output buffer.
2012-12-03 00:47:37,131 DEBUG: org.graylog2.buffers.processors.OutputBufferProcessor – Processing message from OutputBuffer.
2012-12-03 00:47:37,131 DEBUG: org.graylog2.buffers.processors.OutputBufferProcessor – Writing message batch to [ElasticSearch Output]. Size <1>
2012-12-03 00:47:37,131 DEBUG: org.graylog2.outputs.ElasticSearchOutput – Writing <1> messages.
2012-12-03 00:47:37,141 DEBUG: org.graylog2.indexer.EmbeddedElasticSearchClient – Deflector index: Bulk indexed 1 messages, took 9 ms, failures: false
2012-12-03 00:47:37,141 DEBUG: org.graylog2.indexer.EmbeddedElasticSearchClient – Recent index: Bulk indexed 1 messages, took 9 ms, failures: false
2012-12-03 00:47:37,142 DEBUG: org.graylog2.buffers.processors.OutputBufferProcessor – Wrote message to all outputs. Finished handling.
Now exit and start the server using the control script:
cd bin/
sudo ./graylog2ctl start | stop | restart
This will start your graylog2-server in the background. Find a log in nohup.out in the same directory.
# Configure the local rsyslog server to talk to graylog2.
echo "*.* @localhost" | sudo tee /etc/rsyslog.d/32-graylog2.conf
sudo service rsyslog restart
Check that logs are making it into elasticsearch
curl -XGET 'http://localhost:9200/graylog2/_search?pretty=true' -d '
{
"query" : {
"matchAll" : {}
}
}'
Graylog2-web-interface install (download: https://github.com/Graylog2/graylog2-web-interface/downloads)
cd /opt/
sudo wget http://download.graylog2.org/graylog2-web-interface/graylog2-web-interface-0.11.0.tar.gz -O graylog2-web-interface.tar.gz
sudo tar -xf graylog2-web-interface.tar.gz
sudo mv graylog2-web-interface-0.11.0 graylog2-web-interface
sudo rm graylog2-web-interface.tar.gz
# Setup mongoid.yml
sudo nano /opt/graylog2-web-interface/config/mongoid.yml
production:
host: localhost
port: 27017
username:
password:
database: graylog2
# Ruby 1.9.3 install
sudo apt-get install libcurl4-openssl-dev libssl-dev zlib1g-dev
sudo apt-get install ruby1.9.3
OR from source
wget ftp://ftp.ruby-lang.org/pub/ruby/1.9/ruby-1.9.3-p194.tar.gz
tar -xvzf ruby-1.9.3-p194.tar.gz
cd ruby-1.9.3-p194
./configure –prefix=/usr –disable-install-doc
sudo make
sudo make install
# Choose the default ruby version
sudo update-alternatives –config ruby
# Install gems necessary to run graylog2
cd /opt/graylog2-web-interface
sudo gem install bundlersudo bundle install
sudo gem install rakesudo gem install passenger
# Passenger, Nginx install
sudo passenger-install-nginx-module –auto –auto-download –prefix=/opt/nginx
sudo nano /opt/nginx/conf/nginx.conf
http {
…
passenger_root /var/lib/gems/1.9.1/gems/passenger-3.0.19;
passenger_ruby /usr/bin/ruby1.9.1;
…
}server {
listen 80;
server_name localhost;
root /opt/graylog2-web-interface/public;
passenger_enabled on;
}Check the Users Guide for security and
optimization tips and other useful information:/var/lib/gems/1.8/gems/passenger-3.0.19/doc/Users guide Nginx.html
Enjoy Phusion Passenger, a product of Phusion (www.phusion.nl) 🙂
https://www.phusionpassenger.com
Setup a script to allow us to control Nginx. We’re going to grab this from Linode:
wget -O init-deb.sh http://library.linode.com/assets/660-init-deb.sh
sudo mv init-deb.sh /etc/init.d/nginx
sudo chmod +x /etc/init.d/nginx
sudo /usr/sbin/update-rc.d -f nginx defaults
You can now control Nginx with this script. To start and stop the server manually, you run:
sudo /etc/init.d/nginx stop
sudo /etc/init.d/nginx start
The script *************
#! /bin/sh
### BEGIN INIT INFO
# Provides: nginx
# Required-Start: $all
# Required-Stop: $all
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: starts the nginx web server
# Description: starts nginx using start-stop-daemon
### END INIT INFO
PATH=/opt/nginx/sbin:/sbin:/bin:/usr/sbin:/usr/bin
DAEMON=/opt/nginx/sbin/nginx
NAME=nginx
DESC=nginx
test -x $DAEMON || exit 0
# Include nginx defaults if available
if [ -f /etc/default/nginx ] ; then
. /etc/default/nginx
fi
set -e
case “$1” in
start)
echo -n “Starting $DESC: “
start-stop-daemon –start –quiet –pidfile /opt/nginx/logs/$NAME.pid \
–exec $DAEMON — $DAEMON_OPTS
echo “$NAME.”
;;
stop)
echo -n “Stopping $DESC: “
start-stop-daemon –stop –quiet –pidfile /opt/nginx/logs/$NAME.pid \
–exec $DAEMON
echo “$NAME.”
;;
restart|force-reload)
echo -n “Restarting $DESC: “
start-stop-daemon –stop –quiet –pidfile \
/opt/nginx/logs/$NAME.pid –exec $DAEMON
sleep 1
start-stop-daemon –start –quiet –pidfile \
/opt/nginx/logs/$NAME.pid –exec $DAEMON — $DAEMON_OPTS
echo “$NAME.”
;;
reload)
echo -n “Reloading $DESC configuration: “
start-stop-daemon –stop –signal HUP –quiet –pidfile /opt/nginx/logs/$NAME.pid \
–exec $DAEMON
echo “$NAME.”
;;
*)
N=/etc/init.d/$NAME
echo “Usage: $N {start|stop|restart|reload|force-reload}” >&2
exit 1
;;
esac
exit 0
****************************
sudo service nginx start
nginx.conf (https://raw.github.com/matthieudolci/Scripts-GrayLog2/master/nginx/nginx.conf)
#user nobody;
worker_processes 4;
#error_log logs/error.log;
#error_log logs/error.log notice;
#error_log logs/error.log info;
#pid logs/nginx.pid;
events {
worker_connections 1024;
}
http {
passenger_root /var/lib/gems/1.9.1/gems/passenger-3.0.19;
passenger_ruby /usr/bin/ruby1.9.1;
include mime.types;
default_type application/octet-stream;
access_log logs/access.log;
sendfile on;
keepalive_timeout 65;
tcp_nodelay on;
gzip on;
gzip_comp_level 2;
gzip_proxied any;
server {
listen 80;
server_name graylog; # or localhost
root /opt/graylog2-web-interface/public;
passenger_enabled on;
rails_spawn_method smart;
location ~* \.(ico|css|js|gif|jpe?g|png)$ {
expires max;
add_header Pragma public;
add_header Cache-Control “public, must-revalidate, proxy-revalidate”;
}
}
}
Installing Bundler for Ruby in Ubuntu 10.04
When attempting to install Bundler for Ruby on Ubuntu 10.04, I got the following error:
shell> sudo gem install bundler
ERROR: Error installing bundler:
bundler requires RubyGems version >= 1.3.6
Running sudo gem -v I saw that I had 1.3.5. To get around this, simply install the available updater gem, then run it:
shell> sudo gem install rubygems-update
shell> sudo /var/lib/gems/1.8/bin/update_rubygems
Now running gem -v I see that I have 1.8.15 and I am able to install bundler:
shell> gem install bundler
Fetching: bundler-1.0.21.gem (100%)
Successfully installed bundler-1.0.21
1 gem installed
Installing ri documentation for bundler-1.0.21…
Installing RDoc documentation for bundler-1.0.21…
After searching for a solution on how to use graylog2 to receive syslog messages from Cisco switching equipment found a solution. Looks like the problem was related with the counter/sequence number that the equipment was sending by default.
If you see in graylog2 messages without a correct Host coming from a Cisco equipment, try the following config in the same equipment:
conf t
no logging message-counter syslog
no service sequence-numbers
logging trap “level”
logging “syslog server ip”
Install MongoDB
1
2
3
4
|
echo "deb http://downloads-distro.mongodb.org/repo/ubuntu-upstart dist 10gen" > /etc/apt/sources.list.d/10gen.list apt-get update apt-get install mongodb18-10gen start mongod |
If you feel the need to mess around with MongoDB settings, they can be found in:
/etc/mongodb.conf – although they default settings should be fine as they are.
We will proceed by creating a user for greylog in the MongoDB console:
In the shell enter:
1
2
3
4
5
6
7
|
mongo use admin db.addUser('admin', 'grayloguser-password') db.auth('admin', 'grayloguser-password') use graylog2 db.addUser('grayloguser', 'grayloguser-password') db.auth('grayloguser', 'grayloguser-password') |
Remember to note down the user and password you chose.
Install Java
1
|
apt-get install openjdk-6-jre |
Download and install elastic search to /opt
1
2
3
|
tar zxvf elasticsearch-0.19.0.tar.gz ln -s /opt/elasticsearch-0.19.0 /opt/elasticsearch |
Start it
1
|
/opt/elasticsearch/bin/elasticsearch |
Graylog2-server Installation
- Download the latest graylog2-server tarball into /opt: https://github.com/Graylog2/graylog2-server/downloads
- Install into /opt/graylog2-server
1
2
3
4
|
cd /opt/ tar zxvf graylog2-server-0.9.6p1.tar.gz ln -s /opt/graylog2-server-0.9.6p1 /opt/graylog2-server |
- Configure /etc/graylog2.conf
1
2
|
cp /opt/graylog2-server/graylog2.conf.example /etc/graylog2.conf edit /etc/graylog2.conf, change user and password to your likings |
- Create /etc/init.d/graylog2-server
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
|
#!/bin/sh # # graylog2-server: graylog2 message collector # # chkconfig: - 98 02 # description: This daemon listens for syslog and GELF messages and stores them in mongodb # CMD=$1 NOHUP=` which nohup ` JAVA_HOME= /usr/lib/jvm/java-6-openjdk-amd64/ JAVA_CMD=$JAVA_HOME /bin/java GRAYLOG2_SERVER_HOME= /opt/graylog2-server start() { echo "Starting graylog2-server ..." $NOHUP $JAVA_CMD -jar $GRAYLOG2_SERVER_HOME /graylog2-server .jar > /var/log/graylog2 .log 2>&1 & } stop() { "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "PID" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" =` cat /tmp/graylog2 .pid` echo "Stopping graylog2-server ($PID) ..." kill $PID } restart() { echo "Restarting graylog2-server ..." stop start } case "$CMD" in start) start ;; stop) stop ;; restart) restart ;; *) echo "Usage $0 {start|stop|restart}" RETVAL=1 esac |
- Create /etc/logrotate.d/graylog2-server
1
2
3
4
5
6
7
8
9
|
/var/log/graylog2.log { daily rotate 90 copytruncate delaycompress compress notifempty missingok } |
- Start graylog2-server
1
2
3
4
5
|
chmod +x /etc/init.d/graylog2-server service graylog2-server start ps -ef | grep graylog2 root 21159 1 0 15:15 pts/0 00:00:07 java -jar ../graylog2-server.jar |
Graylog2-web-interface install
- Install pre-reqs
1
|
apt-get install libapache2-mod-passenger ruby rubygems ruby-dev |
- Create graylog user
1
|
useradd -m graylog |
- Download the latest graylog2-web-interface tarball: https://github.com/Graylog2/graylog2-web-interface/downloads
1
2
3
4
5
|
cd /opt/ wget https://github.com/downloads/Graylog2/graylog2-web-interface/graylog2-web-interface-0.9.601.tar.gz tar zxvf graylog2-web-interface-0.9.6p1.tar.gz ln -s /opt/graylog2-web-interface-0.9.6p1 /opt/graylog2-web-interface chown -R graylog:graylog /opt/graylog2-web-* |
- Install gems necessary to run graylog2 using Bundler in the ruby 1.9.2 environment
1
2
3
|
cd /opt/graylog2-web-interface gem install bundler bundle install |
- Configure the web interface’s Mongo settings
1
|
edit /opt/graylog2-web-interface/config/mongoid.yml |
The production: settings should look like this:
1
2
3
4
5
6
|
production: host: 127.0.0.1 port: 27017 username: grayloguser password: grayloguser-password database: graylog2 |
NOTE: MongoDB creates databases on the fly, so you do not need to create the database before specifying it within the config! (Unlike MYSQL or MSSQL where you are required to have created the database beforehand)
- We will now configure the database to our needs:
1
2
|
cd /opt/graylog2-web-interface bundle exec rake db:mongoid:create_indexes RAILS_ENV=production |
- Configure the graylog2-web-interface by editing email.yml, general.yml
- Start the graylog2-web-interface
1
2
|
cd /opt/graylog2-web-interface script/rails server -e production |
It will output on the lines of:
1
2
3
4
5
6
7
|
=> Booting WEBrick => Rails 3.1.3 application starting in production on http://0.0.0.0:3000 => Call with -d to detach => Ctrl-C to shutdown server [2012-08-15 12:24:34] INFO WEBrick 1.3.1 [2012-08-15 12:24:34] INFO ruby 1.8.7 (2011-06-30) [x86_64-linux] [2012-08-15 12:24:34] INFO WEBrick::HTTPServer#start: pid=10287 port=3000 |
Connect to http://:3000. If everything is working, graylog2 will ask you to create the first user. Shutdown graylog2 (ctrl+c) after you create the first user
- Create /etc/logrotate.d/graylog2-web-interface
1
2
3
4
5
6
7
8
9
|
/opt/graylog2-web-interface/log/*log size=256M rotate 90 copytruncate delaycompress compress notifempty missingok } |
- Setup the webinterface to be served with Apache Mod Passenger
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
|
cat >> /etc/apache2/sites-available/graylog2 <<EOF Listen 8080 <VirtualHost *:8080> ServerName server.domain.com ServerAlias server.domain.com DocumentRoot /opt/graylog2-web-interface/public AddDefaultCharset UTF-8 RailsAutoDetect on PassengerAppRoot /opt/graylog2-web-interface ErrorLog /var/log/apache2/gl2-error.log LogLevel warn CustomLog /var/log/apache2/gl2-access.log combined </VirtualHost> EOF |
- Enable the newly created site (Basically symlinking it to /etc/apache2/sites-enabled/…)
1
|
a2ensite graylog2 |
- Restart apache
1
|
service apache2 restart |
- You should now be able to view the webinterface at http://server.domain.com:8080 or at the other configured address