By Ghulam Qadir, Alibaba Cloud Tech Share Author. Tech Share is Alibaba Cloud's incentive program to encourage the sharing of technical knowledge and best practices within the cloud community.
Graylog is a powerful open-source log management platform that aggregates and extracts important data from server logs, which are often sent using the Syslog protocol. It also allows you to search and visualize the logs in a web interface.
Graylog is compatible and works well with Alibaba Cloud Elastic Compute Service (ECS) instances. In this tutorial, we'll install and configure Graylog on Ubuntu 16.04, and set up a simple input that receives system logs.
Your ECS Ubuntu 16.04 server must have at least 2 GB of RAM, private networking enabled, and a non-root user set up.
After completing the prerequisites, log in as root user with your root username & password via SSH client (e.g. Putty) or VNC console available in your Alibaba Cloud account dashboard.
Before you installing Graylog, you'll need:
The easiest option for installing Java is using the version packaged with Ubuntu. Specifically, this will install OpenJDK 8, the latest and recommended version.
First, update the package index.
sudo apt-get updateNext, install Java. Specifically, this command will install the Java Runtime Environment (JRE).
sudo apt-get install default-jreThere is another default Java installation called the JDK (Java Development Kit). The JDK is usually only needed if you are going to compile Java programs or if the software that will use Java specifically requires it.
The JDK does contain the JRE, so there are no disadvantages if you install the JDK instead of the JRE, except for the larger file size.
You can install the JDK with the following command:
Sudo apt-get install default-jdkIf you want to install the Oracle JDK, which is the official version distributed by Oracle, you will need to follow a few more steps.
First, add Oracle's PPA, then update your package repository.
sudo add-apt-repository ppa:webupd8team/java
sudo apt-get updateThen, depending on the version you want to install, execute one of the following commands:
This is the latest stable version of Java at time of writing, and the recommended version to install. You can do so using the following command:
sudo apt-get install oracle-java8-installerThis is a developer preview and the general release is scheduled for March 2017. It's not recommended that you use this version because there may still be security issues and bugs. There is more information about Java 9 on the official JDK 9 website.
To install JDK 9, use the following command:
sudo apt-get install oracle-java9-installerThere can be multiple Java installations on one server. You can configure which version is the default for use in the command line by using update-alternatives, which manages which symbolic links are used for different commands.
sudo update-alternatives –-config javaThe output will look something like the following. In this case, this is what the output will look like with all Java versions mentioned above installed.
Output
There are 5 choices for the alternative java (providing /usr/bin/java).
Section Path Priority Status
* 0 /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java 1081 auto mode
1 /usr/lib/jvm/java-6-oracle/jre/bin/java 1 manual mode
2 /usr/lib/jvm/java-7-oracle/jre/bin 2 manual mode
3 /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java 1081 manual mode
4 /usr/lib/jvm/java-8-oracle/jre/bin/java 3 manual mode
5 /usr/lib/jvm/java-9-oracle/bin/java 4 manual mode
Press <enter> to keep the current choice[*], or type selection number:You can now choose the number to use as a default. This can also be done for other Java commands, such as the compiler (javac), the documentation generator (javadoc), the JAR signing tool (jarsigner), and more. You can use the following command, filling in the command you want to customize.
sudo update-alternatives –-config commandMany programs, such as Java servers, use the JAVA_HOME environment variable to determine the Java installation location. To set this environment variable, we will first need to find out where Java is installed. You can do this by executing the same command as in the previous section:
sudo update-alternatives –-config javaCopy the path from your preferred installation and then open /etc/environment using nano or your favorite text editor.
sudo nano /etc/environmentAt the end of this file, add the following line, making sure to replace the highlighted path with your own copied path.
/etc/environment
JAVA_HOME="/usr/lib/jvm/java-8-oracle"Save and exit the file, and reload it.
source /etc/environmentYou can now test whether the environment variable has been set by executing the following command:
echo $JAVA_HOMEThis will return the path you just set.
Elasticsearch is one of the main component which requires Graylog to run, acts as a search server, offers a real-time distributed search and analytics with the RESTful web interface. Elasticsearch stores all the logs sent by the Graylog server and displays the messages whenever user request over the built-in web interface.
This guide covers configuration settings that are required for Graylog.
Let's install the Elasticsearch. First download and install GPG signing key.
wget –q0 – https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key
add –Configure Elasticsearch repository by running below command.
Echo "deb https://packages.elastic.co/elasticsearch/2.x/debian stable main"
| sudo tee-a /etc/apt/sources.list.d/elasticsearch.listUpdate repository cache and install Elasticsearch.
sudo apt-get update && sudo apt-get install –y elasticsearchMake Elasticsearch to start automatically on the system startup.
sudo systemctl enable elasticsearchWe need to modify the Elasticsearch configuration file so that the cluster name matches the one set in the Graylog configuration file. To keep things simple, we'll set the Elasticsearch cluster name to the default Graylog name of graylog. You may set it to whatever you wish, but make sure you update the Graylog configuration file to reflect that change.
Open the Elasticsearch configuration file in your editor:
sudo nano /etc/elasticsearch/elasticsearch.ymlFind the following line:
/etc/elasticsearch/elasticsearch.yml
cluster.name: <CURRENT CLUSTER NAME>Change the cluster.name value to graylog:
/etc/elasticsearch/elasticsearch.yml
cluster.name: graylogSave the file and exit your editor.
Since we modified the configuration file, we have to restart the service for the changes to take effect.
sudo systemctl restart elasticsearchNow that you have configured Elasticsearch, let's move on to installing Graylog on Elasticsearch.
Disable dynamic scripts to avoid remote execution, by adding the following lines to the server.conf.
script.inline: false
script.indexed: false
script.file: falseRestart the Elasticsearch service to read the new configurations.
sudo service elasticsearch restartWait at least a minute to let the Elasticsearch get fully restarted. Elastisearch should be now listening on 9200 for processing HTTP request, use a CURL to check the response.
Ensure that cluster name shows as "graylog"
curl –X GET http://localhost:9200
{
"name" : "Marvin Flumm"
"cluster_name" : "graylog"
"version" : {
"number" : "2.3.3",
"build_hash" : "218bdf10790eef486ff2c41a3df5cfa32dadcfde",
"build_timestamp" : "2018-07-28T15:40:04Z",
"build_snapshot" : false,
"lucene_version" : "5.5.0"
},
"tagline" : "You Know, for Search"
}Optional: Test the health of Elasticsearch cluster, make sure the output yields the cluster status as "green"
curl –XGET 'http://localhost:9200/_cluster/health?pretty=true'
{
"cluster_name" : "graylog"
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 1,
"active_primary_shards" : 1,
"active_shards" : 1,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}Download and install the latest MongoDB from the official website. Import public key on the terminal to begin.
sudo apt-key adv –keyserver hkp://keyserver.ubuntu.com:80 –-recv EA312927Add mongodb repository by creating the /etc/apt/sources.list.d/mongodb-org.list file using following command.
echo "deb http://repo.mongodb.org/apt/debian wheezy/mongodb-org/3.2 main" | sudo tee
/etc/apt/sources.list.d/mongodb-org.listInstall MongoDB using the following command.
sudo apt-get update && sudo apt-get install –y mongodb-orgStart the MongoDB and enable it on the system start-up.
sudo systemctl start mongod
sudo systemctl enable mongodIn this step, we we'll install the Graylog server.
First, download the package file containing the Graylog repository configuration. Visit the Graylog download page to find the current version number. We'll use version 2.2 for this tutorial.
wget https://packages.graylog2.org/repo/packages/graylog-2.2-
repository_latest.debNext, install the repository configuration from the .deb package file, again replacing 2.2 with the version you downloaded.
sudo dpkg –I graylog-2.2-repository_latest.debNow that the repository configuration has been updated, we have to fetch the new list of packages. Execute this command:
sudo apt-get updateNext, install the graylog-server package:
sudo apt-get install graylog-serverLastly, start Graylog automatically on system boot with this command:
sudo systemctl enable graylog-server.serviceGraylog is now successfully installed, but it's not started yet. We have to configure it before it will start.
You must set a secret to secure the user passwords, use the pwgen command to the same.
pwgen – N 1 –s 96
OH9wXpsNZVBA8R5vJQSnkhTB1qDOjCxAh3aE3LvXddtfDlZlKYEyGS24BJAiIxI0sbSTSPovTTnhLkkrUvhSSxodTlzDi5gPIf you get an error like "pwgen: command not found", install pwgen using the following command.
sudo apt-get install pwgenPlace the secret like below.
password_secret = OH9wXpsNZVBA8R5vJQSnkhTB1qDOjCxAh3aE3LvXddtfDlZlKYEyGS24BJAiIxI0sbSTSPovTTnhLkkrUvhSSxodTlzDi5gP Next is to set a hash (sha256) password for the root user (not to be confused with the system user, root user of graylog is admin). You will need this password to login into the web interface, admin's password can't be changed using web interface; you must edit this variable to set.
Replace "yourpassword" with your own.
echo –n yourpassword | sha256sum
e3c652f0ba0b4801205814f8b6bc49672c4c74e25b497770bb89b22cdeb4e951Place the hash password.
root_password_sha2 = e3c652f0ba0b4801205814f8b6bc49672c4c74e25b497770bb89b22cdeb4e951You can setup email address admin user.
root_email = "datamounts@gmail.com"Set time zone of root (admin) user.
root_timezone = UTCGraylog server will try to find the Elasticsearch nodes automatically by using multicast mode. But when it comes to larger network, it is recommended to use unicast mode which is best suited one for production.
Add the following entry to graylog server.conf file, replace ipaddress with your ipaddress. You can add multiple hosts with comma separated.
elasticsearch_discovery_zen_ping_unicast_hosts = ipaddress:9300Set only one master node by defining the below variable, the default setting is true.
If you add any second Graylog node, set it to false to make the node as a slave. Master node does some periodic tasks that slave nodes won't perform.
is_master = trueSet the number of log messages to keep per index; it is recommended to have several smaller indices instead of larger ones.
elasticsearch_max_docs_per_index = 20000000The following parameter defines to have a total number of indices, if this number is reached old index will be deleted.
elasticsearch_max_number_of_indices = 20Shards setting rely on the number of nodes in the particular Elasticsearch cluster, if you have only one node, set it as 1.
elasticsearch_shards = 1This the number of replicas for your indices, if you have only one node in Elasticsearch cluster; set it as 0.
From the version 2.x, no more extra web interface component, the web interface is being served directly by Graylog server.
Configure Graylog web interface by editing the server.conf file.
sudo nano /etc/graylog/server/server.confModify the below entries to let Graylog Web Interface to connect to the Graylog server.
rest_listen_uri = http://your-server-ip:12900/
web_listen_uri = http://your-server-ip:9000/Restart Graylog service.
sudo systemctl daemon-reload
sudo systemctl restart graylog-serverMake Graylog server to start automatically on system startup.
sudo systemctl enable graylog-serverYou can check out the server startup logs; it will be useful for you to troubleshoot Graylog in case of any issue.
sudo tailf /var/log/graylog-server/server.logOn the successful start of graylog-server, you should get the following message in the log file.
2018-07-28T08:21:41.538Z INFO [ServerBootstrap] Graylog server up and running.The web interface will now be listening on port 9000, point your browser to http://ip-add-ress:9000.
Login with username "admin" and the password you configured at root_password_sha2 on server.conf.
Once you logged in, you would see the getting started page.
Click on System/Overview to know the status of Graylog server.
Click on System/Overview to know the status of Graylog server.
Graylog inputs need to be configured to receive the logs from the external source, i.e., Syslog or any logging system.
Click System –> Inputs –> Select Syslog UDP and then click Launch new input. Fill with the values in the screen like below.
Once you have created the inputs, configure rsyslog or forward any system logs to your–ip-address:1514
Following screenshot shows the logs received by Graylog (Graylog console –> Search).
That's all! You have successfully installed Graylog 2.0.3 on Ubuntu 16.04.
You now have a working Graylog server with an input source that can collect logs from other servers.
Next, you might want to look into setting up dashboards, alerts, and streams. Dashboards provide a quick overview of your logs. Streams categorize messages, which you can monitor with alerts. To learn more about configuring the more advanced features of Graylog, you can find instructions in the Graylog documentation.
Deploying WSFC and the SQL Server Failover Cluster on Alibaba Cloud ECS
2,599 posts | 762 followers
FollowAlibaba Clouder - November 13, 2019
Alibaba Clouder - February 13, 2018
Alibaba Clouder - April 23, 2019
Alibaba Clouder - May 7, 2019
Alibaba Clouder - June 4, 2019
Alibaba Clouder - February 13, 2019
You have mistakes in a lot of your commands but this is good article!!!! Examples . "Echo" "tee-a" the "-y (before the install)" the "add - (should replace - with name of file?) etc..
2,599 posts | 762 followers
Follow
ECS(Elastic Compute Service)
Elastic and secure virtual cloud servers to cater all your cloud hosting needs.
Learn More
OSS(Object Storage Service)
An encrypted and secure cloud storage service which stores, processes and accesses massive amounts of data from anywhere in the world
Learn MoreLearn More
More Posts by Alibaba Clouder
5953239330871758 October 12, 2018 at 7:58 am
With this Tutorial, you install Products that have already reached their end of life.> This tutorial uses Elasticsearch 2.4.4 and Graylog 2.2In addition, only Java 8 is supported.Just follow the official documentation guide and you will have one installation with current software versions and installed that in the supported and tested way: http://docs.graylog.org/en/2.4/pages/installation/os/ubuntu.html