ELK+RabbitMQ单机部署收集nginx访问日志for Centos7

安装前准备

1
2
3
4
5
6
1、修改主机名:
hostnamectl set-hostname elk

2、配置hosts:
vim /etc/hosts
192.168.10.21 elk

安装Elasticsearch

1
2
3
4
5
mkdir -p /a01/apps/apps_src
cd /a01/apps/apps_src/
mkdir -p /usr/local/java
tar -zxvf jdk-8u151-linux-x64.tar.gz -C /usr/local/java/
vim /etc/profile
1
2
3
4
JAVA_HOME=/usr/local/java/jdk1.8.0_151
PATH=$JAVA_HOME/bin:$PATH
CLASSPATH=$JAVA_HOME/jre/lib/ext:$JAVA_HOME/lib/tools.jar
export PATH JAVA_HOME CLASSPATH
1
2
3
4
5
6
7
8
source /etc/profile
java -version
useradd elk
tar -zxvf elasticsearch-6.1.1.tar.gz -C /a01/apps/
mv /a01/apps/elasticsearch-6.1.1/ /a01/apps/elasticsearch
cd /a01/apps/elasticsearch/config/
cp elasticsearch.yml elasticsearch.yml.bak
vim elasticsearch.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
cluster.name: nongkaige-cluster
node.name: node-21
path.data: /a01/apps/elasticsearch/data
path.logs: /var/log/elasticsearch
path.work: /a01/apps/elasticsearch/word
path.plugins: /a01/apps/elasticsearch/plugins
bootstrap.memory_lock: true
bootstrap.system_call_filter: false
network.host: 192.168.10.21
http.port: 9200
transport.tcp.port: 9300
transport.tcp.compress: true
discovery.zen.ping.multicast.enabled: false
discovery.zen.fd.ping_timeout: 100s
discovery.zen.ping.timeout: 100s
discovery.zen.minimum_master_nodes: 2
discovery.zen.ping.unicast.hosts: ["192.168.10.21"]
1
2
3
4
5
mkdir -p /a01/apps/elasticsearch/data
mkdir -p /var/log/elasticsearch

chown -R elk:elk /a01/apps/elasticsearch/
chown -R elk:elk /var/log/elasticsearch/
1
2
3
4
5
6
7
8
9
vim /etc/security/limits.conf
* soft nproc 4096
* hard nproc 4096
* soft nofile 65536
* hard nofile 65536
elk soft nproc 4096
elk hard nproc 4096
elk soft memlock unlimited
elk hard memlock unlimited
1
2
vim /etc/sysctl.conf
vm.max_map_count=262144
  • sysctl -p生效,然后断开重新登陆即可

尝试启动是否成功

PRoUaj.png

报错:

1
2
[WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-1] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: unknown setting [discovery.zen.ping.multicast.enabled] please check that any required plugins are installed, or check the breaking changes documentation for removed settings

解决:

  • 配置文件中删除掉该参数[discovery.zen.ping.multicast.enabled],elasticsearch5.0之后已经没有该参数

PRoBR0.png

报错:

1
2
[WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-1] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: unknown setting [discovery.zen.ping.timeout] did you mean any of [discovery.zen.ping_timeout, discovery.zen.fd.ping_timeout, discovery.zen.join_timeout, discovery.zen.publish_timeout, discovery.zen.commit_timeout]?

解决:

  • 删除该参数[discovery.zen.ping.timeout]

PRoysU.png

报错:

1
2
[WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-1] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: unknown setting [path.plugins] please check that any required plugins are installed, or check the breaking changes documentation for removed settings

PRogZ4.png

报错:

1
2
[WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-1] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: unknown setting [path.work] please check that any required plugins are installed, or check the breaking changes documentation for removed settings

PRoRo9.png

报错:

1
[WARN ][o.e.d.z.ZenDiscovery     ] [node-1] not enough master nodes discovered during pinging (found [[Candidate{node={node-1}{9bMckZcHQhmkuVTBHcDIVA}{bj9pFgIYTGK2TghNIr-CJg}{192.168.10.21}{192.168.10.21:9300}, clusterStateVersion=-1}]], but needed [2]), pinging again

解决:

  • 配置文件中写了discovery.zen.minimum_master_nodes:2,但是找不到两台,所以报错,改成1即可

最后的配置文件为

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
vim /a01/apps/elasticsearch/config/elasticsearch.yml

cluster.name: nongkaige-cluster
node.name: node-1
path.data: /a01/apps/elasticsearch/data
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
bootstrap.system_call_filter: false
network.host: 192.168.10.21
http.port: 9200
transport.tcp.port: 9300
transport.tcp.compress: true
discovery.zen.fd.ping_timeout: 100s
discovery.zen.minimum_master_nodes: 1
discovery.zen.ping.unicast.hosts: ["192.168.10.21"]

编写elasticsearch加入系统服务

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
ln -s /usr/local/java/jdk1.8.0_151/bin/java /usr/local/sbin/
vim /etc/systemd/system/elasticsearch.service

[Unit]
Description=Elasticsearch
Wants=network-online.target
After=network-online.target

[Service]
User=elk
Group=elk

ExecStart=/a01/apps/elasticsearch/bin/elasticsearch

LimitNOFILE=65536
LimitNPROC=4096

[Install]
WantedBy=multi-user.target

chmod 754 /etc/systemd/system/elasticsearch.service
systemctl enable elasticsearch
systemctl start elasticsearch
systemctl status elasticsearch

浏览器访问:

PRofiR.png

安装Kibana

1
2
3
4
5
6
7
8
9
tar -zxvf kibana-6.1.1-linux-x86_64.tar.gz -C /a01/apps/
mv /a01/apps/kibana-6.1.1-linux-x86_64/ /a01/apps/kibana

vim /a01/apps/kibana/config/kibana.yml

server.port: 5601
server.host: "192.168.10.21"
elasticsearch.url: http://192.168.10.27:9200 #Elasticsearch的ip地址
kibana.index: ".kibana" #创建索引

编写Kibana加入系统服务

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
vim /etc/systemd/system/kibana.service

[Unit]
Description=Kibana
Wants=network-online.target
After=network-online.target

[Service]
ExecStart=/a01/apps/kibana/bin/kibana


[Install]
WantedBy=multi-user.target

chmod 754 /etc/systemd/system/kibana.service
systemctl enable kibana
systemctl start kibana
systemctl status kibana

安装RabbitMQ

1、安装erlang
1
2
3
4
5
6
7
tar -zxvf otp_src_20.2.tar.gz
cd otp_src_20.2
yum -y install ncurses-devel openssl-devel unixODBC-devel xsltproc fop
./configure --prefix=/a01/apps/erlang --without-javac
make -j 2 && make install
echo "export PATH=$PATH:/a01/apps/erlang/bin">>/etc/profile
source /etc/profile
2、安装RabbitMQ
1
2
3
4
5
xz -d rabbitmq-server-generic-unix-3.7.0.tar.xz
tar -xvf rabbitmq-server-generic-unix-3.7.0.tar
mv rabbitmq_server-3.7.0/ /a01/apps/rabbitmq
echo "export PATH=$PATH:/a01/apps/rabbitmq/sbin">>/etc/profile
source /etc/profile
3、配置
1
2
3
4
5
6
7
8
9
10
vim /a01/apps/rabbitmq/sbin/rabbitmq-defaults
```

![PRooQK.png](https://s1.ax1x.com/2018/08/16/PRooQK.png)

```bash
mkdir -p /etc/rabbitmq
cd /etc/rabbitmq/
touch rabbitmq-env.conf
touch rabbitmq.conf
1
2
3
4
5
6
7
8
9
10
vim /etc/rabbitmq/rabbitmq-env.conf
RABBITMQ_CONFIG_FILE=/etc/rabbitmq/rabbitmq.conf #配置文件
RABBITMQ_MNESIA_BASE=/a01/apps/rabbitmq/data #节点数据库目录
RABBITMQ_LOG_BASE=/var/log/rabbitmq #日志目录
RABBITMQ_PLUGINS_DIR=/a01/apps/rabbitmq/plugins #插件目录
RABBITMQ_NODE_PORT=5672 #默认监听端口
RABBITMQ_PID_FILE=/var/run/rabbitmq/rabbitmq.pid #进程pid文件
RABBITMQ_LOGS=/var/log/rabbitmq/erlang.log #erlang日志
RABBITMQ_SASL_LOGS=/var/log/rabbitmq/sasl.log #sasl日志
RABBITMQ_CONF_ENV_FILE=/etc/rabbitmq/rabbitmq-env.conf #默认env配置文件
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
vim /etc/rabbitmq/rabbitmq.conf
listeners.tcp.default = 5672 #监听AMQP连接(不含TLS)的端口
num_acceptors.tcp = 10 #接受TCP侦听器连接的Erlang进程的数量
handshake_timeout = 10000 #AMQP 0-9-1握手的最大时间,以毫秒为单位
listeners.ssl = none #ssl连接
vm_memory_calculation_strategy = allocated #使用Erlang内存分配器统计内存使用情况
vm_memory_high_watermark_paging_ratio = 0.75 #当内存使用超过总内存百分比多少时,队列开始将消息持久化到磁盘以释放内存
disk_free_limit.absolute = 2GB #RabbitMQ存储数据的分区的磁盘空间限制。当可用的磁盘空间低于这个限制时,就会触发流控制。
log.file.level = info #日志记录,可设置级别:‘none’、‘error’、‘warning’、‘info’、‘debug’,其中下一层级别的日志输出均包含上层级别的日志输出,比如warning包含warning和error,none为不输出日志
channel_max = 0 #与客户协商的最大通道数量。设置为0意味着“无限”
#channel_operation_timeout = 15000
heartbeat = 600 #在有大量连接的情况下,禁用心跳可能改善性能,但可能会导致连接在关闭非活动连接的网络设备的出现
default_vhost = / #当RabbitMQ创建一个新的数据库时,创建一个虚拟主机。交换amq.rabbitmq.logwill存在于这个虚拟主机中
default_user = guest #当RabbitMQ创建一个新的数据库时,需要创建的用户名
default_pass = lianni@com #默认用户guest的密码
default_user_tags.administrator = false #默认用户的角色是否为管理员
default_permissions.configure = .* #在创建时分配给默认用户的权限
default_permissions.read = .*
default_permissions.write = .*
loopback_users.guest = true #是否限制guest用户只能本地登录,其它用户一样的设置
collect_statistics = none #统计数据收集模式,none为不发布统计数据,coarse为发出的每个队列/通道/连接统计信息,fine为发出的每条数据
collect_statistics_interval = 5000 #统计信息收集间隔,以毫秒为单位
#management_db_cache_multiplier = 3
#auth_mechanisms.1 = PLAIN
#auth_mechanisms.2 = AMQPLAIN
#auth_backends.1 = internal
reverse_dns_lookups = false #客户端连接执行反向DNS查询
tcp_listen_options.backlog = 128 #默认的套接字选项,通常不改变这些参数
tcp_listen_options.nodelay = true
tcp_listen_options.linger.on = true
tcp_listen_options.linger.timeout = 0
tcp_listen_options.exit_on_close = false
hipe_compile = false #设置为true,使用HiPE预编译RabbitMQ的部分,这是Erlang的即时编译器。这将增加服务器的吞吐量,以增加启动时间的成本。您可能会看到,在启动时延迟几分钟,您的性能会提高20-50%。这些数据是高度工作负载和硬件依赖的。HiPE支持可能不会编译到您的Erlang安装中。如果不是这样,启用这个选项只会导致一个警告消息被显示,而启动将照常进行。例如,Debian/Ubuntu用户需要安装erlangbase-base-hipe包。
cluster_partition_handling = ignore #如何处理网络分区
cluster_keepalive_interval = 100000 #节点多久发送keepalive消息到其它节点
queue_index_embed_msgs_below = 4096 #在4096字节数中,消息将被直接嵌入到队列索引中
mnesia_table_loading_retry_timeout = 30000 #在集群中等待每个重试时间
mnesia_table_loading_retry_limit = 5 #等待集群可用时,需要重试的次数
proxy_protocol = false #如果设置为true,则打开AMQP连接时,RabbitMQ将期望首先发送代理协议头。这意味着在RabbitMQ前面建立一个符合代理协议的反向代理(例如HAproxy 或AWS ELB)。启用代理协议后,客户端无法直接连接到RabbitMQ,因此所有连接都必须通过反向代理。

先启动rabbitmq:

1
rabbitmq-server start

启用插件

1
rabbitmq-plugins enable rabbitmq_management

添加用户:

1
2
3
rabbitmqctl add_user nong nong
rabbitmqctl set_permissions -p / nong '.*' '.*' '.*'
rabbitmqctl set_user_tags nong administrator

编写rabbitmq加入系统服务

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
ln -s /a01/apps/erlang/bin/* /usr/bin/
vim /etc/systemd/system/rabbitmq.service

[Unit]
Description=RabbitMQ broker
After=network.target
Wants=network.target

[Service]
User=root
Group=root
Type=forking
WorkingDirectory=/a01/apps/rabbitmq
ExecStart=/a01/apps/rabbitmq/sbin/rabbitmq-server -detached
ExecStop=/a01/apps/rabbitmq/sbin/rabbitmqctl stop

[Install]
WantedBy=multi-user.target

chmod 754 /etc/systemd/system/rabbitmq.service
systemctl enable rabbitmq

先把之前启动的rabbitmq关闭,然后再使用以下命令启动rabbitmq

1
2
3
kill -s 9 2530
systemctl start rabbitmq
systemctl status rabbitmq

浏览器访问IP:15672

PRo7LD.png

PRobee.png

安装Logstash服务端

1
2
tar -zxvf logstash-6.1.1.tar.gz -C /a01/apps/
mv /a01/apps/logstash-6.1.1/ /a01/apps/logstash

编写服务端配置文件收集nginx日志

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
mkdir -p /etc/logstash
vim /etc/logstash/logstash.conf

input {
rabbitmq {
host => "192.168.10.21"
port => 5672
key => "nginx_access"
user => "nong"
password => "nong"
exchange => "amq.direct"
exchange_type => "direct"
}


}



output {
if [type] == "nginx_access" {
elasticsearch {
hosts => "192.168.10.21:9200"
index => "nginx_access-%{+YYYY.MM.dd}"
}
}

}

检测配置文件是否正确

1
/a01/apps/logstash/bin/logstash -t -f /etc/logstash/logstash.conf

PRoLod.png

编写logstash加入系统服务

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
vim /etc/systemd/system/logstash.service

[Unit]
Description=Logstash
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
ExecStart=/a01/apps/logstash/bin/logstash -f /etc/logstash/logstash.conf

[Install]
WantedBy=multi-user.target

chmod 754 /etc/systemd/system/logstash.service
systemctl enable logstash
systemctl start logstash
systemctl status logstash

配置nginx日志格式为json(web服务器上)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
vim /etc/nginx/nginx.conf

log_format json '{"@timestamp":"$time_iso8601",'
'"@version":"1",'
'"client":"$remote_addr",'
'"url":"$uri",'
'"status":"$status",'
'"domain":"$host",'
'"host":"$server_addr",'
'"size":$body_bytes_sent,'
'"responsetime":$request_time,'
'"referer": "$http_referer",'
'"ua": "$http_user_agent"'
'}';

access_log /a01/apps/nginx/log/access_nginx_json.log json;

然后重启nignx,动态查看一下nginx的json日志文件,我们看看格式是怎样的:

1
tail -f /a01/apps/nginx/log/access_nginx_json.log

PRoXFA.png

在web服务器上配置Logstash客户端

1
2
3
4
cd /a01/apps/apps_src/
mkdir -p /usr/local/java
tar -zxvf jdk-8u151-linux-x64.tar.gz -C /usr/local/java/
vim /etc/profile
1
2
3
4
JAVA_HOME=/usr/local/java/jdk1.8.0_151
PATH=$JAVA_HOME/bin:$PATH
CLASSPATH=$JAVA_HOME/jre/lib/ext:$JAVA_HOME/lib/tools.jar
export PATH JAVA_HOME CLASSPATH
1
2
3
4
5
source /etc/profile
tar -zxvf logstash-6.1.1.tar.gz -C /a01/apps/
mv /a01/apps/logstash-6.1.1/ /a01/apps/logstash
mkdir -p /etc/logstash/
vim /etc/logstash/logstash.conf
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
input {
file {
path => "/a01/apps/nginx/log/access_nginx_json.log"
type => "nginx_access"
codec => "json"
}

}


output {
if [type] == "nginx_access" {
rabbitmq {
host => "192.168.10.21"
port => 5672
key => "nginx_access"
user => "nong"
password => "nong"
exchange => "amq.direct"
exchange_type => "direct"
}
}
}

检测配置文件是否正确

1
/a01/apps/logstash/bin/logstash -t -f /etc/logstash/logstash.conf

编写logstash加入系统服务

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
vim /etc/systemd/system/logstash.service

[Unit]
Description=Logstash
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
ExecStart=/a01/apps/logstash/bin/logstash -f /etc/logstash/logstash.conf

[Install]
WantedBy=multi-user.target

chmod 754 /etc/systemd/system/logstash.service
systemctl enable logstash
systemctl start logstash
systemctl status logstash

测试ELK+RabbitMQ

好了,到现在为止,所有的服务已经安装完成,我们来访问一下web服务端的nginx产生访问日志,然后去Kibana创建index

PRTiwQ.png

PRTAFs.png

PRTEYn.png

PRTVWq.png

PRTeS0.png

查看rabbitmq状态

PRTuOU.png

PRTlTJ.png

PRTGf1.png

博主QQ:1012405802
技术交流QQ群:830339411
版权声明:网站内容有原创和转载,如有侵权,请联系删除,谢谢!!
感谢打赏,93bok因你们而精彩!!(支付宝支持花呗)
0%