前言
流程,nignx格式化日志成json,通过logstash直接采集到elasticsearch,然后通过kibana gui界面展示分析
要点nignx日志成json格式,避免nignx默认日志是空格,需要正则匹配,导致logstash占过多cpu
elasticsearch机配置防火墙,只让指定的logstash机访问
kibana只监听本地127.0.0.1使用nignx方向代理,nginx中配置Http Basic Auth账号密码登陆
比较粗略的笔记,备忘
安装java
um install java-1.8.0-openjdk*
nginx配置
为了让nignx机跑logstash采集日志负载最低,建议直接生成json的方式,直接就可以用logstash读取写入到Elasticsearch
http{} 中定义 格式化日志成json log_format logstash_json '{"@timestamp":"$time_iso8601",' '"host":"$server_addr",' '"clientip":"$remote_addr",' '"http_x_forwarded_for":"$http_x_forwarded_for",' '"size":$body_bytes_sent,' '"responsetime":$request_time,' '"upstreamtime":"$upstream_response_time",' '"upstreamhost":"$upstream_addr",' '"http_host":"$host",' '"request":"$request",' '"url":"$uri",' '"xff":"$http_x_forwarded_for",' '"referer":"$http_referer",' '"agent":"$http_user_agent",' '"status":"$status"}'; server内输出日志 access_log可以配置多个同时输出,可以保留你以前的 access_log /data/wwwlogs/www.iamle.log iamle.com; access_log /data/wwwlogs/www.iamle.com.logstash_json.log logstash_json; nginx机安装Logstash1.5.x rpm --import http://packages.elasticsearch.org/GPG-KEY-elasticsearch cat > /etc/yum.repos.d/logstash.repo <<EOF [logstash-1.5] name=logstash repository for 1.5.x packages baseurl=http://packages.elasticsearch.org/logstash/1.5/centos gpgcheck=1 gpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch enabled=1 EOF yum clean all yum install logstash
在目录 /etc/logstash/conf.d/
建立配置文件 nginx_json.conf
input { file { path => "/data/wwwlogs/www.iamle.com.logstash_json.log" codec => json } } filter { mutate { split => [ "upstreamtime", "," ] } mutate { convert => [ "upstreamtime", "float" ] } } output { elasticsearch { host => "elk.server.iamle.com" protocol => "http" index => "logstash-%{type}-%{+YYYY.MM.dd}" index_type => "%{type}" workers => 5 template_overwrite => true } } service logstash start
日志存储机安装Elasticsearch1.7.x提供数据底层支持
rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch cat > /etc/yum.repos.d/elasticsearch.repo <<EOF [elasticsearch-1.7] name=Elasticsearch repository for 1.7.x packages baseurl=http://packages.elastic.co/elasticsearch/1.7/centos gpgcheck=1 gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch enabled=1 EOF yum clean all yum install elasticsearch
配置文件
配置数据保存位置
vim /etc/elasticsearch/elasticsearch.yml # Can optionally include more than one location, causing data to be striped across # the locations (a la RAID 0) on a file level, favouring locations with most free # space on creation. For example: # path.data: /data
目录会自动生成,只需要指定一个空目录就可以了
service elasticsearch start centos7 systemctl start elasticsearch systemctl status elasticsearch elasticsearch.service - Elasticsearch Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; disabled) Active: active (running) since Fri 2015-09-04 15:37:08 CST; 1s ago Docs: http://www.elastic.co Main PID: 19376 (java) CGroup: /system.slice/elasticsearch.service └─19376 /bin/java -Xms256m -Xmx1g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -X... Sep 04 15:37:08 elk systemd[1]: Starting Elasticsearch... Sep 04 15:37:08 elk systemd[1]: Started Elasticsearch. 检查是否已经成功开启 ss -ltnp |grep 9200 centos7配置firewalld固定ip可访问elasticsearch systemctl start firewalld.service systemctl status firewalld.service 只允许nignx机访问elasticsearch机9200 9300端口 firewall-cmd --permanent --zone=public --add-rich-rule="rule family="ipv4" \ source address="10.8.8.2" \ port protocol="tcp" port="9200" accept" firewall-cmd --permanent --zone=public --add-rich-rule="rule family="ipv4" \ source address="10.8.8.2" \ port protocol="tcp" port="9300" accept" firewall-cmd --reload iptables -L -n |grep 9200 ACCEPT tcp -- 10.8.8.2 0.0.0.0/0 tcp dpt:9200 ctstate NEW 安装Kibana4展示Elasticsearch中的数据 wget https://download.elastic.co/kibana/kibana/kibana-4.1.1-linux-x64.tar.gz tar zxvf kibana-4.1.1-linux-x64.tar.gz cd kibana-4.1.1-linux-x64 修改配置文件 vim /usr/local/kibana-4.1.1-linux-x64/config/kibana.yml # Kibana is served by a back end server. This controls which port to use. port: 5601 # The host to bind the server to. #监听本地地址 用nignx反向代理 host: "127.0.0.1" nohup ./bin/kibana & 检查是否已经成功开启 ss -ltnp |grep 5601 使用nignx反向代理kibana nginx配置Http Basic Auth账号密码登陆 http://trac.edgewall.org/export/10770/trunk/contrib/htpasswd.py (nginx wiki里推荐的) 运行示例 chmod 777 htpasswd.py ./htpasswd.py -c -b htpasswd username password #-c为生成文件 htpasswd为文件名 server { listen 80; #listen [::]:80; server_name elk.server.iamle.com; location / { auth_basic "Password please"; auth_basic_user_file /usr/local/nginx/conf/htpasswd; proxy_pass http://127.0.0.1:5601/; proxy_redirect off; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; } }
扩展centos7 firewall的使用
检查防火墙状态 firewall-cmd --stat 临时开放ftp服务 firewall-cmd --add-service=ftp 永久开放ftp服务 firewall-cmd --add-service=ftp --permanent 关闭ftp服务 firewall-cmd --remove-service=ftp --permanent 配置防火墙在public区域永久开放http服务 firewall-cmd --permanent --zone=public --add-service=http 加入指定开放端口 firewall-cmd --add-port=1324/tcp 为了让之前的设定生效当然要重启服务咯 systemctl restart firewalld 或者使用下面的命令免去重启服务(防火墙策略配置后重新载入) firewall-cmd --complete-reload firewall-cmd --reload (这两句功能相同) 检查ftp服务的21端口是否开放 iptables -L -n | grep 21 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:21 ctstate NEW 查询ftp服务启用状态 firewall-cmd --query-service ftp 查看当前规则 firewall-cmd --list-all 仅允许部分IP访问本机服务配置 firewall-cmd --permanent --zone=public --add-rich-rule="rule family="ipv4" \ source address="192.168.0.4/24" service name="http" accept" 仅允许部分IP访问本机端口配置 firewall-cmd --permanent --zone=public --add-rich-rule="rule family="ipv4" \ source address="192.168.0.4/24" \ port protocol="tcp" port="8080" accept"
以上是小编为您精心准备的的内容,在的博客、问答、公众号、人物、课程等栏目也有的相关内容,欢迎继续使用右上角搜索按钮进行搜索日志
配置
elasticsearch kibana、nginx elasticsearch、logstash kibana、es logstash kibana、kibana和logstash,以便于您获取更多的相关知识。
时间: 2024-12-24 08:49:55