购买阿里云ecs后所做项目五:nginx 日志分割,格式化json日志,日志分析

一、日志分割

在系统中有每天,每小时,每月,每周的定时任务文件夹,默认logrotate 是在每天执行一次,它会执行/etc/logrotate.d/下的日志分割的配置,我们只需要创建一个nginx的配置在这里吗就行了

[root@iZ2ze1o ~]#ll /etc/cron
cron.d/       cron.daily/   cron.deny     cron.hourly/  cron.monthly/ crontab       cron.weekly/  
[root@iZ2ze1o ~]#cat /etc/cron.daily/logrotate 
# 以上是解释,不需要操作,只需创建下面的文件即可
[root@iZ2ze1o ~]#vi /etc/logrotate.d/nginx 
/apps/nginx/logs/*.log {
        daily                                                                                                                
        rotate 6
        missingok
        notifempty
        create 644 nginx nginx
        postrotate
          if [ -f /apps/nginx/logs/nginx.pid ]; then
                kill -USR1 `cat /apps/nginx/logs/nginx.pid`
          fi
        endscript
}
# daily 每天
# rotate 6 保存6个文件
# missingok 忽略错误
# notifempty忽略空文件
# create 644 nginx nginx 创建的权限和用户
# postrotate 分割后执行的shell
# endscript 结束shell

效果:
[root@iZ2ze1o ~]#ll /apps/nginx/logs/
total 3864
-rw-r--r-- 1 nginx nginx   12263 Jun  9 09:16 access.log
-rw-r--r-- 1 nginx nginx 2827263 Jun  8 16:07 access.log.1
-rw-r--r-- 1 root  root    13189 Jun  9 02:57 access.log-20200609
-rw-r--r-- 1 nginx nginx    6182 Jun  9 08:36 error.log
-rw-r--r-- 1 nginx nginx 1067074 Jun  8 17:25 error.log.1
-rw-r--r-- 1 nginx nginx    6738 Jun  9 02:52 error.log-20200609
-rw-r--r-- 1 root  root        4 Jun  8 14:11 nginx.pid
[root@iZ2ze1o ~]#

二、nginx 格式化json 日志

[root@iZ2ze1o ~]#vi /apps/nginx/conf/nginx.conf
# 在http 定义日志格式,在server 引用日志格式
http {
    log_format access_json '{"@timestamp":"$time_iso8601",'
                           '"host":"$server_addr",'
                           '"clientip":"$remote_addr",'
                           '"size":$body_bytes_sent,'
                           '"responsetime":$request_time,'
                           '"upstreamtime":"$upstream_response_time",'
                           '"upstreamhost":"$upstream_addr",'
                           '"http_host":"$host",'
                           '"url":"$uri",'
                           '"domain":"$host",'
                           '"xff":"$http_x_forwarded_for",'
                           '"referer":"$http_referer",'
                           '"status":"$status"}';
    server {
        access_log  logs/access.log  access_json;
        }
 }
[root@iZ2ze1o ~]#nginx -s reload
[root@iZ2ze1o ~]#tail -1 /apps/nginx/logs/access.log
{"@timestamp":"2020-06-09T09:16:11+08:00","host":"xxxxx","clientip":"216.53.254.149","size":157,"responsetime":0.310,"upstreamtime":"-","upstreamhost":"-","http_host":"localhost","url":"/","domain":"localhost","xff":"-","referer":"-","status":"400"}
[root@iZ2ze1o ~]#

校验格式是否正确
购买阿里云ecs后所做项目五:nginx 日志分割,格式化json日志,日志分析插图

三、日志分析

使用脚本,分析ip访问次数,查看是否遭受某个ip 的攻击

[root@iZ2ze1o scripts]# cat /apps/nginx/logs/access.log | awk -F'"' '{count[$12]+=1;}END{for (i in count){print count[i]" "i;}}' | sort -t" " -k 1 -nr | head 
11 61.160.236.22
4 190.186.205.194
4 176.53.69.2
2 87.8.109.199
2 200.76.4.130
1 95.213.177.126
1 95.156.203.233
1 89.239.168.227
1 80.82.70.187
1 62.210.111.97
[root@iZ2ze1o scripts]#

因为日志已经分割了,下面是我以前的日志

  1. 问题一,某个ip 一直在想尝试登陆我的WordPress

    [root@iZ2ze1o logs]#grep "//wp-login.php" access.log.1 | head
    52.187.188.65 - - [06/Jun/2020:17:15:24 +0800] "GET /wordpress//wp-login.php HTTP/1.1" 200 5251 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0 Safari/537.36"
    52.187.188.65 - - [06/Jun/2020:17:15:24 +0800] "GET /wordpress//wp-login.php HTTP/1.1" 200 5251 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0 Safari/537.36"
    52.187.188.65 - - [06/Jun/2020:17:21:21 +0800] "POST /wordpress//wp-login.php HTTP/1.1" 200 5533 "http://jiansesheying.com/wordpress//wp-login.php" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0 Safari/537.36"
    52.187.188.65 - - [06/Jun/2020:17:21:22 +0800] "POST /wordpress//wp-login.php HTTP/1.1" 200 5533 "http://jianseyingshi.com/wordpress//wp-login.php" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0 Safari/537.36"
    52.187.188.65 - - [06/Jun/2020:17:21:22 +0800] "POST /wordpress//wp-login.php HTTP/1.1" 200 5533 "http://jiansesheying.com/wordpress//wp-login.php" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0 Safari/537.36"
    52.187.188.65 - - [06/Jun/2020:17:21:22 +0800] "POST /wordpress//wp-login.php HTTP/1.1" 200 5533 "http://jianseyingshi.com/wordpress//wp-login.php" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0 Safari/537.36"
    52.187.188.65 - - [06/Jun/2020:17:21:22 +0800] "POST /wordpress//wp-login.php HTTP/1.1" 200 5533 "http://jiansesheying.com/wordpress//wp-login.php" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0 Safari/537.36"
    52.187.188.65 - - [06/Jun/2020:17:21:22 +0800] "POST /wordpress//wp-login.php HTTP/1.1" 200 5533 "http://jianseyingshi.com/wordpress//wp-login.php" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0 Safari/537.36"
    52.187.188.65 - - [06/Jun/2020:17:21:22 +0800] "POST /wordpress//wp-login.php HTTP/1.1" 200 5533 "http://jiansesheying.com/wordpress//wp-login.php" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0 Safari/537.36"
    52.187.188.65 - - [06/Jun/2020:17:21:23 +0800] "POST /wordpress//wp-login.php HTTP/1.1" 200 5533 "http://jianseyingshi.com/wordpress//wp-login.php" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0 Safari/537.36"
    [root@iZ2ze1o logs]#
    
  2. 问题二:
    一些ip 使用xmlrpc.php 来查看暴力破解我的WordPress密码

    [root@iZ2ze1o logs]#grep xmlrpc.php access.log.1 | head
    5.135.165.15 - - [23/May/2020:15:28:14 +0800] "POST /xmlrpc.php HTTP/1.1" 404 27 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36"
    198.71.239.44 - - [23/May/2020:15:39:31 +0800] "POST /xmlrpc.php HTTP/1.1" 404 27 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36"
    198.71.239.44 - - [23/May/2020:15:39:31 +0800] "POST /xmlrpc.php HTTP/1.1" 404 27 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36"
    182.50.132.95 - - [23/May/2020:15:39:31 +0800] "POST /xmlrpc.php HTTP/1.1" 404 27 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36"
    182.50.132.95 - - [23/May/2020:15:39:31 +0800] "POST /xmlrpc.php HTTP/1.1" 404 27 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36"
    198.71.239.7 - - [23/May/2020:15:40:00 +0800] "POST /xmlrpc.php HTTP/1.1" 404 27 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36"
    198.71.239.7 - - [23/May/2020:15:40:04 +0800] "POST /xmlrpc.php HTTP/1.1" 404 27 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36"
    210.209.123.158 - - [23/May/2020:15:42:59 +0800] "POST /xmlrpc.php HTTP/1.1" 404 27 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36"
    50.63.197.102 - - [23/May/2020:15:43:00 +0800] "POST /xmlrpc.php HTTP/1.1" 404 27 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36"
    3.133.43.109 - - [23/May/2020:15:43:00 +0800] "POST /xmlrpc.php HTTP/1.1" 404 27 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36"
    [root@iZ2ze1o logs]#
    
  3. 解决办法
    解决一:使用awk 配合iptables 来拒绝ip ,下面只是样例,自己按需更改

    # 查看前十名ip
    cat /apps/nginx/logs/access.log  | awk -F'"' '{count[$12]+=1;}END{for (i in count){print count[i]" "i;}}' | sort -t" " -k 1 -nr | head
    
    # 查看访问数大于100的ip
    cat /apps/nginx/logs/access.log  | awk -F'"' '{count[$12]+=1;}END{for (i in count){ if(count[i]>100){print i}}}' | sort -t" " -k 1 -nr | head
    
    #把访问数大于100的拒绝掉
    cat /apps/nginx/logs/access.log  | awk -F'"' '{count[$12]+=1;}END{for (i in count){ if(count[i]>100){system("iptables -I INPUT -s " i " -j DROP")}}}' 
    
    # 查看上一分钟的访问次数
    cat /apps/nginx/logs/access.log  | awk -F'"' 'BEGIN{ now=strftime("%H:%M",systime()-60); print now;}$4 ~ now {count[$12]+=1;}END{for (i in count){print count[i]" "i;}}'
    
    # 查看上一个小时的访问次数
    cat /apps/nginx/logs/access.log  | awk -F'"' 'BEGIN{ beg=strftime("%Y-%m-%dT%H:%M",systime()-3660); end=strftime("%Y-%m-%dT%H:%M",systime()-60); print beg; print end;}$4 > beg,$4 < end{count[$12]+=1; print $4}END{for (i in count){print count[i]" "i ;}}'   
    

    解决二:使用nginx 拒绝 xmlrpc.php的访问

     location ~ xmlrpc.php$ {                                                                                             
                deny all;
            }
    

    购买阿里云ecs后所做项目五:nginx 日志分割,格式化json日志,日志分析插图(1)

没有账号? 忘记密码?

社交账号快速登录