Awstats动态分析结果for nginx

前言

Awstats是一个免费非常简洁而且强大有个性的网站日志分析工具,是一款基于 Perl 的 WEB 日志分析工具,AWStats 可以显示你所有的网页/邮件/ FTP统计包括访问,访问者,页面,点击,高峰时间,操作系统,浏览器,搜索引擎,关键字,机器人访问,断开的链接和更多的阻力截图排序

Awstats功能

Awstats 是在 SourceForge 上发展很快的一个基于 Perl 的 WEB 日志分析工具,一个充分的日志分析让 Awstats 显示您下列资料:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
1. 访问次数、独特访客人数,
2. 访问时间和上次访问,
3. 使用者认证、最近认证的访问,
4. 每周的高峰时间(页数,点击率,每小时和一周的千字节),
5. 域名/国家的主机访客(页数,点击率,字节,269域名/国家检测, geoip 检测),
6. 主机名单,最近访问和未解析的 IP 地址名单
7. 大多数看过的进出页面,
8. 档案类型,
9. 网站压缩统计表(mod_gzip 或者 mod_deflate),
10. 使用的操作系统 (每个操作系统的页数,点击率 ,字节, 35 OS detected),
11. 使用的浏览器,
12. 机器人访问(检测 319 个机器人),
13. 蠕虫攻击 (5 个蠕虫家族),
14. 搜索引擎,利用关键词检索找到你的地址,
15. HTTP 协议错误(最近查阅没有找到的页面),
16. 其他基于 URL 的个性报导,链接参数, 涉及综合行销领域目的.
17. 贵网站被加入"最喜爱的书签".次数.
18. 屏幕大小(需要在索引页补充一些 HTML 标签).
19. 浏览器的支持比例: Java, Flash, RealG2 reader, Quicktime reader, WMA reader, PDF reader.
20. 负载平衡服务器比率集群报告.

Awstats官网

注意问题

  • Awstats 的运行是需要 PERL 环境的支持,从 awstats 的文档来看,它对 Apache HTTP Server 的支持是非常完美的,而当我们把 Web 服务器换成 Nginx 后,要运行 awstats 变得很麻烦。首先 Nginx 本身对 Perl 的支持是比较弱的,甚至官方也不建议使用。

Awstats运行原理

  • AWStats提供一系列的perl脚本实现:服务配置,日志读取,报表生成等功能。而功能实现的具体执行过程是:首先,当然是apache将访问情况记录到日志中,AWStats每次执行更新时读取这些日志,分析日志数据,将结果存储到数据库中,(这个数据库是AWStats自带的(就是一文本文件),并不需要第三方软件支持。),最后AWStats提供一个cgi程序通过web页面来显示数据库中所统计的数据。

一、安装CPAN

1
2
3
4
cd /a01/apps/apps_src/
tar -zxvf CPAN-2.16.tar.gz
cd CPAN-2.16
perl Makefile.PL

1
2
3
报错:
Can't locate ExtUtils/MakeMaker.pm in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at Makefile.PL line 6.
BEGIN failed--compilation aborted at Makefile.PL line 6.
1
2
3
4
解决:
yum -y install perl-devel
perl Makefile.PL
make && make install

二、安装FCGI

下载地址:http://search.cpan.org/~ether/FCGI-0.78/FCGI.pm

1
2
3
4
5
cd /a01/apps/apps_src/
tar -zxvf FCGI-0.78.tar.gz
cd FCGI-0.78
perl Makefile.PL
make && make install

三、安装FCGI::ProcManage

1
2
3
4
5
cd /a01/apps/apps_src/
tar -zxvf FCGI-ProcManager-0.28.tar.gz
cd FCGI-ProcManager-0.28
perl Makefile.PL
make && make install

四、创建FCGI启动文件

1
vim /a01/apps/nginx/sbin/fcgi

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
#!/usr/bin/perl
use FCGI;
#perl -MCPAN -e 'install FCGI'
use Socket;
use POSIX qw(setsid);
#use Fcntl;
require 'syscall.ph';
&daemonize;
#this keeps the program alive or something after exec'ing perl scripts
END() { } BEGIN() { }
*CORE::GLOBAL::exit = sub { die "fakeexit\nrc=".shift()."\n"; };
eval q{exit};
if ($@) {
exit unless $@ =~ /^fakeexit/;
};
&main;
sub daemonize() {
chdir '/' or die "Can't chdir to /: $!";
defined(my $pid = fork) or die "Can't fork: $!";
exit if $pid;
setsid or die "Can't start a new session: $!";
umask 0;
}
sub main {
#$socket = FCGI::OpenSocket( "127.0.0.1:8999", 10 );
$socket = FCGI::OpenSocket( "/a01/apps/nginx/fastcgi_temp/perl_cgi-dispatch.sock", 10 );
#use UNIX sockets - user running this script must have w access to the 'nginx' folder!!
$request = FCGI::Request( \*STDIN, \*STDOUT, \*STDERR, \%req_params, $socket );
if ($request) { request_loop()};
FCGI::CloseSocket( $socket );
}
sub request_loop {
while( $request->Accept() >= 0 ) {
#processing any STDIN input from WebServer (for CGI-POST actions)
$stdin_passthrough ='';
$req_len = 0 + $req_params{'CONTENT_LENGTH'};
if (($req_params{'REQUEST_METHOD'} eq 'POST') && ($req_len != 0) ){
my $bytes_read = 0;
while ($bytes_read < $req_len) {
my $data = '';
my $bytes = read(STDIN, $data, ($req_len - $bytes_read));
last if ($bytes == 0 || !defined($bytes));
$stdin_passthrough .= $data;
$bytes_read += $bytes;
}
}
#running the cgi app
if ( (-x $req_params{SCRIPT_FILENAME}) && #can I execute this?
(-s $req_params{SCRIPT_FILENAME}) && #Is this file empty?
(-r $req_params{SCRIPT_FILENAME}) #can I read this file?
){
pipe(CHILD_RD, PARENT_WR);
my $pid = open(KID_TO_READ, "-|");
unless(defined($pid)) {
print("Content-type: text/plain\r\n\r\n");
print "Error: CGI app returned no output - Executing $req_params
{SCRIPT_FILENAME} failed !\n";
next;
}
if ($pid > 0) {
close(CHILD_RD);
print PARENT_WR $stdin_passthrough;
close(PARENT_WR);
while(my $s = <KID_TO_READ>) { print $s; }
close KID_TO_READ;
waitpid($pid, 0);
} else {
foreach $key ( keys %req_params){
$ENV{$key} = $req_params{$key};
}
# cd to the script's local directory
if ($req_params{SCRIPT_FILENAME} =~ /^(.*)\/[^\/]+$/) {
chdir $1;
}
close(PARENT_WR);
close(STDIN);
#fcntl(CHILD_RD, F_DUPFD, 0);
syscall(&SYS_dup2, fileno(CHILD_RD), 0);
#open(STDIN, "<&CHILD_RD");
exec($req_params{SCRIPT_FILENAME});
die("exec failed");
}
}
else {
print("Content-type: text/plain\r\n\r\n");
print "Error: No such CGI app - $req_params{SCRIPT_FILENAME} may not exist or is
not executable by this process.\n";
}
}
}

  • 授权:
    1
    2
    chmod 755 /a01/apps/nginx/sbin/fcgi
    chown -R www:www /a01/apps/nginx/
  • 启动FPM(FastCGI 进程管理器)
    1
    perl /a01/apps/nginx/sbin/fcgi
  • 查看
    1
    ps aux | grep fcgi

Awstats安装配置

  • (nginx安装过程省略,这里直接使用之前安装好的LNMP环境)
1. 解压Awstats
  • 首先要下载源码包上传到服务器,解压
    1
    2
    3
    cd /a01/apps/apps_src/
    tar -zxvf awstats-7.7.tar.gz
    mv awstats-7.7 /a01/apps/awstats
2. 给root权限,否则后边运行.pl报错
1
2
3
chown -R root:root /a01/apps/awstats/
chmod +x /a01/apps/awstats/tools/*.pl
chmod +x /a01/apps/awstats/wwwroot/cgi-bin/*.pl
3. 执行awstats_configure.pl配置向导生成awstats的配置文件
1
2
cd /a01/apps/awstats/tools/
./awstats_configure.pl

  • 此时会出现以下提示
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77

----- AWStats awstats_configure 1.0 (build 20140126) (c) Laurent Destailleur -----
This tool will help you to configure AWStats to analyze statistics for
one web server. You can try to use it to let it do all that is possible
in AWStats setup, however following the step by step manual setup
documentation (docs/index.html) is often a better idea. Above all if:
- You are not an administrator user,
- You want to analyze downloaded log files without web server,
- You want to analyze mail or ftp log files instead of web log files,
- You need to analyze load balanced servers log files,
- You want to 'understand' all possible ways to use AWStats...
Read the AWStats documentation (docs/index.html).

-----> Running OS detected: Linux, BSD or Unix
Warning: AWStats standard directory on Linux OS is '/usr/local/awstats'.
If you want to use standard directory, you should first move all content
of AWStats distribution from current directory:
/a01/apps/awstats
to standard directory:
/usr/local/awstats
And then, run configure.pl from this location.
Do you want to continue setup from this NON standard directory [yN] ? y

-----> Check for web server install

Enter full config file path of your Web server.
Example: /etc/httpd/httpd.conf
Example: /usr/local/apache2/conf/httpd.conf
Example: c:\Program files\apache group\apache\conf\httpd.conf
Config file path ('none' to skip web server setup):
> none #这里让填写网页服务器的配置文件路径,因为我们不是apache,所以写none,如果是apache就写apache的配置文件路径即可

Your web server config file(s) could not be found.
You will need to setup your web server manually to declare AWStats
script as a CGI, if you want to build reports dynamically.
See AWStats setup documentation (file docs/index.html)

-----> Update model config file '/a01/apps/awstats/wwwroot/cgi-bin/awstats.model.conf'
File awstats.model.conf updated.

-----> Need to create a new config file ?
Do you want me to build a new AWStats config/profile
file (required if first install) [y/N] ? y #询问是否创建awstats新的配置文件

-----> Define config file name to create
What is the name of your web site or profile analysis ?
Example: www.mysite.com
Example: demo
Your web site, virtual server or profile name:
> www.hehe.com #填写网站域名,虚拟主机名或者随便配置一个名字

-----> Define config file path
In which directory do you plan to store your config file(s) ?
Default: /etc/awstats
Directory path to store config file(s) (Enter for default):
> #这里填写awstats的配置文件存放路径,我们使用它默认的/etc/awstats就行了,直接回车

-----> Create config file '/etc/awstats/awstats.www.hehe.com.conf'
Config file /etc/awstats/awstats.www.hehe.com.conf created.

-----> Add update process inside a scheduler
Sorry, configure.pl does not support automatic add to cron yet.
You can do it manually by adding the following command to your cron:
/a01/apps/awstats/wwwroot/cgi-bin/awstats.pl -update -config=www.hehe.com
Or if you have several config files and prefer having only one command:
/a01/apps/awstats/tools/awstats_updateall.pl now
Press ENTER to continue... #提示不能自动加入crontab计划任务,稍后需要自己添加,回车继续


A SIMPLE config file has been created: /etc/awstats/awstats.www.hehe.com.conf
You should have a look inside to check and change manually main parameters.
You can then manually update your statistics for 'www.hehe.com' with command:
> perl awstats.pl -update -config=www.hehe.com
You can also build static report pages for 'www.hehe.com' with command:
> perl awstats.pl -output=pagetype -config=www.hehe.com

Press ENTER to finish... #提示配置文件创建完成和如何更新配置及建立静态报告页,回车结束

4. 修改awstats配置文件
1
vim /etc/awstats/awstats.www.hehe.com.conf

1
2
LogFile="/a01/apps/nginx/log/access.log"			#需要分析的nginx日志文件
DirData="/a01/apps/awstats/data" #awstats的数据库(纯文本)

5. 创建awstats数据库路径
1
mkdir -p /a01/apps/awstats/data
6. 开始执行日志分析
  • 现在我们需要用awstats来生成对日志的统计分析信息出来了。由于我们用的是支持perl的FCGI动态化访问页面,所以此处我们只需要直接更新数据库即可。FCGI程序会自动将数据库以动态页面的形式展现出来,无须再手动生成静态页面了。
1
/a01/apps/awstats/wwwroot/cgi-bin/awstats.pl -update -config=www.hehe.com

1
2
3
4
5
6
7
8
9
10
11
12
13
Create/Update database for config "/etc/awstats/awstats.www.hehe.com.conf" by AWStats version 7.7 (build 20180105)
From data in log file "/a01/apps/nginx/log/access.log"...
Phase 1 : First bypass old records, searching new record...
Searching new records from beginning of log file...
Phase 2 : Now process new records (Flush history on disk after 20000 hosts)...
Jumped lines in file: 0
Parsed lines in file: 3
Found 3 dropped records,
Found 0 comments,
Found 0 blank records,
Found 0 corrupted records,
Found 0 old records,
Found 0 new qualified records.

7. 配置nginx配置文件
1
vim /etc/nginx/nginx.conf

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
    server {
listen 80;
server_name www.hehe.com;

location / {
root /a01/apps/awstats/wwwroot/;
index index.php index.html index.htm;
}

location ~* ^/cgi-bin/.*\.pl$ {
root /a01/apps/awstats/wwwroot;
fastcgi_pass unix:/a01/apps/nginx/fastcgi_temp/perl_cgi-dispatch.sock;
fastcgi_index index.pl;
include fastcgi_params;
charset gb2312;
auth_basic "Restricted";
auth_basic_user_file /a01/apps/nginx/htpasswd.pass;
}
location ~ ^/icon/ {
root /a01/apps/awstats/wwwroot/;
index index.html;
access_log off;
error_log off;
charset gb2312;
}
……
……

  • 为了保持nginx.conf主配置文件更加整洁干净,所以我们将fastcgi_param的一系列参数添加到/etc/nginx/fastcgi_params文件的最顶部,然后在nginx.conf里面调用这个文件即可
1
vim /etc/nginx/fastcgi_params

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
fastcgi_param QUERY_STRING $query_string;
fastcgi_param REQUEST_METHOD $request_method;
fastcgi_param CONTENT_TYPE $content_type;
fastcgi_param CONTENT_LENGTH $content_length;
fastcgi_param GATEWAY_INTERFACE CGI/1.1;
fastcgi_param SERVER_SOFTWARE nginx;
fastcgi_param SCRIPT_NAME $fastcgi_script_name;
fastcgi_param REQUEST_URI $request_uri;
fastcgi_param DOCUMENT_URI $document_uri;
fastcgi_param DOCUMENT_ROOT $document_root;
fastcgi_param SERVER_PROTOCOL $server_protocol;
fastcgi_param REMOTE_ADDR $remote_addr;
fastcgi_param REMOTE_PORT $remote_port;
fastcgi_param SERVER_ADDR $server_addr;
fastcgi_param SERVER_PORT $server_port;
fastcgi_param SERVER_NAME $server_name;
fastcgi_read_timeout 60;

8. 针对上面的加密,由于nginx没有好的加密认证工具,需要借助apache的htpasswd来实现加密认证功能
1
2
yum -y install httpd-tools
htpasswd -bc /a01/apps/nginx/htpasswd.pass nongkaige 123456
9. 修改客户机的hosts文件

PrQB90.png

10. 访问http://www.hehe.com/cgi-bin/awstats.pl?config=www.hehe.com,输入账号密码即可

PrQrcT.png

PrQcB4.png

PrQgHJ.png

11. 配置日志分析页面的来访IP的地址位置显示
  • 把软件包上传到服务器,解压到/a01/apps/awstats/wwwroot/cgi-bin/plugins/中
1
2
3
4
5
6
cd /a01/apps/apps_src/
mv Awstats之qqhostinfo插件.zip /a01/apps/awstats/wwwroot/cgi-bin/plugins/
cd /a01/apps/awstats/wwwroot/cgi-bin/plugins/
unzip Awstats之qqhostinfo插件.zip
cd Awstats之qqhostinfo插件/qqhostinfo/
mv * /a01/apps/awstats/wwwroot/cgi-bin/plugins/

1
vim /etc/awstats/awstats.www.hehe.com.conf

1
#LoadPlugin="hostinfo"改为LoadPlugin="qqhostinfo"

PrQWNR.png

  • 这样就可以了,在后面的日志分析中,我们就可以清楚的看到来访IP的地理位置信息了。
12. 配置awstats自动运行
  • 先写个生成awstats统计信息的脚本
1
vim awstats_up.sh

1
2
#!/bin/sh
/a01/apps/awstats/wwwroot/cgi-bin/awstats.pl -update -config=www.hehe.com >/dev/null 2>&1

1
chmod +x /root/awstats_up.sh
  • 为了让整个日志的统计过程可以实现自动化,将awstats_up.sh脚本加入crontab定时任务中去,每分钟执行一次,当然这个统计的时间间隔可自行修改
1
crontab -e

1
*/1 * * * * /bin/sh /root/awstats_up.sh
博主QQ:1012405802
技术交流QQ群:830339411
版权声明:网站内容有原创和转载,如有侵权,请联系删除,谢谢!!
感谢打赏,93bok因你们而精彩!!(支付宝支持花呗)
0%