40.2. Squid - Internet Object Cache (WWW proxy cache)

如果apache 安装了gzip,deflate需要开启cache_vary

cache_vary on

40.2.1. 源码安装

wget http://www.squid-cache.org/Versions/v2/2.6/squid-2.6.STABLE13.tar.gz
./configure --prefix=/usr/local/squid-2.6
make all
make install

mkdir -p /usr/local/squid-2.6/var/cache
chown nobody.nobody -R /usr/local/squid-2.6/var/
ln -s /usr/local/squid-2.6 /usr/local/squid
cd /usr/local/squid

./squid -NCd1

40.2.2. debian/ubuntu 安装

$ sudo apt-get install squid 

$ sudo apt-get install squid3
$ sudo apt-get install squidclient

40.2.3. 配置

查看当前配置参数

当你打开squid.conf文件时，你会头大，因为文件太长了，并且已经启用了部分参数。你可以使用下面命令查看那些参数被开启。

$ grep '^[a-z]' squid.conf

下面是安装squid3后的默认开启选项

$ grep '^[a-z]' squid.conf
acl manager proto cache_object
acl localhost src 127.0.0.1/32
acl to_localhost dst 127.0.0.0/8
acl SSL_ports port 443
acl Safe_ports port 80          # http
acl Safe_ports port 21          # ftp
acl Safe_ports port 443         # https
acl Safe_ports port 70          # gopher
acl Safe_ports port 210         # wais
acl Safe_ports port 1025-65535  # unregistered ports
acl Safe_ports port 280         # http-mgmt
acl Safe_ports port 488         # gss-http
acl Safe_ports port 591         # filemaker
acl Safe_ports port 777         # multiling http
acl CONNECT method CONNECT
http_access allow manager localhost
http_access deny manager
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localhost
http_access deny all
icp_access deny all
htcp_access deny all
http_port 3128
hierarchy_stoplist cgi-bin ?
access_log /var/log/squid3/access.log squid
refresh_pattern ^ftp:           1440    20%     10080
refresh_pattern ^gopher:        1440    0%      1440
refresh_pattern (cgi-bin|\?)    0       0%      0
refresh_pattern .               0       20%     4320
icp_port 3130
coredump_dir /var/spool/squid3

修改squid.conf之前请做好备份。

netkiller@Linux-server:/etc/squid$ sudo cp squid.conf squid.conf.old
netkiller@Linux-server:/etc/squid$ sudo vi squid.conf

生成自己的squid.conf文件,这样比较清晰

$ grep '^[a-z]' squid.conf.old > squid.conf

40.2.3.1. 正向代理

# cat squid.conf
acl manager proto cache_object
acl localhost src 127.0.0.1/32 ::1
acl to_localhost dst 127.0.0.0/8 0.0.0.0/32 ::1
acl SSL_ports port 443
acl Safe_ports port 80          # http
acl Safe_ports port 21          # ftp
acl Safe_ports port 443         # https
acl Safe_ports port 70          # gopher
acl Safe_ports port 210         # wais
acl Safe_ports port 1025-65535  # unregistered ports
acl Safe_ports port 280         # http-mgmt
acl Safe_ports port 488         # gss-http
acl Safe_ports port 591         # filemaker
acl Safe_ports port 777         # multiling http
acl CONNECT method CONNECT
http_access allow manager localhost
http_access deny manager
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localhost
http_access allow all
#http_access deny all
http_port 3128
hierarchy_stoplist cgi-bin ?
coredump_dir /var/spool/squid3
refresh_pattern ^ftp:           1440    20%     10080
refresh_pattern ^gopher:        1440    0%      1440
refresh_pattern -i (/cgi-bin/|\?) 0     0%      0

refresh_pattern -i \.css$ 	1440 50% 129600 reload-into-ims
refresh_pattern -i \.js$ 	1440 90% 129600 reload-into-ims
refresh_pattern -i \.hml$ 	1440 90% 129600 reload-into-ims
refresh_pattern -i \.html$ 	1440 90% 129600 reload-into-ims
refresh_pattern -i \.shtml$ 1440 90% 129600 reload-into-ims
refresh_pattern -i \.xml$ 	1440 50% 129600 reload-into-ims
refresh_pattern -i \.jpg$ 	1440 90% 129600 reload-into-ims
refresh_pattern -i \.png$ 	1440 90% 129600 ignore-reload
refresh_pattern -i \.gif$ 	1440 90% 129600 ignore-reload
refresh_pattern -i \.bmp$ 	1440 90% 129600 ignore-reload

refresh_pattern -i \.mp3$ 	1440 50% 2880 ignore-reload
refresh_pattern -i \.wmv$ 	1440 50% 2880 ignore-reload
refresh_pattern -i \.rm$ 	1440 50% 2880 ignore-reload
refresh_pattern -i \.swf$ 	1440 50% 2880 ignore-reload
refresh_pattern -i \.mpeg$ 	1440 50% 2880 ignore-reload

refresh_pattern -i \.doc$ 		1440    50%     2880      ignore-reload
refresh_pattern -i \.ppt$ 		1440    50%     2880      ignore-reload
refresh_pattern -i \.xls$ 		1440    50%     2880      ignore-reload
refresh_pattern -i \.pdf$ 		1440    50%     2880      ignore-reload
refresh_pattern -i \.rar$       1440    50%     2880      ignore-reload
refresh_pattern -i \.zip$ 		1440    50%     2880      ignore-reload
refresh_pattern -i \.txt$ 		1440    50%     2880      ignore-reload

refresh_pattern .               0       20%     4320

设置代理服务器

declare -x ftp_proxy="192.168.0.1:3128"
declare -x ftps_proxy="192.168.0.1:3128"
declare -x http_proxy="192.168.0.1:3128"
declare -x https_proxy="192.168.0.1:3128"

检查Cache工作情况

# declare -x http_proxy="172.16.0.5:3128"

# curl -I http://www.qq.com
HTTP/1.0 200 OK
Server: squid/3.0
Date: Wed, 15 Jun 2011 07:54:36 GMT
Content-Type: text/html; charset=GB2312
Vary: Accept-Encoding
Expires: Wed, 15 Jun 2011 08:09:36 GMT
Cache-Control: max-age=900
Vary: Accept-Encoding
X-Cache: HIT from rainny.qq.com
X-Cache: MISS from localhost
X-Cache-Lookup: MISS from localhost:3128
Via: 1.0 localhost (squid/3.1.6)
Proxy-Connection: keep-alive

# curl -I http://www.qq.com
HTTP/1.0 200 OK
Server: squid/3.0
Date: Wed, 15 Jun 2011 07:54:36 GMT
Content-Type: text/html; charset=GB2312
Vary: Accept-Encoding
Expires: Wed, 15 Jun 2011 08:09:36 GMT
Cache-Control: max-age=900
Vary: Accept-Encoding
X-Cache: HIT from rainny.qq.com
Age: 2
X-Cache: HIT from localhost
X-Cache-Lookup: HIT from localhost:3128
Via: 1.0 localhost (squid/3.1.6)
Proxy-Connection: keep-alive

当第二次请求同一个URL的时候X-Cache: 由MISS变为HIT，表示已经被缓存

40.2.3.2. 代理服务器

加入权限认证

netkiller@Linux-server:/etc/squid$ sudo htpasswd -c /etc/squid/squid_passwd neo
New password:
Re-type new password:
Adding password for user neo
netkiller@Linux-server:/etc/squid$

netkiller@Linux-server:/etc/squid$ sudo find / -name ncsa_auth
/usr/lib/squid/ncsa_auth

#
# Add this to the auth_param section of squid.conf
#
auth_param basic program /usr/lib/squid/ncsa_auth /etc/squid/squid_passwd

#
# Add this to the bottom of the ACL section of squid.conf
#
acl ncsa_users proxy_auth REQUIRED
acl business_hours time M T W H F 9:00-17:00

#
# Add this at the top of the http_access section of squid.conf
#
http_access allow ncsa_users business_hours

extension_methods REPORT MERGE MKACTIVITY CHECKOUT # subversion

extension_methods REPORT MERGE MKACTIVITY CHECKOUT

默认端口 3128 如果你不想改squid.conf,可以使用iptables映射

iptables -t nat -A PREROUTING -i eth0 -p tcp -s 0.0.0.0/0.0.0.0 --dport 80 -j REDIRECT --to-ports 3128

设置你的浏览器，并测试

40.2.3.3. Squid作为反向代理Cache服务器(Reverse Proxy)

这里我们将apache和squid安装在一台服务器上

过程 40.1. 配置步骤

  1. 配置Apache监听端口

    netkiller@Linux-server:~$ cd /etc/apache2/
netkiller@Linux-server:/etc/apache2$ sudo cp ports.conf ports.conf.old
netkiller@Linux-server:/etc/apache2$ sudo vi ports.conf
Listen 8080
Listen 443
netkiller@Linux-server:/etc/apache2$ sudo /etc/init.d/apache2 restart
 * Forcing reload of apache 2.0 web server...                                                                                          [ ok ]
netkiller@Linux-server:/etc/apache2$

    restart/reload后测试一下

    http://localhost:8080/

  2. squid 2.5 之前的版本 
    netkiller@Linux-server:/etc/apache2$ cd ../squid/
netkiller@Linux-server:/etc/squid$ sudo vi squid.conf
http_port 80
httpd_accel_host localhost
httpd_accel_port 8080
httpd_accel_single_host on
httpd_accel_with_proxy on
httpd_accel_uses_host_header off
netkiller@Linux-server:/etc/squid$ sudo /etc/init.d/squid reload
 * Reloading Squid configuration files
   ...done.
netkiller@Linux-server:/etc/squid$

    squid 2.5 之前的版本

    对公网主机220.201.35.11:80做Cache

    netkiller@Linux-server:/etc/apache2$ cd ../squid/
netkiller@Linux-server:/etc/squid$ sudo vi squid.conf
http_port 80
httpd_accel_host 220.201.35.11
httpd_accel_port 80
httpd_accel_single_host on
httpd_accel_with_proxy on
httpd_accel_uses_host_header off
netkiller@Linux-server:/etc/squid$ sudo /etc/init.d/squid reload
 * Reloading Squid configuration files
   ...done.
netkiller@Linux-server:/etc/squid$

    多台主机做Cache

    netkiller@Linux-server:/etc/apache2$ cd ../squid/
netkiller@Linux-server:/etc/squid$ sudo vi squid.conf
http_port 80
httpd_accel_host virtual
httpd_accel_port 8080
httpd_accel_single_host on
httpd_accel_with_proxy on
httpd_accel_uses_host_header off
netkiller@Linux-server:/etc/squid$ sudo /etc/init.d/squid reload
 * Reloading Squid configuration files
   ...done.
netkiller@Linux-server:/etc/squid$
  3. squid 2.6之后版本的配置

    localhost

    http_port 80 defaultsite=localhost vhost transparent
cache_peer localhost parent 8080 0 no-query originserver

    其它主机

    http_port 80 defaultsite=192.168.1.2 vhost transparent
cache_peer 192.168.1.2 parent 80 0 no-query originserver
  4. 2.7/3.0 版本 
    visible_hostname netkiller.8800.org

http_port 80 accel vhost vport

cache_peer 127.0.0.1 parent 8080 0 no-query originserver name=mainsite
cache_peer 127.0.0.1 parent 8080 0 no-query originserver name=site1
cache_peer_domain mainsite netkiller.8800.org
cache_peer_domain site1 neo.ohyeap.com
http_access allow all
  5. 注意事项

    ERROR

    The requested URL could not be retrieved

    * Access Denied

    出现上面错说，关闭http_access deny all

    # And finally deny all other access to this proxy

    #http_access deny all

#squid.conf
#服务器IP 192.168.1.1
#监听服务器的80端口，透明代理，支持域名和IP的虚拟主机
http_port 192.168.1.1:80 transparent vhost vport

#限制同一IP客户端的最大连接数
acl OverConnLimit maxconn 16
http_access deny OverConnLimit

#防止天涯盗链，转嫁给百度
acl tianya referer_regex -i tianya
http_access deny tianya
deny_info http://www.baidu.com/logs.gif tianya

#防止被人利用为HTTP代理，设置允许访问的IP地址
acl myip dst 192.168.1.1
http_access deny !myip

#防止百度机器人爬死服务器
acl AntiBaidu req_header User-Agent Baiduspider
http_access deny AntiBaidu

#允许本地管理
acl Manager proto cache_object
acl Localhost src 127.0.0.1 192.168.1.1
http_access allow Manager Localhost
http_access deny Manager

#仅仅允许80端口的代理
acl Safe_ports port 80 # http
http_access deny !Safe_ports
http_access allow all

#Squid信息设置
visible_hostname netkiller.8800.org
cache_mgr openunix@163.com

#基本设置
cache_effective_user squid
cache_effective_group squid
tcp_recv_bufsize 65535 bytes

#2.5的反向代理加速配置
#httpd_accel_host 127.0.0.1
#httpd_accel_port 80
#httpd_accel_single_host on
#httpd_accel_uses_host_header on
#httpd_accel_with_proxy on
#2.6的反向代理加速配置
#代理到本机的80端口的服务，仅仅做为原始内容服务器
cache_peer 127.0.0.1 parent 80 0 no-query originserver

#错误文档
error_directory /usr/local/squid/share/errors/Simplify_Chinese

#单台使用，不使用该功能
icp_port 0

40.2.3.4. 代理+反向代理

http_port 80 vhost vport defaultsite=220.201.35.11
http_port 88
......
......
acl Manager proto cache_object
acl Localhost src 127.0.0.1/32
acl Safe_ports port 80
acl all src 0.0.0.0/0.0.0.0
acl ACCEL_DST dst 127.0.0.1/32 220.201.35.11/32

acl ACCEL_MODE  myport 80
acl PROXY_MODE  myport 88
# Authentation
auth_param basic realm Please Login
auth_param basic program /usr/local/squid/libexec/ncsa_auth /usr/local/squid/etc/passwd
acl VALIDUSER proxy_auth plan9

# ACCEL MODE
# -----------------------------------------------------------------------------
cache_peer 10.34.2.93 parent 80 0 no-query originserver
cache_peer_access 220.201.35.11 allow ACCEL_MODE
cache_peer_access 220.201.35.11 deny all

http_access allow ACCEL_DST Safe_ports
http_access allow PROXY_MODE VALIDUSER
http_access deny !Safe_ports
http_access allow ACCEL_MODE
http_access allow Manager Localhost
http_access deny all
icp_access deny all

40.2.4. Squid 管理

40.2.4.1. squidclient

squidclient -- client interface to the squid cache

squidclient 使用方法

  1. 运行状态信息： squidclient -p 80 mgr:info
  2. 内存使用情况： squidclient -p 80 mgr:mem
  3. 磁盘使用情况： squidclient -p 80 mgr:diskd
  4. 已经缓存的列表： squidclient -p 80 mgr:objects. use it carefully,it may crash
  5. 强制更新url：squidclient -p 80 -m PURGE http://netkiller.8800.org/index.html
  6. 查看更多信息：squidclient -h 或者 squidclient -p 80 mgr:
debian:~# squidclient -p 80 mgr:squidaio_counts
HTTP/1.0 200 OK
Server: squid/2.6.STABLE5
Date: Sun, 29 Apr 2007 13:27:09 GMT
Content-Type: text/plain
Expires: Sun, 29 Apr 2007 13:27:09 GMT
Last-Modified: Sun, 29 Apr 2007 13:27:09 GMT
X-Cache: MISS from debian.example.org.example.org
X-Cache-Lookup: MISS from debian.example.org.example.org:80
Via: 1.0 debian.example.org.example.org:80 (squid/2.6.STABLE5)
Connection: close

ASYNC IO Counters:
Operation       # Requests
open    0
close   0
cancel  0
write   0
read    0
stat    0
unlink  0
check_callback  0
queue   0
debian:~#

squidclient -p 80 mgr:5min

40.2.4.2. reset cache

重做 cache

mkdir /var/spool/squid
chown proxy.proxy -R /var/spool/squid
netkiller@Linux-server:~$ sudo squid -z
netkiller@Linux-server:~$ sudo squid -k reconfigure

40.2.5. 禁止页面被Cache

加到head中

  HTML
          <META   HTTP-EQUIV="pragma" CONTENT="no-cache">
          <META   HTTP-EQUIV="Cache-Control" CONTENT="no-cache, must-revalidate">
          <META   HTTP-EQUIV="expires" CONTENT="Wed, 26 Feb 1978 08:21:57 GMT">
  ASP
  <%
          Response.Expires = -1
          Response.ExpiresAbsolute = Now() - 1
          Response.cachecontrol = "no-cache"
  %>
  PHP
          header("Expires: Mon, 26 Jul 1997 05:00:00 GMT");
          header("Cache-Control: no-cache, must-revalidate");
          header("Pragma: no-cache");
  JSP
          response.setHeader("Pragma","No-Cache");
          response.setHeader("Cache-Control","No-Cache");
          response.setDateHeader("Expires",   0);
  C#中禁止cache的方法！
          Response.Buffer=true;
          Response.ExpiresAbsolute=System.DateTime.Now.AddSeconds(-1);
          Response.Expires=0;
          Response.CacheControl="no-cache";

让浏览器发送no-cache头,只需Ctrl+f5刷新

40.2.6. Squid 实用案例

40.2.6.1. Squid Apache/Lighttpd 在同一台服务器上

squid 与 web server 在同一台服务器上,一般情况是squid 监听80端口, web server 监听其它端口(一般是8080)

用户访问时通过80端口访问服务器.不想让用户访问8080.

  1. web server

    Apache httpd.conf文件Listen 8080 改成IP:Port,这样8080端口只允许本地访问

    Listen 127.0.0.1:8080

    lighttpd

    vi /etc/lighttpd/lighttpd.conf
server.port               = 8080
server.bind               = "localhost"

/etc/init.d/lighttpd reload

    本地测试

    curl http://127.0.0.1:8080/
  2. Squid 
    http_port 80 defaultsite=localhost vhost
cache_peer localhost parent 8080 0 no-query originserver

acl our_networks src 172.16.0.0/16
http_access allow our_networks
http_access allow all

    测试

    curl http://127.0.0.1/

    在其它电脑上用IE访问http://your_ip/ 可以看到你的主页

    在其它电脑上用IE访问 http://ip:8080/ 应该是无法访问

  3. 另一种方法是使用 iptables 实现 
    /sbin/iptables -A INPUT -i eth0 -p tcp --dport 8080 -j DROP
/sbin/iptables -A INPUT -i lo -p tcp --dport 8080 -j ACCEPT

使用 nmap 工具还是可以看到8080存在的.

# nmap localhost 

debian:~# nmap localhost

Starting Nmap 4.11 ( http://www.insecure.org/nmap/ ) at 2007-04-29 08:28 EDT
Interesting ports on localhost (127.0.0.1):
Not shown: 1670 closed ports
PORT     STATE SERVICE
22/tcp   open  ssh
25/tcp   open  smtp
53/tcp   open  domain
80/tcp   open  http
111/tcp  open  rpcbind
113/tcp  open  auth
548/tcp  open  afpovertcp
901/tcp  open  samba-swat
953/tcp  open  rndc
8080/tcp open  http-proxy

Nmap finished: 1 IP address (1 host up) scanned in 0.268 seconds

40.2.6.2. 用非 root 用户守护 Squid

squid.conf

http_port 3128 transparent vhost vport

iptables 做端口重定向

iptables -t nat -A PREROUTING -j REDIRECT -p tcp --destination-port 80 --to-ports 3128

40.2.7. squid+icap+clamav

squid+icap+clamav

http://icap-server.sourceforge.net/squid.html http://wiki.squid-cache.org/Features/ICAP

原文出处：Netkiller 系列 手札
本文作者：陈景峯
转载请与作者联系，同时请务必标明文章原始出处和作者信息及本声明。

时间： 2024-09-24 08:44:55

40.2. Squid - Internet Object Cache (WWW proxy cache)的相关文章

shared pool系列三：library cache结构/library cache object的结构

library cache结构/library cache object的结构-dump LibraryHandle Library cache结构 Library cache最主要的功能就是存放用户提交的SQL语句,SQL语句相关的解析树(解析树也就是对SQL语句中所涉及到的所有对象的展现)--->共享SQL区(shared SQL areas),私有SQL区(private SQLareas,如果配置了共享服务器),执行计划,用户提交的PL/SQL程序块(包括匿名程序块,存储过程,包,函数等

构建ASP.NET MVC4+EF5+EasyUI+Unity2.x注入的后台管理系统（40）-精准在线人数统计实现-【过滤器+Cache】

原文:构建ASP.NET MVC4+EF5+EasyUI+Unity2.x注入的后台管理系统(40)-精准在线人数统计实现-[过滤器+Cache] 系列目录 上次的探讨没有任何结果,我浏览了大量的文章和个别系统的参考!决定用Cache来做,这可能有点难以接受但是配合mvc过滤器来做效果非常好! 由于之前的过滤器我们用过了OnActionExecuting这个方法来判断权限 现在在方法被执行后我们用OnActionExecuted来监听用户的操作和刷新用户在线列表 首先下载http://files

Oracle中library cache和dictionary cache概述

library cache最主要的功能就是存放用户提交的SQL语句.SQL语句相关的解析树(解析树也就是对SQL语句中所涉及的所有对象的展现 ). 执行计划.用户提交的PL/SQL程序块(包括匿名程序块.存储过程.包.函数等)以及它们转换后能够被Oracle执行的代码等.为了 对这些 内存结构进行管理,library cache中还存放了很多控制结构,包括lock.pin.dependency table等. library cache也存放了很多的数据库对象的信息,包括表.索引等.有关这些数据

Buffer cache和page cache有何区别

Page cache和buffer cache到底有什么区别呢?很多时候我们不知道系统在做IO操作的时候到底是走了page cache还是buffer cache?其实,buffer cache和page  cache是Linux中两个比较简单的概念,在此对其总结说明. Page cache是vfs文件系统层的cache,例如 对于一个ext3文件系统而言,每个文件都会有一棵radix树管理文件的缓存页,这些被管理的缓存页被称之为page cache.所以,page cache是针对文件系统而言

Java Cache系列之Cache概述和Simple Cache

前记:最近公司在做的项目完全基于Cache(Gemfire)构建了一个类数据库的系统,自己做的一个小项目里用过Guava的Cache,以前做过的项目中使用过EHCache,既然和Cache那么有缘,那就趁这个机会好好研究一下Java中的Cache库.在Java社区中已经提供了很多Cache库实现,具体可以参考http://www.open-open.com/13.htm,这里只关注自己用到的几个Cache库而且这几个库都比较具有代表性:Guava中提供的Cache是基于单JVM的简单实现:EHC

buffer cache实验5-latch:cache buffers chain

1.CBC latch产生的原理: 一次逻辑读时CBC latch锁及Buffer pin锁的获取和释放过程如下: 1.加Latch X 2.进入hash chain,在相应的BH上加Buffer pin S (0-->1) 3.释放Latch X 4.进行逻辑读--也就是通过BH中的buffer adderss找到数据块在内存中真实位置  ---假如读了1MS 5.加Latch X 6.释放Buffer pin S (1-->0)  0:没锁  1:共享锁  2:独占锁 7.释放Latch

squid源码安装的服务启动脚本

#!/bin/bash # squid This shell script takes care of starting and stopping # Squid Internet Object Cache # # chkconfig: - 90 25 # description: Squid - Internet Object Cache. Internet object caching is \ # a way to store requested Internet objects (i.e

Linux下Squid代理服务器的架设与维护经验分享_Linux

通过架设专门的WWW(FTP)代理来满足用户的主要需求,通过架设socks5代理来满足用户的其他需求. 一.对使用者的分析 现有网络情况: 我校校园网通过光缆已将31座建筑物连通,光缆总长度约15Km,绝大多数楼中实现结构化布线,连入校园网的网络多媒体教室.教学基地.实验室.机房等约有数十个,连网计算机达3000多台 .我校目前出口有2个,一条速率为10M bps,通过光纤接入中国教育科研网CERNET,另一条速率为4M bps连入中国电信. 用户的需求: 我校校园网的使用者主体为在校学生及老师

Website Cloud Architecture Best Practices

This article introduces the technical challenges brought by cloud technology to traditional enterprises, and explains in depth the best practices of cloud architecture. It was first published in Lingyun, a magazine jointly sponsored by Alibaba Cloud