Simple is Best

A new telphone system based on voip two weeks ago, the TARGET is a low-cose call center on voip system link with pstn like 800 or 400 hotline.
I need to choose the voip server, client, pstn gateway and ata etc, it's boring but very challengingly, i like it:)

voip protocol:
h323(old man?), sip(common use but nat problem?), iax2(asterisk support, other supported server?),mgcp etc

voip server software:
asterisk(support sip,iax2,conditional h323), gnukeeper(support h323),
voip protocol, ser, sipx, trixbox(based on asterisk), callweaver etc, many servers but we need only one,

voip client:
softphone iax2 client(zoiper?), sip client(x-lite xten?), but the h323 client?, cisco hardphone?

voip pstn linker:
voip gateway, Digium pci card

voip hareware:
one Dell poweredge 1425 enough? or another application pc server?

then the other problem, performance and the stability and the expandability and the last one: COST

total is a long way, but i am on the way:)

don't use squid as reverse proxy(like squid) before a mod_deflate/mod_gzip apache/lighty/web server, especial for dynamtic pages, otherwise the reverse proxy cached page will be special linked to only one client(because of ziped page contain http cookie ?), thus the efficiency of the proxy will become terrible and the load of the behind server will increase a lot(depend on cached your pages and vistors).
deep in finding a solution.

upgrade from squid 2.5 to 2.6 because of the new epoll support under linux kernel 2.6, but the most import reason is squid 2.6 now support ssl communcation between cache server and web server, which is NOT support in squid 2.5 version.
like 2.5, before compile 2.6 u should check max file descriptor and install openssl, then compile and install like below

./configure --prefix=/opt/squid2.6 --enable-large-cache-files --with-large-files --with-aufs-threads=3 --with-pthreads --with-aio --enable-storeio=diskd,ufs,aufs --enable-ssl --with-openssl=/opt/openssl

the squid config file is changed too much from version 2.5, the new reverse proxy config file like below, enable ssl support and no longer need the custom log patch:

http_port 80 accel defaultsite=www.yourdomain.com vhost protocol=http
https_port 443 accel defaultsite=www.yourdomain.com cert=/etc/ssl/cacert.pem key=/etc/ssl/privkey.pem
vhost protocol=https
ssl_unclean_shutdown on
sslproxy_flags DONT_VERIFY_PEER

cache_peer 123.123.123.123 parent 443 0 no-query originserver name=server_www1 ssl sslflags=DONT_VERIFY_PEER

cache_peer_domain server_www1 www.youdomain.com others.yourdomain.com

hierarchy_stoplist cgi-bin ?

acl apache rep_header Server ^Apache
broken_vary_encoding allow apache

cache_mem 256 MB
cache_dir diskd /opt/cachedir_2.6 4000 64 256

logformat combined %>a %ui %un [%{%d/%b/%Y:%H:%M:%S +0000}tl] "%rm %ru HTTP/%rv" %Hs %h" "%{User-Agent}>h"
access_log /opt/squid2.6/var/logs/access.log squid
cache_log /opt/squid2.6/var/logs/cache.log
cache_store_log none

hosts_file /etc/hosts

refresh_pattern -i .jsp 360 25% 720 override-expire ignore-reload
refresh_pattern -i .php 360 20% 720 override-expire ignore-reload
refresh_pattern -i .htm 360 25% 720 override-expire ignore-reload
refresh_pattern -i .html 360 25% 720 override-expire ignore-reload
refresh_pattern -i .js 360 25% 720 override-expire ignore-reload

acl manager proto cache_object
acl localhost src 127.0.0.1/255.255.255.255
acl to_localhost dst 127.0.0.0/8
acl SSL_ports port 443
acl CONNECT method CONNECT

http_access allow manager localhost
http_access deny manager
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports

acl webserver_networks src 123.123.213.64/24
http_access allow webserver_networks

acl www1 dstdomain www.yourdomain.com youdomain.net
http_access allow www1

acl PURGE method PURGE
acl me src 127.0.0.1
http_access allow PURGE me
http_access deny PURGE
http_access deny all
icp_access allow all

cache_mgr admin@yourdomain.com
visible_hostname squid.yourdomain.com

client_persistent_connections off
server_persistent_connections off

coredump_dir /opt/squid2.6/var/cache
when compile with openssl under centos4/rhel4, a known issue if OpenSSL is compiled as a static library will raise, and make will failed, resolve is so simple. after running squid configure, manually edit src/Makefile and add -ldl after -lcrypto, then make& make install will be ok.
the new config file is simple different from the old complex 2.5 version config,  the performance will increase about 20-30% in our test when compile it with epoll under high concurrent speed. but the stability is need  more  firm, we encountered a ssl leak bug in early version, so often watch the new version changelog to resolve your problem.

posted by devil4heaven  # 12:57 AM 0 Comments

old document about 3 years ago
using reverse proxy to increase customer visit speed, concurrent performance etc,we use squid cluster to cache our dynamic jsp file before our web server cluster.

first compile with gcc under linux, without openssl, if u want,pls compile with the -with-openssl option,then make&make install

./configure --prefix=/opt/squid --disable-internal-dns --enable-async-io --enable-storeio=diskd,ufs --enable-removal-policies --e
nable-cache-digests --enable-poll --enable-gnuregex

edit the default squid.conf, open HTTPD-ACCELERATOR OPTIONS and others like http_port, our simple config file below:

http_port 80
#https_port 443 cert=/etc/ssl/cacert.pem key=/etc/ssl/privkey.pem
icp_port 0
hierarchy_stoplist cgi-bin ?
cache_mem 128 MB
cache_swap_low 90
cache_swap_high 95
maximum_object_size 6144 KB
maximum_object_size_in_memory 512 KB
ipcache_size 2048
fqdncache_size 2048
cache_replacement_policy heap GDSF
memory_replacement_policy heap LRU
cache_dir diskd /opt/cachedir 2048 32 256
#cache_access_log /opt/squid/var/logs/access.log
logformat combined %>a %ui %un [%{%d/%b/%Y:%H:%M:%S +0000}tl] "%rm %ru HTTP/%rv" %Hs %h" "%{User-Agent}>h"
cache_access_log /opt/squid/var/logs/access.log combined
cache_log /opt/squid/var/logs/cache.log
cache_store_log none
emulate_httpd_log on
debug_options ALL,1
dns_children 32
hosts_file /etc/hosts
redirect_children 50
redirect_rewrites_host_header off

auth_param basic children 50
auth_param basic realm Squid proxy-caching web server
auth_param basic credentialsttl 2 hours

refresh_pattern -i .jsp 360 25% 720 override-expire ignore-reload
refresh_pattern -i .php 360 20% 720 override-expire ignore-reload
refresh_pattern -i .htm 360 25% 720 override-expire ignore-reload
refresh_pattern -i .html 360 25% 720 override-expire ignore-reload
refresh_pattern -i .js 360 25% 720 override-expire ignore-reload



connect_timeout 2 minute
peer_connect_timeout 60 seconds
read_timeout 10 minutes
request_timeout 60 seconds
persistent_request_timeout 60 seconds
client_lifetime 10 minutes
half_closed_clients off
pconn_timeout 1 seconds
ident_timeout 5 seconds
shutdown_lifetime 45 seconds

acl all src 0.0.0.0/0.0.0.0
acl manager proto cache_object
acl localhost src 127.0.0.1/255.255.255.255
acl to_localhost dst 127.0.0.0/8
acl trust_group 123.123.123.0/255.255.255.0
acl SSL_ports port 443 563i
acl CONNECT method CONNECT
acl PURGE method PURGE
acl Safe_ports port 80 # http
acl Safe_ports port 443 563 # https, snews
acl PURGE method PURGE

http_access allow PURGE localhost
http_access deny PURGE
http_access allow manager localhost
http_access deny manager
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports

http_access allow trust_group

http_access deny all
http_reply_access allow all
icp_access allow all

cache_mgr admin@youdomain.com
visible_hostname squid.youdomain.com

httpd_accel_port 80
httpd_accel_host 222.60.80.65
httpd_accel_single_host off
httpd_accel_with_proxy off
httpd_accel_uses_host_header on

memory_pools off
memory_pools_limit 100 MB

coredump_dir /opt/squid/var/cache
ie_refresh off



in the config file, i patched squid 2.5 with the customlog patch from squid-cache.org, using refresh_pattern to force squid cache the jsp/php file and ie_fresh to anti the F5 fresh under ms ie and open the ssl port as u wish etc.ps: increase system file-max parameters to anti squid run out of filedescriptor(squid max use 32768,suggest to change file-max more than it), otherwise in high traffic squid will run out of file descrptor and response slow and slow.
the last thing is the /etc/hosts file, edit it to  add the reverse cache domain name and  ip address which resolve to the behind web server. DON'T use the outside name server.

one week ago, the user trace table in our mysql database reached about 2,00,000,000 rows, it used about 40G tablespace,
but works great without any crash last 24 months after last change.
we are using innodb engine on old mysql4.0.x version(self compiled) with rhel3/4,it began at 2005,which migrated from mysql4/rhas2.1 to mysql4/rhel3.
but after the upgrade,the db often crashed and the log only gave me an innodb internal error. i try to upgrad mysql to mysql4.0.27 and it still happened,  checked INSTALL-SOURCE from mysql.tar.gz,nothing wrong,but can't resolve the crash.
we used the compile  in INSTALL-SOURCE like below
./configure \
--prefix=/opt/mysql --enable-assembler \
--with-mysqld-ldflags=-all-static
but recompile can't reslove the crash, after about two days search, finally found mysqld manual compile like below works unstable with new glibc2.3  from rhel3 which began support nptl .
the solution is sample, compile the mysql without the ldflags all-static mark,  compile like below:
./configure --prefix=/opt/mysql --with-extra-charsets=complex --enable-thread-safe-client --enable-local-infile --enable-assembler --without-debug
after this compile, the innodb engine works perfect with nptl from glibc2.3, the reason maybe glibc2.3's backward compatibility for LinuxThreads.
Maybe it's time for mysql.tar.gz update the INSTALL-SOURCE txt:)

About two months ago, we changed our database server from five years old IBM x360 to a new Dell poweredge 6850 with four dual-kernel xeon and 32 G memory and raid5 array. Because we encountered the x86_32 memory limit and the x360 is too old for it's stability. 
So we changed from x86_32 to x86_64. the operation system migrated from rhas3 to rhas4(we even used rhas2.1 about seven years ago) x86_64 version, and database still using the Oracle 9i(9208) x86_64, and the not to upgrade to Oracle 10G because our old running applications based on oracle old rbo NOT new cbl. 
After the successfull os/database install, i setup the kernel  shmmax to 30G, Oracle started successful with default 2G sga, then i changed sga to 26G memory for better performance ,but when  startup oracle under sqlplus, it gived me  a strange ora-27102  error.
i checked the kernel shmmni,shmall,sem and file-max parameters etc again,nothing goes wrong,strange, except the ora-27102. 
after many hours search, the reason raised from deep water. the DEFAULT shmall is 219702 under rhas4, so the system max use memory is 219702*PAGE_SIZE(default 4096)=8G, so when the oracle try to use more than 8G memory, kernel can't permit do this, error ora-21702 then come.
The resolve is sample, changed shmall to 32212254720(30G) under /etc/sysctl.conf, reboot system or /sbin/sysctl -p to let kernel accpet the new setting, then successed to startup oracle with 26G memory.
It's the sample but boring shmall parameter under rhel4 x86_64 kernel.

正好最近需要把原来架设的web mail系统整体转移，顺便把原来放了N久的文档补齐吧:p..目的是为用户提供web方式的邮件系统，注意，只有web方式，不提供pop3等等方式，要求系统良好的扩展性以及相应的安全（防病毒、垃圾邮件）等等。
系统中apache+php+hivemail来提供web访问，hivemail通过pipe gateway默认于mta连接，
整个邮件系统涉及：
1.linux rhas3
2.mta postfix
3.database mysql
4.webmail apache+php+hivemail
5.anti-virus,anti-spam f-secure internet gatekeeper

A。操作系统，linux，习惯了rhas3 当然debain、suse等都可以，安装过程忽略。推荐使用较大的硬盘，大用户量下还是感觉mail放在硬盘上比放在mysql数据库里性能要好的多，硬盘大小根据你的用户量以及邮箱容量计算。 注意因为偶是采用的硬盘目录存储邮件，ext3文件系统存在单一目录下子目录不能超过32k的限制，推荐使用reiserfs文件系统。
B。postfix配置 而不是sendmail，可以使用rhas3自带的，也可以手工编译，偶是懒人，直接使用了rhas3自带的postfix，需要编辑更改一下main.cf以及alias文件main.cf需要更改myhostname、mydomain、myorigin、inet_interfaces、mydestination、mynetworks、relay_domains等等，整个相当于postfix作为mta开始对外使用，注意不要打开open relay针对我们的web系统，需要优化一下postfix，例如maximal_queue_lifetime、smtpd_timeout减小之类以避免影响性能最终需要配置main.cf，这个是使用pipe-gateway方式来连接hivemail，注意一下两项都需要
luser_relay = hivemail
local_recipient_maps =
这个是保证postfix把接受的邮件转移个hivemail用户，当然需要在alias文件里添加用户如下
hivemail: /opt/php/bin/hivemail_process.php
然后更新一下aliases.db# postalias aliases最后重启一下postfix服务，看看postfix是不是配置正确。
C。Mysql编译
偶不喜欢rh自带的mysql，使用的是mysql4.0.27源码编译的，参数如下
./configure --prefix=/resource/mysql --with-extra-charsets=complex --enable-thread-safe-client --enable-local-infile --enable-assembler --disable-shared --with-client-ldflags=-all-static --with-mysqld-ldflags=-all-static --without-debug
make & make install
然后参照INSTALL-SOURCE完成数据库安装，具体请查看INSTALL-SOURCE内容，配置/etc/my.cnf等等，包括需要的表类型、数据库引擎类型等等。 最终需要创建我们需要的数据库跟用户
mysql>create database hivemail;
mysql>grant all on hivemail.* to 'user'@'%' identified by 'pwd';
注意这里我们user用户是可以在任何地点连接的，需要通过防火墙阻止非信任ip对mysql 3306端口的访问。
D。apache＋php＋hivemail编译安装
注意这里没有使用lighty或者zeus，zeus4.2完全可以使用hivemail，只要正确配置的fastcgi，但lighty＋fastcgi＋php方式却不能正常使用hivemail，怀疑某个变量没有被传输过来，偶没有时间来继续做测试，有兴趣的可以测试一下。
apache2编译安装，偶需要对apache加装mod_evasive、mod_security、ssl等等，所以编译比较繁琐，正常情况下简化即可
./configure --prefix=/opt/apache2 --enable-so --enable-rewrite --enable-forward --enable-deflate --enable-headers --enable-ssl --with-ssl=/opt/openssl
make & make install
php编译安装，主要是--with-apxs2参数编译mod_php，参数如下，偶打开了fastcgi
./configure --prefix=/opt/php --enable-fastcgi --enable-force-cgi-redirect --with-config-file-path=/etc --with-zlib --with-mysql=/resource/mysql --with-xml --with-mysql-sock=/tmp/mysql.sock --enable-discard-path --with-apxs2=/opt/apache2/bin/apxs
make & make install
如果需要增加其它例如ecacceleator、zend optimizer之类的，请自己手动安装一提高php执行性能。
hivemail安装，这里我们是使用pipe gateway方式来连接postfix，偶这里是hivemail1.2.2，没有采用最新的hivemail1.3，因为偶改了N多东西，所以一直没有升级。
ⅰ。hivemail web安装很简单，仔细看INSTALL安装文件，需要把upload ftp到服务器上，然后修改upload/include/config.php，正确设置mysql连接，然后通过web访问upload/install/index.php通过web一路next即可。
注意这里有个sql_install.php，实际上就是它创建的hivemail各项使用表，默认的为myisam表格式，偶喜欢innodb，所以通过更改sql_install.php，直接创建为innodb表。
2。pipe gateway连接mta，需要把hivemail_process.php上传到我们的php执行路径下，编辑一下hivemail_process.php设置正确的各个路径，包括php路径，hivemail安装目录等等，最后chmod +x hivemail_process.php。
3.使用你创建的admin登陆，然后更改存储方式为storage method，即硬盘存储邮件方式，以减轻大用户量下mysql的压力。
E。防毒防垃圾邮件，推荐f-secure internet gatekeeper，比偶原来使用Trend IMSS性能好的多internet gatekeeper是优秀的网关防毒软件，当然也是商业软件:p，支持http、ftp、smtp、pop3防毒。
到f-secure下载个for linux的30天试用版本，tar zxvf解压，sh INSTALL.sh即默认安装到/opt/f-secure/fsigk下，默认的管理端口为127.0.0.1:9012，更改一下fsigk/etc/fsigk.ini改成公网ip重新启动fsigk_admin服务即可远程管理，这里打开smtp，监听9025端口，parent smtp设置127.0.0.1 25，打开virus以及spam检查，对于收到virus以及spam邮件自个设定如何处理。fsigk_smtp会自动启动，监听9025端口
注意这里需要我们把对25端口的收信转移到9025上以方便gatekeeper检查病毒、spam，然后再转移回postfix，利用iptables的nat端口转换功能来完成此项操作。
iptables -t nat -A PREROUTING -d publicip -p tcp -m tcp --dport 25 -j DNAT --to-destination publicip:9025
系统的优化：
偶使用的是hivemail1.2.2，性能在默认情况下有些问题，经过检查发现一个sql占用了大量的cpu，就是对hive_emailid的操作，表现为系统中有N个hivemail_process.php，mysql占用大量的系统cpu，对此表emailid字段创建index即可坚决此严重的性能问题。此问题曾经在hivemail forum多次出现，官方只推荐升级到hivemail1.
3。偶没有测试过1.3，所以不知道1.3是否解决此性能问题。至少在1.2.2上创建index后在偶的接近10W用户里没再出现相应的性能问题。
题外话：
hivemail作为商业web mail系统，已经2年没更新了，怀疑已经是个dead project，但其用户还是挺多的，对于中小型使用非常合适而且可以根据需要更改php程序，整体界面非常漂亮。这么死掉的话非常可惜。
最近发现的hivemail漏洞是因为对于各个id的检查不够，各位可以在相应的php程序开头加入intme($*id)以防止此注入。
要求安全性高的话可以在apache加装mod_security、mod_evasive等以防止一些常见的攻击。