Mailing lists
Help: performance mod_backhand
As mod_backhand is a web clustering tool, it should imply high performance. One of the many goals of mod_backhand is balancing assistance in environments where high performance is necessary. There are many considerations to make when designing a high performance cluster and mod_backhand will not compensate for poor network topology and poor site design in most cases. mod_backhand is designed to assist in serving your web site using as few resources as possible (using your resource as efficiently as possible).

Statistics transmission and collection can suffer under high load:
The mod_backhand module forks of a daemon that manages collection and transmission of statistics as well as pooled connections to other servers within the cluster. If your machine is highly loaded, this process will not be scheduled often enough and there will be a performance impact. A decent solutions to this is increasing the priority of that daemon. The PID is in the error_log and it will be the process that racks up a serious amount of processor time. Just renice it to -20 or set it to real time priority if your OS supports that.

mod_backhand is a proxy... let your OS help:
mod_bakchand, by default, uses application level proxying to do its dirty work. This means it reads data from another server and writes it back to the client. If the client is slow (over a modem or bad network connection) then the mod_bakchand Apache child stands to spend time doing nothing but waiting for the client to read all of the data.
On Unix machines with BSD compatible sochets, you are allowed to ask the kernel to change its socket buffer size. By default, it is usually around 4 kilobytes. This means that you can only write 4k before you are blocked waiting for the client at the other end to consume the information. Using the default Apache SendBufferSize directive, you can augment the buffer size to be something larger. If you averge document is 20 kilobytes, you should probably set this parameter up to 32k (32768). this will give you enough "room" to write the headers and the content back to the user and allows the kernel to feed it to the client.

mod_backhand can use a lot of files and open many file descriptors:
mod_backhand opens a file for every Apache child plus a handful at start up. It also needs a file descriptor for each pooled connection to another server. You need to make sure you don't overrun any preset limitations by your kernel or shell.

Under Linux, here are a few helpful optimizations:
# echo 32768 > /proc/sys/fs/file-max 
# echo 65536 > /proc/sys/fs/inode-max 

If you still have errors pertaining to a file descriptor shortage, you may need to recompile the kernel with NR_OPEN doubled.

Under Solaris: Add:
set rlim_fd_cur=2048
set rlim_fd_max=4096
to your /etc/system and reboot.

TCP/IP and the "Keep-Alive" blessing:
Establishing a TCP/IP connection to another machine is an operation that requires a substantial amount of work. People take this for granted because on most systems, TCP/IP sessions initiation happens at most a few time per second.

On a extremely busy cluster, you can expect to thousands of TCP/IP sessions built up and torn down every second. The processing time required of the kernel is tolling. The HTTP protocol allows a client to make a request in a fashion so as to not tear down the TCP/IP session. This will allow a client to establish a TCP/IP session to the server make many serialized requests.

Most web sites that use databases and push high volumes of traffic turn this feature off, because it has some bad side effects. The sessions current being managed by your cluster are no longer the length of a request plus response.

Let us assume a request takes and average of 10ms to turn around. If you serve 1000 requests in one second and on average a client is allowed to keep the session alive for 15 seconds without making a request, then you are no managing 15000 open sessions at any given moment instead of 100. This is a serious problem.

On the other hand, mod_backhand proxies requests to your other servers. You know that it will be making many requests and you obviously don't want to pay the overhead of building and tearing down TCP/IP sessions for each request that is backhanded.

You need to enable "Keep-Alive" functionality within the cluster regardless of whether or not it is enabled for the typical client. How do you do this in Apache? Unfortunately, as of Apache v1.3.9, you can't. But sense this can boost performance tremendously, we have a patch to the Apache source tree that will allow you to serve requests in this fashion. (And it is only a difference of two lines).

Here is the patch file and since it is so small, here it is complete:
*** http_protocol.c	Sat Aug 14 05:21:19 1999
---	Sat Jan 29 19:04:04 2000
*** 303,309 ****
                           "chunked") ||
           ((r->proto_num >= HTTP_VERSION(1,1)) &&
  	  (r->chunked = 1))) && /* THIS CODE IS CORRECT, see comment above. */
!         r->server->keep_alive &&
          (r->server->keep_alive_timeout > 0) &&
          ((r->server->keep_alive_max == 0) ||
           (r->server->keep_alive_max > r->connection->keepalives)) &&
--- 303,310 ----
                           "chunked") ||
           ((r->proto_num >= HTTP_VERSION(1,1)) &&
  	  (r->chunked = 1))) && /* THIS CODE IS CORRECT, see comment above. */
!         (r->server->keep_alive ||
! 		ap_table_get(r->headers_in, "BackhandProxied")) &&
          (r->server->keep_alive_timeout > 0) &&
          ((r->server->keep_alive_max == 0) ||
           (r->server->keep_alive_max > r->connection->keepalives)) &&

If you are currently using mod_backhand in a set up that has KeepAlives Off in the Apache configuration, this should remove thousands of TCP/IP connections per second for the overall operation. This could lead to a significant performance boost if your current bottleneck is related to kernel networking.

Copyright © 1999 Theo Schlossnagle. All rights reserved.