Download
Mailing lists
FAQ
Frequently Asked Questions
Does mod_backhand support Apache 2?
What documents should I read to learn more about mod_backhand?
Are there mailing lists?
Does mod_backhand induce much overhead when it is not enabled for a directory?
How does mod_backhand know about the resources on other machines?
How does mod_backhand decide which is the "best" server?
Does mod_backhand only proxy? Can it HTTP redirect?
What do the built-in candidacy functions do?
How do I use the HTTPRedirectToName directive's format string?
Can I write my own candidacy functions?
Why does the load on my machine soar when I start Apache, and why does it take a few seconds (or minutes) to actually be available to serve pages?
mod_backhand won't work.. what is wrong?
I start backhand, but I can only see myself on the status page. Why?
I run Solaris and I get this weird error upon start up: "[error] (22)Invalid argument: shmctl(., IPC_SET, [-1,-1]) could not set segment #number", what is wrong?
When I start up Apache/mod_backhand, the backhand-handler tells me that the number of available servers and the total number of servers is 0. This isn't true, why is it wrong and how do I fix it?
Question:Does mod_backhand support Apache 2?
Answer:Nope. There has been thought, but no real work. The design of mod_backhand is very specific to both UNIX and a multiprocess-model. Apache 2 does away with the latter, so mod_backhand 2.x will need to be written from the ground up.
Question:What documents should I read to learn more about mod_backhand?
Answer:The papers, presentations and assorted documentation:
Question:Are there mailing lists?
Answer:Yes, visit http://lists.backhand.org/mailman/listinfo/ to join the users or developers lists.
Question:Does mod_backhand induce much overhead when it is not enabled for a directory?
Answer:Very little. The same as any other module.
Question:How does mod_backhand know about the resources on other machines?
Answer:When you configure the web server, you supply a mechanism for distributing resource information. The currently supported mechanisms are UDP based via Ethernet broadcast or IP multicast.
Question:How does mod_backhand decide which is the "best" server?
Answer:It uses what we call candidacy functions. A list of ALL the webservers are passed to the first function and that list is modified. Then it is passed to the next function and so on and so forth. The functions are executed in the order in which they appear in the configuration.
Question:Does mod_backhand only proxy? Can it HTTP redirect?
Answer:Version 1.0.9 and later have the ability to issues HTTP redirects to browsers instead of proxying them internally. See the HTTPRedirectTo{IP,Name} directives described in the "What do the built-in candidacy functions do?" section in this FAQ.
Question:What do the built-in candidacy functions do?
Answer:Their are current five built-in candidacy functions.
  • off -- this disables mod_backhand for a directory.
  • addSelf -- adds the local server to the end of the list of candidates if that server does not already exist in the list. If you are using a peer-based topology, you most often want to consider yourself and some configurations (byRandom, byLogWindow) could remove the local machine.
  • removeSelf -- removes the local server from the list of candidates if and only if that server already exists in the list. If you are using a tier-based topology, front-end servers most often would not like to include themselves. Of course, if you have multiple front-end machines you will need to eliminate all of them using a custom candidacy function or somethine like byHostname. removeSelf is great for testing.
  • byAge [time in seconds] -- eliminates servers that you have not received resource information for in a certain amount of time. The default is 20 seconds, but you can pass a parameter that is interpreted as an integer number of seconds.
  • byLoad [bias] -- Reorders the list of candidates from the lowest load to the highest load. The bias (a floating point number) is used to prefer yourself over proxying the request and can be used to approximate the effort involved in proxying a request. It is the amout of load subtracted from your load before you sort the servers.
  • byBusyChildren [bias] -- like the byLoad function, it reorders the list of candidates from least loaded to most loaded. Instead of using the system load (which has poor ganularity), it uses the number of busy Apache children. This should be a decent approximation of the length of the system run queue. The bias (an integer) is used to prefer yourself over proxying the request and can be used to approximate the effort involved in proxying a request. It is the amout of load (in run-queue units) subtracted from your load before you sort the servers. A good number is between 1 and 5.
  • byCPU -- like byLoad except work on the servers with the absolute highest CPU idle. This is mostly useless. Don't use it unless you really know why you are using it.
  • byLogWindow -- eliminates all but the first log base 2 of the n servers passed in. So, if 17 servers are passed in, the first 4 remain.
  • byRandom -- this function randomly (psuedo of course) reorders the list of servers given as input.
  • byCost -- this function attempts to assign a cost to the assingment of a request to each machine in the cluster. It then chooses the assignment that costs the least. The method of cost assignment is based on a cost-benefit framework as discussed in the paper titled "A Cost-Benefit Framework for Online Management of a Metacomputing System" by Amir, Awerbuch, and Borgstrom.
  • HTTPRedirectToIP -- Tell mod_backhand that it is to send clients to servers within the cluster via an HTTP redirect of the form http://1.2.3.4/request/uri rather than the default method of proxying.
  • HTTPRedirectToName [format string] -- Tell mod_backhand that it is to send clients to servers within the cluster via an HTTP redirect of the form http://format string/request/uri rather than the default method of proxying. The format string, if omitted will simply be the ServerName for the Apache server chosen. As this is not always a desirable choice, format string provides a means for a more intelligent hostname creation. It allows the construction of the new hostname based on the left portion of the ServerName (the Hostname on the backhand-handler page) and the right portion of the hostname provided from the Host: header. This facilitates clustered name based vurtual hosting setups. A more detailed example is outlined in the "How do I use the HTTPRedirectToName directive's format string?" section of this FAQ.
  • bySession [identifier] -- this function will attempt to find a cookie named [identifier] or a querystring variable named [identifier]. It will then attempt to hex decode the first 8 bytes of its content into an IPv4 style IP address. It will attempt to find this IP address in the list of candidates and if it is found it will make the server in question the only remaining candidate. If any of the above steps fail, it will not augment the candidacy list. This, plus a bit fo server side application code, can be used to implement sticky user sessions -- where a given user will always be delivered to the same server once a session has been established. [identifier] defaults to "PHPSESSID=" as it was original written to support PHP sessions by Martin Domig. For more information see: README.bySession file included in the distribution.
  • byHostname <regexp> -- this function is an example of a user built function that mod_backhand loads at run-time; it is not actually "built in". It eliminates all servers whose hostnames do not match the regular expression given as the parameter.

This can be cascaded to do fairly complicated things:

Say you want to redirect all database oriented cgi requests to your 64-bit architecture machines. you have the real names of you machines as intel1 through intel10, sun1 through sun10, and alpha1 through alpha10. Of course, you want to choose a random window of those machines (so as not to have EVERYONE go to the least loaded machine... because it would get clobbered). Of that window you want to pick the machine with the highest CPU idle. Oh yeah, of course you would like to avoid any machine you have not heard from in the last 6 seconds.
Backhand byAge 6
BackhandFromSO libexec/byHostname.so byHostname (sun|alpha)
Backhand byRandom
Backhand byLogWindow
Backhand byLoad
Pretty easy, huh?
Question:How do I use the HTTPRedirectToName directive's format string?
Answer:This directive is used to issue HTTP redirect responses to requests. When will it service the request rather than redirect? How will it preserve my domain name if the ServerName doesn't have the same domain name? These are all really a rephrasing of the same question. Here is the answer.
The format string is just like C format string except that it only has two insertion tokens: %#S and %#H (where # is a number).
  • %-#S will be the server name with the right # parts chopped off. If your server name is www-1.jersey.domain.com, %-3S will yield www-1
  • %#S will be the server name with only the # left parts preserved. If your server name is www-1.jersey.domain.com, %2S will yield www-1.jersey
  • %-#H will be the Host: with only the right # parts preserved. If the Host: is www.client.com, %-2S will yield client.com
  • %#H will be the Host: with the left # parts chopped off. If the Host: is www.client.com, %1H will yield client.com
For example, if you run a hosting company hosting.com and you have 5 machines named www[1-5].sanfran.hosting.com. You host www.client1.com and www.client2.com. You also add appropriate DNS names for www[1-5].sanfran.client[12].com.
Backhand HTTPRedirectToName %-2S.%-2H
This will redirect requests to www.client#.com to one of the www[1-5].sanfran.client#.com.
Question:Can I write my own candidacy functions?
Answer:byHostname.c is supplied as an example. The entire serverstats structure is available to you. This provides things like load, number of CPUs, available memory, total memory, number of Apache servers running (and number occupied) and some other things.
Apparently there have been many reported problems on various sytems regarding dynamically loaded objects from dynamically loaded modules in Apache. If you plan on making heavy use of run-time loadable candidacy functions, you should compile the mod_backhand module into Apache statically.

Oh yeah, the answer to this question is "Yes."
Question:Why does the load on my machine soar when I start Apache, and why does it take a few seconds (or minutes) to actually be available to serve pages?
Answer:You are (a) running mod_backhand version 1.1.0 or lower, (b) running mod_backhand for the first time, or (c) have deleted your mod_backhand-Arriba file. There is something we call "Arriba". This is how fast your machine's processor(s) are combined. We calculate this when apache starts... and yes it hurts. If you don't like it, you can modify the source so that it does not recalculate this value at start up. Maybe later this will be a run-time option.
Graceful restarts use the existing mod_backhand-Arriba file to determine the Arriba of your machine. So it doesn't hurt at all.
Question:mod_backhand won't work.. what is wrong?
Answer:Yikes! what a question! Well, a little troubleshooting guide.
  • mod_backhand uses SysV IPC, make sure you have shared memory and AF_UNIX domain sockets available to you (by loading the modules or compiling it into your kernel).
  • mod_backhand users AF_UNIX domain sockets and they are stored in the UnixSocketDir (set in httpd.conf). Make sure this directory is owned by Apache UID/GID pair, usually nobody/nobody. I suggest the permissions to be 0700.
  • mod_backhand logs to the ErrorLog, so look there.
  • mod_backhand has a diagnostic handler (like server-status). Do a
    <Location /backhand-status>
      SetHandler backhand-handler
      ... access permissions here ...
    </Location>
    
    and then do visit it. It is actually a nice monitoring tool for you cluster. I use it for that quite often. Think of it as a cluster like perftool.
  • Make sure that you have Backhand directives in the directory that you want to be balanced by backhand. It doesn't try to redirect requests if you don't ask it to.

Question:I start backhand, but I can only see myself on the status page. Why?
Answer:Hmm, this is complicated and could be caused my many misconfigurations. The most common is that of IP addresses. mod_backhand servers announce themselves to each other via IP datagrams (broadcast or multicast). In that packet, it says: "my IP is aaa.bbb.ccc.ddd". mod_backhand has a Directive called AcceptStats that will allow you to set up ACLs that can control the inclusion of other servers in your "candidates list".

Well, that leads us to: How does mod_backhand determine the IP address to advertise? Unless told otherwise, mod_backhand will call gethostbyname(gethostname()), so whatever IP the 'hostname' command results resolve to will stuffed in that packet.

With that said, how do you change it? The MulticastStats directive takes two forms:
MulticastStats <dest addr>:<port>[,ttl]
MulticastStats <myip addr> <dest addr>:<port>[,ttl]
So, your hostname is www3 and it resolves to 1.2.3.4, but you want to announce stats to 10.0.0.255 port 4445 and only accept stats from 10.0.0.0/24. By default, it will stuff 1.2.3.4 into the packet and the announcement will be trashed because it doesn't match 1.2.3.4. Also, if it annouces 1.2.3.4, it will try to proxy to 1.2.3.4 (and you want to use your high speed private network). For this type of (common) configuration, use the two argument form of MulticastStats, for example:
MulticastStats 10.0.0.4 10.0.0.255:4445

Question:I run Solaris and I get this weird error upon start up: "[error] (22)Invalid argument: shmctl(., IPC_SET, [-1,-1]) could not set segment #number", what is wrong?
Answer:The uid and gid have not been set for Apache in the configuration file yet, we don't know why Apache exhibits this behaviour on Solaris and not Linux. But, more importantly, we know how to fix it. Simply move the
User nobody
Group nobody
directive above the AddModule/LoadModule sections in httpd.conf. Weird, huh?
Question:When I start up Apache/mod_backhand, the backhand-handler tells me that the number of available servers and the total number of servers is 0. This isn't true, why is it wrong and how do I fix it?
Answer:We don't actually know why this is happening, the internals of Apache are somewhat confusing. I believe it has to do with the order of loading modules. The fact is that mod_backhand attaches to the scoreboard file in the parent process NOT in the child, this has somthing to do with it. But, again, we know how to work around this problem. Simply gracefully restart apache and the number will, from then on out, be correct and current.

Copyright © 1999 Theo Schlossnagle. All rights reserved.