Zenoss uses Zope as its web server.  For the most part Zope is tuned to be fairly quick.  However, when handling large numbers of simultaneous requests, its multi-threaded model runs into a bottleneck around the Python Global Interpreter Lock. The solution for this case is to run multiple zope instances with some type of load balancer in front to split requests across the zope instances.  But before we detail how to do this, we should talk about some of the caveats of this technique first.

Caveats

The root cause of all the drawbacks is that each zope instance will maintain its own its own database connection pool.  Because of this you should expect to see an increased load on the ZEO database.  Thus, if you already have high load on ZEO, you will want to find ways to decrease this first.  Further, because database caching occurs per-connection and you are globally increasing the number of connections, you should expect the memory usage of zope to rise significantly and you should tune your caching settings accordingly.  Lastly, you should be aware that when load balancing requests across multiple zope instances, one request may hit a zope instance with all the objects cached while the same request occurring again might hit a different zope instance which has to load objects from the database.  This is mostly mitigated by leaving your zope instances running for a while so that their caches are populated, but you should expect some variance in request speeds especially when the zopes are first started.

How To

1. Stop Zope: zopectl stop
2. Apply the attached patch to your Zenoss instance.  This patch will need to be applied each time you upgrade Zenoss.
3. Edit /opt/zenoss/bin/zopectl, changing the ZOPES parameter to an appropriate number.  The number will vary for each deployment, but a general guideline is to run the same number of Zopes as you have CPUs.  You may be able to go a bit above this depending on your setup and CPU features (like hyperthreading).
   
4. Start multiple zopes: zopectl start
5. Configure your load balancer to split the load across each of the servers.  This can be done trivially with apache.