You have quickly isolated one of the most controversial topics regarding Zenoss: device dependency. Simply put, there is no layer 2 device dependency mechanism in Zenoss. mray (one of the developers) has added a ticket here. So we know the development team is aware of the issue but there does not appear to be a current solution.

I believe your assessment is correct - your collectors are timing out if numerous devices are unreachable. The sum of the timeouts is probably greater than the cycle interval. Your use of multiple collectors is a good idea. If you are using Enterprise, I understand you can set up multiple collectors on a single system. This should alleviate some of the bottlenecks when devices become unreachable. With so many devices to be monitored, it may be necessary to use even more collectors.

I don't think there is any magic formula for determining hardware requirements. They are determined by how you configure your system, not the number of devices monitored. If you are collecting 200 data points per cycle from 300 devices, you will need more hardware resources than if you are simply pinging 1000 devices. There are a number of factors to consider. You need enough bandwidth to send/receive all the data in each cycle, your hard drives need enough space and speed to store all the data you are recording each cycle, event complexity can impact the processors, etc.

agreed, this is actually a showstopper for us, can't have 2-300 alerts coming in when only one device is down.

only "solution" we have found so far is: docs/DOC-3215

And the auto-discover in the route table dependency model is basicly no good, as alot of our equipment are cisco switches with a management vlan on layer 2, meaning we can ping the device, but the device does no routing, it only act as a layer2 switch/bridge.

I'll quoute up a post from july 2009 to remind the devs that this is very much needed:

Matt Ray: "This feature is high on our feature request list but we have not
gotten  to it yet because we have fairly limited development resources
(5  developers). Much of the recent work has been targeted at
improving  stability and reliability with each release while squeezing
in new  features where we can. As seen in the "What do you want in
King  Crab" thread (http://forums.zenoss.com/viewtopic.php?t=9296),
this  is quite popular request and we are very much aware of it."

Ref: thread/9845

Hmmm... alternatively, if you created an alerting rule just for ping events that dictated a delay that was twice as long as the ping cycle, (assuming your ping cycle has been calculated to account for the amount of time it would take to ping a significant number of devices that cannot be found)  that should ensure that all ping events from the previous cycle would be active. When the next cycle begins, the event command would be able to find and clear out any events for "lower" devices, leaving only the ping events for the devices at the top of the hierarchy.

This still doesn't account for IP service monitoring, etc...