"coyote" wrote:

 

The Senior Linux admin gave me a new Zenoss server with 16 CPUs, 36G of ram and lots of disk space.
I have Zenoss 2.4.2 installed and am trying to monitor 1800 devices.
Zenoss uses less that 12Gigs of ram and load is always less than 2. I have configured Zenhub with 8 workers.

vm.vfs_cache_pressure = 0
vm.swappiness = 20
vm.overcommit_memory = 2
vm.overcommit_ratio=75
vm.dirty_background_ratio = 1
vm.dirty_ratio = 100
vm.dirty_expire_centisecs = 3000 
vm.dirty_writeback_centisecs = 500 

echo 0 > /sys/block/cciss!c0d0/queue/iosched/front_merges
echo 150 > /sys/block/cciss!c0d0/queue/iosched/read_expire
echo 1500 > /sys/block/cciss!c0d0/queue/iosched/write_expire

<zodb_db temporary>
    # Temporary storage database (for sessions)
    <temporarystorage>
      name temporary storage for sessioning
    </temporarystorage>
    mount-point /temp_folder
    container-class Products.TemporaryFolder.TemporaryContainer
    cache-size 500000
    pool-size 16 
</zodb_db>

<zodb_db main>
  mount-point /
  # ZODB cache, in number of objects
  cache-size 500000
  pool-size 16
  <zeoclient>
    server localhost:8100
    storage 1
    name zeostorage
    var $INSTANCE/var
    
    # ZEO client cache, in bytes
    cache-size 10000MB
    # Uncomment to have a persistent disk cache
    #client zeo1
  </zeoclient>
</zodb_db>

# Reseting IPs for all Devices
for d in dmd.Devices.getSubDevices():
    d.setManageIp()
# Commiting changes for all devices
commit() 

# Reseting IPs for all Devices
for d in dmd.Devices.getSubDevices():
   # Reset IP and commit change
    d.setManageIp()
    commit() 

tmpfs /tmp     tmpfs rw 0 0

"coyote" wrote:

 

I have 6 SATA drives in a RAID 5 configuration, not the best performance but it is what the admin wants.

I can back off the monitoring and performance collection if I have too. sda8 is where Zenoss is installed.

# iostat
Linux 2.6.18-128.1.16.el5 (zmaster) 07/28/2009

avg-cpu: %user %nice %system %iowait %steal %idle
0.75 0.00 0.08 0.01 0.00 99.15

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 25.28 3.24 632.88 5326012 1040210946
sda1 0.00 0.02 0.00 25980 602
sda2 0.00 0.00 0.00 1877 0
sda3 0.31 0.40 5.03 652578 8261584
sda4 0.00 0.00 0.00 12 0
sda5 6.52 0.21 185.22 336968 304432832
sda6 0.15 1.50 2.63 2465424 4322896
sda7 1.08 0.00 261.88 6096 430430360
sda8 17.21 1.11 177.97 1828235 292514760
sda9 0.00 0.00 0.15 3619 246896
sda10 0.00 0.00 0.00 4607 1016

# Fix deviceSearch
brains = dmd.Devices.deviceSearch()
for d in brains:
    try:
        bah = d.getObject()
    except Exception:
        print "Removing non-existent device from deviceSearch: " + d.getPath()
        dmd.Devices.deviceSearch.uncatalog_object(d.getPath())
commit()

# Fix componentSearch
brains = dmd.Devices.componentSearch()
for d in brains:
    try:
        bah = d.getObject()
    except Exception:
        print "Removing non-existent device from componentSearch: " + d.getPath()
        dmd.Devices.componentSearch.uncatalog_object(d.getPath())
commit()

dmd.Devices.reIndex()
commit()

reindex()
commit()

I have a few questions regarding your environment. We plan to deploy a similar sizing to your environment and I am trying to any leverage lessons learned here.

• Considering the number of monitors you have, how big is your MySQL database?
  
• How much history do you have Zenoss set to keep (i.e.Events and ZenRRD)?
• What is your strategy for backup and recovery (Hot Backup / Cold Backup)?
• Do you have a sample of your my.cnf you can provide?

I would appreciate any information you can provide.

Thank you,

- Ken