Archived community.zenoss.org | full text search
Skip navigation
115084 Views 12 Replies Latest reply: Jan 5, 2010 3:30 PM by proactis RSS
proactis Rank: White Belt 9 posts since
Oct 5, 2009
Currently Being Moderated

Oct 5, 2009 8:20 PM

Clear database to cleanup disk space

Hi folks,

 

I'm fairly new to zenoss and linux, (we're running CentOS 5 and zenoss 2.5.3 (i think)), i took over this from a previous tech who is unavailable to help.

Anyways, my zenoss is not functioning - main problem appears to be disk space:  When attempting to start Zenoss i see the error: IOError [Errno 28] No space left on Device.

This is all running in a VM, so i could rezise the virtual disk, but than i'd have to deal with changing the Guest OS parition.  I have no problem with ditching all old logs that Zenoss has collected.  I'm wondering if there is a command i can run to do that, or if there is a file i can just delete.

 

thanks

 

-c

  • vitaly_il Rank: Green Belt 189 posts since
    Jul 8, 2008
    Currently Being Moderated
    1. Oct 6, 2009 4:38 AM (in response to proactis)
    Re: Clear database to cleanup disk space
    please send us output of "df -h" command.
  • vitaly_il Rank: Green Belt 189 posts since
    Jul 8, 2008
    Currently Being Moderated
    3. Oct 6, 2009 3:13 PM (in response to proactis)
    Re: Clear database to cleanup disk space

    I see.

    You should be able to start linux VM in single-user mode  during boot process; after that you'll be able to clean disk.

    Other option is to mount VM disk image using some VMWARE utility (I don't remember its name)

  • vitaly_il Rank: Green Belt 189 posts since
    Jul 8, 2008
    Currently Being Moderated
    5. Oct 6, 2009 3:51 PM (in response to proactis)
    Re: Clear database to cleanup disk space

    and what is

     

    du /var | sort -rn |head -30

     

    output?

     

    Message was edited by: vitaly_il

  • vitaly_il Rank: Green Belt 189 posts since
    Jul 8, 2008
    Currently Being Moderated
    7. Oct 6, 2009 4:03 PM (in response to proactis)
    Re: Clear database to cleanup disk space

    I asked for "du /var" output, sorted by size using "sort -rn"

     

    du /var | sort -rn

     

    ("|"  sign is for piping output of du to "sort" filter)

  • vitaly_il Rank: Green Belt 189 posts since
    Jul 8, 2008
    Currently Being Moderated
    9. Oct 6, 2009 4:52 PM (in response to proactis)
    Re: Clear database to cleanup disk space

    You were right - DB is taking  more than 4GB. If I understand properly, by default zenoss VM appliance uses Mysql DB with InnoDB engine with "all DB in single file" mode. In this case even after cleaning DB data - size  of DB file won't reduce.

     

    I see two ways for fixing  disk space problem

    1)

    You can free about 300M disk by cleaning /var/log directory - you can delete all files with *.1.*, *.2.* and similar filenames  (like  maillog.1, maillog.3.bz2).

    In addition, there is about 70 MB under /var/cache/yum - you can delete all files here - "rm -rf /var/cache/yum/*"

     

    Hopefully after this  you'll be able to start zenoss. When zenoss is up, go to "Event Manager" and decrease "delete historical events older than" value.

    AFAIK, it should prevent from DB to grow (I didn't try this).

    In addition, it seems that there are some orphaned records into "event" table and google reports some howto's like this docs/DOC-3220/. But for some reason - probably because zenoss site reorganization, this howto is  missed major part.

     

    2)

    add more disk space to zenoss VM and  either increase current "/" partition or add new partition and migrate /var/lib/mysql to the new partition.

     

    (some linux admin skills are requered for this)

  • vitaly_il Rank: Green Belt 189 posts since
    Jul 8, 2008
    Currently Being Moderated
    11. Oct 7, 2009 2:09 AM (in response to proactis)
    Re: Clear database to cleanup disk space

    Here is one way to do this (another way is to add new disk space to your current filesystem, but IMHO it's more tricky):

     

    1) add new disk to Zenoss VM using some VMWARE tool. I suggest to add at least 10GB space.

     

    2) 2.1 create one big partition on this disk.

     

    fdisk /dev/sdb   ("fdisk" is command-line utility for disk partitioning, you will receive text menu)

     

    2.2 create linux filesystem on this disk. Exactly command will depend on name of your new disk - you'll get it from VMWARE.

     

    mkfs.ext3 /dev/sdb1

     

    3) mount new partition ad add appropriate directive to "/etc/fstab" for doing this every boot:

     

    3.1)  mkdir /var/lib/mysql1

     

    mount /dev/sdb1 /var/lib/mysql1

     

    chown -R mysql /var/lib/mysql1

     

    3.2)

    add this line

     

    /dev/sdb1       /var/lib/mysql1       ext3    defaults        1       1

     

    at the end of /etc/fstab file

     

    4) [this should be done in single-user mode]

    copy zenoss mysql DB from old to new location

     

    cp -ar /var/lib/mysql/*  /var/lib/mysql1

     

    5) edit /etc/my.cnf  -

     

    datadir=/var/lib/mysql1

     

    6) reboot!

    in the best case, system will start.

     

    Even in this case, I suggest you:

    - add zenoss box itself to monitoring and send alert regarding  disk space , etc

    - implement some scripts for prevent  zenoss DB from growing

More Like This

  • Retrieving data ...

Legend

  • Correct Answers - 4 points
  • Helpful Answers - 2 points