Oct 5, 2009 8:20 PM
Clear database to cleanup disk space
-
Like (0)
Hi folks,
I'm fairly new to zenoss and linux, (we're running CentOS 5 and zenoss 2.5.3 (i think)), i took over this from a previous tech who is unavailable to help.
Anyways, my zenoss is not functioning - main problem appears to be disk space: When attempting to start Zenoss i see the error: IOError [Errno 28] No space left on Device.
This is all running in a VM, so i could rezise the virtual disk, but than i'd have to deal with changing the Guest OS parition. I have no problem with ditching all old logs that Zenoss has collected. I'm wondering if there is a command i can run to do that, or if there is a file i can just delete.
thanks
-c
hmmm. I'm unable to type any commands in the console - ever since it tried to load zenoss and showed the disk space error, it looks like this:
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . and so on - keeps adding periods.
I see.
You should be able to start linux VM in single-user mode during boot process; after that you'll be able to clean disk.
Other option is to mount VM disk image using some VMWARE utility (I don't remember its name)
screen shot of console attached (ran df - h)
and screen shot of console after running du --max-depth=1 -h
-c
and what is
du /var | sort -rn |head -30
output?
Message was edited by: vitaly_il
Sorry - is that the exact command? I tried it and got the following (attached)
Wow - just looked at this again - i fully don't know why i misread du /var as du -x
I asked for "du /var" output, sorted by size using "sort -rn"
du /var | sort -rn
("|" sign is for piping output of du to "sort" filter)
sorry my lack of linux knowledge is embarrassing.
Attached two outputs - can't find a way to cut and past from my VM console so i did one sorted and then one reverse sorted.
-c
You were right - DB is taking more than 4GB. If I understand properly, by default zenoss VM appliance uses Mysql DB with InnoDB engine with "all DB in single file" mode. In this case even after cleaning DB data - size of DB file won't reduce.
I see two ways for fixing disk space problem
1)
You can free about 300M disk by cleaning /var/log directory - you can delete all files with *.1.*, *.2.* and similar filenames (like maillog.1, maillog.3.bz2).
In addition, there is about 70 MB under /var/cache/yum - you can delete all files here - "rm -rf /var/cache/yum/*"
Hopefully after this you'll be able to start zenoss. When zenoss is up, go to "Event Manager" and decrease "delete historical events older than" value.
AFAIK, it should prevent from DB to grow (I didn't try this).
In addition, it seems that there are some orphaned records into "event" table and google reports some howto's like this docs/DOC-3220/. But for some reason - probably because zenoss site reorganization, this howto is missed major part.
2)
add more disk space to zenoss VM and either increase current "/" partition or add new partition and migrate /var/lib/mysql to the new partition.
(some linux admin skills are requered for this)
thanks - that cleared up 200 MB approx. But not enough for mysql and zenoss to start.
looks like i may have to create another parition and migrate mysql - can you point me to any docs on how to do this?
-c
Here is one way to do this (another way is to add new disk space to your current filesystem, but IMHO it's more tricky):
1) add new disk to Zenoss VM using some VMWARE tool. I suggest to add at least 10GB space.
2) 2.1 create one big partition on this disk.
fdisk /dev/sdb ("fdisk" is command-line utility for disk partitioning, you will receive text menu)
2.2 create linux filesystem on this disk. Exactly command will depend on name of your new disk - you'll get it from VMWARE.
mkfs.ext3 /dev/sdb1
3) mount new partition ad add appropriate directive to "/etc/fstab" for doing this every boot:
3.1) mkdir /var/lib/mysql1
mount /dev/sdb1 /var/lib/mysql1
chown -R mysql /var/lib/mysql1
3.2)
add this line
/dev/sdb1 /var/lib/mysql1 ext3 defaults 1 1
at the end of /etc/fstab file
4) [this should be done in single-user mode]
copy zenoss mysql DB from old to new location
cp -ar /var/lib/mysql/* /var/lib/mysql1
5) edit /etc/my.cnf -
datadir=/var/lib/mysql1
6) reboot!
in the best case, system will start.
Even in this case, I suggest you:
- add zenoss box itself to monitoring and send alert regarding disk space , etc
- implement some scripts for prevent zenoss DB from growing
Follow Us On Twitter »
|
Latest from the Zenoss Blog » | Community | Products | Services Resources | Customers Partners | About Us | ||
Copyright © 2005-2011 Zenoss, Inc.
|
||||||||