Major Subjects: BSD release info, Heartbeat issues, zope catalogue search / AdvancedQuery, multiple collectors on the same physical host, SubConfigurationTaskSplitter, High Availability / HA / DRBD, MIB browsing tools, xkcd
[01-Sep-2011 11:00:49] <mrchippy> hola, everybody
[01-Sep-2011 11:02:01] <user3322> Pryvet
[01-Sep-2011 11:03:21] <Hackman238> mrchippy: Hey, how goes it?
[01-Sep-2011 11:03:47] <mrchippy> it goes crazy awesome
[01-Sep-2011 11:04:19] <Hackman238> user3322: ?
[01-Sep-2011 11:04:24] <Hackman238> mrchippy: Glad to ehar
[01-Sep-2011 11:05:06] <mrchippy> privyet is russian for howdy
[01-Sep-2011 11:05:08] <user3322> i thought it was "say hello in a different language day" so I used Russian… 
[01-Sep-2011 11:05:28] <Hackman238> mrchippy: Ah gotcha
[01-Sep-2011 11:05:48] <Hackman238> user3322: guten tag
[01-Sep-2011 11:05:49] <nyeates> Hi all
[01-Sep-2011 11:06:07] <Hackman238> nyeates: Hey, how goes the war?
[01-Sep-2011 11:07:27] <nyeates> Ok Hackman238... got any stumpers for mrchippy today?
[01-Sep-2011 11:07:30] <rmatte> tsener: I doubt anyone in here has tried the BSD port... bad experiences with running Zenoss on BSD in the past
[01-Sep-2011 11:08:22] <nyeates> The guys at Experts exchange, i think, have made new BSD-compliant code
[01-Sep-2011 11:09:00] <Hackman238> mrchippy: Any idea how I can get my heartbeats to work more reliably in v2.5.2? Most of my DC's have hb issues all the time even though all the daemons are working away flawlessly
[01-Sep-2011 11:09:07] <nyeates> ahh freeBSD: http://www.experts-exchange.com/blogs/EE-Tech-News/B_5409-Experts-Exchange-Employee-Makes-It-Possible-to-Install-Zenoss-on-a-FreeBSD-Machine.html
[01-Sep-2011 11:09:43] <Hackman238> mrchippy: Also, where can I find technical documentation on how the failover for collectors works in Zenoss 4.0.2?
[01-Sep-2011 11:11:08] <nyeates> see here for freeBSD port info: message/60551
[01-Sep-2011 11:12:00] <mrchippy> on the reliable heartbeats, i can't tell you a good answer other than improvements were made in that area in 3 and especially 4. Sorry if that's an asshat answer 
If you have a more specific situation, I might be able to dig out an answer for you
[01-Sep-2011 11:13:10] <nyeates> Hackman238: one of yer production instances still on 2.5.2? understandable - just asking
[01-Sep-2011 11:13:34] <mrchippy> on the tech docs for collector failover in 4.0.2-- what is this "collector failover" you speak of?
[01-Sep-2011 11:13:43] <Hackman238> nyeates: All of our production is on 2.5.2....we're waiting on Zenoss professional services to fix a few packs in the contracts...stil
[01-Sep-2011 11:13:56] <Hackman238> mrchippy: 1 sec, I'll find the listing
[01-Sep-2011 11:14:03] <mrchippy> cool, thx
[01-Sep-2011 11:14:45] <konvert> Hackman238 thx
[01-Sep-2011 11:14:55] <Hackman238> mrchippy: "Collector redundancy'
[01-Sep-2011 11:15:25] <rmatte> Hackman238: join the club, constant heartbeat issues in 2.5.2
[01-Sep-2011 11:15:31] <Hackman238> konvert: NP
[01-Sep-2011 11:15:33] <rmatte> Hackman238: I just ignore them at this point
[01-Sep-2011 11:16:21] <Hackman238> rmatte: yeah its a pain in the balls. We have huge caches, dedicated hubs, insanely fast mysql setups, cached configs true....can't figure it out
[01-Sep-2011 11:16:52] <Hackman238> rmatte: Whats really a pest is that all the DC's are configured exactly the same....but some have the issue, some dont.
[01-Sep-2011 11:17:12] <Hackman238> rmatte: ...and one in a while they'll go all musical chairs and some stop andothers start up with the problem.
[01-Sep-2011 11:17:20] <Hackman238> s/one/once/
[01-Sep-2011 11:17:41] <Hackman238> The madness! Madness!
[01-Sep-2011 11:17:56] <jmp242> lol
[01-Sep-2011 11:18:17] * Sam-I-Am had heartbeat issues too
[01-Sep-2011 11:18:42] <mrchippy> ahhh, ok that's our HA config that ps uses. We have a doc on it. Let me find it.
[01-Sep-2011 11:19:13] <Hackman238> mrchippy: Thanks. I've got folks after me for an explaination and all I can say at this point is 'no one at Zenoss has ever heard of the feature'
[01-Sep-2011 11:19:29] <nyeates> mrchippy: HA as in like DRBD?
[01-Sep-2011 11:19:30] <rmatte> Hackman238: heartbeat events == the boy who cried wolf
[01-Sep-2011 11:19:55] <Hackman238> rmatte: Aye...and you never know which DC is the hypocondriact and which arent!
[01-Sep-2011 11:20:01] <rmatte> lol
[01-Sep-2011 11:20:07] <mrchippy> yeah, it's not a 'feature' as in something built into the product. it's a set of configuration instructions.
[01-Sep-2011 11:20:35] <Hackman238> s/hypocondriact/hypochondriac/
[01-Sep-2011 11:21:11] <bigegor> hi all
[01-Sep-2011 11:21:20] <Hackman238> mrchippy: Careful with that type of advertising. That means that monitoring the air fuel ratio using zenoss command datasources on a car pc is also a feature
[01-Sep-2011 11:21:21] <nyeates> mrchippy: i can provide link to the KB for "How To Configure Zenoss for High-Availability Using DRBD and Heartbeat"
[01-Sep-2011 11:21:38] <nyeates> Heartbeat in this sense being the linux kind
[01-Sep-2011 11:21:54] <Hackman238> nyeates: TY
[01-Sep-2011 11:22:34] <nyeates> Gimme a few minutes tho...its on the new KB and Its hard to provide links to it
[01-Sep-2011 11:22:49] <Hackman238> nyeates: NP man, no rush
[01-Sep-2011 11:22:56] <Hackman238> nyeates: Even later is good
[01-Sep-2011 11:23:04] <Hackman238> mrchippy: Whats your take on message/60974#60974
[01-Sep-2011 11:23:53] <konvert> Guys i have another question, we are running citrix, is there a zenpack for citrix?
[01-Sep-2011 11:24:37] <bigegor> i'm playing now with Zenoss 3.2 beta. It seems like something was changed in ZenCollector framework (new task creation). My question is Products.ZenCollector.tasks.SubConfigurationTaskSplitter class usable for now or it has WIP status.
[01-Sep-2011 11:27:03] <mrchippy> sorry i'm taking so long. trying to digest the forum entry...
[01-Sep-2011 11:27:29] <Hackman238> mrchippy: NP. No rush.
[01-Sep-2011 11:29:06] <Hackman238> mrchippy: Also, bit of a dev question, is it possible to pass operands like 'AND' and 'OR' to the Zope catalog search?
[01-Sep-2011 11:29:37] <mrchippy> yeah. Check out AdvancedQuery. i'll find the link and post
[01-Sep-2011 11:29:45] <Hackman238> mrchippy: TY
[01-Sep-2011 11:30:00] <mrchippy> http://www.dieter.handshake.de/pyprojects/zope/AdvancedQuery.html
[01-Sep-2011 11:30:31] <Hackman238> mrchippy: Danke.
[01-Sep-2011 11:30:35] <mrchippy> np
[01-Sep-2011 11:31:00] <Hackman238> mrchippy: Who are you, by the way? Simon?
[01-Sep-2011 11:32:03] <mrchippy> my name is Chip Holden. i provide soda and snacks for the wieners in the engineering group
[01-Sep-2011 11:32:21] <mrchippy> i code some too
[01-Sep-2011 11:33:09] <nyeates> Public link to the HA / DRBD stuff btw... https://support.zenoss.com/ics/support/default.asp?deptID=16059&task=knowledge&questionID=8
[01-Sep-2011 11:33:56] <Hackman238> mrchippy: LOL. Gotcha. I like to know who's who so I can inform Rusty of who needs a raise for excellent work.
[01-Sep-2011 11:34:09] <Hackman238> nyeates: Thanks Nick
[01-Sep-2011 11:34:51] <dhopp> if I'm creating a distributed collector in core, rather then trying to install all zenpacks, can I just rsync $ZENHOME/zenpacks from the hub to the distributed collector and then restart zenoss on the collector? (I know I'd have to grab /bin/* and /etc/*.conf as well and modify confs)
[01-Sep-2011 11:35:24] <rmatte> dhopp: yes, you can, though I'm pretty sure that Egor's ZenPack takes care of doing the ZenPack syncing
[01-Sep-2011 11:35:58] <dhopp> rmatte: I know his zenpack does handle it..but his zenpack doesn't handle logical collectors right now (running multiple collectors on the same physical host)
[01-Sep-2011 11:36:08] <rmatte> ah, yeh it doesn't
[01-Sep-2011 11:36:12] <dhopp> so I was just making sure it was that simple
[01-Sep-2011 11:36:16] <rmatte> but yeh, you're correct that you just have to rsync or copy
[01-Sep-2011 11:36:24] <mrchippy> dhopp: trying to think if there's location-specific stuff in the deployed zenpacks.... nothing coming to mind.
[01-Sep-2011 11:36:26] <nyeates> bigegor: is that issue halting your zenpack from working in 3.2?
[01-Sep-2011 11:37:02] <rmatte> dhopp: the only thing that matters is that any scripts that the collector may be trying to execute are there, meaning the ZenPacks don't actually have to be installed (since Zope isn't even running)
[01-Sep-2011 11:37:24] <dhopp> mrchippy: I know some zenpacks that create a new daemon have the daemon live in the zenpack dir and then create a symlink in bin..but I can script around that to handle that scenario
[01-Sep-2011 11:37:31] <dhopp> rmatte: that's what I thought..but wanted to verify
[01-Sep-2011 11:37:48] <rmatte> you're right about daemons though, that's a special case
[01-Sep-2011 11:37:59] <rmatte> and like you said, just need to create the symlink
[01-Sep-2011 11:38:08] <bigegor> i have some strange results (error: "Task servername already exist")
[01-Sep-2011 11:38:32] <Hackman238> bigegor: Strange indeed. Where do you see that?
[01-Sep-2011 11:40:09] <nyeates> bigegor: this is for your ZenPacks.community.DistributedCollectors-1.7.egg zenpack?
[01-Sep-2011 11:40:38] <bigegor> zenperfsql ZenPerfSqlTaskSplitter splitConfiguration method
[01-Sep-2011 11:41:19] <bigegor> nyeates: no, my new SQLDataSource
[01-Sep-2011 11:41:47] <jmp242> Yum, new forum thread that I've perhaps gone down the garden path on, or can't help anymore
[01-Sep-2011 11:41:48] <jmp242> message/60979#60979
[01-Sep-2011 11:42:14] <mrchippy> jmp242: looking
[01-Sep-2011 11:43:29] <jplouis> bigegor: do you have a link to your code?
[01-Sep-2011 11:44:17] <bigegor> jplouis: https://github.com/epuzanov/ZenPacks.community.SQLDataSource
[01-Sep-2011 11:47:04] <bigegor> jplouis: today i've played with Products.ZenCollector.tasks.SubConfigurationTaskSplitter class. It seems like it subconfiguration not fully supported by scheduler (error: "Task servername already exist").
[01-Sep-2011 11:47:12] <rocket> Hackman238: regarding ha there has been an KB article for it for ages on our old portal
[01-Sep-2011 11:48:04] <nyeates> rocket: the link to it on the new KB is above
[01-Sep-2011 11:52:22] <nyeates> welcome back davetoo
[01-Sep-2011 11:52:36] <nexex> Monitoring using JMX, if the java processes on the target is restarted, zenoss will say: DataSource Thread Count; Error calling mbean java.lang:type=Threading: Exception: no MBean registered with name: java.lang:type=Threading
[01-Sep-2011 11:52:49] <nexex> it will say that until I connect with jconsole, then it self clears
[01-Sep-2011 11:52:58] <davetoo> hi
[01-Sep-2011 11:53:01] <davetoo> busy, busy, busy
[01-Sep-2011 11:54:10] <dhopp> another question regarding redundancy. I have a hub server in 1 DC and then thinking of a standby in a 2nd DC. DRBD isn't really feasible in this situation. The remote collectors will have hubhost set to a CNAME and in the event of a complete catastrophe I could change that CNAME to the hubhost in the 2nd DC. I'm going to be rsyncing $ZENHOME from primary hub to secondary hub and will have mysql replication setup. Is this something that *should* work or am
[01-Sep-2011 11:54:25] <jplouis> bigegor: We use SubConfigurationTaskSplitter in a few daemons. Probably some sort of undocumented setup needs to happen
[01-Sep-2011 11:55:46] <hmp> dhopp: why not use DRBD?
[01-Sep-2011 11:56:43] <rocket> dhopp: you have to worry about the Data.fs mostly
[01-Sep-2011 11:56:56] <rmatte> hmmm, so I'm trying to load some Mibs and I get an error: IOError: [Errno 2] No such file or directory: '/tmp/mib_extract/MrSEel/mib-jnx-l2ald.txt'
[01-Sep-2011 11:57:07] <rmatte> but, that file is in the directory that I'm trying to load the Mibs from
[01-Sep-2011 11:57:14] <rocket> dhopp: you will need to use repozo to create a backup regularly and then sync that across
[01-Sep-2011 11:57:26] <rmatte> so I don't get why it doesn't get copied to the tmp directory for installation if that's what's required
[01-Sep-2011 11:57:28] <rocket> rmatte: dir perms?
[01-Sep-2011 11:57:36] <rmatte> doubt it but I'll check
[01-Sep-2011 11:57:39] <dhopp> rocket: repozo?
[01-Sep-2011 11:57:53] <rocket> check the source code for zenbackup
[01-Sep-2011 11:58:08] <rocket> you will see that is the utility being used to backup the data.fs file to a new file
[01-Sep-2011 11:58:10] <dhopp> hmp: too many firewalls and other security that would have to be messed with to make that work
[01-Sep-2011 11:58:15] <rmatte> nope, I was able to copy it find from hand
[01-Sep-2011 11:58:29] <rmatte> fine* by hand
[01-Sep-2011 11:58:41] <rocket> since the Data.fs file is live you cannot just copy it between systems or it will be corrupt
[01-Sep-2011 11:58:44] <Simon4> dhopp: we do pretty much exactly that, and I have a cron script that syncs the Data.fs file in the way rocket describes every 15 minutes
[01-Sep-2011 11:59:02] <Simon4> so you have a cold standby install in the other DC
[01-Sep-2011 11:59:11] <nyeates> repozo is zope tool for incremental backup w/o having to shut off zopectl
[01-Sep-2011 11:59:19] <Hackman238> dhopp: Best bet to keep a backup of Data.fs is to zeopack it then rsyn the Data.fs.old to Data.fs on the other side
[01-Sep-2011 11:59:42] <Hackman238> dhopp: Since it is alfterall standby
[01-Sep-2011 11:59:47] <Hackman238> (after all
[01-Sep-2011 12:00:01] [disconnected at Thu Sep 1 12:00:01 2011]
[01-Sep-2011 12:00:01] [connected at Thu Sep 1 12:00:01 2011]
[01-Sep-2011 12:00:16] [zenoss-logger (logger bot) has joined #zenoss]
[01-Sep-2011 12:00:21] <rocket> thats another way to do it too .. basically the same thing with the added advantage that its packed .. :/
[01-Sep-2011 12:01:02] <Hackman238> dhopp: If you zeopack often it wont take long
[01-Sep-2011 12:01:14] <Hackman238> dhopp: So say you pack every 4 hours, at worst you'll be 4 hours behind
[01-Sep-2011 12:01:25] <Hackman238> dhopp: Plus the pack wont take long after the initial pack
[01-Sep-2011 12:02:52] <rmatte> http://fpaste.org/Ei7S/raw/
[01-Sep-2011 12:03:03] <rmatte> That's what I get when trying to install the Mibs
[01-Sep-2011 12:03:15] <rmatte> not really sure why it's tarring them then extracting them from tar
[01-Sep-2011 12:03:24] <davetoo> mibs, mibs, mibs,
[01-Sep-2011 12:03:27] <Hackman238> rmatte: For fun!
[01-Sep-2011 12:03:28] <davetoo> it's a box all full of mibs,
[01-Sep-2011 12:03:29] <davetoo> ....
[01-Sep-2011 12:03:40] <davetoo> oh
[01-Sep-2011 12:03:42] <rmatte> zenmib can be such a pain in the ass
[01-Sep-2011 12:03:45] <Hackman238> rmatte: Burn those CPU cycles!
[01-Sep-2011 12:03:50] * davetoo reminds self to look at rmatte's cisco mib pack
[01-Sep-2011 12:03:50] <rmatte> lol
[01-Sep-2011 12:04:06] <rmatte> yeh, getting those cisco Mibs loaded was a pain
[01-Sep-2011 12:04:10] <dhopp> Hackman238 / rocket: thanks…I didn't realize zeopack created a .old
[01-Sep-2011 12:04:42] <rmatte> Also, apparently zenmib was rewritten by Kells to be better, but instead it actually loads less of the Cisco Mibs now
[01-Sep-2011 12:04:57] <Hackman238> dhopp: NP
[01-Sep-2011 12:05:06] <davetoo> rmatte do you have a favorite standalone mib browser tool? I used to use TkIned, but... it doesn't work with later versions of tcl/tk
[01-Sep-2011 12:05:12] <Hackman238> rmatte: LOL Did he jsut comment out the areas that were broke?
[01-Sep-2011 12:05:15] <rmatte> though it does fix a couple of problems where I had to modify some of the Mibs by hand to get them to load in
[01-Sep-2011 12:05:21] <Hackman238> rmatte: I'm JK, kells is good.
[01-Sep-2011 12:05:24] <rmatte> nah, doubt it
[01-Sep-2011 12:05:27] <rmatte> lol
[01-Sep-2011 12:06:04] <Hackman238> LOL
[01-Sep-2011 12:06:28] <davetoo> try: something; except: give_up()
[01-Sep-2011 12:06:46] <dhopp> Hackman238: so I could so something like this on the secondary: rsync primary:/opt/zenoss —exclude=Data.fs; cp /opt/zenoss/var/Data.fs.old /opt/zenoss/var/Data.fs
[01-Sep-2011 12:06:53] <rmatte> davetoo: I honestly just browse the Mibs online at the oidview site
[01-Sep-2011 12:06:57] <rmatte> works just as well
[01-Sep-2011 12:07:04] <dhopp> or maybe do that from the primary right after a zeopack
[01-Sep-2011 12:07:24] <rmatte> http://www.oidview.com/
[01-Sep-2011 12:07:38] <davetoo> heh.. I'm trying to get permission to buy a copy of oidview; the automagic mib-downloading has already saved me hours, running the trial version.
[01-Sep-2011 12:07:53] <Hackman238> dhopp: yep
[01-Sep-2011 12:08:00] <rmatte> well, I just use the online version of it
[01-Sep-2011 12:08:06] <davetoo> yeah
[01-Sep-2011 12:08:10] <rmatte> if you google for a MIB name the first result is almost always oidview online
[01-Sep-2011 12:08:14] <Hackman238> dhopp: But you'll want to change your cron for the zeopack to be far more frequent
[01-Sep-2011 12:08:50] <Hackman238> dhopp: That works too
[01-Sep-2011 12:10:19] <rmatte> eugh, juniper sucks balls with Mibs, one of these mib files in completely empty
[01-Sep-2011 12:10:23] <davetoo> I get paid to deal with a different set of systems every week or two; anything that saves me time like this ought to be a no-brainer I could spend a bunch of time fussing around with snmpwalk and manual mib management, but this is so much easier.
[01-Sep-2011 12:10:42] <davetoo> __init__.mib ? :
[01-Sep-2011 12:10:45] <Hackman238> rmatte: s/eugh, juniper sucks balls with Mibs/eugh, juniper sucks balls/
[01-Sep-2011 12:10:58] <rmatte> haha, that's what the problem was, empty Mib file packaged in
[01-Sep-2011 12:11:09] <rmatte> zenmib was choking on it
[01-Sep-2011 12:11:45] <rmatte> mystery solved
[01-Sep-2011 12:12:33] <rmatte> these are the Mibs for JunOS
[01-Sep-2011 12:12:37] <rmatte> hadn't loaded them in yet
[01-Sep-2011 12:12:44] <dhopp> Hackman238: thanks..obviously if I do that on the primary I change the cp command slightly :-)
[01-Sep-2011 12:13:04] <Hackman238> rmatte: I think we should write some how to's and FAQ on MIBs on the ZCA wiki
[01-Sep-2011 12:13:17] <rmatte> Hackman238: well, there's really not much to know
[01-Sep-2011 12:13:33] <rmatte> Extract mibs to directory then: zenmib run -v10 *
[01-Sep-2011 12:13:34] <rmatte> lol
[01-Sep-2011 12:13:45] <Hackman238> rmatte: Just tips and tricks on getting zenmib to work.
[01-Sep-2011 12:13:47] <rmatte> If it fails, troubleshoot as best you can
[01-Sep-2011 12:13:49] <rmatte> then give up
[01-Sep-2011 12:13:54] <Hackman238> rmatte: well tricking zenmib in to working.
[01-Sep-2011 12:14:01] <rmatte> also need to restart Zenoss after the Mibs are installed
[01-Sep-2011 12:14:30] <rmatte> yeh, the problem is the tricks aren't exactly general
[01-Sep-2011 12:14:42] <rmatte> it's more about coming up with ways on the fly
[01-Sep-2011 12:15:04] <rmatte> voila, juniper mibs installed
[01-Sep-2011 12:15:12] <nyeates> Im going to change the topic and officiate the dev meeting over....thx everyone
[01-Sep-2011 12:15:17] <Hackman238> rmatte: True, but not all users can troubleshoot by themselves LOL
[01-Sep-2011 12:15:19] <Hackman238> rmatte: Nice
[01-Sep-2011 12:15:27] <davetoo> heh.. just realized that
[01-Sep-2011 12:15:28] <Hackman238> nyeates: Thanks Nick!
[01-Sep-2011 12:15:35] <davetoo> was supposed to be in progresse
[01-Sep-2011 12:16:11] <rmatte> erm...
[01-Sep-2011 12:16:18] <rmatte> true, but you can't really document crafty troubleshooting
[01-Sep-2011 12:16:26] <rmatte> you can document common issues, but that's about it
[01-Sep-2011 12:16:36] <Hackman238> rmatte: Perhaps a webmd for zenoss! 
[01-Sep-2011 12:16:49] <davetoo> it's .. surprisingly nondeterministic
[01-Sep-2011 12:16:50] <rmatte>
[01-Sep-2011 12:17:03] <Hackman238> LOL
[01-Sep-2011 12:17:03] <rmatte> "The doctor is in"
[01-Sep-2011 12:17:04] <davetoo> if it's not to be super-document-heavy.
[01-Sep-2011 12:17:39] * davetoo used to work for a NASA subcontractor on the space shuttle program (never got to see a launch). I had to write down every single step that I took that could be described.
[01-Sep-2011 12:18:17] <nyeates> "next, i removed my bottom from my chair, and got coffee"
[01-Sep-2011 12:18:26] <rmatte> Here's our zenmib troubleshooting documentation: http://xkcd.com/627/
[01-Sep-2011 12:18:53] <rmatte> would need some slight modification
[01-Sep-2011 12:18:56] <Hackman238> LOL
[01-Sep-2011 12:19:05] <davetoo> seriously: google is a great tool.
[01-Sep-2011 12:19:08] <Hackman238> rmatte: Very slight modification
[01-Sep-2011 12:19:15] <Hackman238>
[01-Sep-2011 12:19:16] <rmatte> lol
[01-Sep-2011 12:19:21] <davetoo> except when all it finds is me asking the same question a year before, with no answer.
[01-Sep-2011 12:19:41] <rmatte> s/click/enter command and return/g
[01-Sep-2011 12:19:47] <rmatte> done
[01-Sep-2011 12:19:47] <nyeates> that xkcd is sooo true
[01-Sep-2011 12:19:57] <rmatte> nyeates: yeh, it really is
[01-Sep-2011 12:19:59] <davetoo> they forgot -v10
[01-Sep-2011 12:20:11] <rmatte> that too
[01-Sep-2011 12:20:16] <Hackman238> davetoo: true, need v10
[01-Sep-2011 12:20:25] <Hackman238> davetoo: if not only to look busy
[01-Sep-2011 12:20:28] <Hackman238> LOL
[01-Sep-2011 12:26:06] <rmatte> the best comics on the net though are here: http://theoatmeal.com/
[01-Sep-2011 12:27:10] <rmatte> For instance: http://theoatmeal.com/comics/apple
[01-Sep-2011 12:29:00] <davetoo> http://art.penny-arcade.com/photos/215181106_tvEcr-XL-2.jpg
[01-Sep-2011 12:29:02] <davetoo> that is me.
[01-Sep-2011 12:29:06] <davetoo> every day.
[01-Sep-2011 12:29:31] <rmatte> haha
[01-Sep-2011 12:29:37] <davetoo> Mostly the coffe part,
[01-Sep-2011 12:29:58] <davetoo> but I also talk like a sysadmin-sailor :P)
[01-Sep-2011 12:30:07] <rmatte> http://theoatmeal.com/comics/customer_service <--- "Why I'd rather be punched in the balls than call customer service."
[01-Sep-2011 12:30:53] <davetoo> heh...
[01-Sep-2011 12:31:18] <rmatte> "Groan loudly as if in mid-orgasm every time the support operator starts to speak. It will make them uncomfortable - this is how the battle is won."
[01-Sep-2011 12:32:24] <davetoo> Skippy McSkipperSkip
[01-Sep-2011 12:32:34] <davetoo> ok, back to serious work....
[01-Sep-2011 12:32:41] <rmatte> lol
[01-Sep-2011 12:34:38] <rmatte> "Hello sir! Can I get your first name, phone number, address, date of birth, favorite planet in the solar system, and least favorite african mammal. I won't actually log this information, mind you, so you'll have to repeat it to every other operator I forward you to."
[01-Sep-2011 12:37:53] <Hackman238> rmatte: Oh my. The apple one is hilarious!
[01-Sep-2011 12:38:07] <rmatte> This one is amazing too: http://theoatmeal.com/comics/online_gaming
[01-Sep-2011 12:39:59] <Hackman238> rmatte: "freaky weird sey illegal in 7 states" LOL
[01-Sep-2011 12:40:07] <rmatte> lol
[01-Sep-2011 12:40:08] <Hackman238> s/sey/sex/
[01-Sep-2011 12:40:11] <Hackman238> LOL
[01-Sep-2011 12:40:22] <rmatte> I like the guys running at them "but Mommmmmm...."
[01-Sep-2011 12:40:23] <rmatte> lol
[01-Sep-2011 12:40:30] <Hackman238> lol
[01-Sep-2011 12:40:30] <rmatte> It's so true
[01-Sep-2011 12:41:53] <rmatte> http://theoatmeal.com/comics/airplane_layout
[01-Sep-2011 12:42:11] <rmatte> That is the best layout for a plane ever, I especially like the screaming infant storage kennel
[01-Sep-2011 12:47:46] <Hackman238> LOL
[01-Sep-2011 13:01:55] <Sam-I-Am> hehe, theoatmeal is rock
[01-Sep-2011 13:04:42] <Hackman238> Sam-I-Am: Indeed
