The Challenge

You may not be familiar with the name Geeknet, but you’ve definitely heard of its online properties, Slashdot, ThinkGeek, ohloh and Freshmeat, which are visited by more than 40 million technology professionals each month. As one of the top global online media companies, Geeknet provides the technology infrastructure and content for more than five web sites offering all things geeks crave.

Uriah Welcome, senior director of operations, and Zac Sprackett, senior systems administrator, lead Geeknet’s network operations team. “We are responsible for everything from the IP and OS level down, and our infrastructure is all over the map in terms of services and scale.

When Geeknet opened a new datacenter in Chicago two years ago, the network operations team wanted to centralize management of more than 400 systems, serving the Geeknet web network. Geeknet’s servers are 100% open source running CentOS Linux 5, and their systems span a variety of open source web servers, mail servers, databases (My SQL, Postgress, memchache) and large number of source code repositories running subversion.

Prior to the data center consolidation, the team was using Nagios to run system checks supplemented by a homegrown monitoring system. As they built their next generation data center, they wanted to implement a central database and repository for all system configurations. They also wanted the ability to get systems up and running faster, and the solution needed to be flexible and support the many flavors of open source systems found in the Geeknet infrastructure.

The Solution

Geeknet first found the Zenoss open source systems management solution on the company’s own site, SourceForge, the largest collection of open source tools and applications on the net where Zenoss was consistently highly ranked and recommended. The team initially installed Zenoss to auto-generate configuration files for open source tools Nagios (simple monitoring) and Cacti (graphing engine). In addition to Zenoss, they also considered Groundwork Open Source for the job, but it didn’t provide much functionality beyond what they already had in Nagios. In addition, they had already invested substantial time in a Nagios installation with custom scripts and service checks that they chose not to replace. Open standards and integration were a key to saving time and augmenting the solution to meet their complex needs.

Geeknet now uses Zenoss to created a centralized configuration database that has a complete picture or model of their entire infrastructure. The ability to model all aspects of the network and keep a canonical database of network features was imperative for the Geeknet operations team. The Geeknet team also relies on key automated features of Zenoss, specifically auto-discovery and inventory. Zenoss servers as the master database for all infrastructure and provides the data to auto-generate configurations for other monitoring software and DNS servers. As a result, the team has a very limited amount of manual steps involved in setting up a machine, getting it deployed on teh network, and up-and-running for customers. According to Sprackett, they chose Zenoss for its easy integration points, especially “how easy it is to plug-in third party scripts that you typically have to write as an admin.”

The Results

Zenoss acts as the nervous system for network management, discovering and modeling all servers a they come online, and then automatically generating a BIND configuration to get the DNS deployed on the network. Once the server is up-and-running, Zenoss automatically instructs Nagios to run checks on the server and report back.

The ability to automate DNS entries and set up all the monitors in Zenoss has resulted in huge time savings for the Geeknet network operations team. According to Sprackett, “We couldn’t live without it; we would spend all of our time getting machiens up and running.” After the initial investment made in writing scripts, “we can literally spin up and deploy a machines in 25 minutes versus hours of uploading configuration entries.”

The few times they have had problems, for example when they mistakenly updated Zenoss with the wrong parameters, the team was quick to find support in the Zenoss IRC (Internet Relay Chat) Channel from other community members.

For Geeknet, the ability to automate the configuration process and centralize network management has streamlined the team’s workflows, increased efficiency and the ability to scale. When asked for one piece of advice to share with industry peers based on their experience with Zenoss and netowrk operations, Uriah Welcom said, “simple is fast.”

About Geeknet

Geeknet is the online network for the global geek community. Geekent sites include: SourceForge, Shashdot, ThinkGeek, Ohloh and Freshmeat. Geeknet serves an audience of more than 40 million users each month, and provides the tech-obsessed with content, culture, connections, commerce, and all the things that geeks crave.