I was faced with the decision to implement an additional system on the RHEL 4.x series, or make our first jump to the version 5 releases. I decided to have this additional system to stay on RHEL 4.x because of our support situation. As admins are aware, there are many factors that affect a decision like this one.

RHEL 4.x vs. RHEL 5
RHEL is a stable platform among its competition. At just over a year old, its latest build, RHEL 5, is still new to the scene. But RHEL version5.1 was recently released and has enjoyed initial success thus far. The biggest factor in choosing to remain on the 4.x platform was Red Hat’s recent release of version 4.6, keeping the 4.x a current product. With this release, all of our versions remain within the realm of support, so our internal support requirements have not been impacted by another platform. This also keeps us in line with base configuration of applications that are running on the RHEL systems.

You can’t stay on version 4.x forever!
I know, but we made the decision based on what we can best support internally by not multiplying our scope of platforms. But, the version 5.x test bed is just around the corner, and we will increase our comfort with version 5.x (curiously awaiting a 5.2). At that point we would welcome version 5.x by ceasing the version 4.x installs, and migrating to version 5.x if possible.

What is your strategy?
Do you have multiple versions running in your enterprise? What is your thought process in regards to introducing a new distribution? Share your strategies below in a comment.

It’s a day of blog subject line questions, apparently.

The subject line question is asked today because there’s apparently some concern over Gutsy Gibbon’s documentation. Gibbon is the latest release of Ubuntu, version 7.10, which came out last month to the usual fanfare associated with new Ubuntu releases.

That said, Carla Schroder, writing for Enterprise Networking Planet, has a bone to pick with Canonical and the Ubuntu development team over documentation:

Whatever anyone may think of Ubuntu, you can’t deny they’re busy little critters, stuffing all manner of new things into every release. Which is a splendid thing, and what would make it even better is if they documented all of these wonderful new things. And also the old things. I think it’s the worst of the major Linux distributions for documentation. I spent a considerable amount of time trying to find out what makes Ubuntu’s server kernel different from a desktop kernel, what exactly is the OEM installation, where is the online package search page, what’s new in this release, and what’s included in this release. www.ubuntu.com is poorly-organized and seems more marketing-oriented than informative.

In our overview of Linux support, then and now, from this earlier summer, we identified similar documentation concerns with Linux in general. To see that one of the signature distros (and by that I mean one of the most wildly popular ones outside of Red Hat and Novell) is suffering from poor documentation is troubling to say the least.

In 2003, systems administrator Pati Moss said some online Linux/OSS instructions are very “high level and difficult to decipher.” IT manager Rick Segeberg agreed: “Newbies to Linux (especially non-programmers) find it difficult to follow the very technical documentation and how-tos that are available,” he said. “Most of the technical documentation is done by technical people for technical people.”

So documentation can be too technical and difficult to decipher. Got it. In our overview this was still a “thorn in the side” of IT managers, but at least Red Hat and Novell had stepped up to offer paid, professional support. But with Ubuntu, however, things are apparently too sparse! Not a good sign when Mark Shuttleworth and company at Canonical are trying to get Ubuntu pre-installed on commodity hardware from vendors like Dell.

Schroder again (under the wonderful subject line, “Dammit, Jim, I need documentation!”):

The Ubuntu release notes are quite sparse, and they lump the server and desktop editions together. There are bug reports and workarounds, but where is the list of major and new features? AppArmor is a radically new inclusion, but the only mention of it is that it breaks printing. What is it, and what do you do with it? What are the kernel versions, and versions of major applications like Apache and … well, what exactly comes with this release? What hardware architectures are supported, and what are some of the specific issues for them? And so forth—just cruise the Debian and Fedora release notes to see how it should be done. In fact you can check out older Ubuntu release notes—the farther back you go, the more complete they are, though they’re still short of what they should be.

The problem here, it seems, is that the notes assume a lot on the part of the reader. Do I know what AppArmor is? Sure I do, but I write about OS’s like Novell SUSE Linux and I know that AppArmor has been included in that distro for quite some time.

A huge part of the Linux ecosystem is documentation. Ubuntu Server — at least for one columnist — comes up short: “It’s also typical to include batches of READMEs and CHANGES and other helpful documentation on installation CDs. Don’t bother looking on the Ubuntu Server CD for these, because there aren’t any. However, it does include the “Ubuntu Installation Guide”, which is actually the Debian Installation Guide with some minor modifications, such as changing “Debian” to “Ubuntu”, and adding useful links to online Ubuntu resources,” Schroder said.

Apparently the Ubuntu documentation is Debian stuff dressed up with Ubuntu branding. Sounds kind of like what CentOS does with Red Hat Enterprise Linux (in reverse, of course), but in this case it’s a bad thing. Enterprise users want enterprise level documentation with their Linux OS. If Canonical is truly serious about getting Ubuntu pre-installed on commodity hardware, they’d best be tweaking their docs … do you agree?

When we first wrote about the Oracle Unbreakable Linux support program late last year, Red Hat Enterprise Linux users appeared ready to at least give it a look. The program, officially announced in October 2006, promised deep discounts for Red Hat users that opted to go to Oracle — not Red Hat — for their Linux support. Sometimes these discounts were as much as 50% below what Red Hat was asking.

In our article, “Red Hat users pine for discounted support”, a Pacific Crest Securities survey of 188 enterprise operating system buyers — 86 of whom were Red Hat Inc. support customers — revealed that one-third of respondents said Red Hat would need an Oracle-like discount of 50% to 74% to keep their business.

A story from down under today leads me to believe that Pacific Crest might want to revisit that survey for another go.

Max McLaren, Red Hat Australia managing director, said Red Hat has not been losing business to Oracle for support, with one exception: Melbourne firm Opes Prime Stockbroking said it would switch allegiance to Oracle last year.

McLaren told Australian IT that “out of the top five banks in Australia, four are our customers … (but) we don’t have much penetration in the insurance sector. That’s an area to work on.” He added that the Pacific Rim is another area of huge expansion for Red Hat, which we kind of touched upon in previous posts on the Enterprise Linux Log (or TELL for short).

So that’s Red Hat’s take abroad anyway. I’m sure McLaren made sure his comments went through the spin cycle a few times before he said them, but it’s interesting to see how little of an effect Oracle *might* be having on Red Hat’s business. It’s about one year in now, so we’ll see if Unbreakable is maturing or manuring, if you get my meaning.

Occasionally throughout the summer I’ve been chatting and emailing with Saugatuck Technology analyst Charlie Burns about mainframes, IBM and Linux. Many people have argued over the past year that the mainframe is dying out (again), but Burns and some very telling market trends go against that grain with a 180 degree turn: the mainframe is surging, and it’s all thanks to Linux.

I’ll have an article up a bit later this week (or early next) detailing just exactly what is going on in this space, but for now I thought I’d include one of the recent emails Charlie sent me that covers some of the basic cost advantages of the mainframe.

Architecturally-based advantages in the hardware, the operating systems, and in the virtualization functionality enable mainframes to manage multiple diverse workloads based on business objectives and deliver exceptional cost reductions. If we compare the costs of using mainframes to those of conventional servers as noted earlier, we find the following:

• Technical support and maintenance costs. By consolidating and centralizing the capabilities of dozens of servers into a single platform, use of a mainframe drastically reduces the redundancies and differences that are de rigueur in server farm environments. If we accept conventional industry wisdom that states a minimum of 70 percent of IT costs are labor - and that the majority of labor costs are training and support - it’s easy to see how mainframes can quickly free up IT budgets for more strategic investment such as new application development.
• Software licensing and maintenance costs. Since most operating, middleware, and application software is licensed to each server it is used on, a mainframe offers substantial software savings. In a mainframe, the computing capacity applied to software can scale dramatically. Literally, hundreds of virtualized server images can operate in a single mainframe under a single license, thus, avoiding additional license and maintenance fees. In addition, the IBM System z has the capability of running specialized processors for Linux and for some application workloads. These processors are priced substantially lower than the base processors. Thus, the System z delivers both hardware and software saving on a broad scale when compared to individual x86 server platforms.
• User and IT training costs. Training costs tend to be driven by the number and complexities of multiple applications and operating systems. By enabling the use of all leading operating systems and applications within single platform, mainframes drastically reduce the need for training.
• Utility and environmental costs. Mainframes require substantially smaller amounts of power, UPS capacity, cooling, and floor space when compared to the environmental requirements of an x86 server farm with equivalent processing capacity. The mainframe’s advantage is even more substantial when one considers the reduced amount of storage and inter-connection equipment compared to an x86 server farm.
• Security costs. Mainframes enable centralization of software and application interfaces. Centralization of software enables vastly improved security management by reducing the number and types of access points. Additionally, because of its heritage, security is architected into the mainframe and is uniquely robust. For example the IBM System z family of mainframes provides security against information flow between virtual machines. The System z was first certified in mid-2003 as Evaluation Assurance Level 5 (EAL 5) by meeting the Common Criteria standard ISO 15408. Comparatively, virtualization on x86 server platforms require security to be added and layered as part of the operating system, applications, databases, and so on – further increasing both the complexity and cost of security, while adding more points of vulnerability due to incompatibilities between security systems and other software.

An Elementary Roadmap
Saugatuck recommends that every company with more that 20 x86 servers should perform a thorough evaluation of existing workloads and servers with the following steps in mind:

1. x86 servers yielding the largest savings should be migrated to the mainframe first (e.g., those with unique infrastructure support requirements)
2. x86 servers with the lowest utilization should be migrated early
3. Assets with an upcoming compelling event (e.g., need for capacity upgrade, lease expiration, etc.) should be migrated before incurring the expense
4. x86 servers/workloads should be aggregated by user department to leverage strong buy-in
5. Oldest technology x86 servers should be migrated early
6. Focus on real estate by freeing up contiguous raised floor space or eliminating sites as early as possible

Consider this the first in an occasional, meandering series of articles on Linux done right. These aren’t meant to boost the sales of any particular vendor, but instead are meant to show other end users, IT managers and decision makers what to look for when vetting applications and operating system migrations. It can be support, migrations strategies, execution or anything and everything in between. If it’s Linux done right, then you’ll find it here.

I initially spoke with John Flores, a system administrator with the University of Texas at San Antonio, earlier this year for a broad SearchEnterpriseLinux.com article on Linux support. The article focused on the good, the bad and the ugly of working with commercial Linux distributors, as well as with the alternatives like CentOS and Debian. It was also a comparison of the past, present and future of Linux support as a whole.

Flores and his data center — like many data centers today — were at a crossroads. He was using Windows NT as his domain controller, but it was update time as a few Dell servers were past their prime and new ones were set to be introduced in the summer of 2006.

“We had an old Dell 6300 that was to be put out of service … it was what was running the NT 4.0,” Flores told me. “Rather than move NT 4.0 to a new server, we were looking for an OS that could put onto a new server and it was going to be either Linux or MS.”

But old servers weren’t the only issue at the U of T that summer. Flores explained that NT 4.0 had become “unstable, mostly due to age.” The software configurations were also old and difficult to maintain, he said. and a lot of “junk” had accumulated over the years. The clutter was quickly becoming a maintenance issue for the IT staff, he said.”We were having a server failure almost once every two weeks. A server would have a major problem so we’d have to reboot it and bring it back up again,” Flores said. But then things got even worse.

“Because this is a university environment, we have a whole new set of something like 5,000 users changing over every semester. We have to log all those IDs and passwords every semester.” Read more »

The webcomic xkcd takes on Black Hat Linux support in its latest comic (”Rule 34″).

You can check out the full comic in the link above, and more xkcd comics here.

xkcd’s other Linux-related fare is pretty good too.

[Hat tip Alex for the link!]

As SearchEnterpriseLinux.com gears up to cover the heck out of the state of Linux support today (a summer support series — how’s that for some alliteration, huh?), I’ve been exchanging emails and phone calls with Linux consultant Patrick Green, of Chicago-based Silver Strand Solutions. Here’s a look at what Green (also an Enterprise Linux Log off-and-on contributor) had to say about his line of work.

The Enterprise Linux Log: What sorts of services do consultants offer over and above a support contract?

Patrick Green: I try to be hands off with the actual heavy lifting. I prefer to educate, inform, and empower the staff of a company to get up to speed. Most of what I have done is provide digestible training materials. If I am not the one executing it, I develop a comprehensive teachers guide and student guide with a .ppt presentation. This will range from a simple desktop tour to authenticating to a Windows environment and even how to position Linux in your sales model.

The other thing I do is look at the macro environment the client exists in. Consultants and sales people do not always give the proper thought to the experience of the guy in the mail room or the receptionist at the front desk. There may be an impact on those people as well and you have to ease the migration burden on them. I compare the migration path to a roadmap for a reason. You have where you are and where you want to be. Some parts of the migration will be simple and others are hard. I like to plan migration in such away that after a tough hurdle, there is a “reward period” where the easy stuff is handled. After said breather, another pain point. These pain points may not be sexy and they may not sell a transition to a platform, but if we are honest, we will know all large IT transitions have these.

TELL: How much do they typically charge?

Green: In my case, I prefer to charge by the project. I have an hourly rate in my mind. After speaking with the potential client, I do what any consultant does. I factor in how long the project will take to develop and execute, add in expenses, justify everything in detail, and set a proposal (ready to negotiate). Typically, my projects average around $4,000-$5,500 a project.

TELL: Are they available as a supplement to support contracts, or to implement larger projects?

Green: Some are. In my case, I prefer not to supplement contracts, but will be happy to implement a larger project. When one goes to conventions, one discovered that convention floors are like a small town that travels. I prefer to build relationships with vendors and support specialists who can handle the long term needs of a client. It is not “passing the buck”, it is giving the client the best service and the best advice for the long haul. There have actually been times where I have turned down business and told a client to contact File Engine, Turtol, or some other vendor knowing that what they want and need can be best handled by those people. I then give the vendor a heads up call and let them know the customers needs in brief so they can give the client the best possible solution.

In the case of a larger implementation, my methods are similar. I will gladly draw up a roadmap, dot the i’s and cross the t’s, and help them find the best distro and service that they need for their project.

If I were to use a really bad analogy, it would be this. Picture Red Hat or Novell as Ford or GM looking to provide a company with fleet vehicles. They provide the cars with the options and the warranties and the service contracts. I am a cross between a driver’s ed instructor and a purchasing consultant. Someone has to help them learn to drive and know how many vans they need in their fleet to do what they need to do, where to store them, how large a service contract, how often to change the oil (and are you better served having the oil changed at the dealership or the Jiffy Lube down the street or hire a guy), wash em, etc.

TELL: Look for more on Linux support best practices at SearchEnterpriseLinux.com coming soon.

Earlier this morning I finished up a podcast with Patrick Green, a Linux consultant and migration expert. Green, a native of Chicago, saw his fair share of ups and downs as a Linux consultant and entrepreneur in the late 1990’s and early 00’s, so I thought it would be best to talk with him about the strengths, successes, challenges and even failures of Linux support today.

The query isn’t out of the blue. To close out 2006, SearchEnterpriseLinux.com conducted a broad survey of its readership about a host of present day Linux and open source software issues. Support, surprisingly or unsurprisingly depending on where your experience with the matter lies, made the list as a “barrier to adoption.”

This is not to say that the support structure for Linux and other open source software is viewed as inadequate. Just three years ago SearchEnterpriseLinux.com reported that only the most diligent of IT managers were considering Linux or open source in their data centers. At the time, Linux deployments in mission critical areas just didn’t happen as they do today because they lacked the standard commercial support that businesses were accustomed to receiving from IBM, Microsoft and others. This perception, some of it earned and some not, was one of the main reasons SUSE Linux couldn’t compete and its founders chose to sell to Novell.

Since then, however, Red Hat dropped support of older Red Hat Enterprise Linux (RHEL) versions to streamline the cost of support. Ubuntu, based on Debian, had commercial support in two years–an accomplishment Ubuntu corporate sponsor Canonical Ltd. said indicates the community got the message that support is of extreme importance. HP made a bundle supporting Debian in Europe. The list goes on.

But what also goes on is this residual effect wherein IT managers not accustomed to Linux continue to view it as less supported and more difficult to administer than Windows, for example. Fair or unfair, it’s out there, and as of the end of 2006 IT managers — and these are somewhat pro-Linux guys who read SearchEnterpriseLinux.com, mind you — are still listing support as a barrier. This is even in light of the cost savings and success associated with Linux and open source software. IDC, as you may know, says open source will go gangbusters in 2011,with revenues through the roof.

So some of this is a skewed perception, the result of Old Guard admins who were raised in the late 1980’s and early 90’s on a diet of Unix and Windows, and who are not as familiar with Linux as they could be. These guys are aware of the forums and mailing lists and the fact that Red Hat sells a very robust (albeit premium-priced) support structure, sure, but are they *really* aware?

Green, in our discussion last week and subsequent podcast this morning, doesn’t seem to think so. He thinks some due diligence and re-education is in order; some hard work on the part of IT staffs to demonstrate the savings of open source AND the richness of the support offered by the major distros and leading commercial open source companies to their higher ups and CIOs.

But that’s not all there is to it. There’s also a documentation problem. Documentation exists but it’s not uniform across the board. Green suggests commercial vendors put some serious effort into making their documentation hum. And this is all while they re-educate users on the importance of community forums, chat channels, and mailing lists. In other words, the support outlets that some open source advocates take for granted.

There’s so much more to be written about this topic. It’s as broad as it is deep, and it’s definitely on the radar of more than a few IT pros out there in the trenches of today’s data center. Information and education are key, and it’s something we here at the Log and SearchEnterpriseLinux.com take seriously enough to dedicate a substantial amount of time and effort into addressing.

All that said… What’s your criteria for evaluating an open source company? Linux vendors? Linux communities (Ubuntu, Debian, etc)? How far have things come, in your opinion, from even just 4-5 years ago? Are users today still doing a lot of their own support? As for the criteria for evaluating support question, here are some examples: time of response, quality of first contact support person, how quickly your question moves up the help ladder, exclusions from contract or things not covered, flexibility, cost, etc. Choose your own adventure, but with support.

It’s been a long time coming (perhaps too long, eh Jan?), but SearchEnterpriseLinux.com will soon be running a regularly updated series on the broad topic of Linux support in the data center. Having a user-generated list of support criteria and some testimonials in place will not only help you get your job done better (and save a buck or two), it will also help your fellow open source community leaders do the same. You guys swap code all the time. How about we try the same with support war stories?