First, we must identify what exactly is meant by “the cloud.” There are two major aspects: hosted applications and infrastructure. Both are a form of outsourcing, but the implications are vastly different. We’ll spend a bit of time explaining the differences before exploring the two sides of the cloud debate.

Applications hosted and run by another organization are said to be cloud-based services. Your bank probably provides a portal for managing various accounts. Your payroll company or check printing house provides a mechanism by which you can access their application, via a Web site. Even the most anti-outsourcing businesses use at least a few applications, via the Web, to communicate with partners and service providers.

Taking that one step further, today’s Software as a Service (SaaS) models represent the true spirit of cloud-based applications. Instead of buying a CRM product, for example, and installing it on your servers, you can now simply use the vendor-provided Web access and get all the functionality without the hassle. Moving to hosted applications saves IT staff time, capital budgets, and hassle. It makes a great deal of sense to let the people who wrote the software host it for you, since they are the experts.

Many cloud-only service providers have cropped up lately, to fulfill a market demand for hosted applications. Salesforce is the premiere example, and they have certainly proven that there is space in the market for SaaS companies to re-implement traditional software as hosted solutions.

Infrastructure as a Service is computing outsourcing. Amazon EC2 provides the infrastructure where you can run operating system instances just as you would do on physical servers in a data center. Except you don’t buy servers, you rent time on the cloud. The major selling point is that you can scale up computing resources very quickly with a cloud infrastructure, as opposed to purchasing and installing new servers yourself, but that point is also heavily debated.

The infrastructure component is what most IT people are referring to when they mention clouds. They are struggling with the options: use EC2, implement a compatible private cloud themselves, or skip it altogether.

The cloud infrastructure is essentially a better way to manage virtualization. It is also, some would say, “the right way to run servers.”

Everything Is Moving

The promise of mass deployability of new operating system instances has thankfully forced the IT world as a whole to really embrace automation and configuration management. Long-time sysadmins often scoff at the marketing buzz around the cloud, because they have accomplished automated deployment and on-going configuration management of running systems a long time ago. The cloud concept is simply another smart layer, one that balances resources and possibly auto-migrates virtual machines when resource usage changes.

This is a good thing! It means that everyone agrees we’re headed in the right direction. The right way to manage systems is with automated virtual machines, and the cloud concept is brilliant.

The “everything is moving to a public cloud” camp believes it makes no sense to run your own servers. Data centers, cooling, power — it all is a waste of time and money, unless of course you’re in the hosting business. Large-scale providers can do it all much more efficiently.

It’ll Never Work

Stability is a problem. If Amazon has taught us anything, it’s that they aren’t very good at this cloud thing, or redundancy. Their storage infrastructure, S3, has had many outages and consistently under performs. Why would you move your important business computing needs to a single provider? Nobody is immune to outages, regardless of how redundant they are.

Security is another major concern. If a critical piece of S3 infrastructure becomes compromised, all businesses using S3 will have a major data breach. The servers running EC2 images, too, are a target for criminal hackers. It makes more sense to hide critical servers behind a corporate firewall that can be audited, rather than rely on the security of each individual EC2 instance (and Amazon’s infrastructure).

Final Thoughts

At least, that’s the argument. Personally, I don’t buy the security argument, but stability, reliability and performance are highly suspect.

Another major point of contention is public versus private clouds. If you choose to run your own cloud infrastructure, you need to purchase more computing power than you necessarily need. To scale on-demand, capacity needs to be available. I think you will find that most businesses are currently in a half-virtualized state. They aren’t running cloud-like infrastructure, but a large portion of their servers run as virtual machines. The utilization of those servers is higher (in a good sense) than it has ever been, but there are still a lot of applications running on bare metal.

As has already been happening, businesses will continue to move away from service- or server-based mentalities, toward application-specific uses. Applications can easily be segregated onto individual virtual machine instances, which provides security and manageability benefits (assuming proper automation exists). Those virtual machines can be run the “traditional way,” or they can be part of a cloud infrastructure that eases management even further.

For many larger businesses, regardless their position in the cloud debate, it makes the most sense to implement a private cloud infrastructure that is compatible with EC2. If some instances need to be migrated to a public cloud in the future, and the security department OK’s it, you’re ready to do so.

The post Sorting Out the Debate Over Cloud Computing appeared first on Enterprise Networking Planet.

Now look at what we’ve done. Knowledge is so decentralized we must invent new roles to act as liaisons between all the IT groups. Architects now hold much of the high-level “how it works” knowledge, but without knowing how any one piece actually does work. In organizations with more than a few hundred IT staff and developers, it becomes nearly impossible for one person to do and know everything. This movement toward specializing in individual areas seems almost natural. That, however, does not provide a free ticket for people to turn a blind eye.

Specialization

You know the story: Company installs new application, nobody understands it yet, so an expert is hired. Often, the person with a certification in using the new application only really knows how to run that application. Perhaps they aren’t interested in learning anything else, because their skill is in high demand right now. And besides, everything else in the infrastructure is run by people who specialize in those elements. Everything is taken care of.

Except, how do these teams communicate when changes need to take place? Are the storage administrators teaching the Windows administrators about storage multi-pathing; or worse logging in and setting it up because it’s faster for the storage gurus to do it themselves? A fundamental level of knowledge is often lacking, which makes it very difficult for teams to brainstorm about new ways evolve IT services. The business environment has made it OK for IT staffers to specialize and only learn one thing.

If you hire someone certified in the application, operating system, or network vendor you use, that is precisely what you get. Certifications may be a nice filter to quickly identify who has direct knowledge in the area you’re hiring for, but often they indicate specialization or compensation for lack of experience.

Resource Competition

Does your IT department function as a unit? Even 20-person IT shops have turf wars, so the answer is very likely, “no.” As teams are split into more and more distinct operating units, grouping occurs. One IT budget gets split between all these groups. Often each group will have a manager who pitches his needs to upper management in hopes they will realize how important the team is.

The “us vs. them” mentality manifests itself at all levels, and it’s reinforced by management having to define each team’s worth in the form of a budget. One strategy is to illustrate a doomsday scenario. If you paint a bleak enough picture, you may get more funding. Only if you are careful enough to illustrate the failings are due to lack of capital resources, not management or people. A manager of another group may explain that they are not receiving the correct level of service, so they need to duplicate the efforts of another group and just implement something themselves. On and on, the arguments continue.

Most often, I’ve seen competition between server groups result in horribly inefficient uses of hardware. For example, what happens in your organization when one team needs more server hardware? Assume that another team has five unused servers sitting in a blade chassis. Does the answer change? No, it does not. Even in test environments, sharing doesn’t often happen between IT groups.

With virtualization, some aspects of resource competition get better and some remain the same. When first implemented, most groups will be running their own type of virtualization for their platform. The next step, I’ve most often seen, is for test servers to get virtualized. If a new group is formed to manage the virtualization infrastructure, virtual machines can be allocated to various application and server teams from a central pool and everyone is now sharing. Or, they begin sharing and then demand their own physical hardware to be isolated from others’ resource hungry utilization. This is nonetheless a step in the right direction. Auto migration and guaranteed resource policies can go a long way toward making shared infrastructure, even between competing groups, a viable option.

Blamestorming

The most damaging side effect of splitting into too many distinct IT groups is the reinforcement of an “us versus them” mentality. Aside from the notion that specialization creates a lack of knowledge, blamestorming is what this article is really about. When a project is delayed, it is all too easy to blame another group. The SAN people didn’t allocate storage on time, so another team was delayed. That is the timeline of the project, so all work halted until that hiccup was restored. Having someone else to blame when things get delayed makes it all too easy to simply stop working for a while.

More related to the initial points at the beginning of this article, perhaps, is the blamestorm that happens after a system outage.

Say an ERP system becomes unresponsive a few times throughout the day. The application team says it’s just slowing down, and they don’t know why. The network team says everything is fine. The server team says the application is “blocking on IO,” which means it’s a SAN issue. The SAN team say there is nothing wrong, and other applications on the same devices are fine. You’ve ran through nearly every team, but without an answer still. The SAN people don’t have access to the application servers to help diagnose the problem. The server team doesn’t even know how the application runs.

See the problem? Specialized teams are distinct and by nature adversarial. Specialized staffers often relegate themselves into a niche knowing that as long as they continue working at large enough companies, “someone else” will take care of all the other pieces.

I unfortunately don’t have an answer to this problem. Maybe rotating employees between departments will help. They gain knowledge and also get to know other people, which should lessen the propensity to view them as outsiders.

The post Business Has Killed IT With Overspecialization appeared first on Enterprise Networking Planet.

RiverMuse started in 2008 as a combination of some of the the remaining innovators from Riversoft and Micromuse. Micromuse acquired the struggling Riversoft in 2002, and IBM acquired Micromuse for considerably more money in 2005. Prior, Riversoft had essentially become HP OpenView advanced edition, and with the IBM acquisition Micromuse became IBM Tivoli Netcool. Needless to say, this company has pedigree. So, what does this software do? It’s complex; that is to say, really difficult to explain: Open source manager of managers, uber event correlation, fault message aggregator, message filter and enhancer–all of those descriptions are true. None of those really explain it, either. Instead, let’s start by defining a few of the problems.

Current Problems in Enterprise Systems Monitoring

Continuous change is prevalent in today’s IT world more than it ever has been. Administrators deploy new hardware seamlessly, but that’s not even half of the story. In virtualized environments, virtual machines can spring up as the result of just a few simple commands. They can also physically move, which plays havoc on many monitoring and reporting systems.

Agility, then, is a weak point. What happens to your monitoring system when a node physically moves? Is all the historic data associated with the old MAC or IP address re-associated properly? Or does it linger behind and get associated with the new node to occupy that space? Does your monitoring system even know when virtual machines migrate to different physical hardware?

The lack of agility and ability to handle continuous change means that the two popular enterprise monitoring systems have blind spots. Silent failures can happen if the monitoring system fails to properly recognize how the environment has changed.

Finally, there is the cost. Large IT departments are still paying in excess of six figures for maintenance contracts on this aging monitoring software. Don’t forget the specialized staff required to run these systems.

That is not to say RiverMuse can replace expensive and antiquated network management systems. Instead, it sits and gathers information from your NMS as well as other sources, and from this vantage point can make intelligent decisions about event and fault management. Let’s take a look at what it actually does.

RiverMuse’s Place in the Infrastructure

To make sure we aren’t wrongly portraying RiverMuse as an add-on to the big two NMS providers, let’s talk about the open source solutions briefly.

What is number one complaint with the open source monitoring system Zenoss? It’s the noise. You cannot even define parent-child relationships to stop Zenoss from alerting on 100 hosts when a switch has failed. With the other big option, Nagios, the main complaint is configuration. Alerting, and even adding new nodes, is a manual, time-consuming process. What if you could just send all alerts and data from these systems to a smarter system. With RiverMuse being a manager of managers, this is possible.

RiverMuse correlates events from monitoring systems, applications, and infrastructure components. This means that you aren’t constrained by information from only a few of the critical sources any longer. SNMP traps can be sent directly to RiverMuse from various infrastructure and application sources. Your current NMS can likely do that, but RiverMuse can also digest syslog and Windows events. This means event correlation goes much deeper and that actually identifying the root cause of outages is possible from within the alerting system. RiverMuse, then, is almost like a Splunk but with more enterprisey features and robust alerting capabilities.

RiverMuse will then enhance the information it gathers by taking data from configuration management databases. For example, getting an alert from your NMS that says “node web2347 is down!” is not very useful. Sysadmins even struggle with remembering whether or not they should be in panic mode. Business users and NOC employees less familiar with hostnames in an infrastructure certainly aren’t going to know if that alert means anything.

Finally, once events have been given additional business information, they get filtered. This is where RiverMuse augments many half-baked NMS solutions. It de-duplicates, filters, and summarizes only events that matter before sending them on to operations staff for immediate attention. Some NMSes do this fairly well, and some completely ignore this. RiverMuse, however, owns this functionality as its core function, and it uses more information than is available within a traditional NMS to make its decisions.

Configuring these types of systems is going to be complex. The data and rules RiverMuse is dealing with can only be automated so much before it starts to make guesses about individual business requirements. It does, based on the demos we’ve seen, make this process very easy. We all know that dedicated FTE is required to manage the big two NMSes. This time is often spent updating the NMS with new information about network changes and fighting with alerting rules. With RiverMuse’s vision, it seems entirely possible that after deploying RiverMuse these staffs will be freed up to focus more on the business logic of how the infrastructure interacts.

RiverMuse is partially open source. RiverMuse core includes all the basic components, but seems to be fairly bare-bones compared to the pro offering. But hey, RiverMuse say it has “gifted” this to the open source community. Misunderstandings of “open source” aside, this is a fascinating system built by a fascinating company comprised of veterans in the NMS space. It’s worth a look.

The post RiverMuse Brings Your Legacy NMS Into the Present appeared first on Enterprise Networking Planet.

Migrating all of your existing e-mail doesn’t have to be difficult. The big three providers, Google, Microsoft, and Yahoo, all provide many options to help alleviate the pain associated with moving an entire company’s e-mail infrastructure.

Why?

The decision to outsource e-mail, even to a free provider, is a big one. Once all the naysayers have been satisfied with regards to performance, security, and maintenance, the next big step is to devise a migration mechanism that won’t break and allow for anyone to question the decision.

To create a seamless migration from your local mail storage to the cloud, user accounts must be moved in-tact. It’s more than just the account (user and password), however. Every single mail folder must be migrated, and message timestamps need to be preserved. In short, users should not be able to detect that anything has changed.

The savings of not running 50 or more servers won’t be realized if account migration and routine maintenance tasks aren’t carefully considered. The big providers have APIs for creating, deleting, and generally adjusting user accounts. Most organizations have identity management systems that create accounts on various systems, including e-mail servers. Before making the big switch, make sure to extend these automation tools to begin managing e-mail users via the new provider’s API.

How?

There are four basic steps, once all the planning and other tasks mentioned above are complete. They are:

1. Calculate how much time the actual transfer of mail boxes will take.
2. Create the user accounts in bulk via your new provider’s API or bulk upload function.
3. Change MX records in DNS to point at your new provider.
4. Transfer

Calculating how much time it will take to transfer e-mail is critical, and difficult. If users are actively using e-mail, some “sent mail” may be left behind if the timing is off, so it is important to plan when to point users at the new service. Say you have 4,000 e-mail accounts, averaging 1GB each, and your provider will allow 100 transfers per batch. You can estimate that it should take one hour per batch, so in 40 hours the migration will be complete. Ideally, don’t let users access e-mail during this time, at all.

Next, you will want to create the user accounts at your provider. It is recommended that you initially set the password to something you know, to make sure you have as many options as possible for the transfer mechanism. We’ll cover that in a moment.

You will want to update MX records so that new mail ends up at the provider, but only after the user accounts exist, and preferably during a large window of time that your users are dormant. Make sure to lower the TTL records well before time so then changeover is quick.

Finally, the transfer itself. With most providers, there are three options for transferring all e-mail (and folders):

1. Have users do it themselves
2. IMAP automation, using provider tools or your own scripts
3. API insertion of mail

Having your users do it themselves really only works for small offices full of fairly technical people. Nevertheless, this is how: simply add the new IMAP account from your provider to a mail client, and drag & drop folders between accounts until everything has been copied over.

IMAP automation is the preferred method. There are two options here: do it yourself with imapsync, or use the provider’s tools. Google, for example, provides IMAP mail migration for education and Premiere Google Apps accounts. You must provide Google with remote access to your IMAP server, including a list of usernames and passwords for each user to migrate. Once it has that, Google’s sync tool will automatically transfer all IMAP folders over.

Getting your user’s passwords can be tricky. The best method is to get them to “register” for their new mail account, at which time you can gather the plain text version of their current password. If you’re authenticating users via LDAP, however, there is a cool way to work around this. Simply write a script that records a user’s password hash and then sets their password to something you know. Provide that known password to Google, and then restore their previous hash once the migration is complete.

Finally, most providers have an API to facilitate creating accounts and other administrative tasks. They might also allow for the transfer of e-mail via API. If this is the case with your new provider, API transfer of e-mail folders is a good idea. You’re going to need to know the provider’s API to create or delete accounts anyway, so this is a good time to practice.

What Won’t Work

Mail filters are the big issue. Unix-based procmail or sieve filters will be lost. If you’ve historically provided a Web interface to allow users to create filters, this one will probably bite you. Users that run Outlook or other mail clients, however, probably have defined filters from the client side. If filters were created server-side, you’re probably going to lose them, since Google doesn’t provide a migration path for filters.

The post Move Your E-Mail Hosting to the Cloud appeared first on Enterprise Networking Planet.

Before Zap, a wireless network administrator might have done some manual calculation to determine a best-guess baseline of how a network would perform. Check to make sure there aren’t too many overlapping channels that may cause interference, carefully plan thelayout, and hope. When testing time comes, we might run a large file copy with rsync or even use TTCP, which will tell us, on average, how fast the data transferred. This is fine for bulk transfers, and it may highlight an obvious problem in the network, but network-intensive applications today require more certainty.

How It Works

Testing wireless is difficult because the tester has no ability to control the environmental factors. We might run a crude test, like a file copy, multiple times and get multiple answers. Those answers can vary by MegaBits per second, which isn’t exactly precise.

Zap lets network designers determine the sustained worst-case performance a network can deliver 99.5 percent of the time. Zap does this by sending controlled bursts of data, measuring loss, latency, and other factors that can cause fluctuations in performance. Learning the worst-case sustained rate provides network engineers with the tools to evaluate whether streaming video or other demanding applications will run effectively on the existing network.

Getting Zap

At first, it seemed Ruckus was hopping on the marketing bandwagon that is open source. Its choice of license, however, indicates otherwise. Ruckus released Zap under a great license, the BSD, but simplified a bit. Basically, you’re free to use or even sell it, but keep the copyright notice intact and don’t imply Ruckus is endorsing your product or service.

To get Zap, visit its Google Code page for subversion checkout information:

http://code.google.com/p/zapwireless/source/checkout

Part of the reason for that awkward first impression is that Zap lacks documentation. There is no README file, or other trace of documentation in the checked out code. There are good comments in the source code, but you shouldn’t have to look there. On the Google Code page there is a Word document in the Downloads section, but it is mainly instructions for running Zap on Windows. Windows users, however, don’t get easy access to the binaries.

Anyway, Linux users. We get the source and run ‘make’ to compile Zap. There are a few warnings, but it seems to compile and run fine. Being the Curious George most of us are, we also find that turning on all warnings in gcc (-Wall) yields 78 compile warnings. Regardless, it runs and does its job, and none of the warnings seem to indicate that a grave mathematical error has been made.

In addition to the missing README file, ‘make install’ does not install any man pages. In fact, there are no man pages to speak of. This clearly isn’t Linux software, but it’s useful nonetheless, so let’s move on.

Output

The Windows documentation indicates that Zap has two components: zap and zapd. Running Zap on one node as the server requires simply invoking ‘zapd’ to start the service. Zap, then requires you to specify the source and destination IP address. Running ‘zap -h’ or any other option it doesn’t understand will coincidentally spit out syntax help information.

We first tried to run zapd on one machine, and then zap on the same. This produced a segmentation fault, so maybe testing the loopback interface performance isn’t implemented.

Running zap on two machines led us to discover it isn’t a server/client type of scenario. In reality, you need to run zapd on both machines, and then run zap to instruct them to communicate. This is weird, at best, but keep in mind that this software is generally running on their embedded devices, so it makes some sense.

With zapd running on both nodes, we ran: 

‘zap -s10.1.0.69 -d10.1.0.70 -X30’

The -X30 option instructs Zap to terminate after thirty seconds.

Every burst outputs a line, but the very last line of output is what you’re after:

  274: 10.1.0.69->10.1.0.70 1071333=rx   0=dr   0=oo   0=rp  3880=rx in    50.1ms   912.8mbps   916.8 | 944.3 926.3 883.6 871.4 863.1 856.7

Left to right, here is what this all means:

274: this was the 274th burst test

10.1.0.69: the source (sending) station

10.1.0.70: the destination (receiving) station

1071333=rx: number of received packets

0=dr: number of dropped packets

0=oo: out of order packets

0=rp: retried packets

3880=rx in: received packets in this batch/sample

50.1ms: batch sample time

912.8mbps: throughput

916.8: cumulative average throughput

944.3: peak throughput observed

926.3: median throughput (50%)

883.6: throughput at 90 percentile, i.e. it was better than this 90% of the time

871.4: 95 percentile

863.1: 99 percentile

856.7: 99.5 percentile

As you can see, Zap is fun to use on wired networks, too.

The 99.5 percent number basically tells you that you can depend on the network to deliver that speed almost always. This is especially useful for IPTV traffic, where it is important to be able to sustain a minimum rate to keep the video flowing. 

In summary, Zap provides insights into the true nature of how a wireless network will perform. The unique method of sending bursts really puts wireless (and wired) networks through its paces and demonstrates what real-world throughput performance you can expect. It’s great to have this tool, we just wish it were a little cleaner so that it might one day end up in various Linux distributions.