The Calm Before the Storm

Rakesh Kumar of Gartner published a white paper last fall entitled "U.S. Data Centers: The Calm Before the Storm".

In it he says U.S. Data Centers "are facing considerable disruption during the next three or more years" and they are facing it from a few things:

• Energy
• Green IT initiatives
• Floor space demands
• New technology

No mention of virtualization unless it's the source of all of the above - impacting energy, trying to be a green IT initiative, trying to help with floor space and it is a new technology.

What should CIOs be going now to prepare for this storm?

• Consider Data Center Colocation - see who has a data center nearby and see who has fiber to it - you will need 1 GB or 10 GB links depending on the size of your enterprise and hire some good financial people to determine if there is a decent ROI on moving your Data Center to a third party provider.
  
  Also ask the beancounters to factor in running your own fiber, this may not be as expensive as it once was, carriers may have available strands and you may only have to do the last mile to your location.
  
  
• Worry about your power bill - energy costs have increased, consider off peak times, VMware's introduction of DPM is the ability to power on ESX hosts and then move VMs to take advantage of lower density environments (think your overnight routines that chew up alot of CPU) with DPM you can spread them out to take advantage of lower power costs.
  
  
  Before 6 AM and after 6 PM may be lower rates ($$ per kw/hr).
  
  
  
• Invest in Systems Management tools - get something that helps you identify who is using your resources and driving up your costs. Chargeback by VM will allow you to fairly delivery charges for Data Center usage by the business unit using the most resources.
  
  This type of transparency will priortize which business unit needs to fix their Data Center problems - be it running highly transactional reports during the day that could be run at night, poorly coded applications that use too much memory, too much CPU, etc.
  
  Start a list of power supplies in the Data Center - servers, SANs, etc - you will be shocked to see newer servers may have power supplies running at 900-1300 watts - that's nearly 1 KW per hour per server.

Remember its this simple:

Servers = Power A = Heat = Cooling = Power B.

To fix this:

1. Reducing your physical servers
2. Reduces your power A
3. Reduces your heat
4. Reduces your cooling
5. Reduces your power B.

Now just get the financial data lined up to show that a server reduction project (i.e. virtualization) may costs some $$ but it be offset by the cost saving of reducing power A and B.

It's all about the ratio.

VKernel is an advocate of running your hardware at high levels of capacity. I know that we would see record-breaking ratios of virtual machines to server hardware.

• A major worldwide financial services organization achieved a 12:1 consolidation ratio and increased its central processing unit utilization by 30 percent.
  
  
• An Indian petroleum refining and distribution company achieved a 17:1 consolidation ratio and expects to increase that to 30:1 with additional CPUs and RAM.
  
  
• One of Italy's largest banks improved its server utilization rates by 100 percent.
  
  
• A leading US faucet manufacturer saved $250,000 in hardware costs by reallocating existing units instead of purchasing new, achieving a 10:1 consolidation ratio.
  
  
• A South American energy company consolidated its servers by a 20:1 ratio.
  
  
• A federation of trade unions in Singapore consolidated its servers by 46:1, achieving a 26 percent savings.
  

Smallest is 10:1 and largest is 46:1.

It's all about the ratio.

VMs can cost more than physical servers -- really!!

I am amazed how many virtual environments I have now seen that are severely under utilizing the new hardware and are afraid to increase VM density. They buy expensive server hardware, loaded it with 16Gigs or more for $30K to $50K and are running just a handful of VMs on it. This is analogous to driving a perfectly good Ferrari without ever getting out of first gear!! Say your are running 8 Vms on a $50K hardware. Add the cost of SANs, etc and you can quickly see how the cost of each VM can actually be higher than the physical server it replaced. This of course begs the question why do people underutilize the hardware?

As far as I can tell there are several reasons. Some are just being utilization "ignorant" about their environment, but the majority is simply afraid to "push the metal" and increase utilization because of concerns about running into ESX performance problems or worth yet -- downtime. Since finding capacity bottlenecks using Virtual Center is not trivial and time consuming, and predicting future capacity bottlenecks requires fairly advanced mathematical analysis of all core 4 resource types , disk I/O etc, most Vmware Admins lack the time and experience to do this exercise. So they keep the Ferrari in first gear, keep driving blindfolded, and hope that vm sprawl does not catch up with them. With availability of tools like the Vkernel Capacity Bottleneck Analyzer
VMware admins will gain visibility into current and future capacity problems and steer clear of performance issues. It heps driving with lights on!! Tell us what you think www.vkernel.com

Want awesome performance in VMWARE ESX?

Of course! we all do! So let's talk about how to get there. To achieve stellar VMware ESX performance you have to remove ALL bottlenecks in your environment. Remember your performance will only be as fast as the slowest "link" in your performance equation.

Here is the list to start removing performance bottlenecks:

1. Make sure that all of your hosts have sufficient RAM and do not over commit on memory utilization. If your workloads force the host to start swapping memory to disk, kiss your performance goodbye. There is a huge difference in speed between accessing memory internally and doing physical i/o to disk!

2. Make sure your CPU is not over utilized. Checks the processor ready queue to see how long threads are waiting to run. Extended period of time indicates a problem. Also check the overall processor utilization over a week or a month. Keep the overall utilization under 80%

3. As you scale up your ESX environment you will run into disk i/o bottlenecks. Make sure that you places your most mission critical VMs on the fastest storage available to you

4. Understand the timing of your workloads to identify when or if constraints develop in memory utilization, cpu, and disk i/o. Spread out the workloads by changing when they run i.e timing or move workloads to different hosts to mix it up.

How to predict future capacity bottlenecks

Your virtual data center is growing. You are adding a ton of new VMs every week. Wouldn't it be really cool if you head a "crystal ball" that told you in how many days you will run into capacity bottlenecks and what type of bottlenecks it will be (cpu, memory, storage) ?

Now you can. Join Vkernel's beta program for Capacity Bottleneck Analyzer that will kick off in early March