Data Rich, Insight Poor

This phrase has been gaining popularity.

It's a phrase used by CIO's to describe their business systems, the mountains of data generated by business applications.

This mountain has spawned new phrases and software to address business intelligence, dashboards, scorecards, key performance indicator's, etc.

At the recent MIT CIO Symposium, I heard a CIO talk about how easy it was to become "data rich, insight poor", she was talking primarily about how their BI product didn't make them that smart because the data was all over the place - they were engaged in new initiatives (ETL, Data Warehousing) to try and be data rich, insight rich.

I think a significant tertiary effect of virtualization is the increase of information.

Before you had individual piles of data, but with a virtualized environment you have the system and performance data from the ESX Server, the Host OS, the SAN, the Network, etc.

VKernel's Capacity Bottleneck Analyzer is a "ESX Intelligence" product - its a KPI, Dasboard, to do the analytics and automation to lift and load data out of Virtual Center and start analyzing resource constraints.

This problem is going to get worse.

The density of virtual machines is going to increase, we are going to see more CPU's, more RAM and more VMs per physical boxes.

It's time to look at the mountain of data and start working on the insight.

VMs can cost more than physical servers -- really!!

I am amazed how many virtual environments I have now seen that are severely under utilizing the new hardware and are afraid to increase VM density. They buy expensive server hardware, loaded it with 16Gigs or more for $30K to $50K and are running just a handful of VMs on it. This is analogous to driving a perfectly good Ferrari without ever getting out of first gear!! Say your are running 8 Vms on a $50K hardware. Add the cost of SANs, etc and you can quickly see how the cost of each VM can actually be higher than the physical server it replaced. This of course begs the question why do people underutilize the hardware?

As far as I can tell there are several reasons. Some are just being utilization "ignorant" about their environment, but the majority is simply afraid to "push the metal" and increase utilization because of concerns about running into ESX performance problems or worth yet -- downtime. Since finding capacity bottlenecks using Virtual Center is not trivial and time consuming, and predicting future capacity bottlenecks requires fairly advanced mathematical analysis of all core 4 resource types , disk I/O etc, most Vmware Admins lack the time and experience to do this exercise. So they keep the Ferrari in first gear, keep driving blindfolded, and hope that vm sprawl does not catch up with them. With availability of tools like the Vkernel Capacity Bottleneck Analyzer
VMware admins will gain visibility into current and future capacity problems and steer clear of performance issues. It heps driving with lights on!! Tell us what you think www.vkernel.com

Want awesome performance in VMWARE ESX?

Of course! we all do! So let's talk about how to get there. To achieve stellar VMware ESX performance you have to remove ALL bottlenecks in your environment. Remember your performance will only be as fast as the slowest "link" in your performance equation.

Here is the list to start removing performance bottlenecks:

1. Make sure that all of your hosts have sufficient RAM and do not over commit on memory utilization. If your workloads force the host to start swapping memory to disk, kiss your performance goodbye. There is a huge difference in speed between accessing memory internally and doing physical i/o to disk!

2. Make sure your CPU is not over utilized. Checks the processor ready queue to see how long threads are waiting to run. Extended period of time indicates a problem. Also check the overall processor utilization over a week or a month. Keep the overall utilization under 80%

3. As you scale up your ESX environment you will run into disk i/o bottlenecks. Make sure that you places your most mission critical VMs on the fastest storage available to you

4. Understand the timing of your workloads to identify when or if constraints develop in memory utilization, cpu, and disk i/o. Spread out the workloads by changing when they run i.e timing or move workloads to different hosts to mix it up.

How to predict future capacity bottlenecks

Your virtual data center is growing. You are adding a ton of new VMs every week. Wouldn't it be really cool if you head a "crystal ball" that told you in how many days you will run into capacity bottlenecks and what type of bottlenecks it will be (cpu, memory, storage) ?

Now you can. Join Vkernel's beta program for Capacity Bottleneck Analyzer that will kick off in early March

How many new VMs are you adding per week?

How many new VMs are you adding per week? This is very important question, because it has major implication to capacity availability in your ESX data center and ultimately performance. Every VM you deploy will consume cpu, memory, storage and network resources. It will also add additional disk I/O. It is easy to see how, if uncontrolled, you can quickly run out of resources and develop capacity bottlenecks. Of course the trick is to figure out which resource you are going to run out of first? Will you hit the bottleneck in memory, cpu, storage, disk i/o or network? The answer is it really depends on your environment, but in most cases the first bottleneck is memory. Why? Remember you were able to virtualize servers, because they were under utilizing CPU. That is what enabled you to combine 8+ plus servers on one piece of hardware. When you think about memory, it is a different story. Just because your servers are now virtual, it does not mean they are consuming less memory. Hence that's why in most environments the first capacity bottleneck is memory. What do you think the second capacity bottleneck you are likely to hit? Let me know at abakman@vkernel.com

9 capacity bottlenecks in ESX that kill performance

I have compiled a list of "things" that can cause you to run out of capacity resources in your ESX data center and run into performance problems or even downtime:

1. Adding new VMs though uncontrolled VM sprawl
2. Removing hosts from clusters
3. HA enabling your cluster without accounting for fail over
4. Changing Fail Over Capacity setting in a Cluster
5. Increasing reservations in VMs
6. Changing Resource Pool Configurations
7. Power up many VMs that were powered off or in maintenance
8. Natural growth rates in Storage, CPU, Memory and Network utilization
9. Changes in workloads can result in Disk I/O bottlenecks

Did I miss any? Let me know abakman@vkernel.com

Are you ready to SHARE your resources?

Sharing what? Resources? Memory? CPU? Storage?

There is an entire generation of Sys Admins now that has grown up with a distributed computing data center where one application is normally run on one server. This mostly happened because of Windows instability. Most administrators did not want to deal with trying to troubleshoot OS problems and multiple application problems at the same time. The threat of the infamous "Blue Screen of Death" defacto created this one application one server architecture. In this world admins did not have to think about or worry about sharing of resources.

Welcome to server virtualization where sharing of resources IS the primary idea. Sys Admins now will have to get used to the fact that their VM may suffer performance degradation as a result of its neighbor VM running on the same hardware and consuming a disproportionate amount of CPU and memory. So now Capacity Analysis and Capacity Monitoring becomes important again just as it was back in the mainframe days. Sys Admins now have to really pay attention to "Who is consuming what resources". Capacity Analysis is not a one time event. It is an ongoing activity. In fact many System Administrators I have spoken with are already spending a good chuck of their time troubleshooting capacity related bottlenecks in their environment.

The problem will only get worse. As organizations continue to add Virtual Machines at an exponential rate, this problem in fact will get exponentially more challenging. The unpredictability of work load management will make Capacity Analysis a required activity that will have to be performed at least daily. Just because you have used VMWARE Capacity Planner for your initial P-to-V conversion, you have to realize that it was nothing more than initial sizing. As you continue to add more virtual systems to the mix, many of the previous assumptions made by the Capacity Planner will no longer be accurate.

I love virtualization! Let me know what you think and email to abakman@vkernel.com

Alex Bakman

Virtual server sprawl reality

If you have not heard the term "virtual machine sprawl", welcome to virtualization. While the number of physical hosts in your environment will start shrinking, the number of VMs will grow exponentially once your users figure out just how easy it is to create "another server".

The implications are many:

1. If you thought that you had "too many" servers to manage before, guess what? It will actually get worse. A thousand of anything is too much to manage, ten thousand of anything will send you off the deep end. The management challenge is in the numbers, and there is no relief in sight on this front. To fight it getting organized is the answer. You will have to get really good at keeping asset inventory of your VMs. You have to know how many VMs you have where are they, what's in them, what state they are in.

2. Capacity planing will quickly become an issue. Think about it. Adding VMs at a quick pace will begin to strain your ESX resources. You will be amazed at just how quickly your "plentiful" amount of memory, storage and CPU starts to disappear. Each VMs is consuming recourses and before you know your overburdened hosts begin to develop performance problems. To fight it, you have to get disciplined and control introductions of new VMs. At minimum you need a an approval process to quickly review new VM requests.

3. Audit of VM environment will become even more challenging. With so many VMs being added, knowing who is acting on them,, what changes are being made and where will require a real herculean effort.

Are we having fun yet? What do you think? Drop me a line.

Virtualized Dataceter Brings New Challenges

Right now an average US corporation has about 7% of its Datacenter virtualized. As organizations continue to virtualize servers they will face 3 new management challenges:

1. Explosion in the number of virtual servers. User have already figured out just how easy it is for IT to create new virtual servers. The number of requests for new virtual servers will continue to skyrocket

2. Sharing of resources: memory, cpu, storage and network . In the traditional data center where one application server was dedicated to one application, no sharing of resources took place. That's not the case anymore in the virtualized datacenter

3. Servers have grown "legs". In the traditional datacenter we did not need to worry about servers moving around the network from one location to another. Now we do.

In the next post I will explore how VMWARE administrators can address the 3 new challenges