Virtualization is a wide term.

Good interview with Mendel Rosenblaum (CEO, VMware) at the Structure '08 conference. He shares some of thoughts (via GigaOM) on VMware - without an NDA.

1. Virtualization is wide. x86 or Server Virtualization is only a piece of it. (i.e. Thinapp)
   
   
2. He plugged being able to order V3I from Dell when you order the server.
   
   
3. He reminds us all that VMware started "started as a desktop virtualization company.
   
   He does mention "server side you can virtualize with virtually no overhead, but the desktop side is trailing that." so capacity/performance may be a hot topic in the VDI space.
   
   
4. "The goal is to build a computing environment on a virtualized machine because of availability and reliability."
   
   
5. Arnie Berman, chief technology strategist, Cowen and Company called VMware the "Kleenex" of virtualization.
   
   
6. He spoke about not having to evangelize server virtualization - "We spent a lot of the early times telling people that virtualiztion is a good thing, and we don’t have to do that anymore."
   
   
7. What is next??!?!? "I’m really interested in cloud computing" and spoke of managing those virtual machines and that virtual layer.
   

The Calm Before the Storm

Rakesh Kumar of Gartner published a white paper last fall entitled "U.S. Data Centers: The Calm Before the Storm".

In it he says U.S. Data Centers "are facing considerable disruption during the next three or more years" and they are facing it from a few things:

• Energy
• Green IT initiatives
• Floor space demands
• New technology

No mention of virtualization unless it's the source of all of the above - impacting energy, trying to be a green IT initiative, trying to help with floor space and it is a new technology.

What should CIOs be going now to prepare for this storm?

• Consider Data Center Colocation - see who has a data center nearby and see who has fiber to it - you will need 1 GB or 10 GB links depending on the size of your enterprise and hire some good financial people to determine if there is a decent ROI on moving your Data Center to a third party provider.
  
  Also ask the beancounters to factor in running your own fiber, this may not be as expensive as it once was, carriers may have available strands and you may only have to do the last mile to your location.
  
  
• Worry about your power bill - energy costs have increased, consider off peak times, VMware's introduction of DPM is the ability to power on ESX hosts and then move VMs to take advantage of lower density environments (think your overnight routines that chew up alot of CPU) with DPM you can spread them out to take advantage of lower power costs.
  
  
  Before 6 AM and after 6 PM may be lower rates ($$ per kw/hr).
  
  
  
• Invest in Systems Management tools - get something that helps you identify who is using your resources and driving up your costs. Chargeback by VM will allow you to fairly delivery charges for Data Center usage by the business unit using the most resources.
  
  This type of transparency will priortize which business unit needs to fix their Data Center problems - be it running highly transactional reports during the day that could be run at night, poorly coded applications that use too much memory, too much CPU, etc.
  
  Start a list of power supplies in the Data Center - servers, SANs, etc - you will be shocked to see newer servers may have power supplies running at 900-1300 watts - that's nearly 1 KW per hour per server.

Remember its this simple:

Servers = Power A = Heat = Cooling = Power B.

To fix this:

1. Reducing your physical servers
2. Reduces your power A
3. Reduces your heat
4. Reduces your cooling
5. Reduces your power B.

Now just get the financial data lined up to show that a server reduction project (i.e. virtualization) may costs some $$ but it be offset by the cost saving of reducing power A and B.

Run OS Run...

The above title is a horrible attempt at a Forest Gump reference.

Either way - Novell announces support for VMWare's Virtual Machine Interface - which allows Suse Linux Enterprise Server (SLES) 10 Service Pack 2's kernel to have "increased performance and better interoperability".

Think of it this way - there will be preferred OS's to virtualize - Novell says you can virtualize Suse since its just x86 virtualization but if you do SLES 10SP2 - you can get VMI support.

In the paravirtualization space - this will mean increased density, better running VM's, better running ESX hosts, etc.

In other words - "the guest operating system is modified to work more closely with the underlying hardware and not just with the virtualized environment."

And its a brand differentiator as well - the OS and the VM platform both have to be tuned or tunable - and with VMI - VMWare is saying they are prepared to virtualize an OS like SUSE better than its competitors.

Killer VMs on the loose.

I have been hearing more and more about the "Killer VMs" - think of this as the lovechild of Virtualization and the industry term "Killer Apps".

Wikipedia has it as "is an application so compelling that someone will buy the hardware or software components necessary to run it."

Basically virtualize the "Killer App" and its becomes the "Killer VM".

This will make you visible, put your name on the map, your CIO will thank you, your CEO may even wave at you once in recognition of your stellar service.

Don't get cocky. These "Killer Apps" often have problems and are the application server than when it hiccups, goes down, causing your business pain and makes everyone go "We really should do something about that" but no one does.

Alot of IT departments may spend a ton of money on Active/Passive Clustering, etc - remember its a favorite child, it may get all sorts of resources, dollars spent on it, trying to ensure application SLA's or increase its uptime and performance.

I have seen new servers, more memory, faster CPUs, even SAN's purchased to manage "Killer Apps".

I have seen investment in high-end clustering, low-end clustering for "Killer Apps" - when at the end of the day the "Killer App" may just be a Windows 2000 Server running SQL with an application that is mission critical.

Enter VMWare, enter DRS, VMotion, HA, etc and you pick up some amazing tools to manage these "Killer Apps" and they become "Killer VMs".

Mark Brunnel writes about putting Navision and SQL 2005 into a VM, and had the v-piphany (virtualization epiphany):

"It is amazing to see VMWare running and the management and failover capabilities. For me it means the end of active passive clusters."

Mark's done some rough benchmarking and found that "VMWare is just slightly slower in posting but only 5% maximum." that's compared to the application running in Windows.

More and more people are going past their initial P2V consolidation effors, alot more are building VMs without every having a physical server, and now folks are optimizing environments, in this 2nd or 3rd phase of Virtualization, the Killer VMs are going to start showing up - these VMs will be more important than some others, require more attention, more care and feeding and better capacity management of the resources.

No CIO is going to like to hear that a VM used by 10 people took down a "Killer VM".

No CIO wants to hear that you could have prevented it but hadn't rolled out Resource Pools yet or don't have the right systems management tools to manage/monitor resource utilization.

Water Cooled Doors / Datacenters

On the third day of cloudy skies and rain, I figured it appropriate to talk about water and how its returning to provide cooling in datacenters.

With virtualization usage increasing, server capacity increasing, and VM densities on the rise (100:1 anyone?).

The amount of power in a rack is going to change from 7-8 kW to 30+ kW or more.

That's 300-400% increases in power densities inside a single rack. This explains the increase in discussion/slides on water cooling data centers - IBM has a Rear Door Heat eXchanger and I saw Vette Corp talking about their Rear Door Heat Exchanger.

I also think more folks start looking at Deep Lake Water Cooling (DLWC), which they are doing up at Cornell and Ontario.

Metro Hall, a 27 story office building in Toronto, went online with Enwave's Deep Lake Water Cooling system in June 2006. Energy consumption at Metro Hall will be reduced by 3 million kilowatt-hours per year and reduce CO2 emissions by 732 tons annually - equivalent to taking 160 cars off the road.

I am heading to Kitchener, Ontario (near Toronto) for a VMUG and I am going to try and get over to Toronto and check it out - Enwave distributes steam and chilled water to over 140 buildings via a 40km underground pipe network that covers most of the city's downtown core.

What's great is that they don't take water out of the lake, they use the "coldness" of the deep lake water to chill water in a pipe loop - this is done with a drastically lower carbon footprint that using electricity from coal-fed power plants.

One Quad or Two?

One of the best resources on the Internet for VMWare implementation is the VMTN community forums - its top notch.

This week there was a discussion about budgets and performance (where finance always mixes it up with IT (that and Chargeback)).

The post asks about the value of two medium-speed (1.6 - 2 Ghz) QuadCore CPU's ($$) vs. one high speed (3.33 Ghz) QuadCore CPU ($$$$).

I liked William Bishop from Huntsville Hospital response - "You'll get better density on the dual socket". He prefers the "dual proc, quad core setup" and has been "adminning vmware from some of the first dual cores to the newest quad cores."

I wonder if he has done anything with 4 socket x QuadCores?

Density is important - it's going to help you drive down your per VM costs and generate better ROI on the dollars invested in a virtualization product.

B-Hive acquisition is a real smart move for VMWARE

Since the dawn of computing when it comes to performance management the most critical element is the user. It does not matter what performance metrics say inside the Virtual Center. It is all about how your users perceive the performance to be. Is it fast, Ok, too slow. It is not about the metrics. At the end of the day it is a qualitative experience. As David Marshall
correctly points out in his coverage of the news, the cool factor here is that now VMWare will be able to granularly break down where the time in application response is being spent. Is it in the network, database, application? When combined with vmotion, a vm could be moved to another host based on the analysis of where the problems are. If it is a network problem, move vm to another segment closer to ens users. If it is a host capacity issue, move it to another host where more capacity exists. The angle that really excites me is the ability to monitor more granularly down to the end user level. We would be able to answer how much resource is being utilized by a given user. This can be used in chargeback, capacity management etc. I hope to see Vmware publish this API in the near future!!

Data Rich, Insight Poor

This phrase has been gaining popularity.

It's a phrase used by CIO's to describe their business systems, the mountains of data generated by business applications.

This mountain has spawned new phrases and software to address business intelligence, dashboards, scorecards, key performance indicator's, etc.

At the recent MIT CIO Symposium, I heard a CIO talk about how easy it was to become "data rich, insight poor", she was talking primarily about how their BI product didn't make them that smart because the data was all over the place - they were engaged in new initiatives (ETL, Data Warehousing) to try and be data rich, insight rich.

I think a significant tertiary effect of virtualization is the increase of information.

Before you had individual piles of data, but with a virtualized environment you have the system and performance data from the ESX Server, the Host OS, the SAN, the Network, etc.

VKernel's Capacity Bottleneck Analyzer is a "ESX Intelligence" product - its a KPI, Dasboard, to do the analytics and automation to lift and load data out of Virtual Center and start analyzing resource constraints.

This problem is going to get worse.

The density of virtual machines is going to increase, we are going to see more CPU's, more RAM and more VMs per physical boxes.

It's time to look at the mountain of data and start working on the insight.

Capacity Planning for ESX is a multi-dimensional challenge

Windows Admins listen up. Your world has changed. No more one application running on one Windows server where none of your capacity resources were shared. Once you virtualize your servers, they have to share memory, cpu, storage, network bandwidth, disk i/o, network i/0, etc, with other VMs running on the same hardware. Capacity planning which was a non-event in the Windows world is now a must do, otherwise you will run out of capacity and experience performance problems or even worse - downtime.

Capacity Planning is a multidimensional problems. To do it correctly you must take into account literally hundreds of variables. Here are some of them:

- how many VMs do you deploy?
- where are you going to deploy them
- how much resource to allocate to them
- what happens if you want to change hardware
- will you violate any configuration constraints?
- do you need another host?
-what resource will you run out of first? memory? CPU, Storage -- and where?
- how many more VMs can you fit into each cluster?
- what happens if VMs get vmotioned?
- will you violate DRS affinity rules?
- what configuration constraints will you violate?
- will DRS work?
- will HA work?

I can go on and on. I hope you see just how complex capacity planning has become. As VM density on hosts continues to increase, capacity planning in VMware will become even more critical, because every physical server becomes more business critical and failure is not an option. Systems Management is fun again!!

It's all about the ratio.

VKernel is an advocate of running your hardware at high levels of capacity. I know that we would see record-breaking ratios of virtual machines to server hardware.

• A major worldwide financial services organization achieved a 12:1 consolidation ratio and increased its central processing unit utilization by 30 percent.
  
  
• An Indian petroleum refining and distribution company achieved a 17:1 consolidation ratio and expects to increase that to 30:1 with additional CPUs and RAM.
  
  
• One of Italy's largest banks improved its server utilization rates by 100 percent.
  
  
• A leading US faucet manufacturer saved $250,000 in hardware costs by reallocating existing units instead of purchasing new, achieving a 10:1 consolidation ratio.
  
  
• A South American energy company consolidated its servers by a 20:1 ratio.
  
  
• A federation of trade unions in Singapore consolidated its servers by 46:1, achieving a 26 percent savings.
  

Smallest is 10:1 and largest is 46:1.

It's all about the ratio.

VMs can cost more than physical servers -- really!!

I am amazed how many virtual environments I have now seen that are severely under utilizing the new hardware and are afraid to increase VM density. They buy expensive server hardware, loaded it with 16Gigs or more for $30K to $50K and are running just a handful of VMs on it. This is analogous to driving a perfectly good Ferrari without ever getting out of first gear!! Say your are running 8 Vms on a $50K hardware. Add the cost of SANs, etc and you can quickly see how the cost of each VM can actually be higher than the physical server it replaced. This of course begs the question why do people underutilize the hardware?

As far as I can tell there are several reasons. Some are just being utilization "ignorant" about their environment, but the majority is simply afraid to "push the metal" and increase utilization because of concerns about running into ESX performance problems or worth yet -- downtime. Since finding capacity bottlenecks using Virtual Center is not trivial and time consuming, and predicting future capacity bottlenecks requires fairly advanced mathematical analysis of all core 4 resource types , disk I/O etc, most Vmware Admins lack the time and experience to do this exercise. So they keep the Ferrari in first gear, keep driving blindfolded, and hope that vm sprawl does not catch up with them. With availability of tools like the Vkernel Capacity Bottleneck Analyzer
VMware admins will gain visibility into current and future capacity problems and steer clear of performance issues. It heps driving with lights on!! Tell us what you think www.vkernel.com

Swims like a mainframe.

I love the duck test. If a bird looks like a duck, swims like a duck and quacks like a duck, then it's probably a duck.

HP's new 8-way DL785 G5 looks like a mainframe wannabe.

Replace the mainframe OS ($$$$) and replace it with ESX ($$). Then replace mainframe workloads with virtual machines - basically VM workloads - both are consumers of disk, memory, cpu, network.

Mainframes try to run continuously at over 70% busy. A 90% figure is more typical, and modern mainframes could see sustained periods of 100% CPU utilization. You're going to need a capacity tool.

Typically, a mainframe is repaired without being shut down. Also, memory, storage and processor modules of chips could be added or hot swapped without being shut down. It is not unusual for a mainframe to be continuously switched on for 6 months at a stretch.

So maybe if it runs CPU like a mainframe, and has uptime like a mainframe, is it a mainframe??

Check out the numbers:

8 sockets (up to 32 cores)
64 DIMM slots, (Up to 256 GB of RAM - 4 GB max per slot)
11 PCI-e expansion slots (3 x16 slots, 3 x8 slots and 5 x4 slots)
2.3 terabytes of internal storage

When the 8 GB DIMMs ship - this could be 512 GB of RAM.

HP is aiming these behemoths are two roles:

1) Very Large Database Systems (VLDBS)

Very large database servers with massive data buffer caches.

2) Very Large Virtualization System (VLVS)

These are going to push capacity and virtual machine counts to new historic levels.

A huge issue with these massive systems into production is finding better/smarter management tools that can help you identify potential capacity bottlenecks and gather capacity and performance data. Oh and don't forget about VM chargeback.

The key to these beasts looks like the Opteron chipset - no shared memory bus - each processor has its own memory and I/O bus. Sun's Sun Fire X4600's also running's eight sockets and Opteron's. I can't imagine Intel is going to stand for that - new word of the week - octal core.

The mainframe folks are seeing a return to shared processing of the very large systems, so it may not be a mainframe per se, but this system sure quacks and swims like one. Except it csts like a server.

How to predict future capacity bottlenecks

Your virtual data center is growing. You are adding a ton of new VMs every week. Wouldn't it be really cool if you head a "crystal ball" that told you in how many days you will run into capacity bottlenecks and what type of bottlenecks it will be (cpu, memory, storage) ?

Now you can. Join Vkernel's beta program for Capacity Bottleneck Analyzer that will kick off in early March

How many more VMs can you fit into a Cluster, Resource Pool or Host?

Given the speed at which most admins are adding new VMs to their environment ("vm sprawl"), every admin has to figure out where to deploy the new VMs. Simply guessing about resource availability will lead to performance problems and downtime. What I am proposing here is a simple 4 step process you can use to determine how many more VMs you can fit into a Cluster, Resource Pool or a host. Here it goes

1. Select a cluster, resource pool or a host
2. Get info on available memory, storage, cpu, disk i/o and net i/o
3. Calculate an average VM footprint in the cluster, RP or host in terms memory, cpu, storage, disk i/o
4. Apply average VM footprint to every resource type to see which resource you will run out of first.That’s how many more VMs you can fit into host, cluster or resource pool

o

Servers are no longer a "Resource Boundary"

One of the hardest concepts for System Administrators new to virtualization to understand is the shared resource management. VMware ESX makes it possible to share resources namely memory, cpu, storage and network not only inside a physical host, but also across multiple physical hosts. The resources are pulled together to create one massive resource pool captured in a concept called a cluster. Even resources inside clusters can be further subdivided into many Resource Pools. For admins who are only used to dealing with physical servers as resource boundaries this can be confusing, especially when it comes to planning and management of capacity. For example when monitoring or determining resource capacity, Admins must now take into consideration how all resource boundaries are affected. Looking just at physical servers is no longer an option!

How many new VMs are you adding per week?

How many new VMs are you adding per week? This is very important question, because it has major implication to capacity availability in your ESX data center and ultimately performance. Every VM you deploy will consume cpu, memory, storage and network resources. It will also add additional disk I/O. It is easy to see how, if uncontrolled, you can quickly run out of resources and develop capacity bottlenecks. Of course the trick is to figure out which resource you are going to run out of first? Will you hit the bottleneck in memory, cpu, storage, disk i/o or network? The answer is it really depends on your environment, but in most cases the first bottleneck is memory. Why? Remember you were able to virtualize servers, because they were under utilizing CPU. That is what enabled you to combine 8+ plus servers on one piece of hardware. When you think about memory, it is a different story. Just because your servers are now virtual, it does not mean they are consuming less memory. Hence that's why in most environments the first capacity bottleneck is memory. What do you think the second capacity bottleneck you are likely to hit? Let me know at abakman@vkernel.com

9 capacity bottlenecks in ESX that kill performance

I have compiled a list of "things" that can cause you to run out of capacity resources in your ESX data center and run into performance problems or even downtime:

1. Adding new VMs though uncontrolled VM sprawl
2. Removing hosts from clusters
3. HA enabling your cluster without accounting for fail over
4. Changing Fail Over Capacity setting in a Cluster
5. Increasing reservations in VMs
6. Changing Resource Pool Configurations
7. Power up many VMs that were powered off or in maintenance
8. Natural growth rates in Storage, CPU, Memory and Network utilization
9. Changes in workloads can result in Disk I/O bottlenecks

Did I miss any? Let me know abakman@vkernel.com

Are you ready to SHARE your resources?

Sharing what? Resources? Memory? CPU? Storage?

There is an entire generation of Sys Admins now that has grown up with a distributed computing data center where one application is normally run on one server. This mostly happened because of Windows instability. Most administrators did not want to deal with trying to troubleshoot OS problems and multiple application problems at the same time. The threat of the infamous "Blue Screen of Death" defacto created this one application one server architecture. In this world admins did not have to think about or worry about sharing of resources.

Welcome to server virtualization where sharing of resources IS the primary idea. Sys Admins now will have to get used to the fact that their VM may suffer performance degradation as a result of its neighbor VM running on the same hardware and consuming a disproportionate amount of CPU and memory. So now Capacity Analysis and Capacity Monitoring becomes important again just as it was back in the mainframe days. Sys Admins now have to really pay attention to "Who is consuming what resources". Capacity Analysis is not a one time event. It is an ongoing activity. In fact many System Administrators I have spoken with are already spending a good chuck of their time troubleshooting capacity related bottlenecks in their environment.

The problem will only get worse. As organizations continue to add Virtual Machines at an exponential rate, this problem in fact will get exponentially more challenging. The unpredictability of work load management will make Capacity Analysis a required activity that will have to be performed at least daily. Just because you have used VMWARE Capacity Planner for your initial P-to-V conversion, you have to realize that it was nothing more than initial sizing. As you continue to add more virtual systems to the mix, many of the previous assumptions made by the Capacity Planner will no longer be accurate.

I love virtualization! Let me know what you think and email to abakman@vkernel.com

Alex Bakman

Virtual server sprawl reality

If you have not heard the term "virtual machine sprawl", welcome to virtualization. While the number of physical hosts in your environment will start shrinking, the number of VMs will grow exponentially once your users figure out just how easy it is to create "another server".

The implications are many:

1. If you thought that you had "too many" servers to manage before, guess what? It will actually get worse. A thousand of anything is too much to manage, ten thousand of anything will send you off the deep end. The management challenge is in the numbers, and there is no relief in sight on this front. To fight it getting organized is the answer. You will have to get really good at keeping asset inventory of your VMs. You have to know how many VMs you have where are they, what's in them, what state they are in.

2. Capacity planing will quickly become an issue. Think about it. Adding VMs at a quick pace will begin to strain your ESX resources. You will be amazed at just how quickly your "plentiful" amount of memory, storage and CPU starts to disappear. Each VMs is consuming recourses and before you know your overburdened hosts begin to develop performance problems. To fight it, you have to get disciplined and control introductions of new VMs. At minimum you need a an approval process to quickly review new VM requests.

3. Audit of VM environment will become even more challenging. With so many VMs being added, knowing who is acting on them,, what changes are being made and where will require a real herculean effort.

Are we having fun yet? What do you think? Drop me a line.

Virtualized Dataceter Brings New Challenges

Right now an average US corporation has about 7% of its Datacenter virtualized. As organizations continue to virtualize servers they will face 3 new management challenges:

1. Explosion in the number of virtual servers. User have already figured out just how easy it is for IT to create new virtual servers. The number of requests for new virtual servers will continue to skyrocket

2. Sharing of resources: memory, cpu, storage and network . In the traditional data center where one application server was dedicated to one application, no sharing of resources took place. That's not the case anymore in the virtualized datacenter

3. Servers have grown "legs". In the traditional datacenter we did not need to worry about servers moving around the network from one location to another. Now we do.

In the next post I will explore how VMWARE administrators can address the 3 new challenges