Category vSphere

Hey guys, the other day I was having a conversation with a friend of mine that went something like this.

How did this all start you might say?!? Well, contrary to popular belief, I am a STAUNCH NetApp FUD dispeller.  What that means is, if I hear something said about NetApp by a competitor, peer, partner or customer which I feel is incorrect or just sounds interesting; I task it upon myself to prove/disprove it because well frankly… People still hit me up with NetApp questions all the time :) (And I’d like to make sure I’m supplying them with the most accurate and reflective data! – yea that’s it, and it has nothing to do with how much of a geek I am.. :))

Well, in the defense of the video it didn’t go EXACTLY like that.   Here is a little background on how we got to where that video is today :)   I recently overheard someone say the following:

What I hear over and over is that dedupe rates when using VMware deteriorate over time

And my first response was “nuh uh!”, Well, maybe not my FIRST response.. but quickly followed by; “Let me try and get some foundational data”  because you know me… I like to blog about things and as a result collect way too much data to try to validate and understand and effectively say whatever I say accurately :)

The first thing I did was engage several former NetApp folks who are as agnostic and objective as I am to get their thoughts on the matter (we were on the same page!) – Data collection time!  

For Data Collection… I talked to some good friends of mine regarding how their Dedupe savings have been over time because they were so excited when we first enabled it in the first place (And I was excited for them!)   This is where I learned some… frankly disturbing things (I did talk to numerous guys named Mike interestingly enough, and on the whole all of those who I talked with and their data they shared with me reflected similar findings)

Disturbing things learned!

Yea I’ve heard all the jibber jabber before usually touted as FUD that NetApp systems will deteriorate over time in general (whether it be Performance, whether it be Space Savings) etc etc. 

Well some of the disturbing things learned actually coming from the field on real systems protecting real production data was:

• Space Savings are GREAT, and will be absolutely amazing in the beginning! 70-90% is common… in the beginning. (Call this the POC and the burn-in period)

• As that data starts to ‘change’ ever so slightly as you would expect your data to change (not sit static and RO) you’ll see your savings start to decrease, as much as 45% over a year
• This figure is not NetApp’s fault.  Virtual machines (mainly what we’re discussing here) are not designed to stay uniformly the same no matter what in accordance to 4k blocks, so the very fact that they change is absolutely normal so this loss isn’t a catastrophe, it’s a fact of the longevity of data.

• I found that particularly disturbing because after reviewing the data from these numerous environments… I had the impression VMware data would account for much more!   I saw a 50TB SAN only have ~2TB of data residing in Data stores and of that only 23% of it was deduplicating (I was shocked!)
• I was further shocked that when reviewing the data that over the course of a year on a 60TB SAN, this customer only found 12TB of data they could justify running the dedupe process against and of that they were seeing less than 3TB of ‘duplicate data’ coming in around 18% space savings over that 12TB.    The interesting bit is that the other 48TB of data just continued on un-affected by dedupe.   (Yes, I asked why don’t they try to dedupe it… and they did in the lab and, well it never made it into production)

At this point, I was even more so concerned.   Concerned whether there was some truth to this whole NetApp starts really high in the beginning (Performance/IO way up there, certain datasets will have amazing dedupe ratios to start) etc. and then starts to drop off considerably over time, while the EMC equivalent system performs consistently the entire time.

Warning! Warning Will Robinson!

This is usually where klaxons and red lights would normally go off in my head.    If what my good friends (and customers) are telling me is accurate, it is that not only will my performance degrade just by merely using the system, but my space efficiency will deteriorate over time as well.    Sure we’ll get some deduplication, no doubt about that!  But the long term benefit isn’t any better than compression (as a friend of mine had commented on this whole ordeal)    With the many ways of trying to look at this and understand I discussed it with my friend Scott who had the following analogy and example to cite with this:

The issue that I’ve seen is this:

Since a VMDK is a container file, the nature of the data is a little different than a standard file like a word doc for example.

Normally, if you take a standard windows C: – like on your laptop, every file is stored as 4K blocks.  However, unless the file is exactly divisible by 4K (which is rare), the last block has just a little bit of waste in it.  Doesn’t matter if this is a word doc, a PowerPoint, or a .dll in the \windows\system32 directory, they all have a little bit of waste at the end of that last block.

When converted to a VMDK file, the files are all smashed together because inside the container file, we don’t have to keep that 4K boundary.  Kind of like sliding a bunch of books together on a book shelf eliminating the wasted space.  Now this is one of the cool things about VMware that makes the virtual disk more space efficient than a physical disk – so this is a good thing.

So, when you have a VMDK and you clone it – let’s say create 100 copies and then do a block based dedupe – you’ll get a 99% dedupe rate across those virtual disks.  That’s great – initially.  Netapp tends to calculate this “savings” into their proposals and tell customers that require 10TB of storage, that they can just buy 5TB and dedupe and then they’ll have plenty of space.

What happens is, that after buying ½ the storage they really needed the dedupe rate starts to break down. Here’s why:

When you start running the VMs and adding things like service packs or patches for example – well that process doesn’t always add files to the end of the vmdk.  It often deletes files from the middle, beginning, end and then  replaces them with other files etc.  What happens then is that the bits shift a little to the left and the right – breaking the block boundaries. Imagine adding and removing books of different sizes from the shelf and making sure there’s no wasted space between them.

If you did a file per file scan on the virtual disk (Say a windows C: drive), you might have exactly the same data within the vmdk, however since the blocks don’t line up, the block based dedupe which is fixed at 4K sees different data and therefore the dedupe rate breaks down.

A sliding window technology (like what Avamar does ) would solve this problem, but today ASIS is fixed at 4K. 

Thoughts?

If you have particular thoughts about what Scott shared there, feel free to comment and I’ll make sure he reads this as well; but this raises some interesting questions.   

We’ve covered numerous things in here, and I’ve done everything I can to avoid discussing the guarantees I feel like I’ve talked about to death (linked below) so addressing what we’ve discussed:

• I’m seeing on average 20% of a customers data which merits deduping and of that I’m seeing anywhere from 10-20% space saved across that 20%

• Translation: 100TB of data, 20TB is worth deduping reclaiming about 4TB of space in total; thus on this conservative estimate you’d get about 4-5% space saved!
• Translation: When you have a 20TB data warehouse and you go to dedupe it (You won’t) you’ll see no space gained, with a 100% cost across it.

• With the EMC Unified Storage Guarantee, that same 20TB data warehouse will be covered by the 20% more efficient guarantee (Well, EVERY data type is covered without caveat)   [It’s almost like it’s a shill, but it really bears repeating because frankly this is earth shattering and worth discussing with your TC or whoever]

For more great information on EMC’s 20% Unified Storage Guarantee – check out these links (and other articles I’ve written on the subject as well!)

• EMC didn’t invent Unified Storage; They Perfected it
• EMC 20% Unified Storage Guarantee: Final Reprise
• EMC Unified Storage Capacity Calculator – The Tutorial!
• EMC 20% Unified Storage Guarantee !EXPOSED!
• EMC Unified Storage is 20% more efficient Guaranteed!

I won’t subject you to it, especially because it is over 7 minutes long, but here is a semi funny (my family does NOT find it funny!) video about EMCs Unified Storage Guarantee and making a comparison to NetApp’s Guarantee.   Various comments included in the description of the video – Don’t worry if you never watch it… I won’t hold it against you ;)

Be safe out there, the data jungle is a vicious one!   If you need any help driving truth out of your EMC or NetApp folks feel free to reach out and I’ll do what I can :)

SPOILERS!!!

Read More

Hi Guys! Remember me! I’m apparently the one who upset some of you, enlightened others; and the rest of you.. well, you drove a lot of traffic here to get my blog to even beat out EMC’s main website as the primary source for information on "Unified Storage" (And for that, I appreciate it :))

• EMC 20% Unified Storage Guarantee !EXPOSED!
• EMC Unified Storage Capacity Calculator – The Tutorial!
• EMC 20% Unified Storage Guarantee: Final Reprise

In case any of you forgot some of those "target" posts, here they are for your reference! but I’m not here to start a fight! I’m here to educate and to direct my focus on not what this previously OVERLY discussed Unified Storage Guarantee was or is, but instead to drive down in to what Unified Storage will really bring to bear.   So, without further adieu!

What is Unified Storage?

I’ve seen a lot of definitions of what it is, quite frankly a lot of stupid definitions too. (My GOD I hate stupid definitions!)  But what does it mean when you Unify to you and me?   I could go on and on about the various ‘definitions’ of what it really is (and I even started WRITING that portion of it!) but instead I’m going to scrap all of that so I do not end up on my own list of ‘stupid definitions’ and instead will define Unified Storage at it’s simplest terms.

A unified storage system merges NAS and SAN. Optimized for performance and interoperability, the system simultaneously stores both file data and blocks of application data in virtually any operating environment

You can put your own take and spin on it, but at it’s guts that is seemingly what the basics of a "Unified Storage" system are; nothing special about it, NAS and SAN (hey, lots of people do that right?!)  You bet they do!   And this is by no way the definitive definition on what “Unified Storage” is, and frankly that is not my concern either.   So taking things to the next level; now that we have a baseline of what it takes to ‘get the job done’, now it’s time to evaluate the Cost of Living in a Unified Storage environment.

Unified Storage Architecture Cost of Living

I get it.  No really I do.   And I’m sure by now you’re tired of the conversation of ‘uniqueness’ focused on the following core areas:

• Support for Mixed Clients
• Support for multiple types (tiers) of disk
• Simplified Provisioning
• Thin Provisioning
• Improving Utilization

All of these items are simply a FACT and an expectation when it comes to a Unified Platform.  (Forget unified, a platform in general)   Lack of support of multiple tiers, locking down to a single client, complicated provisioning which can only be done fat which makes you lose out on utilization and likely is a waste of time – That my friend is the cost of living.    You’re not going to introduce a wasteful fat obsolete system and frankly, I’m not sure of any (many) vendors who are actually delivering services which don’t meet on multiple of these criteria; So the question I’m asking is… Why do we continue to discuss these points?   I do not go to a car dealership and say “You know, I’m expecting a transmission in this car, you have a transmission right?”  And feel free to replace transmission with tires and other things you just flat out EXPECT.    It’s time to take the conversation to the next level though; because if you’ve ever talked to me you know how I feel about storage. “There is no inherent value of storage in and of itself without context or application.”   Thus… You don’t want spinning rust just for the sake to have it spin, no you want it to store something for you, and it is with that you need to invest in Perfection.

Unified Storage Perfection

What exactly is the idea of Unified Storage Perfection?   It is an epic nirvana whereby we shift from traditional thinking and takes NAS and SAN out of the business of merely rusty spindles and enable and engage the business to earn its keep.

Enterprise Flash Disks

Still storage, yet sexy in it’s own right.  Why?  First of all, it’s FAST OMFG FLASH IS SO FAST! And second of all, it’s not spinning, so it’s not annoying like the latest and greatest SAS, ATA or FC disk!    But what makes this particular implementation of EFD far sexier than simple consumer grade SSD’s is the fact that these things will guarantee you a consistent speed and latency through and through.   I mean, sure it’s nice that these things can take the sheer number of FC disks you’d need to run an aggressive SQL server configuration and optimize the system to perform, but it goes beyond that.   

Fully Automated Storage Tiering (FAST)

Think back to that high performance SQL workload you had a moment ago, there might come a time in the life of the business where your performance needs change; Nirvana comes a knocking and with the power of FAST enables you to dynamically, non-disruptively move from one tier of Storage (EFD, FC, SATA) to another, so you are guaranteed not only investment protection but scalability which grows and shrinks as your business does.    Gone are the days of ‘buy for what we might use one day’ and welcome are the days of Dynamic and Scalable business.

FAST Cache

Wow, is this the triple whammy or what?  Building upon the previous two points, this realm of Perfection is able to take the performance and speed of Enterprise Flash Disks and the concept of tiering your disks to let you use those same existing EFD disks to extend your READ and WRITE cache on your array!    FAST Cache accelerates performance to address unexpected workload spikes. FAST and FAST Cache are a powerful combination, unmatched in the industry, that provides optimal performance at the lowest possible cost.  (Yes I copied that from a marketing thingie, but it’s true and is soooooo cool!) 

FAST + FAST Cache = Unified Storage Performance Nirvana

So, let’s put some common sense on this then, because this is no joke, nor is it marketing BS.    You assign EFD’s to a specific workload you want to guarantee a certain speed and a certain response time (Win).    You have unpredictable workloads who may need to be fast some times, but may be slow other times on quarterly of yearly basis’s, so you leverage FAST to move that data around, but that’s your friend when you can PREDICT what is going to happen.    What about when it is slow most of the time, but then on June 29th you make a major announcement that you were not expecting to hit as hard as it did, and BAM! Your system goes in the tank because data sitting on FC or SATA couldn’t handle the load.   Hello FAST Cache, how I love you so.     Don’t get me wrong, I absolutely LOVE EFD’s and I wish all of my data could sit on them (At home a lot of it does ;)) and I have massive desire for FAST because I CAN move my workload around based upon predictable or planned patterns (Marry me!)  But FAST Cache is my superman, because he is there to save the day when I least expected it, he caches my reads when BOOM I didn’t know it was coming, but more importantly he holds my massive load of WRITES which come in JUST as unexpectedly.   So for you naysayers or just confused ones who wonder why you’d have one vs the other (vs) the other; Hopefully this example use-case is valuable.   Think about it in terms of your business, you could get away with one or the other, or all three… Either way, you’re a winner.

Block Data Compression

EMC is further advancing its storage efficiency innovation as the first storage provider to introduce block data compression, by allowing customers to compress inactive data and reclaiming valuable storage capacity— data footprints can be reduced by up to 50 percent. A common use case would be compressing inactive data once EMC FAST software has moved that data to the most cost-effective storage tier. Block data compression joins EMC’s existing capabilities, including thin provisioning and data deduplication, to automatically and transparently maximize storage utilization.

Yea, I DID copy that verbatim from a Press Release – And do you know why? Because it’s right! Even addresses a pretty compelling use-case too!   So think about it a moment.  Does this apply to you?  I’d never compress ALL of my data (reminisces back to the days of DoubleSpace where let’s just say, for any of us who lived it… those were interesting times ;)) But think about the volume of data which you have sitting on Primary Storage which is inactive and otherwise wasting space when it continues sitting un-accessed and consuming maximum capacity!  But this is more than just about that data type, unlike some solutions this it not an all or nothing.

Think if you could choose to compress on demand! Compress say… your virtual machine right out of vCenter! But wait there’s more!   And there’s so much more to say on this, let alone the things which are coming.. I don’t want to reveal what is coming, so I’ll let Mark Twomey do it where he did it here:  Storage Services for Clariion Storage Pool LUNs

What does all of this mean for me and Unified Storage?!

Whoa, hey now! What do you mean what does all of this mean?! Are you cutting me short?  Yes.  Yes I am. :)   There are some cool things coming, which I cannot talk about yet… and not to mention some of all of the new stuff coming in Q3 – But things I was talking about… that’s stuff I can talk about –TODAY- there’s only even better things and cake coming tomorrow :)

I can fill this with videos, decks, resources, references, Unisphere and every thing under the sun (You let me know if you really want that.. I’ve done that in the past as well)  But ideally, I want you to make your own decision, come to your own conclusions..  What does this mean for you?   Stop asking “What is Unified Storage” and start asking “What value can my business derive from technologies in order to save money, save time, save waste!”    I’ll try to avoid writing yet another article on this subject unless you so demand it! I look forward to all of your comments and feedback! :)

Hey guys, it’s been a long while since I’ve done a Post-Mortem on an exam.. I just didn’t feel like it from the last few betas I took – So here you go, with so much interest in the Hyper-V exam here is my post-mortem analysis and not to mention what I felt about it, and why I said “Wow” :)

Pro: Windows Server 2008 R2, Virtualization Administrator

About this Exam

This exam validates a candidates knowledge of Microsoft virtualization technologies.

Audience Profile

Candidates should have one to three years of experience using Microsoft virtualization products, including Hyper-V, System Center Virtual Machine Manager, and Remote Desktop Services (RDS), in a Windows Server 2008 R2 infrastructure. Candidates for this exam are IT professionals who have jobs in which managing or deploying virtualization technologies is their main area of responsibility.

Credit Toward CertificationExam 70-693: Pro: Windows Server 2008 R2, Virtualization Administrator: counts as credit toward the following certification(s):

Microsoft Certified IT Professional: Windows Server 2008 R2, Virtualization Administrator

So, there is the high level view of the exam as listed at Pro: Windows Server 2008 R2, Virtualization Administrator and one of the most useful tools you will find on that page is the “Skills Measured” tab which happens to give you a comprehensive overview of what kind of content there is on the exam – If you follow that list and rule, you will indeed be prepared if you study against the skills measured!  I do want to note, I HIGHLY encourage you to check out the ‘Skills Measured’ from TS: Windows Server Virtualization, Configuring – Seriously!  – A slight disclaimer here.. I mistakenly wrote the reference material against last years 70-652 TS: Windows Server Virtualization, Configuring – But take it for what it is.. Combine the two ‘skills measured’ from both exams and your chances of passing will increase exponentially!

Now what may be beneficial is a comprehensive understanding of… competitive pressures? Would you call it that? I have to say, I saw a damn lot of another vendors virtualization product (Some might call it, the largest virtualization product in the industry, not to mention the most deployed)   In the “Installing Hyper-V” section, as seen in Skills Measured, it mentions very briefly a coverage of clustering, storage – shared and otherwise – accounting for 14% of the exam.  To me it honestly felt more like 45% of the exam had some focus on Storage or Clustering.  I haven’t seen that much iSCSI, and FCP touted in a long time! (Take my NFS and CIFS Please! – Oh, yea while not mentioned, you probably want to ensure you’re up on the entire protocol stack, grin :))

Next, if you look across all 4 Skill areas, you’ll notice SCVMM is included in there.   Yea, there’s a reason for that.   Infact, I’d be surprised if there were any questions which DIDN’T include SCVMM! I say ‘mostly’ in jest, because it makes you wonder ‘Is this Hyper-V, or a purely SCVMM exam?!?” :)

As far as annoying faults in the tests go, I only found one major syntactical error which I reported, but on the whole the test itself was well formed and the questions were free of Grammatical mistakes.   Now, let’s get into the Wow section.

Perhaps I was a bit hasty when I said “Wow” about this exam.  Perhaps I should have placed myself more into the category of WTF?!?   So, feel free to see an intermingling of my thoughts on the exam now :)   The questions were well formed, perhaps even a little too well formed.   A number of them looked as though they were struggling to find examples of what WASN’T the right answer, because they were all pretty damn easy to answer in and out!   Am I saying I passed? There’s a pretty good chance, but I place no bets!    If you are NOT up on the competitive landscape as far as where Hyper-V plays in the industry, you better be to take this exam.  I wasn’t sure if I was sitting for the VCP, a minor in Citrix, or if this was infact an actual Microsoft Exam! (Yes, I know it was a Microsoft exam because all of the questions WERE very well formed, and a number of them… were sadly still written to the old adage of ‘Choose the microsoft answer’ ;)

This exam also included the recent name changes to products, so I commend it’s accuracy!    And the intimate level of focus on VDI – was quite amazing, but sadly I reach a saddening point.

If I am to fail in this exam under any circumstances? It is because of the number of ‘it depends’ questions they had in there.   What does that mean?   I’m sure providing details about how many interfaces you should have and factual information backing it is PERFECTLY okay, I can sign off on that – No problem, albeit Best Practice and ‘minimal acceptable’ is further subjective.   But when it comes to degrees of scale and how many VM’s I can actually host on a particular server?   Without raw details, a breakdown of workload, and not to mention this isn’t a different vendors solution so the pure economies of scale require me to be EXTREMELY conservative.   I’m not being negative I’m being factual, we all know that – and we know JUST how subjective things are when it comes to VM density.   With that said be very careful, I have no guidance there other than try to find out what the proverbial ‘microsoft answer’ is for what density looks like I’ve always seen it published as ‘not as much as others’ and some of the deployments in the exam outright scared me – And I don’t get scared by technology, I put fear into it’s heart!

I’m FAIRLY certain I didn’t say anything which violates NDA, since pretty much everything included here is referenced in the Skills Measured page Pro: Windows Server 2008 R2, Virtualization Administrator but incase I did… don’t spank me! preferably fix the questions which are wrong (glares in Liberty’s direction ;)) And… Well, have a good time – Use of the technology and understanding these skills are pretty much all you need in order to pass!

Now on a personal note! I’m going to be running the Boston Marathon in a few months in order to raise money for disabled children and every single dollar helps, so if you can help me in my cause these children and their families will greatly appreciate it!   Even if you can only afford $1 that’s perfectly fine! The more people who contribute the better!

http://www.firstgiving.com/cxi – Help sponsor my run in the Boston Marathon on behalf of disabled children!

So, thank you all and I hope you find something useful from this post-mortem and truly every $1 helps, and I greatly appreciate it!  Thanks!

I was recently on the Internets when it came about that Oracle President Charles Phillips said he he didn’t realize that customers wanted to run their Oracle products on VMware (similar to how Microsoft customers run their  line of products on both VMware and Hyper-V today)   Wow, is the first thing I’d say – normally you expect “Hey, they’re just ignoring our needs, I mean cmon… who DOESN’T want their product to be virtualized? I mean even the greatest OLTP app, I’d rather find a way to scale and virtualize it than taking up even MORE footprint on the floor with power and cooling to run yet another app on yet another physical box! (Let’s not even get into DR!)

With that said though, Charles Phillips needs to hear from you – the customer : Do you want to run Oracle on VMware? Let him know! – The contact information sniped from another location (which I’ll cite inline as well :))

Re: Need a favor – Oracle

You are not going to believe this. Some VMware folks met with Charles Phillips, the president or CEO of Oracle and he said no customers had ever mentioned to him that they wanted Oracle to support their products on VMware. Or modify the licensing scheme. He offered if anyone knew of customers who did want better or more support for Oracle on VMware, or virtualization friendly licensing, to email him directly. His email is Charles.phillips@oracle.com and he really needs to hear that customers run Oracle on VMware, and better support / licensing would be nice!

So, I’d like to send special thanks to Oracle Storage Guy, Duncan Yellow Bricks, Chad SaKac – and I’m sure others may have discussed this already!

Oracle on VMware – it’s time for us to speak up or shut up!

Oracle feels that not many people want to run their apps in a virtual environment!

Charles Phillips Asks for Customer Input on VMware

I feel it bears repeating – Let Charles know how you feel about Oracle on VMware! I’m always a fan of using my existing virtualization investment to run my next app on, instead of having to invest in yet ANOTHER point solution to run one of my existing apps (which is where it looks like they’ve been trying to go with their Xen, Iron and other investments ;))

If you know me (Sure you do, right!? ;)) you know that I like to find good value which is quick, effective and often and especially free! :)

I’d like to introduce you to Mike Laverick, Virtualization guru extraordinaire! And what Mike has produced here is amazing videos (which you can actually see, since he converted them to SWF/Shockwave files) and not to mention they’re human, humourous and even more so, extremely valuable and fast!

Oh, did I mention these videos will also be supplemental to his Book?! (the book isn’t out yet, but I’ll pick it up as I know how valuable it will be!)

So, I highly encourage you to check this out, if you’re familiar with ESX and haven’t touched on vSphere yet, if you’re a master of VMware and want some good insight and value, and most importantly, even if you’ve never touched VMware, or virtualization before – You’ll get a –lot- out of this! So check it out, this is some of the best content you can pay for – and it’s free!