Our 5 year ELA for VMware went from 1.5M USD to 12M USD. I work in Higher ed.
Our Hyper-V environment came online a few months ago. It was already included with our ELA with Microsoft so we were able to splash out a bit for some higher tier support.
Granted, we have a separate team working on "genAI stuff."
We started converting virtual machines about 3 weeks ago and we've gotten through ~500 of about 3500 or so.
Our grant based HPC environment is just moving back to bare metal. The VM conversion is just for ad-hoc HPC and then all of our general infrastructure. Some of our larger application servers (SAP Hana) are possibly staying on VMWare if SAP won't support them on Hyper-V.
This is a hot topic among some of my nerdier SME friends, and our conclusion is that the major players are HPE and Nutanix. At least from our perspective over here in Sweden.
HPE did a big brain move to support multiple hypervisor backends with their own frontend. The only way to go forward imho.
I'm using Proxmox at my current $dayjob, and we're quite happy with it. I come from a big VMware shop and I think most businesses could easily replace VMware with Proxmox.
I think Proxmox should just launch an Enterprise contract, regardless of the cost, just have one. Because right now I think the main obstacle halting adoption is their lack of any Enterprise SLA.
On a personal level I would love to see KubeVirt, or Openshift with KubeVirt, take over more. It just seems like a genius move to use the already established APIs of kubernetes with a hypervisor runtime.
Proxmox is about to miss their window of opportunity here. They are uniquely positioned to take on VMWare, but their outfit seems like a fairly tiny and conservative company with zero ambition to take on the world, so to speak.
If they aren't interested in that business, then it isn't really a window of opportunity for them. In fact I respect a company that chooses to not pursue business opportunities that don't fit their goals, and instead focus on being a good fit for the market they are in. Growth isn't the most important thing.
I've been at multiple companies that wasted millions courting large enterprise contacts only to not make a single sale. It does make the sales update more exciting though—if we just get this one sale…
I can't blame any company for wanting to stay out of that market.
It helps that they’re not a publicly traded company [A]. If you’re beholden to stockholders, you’re beholden to a market demanding growth at all costs. Even if the leadership at the moment wants this stable strategy, all investor pressure tends toward aggressive moves to the contrary.
[A] probably? I couldn’t conclusively determine this, and I’m not an expert
Indeed they are not. They are a Austrian GmbH, which is a special kind of company form that is not really comparable to a e.g. British or American Ltd.
Long story short, for being a publicly traded company, they would need to "transform" to an AG ("Aktiengesellschaft", where "Aktie" means "share of stock").
A GmbH is nothing “special”, its basically the most used corporate structure in Austria (and Germany).
Its a limited liability structure and most businesses from small to large that have private shareholders use it (Red Bull or Porsche Holding are GmbHs for example)
You can have a look at XCP-ng. They have the expertise and it's originally a fork of Citrix XenServer however they are completely on their own feet now delivering some interesting advancements.
Access to Enterprise repository
Complete feature-set
Support via Customer Portal
Unlimited support tickets
Response time: 2 hours* within a business day
Remote support (via SSH)
Offline subscription key activation
You asked for an Enterprise SLA. Not all Enterprise SLAs are 24/7. IM(Professional)E, most are not 24/7.
> What's a business day?
From the FAQ on the page linked to by guerby:
What are the business days/hours for support?
Ticket support provided by the Proxmox Enterprise support team is available on Austrian business days (CET/CEST timezone) for all Basic, Standard, or Premium subscribers, please see all details in the Subscription Agreement.
For different timezones, contact one of our qualified Proxmox resellers who will be able to offer you help with Proxmox solutions in your timezone and your local language.
Check out the actual FAQ entry to chase down the links embedded in those words that I'm too lazy to try to reproduce.
> ...definitely [a] 24/7 SLA is what Proxmox needs to break into the enterprise sector I have experience with.
Well, their FAQ says:
For different timezones, contact one of our qualified Proxmox resellers who will be able to offer you help with Proxmox solutions in your timezone and your local language.
Consulting the list of resellers that that page links to finds one that blatantly advertises 24x7 support, and it's likely that others will offer it if asked. See [0].
That's excellent. The info a bit buried, and the lack of it front and center probably scares away a lot of clients.
Proxmox needs to better their reputation right now if they're going to be counted as a contender, and burying the fact that you can have 24/7 SLA is not a good way of improving that reputation of being mostly for the homebrew crowd.
> ...the lack of it front and center probably scares away a lot of clients.
I've done on-call enterprise support for the products that I and the folks I worked with maintained and extended. We were whatever tier "the folks who work on the product" is. [0]
I can pretty authoritatively tell you that the folks who are scared away by a 60->120 second search to answer the question "It looks like this one vendor doesn't offer 24/7 support, but they do offer a list of certified support vendors. Do one or more of the vendors they certify to provide support and training provide 24/7 support?" are the sort of folks you rather don't want as customers.
> You asked for an Enterprise SLA. Not all Enterprise SLAs are 24/7. IM(Professional)E, most are not 24/7.
Any serious enterprise software or hardware company absolutely has a 24/7 support option. They all have a base option that is not 24/7 for a significantly lower price.
There’s no way you’re replacing VMware in any company of any size without 24/7 support.
Microsoft seems perfectly capable of advertising 24/7 support whilst never managing to call back within 24 hours on business crippling sev1 tickets. Just look at how often someone on /r/sysadmin is shocked to find this is the norm.
I know thst youre right about the wording turning off orgs but I do wonder when the biggest enterprise organisation can barely offer it in practice what really is the show stopper for business.
The trap is you need Microsoft support training & strategy. If you buy unified and open a sev a, they just fuck around and assign an engineer from Antartica who works from 3AM-6:20AM Mongolian time, then reassign at 6:19AM to dude in Japan to reset their 2 hour SLA for the incident manager. In general, if you are big, you're better off buying Premier from a partner, and declaring a crit sit. Many issues are fixable by less dumb third party L2 techs, and you can leverage the partner's juice with Microsoft to get somebody. You have the ability to inflict real pain on the reseller, but all Microsoft will do for a strategic customer is send some VP of something to apologize profusely at great length and suggest the more staff meet with your TAM/CSM so they can get a dramatic reading of a powerpoint. Companies like this only understand pain, so you need leverage.
Microsoft is uniquely bad at this type of stuff. Anyone committing serious infrastructure where bad things are gonna happen when it goes down is insane for using HyperV. But you're also insane expecting a small reseller of some small company to pull your chestnuts out of the fire.
Anyone who is really committed to their infrastructure will not build it on top of highly proprietary stuff where you have 0 visibility into what's actually happening so you can only hope that somebody fixes it sustainably, in a reasonable time frame and permanently.
With open source, if you have the right people, you can find/ bisect down to the commit and function where the problem is exactly, which speeds up the remedy immensely. We have done such a thing with backup restores from the Proxmox Backup Server. The patches are now in Proxmox VE 9.0 because the low hanging fruit problem was actually with the client code not the Proxmox Backup Server.
It’s not about the support. It’s about the blame shifting. The CTO has a piece of paper which means he’s no longer accountable. Gartner says they are good, the occasional sales lunches are expensive, and the golf game can continue.
The show stopper is explaining to your CEO that you don’t have 24/7 support on a piece of software that’s core to the business.
You can explain away horrible 24/7 support and keep your job. Not so much if you buy something that doesn’t even offer it and you have a hard outage at 5pm on a Friday.
I can second technion's observations about Microsoft's "24/7" support SLA.
Anyway, as the FAQ answer that I quoted mentions, there are plenty of qualified Proxmox resellers who offer support for folks who are dissatisfied with what is offered by Proxmox Server Solutions GmbH. One reseller explicitly advertises 24x7 support [0]. I expect others would offer 24x7 support if you asked, but don't see the need to advertise it up-front.
Formally, yes, they are 24/7. However, getting the expert you really need to solve the issue, that can be much harder on weekends. Sometimes it only amounts to handholding till Monday.
Yes, I'd think Openshift with Kubevirt would be positioned to move in. Lots of Openshift in some of the sectors I've worked with so seems like a natural expansion.
I forgot about MSFT's ability to bundle Hyper-V though which seems to come up in this thread a lot.
Broadcom turned up the heat on our pot fast enough that we’re evacuating over to proxmox. I and several others in IT had run it at home for a while, so when Broadcom made the definite losses to continue on VMWare far higher than the likely losses from any migration outage, it became a no-brainer to migrate.
Migrating part of the farm and A/B testing shows good results and we’ll be able to complete it in-place before the next pizzo payment to Broadcom is due.
Thanks for the nudge, Broadcom! As far as I’m concerned, Broadcom and Oracle are tied for first on my “do not voluntarily do business with” list. Equaling Oracle in this way is a feat…
The CEO explicitly said the only customers they're interested in is the 600 customers who are responsible for 80% of their revenue. The are actively trying to shed all others; it's simply not worth the overhead costs otherwise.
That's one way to do business. Run tour product into obscurity so there are few qualified managers and admins in the market after 10 years, and everyone has become familiar with cheaper and now more prevalent alternatives.
There are customers that cost more to keep than you earn - firing those customers is a useful activity at all levels of business. However Broadcom has a lot of products that we would generally expect to scale to a lot of customers and they are acting like they are bespoke things that don't scale. Their business model doesn't match what their products generally need.
I expect things like VMWare to disappear in a few years because some competitors taking those other customers will slowly make things better until those still buying VMWare go elsewhere anywhere because it is good enough. There is only so long they can sell bundles of everything before customers decide the total price is too high - so it might not even be VMWare's competitors are good enough so much as everywhere else is good enough and VMWare needs to be tossed too.
VMWare was for sale in the first place because it is obviously not going to be the case in 10 years that every dentist’s office and car dealership needs 3-10 instances of Windows Server in the closet. It makes sense to sell to someone who can extract as much value as possible from it on the way to zero.
I work for an MSP, mostly with small to medium companies. Licensing costs went up a ton when broadcom acquired vmware. They went up a ton more this year with minimum core counts, current licensing costs are roughly $20k a year minimum. They might hike the price again, even medium businesses that see some value in avoiding an expensive migration want to avoid this uncertainty. Basically they don't want to deal with small and medium sized businesses. I'm sure large businesses are facing price hikes too but I don't have experience with that.
If you are on a perpetual license you can put the management vlan on a network not connected to the internet if it wasn't already and realistically this buys a few years. You will not be able to patch, eventually auditors will not accept that. For the rest not on perpetual licensing, when the licensing expires you will not be able to power on machines, if they reboot they stay off.
About half of clients we are migrating to hyper-v. Most are already running windows servers. There are some differences but hyperv covers the important features and the licensing is basically already included. Beeam makes the virtual to virtual move a lot easier, this is what most of our customers use for backups
For a good chunk they are migrating to azure or another hosted environment. If you don't have a main office with a file server or some more demanding line of business apps this is a pretty easy move.
A few are going to nutanix. Or more of expanding nutanix.
My organisation went down the Nutanix path with about 1/4 of the DC about 18 months ago. They're now just finalising the move away feom Nutanix. From a server and dev admin point of view we had really odd VM behaviour, poorer than expected process performance, and random instability that just couldn't be resolved. I believe other system owners had similar and that the VM admins had their own range of oddities to track down. Something behind the scenes was the catalyst for change away in a short period of time.
Check out libvirtd based stacks, because that's what's supported by upstream Linux.
Some shops here migrate to proxmox as a UI because of certification requirements, but I migrated some of my customers to cockpit dashboard, and some to kubernetes. It's always a matter of scale and provisioning requirements.
Cockpit is my favorite so far because it's easy to setup, but its focus isn't cluster scale, which is what most larger companies need. You have to setup basically two cockpit variants: the webui and lots of cockpit server daemons (aka libvirtd on remote machines). The webui then uses SSH to login to other machines to manage them (e.g. via the known_hosts file on the webui server). [3]
Proxmox is pretty old and Perl, but it's doable. Usually storage clustering is a bit painful because you need something on a filesystem layer like ceph clusters.
There's also openshift but no idea if that is an IBM/RedHat lock-in as well, so the SMEs didn't want that risk.
I'm with a block storage vendor that works with a lot of companies migrating off VMware, and the diversity of KVM-based cloud management platforms we're seeing is fascinating. We have customers moving to OpenNebula, CloudStack, Proxmox, OpenStack, HP VME, Oracle Virtualization, and even some homegrown solutions. The common thread is that they're all looking for a storage backend that is not tied to a specific hypervisor and can deliver predictable high performance. The beauty of the KVM ecosystem is the freedom to choose the best tool for the job, and that extends to the storage layer. A good software-defined block storage solution should have the features (data migration, disaster recovery) and capabilities to make the transition away from VMware as smooth as possible.
Kind of sad seeing businesses getting screwed by closed source proprietary software, then making the same choices all over again.
Nutanix also seeing huge demand.
Not everyone is repeating their mistakes, with Proxmox and Xcp-ng seeing huge new level of business, as well, which is nice.
I'm part of the Apache CloudStack project and that too is seeing unparalleled levels of demand.
The KVM hypervisor has sort of become the de facto choice, thanks to virt-v2v tool which can help migrate VMware guests.
At PriorCo, I did a slide deck presentation of our options at the time (2023/2024) and pitched essentially three pathways: stay on VMware, move to Apache Cloudstack, or move to Nutanix. The deck was roundly ignored in favor of a lift-and-shift to AWS for remaining infrastructure.
If I were running the migration, my preferred pathway would’ve been to Apache Cloudstack. We had the expertise to pull it off, and it would’ve freed us from vendor partners. Nutanix was really only on the list purely from its technology portfolio; its lack of profitability and shifting towards SaaS for features like cost analysis meant that we’d be moving into a similarly bad situation as VMware at the time (wholly beholden to their business priorities instead of our own), which I didn’t care for.
There’s a lot of options out there, most of which are built atop either KVM/QEMU or OpenStack. Virtuozzo’s offerings impressed me, but the lack of a “comprehensive” product was a turnoff. Oxide was incredibly interesting from a simplicity and integration perspective, but the appetite wasn’t there to try a startup’s product. Microsoft and Oracle were both ruled out due to higher costs and more onerous licensing than VMware/Broadcom, while IBM/OpenShift were ruled out as our private cloud estate was 100% VMs with only ~20% of our internal products capable of containerization support.
The biggest advice I can give is to understand your workload today, and determine options accordingly. Everyone’s pitching K8s and containers, but if your estate is majority VMs, then a lot of those options just aren’t worthwhile.
Would you be considering Oxide in the future with their recent $100M Series B funding round, their recent sales to clients like Lawerence Livermore National Labs, and the recent AWS outage in mind?
1. Proxmox cannot even join a network using DHCP requiring manual IP configuration.
2. Disk encryption is a hell instead of checkbox in installer
3. Wi-Fi - no luck (rarely used for real servers, but frequently for r&d racks)
Of course, it is a Debian core underneath and a lot of things are possible given enough time and persistence, but other solutions have them out of the box.
Proxmox wasn’t considered because of the audience (leadership) and Proxmox’s perceived market (SMBs/homelabs). I couldn’t even get them to take Virtuozzo seriously, so Proxmox was entirely a non-starter, unfortunately.
FWIW, I use Proxmox at home. It’s a bit obtuse at times, but it runs like a champ on my N100-based NUCs.
Mostly bitching to corporate IT to make it possible to use alternative tools and workflows.
Not kidding, that’s the main blocker. We have the DevOps knowledge on our team to go to containers, prepackaged dev environments, etc. But corporate cyber tends to respond to our requests to discuss cyber policy and escalate via proper channels with “sorry that’s against policy”.
This is not my experience at one company but multiple good, name brand companies that generally do good engineering and software work.
Because the old “trusted” vendor is now absurdly expensive and switching to another one helps increase profit? E.g. why did organizations switch from bare metal to virtualization in the first place?
Because those policies didn’t account for the workflows of engineering and dev teams. And I’m not even really asking for them to trust 3rd parties but to instead have a workflow to escalate and petition tools and workflows to become supported in house.
For example Docker Desktop being disallowed with vigor for Windows machines because it’s a virtualization tool. But Docker is fine for Linux users. And confirmed it’s not a licensing or purchasing issue.
> We have the DevOps knowledge on our team to go to containers, prepackaged dev environments, etc.
This is lovely to strive towards and going all in on containers (albeit not with Kubernetes) has worked out great for where I work; their resistance to the approach sucks, I'm sorry you have to deal with that. Hope it works out in the end.
Im in the same boat, and it sucks. CyberSec rules the roost but have little to no care or knowledge of good DevOps, or process management considerations, so the result is tpu wind up talking to a human firewall whose response is always "no.". Organisationally we wonder why nothing improves.
If you want a solution that "just works", is simple to manage and can even extend to public cloud infrastructure when needed I can really, really recommend Nutanix. Migrating from VMWare is also really simple, they have a tool which makes that very easy. Do note that they use a heavily modified KVM as hypervisor.
We're driving as many apps as possible to containers, and replacing most of our virtual infrastructure with Talos Linux [1] which is a reasonably hardened OS dedicated to Kubernetes hosting. I strongly recommend using the terraform provider to help manage at scale. The docs seem a little sparse for beginners, but if you already know kubernetes concepts, it's pretty easy to pick up. If you know flux style gitups talhelper+sops is far better than naked talosctl. We're also trying to migrate off of IAAS provided kubernetes and migrate to talos within instances. It's an effort to reduce dependency on specific IAAS while also minimizing number of technologies we need to support.
We're driving anything that cannot be containerized to lift and shift to IAAS and forcing the app owners to pay for it out of their budget as motivation to modernize. They have to explain to the board why their spending increased and they are still on legacy.
Seeing a lot of Nutanix especially for VDI/Citrix heavy workloads or typical 3-tier applications. HP VME is also becoming a thing as an almost drop-in and VERY cost effective alternative to VMWare. In telco Openstack is still king AFAIK.
Non-Profit Liberal Arts Higher Ed here. Trying to move to the Azure Hybrid Cloud as fast as possible. Going to have to eat one more year of 170% price hike in our current VMware contract.
Heard from a peer school on the east coast that had to sign a 600% hike in their most recent contract. Absolutely evil.
I would not call it evil. Orgs are _voluntarily_ making the choice to lock in for expensive useless proprietary software by either being stupid or - actually evil - receiving kickbacks from those proprietary companies.
Every manager signing off the contract with vendor lock in should understand possible repercussions. Its not a rocket science. Yep they will screw you along the way with prices. Yea, that sales rep saying they won't is a lying bastard that can't care ess about your business the moment your pen raised from your signature in contract.
I understand all of that. We made a calculated risk, sure... back in 2007, and we've been happy to pay since. Only now, in one year, do the screws turn so tightly.
But at least shareholder value is going up somewhere.
In my sphere most companies are going to either Hyper-V or the cloud. Hyper-V kinda won by default as a lot of orgs already had Windows Server licenses.
I don’t really consider OpenShift in the same category. VMWare and its enabling software such as vSphere and vCenter are in another category than OpenShift to the point that there is a symbiotic relationship between VMWare and Dell in the corporate/enterprise setting
I can confirm this, from my experience. Many organizations have foregone ms in lieu of redhat and oracle, and redhat is slowly injecting openshift where they can fit it.
For Mac/Linux/Hybrid-Cloud shops -> "Incus" (can also host macOS and Windows instances) for most work and "Firecracker" for specialized appliances.
For Windows shops (you sick bstrds :D) -> Hyper-V
RE: Proxmox, not a hater, but errtime I read that name, I picture "Mike Myers Dieter". Seriously tho, the best they could do for themselves would be to make ProxMox into a UI for the Incus REST API (and their legacy backend) then repackage their enterprise offerings as add-ons for Incus (for email, DRS, etc.) BTW, the first person to release an "I use this exclusively everyday for months before releasing it on Github" gesture-controlled WebGL/Audio and WebXR based UX with agentic Incus REST API sensor analysis and settings controls, will win a valuable prize.
-> Mind earlier warnings that many people in the Industry these days tend to follow each other around on forums creating a sort of "feedback loop" so bad trends (eg, PHP) takeover the game. <cue https://youtu.be/uPWQfAv_qBQ>
I'm seeing a bit of everything: renegotiating (which Broadcom doesn't really do), optimizing and consolidating hosts (to lower costs), public cloud migration (which is why I see the most given my line of work, but may not represent everything), forays into other hypervisors, etc.
Proxmox may come to many an HN visitor's mind (and I use it myself extensively, all my home services run on it), but it actually doesn't have a lot of enterprise features and isn't a drop-in replacement.
Depends on what VMware product you’re using. I’m still stuck on Workstation Pro because Hyper-V takes over my computer and slows down anything virtualized that isn’t Windows.
For servers, there’s things like Proxmox that are super versatile and can be deployed to a 1U server in a few hours. And if whatever you’re virtualizing exists mostly in the application/user layer and doesn’t touch lower level APIs (eg raw TCP), Docker/Portainer is even better (no Kubernates required!).
Hi, im a presales engineer for OVHcloud, a cloud provider. OVH is a "pinacle" VMware partner, we are a huge reseller of VMware licenses, and were able to secure a 3-5 years contract with broadcom with only 5 to 15% of price increase.
Of course, the clients I talk with are mostly interested in moving to the cloud or are already in the cloud, so it won't be applicable to you if you'd stay on-prem. I wanted to share my experience.
For some clients, change of technology would be more expensive than paying a bit more to VMware. You'd have to re-train half your IT department, and the migration could be long, risky and complexe. So in this case, a lift and shift move-to-cloud can be competitive, and frankly a serious option.
I see a lot of projects with Nutanix, but you'd be surprise of the price, which is almost the same as VMware. Nutanix comes with way more features than vanilla vsphere which explains the cost increase. You'd have NSX and vSAN packaged, plus replication features. Nutanix offers a great alternative. Actually, OVH proposes Nutanix too, so we can be agnostic and have a sort of leverage over Broadcom.
Some clients are ok to move to public cloud (equivalent AWS/Azure..). The smaller the infra the easier it is. It can be very cheap at OVH. Also it's great if you do containers because of the universal nature of them, they are easy to migrate.
However, If the client has a lot of Windows Server, the cost is actually higher (at OVH the price of the Windows Server Licence is higher than at Azure..) because we cannot leverage the Windows Datacenter licenses
The cheapest viable option is to go to Proxmox on Baremetal servers. The features are close to a standard vSphere environnement. The lack of enterprise support is the thing that stops most clients to do this move.
All good points, except perhaps the cost of the move. With VMWare licences anecdotallty increasing ~8x, the cost of moving could well be recovered quite quickly. Organisations tend to think and react strategically, and this will mean 3-5 year (or more) financial projections for major projects. If a $1m VMWare annual bill is now $8m, over 5 years thats $5m vs $40m. A change to a $1m annually cluster isn't going to cost you $35m, so you should definitely look at changing to minimise your expenses, assuming you get the support etc that you need elsewhere.
Ah, 5x? At $WORK, the low code tool vendor that is used to build the monolith (and that of our sister company) is bought by a private equity firm. Our sister company will face a 7x increase. Another fun thing is that the license is based on a percentage of licensing cost to their customers.
Their game is clearly to squeeze very hard for a few years, and then deprecate the product. I can't imagine that there are companies that are fine with such price hikes.
I've just enjoyed the realization that I attended a meeting where our VMware support rep used "Just think about how much money using bare metal hardware is costing" as an argument to my management during that meeting after these price increases were becoming common news.
Datacenter edition will give you the right to run unlimited windows server VMs on each host. It's a really hard deal to pass up if you're mostly a Microsoft shop.
Most of the time I don't need a full VM and run inside a container via systemd-nspawn. This runs on the existing kernel instance but isolates everything nicely. Mainly use it for complex builds so they don't bork my system.
I don't think Proxmox is anywhere near ready for that sort of shift. It's interesting what a big hole in the market VMWare is leaving and nothing quite fills it. OpenStack is the closest, but way more complicated than VMWare, and doesn't work at all for smaller deployments.
Former VMware service provider here. We switched about 1,000 VMs from VMware to Proxmox VE a couple years back, and it's been one of our best moves. We run Proxmox hyper-converged now, love the built-in VM firewalls, solid backups with PBS, and Ceph storage. The paid subscription gives us reliable updates, too.
Hardware requirements for Proxmox are way more flexible (and cheaper) than VMware, in our experience. Plus, more MSPs are jumping on the Proxmox train, so the support scene is growing fast.
Proxmox isn’t a perfect VMware clone, but it covers a lot of ground for service providers or mid-sized setups that want reliability, flexibility, and lower costs.
I’m not sure that’s true for larger scale installs but small scale VMware installs are probably less easily replaced by solutions that are also as well supported and have a path for expanding.
Doing a head-on VMware takeout path hasn’t been a good business strategy for companies that tried it.
For small SMBs using Proxmox is reasonably ok-ish. Running in production for 2+ years already our customers are quite happy. We also sent some patches to Proxmox for other much larger clients...
Seems to me everyone is going to Proxmox, Xen, or hyperv. Honourable mention but technically apples to oranges. Kubernetes and docker.
Redhat openshift? I have never heard of anyone trying to go to it. I like how they try to mix vms and containers... but i dunno it seems to suffer from the standard red hat problem.
#1 reason people don't wholesale move off of VMW is 1 word, RISK. Too much risk to move legacy applications of enterprises without a lengthy project to move. This that do and start to, are choosing KVM variants, RH Open shift Virt, or Hyper-V. Promos would be good if it had enterprise support.
Nutanix is just as expensive and also a locked in option.
XCP-ng and XCP-ng Center are excellent no-cost solutions. It can't do highest end VMware functions such as the voodoo of moving a running system to a different VM with no interruptions, but it can do a lot.
Don't a bunch of the newer tools that wrap Virtualization.Framework (which itself wraps/builds on the lower level Hypervisor.framework) already support this?
We participate in a forum where people regularly recommend Hetzner. "Enterprise Grade" reliability is rarely a concern for the folks here. They have no limit of elbow grease to make sure all the shoddy solutions work together. And if they don't, they don't receive enough traffic for the difference to show.
could you please tell what's the difference between VirtualBox and VMWare ESX? I mean what's the VirtualBox lacking as compared to ESX? I've never used an enterprise VMWare so I can't imagine what's different there? Especially if Hyper-V is proposed as an alternative...
--- EDIT ---
I've made some research and now understand that Hyper-V runs alongside the Windows Server (i.e. it is not an in-windows app). Therefore it would be hard to achieve the ESX performance close to ESX or Hyper-V even when deploying on linux. Or wouldn't it? Maybe considering a partition as disk for virtualbox? Idle Linux overhead isn't this much, is it?
Parallels and virtualbox are designed for the end user ad hoc running a few VMs on their client system. The VMWare equivalent was VmWare Workstation, not ESX. ESX, Proxmox etc. are about orchestrating fleets of VMs running on clusters of hardware
It's not really about performance. Imagine comparing AirDrop to a SAN on the basis both provide file sharing functionality. The topic wouldn't be performance, it'd be how they have entirely different use case goals that just happen to both end up using file sharing as one part in achieving that goal.
In the case of ESXi it's about the clustering, filesystem/network virtualization, management and orchestration for hundreds of servers, disaster recovery, enterprise security/software integrations, and so on. That VirtualBox requires a client OS is just a footnote in the comparison of functionality.
Hard to say much given we weren't given much info on how it'd be used.
E.g. Parallel's is only useful for people looking to run VMs locally on their Mac, but Hyper-V can be anything from that for a Windows PC to a full-blown headless hypervisor cluster with HA, shared volumes, replication, etc.
For several of the common categories, these are my takes:
- Traditional Enterprise: Nutanix [paid] if money is available, otherwise Hyper-V [paid] if a large Microsoft contract is already in place. If neither fit: fall through to acting like an SMB.
- SMB/Modern Mid-Sized Enterprise: Cloud [paid] only and/or Proxmox [free/paid]
- Tech Company: Doesn't matter, they'll do whatever sounds cool that year and make it work well enough
- Home Lab: Proxmox [free/paid]
- Windows PC: Hyper-V [free w/ Windows] (it's meh, but it's integrated - doubly so if you plan on using WSL on the side).
- Mac PC: Parallels [paid] if you need a GPU accelerated Windows guest, UTM [free] otherwise.
- Linux PC: QEMU+KVM [free], the choice of (optional) GUI client is up to preference.
Some extra notes by solution:
- Nutanix: Enterprises were staring to use this more and more even prior to the VMware sale. It's definitely the spiritual successor of traditional VMware usage in the data center. A bit less full of themselves, for now at least, than VMware ever managed to keep themselves (IMO).
- Proxmox: Has a bit of a habit of feeling like it always ends up a little broken by the time you've used an install/cluster for 6 months, but is by far the best option for the homelabber type use case (even ignoring that it's free as a reason). It's basically like someone configured KVM with what you want to be able to just (try to) use it without thinking about what's underneath, while still having access to the underneath to un-stick it in certain situations. Also does host-native containers! I never did have the guts to pitch my company try to run anything production on a cluster, but they do have reasonably priced support plans and advanced feature tiers for that.
- Parallels: Kind of sucks for the price, but there isn't anything else on macOS with the same GPU acceleration for Windows.
- Hyper-V: I think this is mostly still around because it helps Microsoft stay sticky at companies when the yearly renewal comes up. That said, it's alright - and it's also integrated into Windows in some pretty nifty ways for local use these days.
- UTM: Fantastic QEMU client for macOS, worth giving a few bucks for even though it's free.
I hate virtualization with a passion, because it's just more crap that doesn't work written by people that shouldn't touch computers.
In theory, it's great. In practice, if you need to get "support" from someone else, it's not so great anymore, as all these companies have been discovering.
I would use VM technology if whoever wrote it would provide me with a contract saying that if anyone were to find just one program that would crash their VM (while not crashing a real machine) or miscompute, that I would get a billion dollars.
To answer your question: I was smart enough to never use it in the first place.
The only kind of person who would say that they hate virtualization is a hardware vendor. Because of conflict of interests.
In this light, Broadcom, a hardware vendor, who buys a popular virtualization product does a "smart" move - it supposedly eliminates the very thing that eats away their profits. But it only looks smart to the vendor itself. For everyone else, the move looks unprofessional and incompetent.
I never sold hardware, so there is that. I am rather anal about correctness and virtualization products are really complicated and as such almost nobody gets them to work reliably (if there is just one known bug, it's trash, IMO).
But, even if you restrict it to 'x86 virtualization', the alternative for the current crop of 'enterprise' OS environments is ...server sprawl. I'm a big fan of discrete hw for some things, but it can be a hard sell for everything.
The primary alternative to full system VMs is containers (or jails, zones... whatever your OS might call them). You don't need to go server sprawl or VMs as the only two options.
If you're so ignorant of the space you think virtualization is 20 years old, you're too ignorant to make proclamations about what anyone else should do.
Our 5 year ELA for VMware went from 1.5M USD to 12M USD. I work in Higher ed.
Our Hyper-V environment came online a few months ago. It was already included with our ELA with Microsoft so we were able to splash out a bit for some higher tier support.
Granted, we have a separate team working on "genAI stuff."
We started converting virtual machines about 3 weeks ago and we've gotten through ~500 of about 3500 or so.
Our grant based HPC environment is just moving back to bare metal. The VM conversion is just for ad-hoc HPC and then all of our general infrastructure. Some of our larger application servers (SAP Hana) are possibly staying on VMWare if SAP won't support them on Hyper-V.
This summer sucked big time but we'll make it.
I think the following quote summarizes Broadcom's strategy with regards to VMWare:
"If you are a Global 2000 company, VMware wants your business — from the rest, not so much,”
[1] https://www.networkworld.com/article/4053783/broadcoms-vmwar...
This is a hot topic among some of my nerdier SME friends, and our conclusion is that the major players are HPE and Nutanix. At least from our perspective over here in Sweden.
HPE did a big brain move to support multiple hypervisor backends with their own frontend. The only way to go forward imho.
I'm using Proxmox at my current $dayjob, and we're quite happy with it. I come from a big VMware shop and I think most businesses could easily replace VMware with Proxmox.
I think Proxmox should just launch an Enterprise contract, regardless of the cost, just have one. Because right now I think the main obstacle halting adoption is their lack of any Enterprise SLA.
On a personal level I would love to see KubeVirt, or Openshift with KubeVirt, take over more. It just seems like a genius move to use the already established APIs of kubernetes with a hypervisor runtime.
Proxmox is about to miss their window of opportunity here. They are uniquely positioned to take on VMWare, but their outfit seems like a fairly tiny and conservative company with zero ambition to take on the world, so to speak.
If they aren't interested in that business, then it isn't really a window of opportunity for them. In fact I respect a company that chooses to not pursue business opportunities that don't fit their goals, and instead focus on being a good fit for the market they are in. Growth isn't the most important thing.
I've been at multiple companies that wasted millions courting large enterprise contacts only to not make a single sale. It does make the sales update more exciting though—if we just get this one sale…
I can't blame any company for wanting to stay out of that market.
It helps that they’re not a publicly traded company [A]. If you’re beholden to stockholders, you’re beholden to a market demanding growth at all costs. Even if the leadership at the moment wants this stable strategy, all investor pressure tends toward aggressive moves to the contrary.
[A] probably? I couldn’t conclusively determine this, and I’m not an expert
Indeed they are not. They are a Austrian GmbH, which is a special kind of company form that is not really comparable to a e.g. British or American Ltd.
Long story short, for being a publicly traded company, they would need to "transform" to an AG ("Aktiengesellschaft", where "Aktie" means "share of stock").
A GmbH is nothing “special”, its basically the most used corporate structure in Austria (and Germany).
Its a limited liability structure and most businesses from small to large that have private shareholders use it (Red Bull or Porsche Holding are GmbHs for example)
As I understood it, GmbH is like an LLC but with more paperwork.
You can have a look at XCP-ng. They have the expertise and it's originally a fork of Citrix XenServer however they are completely on their own feet now delivering some interesting advancements.
Xcp-ng seems better positioned with a familiar vmwareish experience.
Lifestyle business, the antithesis of Y Combinator.
They're a european business. I don't think they're interested in the stress involved in selling to enterprise.
https://www.proxmox.com/en/products/proxmox-virtual-environm...
"Premium"
>Response time: 2 hours* within a business day
What's a business day? I wouldn't call that a 24/7 SLA.
> I wouldn't call that a 24/7 SLA.
You asked for an Enterprise SLA. Not all Enterprise SLAs are 24/7. IM(Professional)E, most are not 24/7.
> What's a business day?
From the FAQ on the page linked to by guerby:
Check out the actual FAQ entry to chase down the links embedded in those words that I'm too lazy to try to reproduce.I should have been more clear then, definitely 24/7 SLA is what Proxmox needs to break into the enterprise sector I have experience with.
It's kind of frustrating because it's such a tiny detail that could make them a real contender in this new power vacuum.
> ...definitely [a] 24/7 SLA is what Proxmox needs to break into the enterprise sector I have experience with.
Well, their FAQ says:
Consulting the list of resellers that that page links to finds one that blatantly advertises 24x7 support, and it's likely that others will offer it if asked. See [0].[0] <https://news.ycombinator.com/item?id=45637767>
That's excellent. The info a bit buried, and the lack of it front and center probably scares away a lot of clients.
Proxmox needs to better their reputation right now if they're going to be counted as a contender, and burying the fact that you can have 24/7 SLA is not a good way of improving that reputation of being mostly for the homebrew crowd.
> The info [is] a bit buried...
No. It took me less than 60 seconds to find it.
> ...the lack of it front and center probably scares away a lot of clients.
I've done on-call enterprise support for the products that I and the folks I worked with maintained and extended. We were whatever tier "the folks who work on the product" is. [0]
I can pretty authoritatively tell you that the folks who are scared away by a 60->120 second search to answer the question "It looks like this one vendor doesn't offer 24/7 support, but they do offer a list of certified support vendors. Do one or more of the vendors they certify to provide support and training provide 24/7 support?" are the sort of folks you rather don't want as customers.
[0] Tier 2? Tier 3? Who knows?
> You asked for an Enterprise SLA. Not all Enterprise SLAs are 24/7. IM(Professional)E, most are not 24/7.
Any serious enterprise software or hardware company absolutely has a 24/7 support option. They all have a base option that is not 24/7 for a significantly lower price.
There’s no way you’re replacing VMware in any company of any size without 24/7 support.
Microsoft seems perfectly capable of advertising 24/7 support whilst never managing to call back within 24 hours on business crippling sev1 tickets. Just look at how often someone on /r/sysadmin is shocked to find this is the norm.
I know thst youre right about the wording turning off orgs but I do wonder when the biggest enterprise organisation can barely offer it in practice what really is the show stopper for business.
They suck balls.
The trap is you need Microsoft support training & strategy. If you buy unified and open a sev a, they just fuck around and assign an engineer from Antartica who works from 3AM-6:20AM Mongolian time, then reassign at 6:19AM to dude in Japan to reset their 2 hour SLA for the incident manager. In general, if you are big, you're better off buying Premier from a partner, and declaring a crit sit. Many issues are fixable by less dumb third party L2 techs, and you can leverage the partner's juice with Microsoft to get somebody. You have the ability to inflict real pain on the reseller, but all Microsoft will do for a strategic customer is send some VP of something to apologize profusely at great length and suggest the more staff meet with your TAM/CSM so they can get a dramatic reading of a powerpoint. Companies like this only understand pain, so you need leverage.
Microsoft is uniquely bad at this type of stuff. Anyone committing serious infrastructure where bad things are gonna happen when it goes down is insane for using HyperV. But you're also insane expecting a small reseller of some small company to pull your chestnuts out of the fire.
Anyone who is really committed to their infrastructure will not build it on top of highly proprietary stuff where you have 0 visibility into what's actually happening so you can only hope that somebody fixes it sustainably, in a reasonable time frame and permanently.
With open source, if you have the right people, you can find/ bisect down to the commit and function where the problem is exactly, which speeds up the remedy immensely. We have done such a thing with backup restores from the Proxmox Backup Server. The patches are now in Proxmox VE 9.0 because the low hanging fruit problem was actually with the client code not the Proxmox Backup Server.
It’s not about the support. It’s about the blame shifting. The CTO has a piece of paper which means he’s no longer accountable. Gartner says they are good, the occasional sales lunches are expensive, and the golf game can continue.
> what really is the show stopper for business.
The show stopper is explaining to your CEO that you don’t have 24/7 support on a piece of software that’s core to the business.
You can explain away horrible 24/7 support and keep your job. Not so much if you buy something that doesn’t even offer it and you have a hard outage at 5pm on a Friday.
I can second technion's observations about Microsoft's "24/7" support SLA.
Anyway, as the FAQ answer that I quoted mentions, there are plenty of qualified Proxmox resellers who offer support for folks who are dissatisfied with what is offered by Proxmox Server Solutions GmbH. One reseller explicitly advertises 24x7 support [0]. I expect others would offer 24x7 support if you asked, but don't see the need to advertise it up-front.
[0] <https://www.proxmox.com/en/partners/find-partner/all/partner...>
Formally, yes, they are 24/7. However, getting the expert you really need to solve the issue, that can be much harder on weekends. Sometimes it only amounts to handholding till Monday.
I proposed that my job move to Openshift on bare metal but there was some pushback because of missing functionnalities ?
In the end we're going with hpe.
Yes, I'd think Openshift with Kubevirt would be positioned to move in. Lots of Openshift in some of the sectors I've worked with so seems like a natural expansion.
I forgot about MSFT's ability to bundle Hyper-V though which seems to come up in this thread a lot.
Love the username.
The lack of support for SANs and ISCSI really bothers me. I like the way this style of setup works, I would like to keep doing it.
Proxmox supports both FC and ISCSI though!?
No equivalent of VMFS though.
Broadcom turned up the heat on our pot fast enough that we’re evacuating over to proxmox. I and several others in IT had run it at home for a while, so when Broadcom made the definite losses to continue on VMWare far higher than the likely losses from any migration outage, it became a no-brainer to migrate.
Migrating part of the farm and A/B testing shows good results and we’ll be able to complete it in-place before the next pizzo payment to Broadcom is due.
Thanks for the nudge, Broadcom! As far as I’m concerned, Broadcom and Oracle are tied for first on my “do not voluntarily do business with” list. Equaling Oracle in this way is a feat…
I place Broadcom higher. At least Oracle seems to want customers that pay. I can't find evidence that of Broadcom wanting customers.
The CEO explicitly said the only customers they're interested in is the 600 customers who are responsible for 80% of their revenue. The are actively trying to shed all others; it's simply not worth the overhead costs otherwise.
That's one way to do business. Run tour product into obscurity so there are few qualified managers and admins in the market after 10 years, and everyone has become familiar with cheaper and now more prevalent alternatives.
slow clap
There are customers that cost more to keep than you earn - firing those customers is a useful activity at all levels of business. However Broadcom has a lot of products that we would generally expect to scale to a lot of customers and they are acting like they are bespoke things that don't scale. Their business model doesn't match what their products generally need.
I expect things like VMWare to disappear in a few years because some competitors taking those other customers will slowly make things better until those still buying VMWare go elsewhere anywhere because it is good enough. There is only so long they can sell bundles of everything before customers decide the total price is too high - so it might not even be VMWare's competitors are good enough so much as everywhere else is good enough and VMWare needs to be tossed too.
VMWare was for sale in the first place because it is obviously not going to be the case in 10 years that every dentist’s office and car dealership needs 3-10 instances of Windows Server in the closet. It makes sense to sell to someone who can extract as much value as possible from it on the way to zero.
Reminds me of the saying that business's are either sowing or reaping investments.
Broadcom definitely seems to be in a reaping stage.
Broadcom (cloud vs on-prem conflict of interest) is sowing cloud silicon for VMware refugees.
I'd throw IBM in that list as well... Unless you literally have money to burn, I've avoided Oracle and IBM for decades.
I work for an MSP, mostly with small to medium companies. Licensing costs went up a ton when broadcom acquired vmware. They went up a ton more this year with minimum core counts, current licensing costs are roughly $20k a year minimum. They might hike the price again, even medium businesses that see some value in avoiding an expensive migration want to avoid this uncertainty. Basically they don't want to deal with small and medium sized businesses. I'm sure large businesses are facing price hikes too but I don't have experience with that.
If you are on a perpetual license you can put the management vlan on a network not connected to the internet if it wasn't already and realistically this buys a few years. You will not be able to patch, eventually auditors will not accept that. For the rest not on perpetual licensing, when the licensing expires you will not be able to power on machines, if they reboot they stay off.
About half of clients we are migrating to hyper-v. Most are already running windows servers. There are some differences but hyperv covers the important features and the licensing is basically already included. Beeam makes the virtual to virtual move a lot easier, this is what most of our customers use for backups
For a good chunk they are migrating to azure or another hosted environment. If you don't have a main office with a file server or some more demanding line of business apps this is a pretty easy move.
A few are going to nutanix. Or more of expanding nutanix.
My organisation went down the Nutanix path with about 1/4 of the DC about 18 months ago. They're now just finalising the move away feom Nutanix. From a server and dev admin point of view we had really odd VM behaviour, poorer than expected process performance, and random instability that just couldn't be resolved. I believe other system owners had similar and that the VM admins had their own range of oddities to track down. Something behind the scenes was the catalyst for change away in a short period of time.
What did you choose to replace?
Check out libvirtd based stacks, because that's what's supported by upstream Linux.
Some shops here migrate to proxmox as a UI because of certification requirements, but I migrated some of my customers to cockpit dashboard, and some to kubernetes. It's always a matter of scale and provisioning requirements.
Cockpit is my favorite so far because it's easy to setup, but its focus isn't cluster scale, which is what most larger companies need. You have to setup basically two cockpit variants: the webui and lots of cockpit server daemons (aka libvirtd on remote machines). The webui then uses SSH to login to other machines to manage them (e.g. via the known_hosts file on the webui server). [3]
Proxmox is pretty old and Perl, but it's doable. Usually storage clustering is a bit painful because you need something on a filesystem layer like ceph clusters.
There's also openshift but no idea if that is an IBM/RedHat lock-in as well, so the SMEs didn't want that risk.
[1] https://cockpit-project.org/
[2] https://www.proxmox.com/en/
[3] https://cockpit-project.org/guide/latest/feature-machines.ht...
Cockpits SSH/multiple machines feature is being deprecated though unfortunately
I'm with a block storage vendor that works with a lot of companies migrating off VMware, and the diversity of KVM-based cloud management platforms we're seeing is fascinating. We have customers moving to OpenNebula, CloudStack, Proxmox, OpenStack, HP VME, Oracle Virtualization, and even some homegrown solutions. The common thread is that they're all looking for a storage backend that is not tied to a specific hypervisor and can deliver predictable high performance. The beauty of the KVM ecosystem is the freedom to choose the best tool for the job, and that extends to the storage layer. A good software-defined block storage solution should have the features (data migration, disaster recovery) and capabilities to make the transition away from VMware as smooth as possible.
Openstack second wind was definitelly not on my 2020s bingo card. But I agree kvm solutions have a lot of momentum.
Microsoft gaining the most I reckon.
Kind of sad seeing businesses getting screwed by closed source proprietary software, then making the same choices all over again.
Nutanix also seeing huge demand.
Not everyone is repeating their mistakes, with Proxmox and Xcp-ng seeing huge new level of business, as well, which is nice.
I'm part of the Apache CloudStack project and that too is seeing unparalleled levels of demand. The KVM hypervisor has sort of become the de facto choice, thanks to virt-v2v tool which can help migrate VMware guests.
I'm seeing a fucking ton of cloud migrations to Azure and AWS from VMware. Even cloud VMs are pretty cheap compared to Broadcom licensing.
At PriorCo, I did a slide deck presentation of our options at the time (2023/2024) and pitched essentially three pathways: stay on VMware, move to Apache Cloudstack, or move to Nutanix. The deck was roundly ignored in favor of a lift-and-shift to AWS for remaining infrastructure.
If I were running the migration, my preferred pathway would’ve been to Apache Cloudstack. We had the expertise to pull it off, and it would’ve freed us from vendor partners. Nutanix was really only on the list purely from its technology portfolio; its lack of profitability and shifting towards SaaS for features like cost analysis meant that we’d be moving into a similarly bad situation as VMware at the time (wholly beholden to their business priorities instead of our own), which I didn’t care for.
There’s a lot of options out there, most of which are built atop either KVM/QEMU or OpenStack. Virtuozzo’s offerings impressed me, but the lack of a “comprehensive” product was a turnoff. Oxide was incredibly interesting from a simplicity and integration perspective, but the appetite wasn’t there to try a startup’s product. Microsoft and Oracle were both ruled out due to higher costs and more onerous licensing than VMware/Broadcom, while IBM/OpenShift were ruled out as our private cloud estate was 100% VMs with only ~20% of our internal products capable of containerization support.
The biggest advice I can give is to understand your workload today, and determine options accordingly. Everyone’s pitching K8s and containers, but if your estate is majority VMs, then a lot of those options just aren’t worthwhile.
Would you be considering Oxide in the future with their recent $100M Series B funding round, their recent sales to clients like Lawerence Livermore National Labs, and the recent AWS outage in mind?
What's the reason for not considering Proxmox?
They seriously need to invest in a well engineered multi node cluster filesystem. VMFS made VMware into the behemoth it is.
Without that your options for HA shared storage is Ceph (which proxmox makes decently easy to run), or NFS.
My 2 cents: Proxmox is too rigid. For example:
1. Proxmox cannot even join a network using DHCP requiring manual IP configuration.
2. Disk encryption is a hell instead of checkbox in installer
3. Wi-Fi - no luck (rarely used for real servers, but frequently for r&d racks)
Of course, it is a Debian core underneath and a lot of things are possible given enough time and persistence, but other solutions have them out of the box.
My Proxmox seems to use DHCP just fine by putting "iface eno1 inet dhcp" in /etc/network/interfaces
Proxmox wasn’t considered because of the audience (leadership) and Proxmox’s perceived market (SMBs/homelabs). I couldn’t even get them to take Virtuozzo seriously, so Proxmox was entirely a non-starter, unfortunately.
FWIW, I use Proxmox at home. It’s a bit obtuse at times, but it runs like a champ on my N100-based NUCs.
Mostly bitching to corporate IT to make it possible to use alternative tools and workflows.
Not kidding, that’s the main blocker. We have the DevOps knowledge on our team to go to containers, prepackaged dev environments, etc. But corporate cyber tends to respond to our requests to discuss cyber policy and escalate via proper channels with “sorry that’s against policy”.
This is not my experience at one company but multiple good, name brand companies that generally do good engineering and software work.
3rd party trust is not a joke. Why should they drop what they're doing to go and audit a new critical vendor?
Because the old “trusted” vendor is now absurdly expensive and switching to another one helps increase profit? E.g. why did organizations switch from bare metal to virtualization in the first place?
Because the old vendor started charging 10x the price
This. This always gets them. Just a matter of time.
Because those policies didn’t account for the workflows of engineering and dev teams. And I’m not even really asking for them to trust 3rd parties but to instead have a workflow to escalate and petition tools and workflows to become supported in house.
For example Docker Desktop being disallowed with vigor for Windows machines because it’s a virtualization tool. But Docker is fine for Linux users. And confirmed it’s not a licensing or purchasing issue.
> We have the DevOps knowledge on our team to go to containers, prepackaged dev environments, etc.
This is lovely to strive towards and going all in on containers (albeit not with Kubernetes) has worked out great for where I work; their resistance to the approach sucks, I'm sorry you have to deal with that. Hope it works out in the end.
Im in the same boat, and it sucks. CyberSec rules the roost but have little to no care or knowledge of good DevOps, or process management considerations, so the result is tpu wind up talking to a human firewall whose response is always "no.". Organisationally we wonder why nothing improves.
"VMware's in court again. Customer relationships rarely go this wrong", 190 comments (2025), https://news.ycombinator.com/item?id=45167239
"Proxmox VE: Import Wizard for Migrating VMware ESXi VMs", 100 comments (2024), https://news.ycombinator.com/item?id=39841363
We use Proxmox.
NVidia are pushing hard in the direction of combined accelerators and ARM CPU (i.e. DGX, Thor, Jetson, etc).
Some of the upcoming hardware hits a sweet spot in terms of performance / $ / W. It's hard to ignore.
But Proxmox is ignoring ARM. Which is a big mistake IMO
> But Proxmox is ignoring ARM.
If you're not already aware of this, then it might be worth keeping an eye on:
https://bugzilla.proxmox.com/show_bug.cgi?id=1314
Happily want to give some options:
* DIY w/ KVM/Xen
* DIY w/ OpenStack - https://www.openstack.org/
* ProxMox - https://www.proxmox.com/en/
* Paid OpenStack through one of their partners - https://www.openstack.org/marketplace/consulting/
* OpenNebula - https://opennebula.io/
* Nutanix - https://www.nutanix.com/
If you want a solution that "just works", is simple to manage and can even extend to public cloud infrastructure when needed I can really, really recommend Nutanix. Migrating from VMWare is also really simple, they have a tool which makes that very easy. Do note that they use a heavily modified KVM as hypervisor.
We're driving as many apps as possible to containers, and replacing most of our virtual infrastructure with Talos Linux [1] which is a reasonably hardened OS dedicated to Kubernetes hosting. I strongly recommend using the terraform provider to help manage at scale. The docs seem a little sparse for beginners, but if you already know kubernetes concepts, it's pretty easy to pick up. If you know flux style gitups talhelper+sops is far better than naked talosctl. We're also trying to migrate off of IAAS provided kubernetes and migrate to talos within instances. It's an effort to reduce dependency on specific IAAS while also minimizing number of technologies we need to support.
We're driving anything that cannot be containerized to lift and shift to IAAS and forcing the app owners to pay for it out of their budget as motivation to modernize. They have to explain to the board why their spending increased and they are still on legacy.
- [1] - https://www.talos.dev/
Seeing a lot of Nutanix especially for VDI/Citrix heavy workloads or typical 3-tier applications. HP VME is also becoming a thing as an almost drop-in and VERY cost effective alternative to VMWare. In telco Openstack is still king AFAIK.
Non-Profit Liberal Arts Higher Ed here. Trying to move to the Azure Hybrid Cloud as fast as possible. Going to have to eat one more year of 170% price hike in our current VMware contract.
Heard from a peer school on the east coast that had to sign a 600% hike in their most recent contract. Absolutely evil.
I would not call it evil. Orgs are _voluntarily_ making the choice to lock in for expensive useless proprietary software by either being stupid or - actually evil - receiving kickbacks from those proprietary companies.
Every manager signing off the contract with vendor lock in should understand possible repercussions. Its not a rocket science. Yep they will screw you along the way with prices. Yea, that sales rep saying they won't is a lying bastard that can't care ess about your business the moment your pen raised from your signature in contract.
But people still believe in tales aren't they
I understand all of that. We made a calculated risk, sure... back in 2007, and we've been happy to pay since. Only now, in one year, do the screws turn so tightly.
But at least shareholder value is going up somewhere.
In my sphere most companies are going to either Hyper-V or the cloud. Hyper-V kinda won by default as a lot of orgs already had Windows Server licenses.
Same here as well
hard to beat the MSFT bundle
Red Hat is offering OpenShift virtualization, which is Kubernetes with Kubevirt. So some people might just use Kubernetes with Kubevirt.
There's also Harvester "open source hyperconverged infrastructure" https://harvesterhci.io/
Or some Xen spinoff like https://xcp-ng.org/
Smaller shops are migrating to Proxmox.
I don’t really consider OpenShift in the same category. VMWare and its enabling software such as vSphere and vCenter are in another category than OpenShift to the point that there is a symbiotic relationship between VMWare and Dell in the corporate/enterprise setting
Red Hat has an implementation of OpenStack, which is more of a VMware replacement than OpenShift is.
Do not ever use openshift
Can you explain why I shouldn't use OpenShift?
I know they pulled a mini-Broadcom on us and sharply raised all our prices after our first two years of having our OpenShift clusters.
They have a dominant percentage of banking workloads at this point.
This is damning with faint praise.
I can confirm this, from my experience. Many organizations have foregone ms in lieu of redhat and oracle, and redhat is slowly injecting openshift where they can fit it.
but it's alright. getting out of openshift is rather easy.
getting out of vmware is a different story.
> getting out of vmware is a different story
Not really, wasn't difficult
I find that regular libvirt/qemu with virt-manager or cockpit front-end on RHEL/Alma/Rocky is perfectly fine for plenty of situations.
For Mac/Linux/Hybrid-Cloud shops -> "Incus" (can also host macOS and Windows instances) for most work and "Firecracker" for specialized appliances.
For Windows shops (you sick bstrds :D) -> Hyper-V
RE: Proxmox, not a hater, but errtime I read that name, I picture "Mike Myers Dieter". Seriously tho, the best they could do for themselves would be to make ProxMox into a UI for the Incus REST API (and their legacy backend) then repackage their enterprise offerings as add-ons for Incus (for email, DRS, etc.) BTW, the first person to release an "I use this exclusively everyday for months before releasing it on Github" gesture-controlled WebGL/Audio and WebXR based UX with agentic Incus REST API sensor analysis and settings controls, will win a valuable prize.
https://aws.plainenglish.io/why-even-google-is-rethinking-ku...
-> Mind earlier warnings that many people in the Industry these days tend to follow each other around on forums creating a sort of "feedback loop" so bad trends (eg, PHP) takeover the game. <cue https://youtu.be/uPWQfAv_qBQ>
| | ----> The World of Incus -> https://youtu.be/TzH2L5Bzlh8
I'm seeing a bit of everything: renegotiating (which Broadcom doesn't really do), optimizing and consolidating hosts (to lower costs), public cloud migration (which is why I see the most given my line of work, but may not represent everything), forays into other hypervisors, etc.
Proxmox may come to many an HN visitor's mind (and I use it myself extensively, all my home services run on it), but it actually doesn't have a lot of enterprise features and isn't a drop-in replacement.
This was question at a very very very slow moving org and industry I was at until about a year ago.
They went to Nutanix right before the broadcom acquisition and never looked back.
They were much happier, and HCI was very nice for k8s nodes.
croit.io, provides 24*7 enterprise support as a Proxmox Gold partner with a follow the sun support team.
Depends on what VMware product you’re using. I’m still stuck on Workstation Pro because Hyper-V takes over my computer and slows down anything virtualized that isn’t Windows.
For servers, there’s things like Proxmox that are super versatile and can be deployed to a 1U server in a few hours. And if whatever you’re virtualizing exists mostly in the application/user layer and doesn’t touch lower level APIs (eg raw TCP), Docker/Portainer is even better (no Kubernates required!).
Hi, im a presales engineer for OVHcloud, a cloud provider. OVH is a "pinacle" VMware partner, we are a huge reseller of VMware licenses, and were able to secure a 3-5 years contract with broadcom with only 5 to 15% of price increase.
Of course, the clients I talk with are mostly interested in moving to the cloud or are already in the cloud, so it won't be applicable to you if you'd stay on-prem. I wanted to share my experience.
For some clients, change of technology would be more expensive than paying a bit more to VMware. You'd have to re-train half your IT department, and the migration could be long, risky and complexe. So in this case, a lift and shift move-to-cloud can be competitive, and frankly a serious option.
I see a lot of projects with Nutanix, but you'd be surprise of the price, which is almost the same as VMware. Nutanix comes with way more features than vanilla vsphere which explains the cost increase. You'd have NSX and vSAN packaged, plus replication features. Nutanix offers a great alternative. Actually, OVH proposes Nutanix too, so we can be agnostic and have a sort of leverage over Broadcom.
Some clients are ok to move to public cloud (equivalent AWS/Azure..). The smaller the infra the easier it is. It can be very cheap at OVH. Also it's great if you do containers because of the universal nature of them, they are easy to migrate. However, If the client has a lot of Windows Server, the cost is actually higher (at OVH the price of the Windows Server Licence is higher than at Azure..) because we cannot leverage the Windows Datacenter licenses
The cheapest viable option is to go to Proxmox on Baremetal servers. The features are close to a standard vSphere environnement. The lack of enterprise support is the thing that stops most clients to do this move.
All good points, except perhaps the cost of the move. With VMWare licences anecdotallty increasing ~8x, the cost of moving could well be recovered quite quickly. Organisations tend to think and react strategically, and this will mean 3-5 year (or more) financial projections for major projects. If a $1m VMWare annual bill is now $8m, over 5 years thats $5m vs $40m. A change to a $1m annually cluster isn't going to cost you $35m, so you should definitely look at changing to minimise your expenses, assuming you get the support etc that you need elsewhere.
I use VMWare Workstation a lot for testing and it's a very good workhorse for that. I hope they won't mess that up.
Out of the loop: what's up with VMware?
Bought out by Broadcom, who realized if you increase prices by 10x and lose 75% of customers, you end with more revenue and less support costs
Hock Tan's only goal is to increase stock value. Period. At the expense of anything and everything else, stock value.
That's how he gets paid - increasing short term stock price.
"Show me the incentives and I'll show you the results" - Charlie Munger
aka fiduciary duty
That is absolutely not what fiduciary duty means, unless you missed scare quotes and a /s.
Broadcom competing with Oracle for most hated company prize:
https://www.reddit.com/r/sysadmin/comments/1jfumvw/broadcom_...
https://www.theregister.com/2025/05/22/euro_cloud_body_ecco_...
Maybe Oracle can buy Broadcom and settle it
Bought by Broadcom, now implementing classic strategy of leveraging vendor lock-in to milk customers.
The increase is massive ( I’ve heard x5 over existing contracts in some places )
Ah, 5x? At $WORK, the low code tool vendor that is used to build the monolith (and that of our sister company) is bought by a private equity firm. Our sister company will face a 7x increase. Another fun thing is that the license is based on a percentage of licensing cost to their customers.
Their game is clearly to squeeze very hard for a few years, and then deprecate the product. I can't imagine that there are companies that are fine with such price hikes.
Not only that, add dip downs in quality. For instance, VMware was famous for stuter-less graphics, now it's a 15 FPS show.
Milking customers is already a thin ice but in combination with declining quality it's a death sentence.
Fork?
Are you under the impression that VMware is free open source software?
Yes, I was, actually.
fair enough, you're one of today's (un)lucky 10,000! (https://xkcd.com/1053/)
VMWare has always been proprietary, there's been some handwringing a few times about the fact that they borrow from FOSS quite a bit.
https://www.zdnet.com/article/linux-developer-abandons-vmwar...
Broadcom bought VMware and changed the pricing.
Imagine seeing a 300% or more cost increase for no particular reason.
the reason is greed, old as time
I've just enjoyed the realization that I attended a meeting where our VMware support rep used "Just think about how much money using bare metal hardware is costing" as an argument to my management during that meeting after these price increases were becoming common news.
Hyper-V and Windows Server 2025. It ticks all the boxes, except being BigScaryCorp for some people.
Datacenter edition will give you the right to run unlimited windows server VMs on each host. It's a really hard deal to pass up if you're mostly a Microsoft shop.
How does one actually orchestrate this at scale? The only API is powershell, right?
Most of the time I don't need a full VM and run inside a container via systemd-nspawn. This runs on the existing kernel instance but isolates everything nicely. Mainly use it for complex builds so they don't bork my system.
A major european bank is about to move everything they got on VMware to Hyper-V
Linux. kvm or lxc.
Hire a couple of sysadmins who know their ass from a hole in the ground.
Will be interesting to see if large organizations move to Proxmox
I don't think Proxmox is anywhere near ready for that sort of shift. It's interesting what a big hole in the market VMWare is leaving and nothing quite fills it. OpenStack is the closest, but way more complicated than VMWare, and doesn't work at all for smaller deployments.
Former VMware service provider here. We switched about 1,000 VMs from VMware to Proxmox VE a couple years back, and it's been one of our best moves. We run Proxmox hyper-converged now, love the built-in VM firewalls, solid backups with PBS, and Ceph storage. The paid subscription gives us reliable updates, too.
Hardware requirements for Proxmox are way more flexible (and cheaper) than VMware, in our experience. Plus, more MSPs are jumping on the Proxmox train, so the support scene is growing fast.
Proxmox isn’t a perfect VMware clone, but it covers a lot of ground for service providers or mid-sized setups that want reliability, flexibility, and lower costs.
I’m not sure that’s true for larger scale installs but small scale VMware installs are probably less easily replaced by solutions that are also as well supported and have a path for expanding.
Doing a head-on VMware takeout path hasn’t been a good business strategy for companies that tried it.
For small SMBs using Proxmox is reasonably ok-ish. Running in production for 2+ years already our customers are quite happy. We also sent some patches to Proxmox for other much larger clients...
Broadcom ruined vmware for me.
Seems to me everyone is going to Proxmox, Xen, or hyperv. Honourable mention but technically apples to oranges. Kubernetes and docker.
Redhat openshift? I have never heard of anyone trying to go to it. I like how they try to mix vms and containers... but i dunno it seems to suffer from the standard red hat problem.
#1 reason people don't wholesale move off of VMW is 1 word, RISK. Too much risk to move legacy applications of enterprises without a lengthy project to move. This that do and start to, are choosing KVM variants, RH Open shift Virt, or Hyper-V. Promos would be good if it had enterprise support.
Nutanix is just as expensive and also a locked in option.
I gave a lightning talk about virt-v2v, VMware and Red Hat at DevConf last year. It's online at https://pretalx.com/devconf-cz-2024/talk/SN93LG/
XCP-ng and XCP-ng Center are excellent no-cost solutions. It can't do highest end VMware functions such as the voodoo of moving a running system to a different VM with no interruptions, but it can do a lot.
I work for a vendor covering subset of functionality from VMware.
I am seeing Nutanix the most, then Proxmox, Openshift.
For some sub products, Avi is often going to HAproxy, Aria to a combination of Terraform, Datadog (and others)
We are moving to proxmox
A mix of Proxmox and Hyper-V. Also, Apache CloudStack using KVM.
Jumping into bed with another single vendor.
You dont think enterprise IT does sensible things like have multiple vendors to avoid single points of failure.
Unfortunately there are effectively 0 alternatives to VMware / Parallels if wanting to do MacOS within MacOS.
Don't a bunch of the newer tools that wrap Virtualization.Framework (which itself wraps/builds on the lower level Hypervisor.framework) already support this?
There's even an example project to do this in code: https://developer.apple.com/documentation/virtualization/run...
What’s next best alternative (regardless of cost)?
Anything else? Which is best?Did you just propose Virtualbox as a replacement for ESX?
We participate in a forum where people regularly recommend Hetzner. "Enterprise Grade" reliability is rarely a concern for the folks here. They have no limit of elbow grease to make sure all the shoddy solutions work together. And if they don't, they don't receive enough traffic for the difference to show.
could you please tell what's the difference between VirtualBox and VMWare ESX? I mean what's the VirtualBox lacking as compared to ESX? I've never used an enterprise VMWare so I can't imagine what's different there? Especially if Hyper-V is proposed as an alternative...
--- EDIT ---
I've made some research and now understand that Hyper-V runs alongside the Windows Server (i.e. it is not an in-windows app). Therefore it would be hard to achieve the ESX performance close to ESX or Hyper-V even when deploying on linux. Or wouldn't it? Maybe considering a partition as disk for virtualbox? Idle Linux overhead isn't this much, is it?
Parallels and virtualbox are designed for the end user ad hoc running a few VMs on their client system. The VMWare equivalent was VmWare Workstation, not ESX. ESX, Proxmox etc. are about orchestrating fleets of VMs running on clusters of hardware
It's not really about performance. Imagine comparing AirDrop to a SAN on the basis both provide file sharing functionality. The topic wouldn't be performance, it'd be how they have entirely different use case goals that just happen to both end up using file sharing as one part in achieving that goal.
In the case of ESXi it's about the clustering, filesystem/network virtualization, management and orchestration for hundreds of servers, disaster recovery, enterprise security/software integrations, and so on. That VirtualBox requires a client OS is just a footnote in the comparison of functionality.
Hard to say much given we weren't given much info on how it'd be used.
E.g. Parallel's is only useful for people looking to run VMs locally on their Mac, but Hyper-V can be anything from that for a Windows PC to a full-blown headless hypervisor cluster with HA, shared volumes, replication, etc.
For several of the common categories, these are my takes:
- Traditional Enterprise: Nutanix [paid] if money is available, otherwise Hyper-V [paid] if a large Microsoft contract is already in place. If neither fit: fall through to acting like an SMB.
- SMB/Modern Mid-Sized Enterprise: Cloud [paid] only and/or Proxmox [free/paid]
- Tech Company: Doesn't matter, they'll do whatever sounds cool that year and make it work well enough
- Home Lab: Proxmox [free/paid]
- Windows PC: Hyper-V [free w/ Windows] (it's meh, but it's integrated - doubly so if you plan on using WSL on the side).
- Mac PC: Parallels [paid] if you need a GPU accelerated Windows guest, UTM [free] otherwise.
- Linux PC: QEMU+KVM [free], the choice of (optional) GUI client is up to preference.
Some extra notes by solution:
- Nutanix: Enterprises were staring to use this more and more even prior to the VMware sale. It's definitely the spiritual successor of traditional VMware usage in the data center. A bit less full of themselves, for now at least, than VMware ever managed to keep themselves (IMO).
- Proxmox: Has a bit of a habit of feeling like it always ends up a little broken by the time you've used an install/cluster for 6 months, but is by far the best option for the homelabber type use case (even ignoring that it's free as a reason). It's basically like someone configured KVM with what you want to be able to just (try to) use it without thinking about what's underneath, while still having access to the underneath to un-stick it in certain situations. Also does host-native containers! I never did have the guts to pitch my company try to run anything production on a cluster, but they do have reasonably priced support plans and advanced feature tiers for that.
- Parallels: Kind of sucks for the price, but there isn't anything else on macOS with the same GPU acceleration for Windows.
- Hyper-V: I think this is mostly still around because it helps Microsoft stay sticky at companies when the yearly renewal comes up. That said, it's alright - and it's also integrated into Windows in some pretty nifty ways for local use these days.
- UTM: Fantastic QEMU client for macOS, worth giving a few bucks for even though it's free.
That's the $XXM question.
Proxmox is a pretty solid alternative!
Production: Hyper V Dev: Proxmox
docker is the way!
I hate virtualization with a passion, because it's just more crap that doesn't work written by people that shouldn't touch computers.
In theory, it's great. In practice, if you need to get "support" from someone else, it's not so great anymore, as all these companies have been discovering.
I would use VM technology if whoever wrote it would provide me with a contract saying that if anyone were to find just one program that would crash their VM (while not crashing a real machine) or miscompute, that I would get a billion dollars.
To answer your question: I was smart enough to never use it in the first place.
The only kind of person who would say that they hate virtualization is a hardware vendor. Because of conflict of interests.
In this light, Broadcom, a hardware vendor, who buys a popular virtualization product does a "smart" move - it supposedly eliminates the very thing that eats away their profits. But it only looks smart to the vendor itself. For everyone else, the move looks unprofessional and incompetent.
I never sold hardware, so there is that. I am rather anal about correctness and virtualization products are really complicated and as such almost nobody gets them to work reliably (if there is just one known bug, it's trash, IMO).
Docker or podman your stuff? There's an image of every OS.
Virtualization is a 20 years old tech. Quit it.
Rather a lot older than that.
But, even if you restrict it to 'x86 virtualization', the alternative for the current crop of 'enterprise' OS environments is ...server sprawl. I'm a big fan of discrete hw for some things, but it can be a hard sell for everything.
The primary alternative to full system VMs is containers (or jails, zones... whatever your OS might call them). You don't need to go server sprawl or VMs as the only two options.
For many organizations & applications, containers are going to be a challenge. This is also why HyperV is a leading VMware replacement.
If you're so ignorant of the space you think virtualization is 20 years old, you're too ignorant to make proclamations about what anyone else should do.
Give a few years and every process will be virtualized on all major operating systems.
Why?