I recently had to upgrade two log insight instances for a customer from 4.x to 8.x. This was a manual process in two phases (you have to upgrade to 4.8 first). After the upgrade Log Insight was running, but we saw intermittent collection/task error messages on the vCenters. (Every instance connects to 4 vCenters).
I found 2 KB articles that solved our problem. If you upgrade it is probably worth taking a look at them. These issues were not documented in the release notes as far as I know.
The first is around WCP collection for Kubernetes. (KB 78107). This is apparently enabled by default after a 8.x upgrade ! Follow the KB to disable it in the advanced config.
The second one occurs because Log Insight can no longer update time stamps on the files it keeps for every vCenter. Follow the KB 70633 and you wil see if files have an old time stamp.
Last week we had a briefing in the vExpert Cloud Management program on the announcements for VMWorld 2020 this week. It is one of the great perks of the program. Product managers and technical marketing take time to present to the team, which is greatly appreciated!
The announcements in a nutshell are: – VMware vRealize Cloud Universal. A new Generation of SaaS. – VMware vRealize AI Cloud. Machine learning comes to your metrics. – Skyline 2.0 – vRealize Operations Cloud/ 8.2 – vRealize Automation Cloud/ 8.2 – vRealize Log Insight Cloud/ 8.2
Now let me start by sharing an insight I got during the presentation and that I verified with an insider:
It is cloud first from now on.
Yes you read that right. What do I mean by that ? Well until now VMware had a SaaS version of every cloud management product and the versions were more or less up to par. With the introduction of Cloud Universal only cloud subscribers will get their hands on AI Cloud. And only the SaaS version will allow you to reduce the infamous 5-minute monitoring interval in vRealize Operations from 5 minutes to 20s for realtime monitoring. And there are other differences.
This means you will want to rethink your cloud management strategy. Fortunately Cloud Universal is a new hybrid subscription that allows a combination of on-prem and SaaS with one license.
AI Cloud is a new offering included in Cloud Universal. It applies Reinforcement Learning to vSAN in this first instalment, continuously analysing and learning and dynamically applying actions to vSAN to optimise performance. We get a glance here of where operations management is going thanks to machine learning techniques.
Apart from updates in cost management, my beloved vROps 8.2 gets a boost in the APM front! It is now possible to integrate the app discovery from Network Insight into vROps! There is also an APM tool to integrate application data from App Dynamics, Datadog, Dynatrace and New Relic into vROps! This means a big step forward in extending full stack monitoring to applications, so that vROps becomes more and more the full stack monitoring platform, across clouds and apps in the enterprise. BUT… Note that the APM integrations are vRealize Operations Cloud only at the moment….
All in all exciting updates coming in cloud management, stay tuned for more details at VMworld 2020 this week. If you would like to discuss you can find me this week on the Orbital Jigsaw Discord server. There are chats and voice meeting rooms to discuss! If you want more info on this great community effort during VMWorld, read the blog post.
OK so you have seen a ton of pictures people post from previous VMWorld conferences, lamenting that it is an online event this year… Well the folks at Orbital Jigsaw have made a big effort to make it more of a community event with live corona-proof interactions!
Their Discord Server will be running a free community event, featuring all tracks, with Watch Parties, BoF Roundtables, and much more, for both days of VMworld in all three time zones. They have been working with the VMWorld event team and the vExperts amongst others.
I will be hanging out in the #Europe #vExperts and #Multi-cloud channels for sure and wherever else time permitting! If you want to chat with me directly there are meeting rooms available, just like the real VMworld, minus the sore feet and queues for lunch!
I am only a user promoting this great effort here. If you want to read more, here is the original blogpost. You can find a link there to sign up. To join the party, this is the address of the Orbital Jigsaw server.
I think Discord is fast becoming the communications platform of choice, I am part of a few servers already. I am a big fan.
Entries have opened again to apply for the VMware vExpert program for the second half of the year. I want to share my path and experiences in the program so far, hoping it might be helpful if you are considering applying (and you should …).
What is the vExpert program? It is a recognition by VMware for people that promote the company and products in certain ways. Evangelists you might call it. You can be a blogger, a public speaker, someone who solves issues in the VMware Communities, an end user, partner, employee. As long as you do something active in some way! It has nothing to do with certification programs like VCP. And it is renewed every year. It is a very active status. This is my version, you will find more info on the VExpert website.
What do I get you might ask? What’s in it for me? Quite a lot as a matter of fact. You get badges for your website or social media, test licenses for software, briefings, some freebies. But above all you get a community on slack. A group of peers there to help or just hang out with.
But allow me to dive into my personal experience. I worked for VMware for a few years, but the vExpert program only came to my attention when I was working for a technology partner – Blue Medora – some 3 years ago. Some guys in the Belgian VMware community decided there were far too few vExperts in Belgium compared to the Netherlands, there is always some healthy competition in the lower countries… (grab your maps app of choice ….). Kim Bottu, Tom Vallons and Stijn Depril convinced a bunch of us to apply. As a matter of fact all of these gentlemen are now vExpert Pro. If you are in the region and need help applying, talk to them too. I can hook you up.
Halfway through my first year I became jobless. I kept writing some blogs and talking to people. I continued as an independent contractor and actually found a few projects through the vExpert community! If you need more reasons to apply, tell me! I also noticed that for customers the vExpert title is becoming important, it is not a certification of course, but it shows that you are in touch with VMware. On top of that I hooked up with the guys at Comdivision and I now work together with them. You can check on our vExpert page that this a very important program for us!
In my second year I became part of the cloud management sub program and this year the security one. This is a specialisation, it means you get briefings on upcoming products and roadmap! Again an important advantage. There is no way I would get access to this kind of information as a lonely contractor… Also the cloud management guys have run some fun competitions…. and I attended the Carbon Black conference thanks to the security people. We also had meetings at VMWorld, but not sure if in-person conferences are ever coming back… Highlight was the vExpert reception in Barcelona that Pat Gelsinger attended!
You need to consider what you can contribute to the community, but do not be too hard on yourself. I wrote a few blog posts last year. I participated in briefings and published some vROps dashboards. I am not an expert on everything VMware. I do realise however that you have to contribute something to be part of this family! I am very grateful for everything I got out of the program so far!
On top of that if you are lucky enough to be in Belgium, you get vBeers and vBBQ meetings! But proof of citizenship is necessary to talk more about that!
After the big vSphere 7 launch last week, VMware this week announced the General Availability of all vRealize goodness … Namely vRealize Operations 8.1, Automation 8.1 and Log Insight 8.1.
Spending some time at home at the moment I decided to upgrade the home lab, starting with vROps 8.1. I am running two Intel NUCs still with ESXi 6.7 and 32-GB RAM each (really…). My vROps was an 8.01 updated from 8.0, updated from 7.5…
I ran the Upgrade assessment utility first, highly recommended. It will not only check versions, but also resources and pointed out some alerts I had in the past that might indicate resource problems… It comes as a PAK file that you load in the admin interface, like a software update.
After the assessment I just ran the update PAK file…. Obviously I have a very small test install, but I have some management packs installed and 3 compliance packs. All went smoothly. After that I upgraded my vCenter from 6.7 to 7.0 because I was feeling lucky… It proved no problem a all, validating the connection in vROps 7 established the blink to the updated VC.
Notice the newly cleaned interface! I added my Azure account, since cloud management pacs are now native. That will be my next blog post!
Adding my old Shuttle NAS server as a vSphere 7 host is next!
VMware has been working hard to add new features to vRealize Operations (vROps), the management tool is unrecognisable from it’s origins as vCops… One of the most overlooked areas for me is compliance. It is very powerful and easy to setup.
Since I talked about vCOps … let’s get back some years to around 2012 when VMware was still actively selling a product called Configuration manager (vCM). It was a product they inherited from EMC with a heavy Windows background… It WAS a Windows program, but extremely powerful. You could make your own rule sets for testing, use the built-in ones (vSphere hardening guidelines) or buy additional ones (SoX, HIPAA, PCI,…). So it was not limited to vSPhere environments. You could for example make a rule to check anti-virus was installed on all your desktops. Or check your MS AD. or…. But it was very complex to setup… I demoed it a lot, especially in the financial world.
The last GA version of vCM is 5.8.5 which is supported until 1/2021 (sse KB), but VMware stopped bundling it with vRops Advanced and Enterprise with version 6.7. They started developing a SaaS version called vRealize Air Compliance, but with the sales of vRealize Air to OVH in 2017 the product stopped. At least I cannot find it anymore.
The good news is that Vmware started adding compliance testing to vROps 6.7 and with each version the possibilities keep improving!
Compliance is based on alert symptoms. In previous releases you had to edit the default policy to enable them… But now Compliance finally has a full-blown dashboard where you can edit the benchmarks and activate them in the policy of your choice. By default vROps includes the vSphere, NSX and vSAN hardening guidelines as benchmarks. Can you imagine ? You can enable them with one click and check your whole SDDC for security weaknesses! I cannot emphasise enough how important this is, can you keep an eye on all settings of hundreds of virtual machines with pen and paper ? Of course not. Use vROps ! On top of that you can create up to 5 of your own benchmarks (or parts of the others) to check. This is the custom benchmark pane in the middle. I will sit back down now…
Of course my homelab reflects the real world .. with a lot of compliance issues! if you click a benchmark you get to the details with triggered symptoms. From their you can decide action. What I still miss here is a ‘remedy’ or ‘action’ button like we had in vCM. Now you have to go through a lot of messages and decide on corrective action. But knowing the team this is probably coming !
And for the ‘piece de résistance’ VMware now includes all the important industry benchmarks like PCI or HIPAA for free ! You just need to download them from the dashboard and enable them. You see in my example that I enabled ISO on my SDDC.
These used to be expensive extras in Configuration Manager and now you just need to download… I hope with the new services discovery and Telegraf agents that more benchmarks are coming, operating systems for example. Maybe partners will provide the too on the Exchange.
And if all of this is not enough, you can unleash these benchmarks on your VMware Managed Cloud (VMC SDDC) environment as well of course. All from the same interface.
I hope this blog post has inspired you to test this out for yourself. Let me know how it goes and I am available for consulting if you need help. You can contact me through my website.
One of the big advantages of VMware’s vRealize Operations Manager tool is plugins. It is also one of the least well known ones. Luckily with vROps 8.0 they cleaned up the admin interface. You can now quickly see which management packs have been installed and upgrade them all from one screen. It also makes it clearer which native management packs can be activated (such as the VMware MP for AWS) or which ones are no longer there (such as the VMware MP for NSX).
There are a lot of third party plugins available, especially for storage devices. You can find them in the solutions exchange (www.vmware.com/cmm) – I memorise that url as two letters before that news site ☺️. As you might know I have been working a lot with the Blue Medora plugins in the past years. I am now involved in a project however where the end user is a large Dell EMC customer and wants to use their toolset.
Dell EMC has published Storage Analytics for vROps for a number of years now. It used to be a paid solution, but since version 4.6 it is free to customers! This seems not well known yet, but you can check it in the documentation. Download, certification and documentation are available on the solutions exchange. They published a new version 5.0 that is compatible with vROps 8.0, although I also installed it on vROps 7.0! (Don’t ask…).
There is more confusion around the installation as some Dell EMC colleagues stated you need to install v4.6 first and then 5.0. That made no sense to me as each version is clearly a pak file (zipped java) with all the files and describe.xml to install the solution and dashboards and alerts. I tested it twice now, I just installed v5.0 and it worked.
Storage Analytics is a solution that can connect to most Dell EMC storage solutions: Avamar, Isilon, EMC RecoverPoint for Virtual Machines, VxFlex OS, Unity, Unity/VSA, VNX, VNXe, VMAX3, VMAX All Flash, PowerMax, VPLEX and XtremIO infrastructures in virtual or physical environments. This is a mixed blessing in my opinion. You need to install and maintain only one management pack, but it installs all dashboards for all solutions out-of-the-box. So you have some deleting to do. Before configuring any storage, you have to add separate vCenter accounts with RO users. My best guess is that they need it to build the relationships. Since my customer uses VMAX (really big ones…) I installed one instance per array as per the instructions. Again you have to keep your wits about since the fields take different values depending on your storage… I left one on Vplex (default) and it did not work of course. For VMAX you need to specify the <ip_adderss>:<port> of your Unisphere instance. This is not well documented! My config screen for an array looks like this:
After you setup the instances they start collecting data in your vROps database. This takes a while in big environments! I have also noticed that detecting the relationships can take even longer. But after a day I could see the relationship between hosts, VM, datastores and VMAX devices:
This is a standard dashboard and very helpful for troubleshooting I think. The management pack also installs symptoms and alerts of course. You might want to tweak these in policies if you get too many capacity related alerts:
And I leave you with these beautiful PowerMax overview dashboard that is included and shows everything you need to know in handy heat maps. I am even winning over storage people with these!