Last year (starting with Prism Central 2023.4 / AOS 6.71) Nutanix added the ability to centrally manage the local users (admin and nutanix) on AOS clusters and Prism Central itself through the PC GUI.
Really useful for getting rid of that pesky “CVM xx.xx.xx.xx is using default password” alert
without going to the command line, it also lets you update the local user password on all* your clusters with one click. (*well, up to 10 clusters at a time)
I’m using PC 2024.2, but I’ve updated this for some minor GUI changes to 2024.3.
You can find “Local Account Password” under Infrastructure/Network & Security
Note that 2024.2 has it as “Local Account Passwords” and 2024.3+ has “System Account Passwords” and 2024.3 also changed the way the components are identified (CVM/Prism Central).
Now click on one user name/component to update – you can click on the same user name/component (so admin/PC) for multiple clusters but you cannot select more than one name or component (so no admin/PC + admin/AOS).
And click Change Password. If you can’t click Change Password you have selected more than one username or more than one component.
Enter the current and next passwords and hit Change Password (again)
Note that PC 2024.3+ will also show the password rules in effect.
As an added bonus, Nutanix would really prefer you not to use password-based SSH and have added a new alert when you have not enabled Cluster Lockdown.
The cluster is using password based ssh access for the cvm xx.xx.xx.xx
As always, if your GUI (or docs) don’t match – check your versions!
How to! I’m working with PC 2024.2-on-AOS 6.10-on-AHV in a nested environment. One 1 node cluster, one 3-node cluster. Each has its own Prism Central (the main difference vs 1 shared PC is not Availability Zone). The clusters are all on the same network (so it mimics a stretched network).
I’ll need to do some other posts on troubleshooting and Recovery Plans.
Step 1: Log into each Prism Central and Enable DR.
You will be prompted to expand the memory in Prism Central. In my case my PCVMs will go from 28
to 32GB after the process runs
Step 2: Connect to the other Availability Zone. For on-prem clusters, each Prism Central (and its clusters) acts as an Availability Zone. If you have two clusters (physically close or separated by distance) you can replicate from one to the other without changing the AZ settings first.
Note the option to connect to DRaas (Nutanix Cloud AS or Nutanix Cloud Cluster AZ). Here we want to pick Physical Location.
This gives us prompts for IP address for the other Prism Central and credentials to Connect.
note the only option here is to disconnect the remote PC – and you have to check the AZ to get that option to popup
If you check on the remote PC you’ll see an AZ has been setup for the primary site automatically.
Step 3:
Before you create a Protection Policy you’ll need to have at least one category associated with the VMs to protect. You can add them a few ways, once added you can check using the “Categories” tab on the VM.
The first time you’ll get an explanation screen, just click Create
Enter a name for the policy and click the appropriate cluster in the drop down.
Click Save to start the Recovery Location widget. And note now you can also add a local schedule to the Primary Location.
You can see that “Auto-select Cluster” is an option but not recommended. Here, pick your desired remote cluster and click Save.
Click Add Schedule
Here you can choose either Asynchronous (1-15min RPO with NearSync or 1+ hour RPO with Async) or Synchronous (0min RPO).
To use NearSynch choose Minutes and then set a value from 1 to 15. No other values are allowed.
Retention Type Soliloquy
When you choose NearSync you are also limited to Roll-up Retention Type – although you only have a limited range of RPOs where you can choose which Retention Type to use.
1-15min RPO/NearSync uses Roll-up 1-4hour RPO (Async) you are allowed to choose Linear or Roll-up 5+hour RPO (Async) uses Linear
So if you ever wonder why you can’t choose which Retention Type to use during Protection Policy creation, check that table.
Linear Retention Type tooltip:
Roll-up Retention Type tooltip:
To use Async mode choose Hours (or Days or even Weeks!) and then set a value from 1 to 99999. No other values are allowed (so no setting Minutes to 60 or 90).
If you want to use Synchronous/Metro choose the Synchronous radio button and then pick Manual or Automatic Failure Detection Mode.
Note that you get these choices because I did not set a Witness up first. But you can Witness a new Blog Post coming soon!
I’m testing Synchronous so lets leave it there and check where we are.
Note that we can choose when to start this schedule, but we want to test it now, so hit Next!
Now we get to pick the category/ies that this Protection Policy will reference.
We’ll pick the one we added to the one VM earlier, click Add then Create.
Step 4:
Check the status.
Prism Central view
Click the policy name to get a summary tab
Here we can see Alerts by date and the settings we picked. You can also see it it reporting one synched.
We can also look at the Protection Summary page which shows 1 protected entity and 0 RPO not met. You can also catch a glimpse at the bottom of the tail end of the initial replication (the VM isn’t changing so nothing on going)
Prism Element view Looking at the Storage Container Usage for the matching (names the same) containers in Prism Element/Storage for each cluster shows the Remote site increasing in size to match.
If you look close there is a bug here I probably ought to report – it shows the peak in TiB but the current in MiB.
Speaking of looking close – the Active/Primary/First cluster is the 1-node cluster with a Replication Factor of 1. The destination cluster is the 3-node with a Replication Factor of 2. Thus doubling the storage used.
New with Prism Central 2023.x you can now allow trunking to a virtual machine from Prism Central.
Where typically the oVS strips the VLAN tag off of incoming network packets (and attaches the tags to out-going packets), if you set up a trunk connection the NIC will get unmolested packets.
Great for security systems (or Wireshark, tho you really also want port mirroring for that) , you’ll probably only be configuring this with docs from your security vendor. Don’t just add this to a VM as unless you configure the guest OS to handle the VLAN tags. Otherwise you won’t be on the network at all (guest OSs assume no VLAN tagging).
Previously you needed CLI to do this :
acli vm.nic_update <vm name> <vm nic mac address> vlan_mode=kTrunked update_vlan_trunk_info=true
Now you can see the option when adding (or updating) a NIC in PC.
Note that you have to be in this window to see it, the Resources and Review tabs don’t show Trunking.
You can also check the CLI to verify you are trunking along.
This assumes you have sourced a qcow2 image (I downloaded SLES15-SP6-Minimal-VM.x86_64-Cloud-GM.qcow2).
Process: Upload Image, Create VM (attaching a copy of the qcow2 as a disk), power on the VM. Profit.
Step 1: Upload the image to Nutanix. I’m using Prism Central but its fairly similar in Prism Element. Except for one step that I’ll note.
In Prism Central navigate to the Images page and click Add Image.
We are adding a qcow2 disk image so leave the Type as Disk. If you want to get fancy you can add the checksum value from whoever made the image.
Since I’m using Prism Central I have the option of using placement policies. This allows you to associate categories with images and with clusters and then set policies which associate images with clusters using either Hard (“must”) or Soft (“should”)policies.
Note that the image will be added to the SelfService container of the cluster the Prism Central VM is running on by default and then redistributed. The images are never saved directly to Prism Central.
But, I’m not using Policies and have only one cluster so I’ll leave it alone and hit save.
Give it time to process and provide updates.
Eventually you’ll see the upload task running.
Step 2: Create a VM with that disk. Navigate to Infrastructure / Compute & Storage / VMs and click Create VM
In Prism Central the VM will default to 1 vCPU/1 core and 4GB of RAM. Which is more than enough for my SUSE plans.
Note that Memory Overcommit is set on a per-VM basis from Prism Central (its not available in Prism Element) and enables powering on VMs with more memory allocated than the host physically has. in case on contention this is managed using a balloon driver in the guest or swapping to disk.
Also, the advanced processor compatibility checkbox allows you to set the processor family version for the VM. This could be useful if you have different versions of processors in a cluster or when you want to Live Migrate between cluster with different processor families.
PE Note: If you are using Prism Element, it defaults to 0 vCPU and 0 Memory so make sure you set that. (You’ll get an error.)
On the second page of the Create VM wizard you’ll set the network and (most important) the disk.
For the disk choose clone from image and select the image you uploaded. You can upload the image if needed. My qcow2 image has no extra space, so I’m kicking it to 20GB.
Prism Element Note: In Prism Central you can edit the Capacity here. In Prism Element this is greyed out when you attach the disk.
If you attach the disk then update the attached disk immediately, the VM will not create with a “Container UUID is required”. However if you attach the disk and create the VM, you can then immediately edit the VM and update the disk as needed.
Continuing with the VM creation, You can just hit next a few times and then save.
Wait for the VM to create then power it on.
I’m using SUSE here but the basic idea is the same. Note that this specific method winds up with the disks needing some partition work.
I used parted /dev/sda | resize | 3 | Yes | <new total size>
and then xfs_growfs /dev/sda3
But I have modest needs and this VM wont live long.
How to! I’m working with AOS 6.10-on-AHV in a nested environment. One 1 node cluster, one 3-node. Each with Prism Central. They are all on the same network (so it mimics a stretched network).
Since I have AHV-to-AHV I can’t do Metro/Synchronous/ 0min-RPO using Prism Element and since I have a freebie license I can’t do NearSync (1-15minRPO) so Async it is!
Step 1: Add each site to the other as a Remote Site. You have to do it from the active site to the remote site, but you will want to setup the remote-to-active to test failback.
From each Prism Element interface. under the menu tab choose Data Protection then Remote Site then Physical Cluster
Name the site appropriately and enter the VIP of the cluster. Don’t worry about entering “All CVMs” – adding the site with the VIP populate all the CVMs appropriately and will remove the VIP.
You can choose to set bandwidth throttling if you’d like.
[Interestingly, “Policy” is usually a Prism Central construct. There are not a lot of “policies” to create in Prism Element]
Set mappings for any container or network whose names do not match between the clusters – best practice is to make mappings for any container or network you will be using, but the storage container at least will use the same name as the VM is created on – if the same name is found.
The Remote Site will be available in the Remote Site tab, and you can Update from there if mappings or Throttle Policy changes
Step 3: Create the Protection Domain. Click Async DR from the +Protection Domain menu option and provide a name
Now you need to pick the VMs or Volume Groups to protect. You can also group them into Consistency Groups which will ensure the VMs in the CG are snapshotted at the same time. You can also enable Application Consistent Snapshots for the CG (tho NGT will be required).
“Auto Protect related entities” is for multiple VMs deployed as one application from Self Service or shared resources like Volume Groups. If you have two VMs which are connected to the same VG, selecting just one VM and leaving “Auto Protect related entities” enabled will grab both VMs and the VG.
The entities selected will populate on the right.
Step 4: Create the Schedule.
The third step of the PD creation is the schedule. No schedule = no replication.
Your options are 1-15min (NearSync) or 60+ minutes (Async). Note that entries between 16 and 30 will give “Please enter a value greater than or equal to 30 minutes”
but entering 30-59 will give you “Error saving schedule: RPO above 00:15:00 and below 01:00:00 hours not supported on remote <remote site name>”
So maybe 16-30 works for a 3rd party tool? No idea. Even Cloud Connect and Cloud Cluster as remote sites have the same 1-55 and 60+ limits.
Anyway, set the time limity9ou deem appropriate and have the licensing for. Note you can specify how many local snapshot to keep – and the remote replication is an option. You can even set when it starts. And here we can set application consistent for the whole PD (instead of just the Consistency Group)
The Schedule tab will populate with the changes, and note you can have multiple schedules. And since you pick the remote site for each schedule, you can run one set of local copies, one set of copies to one DR site and one set of copies to Cloud Connect? Lots of options.
Step 5:
You can check the main Async DR table for an overview. I’m replicating a stable VM so there is not much going on.
But down below you see the Local Snapshots
Remote Snapshots
And a little of the metrics
Jumping to the remote site, you’ll notice it shows no Entities, because this is the remote site
It does show the Local and Remote snapshots (the inverse of what the active site is showing)
Note that Restore will use the snapshot to create a local clone of the VM, Retrieve will copy the remote snapshot to the local site and Reclaimable Space (which is still calculating in my environment) will show how much physical space would be reclaimed if you deleted the local snapshot.
I have a project involving setting up and testing DR with Nutanix so I’ll be making notes about the overall topics and then posting some walk-throughs on setting it up.
First off note the versions this blog post about (AOS 6.10/PC 2024.2). Nutanix DR is changing and evolving pretty quickly. If your GUI (or referenced docs) has slightly different options, make sure you are reading the doc (or blog!) for your version.
A basic Nutanix cluster has 3 main versions to consider – hypervisor, AOS and Prism Central. You can check the Compatibility Matrix and the Upgrade Paths for more information on what fits together.
You can also use the Matrix to see the if the version you have supports the DR features you want:
[Like, “Does Protection Domain DR support CHDR with AOS 6.10?” (Cross Hypervisor DR – the ability to replicate from ESXi-to-AHV or vice versa).]
PE vs PC
There are two ways to configure DR in the GUI: Using Protection Domains in Prism Element (referred to as “Legacy DR” in some docs) and using Protection Policies in Prism Central.
Both can do Synchronous/Metro (0 RPO), NearSync (1-15min RPO) or Asynchronous replication (1+hr RPO), but only Prism Element/Protection Domains can do Synchronous/Metro for ESXi or Hyper-V and only Prism Central/Protection Policies can do Metro for AHV.
There are a ton of other limitations, mostly around Synchronous replication (RTT <5ms, if ESXi you can’t replicate VMs with vGPUs) but the key difference is Metro with ESXi/Hyper-V replicates a whole storage container. All other options are by VMs or Volume Groups, – you select them individually in Prism Element or use categories if you are using Prism Central.
Volume Group Xtras
Also, if using Volume Groups note that: CHAP might not be restored You can’t support Volume Groups with Metro using Prism Element (you can with Prism Central) You can’t have Volume Groups in an Protection Domain with Application-consistent snapshots You might need to manually reattach Volume groups during fail-over and/or failback CHDR (Cross-hypervisor DR) is not supported for VMs with Volume Groups Volume Groups cannot be protected to AWS (using Cloud Connect/NC2 clusters)
Metro? NearSync? Async?
If you are poking around in the GUI wondering where “Metro” is – the word “Metro” is only an option Prism Element (and only if you have ESXi/HyperV) and “NearSync” never shows up. Which replication family you are using is determined by the RPO you choose. 0 for Synchronous/Metro, between 1 and 15 (NearSync) and 60+ (Async). You are not allowed to set a RPO between 16 and 59 minutes.
And while Prism Element is happy with “90 minutes” Prism Central limits you to “x hours” for anything past 59 minutes.
While running through some SalesForce training (mostly on TrailHead) I suddenly found lots of pages were coming up blank.
The menus seemed to work, and switching to Classic worked, but some of the pages I needed were Lightning-specific.
Inspecting the pages turned up
DOMException: Failed to read the 'sessionStorage' property from 'Window': Access is denied for this document
Searching around on that and thinking about access/permissions issues, I found that whitelisting cookies for the SalesForce sites resolved the issue. If you’re using Chrome, open settings and click on “Cookies and other site date” under “Privacy and security.”
Then under “Sites that can always use cookies” add https://[*.]my.salesforce.com https://[*.]lightning.force.com flagging them for “All cookies, on this site only”
I think a Chrome update came out that changed the default for third party site cookies which caused the problem. This resolved it for me.
I needed to do some Salesforce automation (running an existing report and then manipulating the data before send off to a different API) and, already having a CentOS/Python2-based automation VM up in Google cloud, used Python2 as a starting point.
Token Postman
First step, have the SF team create an SF account for the service “user.” They also told me they created a “connected app” but I don’t have details on that.
Suffice to say, I was give a SF password for the email address of my user account plus a Consumer Key and Consumer Secret. These are needed to generate a short-time access token used for the actual request.
Working with the SF developer I got the token request working in Postman (note this is very redacted, everything other than port URL, grant_type, token_type and issued_at are altered)
Then I had to get that working in Python which turned out to be a pain in the ass, thus this post so if I ever need to do it again I have some help.
Token Python Developer
First, I made the four components needed into environment variables so they’re not in the .py or github . Then loaded some needed modules.
import json import os import requests from requests.auth import HTTPBasicAuth
The includeDetails=true is needed if you want all the rows and not just the summary. My initial run was stymied as the report was saved last year, and the dates are set when its manually run. I saved a copy of it with a date of “last 7 days” and voila, now its always current. The Report ID (the <reportID> above) just look in the URL when you are looking at the end of the URL when you open the report in SalesForce.
Ran into an issue while updating one of my 6.5 vCenter servers, where it inexplicably reported:
A problem occurred while getting data from the source vCenter Server.
A quick peek at the installer log showed :
2019-11-23T21:32:47.402Z - info: Stream :: close 2019-11-23T21:32:47.402Z - info: Password not expired 2019-11-23T21:32:47.402Z - error: sourcePrecheck: error in getting source Info: ServerFaultCode: Failed to authenticate with the guest operating system using the supplied credentials. 2019-11-23T21:32:55.877Z - info: Log file was saved at: C:\installer-20191123-163017412.log
Which was interesting as I had to think about what guest OS the installer might need to log into. The I recalled some other issues I’ve had logging into vCenter – the root password likes to expire after a year.
Part of the fun – you can log into the console using root with no issues. it won’t report there is a problem unless you ask.
The chage -l root command will tell you the time since the last password change. And passwd of course will let you update your password.
While passwd complains about simple password, it will take a out anything. Here I reused the one that just expired.
