More and more we see NTP of VMs is getting overlooked. Now we have COVID-19 Infra and System admins need to have a NTP strategy that is robust and does not impact production.
Here is a powercli method to set ntp on all VMs to get from ESXi hosts.
In this blog I advise the prerequisite step of how to set NTP on ESXi.
Save file as TimeSync.ps1Run from powercli. .\TimeSync.ps1
Step 1. Create the temp folder on your Windows machine and generate a CSV for the VMs in vCenter. Get-View -ViewType virtualmachine | Select name,@{N=’ToolsConfigInfo’;E={$_.Config.Tools.syncTimeWithHost } } | Export-Csv “c:\temp\dr\TimeSyncStatus.csv” -NoTypeInformation Step 2 Create the powercli script.
Please read the article below to see how I managed to clear
the VCAP Deploy exam. Both offer a different
set of challenges. The VCAP deploy is
all hands on and navigation of vCenter and deep diving into creating and managing
storage networking vMotion alarms RPO and backups.
The VCAP Design was a different beast and proved very challenging. The exam is heavily weighted towards project management and getting to terms with the best design decisions which can be made for a wide range of scenarios. While I was studying, I found the vBrownbag video series invaluable.
Here is the blue print I created in an easy to follow org chart.
It is very easy to go wrong in the exam. It will take you a full 2 hours to complete. I had
20 minutes to spare at the end as this was the second time I have taken the exam. If you have finished the exam after 20 minutes
then you have likely failed. Each
question I gave a comment for each part of the question and gave my reasons for
making the choice. So you will be taking your time reading and
rereading each question and justifying your answers. The training material was a massive amount
of learning and it took me places I never would have gone: ). Overall I got great benefit and insight from
the exam.
My preparation had taken me to a state where I couldn’t have
done much more so I was quietly confident.
To change the password for the hms-keystore.jks keystore, open the remote console of your vSphere Replication virtual machine and log in as root.
Obtain the current keystore password.
# /opt/vmware/hms/bin/hms-configtool -cmd list | grep keystore
Example of the output hms-keystore-password = old_password
Change the keystore password.
# /usr/java/default/bin/keytool -storepasswd -storepass old_password -new new_password -keystore /opt/vmware/hms/security/hms-keystore.jks
Change the vSphere Replication appliance private key password.
The following command is a long, single command and must be run at once. There are breaks in the command for better visibility. Verify that the command returns a success message.
# /usr/java/default/bin/keytool -keypasswd -alias jetty -keypass
old_password -new new_password -storepass new_password -keystore
/opt/vmware/hms/security/hms-keystore.jks
Update the configuration with the new password.
/opt/vmware/hms/bin/hms-configtool -cmd reconfig -property ‘hms-keystore-password=new_password’
Update the tomcat server.xml file with the new password.
sed -i — ‘s/old_password/new_password/g’ /var/opt/apache-tomcat/conf/server.xml
Reboot the appliance for the changes to take effect.
# reboot
Use a supported browser to log in to the vSphere Replication VAMI.
The URL for the VAMI is https:// vr-appliance-address:5480.
On the VR tab, click Configuration, and click Save and Restart Service.
foreach($esxcli in get-vmhost|get-esxcli){“”|select @{n=’Time’;e={$esxcli.system.time.get()}},@{n=’hostname’;e={$esxcli.system.hostname.get().hostname}} }
Set time manually on esxi host.
Directly on host: esxcli system time set -H 03 -m 29 -s 40
For all hosts.
$esxlist = get-vmhost
foreach ($_ in $esxlist) {$esxcli = Get-EsxCli -VMHost $_; $esxcli.system.time.get();}
foreach($_ in $esxlist){$esxcli = Get-EsxCli -VMHost $_; $esxcli.system.time.set(10,09,53,01,20,2019); }
In SRM-UI only “A general system error occurred: vix error codes = (1, 2).” is shown.
VIX is the failback authentication process and it does not always trigger correctly due to timeouts and other issues that might be happening in vCenter at the time of the test failover.
Below is what I expect to see when VMware Authentication service installed.
It should look like this.
Linux
Checking other articles the service does not install by default and during the installer you need to install the extra service. Here is another article where the .exe gets removed during a VMware tools upgrade !
Many times it’s not practical to modify the IP addresses of every individual VM as they are configured. Luckily VMware has provided a way to bulk upload IP addresses.
From an SRM server, open a command prompt and change the working directory to: c:Program FilesVMwareVMware vCenter Site Recovery Managerbin
NOTE: Path may be different depending on your install location.
Generate a .CSV file to edit your IP Addresses by running dr-ip-customizer.exe with the –cfg, –cmd –vc -i –out switches.
–cfg should be the location of the vmware-dr.xml file. –cmd should be “Generate”, –vc lists the vCenter server, and –out lists the location to generate the .csv file.
Open the .csv file and fill out the information. Notice that there are two entries for the VM. This is because there are two vCenters and in order to do protection and fail back we need the IP Addresses for both sides.
Once the IP Address information is entered, run the customizer again with the –cmd “Apply” and –CSV file location.
IP changes during a SRM failover are a necessity for many companies and SRM 5.8 has both made this process easier as well as giving plenty of options depending on your needs. We can now use network mapping, manual IP customization or bulk IP customization to accomplish our objectives.
For all those using Load balancing with VMware, you will need to configure a NLB portgroup to avoid port flooding. To prevent RARP packets being sent every time vMotion or powering on a VM, you will need to configure no notify switch on the required portgroups. If you are using unicast, you will also need to set the Security Policy Forged Transmit to Accept.
A poorly performing virtual machine is probably one of the topmost ailments you’ll bump into as a virtualization admin. The issue is also one of the hardest nuts to crack due to its multifaceted nature. Regardless, there are a number of things you can do to read the symptoms, narrow down the cause and apply a fix. Taking hints from this VMware KB, today we explore 10 ways to troubleshoot poor vSphere performance. Have a look at this Altaro webinar for a more holistic approach to boosting vSphere performance.
Performance Monitoring
Before we move on, let’s revisit the Performance monitoring function embedded in vSphere clients. This is one tool that will help you examine performance related issues. Figure 1 shows a performance chart for datastore read and write latencies for a VM using the vSphere Web client. The 3ms peaks observed are well within the acceptable range, however, sustained levels exceeding 10ms are indicative of a looming storage issue or perhaps network congestion.
Use alarms wherever possible so you’re always on top of any performance issue. Alternatively, consider deploying vRealize Operations Manager for a more in-depth assessment of your environment.
Figure 1 – Performance graph for datastore read and write latencies
How to troubleshoot poor vSphere performance
The steps, or rather, questions you should ask yourself, are listed in an orderly fashion starting with the most trivial. Re-evaluate the performance of the affected VM after each step if you decide to try out a relevant fix. You can then choose to skip to the next step depending on the level of observed improvement if any. If you come across something as glaringly obvious as a failed disk on a host, it goes without saying that you’d want to fix this first before moving on!
1 – Is it really unexpected behavior?
A VM subjected to a heavy workload can sometimes be perceived as performing poorly. Some examples are virtualized instances of SQL servers, processor intensive or badly written SQL queries or mail servers with large user bases. The performance monitoring charts in vSphere Web client will help you gauge resource utilization across a given period of time so you can assess if the behavior anomaly was a one-off or ongoing and then gauge whether the behavior is expected or not. Products such as MS SQL and Exchange Server, will by design, take up any RAM thrown at them bleeding memory from the VM’s guest OS unless otherwise configured. What that in mind, it’s always a good idea to refer to the product’s documentation.
Updates and new releases may address performance issues in the form of ironed out bugs or improved drivers and code. Sometimes, however, the latest release could, in fact, make the problem even worse. So test, test and test again before taking the leap or at least wait until there’s a sufficient uptake of the new release or update, so you can make an informed decision.
Make sure that vmtools are installed, updated and running on every VM that supports them. The VMware Tools package, above all, provides a set of optimized virtual device drivers that directly affect VM performance (for the better usually). Again, using the vSphere Web client, you can easily check your overall vmtools health as shown in Fig. 2. Remember to add the vmtools fields by right-clicking on the fields header and selecting them accordingly.
Figure 2 – Displaying the vmtools status for VMs managed by vCenter Server
Alternatively, you could cook yourself a PowerCLI script that checks for the vmtools package and its current state. The bulk of the properties related to vmtools is found under <vm>.guest.extensiondata.
Figure 3 – Using PowerCLI to query the state of vmtools on VMs
4 – Is your VM adequately powered in terms of resources?
Though seemingly obvious, you’d be surprised as to how many VMs are not assigned sufficient resources as per the guest OS requirements and the applications running under it. Remember that, regardless of the benefits virtualization brings about, there are always overheads to contend with. For instance, if the VM runs out of RAM, it will start swapping to disk on a more frequent basis. If the underlying storage is flaky, performance will be badly hit. Whenever possible, use reservations, resource pools, and DRS to ensure that the correct amount of resources are assigned to a VM for maximum operational efficiency.
5 – Is antivirus software or similar running on ESXi?
Yes, even though rare in practice, you can, in fact, find antivirus software – think vShield – running on ESXi. This can adversely affect VM performance on a number of counts if it is not configured properly. One should also keep in mind that there is little justification for running AV software on ESXi given its small footprint and inbuilt security features. Best practice, in fact, calls for anti-malware software to be relegated to the VM’s guest OS. If you must install AV on ESXi, do make it a point to exclude VM files such as VMDKs from scanning schedules especially during peak utilization hours.
AV software aside, there’s also Backup and other I/O intensive software that may adversely impact VM performance.
Whether you’re using local or SAN-based datastores, it all boils down to the performance and health of your disks and the underlying sub-systems housing them. Simply put, if VMs do not get their fair share of IOPS, performance will start degrading. Here are a few things you can check and do:
Bad disks: Run regular health checks on your disk / networked storage and replace aging or failing disks immediately.
ESXi OS: Use separate disk(s) for the ESXi host’s OS, the swap partition, and VMs residing on a local datastores. Also, consider using RAID to improve read and write performance.
Snapshots: Delete any unused or redundant snapshots. The more snapshots you have, the greater the disk overheads will be vis-à-vis I/O activity.
Encryption: Use disk encryption only when necessary. Encryption = overheads = decreased performance.
Running a dozen or so VMs configured with 16GB of RAM concurrently on a single ESXi host that has only 96GB of RAM is simply asking for trouble. Consider adding RAM to the host or use DRS – if you have multiple ESXi hosts and proper licensing – for better load distribution.
CPU power management, when enabled on ESXi servers, may introduce speed latency that can be picked up by applications or workloads resulting in a slower performance. If you suspect this to be the case, do consult the vendor documentation on how to disable CPU power management. If disabling it has no effect, re-enable it in the spirit of running energy-friendly data centers.
Make sure that your ESXi host networking does not become a bottleneck preventing VMs from running and operating optimally. Symptoms may include a laggy response when connecting to VMs via remote clients or management consoles to overly lengthy vMotion transfers. Make sure that the network cards on your hosts are correctly configured. If your infrastructure permits it, partition or segregate network traffic. Run services such as management, vMotion and storage on their own dedicated network. Use optimized TCP/IP stacks and things like Jumbo frames where applicable. Make sure that the firmware on any networking hardware thrown in the mix is up to date. Finally, do not exclude issues with the virtual switches. Check your portgroups, vlan assignment and so on.
10 – Have you checked your ESXi OS and hardware health lately?
Just like any other system, ESXi needs regular maintenance for it to operate at full throttle both from a hardware and operating system perspective. Purple screens aside, you cannot immediately tell if there’s some issue brewing just waiting to rear its ugly head on a weekend night. Make sure to monitor disk usage and use the health monitoring software that generally comes bundled with your server(s) or products like PRTG.
This pretty much sums today’s post on how to improve VM performance. That said, there are probably more factors affecting VM performance and ways and means to tackle them. I suggest you read the material referenced by the links provided throughout this post and other posts, such as this one on DRS, for more information. Also, get yourself into the habit of visiting sites such as the VMware Technology Network where you’ll find like-minded people sharing similar queries, problems, and potential solutions.