VMware vCloud error: Batch update returned unexpected row

Ran into a bit of a weird issue the other day. Suddenly non of my vApps could be started and all I got back was the following errors: “Batch update returned unexpected row count from update [0]; actual row count: 0; expected: 1″ (See screenshot). Some google searches didn’t give any results related to VMware, only a few pointing at databases and hibernation modes?????

After checking the database, which was running fine, I checked vCenter Server but this reported no errors. What I could see was some tasks in which the VM settings were being changed by vCloud and then no more new tasks, not even an attempt to power on the VMs. I decided to call VMware Support and after some digging through the log files we finally found the following lines in vcloud-container-debug.log: 

“ERROR: ComputeFabricExecutor-Pool-5 | AllocationPoolVirtualResourcePool | Admission control failed for VRP <snip> during deployment of VM <snip>. Total CPU required for all VMs in the VRP including current VM being deployed 61,000MHz is greater than VRP cpu 60,000MHz.”

Well, that is a very clear message :-) And indeed when checking the OrgVDC I could see that it was near its limit of GHz I set. I shutdown some other vApps and could now power on the vApp I was testing with.

Update: The error occurred on multiple occasions.

I also had this happen when deploying a vApp with a number of static routes between the vApp Networks. In those static routes an IP address was used pointing to an IP from a static pool, which means that with each boot of the vApp, that IP could potentially change and that is what triggers the error. I could easily solve this by editing the static routes to point to the default gateway (edge) and re-edit the static route after a new deployment. During restarts of the vApps the IP will not change because I enabled the option “Retain IP / Mac resources”.

Invalid nextHop ’192.168.0.10′. NextHop must not be a Vnic assigned IP Address. Caused by: java.util.concurrent.ExecutionException: com.vmware.vcloud.fabric.nsm.error.VsmException: VSM response error (13003): [StaticRouting] Invalid nextHop ’192.168.0.10′. NextHop must not be a Vnic assigned IP Address. Caused by: com.vmware.vcloud.fabric.nsm.error.VsmException: VSM response error (13003): [StaticRouting] Invalid nextHop ’192.168.0.10′. NextHop must not be a Vnic assigned IP Address.

Another occurrence was when one of the VMs that was deployed had the “Guest Customization” option enabled but these tools hadn’t been installed on the vCloud cell yet. In vcloud-container-debug.log the following error was seen:

Deployment failed | vcd=88db5971-40a7-4cfe-b883-bea4464b7f42,task=ded2203b-bf34-4104-a25a-4e8c39615d10 com.vmware.ssdc.util.LMException: HTTP NFC Error. Cannot deploy package from /opt/vmware/vcloud-director/guestcustomization/windows_deployment_package_sidgen.cab to ‘VMName (21760366-b3da-4ef6-920d-3263c2cbcdf2)/windows_deployment_package_sidgen.cab’.

I hope VMware will change the error message on this to make the troubleshooting a bit easier :-)

  • http://twitter.com/VMwareKB VMware Knowledgebase

    I’ll definitely pass this on to our product managers to consider… Thanks Gabe!

  • http://twitter.com/marknudelman Mark Nudelman

    Thanks for this post. You are correct, our errors do need help in vCD :) Been fighting with development for a long time to make these better. In the case of this error, there are so many events that throw this specific exception it would take a ton of dev time to break them all out.

    I am with you though, and will keep pushing for better error reporting. I think you will be happy when you see vCD 5.5 come out….has some nice additions.