tomcsanyid's profile - activity

Hi all, we hit the following bug on stable/liberty using WMI Security Groups with neutron-agent-hyperv: An instance has a security group let's say "http" on it. Assume that we'd like to extend the ports allowed by this security group, so we add a new port to it. When we do that the following error could be seen in neutron agent's log: http://paste.openstack.org/show/eK8KA... and the new port rule isn't applied to the machine.

If we remove the security group from the instance, then add the port to it then re apply it to the instance it works fine.

Thanks, Domi

After contacting with CloudBase directly we found out the solution to this one: simply applying this patch to our system solved the issue: https://review.openstack.org/#/c/293438/

Hi all,

We talked about this briefly during an IRC session, but didn't reach any conclusions, so I thought I'll post this here as well. A brand new, fresh installation of FreeRDP doesn't work, and here is the error message we see in the controller's nova-api.log:

http://paste.openstack.org/show/e1bCN...

The network setup is the following:

Horizon -- routing and NAT --> SRV7 (having FreeRDP WebConnect installed) -- same network --> SRV5 (.5.24 in the log) running the VM in question.

Let us know if we can provide you with any logs, info etc. It doesn't matter on which HyperV server the VM is hosted the error is the same and the RDP web console doesn't work.

Thanks, Domi

Without changing anything today we started to get the same error again....so we are back to sq 1.

Worth mentioning that this is a bug: in imagecache.py line 102 the destination_path will be the same as fetch_path if the cache is disabled. This causes the "if destination_path != fetch_path:" in line 71 to fail, and therefore the code will not do any resizing.

After enabling the imagecache subsystem via the two options "cache_fetched_images=True and image_cache_dir=PATH_TO_CACHE" the newly created VHDX files have the right size. Let us do some more testing.

Also let me walk you through our process: we downloaded the QCOW2 image from Ubuntu's cloudarchive then converted it to VHDX on a Mac with qemu-img 2.5.0: qemu-img convert -p -O vhdx -o subformat=dynamic trusty-server-cloudimg-amd64-disk1.img trusty-server-cloudimg-amd64-disk1.vhdx

Just to be clear we have since updated to the master branch (cloned from github and overwrote all the Python files on our Cinder head), but the issue remained that's why we opened it.

Sure thing, I just did it. I didn't even do the second step, I was able to see after the first that the image size is wrong. Is it possible that the new size is only applied accidentally when the image is converted? Because our image isn't converted, it is in native VHDX format in Glance.

Just tested it out now, and indeed the resulting image's size is 8.6 GB instead of the requested 10 GB. I need to not though that cinder create works just fine, we are able to create volumes and attach them to VMs fine.

We are not using cinder create directly, but Horizon to create a machine, so in CLI that would be nova boot like this: nova boot --nic net-id=net_id --flavor our_flavor --block-device source=image,id=3f74e0f8-0e2c-499f-8836-bcae029c8af2,dest=volume,size=10,shutdown=PRESERVE,bootindex=0 test

Indeed, but the disk isn't resize, that's the problem. Let me show you how the disk of a VM looks like using Get-VHD in PowerShell: http://paste.openstack.org/show/XHOr5HLQerN8Cg76ooZu/ do you see the problem? The Size parameter of it isn't modified to match the volume size requested

Hi Alessandro, the 2.3 GB is the reported partition size inside the VM using Linux. We are indeed doing a boot from volume. "The Cinder volume VHD(X) gets resized to the requested volume size, which in turn is attached to the Nova VM." -> this is exactly what should happen, but it doesn't apparently

Currently the Cinder SMB backend only copies the image file from Glance onto the SMB share and then signals the hypervisor that the image is ready, however there is a major step missing from this flow: re-sizing (extending) the image according to its flavor. Basically the VHDX's parameter 'maximum size' needs to be adjusted to the value that's coming from the controller (? this is just an assumption). Example: We have an Ubuntu image in Glance with the size of 2.3 GB. If the user selects a flavor that has a root disk size of 5 GB then he/she should be able to see 5 GB of disk in Linux, however this is not the case, 2.3GB is the reported size. Note that this is totally independent of the VHDX being dynamically expanding type. Please commit a fix to Cinder-SMB as soon as possible.

Cheers, Domi

Hi Luci, AWESOME work, it works flawless! Also it is worth mentioning that we set up SMB constrained delegation, but I don't think that has a direct connection with the issue: http://www.thomasmaurer.ch/2014/02/hyper-v-over-smb-set-smb-constrained-delegation-via-powershell/

This is the link I remembered: https://msdn.microsoft.com/en-us/library/windows/desktop/aa365202.aspx

We are using Scale out FileServer with a Server 2012 R2 head with Cinder-SMB installed.I found a Microsoft article stating that SMB 3.0 and SOFS is capable of handling the locks needed.When watching the cluster via MMC we can see that the VHD is opened by HyperV + a read-lock is placed on it.Thanks!

We are using SMB and iSCSI, experienced the issue with both backends in Kilo. In Liberty I think we only tried with SMB and the bug was there. Thanks

Sure https://review.openstack.org/#/c/18832/ https://review.openstack.org/#/c/18832/1/nova/virt/libvirt/driver.py

It must be possible to open the VHD when using the GET_INFO access mask, but still it fails (because of the read-lock maybe? this file says READ access needed) https://github.com/Microsoft/Virtualization-Documentation/blob/master/hyperv-tools/Convert-WindowsImage/Convert-WindowsImage.ps1#L2631

Hi All,

We recently faced a weird issue: ever since we upgraded to Liberty we were not able to spin up any new VMs because of a strange "No valid host was found" issue. We debugged the issue and we came to the following conclusion: Ever since we have used OpenStack we had absolutely false numbers on graphs in Horizon, showing for example 55 TB used out of 130 GB. We didn't pay attention to this because it was a known bug in Kilo. The bug is causing Nova to compare all the storage used by VMs to the local storage of the hosting hypervisor, meaning that even if the machine is stored on an iSCSI LUN, or on an SMB share the amount of space defined in its flavor will be deducted from the local disk space of the hypervisor. This caused no error in Kilo, because cinder-scheduler ignored these values. However in Liberty the bug was fixed and therefore the developers of OpenStack decided to add the DiskFilter to the enabled DefaultFilters of Cinder scheduler immediately causing the behavior I described above. We solved this problem via creating a workaround: in cinder.conf it is possible to define a new set of filters, so we simply omitted the DiskFilter from that list and everything started to work as it did before.

We would like to however fix this in a better way, and that would require CloudBase to pull the fix from the official OpenStack repository and apply it to the HyperV driver as well (it was applied to the KVM-libvirt driver as far as I remember).

I'd have reported this on Github as an issue of the nova fork you are maintaining but I think issues are disabled for that repository.

Thanks, Domi

Thanks for Luci for helping us with this issue, so for future reference the final fix was then the following:

1. Set up a migration network
2. Set up SMB constrained delegation (it might not has a direct connection with the issue, but it's good to have it): http://www.thomasmaurer.ch/2014/02/hy...
3. Applying Luci's patch: Downstream Cinder Liberty patch: https://github.com/cloudbase/cinder/c... os-win patch: https://review.openstack.org/#/c/280640/

Hi all,

We are trying to test out a live-migration scenario with the following setup: 2 HyperV servers, both joined to the same domain, and one SMB file share failover cluster server with Cinder backend installed. We are using Liberty everywhere. We are simply trying to live migrate one instance via Horizon but the migration fails and we receive the following error (in the cinder-volume.log of the Cinder backend):

http://paste.openstack.org/show/485650/

We have added full control permissions to every single computer or group possible, but we are still facing this error.

We would be really grateful if anyone could give us some pointers...at least telling us why would cinder like to open the vhdx file when we reqeested a live-migration?

thanks.

Sure thing, sorry for the delayed answer: Originating server (has the VM currently): http://pastebin.com/VZKQzMww Target server (should have the VM after migration): http://pastebin.com/SiYA5HuA I used pastebin, because paste.openstack.org limited my input to only 230 lines. Thank you!