Live-Migration fails with cinder SMB backend - error code 5 Access Denied

asked 2016-02-01 20:06:54 +0300

tomcsanyid
16 ●6 ●8 ●8

Hi all,

We are trying to test out a live-migration scenario with the following setup: 2 HyperV servers, both joined to the same domain, and one SMB file share failover cluster server with Cinder backend installed. We are using Liberty everywhere. We are simply trying to live migrate one instance via Horizon but the migration fails and we receive the following error (in the cinder-volume.log of the Cinder backend):

http://paste.openstack.org/show/485650/

We have added full control permissions to every single computer or group possible, but we are still facing this error.

We would be really grateful if anyone could give us some pointers...at least telling us why would cinder like to open the vhdx file when we reqeested a live-migration?

thanks.

edit retag flag offensive close merge delete

add a comment

3 answers

Sort by » oldest newest most voted

answered 2016-02-02 15:50:58 +0300

tomcsanyid
16 ●6 ●8 ●8

updated 2016-02-19 12:14:33 +0300

Thanks for Luci for helping us with this issue, so for future reference the final fix was then the following:

Set up a migration network
Set up SMB constrained delegation (it might not has a direct connection with the issue, but it's good to have it): http://www.thomasmaurer.ch/2014/02/hy...
Applying Luci's patch: Downstream Cinder Liberty patch: https://github.com/cloudbase/cinder/c... os-win patch: https://review.openstack.org/#/c/280640/

edit flag offensive delete link

Comments

Without changing anything today we started to get the same error again....so we are back to sq 1.

tomcsanyid ( 2016-02-19 18:59:57 +0300 )edit

It's worth noting that this issue is happening inconsistently on different environments and might be worth investigating the locks held by Windows on the SMB VHD resources.

alexpilotti ( 2016-02-19 19:15:12 +0300 )edit

Constrained delegation is useful for delegating access rights across hosts, but it's unrelated to the locking issue.

alexpilotti ( 2016-02-19 19:15:39 +0300 )edit

add a comment

answered 2016-02-02 13:05:20 +0300

Lucian Petrut
56 ●1 ●2

Hi,

When a live migration is performed, the destination host will initialize the connection to all the volumes connected to the instance. Cinder plays a role in this process, ensuring that the volume can be accessed by the destination host, providing the relevant connection information. In this case, Cinder will attempt to open the image to retrieve the format.

Now, this seems to cause your issue. It appears that in some environments, the access mask is not handled properly and we are unable to open in-use images. The good news is that I've just sent a fix for this:

https://review.openstack.org/#/c/275102 (os-win, used by Cinder M) https://github.com/cloudbase/cinder/t... (downstream Cinder Liberty repo)

Unfortunately, the Cinder policy related to backports would not allow us to send this fix on the upstream Libery branch as they only accept backporting security bug fixes. Anyway, you can use our downstream repo provided above.

Please let me know if this fixes your issue. We'll provide an updated MSI soon.

Regards, Luci

edit flag offensive delete link

add a comment

answered 2016-02-11 18:25:09 +0300

Tamas Pasztor
1 ●1

Hi Luci,

I apply the patch, but it not working for me. If I try to live migrate vm, get this error:

2016-02-11 17:17:27.135 1728 DEBUG oslo_messaging._drivers.amqpdriver [-] received message msg_id: 0669d7ed76804f21a759162aee846500 reply to reply_d5187308972e47b68fbfc6903e445b8c __call__ C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Python27\lib\site-packages\oslo_messaging\_drivers\amqpdriver.py:193
2016-02-11 17:17:27.230 1728 DEBUG oslo_concurrency.lockutils [req-3700b16d-1515-4828-9368-cd45543eb0c8 01d407fa35af4a0eb873c29bd4c71751 9097b948e1f5440f80a6fe4d0cedc306 - - -] Lock "smbfs-12c25de0-4c31-4c55-b119-13ece392d6ac" acquired by "cinder.volume.drivers.remotefs.lvo_inner2" :: waited 0.000s inner C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Python27\lib\site-packages\oslo_concurrency\lockutils.py:253
2016-02-11 17:17:27.244 1728 DEBUG cinder.volume.drivers.windows.vhdutils [req-3700b16d-1515-4828-9368-cd45543eb0c8 01d407fa35af4a0eb873c29bd4c71751 9097b948e1f5440f80a6fe4d0cedc306 - - -] Trying to access file: C:\OpenStack\b059be7b212fc829e92972d0a0b30e82\volume-12c25de0-4c31-4c55-b119-13ece392d6ac.vhdx _open C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Python27\lib\site-packages\cinder\volume\drivers\windows\vhdutils.py:246
2016-02-11 17:17:27.244 1728 ERROR cinder.volume.drivers.windows.vhdutils [req-3700b16d-1515-4828-9368-cd45543eb0c8 01d407fa35af4a0eb873c29bd4c71751 9097b948e1f5440f80a6fe4d0cedc306 - - -] Executing Win32 API function OpenVirtualDisk failed. Error code: 32. Error message: The process cannot access the file because it is being used by another process.

2016-02-11 17:17:27.244 1728 ERROR cinder.volume.drivers.windows.vhdutils None
2016-02-11 17:17:27.244 1728 ERROR cinder.volume.drivers.windows.vhdutils
2016-02-11 17:17:27.244 1728 DEBUG oslo_concurrency.lockutils [req-3700b16d-1515-4828-9368-cd45543eb0c8 01d407fa35af4a0eb873c29bd4c71751 9097b948e1f5440f80a6fe4d0cedc306 - - -] Lock "smbfs-12c25de0-4c31-4c55-b119-13ece392d6ac" released by "cinder.volume.drivers.remotefs.lvo_inner2" :: held 0.016s inner C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Python27\lib\site-packages\oslo_concurrency\lockutils.py:265
2016-02-11 17:17:27.244 1728 ERROR cinder.volume.manager [req-3700b16d-1515-4828-9368-cd45543eb0c8 01d407fa35af4a0eb873c29bd4c71751 9097b948e1f5440f80a6fe4d0cedc306 - - -] Driver initialize connection failed (error: Executing Win32 API function OpenVirtualDisk failed. Error code: 32. Error message: The process cannot access the file because it is being used by another process.).
2016-02-11 17:17:27.260 1728 ERROR oslo_messaging.rpc.dispatcher [req-3700b16d-1515-4828-9368-cd45543eb0c8 01d407fa35af4a0eb873c29bd4c71751 9097b948e1f5440f80a6fe4d0cedc306 - - -] Exception during message handling: Bador unexpected response from the storage volume backend API: Driver initialize connection failed (error: Executing Win32 API function OpenVirtualDisk failed. Error code: 32. Error message: The process cannot access the file because it is being used by another process.).
2016-02-11 17:17:27.260 1728 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-02-11 17:17:27.260 1728 ERROR oslo_messaging.rpc.dispatcher   File "C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Python27\lib\site-packages\oslo_messaging\rpc\dispatcher.py", line 142, in _dispatch_and_reply
2016-02-11 17:17:27.260 1728 ERROR oslo_messaging.rpc.dispatcher     executor_callback))
2016-02-11 17:17:27.260 1728 ERROR oslo_messaging.rpc.dispatcher   File "C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Python27\lib\site-packages\oslo_messaging\rpc\dispatcher.py", line 186, in _dispatch
2016-02-11 17:17:27.260 1728 ERROR oslo_messaging.rpc.dispatcher     executor_callback)
2016-02-11 17:17:27.260 1728 ERROR oslo_messaging.rpc.dispatcher   File "C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Python27\lib\site-packages\oslo_messaging\rpc\dispatcher.py", line 129, in _do_dispatch
2016-02-11 17:17:27.260 1728 ERROR oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
2016-02-11 17:17:27.260 1728 ERROR oslo_messaging.rpc.dispatcher   File "C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Python27\lib\site-packages\osprofiler\profiler ...

(more)

edit flag offensive delete link

Comments

It must be possible to open the VHD when using the GET_INFO access mask, but still it fails (because of the read-lock maybe? this file says READ access needed) https://github.com/Microsoft/Virtualization-Documentation/blob/master/hyperv-tools/Convert-WindowsImage/Convert-WindowsImage.ps1#L2631

tomcsanyid ( 2016-02-13 16:00:37 +0300 )edit

What SMB share backend are you guys using? Is it a Windows share, Samba share or a NAS based one? I'm wondering if the locks are handled properly by the share backend, or perhaps it's the share mode flags. Either way, I'm looking further into this.

Lucian Petrut ( 2016-02-15 13:05:48 +0300 )edit

We are using Scale out FileServer with a Server 2012 R2 head with Cinder-SMB installed.I found a Microsoft article stating that SMB 3.0 and SOFS is capable of handling the locks needed.When watching the cluster via MMC we can see that the VHD is opened by HyperV + a read-lock is placed on it.Thanks!

tomcsanyid ( 2016-02-15 13:15:12 +0300 )edit

This is the link I remembered: https://msdn.microsoft.com/en-us/library/windows/desktop/aa365202.aspx

tomcsanyid ( 2016-02-15 15:21:54 +0300 )edit

I was able to replicate your issue and submitted a new patch. Please let me know how this works for you: Downstream Cinder Liberty patch: https://github.com/cloudbase/cinder/commit/c57b85ea06d899f2a6d5fb6ecc37f0da3b6fca6b os-win patch: https://review.openstack.org/#/c/280640/

Lucian Petrut ( 2016-02-16 15:34:24 +0300 )edit

see more comments

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Live-Migration fails with cinder SMB backend - error code 5 Access Denied

3 answers

Comments

Comments

Your Answer

Question Tools

Stats

Related questions

Live-Migration fails with cinder SMB backend - error code 5 Access Denied edit

3 answers

Comments

Comments

Your Answer

Question Tools

Stats

Related questions

Live-Migration fails with cinder SMB backend - error code 5 Access Denied