New Question
0

Cinder local loopback

asked Mar 31 '17

jrack gravatar image

I am using cinder AZs to expose local disk for volume services running on a Hyper-V 2016 instance. Using the mitaka cinder driver it can do the local detect (for S2D efficiency) which works most of the time. But after a while I will start getting what I would imagine is a token expiration that breaks attaching volumes to instances.

2017-03-31 13:07:02.532 3436 ERROR cinder.volume.drivers.remotefs [req-9695f713-81a8-4b0e-b04f-1b9d26f9bf2e - - - - -] Exception during mounting Unable to mount SMBFS share: \localhost\Cinder$ WMI exception: <x_wmi: a="" specified="" logon="" session="" does="" not="" exist.="" it="" may="" already="" have="" been="" terminated.="" &gt;<="" p="">

I am betting I just need to do a more precise configuration in my New-SmbShare with perms or something, but wasn't sure. And HV server is a bit of a black box so harder to track down might might be going on. As a note volume creation and nova ops all work, but that attachment is what is blowing up.

Preview: (hide)

1 answer

Sort by » oldest newest most voted
0

answered Apr 4 '17

lpetrut gravatar image

I think that happens when it tries to cleanup a failed mount. Could you please include the whole trace? Thank you.

I guess you're using CSVs, right? We're usually placing scale-out shares on top of them, simplifying share address resolution. I'd suggest using share locations such as //<sofs-name>/<volumes_share>. Those will be resolved to their local mount points.

Lucian

Preview: (hide)
link

Comments

Nope just a local SMB share configured as //localhost/Cinder$ no CSV or anything complex. Just trying to use the local storage in the box. I will flip on debug as the local cinder-volume-smb just contains the above over and over.

jrack gravatar imagejrack (Apr 4 '17)

Flipped on verbose in cinder-volume-smb.conf and nothing really enlightening. http://paste.openstack.org/show/605417/

jrack gravatar imagejrack (Apr 4 '17)

And added a debug=true statement to the conf and oddly I did bsod the box, but after booting still getting the context error. http://paste.openstack.org/show/605425/

jrack gravatar imagejrack (Apr 4 '17)

I mean honestly I don't even want to use the SMB share which I thought the mitaka driver handled internally to make the S2D lookup try local first.

jrack gravatar imagejrack (Apr 4 '17)

As a workaround for now, just set the actual hostname instead of 'localhost'. This should let the driver pick it up as a local share (which is simply resolving addresses). I'm going to send a fix ASAP, sorry for the inconvenience. Let me know how it went.

lpetrut gravatar imagelpetrut (Apr 5 '17)

I had the same thought yesterday, but it didn't resolve the core issue with the attach. I am not getting the WMI error so that is good. Likely in my TP5 test I was using hostname, but used localhost in the deploy script thinking it wouldn't matter. http://paste.openstack.org/show/605513/

jrack gravatar imagejrack (Apr 5 '17)

So I figured out one err... using gen1s and you can't attach when powered on. so I need to handle that better as the smbfs driver seems to just do a terminate with no error. After powering down though I get the same wmi session err even w \\hostname\share$

jrack gravatar imagejrack (Apr 5 '17)

So the issue is on the Nova side. Could you please check the logs and paste them here? What Nova MSI are you using? You can in fact attach disks to powered on Gen1 VMS. It's just that you can't attach/detach IDE disks in this case (we're using the SCSI bus in this case, except for BFV).

lpetrut gravatar imagelpetrut (Apr 6 '17)

Not sure on that, but it's the mitaka release? I had tested using hv manager to try and attach the vmdk and that's where I noticed it would fail if the g1 vm's were on. I know I need to set the metadata for g2. I'd there any more granular debugging in cinder?

jrack gravatar imagejrack (Apr 6 '17)

On pto till next week, but we'll try to grab versions and any Nova logs that corelate to the attach events when I am back in the office.

jrack gravatar imagejrack (Apr 6 '17)

So, at this point, everything works fine on the Cinder side, there's nothing else to debug there. When Nova fails to attach a disk, it will request Cinder to terminate the volume connection, which is expected.

lpetrut gravatar imagelpetrut (Apr 6 '17)

Ahh I follow now. Honestly didn't even think of checking Nova since it was all volume work, but makes more sense now. We'll try to remote in and take a look.

jrack gravatar imagejrack (Apr 6 '17)

You can issue the following powershell command to fetch the exact Nova product version: http://paste.openstack.org/raw/605618/ The idea is that from time to time we release updated MSIs for stable releases.

lpetrut gravatar imagelpetrut (Apr 6 '17)

Man feel like a dunce now as there is some solid info in there. http://paste.openstack.org/show/sQzF5Fox30oXmi3VMaOe/ including the session expire error. I don't have an A record for this lab and it is not a domain member, but maybe just create a hosts entry if the hostname isn't cutting it?

jrack gravatar imagejrack (Apr 6 '17)

And compute version 13.0.0.0

jrack gravatar imagejrack (Apr 6 '17)

@lpetrut any additional thoughts on this one?

jrack gravatar imagejrack (Apr 10 '17)

From Nova and trimmed to fit comment max. HyperVException: WMI job failed with status 10. Error details: Failed to add device 'Physical Disk Drive'. - 'instance-0000048b' failed to add device 'Physical Disk Drive'. (Virtual machine ID ...) - Error code: 32773

jrack gravatar imagejrack (Apr 10 '17)

That 32773 seems to reference a bad param (https://msdn.microsoft.com/en-us/library/hh850023(v=vs.85).aspx) but couldn't find any more details on what params were used in evtLogs.

jrack gravatar imagejrack (Apr 10 '17)

Sorry for the delayed answer. For some reason, the Nova Mitaka MSI hasn't been updated in a while and a known issue remained there. We've been testing a new Mitaka release for the last few days, so I think today we'll update the MSI from our website, including the fix for this issue as well.

lpetrut gravatar imagelpetrut (Apr 11 '17)

Is there a patchId for the fix I could snag and see if it covers it? If not I can hold tight, but figured I could make the best of the down time before the push is complete.

jrack gravatar imagejrack (Apr 11 '17)

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

Stats

Asked: Mar 31 '17

Seen: 564 times

Last updated: Apr 04 '17