New Question
0

Live migration support when using permanent volumes not ephemeral disks

asked 2015-10-31 01:22:22 +0300

tomcsanyid gravatar image

Hi all,

I'd like to request a new feature for the nova-compute service: it should support Live Migration when not the default ephemeral disk (local storage) is used to store the VHD, but a permanent Cinder volume (for example in our case it is attached via iSCSI as a direct disk - so when creating the instance the image contents gets written to a new iSCSI volume which is then attached to the instance). Currently the nova-compute driver dies with the following error message:

2015-09-29 17:00:47.331 3924 DEBUG nova.compute.resource_tracker [req-e55effff-e3ba-47be-add7-33c335a703f2 - - - - -] Migration instance not found: Instance 5e8ce432-7792-452c-b966-d427367d0c4d could not be found.
Traceback (most recent call last):

  File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 422, in _object_dispatch
    return getattr(target, method)(*args, **kwargs)

  File "/usr/lib/python2.7/dist-packages/nova/objects/base.py", line 163, in wrapper
    result = fn(cls, context, *args, **kwargs)

  File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 559, in get_by_uuid
    use_slave=use_slave)

  File "/usr/lib/python2.7/dist-packages/nova/db/api.py", line 651, in instance_get_by_uuid
    columns_to_join, use_slave=use_slave)

  File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 233, in wrapper
    return f(*args, **kwargs)

  File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 1744, in instance_get_by_uuid
    columns_to_join=columns_to_join, use_slave=use_slave)

  File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 1756, in _instance_get_by_uuid
    raise exception.InstanceNotFound(instance_id=uuid)

InstanceNotFound: Instance 5e8ce432-7792-452c-b966-d427367d0c4d could not be found.

Steps to reproduce the issue: We try to perform a simple live migration from Horizon - simply going to Admin -> Instances, selecting an instance and then saying "Live migrate" and choosing the other hypervisor as target. The instance was created by using the Boot from an image (create volume) option. We tried this so far with Ubuntu guests, but I guess this shouldn't matter. What we expect from the nova-compute service: to behave the same as the KVM service does or the way HyperV behaves with ephemeral-local disks used. I'm not sure how much of this is implemented already in Cinder, but this is kind of the steps we thought it should go through: 1. Attach volume to target hypervisor 2. Do HyperV's live migrate 3. Detach volume from previous hypervisor

Thanks a lot! Let us know if you need anything - more logs, testing code, whatever, we are ready to assist as much as we can!

Domi

edit retag flag offensive close merge delete

3 answers

Sort by » oldest newest most voted
0

answered 2015-11-10 18:32:34 +0300

alexpilotti gravatar image

This is an already supported scenario. We're going to reproduce your steps and validate if there are pending issues.

edit flag offensive delete link more
0

answered 2015-11-10 17:53:25 +0300

tomcsanyid gravatar image

Bump, I'd really like you to have a look at this issue if possible. Also would love to help, if possible, I'm quite sure many people are affected by this.

edit flag offensive delete link more
0

answered 2015-11-11 14:02:38 +0300

updated 2015-11-25 06:50:53 +0300

This is odd. Could you please tell me what branch are you using? Also, if you can paste the full logs from c:\Openstack\Logs\nova-compute.log from both hosts, that would be helpful. Please use pastebin or paste.openstack.org

EDIT:

Sorry for answering so late, my original response didn’t get posted from my phone for some reason.

I tried to reproduce, but couldn’t find any issue. I tried with both Kilo and Liberty branches, both work fine. You didn’t said which branch you were using, but judging by the line numbers in the trace I assumed Kilo.

You posted the logs from both Hyper-V Nodes, but unfortunately there is not much information there. I suspect that this exception is masking the real problem, even so, it is odd that there is no Hyper-V error in the logs.

Given that I can confirm that live-migration works, I suspect that the problem is related to the configuration of the Hyper-V nodes. One thing you can try is to start a migration and immediately after it fails, check to see if there is any error with Event Viewer on both Hyper-V Hosts. You should look under “Application and Services\Microsoft\Windows\Hyper-V VMMS”. Also, please set your log level to debug=true in the nova.conf file on both hosts, that should give us a clearer picture of what is happening.

edit flag offensive delete link more

Comments

Sure thing, sorry for the delayed answer: Originating server (has the VM currently): http://pastebin.com/VZKQzMww Target server (should have the VM after migration): http://pastebin.com/SiYA5HuA I used pastebin, because paste.openstack.org limited my input to only 230 lines. Thank you!

tomcsanyid gravatar imagetomcsanyid ( 2015-11-17 11:27:53 +0300 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

1 follower

Stats

Asked: 2015-10-31 01:22:22 +0300

Seen: 398 times

Last updated: Nov 25 '15