New Question
0

Security Groups Not Applying

asked 2019-01-11 07:44:03 +0200

zimosworld gravatar image

updated 2019-01-14 22:31:07 +0200

Hi,

I have an issue with Security Group don't apply and all traffic is allowed to all VM's. After looking into it I was able to confirm there are no errors in the controller/network node logs but errors in the neutron logs on the compute nodes (Both Compute Nodes have the same error with all VM's Linux/Windows)

WMIJobFailed: WMI job failed with status 10. Error summary description: Failed to add device 'Ethernet Connection'.. Error description: 'instance-00000051' failed to add device 'Ethernet Connection'. (Virtual machine ID C5F26B60-6498-4685-AA09-F31FD95F46ED) Error code: 32768.

Running "Get-VMNetworkAdapterExtendedAcl" returns 0 results, I can run the below command manually which then when I run Get-VMNetworkAdapterExtendedAcl returns the record I added.

"Add-VMNetworkAdapterExtendedAcl -VMName "instance-00000051" -Action Allow -Direction Inbound -LocalPort "3389" -Protocol "TCP" -Weight 10 -Stateful $True"

Controller/Network Node: Centos 7 (Same Machine)
Compute Nodes (x2): Hyper-V Server 2016

Network Type: vlan
Physical Network: physnet1

==== Start Compute Node Neutron Config ====
[DEFAULT]
debug=true
control_exchange=neutron
transport_url=rabbit://***:***@controller:5672
log_dir=C:\OpenStack\Log
log_file=neutron-hyperv-agent.log
[AGENT]
polling_interval=2
physical_network_vswitch_mappings=*:external
enable_metrics_collection=false
enable_qos_extension=false
[SECURITYGROUP]
firewall_driver=hyperv
enable_security_group=true
==== End Compute Node Neutron Config ====

==== Start Compute Node Nova Config ====
[DEFAULT]
debug=false
compute_driver=compute_hyperv.driver.HyperVDriver
instances_path=C:\OpenStack\Instances
use_cow_images=true
force_config_drive=true
flat_injected=true
mkisofs_cmd=C:\Program Files\Cloudbase Solutions\OpenStack\Nova\bin\mkisofs.exe
allow_resize_to_same_host=true
running_deleted_instance_poll_interval=120
resize_confirm_window=5
resume_guests_state_on_host_boot=true
transport_url=rabbit://****:****@controller:5672/
rpc_response_timeout=1800
lock_path=C:\OpenStack\Log
vif_plugging_is_fatal=false
vif_plugging_timeout=60
block_device_allocate_retries=600
log_dir=C:\OpenStack\Log
log_file=nova-compute.log
use_neutron=true
[placement]
auth_strategy=keystone
auth_type=password
auth_url=http://controller:35357/v3
project_name=services
username=placement
password=****
project_domain_name=Default
user_domain_name=Default
os_region_name=RegionOne
[notifications]
[glance]
api_servers=http://controller:9292
[hyperv]
limit_cpu_features=false
config_drive_inject_password=true
qemu_img_cmd=C:\Program Files\Cloudbase Solutions\OpenStack\Nova\bin\qemu-img.exe
config_drive_cdrom=true
dynamic_memory_ratio=1
enable_instance_metrics_collection=false
vswitch_name=external
[os_win]
cache_temporary_wmi_objects=false
[rdp]
enabled=true
html5_proxy_base_url=https://VMS-1:4430
[neutron]
url=http://controller:9696
auth_strategy=keystone
project_name=services
username=neutron
password=****
auth_url=http://controller:35357/v3
project_domain_name=Default
user_domain_name=Default
os_region_name=RegionOne
auth_type=password
==== Start Compute Node Nova Config ====

==== Start Compute Node Neutron Log ====
2019-01-11 07:13:43.750 6436 DEBUG neutron.api.rpc.handlers.securitygroups_rpc [req-ce3168dc-58cf-4d3f-999a-09ccf7010e71 689e1532bbfa4e0a8143803416d2f63b 4ca6b6265e94471a9a5d7978f8520df1 - - -] Security group member updated on remote: [u'24a11562-281f-4ad6-ba5a-53e86ce01d16', u'94b5b78e-fba3-4e3d-91a5-4fb4dbc27ef5', u'215fb40a-3c59-4613-8014-0e64a7746989', u'6e536d86-49f9-404a-be6e-9d11d0bf93ee'] security_groups_member_updated C:\PROGRA~1\CLOUDB~1\OPENST~1\Nova\Python27\lib\site-packages\neutron\api\rpc\handlers\securitygroups_rpc.py:198
2019-01-11 07:13:43.780 6436 INFO neutron.agent.securitygroups_rpc [req-ce3168dc-58cf-4d3f-999a-09ccf7010e71 689e1532bbfa4e0a8143803416d2f63b 4ca6b6265e94471a9a5d7978f8520df1 - - -] Security group member updated [u'24a11562-281f-4ad6-ba5a-53e86ce01d16', u'94b5b78e-fba3-4e3d-91a5-4fb4dbc27ef5', u'215fb40a-3c59-4613-8014-0e64a7746989', u'6e536d86-49f9-404a-be6e-9d11d0bf93ee']
2019-01-11 07:13:43.921 6436 DEBUG networking_hyperv.neutron.agent.layer2 [req-ce3168dc-58cf-4d3f-999a-09ccf7010e71 689e1532bbfa4e0a8143803416d2f63b 4ca6b6265e94471a9a5d7978f8520df1 - - -] port_update received: d407f7bd-eacf-4cc6-ac13-011afb6c5c23 port_update C:\PROGRA~1\CLOUDB~1\OPENST~1\Nova\Python27\lib\site-packages\networking_hyperv\neutron\agent\layer2.py:436
2019-01-11 07:13:43.983 6436 DEBUG oslo_concurrency.lockutils [req-ce3168dc-58cf-4d3f-999a-09ccf7010e71 689e1532bbfa4e0a8143803416d2f63b 4ca6b6265e94471a9a5d7978f8520df1 - - -] Lock "n-hv-agent-port-lock-d407f7bd-eacf-4cc6-ac13-011afb6c5c23" acquired by "networking_hyperv.neutron._common_utils.inner" :: waited 0.000s inner C:\PROGRA~1\CLOUDB~1\OPENST~1\Nova\Python27\lib\site-packages\oslo_concurrency\lockutils.py:273 ...
(more)
edit retag flag offensive close merge delete

2 answers

Sort by » oldest newest most voted
0

answered 2019-01-16 04:12:42 +0200

zimosworld gravatar image

Hi,

Thanks for the response, no it still failed.

Below is what is returned, but something I forgot to add was on creation of instances the network is never attached, no Virtual switch or VLAN is selected (https://imgur.com/a/3lbNVYd screenshot from Windows Admin Center).

If I select a virtual switch and enter the VLAN the instance can connect to the internal and I can talk to the instances on any port without adding anything to the default security group or adding a another security group to the instance.

SwitchName
----------
external

Both the Controller/Network and Compute Nodes are using the Queens release, used the "HyperVNovaComputeQueens1700.msi" installer on the Compute Nodes.

Controller/Network pip package versions

pip freeze
alembic==1.0.5
amqp==2.3.2
ansible==2.7.4
ansible-modules-hashivault==3.11.0
anyjson==0.3.3
appdirs==1.4.3
asn1crypto==0.24.0
automaton==1.14.0
Babel==2.6.0
backports.ssl-match-hostname==3.5.0.1
bcrypt==3.1.5
Beaker==1.5.4
beautifulsoup4==4.6.3
boto==2.34.0
cachetools==3.0.0
castellan==0.17.0
certifi==2018.11.29
cffi==1.11.5
chardet==3.0.4
cinder==12.0.3
cliff==2.14.0
cmd2==0.8.9
configobj==4.7.2
configshell-fb==1.1.23
contextlib2==0.5.5
cryptography==2.4.2
cursive==0.2.1
debtcollector==1.20.0
decorator==4.3.0
defusedxml==0.5.0
deprecation==1.0
Django==1.11.10
django-appconf==1.0.1
django-babel==0.6.2
django-compressor==2.1
django-pyscss==2.0.2
dnspython==1.16.0
docutils==0.11
dogpile.cache==0.6.8
dogpile.core==0.4.1
enum-compat==0.0.2
enum34==1.1.6
ethtool==0.8
eventlet==0.20.0
extras==1.0.0
fasteners==0.14.1
fixtures==3.0.0
funcsigs==1.0.2
futures==3.2.0
futurist==1.8.0
gevent==1.1.2
glance==16.0.1
glance-store==0.23.0
google-api-python-client==1.4.2
greenlet==0.4.15
horizon==13.0.1
httplib2==0.12.0
hvac==0.7.2
idna==2.8
iniparse==0.4
ipaddress==1.0.22
IPy==0.75
iso8601==0.1.12
Jinja2==2.10
jmespath==0.9.3
jsonpatch==1.23
jsonpointer==2.0
jsonschema==2.6.0
jwcrypto==0.4.2
kazoo==2.2.1
keyring==5.7.1
keystone==13.0.2
keystoneauth1==3.11.2
keystonemiddleware==5.3.0
kitchen==1.1.1
kmod==0.1
kombu==4.2.2
ldappool==1.0
lesscpy===0.9j
linecache2==1.0.0
logutils==0.3.5
lxml==3.2.1
Mako==1.0.7
MarkupSafe==1.1.0
microversion-parse==0.1.4
mock==2.0.0
monotonic==1.5
msgpack==0.6.0
msgpack-python==0.4.6
munch==2.3.2
MySQL-python==1.2.5
ncclient==0.4.7
netaddr==0.7.19
netifaces==0.10.7
networking-hyperv==6.0.0
networkx==1.10
neutron==13 ...
(more)
edit flag offensive delete link more

Comments

Ok, so you're using Queens. Is the name of the vswitch correct ('external')? Also, could you provide the output of 'systeminfo.exe' and 'get-vmswitch''? I'm mostly interested in Windows build, hotfixes and network adapters. Please use paste.openstack.org or pastebin, it's easier to see the outut.

lpetrut gravatar imagelpetrut ( 2019-01-16 15:42:48 +0200 )edit

Yep 'external', below is the the paste links to those outputs: Systeminfo.exe - http://paste.openstack.org/show/742780/ get-vmswitch - http://paste.openstack.org/show/742781/

zimosworld gravatar imagezimosworld ( 2019-01-16 19:56:27 +0200 )edit

Are the instances created from scratch or just imported from backups? I remember a similar issue: https://ask.cloudbase.it/question/2758/wmi-job-failed-with-status-10-error-details-failed-to-add-device-ethernet-connection/#2759.

lpetrut gravatar imagelpetrut ( 2019-01-17 15:59:57 +0200 )edit

This script will help networking-hyperv pick up ports that were connected by a 3rd party: http://paste.openstack.org/raw/725812/

lpetrut gravatar imagelpetrut ( 2019-01-17 16:00:21 +0200 )edit

Another thing that I can think of is some caching issue, so please try setting the following in the neutron agent conf: [os_win] cache_temporary_wmi_objects=false

lpetrut gravatar imagelpetrut ( 2019-01-17 16:01:57 +0200 )edit

Some instances were backups and others were new created from cloud images (Centos and CoreOS) but all had the same issue. I try adding the config, restart the service and creating a new VM but no change :(

zimosworld gravatar imagezimosworld ( 2019-01-17 21:21:28 +0200 )edit
0

answered 2019-01-15 17:43:33 +0200

lpetrut gravatar image

Hi,

Looks like the agent couldn't connect the port to a vswitch. Did it manage to do so after subsequent retries? You can check using something like:

Get-VMNetworkAdapter -VMName <vm_name>| Select-Object -Property SwitchName

We're setting port ACLs only after connecting it to the configured vswitch.

To better understand the issue, it might help knowing which OpenStack version you're using.

edit flag offensive delete link more

Comments

Sorry my response was abit big for a comment so I added it an answer.

zimosworld gravatar imagezimosworld ( 2019-01-16 04:13:26 +0200 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

1 follower

Stats

Asked: 2019-01-11 07:44:03 +0200

Seen: 1,407 times

Last updated: Jan 16 '19