Tuesday, 2020-07-14

*** d34dh0r53 has quit IRC00:09
*** d34dh0r53 has joined #openstack-ansible00:12
*** cshen has quit IRC00:25
*** gyee has quit IRC00:52
*** cshen has joined #openstack-ansible00:52
*** also_stingrayza has joined #openstack-ansible01:23
*** rh-jelabarre has quit IRC01:23
*** stingrayza has quit IRC01:25
*** cshen has quit IRC01:36
*** spatel has joined #openstack-ansible01:55
openstackgerritws proposed openstack/openstack-ansible master: Add rsync package for New CentOS-8 minimal installs  https://review.opendev.org/74084302:01
*** cshen has joined #openstack-ansible02:04
*** spatel has quit IRC02:34
*** spatel has joined #openstack-ansible02:51
*** cshen has quit IRC02:54
*** schwicht has quit IRC03:09
*** schwicht has joined #openstack-ansible03:15
*** schwicht has quit IRC03:20
*** cshen has joined #openstack-ansible03:22
*** arkan has joined #openstack-ansible03:30
*** schwicht has joined #openstack-ansible03:51
*** arkan has quit IRC03:51
*** miouge36253686 has quit IRC03:54
*** mbuil has quit IRC03:54
*** miouge362536868 has joined #openstack-ansible03:54
*** mbuil has joined #openstack-ansible03:55
*** schwicht has quit IRC04:06
*** cshen has quit IRC04:19
*** evrardjp has quit IRC04:33
*** evrardjp has joined #openstack-ansible04:33
*** cshen has joined #openstack-ansible04:51
*** arkan has joined #openstack-ansible05:03
*** spatel has quit IRC05:09
jannojrosser: no worries, thank you05:28
*** udesale has joined #openstack-ansible05:29
*** cshen has quit IRC05:46
*** arkan has quit IRC05:51
openstackgerritChandan Kumar (raukadah) proposed openstack/openstack-ansible master: [WIP] improve ironic tempest testing  https://review.opendev.org/73650705:59
openstackgerritChandan Kumar (raukadah) proposed openstack/openstack-ansible master: [WIP] improve ironic tempest testing  https://review.opendev.org/73650705:59
*** arkan has joined #openstack-ansible06:16
*** cshen has joined #openstack-ansible06:19
*** halali_ has quit IRC06:30
*** halali_ has joined #openstack-ansible06:51
*** this10nly has joined #openstack-ansible06:55
*** cshen has quit IRC06:57
*** cshen has joined #openstack-ansible07:13
*** halali_ has quit IRC07:34
*** tosky has joined #openstack-ansible07:37
*** arkan has quit IRC07:51
*** arkan has joined #openstack-ansible07:52
*** donnyd has quit IRC08:53
*** donnyd has joined #openstack-ansible08:53
*** also_stingrayza is now known as stingrayza08:58
*** cshen has quit IRC08:59
ionihi guys, what's the procedure to update ceph ? i've updated an aio system from train nto ussuri and ceph wasn't updated on mons09:12
ioniit was updated only on osds09:12
jrosserioni: this may be useful https://review.opendev.org/#/c/711440/09:21
ionijrosser: ok, i'll apply the patch and retry09:23
ionijrosser: is useful for all branches not only for stable/stein09:24
*** cshen has joined #openstack-ansible09:26
jrosserioni: if you can check that works we should look to merge and backport it09:29
*** soren has quit IRC10:01
*** soren has joined #openstack-ansible10:03
*** grantza has quit IRC10:04
*** cshen has quit IRC10:08
ionijrosser: seems to work, i only used the upgrade yml10:09
*** bl0m1 has quit IRC10:22
*** gouthamr has quit IRC10:22
*** gouthamr has joined #openstack-ansible10:23
*** bl0m1 has joined #openstack-ansible10:23
*** Jeffrey4l has quit IRC10:23
*** Jeffrey4l has joined #openstack-ansible10:24
*** cshen has joined #openstack-ansible10:38
*** cshen has quit IRC10:43
openstackgerritMerged openstack/ansible-role-systemd_service stable/ussuri: Remove fail: tasks  https://review.opendev.org/73956610:51
*** arkan has quit IRC10:51
*** gokhani has joined #openstack-ansible10:51
*** cshen has joined #openstack-ansible11:18
*** arkan has joined #openstack-ansible11:19
*** cshen has quit IRC11:22
*** arkan has quit IRC11:52
*** arkan has joined #openstack-ansible11:52
mgariepymorning everyone11:55
*** spatel has joined #openstack-ansible11:58
*** schwicht has joined #openstack-ansible12:03
*** spatel has quit IRC12:03
*** rh-jelabarre has joined #openstack-ansible12:09
*** udesale_ has joined #openstack-ansible12:22
*** udesale has quit IRC12:25
*** cshen has joined #openstack-ansible12:53
*** spatel has joined #openstack-ansible12:58
*** cshen has quit IRC13:03
jrossermgariepy: CeeMac are either of you using a network thats not br-mgmt as the hosts in /etc/openstack_deploy/openstack_user_config.yml ?13:30
jrosserlike a dedicated provisioning network13:30
*** pcaruana has quit IRC13:31
*** pcaruana has joined #openstack-ansible13:43
spatelmorning folks13:48
spateljrosser: you may help with this - https://review.opendev.org/#/c/740513/13:49
spatelThis patch is totally good but somehow build worker not picking up correctly and still showing same error13:50
spatelit seem build workers are having issue..13:50
CeeMacJrosser yes I use an OOB management network for my hosts connections13:51
*** arkan has quit IRC13:51
jrosserCeeMac: i wonder if you can do a sanity check for me13:53
jrosserwe are just looking at this here https://opendev.org/openstack/openstack-ansible/src/branch/master/playbooks/common-playbooks/nova.yml#L69-L7413:53
jrosserthat does some magical stuff rooting round in the hostvars to pick an address for the nova host13:53
jrosserit then sticks it here https://opendev.org/openstack/openstack-ansible/src/branch/master/playbooks/common-playbooks/nova.yml#L14513:54
jrosserwhich means that IP ends up in places like this https://opendev.org/openstack/openstack-ansible-os_nova/src/branch/master/templates/nova.conf.j2#L4113:55
jrosserwould be interesting to see if you have the IP you expect to have there in nova.conf13:55
CeeMacSo that's management as in br-mgmt as opposed to the host local management address?13:55
jrosserthat is a very good question :)13:56
CeeMacI'm not at my computer right now but I'll double check when I get back to my desk later. Working split shifts atm13:56
jrosserah ok, no worries13:56
jrosserwe are thinking that "management_address" is wrong and does not exist in container_networks in hostvars13:57
CeeMacI'll be back online by 1700 latest13:57
jrosserthere is a fallback which would be ansible_host13:58
jrosserso the address of the OOB network ends up in places that it maybe shouldnt13:58
*** dave-mccowan has joined #openstack-ansible14:02
*** cshen has joined #openstack-ansible14:08
CeeMacHas anything changed in how this works since rocky or should it still be the same14:17
*** this10nly has quit IRC14:26
*** cshen has quit IRC14:27
mgariepyjrosser, not me.14:31
*** dave-mccowan has quit IRC14:33
jrosserCeeMac: it should be the same14:35
CeeMacOk14:39
*** dave-mccowan has joined #openstack-ansible14:39
*** cshen has joined #openstack-ansible14:54
*** arkan has joined #openstack-ansible15:00
jrosserspatel: the issue with your patch to gnocchi is that the test fails when installing ceilometer, becasue this is not merged https://review.opendev.org/#/c/739400/15:09
jrosserjust here the error is in os_ceilometer https://zuul.opendev.org/t/openstack/build/6424de617c0240719b18a44eb350bae1/log/job-output.txt#1364315:10
spateljrosser: you are saying its dependency and without merge it won't work?15:10
jrosserit is a dependacy yes, for the gnocci role see the test scenario here https://zuul.opendev.org/t/openstack/build/6424de617c0240719b18a44eb350bae1/log/job-output.txt#523815:12
jrosser'telemetry' means it deploys the whole telemetry stack15:12
spatelhmm that make sense15:13
jrosseri do not think we should be too concerned about the peripheral things like this15:13
jrosserwe should get centos-8 basic stuff passing on ussuri first15:14
spateljrosser: yes, also this patch is very minor change just rename of rehat-7.yml file and in some case just package name match for centos815:14
spateli think its totally safe to merge if people review them15:14
jrosseryou can't :)15:14
jrosserunless the tests pass it will never merge15:15
jrosseror the tests are set to not-voting15:15
jrosserthere is no "merge now regardless"15:15
spatelin this case what do you suggest?15:15
spatelalso we are very close to centos-8 beta release, as i am seeing all major patches has been merged15:17
jrosserso two different things here15:17
jrosseryour test is failing for gnocchi becasue os_ceilometer patch is not yet merged15:18
jrosserand to merge the os_ceilometer patch we need to merge the gnocchi one15:18
spatel:) chicken egg15:18
jrosserso that is clearly an impossible situation and the only was to resolve that is to set the centos-8 tests to not voting and merge some of the patches with a failing test15:18
jrosser*but*15:18
jrosseryou can still see beforehand if it is going to work15:19
jrosserfor example, you can make the os_ceilometer patch also take account of the unmerged patch to gnocchi15:19
jrossermake it do a "what if" test15:20
jrosserhere is an unrelated example https://review.opendev.org/#/c/618305/15:20
jrossersee the 'Depends-On' line in the commit message, this means two things15:20
jrosser"do not allow this patch to merge before the depends-on one is merged"15:21
jrosser"when testing this patch also apply the depends-on patches in the test"15:21
jrosserso if you edit the commit message of the os_ceilometer patch to include depends-on lines for the other parts of the telemetry stack we will see further on how far it works15:22
spatelhmm! interesting.. let me see15:23
spateljrosser: thanks, let me try and i will ping you back if any question15:24
jrossersure no worries15:24
jrosserthis is one of the powerful bits of testing with zuul that you can make a change in one repo and test the effect that would have in another, before merging anything15:25
arkanHi guys, how are you all ? I hope you are doing well.15:26
arkanI got a hint from @CeeMac to use magnum in stead of using terraform + kubespray, so I went and installed it :))15:26
arkanwith barbican also15:26
arkandoes coreos works with magnum (stable/train) ?15:28
arkanI've created the template successfully on horizon15:28
arkanand also on cli15:28
arkanonly the cluster when it is in creation in progress15:29
arkanthen it will through this error15:29
arkanERROR: The Parameter (fixed_network_name) was not defined in template.15:29
arkanand I have specified it in my template15:29
arkanfrom horizon15:30
arkanthere are combo boxes for both "Fixed Network" and "Fixed Subnet" in the Network section15:31
*** udesale_ has quit IRC15:35
arkanhumm15:38
arkanmaybe this is have to do with coreos15:38
arkani will try fedora-atomic15:38
arkanI will need to download it first15:39
*** gyee has joined #openstack-ansible15:50
*** cshen has quit IRC15:50
admin0is adding  magnum_infra_hosts:  and then rerunning bootstrap/setup is enough to get magnum support for deploying kubernetes ?15:52
*** spatel has quit IRC15:54
CeeMacjrosser: in nova.conf my_ip is set to the OOB management IP15:54
jrosserCeeMac: i'm not completely clear on the detail of why just yet but the upshot is that live migration seems to happen across that network15:56
jrosserwe had some migrations that took eternity to complete and that was why15:57
CeeMacoh16:04
CeeMacwell, thats not goodf16:04
jrosserwe've not fully understood whats happening as the nova docs are a bit confusing16:06
*** sshnaidm is now known as sshnaidm|afk16:08
CeeMacit should be using br-mgmt presumably?16:08
jrosseryes so it starts here https://github.com/openstack/openstack-ansible/blob/master/playbooks/common-playbooks/nova.yml#L69-L7416:12
jrosserand that takes the dict container_networks for the host, and tries to find the key "managment_address" in there16:13
jrosserbut i think that is wrong as i've not found good reference to managment_address in the code or documentation16:14
jrosserthat would come about by having a section "management" in cidr_networks in openstack_user_config.yml which none of the examples have16:14
CeeMacyes, i was just looking at dynamic-address-fact.yml trying to match it back16:17
CeeMacis not pulling the "management_address" value out of the query in dynamic-address-fact.yml16:19
CeeMacwhich is looking for "\"ansible_{{ hostvars[inventory_hostname] in metal_query?16:20
jrosserif you do ansible -m debug -a 'var=container_networks' then i don't think you'll find a section in the dict for managment_address16:21
jrosserwould be interesting to see if you do16:21
jrosseransible <host> -m debug ....16:22
CeeMacok16:22
CeeMacwas just looking in openstack_inventory and the IP is references in ansible_host and container_address and container_networks: container_address: address16:23
*** cshen has joined #openstack-ansible16:23
*** aedc_ has joined #openstack-ansible16:24
*** aedc_ has quit IRC16:24
CeeMacno its definitely container_address now management_address16:25
arkanadmin0: setup-hosts, setup-infrastructure, os-horizon, os-magnum16:25
arkanadmin0: it has problem with coreos16:25
jrosserCeeMac: for a metal host the query should look at the "bridge" field16:26
jrosserthen take the ip from the bridge of that name16:26
CeeMacah, i couldn't work it if it was using | as 'or' in an order of priority16:26
CeeMacbut i dont have my management interface in a bridge....16:27
*** aedc has quit IRC16:27
arkanadmin0: now the problem disappeared when I used fedora-atomic-27 image but another problem I have now in magnum container: Failed to load default keystone auth policy: FileNotFoundError: [Errno 2] No such file or directory: '/etc/magnum/keystone_auth_default_policy.json'16:27
CeeMacjson_query(find_bridge) <- that bit?16:27
jrosser:) br-mgmt is kind of expected i think16:27
CeeMacright16:28
jrosserbut also then just look here https://docs.openstack.org/nova/latest/configuration/config.html16:28
CeeMacexcept it isnt, hmm16:28
*** cshen has quit IRC16:28
jrosserat the definition of live_migration_uri16:28
jrosserthen we all collectively go "oh no!"16:28
jrosserCeeMac: anyway just good to see that for you with a similar setup it also makes the same outcome as us16:29
jrosserwe're going to have to (tomorrow) dive into the nova code and see really how that migration uri is used16:29
CeeMacit definitely doesn't read well16:30
CeeMacIf live_migration_inbound_addr value is not None and live_migration_tunnelled is False, the ip/hostname address of target compute node is used instead of live_migration_uri as the uri for live migration.16:30
CeeMacand the hosts file is built from the inventory/management address, in my case the OOB one16:31
CeeMacjrosser: just looking at openstack_inventory again16:34
CeeMaccontainer_address has the OOB management IP but br-mgmt as the bridge16:35
jrosseryes, same here16:35
CeeMacso looking for br-mgmt would return the wrong IP in that case16:35
jrosserso for us we would get part of the way there by changing "management_address" (confusingly) to "container_address"16:35
jrosserin this case for nova i think the ip off br-mgmt is the thing you want it to be using16:36
CeeMacbut then why isn't the dynamic inventory loading the br-mgmt IP as the container-address16:36
jrosserfor the compute host?16:36
CeeMacyes16:36
jrosseri see something like this http://paste.openstack.org/show/795918/16:38
CeeMacthe the ansible_host is the OOB mgmt IP if you have similar setup to me16:39
jrosseronly for the things that are hosts though, the 10.84 ones16:40
CeeMac?16:40
CeeMacgot you16:40
jrosser10.80 is the openstack mgmt network16:40
jrosserwhich kind of makes sense for ansible_host16:40
CeeMacin the 'standard' way of deploying yes16:40
jrosserbut not so much when the dynamic address lookup tries to lookup something non-existing (in any deployment) and falls back to ansible_host16:41
jrossernormally this would be OK as mgmt address == ansible address16:41
CeeMachowever, with  management_bridge: "br-mgmt" set would it not gather the br-mgmt ip for the container address?16:41
jrosseri'm not sure that variable is actually doing anything though16:42
CeeMaci see, so potentially its falling back to "non_metal_query: "container_networks.{{ network_address }}.address" first16:42
CeeMacwhich in our case is populated by the OOB IP not the br-mgmt IP for metal hosts16:43
*** cshen has joined #openstack-ansible16:43
CeeMacactually, i missed the find_bridge bit16:44
jrosseryeah :) it's mind boggling isnt it16:44
CeeMacsaying that, I don't think the issue is necessarily in dynamic-address-fact.yml?16:45
CeeMacif the wrong IP is in the dynamic inventory for the mr-mgmt bridge in the first place?16:45
CeeMac*br-mgmt16:45
jrosserthe issue is that its trying to find a somthing called "management_address"16:45
jrosserwhich will never exist16:46
CeeMaci still haven't quite worked that bit out :)16:46
CeeMacok16:46
CeeMacnow i have16:46
jrosserhere https://github.com/openstack/openstack-ansible/blob/master/playbooks/common-playbooks/nova.yml#L7216:46
CeeMacso its replacing {{ network_address }} with management_address in the queries16:46
CeeMaci was trying to work out which way it was passing the variable16:47
CeeMacim a bit rusty :)16:47
jrosseryes its setting a fact "management_address" by looking up "management_address" in container_networks then performing the jq on it to get the address16:47
jrosserwhich is umm, 'esoteric'16:48
CeeMacwhich, as you say, isn't going to get anywhere16:48
CeeMacgood word :)16:48
*** sshnaidm|afk is now known as sshnaidm16:48
jrosserright - so i think this works for standard deploys becasue it falls back to the right thing16:48
jrossersort of by accident16:48
CeeMacyes16:49
CeeMacbut using the ansible_host value16:49
CeeMacs/but/by16:49
jrosserright16:49
CeeMacas the rest of the queries are going to fail16:49
jrosserwe'll dig into this a bit more, becasue it seems that getting the value you want in my_ip isnt enough to steer the migration traffic to br-mgmt16:50
jrosser*tomorrow16:50
*** arkan has quit IRC16:51
CeeMacyes16:51
CeeMacthats something else again then16:51
CeeMacbut those live_migration vars are pretty out there too16:51
*** arkan has joined #openstack-ansible16:52
openstackgerritws proposed openstack/openstack-ansible master: Add rsync package for New CentOS-8 minimal installs  https://review.opendev.org/74084316:56
CeeMacok, so my_ip is set by nova_management_address which is set from management_address16:58
CeeMacinteresting16:59
CeeMacjrosser: https://opendev.org/openstack/openstack-ansible-os_nova/src/branch/master/tests/group_vars/all_containers.yml#L1716:59
*** npalladium has joined #openstack-ansible17:03
CeeMacso nova.conf.j2 sets live_migration_tunnelled = true, and configures live_migration_uri.17:15
CeeMacwhich means that live_migration_inbound_addr won't be used17:15
CeeMacbut %s replaced with the migration target hostname, which in our case will resolve to the OOB management address17:15
CeeMacbut then what could you use to replace %s with that would resolve the br-mgmt ip dynamically as part the uri buildout17:17
CeeMacoh, actually hosts has an entry for both the OOB IP and br-mgmt17:19
CeeMacso thats going to be confusing17:19
CeeMacis that going to round-robin on resolution requests?17:20
jrosseri am wondering if using %s is a bad idea17:24
jrosserbecasue it makes some assumptions about what dns may or may not say17:24
CeeMacyes17:24
jrosserit might be better to template out the actual IP that we want then there is no confusion17:25
CeeMacbut what could you replace it with that would be dynamic across target migration hosts?17:25
jrosser{{ nova_managment_address }} i guess, if that could be fixed to be the right thing17:25
CeeMacas i assume the URL would change based on where you are migrating to?17:25
jrosseri think its the "from"17:25
CeeMacthis isn't proxied via haproxy?17:26
jrosserto the "to" would connect to this, but this is kind of where my understanding is running out17:26
jrosserits an ssh17:26
CeeMac"qemu+ssh://nova@%s/system?no_verify=1&keyfile={{ nova_system_home_folder }}/.ssh/id_rsa"17:26
CeeMacis what it is being set to17:26
jrosserhttps://github.com/openstack/openstack-ansible-os_nova/blob/master/templates/nova.conf.j2#L23617:26
CeeMacwhich is a destination path as I understand it17:26
CeeMacyes17:27
jrosseroh right17:27
CeeMac"Override the default libvirt live migration target URI"17:27
CeeMachence the conundrum17:27
CeeMacas that needs to be explicitly set for the destination host in the migration request17:28
jrosserso i figure that what matters is from the "initiator" or whatever we want to call, reaching that URI uses the right interface17:28
CeeMacyes, based on what address the destination host address resolves to17:28
jrosserhrrm, i'm a bit confused about it really17:29
jrosserif that uri and %s is used every time a migration is done, or else it's saying "i am this"17:29
CeeMacyou could hack it by adding a unique hosts entry based on <hostname>_uri and setting that to the correct ip, but will hacking that into the uri work with %s_url17:30
CeeMac"Any included ā€œ%sā€ is replaced with the migration target hostname."17:30
CeeMaci think its set at the migration creation17:30
*** cshen has quit IRC17:31
jrosserlive_migration_inbound_addr sounds like a much more useful construct17:32
jrosserbecasue it's data about the host itself17:32
CeeMacyou'd have to unset live_migration_tunnelled then17:33
CeeMacor do you mean wrt my hack?17:34
CeeMacok, i'm catching up on your confusion now17:35
CeeMacone sec17:36
*** mgariepy has quit IRC17:36
CeeMacok so its actually written in to nova.conf with the %s too17:37
CeeMacso, i guess the question is, is live_migration_uri on compute1 used to work out the correct path to a migration target17:37
CeeMacor is it provided to the compute node initiating the migration in a "send traffic here" kind of way17:38
jrosseroh my goodness https://review.opendev.org/#/c/456571/17:39
CeeMaclive_migration_uri is actually deprecated anyway.....17:39
CeeMacah, you found something similar a different way :)17:40
CeeMacdigging in think having tunnel enabled is still going to cause some issues17:46
CeeMac"It is typically only necessary to specify this URI if the destination host has multiple interfaces and a specific interface is required to transmit migration data"17:48
CeeMacthat seems promising17:48
CeeMachttps://libvirt.org/html/libvirt-libvirt-domain.html#VIR_MIGRATE_PARAM_URI17:48
jrosseri think i'm going to leave this until tomorrow17:48
jrosserbut if you have any useful insights for when we pick it up in the morning do let me know17:48
CeeMaci'll be clocking off shortly too, but i'll take a peak again first thing17:49
jrosserok, no worries, catch up tomorrow17:49
CeeMacsure, have a good evening :)17:49
CeeMacI'll go back to lurking17:50
*** schwicht has quit IRC17:56
*** schwicht has joined #openstack-ansible18:02
*** cshen has joined #openstack-ansible18:04
*** d34dh0r53 has quit IRC18:08
*** cshen has quit IRC18:08
*** d34dh0r53 has joined #openstack-ansible18:15
*** mgariepy has joined #openstack-ansible18:22
*** rmcall has joined #openstack-ansible18:30
*** spatel has joined #openstack-ansible18:37
CeeMacjrosser: for the morning (because apparently I'm incapable of letting things go once I'm curious). I think setting live_migration_inbound_addr to nova_management_address and setting live_migration_tunnelled to false could be the way forward. That leaves changing 'management_address' to 'container_adress', and ensuring the correct br-mgmt ip is collected on the container_networks dict for all use cases (being19:01
CeeMacstandard collapsed mgmt and OOB mgmt mainly)19:01
*** mgariepy has quit IRC19:03
*** npalladium has quit IRC19:12
*** cshen has joined #openstack-ansible20:01
*** also_stingrayza has joined #openstack-ansible20:03
*** cshen has quit IRC20:05
*** stingrayza has quit IRC20:07
*** arkan has quit IRC20:51
*** markvoelker has joined #openstack-ansible21:23
CeeMacAlso, need to re-read and try and understand what is happening in this block https://opendev.org/openstack/openstack-ansible/src/branch/master/osa_toolkit/generate.py#L66521:26
*** markvoelker has quit IRC21:27
*** spatel has quit IRC21:44
*** spatel has joined #openstack-ansible21:45
*** cshen has joined #openstack-ansible22:02
towhi guys, we are kind of struggling with 20.1.3, the setup-openstack deployment gets stuck at different points  for the same reason, git not found inside some services containers (placement, heat-api, aodh). If we get into the containers and install it manually, then it seems to go through. Haven't found anything in the bugtracker.22:04
*** spatel has quit IRC22:04
*** cshen has quit IRC22:07
*** spatel has joined #openstack-ansible22:23
*** spatel has quit IRC22:23
*** tosky has quit IRC22:50

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!