Tuesday, 2022-06-21

opendevreviewOpenStack Proposal Bot proposed openstack/openstack-ansible master: Imported Translations from Zanata  https://review.opendev.org/c/openstack/openstack-ansible/+/84687103:44
*** ysandeep|out is now known as ysandeep04:34
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible master: Bump roles and services for Yoga RC1  https://review.opendev.org/c/openstack/openstack-ansible/+/84679706:06
noonedeadpunksnadge: could it be some race condition, that check was faster then service actually came up?06:14
jrosser_would that be https in that case?06:18
jrosser_hmm06:19
noonedeadpunkwe defaulted internal to https only in CI06:20
noonedeadpunkit's done in bootstrap-host06:20
noonedeadpunkhttps://opendev.org/openstack/openstack-ansible/src/branch/master/tests/roles/bootstrap-host/templates/user_variables.aio.yml.j2#L26206:21
noonedeadpunkso for normal deployments, internal is not protected by default06:21
jrosser_oh right yes we did talk about this before06:22
noonedeadpunkI guess we did that because of non-trivial upgrade path06:22
jrosser_and if we needed to drop CI back to http in order to then test upgrades06:22
noonedeadpunkfor upgrades we actually test http indeed now06:23
noonedeadpunkI wonder if for Z we should just enable https by default and create an upgrade playbook that would preserve http in variables06:24
noonedeadpunkOr we can land that for Y :D06:25
noonedeadpunkBut I'd rather not06:25
opendevreviewMerged openstack/openstack-ansible master: Imported Translations from Zanata  https://review.opendev.org/c/openstack/openstack-ansible/+/84687106:38
*** ysandeep is now known as ysandeep|afk06:42
noonedeadpunkhttps://review.opendev.org/c/openstack/ansible-config_template/+/846391 finally passed. I wonder if it will just work as expected 06:59
opendevreviewDamian Dąbrowski proposed openstack/openstack-ansible-os_tempest master: Define fixed_network_name when dynamic credentials are not used  https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/84695407:31
snadgei dont think so.. it retries the test 10 times or something then gives up07:43
noonedeadpunkbut manually it works?07:45
snadgei haven't looked into it much further than that.. other than trying to connect to that ip and port, which it does07:45
snadgethe error isn't connection refused though.. its service unavailable07:46
snadgei just used telnet07:46
noonedeadpunkwell it sounds like haproxy marked backend as down for some reason07:46
noonedeadpunkaha07:46
noonedeadpunkso yeah, worth checking why haproxy marks it as down and then check keystone itself. I guess you mentioned Xena, so it should be apache status and keystone-api service07:47
jrosser_snadge: try curl/wget against the backend rather than telnet, haproxy is looking for a proper http response rather than just if the port is connectable07:51
snadgeyeah its coming back 503 so if i dig through the logs i should find out why07:59
anskiysounds like that old error when mariadb on debian-based systems couldn't properly start after installation08:00
jrosser_snadge: you mean 503 from the keystone backend or 503 from haproxy?08:01
noonedeadpunkI recently catched issue when tmpfiles were not created due to race condition on systemd-tmpfiles service startup. that was on centos though. But that lead to apache not being able to start08:02
noonedeadpunkI believe 503 can go only from haproxy08:02
snadgefrom the address it was trying to test which is the br-mgmt address on the controller port 500008:02
noonedeadpunktry curl to backend directly08:03
snadgekeystone_service-front-2 its configured as in haproxy.cfg08:04
noonedeadpunk503 is no available backends error iirc08:05
noonedeadpunkso look on backends not frontends :)08:05
jrosser_"from the address it was trying to test which is the br-mgmt address on the controller port 5000" <- this is the frontend, that HAProxy presents the loadbalanced service on08:05
anskiysnadge: there is a block keystone_service-back right after the one you've mentioned: try to curl one of those hosts08:05
jrosser_you need to look in the keystone backend config, it will list the actual keystone service IP/ports and it's those you want to test with curl08:06
snadgeyeah thats what i just did and its not connecting.. just timing ou08:07
snadgeok so the keystone container should have this 172.29.239.91:500008:09
jrosser_the next thing to do would be check on that container for the status of apache and the keystone api service08:09
snadgeapache2 is loaded and running and so is keystone-wsgi-public.service08:13
opendevreviewDanny Nelhams proposed openstack/openstack-ansible-os_magnum stable/xena: Fixed dest typo in config_template  https://review.opendev.org/c/openstack/openstack-ansible-os_magnum/+/84693208:14
snadgei can ping that ip address but port 5000 isn't listening08:14
jrosser_there should be some config a bit like this https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_073/845684/1/gate/openstack-ansible-deploy-aio_lxc-ubuntu-focal/0739c11/logs/etc/openstack/aio1_keystone_container-36e9f45e/apache2/sites-available/keystone-httpd.conf.txt08:21
opendevreviewJonathan Rosser proposed openstack/openstack-ansible master: Bump roles and services for Yoga RC1  https://review.opendev.org/c/openstack/openstack-ansible/+/84679708:23
snadgeyep i see it, theres a listen on that address and a virtualhost directive that passes it to 127.0.0.1:3535808:24
snadgeahh you're kidding me i just restarted apache08:26
opendevreviewDanny Nelhams proposed openstack/openstack-ansible-os_magnum stable/wallaby: Fixed dest typo in config_template  https://review.opendev.org/c/openstack/openstack-ansible-os_magnum/+/84696408:33
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible master: Add timestamp to written journal file  https://review.opendev.org/c/openstack/openstack-ansible/+/84643908:47
jrosser_snadge: apache should have been restarted during the installation automatically09:06
jrosser_like this (and a few other places) https://github.com/openstack/openstack-ansible-os_keystone/blob/e26aabe440b75d134ee90907656207445ddfcf64/tasks/keystone_apache.yml#L5609:07
snadgeyeah odd.. then i had another error.. restarted it and its going past now.. im not too fussed about this, as its a "test" ubuntu installation before we actually install it on production hardware09:09
snadgetheres issue with licensing, drivers.. and it seems easier to go this way09:09
snadgecentos hasn't been much fun lately09:09
jrosser_if you ctrl-c then you can end up in a situation where the config files have been dropped but the handlers wont run09:09
snadgei think at one point i tried blowing away the containers and starting the install from scratch09:10
*** ysandeep|afk is now known as ysandeep09:10
snadgei have one of those environments where everything is behind a proxy so i have to use config to assist with that09:15
jrosser_snadge: same here re. proxy, if theres anything we don't cover properly for you or it's difficult, let me know09:29
snadgeSlurp up the constraints file for later re-deployment is failing now09:32
snadgeits saying it cant find "/var/www/repo/os-releases/24.2.0/ubuntu-20.04-x86_64/requirements/placement-24.2.0-constraints.txt"09:37
snadgelooks like this is building the placement container so at least im past the keystone issue now09:38
opendevreviewMerged openstack/openstack-ansible-os_tempest master: Check return value for router create  https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/84680209:38
jrosser_snadge: it helps if you can paste the actual errors, and say which playbook it is, etc09:41
jrosser_paste.opendev.org of course, not into IRC09:41
jrosser_the repo server should be hosting those files (placement-24.2.0-constraints.txt) after the python wheels for that service have been built09:42
jrosser_similar thing with checking that you can get those with curl, should be port 8181 on the internal vip i think09:43
anskiysnadge: this could be because of interrupted execution some time ago. There are four files per each virtualenv in /var/www/repo/os-releases/24.2.0/ubuntu-20.04-x86_64/requirements/ and it could be so you're missing some. Deleting all of them like '/var/www/repo/os-releases/24.2.0/ubuntu-20.04-x86_64/requirements/placement-* could solve the issue, effectively causing rebuild of all the dependent packages for that env, AFAIR.09:46
noonedeadpunkjrosser_: as you have some insight about zun, I have question about etcd there09:48
* jrosser_ summons andrewbonney :)09:49
noonedeadpunkso we install etcd as part of zun https://opendev.org/openstack/openstack-ansible-os_zun/src/branch/master/tasks/main.yml#L188-L19609:49
noonedeadpunkBut we also have playbook for standalone etcd installation https://opendev.org/openstack/openstack-ansible/src/branch/master/playbooks/etcd-install.yml09:49
noonedeadpunkI'm not really aware about etcd functionality, but why we can't leverage the one installed with infra?09:50
noonedeadpunkAsking mostly as trying to triage https://bugs.launchpad.net/openstack-ansible/+bug/1979208 and way to solve that09:51
noonedeadpunkAnd I believe this issue indeed happens because of serial09:51
andrewbonneyI can't remember a specific reason it couldn't be changed to use an external cluster, but I clearly didn't get that far when trying to tidy the role up09:52
noonedeadpunkBut likely to solve that we need to install etcd outside of zun role. And it's added right at the end of main.yml... 09:53
noonedeadpunkso sounds like zun even don't need it on config stage.09:53
jrosser_we had quite a big look at zun in V(?) but eventually did not deploy it09:58
noonedeadpunkI believe I broke things on X for real deployments10:04
noonedeadpunkwith changes to playbooks regarding serial/facts gathering10:04
noonedeadpunkeventually etcd role deployment isn't designed to run in serial...10:05
jrosser_it does seem that the use of etcd in the zun role is really minimal10:05
jrosser_tbh not sure why this is not using the infra role10:06
noonedeadpunkthe only reason I see is different cluster name... But I'm not sure about etcd overall - like can it be used by different services at same time...10:08
noonedeadpunknever looked into it deeply10:08
jrosser_the other use case was calico, which i guess we are really close to deprecating anyway10:17
jrosser_and also we never addressed needing a lock manager for stuff like designate10:17
jrosser_noonedeadpunk: we could move the calling of the etcd role out of the os_zun role into the playbook10:20
jrosser_a bit like we do for installing glusterfs ahead of the repo server role, thats also a thing which just won't work for serial=110:20
noonedeadpunkyeah. but still I'd say it's worth fixing running role in serial manner...10:21
noonedeadpunkand I don't like idea to run some change across ha cluster at same time...10:21
noonedeadpunkor dunno10:21
jrosser_is it more that it needs to be two plays "do etcd first in serial" "do zun service in serial"10:22
noonedeadpunkfor lock driver etcd feature-wise sounds like not best option tbh10:25
jrosser_yup10:25
noonedeadpunkbut that's different topic anyway10:25
noonedeadpunkso the problem I see that etcd in serial on itself breaks now10:26
noonedeadpunkbecause it expects for other members to exists I believe and they're not yet10:26
noonedeadpunkBut need to read more about etcd to understand it more. I bet there should be some flag to start regardless of other members...10:27
*** dviroel|out is now known as dviroel11:30
mgariepygood morning13:15
jrosser_hello13:22
mgariepyanything need a push this morning ?13:23
jrosser_well - this is top priority 84679713:24
jrosser_oh13:24
jrosser_https://review.opendev.org/c/openstack/openstack-ansible/+/84679713:24
mgariepyhmm c9s failing..13:25
mgariepysurprising ;) 13:25
mgariepyhttps://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_22b/846797/4/check/openstack-ansible-deploy-aio_lxc-centos-9-stream/22b5585/logs/ara-report/results/2968.html14:11
mgariepyhttps://paste.openstack.org/show/bBifbZuzcQDJ21e6tNoB/14:11
mgariepyseems to be kinda ok on the cinder logs..14:11
jrosser_i looked in the haproxy log and it just says the backend is down14:11
jrosser_i wonder if it took too long to create that volume or something and didnt respond to the healthcheck in time14:12
*** ysandeep is now known as ysandeep|afk14:19
mgariepycan this be an issue with the thread/process config ?14:34
fanfilj5p;< zin 8uy h aioik,,mda< J<Z NNSSOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOONNNNNNNNNNNNNNNNNNNNNKOOOOOOOOOACK LEEEEBZ                           cvvxsaassdczdcd4,14:37
fanfi90014:37
fanfi999999999999999999999999999n14:37
fanfii14:37
fanfij14:37
spatelLook like keyboard issue.. lol 14:48
mgariepyor an app focus one.14:48
noonedeadpunk#startmeeting openstack_ansible_meeting15:02
opendevmeetMeeting started Tue Jun 21 15:02:38 2022 UTC and is due to finish in 60 minutes.  The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot.15:02
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:02
opendevmeetThe meeting name has been set to 'openstack_ansible_meeting'15:02
noonedeadpunk#topic roll call15:02
noonedeadpunkhey o/ sorry for being late15:02
NeilHanlono/15:03
mgariepyhello15:03
jrosser_o/ hello15:04
noonedeadpunk#topic office hours15:05
noonedeadpunkso. We almost branched Yoga for integrated repo and branched all repos15:05
noonedeadpunkexcept it failing in CI :)15:06
noonedeadpunkAfter we branch we have to drop centos 815:06
noonedeadpunkand Rocky 8 as well as a result mostly because py36 being unsupported by Zed and no python-libvirt bindings for py3815:07
noonedeadpunkNeilHanlon: any info about Rocky 9 ?15:07
jrosser_NeilHanlon: ^15:07
jrosser_:)15:07
noonedeadpunkok, next to that I wonder how you feel about publishing config_template collection on galaxy?15:10
noonedeadpunkshould we drop symlinks created for compatability as module for that?15:11
noonedeadpunkreleases team suggested to tag it as 2.0.0 regardless of publishing15:12
noonedeadpunkbecause we dropped py27 support basically15:12
jrosser_that would be good15:12
jrosser_then its very easily consumable15:12
NeilHanlonjrosser_: we have a release candidate! I should have an image available soon-ish and am going to put in a few commits for dib, too15:12
*** ysandeep|afk is now known as ysandeep15:12
noonedeadpunkNeilHanlon: these are very good news!15:12
jrosser_fun upgrade path15:13
noonedeadpunkthen I believe we should backport Rocky 9 support to Y...15:13
noonedeadpunkshould be quite straightforward?15:13
noonedeadpunkgiven no huge surprises there....15:13
NeilHanloni can help with the python-libvirt bindings for python3.8, if that would help. but also just as happy to move towards 9 and revisit 8 if I have time15:13
jrosser_should be - i expect maybe a few places we specifically check for centos-9 or named vars files15:14
jrosser_but that can be fixed15:14
NeilHanlontotally, i kept a list of the places I had to touch last time so hopefully I'll catch them before review ;) 15:14
jrosser_i expect it to be much much less work than the original rocky or centos-915:14
noonedeadpunkI believe we could be more carefull now as well15:14
noonedeadpunkI guess moving towards 9 might be more perspecitve path. And libvirt thing is just first I recalled but there were likely more15:16
NeilHanlonundoubtedly 15:16
jrosser_if we backport 9 to Y we can drop 8 from master15:16
jrosser_that would be nice15:16
jrosser_then a bunch more logic / ternary / when: can get removed15:17
NeilHanlonthat sounds like a reasonable plan. it'd be nice to only support 9 which ideally has less issues than 8 with dependencies15:17
noonedeadpunkI bet devstack and projects already dropped 8 for master15:17
noonedeadpunkok, great that Rocky 9 is coming!15:17
noonedeadpunkI want to return back to config template. jrosser_ "that would be good" was it about publishing or removing legacy ?:)15:18
noonedeadpunkI think for publishing we just need to add this job https://review.opendev.org/c/openstack/ansible-config_template/+/846391/6/zuul.d/project.yaml15:19
jrosser_oh well all of it really :)15:19
noonedeadpunkI'm not sure how tripleo consumes it actually....15:19
noonedeadpunkWon't be surprised if with python setup.py install ....15:20
noonedeadpunkBut I can recall pushing smth there to switch to collection15:20
jrosser_i think that got undone15:20
jrosser_when the setup.py modules stuff all went odd15:20
noonedeadpunkOk, i abandoned it https://review.opendev.org/c/openstack/tripleo-ansible/+/82014015:21
noonedeadpunkarxcruz|rover: do you recall how you install/consume config_template collection now? 15:22
noonedeadpunkdo you checkout to some specific tag at least?15:22
arxcruz|rovernot really 15:22
noonedeadpunkwell.. I can propose patch to CI and witness how it fails...15:24
opendevreviewDmitriy Rabotyagov proposed openstack/ansible-config_template master: Remove possibility to install as plugin  https://review.opendev.org/c/openstack/ansible-config_template/+/84704515:37
noonedeadpunkhere we go15:38
noonedeadpunkI kind of wonder if we should publish it in galaxy first and then tripleo case would be solved15:39
noonedeadpunkas we have that https://opendev.org/openstack/openstack-ansible-os_tempest/src/branch/master/meta/main.yml#L44-L4515:39
noonedeadpunkwhich kind of should pull collection in if it's absent?15:39
noonedeadpunkWe also have interesting bug that I spoilered during the day regarding zun and etcd15:43
noonedeadpunk#link https://bugs.launchpad.net/openstack-ansible/+bug/197920815:43
noonedeadpunkthe issue there is for some reason we have etcd installation inside zun role while we also have etcd in playbooks and even have env.d file for it15:44
noonedeadpunkSo likely this needs some love and refactoring15:44
noonedeadpunkwill try to look into that 15:44
noonedeadpunkI beleive that's it for today from myself15:45
noonedeadpunkand thanks everybody for working on Y!15:46
noonedeadpunkI know last week was tough on amount of things we worked on15:46
jrosser_i don't remeber why we have a separate etcd15:47
noonedeadpunkwe'll figure that out :)15:47
noonedeadpunkmaybe it's really necessity as in if they can't share the one15:47
noonedeadpunk*as in case15:47
noonedeadpunk*in case 15:48
noonedeadpunklike trove is good example why you might want to have another rabbitmq cluster15:50
noonedeadpunkmaybe here it's the same. 15:50
*** dviroel is now known as dviroel|lunch15:52
noonedeadpunk#endmeeting15:54
opendevmeetMeeting ended Tue Jun 21 15:54:51 2022 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)15:54
opendevmeetMinutes:        https://meetings.opendev.org/meetings/openstack_ansible_meeting/2022/openstack_ansible_meeting.2022-06-21-15.02.html15:54
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/openstack_ansible_meeting/2022/openstack_ansible_meeting.2022-06-21-15.02.txt15:54
opendevmeetLog:            https://meetings.opendev.org/meetings/openstack_ansible_meeting/2022/openstack_ansible_meeting.2022-06-21-15.02.log.html15:54
odyssey4menoonedeadpunk jrosser_ o/ I think the etcd was extracted because there was a network service using it... Logan Vig was using it that way IIRC15:56
odyssey4meie it was used across multiple services15:57
noonedeadpunkyeah, it was calico15:57
noonedeadpunkbut can't etcd be shared across services by design?15:58
jrosser_the odd thing is that the zun role came after calico iirc but makes it's own etcd cluster using the same role15:59
jrosser_*same etcd role as calico16:00
noonedeadpunkit has different cluster name though16:00
*** dviroel|lunch is now known as dviroel16:38
mgariepywoohoo rc1 passed.. 17:02
jrosser_lets try for 2 in a row17:07
mgariepyshould i go get lottery ?17:08
jrosser_we need some luck17:10
jrosser_over in #qa they discuss the value of having centos-9 jobs when they're constantly breaking17:11
mgariepywonder about the thread/process issue tho.17:11
*** ysandeep is now known as ysandeep|out17:37
noonedeadpunkI can't say we have tons of issues with centos rather then with post_failures as of today18:27
noonedeadpunkwhat a surprise - https://review.opendev.org/c/openstack/ansible-config_template/+/847045 failed on our functional jobs18:33
opendevreviewDmitriy Rabotyagov proposed openstack/ansible-config_template master: Remove possibility to install as plugin  https://review.opendev.org/c/openstack/ansible-config_template/+/84704518:37
jrosser_oh well like I found the other day I think the functional config template jobs are very odd18:37
jrosser_I don’t think we have anything to symlink to the zuul repo in the tox env18:37
jrosser_or otherwise install from there18:38
noonedeadpunkyeah.... I think this is true18:39
noonedeadpunkWe indeed need to proceed with molecule18:40
noonedeadpunkI got a bit hands-on with it :D18:40
noonedeadpunkWe just need good way of managing variables for it in one place18:40
jrosser_maybe this is silly but can it go in plugins collection18:41
noonedeadpunktbh I'd rather did that in integrated repo. As plugins we likely need to sort out one day.18:42
noonedeadpunkfor example I was thinking recently about moving service_setup to ansible-collections-openstack.18:42
jrosser_we just need to make sure that the workflow out of zuul is sensible18:43
noonedeadpunkas what I found here tells me we do exactly smae thing in 2 repos https://github.com/openstack/bifrost/blob/7271695714739af10d741a39dc4acd5e68465cb5/playbooks/roles/bifrost-ironic-install/tasks/keystone_setup.yml18:45
noonedeadpunkI evene started a bit wondering why bifrost and os_ironic even co-exist, but I'd rather not go there now :D18:47
jrosser_one step at a time18:54
noonedeadpunkyeah19:31
mgariepylook like it's going to merge now :D19:35
opendevreviewMerged openstack/openstack-ansible master: Bump roles and services for Yoga RC1  https://review.opendev.org/c/openstack/openstack-ansible/+/84679719:47
*** dviroel is now known as dviroel|afk20:06

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!