Tuesday, 2022-07-05

*** ysandeep|out is now known as ysandeep|ruck04:02
*** ysandeep|ruck is now known as ysandeep|ruck|afk04:38
*** ysandeep|ruck|afk is now known as ysandeep|ruck05:08
*** ysandeep|ruck is now known as ysandeep|ruck|afk06:18
noonedeadpunkianw: that is neat, thanks!06:40
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible master: Continue Zed development  https://review.opendev.org/c/openstack/openstack-ansible/+/84727206:43
noonedeadpunkjrosser_: btw I believe we have circular dependency here https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/84858906:44
jrosser_noonedeadpunk: yeah that looks unfortunate06:57
jrosser_maybe we just remove all the jobs from os_horizon in the same patch and merge 848589 ASAP06:57
noonedeadpunk well, yes, I guess so. But for 847272 there's some weird issue with distro installs at the very beginning. Which sounds more like some module issue tbh... Which is weird, as with that still Yoga packages are being used07:02
jrosser_"msg": "Invalid query params: region_id"07:04
jrosser_well - i was thinking for the distro jobs where does the openstacksdk come from07:06
jrosser_here https://github.com/openstack/openstack-ansible/blob/master/playbooks/utility-install.yml#L82-L9607:07
jrosser_hmm python3-openstacksdk:amd64 (0.61.0-0ubuntu1~cloud1, automatic)07:08
*** ysandeep|ruck|afk is now known as ysandeep|ruck07:11
noonedeadpunkbut utility should be setup with distro packages as well then.... but for this task I'm not sure it's utility version we should check or ansible venv?07:12
noonedeadpunkbut yeah, utility I guess indeed07:12
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-os_horizon master: Temporary disable horizon functional jobs  https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/84869307:14
noonedeadpunkoh! I think it's because I set ansible-collections-openstack version to master07:15
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible master: Continue Zed development  https://review.opendev.org/c/openstack/openstack-ansible/+/84727207:15
noonedeadpunkBut then likely everything else would fail07:16
noonedeadpunklet's see07:16
jrosser_thats odd why only the distro job fails07:17
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-os_horizon master: Release pyscss constraints  https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/84858907:17
jrosser_i think the upgrade jobs were failing rally so we might have also something separate there to look at07:22
noonedeadpunkupgrade jobs were failing in other place which I covered I believe07:26
noonedeadpunkas they were trying to upgrade from Xena07:26
noonedeadpunkor well, at least centos 9 was failing there07:28
noonedeadpunkI was also wondering what are concerns regarding https://review.opendev.org/c/openstack/ansible-config_template/+/846391 ?07:29
jrosser_concerns? you mean why it's not merged yet?07:34
noonedeadpunkWell, and no votes or comments:)07:35
noonedeadpunkas we were discussing it several times I guess and there was no objections from anybody but no reviews either 07:36
jrosser_oh well i think i concentrated on getting through the POST_FAILURES stuff and ignored everything that wasn't that or release patches07:36
noonedeadpunkthat's totally fair07:36
noonedeadpunkI just started looking through review board and noticed it :p07:37
noonedeadpunkcan't say it's the only thing I noticed, but well07:37
noonedeadpunkto be completely fair, I don't know if it will "just work" and dunno how to check except try out 07:38
*** ysandeep|ruck is now known as ysandeep|lunch08:15
damiandabrowski[m]noonedeadpunk:  I've found one disadvantage of making `tempest_plugins` a list of dicts.09:23
damiandabrowski[m]when tempest_plugins is a dict, we can use combine() filter for tempest_extra_plugins which allows to override specific plugins.09:23
damiandabrowski[m]when tempest_plugins is a list, we can just concatenate 2 lists together but it's not that smart as combine()09:23
damiandabrowski[m]do we still want to make it a list?09:23
opendevreviewMerged openstack/openstack-ansible-os_horizon master: Release pyscss constraints  https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/84858909:36
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-os_horizon master: Return zuul integrated jobs  https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/84873209:47
noonedeadpunkdamiandabrowski[m]: well, it's fair point09:47
noonedeadpunkbut looking at _tempest_plugins seems there's nothing not configurable in it...09:48
damiandabrowski[m]repo url / branch :D 09:49
noonedeadpunkbut these are variables?09:49
noonedeadpunkhttps://opendev.org/openstack/openstack-ansible-os_tempest/src/branch/master/vars/main.yml#L56-L5709:49
damiandabrowski[m]ah, You're right09:50
noonedeadpunkso if you want to change behaviour of anything that's pre-defined, you don't actually need to combine, as you have variables in place kind of...09:50
damiandabrowski[m]ack09:52
noonedeadpunkbut dunno, maybe you're right. it's just a bit pita to deal with dicts when you actually don't need them09:52
noonedeadpunkjrosser_: yeah, with old collection the only thing that is passing is distro job lol09:53
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible master: Continue Zed development  https://review.opendev.org/c/openstack/openstack-ansible/+/84727209:56
*** ysandeep|lunch is now known as ysandeep|ruck10:16
damiandabrowski[m]do You think we can finally remove 'tests/' directory in openstack-ansible-os_* repos? I wonder what else should be cleaned up10:36
*** arxcruz is now known as arxcruz|brb10:59
noonedeadpunkI can't recall if we need that for docs/releasenotes? I guess not?11:03
damiandabrowski[m]i think we don't11:11
jrosser_the only places that would need to keep it would be where we still have functional tests11:13
jrosser_and thats not very many at all now, probably non of the os_ roles11:13
noonedeadpunkfor sure we have that for config_template,systemd, qrouterd and maybe smth else 11:14
opendevreviewDamian DÄ…browski proposed openstack/openstack-ansible-os_tempest master: Make _tempest_plugins a list of dicts.  https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/84864211:23
*** dviroel|biab is now known as dviroel11:29
opendevreviewDamian DÄ…browski proposed openstack/openstack-ansible-os_tempest master: Allow to define extra tempest plugins  https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/84864311:30
damiandabrowski[m]okok thanks11:34
noonedeadpunkdamiandabrowski[m]: with removing tests folder we should also cleanup tox.ini and drop run_tests.sh11:44
noonedeadpunkI think it's also time to deprecate rsyslog roles 11:45
damiandabrowski[m]ack11:55
*** arxcruz|brb is now known as arxcruz12:03
*** ysandeep|ruck is now known as ysandeep|brb12:52
*** ysandeep|brb is now known as ysandeep|ruck13:01
jrosser_damiandabrowski[m]: i think there is already a patch to os_ironic for a big cleanup to take as a starting point13:25
damiandabrowski[m]ah this one? https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/82421413:27
damiandabrowski[m]thanks, i was just trying to figure out how everything works :D 13:27
spateljamesdenton morning!13:55
spatelIn ovn multi-node deployment i have noticed our controller node doesn't install openvswitch so it doesn't act as network gateway for deployment. 13:57
jamesdentonmornin13:58
spateli have deployed multi-node ovn and noticed one of compute node is gw 13:58
jamesdentonthere may be something in the playbooks setting 'chassis-is-gw' or whatever that is13:59
jamesdentonhttps://github.com/openstack/openstack-ansible-os_neutron/blob/bbdd12561f4e5800e6504d14a7810fad8631a42b/tasks/providers/setup_ovs_ovn.yml#L23-L2713:59
spatelYes chassis-is-gw tell you which node will be gateway but in OSA deployment we don't install openvswitch on infra node so how can infra node will be gateway?14:00
spatelWe need to install neutron-ovn-controller on infra nodes to deploy OVS switch 14:01
spatelam i missing something here?14:01
jamesdentonhttps://github.com/openstack/openstack-ansible-os_neutron/blob/6c722ead673f1c80aee4f439bf5f6a41841e85a5/vars/distro_install.yml#L46-L4814:03
jamesdentonthe controller needs to be a member of one of those groups14:03
jamesdentonerrr, one sec14:04
spateltake your time because i need coffee :) 14:05
spatelall i can see we need to install neutron-ovn-controller  on infra metal box which will make that node part of SDN and then you can set chassis-is-gw to route traffic. 14:06
spatelI have deployed 3 (infra) and 2 (computes) nodes deployment and found only compute nodes act as gateway. controller nodes doesn't have any OVN information to be part of SDN. 14:08
jamesdentondid you manipulate the env.d files as part of your deployment?14:10
jamesdentoni think the 'ref arch' only has computes in the ovn controller role: https://github.com/openstack/openstack-ansible/blob/master/etc/openstack_deploy/env.d/nova_ovn.yml.example#L1014:10
jamesdentonand by ref arch, i mean the OSA arch14:10
jamesdentonbut that's definitely tunable14:11
spatelyes i did env.d stuff 14:11
spatelwhat i need to do to deploy ovn-controller on infra nodes?14:11
spatelwe should make that deployment default instead compute nodes act as a gateway. (Let user decide if they want distributes FIP or not)14:12
mgariepywith ovn it's kinda free tho.14:13
spatelinfra nodes should be gateway by default for ovn deployment in my opinion 14:13
mgariepyand nova-compute shouldn't?14:14
jamesdentonyou'll need to update the env.d files to make sure the infra nodes are members of the neutron_ovn_controller group; i don't have an example handy14:14
jamesdentoni agree there needs to be a more graceful way to determine what is or isn't a gateway node14:16
spatelnova-compute can be but let user decide what they want (everything is tunable in ovn but better out of box when someone deploy cloud infra should be gateway and later you can decide if you want compute to be gateway14:17
spatelIn OVN you have priority option to tell who will be gateway when you have multiple gateway so we can set infra node with high priority bit. 14:19
jamesdentonthe original intent was to mirror DVR, where the compute is gateway, but reality is i don't really care. We just need to remove hardcoding from the playbooks to ensure we don't overwrite anything the user has done. So, that may mean not setting any gateway and leaving that to admin guide14:19
mgariepythe admin guide might need some updating.14:20
jamesdenton^^14:20
mgariepyhttps://docs.openstack.org/openstack-ansible-os_neutron/latest/app-ovn.html#optional-dvr-or-distributed-l3-routing14:20
spatelwe need more people to use OVN so we get better feedback :)14:21
spatelI have started playing with ovn and my goal is to make it better with OSA 14:22
anskiyi don't think making infra node act as gateway by default is good idea: that's mixing control and data planes14:22
spatelI have started building ovn-bgp-agent lab so hope we have that piece available in OSA 14:22
mgariepyif you have 2 vms with fips on 2 compute node, do they all go through the same compute host gw or they are each hosted on their compute node?14:23
spatelI am looking at current deployment of osa (LXD based) where infra nodes are gateway. (Yes you can do whatever you want later)14:24
spatelmgariepy i have create blog here on OVN non-distributed vs distributed FIPs - https://satishdotpatel.github.io/openstack-ansible-ovn-deployment-part2/14:25
spatelOVN is best solution for DVR style deployment and freaking easy. 14:26
mgariepyi guess ovn is the way to go now14:27
spatelindeed, no need to worry about keepalived/namespace etc.. :) 14:28
mgariepyi have an ovs deploy that needs to be migrated to ovn.14:28
spatelI am building FRR lab to test ovn-bgp-agent deployment to see how good it can fit with osa 14:30
mgariepydid you assist the talk about the migration on tripleO at the summit?14:38
*** ysandeep|ruck is now known as ysandeep|out14:39
mgariepyho. just remembered while watching the talk you did told me already :) sorry14:44
noonedeadpunk#startmeeting openstack_ansible_meeting15:03
opendevmeetMeeting started Tue Jul  5 15:03:30 2022 UTC and is due to finish in 60 minutes.  The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot.15:03
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:03
opendevmeetThe meeting name has been set to 'openstack_ansible_meeting'15:03
noonedeadpunk#topic rollcall 15:03
noonedeadpunkhey o/15:03
damiandabrowski[m]hi!15:03
NeilHanlonhiya!15:03
mgariepyhey !15:03
noonedeadpunk#topic office hours15:06
noonedeadpunkok, so seems all our branches are now blocked because of rally15:06
*** dviroel is now known as dviroel|lunch15:07
damiandabrowski[m]:(15:08
noonedeadpunkbasically this is the fix https://review.opendev.org/c/openstack/openstack-ansible-os_rally/+/848666 but upgrade jobs needs to be fixed first15:08
noonedeadpunkAnd not sure how long we should backport it15:08
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-os_rally stable/victoria: Control rally-openstack installed version  https://review.opendev.org/c/openstack/openstack-ansible-os_rally/+/84871815:08
noonedeadpunkeventually I wonder, if we should not install rally in first run of upgrade jobs15:10
noonedeadpunkas we don't run tempest15:11
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible master: Do not install rally on first upgrade run  https://review.opendev.org/c/openstack/openstack-ansible/+/84876015:13
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-os_rally master: Control rally-openstack installed version  https://review.opendev.org/c/openstack/openstack-ansible-os_rally/+/84866615:14
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible master: Continue Zed development  https://review.opendev.org/c/openstack/openstack-ansible/+/84727215:14
noonedeadpunkI wonder if that would be enough to fix the issue:)15:14
noonedeadpunkOther then that - stable 25.0.0 tag was pushed on 1st of July, but I think it's still hasn't been merged15:15
noonedeadpunkI've already reminded release team about it15:15
damiandabrowski[m]fingers crossed15:16
noonedeadpunkthen for master we had to set distro jobs to NV for now. As with openstacksdk 0.99 which is set in u-c we should use master ansible collections. Which does not work with older sdk....15:17
noonedeadpunkI expect things to be fixed only when packages for Z will be published and it will take a while15:18
noonedeadpunkalso jrosser_did good job on updating our grafana board  https://grafana.opendev.org/d/8cbc9d7032/openstack-ansible?orgId=1&from=now-90d&to=now15:19
noonedeadpunkwhich gives good idea about our CI state15:20
noonedeadpunkI think we also can now try out shared queus in zuul. wdyt?15:20
damiandabrowski[m]awesome, thanks jrosser!15:21
damiandabrowski[m]not sure what prevented us to do that before, but I don't have any objections :D 15:21
noonedeadpunkwell, I think if any task that is in gate fails, it leads to failure of all rest patches15:22
noonedeadpunkif I got that correctly15:22
noonedeadpunkand that sounds quite troublesome for me15:22
damiandabrowski[m]ouh, yeah :/15:23
noonedeadpunkBut I might be wrong in my understanding of the concept and how it works...15:23
noonedeadpunkbut rechecking each time when +w is set in incorrect order also nasty...15:24
jrosser_o/ hello15:31
* jrosser_ late15:31
noonedeadpunkI'd say in time:)15:32
NeilHanlonRocky 9 should be coming out "soon", so i'll probably start on that this week15:58
noonedeadpunkthat are good news!16:05
noonedeadpunkjust let us know where we can help16:06
noonedeadpunk#endmeeting16:06
opendevmeetMeeting ended Tue Jul  5 16:06:04 2022 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)16:06
opendevmeetMinutes:        https://meetings.opendev.org/meetings/openstack_ansible_meeting/2022/openstack_ansible_meeting.2022-07-05-15.03.html16:06
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/openstack_ansible_meeting/2022/openstack_ansible_meeting.2022-07-05-15.03.txt16:06
opendevmeetLog:            https://meetings.opendev.org/meetings/openstack_ansible_meeting/2022/openstack_ansible_meeting.2022-07-05-15.03.log.html16:06
*** dviroel|lunch is now known as dviroel16:22
jrosser_has anyone ever implemented a plugin (like a keystone auth plugin) and have boilerplate for making all the stevedore plugin loading stuff work16:27
jrosser_RITICAL keystone [-] Unhandled error: stevedore.exception.NoMatches: No 'keystone.auth.password' driver found, looking for <my-plugin....>16:27
noonedeadpunkwell, I used to debug stevedore plugins behaviour in barbican. And the tricky thing there for me was how to discover plugin name. As it's not path to the file or class, it's another thing, so I had to print list of all found plugins to realize, that plugin I'm looking for named quite differently from what I expected or was documented16:38
noonedeadpunkjrosser_: btw yeah, in https://review.opendev.org/c/openstack/openstack-ansible/+/848659 it's worth using haproxy_allowlist_networks 16:43
jrosser_right i think i'm gettig mixed up with plugin names / namespaces / other things16:43
noonedeadpunkor well, create haproxy_stick_table_allowlist_networks: "{{ haproxy_allowlist_networks }}"16:43
noonedeadpunkit's super unobvious in stvedore for me. Even when I found actual name I failed to understand where it comes from....16:44
noonedeadpunkI commented in task. Join looks nice in test playbook - get result "http-request deny deny_status 429 if { sc_http_err_rate(0) gt 20 } !{ src 192.168.0.0/16 } !{ src 172.16.0.0/12 } !{ src 10.0.0.0/8  }" for multiple networks and "http-request deny deny_status 429 if { sc_http_err_rate(0) gt 20 } !{ src 192.168.0.0/16 }" for single one16:53
jrosser_noonedeadpunk: this was helpful https://pypi.org/project/entry-point-inspector/17:17
jrosser_i made it work but omg the documentation is just useless17:17
jrosser_or the whole thing is so abstract it's impossible to get a grip on actually whats happening17:17
NeilHanlonwhat in the world is a stevedore17:49
NeilHanlonlol17:49
noonedeadpunkwell, I just putting print() somewhere in the code, as I didn't know about that tool, so....18:59
noonedeadpunk*I was just18:59
noonedeadpunknot sure whos way was more challenging though hehe19:00
noonedeadpunkdoh, we have another circular dependency19:15
noonedeadpunkfor https://review.opendev.org/c/openstack/openstack-ansible/+/848760 and https://review.opendev.org/c/openstack/openstack-ansible-os_rally/+/84866619:16
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-os_rally stable/wallaby: Control rally-openstack installed version  https://review.opendev.org/c/openstack/openstack-ansible-os_rally/+/84872119:18
noonedeadpunkpff, and for backport since victoria it's erlang repo that is broken for bionic19:19
noonedeadpunkthat is annoying I must say19:24
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-os_rally master: Control rally-openstack installed version  https://review.opendev.org/c/openstack/openstack-ansible-os_rally/+/84866619:27
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible master: Do not install rally on first upgrade run  https://review.opendev.org/c/openstack/openstack-ansible/+/84876019:29
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible master: Continue Zed development  https://review.opendev.org/c/openstack/openstack-ansible/+/84727219:30
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-os_rally master: Return upgrade jobs to voting  https://review.opendev.org/c/openstack/openstack-ansible-os_rally/+/84877819:30
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible master: Continue Zed development  https://review.opendev.org/c/openstack/openstack-ansible/+/84727219:31
noonedeadpunkand we need to do the same for each damn branch....19:33
spatelI have haproxy related question - https://paste.opendev.org/show/bhM1FR1Y0fzOyaSPexiN/20:04
jrosser_spatel: hit 35k what?20:36
spatelwe have loadtest tool develop for our customer application benchmarking, i am generating load to see how far i can go with haproxy 20:37
jrosser_oh sure, but 35k bytes / kilograms / req/sec / concurrent clients ? :)20:38
spatelI have 5 backend servers and 1 source ( 55k per backend server that means i can hit 250k concurrent connection but look like getting error at 35k related running out of ports)20:43
jrosser_i am sure we have talked about this before20:44
mgariepywhat are the ulimit for your haproxy ?20:45
mgariepyhttps://www.freecodecamp.org/news/how-we-fine-tuned-haproxy-to-achieve-2-000-000-concurrent-ssl-connections-d017e61a4d27/20:45
mgariepypretty sure haproxy can scale a lot. but it will need to have the correct config/tuning and so on.20:46
spatelI have 10million 20:48
spatelI have 10million  (ulimit + openfile + whatever tuning required by google search)20:48
spatelIf i add more IP in source then i can pass 50k without issue. by my math saying with single IP i should be able to do that 20:49
spatelI have 5 backend servers that means enough socket:ports available 20:50
*** dviroel is now known as dviroel|afk21:59
opendevreviewJonathan Rosser proposed openstack/openstack-ansible-os_rally master: Control rally-openstack installed version  https://review.opendev.org/c/openstack/openstack-ansible-os_rally/+/84866622:03
*** dviroel|afk is now known as dviroel23:27
*** dviroel is now known as dviroel|out23:50

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!