Wednesday, 2024-01-24

opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Update horizon local settings for Django 4  https://review.opendev.org/c/openstack/kolla-ansible/+/90625406:49
fricklerhmm, kevko does a rebase of my patch and when I rebase it again, gerrit says I'm doing this on behalf of kevko? weird. but let's please get this ^^ fix in now and check the local_settings.d stuff later07:01
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: [DNM] Just test temporarily  https://review.opendev.org/c/openstack/kolla-ansible/+/90642307:32
opendevreviewRafal Lewandowski proposed openstack/kolla-ansible master: [WIP] Enable ML2/OVN and distributed FIP by default  https://review.opendev.org/c/openstack/kolla-ansible/+/90495907:45
kevkofrickler: i have a question ... what version should be installed before kolla-ansible upgrade in upgrade.sh ? i thought there should be 2023.2, shouldn't be ? 08:28
kevkofrickler: why there is master and then tag change_xxxx ? 08:29
kevkofrickler: let's merge it ..as it is CI blocker ...It will fix the gate ...but I still don't understand why my patch is not working 08:33
kevkofrickler: deploy jobs are working ..because it is master images and master kolla-ansible where it is everything ok,  why upgrade job is not working ? i thought that upgrade job is zuul-mirror 2023.2 tag and kolla-ansible 2023.2 ....then tag zuul-mirror:master (in case not depends-on) and then gerrit review upgrade ... or change_XXX (if depends-on08:36
kevkokolla review) tag ...with the current gerrit review kolla-ansible patch ...or no ? 08:36
kevkowe should draw some pipeline workflow diagram :D 08:37
fricklerkevko: it is well possible that there is a bug in our testing, I don't claim to understand much of it either. probably mnasiadka is the one who remains with the most insight09:16
kevkofrickler: bbezak promised me that he will check  it today :)09:17
kevkofrickler: okay, let's merge your django local_settings as it's gate blocker 09:18
opendevreviewRafal Lewandowski proposed openstack/kolla-ansible master: [WIP] Enable ML2/OVN and distributed FIP by default  https://review.opendev.org/c/openstack/kolla-ansible/+/90495909:32
* SvenKieske has to read up about all the CI breakage09:59
opendevreviewDr. Jens Harbott proposed openstack/kolla master: Revert "Fix gnocchi and skyline after requirements change"  https://review.opendev.org/c/openstack/kolla/+/90647910:09
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: [DNM] Test jobs workflow  https://review.opendev.org/c/openstack/kolla-ansible/+/90650310:32
opendevreviewMerged openstack/kolla-ansible master: Update horizon local settings for Django 4  https://review.opendev.org/c/openstack/kolla-ansible/+/90625410:50
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: [CI] Fix podman cross-dependency build  https://review.opendev.org/c/openstack/kolla-ansible/+/90622911:10
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Rework horizon role to support local_settings.d  https://review.opendev.org/c/openstack/kolla-ansible/+/90634711:10
opendevreviewBartosz Bezak proposed openstack/kolla-ansible stable/2023.2: use 2023.2 images in stable/2023.2 branch  https://review.opendev.org/c/openstack/kolla-ansible/+/90650911:48
kevkofrickler ^^ bbezak is my savior ..I almost lost my temper11:49
bbezak:)11:51
bbezakI've seen couple of other gems, will post some patches11:51
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Rework horizon role to support local_settings.d  https://review.opendev.org/c/openstack/kolla-ansible/+/90634711:52
kevkobbezak: can u share them in short ? :) 11:52
bbezakwe didn't update openstack_previous_release_name in master - that is used for ironic rolling update11:54
bbezakand we didn't update previous release in kayobe11:54
kevkobbezak: yes i saw this  at night 11:54
bbezakI'm updating this shortly11:55
kevko^ ironic11:55
kevkowhen i was grepping almost everytihing in kolla-ansible repo :D 11:55
bbezak:)11:55
opendevreviewBartosz Bezak proposed openstack/kolla-ansible master: update openstack_previous_release_name var for bobcat  https://review.opendev.org/c/openstack/kolla-ansible/+/90651111:57
opendevreviewBartosz Bezak proposed openstack/kayobe master: update previous release to 2023.2  https://review.opendev.org/c/openstack/kayobe/+/90651312:00
kevkobbezak: shouldn't we have some script to do this automatically when stable release from "previous" master is released ? 12:01
kevkobbezak: btw, do exist some way how i can access logs dir even if tests are still running ? :D 12:02
bbezakwe've got a procedure to follow - https://docs.openstack.org/kolla/latest/contributor/release-management.html looks like it was not followed properly this time :)12:05
kevkothat's what i am saying all the time ...noone reads docs :D 12:06
bbezakI think for that you would need to get access to the host directly, but I think frickler could help you with that if that's really needed12:06
kevkobbezak: i am asking for the future ...12:15
kevkobbezak: here you go -> podman depends on fail ... (i rebased and dropped that patch which was solving podman cross-dependency issue ... and you see..)  https://zuul.opendev.org/t/openstack/build/c95b2d80c13145dfbff71973966c7dad12:16
bbezakI'll look into podman thingy in a bit12:17
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: [CI] Fix podman cross-dependency build  https://review.opendev.org/c/openstack/kolla-ansible/+/90622912:17
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Rework horizon role to support local_settings.d  https://review.opendev.org/c/openstack/kolla-ansible/+/90634712:17
kevkobbezak: it's simple, if job is running with podman container engine ...it's built also using podman --> https://github.com/openstack/kolla-ansible/blob/6ec1dc97e2246442fc601b022a4baf783ce052ec/tests/setup_gate.sh#L82C1-L82C2912:19
kevkobbezak: as I saw fail, i've already rebased and included fix for cross-dependency again 12:20
bbezakcool12:21
opendevreviewPiotr Parczewski proposed openstack/kolla master: Use full binary path when invoking ip  https://review.opendev.org/c/openstack/kolla/+/90651612:24
SvenKieskekevko: bbezak: +1 on automating release management. I fail to see why one would need host access for that? I must overlook something.13:23
kevkoSvenKieske: during debugging I wrote some own output files which ran early on pipeline beggining ..and because I am impatient most of the time .. I was just curious if it is possible to check my debug even if job is still running13:25
kevkoSvenKieske: sometimes it's really helpful to try some scenarios manually ... you probably agree that if you develop some feature ..you execute code locally 20x more than in one review ...13:26
SvenKieskesure, but I was just talking about automation of release management, not about debugging :) maybe I misunderstood.13:27
kevkoSvenKieske: for me - for example it would be helpful to create bucket in swift on the beginning and  upload intermediate logfiles to swift even if job is already running ... then you will be able to check the URL for logs from running console job and just open in your browser ...unfortuntatelly ... current implementation is that logs are uploaded13:29
kevkoat once on the end 13:29
kevkoSvenKieske: aaaa ..  well there were several variables misconfigured ..and because of this ..we tested (for example) master->master  ..because variable was master in 2023.2 branch (:() 13:30
kevkobbezak sent me docs ..what is the process of release ..13:30
kevkoI was asking ..why there is not some script to do it automatically and ask user on the end ...is this everything OK ? and he just click ..yes 13:31
SvenKieskekevko: yes, but it at least seems we should be able to take the docs and replace them by some automation script thingy, no? :)13:31
SvenKieskeagreed :)13:31
kevkoyeah, of course ..13:32
kevkoif the release process is always same ..i mean, create new branch from the master branch, change previous version in new master to previous released version ..change version in stable release to stbale release not master ...etc ...etc 13:32
kevkofrom my point of view it's really easy script 13:33
kevkosome greps, seds, ...or python + yaml 13:33
kevkothat's it 13:33
fricklerso just write it and submit it as a tool, I don't think anyone would object13:35
bbezakKolla meeting in 10 minutes - mgoddard mnasiadka bbezak frickler kevko SvenKieske mmalchuk gkoper jangutter jsuazo jovial osmanlicilegi mattcrees13:50
bbezak#startmeeting kolla14:00
opendevmeetMeeting started Wed Jan 24 14:00:33 2024 UTC and is due to finish in 60 minutes.  The chair is bbezak. Information about MeetBot at http://wiki.debian.org/MeetBot.14:00
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.14:00
opendevmeetThe meeting name has been set to 'kolla'14:00
bbezak#topic rollcall14:00
SvenKieskeo/14:00
frickler\o14:00
kevko\o/14:00
mattcreeso/14:01
bbezak#topic agenda14:02
bbezak* Roll-call14:02
bbezak* Agenda14:02
bbezak* Announcements14:02
bbezak* Review action items from the last meeting14:02
bbezak* CI status14:02
bbezak* Release tasks14:02
bbezak* Regular stable releases (first meeting in a month)14:02
bbezak* Current cycle planning14:02
bbezak* Additional agenda (from whiteboard)14:02
bbezak* Open discussion14:02
bbezak#topic CI status14:02
bbezakI think looks good after yesterday fixes by kevko and frickler14:03
bbezak(I've been on sick leave for couple of days)14:03
kevkopodman cross-dependency missing 14:03
SvenKieske(me too ftr)14:03
bbezakthx SvenKieske - sorry I was not around14:04
mmalchuko/14:04
SvenKieskeah no problem, health comes first :)14:04
bbezakkevko: yeah I wanted first to merge the 2023.2 k-a fix and then retest upgrades in your patch14:04
bbezakif everything is sorted as it should14:05
kevkobbezak: we have the same thoughts14:05
kevko+114:05
bbezaknice14:05
bbezakthere was red status in kolla whiteboard, I've changed it to green, as it is If I'm not mistaken14:05
kevkoI hypnotize zuul status with a look14:06
bbezak:)14:06
bbezak#topic Release tasks14:06
bbezakwe were lacking several patches for release tasks unfortunately14:06
bbezakI've raised them today14:07
bbezakkevko has some automation/tooling ideas14:07
bbezakpatches are welcome14:07
kevkoi will propose some maybe ...if I will have some time :) 14:08
bbezakok, thx14:08
bbezakI've raised monthly stable release last week14:08
kevkoas you surely know, I have never done a release, I was completely surprised that there is no automated procedure for this14:08
bbezakhttps://review.opendev.org/c/openstack/releases/+/90604614:09
bbezakthe process is pretty well documented, however it surely need an update - https://docs.openstack.org/kolla/latest/contributor/release-management.html14:09
bbezakwill add a note to myself to update this doc14:11
kevko+114:11
bbezak#topic Current cycle planning14:12
bbezakhttps://etherpad.opendev.org/p/KollaWhiteBoard#L20914:12
bbezakany updates/questions on those priorities?14:12
bbezakI don't have any update myself this week14:13
SvenKieskenot from my part at least, partly due to being sick the last two days :)14:13
bbezakas far as my tasks14:13
SvenKieske*technically three days14:15
bbezakwe've been mostly fire fighting CI since last week I can see. Let's hope next week will be more fruitful 14:16
bbezak#topic Additional agenda (from whiteboard)14:17
bbezakI can see Container engine migration from mhiner14:17
bbezakhowever I recall it was covered last week14:17
SvenKieskeI don't remember if this question was answered? > current CI tests don't gather container logs after migrating, any hints how to fix?14:18
bbezakoh I remember that I didn't want to remove it from whiteboard as it has some valuable info14:18
kevkoSvenKieske: i think it is ..14:19
SvenKieskein general I think it would make sense to update the whiteboard if questions are brought up and answered, no? not sure about the current approach here.14:19
fricklerI think I still need to look at those tests14:19
SvenKieskeat least - if I'm not misinterpreting - I don't see any answers to any of the questions there, like which cases we want to test. these answers are also missing from the patchset afaik.14:20
bbezakyeah, it got valuable info/question - which is not yet fully answered, we can leave it here to track every week. if that makes sense14:21
SvenKieskeimho it only makes sense if it either contains the actual status of info/question, or a link to the status, or at the very list a link to the person working on answering the questions/providing the needed information14:22
bbezakit probably should be discussed in the change itself14:22
SvenKieskeI'm fine with any solution, but I think only having the question there with no update serves little benefit. afaik the CI question is already in the review, but no answers :)14:23
bbezakok, let's continue discussion on that in the change. I'll double check if this info is in the change and then will remove from additional agenda part for now14:25
SvenKieskeok14:25
bbezak#topic Open discussion14:25
bbezakAnything to discuss today? some changes to review?14:27
kevkowell, I wanted to review horizon patches :P 14:27
kevkoand DNS14:27
mmalchukhttps://review.opendev.org/c/openstack/kolla-ansible/+/90440014:28
kevkoI am sure that after CI patches will be merged ..it will pass ..so if anyone have a time 14:28
mmalchukplease merge last fix to Yoga14:28
mmalchukand this: https://review.opendev.org/c/openstack/kolla-ansible/+/90238214:28
mmalchukshould be abandoned since not needed14:28
mmalchukhttps://review.opendev.org/c/openstack/tenks/+/90433314:29
mmalchuknot K/K-A but please review too14:29
mattcreesI'm looking for second core reviews on two bugfixes, if anyone has the time: https://review.opendev.org/c/openstack/kolla-ansible/+/905500 https://review.opendev.org/c/openstack/kolla-ansible/+/90585114:29
frickleradded to my list14:31
bbezakspeaking of tenks, I've just notice it is failing in kayobe and k-a ironic jobs in master14:32
bbezak"msg": "exceeded retry limit of 15 whilst waiting for resources to become available"14:32
bbezakwe need to look into that14:32
bbezakI'll add info to whiteboard14:33
mmalchukhave a link to failed job?14:33
bbezakhttps://zuul.opendev.org/t/openstack/build/ea7e2d6d8f0d471e849e66cee46d342214:34
bbezakand in k-a https://zuul.opendev.org/t/openstack/build/935effc2148b436287d1ab1a057c7d7f14:34
bbezaklooks pretty fresh case14:35
mmalchukinteresting14:36
mmalchuksome regression in ironicclient?14:36
mmalchukoh... placementclinet14:36
bbezaknot sure, I literally just seen it in my patch for kayobe14:37
mmalchukstrange... 4.2.0 released in Aug 202314:37
SvenKieskedon't know if related but I proposed a fix for networking-baremetal, which I think is used in ironic, with regards to quorum queues: https://review.opendev.org/c/openstack/networking-baremetal/+/90399514:37
SvenKieskedidn't look into your issue though, so maybe totally unrelated.14:38
mmalchukimho not related since not used even if exists in container14:38
bbezakinteresting, let's see. I'll try to look into that tenks case probably tomorrow14:38
mmalchukit used when segments activated14:38
kevkobtw, i am too lazy to study code ..but from where is tenks installed in ci ? 14:39
mmalchukhave a look too14:39
kevkotenks.yml 14:39
kevkowhere i can find it ? 14:40
mmalchukdeploy-tenks.sh14:40
kevkohh, i know 14:41
mmalchuktests/deploy-tenks.sh14:41
kevkoand there is tenks.yml14:41
mmalchukfunction deploy_tenks_logged14:41
mmalchukline #1014:41
mmalchukinstalled from source14:41
kevkothanks 14:42
bbezaknice, so more firefighting, as usual14:42
bbezakanything more to discuss?14:43
bbezakok, I guess no. thank you everybody!14:44
bbezak#endmeeting14:44
opendevmeetMeeting ended Wed Jan 24 14:44:21 2024 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)14:44
opendevmeetMinutes:        https://meetings.opendev.org/meetings/kolla/2024/kolla.2024-01-24-14.00.html14:44
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/kolla/2024/kolla.2024-01-24-14.00.txt14:44
opendevmeetLog:            https://meetings.opendev.org/meetings/kolla/2024/kolla.2024-01-24-14.00.log.html14:44
mmalchukthanks bbezak 14:44
SvenKieskethank you for running the meeting :)14:44
mattcreesCheers all14:45
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Fix ironic quorum queues don't support auto_delete property  https://review.opendev.org/c/openstack/kolla-ansible/+/90652715:04
kevkolet's see15:05
opendevreviewMerged openstack/kolla-ansible master: update openstack_previous_release_name var for bobcat  https://review.opendev.org/c/openstack/kolla-ansible/+/90651115:36
opendevreviewMerged openstack/kolla-ansible stable/2023.2: use 2023.2 images in stable/2023.2 branch  https://review.opendev.org/c/openstack/kolla-ansible/+/90650915:56
opendevreviewMatt Crees proposed openstack/kolla-ansible stable/2023.1: Support older Octavia var names in Antelope  https://review.opendev.org/c/openstack/kolla-ansible/+/90550016:10
opendevreviewMerged openstack/kolla-ansible master: Fix OpenSearch upgrade tasks idempotency  https://review.opendev.org/c/openstack/kolla-ansible/+/90585116:45
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: [CI] Fix podman cross-dependency build  https://review.opendev.org/c/openstack/kolla-ansible/+/90622917:03
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Rework horizon role to support local_settings.d  https://review.opendev.org/c/openstack/kolla-ansible/+/90634717:03
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Enable instance usage audit only when ceilometer is enabled  https://review.opendev.org/c/openstack/kolla-ansible/+/90550218:51
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Fix neutron DNS integration  https://review.opendev.org/c/openstack/kolla-ansible/+/90585218:51
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: [CI] Test neutron DNS integration and designate  https://review.opendev.org/c/openstack/kolla-ansible/+/90564418:51
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: [DNM] Trigger octavia tests - neutron internal DNS integration  https://review.opendev.org/c/openstack/kolla-ansible/+/90609918:51
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Fix neutron DNS integration  https://review.opendev.org/c/openstack/kolla-ansible/+/90585220:52
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: [CI] Test neutron DNS integration and designate  https://review.opendev.org/c/openstack/kolla-ansible/+/90564420:52
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: [DNM] Trigger octavia tests - neutron internal DNS integration  https://review.opendev.org/c/openstack/kolla-ansible/+/90609920:52
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Fix neutron DNS integration  https://review.opendev.org/c/openstack/kolla-ansible/+/90585220:57
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: [CI] Test neutron DNS integration and designate  https://review.opendev.org/c/openstack/kolla-ansible/+/90564420:57
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: [DNM] Trigger octavia tests - neutron internal DNS integration  https://review.opendev.org/c/openstack/kolla-ansible/+/90609920:57
opendevreviewMerged openstack/kolla-ansible stable/2023.1: Support older Octavia var names in Antelope  https://review.opendev.org/c/openstack/kolla-ansible/+/90550021:05
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: [CI] Test neutron DNS integration and designate  https://review.opendev.org/c/openstack/kolla-ansible/+/90564421:09
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: [DNM] Trigger octavia tests - neutron internal DNS integration  https://review.opendev.org/c/openstack/kolla-ansible/+/90609921:09
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Fix neutron DNS integration  https://review.opendev.org/c/openstack/kolla-ansible/+/90585221:33
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: [CI] Test neutron DNS integration and designate  https://review.opendev.org/c/openstack/kolla-ansible/+/90564421:33
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: [DNM] Trigger octavia tests - neutron internal DNS integration  https://review.opendev.org/c/openstack/kolla-ansible/+/90609921:33
opendevreviewMerged openstack/kolla-ansible master: ironic: only bootstrap ironic-tftp when service is enabled  https://review.opendev.org/c/openstack/kolla-ansible/+/89682023:41
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Rework horizon role to support local_settings.d  https://review.opendev.org/c/openstack/kolla-ansible/+/90634723:58

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!