Monday, 2022-09-05

*** rlandy is now known as rlandy|holiday00:08
marios\o05:06
chkumar|ruckmarios: o/ welcome back mate :-)05:09
marioso/ chkumar|ruck thanks :) 05:12
*** ysandeep|out is now known as ysandeep05:38
ysandeepgood morning oooci o/05:38
ysandeepmarios, welcome back! How was your vacation? 05:38
marioso/ ysandeep it was nice thank you05:42
ysandeep:) 05:47
mariosysandeep: is that right? https://review.opendev.org/c/openstack/tripleo-ci/+/854751/comment/6b287355_e5f48518/ 06:04
mariosysandeep: hmm no just confusing that they are both about dns in the end i think so probably that is the right one based on the comments in https://bugs.launchpad.net/tripleo/+bug/1983718 06:06
mariosysandeep: i was going to add a note to https://bugs.launchpad.net/tripleo/+bug/1983817 but now i think it is ok 06:06
ysandeepmarios, yes the one I mentioned is correct, slow firewall was dns issue06:06
mariosysandeep: thanks for confirming06:07
ysandeepfyi.. more details are on the bug.. but tl;dr version soon we will get a c9 node build via dib that will have unbound installed by default and we don't need to override nameservers anymore.06:08
mariosthanks ysandeep i read some of those comments interesting and good find and fix06:09
ykarelERROR! Error when getting the collection info for ansible.utils from default (https://galaxy.ansible.com/api/) (HTTP Code: 502, Message: Bad Gateway Code: Unknown)06:56
ykarelis ^ something known already?06:57
ykarelseen in https://zuul.opendev.org/t/openstack/build/e50d1b5f60c2412ba7db0ee79abc510b06:57
chkumar|ruckykarel: yes seen in one job06:59
chkumar|ruckykarel: ah you have more 06:59
chkumar|ruckinteresting06:59
ysandeepchkumar|ruck, ykarel, I have seen that kind of issue once in a while.07:00
ysandeepbut we can check if noticed in multiple jobs.07:01
ykarelchkumar|ruck, ysandeep that's fetching from external network and that's error prone07:01
*** ysandeep is now known as ysandeep|afk07:02
ykareli see difference in quickstart and job config likely that's where needs to be fixed07:03
ykarelhttps://opendev.org/openstack/tripleo-ci/src/branch/master/zuul.d/base.yaml vs https://opendev.org/openstack/tripleo-quickstart/src/branch/master/ansible-collection-requirements.yml07:03
ykarelor get those installed from rpms instead07:03
arxcruz|roverchkumar|ruck good morning07:05
arxcruz|roverchkumar|ruck need 10 min and we can sync :)07:05
chkumar|ruckarxcruz|rover: yes sure07:06
chkumar|rucklet me know when free, thank you :-)07:06
arxcruz|roverchkumar|ruck now :)07:13
jm1good morning folks :)07:26
anbanerj0/07:27
*** anbanerj is now known as frenzyfriday07:27
*** jpena|off is now known as jpena07:37
mariosakahat: o/ please see pvt when you have a chance07:55
marioso/ jm1 frenzyfriday arxcruz|rover 07:55
arxcruz|rovermarios yo07:55
frenzyfridaymarios, hey07:58
jpodivinreview list https://review.rdoproject.org/r/c/rdo-jobs/+/4469208:16
jpodivinreviewbot,  add in review list https://review.rdoproject.org/r/c/rdo-jobs/+/4469208:17
reviewbotI could not add the review to Review List08:17
frenzyfridaypls ignore the bot's response - it added the review to the hackmd08:19
frenzyfridayhey, did the downstream cockpit again go down?08:43
arxcruz|roverdpawlik hey, meeting?09:01
mariosfrenzyfriday: yes it has been down for me since this morning forgot to ask here chkumar|ruck arxcruz|rover do you know about d/stream cockpit not responding09:20
arxcruz|rovermarios i wasn't aware 09:21
chkumar|ruckyes it seems to be down, http://tripleo-cockpit.lab4.eng.bos.redhat.com/ (just checked right now)09:22
mariosarxcruz|rover: chkumar|ruck: k thanks let us know if you need someone else to handle it (have call in couple mins but can check after if you are busy with other stuff)09:23
chkumar|ruckmarios: anyone else take a look at this downstream cockpit, that would be great09:26
mariosok chkumar|ruck ... arxcruz|rover are you able to check it? ^^ 09:27
mariosarxcruz|rover: i just logged into psi ( chkumar|ruck do you know which tenant/instance it is on psi?)09:27
mariosarxcruz|rover: i will join call in 3 mins but can help you afer if you need09:28
arxcruz|rovermarios sorry, i'm in a meeting now, i'll ckeck soon, i don't know anything about downstream cockpit :) 09:28
mariosah the one called 'downstream cockpit' :D09:28
arxcruz|roverneed to check 09:28
mariosarxcruz|rover: k ^^^ can you please try that and let me know if you need help09:28
mariosarxcruz|rover: sending you private message now sec 09:28
mariosarxcruz|rover: see private and i can help after call if you need09:29
mariosakahat: o/ joining09:30
* frenzyfriday can check in some time why the downstream cockpit is down09:30
akahatmarios, o/ joining in a min.. 09:31
mariosthanks frenzyfriday just write here so we can coordinate 09:31
mariosakahat: ok 09:31
frenzyfridayprobably again something with c8/c8 docker stuff09:31
mariosakahat: can you hear me? cant hear you 09:32
jm1frenzyfriday: looking into downstream cockpit or shall i?09:34
*** ysandeep|afk is now known as ysandeep09:36
frenzyfridayjm1, sure, there is a tmux running for root user in the downstream cockpit server09:44
frenzyfridayI am checking there09:44
frenzyfridayall containers are up though..09:49
jm1frenzyfriday: looks like ansible-pull got stuck on 2022-08-03. killed all ansible processes and removed the lock. lets wait for ansible pull to rerun10:04
frenzyfridayoh, cool10:04
mariosthank you jm1 and frenzyfriday 10:12
arxcruz|rovergthiemon1e around? we are getting octavia failures on our jobs10:14
arxcruz|rovergthiemon1e https://logserver.rdoproject.org/45/44745/2/check/periodic-tripleo-ci-centos-9-scenario010-ovn-provider-standalone-master/c02c44d/logs/undercloud/var/log/tempest/stestr_results.html.gz10:14
arxcruz|rovergthiemon1e Details: b'{"faultcode": "Server", "faultstring": "Provider \'ovn\' reports error: create_vip_port() takes 4 positional arguments but 5 were given", "debuginfo": null}'10:14
arxcruz|rovergthiemon1e it seems related to this https://opendev.org/openstack/octavia/commit/d9ee63f561019c247a49de5805b6d9dcbafeeadf10:16
mariosjm1: frenzyfriday: do you have a call going (should i join) /do you need more eyes there ? 10:17
frenzyfridaymarios, we are working on root's tmux in the server10:17
jm1marios, frenzyfriday: downstream cockpit is nearly fixed..10:18
frenzyfridayjm1, thanks, I can access the dashboard now10:19
jm1marios, frenzyfriday: cockpit is recovering. will take some time till everything is updated10:19
jm1chkumar|ruck, frenzyfriday, marios: please vote https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/4474910:20
mariosjm1: thank you10:20
jm1its working as downstream cockpit shows10:20
mariosfrenzyfriday: for future reference, i cannot login do you need to add my keys? ping me/private with info please? (username?)10:20
jm1marios: please try to ssh into http://tripleo-cockpit.lab4.eng.bos.redhat.com/10:22
arxcruz|rovergthiemon1e the root cause is that the create_vip_port was updated to have 5 args but the ovn driver wasn't https://github.com/openstack/ovn-octavia-provider/blob/master/ovn_octavia_provider/driver.py#L44310:22
mariosjm1: re /ci-config/+/44749 - voted, but had to dig a bit to find that the lookup accepts a default= parameter it seemed 'wrong' that |default wasn't working there at first glance10:27
jm1marios: oh sorry, i forgot to link the docs 😬10:27
mariosjm1: frenzyfriday: thanks but root? centos? mandreou? dont seem to work 10:28
mariosfrenzyfriday: gimme user name in private please? 10:28
*** ysandeep is now known as ysandeep|afk10:30
jm1marios, frenzyfriday: nothing private, all in the open https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/infra-setup/tenant_vars/common.yml#L4310:30
mariosthanks jm1 frenzyfriday 10:30
mariosjm1: ;)10:30
jm1frenzyfriday, marios: downstream cockpit has issues updating its data10:31
frenzyfridayhm.. says zuulv3_downstream.py is missing10:32
jm1frenzyfriday: this and other messages before10:32
mariosjm1: thanks i was about to say 'its working' 10:32
mariosat least there is a response now anyway so progress ;)10:33
*** ysandeep|afk is now known as ysandeep10:39
*** marios is now known as marios|fooood10:50
*** marios|fooood is now known as marios11:13
mariosanyone need to discuss reviews? 11:15
reviewbotDo you want me to add your patch to the Review list? Please type something like add to review list <your_patch> so that I can understand. Thanks.11:15
mariosonly see akahat patches there and we discussed those earlier11:16
*** dviroel|out is now known as dviroel11:28
bhagyashrismarios, Done11:34
bhagyashristhanks :)11:34
mariosbhagyashris: ack will check in sec 11:36
mariosbhagyashris: do you want to sanity recheck the test too? 11:48
mariosalso minor comment about the vars: are they needed bhagyashris ? 11:50
*** frenzyfriday is now known as frenzyfriday|lunch12:02
mariosakahat: please check when you have a minute https://review.opendev.org/c/openstack/tripleo-quickstart/+/853860/comment/8bf81e6b_5a8a1905/ why did you add those?12:04
akahatmarios, if we want to run component line then we need to do changes in CentOS-8-9/wallaby.yml right?12:06
mariosakahat: yes... earlier you had the promotion testing hash (and the testproject was using that) and you said you'd update those files. but indeed looking at the other networks it is using the non promotion testing hash ones e.g. there 12:11
marioshttps://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-network-wallaby/494480c/logs/quickstart_files/playbook_executions.log12:11
mariosakahat: so then remove the promotion testing hash change as commented please 12:11
mariosakahat: ok?12:14
bhagyashrismarios, let me check12:14
mariosbhagyashris: thanks, maybe worth removeing those vars and re-run the test anyway to check both reparenting and vars are OK then 12:15
bhagyashrismarios, ack12:15
akahatmarios, ack.12:16
mariosakahat: but otherwise i think this should be good https://review.opendev.org/c/openstack/tripleo-quickstart/+/853860/10/config/release/tripleo-ci/CentOS-9/wallaby.yml 12:16
mariosakahat: well done12:17
gthiemon1earxcruz|rover: sorry I have missed your messages, there's a fix in the ovn-octavia-provider for this issue: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/85476412:17
mariosi think it should be ok since those ones are going to be set only for periodic definitions12:17
mariosakahat: but the test will show us 12:17
*** gthiemon1e is now known as gthiemonge12:17
arxcruz|rovergthiemonge great, thanks12:18
akahatmarios, but we tests are not good. job.provider_dlrn_hash_tag_branch and provider_dlrn_hash_distro vars are not present.12:18
akahathttps://logserver.rdoproject.org/99/44499/9/check/periodic-tripleo-ci-centos-8-9-multinode-mixed-os-compute-wallaby/7e25f83/job-output.txt12:18
akahatmay be we need to add checks for this also.12:18
mariosakahat: k was staring at it but have not checked your tests yet (didn't think it ran yet, are you not going toupdate the job definitions too?)12:19
mariosakahat: the tag_branch and hash_distro are used with content providers so wont be available here no 12:20
mariosah but akahat 12:20
marios  {{ job.provider_dlrn_hash_tag_branch[release]|default('current-tripleo') }}12:20
mariosit should be saved by that default 12:20
akahatmarios, yes. that's the point. 12:21
marioswhich is what the component jobs are currently using 12:21
mariosakahat: i.e. |default('current-tripleo')12:21
*** tosky_ is now known as tosky12:23
akahatmarios, ok. let me try something else. if it's default value is current-tripleo then we don't need to create any change for tq.12:23
mariosakahat: k 12:25
mariosakahat: problem is there https://opendev.org/openstack/tripleo-ci/src/commit/0b87643560e678c9847b411db313a62e6a3b3d1d/roles/run-test/templates/toci_quickstart.sh.j2#L104-L10812:26
mariosakahat: cos we are setting registry_distro so it is going to be present/defined in the component jobs too 12:27
mariosakahat: conditional needs some more thought12:27
mariosakahat: maybe something with " job.component is defined" to check that it is a component job12:28
akahatyeah.. that is another way around.12:28
mariosakahat: k have a go/stare at it a bit ;) and update/rerun test 12:29
jm1afuscoar, frenzyfriday|lunch: did downstream dashboard by cre ever work?12:32
frenzyfriday|lunchjm1, no, I think the last time it had some missing data as well12:38
jm1frenzyfriday|lunch, afuscoar: yeah it should have never worked. only with manual tweaking. fixing it right now12:40
jm1ysandeep: hello :) here today?13:04
ysandeepjm1, hello o/13:05
ysandeepjm1, yes just working quietly :D13:05
jm1ysandeep: oh great :) will pm you ^^13:05
afuscoarHello jm1, frenzyfriday|lunch, no, it didn't, sorry for not answering before and thanks for fixing it13:24
pojadhavscrum : ysandeep chkumar|ruck akahat 13:31
arxcruz|rovermarios https://logserver.rdoproject.org/openstack-periodic-integration-stable1-cs8/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-build-containers-centos-8-quay-push-wallaby/deccc8c/logs/container-builds/13:34
jm1afuscoar, frenzyfriday|lunch: please review :) https://code.engineering.redhat.com/gerrit/c/openstack/rrcockpit/+/42736813:43
*** frenzyfriday|lunch is now known as frenzyfriday13:43
jm1afuscoar, frenzyfriday|lunch: the script still has an issue as noted in the comment but that can be fixed later13:43
mariosarxcruz|rover: but what failed there cant see the error in https://logserver.rdoproject.org/openstack-periodic-integration-stable1-cs8/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-build-containers-centos-8-quay-push-wallaby/deccc8c/job-output.txt 13:46
mariosakahat: o/ scrum?13:46
arxcruz|rovermarios the failure there is just a pytest to check if the containers were created properly 13:46
arxcruz|roverif you check tne containers-build directory13:46
mariosarxcruz|rover: how about these logs did you already check there https://logserver.rdoproject.org/openstack-periodic-integration-stable1-cs8/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-build-containers-centos-8-quay-push-wallaby/deccc8c/logs/container-builds/017f7a87-e5ca-4a02-987e-ddeb8035c5ca/base/os/nova-base/ 13:47
arxcruz|roverhttps://logserver.rdoproject.org/openstack-periodic-integration-stable1-cs8/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-build-containers-centos-8-quay-push-wallaby/deccc8c/logs/container-builds/017f7a87-e5ca-4a02-987e-ddeb8035c5ca/base/os/nova-base/nova-compute/13:47
arxcruz|rovermarios yes, nova-base container is build properly13:47
arxcruz|roverbut nova-compute, nova-api, nova-scheduler for example doesn't 13:47
arxcruz|roverand there is no build log for those 13:47
jm1dasm, frenzyfriday: downstream cockpit is in development state aka its not running ansible-pull and i changed some code in the repos14:06
jm1dasm, frenzyfriday: i will revert it to normal operation once my patches have been merged. please do not touch it for now, thanks :)14:06
rlandyysandeep++ for getting me off the mark on first PR14:10
rlandydviroel: ^^ I started :)14:10
jm1arxcruz|rover, chkumar|ruck, marios: please ping me when downstream cockpit is returning strange or no results :)14:10
dviroelrlandy: nice, I can take a look afterwards :)14:11
rlandydviroel: thanks - going to try doc newbie process now14:11
arxcruz|roverjm1 oky doky 14:11
mariosok jm1 14:11
mariosarxcruz|rover: first error in that build.log is for cinder not nova see [Errno 2] No such file or directory: '/home/zuul/container-builds/017f7a87-e5ca-4a02-987e-ddeb8035c5ca/base/os/cinder-base/cinder-backup/cinder-backup-build.log':14:14
mariosarxcruz|rover: (from there https://logserver.rdoproject.org/openstack-periodic-integration-stable1-cs8/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-build-containers-centos-8-quay-push-wallaby/deccc8c/logs/build.log )14:14
arxcruz|rovermarios no, this is already the checking 14:14
arxcruz|roverafter everything is build14:14
mariosah 14:15
arxcruz|rovermarios and yes, nova is just the ones i'm tracking for now, but there were others failing 14:15
rlandymarios: hey - welcome back14:15
mariosarxcruz|rover: interesting seems like it was truncated there (last text is 'le')14:18
mariosrlandy: o/ 14:18
mariosrlandy: i was told you're out today 14:18
rlandymarios: yep - public holoday here14:19
rlandyholiday14:19
mariosrlandy: thanks good to be back amongst the cis14:19
mariosarxcruz|rover: truncated 'there' https://logserver.rdoproject.org/openstack-periodic-integration-stable1-cs8/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-build-containers-centos-8-quay-push-wallaby/deccc8c/logs/container-builds/017f7a87-e5ca-4a02-987e-ddeb8035c5ca/base/base-build.log 14:19
mariosarxcruz|rover: ah no... refresh fixed it :D14:20
mariosarxcruz|rover: so start by filing the bug anyway and then maybe someone else can tell us how to get th elogs we need14:20
mariosarxcruz|rover: but without a bug you'll have to explain each time 14:20
arxcruz|rovermarios in a few, working on another issue14:23
afuscoarjm1 I've fixed it and published a new patchset14:23
mariosarxcruz|rover: the other option is to get a testproject and hold the node maybe you can run the build command and get more info there 14:24
*** ysandeep is now known as ysandeep|out14:26
jm1afuscoar: oh very nice, thank you!14:28
jm1afuscoar: let me rebuild the downstream cockpit14:28
afuscoarThank you jm114:30
jm1reviewbot: add to review list https://code.engineering.redhat.com/gerrit/c/openstack/rrcockpit/+/427368/14:35
reviewbotI could not add the review to Review List14:35
jm1rlandy, ysandeep: please +w this when you are back :) https://code.engineering.redhat.com/gerrit/c/openstack/rrcockpit/+/427368/14:38
afuscoarjm1 btw is http://tripleo-cockpit.lab4.eng.bos.redhat.com/ not accessible yet right?14:38
jm1afuscoar: i am rebuilding it right now. updating the script requires rebuilding the container image 14:39
jm1afuscoar: (of course we could manually patch the container image but.. no.. XD)14:40
afuscoarThat's right :D14:41
jm1afuscoar: up again and it starts collecting data14:43
jm1afuscoar: due to config changes the commands will not all be run at the same time. they will run random withing 180 seconds to not overload zuul14:44
jm1afuscoar: so it might take some minutes till everything is updated14:45
afuscoarIt's totally okay, now I see the data in the dashboard. Thank you very much jm1 and frenzy_friday14:45
jm1afuscoar: oh nice pie :)14:45
jm1afuscoar: next time you submit changes please remind us to rebuild the container image in case we forget ^^14:46
afuscoarPerfect, thanks for the help! 14:47
rlandylooks like it's w+'ed14:47
afuscoarjm1++14:47
rlandyjm1: now it's submitted: https://code.engineering.redhat.com/gerrit/c/openstack/rrcockpit/+/42736814:48
jm1rlandy: oh interesting workflow. thanks anyway :)14:49
rlandydviroel: hi ... https://github.com/openshift/release/pull/31997 "tide Pending — Not mergeable. Must be by author openshift-bot OR Needs lgtm label."15:18
rlandywhat do I do there?15:18
rlandyhttps://github.com/openstack-k8s-operators/cinder-operator/pull/10 - still waiting on merge15:19
dviroelrlandy: "Needs lgtm label."15:23
dviroelmissing just a /lgtm15:23
rlandyso I add that or a reviewer?15:24
chkumar|ruckarxcruz|rover: see ya tomorrow! leave notes, if anything to pickup in morning.15:24
arxcruz|roverchkumar|ruck cya, i'll stay for 30 more min 15:24
dviroelrlandy: a reviewer, I think that we can do that, let me review15:25
arxcruz|roverchkumar|ruck but i'll monitor diring night 15:25
rlandydviroel: thanks15:25
dviroelrlandy: lgtm, should merge soon15:31
* dviroel lunch15:32
rlandyawesomeness15:32
*** dviroel is now known as dviroel|lunch15:32
rlandydoc'ing my notes now15:32
*** marios is now known as marios|out15:51
rlandylunch brb16:13
*** jpena is now known as jpena|off16:38
*** dviroel|lunch is now known as dviroel16:41
* jm1 out for today, have a nice evening :)18:33
* rcastillo out for the day21:05
rcastilloo/ have a good evening21:05
dviroelrlandy: around?21:28
dviroelhope not, it is holiday :)21:28
* dviroel out, openshift-ci seems to be broken21:29
* dviroel i hope that isn't my fault 21:29
* dviroel /o/21:29
*** dviroel is now known as dviroel|out21:29
rlandydviroel: how would that be your fault?22:50
rlandyysandeep|out: ^^ when you are in ... we have an issue with tests 22:53
rlandyhttps://github.com/openstack-k8s-operators/cinder-operator/pull/1022:53
*** rlandy is now known as rlandy|out22:54

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!