Thursday, 2022-02-03

« Wednesday, 2022-02-02 Index Friday, 2022-02-04 »
*** rlandy|ruck is now known as rlandy|out00:22
*** pojadhav|afk is now known as pojadhav03:16
bhagyashrisfrenzyfriday, hi ananya04:27
bhagyashrisi have added some comments here https://review.rdoproject.org/r/c/rdo-jobs/+/3745704:39
bhagyashrisi have few questions there could you help me to understand those? thanks 04:39
frenzyfridaybhagyashris, Hey, checking in some time04:42
bhagyashrisnp04:42
*** marios is now known as marios|ruck06:03
anbanerjbhagyashris, /me creating hackmd for trigger jobs06:04
*** anbanerj is now known as frenzyfriday06:04
bhagyashrisfrenzyfriday, yeah just see...06:06
*** ysandeep|out is now known as ysandeep06:23
frenzyfridaybhagyashris, https://hackmd.io/BZvWtssoRQGsJBYETKWq2A Lemme know if this makes sense 06:27
bhagyashrisfrenzyfriday, hey06:28
bhagyashriswdyt about this comment https://review.rdoproject.org/r/c/rdo-jobs/+/37457/4/zuul.d/component-jobs-master-centos9.yaml#68206:29
marios|ruckakahat: can you please check asap at https://review.opendev.org/c/openstack/tripleo-ci/+/823688/26#message-e3302a560df0e9dbd4123faadf2585bfdfca35b2 - mainly that https://review.opendev.org/c/openstack/tripleo-ci/+/823688/26/playbooks/tripleo-buildcontainers/pre.yaml#22 (nit on task name) and https://review.opendev.org/c/openstack/tripleo-ci/+/823688/26/playbooks/tripleo-buildcontainers/run.yaml#6 07:18
marios|ruck(mirrors not picked up for 8)07:18
marios|ruckakahat: the second one as commented there not sure if the test ran before you updated? 07:19
marios|ruckakahat: let me know if you want to jump on the phone might be quicker to discuss 5 mins07:19
marios|ruckakahat: otherwise looks good lets try get it in today 07:19
marios|ruckakahat: and please add periodic 9 container build into the test https://review.rdoproject.org/r/c/testproject/+/38921/3/.zuul.yaml#1907:21
*** amoralej|off is now known as amoralej07:26
akahatmarios|ruck, ack.. looking.07:27
marios|ruckakahat: thanks mate07:27
marios|ruckakahat: let me know if you want to chat07:27
akahatyeah sure.07:27
marios|ruckakahat: got it.. the problem is there https://opendev.org/openstack/tripleo-ci/src/commit/39e59fcb9923d1ab59970136a1b97e33faba5b0b/playbooks/tripleo-buildimages/run-v3.yaml#L6 07:28
marios|ruckakahat: ^^^ you need to update that one too 07:28
marios|ruckakahat: then mirrors should be good07:28
marios|ruckakahat: please check the rest of the comments should be quick update so lets get it through check & rerunt estproject and we can merge 07:29
akahatmarios|ruck, yes. i'm on it. 07:29
marios|ruckakahat: thanks ;) sorry i need to slow down on the caffeine i will stop harrassing you now07:30
marios|ruck(for a bit) ;)07:30
akahatmarios|ruck, updated both patches. let's wait for zuul run.07:48
marios|ruckakahat: thanks 07:48
marios|ruckakahat: one nit please update before the test runs too far? 07:50
marios|ruckakahat: i'm updating (tell me if you are i'll stop)07:51
marios|ruckakahat: done07:52
amoralejsome changes related to centos have been merged in upstream infra roles, let me know if you observe anything abnormal08:04
marios|ruckthanks amoralej 08:04
slaweqamoralej: hi, can You maybe put for me node where job periodic-tripleo-ci-centos-9-scenario007-multinode-oooq-container-wallaby from patch https://review.rdoproject.org/r/c/testproject/+/38969 runs? It's related to the bug https://bugs.launchpad.net/tripleo/+bug/1959582 which I would like to investigate a bit more if possible08:43
slaweqthx in advance08:43
amoralejslaweq, you mean to hold the node?08:44
marios|ruckamoralej: i will help slaweq thanks08:44
slaweqamoralej yes, can You do that?08:45
marios|ruckslaweq: there is nit on the test08:45
amoralejack08:45
marios|ruckslaweq: update it 08:45
slaweqahh, ok08:45
marios|ruckslaweq: and will hold the node for you 08:45
slaweqthx marios|ruck08:45
marios|ruckslaweq: will you update the testproject 10:45 < marios|ruck> slaweq: there is nit on the test08:47
marios|ruckslaweq: ah sec probably the job isn't merged yet let me find it you can add depends on08:47
slaweqmarios|ruck thx08:48
marios|ruckysandeep: where is your node hold tool? I am getting expired password from my zuul/client.conf but i don't have access to the node where i can generate new one 08:59
ysandeepmarios|ruck, in ci-config, let me grab that link for you08:59
marios|ruckysandeep: thx 09:00
marios|ruckysandeep: but do i have to anyway create my own client.conf right? so i think i have the same problem 09:00
ysandeepmarios|ruck, pm09:00
marios|ruckysandeep: thanks and i see it in ci-config now 09:00
marios|ruckk thanks ysandeep i have same issue with token will see who can get me access to that machine or refresh tokens 09:02
marios|ruckthanks ysandeep !09:05
marios|ruckslaweq: let me know when the job fails and we can get you on there 09:10
marios|ruckslaweq: ( https://review.rdoproject.org/zuul/status#38969 )09:15
marios|ruckbhagyashris: o/ hey any idea why we have no victoria logs @ http://promoter.rdoproject.org/promoter_logs/09:23
*** ysandeep is now known as ysandeep|lunch09:23
marios|ruckbhagyashris: ah there http://10.0.148.74/promoter_logs/ 09:24
akahatmarios|ruck, we have victoria logs here: http://10.0.148.74/promoter_logs/?C=M;O=D09:24
marios|ruckheh you just wrote on google chat too :D09:24
marios|ruckthanks bhagyashris akahat 09:25
akahatmarios|ruck, testproeject looks good: https://review.rdoproject.org/r/c/testproject/+/3892109:26
marios|ruckakahat: nice can you please sanity check the mirrors and add links/pointers on the review or testproject 09:30
marios|ruckakahat: (i.e. make sure the repo have right baseurl 09:30
marios|ruckbrb09:30
marios|ruckdviroel|out: o/ can you check when you can please https://bugs.launchpad.net/tripleo/+bug/1956475/comments/7 (i see wallaby job isn't merged yet you are working there with testproject/depends-on)10:02
akahatmarios|ruck, looks like opstools is by default pointing to the master10:28
akahatwe have all repos good.. except opstool10:28
akahatit's even in the delorean-deps also10:29
akahathttps://logserver.rdoproject.org/21/38921/4/check/periodic-tripleo-ci-build-containers-ubi-8-push-wallaby/d82c346/etc/yum.repos.d/quickstart-centos-opstools.repo10:29
akahathttps://logserver.rdoproject.org/21/38921/4/check/periodic-tripleo-ci-build-containers-ubi-8-push-wallaby/d82c346/etc/yum.repos.d/delorean-wallaby-deps.repo10:29
*** ysandeep|lunch is now known as ysandeep10:30
marios|ruckakahat: yeah that one is hardcoded as temp workaroudn so find (opstools)10:33
marios|rucks/find/fine10:34
akahatmarios|ruck, okay. if this is the case then it's okay.10:35
akahatand cepth-nautilus also: https://logserver.rdoproject.org/21/38921/4/check/periodic-tripleo-ci-build-containers-ubi-8-push-victoria/16c25a2/etc/yum.repos.d/quickstart-centos-ceph-nautilus.repo10:36
marios|ruckakahat: yes10:36
marios|ruckakahat: https://review.opendev.org/c/openstack/tripleo-quickstart/+/82727910:36
marios|ruckakahat: opstools and ceph-nautilus are OK they should be using trunk.rdoproject.org for now will be fiexed when we revert in the release files10:37
akahatokay.. good then.10:37
*** rlandy|out is now known as rlandy|ruck11:13
*** dviroel|out is now known as dviroel11:16
marios|ruckslaweq: o/ gimme pub key so i can add you to node11:19
marios|ruckslaweq: node is ready 11:19
slaweqmarios|ruck https://paste.opendev.org/show/812499/11:19
marios|ruckslaweq: k sec 11:20
dviroelmarios|ruck: o/ - wrt bug/1956475, we need to testproject + revert of skiplist - both master and wallaby11:21
marios|ruckslaweq: pvt11:22
marios|ruckslaweq: let me know if you can get in 11:23
ysandeeprlandy|ruck, this is amol's patch https://review.opendev.org/c/openstack/tripleo-ci/+/823688 ?11:25
marios|ruckslaweq: (I gave you IP in pvt )11:25
slaweqthx marios|ruck11:28
slaweqI will get to it in about 10-15 minutes11:28
marios|ruckslaweq: k 11:28
marios|ruckdviroel: thanks - are you doing that or should I ? 11:30
marios|ruckdviroel: in call now but can check after11:30
rlandy|ruckgthiemonge: hey ... just touching base on: https://review.opendev.org/c/openstack/octavia-tempest-plugin/+/81636911:33
rlandy|ruckhttps://review.opendev.org/c/openstack/octavia/+/81637011:33
rlandy|ruckanything we can do to help get these patches in?11:33
rlandy|ruckthanks11:33
dviroelmarios|ruck: i can do it11:34
dviroelmarios|ruck: only for master now, they are still backporting the fix11:34
dviroelmarios|ruck: ah, looking at the wrong bug-topic, we should be able to test master and wallaby11:37
marios|ruckdviroel: thank you 11:37
marios|ruckslaweq: it expires in 24 hours if you need we can rehold later11:37
rlandy|ruckysandeep: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/3879711:41
rlandy|ruckysandeep: marios|ruck: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/3882311:43
marios|ruckbhagyashris: do we have an outstanding patch to wire up the wallaby 9 component line? 11:46
marios|ruckbhagyashris: we see https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/3882311:46
marios|ruckbhagyashris: jobs are merged but ... k seems we found it ;)11:47
rlandy|ruckhttps://review.rdoproject.org/r/c/config/+/3870911:47
rlandy|ruckwas in merge conflict11:47
ysandeeprlandy|ruck, marios|ruck https://review.rdoproject.org/r/c/rdo-jobs/+/3836611:50
rlandy|ruckbhagyashris: periodic-tripleo-ci-centos-9-scenario004-standalone-tripleo-wallaby  is not defined ... pls see11:50
rlandy|ruckhttps://review.rdoproject.org/r/c/config/+/3870911:50
rlandy|ruckzuul comments11:50
marios|ruckbhagyashris: (same topic^^^ https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/38823/3#message-0ee8d2e2aebbfbebe3b5e0f94ca3a5f64474c9e2 )11:51
gthiemongerlandy|ruck: hi, I will ping the core reviewers to merge them quickly11:53
rlandy|ruckgthiemonge: great - thanks11:54
marios|ruckbhagyashris: ysandeep: is doing that btw so it is OK sorry for noise ;)11:54
ysandeepbhagyashris, https://review.rdoproject.org/r/c/rdo-jobs/+/3897711:57
rlandy|ruckdviroel: hey - can you review/merge https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/38967?11:57
*** pojadhav is now known as pojadhav|brb11:57
dviroelrlandy|ruck: looking12:03
* ysandeep afk for ~40 mins 12:08
*** ysandeep is now known as ysandeep|afk12:08
rlandy|ruckakahat: tooling meeting?12:08
rlandy|ruckabregman: ^^?12:09
*** dviroel is now known as dviroel|biab12:10
*** pojadhav|brb is now known as pojadhav12:10
akahatrlandy|ruck, done. Arie scheduled one week later.12:19
akahati'll sent invite to the team.12:19
rlandy|ruckakahat: sure12:20
akahatrlandy|ruck, those jobs are not running zuul, do we need to remove them: https://paste.opendev.org/show/812470/12:24
rlandy|ruckakahat: they run in downstream zuul12:25
rlandy|ruckremember? the downstream trigger project12:26
rlandy|ruckplease don't remove them from anywhere12:26
rlandy|ruckthey run and report to dlrn12:26
akahatyes. i remember12:26
akahatokay then that should not be a problem12:26
*** dviroel|biab is now known as dviroel12:27
dviroelrlandy|ruck: o/ - need to ask your vote again on this one https://review.rdoproject.org/r/c/rdo-jobs/+/36893 - I will merge then - promise12:47
dviroelrlandy|ruck: small fix after your last vote (https://review.rdoproject.org/r/c/rdo-jobs/+/36893/25..27/playbooks/dependency/setup-compose-repos.yml)12:47
rlandy|rucklooking12:47
rlandy|ruckdone12:48
dviroeltyvm12:48
rlandy|ruckdviroel: hi ...12:59
rlandy|ruckhttp://promoter.rdoproject.org/config/CentOS-9/12:59
rlandy|ruckwallaby 912:59
rlandy|ruckneeds to be updated13:00
rlandy|ruckbhagyashris: ^^13:00
rlandy|ruckto promote images13:00
dviroelrlandy|ruck: ack, will do13:00
bhagyashrisrlandy|ruck, ack13:00
*** amoralej is now known as amoralej|lunch13:00
rlandy|ruckand add all the new tests we have13:00
rlandy|ruckthank you!13:00
dviroelrlandy|ruck: yesterday c9-wallaby was failing image builds - running tests atm13:00
rlandy|ruckk13:01
dviroelrlandy|ruck: s/image/container13:01
bhagyashrisrlandy|ruck, dasm|off rcastillo marios|ruck 13:01
bhagyashrisscrum time13:01
*** ysandeep|afk is now known as ysandeep13:01
bhagyashrisreview list from scrum13:06
bhagyashrishttps://review.rdoproject.org/r/c/rdo-infra/ci-config/+/38479, https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/38262, https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/37946, https://hackmd.io/BZvWtssoRQGsJBYETKWq2A, https://review.opendev.org/c/openstack/ansible-collections-openstack/+/827274 13:06
sshnaidmfrenzyfriday, you can ask jm1 about devstack, he might have one13:08
frenzyfridaysshnaidm, yep, I'll steal his devstack if it doesnt work on mine :D13:09
slaweqmarios|ruck thx for the node, I think I found something there. I proposed backport https://review.opendev.org/c/openstack/tripleo-heat-templates/+/827642 as I think this may be the root cause of the problem. Lets test it in https://review.rdoproject.org/r/c/testproject/+/38969 to see how it will go :)13:11
slaweqYou can clean the hold node now13:11
*** dasm|off is now known as dasm|rover13:14
dasm|rovero/13:14
marios|ruckthanks slaweq 13:44
*** dviroel is now known as dviroel|ruck13:44
marios|ruckslaweq: ok clearing the node hold now13:44
*** akahat is now known as akahat|rover13:45
dasm|roverakahat|rover: i got an invitation for Feb 14 meeting. it starts at 4am my time. Are planning to record that?13:45
marios|ruckdasm|rover: akahat|rover: are you in such a hurry ;D13:45
marios|rucki think you folks can start tomorrow otherwise you're cheating yourself 13:46
marios|ruckbut sure 13:46
* marios|ruck hands over the launch codes and runs away 13:46
marios|ruck;)13:46
dasm|rovermarios|ruck: Nooooooooo!!!!111113:46
akahat|roverdasm|rover, okay. I'll ask them to record it.13:46
dasm|roverakahat|rover: thank you!13:46
marios|ruckdasm|rover: ;)13:46
dasm|rovermarios|ruck: this: http://www.nooooooooooooooo.com/13:47
*** dviroel|ruck is now known as dviroel13:48
marios|ruckhaha dasm|rover 13:48
akahat|rover:)13:48
dviroelok, i will start ruck after marios eod13:48
dviroel;)13:48
rlandy|ruckrechecking wallaby component wire-up patch13:49
dviroelbut ack, should start tomorrow13:49
* dviroel coffee13:49
*** akahat|rover is now known as akahat13:56
marios|ruckysandeep: please see pvt when you have a sec thanks13:59
marios|ruckthanks ysandeep 14:01
ysandeepyou'r welcome :)14:01
marios|ruckysandeep: node_holder is great much nicer than those huge podman run commands 14:02
marios|ruckgreat job ysandeep 14:02
ysandeepmarios|ruck: thanks14:03
ysandeepIf it helps i am thinking about adding a script that can trigger our pipelines - so that we don't have to run long podman commands to trigger pipelines on purpose - but we don't trigger that daily - may be once in a while14:04
ysandeepmarios|ruck, rlandy|ruck ^^ thoughts 14:04
marios|ruckysandeep: you mean to run e.g. 'openstack-integration-main' like that? 14:04
ysandeepmarios|ruck, yes14:04
ysandeepor any component line 14:05
dviroelysandeep: I think that is a good idea, you should be able to reuse your code/classes from node_holder tool14:06
dviroelright?14:07
ysandeepdviroel, yes.. extend node_holder.py or reuse classes/functions from there14:07
dviroel+114:07
marios|ruckysandeep: yea it would be useful... now if we want to 'run the train line' you have to copy/paste all 14:07
ysandeepdviroel, marios|ruck ack, thanks.. adding to list after we are done with c9/rhel914:08
marios|ruckysandeep: for example i was trying to get something like this to run https://review.rdoproject.org/r/c/testproject/+/38885 but couldn't14:08
marios|ruckysandeep: (e.g. just use the template) but must be something in our config that prevents it i didn't have time to dig much on it14:08
rlandy|rucksure14:09
ysandeepmarios|ruck, hmm, I see you were using depends-on there, not sure the way i had in mind can work with depends-on - but i will explore14:10
*** amoralej|lunch is now known as amoralej14:19
ysandeeprlandy++14:29
rlandy|ruck:)14:32
rlandy|rucknp14:32
rlandy|ruckstarting on the trigger job14:32
marios|ruckrlandy|ruck: did you say you already re-ran 20 somewhere or should i recheck ? https://review.rdoproject.org/r/c/testproject/+/38972/1#message-99dc5aac9621451c02ac08aa317e6278426538ce14:48
marios|ruckrlandy|ruck: s/recheck/rerun 20 there 14:48
dviroelrlandy|ruck: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/3899214:48
rlandy|ruckmarios|ruck 21853,4014:49
rlandy|ruckalready in recheck14:49
marios|ruckrlandy|ruck: ack thanks14:49
rlandy|ruckdviroel: pls add to review list14:49
rlandy|ruckwill take care after I am done with trigger job14:49
dviroelsure14:50
slaweqdviroel marios|ruck: with missing backport test patch https://review.rdoproject.org/r/c/testproject/+/38969 is green now15:00
slaweqso if You can review https://review.opendev.org/c/openstack/tripleo-heat-templates/+/827642 that would be great15:01
marios|ruckslaweq: nice thanks15:01
slaweqyw15:02
bhagyashrisrlandy|ruck, dviroel marios|ruck fyi https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/38997 Enable image promotion for cs9 wallaby15:03
dviroelslaweq: thanks o/15:03
ysandeeprlandy|ruck, https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/308841/3/playbooks/tripleo-buildcontainers/pre_minimal.yaml 15:03
ysandeeptrying here https://code.engineering.redhat.com/gerrit/c/testproject/+/21019715:03
* dviroel manila community meeting15:04
marios|ruckbhagyashris: ack but is it what we have in other branches? https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/38997/1#message-f0485ea2eca00d065ece9fb60be3359197e742b0 i.e. should we instead just add qcow_client there in wallaby config? 15:05
rlandy|ruckysandeep: https://code.engineering.redhat.com/gerrit/q/topic:%22trigger-https%22+(status:open%20OR%20status:merged)15:05
ysandeeprlandy|ruck, thanks will look them after checkpoint call15:06
* frenzyfriday afk15:07
marios|ruckbhagyashris: k looks same for master 15:08
* akahat dinner15:12
rlandy|ruckperiodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-network-master (2nd attempt)15:16
rlandy|ruck^^ failed15:16
rlandy|ruckneed to rerun?15:16
rlandy|ruckttps://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-network-master/4c4edc3/logs/undercloud/var/log/tempest/stestr_results.html.gz15:17
rlandy|ruckholy cow15:17
rlandy|rucknot one working test there15:17
rlandy|ruckmarios|ruck: dasm|rover: dviroel: ^^15:18
rlandy|ruckgoing to rerun15:18
rlandy|ruckmay be legit here15:18
dviroelOSError: [Errno 101] Network is unreachable15:18
dviroelyeah, try again15:18
marios|ruckrlandy|ruck: thanks 15:19
dasm|roverunbelievable15:19
marios|ruckthat job doesn't look too stable in general :/ https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-network-master15:19
dasm|rover> Failed to establish a new connection: [Errno 101] Network is unreachable'))15:19
marios|ruckbut i would hope those tempest results are not reproducible rlandy|ruck 15:20
marios|rucklets see on recheck 15:20
rlandy|ruckhttps://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-network-master/af6b4bd/logs/undercloud/var/log/tempest/stestr_results.html.gz15:22
rlandy|rucksame on 815:22
marios|ruck:(15:22
marios|ruckwhy does promoting c9 ovb have to be so complicated 15:23
rlandy|ruckthat's why they pay us the big bucks15:23
rlandy|rucklooks legit15:23
rlandy|ruckreal failure across the board here15:23
marios|ruckrlandy|ruck: yeah see https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-network-master 15:23
rlandy|ruckdasm|rover: ^^ want to check this out??15:23
dasm|roveryeah15:24
dasm|roverlet's see what we got here15:24
marios|ruckdasm|rover: we'll need a new bug please for that see ^^^ both 8 and 9 looks like for network component fs1 15:25
rlandy|ruckfailure15:25
rlandy|ruckwas in deploy15:25
dasm|rover> Provisioning started on node baremetal-12059-1 (UUID c8b7cd9d-d049-498d-904d-f3422a245e10)\n", "msg": "Node 6d112eed-c1d9-428f-92c8-d772b07ad504 reached failure state \"deploy failed\"; the last error is Timeout reached while waiting for callback for node 6d112eed-c1d9-428f-92c8-d772b07ad504"}15:28
ysandeeprlandy|ruck, moved forward and hitting below now in container build 15:28
ysandeep2022-02-03 15:12:13.635295 | LOOP [build-containers : Install pytest]15:28
ysandeep2022-02-03 15:12:13.783141 | primary | ERROR15:28
ysandeep2022-02-03 15:12:13.783651 | primary | {15:28
ysandeep2022-02-03 15:12:13.783718 | primary |   "msg": "The task includes an option with an undefined variable. The error was: 'ensure_pip_virtualenv_command' is undefined\n\nThe error appears to be in '/var/lib/zuul/builds/47242da13ae24ca49feb4662b4aedee4/untrusted/project_3/opendev.org/openstack/tripleo-ci/roles/build-containers/tasks/build-report.yaml': line 77, column 7, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe 15:28
ysandeepoffending line appears to be:\n\n\n    - name: Install pytest\n      ^ here\n"15:28
ysandeep2022-02-03 15:12:13.783755 | primary | }15:28
ysandeep2022-02-03 15:12:13.783806 | primary | ERROR: Ignoring Errors15:28
ysandeepakahat, ^^ i remember you hit something similiar a couple of days back 15:29
dasm|roverysandeep: this should be ignored15:29
dasm|rover"Ignoring Errors"15:29
ysandeepdasm|rover, yeah.. correct, it failed before that 15:30
rlandy|ruckmarios|ruck: dasm|rover: dviroel: ugh - sorry - I killed the fs020 victoria rerun by mistake15:31
rlandy|ruckreadded15:31
dasm|rovernw15:31
* ysandeep updating patch15:33
bhagyashrismarios|ruck, yup15:42
bhagyashrissorry went for dinner 15:43
rlandy|ruckysandeep: the trigger patch incorporates two changes15:45
marios|rucknp bhagyashris it was fine thanks for checking15:45
rlandy|rucklet me know if you want me to separate them15:45
rlandy|ruckand we can merge the trigger specfic part now15:45
ysandeeprlandy|ruck, looking now15:47
dasm|roverrlandy|ruck: created a bug for failing centos8-ovb-network: https://bugs.launchpad.net/tripleo/+bug/195991415:59
* dviroel lunch16:02
*** dviroel is now known as dviroel|lunch16:02
amoralejwrt https://review.opendev.org/c/openstack/tripleo-quickstart/+/827607 we need the +1 from RDO Third party? 16:02
amoralejone of the ovb jobs failed while provisioning a node16:03
amoralejhttps://logserver.rdoproject.org/07/827607/1/openstack-check/tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001/6120926/logs/undercloud/home/zuul/overcloud_node_provision.log.txt.gz16:03
amoraleji guess it's totally unrelated, as that part of the code i'd say it's not used there16:03
amoralejah, i see some reviews were merged with -1 from third party, forget my question, sorry for the noise16:04
rlandy|ruckdasm|rover: ugh - thanks16:05
rlandy|ruckso close16:05
rlandy|ruckamoralej: nah16:05
ysandeeprlandy|ruck, https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-config/+/308842/1#message-151c036bb9c4c2a4a854bb982336ce367606b0c3 , comment - we can merge and try16:05
rlandy|ruckwe can merge that for you16:05
dasm|roveramoralej: actually, this might be even similar to https://bugs.launchpad.net/tripleo/+bug/195991416:05
dasm|rovers/similar/the same (?)16:06
rlandy|ruckysandeep: ok - if we have an issue, I will revert16:06
amoralejdasm|rover, yep, looks similar16:06
ysandeeprlandy|ruck, sure thanks!16:06
ysandeeprlandy|ruck, marios|ruck 16.2 container build failing, logs shared in internal channel16:10
marios|ruckysandeep: looking16:18
marios|ruckysandeep: ack lets continue tomorrow 16:23
ysandeepmarios++ thanks for pointer, i will try that 16:23
rlandy|ruckysandeep: ugh - I messed up the role16:28
rlandy|ruckreverting16:28
ysandeeprlandy|ruck, ack16:28
rlandy|ruckand just adding the playbook for r=tigger16:28
ysandeepaye16:28
rlandy|ruckwaiting for merge then putting back trigger change16:33
ysandeepack16:33
marios|ruckUnable to freeze job graph: Job tripleo-ci-centos-9-scenario001-standalone depends on tripleo-ci-centos-9-content-provider which was not run.16:34
marios|rucklavly16:34
marios|ruckoff in a few rlandy|ruck dasm|rover o/ 16:39
dasm|roverack16:40
rlandy|ruckmarios|ruck: have a good night16:42
* ysandeep out for the day, see you guys tomorrow o/ have a great rest of your shift 16:44
*** ysandeep is now known as ysandeep|out16:44
dasm|rover\o16:44
rlandy|ruckysandeep|out: rechecking trigger jobs16:45
rlandy|ruckhave a good night16:45
marios|ruckyou too rlandy|ruck o/16:45
*** marios|ruck is now known as marios|out16:45
ysandeep|outyou too :D16:45
*** dviroel|lunch is now known as dviroel17:02
*** amoralej is now known as amoralej|off17:16
*** sshnaidm is now known as sshnaidm|afk17:30
rlandy|ruckrcastillo: dviroel: just waiting on the the rhel8 jobs from https://code.engineering.redhat.com/gerrit/c/testproject/+/21019717:57
rlandy|ruckrhel 9 already passed17:57
rlandy|ruckthen we can merge amol's patch17:57
rlandy|ruckdviroel: is there a patch yet to update wallaby c9 promotion config?17:59
rlandy|ruckso images get promoted?17:59
dviroelrlandy|ruck: yes, I think so /me gets18:11
dviroelrlandy|ruck: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/38997 ?18:12
rlandy|ruckdviroel++ thanks 18:26
rlandy|ruckmerged18:26
rlandy|ruckwe need to add your new jobs to criteria there 18:26
dviroelrlandy|ruck: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/3899218:27
rlandy|ruckperiodic-tripleo-build-containers-ubi-9-internal-rhel-9-build-push-upload-rhos-17testprojectmastercheck210197,4518:27
rlandy|ruck57 mins 51 secs2022-02-03 16:42:11SUCCESS18:27
rlandy|ruckperiodic-tripleo-build-containers-ubi-8-internal-rhel-8-build-push-upload-rhos-17testprojectmastercheck210197,4518:27
rlandy|ruck1 hr 28 mins 51 secs2022-02-03 16:41:30SUCCESS18:27
rlandy|ruckperiodic-tripleo-build-containers-ubi-8-internal-rhel-8-build-push-upload-rhos-16.2testprojectmastercheck210197,4518:27
rlandy|ruck1 hr 28 mins 1 sec2022-02-03 16:41:40SUCCESS18:27
rlandy|ruckperiodic-tripleo-rhel-8-upload-job-trigger-rhos-17testprojectmastercheck190672,19918:27
rlandy|ruck3 mins 58 secs2022-02-03 16:48:37SUCCESS18:27
rlandy|ruckperiodic-tripleo-rhel-8-upload-job-trigger-rhos-16.2testprojectmastercheck190672,19918:27
rlandy|ruck3 mins 57 secs2022-02-03 16:47:34SUCCESS18:27
rlandy|ruckrcastillo: ^^ we're a go on downstrem18:27
rlandy|ruckdviroel: rebasing18:28
rlandy|ruckhttps://review.rdoproject.org/r/c/rdo-infra/ci-config/+/38992/1/ci-scripts/dlrnapi_promoter/config_environments/rdo/CentOS-9/wallaby.yaml18:28
rlandy|ruckwill clash, right?18:28
dviroelit shouldn't, different lines18:29
dviroellet me rebase on top that one18:30
rlandy|ruckdviroel: lol18:30
rlandy|ruckI think we both tried18:31
dviroelrlandy|ruck: done18:31
dviroelno conflicts18:31
rlandy|ruckcool18:31
rlandy|ruckwill vote on that18:31
dviroel++18:31
rlandy|ruckdviroel: voted - feel free to w+ when ready18:32
rlandy|ruckdviroel: also - we can go with amol'd patch18:32
rlandy|rucksee above for downstream results18:32
dviroelamols patch need a recheck 18:35
dviroeli will take a look on his code18:35
dviroeloh, he did 1 hour ago18:37
dviroelah, easy review since he abandoned c7 idea too :)18:40
dviroelso, featureset001-network failed again, but in tempest18:58
rlandy|ruckdviroel: re: https://review.opendev.org/c/openstack/tripleo-ci/+/823688/ will vote after zuul results18:58
dviroelack18:58
rlandy|ruckwant to give sandeep a heads up about 18:58
rlandy|ruckwhen to merge the downstream patch18:59
rlandy|rucklooking at network failure18:59
rlandy|ruckdviroel: dasm|rover: on the upside, I think victoria should promote19:00
rlandy|ruckdviroel: dasm|rover: let's meet re: network failures19:01
rlandy|ruckhttps://meet.google.com/hid-kwmq-pec?pli=1&authuser=019:01
rlandy|ruckarxcruz: hey - you still around?19:04
dviroeljoining19:04
dasm|roverrlandy|ruck: https://bugs.launchpad.net/tripleo/+bug/195991419:07
dasm|rover16:03 <amoralej> https://logserver.rdoproject.org/07/827607/1/openstack-check/tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001/6120926/logs/undercloud/home/zuul/overcloud_node_provision.log.txt.gz19:08
dasm|roverrlandy|ruck: ^19:08
rlandy|ruckhttps://bugs.launchpad.net/tripleo/+bug/195993019:25
rlandy|ruckdasm|rover: dviroel: ^^19:25
rlandy|ruckstarting to collect debug data there19:25
rlandy|ruckhttps://opendev.org/openstack/tripleo-heat-templates/commit/66f968be90a0797118ff483f96fca31e90d8304419:28
dviroel^ good candidate19:30
* rlandy|ruck checks who submitted that19:30
dviroeltakashi19:31
dasm|roverit's a day ago. when did we start seeing issues?19:31
rlandy|ruckhard to tell19:31
rlandy|ruckwe have so many issue19:31
dasm|rovermhm19:31
rlandy|ruckwhere is ade_lee when you need him19:31
rlandy|ruckmaybe he can tell us if it look vaguely suited19:32
rlandy|ruckhttps://opendev.org/openstack/tripleo-heat-templates/commit/ad97bd70592a57708313d2b87a61f0be01c7b00419:32
rlandy|ruckbunch of those19:32
rlandy|ruckpinging him on #tripleo19:32
rlandy|ruckwe could try a revert19:32
rlandy|ruckif no response19:32
dasm|roverwe can actually try revert and run testproject now19:33
dasm|roverlemme do that now19:33
rlandy|ruck [Zuul] Zuul ad97bd7059 Merge "Use Internal API by default for Keystone admin endpoint" 1 day ago19:33
rlandy|ruck[Zuul] Zuul 66f968be90 Merge "Nova: Use internal endpoint to access Keystone" 1 day ago19:33
rlandy|ruckdasm|rover: ^^ two19:33
dasm|roverdo we want to try reverting both? or just one of these?19:33
rlandy|ruckdasm|rover: thanks - let us know19:33
rlandy|ruckI would guess they are related19:34
rlandy|ruckwe probably need a tempset update 19:34
dviroeltempest error is: 19:34
dviroel2022-02-02 22:44:12.012 457856 WARNING urllib3.connectionpool [-] Retrying (Retry(total=7, connect=None, read=None, redirect=5, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb4569bdc88>: Failed to establish a new connection: [Errno 101] Network is unreachable',)': /v3/OS-FEDERATION/service_providers/57963b3163eb44748a0d13f1c3fa3d4819:34
dviroelLooks like network went down19:35
rlandy|ruckhey ade_lee19:35
ade_leerlandy|ruck, yo19:35
rlandy|ruckwe're looking at some horribly failing tempest tests ...19:35
rlandy|ruckmaster only 19:35
rlandy|ruckhttps://logserver.rdoproject.org/53/21853/42/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-network-master/43e33cb/logs/undercloud/var/log/tempest/testrepository.subunit.gz19:36
rlandy|ruckexample: ^^19:36
ade_leerlandy|ruck, yeah - I figured - I never get called for anything else ..19:36
dasm|rover:D19:36
rlandy|ruckwe were wondering if either of these two patches look suspicious to you 19:36
rlandy|ruckhttps://logserver.rdoproject.org/53/21853/42/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-network-master/43e33cb/logs/undercloud/var/log/tempest/testrepository.subunit.gz19:36
rlandy|ruckhttps://opendev.org/openstack/tripleo-heat-templates/commit/66f968be90a0797118ff483f96fca31e90d8304419:36
rlandy|ruckade_lee: sorry - we will call you just to say hi some day :)19:37
ade_leerlandy|ruck, you said two patches -- youjust pointed me to one, right?19:39
rlandy|ruckok - let's try this again ,,,19:39
rlandy|ruckhttps://opendev.org/openstack/tripleo-heat-templates/commit/ad97bd70592a57708313d2b87a61f0be01c7b00419:39
rlandy|ruckand19:39
rlandy|ruckhttps://opendev.org/openstack/tripleo-heat-templates/commit/66f968be90a0797118ff483f96fca31e90d8304419:39
rlandy|ruckwe could be totally off here ... just keystone tests almost never fail19:40
rlandy|ruckhttps://opendev.org/openstack/tripleo-heat-templates/commits/branch/master19:41
rlandy|ruckscanning that list19:41
ade_leerlandy|ruck, when did things start to fail?19:41
rlandy|ruckwell, that's a complicated question to answer19:42
rlandy|ruckwe just got through a bug on node provision19:42
rlandy|ruckso we see this now19:42
rlandy|ruckbut it could be a couple days19:42
rlandy|ruckand it's in integration and component line together19:42
rlandy|ruckso it feels like a patch merge that gets taken from current19:43
rlandy|rucklist here:19:43
rlandy|ruckhttps://github.com/openstack/tripleo-quickstart/blob/master/config/release/tripleo-ci/CentOS-8/promotion-testing-hash-master.yml#L10019:43
rlandy|ruck2022-02-03 11:38:27.586 ERROR /var/log/containers/neutron/server.log: 15 ERROR neutron.plugins.ml2.managers [req-4c604be7-0f9d-4586-9f71-6bd354614ef8 - fake_project_id - - -] Failed to bind port 849578a9-66f9-452e-a19d-65bd63959ac1 on host 71d6c58b-ef26-4ad9-a0db-987ee791d4e8 for vnic_type normal using segments [{'id': '1b87e441-216e-4358-9f3f-490fed0000d0', 'network_type': 'flat', 'physical_network': 'ctlplane', 19:44
rlandy|ruck'segmentation_id': None, 'network_id': '6c917854-3cdc-4398-9842-6510245c7b86'}]19:44
rlandy|ruckcould be that19:44
ade_leerlandy|ruck, that commit looks like the most likely culprit here - curious how it got through ci to begin with19:44
ade_leeyup19:45
rlandy|ruckade_lee: sorry - which one?19:45
ade_leerlandy|ruck, so is this failing on master upstream now?19:45
rlandy|ruckit got through ci because we only see it in OVB19:45
rlandy|ruckyep19:46
rlandy|ruckc8 and c919:46
rlandy|ruckhttps://bugs.launchpad.net/tripleo/+bug/195993019:46
ade_leerlandy|ruck, so the fix that takashi put in doesn't look fishy -- but maybe its uncovered some other issue with reaching the internal api on ovb systems?19:50
rlandy|ruckor it's something else completely19:50
ade_leerlandy|ruck, that neutron error is worth looking into19:51
rlandy|ruckok19:51
rlandy|ruckon it19:51
ade_leerlandy|ruck, but if its true that it works on some systems, but not ovb ones, maybe points to network config or something to do with ovb19:52
rlandy|ruckade_lee: ok - we will look more  - thanks for checking in 19:53
dasm|roverthis are actual takashi's fixes: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/826945 and https://review.opendev.org/c/openstack/tripleo-heat-templates/+/82694519:53
ade_leerlandy|ruck, no prob -- I'll check in again tomorrow in case you need me19:53
rlandy|ruckthank you!!19:53
dasm|roveri meant this: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/82679319:55
ade_leerlandy|ruck, dviroel , dasm|rover - just fyi  - I tried using the test python to get around the python+fips error in centos 9 and ran into other issues.19:56
ade_leehttps://review.opendev.org/c/openstack/tripleo-ci/+/82447919:56
dasm|roverchecking19:56
ade_leethe job got further, but I'll probably need to also update python within the containers ..19:56
ade_leeand first we'll need to fix a mysql issue 19:57
rlandy|ruckk - we can check in on those reults19:57
rlandy|ruckresults19:57
dasm|roverthat's something new to me: Problem 1: conflicting requests\n  - nothing provides python3-pip-wheel >= 21.2.3-4 needed by python3-libs-3.9.10-1.el9.x86_64\19:57
ade_leewhere we get a mysql crash under fips when setting up the db19:58
ade_leedciabrin has reproduced this outside of ci -- so we'll need a fix for that from rhel ..19:58
ade_leedasm|rover, yeah some things will fail coz we're just pulling in some python and not others -- I basically only looked at the scenario tests for centos920:00
dasm|roverack20:00
dasm|roverrlandy|ruck: fyi: i reverted takashi's changes and started testproject to see if we're seeing ovb issues: https://review.rdoproject.org/r/c/testproject/+/3901820:01
dasm|roverrlandy|ruck: lmk if that looks right20:01
dviroelrlandy|ruck: ack, hope that works :)20:02
rlandy|ruckdviroel: we're guessing here :)20:02
rlandy|ruckbut sure - pls just abandon if they don't fix it20:02
dviroelade_lee: any idea when these packages will be available on centos mirrors?20:02
rlandy|ruckstill following up neutron errors20:03
* rlandy|ruck checking last passing test20:04
rlandy|ruck01/3120:05
rlandy|ruckhttps://opendev.org/openstack/neutron/commit/e7b70521d0e230143a80974e7e4795a2acafcc9b?20:12
dviroelPassing job: PUT http://192.168.24.12:35357/v3/OS-FEDERATION/service_providers/c4774d106d384c7a8e19b73f435f0b3920:41
dviroelctrlplane ip20:41
dviroelwhich doesn't exist in the tempest log of the failing job20:41
*** dviroel is now known as dviroel|afk20:50
rlandy|ruckthat's a good start20:55
rlandy|ruckdviroel|afk: akahat: left notes on https://bugs.launchpad.net/tripleo/+bug/195993021:33
rlandy|ruckarxcruz: ^^ can you help out here??21:33
rlandy|ruckhttps://opendev.org/openinfra/python-tempestconf change21:36
dasm|roverhttps://opendev.org/openstack/openstack-ansible-os_tempest/src/branch/master/templates/tempest.conf.j2#L6521:39
rlandy|ruckdasm|rover: k - let's check what the testproject with harold's change gives us21:46
dasm|roverthe link is here: https://review.rdoproject.org/r/c/testproject/+/3902221:46
dasm|roveri started it just for periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-network-master21:47
dasm|roverrlandy|ruck: you mentioned we should test it more thorough. are you thinking about some particular jobs?21:47
rlandy|ruckdasm|rover: so you'll want to run the fs001 test from wallaby, train, ussuri, victoria21:49
rlandy|ruckas well as master21:49
rlandy|ruckin your testproject21:49
rlandy|ruckmake sense?21:49
rlandy|ruckdasm|rover: you can start that job21:49
dasm|roverk, i'll update the zuul def21:49
rlandy|ruckand then leave notes for akahat and dviroel|afk in tripleo-ci chat21:49
rlandy|ruckdasm|rover++ thanks21:55
dasm|roverrlandy|ruck: https://review.rdoproject.org/r/c/testproject/+/39022 does it look good?21:57
rlandy|ruckdasm|rover: looks good - wallaby is missing but if it works on master and victoria, we're fine21:58
rlandy|ruckthanks!!21:58
dasm|roveradded wallaby.22:00
* dasm|rover leaves22:02
dasm|roverhave a good night y'all!22:02
*** dasm|rover is now known as dasm|off22:02
rlandy|ruckhave a good night22:07
rcastillocalling it a day as well. See you tomorrow!22:09
rlandy|ruckrcastillo: see you22:24
arxcruzrlandy|ruck sorry, didn't saw your messages, it's late here, but checking the tempest error, it seems the undercloud where tempest run are not being able to connect to overcloud (and do the rest api calls)23:28
rlandy|ruckarxcruz: no wirris23:42
rlandy|ruckworries - we have a patch23:42
*** rlandy|ruck is now known as rlandy|out23:44
« Wednesday, 2022-02-02 Index Friday, 2022-02-04 »

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!