Monday, 2020-06-29

*** ysandeep|away is now known as ysandeep04:00
*** ykarel|away is now known as ykarel04:17
*** udesale has joined #oooq04:28
*** jtomasek has joined #oooq05:00
*** bhagyashris is now known as bhagyashris|pto05:16
*** ratailor has joined #oooq05:17
*** jfrancoa has joined #oooq05:52
*** dpawlik4 has joined #oooq06:20
*** amoralej|off is now known as amoralej06:37
*** matbu has joined #oooq06:42
*** ysandeep is now known as ysandeep|brb06:45
*** ccamacho has joined #oooq06:47
*** ccamacho has quit IRC07:02
*** jbadiapa has joined #oooq07:05
*** ysandeep|brb is now known as ysandeep07:06
*** sshnaidm is now known as sshnaidm|ruck07:24
*** dtantsur|afk is now known as dtantsur07:28
sshnaidm|ruckweshay_, fyi https://review.rdoproject.org/r/#/c/28277/07:28
*** ccamacho has joined #oooq07:29
ykarelsshnaidm|ruck, does that work?07:43
ykarel^^07:44
ykareli think issue is different, let me fetch link07:44
ykarelis search should have also worked07:44
sshnaidm|ruckykarel, "release in" always worked, idk what is the reason behind "search"07:47
sshnaidm|ruckbut it failed in ceph jobs07:48
ykarelsshnaidm|ruck, i think it's caused with https://review.rdoproject.org/r/#/c/28003/07:53
ykarelnot directly, but missing release: var in job07:53
ykareland usage of release_name:07:54
ykareland my comment in https://review.rdoproject.org/r/#/c/28234/ should fix it07:54
ykarelsimilar for ceph-ansible, ^ for podman jobs07:55
*** jpena|off is now known as jpena07:56
chandankumarsshnaidm|ruck, https://review.rdoproject.org/r/#/c/28277/ can we remove release condition and keep only distro version condition07:57
chandankumar?07:57
ykarelsshnaidm|ruck, small reproducer ansible localhost -c local -m debug -a "msg={{ release is search("queens|rocky|stein|train") }}"07:58
ykarelfor root cause of issue07:58
ykarelrelease: var is undefined07:58
sshnaidm|ruckchandankumar, I'm not sure it's a good solution here07:59
sshnaidm|ruckykarel, so "release" is undefined there?07:59
ykarelsshnaidm|ruck, yes07:59
chandankumarsshnaidm|ruck, train is a special case so that distro version works like that07:59
sshnaidm|ruckhow the "search" is supposed to help then?07:59
ykarelsearch was working earlier too and should work now as well, issue is release: not defined08:00
sshnaidm|ruckchandankumar, component are branch depending, not OS, so better not to mix dependencies here08:00
sshnaidm|ruckykarel, ack, looking..08:01
chandankumarsshnaidm|ruck, with in each job definition, we pass release and branch_override var so i donot think it will cause issue and in get hash role, we have mapping defined https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/infra-setup/roles/get_hash/defaults/main.yml#L408:04
ysandeepsshnaidm|ruck, fyi.. recently i proposed https://review.opendev.org/#/c/737923/ to move from Release to OS08:07
ysandeepchandankumar, need some confirmation for periodic c8 jobs - https://hackmd.io/oiRjzroqTE6xLsLR0vfpKw?view08:08
chandankumarysandeep, will take a look in sometime08:14
ysandeepchandankumar, thank you08:14
*** ysandeep is now known as ysandeep|lunch08:17
*** ratailor has quit IRC08:26
*** ratailor has joined #oooq08:26
sshnaidm|ruckchandankumar, https://review.rdoproject.org/zuul/build/9ceed77761e64013934e2aae75cb6915/log/job-output.txt#85508:33
sshnaidm|ruckchandankumar, there is a problem ^08:33
soniya29|roversshnaidm|ruck, Can you have a look at this card, https://trello.com/c/mJvj47I8/1528-cixlp1881592tripleociproa-ussuri-tempestapicomputeimagestestimagesoneservernegativeimagesoneservernegativetestjson?08:34
sshnaidm|rucksoniya29|rover, sure08:36
sshnaidm|rucksoniya29|rover, afaiu the test is passing now?08:37
soniya29|roversshnaidm|ruck, we are clean on periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-ussuri08:37
sshnaidm|rucksoniya29|rover, ?08:37
soniya29|roversshnaidm|ruck, the test is passing consectively08:37
sshnaidm|rucksoniya29|rover, how is 001 related?08:38
soniya29|roversshanidm|ruck, actually bug was filed with 001 logs08:38
chandankumarsshnaidm|ruck, https://review.rdoproject.org/r/#/c/28277/1/ci-scripts/infra-setup/roles/get_hash/tasks/get_hash.yaml@22 it is break for c7 train, that is the only concern i have08:39
soniya29|roversshnaidm|ruck, full-tempest-api-ussuri is also passing08:39
chandankumarmay be we can add one more condition - ansible_distribution_major_version is version("8", ">=")08:39
sshnaidm|rucksoniya29|rover, so problem is solved08:39
soniya29|roversshnaidm|ruck, I think so08:40
sshnaidm|ruckchandankumar, why is it breaking c7?08:40
sshnaidm|ruckchandankumar, I just change "search" by "in []"08:41
sshnaidm|rucksoniya29|rover, moved it to "done"08:42
chandankumarsshnaidm|ruck, ah got your point08:44
*** derekh has joined #oooq08:44
arxcruzsshnaidm|ruck: chandankumar https://review.opendev.org/#/c/738022/ when you guys have some time08:48
*** ysandeep|lunch is now known as ysandeep08:50
soniya29|roversshnaidm|ruck, ack08:50
*** apetrich has joined #oooq08:59
chandankumarysandeep, commented on hackmd09:04
sshnaidm|ruckchandankumar, ykarel any ideas about https://bugs.launchpad.net/tripleo/+bug/1885315 ?09:05
openstackLaunchpad bug 1885315 in tripleo "tripleo-buildimage-overcloud-full-centos-8 is failing on the update of libnghttp2 package" [Critical,Triaged]09:05
chandankumarsshnaidm|ruck, not yet looked09:07
ysandeepchandankumar, thanks! I think minion should be present in train, our osp16 docs(which is based on train) have minion reference - https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.0/html-single/director_installation_and_usage/index#installing-undercloud-minions09:07
chandankumarysandeep, ok, then we can start adding job then09:08
chandankumarbut confrim with Alex once he comes back online09:08
arxcruzkopecmartin: chandankumar https://review.opendev.org/#/c/734775/09:10
arxcruzsoniya29|rover: ^09:10
ysandeepchandankumar, ack thank you, I will check if jobs fail but I am confident as its present in OSP16 release docs - https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.0/html-single/release_notes/index#technology-previews09:11
ykarelsshnaidm|ruck, it's likely failing due to epel repo09:16
ykarelwe hit it in packstack too, and removing epel repos cleared for us09:17
ykarelsshnaidm|ruck, https://review.opendev.org/#/c/738376/3/playbooks/packstack-centos8-pre.yaml09:17
ykarelu can do similar in affecting jobs09:18
ykarelremove usage of epel repos wherever used09:19
ykarelit's prone to errors mixing c8 repos09:19
arxcruzgithub is down aparently09:20
arxcruzsoniya29|rover: sshnaidm|ruck ^09:20
arxcruzin case09:20
sshnaidm|ruckarxcruz, ack09:20
sshnaidm|ruckdamn microsoft..09:20
arxcruzlol09:21
ykarelsshnaidm|ruck, i commented, u can try that out09:26
sshnaidm|ruckykarel, ack, trying09:26
ykarelack09:27
chandankumarsshnaidm|ruck, arxcruz weshay_ I have internet outage in my area, I am not sure how much will i survive with phone internet, might skip all meetings09:33
arxcruzchandankumar: weshay_ sshnaidm|ruck I also have doctor appointment today, i might skip the scrum meeting09:34
*** ratailor_ has joined #oooq10:09
*** ratailor has quit IRC10:12
zbrsshnaidm|ruck: what is the status with vexx host? the URL from our secrets does not load.10:12
*** ratailor_ has quit IRC10:13
*** ratailor has joined #oooq10:13
jfrancoasshnaidm|ruck: hey, one question about the tripleo-buildimage-overcloud-full-centos-8. Is there any open issue wiht them? they are consistently failing in few of the patches10:41
*** tosky has joined #oooq10:57
pojadhavysandeep, pls review : Add Centos8 multinode ipa train job  https://review.rdoproject.org/r/2828410:58
*** pojadhav is now known as pojadhav|brb10:58
ysandeeppojadhav|brb, Can you please run a testrun against it , temporarily you can add something like - https://review.rdoproject.org/r/#/c/28282/2/zuul.d/projects.yaml11:00
*** udesale has quit IRC11:02
*** udesale has joined #oooq11:02
*** ysandeep is now known as ysandeep|brb11:02
*** skramaja has joined #oooq11:06
pojadhav|brbysandeep|brb, ack11:18
*** pojadhav|brb is now known as pojadhav11:18
sshnaidm|ruckjfrancoa, WIP https://review.opendev.org/#/c/738406/11:23
sshnaidm|ruckzbr, not sure I understand the issue11:23
zbrthere are some credentials to login to vexx host in our creds doc, even the url does not work.11:24
weshay_zbr, sshnaidm|ruck 0/  need to chat about upstream infra and retries in a few minutes11:29
* weshay_ grabs coffee etc11:29
*** jpena is now known as jpena|lunch11:32
sshnaidm|ruckzbr, need details11:35
jfrancoasshnaidm|ruck: thanks!11:35
weshay_sshnaidm|ruck, zbr ping me when you have a moment fellas11:42
*** ysandeep|brb is now known as ysandeep11:43
ysandeepHey! weshay_ o/ , Hope you had great PTOs :) Could you please add https://review.opendev.org/#/c/737923/ and https://review.opendev.org/#/c/737975/ to your review list.11:46
weshay_aye11:46
weshay_sshnaidm|ruck, zbr http://dashboard-ci.tripleo.org/d/3sUHfh9Wz/jobs-exploration?orgId=1&from=now-30d&to=now&var-influxdb_filter=result%7C%3D%7CRETRY_LIMIT&var-influxdb_filter=cloud%7C%3D%7Cnull11:46
zbrweshay_: ping11:47
weshay_zbr, :) https://meet.google.com/xzh-pwzm-roa sshnaidm|ruck11:48
weshay_soniya29|rover, fyi ^11:48
*** rfolco has joined #oooq11:56
sshnaidm|ruckweshay_, https://review.rdoproject.org/zuul/builds?result=RETRY_LIMIT11:59
*** rlandy has joined #oooq11:59
*** amoralej is now known as amoralej|lunch12:00
rlandysshnaidm|ruck: soniya29|rover: hi - I'm in training today12:01
soniya29|roverrlandy, sure12:01
rlandysshnaidm|ruck: soniya29|rover: https://ci.centos.org/job/weirdo-master-promote-packstack-scenario003/ - fails consistently12:02
rlandy^^ real issue12:02
*** ratailor has quit IRC12:02
*** ratailor has joined #oooq12:02
sshnaidm|ruckrlandy, ack12:02
rlandyreran a few times12:02
weshay_ykarel, https://meet.google.com/xzh-pwzm-roa can you join for a min re: retries12:03
sshnaidm|ruckrlandy, weirdo better to delegate to #rdo12:03
rlandyack  - will ping there12:03
ykarelweshay_, ack12:03
zbrykarel: can you please joing xzh-pwzm-roa -- re centos-8.2 issue?12:04
*** ratailor has quit IRC12:08
rlandysshnaidm|ruck: https://bugs.launchpad.net/tripleo/+bug/188531412:11
openstackLaunchpad bug 1885314 in tripleo "OVB master job running on vexxhost show some nodes failing introspection step" [Critical,In progress] - Assigned to Ronelle Landy (rlandy)12:11
rlandy^^ you're right about vexxhost12:11
rlandykeeps failing one or two nodes12:11
rlandytried to test out your patch = failed out12:12
*** udesale_ has joined #oooq12:20
weshay_zbr, go eat .. we'll catch up thanks :)12:20
*** udesale has quit IRC12:22
*** derekh has quit IRC12:25
*** weshay_ is now known as weshay_ruck12:26
*** rlandy is now known as rlandy|training12:29
sshnaidm|ruckykarel, this one? https://zuul-ci.org/docs/zuul/reference/job_def.html#attr-job.attempts12:34
ykarelsshnaidm|ruck, yes12:34
*** derekh has joined #oooq12:46
*** Goneri has joined #oooq12:48
chandankumarrlandy|training, hello12:48
rlandy|trainingchandankumar: hey12:49
chandankumarrlandy|training, https://opendev.org/openstack/tripleo-ci/src/branch/master/roles/build-containers/vars/main.yaml there are multiple exclude list12:49
chandankumarexclude_containers used with kolla and excl_containers for redhat12:49
chandankumarwith new container build which one to use?12:50
rlandy|trainingchandankumar: that's  a hack I used to differenciate between new and old methos w/o12:50
rlandy|trainingmethods12:50
rlandy|trainingexcl_containers is the new method12:50
rlandy|trainingexclude_containers the old one12:50
rlandy|trainingI experimented with some jinja if else conditions12:51
rlandy|trainingbut they were messy12:51
chandankumarrlandy|training, oh, I modified my patch to use old one12:51
chandankumarpanda, ^^12:51
rlandy|trainingchandankumar: basically there is with kolla and without kolla12:51
rlandy|trainingchandankumar: you can find another solution12:52
rlandy|trainingor I can change my work12:52
chandankumarrlandy|training, let me get the container build working then we look for a better solution12:52
pandaok, I'll create a generic solution anyway, we can change the url at will after it's merged.12:52
chandankumarrlandy|training, thanks panda thanks :-)12:53
rlandy|trainingyeah - just a hack so I didn't break running12:53
rfolcoping scrum arxcruz, rfolco, zbr, panda, sshnaidm, rlandy, marios, ysandeep, bhagyashris, soniya29, pojadhav, akahat, weshay, chandankumar13:01
rfolcohttps://meet.google.com/oiv-geho-mai?authuser=113:01
weshay_ruckakahat, pojadhav scrum13:02
pojadhavjoining13:02
*** amoralej|lunch is now known as amoralej13:05
*** TrevorV has joined #oooq13:30
*** pojadhav is now known as pojadhav|afk13:32
weshay_ruckysandeep, add https://review.opendev.org/#/c/738434/ as depends-on for https://review.opendev.org/#/c/73837513:43
ysandeepweshay_ruck, thanks! I will once c8 train containers are available13:44
ysandeepwe can't merge that change till c8 train contaienrs are available13:45
*** holser_ has quit IRC13:48
*** ysandeep is now known as ysandeep|afk13:50
*** holser has joined #oooq13:52
rfolcozbr, https://meet.google.com/ssb-free-yne14:03
*** jpena|lunch is now known as jpena14:29
pandarfolco: zbr https://review.rdoproject.org/r/2602214:41
*** ykarel is now known as ykarel|away14:55
*** ysandeep|afk is now known as ysandeep|away14:57
sshnaidm|ruckykarel|away, weshay_ruck we hit again in build images "failed to glob pattern /etc/rc0.d/[SK][0-9][0-9]network: No such file or directory"15:08
*** skramaja has quit IRC15:12
sshnaidm|ruckhttps://bugs.launchpad.net/tripleo/+bug/185127415:12
openstackLaunchpad bug 1851274 in tripleo "periodic-tripleo-rhel-8-buildimage-overcloud-full-master is failing - failed to glob pattern /etc/rc0.d/[SK][0-9][0-9]network: No such file or directory" [Critical,Fix released] - Assigned to Alex Schultz (alex-schultz)15:12
ykarel|awaysshnaidm|ruck, how was it fixed that time, i have a feeling it's related to the directory cleanup issue15:15
sshnaidm|ruckykarel|away, see on #tripleo, wasn't fixed explicitly15:15
ykarel|awaysshnaidm|ruck, ack looking15:16
*** pojadhav|afk is now known as pojadhav15:29
*** dsneddon has joined #oooq15:40
zbrweshay_ruck: fyi, very soon zuul (next restart) will report retries run, and we will be able to query on them too.15:47
*** dsneddon has quit IRC15:50
weshay_ruckzbr, fuck ya!15:50
weshay_ruckzbr++15:50
weshay_rucksshnaidm|ruck, cool thank you15:51
weshay_rucksshnaidm|ruck, wait.. are you putting up a sova pattern or are you referencing that epel may not have fixed it15:51
sshnaidm|ruckzbr++15:52
sshnaidm|ruckweshay_ruck, wdym?15:52
weshay_ruck[09:08:42] <sshnaidm|ruck> ykarel|away, weshay_ruck we hit again in build images "failed to glob pattern /etc/rc0.d/[SK][0-9][0-9]network: No such file or directory"15:52
weshay_rucksshnaidm|ruck, any luck w/ held nodes?15:53
sshnaidm|ruckweshay_ruck, yeah, it's a ghost issue from the past, going to submit a patch now..15:53
weshay_ruckk15:53
sshnaidm|ruckweshay_ruck, nope (about helding)15:54
sshnaidm|ruckweshay_ruck, although epel patch worked, that's how we reached this point..15:55
*** dsneddon has joined #oooq15:57
*** amoralej is now known as amoralej|off15:58
sshnaidm|ruckweshay_ruck, rlandy|training you're off in Fri, right?16:02
weshay_ruckaye16:05
sshnaidm|ruckneed somebody to cover then16:06
*** Goneri has quit IRC16:20
weshay_rucksshnaidm|ruck, aye.. I'll figure it out16:21
*** udesale_ has quit IRC16:23
rlandy|trainingsshnaidm|ruck: weshay_ruck: yeah - officially off on Friday - but I can be on for some hours16:25
rlandy|traininghave an appointement around 11 my time - otherwise could be on line16:26
*** Goneri has joined #oooq16:33
weshay_ruckrlandy|training, you covered enough last week :)16:42
weshay_ruckzbr, any clues to when a zuul restart will happen?16:42
zbrno, usually is less than a week16:43
weshay_rucklolz16:43
weshay_ruck k16:43
zbrusually is "when things go shitty for another reason" afaik16:43
weshay_ruckok.. then we have rdo sf and the reproducer16:43
zbrbut we will not have the same feature in rdo, rdo upgrades much slower.16:44
zbrit was fix merged yesterday.16:44
weshay_ruckzbr, where's the fix16:50
weshay_ruckmaybe we can get them to do it16:50
zbrlooking now, i did not check it myself16:51
zbrfirst we need to see landed upstream16:52
zbri think this is https://review.opendev.org/#/c/633501/16:53
zbris fun that some long waiting features go in only when shit hit the fan16:54
weshay_ruckhttps://review.opendev.org/#/q/topic:bloeffler-report-reties+(status:open+OR+status:merged)16:54
zbrwe need to wait, cannot push on stuff that was not yet tested upstream on zuul16:55
zbrthey may discover issues. believe me I already asked rdo to do an upgrade several times for other reasons16:56
zbronce we get this deployed, we can a very good reason to escalate the need to upgrade16:56
zbrbut i want to see the new field working first16:57
weshay_ruckk16:59
*** derekh has quit IRC17:03
*** jpena is now known as jpena|off17:09
chandankumarweshay_ruck, rlandy|training https://review.opendev.org/#/c/738049/ needed for c8 train17:17
chandankumartested here https://review.rdoproject.org/r/#/c/28204/17:17
*** ccamacho has quit IRC17:26
*** ccamacho has joined #oooq17:34
*** dtantsur is now known as dtantsur|afk17:51
sshnaidm|ruckchandankumar, weshay_ruck fyi https://bugs.launchpad.net/tripleo/+bug/188561718:05
openstackLaunchpad bug 1885617 in tripleo "CentOS 8 train jobs fail with "tripleo-podman not found" error" [High,Triaged] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm)18:05
*** EmilienM has joined #oooq18:18
EmilienMhi18:18
*** Goneri has quit IRC18:43
*** Goneri has joined #oooq18:55
*** sshnaidm|ruck is now known as sshnaidm|afk19:12
sshnaidm|afk2020-06-29 17:21:04.269660 | TASK [oooci-build-images : Download TripleO source image]19:25
sshnaidm|afk2020-06-29 18:03:04.967052 | primary | changed: OK (854851584 bytes)19:25
sshnaidm|afk40 mins of download19:25
*** jtomasek has quit IRC19:57
*** Goneri has quit IRC20:06
*** jfrancoa has quit IRC20:25
*** jmasud has quit IRC20:33
*** sshnaidm|afk has quit IRC20:51
*** sshnaidm|afk has joined #oooq20:53
*** rlandy|training is now known as rlandy21:30
*** TrevorV has quit IRC21:33
*** Goneri has joined #oooq21:37
*** TrevorV has joined #oooq21:37
*** jmasud has joined #oooq21:45
*** whoami-rajat has quit IRC22:04
*** whoami-rajat has joined #oooq22:07
*** Goneri has quit IRC22:22
*** TrevorV has quit IRC22:31
*** tosky has quit IRC22:46
*** Goneri has joined #oooq22:47
*** rlandy has quit IRC23:00
*** jmasud has quit IRC23:26
*** jmasud has joined #oooq23:48

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!