Thursday, 2021-11-18

« Wednesday, 2021-11-17 Index Friday, 2021-11-19 »
*** rlandy|ruck|bbl is now known as rlandy|ruck02:40
*** rlandy|ruck is now known as rlandy|out02:44
*** ysandeep|PTO is now known as ysandeep03:54
*** pojadhav|afk is now known as pojadhav04:01
*** rcastillo is now known as rcastillo|away04:24
*** ysandeep is now known as ysandeep|afk05:13
*** ysandeep|afk is now known as ysandeep06:19
*** soniya29|ruck is now known as soniya29|ruck|lunch07:13
*** ysandeep is now known as ysandeep|lunch07:41
*** pojadhav- is now known as pojadhav08:08
*** soniya29|ruck|lunch is now known as soniya29|ruck08:11
*** amoralej|off is now known as amoralej08:17
*** jpena|off is now known as jpena08:36
*** ysandeep|lunch is now known as ysandeep08:38
pojadhavsshnaidm, 0/ I am adding c9 ipa job https://review.rdoproject.org/r/c/rdo-jobs/+/36718 and testing here https://review.rdoproject.org/r/c/testproject/+/36719 but facing ansible related issue https://logserver.rdoproject.org/19/36719/2/check/periodic-tripleo-ci-centos-9-standalone-on-multinode-ipa-master/6f333d5/job-output.txt could you please check when you free..?08:49
*** ykarel is now known as ykarel|lunch09:07
sshnaidmpojadhav, looking09:23
jbadiapagm,09:35
jbadiapais there a problem with the repos? 09:35
jbadiapahttps://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-8-undercloud-containers+ 09:37
ysandeepjbadiapa, looks like issue with mirrors, CI team ruck-rovers have reached out to infra folks09:38
jbadiapaysandeep, I've just seen the comments on the tripleo channel, thx09:38
ysandeepWe have less infra folks in this timezone, we may need to wait for a bit 09:39
*** ysandeep is now known as ysandeep|afk10:09
* akahat|rover need to step out.. back in few hours.10:48
*** ykarel|lunch is now known as ykarel11:03
*** dviroel|afk is now known as dviroel11:03
*** ysandeep|afk is now known as ysandeep11:06
*** rlandy|out is now known as rlandy|ruck11:11
rlandy|ruckysandeep: hi - reading back - mirror issue?11:11
rlandy|ruckoh gosh - what happened on the gate?11:13
rlandy|ruckakahat|rover; soniya29|ruck: ^^?11:13
rlandy|ruckok - I see on #opendev11:16
arxcruzrlandy|ruck: akahat|rover just talked with kopecmartin we can go ahead and merge the scen10 voting patch, we are blocked already in other issues so this won't be a problem 11:17
rlandy|ruckarxcruz: thanks ... getting patch11:18
sshnaidmpojadhav, seems like it requires more work, I'll handle it11:21
rlandy|ruckysandeep: when you have a moment - let's sync about downstream11:22
sshnaidmrlandy|ruck, some overlooked bits for 2.11 in CI, it appeared in c9 quickstart jobs, solving it11:22
ysandeeprlandy|ruck, I am available now11:22
rlandy|rucksshnaidm: k- thanks11:22
rlandy|ruckysandeep: https://meet.google.com/mrb-qprj-mrk?pli=1&authuser=011:24
soniya29|ruckrlandy|ruck, mirrors weren't reachable ..i discussed this issue with infra folks, they said we need to check weather issue still persists on next update11:28
rlandy|rucksoniya29|ruck: thanks - talking with ysandeep11:28
soniya29|ruckrlandy|ruck, akahat|rover will be back in an hour11:29
pojadhavsshnaidm, ack11:35
*** pojadhav is now known as pojadhav|brb11:35
rlandy|ruckysandeep: https://bugs.launchpad.net/tripleo/+bug/195081511:48
rlandy|ruckysandeep: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/81798111:50
rlandy|ruckysandeep: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/81798211:50
*** amoralej is now known as amoralej|lunch12:01
rlandy|ruckysandeep: https://code.engineering.redhat.com/gerrit/c/openstack-tripleo-common/+/291903/1/container-images/tcib/base/base.yaml12:02
ysandeeprlandy|ruck, https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-17/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-standalone-full-tempest-api-rhos-17/8622b89/logs/undercloud/var/log/tempest/stestr_results.html12:05
rlandy|ruckysandeep: https://code.engineering.redhat.com/gerrit/c/testproject/+/19067212:09
rlandy|ruckhttps://code.engineering.redhat.com/gerrit/c/testproject/+/18944012:09
rlandy|ruckysandeep: ^^12:09
rlandy|ruckakahat|rover: soniya29|ruck: let's ruck rover sync when you are both in12:13
soniya29|ruckrlandy|ruck, i am here12:13
rlandy|rucksoniya29|ruck: ack -  waiting for akahat|rover 12:13
soniya29|ruckrlandy|ruck, sure12:13
rlandy|rucklooks like our gate is blocked as well12:20
rlandy|rucksoniya29|ruck: ysandeep: did you report the blocked gate as well?12:20
rlandy|ruck818161,212:21
rlandy|ruckunknown12:21
rlandy|ruck13 hr 44 min12:21
ysandeeprlandy|ruck, I am not aware of that.. 12:22
* ysandeep checks12:22
rlandy|ruckmaybe it's just taking time12:22
rlandy|ruck13hr12:22
soniya29|ruckrlandy|ruck, nope12:22
rlandy|ruckwaiting for infra guys anyways12:22
ysandeeprlandy|ruck, the top most job in gate is progressing: https://zuul.openstack.org/stream/5d4e523f53c640bd840efaef5760fc14?logfile=console.log , Looks good to me12:23
*** jpena is now known as jpena|lunch12:23
rlandy|ruckysandeep: pls vote on https://review.opendev.org/c/openstack/tripleo-ci/+/80261612:24
rlandy|ruckthen we can w+ once the gate returns12:25
rlandy|ruckand check12:25
ysandeeprlandy|ruck: checking12:25
*** pojadhav|brb is now known as pojadhav12:31
ysandeeprlandy|ruck, making sc010 voting looks okay but I didn't get why we are adding them in tripleo-standalone-scenarios-pipeline template, Looks like ton of sc010 jobs for different branches will trigger on tripleo-ci repo now12:32
ysandeepthere are in total 10 jobs in check and 10 jobs in gate just for sc010 if we got a match in files: &scen10_files12:34
* rlandy|ruck looks12:35
* ysandeep commenting12:35
rlandy|ruckysandeep: the files are pretty limited12:37
rlandy|ruckthose jobs are branched12:38
rlandy|ruckysandeep: so only scenario010 and ovn provider were kicked on tripleo-ci12:39
ysandeeprlandy|ruck, yeah but still looks like lot of jobs ... 20 in one patch just for sc010 - if we got a hit12:41
* rlandy|ruck will read comment12:41
rlandy|ruckonly if the branch matched12:42
rlandy|ruckthere is no overried12:43
rlandy|ruckoverride12:43
rlandy|ruckjust a branch12:43
ysandeeprlandy|ruck, what about when they run against brancless project? like tq/tqe12:43
rlandy|ruckwe can test it12:44
rlandy|ruckwill test config/general_config/featureset062.yml12:45
ysandeepack12:48
ysandeeprlandy|ruck, thanks for test, +2 12:51
rlandy|ruckysandeep: sure - always good to test :)12:51
* ysandeep was wrong, I was under impression for branchless project all the jobs will trigger, but looks like only master jobs triggered here12:52
* rlandy|ruck w+s12:54
rlandy|rucklet's hope we don't break the world12:54
rlandy|ruckgthiemonge: hey ... https://review.opendev.org/c/openstack/tripleo-ci/+/80261612:55
rlandy|ruck^^ w+'ed12:55
rlandy|ruckscenario010 going voting12:55
rlandy|ruckakahat|rover++ for his work here12:56
ysandeepakahat|rover++ thanks! We now have the ability to individually mark sc010 non-voting for different branches12:56
* akahat|rover back12:57
akahat|roveroh.. sc 10 got merged. great. :)12:58
gthiemongewow!12:58
gthiemongerlandy|ruck: akahat|rover: thanks!!12:58
akahat|rover:)12:58
rlandy|ruckakahat|rover: not yet - gate is hosed due to mirror issues12:58
rlandy|ruckbut it will when gate clears12:58
akahat|roveroh.. that is still not resolved?12:59
rlandy|ruckakahat|rover: w+ed https://review.opendev.org/c/openstack/tripleo-quickstart/+/818250 as well12:59
rlandy|ruckscrum time13:00
akahat|roverwell we are clear with: https://review.opendev.org/c/openinfra/python-tempestconf/+/81437313:00
rlandy|ruckakahat|rover: yep - arxcruz agreed13:00
rlandy|ruckakahat|rover: soniya29|ruck: scrum13:01
rlandy|ruckreview:13:06
arxcruzhttps://review.rdoproject.org/r/c/rdo-infra/ci-config/+/3676013:07
arxcruzrlandy|ruck: ^13:07
rlandy|ruckdashboard13:07
rlandy|ruckhttps://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/815451 - build packages13:11
arxcruzsshnaidm: https://review.opendev.org/c/openstack/tripleo-ci/+/81699113:12
*** amoralej|lunch is now known as amoralej13:14
dviroelhttps://review.rdoproject.org/r/c/rdo-infra/ci-config/+/3681613:15
*** jpena|lunch is now known as jpena13:16
rlandy|ruckhttps://review.opendev.org/c/openstack/tripleo-ci/+/818222  https://review.rdoproject.org/r/c/testproject/+/3681713:17
rlandy|ruckhttps://review.opendev.org/c/openstack/tripleo-ci/+/818222/10/roles/tripleo-build-jobs-repos/tasks/main.yaml13:17
pojadhavrlandy|ruck, patch : https://review.rdoproject.org/r/c/rdo-jobs/+/36718 and testing patch : https://review.rdoproject.org/r/c/testproject/+/3671913:18
dviroelarxcruz: o/13:32
dviroelarxcruz: about your question wrt collect logs in fips jobs. scen002 didn't fail to collect them in latest run13:32
dviroelarxcruz: it was able to collect all container logs within the default 60 retries (I increased to 120 in my DNM patch)13:33
dviroelhttps://3ebe69169634ca122567-8e9a9f995b4a6012c53e1bcc573a9fc8.ssl.cf2.rackcdn.com/808215/54/check/tripleo-ci-centos-8-scenario002-standalone/7150c66/logs/quickstart_collect_logs.log13:33
dviroellast fail was: "RETRYING: Wait for container logs collection if not finished yet (68 retries left)"13:33
rlandy|ruckarxcruz: dviroel: marios: sshnaidm: pojadhav: akahat|rover: soniya29|ruck: review list updated https://hackmd.io/FGMoCiRfSNa8puA1BpTQ-Q13:34
soniya29|ruckrlandy|ruck, ack13:35
rlandy|ruckakahat|rover: is https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/817002 still needed?13:35
mariosack rlandy|ruck thanks13:35
akahat|roverlooking13:36
akahat|roverabandon: https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/817002, since test is passing.. no need to add it in skiplist13:38
pojadhavrlandy|ruck, ack13:49
rlandy|ruckakahat|rover: soniya29|ruck: let's ruck/rover sync https://meet.google.com/fre-agvw-bgt?pli=1&authuser=013:57
rlandy|ruckysandeep: your test job on containers14:01
rlandy|ruckon rhos-1714:01
rlandy|ruckis that depends-on taking14:01
ysandeeprlandy|ruck, yeah not taking depends-On, thanks14:02
* ysandeep checking what's wrong14:03
*** rcastillo|away is now known as rcastillo14:04
ysandeeprlandy|ruck, turns out a additional space between Depends-On and https:// can cause that 14:04
rlandy|ruckyes - suspected14:04
akahat|roverrlandy|ruck, https://review.rdoproject.org/r/gitweb?p=rdo-jobs.git;a=blob;f=zuul.d/nodesets.yaml#l30114:10
*** ysandeep is now known as ysandeep|dinner14:10
rlandy|ruckakahat|rover: https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/standalone-jobs.yaml#L33514:13
dviroel ade_lee o/14:28
dviroelade_lee: PS54 has scenario002 logs https://zuul.opendev.org/t/openstack/build/7150c66f5f0142e38309165a7db6ce2914:28
rlandy|ruckha  I think mirrors are back14:41
rlandy|ruckrecheck ...14:44
*** ysandeep|dinner is now known as ysandeep14:49
rlandy|ruckysandeep: I will miss the DF call - have to listen to some manager info session14:50
ysandeeprlandy|ruck, ack, want me to report something apart from mirror issue?14:51
rlandy|ruckysandeep: ack14:51
rlandy|ruckthey are back14:51
rlandy|ruckrecheck away14:51
ysandeepgreat!14:51
rlandy|ruckysandeep: rechecked train ubi patch14:51
rlandy|ruckmarios: ^^ fyi - will be missing DF call14:52
rlandy|ruckysandeep will report on mirror fun14:52
rlandy|ruckand ubi decisions14:52
rlandy|ruckakahat|rover: soniya29|ruck: pls ping when you are EoD so and let me know what you are watching14:53
ysandeepdownstream testprojects are kicking, we will now in few hours what worked / what not for 17.. 14:53
ysandeepknow* in14:53
rlandy|ruckysandeep++ yep14:53
rlandy|ruckalso pls ping when you are EoD 14:53
rlandy|ruckwill watch there and leave notes14:53
akahat|roverrlandy|ruck, ack14:53
* rlandy|ruck gets fs file changes in14:53
ysandeepack, I will go afk after df call and will comes back after ~2 hours - will handover then14:54
rlandy|ruckk14:55
*** dviroel is now known as dviroel|lunch14:57
mariosrlandy|ruck: o/15:13
mariosrlandy|ruck: digging on the c9 but 16:52 < rlandy|ruck> marios: ^^ fyi - will be missing DF call15:13
rlandy|ruckk - no worries15:13
rlandy|ruckjuts fyi15:13
mariosrlandy|ruck: k 15:13
*** ykarel is now known as ykarel|away15:13
*** pojadhav is now known as pojadhav|afk15:14
*** akahat|rover is now known as akahat|dinner15:19
ade_leedviroel|lunch, hey - yes adding more did allow the logs to collect15:25
ade_leedviroel|lunch, progress so far on the ci jobs -- scenario 2 is now collecting logs and we see that the test that we were trying to skip isn't being skipped15:30
ade_leethe ipa and standalone jobs are failing because the version of erlang thats there doesn't support fips, and even if it did, rabbot would need to be configured to start in fips mode.  eck is investigating exactly what needs to be there. 15:31
ade_leebut it fails because it tries to use a disallawed ecc key when connecting15:32
ade_leescenario 10 appears to be failing because we don't have the fips environment file for some reason -- may need your help to figure out why not.15:34
ade_leeand just need to figure out scenario 4 ..15:35
soniya29|ruckrlandy|ruck, leaving for the day15:36
rlandy|ruckack - have a goo dnight15:36
soniya29|ruckrlandy|ruck, :)15:36
*** ysandeep is now known as ysandeep|afk15:37
*** dviroel|lunch is now known as dviroel15:54
dviroelade_lee: yes, the script that captures all container logs is taking longer in fips env, not sure why16:04
dviroelade_lee: i'll take a look on scn10 too16:04
ade_leedviroel, yeah - not sure why either 16:05
sshnaidmrlandy|ruck, that's why important to have same users in CI: https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/ipa-multinode/tasks/ipaserver-undercloud-setup.yml#L13216:11
sshnaidmpojadhav|afk, fyi ^16:11
sshnaidmrlandy|ruck, did you here something back from SF folks about user in c9 images?16:12
rlandy|rucksshnaidm: yep- we merged that patch16:13
rlandy|ruckto switch to zuul user16:13
rlandy|ruckyou need to change the nodeset to get tha16:13
rlandy|ruckthat16:13
sshnaidmrlandy|ruck, oh, great16:14
rlandy|rucksshnaidm: in meeting - will send shortly16:14
dviroelmarios: you adding tripleo-repos cause needs get-hash right? we can think on yum-config as future improvement16:15
mariosdviroel: yeah it couldn't find the module so i am trying a ansible-galaxy install but i don't think it will work like that16:17
mariosdviroel: lets do a sync in 10 mins before i leave? half past the hour ok? 16:18
dviroelmarios: ok16:18
mariosdviroel: meet.google.com/sgz-oujz-vwr16:31
rlandy|rucksshnaidm: https://softwarefactory-project.io/r/c/config/+/2305716:57
rlandy|ruckmerged16:57
*** marios is now known as marios|out16:58
*** akahat|dinner is now known as akahat|rover16:58
rlandy|rucksshnaidm: pasting conversation in pvt chat16:58
*** ysandeep|afk is now known as ysandeep16:59
akahat|roverrlandy|ruck, sc10 voting patch got merged. https://review.opendev.org/c/openstack/tripleo-ci/+/80261617:02
rlandy|ruckyay17:04
rlandy|ruckhappy dance17:04
rlandy|ruckakahat|rover: give yourself a clap for that one!!17:05
rlandy|ruckwell lucky all branches promoted yesterday17:06
rlandy|rucklunch - brb17:06
ysandeeparxcruz, kopecmartin do you know any recent changes that could cause : https://sf.hosted.upshift.rdu2.redhat.com/logs/43/211643/95/check/periodic-tripleo-ci-rhel-8-scenario010-standalone-rhos-17/65314f4/job-output.txt 17:10
ysandeepModuleNotFoundError: No module named 'certifi' during TASK [os_tempest : List installed tempest plugins] 17:10
arxcruzysandeep: nope, i can investigate though 17:10
toskyysandeep, arxcruz : it's a new dependency of octavia-tempest-plugin through python3-httpx17:10
toskyI've seen that downstream; the package is fixed in centos9 but for centos8/rhel8 it requires some fixes in the dependency17:11
ysandeeptosky++ thanks was quick, thanks! do you by any chance also know where we added dependency on python3-httpx, any reviews/bug ?17:13
arxcruztosky: which dependency? i can submit a patch to os_tempest17:13
toskyysandeep, arxcruz: you just need to add a dependency to that package (there is already a custom fix to depend on httpcore)17:13
toskyarxcruz: it's not os_tempest, you need to patch python-httpx17:14
arxcruzahh 17:14
toskysee the last fix here, which adds a dependency on httpcore? https://cbs.centos.org/koji/buildinfo?buildID=3142417:14
toskythis one is the same problem, httpx needs *also* a dependency on python-certifi17:14
arxcruztosky: what i meant is, i can add the python-certifi on the list of pacakages that needs to be installed by os_tempest 17:15
toskyarxcruz: but that's not the proper fix17:15
toskyarxcruz: the fix is the dependency17:15
toskyin the RPM17:15
arxcruzok17:15
*** dviroel is now known as dviroel|afk17:30
ysandeeprlandy|ruck, fyi.. https://bugzilla.redhat.com/show_bug.cgi?id=2024700 , Its late for me now but I will check with yatin tomorrow how can we fix  python-httpx package as tosky suggested ^^17:34
rlandy|ruckysandeep: thanks17:38
rlandy|ruckreading back17:38
*** jpena is now known as jpena|off17:39
toskyysandeep: sorry, is that a downstream bug? I thought it was an upstream issue17:40
toskyysandeep: the downstream issue is already fixed (in the next compose)17:40
toskyrlandy|ruck, arxcruz ^^17:40
ysandeeptosky: in which rpm?17:41
ysandeeptosky, we are facing in integration line, integration line is ahead of compose17:41
toskyysandeep: python-httpx, as I've written above17:42
ysandeeptosky, I mean in which version of python-httpx issue is fixed in downstream?17:43
toskyysandeep: again: python-httpx needs that dependency; if we are talking about centos, that needs to be fixed; if it's about downstream, that's a different story and the package was just fixed17:43
toskyysandeep: can we talk about this in an internal channel? I don't feel confortable talking downstream-only stuff  here17:43
ysandeeptosky, sure , ping you there17:44
*** amoralej is now known as amoralej|off17:47
ysandeeprlandy|ruck, me out now, I have send a pm with some handover details18:08
*** ysandeep is now known as ysandeep|out18:12
ysandeep|outrlandy|ruck, could you please retriiger fs020 for 17 if fs001 passes there - still running18:13
rlandy|ruckyep18:39
rlandy|ruckfs001 failed18:57
*** dviroel|afk is now known as dviroel19:53
*** dviroel is now known as dviroel|afk21:34
*** chem is now known as Guest624423:38
« Wednesday, 2021-11-17 Index Friday, 2021-11-19 »

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!