| *** rlandy is now known as rlandy|out | 00:36 | |
| *** dasm|afk is now known as dasm | 01:17 | |
| *** dasm is now known as dasm|gone | 03:41 | |
| *** soniya29 is now known as soniya29|rover | 03:41 | |
| soniya29|rover | o/ | 03:52 |
|---|---|---|
| soniya29|rover | Is https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds down? | 03:52 |
| *** ysandeep|out is now known as ysandeep | 04:26 | |
| ysandeep | soniya29|rover, There is an ongoing outage in psi as per outage list notification, I think that's related. | 04:32 |
| soniya29|rover | ysandeep: okay, thanks | 04:34 |
| *** marios is now known as marios|ruck | 05:06 | |
| marios|ruck | \o/ | 05:06 |
| ysandeep | o/ | 05:19 |
| soniya29|rover | marios|ruck, hello | 05:50 |
| soniya29|rover | marios|ruck, downstream zuul is facing issues may be because of psi outage | 05:51 |
| marios|ruck | soniya29|rover: thanks - can you see if you can find out more in rhos-ops? | 05:54 |
| * ysandeep already left a message on rhos-ops | 05:57 | |
| marios|ruck | thanks ysandeep soniya29|rover | 05:59 |
| soniya29|rover | ysandeep, thanks :) | 05:59 |
| soniya29|rover | marios|ruck, cloups component is also facing trasient failures on master..failures are random | 06:27 |
| soniya29|rover | it is also a promotion blocker | 06:27 |
| * soniya29|rover will be back in 10-15 mins | 06:30 | |
| marios|ruck | soniya29|rover: thank you | 06:31 |
| marios|ruck | soniya29|rover: when you come back can you please dig a bit there and see if you can find any new bugs? for example if any two of those jobs are failing for the same thing, then it is a new bug | 06:31 |
| marios|ruck | soniya29|rover: ping if you want to discuss | 06:32 |
| marios|ruck | coffeee brb | 06:36 |
| marios|ruck | arxcruz: please vote when you get a chance https://review.opendev.org/c/openstack/tripleo-ci/+/835776 (https://zuul.openstack.org/status#835776) | 06:42 |
| soniya29|rover | marios|ruck, pong | 06:47 |
| marios|ruck | soniya29|rover: o/ | 06:47 |
| soniya29|rover | marios|ruck, job is failing on different issues not on same thing, i wll re run them to see if any of the failure is consistent enough to file a bug | 06:49 |
| marios|ruck | soniya29|rover: is it one job or different jobs with different fails? | 06:49 |
| marios|ruck | soniya29|rover: ok sounds good to try a test | 06:49 |
| soniya29|rover | marios|ruck, sure | 06:49 |
| soniya29|rover | marios|ruck, same job | 06:50 |
| marios|ruck | soniya29|rover: which one can you give me link to zuul builds please? i'll try have a loko in a bit as well | 06:52 |
| marios|ruck | soniya29|rover: see private message too | 06:53 |
| soniya29|rover | https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-scenario000-multinode-oooq-container-updates-cloudops-master | 06:54 |
| soniya29|rover | marios|ruck, sure | 06:54 |
| marios|ruck | soniya29|rover: doesn't appear to be random | 06:56 |
| marios|ruck | soniya29|rover: i see 2 things that are the same | 06:56 |
| marios|ruck | soniya29|rover: for example this is one issue https://logserver.rdoproject.org/openstack-component-cloudops/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-scenario000-multinode-oooq-container-updates-cloudops-master/642c12f/logs/undercloud/home/zuul/undercloud_install.log.txt.gz && | 06:57 |
| marios|ruck | https://logserver.rdoproject.org/openstack-component-cloudops/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-scenario000-multinode-oooq-container-updates-cloudops-master/b88bb92/logs/undercloud/home/zuul/undercloud_install.log.txt.gz same problem | 06:57 |
| marios|ruck | 2022-03-28 01:03:17.206841 | fa163eb7-de29-3e46-a313-0000000012c2 | FATAL | Container image prepare | undercloud | error={"changed": false, "error": "Not found image: https://trunk.registry.rdoproject.org/v2/tripleomastercentos9/openstack-iscsid/manifests/c16775607f2619e1086d495b6ddb8612", "msg": "Error running container image prepare: Not found image: | 06:57 |
| marios|ruck | https://trunk.registry.rdoproject.org/v2/tripleomastercentos9/openstack-iscsid/manifests/c16775607f2619e1086d495b6ddb8612", "params": {}, "success": false} | 06:57 |
| marios|ruck | soniya29|rover: and then there is a different one | 06:57 |
| marios|ruck | soniya29|rover: here: https://logserver.rdoproject.org/openstack-component-cloudops/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-scenario000-multinode-oooq-container-updates-cloudops-master/fe56b06/job-output.txt && | 06:57 |
| marios|ruck | https://logserver.rdoproject.org/openstack-component-cloudops/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-scenario000-multinode-oooq-container-updates-cloudops-master/bf575c9/job-output.txt | 06:57 |
| marios|ruck | 2022-03-26 01:36:06.891581 | primary | fatal: [undercloud]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: kex_exchange_identification: Connection closed by remote host\r\nConnection closed by 127.0.0.2 port 22", "unreachable": true} | 06:57 |
| soniya29|rover | marios|ruck, correct, there is different on also there | 06:57 |
| marios|ruck | soniya29|rover: k cos you said it is all random | 06:58 |
| marios|ruck | soniya29|rover: so we may have 2 bugs here... go ahead and post a testproject and see .. i expect it may hit that one ^^ ssh one is the most recent... but depending on the test can yu please file bugs? | 06:59 |
| marios|ruck | soniya29|rover: is it clear? | 06:59 |
| marios|ruck | soniya29|rover: we can have a call if it is not let me know | 06:59 |
| soniya29|rover | marios|ruck, got it | 06:59 |
| marios|ruck | soniya29|rover: thank you | 07:00 |
| soniya29|rover | marios|ruck, i will two file bugs about the issues and post a testproject | 07:00 |
| soniya29|rover | marios|ruck, thanks | 07:00 |
| marios|ruck | soniya29|rover: i think only file 1 bug | 07:00 |
| marios|ruck | soniya29|rover: the one about the missing container is probably a transient thing and we now have new containers so don't see it any more? | 07:01 |
| marios|ruck | soniya29|rover: for now its probably only worth filing the bug for the ssh thing OK? | 07:01 |
| marios|ruck | soniya29|rover: ? | 07:01 |
| soniya29|rover | marios|ruck, okay | 07:02 |
| marios|ruck | soniya29|rover: thanks | 07:02 |
| marios|ruck | arxcruz: can you see my comment @ https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/835475/1#message-92d9c6e7185a503b2312861fea1676ed84c86469 sanity check wrt the URL is it OK ? | 07:08 |
| arxcruz | marios|ruck yes | 07:15 |
| *** arxcruz is now known as arxcruz|out | 07:15 | |
| *** amoralej|off is now known as amoralej | 07:19 | |
| jm1 | o/ | 07:25 |
| marios|ruck | \o/ | 07:26 |
| marios|ruck | (/me surrendering to be clear jm1 ^^) ;) | 07:26 |
| jm1 | He? | 07:27 |
| marios|ruck | zuul wins i surrender | 07:27 |
| jm1 | marios|ruck: zuul always wins ^^ | 07:27 |
| jm1 | marios|ruck: zuul is from the dark side | 07:28 |
| marios|ruck | :) | 07:28 |
| jm1 | marios|ruck: anything you need help with zuul? | 07:28 |
| marios|ruck | jm1: no mainly referring to ruck/lots of failed things | 07:28 |
| marios|ruck | thanks jm1 :) | 07:28 |
| jm1 | marios|ruck: took me only 2+x weeks until i got our zuul problems solved. but hey, finally it looks like zuul follows orders ^^ | 07:29 |
| jm1 | marios|ruck: btw is there any ci job for centos8 where i can see which rpm versions (e.g. for ansible-collection-containers-podman) are being used? | 07:32 |
| jm1 | marios|ruck: (i want to check if our updated packages have arrived in rdo centos8 repos) | 07:33 |
| marios|ruck | jm1: in any of the jobs look at /var/log/extra there is an installed package list there | 07:35 |
| *** jpena|off is now known as jpena | 07:37 | |
| * jm1 looks for logs | 07:40 | |
| jm1 | marios|ruck: thanks, found the package list :) | 07:45 |
| soniya29|rover | marios|ruck, https://bugs.launchpad.net/tripleo/+bug/1967073 | 07:45 |
| jm1 | ysandeep: you were hit by a podman bug some time before. which job was this? | 07:46 |
| ysandeep | jm1, no job - brendan hit that locally - that's when I mentioned we should consider adding a job to redeploy. | 07:47 |
| soniya29|rover | chasing here:- https://review.rdoproject.org/r/c/testproject/+/41131 | 07:47 |
| soniya29|rover | marios|ruck, ^^ | 07:47 |
| jm1 | ysandeep: ah, of course 🤦 but do you know a job which job does the same without redeploy? i am looking for a job that uses the ansible-collection-containers-podman rpm | 07:49 |
| marios|ruck | thanks soniya29|rover | 07:51 |
| marios|ruck | ack jm1 | 07:51 |
| marios|ruck | ysandeep: fyi https://review.opendev.org/c/openstack/tripleo-ci/+/835541/5#message-04969223f48658dc1eaca027ef8430d21c89114b | 07:54 |
| ysandeep | jm1, don't we use ansible-collection-containers-podman by default in tripleo? | 07:56 |
| ysandeep | jm1, for ex. i see its installed here: | 07:56 |
| ysandeep | https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-wallaby/5f39d5b/logs/undercloud/var/log/extra/package-list-installed.txt.gz | 07:56 |
| ysandeep | ansible-collection-containers-podman.noarch 1.8.2-1.el8 @delorean-wallaby-testing | 07:56 |
| ysandeep | marios|ruck, thanks done | 07:58 |
| * ysandeep afk for lunch | 07:59 | |
| jm1 | ysandeep: thanks :) | 07:59 |
| *** ysandeep is now known as ysandeep|lunch | 07:59 | |
| *** pojadhav is now known as pojadhav|afk | 08:08 | |
| *** pojadhav|afk is now known as pojadhav | 08:39 | |
| marios|ruck | soniya29|rover: you filed the image prepare one instead of the other (ssh one) ? was it intentional 10:45 < soniya29|rover> marios|ruck, https://bugs.launchpad.net/tripleo/+bug/1967073 | 08:57 |
| marios|ruck | 10:01 < marios|ruck> soniya29|rover: the one about the missing container is probably a transient thing and we now have new containers so don't see it any more? | 08:58 |
| marios|ruck | 10:01 < marios|ruck> soniya29|rover: for now its probably only worth filing the bug for the ssh thing OK? | 08:58 |
| *** ysandeep|lunch is now known as ysandeep | 08:58 | |
| marios|ruck | soniya29|rover: ? | 08:59 |
| soniya29|rover | marios|ruck, okay | 09:04 |
| soniya29|rover | marios|ruck, i think i filed something different mistakenly | 09:05 |
| soniya29|rover | i will close it and file new one..probably after lunch | 09:05 |
| *** soniya29|rover is now known as soniya29|rover|lunch | 09:05 | |
| marios|ruck | soniya29|rover: thanks | 09:05 |
| marios|ruck | soniya29|rover|lunch: is this one a duplicate https://bugs.launchpad.net/tripleo/+bug/1966972 (for https://bugs.launchpad.net/tripleo/+bug/1967073 ) ? I think it may be | 09:26 |
| marios|ruck | soniya29|rover|lunch: i suspect if you rerun it will clear (same as https://bugs.launchpad.net/tripleo/+bug/1967073/comments/1) can you please check when you find time thanks | 09:27 |
| marios|ruck | soniya29|rover|lunch: actually it is a different issue but i marked it duplicate for another bug fyi https://bugs.launchpad.net/tripleo/+bug/1966972/comments/1 | 09:37 |
| marios|ruck | amoralej: o/ hey any idea how we can disable extras-common for build-test-packages (the repo seems to be pulled in for the dlrn build ( re https://bugs.launchpad.net/tripleo/+bug/1967012/comments/4 ) | 09:59 |
| * marios|ruck be right back foooooood | 10:01 | |
| *** marios|ruck is now known as marios|ruck|lunch | 10:01 | |
| *** soniya29|rover|lunch is now known as soniya29|rover | 10:07 | |
| soniya29|rover | marios|ruck|lunch, i am getting confused?...what is different issue of whom | 10:09 |
| *** ysandeep is now known as ysandeep|afk | 10:09 | |
| soniya29|rover | we can clear this issue when you get back | 10:10 |
| marios|ruck|lunch | soniya29|rover: read the bugs if there are questions ask here thanks (I added comments on both) | 10:13 |
| marios|ruck|lunch | soniya29|rover: https://bugs.launchpad.net/tripleo/+bug/1967073/comments/1 and https://bugs.launchpad.net/tripleo/+bug/1966972/comments/1 | 10:14 |
| soniya29|rover | marios|ruck|lunch, thanks | 10:14 |
| soniya29|rover | i have got a green run over the above issue, i think we don't need a bug for ssh issue either, because its a fresh run | 10:15 |
| soniya29|rover | too many transient errors these days | 10:15 |
| amoralej | marios|ruck|lunch, lemme check | 10:16 |
| amoralej | i think that is done in build-test-packages ... | 10:18 |
| amoralej | iirc it added repos somehow | 10:18 |
| *** rlandy|out is now known as rlandy | 10:22 | |
| amoralej | marios|ruck|lunch, i replied in the lp | 10:23 |
| rlandy | marios|ruck|lunch: soniya29|rover: hello | 10:24 |
| amoralej | it has been added recently in the official centos repos | 10:24 |
| rlandy | I se PSI is still out :( | 10:24 |
| rlandy | amoralej: hi ... thanks for responding to https://bugs.launchpad.net/tripleo/+bug/1967012 | 10:25 |
| rlandy | it's only build-test-packages | 10:25 |
| rlandy | checking the latest | 10:25 |
| rlandy | https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-8-content-provider&skip=0 | 10:26 |
| soniya29|rover | rlandy, hello | 10:26 |
| rlandy | looks like we had some passes and some fails | 10:26 |
| rlandy | yesterday it was all red | 10:26 |
| soniya29|rover | rlandy, too many transient errors are happening | 10:27 |
| rlandy | maybe mirror availability is could related | 10:27 |
| rlandy | soniya29|rover: will sync when marios|ruck|lunch is back | 10:27 |
| rlandy | amoralej: https://953231453cf8ba1b2f86-0951bd3f7386b5b69d3017bc03aa7f0f.ssl.cf1.rackcdn.com/834726/3/check/tripleo-ci-centos-8-content-provider/a4bfa97/job-output.txt - still happening | 10:27 |
| amoralej | rlandy, i've just added the code in build-test-packages wchich pulled it | 10:27 |
| rlandy | Provider: rax-dfw failing | 10:29 |
| rlandy | ysandeep|afk: jpodivin: we have one more ... https://38c2e58cc4c278c49d69-abb8a8f0319292cc75f8cf2775f900b3.ssl.cf2.rackcdn.com/791949/3/gate/openstack-tox-molecule/17c6aa4/job-output.txt | 10:30 |
| rlandy | upgrades | 10:30 |
| rlandy | soniya29|rover: too many transient failures where? | 10:30 |
| *** marios|ruck|lunch is now known as marios|ruck | 10:30 | |
| marios|ruck | thank you amoralej checking | 10:31 |
| jpodivin | rlandy: I see | 10:31 |
| soniya29|rover | rlandy, cloudops component line | 10:31 |
| marios|ruck | ah thanks amoralej lets try that | 10:31 |
| rlandy | marios|ruck: hello | 10:32 |
| amoralej | i'm still not sure why it fails to get it from mirrorlist but anyway we could just get rid of that repo | 10:32 |
| marios|ruck | rlandy: hello ... gimme sec posting something for the c8 content provider bug (amoralej++ https://bugs.launchpad.net/tripleo/+bug/1967012/comments/6 ) | 10:32 |
| jpodivin | rlandy: that's ... interesting, the triple-upgrade repo was patched from what I can tell https://opendev.org/openstack/tripleo-upgrade/src/branch/master/setup.py | 10:33 |
| rlandy | jpodivin: weird becuase the gate should be rebased | 10:33 |
| rlandy | marios|ruck: sure - just updating the downstream infro on program doc | 10:34 |
| marios|ruck | rlandy: i updated but please check? | 10:34 |
| rlandy | missing BZ on 16.2 | 10:34 |
| jpodivin | rlandy: oh wait, the patch is on train | 10:35 |
| jpodivin | rlandy: that makes sense, the fix wasn't merged there yet https://review.opendev.org/c/openstack/tripleo-upgrade/+/835791 | 10:36 |
| amoralej | marios|ruck, you remember where the standard repos are removed to be replaced by quickstart-centos-base, etc...? | 10:37 |
| rlandy | marios|ruck: https://bugzilla.redhat.com/show_bug.cgi?id=2069171 is on 16.2 not 17 - correcting that | 10:38 |
| soniya29|rover | rlandy, yes, that bug is for 16.2 | 10:38 |
| rlandy | updates | 10:39 |
| rlandy | updated | 10:39 |
| rlandy | amoralej: marios|ruck: release files: https://github.com/openstack/tripleo-quickstart/blob/master/config/release/tripleo-ci/CentOS-8/wallaby.yml#L168 | 10:41 |
| rlandy | example | 10:41 |
| amoralej | ah, i know what's the problem | 10:41 |
| rlandy | quickstart-* repos are added | 10:41 |
| amoralej | it's tricky | 10:41 |
| amoralej | it does dnf config-manager --disable '*' | 10:41 |
| rlandy | amoralej: but pls see line https://github.com/openstack/tripleo-quickstart/blob/master/config/release/tripleo-ci/CentOS-8/wallaby.yml#L74 | 10:41 |
| amoralej | that disables all repos | 10:41 |
| rlandy | ack | 10:41 |
| rlandy | that did not work | 10:41 |
| amoralej | but, then it updates centos-stream-repos | 10:41 |
| amoralej | https://953231453cf8ba1b2f86-0951bd3f7386b5b69d3017bc03aa7f0f.ssl.cf1.rackcdn.com/834726/3/check/tripleo-ci-centos-8-content-provider/a4bfa97/logs/undercloud/home/zuul/repo_setup.log | 10:41 |
| amoralej | that introduces a new repo extras-common | 10:42 |
| amoralej | which is disabled by default | 10:42 |
| * rlandy checks where the update is | 10:42 | |
| amoralej | it's part of dnf update -y | 10:42 |
| rlandy | ugh - so that brings in a new repo | 10:43 |
| amoralej | https://github.com/openstack/tripleo-quickstart/blob/master/config/release/tripleo-ci/CentOS-8/master.yml#L228 | 10:43 |
| amoralej | yes | 10:43 |
| amoralej | one option would be to do a | 10:43 |
| amoralej | dnf update centos-stream-repo right before https://github.com/openstack/tripleo-quickstart/blob/master/config/release/tripleo-ci/CentOS-8/master.yml#L74 | 10:43 |
| rlandy | should we have a quickstart- version of that repo? | 10:44 |
| rlandy | or just flat disable it? | 10:44 |
| rlandy | amoralej: ok - let me try that | 10:44 |
| amoralej | i doubt we will need that tbh | 10:44 |
| rlandy | some tests are passing now - so my guess is some mirrors have started to update | 10:45 |
| amoralej | but if we need it, the right thing would be to add the quickstart- version, yep | 10:45 |
| rlandy | but this is safer | 10:45 |
| marios|ruck | rlandy: amoralej: posted that https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/835815 | 10:45 |
| amoralej | rlandy, yes, that's the other part | 10:45 |
| marios|ruck | reading back here.. | 10:45 |
| amoralej | even if the repo definition is there, it shouldn't fail | 10:45 |
| marios|ruck | rlandy: phone? | 10:45 |
| rlandy | marios|ruck: gmeet | 10:45 |
| marios|ruck | yeah | 10:45 |
| rlandy | amoralej: marios|ruck: I think amoralej suggestion is better | 10:46 |
| marios|ruck | amoralej: so you think it is something other than https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/835815 | 10:46 |
| marios|ruck | rlandy: that is amoralej suggestion :) | 10:46 |
| rlandy | it will save us in the future | 10:46 |
| rlandy | amoralej> dnf update centos-stream-repo right before https://github.com/openstack/tripleo-quickstart/blob/master/config/release/tripleo-ci/CentOS-8/master.yml#L74 | 10:46 |
| rlandy | ^^ that ione | 10:46 |
| rlandy | but let's discuss | 10:46 |
| marios|ruck | k | 10:46 |
| rlandy | https://meet.google.com/xwo-uevu-rcj | 10:47 |
| rlandy | soniya29|rover" ^^ pls join | 10:47 |
| marios|ruck | rlandy: https://zuul.openstack.org/status#835762 | 11:00 |
| * soniya29|rover coffee break | 11:09 | |
| *** soniya29|rover is now known as soniya29|rover|brb | 11:09 | |
| amoralej | rlandy, marios|ruck i did two different suggestions, sorry :) | 11:14 |
| marios|ruck | amoralej: ack we are trying both ;) | 11:15 |
| rlandy | amoralej; no worries - yep - what marios|ruck said | 11:16 |
| amoralej | i explained in the lp the long version | 11:16 |
| *** dviroel|out is now known as dviroel | 11:17 | |
| *** ysandeep|afk is now known as ysandeep | 11:18 | |
| ysandeep | rlandy, fyi.. downstream ci is completely dead due to psi outage. | 11:38 |
| dviroel | :( | 11:38 |
| rlandy | ysandeep: yep - outage started all of yesterday afternoon | 11:40 |
| rlandy | and went all night | 11:41 |
| rlandy | and is still rocking | 11:41 |
| rlandy | in fact two of them :) | 11:41 |
| rlandy | we reported that on the program call doc | 11:41 |
| *** soniya29|rover|brb is now known as soniya29|rover | 11:42 | |
| ysandeep | >> we reported that on the program call doc - ack, thanks! | 11:43 |
| marios|ruck | rlandy: i think it should be OK now? (https://review.opendev.org/c/openstack/tripleo-ci/+/835674/5#message-ed7c1270d1babb023fd1849dee70e6797293d6b3 https://zuul.openstack.org/status#835674 | 11:51 |
| marios|ruck | rlandy: i added the fix directly into that ^^^ as there are a few patches that are rebased onto the parent and don't want to disrupt them | 11:51 |
| rlandy | checking | 11:54 |
| rlandy | marios|ruck: looks good - thanks | 11:55 |
| rlandy | also checking the THT DNM test patch | 11:55 |
| rlandy | marios|ruck++ amoralej++ https://zuul.opendev.org/t/openstack/stream/b132add59df64031bafc8d49e40a8d77?logfile=console.log looking better | 11:57 |
| rlandy | hopefully that was not just one cloud with an updated mirro | 11:57 |
| rlandy | marios|ruck: soniya29|rover: who is presenting at program call? | 11:57 |
| soniya29|rover | rlandy, i can present | 11:58 |
| marios|ruck | rlandy: i will, unless soniya29|rover would like to? | 11:58 |
| marios|ruck | soniya29|rover: ok | 11:58 |
| soniya29|rover | marios|ruck, thanks :) | 11:59 |
| marios|ruck | soniya29|rover: i will be standby if you need anything | 11:59 |
| soniya29|rover | marios|ruck, sure | 11:59 |
| rlandy | soniya29|rover: pls raise the PSI outage issue | 12:00 |
| amoralej | rlandy, probably both, the patch fixes it and the mirrorlist is updated, we'll see it in logs | 12:00 |
| rlandy | as a downstream blocker | 12:00 |
| marios|ruck | rlandy: soniya29|rover: its on the doc (psi outage) | 12:01 |
| soniya29|rover | rlandy, sure, i will raise it | 12:01 |
| amoralej | note that, if mirrorlist is not getting extras-repos, it may fail in a later step, after build-test-packages | 12:01 |
| rlandy | ahead of April 6th clode freeze for import | 12:01 |
| rlandy | ysandeep: jpodivin: marios|ruck: I w+ed all patches that had zuul +1 on https://review.opendev.org/q/topic:lp%252F1966382 | 12:13 |
| rlandy | some failures there | 12:13 |
| rlandy | some are in recheck | 12:13 |
| ysandeep | rlandy, thanks! Will check the failing once in sometime. | 12:14 |
| rlandy | ysandeep: some are waiting on marios|ruck patch to fix c8 content provider | 12:14 |
| ysandeep | okay | 12:15 |
| rlandy | some are tox failures | 12:15 |
| *** amoralej is now known as amoralej|lunch | 12:16 | |
| marios|ruck | rlandy: thank you | 12:17 |
| *** ysandeep is now known as ysandeep|afk | 12:20 | |
| rlandy | ysandeep|afk: bhagyashris: amol is taking over the ruck rover shift on friday but he is PTO | 12:22 |
| rlandy | dasm|gone is the other person on the shift | 12:22 |
| bhagyashris | rlandy, ack | 12:22 |
| rlandy | pls add to planning if there is one person who can help cover friday | 12:22 |
| bhagyashris | ok | 12:22 |
| rlandy | preferable someone in earlier TZ | 12:23 |
| jpodivin | rlandy: thanks for the votes. | 12:24 |
| jpodivin | Those time outs ... is something happening with the mirrors? | 12:24 |
| rlandy | jpodivin: always :( | 12:25 |
| jpodivin | rlandy: yes there seems to be sort of a ... trend lately. But maybe I'm just seeing things. | 12:26 |
| rlandy | jpodivin: checking | 12:27 |
| rlandy | I see a few fails on tox | 12:27 |
| rlandy | the content provider one we know about marios|ruck has a fix there | 12:28 |
| jpodivin | rlandy: There's this one https://zuul.opendev.org/t/openstack/build/f4e543127d7f4445adf9796f56a8abfd ... mirrors for tripleo-centos-highavailability don't seem to be ... highly available. | 12:31 |
| jpodivin | The same job had another failure today, this time due to the 'centos-nfv-ovs' repo not being available. | 12:33 |
| rlandy | ugh ... adding this to the list of joy | 12:33 |
| marios|ruck | jpodivin: rlandy: doesn't seem to be a general thing? (yet?) https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-tox-molecule&project=openstack/tripleo-ci | 12:34 |
| jpodivin | marios|ruck: well, no it isn't failing all the time yey. | 12:34 |
| jpodivin | s/yey/yet/ | 12:34 |
| rlandy | mirror issues are usually transient | 12:34 |
| rlandy | c9 compose updated on 28 | 12:35 |
| jpodivin | rlandy: that could be it. | 12:35 |
| rlandy | on we should expect some providers to lag that | 12:35 |
| rlandy | acording to infa 6 hours is a usual upgrade window | 12:35 |
| marios|ruck | rlandy: provider 8 is green there https://zuul.openstack.org/status#835762 but let me check logs see if it excluded that repo | 12:36 |
| jpodivin | ... it has been two days now. OTOH, we aren't seeing this everywhere/all the time yet. So I guess we don't need to devote much time to it. | 12:36 |
| rlandy | marios|ruck: check repos there | 12:38 |
| rlandy | https://c9fbb7c04753bc93e94d-0bcd974829c5636eec53c54a828a88e5.ssl.cf2.rackcdn.com/835762/3/check/tripleo-ci-centos-8-content-provider/b132add/logs/undercloud/etc/yum.repos.d/CentOS-Stream-Extras-common.repo | 12:38 |
| rlandy | marios|ruck: ^^ | 12:38 |
| rlandy | did I miss something? | 12:38 |
| marios|ruck | rlandy: https://bugs.launchpad.net/tripleo/+bug/1967012/comments/9 | 12:40 |
| marios|ruck | rlandy: yeah | 12:40 |
| marios|ruck | rlandy: that is laid down but the problem was it being used in build-test-packages | 12:40 |
| marios|ruck | rlandy: i.e. my patch stops that repo you pointed at from being included in the dlrnbuild | 12:40 |
| rlandy | marios|ruck: got it | 12:41 |
| rlandy | then we're golden | 12:41 |
| marios|ruck | rlandy: i think it is oK... let me remove WIP from my patch then you can decide after | 12:41 |
| rlandy | justcheck dlrn build log and I'l vote | 12:41 |
| rlandy | marios|ruck: you're right ... https://c9fbb7c04753bc93e94d-0bcd974829c5636eec53c54a828a88e5.ssl.cf2.rackcdn.com/835762/3/check/tripleo-ci-centos-8-content-provider/b132add/logs/undercloud/home/zuul/DLRN/data/repos/component/tripleo/52/f2/52f25290d7058d15885ff7a6c9cc1bdc0a4bb6d2_dev/rpmbuild.log | 12:42 |
| rlandy | will vote | 12:42 |
| marios|ruck | rlandy: thanks updated | 12:43 |
| rlandy | marios|ruck: +2'ed with reference to DLRN build repo log | 12:43 |
| rlandy | let's w+ that | 12:43 |
| rlandy | so we can clear gate | 12:44 |
| rlandy | dviroel: ^^ can you vote pls | 12:44 |
| marios|ruck | k | 12:44 |
| marios|ruck | amoralej|lunch:++ thanks for the pointer there it would have taken ages to track it down! | 12:45 |
| rlandy | marios|ruck: other burning issues you want help with?? | 12:45 |
| marios|ruck | rlandy: not currently, going to try tidy up our buglist/cix after this call | 12:45 |
| rlandy | ++ | 12:46 |
| dviroel | https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/835815 ? | 12:48 |
| rlandy | dviroel: ack | 12:52 |
| rlandy | soniya29|rover: after CIX let's sync on where you are with upstream components | 12:52 |
| rlandy | oh in fact we can do that now if you are around | 12:53 |
| rlandy | soniya29|rover: | 12:53 |
| rlandy | marios|ruck: ^^ just fyi - soniya29|rover and I will review that | 12:53 |
| marios|ruck | rlandy: soniya29|rover: thank you | 12:54 |
| soniya29|rover | rlandy, okay | 12:58 |
| *** ysandeep|afk is now known as ysandeep | 12:58 | |
| soniya29|rover | rlandy, meet.google.com/kzx-embe-qtj | 12:59 |
| soniya29|rover | marios|ruck, ack | 13:00 |
| rlandy | soniya29|rover: joining - was just checking on psi | 13:03 |
| rlandy | network line started | 13:03 |
| *** amoralej|lunch is now known as amoralej | 13:03 | |
| rlandy | so we can check if the patch is in play | 13:03 |
| rlandy | we can check if we are missing some upstream kvm jobs | 13:04 |
| rlandy | psi should kick them now | 13:04 |
| bhagyashris | arxcruz, rlandy, marios, ysandeep, bhagyashris, svyas, soniya29, pojadhav, akahat, chandankumar, frenzy_friday, dviroel, rcastillo, jm1, rlandy|ruck, dasm soniya29|rover marios|ruck | 13:22 |
| bhagyashris | Tripleo CI Retro in 7 mins | 13:23 |
| bhagyashris | https://miro.com/app/board/uXjVOBV0kFk=/ | 13:23 |
| rlandy | soniya29|rover: https://docs.openstack.org/tripleo-docs/latest/ci/stages-overview.html | 13:24 |
| *** dasm|gone is now known as dasm | 13:26 | |
| dasm | o/ | 13:26 |
| dasm | brb | 13:27 |
| marios|ruck | ykarel: o/ hey any updates on https://bugs.launchpad.net/tripleo/+bug/1964940/comments/10 (https://review.opendev.org/c/openstack/neutron/+/835340 ) did you manage to test with wallaby? | 13:30 |
| ykarel | marios|ruck, i attempted in https://review.rdoproject.org/r/c/testproject/+/41033 | 13:32 |
| ykarel | but the job looks unstable as failing with different issues | 13:32 |
| bhagyashris | dasm, rcastillo rlandy soniya29|rover pojadhav | 13:33 |
| bhagyashris | retro time | 13:33 |
| bhagyashris | dasm, rcastillo ^ | 13:33 |
| marios|ruck | ykarel: ok thanks just going through cix checking for udpates | 13:33 |
| pojadhav | bhagyashris, I am already in | 13:33 |
| ykarel | marios|ruck, will push a patch to enable debug mode for ovn, but for that to work need the issue to reproduce | 13:35 |
| ykarel | but job failing for one or other reason | 13:36 |
| bhagyashris | dasm, rcastillo around plz join we are waiting | 13:36 |
| marios|ruck | ysandeep: that one https://review.opendev.org/c/openstack/project-config/+/834199 let me go ask in opendev if somene can merge | 13:49 |
| ysandeep | marios|ruck, thanks! | 13:56 |
| ysandeep | rlandy, I will propose a fix for component lines in rr tooling, I noticed that too. | 14:20 |
| *** pojadhav is now known as pojadhav|afk | 14:31 | |
| dasm | ysandeep: +1 for the bot. Are you thinking about creating such bot? Or do you have it ready? | 14:31 |
| jm1 | rcastillo: here today or out? | 14:31 |
| rlandy | bhagyashris++ nice retro | 14:32 |
| ysandeep | dasm: just brainstorming currently for my weekend sideproject | 14:32 |
| dasm | ysandeep: nice. that could be useful. | 14:33 |
| ysandeep | dasm, I hope I will have something before by next rr shift :D | 14:33 |
| dasm | no worries. if you won't, it won't affect us at all :) | 14:34 |
| ysandeep | :D true.. | 14:35 |
| *** ysandeep is now known as ysandeep|out | 14:37 | |
| rlandy | marios|ruck: soniya29|rover: https://review.opendev.org/c/openstack/tripleo-ci/+/835101 is still not merged | 14:39 |
| rlandy | will rekick train c7 after that merges | 14:39 |
| marios|ruck | thanks for recheck rlandy | 14:41 |
| bhagyashris | rlandy, thanks | 14:42 |
| soniya29|rover | rlandy, marios|ruck, anything you want me to look in morning? | 14:56 |
| marios|ruck | soniya29|rover: check d/stream if it is back please - 16/17 lines? | 14:57 |
| soniya29|rover | marios|ruck, ack | 14:57 |
| * soniya29|rover leaving for the day | 14:59 | |
| rlandy | marios|ruck; no quite back | 14:59 |
| marios|ruck | rlandy: ack (hoping it will be by tomorrow morning ;)) | 14:59 |
| rlandy | yep - there's hoping | 15:00 |
| rlandy | marios|ruck: also asked soniya to continue component line check | 15:00 |
| marios|ruck | rlandy: sec have to re-nick to chekc something with the chan (https://review.opendev.org/c/openstack/project-config/+/834199 merged) | 15:00 |
| *** marios|ruck is now known as marios | 15:00 | |
| *** ChanServ sets mode: +o marios | 15:02 | |
| *** dviroel is now known as dviroel_ | 15:04 | |
| *** dviroel_ is now known as dviroel | 15:04 | |
| *** marios changes topic to "tripleo-ci || rr status: https://hackmd.io/mXb6zsGsTNWWO6Y6UpUNAg?view || https://docs.openstack.org/tripleo-docs/latest/ci/index.html https://docs.openstack.org/tripleo-quickstart/latest/" | 15:10 | |
| *** marios is now known as marios|ruck | 15:10 | |
| rlandy | marios|ruck: pls ping when you are eod and what you want watched in my afternoon | 15:11 |
| marios|ruck | rlandy: ack will do | 15:16 |
| ade_lee | rlandy, dviroel, dasm hey - can you guys point me to the script in tripleo-ci where you set up the ipa server? | 15:29 |
| dviroel | ade_lee: which job? multinode-ipa? | 15:30 |
| rlandy | ade_lee: there are two ... | 15:30 |
| ade_lee | dviroel, sure -- do we do it differently for both? | 15:30 |
| rlandy | https://github.com/openstack/tripleo-quickstart-extras/tree/master/roles/ipa-multinode/tasks | 15:31 |
| rlandy | https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/ipa-multinode/tasks/ipaserver-subnode-install.yml | 15:31 |
| rlandy | ^^ for the multinode-ipa job and then for fs039 ... | 15:31 |
| rlandy | https://github.com/openstack/tripleo-quickstart-extras/tree/master/roles/freeipa-setup | 15:32 |
| rlandy | ade_lee: ^^ those two | 15:32 |
| ade_lee | rlandy, thanks! | 15:32 |
| *** dviroel is now known as dviroel|lunch | 15:44 | |
| *** amoralej is now known as amoralej|off | 15:50 | |
| marios|ruck | rlandy: am off in a few ... nothing major at the moment - all latest on the hackmd .. couple testprojects there you may want to check/rekick later? https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/835815/ in gate (if folks complain about broken c8 content providers need that) ... ci gate should be unblocked (merged @ https://review.opendev.org/c/openstack/tripleo-ci/+/835776 ) | 16:13 |
| rlandy | marios|ruck: sounds good | 16:14 |
| rlandy | I rekicked some failing component jobs | 16:14 |
| rlandy | so hopefully we will clear up | 16:14 |
| rlandy | watching for downstream news | 16:14 |
| marios|ruck | rlandy: we managed a day without filing a new bug \o/ | 16:14 |
| rlandy | marios|ruck: have a good night | 16:14 |
| rlandy | marios|ruck: woohoo | 16:14 |
| marios|ruck | ;) | 16:14 |
| rlandy | we filed enough | 16:14 |
| rlandy | lunch - brb | 16:15 |
| *** marios|ruck is now known as marios|out | 16:20 | |
| *** jpena is now known as jpena|off | 16:28 | |
| *** dviroel|lunch is now known as dviroel | 17:00 | |
| rlandy | dasm: rcastillo: need anything from me atm? | 17:15 |
| dasm | rlandy: i'm good. thanks | 17:15 |
| rcastillo | rlandy: I'm doing fine, thanks | 17:51 |
| rlandy | k good | 17:51 |
| dasm | dviroel: do you know where "release" is getting injected? I see errors about missing var in my patch: https://review.rdoproject.org/r/c/config/+/41127 I followed similar path to: https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/infra-setup/roles/get_hash/tasks/get_hash.yaml#L12 | 20:36 |
| dasm | I'd like to understand what I'm missing here | 20:36 |
| dviroel | dasm: in the release file, lol | 20:44 |
| dviroel | dasm: must get from branch somewhere | 20:46 |
| dasm | what do you mean by release file? | 20:48 |
| dasm | the most probably I'm still misunderstanding dependencies here | 20:49 |
| dasm | dviroel: ^ | 20:49 |
| dasm | otherwise, no worries, i'm gonna check that later too | 20:49 |
| dviroel | dasm: for repo-setup role we always need to load vars from release file (https://github.com/openstack/tripleo-quickstart/blob/master/config/release/tripleo-ci/CentOS-9/master.yml) which has all repo vars needed | 20:58 |
| dviroel | dasm: but the release itself might be defined based on branch info | 20:58 |
| dviroel | dasm: and at the point that ovb-manage is being called, you still don't have release set | 20:59 |
| dviroel | you might want to try get this info from zuul | 20:59 |
| dviroel | zuul.branch ? | 21:01 |
| dviroel | dasm: we use set_ci_branch in many places, you might want to take a look https://opendev.org/openstack/tripleo-ci/src/branch/master/roles/build-containers/tasks/set_ci_branch.yml#L4 | 21:08 |
| rcastillo | dasm: would it be better to condition it on c8 and c9, since we really only want uefi on master and w? | 21:08 |
| * dviroel out o/ | 21:16 | |
| *** dviroel is now known as dviroel|out | 21:16 | |
| dasm | rlandy: we need to test wallaby as well. at least for one scenario | 21:16 |
| dasm | dviroel|out: i'll check that. thanks | 21:17 |
| dasm | rcastillo: ^ | 21:17 |
| dasm | sorry rlandy | 21:17 |
| dasm | rcastillo: i was thinking about making it relying on openstack releases instead of centos releases/ | 21:18 |
| rlandy | np | 21:19 |
| rlandy | we need wallaby c8 and c9 on uefi boot as default | 21:19 |
| rlandy | and one test for w c8 on bios | 21:20 |
| dasm | yep | 21:20 |
| dasm | i was thinking about scenario001 for bare minimum (iirc it's ovb) for wallaby with legacy bios and others for uefi | 21:21 |
| rlandy | woohoo "tripleowallabycentos9" promoting | 21:25 |
| dasm | neat | 21:37 |
| * dasm disappears into bushes | 21:43 | |
| dasm | https://www.youtube.com/watch?v=_lPJ9J-6vDw | 21:43 |
| *** dasm is now known as dasm|off | 21:43 | |
| *** rlandy is now known as rlandy|bbl | 22:11 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!