| arxcruz | rlandy|out it won't work, i put a patch up, i'll check tomorrow morning if it will work, added a testproject as well | 00:04 |
|---|---|---|
| arxcruz | rlandy|out https://review.rdoproject.org/r/c/testproject/+/39025 https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/827779 | 00:04 |
| dasm|off | arxcruz: nice, thanks! | 01:03 |
| *** ysandeep|out is now known as ysandeep | 01:14 | |
| *** dviroel|afk is now known as dviroel | 01:16 | |
| dviroel | ysandeep: early bird | 01:17 |
| ysandeep | dviroel, hey o/ yeah | 01:17 |
| ysandeep | My daughter woke me up early and then she went back to sleep but i can't :) | 01:18 |
| dviroel | ysandeep: yeah, happens | 01:18 |
| ysandeep | dviroel, what's the current time at your place? | 01:20 |
| dviroel | 10:20PM | 01:20 |
| ysandeep | :) go to sleep my friend | 01:21 |
| dviroel | ysandeep: just looking the outputs from some testproject - I updated the review list - going to take the kiddo to bed now | 01:21 |
| dviroel | ysandeep: after that, will be me | 01:21 |
| dviroel | ysandeep: see you later o/ | 01:21 |
| ysandeep | dviroel, gd night to you and your kiddo o/ | 01:22 |
| dviroel | ysandeep: tks bye | 01:22 |
| *** dviroel is now known as dviroel|out | 01:22 | |
| rlandy|out | arxcruz; thank you | 01:24 |
| rlandy|out | ysandeep: hey | 01:24 |
| rlandy|out | trigger jobs is good now | 01:24 |
| ysandeep | rlandy|out, we got a green run, nice | 01:24 |
| rlandy|out | 17 was looking good | 01:24 |
| rlandy|out | ysandeep: I think we're ok on amol's patch | 01:25 |
| rlandy|out | pls w+ if you agree | 01:25 |
| ysandeep | yes, looks from downstream side now | 01:25 |
| * ysandeep checking amol's patch now | 01:25 | |
| ysandeep | it had -1 last night from zuul, let me check current status | 01:26 |
| rlandy|out | ok - really going out now | 01:26 |
| rlandy|out | see you later | 01:26 |
| ysandeep | o/ bye | 01:27 |
| *** ysandeep is now known as ysandeep|afk | 01:40 | |
| *** akahat is now known as akahat|rover | 05:29 | |
| marios | o/ akahat|rover let me know if you need sthing else we can get on phone too | 06:54 |
| akahat|rover | marios, yeah. sure. I'll ping :) | 06:55 |
| akahat|rover | thank you :) | 06:55 |
| *** ysandeep|afk is now known as ysandeep | 07:09 | |
| *** amoralej|off is now known as amoralej | 07:28 | |
| ysandeep | akahat|rover: I have a promoter related query - if you are aware about that https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/308787/2#message-d76f89f2ba8c389c41bf217e77811fd4f23c9b07 | 08:25 |
| *** akahat|rover is now known as akahat|lunch | 08:30 | |
| *** ysandeep is now known as ysandeep|lunch | 08:36 | |
| frenzy_friday | hey folks, pls add to your review list: https://review.rdoproject.org/r/c/rdo-jobs/+/39031 | 08:55 |
| *** akahat|lunch is now known as akahat|rover | 09:22 | |
| akahat|rover | ysandeep|lunch, i've added comments here: https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/308787/2#message-1c9a529b6fb910a125fc182d062e44d197e31d69 | 09:37 |
| *** ysandeep|lunch is now known as ysandeep | 10:04 | |
| ysandeep | akahat|rover: ++ thanks for your comment, so we don't need to list them in config file as we are not doing checking what's mentioned in config vs what is in directory.. we promote everything in directory. | 10:05 |
| akahat|rover | ysandeep, yes. working on the fix for it. :) If we defined there then we should use it. | 10:06 |
| ysandeep | ack thanks akahat|rover | 10:07 |
| ysandeep | please feel free to work on this later if you have occupied with other rr work. | 10:08 |
| akahat|rover | ysandeep, tests to validate criteria and component jobs: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/38990 | 10:14 |
| ysandeep | akahat|rover: great idea +100, I will review in my next round | 10:16 |
| akahat|rover | ysandeep, ack | 10:16 |
| *** pojadhav is now known as pojadhav|brb | 10:24 | |
| *** chem is now known as Guest1752 | 10:45 | |
| *** dviroel|out is now known as dviroel|ruck | 10:46 | |
| *** beagles is now known as beagles_pto | 10:53 | |
| dviroel|ruck | arxcruz: hey, news on identity endpoint fix? | 11:02 |
| arxcruz | dviroel|ruck yes, working on a patch :) | 11:02 |
| dviroel|ruck | arxcruz: ah ok, was about to recheck - will wait | 11:02 |
| dviroel|ruck | arxcruz: :* | 11:02 |
| arxcruz | dviroel|ruck https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/827779 | 11:06 |
| arxcruz | still testing, i have a testproject, but the patch fail so i'm testing again | 11:06 |
| *** rlandy|out is now known as rlandy|ruck | 11:10 | |
| *** pojadhav|brb is now known as pojadhav | 11:14 | |
| *** ysandeep is now known as ysandeep|mtg | 11:17 | |
| marios | akahat|rover: please check c8 master looks veeery red integration line | 11:19 |
| dviroel|ruck | marios: looking nw | 11:20 |
| dviroel|ruck | akahat|rover: "Error running container image prepare: [tripleomaster/openstack-nova-conductor] Write Failure: HTTPSConnectionPool(host='trunk.registry.rdoproject.org', port=443): Read timed out.", "params": {}, "success": false} | 11:20 |
| rlandy|ruck | marios: ysandeep|mtg: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/38823/3/ci-scripts/dlrnapi_promoter/config/CentOS-9/component/wallaby.yaml | 11:20 |
| marios | akahat|rover: running now openstack-periodic-integration-main-centos8 1 https://review.rdoproject.org/zuul/status | 11:21 |
| akahat|rover | marios, ack | 11:23 |
| akahat|rover | dviroel|ruck, where you seen that logs? | 11:26 |
| rlandy|ruck | https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/39012 | 11:26 |
| rlandy|ruck | marios: ^^ | 11:26 |
| ysandeep|mtg | rlandy|ruck, marios https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/308841 | 11:27 |
| dviroel|ruck | akahat|rover: standalone jobs - standalone install | 11:27 |
| dviroel|ruck | akahat|rover: on any job openstack-periodic-integration-main-centos8 | 11:28 |
| marios | rlandy|ruck: akahat|rover: let me know if you want me re join ;) | 11:29 |
| dviroel|ruck | akahat|rover: happened ~ 2022-02-04 07:56:11 | 11:30 |
| akahat|rover | dviroel|ruck, it says write failure.. trunk.registry.rdoproject.org | 11:31 |
| dviroel|ruck | akahat|rover: there are green jobs on component lines, that are more recent | 11:32 |
| dviroel|ruck | akahat|rover: we should trigger again and watch | 11:34 |
| akahat|rover | dviroel|ruck, okay | 11:36 |
| * akahat|rover brb | 11:44 | |
| ysandeep|mtg | rlandy|ruck, https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/infra-setup/roles/get_hash/tasks/get_hashes_from_commit_yaml.yaml#L4 | 11:49 |
| ysandeep|mtg | rlandy|ruck, https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/infra-setup/roles/get_hash/defaults/main.yml#L44 | 11:51 |
| rlandy|ruck | ysandeep|mtg: https://osp-trunk.hosted.upshift.rdu2.redhat.com/rhel9-osp17/consistent/b6/0c/b60cf0b607d0ecd2e6347fe0c0e45ab5/delorean.repo | 11:59 |
| *** ysandeep|mtg is now known as ysandeep | 12:02 | |
| rlandy|ruck | dviroel|ruck: akahat|rover: he - do you want to ruck/rover sync or doing ok? | 12:02 |
| dviroel|ruck | rlandy|ruck: lets quick sync | 12:03 |
| rlandy|ruck | dviroel|ruck: akahat|rover: https://meet.google.com/nkz-qsny-ffi?pli=1&authuser=0 | 12:03 |
| dviroel|ruck | akahat|rover: around? | 12:04 |
| akahat|rover | dviroel|ruck, joining | 12:04 |
| rlandy|ruck | dviroel|ruck: ysandeep: you guys both have CIX's on scenario007 | 12:26 |
| rlandy|ruck | related? | 12:26 |
| ysandeep | checking, should not be related in upstream we set net.ipv6.conf.lo.disable_ipv6 = 0 in image | 12:27 |
| dviroel|ruck | rlandy|ruck: let me see, my cix is fixed already | 12:27 |
| ysandeep | rlandy|ruck, dviroel|ruck not related | 12:28 |
| dviroel|ruck | yeah, not related | 12:28 |
| rlandy|ruck | ok - just cleaning up CIX board | 12:28 |
| ysandeep | rlandy|ruck, for my bug i think we are missing https://softwarefactory-project.io/r/c/config/+/23153 in our downstream rhel9 image | 12:28 |
| * ysandeep and pojadhav testing | 12:29 | |
| rlandy|ruck | ysandeep: pojadhav: ok - did you proposed a change to the images or you want me to? | 12:32 |
| ysandeep | rlandy|ruck, i will, testing if movin to net.ipv6.conf.lo.disable_ipv6 = 0 helps | 12:33 |
| ysandeep | moving* | 12:33 |
| rlandy|ruck | k - let me know | 12:33 |
| rlandy|ruck | amoralej++ ... https://jenkins-cloudsig-ci.apps.ocp.ci.centos.org/job/tripleo-quickstart-promote-master-centos9-current-tripleo-delorean-minimal/15/consoleFull actually started | 12:35 |
| rlandy|ruck | that's a step forward | 12:35 |
| rlandy|ruck | ysandeep: marios: ^^ we got a started c9 job on cico | 12:36 |
| amoralej | rlandy|ruck, yes, i sent a new patch | 12:36 |
| marios | rlandy|ruck: thanks | 12:36 |
| rlandy|ruck | woohoo | 12:36 |
| amoralej | rlandy|ruck, https://review.opendev.org/c/openstack/tripleo-quickstart/+/827828 | 12:36 |
| rlandy|ruck | appears to be in '/tmp/workspace/tripleo-quickstart-promote-master-centos9-current-tripleo-delorean-minimal/tripleo-quickstart/roles/parts/libvirt/tasks/main.yml': line 18, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n# different list of packages.\n- name: Install packages for libvirt\n ^ here\n"} | 12:36 |
| rlandy|ruck | yeah - I see | 12:36 |
| amoralej | i hope that will move it more ahead | 12:36 |
| amoralej | i tested locally and reach the point where it breaks because of opstools url, so i think it's the right track :) | 12:37 |
| rlandy|ruck | amoralej: great - let's try merge that | 12:37 |
| rlandy|ruck | and run again | 12:37 |
| rlandy|ruck | dviroel|ruck: ysandeep: pls review vote for ^^ https://review.opendev.org/c/openstack/tripleo-quickstart/+/827828 | 12:38 |
| amoralej | btw, in the other front, we got a weirdo job passing and the other needs promotions | 12:38 |
| rlandy|ruck | at least we have a jobs starting | 12:38 |
| rlandy|ruck | amoralej: yep - heard from akahat|rover | 12:38 |
| rlandy|ruck | will watch those on next promotion | 12:38 |
| ysandeep | rlandy|ruck, lgtm, voted | 12:40 |
| * akahat|rover back in few hours | 12:54 | |
| rlandy|ruck | arxcruz: hello ... thanks for picking up the tempest endpoint issue ... | 13:12 |
| rlandy|ruck | pls let us know when you are EoD | 13:12 |
| rlandy|ruck | so ruck/rover can pick up | 13:12 |
| arxcruz | rlandy|ruck ack | 13:12 |
| dviroel|ruck | rlandy|ruck: akahat|rover https://review.rdoproject.org/zuul/buildset/a9b88b4436974b2c9c6e45525addf6b8 | 13:13 |
| dviroel|ruck | not sure what happened there | 13:13 |
| rlandy|ruck | dviroel|ruck: so it died on container push? | 13:13 |
| * rlandy|ruck looks | 13:14 | |
| dviroel|ruck | rlandy|ruck: i see a failure in post-run only | 13:14 |
| rlandy|ruck | there is no run | 13:15 |
| rlandy|ruck | 2022-02-04 12:46:21.510835 | PRE-RUN END RESULT_NORMAL: [untrusted : opendev.org/openstack/tripleo-ci/playbooks/tripleo-buildcontainers/pre.yaml@master] | 13:15 |
| rlandy|ruck | 2022-02-04 12:46:21.557337 | POST-RUN START: [trusted : review.rdoproject.org/config/playbooks/tripleo-ci-periodic-base/post.yaml@master] | 13:15 |
| dviroel|ruck | (´･_･`) | 13:16 |
| rlandy|ruck | dviroel|ruck: ok - so ... | 13:16 |
| rlandy|ruck | we need to start a node | 13:16 |
| rlandy|ruck | hold it and see what's dying | 13:16 |
| rlandy|ruck | my first guess is registry login | 13:16 |
| rlandy|ruck | 2022-02-04 12:46:20.496827 | primary | ok: 44 changed: 19 unreachable: 0 failed: 0 skipped: 26 rescued: 0 ignored: 0 | 13:17 |
| rlandy|ruck | nothing failed | 13:17 |
| rlandy|ruck | so maybe there was no host to run the RUN playbook | 13:18 |
| rlandy|ruck | the order is like ... | 13:18 |
| rlandy|ruck | compare to: https://logserver.rdoproject.org/openstack-periodic-integration-main-centos8/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-build-containers-ubi-8-push/76636f5/job-output.txt | 13:19 |
| rlandy|ruck | 2022-02-04 05:15:08.826660 | PRE-RUN END RESULT_NORMAL: [untrusted : opendev.org/openstack/tripleo-ci/playbooks/tripleo-buildcontainers/pre.yaml@master] | 13:20 |
| rlandy|ruck | 2022-02-04 05:15:08.867182 | PRE-RUN START: [trusted : review.rdoproject.org/config/playbooks/tripleo-rdo-base/pre.yaml@master] | 13:20 |
| rlandy|ruck | ^^ right order | 13:20 |
| rlandy|ruck | 2022-02-04 12:46:19.886127 | TASK [Configure tripleo registry] | 13:21 |
| rlandy|ruck | 2022-02-04 12:46:20.165991 | primary | ok | 13:21 |
| rlandy|ruck | diff is there | 13:21 |
| dviroel|ruck | yes | 13:22 |
| dviroel|ruck | missing from failed one | 13:23 |
| rlandy|ruck | dviroel|ruck: finding the line | 13:23 |
| rlandy|ruck | in code | 13:23 |
| rlandy|ruck | something up with registry | 13:23 |
| rlandy|ruck | maybe master c8 got deleted | 13:23 |
| rlandy|ruck | https://trunk.registry.rdoproject.org:8443/oapi/v1/namespaces/tripleomaster/imagestreamtags/ | 13:24 |
| rlandy|ruck | exists | 13:24 |
| rlandy|ruck | https://opendev.org/openstack/tripleo-ansible/src/branch/master/tripleo_ansible/roles/tripleo_image_serve/ | 13:28 |
| rlandy|ruck | ^^ failure | 13:28 |
| rlandy|ruck | https://opendev.org/openstack/tripleo-ansible/commit/8378ac7ae4f21dc99347a432df878d911ec8efc4 | 13:29 |
| rlandy|ruck | maybe? | 13:29 |
| dviroel|ruck | not sure | 13:30 |
| dviroel|ruck | rlandy|ruck: running testproject | 13:31 |
| rlandy|ruck | Tengu: hey | 13:31 |
| Tengu | rlandy|ruck: heya | 13:31 |
| rlandy|ruck | ^^ blame points at you :) | 13:31 |
| Tengu | what? | 13:31 |
| rlandy|ruck | Tengu: so :) | 13:31 |
| rlandy|ruck | our container build jobs on c8 master are dying | 13:32 |
| rlandy|ruck | with no trace | 13:32 |
| Tengu | uho | 13:32 |
| rlandy|ruck | it looks like the role tripleo_image_serve never runs | 13:32 |
| rlandy|ruck | only master | 13:32 |
| rlandy|ruck | only c8 | 13:32 |
| Tengu | o_O | 13:32 |
| rlandy|ruck | comparing with a passing log, that looks like the difference | 13:33 |
| rlandy|ruck | I see your patch from yesterday ... | 13:33 |
| rlandy|ruck | do we need to do something to make our jobs work with that | 13:33 |
| Tengu | it's in master only? | 13:33 |
| rlandy|ruck | it looks like we flat skip it | 13:33 |
| rlandy|ruck | yeah - master only, right, dviroel|ruck? | 13:33 |
| Tengu | I don't see why it would be skipped... | 13:34 |
| rlandy|ruck | or maybe dies silently? | 13:34 |
| Tengu | even that... | 13:35 |
| rlandy|ruck | I am guessing here atm | 13:35 |
| Tengu | care to point to a job? | 13:35 |
| rlandy|ruck | sure ... pls compare logs ... | 13:35 |
| dviroel|ruck | rlandy|ruck: yeah, so far | 13:35 |
| rlandy|ruck | https://logserver.rdoproject.org/openstack-periodic-integration-main-centos8/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-build-containers-ubi-8-push/76636f5/job-output.txt | 13:35 |
| rlandy|ruck | ^^ passing | 13:35 |
| rlandy|ruck | https://logserver.rdoproject.org/openstack-periodic-integration-main-centos8/opendev.org/openstack/tripleo-ci/e05a34f546c9f25321f2a884fd16f84aa87cac3b/periodic-tripleo-ci-build-containers-ubi-8-push/0639d26/job-output.txt | 13:35 |
| rlandy|ruck | ^^ failing | 13:35 |
| rlandy|ruck | no obvious error | 13:36 |
| *** dasm|off is now known as dasm | 13:36 | |
| rlandy|ruck | just ... | 13:36 |
| rlandy|ruck | 2022-02-04 12:46:19.886127 | TASK [Configure tripleo registry] | 13:36 |
| rlandy|ruck | 2022-02-04 12:46:20.165991 | primary | ok | 13:36 |
| rlandy|ruck | 2022-02-04 12:46:20.323144 | | 13:36 |
| rlandy|ruck | 2022-02-04 12:46:20.323352 | TASK [bindep : Remove bindep temp dir] | 13:36 |
| rlandy|ruck | in failing | 13:36 |
| dasm | o/ | 13:36 |
| rlandy|ruck | and | 13:36 |
| rlandy|ruck | 2022-02-04 05:15:01.159987 | TASK [Configure tripleo registry] | 13:36 |
| rlandy|ruck | 2022-02-04 05:15:01.223218 | primary | ok | 13:36 |
| rlandy|ruck | 2022-02-04 05:15:01.274443 | | 13:36 |
| rlandy|ruck | 2022-02-04 05:15:01.274558 | TASK [tripleo_image_serve : ensure apache is installed] | 13:36 |
| rlandy|ruck | 2022-02-04 05:15:03.270318 | primary | ok: Nothing to do | 13:36 |
| rlandy|ruck | 2022-02-04 05:15:03.324262 | | 13:36 |
| rlandy|ruck | in passing | 13:36 |
| rlandy|ruck | hey dasm - good morning | 13:36 |
| Tengu | seems to be failing before anything. | 13:36 |
| Tengu | https://paste.openstack.org/show/812523/ | 13:36 |
| Tengu | that's apparently the failure? | 13:36 |
| Tengu | rlandy|ruck: -^^ | 13:37 |
| dasm | good morning rlandy|ruck | 13:37 |
| Tengu | yep. that's the issue. | 13:38 |
| Tengu | rlandy|ruck: so the playbook fails due to what I pasted, and doesn't reach the image-serve things. | 13:38 |
| Tengu | it's pretty early in the run. | 13:38 |
| rlandy|ruck | so you're right that the dlrn failure is the obvious error .... | 13:39 |
| rlandy|ruck | but it errors because the 'run' part never happens | 13:39 |
| rlandy|ruck | dviroel|ruck: ok ... we need to hold a node here | 13:39 |
| rlandy|ruck | get on the node | 13:39 |
| rlandy|ruck | and try these commands one by one | 13:39 |
| rlandy|ruck | until we see what happens | 13:40 |
| rlandy|ruck | Tengu: thanks - we will get back to you | 13:40 |
| dviroel|ruck | rlandy|ruck: https://review.rdoproject.org/zuul/status#36356 | 13:40 |
| Tengu | rlandy|ruck: I'm pretty sure my change isn't a cause of silent failure. if really, it would spit a trace at some point | 13:40 |
| dviroel|ruck | rlandy|ruck: hold in place | 13:41 |
| rlandy|ruck | Tengu: k - it was just the only change on that role so started here | 13:41 |
| rlandy|ruck | keys on that node? | 13:41 |
| Tengu | sure :). but.... nope. I'd be really surprised if it was the issue. | 13:41 |
| rlandy|ruck | idk if that is one of those jobs where we gets keys | 13:41 |
| Tengu | rlandy|ruck: seems like the containers aren't even built. | 13:47 |
| Tengu | but I don't find build logs in the tree. | 13:47 |
| Tengu | https://logserver.rdoproject.org/openstack-periodic-integration-main-centos8/opendev.org/openstack/tripleo-ci/e05a34f546c9f25321f2a884fd16f84aa87cac3b/periodic-tripleo-ci-build-containers-ubi-8-push/0639d26/logs/containers-built.log is empty | 13:47 |
| rlandy|ruck | yep | 13:50 |
| rlandy|ruck | it never starts | 13:50 |
| *** pojadhav- is now known as pojadhav | 13:53 | |
| rlandy|ruck | 2022-02-04 13:48:40.842334 | TASK [Configure tripleo registry] | 13:54 |
| rlandy|ruck | 2022-02-04 13:48:41.012332 | primary | ok | 13:54 |
| rlandy|ruck | w/o any trace of running anything | 13:54 |
| Tengu | rlandy|ruck: do you want to try a revert and depends-on the revert? | 13:54 |
| rlandy|ruck | let's try run the role on the node | 13:54 |
| rlandy|ruck | and see what happens | 13:55 |
| rlandy|ruck | maybe I'm completely off base here | 13:55 |
| Tengu | rlandy|ruck: my patch was merged today at 12:22 | 13:55 |
| Tengu | probably not promoted yet - would it end in the CI directly? | 13:55 |
| rlandy|ruck | Tengu: tripleo-ansible should come from current | 13:56 |
| Tengu | not to mention - it passed the content-provider runs in CI before merging | 13:56 |
| rlandy|ruck | yeah | 13:56 |
| Tengu | hmmm current being the closest to the merge right? | 13:56 |
| rlandy|ruck | yep | 13:56 |
| rlandy|ruck | dviroel|ruck: need to get on meeting now | 13:56 |
| Tengu | ok. so yep. it's probably used then. Still... the change passed the jobs :/ | 13:56 |
| rlandy|ruck | let me know if you can get on the node and run that role | 13:57 |
| rlandy|ruck | it may be the registry itself | 13:57 |
| rlandy|ruck | we don't know yet | 13:57 |
| dviroel|ruck | rlandy|ruck: ack | 13:57 |
| rlandy|ruck | if we're desperate, we'll try a revert | 13:57 |
| ykarel | community.general.seport looks suspecious | 13:57 |
| ykarel | considering ansible version | 13:58 |
| Tengu | no mention of it in the log :/ | 13:58 |
| ykarel | yes zuul likely hiding it :( | 13:58 |
| Tengu | (I mean - no seport at all in there) | 13:58 |
| * Tengu hurts zuul with an extra large camel | 13:58 | |
| Tengu | though, again: https://review.opendev.org/c/openstack/tripleo-ansible/+/824524 is green all the way, also for tripleo-ci-centos-8-content-provider | 13:59 |
| *** ysandeep is now known as ysandeep|mtg | 13:59 | |
| ykarel | content-provider runs differently | 13:59 |
| ykarel | so that part is not tested | 13:59 |
| Tengu | .... | 13:59 |
| Tengu | there are other community.general in there. | 14:00 |
| Tengu | well. one actually. | 14:00 |
| rlandy|ruck | lol "Tengu hurts zuul with an extra large camel" | 14:00 |
| ykarel | Tengu, in build-containers? | 14:00 |
| Tengu | ykarel: nope, in tripleo_ansible/roles/tripleo_frr/tasks/main.yml | 14:00 |
| ykarel | Tengu, ack that's different | 14:00 |
| Tengu | though it's in tripleo_ansible/requirements.yml as well | 14:01 |
| Tengu | tripleo_ansible/requirements.yml: - community.general | 14:01 |
| ykarel | so whatever not triggerd by zuul ansible is different | 14:01 |
| rlandy|ruck | meeting - will check back later | 14:01 |
| Tengu | "great". so it may be it after all, but since zull is masking things, it will require some manual debugging | 14:01 |
| Tengu | and since CI didn't catch it, it's even more fun. | 14:02 |
| ykarel | so corresponding ci job not running, so that's coverage related | 14:02 |
| ykarel | and yes zuul should have given error on what's wrong | 14:02 |
| Tengu | though we probably don't want to build containers on every image-serve change. | 14:02 |
| Tengu | but zuul (probably) masking things is a bigger issue imho. | 14:03 |
| ykarel | yes agree, but if the job ran in that patch issue would have caught | 14:04 |
| ykarel | to fix need something that works with ansible version used in zuul executor | 14:05 |
| ykarel | i think i can check zuul logs, there it should be showing the error | 14:06 |
| Tengu | while working with what is present on the Director/Undercloud node. | 14:06 |
| ykarel | yes there ansible is higher version | 14:06 |
| Tengu | since this role is used during the UC deploy. | 14:06 |
| ykarel | yeap that's different workflow, different ansible version | 14:06 |
| Tengu | would be too easy otherwise. | 14:06 |
| * anbanerj going afk in some time. Need to collect some documents for visa | 14:08 | |
| Tengu | rlandy|ruck: I'm about to drop - if that patch is the issue, we can revert it I guess. I won't have direct effect on tripleo deploy. | 14:11 |
| ykarel | Tengu, yes confirmed that caused it | 14:12 |
| ykarel | from zuul logs https://paste.opendev.org/show/812525/ | 14:12 |
| Tengu | ykarel: pfrrtt.. bad zuul. | 14:12 |
| Tengu | ykarel, rlandy|ruck https://review.opendev.org/c/openstack/tripleo-ansible/+/827748 | 14:13 |
| Tengu | zuul-- | 14:13 |
| rlandy|ruck | Tengu: thank you | 14:14 |
| Tengu | rlandy|ruck: sorry for that one :/. | 14:14 |
| rlandy|ruck | dviroel|ruck: ^^ can you testproject that | 14:14 |
| rlandy|ruck | Tengu: no worries | 14:14 |
| dviroel|ruck | ykarel: thanks for debugging o/ | 14:15 |
| dviroel|ruck | rlandy|ruck: sure | 14:15 |
| Tengu | guess it's time to sign off before crashing other things :) | 14:20 |
| rlandy|ruck | Tengu: have a good weekend | 14:38 |
| amoralej | marios, fyi opstools repo should be fine https://review.rdoproject.org/r/c/rdo-infra/ansible-role-dlrn/+/39069 but i think i will merge it on monday morning | 14:45 |
| amoralej | it's also mirrored in AFS | 14:45 |
| marios | amoralej: thanks yeah next week is good we can revert the release file stuff | 14:46 |
| marios | thanks | 14:46 |
| *** ysandeep|mtg is now known as ysandeep|out | 14:53 | |
| dviroel|ruck | node provision errors again? https://logserver.rdoproject.org/25/39025/2/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-network-master/2a0e227/ | 15:10 |
| dviroel|ruck | (╯°□°)╯︵ ┻━┻ | 15:10 |
| * dviroel|ruck lunch | 15:17 | |
| *** dviroel|ruck is now known as dviroel|ruck|lunch | 15:17 | |
| rlandy|ruck | dviroel|ruck|lunch: ack dasm logged that bug | 15:31 |
| dasm | dviroel|ruck|lunch: https://bugs.launchpad.net/tripleo/+bug/1959914 | 15:33 |
| rlandy|ruck | periodic-tripleo-ci-build-containers-ubi-8-pushtestprojectmastercheck36489,16 | 15:45 |
| rlandy|ruck | 1 hr 3 mins 38 secs2022-02-04 14:40:42 | 15:45 |
| rlandy|ruck | success there | 15:45 |
| *** akahat|rover is now known as akahat|dinner | 15:45 | |
| rlandy|ruck | https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/827779 | 15:47 |
| rlandy|ruck | looking good to go? | 15:47 |
| *** dviroel|ruck|lunch is now known as dviroel|ruck | 16:22 | |
| marios | rlandy|ruck: can you add this into your list to start staring at its quite a big patch https://review.opendev.org/c/openstack/tripleo-ci/+/826317/15#message-7394cf1316a2cdecf6730c89d035116faa784297 | 16:23 |
| rlandy|ruck | marios: oh wow - ok :) | 16:24 |
| dviroel|ruck | rlandy|ruck: 1. the revert solves container-buils in c8 | 16:24 |
| marios | rlandy|ruck: maybe merge the depends-on (fake nodeset please?) | 16:24 |
| dviroel|ruck | rlandy|ruck: 2. arxcruz is testing it - no successful runs yet | 16:24 |
| marios | rlandy|ruck: and see topic branch for layout updates & tests https://review.opendev.org/q/topic:c8_teardown_master (eg. still problem there https://review.opendev.org/c/openstack/python-tripleoclient/+/827858/1#message-f281b7608cb728cc5bb4c715f55adfd77ead8df0 | 16:25 |
| arxcruz | rlandy|ruck dviroel|ruck yup one of the jobs are finishing right now | 16:25 |
| dviroel|ruck | arxcruz: failed too in tempest tests | 16:26 |
| rlandy|ruck | marios, https://review.rdoproject.org/r/c/rdo-jobs/+/39045 - one comment there | 16:26 |
| arxcruz | dviroel|ruck same issue ? | 16:26 |
| rlandy|ruck | then we can merge | 16:26 |
| dviroel|ruck | arxcruz: don't know, waiting for the logs | 16:26 |
| rlandy|ruck | dviroel|ruck; ok - let's get the revert merged | 16:26 |
| dviroel|ruck | rlandy|ruck: +1 | 16:26 |
| rlandy|ruck | marios: can you w+ before EoD | 16:26 |
| rlandy|ruck | https://review.opendev.org/c/openstack/tripleo-ansible/+/827748 | 16:27 |
| rlandy|ruck | marios: ^^ pretty pls for us | 16:27 |
| marios | rlandy|ruck: see https://review.rdoproject.org/r/c/rdo-jobs/+/39045/1#message-34d8bf042075bafb418712b11b354f80159c5e27 there is already a comment there | 16:27 |
| marios | rlandy|ruck: checking | 16:27 |
| rlandy|ruck | marios: ak - ok (scroll up) | 16:28 |
| rlandy|ruck | merging | 16:28 |
| rlandy|ruck | done | 16:28 |
| rlandy|ruck | dviroel|ruck: arxcruz: ok - let's review when tests finish | 16:29 |
| dviroel|ruck | rlandy|ruck: will investigate if node provision on c8 is only a network component issue, not sure if failed somewhere else | 16:30 |
| marios | rlandy|ruck: thanks | 16:30 |
| rlandy|ruck | dviroel|ruck: ok ... need to run out for a few minutes ... biab | 16:30 |
| *** rlandy|ruck is now known as rlandy|ruck|brb | 16:30 | |
| arxcruz | rlandy|ruck|brb dviroel|ruck so, the endpoint was set properly | 16:30 |
| arxcruz | BUT... | 16:31 |
| arxcruz | hmmm... interesting, let me check something | 16:31 |
| dviroel|ruck | arxcruz: errors are bit different now | 16:32 |
| arxcruz | dviroel|ruck yeah... and less error i think | 16:33 |
| dviroel|ruck | only keystone_tempest_plugin.tests.api.identity.v3 is failing now | 16:33 |
| dviroel|ruck | but not all | 16:33 |
| arxcruz | dviroel|ruck well, the patch works as it adds the proper config on identity | 16:35 |
| dviroel|ruck | arxcruz: yes, we made some progress | 16:36 |
| *** marios is now known as marios|out | 16:38 | |
| *** rlandy|ruck|brb is now known as rlandy|ruck | 16:54 | |
| rlandy|ruck | arxcruz: worth merging? | 16:54 |
| rlandy|ruck | some fixes | 16:54 |
| dviroel|ruck | rlandy|ruck: some clients are still pointing to adminURL: https://github.com/openstack/keystone-tempest-plugin/blob/f57af91f9a97a822d2deee1fd32857ad4571d210/keystone_tempest_plugin/services/identity/clients.py#L39 | 16:57 |
| dviroel|ruck | for those tests, we continue failing in the same way | 16:58 |
| rlandy|ruck | sec 0 just rekicking downstream | 16:59 |
| rlandy|ruck | dviroel|ruck: sorry - reading back | 17:02 |
| arxcruz | rlandy|ruck dviroel|ruck i think the patch worth merge because it is required anyway to set to publicURL | 17:02 |
| arxcruz | rlandy|ruck the failing tests, have adminURL hardcoded | 17:03 |
| arxcruz | so even if we set conf.identity.v3_endpoint | 17:03 |
| arxcruz | to public | 17:03 |
| rlandy|ruck | arxcruz: dviroel|ruck: so iiuc ... | 17:03 |
| arxcruz | it will still fail | 17:03 |
| rlandy|ruck | 1. merge arx's patch | 17:03 |
| rlandy|ruck | 2. search for hardcoded adminURL and change | 17:03 |
| rlandy|ruck | right? | 17:04 |
| arxcruz | rlandy|ruck yes | 17:04 |
| arxcruz | so, half of the problem is already fixed | 17:04 |
| rlandy|ruck | ok ... | 17:04 |
| dviroel|ruck | rlandy|ruck: yeah, who can we ask from keystone? | 17:04 |
| arxcruz | now need to see with keystone guys, if the federation tests are supposed to run only in adminURL | 17:04 |
| rlandy|ruck | dviroel|ruck: we need to ping harry - someone on sec should know | 17:05 |
| arxcruz | because basically, the are creating a class hardcoded adminURL for federation tests | 17:05 |
| rlandy|ruck | or douglas (other one) | 17:05 |
| dviroel|ruck | rlandy|ruck: dmendiza? | 17:05 |
| rlandy|ruck | dviroel|ruck: ack - ask him if he knows | 17:06 |
| rlandy|ruck | or dwilde | 17:06 |
| rlandy|ruck | one of those guys | 17:06 |
| rlandy|ruck | https://rover.redhat.com/people/profile/hrybacki | 17:07 |
| rlandy|ruck | ^^ pick a victim from here | 17:07 |
| arxcruz | dviroel|ruck maybe hrybacki | 17:07 |
| rlandy|ruck | dviroel|ruck: you can reach them on #tripleo or internal | 17:07 |
| arxcruz | he's a nice guy | 17:07 |
| rlandy|ruck | dviroel|ruck: join #sec-interop | 17:07 |
| rlandy|ruck | on internal | 17:08 |
| rlandy|ruck | arxcruz: ^^ you can find them all there | 17:08 |
| rlandy|ruck | Caesar: dwilde | 17:08 |
| rlandy|ruck | ^^ start there | 17:08 |
| dviroel|ruck | rlandy|ruck: ack | 17:09 |
| *** amoralej is now known as amoralej|off | 17:41 | |
| dviroel|ruck | rlandy|ruck: https://review.opendev.org/c/openstack/keystone-tempest-plugin/+/827910 | 17:49 |
| dviroel|ruck | lets wait a bit if does't break keystone CI :) | 17:50 |
| *** akahat|dinner is now known as akahat|rover | 17:59 | |
| rlandy|ruck | dviroel|ruck: sorry - long meeting | 18:58 |
| rlandy|ruck | reading back | 18:58 |
| rlandy|ruck | looks like amol's patch will mereg | 18:59 |
| rlandy|ruck | dviroel|ruck: need anything from me? | 19:00 |
| rlandy|ruck | hmmm ... | 19:01 |
| rlandy|ruck | I think donwstream is stuck | 19:01 |
| dviroel|ruck | rlandy|ruck: no, testing with keystone-tempest-plugin + arx fix | 19:03 |
| dviroel|ruck | rlandy|ruck: master c8 still waiting for revert to merge | 19:03 |
| rlandy|ruck | dviroel|ruck: k - let me know where we are EoD | 19:03 |
| dviroel|ruck | rlandy|ruck: triggered some integration jobs that failed | 19:03 |
| rlandy|ruck | will check in on sunday | 19:03 |
| dviroel|ruck | rlandy|ruck: will do | 19:03 |
| rlandy|ruck | ok - now downstream is running again | 19:23 |
| * rlandy|ruck need tso merge sandeep's patch | 19:24 | |
| rlandy|ruck | after amol's merges | 19:24 |
| rlandy|ruck | hmm ... I think we may be out of space on images dir | 19:59 |
| dviroel|ruck | we should automate deletion of old images | 20:03 |
| dviroel|ruck | or there is an script for that already? | 20:04 |
| rlandy|ruck | dviroel|ruck: I think the prune script is missing in c9 | 20:12 |
| rlandy|ruck | we need to tell infra about that | 20:12 |
| rlandy|ruck | just don't want to make them do it on friday afternoon | 20:12 |
| rlandy|ruck | we can ping them on monday morning to ask | 20:12 |
| dviroel|ruck | ahg ok | 20:14 |
| rlandy|ruck | dviroel|ruck: ok - I just nuke a bunch of old image dir | 20:17 |
| rlandy|ruck | that should spring us for a bit | 20:17 |
| dviroel|ruck | \o/ | 20:22 |
| rlandy|ruck | dasm: nice ... you got a uefi boot job going | 20:25 |
| rlandy|ruck | cool beans! | 20:25 |
| jm1 | rlandy|ruck: devstack definitely needs some love. took hours to hunt down this bug... https://review.opendev.org/c/openstack/devstack/+/827918 | 20:35 |
| rlandy|ruck | jm1: wow - late for you | 20:35 |
| rlandy|ruck | nice one | 20:36 |
| rlandy|ruck | go have a weekend!! | 20:36 |
| jm1 | rlandy|ruck: devstack has so many roadblocks... it took most of this week. had to get it up and running before weekend, dont want to spent more time on it next week | 20:38 |
| rlandy|ruck | understand | 20:39 |
| rlandy|ruck | we really never touch it | 20:39 |
| rlandy|ruck | but the component devs do | 20:39 |
| jm1 | the last batch of problems did not show up in CI because it requires a reboot to expose them | 20:41 |
| jm1 | so everything is fine unless you date to reboot your vm | 20:41 |
| jm1 | -date +dare | 20:41 |
| rlandy|ruck | never reboot - golden law | 20:42 |
| rlandy|ruck | amol's patch is about to merge | 20:42 |
| jm1 | devstack still has the same kind of persistence bug for cinder, but at least there is a workaround (which is not documented) | 20:42 |
| rlandy|ruck | getting downstream patch in w+ | 20:42 |
| jm1 | never reboot... haha next power outage will come, for sure | 20:43 |
| * jm1 to tired to document the cinder workaround... out for today | 20:44 | |
| jm1 | @team: have a nice weekend | 20:45 |
| dviroel|ruck | jm1: o/ | 20:48 |
| dasm | rlandy|ruck: yeah. waiting for a full results | 20:58 |
| rlandy|ruck | just w+'ed sandeep's patch | 21:00 |
| rlandy|ruck | dviroel|ruck: what do you want to do about the keystone and tempest patches? | 21:05 |
| rlandy|ruck | merhe arx's or leave them both until next week | 21:05 |
| dviroel|ruck | rlandy|ruck: waiting tempest results, after that we can merge arx's patch and ask reviews on keystone | 21:07 |
| rlandy|ruck | ack ok | 21:08 |
| dviroel|ruck | https://review.rdoproject.org/zuul/stream/0f99befa3f5b47b4abd63cc43d69aba7?logfile=console.log | 21:08 |
| rlandy|ruck | yep - watching those | 21:09 |
| dviroel|ruck | we need these https://review.opendev.org/q/topic:bug%252F1959601 | 21:10 |
| dviroel|ruck | I tested and fixes the issue, but we need backports to wallaby | 21:11 |
| dviroel|ruck | https://images.rdoproject.org/centos8/train/rdo_trunk/tripleo-ci-testing/ | 21:25 |
| dasm | hmmm... the job with uefi failed. now, let's see why :) | 21:31 |
| rlandy|ruck | ok - well at least we got one step | 21:36 |
| rlandy|ruck | fixes master | 21:36 |
| rlandy|ruck | dasm: failure may not be uefi | 21:37 |
| rlandy|ruck | compare against other runs | 21:37 |
| dasm | it booted, but i want to double check :) | 21:37 |
| rlandy|ruck | god | 21:38 |
| rlandy|ruck | good | 21:38 |
| dasm | i was looking at logs. it took about 30 minutes to start "baremetal" VMs. Much longer than for regular boot. | 21:39 |
| dviroel|ruck | rlandy|ruck: good to go https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/827779 | 21:42 |
| dviroel|ruck | rlandy|ruck: don't have the logs, killed the jobs before that - but they passed | 21:43 |
| rlandy|ruck | looking | 21:49 |
| rlandy|ruck | dviroel|ruck; ok - let's merge it | 21:49 |
| rlandy|ruck | hopefully it wont hose any branches | 21:49 |
| rlandy|ruck | it's a start | 21:50 |
| rlandy|ruck | dasm: ok - we can send those to steve | 21:50 |
| dasm | it actually looks like baremetal didn't start. i'm gonna need some more time to get through that | 21:50 |
| dasm | pleo_baremetal_node_introspection: Introspection of node 28d229ae-bc3e-4759-a8a2-7d46e3c8f661 failed on attempt 1: Timeout waiting for node 28d229ae-bc3e-4759-a8a2-7d46e3c8f661 to reach target state 'manageable'", | 21:50 |
| dasm | "2022-02-04 16:11:43,204 ERROR os_tripleo_baremetal_node_introspection: Introspection of node f72e03cd-8660-4804-a83b-e6bc449ccc20 failed on attempt 1: Timeout waiting for node f72e03cd-8660-4804-a83b-e6bc449ccc20 to reach target s | 21:50 |
| rlandy|ruck | I have a testproject running now | 21:50 |
| rlandy|ruck | so you can compare with that ine | 21:51 |
| rlandy|ruck | one | 21:51 |
| dasm | ack | 21:51 |
| rlandy|ruck | trying to promote 17 | 21:51 |
| dasm | looks like the job i tried to run with uefi is hosed. there was no successful test run in last 2 weeks | 21:57 |
| dviroel|ruck | rlandy|ruck: going eod | 22:00 |
| dviroel|ruck | rlandy|ruck: we need those w'ed patches merged, i added keystone reviewers to keystone-tempest-plugin change | 22:01 |
| dviroel|ruck | rlandy|ruck: and then have network component promoted to fix centos-9-master | 22:03 |
| dviroel|ruck | rlandy|ruck: but we should be able to continue with centos-8 | 22:03 |
| dviroel|ruck | rlandy|ruck: will take a look tomorrow how things are going | 22:03 |
| rlandy|ruck | dviroel|ruck: thanks so much | 22:05 |
| rlandy|ruck | have a great weekend | 22:05 |
| dviroel|ruck | rlandy|ruck: you too o/ | 22:05 |
| * rlandy|ruck also out now | 22:05 | |
| dviroel|ruck | dasm: hey, you too, have a great weekend | 22:05 |
| dasm | dviroel|ruck: \o | 22:06 |
| *** dviroel|ruck is now known as dviroel|out | 22:06 | |
| dasm | take care team :) | 22:06 |
| rlandy|ruck | dasm: bye - enjoy the weekend | 22:06 |
| rlandy|ruck | you too rcastillo | 22:06 |
| dasm | rlandy|ruck: you too! o/ | 22:06 |
| rlandy|ruck | see you all next week | 22:06 |
| * dasm is also leaving! | 22:12 | |
| *** dasm is now known as dasm|off | 22:13 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!