| *** ysandeep|out is now known as ysandeep | 05:22 | |
| akahat|rover | folks please take a look when you are free: https://review.rdoproject.org/r/c/rdo-jobs/+/41280 | 06:01 |
|---|---|---|
| marios | akahat|rover: morning | 06:17 |
| akahat|rover | marios, morning. | 06:17 |
| ysandeep | good morning everyone o/ | 06:20 |
| ysandeep | akahat|rover, looking | 06:20 |
| akahat|rover | review.rdorpoject.org/zuul is down | 06:24 |
| ysandeep | akahat|rover, left a comment: https://review.rdoproject.org/r/c/rdo-jobs/+/41280/1#message-5c4b48b62b034615d8920ef375d5aeb0080fb30f | 06:27 |
| ysandeep | ahh, marios have same comment - I didn't hit refresh before. | 06:28 |
| marios | ysandeep: ack jinx ;) | 06:30 |
| ysandeep | :D | 06:31 |
| ysandeep | marios: +wing this one https://review.opendev.org/c/openstack/tripleo-ci/+/834861, any objections? | 06:33 |
| marios | ysandeep: thanks yeah lets go | 06:35 |
| ysandeep | thanks! done | 06:36 |
| bhagyashris | marios, hi, good morning! i have one doubt... i have set my downstram reparent patch https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/399683 Depends-On in your patch https://review.opendev.org/c/openstack/tripleo-ci/+/834861 but it's giving me some zuul error "This change depends on a change with an invalid configuration." | 06:36 |
| bhagyashris | and i see your patch has no any invalid configration issues | 06:37 |
| bhagyashris | so do you have any idea about this issue | 06:37 |
| ysandeep | reviewbot: please add in review list: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/41569 | 06:37 |
| reviewbot | I have added your review the to Review list | 06:38 |
| ysandeep | rlandy_: yeah, I see push failed today as well. | 06:42 |
| ysandeep | akahat|rover, https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-17-rhel9/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-build-containers-ubi-9-internal-rhel-9-build-push-upload-rhos-17/80d9a6c/logs/build.log | 06:47 |
| ysandeep | akahat|rover, ^^ can you push an update on ticket that we are still facing issue today. | 06:48 |
| ysandeep | I will keep trying to delete images from our namespace with daniel help in background. | 06:48 |
| akahat|rover | ysandeep, oh.. that's not good. I"ll update the ticket. | 06:49 |
| bhagyashris | i see your patch got +W let's wait to merge that and then i will remove the Depends-On | 06:55 |
| marios | akahat|rover: ysandeep: is this related to the missing tag from yesterday (that was upstream) or something new? | 07:02 |
| marios | akahat|rover: was that resolved after you re-ran the push job (yesterday issue i mean) | 07:02 |
| marios | bhagyashris: sorry reading back | 07:03 |
| marios | bhagyashris: not sure ... the rdo patch at https://review.rdoproject.org/r/c/rdo-jobs/+/40842 also depends-on it and we don't have that... i wonder if it is because it didn't merge yet? if it persists after the depends-on merges i can look again but cant spot something on first pass | 07:06 |
| marios | bhagyashris: (the depends-on has been workflowed so should be in today | 07:08 |
| marios | fyi folks http://lists.openstack.org/pipermail/openstack-discuss/2022-April/028035.html | 07:11 |
| marios | [dev][infra][qa][tact-sig] Zuul behavior change with Depends-On across queues | 07:11 |
| akahat|rover | marios, ysandeep update the jobs. but it got failed. arxcruz copied containers back to trunk.registry | 07:25 |
| akahat|rover | after that we didn't see any failures. | 07:25 |
| * akahat|rover lunch | 07:34 | |
| *** jpena|off is now known as jpena | 07:36 | |
| jpodivin | akahat|rover: I'm not sure if this is the right place to raise this but it has been bugging me for a while so ... Why are we attempting to install atop during undercloud setup? | 08:13 |
| jpodivin | It's consistently failing on centos9 and the package isn't available on centos8-stream either https://pkgs.org/search/?q=atop | 08:14 |
| jpodivin | I mean, at very least we should limit the task somewhat, unless the package is going to get released for OS we use. | 08:16 |
| jpodivin | but even if it was there, I don't really understand what value does it provide. | 08:16 |
| bhagyashris | marios, ack thanks | 08:29 |
| *** raukadah is now known as chandankumar | 08:29 | |
| akahat|rover | jpodivin, atop is used to collect data. And later that used to show in graph. I'm also not sure now it's used in our jobs. | 08:48 |
| akahat|rover | jpodivin, if it is blocker to you then we can do something around it. | 08:48 |
| jpodivin | akahat|rover: it's not a blocker. not at all. But the task to install it fails consistently and it's just gobbling up time. | 08:49 |
| jpodivin | akahat|rover: essentially, whatever purpose it once served, it doesn't anymore. At least not in our jobs. | 08:50 |
| ysandeep | rlandy_, akahat|rover chandankumar fyi.. we had 103k images just for 16.2, delete is ongoing from last ~1.5 hour. | 08:50 |
| chandankumar | ysandeep: so many images | 08:50 |
| chandankumar | ysandeep: ++ thank you thank you for taking care of that :-) | 08:51 |
| akahat|rover | jpodivin, If package is not present in repo then there is no point to run play. | 08:52 |
| akahat|rover | jpodivin, I'll suggest report one lp and let maintainer team decide to include it or not. | 08:53 |
| akahat|rover | ysandeep, thanks. | 08:54 |
| akahat|rover | ysandeep++ | 08:54 |
| jpodivin | akahat|rover: ack. I'll also put up review limiting the task execution. | 08:55 |
| akahat|rover | ysandeep, can you please recheck downstream jobs again? | 08:55 |
| ysandeep | akahat|rover, chandankumar All kudos to dpawlik++ who helped me. | 08:55 |
| ysandeep | akahat|rover, lets wait for some cleanup.. | 08:55 |
| akahat|rover | ysandeep, okay. | 08:56 |
| akahat|rover | thank you dpawlik!! | 08:56 |
| akahat|rover | dpawlik++ | 08:56 |
| chandankumar | dpawlik++ thank you for doing that :-) | 08:57 |
| dpawlik | ysandeep do not be so modest! Each of us made a piece and started working ;) | 08:58 |
| ysandeep | :) | 08:59 |
| akahat|rover | arxcruz, o/ there? | 09:00 |
| *** ysandeep is now known as ysandeep|lunch | 09:08 | |
| jpodivin | akahat|rover: LP is up and triaged, as well as are the patches. The first one just disables, the followup is marked as WIP, because it get's rid of the task completely. | 09:15 |
| arxcruz | akahat|rover hey sorry, did not saw your message | 09:33 |
| arxcruz | sup ? | 09:33 |
| frenzy_friday | Hey ysandeep|lunch, is https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/41569 for the volume mount issue that we were seeing in the development cockpit server? | 09:33 |
| frenzy_friday | ok, it is for the selinux thing fro the certs, sorry, pls ignore | 09:37 |
| akahat|rover | arxcruz, https://meet.google.com/jsr-actw-kzr | 09:41 |
| arxcruz | akahat|rover give me one second | 09:41 |
| arxcruz | dpawlik hey, are you responsible for the rdo trunk prune containers ?> | 09:59 |
| dpawlik | arxcruz: yes | 10:00 |
| arxcruz | dpawlik https://meet.google.com/jsr-actw-kzr | 10:00 |
| arxcruz | quickly pleae | 10:00 |
| arxcruz | please :) | 10:01 |
| *** ysandeep|lunch is now known as ysandeep | 10:03 | |
| ysandeep | frenzy_friday: yes that's a separate issue. The issue we are facing on development server may be caused by the package updates I did on Monday's evening. | 10:05 |
| ysandeep | ^^ not sure about this part, just suspecting | 10:06 |
| chandankumar | jpodivindist: hello, left a comment on review regarding atop | 10:16 |
| jpodivindist | chandankumar: Thanks, I will take a look once I'm back at the desk. | 10:17 |
| *** rlandy_ is now known as rlandy | 10:19 | |
| rlandy | marios: chandankumar: ysandeep: akahat|rover: let's sync pls | 10:21 |
| rlandy | https://meet.google.com/egp-ujge-hsj?pli=1&authuser=0 | 10:22 |
| marios | rlandy: o/ sure joining | 10:23 |
| marios | rlandy: https://review.opendev.org/c/openstack/puppet-pacemaker/+/836767/2#message-3d78834c590f4ee83a967c646e406f0be90874b8 | 10:31 |
| ysandeep | chandankumar: just to confirm we need to parent off this job in downstream right? https://review.opendev.org/c/openstack/tripleo-ci/+/834100/4/zuul.d/base-upstream.yaml#275 | 10:31 |
| dpawlik | arxcruz: merging https://review.rdoproject.org/r/c/rdo-infra/rdo-infra-playbooks/+/41574 , Will be applied with next patch set for sf-infra | 10:31 |
| arxcruz | dpawlik thanks | 10:32 |
| dpawlik | np, thanks for notify | 10:32 |
| arxcruz | rlandy ^ | 10:32 |
| chandankumar | ysandeep: yes correct | 10:32 |
| arxcruz | rlandy akahat|rover probably already explain, but the containers were being pruned because it wasn't on the exclude list | 10:32 |
| ysandeep | chandankumar, thanks, proposing a patch | 10:33 |
| arxcruz | we should no longer have prune issues | 10:33 |
| ysandeep | arxcruz++ | 10:37 |
| ysandeep | reviewbot, please add to review list: https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-config/+/400141 | 10:43 |
| reviewbot | I have added your review the to Review list | 10:43 |
| ysandeep | chandankumar, ^^ fyi | 10:43 |
| chandankumar | ysandeep: ack will take a look | 10:45 |
| ysandeep | thanks, I will trigger testproject to trigger content-provier but as this is config project, I need to wait before ^^ merges. | 10:46 |
| chandankumar | ysandeep: good to trigger testproject noew | 10:58 |
| chandankumar | *now | 10:58 |
| chandankumar | marios: https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/rdoinfo-jobs.yaml and https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/distgit-jobs.yaml | 11:00 |
| marios | rlandy: https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/399683/4#message-68dbd9b0568f3caa60a439e16380b796949a975b | 11:06 |
| bhagyashris | hello folks ... review time.... | 11:18 |
| *** dviroel|out is now known as dviroel | 11:21 | |
| bhagyashris | rlandy, ysandeep marios chandankumar plz add into ypur review list https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/400144 this is braking downstram tripleo-ci-internal-jobs master branch | 11:22 |
| reviewbot | I have added your review the to Review list | 11:23 |
| bhagyashris | I have added your review *to the* Review list | 11:23 |
| chandankumar | bhagyashris: commented | 11:27 |
| bhagyashris | chandankumar, you mean allpodman jobs | 11:28 |
| bhagyashris | ? | 11:28 |
| chandankumar | bhagyashris: yes | 11:28 |
| chandankumar | bhagyashris: those are no longer runned | 11:28 |
| bhagyashris | chandankumar, ack | 11:28 |
| bhagyashris | thanks | 11:28 |
| *** ysandeep is now known as ysandeep|afk | 11:37 | |
| marios | bhagyashris: checking | 11:41 |
| soniya29 | kopecmartin, chandankumar, ysandeep|afk, rlandy, arxcruz, please add/edit today's agenda for tempest meeting - https://hackmd.io/fIOKlEBHQfeTZjZmrUaEYQ | 11:49 |
| soniya29 | kopecmartin, chandankumar, ysandeep|afk, rlandy, arxcruz, tempest meeting is overlapping with ptg session so its best we cancel the meeting and cover the topic in next meeting | 11:52 |
| * bhagyashris tea brb | 11:52 | |
| arxcruz | ok | 11:52 |
| soniya29 | if that is okay with all of us, then i can cancel the meeting | 11:53 |
| chandankumar | arxcruz: ok | 11:55 |
| kopecmartin | soniya29: ok | 11:59 |
| *** amoralej is now known as amoralej|lunch | 12:06 | |
| rlandy | arxcruz: https://logserver.rdoproject.org/59/836359/3/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001/1bf1d9d/job-output.txt - just FYI - failing stackviz install | 12:08 |
| rlandy | oslo | 12:08 |
| arxcruz | rlandy isn't this related to clark boylan email yesterday ? | 12:09 |
| rlandy | possibly | 12:10 |
| arxcruz | rlandy dasm|off also send this https://status.python.org/incidents/mxgkk3xxr9v7?u=v8pzlr5n28h8 | 12:10 |
| rlandy | arxcruz: still going on | 12:10 |
| rlandy | not related | 12:10 |
| arxcruz | rlandy in order to not block, we can set to not run stackviz | 12:11 |
| rlandy | arxcruz: going to raise them at CIX | 12:12 |
| rlandy | arxcruz: we might need that to keep train promoting | 12:12 |
| arxcruz | rlandy ok, i'm writing the patch now | 12:13 |
| arxcruz | rlandy do you have the bug easy so i can add on the patch ? | 12:13 |
| rlandy | https://bugs.launchpad.net/tripleo/+bug/1967943 | 12:13 |
| rlandy | ysandeep|afk: no show on container push /tripleorhos-16-2/openstack-base:21141c6945f2756a11ad74cd31d4530a": Error writing blob: Error initiating layer upload to /v2/tripleorhos-16-2/openstack-base/blobs/uploads/ in docker-registry.upshift.redhat.com: received unexpected HTTP status: 500 Internal Server Error\n' | 12:15 |
| chandankumar | rlandy: https://bugs.launchpad.net/tripleo/+bug/1967943/comments/2 | 12:15 |
| chandankumar | I tried to reproduce the issue but not able to reproduce it | 12:16 |
| rlandy | chandankumar: can you repro https://bugs.launchpad.net/tripleo/+bug/1967945 | 12:17 |
| chandankumar | I am not sure, it might be linked with http://mirror.regionone.vexxhost-nodepool-tripleo.rdoproject.org/wheel/centos-8-x86_64 part? | 12:17 |
| rlandy | that's the bigger issue | 12:17 |
| rlandy | yeah - will raise that at CIX | 12:17 |
| chandankumar | ok let me try that one | 12:17 |
| arxcruz | rlandy https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/836817 | 12:19 |
| arxcruz | reviewbot please add to review list https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/836817 | 12:20 |
| reviewbot | I could not add the review to Review List | 12:20 |
| arxcruz | how dare you! | 12:20 |
| jpodivin | akahat|rover: I've read your comment and I think it makes sense. But I think it might be better if we could raise this at the PTG, you know, after the scheduled talks are over. | 12:22 |
| arxcruz | jpodivin sorrye what are we talking about ? | 12:31 |
| arxcruz | xD | 12:31 |
| jpodivin | arxcruz: Oh nothing. Just crippling or removing a vital oooq task. ;) | 12:31 |
| * jpodivin joking, obviously. | 12:32 | |
| jpodivin | arxcruz: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/836752/ <<< the important info is all there. | 12:33 |
| rlandy | ysandeep|afk: akahat|rover: on the PNT escalation chat - other are reporting the same push failure | 12:38 |
| rlandy | Dan is looking into it | 12:38 |
| rlandy | ysandeep|afk: added you to the chat | 12:38 |
| rlandy | chandankumar: The user requested python-heatclient>=1.10.0\n The user requested (constraint) python-heatclient===2.3.1 | 12:41 |
| rlandy | lol - that should not conflict :/ | 12:41 |
| chandankumar | rlandy: ok | 12:42 |
| marios | jm1: see private i can send it to you if you need (different for different rooms) | 12:46 |
| *** ysandeep|afk is now known as ysandeep | 12:56 | |
| ysandeep | rlandy: thanks! | 12:57 |
| *** amoralej|lunch is now known as amoralej | 13:05 | |
| marios | rlandy: a little bird just told me in pvt that its a special day for you today? :D we got you some gate blockers \o/ | 13:36 |
| marios | happy birthday rlandy !! | 13:36 |
| rlandy | marios: thank you .... you forgot the broken registry | 13:36 |
| marios | oh yeah ... nothing but the best for your special day ! | 13:37 |
| rlandy | you guys rock as usual :) | 13:37 |
| chandankumar | wishing you a very happy birthday rlandy :-) | 13:37 |
| rlandy | chandankumar: thanks :) | 13:37 |
| marios | (w) obviously I won't (e) tell you (s) who gave me (h) this (a) information (y) | 13:39 |
| jm1 | marios++ 😂😂😂 | 13:42 |
| jm1 | rlandy: Congratulations on your birthday 🥂 | 13:43 |
| marios | ;) | 13:43 |
| rcastillo | rlandy: happy birthday :) | 13:44 |
| *** dasm|off is now known as dasm|ruck | 13:44 | |
| dasm|ruck | o/ | 13:45 |
| dviroel | rlandy: happy birthday o/ | 13:45 |
| dasm|ruck | happy birthday rlandy! | 13:45 |
| pojadhav | rlandy, wishing you a very very happy birthday :);) \o/ | 13:46 |
| rlandy | thanks all! | 13:50 |
| rlandy | marios: unnamed person already wished me happy birthday today - so I guessed :) | 13:51 |
| arxcruz | oh happy birtyday rlandy | 13:53 |
| arxcruz | should be celebrating no ? :D | 13:53 |
| arxcruz | i'll drink a beer for you today :D | 13:53 |
| rlandy | arxcruz: great idea - thank you | 13:53 |
| dasm|ruck | arxcruz: your fix to not compile tempest doesn't work, because it fails on missing heat client | 13:55 |
| dasm|ruck | https://logserver.rdoproject.org/17/836817/1/openstack-check/tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby/bb7ed40/job-output.txt | 13:56 |
| dasm|ruck | wrt: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/836817 | 13:56 |
| arxcruz | dasm|ruck this seems to be a different issue, because it doesn't even get to the point of my patch | 13:57 |
| akahat|rover | rlandy, Happy Birthday!! :) | 13:57 |
| dasm|ruck | arxcruz: ack. i initially though this could be related to pypi outage, but i also hoped to see it clearing by now. | 13:58 |
| arxcruz | dasm|ruck TASK [ovb-manage : Install openstack clients in venv] | 13:58 |
| rlandy | akahat|rover: thank you | 13:58 |
| arxcruz | it's in the ovb-manage task, even before the quickstart | 13:58 |
| dasm|ruck | not sure why openstack cdn didn't pick up the change. | 13:58 |
| dasm|ruck | arxcruz: why do we need to disable stackviz? | 13:59 |
| rcastillo | stackviz was failing pip install a lot | 13:59 |
| arxcruz | dasm|ruck rlandy point me to https://logserver.rdoproject.org/59/836359/3/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001/1bf1d9d/job-output.txt | 14:00 |
| arxcruz | i saw your email, about the pypi issue, but it seems to not be related | 14:00 |
| arxcruz | so, disabling just to clear the queue and then find a solution | 14:00 |
| dasm|ruck | arxcruz: i see stackviz. yeah. | 14:00 |
| dasm|ruck | wrt pypi -- i'm stubborn, but i still believe that it's related. | 14:01 |
| arxcruz | dasm|ruck am i understood wrongly ? | 14:01 |
| dasm|ruck | the outage is gone, but we're dealing with it's side effect | 14:01 |
| arxcruz | dasm|ruck because we are using mirror, so the vexxhost mirror might not be yet updated ? | 14:02 |
| dasm|ruck | arxcruz: no, you're right, i misread logs | 14:02 |
| dasm|ruck | arxcruz: indeed | 14:02 |
| arxcruz | rlandy did akahat|rover updated you regarding the missing containers on rdo registry ? | 14:02 |
| akahat|rover | arxcruz, yes. i did :) | 14:04 |
| arxcruz | akahat|rover++ | 14:04 |
| arxcruz | dpawlik++ | 14:04 |
| dasm|ruck | chandankumar++ thanks for reproducer for #1967945 | 14:05 |
| rlandy | arxcruz: ack - updated patch | 14:06 |
| rlandy | he pointed me to it | 14:06 |
| chandankumar | dasm|ruck: we need latest version of pip | 14:06 |
| dasm|ruck | chandankumar: hmm... nice catch. i looked at versions yesterday and it looked 22.0.4 for pip. Is there a newer one? | 14:07 |
| marios | akahat|rover: dasm|ruck: are y ou aware of ImportError: cannot import name 'OrderedDict' from 'typing_extensions' (/home/zuul/src/opendev.org/openstack/tripleo-heat-templates/.tox/pep8/lib/python3.8/site-packages/typing_extensions.py) | 14:07 |
| marios | i don't see it on the hackmd | 14:07 |
| chandankumar | 22.0.4 | 14:07 |
| chandankumar | dasm|ruck: https://review.rdoproject.org/zuul/stream/7d4e940c03904443bc98a1b5d8df208e?logfile=console.log | 14:07 |
| marios | akahat|rover: dasm|ruck: it blocks tht/wallaby https://zuul.opendev.org/t/openstack/builds?job_name=openstack-tox-pep8&project=openstack%2Ftripleo-heat-templates&branch=stable%2Fwallaby&skip=0 | 14:08 |
| chandankumar | here Install deps from upper-constraints task passed right now | 14:08 |
| dasm|ruck | marios: no, i'm not | 14:08 |
| dasm|ruck | that must be something new, marios | 14:08 |
| marios | dasm|ruck: k making a note on the hackmd for now | 14:08 |
| bhagyashris | rlandy, Happy Birthday ! :) | 14:08 |
| akahat|rover | marios, no. that is new. | 14:08 |
| chandankumar | dasm|ruck: https://review.rdoproject.org/r/c/testproject/+/40195 I am going to do a rerun of container and image build | 14:09 |
| dasm|ruck | chandankumar: thanks! | 14:09 |
| dasm|ruck | chandankumar: so the solution is to rebuild images, because we had some stale data in them? | 14:10 |
| rlandy | thank you | 14:10 |
| * dasm|ruck needs to learn more about images... | 14:10 | |
| chandankumar | dasm|ruck: nono | 14:11 |
| rlandy | chandankumar: where did you update pip? | 14:12 |
| chandankumar | dasm|ruck: build images job went into 3rd attempt | 14:12 |
| dasm|ruck | marios: it looks similar to what we're seeing downstream. and it started around similar time frame | 14:12 |
| chandankumar | and there install deps from uc task passed | 14:12 |
| chandankumar | somenow | 14:12 |
| chandankumar | I just wanted to make sure I have seen it correctly | 14:12 |
| dasm|ruck | chandankumar: so, are you saying -- try/repeat until passes? | 14:13 |
| chandankumar | or pip just got updated or working fine itself | 14:13 |
| *** dviroel is now known as dviroel|ptg | 14:13 | |
| dasm|ruck | ah | 14:13 |
| marios | dasm|ruck: ack i will try dig into it a bit more in a little while (seems like somethign was updated, or we need to update soemthing ;) e.g. https://github.com/allenai/allennlp/issues/5412 but that is pretty old | 14:13 |
| * dviroel|ptg on manila ptg | 14:13 | |
| dasm|ruck | chandankumar: it might be working if our local CDN is updated | 14:13 |
| chandankumar | dasm|ruck: just last run, if it does not work, will propose a fix to update pip to latest | 14:13 |
| dasm|ruck | chandankumar: let's try your reproducer. do you have it handy? | 14:13 |
| rlandy | dasm|ruck: we need to rekick victoria line- stable-2 | 14:14 |
| rlandy | now that some issues are resolved | 14:14 |
| dasm|ruck | rlandy: is it working now? | 14:14 |
| chandankumar | dasm|ruck: it is in my local container | 14:14 |
| rlandy | do you want to give that a shot? | 14:14 |
| dasm|ruck | rlandy: sure, let's try | 14:14 |
| rlandy | stable c8 is building containers - so we can try | 14:15 |
| rlandy | chandankumar: quick check in on pip ... https://meet.google.com/uho-ystc-tpn?pli=1&authuser=0 | 14:15 |
| dasm|ruck | rlandy: i kicked stable1 along with stable2 | 14:17 |
| dasm|ruck | oh wait. i couldn't kick stable1 because it doesn't exist anymore. there is stable1-cs8 which is already running | 14:19 |
| dasm|ruck | stable2: https://review.rdoproject.org/zuul/status/change/da881d902ef3da91a89790917edc3d8afaeb3f23 | 14:19 |
| dasm|ruck | and i'm seeing green statuses there | 14:20 |
| dasm|ruck | not jumping on a gun, but we might be in a better state right now | 14:20 |
| dasm|ruck | ftr: wallaby: c8 and c9; victoria c8 -- all are started | 14:21 |
| *** pojadhav is now known as pojadhav|afk | 14:23 | |
| chandankumar | dasm|ruck: https://review.rdoproject.org/zuul/status#40195 looking good, both reached at building containers and images | 14:31 |
| chandankumar | if that passes, we can rekick the failed line | 14:31 |
| dasm|ruck | chandankumar: great! | 14:31 |
| dasm|ruck | chandankumar: all 3 lines are running now, please see ^ | 14:31 |
| rlandy | ok - https://review.rdoproject.org/zuul/stream/fb2d901a8b764b8c819621bf20c98fab?logfile=console.log got through setup | 14:34 |
| dasm|ruck | are we savedd? | 14:35 |
| rlandy | dasm|ruck: maybe | 14:35 |
| dasm|ruck | i'd like to hear/understand post mortem. | 14:35 |
| rlandy | 16.2 is pushing containers now | 14:35 |
| dasm|ruck | at the end - did we do anything on our end? | 14:36 |
| ysandeep | rlandy, bhagyashris hey looks like jm1 have ptg topic tomorrow for Ansible collection: https://etherpad.opendev.org/p/aom-zed-ptg , Timing conflicts with our scrum mtg time, Since many of our team members are contributing there including rcastillo frenzy_friday arxcruz - I think its worth for them to join the session as well, Do you want to cancel tomorrow's scrum? | 14:38 |
| arxcruz | ysandeep indeed | 14:38 |
| ysandeep | rlandy, fyi.. retriggering network component job on 17/rhel9 so that network component can promote - to clear container build issue - missing ovn-bpg agent bug. | 15:01 |
| dasm|ruck | > 2022-04-06 14:58:58.186164 | primary | fatal: [undercloud]: FAILED! => {"changed": true, "cmd": "set -o pipefail && /home/zuul/install_packages.sh 2>&1 | awk '{ print strftime(\"%Y-%m-%d %H:%M:%S |\"), $0; fflush(); }' > /home/zuul/install_packages.sh.log\n", "delta": "0:00:03.991482", "end": "2022-04-06 14:58:57.977900", "msg": "non-zero return code", "rc": 1, "start": "2022-04-06 | 15:02 |
| *** pojadhav|afk is now known as pojadhav | 15:02 | |
| dasm|ruck | 14:58:53.986418", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} | 15:02 |
| dasm|ruck | hmm... i'm seeing errors in https://review.rdoproject.org/zuul/stream/9e176982f3ab43eb9f08298358fbda80?logfile=console.log | 15:02 |
| dasm|ruck | so we might be hitting another issue here | 15:02 |
| dasm|ruck | > Run the package installation script | 15:03 |
| dasm|ruck | during ^ step | 15:03 |
| chandankumar | rlandy: dasm|ruck https://review.rdoproject.org/r/c/testproject/+/40195 | 15:05 |
| chandankumar | it passed | 15:05 |
| chandankumar | https://logserver.rdoproject.org/95/40195/3/check/periodic-tripleo-ci-build-containers-centos-9-push-master/85af6c0/job-output.txt | 15:05 |
| dasm|ruck | chandankumar: we're seeing mixed results with package installation. 404 on some packages | 15:05 |
| dasm|ruck | https://review.rdoproject.org/zuul/status/change/da881d902ef3da91a89790917edc3d8afaeb3f23 | 15:05 |
| chandankumar | Successfully installed pip-22.0.4 wheel-0.37.1 | 15:05 |
| chandankumar | pip version got updated | 15:06 |
| dasm|ruck | but we're not having issues with pip anymore | 15:06 |
| dasm|ruck | ack | 15:06 |
| dasm|ruck | yeah, flushing cache and upgrading pip solved the issue | 15:06 |
| dasm|ruck | chandankumar++ | 15:06 |
| dasm|ruck | bbaib: dropping off the car at dealership. | 15:07 |
| *** dasm|ruck is now known as dasm|ruck|bbl | 15:08 | |
| rlandy | ysandeep: ack - thanks - ws just discussing that in cix | 15:10 |
| chandankumar | jm1: dasm|ruck|bbl dviroel|ptg rlandy please have a look at https://review.opendev.org/c/openstack/tripleo-docs/+/835782 and published doc https://e842593f2193c9a738b6-4db814fce3c49764d252a4e9ed44d242.ssl.cf5.rackcdn.com/835782/9/check/openstack-tox-docs/c513620/docs/ci/tripleo_ci_job_parenting.html , thanks once again for previous review, it was very helpful. :-) | 15:13 |
| ysandeep | rlandy, fyi.. old container are still getting purged for 16.2, In last run I believe token expired in between and gave me false hope that it completed, I have added token in script itself and we are doing purge in batches now, retention - 500 days -> than 400 days and so on.. Based on current rate might take entire day.. | 15:15 |
| ysandeep | running on a server in tmux, hope I to see good results tomorrow morning :D | 15:16 |
| rlandy | ysandeep++ thanks or taking care of this | 15:19 |
| rlandy | chandankumar: will do when off meetings | 15:19 |
| ysandeep | I am out for the day, bhagyashris rlandy I will check in my morning what you guys decides about scrum mtg - whether to keep it or cancel it for Ansible collection session. | 15:19 |
| rlandy | ysandeep: bhagyashris: can we move scrum | 15:20 |
| rlandy | we should attend jm1's session | 15:20 |
| rlandy | I need to be there | 15:20 |
| rlandy | good call | 15:20 |
| rlandy | bhagyashris: can we move scrum to friday? | 15:21 |
| ysandeep | +1 | 15:21 |
| *** ysandeep is now known as ysandeep|out | 15:21 | |
| ysandeep|out | triggered network component jobs on 17/rhel9 here: https://code.engineering.redhat.com/gerrit/c/testproject/+/210197 | 15:24 |
| chandankumar | see ya! | 15:33 |
| marios | rlandy: chandankumar: ysandeep|out: please add to your next reviews https://review.opendev.org/c/openstack/tripleo-ci/+/836803/2#message-f036f5560f444856848ecc45d3b02e4ed669976a | 15:37 |
| reviewbot | Do you want me to add your patch to the Review list? Please type something like add to review list <your_patch> so that I can understand. Thanks. | 15:37 |
| marios | thank you reviewbot that is very kind | 15:37 |
| reviewbot | You are welcome :) | 15:37 |
| marios | :D | 15:37 |
| rlandy | so polite | 15:39 |
| marios | ikr? | 15:39 |
| *** dviroel|ptg is now known as dviroel|ptg|lunch | 15:42 | |
| * dviroel|ptg|lunch lunch | 15:42 | |
| rlandy | dasm|ruck|bbl: pls check lagging components when you get back | 15:50 |
| rlandy | forget ussuri | 15:50 |
| *** dasm|ruck|bbl is now known as dasm|ruck | 15:52 | |
| dasm|ruck | rlandy: ack | 15:52 |
| rlandy | going to rekick the network line | 15:52 |
| rlandy | it's all failing | 15:52 |
| dasm|ruck | ack | 15:52 |
| dasm|ruck | i see mixed results from c8 wallaby line: rlandy after it's done, do we want to attempt one more promotion, or manually re-running failed jobs? | 15:56 |
| rlandy | dasm|ruck: the same hash ran before | 15:59 |
| rlandy | so just looking for the missing jobs | 15:59 |
| rlandy | dasm|ruck: you can check me with the rr tool | 15:59 |
| dasm|ruck | ack | 16:00 |
| rlandy | dasm|ruck: https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-7-content-provider&skip=0 - here we go again | 16:01 |
| rlandy | 2022-04-06 15:55:42.276555 | primary | --> Processing Dependency: openssl-libs(x86-64) = 1:1.0.2k-24.el7_9 for package: 1:openssl-devel-1.0.2k-24.el7_9.x86_64 | 16:01 |
| rlandy | 2022-04-06 15:55:42.277763 | primary | ---> Package pcre-devel.x86_64 0:8.32-17.el7 will be installed | 16:01 |
| rlandy | 2022-04-06 15:55:42.506065 | primary | --> Finished Dependency Resolution | 16:01 |
| rlandy | 2022-04-06 15:55:42.534318 | primary | Error: Package: 1:openssl-devel-1.0.2k-24.el7_9.x86_64 (updates) | 16:01 |
| dasm|ruck | why?! why?!?! | 16:01 |
| *** dviroel|ptg|lunch is now known as dviroel|ptg | 16:03 | |
| marios | dasm|ruck: filing bug now for that issue i pinged earlier on wallaby/tht with fix incoming | 16:14 |
| marios | dasm|ruck: (sanity check did you already file one? ) | 16:14 |
| dasm|ruck | rlandy: i didn't file a bug. | 16:15 |
| marios | dasm|ruck: ack ok doing thanks | 16:15 |
| dasm|ruck | ack, thamhs | 16:15 |
| dasm|ruck | *thanks | 16:15 |
| *** amoralej is now known as amoralej|off | 16:16 | |
| rlandy | marios: dasm|ruck: so ok one of you is filing a bug | 16:17 |
| rlandy | gate blocker fix | 16:17 |
| rlandy | marios: you have a fix? | 16:17 |
| marios | rlandy: yeah luckily easy fix but surprised no one has asked us aabout it yet https://zuul.opendev.org/t/openstack/builds?job_name=openstack-tox-pep8&project=openstack%2Ftripleo-heat-templates&branch=stable%2Fwallaby&skip=0 | 16:18 |
| dasm|ruck | marios: sorry, i pinged Ronelle automatically, because she's the only person pinging me during this time :D | 16:18 |
| rlandy | marios: I was talking about content provider | 16:18 |
| marios | rlandy: ah this is another issue then for tripleo-heat-templates only | 16:18 |
| marios | gate blocker | 16:19 |
| rlandy | dasm|ruck: marios: so we still need a c7 provider bug? | 16:19 |
| dasm|ruck | rlandy: yes | 16:19 |
| rlandy | dasm|ruck: pls file that | 16:19 |
| dasm|ruck | k | 16:19 |
| marios | dasm|ruck: i filed it sec | 16:20 |
| marios | dasm|ruck: rlandy: https://bugs.launchpad.net/tripleo/+bug/1968065 openstack-tox-pep8 failing on stable/wallaby tripleo-heat-templates gate blocker | 16:20 |
| marios | dasm|ruck: pin down works let me try the other way too https://bugs.launchpad.net/tripleo/+bug/1968065/comments/1 | 16:23 |
| dasm|ruck | ack | 16:23 |
| rlandy | marios: thank you | 16:26 |
| marios | rlandy: np | 16:28 |
| *** jpena is now known as jpena|off | 16:35 | |
| *** marios is now known as marios|out | 16:38 | |
| *** marios|out is now known as marios | 16:43 | |
| marios | dasm|ruck: sorry... looks like there is some issue still there but will have to dig tomorrow fyi https://bugs.launchpad.net/tripleo/+bug/1968065/comments/2 | 16:44 |
| marios | k now going | 16:44 |
| *** marios is now known as marios|out | 16:44 | |
| dasm|ruck | marios|out: thanks | 16:45 |
| rlandy | dasm|ruck: today we need to promo victoria and stable -1 - wallaby c9 | 16:50 |
| dasm|ruck | ack | 16:50 |
| rlandy | pls lagging components there | 16:50 |
| dasm|ruck | wallaby c9 is in progress | 16:50 |
| rlandy | pls rerun jobs that failed on those lines | 16:50 |
| dasm|ruck | ack | 16:50 |
| rlandy | 8 days agobaremetalpromoted-components2 days ago | 16:50 |
| rlandy | 16 days agotripleopromoted-components6 hours ago | 16:50 |
| rlandy | dasm|ruck: ^^ wallaby c8 component | 16:51 |
| rlandy | baremetal needs some love | 16:51 |
| rlandy | and tripleo after it next kickc | 16:51 |
| dasm|ruck | c8 or c9? | 16:51 |
| rlandy | pls keep an eye on those | 16:51 |
| rlandy | c8 | 16:51 |
| dasm|ruck | ack | 16:51 |
| rlandy | sorry - that I pasted was master | 16:52 |
| rlandy | dasm|ruck; anyways you know where to look | 16:52 |
| dasm|ruck | (venv) [dasm@fedora telegraf_py3]$ ./ruck_rover.py --distro centos-8 --release wallaby --component all | 16:52 |
| dasm|ruck | baremetal component status=Red last_promotion=2022-03-27 06:03:33 | 16:52 |
| rlandy | I'm concentrating on promoting 16.2 now | 16:52 |
| dasm|ruck | yeah, some love needed | 16:52 |
| dasm|ruck | ack, thank you rlandy++ | 16:52 |
| rlandy | so yeah - pls check what upstream components need attention | 16:53 |
| rlandy | so we hand off cleanly | 16:53 |
| rlandy | lunch - brb | 16:54 |
| * dasm|ruck stepping away for a bit. picking up the car | 17:06 | |
| *** dasm|ruck is now known as dasm|ruck|bbl | 17:06 | |
| *** dviroel|ptg is now known as dviroel | 17:22 | |
| *** dasm|ruck|bbl is now known as dasm|ruck | 17:36 | |
| * dasm|ruck is back | 17:37 | |
| dasm|ruck | dviroel: how to re-run failing component? the same way like other jobs, with dlrn hash? | 17:40 |
| dasm|ruck | dviroel: let's say this: https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-baremetal-wallaby failed due to pip issues | 17:40 |
| dasm|ruck | and i'd like to run it agai | 17:40 |
| dasm|ruck | *again | 17:40 |
| rcastillo | dasm|ruck: you can pass --component to the rr script | 17:41 |
| dasm|ruck | ok, i found an example: https://review.rdoproject.org/r/c/testproject/+/33582/23/.zuul.yaml | 17:41 |
| dasm|ruck | rcastillo: it lists me all passing/failing. i wanted te re-run jobs | 17:42 |
| rcastillo | ahh, right | 17:42 |
| * dasm|ruck is gonna add some code to print that out immediately ^^ | 17:42 | |
| dviroel | dasm|ruck: yeah, no hash, no force_periodic | 17:42 |
| dasm|ruck | ack | 17:42 |
| rcastillo | dasm|ruck: yeah that'd be handy | 17:42 |
| dasm|ruck | just names. gotcha | 17:43 |
| *** dviroel is now known as dviroel|mtg | 17:47 | |
| *** dviroel|mtg is now known as dviroel | 18:43 | |
| rlandy | dasm|ruck: c7 content provider job that failed is in rerun - may be passing now | 19:36 |
| rlandy | will let you know | 19:36 |
| dasm|ruck | ack | 19:38 |
| * rlandy needs to drop off package - brb | 19:48 | |
| *** rlandy is now known as rlandy|biab | 19:48 | |
| *** rlandy|biab is now known as rlandy | 20:04 | |
| rlandy | dasm|ruck: need anything? | 20:15 |
| rlandy | https://review.opendev.org/c/openstack/tripleo-ansible/+/836600/ has c7 passing now | 20:16 |
| dasm|ruck | rlandy: no, thank you | 20:21 |
| rlandy | dasm|ruck: distracted day - friends/family calling on and off for birthday :) | 20:21 |
| dasm|ruck | rlandy: that's a good thing! I hope you have time to celebrate? :) | 20:23 |
| rlandy | dasm|ruck: yep - dinner tonight | 20:23 |
| dasm|ruck | nice! | 20:23 |
| *** dviroel is now known as dviroel|afk | 20:34 | |
| * dasm|ruck must be running. Will try check on lines later | 21:12 | |
| *** dasm|ruck is now known as dasm|off | 21:12 | |
| rlandy | dasm|off: there is a possibility we may move scrum tomorrow to attend jm1's PTG session | 21:19 |
| rlandy | so no need to get up early for that one | 21:19 |
| dasm|off | ack, thanks rlandy | 21:21 |
| rlandy | we're one job away from promoting 16.2 | 21:22 |
| dasm|off | that's awesome! | 21:22 |
| rlandy | woohoo stable 1 promoted | 21:24 |
| rlandy | tripleo-ci-centos-7-content-provider cleared | 21:25 |
| rlandy | so we're doing much better | 21:25 |
| dasm|off | nice | 21:27 |
| rcastillo | frenzy_friday: there was no openstack-ovn-bgp-agent in the quay tripleomastercentos9 namespace. I added it and the test periodic-tripleo-build-containers-base-c9-quay-master is passing now and the containers are showing up | 21:30 |
| rlandy | current-tripleo/2022-04-06 18:05 | 22:05 |
| rlandy | woohoo 16.2 promoted | 22:05 |
| *** rlandy is now known as rlandy|out | 22:46 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!