| *** ysandeep|out is now known as ysandeep | 05:13 | |
| *** pojadhav- is now known as pojadhav | 05:36 | |
| *** ysandeep is now known as ysandeep|lunch | 07:35 | |
| *** amoralej|off is now known as amoralej | 07:42 | |
| *** jpena|off is now known as jpena | 08:13 | |
| *** ysandeep|lunch is now known as ysandeep | 08:33 | |
| *** chandankumar is now known as chkumar|rover | 08:42 | |
| jm1 | o/ | 09:06 |
|---|---|---|
| jm1 | will we meet today for talking about how to do ruck rover in future? | 09:07 |
| ysandeep | jm1, yes looks like bhagyashris missed to add you in invite | 09:16 |
| bhagyashris | jm1, yes | 09:16 |
| jm1 | bhagyashris: please send me an invititation :) | 09:17 |
| bhagyashris | my apology forgot to add in invite | 09:17 |
| bhagyashris | sent an invite to you | 09:17 |
| jm1 | bhagyashris: thank you :) | 09:18 |
| *** chem is now known as Guest1864 | 09:50 | |
| dpawlik | chkumar|rover, marios: hey, could you push container images also to tripleomastercentos9 on quay.rdoproject.org? | 10:03 |
| marios | dpawlik: o/ i must be missing some context chkumar|rover ? is there a problem with the promoter? | 10:08 |
| ysandeep | dpawlik, ananya is doing some testing to push containers to quay.rdoproject.org for ussuri/victoria first. | 10:09 |
| marios | something wrong with rdo/infra today ? waiting 45 mins for testproject didn't kick yet | 10:10 |
| marios | anyone else seeing it? | 10:10 |
| ysandeep | dpawlik, We want to target u/v branches first and once we confirm all okay for those branches - we will start pushing containers to quay.rdoproject.org for all branches. | 10:10 |
| dpawlik | ysandeep: ack | 10:10 |
| dpawlik | ysandeep: exactly | 10:10 |
| marios | oh i see what dpawlik meant now ysandeep sorry i though there was problem with our promoter pushing images | 10:10 |
| dpawlik | ysandeep: but I don't see Ananya on irc, thats why I'm writing here :) | 10:11 |
| ysandeep | marios: that's the new registry we need to use. | 10:11 |
| marios | ysandeep: yeah 'new' ;) | 10:11 |
| ysandeep | dpawlik, she is out today | 10:11 |
| dpawlik | ah, ok ;) | 10:11 |
| marios | dpawlik: do you need them on quay for something specific/do we have a deadline on that? or just following it? | 10:13 |
| dpawlik | marios: no deadline, so far :) | 10:14 |
| marios | k | 10:14 |
| dpawlik | marios: I'm mostly improving quay and automatize it, but I need to focus now on opensearch opendev | 10:14 |
| ysandeep | dpawlik, I will chat with Ananya once she is back on Monday - if we can validate soon new registry is working for us. | 10:15 |
| dpawlik | ysandeep: easy. Just one branch is enough to test now. I'm still working on improving automation new organizations | 10:16 |
| ysandeep | dpawlik, we are not in hurry unless there is a deadline :D just one branch for validation :D and for that branch too - I think plan is to push to both registries for the timebeing. | 10:18 |
| chkumar|rover | dpawlik: Ananya was working on testproject to push ussuri containers on rdo quay | 10:19 |
| chkumar|rover | dpawlik: will add a job in the pipeline to push containers on quay | 10:20 |
| chkumar|rover | may be by monday | 10:20 |
| chkumar|rover | from cs9master | 10:20 |
| dpawlik | ack, thank you all | 10:20 |
| chkumar|rover | marios: opening a bug for this https://review.opendev.org/c/openstack/tripleo-ci/+/831932/6#message-b219ca9074fa8579e4cdc137ca48777fb24d9af1 | 10:40 |
| marios | chkumar|rover: are you sure? doesn't seem to be a general thing yet? https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-9-content-provider-wallaby | 10:41 |
| chkumar|rover | oh checking the job | 10:42 |
| marios | chkumar|rover: thats why i did the recheck i looked earlier | 10:43 |
| marios | chkumar|rover: it is green now in the latest run | 10:43 |
| marios | chkumar|rover: https://zuul.openstack.org/status#831932 | 10:43 |
| marios | chkumar|rover: heh lets see maybe it only hits gate :/ :D | 10:45 |
| * marios prepares sacrifice for zuul | 10:45 | |
| chkumar|rover | dlrn build was a legit failure | 10:45 |
| marios | i love your optimism | 10:45 |
| chkumar|rover | https://github.com/openstack/validations-libs/commit/9163d8bcbcf364db9c04323436de35fc3946b380#diff-fac4c6890301d4de5c3f4266837803d5240c84a3d8b6c735bbc6a64c39d2f94eR16 | 10:46 |
| chkumar|rover | https://github.com/rdo-packages/validations-libs-distgit/blob/wallaby-rdo/python-validations-libs.spec | 10:46 |
| chkumar|rover | there is no python3-oslotest in Buildrequires | 10:46 |
| chkumar|rover | anyway will wait | 10:46 |
| marios | chkumar|rover: agree it was a legit issue but we often get things that happen 1/2 then disapear... e.g. | 10:46 |
| marios | chkumar|rover: some part of the thing you pointed to merged | 10:46 |
| marios | chkumar|rover: and some delay before another dependent part also merged | 10:46 |
| marios | in between breakage | 10:47 |
| marios | or something | 10:47 |
| chkumar|rover | ok will add a note to hackmd | 10:47 |
| marios | chkumar|rover: but we should avoid filing bugs for something we see 1 time otherwise we will become even crazier | 10:47 |
| marios | ;) | 10:47 |
| marios | chkumar|rover: if we see it again then file it | 10:47 |
| marios | at least 2 x | 10:47 |
| chkumar|rover | yes yes | 10:48 |
| marios | chkumar|rover: i think something up with rdo/infra | 10:54 |
| marios | chkumar|rover: i see a lot of POST_FAILURE in tests | 10:54 |
| marios | chkumar|rover: look at your testproject has ^^^ | 10:54 |
| *** soniya is now known as soniya29 | 10:54 | |
| marios | chkumar|rover: (40340) | 10:54 |
| chkumar|rover | marios: ah my patch | 10:55 |
| marios | chkumar|rover: but also for me | 10:55 |
| marios | chkumar|rover: it was waiting 1 hour to start a job | 10:55 |
| marios | chkumar|rover: and now is going 2nd attempt before even running | 10:55 |
| marios | chkumar|rover: something wrong i think | 10:55 |
| chkumar|rover | we are aware of node failure and retry limit | 10:55 |
| marios | chkumar|rover: ah ok | 10:55 |
| marios | chkumar|rover: they have some upgrade ongoing or something? | 10:56 |
| marios | vexx? | 10:56 |
| chkumar|rover | marios: https://hackmd.io/TBlzcsVTSv6nAKg9ZY-8cA?view | 10:56 |
| marios | lots of jobs hitting retry_limit | 10:56 |
| marios | looking | 10:56 |
| marios | chkumar|rover: ok great | 10:57 |
| marios | dasm|off: ++ good idea to resurrect the ruck|rover pad :) even if i was grumpy about it to begin with ;) | 10:57 |
| * marios grumpy old man | 10:57 | |
| marios | get off my lawn! | 10:57 |
| marios | well that sucks | 10:58 |
| marios | so i guess we don't get any rdo testing today | 10:58 |
| marios | :/ | 10:59 |
| marios | END OF STREAM https://review.rdoproject.org/zuul/stream/b50997ed327f4d8f984c5d518b8e9bca?logfile=console.log | 10:59 |
| marios | wont start the run | 10:59 |
| *** dviroel|out is now known as dviroel|ruck | 11:12 | |
| dviroel|ruck | mornings | 11:12 |
| chkumar|rover | dviroel|ruck: Good morning | 11:13 |
| *** rlandy|out is now known as rlandy|ruck | 11:13 | |
| rlandy|ruck | hello | 11:15 |
| rlandy|ruck | dviroel|ruck: chkumar|rover: review time? | 11:16 |
| rlandy|ruck | we can go through the queue status | 11:16 |
| dviroel|ruck | ok | 11:16 |
| dviroel|ruck | train c8 promoted, nice | 11:16 |
| chkumar|rover | dviroel|ruck: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/40345 please +w | 11:34 |
| chkumar|rover | dviroel|ruck: rlandy|ruck fyi! I will be travelling in the morning on Monday, will start my day during review time | 11:35 |
| dviroel|ruck | chkumar|rover: ack | 11:36 |
| rlandy|ruck | chkumar|rover: ok | 11:37 |
| rlandy|ruck | chkumar|rover: dviroel|ruck; ok if I dequeue the stable-1 line | 11:39 |
| rlandy|ruck | it will not promote and we need the nodes | 11:39 |
| chkumar|rover | yes please | 11:39 |
| rlandy|ruck | akahat: so we still have the option to comment out on the promoter itself and restart the service? | 11:41 |
| akahat | rlandy|ruck, yes. | 11:41 |
| akahat | which job ? | 11:42 |
| chkumar|rover | dviroel|ruck: cs9 container build is passing now in testproject, need to wait final main integration line run then we can close deps bug | 11:43 |
| chkumar|rover | *wait for | 11:43 |
| rlandy|ruck | akahat: periodic-tripleo-ci-centos-9-standalone-full-tempest-scenario-wallaby - waiting for my job to start ... but maybe let's just do this on the promoter? | 11:44 |
| dviroel|ruck | chkumar|rover: ack - I can close the bug in my afternoon | 11:44 |
| akahat | rlandy|ruck, okay. | 11:44 |
| rlandy|ruck | akahat: giving my job 15 mins to start - if not by then - will ping you | 11:45 |
| rlandy|ruck | and we can do this on the promoter itself | 11:45 |
| rlandy|ruck | and then revert it | 11:45 |
| akahat | oh.. i restarted the service. | 11:45 |
| soniya29 | tosky, arxcruz, please review this patch whenever you have time:- https://review.opendev.org/c/openinfra/python-tempestconf/+/820169 | 11:47 |
| rlandy|ruck | akahat: you already did it? ok | 11:48 |
| akahat | rlandy|ruck, yes. i thought it's needs to done right away. | 11:49 |
| rlandy|ruck | akahat: let's chat ... https://meet.google.com/osd-sxmm-sgo?pli=1&authuser=0 | 11:49 |
| chkumar|rover | interesting | 11:49 |
| chkumar|rover | we are already consuming podman.x86_64 2:4.0.0-6.el9 @quickstart-centos-appstreams | 11:50 |
| chkumar|rover | in our jobs https://logserver.rdoproject.org/56/36356/26/check/periodic-tripleo-ci-build-containers-centos-9-push-master/7eb5a6a/logs/system/installed_pkgs.txt | 11:50 |
| dviroel|ruck | marios: arxcruz: when you have some time, please take a look on this https://review.opendev.org/c/openstack/tripleo-quickstart/+/830884 - arxcruz you might remember that issue | 11:52 |
| dviroel|ruck | marios: arxcruz - needed for fips c9 testing | 11:53 |
| rlandy|ruck | jm1: hey - are you joining the rr improvements meeting?> | 12:03 |
| * dviroel|ruck rebooting laptop | 12:04 | |
| akahat | http://promoter.rdoproject.org/promoter_logs/centos9_wallaby.log | 12:05 |
| rlandy|ruck | jm1: pls bring your suggestion to that meeting | 12:05 |
| marios | ack dvd | 12:07 |
| marios | sorry dvd was meant for dviroel but he quit :) | 12:07 |
| rlandy|ruck | chkumar|rover: marios: wrt ceph failures ... was there a fix for scen001 and 004 as well? | 12:10 |
| rlandy|ruck | or just molecule | 12:10 |
| rlandy|ruck | https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-9-scenario004-standalone&skip=0 | 12:11 |
| rlandy|ruck | https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-9-scenario001-standalone&skip=0 | 12:11 |
| chkumar|rover | rlandy|ruck: just molecule | 12:11 |
| rlandy|ruck | ok - so there is no bug for ^^? | 12:12 |
| marios | rlandy|ruck: so there was an update from franscesco for versions .. sec let me find | 12:12 |
| rlandy|ruck | marios: can just ask him | 12:13 |
| marios | rlandy|ruck: https://review.opendev.org/c/openstack/tripleo-common/+/832810 | 12:13 |
| marios | rlandy|ruck: see related bug there | 12:13 |
| marios | rlandy|ruck: master merged waiting for that ^^^ and it has a quickstart which dependson it there https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/832756 | 12:13 |
| chkumar|rover | in order to fix wallaby jobs, we need a cs9 promotion | 12:14 |
| rlandy|ruck | https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-9-scenario001-standalone&skip=0 | 12:14 |
| rlandy|ruck | has failures on master | 12:14 |
| rlandy|ruck | am I confused here? | 12:15 |
| marios | rlandy|ruck: for that one https://f5c97d48dcd7c256d91c-c98b0e913e593910e72c88f6f60b9c46.ssl.cf1.rackcdn.com/778915/100/check/tripleo-ci-centos-9-scenario004-standalone/ff8a474/logs/undercloud/home/zuul/standalone_deploy.log (from th ebuilds you pointed to) it is the 2022-03-11 11:17:44.182820 | fa163e1e-9a3b-92ce-b191-000000001c21 | FATAL | genereate ceph_spec for bootstrap | undercloud | | 12:15 |
| chkumar|rover | The tripleo_ansible_inventory or deployed_metalsmith parameter is required | 12:15 |
| marios | error={"changed": false, "msg": "The tripleo_ansible_inventory or deployed_metalsmith parameter is required.", "specs": []} | 12:15 |
| marios | thing that merged today | 12:15 |
| rlandy|ruck | yes | 12:16 |
| rlandy|ruck | thing == fix? | 12:16 |
| rlandy|ruck | marios: ^^? | 12:17 |
| tosky | soniya29: I asked for a fresh link to a job which tests it, and I'd like to hear from akekane too | 12:18 |
| marios | rlandy|ruck: https://review.opendev.org/c/openstack/tripleo-ansible/+/833178 | 12:18 |
| rlandy|ruck | correct but that only fixes molecule | 12:19 |
| chkumar|rover | marios: rlandy|ruck https://review.opendev.org/c/openstack/tripleo-ansible/+/833182/2 | 12:19 |
| chkumar|rover | we need this | 12:19 |
| chkumar|rover | another patch on top of that | 12:19 |
| marios | chkumar|rover: no not to unblock we don't | 12:20 |
| marios | chkumar|rover: first patch should be enough according to fulton | 12:20 |
| marios | chkumar|rover: the second one is improvement | 12:20 |
| *** dviroel is now known as dviroel|ruck | 12:20 | |
| rlandy|ruck | chkumar|rover: hmmm .. scen001 did not run on that | 12:21 |
| * dviroel|ruck fixed krbs issue | 12:21 | |
| chkumar|rover | https://review.opendev.org/c/openstack/tripleo-ansible/+/830998 got merged then they proposed this https://review.opendev.org/c/openstack/tripleo-ansible/+/833178 | 12:22 |
| * rlandy|ruck just asks at this point | 12:22 | |
| * chkumar|rover fires a dnm patch | 12:22 | |
| marios | chkumar|rover: rlandy|ruck: https://bugs.launchpad.net/tripleo/+bug/1964524/comments/2 only first patch needed to unblock | 12:24 |
| rlandy|ruck | tripleo-ansible-centos-8-molecule-tripleo_cephadm https://zuul.opendev.org/t/openstack/build/eb7ee7ced78b48bc9192b941a55de17d : SUCCESS in 19m 50s is green | 12:25 |
| rlandy|ruck | sc001 did not run on that | 12:25 |
| marios | now this is a new issue rlandy|ruck chkumar|rover AnsibleUndefinedVariable: 'ansible.parsing.yaml.objects.AnsibleUnicode object' has no attribute 'keys' | 12:25 |
| marios | https://79691c100660a74b7818-f833862f6fa359097814ed1d3103e043.ssl.cf1.rackcdn.com/830987/20/check/tripleo-ci-centos-9-scenario004-standalone/ff3803d/ | 12:25 |
| marios | rlandy|ruck: jobs fail in ~ 10 mins you can see there 3 of them there https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-9-scenario004-standalone&skip=0 | 12:26 |
| rlandy|ruck | dviroel|ruck: ^^ can you look into the scne001 and 004 failures? | 12:26 |
| rlandy|ruck | idk which ones are still relevant | 12:26 |
| dviroel|ruck | looking already | 12:26 |
| rlandy|ruck | dviroel|ruck: thanks - left note for fmount on #tripleo as well | 12:27 |
| rlandy|ruck | in process of promoting w c9 - thanks akahat | 12:27 |
| dviroel|ruck | those ~10min failures are related to chandan's parenting change - I will not consider | 12:30 |
| chkumar|rover | post_failure you can ignore | 12:30 |
| chkumar|rover | or is there any other failures? | 12:30 |
| dviroel|ruck | chkumar|rover: those ~10m failures only happens with your patch | 12:32 |
| dviroel|ruck | chkumar|rover: https://review.opendev.org/c/openstack/tripleo-ci/+/830987/ | 12:32 |
| chkumar|rover | yes you can ignore those | 12:32 |
| dviroel|ruck | ack | 12:32 |
| dviroel|ruck | hum, changes in tripleo-ansible/roles/tripleo_cephadm could trigger scn001 and scn004 | 12:36 |
| rlandy|ruck | akahat: promotion failed - pls check it | 12:47 |
| * pojadhav stepping out for hour... | 12:55 | |
| *** rlandy|ruck is now known as rlandy|ruck|mtg | 12:57 | |
| akahat | images pused.. promoter failed at the end. | 12:58 |
| chkumar|rover | akahat: rlandy|ruck|mtg https://trunk.rdoproject.org/centos9-wallaby/current-tripleo/ | 13:00 |
| chkumar|rover | it promoted | 13:00 |
| chkumar|rover | Does image got pushed? | 13:00 |
| chkumar|rover | oh image does not got pushed there | 13:01 |
| akahat | it's got pushed: https://hub.docker.com/r/tripleowallabycentos9/openstack-base | 13:01 |
| chkumar|rover | akahat: overcloudf images | 13:02 |
| chkumar|rover | akahat: https://images.rdoproject.org/centos9/wallaby/rdo_trunk/current-tripleo/ | 13:02 |
| akahat | chkumar|rover, okay let me check | 13:02 |
| akahat | manually promoting | 13:03 |
| chkumar|rover | yes | 13:04 |
| akahat | chkumar|rover, images promoted. please check. | 13:18 |
| dviroel|ruck | c9-wallaby promoted right? https://trunk.rdoproject.org/centos9-wallaby/current-tripleo/ | 13:18 |
| chkumar|rover | thanks | 13:18 |
| chkumar|rover | dviroel|ruck: yes | 13:18 |
| dviroel|ruck | tks | 13:18 |
| akahat | yes c9-wallaby | 13:18 |
| chkumar|rover | akahat++ | 13:18 |
| dviroel|ruck | tks akahat | 13:19 |
| chkumar|rover | dviroel|ruck: stepping out for a bit, will back before meeting | 13:23 |
| dviroel|ruck | ok | 13:25 |
| soniya29 | tosky, ack | 13:45 |
| *** amoralej is now known as amoralej|lunch | 13:48 | |
| rlandy|ruck|mtg | akahat: sorry - reading back | 14:02 |
| rlandy|ruck|mtg | akahat: thank you - looks like the change is reverted on config | 14:03 |
| akahat | rlandy|ruck|mtg, promoter is restored. | 14:03 |
| rlandy|ruck|mtg | https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/40345 abandoned | 14:03 |
| *** dasm|off is now known as dasm | 14:06 | |
| dasm | marios: that's what a new comers are for. To make old people more grumpy ;) | 14:06 |
| *** rlandy|ruck|mtg is now known as rlandy|ruck | 14:09 | |
| rlandy|ruck | bhagyashris: hello ... | 14:12 |
| rlandy|ruck | bhagyashris: can you add the rr ideas for improvement to hackmed fir today's meeting | 14:12 |
| rlandy|ruck | so we can take notes | 14:12 |
| rlandy|ruck | and find some good ways to go forward | 14:12 |
| marios | dasm: :) | 14:12 |
| rlandy|ruck | bhagyashris: let's separate the ideas into scheduling and tooling an dinfo transfer | 14:13 |
| bhagyashris | rlandy|ruck, ack | 14:14 |
| rlandy|ruck | bhagyashris: thank you | 14:14 |
| marios | akahat: chkumar|rover: ysandeep: rlandy|ruck: dviroel|ruck: please review when you have time https://review.opendev.org/c/openstack/tripleo-ci/+/831543 rebased onto chkumar/akahat standalone patch ready for reviews | 14:15 |
| *** amoralej|lunch is now known as amoralej | 14:22 | |
| bhagyashris | rlandy|ruck, https://hackmd.io/FK0JgBvRQxmxqmqjZxK_PQ | 14:22 |
| rlandy|ruck | bhagyashris: thanks | 14:24 |
| bhagyashris | chkumar|rover, marios dviroel|ruck ysandeep | 14:30 |
| bhagyashris | meeting time | 14:30 |
| bhagyashris | arxcruz, | 14:30 |
| dviroel|ruck | ack | 14:31 |
| arxcruz | bhagyashris i'm there | 14:31 |
| bhagyashris | marios, chkumar|rover plz join | 14:32 |
| marios | bhagyashris: yes sorry had to deal with local issue joining now | 14:32 |
| bhagyashris | ok np | 14:32 |
| * dasm thinks it war a very useful meeting \o/ | 15:40 | |
| rlandy|ruck | all++ | 15:40 |
| rlandy|ruck | great team work | 15:40 |
| rlandy|ruck | on making a new path forward | 15:40 |
| rlandy|ruck | from monday /me will return to just rlandy | 15:40 |
| rlandy|ruck | rlandy|ruck will go bye bye | 15:41 |
| dviroel|ruck | omg | 15:41 |
| dasm | congrats rlandy|ruck ! | 15:41 |
| dasm | how long it was since you were you, rlandy|ruck ? | 15:42 |
| marios | rlandy|forever_ruck | 15:42 |
| dasm | rlandy|fornever_ruck | 15:42 |
| dasm | ^^ | 15:42 |
| marios | :D | 15:42 |
| rlandy|ruck | dasm: since weshay|ruck stopped existing | 15:42 |
| rlandy|ruck | 8 months or so | 15:42 |
| dasm | ouch. that's a pretty long shift | 15:43 |
| dviroel|ruck | chkumar|rover: will lunch now - let me know, before your eod, if you want me to continue track/investigate something you are doing now | 15:44 |
| chkumar|rover | dviroel|ruck: sc1/4 dnm patch with fix https://review.opendev.org/c/openstack/tripleo-ci/+/833337 | 15:44 |
| chkumar|rover | and nothing else | 15:45 |
| soniya29 | rlandy|ruck, congratulations :) | 15:45 |
| dviroel|ruck | chkumar|rover: ok, aware of that - tks | 15:47 |
| chkumar|rover | dviroel|ruck: see ya on monday! | 15:48 |
| dviroel|ruck | chkumar|rover: o/ | 15:48 |
| *** dviroel|ruck is now known as dviroel|ruck|lunch | 15:48 | |
| chkumar|rover | ysandeep: marios can you swap from 4th apr to 28th mar? both will be travelling | 15:50 |
| chkumar|rover | both == akahat and me | 15:50 |
| bhagyashris | chkumar|rover, hey will need to reschedule the rr schedule as per todays discussion | 15:51 |
| rlandy|ruck | dviroel|ruck|lunch: lol - will still be around | 15:51 |
| rlandy|ruck | ok - now I guess I have to do some manager stuff??? | 15:52 |
| rlandy|ruck | what is that again? | 15:52 |
| rlandy|ruck | starting by cutting back lines | 15:53 |
| rlandy|ruck | marios: ^^ ussuri patch coming in | 15:53 |
| marios | chkumar|rover: rlandy|ruck: you mean ussuri extended maintenance? | 15:55 |
| marios | chkumar|rover: sorry was for rlandy|ruck ^ | 15:55 |
| marios | chkumar|rover: checking | 15:55 |
| marios | chkumar|rover: sure i can sawp | 15:55 |
| rlandy|ruck | marios: akc - pairing back vic | 15:55 |
| rlandy|ruck | toria | 15:55 |
| rlandy|ruck | moving components to diff schedule | 15:56 |
| rlandy|ruck | etc. | 15:56 |
| marios | rlandy|ruck: victoria not yet | 15:57 |
| marios | rlandy|ruck: the release team had to make some change to allow us to move to extended maintenance, i think it merged yesterday let me check | 15:58 |
| marios | rlandy|ruck: yes https://review.opendev.org/c/openstack/releases/+/817623 | 15:58 |
| marios | rlandy|ruck: so next week i can start to propose the end of life patch | 15:59 |
| rlandy|ruck | marios: nice | 16:05 |
| rlandy|ruck | so I'll put in ussuri patches and -1 them until ready | 16:05 |
| marios | rlandy|ruck: which ussuri patches? | 16:05 |
| rlandy|ruck | remove lines | 16:05 |
| rlandy|ruck | or pair back | 16:05 |
| marios | rlandy|ruck: ah i see :) ok | 16:05 |
| marios | sure | 16:05 |
| rlandy|ruck | dasm: rcastillo: any luck on uefi boot? | 16:14 |
| *** ysandeep is now known as ysandeep|out | 16:15 | |
| dasm | not yesterday. i'm continuing today, but i think i know what's the issue | 16:16 |
| rlandy|ruck | dviroel|ruck|lunch: just rekicked https://jenkins-cloudsig-ci.apps.ocp.ci.centos.org/view/phase-1-pipelines/job/rdo_trunk-promote-victoria-centos8-current-tripleo/ | 16:16 |
| dasm | rlandy|ruck: long story short: ipxe-boot is polling 10 times and if fails, it reboots and starts over again | 16:16 |
| rlandy|ruck | if weirdo scenario 3 fails again - we will need to raise it to #rdo | 16:17 |
| dasm | uefi gets stuck on "console" | 16:17 |
| rlandy|ruck | dasm: does adding the other property help? | 16:17 |
| rlandy|ruck | dasm: if not, pls write bug/notes to steve | 16:17 |
| dasm | it didn't change anything. but i'm cross-checking with a regular boot to confirm findings | 16:17 |
| rlandy|ruck | that we have the uefi supported cloud | 16:17 |
| dasm | i'll do | 16:18 |
| rlandy|ruck | and still no boot | 16:18 |
| rlandy|ruck | dasm: Tengu also reported issues with uefi boot on master | 16:18 |
| rlandy|ruck | but iiuc, failing provision | 16:18 |
| dasm | i wouldn't be surprised if the issue is the same | 16:18 |
| rcastillo | rlandy|ruck: I managed to use overcloud-full for undercloud but I ran into a different error during provision | 16:18 |
| rcastillo | might be what Tengu is running into | 16:19 |
| rlandy|ruck | so pls sync with Tengu and see if same issue | 16:19 |
| dasm | yeah, that would be the same -- it can boot, because uefi is not asking for pxe | 16:19 |
| rlandy|ruck | rcastillo: ok | 16:19 |
| dasm | ack, will do | 16:19 |
| rlandy|ruck | rcastillo: dasm: Tengu: we need to get together and ensure we all have the same bug logged and identified | 16:19 |
| rlandy|ruck | so steve has one clear issue to work on | 16:19 |
| rlandy|ruck | rcastillo: dasm: thank you for following that down | 16:20 |
| dasm | i'll continue investigation today to get better understanding | 16:20 |
| rlandy|ruck | rcastillo: can you try a or wallaby c8 job on psi | 16:20 |
| rlandy|ruck | and see if we can get a working deploy there? | 16:20 |
| rcastillo | would rhos-dev-stage be the right tenant for this? | 16:21 |
| ade_lee__ | dviroel|ruck|lunch, hey - ping me when you get back. would like to make some changes to the fips job to test out new ceph image, and disable fips testing in ipa job | 16:27 |
| Tengu | dasm: ipxe was working until now - issue was on the boot on the installed OS (meaning after the provisioning) | 16:37 |
| Tengu | didn't test today, was working on adding "pre-provisionned node" support in my lab to be able to overcome the provisioning issues and, at last, test some OC things... | 16:38 |
| Tengu | (especially on osp-17) | 16:38 |
| dasm | Tengu: hmm.. do you mean vexxhost or psi? | 16:44 |
| Tengu | dasm: local lab | 16:44 |
| Tengu | i.e. my own hardware, with libvirt VMs | 16:45 |
| dasm | Tengu: i watched console yesterday and uefi was stuck on a shell. | 16:45 |
| dasm | ack. i'll rerun tests and i'll see if that chandes anything | 16:45 |
| Tengu | dasm: hmmmm with dracut issues about unavailable UUID? | 16:45 |
| dasm | thanks | 16:45 |
| Tengu | or was it really uefi? | 16:45 |
| dasm | it was. i built uefi locally and run it. it behaved the same way | 16:46 |
| Tengu | ok | 16:46 |
| dasm | for a comparison i ran non-uefi version | 16:46 |
| *** dviroel|ruck|lunch is now known as dviroel|ruck | 16:46 | |
| Tengu | 'k | 16:46 |
| dasm | when i started uefi, it waited 5 secs, and then entered "shell >" | 16:47 |
| Tengu | it usually fails after uefi - and usually after the image deploy on the host | 16:47 |
| dasm | that was it | 16:47 |
| Tengu | "great" | 16:47 |
| dasm | in the case of legacy ipxe, it polls for all interfaces 10 times. when it exceeds number, it reboots itself. | 16:48 |
| dasm | the behavior is different. but lemme check on that again | 16:48 |
| dasm | (good thing i have 3 monitors. i can watch 3x more windows ;) ) | 16:49 |
| Tengu | I'm off for today, we can discuss on Monday if you want. I also should get a pariing session with Steve in order to track the issues I'm seeing on my env. | 16:49 |
| dasm | sure. have a good weekend Tengu ! | 16:49 |
| dasm | sorry for interruption | 16:49 |
| Tengu | I've been hitting uefi things for the past...... 4-5 weeks? yeah. | 16:49 |
| dasm | i'm just starting again. this time with vexx | 16:50 |
| dviroel|ruck | ade_lee__: hey | 16:50 |
| marios | have a good weekend all o/ | 16:57 |
| *** marios is now known as marios|out | 16:57 | |
| *** jpena is now known as jpena|off | 17:11 | |
| ade_lee__ | dviroel|ruck, hey there -- gmeet? | 17:17 |
| dviroel|ruck | ade_lee__: sure | 17:20 |
| ade_lee__ | meet.google.com/fnq-qcao-rnm | 17:20 |
| ade_lee__ | dviroel|ruck, ^^ | 17:20 |
| rlandy|ruck | current-tripleo/2022-03-11 12:32 - rhos-17 promoted | 17:33 |
| rlandy|ruck | dviroel|ruck++ nice | 17:33 |
| dviroel|ruck | ~o~ | 17:36 |
| * dviroel|ruck going though cix cards now - lots of them there | 17:38 | |
| *** amoralej is now known as amoralej|off | 17:39 | |
| ade_lee__ | dviroel|ruck, dependent patch updated | 17:49 |
| ade_lee__ | dviroel|ruck, https://review.opendev.org/c/openstack/tripleo-ansible/+/830308 | 17:49 |
| ade_lee__ | so we should be good to go .. | 17:49 |
| dviroel|ruck | rlandy|ruck: a miracle: https://review.rdoproject.org/r/c/testproject/+/38635/15#message-999b6b81a96091412b96635e2d55688522bca7fe | 17:54 |
| dviroel|ruck | rlandy|ruck: related to this card: https://trello.com/c/F00VnyBm/2336-cixlp1956475tripleociproa-centos9-nova-migration-breaks-because-sftp-server-is-not-part-of-the-migration-wrapper | 17:55 |
| dviroel|ruck | we need to merge those two skiplists | 17:56 |
| dviroel|ruck | and close bug + card | 17:56 |
| dviroel|ruck | ade_lee__: ack - you taking care of those ceph and ipa changes? need anything? | 17:57 |
| ade_lee__ | dviroel|ruck, actually I thought you were going to do it -- but I can, they're easy enough | 17:58 |
| dviroel|ruck | ade_lee__: i can do too, np | 18:00 |
| dviroel|ruck | ade_lee__: ipa ones can be made on gerrit edit - lol | 18:00 |
| ade_lee__ | dviroel|ruck, yup - its all gerrit edit :) | 18:00 |
| ade_lee__ | dviroel|ruck, sorry -- so I'm confused? am I doing it or you? | 18:01 |
| dviroel|ruck | ade_lee__: i will do - is quay.ceph.io/ceph-ci/daemon:latest-pacific-devel the version that you want to test? | 18:02 |
| ade_lee__ | yes | 18:02 |
| * ade_lee__ lunchin | 18:08 | |
| dviroel|ruck | ade_lee__: done | 18:08 |
| ade_lee__ | looking | 18:08 |
| dviroel|ruck | ade_lee__: adde ceph container devel, disable fips on stand-on-multinode-ipa, removed python3-crypto repo | 18:09 |
| ade_lee__ | dviroel|ruck, yup - looks good | 18:09 |
| ade_lee__ | dviroel|ruck, lets see if the ci gods like us :) | 18:09 |
| dviroel|ruck | ++ | 18:09 |
| rlandy|ruck | dviroel|ruck: wow ... just wow periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-wallaby https://review.rdoproject.org/zuul/build/072ae57419704c40abfd2521309057c7 : SUCCESS in 3h 19m 03s | 18:12 |
| rlandy|ruck | dviroel|ruck: ok to merge? https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/827640 | 18:13 |
| dviroel|ruck | rlandy|ruck: yes pls | 18:21 |
| dviroel|ruck | rlandy|ruck: after that, merge skiplist for master: https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/827641/1 | 18:21 |
| rlandy|ruck | we can actually remove the whole thing | 18:23 |
| dviroel|ruck | rlandy|ruck: what about that: https://jenkins-cloudsig-ci.apps.ocp.ci.centos.org/view/phase-1-pipelines/job/rdo_trunk-promote-victoria-centos8-current-tripleo/ | 18:24 |
| rlandy|ruck | dviroel|ruck: we can ping amoralej|off or joel on that | 18:25 |
| rlandy|ruck | dviroel|ruck: you cab bug or wait until monday and ping - up to you | 18:25 |
| dviroel|ruck | rlandy|ruck: will leave a message for them - so they can start looking at their monday morning | 18:27 |
| rlandy|ruck | great - thank you | 18:29 |
| rlandy|ruck | dviroel|ruck: need help with anything? | 18:56 |
| rlandy|ruck | if not, will try clean up CIX board a little | 18:57 |
| rlandy|ruck | dviroel|ruck: is https://bugs.launchpad.net/tripleo/+bug/1964468 still relevant? | 18:59 |
| dviroel|ruck | rlandy|ruck: no - we are good now | 19:00 |
| rlandy|ruck | k - closing out | 19:00 |
| dviroel|ruck | rlandy|ruck: only thing now - jobs are failing to pull ceph daemon container - should be related to recent promotion https://review.opendev.org/q/topic:ceph_pacific_promotion | 19:01 |
| rlandy|ruck | dviroel|ruck: waiting on your ceph line :) | 19:02 |
| dviroel|ruck | rlandy|ruck: indeed | 19:02 |
| rlandy|ruck | ok - so fixes in progress | 19:02 |
| rlandy|ruck | dviroel|ruck: https://bugs.launchpad.net/tripleo/+bug/1963907 - closable? I think we are past this? | 19:03 |
| * rlandy|ruck tries to clean up a little on last ruck day | 19:03 | |
| * dviroel|ruck looking | 19:04 | |
| dviroel|ruck | rlandy|ruck: yes - we can close - related to ssh fix | 19:05 |
| rlandy|ruck | ok - closed out a bunch | 19:12 |
| rlandy|ruck | rechecked two testprojects | 19:12 |
| rlandy|ruck | we may finally be climbing out of last week's mess | 19:12 |
| dviroel|ruck | rlandy|ruck: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/832756/1/roles/extras-common/defaults/main.yml | 19:25 |
| dviroel|ruck | master merged with 6.0.7 but not yet wallaby | 19:25 |
| dviroel|ruck | if we don't updade tripleo-quickstart - master will continue to fail | 19:26 |
| dviroel|ruck | or we split this code to have a version for master and one for wallaby =/ | 19:26 |
| * dviroel|ruck will move to promotion-blocker https://bugs.launchpad.net/tripleo/+bug/1964530 = so we can track in CIX - there are too many patches involved - wont fix so fast | 19:31 | |
| * rlandy|ruck looks | 19:50 | |
| rlandy|ruck | dviroel|ruck: how far out are we from merging wallaby? | 19:51 |
| rlandy|ruck | if it's far out - split the code | 19:51 |
| rlandy|ruck | {%- elif release in ['stein','train','ussuri','victoria'] -%} | 19:52 |
| dviroel|ruck | rlandy|ruck: 16.2 will promote soon | 19:52 |
| rlandy|ruck | already release based conditions in that file | 19:52 |
| dviroel|ruck | rlandy|ruck: rechecked minutes ago | 19:52 |
| rlandy|ruck | dviroel|ruck: nice on 16.2 | 19:52 |
| rlandy|ruck | if we can merge wallaby by monday, leave it | 19:52 |
| rlandy|ruck | if notsplit ut | 19:52 |
| rlandy|ruck | split it | 19:52 |
| dviroel|ruck | rlandy|ruck: ack - already added to RR hackmd | 19:53 |
| dviroel|ruck | bug is now promotion blocker | 19:53 |
| dviroel|ruck | we also need fix that cloudnull and rcastillo were debugging | 19:53 |
| rlandy|ruck | dviroel|ruck: the molecule error? | 19:54 |
| dviroel|ruck | yes | 19:54 |
| rlandy|ruck | rcastillo had a fix for that | 19:54 |
| rlandy|ruck | Rafael Castillo proposed openstack/tripleo-ansible master: Don't specify inventory in base molecule config file https://review.opendev.org/c/openstack/tripleo-ansible/+/833452 | 19:54 |
| dviroel|ruck | ok - great | 19:54 |
| dviroel|ruck | rcastillo++ | 19:54 |
| rlandy|ruck | reading backchat on #tripleo | 19:57 |
| rlandy|ruck | rcastillo: ^^ that still he right fix? | 19:58 |
| rlandy|ruck | reading cloudnull's comments | 19:58 |
| rcastillo | rlandy|ruck: it should fix role-addition, I think they're saying role-addition shouldn't be running on these changes | 20:03 |
| rlandy|ruck | rcastillo++ ok | 20:03 |
| jm1 | rlandy|ruck: just checked, cloudnull is core on ansible openstack collection | 20:05 |
| rlandy|ruck | jm1: ok - good | 20:05 |
| rlandy|ruck | that's a start | 20:05 |
| rlandy|ruck | rcastillo: fyi https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/39665 | 20:28 |
| rlandy|ruck | removing c8 from cico | 20:28 |
| * jm1 out for today, have a nice weekend :) | 20:29 | |
| * dviroel|ruck almost weekend :) | 20:48 | |
| rlandy|ruck | dviroel|ruck: have a great weekend | 20:57 |
| rlandy|ruck | see you monday | 20:57 |
| dviroel|ruck | rlandy|ruck: you too o/ | 20:58 |
| rlandy|ruck | thanks | 20:58 |
| rlandy|ruck | dviroel|ruck: what needs to merge? | 20:59 |
| rlandy|ruck | over weekend? | 20:59 |
| rlandy|ruck | dviroel|ruck: w+ https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/832756? or you are splitting conditions? | 21:00 |
| dviroel|ruck | rlandy|ruck: ceph related ones | 21:00 |
| rlandy|ruck | dviroel|ruck: ^^ anything other than ^^ above? | 21:01 |
| dviroel|ruck | rlandy|ruck: you can W+, but depends on wallaby one yet - we need a W+ there too | 21:01 |
| dviroel|ruck | we need this one too: https://review.opendev.org/c/openstack/tripleo-ansible/+/833182/ | 21:01 |
| rlandy|ruck | dviroel|ruck: so 832810 is waiting for zuul +1 | 21:01 |
| dviroel|ruck | correct | 21:02 |
| rlandy|ruck | if zuul +1's that , w+ both? | 21:02 |
| dviroel|ruck | yes | 21:02 |
| rlandy|ruck | ok | 21:02 |
| rlandy|ruck | will be around for about another hour | 21:02 |
| rlandy|ruck | can do that | 21:02 |
| dviroel|ruck | ok | 21:03 |
| * dviroel|ruck out now | 21:18 | |
| dviroel|ruck | have a good weekend team o/ | 21:19 |
| rcastillo | dviroel|ruck: have a good weekend | 21:19 |
| *** dviroel|ruck is now known as dviroel|out | 21:19 | |
| dviroel|out | rcastillo: o/ | 21:19 |
| dasm | dviroel|out: \o | 21:22 |
| * dasm outs | 22:27 | |
| dasm | o/ have a good weekend! | 22:27 |
| *** dasm is now known as dasm|off | 22:27 | |
| rlandy|ruck | bye | 22:32 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!