| *** pojadhav|out is now known as pojadhav | 04:59 | |
| *** ysandeep|out is now known as ysandeep | 05:10 | |
| bhagyashris | hey pojadhav around? | 05:33 |
|---|---|---|
| *** bhagyashris is now known as bhagyashris|ruck | 05:33 | |
| pojadhav | bhagyashris|ruck, yes | 05:34 |
| bhagyashris|ruck | should we meet | 05:34 |
| bhagyashris|ruck | ? | 05:34 |
| pojadhav | lets meet, https://meet.google.com/qqu-bqhb-bai | 05:34 |
| ysandeep | Anyone want reviews from me :) https://drive.google.com/file/d/1mscpQPHQT52FPxgb3XXZKmiKd-UK_jEh/view?usp=sharing | 05:44 |
| reviewbot | Do you want me to add your patch to the Review list? Please type something like add to review list <your_patch> so that I can understand. Thanks. | 05:44 |
| chandankumar | ysandeep: new member is so eager and enthuastic to help us on review :-) So adorable :-) | 05:47 |
| bhagyashris|ruck | https://hackmd.io/VVzkho6RS327qkFwTv_PFA | 05:49 |
| ysandeep | chandankumar, hahaha thanks.. she is super enthuastic and pressing all the buttons on keyboard with full speed :D | 05:49 |
| pojadhav | ysandeep, so cute she is :) | 06:14 |
| chandankumar | pojadhav: https://bugs.launchpad.net/tripleo/+bug/1969613/ | 06:16 |
| chandankumar | bhagyashris|ruck: ^^ | 06:16 |
| *** pojadhav is now known as pojadhav|lunch | 06:27 | |
| ysandeep | pojadhav|lunch, thanks! | 06:27 |
| *** jpena|off is now known as jpena | 06:59 | |
| *** ysandeep is now known as ysandeep|lunch | 07:09 | |
| *** pojadhav|lunch is now known as pojadhav | 07:11 | |
| bhagyashris|ruck | chandankumar, pojadhav hey do you the failure https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master&pipeline=openstack-periodic-integration-main | 07:15 |
| chandankumar | checking | 07:15 |
| bhagyashris|ruck | it's failing from 13th april | 07:16 |
| bhagyashris|ruck | wallaby as well https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-wallaby&pipeline=openstack-periodic-integration-stable1+ | 07:16 |
| chandankumar | https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master/d8a66ff/logs/supplemental/var/log/ipaserver-install.log.txt.gz | 07:17 |
| chandankumar | bhagyashris|ruck: checking the freeipa bug | 07:17 |
| chandankumar | on hackmd | 07:17 |
| chandankumar | bhagyashris|ruck: https://bugs.launchpad.net/tripleo/+bug/1969613/ | 07:18 |
| bhagyashris|ruck | chandankumar, do we have fix for that? | 07:23 |
| chandankumar | working on it | 07:29 |
| bhagyashris|ruck | ok | 07:30 |
| bhagyashris|ruck | thanks | 07:30 |
| * bhagyashris|ruck lunch | 07:32 | |
| chandankumar | bhagyashris|ruck: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/838996 | 07:38 |
| akahat | https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/42139, https://review.rdoproject.org/r/c/config/+/42134 please take a look when you are free. | 08:13 |
| akahat | thanks !! | 08:13 |
| bhagyashris|ruck | chandankumar, ack thanks | 08:24 |
| *** ysandeep|lunch is now known as ysandeep | 09:11 | |
| ysandeep | bhagyashris|ruck, node provisioning failure in ovb third party jobs is known? | 09:26 |
| ysandeep | https://logserver.rdoproject.org/90/838990/1/openstack-check/tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001/5cad192/logs/baremetal_1_19542_0-console.log | 09:26 |
| ysandeep | https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001&project=openstack/tripleo-quickstart | 09:26 |
| bhagyashris|ruck | ysandeep, not sure just catching up the things | 09:27 |
| ysandeep | may be pojadhav knows ^^ | 09:28 |
| bhagyashris|ruck | will lokk into it | 09:28 |
| ysandeep | reviewbot: Please add in review list: https://review.opendev.org/c/openstack/tripleo-quickstart/+/838990 | 09:37 |
| reviewbot | I have added your review to the Review list | 09:37 |
| frenzy_friday | hey folks, pls add to review list https://review.rdoproject.org/r/c/config/+/42140 (quay username) | 10:07 |
| reviewbot | I have added your review to the Review list | 10:07 |
| ysandeep | pojadhav: hey o/ I see https://review.opendev.org/c/openstack/tripleo-ci/+/835460 is in review list but its depends-on patches have a -W from you, Could you please check if those -W can be removed now. | 10:26 |
| chandankumar | ysandeep: hello | 11:00 |
| chandankumar | ysandeep: https://logserver.rdoproject.org/96/838996/2/openstack-check/tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby/23c6774/logs/undercloud/home/zuul/undercloud_install.log.txt.gz | 11:00 |
| chandankumar | ysandeep: i think https://review.opendev.org/c/openstack/python-tripleoclient/+/831425 caused it | 11:00 |
| chandankumar | bhagyashris|ruck: pojadhav ^^ need a bug | 11:00 |
| chandankumar | or may be we need a promotion https://review.opendev.org/c/openstack/tripleo-validations/+/831424/ | 11:01 |
| chandankumar | for tripleo-validations | 11:01 |
| chandankumar | we need the promotion of tripleo-component https://github.com/redhat-openstack/rdoinfo/blob/master/rdo.yml#L1192 | 11:02 |
| bhagyashris|ruck | chandankumar, ack | 11:02 |
| chandankumar | bhagyashris|ruck: check wallaby line | 11:03 |
| bhagyashris|ruck | c9 or c8? | 11:03 |
| chandankumar | c9 | 11:04 |
| bhagyashris|ruck | lot many failures looking one by one | 11:04 |
| ysandeep | chandankumar, openstack-tripleo-validations.noarch 14.2.2-0.20220408135359.8f894bc.el9 @delorean-component-validation | 11:06 |
| ysandeep | i think https://review.opendev.org/c/openstack/tripleo-validations/+/831424/ is missing | 11:06 |
| ysandeep | validation promotion should fix the issue | 11:06 |
| chandankumar | RDO third party getting impacted | 11:07 |
| chandankumar | as tripleo gets pulled from current-tripleo | 11:08 |
| chandankumar | *as tripleo packages gets pulled from current | 11:09 |
| *** dviroel|out is now known as dviroel|rover | 11:14 | |
| dviroel|rover | hi @all o/ | 11:14 |
| bhagyashris|ruck | dviroel|rover, hey hi | 11:23 |
| bhagyashris|ruck | dviroel|rover, i have created hackmd for rr https://hackmd.io/VVzkho6RS327qkFwTv_PFA | 11:23 |
| chandankumar | https://review.rdoproject.org/zuul/status#838996 | 11:24 |
| dviroel|rover | bhagyashris|ruck: great - anything that you want me to help now? | 11:24 |
| bhagyashris|ruck | dviroel|rover, there are lots of failure on c9 master and wallaby currently i am looking at master one | 11:26 |
| bhagyashris|ruck | so it would be great if you look at c9 wallaby | 11:27 |
| bhagyashris|ruck | periodic integration line RDO | 11:27 |
| ysandeep | chandankumar, https://sf.hosted.upshift.rdu2.redhat.com/logs/76/398576/4/check/periodic-tripleo-ci-centos-9-bm_envC-3ctlr_1comp-featureset001-master/e47ace1/job-output.txt | 11:27 |
| dviroel|rover | bhagyashris|ruck: ack - saw lots op ipa failures on master line | 11:27 |
| bhagyashris|ruck | yeah | 11:27 |
| dviroel|rover | bhagyashris|ruck: will take a look on wallaby | 11:27 |
| bhagyashris|ruck | dviroel|rover, for ipa we have one bug https://bugs.launchpad.net/tripleo/+bug/1969613/ and fix fo that is here https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/838996 | 11:28 |
| chandankumar | https://opendev.org/openstack/tripleo-operator-ansible/src/branch/master/roles/tripleo_undercloud_install/tasks/main.yml#L32 | 11:29 |
| dviroel|rover | chandankumar: goos stuff | 11:30 |
| chandankumar | https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/roles/undercloud-deploy/tasks/main.yml#L65 | 11:30 |
| chandankumar | jm1: ^^ | 11:30 |
| chandankumar | https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/playbooks/baremetal-full-undercloud.yml#L6 | 11:31 |
| chandankumar | {{ ansible_env.HOME }} | 11:33 |
| chandankumar | https://sf.hosted.upshift.rdu2.redhat.com/logs/76/398576/4/check/periodic-tripleo-ci-centos-9-bm_envC-3ctlr_1comp-featureset001-master/e47ace1/logs/undercloud/var/log/extra/dump_variables_hostvars.yaml | 11:35 |
| ysandeep | jm1, chandankumar frenzy_friday https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-17-rhel9/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-9-bm_envB-3ctlr_1comp-featureset001-rhos-17/03057bc/logs/undercloud/var/log/extra/dump_variables_vars.yaml | 11:39 |
| chandankumar | ysandeep: https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/roles/undercloud-deploy/tasks/main.yml#L66 | 11:40 |
| chandankumar | ysandeep: basically this var | 11:43 |
| chandankumar | ansible_env.HOME | 11:43 |
| chandankumar | https://opendev.org/openstack/tripleo-operator-ansible/src/branch/master/roles/tripleo_undercloud_install/defaults/main.yml#L9 | 11:43 |
| ysandeep | chandankumar, okay, just to confirm you are suggesting to add a debug task to print ansible_env.HOME here https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/roles/undercloud-deploy/tasks/main.yml#L61 | 11:45 |
| chandankumar | ysandeep: yes | 11:45 |
| ysandeep | chandankumar: ack, thanks! | 11:45 |
| chandankumar | jm1: ^^ any other suggestion to try | 11:45 |
| chandankumar | ? | 11:45 |
| jm1 | ysandeep, chandankumar: ansible_env.HOME is root, else it would not try to write to /root/... Question is, why is it suddenly trying to run this role as root? something up the call chain something has changed. | 11:48 |
| jm1 | ysandeep: where can i find the job def? | 11:49 |
| ysandeep | jm1: This is a new job, I mean rhel 17 baremetal jobs are already working, I am writing a new job for master centos9 stream. | 11:50 |
| ysandeep | jm1: https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/398576/4/zuul.d/baremetal-jobs.yaml | 11:50 |
| *** ysandeep is now known as ysandeep|afk | 11:58 | |
| jm1 | ysandeep: or maybe ansible has changed its behavior. the successful job has ansible-core 2.12.2, while the failing one has 2.12.4 | 11:59 |
| dviroel|rover | bhagyashris|ruck: with the ipa fix, we can (probably) promote wallaby-cs9 with a previous hash (yesterday) | 12:06 |
| dviroel|rover | bhagyashris|ruck: running more tests now | 12:06 |
| bhagyashris|ruck | dviroel|rover, ack | 12:06 |
| dviroel|rover | bhagyashris|ruck: only fs020 is missing to promote a train hash, rerunning now | 12:08 |
| bhagyashris|ruck | dviroel|rover, not sure but just observe one thing most of the jobs are failing with bunch tempest test failure (it's very hard to add in the skip and check the consistency of failure ) | 12:08 |
| bhagyashris|ruck | chandankumar, ^ | 12:09 |
| bhagyashris|ruck | with bunch of tempest test failure ... | 12:09 |
| dviroel|rover | bhagyashris|ruck: yes, i see that in master | 12:09 |
| bhagyashris|ruck | yeah | 12:09 |
| dviroel|rover | chandankumar: "Failed to establish a new connection: [Errno 111] Connection refused'" seems to be everywhere | 12:13 |
| dviroel|rover | chandankumar: sorry, it is for bhagyashris|ruck ^ | 12:13 |
| chandankumar | dviroel|rover: for node provisioning issue | 12:14 |
| chandankumar | https://logserver.rdoproject.org/96/838996/2/openstack-check/tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001/ccec871/logs/baremetal_2_45854_0-console.log | 12:14 |
| chandankumar | [H[J[1;1Herror: ../../grub-core/disk/diskfilter.c:916:diskfilter writes are not | 12:14 |
| chandankumar | is this seen in other jobs also? | 12:15 |
| bhagyashris|ruck | dviroel|rover, yeah. we have fix in tripleo-validation will need to wait for validation component to promote | 12:15 |
| * bhagyashris|ruck tea brb | 12:16 | |
| dviroel|rover | chandankumar: tks for that info - not sure yet, will keep an eye on it | 12:18 |
| chandankumar | https://review.opendev.org/c/openstack/diskimage-builder/+/826976 merged in dib | 12:18 |
| dviroel|rover | chandankumar: on network component line, doesn't seems to be same issue https://logserver.rdoproject.org/26/41926/5/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-network-master/b77578c/logs/undercloud/home/zuul/overcloud_node_provision.log.txt.gz | 12:34 |
| jm1 | ysandeep|afk: can you force an older version of ansible, e.g. ansible-core 2.12.2? | 12:37 |
| * pojadhav reading back | 12:50 | |
| pojadhav | ysandeep|afk, there are some discussion happened in the scrum like marios will take action on tripleo-upgrade patches, i think that was missed to tear down for ussuri. We need to wait for marios for this patch : https://review.opendev.org/c/openstack/tripleo-ci/+/835460 | 12:53 |
| *** dasm|off is now known as dasm | 13:16 | |
| dasm | o' | 13:16 |
| *** ysandeep|afk is now known as ysandeep | 13:16 | |
| ysandeep | jm1: thanks for pointers, Will check if downgrading tripleo-ansible is an option. | 13:17 |
| ysandeep | chandankumar, dviroel|rover we should add some formating/layout to the tripleo-all hands ppt. | 13:18 |
| ysandeep | chandankumar, do you want to meet today so that we can fill dviroel|rover about the plan. | 13:21 |
| chandankumar | ysandeep: yes sure | 13:21 |
| dviroel|rover | ysandeep: chandankumar: good idea :) | 13:21 |
| ysandeep | dviroel|rover, chandankumar do you want to meet now? | 13:22 |
| chandankumar | works for me | 13:22 |
| dviroel|rover | sure | 13:23 |
| *** pojadhav is now known as pojadhav|afk | 13:23 | |
| ysandeep | meet.google.com/xjk-djtj-tgk | 13:24 |
| ysandeep | chandankumar, dviroel|rover ^^ | 13:24 |
| bhagyashris|ruck | dviroel|rover, hey i am rerunning the 16.2 failed jobs here https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status/change/315285,2 | 13:38 |
| bhagyashris|ruck | two of them passed | 13:38 |
| bhagyashris|ruck | you can killed yours if possible https://code.engineering.redhat.com/gerrit/c/testproject/+/310670 | 13:38 |
| bhagyashris|ruck | as we have resource contraint issue at downstream side | 13:39 |
| bhagyashris|ruck | constraint* | 13:42 |
| dviroel|rover | bhagyashris|ruck: sorry, missed that one there, but those for 17 are still relevant, no? | 13:44 |
| bhagyashris|ruck | yeah rhel8 on rhos17 | 13:46 |
| bhagyashris|ruck | dviroel|rover, ok then no need to discard | 13:47 |
| bhagyashris|ruck | i missed rhos17 on rhel8 | 13:47 |
| dviroel|rover | ack, np | 13:48 |
| bhagyashris|ruck | i was focused on rhel8-rhos16.2 and rhos17-rhel9 | 13:48 |
| dviroel|rover | chandankumar: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/838996/3/roles/ipa-multinode/tasks/ipaserver-subnode-install.yml#45 | 13:48 |
| frenzy_friday | looks like the container push jobs are able to push to both master and train quay https://review.rdoproject.org/r/c/rdo-jobs/+/41528 \O/ | 13:56 |
| dpawlik | o/\o | 13:59 |
| arxcruz | no happy friday today i guess :) | 14:01 |
| *** rcastillo|rover is now known as rcastillo | 14:03 | |
| rcastillo | o/ | 14:03 |
| dviroel|rover | chandankumar: hey, can you update https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/838996/ before your end of day? | 14:29 |
| chandankumar | dviroel|rover: done | 14:31 |
| bhagyashris|ruck | dviroel|rover, hey i have updated the todays rr stuff from my end https://hackmd.io/VVzkho6RS327qkFwTv_PFA anything you want from my side before i am closing out the day | 14:32 |
| dviroel|rover | chandankumar: tks | 14:32 |
| dviroel|rover | bhagyashris|ruck: thanks, i don't think so, i believe that fixing ipa issue will let us promote something, lets see, thanks | 14:33 |
| bhagyashris|ruck | yesterday in planning we decided that i will take hand off from pooja and yoy will take hand off from rafael | 14:34 |
| bhagyashris|ruck | so if you need anything then contact rafael | 14:34 |
| bhagyashris|ruck | or else i am also ready to give hand off | 14:34 |
| dviroel|rover | bhagyashris|ruck: don't worry, i can ping rcastillo if needed, but i was following up RR during the week | 14:37 |
| dviroel|rover | no new so far I think | 14:37 |
| dviroel|rover | new* | 14:37 |
| * dviroel|rover news* | 14:38 | |
| chandankumar | dasm: thank you :-) | 14:39 |
| dviroel|rover | chandankumar: running tp | 14:39 |
| dviroel|rover | lets see | 14:39 |
| chandankumar | dviroel|rover: feel free to update the patch | 14:39 |
| ysandeep | chandankumar, dviroel|rover dasm thank you guys! | 14:39 |
| jm1 | ysandeep: you have ssh access to this failing machine, right? you could just manually downgrade ansible-core (only this package!) to 2.12.2 and try to run the playbook again as written in log output. | 14:39 |
| dviroel|rover | chandankumar: ack, dasm, rcastillo and I can review and merge if it works | 14:40 |
| jm1 | ysandeep: in case you did not already fix it 🙈 | 14:40 |
| ysandeep | jm1: thanks for suggestions, I will try that on Monday. | 14:42 |
| * ysandeep out, Have a good weekend everyone. | 14:43 | |
| chandankumar | see ya, happy weekend! | 14:43 |
| *** ysandeep is now known as ysandeep|out | 14:43 | |
| jm1 | ysandeep: ack. i had a look through the commit history and there might be one patch which could have caused this, but i suspect its not in ansible. but worth to check. | 14:43 |
| jm1 | ysandeep|out: have a nice weekend :) | 14:43 |
| *** dviroel|rover is now known as dviroel|rover|lunch | 15:09 | |
| bhagyashris|ruck | dviroel|rover|lunch, ack thanks :) | 15:21 |
| *** jpena is now known as jpena|off | 15:55 | |
| *** dviroel|rover|lunch is now known as dviroel|rover | 16:14 | |
| *** amoralej is now known as amoralej|off | 16:14 | |
| dviroel|rover | dasm: rcastillo: please review https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/838996 - wanna merge this to unblock some jobs | 18:33 |
| rcastillo | LGTM, the command looks fine on my machine too | 18:36 |
| dviroel|rover | dasm: if you has a chance to see and find something wrong, i can remove the W | 18:45 |
| rcastillo | sent in the patch to switch off ping for testing dns https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/839094 | 18:58 |
| dviroel|rover | rcastillo: nice, will take a look | 19:04 |
| dviroel|rover | rcastillo: can you add to related CIX card, pls? that we are working on a proper fix | 19:05 |
| rcastillo | dviroel|rover: ack, done | 19:08 |
| dviroel|rover | tksd | 19:08 |
| dasm | dviroel|rover: sorry, i didn't see your ping. One typo which can be ignored. But another thing which might be still an issue: you're downgrading the package after installing IPA. | 19:14 |
| dasm | Wouldn't it be a problem> | 19:14 |
| dviroel|rover | dasm: as far as I can see, both ways are installing before IPA install | 19:19 |
| dviroel|rover | 1. https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/838996/4/roles/ipa-multinode/tasks/ipaserver-subnode-install.yml#102 | 19:20 |
| dviroel|rover | 2. https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/838996/4/roles/freeipa-setup/templates/deploy_freeipa.sh.j2#65 | 19:20 |
| dasm | in that case, no worries | 19:20 |
| dviroel|rover | can't believe that i will promote something today on upstream | 20:11 |
| * dviroel|rover biab | 20:30 | |
| *** dviroel|rover is now known as dviroel|rover|biab | 20:31 | |
| *** dviroel|rover|biab is now known as dviroel|rover | 21:02 | |
| dviroel|rover | train 2022-04-22 20:37 | 21:02 |
| dviroel|rover | rdo zuul is stuck | 21:16 |
| dviroel|rover | should be back soon | 21:16 |
| * dviroel|rover biab | 21:16 | |
| *** dviroel|rover is now known as dviroel|rover|biab | 21:16 | |
| *** dasm is now known as dasm|off | 22:03 | |
| dasm|off | o/ | 22:03 |
| rcastillo | have a good weekend dasm|off | 22:03 |
| dasm|off | you too rcastillo | 22:03 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!