| *** rlandy|ruck is now known as rlandy|out | 00:54 | |
| *** ysandeep|out is now known as ysandeep | 05:27 | |
| jfrancoa | ysandeep: good morning Sandeep, sorry to ask again but I need to ask you again to hold the node from this job https://review.rdoproject.org/r/c/testproject/+/36859 ... could you do it, please? we were pretty sure we found out where the issue lays at, but the job is not collecting any logs at all now...not even in the rdo testproject | 05:56 |
|---|---|---|
| jfrancoa | ysandeep: so I would like to jump into the node and have a look | 05:57 |
| ysandeep | jfrancoa, sure, let me see how we can push your keys using testproject itself. | 05:58 |
| jfrancoa | ysandeep: thanks a lot | 05:58 |
| jfrancoa | ysandeep++ | 05:58 |
| ysandeep | jfrancoa, can we remove the node from hold which we put yesterday? | 05:59 |
| jfrancoa | ysandeep: yes please | 05:59 |
| ysandeep | ack, thanks! | 05:59 |
| ysandeep | jfrancoa, I have added node on hold and pinged infra to add your keys, Once they add your keys.. I will ping you credentials.. | 06:13 |
| jfrancoa | ysandeep: thanks a lot, let's see if I can find out this time whats going on | 06:15 |
| *** sshnaidm|afk is now known as sshnaidm | 06:57 | |
| ysandeep | folks o/ please add https://review.rdoproject.org/r/c/rdo-jobs/+/37139 on your review list | 07:16 |
| bhagyashris | folks kindly add into your review list https://review.rdoproject.org/r/c/rdo-jobs/+/36994 thanks :) | 07:22 |
| *** ysandeep is now known as ysandeep|lunch | 07:26 | |
| *** amoralej|off is now known as amoralej | 08:15 | |
| *** ysandeep|lunch is now known as ysandeep | 08:41 | |
| akahat | ysandeep, o/ | 09:09 |
| ysandeep | hello akahat | 09:10 |
| akahat | ysandeep, looks like rlandy found way to run sc-10 in downstream. | 09:12 |
| ysandeep | akahat, :) I saw her comments on chat, but was not sure where she left off, Lets wait for her to come online then we can discuss | 09:13 |
| akahat | ysandeep, sure np. | 09:14 |
| chandankumar | anbanerj|ruck: hello, please have a look https://review.opendev.org/c/openstack/tripleo-ci/+/780950, may be rebase will help, when free, thanks! | 09:22 |
| marios | chandankumar: looks like that revert helped ? | 09:24 |
| marios | chandankumar: i think provision passed now in your test | 09:24 |
| marios | https://logserver.rdoproject.org/13/36713/7/check/periodic-tripleo-ci-centos-9-ovb-1ctlr_1comp-featureset002-master/3cef704/logs/undercloud/home/zuul/overcloud_node_provision.log.txt.gz | 09:24 |
| chandankumar | marios: yes, but there are objections from Ian | 09:25 |
| marios | chandankumar: k thanks i'll add it for fs1 for now | 09:25 |
| chandankumar | marios: https://bugs.launchpad.net/tripleo/+bug/1954456/comments/7 | 09:25 |
| marios | chandankumar: ah but fs1 isn't building though hmm | 09:25 |
| marios | chandankumar: no it is actually ... k | 09:26 |
| *** ysandeep is now known as ysandeep|afk | 09:35 | |
| anbanerj|ruck | chandankumar, ack, checking | 09:37 |
| chandankumar | anbanerj|ruck: please recheck this patch also https://review.rdoproject.org/r/c/testproject/+/36508 so that we can clear sc7 cs9 | 09:40 |
| jpodivin | I've found a duplicate definition for one of VF jobs, no idea how we missed it, but it didn't cause any problems so far. https://review.rdoproject.org/r/c/rdo-jobs/+/37196 | 09:41 |
| anbanerj|ruck | chandankumar, ack, rebased the job def and restarted the testP | 09:45 |
| chandankumar | anbanerj|ruck: thanks! | 09:45 |
| anbanerj|ruck | hey chandankumar, I saw this patch https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/821106 . But the C8 standalone-upgrade jobs ussuri/victoria are again hitting https://bugs.launchpad.net/tripleo/+bug/1953326 | 10:11 |
| anbanerj|ruck | chandankumar, https://logserver.rdoproject.org/95/37195/1/check/periodic-tripleo-ci-centos-8-standalone-upgrade-ussuri/a99e761/job-output.txt I can see Downgrade pcs running. Why are we running Return current pcs version after that? | 10:11 |
| chandankumar | anbanerj|ruck: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/821521 | 10:12 |
| chandankumar | anbanerj|ruck: can you test the ussuri/victoria job with above patch? | 10:13 |
| anbanerj|ruck | thanks, putting depends on | 10:13 |
| jbadiapa | hi, I've just tried to deploy tripleo-quickstart with cs9. | 10:21 |
| jbadiapa | but it failed | 10:21 |
| jbadiapa | on the provisioning stage | 10:21 |
| jbadiapa | It failed on the "modify-image : Run virt-customize on the provided image" | 10:21 |
| jbadiapa | the error is "command: mount '-o' 'ro' '/dev/sda' '/sysroot//'" | 10:22 |
| jbadiapa | '[ 3.024065] XFS (sda): SB validate failed with error -22.' | 10:22 |
| jbadiapa | is there any package that is needed to be able to run cs9 as guest on cs8 host? | 10:23 |
| jbadiapa | should I file a launchpad? | 10:23 |
| marios | jbadiapa: o/ hey i don't know if it is the same issue but, we had some problems with virt-customize for cs9 (no solution yet/it has been low priority) bug there... https://bugs.launchpad.net/tripleo/+bug/1950453 ... in that case the command it tries to run is https://opendev.org/openstack/tripleo-ci/src/commit/e6a036545aa0d3dd543562703b0f87f3267dacd2/roles/oooci-build-images/tasks/image_sanity.yaml#L24 ... | 10:24 |
| marios | fyi | 10:25 |
| jbadiapa | marios, thanks | 10:26 |
| *** ysandeep|afk is now known as ysandeep | 10:32 | |
| *** dviroel|rover|afk is now known as dviroel|rover | 11:01 | |
| chandankumar | sshnaidm: I am creating a new review on top of yours to put remaining standalone jobs | 11:11 |
| *** rlandy is now known as rlandy|ruck | 11:13 | |
| dviroel|rover | oh, train c7 promoted | 11:16 |
| rlandy|ruck | dviroel|rover: ussuri still not | 11:17 |
| rlandy|ruck | dviroel|rover: did you rekick the testproject? | 11:17 |
| dviroel|rover | rlandy|ruck: yes, yesterday, but with no sucess | 11:18 |
| dviroel|rover | anbanerj|ruck: is tp'ing atm | 11:18 |
| dviroel|rover | anbanerj|ruck: i think that you already tested with pcs revert too, right? | 11:19 |
| rlandy|ruck | we may need to pull marios back into this one | 11:19 |
| dviroel|rover | rlandy|ruck: ^ | 11:19 |
| rlandy|ruck | no luck | 11:19 |
| anbanerj|ruck | dviroel|rover, rlandy|ruck Hey 0/ Yes, testing with pcs revert - https://review.rdoproject.org/r/c/testproject/+/37195 | 11:19 |
| marios | rlandy|ruck: o/ ? ok i can have a look after i finish what i'm currently doing | 11:20 |
| dviroel|rover | rlandy|ruck: will dig into error to see if i can spot something. Try to check diffs between friday and monday, since it was passing on friday | 11:21 |
| marios | ussuri blocked again? | 11:21 |
| dviroel|rover | marios: standalone-upgrade-ussuri is fighting a war with us | 11:22 |
| marios | sounds about right . k dviroel|rover i am working on arxcruz upgrade patch right now but will have a look after this | 11:23 |
| dviroel|rover | marios: sure, tks | 11:23 |
| *** jpena|off is now known as jpena | 11:42 | |
| sshnaidm | chandankumar, replied in https://review.opendev.org/c/openstack/tripleo-ci/+/821241 - we don't vote now, so it's fine to run on wallaby | 11:43 |
| sshnaidm | chandankumar, and we can know when it's ok if job starts passing | 11:43 |
| jfrancoa | anbanerj|ruck: hey, as the ruck...have you seen any ussury deployment work fail with https://bugs.launchpad.net/tripleo/+bug/1871703 recently? I've triggered my upgrades job from ussuri to victoria and it has failed due to exactly that bug: https://zuul.opendev.org/t/openstack/build/d0dd2d838f9840d2bcc42dcd58580779/log/logs/undercloud/home/zuul/standalone_deploy.log | 11:48 |
| jfrancoa | during ussuri deployment | 11:48 |
| anbanerj|ruck | jfrancoa, not yet, these jobs are failing today with https://bugs.launchpad.net/tripleo/+bug/1953326. But lemme check | 11:51 |
| rlandy|ruck | bhagyashris: pls put back the planning meeting for thursday | 11:52 |
| rlandy|ruck | it's our next year's planning | 11:52 |
| rlandy|ruck | ^^ | 11:52 |
| rlandy|ruck | anbanerj|ruck: failed again | 11:56 |
| rlandy|ruck | jfrancoa: we;ve been going on a few bugs for ussuri upgrades | 11:57 |
| rlandy|ruck | blocked our promotion for 13 days | 11:57 |
| rlandy|ruck | only the standalone | 11:57 |
| rlandy|ruck | the current failure is logging now | 11:58 |
| rlandy|ruck | ussuri itself should be passing | 11:58 |
| anbanerj|ruck | rlandy|ruck, checking after the logs are generated | 11:58 |
| jfrancoa | rlandy|ruck: I saw that you had to revert the pcs package (I though for a moment that was the issue I was facing, but I'm still stuck on finding out why are my attempts to improve the stanadlone-upgrade failing and generating so much logs that the disk gets full) | 11:59 |
| jfrancoa | everytime I say, I think I got it! It's not it.. :-D | 11:59 |
| rlandy|ruck | jfrancoa: https://trunk.rdoproject.org/centos8-ussuri/component/tripleo/current/ - has marios revert on what was the previous bug | 12:00 |
| rlandy|ruck | so this testproject run should include both | 12:00 |
| bhagyashris | rlandy|ruck, ack | 12:01 |
| rlandy|ruck | jfrancoa: anbanerj|ruck: ussuri just kicked again | 12:02 |
| rlandy|ruck | the whole line | 12:02 |
| rlandy|ruck | dviroel|rover: do pcs was upgraded | 12:03 |
| rlandy|ruck | so we need to revert that workaround | 12:03 |
| rlandy|ruck | anbanerj|ruck included that patch | 12:03 |
| jfrancoa | rlandy|ruck: oh..good to know. I'll wait for the revert and do the same on my changes | 12:04 |
| jfrancoa | rlandy|ruck: thanks for hte heads up | 12:04 |
| rlandy|ruck | jfrancoa: even with the revert we have a problem | 12:04 |
| dviroel|rover | it seems that even with the pcs downgrade fix, we have the jobs running with pcs-0.10.11-2.el8.x86_64 (instead of pcs-0.10.11-1.el8.x86_64) | 12:04 |
| *** soniya29 is now known as soniya29|afk | 12:04 | |
| dviroel|rover | which version we have with the revert? checking.. | 12:05 |
| anbanerj|ruck | dviroel|rover, with revert pcs-0.10.12 | 12:05 |
| rlandy|ruck | dviroel|rover: anbanerj|ruck: jfrancoa: checking those logs now | 12:05 |
| anbanerj|ruck | https://logserver.rdoproject.org/95/37195/3/check/periodic-tripleo-ci-centos-8-standalone-upgrade-ussuri/48079dd/logs/undercloud/var/log/extra/rpm-list.txt.gz | 12:05 |
| rlandy|ruck | https://logserver.rdoproject.org/95/37195/3/check/periodic-tripleo-ci-centos-8-standalone-upgrade-ussuri/48079dd/logs/undercloud/home/zuul/standalone_upgrade.log.txt.gz | 12:06 |
| rlandy|ruck | still issue has pcs | 12:06 |
| dviroel|rover | pcs-0.10.12-1.el8.x86_64 is the newer, still not working | 12:06 |
| rlandy|ruck | dviroel|rover: anbanerj|ruck: instead of reverting the workaround | 12:07 |
| rlandy|ruck | we need to change it | 12:07 |
| rlandy|ruck | to install the exact version we know works | 12:07 |
| anbanerj|ruck | I tried with version https://logserver.rdoproject.org/95/37195/2/check/periodic-tripleo-ci-centos-8-standalone-upgrade-ussuri/12dd58b/logs/undercloud/var/log/extra/rpm-list.txt.gz as well | 12:07 |
| anbanerj|ruck | in case it was fixed | 12:08 |
| rlandy|ruck | anbanerj|ruck: dviroel|rover: let's meet | 12:08 |
| anbanerj|ruck | checking which version worked last | 12:08 |
| rlandy|ruck | https://meet.google.com/smf-tjbw-wjt?pli=1&authuser=0 | 12:08 |
| rlandy|ruck | anbanerj|ruck: dviroel|rover: ^^ | 12:08 |
| rlandy|ruck | jfrancoa: ^^ if you are interested | 12:09 |
| rlandy|ruck | pcs-0.10.11-1.el8.x86_64 | 12:15 |
| bhagyashris | rlandy|ruck, hey for logrotate testing is it ok to do the ussuri promotion using old promoter server? | 12:18 |
| bhagyashris | old promoter server http://10.0.148.74/promoter_logs/ | 12:20 |
| rlandy|ruck | bhagyashris: pls hold on using ussuri - we are trying to get that release promoted | 12:20 |
| rlandy|ruck | can you try victoria? | 12:20 |
| *** ysandeep is now known as ysandeep|brb | 12:21 | |
| bhagyashris | rlandy|ruck, np i will try victoria | 12:21 |
| rlandy|ruck | bhagyashris: thanks | 12:22 |
| rlandy|ruck | bhagyashris: can we move the retro an hour later? | 12:24 |
| rlandy|ruck | very early for daruisz and rafael | 12:24 |
| bhagyashris | rlandy|ruck, ok will do that | 12:24 |
| rlandy|ruck | bhagyashris: also - do we usually use 2 hours for planning? | 12:25 |
| rlandy|ruck | 3 hours? | 12:25 |
| bhagyashris | 3 hr | 12:25 |
| rlandy|ruck | bhagyashris: do we usually use all that time? | 12:25 |
| bhagyashris | nope | 12:25 |
| rlandy|ruck | bhagyashris: do you think we could do it in 2 hours? | 12:25 |
| rlandy|ruck | and start an hour later? | 12:26 |
| bhagyashris | in reality it's just 2 hrs | 12:26 |
| bhagyashris | rlandy|ruck, sure | 12:26 |
| rlandy|ruck | bhagyashris: thanks - just trying to accommodate the newer people | 12:26 |
| rlandy|ruck | ysandeep|brb: hey - pls ping when you have time to sync | 12:27 |
| bhagyashris | rlandy|ruck, yeah | 12:27 |
| rlandy|ruck | bhagyashris: lastly - we added c9 topics to today's community call | 12:27 |
| rlandy|ruck | and invited Tengu to join us | 12:27 |
| bhagyashris | rlandy|ruck, ack | 12:27 |
| rlandy|ruck | akahat: so think we have a way to run sc010 | 12:28 |
| rlandy|ruck | just need to check with ysandeep|brb that we are not hosing anything | 12:28 |
| rlandy|ruck | bhagyashris++ thank you | 12:29 |
| * Tengu waves | 12:31 | |
| dviroel|rover | rlandy|ruck: anbanerj|ruck https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/821699 | 12:38 |
| dviroel|rover | sanity check please | 12:38 |
| dviroel|rover | anbanerj|ruck: do you mind to testproject in one of your patches? | 12:38 |
| anbanerj|ruck | dviroel|rover, thanks! adding to the original tp of ussuri | 12:39 |
| rlandy|ruck | dviroel|rover++ nice | 12:41 |
| rlandy|ruck | thank you | 12:41 |
| anbanerj|ruck | https://review.rdoproject.org/r/c/testproject/+/37195 - testing here | 12:41 |
| * anbanerj|ruck tea | 12:43 | |
| dviroel|rover | tks | 12:43 |
| * dviroel|rover coffee | 12:44 | |
| * bhagyashris tea | 12:48 | |
| *** ysandeep|brb is now known as ysandeep | 12:48 | |
| ysandeep | rlandy|ruck, back | 12:49 |
| rlandy|ruck | ysandeep: let's chat for 10 ... https://meet.google.com/ydx-sfhb-bpn?pli=1&authuser=0 | 12:50 |
| Tengu | rlandy|ruck: fyi, the next LADA release will support package lookup for osp17-rhel9 | 13:03 |
| rlandy|ruck | nice | 13:03 |
| Tengu | and this kind of things will be based on ospinfo, using a new "translator" I've provided for PCD (osp_release.yml). | 13:03 |
| Tengu | took me a while, with a dead brain, but.... it's almost in. | 13:04 |
| dviroel|rover | anbanerj|ruck: https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-8-standalone-on-multinode-ipa - 3 consecutive failures, same package issue | 13:04 |
| anbanerj|ruck | dviroel|rover, checking | 13:06 |
| marios | dviroel|rover: so do we have a new bug for ussuri or is it still this one https://bugs.launchpad.net/tripleo/+bug/1953326 | 13:10 |
| marios | dviroel|rover: i see... the workaround fails and we are trying now with https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/821699 specific version instead | 13:10 |
| dviroel|rover | marios: it is still pcs issue, now we created a patch that downgrades to a specific pcs version | 13:10 |
| dviroel|rover | marios: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/821699 | 13:11 |
| dviroel|rover | yes | 13:11 |
| dviroel|rover | anbanerj|ruck has a testproject running | 13:11 |
| jfrancoa | rlandy|ruck: sorry, I was having lunch...I missed the party | 13:13 |
| marios | anbanerj|ruck: sanity check did you mean to use that specific hash (i guess you need it for promotion?) dlrn_hash_tag: 0c303674a982c099b14bcb4198325ec7 | 13:15 |
| marios | dviroel|rover: anbanerj|ruck: so you mean, it failed in the periodic line, even though we have the workaround in place? i guess this is why the revert was posted, and finally we are instead trying to downgrade to a specific version | 13:15 |
| *** amoralej is now known as amoralej|lunch | 13:16 | |
| anbanerj|ruck | marios, yep - 0c30.. is the current hash under test. I copied directly from the rick rover script :D | 13:16 |
| marios | anbanerj|ruck: thanks i see it http://promoter.rdoproject.org/promoter_logs/centos8_ussuri.log | 13:17 |
| dviroel|rover | marios: the revert that test newer versions didn't work | 13:18 |
| anbanerj|ruck | marios, with the downgrade it passed in the original (sandeeps) testproject - there pcs was 10.11-1. Right now with downgrade the version is 10.11-2 which is not working. With revert the version is 10.12-1 which is not working either. So we are trying to go back to 10.11-1 | 13:18 |
| marios | anbanerj|ruck: dviroel|rover: k so it sounds like it didnt' fail in the periodic line just in testproject (i.e. the workaround is still working right now?) | 13:22 |
| marios | anbanerj|ruck: dviroel|rover: any idea why it didjn't run yesterday though? https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-standalone-upgrade-ussuri&pipeline=openstack-periodic-integration-stable3 i see run from 12th (failing on different issue) | 13:22 |
| marios | anbanerj|ruck: dviroel|rover: i was just trying to find if it fails in periodic with the workaround but sounds like no | 13:22 |
| * marios fetch coffee | 13:22 | |
| anbanerj|ruck | marios, yes, the job initially failed on a different issue in the periodic line. I testprojected the job after https://review.opendev.org/c/openstack/tripleo-heat-templates/+/821264 i merged - then it started failing on the pcs issue | 13:23 |
| bhagyashris | arxcruz, sshnaidm, rlandy, rlandy|ruck, marios, ysandeep, bhagyashris, svyas, soniya29, pojadhav, akahat, chandankumar, frenzy_friday, anbanerj, dviroel, rcastillo, dasm | 13:27 |
| bhagyashris | Tengu, | 13:27 |
| bhagyashris | TripleO CI community call in 3 mins | 13:27 |
| Tengu | yup | 13:27 |
| Tengu | preparing my side :) | 13:27 |
| bhagyashris | folks feel free to add topic at line no 29 here https://hackmd.io/MMg4WDbYSqOQUhU2Kj8zNg?both#2021-12-14-Community-Call | 13:28 |
| bhagyashris | https://meet.google.com/bqx-xwht-wky?authuser=0 | 13:28 |
| dviroel|rover | marios: yesterday rdo zuul was restarted, maybe ussuri was affected | 13:28 |
| marios | dviroel|rover: thanks | 13:29 |
| ysandeep | rlandy|ruck, approach i tried worked, details in https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/298826/2#message-f367e18389b84a7a2a910a87316a7260cfa25e75 , lets merge that if you agree | 13:29 |
| marios | anbanerj|ruck: i see.. so it *would* have failed in the next periodic run i guess ? | 13:29 |
| marios | :D | 13:29 |
| ysandeep | bhagyashris, will join few mins late today | 13:30 |
| bhagyashris | ysandeep, np | 13:30 |
| anbanerj|ruck | marios, yes, i think so | 13:30 |
| bhagyashris | rlandy|ruck, pojadhav- anbanerj|ruck chandankumar akahat | 13:30 |
| bhagyashris | ^ | 13:31 |
| bhagyashris | soniya29|afk, ^ | 13:31 |
| Tengu | chandankumar: sorry, forgot to re-check about that nsenter thing -.-. As said, I'll take some time now. | 13:48 |
| chandankumar | Tengu: I am reading about nsenter | 13:48 |
| Tengu | for that I need a local reproducer though. | 13:48 |
| Tengu | chandankumar: there's a "--cgroup[=file]" param, but I don't think this is what we're looking for. | 13:49 |
| Tengu | chandankumar: do you happen to have a simple reproducer? | 13:49 |
| chandankumar | Tengu: Nope | 13:49 |
| Tengu | dang.. | 13:49 |
| Tengu | getting a centos 9 env shortly. | 13:50 |
| rlandy|ruck | anbanerj|ruck: hey - any word on what's killing master/wallaby? | 13:50 |
| rlandy|ruck | lots of failures there | 13:50 |
| rlandy|ruck | network line is also red | 13:51 |
| rlandy|ruck | dviroel|rover: ^^ | 13:51 |
| anbanerj|ruck | master errors: | 13:51 |
| anbanerj|ruck | - periodic-tripleo-ci-centos-8-scenario002-standalone-master - barbican_tempest_plugin.tests.scenario.test_image_signing.ImageSigningSnapshotTest, .ImageSigningTest, test_volume_encryption.VolumeEncryptionTest failed - server didnt reach active state - maybe this is a one time issue | 13:51 |
| anbanerj|ruck | - periodic-tripleo-ci-centos-8-scenario001-standalone-master - couldnt reach keystone endpoint - but no errors in keystone containet logs - rerunning | 13:51 |
| anbanerj|ruck | - periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-master - package sssd-common-pac-2.5.2-2.el8_5.1.x86_64 requires libndr.so.1()(64bit), but none of the providers can be installed", "2021-12-14 10:05:29 | - package sssd-common-pac-2.5.2-2.el8_5.1.x86_64 requires libndr.so.1(NDR_0.0.1)(64bit), but none of the providers can be installed", "2021-12-14 10:05:29 | - cannot install both samba-client-libs-4.15.2-2.el8.x86_64 and | 13:51 |
| anbanerj|ruck | samba-client-libs-4.14.5-2.el8.x86_64", "2021-12-14 10:05:29 | | 13:51 |
| anbanerj|ruck | - cannot install both libipa_hbac-2.4.0-9.el8.x86_64 and libipa_hbac-2.5.2-2.el8_5.1.x86_64", "2021-12-14 10:05:29 | - cannot install both libipa_hbac-2.4.0-9.el8_4.1.x86_64 and libipa_hbac-2.5.2-2.el8_5.1.x86_64", "2021-12-14 10:05:29 | 13:51 |
| anbanerj|ruck | ^ this might be a real issue | 13:51 |
| anbanerj|ruck | testproj: https://review.rdoproject.org/r/c/testproject/+/37199 | 13:51 |
| chandankumar | Tengu: may be recheck the testproject and get a node hold? | 13:51 |
| * anbanerj|ruck checking the rpm failures | 13:51 | |
| rlandy|ruck | 2021-12-14 11:11:58.183395 | fa163e20-c6f4-4011-a861-000000003745 | FATAL | Check Keystone admin endpoint status | undercloud | item=neutron | | 13:52 |
| dviroel|rover | anbanerj|ruck: yes | 13:52 |
| rlandy|ruck | FATAL | Check Keystone admin endpoint status | undercloud | item=swift | | 13:52 |
| sshnaidm | marios, for running on all ci repos we'll need to add c9 jobs to every repo separately, or to include it in template "tripleo-multinode-branchful" which is only one that runs on CI repos only | 13:53 |
| sshnaidm | marios, although it's neither multinode and not branchful :) | 13:53 |
| sshnaidm | marios, I think we can include in this template for a few days? Just to save on patches | 13:54 |
| sshnaidm | rlandy|ruck, ^ | 13:54 |
| marios | sshnaidm: yeah sure | 13:54 |
| marios | sshnaidm: i think its fine... mainly just to see them running at least :) | 13:54 |
| rlandy|ruck | sshnaidm: yep | 13:54 |
| rlandy|ruck | sounds fine | 13:54 |
| ysandeep | rlandy|ruck, any objections merging https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/298826 ? | 13:54 |
| rlandy|ruck | let's get the minimal run in ci repos merged | 13:54 |
| rlandy|ruck | ysandeep: works? | 13:55 |
| rlandy|ruck | if so, let's merge that | 13:55 |
| ysandeep | rlandy|ruck, yes https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/298826/2#message-f367e18389b84a7a2a910a87316a7260cfa25e75 | 13:55 |
| Tengu | chandankumar: nah - I'd rather get a simple, stupid container and try to talk to another one with a simple command in order to understand better the whole issue. | 13:55 |
| Tengu | chandankumar: shouldn't take much. | 13:55 |
| chandankumar | ok | 13:55 |
| rlandy|ruck | ovn-2021 x86_64 21.06.0-29.el8fdp rhosp-rhel-8.4-fdp | 13:56 |
| rlandy|ruck | good | 13:56 |
| ysandeep | rlandy|ruck, thanks for review ++ | 13:56 |
| *** soniya29|afk is now known as soniya29 | 13:58 | |
| chandankumar | marios: sshnaidm: for standalone scenario will I move to tripleo-multinode-branchful ? | 13:58 |
| marios | chandankumar: yeah that's what sshnaidm proposed for now so it is easier to wire it up for ci repos | 14:01 |
| sshnaidm | chandankumar, yeah, I'll move sa job there | 14:01 |
| chandankumar | sshnaidm: can we nuke for wallaby for now? Since nothing is ready there | 14:01 |
| chandankumar | it does not looks good by seeing failing job | 14:02 |
| sshnaidm | chandankumar, I'll put them only for ci repos for now | 14:02 |
| chandankumar | ok | 14:02 |
| sshnaidm | chandankumar, ok | 14:02 |
| rlandy|ruck | dviroel|rover: anbanerj|ruck: still not on ussuri upgrade | 14:03 |
| rlandy|ruck | jfrancoa: no worries the party is still on going | 14:04 |
| rlandy|ruck | https://review.rdoproject.org/zuul/stream/f8617286044c44f29b657b1f4d5eff53?logfile=console.log | 14:04 |
| rlandy|ruck | ^^ still no pass on ussuri upgrades | 14:04 |
| *** amoralej|lunch is now known as amoralej | 14:04 | |
| dviroel|rover | rlandy|ruck: anbanerj|ruck "msg": "Last metadata expiration check: 0:02:16 ago on Tue 14 Dec 2021 01:35:05 PM UTC.\nNo package pcs-0.10.11-1 available.\npcs-0.10.12-1.el8.x86_64" | 14:05 |
| dviroel|rover | pcs-0.10.11-1 is not available anymore | 14:06 |
| rlandy|ruck | dviroel|rover: anbanerj|ruck: this will fail gates as well | 14:06 |
| rlandy|ruck | marios: ^^ | 14:07 |
| rlandy|ruck | I think we need to remove this job from criteria and promote now | 14:07 |
| rlandy|ruck | the error is in gates as well | 14:07 |
| rlandy|ruck | and mark this job non-voting again | 14:07 |
| rlandy|ruck | this is going on too long | 14:08 |
| marios | rlandy|ruck: thanks, waiting for that to report so we can get the logs | 14:08 |
| rlandy|ruck | open it again on CIX | 14:08 |
| marios | rlandy|ruck: so we got a promotion but the workaround only lasted a day or two | 14:08 |
| rlandy|ruck | marios: no promotion | 14:08 |
| rlandy|ruck | that hash failed on OVB | 14:08 |
| marios | rlandy|ruck: but the ussuri promotion is 'only' 4 days old at this point right? | 14:08 |
| marios | rlandy|ruck: ah :/ | 14:08 |
| rlandy|ruck | 13 days agoussuricurrent-tripleocentos-83 days ago | 14:09 |
| marios | rlandy|ruck: we have a bug for that ovb issue ? | 14:09 |
| rlandy|ruck | marios: that was fix | 14:09 |
| rlandy|ruck | ussuri is running now | 14:09 |
| marios | rlandy|ruck: so... in the time frame the workaround *was* working, ovb failed. now ovb is fixed but the workaround expired :/ | 14:09 |
| marios | :/ | 14:09 |
| marios | lavly | 14:09 |
| rlandy|ruck | exactly | 14:09 |
| rlandy|ruck | looking at the line now | 14:10 |
| rlandy|ruck | we'll have to see what's the story | 14:10 |
| anbanerj|ruck | dviroel|rover, do we need maybe dnf downgrade pcs-0.10.12-1.el8 -y instead of cs-0.10.12-1 ? | 14:10 |
| marios | anbanerj|ruck: can you let the test report before you update it though so we can get th elogs | 14:10 |
| marios | ah it reported already anbanerj|ruck k | 14:11 |
| * ysandeep out for the day | 14:12 | |
| *** ysandeep is now known as ysandeep|out | 14:13 | |
| dviroel|rover | marios: actually, your testproject that had upgrade-ussuri green was with report_dlrn: false | 14:14 |
| marios | dviroel|rover: :/ | 14:16 |
| dviroel|rover | https://mirror.regionone.vexxhost-nodepool-tripleo.rdoproject.org/centos/8-stream/HighAvailability/x86_64/os/Packages/ | 14:19 |
| marios | rlandy|ruck: 2021-12-14 13:37:22.929736 | primary | "msg": "Last metadata expiration check: 0:02:16 ago on Tue 14 Dec 2021 01:35:05 PM UTC.\nNo package pcs-0.10.11-1 available.\npcs-0.10.12-1.el8.x86_64" | 14:19 |
| dviroel|rover | pcs-0.10.11-1 is still there | 14:19 |
| rlandy|ruck | marios: no longer carried | 14:19 |
| rlandy|ruck | you see it? | 14:19 |
| dviroel|rover | anbanerj|ruck: lets try with pcs-0.10.11-1.el8 instead, might be it | 14:28 |
| * anbanerj|ruck crosses finger and prays to the ci god | 14:28 | |
| marios | dviroel|rover: :) wrote something similar just now https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/821699/1#message-3bd316ec872e38470bb01f109b4c31b869fc36da | 14:28 |
| rlandy|ruck | amoralej: hello - touching base re: weirdo jobs in master | 14:31 |
| amoralej | rlandy|ruck, working on it, https://review.opendev.org/c/x/packstack/+/821594 will fix it | 14:31 |
| rlandy|ruck | amoralej: great - thank you | 14:31 |
| amoralej | i hope... | 14:31 |
| dviroel|rover | marios: just checking in my centos env here | 14:32 |
| marios | dviroel|rover: same but still not working anbanerj|ruck hold a sec not sure yet | 14:32 |
| dviroel|rover | 0.10.8-1.el8 - is the version here, so we need to add el8 | 14:33 |
| anbanerj|ruck | for my fedora sudo dnf install pcs-0.10.8-1.fc32.x86_64 worked | 14:33 |
| marios | dviroel|rover: is it working for you with ^^ ? .el8? if so lets try it | 14:33 |
| marios | anbanerj|ruck: so that is another thing we could try you know good idea anbanerj|ruck :D | 14:33 |
| marios | dviroel|rover: i mean anbanerj|ruck | 14:33 |
| marios | instead of downgrade | 14:34 |
| marios | we can remove and install specific version we want | 14:34 |
| dviroel|rover | marios: thats what dnf downgrade does, if you try there | 14:34 |
| marios | maybe rpm -e --no-deps for the removal so we don't get rid of stuffs | 14:34 |
| marios | dviroel|rover: ah i didn't realise it actually removes and installs the one you want | 14:34 |
| dviroel|rover | marios: downgrade of my telegraf here https://paste.opendev.org/show/811663/ | 14:36 |
| dviroel|rover | with dnf downgrade | 14:36 |
| dviroel|rover | lets add full package name | 14:36 |
| marios | k lets try it | 14:37 |
| dviroel|rover | will not add arch since the OS should solve that right | 14:37 |
| marios | ack | 14:37 |
| dviroel|rover | seems not right to force an arch | 14:37 |
| amoralej | marios, sorry for interfere but, why do we need to downgrade pcs? any known issue? | 14:38 |
| amoralej | or what's the issue? | 14:38 |
| marios | amoralej: https://bugs.launchpad.net/tripleo/+bug/1953326 | 14:38 |
| marios | amoralej: its a workaround for now | 14:38 |
| amoralej | ups | 14:39 |
| amoralej | it may be better to blacklist in the repo config? | 14:39 |
| marios | amoralej: ah pin to the version we want ? yes rdo-info i think? | 14:40 |
| amoralej | no, i mean in the .repo file created by oooq ? | 14:40 |
| marios | amoralej: ah i see | 14:40 |
| amoralej | we don't manage OS packages in rdoinfo | 14:40 |
| marios | amoralej: yes this is also a good idea | 14:40 |
| marios | amoralej: k ... dviroel|rover ^^ we can try that as a next step update the ha .repo file in the release config | 14:41 |
| marios | thanks amoralej | 14:41 |
| dviroel|rover | yes, the right way to do it, this workaround was suppose to be very temp | 14:42 |
| amoralej | ah i see you put the feedback in the bz, nice | 14:42 |
| anbanerj|ruck | dviroel|rover, thanks for rechecking https://review.rdoproject.org/r/c/testproject/+/37195. I am eod in ~10 mins. Updated the master tp in gchat | 14:43 |
| dviroel|rover | anbanerj|ruck: np, thanks anbanerj|ruck | 14:43 |
| dasm | o/ | 14:54 |
| *** dviroel|rover is now known as dviroel|rover|lunch | 14:59 | |
| chandankumar | anbanerj|ruck: can I update your sc7 cs9 testproject? | 15:01 |
| anbanerj|ruck | chandankumar, sure | 15:02 |
| chandankumar | we have got fix for sc7 cs9 https://review.opendev.org/c/openstack/tripleo-heat-templates/+/821709 | 15:02 |
| marios | tripleo-ci o/ please add to your reviews list https://review.opendev.org/c/openstack/tripleo-ci/+/816991/16#message-199f53afaae2f5a30fe54be0bdf3c57b348e8b54 | 15:13 |
| rlandy|ruck | https://review.rdoproject.org/zuul/builds?pipeline=openstack-component-network - network line is pretty bad | 15:20 |
| marios | chandankumar: well, fs1 also gets passed provision with https://review.opendev.org/c/openstack/diskimage-builder/+/821526 | 15:25 |
| marios | chandankumar: waiting for it to report and will add pointer in the bug | 15:26 |
| chandankumar | marios: awesome ++ | 15:26 |
| marios | chandankumar: also fails in tempest like fs2 | 15:26 |
| rlandy|ruck | dviroel|rover|lunch: hey - going to rekick the network line when it is done | 15:38 |
| rlandy|ruck | very red on so many branches | 15:38 |
| dviroel|rover|lunch | rlandy|ruck: ack, thanks | 15:59 |
| *** dviroel|rover|lunch is now known as dviroel|rover | 15:59 | |
| dviroel|rover | Downgraded:\n pcs-0.10.11-1.el8.x86_64 \n\nComplete!\npcs-0.10.11-1.el8.x86_64 | 16:08 |
| dviroel|rover | :) | 16:08 |
| dviroel|rover | marios ^ | 16:08 |
| dviroel|rover | lets see if the job will pass now | 16:08 |
| marios | dviroel|rover: ack good | 16:10 |
| marios | 'good' ;) dviroel|rover | 16:10 |
| dviroel|rover | :) | 16:10 |
| marios | tripleo-ci o/ please add to your reviews list https://review.opendev.org/c/openstack/tripleo-ci/+/816991/16#message-199f53afaae2f5a30fe54be0bdf3c57b348e8b54 | 16:27 |
| marios | rlandy|ruck: when you next have time thanks ^ | 16:27 |
| *** marios is now known as marios|out | 16:46 | |
| rlandy|ruck | dviroel|rover: upgrade done?? | 16:49 |
| rlandy|ruck | finally :) | 16:49 |
| dviroel|rover | oh yes, running tempest tests now | 16:50 |
| dviroel|rover | :) | 16:50 |
| rlandy|ruck | dviroel|rover: great - let's get https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/821699 merged | 17:07 |
| dviroel|rover | rlandy|ruck: +1 | 17:10 |
| dviroel|rover | 0c303674a982c099b14bcb4198325ec7 is good for the promoter now :) | 17:13 |
| rlandy|ruck | dviroel|rover++ yep | 17:14 |
| rlandy|ruck | promoting now | 17:14 |
| rlandy|ruck | dviroel|rover: k ... will merge https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/821699 when it passes check | 17:15 |
| rlandy|ruck | gates must be breaking now | 17:15 |
| dviroel|rover | component-network looks bad | 17:16 |
| *** amoralej is now known as amoralej|off | 17:26 | |
| *** jpena is now known as jpena|off | 17:37 | |
| akahat | moni_l | 18:00 |
| rlandy|ruck | dviroel|rover: and now zuul is gone | 18:47 |
| rlandy|ruck | dviroel|rover: on the upside ussuri promoted | 18:47 |
| dviroel|rover | yes :) and yes :( | 18:50 |
| rlandy|ruck | dviroel|rover: meeting | 19:04 |
| dviroel|rover | ah, true, joining | 19:05 |
| rlandy|ruck | dviroel|rover++ rcastillo++ looking very nice - thanks for that work | 19:40 |
| dviroel|rover | \o/ | 19:41 |
| rlandy|ruck | dviroel|rover: updating the program doc for tomorrow's meeting you can decide with ananya who wants to present | 19:46 |
| dviroel|rover | rlandy|ruck: ok | 19:46 |
| dviroel|rover | rechecking pcs fix =/ | 19:46 |
| rlandy|ruck | ugh - zuul still awol | 20:05 |
| rlandy|ruck | dviroel|rover: pls ping when you are EoD - I have one meeting left - then I can pick up what's left to rerun | 20:05 |
| dviroel|rover | ok | 20:05 |
| dviroel|rover | rlandy|ruck: going eod, there isn't too much to see while we didn't kick missed pipeline | 20:26 |
| dviroel|rover | rlandy|ruck: we need to run master and train c8 | 20:27 |
| dviroel|rover | rlandy|ruck: network-component has different failures, but I've seen pacemaker error more than once | 20:27 |
| dviroel|rover | rlandy|ruck: Error: Execution of '/bin/dnf -d 0 -e 1 -y install pacemaker' returned 1: Error: Error downloading packages:\n<13>Dec 14 13:05:25 puppet-user: libwbclient-4.15.2-2.el8.x86_64: Cannot download, all mirrors were already tried without success\n | 20:28 |
| rlandy|ruck | dviroel|rover: ok - I'll log the network one | 20:28 |
| rlandy|ruck | and rerun the lines | 20:29 |
| rlandy|ruck | thanks dviroel|rover | 20:29 |
| dviroel|rover | network component error: https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-network-master/bc0b0b5/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz | 20:29 |
| dviroel|rover | o/ | 20:31 |
| *** dviroel|rover is now known as dviroel|out | 20:31 | |
| rlandy|ruck | https://bugs.launchpad.net/tripleo/+bug/1954778 - I see master bug | 20:39 |
| rlandy|ruck | train 8 rekicked | 20:44 |
| rlandy|ruck | master rekicked | 20:46 |
| rlandy|ruck | ugh - we hosed container builds on wallaby | 21:57 |
| rlandy|ruck | chandankumar: ^^ | 21:57 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!