| *** dviroel|out is now known as dviroel | 00:33 | |
| dviroel | o/ | 00:33 |
|---|---|---|
| dviroel | rlandy|ruck: still around? | 00:33 |
| dviroel | rlandy|ruck: do you want vote/merge the workaround? | 00:33 |
| rlandy|ruck | dviroel: hi | 00:35 |
| rlandy|ruck | dviroel: I have an edit | 00:35 |
| rlandy|ruck | to address your comment | 00:35 |
| rlandy|ruck | and add the tripleo-ci-testing version | 00:35 |
| rlandy|ruck | I am just waiting for the current one to finish reporting then will update | 00:36 |
| dviroel | ack, you going to try only sqlalchemy or merge both? | 00:36 |
| dviroel | and remove later? | 00:36 |
| rlandy|ruck | should be 5 mins | 00:36 |
| rlandy|ruck | I am going with both | 00:36 |
| dviroel | ok | 00:36 |
| rlandy|ruck | tomorrow we can remove one at a time | 00:36 |
| rlandy|ruck | and see which one or both are needed | 00:36 |
| dviroel | if you want, we can merge this PS too | 00:36 |
| dviroel | just send it to gate | 00:36 |
| rlandy|ruck | it will report in 5 | 00:36 |
| dviroel | address my comment in another patch | 00:37 |
| rlandy|ruck | I need to add the integration line one | 00:37 |
| dviroel | oh yeagh | 00:37 |
| rlandy|ruck | I have the change ready | 00:37 |
| dviroel | tru | 00:37 |
| dviroel | true | 00:37 |
| rlandy|ruck | just waiting to click | 00:37 |
| rlandy|ruck | 2 tests left are just uploading logs | 00:38 |
| rlandy|ruck | at least it worked | 00:38 |
| dviroel | yeah | 00:38 |
| dviroel | ++ | 00:38 |
| rlandy|ruck | dviroel: thanks for coming back to check in | 00:38 |
| rlandy|ruck | I think we can just w+ the second patch | 00:39 |
| dviroel | sure | 00:39 |
| rlandy|ruck | it can't be worse than the situation now | 00:39 |
| dviroel | this patch also fixes https://bugs.launchpad.net/tripleo/+bug/1982195 | 00:41 |
| dviroel | you can add to commit message too | 00:41 |
| dviroel | https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-9-undercloud-upgrade&skip=0 | 00:42 |
| rlandy|ruck | yep added both to new commit message | 00:43 |
| rlandy|ruck | two bugs, one workaround | 00:44 |
| rlandy|ruck | ugh - slowest log upload in history | 00:44 |
| rlandy|ruck | https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-9-containers-multinode-wallaby&skip=0 | 00:49 |
| rlandy|ruck | it works :) | 00:49 |
| dviroel | great :) | 00:50 |
| rlandy|ruck | dviroel: new patch uploaded | 00:57 |
| rlandy|ruck | dviroel: pls vote and I'll w+ and we can both go to sleep | 00:58 |
| dviroel | ok | 00:58 |
| rlandy|ruck | pls check I got your comment right | 00:59 |
| dviroel | rlandy|ruck: yes, +2 | 00:59 |
| rlandy|ruck | great - thanks | 00:59 |
| rlandy|ruck | have a good night | 01:00 |
| dviroel | rlandy|ruck: you too | 01:00 |
| dviroel | ttyt o/ | 01:00 |
| rlandy|ruck | thanks | 01:00 |
| *** dviroel is now known as dviroel|out | 01:00 | |
| *** rlandy|ruck is now known as rlandy|out | 01:00 | |
| *** ysandeep|out is now known as ysandeep | 01:52 | |
| *** ysandeep is now known as ysandeep|afk | 03:23 | |
| *** amoralej|off is now known as amoralej | 06:09 | |
| jm1 | good morning folks :) | 06:13 |
| jm1 | chkumar|rover: any fallout from my breaking change in aoc yesterday? | 06:18 |
| chkumar|rover | jm1: I have not looked ove the integration line logs | 06:43 |
| chkumar|rover | currently we have cs9 update broke the upstream gate working on fixing that | 06:43 |
| chkumar|rover | *over | 06:43 |
| chkumar|rover | if I see anything suspecious will let you know , thanks for the ping :-) | 06:43 |
| akahat | 06:48 | |
| akahat | Good Morning o/ | 06:48 |
| tweining | hi. the gate pipeline fails in the "Run DLRN" for one of my changes and I wonder if it something that on the team's radar already. Could you please have a quick look and tell me if I need to report a bug or not? | 07:41 |
| tweining | https://zuul.opendev.org/t/openstack/build/d760a50c038548989a6d4e05ea2c0fda | 07:41 |
| tweining | the relevant logs seem to be in https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d76/850150/1/gate/tripleo-ci-centos-9-content-provider/d760a50/logs/undercloud/home/zuul/DLRN/data/repos/component/validation/04/ed/04ed7b2d47190f9bcc8e679085429ce8713f0ee2_dev/ | 07:42 |
| chkumar|rover | ykarel: on kernel 130, recent run of fs01 https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-master/eedf507/logs/baremetal_0-console.log failed at overcloud deploy during nova db sync | 08:14 |
| ykarel | chkumar|rover, okk so it's not failing as before? | 08:21 |
| ykarel | i seee Kernel 5.14.0-130.el9.x86_64 on an x86_64 in above logs | 08:21 |
| ykarel | if ^ happening only in periodic pipeline then issue is related when kernel is upgraded within job | 08:23 |
| ykarel | vs latest kernel already present in overcloud image | 08:23 |
| ykarel | looking db sync error | 08:23 |
| chkumar|rover | ykarel: db sync error is related to deadlock exception | 08:23 |
| ykarel | and if true we would need promotion to clear check jobs | 08:24 |
| ykarel | chkumar|rover, but that deadlock issue was not only for wallaby? | 08:24 |
| chkumar|rover | let me check fs01 wallaby job also | 08:25 |
| ykarel | pymysql.err.OperationalError: (1205, 'Lock wait timeout exceeded; try restarting transaction') | 08:26 |
| ykarel | i see errors in pcs, so likely that caused that, but doesn't look it will be consistent | 08:26 |
| ykarel | likely it's https://bugs.launchpad.net/tripleo/+bug/1981478 | 08:28 |
| chkumar|rover | ykarel: Does fs035 rhos0-17 rhel-9 fix got merged? | 09:15 |
| ykarel | chkumar|rover, yes that's merged | 09:18 |
| ykarel | and is in component testing | 09:19 |
| ykarel | when i checked last | 09:19 |
| *** ysandeep|afk is now known as ysandeep | 10:14 | |
| chkumar|rover | amoralej: hello | 10:18 |
| chkumar|rover | amoralej: is there a way to avoid package updates which are coming from dlrn-deps repo? | 10:19 |
| chkumar|rover | on cs9 wallaby, sqlachemy got updated it broke the job | 10:19 |
| chkumar|rover | https://review.opendev.org/c/openstack/tripleo-quickstart/+/850428 | 10:20 |
| chkumar|rover | is there a way to avoid that for any package coming from dlrn-deps repo? | 10:21 |
| amoralej | chkumar|rover, mmmm if there is an actual problem we can revert | 10:26 |
| amoralej | what's the problem? | 10:26 |
| chkumar|rover | amoralej: https://bugs.launchpad.net/tripleo/+bug/1982195 | 10:26 |
| chkumar|rover | please use ``inspect(some_engine).has_table(<tablename>>)`` for public API use. | 10:26 |
| amoralej | but we haven't updated sqlalchemy in wallaby in ages | 10:28 |
| chkumar|rover | https://review.opendev.org/c/openstack/barbican/+/796284 - merged in jun 16, 2021 | 10:28 |
| chkumar|rover | python-sqlalchemy-1.3.24-1.el9s is cs9 wallaby testing | 10:29 |
| amoralej | but that's master | 10:29 |
| amoralej | not wallaby | 10:29 |
| chkumar|rover | the failure was coming in upgrade job | 10:30 |
| amoralej | upgrade job from wallaby to master? | 10:30 |
| amoralej | i understand in master it uses sqlalchemy from zed, right? | 10:31 |
| amoralej | in zed we have python-sqlalchemy-1.4.36-1.el9s | 10:31 |
| amoralej | i don't understand it | 10:31 |
| chkumar|rover | amoralej: yes, it is a upgrade job from wallaby to master | 10:33 |
| *** rlandy|out is now known as rlandy | 10:33 | |
| amoralej | chkumar|rover, that change is 2018 2021! | 10:33 |
| amoralej | so that change must be in wallaby | 10:34 |
| amoralej | > one year old | 10:34 |
| amoralej | mmm as per yatin comments probably xena | 10:34 |
| chkumar|rover | amoralej: i found that change from gerrit search | 10:34 |
| amoralej | so, let me see the log | 10:34 |
| amoralej | i don't understand what is the problem yet | 10:34 |
| chkumar|rover | https://536ba7636eca1576ae9f-4e53ffd70f9f0b38678a1c719bbe7de3.ssl.cf2.rackcdn.com/850349/1/check/tripleo-ci-centos-9-undercloud-upgrade/f43c1b0/logs/undercloud/var/log/extra/podman/containers/barbican_api_db_sync/stdout.log | 10:35 |
| chkumar|rover | logs here | 10:35 |
| rlandy | chkumar|rover: hey | 10:35 |
| chkumar|rover | rlandy: hello | 10:35 |
| rlandy | chkumar|rover: want to sync about c9? | 10:35 |
| chkumar|rover | rlandy: yes | 10:35 |
| amoralej | chkumar|rover, | 10:36 |
| amoralej | python3-sqlalchemy.x86_64 1.4.37-3.el9 @quickstart-centos-appstreams | 10:36 |
| amoralej | thats the sqlalchemy in the container | 10:36 |
| rlandy | chkumar|rover: https://meet.google.com/dtk-wtiy-ppe?pli=1&authuser=0 | 10:36 |
| amoralej | it comes from OS, not from RDO dep | 10:36 |
| amoralej | it was updated one week ago | 10:38 |
| amoralej | in centos | 10:38 |
| chkumar|rover | rlandy: https://zuul.opendev.org/t/openstack/status#850442 | 10:38 |
| chkumar|rover | https://review.opendev.org/c/openstack/tripleo-quickstart/+/850442/ | 10:39 |
| amoralej | actually, it seems it has been added to centos, not updated | 10:39 |
| amoralej | it wasn't in the past | 10:39 |
| amoralej | https://gitlab.com/redhat/centos-stream/rpms/python-sqlalchemy/-/commit/2a8b96be86e7209ce54a2f047498427262b677d3 | 10:40 |
| amoralej | so that's probably what is breaking it | 10:40 |
| amoralej | mmm | 10:40 |
| amoralej | chkumar|rover, https://bugzilla.redhat.com/show_bug.cgi?id=2084556 | 10:43 |
| amoralej | that's the actual problem | 10:43 |
| amoralej | it's ahead of u-c | 10:43 |
| chkumar|rover | rlandy: https://code.engineering.redhat.com/gerrit/c/tripleo-ansible/+/420543 | 10:49 |
| chkumar|rover | rlandy: https://review.opendev.org/c/openstack/tripleo-quickstart/+/850442/ | 10:51 |
| amoralej | what i don't understand is why it's not failing in p-o-i jobs | 10:54 |
| chkumar|rover | https://review.rdoproject.org/r/c/testproject/+/44080 | 10:59 |
| chkumar|rover | rlandy: https://review.rdoproject.org/r/c/testproject/+/44037 | 11:00 |
| chkumar|rover | https://review.rdoproject.org/r/c/testproject/+/42692 | 11:04 |
| marios | jm1: o/ fyi https://review.opendev.org/q/topic:oooci_remove_enable_paunch ( https://review.opendev.org/c/openstack/tripleo-quickstart/+/841582/29/config/general_config/featureset066.yml#69 ) when you next have time for reviews please | 11:05 |
| reviewbot | Do you want me to add your patch to the Review list? Please type something like add to review list <your_patch> so that I can understand. Thanks. | 11:05 |
| chkumar|rover | amoralej: sorry was in a meeting | 11:05 |
| amoralej | np | 11:06 |
| chkumar|rover | amoralej: it is coming in upgrade job only | 11:06 |
| amoralej | dunno why | 11:06 |
| chkumar|rover | Do we have a upgrade job in poi line? | 11:06 |
| amoralej | i'd expect it to fail on all jobs, tbh | 11:06 |
| amoralej | nop | 11:06 |
| chkumar|rover | amoralej: https://de0765cc74f1c1de732f-60e7154167fad33e01f6d4d5448fd7d1.ssl.cf5.rackcdn.com/847237/1/check/tripleo-ci-centos-9-undercloud-containers/e1bfc86/logs/undercloud/home/zuul/undercloud_install.log | 11:08 |
| chkumar|rover | it is alos seen now in undercloud job | 11:08 |
| chkumar|rover | checking failure at this one https://review.opendev.org/c/openstack/puppet-tripleo/+/847237 | 11:08 |
| chkumar|rover | Not sure, it might pop up in poi also | 11:08 |
| chkumar|rover | amoralej: thanks for the rhbz sqlalchemy bug link | 11:10 |
| jm1 | marios: want to remove undercloud_enable_paunch from here as well? https://opendev.org/openstack/tripleo-upgrade/src/branch/master/tasks/common/configure_uc_containers.yml | 11:16 |
| marios | review time folks o/ | 11:16 |
| marios | jm1: yeah saw that one we should yes (if you want to post go ahead and use the same topic otherwise will do later) | 11:16 |
| amoralej | chkumar|rover, the options i see is trying to use < 1.4.0 in spec | 11:17 |
| amoralej | or backporting the fix | 11:17 |
| amoralej | for barbican | 11:17 |
| amoralej | using < can lead to issues | 11:17 |
| amoralej | but ... | 11:17 |
| amoralej | i'm not sure if https://review.opendev.org/c/openstack/barbican/+/796059 would be enough | 11:20 |
| chkumar|rover | amoralej: currently going with this https://review.opendev.org/c/openstack/tripleo-quickstart/+/850442 | 11:20 |
| chkumar|rover | there is a cix open for that may be barbican team can provide their input | 11:20 |
| amoralej | ok | 11:21 |
| chkumar|rover | I will check with adelee on this and let you know :-) | 11:22 |
| jm1 | marios: https://review.opendev.org/c/openstack/tripleo-upgrade/+/850509 | 11:22 |
| jm1 | reviewbot: add https://review.opendev.org/c/openstack/tripleo-upgrade/+/850509 | 11:22 |
| jm1 | reviewbot: add review https://review.opendev.org/c/openstack/tripleo-upgrade/+/850509 | 11:22 |
| jm1 | frenzy_friday: reviewbot down again? | 11:23 |
| amoralej | in the meanwhile i'll try to find out why we are not getting it in p-o-i | 11:23 |
| chkumar|rover | amoralej: sure | 11:23 |
| anbanerj | jm1, restarted | 11:27 |
| anbanerj | probably I should add a cronjob to restart reviewbot everyday | 11:28 |
| jm1 | anbanerj: thanks :) now we have anbanerj, frenzy_friday, frenzyfriday and frenzyfriday_ 🤨 | 11:30 |
| anbanerj | jm1, yeah my internet disconnected and something happened XD | 11:30 |
| *** anbanerj is now known as frenzy_friday | 11:33 | |
| *** dviroel|out is now known as dviroel | 11:34 | |
| ysandeep | marios, fyi.. based on this patch.. looks like paunch was deprecated in U cycle: https://opendev.org/openstack/tripleo-heat-templates/commit/3a00c029f28a215025a2c7e75ce7d01947223591 | 11:51 |
| marios | thanks ysandeep apparently still used for train Tengu confirmed in tripleo just now | 11:51 |
| marios | i'll revisit those patches and add branch if necessary (but e.g. we are setting it explicitly false right now eg see https://review.opendev.org/c/openstack/tripleo-quickstart/+/850507/1/config/general_config/featureset010.yml we don't have branch conditional for train ... ? | 11:52 |
| amoralej | chkumar|rover, https://github.com/openstack/ec2-api/blob/stable/wallaby/requirements.txt#L32 | 11:55 |
| amoralej | that's why it's working in p-o-i | 11:55 |
| amoralej | maybe we could set max version in barbican in wallaby so far | 11:55 |
| amoralej | that should work | 11:55 |
| ysandeep | because default is true for enable_paunch, we may need some guard there, ack for revisit.. thanks! | 11:58 |
| *** anbanerj is now known as frenzy_friday | 11:58 | |
| rlandy | chkumar|rover: did the master promo work? | 12:04 |
| chkumar|rover | rlandy: sorry got side tracked , just proposed the config patch https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/44083 | 12:08 |
| rlandy | chkumar|rover: thanks | 12:09 |
| *** amoralej is now known as amoralej|lunch | 12:10 | |
| rlandy | dviroel: you got a shout out at the program call | 12:23 |
| dviroel | oh, really? lost it :( | 12:24 |
| rlandy | fips | 12:25 |
| rlandy | chkumar|rover: ah - we need to start reporting on 17.1 when ready | 12:26 |
| rlandy | you can say 17.1 standup is in progress | 12:26 |
| rlandy | chkumar|rover++ nice | 12:29 |
| chkumar|rover | rlandy: we got another fix for sqlachemy https://review.rdoproject.org/r/c/openstack/barbican-distgit/+/44082 | 12:30 |
| chkumar|rover | will revert tq patch once we have tripleo wallaby promotion | 12:32 |
| rlandy | chkumar|rover: ah ok - much better | 12:33 |
| rlandy | chkumar|rover: ack - let's keep the w/ around until we get that | 12:33 |
| chkumar|rover | sure | 12:35 |
| chkumar|rover | rlandy: no hash available to promote master | 12:46 |
| chkumar|rover | http://promoter.rdoproject.org/promoter_logs/centos9_master.log | 12:46 |
| chkumar|rover | if we remove fs01 and fs035 then we might promote one hash | 12:47 |
| chkumar|rover | if you are ok? | 12:47 |
| * chkumar|rover prepare a patch | 12:48 | |
| chkumar|rover | sorry https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/44083 is not yet merged | 12:49 |
| rlandy | chkumar|rover: at this point - yeah | 12:50 |
| rlandy | oh ok | 12:50 |
| rlandy | chkumar|rover: let me know by your eod | 12:50 |
| chkumar|rover | sure sure | 12:50 |
| rlandy | frenzy_friday: bhagyashris: hey - ps let me know about your 17.1 patch by your eod - of there is something to re- review | 12:51 |
| bhagyashris | rlandy, image build and conatiner build patches are ready | 12:53 |
| bhagyashris | i mean addressed review comments about override_checkout | 12:54 |
| rlandy | bhagyashris++ thank you | 12:54 |
| rlandy | can we re-testproject them | 12:54 |
| rlandy | so we can check the versions now? | 12:54 |
| bhagyashris | will need to check with release file patch | 12:54 |
| bhagyashris | rlandy, sure let me recheck the testproject | 12:54 |
| rlandy | bhagyashris: perfect - thanks | 12:55 |
| frenzy_friday | bhagyashris, I updated release file patch. We can rechevck testproj now | 13:00 |
| rlandy | company meeting all | 13:03 |
| *** amoralej|lunch is now known as amoralej | 13:20 | |
| akahat | marios, https://review.rdoproject.org/r/c/testproject/+/44048 | 13:21 |
| marios | akahat: thanks. once the upstream patches merge we can merge https://review.rdoproject.org/r/c/config/+/44066 and start testing | 13:23 |
| marios | akahat: but not master should be wallaby only | 13:23 |
| akahat | marios, may be we don't need to add required projects. it should use already created multinode job. | 13:24 |
| akahat | and job is running currently.. https://review.rdoproject.org/zuul/stream/419acfd397bf47459e7ab68e53a4bd34?logfile=console.log | 13:24 |
| akahat | yes.. wallaby: https://review.rdoproject.org/r/c/testproject/+/44048/2/.zuul.yaml#4 | 13:25 |
| marios | akahat: k add comments on the review re required projects? problem is we need a different parent here | 13:25 |
| akahat | marios, ok. I'll add comments there. | 13:26 |
| marios | akahat: there is a typo you're not running the mixed os job there see periodic-tirpleo-ci-mixed-os-8-9-master | 13:27 |
| akahat | marios, yeah.. thanks.. fixing. | 13:28 |
| *** ysandeep is now known as ysandeep|afk | 13:37 | |
| *** dasm|off is now known as dasm|ruck | 13:48 | |
| dasm|ruck | o/ | 13:48 |
| jm1 | dasm|ruck: o/ (folks watching company meeting) | 13:49 |
| rlandy | dasm|ruck: ^^ yyou can catch the last of meeting | 13:51 |
| dasm|ruck | oh | 13:52 |
| dasm|ruck | probably i'm gonna watch replay then | 13:53 |
| dasm|ruck | chkumar|rover: i saw your comments on the launchpad. Thanks for triaging that | 13:56 |
| chkumar|rover | dasm|ruck: tldr of today | 13:56 |
| chkumar|rover | python3-sqlalchemy issue is handled | 13:56 |
| dasm|ruck | i saw it being merged | 13:56 |
| chkumar|rover | we have skipped ovn-provider job from master criteria and checking if any hash fits the criteria | 13:57 |
| dasm|ruck | ack | 13:57 |
| chkumar|rover | if it does not work, then remove fs01 and fs035 from master and make master promotion happen | 13:57 |
| chkumar|rover | coming to downstream side | 13:57 |
| chkumar|rover | on RHOS-17 RHEL-9, we are currently node failure on psi | 13:57 |
| chkumar|rover | when jobs are back. | 13:57 |
| chkumar|rover | please promote tripleo component, all jobs are passing there | 13:58 |
| chkumar|rover | and kick rhos-17 rhel-9 line | 13:58 |
| chkumar|rover | that's the end of story | 13:58 |
| dasm|ruck | chkumar|rover: what do you mean by "promote"? if jobs are passing, shouldn't it automatically promote? | 13:58 |
| chkumar|rover | dasm|ruck: yes | 13:58 |
| chkumar|rover | it will promote | 13:59 |
| dasm|ruck | ack | 13:59 |
| chkumar|rover | config ovn job removal https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/44083 | 14:00 |
| dasm|ruck | chkumar|rover: so we're behind just with cs9 master and rhel9 osp17 | 14:00 |
| chkumar|rover | sqlalchemy patch: https://review.opendev.org/c/openstack/tripleo-quickstart/+/850442 | 14:00 |
| chkumar|rover | please ^^ get it merged | 14:00 |
| dasm|ruck | chkumar|rover: do we still need that? we already have barbican spec merged. | 14:01 |
| dasm|ruck | https://review.rdoproject.org/r/c/openstack/barbican-distgit/+/44082 | 14:01 |
| chkumar|rover | RHOS-17 RHEL-9 tripleo component pipeline - https://code.engineering.redhat.com/gerrit/c/testproject/+/420692 | 14:01 |
| chkumar|rover | dasm|ruck: yes, till we have a wallaby cs9 promotion | 14:01 |
| dasm|ruck | dviroel: can we get your attention here? https://review.opendev.org/c/openstack/tripleo-quickstart/+/850442 | 14:02 |
| chkumar|rover | https://review.rdoproject.org/r/c/openstack/barbican-distgit/+/44082 will take time to reach upstream and check gate | 14:02 |
| chkumar|rover | dasm|ruck: that's the end of story now | 14:03 |
| dasm|ruck | ack | 14:03 |
| dasm|ruck | chkumar|rover: any other issues with new centos stream? i noticed some chatter | 14:03 |
| chkumar|rover | dasm|ruck: good question | 14:04 |
| chkumar|rover | dasm|ruck: https://bugs.launchpad.net/tripleo/+bug/1982137 | 14:04 |
| chkumar|rover | kernel update brought that | 14:04 |
| chkumar|rover | but it seems to resolved itself | 14:04 |
| chkumar|rover | we just need a wallaby promotion to clear the fs01 check jobs | 14:04 |
| dasm|ruck | ack | 14:04 |
| dviroel | dasm|ruck: will get there in a minute | 14:05 |
| chkumar|rover | dasm|ruck: kernel exclude patch https://review.opendev.org/c/openstack/tripleo-quickstart/+/850444 | 14:05 |
| chkumar|rover | dasm|ruck: regarding ovn provide side, we have also got reverts https://review.rdoproject.org/r/q/project:openstack%252Foctavia-distgit | 14:06 |
| chkumar|rover | I did not get the time to poke at those | 14:07 |
| chkumar|rover | that's it | 14:07 |
| dasm|ruck | chkumar|rover: soo... your patch cannot be merged https://review.opendev.org/c/openstack/tripleo-quickstart/+/850444 due to barbican issue, which is partially resolved already | 14:07 |
| dasm|ruck | chkumar|rover: Oo it just got merged... | 14:08 |
| dasm|ruck | Heh | 14:08 |
| chkumar|rover | dasm|ruck: it is for node provisioning issue | 14:08 |
| chkumar|rover | if no questions, then I can log out | 14:08 |
| dasm|ruck | chkumar|rover: as long as you kept hackmd, there are no more questions ^^ | 14:09 |
| chkumar|rover | dasm|ruck: I might have not updated the hackmd but these chats might help | 14:09 |
| dasm|ruck | :( | 14:09 |
| * dasm|ruck is disappointed | 14:09 | |
| dasm|ruck | but what can i do? | 14:09 |
| dasm|ruck | probably i'll update them by myself :) | 14:09 |
| chkumar|rover | except ovn revert info and kernel patch, rest is there sir | 14:10 |
| dasm|ruck | ack, ack | 14:10 |
| dasm|ruck | no worries | 14:10 |
| chkumar|rover | dasm|ruck: rlandy see ya then! | 14:10 |
| dasm|ruck | chkumar|rover: o/ | 14:10 |
| *** ysandeep|afk is now known as ysandeep | 14:13 | |
| dviroel | chkumar|rover: you can W+1 https://review.opendev.org/c/openstack/tripleo-quickstart/+/850442 if you want - thanks++ | 14:13 |
| dviroel | dasm|ruck: ^ | 14:13 |
| dasm|ruck | dviroel: i think chkumar|rover left already | 14:16 |
| dviroel | dasm|ruck: oh yeah | 14:18 |
| dviroel | dasm|ruck: rlandy: https://review.opendev.org/c/openstack/tripleo-quickstart/+/850442 - W+1 to clear ci | 14:19 |
| dasm|ruck | dviroel++ | 14:20 |
| * frenzy_friday lunch | 14:26 | |
| bhagyashris | rlandy, hey all the rhso17.1 patches are updated waiting for the testproject result but on testproject patches there is node_failure | 14:40 |
| bhagyashris | jfyi^ | 14:40 |
| rlandy | bhagyashris: ack - psi is having another bad day | 14:40 |
| rlandy | will try recheck later if things improve | 14:40 |
| rlandy | ty | 14:40 |
| marios | this known chkumar|rover ERROR: The Resource Type (OS::TripleO::Services::Rear) could not be found | 14:43 |
| marios | ? | 14:43 |
| * marios checks rr pad | 14:43 | |
| Tengu | rlandy: https://logserver.rdoproject.org/54/31954/43/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-master/6161cf7/logs/undercloud/var/log/extra/dropped-packets.txt.gz the new logfile exists! | 14:43 |
| rlandy | Tengu: cool | 14:43 |
| rlandy | will revote | 14:43 |
| Tengu | uho..... weird. | 14:44 |
| Tengu | the dropped packets... it looks like the localhost access is prevented ?! | 14:44 |
| Tengu | what the actual f.. | 14:45 |
| Tengu | humpf. and the audit level is terrible. Just source/dest/proto. No port info. "great". | 14:48 |
| Tengu | rlandy: getting the log creation would be nice indeed - but we'll need the 2 others https://review.opendev.org/q/topic:nftables%252Flogging in order to actually GET content :). And I have to check if changing the log level (currently "audit") will change anything. | 14:49 |
| Tengu | rlandy: I -W for now the dedicated file - it may need some changes depending on my coming investigations... | 14:50 |
| Tengu | but the theory is good. Just... incomplete. | 14:51 |
| rlandy | sorry on - cix | 14:51 |
| Tengu | np | 14:51 |
| frenzy_friday | bhagyashris, rlandy the testproject for containerbuild finally passed | 14:54 |
| rlandy | frenzy_friday: cool - can you confirm you hav 17.1 versions of repos? | 14:55 |
| rlandy | link to testproject? | 14:55 |
| frenzy_friday | checking | 14:55 |
| rlandy | you should see it on zuul file | 14:55 |
| frenzy_friday | https://code.engineering.redhat.com/gerrit/c/testproject/+/417334 | 14:55 |
| rlandy | inventory | 14:55 |
| frenzy_friday | rechecking image build testproj | 14:55 |
| rlandy | canonical_name: code.engineering.redhat.com/python-tripleoclient | 14:56 |
| rlandy | checkout: rhos-17.1-trunk-patches | 14:56 |
| rlandy | override_checkout: rhos-17.1-trunk-patches | 14:56 |
| rlandy | code.engineering.redhat.com/openstack-tripleo-common: | 14:57 |
| rlandy | canonical_hostname: code.engineering.redhat.com | 14:57 |
| rlandy | canonical_name: code.engineering.redhat.com/openstack-tripleo-common | 14:57 |
| rlandy | checkout: rhos-17.0-trunk-patches | 14:57 |
| rlandy | ^^ | 14:57 |
| frenzy_friday | oh, updating | 14:58 |
| rlandy | that one is off | 14:58 |
| rlandy | rest looks ok | 14:58 |
| frenzy_friday | rechecked testproj, updated job def | 15:01 |
| rlandy | dasm|ruck: what the status of master promo? | 15:19 |
| rlandy | merged patch to skip ovn, right? | 15:20 |
| rlandy | still no hash match? | 15:20 |
| dasm|ruck | yes, ovn is in skiplist: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/44083 | 15:21 |
| dasm|ruck | last rerun by chkumar|rover failed: https://review.rdoproject.org/r/c/testproject/+/44080 | 15:22 |
| dasm|ruck | currently there are 4 more jobs for the same hash | 15:22 |
| rlandy | it should have promoted a previous hash | 15:25 |
| dasm|ruck | checking | 15:26 |
| dasm|ruck | currently there is no promo | 15:26 |
| rlandy | yeah | 15:27 |
| rlandy | so monday's hash that was just missing ovn is no longer available | 15:28 |
| rlandy | dasm|ruck: what's our next best option | 15:28 |
| dasm|ruck | checking what we got | 15:28 |
| rlandy | chkumar|rover and I were looking at the last to hashes | 15:28 |
| rlandy | pls see what looks workable | 15:28 |
| rlandy | waiting for promo of tripleo component on rhel-9 to run | 15:29 |
| rlandy | dasm|ruck: ^^ | 15:29 |
| dasm|ruck | ack | 15:29 |
| rlandy | then will reckick integration line there | 15:29 |
| rlandy | downstream | 15:29 |
| dasm|ruck | k | 15:29 |
| rlandy | frenzy_friday: want to meet now rather? | 15:30 |
| rlandy | meeting in that slot was moved | 15:30 |
| frenzy_friday | rlyep sure | 15:32 |
| frenzy_friday | rlandy, yep sure | 15:33 |
| rlandy | frenzy_friday: joining | 15:34 |
| rlandy | dviroel: can you vote/w+ https://review.opendev.org/c/openstack/tripleo-ci/+/850359 | 15:38 |
| ysandeep | dviroel++ | 15:52 |
| dviroel | ysandeep++ great chat | 15:53 |
| Tengu | rlandy: fyi, I found the right way to get my logs for nftables. Reviews have been updated accordingly. | 15:54 |
| rlandy | awesome thaks | 15:54 |
| rlandy | thanks | 15:54 |
| Tengu | the "audit" level is terrible - and we can't extend it. So I'm pushing everything in the syslog, with *all* the available data. | 15:55 |
| Tengu | and the file is now created using a crafted "journalctl" command :). | 15:55 |
| dviroel | rlandy: done | 15:55 |
| Tengu | wokay. that's a good thing. Log is always good. | 15:56 |
| Tengu | now, time to drop :) | 15:56 |
| dasm|ruck | dviroel: do you have a few? i'm hunting for a good dlrn hash to promo cs9 master. currently i'm finding only empty ones. and the ones i found usually have about 5 broken jobs. | 15:59 |
| dasm|ruck | tdf, i'm having hard times doing that all manually | 16:00 |
| rlandy | lunch - brb | 16:01 |
| dviroel | dasm|ruck: hum, what about this one: https://review.rdoproject.org/zuul/buildset/ef55f894273c4db19947ebe9022c8554 | 16:02 |
| dasm|ruck | lemme see what we got here | 16:02 |
| dviroel | 54efed52e5e55b97639d996485251bdf | 16:02 |
| dasm|ruck | thx | 16:02 |
| dasm|ruck | 3 jobs. not great, not bad. i'll try this one | 16:03 |
| dviroel | yep, 3 ovbs | 16:03 |
| dasm|ruck | recently there was no clear run (that i've seen) with ovb | 16:04 |
| dasm|ruck | and it's a lot of random errors, sometimes it doesn't even get to tempest | 16:04 |
| dviroel | dasm|ruck: check internal ones, to see if you have better results there | 16:05 |
| dviroel | that was that saved my shift | 16:05 |
| dasm|ruck | that's an idea. i'll do that, thanks | 16:05 |
| dviroel | ++ | 16:05 |
| * dviroel lunch | 16:05 | |
| dasm|ruck | dviroel: but, we're talking about cs9 master. there is no internal one, right? | 16:05 |
| *** dviroel is now known as dviroel|lunch | 16:06 | |
| dviroel|lunch | dasm|ruck: you can still trigger those jobs thre | 16:06 |
| dasm|ruck | i'll need to learn how to do that | 16:06 |
| dviroel|lunch | dasm|ruck: like periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-internal-master | 16:07 |
| dviroel|lunch | if needed | 16:08 |
| dasm|ruck | ack, thx | 16:08 |
| *** ysandeep is now known as ysandeep|out | 16:11 | |
| * ysandeep|out out, see you all tomorrow o/ | 16:12 | |
| dasm|ruck | ysandeep|out: o/ | 16:12 |
| dasm|ruck | hmm.. there are no internal jobs for fs020, fs064, fs039 | 16:13 |
| frenzy_friday | rlandy, container build testproj passed. Image build running | 16:55 |
| rlandy | great | 16:55 |
| rlandy | dasm|ruck: hey ... | 16:56 |
| rlandy | if your current testproject fails, let's look at failures and see if it's safe to skip | 16:57 |
| dasm|ruck | rlandy: ack | 16:58 |
| dasm|ruck | that's the one running atm: https://review.rdoproject.org/r/c/testproject/+/42319 | 16:59 |
| dasm|ruck | here: https://review.rdoproject.org/zuul/status/change/42319,21 | 16:59 |
| rlandy | ok | 17:00 |
| dasm|ruck | another master promo is running atm. but it won't include https://review.opendev.org/c/openstack/tripleo-quickstart/+/850442 | 17:05 |
| dasm|ruck | this one just got merged few secs ago | 17:05 |
| *** dviroel|lunch is now known as dviroel | 17:23 | |
| rlandy | ok great | 17:54 |
| dasm|ruck | kicked off https://review.rdoproject.org/r/c/testproject/+/42374 to chase recent hash too | 18:35 |
| rlandy | dasm|ruck: ok - let's chat when you have both results | 18:59 |
| dasm|ruck | first one is red. it doesn't look as a valid hash, because it failed overcloud deployment | 19:00 |
| dasm|ruck | second one is still in progress | 19:00 |
| * dviroel stepping out for a few to run an errand | 19:34 | |
| *** dviroel is now known as dviroel|afk | 19:34 | |
| dasm|ruck | started another testproject with missing master hash for cs9: https://review.rdoproject.org/r/c/testproject/+/42319 | 19:40 |
| rlandy | periodic-tripleo-rhel-9-rhos-17-component-tripleo-promote-to-promoted-componentstestprojectmastercheck420692,4 | 19:59 |
| rlandy | 2 mins 8 secs2022-07-20 19:28:08SUCCESS | 19:59 |
| rlandy | ha - finally | 19:59 |
| rlandy | rekicking 17 on rhel9 | 20:00 |
| dasm|ruck | nice. at least something | 20:03 |
| dasm|ruck | rlandy: cs9 master is miserable | 20:03 |
| dasm|ruck | https://review.rdoproject.org/r/c/testproject/+/42319 and https://review.rdoproject.org/r/c/testproject/+/42374 are current jobs which need to pass | 20:03 |
| rlandy | dasm|ruck: ok - sec - just checking ceph stuff | 20:04 |
| rlandy | and then will look at master | 20:04 |
| dasm|ruck | k | 20:04 |
| dasm|ruck | looks like "periodic-tripleo-ci-centos-9-scenario002-standalone-master" is gonna pass. | 20:07 |
| dasm|ruck | so we would be down to 4 + (2 jobs still running) | 20:07 |
| rlandy | checking | 20:10 |
| rlandy | let's see what's actually failing | 20:11 |
| rlandy | 039 and 064 seem to be issues | 20:11 |
| dasm|ruck | last several times there were no consistent results | 20:11 |
| dasm|ruck | sometimes overcloud deployment failed. another times undercloud. | 20:11 |
| rlandy | there was an ipa upgrade in c9 | 20:11 |
| rlandy | let me check | 20:11 |
| dasm|ruck | hmm | 20:11 |
| rlandy | we had one pass: periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-masteropenstack/tripleo-cimasteropenstack-periodic-integration-main2 hrs 29 mins 27 secs2022-07-20 01:50:22SUCCESS | 20:13 |
| dasm|ruck | but the buildset was bad: https://review.rdoproject.org/zuul/buildset/c5c2da62331a4f59be0bd714f1700384 | 20:14 |
| dasm|ruck | afair that was the one started by chkumar|rover which i rekicked. and it failed | 20:14 |
| dasm|ruck | chkumar|rover tried this hash: https://review.rdoproject.org/r/c/testproject/+/44080 | 20:16 |
| dasm|ruck | fcf18e1661b4add7eb1b3b2c761e1c63 | 20:16 |
| rlandy | dasm|ruck: so this hash looks best: | 20:18 |
| rlandy | https://trunk.rdoproject.org/api-centos9-master-uc/api/civotes_agg_detail.html?ref_hash=fcf18e1661b4add7eb1b3b2c761e1c63 | 20:18 |
| rlandy | correct | 20:18 |
| rlandy | it passed fs035 | 20:18 |
| rlandy | fs001 internal | 20:18 |
| rlandy | fs064 | 20:18 |
| rlandy | looking at full-tempest-api | 20:19 |
| dasm|ruck | ack | 20:19 |
| rlandy | https://logserver.rdoproject.org/80/44080/1/check/periodic-tripleo-ci-centos-9-standalone-full-tempest-api-master/c9daad9/logs/undercloud/var/log/tempest/stestr_results.html.gz | 20:20 |
| rlandy | and | 20:20 |
| rlandy | https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-full-tempest-api-master/3737f08/logs/undercloud/var/log/tempest/stestr_results.html.gz | 20:20 |
| rlandy | are diff | 20:20 |
| rlandy | so the tests that failed in one, passed in the other | 20:21 |
| dasm|ruck | yes. looks promising | 20:21 |
| rlandy | check fs020 | 20:21 |
| dasm|ruck | one failed undercloud, another overcloud | 20:22 |
| dasm|ruck | no tempest runs | 20:22 |
| dasm|ruck | https://logserver.rdoproject.org/80/44080/1/check/periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-master/f922999/job-output.txt | 20:22 |
| dasm|ruck | https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-master/7212730/job-output.txt | 20:22 |
| rlandy | stack deploy | 20:23 |
| rlandy | yeah - checking | 20:23 |
| rlandy | https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-master/30ed97d/logs/undercloud/var/log/tempest/stestr_results.html.gz | 20:24 |
| rlandy | nest hash gets to tempest and fails one test | 20:24 |
| rlandy | dasm|ruck: is there a rerun of fcf18e1661b4add7eb1b3b2c761e1c63 currently going? | 20:24 |
| dasm|ruck | no | 20:25 |
| dasm|ruck | i'm rerunning promotion ongoing promotion | 20:25 |
| dasm|ruck | we can rekick this one: https://review.rdoproject.org/r/c/testproject/+/44080 | 20:26 |
| rlandy | dasm|ruck: let;s try promote fcf18e1661b4add7eb1b3b2c761e1c63 | 20:26 |
| rlandy | pls put up a patch to skip what is missing there | 20:26 |
| rlandy | we will revert afterwards | 20:26 |
| rlandy | otherwise we will be too far out | 20:26 |
| rlandy | enough here | 20:26 |
| dasm|ruck | ack | 20:26 |
| dasm|ruck | i'm starting internal fs020 here: https://code.engineering.redhat.com/gerrit/c/testproject/+/419956 | 20:34 |
| dasm|ruck | if we're gonna have it, we can skip jobs | 20:35 |
| dasm|ruck | oh, we don't have internal fs020 O/ | 20:36 |
| dasm|ruck | :/ | 20:36 |
| dasm|ruck | rlandy: we're missing fs020, i wanted to run internal one, but there is no internal one. | 20:38 |
| dasm|ruck | skiptest: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/44087 | 20:38 |
| dasm|ruck | so we're short of fs020 | 20:38 |
| * dasm|ruck is checking how to quickly bring up internal job. | 20:38 | |
| dasm|ruck | ok, updated fs020 internal: https://code.engineering.redhat.com/gerrit/c/testproject/+/419956 | 20:43 |
| rlandy | dasm|ruck: https://code.engineering.redhat.com/gerrit/gitweb?p=openstack/tripleo-ci-internal-jobs.git;a=blob;f=zuul.d/ovb-rdo.yaml;h=f147867a62294339ec3edd3e3791147291e233f4;hb=HEAD | 20:43 |
| dasm|ruck | i added the job, based on this file, to testproject | 20:43 |
| rlandy | dasm|ruck: need to run out for a few - will be back | 21:04 |
| rlandy | if you go before I am back, please leave notes or a patch I should merge/revert | 21:04 |
| dasm|ruck | k | 21:04 |
| dasm|ruck | i left notes already | 21:04 |
| rlandy | dasm|ruck: so there is no skip patch yet? | 21:08 |
| rlandy | no roblem if not | 21:08 |
| *** rlandy is now known as rlandy|biab | 21:10 | |
| dasm|ruck | rlandy|biab: there is https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/44087 | 21:32 |
| dasm|ruck | but still waiting for fs020 | 21:32 |
| *** rlandy|biab is now known as rlandy | 21:33 | |
| rlandy | dasm|ruck: thanks - looking | 21:33 |
| dasm|ruck | those are jobs which tempest passed one way or another | 21:33 |
| rlandy | ack | 21:33 |
| rlandy | best we can do atm | 21:34 |
| dasm|ruck | the only one which is still unknown is this fs020 | 21:34 |
| dasm|ruck | https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status/change/419956,7 | 21:34 |
| rlandy | ack ok | 21:36 |
| rlandy | dasm|ruck: stepping out - will be back in about 45 mins | 22:14 |
| *** rlandy is now known as rlandy|bbl | 22:15 | |
| dasm|ruck | ack | 23:11 |
| dasm|ruck | internal fs020 > TASK [os_tempest : Execute tempest tests] | 23:12 |
| dasm|ruck | running recheck again on https://review.rdoproject.org/r/c/testproject/+/42319 (latest master promo) | 23:20 |
| dasm|ruck | for fs020 internal > Playbook run of ovb.yml failed | 23:20 |
| dasm|ruck | checking error in a minute | 23:20 |
| dasm|ruck | (it's not finished yet) | 23:21 |
| dasm|ruck | be back in few, to see how fs020 done | 23:22 |
| *** rlandy|bbl is now known as rlandy | 23:30 | |
| rlandy | ok | 23:30 |
| rlandy | dasm|ruck: think they bothe failed fs020 | 23:33 |
| rlandy | let's just skip and promote | 23:33 |
| rlandy | they are tempest failures this time | 23:34 |
| rlandy | not deploy | 23:34 |
| dasm|ruck | k | 23:35 |
| dasm|ruck | rlandy: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/44087 | 23:35 |
| dasm|ruck | rlandy: i updated the patch ^ it disables all failed jobs for aforementioned hash | 23:40 |
| dasm|ruck | chkumar|rover: please take a look at ^ cs9 master is in decend-ish shape. | 23:40 |
| dasm|ruck | i hope it's gonna be better tomorrow with more recent hash. | 23:41 |
| * dasm|ruck => offline | 23:41 | |
| *** dasm|ruck is now known as dasm|off | 23:41 | |
| rlandy | dasm|off: thanks | 23:48 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!