| *** tosky has quit IRC | 00:10 | |
| weshay|ruck | rlandy, shall we promote before it's really late? | 00:22 |
|---|---|---|
| rlandy | weshay|ruck: if you want to | 00:22 |
| weshay|ruck | k.. I'll do it | 00:22 |
| rlandy | but multinode is still broken | 00:22 |
| rlandy | weshay|ruck: I think it's the av virt module | 00:22 |
| rlandy | watched the deploy and when we lost ssh | 00:22 |
| rlandy | looks like some virt actions | 00:23 |
| weshay|ruck | oh.. the problem w/ the multinode job is the virt av module | 00:23 |
| weshay|ruck | hrm.. well that may bite us upstream | 00:24 |
| weshay|ruck | odd that it doesn't hit the other jobs... we sure we have virt av on the other jobs? | 00:24 |
| rlandy | weshay|ruck: would any job other than multinode care? | 00:25 |
| rlandy | idk | 00:25 |
| rlandy | after you promote we need to merge the two patches posted | 00:25 |
| weshay|ruck | k.. it's running | 00:25 |
| weshay|ruck | rlandy, k.. it's done | 00:29 |
| rlandy | ok | 00:29 |
| rlandy | weshay|ruck: let's merge | 00:29 |
| weshay|ruck | you have +2 on both from me | 00:29 |
| rlandy | weshay|ruck: one submited | 00:29 |
| rlandy | 1 w'ed | 00:29 |
| weshay|ruck | rlandy, k.. so let's check in tomorrow and check things out? and continue the featureset010 debug? | 00:30 |
| rlandy | weshay|ruck: ack | 00:31 |
| weshay|ruck | rlandy, cool cool.. go relax :) | 00:31 |
| weshay|ruck | have a good night! | 00:31 |
| rlandy | thanks - you too | 00:31 |
| *** rlandy has quit IRC | 00:46 | |
| *** jmasud has joined #oooq | 00:55 | |
| *** jmasud_ has quit IRC | 00:58 | |
| *** jmasud has quit IRC | 01:04 | |
| *** jmasud has joined #oooq | 01:06 | |
| *** jmasud has quit IRC | 02:08 | |
| *** jmasud has joined #oooq | 02:15 | |
| *** jmasud has quit IRC | 02:58 | |
| *** jmasud has joined #oooq | 03:05 | |
| *** ysandeep|away is now known as ysandeep|ruck | 03:41 | |
| *** jmasud has quit IRC | 04:10 | |
| *** ykarel has joined #oooq | 04:11 | |
| *** jmasud has joined #oooq | 04:20 | |
| *** jmasud has quit IRC | 04:57 | |
| *** jmasud has joined #oooq | 05:00 | |
| *** ratailor has joined #oooq | 05:01 | |
| *** udesale has joined #oooq | 05:21 | |
| *** marios has joined #oooq | 05:57 | |
| *** saneax has joined #oooq | 06:07 | |
| *** jbadiapa has quit IRC | 06:45 | |
| ysandeep|ruck | arxcruz, o/ https://bugs.launchpad.net/tripleo/+bug/1916695 | 06:59 |
| openstack | Launchpad bug 1916695 in tripleo "periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset010-rocky is failing with NeutronError: "Invalid input for operation: segmentation_id requires physical_network for VLAN provider network"" [Critical,Triaged] | 06:59 |
| *** jpodivin has joined #oooq | 07:04 | |
| *** jpodivin has joined #oooq | 07:21 | |
| *** slaweq has joined #oooq | 07:27 | |
| *** sshnaidm|afk is now known as sshnaidm | 07:38 | |
| *** jmasud has quit IRC | 07:38 | |
| ysandeep|ruck | slaweq, Hey o/ When you have time could you please look at https://bugs.launchpad.net/tripleo/+bug/1916695 , rocky is not very active these days.. but its still not eol upstream.. | 07:55 |
| openstack | Launchpad bug 1916695 in tripleo "periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset010-rocky is failing with NeutronError: "Invalid input for operation: segmentation_id requires physical_network for VLAN provider network"" [Critical,Triaged] | 07:55 |
| *** amoralej|off is now known as amoralej | 08:00 | |
| *** ysandeep|ruck is now known as ysandeep|lunch | 08:00 | |
| slaweq | ysandeep|lunch: hi | 08:07 |
| slaweq | sure, looking | 08:07 |
| slaweq | ysandeep|lunch: I think that we missed that patch in rocky https://review.opendev.org/q/I021015709c8254ff5936ad08d61e1e02ab58c44f | 08:18 |
| slaweq | I proposed backports just now | 08:18 |
| slaweq | I hope it will help | 08:18 |
| *** ykarel_ has joined #oooq | 08:31 | |
| *** ykarel has quit IRC | 08:34 | |
| *** ykarel_ is now known as ykarel | 08:38 | |
| soniya29 | arxcruz, kopecmartin , please edit/add the today's agenda for tempest meeting:- https://hackmd.io/fIOKlEBHQfeTZjZmrUaEYQ | 08:50 |
| *** jpena|off is now known as jpena | 08:57 | |
| zbr | apparently i am assaulted by private spam messages on irc, i am the only one? got ~20-30 this morning and they still appear to happen. | 08:58 |
| zbr | they seem like random texts with no clear meaning (drunken AI style) | 08:58 |
| *** ysandeep|lunch is now known as ysandeep|ruck | 09:10 | |
| ysandeep|ruck | slaweq, thank you :D | 09:10 |
| *** holser_ has joined #oooq | 09:10 | |
| chandankumar | zbr: same here, you can disable it by /mode <your irc nick> + R make sure your nick is authenticated with freenode server | 09:11 |
| chandankumar | marios: zbr sshnaidm arxcruz ysandeep|ruck https://review.opendev.org/c/openstack/tripleo-docs/+/776921 when free | 09:12 |
| zbr | chandankumar: thanks! Is that persistent it should I add it to my login script for the user? | 09:14 |
| chandankumar | zbr: it is going to be persistant, as I enabled it in weechat | 09:14 |
| chandankumar | may be put it in login script after authentication command | 09:15 |
| marios | chandankumar: zbr: i was wondering what the +Ri is next to my nick | 09:15 |
| marios | chandankumar: zbr: either i had it in my config or cos my nick is authenticated perhaps set by the server? | 09:15 |
| chandankumar | marios: iR got setted against nick after setting the mode +R | 09:16 |
| chandankumar | if +R mode is not setted, freenode admins might notified in the freenode tab | 09:17 |
| marios | chandankumar: i mean i didn't explicitly set anything myself, just my nick is authenticated | 09:17 |
| chandankumar | 01:57 -- glguy_ (x@freenode/staff/haskell.developer.glguy): [global notice] Due to today's high wave of spam, you might want to set yourself +R to block PMs from unidentifie | 09:17 |
| chandankumar | d users. In many clients this is "/mode YourNickHere +R" and we've enabled this as a default mode for the duration. | 09:17 |
| marios | chandankumar: so possibly the server sets +R for all authenticated nicks today cos there is some issue? don't know | 09:17 |
| chandankumar | marios: may be | 09:17 |
| marios | chandankumar: ah ack 11:17 < chandankumar> d users. In many clients this is "/mode YourNickHere +R" and we've enabled this as a default mode for the duration. | 09:18 |
| marios | default mode | 09:18 |
| chandankumar | yes | 09:18 |
| *** tosky has joined #oooq | 09:18 | |
| *** ksambor has quit IRC | 09:36 | |
| *** derekh has joined #oooq | 09:36 | |
| bhagyashris | ysandeep|ruck, hi, do we have new hash to promote for ussuri or victoria | 09:47 |
| bhagyashris | we are testing container-push on quay.io and we need new hash | 09:47 |
| ysandeep|ruck | bhagyashris, checking | 09:50 |
| ysandeep|ruck | ussuri - current-tripleo and triple-ci-testing is same.. checking victoria | 09:52 |
| arxcruz | ysandeep|ruck: hey, sorry, had some emergency this morning here | 09:52 |
| ysandeep|ruck | arxcruz, nw :D , everything is under control.. happy last day as ruck/rover this sprint | 09:53 |
| arxcruz | ysandeep|ruck: yes, it's TODAY!!!! | 09:53 |
| arxcruz | ysandeep|ruck: i'll drink two beers tonight | 09:54 |
| arxcruz | one for me, one for you :P | 09:54 |
| ysandeep|ruck | arxcruz, :D yeah! enjoy my first beer | 09:55 |
| ysandeep|ruck | bhagyashris, we don't have a new hash for ussuri, we have a new hash for victoria.. but line just triggered.. we need to wait for results | 09:56 |
| bhagyashris | ysandeep|ruck, thanks :) | 09:57 |
| ysandeep|ruck | arxcruz, any luck with stein containers :D | 09:58 |
| * ysandeep|ruck brb in few mins | 09:59 | |
| arxcruz | ysandeep|ruck: well, patch is up, and it works | 09:59 |
| ysandeep|ruck | arxcruz, can we repush the containers.. so that we unblock check/gate | 10:00 |
| *** dtantsur|afk is now known as dtantsur | 10:07 | |
| *** ysandeep|ruck is now known as ysandeep|brb | 10:10 | |
| arxcruz | ysandeep|brb: i update the script to get the x86 and remove it as well and i'll push manually db93471c423322fc24743db3545da0a11b073599_a1cf5a25 | 10:24 |
| chandankumar | ysandeep|brb: we got an issue for you | 10:24 |
| arxcruz | ysandeep|brb: ok? | 10:24 |
| chandankumar | ysandeep|brb: https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-build-containers-ubi-8-push-victoria/d2783f2/job-output.txt | 10:24 |
| *** ysandeep|brb is now known as ysandeep|ruck | 10:24 | |
| ysandeep|ruck | arxcruz, yes thank you | 10:25 |
| ysandeep|ruck | chandankumar, looking | 10:26 |
| chandankumar | ysandeep|brb: http://paste.openstack.org/show/802954/ | 10:27 |
| arxcruz | ysandeep|ruck: it's being pushed | 10:27 |
| arxcruz | ysandeep|ruck: https://quay.io/repository/tripleostein/centos-binary-octavia-api?tab=tags | 10:27 |
| arxcruz | you can see db93471c423322fc24743db3545da0a11b073599_a1cf5a25_x86_64 and db93471c423322fc24743db3545da0a11b073599_a1cf5a25 there | 10:27 |
| ysandeep|ruck | arxcruz, great! | 10:28 |
| chandankumar | ysandeep|ruck: may be setting buildah --log-level error will fix the issue | 10:29 |
| ysandeep|ruck | chandankumar, https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-build-containers-ubi-8-push-victoria job was working good so far.. comparing older results | 10:30 |
| ysandeep|ruck | buildah.x86_64 1.15.1-2.module_el8.3.0+475+c50ce30b @appstream | 10:34 |
| ysandeep|ruck | buildah.x86_64 1.16.7-4.module_el8.3.0+699+d61d9c41 @appstream | 10:34 |
| ysandeep|ruck | chandankumar, ^^ new buildah rpm | 10:34 |
| * ysandeep|ruck reporting bug | 10:35 | |
| ysandeep|ruck | https://bugs.launchpad.net/tripleo/+bug/1916742 | 10:45 |
| openstack | Launchpad bug 1916742 in tripleo "Victoria Container build job is failing, buildah deprecations warning is added as a first item in the list instead of container name" [Critical,Triaged] | 10:45 |
| ysandeep|ruck | I think other branches could be affected as well.. but ussuri job triggered in the morning was good.. had older buildah version. | 10:45 |
| arxcruz | ysandeep|ruck: done, i think we are good now, can you verify ? | 10:59 |
| ysandeep|ruck | arxcruz, https://quay.io/repository/tripleostein/centos-binary-aodh-api?tab=tags looks good, let me post recheck on failing patch | 10:59 |
| ysandeep|ruck | arxcruz, posted recheck on marios's failing patch https://review.opendev.org/c/openstack/paunch/+/774581 | 11:00 |
| marios | thank you ysandeep|ruck arxcruz | 11:01 |
| ysandeep|ruck | folks o/ could you please add to your review list https://review.rdoproject.org/r/#/c/32054/ | 11:03 |
| marios | ysandeep|ruck: ack but please check minor nits needs an update | 11:05 |
| ysandeep|ruck | marios, sry wrong copy paste | 11:06 |
| ysandeep|ruck | marios, this is the correct patch https://review.rdoproject.org/r/#/c/32081/ | 11:06 |
| ysandeep|ruck | ^^ above was testproject for ^^ | 11:07 |
| marios | ysandeep|ruck: thanks ;) | 11:07 |
| ysandeep|ruck | ++ thank you | 11:09 |
| marios | ysandeep|ruck: heh sorry i mean seriously... reviewing a testproject patch ;) in my defense i have failing test-requirements issue in terminal and thought i'd quickly check your patch ;) | 11:10 |
| ysandeep|ruck | :D | 11:11 |
| *** udesale_ has joined #oooq | 11:14 | |
| *** udesale has quit IRC | 11:18 | |
| ysandeep|ruck | arxcruz, we have a new promotion blocker https://bugs.launchpad.net/tripleo/+bug/1916742 , I am checking as well | 11:29 |
| openstack | Launchpad bug 1916742 in tripleo "Victoria Container build job is failing, buildah deprecations warning is added as a first item in the list instead of container name" [Critical,Triaged] | 11:30 |
| ysandeep|ruck | we have new podman and buildah and some other packages in job.. | 11:30 |
| ysandeep|ruck | chandankumar, i found https://github.com/containers/podman/issues/6985 , could podman log level instead of buildah | 11:35 |
| chandankumar | sshnaidm: marios zbr https://review.rdoproject.org/r/#/c/32059/ and https://review.rdoproject.org/r/32092 please | 11:40 |
| marios | chandankumar: will do added to list | 11:41 |
| sshnaidm | chandankumar, thanks | 11:41 |
| arxcruz | ysandeep|ruck: for the program call today, do you want to do it or should i ? | 12:02 |
| arxcruz | ysandeep|ruck: i saw you said you would attend but up to you | 12:02 |
| ysandeep|ruck | arxcruz, i can cover it | 12:03 |
| *** akahat has quit IRC | 12:05 | |
| *** akahat has joined #oooq | 12:05 | |
| ysandeep|ruck | arxcruz, i will not be able to cover cix today if you can cover that, i have updated the card on which i was working/ moved some to done which are fixed | 12:07 |
| arxcruz | ysandeep|ruck: sure | 12:07 |
| ysandeep|ruck | arxcruz, thanks! | 12:07 |
| *** ratailor has quit IRC | 12:07 | |
| ysandeep|ruck | arxcruz, stein recheck failed.. https://47068408c5c86a544ba7-1e843a574593cd5401f601b6248e91e4.ssl.cf5.rackcdn.com/774581/2/check/tripleo-ci-centos-7-scenario002-standalone/b5fa2b2/logs/undercloud/home/zuul/standalone_deploy.log , looks like we don't have all the containers with db hash for ex | 12:14 |
| ysandeep|ruck | https://quay.io/repository/tripleostein/centos-binary-nova-compute?tab=tags | 12:14 |
| arxcruz | ysandeep|ruck: just notice this: | 12:16 |
| arxcruz | ERRO[0474] Failed to copy container image: Error in copy the image: Error initializing source docker://trunk.registry.rdoproject.org/tripleostein/centos-binary-ovn-base:current-tripleo: Error reading manifest current-tripleo in trunk.registry.rdoproject.org/tripleostein/centos-binary-ovn-base: manifest unknown: manifest unknown | 12:16 |
| arxcruz | ysandeep|ruck: there are a bunch of containers with this failure | 12:19 |
| *** rlandy has joined #oooq | 12:32 | |
| *** jpena is now known as jpena|lunch | 12:38 | |
| chandankumar | rlandy: please+w it https://review.rdoproject.org/r/32059 | 12:54 |
| weshay|ruck | ysandeep|ruck, you need me to force promote tripleo component? | 12:57 |
| soniya29 | chandankumar, kopecmartin, arxcruz, weshay|ruck tempest meeting in 3 minutes | 12:57 |
| weshay|ruck | ysandeep|ruck, for https://bugs.launchpad.net/bugs/1915519 | 12:57 |
| openstack | Launchpad bug 1915519 in tripleo "[Train][CentOS7][scenario004] Failing with Error: 'ip-192.168.24.3' already exists. Too many tries"" [Critical,Triaged] - Assigned to Michele Baldessari (michele) | 12:57 |
| ysandeep|ruck | weshay|ruck, no.. current -> consistent promotion is stuck.. because of validation rpm failure | 12:59 |
| ysandeep|ruck | https://trunk.rdoproject.org/centos7-train/status_report.html | 12:59 |
| ysandeep|ruck | weshay|ruck, ykarel is working with validations squad | 12:59 |
| *** amoralej is now known as amoralej|lunch | 12:59 | |
| *** ysandeep|ruck is now known as ysandeep|prgm_ca | 12:59 | |
| soniya29 | akahat, ^^ | 12:59 |
| weshay|ruck | jebus | 12:59 |
| akahat | soniya29, joining | 13:00 |
| rlandy | chandankumar: done | 13:12 |
| rlandy | chandankumar: sorry - you had some other reviews you wanted looked at? | 13:12 |
| chandankumar | rlandy: https://review.opendev.org/c/openstack/tripleo-docs/+/776921 https://e55d75885c9c562cc63e-7ca08d7dc339532aeab9467cc90f03e8.ssl.cf5.rackcdn.com/776921/5/check/openstack-tox-docs/3f23faa/docs/ci/tripleo_dependencies_pipelines.html you might like it | 13:18 |
| rlandy | nice - reading | 13:19 |
| rlandy | chandankumar: looks good - few minor comments in line ... https://review.opendev.org/c/openstack/tripleo-docs/+/776921 | 13:23 |
| chandankumar | rlandy: done | 13:26 |
| rlandy | chandankumar: thanks +2 | 13:30 |
| *** dtantsur is now known as dtantsur|brb | 13:35 | |
| *** jpena|lunch is now known as jpena | 13:39 | |
| *** ysandeep|prgm_ca is now known as ysandeep | 13:51 | |
| *** ysandeep is now known as ysandeep|mtg | 13:55 | |
| *** amoralej|lunch is now known as amoralej | 13:58 | |
| ykarel | ysandeep|mtg, https://review.opendev.org/c/openstack/validations-common/+/777308 will fix that | 14:05 |
| ysandeep|mtg | ykarel, ack | 14:10 |
| ysandeep|mtg | chandankumar, fyi.. patch didn't work.. on every buildah command i am seeing a warning - http://paste.openstack.org/ | 14:29 |
| chandankumar | ysandeep|mtg: link? | 14:30 |
| ysandeep|mtg | chandankumar, periodic-tripleo-ci-build-containers-ubi-8-push-victoria https://review.rdoproject.org/zuul/build/4862ed44a71748eaab2944b2ffe09074 : FAILURE in 56m 36s | 14:30 |
| ysandeep|mtg | chandankumar, testproject https://review.rdoproject.org/r/#/c/28446/ | 14:30 |
| ysandeep|mtg | chandankumar, i have the node hold.. if you want to look at env directly | 14:30 |
| ysandeep|mtg | weshay|ruck, bhagyashris, soniya29 arxcruz want to do ruck/rover sync now | 14:33 |
| chandankumar | ysandeep|mtg: oh, I need to make one more changes | 14:33 |
| chandankumar | ysandeep|mtg: can I update your patch | 14:34 |
| ysandeep|mtg | chandankumar, sure | 14:34 |
| chandankumar | ysandeep|mtg: https://opendev.org/openstack/tripleo-ci/src/branch/master/roles/build-containers/tasks/main.yaml#L242 | 14:34 |
| chandankumar | may be here | 14:34 |
| marios | arxcruz: see pvt chat | 14:34 |
| chandankumar | ysandeep|mtg: http://paste.openstack.org/raw/802961/ add the keys there | 14:35 |
| chandankumar | and ip please want to see something there first | 14:35 |
| arxcruz | soniya29: bhagyashris https://review.opendev.org/c/openstack/tripleo-ci/+/777378 | 14:38 |
| ysandeep|mtg | chandankumar, give me 5 mins.. in ruck/rover sync | 14:38 |
| weshay|ruck | ysandeep|mtg, bhagyashris soniya29 arxcruz can you folks hold 22 min | 14:38 |
| weshay|ruck | and I can join as well? | 14:38 |
| arxcruz | weshay|ruck: we are doing it right now | 14:39 |
| weshay|ruck | bhagyashris, akahat is the promoter ready to try centos8 train? | 14:39 |
| ysandeep|mtg | chandankumar, ip in pm | 14:43 |
| ysandeep|mtg | added your keys | 14:43 |
| bhagyashris | weshay|ruck, we stop the dlrn-promoter service to test quay.io | 14:43 |
| weshay|ruck | bhagyashris, k.. /me notes we need to promote c8-train | 14:43 |
| akahat | weshay|ruck, y'day it ran but it did't found the containers: http://10.0.148.74/promoter_logs/centos8_train.log-20210222 | 14:43 |
| weshay|ruck | and we need to attempt it today | 14:43 |
| weshay|ruck | akahat, is it set to how many hashes..5 or 50? | 14:44 |
| akahat | weshay|ruck, 50 | 14:44 |
| bhagyashris | weshay|ruck, in rr sync will update soon | 14:46 |
| *** jlarriba has quit IRC | 14:50 | |
| ysandeep|mtg | arxcruz, podman pull docker.io/tripleostein/centos-binary-nova-compute:db93471c423322fc24743db3545da0a11b073599_a1cf5a25 | 14:51 |
| *** dsneddon has quit IRC | 14:52 | |
| weshay|ruck | ysandeep|mtg, bhagyashris link to the sync? | 14:52 |
| ysandeep|mtg | weshay|ruck, https://meet.google.com/dtu-qdgf-tga?authuser=0 | 14:53 |
| bhagyashris | weshay|ruck, https://meet.google.com/dtu-qdgf-tga?authuser=0 | 14:53 |
| *** dsneddon has joined #oooq | 14:54 | |
| frenzy_friday | rlandy, do you know what this error might be " findpkg: No package found in distroinfo for query: rdo-jobs " ? | 14:55 |
| *** chandankumar has quit IRC | 14:58 | |
| arxcruz | podman pull docker.io/tripleostein/centos-binary-redis-base:db93471c423322fc24743db3545da0a11b073599_a1cf5a25 | 14:59 |
| ysandeep|mtg | weshay|ruck, fyi.. https://bugs.launchpad.net/tripleo/+bug/1916742 we have already gone through this with bhagyashris and soniya29 | 15:01 |
| openstack | Launchpad bug 1916742 in tripleo "Victoria Container build job is failing, buildah deprecations warning is added as a first item in the list instead of container name" [Critical,Triaged] | 15:01 |
| * akahat will be back after some time | 15:03 | |
| *** dsneddon has quit IRC | 15:06 | |
| *** dsneddon has joined #oooq | 15:09 | |
| rlandy | frenzy_friday: pls link | 15:09 |
| frenzy_friday | rlandy, https://logserver.rdoproject.org/80/32080/4/check/periodic-tripleo-ci-centos-8-standalone-full-tempest-api-master/6ca069e/logs/quickstart_install.log - | 15:10 |
| frenzy_friday | rdopkg findpkg -s rdo-jobs -i rdoinfo/rdo-full.yml | 15:10 |
| frenzy_friday | + awk '/^name/{print $2}; {print "findpkg: " $0 > "/dev/stderr"}' | 15:10 |
| frenzy_friday | findpkg: | 15:10 |
| frenzy_friday | findpkg: No package found in distroinfo for query: rdo-jobs | 15:10 |
| rlandy | frenzy_friday: one sec - putting in patch - will take a look | 15:12 |
| rlandy | sorry - just out of prod chain meeting | 15:13 |
| frenzy_friday | no prob | 15:13 |
| *** dtantsur|brb is now known as dtantsur | 15:18 | |
| *** saneax has quit IRC | 15:19 | |
| *** ysandeep|mtg is now known as ysandeep | 15:20 | |
| *** bhagyashris is now known as bhagyashri|ruck | 15:20 | |
| rlandy | ysandeep: weshay|ruck: going to BZ the multinode | 15:20 |
| rlandy | 8.4 issue | 15:20 |
| rlandy | in case someone has a clue while our debug is in place | 15:21 |
| rlandy | ysandeep: weshay|ruck: fyi - to fix missing netstat https://code.engineering.redhat.com/gerrit/#/c/228956/ | 15:21 |
| *** soniya29 is now known as soniya29|ruck | 15:22 | |
| ysandeep | rlandy, i didn't get time to look at that but could be iptables related.. will try to debug tomorrow.. fyi.. i had to add some iptables rule in images in the past | 15:24 |
| ysandeep | rlandy, https://code.engineering.redhat.com/gerrit/#/q/owner:sandyada%2540redhat.com+project:openstack/sf-config | 15:24 |
| rlandy | ysandeep: thanks - will look into that later | 15:25 |
| rlandy | it's our last hurdle | 15:25 |
| ysandeep | rlandy, ack otherwise tomorrow.. i will changes root password of secondary node before deployment.. capture iptables rules... after deployment connect to vm using horizon console.. and compare iptable rules/ ssh config n all | 15:25 |
| rlandy | {# Enable iptables to ensure it is started on boot #} | 15:26 |
| rlandy | --run-command 'systemctl enable iptables.service' | 15:26 |
| rlandy | ^^ in 8.4 image | 15:26 |
| ysandeep | ack | 15:27 |
| *** chandan_kumar has joined #oooq | 15:29 | |
| ysandeep | weshay|ruck, arxcruz cix time | 15:30 |
| bhagyashri|ruck | akahat, | 15:30 |
| *** chandan_kumar is now known as chandankumar | 15:32 | |
| arxcruz | https://review.opendev.org/c/openstack/tripleo-ci/+/777378 | 15:33 |
| arxcruz | weshay|ruck: https://review.opendev.org/c/openstack/tripleo-ci/+/777378 | 15:34 |
| rlandy | frenzy_friday: looks like you have a bunch of tempest failures there ... https://logserver.rdoproject.org/80/32080/4/check/periodic-tripleo-ci-centos-8-standalone-full-tempest-api-master/6ca069e/logs/undercloud/var/log/tempest/stestr_results.html.gz | 15:34 |
| rlandy | maybes skiplist stuff | 15:34 |
| *** jbadiapa has joined #oooq | 15:35 | |
| rlandy | few more minutes - will give it some time | 15:35 |
| frenzy_friday | thanks, lemme check the tempest logs | 15:39 |
| ysandeep | sshnaidm, fyi.. recheck on https://review.opendev.org/c/openstack/paunch/+/774581 was intentional.. arxcruz pushed the containers.. but turns out we are still missing some containers | 15:42 |
| sshnaidm | ysandeep, ack, didn't know that | 15:43 |
| sshnaidm | ysandeep, good we have it worked on.. | 15:43 |
| bhagyashri|ruck | weshay|ruck, reverted changes , now promoter server is ready for docker.io | 15:46 |
| * weshay|ruck logs in | 15:46 | |
| weshay|ruck | and looks | 15:46 |
| weshay|ruck | bhagyashri|ruck, ok.. so I'll enable "JUST TRAIN" and turn the promoter on | 15:47 |
| weshay|ruck | ack? | 15:47 |
| bhagyashri|ruck | weshay|ruck, sure | 15:48 |
| weshay|ruck | k.. 1 sec | 15:48 |
| bhagyashri|ruck | ok | 15:49 |
| weshay|ruck | bhagyashri|ruck, k.. thanks it's promoting | 15:49 |
| bhagyashri|ruck | weshay|ruck, ok | 15:50 |
| weshay|ruck | arxcruz, ysandeep bhagyashri|ruck soniya29|ruck centos-8 train is promoting now | 15:50 |
| bhagyashri|ruck | weshay|ruck, leaving for the day | 15:50 |
| weshay|ruck | bhagyashri|ruck, 0/ | 15:50 |
| * bhagyashri|ruck out | 15:50 | |
| *** jpodivin has quit IRC | 15:50 | |
| *** jlarriba has joined #oooq | 15:52 | |
| weshay|ruck | akahat, you around? | 15:53 |
| weshay|ruck | akahat, on the c8 promoter | 15:53 |
| weshay|ruck | 2021-02-24 15:52:22,779 2771206 ERROR promoter Failed to promote overcloud images │················ | 15:53 |
| weshay|ruck | akahat, bhagyashri|ruck https://review.rdoproject.org/r/32100 | 16:06 |
| chandankumar | ysandeep: got the solution but as a workaround and found another bug in buildha | 16:07 |
| chandankumar | ysandeep: http://paste.openstack.org/show/802964/ | 16:07 |
| chandankumar | ysandeep: just remove libpod.conf file | 16:07 |
| ysandeep | chandankumar, ack.. /usr/share/containers/libpod.conf is provided by podman rpm.. i just checked that file was present in previous podman version as well | 16:09 |
| weshay|ruck | arxcruz, rlandy do you want to go early to get arxcruz out of here for the day? | 16:11 |
| arxcruz | weshay|ruck: fine to me | 16:11 |
| rlandy | weshay|ruck: ok | 16:12 |
| weshay|ruck | lets start in 3min | 16:12 |
| rlandy | ok | 16:12 |
| *** ysandeep is now known as ysandeep|dinner | 16:12 | |
| *** rlandy is now known as rlandy|mtg | 16:17 | |
| *** jmasud has joined #oooq | 16:22 | |
| chandankumar | sshnaidm: weshay|ruck rlandy|mtg ysandeep|dinner https://github.com/containers/buildah/issues/3040 fyi | 16:26 |
| chandankumar | ysandeep|dinner: let me know when you update the patch with work around | 16:27 |
| * akahat back | 16:29 | |
| akahat | weshay|ruck, chandankumar bhagyashri|ruck let's get this merged: https://review.rdoproject.org/r/#/c/32100/ | 16:30 |
| *** ysandeep|dinner is now known as ysandeep | 16:33 | |
| chandankumar | akahat: done | 16:41 |
| *** jmasud has quit IRC | 16:41 | |
| *** jmasud has joined #oooq | 16:44 | |
| ysandeep | chandankumar, hey o/ i am thinking about adding http://paste.openstack.org/show/802966/ here https://opendev.org/openstack/tripleo-ci/src/branch/master/roles/build-containers/tasks/main.yaml#L240 , will that work? | 16:47 |
| ysandeep | become: true as well | 16:48 |
| *** dtantsur is now known as dtantsur|brb | 16:50 | |
| *** udesale_ has quit IRC | 16:54 | |
| *** marios has quit IRC | 16:54 | |
| bhagyashri|ruck | weshay|ruck, akahat chandankumar train promoted https://trunk.rdoproject.org/centos8-train/current-tripleo/delorean.repo.md5 | 16:56 |
| chandankumar | ysandeep: yes sounds good | 16:58 |
| ysandeep | chandankumar, https://review.opendev.org/c/openstack/tripleo-ci/+/777378 | 16:59 |
| ysandeep | rechecking with testproject, let me know if any update needed in the meantime | 17:00 |
| chandankumar | ysandeep: we can use copy module for moving the file | 17:01 |
| ysandeep | chandankumar, copy module removes source file? let me check that | 17:02 |
| chandankumar | ysandeep: got the answer | 17:03 |
| chandankumar | it is ok | 17:03 |
| chandankumar | ysandeep: https://stackoverflow.com/questions/24162996/how-to-move-rename-a-file-using-an-ansible-task-on-a-remote-system | 17:03 |
| ysandeep | chandankumar, ack :D lets see how it goes | 17:04 |
| *** ykarel has quit IRC | 17:05 | |
| ysandeep | weshay|ruck, might need your help as its getting really late for me and chandankumar.. for the container build issue - here is the fix https://review.opendev.org/c/openstack/tripleo-ci/+/777378 , running in testproject https://review.rdoproject.org/r/#/c/28446/ here .. please feel free to update if needed.. otherwise we will continue tomorrow | 17:07 |
| ysandeep | bhagyashri|ruck, soniya29|ruck ^^ fyi.. | 17:08 |
| weshay|ruck | ysandeep, k.. thanks | 17:08 |
| weshay|ruck | ysandeep, thanks for your ruck / rover time :) | 17:08 |
| bhagyashri|ruck | ysandeep, ack | 17:08 |
| ysandeep | :D my pleasure | 17:08 |
| bhagyashri|ruck | thanks :) | 17:08 |
| rlandy|mtg | weshay|ruck: frenzy_friday: after compass discussion, we should talk about promotions | 17:08 |
| ysandeep | chandankumar, thank you for helping me :D | 17:08 |
| rlandy|mtg | master promotion requested on #tripleo | 17:09 |
| ysandeep | chandankumar, good night :) | 17:09 |
| *** ysandeep is now known as ysandeep|away | 17:11 | |
| *** amoralej is now known as amoralej|off | 17:14 | |
| *** derekh has quit IRC | 17:23 | |
| akahat | weshay|ruck, bhagyashri|ruck chandankumar why C-8/train promotion to current-tripleo-rdo is commented? | 17:23 |
| weshay|ruck | akahat, ya.. I'll put that back | 17:23 |
| *** derekh has joined #oooq | 17:23 | |
| weshay|ruck | just wanted to see the upstream promotion alone | 17:24 |
| akahat | okay cool | 17:24 |
| weshay|ruck | fixed | 17:24 |
| chandankumar | weshay|ruck: we tried pushing container to quay, we need to do some cleanup on the container-push role side | 17:24 |
| weshay|ruck | promoter is off | 17:24 |
| weshay|ruck | akahat, ^ | 17:24 |
| weshay|ruck | chandankumar, ok.. let's weigh that work against.. what it takes to integrate arxcruz's tool | 17:25 |
| akahat | weshay|ruck, yes. it's off. Do you want me to turn it on? | 17:25 |
| chandankumar | weshay|ruck: notes here https://hackmd.io/M1IjwgcPQLK_ttOLFUSPHQ#Uploading-cloud-images-to-quay | 17:25 |
| weshay|ruck | akahat, w/ regards to promotions.. we're good for a few days if you want / need to experiment a bit more | 17:25 |
| weshay|ruck | akahat, we can also turn it on for all non-master promotions for c8.. and see if hit issues | 17:26 |
| weshay|ruck | akahat, up to you... | 17:26 |
| *** derekh has quit IRC | 17:26 | |
| akahat | weshay|ruck, yeah.. sounds good. I'll trun it on. And keep eye on it. :) | 17:26 |
| *** derekh has joined #oooq | 17:26 | |
| weshay|ruck | akahat, k | 17:27 |
| weshay|ruck | akahat, update the dlrn-promoter.sh script | 17:27 |
| weshay|ruck | re: branches | 17:27 |
| weshay|ruck | aye | 17:27 |
| weshay|ruck | that one | 17:27 |
| akahat | weshay|ruck, what about c7? | 17:28 |
| weshay|ruck | akahat, that's one a different server right? | 17:29 |
| weshay|ruck | akahat, and we have a production one in vex | 17:29 |
| weshay|ruck | akahat, let's talk about c7 tomorrow | 17:29 |
| weshay|ruck | I'll schedule 15min | 17:29 |
| akahat | yeah.. i forgot. | 17:29 |
| akahat | okay that will work. | 17:29 |
| weshay|ruck | :) | 17:29 |
| chandankumar | rlandy|mtg: let me know when free wanted to talk on container tools deps pipeline https://review.rdoproject.org/r/#/c/31759/ not sure why https://logserver.rdoproject.org/59/31759/2/check/periodic-tripleo-ci-centos-8-standalone-container-tools-container-build-master/eae1fc9/logs/undercloud/var/log/extra/rpm-list.txt.gz podman 3 not getting installed but it showing here | 17:31 |
| chandankumar | https://logserver.rdoproject.org/59/31759/2/check/periodic-tripleo-ci-centos-8-standalone-container-tools-container-build-master/eae1fc9/logs/undercloud/home/zuul/control_test_diff_table.log.txt.gz please have a look, when free, thanks! | 17:31 |
| chandankumar | weshay|ruck: https://review.opendev.org/c/openstack/tripleo-docs/+/776921 please +w it then dependency pipeline docs will be live | 17:32 |
| *** derekh has quit IRC | 17:46 | |
| *** derekh has joined #oooq | 17:47 | |
| pojadhav | weshay|ruck, sshnaidm : when you are free please take a look at https://code.engineering.redhat.com/gerrit/#/c/228633/ | 17:49 |
| * pojadhav out | 17:49 | |
| weshay|ruck | k.. sec | 17:49 |
| pojadhav | also need reviews on https://review.rdoproject.org/r/#/c/31967/ and https://review.rdoproject.org/r/#/c/31970/ | 17:58 |
| *** jpena is now known as jpena|off | 17:59 | |
| *** derekh has quit IRC | 18:00 | |
| ysandeep|away | chandankumar, updating patchset | 18:15 |
| chandankumar | zbr: git-review is so slow on fedora 33? any things to do for optimizing, currently running python 3.9 | 18:21 |
| * ysandeep|away updated https://review.opendev.org/c/openstack/tripleo-ci/+/777378/, rechecked the testproject and goes away | 18:23 | |
| *** dtantsur|brb is now known as dtantsur | 18:31 | |
| weshay|ruck | pojadhav :) http://tripleo-cockpit.usersys.redhat.com/d/qvqIzkFGz/osp-dependency-pipeline?orgId=1&from=now-2d&to=now | 18:56 |
| weshay|ruck | rlandy|mtg, fyi ^ | 18:56 |
| *** rlandy|mtg is now known as rlandy | 18:56 | |
| rlandy | cool | 18:57 |
| chandankumar | rlandy: https://review.opendev.org/c/openstack/tripleo-docs/+/776921 +w please | 18:58 |
| rlandy | chandankumar: done - thanks | 18:58 |
| rlandy | chandankumar: you still want to talk about the container tools line? | 18:59 |
| chandankumar | rlandy: please have a look at container tools deps pipeline query | 18:59 |
| chandankumar | rlandy: yes sure | 18:59 |
| rlandy | chandankumar: k - give me 5 minutes | 18:59 |
| rlandy | just checking job | 18:59 |
| rlandy | chandankumar: ok | 19:02 |
| rlandy | chandankumar: meet.google.com/xch-owgy-cjv | 19:03 |
| zbr | chandankumar: not git-reviews is slow, is opendev gerrit which became very slow after the upgrade. | 19:09 |
| zbr | git-review is very thin, any delay is server side caused. | 19:10 |
| frenzy_friday | rlandy, weshay|ruck the integration line testprojects are commented out now. Wont run on accidental recheck - https://review.rdoproject.org/r/#/c/32041/, https://review.rdoproject.org/r/#/c/32080/ | 19:13 |
| rlandy | frenzy_friday: perfect - thanks | 19:14 |
| rlandy | weshay|ruck: check it out ... https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/stream/b063371ee05341558bb6dc945320f485?logfile=console.log | 19:17 |
| rlandy | fs010 running tempest | 19:17 |
| rlandy | passed the ssh issue | 19:17 |
| rlandy | hack fix | 19:17 |
| weshay|ruck | rlandy, ah nice | 19:24 |
| rlandy | two things - netstat on node | 19:24 |
| weshay|ruck | thanks frenzy_friday | 19:24 |
| rlandy | and readded iptables | 19:24 |
| rlandy | not sure which one helped | 19:25 |
| weshay|ruck | both rpm installs? | 19:25 |
| rlandy | guessing itables | 19:25 |
| chandankumar | rlandy: sshnaidm weshay|ruck podman pull quay.io/centos/centos:stream8 c8 stream container images | 19:26 |
| rlandy | cool | 19:26 |
| rlandy | will check where that is locked | 19:27 |
| rlandy | periodic-tripleo-ci-rhel-8-multinode-1ctlr-featureset010-rhos-16.2testprojectmastercheck200295,563 hrs 13 mins 14 secs2021-02-24 16:30:57SUCCESS | 19:44 |
| rlandy | weshay|ruck: happiness is ^^ | 19:45 |
| rlandy | weshay|ruck: frenzy_friday: https://review.rdoproject.org/r/#/c/32105/1/zuul.d/upstream.yaml | 19:50 |
| rlandy | revert of patch | 19:50 |
| *** jmasud has quit IRC | 19:51 | |
| frenzy_friday | rlandy, thanks | 19:52 |
| chandankumar | rlandy: weshay|ruck https://review.opendev.org/c/openstack/tripleo-ci/+/777378 merge this one, test project working | 19:53 |
| chandankumar | rlandy: weshay|ruck proof https://review.rdoproject.org/zuul/stream/690d768b9c804d25b7213e21e77fcf9f?logfile=console.log | 19:54 |
| rlandy | voted | 19:55 |
| *** dtantsur is now known as dtantsur|afk | 19:56 | |
| *** sshnaidm is now known as sshnaidm|afk | 20:28 | |
| rlandy | weshay|ruck: merging revert to restart master line | 20:37 |
| weshay|ruck | thanks | 20:37 |
| weshay|ruck | review please https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/776707/ | 20:51 |
| weshay|ruck | rlandy, safe to turn on promotions for master? | 20:53 |
| rlandy | weshay|ruck: ack | 20:53 |
| weshay|ruck | rlandy, thanks | 20:53 |
| rlandy | done | 20:54 |
| rlandy | weshay|ruck: wrt donwstream ovb - I thnk we may be overloading the cloud with so man ovb jobs starting at once | 20:58 |
| rlandy | just a guess | 20:58 |
| weshay|ruck | well.. master rdo is promotiong | 20:58 |
| rlandy | we almost never get them all to pass at once | 20:58 |
| weshay|ruck | rlandy, so perhaps we kill fs001 | 20:58 |
| weshay|ruck | in favor of only running 035 | 20:59 |
| rlandy | weshay|ruck: or 35 if we have bm | 20:59 |
| rlandy | yep | 20:59 |
| rlandy | weshay|ruck: so the multinode issue is that we are missing http://git.app.eng.bos.redhat.com/git/openstack/sf-config.git/tree/nodepool/elements/virt-customize/rhel8.4-cloud.yaml#n165 | 20:59 |
| rlandy | those iptables rules on the secondary node | 21:00 |
| rlandy | see #rhos-ops for discussion | 21:00 |
| rlandy | maybe never added | 21:00 |
| rlandy | or they are overwritten | 21:00 |
| rlandy | if we add them back, it passes | 21:00 |
| weshay|ruck | meh | 21:01 |
| weshay|ruck | not pleased.. but will leave it alone :) | 21:01 |
| weshay|ruck | rlandy, if we've given up trying to sync how images are built internally | 21:02 |
| weshay|ruck | at least create a check list of items to ensure are updated before revving the releasse | 21:02 |
| rlandy | weshay|ruck: no - I had that discussion today as well | 21:02 |
| * rlandy copies | 21:02 | |
| rlandy | <nhicher> rlandy: If I remember correctly, it was because centos-8 was not release when we started to build rhel-8 image | 21:03 |
| weshay|ruck | rlandy, ya.. totally | 21:03 |
| rlandy | nhicher> rlandy: I think we've got issue in the past with dib and rhel, but feel free to use dib if needed, it's not required to use virt-image | 21:03 |
| rlandy | weshay|ruck: ^^ so we can change the process | 21:03 |
| weshay|ruck | k.. | 21:03 |
| weshay|ruck | rlandy, let's ask to use the process for rhel9 | 21:03 |
| weshay|ruck | but use dib for stable rhel8 | 21:03 |
| rlandy | weshay|ruck: well usually we build the images | 21:04 |
| rlandy | so we can change the process | 21:04 |
| rlandy | (the royal we) | 21:04 |
| rlandy | builds the images | 21:04 |
| rlandy | idk why this does not work on rhel 8.4 | 21:04 |
| rlandy | was the same as 8.3 | 21:04 |
| rlandy | {# Ensure necessary iptables rules are present #} | 21:05 |
| rlandy | --run-command '/sbin/iptables -I INPUT -p tcp -m state --state NEW -m tcp --dport 19885 -j ACCEPT' | 21:05 |
| rlandy | --run-command '/sbin/iptables -I INPUT -p tcp -m tcp --dport 22 -j ACCEPT' | 21:05 |
| rlandy | --run-command '/sbin/iptables-save > /etc/sysconfig/iptables' | 21:05 |
| rlandy | {# Enable iptables to ensure it is started on boot #} | 21:05 |
| rlandy | --run-command 'systemctl enable iptables.service' | 21:05 |
| rlandy | didn't work | 21:05 |
| weshay|ruck | is there a log? | 21:05 |
| rlandy | at least we know why now | 21:05 |
| rlandy | yep | 21:05 |
| rlandy | image build log? | 21:05 |
| weshay|ruck | aye | 21:05 |
| rlandy | if you ask infra, they will point you to it | 21:06 |
| rlandy | getting on #rhos=ops | 21:07 |
| weshay|ruck | rlandy, ok... so my understanding is the following.. | 21:08 |
| weshay|ruck | 1. we're patching to resolve 8.4 now | 21:08 |
| weshay|ruck | 2. rhel8 builds will start using dib.. and the logs will be available ( hopefully ) | 21:08 |
| weshay|ruck | 3. rhel9 will continue to probably use virt-customize | 21:09 |
| weshay|ruck | until it's stable | 21:09 |
| rlandy | weshay|ruck: ... | 21:10 |
| rlandy | 1. ack - patching | 21:10 |
| rlandy | 2. idk - if there are more rhel8 builds, yes | 21:10 |
| rlandy | 3. ack | 21:10 |
| rlandy | actually | 21:10 |
| rlandy | rhel9 shoudl use dib if possible | 21:11 |
| rlandy | with stream - that may be possible | 21:11 |
| rlandy | weshay|ruck: ^^ | 21:11 |
| weshay|ruck | rlandy, dib probabaly won't be ready | 21:12 |
| weshay|ruck | that's why we use virt-customize for rhel8... | 21:12 |
| weshay|ruck | but we never stopped using it | 21:12 |
| rlandy | correct | 21:12 |
| rlandy | we just sort of carried it on | 21:12 |
| rlandy | because it was easier | 21:12 |
| rlandy | dib required upstream changes | 21:13 |
| rlandy | but for rhel9, we can do it differently | 21:13 |
| *** slaweq has quit IRC | 21:48 | |
| rlandy | ysandeep|away: hi - found the issues for fs010 on rhel 8.4. You wee right - iptables - bit issue with firewalld. testing a change to the node | 23:03 |
| rlandy | if not, I have a ci fix | 23:03 |
| rlandy | back later | 23:03 |
| *** rlandy is now known as rlandy|bbl | 23:13 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!