| *** ysandeep|PTO is now known as ysandeep | 02:00 | |
| *** rlandy|bbl is now known as rlandy | 02:16 | |
| *** ykarel_ has joined #oooq | 03:49 | |
| *** skramaja has joined #oooq | 03:50 | |
| *** rfolco has quit IRC | 03:53 | |
| *** ykarel_ is now known as ykarel | 04:11 | |
| *** jfrancoa has joined #oooq | 04:20 | |
| *** ysandeep is now known as ysandeep|ruck | 04:32 | |
| *** udesale has joined #oooq | 04:33 | |
| *** ratailor has joined #oooq | 05:14 | |
| *** saneax has joined #oooq | 05:25 | |
| *** bogdando has joined #oooq | 06:06 | |
| *** marios has joined #oooq | 06:20 | |
| *** dmellado has quit IRC | 06:27 | |
| *** bogdando has quit IRC | 06:33 | |
| akahat | ysandeep|ruck, o/ | 06:44 |
|---|---|---|
| akahat | ysandeep|ruck, do you have any updates on fs39 train job? | 06:45 |
| ysandeep|ruck | akahat, hello o/ | 06:46 |
| ysandeep|ruck | akahat, ovb fs39 master job ran successfull today - https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-master | 06:47 |
| ysandeep|ruck | akahat, can you recheck and see the current status | 06:47 |
| akahat | ysandeep|ruck, i didn't ran it recently.. let me recheck. | 06:47 |
| ysandeep|ruck | akahat, ack | 06:47 |
| *** jpena|off is now known as jpena | 07:11 | |
| *** amoralej|off is now known as amoralej | 07:31 | |
| *** tosky has joined #oooq | 07:37 | |
| chandankumar | marios: yes please+w | 07:44 |
| marios | nope | 07:45 |
| marios | i | 07:45 |
| marios | r | 07:45 |
| marios | e | 07:45 |
| marios | f | 07:45 |
| marios | u | 07:45 |
| marios | s | 07:45 |
| marios | e | 07:45 |
| chandankumar | marios: thanks :-) | 07:46 |
| * chandankumar offers milk tea to marios :-) | 07:46 | |
| marios | thanks | 07:46 |
| * marios sips massala chai | 07:47 | |
| *** ysandeep|ruck is now known as ysandeep|lunch | 07:49 | |
| *** bogdando has joined #oooq | 07:53 | |
| *** dmellado has joined #oooq | 08:20 | |
| *** jbadiapa has joined #oooq | 08:23 | |
| *** ysandeep|lunch is now known as ysandeep|ruck | 08:31 | |
| *** pojadhav is now known as pojadhav|lunch | 08:42 | |
| chandankumar | bhagyashris: pojadhav|lunch ysandeep|ruck akahat https://review.opendev.org/#/c/736752/14 when free, thanks! | 08:44 |
| ysandeep|ruck | chandankumar, ack | 08:48 |
| *** derekh has joined #oooq | 08:51 | |
| *** dtantsur|afk is now known as dtantsur | 08:53 | |
| *** skramaja has quit IRC | 09:03 | |
| *** skramaja has joined #oooq | 09:04 | |
| *** ratailor_ has joined #oooq | 09:06 | |
| *** ratailor has quit IRC | 09:08 | |
| *** pojadhav|lunch is now known as pojadhav | 09:10 | |
| pojadhav | chandankumar, ack | 09:11 |
| arxcruz | chandankumar: kopecmartin https://review.opendev.org/#/c/741625/ | 09:21 |
| chandankumar | arxcruz: will take a look | 09:22 |
| marios | panda: did you see the invite for the call this afternoon? | 09:25 |
| marios | panda: i just noticed you didn't ack/nack yet | 09:25 |
| *** zbr is now known as zbr|ruck | 09:26 | |
| zbr|ruck | if anyone needs my help let me know | 09:26 |
| zbr|ruck | i am happy that I no longer see the confusing image related errors from yesterday on the promoter | 09:27 |
| marios | zbr|ruck: my fence needs painting please | 09:27 |
| marios | thanks! | 09:27 |
| zbr|ruck | marios: i do a poor job with the brush, but no problem with power tool. warning, I do not clean after myself. | 09:28 |
| chandankumar | zbr|ruck: please review my j2 removal promoter patch | 09:31 |
| marios | zbr|ruck: ;) | 09:32 |
| zbr|ruck | chandankumar: if only you could post a link, i was not able to find it in ci-config | 09:37 |
| chandankumar | zbr|ruck: https://review.rdoproject.org/r/28624 | 09:37 |
| chandankumar | it is on rlandy name | 09:37 |
| chandankumar | i have taken over it | 09:37 |
| zbr|ruck | marios, abandon https://review.rdoproject.org/r/#/c/23212/ | 09:38 |
| zbr|ruck | we really need to fix permissions on rdo gerrit to allow us to abandon others reviews, helps w/ cleanups | 09:38 |
| zbr|ruck | on upstream we can | 09:38 |
| marios | zbr|ruck: ack very old one :D | 09:39 |
| zbr|ruck | marios: chandankumar : approve https://review.rdoproject.org/r/#/c/28688/ | 09:40 |
| akahat | chandankumar, ysandeep|ruck https://logserver.rdoproject.org/23/27523/6/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-train/347dda6/logs/undercloud/home/zuul/undercloud_install.log.txt.gz | 09:40 |
| marios | zbr|ruck: added to next reviews queue | 09:41 |
| zbr|ruck | that is permissions fix, not need to "study" it | 09:41 |
| zbr|ruck | so i can abandon your reviews! ;) | 09:42 |
| chandankumar | zbr|ruck: commented | 09:42 |
| ysandeep|ruck | akahat, that's because periodic-tripleo-centos-8-train-containers-build-push failed today with node failure | 09:42 |
| marios | zbr|ruck: ack | 09:42 |
| chandankumar | jpena: please have a look at this https://review.rdoproject.org/r/#/c/28688/ | 09:42 |
| akahat | ysandeep|ruck, ack. | 09:43 |
| marios | panda: o/ hello please confirm when you get a chance, if the time doesn't work we can change it | 09:43 |
| jpena | chandankumar: I think that's not the right way to change that, let me find it and comment in the review | 09:44 |
| chandankumar | zbr|ruck: ^6 jpena thanks :-) | 09:44 |
| ysandeep|ruck | akahat, can you try something like this - http://paste.openstack.org/show/796279/ | 09:48 |
| zbr|ruck | jpena: thanks for the update, i have no clue about how to navigate that "encrypted" file | 09:52 |
| zbr|ruck | filenames + keys with guids,... | 09:52 |
| zbr|ruck | but i am happy to help on fixing the issue, i only need bit of guidance | 09:54 |
| zbr|ruck | the idea is that cores should be able to abandon reviews, in general | 09:54 |
| zbr|ruck | afaik on upstream gerrit this is implicit and makes sense, as cores should be allowed to cleanup the review queue | 09:55 |
| jpena | zbr|ruck: let me create a review to extract the config from that messy file, we can take over from there | 09:55 |
| zbr|ruck | thanks! | 09:56 |
| * zbr|ruck preparing to abandon all panda's changes, muhaha, muhaha... | 09:57 | |
| chandankumar | akahat: arxcruz kopecmartin svyas agenda for today's tempest meeting https://hackmd.io/fIOKlEBHQfeTZjZmrUaEYQ?view#2020-07-24 please add, update or remove | 09:59 |
| zbr|ruck | chandankumar: i am afraid you still need to work on it to make it green | 10:01 |
| chandankumar | zbr|ruck: comment, please comment | 10:01 |
| chandankumar | staging promoter code need some fixes | 10:01 |
| jpena | zbr|ruck: https://review.rdoproject.org/r/28689, let's refine it | 10:03 |
| *** Tengu has quit IRC | 10:11 | |
| *** Tengu has joined #oooq | 10:13 | |
| zbr|ruck | jpena: i did few edits. i think i started to realise what we need to do. | 10:21 |
| zbr|ruck | ideally we should endup cleaning the missing resources files | 10:22 |
| zbr|ruck | and i also discoverd how dms appeared all over on reviews | 10:23 |
| chandankumar | akahat: ping | 10:32 |
| chandankumar | akahat: any comment on this https://review.opendev.org/#/c/741625/ before merging | 10:33 |
| chandankumar | ? | 10:33 |
| * akahat looking | 10:33 | |
| akahat | chandankumar, no comments. looks good. | 10:35 |
| chandankumar | akahat: thanks! | 10:42 |
| panda | marios:sorry, it's a ack | 10:44 |
| *** holser has joined #oooq | 10:57 | |
| ysandeep|ruck | marios, quick query on naming convention of upgrades job - tripleo-ci-centos-7-standalone-upgrade-stein ---> means rocky to stein upgrade , right? | 11:01 |
| marios | ysandeep|ruck: yes | 11:09 |
| marios | ysandeep|ruck: always upgrade to | 11:09 |
| marios | panda: praise be | 11:10 |
| marios | panda: thanks | 11:10 |
| *** udesale_ has joined #oooq | 11:11 | |
| ysandeep|ruck | marios, thanks, just wondering why we have R-> S upgrade job in tht train branch tripleo-ci-centos-7-standalone-upgrade-stein https://github.com/openstack/tripleo-heat-templates/blob/stable/train/zuul.d/layout.yaml#L22 , Not sure how useful it will be here. | 11:13 |
| *** udesale has quit IRC | 11:13 | |
| marios | ysandeep|ruck: well it wouldn't run on train ... looks like maybe left over from when the train branch was made | 11:14 |
| marios | ysandeep|ruck: i wouldn't run cos of https://github.com/openstack/tripleo-ci/blob/4705cc4002812d8de8681efedf731a71ee303d70/zuul.d/standalone-jobs.yaml#L98 | 11:14 |
| ysandeep|ruck | marios, yes, seems like left over to me as well, thanks for confirming | 11:15 |
| ysandeep|ruck | bhagyashris, pojadhav feel free to remove tripleo-ci-centos-7-standalone-upgrade-stein from train branches layouts.. its seems leftover | 11:17 |
| chandankumar | akahat: regarding os_tempest ironic issue https://review.opendev.org/#/c/727067/ is testing here | 11:17 |
| chandankumar | with all the patches | 11:17 |
| *** holser_ has joined #oooq | 11:17 | |
| chandankumar | akahat: let me use the ironic tempest plugin patch also | 11:17 |
| pojadhav | ysandeep|ruck, ack | 11:17 |
| akahat | chandankumar, oh.. you put everything there. :P | 11:19 |
| *** holser_ has quit IRC | 11:19 | |
| *** holser_ has joined #oooq | 11:20 | |
| *** holser has quit IRC | 11:20 | |
| chandankumar | akahat: I was talking about this error https://zuul.opendev.org/t/openstack/build/adfad8d2d13540adad306ce6c20c9d61/log/job-output.txt#13864 | 11:20 |
| *** holser_ has quit IRC | 11:21 | |
| akahat | chandankumar, in recent ran i didn't seen this issue: https://review.opendev.org/#/c/720705/ | 11:22 |
| marios | ack ysandeep|ruck | 11:22 |
| *** holser has joined #oooq | 11:22 | |
| chandankumar | akahat: not there | 11:22 |
| chandankumar | akahat: https://review.opendev.org/#/c/727067/ it is getting tested | 11:23 |
| chandankumar | akahat: os_tempest ironic changes are not getting executed there | 11:23 |
| akahat | chandankumar, okay. let's wait for it. | 11:23 |
| chandankumar | akahat: one more thing, https://review.opendev.org/#/c/723876 which patch is not working | 11:26 |
| *** holser has quit IRC | 11:27 | |
| *** holser has joined #oooq | 11:28 | |
| *** holser has quit IRC | 11:30 | |
| *** jpena is now known as jpena|lunch | 11:32 | |
| *** holser has joined #oooq | 11:32 | |
| *** pojadhav is now known as pojadhav|brb | 11:41 | |
| chandankumar | ysandeep|ruck: pojadhav|brb bhagyashris added comments for missing jobs here https://hackmd.io/oFX0m-T8SHWqqwzoHmC4mg?view#Missing-centos-8-train-jobs let us know if any queries | 11:43 |
| ysandeep|ruck | chandankumar, ack, thank you! | 11:46 |
| *** holser_ has joined #oooq | 11:49 | |
| *** holser has quit IRC | 11:50 | |
| weshay|ruck | marios, re: packages.. https://bugs.launchpad.net/tripleo/+bug/1888821 | 11:54 |
| openstack | Launchpad bug 1888821 in tripleo "Some tasks are still using yum instead of dnf making the command always fail." [Critical,Triaged] - Assigned to Sofer Athlan-Guyot (sofer-athlan-guyot) | 11:54 |
| marios | weshay|ruck: o/ packages? looking | 11:54 |
| weshay|ruck | marios, it looks like ur job installs the dlrn built packages after upgrade.. in ussuri https://4c9bea7f28088149dfcc-1634c0891e365f9cbd205121e56f3649.ssl.cf1.rackcdn.com/739457/11/check/tripleo-ci-centos-8-standalone-upgrade-ussuri/36584f0/job-output.txt | 11:55 |
| marios | weshay|ruck: we had a similar one https://bugs.launchpad.net/tripleo/+bug/1886837 | 11:55 |
| openstack | Launchpad bug 1886837 in tripleo "centos8 master undercloud upgrade failing - yum instead of dnf" [High,Fix released] - Assigned to Marios Andreou (marios-b) | 11:55 |
| marios | weshay|ruck: didn't come across the one Sofer filed though | 11:55 |
| marios | weshay|ruck: well, i haven't been paying too much attention to master | 11:55 |
| marios | weshay|ruck: maybe that's why | 11:55 |
| marios | weshay|ruck: oh you mean https://bugs.launchpad.net/tripleo/+bug/1887159/comments/3 | 11:56 |
| openstack | Launchpad bug 1887159 in tripleo "centos8 standalone-upgrade ussuri fails Error: cluster is not configured " [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 11:56 |
| * weshay|ruck just guessing but wanted to make sure you saw it | 11:56 | |
| marios | weshay|ruck: ack noting | 11:56 |
| *** holser has joined #oooq | 11:58 | |
| weshay|ruck | arxcruz, you still working on 2020-07-24 07:30:34.195 | TASK [install_skiplist : Install tempest-skiplist] ***************************** | 11:58 |
| weshay|ruck | https://ci.centos.org/artifacts/rdo/jenkins-tripleo-quickstart-promote-master-current-tripleo-delorean-minimal-780/console.log | 11:58 |
| *** holser has quit IRC | 11:58 | |
| *** holser has joined #oooq | 11:59 | |
| *** holser_ has quit IRC | 11:59 | |
| arxcruz | weshay|ruck: yes | 11:59 |
| weshay|ruck | arxcruz, k... and just to close the loop.. you know the zuul repo is n/a in this case | 12:00 |
| weshay|ruck | right? | 12:00 |
| weshay|ruck | just a standard quickstart.sh invocation | 12:00 |
| bhagyashris | chandankumar, ack thank you :) | 12:00 |
| arxcruz | weshay|ruck: yes | 12:00 |
| weshay|ruck | k | 12:00 |
| arxcruz | I'm thinking download from git if zuul is not defined | 12:01 |
| arxcruz | and install it | 12:01 |
| *** holser has quit IRC | 12:03 | |
| *** holser has joined #oooq | 12:03 | |
| *** rfolco has joined #oooq | 12:03 | |
| *** holser has quit IRC | 12:08 | |
| *** holser has joined #oooq | 12:08 | |
| weshay|ruck | arxcruz, k.. you have a box for the recreate ya? | 12:09 |
| arxcruz | weshay|ruck: yup | 12:09 |
| arxcruz | weshay|ruck: but i don't think it will be required | 12:10 |
| chandankumar | rfolco: thanks for looking into my patch, I need to fix few stuff, may be a new version come on monday | 12:14 |
| rfolco | chandankumar, np | 12:14 |
| *** ysandeep|ruck is now known as ysandeep|afk | 12:15 | |
| rlandy | chandankumar: hey ... https://review.rdoproject.org/r/#/c/28489/9/zuul.d/component-jobs.yaml | 12:19 |
| chandankumar | rlandy: Hello | 12:19 |
| rlandy | can you explain more about what we need to do with the md5? | 12:19 |
| rlandy | chandankumar: that job doesn;t report to dlrn ... so wanted to ask you about tracking versions | 12:20 |
| *** pojadhav|brb is now known as pojadhav | 12:20 | |
| rlandy | how can we know what repo we tested? | 12:20 |
| rlandy | chandankumar: test result ... https://review.rdoproject.org/r/#/c/25325/ | 12:20 |
| chandankumar | rlandy: sorry | 12:21 |
| chandankumar | rlandy: i havenot seen the test patch https://logserver.rdoproject.org/25/25325/35/check/periodic-tripleo-ci-centos-8-standalone-baseos-centos8cr-master/f0bf485/logs/undercloud/etc/yum.repos.d/centos8cr-component.repo.txt.gz | 12:22 |
| rlandy | np | 12:22 |
| chandankumar | rlandy: we donot need md5 now | 12:22 |
| rlandy | chandankumar: this is just POC | 12:22 |
| rlandy | chandankumar: I think you are right though | 12:22 |
| chandankumar | rlandy: but we also wanted to put to track it as a part of promotion criteria | 12:22 |
| chandankumar | so that none misses it | 12:22 |
| chandankumar | if it fails lets; block the pipeline | 12:22 |
| rlandy | chandankumar: ok- so that will work if we have a version to block on | 12:23 |
| rlandy | how do I tell what version of that CR repo I am working with? | 12:23 |
| rlandy | they don't keep an md5 | 12:24 |
| rlandy | we can take one | 12:24 |
| *** derekh has quit IRC | 12:24 | |
| chandankumar | rlandy: regarding version hard to find out | 12:25 |
| chandankumar | as they get populate few weeks before a new release comes up | 12:26 |
| weshay|ruck | ya.. it's not versioned | 12:26 |
| weshay|ruck | rlandy, chandankumar what would we block when it fails? | 12:27 |
| chandankumar | weshay|ruck: in ideal case, new component pipeline will be promoting always | 12:27 |
| weshay|ruck | may the force be with you, always | 12:27 |
| chandankumar | sorry component comes from dlrn | 12:28 |
| chandankumar | sorry component repo provided by dlrn | 12:28 |
| weshay|ruck | right. which is not the case here.. w/ cr | 12:28 |
| rlandy | we have no way then to determine block/unblock | 12:28 |
| rlandy | based on what repo we actually tested | 12:29 |
| chandankumar | so here, we can need add few jobs in each component which will be part of promotion | 12:29 |
| rlandy | unless we played dlrn ourselves ... | 12:29 |
| weshay|ruck | imho... open to suggestions.. | 12:29 |
| chandankumar | if fails then block the promotion | 12:29 |
| weshay|ruck | standalone | 12:29 |
| rlandy | we took the md5 of th repo we tested | 12:29 |
| weshay|ruck | standalone w/ ceph, so 001,004 | 12:29 |
| weshay|ruck | ovb | 12:29 |
| rlandy | and ublocked cr if it was the same md5 | 12:29 |
| rlandy | weshay|ruck: chandankumar: chat for a few? or ideas on hackmd? | 12:30 |
| chandankumar | rlandy: going in tempest meeting | 12:30 |
| weshay|ruck | chandankumar, if these jobs fail.. I'd think about auto creating a bug.. w/ cix | 12:30 |
| chandankumar | may be chat after an hour? | 12:30 |
| rlandy | chandankumar: ack | 12:30 |
| rlandy | then I can finish this for you all | 12:30 |
| rlandy | marios: looking into your comment on https://review.opendev.org/#/c/742677/ - maybe it's only multinode where the logs exploded. | 12:33 |
| *** ratailor_ has quit IRC | 12:34 | |
| marios | rlandy: ack i wasn't sure if i was looking at the right place/if the problem was general/specific etc | 12:35 |
| *** jpena|lunch is now known as jpena | 12:39 | |
| *** ysandeep|afk is now known as ysandeep|ruck | 12:40 | |
| *** amoralej is now known as amoralej|lunch | 12:44 | |
| *** derekh has joined #oooq | 12:51 | |
| *** ysandeep|ruck is now known as ysandeep|away | 12:57 | |
| *** ysandeep|away is now known as ysandeep | 12:58 | |
| marios | rlandy: wana join? | 13:03 |
| arxcruz | weshay|ruck: yup, the command is there | 13:05 |
| weshay|ruck | :) | 13:05 |
| arxcruz | assuming .quickstart/bin/activate is being called and it's on path | 13:06 |
| weshay|ruck | ya | 13:07 |
| weshay|ruck | arxcruz, see... quickstart.sh and install-deps.sh | 13:07 |
| weshay|ruck | in tq | 13:07 |
| weshay|ruck | zbr|ruck, dear lord sotp | 13:08 |
| weshay|ruck | stop | 13:08 |
| weshay|ruck | lolz | 13:08 |
| weshay|ruck | zoom out | 13:08 |
| *** skramaja has quit IRC | 13:08 | |
| *** amoralej|lunch is now known as amoralej | 13:28 | |
| arxcruz | weshay|ruck: 1-1? | 13:32 |
| weshay|ruck | arxcruz, aye.. coming | 13:32 |
| *** TrevorV has joined #oooq | 13:34 | |
| arxcruz | https://review.rdoproject.org/zuul/api/builds | 13:39 |
| chandankumar | rlandy: weshay|ruck want to talk now? | 13:44 |
| rfolco | marios, added link to test hackmd at the top of the page | 13:44 |
| marios | rfolco: ack | 13:44 |
| *** ysandeep is now known as ysandeep|away | 13:44 | |
| rlandy | chandankumar: ack | 13:45 |
| chandankumar | rlandy: https://meet.google.com/qei-hciz-eyx?pli=1&authuser=0 | 13:46 |
| marios | akahat: bhagyashris: chandankumar: panda: rfolco: rlandy: ysandeep|away: zbr|ruck: weshay|ruck: as just discussed I sent an invite for wednesday 29 promoter sync fyi check inboxes | 13:47 |
| pojadhav | marios, add me also :) | 13:49 |
| marios | recheck awoo https://review.opendev.org/#/c/742182/3 here we go it's been a while its the recheck roullette - never know what'cha gonna get! | 13:49 |
| marios | pojadhav: ack added check inbox | 13:50 |
| pojadhav | thanks | 13:51 |
| marios | welcome | 13:51 |
| chandankumar | weshay|ruck: https://review.rdoproject.org/r/#/c/28650/ please +w ot | 14:07 |
| chandankumar | it | 14:07 |
| chandankumar | rlandy: I am testing new container build here https://review.rdoproject.org/r/28687 and namespace does not get populated | 14:07 |
| chandankumar | once it's up, we will have a standalone run then we will move it to prod | 14:08 |
| *** dpawlik2 has quit IRC | 14:11 | |
| chandankumar | marios: weshay|ruck rlandy https://review.opendev.org/#/c/742662/ please have a look when free | 14:13 |
| weshay|ruck | rlandy, chandankumar fyi https://review.rdoproject.org/r/#/c/28511/ | 14:15 |
| rlandy | chandankumar: ok - thanks - I | 14:16 |
| rlandy | 'll use it to test | 14:16 |
| chandankumar | rlandy: containers are build successully | 14:17 |
| chandankumar | rlandy: https://logserver.rdoproject.org/87/28687/1/check/periodic-tripleo-ci-build-containers-ubi-8-push-ussuri/d6693bb/logs/containers-successfully-built.log and https://logserver.rdoproject.org/87/28687/1/check/periodic-tripleo-ci-build-containers-ubi-8-push-train/cb08655/logs/containers-successfully-built.log | 14:17 |
| *** bogdando has quit IRC | 14:19 | |
| rlandy | weshay|ruck: re: https://review.rdoproject.org/r/#/c/28511/ - when we see it in dlrn, I'll add the line for it | 14:19 |
| *** Goneri has joined #oooq | 14:26 | |
| rlandy | chandankumar: should I abandon https://review.rdoproject.org/r/#/c/28673/? | 14:39 |
| chandankumar | rlandy: nope | 14:39 |
| chandankumar | rlandy: we will merge it on monday | 14:39 |
| rlandy | ok | 14:39 |
| arxcruz | chandankumar: need your help here https://6e4d1e6e297929d2c578-1c888752712004e424d036d0f1b76493.ssl.cf1.rackcdn.com/740662/5/check/tripleo-ci-centos-7-containers-multinode-queens/5c4a50b/job-output.txt | 14:48 |
| arxcruz | is it possible we have openstacksdk >= 12 on queens ? | 14:49 |
| chandankumar | arxcruz: need to check the version in rdo | 14:50 |
| chandankumar | and upstream requirements | 14:51 |
| chandankumar | arxcruz: https://github.com/redhat-openstack/rdoinfo/blob/master/tags/queens.yml#L453 | 14:51 |
| chandankumar | arxcruz: https://opendev.org/openstack/requirements/src/branch/stable/queens/upper-constraints.txt#L414 | 14:52 |
| arxcruz | chandankumar: this is what we need to make it work, the stein is passing with os_tempest | 14:52 |
| chandankumar | arxcruz: answer is no | 14:52 |
| chandankumar | arxcruz: we need to bump in upstream requirments | 14:52 |
| arxcruz | chandankumar: maybe we check os_tempest side... | 14:53 |
| arxcruz | chandankumar: is that possible ? | 14:53 |
| chandankumar | then it will get worked | 14:53 |
| chandankumar | arxcruz: https://opendev.org/openstack/openstack-ansible-os_tempest/src/branch/master/tasks/tempest_resources.yml#L81 | 14:54 |
| arxcruz | chandankumar: https://3a40dd74807b16b826bd-be2647135e35587b155fc0ff22bf704c.ssl.cf5.rackcdn.com/740662/5/check/tripleo-ci-centos-7-containers-multinode-stein/1896543/logs/undercloud/var/log/tempest/stestr_results.html | 14:54 |
| chandankumar | these are not conditonal | 14:54 |
| arxcruz | stein passes with os_tempest | 14:54 |
| chandankumar | arxcruz: source-branch: 0.27.0 | 14:55 |
| *** udesale_ has quit IRC | 14:55 | |
| chandankumar | becuase of higher openstacksdk version | 14:55 |
| arxcruz | chandankumar: yup, i'm aware | 14:55 |
| arxcruz | so, i'm checking the ansible module os_ | 14:55 |
| chandankumar | arxcruz: may be we ask release team to update bump openstasksk version | 14:55 |
| arxcruz | chandankumar: how we do that? mail, submit review? | 14:56 |
| arxcruz | never did that before :) | 14:56 |
| chandankumar | arxcruz: yes | 14:56 |
| chandankumar | arxcruz: submit review | 14:56 |
| rlandy | marios: so I looked into the overcloud-deploy logging issue | 14:56 |
| rlandy | the explosion of the logs is only on multinode | 14:57 |
| rlandy | but all releases | 14:57 |
| rlandy | so I will change that patch to be only multinode featuresets | 14:57 |
| chandankumar | arxcruz: propose the patch add sagi and odyssey4me they can help on pitching in | 14:58 |
| *** derekh has quit IRC | 15:03 | |
| arxcruz | chandankumar: sshnaidm|off odyssey4me: https://review.opendev.org/#/c/742906/ | 15:04 |
| chandankumar | arxcruz: you need to fix the commit msg | 15:06 |
| chandankumar | arxcruz: on train jobs. -> queens job | 15:07 |
| weshay|ruck | arxcruz, isn't that a rocky tag? | 15:08 |
| rlandy | zbr|ruck: did we stop gzipping our logs? | 15:11 |
| rlandy | or is that only an rdocloud thing? | 15:11 |
| chandankumar | weshay|ruck: do we still care about rocky? | 15:11 |
| zbr|ruck | rlandy: no idea, i did not touch anything around. | 15:11 |
| zbr|ruck | as you probably know, I am against gzipping | 15:12 |
| rlandy | zbr|ruck: I didn't know - but I don't want to turn off verbose logs if the issue is gzip | 15:12 |
| *** holser_ has joined #oooq | 15:14 | |
| *** holser has quit IRC | 15:15 | |
| rlandy | zbr|ruck: why are you against gzip? | 15:18 |
| * rlandy just asking | 15:18 | |
| weshay|ruck | chandankumar, for now yes | 15:18 |
| *** holser_ has quit IRC | 15:19 | |
| *** holser has joined #oooq | 15:19 | |
| zbr|ruck | because they prevent browsing the files. all modern storage backends can take care of compressing files at rest and for transport in a transparent way. | 15:19 |
| weshay|ruck | zbr|ruck, I would like our logs to work the way they used to | 15:19 |
| zbr|ruck | only exception is when we have a huge number of files, ones that nobody expects to browse. | 15:19 |
| weshay|ruck | our logs need to 100% browsable | 15:20 |
| zbr|ruck | weshay|ruck: as I said, i did not make any change myself. | 15:20 |
| *** ykarel is now known as ykarel|away | 15:20 | |
| zbr|ruck | probably you remember my round with infra around .sh files, still not sorted. | 15:20 |
| weshay|ruck | k | 15:20 |
| weshay|ruck | right.. I was going to say.. the work was never completed | 15:21 |
| -openstackstatus- NOTICE: We are renaming projects in Gerrit and review.opendev.org will experience a short outage. Thank you for your patience. | 15:21 | |
| zbr|ruck | it takes only one smart... to say he prefers download instead of browse to block anything | 15:21 |
| weshay|ruck | aye | 15:21 |
| weshay|ruck | and they may never approve it | 15:21 |
| weshay|ruck | at what point do we take ownership of it.. and do what we can to return to 100% browsable | 15:21 |
| marios | rlandy: reading back | 15:22 |
| zbr|ruck | it will eventually happen, i started to go deepen in zuul changes, but i need time, time to learn more Reac for example | 15:22 |
| marios | rlandy: ack ok i will revisit on next reviews then noting | 15:22 |
| rlandy | marios: actually, after more investigation, we zip the ovb logs | 15:23 |
| rlandy | we don't zip upstream | 15:23 |
| rlandy | still following this one up | 15:23 |
| zbr|ruck | where are the files that are not browsable but there were few days ago? gime me bit more context, maybe we discover the issue | 15:23 |
| marios | rlandy: ack thanks for update np i would have revisited anyway on next reviews | 15:23 |
| arxcruz | weshay|ruck: yeah, now i have no idea how to manage that | 15:24 |
| arxcruz | :/ | 15:24 |
| weshay|ruck | arxcruz, we can bump the release in openstacksdk for queens.. but I need more context of what the issue is | 15:24 |
| arxcruz | weshay|ruck: os_tempest | 15:25 |
| weshay|ruck | aye | 15:25 |
| arxcruz | weshay|ruck: in order to move queens to use os_tempest | 15:25 |
| arxcruz | it requires openstacksdk >= 1.12 | 15:25 |
| arxcruz | ops, 0.12 | 15:25 |
| rlandy | zbr|ruck: so this all started because the multinode deploy logs were running ~6M | 15:25 |
| rlandy | I'm looking back to see if this was always the case | 15:25 |
| rlandy | in rdocloud, we don;t see this because we gzip the logs | 15:26 |
| rlandy | dropping verbose may be a bad idea | 15:26 |
| rlandy | if it decreases our ability to debug | 15:26 |
| akahat | quick question: where ironic deployment logs got stored ? | 15:26 |
| rlandy | which is why I w-1'ed the patch | 15:26 |
| weshay|ruck | arxcruz, sec.. checking something | 15:27 |
| akahat | /var/log/containers/ironic/deploy.. this directory is not exists. | 15:27 |
| akahat | so in tripleo it's path is different? | 15:28 |
| weshay|ruck | arxcruz, last tag was from march | 15:29 |
| weshay|ruck | https://opendev.org/openstack/releases/src/branch/master/deliverables/queens/openstacksdk.yaml#L29-L32 | 15:29 |
| weshay|ruck | arxcruz, so.. does queens since march have the feature you need | 15:29 |
| weshay|ruck | or just in rocky | 15:29 |
| arxcruz | just in 0.12 | 15:29 |
| weshay|ruck | arxcruz, forget the tag | 15:29 |
| weshay|ruck | because we can 11.5 | 15:30 |
| weshay|ruck | but that only helps if the feature you need is in queens at all | 15:30 |
| arxcruz | weshay|ruck: https://6e4d1e6e297929d2c578-1c888752712004e424d036d0f1b76493.ssl.cf1.rackcdn.com/740662/5/check/tripleo-ci-centos-7-containers-multinode-queens/5c4a50b/job-output.txt | 15:30 |
| weshay|ruck | arxcruz, if not.. I think it would be fine to use $any_version of openstacksdk | 15:30 |
| weshay|ruck | w/ tempest | 15:30 |
| weshay|ruck | tbh | 15:30 |
| weshay|ruck | or... wait | 15:30 |
| arxcruz | weshay|ruck: 2020-07-24 13:29:48.918368 | primary | To utilize this module, the installed version of the openstacksdk library MUST be >=0.12. | 15:30 |
| weshay|ruck | re: the setup for tempest | 15:30 |
| weshay|ruck | not tempest itself | 15:30 |
| weshay|ruck | arxcruz, which bits of os_tempest are failing | 15:31 |
| weshay|ruck | the setup or the test execution | 15:31 |
| arxcruz | weshay|ruck: setup | 15:31 |
| weshay|ruck | arxcruz, for all I care.. use master for the setup | 15:31 |
| weshay|ruck | doesn't matter | 15:31 |
| weshay|ruck | arxcruz, does that help? | 15:31 |
| arxcruz | weshay|ruck: i'll test | 15:31 |
| arxcruz | brb | 15:32 |
| weshay|ruck | k | 15:32 |
| *** holser_ has joined #oooq | 15:36 | |
| *** holser__ has joined #oooq | 15:39 | |
| *** holser has quit IRC | 15:39 | |
| *** holser_ has quit IRC | 15:40 | |
| *** marios has quit IRC | 15:46 | |
| *** dtantsur is now known as dtantsur|afk | 15:53 | |
| *** amoralej is now known as amoralej|off | 16:08 | |
| *** jbadiapa has quit IRC | 16:10 | |
| chandankumar | see ya on monday, Have a nice weekend ahead :-) | 16:18 |
| *** chandankumar is now known as raukadah | 16:18 | |
| *** holser__ has quit IRC | 16:19 | |
| *** holser has joined #oooq | 16:19 | |
| *** jpena is now known as jpena|off | 16:21 | |
| *** holser has quit IRC | 16:24 | |
| rlandy | weshay|ruck: have a minute? | 16:33 |
| rlandy | weshay|ruck: re: deploy log size | 16:34 |
| *** zbr|ruck is now known as zbr | 16:41 | |
| *** TrevorV has quit IRC | 16:49 | |
| *** ykarel|away has quit IRC | 16:58 | |
| rlandy | raukadah: https://code.engineering.redhat.com/gerrit/207276 Sync with RDO - container login with buildah role | 18:32 |
| rlandy | weshay|ruck: ping re: 16-2 naming | 18:52 |
| weshay|ruck | rlandy, hey | 19:30 |
| weshay|ruck | sorry.. that took longer than I thought | 19:30 |
| rlandy | weshay|ruck: np | 19:43 |
| weshay|ruck | rlandy, the deploy_log size, that Emilien was working? | 19:45 |
| rlandy | weshay|ruck: yeah - it seems like it's been this way for a log time | 19:45 |
| rlandy | also, I noticed we were zip'ping the logs in rdo but not upstrea, | 19:45 |
| weshay|ruck | re: the logs being tooo long? | 19:45 |
| rlandy | removing the debug output helped somewhat | 19:45 |
| rlandy | but I have concerns about removing verbise output | 19:46 |
| rlandy | verbose | 19:46 |
| rlandy | maybe it will hurt our debugging | 19:46 |
| weshay|ruck | rlandy, give me an example.. just so we're looking at the exact same thing | 19:46 |
| rlandy | deploy logs are edging upwards of 6M | 19:46 |
| *** jfrancoa has quit IRC | 19:47 | |
| weshay|ruck | rlandy, I'm a little paralized in my thinking about it tbh.. because it cuts both ways | 19:47 |
| weshay|ruck | I'm afraid if we reduce the logging too much, our debugging will suffer | 19:47 |
| weshay|ruck | generally speaking I always feel that way | 19:47 |
| rlandy | weshay|ruck: https://review.opendev.org/#/c/742517/ merged ... | 19:47 |
| EmilienM | we don't want --verbose | 19:47 |
| weshay|ruck | https://ee771230d40e7b2547ad-f6d9d58f0487401586e66f66be7f6106.ssl.cf1.rackcdn.com/742517/1/check/tripleo-ci-centos-8-containers-multinode/1f2335d/logs/quickstart_files/log-size.txt | 19:49 |
| rlandy | weshay|ruck: compare: https://review.opendev.org/#/c/742575/ https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_b0a/742575/1/check/tripleo-ci-centos-8-containers-multinode/b0a1071/logs/undercloud/home/zuul/ | 19:49 |
| rlandy | with | 19:50 |
| rlandy | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_c2d/742677/1/check/tripleo-ci-centos-8-containers-multinode/c2d33a7/logs/undercloud/home/zuul/ | 19:50 |
| weshay|ruck | k | 19:50 |
| rlandy | ansible.logThu Jul 23 19:20:27 2020131.4M | 19:50 |
| rlandy | overcloud_deploy.logThu Jul 23 19:20:30 2020105.4M | 19:50 |
| rlandy | ansible.logThu Jul 23 04:16:50 2020841.5K | 19:50 |
| weshay|ruck | ansible log is much smaller in the first | 19:50 |
| rlandy | overcloud_deploy.logThu Jul 23 04:16:50 2020843.6K | 19:50 |
| rlandy | maybe we shot ourselves in the foot??? | 19:51 |
| weshay|ruck | with regards to the logs being not verbose enough? | 19:52 |
| rlandy | marios also objected to the blanket change https://review.opendev.org/#/c/742677/ | 19:53 |
| rlandy | considering we zip OVB logs | 19:53 |
| rlandy | we could leave those verbose? | 19:53 |
| EmilienM | we should only enable verbose when things are broken | 19:53 |
| EmilienM | this storage isn't free | 19:54 |
| weshay|ruck | I would also start using /var/log/extra/errors.txt | 19:54 |
| weshay|ruck | more | 19:54 |
| rlandy | EmilienM: I got some push back from the team to https://review.opendev.org/#/c/742677/ | 19:54 |
| rlandy | which is why I am asking | 19:54 |
| EmilienM | meh | 19:54 |
| EmilienM | no more energy to deal with it today | 19:54 |
| EmilienM | go deal with 200M+ logs then :D | 19:54 |
| weshay|ruck | it's not saving space | 19:55 |
| * weshay|ruck goes back prior to doing so | 19:56 | |
| weshay|ruck | it saves time in the log loading in my browser for sure | 19:57 |
| EmilienM | it saves storage | 19:57 |
| EmilienM | storing 200M just for overcloud per job is useless | 19:57 |
| EmilienM | we should have this talk on #tripleo | 19:57 |
| EmilienM | alex is not here | 19:57 |
| weshay|ruck | something else is at play.. looking at a job from july3rd.. https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_407/736134/4/gate/tripleo-ci-centos-8-containers-multinode/407b050/logs/quickstart_files/log-size.txt | 19:58 |
| EmilienM | weshay|ruck: you remember when you said "pick your battles" | 20:01 |
| EmilienM | i'm putting that in action right now | 20:01 |
| *** EmilienM has left #oooq | 20:01 | |
| rlandy | consider standalone ... | 20:01 |
| weshay|ruck | lolz | 20:02 |
| weshay|ruck | Emilien .. such a drama boy | 20:02 |
| rlandy | standalone_deploy.logWed Jul 22 21:04:57 2020730.4K | 20:02 |
| rlandy | weshay|ruck: idk - we can do through it more ... but the logs have been large for a while | 20:04 |
| rlandy | don;t know why it attracted attention all of a sudden | 20:05 |
| weshay|ruck | rlandy, so you are concerned we can't get what we need from the logs eh? | 20:05 |
| weshay|ruck | because you haven't said that yet | 20:06 |
| * rlandy doesn't expect weshay|ruck to read between the lines ... | 20:07 | |
| rlandy | "<rlandy> maybe it will hurt our debugging" | 20:07 |
| weshay|ruck | ya.. I would let it run like this for a bit.. and see how it goes | 20:07 |
| weshay|ruck | experiment.. | 20:07 |
| rlandy | weshay|ruck: run like how? | 20:08 |
| weshay|ruck | the deploy log doesn't often point at the root cause of any failure | 20:08 |
| weshay|ruck | rlandy, leave the less verbose logging on for a bit | 20:08 |
| rlandy | weshay|ruck: so put the patch in for merge? | 20:08 |
| rlandy | debug removal is merged | 20:08 |
| rlandy | question is around verbose patch | 20:09 |
| weshay|ruck | https://review.opendev.org/#/c/742575/1/config/general_config/featureset010.yml | 20:09 |
| rlandy | nvm merged - /me will recheck | 20:09 |
| rlandy | https://review.opendev.org/#/c/742677/ | 20:10 |
| rlandy | weshay|ruck: ^^ ok - then we can decide on | 20:10 |
| rlandy | weshay|ruck: so second ping was about 16.2 renaming ... | 20:11 |
| rlandy | well naming | 20:11 |
| rlandy | https://code.engineering.redhat.com/gerrit/#/c/207282 | 20:11 |
| rlandy | got a couple patches in the works like ^^ | 20:12 |
| rlandy | the dlrn repo is 16-2 | 20:12 |
| rlandy | in code-eng the repos are 16.2 | 20:12 |
| rlandy | mostly I went with 16.2 | 20:12 |
| rlandy | pls comment on the review if any issues | 20:12 |
| weshay|ruck | ya.. I have it up .. couple side conversations atm.. might be slow | 20:24 |
| weshay|ruck | rlandy, I think the internal review looks fine | 20:36 |
| weshay|ruck | there may be an issue w/ 16.2 vs something like 16_2 etc | 20:36 |
| rlandy | adding more jobs | 20:36 |
| rlandy | weshay|ruck: yeah - that's exactly waht I was asking | 20:36 |
| weshay|ruck | but until we KNOW of a problem w/ a "." I'm fine w/ it | 20:36 |
| rlandy | we'll find out | 20:36 |
| weshay|ruck | aye | 20:36 |
| rlandy | but juts adding the jobs so we have something to start with | 20:37 |
| rlandy | weshay|ruck: we need a new namespace in the registry | 20:38 |
| rlandy | will discuss the 16.2 vs 16-2 in monday's meeting | 20:39 |
| rlandy | and we can have a final decision | 20:39 |
| rlandy | no big deal either way | 20:39 |
| weshay|ruck | k | 20:39 |
| weshay|ruck | is the registry link in the infra doc? | 20:39 |
| weshay|ruck | for 17? | 20:39 |
| rlandy | should be - checking | 20:40 |
| rlandy | Internal upshift container registry | 20:41 |
| rlandy | weshay|ruck: ^^ | 20:41 |
| rlandy | and new release files | 20:43 |
| weshay|ruck | thanks | 20:44 |
| weshay|ruck | rlandy, anything else I can help w/? | 20:52 |
| rlandy | weshay|ruck: I don't think so - thanks | 20:57 |
| *** rfolco has quit IRC | 21:07 | |
| *** rfolco has joined #oooq | 21:08 | |
| *** rfolco has quit IRC | 21:12 | |
| rlandy | weshay|ruck: and in the shocker of the week, OVB downsteam baremetal component is running well :) | 21:21 |
| weshay|ruck | rlandy, NICE | 21:24 |
| * weshay|ruck looks | 21:24 | |
| * weshay|ruck see's two passes in bm | 21:25 | |
| weshay|ruck | ironic | 21:26 |
| rlandy | that's as long as it's been in the component | 21:28 |
| rlandy | just merged that change | 21:28 |
| weshay|ruck | hrm | 21:30 |
| weshay|ruck | http://tripleo-cockpit.usersys.redhat.com/d/VTiDErfWk/jobs-exploration?orgId=1&var-influxdb_filter=type%7C%3D%7Cinternal&var-influxdb_filter=job_name%7C%3D~%7C%2Ffeatureset001%2F | 21:30 |
| weshay|ruck | signs of success | 21:30 |
| * weshay|ruck checks the other jobs.. maybe we have failing copoment | 21:30 | |
| * weshay|ruck thinks we need a promotion of rhos-17 still | 21:31 | |
| weshay|ruck | ya.. it's the components | 21:32 |
| weshay|ruck | rlandy, let's look monday.. it's getting late for you | 21:32 |
| rlandy | weshay|ruck: we are still waiting for a change from brew ... | 21:32 |
| rlandy | getti g | 21:32 |
| weshay|ruck | ya.. I recall.. so that's going to suck if we have to do that a lot | 21:33 |
| rlandy | weshay|ruck: https://trello.com/c/IYkmzCCf/1610-update-python-importlib-resources-and-python-importlib-metadata-for-osp-170 | 21:33 |
| rlandy | it's in backlog | 21:33 |
| rlandy | until that - we won't get a promotion | 21:33 |
| rlandy | and all those components will be broken | 21:33 |
| * weshay|ruck cix's it | 21:34 | |
| weshay|ruck | https://trello.com/c/JXICzDz5/1606-cixmissing-deposp-170python-importlib-resources-and-python-importlib-metadata | 21:36 |
| weshay|ruck | that's what burns wanted :) | 21:37 |
| rlandy | weshay|ruck: https://code.engineering.redhat.com/gerrit/#/q/topic:prep-16-2+(status:open+OR+status:merged) - I am stopping there - until we geta 16.2 61-2 decision on monday | 21:37 |
| rlandy | and the train container backports | 21:37 |
| weshay|ruck | I would _ | 21:37 |
| weshay|ruck | not - | 21:37 |
| weshay|ruck | but good to rollcall it | 21:37 |
| rlandy | let's get a final decision | 21:37 |
| weshay|ruck | aye | 21:38 |
| * rlandy has CR work to do now | 21:38 | |
| weshay|ruck | underscores are usually a safe bet | 21:38 |
| weshay|ruck | rlandy, :) | 21:38 |
| rlandy | ack | 21:38 |
| *** rlandy has quit IRC | 22:14 | |
| *** rfolco has joined #oooq | 22:54 | |
| *** holser has joined #oooq | 22:57 | |
| *** rfolco has quit IRC | 22:58 | |
| *** holser has quit IRC | 23:19 | |
| *** holser has joined #oooq | 23:29 | |
| *** tosky has quit IRC | 23:55 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!