| *** rfolco has quit IRC | 00:15 | |
| *** yolanda has quit IRC | 01:26 | |
| *** yolanda has joined #oooq | 01:26 | |
| *** yolanda has quit IRC | 01:27 | |
| *** yolanda has joined #oooq | 01:27 | |
| *** jmasud has joined #oooq | 04:02 | |
| *** raukadah is now known as chkumar|rover | 04:20 | |
| *** ykarel has joined #oooq | 04:25 | |
| *** ysandeep|away is now known as ysandeep | 04:34 | |
| *** skramaja has joined #oooq | 04:46 | |
| *** ratailor has joined #oooq | 04:56 | |
| *** saneax has joined #oooq | 05:04 | |
| *** marios has joined #oooq | 05:10 | |
| *** marios is now known as marios|ruck | 05:10 | |
| chkumar|rover | marios|ruck: Good morning | 05:19 |
|---|---|---|
| chkumar|rover | marios|ruck: c8 master and train got promoted yesterday | 05:20 |
| chkumar|rover | marios|ruck: thank you for taking care of rovering on my absence. :-) | 05:20 |
| *** jbadiapa has joined #oooq | 05:28 | |
| marios|ruck | chkumar|rover: o/ | 05:29 |
| marios|ruck | chkumar|rover: morning | 05:29 |
| marios|ruck | chkumar|rover: didn't even notice that this was stuck ! https://review.rdoproject.org/r/#/c/28928/ :D | 05:30 |
| marios|ruck | i thought it merged already :/ | 05:30 |
| chkumar|rover | sometime zuul ignores us :-) | 05:37 |
| * marios|ruck shakes fist at zuul | 05:38 | |
| *** udesale has joined #oooq | 05:43 | |
| *** udesale has quit IRC | 06:10 | |
| chkumar|rover | marios|ruck: https://trunk.rdoproject.org/centos8-master/component/cloudops/current-tripleo/ cloudops component is fine now | 06:15 |
| marios|ruck | chkumar|rover: thanks | 06:15 |
| chkumar|rover | it also got promoted on 15th Aug | 06:15 |
| chkumar|rover | marios|ruck: regarding rocky tempest bug I think I know the fix https://bugs.launchpad.net/tripleo/+bug/1891372 | 06:25 |
| openstack | Launchpad bug 1891372 in tripleo "rocky periodic jobs are failing with " Error: image tripleorocky/centos-binary-tempest:9801dc7461cbd6cbd73868e72e74d21d586c6708_fbb4de96-updated-20200812131025 not found"" [Critical,Triaged] | 06:25 |
| chkumar|rover | let me propose a patch | 06:25 |
| marios|ruck | ok | 06:26 |
| marios|ruck | chkumar|rover: fyi https://bugs.launchpad.net/tripleo/+bug/1891000/comments/8 (https://review.rdoproject.org/r/28981 ) | 06:32 |
| openstack | Launchpad bug 1891000 in tripleo "master network component failing tempest - neutron containers unexpected keyword argument 'libc'" [Critical,In progress] - Assigned to yatin (yatinkarel) | 06:32 |
| chkumar|rover | marios|ruck: aye thanks! | 06:32 |
| ykarel | marios|ruck, ^ can be closed i commented some time back | 06:33 |
| marios|ruck | ykarel: which comment do you mean | 06:33 |
| *** apetrich has joined #oooq | 06:34 | |
| marios|ruck | ykarel: see https://bugs.launchpad.net/tripleo/+bug/1891000/comments/8 (i think green but there is one suspicious log with tempest fail might be related so holding for now) | 06:34 |
| openstack | Launchpad bug 1891000 in tripleo "master network component failing tempest - neutron containers unexpected keyword argument 'libc'" [Critical,In progress] - Assigned to yatin (yatinkarel) | 06:34 |
| ykarel | marios|ruck, comment 7 | 06:34 |
| ykarel | scenario010 failure is not relatecd | 06:34 |
| ykarel | thats different bug already logged | 06:34 |
| marios|ruck | ykarel: ah ok we were commenting at same time.. and referring to the same buildset | 06:34 |
| marios|ruck | ykarel: which bug | 06:35 |
| ykarel | marios|ruck, scenario010 that i shared last week | 06:35 |
| marios|ruck | ykarel: k i don't recall it do you have it handy? (is it already there? https://hackmd.io/QnprH9-yRTi6uWlEfaahoQ?both | 06:36 |
| ykarel | ok looking fo rbug | 06:36 |
| marios|ruck | https://bugs.launchpad.net/tripleo/+bug/1891293 | 06:37 |
| openstack | Launchpad bug 1891293 in tripleo "periodic centos-8 scen10 standalone master fails tempest - octavia_tempest_plugin scenario.v2" [Critical,Triaged] | 06:37 |
| ykarel | it's on CIX too | 06:37 |
| marios|ruck | maybe that ykarel ^ | 06:37 |
| ykarel | yes that one | 06:37 |
| marios|ruck | ykarel: k thanks | 06:37 |
| ykarel | i proposed for fix that too and it's also clear | 06:37 |
| ykarel | but for network component you would need one more master promotion | 06:37 |
| ykarel | for pyroute2-0.5.13-1.1 | 06:37 |
| marios|ruck | ykarel: chkumar|rover: then fix-released https://bugs.launchpad.net/tripleo/+bug/1891000/comments/9 | 06:38 |
| openstack | Launchpad bug 1891000 in tripleo "master network component failing tempest - neutron containers unexpected keyword argument 'libc'" [Critical,Fix released] - Assigned to yatin (yatinkarel) | 06:38 |
| ykarel | ack | 06:38 |
| chkumar|rover | ack | 06:39 |
| marios|ruck | chkumar|rover: this needs some digging if you have time please https://bugs.launchpad.net/tripleo/+bug/1891179/comments/7 | 06:46 |
| openstack | Launchpad bug 1891179 in tripleo "periodic OVB train centos 7 failing ovb-manage No server with a name or ID" [Critical,Triaged] | 06:46 |
| marios|ruck | chkumar|rover: reached out to rhos-ops on fri but might need to check again | 06:46 |
| marios|ruck | err.. on thursday even ^ | 06:46 |
| chkumar|rover | marios|ruck: sure will check with them | 06:47 |
| marios|ruck | chkumar|rover: ykarel: thanks moving fix-released https://bugs.launchpad.net/tripleo/+bug/1891293/comments/4 | 06:52 |
| openstack | Launchpad bug 1891293 in tripleo "periodic centos-8 scen10 standalone master fails tempest - octavia_tempest_plugin scenario.v2" [Critical,Triaged] | 06:52 |
| ykarel | marios|ruck, ack then it's promotion should clear network component too | 06:58 |
| ykarel | marios|ruck, i think i known No server with a name or ID issue in train ovb | 06:58 |
| ykarel | let me check and fire up patch | 06:59 |
| marios|ruck | ykarel: well looks like a good buildset https://review.rdoproject.org/zuul/buildset/c148bbc632ea4be2893d8cf2f5557217 so should promote | 06:59 |
| ykarel | hmm i seen that | 06:59 |
| ykarel | marios|ruck, https://review.rdoproject.org/r/28982 should clear : No server with a name or ID issue in train ovb | 07:10 |
| marios|ruck | ykarel: oh nice | 07:13 |
| marios|ruck | ykarel: thanks | 07:13 |
| marios|ruck | chkumar|rover: fyi for the centos7 issue https://bugs.launchpad.net/tripleo/+bug/1891179/comments/8 | 07:14 |
| openstack | Launchpad bug 1891179 in tripleo "periodic OVB train centos 7 failing ovb-manage No server with a name or ID" [Critical,Triaged] | 07:14 |
| ykarel | marios|ruck, also comented on ^ | 07:14 |
| ykarel | wrt issue and fix | 07:14 |
| marios|ruck | ack thanks | 07:15 |
| chkumar|rover | ykarel: nice catch :-) | 07:16 |
| marios|ruck | chkumar|rover: looks like ykarel got this... pub? | 07:16 |
| marios|ruck | chkumar|rover: meet you there | 07:16 |
| *** matbu has joined #oooq | 07:20 | |
| chkumar|rover | ykarel: marios|ruck https://review.opendev.org/#/c/746446/ when free | 07:53 |
| chkumar|rover | posting a test project patch | 07:54 |
| *** dtantsur|afk is now known as dtantsur | 07:55 | |
| *** tosky has joined #oooq | 07:56 | |
| *** jpena|off is now known as jpena | 07:58 | |
| ykarel | chkumar|rover, ack, changing defaults doesn't looks good though | 08:02 |
| ykarel | need to check if it doesn't impact other releases | 08:02 |
| chkumar|rover | ykarel: https://review.rdoproject.org/r/#/c/28983/ testing with other release also | 08:04 |
| marios|ruck | chkumar|rover: ack | 08:06 |
| ykarel | chkumar|rover, ack let' see results then, but i think actual issue is in container preps | 08:10 |
| ykarel | containers should have been prepared with updated tag | 08:10 |
| chkumar|rover | ykarel: i think https://review.opendev.org/745718 caused it | 08:10 |
| chkumar|rover | as older one was passing | 08:11 |
| chkumar|rover | ykarel: weired thing is that fs02 is passing | 08:11 |
| chkumar|rover | https://logserver.rdoproject.org/openstack-periodic-integration-stable4-5/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-rocky-upload/2f13ebb/logs/undercloud/home/zuul/tempest.log.txt.gz | 08:12 |
| ykarel | chkumar|rover, but https://review.opendev.org/#/c/745718/ is just standalone, non standalone jobs shouldn't be impacted by that change | 08:24 |
| ykarel | so something else caused it | 08:25 |
| chkumar|rover | marios|ruck: https://review.rdoproject.org/r/#/c/28979/ -> periodic-tripleo-ci-centos-8-multinode-1ctlr-featureset030-ussuri SUCCESS in 1h 35m 58s | 08:51 |
| marios|ruck | chkumar|rover: thanks | 08:55 |
| *** jfrancoa has joined #oooq | 09:07 | |
| chkumar|rover | kopecmartin: arxcruz akahat svyas check this out https://opendev.org/vexxhost/tempest-pushgateway | 09:48 |
| chkumar|rover | may be we can fix some workaround used there like https://opendev.org/vexxhost/tempest-pushgateway/src/branch/master/tempest_pushgateway/__init__.py#L104 | 09:49 |
| arxcruz | cool, they use tempestconf | 09:49 |
| chkumar|rover | arxcruz: yup | 09:49 |
| chkumar|rover | arxcruz: kopecmartin https://twitter.com/_mnaser/status/1295112003070156805 | 09:50 |
| kopecmartin | interesting | 09:51 |
| chkumar|rover | arxcruz: can we prepare the final patch to move all stein, queens, and rocky to os_tempest? | 09:57 |
| arxcruz | chkumar|rover: yes and no | 09:57 |
| arxcruz | chkumar|rover: I just realize last week, that we do have rocky jobs | 09:57 |
| chkumar|rover | arxcruz: any thing still missing? | 09:57 |
| arxcruz | which was using validate-tempest | 09:57 |
| arxcruz | instead of os_tempest | 09:58 |
| arxcruz | chkumar|rover: what are the featuresets we are using at the moment? | 10:29 |
| chkumar|rover | arxcruz: hard to tell, let me get the pipeline | 10:32 |
| chkumar|rover | arxcruz: https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/integration-pipeline-stable3.yaml and https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/integration-pipeline-stable4-5.yaml | 10:32 |
| chkumar|rover | arxcruz: Do we want to do a cleanup of all featureset also? | 10:33 |
| marios|ruck | chkumar|rover: when you hav some time can you please check promoter re ussuri current-tripleo-rdo there is a trace about missing containers | 10:34 |
| marios|ruck | chkumar|rover: added note on the hackmd | 10:34 |
| chkumar|rover | marios|ruck: looking | 10:35 |
| chkumar|rover | it means all the containers are missing on rdo registry or not built | 10:39 |
| chkumar|rover | let me check the job with that hash | 10:39 |
| marios|ruck | chkumar|rover: thanks | 10:39 |
| chkumar|rover | marios|ruck: https://trunk.rdoproject.org/centos8-ussuri/tripleo-ci-testing/29/4c/294c650b84e58da9874de4ba41d2a08a/ it has taken the hash of 04th Aug | 10:43 |
| chkumar|rover | those containers does not exists now in rdo registry so failure is legit | 10:44 |
| chkumar|rover | may be we can improve the messaging part | 10:44 |
| chkumar|rover | let me propose a patch for the same | 10:44 |
| marios|ruck | thanks chkumar|rover | 10:48 |
| marios|ruck | chkumar|rover: and probably we need to do something about that it seems too strict... i mean 2 weeks and the containers are scrubbed ? | 10:50 |
| chkumar|rover | marios|ruck: may we need to fix something on promoter side to look for latest hash | 10:52 |
| chkumar|rover | instead of looking on older one | 10:52 |
| chkumar|rover | *may be | 10:52 |
| chkumar|rover | https://trunk.rdoproject.org/api-centos8-ussuri/api/civotes_agg_detail.html?ref_hash=294c650b84e58da9874de4ba41d2a08a -> this hash might fulfill the criteria | 10:55 |
| marios|ruck | chkumar|rover: k maybe | 10:55 |
| chkumar|rover | marios|ruck: https://review.rdoproject.org/r/#/c/28986/ | 11:00 |
| chkumar|rover | marios|ruck: for ussuri promotion, do we want to make upgrade job nv and promote and then make voting again once fix lands? | 11:03 |
| marios|ruck | chkumar|rover: no rather not i mean at least not yet ... it is established to be valid issue so why promote bad content | 11:10 |
| chkumar|rover | yes, I agree | 11:12 |
| *** udesale has joined #oooq | 11:20 | |
| *** jpena is now known as jpena|lunch | 11:33 | |
| *** weshay_ is now known as weshay|ruck | 11:36 | |
| weshay|ruck | marios|ruck, chkumar|rover 0/ | 11:37 |
| weshay|ruck | you folks have a handle on ussuri? | 11:39 |
| arxcruz | chkumar|rover: the easier would be just ignore the featureset and foce the use_os_tempest on tempest playbook | 11:41 |
| arxcruz | s/foce/force | 11:41 |
| weshay|ruck | marios|ruck, chkumar|rover we have to clean up rdo cloud heat stacks again | 11:42 |
| *** rfolco has joined #oooq | 11:43 | |
| marios|ruck | weshay|ruck: o/ | 11:45 |
| chkumar|rover | weshay|ruck: \o/ | 11:45 |
| weshay|ruck | howdy folks | 11:45 |
| chkumar|rover | arxcruz: if it works then we are good | 11:46 |
| weshay|ruck | is a sync a good idea? | 11:46 |
| weshay|ruck | chkumar|rover, arxcruz pretty cool there.. mnaser's tweet | 11:46 |
| marios|ruck | weshay|ruck: yeah ... so pinged about the heat stacks on thursday in rhos-ops... we *thought* (mistakenly) it was related to https://bugs.launchpad.net/tripleo/+bug/1891179/comments/7 | 11:46 |
| openstack | Launchpad bug 1891179 in tripleo "periodic OVB train centos 7 failing ovb-manage No server with a name or ID" [Critical,Triaged] | 11:46 |
| marios|ruck | weshay|ruck: have you checked https://hackmd.io/QnprH9-yRTi6uWlEfaahoQ?both all known things there | 11:47 |
| marios|ruck | chkumar|rover: was going to ping rhos-ops about the stacks today (for same bug) but then ykarel noticed the actual problem (https://review.rdoproject.org/r/28982 ) so not sure if you did already chkumar|rover ? | 11:48 |
| chkumar|rover | marios|ruck: I have merged the patch | 11:48 |
| marios|ruck | chkumar|rover: great ... but we need to sort out the heat stacks 14:42 < weshay|ruck> marios|ruck, chkumar|rover we have to clean up rdo cloud heat stacks again | 11:48 |
| marios|ruck | weshay|ruck: chkumar|rover: remind me is that something we do or that we ask rhos-ops to do for us? | 11:49 |
| marios|ruck | http://dashboard-ci.tripleo.org/d/wb8HBhrWk/cockpit?orgId=1&fullscreen&panelId=231 | 11:49 |
| chkumar|rover | marios|ruck: weshay|ruck has access to the box and he can do a cleanup | 11:49 |
| weshay|ruck | marios|ruck, I'll show you guys.. I think we need to fix the cron or script | 11:49 |
| chkumar|rover | last time we had done over tmate | 11:49 |
| weshay|ruck | you all should have access | 11:49 |
| weshay|ruck | chkumar|rover, marios|ruck https://docs.google.com/document/d/1bHVPN-lcXyY7imLdvj2J7I55s-vtF3ffPG-y2Cj8u38/edit#heading=h.und2p35wlqs7 | 11:50 |
| zbr | is the promoter daily sync still a thing? | 11:51 |
| marios|ruck | weshay|ruck: ack... chkumar|rover but if those scrips are running on cron then why/what should we do? re-run? they are going every 30/40 mins? | 11:51 |
| marios|ruck | (am on the box) | 11:51 |
| chkumar|rover | marios|ruck: I am not able to enter into that box | 11:51 |
| weshay|ruck | marios|ruck, unclear.. something is not happening automatically.. not sure what atm | 11:52 |
| marios|ruck | weshay|ruck: k chkumar|rover call in 10 mins at top of hour for sync ok? | 11:52 |
| chkumar|rover | marios|ruck: ok | 11:52 |
| marios|ruck | weshay|ruck: are you available for tmate to show us? otherwise after the cix/before scrum, otherwise after scrum :) | 11:52 |
| marios|ruck | brb | 11:52 |
| weshay|ruck | marios|ruck, after scrum.. rfolco I will be late to the scrum call.. I have to go to the network DFG scrum to explain the component pipeline | 11:53 |
| weshay|ruck | bhagyashris, want to 1-1 now? | 11:53 |
| rfolco | weshay|ruck, ack | 11:53 |
| weshay|ruck | ysandeep, do we know why rhos-17 is all red? in the compnent pipeline? | 11:55 |
| bhagyashris | weshay|ruck, give me 5 min | 11:56 |
| weshay|ruck | bhagyashris, k | 11:57 |
| bhagyashris | weshay|ruck, joined | 11:58 |
| marios|ruck | weshay|ruck: ack | 11:59 |
| ysandeep | weshay|ruck, looking | 12:00 |
| chkumar|rover | rfolco: https://review.rdoproject.org/r/#/c/28986 please have a look | 12:00 |
| *** rlandy has joined #oooq | 12:05 | |
| weshay|ruck | rlandy, 0/ good morning.. looks like there was a change for rhos-release | 12:06 |
| weshay|ruck | http://tripleo-cockpit.usersys.redhat.com/d/v8gltz4Mz/rhos-17-full-component-pipeline?orgId=1&from=now-7d&to=now | 12:07 |
| weshay|ruck | ysandeep, is checking it out | 12:07 |
| rlandy | weshay|ruck: whatever | 12:07 |
| rlandy | trust me least of our problems | 12:07 |
| weshay|ruck | oh boy | 12:07 |
| weshay|ruck | what's up? | 12:07 |
| rlandy | we have so many - 17 is who cares | 12:07 |
| rlandy | I don't | 12:07 |
| *** ykarel has quit IRC | 12:08 | |
| *** ykarel has joined #oooq | 12:09 | |
| chkumar|rover | rlandy: regarding image build issue https://bugzilla.redhat.com/show_bug.cgi?id=1828681 | 12:11 |
| openstack | bugzilla.redhat.com bug 1828681 in qemu-kvm "Unable to initialize gcrypt will prevent qemu related commands from running" [Urgent,Verified] - Assigned to ddepaula | 12:11 |
| chkumar|rover | RHEL-8.3 and 16.2 | 12:12 |
| rlandy | chkumar|rover: thank you for looking into it | 12:13 |
| rlandy | was getting nowhere | 12:13 |
| chkumar|rover | rlandy: Package libgcrypt-devel.x86_64 1.8.3-4.el8 will be installed | 12:13 |
| chkumar|rover | just update this package | 12:13 |
| chkumar|rover | and we are done | 12:14 |
| chkumar|rover | rlandy: can I update your patches? | 12:14 |
| rlandy | chkumar|rover: sure | 12:14 |
| *** udesale_ has joined #oooq | 12:14 | |
| rlandy | weshay|ruck: tbh - we have taken on a much bigger undertaking that we estimated | 12:14 |
| rlandy | weshay|ruck: we are building out 16.2 on an unstable rhel 8.3 base | 12:15 |
| weshay|ruck | ok.. you'll have to fill me in today. sorry to hear that | 12:15 |
| rlandy | to be come 8.4 | 12:15 |
| rlandy | with unstable container toold | 12:15 |
| rlandy | changing image bases | 12:15 |
| weshay|ruck | and you spoke to that in prod-chain? | 12:15 |
| rlandy | weshay|ruck: no consensus on how to import to 16.2 w/o impacting 16.1 | 12:16 |
| weshay|ruck | jeez | 12:16 |
| rlandy | weshay|ruck: i did, alex did | 12:16 |
| rlandy | mike sent a whole doc to the leadership team | 12:16 |
| rlandy | nobody has any idea how to handle all this | 12:16 |
| *** udesale has quit IRC | 12:17 | |
| rlandy | and yeah - and rhos-release - that's the second time we dealt with this since last week | 12:17 |
| rlandy | so, yeah, least of our problems | 12:17 |
| *** ratailor has quit IRC | 12:21 | |
| rlandy | ysandeep: hey - so standalone got somewhere on Thursday | 12:22 |
| rlandy | chatted with Emilien - maybe real issue with the containers | 12:22 |
| ysandeep | rlandy, yes trying to debug that since morning - http://pastebin.test.redhat.com/893831 | 12:23 |
| rlandy | ysandeep: will fill you in at scrum | 12:23 |
| ysandeep | rlandy, ack | 12:24 |
| rlandy | marios|ruck: chkumar|rover: I see we are leaking stacks again on RDO cloud | 12:25 |
| rlandy | weshay|ruck: ^^ fyi | 12:25 |
| rlandy | something must be up with delete | 12:26 |
| chkumar|rover | rlandy: we will be deleting after scrum | 12:26 |
| rlandy | chkumar|rover: it's a consistent step every day | 12:26 |
| rlandy | same last week | 12:26 |
| rlandy | we leak a certain amount of stacks every day | 12:26 |
| rlandy | chkumar|rover: see the pattern on cockpit? | 12:27 |
| chkumar|rover | rlandy: http://dashboard-ci.tripleo.org/d/wb8HBhrWk/cockpit?orgId=1&fullscreen&panelId=231 | 12:27 |
| rlandy | it's about 10 stacks a day | 12:27 |
| rlandy | chkumar|rover: weird right? | 12:27 |
| chkumar|rover | yes | 12:27 |
| rlandy | maybe stein or a once a day job is leaking stacks | 12:28 |
| rlandy | train? | 12:28 |
| rlandy | started tracking it last week with marios|ruck | 12:28 |
| rlandy | chkumar|rover: I didn't notice the same on vexxhost | 12:28 |
| chkumar|rover | last to last friday weshay|ruck clean up | 12:29 |
| marios|ruck | rlandy: weshay|ruck: could be related to the issue that ykarel found? https://bugs.launchpad.net/tripleo/+bug/1891179/comments/8 | 12:29 |
| openstack | Launchpad bug 1891179 in tripleo "periodic OVB train centos 7 failing ovb-manage No server with a name or ID" [Critical,Triaged] | 12:29 |
| * marios|ruck joining cix | 12:29 | |
| rlandy | looking | 12:29 |
| rfolco | arxcruz, this is failing on my env --> https://review.rdoproject.org/r/#/c/28956/2/ci-scripts/infra-setup/roles/promoter/tasks/install_packages.yml | 12:32 |
| arxcruz | rfolco: can you paste me the failure? | 12:32 |
| arxcruz | it's working on mine :P | 12:32 |
| rfolco | arxcruz, because I only have python3... yum only works if you have python2 | 12:33 |
| rfolco | arxcruz, The Python 2 yum module is needed for this module. If you require Python 3 support use the `dnf` Ansible module instead | 12:33 |
| arxcruz | rfolco: hmmmm | 12:33 |
| arxcruz | rfolco: i only have python3 also | 12:34 |
| rfolco | weird | 12:34 |
| arxcruz | rfolco: maybe we should move to package: | 12:34 |
| arxcruz | instead of dnf/yum | 12:34 |
| rfolco | yep lets do it | 12:34 |
| arxcruz | rfolco: ok | 12:34 |
| rfolco | arxcruz, please fix typo in comment as well | 12:35 |
| arxcruz | rfolco: yum_repository is also failing, or passing for you ? | 12:35 |
| rfolco | arxcruz, yeah... its not getting there, but I guess it will fail... can we switch to python3 equivalent ? | 12:36 |
| *** jpena|lunch is now known as jpena | 12:38 | |
| rfolco | arxcruz, maybe enablerepo option from dnf module, I am not quite sure | 12:41 |
| arxcruz | rfolco: there's no dnf_repository or silimar for dnf, only for yum | 12:42 |
| arxcruz | rfolco: https://github.com/ansible/ansible/issues/46963 | 12:42 |
| arxcruz | unless zbr knows something | 12:42 |
| sshnaidm | ykarel, jpena rlandy tristanC can you take a look please: https://review.rdoproject.org/r/#/c/28977/ https://review.rdoproject.org/r/#/c/28976/ | 12:42 |
| zbr | if it remember well ansible modules where incomplete and configuring via cli was the only reliable way | 12:45 |
| zbr | i doubt the repo module was improved recently | 12:45 |
| zbr | but do not get confused by yum_ prefix name, it applies as well to dnf, nobody will bother to rename it | 12:45 |
| zbr | ideally they should have being called "rpm_repository" to avoid implementation detail | 12:46 |
| zbr | still, ini_file works better | 12:46 |
| rfolco | arxcruz, if its working with py3, lets keep it | 12:47 |
| rlandy | marios|ruck: chkumar|rover: ack- this makes sense ... https://bugs.launchpad.net/tripleo/+bug/1891179/comments/9 | 12:49 |
| openstack | Launchpad bug 1891179 in tripleo "periodic OVB train centos 7 failing ovb-manage No server with a name or ID" [Critical,Triaged] | 12:49 |
| rlandy | as to why we would be leaking the same number of stacks every day | 12:49 |
| rlandy | ykarel++ | 12:49 |
| rlandy | chkumar|rover++ ... image builds looking better :) ... nice work | 12:51 |
| chkumar|rover | rlandy: weshay|ruck https://code.engineering.redhat.com/gerrit/#/c/207510/ | 12:55 |
| chkumar|rover | periodic-tripleo-rhel-8-buildimage-overcloud-full-rhos-16.2 https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/build/12539b20f65e4ff287c309479770c052 : SUCCESS in 29m 00s | 12:55 |
| chkumar|rover | periodic-tripleo-rhel-8-buildimage-ironic-python-agent-rhos-16.2 https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/build/1b65639394674dc582161f2a24ab77aa : SUCCESS in 12m 24s | 12:55 |
| chkumar|rover | perfectoo | 12:56 |
| rlandy | chkumar|rover: yay! | 12:56 |
| chkumar|rover | rlandy: Does container build for rhos-16.2 ready? | 12:56 |
| rlandy | chkumar|rover: ack | 12:57 |
| rlandy | chkumar|rover: build/push/upload all in place | 12:57 |
| chkumar|rover | cool | 12:57 |
| rlandy | a lot of upstream patches need to merge though | 12:57 |
| chkumar|rover | then I think we are ready to roll the job definitions | 12:57 |
| rlandy | before we can get periodic pipeline passing | 12:57 |
| weshay|ruck | nice | 12:57 |
| chkumar|rover | rlandy: we can push the containers and consume in periodic jobs running in check | 12:58 |
| rlandy | chkumar|rover: let's talk it through at scrum | 12:58 |
| chkumar|rover | rlandy: sure | 12:58 |
| rlandy | chkumar|rover: https://code.engineering.redhat.com/gerrit/#/c/208453/ can probably merge though | 12:59 |
| rlandy | chkumar|rover: https://code.engineering.redhat.com/gerrit/#/c/207802/ container builds | 13:00 |
| chkumar|rover | rlandy: https://code.engineering.redhat.com/gerrit/208453 good to go | 13:00 |
| rfolco | arxcruz, rfolco, zbr, sshnaidm, rlandy, marios, ysandeep, bhagyashris, svyas, pojadhav, akahat, weshay, chandankumar | 13:00 |
| rfolco | ping scrum | 13:00 |
| *** udesale_ has quit IRC | 13:09 | |
| *** Goneri has joined #oooq | 13:13 | |
| *** dtantsur has quit IRC | 13:32 | |
| rlandy | ysandeep: I'm going to separate the container builds patch - so that the parts that standalone relies on are not clashing | 13:36 |
| rlandy | then try build containers again | 13:36 |
| rlandy | merged image builds | 13:36 |
| ysandeep | rlandy, ack | 13:36 |
| weshay|ruck | rfolco, you guys done w/ scrum? | 13:38 |
| rlandy | ysandeep: or patch went through train https://github.com/openstack/tripleo-common/commit/082d3bed48911cafcea56c020c01ba561bd913b4 and http://git.app.eng.bos.redhat.com/git/openstack-tripleo-common.git/commit/?h=rhos-16.2-trunk-patches&id=21b4a1f3511fec6eab2c25db52acd85e1693ab84 | 13:38 |
| *** dtantsur has joined #oooq | 13:39 | |
| rlandy | http://git.app.eng.bos.redhat.com/git/openstack-tripleo-common.git/tree/container-images/tcib/base/os/gnocchi-base/gnocchi-base.yaml?h=rhos-16.2-trunk-patches | 13:39 |
| rlandy | still missing gnocchi | 13:39 |
| weshay|ruck | rlandy, we out of scrum? | 13:39 |
| rlandy | weshay|ruck: yes | 13:39 |
| weshay|ruck | rlandy, ok.. want to download on me? | 13:40 |
| ysandeep | rlandy, i don't think we need gnocchi changes, that was because of conflict of rpms coming from 8.2 vs 8.3 | 13:40 |
| ysandeep | those gnocchi changes never helped us | 13:40 |
| weshay|ruck | rlandy, https://meet.google.com/yky-javb-bsz?authuser=1 | 13:41 |
| rlandy | ysandeep: k - will kick containers build again | 13:41 |
| rlandy | maybe we'll get lucky there on what is in place | 13:41 |
| rlandy | weshay|ruck: joining in a sec | 13:41 |
| weshay|ruck | chkumar|rover, marios|ruck one of you cleaning stacks? | 13:46 |
| ysandeep | weshay|ruck, we are installing and running rhos-relase in multiple places, testing removal of pre-rhos.yaml - https://code.engineering.redhat.com/gerrit/#/c/201382/ to fix current rhosp-17 issues | 13:47 |
| weshay|ruck | k.. thanks | 13:47 |
| rfolco | weshay|ruck, sorry , we're done, let me know if you want to sync/chat | 13:48 |
| chkumar|rover | weshay|ruck: not me, https://review.rdoproject.org/r/#/c/28982/ will take care of the issue, may be cron job will delte the stacks | 13:49 |
| *** ykarel is now known as ykarel|away | 13:53 | |
| marios|ruck | weshay|ruck: o/ no i thought we were going to do that together | 13:58 |
| chkumar|rover | marios|ruck: weshay|ruck rlandy https://review.rdoproject.org/r/#/c/28785/ | 13:59 |
| arxcruz | chkumar|rover: https://review.opendev.org/#/c/746509/ | 14:00 |
| marios|ruck | chkumar|rover: ack | 14:01 |
| chkumar|rover | arxcruz: ack | 14:03 |
| * marios|ruck back in 15 | 14:14 | |
| *** ykarel_ has joined #oooq | 14:20 | |
| chkumar|rover | rfolco: can you pass me the fs02 patch? | 14:21 |
| *** ykarel|away has quit IRC | 14:22 | |
| rfolco | chkumar|rover, I just changed something that may fix it. Will add you to the review after it runs... | 14:22 |
| chkumar|rover | rfolco: ok | 14:22 |
| rfolco | chkumar|rover, thx | 14:23 |
| *** sshnaidm is now known as sshnaidm|MYG | 14:27 | |
| *** sshnaidm|MYG is now known as sshnaidm|mtg | 14:27 | |
| rlandy | chkumar|rover: W+'ed https://review.rdoproject.org/r/#/c/28785/ | 14:28 |
| chkumar|rover | rlandy: thanks! | 14:35 |
| chkumar|rover | rlandy: https://review.rdoproject.org/r/#/c/28986/ this also | 14:36 |
| rlandy | chkumar|rover: also w+'ed - much better message | 14:39 |
| chkumar|rover | marios|ruck: anything else needed? I am logging out now | 14:44 |
| marios|ruck | chkumar|rover: ack will catch up tomorrow then have a good one | 14:45 |
| *** chkumar|rover is now known as raukadah | 14:45 | |
| raukadah | see ya then! | 14:45 |
| marios|ruck | we're almost there raukadah ;) | 14:46 |
| marios|ruck | ~ 52 hours | 14:46 |
| weshay|ruck | marios|ruck, raukadah hand off anything you want me to drive | 14:56 |
| marios|ruck | weshay|ruck: o/ for the heat stacks are we waiting for cron to take care ? (hasn't yet looking at http://dashboard-ci.tripleo.org/d/wb8HBhrWk/cockpit?orgId=1&fullscreen&panelId=231 | 14:58 |
| marios|ruck | weshay|ruck: i think the fix as discussed earlier re the vexx vs rdo https://review.rdoproject.org/r/28982 should help | 14:59 |
| marios|ruck | weshay|ruck: otherwise, latest i know about on the hackmd | 14:59 |
| marios|ruck | brb | 15:00 |
| marios|ruck | back | 15:01 |
| weshay|ruck | marios|ruck, right.. but I think there is some issue w / the cron on the toolbox | 15:02 |
| weshay|ruck | we should see it auto cleanup quite often and it is not | 15:02 |
| weshay|ruck | waiting is not going to help | 15:02 |
| weshay|ruck | either we run it manually, or fix cron / scripts on that box | 15:02 |
| weshay|ruck | afaict it's cron | 15:02 |
| marios|ruck | weshay|ruck: ok, should i just log on and run them manually now? | 15:02 |
| marios|ruck | weshay|ruck: and maybe make a minor edit comment/uncomment and reinstall the crontab | 15:03 |
| marios|ruck | weshay|ruck: running | 15:06 |
| weshay|ruck | ++ | 15:06 |
| marios|ruck | weshay|ruck: there was a typo in the crotab /opt/stack/ci-config/ci-scripts/infra-cleanup/ovb-tenant-cleanup.sh &>>/var/log/clean_stacks_rdo.log' | 15:11 |
| marios|ruck | extra ' at edn | 15:11 |
| rlandy | ysandeep: hey ... https://code.engineering.redhat.com/gerrit/#/c/208964/ | 15:12 |
| rlandy | separated the required projects for standalone jobs | 15:12 |
| ysandeep | rlandy, thank you | 15:13 |
| *** ykarel_ has quit IRC | 15:14 | |
| *** ykarel_ has joined #oooq | 15:15 | |
| ysandeep | rlandy, weshay|ruck https://code.engineering.redhat.com/gerrit/#/c/208957/ to fix current rhos-release issues with 17 jobs ( https://sf.hosted.upshift.rdu2.redhat.com/logs/82/201382/12/check/periodic-tripleo-ci-rhel-8-standalone-clients-rhos-17/a90df4d/job-output.txt ) | 15:23 |
| ysandeep | There is an another issue with 17 line too - https://sf.hosted.upshift.rdu2.redhat.com/logs/82/201382/13/check/periodic-tripleo-ci-rhel-8-standalone-clients-rhos-17/f79a7a3/logs/undercloud/home/zuul/standalone_deploy.log , will check that tomorrow morning | 15:24 |
| weshay|ruck | ysandeep, thank you | 15:25 |
| rlandy | ysandeep: looks good - thanks | 15:25 |
| *** ykarel_ has quit IRC | 15:25 | |
| ysandeep | thank you :) | 15:25 |
| *** ysandeep is now known as ysandeep|away | 15:42 | |
| marios|ruck | weshay|ruck: seems stuck on ovb-tenant-cleanup been running ~40 mins now | 15:51 |
| marios|ruck | weshay|ruck: i will dig further tomorrow... but there *was* a typo in the crontab so fixed it and hopefully that will help | 15:51 |
| marios|ruck | weshay|ruck: going in couple mins | 15:51 |
| *** skramaja has quit IRC | 15:59 | |
| *** marios|ruck is now known as marios|out | 16:00 | |
| weshay|ruck | marios|out, 0/ | 16:01 |
| *** marios|out has quit IRC | 16:05 | |
| *** sshnaidm|mtg is now known as sshnaidm | 16:08 | |
| *** jpena is now known as jpena|off | 16:14 | |
| rfolco | zbr, these 2 are green and ready to review: https://review.rdoproject.org/r/#/c/28944/ and https://review.rdoproject.org/r/#/c/28963/ | 16:34 |
| rfolco | anyone else welcome to review these 2 changes ^ | 16:35 |
| rfolco | arxcruz, ^ | 16:35 |
| rlandy | https://sf.hosted.upshift.rdu2.redhat.com/logs/02/207802/46/check/tripleo-build-containers-ubi-8-internal-rhel-8-rhos-17/f82637f/logs/buildah-builds/container-builds/1ce4fb44-ede8-4b95-9f5f-5ea15b4b63a4/base/os/tempest/tempest-build.log | 16:50 |
| rlandy | did we see this in the upstream at all? | 16:50 |
| *** dtantsur is now known as dtantsur|afk | 16:59 | |
| rfolco | zbr, ping | 17:29 |
| rfolco | still around ? | 17:30 |
| zbr | sort off | 17:30 |
| rfolco | hmm ok I know its late | 17:30 |
| rfolco | please see my answer on https://review.rdoproject.org/r/#/c/28963/ | 17:30 |
| rfolco | when you get a chance | 17:30 |
| zbr | ok. | 17:31 |
| zbr | we will see if it causes a problem anyway | 17:31 |
| rfolco | thanks zbr | 17:32 |
| rlandy | weshay|ruck: updated https://hackmd.io/I_CFKPXHTza-5i2kFDIs5Q | 18:37 |
| weshay|ruck | k | 18:38 |
| rlandy | weshay|ruck: basically image builds work, container build/push/upload works with deps patches and standalone fails https://bugzilla.redhat.com/show_bug.cgi?id=1869379 | 18:55 |
| openstack | bugzilla.redhat.com bug 1869379 in rabbitmq-server "[OSP 16.2] Containers fail to start for step 2 - mysql_init_bundle error" [Unspecified,New] - Assigned to plemenko | 18:55 |
| rlandy | pick your poison | 18:55 |
| weshay|ruck | k.. few min | 18:55 |
| rlandy | possibly related to the rhos-17 failure https://sf.hosted.upshift.rdu2.redhat.com/logs/82/201382/12/check/periodic-tripleo-ci-rhel-8-standalone-clients-rhos-17/a90df4d/job-output.txt | 18:55 |
| rlandy | no rush | 18:56 |
| *** jbadiapa has quit IRC | 19:21 | |
| *** saneax has quit IRC | 19:38 | |
| weshay|ruck | rlandy, omg.. you still around? | 20:02 |
| rlandy | weshay|ruck: yeah :) | 20:02 |
| rlandy | it's only 4:02 | 20:02 |
| rlandy | all info is on hackmd | 20:03 |
| weshay|ruck | k.. /me looks | 20:03 |
| rlandy | standalone has a BZ | 20:03 |
| weshay|ruck | rlandy, so container builds are on hold until 8/26? | 20:04 |
| rlandy | weshay|ruck: why would they be? | 20:04 |
| rlandy | containers are building and pushing | 20:04 |
| rlandy | that's just the node | 20:04 |
| rlandy | we can run on an 8.2 node - with rhos-release -u -r 8.3 | 20:05 |
| rlandy | in fact we were running on 8.1 for a long time | 20:05 |
| weshay|ruck | https://meet.google.com/cfq-jfuq-gbm?authuser=1 | 20:06 |
| rlandy | weshay|ruck: https://bugzilla.redhat.com/show_bug.cgi?id=1869379 | 20:31 |
| openstack | bugzilla.redhat.com bug 1869379 in rabbitmq-server "[OSP 16.2] Containers fail to start for step 2 - mysql_init_bundle error" [Unspecified,New] - Assigned to plemenko | 20:31 |
| *** apetrich has quit IRC | 20:32 | |
| rlandy | weshay|ruck: https://review.rdoproject.org/r/#/c/28775/ | 20:46 |
| *** rfolco has quit IRC | 20:51 | |
| rlandy | weshay|ruck: cix sent .. thanks | 21:08 |
| *** rlandy has quit IRC | 21:08 | |
| *** rfolco has joined #oooq | 22:34 | |
| *** tosky has quit IRC | 22:57 | |
| *** saneax has joined #oooq | 23:14 | |
| *** rfolco has quit IRC | 23:29 | |
| *** saneax has quit IRC | 23:50 | |
| *** Goneri has quit IRC | 23:54 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!