| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp- (3 more messages) | 00:22 |
|---|---|---|
| *** d0ugal has quit IRC | 01:06 | |
| *** d0ugal has joined #oooq | 01:12 | |
| *** gkadam_ has joined #oooq | 02:11 | |
| *** gkadam_ has quit IRC | 02:17 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp- (3 more messages) | 02:22 |
| *** irclogbot_0 has quit IRC | 02:23 | |
| *** ykarel has joined #oooq | 02:52 | |
| *** udesale has joined #oooq | 04:08 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp- (3 more messages) | 04:22 |
| *** rodrigods has quit IRC | 05:01 | |
| *** skramaja has joined #oooq | 05:11 | |
| *** fultonj has quit IRC | 05:22 | |
| *** fultonj has joined #oooq | 05:30 | |
| *** fultonj has quit IRC | 05:41 | |
| *** ykarel_ has joined #oooq | 05:43 | |
| *** fultonj has joined #oooq | 05:43 | |
| *** ykarel has quit IRC | 05:45 | |
| *** jaosorior has joined #oooq | 05:45 | |
| *** ykarel_ is now known as ykarel | 05:51 | |
| ykarel | arxcruz, jobs are still failing: OSError: [Errno 2] No such file or directory: '/home/stack/tempest' even after https://review.openstack.org/#/c/642420/ | 05:55 |
| ykarel | is the patch ^^ was not sufficient for the bug? | 05:56 |
| *** saneax has joined #oooq | 05:57 | |
| *** jbadiapa has joined #oooq | 06:06 | |
| *** ratailor has joined #oooq | 06:10 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp- (3 more messages) | 06:22 |
| *** brault has joined #oooq | 06:46 | |
| *** brault has quit IRC | 06:55 | |
| *** jaosorior has quit IRC | 06:58 | |
| *** jfrancoa has joined #oooq | 06:58 | |
| *** kopecmartin|off is now known as kopecmartin | 07:03 | |
| *** jaosorior has joined #oooq | 07:11 | |
| *** jtomasek has joined #oooq | 07:12 | |
| *** jaosorior has quit IRC | 07:35 | |
| *** apetrich has joined #oooq | 07:38 | |
| *** jaosorior has joined #oooq | 07:44 | |
| *** jaosorior has quit IRC | 07:44 | |
| arxcruz | ykarel: it should be according chandankumar | 07:47 |
| chandankumar | ykarel: do you have the error log? | 07:47 |
| arxcruz | ykarel: can you check if https://review.openstack.org/#/c/643237/ is there as well ? | 07:48 |
| ykarel | chandankumar, https://ci.centos.org/job/tripleo-quickstart-promote-rocky-rdo_trunk-minimal/125/consoleText | 07:48 |
| ykarel | arxcruz, the job ran 15 hours ago so https://review.openstack.org/#/c/643237/ should be there | 07:48 |
| ykarel | arxcruz, no ^^ will not be there | 07:49 |
| ykarel | as it's need promotion to current-tripleo first | 07:49 |
| arxcruz | ykarel: so, the https://review.openstack.org/#/c/643237/ is required as well | 07:49 |
| chandankumar | arxcruz: ykarel error is this Could not find or access '/var/lib/tempestdata/tempest' | 07:49 |
| ykarel | arxcruz, but that patch is master, the error i linked is from rocky | 07:50 |
| chandankumar | ykarel: temepst workpsace is backported | 07:51 |
| chandankumar | to rocky | 07:51 |
| chandankumar | ykarel: https://review.openstack.org/#/c/643237/ | 07:52 |
| ykarel | chandankumar, ^^ is in the job | 07:53 |
| ykarel | still failing | 07:53 |
| ykarel | Could not find or access '/var/lib/tempestdata/tempest' was in the logged bug as well | 07:53 |
| arxcruz | ykarel: any chance i can log into the machine? | 07:56 |
| chandankumar | https://ci.centos.org/artifacts/rdo/jenkins-tripleo-quickstart-promote-rocky-rdo_trunk-minimal-125/undercloud/home/stack/tempest.log.gz temepst works successfully | 07:56 |
| ykarel | arxcruz, nope | 07:58 |
| arxcruz | sshnaidm|afk: hey, do you know a easy way to reproduce tripleo-quickstart-promote-rocky-rdo_trunk-minimal job ? | 07:59 |
| ykarel | chandankumar, tempest was running successfully earlier as well | 07:59 |
| ykarel | https://ci.centos.org/artifacts/rdo/jenkins-tripleo-quickstart-promote-rocky-rdo_trunk-minimal-118/undercloud/home/stack/tempest.log.gz | 07:59 |
| ykarel | in the logged bug i mean | 07:59 |
| arxcruz | ok, i'm trying to reproduce here in a baremetal | 08:04 |
| arxcruz | let's hope i can do it :) | 08:04 |
| *** tosky has joined #oooq | 08:05 | |
| *** tosky has quit IRC | 08:07 | |
| ykarel | arxcruz, i remember you posted some fix related to remote_src, where is it? | 08:11 |
| arxcruz | ykarel: i change my mind, because it was running locally | 08:12 |
| ykarel | running locally? you ran similar to job? | 08:12 |
| arxcruz | ykarel: my thought was: it's standalone, ansible is running from standalone so it will be created in the same machine that ansible is running... but now... | 08:16 |
| arxcruz | because i was thinking about upstream... | 08:17 |
| * arxcruz stupid | 08:17 | |
| ykarel | jobs are other than standalone as well | 08:17 |
| arxcruz | ykarel: yeah... | 08:17 |
| arxcruz | ykarel: I'm deploying right now on a baremetal | 08:17 |
| arxcruz | let me see if i can figure out what's going on | 08:17 |
| ykarel | arxcruz, ack afaikt there are atleast two issues | 08:18 |
| arxcruz | meanwhile, i'll cut my hair to became sexy | 08:18 |
| ykarel | one is remote_src and other is ignore_errors | 08:18 |
| ykarel | in the patch that caused the issue | 08:18 |
| arxcruz | yeah... i don't want to get ignore_errors on this part, it's clearly a problem and we need to fix | 08:19 |
| *** ccamacho has joined #oooq | 08:19 | |
| arxcruz | then weshay|rover will complaing it's not generating the tempest.html :P | 08:19 |
| ykarel | arxcruz, yeah but there are multiple things | 08:19 |
| ykarel | see https://review.openstack.org/#/c/639651/15/roles/validate-tempest/tasks/tempest-results.yml | 08:19 |
| ykarel | this removed ignore errors from task where it was aleardy set | 08:19 |
| ykarel | so to remove ignore_errors you need to fix all those tasks | 08:20 |
| arxcruz | ykarel: i see | 08:20 |
| arxcruz | ykarel: let the deploy finishes and i'll check what really goin on | 08:21 |
| ykarel | ack | 08:21 |
| *** ykarel is now known as ykarel|lunch | 08:21 | |
| *** amoralej|off is now known as amoralej | 08:22 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp- (3 more messages) | 08:22 |
| *** tosky has joined #oooq | 08:24 | |
| *** bogdando has joined #oooq | 08:28 | |
| *** jpena|off is now known as jpena | 08:29 | |
| *** sshnaidm|afk is now known as sshnaidm|rover | 08:42 | |
| sshnaidm|rover | marios_|ruck, o/ | 08:45 |
| marios_|ruck | o/ sshnaidm|rover | 08:47 |
| chandankumar | marios_|ruck: sshnaidm|rover \p/ | 08:49 |
| marios_|ruck | o/ chandankumar | 08:51 |
| sshnaidm|rover | chandankumar, \o | 08:51 |
| marios_|ruck | sshnaidm|rover: any news from yesterday? didn't see something on etherpad. updated it this morning for latest things am aware of (nothing new and the mistral one should hopefully be fixed in next run) | 08:52 |
| sshnaidm|rover | ykarel|lunch, did you pin mistral client in rdo? | 08:53 |
| sshnaidm|rover | marios_|ruck, seems like we have more or less stable periodic run today for master, thanks to ykarel|lunch | 08:54 |
| sshnaidm|rover | marios_|ruck, rocky containers build job fails, didn't submit bug yet | 08:54 |
| marios_|ruck | ykarel|lunch++ | 08:54 |
| hubbot1 | marios_|ruck: ykarel|lunch's karma is now 1 | 08:54 |
| marios_|ruck | ykarel++ | 08:55 |
| marios_|ruck | we really need that promotion sshnaidm|rover :D | 08:55 |
| marios_|ruck | srsly | 08:55 |
| sshnaidm|rover | marios_|ruck, oh yea | 08:55 |
| marios_|ruck | sshnaidm|rover: ack sory i missed that rocky fail via grafana or ci status? /me was reviewing code | 08:55 |
| marios_|ruck | sshnaidm|rover: ah i see there are lots of fail on rocky in grafana periodic rocky | 08:56 |
| sshnaidm|rover | marios_|ruck, both, see periodic-tripleo-centos-7-rocky-containers-build | 08:56 |
| marios_|ruck | sshnaidm|rover: missing images | 08:58 |
| marios_|ruck | sshnaidm|rover: and this is the 'old' rdo kolla build job not the new one we just merged | 08:58 |
| sshnaidm|rover | marios_|ruck, do you remember how to login into container registry? | 08:58 |
| marios_|ruck | sshnaidm|rover: did you already file the bug? | 08:58 |
| marios_|ruck | sshnaidm|rover: i didn't log in just with skopeo | 08:59 |
| sshnaidm|rover | marios_|ruck, no | 08:59 |
| marios_|ruck | sshnaidm|rover: k should i ? | 08:59 |
| sshnaidm|rover | marios_|ruck, go for it | 08:59 |
| marios_|ruck | sshnaidm|rover: k gonna collect some more pointers logs and file it in a few mins i'll add to the ether | 08:59 |
| chandankumar | sshnaidm|rover: https://tree.taiga.io/project/tripleo-ci-board/us/852 feel free to take a look this US when free! | 09:00 |
| sshnaidm|rover | chandankumar, sure | 09:00 |
| marios_|ruck | sshnaidm|rover: like this pastebin.test.redhat.com/733190 example fyi | 09:01 |
| marios_|ruck | if that's what you were after | 09:01 |
| *** brault has joined #oooq | 09:03 | |
| sshnaidm|rover | marios_|ruck, thanks | 09:04 |
| sshnaidm|rover | marios_|ruck, we also had some openshift ui for that, but didn't log in there yet.. | 09:04 |
| *** brault has quit IRC | 09:09 | |
| *** holser_ has joined #oooq | 09:10 | |
| sshnaidm|rover | marios_|ruck, arxcruz is busy with phase1 tempest failure in rocky, right arxcruz ? | 09:11 |
| *** dtantsur|afk is now known as dtantsur | 09:11 | |
| sshnaidm|rover | marios_|ruck, I think we need a bug about it too | 09:11 |
| *** rascasoft has joined #oooq | 09:12 | |
| marios_|ruck | sshnaidm|rover: ok will you file that one? | 09:12 |
| marios_|ruck | sshnaidm|rover: gonna check out queens (weshay email on friday about the import today ) | 09:12 |
| sshnaidm|rover | marios_|ruck, queens..? | 09:19 |
| marios_|ruck | sshnaidm|rover: email "Fwd: [openstack-program] 13z6 / 14z2 Imports planned for 18-Mar-19" | 09:21 |
| marios_|ruck | sshnaidm|rover: looking at http://logs.rdoproject.org/openstack-periodic-24hr/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset021-queens/7f51dec/job-output.txt.gz | 09:22 |
| marios_|ruck | looks like collect logs issue | 09:22 |
| marios_|ruck | 2019-03-17 09:00:25.973720 | primary | "cmd": "set -o pipefail &&subunit2junitxml /home/zuul/tempest/testrepository.subunit --output-to /home/zuul/tempest/tempest.xml 2>&1 >> tempest.log", | 09:23 |
| marios_|ruck | ah | 09:23 |
| *** chem has joined #oooq | 09:25 | |
| arxcruz | sshnaidm|rover: marios_|ruck what's up ? | 09:26 |
| *** ykarel|lunch is now known as ykarel | 09:27 | |
| marios_|ruck | 11:11 < sshnaidm|rover> marios_|ruck, arxcruz is busy with phase1 tempest failure in rocky, right arxcruz ? | 09:28 |
| marios_|ruck | arxcruz: are you looking at that? ^ | 09:29 |
| arxcruz | marios_|ruck: trying to reproduce it, yes | 09:29 |
| marios_|ruck | arxcruz: also looks like queens failing tempest gonna file a bug in a sec | 09:29 |
| marios_|ruck | http://logs.rdoproject.org/openstack-periodic-24hr/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset021-queens/7f51dec/logs/tempest.html.gz | 09:29 |
| arxcruz | marios_|ruck: pretty much something in neutron, but i'll take a look later | 09:32 |
| *** rfolco has joined #oooq | 09:34 | |
| ykarel | marios_|ruck, mmm i thought queens promoting today, but looks like wes udpated wrong file and some new bug introduced | 09:37 |
| marios_|ruck | ykarel: and a different error here :/ http://logs.rdoproject.org/openstack-periodic-24hr/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-queens/3707869/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz | 09:38 |
| marios_|ruck | CREATE_FAILED Resource CREATE failed: Error: resources.AllNodesDeploySteps.resources.ComputeDeployment_Step5.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non | 09:38 |
| ykarel | marios_|ruck, yup this is new bug | 09:38 |
| ykarel | but we should have a promotion for https://trunk.rdoproject.org/centos7-queens/5d/a8/5da8a104057bf9c69250a9b2b72c05d55deb7817_0d83d3bc/ | 09:39 |
| marios_|ruck | ykarel: ah there is also tempest fail here http://logs.rdoproject.org/openstack-periodic-24hr/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset021-queens/7f51dec/logs/tempest.html.gz | 09:39 |
| marios_|ruck | so we have 2 isues | 09:39 |
| ykarel | marios_|ruck, ^^ is not i promotion criteria | 09:39 |
| ykarel | fs021 | 09:39 |
| marios_|ruck | ykarel: ah ok i didn't check that yet | 09:39 |
| marios_|ruck | ykarel: cool | 09:39 |
| marios_|ruck | ykarel: but still worth a bug | 09:39 |
| ykarel | marios_|ruck, so good to get promotion of https://trunk.rdoproject.org/centos7-queens/5d/a8/5da8a104057bf9c69250a9b2b72c05d55deb7817_0d83d3bc/ | 09:40 |
| ykarel | ^^ hash is from 14th March | 09:40 |
| ykarel | and only missing job is fs030-queens | 09:40 |
| ykarel | so wes updated https://review.rdoproject.org/r/#/c/19437/ but these files are outdated afaik | 09:40 |
| ykarel | new files are CentOS7 someting | 09:40 |
| ykarel | marios_|ruck, can you fix ^^ and have queens promotion, also file bug for fs030(possible cause:- https://review.openstack.org/#/c/639187/) and fs035(possible cause:- https://review.openstack.org/#/c/641968/) | 09:45 |
| ykarel | fs035 might have fix as well | 09:46 |
| ykarel | https://review.openstack.org/#/c/641977/1, but good to rebase and see | 09:46 |
| ykarel | ok it's already on the patch that break, so seems ^^ fixing the issue | 09:47 |
| ykarel | already on top | 09:47 |
| marios_|ruck | ykarel: looking | 09:47 |
| apetrich | ykarel, so mistralclient pin worked? | 09:52 |
| apetrich | did^ | 09:52 |
| ykarel | apetrich, yes | 09:52 |
| apetrich | ykarel, cool. | 09:52 |
| apetrich | ykarel, thanks for looking after it | 09:52 |
| marios_|ruck | ykarel: ack posting the patch for the right file but filing bug first sec | 09:57 |
| marios_|ruck | ykarel: ack on 35 and the tempest one though they are both not in the critiera | 09:58 |
| * ykarel looking | 09:59 | |
| marios_|ruck | ykarel: well like the thing you pointed at man :D https://review.rdoproject.org/r/#/c/19437/1/ci-scripts/dlrnapi_promoter/config/queens.ini | 09:59 |
| marios_|ruck | ykarel: but i'll file bugs them too in a bit | 09:59 |
| ykarel | marios_|ruck, correct file is https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/dlrnapi_promoter/config/CentOS-7/queens.ini | 10:00 |
| marios_|ruck | ykarel: yeah | 10:00 |
| marios_|ruck | ykarel: thanks | 10:00 |
| ykarel | wes update wrong file | 10:00 |
| marios_|ruck | ykarel: i did it already just in git commit message now for bug :D | 10:01 |
| sshnaidm|rover | weshay|rover, you're not a rover anymore | 10:06 |
| * sshnaidm|rover is taking rover star from weshay|rover | 10:07 | |
| marios_|ruck | ykarel: sshnaidm|rover https://review.rdoproject.org/r/#/c/19441/ https://bugs.launchpad.net/tripleo/+bug/1820578 | 10:08 |
| openstack | Launchpad bug 1820578 in tripleo "[queens ]periodic-centos-7-multinode-1ctlr-featureset030-queens fails for ovn image" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 10:08 |
| ykarel | marios_|ruck, nit | 10:10 |
| ykarel | posted a comment | 10:10 |
| ykarel | marios_|ruck, and in bug good to link https://review.openstack.org/#/c/639187/ | 10:10 |
| ykarel | fs030 bug, as it's happeing post ^^ | 10:10 |
| marios_|ruck | ykarel: thanks done | 10:10 |
| ykarel | marios_|ruck, for future you can use lp#<bug number> in commit message to create links | 10:11 |
| marios_|ruck | ykarel: thanks | 10:11 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp- (3 more messages) | 10:22 |
| arxcruz | sshnaidm|rover: marios_|ruck have you guys seen this error ? | 10:27 |
| arxcruz | 18 Mar 10:31:06 ntpdate[9159]: no server suitable for synchronization found | 10:27 |
| *** udesale has quit IRC | 10:28 | |
| sshnaidm|rover | arxcruz, where is it? | 10:28 |
| arxcruz | sshnaidm|rover: in one machine i'm using to install rocky, it's on rh network | 10:28 |
| marios_|ruck | arxcruz: yeah | 10:30 |
| marios_|ruck | ykarel: is it beaker box? | 10:30 |
| ykarel | ? | 10:30 |
| marios_|ruck | ykarel: i mean i had some issue on a beaker box sec lemme get you link | 10:30 |
| arxcruz | marios_|ruck: nope, it's a machine I have for a long long time | 10:30 |
| marios_|ruck | ykarel: err sorry | 10:30 |
| marios_|ruck | ykarel: was meant for arxcruz | 10:30 |
| ykarel | ack | 10:30 |
| marios_|ruck | arxcruz: well 'rh network' anyway maybe related | 10:31 |
| arxcruz | marios_|ruck: gimme the link! :D | 10:31 |
| ykarel | iirc clock.redhat.com needs to be used inside network machines for ntp | 10:31 |
| marios_|ruck | https://tree.taiga.io/project/tripleo-ci-board/task/708?kanban-status=1447277 arxcruz see note about (beaker specific dnsservers ntpserver | 10:32 |
| arxcruz | man.. ykarel is da man! | 10:33 |
| arxcruz | marios_|ruck: thanks, how do i do that? I'm not running standalone | 10:34 |
| arxcruz | marios_|ruck: I'm running ci-script/full-deploy.sy because i want to reproduce the promote rocky job | 10:34 |
| marios_|ruck | arxcruz: well just add those to some environment file you're using | 10:34 |
| marios_|ruck | (or create a new env file and use it) | 10:35 |
| arxcruz | i have no idea where to put it :/ | 10:41 |
| marios_|ruck | chandankumar: arxcruz: can i assign it to one of you ? (tempest queens not blocking promotion) https://bugs.launchpad.net/tripleo/+bug/1820585 | 10:41 |
| openstack | Launchpad bug 1820585 in tripleo "[queens] periodic centos-ovb-1ctlr_1comp-featureset021-queens fails tempest test_qos & test_force_detach_volume" [High,Triaged] | 10:41 |
| arxcruz | yup | 10:42 |
| sshnaidm|rover | you need to use clock.redhat.com | 10:43 |
| sshnaidm|rover | arxcruz, usually you can't use external ntp from rh net | 10:43 |
| arxcruz | sshnaidm|rover: yeah, i got that, but where do i set the clock.redhat.com | 10:43 |
| sshnaidm|rover | arxcruz, depends where you get the error | 10:44 |
| arxcruz | sshnaidm|rover: i'm running bash ci-scripts/full-deploy.sh rocky rdo_trunk minimal promote | 10:44 |
| arxcruz | on my baremetal | 10:44 |
| arxcruz | no idea where to put it | 10:44 |
| *** jbadiapa has quit IRC | 10:44 | |
| *** saneax has quit IRC | 10:57 | |
| sshnaidm|rover | arxcruz, /etc/ntp.conf? | 11:00 |
| arxcruz | it seems there's an option NtpServer on heat template | 11:01 |
| arxcruz | but i don't know where exactly to put to make it work, i need to check which yml file the undercloud install is calling to edit there | 11:01 |
| sshnaidm|rover | marios_|ruck, ykarel standalone002 reports success, waiting for ovb now.. | 11:02 |
| ykarel | sshnaidm|rover, cool | 11:02 |
| marios_|ruck | arxcruz: if its undercloud you're after then you can use these maybe https://github.com/openstack/tripleo-quickstart-extras/blob/679b96f2d0eb17a6310b271b572424e376142145/roles/undercloud-deploy/templates/undercloud.conf.j2#L99-L106 | 11:03 |
| marios_|ruck | sshnaidm|rover: for which bug/promotion? | 11:03 |
| sshnaidm|rover | marios_|ruck, for master | 11:03 |
| marios_|ruck | sshnaidm|rover: we have bug? | 11:04 |
| sshnaidm|rover | marios_|ruck, but about what? | 11:04 |
| marios_|ruck | 13:02 < sshnaidm|rover> marios_|ruck, ykarel standalone002 reports success, waiting for ovb now.. | 11:04 |
| arxcruz | marios_|ruck: and what if is overcloud? | 11:04 |
| *** ratailor has quit IRC | 11:04 | |
| arxcruz | jesus, now i can't remember :( | 11:04 |
| sshnaidm|rover | marios_|ruck, it's about rerunning periodic: https://review.rdoproject.org/r/#/c/19422/ | 11:05 |
| marios_|ruck | arxcruz: i am not familiar with the script you're using arxcruz thats why i didn't reply. but somewhere it is using tripleo heat templates to deploy so... maybe just directly patch the /usr/shar/openstack-tripleo-heat-templates if its using that... | 11:05 |
| marios_|ruck | arxcruz: so worst case hard code it into <pre> reports success, waiting for ovb now.. | 11:06 |
| marios_|ruck | 13:02 <font color="#555753"><b><</b></font> ykarel<font color="#555753"><b>></b></font> sshnaidm|rover, cool | 11:06 |
| marios_|ruck | arxcruz: wow | 11:06 |
| marios_|ruck | arxcruz: into this https://github.com/openstack/tripleo-heat-templates/blob/fa3d65ff4bf4c8c8942fc22e7f0afa7abd05c109/overcloud-resource-registry-puppet.j2.yaml#L361 | 11:06 |
| sshnaidm|rover | marios_|ruck, why so htmlish | 11:06 |
| sshnaidm|rover | ykarel, so the latest result will override failure in dlrn, right? | 11:06 |
| marios_|ruck | i think i copy pasted from the chan by mistak sshnaidm|rover | 11:07 |
| ykarel | sshnaidm|rover, yes | 11:07 |
| marios_|ruck | (irssi) | 11:07 |
| *** udesale has joined #oooq | 11:45 | |
| *** jaosorior has joined #oooq | 11:51 | |
| marios_|ruck | sshnaidm|rover: you joining escalation call? | 12:00 |
| marios_|ruck | sshnaidm|rover: ah not now in half hour | 12:00 |
| *** rlandy has joined #oooq | 12:01 | |
| *** udesale has quit IRC | 12:01 | |
| *** udesale has joined #oooq | 12:02 | |
| sshnaidm|rover | amoralej, no | 12:07 |
| sshnaidm|rover | marios_|ruck, no, it's ruck that should join it | 12:07 |
| sshnaidm|rover | amoralej, oops, mistake | 12:07 |
| amoralej | :) | 12:07 |
| marios_|ruck | sshnaidm|rover: ok i was planning to anyway i thought we both join but ok np | 12:08 |
| sshnaidm|rover | marios_|ruck, let's sync after it | 12:09 |
| *** trown|outtypewww is now known as trown | 12:09 | |
| *** saneax has joined #oooq | 12:11 | |
| *** ksambor has joined #oooq | 12:16 | |
| ksambor | hi guys can you take a look here and help with creating of acceptable solution? https://review.openstack.org/#/c/640037/1/roles/validate-tempest/templates/cleanup-network.sh.j2 | 12:18 |
| weshay|rover | dam... so close on getting ovb working | 12:18 |
| weshay|rover | marios_|ruck we are familiar w/ this one :) 2019-03-18 08:34:51 | try: 1, refid: 00000000, correction: 0.000000000, skew: 0.000 | 12:19 |
| weshay|rover | dns issues in rdo? | 12:19 |
| marios_|ruck | weshay|rover: is it chrony? | 12:20 |
| marios_|ruck | or ntp issue anyway | 12:20 |
| weshay|rover | ya.. chrony.. so dns/ntp are suspect for sure | 12:21 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039 @ https://review.openstack.org/602248, (3 more messages) | 12:22 |
| weshay|rover | sshnaidm|rover arxcruz what's up w/ the dup base path for the cirros image | 12:23 |
| *** agopi__ has quit IRC | 12:23 | |
| weshay|rover | https://github.com/openstack-infra/tripleo-ci/blob/897505711b11edbc6c618d1e8d2a6574d2ca924d/toci-quickstart/config/testenv/multinode.yml#L33 | 12:23 |
| arxcruz | weshay|rover: ? | 12:24 |
| sshnaidm|rover | weshay|rover, ? | 12:24 |
| arxcruz | weshay|rover: if i recall correct, it was downloading the cirros image, adding a few minutes while we had it in the image | 12:24 |
| weshay|rover | vs https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/validate-tempest/templates/cleanup-network.sh.j2#L57 | 12:24 |
| weshay|rover | s/\/opt\/cache/files/cirros-0.3.5-x86_64-disk.img cirros/{{ tempest_test_image_path }} ? | 12:25 |
| weshay|rover | brb | 12:25 |
| weshay|rover | coffee | 12:25 |
| arxcruz | weshay|rover: no idea... it's been there for a while | 12:25 |
| sshnaidm|rover | weshay|rover, there is a patch somewhere to fix it | 12:26 |
| ksambor | sshnaidm|rover, this one? https://review.openstack.org/#/c/640037/1/roles/validate-tempest/templates/cleanup-network.sh.j2 | 12:26 |
| sshnaidm|rover | ksambor, yep, right | 12:27 |
| *** udesale has quit IRC | 12:27 | |
| *** udesale has joined #oooq | 12:28 | |
| ksambor | sshnaidm|rover, so question, there is any way to fix this template to use some dynamic value for cirros image version? how we do this in quickstart? | 12:29 |
| sshnaidm|rover | ksambor, what does mean dynamic value? | 12:32 |
| sshnaidm|rover | ksambor, if we need to move to 3.6 version, let's change this variable in defaults to point to 3.6 image | 12:32 |
| sshnaidm|rover | ksambor, better to have all jobs to use the same image | 12:33 |
| weshay|rover | and my quick look at it.. it looked like we need a path var, and image var ( with version ) | 12:34 |
| sshnaidm|rover | ksambor, I don't think we want to run different cirros version in different jobs, it breaks all maintainability | 12:34 |
| sshnaidm|rover | weshay|rover, we have both I think | 12:34 |
| weshay|rover | k | 12:34 |
| ksambor | kk, but tempest_test_image_path is provided in every testenv separately, shouldn't we centralize it sshnaidm|rover ? | 12:35 |
| sshnaidm|rover | chandankumar, can you take a look please on 039 tempest failure in rocky? https://review.openstack.org/#/c/643259/ maybe we need to port something to rocky? | 12:35 |
| sshnaidm|rover | ksambor, not sure I understand | 12:35 |
| weshay|rover | arxcruz morning.. did you see my email re: that ci.centos job | 12:35 |
| chandankumar | sshnaidm|rover: sure | 12:36 |
| sshnaidm|rover | chandankumar, it still fails on master afaik, but can be tested in rocky | 12:36 |
| ksambor | sshnaidm|rover, kk, give me a moment probably I mixed up something | 12:36 |
| sshnaidm|rover | marios_|ruck, weshay|rover ykarel ovb finished and passed, so master promotion is matter of couple hours \o/ | 12:39 |
| ykarel | sshnaidm|rover, yeah just saw | 12:39 |
| ykarel | queens also promoted | 12:39 |
| weshay|rover | very nice | 12:39 |
| arxcruz | weshay|rover: yes, and i'm working on it | 12:41 |
| arxcruz | weshay|rover: deploying the same job here to check why still not working | 12:41 |
| arxcruz | marios_|ruck: it seems -e undercloud_ntp_servers=clock.redhat.com worked :) | 12:43 |
| weshay|rover | arxcruz check the hash of tqe | 12:44 |
| weshay|rover | jenkins is peice of shit | 12:44 |
| weshay|rover | might be pulling something old | 12:44 |
| arxcruz | weshay|rover: i'm using the full-deploy.sh | 12:44 |
| arxcruz | weshay|rover: ykarel told that the hash contains both patches | 12:44 |
| sshnaidm|rover | marios_|ruck, I'm not sure I kept up with last changes to containers jobs.. Seems like in rocky containers-build job tries to pull images before it builds them. Could it be consequences of these last changes? | 12:48 |
| *** agopi__ has joined #oooq | 12:50 | |
| weshay|rover | arxcruz ah.. see ur email now.. thanks | 12:50 |
| sshnaidm|rover | rlandy, weshay|rover https://review.openstack.org/#/c/643906 | 12:50 |
| *** zul has joined #oooq | 12:50 | |
| arxcruz | weshay|rover: u need more coffee | 12:51 |
| arxcruz | i'm in my 4th cup | 12:51 |
| weshay|rover | :) | 12:53 |
| weshay|rover | marios_|ruck we might as well chat prior to the scrum | 12:57 |
| *** jpena is now known as jpena|lunch | 12:57 | |
| weshay|rover | moved up the mtg | 12:57 |
| marios_|ruck | weshay|rover: ack seen it | 12:57 |
| *** jtomasek has quit IRC | 13:03 | |
| *** jtomasek_ has joined #oooq | 13:03 | |
| arxcruz | marios_|ruck: chandankumar weshay|rover so, /var/lib/tempest{data} does exist | 13:03 |
| arxcruz | it's finishing prepare the stuff, let's see | 13:04 |
| *** jtomasek_ has quit IRC | 13:05 | |
| *** jtomasek has joined #oooq | 13:05 | |
| sshnaidm|rover | marios_|ruck, tell me when you're available please, need to sync about containers | 13:06 |
| ykarel | sshnaidm|rover, master not promoting due to fs020 failed | 13:09 |
| ykarel | missing successful jobs: [u'periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-master'] | 13:09 |
| weshay|rover | sshnaidm|rover join my blue we're chatting | 13:10 |
| sshnaidm|rover | ykarel, looking.. | 13:13 |
| ykarel | ack | 13:13 |
| weshay|rover | https://etherpad.openstack.org/p/ruckroversprint7 | 13:24 |
| weshay|rover | https://bugs.launchpad.net/tripleo/+bug/1820045 - mistral typo periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset010-master failure | 13:24 |
| openstack | Launchpad bug 1820045 in tripleo "periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset010-master times out during deploy: Mistral client fails to initialize base action" [Critical,In progress] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm) | 13:24 |
| weshay|rover | https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset010-master - this morning run (18mar19) should include the fix from https://review.rdoproject.org/r/#/c/19435 merged 3 hrs ago | 13:24 |
| weshay|rover | according to yatin is caused by https://review.openstack.org/554950 | 13:24 |
| weshay|rover | testing patch: https://review.openstack.org/643305 and https://review.openstack.org/643304 fix merged | 13:24 |
| weshay|rover | MORE https://review.rdoproject.org/r/#/c/19435/ https://review.openstack.org/#/c/643319/ | 13:24 |
| weshay|rover | arxcruz ^ | 13:24 |
| weshay|rover | arxcruz hold on that | 13:25 |
| weshay|rover | arxcruz nvrmind | 13:26 |
| arxcruz | weshay|rover: ? | 13:26 |
| marios_|ruck | o/ weshay|rover do you mean https://bugs.launchpad.net/tripleo/+bug/1820045 | 13:28 |
| openstack | Launchpad bug 1820045 in tripleo "periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset010-master times out during deploy: Mistral client fails to initialize base action" [Critical,In progress] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm) | 13:28 |
| marios_|ruck | sshnaidm|rover: ^ | 13:28 |
| *** vinaykns has joined #oooq | 13:29 | |
| weshay|rover | promoter Skipping promotion of {'timestamp': 1552885945, 'distro_hash': 'b0217950876ab5861652a169206dd93fc796b135', 'promote_name': 'tripleo-ci-testing', 'user': 'review_rdoproject_org', 'repo_url': 'https://trunk.rdoproject.org/centos7/0c/d6/0cd6b282920214d2995d1203aa0ad7d33c825b10_b0217950', 'full_hash': '0cd6b282920214d2995d1203aa0ad7d33c825b10_ | 13:35 |
| weshay|rover | b0217950', 'repo_hash': '0cd6b282920214d2995d1203aa0ad7d33c825b10_b0217950', 'commit_hash': '0cd6b282920214d2995d1203aa0ad7d33c825b10'} from tripleo-ci-testing to current-tripleo, missing successful jobs: [u'periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-master'] | 13:35 |
| *** skramaja_ has joined #oooq | 13:37 | |
| *** skramaja has quit IRC | 13:38 | |
| marios_|ruck | NEEDS BUG promotion blocker http://logs.rdoproject.org/openstack-periodic-24hr/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-queens/3707869/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz | 13:38 |
| marios_|ruck | sshnaidm|rover: ^ | 13:38 |
| weshay|rover | marios_|ruck sshnaidm|rover https://review.rdoproject.org/r/19443 | 13:40 |
| weshay|rover | sshnaidm|rover https://review.openstack.org/#/c/643187/ | 13:47 |
| sshnaidm|rover | weshay|rover, marios_|ruck http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-centos-7-rocky-containers-build/7ec234b/logs/undercloud/home/zuul/undercloud_install.log.txt.gz | 13:47 |
| marios_|ruck | https://bugs.launchpad.net/tripleo/+bug/1820574 | 13:48 |
| openstack | Launchpad bug 1820574 in tripleo "[rocky] periodic-rocky-containers-build fails for missing centos-binary-cron" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 13:48 |
| marios_|ruck | weshay|rover: sshnaidm|rover ^ | 13:48 |
| weshay|rover | marios_|ruck sshnaidm|rover I'll open bugs on the master barbican tempest failures http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-master/59b2b40/logs/tempest.html.gz | 13:54 |
| weshay|rover | and add to skip | 13:54 |
| ykarel | weshay|rover, good to fix ^^, those tests are running where barbican is not deployed | 13:55 |
| ykarel | i think barbican_tempest_plugin recently enabled | 13:55 |
| ykarel | i mean if fix is easy enough good to fix rather than skip | 13:55 |
| ykarel | as those tests must be running on barbican jobs | 13:55 |
| weshay|rover | ykarel ya.. it's good to fix I'm not suggesting it shouldn't be :) | 13:56 |
| sshnaidm|rover | who can fix barbican stuff at all? | 13:57 |
| weshay|rover | juan, harry, ada | 13:57 |
| weshay|rover | I think | 13:57 |
| ykarel | sshnaidm|rover, in that job barbican is not running, good to check that first | 13:57 |
| ykarel | and should not run tests for service, which is not running at all | 13:58 |
| ykarel | so seems framework issue | 13:58 |
| zbr | weshay|rover: re my question from planning meeting regarding collect-logs repo: can't we make it more generic? like creating a repo for hosting reusable roles instead of one "collect-logs" repo. collect-logs would be first and only role inside for start, but at least it would be much easier to add new ones. | 13:59 |
| weshay|rover | arxcruz chandankumar ^ | 13:59 |
| sshnaidm|rover | zbr, do you mean tqe? | 13:59 |
| weshay|rover | sshnaidm|rover ya.. that would be tqe | 14:00 |
| rfolco | ping scrum marios, quiquell, sshnaidm, weshay, panda, rlandy, arxcruz, mwhahaha, rfolco, chkumar, ssbarnea, kopecmartin | 14:00 |
| *** amoralej is now known as amoralej|lunch | 14:02 | |
| *** skramaja_ has quit IRC | 14:03 | |
| *** jpena|lunch is now known as jpena | 14:04 | |
| *** dtrainor has joined #oooq | 14:05 | |
| ykarel | chandankumar, arxcruz hi we are branching stein https://review.rdoproject.org/etherpad/p/stein-release-preparation | 14:07 |
| ykarel | chandankumar, arxcruz can you guys take care of tempest and tempest-plugins? | 14:08 |
| arxcruz | ykarel: sure | 14:08 |
| arxcruz | chandankumar: can you walk with me on that? I would like to re-learn it | 14:08 |
| arxcruz | i think i did only once last time | 14:08 |
| ykarel | arxcruz, Thanks | 14:11 |
| ykarel | let me know in case any help needed | 14:11 |
| ykarel | anyway chandan is good on that | 14:11 |
| arxcruz | ykarel: do you have some documentation ? | 14:22 |
| arxcruz | i'm working right now on the rocky tempest issue | 14:22 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039 @ https://review.openstack.org/602248, (3 more messages) | 14:22 |
| ykarel | arxcruz, documentation about reqcheck? arxcruz so currently step 2 needs to be done:- https://www.rdoproject.org/documentation/branching/ | 14:26 |
| ykarel | and once all tempest plugin, tempest done, step 3 can be done | 14:26 |
| arxcruz | ykarel: so, all i need to do is get the current hash from github, and add the rocky right ? | 14:30 |
| chandankumar | ykarel: i think upstream will be releasing a new tag for all temepst plugins soo for rocky | 14:35 |
| chandankumar | it might depends on tempest new tag | 14:36 |
| weshay|rover | marios_|ruck sshnaidm|rover 2019-03-18 13:52:30,243 25143 INFO promoter Promoting the container images for dlrn hash 0cd6b282920214d2995d1203aa0ad7d33c825b10 on master to current-tripleo | 14:36 |
| weshay|rover | :) | 14:36 |
| *** saneax has quit IRC | 14:50 | |
| *** amoralej|lunch is now known as amoralej | 14:54 | |
| ykarel | chandankumar, rocky? | 15:00 |
| ykarel | chandankumar, we are looking for stein | 15:00 |
| chandankumar | ykarel: sorry stein | 15:00 |
| ykarel | chandankumar, okk, so any idea what tempest tag is targetted for stein? | 15:01 |
| ykarel | and when upstream release for tempest plugin is going to happen? | 15:01 |
| chandankumar | ykarel: it might be 20, I will check tomorrow | 15:01 |
| ykarel | chandankumar, okk perfect, Thanks | 15:01 |
| zbr | weshay|rover: please send me link to compass template as i already wasted 15mins searching for it... (or just create a copy for me). thanks. | 15:10 |
| zbr | mojo search was useless and google docs seems to be unable to search in templates.... | 15:11 |
| weshay|rover | zbr is it in the epic? | 15:28 |
| zbr | weshay|rover: out of context... what epic. | 15:30 |
| weshay|rover | oh sorry | 15:30 |
| weshay|rover | sec | 15:30 |
| zbr | yeah, its "epic" in its own way... ;) | 15:31 |
| weshay|rover | zbr sending in private | 15:31 |
| *** ykarel is now known as ykarel|away | 15:41 | |
| chandankumar | weshay|rover: I will be not available on monday, need to attend this event https://www.meetup.com/Vadodara-Open-Infra-Meetup/events/258838945/ | 15:43 |
| chandankumar | it is going to be full day! | 15:44 |
| weshay|rover | k | 15:48 |
| weshay|rover | marios_|ruck sshnaidm|rover master proomoted tripleo | 15:49 |
| weshay|rover | atm w/ have 0 failing in the gate, in the cockpit.. keep an eye out | 15:49 |
| marios_|ruck | weshay|rover: ack thanks | 15:55 |
| *** weshay|rover is now known as weshay | 16:02 | |
| sshnaidm|rover | going green today | 16:10 |
| marios_|ruck | weshay: sshnaidm|rover filed that fs35 queens promotion blocker fyi https://bugs.launchpad.net/tripleo/+bug/1820667 some weird errors mistral/ironic but i don't know what the root is yet overcloud deploy fails | 16:12 |
| openstack | Launchpad bug 1820667 in tripleo "[queens] periodic-centos-ovb-3ctlr_1comp-featureset035-queens fails overcloud deploy overcloud.AllNodesDeploySteps.ComputeDeployment_Step5.0:" [High,Triaged] - Assigned to Marios Andreou (marios-b) | 16:12 |
| sshnaidm|rover | rlandy, isn't it related to external net question we talked about? ^^ | 16:15 |
| sshnaidm|rover | rlandy, or you changed only bm stuff..? | 16:16 |
| *** udesale has quit IRC | 16:21 | |
| marios_|ruck | weshay: sshnaidm|rover so for queens, looks like fs35 is the only one thats failing right now. see comment #1 https://bugs.launchpad.net/tripleo/+bug/1820667 | 16:22 |
| openstack | Launchpad bug 1820667 in tripleo "[queens] periodic-centos-ovb-3ctlr_1comp-featureset035-queens fails overcloud deploy overcloud.AllNodesDeploySteps.ComputeDeployment_Step5.0:" [High,Triaged] - Assigned to Marios Andreou (marios-b) | 16:22 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039 @ https://review.openstack.org/602248, (3 more messages) | 16:22 |
| rlandy | sshnaidm|rover: not sure what you are referring to? | 16:23 |
| rlandy | I changed this ... | 16:23 |
| sshnaidm|rover | rlandy, just a dejavu about ipv6 services unreachable, may be not related | 16:24 |
| weshay | sshnaidm|rover marios_|ruck https://review.rdoproject.org/r/#/c/19446/1/ci-scripts/dlrnapi_promoter/config/CentOS-7/master.ini | 16:25 |
| sshnaidm|rover | rlandy, we talked about having external network ipv4 in ipv6 jobs, not sure about what it was | 16:25 |
| marios_|ruck | weshay: what about it | 16:25 |
| rlandy | sshnaidm|rover: I worked around the issue with https://review.openstack.org/#/c/642924/7/roles/undercloud-deploy/templates/hieradata-overrides-t-h-t-undercloud.yaml.j2 | 16:25 |
| weshay | marios_|ruck sshnaidm|rover let's merge that.. | 16:25 |
| weshay | so we don't over promote | 16:25 |
| marios_|ruck | weshay: oh sorry its a review ;D i thought it was github /me needs screen break | 16:25 |
| marios_|ruck | revieweing | 16:25 |
| rlandy | sshnaidm|rover: undercloud_external_network resolves to ipv6 for fs035 | 16:26 |
| sshnaidm|rover | weshay, commented | 16:27 |
| rlandy | but that did not work for routing | 16:27 |
| weshay | sshnaidm|rover there is one change, 1 comment | 16:27 |
| weshay | sshnaidm|rover what chu talkin bout willis? | 16:28 |
| sshnaidm|rover | weshay, I see three changes | 16:28 |
| weshay | 2 are comments, sorry, 1 change | 16:28 |
| chandankumar | sshnaidm|rover: can you pass me the nova join test review, not able to find it | 16:28 |
| chandankumar | ? | 16:28 |
| ykarel|away | marios_|ruck, sshnaidm|rover, i remember i mentioned the reasoning for fs035 failure, seems to be missed | 16:28 |
| marios_|ruck | weshay: sshnaidm|rover headline is fs20 restore right? | 16:29 |
| * ykarel|away rechecking | 16:29 | |
| marios_|ruck | ykarel|away: you mean you know the root cause for fs35 queens bug? | 16:29 |
| marios_|ruck | weshay: sshnaidm|rover i mean in https://review.rdoproject.org/r/#/c/19446/1/ci-scripts/dlrnapi_promoter/config/CentOS-7/master.ini | 16:29 |
| ykarel|away | marios_|ruck, yes | 16:29 |
| ykarel|away | marios_|ruck, can you fix ^^ and have queens promotion, also file bug for fs030(possible cause:- https://review.openstack.org/#/c/639187/) and fs035(possible cause:- https://review.openstack.org/#/c/641968/) | 16:29 |
| ykarel|away | <ykarel> fs035 might have fix as well | 16:29 |
| ykarel|away | <ykarel> https://review.openstack.org/#/c/641977/1, but good to rebase and see | 16:29 |
| ykarel|away | <ykarel> ok it's already on the patch that break, so seems ^^ fixing the issue | 16:29 |
| ykarel|away | <ykarel> already on top | 16:29 |
| sshnaidm|rover | chandankumar, for rocky: https://review.openstack.org/#/c/643259/ | 16:29 |
| sshnaidm|rover | chandankumar, for master, but might not work now: https://review.openstack.org/#/c/639324/ | 16:29 |
| sshnaidm|rover | chandankumar, that's why I | 16:29 |
| sshnaidm|rover | m trying in rocky | 16:29 |
| * chandankumar takes a look@ | 16:30 | |
| * chandankumar loves weechat customization | 16:30 | |
| marios_|ruck | ykarel|away: cool so you mean this https://review.openstack.org/#/c/641977/1 queens cherrypick might fix | 16:30 |
| marios_|ruck | ykarel|away: thanks adding on the bug and etherpad | 16:30 |
| ykarel|away | marios_|ruck, yes as per me, you can see the same symptoms in fs035 openstack-check job | 16:31 |
| ykarel|away | the bug is hitting not just promotion job | 16:31 |
| chandankumar | sshnaidm|rover: tempest failure looks interesting, somethign is not in the list as expected | 16:32 |
| chandankumar | sshnaidm|rover: I can dig it | 16:32 |
| marios_|ruck | sshnaidm|rover: looking | 16:33 |
| sshnaidm|rover | chandankumar, I wonder if we have everything in rocky for that to work | 16:33 |
| sshnaidm|rover | chandankumar, so it is not branch related | 16:33 |
| chandankumar | sshnaidm|rover: nope | 16:34 |
| ykarel|away | marios_|ruck, https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035&branch=stable%2Fqueens | 16:34 |
| ykarel|away | only passing in that patch, else failing since 14th | 16:34 |
| ykarel|away | when the related patch merged | 16:34 |
| chandankumar | sshnaidm|rover: weshay regarding collect-logs https://tree.taiga.io/project/tripleo-ci-board/us/852 -> can I create the task list if DOD/design is ok? | 16:34 |
| marios_|ruck | sshnaidm|rover: why not i mean its in the promotion criteria no? https://github.com/rdo-infra/ci-config/blob/f964b950ce8a6d91dd7a549a2daa377ead50fafd/ci-scripts/dlrnapi_promoter/config/CentOS-7/master.ini#L15 | 16:35 |
| marios_|ruck | https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master | 16:35 |
| weshay | chandankumar sure, I'll review it | 16:35 |
| marios_|ruck | sshnaidm|rover: or you mean because it has green run in builds ^ | 16:35 |
| marios_|ruck | thank you ykarel|away | 16:35 |
| sshnaidm|rover | marios_|ruck, well, now it doesn't fail, seems like random error | 16:36 |
| sshnaidm|rover | marios_|ruck, I mean it doesn't block promotion currently, but should be fixed of course | 16:36 |
| weshay | rlandy gpg keyserver is down today.. lolz https://keys.fedoraproject.org/ | 16:36 |
| marios_|ruck | sshnaidm|rover: ack noted coment #1 and removed tag | 16:37 |
| rlandy | weshay: lol - maybe we should take the hint that it's not meant to be?? | 16:38 |
| weshay | right | 16:38 |
| chandankumar | weshay: ah, fedoraproject they just migrated openstack env from rhos-7 to rhos-13 :-) | 16:39 |
| chandankumar | but they donot like openstack | 16:39 |
| weshay | heh | 16:40 |
| sshnaidm|rover | rlandy, did you want to talk about https://review.openstack.org/#/c/643906/ ? | 16:41 |
| rlandy | sshnaidm|rover: just wanted to raise the concern | 16:41 |
| rlandy | for example, project keys. | 16:41 |
| rlandy | even if you call the job periodic, it won't help if check pipeline is not post-review | 16:42 |
| rlandy | that may only come up in internal | 16:42 |
| rlandy | and called the job periodic means it should kick from post-review | 16:42 |
| rlandy | so, in essence if we don;t do stupid naming it will be ok | 16:42 |
| chandankumar | one stupid question, http://logs.openstack.org/06/643906/1/check/tripleo-ci-centos-7-scenario001-standalone/0bd29d1/logs/ -> do I need to use reproducer-quickstart.sh or reproducer-zuul-based-quickstart.sh for the new workflow? | 16:42 |
| rlandy | idk - I guess I just wanted to raise that | 16:43 |
| rlandy | as long as team understand possible problems, it's ok | 16:43 |
| sshnaidm|rover | rlandy, not sure I understand - what is post-review? | 16:43 |
| *** brault has joined #oooq | 16:45 | |
| rlandy | sshnaidm|rover: https://zuul-ci.org/docs/zuul/user/config.html#attr-pipeline.post-review | 16:45 |
| rlandy | ^^ we use that in internal zuul | 16:45 |
| rlandy | I see it's not defined here ... https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/zuul.d/tripleo.yaml | 16:46 |
| rlandy | it's here though ... https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/zuul.d/pipelines.yaml#L141 | 16:47 |
| sshnaidm|rover | rlandy, you concern is if we call some non-periodic job as "periodic" job? | 16:47 |
| rlandy | sshnaidm|rover: no - if we call a periodic job from check that relies on it being in a post-review pipeline | 16:48 |
| sshnaidm|rover | rlandy, this is our pipeline: https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/zuul.d/upstream.yaml#L60 | 16:48 |
| sshnaidm|rover | rlandy, I don't really understand the problem.. bluejeans? | 16:49 |
| sshnaidm|rover | rlandy, https://bluejeans.com/u/sshnaidm/ | 16:49 |
| *** brault has quit IRC | 16:50 | |
| marios_|ruck | ttyl folks sshnaidm|rover weshay /me off | 16:57 |
| weshay | marios_|ruck thanks man.. good night | 16:58 |
| rfolco | weshay, if still willing to do bug triage https://bluejeans.com/5878458097 | 17:01 |
| weshay | k ya.. 2 min | 17:01 |
| *** trown is now known as trown|lunch | 17:03 | |
| rlandy | sshnaidm|rover: wrt masquerade network and ipv6, pls review https://review.openstack.org/#/c/643140 and https://review.openstack.org/#/c/642924/ ( see notes added to the review with results) | 17:06 |
| rlandy | weshay: ^^ reviews required for bm ... thanks | 17:07 |
| ykarel|away | weshay, sshnaidm|rover, can you review https://review.openstack.org/#/c/643857/, it's needed for phase1 for all releases | 17:12 |
| ykarel|away | a bug was there, but earlier fix missed it | 17:12 |
| *** bogdando has quit IRC | 17:13 | |
| * ykarel|away out | 17:14 | |
| weshay | rfolco https://bugs.launchpad.net/tripleo/+bug/1818860 | 17:18 |
| openstack | Launchpad bug 1818860 in tripleo "tempest.api.network.test_ports.PortsIpV6TestJSON.test_update_port_with_two_security_groups_and_extra_attributes failing with SecurityGroupNotFound" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz) | 17:18 |
| *** ykarel|away has quit IRC | 17:18 | |
| arxcruz | rfolco: weshay i'm not being able to reproduce it locally... | 17:22 |
| arxcruz | https://tree.taiga.io/project/tripleo-ci-board/issue/829?kanban-status=2027734 | 17:22 |
| *** kopecmartin is now known as kopecmartin|off | 17:22 | |
| weshay | arxcruz can you double double check centos.ci has the right hash from qs/qse | 17:23 |
| arxcruz | weshay: i'm getting the current-tripleo, but yeah, i'll double check | 17:23 |
| arxcruz | weshay: i mean the https://bugs.launchpad.net/tripleo/+bug/1818860 | 17:24 |
| openstack | Launchpad bug 1818860 in tripleo "tempest.api.network.test_ports.PortsIpV6TestJSON.test_update_port_with_two_security_groups_and_extra_attributes failing with SecurityGroupNotFound" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz) | 17:24 |
| arxcruz | i'm not being able to reproduce this one | 17:24 |
| weshay | rfolco https://bugs.launchpad.net/tripleo/+bug/1814123 | 17:28 |
| openstack | Launchpad bug 1814123 in tripleo "rocky ovb test_snapshot_create_offline_delete_online failing" [Low,Confirmed] - Assigned to Quique Llorente (quiquell) | 17:28 |
| zbr | weshay: safe to wf https://review.openstack.org/#/c/613795/ (ara related) | 17:35 |
| weshay | rfolco https://review.openstack.org/#/c/635478/ | 17:39 |
| *** amoralej is now known as amoralej|off | 17:41 | |
| sshnaidm|rover | weshay, reproducer seems to be broken for me, will work on it tomorrow.. | 17:42 |
| weshay | sshnaidm|rover :( | 17:51 |
| *** chandankumar is now known as raukadah | 17:51 | |
| *** trown|lunch is now known as trown | 18:00 | |
| *** jpena is now known as jpena|off | 18:06 | |
| *** jtomasek has quit IRC | 18:18 | |
| *** jtomasek has joined #oooq | 18:18 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039 @ https://review.openstack.org/602248, (3 more messages) | 18:22 |
| *** dtantsur is now known as dtantsur|afk | 18:23 | |
| rfolco | weshay, tripleo projects are not branched yet, there is no stable/stein. How can we test our triggers w/ a DNM change for stein on these projects ? | 18:47 |
| weshay | rfolco raise it at the #tripleo mtg tomorrow | 18:47 |
| weshay | put it on the agenda | 18:47 |
| rfolco | weshay, deal | 18:47 |
| rfolco | thx | 18:48 |
| *** saneax has joined #oooq | 18:51 | |
| *** saneax has quit IRC | 19:00 | |
| *** sshnaidm|rover is now known as sshnaidm|afk | 19:13 | |
| *** jfrancoa has quit IRC | 19:28 | |
| zbr | rfolco: weshay: i added a poc about stein on tq at https://review.openstack.org/#/c/643963/ | 20:05 |
| zbr | lets use "oooq/branching" topic for this subject. | 20:05 |
| weshay | fine by me :) | 20:10 |
| weshay | thanks for getting in there | 20:10 |
| *** irclogbot_0 has joined #oooq | 20:12 | |
| zbr | weshay: please have a look at it and add comments, there or on the ticket, as I am not sure exactly what are the next steps | 20:13 |
| *** holser_ has quit IRC | 20:17 | |
| weshay | \0/ | 20:18 |
| weshay | http://dashboard-ci.tripleo.org/d/cEEjGFFmz/cockpit?orgId=1&panelId=61&fullscreen | 20:18 |
| weshay | 0 failed gate jobs in 3 days | 20:18 |
| zbr | weshay: "24 hours" for that graph. | 20:19 |
| weshay | it's 3 days | 20:20 |
| zbr | 3 graphs do use fixed time, with their top right corner reporting always "24h". | 20:21 |
| zbr | the cyan text "Last 24 hours" is wrong? | 20:22 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039 @ (2 more messages) | 20:22 |
| weshay | zbr go find out :P | 20:22 |
| weshay | read the source luke | 20:22 |
| zbr | not enough time | 20:23 |
| *** irclogbot_0 has quit IRC | 20:25 | |
| *** irclogbot_0 has joined #oooq | 20:27 | |
| *** jfrancoa has joined #oooq | 20:52 | |
| *** holser_ has joined #oooq | 20:53 | |
| *** trown is now known as trown|outtypewww | 20:59 | |
| *** irclogbot_0 has quit IRC | 21:05 | |
| *** irclogbot_0 has joined #oooq | 21:06 | |
| *** jfrancoa has quit IRC | 21:15 | |
| *** irclogbot_0 has quit IRC | 21:26 | |
| *** irclogbot_0 has joined #oooq | 21:28 | |
| *** vinaykns has quit IRC | 21:58 | |
| *** rfolco has quit IRC | 22:04 | |
| *** jtomasek has quit IRC | 22:05 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039 @ (2 more messages) | 22:22 |
| *** holser_ has quit IRC | 22:50 | |
| *** tosky has quit IRC | 23:48 | |
| *** holser_ has joined #oooq | 23:57 | |
| *** agopi__ has quit IRC | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!