| *** jaganathan has joined #oooq | 00:00 | |
| *** jaganathan_ has joined #oooq | 00:00 | |
| *** jaganathan has quit IRC | 00:01 | |
| *** Goneri has quit IRC | 00:01 | |
| *** jaganathan_ has quit IRC | 00:05 | |
| *** jaganathan_ has joined #oooq | 00:05 | |
| *** saneax has quit IRC | 00:08 | |
| *** jaganathan__ has joined #oooq | 00:22 | |
| *** jaganathan_ has quit IRC | 00:22 | |
| *** jaganathan__ has quit IRC | 00:34 | |
| hubbot1 | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario009-multinode-oooq, legacy-tripleo-ci-centos-7-containers-multinode-upgrades-pike-branch, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, legacy- (1 more message) | 01:40 |
|---|---|---|
| *** ykarel|away has joined #oooq | 02:53 | |
| *** apetrich_things_ has joined #oooq | 02:55 | |
| *** ykarel|away has quit IRC | 02:57 | |
| *** ykarel|away has joined #oooq | 02:58 | |
| *** apetrich_ has quit IRC | 02:58 | |
| *** zul has joined #oooq | 03:02 | |
| *** ykarel|away has quit IRC | 03:31 | |
| hubbot1 | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-queens, tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci- (2 more messages) | 03:40 |
| *** ykarel|away has joined #oooq | 03:55 | |
| *** udesale has joined #oooq | 04:01 | |
| *** ykarel|away has quit IRC | 04:52 | |
| *** jaganathan__ has joined #oooq | 04:57 | |
| *** pgadiya has joined #oooq | 05:09 | |
| *** pgadiya has quit IRC | 05:09 | |
| *** ykarel|away has joined #oooq | 05:11 | |
| *** skramaja has joined #oooq | 05:21 | |
| *** apetrich_things_ has quit IRC | 05:30 | |
| *** dtrainor_ has joined #oooq | 05:30 | |
| *** dtrainor has quit IRC | 05:32 | |
| hubbot1 | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-queens, tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci- (2 more messages) | 05:40 |
| *** ratailor has joined #oooq | 05:42 | |
| *** quiquell has joined #oooq | 05:47 | |
| *** quiquell is now known as quiquell|rover | 05:47 | |
| quiquell|rover | chkumar|ruck: Goor morning | 05:47 |
| *** saneax has joined #oooq | 06:09 | |
| quiquell|rover | promotion jobs are enqueue | 06:12 |
| *** gkadam has joined #oooq | 06:25 | |
| chkumar|ruck | quiquell|rover: Good morning | 07:09 |
| chkumar|ruck | quiquell|rover: jobs are still in queue state | 07:13 |
| chkumar|ruck | from very long time | 07:13 |
| *** ykarel|away is now known as ykarel | 07:14 | |
| quiquell|rover | chkumar|ruck: RDO enqueue time is bad | 07:15 |
| quiquell|rover | also upstream queue is bad | 07:16 |
| *** tesseract has joined #oooq | 07:18 | |
| *** ykarel is now known as ykarel|lunch | 07:26 | |
| hubbot1 | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-queens, tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci- (2 more messages) | 07:40 |
| *** apetrich has joined #oooq | 07:54 | |
| *** jaganathan__ has quit IRC | 08:08 | |
| *** apetrich has quit IRC | 08:10 | |
| *** apetrich has joined #oooq | 08:11 | |
| *** ykarel|lunch is now known as ykarel | 08:19 | |
| *** apetrich has quit IRC | 08:24 | |
| *** bogdando has joined #oooq | 08:25 | |
| *** apetrich has joined #oooq | 08:25 | |
| *** apetrich has quit IRC | 08:25 | |
| *** apetrich has joined #oooq | 08:26 | |
| *** apetrich has quit IRC | 08:27 | |
| *** apetrich has joined #oooq | 08:28 | |
| *** apetrich has quit IRC | 08:29 | |
| *** apetrich has joined #oooq | 08:29 | |
| *** apetrich has quit IRC | 08:31 | |
| *** apetrich has joined #oooq | 08:32 | |
| *** apetrich has quit IRC | 08:45 | |
| *** apetrich has joined #oooq | 08:46 | |
| *** apetrich has quit IRC | 08:49 | |
| *** dtantsur|afk is now known as dtantsur | 09:05 | |
| chkumar|ruck | quiquell|rover: can we add this to alerts? https://bugs.launchpad.net/tripleo/+bug/1792091 | 09:12 |
| openstack | Launchpad bug 1792091 in tripleo "overcloud-prep-containers not executing at featureset016" [Critical,Triaged] - Assigned to Quique Llorente (quiquell) | 09:12 |
| quiquell|rover | chkumar|ruck: I think it's invalid, red herring | 09:13 |
| quiquell|rover | chkumar|ruck: https://review.openstack.org/#/c/573476/ | 09:14 |
| quiquell|rover | But this is just before duration increases | 09:15 |
| chkumar|ruck | quiquell|rover: yes | 09:16 |
| quiquell|rover | it's around 29-31 or aughust when it start to increase | 09:16 |
| quiquell|rover | august | 09:16 |
| quiquell|rover | Maybe this container-prepare-workflow increase the time | 09:16 |
| chkumar|ruck | might be possible | 09:17 |
| quiquell|rover | chkumar|ruck: Do you know who we can ask apart from steve baker ? | 09:18 |
| chkumar|ruck | quiquell|rover: may be James Slagle or steven hardy | 09:20 |
| chkumar|ruck | but both might be at PTG | 09:21 |
| quiquell|rover | damn... | 09:21 |
| hubbot1 | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario009-multinode-oooq, legacy-tripleo-ci-centos-7-containers-multinode-upgrades-pike-branch, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, legacy- (1 more message) | 09:40 |
| *** d0ugal has quit IRC | 09:52 | |
| *** udesale has quit IRC | 10:30 | |
| *** d0ugal has joined #oooq | 10:41 | |
| *** d0ugal has quit IRC | 10:46 | |
| *** d0ugal has joined #oooq | 10:47 | |
| *** ratailor has quit IRC | 11:23 | |
| chkumar|ruck | quiquell|rover: jobs are started moving from queued to running | 11:24 |
| quiquell|rover | chkumar|ruck: ack | 11:25 |
| *** apetrich has joined #oooq | 11:28 | |
| hubbot1 | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/567224, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario009-multinode-oooq, tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates @ (1 more message) | 11:40 |
| weshay | quiquell|rover, chkumar|ruck you guys hanging in? | 11:44 |
| chkumar|ruck | weshay: sensei needs blessing here https://review.openstack.org/#/c/599355/ | 11:44 |
| quiquell|rover | weshay: kind of, promotion jobs are kicking in, and looks like timeouts are related to not having promotions, so containers need to be build | 11:45 |
| weshay | built or updated | 11:46 |
| quiquell|rover | weshay: yep | 11:46 |
| *** d0ugal has quit IRC | 11:47 | |
| weshay | quiquell|rover, we're only at 6days on master | 11:47 |
| chkumar|ruck | weshay: needs +w on that | 11:48 |
| chkumar|ruck | in order to pass fs020 queens job for promotion | 11:48 |
| weshay | chkumar|ruck, what job is that fixing man? | 11:48 |
| weshay | chkumar|ruck, I see several queens jobs failing | 11:49 |
| *** panda|ptg has joined #oooq | 11:50 | |
| chkumar|ruck | weshay: https://bugs.launchpad.net/tripleo/+bug/1791298 | 11:51 |
| openstack | Launchpad bug 1791298 in tripleo "queens + featureset020 failing at promoting icroversion header 'openstack-api-version' with value 'volume 3.0' does not" [Critical,In progress] - Assigned to Quique Llorente (quiquell) | 11:51 |
| quiquell|rover | panda|ptg: How you doing | 11:51 |
| weshay | chkumar|ruck, update the commit message | 11:51 |
| chkumar|ruck | weshay: actually the fix was proposed long time ago till the bug appears in CI | 11:51 |
| quiquell|rover | panda|ptg: I am doing a little refactor on https://review.openstack.org/#/c/601601/ | 11:51 |
| panda|ptg | quiquell|rover: what are you refactoring on ? | 11:51 |
| quiquell|rover | panda|ptg: Trying to reduce duplication using include: | 11:52 |
| panda|ptg | aah optimization | 11:53 |
| weshay | lol | 11:53 |
| panda|ptg | "we don't do that here" | 11:53 |
| weshay | lolz | 11:53 |
| weshay | panda|ptg, quiquell|rover there is a 50% chance alex -2's the patch | 11:53 |
| quiquell|rover | weshay: ack, then I forget about optimizing it | 11:54 |
| chkumar|ruck | weshay: done | 11:54 |
| quiquell|rover | weshay: btw I have change importance of this https://bugs.launchpad.net/tripleo/+bug/1790489 | 11:54 |
| openstack | Launchpad bug 1790489 in tripleo "ERROR heat.engine.resource ResourceFailure: Error: resources.ControllerDeployment_Step1.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 6" [High,New] - Assigned to Quique Llorente (quiquell) | 11:54 |
| weshay | ya.. the issue is w/ infra and we're coding around infa bs | 11:55 |
| quiquell|rover | weshay: doesn't make sense to expense CIX time with it | 11:55 |
| weshay | quiquell|rover, ocata is VERY low priority atm | 11:56 |
| panda|ptg | weshay: you distract him this mornin while I use his laptop to +2 it. | 11:56 |
| weshay | panda|ptg, sounds like a plan | 11:56 |
| weshay | quiquell|rover, chkumar|ruck focus on a master promotion please | 11:57 |
| weshay | quiquell|rover, chkumar|ruck I think we can drop ocata jobs | 11:59 |
| chkumar|ruck | weshay: i am infaoir of dropping | 11:59 |
| weshay | but best to signal that to the openstack-dev[tripleo] first | 11:59 |
| quiquell|rover | weshay: also this, don't thing is critical https://bugs.launchpad.net/tripleo/+bug/1791053 | 11:59 |
| openstack | Launchpad bug 1791053 in tripleo "delorean: Get the non-root user UI fails after user is created" [High,In progress] - Assigned to Sorin Sbarnea (ssbarnea) | 11:59 |
| weshay | quiquell|rover, is that fixed? https://ci.centos.org/view/rdo/view/tripleo-gate/ | 12:01 |
| *** jtomasek has joined #oooq | 12:01 | |
| *** jtomasek has quit IRC | 12:01 | |
| weshay | rfolco, ping | 12:01 |
| rfolco | weshay, hi | 12:01 |
| weshay | rfolco, hey buddy | 12:02 |
| rfolco | morning | 12:02 |
| quiquell|rover | weshay: this looks good https://ci.centos.org/view/rdo/view/tripleo-gate/job/tripleo-quickstart-extras-gate-master-delorean-quick-basic/ | 12:02 |
| weshay | rfolco, can you please help us w/ https://trello.com/c/TVsZ3Ut6/877-clean-up-rdo-sf-legacy-code-after-zuulv3-migration | 12:02 |
| weshay | moving that card forward | 12:02 |
| rfolco | weshay, sure | 12:03 |
| chkumar|ruck | weshay: I will do that | 12:09 |
| *** udesale has joined #oooq | 12:21 | |
| quiquell|rover | panda|ptg: Added retries to retrieve 'id' from stack user | 12:24 |
| quiquell|rover | panda|ptg: https://review.openstack.org/#/c/602015/ | 12:24 |
| quiquell|rover | what do you think ? | 12:24 |
| weshay | reviews https://review.openstack.org/#/c/518043/ | 12:26 |
| *** jtomasek has joined #oooq | 12:26 | |
| *** jtomasek has quit IRC | 12:28 | |
| *** jtomasek has joined #oooq | 12:30 | |
| weshay | chkumar|ruck, you should be on the program call | 12:32 |
| *** rlandy has joined #oooq | 12:34 | |
| *** jtomasek has quit IRC | 12:35 | |
| weshay | chkumar|ruck, you join? | 12:36 |
| weshay | quiquell|rover, ? | 12:36 |
| dmellado | quiquell|rover: chkumar|ruck weshay any news on https://bugs.launchpad.net/tripleo/+bug/1791637 | 12:37 |
| openstack | Launchpad bug 1791637 in tripleo "tempest kuryr plugin ImportError: No module named openshift" [Critical,In progress] - Assigned to chandan kumar (chkumar246) | 12:37 |
| quiquell|rover | weshay: Do we have it today, there is program call | 12:37 |
| weshay | quiquell|rover, ya.. chkumar|ruck should be attending it | 12:37 |
| dmellado | I keep saying that for me it wouldn't be a critical bug at all | 12:37 |
| quiquell|rover | dmellado: waitting for promotions, RDO zuul queue was huge | 12:37 |
| weshay | quiquell|rover, however if he is not.. can you jump on | 12:38 |
| quiquell|rover | weshay: sure | 12:38 |
| dmellado | weshay: o/ how's the PTG, are you there this time as well? | 12:38 |
| weshay | dmellado, ya.. it's in denver so | 12:38 |
| weshay | I'm about 1/2 mile | 12:39 |
| weshay | away | 12:39 |
| dmellado | yeah, that's why I asked xD | 12:39 |
| weshay | dmellado, how r u? | 12:39 |
| quiquell|rover | weshay: nobody there | 12:39 |
| weshay | quiquell|rover, hrm | 12:39 |
| dmellado | weshay: cool, I had to skip the trains this time as I've an oncoming baby | 12:39 |
| weshay | quiquell|rover, dbecker just pinged me | 12:39 |
| quiquell|rover | Ahh thats tomorrows | 12:39 |
| dmellado | and it collided but a lotsa checkups, but all good! | 12:39 |
| quiquell|rover | weshay: I don't have seals today at my cal, do you have the bluejeans id ? | 12:40 |
| weshay | quiquell|rover, https://bluejeans.com/7253947007/ | 12:40 |
| weshay | chkumar|ruck, you have prg call duty next week | 12:40 |
| quiquell|rover | weshay: Do I have to fill in the doc ? | 12:44 |
| weshay | quiquell|rover, yes sir.. | 12:44 |
| weshay | quiquell|rover, for rocky.. | 12:44 |
| weshay | looks green.. you can note the rdo-cloud outtages etc.. if you want | 12:45 |
| weshay | http://rhos-release.virt.bos.redhat.com:3030/rhosp | 12:45 |
| weshay | quiquell|rover, also note that rdo p2 rocky is down due to hardware | 12:45 |
| rlandy | weshay: thanks for rebasing those patches | 12:47 |
| chkumar|ruck | weshay: not today? | 12:47 |
| rlandy | will try for another round of reviews and merge today | 12:47 |
| weshay | chkumar|ruck, sorry? | 12:47 |
| chkumar|ruck | weshay: I was asking we are not covering prg call today? | 12:47 |
| chkumar|ruck | weshay: for next week, I can do that | 12:48 |
| quiquell|rover | chkumar|ruck: For pike what issues we have ? | 12:48 |
| quiquell|rover | chkumar|ruck: Sames as queens ? | 12:49 |
| chkumar|ruck | quiquell|rover: https://review.openstack.org/601522 | 12:49 |
| rlandy | panda|ptg: sshnaidm|afk: marios: rfolco: requesting reviews on the four patches on https://trello.com/c/TVsZ3Ut6/877-clean-up-rdo-sf-legacy-code-after-zuulv3-migration?menu=filter&filter=label:Sprint%2018%20CI | 12:49 |
| *** gkadam has quit IRC | 12:49 | |
| chkumar|ruck | quiquell|rover: yes currnetly same as queens | 12:49 |
| rlandy | actually make that 5 patches | 12:49 |
| rlandy | including the change to the reproducer | 12:49 |
| weshay | chkumar|ruck, GET ON THE CALL | 12:50 |
| rfolco | rlandy, I am looking at these now. Quick question... where do we use /tmp/deployed_nodes.yml | 12:50 |
| quiquell|rover | chkumar|ruck: So we are will not have the promotion | 12:50 |
| rlandy | we pass it to quickstart | 12:50 |
| quiquell|rover | chkumar|ruck: Fill-in if you miss something in the program doc https://docs.google.com/document/d/13StMviF7pnXve42F2vigeZoopoofEtqOyZj8CxVpxko/edit | 12:55 |
| quiquell|rover | At Delivery Teams -> OOOCI | 12:55 |
| weshay | quiquell|rover, chkumar|ruck note.. this is a program call for rocky | 12:56 |
| weshay | quiquell|rover, chkumar|ruck there are times where we have to report on two releases | 12:57 |
| quiquell|rover | weshay: ups, so we are good there | 12:57 |
| weshay | however that is rare | 12:57 |
| weshay | quiquell|rover, chkumar|ruck right.. so your job is to report on the rocky pipeline | 12:57 |
| weshay | rocky primarily... | 12:57 |
| weshay | right.. I think it's good to note.. we're having infra issues etc.. | 12:58 |
| weshay | but they won't care about master or pike etc.. to be clear | 12:58 |
| weshay | quiquell|rover, chkumar|ruck make sense? | 12:58 |
| chkumar|ruck | weshay: yes | 12:58 |
| weshay | quiquell|rover, chkumar|ruck thanks | 12:59 |
| chkumar|ruck | quiquell|rover: we need to highlight status to Green as it is for ROCKy | 12:59 |
| weshay | chkumar|ruck, chkumar|ruck feel free at the end of the call to say you joined and rocky is green | 13:00 |
| weshay | briefly | 13:00 |
| quiquell|rover | weshay: ack, chkumar|ruck you do ? | 13:00 |
| chkumar|ruck | quiquell|rover: sure I can do | 13:00 |
| quiquell|rover | chkumar|ruck: I can do program next week | 13:00 |
| quiquell|rover | chkumar|ruck: I drop now | 13:01 |
| weshay | meh.. | 13:03 |
| weshay | no worries | 13:03 |
| *** agopi|brb has quit IRC | 13:03 | |
| chkumar|ruck | quiquell|rover: the call just ended | 13:03 |
| quiquell|rover | chkumar|ruck: ack | 13:04 |
| panda|ptg | weshay: will you make it to breakfast ? | 13:07 |
| weshay | panda|ptg, I'll be there in 30min or so | 13:08 |
| weshay | panda|ptg, don't wait to eat :) | 13:08 |
| weshay | I'll join you for coffee, I have some tea for you | 13:08 |
| quiquell|rover | chkumar|ruck: Have to drop earlier today | 13:09 |
| chkumar|ruck | quiquell|rover: ack | 13:09 |
| quiquell|rover | chkumar|ruck: Let's see if we have promotions in a few hours | 13:09 |
| *** d0ugal has joined #oooq | 13:10 | |
| myoung | chkumar|ruck: re: rdo2, rocky, seeing on the fs20 job failing during pip install https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/periodic-rocky-rdo_trunk-featureset020-1ctlr_1comp_64gb/1/console | 13:10 |
| myoung | 00:24:05.918 Collecting tripleo-quickstart-extras from git+https://git.openstack.org/openstack/tripleo-quickstart-extras/#egg=tripleo-quickstart-extras (from -r quickstart-extras-requirements.txt (line 1)) | 13:10 |
| myoung | 00:24:05.919 Cloning https://git.openstack.org/openstack/tripleo-quickstart-extras/ to /tmp/pip-install-X3_Jph/tripleo-quickstart-extras | 13:10 |
| myoung | 00:24:05.920 Collecting tripleo-ha-utils from git+https://git.openstack.org/openstack/tripleo-ha-utils/#egg=tripleo-ha-utils (from -r quickstart-extras-requirements.txt (line 2)) | 13:10 |
| myoung | 00:24:05.921 Cloning https://git.openstack.org/openstack/tripleo-ha-utils/ to /tmp/pip-install-X3_Jph/tripleo-ha-utils | 13:10 |
| myoung | 00:24:05.922 Collecting browbeat from git+https://git.openstack.org/openstack/browbeat/#egg=browbeat (from -r quickstart-extras-requirements.txt (line 3)) | 13:10 |
| myoung | 00:24:05.924 Cloning https://git.openstack.org/openstack/browbeat/ to /tmp/pip-install-X3_Jph/browbeat | 13:10 |
| myoung | 00:24:05.924 Could not install packages due to an EnvironmentError: [Errno 2] No such file or directory: '/tmp/pip-req-tracker-m2hVFB/244aaf840d71445cf3dff9a2c7d47ea6eed6f4b132d79e827886ab0e' | 13:10 |
| myoung | chkumar|ruck: is this already understood or is it useful for me to spend some time there this morning? | 13:11 |
| *** quiquell|rover has quit IRC | 13:11 | |
| *** ykarel has quit IRC | 13:11 | |
| chkumar|ruck | myoung: chekcing | 13:11 |
| myoung | bare metal jobs are passing on https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/rdo-promote-rocky-rdo_trunk/3, but rascasoft pidone / cygnus job is failing | 13:12 |
| myoung | brb | 13:12 |
| rascasoft | myoung, /me tries to figure out why | 13:12 |
| *** jtomasek has joined #oooq | 13:13 | |
| *** ykarel has joined #oooq | 13:18 | |
| chkumar|ruck | myoung: trying locally | 13:20 |
| *** jtomasek has quit IRC | 13:22 | |
| chkumar|ruck | rascasoft: myoung I have found similar issue on pip https://github.com/pypa/pip/issues/5271 | 13:28 |
| chkumar|ruck | but not able to reproduce locally | 13:28 |
| *** agopi|brb has joined #oooq | 13:29 | |
| *** ykarel_ has joined #oooq | 13:34 | |
| *** ykarel has quit IRC | 13:36 | |
| rlandy | arxcruz|ptg: chkumar|ruck: is there any value in setting tempest_workers=4 if there is only one tempest test run? | 13:38 |
| *** agopi|brb is now known as agopi | 13:39 | |
| chkumar|ruck | rlandy: nope | 13:40 |
| hubbot1 | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-ocata, legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario009-multinode-oooq, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master, legacy-tripleo-ci- (3 more messages) | 13:40 |
| chkumar|ruck | rlandy: if there are multiple tests, then we can increase the workers | 13:40 |
| rascasoft | chkumar|ruck, myoung, the reason why rocky deployment failed on cygnus is 2018-09-10 08:03:37 | ERROR: Property error: : resources.Compute<nested_stack>.resources.0<https://192.168.24.2:13808/v1/AUTH_b971aac11ef24ab69f90df5f2c3cbdfe/overcloud/puppet/compute-role.yaml>.resources.NetworkConfig.properties: : Unknown Property StorageMgmtInterfaceRoutes | 13:41 |
| rascasoft | chkumar|ruck, myoung, why are we talking about pip? | 13:41 |
| chkumar|ruck | why jobs are getting killed https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-master-upload/5d5f242/job-output.txt.gz#_2018-09-12_13_31_45_998506 | 13:42 |
| chkumar|ruck | 2018-09-12 13:31:45.998506 | tripleo-ovb-centos-7 | bash: line 1: 6004 Killed bash -xe /opt/stack/new/tripleo-ci/toci_gate_test.sh | 13:42 |
| rascasoft | chkumar|ruck, ok, but this has nothing to do with baremetal jobs on rdophase2 | 13:43 |
| chkumar|ruck | rascasoft: sorry | 13:43 |
| rascasoft | chkumar|ruck no worries I'm trying to understand too | 13:44 |
| chkumar|ruck | rascasoft: based on the error message posted on irc by myoung I was suspecting something related to pip that's why I passed the url | 13:44 |
| *** skramaja has quit IRC | 13:46 | |
| rlandy | chkumar|ruck: any issues in running tempest with selinux enforcing? I am looking at a failure on rhos-13 https://thirdparty.logs.rdoproject.org/jenkins-tripleo-quickstart-rhos-13-passed-p2-baremetal-hp_dl360_envE-single_nic_vlans-2/tempest.html | 13:47 |
| rlandy | I lost the machine but I am will rerun to get the machine for debug | 13:47 |
| rlandy | that job also had tempest_workers set for 4 - which I removed | 13:48 |
| rlandy | rascasoft: do you run tempest with your rhos-13 jobs? | 13:48 |
| rascasoft | rlandy, no, just validation ha tests | 13:49 |
| rlandy | rascasoft: k - thanks | 13:49 |
| chkumar|ruck | rlandy: one more thing, does the test is simple api test or scenario tests | 13:52 |
| chkumar|ruck | if scenario we need to set the tempest_worker value so that it shoudl the test | 13:53 |
| rlandy | chkumar|ruck: it's a fs001 test | 13:54 |
| rlandy | so whatever fs001 runs | 13:54 |
| rlandy | 'tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops' | 13:54 |
| rlandy | so it is an scenario test | 13:55 |
| rlandy | chkumar|ruck; if I don;t explicitly set tempest_worker value, is there a default? setting it to 1 ok? | 13:55 |
| rlandy | still think we have an selinux issue | 13:56 |
| rlandy | but that will be easier to prove when I have the machine back | 13:56 |
| rlandy | running again now | 13:57 |
| chkumar|ruck | rlandy: https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/validate-tempest/tasks/pre-tempest.yml#L2 | 13:58 |
| chkumar|ruck | rlandy: by default it takes no. of available cpu on host | 13:58 |
| rlandy | chkumar|ruck: ah ok - I set it to 1 - but really I think we will see the error again | 13:59 |
| rlandy | chkumar|ruck: getting tempest to pass is the last hurdle in getting rhos-13 (and rhos-14) testing back | 13:59 |
| *** zul has quit IRC | 14:00 | |
| *** vinaykns has joined #oooq | 14:00 | |
| *** rootameen has joined #oooq | 14:08 | |
| *** rootameen has quit IRC | 14:08 | |
| *** rootameen has joined #oooq | 14:08 | |
| *** ykarel_ is now known as ykarel|away | 14:12 | |
| *** rootameen has quit IRC | 14:12 | |
| *** rootameen has joined #oooq | 14:12 | |
| *** jtomasek has joined #oooq | 14:12 | |
| *** udesale_ has joined #oooq | 14:13 | |
| *** udesale has quit IRC | 14:16 | |
| *** ykarel|away has quit IRC | 14:17 | |
| myoung | chkumar|ruck, rlandy, the pip error is being seein on the libvirt based fs020 jobs in rocky rdo2 | 14:21 |
| *** zul has joined #oooq | 14:22 | |
| myoung | chkumar|ruck: seeing if it repros here: https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/periodic-rocky-rdo_trunk-featureset020-1ctlr_1comp_64gb/2/console | 14:23 |
| myoung | chkumar|ruck: we have seen other spurious pip errors caused by the slave VM being an older version of fedora, most have been updated to 26, checking to see if the failure came from one of the remaining f22 slaves | 14:24 |
| rlandy | is there a way to remotely join this afternoon's ptg discussion? | 14:29 |
| rlandy | weshay: ^^? | 14:30 |
| weshay | rlandy, I will certainly ask.. I think we can get that worked out | 14:31 |
| weshay | we're starting in 30min | 14:31 |
| weshay | w/ ci | 14:31 |
| weshay | first | 14:31 |
| weshay | so stay tuned | 14:31 |
| rlandy | thanks | 14:31 |
| weshay | rlandy, fyi.. infra nuked the te-broker | 14:32 |
| rlandy | weshay: lol - at least it wasn't our fault this time | 14:32 |
| rlandy | saw discussion on #rdo | 14:32 |
| *** sshnaidm|afk has quit IRC | 14:40 | |
| *** jtomasek has quit IRC | 14:45 | |
| *** sshnaidm|afk has joined #oooq | 14:46 | |
| *** jtomasek has joined #oooq | 14:52 | |
| *** dtrainor__ has joined #oooq | 14:55 | |
| *** ykarel has joined #oooq | 14:57 | |
| *** dtrainor_ has quit IRC | 14:57 | |
| chkumar|ruck | myoung: is it possible to update it to f28 or wiat for few weeks and update it to f29 directly? | 14:58 |
| myoung | chkumar|ruck: all things are possible given time and money | 14:58 |
| myoung | chkumar|ruck: I would strongly advise against updating to f29 at this time, there's typically (usually) a raft of packaging issues exposed. We were at f20, and I've updated most from 20 --> 22 --> 24 --> 26 | 14:59 |
| myoung | 28 would make sense, when there's time and the world is not on fire :) | 14:59 |
| myoung | back in a few, juggling things here on my end apologies for response delays | 14:59 |
| weshay | ptg fyi https://bluejeans.com/u/emacchi/ | 15:02 |
| weshay | going over edge now | 15:02 |
| *** holser|ptg has joined #oooq | 15:03 | |
| arxcruz|ptg | rlandy: no, because it will have paralelism anyway :) | 15:04 |
| rlandy | arxcruz|ptg: k - will see what shakes out of the current run. I set it to 1 but I can change workers back | 15:05 |
| *** jaosorior has joined #oooq | 15:05 | |
| arxcruz|ptg | rlandy: do you have the logs ? | 15:06 |
| rlandy | arxcruz|ptg: of the previous run we looked at .. https://thirdparty.logs.rdoproject.org/jenkins-tripleo-quickstart-rhos-13-passed-p2-baremetal-hp_dl360_envE-single_nic_vlans-2/tempest.html | 15:07 |
| rlandy | current run is in progress | 15:07 |
| rlandy | we looked at this before | 15:07 |
| rlandy | were investigating selinux | 15:07 |
| rlandy | when I lost the machine | 15:08 |
| rlandy | bm hardware | 15:08 |
| arxcruz|ptg | oh yeah, i see, the kvm error right ? | 15:08 |
| arxcruz|ptg | rlandy: if you need i have a qemu/kvm guy that might be able to help you | 15:08 |
| arxcruz|ptg | although he's not openstacker | 15:08 |
| rlandy | always useful to keep a qemu/kvm guy on call | 15:08 |
| rlandy | but this is rhos-13 | 15:08 |
| rlandy | so we have to deal with the selinux fun | 15:09 |
| rlandy | will ping when I have the env available for debug | 15:09 |
| arxcruz|ptg | i think some nods are messed | 15:15 |
| arxcruz|ptg | got this error | 15:15 |
| arxcruz|ptg | http://logs.openstack.org/84/581084/28/check/tripleo-ci-centos-7-scenario002-multinode-oooq-container-refstack/5e116e0/job-output.txt.gz#_2018-09-12_10_00_42_423473 | 15:16 |
| arxcruz|ptg | chkumar|ruck: ^ | 15:16 |
| arxcruz|ptg | are you aware about this ? | 15:16 |
| *** tosky has joined #oooq | 15:17 | |
| chkumar|ruck | arxcruz|ptg: nope it is somehting new | 15:18 |
| ykarel | chkumar|ruck, arxcruz|ptg but it's not failing it's in rescue block | 15:18 |
| arxcruz|ptg | chkumar|ruck: happened once, i did a recheck let's see | 15:18 |
| *** jtomasek has quit IRC | 15:19 | |
| *** jtomasek has joined #oooq | 15:19 | |
| ykarel | it's failing at some tempest test | 15:19 |
| arxcruz|ptg | ykarel: oh, you're right | 15:19 |
| arxcruz|ptg | sorry | 15:20 |
| arxcruz|ptg | yes | 15:20 |
| arxcruz|ptg | my bad | 15:20 |
| chkumar|ruck | arxcruz|ptg: for this review, some how password is not found by tests | 15:20 |
| chkumar|ruck | that's why refstack tests are failing | 15:20 |
| arxcruz|ptg | chkumar|ruck: yeah | 15:20 |
| arxcruz|ptg | just notice that now, i went straight to the logs file because i didn't saw the tempest.html | 15:20 |
| arxcruz|ptg | so i assume tempest didn't run | 15:21 |
| arxcruz|ptg | chkumar|ruck: http://192.168.24.13:5000/v3/auth/tokens is where refstack is looking for credentials, but it should be without the /v3 right ? | 15:22 |
| arxcruz|ptg | chkumar|ruck: http://logs.openstack.org/84/581084/28/check/tripleo-ci-centos-7-scenario002-multinode-oooq-container-refstack/5e116e0/logs/undercloud/home/zuul/tempest.log.txt.gz#_2018-09-12_11_24_23 | 15:23 |
| *** jrist has joined #oooq | 15:23 | |
| chkumar|ruck | arxcruz|ptg: nope | 15:24 |
| *** kopecmartin has joined #oooq | 15:24 | |
| chkumar|ruck | arxcruz|ptg: that was used for some other purposes | 15:24 |
| arxcruz|ptg | chkumar|ruck: but it's failing right ? | 15:25 |
| chkumar|ruck | arxcruz|ptg: password comes from accounts.yaml, http://logs.openstack.org/84/581084/28/check/refstack-client-devstack-tempestconf/f9520dc/job-output.txt.gz#_2018-09-12_10_00_34_744366 | 15:26 |
| chkumar|ruck | arxcruz|ptg: there might be some issue with accounts.yaml or overcloudrc fike | 15:27 |
| *** d0ugal has quit IRC | 15:28 | |
| arxcruz|ptg | chkumar|ruck: i'm not seeing reference to accounts.yaml | 15:28 |
| arxcruz|ptg | only overcloudrc | 15:28 |
| chkumar|ruck | arxcruz|ptg: accounts.yaml refence in tempest.conf | 15:28 |
| arxcruz|ptg | chkumar|ruck: we don't even get into create the tempest.conf file | 15:29 |
| arxcruz|ptg | it fails before | 15:29 |
| *** chem has joined #oooq | 15:30 | |
| chkumar|ruck | arxcruz|ptg: http://logs.openstack.org/84/581084/28/check/tripleo-ci-centos-7-scenario002-multinode-oooq-container-refstack/5e116e0/job-output.txt.gz#_2018-09-12_11_22_43_326027 | 15:32 |
| arxcruz|ptg | chkumar|ruck: so, why i'm not seeing tempest.conf file in the logs ? | 15:33 |
| *** ykarel is now known as ykarel|away | 15:33 | |
| arxcruz|ptg | chkumar|ruck: "2018-09-12 11:22:58,059 refstack_client:478 INFO Tempest Configuration successfully generated in 13 second at /home/zuul/refstack_git/tempest.conf" | 15:34 |
| arxcruz|ptg | chkumar|ruck: so, you generate at /home/zuul/refstack_git and run tempest at /home/zuul/tempest right ? | 15:35 |
| chkumar|ruck | arxcruz|ptg: in the last step where tempest.conf is getting copied | 15:35 |
| chkumar|ruck | is not happening | 15:35 |
| chkumar|ruck | arxcruz|ptg: tempest running and tempest config genration all happen in refstack_git folder | 15:35 |
| chkumar|ruck | ykarel|away: http://logs.openstack.org/55/599355/11/check/tripleo-ci-centos-7-undercloud-upgrades/896bdd8/job-output.txt.gz#_2018-09-12_12_08_33_982387 not sure why it is happening | 15:36 |
| arxcruz|ptg | chkumar|ruck: i'll try to reproduce | 15:37 |
| chkumar|ruck | arxcruz|ptg: thanks :-) | 15:39 |
| hubbot1 | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-ocata, legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario009-multinode-oooq, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master, legacy-tripleo-ci- (3 more messages) | 15:40 |
| ykarel|away | chkumar|ruck, looks like infra issue, seeing at multiple places? | 15:42 |
| chkumar|ruck | ykarel|away: currently one | 15:42 |
| ykarel|away | okk | 15:43 |
| *** jaganathan has joined #oooq | 15:48 | |
| *** ykarel|away has quit IRC | 15:48 | |
| arxcruz|ptg | chkumar|ruck: do you know the name of discover package in centos ? | 15:53 |
| arxcruz|ptg | https://github.com/openstack/stackviz/blob/master/test-requirements.txt#L9 | 15:53 |
| chkumar|ruck | arxcruz|ptg: https://github.com/openstack/stackviz/search?q=discover&unscoped_q=discover | 15:55 |
| chkumar|ruck | not impoered any where | 15:55 |
| arxcruz|ptg | chkumar|ruck: ok | 15:55 |
| chkumar|ruck | arxcruz|ptg: also not packaged in fedora | 15:56 |
| chkumar|ruck | sshnaidm|afk: is this fix https://review.rdoproject.org/r/#/c/16245/ related to this bz https://bugs.launchpad.net/tripleo/+bug/1792158 ? | 15:58 |
| openstack | Launchpad bug 1792158 in tripleo "te-broker was removed from the openstack-nodepool tenant" [Critical,Triaged] | 15:58 |
| rlandy | arxcruz|ptg: chkumar|ruck: ok - got a reproducer on the failing tempest test | 15:58 |
| rlandy | https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/tripleo-quickstart-rhos-13-passed-p2-baremetal-hp_dl360_envE-single_nic_vlans/4/console | 15:58 |
| rlandy | is collecting logs now but I am on the undercloud | 15:58 |
| rlandy | will try run tempest again w selinux set to permissive | 15:59 |
| arxcruz|ptg | rlandy: for some reason my vpn is messed, i can't access | 15:59 |
| *** saneax has quit IRC | 15:59 | |
| rlandy | arxcruz|ptg: will send you publicly available logs when they complete | 15:59 |
| arxcruz|ptg | ok | 15:59 |
| arxcruz|ptg | rlandy: nova list --all-tenants | 16:00 |
| arxcruz|ptg | see if there are some resources created already | 16:00 |
| sshnaidm|afk | chkumar|ruck, yep, right | 16:00 |
| rlandy | sec -in the middle of rerunning | 16:01 |
| *** d0ugal has joined #oooq | 16:01 | |
| chkumar|ruck | sshnaidm|afk: assigned to you :-) | 16:01 |
| *** ykarel|away has joined #oooq | 16:02 | |
| sshnaidm|afk | chkumar|ruck, ok, didn't know bug exists :) | 16:03 |
| rlandy | arxcruz|ptg: setting permissive didn't help | 16:05 |
| arxcruz|ptg | :( | 16:05 |
| arxcruz|ptg | rlandy: and disabling? | 16:05 |
| arxcruz|ptg | just to check | 16:05 |
| rlandy | https://paste.fedoraproject.org/paste/n4bz~4SuNB-mAjcyf7QFZQ | 16:06 |
| rlandy | instance has a build failure | 16:06 |
| rlandy | hmmm ... | 16:06 |
| rlandy | wonder if there is an issue with the instance | 16:06 |
| chkumar|ruck | arxcruz|ptg: for stackviz package how we install npm related stuff in ci? | 16:07 |
| rlandy | arxcruz|ptg: nova list --all-tenants returns empty for the overcloud | 16:08 |
| arxcruz|ptg | chkumar|ruck: don't need, there's a tarball with the html/javascript already generated | 16:08 |
| chkumar|ruck | arxcruz|ptg: i think that needs to be kept somewhere in the package itself | 16:08 |
| chkumar|ruck | so that during rpm install it wil be used | 16:08 |
| arxcruz|ptg | rlandy: did you try to disable selinux ? | 16:09 |
| arxcruz|ptg | just to check | 16:09 |
| rlandy | trying now | 16:09 |
| arxcruz|ptg | chkumar|ruck: well, maybe a stackviz-html package, but i don't know how that would be created to be honest | 16:09 |
| arxcruz|ptg | i don't have the knowledge | 16:09 |
| rlandy | but permissive is pretty open | 16:09 |
| chkumar|ruck | arxcruz|ptg: i will figure out tomorrow morning | 16:09 |
| arxcruz|ptg | rlandy: kvm/qemu throws the error again ? | 16:09 |
| chkumar|ruck | arxcruz|ptg: drink more beer from kopecmartin on my behalf :-) | 16:10 |
| arxcruz|ptg | chkumar|ruck: he owns you beer? if so, i'll drink all of them :) | 16:10 |
| rlandy | arxcruz|ptg: hard to share with you if can;t get on vpn ... waiting for logs to collect to sen dyou | 16:10 |
| chkumar|ruck | arxcruz|ptg: I donot know the count, but he owns :-) | 16:11 |
| kopecmartin | do I? :D | 16:11 |
| arxcruz|ptg | rlandy: let me try again | 16:11 |
| chkumar|ruck | kopecmartin: yes you do :-) | 16:11 |
| arxcruz|ptg | let me restart my computer | 16:12 |
| rlandy | k | 16:12 |
| rlandy | arxcruz|ptg: https://thirdparty.logs.rdoproject.org/jenkins-tripleo-quickstart-rhos-13-passed-p2-baremetal-hp_dl360_envE-single_nic_vlans-4/ | 16:14 |
| rlandy | https://thirdparty.logs.rdoproject.org/jenkins-tripleo-quickstart-rhos-13-passed-p2-baremetal-hp_dl360_envE-single_nic_vlans-4/undercloud/home/stack/tempest.log.txt.gz | 16:15 |
| arxcruz|ptg | rlandy: it was just a matter of reboot, i can log into vpn now | 16:17 |
| weshay | please read through https://specs.openstack.org/openstack/tripleo-specs/specs/rocky/split-controlplane.html | 16:18 |
| weshay | panda|ptg, marios sshnaidm|afk https://docs.google.com/document/d/1-NkzNU487-ph9e9Ck6Io4MPbv4wGiUEexy3_sIYMNTI/edit | 16:20 |
| weshay | keeping some notes here | 16:20 |
| rlandy | arxcruz|ptg: cool - will send you login instructions | 16:22 |
| *** sshnaidm|afk is now known as sshnaidm|ptg | 16:23 | |
| rlandy | arxcruz|ptg: pretty much what we were looking at before | 16:24 |
| *** dtantsur is now known as dtantsur|afk | 16:24 | |
| rlandy | but no multiple workers | 16:24 |
| rlandy | and the last no selinux permissive | 16:24 |
| rlandy | disable requires reboot | 16:24 |
| rlandy | and I really don't think it makes a diff over permissive | 16:24 |
| rlandy | Exhausted all hosts available for retrying build failures for instance 1684d52f-04bc-4ed4-82b0-1c9996e43878.', u'code': 500, u'created': u'2018-09-12T15:57:35Z' | 16:25 |
| rlandy | ^^ not sure what causes that | 16:25 |
| *** vinaykns has quit IRC | 16:26 | |
| *** jaosorior has quit IRC | 16:28 | |
| *** Goneri has joined #oooq | 16:32 | |
| rlandy | arxcruz|ptg: still there? | 16:35 |
| *** jaosorior has joined #oooq | 16:36 | |
| arxcruz|ptg | rlandy: yes | 16:37 |
| rlandy | arxcruz|ptg: ok -don't want to keep you from your ptg duties :) | 16:38 |
| rlandy | arxcruz|ptg: so my question - is anyone running successfully running tempest on rhos? | 16:39 |
| *** jtomasek has quit IRC | 16:39 | |
| chkumar|ruck | rlandy: better to check with filip on #rhos-delivery | 16:40 |
| rlandy | chkumar|ruck: ack - will do | 16:40 |
| chkumar|ruck | fhubik is the irc nick | 16:40 |
| rlandy | hmm - don;t see fhubik atm but will check with others there | 16:41 |
| *** jtomasek has joined #oooq | 16:41 | |
| arxcruz|ptg | rlandy: https://paste.fedoraproject.org/paste/b99sbjFOEJo8MadOE-lmqQ | 16:44 |
| arxcruz|ptg | rlandy: setting selinux to enforcing in compute node made the test pass :) | 16:45 |
| rlandy | arxcruz|ptg: hmmm ... | 16:45 |
| rlandy | so we had it as enforcing on the undercloud but not the compute node | 16:45 |
| arxcruz|ptg | rlandy: https://bugzilla.redhat.com/show_bug.cgi?id=1538651#c24 seems related | 16:45 |
| openstack | bugzilla.redhat.com bug 1538651 in openstack-selinux "When using --libvirttype qemu to deploy an overcloud, VMs can't be launched because SELinux blocks it" [Low,New] - Assigned to zcaplovi | 16:45 |
| arxcruz|ptg | rlandy: yeah, doesn't matter enforcing in the undercloud since the vim will be execute in the compute node | 16:46 |
| rlandy | arxcruz|ptg: interesting | 16:48 |
| rlandy | not sure we attempt to set that | 16:48 |
| arxcruz|ptg | rlandy: i set again to enforcing and running the test | 16:48 |
| rlandy | arxcruz|ptg: I understand | 16:48 |
| rlandy | looking at #rhos-delivery results | 16:48 |
| rlandy | https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/ReleaseDelivery/view/OSP13/job/phase2-13_director-rhel-7.5-virthost-3cont_1comp-ipv6-vlan-lvm-containers/89/testReport/tempest.scenario.test_network_basic_ops/TestNetworkBasicOps/ | 16:48 |
| arxcruz|ptg | rlandy: yeah, didn't work, so selinux must set to permissive in the compute node | 16:48 |
| rlandy | https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/ReleaseDelivery/view/OSP13/job/phase2-13_director-rhel-7.5-virthost-3cont_1comp-ipv6-vlan-lvm-containers/89/testReport/tempest.scenario.test_network_basic_ops/TestNetworkBasicOps/test_network_basic_ops_compute_id_f323b3ba_82f8_4db7_8ea6_6a895869ec49_network_smoke_/ | 16:49 |
| rlandy | ^^ passes | 16:49 |
| rlandy | arxcruz|ptg: hmmm ... how does the tempest guru feel about adding a step to set setlinux to enforcing before tempest starts? | 16:52 |
| rlandy | any objection? | 16:52 |
| arxcruz|ptg | no | 16:52 |
| arxcruz|ptg | not from my side | 16:52 |
| arxcruz|ptg | maybe until https://bugzilla.redhat.com/show_bug.cgi?id=1538651 get the right permissions on selinux for containers | 16:53 |
| openstack | bugzilla.redhat.com bug 1538651 in openstack-selinux "When using --libvirttype qemu to deploy an overcloud, VMs can't be launched because SELinux blocks it" [Low,New] - Assigned to zcaplovi | 16:53 |
| rlandy | arxcruz|ptg: I wonder if that is a deploy option | 16:53 |
| rlandy | selinux status? | 16:53 |
| rlandy | then we could set it there and not bother tempest | 16:53 |
| *** udesale_ has quit IRC | 16:54 | |
| arxcruz|ptg | rlandy: yup sounds good to me | 16:54 |
| arxcruz|ptg | selinux to permissive | 16:54 |
| rlandy | arxcruz|ptg: selinux to enforcing | 16:54 |
| rlandy | for the deploy | 16:54 |
| * rlandy checks if there is such an option | 16:55 | |
| arxcruz|ptg | k | 16:55 |
| rlandy | arxcruz|ptg++ for finding that | 16:55 |
| hubbot1 | rlandy: arxcruz|ptg's karma is now 2 | 16:55 |
| rlandy | thank you | 16:55 |
| arxcruz|ptg | :) | 16:55 |
| *** bogdando has quit IRC | 17:02 | |
| rootameen | rocky aio (2 nics), failing with (No more hosts left): https://pastebin.com/hB40CEZV any ideas? | 17:04 |
| rootameen | correct paste: https://pastebin.com/FuVnBiEG | 17:07 |
| panda|ptg | rootameen: the actual error is in line 34. | 17:21 |
| *** vinaykns has joined #oooq | 17:23 | |
| *** vinaykns has joined #oooq | 17:27 | |
| *** d0ugal has quit IRC | 17:28 | |
| sshnaidm|ptg | weshay, panda|ptg https://review.openstack.org/#/c/602096 | 17:33 |
| weshay | panda|ptg, https://review.openstack.org/#/q/topic:standalone-f28+(status:open+OR+status:merged) | 17:39 |
| hubbot1 | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-ocata, legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario009-multinode-oooq, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master, legacy-tripleo-ci- (3 more messages) | 17:40 |
| rlandy | arxcruz|ptg: so there are a couple of layers to this ... | 17:46 |
| rlandy | one of which is why we are deploying iwth wemu over kvm | 17:46 |
| rlandy | with qemu over kvm | 17:46 |
| rlandy | and the other is how we land up with a permissive overcloud | 17:46 |
| arxcruz|ptg | rlandy: the other run that you point that pass is doing how? | 17:48 |
| rlandy | we are using the stable branches | 17:48 |
| rlandy | I want to switch to master | 17:48 |
| *** tesseract has quit IRC | 17:49 | |
| rootameen | panda|ptg, yeah I have no idea what's causing it, tried a re-run and ended on the same spot. any suggestions? | 17:49 |
| *** jtomasek has quit IRC | 17:52 | |
| *** panda|ptg has quit IRC | 17:53 | |
| rlandy | arxcruz|ptg: using master running again | 17:53 |
| arxcruz|ptg | rlandy: k | 17:53 |
| rlandy | pls log out from tripleo-quickstart-rhos-13-passed-p2-baremetal-hp_dl360_envE-single_nic_vlans | 17:54 |
| arxcruz|ptg | rlandy: i don't think i'm there anymore | 17:54 |
| rlandy | k - thanks | 17:54 |
| rlandy | arxcruz|ptg: you are right - that bug is relevant | 17:54 |
| rlandy | but using the latest deply, we may get by | 17:55 |
| rlandy | https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/set-libvirt-type/tasks/main.yml | 17:55 |
| *** agopi is now known as agopi|brb | 17:57 | |
| *** jrist has quit IRC | 17:57 | |
| *** sshnaidm|ptg has quit IRC | 17:58 | |
| *** agopi|brb has quit IRC | 18:01 | |
| *** ykarel|away has quit IRC | 18:03 | |
| *** chem has quit IRC | 18:06 | |
| *** holser|ptg has quit IRC | 18:09 | |
| *** zul has quit IRC | 18:21 | |
| *** kopecmartin has quit IRC | 18:23 | |
| *** tosky has quit IRC | 18:24 | |
| *** dsneddon has quit IRC | 18:28 | |
| *** agopi|brb has joined #oooq | 18:33 | |
| *** agopi|brb is now known as agopi | 18:33 | |
| *** dtrainor__ has quit IRC | 19:05 | |
| *** panda|ptg has joined #oooq | 19:15 | |
| *** zul has joined #oooq | 19:26 | |
| *** zul has quit IRC | 19:30 | |
| rlandy | panda|ptg: https://review.openstack.org/#/c/596422/ updated for te_broker_ip | 19:30 |
| *** zul has joined #oooq | 19:30 | |
| panda|ptg | rlandy: thanks. | 19:31 |
| rlandy | arxcruz|ptg: interesting ... deploy in progress and ... | 19:32 |
| rlandy | [heat-admin@overcloud-compute-0 ~]$ getenforce | 19:32 |
| rlandy | Enforcing | 19:32 |
| *** sshnaidm|ptg has joined #oooq | 19:33 | |
| *** holser|ptg has joined #oooq | 19:34 | |
| *** jrist has joined #oooq | 19:39 | |
| hubbot1 | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario009-multinode-oooq, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master, legacy-tripleo-ci- (2 more messages) | 19:40 |
| *** holser|ptg has quit IRC | 19:43 | |
| *** holser|ptg has joined #oooq | 19:43 | |
| *** jtomasek has joined #oooq | 19:51 | |
| *** tosky has joined #oooq | 19:58 | |
| rlandy | arxcruz|ptg: selinux is enforcing on this run - still see the failure | 19:58 |
| rlandy | trying with kvm deployment | 19:58 |
| *** rootameen has quit IRC | 20:00 | |
| arxcruz|ptg | rlandy: but on compute node ? | 20:04 |
| rlandy | arxcruz|ptg: ack, see above paste | 20:04 |
| arxcruz|ptg | rlandy: i did this: | 20:04 |
| rlandy | I check a bunch of times | 20:04 |
| arxcruz|ptg | echo 0 > /sys/fs/selinux/enforce | 20:04 |
| arxcruz|ptg | on the compute node | 20:04 |
| rlandy | all nodes are enforcing | 20:04 |
| rlandy | as they should be | 20:04 |
| rlandy | that sets permissive, no? | 20:05 |
| rlandy | not enforcing | 20:05 |
| arxcruz|ptg | rlandy: yes, permissive | 20:05 |
| rlandy | anyways, the bug stands with qemu | 20:05 |
| arxcruz|ptg | works as permissive, enforce fails | 20:05 |
| rlandy | ok - so that makes sense | 20:05 |
| arxcruz|ptg | (╯°□°）╯︵ ┻━┻ | 20:05 |
| rlandy | the reverse did not | 20:05 |
| rlandy | which is what I thought you changed | 20:05 |
| arxcruz|ptg | sorry, i sayd the opposite | 20:06 |
| arxcruz|ptg | said* | 20:06 |
| arxcruz|ptg | so, permissive works, enforce don't | 20:06 |
| rlandy | arxcruz|ptg; no worries - we just confirmed the bug | 20:06 |
| rlandy | we can't run rhel w/o selinux enforcing | 20:06 |
| rlandy | trying kvm might work | 20:06 |
| rlandy | that will be my next run | 20:06 |
| *** kopecmartin has joined #oooq | 20:07 | |
| rlandy | rhos is turning out to be a whole new party | 20:08 |
| *** chem has joined #oooq | 20:42 | |
| *** jrist has quit IRC | 20:46 | |
| *** jtomasek has quit IRC | 20:48 | |
| rlandy | rfolco++ | 20:51 |
| hubbot1 | rlandy: rfolco's karma is now 2 | 20:51 |
| rlandy | https://review.openstack.org/#/c/601584 | 20:51 |
| rlandy | ^^ thank you | 20:51 |
| rlandy | I needed that | 20:52 |
| *** jrist has joined #oooq | 20:52 | |
| *** dsneddon has joined #oooq | 21:06 | |
| *** d0ugal has joined #oooq | 21:19 | |
| *** tosky has quit IRC | 21:20 | |
| *** jtomasek has joined #oooq | 21:35 | |
| *** kopecmartin has quit IRC | 21:37 | |
| *** agopi is now known as agopi|off | 21:37 | |
| hubbot1 | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-queens, tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata, (2 more messages) | 21:40 |
| *** agopi|off has quit IRC | 21:40 | |
| *** agopi|off has joined #oooq | 21:41 | |
| *** d0ugal has quit IRC | 21:43 | |
| *** tosky has joined #oooq | 21:48 | |
| *** agopi|off has quit IRC | 21:52 | |
| *** vinaykns has quit IRC | 22:03 | |
| rlandy | panda|ptg: sshnaidm|ptg: if you have time after during ptg ... would like to start merging some of the depends-on patches ... https://review.openstack.org/#/c/596906/ https://review.openstack.org/#/c/596887/ https://review.openstack.org/#/c/600860 - pls comment/vote so I can make any changes | 22:10 |
| *** cibott has joined #oooq | 22:11 | |
| sshnaidm|ptg | rlandy, yeah, will look at them during ptg. | 22:14 |
| *** cibott has quit IRC | 22:15 | |
| *** cibott has joined #oooq | 22:16 | |
| rlandy | thank you | 22:18 |
| *** cibott has quit IRC | 22:20 | |
| *** cibott has joined #oooq | 22:20 | |
| rlandy | arxcruz|ptg: and kvm is the winner!!! | 22:21 |
| *** kopecmartin has joined #oooq | 22:22 | |
| rlandy | tempest is a go - and rhos-1 is on it's way | 22:22 |
| rlandy | rhos-13 | 22:22 |
| arxcruz|ptg | rlandy: so, we are going with kvm ? | 22:24 |
| rlandy | arxcruz|ptg: really we're going with https://review.openstack.org/#/c/601584 | 22:25 |
| rlandy | which hopefully should give us kvm | 22:25 |
| rlandy | if not, yes, I am setting kvm | 22:25 |
| rlandy | until qemu bugs sorts itself out | 22:25 |
| rlandy | we need rhos testing back | 22:25 |
| arxcruz|ptg | okay | 22:26 |
| arxcruz|ptg | you're da woman :D | 22:26 |
| rlandy | lol | 22:26 |
| arxcruz|ptg | rlandy: i ask my friend to take a look at that bug, he told me tomorrow he will | 22:26 |
| rlandy | arxcruz|ptg: thanks | 22:26 |
| rlandy | green - it's a beautiful thing!! | 22:31 |
| rlandy | https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/tripleo-quickstart-rhos-13-passed-p2-baremetal-hp_dl360_envE-single_nic_vlans/ | 22:31 |
| *** zul has quit IRC | 22:31 | |
| *** jtomasek has quit IRC | 22:33 | |
| rlandy | weshay: fyi ^^ running a test on rhos-14 now | 22:36 |
| weshay | ah cool | 22:36 |
| *** jaosorior has quit IRC | 22:40 | |
| *** cibott has quit IRC | 22:46 | |
| rlandy | panda|ptg: rechecking 596422 for ovb | 23:03 |
| *** jrist has quit IRC | 23:03 | |
| *** kopecmartin has quit IRC | 23:09 | |
| sshnaidm|ptg | rlandy, ovb won't work until https://review.openstack.org/602096 is merged | 23:11 |
| rlandy | sshnaidm|ptg: ack - will watch that | 23:12 |
| *** saneax has joined #oooq | 23:18 | |
| *** rlandy is now known as rlandy|bbl | 23:27 | |
| panda|ptg | rlandy|bbl: https://review.openstack.org/596422 have you reintroduced /etc/nodepool because there are too many pleaces that still use it ? | 23:29 |
| panda|ptg | if we are still going to use /etc/nodepool there is a lot of code in other patches that doesn't make sense ... | 23:31 |
| *** panda|ptg has quit IRC | 23:33 | |
| hubbot1 | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-queens, tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata, (2 more messages) | 23:40 |
| *** holser|ptg has quit IRC | 23:43 | |
| *** sshnaidm|ptg has quit IRC | 23:48 | |
| *** holser|ptg has joined #oooq | 23:49 | |
| *** sshnaidm|ptg has joined #oooq | 23:51 | |
| *** holser|ptg has quit IRC | 23:51 | |
| *** chem has quit IRC | 23:53 | |
| *** tosky has quit IRC | 23:57 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!