| *** agopi has joined #oooq | 00:23 | |
| hubbot` | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci- (1 more message) | 01:02 |
|---|---|---|
| hubbot` | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, tripleo-ci- (1 more message) | 03:02 |
| *** skramaja has joined #oooq | 03:10 | |
| *** skramaja_ has joined #oooq | 03:15 | |
| *** skramaja has quit IRC | 03:15 | |
| *** skramaja_ is now known as skramaja | 03:40 | |
| *** ykarel has joined #oooq | 03:40 | |
| *** ykarel has quit IRC | 04:02 | |
| *** udesale has joined #oooq | 04:16 | |
| *** sanjayu__ has joined #oooq | 04:20 | |
| *** sanjayu__ is now known as saneax | 04:20 | |
| *** ykarel has joined #oooq | 04:30 | |
| *** ratailor has joined #oooq | 04:38 | |
| *** saneax has quit IRC | 04:52 | |
| *** saneax has joined #oooq | 05:00 | |
| hubbot` | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, legacy-tripleo-ci-centos-7-container-to- (1 more message) | 05:02 |
| *** ykarel_ has joined #oooq | 05:22 | |
| *** holser_ has joined #oooq | 05:24 | |
| *** jaganathan has joined #oooq | 05:24 | |
| *** ykarel has quit IRC | 05:25 | |
| *** kopecmartin has joined #oooq | 05:27 | |
| *** brault has joined #oooq | 05:29 | |
| *** brault has quit IRC | 05:29 | |
| *** brault has joined #oooq | 05:29 | |
| *** ykarel__ has joined #oooq | 05:43 | |
| *** brault has quit IRC | 05:44 | |
| *** ykarel_ has quit IRC | 05:46 | |
| *** links has joined #oooq | 05:50 | |
| *** ykarel__ is now known as ykarel | 05:57 | |
| *** brault has joined #oooq | 06:01 | |
| *** holser_ has quit IRC | 06:12 | |
| *** jfrancoa has joined #oooq | 06:20 | |
| *** pliu_ has joined #oooq | 06:28 | |
| *** pliu_ has quit IRC | 06:38 | |
| *** jtomasek has joined #oooq | 06:56 | |
| chkumar|ruck | sshnaidm: ykarel for stable/queens periodic failure https://bugs.launchpad.net/tripleo/+bug/1781862 | 06:58 |
| openstack | Launchpad bug 1781862 in tripleo "[stable/queens][periodic]keystone keystone-manage db_sync failed on overcloud deploy in /legacy-periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset010-queens" [Critical,Triaged] | 06:58 |
| ykarel | chkumar|ruck, ack . is this consistent | 06:58 |
| chkumar|ruck | ykarel: it is on periodic job, how to check for consitent? | 07:01 |
| ykarel | chkumar|ruck, i meant it's failed once or multiple times | 07:01 |
| ykarel | and is this reproducable | 07:01 |
| *** bogdando has joined #oooq | 07:01 | |
| *** amoralej|off is now known as amoralej | 07:02 | |
| chkumar|ruck | ykarel: one time | 07:02 |
| hubbot` | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, legacy-tripleo-ci-centos-7-container-to- (1 more message) | 07:02 |
| *** quiquell|off is now known as quiquell | 07:04 | |
| ykarel | chkumar|ruck, ok then it's good to try to reproduce it | 07:04 |
| chkumar|ruck | ykarel: ack on it | 07:05 |
| ykarel | chkumar|ruck, and about periodic 24 failures, it's good check with rdo cloud, as same issue was seen day before yesterday | 07:07 |
| ykarel | i mean in periodic jobs | 07:07 |
| chkumar|ruck | ykarel: only 24 hr affected with ssh | 07:08 |
| chkumar|ruck | ykarel: 9 jobs | 07:08 |
| chkumar|ruck | ykarel: https://review.rdoproject.org/zuul3/builds.html | 07:08 |
| ykarel | yesterday it was not seen | 07:08 |
| chkumar|ruck | all periodic job with post_failure | 07:08 |
| ykarel | day before yesterday it was seen | 07:08 |
| chkumar|ruck | termination time 7k-8k | 07:09 |
| ykarel | so good to check with rdo cloud if there was some issue day before yesterday and today | 07:09 |
| *** ccamacho has joined #oooq | 07:09 | |
| *** ratailor_ has joined #oooq | 07:10 | |
| *** ratailor has quit IRC | 07:13 | |
| *** florianf has joined #oooq | 07:18 | |
| *** tesseract has joined #oooq | 07:19 | |
| *** zoli is now known as zoli|wfh | 07:30 | |
| *** zoli|wfh is now known as zoli | 07:30 | |
| quiquell | chkumar|ruck: I am reviewing the RR Cockpit | 07:32 |
| quiquell | chkumar|ruck: is RDO so bad ? Or do I have to look into something ? | 07:32 |
| quiquell | ykarel: Do you know if periodic jobs are already at RDO zuulv3 ? | 07:33 |
| ykarel | quiquell, yes | 07:33 |
| ykarel | they are | 07:33 |
| *** panda|off is now known as panda | 07:34 | |
| *** kopecmartin has quit IRC | 07:34 | |
| *** kopecmartin has joined #oooq | 07:38 | |
| *** tosky has joined #oooq | 07:40 | |
| quiquell | ykarel: I see all the periodic jobs failing | 07:41 |
| chkumar|ruck | quiquell: https://bugs.launchpad.net/tripleo/+bug/1781871 | 07:42 |
| openstack | Launchpad bug 1781871 in tripleo "[All branches][openstack-check][openstack-periodic] failing with SSH Error: data could not be sent to remote host \"38.145.32.100\". Make sure this host can be reached over ssh" [Critical,Triaged] | 07:42 |
| quiquell | chkumar|ruck: Ahh ok, then the data is right about RDO kind of broken | 07:42 |
| quiquell | chkumar|ruck: Ok no false positive | 07:42 |
| chkumar|ruck | quiquell: I dumped all affected jobs till seen | 07:42 |
| quiquell | chkumar|ruck: Was just checking the RR Cockpit tool, looks legit | 07:43 |
| quiquell | chkumar|ruck, ykarel: thanks | 07:43 |
| chkumar|ruck | quiquell: cool, I will update my patch | 07:43 |
| *** holser_ has joined #oooq | 07:54 | |
| quiquell | panda: Are use bluejeaneable ? | 08:00 |
| *** ykarel is now known as ykarel|lunch | 08:01 | |
| panda | quiquell: no, but I'm pajamable | 08:05 |
| panda | quiquell: bj/gcerami ? | 08:05 |
| quiquell | panda: Deal, going there | 08:07 |
| panda | quiquell: I have prolems connecting | 08:10 |
| quiquell | let me connect again | 08:10 |
| panda | quiquell: it doesn't let me in | 08:11 |
| quiquell | panda: Go to my room | 08:11 |
| quiquell | panda: https://bluejeans.com/7891065232 | 08:11 |
| *** lucas-afk is now known as lucasagomes | 08:12 | |
| marios | arxcruz: o/ hey so i asked you about this last week (I think you said 'concurrency' was the problem) ... do we have a bug for it or is there something we can do about it please https://review.openstack.org/#/c/578023/ (see last comment there from me on Jul 12 please) | 08:33 |
| marios | fails consistently but afaics nothing to do with this patch from ara logs http://logs.openstack.org/23/578023/5/gate/tripleo-ci-centos-7-scenario002-multinode-oooq-container/3ce1249/logs/ara_oooq/ looks like tempest related : | 08:34 |
| marios | arxcruz: ^ | 08:34 |
| marios | arxcruz: so does it mean it affects anything in tqe? I mean I see other things merging but then i wonder what is it about my patch afaics has nothing to do with tempest | 08:35 |
| chkumar|ruck | marios: ^^ regarding above issue https://bugs.launchpad.net/tripleo/+bug/1773325 | 08:35 |
| openstack | Launchpad bug 1773325 in tripleo "tempest.api.object_storage.test_object_services is failing on scenario002" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz) | 08:35 |
| marios | arxcruz: i can look into it but if there is some extra info please it would be useful (it just sounded like you may have known what the issue was) | 08:35 |
| marios | thanks chkumar|ruck looking | 08:35 |
| marios | chkumar|ruck: arxcruz ack i see so ongoing /me reads bug | 08:36 |
| marios | chkumar|ruck: yeah looks like it is that for sure http://logs.openstack.org/23/578023/5/gate/tripleo-ci-centos-7-scenario002-multinode-oooq-container/3ce1249/logs/tempest.html.gz | 08:38 |
| chkumar|ruck | marios: we are seeing this issue few times, sometimes it passes and sometimes it failed, I tried reproducing locally but failed to do that | 08:39 |
| chkumar|ruck | it is not a concurrency issue, i tried playing with concurrency by changing values but not able to reproduce that | 08:40 |
| marios | thanks chkumar|ruck unfortunately it is failing consistently on that review and so it is stuck in the gate | 08:40 |
| marios | for eva | 08:40 |
| chkumar|ruck | marios: we currently added this test to skip list till we get a proper solution https://review.openstack.org/#/c/582339/ | 08:41 |
| marios | chkumar|ruck: cool thanks so recheck should help now then | 08:42 |
| *** ykarel|lunch is now known as ykarel | 08:42 | |
| *** ssbarnea1 has joined #oooq | 08:45 | |
| marios | panda: o/ so does it mean the 3multinode issue is fixed now? | 08:50 |
| marios | panda: or at least you know what the problem was | 08:50 |
| marios | panda: ah sorry i re-read the notification, was from rlandy | 08:51 |
| panda | my bluejeans is officially KO | 08:51 |
| panda | marios: yep | 08:51 |
| *** chem has joined #oooq | 08:52 | |
| panda | marios: I don't even remember how we do the HostnameMap | 08:53 |
| quiquell | marios, panda: Let's try to workflow the toci_jobtype as job vars https://review.openstack.org/#/c/582465 | 08:53 |
| marios | quiquell: did you update i wasn't sure if it was a nit | 08:55 |
| marios | quiquell: ah thanks revoting | 08:55 |
| quiquell | marios: Yep, thanks so much !!! | 08:55 |
| marios | panda: i'd have to grep for it to be honest | 08:55 |
| marios | panda: https://github.com/openstack/tripleo-heat-templates/blob/58cb630f75f644cbbd2001be3f7e24e3cbc72dd0/overcloud.j2.yaml#L546 and then https://github.com/openstack/tripleo-heat-templates/blob/58cb630f75f644cbbd2001be3f7e24e3cbc72dd0/puppet/role.role.j2.yaml#L692 | 08:58 |
| marios | panda: lots of yaql and j2 \o/ | 08:58 |
| panda | yay | 08:58 |
| hubbot` | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode- (1 more message) | 09:02 |
| chkumar|ruck | sshnaidm: I filed this bug in the morning https://bugs.launchpad.net/tripleo/+bug/1781862 | 09:06 |
| openstack | Launchpad bug 1781862 in tripleo "[stable/queens][periodic]keystone keystone-manage db_sync failed on overcloud deploy in /legacy-periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset010-queens" [Critical,Triaged] | 09:06 |
| chkumar|ruck | tried the reproducer on rdocloud it does not failed | 09:06 |
| chkumar|ruck | sshnaidm: will i mark this as invalid or i wait for next run? | 09:07 |
| chkumar|ruck | till tomorrow? | 09:07 |
| quiquell | panda: Do you have the DNS latency web link shared by rlandy ? | 09:09 |
| panda | quiquell: no | 09:11 |
| sshnaidm | chkumar|ruck, how many times did it happen? | 09:11 |
| chkumar|ruck | sshnaidm: currently one time | 09:12 |
| chkumar|ruck | sshnaidm: I am comparing with the passing one | 09:12 |
| sshnaidm | chkumar|ruck, if it didn't happen twice, it didn't happen :) | 09:12 |
| sshnaidm | chkumar|ruck, please don't open bugs about one failure, just hold it on until you see it twice, of course if it's not something trivial and obvious | 09:13 |
| chkumar|ruck | sshnaidm: sure! | 09:13 |
| sshnaidm | chkumar|ruck, in this case for example you can see other multinode jobs that don't have this problem | 09:13 |
| sshnaidm | chkumar|ruck, and you did right when reproduced it, if it's not reproducible - looks like not real issue | 09:14 |
| chkumar|ruck | sshnaidm: yup | 09:15 |
| sshnaidm | chkumar|ruck, do we have bug about failing tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades with timeout in noop jobs? https://review.openstack.org/#/c/567224/ | 09:18 |
| chkumar|ruck | sshnaidm: nope | 09:19 |
| sshnaidm | chkumar|ruck, can you make one please? | 09:19 |
| chkumar|ruck | sshnaidm: yup | 09:19 |
| sshnaidm | chkumar|ruck, also need to notify pabelanger, because logs are not copied there.. | 09:20 |
| quiquell | panda, sshnaidm: +2 for toci_jobtype variables https://review.openstack.org/#/c/582465/ | 09:29 |
| *** dtantsur|afk is now known as dtantsur | 09:32 | |
| chkumar|ruck | sshnaidm: https://bugs.launchpad.net/tripleo/+bug/1781888 | 09:33 |
| openstack | Launchpad bug 1781888 in tripleo "[stable/queens]tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades job timing out on stable/queens noop jobs " [Critical,Triaged] | 09:33 |
| sshnaidm | chkumar|ruck, great, thanks! | 09:34 |
| chkumar|ruck | %gatestatus | 09:38 |
| hubbot` | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000 (1 more message) | 09:38 |
| chkumar|ruck | brb | 09:40 |
| arxcruz | lucasagomes: ping | 10:01 |
| ykarel | arxcruz, is tempestmail 38.145.34.221 used in ci jobs | 10:03 |
| ykarel | kforde is checking some network issue ^^ | 10:03 |
| arxcruz | ykarel: yes, it is | 10:04 |
| ykarel | arxcruz, so this ip is reachable from vm, right | 10:04 |
| arxcruz | ykarel: yes, i'm in the vm right now | 10:05 |
| ykarel | arxcruz, ok | 10:05 |
| ykarel | Thanks | 10:05 |
| *** udesale has quit IRC | 10:24 | |
| *** dtantsur is now known as dtantsur|brb | 10:35 | |
| arxcruz | apetrich: https://www.amazon.de/dp/B06XH2R7ZD/ref=sspa_dk_detail_0?psc=1&pd_rd_i=B06XH2R7ZD&pf_rd_m=A3JWKAKR8XB7XF&pf_rd_p=2444543729105426210&pf_rd_r=XY3QB3H8X28V7QHKAJ6Y&pd_rd_wg=RkRFW&pf_rd_s=desktop-dp-sims&pf_rd_t=40701&pd_rd_w=8eK40&pf_rd_i=desktop-dp-sims&pd_rd_r=0b5c2849-88e4-11e8-bcad-21bcac3ae2ea | 10:38 |
| *** zoli is now known as zoli|lunch | 10:41 | |
| hubbot` | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode- (1 more message) | 11:02 |
| *** jjoyce has quit IRC | 11:03 | |
| *** jjoyce has joined #oooq | 11:04 | |
| bogdando | sshnaidm: hi, please reconsider https://review.openstack.org/#/c/465047/12 | 11:06 |
| bogdando | there is no more reasons for -2 I think, no global changes | 11:06 |
| sshnaidm | bogdando, yeah, looking at it | 11:07 |
| bogdando | can we also merge https://review.openstack.org/#/c/582508/ please> | 11:09 |
| *** ratailor__ has joined #oooq | 11:10 | |
| bogdando | sshnaidm: thanks | 11:12 |
| *** ratailor_ has quit IRC | 11:13 | |
| *** verdurin has quit IRC | 11:19 | |
| *** verdurin has joined #oooq | 11:25 | |
| *** amoralej is now known as amoralej|lunch | 11:35 | |
| rasca | hey folks, hi, can I ask for some cores on this one: https://review.openstack.org/#/c/573255/ | 11:40 |
| rasca | chkumar|ruck ^^^ | 11:41 |
| rasca | (don't know who's rover this week) | 11:41 |
| *** udesale has joined #oooq | 11:41 | |
| chkumar|ruck | sshnaidm: ^^ | 11:42 |
| sshnaidm | rasca, seems fine, but I'd like to have a dummy patch that tests it actually with artcl_collect_list_append set | 11:42 |
| sshnaidm | rasca, can you please bring such up? if not, I'll try it later | 11:43 |
| *** sshnaidm is now known as sshnaidm|rover | 11:43 | |
| arxcruz | ykarel: so. the PYTHONWARNINGS are not solving the problem with ssl | 11:43 |
| rasca | sshnaidm|rover, it should be not a problem, even if I haven't done it before | 11:43 |
| arxcruz | ykarel: https://logs.rdoproject.org/80/577780/4/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master/3203551/logs/undercloud/home/zuul/tempest.log.txt.gz | 11:44 |
| arxcruz | i'm out of ideas | 11:44 |
| rasca | sshnaidm|rover, do I need to use some special code in the review description? I can imagine a Depends-On: to make the two fit together... right? | 11:44 |
| sshnaidm|rover | rasca, yeah, or just commit over this one, it doesn't matter. Just set "DNM: " in commit title, so people know this is not to review/merge | 11:45 |
| sshnaidm|rover | rasca, y'know, I'd rather be safe than sorry | 11:46 |
| rasca | sshnaidm|rover, sure, doing it at once | 11:46 |
| rasca | sshnaidm|rover, and in any case I'd say that to test this I'd better change some feature set config in tripleo-quickstart. What do you think? | 11:47 |
| sshnaidm|rover | rasca, sure | 11:48 |
| rasca | sshnaidm|rover, ack, doing it at once | 11:48 |
| sshnaidm|rover | rasca, seems totally dine | 11:48 |
| sshnaidm|rover | *fine | 11:48 |
| * sshnaidm|rover needs to dine.. | 11:48 | |
| *** dtantsur|brb is now known as dtantsur | 11:57 | |
| sshnaidm|rover | weshay, ping about rhos-dev-ci | 12:01 |
| quiquell | marios, panda, sshnaidm|rover: +1w for the toci_jobtype variables https://review.openstack.org/#/c/582465/ | 12:04 |
| sshnaidm|rover | quiquell, yay | 12:04 |
| ykarel | arxcruz, hmm you told PYTHONWARNING should fix ssl issue, let me check if i can find something | 12:07 |
| arxcruz | ykarel: yeah, but doesn't seems to fix it :( | 12:07 |
| ykarel | arxcruz, is it happening always | 12:08 |
| weshay | sshnaidm|rover, hey | 12:09 |
| quiquell | sshnaidm|rover: Thanks | 12:09 |
| sshnaidm|rover | weshay, morning | 12:09 |
| *** zoli|lunch is now known as zoli | 12:10 | |
| *** zoli is now known as zoli|wfh | 12:10 | |
| *** zoli|wfh is now known as zoli | 12:10 | |
| sshnaidm|rover | weshay, i have on rhos-dev-ci : Resource CREATE failed: OverQuotaClient: resources.provision_network: Quota exceeded for resources: ['network']. |\n| | Neutron server returns request_ids: ['req-1259b092-7673-4df9-aea4-f2bf3aca706a'] | 12:10 |
| sshnaidm|rover | weshay, tried to login, but no creds are available | 12:10 |
| sshnaidm|rover | weshay, what to do about it | 12:11 |
| weshay | sec | 12:11 |
| *** quiquell is now known as quiquell|lunch | 12:12 | |
| weshay | sshnaidm|rover, see pvt | 12:12 |
| sshnaidm|rover | weshay, see there | 12:15 |
| *** rfolco_ has joined #oooq | 12:16 | |
| *** agopi has quit IRC | 12:17 | |
| *** rlandy has joined #oooq | 12:23 | |
| *** ratailor__ has quit IRC | 12:26 | |
| sshnaidm|rover | rasca, if we are talking.. can you please take a look at failing job in rhos? pasting into private | 12:26 |
| sshnaidm|rover | rasca, it's always stuck on provisioning script | 12:26 |
| sshnaidm|rover | rasca, and I can't find this script anywhere.. | 12:26 |
| rasca | sshnaidm|rover, looking immediately | 12:27 |
| rasca | sshnaidm|rover, or at least after the CIX call | 12:28 |
| rasca | := | 12:28 |
| rasca | :) | 12:28 |
| sshnaidm|rover | rasca, oh, right | 12:28 |
| panda | rfolco_: throw me a note next time you fix something for me :) | 12:32 |
| panda | rfolco_: going to merge my change so I can start testing the ovb-manage integration | 12:32 |
| weshay | rasca, is your mic on? | 12:33 |
| rasca | weshay, can't understand what is happening | 12:34 |
| rasca | weshay, Bluejeans is not working | 12:34 |
| *** tcw has quit IRC | 12:36 | |
| rfolco_ | panda, I commented on your patch, email is old fashion dinosaur era | 12:36 |
| *** tcw has joined #oooq | 12:36 | |
| *** quiquell|lunch is now known as quiquell | 12:37 | |
| quiquell | rfolco_: Good morning | 12:37 |
| rfolco_ | quiquell, morning | 12:38 |
| weshay | arxcruz, you have a recreate for Thiago | 12:38 |
| rfolco_ | afternoon | 12:38 |
| quiquell | rfolco_: Have do some refactoring on where to put the variables | 12:38 |
| arxcruz | weshay: i don't | 12:38 |
| quiquell | rfolco_: Just from a toci_quickstart refact oring | 12:38 |
| arxcruz | weshay: but i'll create one | 12:39 |
| quiquell | rfolco_: Take a look a this patches https://review.openstack.org/#/c/582885/, https://review.openstack.org/#/c/582466/ | 12:39 |
| quiquell | rfolco_: I think they are a good place to move variables from all the toci scripts | 12:39 |
| quiquell | rfolco_: We can talk about in the scrum or after it | 12:40 |
| rfolco_ | quiquell, I am not changing these variables on my patches, so perhaps you can rebase yours on mine | 12:42 |
| chkumar|ruck | quiquell: I am confused https://review.rdoproject.org/r/#/c/14598/ for this review what i needs to fix? | 12:44 |
| rfolco_ | quiquell, I still need to add one more patch to the series... to completely remove toci_jobtype... wip | 12:44 |
| quiquell | chkumar|ruck: The result field is needed to show the result reason | 12:45 |
| panda | rfolco_: trello card is the new fashion dinosaur. I saw your comment and patch, didn't know it was solving my problem. Also this morning tristan was talking about shadowing projects | 12:45 |
| quiquell | rfolco_: ack, add your reviews to the card so I can look at them | 12:46 |
| *** agopi has joined #oooq | 12:46 | |
| panda | rfolco_: also, I have to say that the lack of certificate is getting more and more troublesome. I'm strugging to find new creative ways to not let ssl verify the erver certificates in all the tools involved | 12:46 |
| sshnaidm|rover | quiquell, discovered some problem in cockpit dash.. | 12:46 |
| sshnaidm|rover | quiquell, we have job status "SKIPPED" and this is not "FAILED" | 12:47 |
| sshnaidm|rover | quiquell, when dashboard are showing it as failed | 12:47 |
| quiquell | sshnaidm|rover: Paste me the url to the panel | 12:47 |
| sshnaidm|rover | quiquell, https://review.rdoproject.org/zuul3/builds.html?job_name=legacy-periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset010-pike | 12:48 |
| sshnaidm|rover | quiquell, just look at "last jobs" | 12:48 |
| *** trown|outtypewww is now known as trown | 12:49 | |
| quiquell | sshnaidm|rover: Let me do a chang, and tell me if this is what you expect | 12:49 |
| rfolco_ | panda, is the lack of certificate the root of decryption error ? | 12:51 |
| quiquell | sshnaidm|rover: Check it now | 12:52 |
| panda | rfolco_: no, the decryption error fails only on your repo, because secrets are valid only on the repos they are created for | 12:53 |
| weshay | sshnaidm|rover, thanks for going after the p2 items | 12:54 |
| panda | rfolco_: I created thos secrets with the kjey for the config project, not the one for 3oci2 | 12:54 |
| rfolco_ | panda, but I created the secret on my repo... | 12:54 |
| panda | rfolco_: hmm | 12:54 |
| panda | rfolco_: how did you create them ? | 12:54 |
| rfolco_ | in zuul.yaml.... | 12:54 |
| rfolco_ | - secrets | 12:54 |
| myoung | o/ | 12:55 |
| *** dtrainor has joined #oooq | 12:55 | |
| panda | rfolco_: how did you go from the password to the oweijfwoijwoiejfwoieowjowijoiwoiefjwoeif | 12:55 |
| panda | ? | 12:55 |
| *** dtrainor has joined #oooq | 12:55 | |
| *** dtrainor_ has quit IRC | 12:55 | |
| rfolco_ | panda, I just copied it from your patch, I did not play much with secrets coz I am open guy | 12:56 |
| panda | rfolco_: can you check if the patch is going to break zuul before I merge it ? | 12:56 |
| rfolco_ | panda, k | 12:57 |
| panda | rfolco_: that's what is not working, I created those blobs using the key for the config repo, they won't get decrypted in any other project | 12:57 |
| panda | rfolco_: if you think about it, in this way I can't just copy your secrets in my project, so I can decrypt them in my zuul jobs. | 12:58 |
| rfolco_ | panda, I don't think config is the best place for this change... | 12:59 |
| panda | rfolco_: config is the ONLY place for this change, since it's the only trusted project | 12:59 |
| panda | rfolco_: and only trusted projects can use secrets | 12:59 |
| panda | oh, meeting | 13:00 |
| marios | panda: trying to connect to bluejeans | 13:00 |
| marios | having issues | 13:00 |
| panda | marios: mee too | 13:00 |
| panda | marios: I have to use the app | 13:00 |
| weshay | https://bluejeans.com/7050859455/ | 13:00 |
| marios | panda: refresh helped ;) | 13:00 |
| marios | panda: well... ish | 13:01 |
| panda | marios: not me | 13:01 |
| marios | panda: still trying to connect i think am connected | 13:01 |
| sshnaidm|rover | quiquell, kind of.. but need to remove skipped from all statistics as failures | 13:01 |
| marios | but not fully loading | 13:01 |
| myoung | lagging here too | 13:01 |
| marios | myoung: welcome back :) | 13:02 |
| sshnaidm|rover | quiquell, maybe to remove them from "last_jobs" too.. | 13:02 |
| myoung | marios: thx...i've been online for 10 mins ;) | 13:02 |
| quiquell | sshnaidm|rover: We don't want them ? | 13:02 |
| * myoung watches the bluejeans spinny thingy | 13:02 | |
| hubbot` | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-scenario009-multinode-oooq, legacy-tripleo-ci-centos-7 (1 more message) | 13:02 |
| quiquell | Cannot connet to bj | 13:02 |
| rfolco_ | bj seems normal to me | 13:02 |
| panda | me too | 13:02 |
| rfolco_ | using app | 13:02 |
| panda | not even from the app | 13:02 |
| sshnaidm|rover | quiquell, they didn't run, so no need to put them in "last jobs" | 13:02 |
| myoung | i've tried app + browser...no dice | 13:02 |
| rlandy | ugh - no meeting here on BJ? | 13:03 |
| sshnaidm|rover | quiquell, maybe to put them in "last promotion jobs".. | 13:03 |
| rlandy | panda: can't connect either | 13:03 |
| panda | sorry guys, I'm cut out, no bluejeans is working for me | 13:03 |
| quiquell | sshnaidm|rover: We need to show finixhed jobs | 13:03 |
| weshay | panda, can you just dial in? | 13:03 |
| marios | panda: me too myoung weshay sorry still trying to connect | 13:03 |
| rlandy | The server encountered an internal error or misconfiguration and was unable to complete your request. | 13:03 |
| myoung | worked when coming in as a guest vs. authenticated | 13:03 |
| myoung | auth --> forever hang. | 13:04 |
| weshay | folks.. try to just dial in | 13:04 |
| sshnaidm|rover | quiquell, yeah, but skipped are not finished, and they break pass/fail statistics.. | 13:04 |
| myoung | (for me) | 13:04 |
| ssbarnea1 | indeed, BJ is down | 13:04 |
| panda | weshay: trying | 13:04 |
| weshay | please sign in https://etherpad.openstack.org/p/tripleo-ci-squad-meeting | 13:05 |
| quiquell | sshnaidm|rover: Ok, will try to skip skipped :-) | 13:05 |
| quiquell | blue jeans is spining | 13:05 |
| marios | weshay: i can't call in don't have any toll free numers like intercall used to (for cyprus) | 13:06 |
| quiquell | me neither | 13:06 |
| ssbarnea1 | ok joined by phone | 13:08 |
| weshay | quiquell, marios https://www.bluejeans.com/numbers | 13:10 |
| marios | weshay: thanks, looking (and trying to download the app sorry ) | 13:10 |
| marios | weshay: yeah no cyprus unfortunately | 13:11 |
| marios | weshay: in with app | 13:14 |
| weshay | quiquell, go on mute please | 13:16 |
| weshay | myoung, mute him ^ | 13:17 |
| quiquell | weshay: Sorry | 13:17 |
| *** matbu has quit IRC | 13:22 | |
| *** matbu has joined #oooq | 13:23 | |
| rasca | sshnaidm|rover, I think we have an hardware problem with that deployment | 13:24 |
| *** amoralej|lunch is now known as amoralej | 13:24 | |
| *** brault has quit IRC | 13:27 | |
| *** brault has joined #oooq | 13:27 | |
| sshnaidm|rover | rasca, ok, so what can we do? | 13:29 |
| rasca | sshnaidm|rover, I'm working on it, just need to accelerate the process of making the virtual undercloud. Stay tuned. | 13:30 |
| sshnaidm|rover | rasca, cool, thanks | 13:30 |
| rasca | sshnaidm|rover, at you service | 13:30 |
| ykarel | trown, sshnaidm|rover can you please review https://review.openstack.org/#/c/582462/ | 13:33 |
| chkumar|ruck | arxcruz: kopecmartin feel free to update the cards, thanks :-) | 13:33 |
| agopi | quiquell: ping | 13:36 |
| quiquell | agopi: Hello there | 13:36 |
| agopi | hello quiquell, rlandy said rdo cloud has its own ES host, can you please share the host details? | 13:37 |
| agopi | any chance, there's a graphite host too? | 13:37 |
| quiquell | agopi: Give me a minute I am in mtg now | 13:38 |
| agopi | sure thing quiquell. | 13:38 |
| rlandy | agopi: we need to talk about this ... | 13:38 |
| ssbarnea1 | do we have somewhre some bash code that is can parse ZUUL_CHANGES variable? | 13:38 |
| rlandy | weshay wants to ignore the actual data upload to grafana or now | 13:38 |
| rlandy | agopi: we are in meeting now - will ping you after the meeting | 13:39 |
| agopi | okay rlandy | 13:39 |
| *** holser_ has quit IRC | 13:39 | |
| *** holser_ has joined #oooq | 13:40 | |
| ykarel | sshnaidm|rover, please review https://review.openstack.org/#/c/581640/ also | 13:47 |
| ykarel | other one i replied | 13:48 |
| quiquell | sshnaidm|rover, chkumar|ruck: have to drop builds from influxdb for RR Cockpit | 13:49 |
| sshnaidm|rover | quiquell, ack | 13:49 |
| sshnaidm|rover | ykarel, so you rename image *.initramfs to be *.initrd? | 13:50 |
| *** vinaykns has joined #oooq | 13:51 | |
| ykarel | sshnaidm|rover, rename initramfs --> overcloud-full.initrd and vmlinuz --> overcloud-full.vmlinuz | 13:51 |
| ykarel | sshnaidm|rover, it's used to be ^^ way https://review.openstack.org/#/c/408773, | 13:52 |
| ykarel | but it didn'c consider overcloud_as_undercloud: false | 13:52 |
| sshnaidm|rover | ykarel, ack | 13:54 |
| ykarel | sshnaidm|rover, Thanks | 13:54 |
| *** sshnaidm|rover is now known as sshnaidm|afk | 13:54 | |
| *** skramaja has quit IRC | 13:55 | |
| quiquell | sshnaidm|afk: Now it's ok ? | 13:59 |
| rlandy | agopi: hello | 14:01 |
| rlandy | agopi: following our discussion on friday | 14:01 |
| rlandy | agopi: I spoke with weshay regarding the issue of uploading the browbeat data to a grafana that rdocloud can reach | 14:02 |
| quiquell | sshnaidm|afk: To remove skipped https://review.rdoproject.org/r/#/c/14852/ | 14:03 |
| quiquell | sshnaidm|afk: It's already in place though | 14:03 |
| *** links has quit IRC | 14:03 | |
| rlandy | agopi: the decision was that we should not be logging the data to grafana as that is not a test of the tools as such | 14:03 |
| rlandy | could you run an effective test without uploading data? | 14:04 |
| rlandy | ie: you'd run the collect metrics piece | 14:04 |
| rlandy | just not upload the data ( to start with) | 14:04 |
| rlandy | we can discuss the data upload with rdo infra people once the test works | 14:04 |
| rlandy | ^^ is that a workable situation for you? | 14:05 |
| rlandy | rook: ^^ pls see above | 14:05 |
| panda | rfolco_: qiuck, take cover | 14:07 |
| panda | rfolco_: I'm merging that patch | 14:07 |
| weshay | rook, first pass at this... sending perf data is not criteria. Please ping us if you have concerns / questions about that | 14:07 |
| rfolco_ | panda, fear | 14:07 |
| rfolco_ | panda, you are changing __base-jobs.yaml in config | 14:07 |
| rfolco_ | panda, we need somebody from SF DFG to revert it if we break zuul | 14:08 |
| rlandy | rook: agopi: we need to know if there is a workable minimal workflow without ES and grafana upload | 14:08 |
| *** jaganathan has quit IRC | 14:08 | |
| rook | weshay: how about the results of the tools ran | 14:09 |
| rook | not the collectd crud but the rally results | 14:09 |
| agopi | rlandy, you said ideally we can have 2-3 hours of job time right? | 14:10 |
| rlandy | agopi: job time is 3 hrs atm | 14:11 |
| rfolco_ | panda, I am trying to read docs to see how to work with secrets... imagine any change in our repos in sf... they will run a ovb job ? does not make any sense to me | 14:11 |
| panda | rfolco_: does this need 2x+2 ? | 14:11 |
| rfolco_ | panda, no, one is enough | 14:11 |
| rfolco_ | panda, can you w+ ? | 14:11 |
| panda | rfolco_: I can | 14:11 |
| panda | rfolco_: gates started | 14:12 |
| panda | rfolco_: I don't understand your question | 14:12 |
| rfolco_ | panda, with this change I cannot submit any patch to 3oci2-jobs for example | 14:12 |
| rfolco_ | oh I see you filter some files | 14:13 |
| *** matbu has quit IRC | 14:13 | |
| panda | rfolco_: yep , only in checks, and only for the files involved in the change | 14:13 |
| rook | rlandy agopi could we not index to our public server? | 14:13 |
| panda | so even if jobs fail, a +1 will run the standard gate, and no harm done | 14:13 |
| panda | maybe | 14:13 |
| agopi | rook, apparently not | 14:13 |
| rook | agopi: that shouldn't impact time | 14:13 |
| weshay | rook, logging them is fine for the first pass | 14:14 |
| rlandy | rook: I was not aware you have any public server | 14:14 |
| agopi | we've one rlandy | 14:14 |
| agopi | http://elk.browbeatproject.org | 14:14 |
| weshay | rook, agopi here is what I would like to see | 14:14 |
| *** matbu has joined #oooq | 14:15 | |
| weshay | rook, agopi rlandy will get the job up and running.. end to end | 14:15 |
| weshay | rook, agopi will the job fail if the data is not sent to an elk server? | 14:15 |
| rook | if the job never gets to running browbeat. | 14:15 |
| agopi | no it wont | 14:15 |
| rook | oh | 14:15 |
| rook | so if it is running browbeat and fails to index | 14:15 |
| rook | no | 14:15 |
| rook | it will give up and write locally | 14:16 |
| rook | so you could tar it u | 14:16 |
| rook | up* | 14:16 |
| weshay | rook, right.. let's take it that far | 14:16 |
| weshay | once we see that | 14:16 |
| weshay | agopi, rook jump in and hook it up to an elk server | 14:16 |
| weshay | and we'll watch the results from there | 14:16 |
| rlandy | what do you mean by jump in? | 14:16 |
| rook | weshay agopi I think we need to open up the rules to let a new source index to that es server | 14:17 |
| rook | is that source inside or ourside of RH? | 14:17 |
| panda | rfolco_: do we have access to zuul logs ? | 14:17 |
| panda | rfolco_: I already broke everything | 14:17 |
| rfolco_ | panda, no | 14:17 |
| panda | rfolco_: wonderful | 14:17 |
| rfolco_ | panda, need to talk to sf dfg and see if they can help / revert | 14:18 |
| rlandy | rook: rdocloud is outside | 14:18 |
| panda | rfolco_: just kidding, the job just fails with node_failure error | 14:18 |
| rlandy | ie: has no access to RH servers | 14:18 |
| rfolco_ | panda, change to upstream-centos-7 | 14:19 |
| *** ykarel is now known as ykarel|away | 14:19 | |
| rlandy | agopi: btw: I have not forgotten about the reproducer | 14:20 |
| rlandy | agopi: some reported network issues on rdocloud now | 14:21 |
| panda | rfolco_: yep, | 14:21 |
| rlandy | will retry when we are cleared there | 14:21 |
| agopi | rook, we can get the ip range for rdo cloud and add it to rules, shouldn't be a problem i guess? | 14:21 |
| rook | right | 14:21 |
| agopi | and right now only browbeat-ci is pushing data to elk.browbeatproject.org | 14:22 |
| agopi | ill just move al the old data from there to one of our new hosts. | 14:22 |
| agopi | haha okay rlandy, thanks | 14:23 |
| agopi | also, rlandy weshay what's the optimal run time for browbeat roles? | 14:23 |
| agopi | ^ what would you like it to be? | 14:24 |
| rlandy | agopi: rook: if you guys can get that into the browbeat-minimal.yml playbook we are expecting here, https://review.openstack.org/#/c/581488/3/playbooks/baremetal-quickstart-extras.yml | 14:24 |
| rlandy | we can run | 14:24 |
| rlandy | agopi: anything that keeps the job under the run time max - which is 3 hours in total now | 14:24 |
| rlandy | we can up the job time | 14:24 |
| rlandy | what is your estimate on how long it will take? | 14:25 |
| agopi | 3 hours for us to run browbeat ? or does it include deploying UC and such? | 14:25 |
| rlandy | including everything | 14:25 |
| rlandy | we do have flexibility to up the job time | 14:25 |
| rlandy | how ling would you need? | 14:26 |
| *** ykarel|away has quit IRC | 14:26 | |
| rlandy | long | 14:26 |
| rlandy | could you work with3.5 hours? | 14:26 |
| agopi | it should work i believe. | 14:26 |
| weshay | agopi, ideally we're keeping it at the same run time as our full tempest run | 14:27 |
| weshay | agopi, which is.. /me gets | 14:27 |
| agopi | how long does it take to get to the part from where we can run the minimal playbook for browbeat? | 14:27 |
| ssbarnea1 | weshay: can you please have a brief look at https://review.openstack.org/#/c/582963/ and tell me if that's desired or not. | 14:27 |
| weshay | ssbarnea1, k.. in a bit | 14:27 |
| agopi | okay weshay | 14:27 |
| weshay | agopi, rlandy 4hr https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-master/566903e/job-output.txt.gz | 14:28 |
| weshay | agopi, is that workable? | 14:29 |
| weshay | agopi, reminder.. the purpose of this job is to exercise browbeat to ensure it does not break | 14:29 |
| weshay | agopi, we can setup a longer job in a diff queue | 14:29 |
| weshay | if the results are meaningful to you guys | 14:29 |
| weshay | agopi, the other queue can run for 23hrs if needed :) | 14:30 |
| *** weshay is now known as weshay_mtg | 14:30 | |
| rlandy | let's stay away from 23hrs if we can | 14:31 |
| rlandy | agpoi: weshay_mtg: rook: ok - I'll increase the job time out to four hours - let's work within that | 14:31 |
| weshay_mtg | lolz | 14:31 |
| weshay_mtg | rlandy, thanks | 14:32 |
| agopi | okay yes, but this 4 hour job includes running tempest, it takes 2 hours to get to validate-overcloud | 14:32 |
| *** quiquell is now known as quiquell|off | 14:32 | |
| rlandy | agopi: we can get rid of tempest | 14:32 |
| agopi | so yes we'll be able to work within the 3.5 hours easily | 14:32 |
| rlandy | for your run if need be | 14:32 |
| weshay_mtg | agopi, very good | 14:32 |
| weshay_mtg | arxcruz, you have a recreate up w/ swift? | 14:33 |
| rlandy | cool beans - updating job definition | 14:33 |
| agopi | yes we can get rid of tempest so that gives us 1.5-2 hours i believe | 14:33 |
| arxcruz | weshay_mtg: not yet, on the way | 14:33 |
| rook | for sure | 14:33 |
| weshay_mtg | arxcruz, thanks | 14:33 |
| agopi | rook, the minimal version of browbeat takes 15-20 mins to run the config file for browbeat-ci | 14:33 |
| *** panda is now known as panda|travel | 14:35 | |
| ssbarnea1 | does anyone knows the minimal supported version on git on ci? | 14:39 |
| ssbarnea1 | i want to know if assuming git>=1.8.5 is acceptable or not. | 14:40 |
| rlandy | agopi: rook: weshay_mtg: https://review.rdoproject.org/r/#/c/14808/ job is set for 4 hours now. https://review.openstack.org/#/c/581484/3/config/general_config/featureset053.yml run tempest but it's a sort test | 14:44 |
| rlandy | sshnaidm|afk still has a -1 on this - we discussed it - will have to touch base tomorroqw | 14:45 |
| *** links has joined #oooq | 14:47 | |
| *** sshnaidm|afk is now known as sshnaidm|rover | 14:47 | |
| arxcruz | sshnaidm|afk: rlandy weshay_mtg trown panda|travel https://review.openstack.org/#/c/572347/ when you guys have time | 14:48 |
| agopi | okay rlandy. | 14:48 |
| rlandy | agopi; let's see how far 4 hours takes you | 14:49 |
| rlandy | what's the status of rdocloud now - usable? | 14:57 |
| rook | nice rlandy!! | 14:59 |
| *** bogdando has quit IRC | 15:01 | |
| hubbot` | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-scenario009-multinode-oooq, legacy-tripleo-ci-centos-7 (1 more message) | 15:02 |
| amoralej | can i recheck https://review.openstack.org/#/c/582327/ ? | 15:11 |
| rasca | rlandy, may I have a word with you about ha jobs integration upstream? | 15:12 |
| *** weshay_mtg is now known as weshay | 15:13 | |
| rlandy | rasca: sure - 10 mins? just fixing a merge issue | 15:13 |
| rasca | weshay, if you're free maybe you can help me here too ^^ | 15:13 |
| rasca | rlandy, sure | 15:13 |
| weshay | yes.. | 15:13 |
| weshay | please ;) | 15:13 |
| rasca | rlandy, weshay, for the moment is just a question: I want to make all the reviews consistent, and as a first step I want tripleo-ha-utils to be available in quickstart (to enable for example fs41, like in https://review.openstack.org/#/c/538952/) | 15:15 |
| weshay | amoralej, yes sir.. recheck away | 15:15 |
| rasca | rlandy, weshay, is the right approach to add tripleo-ha-utils as a requirement for quickstart? | 15:15 |
| weshay | rlandy, and I thought so.. but sshnaidm|rover also had some other ideas | 15:15 |
| weshay | not sure where we landed on that | 15:15 |
| rasca | I mean, inside quickstart-extras-requirements.txt | 15:15 |
| weshay | rasca, yes | 15:16 |
| rlandy | rasca is asking the same question above | 15:16 |
| rasca | weshay, rlandy, great, so I'm posting the review | 15:16 |
| rlandy | weshay: sshnaidm|rover suggests a separate file is better | 15:16 |
| weshay | rasca, imho I think that is fine.. I would take browbeat and ha from quickstart-extras and see if I can rework that into sshnaidm|rover's path | 15:16 |
| rlandy | what is your take on that | 15:16 |
| rlandy | I am -1 on using sshnaidm|rover's patch to put the requirements in quickstart | 15:16 |
| weshay | rlandy, I'm unclear how the zuul job would change | 15:16 |
| rlandy | meaning? | 15:17 |
| rasca | rlandy, sshnaidm|rover, so you mean having like : ${OOOQ_HA_REQUIREMENTS:=quickstart-ha-requirements.txt} in quickstart.sh ? | 15:17 |
| weshay | rlandy, ok.. rasca so for now.. as long as.. the repo is in openstack.. not github I'm fine adding it directly to the requirements | 15:17 |
| rlandy | if we have multiple requirements files | 15:17 |
| weshay | because that has been the plan and design to-date | 15:17 |
| rlandy | I have a -1 on my patch | 15:17 |
| sshnaidm|rover | weshay, rlandy I think we can either to support this from command line (https://review.openstack.org/#/c/576816/) or to use additional requirement files | 15:17 |
| rlandy | sshnaidm|rover: I am -1 on the command line add | 15:18 |
| rlandy | ok with multiple req files | 15:18 |
| sshnaidm|rover | rlandy, why not command? | 15:18 |
| weshay | amoralej, one needs to have an email alert when files change that break the workflows we care about :) | 15:18 |
| * weshay feels ur pain | 15:18 | |
| rlandy | sshnaidm|rover: this needs to be supported out of the box | 15:18 |
| rlandy | not an add on after | 15:19 |
| sshnaidm|rover | rlandy, not sure I understand | 15:19 |
| rlandy | we have ha, browbeat, tempest | 15:19 |
| weshay | sshnaidm|rover, I think your change introduces an interesting option.. | 15:19 |
| rlandy | all needing support not reliant on quickstart.sh | 15:19 |
| weshay | sshnaidm|rover, however I think it should be considered after we get browbeat, ha etc.. integrated upstream | 15:20 |
| rlandy | tq should not need quickstart to run | 15:20 |
| sshnaidm|rover | rlandy, if somebody want to run oooq with his role from infeared, one can just run -u path/to/my/role | 15:20 |
| rlandy | we are moving to an ansible-playbook workflow | 15:20 |
| rlandy | I am -1 on 'just run with' | 15:20 |
| weshay | sshnaidm|rover, sure.. they can also just add it to the file | 15:20 |
| sshnaidm|rover | weshay, rlandy let's leave browbeat aside, it's not a point | 15:20 |
| rlandy | this needs to be a supported workflow | 15:20 |
| weshay | sshnaidm|rover, it's not a huge change.. however it does impact how the jobs would be defined | 15:21 |
| rlandy | I am fine with the review bit not as a solution to this problem | 15:21 |
| sshnaidm|rover | rlandy, we don't remove quickstart.sh, we can't replace it with ansible workflow, and this is not a goal | 15:21 |
| rlandy | it is not he same thing | 15:21 |
| weshay | rlandy, sshnaidm|rover let's discuss this for a sec | 15:21 |
| weshay | https://bluejeans.com/u/whayutin/ | 15:21 |
| rasca | weshay, can I join? | 15:22 |
| weshay | sure | 15:22 |
| *** ccamacho has quit IRC | 15:23 | |
| myoung | chkumar|ruck, weshay, tempest scrum | 15:32 |
| chkumar|ruck | myoung: I will miss the scrum, I am away from laptop | 15:32 |
| myoung | ok | 15:32 |
| myoung | weshay: are you joining? | 15:34 |
| weshay | bah.. ya.. in a minute | 15:34 |
| myoung | ^^ https://bluejeans.com/7050859455 | 15:34 |
| myoung | ack | 15:34 |
| *** saneax has quit IRC | 15:34 | |
| *** ccamacho has joined #oooq | 15:35 | |
| *** d0ugal has quit IRC | 15:41 | |
| *** d0ugal has joined #oooq | 15:41 | |
| *** d0ugal has quit IRC | 15:41 | |
| *** d0ugal has joined #oooq | 15:41 | |
| sshnaidm|rover | rlandy, remind me please which patch it is | 15:55 |
| rlandy | sshnaidm|rover, https://review.openstack.org/#/c/581484/ | 15:55 |
| rlandy | sshnaidm|rover: +2'ed your patch | 15:56 |
| sshnaidm|rover | rlandy, thanks | 15:56 |
| sshnaidm|rover | rlandy, added comments, about ntp: https://review.openstack.org/#/c/582802/ | 15:58 |
| weshay | arxcruz, https://review.openstack.org/#/q/I398980184f3c2ae2057b7cc5c56a6a1f42774be0 | 15:59 |
| sshnaidm|rover | rasca, about https://review.openstack.org/#/c/582932 - can you please use featureset 010? 002 is featureset that we don't use in CI. Sorry that I missed this.. | 15:59 |
| rlandy | sshnaidm|rover: ack - there is merge issue on the fs doc as well - I am fixing the ntp thing with the merge | 15:59 |
| sshnaidm|rover | rlandy, cool | 16:00 |
| *** links has quit IRC | 16:00 | |
| rasca | sshnaidm|rover, I'll do it in a sec, don't worry | 16:01 |
| *** d0ugal has quit IRC | 16:03 | |
| ssbarnea1 | sshnaidm|rover: i fixed https://review.openstack.org/#/c/581788/ few days ago but I am afraid that posting a log is useless. Cannot really highlight in the log the fact that warnings are no longer displayed ;) | 16:05 |
| rlandy | agopi: rook: weshay: wrt your additions to https://review.openstack.org/#/c/581488/ - we need to decide on what goes into the tq/tqe repos directly as opposed to what stays in browbeat and gets called from tq/tqe | 16:09 |
| rlandy | agopi: rook: weshay: it was my impression that all the role and playbook would be committed to the browbeat repo | 16:09 |
| rlandy | and that we would only call that playbook from tqe | 16:09 |
| rlandy | pls confirm | 16:10 |
| sshnaidm|rover | ssbarnea1, I'll look. I asked log file just to see if nothing is broken, because CI doesn't test most of this code :) | 16:10 |
| *** d0ugal has joined #oooq | 16:11 | |
| *** udesale has quit IRC | 16:15 | |
| agopi | rlandy, yes all the roles and playbooks would be in the browbeat repo, the only thing that goes into tqe would be the task to call the bootstrapping script in the repo. | 16:17 |
| agopi | idea is to keep most of the work in browbeat, only those that can be mpoved to browbeat repo would be in tqe | 16:20 |
| *** kopecmartin has quit IRC | 16:22 | |
| rlandy | left a comment | 16:22 |
| rlandy | I can fix it if that stays in tqe | 16:23 |
| rlandy | I have to fix a merge issue on the fs review in the mean time | 16:27 |
| *** sshnaidm|rover is now known as sshnaidm|bbl | 16:28 | |
| agopi | from what i understand, the only way to make it to work is to keep in tqe. unless im missing somehting. | 16:30 |
| *** sshnaidm_away has joined #oooq | 16:31 | |
| *** sshnaidm|bbl has quit IRC | 16:31 | |
| *** sshnaidm_away is now known as sshnaidm|bbl | 16:32 | |
| rlandy | agopi: https://review.openstack.org/#/c/581484/3/quickstart-extras-requirements.txt makes browbeat role and playbooks available | 16:32 |
| chkumar|ruck | weshay: myoung rlandy please have a look at this https://review.openstack.org/#/c/582503/ | 16:36 |
| agopi | rlandy, won't that mean that there'll be problems if browbeat repo happens to have same roles that tq/tqe has? | 16:37 |
| rlandy | does it? | 16:39 |
| *** trown is now known as trown|lunch | 16:43 | |
| *** tesseract has quit IRC | 16:43 | |
| agopi | currently i don't think so, but moving forward it be a hassle right? | 16:43 |
| *** dtantsur is now known as dtantsur|afk | 16:44 | |
| *** agopi is now known as agopi|lunch | 16:44 | |
| weshay | rlandy, re: your questions | 16:47 |
| weshay | <rlandy> agopi: rook: weshay: it was my impression that all the role and playbook would be committed to the browbeat repo | 16:48 |
| weshay | agopi|lunch, rlandy in the cases we need to fix something | 16:48 |
| weshay | agopi|lunch, how does one get core on browbeat rook ? | 16:49 |
| rook | weshay we all go out drinking, and if you drink the most, you get core. | 16:51 |
| weshay | challenge accepted .. < said in barney stinson voice > | 16:51 |
| rook | :D | 16:51 |
| rook | weshay there isn't a "path". Mostly based on commits/reviews and being present in the project. | 16:51 |
| weshay | team w/ highest drinking performance wins | 16:51 |
| rook | haha | 16:52 |
| rook | Our project is pretty much just our team | 16:52 |
| *** d0ugal has quit IRC | 16:52 | |
| rook | We have had commits here and there.. | 16:52 |
| rook | from other companies | 16:52 |
| weshay | rook, k.. so if something fails say in the upstream browbeat job.. we can make it non-voting, fix it, or temp remove the job | 16:52 |
| rook | but nothing consistent.. | 16:52 |
| * weshay just figuring out the logistics of that | 16:52 | |
| rook | ack - yeah | 16:52 |
| rook | I think that is how myoung operated with it when Justin was around | 16:52 |
| weshay | rasca, same goes for you | 16:53 |
| rook | like, browbeat is fucked. disabling it. | 16:53 |
| weshay | rook, ya.. that won't fly so well upstream.. but if the long term path is to have some cross section of cores | 16:53 |
| weshay | I'm ok w/ it | 16:53 |
| weshay | that would include folks like tripleo cores | 16:53 |
| weshay | get more folks working in browbeat, reviewing it.. etc.. | 16:54 |
| weshay | probably a good soft goal anyway | 16:54 |
| rook | ack weshay | 16:54 |
| rasca | weshay, I read just the part in which you talked about drinking with rook. I'm totally fine with that. | 16:54 |
| rook | :D | 16:54 |
| weshay | rasca, I think ha-utils is under the tripleo umbrella | 16:54 |
| rook | PTG, game on. | 16:54 |
| weshay | so tripleo cores have core | 16:54 |
| * weshay puts on my drinking boots | 16:55 | |
| rlandy | drinking aside - what's the verdict on where the playbook/roles reside? | 16:57 |
| rlandy | weshay: agopi|lunch: rook: ^^ I am fine with placing the role/playbook in either place - just want agreement on it | 16:59 |
| weshay | rlandy, right.. so if it's all in browbeat rook is responsible for approving fixes if it breaks | 16:59 |
| rook | I could make rlandy core | 16:59 |
| *** zoli is now known as zoli|gone | 16:59 | |
| rook | w/ agopi | 16:59 |
| agopi|lunch | weshay, rlandy rook i'd prefer having most of it on browbeat so that the roles dont need to be tied up | 16:59 |
| rlandy | unless you out drink him? | 16:59 |
| rook | for browbeat | 16:59 |
| *** zoli|gone is now known as zoli | 16:59 | |
| weshay | rlandy, ideally we earn some trust and faith and have some cross over cores | 16:59 |
| weshay | agopi|lunch, ya.. I think having the playbooks and roles in browbeat is the right thing | 17:00 |
| rook | I trust agopi and rlandy that if it isn't CI/Infra for CI, they will pass on pulling in things... | 17:00 |
| *** amoralej is now known as amoralej|off | 17:00 | |
| weshay | rlandy, so.. if it's a check job and failing.. we'd have to move it non-voting | 17:00 |
| rook | Yeah, I think we are in total agreement on that. | 17:00 |
| weshay | that's a slight pita | 17:00 |
| weshay | rlandy, and in promotion jobs.. remove it from criteria | 17:00 |
| rlandy | weshay: ok - so if we keep it as is in https://review.openstack.org/#/c/581488/ | 17:01 |
| rlandy | we are not holding much code | 17:01 |
| rlandy | we can always edit the fs to not run | 17:01 |
| weshay | agopi|lunch, rook we'll probably want to train up a couple folks from your team on upstream ci | 17:01 |
| weshay | maybe get them to be oooq core as well | 17:01 |
| rlandy | weshay: being oooq core is a lot of responsibility for one role and one playbook | 17:02 |
| hubbot` | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-scenario009-multinode-oooq, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000 (1 more message) | 17:02 |
| weshay | rlandy, the right idea though is to do more work upstream | 17:03 |
| weshay | from both groups | 17:03 |
| weshay | rlandy, what does trigger-browbeat do? | 17:03 |
| agopi|lunch | weshay, not sure about having to be core but i'd like to be trained on upstream ci so that we move along with yall. | 17:04 |
| weshay | agopi|lunch, ya.. | 17:04 |
| weshay | +1 | 17:04 |
| agopi|lunch | weshay, it'll just call a bootstrapping script that i'nm gonna add on the browbeat repo that will setup adn do the work | 17:04 |
| weshay | rlandy, minus my question, looks fine https://review.openstack.org/#/c/581488/7 | 17:04 |
| agopi|lunch | sorry weshay i use vi | 17:05 |
| rlandy | trigger-browbeat call the .sh script to do <some magic> | 17:05 |
| rlandy | ok - to be pedantic - the role and the playbook stays in tqe??? | 17:05 |
| *** d0ugal has joined #oooq | 17:06 | |
| weshay | the role? | 17:06 |
| weshay | rlandy, playbooks can live anywhere | 17:06 |
| weshay | that doesn't matter so much | 17:06 |
| rlandy | https://review.openstack.org/#/c/581488/7/roles/trigger-browbeat/tasks/main.yml | 17:06 |
| weshay | the role does | 17:06 |
| weshay | rlandy, ya.. I don't know what that does | 17:06 |
| * rlandy neither | 17:07 | |
| weshay | jebus | 17:07 |
| * rlandy gets link | 17:07 | |
| weshay | agopi|lunch, rlandy it should just be.. browbeat is added to quickstart-extras-requirements.txt | 17:07 |
| weshay | agopi|lunch, can you explain a bit when you are back from lunch | 17:08 |
| weshay | I don't think we want or need a role here | 17:08 |
| agopi|lunch | weshay, i was thinking of removing browbeat from quickstart-extras-requirements.txt | 17:08 |
| weshay | agopi|lunch, and replacing it w/ a role? | 17:08 |
| agopi|lunch | yeah, browbeat will be cloned anyways amirite? | 17:09 |
| rlandy | https://github.com/openstack/browbeat/tree/master/ci-scripts/tripleo | 17:09 |
| agopi|lunch | i've not yet created it rlandy | 17:09 |
| agopi|lunch | i'm working on it | 17:09 |
| weshay | agopi|lunch, so which script there? | 17:09 |
| agopi|lunch | i've not yet created it weshay, i'm on it. | 17:10 |
| agopi|lunch | it'll be something liek https://github.com/openstack/browbeat/blob/master/ci-scripts/tripleo/microbrow-browbeat-ci.sh | 17:10 |
| agopi|lunch | but less variables | 17:10 |
| rlandy | will rut run quickstart? | 17:10 |
| rlandy | run | 17:10 |
| weshay | agopi|lunch, I don't think any of that is needed | 17:10 |
| weshay | agopi|lunch, let me know when you have a minute to chat | 17:11 |
| agopi|lunch | sure thing weshay | 17:11 |
| rook | we are bolting this on after a overcloud is built right? | 17:11 |
| agopi|lunch | yes rook | 17:11 |
| rlandy | yes | 17:11 |
| myoung | rook, weshay, regarding how Justin and I operated in the deep past w.r.t. voting. INitially when getting all this working (prior to featuresets, and the other improvements made over the last year - including the notion of the promoter(s), we create a non-voting jenkins pipeline, so that browbeat jobs could run and populate data/history, but would not block promotions. | 17:11 |
| agopi|lunch | we arent gonna run quickstart | 17:11 |
| rlandy | all set up | 17:11 |
| rook | right | 17:11 |
| * agopi|lunch waves to myoung | 17:11 | |
| myoung | rook, weshay, justin presented a failly detailed overview to RelDel, and the intent was to move in future releases to having browbeat (and perf ML metrics / alerts) actually vote | 17:12 |
| * myoung waves at agopi|lunch | 17:12 | |
| rook | yup myoung that work was what agopi|lunch was cranking on | 17:12 |
| rook | :) | 17:12 |
| rook | ML/AI | 17:12 |
| rlandy | maybe let's see what agopi|lunch has in mind for the script first | 17:12 |
| rlandy | then we can review | 17:12 |
| weshay | rlandy, it's the wrong course | 17:12 |
| myoung | brb | 17:12 |
| agopi|lunch | okay weshay im open for suggestions | 17:13 |
| weshay | agopi|lunch, we have a config file just for browbeat.. a featureset file like anything else | 17:13 |
| weshay | turn on / off what ever you want ... grafrana_enable: true etc | 17:14 |
| weshay | do not pass that via a shell script | 17:14 |
| weshay | agopi|lunch, a playbook is designed to call any set of roles.. the playbook that runs browbeat in tripleo could be hosted in browbeat or tq.. doesn't really matter | 17:15 |
| weshay | the role is obviously hosted in browbeat | 17:15 |
| agopi|lunch | https://review.openstack.org/#/c/581484/3/config/general_config/featureset053.yml you mean add the vars here | 17:15 |
| rook | so - we just tell/pass you (being tq) what playbook to run, ie browbeat/ansible/install.yaml with a specific group_vas/all.yml... Then pass a minimal browbeat_run.yml to browbeat. | 17:15 |
| weshay | agopi|lunch, calling quickstart.sh is also deprecated.. calling ansible-playbook directly is required | 17:15 |
| weshay | agopi|lunch, yes.. that is exactly what the file is for | 17:16 |
| agopi|lunch | ahhh got it | 17:17 |
| agopi|lunch | rook, so we just create a version of https://github.com/openstack/browbeat/blob/master/ansible/oooq/baremetal-virt-undercloud-int-browbeat.yml | 17:17 |
| weshay | rook, so each part of upstream ci is broken down in sections... called via ansible-playbook | 17:17 |
| weshay | ansible-playbook install undercloud | 17:17 |
| weshay | ansible-playbook prep overcloud | 17:17 |
| weshay | ansible-playbook deploy overcloud | 17:17 |
| weshay | etc | 17:17 |
| rook | sure | 17:17 |
| weshay | rook, so after the deployments runs and we run some minimal validation | 17:18 |
| rlandy | rook: correct agopi|lunch had a minimal playbook | 17:18 |
| rlandy | we already discussed this | 17:18 |
| weshay | we run ansible-playbook minimal-browbeat | 17:18 |
| rook | so ansible-playbook prep-browbeat, run-browbeat | 17:18 |
| weshay | ya | 17:18 |
| rlandy | that the env would be set up | 17:18 |
| weshay | how does that sound? | 17:18 |
| weshay | https://review.openstack.org/#/c/581488/7/playbooks/baremetal-quickstart-extras.yml | 17:19 |
| rlandy | https://gist.github.com/aakarshg/73b4a1a7a87b0867d816ce94bb83f76e | 17:19 |
| weshay | so this makes sense to me ^ | 17:19 |
| weshay | but need to add browbeat-prep.yml | 17:19 |
| weshay | rook, agopi|lunch and in the feature config file https://review.openstack.org/#/c/581484/3/config/general_config/featureset053.yml | 17:19 |
| weshay | enable_minimal_browbeat is set to true | 17:20 |
| weshay | no shell script | 17:20 |
| weshay | nothing unique in the cli invocation that is different than any other job | 17:20 |
| rlandy | pls hold off on editing https://review.openstack.org/#/c/581484/ - I am fixing the merge issue | 17:22 |
| *** agopi|lunch is now known as agopi | 17:32 | |
| rlandy | https://review.openstack.org/#/c/581484/ - updated | 17:33 |
| rlandy | agopi: weshay: ^^ | 17:33 |
| agopi | ack rlandy | 17:34 |
| rlandy | marios: sshnaidm|bbl: ^^ | 17:35 |
| rlandy | that review can merge - won't do any damage w/o the call to run it | 17:35 |
| *** florianf has quit IRC | 17:37 | |
| *** trown|lunch is now known as trown | 17:37 | |
| agopi | rlandy, i'll be adding vars to fs053.yml | 17:38 |
| weshay | agopi, +1 | 17:47 |
| weshay | agopi, that config file is all yours | 17:47 |
| weshay | agopi, just don't post any passwords to it :)) | 17:47 |
| agopi | weshay, lol | 17:47 |
| *** ykarel|away has joined #oooq | 17:49 | |
| myoung | agopi, rook, are you folks changing the internal browbeat (microbrow) rigs (actual BM with metrics history) to also use the featureset design going into upstream configs as well? | 17:57 |
| myoung | or is that another phase/step/hunk of work? | 17:57 |
| myoung | (apologies, been on PTO for a few weeks) | 17:57 |
| agopi | no myoung, the perfci remains as it is. | 17:57 |
| myoung | agopi: is there a plan to have those running with the same (tested) config as upstream? | 17:58 |
| myoung | eventually? | 17:58 |
| agopi | not that i know of. | 18:00 |
| rlandy | agpoi: ack - when you are done, can edit https://review.openstack.org/#/c/581488/ accordingly | 18:00 |
| agopi | rlandy, removing the files i added. | 18:01 |
| rlandy | agopi: sorry for confusing | 18:01 |
| agopi | now i understand why theyre not needed | 18:01 |
| agopi | no rlandy, it was i who couldn't quite catch the flow. | 18:01 |
| *** d0ugal has quit IRC | 18:04 | |
| *** jfrancoa has quit IRC | 18:13 | |
| myoung | agopi, rook, ack, IMHO would make sense to create a plan to converge the actual perfci (kicked off from internal jenkins today) to run the same configs/tooling as upstream, once that stabilizes. This could perhaps be facilitated by internal SF in the future (or some other SF deployment), or done some other way (I can think of a few)...but IMHO given our experience over the past couple years upstream/downstream drift never ends well. | 18:15 |
| myoung | weshay: ^^ | 18:15 |
| weshay | ya.. that is the plan. It just hasn't soaked in yet | 18:16 |
| *** d0ugal has joined #oooq | 18:17 | |
| myoung | weshay: ack, cool. | 18:24 |
| myoung | weshay: don't want to replicate drift-induced pain (ie 2017) if we can avoid it. | 18:24 |
| *** ykarel|away has quit IRC | 18:28 | |
| rook | agopi: join | 18:30 |
| *** d0ugal has quit IRC | 18:46 | |
| *** ykarel|away has joined #oooq | 18:49 | |
| *** d0ugal has joined #oooq | 18:50 | |
| hubbot` | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-scenario009-multinode-oooq, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000 (1 more message) | 19:02 |
| *** d0ugal has quit IRC | 19:02 | |
| *** d0ugal has joined #oooq | 19:05 | |
| *** chem has quit IRC | 19:11 | |
| radez | rlandy: hey, I've gotten a base delpoyment to succeed now and I'm trying to get my feature to deploy. When you have a second would you mind looking at an error with me? | 19:12 |
| rlandy | radez: sure - pls paste | 19:15 |
| radez | this is the error: http://paste.openstack.org/show/726043/ | 19:16 |
| weshay | # uptime | 19:16 |
| weshay | 15:16:00 up 665 days, 5:18, 1 user, load average: 0.00, 0.01, 0.05 | 19:16 |
| weshay | radez, I think it's OUR feature | 19:16 |
| radez | this is the puppet module https://review.openstack.org/#/c/577073/3/manifests/plugins/ml2/networking_ansible.pp | 19:16 |
| weshay | this IS open source | 19:16 |
| radez | weshay: just you wait... it sure will be :) | 19:16 |
| weshay | ah there's ur problem.. it uses puppet | 19:17 |
| * weshay runs | 19:17 | |
| radez | this is the THT template that defines that puppet variable | 19:17 |
| radez | https://review.openstack.org/#/c/577620/5/puppet/services/neutron-plugin-ml2-ansible.yaml | 19:17 |
| radez | rlandy: so my question is, can I put that in my env_template some how? | 19:17 |
| radez | i tried this... and it didn't work: (pasting now) | 19:17 |
| *** ykarel|away has quit IRC | 19:18 | |
| radez | http://paste.openstack.org/show/726044/ | 19:18 |
| radez | I put that in my env_settings file | 19:18 |
| radez | but got the same error with and without that section in it | 19:18 |
| rlandy | ah .. ok - that should be included with the deploy args? | 19:19 |
| radez | YES? i THINK THAT'S RIGHT | 19:20 |
| radez | oops, sry | 19:20 |
| radez | so maybe I need and extra file with that section in it that gets passed with the -e args right? | 19:21 |
| rlandy | we have a bunch of examples ... | 19:21 |
| radez | is there a way to inject that into oooq? | 19:21 |
| rlandy | one of which is the network parameters | 19:21 |
| radez | oooh, examples :) | 19:21 |
| rlandy | https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/overcloud-prep-config/templates/network-environment.yaml.j2 | 19:22 |
| radez | I may need to add to network params too so that would be a helpful exmaple | 19:22 |
| rlandy | with vars set in | 19:22 |
| rlandy | https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/overcloud-prep-config/defaults/main.yml#L15 | 19:23 |
| rlandy | you'd need to add your file to the deploy command | 19:23 |
| rlandy | https://github.com/openstack/tripleo-quickstart-extras/search?q=parameter_defaults&unscoped_q=parameter_defaults | 19:23 |
| rlandy | ^^ will show diff example about how we have done that | 19:24 |
| rlandy | if you create a new file, and you want to add it to the deploy command, | 19:24 |
| radez | ah, I see | 19:24 |
| rlandy | you can add it to ttps://github.com/openstack/tripleo-quickstart/blob/master/config/general_config/featureset001.yml#L77 | 19:25 |
| rlandy | extra_args | 19:25 |
| radez | ok so for now I could just jam that stuff into the network params to test it and then later split it out to its own file? | 19:25 |
| rlandy | you can create a new file | 19:25 |
| rlandy | and we will add new args | 19:25 |
| rlandy | in the mean time, you can add your file to extra_args | 19:25 |
| rlandy | if you jam it into network params just to test, that is fine | 19:26 |
| radez | perfect, thx much rlandy | 19:26 |
| rlandy | but to be accepted into CI, we should be more exact and give you a new file - of that is required | 19:26 |
| rlandy | test it out and let us know, we will help you from there | 19:26 |
| rlandy | https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/overcloud-deploy/defaults/main.yml#L108 | 19:30 |
| rlandy | ^^ are all the deploy args - you can add one to the list | 19:30 |
| rook | lol @ weshay it uses puppet | 19:39 |
| rook | q | 19:39 |
| *** holser_ has quit IRC | 19:56 | |
| *** sshnaidm|bbl has quit IRC | 20:00 | |
| weshay | :) | 20:00 |
| weshay | anything to mess w/ radez | 20:00 |
| *** sshnaidm|bbl has joined #oooq | 20:00 | |
| radez | heh, well I just got called out :) | 20:00 |
| *** holser_ has joined #oooq | 20:01 | |
| *** d0ugal has quit IRC | 20:16 | |
| rlandy | weshay: hello ... https://github.com/openstack/tripleo-quickstart/blob/master/config/nodes/2ctlr.yml - okie dokie ... why do we reference hostvars for the hostnamemap and /etc/nodepool/sub_nodes_private for the deployed_server_overcloud_roles? | 20:22 |
| *** d0ugal has joined #oooq | 20:22 | |
| rlandy | http://logs.openstack.org/76/581376/6/check/tripleo-ci-centos-7-3nodes-multinode/4708903/logs/undercloud/etc/nodepool/ | 20:23 |
| rlandy | are empty archives | 20:23 |
| weshay | rlandy, that was used to validate hostname maps | 20:23 |
| weshay | oh.. this may explain why 3ctrl is not working? | 20:24 |
| weshay | er.. 3node? | 20:24 |
| rlandy | I think so | 20:24 |
| rlandy | but I can't see what is in /etc/nodepool/sub_nodes_private | 20:24 |
| weshay | rlandy, it should work w/ or w/o hostname maps though | 20:24 |
| rlandy | if we use nodes/2ctlr.yml we are using hostnamemaps | 20:24 |
| weshay | ya | 20:25 |
| rlandy | I can see hosts file | 20:25 |
| rlandy | http://logs.openstack.org/76/581376/6/check/tripleo-ci-centos-7-3nodes-multinode/4708903/logs/undercloud/etc/hosts.txt.gz | 20:25 |
| weshay | rlandy, can you see the hostvars for the 3node job? | 20:25 |
| rlandy | http://logs.openstack.org/76/581376/6/check/tripleo-ci-centos-7-3nodes-multinode/4708903/logs/undercloud/var/log/extra/dump_variables_hostvars.json.txt.gz | 20:26 |
| rlandy | I am comparing the twp | 20:26 |
| rlandy | two | 20:26 |
| rlandy | parameter_defaults: | 20:26 |
| rlandy | HostnameMap: | 20:26 |
| rlandy | overcloud-controller-0: centos-7-inap-mtl01-0000693370 | 20:26 |
| rlandy | overcloud-controllerapi-0: centos-7-inap-mtl01-0000693368 | 20:26 |
| rlandy | looks correct | 20:26 |
| weshay | http://logs.openstack.org/76/581376/6/check/tripleo-ci-centos-7-3nodes-multinode/4708903/logs/undercloud/home/zuul/hostnamemap.yaml.txt.gz | 20:27 |
| weshay | ya.. but it's only picking up two | 20:28 |
| rlandy | http://logs.openstack.org/76/581376/6/check/tripleo-ci-centos-7-3nodes-multinode/4708903/logs/undercloud/var/lib/mistral/4ab3a2fa-a5cc-418a-b3d8-6e27e8a5f07c/ControllerApi/centos-7-inap-mtl01-0000693368/ correct | 20:28 |
| weshay | oh that's ok | 20:28 |
| rlandy | this is incorrect ... | 20:28 |
| rlandy | http://logs.openstack.org/76/581376/6/check/tripleo-ci-centos-7-3nodes-multinode/4708903/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2018-07-12_22_59_54 | 20:28 |
| rlandy | ControllerApi/centos-7-inap-mtl01-0000693370/UpgradeInitDeployment would not exist | 20:29 |
| rlandy | overcloud-controllerapi-0: centos-7-inap-mtl01-0000693368 | 20:29 |
| rlandy | https://github.com/openstack/tripleo-common/blob/master/tripleo_common/templates/deployments.yaml#L3 reads the wrong host | 20:29 |
| rlandy | what is ~ item | 20:30 |
| rlandy | that is the question | 20:30 |
| rlandy | undercloud/etc/hosts.txt.gz | 20:30 |
| rlandy | --------------------------- | 20:30 |
| rlandy | 198.72.124.176 centos-7-inap-mtl01-0000693368 | 20:30 |
| rlandy | 198.72.124.176 secondary-1 | 20:30 |
| rlandy | 198.72.124.184 centos-7-inap-mtl01-0000693370 | 20:30 |
| rlandy | 198.72.124.184 secondary-2 | 20:30 |
| weshay | "groups": { | 20:31 |
| weshay | "all": [ | 20:31 |
| weshay | "undercloud", | 20:31 |
| weshay | "localhost", | 20:31 |
| weshay | "subnode-2", | 20:31 |
| weshay | "subnode-3", | 20:31 |
| weshay | "127.0.0.2" | 20:31 |
| weshay | ], | 20:31 |
| weshay | "overcloud": [ | 20:31 |
| weshay | "subnode-2", | 20:31 |
| weshay | "subnode-3" | 20:31 |
| weshay | ], | 20:31 |
| weshay | "ungrouped": [ | 20:31 |
| weshay | "undercloud", | 20:31 |
| weshay | "localhost" | 20:31 |
| weshay | ], | 20:31 |
| rlandy | that is ok | 20:31 |
| weshay | "virthost": [ | 20:31 |
| weshay | "127.0.0.2" | 20:31 |
| weshay | ] | 20:31 |
| rlandy | "subnode-3" is secondary-1 which is centos-7-inap-mtl01-0000693368 | 20:32 |
| weshay | k.. this is what turns it on btw https://github.com/openstack-infra/tripleo-ci/blob/master/toci-quickstart/config/testenv/multinode.yml#L15 | 20:32 |
| rlandy | so ... if | 20:33 |
| rlandy | https://github.com/openstack/tripleo-common/blob/master/tripleo_common/templates/deployments.yaml#L3 | 20:33 |
| rlandy | references | 20:33 |
| rlandy | overcloud groups | 20:33 |
| rlandy | directly as item | 20:33 |
| rlandy | then subnode-2 is first | 20:34 |
| *** d0ugal has quit IRC | 20:34 | |
| rlandy | but nowhere is that defined as the controllerapi | 20:34 |
| rlandy | so the question stands ... /etc/nodepool/sub_nodes_private what is in here? | 20:36 |
| rlandy | and | 20:36 |
| rlandy | what is https://github.com/openstack/tripleo-common/blob/master/tripleo_common/templates/deployments.yaml#L3 referencing? | 20:36 |
| weshay | rlandy, have you tried to getthelogs? | 20:36 |
| rlandy | unless we can see /etc/nodepool/sub_nodes_private, idk | 20:36 |
| weshay | rlandy, /me trying getthelogs | 20:37 |
| rlandy | http://logs.openstack.org/76/581376/6/check/tripleo-ci-centos-7-3nodes-multinode/4708903/logs/undercloud/etc/nodepool/ | 20:38 |
| rlandy | ^^ need those | 20:38 |
| rlandy | ha - got it | 20:39 |
| rlandy | weshay: ^^ | 20:39 |
| rlandy | [rlandy@rlandy temp]$ cat sub_nodes_private | 20:40 |
| rlandy | 198.72.124.184 | 20:40 |
| rlandy | 198.72.124.176 | 20:40 |
| rlandy | weshay: ok - got it now - | 20:41 |
| rlandy | https://github.com/openstack/tripleo-quickstart/blob/master/config/nodes/2ctlr.yml#L16 | 20:41 |
| rlandy | ^^ would set ControllerApi to 198.72.124.184 - which is centos-7-inap-mtl01-0000693370 | 20:41 |
| *** d0ugal has joined #oooq | 20:47 | |
| hubbot` | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-3nodes-multinode @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-scenario009-multinode- (1 more message) | 21:02 |
| weshay | rlandy, where did you find the content to sub_nodes_private? | 21:05 |
| rlandy | I just did a wget on the file | 21:09 |
| rlandy | which worked | 21:09 |
| rlandy | http://logs.openstack.org/76/581376/6/check/tripleo-ci-centos-7-3nodes-multinode/4708903/logs/undercloud/etc/nodepool/sub_nodes_private.gz | 21:09 |
| rlandy | idk if /etc/nodepool/sub_nodes_private and hostvars[groups['overcloud'] are **required** to match | 21:10 |
| weshay | huh k | 21:13 |
| rlandy | they could have just matched by default before | 21:17 |
| rlandy | or maybe that is a requirement | 21:17 |
| rlandy | weshay: ^^? | 21:23 |
| weshay | I didn't think the two were DIRECTLY related tbh | 21:24 |
| weshay | it's been awhile since I touched that code | 21:24 |
| rlandy | so then, we could use hostvars in both | 21:24 |
| weshay | rlandy, this was the feature request from MTB minus 1 letter on each | 21:24 |
| weshay | oops | 21:24 |
| weshay | OTB | 21:24 |
| rlandy | oh | 21:26 |
| rlandy | I see | 21:26 |
| weshay | I'm a little out of context | 21:26 |
| weshay | on how I did that | 21:26 |
| weshay | need to spend a few minutes w/ it | 21:26 |
| weshay | unless you are good now | 21:26 |
| weshay | sounds like you are | 21:26 |
| weshay | but happy to dig into it | 21:26 |
| rlandy | I'll leave some notes - we can review later | 21:27 |
| weshay | k | 21:27 |
| weshay | rlandy, fyi.. I have a local osp13 deployment going | 21:30 |
| weshay | on libvirt | 21:30 |
| rlandy | did your undercloud install pass? | 21:30 |
| weshay | yes | 21:31 |
| rlandy | oh how nice | 21:31 |
| weshay | I'm on prep containers | 21:31 |
| rlandy | same release file? | 21:31 |
| weshay | rlandy, so the undercloud is also failing in master rdo atm | 21:31 |
| weshay | rhos-13 | 21:31 |
| weshay | rlandy, I did see some config missing | 21:33 |
| weshay | from fs001 | 21:33 |
| rlandy | rebase> | 21:33 |
| weshay | rlandy, can I send a test patch to fs001 and we can try it? | 21:33 |
| weshay | on osp-13? | 21:33 |
| rlandy | sure | 21:33 |
| weshay | k sec | 21:33 |
| weshay | rlandy, https://review.openstack.org/583051 | 21:35 |
| rlandy | maybe that was it? | 21:36 |
| rlandy | will try in a bit | 21:36 |
| weshay | rlandy, thanks | 21:38 |
| rlandy | https://github.com/openstack-infra/openstack-zuul-jobs/blob/master/playbooks/legacy/pre.yaml#L21 just checking how that is populated | 21:38 |
| weshay | rlandy, /me also noticing in libvirt the centos image is selinux enforcing | 21:38 |
| weshay | https://review.openstack.org/583054 | 21:40 |
| weshay | rlandy, that probably also applies to fs35 | 21:44 |
| rlandy | weshay: ^^ that prevents us running on rhel by default? | 21:45 |
| weshay | rlandy, there would be an env file for rhel | 21:45 |
| weshay | rlandy, and rhel should be permissive for rdo | 21:46 |
| weshay | https://code.engineering.redhat.com/gerrit/gitweb?p=tripleo-environments.git;a=tree;f=config/environments;h=fb11ba49b560911180c0a3feed15daf30483b021;hb=HEAD | 21:46 |
| weshay | rlandy, atm introspection is hanging for me on centos + master | 21:47 |
| weshay | pretty sure it's selinux | 21:47 |
| rlandy | hmm - ok | 21:48 |
| agopi | rlandy, so one of the scenario/tests we run, requires the overcloud to have images. Is it possible to have overcloud have centos7 and cirros images? | 21:48 |
| weshay | agopi, why wouldn't the browbeat role pull those if they were not there | 21:48 |
| weshay | if browbeat requires it | 21:48 |
| rlandy | something different to what tempest uses? | 21:48 |
| weshay | that is what ansible does :) | 21:49 |
| panda|travel | rfolco_: still here ? | 21:49 |
| *** panda|travel is now known as panda | 21:49 | |
| agopi | yes there's a browbeat role that can do that, but it goes https://github.com/openstack/browbeat/blob/master/ansible/install/group_vars/all.yml#L98 and then downloads it | 21:49 |
| weshay | agopi, k.. so what is the question then.. it should pull those two images as you expect | 21:50 |
| agopi | i was thinking in order to reduce time further more it takes around 4-5 mins | 21:51 |
| weshay | agopi, rlandy I would not optimize for time on the first pass | 21:51 |
| weshay | optimize on getting the job up and running :) | 21:51 |
| agopi | ack weshay | 21:52 |
| weshay | agopi, rlandy it's only going to check two repos.. tqe and browbeat | 21:52 |
| rlandy | tq also | 21:52 |
| weshay | sure | 21:52 |
| *** holser_ has quit IRC | 22:08 | |
| agopi | rook, https://review.openstack.org/#/c/583011/ can we merge this. | 22:10 |
| *** holser_ has joined #oooq | 22:18 | |
| agopi | rlandy, once the above commit lands, we can try triggering. | 22:18 |
| agopi | fingers crossed. i feel like im missing something, but it's not hitting me | 22:19 |
| rlandy | cool - looking forward | 22:19 |
| rlandy | weshay: trying this for the hosts ordering issue - https://review.openstack.org/#/c/581376/7/playbooks/openstack-zuul-jobs/legacy/pre.yaml | 22:20 |
| rlandy | not sure it will make a difference | 22:20 |
| rlandy | ok - trying rhos-13 stuff now | 22:26 |
| *** agopi is now known as agopi|brb | 22:29 | |
| *** dalvarez has quit IRC | 22:30 | |
| *** radez has quit IRC | 22:33 | |
| *** agopi|brb has quit IRC | 22:34 | |
| *** amoralej|off has quit IRC | 22:34 | |
| weshay | rlandy, osp-13 | 22:43 |
| weshay | Monday 16 July 2018 18:42:16 -0400 (0:00:00.051) 2:26:45.816 *********** | 22:43 |
| weshay | ok: [undercloud] => { | 22:43 |
| weshay | "overcloud_deploy_result": "passed" | 22:43 |
| weshay | } | 22:43 |
| rlandy | nice | 22:43 |
| rlandy | the libvirt flavors are different | 22:45 |
| rlandy | so I suspect you never hit the node issue we are hitting in ovb | 22:46 |
| *** ccamacho has quit IRC | 22:51 | |
| weshay | rlandy, oh the size issue | 22:55 |
| weshay | we're still stuck there.. waiting on IT ya? | 22:56 |
| *** holser_ has quit IRC | 22:56 | |
| weshay | rlandy, can you please review this https://review.openstack.org/#/c/581789/ | 22:56 |
| weshay | marios has +2 on it too | 22:56 |
| weshay | but.. ya know | 22:57 |
| rlandy | I have a ptach to modify falvors | 22:57 |
| weshay | rlandy, k | 22:57 |
| weshay | rlandy, should I merge your flavor patch? | 22:57 |
| weshay | where is that | 22:57 |
| weshay | https://review.openstack.org/#/c/579406/4/roles/overcloud-prep-flavors/templates/overcloud-prep-flavors.sh.j2 | 22:57 |
| rlandy | https://review.openstack.org/#/c/579406/ | 22:58 |
| rlandy | reviewing | 23:00 |
| weshay | f.. I think master has an ironic problem | 23:00 |
| hubbot` | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-3nodes-multinode @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-scenario009-multinode- (1 more message) | 23:02 |
| weshay | ya. introspection is borked | 23:04 |
| rlandy | how so if deploy passed? | 23:06 |
| rlandy | introspection is false on vibvirt | 23:06 |
| weshay | rlandy, so master it's failing, osp-13 worked | 23:07 |
| rlandy | oh | 23:07 |
| weshay | rlandy, it's just timing out | 23:07 |
| rlandy | just | 23:07 |
| *** tosky has quit IRC | 23:15 | |
| *** vinaykns has quit IRC | 23:24 | |
| *** vinaykns has joined #oooq | 23:24 | |
| weshay | rlandy, heh | 23:28 |
| *** vinaykns has quit IRC | 23:28 | |
| *** agopi|brb has joined #oooq | 23:30 | |
| *** agopi|brb is now known as agopi | 23:31 | |
| *** dtrainor has quit IRC | 23:31 | |
| *** radez has joined #oooq | 23:47 | |
| *** vinaykns has joined #oooq | 23:55 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!