*** eil397 has quit IRC | 00:00 | |
*** ooolpbot has joined #tripleo | 00:10 | |
ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 00:10 |
---|---|---|
ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1543810 | 00:10 |
*** ooolpbot has quit IRC | 00:10 | |
openstack | Launchpad bug 1543810 in tripleo "Jenkins errors killing many CI jobs" [Critical,Triaged] | 00:10 |
*** Marga__ has joined #tripleo | 00:15 | |
*** Marga_ has quit IRC | 00:17 | |
openstackgerrit | ayoung proposed openstack/tripleo-heat-templates: puppet: run keystone in wsgi https://review.openstack.org/213175 | 00:21 |
*** trown is now known as trown|outtypewww | 00:23 | |
openstackgerrit | John Trowbridge proposed openstack/instack-undercloud: Update nova-neutron configuration https://review.openstack.org/278137 | 00:30 |
*** xarses has quit IRC | 00:35 | |
*** eil397 has joined #tripleo | 00:45 | |
*** lblanchard has joined #tripleo | 00:53 | |
*** xarses has joined #tripleo | 00:56 | |
openstackgerrit | greghaynes proposed openstack/diskimage-builder: Fail functests if refusing to run tests https://review.openstack.org/177002 | 01:00 |
*** penick has joined #tripleo | 01:02 | |
*** eil397 has quit IRC | 01:06 | |
*** Marga__ has quit IRC | 01:06 | |
*** Marga_ has joined #tripleo | 01:07 | |
*** ron___ has joined #tripleo | 01:08 | |
*** ooolpbot has joined #tripleo | 01:10 | |
ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 01:10 |
ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1543810 | 01:10 |
*** ooolpbot has quit IRC | 01:10 | |
openstack | Launchpad bug 1543810 in tripleo "Jenkins errors killing many CI jobs" [Critical,Triaged] | 01:10 |
*** weshay_xchat has quit IRC | 01:10 | |
*** weshay_xchat has joined #tripleo | 01:16 | |
*** jayg is now known as jayg|g0n3 | 01:17 | |
*** dmacpher-afk has quit IRC | 01:19 | |
*** xarses has quit IRC | 01:19 | |
openstackgerrit | Graeme Gillies proposed openstack/instack-undercloud: Added a new element to configure ansible inventory from nova https://review.openstack.org/277688 | 01:31 |
*** shakamunyi has quit IRC | 01:40 | |
*** xarses has joined #tripleo | 01:44 | |
*** tiswanso has joined #tripleo | 01:50 | |
*** tiswanso has quit IRC | 01:50 | |
*** tiswanso has joined #tripleo | 01:51 | |
*** bswartz has joined #tripleo | 01:51 | |
*** weshay_xchat has quit IRC | 01:58 | |
*** dmacpher has joined #tripleo | 02:03 | |
*** ooolpbot has joined #tripleo | 02:10 | |
ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 02:10 |
ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1543810 | 02:10 |
*** ooolpbot has quit IRC | 02:10 | |
openstack | Launchpad bug 1543810 in tripleo "Jenkins errors killing many CI jobs" [Critical,Triaged] | 02:10 |
*** shivrao has quit IRC | 02:11 | |
*** yamahata has quit IRC | 02:19 | |
*** links has joined #tripleo | 03:00 | |
openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Fix startup race with growroot/systemd https://review.openstack.org/278187 | 03:04 |
openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Refactor growroot for debuggabilty https://review.openstack.org/278188 | 03:04 |
*** Marga_ has quit IRC | 03:05 | |
*** ooolpbot has joined #tripleo | 03:10 | |
ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 03:10 |
ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1543810 | 03:10 |
*** ooolpbot has quit IRC | 03:10 | |
openstack | Launchpad bug 1543810 in tripleo "Jenkins errors killing many CI jobs" [Critical,Triaged] | 03:10 |
*** shivrao has joined #tripleo | 03:19 | |
*** ron___ has quit IRC | 03:19 | |
*** shivrao has quit IRC | 03:29 | |
openstackgerrit | Matthew Thode proposed openstack/diskimage-builder: Remove eclean-dist as it's not available by default https://review.openstack.org/276581 | 04:01 |
*** ooolpbot has joined #tripleo | 04:10 | |
ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 04:10 |
ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1543810 | 04:10 |
openstack | Launchpad bug 1543810 in tripleo "Jenkins errors killing many CI jobs" [Critical,Triaged] | 04:10 |
*** ooolpbot has quit IRC | 04:10 | |
*** tiswanso has quit IRC | 04:20 | |
*** bhunter has quit IRC | 04:33 | |
*** michchap_ has joined #tripleo | 04:33 | |
*** bhunter has joined #tripleo | 04:34 | |
*** michchap has quit IRC | 04:35 | |
*** rbrady has quit IRC | 04:44 | |
openstackgerrit | greghaynes proposed openstack/diskimage-builder: Install docker for tests https://review.openstack.org/278021 | 04:45 |
openstackgerrit | greghaynes proposed openstack/diskimage-builder: Fail functests if refusing to run tests https://review.openstack.org/177002 | 04:47 |
*** shivrao has joined #tripleo | 04:48 | |
*** lblanchard has quit IRC | 04:57 | |
*** stendulker has joined #tripleo | 04:57 | |
*** rlandy has quit IRC | 05:07 | |
*** anande has joined #tripleo | 05:07 | |
*** anande has quit IRC | 05:08 | |
*** ooolpbot has joined #tripleo | 05:10 | |
ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 05:10 |
ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1543810 | 05:10 |
*** ooolpbot has quit IRC | 05:10 | |
openstack | Launchpad bug 1543810 in tripleo "Jenkins errors killing many CI jobs" [Critical,Triaged] | 05:10 |
*** shivrao_ has joined #tripleo | 05:10 | |
*** shivrao has quit IRC | 05:10 | |
*** shivrao_ is now known as shivrao | 05:10 | |
*** masco has joined #tripleo | 05:39 | |
openstackgerrit | Matthew Thode proposed openstack/diskimage-builder: Reorder the package-uninstall action https://review.openstack.org/278211 | 05:43 |
openstackgerrit | Matthew Thode proposed openstack/diskimage-builder: Remove eclean-dist as it's not available by default https://review.openstack.org/276581 | 05:44 |
*** michchap_ has quit IRC | 05:45 | |
*** michchap has joined #tripleo | 05:46 | |
*** Marga_ has joined #tripleo | 05:50 | |
*** Marga_ has quit IRC | 05:55 | |
*** ooolpbot has joined #tripleo | 06:10 | |
ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 06:10 |
ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1543810 | 06:10 |
openstack | Launchpad bug 1543810 in tripleo "Jenkins errors killing many CI jobs" [Critical,Triaged] | 06:10 |
*** ooolpbot has quit IRC | 06:10 | |
*** dsneddon_brno has joined #tripleo | 06:12 | |
*** Marga_ has joined #tripleo | 06:50 | |
*** Marga_ has quit IRC | 06:50 | |
*** Marga_ has joined #tripleo | 06:51 | |
*** aufi has joined #tripleo | 06:55 | |
*** shivrao has quit IRC | 07:03 | |
*** tzumainn has quit IRC | 07:05 | |
*** ooolpbot has joined #tripleo | 07:10 | |
ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 07:10 |
ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1543810 | 07:10 |
openstack | Launchpad bug 1543810 in tripleo "Jenkins errors killing many CI jobs" [Critical,Triaged] | 07:10 |
*** ooolpbot has quit IRC | 07:10 | |
*** jprovazn has joined #tripleo | 07:12 | |
*** rcernin has joined #tripleo | 07:13 | |
*** ukalifon1 has joined #tripleo | 07:16 | |
*** Marga_ has quit IRC | 07:33 | |
*** panda has quit IRC | 07:37 | |
*** panda has joined #tripleo | 07:37 | |
*** dsneddon_brno has quit IRC | 07:41 | |
*** mkovacik has quit IRC | 07:41 | |
*** openstackgerrit has quit IRC | 07:47 | |
*** openstackgerrit has joined #tripleo | 07:47 | |
*** jtomasek_ has joined #tripleo | 07:50 | |
*** dmacpher has quit IRC | 07:51 | |
*** rcernin has quit IRC | 07:52 | |
*** gfidente has joined #tripleo | 07:55 | |
*** gfidente has quit IRC | 07:55 | |
*** gfidente has joined #tripleo | 07:55 | |
*** fgimenez has joined #tripleo | 08:08 | |
*** fgimenez has quit IRC | 08:08 | |
*** fgimenez has joined #tripleo | 08:08 | |
*** ooolpbot has joined #tripleo | 08:10 | |
ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 08:10 |
ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1543810 | 08:10 |
openstack | Launchpad bug 1543810 in tripleo "Jenkins errors killing many CI jobs" [Critical,Triaged] | 08:10 |
*** ooolpbot has quit IRC | 08:10 | |
*** oshvartz has joined #tripleo | 08:13 | |
*** rcernin has joined #tripleo | 08:14 | |
*** d0ugal has joined #tripleo | 08:17 | |
*** liverpooler has joined #tripleo | 08:19 | |
*** devvesa has joined #tripleo | 08:20 | |
*** jistr|doc has joined #tripleo | 08:22 | |
*** hjensas has joined #tripleo | 08:30 | |
*** hjensas has joined #tripleo | 08:30 | |
*** athomas has joined #tripleo | 08:32 | |
openstackgerrit | Merged openstack/diskimage-builder: Only match #!/bin/bash in scripts https://review.openstack.org/273879 | 08:35 |
*** jaosorior has joined #tripleo | 08:48 | |
jaosorior | shardy: any clue why the HA gate is failing now? For some reason I no longer see the mongodb error, so it seems my patch is no longer relevant | 08:53 |
openstackgerrit | Moshe Levi proposed openstack/diskimage-builder: Adding InfiniBand Support https://review.openstack.org/274781 | 08:54 |
*** morazi has joined #tripleo | 08:59 | |
*** ifarkas has joined #tripleo | 09:02 | |
*** jcoufal has joined #tripleo | 09:03 | |
*** ooolpbot has joined #tripleo | 09:10 | |
ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 09:10 |
ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1543810 | 09:10 |
*** ooolpbot has quit IRC | 09:10 | |
openstack | Launchpad bug 1543810 in tripleo "Jenkins errors killing many CI jobs" [Critical,Triaged] | 09:10 |
*** shardy has joined #tripleo | 09:11 | |
jaosorior | shardy: Hey dude, how's it going? any clue why the HA gate is failing now? For some reason I no longer see the mongodb error, so it seems my patch is no longer relevant | 09:13 |
*** lucas-dinner is now known as lucasagomes | 09:19 | |
shardy | jaosorior: not sure, sorry - not yet had time to look into it | 09:21 |
*** mbound has joined #tripleo | 09:21 | |
shardy | marios: re https://review.openstack.org/#/c/260413/ - it's passed all CI on the current revision in different runs, and the latest ceph failure appears to be a job timeout and not anything linked to the patch | 09:22 |
shardy | I'm tempted to just approve it, wdyt? | 09:22 |
marios | shardy: heh, am watching the console at https://jenkins04.openstack.org/job/gate-tripleo-ci-f22-ha/373/console | 09:23 |
openstackgerrit | Juan Antonio Osorio Robles proposed openstack/os-cloud-config: Enable ssl for ceilometer https://review.openstack.org/267660 | 09:23 |
marios | shardy: (for the ha job anyway, if it fails on the pingtest again or some other issue, i may have an actual fit) | 09:24 |
marios | shardy: shall we see what happens on this last run? but yeah i'd be in favor of it, though would be nice to see a green run since we've come this far | 09:25 |
shardy | marios: maybe I need more coffee - it looked like the ha and nonha jobs both passed on the last run? | 09:25 |
marios | shardy: revisit later? | 09:25 |
shardy | marios: ack, sure | 09:25 |
marios | shardy: yeah it did, as you say it was the ceph job that failed | 09:25 |
marios | shardy: i rechecked to see that (and wanted a green run) | 09:25 |
shardy | Ok, cool, lets see what it does this time - would be another datapoint re the failures anyway I guess | 09:26 |
marios | thanks shardy | 09:27 |
*** dtantsur|afk is now known as dtantsur | 09:28 | |
*** mkovacik has joined #tripleo | 09:29 | |
*** nico_auv has joined #tripleo | 09:30 | |
*** Marga_ has joined #tripleo | 09:33 | |
gfidente | shardy, marios can't we just do that from hiera? | 09:34 |
gfidente | oh yes but no for the pcmk scenario :( | 09:35 |
marios | gfidente: o/ don't follow | 09:35 |
gfidente | https://review.openstack.org/#/c/260413/ | 09:35 |
marios | gfidente: oh yeah is also a cherrypick | 09:35 |
gfidente | but we can still drop the templating part though | 09:35 |
gfidente | oh right | 09:35 |
gfidente | I saw it too late :( | 09:35 |
shardy | gfidente: Hey, have you seen https://review.openstack.org/#/c/275437 ? | 09:36 |
shardy | it reworks the endpoint map to avoid DoSing heat, which IIRC was an issue you were encountering | 09:36 |
marios | shardy: i had a look at that this morning | 09:36 |
gfidente | shardy, yeah that's what we I was talking about the other day | 09:36 |
gfidente | you hadn't drink yet as I remember | 09:36 |
marios | shardy: gfidente: th eimplication is that operator would have to change the endpoint_data.yaml and rerun the script before deploy for any changes there right | 09:37 |
*** akrivoka has joined #tripleo | 09:37 | |
shardy | I'm wondering if that will help us with our CI performance issues | 09:37 |
*** Marga_ has quit IRC | 09:38 | |
shardy | given that we create the endpoint map for all tests, and it's clearly overloading heat and using a lot of memory | 09:38 |
marios | shardy: gfidente: it essentially replaces the need for the Endpoint resource itself with the input file and the python script | 09:38 |
marios | shardy: gfidente which is nice. i just hate that it is another script :/ and it goes back to template munging | 09:38 |
shardy | marios: well yes, but it's more a developer thing - until we have support for out-of-tree pluggable services | 09:38 |
shardy | atm it'd be adding a new entry to the template consumed by the generator script whenever you integrate a new service into t-h-t | 09:39 |
shardy | we can potentially automate running that via a check job, and enforce updating the input data | 09:39 |
shardy | but as a first step it'll be manual | 09:39 |
marios | shardy: yes this is a good point. you'd have to check in the edit with your change | 09:39 |
marios | (anyway/currently)( | 09:39 |
shardy | marios: yup, that's discussed in the review, it seems OK until we can wire in the automation (which probably won't be that hard) | 09:40 |
marios | shardy: no i meant, right now, you have to add your entry anyway (like I have to add to endpointmap right now, when say i want to add zookeeper or whatever service endpoints) | 09:40 |
shardy | marios: ah, yeah, but now you have to add them and run a script | 09:41 |
shardy | I agree it's not ideal, long term we need a better abstraction for these indirect mappings | 09:41 |
shardy | personally I've never been keen on all the data-only templates munging parameters into outputs with no resources | 09:42 |
*** electrofelix has joined #tripleo | 09:42 | |
shardy | but the fact we're needing to do it probably indicates we're lacking a mapping feature in heat | 09:42 |
marios | shardy: so getting over that ^^^ part.. my other possibly oconern was implications around updates... the resource itself has changed (Endpoint is gone). | 09:44 |
shardy | marios: I think that doesn't matter in this case, because there are no resources inside the Endpoint abstraction | 09:44 |
marios | shardy: i didn't vote cos i wasn't sure about these two things (gfidente actually mentioned this change to me last week, i hadn't realised what a heroic feat zaneb had pulled off so quickly) | 09:44 |
shardy | it's just data manipulation | 09:44 |
gfidente | marios, eroic | 09:45 |
shardy | e.g provided the before/after data output is the same, all should be OK | 09:45 |
shardy | AFAICS anyway | 09:45 |
marios | shardy: right yes that makes sense (the no resource inside endpoint - just params and outputs) | 09:46 |
jaosorior | only con for that is usability, I guess, since there will be one extra step for doing the overcloud deployment, IIUC | 09:46 |
*** jistr|doc has quit IRC | 09:46 | |
shardy | jaosorior: true, but in most cases (outside of super-advanced usage and third-party integrators) won't the stock endpoints be enough? | 09:47 |
marios | shardy: wih that concern addressed too, it should make things MUCH faster based on zane's commit msg | 09:47 |
jaosorior | shardy: I wouldn't call ssl super-advanced | 09:48 |
shardy | well, interestingly the HA job passed there in 1hr30, so I'm wondering if it's related to the frequent timeouts we're seeing | 09:48 |
jaosorior | wow | 09:48 |
jaosorior | how does the non-ha look like then, did the time decrease there too? | 09:48 |
shardy | all the jobs passed in 1hr30 approximately (range from 1hr24 to 1hr34) | 09:49 |
jaosorior | if the performance inpact is that big then I guess it makes sense to address usability later, after this is merged | 09:49 |
gfidente | yeah ha is not slower, false mith | 09:49 |
gfidente | maybe nonha could be made slightly faster by removing a step | 09:50 |
*** derekh has joined #tripleo | 09:50 | |
jaosorior | gfidente: Well, the endpoint map shouldn't realy vary that much between non-ha and ha | 09:50 |
gfidente | but that's the most of it | 09:50 |
jaosorior | *really | 09:50 |
gfidente | jaosorior, yeah endpoint is not a factor | 09:50 |
*** dtantsur is now known as dtantsur|brb | 09:51 | |
shardy | jaosorior: AFAICT it won't impact the SSL use-case at all: | 09:53 |
shardy | https://review.openstack.org/#/c/275437/1/network/endpoints/endpoint_map.yaml | 09:53 |
shardy | You can still pass in a different EndpointMap just like before | 09:54 |
shardy | it only becomes a little bit less convenient if you want to add a new endpoint | 09:54 |
shardy | See line 1715 - it's a bit confusing because the parameters are after the outputs | 09:54 |
jaosorior | shardy: Actually I got quite confused because of that | 10:02 |
jaosorior | ok, now it makes more sense | 10:03 |
*** mgould has joined #tripleo | 10:08 | |
*** ooolpbot has joined #tripleo | 10:10 | |
ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 10:10 |
ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1543810 | 10:10 |
*** ooolpbot has quit IRC | 10:10 | |
openstack | Launchpad bug 1543810 in tripleo "Jenkins errors killing many CI jobs" [Critical,Triaged] | 10:10 |
*** mcornea has joined #tripleo | 10:16 | |
*** r-mibu has quit IRC | 10:20 | |
*** r-mibu has joined #tripleo | 10:21 | |
*** jaosorior has quit IRC | 10:21 | |
*** shardy has quit IRC | 10:25 | |
*** shardy has joined #tripleo | 10:25 | |
*** dtantsur|brb is now known as dtantsur | 10:42 | |
marios | shardy: hmm looking like that ceph job is going to timeout again https://jenkins05.openstack.org/job/gate-tripleo-ci-f22-ceph/395/console for /#/c/260413/ | 10:42 |
openstackgerrit | Merged openstack-infra/tripleo-ci: Add support for periodic jobs https://review.openstack.org/271370 | 10:42 |
marios | ha job passed again though so that's good | 10:42 |
openstackgerrit | Merged openstack-infra/tripleo-ci: Add support for command line job names https://review.openstack.org/235421 | 10:42 |
shardy | marios: Hmm, it's a bit hard to see where it's stuck | 10:47 |
shardy | we should add a dump of all IN_PROGRESS resources to CI (or tripleoclient) sorted by the amount of time they've been running | 10:48 |
*** Marga_ has joined #tripleo | 10:52 | |
*** Marga_ has quit IRC | 10:56 | |
*** fgimenez has quit IRC | 11:02 | |
*** fgimenez has joined #tripleo | 11:03 | |
*** stendulker_ has joined #tripleo | 11:04 | |
*** stendulker_ has quit IRC | 11:07 | |
*** stendulker has quit IRC | 11:08 | |
cmyster | shardy: I suggested at one time to just dump heat schema, it should be pretty empty in CI machines | 11:29 |
*** Marga_ has joined #tripleo | 11:31 | |
*** panda has quit IRC | 11:37 | |
*** panda has joined #tripleo | 11:38 | |
*** Marga_ has quit IRC | 11:46 | |
*** Marga_ has joined #tripleo | 11:46 | |
derekh | marios shardy that stuck HA job ye were talking about, could possibley be fixed by this https://review.openstack.org/#/c/275149/ | 11:51 |
*** links has quit IRC | 11:56 | |
openstackgerrit | Merged openstack/python-tripleoclient: Simplify checking for stack complete https://review.openstack.org/275149 | 11:57 |
dtantsur | EmilienM, hi! could you please take a look at https://review.openstack.org/#/c/277780/ ? this is part of profile matching work that people constantly ping me about :) | 11:58 |
shardy | derekh: thanks, looks like that's ready to land to stable/liberty too now | 11:59 |
derekh | shardy: yup, +2'd | 12:00 |
shardy | hehe, snap ;) | 12:00 |
derekh | lol | 12:00 |
*** olaph has joined #tripleo | 12:04 | |
openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Split the deploy script into its own file https://review.openstack.org/275668 | 12:04 |
marios | derekh: thanks looking | 12:05 |
marios | derekh: prophet slagle at https://review.openstack.org/#/c/274593/2/tripleoclient/utils.py | 12:07 |
derekh | marios: ya, that patch was the first attempt to fix the problem, | 12:09 |
marios | derekh: you know i really amo not familiar with this utils stuff in the client/haven't looked for long time | 12:09 |
* marios fires up vim | 12:10 | |
derekh | marios: its merged now anyways ;-), nor was I until I tried to track down the cause of the problem | 12:10 |
marios | derekh: yeah am trying to understand what was going on with the wait_for_stack_ready to cause the loop in the first place | 12:11 |
marios | derekh: thanks, will try a recheck on those reviews in a sec then | 12:11 |
marios | derekh: good to see ci otherwise running OK this morning, at least for tht afaics (except the ceph jobs which might be this) | 12:12 |
derekh | marios: when I reproduced the probem, the heat stack was actually CREATE_COMPLETE but the tripleoclient was waiting for more events | 12:12 |
shardy | basically we were waiting on an event for definitive flag of stack status, which can be racy because we're also using an incrementing marker | 12:12 |
derekh | marios: the preoblem appears to be that the event is generated and checked by the client befor the status was updated | 12:12 |
shardy | heat already provides a definitive stack status, so it's much better to just use that | 12:13 |
*** masco has quit IRC | 12:13 | |
marios | derekh: i wonder what is special about the ceph job (if it does run green with this fix in place) to reliably repro/hit that | 12:13 |
derekh | marios: donno, if its still happening after this I'll try and reproduce it again, for the one I did reproduce I think we have the right fix | 12:14 |
marios | derekh: are we pulling from master for the tripleoclient? | 12:15 |
marios | (in ci) | 12:15 |
marios | derekh: really i am asking if i should recheck straight away or not | 12:16 |
derekh | marios: yes, but make sure to wait for it to appear in the delorean /current | 12:16 |
derekh | marios: http://trunk.rdoproject.org/centos7/report.html when you see it appear here, recheck | 12:16 |
marios | derekh: thanks ! | 12:16 |
derekh | finally http://tripleo.org/tripleo-periodic-jobs.html | 12:21 |
shardy | woot! | 12:24 |
shardy | there's even some green on it ;) | 12:24 |
*** thrash|g0ne is now known as thrash | 12:25 | |
derekh | shardy: crazy world we live in | 12:26 |
shardy | simple docs patch if someone has a moment: https://review.openstack.org/#/c/273459/ | 12:27 |
trown|outtypewww | derekh: nice one! | 12:29 |
*** trown|outtypewww is now known as trown | 12:29 | |
*** fgimenez has quit IRC | 12:31 | |
*** fgimenez has joined #tripleo | 12:32 | |
*** tremble has quit IRC | 12:33 | |
*** tremble has joined #tripleo | 12:33 | |
d0ugal | thrash: When you get a moment, can you take a look at https://review.openstack.org/#/c/275661/ | 12:34 |
openstackgerrit | Merged openstack/tripleo-docs: Fix NODE_MEM/UNDERCLOUD_NODE_MEM error https://review.openstack.org/273459 | 12:34 |
*** saneax has joined #tripleo | 12:34 | |
*** links has joined #tripleo | 12:35 | |
openstackgerrit | Merged openstack/tripleo-heat-templates: Makes the iSCSI initiator name unique for compute nodes https://review.openstack.org/275890 | 12:39 |
*** Goneri has quit IRC | 12:40 | |
shardy | tripleoclient patch fixing nasty bug mentioned in yesterdays meeting: | 12:45 |
shardy | https://review.openstack.org/#/c/275661/ | 12:46 |
shardy | passing CI, needs a +2A | 12:46 |
thrash | d0ugal: sure | 12:51 |
*** weshay_xchat has joined #tripleo | 12:52 | |
d0ugal | thrash: Thanks! | 12:53 |
*** lucasagomes is now known as lucas-hungry | 12:56 | |
*** rlandy has joined #tripleo | 13:02 | |
openstackgerrit | Merged openstack/python-tripleoclient: Don't regenerate the overcloud passwords if the Heat stack exists https://review.openstack.org/275661 | 13:02 |
*** saneax is now known as saneax_AFK | 13:03 | |
*** rbrady has joined #tripleo | 13:04 | |
*** jcoufal has quit IRC | 13:07 | |
*** dprince has joined #tripleo | 13:09 | |
*** egafford has joined #tripleo | 13:11 | |
*** olaph is now known as loaf | 13:13 | |
*** jayg|g0n3 is now known as jayg | 13:17 | |
trown | derekh: I think we will have to unpin puppet-nova for https://review.openstack.org/278137 to pass, but there is also a similar patch needed for the overcloud | 13:19 |
trown | derekh: I am curious though if those patches are strictly needed to make current pass if we keep the puppet-nova pin | 13:20 |
derekh | trown I gues if we keep the pin we could move current without them but preferably we would remove the pin ASAP | 13:22 |
derekh | trown: iirc we can remove the pin once this merges https://review.openstack.org/#/c/275607/ | 13:22 |
derekh | trown: actaully maybe I'm getting mixed up now | 13:22 |
trown | derekh: ya I am also having a bit of trouble keeping this all straight | 13:23 |
derekh | we can't move onto the new nova package until that merges | 13:23 |
trown | since I am testing current without the nova pin | 13:23 |
trown | I guess we can see if we get https://review.openstack.org/#/c/275607/ and https://review.openstack.org/#/c/276790/ merged if the nightly periodic job passes | 13:25 |
trown | derekh: that periodic job still uses the puppet pins from the gates? | 13:26 |
derekh | trown: yum it does | 13:26 |
derekh | *yup | 13:26 |
trown | cool, the THT patch above needs some work, but dmsimard should be around soon | 13:27 |
openstackgerrit | Derek Higgins proposed openstack/instack-undercloud: Send a refresh event to all db sync commands https://review.openstack.org/244136 | 13:27 |
derekh | trown: ack | 13:28 |
*** dcain1 has joined #tripleo | 13:28 | |
*** dcain1 has left #tripleo | 13:28 | |
*** bvandenh_ has joined #tripleo | 13:29 | |
*** Marga_ has quit IRC | 13:29 | |
openstackgerrit | James Slagle proposed openstack/os-cloud-config: Add []'s around ipv6 address when creating keystone clients https://review.openstack.org/278111 | 13:31 |
openstackgerrit | James Slagle proposed openstack/os-cloud-config: Add []'s around ipv6 address when creating keystone clients https://review.openstack.org/278111 | 13:32 |
marios | OS::TripleO::CephStorage::Ports::StoragePort: /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml | 13:34 |
marios | OS::TripleO::CephStorage::Ports::StorageMgmtPort: /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml | 13:34 |
openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Source undercloud environment variable from a file https://review.openstack.org/275667 | 13:37 |
openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Split the deploy script into its own file https://review.openstack.org/275668 | 13:37 |
openstackgerrit | Merged openstack/python-tripleoclient: Wrap around list of events https://review.openstack.org/275261 | 13:40 |
openstackgerrit | Merged openstack/python-tripleoclient: Simplify checking for stack complete https://review.openstack.org/275263 | 13:41 |
*** bvandenh_ has quit IRC | 13:43 | |
*** mcornea has quit IRC | 13:50 | |
*** mcornea has joined #tripleo | 13:51 | |
*** lucas-hungry is now known as lucasagomes | 13:53 | |
*** bvandenh_ has joined #tripleo | 13:54 | |
*** dcain has joined #tripleo | 13:58 | |
*** rhallisey has joined #tripleo | 14:00 | |
derekh | dprince: I updated your generate site script to add the report fopr the periodic job http://tripleo.org/tripleo-periodic-jobs.html | 14:00 |
derekh | dprince: will figure out how to add it to the main site later | 14:01 |
*** Goneri has joined #tripleo | 14:01 | |
dprince | derekh: want me to add it? | 14:02 |
dprince | derekh: I still need to actually push that code somewhere? | 14:02 |
derekh | fire ahead if its not a PITA | 14:02 |
dprince | derekh: Yeah, I think it should be fairly easy | 14:02 |
derekh | dprince: ya, wouldn't do any hard to have it somewhere | 14:02 |
derekh | dprince: bbiab | 14:02 |
gfidente | derekh, regarding the upgrade job | 14:03 |
gfidente | I replied on gerrit but maybe both your comments are worth discussing here for more audience | 14:04 |
gfidente | for the upgrade job I think we can run the pcmk scenario with a single controller | 14:04 |
gfidente | but at the same time I'd like to have ceph there to make sure we don't break things | 14:04 |
gfidente | on the other hand, we're not actually testing computes or ceph nodes upgrade | 14:06 |
gfidente | so they aren't of great use | 14:06 |
*** weshay_xchat is now known as weshay | 14:07 | |
*** fgimenez has quit IRC | 14:08 | |
*** fgimenez has joined #tripleo | 14:10 | |
tremble | gfidente: however testing the upgrade would provide very basic validation of the type that keeps customers from thinking "does anyone even try upgrading this" and is better than nothing... | 14:10 |
gfidente | tremble, oh indeed I was for keeping the nodes | 14:11 |
gfidente | https://review.openstack.org/#/c/260466 | 14:11 |
gfidente | but I am not sure how much that impacts CI, which is also a factor | 14:11 |
*** Matt4 has quit IRC | 14:13 | |
tremble | Yeah unit testing the install/upgrade pieces is if anything harder than unit testing individual components. | 14:13 |
tremble | personally I'd rather see a more complex case tested as well, but that may be better suited to a periodic task than a gate task | 14:15 |
openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Generate a unique DeployIdentifier on updates https://review.openstack.org/268126 | 14:15 |
*** julim has joined #tripleo | 14:15 | |
*** tzumainn has joined #tripleo | 14:16 | |
d0ugal | shardy: I seem to remember that you had a bash script to find the error in Heat, I can't seem to find it now. Am I remembering correctly? | 14:17 |
openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Install mongodb client package https://review.openstack.org/278394 | 14:22 |
shardy | d0ugal: I had a script in a blog post, but it's easy enough, just do heat resource-list -n5 overcloud | grep FAILED | grep "Deployment " | 14:24 |
shardy | note the space after Deployment | 14:24 |
*** tiswanso has joined #tripleo | 14:24 | |
d0ugal | shardy: Yeah, that is the script I was looking for :) | 14:24 |
shardy | then do heat deployment-show <id of deployment resource> | 14:24 |
shardy | d0ugal: it only really helps with software configuration errors, but we should probably wire that in so it always runs on a failed deployment | 14:24 |
*** apetrich has quit IRC | 14:25 | |
*** apetrich has joined #tripleo | 14:25 | |
d0ugal | shardy: right, thanks. | 14:27 |
*** links has quit IRC | 14:27 | |
*** masco has joined #tripleo | 14:28 | |
*** lblanchard has joined #tripleo | 14:28 | |
dmsimard | I'm around | 14:29 |
dmsimard | tremble: I'll submit a fix | 14:30 |
dmsimard | er, sorry tremble | 14:30 |
dmsimard | trown: ^ | 14:30 |
trown | dmsimard: sweet thanks | 14:32 |
*** jcoufal has joined #tripleo | 14:33 | |
*** dmacpher has joined #tripleo | 14:33 | |
*** rpothier has joined #tripleo | 14:35 | |
openstackgerrit | David Moreau Simard proposed openstack/tripleo-heat-templates: Nova now requires an api database to be created https://review.openstack.org/276790 | 14:37 |
dmsimard | trown: ^ good find | 14:37 |
* dmsimard starts hacking on that nova-neutron patch | 14:37 | |
*** lblanchard has quit IRC | 14:39 | |
*** w_ has joined #tripleo | 14:40 | |
*** loaf has quit IRC | 14:43 | |
dmsimard | trown: meanwhile crisis averted with heat changes that broke mitaka | 14:43 |
dmsimard | EmilienM and I yelled loud enough :) | 14:43 |
EmilienM | :) | 14:44 |
trown | nice one | 14:44 |
dmsimard | still need a tempest revert merged but the necessary fix is merged in heat (should be picked up by delorean shortly) | 14:44 |
trown | that is the most stressful thing about chasing trunk for all of openstack, finding all oustanding issues and getting patches up, then having something else break before they get merged | 14:45 |
shardy | dmsimard: what's the heat patch/bug out of interest? | 14:45 |
dmsimard | shardy: so there was this revert that we needed in tempest: https://review.openstack.org/#/c/278068/ and https://review.openstack.org/#/c/278081/ in heat (see related bug in reviews for details) | 14:47 |
dmsimard | shardy: fyi, back to a comment bnemec pointed out yesterday | 14:47 |
dmsimard | nova-neutron configuration does need a versioned endpoint (as in /v3), unversioned does not work :) | 14:47 |
shardy | dmsimard: thanks for the links | 14:48 |
shardy | that sounds like something wrong with their auth plugin integration | 14:48 |
dmsimard | I'm not the guy that's going to dig into that | 14:48 |
dmsimard | But maybe, yeah :) | 14:48 |
shardy | AFAIK if you use the keystoneclient plugin stuff and sessions correctly versions are detected automagically | 14:49 |
shardy | but yeah, someone can hopefully look into that in due course ;) | 14:49 |
shardy | I removed all the v2.0 stuff from the default heat config recently, and it mostly "just works" once you get the right mixture of values into the keystoneclient code | 14:50 |
dmsimard | automagically -> some wizard isn't doing his job :P | 14:50 |
*** jpeeler has joined #tripleo | 14:50 | |
EmilienM | shardy: it basically broke backward compatibility | 14:51 |
dmsimard | yeah, I mean, not only is /v3 mandatory now but the previous config doesn't work at all | 14:53 |
dmsimard | but hey, it worked in devstack | 14:53 |
EmilienM | dmsimard: still not friday | 14:53 |
trown | EmilienM: I like to say "its almost friday" | 14:54 |
trown | more glass half full | 14:54 |
trown | (nevermind that it is not) | 14:55 |
*** w_ has quit IRC | 14:55 | |
shardy | well that's one disadvantage to devstack moving everything to v3 I guess | 14:55 |
*** pradk has joined #tripleo | 14:55 | |
shardy | FWIW I reported the lack of member role thing against keystone a long time ago: | 14:56 |
shardy | https://bugs.launchpad.net/keystone/+bug/1366133 | 14:56 |
openstack | Launchpad bug 1366133 in OpenStack Identity (keystone) "User create via v3 API doesn't add _member_ role in default project" [Undecided,Won't fix] | 14:56 |
shardy | that was wonfixed thus one aspect of the breakage | 14:56 |
shardy | at least we got a fix for the heat part of the problem posted pretty fast | 14:56 |
*** xarses has quit IRC | 14:56 | |
*** w_ has joined #tripleo | 14:57 | |
dmsimard | EmilienM: is that restricted to fridays ? | 14:57 |
EmilienM | dmsimard: yes | 14:57 |
dmsimard | :( | 14:57 |
dmsimard | At least I'm not posting it on TWITTER on non-fridays | 14:57 |
* dmsimard pouts | 14:58 | |
EmilienM | twitter is a nice place to report bugs :-) | 14:59 |
trown | lol | 14:59 |
*** masco has quit IRC | 15:01 | |
openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Show os-net-config applied via script, not oac https://review.openstack.org/271450 | 15:04 |
derekh | gfidente: I'm not sure about running the upgrade on a single node, there is coordination between the controllers that we will want to exercise | 15:06 |
gfidente | yeah I read that | 15:07 |
gfidente | but it's heat taking care of that | 15:07 |
gfidente | I mean, it's not like we can desync the nodes in the same resource group? | 15:08 |
gfidente | maybe shardy knows better | 15:08 |
*** deva_ has joined #tripleo | 15:08 | |
*** NobodyCa1 has joined #tripleo | 15:09 | |
*** NobodyCa1 has quit IRC | 15:14 | |
*** deva_ has quit IRC | 15:14 | |
*** nico_auv has quit IRC | 15:18 | |
*** nico_auv has joined #tripleo | 15:19 | |
*** rbrady has quit IRC | 15:19 | |
*** rbrady has joined #tripleo | 15:20 | |
*** dustins has joined #tripleo | 15:21 | |
*** lblanchard has joined #tripleo | 15:22 | |
*** xarses has joined #tripleo | 15:22 | |
*** fgimenez has quit IRC | 15:22 | |
*** jaosorior has joined #tripleo | 15:24 | |
*** fgimenez has joined #tripleo | 15:25 | |
*** w__ has joined #tripleo | 15:26 | |
jaosorior | bnemec: ping | 15:27 |
*** w_ has quit IRC | 15:28 | |
*** rcernin has quit IRC | 15:33 | |
bnemec | jaosorior: pong | 15:33 |
*** saneax_AFK is now known as saneax | 15:34 | |
jaosorior | bnemec: Hey dude, I hadn't noticed you had already backported the enabling of the ssl port for ceilometer in os-cloud-config | 15:35 |
jaosorior | just a heads up that I accidentally uploaded it again | 15:35 |
*** oshvartz has quit IRC | 15:36 | |
jaosorior | benemec: https://review.openstack.org/#/c/267660/ And it seems to pass HA too :D | 15:36 |
*** jaosorior has quit IRC | 15:36 | |
*** panda has quit IRC | 15:37 | |
*** panda has joined #tripleo | 15:38 | |
openstackgerrit | James Slagle proposed openstack/tripleo-puppet-elements: Add puppet-kmod into image https://review.openstack.org/278456 | 15:40 |
*** jaosorior has joined #tripleo | 15:42 | |
*** liverpooler has quit IRC | 15:46 | |
*** rcernin has joined #tripleo | 16:00 | |
*** dmacpher is now known as dmacpher-afk | 16:02 | |
*** NobodyCa1 has joined #tripleo | 16:05 | |
*** deva_ has joined #tripleo | 16:06 | |
*** jaosorior_ has joined #tripleo | 16:09 | |
*** deva_ has quit IRC | 16:11 | |
*** NobodyCa1 has quit IRC | 16:11 | |
openstackgerrit | Dustin Schoenbrun proposed openstack/tripleo-heat-templates: Enable Manila integration https://review.openstack.org/188137 | 16:13 |
jaosorior_ | can someone check this commit out? https://review.openstack.org/#/c/267660/ it's a one liner :D | 16:16 |
*** NobodyCa1 has joined #tripleo | 16:19 | |
openstackgerrit | David Moreau Simard proposed openstack/tripleo-heat-templates: Nova Neutron configuration now uses keystone v3 endpoint https://review.openstack.org/278490 | 16:19 |
dmsimard | trown: ^ | 16:19 |
dmsimard | bnemec, shardy: ^ :) | 16:20 |
trown | dmsimard: oh nice, I just kicked a deploy with everything in the etherpad, and removing the puppet-nova from source since it looks like that change made it to OPM | 16:20 |
dmsimard | This could or could not make sense, feel free to comment | 16:21 |
dmsimard | I *think* it should be good | 16:21 |
trown | dmsimard: I will test your change above next... maybe that even gets us a functioning overcloud :) | 16:21 |
*** weshay has quit IRC | 16:21 | |
dmsimard | oh wait, there's something I missed I think. | 16:21 |
*** weshay_xchat has joined #tripleo | 16:21 | |
trown | oh cool even generated with the new generator | 16:22 |
jaosorior_ | dmismard: So there will be both v2.0 and v3 endpoints? | 16:22 |
trown | hopefully we can get that endpoint map generator patch merged | 16:23 |
jaosorior_ | wasn't the recommendation to have versionless endpoints for keystone? | 16:23 |
*** weshay_xchat has quit IRC | 16:23 | |
*** weshay has joined #tripleo | 16:23 | |
jaosorior_ | ayoung ^^ | 16:23 |
dmsimard | jaosorior_: it doesn't work for nova-neutron, see https://bugs.launchpad.net/puppet-nova/+bug/1542486 | 16:23 |
openstack | Launchpad bug 1542486 in puppet-nova "nova-compute stack traces with BadRequest: Specifying 'tenant_id' other than authenticated tenant in request requires admin privileges" [Critical,New] - Assigned to Emilien Macchi (emilienm) | 16:23 |
ayoung | jaosorior_, keystone v3!?!1One! | 16:23 |
*** NobodyCa1 has quit IRC | 16:23 | |
*** rcernin has quit IRC | 16:24 | |
jaosorior_ | ayoung: Yes, hopefully we get there soon. I just had the question regarding if we should be using versionless endpoints to use v3 | 16:24 |
dmsimard | jaosorior_: we should, but for this particular case it does not work | 16:24 |
jaosorior_ | dmsimard: But yeah, thanks for pointing me to that bug. bummer it doesn't work :/ | 16:24 |
ayoung | jaosorior_, dead on, thanks for notifying me, -1 and comment already posted | 16:25 |
openstackgerrit | Derek Higgins proposed openstack/tripleo-heat-templates: Makes the iSCSI initiator name unique for compute nodes https://review.openstack.org/278496 | 16:26 |
ayoung | dmsimard, let me look at that, and maybe I can help figure out why | 16:26 |
dmsimard | ayoung: nova-neutron *does not work* with unversioned endpoint, please see the bug | 16:26 |
dmsimard | ayoung: even devstack has a versioned endpoint | 16:26 |
ayoung | but let's not add an additional layer of paint to the floor outside our corner | 16:26 |
ayoung | if devstack was jumping off a bridge, I would not follow it | 16:26 |
ayoung | even with the XKCD rule about that in effect | 16:26 |
dmsimard | ayoung: feel free to work this out but in the meantime trunk is broken and I don't want it to stay broken for weeks | 16:26 |
ayoung | https://xkcd.com/1170/ | 16:26 |
ayoung | dmsimard, yuck | 16:27 |
jaosorior_ | ayoung: Seems that the patch that fixes the bug report is explicitly using v3password as auth_plugin https://review.openstack.org/#/c/276932/5/manifests/network/neutron.pp So that would explain why v3 is needed | 16:27 |
ayoung | dmsimard, so, this only has to be true for Keystone today | 16:27 |
jaosorior_ | or why it needs to be set in the endpoint | 16:27 |
ayoung | so don;'t use that plugin | 16:27 |
ayoung | there is a versionless plugin | 16:27 |
jaosorior_ | yep | 16:27 |
jaosorior_ | simply called 'password' | 16:27 |
ayoung | this is the problem with jamielennox being in Australia | 16:28 |
ayoung | he's asleep now, but this is his bailywick | 16:28 |
*** saneax is now known as saneax_AFK | 16:28 | |
trown | ayoung: we can not block tripleo moving forward because of breakages in other projects | 16:29 |
ayoung | trown give me a moment to grok what is going on, but explicit v3 is a bad pattern | 16:29 |
trown | k | 16:30 |
dmsimard | ayoung: I don't think anyone disagrees with that statement | 16:30 |
jaosorior_ | ayoung: Seems to me the solution is to patch puppet-nova (again) to get it to use the versionless plugin as a default. And then we need to change tripleo-heat-templates again | 16:30 |
dmsimard | We're just doing what needs to be done for things to work | 16:30 |
dmsimard | jaosorior_: the versionless plugin does not work | 16:30 |
dmsimard | jaosorior_: that is why we changed it | 16:30 |
dmsimard | in the first place | 16:30 |
ayoung | Ah...so puppet is the real culprit here.... | 16:30 |
ayoung | dmsimard, it does work. | 16:30 |
ayoung | If it does not work there, the qweschunis why | 16:31 |
dmsimard | ayoung: it worked before | 16:31 |
*** tiswanso has quit IRC | 16:31 | |
dmsimard | We didn't change it for fun | 16:31 |
dmsimard | Something happened in nova or somewhere else that prompted us to align the configuration to be similar to devstack | 16:31 |
ayoung | dmsimard, did you ask any of the keystone folks why it would be broken? | 16:31 |
dmsimard | We asked #openstack-nova | 16:31 |
dmsimard | Their answer was "Well, it works in devstack" | 16:32 |
ayoung | Cargo cult is why we are stuck with brain dead keystone decisions from 4 + years ago | 16:32 |
trown | well tripleo is not the place to fix that | 16:32 |
trown | nova is the place to fix that | 16:32 |
gfidente | derekh, https://review.openstack.org/#/c/277517/ | 16:32 |
ayoung | trown, I have been working on Tripleo using Keystone HTTPD. | 16:32 |
ayoung | That is pure cargo cult at this point | 16:32 |
ayoung | we had it fixed in Puppet even | 16:32 |
ayoung | So, getting the the root of problems is esssential to me not going more insane than I already \am | 16:33 |
trown | until nova has a gate job for tripleo, we have to take devstack as their recommended configuration | 16:33 |
ayoung | AAAAH too late!@ | 16:33 |
ayoung | Heh | 16:33 |
trown | if that recommended configuration is wrong... that is on nova | 16:33 |
derekh | gfidente: is that related to the initiator thing ? | 16:34 |
*** tiswanso has joined #tripleo | 16:34 | |
dmsimard | ayoung: http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2016-02-05.log.html#t2016-02-05T21:06:32 | 16:35 |
ayoung | dmsimard, you can't trust devstack | 16:35 |
ayoung | they have other demands | 16:35 |
*** mgould has quit IRC | 16:35 | |
ayoung | like not breaking gate | 16:35 |
*** tiswanso has quit IRC | 16:35 | |
dmsimard | ayoung: no kidding | 16:35 |
ayoung | that keeps it from effectively being the reference | 16:35 |
derekh | marios: I got lost in a bunch of browser tabs, which jenkins job was I going to look at ? | 16:35 |
ayoung | we are working with sdague on that | 16:35 |
*** tiswanso has joined #tripleo | 16:35 | |
trown | the crux of the issue is that we (tripleo) have to treat delorean/current being broken as if CI itself is broken or we have no chance of keeping up | 16:35 |
trown | and it has been broken for 2 weeks | 16:36 |
ayoung | dmsimard, are you telling me I need to beat up dims? | 16:36 |
ayoung | Damhnit, I like dims. | 16:36 |
ayoung | trown, fair enough. So long as this is temporary | 16:36 |
ayoung | I will remove the -1 if you open a bug on it | 16:37 |
dmsimard | ayoung: do what you gotta do, I asked for more details on what could have cause the previous configuration to suddenly stop working but was mostly ignored | 16:37 |
trown | and in order to deploy with current nova, we need explicit v3 | 16:37 |
trown | there is a bug already | 16:37 |
ayoung | Is there a bug for that? | 16:37 |
ayoung | cuz explicit v3 is a bug | 16:37 |
dmsimard | There is/was one for puppet-nova, I haven't filed anything specifically against nova | 16:37 |
dmsimard | (personally) | 16:37 |
ayoung | dmsimard, how about add an "also effects" on the puppet bug then...link? | 16:38 |
ayoung | https://bugs.launchpad.net/puppet-nova/+bug/1542486 | 16:38 |
openstack | Launchpad bug 1542486 in puppet-nova "nova-compute stack traces with BadRequest: Specifying 'tenant_id' other than authenticated tenant in request requires admin privileges" [Critical,New] - Assigned to Emilien Macchi (emilienm) | 16:38 |
dmsimard | ayoung: https://bugs.launchpad.net/puppet-nova/+bug/1542486 | 16:38 |
gfidente | derekh, no no, just pulling votes :P | 16:38 |
dtantsur | trown, hey! do tripleoci runs save their images somewhere? | 16:38 |
marios | derekh: https://jenkins07.openstack.org/job/gate-tripleo-ci-f22-ceph/342/console but looks like it is at pingtest now so may be ok | 16:38 |
marios | derekh: for https://review.openstack.org/#/c/260413/9 | 16:39 |
derekh | marios: ack | 16:39 |
trown | dtantsur: nope :( | 16:39 |
dtantsur | sign.. | 16:39 |
trown | dtantsur: one of the things that could be solved by https://review.openstack.org/#/c/276810 | 16:39 |
ayoung | dmsimard, OK | 16:40 |
ayoung | I've updated the bug | 16:40 |
ayoung | I'll remiove the -1 and comment | 16:40 |
trown | thanks | 16:40 |
dmsimard | ayoung: thanks. | 16:40 |
openstackgerrit | ayoung proposed openstack/tripleo-heat-templates: Nova Neutron configuration now uses keystone v3 endpoint https://review.openstack.org/278490 | 16:42 |
dmsimard | trown: let me know if there's anything broken in my patches that you notice -- note that the t-h-t update is based on the endpoint rework | 16:42 |
ayoung | dmsimard, added comments in yaml | 16:42 |
*** jtomasek_ has quit IRC | 16:42 | |
dmsimard | ayoung: sure | 16:42 |
dmsimard | ayoung: hum actually don't put a comment there | 16:43 |
dmsimard | that's a generated file (the patch is based on the endpoint map rework) | 16:43 |
ayoung | dmsimard, feel free to move it, but I don't know where to. | 16:43 |
dmsimard | but the generated file is versioned | 16:43 |
dmsimard | ok | 16:43 |
marios | derekh: well, it is going to fail... pingtest 2016-02-10 16:42:07.934 | ERROR: Failed to stop stack (tenant-stack) on other engine (a3cae6bb-e9cc-46c9-a2fd-b4b588b0bd6c) | 16:43 |
marios | derekh: https://jenkins07.openstack.org/job/gate-tripleo-ci-f22-ceph/342/console | 16:43 |
trown | dmsimard: might be a good idea to comment on your patch that it is based on the endpoint generator patch with link... I would not have known about that had I not seen it come up here last night | 16:44 |
trown | in which case you patch looks pretty weird | 16:44 |
*** ukalifon1 has quit IRC | 16:44 | |
marios | derekh: not to worry, i will look again tomorrow morning it is getting on in the day for you too (thi may be another memory issue affecting overcloud heat in this case. doesn't seem related to the tripleoclient timeouts anyway) | 16:45 |
marios | thanks | 16:45 |
devvesa | marios: let me know if you need something from me | 16:45 |
derekh | marios: ya, this is different, I'll try and take a look also | 16:46 |
dmsimard | trown: well, it's rebased on top of the generator patch - I would hope gerrit had a better way of pointing that out | 16:46 |
devvesa | I was looking at the same patch the ceph job: https://jenkins04.openstack.org/job/gate-tripleo-ci-f22-ha/381/console | 16:46 |
dmsimard | trown: but you can kind of see it in the dependencies | 16:46 |
dmsimard | or related changes rather | 16:46 |
devvesa | and it fails in the pingtest too, but for another reason: http://fpaste.org/320828/51223181/ | 16:46 |
dmsimard | otherwise git log will show that it's on top of that tree | 16:46 |
marios | devvesa: o/ jaume | 16:46 |
openstackgerrit | David Moreau Simard proposed openstack/tripleo-heat-templates: Nova Neutron configuration now uses keystone v3 endpoint https://review.openstack.org/278490 | 16:46 |
dmsimard | ayoung: are you cool with this https://review.openstack.org/#/c/278490/3/network/endpoints/endpoint_data.yaml | 16:47 |
ayoung | dmsimard, ++ | 16:47 |
dmsimard | cool. | 16:47 |
trown | dmsimard: ya, it is unfortunately non-obvious to me from gerrit webui, but maybe it is a non issue | 16:48 |
dmsimard | trown: it was better at showing commit was the parent before the gerrit upgrade | 16:49 |
trown | yep | 16:49 |
*** mgould has joined #tripleo | 16:49 | |
*** marcusvrn_ has joined #tripleo | 16:53 | |
*** jaosorior_ has quit IRC | 16:58 | |
*** jaosorior_ has joined #tripleo | 17:00 | |
*** hjensas has quit IRC | 17:09 | |
*** morazi has quit IRC | 17:09 | |
*** jcoufal has quit IRC | 17:13 | |
*** devvesa has quit IRC | 17:20 | |
*** jcoufal has joined #tripleo | 17:20 | |
*** aufi has quit IRC | 17:20 | |
*** jcoufal has quit IRC | 17:23 | |
*** eggmaster has quit IRC | 17:24 | |
*** mbound has quit IRC | 17:27 | |
*** ifarkas has quit IRC | 17:27 | |
*** dtantsur is now known as dtantsur|afk | 17:28 | |
*** jaosorior_ has quit IRC | 17:28 | |
*** trown is now known as trown|meeting | 17:29 | |
derekh | Looks like the slaves loosing communication with jenkins were because of the hosts going OOM and killing them, its was particularly bad yesterday because we had a lot of jobs start at the same time, If added 15G of RAM to each compute node to see if the situation improves | 17:32 |
derekh | dprince: ^ | 17:32 |
dprince | derekh: possibly related to us overcommitting? | 17:33 |
derekh | dprince: yup | 17:34 |
*** penick has quit IRC | 17:36 | |
*** tiswanso has quit IRC | 17:36 | |
*** trown|meeting is now known as trown | 17:48 | |
*** jtomasek has joined #tripleo | 17:48 | |
*** dustins has quit IRC | 17:49 | |
trown | woot woot tripleo.sh -- Overcloud pingtest, SUCCESS | 17:49 |
trown | dmsimard: ^ with your latest tht patch | 17:50 |
trown | nice work | 17:50 |
dmsimard | trown: wahoo | 17:50 |
dmsimard | trown: there are ci failures on the review though | 17:50 |
dmsimard | trown: well, on this one https://review.openstack.org/#/c/278134/ | 17:50 |
trown | derekh: dprince, that means we have all the patches we need to get current working... minus I guess one to unpin puppet-nova | 17:50 |
trown | maybe we can create that patch with a dependency on the 4 others | 17:51 |
trown | dmsimard: ya, we have puppet-nova pinned so I would not expect these later ones to patch until we unpin it | 17:51 |
*** fgimenez has quit IRC | 17:51 | |
dmsimard | ok | 17:52 |
*** rcernin has joined #tripleo | 17:52 | |
trown | dmsimard: I will create a tripleo-ci patch to tie all the pieces together so we can test current and puppet-nova master with them all | 17:53 |
bnemec | So, it looks to me like the puppet-nova change is the problem here. | 17:53 |
derekh | trown: ack, worth trying them all together | 17:53 |
bnemec | I can run the pingest successfully against a current trunk overcloud as long as I leave the puppet-nova change out. | 17:53 |
trown | hmm, I had not tried that bit | 17:53 |
trown | what does current mean then though | 17:54 |
bnemec | I'm pointed at http://trunk.rdoproject.org/centos7-liberty/eb/5a/eb5a1b841df1130a8f4691406a0f60b7d107bb7d_457a6559/ | 17:54 |
bnemec | Shit | 17:54 |
trown | bnemec: liberty :) | 17:54 |
* bnemec notices that's liberty | 17:54 | |
trown | which is working great :) | 17:54 |
bnemec | WTF does our documentation use liberty as current trunk? | 17:55 |
* bnemec can't read | 17:56 | |
* bnemec needs a vacation | 17:56 | |
trown | good call | 17:56 |
* trown calls it a year, see you all next year | 17:57 | |
*** morazi has joined #tripleo | 17:58 | |
*** tiswanso has joined #tripleo | 18:00 | |
trown | hmm... we really need to move tripleo.sh to tripleo-ci repo... just realized I need 2 patches to tie all of this together | 18:01 |
*** derekh has quit IRC | 18:01 | |
*** dustins has joined #tripleo | 18:01 | |
*** w__ has quit IRC | 18:02 | |
*** akrivoka has quit IRC | 18:02 | |
trown | err... actually nope, both changes go in tripleo.sh nevermind | 18:03 |
*** tiswanso has quit IRC | 18:05 | |
*** tiswanso has joined #tripleo | 18:05 | |
*** jaosorior has quit IRC | 18:09 | |
shardy | trown: I proposed https://review.openstack.org/#/c/272210/ to move it, but we really need to pause landing things to tripleo-common or I'll just keep rebase racing those patches | 18:11 |
openstackgerrit | John Trowbridge proposed openstack/tripleo-common: DO NOT MERGE testing current with master puppet-nova https://review.openstack.org/278553 | 18:12 |
shardy | ramishra: ^^ | 18:12 |
shardy | trown: will that also pick up latest trunk heat? | 18:12 |
shardy | ramishra has been looking to test a recent change which has been reported to cause issues w/magnum | 18:12 |
shardy | we're interested to prove if TripleO is OK, and I've not yet pulled and locally tested | 18:13 |
shardy | https://github.com/openstack/heat/commit/0388da504765f9975aeadc509672c0a47bbcc244 | 18:13 |
trown | shardy: ya it just moves the delorean pin from current-tripleo to current | 18:13 |
shardy | that's the commit under discussion FYI | 18:13 |
trown | so whatever is on current for heat will get pulled in | 18:13 |
shardy | Ok, thanks | 18:13 |
*** w__ has joined #tripleo | 18:14 | |
trown | shardy: I just tested with http://trunk.rdoproject.org/centos7/57/a1/57a16b4571af82276a813d7f07f9dd47bc48f677_9d3dd879 with all of the depends on patches in that review above and it passed... and there is not a newer heat build in delorean | 18:15 |
shardy | trown: ack, thanks, may be an issue specific to magnum then | 18:15 |
shardy | ramishra: ^^ | 18:16 |
ramishra | shardy: thanks | 18:16 |
trown | I did not test HA, but I dont see that heat commit being an HA specific break | 18:17 |
shardy | Yeah, although we may need to test w/network-isolation to be sure all is OK | 18:17 |
trown | yep, I did not test that path either | 18:18 |
*** Marga_ has joined #tripleo | 18:21 | |
*** rhallisey has quit IRC | 18:21 | |
*** electrofelix has quit IRC | 18:22 | |
openstackgerrit | Rabi Mishra proposed openstack/tripleo-heat-templates: [DNM] Test with heat patch https://review.openstack.org/278556 | 18:23 |
*** lucasagomes is now known as lucas-afk | 18:25 | |
*** mgould has quit IRC | 18:36 | |
*** nico_auv has quit IRC | 18:40 | |
*** morazi has quit IRC | 18:45 | |
*** liverpooler has joined #tripleo | 18:47 | |
*** athomas has quit IRC | 18:47 | |
*** mkovacik has quit IRC | 18:47 | |
*** shivrao has joined #tripleo | 18:47 | |
*** penick has joined #tripleo | 18:57 | |
*** w_ has joined #tripleo | 19:01 | |
*** w__ has quit IRC | 19:03 | |
*** mcornea has quit IRC | 19:05 | |
*** trown is now known as trown|lunch | 19:05 | |
openstackgerrit | Dustin Schoenbrun proposed openstack/tripleo-heat-templates: Enable Manila integration https://review.openstack.org/188137 | 19:08 |
*** w_ has quit IRC | 19:14 | |
*** w_ has joined #tripleo | 19:14 | |
*** w_ has quit IRC | 19:15 | |
*** jprovazn has quit IRC | 19:20 | |
*** Marga_ has quit IRC | 19:21 | |
*** electrofelix has joined #tripleo | 19:30 | |
openstackgerrit | Dan Sneddon proposed openstack/tripleo-heat-templates: Make External Load Balancer templates work with IPv6 https://review.openstack.org/270700 | 19:36 |
openstackgerrit | Emilien Macchi proposed openstack/instack-undercloud: puppet: update mistral deployment https://review.openstack.org/277404 | 19:37 |
*** panda has quit IRC | 19:37 | |
*** panda has joined #tripleo | 19:38 | |
*** electrofelix has quit IRC | 19:50 | |
*** trown|lunch is now known as trown | 19:52 | |
*** Goneri has quit IRC | 20:06 | |
*** Goneri has joined #tripleo | 20:09 | |
*** jtomasek has quit IRC | 20:10 | |
*** Marga_ has joined #tripleo | 20:23 | |
*** yamahata has joined #tripleo | 20:37 | |
openstackgerrit | Colleen Murphy proposed openstack/diskimage-builder: Fix dpkg element for Ubuntu Xenial https://review.openstack.org/278617 | 20:40 |
trown | dmsimard: found an issue with https://review.openstack.org/#/c/276790/6 which causes HA job to fail | 20:40 |
trown | ceph job looks like spurious failure to me | 20:41 |
dmsimard | checkling | 20:41 |
dmsimard | checking, to | 20:41 |
dmsimard | sigh | 20:41 |
* dmsimard needs to get used to new keyboard | 20:41 | |
trown | slagle: bnemec, any chance you want to approve https://review.openstack.org/#/c/275607/ | 20:41 |
dmsimard | trown: you're right | 20:42 |
trown | I didnt catch it because I didnt test HA | 20:42 |
slagle | i want to approve many things | 20:43 |
trown | thanks bnemec | 20:44 |
*** jpeeler has quit IRC | 20:44 | |
openstackgerrit | David Moreau Simard proposed openstack/tripleo-heat-templates: Nova now requires an api database to be created https://review.openstack.org/276790 | 20:44 |
slagle | oh man, bnemec stole my thanks | 20:44 |
dmsimard | trown: ^ | 20:44 |
bnemec | slagle: Ninja'd :-P | 20:45 |
trown | slagle: thanks for wanting to approve it :) | 20:45 |
dmsimard | Dang, https://review.openstack.org/#/c/275607/ was double approved | 20:45 |
dmsimard | that makes it double awesome | 20:45 |
trown | workflow +2 | 20:45 |
* bnemec needed a break from trying to figure out how the client validations aren't breaking the world anyway | 20:46 | |
*** eil397 has joined #tripleo | 20:47 | |
openstackgerrit | Merged openstack/instack-undercloud: Update puppet nova to sync new Nova API database https://review.openstack.org/275607 | 20:50 |
trown | 1 down 3 to go | 20:56 |
openstackgerrit | Ben Nemec proposed openstack/python-tripleoclient: Don't warn about missing profiles when they're not used https://review.openstack.org/276688 | 20:58 |
*** bvandenh_ has quit IRC | 21:02 | |
*** egafford has quit IRC | 21:03 | |
*** yamahata has quit IRC | 21:06 | |
dustins | Hey all! Now that I've got the gate checks working, could I get some eyes on https://review.openstack.org/#/c/188137? | 21:10 |
dustins | The Manila Integration patch | 21:10 |
*** Marga_ has quit IRC | 21:16 | |
*** tiswanso has quit IRC | 21:24 | |
openstackgerrit | John Trowbridge proposed openstack/tripleo-common: DO NOT MERGE testing current with master puppet-nova https://review.openstack.org/278553 | 21:24 |
openstackgerrit | Dan Sneddon proposed openstack/tripleo-heat-templates: Add sysctl settings to disable IPv6 on unconfigured interfaces https://review.openstack.org/277182 | 21:25 |
openstackgerrit | Ben Swartzlander proposed openstack/tripleo-heat-templates: Add integration with NetApp Manila driver https://review.openstack.org/188138 | 21:28 |
*** dustins_ has joined #tripleo | 21:29 | |
*** dustins has quit IRC | 21:32 | |
*** dustins_ is now known as dustins | 21:33 | |
*** jayg is now known as jayg|g0n3 | 21:37 | |
*** tremble has quit IRC | 21:37 | |
*** pradk_ has joined #tripleo | 21:38 | |
*** dprince has quit IRC | 21:40 | |
*** mburned` has joined #tripleo | 21:41 | |
*** crinkle_ has joined #tripleo | 21:41 | |
openstackgerrit | Merged openstack/tripleo-heat-templates: Fix endpoint names https://review.openstack.org/275436 | 21:42 |
*** CaptTofu__ has joined #tripleo | 21:45 | |
*** crinkle has quit IRC | 21:47 | |
*** HenryG_ has joined #tripleo | 21:47 | |
*** crinkle_ is now known as crinkle | 21:47 | |
openstackgerrit | Merged openstack/tripleo-common: Output some debug info when pingtest fails https://review.openstack.org/274367 | 21:47 |
*** pradk has quit IRC | 21:48 | |
*** gfidente has quit IRC | 21:48 | |
*** jrist has quit IRC | 21:48 | |
*** mburned has quit IRC | 21:48 | |
*** HenryG has quit IRC | 21:48 | |
*** dhellmann has quit IRC | 21:48 | |
*** CaptTofu_ has quit IRC | 21:48 | |
*** ramishra has quit IRC | 21:48 | |
*** larsks has quit IRC | 21:48 | |
*** HenryG_ is now known as HenryG | 21:48 | |
*** larsks has joined #tripleo | 21:49 | |
*** dhellmann has joined #tripleo | 21:49 | |
openstackgerrit | Merged openstack/python-tripleoclient: Revert "Pass environment variables of proxy to tox" https://review.openstack.org/267743 | 21:50 |
openstackgerrit | Merged openstack/diskimage-builder: Fix growroot for Gentoo's openrc https://review.openstack.org/276578 | 21:50 |
*** tiswanso has joined #tripleo | 21:52 | |
*** HenryG has quit IRC | 21:52 | |
*** jrist has joined #tripleo | 21:54 | |
*** jrist has joined #tripleo | 21:54 | |
*** xinwu has joined #tripleo | 21:55 | |
*** HenryG has joined #tripleo | 21:56 | |
*** ramishra has joined #tripleo | 21:56 | |
*** lblanchard has quit IRC | 22:05 | |
*** rcernin has quit IRC | 22:07 | |
*** dustins has quit IRC | 22:10 | |
*** trown is now known as trown|outtypewww | 22:13 | |
*** shardy has quit IRC | 22:14 | |
*** julim has quit IRC | 22:16 | |
*** Marga_ has joined #tripleo | 22:16 | |
*** Goneri has quit IRC | 22:17 | |
*** julim has joined #tripleo | 22:17 | |
*** julim has quit IRC | 22:17 | |
*** sthillma has joined #tripleo | 22:20 | |
*** dustins has joined #tripleo | 22:20 | |
*** dcain1 has joined #tripleo | 22:20 | |
*** chlong_ has joined #tripleo | 22:22 | |
*** dhellmann has quit IRC | 22:22 | |
*** chlong has quit IRC | 22:22 | |
*** dcain has quit IRC | 22:22 | |
*** dhellmann has joined #tripleo | 22:23 | |
*** bvandenh_ has joined #tripleo | 22:23 | |
*** tiswanso has quit IRC | 22:25 | |
openstackgerrit | Merged openstack/python-tripleoclient: Remove tripleoclient.baremetal wrapper https://review.openstack.org/265336 | 22:32 |
openstackgerrit | Merged openstack/tripleo-puppet-elements: Add puppet-kmod into image https://review.openstack.org/276249 | 22:36 |
*** bvandenh_ has quit IRC | 22:37 | |
openstackgerrit | Merged openstack/tripleo-heat-templates: Remove not needed completion-signal https://review.openstack.org/255927 | 22:39 |
*** penick has quit IRC | 22:39 | |
openstackgerrit | Merged openstack/diskimage-builder: Mark ironic-discoverd-ramdisk as deprecated in favor of ironic-agent https://review.openstack.org/270267 | 22:48 |
openstackgerrit | Merged openstack/tripleo-heat-templates: Set 'host' globally in Cinder instead of per-backend basis https://review.openstack.org/277517 | 22:48 |
openstackgerrit | Merged openstack/tripleo-heat-templates: Pass -q option to yum https://review.openstack.org/277325 | 22:49 |
openstackgerrit | Merged openstack-infra/tripleo-ci: scp/ssh do not check for know host keys https://review.openstack.org/269660 | 22:51 |
*** rpothier has quit IRC | 22:56 | |
*** saneax_AFK is now known as saneax | 22:57 | |
*** eil397 has quit IRC | 22:58 | |
*** eil397 has joined #tripleo | 23:01 | |
openstackgerrit | Merged openstack/tripleo-docs: Add a note about rerunning the installer https://review.openstack.org/222302 | 23:02 |
*** rwsu has quit IRC | 23:06 | |
*** egafford has joined #tripleo | 23:12 | |
*** yamahata has joined #tripleo | 23:14 | |
*** dustins has quit IRC | 23:18 | |
*** penick has joined #tripleo | 23:23 | |
openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Source undercloud environment variable from a file https://review.openstack.org/275667 | 23:26 |
openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Split the deploy script into its own file https://review.openstack.org/275668 | 23:26 |
ayoung | marios, can you take another look at https://review.openstack.org/#/c/213175/ and make sure I got it looking right. The fact that the previous version passed CI has me concerned that I could miss something and then break someone down the road | 23:31 |
ayoung | "puppet: run keystone in wsgi" https://review.openstack.org/#/c/213175/ EmilienM you, too, please. If it meets your standards, +1. | 23:34 |
*** egafford has quit IRC | 23:36 | |
openstackgerrit | Merged openstack/diskimage-builder: py26 is no longer supported by Infra's CI https://review.openstack.org/261656 | 23:36 |
*** egafford has joined #tripleo | 23:37 | |
*** panda has quit IRC | 23:37 | |
*** panda has joined #tripleo | 23:38 | |
*** chlong_ has quit IRC | 23:46 | |
*** weshay has quit IRC | 23:49 | |
*** egafford has quit IRC | 23:51 | |
*** egafford has joined #tripleo | 23:52 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!