| *** tiswanso has joined #tripleo | 00:08 | |
| *** eil397 has quit IRC | 00:13 | |
| *** tiswanso has quit IRC | 00:13 | |
| *** dmacpher-afk has quit IRC | 00:20 | |
| *** sshnaidm has quit IRC | 00:25 | |
| *** sshnaidm has joined #tripleo | 00:31 | |
| *** thrash is now known as thrash|g0ne | 00:33 | |
| *** sshnaidm has quit IRC | 00:36 | |
| *** sshnaidm has joined #tripleo | 00:38 | |
| *** lblanchard has joined #tripleo | 00:49 | |
| openstackgerrit | Merged openstack/tripleo-common: Allow updating images via `tripleo.sh --overcloud-images` https://review.openstack.org/283607 | 00:55 |
|---|---|---|
| *** saneax is now known as saneax_AFK | 00:56 | |
| *** Goneri has joined #tripleo | 00:59 | |
| *** dmacpher has joined #tripleo | 01:09 | |
| *** tiswanso has joined #tripleo | 01:10 | |
| *** masco has joined #tripleo | 01:11 | |
| *** Goneri has quit IRC | 01:12 | |
| *** Goneri has joined #tripleo | 01:12 | |
| *** tiswanso has quit IRC | 01:20 | |
| *** penick has quit IRC | 01:28 | |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-docs: Update list of pkgs in includepkgs https://review.openstack.org/283899 | 01:34 |
| *** shivrao has quit IRC | 01:35 | |
| *** trown|outtypewww has quit IRC | 01:37 | |
| openstackgerrit | Merged openstack/instack: Remove Py33 support https://review.openstack.org/266116 | 01:39 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/tripleo-common: Updated from global requirements https://review.openstack.org/268549 | 01:40 |
| openstackgerrit | Merged openstack/tripleo-common: Reuse puppet modules already downloaded https://review.openstack.org/283699 | 01:52 |
| *** tiswanso has joined #tripleo | 01:54 | |
| *** tiswanso has quit IRC | 01:54 | |
| *** tiswanso has joined #tripleo | 01:55 | |
| openstackgerrit | Michael Chapman proposed openstack/tripleo-heat-templates: Adds OpenDaylight support https://review.openstack.org/200253 | 01:57 |
| *** yamahata has quit IRC | 02:00 | |
| *** masco has quit IRC | 02:02 | |
| *** Goneri has quit IRC | 02:06 | |
| *** pcaruana has quit IRC | 02:07 | |
| *** shivrao has joined #tripleo | 02:17 | |
| *** shivrao_ has joined #tripleo | 02:19 | |
| *** pcaruana has joined #tripleo | 02:19 | |
| *** lblanchard has quit IRC | 02:21 | |
| *** shivrao has quit IRC | 02:21 | |
| *** shivrao_ is now known as shivrao | 02:21 | |
| *** Marga_ has quit IRC | 02:39 | |
| *** shivrao has quit IRC | 02:47 | |
| *** shivrao has joined #tripleo | 02:49 | |
| *** shivrao has quit IRC | 02:54 | |
| *** pcaruana has quit IRC | 03:01 | |
| *** pcaruana has joined #tripleo | 03:16 | |
| *** links has joined #tripleo | 03:46 | |
| *** rlandy has quit IRC | 03:49 | |
| *** pcaruana has quit IRC | 04:01 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/tripleo-common: Updated from global requirements https://review.openstack.org/268549 | 04:02 |
| *** xinwu has quit IRC | 04:09 | |
| openstackgerrit | Merged openstack/puppet-tripleo: Override X-Forwarded-Proto header https://review.openstack.org/283074 | 04:15 |
| *** pcaruana has joined #tripleo | 04:15 | |
| *** masco has joined #tripleo | 04:25 | |
| *** xinwu has joined #tripleo | 04:35 | |
| *** Marga_ has joined #tripleo | 04:36 | |
| *** Marga_ has quit IRC | 04:40 | |
| *** Marga_ has joined #tripleo | 04:43 | |
| *** tiswanso has quit IRC | 04:48 | |
| *** penick has joined #tripleo | 04:48 | |
| *** penick_ has joined #tripleo | 04:50 | |
| *** penick has quit IRC | 04:53 | |
| *** penick_ is now known as penick | 04:53 | |
| *** Marga_ has quit IRC | 04:57 | |
| *** Marga_ has joined #tripleo | 04:58 | |
| *** anande has joined #tripleo | 04:59 | |
| openstackgerrit | Martin André proposed openstack/tripleo-specs: Spec for TripleO validations https://review.openstack.org/255792 | 05:08 |
| *** trozet has quit IRC | 05:22 | |
| *** liverpooler has quit IRC | 05:27 | |
| *** anande has quit IRC | 05:36 | |
| *** weshay has quit IRC | 05:36 | |
| *** pcaruana has quit IRC | 05:38 | |
| *** pcaruana has joined #tripleo | 05:53 | |
| *** akuznetsov has joined #tripleo | 05:56 | |
| *** saneax_AFK is now known as saneax | 06:16 | |
| *** akuznetsov has quit IRC | 06:19 | |
| *** openstack has joined #tripleo | 13:21 | |
| *** openstackstatus has joined #tripleo | 13:22 | |
| *** ChanServ sets mode: +v openstackstatus | 13:22 | |
| bandini_ | jistr: yeah I saw that the wsgi change went in only after the fact. I would have supported it only within the lightweight arch discussion. As is, things get rather complicated not only upgrade wise, but also the whole constraint setup | 13:26 |
| bandini_ | aka we make everything depend on httpd which hosts keystone and horizon, which sounds really wrong | 13:26 |
| trown | pradk: tripleo pin for delorean has been updated, so you should have a recent version of ceilo packages to test against now | 13:27 |
| trown | derekh++ | 13:27 |
| derekh | trown: actually didn't update the current-tripleo link as I was giving existing jobs a chance to finish, I guess thats why you set it in tripleo.sh | 13:28 |
| derekh | trown: will update it now | 13:28 |
| trown | derekh: ya, that part is more a formality though, since we are pinned in tripleo.sh | 13:29 |
| derekh | trown: hmm, looks like I don't have permission on the new server, will ask jpena | 13:29 |
| derekh | trown: yup | 13:29 |
| trown | derekh: I think he copied keys, but you would need to ssh as root | 13:29 |
| derekh | trown: yup, root works, thanks | 13:30 |
| bandini_ | jistr: could you send me a cib file of a latest deployment, so I can take a look? | 13:31 |
| jistr | bandini_: yeah i thought so too... EmilienM, jayg: ^^ can we revert keystone WSGI and land it again with correct constraints and upgrade support? For one thing, i think the lack of migration logic will block us from getting the upgrade job from liberty to mitaka working, and given that the constraints seem incorrect, we can't just jump into implementing the migration logic for what has been merged. It might be better to revert it first and | 13:32 |
| jistr | land it again with correct constraints and migration support. | 13:32 |
| jistr | bandini_: i should have a fresh deployment shortly, will do | 13:33 |
| jistr | EmilienM, jayg: i mean, if we had an upgrade CI job working already, the patch wouldn't be allowed to land in the first place | 13:33 |
| derekh | trown: done | 13:34 |
| trown | derekh: can we switch this to consistent? https://github.com/openstack-infra/tripleo-ci/blob/master/toci_gate_test.sh#L62 | 13:34 |
| trown | I will put up a patch for that and for tripleo.sh to put back current-tripleo | 13:35 |
| derekh | trown: yes, make more sense | 13:35 |
| ayoung | bandini_, we need to rip off this bandage | 13:35 |
| derekh | *makes | 13:36 |
| ayoung | tripleo should never have been written with Keystone in Eventlet in the first place. | 13:36 |
| ayoung | so...lets solve this, and solve it right | 13:36 |
| jaosorior | jistr: Do we really need to revert it? Isn't it possible to propose another patch with the logic to upgrade? | 13:36 |
| jayg | jistr: could there not just be a new patch fixing the constraints instead of reverting whatever made it in? ayoung was pretty adamant that this is a non-negotiable change since eventlet support is gone in mitaka | 13:37 |
| jayg | and he beat me :) | 13:37 |
| hewbrocca | ayoung: there are several things we are trying to solve, and solve right | 13:37 |
| fabbione | fixing the constraints is easy | 13:37 |
| fabbione | and safe | 13:37 |
| hewbrocca | and they are in conflict at the moment | 13:37 |
| fabbione | i gave you the steps | 13:37 |
| fabbione | but that bug keeps being pushed around from "urgent" to "nobody-cares" | 13:38 |
| ayoung | fabbione, link? | 13:38 |
| jayg | which bug is this? | 13:38 |
| fabbione | and i understand why | 13:38 |
| openstackgerrit | John Trowbridge proposed openstack/tripleo-common: Return current-tripleo symlink https://review.openstack.org/284130 | 13:38 |
| fabbione | sec | 13:38 |
| hewbrocca | fabbione: is referring to the change to add the dummy resource to our pacemaker config | 13:38 |
| fabbione | i need to dig the num | 13:38 |
| fabbione | yes that one | 13:38 |
| hewbrocca | so that restarting keystone doesn't restart the entire cluster | 13:38 |
| jistr | jaosorior, jayg: yeah i know it's gone. We can surely have a patch which fixes the constraints *and* prevents upgrades from breaking. My only concern is that until we do have that patch, we cannot start gating on upgrades, because liberty->mitaka upgrade will not work with the current state of things. (It wouldn't work even with correct constraints. This is mainly about the missing upgrade logic.) | 13:38 |
| hewbrocca | (or httpd) | 13:38 |
| hewbrocca | And | 13:39 |
| hewbrocca | the feature freeze is in one week | 13:39 |
| fabbione | https://bugzilla.redhat.com/show_bug.cgi?id=1290121 | 13:39 |
| openstack | bugzilla.redhat.com bug 1290121 in rhel-osp-director "[RFE] Remove keystone constraints and add the openstack-core dummy resource in its place" [High,Assigned] - Assigned to jstransk | 13:39 |
| ayoung | so really https://bugs.launchpad.net/tripleo/+bug/1537885 | 13:39 |
| openstack | Launchpad bug 1537885 in tripleo "Remove keystone constraints and add the openstack-core dummy resource in its place" [Undecided,New] | 13:40 |
| fabbione | i mentioned several times that moving keystone to http requires that change | 13:40 |
| jistr | ayoung: yup, this, plus a migration logic for the same thing | 13:40 |
| pradk | trown, cool, thx | 13:40 |
| ayoung | Make it happen. It would take me months to figure out the logic here | 13:40 |
| ayoung | I am sorry, I can't master every technology in OpenStack | 13:40 |
| jistr | yea i guess we need to find someone who isn't tasked through the roof with Liberty things | 13:41 |
| ayoung | I am heads down tryin to do about 5 different things, and I trust you guys to do the right thing by Keystone. | 13:41 |
| jistr | or un-task someone from Liberty things | 13:41 |
| jayg | jistr: did this patch go to liberty as well as master? can you repaste the review link, I dont have it handy | 13:41 |
| *** tzumainn has joined #tripleo | 13:41 | |
| jistr | jayg: just master https://review.openstack.org/#/c/213175/ | 13:42 |
| openstackgerrit | John Trowbridge proposed openstack-infra/tripleo-ci: Use consistent delorean symlink for periodic job https://review.openstack.org/284131 | 13:42 |
| jayg | so if we are worried about feature freeze for liberty, how is this a problem for right now? | 13:43 |
| hewbrocca | jayg: +1 | 13:43 |
| ayoung | It went to master only, AFAICT | 13:43 |
| jayg | I agree it needs to be fixed, lets put it on the backlog | 13:43 |
| hewbrocca | to be clear, the problem is | 13:43 |
| jistr | jayg: that means kilo->liberty upgrades (main downstream concern) aren't broken by that, but liberty->mitaka upgrades (upstream upgrade job which is being baked) are broken | 13:43 |
| hewbrocca | yes | 13:43 |
| ayoung | I didn't backport, and would not suggest it get backported to liberty. | 13:43 |
| jistr | jayg: we're worried about Mitaka freeze, not Liberty freeze | 13:43 |
| jayg | ah, ok | 13:44 |
| jayg | so switching to this dummy resource doesn't look hard froma quick read, we have used such a thing before | 13:44 |
| bandini_ | when is the mitaka freeza again? | 13:44 |
| jayg | so why not do a patch to get that in, and another for whatever upgrade stuff is needed? | 13:44 |
| hewbrocca | If you can do it quickly | 13:45 |
| hewbrocca | I mean, it's either that, or revert the whole thing | 13:45 |
| hewbrocca | those are the two options | 13:45 |
| jayg | well, I am not suggesting I can, but I dont see how a revert is an option | 13:45 |
| jayg | we have broken keystone or broken upgrades without this | 13:45 |
| jistr | jayg: +1 such two patches would fix it. I brought it up because if we can't do that in time (as i mentioned folks are tasked with Liberty a lot), then we have a broken Mitaka on the freeze date. | 13:46 |
| jayg | jistr: ok, then I am agreeing, I think I missed some of the backchat | 13:46 |
| ayoung | jistr, lets not focus on Freeze date. That is feature freeze, and this would be a bug fix | 13:46 |
| jayg | ah, that is a good point | 13:47 |
| jistr | ok yea | 13:47 |
| ayoung | the time from M3 to release is for stabilization, just this sort of thing. | 13:47 |
| EmilienM | jistr: I'm not in favor of reverts in general, except if CI is broken or if something is blocked. | 13:47 |
| hewbrocca | the problem is it is blocking upgrade CI | 13:47 |
| *** Goneri has joined #tripleo | 13:47 | |
| ayoung | hewbrocca, agreed, so lets get that working. | 13:47 |
| hewbrocca | and that is a huge problem for us right this second | 13:47 |
| EmilienM | but since OOO CI does not have testing for upgrade, I'm not in favor of any revert around keystone/wgsi | 13:47 |
| jayg | I think we have agreed it just needs to be fixed, not reverted | 13:48 |
| EmilienM | sorry I did not follow all discussion | 13:48 |
| jayg | it is just a prioritization and timing discussion now | 13:48 |
| jayg | s'okay, it was long :) | 13:48 |
| ayoung | fabbione, can you work up a WIP patch along the lines of what you wrote in https://bugs.launchpad.net/tripleo/+bug/1537885 | 13:48 |
| openstack | Launchpad bug 1537885 in tripleo "Remove keystone constraints and add the openstack-core dummy resource in its place" [Undecided,New] | 13:48 |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates: Deploy Aodh services, replacing Ceilometer Alarm https://review.openstack.org/241408 | 13:48 |
| fabbione | ayoung: no | 13:48 |
| hewbrocca | All right, but if it is not fixed *very quickly*, then we need to take another look at reverting | 13:48 |
| ayoung | who can? | 13:48 |
| fabbione | ayoung: i never ever touched tripleo or puppet | 13:48 |
| fabbione | people in this channel | 13:49 |
| fabbione | ? | 13:49 |
| hewbrocca | I think you guys are missing the point of the time crunch we are in here | 13:49 |
| jayg | so we need to have some people focused on mitaka, since others are focused on liberty/downstream is basically the thing, imo | 13:49 |
| jayg | hewbrocca: I dont see how a revert is an option, that may fix upgrade, but it breaks the deployment | 13:49 |
| fabbione | ok extra question | 13:49 |
| jistr | yeah, this is not as much of what needs to happen kind of problem, it's probably more of a "who is able *and* has bandwidth to do this before things go very bad" problem | 13:49 |
| fabbione | can somebody show me the upgrade error? | 13:50 |
| fabbione | maybe there is a simpler fix | 13:50 |
| ayoung | hewbrocca, Tripleo of a large community here, and by going with the wrong qway of deploying Keystone it has essentially introduced a regression that kills a huge swath of functionality. | 13:50 |
| ayoung | so, lets fix it | 13:50 |
| ayoung | fabbione, what *do* you know here? Pacemaker? | 13:50 |
| hewbrocca | fabbione: the "upgrade error" is that the wsgi patch doesn't include any code to change a running eventlet setup over to become a running wsgi setup | 13:50 |
| hewbrocca | that needs to be written | 13:51 |
| jayg | ayoung fabbione: be nice :) | 13:51 |
| ayoung | jayg, no, that was not a ding | 13:51 |
| fabbione | jayg: uh? | 13:51 |
| ayoung | jayg, it was me asking what is his specialty | 13:51 |
| jayg | ok ok | 13:51 |
| jayg | yes, he is the pacemaker guru | 13:51 |
| fabbione | ayoung: look at orgchart | 13:51 |
| jistr | fabbione: yeah, i got an error when running our current upgrade code from mitaka->mitaka too, because it doesn't count with openstack-keystone resource not existing. I already fixed that. But the greater problem is migration for existing eventlet deployments to wsgi, as hewbrocca mentioned. | 13:51 |
| ayoung | I don't know who knows what. We need to pull the pieces together. I know Keystone, and can gfake a few other things | 13:52 |
| fabbione | hewbrocca: ok, then i don't think changing constraint is going to fix it | 13:52 |
| ayoung | I have the utmost respect for all the people here, just trying to let everyone know how important this is | 13:52 |
| fabbione | hewbrocca: it's going to make it hella easier to upgrade | 13:52 |
| hewbrocca | fabbione: no. It's a *prerequisite* for fixing it | 13:52 |
| hewbrocca | but it is not itself going to fix it | 13:52 |
| jistr | exactly | 13:52 |
| fabbione | ayoung: we got it. you told that N times, we are overloaded, not hard-headed | 13:52 |
| jistr | fabbione: +1 :) | 13:53 |
| ayoung | fabbione, I'm the heard headed one | 13:53 |
| fabbione | ayoung: and yes I do pacemkaer/HA and a bunch of other stuff | 13:53 |
| ayoung | hard | 13:53 |
| fabbione | i just don't write code for OOO | 13:53 |
| ayoung | fabbione, and you are awesome. Never let it be thought that I indicated differently... | 13:53 |
| ayoung | Cool. | 13:53 |
| bandini_ | jistr, hewbrocca: so the fix is 1) conversion from eventlet to wsgi part 2) fixing the constraints, yes? | 13:53 |
| ayoung | bandini_, in that order? | 13:53 |
| fabbione | bandini_: fix constraints then conversation | 13:54 |
| jistr | bandini_: yup, exactly. In the other order though. | 13:54 |
| jistr | yea | 13:54 |
| hewbrocca | right | 13:54 |
| bandini_ | ok | 13:54 |
| *** ohamada has joined #tripleo | 13:54 | |
| bandini_ | ayoung: reverse roder ;) | 13:54 |
| ayoung | OK.. fabbione did a pretty clear layout in the bug of what needs to change for the constraints....who can do a WIP patch for that? | 13:54 |
| bandini_ | jistr: while we have a fairly good idea in terms of constraint fixing. what's the story about the eventlet conversion? what needs to be done exactly? | 13:55 |
| fabbione | bandini_: basically, stop delete the keystone systemd resource is the first step | 13:55 |
| jistr | bandini_: i think stop and undefine openstack-keystone resource and all its constraints | 13:55 |
| jistr | bandini_: then create the openstack-core and its constraints | 13:55 |
| fabbione | bandini_: then i guess it's moving openstack-keystone under httpd and restart httpd in pacemaker | 13:55 |
| jistr | bandini_: since this can happen during major upgrade, we will have a window where everything is stopped and we can work with the constraints on stopped rather than running resources, if that helps | 13:56 |
| ayoung | bandini_, my guess is that we need to trigger the HTTPD support in the puppet modules. We do that on a clean install, but I don't know what it means for upgrade | 13:56 |
| ayoung | I know what the end state is supposed to look like | 13:56 |
| bandini_ | fabbione, jistr: do we care about the fact that "httpd" at that point mens both keystone and horizon (and possibly more)? | 13:56 |
| bandini_ | s/mens/means/ | 13:57 |
| fabbione | bandini_: no, we can worry about renaming it at some other times | 13:57 |
| fabbione | for mikata+1 | 13:57 |
| fabbione | or whatever | 13:57 |
| ohamada | hello guys, could you please help - i need to reduce the keystone timeout in the overcloud before deploying the overcloud - like to edit the templates. Could you please point me where to look? | 13:57 |
| bandini_ | fabbione: am not worried about the name, am more concerned about "pcs resource restart httpd" now means *two* things in one | 13:57 |
| jistr | bandini_: that's bad, but i'm not sure we can do anything about it | 13:58 |
| bandini_ | if we care about that, that is | 13:58 |
| gfidente | jistr, fabbione, ayoung ... any chance we switch to httpd only for new deployments? | 13:58 |
| ayoung | bandini_, from a depednecy graph, we should be OK. If we restart, say, Nova, we should not restart HTTPD when it means only Horizon. And with Keystone in there, obviously not restarting HTTPD is the right thing. | 13:58 |
| ayoung | gfidente, It means we are going to be chasing Eventlet bugs in the new release | 13:58 |
| bandini_ | ayoung: ack | 13:59 |
| jistr | gfidente: i don't think so... puppet manifest in new t-h-t will try to create the new constraints, so you'd have 2 sets of competing constraints (old + new), or something would plainly error out, if you don't have the migration logic | 13:59 |
| bandini_ | ayoung: the only minor thing is that now restarting the httpd service means restarting both horizon and keystone. no biggie I guess, but worth being aware of | 13:59 |
| ayoung | it also means there is no way to get the Keystone HTTPD specific features, and that is a pretty major bummer. And eventlet support is depracated. | 13:59 |
| gfidente | jistr, yeah was just raising a point about making httpd an optional thing | 13:59 |
| jayg | bandini_: I would propose dealing with that is patch 3, and maybe can fall to bugfix time? | 14:00 |
| jistr | gfidente: oh you mean support *both* options in t-h-t... hmm i think it's easier to do the fix maybe.. | 14:00 |
| gfidente | ahahah | 14:00 |
| gfidente | I meant both yes | 14:00 |
| jayg | also, there is no eventlet support in mitaka, so how can we support it? | 14:00 |
| ayoung | bandini_, it won't hurt anything if it is just a restart. Shutting it off and leaving it off is no different than doing that with Just Keystone...I think it is status quo? | 14:00 |
| bandini_ | jayg: +1 definitely the last thing to look at in terms of priorities | 14:00 |
| gfidente | ok sorry about noise | 14:01 |
| bandini_ | ayoung: yeah I think it is not a big deal, as long as we document it somewhere | 14:01 |
| ayoung | OK...I had a WIP patch that I killed...let me see: | 14:01 |
| fabbione | bandini_: I am still waiting a response for that. | 14:01 |
| ayoung | https://review.openstack.org/#/c/276285/ | 14:01 |
| fabbione | bandini_: ideally we would have separated http resources for each WSGI service | 14:01 |
| jaosorior | When is the freeze? | 14:02 |
| *** saneax_AFK is now known as saneax | 14:02 | |
| hewbrocca | 3 March | 14:02 |
| hewbrocca | -ish | 14:02 |
| fabbione | bandini_: but apparently we don't have time to get it right and IIRC OSPd right now deploys with systemd:httpd and not ocf:: | 14:02 |
| ayoung | jaosorior, M3. But that is Feature freeze. The real deal is that CI is broken | 14:02 |
| fabbione | bandini_: multiple httpd instance require ocf | 14:02 |
| ayoung | for upgrades from L->M | 14:02 |
| jistr | to clarify: we don't have that ^ CI yet, but this would be another blocker on the road between now and having it | 14:03 |
| ayoung | OK...so, first question. Is https://review.openstack.org/#/c/276285/1/manifests/wsgi/apache.pp even the right file? | 14:03 |
| *** weshay has joined #tripleo | 14:03 | |
| bandini_ | fabbione: ah righto, so that would be another step to go through | 14:04 |
| fabbione | yes | 14:04 |
| ayoung | Back in a bit..appointment | 14:04 |
| *** ayoung has quit IRC | 14:04 | |
| fabbione | somebody decided to go for this intermediate dirty solution as interim step | 14:04 |
| fabbione | not my decision, but who am I to enforce it? | 14:05 |
| jayg | jistr: could the upgrade CI perhaps be held off until bugfix time, to allow a bit of breathing room for this? | 14:05 |
| hewbrocca | jayg: unfortunately it means we have no way to test upgrades upstream | 14:06 |
| hewbrocca | which is really important | 14:06 |
| jayg | fwiw, this patch has been wip for many months, and we have been unable to get everyone who should care to be involved, largely due to focus on liberty instead of mitaka | 14:06 |
| hewbrocca | jayg: *facepalm* | 14:06 |
| jistr | jayg: i'm actually thinking now that it's a possibility that we'll only get the upgrade CI working from mitaka to newton. Not that i like that, but it might happen. That doesn't mean we can skip solving the problem, it only means we'll hit it downstream rather than upstream (sigh, again). | 14:07 |
| jayg | I think we need to get a bit of split here to allow people to focus on each release | 14:07 |
| hewbrocca | Well, hopefully we can fix it | 14:07 |
| hewbrocca | EmilienM, mburned WDYT | 14:07 |
| jayg | so a team focused on master +upgrade, and another focused on stabilization of master -1 | 14:07 |
| jayg | I feel everyones attention is currently too divided | 14:08 |
| hewbrocca | jayg: it's not a bad idea, although I think the first thing is a prequisite for the second thing | 14:08 |
| hewbrocca | because we're so late with Liberty | 14:08 |
| *** chem has joined #tripleo | 14:08 | |
| hewbrocca | (otherwise it wouldn't be, necessarily) | 14:08 |
| jayg | yeah, but if we keep focusing on only master-1, it will keep happening that way for each release | 14:09 |
| *** bandini_ is now known as bandini | 14:09 | |
| hewbrocca | jayg: totally agree. We should be focusing on master +upgrade right now | 14:09 |
| hewbrocca | b/c if we don't we have no CI and we're just kicking the can down the road | 14:09 |
| hewbrocca | (which is why I would rather not land this wsgi thing right this minute, but apparently we have no choice) | 14:10 |
| jayg | right, maybe this should be a 'feature team', even though those are mostly right now for liberty? | 14:10 |
| jayg | sadly we do not, we got to this point because we did not have enough people focused on mitaka in the particular area | 14:10 |
| jayg | so we have steps, we just need to allocate people to fix it | 14:11 |
| hewbrocca | EmilienM: ^^^ this is for you to sort out when you get back I think | 14:11 |
| dtantsur | folks, I have an excellent offer for you! you can reduce your agent images by ~ 80 MiB (!!) by only approving https://review.openstack.org/#/c/279453/ :) | 14:14 |
| dtantsur | already has 1 +2 and passed the gate | 14:14 |
| hewbrocca | dprince: ^^^ little help please? | 14:14 |
| *** julim has joined #tripleo | 14:15 | |
| jayg | while we wait for EmilienM to return, anyone have a pointer to the doc ohamada needs for configuring keystone timeout in the overcloud? I do not know where to point him | 14:15 |
| jistr | bandini: this is current cib http://paste.fedoraproject.org/328410/32331314/ | 14:15 |
| * jayg uses this lull to take the dogs for a walk, bbs | 14:15 | |
| *** jaosorior has quit IRC | 14:18 | |
| *** jaosorior has joined #tripleo | 14:19 | |
| *** lblanchard has joined #tripleo | 14:22 | |
| jistr | ohamada, jayg: depends on what timeout is meant, but if it's something in keystone.conf, then that can be customized in hiera here https://github.com/openstack/tripleo-heat-templates/blob/838c998d3d6b3f3703f1ba3fea7a47c862e09212/puppet/hieradata/controller.yaml#L50-L54 | 14:22 |
| *** tiswanso has joined #tripleo | 14:22 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/instack-undercloud: Enable extra hardware data collection and processing for ironic-inspector https://review.openstack.org/277974 | 14:23 |
| jistr | ohamada: instead of editing that file, the hiera parameter can be overriden via the ExtraConfig heat stack parameter: https://github.com/openstack/tripleo-heat-templates/blob/9e473e4b54197811ede59b07bfb3d0e79475e2e1/overcloud.yaml#L406 | 14:23 |
| jistr | ohamada: could be done via a Heat environment file passed into `openstack overcloud deploy`. The environment file would look something like this: http://fpaste.org/328418/23926145/ | 14:26 |
| ohamada | jistr: thank you, i want to reduce the keystone default timeout of 14400 secs - my deployment of overcloud is failing constantly but i do not want to wait four hours each time | 14:26 |
| jistr | oh then i guess you want to set *undercloud* keystone timeout | 14:26 |
| jistr | ohamada: and i guess you mean the keystone token validity timeout? | 14:27 |
| ohamada | jistr: probably - i'm not sure. The deployment always get stuck and then 'Authentication error' occurs which kills it | 14:28 |
| jistr | ohamada: alternatively, i think if you want to get a heat deployment stuck in CREATE_IN_PROGRESS to go to CREATE_FAILED, you can `systemctl restart openstack-heat-engine` on the undercloud. I think (but i'm not sure) that this shouldn't have any disastrous side effects for the overcloud stack. | 14:29 |
| hewbrocca | jistr: when are we getting abort support in Heat | 14:29 |
| hewbrocca | Mitaka? | 14:29 |
| hewbrocca | I know there was talk of it | 14:29 |
| * jistr doesn't know | 14:29 | |
| hewbrocca | zaneb: ^^^ "abort" support in Heat was coming, wasn't it? | 14:29 |
| jistr | ohamada: the token expiration can be set in keystone.conf in [token] section in the expiration key, and then you would restart keystone for the change to take effect. But i think the heat-engine restart might be a preferred solution. | 14:33 |
| dprince | hewbrocca: sorry, what exactly did you need help with? | 14:34 |
| ohamada | jistr: thanks, the heat-engine restart works, however there are more failed jobs then before | 14:34 |
| *** saneax is now known as saneax_AFK | 14:37 | |
| hewbrocca | dprince: dtantsur needs a +A | 14:41 |
| hewbrocca | <dtantsur> folks, I have an excellent offer for you! you can reduce your agent | 14:41 |
| hewbrocca | images by ~ 80 MiB (!!) by only approving | 14:41 |
| hewbrocca | https://review.openstack.org/#/c/279453/ :) [15:14] | 14:41 |
| hewbrocca | <dtantsur> already has 1 +2 and passed the gate | 14:41 |
| hewbrocca | 14:41 | |
| dtantsur | :) | 14:41 |
| mburned | +1 | 14:42 |
| mburned | hewbrocca: sorry, was not watching this channel | 14:43 |
| mburned | hewbrocca: the thought of splitting focus between master+1 and master-1 for upgrades? | 14:43 |
| fabbione | bandini: ping | 14:44 |
| bandini | fabbione: pong | 14:44 |
| fabbione | bandini: change of plans, can you please help hewbrocca and jistr with those pacemaker constraints 100% of your time? | 14:44 |
| fabbione | hewbrocca: ^^ please make sure somebody can at least review the patches and help if necessary | 14:45 |
| fabbione | bandini: put autofencing and IHA on hold | 14:45 |
| *** ayoung has joined #tripleo | 14:46 | |
| bandini | fabbione: ack | 14:47 |
| bandini | jistr: would you have 10mins this afternoon for a quick call (I need to understand how I am going to test the upgrade exactly as I am not sure atm)? | 14:48 |
| dtantsur | folks, also an important patch bring UEFI local boot support: https://review.openstack.org/#/c/277842/ (backport to liberty, CI passed) | 14:49 |
| hewbrocca | wut UEFI | 14:49 |
| jistr | bandini: sure | 14:51 |
| ayoung | did you guys solve everything while I was away? | 14:52 |
| jayg | ayoung: yep, we are going to just not use keystone ;) | 14:52 |
| ayoung | jayg, I'd be fine with that | 14:52 |
| bandini | lol | 14:53 |
| jayg | haha | 14:53 |
| ayoung | Keystone really should not exist anyway | 14:53 |
| fabbione | direct access to /etc/passwd | 14:53 |
| fabbione | un-encrypted! | 14:53 |
| jayg | but then won't the arch collapse? | 14:53 |
| * jayg steps out of the entryway | 14:53 | |
| ayoung | fabbione, Kerberos and LDAP. | 14:53 |
| ayoung | jayg, the only thing that Keystone really provides is the service catalog. It could be replaced with a static JSON page rturned when you do curl $OS_AUTH_URL | 14:54 |
| ayoung | but then you would have to authenticate at Nova, Neutron, and Glance etc...and then all those services would have to be run in HTTPD | 14:55 |
| ayoung | :) | 14:55 |
| jayg | sounds simple enough </sarcasm> | 14:55 |
| *** ayoung has quit IRC | 14:59 | |
| lucasagomes | hi all, someone can take a look at https://review.openstack.org/#/c/277842/ please? This is the liberty backport needed for EFI | 15:02 |
| *** pblaho has quit IRC | 15:05 | |
| *** eggmaste` is now known as eggmaster | 15:06 | |
| zaneb | hewbrocca: depends what you mean by 'abort'. if something fails then all of the nested stacks will be aborted as of Liberty (in fact they also were pre-Kilo - only Kilo was broken) | 15:07 |
| zaneb | hewbrocca: we also have had an 'update-cancel' API since at least Liberty, BUT it always does a rollback | 15:08 |
| zaneb | and rollback is basically never what you want for TripleO | 15:08 |
| *** fgimenez has quit IRC | 15:11 | |
| *** fgimenez has joined #tripleo | 15:13 | |
| *** fgimenez has quit IRC | 15:13 | |
| *** fgimenez has joined #tripleo | 15:13 | |
| *** xinwu has joined #tripleo | 15:15 | |
| EmilienM | jistr: why not landing https://review.openstack.org/#/c/270700/ ? | 15:16 |
| EmilienM | CI is green, 3 +2 !! | 15:16 |
| hewbrocca | zaneb: ahh, cancel | 15:16 |
| hewbrocca | that was the thing we were looking for | 15:16 |
| hewbrocca | except, you're right, always doing a rollback is not what's needed I don't guess | 15:17 |
| zaneb | hewbrocca: yeah, you don't want to try rolling back a tripleo deployment | 15:17 |
| hewbrocca | weeeeeeeee | 15:17 |
| jistr | EmilienM: it depends on an unmerged patch | 15:17 |
| EmilienM | jistr: what do we want to land it? | 15:18 |
| *** fgimenez has quit IRC | 15:20 | |
| *** tosky has quit IRC | 15:22 | |
| *** nkinder has joined #tripleo | 15:23 | |
| *** tosky has joined #tripleo | 15:23 | |
| *** fgimenez has joined #tripleo | 15:24 | |
| *** fgimenez has quit IRC | 15:24 | |
| *** fgimenez has joined #tripleo | 15:24 | |
| *** mcornea has quit IRC | 15:26 | |
| derekh | FYI : Via #rdo meeting #agreed rename openstack-packages/delorean to fluzo | 15:28 |
| *** penick has joined #tripleo | 15:29 | |
| paramite | fluzo? | 15:29 |
| paramite | derekh, what's wrong with delorean? | 15:30 |
| trown | paramite: already a project on pypi | 15:30 |
| derekh | dprince: the next time you are playing on the rh1 rack, sshuttle is extremely useful, no more messing with tunnels and X forwarding https://goodsquishy.com/2016/02/sshuttle-where-have-you-been-all-my-life/ | 15:30 |
| *** mcornea has joined #tripleo | 15:30 | |
| derekh | paramite: and a possible trademark issue | 15:30 |
| paramite | ah I see | 15:30 |
| dprince | derekh: nice | 15:31 |
| * derekh didn't worry about such things 2 years ago | 15:31 | |
| * trown has to change his github picture now | 15:32 | |
| *** paramite is now known as paramite|afk | 15:32 | |
| dprince | derekh/paramite: I'm a holdout for delorean | 15:32 |
| dprince | fluzo sounds like a disease man | 15:33 |
| trown | lol | 15:33 |
| derekh | :-) | 15:33 |
| trown | it is a mistranslation of flux in spanish that was used in the subtitles for back to the future | 15:33 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Source undercloud environment variable from a file https://review.openstack.org/275667 | 15:35 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Split the deploy script into its own file https://review.openstack.org/275668 | 15:35 |
| *** thrash is now known as thrash|biab | 15:37 | |
| *** panda has quit IRC | 15:37 | |
| *** paramite|afk has quit IRC | 15:38 | |
| *** panda has joined #tripleo | 15:38 | |
| *** yamahata has joined #tripleo | 15:39 | |
| *** fgimenez has quit IRC | 15:51 | |
| *** fgimenez has joined #tripleo | 15:52 | |
| *** fgimenez has quit IRC | 15:52 | |
| *** fgimenez has joined #tripleo | 15:52 | |
| *** paramite|afk has joined #tripleo | 15:53 | |
| *** jaosorior has quit IRC | 15:55 | |
| *** penick has quit IRC | 15:58 | |
| *** paramite|afk has quit IRC | 15:58 | |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates: Set notification driver for nova to send https://review.openstack.org/283686 | 15:58 |
| *** oshvartz has quit IRC | 16:00 | |
| *** hjensas has quit IRC | 16:10 | |
| *** yamahata has quit IRC | 16:10 | |
| *** leanderthal is now known as leanderthal|afk | 16:14 | |
| openstackgerrit | Evgeny Bagdasaryan proposed openstack/tripleo-heat-templates: Change the default value for NetworkNexusVxlanGlobalConfig https://review.openstack.org/281957 | 16:14 |
| *** rajinir has joined #tripleo | 16:19 | |
| *** ayoung has joined #tripleo | 16:20 | |
| *** mcornea has quit IRC | 16:21 | |
| *** liverpooler has quit IRC | 16:22 | |
| *** rpothier has joined #tripleo | 16:26 | |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Add IPv6 Support to Isolated Networks https://review.openstack.org/235423 | 16:28 |
| *** tiswanso has quit IRC | 16:31 | |
| *** tiswanso has joined #tripleo | 16:32 | |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-heat-templates: Nova RPC unpinning https://review.openstack.org/284239 | 16:32 |
| openstackgerrit | Ben Swartzlander proposed openstack/tripleo-heat-templates: Enable Manila integration https://review.openstack.org/188137 | 16:36 |
| openstackgerrit | Ben Swartzlander proposed openstack/tripleo-heat-templates: Add integration with NetApp Manila driver https://review.openstack.org/188138 | 16:37 |
| *** jdob has quit IRC | 16:43 | |
| dmsimard | zaneb: re: endpoint map | 16:44 |
| zaneb | dmsimard: yeah | 16:44 |
| dmsimard | tbh I have no idea, there's a lot of things I don't understand in tripleo - I made the patch with the help of other people here that pointed me in the right direction | 16:44 |
| *** jistr has quit IRC | 16:45 | |
| dmsimard | If there's something wrong, it didn't come up in the reviews until now | 16:45 |
| trown | zaneb: we can always do it right now that we have moved CI into the present wrt delorean | 16:45 |
| zaneb | dmsimard: that's fine, I don't think it's critical. we can easily fix up with a follow-up patch | 16:45 |
| trown | zaneb: we were testing tripleo against code from 30 days ago in other projects though | 16:46 |
| openstackgerrit | Zane Bitter proposed openstack/tripleo-heat-templates: Generate the endpoint map statically https://review.openstack.org/275437 | 16:48 |
| trown | zaneb: glad to be fodder for your argument on that patch :) | 16:49 |
| trown | I definitely agree | 16:49 |
| *** Goneri has quit IRC | 16:49 | |
| zaneb | trown: can't overlook a gift like that ;) | 16:50 |
| *** pblaho has joined #tripleo | 16:50 | |
| *** aufi has quit IRC | 16:51 | |
| *** mcornea has joined #tripleo | 16:52 | |
| *** chem has quit IRC | 16:52 | |
| dmsimard | haha, sorry, it was definitely a time sensitive patch and even then it took weeks to get it merged | 16:54 |
| dmsimard | we definitely wanted to base our patch on top of yours | 16:54 |
| trown | dmsimard: funny (well not so funny really) thing is that if we stuck with the endpoint generator based patch, it was correct | 16:55 |
| trown | so we could have updated trunk 2 weeks ago | 16:56 |
| dmsimard | bleh | 16:56 |
| dmsimard | but the generator patch hasn't landed yet | 16:56 |
| *** Guest51435 is now known as mgagne | 16:56 | |
| *** mgagne has quit IRC | 16:56 | |
| *** mgagne has joined #tripleo | 16:56 | |
| trown | I switched to the master one to try to speed things up, but then got lost in an issue I did not attribute to a mistake in that patch | 16:56 |
| trown | a pretty strong argument for the generator patch | 16:57 |
| *** trown is now known as trown|lunch | 16:57 | |
| *** nico_auv has quit IRC | 17:01 | |
| *** ifarkas has quit IRC | 17:02 | |
| *** mikelk has quit IRC | 17:02 | |
| EmilienM | ayoung: chem (offline now) is working on the puppet-pacemaker provider | 17:04 |
| EmilienM | we hope we'll be able to tell puppet "do not manage the service with systemd, just run pcs" | 17:04 |
| ayoung | EmilienM, cool. Feel free to bug me (and pass that on) if there are any questions on the Eventlet to HTTPD transition on Upgrade. Or anything else around this, of course | 17:05 |
| *** yamahata has joined #tripleo | 17:06 | |
| *** CaptTofu___ is now known as CaptTofu | 17:07 | |
| *** tosky has quit IRC | 17:12 | |
| openstackgerrit | Zane Bitter proposed openstack/tripleo-heat-templates: Don't have separate protocols/ports for Keystone v3 https://review.openstack.org/284265 | 17:13 |
| zaneb | dmsimard, trown|lunch: follow-up ^ | 17:14 |
| dmsimard | zaneb: I think that might require a change in the heat/hiera files but I'm a noob | 17:15 |
| dmsimard | i.e https://review.openstack.org/#/c/279184/3/puppet/compute.yaml and https://review.openstack.org/#/c/279184/3/puppet/controller.yaml | 17:15 |
| zaneb | dmsimard: I don't believe so. those files only look at the *output* of that stack. I'm changing the *input* | 17:15 |
| dmsimard | but those are expecting a key in the endpoint map to be called KeystoneV3Admin (for example) no ? | 17:16 |
| dmsimard | with your patch that no longer exists iiuc | 17:17 |
| zaneb | so what's confusing is that both the input and the output are called EndpointMap | 17:17 |
| dmsimard | confusing things? in tripleo? :P | 17:17 |
| zaneb | but there's nothing called "uri" in the input (only host, port, protocol). that's definitely the output | 17:18 |
| dmsimard | you're the expert though, just wanted to point that out and if you see it as a non-issue that's fine | 17:18 |
| zaneb | lol, I'm really not ;) | 17:18 |
| zaneb | but how it works is you have an input called 'EndpointMap' with host, port, protocol. you then launder it through that stack, and also call the output (with host, port, protocol, uri, base_uri) EndpointMap. then you use the output as an input to other things like puppet/compute.yaml | 17:20 |
| openstackgerrit | Evgeny Bagdasaryan proposed openstack/tripleo-heat-templates: Add BondInterfaceOvsOptions parameter to net-config-bond.yaml https://review.openstack.org/245086 | 17:23 |
| openstackgerrit | Jason Guiditta proposed openstack/puppet-tripleo: loadbalancer: fix MySQL timeout HAproxy config https://review.openstack.org/284274 | 17:26 |
| *** fgimenez has quit IRC | 17:27 | |
| *** dtantsur is now known as dtantsur|afk | 17:31 | |
| *** Goneri has joined #tripleo | 17:33 | |
| *** xinwu has quit IRC | 17:45 | |
| *** trown|lunch is now known as trown | 17:50 | |
| *** lucasagomes is now known as lucas-hungry | 17:51 | |
| *** rdopiera has quit IRC | 17:54 | |
| *** Marga_ has quit IRC | 17:55 | |
| *** mkovacik_ has quit IRC | 17:59 | |
| *** thrash|biab is now known as thrash | 18:00 | |
| *** jdob_lt has joined #tripleo | 18:01 | |
| *** eil397 has joined #tripleo | 18:01 | |
| *** derekh has quit IRC | 18:01 | |
| *** olap has quit IRC | 18:02 | |
| *** julim has quit IRC | 18:03 | |
| *** julim has joined #tripleo | 18:07 | |
| *** gfidente has quit IRC | 18:14 | |
| *** xinwu has joined #tripleo | 18:16 | |
| *** jtomasek has quit IRC | 18:21 | |
| *** ohamada has quit IRC | 18:30 | |
| *** electrofelix has quit IRC | 18:30 | |
| *** shivrao has joined #tripleo | 18:31 | |
| *** Marga_ has joined #tripleo | 18:34 | |
| *** olap has joined #tripleo | 18:38 | |
| *** liverpooler has joined #tripleo | 18:41 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Enable configuration of Neutron QoS https://review.openstack.org/272131 | 18:42 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Adds v6 capability to the deploy validation test (pings) https://review.openstack.org/268103 | 18:46 |
| *** jcoufal has quit IRC | 18:48 | |
| *** Marga_ has quit IRC | 18:50 | |
| *** Marga_ has joined #tripleo | 18:51 | |
| *** paramite|afk has joined #tripleo | 18:51 | |
| *** rajinir has quit IRC | 18:55 | |
| *** dustins has joined #tripleo | 18:57 | |
| *** mgould has quit IRC | 19:02 | |
| *** paramite|afk has quit IRC | 19:09 | |
| zigo | Could someone review this one? https://review.openstack.org/#/c/211859/ It's blocking the addition of a Debian image in the infra, and it's really a huge blocker for me. | 19:14 |
| *** athomas has quit IRC | 19:14 | |
| zigo | It's been there since October, so I'm really worried here... | 19:14 |
| zigo | SpamapS: Since nobody is reviewing it, maybe you could workflow it? ^ | 19:15 |
| *** mcornea has quit IRC | 19:17 | |
| *** mkovacik has joined #tripleo | 19:24 | |
| *** eil397 has quit IRC | 19:29 | |
| SpamapS | zigo: ACK, I'll pioritize today | 19:29 |
| SpamapS | zigo: oh, I +2'd ... | 19:30 |
| SpamapS | zigo: greghaynes is usually pretty quick on those reviews but he's out on leave for a bit. | 19:30 |
| SpamapS | ianw: can you maybe take a look at 211859 ? | 19:30 |
| b3nt_pin | sanity check: are the instructions on http://docs.openstack.org/developer/tripleo-incubator/devtest.html expected to work? (more or less) | 19:32 |
| trown | does anyone know if these scale docs are accurate? http://docs.openstack.org/developer/tripleo-docs/post_deployment/scale_roles.html | 19:32 |
| trown | b3nt_pin: highly doubtful | 19:32 |
| b3nt_pin | trown: heh okay | 19:32 |
| b3nt_pin | trown, the next question is do we *want* them to work? ;) | 19:33 |
| b3nt_pin | in other words, is it basically obsolete or is it just something that is bit-rotting at the moment | 19:33 |
| trown | b3nt_pin: I think they are obsolete, devtest is not used in CI | 19:34 |
| b3nt_pin | trown: okay cool | 19:34 |
| trown | b3nt_pin: http://docs.openstack.org/developer/tripleo-docs/ is what you want | 19:34 |
| *** dustins_ has joined #tripleo | 19:34 | |
| trown | b3nt_pin: if those don't work, bugs and patches are welcome | 19:35 |
| *** dustins has quit IRC | 19:35 | |
| b3nt_pin | trown: is that how people are making patches? hand craft in that deployed environment and then patch the code in git? | 19:36 |
| trown | b3nt_pin: probably depends on the person, and on the patch... for tripleo-heat-templates, I just git clone to the home dir and deploy using those templates | 19:37 |
| trown | so putting up a patch is simple from taht | 19:37 |
| *** dustins_ is now known as dustins | 19:38 | |
| trown | other stuff, I build a package for the change using delorean and then install that package to test | 19:38 |
| trown | which is what CI does | 19:38 |
| b3nt_pin | trown: gotcha | 19:39 |
| *** yamahata has quit IRC | 19:41 | |
| *** trown is now known as trown|brb | 19:45 | |
| *** trown|brb is now known as trown | 19:49 | |
| *** dustins has quit IRC | 19:50 | |
| openstackgerrit | Ben Nemec proposed openstack/python-tripleoclient: Refresh node power state at start of loop https://review.openstack.org/283809 | 19:50 |
| *** rpothier has left #tripleo | 19:56 | |
| *** penick has joined #tripleo | 20:01 | |
| dmsimard | dang | 20:02 |
| dmsimard | the tripleo owl has some serious competition | 20:02 |
| dmsimard | https://github.com/openstack/dragonflow | 20:02 |
| *** mkovacik has quit IRC | 20:05 | |
| trown | fire-breathing cat5? no thanks | 20:11 |
| *** dustins has joined #tripleo | 20:17 | |
| Erming_ | trown: can I have question for you? | 20:32 |
| trown | Erming_: sure | 20:33 |
| Erming_ | trown: awesome. After your undercloud env setup (undercloud.qcow2), then I run the following command: | 20:33 |
| Erming_ | oenstack overcloud deploy --templates --control-scale 3 -e /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml --ntp-server pool.ntp.org --ceph-storage-scale 1 --templates -e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml --libvirt-type qemu | 20:34 |
| trown | Erming_: with you so far :) | 20:34 |
| Erming_ | trown: while it had a ' no valid host' error. I found it could be because that it creates a new instance on a already assigned virtual node. | 20:35 |
| Erming_ | that is, it may create more vms than the number of nodes specified in instackenv.json. (I didn't change it at all) | 20:36 |
| Erming_ | trown: so do you think my command is right? | 20:36 |
| trown | Erming_: the command is right, I do not follow your logic on the no valid host though | 20:37 |
| Erming_ | trown: because in your instackenv.json, there are 3 controls, 1 compute, 1 ceph | 20:37 |
| trown | Erming_: for that, you want to look in nova-scheduler.log for which filter returned 0 hosts | 20:38 |
| Erming_ | I give 3 controls and 1 ceph in the command. But in the log file, I saw | 20:38 |
| Erming_ | an instance was trying to be created on a node that has already a VM running. | 20:39 |
| trown | Erming_: so there is a nova scheduler bug like that, but I do not see it often these days | 20:40 |
| trown | Erming_: https://bugs.launchpad.net/nova/+bug/1341420 | 20:40 |
| openstack | Launchpad bug 1341420 in OpenStack Compute (nova) "gap between scheduler selection and claim causes spurious failures when the instance is the last one to fit" [Wishlist,In progress] | 20:40 |
| trown | Erming_: do you reproduce it every time? | 20:41 |
| Erming_ | trown: I just tried once. and just get time to resume this stuff. | 20:43 |
| trown | Erming_: ok I would try to delete the stack and redeploy | 20:46 |
| Erming_ | trown: another suspicion is that I only have 90GB left in / space, could it be the no valid host cause? | 20:46 |
| trown | Erming_: well, no valid host is a super generic error, that is why I said look in the scheduler logs | 20:46 |
| Erming_ | trown: unfortunately I had removed the stack. so scheduler logs .. let me find the original error for you. | 20:47 |
| *** Marga_ has quit IRC | 20:48 | |
| *** Marga_ has joined #tripleo | 20:48 | |
| Erming_ | trown: I checked the flavor and the node themselves, don't see inconsistencies. | 20:49 |
| *** mkovacik has joined #tripleo | 20:51 | |
| Erming_ | trown: sorry I don't know why the previous error log is missing in the nova-scheduler log. I am going to redo it. you think the 90GB / space is not a issue? and the openstack undercloud deploy command is also good? | 20:52 |
| trown | Erming_: deploy command definitely looks right... I doubt that the 90GB of storage was related | 20:53 |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates: Deploy Aodh services, replacing Ceilometer Alarm https://review.openstack.org/241408 | 20:54 |
| trown | Erming_: if you saw nodes being scheduled to already scheduled nodes, that sounds like that nova scheduler bug I posted... which is a race condition, so maybe you will get luckier the second time around :) | 20:54 |
| Erming_ | trown: Hmm. OK. I will do lvresize, and then re-try. before that, just to confirm, for doing it on virtual environment instead of baremetal, I don't need do anything extra other than the this instrcution: | 20:55 |
| Erming_ | "Then proceed with the upstream documentation for the rest of the deployment." --> http://docs.openstack.org/developer/tripleo-docs/basic_deployment/basic_deployment_cli.html#upload-images | 20:56 |
| trown | Erming_: yep, assuming you ran `openstack undercloud install` before that | 20:56 |
| *** penick has quit IRC | 21:00 | |
| Erming_ | trown: if one vm needs 40GB, 5 needs 200GB, where are the VM files located? | 21:08 |
| Erming_ | trown: I am looking at the bug you mentioned at the same time. | 21:08 |
| trown | Erming_: by default they would be in /var/lib/libvirt/images/, but the backing storage is thin provisioned | 21:10 |
| *** pcaruana has quit IRC | 21:10 | |
| trown | Erming_: so you probably only need 20-40G of space | 21:10 |
| Erming_ | trown: OK if so , it shouldn't an issue. as I linked /var/lib/libvirt/images/ to /cloud which has 300GB | 21:11 |
| Erming_ | trown: sounds that I only need try again with running the same command. | 21:11 |
| trown | Erming_: ya, and get scheduler logs if it fails again... ironic-conductor logs would be useful if it was actually some storage related issue | 21:12 |
| trown | but I dont think that is the problem | 21:12 |
| Erming_ | OK. trying! | 21:13 |
| *** Goneri has quit IRC | 21:13 | |
| trown | Erming_: I am taking off, but let me know how it goes | 21:14 |
| *** trown is now known as trown|outtypewww | 21:14 | |
| Erming_ | trown|outtypewww: I will. thanks! | 21:14 |
| ianw | SpamapS: sorry, i probably should have stated my opinion with a vote there. but i just see it as a really undefined layer that will accumulate "stuff" that makes sense for someone at some time, but never leaves | 21:15 |
| ianw | SpamapS: i much prefer to see a more fine grained approach where elements that need things add those dependencies directly | 21:16 |
| *** dustins has quit IRC | 21:36 | |
| *** dustins has joined #tripleo | 21:38 | |
| *** rbrady-run is now known as rbrady | 21:40 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Add TripleO Heat Template Parameters for Neutron Tenant MTU https://review.openstack.org/280852 | 21:41 |
| *** dshulyak has quit IRC | 21:45 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Allow the deployer to pick a predefined IP for VIPs https://review.openstack.org/276316 | 21:47 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/python-tripleoclient: Updated from global requirements https://review.openstack.org/268528 | 21:50 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/tripleo-common: Updated from global requirements https://review.openstack.org/268549 | 21:51 |
| openstackgerrit | Ben Nemec proposed openstack/python-tripleoclient: Refresh node power state at start of loop https://review.openstack.org/283809 | 21:52 |
| *** jdob_lt has quit IRC | 22:01 | |
| openstackgerrit | xin wu proposed openstack/os-net-config: Enable os_net_config to configure IVS https://review.openstack.org/274492 | 22:07 |
| openstackgerrit | xin wu proposed openstack/os-net-config: Enable os_net_config to configure IVS https://review.openstack.org/274492 | 22:08 |
| *** jayg is now known as jayg|g0n3 | 22:09 | |
| openstackgerrit | Merged openstack/instack: remove python 2.6 trove classifier https://review.openstack.org/260790 | 22:11 |
| *** lblanchard has quit IRC | 22:14 | |
| *** dustins has quit IRC | 22:17 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Rename tox env to pep8 https://review.openstack.org/279225 | 22:25 |
| *** dprince has quit IRC | 22:26 | |
| *** penick has joined #tripleo | 22:27 | |
| *** julim has quit IRC | 22:33 | |
| *** tiswanso has quit IRC | 22:37 | |
| openstackgerrit | Ben Nemec proposed openstack/instack-undercloud: Unset DIB_YUM_REPO_CONF before running undercloud install https://review.openstack.org/284444 | 22:43 |
| openstackgerrit | Merged openstack/instack-undercloud: Unset DIB_YUM_REPO_CONF before running undercloud install https://review.openstack.org/271557 | 22:46 |
| openstackgerrit | Merged openstack/os-apply-config: Deprecated tox -downloadcache option removed https://review.openstack.org/256803 | 22:55 |
| openstackgerrit | Merged openstack/os-refresh-config: Deprecated tox -downloadcache option removed https://review.openstack.org/256805 | 22:56 |
| openstackgerrit | Merged openstack/python-tripleoclient: Install bigswitch networking agent by default https://review.openstack.org/282534 | 22:56 |
| openstackgerrit | Merged openstack/tripleo-puppet-elements: Deprecated tox -downloadcache option removed https://review.openstack.org/256842 | 23:01 |
| openstackgerrit | Ben Nemec proposed openstack/instack-undercloud: Enable notifications on undercloud https://review.openstack.org/241341 | 23:08 |
| openstackgerrit | Ben Nemec proposed openstack/instack-undercloud: Enable notifications on undercloud https://review.openstack.org/241341 | 23:09 |
| *** thrash is now known as thrash|g0ne | 23:12 | |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder: Add dib element to generate logical volumes https://review.openstack.org/252041 | 23:12 |
| openstackgerrit | Ryan Brady proposed openstack/tripleo-common: Add Heat Capabilities Actions https://review.openstack.org/284460 | 23:19 |
| *** saneax_AFK is now known as saneax | 23:21 | |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder: Add dib element to generate logical volumes https://review.openstack.org/252041 | 23:22 |
| *** pradk has quit IRC | 23:24 | |
| *** panda has quit IRC | 23:37 | |
| *** panda has joined #tripleo | 23:38 | |
| openstackgerrit | Ben Nemec proposed openstack/instack-undercloud: Enable notifications on undercloud https://review.openstack.org/241341 | 23:44 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!