| openstackgerrit | Mike Burns proposed openstack/tripleo-heat-templates: change default compute hostnames to compute https://review.openstack.org/304342 | 00:03 |
|---|---|---|
| openstackgerrit | Dan Radez proposed openstack/tripleo-heat-templates: Enable deployment of Ceph Storage (OSD) on the Compute Nodes https://review.openstack.org/273754 | 00:11 |
| *** rhallisey has quit IRC | 00:44 | |
| *** tiswanso has joined #tripleo | 00:54 | |
| *** dtrainor has quit IRC | 00:54 | |
| *** apetrich has quit IRC | 00:58 | |
| *** tiswanso has quit IRC | 01:02 | |
| *** tiswanso has joined #tripleo | 01:02 | |
| *** akshai has joined #tripleo | 01:09 | |
| *** akshai_ has joined #tripleo | 01:10 | |
| *** akshai has quit IRC | 01:14 | |
| *** thrash is now known as thrash|g0ne | 01:27 | |
| *** yamahata has quit IRC | 02:00 | |
| *** cwolferh has quit IRC | 02:11 | |
| *** yuanying has quit IRC | 02:12 | |
| *** yuanying has joined #tripleo | 02:12 | |
| *** liverpooler has quit IRC | 02:13 | |
| *** kbyrne has quit IRC | 02:13 | |
| *** tristanC has quit IRC | 02:13 | |
| *** cmyster has quit IRC | 02:13 | |
| *** tristanC has joined #tripleo | 02:14 | |
| *** cmyster has joined #tripleo | 02:14 | |
| *** akshai_ has quit IRC | 02:14 | |
| *** liverpooler has joined #tripleo | 02:14 | |
| *** julim has joined #tripleo | 02:15 | |
| *** kbyrne has joined #tripleo | 02:24 | |
| *** MaxPC has joined #tripleo | 02:34 | |
| *** tzumainn has quit IRC | 02:42 | |
| *** dmacpher is now known as dmacpher-afk | 02:46 | |
| *** MaxPC has quit IRC | 02:46 | |
| *** yuanying has quit IRC | 02:52 | |
| *** Marga_ has quit IRC | 03:14 | |
| *** links has joined #tripleo | 03:20 | |
| *** dmacpher-afk is now known as dmacpher | 03:28 | |
| *** rbrady has quit IRC | 03:28 | |
| *** tiswanso has quit IRC | 03:45 | |
| *** yuanying has joined #tripleo | 03:50 | |
| *** yamahata has joined #tripleo | 04:02 | |
| *** Marga_ has joined #tripleo | 04:04 | |
| *** aparnav has joined #tripleo | 04:07 | |
| *** penick has joined #tripleo | 04:08 | |
| *** penick_ has joined #tripleo | 04:12 | |
| *** Marga__ has joined #tripleo | 04:12 | |
| *** penick has quit IRC | 04:13 | |
| *** penick_ is now known as penick | 04:13 | |
| *** Marga_ has quit IRC | 04:15 | |
| *** liverpooler has quit IRC | 04:22 | |
| *** rwsu_ has joined #tripleo | 04:32 | |
| *** penick has quit IRC | 04:33 | |
| *** rwsu has quit IRC | 04:34 | |
| *** r-mibu has quit IRC | 04:35 | |
| *** Marga__ is now known as fragatina | 04:39 | |
| *** rcernin has joined #tripleo | 04:52 | |
| *** saneax_AFK is now known as saneax | 04:59 | |
| openstackgerrit | Jason E. Rist proposed openstack/tripleo-ui: Modal fixes: header, footer, width, fluidity https://review.openstack.org/304390 | 05:03 |
| openstackgerrit | Jason E. Rist proposed openstack/tripleo-ui: Modal fixes: header, footer, width, fluidity https://review.openstack.org/304390 | 05:04 |
| *** aparnav has quit IRC | 05:05 | |
| *** appu103_ has joined #tripleo | 05:05 | |
| *** appu103_ is now known as aparnav | 05:05 | |
| *** apetrich has joined #tripleo | 05:13 | |
| *** r-mibu has joined #tripleo | 05:20 | |
| *** liverpooler has joined #tripleo | 05:26 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: yum-minimal: strip locale archive to en_* locales https://review.openstack.org/304399 | 05:35 |
| *** apetrich has quit IRC | 05:52 | |
| *** oshvartz has joined #tripleo | 06:03 | |
| *** tesseract has joined #tripleo | 06:24 | |
| *** tesseract is now known as Guest15115 | 06:24 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: yum-minimal: strip locale archive to en_* locales https://review.openstack.org/304399 | 06:31 |
| *** shivrao has quit IRC | 06:34 | |
| *** jprovazn has joined #tripleo | 06:37 | |
| *** florianf has joined #tripleo | 06:52 | |
| *** florianf has quit IRC | 06:52 | |
| *** florianf has joined #tripleo | 06:52 | |
| *** tremble has joined #tripleo | 06:55 | |
| *** tremble has joined #tripleo | 06:55 | |
| *** mgould has joined #tripleo | 07:06 | |
| *** athomas has joined #tripleo | 07:08 | |
| *** paramite has joined #tripleo | 07:08 | |
| *** paramite is now known as paramite|afk | 07:08 | |
| *** ifarkas has joined #tripleo | 07:13 | |
| *** mikelk has joined #tripleo | 07:13 | |
| *** jaosorior has joined #tripleo | 07:19 | |
| *** aufi has joined #tripleo | 07:22 | |
| *** mkovacik has joined #tripleo | 07:24 | |
| *** paramite|afk is now known as paramite | 07:27 | |
| ccamacho | Morning! o/ | 07:31 |
| *** jpena|off is now known as jpena | 07:35 | |
| *** panda has quit IRC | 07:41 | |
| *** panda has joined #tripleo | 07:42 | |
| *** mbound has joined #tripleo | 07:47 | |
| *** jaosorior has quit IRC | 07:47 | |
| *** jaosorior has joined #tripleo | 07:48 | |
| *** shardy has joined #tripleo | 07:48 | |
| *** shivrao has joined #tripleo | 07:49 | |
| *** shivrao_ has joined #tripleo | 07:58 | |
| *** shivrao has quit IRC | 08:01 | |
| *** shivrao_ is now known as shivrao | 08:01 | |
| *** mcornea has joined #tripleo | 08:02 | |
| *** andrearosa has joined #tripleo | 08:06 | |
| *** andrearosa has left #tripleo | 08:06 | |
| *** bvandenh has joined #tripleo | 08:09 | |
| *** fragatina has quit IRC | 08:11 | |
| *** qasims has joined #tripleo | 08:20 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Always use parameter_defaults in environment files https://review.openstack.org/303305 | 08:22 |
| *** jistr has joined #tripleo | 08:25 | |
| *** jcoufal has joined #tripleo | 08:28 | |
| *** yamahata has quit IRC | 08:28 | |
| *** shivrao has quit IRC | 08:33 | |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-heat-templates: change default compute hostnames to compute https://review.openstack.org/304342 | 08:34 |
| *** bvandenh has quit IRC | 08:35 | |
| shardy | marios: Hey, wanted to chat about https://review.openstack.org/#/c/304342/1 | 08:37 |
| shardy | So, it seems we have a downstream patch which names the nodes compute, not novacompute | 08:37 |
| shardy | but if we switch that now, anyone upgrading between the upstream versions will be broken, right? | 08:37 |
| shardy | because it's always been novacompute upstream | 08:38 |
| *** derekh has joined #tripleo | 08:40 | |
| *** sshnaidm has quit IRC | 08:42 | |
| *** dtantsur|afk is now known as dtantsur | 08:42 | |
| jaosorior | fdamn | 08:42 |
| jaosorior | the gate looks pretty broken | 08:42 |
| *** coolsvap has joined #tripleo | 08:44 | |
| ccamacho | jaosorior, good morning :), speaking about broken gates... I'm trying to debug the Jenkins tests for a tripleo.sh patch.. And i'm not able the get the error... I mean... Is not working fine, right? | 08:45 |
| ccamacho | i.e the upgrades, ha, non ha jobs.. | 08:46 |
| jaosorior | ccamacho: Yep, it's all red | 08:46 |
| jaosorior | seems to me that it's broken | 08:46 |
| jaosorior | trying to dig out what's up | 08:46 |
| ccamacho | I will like to debug it also... but not sure where to start | 08:46 |
| ccamacho | for example.. http://fpaste.org/354547/46045084/ | 08:47 |
| jaosorior | well | 08:47 |
| jaosorior | for isntance | 08:47 |
| jaosorior | most of the failures I've been looking at in the past minutes are pretty similar | 08:47 |
| jaosorior | in the sense that | 08:47 |
| jaosorior | 2016-04-12 02:31:52.621 | | Controller | 77046b52-c8d1-4540-9b46-ba318171d744 | CREATE aborted | CREATE_FAILED | 2016-04-12T02:31:32 | | 08:47 |
| jaosorior | and then the overcloud ends up timing out becausae of that | 08:48 |
| *** gfidente has joined #tripleo | 08:48 | |
| *** gfidente has quit IRC | 08:48 | |
| *** gfidente has joined #tripleo | 08:48 | |
| marios | shardy: o/ reading | 08:48 |
| marios | shardy: yes, this is a good point. this 'workaround' was originally going to be delivered with an environment file during upgrades | 08:49 |
| *** paramite is now known as paramite|afk | 08:49 | |
| shardy | marios: Yeah having ComputeHostnameFormat obviously would work around it | 08:50 |
| marios | shardy: well, i think the point is to get this into liberty | 08:50 |
| marios | shardy: because its an upgrades thing | 08:50 |
| shardy | marios: Yeah, if we get it into liberty and mitaka then it removes the problem for new deployments | 08:50 |
| marios | shardy: but it still istn' the right fix, because ultimately you will have to continue to set that always and forever amen | 08:50 |
| shardy | but anyone who already deployed them cannot update | 08:50 |
| jaosorior | ccamacho: For instance, I see a bunch of errors in the ironic log like this: Client-side error: Node 8cef9838-3298-4203-a133-de298c4b68d5 is locked by host localhost.localdomain, please retry after the current operation is completed. | 08:50 |
| shardy | I know given the preliminary state of upgrades upstream we've not made huge commitments re upgrades (yet), but I'm worried about RDO fallout | 08:51 |
| *** paramite|afk is now known as paramite | 08:51 | |
| shardy | so we probably need to at least get trown's input on it when he wakes up | 08:51 |
| shardy | also if we do backport it, we need to test the impact of changing this on an existing deployment via update | 08:51 |
| shardy | e.g it clearly breaks upgrades, but will we break all existing users if they do another overcloud deploy to e.g apply a config change? | 08:52 |
| ccamacho | jaosorior, I will try to find something locked.. I will let you know if I manage to find it | 08:52 |
| marios | shardy: ack.i need to still test the explicit setting workaround ... hopefully we can understand more about what cloud-init is doing and if we're using it wrong or it needs a fix. | 08:52 |
| ccamacho | jaosorior, using your clues BTW :) | 08:52 |
| shardy | marios: Yeah the name can be updated, but cloud-init won't run again: | 08:53 |
| shardy | http://docs.openstack.org/developer/heat/template_guide/openstack.html#OS::Nova::Server-prop-name | 08:53 |
| marios | shardy: thanks for ping these concerns make sense wrt upgrades (I thought well get it into liberty so np, but if you've already got a 'overcloud-novacompute' then upgrading to 'overcloud-copmute' is the same problem in reverse to https://bugzilla.redhat.com/show_bug.cgi?id=1324739 | 08:54 |
| openstack | bugzilla.redhat.com bug 1324739 in openstack-tripleo-heat-templates "Duplicate nova hypervisors show up in nova hypervisor-list post 7.3 -> 8 upgrade" [Urgent,On_dev] - Assigned to mandreou | 08:54 |
| shardy | marios: Yeah, that's my worry - and we do have RDO users of TripleO now to consider | 08:55 |
| marios | shardy: so do you think we first need to do a stack update to rename our compute nodes | 08:55 |
| marios | shardy: and then we can run the upgrade | 08:55 |
| shardy | marios: I don't think a stack update will rename them | 08:56 |
| shardy | but yeah we at least need to test that | 08:56 |
| marios | shardy: oh i thought that's what you meant - ok | 08:56 |
| shardy | marios: the stack update will *try* to update them, but I think we'll end up in an inconsistent, possibly broken state | 08:56 |
| shardy | because the OS::Nova::Server "name" will get updated, but then nothing will make cloud-init adjust the node hostname | 08:57 |
| marios | shardy: well, that is essentially what is happening here no? it is a stack update with the new templates, which hav ecompute hostname change | 08:57 |
| marios | shardy: and as you say, even though nova knows the new node as 'overcloud-novacompute' and it is set as such in /etc/hosts on all nodes | 08:57 |
| marios | shardy: cloud-init keeps setting /etc/hostname to 'overcluod-compute-0'. I had to explicitly disable cloud-init to get the 'new' name to stick on a reboot | 08:58 |
| shardy | marios: Yeah, it's picking up the old nova user/metadata I guess | 09:00 |
| shardy | I didn't realize it runs every boot tho, I thought preserve_hostname was the default | 09:00 |
| marios | shardy: we don't have that set, i had to set it | 09:00 |
| shardy | marios: ack, hmm, tricky - so whatever we do, changing the default is a problem unless we explicitly don't support folks with existing deployments from the upstream branches | 09:03 |
| shardy | Or we figure out a way to collect the new hostname from nova and make it stick | 09:03 |
| marios | shardy: yeah that's the real fix | 09:03 |
| *** julim has quit IRC | 09:04 | |
| *** bvandenh has joined #tripleo | 09:04 | |
| *** julim has joined #tripleo | 09:05 | |
| gfidente | so there is a bug with cloud-init here too, where it runs at every boot | 09:06 |
| gfidente | when it shouldn't | 09:06 |
| gfidente | I think marios pointed that out in the BZ | 09:06 |
| jistr | bandini: discovered we have openstack services still running after `pcs resource disable openstack-core` -- openstack-ceilometer-notification-clone, openstack-sahara-engine-clone, openstack-aodh-listener-clone -- should we hook those to openstack-core too? | 09:08 |
| jistr | specifically not having openstack-ceilomenter-notification-clone stopped seems to cause crm_resource --wait to hang forever when stopping openstack-core | 09:08 |
| bandini | jistr: could you send me a CIB of this system? Is this master (aka newton or mitaka)( | 09:09 |
| shardy | gfidente: Yeah, and despite running every boot it always uses the old locally stored nova metadata | 09:09 |
| jistr | yes, it's master | 09:09 |
| shardy | https://review.openstack.org/#/c/220057/1/firstboot/userdata_heat_admin.yaml | 09:09 |
| * bandini fighting with failed deployments :/ | 09:10 | |
| shardy | gfidente, marios: If we can do preserve_hostname I wonder if we can have another script which pulls the nova metadata (I think the hostname is reflected in the ec2 metadata?) and updates the hostname on boot if it changes | 09:10 |
| shardy | you still have to reboot the computes then tho | 09:10 |
| jistr | bandini: CIB http://chunk.io/f/64955f08cc644639bb8e2bb9fbc9549c | 09:11 |
| marios | shardy: yeah like a firstboot. we can even set/deliver/create that during the 'upgrades init' step - we already have a softwareconfig there that delivers the upgrade script to computes | 09:12 |
| shardy | marios: ah, could work then | 09:12 |
| * shardy looks to see if the hostname is updated by nova in the metadata | 09:13 | |
| marios | jistr: gfidente: o/ | 09:13 |
| marios | bandini: o/ | 09:13 |
| jaosorior | ccamacho: There also seems to have been a problem with swift | 09:13 |
| marios | o/ jaosorior ccamacho | 09:13 |
| * marios coffeee | 09:13 | |
| gfidente | ahahaa | 09:13 |
| gfidente | marios++ | 09:13 |
| jistr | bandini, rasca: the crm_resource --wait i discussed with rasca yesterday on rhos-pidone is caused by pending actions to start heat-engine and heat-api-cloudwatch after disabling openstack-core. I don't know why that is happening, i think it's a bug in pacemaker constraint resolution. But if i hook ceilometer-notification after openstack-core, so that it gets stopped too, the pending start actions for those heat services don't appear. | 09:13 |
| jaosorior | marios: Hey dude | 09:14 |
| ccamacho | marios++ | 09:14 |
| jistr | bandini, rasca: funny thing is that heat-api also depends on ceilometer-notification and openstack-core, but it doesn't have a pending start action -- that's why i think it's a bug in constraint propagation/resolution | 09:14 |
| jaosorior | ccamacho: I see | 09:14 |
| jaosorior | Apr 12 08:20:59 instack.localdomain glance-api[17675]: 2016-04-12 05:20:59.325 17747 ERROR swiftclient Traceback (most recent call last): | 09:14 |
| *** akrivoka has joined #tripleo | 09:14 | |
| jaosorior | Apr 12 08:20:59 instack.localdomain glance-api[17675]: 2016-04-12 05:20:59.325 17747 ERROR swiftclient File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 1565, in _retry | 09:14 |
| jaosorior | Apr 12 08:20:59 instack.localdomain glance-api[17675]: 2016-04-12 05:20:59.325 17747 ERROR swiftclient service_token=self.service_token, **kwargs) | 09:14 |
| jaosorior | Apr 12 08:20:59 instack.localdomain glance-api[17675]: 2016-04-12 05:20:59.325 17747 ERROR swiftclient File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 929, in head_container | 09:14 |
| jaosorior | Apr 12 08:20:59 instack.localdomain glance-api[17675]: 2016-04-12 05:20:59.325 17747 ERROR swiftclient http_response_content=body) | 09:14 |
| jaosorior | Apr 12 08:20:59 instack.localdomain glance-api[17675]: 2016-04-12 05:20:59.325 17747 ERROR swiftclient ClientException: Container HEAD failed: http://192.0.2.1:8080/v1/AUTH_cf8a3fb0684b4ee7b87595effb356530/glanc | 09:15 |
| jaosorior | Apr 12 08:20:59 instack.localdomain glance-api[17675]: 2016-04-12 05:20:59.325 17747 ERROR swiftclient | 09:15 |
| jaosorior | and after that a bunch of glance errors | 09:15 |
| jistr | marios: o/ | 09:15 |
| bandini | marios: o/ | 09:15 |
| jaosorior | I guess cause it tries to fetch that image | 09:15 |
| ccamacho | jaosorior, I see https://s-media-cache-ak0.pinimg.com/736x/f0/56/46/f056465cb8a29fab9e796f73a543d6a0.jpg | 09:16 |
| jaosorior | lol | 09:16 |
| jaosorior | true | 09:16 |
| jaosorior | I should brew some coffee too | 09:16 |
| jaosorior | now that I think about it | 09:16 |
| rasca | jistr, ok so the point is: stated that we have a bug but the developers themselves does not suggest to rely crm_resource, what is the best way to achieve the result we want? | 09:17 |
| *** openstackgerrit has quit IRC | 09:17 | |
| *** openstackgerrit has joined #tripleo | 09:17 | |
| *** coolsvap has quit IRC | 09:18 | |
| bandini | jistr: so openstack-ceilometer-notification-clone was never been dependant on keystone before, while openstack-sahara-engine-clone and openstack-aodh-listener-clone are new | 09:19 |
| rasca | bandini, openstack-ceilometer-notification-clone WAS dependent from keystone, as you can see from the schema in our gdoc | 09:21 |
| jistr | rasca: i think we were suggested to use crm_resource about 3/4 months ago, no? if the suggestion now changed to the opposite, i'm fine with that but i'd like to at least understand what's the issue. I think we have either broken constraints, or a bug in pacemaker, or both. I don't like that we're trying to be so quick to dismiss a solution that worked perfectly until now, without first properly investigating what is the problem in the first | 09:21 |
| jistr | place. | 09:21 |
| derekh | TripleO | CI cloud is down | CI status: http://tripleo.org/cistatus.html | Docs: http://tripleo.org/ | 09:23 |
| derekh | ^^^^ | 09:23 |
| rasca | jistr, I totally agree, even if the fact that it worked until now I don't think must be the only thing to consider while choosing to change something | 09:23 |
| bandini | rasca: do you have a link handy? that is not what I see from my docs | 09:23 |
| *** derekh changes topic to "TripleO | CI cloud is down | CI status: http://tripleo.org/cistatus.html | Docs: http://tripleo.org/" | 09:23 | |
| bandini | jistr: agreed we first need to get to the bottom of this issue fully | 09:24 |
| rasca | bandini, https://docs.google.com/document/d/1aXQ07CNazxt6xWbegIfYlP4VyZL3Dc9ZIkI8bRiB6rc/ if you look at the first schema | 09:24 |
| *** coolsvap has joined #tripleo | 09:24 | |
| jaosorior | derekh: Been trying to look into it | 09:25 |
| jistr | rasca: yeah agreed, my point wasn't just about "worked until now", it was more about "worked until now and we still have no idea why it stopped working" | 09:25 |
| bandini | rasca: it is not what is deployed by tripleo | 09:25 |
| jaosorior | derekh: But haven't figured out much... all I know is that ironic gets some error related to it not being able to do an operation because a node is locked. Then swift not being able to write an image (for some reason) and because of that, glance getting a bunch of resource-not-found's | 09:26 |
| derekh | jaosorior: I've just lost access to the cloud (in the last 10 minutes), trying to get back on now | 09:26 |
| jaosorior | oh, even that is down now? | 09:26 |
| jaosorior | damn | 09:26 |
| rasca | bandini, in osp8 there is start openstack-ceilometer-alarm-notifier-clone then start openstack-ceilometer-notification-clone | 09:26 |
| rasca | bandini, so IT IS dependent | 09:27 |
| rasca | bandini, maybe things changed while inserting openstack-core | 09:27 |
| derekh | jaosorior: yup, although if you want to keep looking into what ever the problem was befor this fire ahead | 09:27 |
| derekh | jaosorior: it might still be a problem when the cloud is back up | 09:27 |
| rasca | bandini, but that constraint make a dependency like the one I have in my docs | 09:27 |
| bandini | rasca: no likely that was before mitaka (http://acksyn.org/files/tripleo/wsgi-2016-02-24-cib.pdf is *pre* openstack-core) | 09:28 |
| jistr | bandini, rasca: re constraints on those openstack services -- i think the issue is that they're not really hooked to any dependencies whatsoever. E.g. i'd imagine they rely at least on RabbitMQ, but they're not hooked to that either. So if we stop Rabbit, ceilometer-notification will still keep running. | 09:28 |
| bandini | jistr: yes we need to tweak them for the full-pacemaker architecture for sure | 09:29 |
| * bandini mumbles something about lightweight arch | 09:29 | |
| jistr | hehe | 09:29 |
| *** sambetts|afk is now known as sambetts|afk] | 09:30 | |
| *** sambetts|afk] is now known as sambetts | 09:30 | |
| gfidente | bandini what do we do with cinder-volume in lightweight? | 09:31 |
| jistr | yeah i thought for full arch we aimed for "backend services -> openstack-core -> openstack services" type of dep, so it would seem logical to me to hook those 3 services after openstack-core | 09:31 |
| bandini | gfidente: that stays A/P (http://acksyn.org/files/tripleo/light-cib.pdf - note that I removed mongod as well from the last templates) | 09:33 |
| gfidente | bandini jistr that conversation abou migrations | 09:34 |
| bandini | jistr: agreed (http://acksyn.org/files/tripleo/newton-jistr-2016-04-12.pdf -> we need to hook up those three services) | 09:34 |
| bandini | gfidente: aye we need to get closure there too :/ | 09:34 |
| gfidente | I wonder if it isn't easy to just remove all the services/constraints and let puppet run with the updated manifest where it starts them as needed | 09:34 |
| jistr | hmm the question is when and where you let puppet run | 09:35 |
| gfidente | after we removed the constraints | 09:35 |
| gfidente | but this is to migrate to lightweight | 09:36 |
| jistr | and what about the other effects of puppet, like the example i keep bringing up -- from 7 to 8 a rabbitmq passwd change is very likely to happen | 09:36 |
| jistr | and if you run puppet only on controllers, it will break rabbitmq comms with other nodes | 09:36 |
| jistr | and if you run puppet everywhere, then you're running new templates on un-upgraded nodes | 09:37 |
| *** mgould has quit IRC | 09:37 | |
| jistr | the problem is that puppet is trying to do the full converge of the whole cloud, while we cannot let that happen just yet when some nodes haven't been upgraded yet, IIUC | 09:38 |
| gfidente | I want to let puppet do that indeed | 09:38 |
| jistr | so we could do that migration with puppet, but not with t-h-t manifest. It would have to be specifically crafted mini-manifest just for the migration, to avoid applying unwanted changes alongside the migration. | 09:38 |
| gfidente | I actually liked the yum_update approach | 09:39 |
| gfidente | go in maintenance, remove constraints or services as needed | 09:39 |
| gfidente | run yum update | 09:39 |
| gfidente | run puppet and let it bring back in known state | 09:39 |
| jistr | yeah, but IIRC there puppet ran also only after *all* nodes have finished the minor update | 09:40 |
| gfidente | yes | 09:40 |
| jistr | if we do it with t-h-t templates then it could be option 4 here https://etherpad.openstack.org/p/tripleo-migrations | 09:42 |
| *** mgould has joined #tripleo | 09:42 | |
| *** sshnaidm has joined #tripleo | 09:42 | |
| gfidente | jistr well 3 to me | 09:44 |
| jistr | gfidente: or if you are suggesting we do the migration in a minor update instead of major update, then it's option 1 in that etherpad | 09:44 |
| jistr | gfidente: well that doesn't work very well AFAICT, see line 67 in the etherpad | 09:44 |
| gfidente | you mean it doesn't work very because the rpms on the computes could be from the previous release when trying to run puppet? | 09:46 |
| aparnav | Hey, Can someone take a look at this patch https://review.openstack.org/#/c/295203/ ? | 09:46 |
| jistr | gfidente: yea. The new puppet modules and t-h-t might contain things specific to the new release. | 09:47 |
| *** apetrich has joined #tripleo | 09:47 | |
| jistr | i don't think we can guarantee that we can safely run mitaka tht&puppet on top of liberty RPMs | 09:47 |
| jistr | and also, given our current distribution model of puppet modules | 09:48 |
| gfidente | but we can update rpms on all nodes though | 09:48 |
| gfidente | like yum_update did | 09:48 |
| jistr | in theory we could but we got an explicit requirement not to tamper with all computes at once | 09:49 |
| jistr | i didn't ask why, but i assume the reason were concerns that if something goes wrong during a mass-upgrade of computes, we might in theory bring down the whole cloud incl. all running workloads | 09:51 |
| *** shardy has quit IRC | 09:52 | |
| jistr | gfidente: btw even yum_update didn't just remove things and wait for puppet to add the new ones. yum_update also added the new constraints too because we had to keep a good cluster state between yum_update.sh and puppet too | 09:53 |
| *** openstackstatus has quit IRC | 09:57 | |
| *** openstack has joined #tripleo | 09:59 | |
| *** openstackstatus has joined #tripleo | 10:00 | |
| *** ChanServ sets mode: +v openstackstatus | 10:00 | |
| *** shardy has joined #tripleo | 10:02 | |
| *** jcoufal has quit IRC | 10:03 | |
| *** miles has joined #tripleo | 10:03 | |
| *** miles is now known as mgould_ | 10:04 | |
| *** mgould has quit IRC | 10:04 | |
| *** bvandenh has quit IRC | 10:05 | |
| openstackgerrit | Merged openstack/tripleo-quickstart: Add libselinux-python to install_deps https://review.openstack.org/304167 | 10:06 |
| *** ccamacho is now known as ccamacho|lunch | 10:21 | |
| *** jaosorior has left #tripleo | 10:21 | |
| *** jaosorior has quit IRC | 10:21 | |
| *** jaosorior has joined #tripleo | 10:22 | |
| openstackgerrit | Imre Farkas proposed openstack/tripleo-docs: Document ready-state configuration https://review.openstack.org/299481 | 10:29 |
| openstackgerrit | Merged openstack/tripleo-ui: Update license string to use SPDX Identifier https://review.openstack.org/301849 | 10:30 |
| *** ccamacho|lunch is now known as ccamacho | 10:38 | |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui: Move Environment and Parameters config to single modal https://review.openstack.org/302272 | 10:45 |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui: Move Validations to right sidebar https://review.openstack.org/304127 | 10:45 |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui: Deployment Plan page updates https://review.openstack.org/303958 | 10:45 |
| *** rook has quit IRC | 10:58 | |
| *** lblanchard has joined #tripleo | 11:03 | |
| *** jcoufal has joined #tripleo | 11:07 | |
| *** ramishra has joined #tripleo | 11:11 | |
| *** ramishra_ has joined #tripleo | 11:14 | |
| *** thrash|g0ne is now known as thrash | 11:16 | |
| *** andrearosa has joined #tripleo | 11:18 | |
| *** ramishra_ has quit IRC | 11:22 | |
| *** rook has joined #tripleo | 11:22 | |
| *** ramishra_ has joined #tripleo | 11:22 | |
| *** ramishra_ has quit IRC | 11:23 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart: add ipv4 network-isolation to quickstart for virt deployments https://review.openstack.org/303030 | 11:25 |
| *** mgould_ has quit IRC | 11:25 | |
| *** ramishra_ has joined #tripleo | 11:27 | |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates: Add missing ManagementIpSubnet https://review.openstack.org/301266 | 11:32 |
| *** thrash has quit IRC | 11:32 | |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates: Add net-config-static.yaml https://review.openstack.org/301267 | 11:32 |
| jaosorior | slagle: CI is broken :/ | 11:32 |
| openstackgerrit | James Slagle proposed openstack/os-net-config: Add support for OVS tunnels https://review.openstack.org/304215 | 11:33 |
| jaosorior | so, no use rechecking ATM | 11:33 |
| *** mcornea_ has joined #tripleo | 11:33 | |
| openstackgerrit | James Slagle proposed openstack/os-net-config: Fix typos https://review.openstack.org/304219 | 11:34 |
| slagle | jaosorior: yea, i know that | 11:34 |
| slagle | was rebasing | 11:34 |
| jaosorior | alright | 11:34 |
| jaosorior | didn't see your nick in this channel, that's why I repeated on the other one | 11:35 |
| *** mcornea has quit IRC | 11:35 | |
| *** thrash has joined #tripleo | 11:38 | |
| *** thrash has joined #tripleo | 11:38 | |
| *** rhallisey has joined #tripleo | 11:39 | |
| *** panda has quit IRC | 11:41 | |
| *** panda has joined #tripleo | 11:42 | |
| *** MaxPC has joined #tripleo | 11:45 | |
| *** Goneri has quit IRC | 11:49 | |
| derekh | slagle: bnemec rh1 down, brought it back up, instances now not getting ipaddresses | 11:49 |
| *** trozet has quit IRC | 11:56 | |
| bandini | dtantsur: ever seen this? http://fpaste.org/354619/60462136/ I get this during introspection on a mitaka BM env. I am moderately sure it worked like a month ago (aka last time I tried) | 11:56 |
| bandini | I will look into it more, but maybe it rings a bell | 11:57 |
| dtantsur | bandini, looks like https://bugzilla.redhat.com/show_bug.cgi?id=1322892 | 11:58 |
| openstack | bugzilla.redhat.com bug 1322892 in openstack-ironic-discoverd "No valid interfaces found during introspection" [Unspecified,Modified] - Assigned to dtantsur | 12:04 |
| *** dmacpher has quit IRC | 12:04 | |
| *** dprince has joined #tripleo | 12:04 | |
| *** dmacpher has joined #tripleo | 12:04 | |
| dtantsur | unfortunately, the fix is not backportable... | 12:04 |
| bandini | dtantsur: yep that's the one, let me look at it more in detail after some coffee ;) | 12:04 |
| dtantsur | bandini, tl;dr: there is a fix in newton, but it can't be directly backported for mitaka | 12:04 |
| *** andrearosa has left #tripleo | 12:04 | |
| *** openstack has quit IRC | 12:04 | |
| *** openstack has joined #tripleo | 12:05 | |
| dtantsur | or do we have like builds of images from master? | 12:06 |
| trown | bandini: ya I am contemplating just building IPA from master for all RDO releases until there is some issue that breaks | 12:06 |
| EmilienM | trown: details? anything related to puppet? we merged lot of stuff monday | 12:06 |
| trown | dtantsur: not yet, working on that today | 12:06 |
| EmilienM | (in puppet modules) | 12:06 |
| trown | EmilienM: check topic, CI cloud is down | 12:06 |
| dtantsur | trown, great! then we can allow people to choose even: use IPA stable or IPA master | 12:06 |
| dtantsur | your tripleo-quickstart could have an option for that | 12:06 |
| *** mgould has joined #tripleo | 12:07 | |
| trown | dtantsur: suppose so, the option would have to download the non-default image, but it is doable | 12:07 |
| sshnaidm | trown, hi | 12:11 |
| trown | sshnaidm: hi :) | 12:11 |
| sshnaidm | trown, do you know ways to connect overcloud directly without creating 3 tunnels? I mean auth_url or even vms on it | 12:12 |
| *** ramishra_ has quit IRC | 12:12 | |
| trown | sshnaidm: larsks wrote some docs for tripleo-quickstart on that https://github.com/openstack/tripleo-quickstart/blob/master/docs/accessing-overcloud.md | 12:12 |
| *** jayg|g0n3 is now known as jayg | 12:13 | |
| trown | they should be mostly relevant to other environments | 12:13 |
| *** lblanchard has joined #tripleo | 12:13 | |
| sshnaidm | trown, great, thanks | 12:13 |
| *** ramishra_ has joined #tripleo | 12:14 | |
| *** ramishra_ has quit IRC | 12:14 | |
| openstackgerrit | Carlos Camacho proposed openstack-infra/tripleo-ci: Removing previously created resourses in pingtest https://review.openstack.org/304560 | 12:17 |
| *** lblanchard has quit IRC | 12:18 | |
| *** ramishra_ has joined #tripleo | 12:19 | |
| derekh | dprince: slagle: bnemec rh1 down, brought it back up, instances now not getting ipaddresses, if anybody has any idea jump in and poke around | 12:20 |
| dprince | derekh: ack, will look in a bit | 12:21 |
| jaosorior | derekh: I have no clue honestly. But does this have anything to do? https://review.openstack.org/#/c/303850/ | 12:21 |
| derekh | jaosorior: it shouldn't, thats the config of how nodpool talks to our cloud, at the moment we're not evening getting that far | 12:23 |
| jaosorior | oh crap, alright :/ | 12:23 |
| openstackgerrit | David Sariel proposed openstack/tripleo-heat-templates: Enable cinder-backup service start https://review.openstack.org/304563 | 12:24 |
| *** rodrigods has quit IRC | 12:24 | |
| *** rodrigods has joined #tripleo | 12:24 | |
| *** aufi has quit IRC | 12:25 | |
| *** coolsvap is now known as coolsvap|away | 12:25 | |
| *** ramishra has quit IRC | 12:26 | |
| trown | shardy: mind putting a PTL stamp of approval on https://review.openstack.org/#/c/304145/ | 12:29 |
| *** ramishra_ has quit IRC | 12:30 | |
| *** jpena is now known as jpena|lunch | 12:30 | |
| bandini | trown: can you cc me on the quickstart IPA change? | 12:32 |
| openstackgerrit | Carlos Camacho proposed openstack-infra/tripleo-ci: Removing previously created resourses in pingtest https://review.openstack.org/304560 | 12:32 |
| *** rbrady has joined #tripleo | 12:33 | |
| *** ramishra has joined #tripleo | 12:33 | |
| trown | bandini: actually would you mind filing a wishlist bug for it? https://bugs.launchpad.net/tripleo-quickstart | 12:33 |
| trown | I need to sync over all the open issues from the redhat-openstack github still | 12:34 |
| bandini | trown: ack sure | 12:34 |
| *** ramishra_ has joined #tripleo | 12:35 | |
| *** saneax is now known as saneax_AFK | 12:35 | |
| shardy | trown: done | 12:37 |
| trown | shardy: thanks! | 12:37 |
| *** ramishra_ has quit IRC | 12:38 | |
| *** morazi has quit IRC | 12:39 | |
| bandini | trown: https://bugs.launchpad.net/tripleo-quickstart/+bug/1569322 | 12:40 |
| openstack | Launchpad bug 1569322 in tripleo-quickstart "RFE: support IPA ramdisk images from master" [Undecided,New] | 12:40 |
| *** ramishra_ has joined #tripleo | 12:40 | |
| trown | bandini: thanks! | 12:41 |
| *** pradk has joined #tripleo | 12:50 | |
| *** fragatina has joined #tripleo | 12:51 | |
| *** links has quit IRC | 12:52 | |
| *** liverpooler has quit IRC | 12:55 | |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-heat-templates: Make sure openstack services are dependant on openstack-core https://review.openstack.org/304592 | 12:56 |
| *** jdob has quit IRC | 12:57 | |
| *** jdob has joined #tripleo | 12:58 | |
| jistr | bandini, rasca: ^ the constraint additions -- i hooked things up the way that what made most sense to me, and it did unblock the services restart, but i'm not sure if we want some changes there. So i marked it WIP for now to prevent merging too early. (Once we agree on the constraint changes, i'll add a .sh counterpart to those changes to have something to migrate existing deployments too) | 13:00 |
| *** eggmaster has joined #tripleo | 13:00 | |
| *** jprovazn has quit IRC | 13:01 | |
| bandini | jistr: would you have a cib around with your change included? | 13:02 |
| jistr | bandini: ah yea, forgot to paste it http://chunk.io/f/4febafdcb9cf4ea58ca958b1a47d4def | 13:02 |
| jistr | i knew you'd like to view it in PDF :D | 13:02 |
| jistr | there might be a superfluous openstack-core -> heat-api constraint now | 13:03 |
| *** Goneri has joined #tripleo | 13:03 | |
| bandini | jistr: my brain is too small to read constraints from the terminal ;) | 13:03 |
| jistr | due to openstack-core -> ceilometer-notification -> heat api ordering | 13:03 |
| *** ramishra_ has quit IRC | 13:03 | |
| *** yamahata has joined #tripleo | 13:04 | |
| *** jdob has quit IRC | 13:05 | |
| *** jdob has joined #tripleo | 13:05 | |
| openstackgerrit | Merged openstack/os-net-config: Fix typos https://review.openstack.org/304219 | 13:09 |
| *** ramishra_ has joined #tripleo | 13:09 | |
| dprince | derekh: is there an etherpad that shows what you've done so far? | 13:10 |
| dprince | derekh: for starters: could we reboot the controller, and re-create the computes from scratch? | 13:11 |
| *** dtantsur is now known as dtantsur|mtg | 13:11 | |
| *** qasims has quit IRC | 13:12 | |
| *** ramishra_ has quit IRC | 13:12 | |
| *** ayoung has quit IRC | 13:12 | |
| derekh | dprince: we were being arp flooded again, I've rebooted the controller and compute nodes, basically following the same steps at the last time https://etherpad.openstack.org/p/snqXyvqubi | 13:12 |
| *** ramishra_ has joined #tripleo | 13:14 | |
| *** tzumainn has joined #tripleo | 13:15 | |
| *** tzumainn has quit IRC | 13:15 | |
| derekh | dprince: although I havn't rebuild the compute nodes, just rebooted | 13:15 |
| *** tzumainn has joined #tripleo | 13:15 | |
| bandini | jistr: agreed, I'd remove the openstack-core -> openstack-heat-api as you mentioned | 13:15 |
| *** ramishra_ has quit IRC | 13:16 | |
| dprince | derekh: yeah. for some reason I recall rebuilding them all | 13:17 |
| dprince | derekh: as being the fix for the ARP flood issue | 13:17 |
| dprince | derekh: just the computes | 13:17 |
| derekh | dprince: iirc, yup we did, the reboot seems to have stoped the flooding, but obvously something is still wrong | 13:18 |
| derekh | dprince: we can rebuild and see what happens | 13:18 |
| dprince | derekh: so you say you get IPs, but they don't work? | 13:18 |
| derekh | dprince: if the problem isn't obvious | 13:18 |
| derekh | dprince: the instances are booting but not getting an IP from dhcp | 13:19 |
| *** ramishra_ has joined #tripleo | 13:20 | |
| *** aparnav has quit IRC | 13:21 | |
| *** tiswanso has joined #tripleo | 13:22 | |
| *** lblanchard has joined #tripleo | 13:25 | |
| *** ramishra_ has quit IRC | 13:25 | |
| *** ramishra_ has joined #tripleo | 13:26 | |
| dprince | derekh: can I clean out all of the shutoff instances from nodepool? | 13:28 |
| *** morazi has joined #tripleo | 13:28 | |
| derekh | dprince: I got no issue with it, as long as nodepool will be ok with it | 13:29 |
| dprince | derekh: it will recover | 13:29 |
| derekh | dprince: ack | 13:29 |
| dprince | derekh: unlikely they would come back up anyways... especially if we resort to rebuilding the computes | 13:29 |
| derekh | dprince: yup | 13:29 |
| openstackgerrit | Ryan Hallisey proposed openstack/tripleo-heat-templates: Composable Keystone Containers https://review.openstack.org/304282 | 13:30 |
| openstackgerrit | Ryan Hallisey proposed openstack/tripleo-heat-templates: composable neutron dhcp service https://review.openstack.org/303386 | 13:30 |
| openstackgerrit | Ryan Hallisey proposed openstack/tripleo-heat-templates: composable neutron metadata service https://review.openstack.org/303618 | 13:30 |
| openstackgerrit | Ryan Hallisey proposed openstack/tripleo-heat-templates: composable neutron l3 service https://review.openstack.org/303562 | 13:30 |
| openstackgerrit | Ryan Hallisey proposed openstack/tripleo-heat-templates: composable glance services https://review.openstack.org/237370 | 13:30 |
| openstackgerrit | Ryan Hallisey proposed openstack/tripleo-heat-templates: Add GlanceRegistry to the endpoint map https://review.openstack.org/303728 | 13:30 |
| *** trozet has joined #tripleo | 13:31 | |
| *** jpena|lunch is now known as jpena | 13:36 | |
| *** links has joined #tripleo | 13:36 | |
| *** ramishra_ has quit IRC | 13:37 | |
| derekh | dprince: my latest attempt got an IP, I restart nova-compute and neutron-ovs-agent on the compute, not sure if it made a difference | 13:42 |
| derekh | dprince: did you cahnge anything ? | 13:42 |
| derekh | dprince: floating ip works also, gonna restart them on all compute node and see if it helps | 13:43 |
| dprince | derekh: I deleted all the nodepool instacnces | 13:43 |
| dprince | derekh: and then restarted the neutron DHCP and OVS processes on the controller | 13:43 |
| dprince | derekh: just a hunch | 13:43 |
| *** ramishra_ has joined #tripleo | 13:43 | |
| derekh | dprince: I've also restarted them a few times but maybe you tickled something I didn't | 13:44 |
| derekh | dprince: floating ip works also, gonna restart them on all compute node and see if it helps | 13:45 |
| derekh | 13:45 | |
| derekh | 13:45 | |
| derekh | 13:45 | |
| derekh | 13:45 | |
| derekh | 13:45 | |
| dprince | derekh: cool | 13:45 |
| dprince | derekh: we might go on and clean out the floatingips' too | 13:45 |
| dprince | derekh: just to clean house... nodepool should fix itself | 13:46 |
| *** dtrainor has joined #tripleo | 13:46 | |
| derekh | dprince: yup | 13:46 |
| *** akshai has joined #tripleo | 13:48 | |
| *** egafford has joined #tripleo | 13:48 | |
| dprince | derekh: looks like only 2 are assigned. I'll delete the rest of them | 13:49 |
| derekh | ok | 13:49 |
| dprince | derekh: I'm slighly concerned one of the computes is still broken. and if something got scheduled on it it might hose us again | 13:49 |
| *** ramishra_ has quit IRC | 13:49 | |
| dprince | derekh: if that happens I suppose we'll find out soon enough | 13:50 |
| derekh | dprince: not sure what you mean? | 13:50 |
| dprince | derekh: just that the ARP floodding we resume once an instance gets spawned on one of (the broken) compute nodes | 13:50 |
| *** ramishra_ has joined #tripleo | 13:50 | |
| *** dtrainor has quit IRC | 13:50 | |
| *** dtrainor has joined #tripleo | 13:51 | |
| derekh | dprince: which broken compute node? | 13:51 |
| *** sanjay__u has joined #tripleo | 13:51 | |
| dprince | derekh: I don't know which one :) | 13:51 |
| derekh | dprince: ahhh, you think one might be broken | 13:52 |
| dprince | derekh: or how to check it even. Just that there might be one of them that caused this... and as soon as it gets an instance spawned on it we'd be back where we started | 13:52 |
| dprince | derekh: okay, floatingip's cleaned up | 13:52 |
| dprince | derekh: my instance was pinging fine | 13:52 |
| openstackgerrit | Carlos Camacho proposed openstack-infra/tripleo-ci: Removing previously created resources in pingtest https://review.openstack.org/304560 | 13:53 |
| dprince | derekh: if connectivity is working for you too we might try opening up to nodepool again... | 13:53 |
| derekh | dprince: wait | 13:53 |
| dprince | derekh: I won't touch that. I was going to let you kick the tires... | 13:54 |
| derekh | dprince: rebooting the proxy, gearman and mirror servers so see if they get an IP tis time | 13:54 |
| dprince | derekh: yep, gotcha | 13:54 |
| *** dustins has joined #tripleo | 13:54 | |
| derekh | dprince: once their confirmed and testenvs are gonnecting to gearman we can open back up to nodepool | 13:54 |
| derekh | *connecting | 13:54 |
| dprince | derekh: I think they will recover | 13:55 |
| dprince | once you reboot them | 13:55 |
| *** ramishra_ has quit IRC | 13:56 | |
| derekh | dprince: mirror server up, had to start httpd | 13:56 |
| *** ramishra_ has joined #tripleo | 13:56 | |
| derekh | dprince: the other 2 not up yet | 13:57 |
| shardy | Hey folks, the meeting starts in 2 mins in #openstack-meeting-alt | 13:58 |
| slagle | there's a new sheriff in town | 14:00 |
| *** jhesketh has left #tripleo | 14:00 | |
| derekh | dprince: those 3 servers are back now, will open up the iptables rule in a couple of minutes once I make sure testenvs are registering ok | 14:05 |
| *** ramishra_ has quit IRC | 14:06 | |
| dprince | derekh: sounds good | 14:06 |
| *** Ryjedo has joined #tripleo | 14:07 | |
| *** ramishra_ has joined #tripleo | 14:07 | |
| *** mcornea_ has quit IRC | 14:07 | |
| *** links has quit IRC | 14:08 | |
| *** ramishra_ has quit IRC | 14:11 | |
| *** ramishra_ has joined #tripleo | 14:12 | |
| *** ramishra_ has quit IRC | 14:15 | |
| openstackgerrit | Carlos Camacho proposed openstack-infra/tripleo-ci: Removing previously created resources in pingtest and --skip-pingtest-cleanup option https://review.openstack.org/304560 | 14:16 |
| *** ramishra_ has joined #tripleo | 14:17 | |
| derekh | dprince: the testenvs dont appear to be connecting to geard, gonna rebuild them now | 14:17 |
| openstackgerrit | Carlos Camacho proposed openstack-infra/tripleo-ci: Removing previously created resources in pingtest and --skip-pingtest-cleanup option https://review.openstack.org/304560 | 14:17 |
| *** aufi has joined #tripleo | 14:18 | |
| openstackgerrit | Florian Fuchs proposed openstack/tripleo-ui: Adds a progress status for the current deployment https://review.openstack.org/303436 | 14:18 |
| dprince | derekh: :/. maybe they lost a port in there somewhere? | 14:18 |
| dprince | derekh: might be worth cleaning up all the ports if we are rebuilding those too.... | 14:19 |
| dprince | derekh: ports on the 192. network that is | 14:19 |
| derekh | dprince: perhapes, will be doing that too | 14:19 |
| dprince | derekh: cool | 14:19 |
| *** mcornea_ has joined #tripleo | 14:22 | |
| *** liverpooler has joined #tripleo | 14:25 | |
| *** ramishra_ has quit IRC | 14:28 | |
| *** ramishra_ has joined #tripleo | 14:28 | |
| gfidente | michchap dprince hey I was following your steps with the puppet restructuring for ceph | 14:33 |
| gfidente | I think have the puppet split ready | 14:33 |
| gfidente | though I figured I want to reuse ceph_osd profile for the ceph storage node | 14:34 |
| gfidente | except it has understanding of 'steps' which we don't use on non-controller nodes | 14:34 |
| gfidente | have you faced something like this before and have ideas? | 14:34 |
| michchap | gfidente: oh neat I was just talking to someone else on the opnfv team about doing the ceph profiles, since we have another change we wanted to get merged that would depend onit | 14:34 |
| gfidente | oh I can submit those | 14:35 |
| michchap | there's a bunch of options ranging from really hacky to mildly hacky | 14:35 |
| openstackgerrit | Giulio Fidente proposed openstack/puppet-tripleo: Add ceph profiles https://review.openstack.org/304675 | 14:35 |
| *** radez has joined #tripleo | 14:35 | |
| gfidente | I have the tht change ready too, except for the cephstorage nodes | 14:36 |
| michchap | gfidente: radez is driving it from the opnfv side | 14:36 |
| gfidente | because of the issue with step | 14:36 |
| radez | been working on the hyper converged stuff | 14:37 |
| radez | https://review.openstack.org/#/c/273754 | 14:37 |
| gfidente | radez ack, I'm getting there | 14:37 |
| *** liverpooler has quit IRC | 14:37 | |
| radez | but dprince suggested not merging that since the composable stuff is coming | 14:37 |
| gfidente | I'm working on the puppet manifests split first | 14:37 |
| gfidente | yep I read that and commented | 14:37 |
| radez | ah, gotcha, hadn't seen your comment yet | 14:38 |
| michchap | gfidente: as far as steps, the issue is the OSD fails when it starts before the MON right? | 14:38 |
| gfidente | we do have control over that in heat | 14:38 |
| michchap | gfidente: and without step on non-controllers it's difficult to do that | 14:38 |
| gfidente | the problem is the osd profile needs 'steps' to be successfully applied on a controller | 14:39 |
| gfidente | but we don't have 'steps' on the cephstorage nodes | 14:39 |
| michchap | is there a flag that indicates that a node is a controller? | 14:39 |
| michchap | if (!hiera('is_controller') or $step >= N) | 14:40 |
| *** rbrady has quit IRC | 14:41 | |
| *** rbrady has joined #tripleo | 14:41 | |
| *** rbrady has quit IRC | 14:41 | |
| gfidente | I was thinking about checking if step is undef | 14:42 |
| gfidente | but I wanted dprince feedback here too on the steps story on non-controllers | 14:43 |
| gfidente | jistr marios ^^ | 14:43 |
| michchap | is puppet only run a single time on non-controllers? | 14:43 |
| gfidente | currently yes | 14:44 |
| dprince | gfidente: we will use steps on all the roles | 14:44 |
| jistr | currently yes | 14:44 |
| dprince | gfidente: that is the way things truely become composable | 14:44 |
| gfidente | dprince right | 14:44 |
| dprince | gfidente: We are de-composing the controller first | 14:44 |
| gfidente | and steps will be identical on all role-types then | 14:44 |
| dprince | gfidente: once that is done we can move towards making the other roles support adding in the functionality | 14:44 |
| dprince | gfidente: yes, that is the plan | 14:44 |
| gfidente | ok I see it now, thanks | 14:45 |
| dprince | gfidente: we may in fact be able to share the same base template or something. I haven't modelled that yet. | 14:45 |
| *** rbrady has joined #tripleo | 14:46 | |
| *** ramishra_ has quit IRC | 14:47 | |
| *** apetrich has quit IRC | 14:47 | |
| *** tiswanso has quit IRC | 14:48 | |
| *** ramishra_ has joined #tripleo | 14:48 | |
| derekh | dprince: ok, gonna open up the flood gates | 14:48 |
| *** tiswanso has joined #tripleo | 14:48 | |
| gfidente | derekh++ | 14:49 |
| *** liverpooler has joined #tripleo | 14:50 | |
| *** apetrich has joined #tripleo | 14:52 | |
| *** apetrich has quit IRC | 14:53 | |
| *** ramishra_ has quit IRC | 14:53 | |
| *** apetrich has joined #tripleo | 14:53 | |
| *** paramite is now known as paramite|afk | 14:54 | |
| *** florianf has quit IRC | 14:55 | |
| *** dustins has quit IRC | 14:56 | |
| openstackgerrit | Giulio Fidente proposed openstack/puppet-tripleo: Add ceph profiles https://review.openstack.org/304675 | 14:57 |
| *** aufi has quit IRC | 14:58 | |
| *** derekh changes topic to "TripleO | CI cloud is back, currently processing the backlog | CI status: http://tripleo.org/cistatus.html | Docs: http://tripleo.org/" | 14:58 | |
| *** jprovazn has joined #tripleo | 15:01 | |
| *** bvandenh has joined #tripleo | 15:01 | |
| *** dustins has joined #tripleo | 15:01 | |
| beagles | shardy: was a bit to slow at the end of the meeting typing this in... I ran into a snag last week because I had to recreate my virt environment and for some reason went with the online docs, not the tripleo script | 15:02 |
| beagles | shardy, I ran into what looks like puppet-concat related issues, this one in the puppet-swift modules. | 15:02 |
| shardy | beagles: Yeah, that's something we need to address - either via everyone (including CI) using tripleo-quickstart | 15:02 |
| beagles | shardy, k... | 15:02 |
| shardy | or by generating the docs from the CI scriptt (or vice-versa) | 15:02 |
| beagles | shardy, that's what I wanted to get and idea of - what is the preferred approach to resolving | 15:03 |
| beagles | shardy, or rather, whether we want to implement the docs (docs take precedence) or document what we actually prefer to do ;) | 15:04 |
| *** ramishra_ has joined #tripleo | 15:04 | |
| *** coolsvap|away is now known as coolsvap | 15:05 | |
| beagles | shardy, the other thing that was interesting was the reason I had to do this in the first place.. my undercloud VM's filesystem got corrupted. I'm not sure what the root cause was, but I suspect the cache settings for the VM when it was created | 15:05 |
| *** ramishra_ has quit IRC | 15:05 | |
| beagles | shardy, just a heads up that this might be something that needs attention. I'm looking into whether it is a plausible cause | 15:06 |
| beagles | dprince, regarding composable patches - I presume you welcome review feedback? ;) | 15:07 |
| beagles | dprince, or is it early days at the moment? I'm referring specifically to neutron-ish patches | 15:07 |
| dprince | beagles: sure, jump in and comment on those reviews... | 15:08 |
| beagles | dprince, awesome | 15:08 |
| *** afazekas has quit IRC | 15:09 | |
| *** ramishra_ has joined #tripleo | 15:09 | |
| *** afazekas has joined #tripleo | 15:09 | |
| *** florianf has joined #tripleo | 15:10 | |
| openstackgerrit | Jason Dunsmore proposed openstack/os-collect-config: Convert collectors option to a ListOpt https://review.openstack.org/304687 | 15:13 |
| openstackgerrit | Giulio Fidente proposed openstack/puppet-tripleo: Add ceph profiles https://review.openstack.org/304675 | 15:13 |
| openstackgerrit | Jason Dunsmore proposed openstack/os-collect-config: Convert collectors option to a ListOpt https://review.openstack.org/304687 | 15:13 |
| *** fpan has joined #tripleo | 15:15 | |
| openstackgerrit | Giulio Fidente proposed openstack/puppet-tripleo: Add ceph profiles https://review.openstack.org/304675 | 15:17 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Use ceph profiles https://review.openstack.org/304692 | 15:17 |
| *** openstackgerrit has quit IRC | 15:18 | |
| *** openstackgerrit has joined #tripleo | 15:18 | |
| *** yamahata has quit IRC | 15:19 | |
| *** ayoung has joined #tripleo | 15:20 | |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Use ceph profiles https://review.openstack.org/304692 | 15:21 |
| gfidente | michchap it misses the hiera call because I'm passing it as class param from https://review.openstack.org/304692 | 15:23 |
| gfidente | so it's a bit of a cleanup | 15:24 |
| *** jaosorior has quit IRC | 15:24 | |
| michchap | gfidente: oh nice | 15:25 |
| michchap | gfidente: in that case, it probably needs to be the first param - usually required params go above optional. | 15:25 |
| gfidente | michchap ok | 15:25 |
| *** paramite|afk has quit IRC | 15:25 | |
| michchap | gfidente: I can't remember if the linter will complain. Probably. | 15:25 |
| *** oshvartz has quit IRC | 15:26 | |
| *** ramishra_ has quit IRC | 15:26 | |
| *** ramishra_ has joined #tripleo | 15:32 | |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-heat-templates: Make sure openstack services are dependant on openstack-core https://review.openstack.org/304592 | 15:35 |
| gfidente | jistr so the .sh counterpart | 15:35 |
| *** ramishra_ has quit IRC | 15:35 | |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: Actually test stack-update to new tht https://review.openstack.org/304707 | 15:36 |
| gfidente | I think the .sh without the constraints should go to mitaka | 15:36 |
| slagle | dprince: i think that is all that is needed to test stack-update's actually ^^ | 15:36 |
| dprince | slagle: nice, lets see what happens :) | 15:37 |
| gfidente | slagle that is to test the actual tht submission on update right? | 15:37 |
| slagle | gfidente: yes | 15:37 |
| shardy | dprince: Hey, was wondering on your thoughts re container integration with the new ResourceChain composition model | 15:37 |
| gfidente | slagle++ | 15:37 |
| rhallisey | shardy dprince how would the use specifcy what service will land on the controller | 15:37 |
| jistr | gfidente: yea, i'm not exactly sure when the restart stopped working, but i think the constraint fix should probably happen with the migration to openstack-core & keystone WSGI, so yeah that means a mitaka backport | 15:37 |
| rhallisey | s/use/user | 15:38 |
| shardy | rhallisey is looking into it, and it seems the needed flow is somewhat backwards to the current model | 15:38 |
| dprince | rhallisey: there is a parameter | 15:38 |
| jistr | gfidente: or actually... depends how we'll decide to do the migrations | 15:38 |
| dprince | rhallisey: ControllerServices | 15:38 |
| dprince | rhallisey: I expect we would add this parameter for all roles | 15:38 |
| shardy | e.g atm we have ControllerServiceChain assemble all the configs which are passed into ControllerNodesPostDeployment | 15:38 |
| openstackgerrit | Ben Nemec proposed openstack/os-net-config: Normalize operstate value for interfaces https://review.openstack.org/304712 | 15:38 |
| gfidente | jistr yeah ... I was trying to split submissions in a way which allows us to backport only those relevant to mitaka | 15:38 |
| gfidente | building on top those which will cope instead with gnocchi | 15:38 |
| shardy | dprince: I was thinking it'd be good if e.g docker/services/keystone.yaml actually deployed the service vs just creating the config | 15:39 |
| dprince | rhallisey: once we decompose the controller I expect we can probably refactor all the roles to use a shared -post.yaml template | 15:39 |
| jistr | gfidente: anyway, i'll add it to pacemaker_migrations.sh and i'll make it idempotent, then we can call the function from wherever we want (even the restart.sh if need be) | 15:39 |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: Actually test stack-update to new tht https://review.openstack.org/304707 | 15:39 |
| rhallisey | dprince, right it's listed then, but do users edit overcloud.yaml? | 15:39 |
| shardy | because that isolation is one of the advantages vs the puppet model | 15:39 |
| dprince | shardy: that is the idea | 15:39 |
| jistr | gfidente: i'm now working on reproducing the issue to gather sosreports for bandini, after that i'll do the .sh | 15:39 |
| dprince | shardy: oh, wait. Are you talking about passing OS::Nova::Server in there again? | 15:39 |
| gfidente | jistr sure I was worrying which of keystone/wsgi, aodh, gnocchi land in mitaka and which in newton | 15:40 |
| shardy | dprince: Yes I'm wonderingif we have to | 15:40 |
| rhallisey | so the interface is OS::TripleO::Services::Keystone -> I want keystone | 15:40 |
| shardy | e.g for the docker case | 15:40 |
| dprince | shardy: passing in the resource group would be okay, perhaps | 15:40 |
| jistr | not sure if gnocchi, but aodh and keystone/wsgi land in mitaka | 15:40 |
| gfidente | jistr to have a better understanding of what the migration scripts are meant to deal with | 15:40 |
| dprince | shardy: I actually don't think we have to do it that way though | 15:40 |
| gfidente | ;( | 15:40 |
| dprince | shardy: if we have a better interface | 15:40 |
| shardy | dprince: Ok, I guess I'm not clear how we get from the ResourceChain to deploying all the containers | 15:41 |
| dprince | shardy: I had expected we'd tackle this after decomposing the controller so I haven't prototyped it entirely | 15:41 |
| rhallisey | shardy, right now I run the existing roles then do containers in post | 15:42 |
| shardy | dprince: Yup, cool - I think rhallisey is trying to prototype it now hence throwing some ideas around :) | 15:42 |
| dprince | shardy: but I do have a pretty good idea | 15:42 |
| *** tiswanso has quit IRC | 15:42 | |
| dprince | shardy: yep, I'm aware of it | 15:42 |
| *** dustins has quit IRC | 15:42 | |
| *** tiswanso has joined #tripleo | 15:43 | |
| *** mikelk has quit IRC | 15:43 | |
| rhallisey | shardy, I was thinking of having some like a ServiceList: keystone,neutron ... | 15:43 |
| rhallisey | would there be a way to map those to a resource | 15:44 |
| *** tiswanso has joined #tripleo | 15:44 | |
| rhallisey | vs tripleo::docker::services::keystone | 15:44 |
| dprince | rhallisey: so let me ask this. Do we want the docker interface refined before we go any further on the compible roles stuff | 15:44 |
| *** ccamacho has quit IRC | 15:45 | |
| rhallisey | dprince, no, you can continue. I just absorb what you have | 15:45 |
| dprince | rhallisey: well, I'm basically asking the core team not to land any features until the controller is done | 15:45 |
| rhallisey | I use the keystone role as is and plug it into the heat-docker-agents container | 15:45 |
| *** akrivoka has quit IRC | 15:46 | |
| shardy | dprince: I think it's fine to continue, but we do need to ensure the abstraction works for multiple implementations | 15:46 |
| *** akrivoka has joined #tripleo | 15:46 | |
| dprince | shardy: I think the abstractions can evolve with regards to where we call them | 15:46 |
| *** ramishra_ has joined #tripleo | 15:46 | |
| dprince | shardy: we aren't committing to an interface yet | 15:46 |
| rhallisey | shardy, I think it does work | 15:46 |
| dprince | rhallisey: it can work, but I do think we'll refine this a bit further too | 15:47 |
| rhallisey | dprince, I do eveythign in post | 15:47 |
| dprince | rhallisey: yes, that is good I think | 15:47 |
| dprince | rhallisey: but we can't mention Puppet in overcloud.yaml | 15:47 |
| rhallisey | dprince, so I don't interact much with what you have | 15:47 |
| rhallisey | I just consume it | 15:47 |
| dprince | rhallisey: so where we create the resource chain might need to change I think.... | 15:47 |
| dprince | rhallisey: I can fix that | 15:47 |
| rhallisey | dprince, my current patch uses the puppet resource change. I'm changing it | 15:48 |
| dprince | rhallisey: we can fix that :) | 15:48 |
| rhallisey | dprince, you can leave it as is. I'm having a container resource chain in post | 15:48 |
| *** ccamacho has joined #tripleo | 15:48 | |
| rhallisey | I use your resource chain to figure out what puppet modules I need | 15:48 |
| dprince | rhallisey: so we'll want to watch how many chains we create I think | 15:48 |
| dprince | rhallisey: specifically because it could effect the output of 'heat stack-validate' or something | 15:49 |
| rhallisey | right.. so this would be 2n | 15:49 |
| shardy | Yeah, I was thinking it could be much simpler if we just passed the Controllers into ControllerServiceChain, then deploy the containers for each service | 15:49 |
| shardy | and do nothing at all in Post for the container case | 15:49 |
| dprince | shardy: that means each services can add anything it wants to Heat software configs | 15:50 |
| dprince | shardy: which means a free-for-all | 15:50 |
| dprince | shardy: i.e. not much of an interface | 15:50 |
| *** ramishra_ has quit IRC | 15:50 | |
| shardy | dprince: So you're proposing we pass out say the container image and the service config? | 15:51 |
| dprince | shardy: I would expect the docker services to extend the puppet ones | 15:51 |
| shardy | then deploy all the containers in Post? | 15:51 |
| dprince | shardy: they would 'extend' it like I do the base services for pacemaker | 15:51 |
| dprince | shardy: http://git.openstack.org/cgit/openstack/tripleo-heat-templates/tree/puppet/services/pacemaker/keystone.yaml#n18 | 15:52 |
| dprince | shardy: that is where the 'config_settings' come from | 15:52 |
| dprince | shardy: then the docker services would have extra output parameters for the: | 15:52 |
| dprince | shardy: 1) docker container | 15:52 |
| shardy | dprince: Yeah I commented on the neutron patch about that | 15:52 |
| dprince | 2) docker compose section | 15:52 |
| shardy | it's fine, but we'll have a lot of layers of nesting just to abstract extending the interface | 15:52 |
| rhallisey | dprince, I was thinking a per service output | 15:53 |
| dprince | 3) puppet tags to apply (this is how we generate just configs w/ puppet) | 15:53 |
| shardy | which is proving really expensive from a heat perspective | 15:53 |
| rhallisey | dprince, ovs_config ovs_container | 15:53 |
| dprince | and then in the -post.yaml template we can re-combine those or run them separately as we see fit in an organized fashion | 15:53 |
| rhallisey | dprince, keystone_config keystone_container | 15:53 |
| dprince | shardy: 2 layers of nesting | 15:53 |
| shardy | dprince: sure, I guess it's the re-combining I'm not clear on in the container case | 15:53 |
| dprince | shardy: and do keep in mind tht these stacks are global, they aren't created for each server... | 15:54 |
| shardy | because we don't recombine anything, we keep it separate and launch a bunch of containers | 15:54 |
| shardy | dprince: yeah that will help limit things somewhat | 15:54 |
| dprince | shardy: we can run things separate (per service) for docker | 15:54 |
| *** jcoufal has quit IRC | 15:54 | |
| *** dtantsur|mtg is now known as dtantsur|afk | 15:54 | |
| shardy | In the pacemaker case, assuming say 25 services, it could still be an extra 50 stacks though | 15:54 |
| dprince | shardy: regardless of how we manage the resource chain | 15:54 |
| shardy | the EndpointMap unrolling was required due to ~70 IIRC | 15:55 |
| *** athomas has quit IRC | 15:55 | |
| shardy | I'd like to see us improve that inside heat, but it's a known issue atm | 15:55 |
| *** mgould has quit IRC | 15:55 | |
| dprince | shardy: right, the distinction with these containers is they are nested stacks which are created once and re-used | 15:56 |
| derekh | dprince: shardy So, all the jobs that were queued have failed, squid server didn't have a dns server after the reboot, fixed now, on the bright side the ZUUL queue is now clear... | 15:56 |
| dprince | derekh: way to go. Sounds like you just implemented a 'clear queue' button for us :) | 15:56 |
| dprince | rhallisey: I'm glad to see you prototyping this stuff | 15:57 |
| * derekh files that under "handy tricks" | 15:57 | |
| shardy | derekh: nice ;) | 15:57 |
| shardy | dprince: Cool, I'm fine with it for now but looking for optimisations which reduces the stack load a bit | 15:57 |
| openstackgerrit | Ben Nemec proposed openstack/os-net-config: Add explicit check for no active nics https://review.openstack.org/304724 | 15:58 |
| shardy | like, if all that changes is the step_config, we could potentially select the appropriate config from a map (json parameter) | 15:58 |
| rhallisey | dprince, so you think I should merge into the service.yaml resource chain? | 15:58 |
| *** ramishra_ has joined #tripleo | 15:58 | |
| rhallisey | dprince, versus a container one in post | 15:58 |
| dprince | rhallisey: I think we might should combine them, yes. | 15:58 |
| *** ifarkas has quit IRC | 15:58 | |
| dprince | rhallisey: my plan was to gradually move all of the config stuff out of controller.yaml first | 15:59 |
| rhallisey | my current patch reflects that. The patch in my local branch doesnt' | 15:59 |
| rhallisey | kk | 15:59 |
| dprince | rhallisey: once that happens I can move the resource chain into -post.yaml | 15:59 |
| dprince | rhallisey: you are pushing me to do that sooner... :) | 15:59 |
| *** lucasagomes is now known as lucas-brno | 15:59 | |
| rhallisey | :) | 15:59 |
| rhallisey | ok cool | 16:00 |
| dprince | rhallisey: which is fine, but would require a mega-patch to get the configs out in a massive blob | 16:00 |
| dprince | rhallisey: does that make sense? | 16:00 |
| rhallisey | dprince, yes | 16:00 |
| rhallisey | dprince, I'll take another stab at that patch today | 16:00 |
| dprince | rhallisey: so maybe you duplicate the Chain for now w/ a comment that our plan is to combine them once the roles are decomposed entirely | 16:00 |
| rhallisey | dprince, ok | 16:01 |
| dprince | rhallisey: one thing to check would be the output from a 'heat stack-validate' | 16:02 |
| *** mkovacik has quit IRC | 16:02 | |
| dprince | rhallisey: just to see how your docker service would get exposed to the UI (and eventually the CLI) via its parameters | 16:02 |
| rhallisey | dprince, gotcha.. I haven't gotten my patch completely working yet because I'm passing a string to docker-compose resource vs a json | 16:02 |
| rhallisey | kk | 16:03 |
| openstackgerrit | Lars Kellogg-Stedman proposed openstack/tripleo-quickstart: add scripts for performing YAML validation https://review.openstack.org/304737 | 16:12 |
| openstackgerrit | Merged openstack/tripleo-quickstart: add ipv4 network-isolation to quickstart for virt deployments https://review.openstack.org/303030 | 16:12 |
| *** apetrich has quit IRC | 16:13 | |
| *** ramishra_ has quit IRC | 16:13 | |
| *** ramishra_ has joined #tripleo | 16:14 | |
| *** oshvartz has joined #tripleo | 16:16 | |
| *** rbrady has quit IRC | 16:18 | |
| *** rbrady has joined #tripleo | 16:19 | |
| *** derekh has quit IRC | 16:20 | |
| *** shivrao has joined #tripleo | 16:21 | |
| *** shivrao_ has joined #tripleo | 16:21 | |
| dprince | bnemec: I just hit the bug you filed pointing to this https://review.openstack.org/#/c/291243 | 16:22 |
| bnemec | dprince: Yeah, should be fixed by https://review.openstack.org/#/c/304712/ | 16:23 |
| dprince | bnemec: yeah, I would have suggested we kick the tires on the breaking commit before landing it | 16:23 |
| *** ramishra has quit IRC | 16:23 | |
| dprince | bnemec: I think most cores don't realize that os-net-config is somewhat sensative to real baremetal in this code | 16:24 |
| bnemec | dprince: Yeah, I'm not even hitting it on baremetal though. It's breaking all of my virtual deployments too. | 16:24 |
| dprince | bnemec: oh, well that too :) | 16:25 |
| dprince | bnemec: so wait, how did this land then? | 16:25 |
| bnemec | I'm not sure. | 16:25 |
| *** shivrao has quit IRC | 16:25 | |
| *** shivrao_ is now known as shivrao | 16:25 | |
| openstackgerrit | Pradeep Kilambi proposed openstack/puppet-tripleo: Add redis profile https://review.openstack.org/304754 | 16:25 |
| bnemec | It doesn't make sense to me that it passed CI. | 16:25 |
| shardy | https://review.openstack.org/#/c/291243 did pass CI | 16:25 |
| dprince | bnemec: was this the we weren't building os-net-config packages problem? | 16:25 |
| dprince | or perhaps we still aren't!? | 16:26 |
| bnemec | dprince: It's possible, but stuff passed CI after it merged too so I don't think it's actually broken in CI. | 16:26 |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-heat-templates: Make sure openstack services are dependent on openstack-core https://review.openstack.org/304592 | 16:26 |
| *** dmacpher is now known as dmacpher-afk | 16:26 | |
| *** ebalduf_ has joined #tripleo | 16:26 | |
| bnemec | Unless the package build was delayed a bunch and it only showed up in the repo yesterday about the time everything else blew up. | 16:26 |
| openstackgerrit | Ben Nemec proposed openstack/os-net-config: Test breaking change in os-net-config https://review.openstack.org/304760 | 16:29 |
| * bnemec pushes a broken os-net-config patch just to see if we're testing it properly | 16:29 | |
| dprince | shardy: this fixes it for me https://review.openstack.org/#/c/304712/1 | 16:30 |
| dprince | bnemec: thanks for pushing this. I've been blocked since yesterday due to the sahara patch, I rebuilt my overcloud image and then I hit this | 16:30 |
| bnemec | dprince: np. I also failed to deploy a single overcloud yesterday, thanks to this and various other issues. | 16:31 |
| *** ramishra_ has quit IRC | 16:33 | |
| *** sshnaidm has quit IRC | 16:33 | |
| dprince | bnemec: -1 on your second patch though. | 16:33 |
| dprince | bnemec: I think that would break the case where I had no active NICs, but wanted to configure a bridge or something | 16:34 |
| dprince | bnemec: or it could. if that function got called, either now or in the future | 16:34 |
| shardy | bnemec: if it broke yesterday doesn't that imply we're not building from the repo, e.g we released 0.2.4 yesterday? | 16:35 |
| bnemec | dprince: Is it even possible to configure a working bridge without an interface? | 16:35 |
| bnemec | shardy: Oh, was there a release? That could be it. | 16:36 |
| dprince | bnemec: brctl addbr foo | 16:36 |
| * bnemec looks at what is installed | 16:36 | |
| jistr | marios: re your question on scrum -- reported the bug https://bugs.launchpad.net/tripleo/+bug/1569444 | 16:36 |
| openstack | Launchpad bug 1569444 in tripleo "pacemaker_resource_restart.sh hangs on crm_resource --wait" [High,In progress] - Assigned to Jiří Stránský (jistr) | 16:36 |
| bnemec | shardy: My image has os-net-config-0.2.5-0.20160411183356.2ab73df.el7.centos.noarch | 16:36 |
| marios | jistr: ack tx | 16:37 |
| shardy | bnemec: ack, OK I guess it's just coincidence as that patch merged yesterday also | 16:37 |
| bnemec | dprince: But will it do anything? | 16:38 |
| bnemec | I guess I can just make it a warning in any case. At least the information is there then. | 16:38 |
| shardy | which actually was from 2188cf1651648af1900b7bb070f9b1eb3f982c3b just before it merged | 16:38 |
| bnemec | shardy: I pushed a change that should break, so we'll see if CI is testing properly: https://review.openstack.org/#/c/304760/1/os_net_config/objects.py | 16:38 |
| shardy | bnemec: sounds good, thanks | 16:39 |
| *** dmacpher-afk is now known as dmacpher | 16:40 | |
| *** dustins has joined #tripleo | 16:42 | |
| gfidente | bnemec though last release is 0.2.4, so I have no clue what 0.2.5 is for | 16:43 |
| openstackgerrit | Pradeep Kilambi proposed openstack/puppet-tripleo: Add mongodb profiles https://review.openstack.org/304780 | 16:43 |
| *** shivrao has quit IRC | 16:43 | |
| bnemec | gfidente: I assume it has to do with how pbr computes versions. | 16:44 |
| bnemec | Not that I have any clue what the semantics around that are these days. | 16:44 |
| *** sambetts is now known as sambetts|afk | 16:45 | |
| trown | gfidente: delorean takes its version from `python setup.py --version` which takes version from pbr | 16:46 |
| *** akshai has quit IRC | 16:47 | |
| *** liverpooler has quit IRC | 16:47 | |
| *** dprince has quit IRC | 16:47 | |
| * trown should not have +A'd that os-net-config patch... | 16:48 | |
| *** tremble has quit IRC | 16:48 | |
| openstackgerrit | Pradeep Kilambi proposed openstack/puppet-tripleo: Add mongodb profiles https://review.openstack.org/304780 | 16:48 |
| trown | I put +1 originally because it looked good, but then saw it had passing CI and 2 +2's so just +2'd to +A | 16:48 |
| *** dmacpher is now known as dmacpher-afk | 16:50 | |
| *** akshai has joined #tripleo | 16:52 | |
| *** ramishra has joined #tripleo | 16:54 | |
| *** ramishra has quit IRC | 16:54 | |
| *** Guest15115 has quit IRC | 16:55 | |
| *** trown is now known as trown|lunch | 16:57 | |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: Do Not Merge: Test stack-update https://review.openstack.org/304787 | 16:59 |
| openstackgerrit | Ben Nemec proposed openstack/os-net-config: Add warning for no active nics https://review.openstack.org/304724 | 17:00 |
| *** tiswanso has quit IRC | 17:00 | |
| *** manous has joined #tripleo | 17:01 | |
| *** akshai has quit IRC | 17:03 | |
| *** akshai has joined #tripleo | 17:03 | |
| *** ramishra has joined #tripleo | 17:04 | |
| *** sambetts|afk has quit IRC | 17:04 | |
| *** tiswanso has joined #tripleo | 17:04 | |
| *** sambetts has joined #tripleo | 17:06 | |
| *** ramishra has quit IRC | 17:06 | |
| *** ramishra has joined #tripleo | 17:09 | |
| shardy | trown|lunch, larsks: Hey I just raised a few quickstart bugs - they're mostly known issues or things we've previously discussed but I wanted to keep track of them | 17:13 |
| larsks | shardy: thanks! | 17:14 |
| *** gfidente has quit IRC | 17:14 | |
| larsks | shardy: note re: https://bugs.launchpad.net/tripleo-quickstart/+bug/1569472 that there is actually documetnation about that in https://github.com/redhat-openstack/tripleo-quickstart/blob/master/docs/accessing-libvirt.md | 17:15 |
| openstack | Launchpad bug 1569472 in tripleo-quickstart "VMs not reflected in virt-manager" [Undecided,New] | 17:15 |
| *** jistr has quit IRC | 17:15 | |
| larsks | (although that is for virsh rather than virt-manager, which is a little trickier) | 17:15 |
| shardy | When we figure out where the upstream images are coming from it'd be great if we could fully automate everything, so you run quickstart then everything is ready to go | 17:15 |
| shardy | larsks: thanks, I'd not spotted that | 17:16 |
| shardy | larsks: I still think it's confusing for folks running it on their local box, already via an unprivileged account | 17:16 |
| shardy | so it'd be good to consider if there's more we can do to streamline things, or make the "where are my VMs" docs more prominent | 17:17 |
| shardy | what *really* tripped me up was accidentally running with sudo | 17:17 |
| larsks | That should pretty much work, actually. As should getting to an unprivileged account via 'su -'. | 17:17 |
| larsks | But yeah, we can make the docs more obvious. | 17:17 |
| shardy | then you see the VMs, but running without sudo they are invisible | 17:17 |
| *** ramishra has quit IRC | 17:17 | |
| shardy | larsks: I've just been running via my normal shardy account | 17:18 |
| shardy | first attempt I made a mistake and did sudo quickstart.sh localhost | 17:18 |
| shardy | that creates the VMs, visible in virt-manager, but then fails to boot the undercloud | 17:18 |
| larsks | shardy: Actually, if you want to update the bug with the specific scenarios you tried that didn't work as expected, that would be a useful reference to have handy. | 17:19 |
| shardy | running it again works fine without the sudo, but as an ex instack-virt-setup user, it's confusing | 17:19 |
| *** qasims has joined #tripleo | 17:19 | |
| shardy | larsks: done | 17:23 |
| *** manous has quit IRC | 17:24 | |
| shardy | some of this is completely user error, but I'm trying to highlight pitfalls other folks may encounter :) | 17:24 |
| openstackgerrit | Ethan Gafford proposed openstack/python-tripleoclient: Trove integration https://review.openstack.org/233241 | 17:25 |
| openstackgerrit | Ethan Gafford proposed openstack/tripleo-heat-templates: Trove Integration https://review.openstack.org/233240 | 17:25 |
| *** jpena is now known as jpena|off | 17:28 | |
| *** ramishra has joined #tripleo | 17:29 | |
| larsks | shardy: highlighting pitfalls is extremely useful! | 17:29 |
| *** sshnaidm has joined #tripleo | 17:29 | |
| *** davidlenwell has quit IRC | 17:29 | |
| *** davidlenwell has joined #tripleo | 17:36 | |
| *** ChanServ sets mode: +v davidlenwell | 17:36 | |
| *** ramishra has quit IRC | 17:37 | |
| *** dujelly has joined #tripleo | 17:42 | |
| *** ramishra has joined #tripleo | 17:43 | |
| *** jaosorior has joined #tripleo | 17:45 | |
| *** cwolferh has joined #tripleo | 17:46 | |
| *** shardy has quit IRC | 17:47 | |
| *** ramishra has quit IRC | 17:48 | |
| *** ramishra has joined #tripleo | 17:49 | |
| *** rcernin has quit IRC | 17:53 | |
| openstackgerrit | Pradeep Kilambi proposed openstack/puppet-tripleo: Add redis profile https://review.openstack.org/304754 | 17:56 |
| *** trown|lunch is now known as trown | 17:57 | |
| *** manous_ has joined #tripleo | 17:58 | |
| *** shivrao has joined #tripleo | 17:59 | |
| *** ramishra has quit IRC | 17:59 | |
| *** jaosorior has quit IRC | 17:59 | |
| *** ramishra has joined #tripleo | 18:00 | |
| *** chem has quit IRC | 18:02 | |
| bnemec | https://review.openstack.org/#/c/304802/ fixes the duplicate sections problem in oslo.config for me. | 18:05 |
| trown | bnemec: nice, I will give that a go. | 18:06 |
| bnemec | It's a little ugly, but I got 100% unit test coverage of the new code so I'm reasonably confident it's correct. | 18:07 |
| bnemec | It also looks like we are at least testing the changes in os-net-config: http://logs.openstack.org/60/304760/1/check-tripleo/gate-tripleo-ci-f22-ha/cc7b9a0/console.html#_2016-04-12_16_58_38_202 | 18:08 |
| bnemec | I suppose it's possible we aren't installing it in the overcloud image for some reason though. | 18:09 |
| *** ramishra has quit IRC | 18:09 | |
| *** coolsvap has quit IRC | 18:10 | |
| trown | it is also possible (though a bit weirder) that operstate is upper case in our CI env | 18:10 |
| trown | oh... but then CI would not be broken | 18:10 |
| *** ramishra has joined #tripleo | 18:10 | |
| *** dujelly has quit IRC | 18:12 | |
| *** ramishra has quit IRC | 18:16 | |
| *** ramishra has joined #tripleo | 18:17 | |
| *** ramishra has quit IRC | 18:18 | |
| *** mcornea_ has quit IRC | 18:18 | |
| *** ramishra has joined #tripleo | 18:23 | |
| openstackgerrit | Ben Nemec proposed openstack/os-net-config: Nothing to see here https://review.openstack.org/304760 | 18:25 |
| bnemec | I think I owe derek royalties now or something. :-) | 18:25 |
| *** ramishra has quit IRC | 18:28 | |
| *** ramishra has joined #tripleo | 18:28 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart: ignore errors on virsh net-undefine in libvirt cleanup https://review.openstack.org/304810 | 18:29 |
| bnemec | CI networking looks to be in a bad way again: http://logstash.openstack.org/#dashboard/file/logstash.json?query=build_name%3A%20*tripleo-ci*%20AND%20build_status%3A%20FAILURE%20AND%20(message%3A%20%5C%22Could%20not%20resolve%20host%3A%20github.com%5C%22%20OR%20message%3A%20%5C%22fatal%3A%20The%20remote%20end%20hung%20up%20unexpectedly%5C%22) | 18:30 |
| bnemec | Every job I've looked at has died on a network failure of some sort. | 18:30 |
| qasims | +1 | 18:32 |
| slagle | bnemec: any object to just merging this? https://review.openstack.org/#/c/304712 | 18:37 |
| slagle | i'm seeing CI failures where os-net-config finds no nics | 18:38 |
| *** ramishra has quit IRC | 18:39 | |
| openstackgerrit | Merged openstack/tripleo-quickstart: Add ignore error on virsh destroy https://review.openstack.org/304286 | 18:39 |
| bnemec | slagle: It's somewhat concerning to me that that patch failed and has no controller logs. Maybe we really aren't testing os-net-config on the overcloud properly. | 18:39 |
| *** ramishra has joined #tripleo | 18:39 | |
| bnemec | In which case I guess we might as well merge it though. | 18:40 |
| slagle | yea it can't get the logs b/c there are no nics | 18:40 |
| *** florianf has quit IRC | 18:41 | |
| trown | there are no nics in CI on the patch that is meant to fix there being no nics? | 18:41 |
| bnemec | Yeah, that's what concerns me. | 18:41 |
| slagle | bnemec: this is how i'm seeing it fail: http://paste.openstack.org/show/493857/ | 18:42 |
| slagle | i guess what's odd is that there should be an eth2 | 18:42 |
| bnemec | No logs at all on the ha job where os-net-config is used on all the nodes. | 18:42 |
| trown | bnemec: I bet we are not building it and just pulling from delorean current | 18:42 |
| bnemec | slagle: Yeah, that looks related to this issue. | 18:42 |
| slagle | ok | 18:42 |
| trown | bnemec: and yesterday there were some delorean outages that meant we were on the backup server so it took a while to get the broken package | 18:42 |
| bnemec | trown: We're definitely building it. My test broken patch blew up on the undercloud. | 18:42 |
| trown | oh right | 18:43 |
| trown | rats | 18:43 |
| bnemec | But that doesn't mean it's getting installed in the overcloud image. | 18:43 |
| slagle | let's just merge | 18:43 |
| bnemec | slagle: Yeah, I'm fine with that. | 18:43 |
| trown | maybe overcloud image is getting delorean package? | 18:43 |
| bnemec | It's a tiny change and everyone who's looked at it says it matches their environment. | 18:43 |
| trown | ya and we cant break broken | 18:44 |
| bnemec | trown: It's possible. I don't actually understand how the test stuff is injected into the overcloud image build. Maybe it isn't. :-/ | 18:44 |
| *** tiswanso has quit IRC | 18:44 | |
| bnemec | Now I'm really curious to see the results on https://review.openstack.org/#/c/304760/ | 18:45 |
| bnemec | Although if it's broken we won't see any useful results. :-( | 18:45 |
| *** apetrich has joined #tripleo | 18:46 | |
| trown | oh ya, cause no logs... though it would be interesting to recheck after we get CI fixed | 18:46 |
| trown | I would put money on that warning not showing up | 18:47 |
| openstackgerrit | Merged openstack/os-net-config: Normalize operstate value for interfaces https://review.openstack.org/304712 | 18:47 |
| *** ramishra has quit IRC | 18:49 | |
| *** ramishra has joined #tripleo | 18:50 | |
| *** apetrich_ has joined #tripleo | 18:52 | |
| *** apetrich has quit IRC | 18:53 | |
| *** ramishra has quit IRC | 18:54 | |
| *** ramishra has joined #tripleo | 18:54 | |
| *** ramishra has quit IRC | 18:54 | |
| *** chem has joined #tripleo | 19:01 | |
| *** tiswanso has joined #tripleo | 19:01 | |
| *** yuanying has quit IRC | 19:04 | |
| *** dprince has joined #tripleo | 19:06 | |
| *** ramishra has joined #tripleo | 19:06 | |
| openstackgerrit | Ben Nemec proposed openstack/os-net-config: Nothing to see here...probably https://review.openstack.org/304760 | 19:08 |
| *** ramishra has quit IRC | 19:11 | |
| *** ramishra has joined #tripleo | 19:13 | |
| bnemec | Better logstash query for the networking issues: http://logstash.openstack.org/#dashboard/file/logstash.json?query=build_name%3A%20*tripleo-ci*%20AND%20build_status%3A%20FAILURE%20AND%20(message%3A%20%5C%22Could%20not%20resolve%20host%3A%20github.com%5C%22%20OR%20message%3A%20%5C%22fatal%3A%20The%20remote%20end%20hung%20up%20unexpectedly%5C%22) | 19:15 |
| bnemec | Although logstash is still missing a bunch of failures for some reason. | 19:15 |
| *** yamahata has joined #tripleo | 19:20 | |
| *** ramishra has quit IRC | 19:21 | |
| *** ramishra has joined #tripleo | 19:22 | |
| *** qasims has quit IRC | 19:25 | |
| *** ramishra has quit IRC | 19:31 | |
| *** ramishra has joined #tripleo | 19:31 | |
| *** dustins has quit IRC | 19:39 | |
| *** ramishra has quit IRC | 19:41 | |
| *** dustins has joined #tripleo | 19:42 | |
| *** ramishra has joined #tripleo | 19:42 | |
| *** ramishra has quit IRC | 19:45 | |
| *** ramishra has joined #tripleo | 19:47 | |
| *** ramishra has quit IRC | 19:56 | |
| *** ramishra has joined #tripleo | 19:57 | |
| *** Goneri has quit IRC | 19:57 | |
| *** lblanchard has quit IRC | 19:58 | |
| *** apetrich_ has quit IRC | 20:00 | |
| *** tiswanso has quit IRC | 20:00 | |
| *** jprovazn has quit IRC | 20:03 | |
| *** tiswanso has joined #tripleo | 20:04 | |
| *** bandini has quit IRC | 20:10 | |
| *** bandini has joined #tripleo | 20:10 | |
| *** ramishra has quit IRC | 20:11 | |
| *** qasims has joined #tripleo | 20:12 | |
| *** ramishra has joined #tripleo | 20:12 | |
| *** akuznetsov has joined #tripleo | 20:17 | |
| *** ramishra has quit IRC | 20:24 | |
| *** ramishra has joined #tripleo | 20:24 | |
| *** jayg is now known as jayg|g0n3 | 20:26 | |
| *** ramishra has quit IRC | 20:26 | |
| *** manous_ has quit IRC | 20:27 | |
| *** akuznetsov has quit IRC | 20:27 | |
| *** ramishra has joined #tripleo | 20:29 | |
| *** ccamacho has quit IRC | 20:38 | |
| *** ramishra has quit IRC | 20:41 | |
| *** ramishra has joined #tripleo | 20:42 | |
| *** dprince has quit IRC | 20:43 | |
| openstackgerrit | Ben Nemec proposed openstack/tripleo-heat-templates: Explicitly set nova and neutron host on controllers https://review.openstack.org/304858 | 20:51 |
| openstackgerrit | John Trowbridge proposed openstack/tripleo-quickstart: Switch to using standalone role for image building https://review.openstack.org/304860 | 20:56 |
| *** ramishra has quit IRC | 20:56 | |
| openstackgerrit | John Trowbridge proposed openstack/tripleo-quickstart: Switch to using standalone role for image building https://review.openstack.org/304860 | 20:59 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Add final success/failure message to ping test https://review.openstack.org/304864 | 21:04 |
| *** ramishra has joined #tripleo | 21:05 | |
| *** MaxPC has quit IRC | 21:07 | |
| openstackgerrit | John Trowbridge proposed openstack/tripleo-quickstart: Switch to using standalone role for image building https://review.openstack.org/304860 | 21:07 |
| openstackgerrit | Dmitry Ilyin proposed openstack/puppet-pacemaker: Merge with fuel-infra/puppet-pacemaker https://review.openstack.org/296440 | 21:08 |
| *** julim has quit IRC | 21:11 | |
| *** qasims has quit IRC | 21:11 | |
| *** trown is now known as trown|outtypewww | 21:11 | |
| *** ramishra has quit IRC | 21:17 | |
| *** ramishra has joined #tripleo | 21:20 | |
| *** myoung has quit IRC | 21:21 | |
| *** myoung has joined #tripleo | 21:21 | |
| *** oshvartz has quit IRC | 21:36 | |
| *** ramishra has quit IRC | 21:42 | |
| *** ayoung has quit IRC | 21:46 | |
| *** ramishra has joined #tripleo | 21:48 | |
| *** julim has joined #tripleo | 21:49 | |
| *** oshvartz has joined #tripleo | 21:50 | |
| *** fragatina has quit IRC | 21:54 | |
| *** akrivoka has quit IRC | 21:54 | |
| *** tiswanso has quit IRC | 21:55 | |
| *** ramishra has quit IRC | 21:57 | |
| *** ramishra has joined #tripleo | 21:58 | |
| *** dustins has quit IRC | 21:58 | |
| *** akshai has quit IRC | 22:07 | |
| *** lblanchard has joined #tripleo | 22:10 | |
| *** morazi has quit IRC | 22:11 | |
| *** ramishra has quit IRC | 22:19 | |
| *** ramishra has joined #tripleo | 22:20 | |
| *** fragatina has joined #tripleo | 22:24 | |
| *** ramishra has quit IRC | 22:25 | |
| *** derekh has joined #tripleo | 22:25 | |
| derekh | slagle: bnemec just incase yer looking into it at the moment, looks like the net problems, don't seem to be with our openstack deployment | 22:27 |
| derekh | I've been trying to locate where packets are being dropped, and it can be reproduced fairly easily from the bastion | 22:28 |
| *** ramishra has joined #tripleo | 22:28 | |
| derekh | after running a bunch of these [derekh@host01-rack01 ~]$ host -v -t A git.openstack.org 8.8.8.8 | 22:29 |
| derekh | about 1 in 10 or so timout ;; connection timed out; trying next origin | 22:29 |
| derekh | tpcdump shows the UDP packet going out but nothing coming back | 22:29 |
| derekh | I'll ping the lab guys in the morning | 22:30 |
| derekh | slagle: bnemec ^ | 22:30 |
| *** fragatina has quit IRC | 22:30 | |
| *** ayoung has joined #tripleo | 22:31 | |
| *** Marga_ has joined #tripleo | 22:32 | |
| *** ramishra has quit IRC | 22:33 | |
| *** derekh has quit IRC | 22:33 | |
| *** ramishra has joined #tripleo | 22:34 | |
| *** ramishra has quit IRC | 22:35 | |
| *** Marga_ has quit IRC | 22:35 | |
| *** Marga_ has joined #tripleo | 22:36 | |
| *** cwolferh has quit IRC | 22:37 | |
| *** cwolferh has joined #tripleo | 22:38 | |
| *** ramishra has joined #tripleo | 22:39 | |
| *** ebalduf_ has quit IRC | 22:47 | |
| *** ramishra has quit IRC | 22:52 | |
| *** ramishra has joined #tripleo | 22:53 | |
| *** sanjay__u has quit IRC | 22:55 | |
| *** ramishra has quit IRC | 23:07 | |
| *** ramishra has joined #tripleo | 23:08 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/tripleo-common: Updated from global requirements https://review.openstack.org/300629 | 23:15 |
| *** akshai has joined #tripleo | 23:16 | |
| *** akshai_ has joined #tripleo | 23:17 | |
| *** yuanying has joined #tripleo | 23:18 | |
| *** ramishra has quit IRC | 23:19 | |
| *** ramishra has joined #tripleo | 23:20 | |
| *** akshai has quit IRC | 23:21 | |
| *** ramishra has quit IRC | 23:24 | |
| *** ramishra has joined #tripleo | 23:25 | |
| *** akshai_ has quit IRC | 23:30 | |
| *** yuanying has quit IRC | 23:36 | |
| *** panda has quit IRC | 23:41 | |
| *** panda has joined #tripleo | 23:42 | |
| *** ramishra has quit IRC | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
