| SpamapS | dkehn: I have to agree. Seattle is a much nicer experience. | 00:01 |
|---|---|---|
| SpamapS | Sunnyvale feels like a trip to the suburbs. :-P | 00:02 |
| dkehn | clarkb: I think I've been there in a past life, it is in the middle of nowwhere | 00:02 |
| dkehn | SpamapS: there is always Ft. Collins, lol | 00:02 |
| dkehn | speaking of in the middle of nowhere | 00:02 |
| *** cd-undercloud has joined #tripleo | 00:04 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 00:04 |
| *** cd-undercloud has quit IRC | 00:04 | |
| SpamapS | dkehn: actually they have better beer than sunnyvale :) | 00:05 |
| SpamapS | interesting | 00:05 |
| dkehn | SpamapS: Seattle or Ft. Collins | 00:05 |
| SpamapS | so we got our nodes stuck in "REBUILD" | 00:05 |
| SpamapS | dkehn: Ft. Collins. | 00:05 |
| dkehn | SpamapS: yes, but so does Seattle | 00:05 |
| SpamapS | Yeah, bellies will be happier in Seattle for sure. | 00:06 |
| cody-somerville | Are we talking about Sprint location? | 00:08 |
| *** boris-42 has quit IRC | 00:19 | |
| dkehn | cody-somerville: yepper | 00:27 |
| *** sballe has quit IRC | 00:30 | |
| *** epim has joined #tripleo | 00:31 | |
| *** epim has quit IRC | 00:31 | |
| NobodyCam | SpamapS: look at your first comment on https://review.openstack.org/#/c/66461/1/elements/nova-ironic/os-refresh-config/configure.d/81-nat-metadata | 00:34 |
| *** UtahDave has quit IRC | 00:34 | |
| NobodyCam | Fail early, fail often is the device? | 00:34 |
| *** CaptTofu has joined #tripleo | 00:40 | |
| SpamapS | NobodyCam: my point there is to not use --type raw.. like.. ever. :) | 00:40 |
| NobodyCam | :) ack ok :) | 00:41 |
| SpamapS | NobodyCam: netdevice will cause an earlier failure if you try to put something that is clearly not a netdevice in for that field. | 00:41 |
| SpamapS | NobodyCam: oh also while looking at that, '' is not an option, so you should not have a --key-default | 00:42 |
| SpamapS | NobodyCam: '' will produce an error below | 00:42 |
| NobodyCam | ok will test and have a patch up soonish | 00:43 |
| NobodyCam | just have to work out how to address ghe's comment :-p | 00:44 |
| *** jayg is now known as jayg|g0n3 | 00:53 | |
| *** dkehn has quit IRC | 00:56 | |
| *** dkehn has joined #tripleo | 00:58 | |
| SpamapS | interesting.. our boxes are wedged in REBUILD | 00:59 |
| SpamapS | ERROR: Instance 61436e6d-864a-44ab-904a-dd77e57a616e in task_state rebuild_spawning. Cannot stop while the instance is in this state. (HTTP 400) (Request-ID: req-9d688ba2-1d0d-462d-b632-580644524989) | 01:00 |
| SpamapS | ugh | 01:09 |
| SpamapS | doesn't seem to be much we can do if a REBUILD gets wedged like this | 01:09 |
| SpamapS | can't stop.. can't force rebuild to start over | 01:09 |
| *** CaptTofu has quit IRC | 01:15 | |
| *** CaptTofu has joined #tripleo | 01:18 | |
| lifeless | SpamapS: task state is cleared if you restart nova-compute, I think. | 01:21 |
| lifeless | SpamapS: but even so - bug filing time ;) | 01:21 |
| SpamapS | yeah | 01:21 |
| SpamapS | I started with this https://bugs.launchpad.net/tripleo/+bug/1272595 | 01:21 |
| lifeless | I'll look later :) gotta run again, or will miss movie | 01:22 |
| SpamapS | lifeless: actually even weirder, if you nova delete them .. they go back to active | 01:22 |
| SpamapS | but they are _not_ active :-P | 01:22 |
| SpamapS | ok .. not easy to get those deleted | 01:39 |
| *** cwolferh has quit IRC | 01:42 | |
| SpamapS | https://bugs.launchpad.net/nova/+bug/1272600 filed in Nova | 01:45 |
| *** greghaynes has quit IRC | 01:46 | |
| *** greghaynes has joined #tripleo | 01:55 | |
| *** CaptTofu has quit IRC | 01:56 | |
| *** flashgordon is now known as jog0 | 01:57 | |
| *** amrit has quit IRC | 02:21 | |
| *** cd-undercloud has joined #tripleo | 02:48 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 02:48 |
| *** cd-undercloud has quit IRC | 02:48 | |
| *** beekneemech has quit IRC | 03:21 | |
| *** bnemec has joined #tripleo | 03:25 | |
| *** taps has quit IRC | 03:44 | |
| *** CaptTofu has joined #tripleo | 03:52 | |
| *** cd-undercloud has joined #tripleo | 03:55 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 03:55 |
| *** cd-undercloud has quit IRC | 03:55 | |
| *** coolsvap has quit IRC | 04:08 | |
| *** rwsu has quit IRC | 04:25 | |
| *** ramishra has joined #tripleo | 04:48 | |
| *** cd-undercloud has joined #tripleo | 05:04 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 05:04 |
| *** cd-undercloud has quit IRC | 05:04 | |
| *** greghaynes has quit IRC | 05:05 | |
| *** e0ne has quit IRC | 05:07 | |
| *** greghaynes has joined #tripleo | 05:10 | |
| *** CaptTofu has quit IRC | 05:12 | |
| *** CaptTofu has joined #tripleo | 05:13 | |
| *** ramishra has quit IRC | 05:14 | |
| *** AaronGr is now known as AaronGr_Zzz | 05:16 | |
| *** ramishra has joined #tripleo | 05:17 | |
| *** CaptTofu has quit IRC | 05:18 | |
| *** ramishra_ has joined #tripleo | 05:23 | |
| *** ramishra has quit IRC | 05:23 | |
| *** ramishra_ has quit IRC | 05:32 | |
| *** jprovazn has quit IRC | 05:50 | |
| *** jprovazn has joined #tripleo | 05:51 | |
| *** ramishra has joined #tripleo | 05:54 | |
| *** ramishra_ has joined #tripleo | 06:03 | |
| *** tzumainn has quit IRC | 06:05 | |
| SpamapS | weird.. things seem totally screwed on undercloud nova now | 06:06 |
| *** ramishra_ has quit IRC | 06:06 | |
| SpamapS | | fault | {u'message': u'{"error": {"message": "object of type \'NoneType\' has no len()", "code": 400, "title": "Bad Request"}}', u'code': 500, u'details': u' File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 259, in decorated_function | | 06:06 |
| *** ramishra has quit IRC | 06:06 | |
| SpamapS | seems like there is some kind of cleanup going on | 06:06 |
| *** ramishra has joined #tripleo | 06:06 | |
| mordred | SpamapS: doesn't seem like clean to me | 06:10 |
| SpamapS | yeah its weird | 06:10 |
| SpamapS | it is getting some kind of unauthorized error while trying to start instances | 06:10 |
| *** ramishra has quit IRC | 06:11 | |
| *** cd-undercloud has joined #tripleo | 06:11 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 06:11 |
| *** cd-undercloud has quit IRC | 06:11 | |
| SpamapS | mordred: what is frustrating is that nova now won't delete these instances | 06:15 |
| *** edmund has quit IRC | 06:16 | |
| mordred | SpamapS: wow | 06:29 |
| mordred | SpamapS: I believe I've heard of this | 06:29 |
| mordred | but I don't know what's up | 06:29 |
| SpamapS | just tried upgrading neutron to latest.. no help | 06:29 |
| *** coolsvap has joined #tripleo | 06:34 | |
| *** coolsvap has quit IRC | 06:34 | |
| *** ramishra has joined #tripleo | 06:35 | |
| *** akuznetsov has joined #tripleo | 06:39 | |
| lifeless | SpamapS: oh hai | 06:40 |
| SpamapS | lifeless: any insight into how to unbreak nova? | 06:43 |
| lifeless | SpamapS: whats up | 06:43 |
| SpamapS | I'm trying to find where neutron or keystone are raising this NoneType has no len error. | 06:43 |
| lifeless | logging in now to poke | 06:44 |
| SpamapS | neither is logging it | 06:44 |
| lifeless | instances have no ports | 06:44 |
| SpamapS | right, they're getting refused auth by keystone | 06:45 |
| SpamapS | or rather, nova is | 06:45 |
| SpamapS | at /tokens | 06:45 |
| lifeless | here's your problem | 06:46 |
| lifeless | File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/virt/baremetal/db/sqlalchemy/api.py", line 152, in bm_node_get_by_instance_uuid | 06:46 |
| lifeless | raise exception.InstanceNotFound(instance_id=instance_uuid) | 06:46 |
| lifeless | InstanceNotFound: Instance 84c6090b-bf42-4c6a-b2ff-afb22b5ff156 could not be found. | 06:46 |
| SpamapS | | 84c6090b-bf42-4c6a-b2ff-afb22b5ff156 | overcloud-NovaCompute0-scls5xj54oym | ERROR | deleting | NOSTATE | | | 06:47 |
| SpamapS | it never got assigned to a bm node | 06:47 |
| lifeless | just looking up the bug for you | 06:47 |
| SpamapS | is this where if we fail to get a bm node, we can't do "neutron things" ? | 06:48 |
| lifeless | it should be in https://bugs.launchpad.net/nova/+bugs?field.tag=baremetal | 06:48 |
| lifeless | there may be multiple issues | 06:48 |
| lifeless | but this one | 06:48 |
| lifeless | is bug 1177584 | 06:48 |
| lifeless | I think | 06:48 |
| lifeless | well, same umbrella | 06:49 |
| SpamapS | so is the fix to assign it manually to a bm node and try to delete it again? | 06:50 |
| lifeless | its the reverse of https://bugs.launchpad.net/nova/+bug/1226342 | 06:50 |
| lifeless | filing a bug | 06:50 |
| lifeless | nah, lets fix it. | 06:50 |
| SpamapS | lifeless: this started after we updated to latest nova | 06:50 |
| SpamapS | lifeless: I thought it was the REBUILD getting them wedged but now I think maybe it is just that deleting bm nodes might be badly broken. | 06:51 |
| SpamapS | or maybe even provisioning them | 06:51 |
| SpamapS | since I had errors before I tried to delete | 06:51 |
| SpamapS | same neutronclient failing to auth errors | 06:51 |
| *** ramishra has quit IRC | 06:52 | |
| *** ramishra has joined #tripleo | 06:52 | |
| lifeless | I think we're in a cascade fail situation | 06:53 |
| lifeless | one thing masking / causing another | 06:53 |
| lifeless | anyhow | 06:53 |
| SpamapS | agreed :( | 06:54 |
| SpamapS | lifeless: could be an old keystone bug | 06:54 |
| lifeless | https://bugs.launchpad.net/nova/+bug/1272623 | 06:54 |
| lifeless | oh this is in init_host | 06:55 |
| lifeless | failing to startup | 06:55 |
| lifeless | I'm sure I filed this before | 06:55 |
| lifeless | ah yes | 06:55 |
| lifeless | bug 1168610 | 06:55 |
| lifeless | different trace | 06:55 |
| lifeless | same cause I think | 06:55 |
| lifeless | new title on the bug I filed | 06:56 |
| SpamapS | lifeless: ok, I'm out of juice.. going to sleep now. | 06:58 |
| lifeless | k | 06:58 |
| lifeless | I'll fix and monkey that on | 06:58 |
| SpamapS | stopped tripleo-cd | 06:58 |
| SpamapS | going to buy some champagne for the next time it status=0's twice in a row | 06:58 |
| lifeless | SpamapS: right, past that, onto te unauthorized | 07:04 |
| lifeless | SpamapS: what else did you alter? | 07:06 |
| rpodolyaka | morning tripleo | 07:11 |
| *** ramishra has quit IRC | 07:13 | |
| *** CaptTofu has joined #tripleo | 07:14 | |
| *** ramishra has joined #tripleo | 07:14 | |
| lifeless | SpamapS: http://lists.openstack.org/pipermail/openstack/2013-November/003478.html | 07:15 |
| *** CaptTofu has quit IRC | 07:18 | |
| *** ramishra has quit IRC | 07:19 | |
| *** ramishra has joined #tripleo | 07:26 | |
| *** akuznetsov has quit IRC | 07:37 | |
| lifeless | tried git keystoneclient, no change | 07:48 |
| lifeless | tried git neutronclient, no change | 07:50 |
| *** ramishra has quit IRC | 07:50 | |
| SpamapS | lifeless: git keystone maybe? | 07:51 |
| *** ramishra has joined #tripleo | 07:51 | |
| lifeless | SpamapS: maybe | 07:51 |
| * SpamapS should really be asleep :-P | 07:51 | |
| lifeless | SpamapS: what had you tried ? | 07:51 |
| SpamapS | lifeless: I'm looking at len() calls in keystone at the commit we have now.. | 07:51 |
| SpamapS | lifeless: oh I should have said, I tried git neutron | 07:51 |
| SpamapS | nothing like dragging ye olde undercloud into the future ;) | 07:51 |
| lifeless | oh god, it's going to be a disaster | 07:52 |
| SpamapS | hm with keystone the token backend len() calls are all guarded against None.. | 07:53 |
| *** ramishra has quit IRC | 07:55 | |
| lifeless | https://bugs.launchpad.net/python-keystoneclient/+bug/1257541 looked suspiciously relevant | 07:55 |
| lifeless | which is why I tried git python-keystoneclient | 07:55 |
| lifeless | oh god | 07:57 |
| lifeless | I'm blind | 07:57 |
| lifeless | hang on while I make neutronclient better | 07:57 |
| SpamapS | :) | 07:57 |
| lifeless | I wish python had SSA | 07:57 |
| SpamapS | no idea what that is.. :) | 07:58 |
| lifeless | single static assignment | 07:58 |
| lifeless | look at the variable body in neutronclient/client.py line 232 | 07:59 |
| *** ramishra has joined #tripleo | 07:59 | |
| lifeless | SpamapS: https://review.openstack.org/69110 | 08:01 |
| SpamapS | lifeless: seems legit.. | 08:01 |
| SpamapS | lifeless: ok this time I really am going to go lay my head on a pillow and sleep. | 08:03 |
| lifeless | {'auth': {'tenantName': None, 'passwordCredentials': {'username': None, 'password': None}}} | 08:05 |
| lifeless | that seems broken | 08:05 |
| *** ramishra has quit IRC | 08:06 | |
| lifeless | I smell thread local breakage again | 08:07 |
| lifeless | IMBW | 08:08 |
| lifeless | ahhh | 08:08 |
| lifeless | I bet they dropped the quantum backwards compat code | 08:08 |
| *** akuznetsov has joined #tripleo | 08:09 | |
| lifeless | yup | 08:09 |
| *** ramishra has joined #tripleo | 08:09 | |
| lifeless | tripleo-cd restarted | 08:11 |
| *** ramishra has quit IRC | 08:11 | |
| lifeless | commit ee50436938eb8122185b08681803d07f16dcd3c1 | 08:12 |
| lifeless | from jog0 :( | 08:12 |
| lifeless | anyhow | 08:13 |
| lifeless | fixed | 08:13 |
| *** lsmola_ has quit IRC | 08:17 | |
| *** openstack has joined #tripleo | 08:22 | |
| *** ramishra has joined #tripleo | 08:26 | |
| *** ramishra has quit IRC | 08:30 | |
| *** csd has quit IRC | 09:04 | |
| *** e0ne has joined #tripleo | 09:06 | |
| *** CaptTofu has joined #tripleo | 09:15 | |
| *** cd-undercloud has joined #tripleo | 09:17 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 09:17 |
| *** cd-undercloud has quit IRC | 09:17 | |
| *** CaptTofu has quit IRC | 09:19 | |
| *** e0ne has quit IRC | 09:21 | |
| *** max_lobur_afk is now known as max_lobur | 10:02 | |
| *** cd-undercloud has joined #tripleo | 10:24 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 10:24 |
| *** cd-undercloud has quit IRC | 10:24 | |
| *** viktors has joined #tripleo | 10:30 | |
| *** akuznetsov has quit IRC | 10:31 | |
| *** e0ne has joined #tripleo | 10:32 | |
| *** e0ne has quit IRC | 10:34 | |
| *** boris-42 has joined #tripleo | 10:49 | |
| *** spzala has quit IRC | 10:56 | |
| *** CaptTofu has joined #tripleo | 11:15 | |
| *** CaptTofu has quit IRC | 11:20 | |
| *** ramishra has joined #tripleo | 11:28 | |
| openstackgerrit | Dirk Mueller proposed a change to openstack/tripleo-incubator: Fix openSUSE detection https://review.openstack.org/68602 | 11:28 |
| *** cd-undercloud has joined #tripleo | 11:32 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 11:32 |
| *** cd-undercloud has quit IRC | 11:32 | |
| openstackgerrit | Dirk Mueller proposed a change to openstack/tripleo-incubator: Setup libvirt/kvm permissions on openSUSE https://review.openstack.org/68794 | 11:32 |
| *** ramishra has quit IRC | 11:34 | |
| *** ramishra has joined #tripleo | 11:40 | |
| *** ramishra_ has joined #tripleo | 11:43 | |
| *** ramishr__ has joined #tripleo | 11:45 | |
| *** ramishra has quit IRC | 11:45 | |
| *** ramishr__ has quit IRC | 11:45 | |
| *** ramishra_ has quit IRC | 11:46 | |
| *** ramishra has joined #tripleo | 11:46 | |
| *** ramishra has quit IRC | 11:51 | |
| *** ramishra has joined #tripleo | 11:52 | |
| *** coolsvap has joined #tripleo | 11:54 | |
| *** ramishra has quit IRC | 11:57 | |
| *** panda has joined #tripleo | 12:06 | |
| *** panda__ has quit IRC | 12:09 | |
| *** coolsvap has quit IRC | 12:30 | |
| *** coolsvap has joined #tripleo | 12:32 | |
| *** cd-undercloud has joined #tripleo | 12:39 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 12:39 |
| *** cd-undercloud has quit IRC | 12:39 | |
| *** CaptTofu has joined #tripleo | 12:53 | |
| *** CaptTofu has quit IRC | 12:57 | |
| *** rpodolyaka has quit IRC | 12:59 | |
| *** CaptTofu has joined #tripleo | 13:04 | |
| *** coolsvap has quit IRC | 13:33 | |
| *** akuznetsov has joined #tripleo | 13:42 | |
| *** cd-undercloud has joined #tripleo | 13:45 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 13:45 |
| *** cd-undercloud has quit IRC | 13:45 | |
| *** akuznetsov has quit IRC | 13:47 | |
| *** michchap has quit IRC | 13:58 | |
| *** michchap has joined #tripleo | 13:58 | |
| *** michchap has quit IRC | 14:03 | |
| *** CaptTofu has quit IRC | 14:13 | |
| *** CaptTofu has joined #tripleo | 14:14 | |
| *** CaptTofu has quit IRC | 14:18 | |
| *** cd-undercloud has joined #tripleo | 14:51 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 14:51 |
| *** cd-undercloud has quit IRC | 14:51 | |
| *** CaptTofu has joined #tripleo | 14:51 | |
| *** ccrouch has quit IRC | 15:01 | |
| *** ccrouch has joined #tripleo | 15:02 | |
| *** ccrouch has quit IRC | 15:06 | |
| SpamapS | hm | 15:12 |
| SpamapS | getting NoValidHost for nova boots now | 15:12 |
| *** viktors has left #tripleo | 15:29 | |
| *** cody-somerville has quit IRC | 15:30 | |
| *** blamar has quit IRC | 15:45 | |
| *** cody-somerville has joined #tripleo | 15:45 | |
| *** akuznetsov has joined #tripleo | 15:46 | |
| *** cd-undercloud has joined #tripleo | 15:58 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 15:58 |
| *** cd-undercloud has quit IRC | 15:58 | |
| *** e0ne has joined #tripleo | 15:59 | |
| *** e0ne has quit IRC | 16:12 | |
| *** openstack has joined #tripleo | 16:19 | |
| *** CaptTofu has quit IRC | 16:26 | |
| *** CaptTofu has joined #tripleo | 16:27 | |
| *** CaptTofu has quit IRC | 16:31 | |
| *** max_lobur is now known as max_lobur_afk | 16:38 | |
| *** michchap has joined #tripleo | 16:40 | |
| *** michchap has quit IRC | 16:45 | |
| *** rpodolyaka1 has joined #tripleo | 16:46 | |
| *** cd-undercloud has joined #tripleo | 17:04 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 17:04 |
| *** cd-undercloud has quit IRC | 17:04 | |
| *** e0ne has joined #tripleo | 17:23 | |
| *** e0ne has quit IRC | 17:26 | |
| *** michchap has joined #tripleo | 17:41 | |
| *** michchap has quit IRC | 17:45 | |
| greghaynes | SpamapS: in filter_scheduler.py? | 17:47 |
| *** rpodolyaka1 has quit IRC | 18:04 | |
| *** e0ne has joined #tripleo | 18:06 | |
| *** cd-undercloud has joined #tripleo | 18:10 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 18:10 |
| *** cd-undercloud has quit IRC | 18:10 | |
| *** noslzzp has quit IRC | 18:11 | |
| lifeless | morning | 18:34 |
| lifeless | nova list | 18:35 |
| lifeless | : f9b7d1a3-e057-4efa-b20c-8dee27234a29] Error from last host: undercloud (node a9402861-306e-4f49-96af-789c1fab5762): [u'Traceback (most recent call last):\n', u' File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 1058, in _build_instance\n set_access_ip=set_access_ip)\n', u' File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 357, in decorated_f | 18:38 |
| lifeless | wheee sorry | 18:38 |
| *** coolsvap has joined #tripleo | 18:39 | |
| lifeless | greghaynes: SpamapS: http://paste.openstack.org/show/61887/ | 18:39 |
| greghaynes | aha, my derpd overcloud is unrelated | 18:40 |
| lifeless | to manually fix this one: | 18:40 |
| lifeless | select * from bm_nodes where uuid='a9402861-306e-4f49-96af-789c1fab5762'\G | 18:41 |
| lifeless | then check the instance_uuid | 18:41 |
| lifeless | which is usually a deleted instance | 18:41 |
| lifeless | then update the instance uuid | 18:41 |
| *** coolsvap has quit IRC | 18:42 | |
| *** michchap has joined #tripleo | 18:43 | |
| SpamapS | lifeless: so this is a baremetal bug? | 18:43 |
| lifeless | SpamapS: maybe or maybe a side effect of the other bug | 18:43 |
| lifeless | anyhow I'm going to cross reference the dbs and reconcile | 18:43 |
| SpamapS | lifeless: It's times like these where I think we need to think hard if we can just switch to Ironic, warts and all, so that we can actually move forward with fixing bugs. | 18:46 |
| SpamapS | IIRC there is just one neutron thing left and then we can reasonably think about doing that. | 18:46 |
| *** michchap has quit IRC | 18:47 | |
| lifeless | SpamapS: I think we should as soon as it has the features | 18:58 |
| lifeless | SpamapS: theres chris and ghe and deva ft on it atm | 18:58 |
| lifeless | SpamapS: I'm not worried about warts | 18:58 |
| SpamapS | lifeless: feels a lot like when juju python was "the one that works" but we couldn't actually improve it. :( | 18:59 |
| *** cody-somerville has quit IRC | 18:59 | |
| lifeless | SpamapS: well, there are some differences - we've been improving nova-bm as needed | 19:00 |
| lifeless | SpamapS: so we're not blocked, we have a protocol for signalling that a new feature is needed for ironic too | 19:00 |
| *** coolsvap has joined #tripleo | 19:00 | |
| SpamapS | lifeless: right, but time spent on baremetal is wasted. :-/ | 19:00 |
| SpamapS | lifeless: I rewrote the entire juju local provider after juju python was deprecated. Fixed most of the bugs, but felt empty. :-/ | 19:01 |
| lifeless | SpamapS: I don't entirely agree that its wasted | 19:01 |
| lifeless | whats the goal we're optimising for | 19:01 |
| devananda | there's 2 things still missing from ironic... links coming ... | 19:01 |
| lifeless | does it help that goal ? | 19:01 |
| lifeless | I don't mind us doing thing two or even three times | 19:01 |
| lifeless | if we learn something | 19:02 |
| devananda | https://review.openstack.org/#/c/66071/ | 19:02 |
| lifeless | and help something | 19:02 |
| lifeless | along the way | 19:02 |
| devananda | https://review.openstack.org/#/c/51328/9 | 19:02 |
| devananda | neutron binding & nova driver | 19:02 |
| * lifeless blinks at the config setting changes there | 19:03 | |
| devananda | auto-cnofig foo. blame oslo for that | 19:04 |
| lifeless | no | 19:04 |
| lifeless | stuff shouldn't be in there | 19:04 |
| lifeless | oh | 19:04 |
| lifeless | you're saying its just rearranged. | 19:04 |
| devananda | yes | 19:04 |
| lifeless | I see - but deploy_ramdisk=<None> | 19:04 |
| lifeless | I just wrote a review comment of | 19:04 |
| lifeless | I don't think we should put this in the config file at all - it was a mistake in nova baremetal, and it would be a mistake to replicate it. flavor works /much/ better. | 19:04 |
| lifeless | and for | 19:04 |
| lifeless | 623#net_config_template=$pybasedir/ironic/net-dhcp.ubuntu.template | 19:05 |
| lifeless | Draft saved at 8:03 AM | 19:05 |
| lifeless | We're not doing file injection, so this isn't needed. | 19:05 |
| GheRivero | hi all! | 19:05 |
| *** e0ne has quit IRC | 19:05 | |
| devananda | lifeless: oh, i thought you were talking abbout the neutron patch's conf changes | 19:06 |
| lifeless | devananda: I thought I was | 19:06 |
| devananda | lifeless: i agree re: net_config_template. but i dont see a reason to remove teh capability to have a default deploy_ramdisk | 19:06 |
| lifeless | devananda: because it doesn't make any sense | 19:07 |
| devananda | neutron patch conf changes: https://review.openstack.org/#/c/66071/11/ironic/common/neutron.py | 19:07 |
| GheRivero | net_config_template is not used anywhere in the code now | 19:07 |
| GheRivero | so +1 to remove it | 19:08 |
| devananda | GheRivero: have time this week to do a cleanup of pxe.py? | 19:08 |
| GheRivero | sure | 19:09 |
| devananda | great, thanks much | 19:09 |
| devananda | there are bunch of TODO you left in there, too :) | 19:09 |
| devananda | at least we should revisit if they are still valid TODO | 19:09 |
| GheRivero | I should have removed my name from the TODOs | 19:09 |
| lifeless | devananda: it doesn't make sense because unlike nova-bm ironic is multi-arch | 19:10 |
| devananda | eh, anyone can do them. it's more a record of who thought it should be done in the first place | 19:10 |
| lifeless | devananda: having a default there is a great way to utterly break people that start with amd64 and add an i386 machine | 19:10 |
| lifeless | devananda: because they will then be going 'wtf' 'wtf' until they learn they had been using a special case feature for $however long | 19:11 |
| devananda | lifeless: ironic is multi-driver | 19:11 |
| lifeless | devananda: it also forces config file updates when the deploy ramdisk is updated, which is much harder than API calls for deployers | 19:11 |
| lifeless | devananda: yes, but there is one 'pxe' default there, but pxe can deploy to multiple architectures | 19:12 |
| lifeless | devananda: no? | 19:12 |
| devananda | eg. for deploying to windows, the pxe driver will need to run slightly different deploy process | 19:13 |
| devananda | not jst different deply ramdisk | 19:13 |
| lifeless | devananda: I'm not talking about operating system | 19:13 |
| lifeless | devananda: same os, same deploy driver, different cpu architecture | 19:13 |
| devananda | sure | 19:13 |
| devananda | so your point is valid and not the complete answer | 19:14 |
| devananda | a default ramdisk has the potential to confuse pppl, i agree | 19:14 |
| devananda | mind filing a bug? I've got to run | 19:14 |
| devananda | it's saturday ;) | 19:14 |
| lifeless | sunday :) | 19:14 |
| lifeless | filing | 19:14 |
| devananda | ty | 19:15 |
| SpamapS | sunday bloody sunday | 19:15 |
| *** cd-undercloud has joined #tripleo | 19:17 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 19:17 |
| *** cd-undercloud has quit IRC | 19:17 | |
| lifeless | now, stack popped, where was I | 19:22 |
| lifeless | thats right, reconciling | 19:22 |
| lifeless | select uuid, instance_uuid from bm_nodes where instance_uuid not in (select uuid from nova.instances where not deleted); | 19:23 |
| lifeless | ^ useful in future | 19:23 |
| lifeless | | f9b7d1a3-e057-4efa-b20c-8dee27234a29 | test2 | BUILD | deleting | NOSTATE | ctlplane=10.10.16.171 | | 19:25 |
| lifeless | hmm | 19:25 |
| lifeless | that might have found a bad machine | 19:25 |
| lifeless | anyhow | 19:25 |
| lifeless | lets see what happens now | 19:25 |
| *** coolsvap has quit IRC | 19:29 | |
| *** michchap has joined #tripleo | 19:43 | |
| *** michchap has quit IRC | 19:43 | |
| *** michchap has joined #tripleo | 19:43 | |
| *** jcooley_ has joined #tripleo | 19:47 | |
| *** michchap has quit IRC | 19:48 | |
| *** jcooley_ has quit IRC | 19:55 | |
| *** jcooley_ has joined #tripleo | 19:55 | |
| *** jcooley_ has quit IRC | 20:00 | |
| *** jcooley_ has joined #tripleo | 20:00 | |
| *** jcooley_ has quit IRC | 20:04 | |
| *** jcooley_ has joined #tripleo | 20:05 | |
| *** panda_ has joined #tripleo | 20:17 | |
| *** panda has quit IRC | 20:20 | |
| *** jcooley_ has quit IRC | 20:21 | |
| *** jcooley_ has joined #tripleo | 20:21 | |
| *** cd-undercloud has joined #tripleo | 20:24 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 20:24 |
| *** cd-undercloud has quit IRC | 20:24 | |
| lifeless | ok, instances stuck in spawning now | 20:25 |
| lifeless | greghaynes: you said you're having a filter problem? | 20:26 |
| lifeless | greghaynes: do yo want to debug that ? | 20:26 |
| *** jcooley_ has quit IRC | 20:27 | |
| greghaynes | I have to afk for a bit but when I return yesplz :) | 20:28 |
| lifeless | ping me, if I'm here I'll be delighted to help | 20:28 |
| greghaynes | will do | 20:28 |
| *** panda_ has quit IRC | 20:29 | |
| *** e0ne has joined #tripleo | 20:31 | |
| lifeless | yay mellanox flexboot | 20:33 |
| lifeless | lets see if I managed to hit the interactive jaackpot | 20:34 |
| lifeless | SpamapS: tcpdumping br-ctlplane shows 20:35:16.903058 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 78:e7:d1:23:93:dd, length 381 | 20:36 |
| lifeless | 20:35:18.880251 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 78:e7:d1:23:93:dd, length 381 | 20:36 |
| lifeless | e.g. no response from neutron | 20:36 |
| lifeless | I've restarted the dhcp agent | 20:36 |
| lifeless | SpamapS: and am now in the ipxe command line manualyl triggering dhcp to see wtf is | 20:39 |
| lifeless | up | 20:39 |
| lifeless | next step is tcpdump in the netns | 20:39 |
| *** cd-undercloud has joined #tripleo | 20:40 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 20:40 |
| *** cd-undercloud has quit IRC | 20:40 | |
| lifeless | the tap device there isn't seeing any traffic | 20:40 |
| lifeless | which suggests a wrong or misconfigured openvswitch config | 20:41 |
| *** michchap has joined #tripleo | 20:43 | |
| lifeless | traffic isn't hitting br-int AFAICT | 20:43 |
| *** michchap has quit IRC | 20:44 | |
| lifeless | SpamapS: ah, I think you broke us with git neutron | 20:44 |
| *** michchap has joined #tripleo | 20:44 | |
| *** panda_ has joined #tripleo | 20:44 | |
| lifeless | SpamapS: rolled neutron back, booting proceeding accordingly | 20:46 |
| *** michchap has quit IRC | 20:48 | |
| lifeless | 20:50 | |
| lifeless | Trying to load: pxelinux.cfg/01-78-e7-d1-23-93-dd ok | 20:50 |
| lifeless | Loading /tftpboot/7e032507-93f9-434f-bbba-1bbb8fd02bb9/deploy_kernel... | 20:50 |
| lifeless | further along | 20:50 |
| *** cd-undercloud has joined #tripleo | 20:50 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 20:50 |
| *** cd-undercloud has quit IRC | 20:50 | |
| lifeless | and dd is off and running | 20:58 |
| lifeless | oh nuts | 20:58 |
| lifeless | 2014-01-25 20:54:47,600.600 13789 DEBUG nova.openstack.common.processutils [req-eae1f4fd-e4fc-4b85-94a4-06700e5cf4c3 None None] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf mkfs -t ext3 -F -L ephemeral0 /dev/disk/by-path/ip-10.10.16.171:3260-iscsi-iqn-7e032507-93f9-434f-bbba-1bbb8fd02bb9-lun-1-part1 execute /opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/openstack/common/processutils.py:147 | 20:58 |
| lifeless | lost the ext4 patch along the way | 20:59 |
| lifeless | SpamapS: ^ we have to do it with config now... | 20:59 |
| clarkb | about that | 20:59 |
| lifeless | clarkb: yo | 21:00 |
| clarkb | I don't know that the argument was made, but doesn't ext3 have a much greater potential for losing your data in a variety of situations common to cloud environments? | 21:00 |
| lifeless | clarkb: not really | 21:00 |
| lifeless | clarkb: 3 and 4 both journal | 21:00 |
| clarkb | hrm I thought that came up with upstart not nicely syncing on shutdown | 21:00 |
| lifeless | clarkb: if you care about data, you should set ordered to both | 21:00 |
| lifeless | clarkb: *4* had early bugs that were terrible when combined with powerloss or hard off situations | 21:01 |
| clarkb | I remember that | 21:01 |
| *** akuznetsov has quit IRC | 21:02 | |
| *** akuznetsov has joined #tripleo | 21:03 | |
| clarkb | I find the default of ext3 to be annoying because all of our rackspace nodes come with ext3 filesystems. Would be nice to see it changed | 21:03 |
| *** akuznetsov has quit IRC | 21:03 | |
| lifeless | yup | 21:03 |
| lifeless | clarkb: I feel your pain | 21:09 |
| clarkb | I know | 21:09 |
| lifeless | clarkb: you can online convert to ext4 | 21:09 |
| lifeless | ext4 so much faster :) | 21:11 |
| clarkb | and you can have more dirs in dir :) | 21:11 |
| lifeless | hmm? | 21:11 |
| clarkb | ext3 has a limit of 32k subdirectorys in one directory. ext4 doubles that | 21:11 |
| lifeless | .. isn't a hardlink anymore? | 21:11 |
| lifeless | oh, just a different compile time setting :> | 21:12 |
| lifeless | and yay, one node active | 21:12 |
| clarkb | apparnetly the numbers are related to performance things. It became important on the logs server | 21:12 |
| lifeless | two nodes active | 21:12 |
| clarkb | we worked around it by remounting ext4 which didn't do a conversion but did bump the limit | 21:13 |
| lifeless | so each subdir has a child .. which is a hardlink to it's parent | 21:13 |
| lifeless | the actual thing that changed is how many hardlinks a single inode can have | 21:13 |
| clarkb | ah | 21:13 |
| lifeless | SpamapS: ok, tripleo-cd stopped, overcloud deleted, tripleo-cd started | 21:14 |
| clarkb | but all of the elasticsearch data is on ext3 and so are the git repos. I am sure there would be noticeable impacts if we were ext4 for all of that stuff | 21:14 |
| lifeless | fsck is faster if you have extents | 21:15 |
| lifeless | but you need to convert (tune2fs enable extents) and then rewrite everything to migrate the metadata for files to fix thaat | 21:16 |
| lifeless | late allocation is (AIUI) entirely in kernel memory, so remounting gets you that | 21:16 |
| lifeless | but writing the thing is again faster if you have extents | 21:16 |
| lifeless | right, time to go do some shopping | 21:18 |
| lifeless | bbiaw | 21:18 |
| *** michchap has joined #tripleo | 21:44 | |
| *** michchap has quit IRC | 21:44 | |
| *** michchap has joined #tripleo | 21:45 | |
| SpamapS | lifeless: we landed the config patch but have not refreshed nova config yet :-P | 21:49 |
| *** michchap has quit IRC | 21:50 | |
| lifeless | done by hand | 21:52 |
| SpamapS | lifeless: \o/ :) | 21:52 |
| SpamapS | Would be great if we could somehow get ironic to facilitate 'nova console-log' :) | 21:55 |
| SpamapS | Assuming ipmi serial console is a thing we can leave on without angering the gods. | 21:55 |
| lifeless | there is a BUG | 21:57 |
| SpamapS | sshd conn refused | 21:58 |
| SpamapS | lifeless: you looking at the consoles? | 21:58 |
| *** michchap has joined #tripleo | 21:58 | |
| lifeless | no | 21:58 |
| lifeless | 2014-01-25 21:48:34,679.679 4148 INFO nova.virt.baremetal.deploy_helper [-] deployment to node 11 done | 21:58 |
| SpamapS | lifeless: used fallback data source | 22:00 |
| SpamapS | lifeless: so ssh keys not installed | 22:01 |
| lifeless | restart nova-api ? | 22:01 |
| lifeless | or log in on console and check metadata access? | 22:01 |
| SpamapS | yeah trying to do that | 22:01 |
| SpamapS | no getty on tty1 | 22:01 |
| SpamapS | or on the virtual serial | 22:02 |
| SpamapS | lifeless: worked fine for the compute node | 22:03 |
| SpamapS | or not | 22:04 |
| SpamapS | weird | 22:04 |
| SpamapS | ok saturday beckons | 22:04 |
| SpamapS | lifeless: things certainly still a little ????? around here :-P | 22:04 |
| * SpamapS goes back to afk for a while | 22:05 | |
| lifeless | ok, so I'll debug metadata server later | 22:06 |
| *** boris-42 has quit IRC | 22:16 | |
| *** cd-undercloud has joined #tripleo | 22:21 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 22:21 |
| *** cd-undercloud has quit IRC | 22:21 | |
| *** ccrouch has joined #tripleo | 22:37 | |
| *** panda has joined #tripleo | 22:56 | |
| *** panda_ has quit IRC | 22:59 | |
| *** cd-undercloud has joined #tripleo | 23:27 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 23:27 |
| *** cd-undercloud has quit IRC | 23:27 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!