| lifeless | clarkb: how is RH region looking ? | 00:00 |
|---|---|---|
| clarkb | it looks good | 00:00 |
| lifeless | SpamapS: 2 looks appealing, be good to finish the automation for it off | 00:01 |
| lifeless | SpamapS: OTOH - we ran for several weeks ok | 00:01 |
| lifeless | SpamapS: perhaps a rolling reboot of all the nodes, then we can work on the automation async ? | 00:01 |
| lifeless | SpamapS: or are we actually up and happy right now? | 00:01 |
| SpamapS | lifeless: https://review.openstack.org/#/c/88384/ | 00:01 |
| SpamapS | lifeless: I don't think we're up and happy no | 00:02 |
| lifeless | greghaynes: https://review.openstack.org/#/c/88384/ | 00:02 |
| lifeless | SpamapS: ok, trying a reboot | 00:02 |
| SpamapS | lifeless: wait.. | 00:03 |
| lifeless | SpamapS: waiting | 00:03 |
| SpamapS | lifeless: which ones are you rebooting? | 00:03 |
| lifeless | SpamapS: NovaComputeX | 00:03 |
| SpamapS | lifeless: I just booted 8 | 00:03 |
| SpamapS | 2 hours ago | 00:03 |
| SpamapS | after it froze | 00:03 |
| lifeless | ok, so are we seeing 1/9 vms working ? | 00:05 |
| *** CaptTofu has joined #tripleo | 00:05 | |
| lifeless | hypothesis : all computes are in trouble | 00:05 |
| lifeless | if 8 has one vm on it and the others have 0 working, all load will go to the others | 00:05 |
| SpamapS | 8 has one that is failing networking | 00:06 |
| SpamapS | but traffic showing on br-tun | 00:06 |
| greghaynes | +A'd.... WCPGW changing the default distro :) | 00:06 |
| SpamapS | greghaynes: :-D | 00:07 |
| SpamapS | lifeless: I take that back, 8 has vms working | 00:07 |
| lifeless | SpamapS: ok, so I'll reboot the others? | 00:08 |
| SpamapS | hm no .. weird.. eth1 has an IP (192.168.1.228) but eth0 still waiting | 00:08 |
| lifeless | SpamapS: which network is eth0 on | 00:08 |
| SpamapS | lifeless: default | 00:08 |
| lifeless | SpamapS: 192.168.1.228 should be eth1 | 00:08 |
| lifeless | the default tenant net is 10.x | 00:08 |
| SpamapS | lifeless: yeah, but eth0 is getting nothing on the VM | 00:08 |
| lifeless | SpamapS: whats the vm id ? | 00:08 |
| SpamapS | Waiting up to 60 more seconds for network configuration... | 00:08 |
| SpamapS | ea3290ef-b226-46a4-81ba-a333f5ba4b29 | 00:09 |
| SpamapS | doh | 00:09 |
| SpamapS | deleted | 00:09 |
| lifeless | so thats the nova bug with reversed ports | 00:09 |
| lifeless | its not a hypervisor problem | 00:09 |
| SpamapS | e8369267-e7c4-4665-b75a-6ef3ed3b9c03 | 00:10 |
| SpamapS | lifeless: not so sure. eth0 _did not get an ip at all_ | 00:10 |
| SpamapS | lifeless: and e8369267-e7c4-4665-b75a-6ef3ed3b9c03 is working similarly... no DHCP replies | 00:10 |
| lifeless | ok eth1 is good | 00:10 |
| lifeless | 192.168.1.x | 00:10 |
| lifeless | eth0 failed | 00:10 |
| lifeless | dhcp agent may need restarting on control | 00:10 |
| SpamapS | lifeless: restarted dnsmasq forcibly last night | 00:11 |
| SpamapS | all this reboot talk.. feels windowsy :-/ | 00:11 |
| lifeless | restarting that reglues the flows | 00:11 |
| tchaypo | So I've been revisiting the pip case-insensitivy thing; it turns out that if we add "find-links=file://tmp/pypi" to pip.conf *and* set --no-index on the commandline, the case-insensitivy happens fine | 00:12 |
| lifeless | tchaypo: because its not using the index :P | 00:12 |
| lifeless | tchaypo: has dstufft landed your patch yet? | 00:12 |
| SpamapS | ok restarting dhcp agent | 00:12 |
| lifeless | SpamapS: did that | 00:12 |
| tchaypo | it's easy to set the former in the pypi element, but I think that to get --no-index on the commandline, I need the os-svc-install element to change its commandline *only* if we're using the pypi element, and I'm not sure how to do that | 00:12 |
| SpamapS | and observing new vm on 8.. | 00:13 |
| SpamapS | b5c566c4-ac5f-4cc4-94d5-6a5c6cb659de | 00:13 |
| lifeless | b5c566c4-ac5f-4cc4-94d5-6a5c6cb659de | 00:13 |
| tchaypo | no - he asked me to add some tests, and it was in testing it out that I realised they always test with "-f file://blah --no-index", hence their tests pass | 00:13 |
| SpamapS | lifeless: ok, let's call things out here and wait 30 seconds for a nack so we don't step on eachothers' toes | 00:13 |
| lifeless | ok | 00:13 |
| lifeless | tchaypo: ok; so they have a different code path, thus they *aren't testing it* | 00:13 |
| *** epim has quit IRC | 00:13 | |
| lifeless | SpamapS: looking for fa:16:3e:95:14:2c in the dnsmasq | 00:14 |
| tchaypo | sure, and that's something I can tweak and show in my testing | 00:14 |
| SpamapS | not seeing traffic from eth0's mac | 00:14 |
| tchaypo | but in the meantime it seems like we could also tweak how we're using pip to match the tested code paths that do work | 00:14 |
| SpamapS | aaaand it's deleted again | 00:15 |
| tchaypo | if i can figure out how to make os-svc-install only add the --no-index flag when we're using hte pypi element | 00:15 |
| *** funzo has quit IRC | 00:17 | |
| SpamapS | lifeless: no I think the traffic isn't making it back | 00:17 |
| lifeless | tchaypo: but --no-index isn't the right thing for bandersnatch mirrors | 00:17 |
| tchaypo | is there a standard mechanism for passing flags between elements | 00:17 |
| SpamapS | lifeless: mellanox fail would explain this, as we saw it drop packets sporadically in the past | 00:17 |
| lifeless | tchaypo: I think we're best off just fixing it | 00:17 |
| *** funzo has joined #tripleo | 00:18 | |
| tchaypo | right - hence why we don't want to use it in all cases, only when we're using a pypi-mirror mirror (as provided by the pypi element) | 00:18 |
| *** funzo is now known as Guest15550 | 00:18 | |
| SpamapS | lifeless: so, should we try to be super awesome and roll out a trusty nova compute image via rebuild? | 00:19 |
| lifeless | tchaypo: the pypi element can point at real mirrors too | 00:19 |
| SpamapS | that sounds like the most fun | 00:19 |
| openstackgerrit | Derek Higgins proposed a change to openstack/diskimage-builder: Place /usr/lib64/ccache in PATH https://review.openstack.org/89724 | 00:19 |
| derekh | possibly one of the reason f20 jobs run slower ^^ (hopefully) | 00:20 |
| SpamapS | derekh: oh definitely | 00:20 |
| derekh | there is something else weird going on with workspace setup timings, gotta get to the bottom of that on yet | 00:22 |
| tchaypo | lifeless: good point. We actually only want to use --no-index if we're using pypi element *and* we have DIB_NOPYPI_PIP set, right? | 00:25 |
| lifeless | tchaypo: nope, not then either :) | 00:26 |
| lifeless | tchaypo: I don't actually know of a heuristic that is correct here | 00:26 |
| tchaypo | blargh. | 00:26 |
| * tchaypo goes back to looking at upstream pip | 00:26 | |
| *** matsuhashi has joined #tripleo | 00:29 | |
| vinsh | Are there certain times when os-apply-config can "see" certain values? For example, when I look at the os-collect-config.log.. I see it running "++ os-apply-config --key rabbit.nodes --type raw --key-default '' | 00:32 |
| vinsh | + NODES='overcloud-controller0-zjurfenp7u4v | 00:32 |
| vinsh | overcloud-controller1-xcpl7j5f7ysc,overcloud-controller2-figxzb7c3vsb'" | 00:32 |
| vinsh | so its getting values back from os-collect-config | 00:32 |
| vinsh | but when I manually run that command on the same node .. I get nothing. | 00:32 |
| lifeless | yeah, oac only knows the right files when its running from within occ | 00:32 |
| vinsh | d'oh | 00:33 |
| greghaynes | os-collect-config --print for great good! | 00:33 |
| greghaynes | sounds like a heat template issue, though | 00:33 |
| vinsh | oh nice.. that is what I was looking for --print .. seems only to be populated ONCE that config has started | 00:34 |
| vinsh | ah-hah.. looks like these nodes are trying to use the virtual IP to connect. maybe thats whats busted. | 00:35 |
| vinsh | or maybe tripleo. :) | 00:35 |
| lifeless | SpamapS: ok so we paused | 00:35 |
| lifeless | SpamapS: but - what next ? | 00:35 |
| *** derekh has quit IRC | 00:37 | |
| lifeless | SpamapS: I agree though, not seeing the dhcp traffic on the control plane | 00:39 |
| *** derekh has joined #tripleo | 00:43 | |
| derekh | ok, so the RH rack is at least capable of running overcloud jobs https://jenkins04.openstack.org/job/check-tripleo-overcloud-f20/143/consoleFull | 00:45 |
| derekh | but look like we aslo have some net problems to sort out https://jenkins05.openstack.org/job/check-tripleo-overcloud-f20/113/consoleFull | 00:46 |
| lifeless | derekh: that would also happen if nodepool killed it | 00:48 |
| lifeless | clarkb: are you able to tell ^ ? | 00:49 |
| clarkb | if zuul cancels the job that can happen | 00:49 |
| clarkb | the way to check is to go to that change and see if zuul reported that test | 00:49 |
| SpamapS | lifeless: ok, have to stop for family time in a few minutes | 00:49 |
| clarkb | if zuul did not report the test then you should be fine | 00:49 |
| clarkb | if it did report the test there is something to debug | 00:50 |
| lifeless | SpamapS: tag, its on me then | 00:50 |
| derekh | lifeless: yup but there seems to be a few of them, actually seems to be happening some jobs on HP also | 00:50 |
| SpamapS | lifeless: I think it's a valid course of action to build a new trusty image and deploy on it. | 00:51 |
| clarkb | if we look at the change for that particular test there is no report so I think you are fine | 00:51 |
| lifeless | derekh: right. so that being exactly 40m in on that one seems uncoicidental | 00:51 |
| lifeless | derekh: is it the same for the others? | 00:51 |
| SpamapS | lifeless: though we may have to manually use nova rebuild commands to do so | 00:51 |
| greghaynes | also, hopefully there are trusty mirrors in place already? | 00:51 |
| lifeless | derekh: clarkb: I'm speculating here but - 30m liveness check from nodepool | 00:51 |
| SpamapS | greghaynes: we'll find out soon enough :) | 00:51 |
| lifeless | + 10m timeout in jenkins for the tcp connection to error | 00:52 |
| lifeless | == 40m | 00:52 |
| clarkb | lifeless: nodepool only does the liveliness check prior to adding the node to jenkins | 00:52 |
| lifeless | clarkb: oh, I thought it checked periodically after that | 00:52 |
| clarkb | nope | 00:52 |
| greghaynes | I bet once that patch merges were going to have a lot of people asking about using DIB_RELEASE until they get a trusty mirror up.... | 00:52 |
| clarkb | once it is able to assert the slave is ready it hands it completely off to jenkins | 00:52 |
| SpamapS | greghaynes: hence my preference to just have a local squid. :P | 00:53 |
| SpamapS | not as fast, but certainly effective | 00:53 |
| greghaynes | yep, thats valid. Fortunately I setup a trusty mirror over the weekend | 00:54 |
| *** TravT has quit IRC | 00:54 | |
| tchaypo | wooo | 00:54 |
| tchaypo | first ever full devtest.sh run on my laptop | 00:54 |
| greghaynes | No parts melted? | 00:54 |
| tchaypo | 2279s total, 1471s of which was overcloud, even with -c | 00:54 |
| SpamapS | errr | 00:55 |
| lifeless | clarkb: I believe you are wrong. | 00:55 |
| SpamapS | tchaypo: so you already have overcloud images? | 00:55 |
| lifeless | clarkb: start at def _doPeriodicCheck(self): | 00:55 |
| derekh | lifeless: timings seem different in them all, but I havn't seen any reported back to jenkins so maybe we just wait and see what zuul does | 00:55 |
| clarkb | lifeless: does it do that for used nodse? | 00:55 |
| lifeless | clarkb: every node in READY state | 00:56 |
| clarkb | lifeless: ya so no used | 00:56 |
| clarkb | which nodes running tests should be | 00:56 |
| clarkb | I suppose it is possible the nodes didn't transition state properly | 00:57 |
| lifeless | clarkb: handleStartPhase looks odd to me | 00:58 |
| lifeless | clarkb: whats the early return on line 196 for ? | 00:58 |
| *** matsuhashi has quit IRC | 00:59 | |
| clarkb | lifeless: nodepool runs a quick test job on the node via jenkins to make sure jenkins si speaking to it properly | 00:59 |
| clarkb | s/nodepool/zuul/ | 00:59 |
| clarkb | in cases where nodes derp iirc | 00:59 |
| lifeless | oh, I see | 01:00 |
| tchaypo | SpamapS: I'm farily certain i already had the images that time | 01:00 |
| tchaypo | the previous run had time wout on the wait_for stage, so I'm fairly sure it had made the images | 01:00 |
| lifeless | clarkb: so its not 'handed to jenkins' | 01:00 |
| lifeless | clarkb: its 'chosen by jenkins for a job' | 01:00 |
| clarkb | lifeless: nodepool hands the salve to jenkins | 01:00 |
| clarkb | then jenkins does things with it | 01:01 |
| *** matsuhashi has joined #tripleo | 01:01 | |
| lifeless | clarkb: yes, but it stays READY until jenkins does something to it | 01:01 |
| clarkb | yes | 01:01 |
| vinsh | Should one overcloud node be able to ping/lookup another overcloud node by hostname only? | 01:02 |
| lifeless | vinsh: compute -> compute yes | 01:02 |
| lifeless | vinsh: we haven't precalculated the hosts content for the control plane yet | 01:03 |
| vinsh | ACK. what I wondered. thx. | 01:03 |
| vinsh | probably a big next step to getting any of this clustering of rabbit or whatever working :) | 01:03 |
| lifeless | we put the ips for rabbit etc in the rabbit config | 01:04 |
| vinsh | uses hostnames | 01:04 |
| vinsh | https://github.com/openstack/tripleo-image-elements/commit/f23fba2db4fa60dc9b7798bce0dfc99b5a69601a | 01:04 |
| lifeless | ahahahahahahaahahahahaahah. | 01:04 |
| vinsh | 0_o | 01:05 |
| lifeless | so the same evil string manipulation we do on the hypervisors needs to be done on the compute set | 01:05 |
| lifeless | now this is odd, cd-undercloud just went under | 01:05 |
| lifeless | I have to wonder if there's some network condition that just takes out mellanox drivers across the board | 01:05 |
| lifeless | rebooting the undercloud | 01:06 |
| openstackgerrit | A change was merged to openstack/diskimage-builder: Small fixes for dhcp-all-interfaces https://review.openstack.org/88131 | 01:07 |
| greghaynes | oh, so the issue is the hostnames for control nodes arent in /etc/hosts so rabbit is getting mad about lookiung each other up by hostname? | 01:09 |
| greghaynes | womp womp | 01:09 |
| vinsh | yeah. I just populated /etc/hosts on a controll and rabbit is now happy | 01:10 |
| vinsh | so. how to do that with the thing that does the stuff. | 01:10 |
| derekh | clarkb: lifeless: I gotta call it a night, looks like we have the same error with jobs on both clouds, I can keep looking in the morning if its still an issue, of the ones I have checked nothing has been reported back to gerrit so I'm kind of hoping zuul will rerun them | 01:11 |
| lifeless | derekh: could be a rackspace network issue | 01:12 |
| lifeless | vinsh: ok so look at overcloud-source.yaml | 01:13 |
| greghaynes | vinsh: see line 451 in overcloud-source.yaml | 01:13 |
| lifeless | vinsh: see StaticHosts | 01:13 |
| greghaynes | wah? | 01:13 |
| derekh | lifeless: possibly, a lot of jobs just failed | 01:13 |
| greghaynes | I think that might of changed with software-config | 01:13 |
| lifeless | vinsh: see how there is a Merge::Map across NovaCompute0 | 01:13 |
| greghaynes | ogod | 01:14 |
| lifeless | vinsh: we need to do two things; we need to convert the controller entry at the bottom to a Merge::Map as well | 01:14 |
| lifeless | vinsh: and secondly we need to use StaticHosts on the controllers as well | 01:14 |
| lifeless | vinsh: which is a related but separate thing | 01:14 |
| *** derekh has quit IRC | 01:14 | |
| * vinsh parsing | 01:15 | |
| *** mestery has quit IRC | 01:15 | |
| greghaynes | I dont actually see where any element uses StaticHosts | 01:15 |
| greghaynes | the hosts element uses {{hosts}} | 01:15 |
| vinsh | I follow now. Had to look around as my overcloud-source is tricked out. | 01:17 |
| lifeless | greghaynes: static_hosts: {Ref: StaticHosts} | 01:17 |
| greghaynes | ah, in nova-compute. I still dont see any element that actually uses that data though | 01:18 |
| vinsh | Isn't he saying thats what populates /etc/hosts? | 01:19 |
| lifeless | greghaynes: then follow the needle | 01:19 |
| lifeless | greghaynes: grep for static_hosts | 01:19 |
| * greghaynes headdesk | 01:19 | |
| *** eguz has joined #tripleo | 01:20 | |
| *** eguz has quit IRC | 01:20 | |
| vinsh | pure wizardry. :) | 01:22 |
| clarkb | git grep for extra win | 01:22 |
| greghaynes | heh, too bad that doesnt work cross repo :p | 01:22 |
| clarkb | `find` says lolwut | 01:23 |
| greghaynes | so the mixup is that the controller deployment resources dont use that | 01:23 |
| greghaynes | they have a hosts: fn::join defined in overcloud-source.yaml | 01:23 |
| greghaynes | which I think means your fix will let compute nodes talk to controller nodes, but not controller nodes to each other | 01:24 |
| *** eghobo has quit IRC | 01:24 | |
| openstackgerrit | lifeless proposed a change to openstack/tripleo-heat-templates: Use the same StaticHosts on the control plane. https://review.openstack.org/89732 | 01:24 |
| lifeless | greghaynes: ^ | 01:24 |
| vinsh | woah | 01:25 |
| greghaynes | haha, that works :) | 01:25 |
| lifeless | greghaynes: as I said, 'whole other thing' :P | 01:25 |
| *** mestery has joined #tripleo | 01:26 | |
| lifeless | the CloudName ref probably wants to point at the VIP eventually | 01:26 |
| vinsh | how does the way you wrote that differ from " hosts: | 01:27 |
| vinsh | get_input: static_hosts" | 01:27 |
| vinsh | oh. template value vs not | 01:27 |
| lifeless | ah, I think my patch might not work | 01:28 |
| lifeless | iterating | 01:28 |
| *** vinsh is now known as vinsh_brb | 01:34 | |
| greghaynes | I wonder if that one was intended as a list of purely controller hosts | 01:40 |
| greghaynes | which is a need to have :/ | 01:40 |
| openstackgerrit | lifeless proposed a change to openstack/tripleo-heat-templates: Scale the control plane in hosts files. https://review.openstack.org/89732 | 01:41 |
| *** nosnos has joined #tripleo | 01:41 | |
| lifeless | greghaynes: so the original idea was we'd put this in a variable aka parameter | 01:41 |
| lifeless | greghaynes: but then that broke in heat | 01:41 |
| lifeless | greghaynes: so, sadface headdesk it has to be copied around | 01:41 |
| greghaynes | whats especially fun is that rabbit/galera are going to want a list of just controller hosts, and the galera one wants to be comma delimited... | 01:46 |
| greghaynes | which really exacerbates the badness of that pattern | 01:46 |
| greghaynes | -> dinner | 01:50 |
| lifeless | ok this is weird | 02:04 |
| lifeless | undercloud host is now unhappy :( | 02:04 |
| openstackgerrit | Steve Kowalik proposed a change to openstack/os-cloud-config: Add a register-nodes command and associated API https://review.openstack.org/84933 | 02:05 |
| *** jpeeler has quit IRC | 02:13 | |
| *** marun has quit IRC | 02:26 | |
| lifeless | ok wow this rot just goes all the way to the core | 02:28 |
| lifeless | the seed vm host is having network pauses now | 02:28 |
| lifeless | stopping the undercloud reboot working | 02:28 |
| lifeless | upgrading the seed vm host machine to trusty | 02:40 |
| *** chuckC has joined #tripleo | 02:41 | |
| *** vinsh_brb is now known as vinsh | 02:43 | |
| *** tzumainn has quit IRC | 02:44 | |
| tchaypo | SpamapS: fwiw, ran again, same timings | 02:44 |
| tchaypo | so i think that's going to be fairly stable | 02:44 |
| *** chuckC has quit IRC | 02:45 | |
| openstackgerrit | A change was merged to openstack/diskimage-builder: Make "trusty" (Ubuntu 14.04) the default release https://review.openstack.org/88384 | 02:46 |
| *** chuckC has joined #tripleo | 02:47 | |
| *** chuckC has quit IRC | 02:48 | |
| *** chuckC has joined #tripleo | 02:49 | |
| vinsh | ! | 02:49 |
| *** chuckC has quit IRC | 02:50 | |
| vinsh | lifeless, " so the original idea was we'd put this in a variable aka parameter" vs "so, sadface headdesk it has to be copied around" | 02:51 |
| vinsh | does "copied around" mean.. using os-apply-config and writing to file? | 02:51 |
| *** chuckC has joined #tripleo | 02:51 | |
| lifeless | vinsh: no - see my patch, it shows | 02:52 |
| vinsh | wondering how that is different to a parameter.. to the over cloud node | 02:52 |
| vinsh | ah | 02:52 |
| tchaypo | does this mean my apt-mirror only needs trusty now - no more saucy, no precise, unless I'm doing something special? | 02:52 |
| vinsh | big stubbed out sections | 02:52 |
| lifeless | tchaypo: ai ai | 02:52 |
| tchaypo | woo | 02:52 |
| lifeless | vinsh: multiple copies of the same expression just in different places | 02:53 |
| * tchaypo freeds up disk space | 02:53 | |
| vinsh | Agree, makes human parsing even worse. | 02:53 |
| vinsh | I need to view the heat templates in some kind of editor that can contract and expand yaml tags :) | 02:53 |
| lifeless | vinsh: like vim ? :) | 02:55 |
| *** untriaged-bot has joined #tripleo | 03:00 | |
| untriaged-bot | Untriaged bugs so far: | 03:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1302881 | 03:00 |
| uvirtbot | Launchpad bug 1302881 in tripleo "incloud CIDR can overlap custom baremetal-network" [Undecided,Incomplete] | 03:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1309027 | 03:00 |
| *** untriaged-bot has quit IRC | 03:00 | |
| uvirtbot | Launchpad bug 1309027 in tripleo "rabbitmq nodes repeatedly reporting the master node as being 'down' then 'up'" [Undecided,New] | 03:00 |
| *** cody-somerville has quit IRC | 03:02 | |
| *** openstackgerrit has quit IRC | 03:04 | |
| *** openstackgerrit has joined #tripleo | 03:04 | |
| *** matsuhashi has quit IRC | 03:14 | |
| *** matsuhashi has joined #tripleo | 03:15 | |
| *** chuckC has quit IRC | 03:16 | |
| *** cody-somerville has joined #tripleo | 03:17 | |
| *** cody-somerville has joined #tripleo | 03:17 | |
| greghaynes | lifeless: For the control node vip - should we make a new subnet with neutron? | 03:19 |
| *** matsuhashi has quit IRC | 03:19 | |
| lifeless | greghaynes: new port yes, subnet no | 03:19 |
| greghaynes | just put it on the 192.0.2? | 03:20 |
| lifeless | greghaynes: we'll need to accept the subnet as a parameter to the template so heat can ask neutron for the ip | 03:20 |
| greghaynes | ah. So basically its someone elses problem where it goes | 03:20 |
| lifeless | for the seed we can manually choose a new ip | 03:20 |
| *** Rakesh6 has joined #tripleo | 03:23 | |
| openstackgerrit | A change was merged to openstack/tripleo-image-elements: Wire in _EXTRA_INSTALL_OPTS... https://review.openstack.org/76966 | 03:24 |
| openstackgerrit | A change was merged to openstack/tripleo-image-elements: Fixup testenv config for interface names. https://review.openstack.org/84326 | 03:24 |
| greghaynes | oo looks like trusty got merged while I was at dinner | 03:24 |
| tchaypo | nice desert? | 03:28 |
| greghaynes | for now | 03:28 |
| tchaypo | I'm rebuilding without -c this time ;) | 03:28 |
| *** Guest15550 has quit IRC | 03:29 | |
| *** nosnos has quit IRC | 03:29 | |
| *** funzo has joined #tripleo | 03:29 | |
| *** funzo is now known as Guest37154 | 03:30 | |
| *** spzala has quit IRC | 03:33 | |
| *** Guest37154 has quit IRC | 03:34 | |
| StevenK | I spent a few hours rebuilding Cinnamon 2.2 for Trusty last night, so I might be able to actually upgrade my laptop soon | 03:36 |
| *** ramishra has joined #tripleo | 03:36 | |
| StevenK | I'll be out for about an hour in ten minutes, I am visiting the vampires. | 03:36 |
| *** cwolferh has quit IRC | 03:38 | |
| *** cwolferh has joined #tripleo | 03:47 | |
| *** eghobo has joined #tripleo | 03:48 | |
| *** rwsu has quit IRC | 03:50 | |
| vinsh | lifeless, Do you do yaml folding in vim? Makes sense but I can't find solid info on how to add it. Just basic folds so far | 03:58 |
| lifeless | vinsh: I don't, but I've seen vim fold just about anything :) | 03:58 |
| vinsh | lots of home brew projects out there with yaml syntax for vim.. I suppose one of those can be used to specify the folding syntax | 03:59 |
| clarkb | vinsh: you can zf | 04:02 |
| *** ilives has joined #tripleo | 04:10 | |
| *** eghobo has quit IRC | 04:10 | |
| *** eghobo has joined #tripleo | 04:10 | |
| SpamapS | vinsh: fdm=indent works perfectly for yaml | 04:10 |
| SpamapS | lifeless: "upgrading the seed vm host to trusty" ... | 04:12 |
| SpamapS | lifeless: http://cdn.memegenerator.net/instances/500x/43836098.jpg | 04:13 |
| *** eguz has joined #tripleo | 04:13 | |
| *** killer_prince has quit IRC | 04:15 | |
| *** eghobo has quit IRC | 04:17 | |
| *** matsuhashi has joined #tripleo | 04:19 | |
| *** nosnos has joined #tripleo | 04:22 | |
| *** ramishra has quit IRC | 04:28 | |
| vinsh | SpamapS, overcloud-source.yaml.. : set fdm=indent.. zf gives me E350: Cannot create fold with current 'foldmethod' | 04:29 |
| * vinsh vim's wrong | 04:29 | |
| *** ramishra_ has joined #tripleo | 04:30 | |
| *** funzo has joined #tripleo | 04:30 | |
| *** funzo is now known as Guest69699 | 04:30 | |
| *** Guest69699 has quit IRC | 04:34 | |
| *** cwolferh_ has joined #tripleo | 04:36 | |
| *** tserong has quit IRC | 04:38 | |
| *** cwolferh has quit IRC | 04:39 | |
| *** andreaf has joined #tripleo | 04:48 | |
| *** rha has quit IRC | 04:48 | |
| *** andreaf_ has joined #tripleo | 04:49 | |
| openstackgerrit | Jufang Wang proposed a change to openstack/tripleo-incubator: configure keystone with apache https://review.openstack.org/89742 | 04:51 |
| *** killer_prince has joined #tripleo | 04:52 | |
| *** andreaf has quit IRC | 04:53 | |
| *** akuznetsov has joined #tripleo | 04:54 | |
| *** rha has joined #tripleo | 04:58 | |
| openstackgerrit | Jufang Wang proposed a change to openstack/tripleo-image-elements: configure keystone with apache https://review.openstack.org/89744 | 04:59 |
| *** tserong has joined #tripleo | 05:03 | |
| *** tserong has joined #tripleo | 05:03 | |
| * StevenK returns, unkilled by the vampires. | 05:05 | |
| *** nati_ueno has quit IRC | 05:16 | |
| *** cwolfe__ has joined #tripleo | 05:21 | |
| tchaypo | Vampires? | 05:22 |
| tchaypo | Donationing blood? | 05:22 |
| StevenK | tchaypo: Yeah | 05:22 |
| *** cwolferh_ has quit IRC | 05:24 | |
| *** funzo has joined #tripleo | 05:30 | |
| *** funzo is now known as Guest45155 | 05:31 | |
| *** Guest45155 has quit IRC | 05:35 | |
| *** matsuhashi has quit IRC | 05:37 | |
| *** matsuhashi has joined #tripleo | 05:37 | |
| *** matsuhashi has quit IRC | 05:43 | |
| *** matsuhashi has joined #tripleo | 05:44 | |
| openstackgerrit | Om Kumar proposed a change to openstack/diskimage-builder: Refactor code to select boot kernel https://review.openstack.org/79873 | 05:46 |
| *** matsuhas_ has joined #tripleo | 05:50 | |
| vinsh | lifeless, I wonder about patch 89732 ... should the control node section be renamed to "static_hosts" from "hosts" | 05:51 |
| vinsh | using your patch.. I get a different result everytime I try to send the template to heat | 05:51 |
| vinsh | ERROR: Property error : NovaCompute0Config: input_values "Fn::Join" must operate on a list | 05:52 |
| vinsh | ERROR: Property error : controller0Deployment: input_values "Fn::Join" must operate on a list | 05:52 |
| vinsh | it just keeps cycling through them. | 05:52 |
| *** matsuhashi has quit IRC | 05:53 | |
| vinsh | does use of the merge map require a build of the template first? I'm just sending a version to heat that is standalone overcloud.. no template values in it. | 05:54 |
| *** cwolfe__ has quit IRC | 05:56 | |
| *** cwolfe__ has joined #tripleo | 05:56 | |
| *** matsuhas_ has quit IRC | 05:58 | |
| *** matsuhashi has joined #tripleo | 05:58 | |
| *** jtomasek has joined #tripleo | 06:00 | |
| *** ilives has quit IRC | 06:07 | |
| *** ilives has joined #tripleo | 06:07 | |
| *** jtomasek has quit IRC | 06:17 | |
| *** nati_ueno has joined #tripleo | 06:19 | |
| *** cwolferh_ has joined #tripleo | 06:19 | |
| *** cwolfe__ has quit IRC | 06:22 | |
| lifeless | vinsh: you need to run make overcloud.yaml and then stack-create with appropriate parameters | 06:23 |
| vinsh | oh, I have to adopt this to this big static 3 control node yaml I got from Tom H | 06:24 |
| vinsh | calls out control0-2 deployment | 06:24 |
| vinsh | so.. the merge map stuff doesn't fly. | 06:24 |
| lifeless | vinsh: the merge::map should replace that | 06:24 |
| vinsh | Ok, i'll have to look into that more then. | 06:26 |
| vinsh | does your patch depend on another one that scales out more then one control node? | 06:28 |
| *** ifarkas has joined #tripleo | 06:28 | |
| vinsh | bah. I'll mess with it more. see what I can get :) | 06:28 |
| *** funzo has joined #tripleo | 06:29 | |
| *** funzo is now known as Guest41031 | 06:30 | |
| *** Guest41031 has quit IRC | 06:34 | |
| lifeless | vinsh: my patch is part of such scaling | 06:35 |
| vinsh | Cool, I got this to work in the yaml I have.. by duplicating the old way compute nodes worked. in static_hosts but in "hosts" for control nodes. I now have an populated /etc/hosts and happy rabbit on control nodes | 06:36 |
| vinsh | I need to get back to master and pull in your change.. try that method. | 06:37 |
| StevenK | lifeless: So, something about PyCon. Do you think a TripleO talk would work for the audience? | 06:40 |
| *** eguz has quit IRC | 06:42 | |
| *** lazy_prince has joined #tripleo | 06:44 | |
| *** lsmola has joined #tripleo | 06:48 | |
| *** jprovazn has joined #tripleo | 06:51 | |
| openstackgerrit | Om Kumar proposed a change to openstack/tripleo-image-elements: Adds local storage Boot support in case PXE boot fails. https://review.openstack.org/79289 | 06:58 |
| SpamapS | StevenK: a talk about python packaging and how it relates to things like TripleO might resonate. | 06:58 |
| *** viktors_away is now known as viktors | 06:59 | |
| SpamapS | lifeless: where are we at? Can't seem to find something in cloud-outage | 07:00 |
| StevenK | SpamapS: Hm. How we prefer git HEAD and not packaging? | 07:01 |
| SpamapS | StevenK: tox, dependencies, venvs, etc. | 07:01 |
| StevenK | SpamapS: Sorry, I donated blood today, and my brain isn't sparking connections. A little more detail with small words, please? | 07:02 |
| *** cwolferh_ has quit IRC | 07:02 | |
| *** jtomasek has joined #tripleo | 07:03 | |
| SpamapS | StevenK: We are running a high scale python service from git using pip and virtualenv for installation, and tox as a test harness abstraction layer.. I think other python shops would be interested to hear how that works, and what we think might need to change. | 07:03 |
| SpamapS | StevenK: http://xkcd.com/1133/ | 07:03 |
| SpamapS | "this end should point to the ground if you want to go to space" ... so good | 07:04 |
| StevenK | Haha | 07:04 |
| SpamapS | "if it starts pointing toward space, you are having a problem and you will not go to space today" | 07:05 |
| StevenK | Bad problem :-P | 07:06 |
| openstackgerrit | Matthew Gilliard proposed a change to openstack/tripleo-incubator: Remove quotas on seed and undercloud's nova https://review.openstack.org/87126 | 07:06 |
| StevenK | SpamapS: However, that is a good plan | 07:07 |
| *** jcoufal has joined #tripleo | 07:08 | |
| StevenK | I was going to submit a talk proposal tomorrow | 07:08 |
| lifeless | SpamapS: sorry | 07:11 |
| lifeless | SpamapS: ETHINGS | 07:11 |
| lifeless | SpamapS: I've updated the seed vm host to trusty | 07:11 |
| lifeless | SpamapS: now trying to reboot the undercloud again | 07:11 |
| *** pblaho has joined #tripleo | 07:15 | |
| *** nati_ueno has quit IRC | 07:16 | |
| *** martyntaylor has joined #tripleo | 07:18 | |
| openstackgerrit | Juerg Haefliger proposed a change to openstack/diskimage-builder: Add sysv support to elements/dhcp-all-interfaces https://review.openstack.org/86299 | 07:23 |
| *** mrunge has joined #tripleo | 07:23 | |
| *** sdake_ has quit IRC | 07:25 | |
| *** ilives has quit IRC | 07:29 | |
| *** ilives has joined #tripleo | 07:30 | |
| *** funzo has joined #tripleo | 07:30 | |
| *** funzo is now known as Guest59931 | 07:30 | |
| *** Guest59931 has quit IRC | 07:34 | |
| *** morganfainberg is now known as morganfainberg_Z | 07:38 | |
| jprovazn | greghaynes: hi, you still around? | 07:46 |
| greghaynes | jprovazn: Hey, yep | 07:46 |
| greghaynes | jprovazn: Didnt get a WIP up for you, sorry :( | 07:46 |
| greghaynes | turns out theres a lot I dont understand about how we do networking | 07:46 |
| *** ilives has quit IRC | 07:47 | |
| jprovazn | greghaynes: np, I'll pick something else if help is not needed | 07:47 |
| lxsli | greghaynes: mind taking a second look at https://review.openstack.org/#/c/86314/ please? | 07:47 |
| greghaynes | jprovazn: Sounds good. Id kind of like to learn how this works :) | 07:48 |
| jprovazn | greghaynes: great! good luck ;) | 07:48 |
| *** ilives has joined #tripleo | 07:53 | |
| openstackgerrit | Alexis Lee proposed a change to openstack/diskimage-builder: Sort rhel/bin/map-packages https://review.openstack.org/89765 | 07:53 |
| greghaynes | lxsli: updated | 07:53 |
| openstackgerrit | Alexis Lee proposed a change to openstack/diskimage-builder: Map openjdk-7-jre to RHEL+SUSE https://review.openstack.org/89531 | 07:54 |
| *** nati_ueno has joined #tripleo | 07:55 | |
| lxsli | greghaynes: thanks! I just split 89531 as requested | 07:56 |
| marios | greghaynes: hey - some comments @ /#/c/83296/12 | 07:56 |
| greghaynes | lxsli: Actually, what do you think of switching to the openjdk-7-jre-headless? | 07:57 |
| greghaynes | itll mess up your package mappings (sorry) but then we dont pull in all the X libs as dependencies | 07:57 |
| lxsli | greghaynes: hmm didn't know about that, sure sounds fine. Give me 2 mins | 07:58 |
| marios | greghaynes: thanks for doing that so quickly | 07:58 |
| SpamapS | lifeless: I don't see either screen shared session active. Anything going on? | 07:59 |
| lifeless | SpamapS: I'm on the seed, networking is off somehow | 07:59 |
| openstackgerrit | Dmitry Shulyak proposed a change to openstack/tripleo-incubator: Add virtual ip create to devtest_overcloud.sh https://review.openstack.org/89613 | 08:01 |
| *** derekh has joined #tripleo | 08:02 | |
| rpodolyaka | morning tripleo | 08:02 |
| lxsli | greghaynes: know offhand if there's a yum pkg for that? Can't find one on rpmfind.net | 08:02 |
| marios | o/ roman | 08:03 |
| greghaynes | lxsli: no idea :/ | 08:03 |
| greghaynes | lxsli: actually java-1.7.0-openjdk-headless | 08:04 |
| greghaynes | for fedora | 08:04 |
| lxsli | ... well how about that | 08:04 |
| lxsli | thanks | 08:04 |
| dshulyak | morning | 08:05 |
| *** tzumainn has joined #tripleo | 08:06 | |
| StevenK | greghaynes: 1.7.0. Because that isn't confusing *at all*. | 08:06 |
| greghaynes | hehe | 08:07 |
| greghaynes | marios: replied | 08:07 |
| *** vinsh is now known as vinsh_zzzz | 08:07 | |
| dshulyak | greghaynes: hi, can you specify what concerns do you have about this one https://review.openstack.org/#/c/89517/ ? i've responded in comments | 08:07 |
| StevenK | [119536.610559] traps: enlightenment_f[10264] general protection ip:7fd45b67f5c6 sp:7fffe069f568 error:0 in libc-2.17.so[7fd45b546000+1bd000] | 08:07 |
| StevenK | Handy | 08:07 |
| greghaynes | so many reviews ;) | 08:07 |
| greghaynes | dshulyak: So if that isnt set we have changed the line from being "bind *:someport" to "bind :someport" | 08:08 |
| greghaynes | Or maybe that * is superfluous? | 08:09 |
| dshulyak | yeah, and :someport is valid syntax | 08:09 |
| greghaynes | ah, sorry then | 08:10 |
| proffalken | morning all, who do I have to sleep with to get https://review.openstack.org/#/c/87223/ and https://review.openstack.org/#/c/87226/ merged? ;) | 08:10 |
| dshulyak | nvm, thanks for reviewing ) | 08:10 |
| StevenK | proffalken: If you have to ask ... | 08:11 |
| StevenK | :-P | 08:11 |
| proffalken | StevenK: ah, ok, I didn't realise it was such an exclusive group... :P | 08:12 |
| proffalken | "I won't belong to any club that will accept people like me as a member..." | 08:12 |
| openstackgerrit | Alexis Lee proposed a change to openstack/diskimage-builder: Map openjdk-7-jre-headless to RHEL+SUSE https://review.openstack.org/89531 | 08:13 |
| lifeless | SpamapS: sorry, more calls :/ | 08:19 |
| *** ilives has quit IRC | 08:20 | |
| *** jistr has joined #tripleo | 08:21 | |
| dshulyak | also there is a couple of patches with support for overcloud vip (https://review.openstack.org/#/c/89556/ , https://review.openstack.org/#/c/89613/) | 08:22 |
| *** jang1 has joined #tripleo | 08:22 | |
| openstackgerrit | Alexis Lee proposed a change to openstack/tripleo-image-elements: Add elasticsearch element https://review.openstack.org/86316 | 08:22 |
| openstackgerrit | Alexis Lee proposed a change to openstack/tripleo-image-elements: Add openjdk-7-jre-headless element https://review.openstack.org/86314 | 08:22 |
| openstackgerrit | Alexis Lee proposed a change to openstack/tripleo-image-elements: Add logstash element https://review.openstack.org/86315 | 08:22 |
| derekh | jobs still failing, looks like each time they remain in the queue to be restarted | 08:22 |
| *** ilives has joined #tripleo | 08:22 | |
| greghaynes | dshulyak: Nice! | 08:23 |
| lxsli | greghaynes: done | 08:23 |
| *** matsuhashi has quit IRC | 08:27 | |
| *** e0ne has joined #tripleo | 08:30 | |
| *** funzo has joined #tripleo | 08:31 | |
| *** funzo is now known as Guest89693 | 08:31 | |
| *** ramishra_ has quit IRC | 08:32 | |
| *** ramishra has joined #tripleo | 08:32 | |
| lifeless | derekh: just repairing the seed; not getting any traffic across brbm to vnet1 (the seed eth1) | 08:34 |
| *** matsuhashi has joined #tripleo | 08:34 | |
| *** Guest89693 has quit IRC | 08:35 | |
| derekh | lifeless: the seed on the HP rack? | 08:35 |
| lifeless | yes | 08:35 |
| derekh | lifeless: anything I can do or should I just keep trying to figure out if I can get to the bottom of the disconnect errors | 08:36 |
| *** ramishra has quit IRC | 08:37 | |
| lifeless | derekh: divide and conquer - leave this with me | 08:39 |
| derekh | k | 08:40 |
| *** lucasagomes has joined #tripleo | 08:40 | |
| *** BadCub has quit IRC | 08:41 | |
| *** giulivo has joined #tripleo | 08:42 | |
| openstackgerrit | Petr Blaho proposed a change to openstack/tripleo-incubator: Removes forgotten #nodoc from devtest_testenv.sh https://review.openstack.org/89774 | 08:42 |
| *** ilives has quit IRC | 08:42 | |
| *** ilives has joined #tripleo | 08:43 | |
| *** bauzas has joined #tripleo | 08:48 | |
| lxsli | SpamapS: replied to your comments here https://etherpad.openstack.org/p/oac-header | 08:50 |
| *** ilives has quit IRC | 08:50 | |
| *** ramishra has joined #tripleo | 08:51 | |
| lxsli | SpamapS: split out https://etherpad.openstack.org/p/oac-retemplate , this or a similar capability is essential for me | 08:52 |
| openstackgerrit | Nicholas Randon proposed a change to openstack/tripleo-incubator: SSH key and virtual_power_driver not used on H/W https://review.openstack.org/83770 | 08:52 |
| *** ilives has joined #tripleo | 08:52 | |
| openstackgerrit | Cian O'Driscoll proposed a change to openstack/tripleo-image-elements: Store ssh host keys on ephemeral partition https://review.openstack.org/89529 | 08:53 |
| gilliard | Any chance of reviews on https://review.openstack.org/83770 please? It's nearly a month old and we need it (or similar) to work on baremetal. | 08:57 |
| *** untriaged-bot has joined #tripleo | 09:00 | |
| untriaged-bot | Untriaged bugs so far: | 09:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1302881 | 09:00 |
| uvirtbot | Launchpad bug 1302881 in tripleo "incloud CIDR can overlap custom baremetal-network" [Undecided,Incomplete] | 09:00 |
| *** untriaged-bot has quit IRC | 09:00 | |
| lifeless | gilliard: its not suitable for landing yet | 09:00 |
| lifeless | gilliard: my comments going back to patch set 8 seem to be unaddressed. | 09:01 |
| gilliard | lifeless: you would prefer 2 patchsets? | 09:02 |
| lifeless | gilliard: there seem to be two very different changes here | 09:03 |
| *** rcarrillocruz has joined #tripleo | 09:03 | |
| *** rcarrillocruz1 has quit IRC | 09:05 | |
| lifeless | gilliard: one of which is at best unneeded, at worst fundamentally wrong. | 09:06 |
| lifeless | more details put in the review | 09:08 |
| *** rcarrillocruz has quit IRC | 09:08 | |
| gilliard | lifeless: Thanks. Will go read & digest. | 09:09 |
| *** e0ne_ has joined #tripleo | 09:12 | |
| *** bauzas has quit IRC | 09:12 | |
| *** bauzas has joined #tripleo | 09:13 | |
| *** jcoufal has quit IRC | 09:14 | |
| *** e0ne has quit IRC | 09:15 | |
| *** akrivoka has joined #tripleo | 09:19 | |
| openstackgerrit | A change was merged to openstack/tripleo-incubator: Make time out 60 mins like the Heat default. https://review.openstack.org/87118 | 09:24 |
| lifeless | ok undercloud booted again. whew. | 09:27 |
| ccorrigan | SpamapS: do you think my approach for reset-db change is okay, or should I allow the original defaults shine through (db-password) if the os-apply-config doesn't have a dsn entry for any of the dbs ? | 09:27 |
| ccorrigan | context: - https://review.openstack.org/#/c/88340 | 09:28 |
| *** funzo has joined #tripleo | 09:32 | |
| *** funzo is now known as Guest29012 | 09:32 | |
| *** ilives has quit IRC | 09:34 | |
| openstackgerrit | Radomir Dopieralski proposed a change to openstack/tuskar-ui: Include default configuration for Sphinx https://review.openstack.org/89780 | 09:34 |
| *** Guest29012 has quit IRC | 09:36 | |
| *** hashar has joined #tripleo | 09:37 | |
| openstackgerrit | Ana Krivokapic proposed a change to openstack/tuskar-ui: Improve developer documentation https://review.openstack.org/87928 | 09:38 |
| *** andrearosa has quit IRC | 09:46 | |
| *** jang1 has quit IRC | 09:54 | |
| *** bauzas has quit IRC | 09:57 | |
| *** mrunge has quit IRC | 10:01 | |
| lifeless | derekh: undercloud didn't come up right - the 'native' ovs /e/n/i integration was at fault, have got rid of that and am re-running | 10:05 |
| derekh | lifeless: ok, RH rack seems to be stable, of the 96 tests in the last 11 hours that give out enough info (some fail before I can tell which rack they ran on) | 10:09 |
| derekh | the RH rack has had 30 successfull tests out of 37 | 10:10 |
| *** akrivoka has quit IRC | 10:10 | |
| lifeless | great | 10:10 |
| derekh | lifeless: the HP rack isn't doing so good at the moment, 11 out of 59 but hopfully that should be better once we're upgraded | 10:11 |
| *** matsuhashi has quit IRC | 10:16 | |
| *** jcoufal has joined #tripleo | 10:18 | |
| lifeless | derekh: trying to figure out why I have 66% packet loss to the undercloud atm | 10:18 |
| *** andrearosa has joined #tripleo | 10:19 | |
| gilliard | lifeless: understood your comments better now. We'll split into 2 patches and discuss the merits of the second one once it's up. | 10:20 |
| *** jang1 has joined #tripleo | 10:22 | |
| openstackgerrit | Nicholas Randon proposed a change to openstack/tripleo-incubator: Only use SSH keys for power mgmt if using VMs https://review.openstack.org/89787 | 10:22 |
| *** matsuhashi has joined #tripleo | 10:23 | |
| derekh | lifeless: let me see if I can find some correlation to compute nodes | 10:23 |
| openstackgerrit | Cian O'Driscoll proposed a change to openstack-infra/tripleo-ci: Test the upgrade codepath works as well. https://review.openstack.org/87758 | 10:31 |
| derekh | lifeless: as far as I can see all of the SUCCESS runs were on ci-overcloud-novacompute1-bvj3nddymido.novalocal | 10:31 |
| derekh | for n in 3771592 3771592 3776515 3776515 3776577 3776577 3777769 3777769 3776841 3776841 3777368 3777368 3776278 3776278 3771525 3771525 3775944 3775944 3775634 3775634 3776041 3776041 ; do echo select host from instances where hostname like \"%3771592%\" | mysql nova ; done | 10:31 |
| *** e0ne_ has quit IRC | 10:32 | |
| *** funzo has joined #tripleo | 10:33 | |
| *** funzo is now known as Guest25484 | 10:33 | |
| derekh | lifeless: scrap that, one sec | 10:34 |
| derekh | lifeless: that was too easy, stupid me, there on a misture of compute node 0,1,6,7 and 3 | 10:35 |
| *** giulivo has quit IRC | 10:36 | |
| *** Guest25484 has quit IRC | 10:37 | |
| lifeless | ohhhh | 10:39 |
| lifeless | I think I know exactly whats going on | 10:39 |
| lifeless | the manual fix for mellanox cannot be sticky across boots | 10:40 |
| lifeless | because we aren't updating the initrd nova-bm manages | 10:40 |
| lifeless | thats why the undercloud keeps going south | 10:40 |
| lifeless | SpamapS: ^ derekh | 10:40 |
| *** akrivoka has joined #tripleo | 10:44 | |
| derekh | o, so can we set it to load post boot? | 10:45 |
| lifeless | so for the undercloud I'm going to upgrade it to trust | 10:46 |
| lifeless | then copy the new kernel and ramdisk to the tftp tree | 10:47 |
| lifeless | the same kernel and ramdisk should be able to be copied down to get us a working updated kernel on the saucy overcloud nodes | 10:47 |
| lifeless | but one step at a time | 10:48 |
| openstackgerrit | Victor Sergeyev proposed a change to openstack/tuskar-ui: Replace Integer to Number in Type check https://review.openstack.org/89789 | 10:50 |
| *** giulivo has joined #tripleo | 10:54 | |
| *** mrunge has joined #tripleo | 10:55 | |
| *** e0ne has joined #tripleo | 11:03 | |
| openstackgerrit | Coleman Corrigan proposed a change to openstack/tripleo-image-elements: reset-db to get all db parameters from the config https://review.openstack.org/88340 | 11:04 |
| *** e0ne has quit IRC | 11:04 | |
| *** matsuhashi has quit IRC | 11:04 | |
| *** e0ne has joined #tripleo | 11:05 | |
| *** e0ne has quit IRC | 11:06 | |
| *** e0ne has joined #tripleo | 11:06 | |
| *** jang1 has quit IRC | 11:10 | |
| openstackgerrit | A change was merged to openstack/tuskar-ui: Improve developer documentation https://review.openstack.org/87928 | 11:10 |
| *** killer_prince has quit IRC | 11:10 | |
| *** e0ne has quit IRC | 11:12 | |
| *** e0ne has joined #tripleo | 11:18 | |
| lifeless | greghaynes: https://review.openstack.org/#/c/85100/ | 11:18 |
| *** killer_prince has joined #tripleo | 11:21 | |
| *** slagle has joined #tripleo | 11:28 | |
| *** lazy_prince has quit IRC | 11:31 | |
| *** funzo has joined #tripleo | 11:33 | |
| *** funzo is now known as Guest58713 | 11:34 | |
| openstackgerrit | Dmitry Shulyak proposed a change to openstack/tripleo-image-elements: Add pacemaker and corosync as tripleo elements https://review.openstack.org/86069 | 11:36 |
| *** Guest58713 has quit IRC | 11:38 | |
| *** olaph has quit IRC | 11:47 | |
| *** olaph has joined #tripleo | 11:47 | |
| *** Rakesh6 has quit IRC | 11:50 | |
| dshulyak | lifeless: are you around? you mentioned here https://review.openstack.org/#/c/89613/, that heat should create vip for us | 11:52 |
| dshulyak | so it would be some kind of custom resource, like OS::Heat::VirtualIP? | 11:52 |
| lifeless | dshulyak: http://docs.openstack.org/developer/heat/template_guide/openstack.html#OS::Neutron::Port | 11:53 |
| dshulyak | thanks | 11:54 |
| *** lucasagomes is now known as lucas-hungry | 12:00 | |
| *** ramishra_ has joined #tripleo | 12:02 | |
| *** ramishra has quit IRC | 12:03 | |
| *** jistr is now known as jistr|english | 12:06 | |
| *** nosnos has quit IRC | 12:10 | |
| *** jistr|mobi has joined #tripleo | 12:11 | |
| *** morazi has joined #tripleo | 12:12 | |
| *** dprince has joined #tripleo | 12:15 | |
| *** pblaho has quit IRC | 12:23 | |
| *** mrunge has quit IRC | 12:24 | |
| *** bauzas1 has joined #tripleo | 12:26 | |
| openstackgerrit | tom-howley proposed a change to openstack/tripleo-image-elements: Fix parsing of node list in rabbitmq element https://review.openstack.org/89812 | 12:28 |
| *** sballe has joined #tripleo | 12:29 | |
| *** weshay has joined #tripleo | 12:32 | |
| *** lazy_prince has joined #tripleo | 12:32 | |
| *** killer_prince has quit IRC | 12:33 | |
| *** lazy_prince is now known as killer_prince | 12:33 | |
| *** funzo has joined #tripleo | 12:34 | |
| *** funzo is now known as Guest17165 | 12:34 | |
| *** w_ has joined #tripleo | 12:35 | |
| *** olaph has quit IRC | 12:37 | |
| *** sballe_ has joined #tripleo | 12:38 | |
| *** rlandy has joined #tripleo | 12:39 | |
| dprince | derekh: what is with the .5 session votes? :) | 12:39 |
| *** Guest17165 has quit IRC | 12:39 | |
| dprince | derekh: do you break your poker chips in two as well? | 12:40 |
| *** sballe has quit IRC | 12:40 | |
| Ng | morning | 12:41 |
| derekh | dprince: wanted to show a vote for sessions I thought could be merged, e.g. there is two network related session that I though could be merged | 12:41 |
| derekh | dprince: that depends on what kind of a hand I'm holding | 12:42 |
| * dprince raises derekh .5 | 12:42 | |
| *** sballe_ has quit IRC | 12:43 | |
| dprince | derekh: which two networking sessions? | 12:43 |
| *** sballe_ has joined #tripleo | 12:43 | |
| derekh | summit.openstack.org/cfp/details/274 summit.openstack.org/cfp/details/101 | 12:44 |
| derekh | dprince: ^ | 12:44 |
| dprince | derekh: ah. So one is about Neutron, and the other is about everything that Neutron doesn't touch. Very different topics IMO | 12:45 |
| derekh | dprince: yup, I see the difference, but both important topics and would like to represent the fact that I think both should be represented | 12:46 |
| *** andreaf_ has quit IRC | 12:48 | |
| *** jdob has joined #tripleo | 12:48 | |
| *** andreaf has joined #tripleo | 12:48 | |
| *** sballe_ has quit IRC | 12:49 | |
| derekh | dprince: btw, as of last night RH rack is now processing jobs, although zuul seems to be hitting system wide problems this morning | 12:50 |
| dprince | derekh: cool :) | 12:50 |
| *** w_ is now known as olaph | 12:57 | |
| *** markmc has joined #tripleo | 13:02 | |
| openstackgerrit | Dmitry Shulyak proposed a change to openstack/tripleo-incubator: Change CtlVirtualInterface to br-ex https://review.openstack.org/89613 | 13:03 |
| gilliard | Is there a fix for the "Cannot find /home/mjg/.cache/tripleo/tripleo-incubator/scripts" issue with devtest? | 13:09 |
| gilliard | i.e. devtest fails almost immediately on a frresh checkout | 13:09 |
| *** lucas-hungry is now known as lucasagomes | 13:09 | |
| openstackgerrit | A change was merged to openstack/os-refresh-config: Add basic unit tests https://review.openstack.org/83633 | 13:10 |
| *** funzo has joined #tripleo | 13:11 | |
| *** funzo is now known as Guest57336 | 13:11 | |
| *** matty_dubs|gone is now known as matty_dubs | 13:11 | |
| gilliard | letting $TRIPLEO_ROOT default to ~/.cache/tripleo doesn't work any more, unless you have a copy of tripleo-incubator checked out in there already? | 13:13 |
| *** Guest57336 is now known as funzo | 13:14 | |
| gilliard | TRIPLEO_ROOT should default to `$(git rev-parse --show-toplevel)` (ie the root of the current repo), no? | 13:18 |
| gilliard | have I missed something obvious? | 13:18 |
| *** ramishra_ has quit IRC | 13:18 | |
| lifeless | gilliard: see cd6c740ebd73669dfc6d09e1dac62d8b59fb099c in incubator | 13:19 |
| lifeless | gilliard: it should still be defaulting to ~/.cache/tripleo for now, though I think everyone wants to remove that default | 13:20 |
| gilliard | lifeless: I've got that open in front of me, but it doesn't say why the default TRIPLEO_ROOT is still in ~/.cache | 13:20 |
| gilliard | well I guess I know how to work around it. What kind of patch would you accept to fix it? | 13:21 |
| *** nati_ueno has quit IRC | 13:21 | |
| lifeless | remove the default :) | 13:21 |
| lifeless | I would assume | 13:21 |
| lifeless | I don't have cells to think about it right now, 0130 in the morning | 13:21 |
| *** jprovazn has quit IRC | 13:22 | |
| *** jprovazn has joined #tripleo | 13:22 | |
| gilliard | .zZ | 13:23 |
| gilliard | I'll put a patch up to fix | 13:23 |
| lxsli | lifeless: please would you check the comments on https://review.openstack.org/#/c/87512/ ? In the (later) morning if you like | 13:27 |
| *** jprovazn has quit IRC | 13:27 | |
| lifeless | lxsli: sure | 13:30 |
| *** sballe has joined #tripleo | 13:32 | |
| openstackgerrit | Matthew Gilliard proposed a change to openstack/tripleo-incubator: Sets TRIPLEO_ROOT to the base of the current clone https://review.openstack.org/89838 | 13:38 |
| openstackgerrit | Matthew Gilliard proposed a change to openstack/tripleo-incubator: Defaults TRIPLEO_ROOT to base of the current clone https://review.openstack.org/89838 | 13:41 |
| *** jistr|english is now known as jistr | 13:43 | |
| *** jprovazn has joined #tripleo | 13:51 | |
| *** jpeeler has joined #tripleo | 13:51 | |
| *** jpeeler has joined #tripleo | 13:51 | |
| slagle | dprince: you didn't even vote for your own session :) | 13:52 |
| *** ramishra has joined #tripleo | 13:55 | |
| *** martyntaylor has left #tripleo | 13:56 | |
| openstackgerrit | Gerry Drudy proposed a change to openstack/tripleo-image-elements: Store swift account, container & object data on /mnt filesystem https://review.openstack.org/89847 | 13:56 |
| openstackgerrit | Dmitry Shulyak proposed a change to openstack/tripleo-heat-templates: Introduce configurable virtual ip in templates https://review.openstack.org/89556 | 13:57 |
| *** darraghb has joined #tripleo | 13:59 | |
| dprince | slagle: I did, I thought | 13:59 |
| slagle | dprince: not the HACKING guidelines one | 14:00 |
| openstackgerrit | Dmitry Shulyak proposed a change to openstack/tripleo-heat-templates: Introduce configurable virtual ip in templates https://review.openstack.org/89556 | 14:00 |
| openstackgerrit | Dmitry Shulyak proposed a change to openstack/tripleo-incubator: Make CtlVirtualInterface configurable https://review.openstack.org/89613 | 14:12 |
| *** weshay has quit IRC | 14:12 | |
| dprince | slagle: Given our existing time slots are limited to just 6 slots we should punt on that. HACKING is not the most important thing ATM. | 14:13 |
| dprince | slagle: I think I added that session idea up there almost a month ago when there wasn't anything else posted... | 14:13 |
| *** rwsu has joined #tripleo | 14:14 | |
| slagle | dprince: cool :) | 14:15 |
| dprince | slagle: can you rebase your MTU patch: https://review.openstack.org/#/c/82803/ | 14:15 |
| slagle | dprince: yea, i need to | 14:18 |
| openstackgerrit | Gerry Drudy proposed a change to openstack/tripleo-image-elements: Store swift account, container & object data on /mnt filesystem https://review.openstack.org/89847 | 14:19 |
| *** cody-somerville has quit IRC | 14:20 | |
| *** bauzas1 has quit IRC | 14:20 | |
| *** killer_prince has quit IRC | 14:23 | |
| openstackgerrit | Stuart McLaren proposed a change to openstack/tripleo-image-elements: Add scripts for managing iptables https://review.openstack.org/89860 | 14:26 |
| *** julim has joined #tripleo | 14:26 | |
| lifeless | derekh: can I tag you in ? | 14:29 |
| *** giulivo has quit IRC | 14:30 | |
| derekh | lifeless: yup, on phone at the moment but should be free in a few minutes, let me know where you are | 14:30 |
| lifeless | derekh: 2014-04-24 HP undercloud - upgraded to trusty, venvs all broken. | 14:31 |
| lifeless | derekh: I will spare you for now the headaches in getting the upgrade done at all | 14:31 |
| *** weshay has joined #tripleo | 14:32 | |
| lifeless | derekh: but - there still seem to be regular network pauses - I'm starting to suspect STP timers or something, but hopefully no more mellanox lockups | 14:33 |
| lifeless | derekh: on the undercloud; overcloud not touched yet | 14:34 |
| *** hashar has quit IRC | 14:34 | |
| *** hashar has joined #tripleo | 14:35 | |
| lifeless | derekh: notes in the etherpad | 14:36 |
| derekh | lifeless: ok, would you like me fucose on under or overcloud ? | 14:36 |
| openstackgerrit | Stuart McLaren proposed a change to openstack/tripleo-image-elements: Add scripts for managing iptables https://review.openstack.org/89860 | 14:36 |
| derekh | *focus | 14:36 |
| lifeless | derekh: we need the undercloud back up or the overcloud will eventually all go offline when dhcp fails | 14:36 |
| derekh | lifeless: ok | 14:36 |
| lifeless | derekh: I'm seeing terrible packetloss to that machine still; it may be time to grab ng and get a DC ticket filed on it | 14:37 |
| lifeless | since its now running latest drivers for sure | 14:37 |
| Ng | fun times | 14:37 |
| lifeless | derekh: but! | 14:37 |
| lifeless | derekh: the venvs in /opt/stack are bust | 14:38 |
| Ng | which machine are we talking about, re packetloss? | 14:38 |
| lifeless | derekh: I think its the 'config' symlink (use find to find it) | 14:38 |
| NobodyCam | good morning TripleO - just checking to see if anyone has a few minute to take a look at https://review.openstack.org/#/c/89703/ | 14:38 |
| NobodyCam | simeple one line change :) | 14:38 |
| derekh | lifeless: ok, I'll try and get the venvs working | 14:38 |
| lifeless | derekh: which on quantal pointed at a python2.7 dir, and on trusty should point at a slightly different dir name | 14:38 |
| lifeless | derekh: if I'm right just fixing that will get the venvs sorted | 14:38 |
| *** akrivoka has quit IRC | 14:39 | |
| lifeless | derekh: rebuilding everything should absolutely not be needed. | 14:39 |
| derekh | lifeless: ok, got ya | 14:39 |
| lifeless | e.g | 14:39 |
| lifeless | heat-admin@undercloud-notcompute-jws3awlsb2kh:/opt/stack/venvs/os-apply-config/lib/python2.7$ ls -l | 14:39 |
| lifeless | ls -l config | 14:39 |
| lifeless | lrwxrwxrwx 1 root root 25 Sep 30 2013 config -> /usr/lib/python2.7/config | 14:39 |
| lifeless | but that appears to need to point at /usr/lib/python2.7/config-x86_64-linux-gnu/ | 14:40 |
| *** TravT has joined #tripleo | 14:40 | |
| lifeless | hmm,its more than that | 14:40 |
| lifeless | derekh: test via /opt/stack/venvs/os-apply-config/bin/python -c 'import io' | 14:41 |
| derekh | lifeless: ok, still on phone but will get at it as soon as I'm off | 14:42 |
| lifeless | Ng: cd-undercloud.tripleo.org | 14:44 |
| lifeless | Ng: e.g. | 14:44 |
| lifeless | 64 bytes from 138.35.77.3: icmp_seq=1407 ttl=46 time=191 ms | 14:44 |
| lifeless | 64 bytes from 138.35.77.3: icmp_seq=1409 ttl=46 time=191 ms | 14:44 |
| lifeless | --- cd-undercloud.tripleo.org ping statistics --- | 14:45 |
| lifeless | 1415 packets transmitted, 220 received, 84% packet loss, time 2339066ms | 14:45 |
| lifeless | rtt min/avg/max/mdev = 191.136/191.955/198.603/0.956 ms | 14:45 |
| Ng | lifeless: and that's coming from the bastion? | 14:45 |
| lifeless | Ng: that ping was from *here*, but gather some data thyself | 14:45 |
| Ng | lifeless: I will definnitely seek to rule out the pacific ocean :) | 14:46 |
| *** newell has joined #tripleo | 14:47 | |
| lifeless | derekh: ok, fix appears to be the config thing + copying the current python executable into e.g /opt/stack/venvs/os-apply-config/bin/python | 14:48 |
| lifeless | derekh: that got os-apply-config fixed, rest are up to you; I'm broken now :) | 14:49 |
| derekh | lifeless: ok, will verify and update the others | 14:49 |
| Ng | not saying a ticket isn't needed, but fwiw I get very consistent pings from cd-undercloud atm | 14:50 |
| *** bauzas has joined #tripleo | 14:51 | |
| *** lazy_prince has joined #tripleo | 14:51 | |
| *** lazy_prince is now known as killer_prince | 14:51 | |
| *** giulivo has joined #tripleo | 14:52 | |
| *** TravT has quit IRC | 14:56 | |
| *** geerdest has joined #tripleo | 14:58 | |
| *** jcoufal has quit IRC | 14:58 | |
| *** jistr has quit IRC | 14:59 | |
| *** untriaged-bot has joined #tripleo | 15:00 | |
| untriaged-bot | Untriaged bugs so far: | 15:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1311631 | 15:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1302881 | 15:00 |
| uvirtbot | Launchpad bug 1311631 in tripleo "rabbitmq os-refresh config fails on the master due to mis-parsing of nodes list" [Undecided,In progress] | 15:00 |
| uvirtbot | Launchpad bug 1302881 in tripleo "incloud CIDR can overlap custom baremetal-network" [Undecided,Incomplete] | 15:00 |
| untriaged-bot | https://bugs.launchpad.net/tuskar/+bug/1311695 | 15:00 |
| uvirtbot | Launchpad bug 1311695 in tuskar "Error during the validation of Overcloud `attributes` field" [Undecided,New] | 15:00 |
| *** untriaged-bot has quit IRC | 15:00 | |
| *** jcoufal has joined #tripleo | 15:01 | |
| *** jistr has joined #tripleo | 15:01 | |
| *** TravT has joined #tripleo | 15:02 | |
| *** jprovazn is now known as jprovazn_afk | 15:05 | |
| *** julim has quit IRC | 15:11 | |
| *** viktors is now known as viktors|afk | 15:16 | |
| openstackgerrit | Stuart McLaren proposed a change to openstack/tripleo-image-elements: Add scripts for managing iptables https://review.openstack.org/89860 | 15:17 |
| openstackgerrit | Dmitry Shulyak proposed a change to openstack/tripleo-incubator: Make CtlVirtualInterface configurable https://review.openstack.org/89613 | 15:19 |
| *** jeblair has joined #tripleo | 15:25 | |
| SpamapS | Ng: I have evidence that cd-undercloud had trouble pinging at least one host from 14:00 UTC to 14:26 UTC ... | 15:28 |
| SpamapS | Ng: I left my ping thing setup on cd-undercloud .. so it tells me when 10.10.16.135 isn't pingable | 15:29 |
| SpamapS | 135 is just a compute host now.. but either way.. connectivity problems were there | 15:29 |
| SpamapS | Oh and sporadically through the night | 15:29 |
| SpamapS | Ng: I suggest we make use of our nagios element (or the icinga element that is in the review queue.. :) and gather data by pinging routers as well as our hosts. | 15:30 |
| *** lazy_prince has joined #tripleo | 15:33 | |
| *** e0ne has quit IRC | 15:34 | |
| openstackgerrit | A change was merged to openstack/tripleo-image-elements: RabbitMQ - Support clusters with more than 2 nodes https://review.openstack.org/86339 | 15:34 |
| *** e0ne has joined #tripleo | 15:35 | |
| Ng | SpamapS: yeah that's not a terrible idea | 15:36 |
| SpamapS | Ng: probably would be good to have the two regions ping eachother as well. | 15:37 |
| SpamapS | dprince: ^ | 15:37 |
| derekh | Ng: SpamapS venvs fixed on undercloud, all our undercloud python services appear to be running again | 15:37 |
| Ng | cool | 15:37 |
| derekh | Ng: SpamapS is it possible the connectivity problems are related to the problem infra are/were seeing in hpcloud region b ? | 15:42 |
| SpamapS | what happened to the venvs? | 15:43 |
| lazy_prince | Hi all, are there any issues with ci..? seems like some are still stuck.. | 15:43 |
| SpamapS | derekh: Entirely possible yes | 15:43 |
| derekh | SpamapS: lifeless upgraded the undercloud to trusty, which resulted in busted venvs, he left me with a suggested fix that I applied to each of them and appears to have worked | 15:44 |
| SpamapS | derekh: upgraded.. like, in-place? | 15:45 |
| openstackgerrit | A change was merged to openstack/diskimage-builder: Sort rhel/bin/map-packages https://review.openstack.org/89765 | 15:45 |
| openstackgerrit | A change was merged to openstack/tripleo-image-elements: Haproxy should listen only on vip https://review.openstack.org/89517 | 15:45 |
| openstackgerrit | A change was merged to openstack/tripleo-image-elements: Make bin/ensure-bridge executable https://review.openstack.org/89593 | 15:45 |
| derekh | SpamapS: yup | 15:45 |
| *** cody-somerville has joined #tripleo | 15:46 | |
| *** cody-somerville has joined #tripleo | 15:46 | |
| SpamapS | ahh | 15:49 |
| SpamapS | thats.. an odd choice | 15:49 |
| derekh | iirc it was for the newer mellanox driver | 15:51 |
| SpamapS | yes but that makes no sense because the undercloud already had the latest upstream mellanox | 15:51 |
| derekh | <lifeless> I think I know exactly whats going on | 15:52 |
| derekh | <lifeless> the manual fix for mellanox cannot be sticky across boots | 15:52 |
| derekh | <lifeless> because we aren't updating the initrd nova-bm manages | 15:52 |
| derekh | <lifeless> thats why the undercloud keeps going south | 15:52 |
| SpamapS | anyway, ok so undercloud is working now? | 15:52 |
| SpamapS | ahhhhh | 15:52 |
| SpamapS | lurvely | 15:52 |
| derekh | SpamapS: yes, appears to be (unless we still have connectivity problems) | 15:52 |
| SpamapS | though the answer to that is to rmmod mlx4_en and then modprobe it again. :-P | 15:52 |
| derekh | so now we have about 20 nodes on the overcloud in ERROR state, which take from capacity, if there still there in a few minutes, I think we should reset-state them so that nodepool can delete them | 15:53 |
| derekh | lazy_prince: yup, lots for slowness in CI today | 15:54 |
| lazy_prince | derekh: aha.. | 15:54 |
| lazy_prince | it seems like there are tasks in queue from last 8hrs.. | 15:55 |
| *** cody-somerville has quit IRC | 15:55 | |
| *** e0ne has quit IRC | 15:55 | |
| *** eghobo has joined #tripleo | 15:55 | |
| *** rpodolyaka has quit IRC | 15:56 | |
| SpamapS | derekh: I think we should just reset-state them. | 15:56 |
| derekh | SpamapS: ok, I'll do it now | 15:56 |
| derekh | done | 15:59 |
| *** cwolferh has joined #tripleo | 16:01 | |
| *** matty_dubs is now known as matty_dubs|lunch | 16:02 | |
| *** lazy_prince has quit IRC | 16:06 | |
| *** cody-somerville has joined #tripleo | 16:07 | |
| *** blamar has quit IRC | 16:08 | |
| SpamapS | seems like most of our ERROR states are still weird things where neutron fails | 16:11 |
| *** vinsh_zzzz is now known as vinsh | 16:11 | |
| *** chuckC has joined #tripleo | 16:11 | |
| *** mkerrin has quit IRC | 16:16 | |
| *** mkerrin has joined #tripleo | 16:26 | |
| *** jistr_ has joined #tripleo | 16:27 | |
| openstackgerrit | James Slagle proposed a change to openstack/tripleo-heat-templates: Expose dnsmasq options https://review.openstack.org/82803 | 16:28 |
| *** jistr has quit IRC | 16:31 | |
| *** hashar has quit IRC | 16:31 | |
| derekh | root@ci-overcloud-novacompute1-bvj3nddymido:/var/log/upstart# ls | wc -l | 16:35 |
| derekh | 75129 | 16:35 |
| *** jcoufal has quit IRC | 16:35 | |
| *** jistr|mobi has quit IRC | 16:36 | |
| *** jistr_ has quit IRC | 16:37 | |
| SpamapS | derekh: yeah, https://review.openstack.org/#/c/84561/ | 16:39 |
| *** fandi has joined #tripleo | 16:41 | |
| derekh | SpamapS: will that flood the console with nework messages (if using textcons for example?) | 16:43 |
| *** darraghb has quit IRC | 16:44 | |
| *** darraghb has joined #tripleo | 16:44 | |
| *** matty_dubs|lunch is now known as matty_dubs | 16:48 | |
| *** giulivo has quit IRC | 16:49 | |
| *** dprince has quit IRC | 16:50 | |
| *** blamar has joined #tripleo | 16:50 | |
| SpamapS | derekh: it's a slow trickle | 16:53 |
| SpamapS | derekh: if we do 'console none' they're lost .. not sure I like that. :-/ | 16:53 |
| derekh | SpamapS: ok, fair enough | 16:54 |
| SpamapS | derekh: especially if there are problems in early boot before all these taps go crazy | 16:54 |
| *** ramishra has quit IRC | 16:57 | |
| openstackgerrit | A change was merged to openstack/tripleo-image-elements: RabbitMQ - Consistent PID file location https://review.openstack.org/85604 | 16:57 |
| *** ramishra has joined #tripleo | 16:57 | |
| NobodyCam | ping funzo | 16:57 |
| openstackgerrit | A change was merged to openstack/tripleo-image-elements: Make innodb pool size configurable https://review.openstack.org/86889 | 16:58 |
| NobodyCam | funzo: just wanted to check on the RHEL dib elements, do you happen to recall are they both in a working state? | 16:58 |
| *** lucasagomes_ has joined #tripleo | 16:59 | |
| vinsh | SpamapS, lifeless, greghaynes Wanted to toss another "thank you" your way.. you guys really brought my tripleo-fu up a few levels yesterday. ++11 | 16:59 |
| openstackgerrit | Matthew Gilliard proposed a change to openstack/tripleo-heat-templates: Increase PXE deployment timeout for overcloud https://review.openstack.org/86523 | 17:00 |
| *** lucasagomes has quit IRC | 17:00 | |
| derekh | SpamapS: gotta run, I +2'd that change, I think lifeless mentioned redeploying an overcloud yesterday now that the RH rack is running, should be back later to help out if its needed | 17:01 |
| *** ramishra has quit IRC | 17:02 | |
| *** lucasagomes has joined #tripleo | 17:02 | |
| *** derekh has quit IRC | 17:02 | |
| *** spzala has joined #tripleo | 17:05 | |
| *** lucasagomes_ has quit IRC | 17:05 | |
| *** nati_ueno has joined #tripleo | 17:06 | |
| *** nati_uen_ has joined #tripleo | 17:12 | |
| davidlenwell | how up to date is this? http://docs.openstack.org/developer/tripleo-incubator/deploying.html | 17:13 |
| *** nati_ueno has quit IRC | 17:15 | |
| *** lucasagomes is now known as lucas-dinner | 17:16 | |
| openstackgerrit | Coleman Corrigan proposed a change to openstack/tripleo-image-elements: reset-db to get all db parameters from the config https://review.openstack.org/88340 | 17:16 |
| *** eghobo has quit IRC | 17:17 | |
| *** nekron99 has joined #tripleo | 17:18 | |
| openstackgerrit | Ben Nemec proposed a change to openstack/tripleo-incubator: Allow to set seed node cpus and memory https://review.openstack.org/84078 | 17:19 |
| *** eghobo has joined #tripleo | 17:19 | |
| *** eghobo has quit IRC | 17:20 | |
| *** eghobo has joined #tripleo | 17:20 | |
| mordred | SpamapS: lifeless and I were talking about my patch to get nodepool to use dib - and one of the thigns I need is a dynamic list of repos created | 17:21 |
| mordred | he suggested that I do it somewhere in pre-something before something | 17:21 |
| mordred | any ideas? | 17:21 |
| SpamapS | mordred: source-repositories does the bulk of the work of cloning repos and stuff | 17:23 |
| SpamapS | mordred: so elements/source-repositories/extra-data.d/98-source-repositories is the script you probably want to inspect | 17:24 |
| mordred | yah. I think he was suggesting I do an element in extra-data before 98 then | 17:25 |
| mordred | that creates a source-repositories file? | 17:25 |
| mordred | does that sound reasonable? | 17:25 |
| SpamapS | well wait let's back up so I know what you're trying to do | 17:26 |
| mordred | SpamapS: the problem wanting to be solved is that instead of listing all of the repos by hand in a file | 17:26 |
| mordred | I want to dynamically create that list at runtime | 17:26 |
| mordred | (specifically by querying gerrit for the list of existing projects) | 17:26 |
| SpamapS | Ok | 17:26 |
| SpamapS | so yeah, in extra-data.d before 98-source-repositories, you'd do that | 17:26 |
| *** jtomasek has quit IRC | 17:26 | |
| mordred | and I want that script to make a file in TMP_HOOKS_PATH right? | 17:27 |
| SpamapS | and dump it into $TMP_HOOKS_PATH | 17:27 |
| SpamapS | source-repository-${SOMETHING_UNIQUE} | 17:27 |
| mordred | yah. awesome | 17:27 |
| mordred | thanks! | 17:27 |
| SpamapS | mordred: is this intended to replace the template/snap thing? | 17:28 |
| mordred | yeah | 17:28 |
| mordred | current non-functional proto-patch is here: https://review.openstack.org/#/c/88479/ | 17:28 |
| mordred | it's both old and broken- so I need to update the contents of the elements to match recent changes in launch scripts | 17:29 |
| mordred | but also I need to not have a hard-coded list of repos anymore | 17:29 |
| *** bauzas has quit IRC | 17:30 | |
| *** jang1 has joined #tripleo | 17:30 | |
| *** panda has quit IRC | 17:32 | |
| *** panda has joined #tripleo | 17:32 | |
| *** marun has joined #tripleo | 17:34 | |
| *** darraghb has quit IRC | 17:34 | |
| funzo | NobodyCam: I haven't ran the dib build for rhel in a while, but as of the havana release they were working | 17:40 |
| funzo | NobodyCam: did you see behavior to the contrary? | 17:40 |
| NobodyCam | funzo: Nope, I am working on a project that may be able to use them. and I just wanted to check before I spoke to soon :) | 17:41 |
| NobodyCam | lucas said you'd prob know.. So I asked :) | 17:42 |
| *** markmc has quit IRC | 17:44 | |
| *** morganfainberg_Z is now known as morganfainberg | 17:48 | |
| funzo | NobodyCam: pretty sure it's fine right now | 17:49 |
| funzo | works with the 6.5 cloud image that is on rhn | 17:49 |
| NobodyCam | sweet :) | 17:50 |
| NobodyCam | TY funzo :) | 17:50 |
| funzo | np | 17:50 |
| *** lsmola has quit IRC | 17:51 | |
| *** saurabhs has joined #tripleo | 18:00 | |
| *** eguz has joined #tripleo | 18:01 | |
| *** dprince has joined #tripleo | 18:03 | |
| *** jtomasek has joined #tripleo | 18:04 | |
| *** eghobo has quit IRC | 18:05 | |
| *** epim has joined #tripleo | 18:05 | |
| *** e0ne has joined #tripleo | 18:07 | |
| *** e0ne has quit IRC | 18:14 | |
| *** cwolferh has quit IRC | 18:17 | |
| *** sdake_ has joined #tripleo | 18:18 | |
| openstackgerrit | Radomir Dopieralski proposed a change to openstack/tuskar-ui: Include default configuration for Sphinx https://review.openstack.org/89780 | 18:18 |
| *** epim has quit IRC | 18:19 | |
| *** e0ne has joined #tripleo | 18:28 | |
| *** blamar has quit IRC | 18:28 | |
| *** blamar has joined #tripleo | 18:29 | |
| *** jtomasek has quit IRC | 18:30 | |
| *** blamar has quit IRC | 18:33 | |
| *** e0ne has quit IRC | 18:35 | |
| jeblair | BadRequest: Error. Unable to associate floating ip (HTTP 400) (Request-ID: req-c0350822-ae5d-4cfd-b045-4ec8a7d0252b) | 18:35 |
| jeblair | SpamapS: ^ seen in infra nodepool | 18:35 |
| *** pblaho has joined #tripleo | 18:37 | |
| SpamapS | jeblair: ack, investigating | 18:38 |
| SpamapS | jeblair: /var/log/upstart/nova-api.log:2014-04-23 18:34:55.589 29336 WARNING nova.api.openstack.compute.contrib.floating_ips [req-c0350822-ae5d-4cfd-b045-4ec8a7d0252b d5af62d2183d431796d74c5bb119ec9f e01e473a9250498883955b80966a1e58] multiple fixed_ips exist, using the first: 192.168.1.111 | 18:38 |
| *** spzala has quit IRC | 18:40 | |
| *** chuckC has quit IRC | 18:42 | |
| *** hashar has joined #tripleo | 18:42 | |
| *** chuckC has joined #tripleo | 18:51 | |
| *** dshulyak_ has joined #tripleo | 19:00 | |
| openstackgerrit | A change was merged to openstack/tripleo-image-elements: Get the test env overcloud password from the correct place https://review.openstack.org/88471 | 19:03 |
| *** jprovazn_afk has quit IRC | 19:03 | |
| *** nati_uen_ has quit IRC | 19:08 | |
| *** jdob_ has joined #tripleo | 19:10 | |
| *** nati_ueno has joined #tripleo | 19:14 | |
| *** e0ne has joined #tripleo | 19:26 | |
| openstackgerrit | Adam Vinsh proposed a change to openstack/tripleo-image-elements: haproxy make element balance type configurable https://review.openstack.org/88105 | 19:27 |
| *** dshulyak_ has quit IRC | 19:30 | |
| openstackgerrit | Adam Vinsh proposed a change to openstack/tripleo-image-elements: haproxy make element balance type configurable https://review.openstack.org/88105 | 19:30 |
| *** nati_ueno has quit IRC | 19:30 | |
| *** epim has joined #tripleo | 19:34 | |
| *** ifarkas has quit IRC | 19:35 | |
| *** e0ne has quit IRC | 19:35 | |
| *** nekron99 has quit IRC | 19:40 | |
| *** julim has joined #tripleo | 19:41 | |
| openstackgerrit | Gregory Haynes proposed a change to openstack/tripleo-image-elements: Allow multiple binds per service in haproxy https://review.openstack.org/89925 | 19:53 |
| *** bauzas1 has joined #tripleo | 19:57 | |
| lifeless | SpamapS: yes, the undercloud locked up with sa_alloc, and I was like W T F | 20:03 |
| lifeless | SpamapS: then went on a rage fueled fix which took me recursively back to the seed host (quantal! - couldn't reliably boot the undercloud) and bastion jumphost (also quantal, also unreliable) | 20:04 |
| lifeless | SpamapS: all three are now dum dum dum duuuuum trusty | 20:04 |
| lifeless | Ng: how did we do on networking ? | 20:05 |
| lifeless | Ng: was it just the pacific? | 20:05 |
| *** nati_ueno has joined #tripleo | 20:06 | |
| Ng | lifeless: yeah the pings were very smooth and consistent for me, but it does seem like we are getting intermittent issues. I think SpamapS is right and we should set up more consistent monitoring of this stuff. transient internet stuff doesn't seem like the only thing at play here | 20:08 |
| openstackgerrit | Gregory Haynes proposed a change to openstack/tripleo-image-elements: Add os-is-bootstrap-host element and script https://review.openstack.org/86435 | 20:08 |
| *** pblaho has quit IRC | 20:09 | |
| openstackgerrit | Gregory Haynes proposed a change to openstack/tripleo-incubator: Slight increase in testenv disk space https://review.openstack.org/88459 | 20:09 |
| *** andreaf has quit IRC | 20:11 | |
| *** nati_ueno has quit IRC | 20:11 | |
| Ng | are we actually running a nagios somewhere already? | 20:11 |
| Ng | curious how the config would get built out, for adding machine pings | 20:12 |
| *** marun has quit IRC | 20:12 | |
| *** marun has joined #tripleo | 20:12 | |
| SpamapS | Ng: we have a nagios element which takes nova client creds | 20:15 |
| SpamapS | Ng: and ping/ssh monitors all the boxes | 20:15 |
| SpamapS | Ng: so we can just stick that on the undercloud | 20:15 |
| Ng | ah right, so we don't have it running somewhere already | 20:16 |
| *** chuckC has quit IRC | 20:19 | |
| *** eguz has quit IRC | 20:19 | |
| *** eghobo has joined #tripleo | 20:20 | |
| *** nati_ueno has joined #tripleo | 20:24 | |
| *** julim has quit IRC | 20:33 | |
| *** ccrouch1 has joined #tripleo | 20:38 | |
| *** ccrouch has quit IRC | 20:39 | |
| *** marun has quit IRC | 20:41 | |
| *** jdob_ has quit IRC | 20:44 | |
| SpamapS | FYI: just released os-collect-config 0.1.16 (since it fixes Heat software-config) | 20:45 |
| lifeless | slagle: you saw you got voluntered for release duty by your compatriot ? | 20:50 |
| slagle | lifeless: i did! | 20:53 |
| slagle | lifeless: will push stuff out tomorrow...unless someone is waiting for it earlier? | 20:53 |
| lifeless | I don't think so other than occ which SpamapS just ninjad | 20:54 |
| lifeless | slagle: thanks ;) | 20:54 |
| slagle | cool | 20:54 |
| lifeless | oh, note there is a new occ too - os-cloud-config that needs to get on the 0.0.x bandwagon | 20:56 |
| *** untriaged-bot has joined #tripleo | 21:00 | |
| untriaged-bot | Untriaged bugs so far: | 21:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1311631 | 21:00 |
| uvirtbot | Launchpad bug 1311631 in tripleo "rabbitmq os-refresh config fails on the master due to mis-parsing of nodes list" [Undecided,In progress] | 21:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1302881 | 21:00 |
| *** untriaged-bot has quit IRC | 21:00 | |
| uvirtbot | Launchpad bug 1302881 in tripleo "incloud CIDR can overlap custom baremetal-network" [Undecided,Incomplete] | 21:00 |
| lifeless | NobodyCam: hows the ironic job looking ? do youneed any debug / analysis assistance? | 21:01 |
| NobodyCam | ieek I need a good run from the gate... Thank you for ping.... | 21:05 |
| *** akrivoka has joined #tripleo | 21:05 | |
| *** akrivoka has quit IRC | 21:11 | |
| *** jdob has quit IRC | 21:17 | |
| *** hashar has quit IRC | 21:17 | |
| *** andreaf has joined #tripleo | 21:21 | |
| *** yamahata has joined #tripleo | 21:26 | |
| *** rha has quit IRC | 21:45 | |
| *** jang1 has quit IRC | 21:47 | |
| *** dprince has quit IRC | 21:52 | |
| *** matty_dubs is now known as matty_dubs|gone | 21:58 | |
| *** nati_ueno has quit IRC | 21:59 | |
| openstackgerrit | Chris Krelle proposed a change to openstack/tripleo-image-elements: Update mysql element to work better with OpenSuSe https://review.openstack.org/89947 | 22:08 |
| tchaypo | morninges | 22:08 |
| greghaynes | O/ | 22:09 |
| vinsh | oy oy oy.. with keepalived running on 3 control nodes... all of them stay as master | 22:13 |
| vinsh | even once they get an advert from a node with a higher priority | 22:13 |
| greghaynes | ah yes, that sounds like the keepalived I know and love | 22:13 |
| * vinsh screams :) | 22:14 | |
| vinsh | any thing come to mind.. as a typical gotcha? | 22:14 |
| *** nati_ueno has joined #tripleo | 22:15 | |
| greghaynes | Nerp, I usually have to break out the tcpdump with that one | 22:15 |
| greghaynes | got a patch? | 22:15 |
| greghaynes | er, review | 22:15 |
| vinsh | I don't is just the out of box config here | 22:15 |
| greghaynes | qh | 22:15 |
| vinsh | root@overcloud-controller2-tk4sygwhq3jf:~# tcpdump -v -i eth0 host 224.0.0.18 | 22:16 |
| vinsh | tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes | 22:16 |
| vinsh | 22:07:50.667318 IP (tos 0xc0, ttl 255, id 24, offset 0, flags [none], proto VRRP (112), length 40) | 22:16 |
| vinsh | 192.0.2.254 > 224.0.0.18: vrrp 192.0.2.254 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 101, authtype none, intvl 1s, length 20, addrs: 192.0.2.254 | 22:16 |
| vinsh | 22:07:50.803081 IP (tos 0xc0, ttl 255, id 16, offset 0, flags [none], proto VRRP (112), length 40) | 22:16 |
| vinsh | 192.0.2.254 > 224.0.0.18: vrrp 192.0.2.254 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 100, authtype none, intvl 1s, length 20, addrs: 192.0.2.254 | 22:16 |
| vinsh | 22:07:51.668682 IP (tos 0xc0, ttl 255, id 25, offset 0, flags [none], proto VRRP (112), length 40) | 22:16 |
| vinsh | 192.0.2.254 > 224.0.0.18: vrrp 192.0.2.254 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 101, authtype none, intvl 1s, length 20, addrs: 192.0.2.254 | 22:16 |
| greghaynes | patste paste paste! | 22:16 |
| vinsh | sometimes.. it sees controller 0 or 1.. that have priorities 200 and 300 respectively | 22:16 |
| vinsh | yet it stays master. | 22:16 |
| greghaynes | hrmm, its been a while and id need to read some docs | 22:17 |
| ccrouch1 | lifeless: do you know when our tirpleo design sessions will be yet? | 22:17 |
| ccrouch1 | /me is trying to plan his summit: http://junodesignsummit.sched.org/ | 22:17 |
| vinsh | greghaynes, i'll capture more relevent stuffs and toss it in a pastebin for ya. going to keep trying different configs here. | 22:19 |
| greghaynes | ok | 22:20 |
| greghaynes | also I think I saw some patches up for doing VIP master election already | 22:20 |
| greghaynes | https://review.openstack.org/#/c/87873/ | 22:21 |
| *** noslzzp has quit IRC | 22:31 | |
| lifeless | ccrouch1: end of the week | 22:37 |
| lifeless | thursday night + friday | 22:38 |
| *** rcarrillocruz has joined #tripleo | 22:42 | |
| *** yamahata has quit IRC | 22:47 | |
| *** bauzas1 has quit IRC | 23:04 | |
| mordred | lifeless: hey - is there a way to run a script in an element as a particular user? | 23:07 |
| clarkb | mordred: you should be able to su right? dib runs as root? just be sure to exit when done as that user | 23:07 |
| mordred | clarkb: right - but if I want to run an entire script as a user | 23:08 |
| *** bnemec has quit IRC | 23:09 | |
| *** chuckC has joined #tripleo | 23:09 | |
| *** noslzzp has joined #tripleo | 23:10 | |
| *** bnemec has joined #tripleo | 23:10 | |
| lifeless | mordred: exec | 23:11 |
| lifeless | within the script | 23:11 |
| ccrouch1 | lifeless: thanks | 23:12 |
| mordred | lifeless: thanks. next ? - is there an install-package equiv to remove a package? | 23:14 |
| hewbrocca | yum remove? | 23:15 |
| * hewbrocca hides | 23:15 | |
| mordred | :) | 23:15 |
| * greghaynes is afraid to ask why | 23:17 | |
| *** andreaf has quit IRC | 23:18 | |
| greghaynes | pretty sure the answer is no, though | 23:18 |
| mordred | greghaynes: well - if you were using a thing, such as a downloaded script | 23:19 |
| mordred | and it did 95% of what you wanted, but it installed an extra thign you did not want | 23:19 |
| mordred | and you wanted to make sure that thing was not actually in the image | 23:19 |
| greghaynes | ah. knowing us there might be some sed s/<suff>// on that script | 23:19 |
| mordred | greghaynes: s/if you were using a downloaded script/if you were running a very large amount of puppet which as a side effect caused python-pip to be installed/ | 23:20 |
| greghaynes | :( | 23:20 |
| mordred | (it's ok that the answer is no - I can deal with it - was just trying to make sure I wasn't not using magic if it was there) | 23:21 |
| *** chuckC has quit IRC | 23:21 | |
| greghaynes | Ah, well sounds like fun | 23:22 |
| lifeless | mordred: there is not yet but sure there should be | 23:24 |
| mordred | greghaynes: so much fun | 23:26 |
| mordred | lifeless: last question - are you guys still using workspace-cache/$git_repos - or have you transitioned to using /opt/git? | 23:27 |
| mordred | lifeless: (working on translating nodepool-scripts into elements) | 23:27 |
| lifeless | mordred: would love patches to migrate us | 23:27 |
| *** adam_g has joined #tripleo | 23:35 | |
| *** CaptTofu has quit IRC | 23:39 | |
| *** CaptTofu has joined #tripleo | 23:39 | |
| *** CaptTof__ has joined #tripleo | 23:41 | |
| *** CaptTofu has quit IRC | 23:43 | |
| mordred | lifeless: what repos are those in? toci? others? | 23:50 |
| lifeless | mordred: should be all in the nodepool scripts + a little adapter glue in toci | 23:50 |
| mordred | lifeless: well, the creation of the repos in nodepool is the thing I was hoping I could delete | 23:51 |
| mordred | lifeless: so lemme go make you a toci patch | 23:52 |
| lifeless | awesome sauce | 23:53 |
| mordred | lifeless: | 23:53 |
| mordred | mordred@camelot:~/src/openstack-infra/tripleo-ci$ git grep workspace-cache | 23:53 |
| mordred | nothing | 23:53 |
| mordred | lifeless: I'm going to take that to mean that, perhaps, you are in fact already moved off | 23:53 |
| lifeless | # set DIB_REPOLOCATION_<project> for each of the projects cloned by devstack-vm-gate-wrap.sh | 23:54 |
| lifeless | # built images will then pull git repository dependencies from local disk. | 23:54 |
| lifeless | for GITDIR in $(ls -d /opt/stack/new/*/.git) ; do | 23:54 |
| lifeless | toci_devtest.sh | 23:54 |
| mordred | oh. you're being driven by d-g still. gotcha | 23:56 |
| lifeless | we don't use d-g at all, but we have common layout | 23:56 |
| mordred | what does the copy from workspace-cache to opt/stack/new then? | 23:56 |
| lifeless | we do use d-v-g-wrap | 23:57 |
| lifeless | cp devstack-gate/devstack-vm-gate-wrap.sh ./safe-devstack-vm-gate-wrap.sh | 23:57 |
| mordred | that's what I mean | 23:57 |
| mordred | meant | 23:57 |
| lifeless | ./safe-devstack-vm-gate-wrap.sh | 23:57 |
| lifeless | ok | 23:57 |
| * mordred groks now | 23:57 | |
| lifeless | so yes ^ | 23:57 |
| lifeless | from ./modules/openstack_project/files/jenkins_job_builder/config/tripleo.yaml | 23:57 |
| lifeless | omg windows 19G for the base install. WTF | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
