| *** CaptTofu_ has joined #tripleo | 00:09 | |
| *** marun has quit IRC | 00:11 | |
| *** marun has joined #tripleo | 00:11 | |
| *** marun has quit IRC | 00:12 | |
| *** nati_ueno has quit IRC | 00:14 | |
| *** xuhaiwei has joined #tripleo | 00:15 | |
| lifeless | oh wow proliant. Overridingthe IPMI boot source option. win. | 00:20 |
|---|---|---|
| *** spzala has quit IRC | 00:34 | |
| tchaypo | well that was a miserable driving lesson | 00:36 |
| tchaypo | any news about the overcloud? | 00:36 |
| *** Penick has quit IRC | 00:38 | |
| *** CaptTofu_ has quit IRC | 00:39 | |
| lifeless | tchaypo: rainy and miserable | 00:40 |
| *** yamahata has quit IRC | 00:41 | |
| openstackgerrit | James Polley proposed a change to openstack-infra/tripleo-ci: Add a vlan check https://review.openstack.org/100078 | 00:42 |
| openstackgerrit | OpenStack Proposal Bot proposed a change to openstack/os-apply-config: Updated from global requirements https://review.openstack.org/99083 | 00:46 |
| openstackgerrit | OpenStack Proposal Bot proposed a change to openstack/os-cloud-config: Updated from global requirements https://review.openstack.org/93253 | 00:46 |
| openstackgerrit | OpenStack Proposal Bot proposed a change to openstack/os-collect-config: Updated from global requirements https://review.openstack.org/96234 | 00:46 |
| *** TravT has quit IRC | 00:46 | |
| openstackgerrit | OpenStack Proposal Bot proposed a change to openstack/os-refresh-config: Updated from global requirements https://review.openstack.org/96235 | 00:46 |
| openstackgerrit | James Polley proposed a change to openstack-infra/tripleo-ci: Add a vlan check https://review.openstack.org/100078 | 00:49 |
| *** CaptTofu_ has joined #tripleo | 00:50 | |
| lifeless | tchaypo: let clarkb know when you've fixed up the commit | 00:57 |
| *** rwsu has quit IRC | 00:58 | |
| *** chuckC has quit IRC | 01:17 | |
| *** matsuhashi has joined #tripleo | 01:17 | |
| *** saurabhs has left #tripleo | 01:22 | |
| *** yamahata has joined #tripleo | 01:26 | |
| *** eghobo has quit IRC | 01:28 | |
| *** fandi has quit IRC | 01:31 | |
| *** nosnos has joined #tripleo | 01:31 | |
| *** spzala has joined #tripleo | 01:37 | |
| *** chuckC has joined #tripleo | 01:38 | |
| *** weshay has joined #tripleo | 01:38 | |
| *** CaptTofu_ has quit IRC | 01:42 | |
| *** yjiang5 is now known as yjiang5_away | 01:43 | |
| lifeless | tchaypo: any luck with local reproduction of the issue ? | 01:43 |
| *** giulivo has quit IRC | 01:44 | |
| lifeless | ok -> dr. | 01:49 |
| *** ramishra has joined #tripleo | 01:50 | |
| *** weshay has quit IRC | 02:01 | |
| greghaynes | tchaypo: I rather liked the sleep 1 vlan test :p | 02:04 |
| *** mestery has joined #tripleo | 02:19 | |
| *** jml has quit IRC | 02:21 | |
| *** jml has joined #tripleo | 02:25 | |
| *** matsuhashi has quit IRC | 02:29 | |
| *** matsuhashi has joined #tripleo | 02:32 | |
| *** spzala has quit IRC | 02:37 | |
| *** untriaged-bot has joined #tripleo | 03:00 | |
| untriaged-bot | Untriaged bugs so far: | 03:00 |
| untriaged-bot | https://bugs.launchpad.net/diskimage-builder/+bug/1329746 | 03:00 |
| uvirtbot | Launchpad bug 1329746 in diskimage-builder "disk-image-create should allow sending compat flags to qemu-img" [Undecided,In progress] | 03:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1329238 | 03:00 |
| *** matsuhashi has quit IRC | 03:00 | |
| *** untriaged-bot has quit IRC | 03:00 | |
| uvirtbot | Launchpad bug 1329238 in tripleo "OVS isn't persisting mac addresses on OVS bridges" [Undecided,In progress] | 03:00 |
| *** CaptTofu_ has joined #tripleo | 03:02 | |
| tchaypo | nope | 03:12 |
| tchaypo | my first try failed with the same issue as yesterday | 03:12 |
| tchaypo | so i ran it again with -c and this time it ran to completion while I was away having lunch | 03:12 |
| greghaynes | Yes, I also didnt run into it locally :/ | 03:12 |
| tchaypo | but I did do it with --no-undercloud though | 03:13 |
| greghaynes | I did full run | 03:13 |
| tchaypo | because somepone is making changes that mean I no longer have the ability to do a full run *innocent look* | 03:13 |
| greghaynes | :) Just wait until UNDERCLOUD_SCALE=3 | 03:13 |
| greghaynes | I do hope you can actually make a full run, still | 03:15 |
| tchaypo | We'll see. *shrug* | 03:16 |
| tchaypo | if we can make progress on folding the seed into the undercloud and then run one cloud in full-HA and hte other degraded... | 03:16 |
| tchaypo | btw SpamapS - thanks | 03:17 |
| tchaypo | I'm going to give a talk about "the tripleo project" in the near future | 03:18 |
| greghaynes | well, you could always just set OVERCLOUD_CONTROLSCALE=1 for your box | 03:18 |
| tchaypo | and start by correcting myself and explaining that it's not a project but a program | 03:18 |
| tchaypo | greghaynes: yeah, or that | 03:18 |
| tchaypo | but then I expect that when I run into problems I'll be told "meh, untested config, you're on your own" | 03:18 |
| StevenK | OVERCLOUD_CONTROLSCALE=1 is the current method | 03:19 |
| tchaypo | no-HA is the current method | 03:19 |
| tchaypo | the latter needs to change, so the former is going to | 03:19 |
| greghaynes | oh no, AIUI our decision was to support no-ha, or 3 node | 03:19 |
| greghaynes | (I also hope 5 node) | 03:19 |
| tchaypo | to be clear, I'm not opposed to the change, just not sure how I'll work with it | 03:20 |
| tchaypo | but that's something I'll be work out once it lands | 03:20 |
| greghaynes | I do think its important we do support a non-ha setup which minimises ram needs | 03:20 |
| vinsh | Hi dshulyak , do you see your review: https://review.openstack.org/#/c/92550/9 changing or staying in light of https://review.openstack.org/#/c/61376/5 ? | 03:20 |
| greghaynes | StevenK: referring to https://review.openstack.org/#/c/100000/ | 03:20 |
| tchaypo | so about bug 1292105 | 03:21 |
| uvirtbot | Launchpad bug 1292105 in tripleo "CI failed pinging overcloud instance" [Critical,Triaged] https://launchpad.net/bugs/1292105 | 03:21 |
| greghaynes | vinsh: pretty sure 92550 isnt neeeded | 03:21 |
| tchaypo | we can't reproduce locally. What's our next step | 03:21 |
| vinsh | greghaynes, thought so, just checking | 03:22 |
| greghaynes | tchaypo: is there some way to prevent a CI run like that from getting completed so you can poke at it? | 03:22 |
| * tchaypo doesn't know | 03:22 | |
| greghaynes | because it seems like we dont have any leads other than 'the network dun work' | 03:22 |
| clarkb | we can hold a node or node(s) | 03:23 |
| clarkb | then poke at them later | 03:23 |
| clarkb | if the failure is intermittent that becomes less useful | 03:23 |
| tchaypo | If I remember from sunnyvale, someone whose name I don't remember was talking about the test environment and my memory is that it might be possible to figure out which nodes were used and then jump on them quickly before they get recycled | 03:23 |
| greghaynes | clarkb: its consistent | 03:23 |
| tchaypo | clarkb: as far as we can tell it's pretty consistent on f20 and ubuntu both | 03:23 |
| clarkb | actually that may not work | 03:23 |
| greghaynes | tchaypo: that was derekh | 03:23 |
| clarkb | because the gearman lock will be released | 03:24 |
| clarkb | so I don't actually know how to do that | 03:24 |
| greghaynes | hrm | 03:24 |
| tchaypo | every time i hear the name derekh, the face I see is sdague. unless it's the other way around. | 03:24 |
| clarkb | so it is probably better for one of you to just grab a gearman loc kand run the test | 03:24 |
| tchaypo | so I guess what we could do is find the spot that does the ping | 03:24 |
| tchaypo | oh, right. | 03:24 |
| tchaypo | clarkb: do you have time to walk us through that? | 03:25 |
| greghaynes | welp, maybe make a review int toci which runs a whole lot of network debug commands, outputs them to files, then tars them up | 03:25 |
| greghaynes | so we can check it out | 03:25 |
| clarkb | tchaypo: uh its like 99% you guys | 03:25 |
| clarkb | tchaypo: boot node in test VM cloud, run whatever you run to test | 03:25 |
| StevenK | tchaypo: Did you sort out the documenation for --no-undercloud ? | 03:25 |
| tchaypo | What is this "test vm cloud" of which you speak? | 03:25 |
| tchaypo | StevenK: no. | 03:25 |
| clarkb | tchaypo: so the way the CI system work is there is a cloud that nodepool talks to. It boots nodes with public and internal interfaces | 03:26 |
| StevenK | Sprinkle spelling to taste. | 03:26 |
| clarkb | via the internal interface it talks to gearman and test clusters and gets lock and runs test | 03:26 |
| tchaypo | StevenK: because i want to make that review depend on my "fix all the errors" patch, which depends on the other patch, which currently can't land because of but 1292105 | 03:26 |
| clarkb | tchaypo: so if you boot a node on the same network you can run the test as if you were jenkins | 03:27 |
| tchaypo | clarkb: So to tsart this off, I need to ssh into one of the bastions? | 03:27 |
| StevenK | tchaypo: So? A pipeline of patches is not something to be avoided | 03:27 |
| tchaypo | or is this what the hpcs vpn is for? | 03:27 |
| clarkb | tchaypo: oh this might be a completely different CI infra | 03:27 |
| tchaypo | StevenK: sure, but 1292105 is blocking everyone, it's a higher priority | 03:27 |
| *** nosnos has quit IRC | 03:27 | |
| clarkb | tchaypo: we use zero bastions and zero VPNs to run upstream tests | 03:28 |
| clarkb | unless of course you treat the test VM connected to jenkins as a bastion | 03:28 |
| tchaypo | Ah. | 03:28 |
| tchaypo | which is why you're being generic. | 03:28 |
| clarkb | yes I have no idea what you are talking about when you say bastion or vm :) | 03:28 |
| tchaypo | very good :) | 03:28 |
| tchaypo | What would I need to do to get started with a setup that can run these tests? | 03:29 |
| tchaypo | I think I've seen this done once before but I don't remember how it was done | 03:29 |
| clarkb | you need at least 5 VMs | 03:29 |
| clarkb | or 5 phsical nodes | 03:30 |
| clarkb | or a mix | 03:30 |
| tchaypo | 5 VMs is easy enough | 03:30 |
| tchaypo | 2Gb each? | 03:30 |
| tchaypo | sounds a lot like the standard setup | 03:30 |
| clarkb | no clue that is all you guys | 03:30 |
| tchaypo | haha | 03:30 |
| tchaypo | you'd think we'd know what we do. | 03:30 |
| clarkb | like really we have so little insight into what happens | 03:31 |
| clarkb | my job is to run the test hundreds/thousands/millions of times per day | 03:31 |
| clarkb | it isn't really to understand the intricate details :P | 03:31 |
| clarkb | anyways I think one of the nodes needs 8GB for DIB | 03:31 |
| clarkb | maybe 16GB | 03:31 |
| tchaypo | from memory the main difference between what I've been doing and CI is that CI uses toci_devtest.sh rather than devtest.sh, and it skips a few steps because it uses pre-built environments | 03:31 |
| tchaypo | I'm going to let my current run go through to completion (or failure) and then see if I can reproduce the last few steps of toci_devtest.sh | 03:32 |
| clarkb | but you register the other 4 nodes in a gearman server that the big 5th node talks to and runs tests on | 03:32 |
| clarkb | so the 5th node is a like a control node and the other four are the machines that get properly tested on | 03:32 |
| clarkb | in the upstream setup infra boots the fifth node but does not do anything with gearman or the test clusters of four nodes | 03:33 |
| lifeless | tchaypo: how much memory is your browser using? | 03:38 |
| lifeless | greghaynes: 5-node, 7-node etc I presume. | 03:39 |
| lifeless | clarkb: tchaypo: you have account on the cloud | 03:41 |
| *** CaptTofu_ has quit IRC | 03:42 | |
| lifeless | tchaypo: what you need is a) your overcloud credentials for ci-overcloud.rh1.tripleo.org | 03:43 |
| lifeless | tchaypo: b) spawn a vm there, c) off you go; but the confusion you're to run into is that the images from nodepool won't be accessible to you | 03:45 |
| lifeless | tchaypo: so you may want to use your admin account | 03:45 |
| lifeless | tchaypo: to get that, ssh to bastion.rh1.tripleo.org using the port from the cd admin spreadsheet | 03:46 |
| lifeless | tchaypo: from there (see the spreadsheet) you ssh to root@ the seed host, and on there are the build scripts to deploy stuff, which possibly need the os-assert-users stuff run again to create your creds | 03:46 |
| greghaynes | basically just kicking off a test run manually? | 03:47 |
| lifeless | yeah | 03:48 |
| lifeless | it will be a little tricky | 03:48 |
| lifeless | but should be doable | 03:49 |
| lifeless | that said, saving the node is the easiest thing | 03:49 |
| clarkb | lifeless: it won't help as the lock will be released | 03:49 |
| lifeless | just put up a toci patch that traps and sleeps for like 4 hours | 03:49 |
| *** jrist has quit IRC | 03:49 | |
| clarkb | oh you could do something horrible like that | 03:49 |
| greghaynes | hahaha | 03:50 |
| clarkb | the test will timeout before you hit that | 03:50 |
| clarkb | so you will have ~ (time to hit failure - time left until timeout) time to debug | 03:51 |
| *** eghobo has joined #tripleo | 03:51 | |
| lifeless | right | 03:52 |
| *** matsuhashi has joined #tripleo | 03:53 | |
| lifeless | clarkb: which of bnemec comments did I not address? | 03:54 |
| clarkb | lifeless: you addressed it | 03:54 |
| clarkb | did you not address one? | 03:54 |
| lifeless | oh | 03:54 |
| lifeless | gerttyshowed me your patc set 9 comment as more recentthan your 10 | 03:55 |
| clarkb | huh | 03:55 |
| lifeless | I had it open, its fine on reentry | 03:55 |
| *** tzumainn has quit IRC | 03:59 | |
| *** mestery has quit IRC | 04:00 | |
| tchaypo | ho hum. usual suite of usernames not working on the bastion | 04:00 |
| *** mestery has joined #tripleo | 04:02 | |
| lifeless | tchaypo: tried tchaypo ? | 04:04 |
| *** mestery has quit IRC | 04:04 | |
| lifeless | tchaypo: huh its not been added | 04:06 |
| lifeless | and I don't know my own password to do sudo and run the script | 04:06 |
| lifeless | tchaypo: ping dprince via email please | 04:06 |
| *** nosnos has joined #tripleo | 04:07 | |
| tchaypo | wheee | 04:07 |
| lifeless | so I think the sleep 36000 patch is the thing to do | 04:09 |
| lifeless | before clarkb is asleep | 04:09 |
| clarkb | no I am already asleep | 04:09 |
| clarkb | (I don't ahve access to stuff anymore) | 04:09 |
| lifeless | clarkb: does sergey have access and know-how ? | 04:10 |
| clarkb | yes he should | 04:10 |
| clarkb | he has access and is pretty nimble with zuul/nodepool | 04:10 |
| lifeless | tchaypo: ^' | 04:10 |
| tchaypo | greghaynes: I'm assuming you have trouble getting access to rh1 bastion as well? | 04:12 |
| *** matsuhashi has quit IRC | 04:12 | |
| greghaynes | tchaypo: no access till cd-admins review gets merged | 04:12 |
| greghaynes | so, no idea :) | 04:12 |
| *** matsuhashi has joined #tripleo | 04:12 | |
| tchaypo | lifeless: clarkb: who is sergey, and why do we need his access? Can I just give a new patch on 100374 that includes the sleep? | 04:12 |
| tchaypo | greghaynes: oh yeah, that. | 04:12 |
| tchaypo | CC team, or just dan? | 04:15 |
| clarkb | he is another infra rooter and you need an infra rooter to hold test nodes for you | 04:15 |
| tchaypo | ah. But if we're doing the sleep 360000 thing, do we still need him? | 04:15 |
| *** jml has quit IRC | 04:16 | |
| clarkb | yes, otherwise you can't get on the node (unless you can with hax then nvm) | 04:16 |
| clarkb | or yo ucould bypass test VM and got on test cluster directly? | 04:17 |
| *** matsuhashi has quit IRC | 04:17 | |
| tchaypo | oh | 04:17 |
| tchaypo | we're talking about SergeyLukjanov | 04:17 |
| tchaypo | right now I think I'm going to give up on rh1 and jump in hp1, since I believe this is happening there too | 04:17 |
| tchaypo | yeah, that sounds feasible, if I know where to get credentials and IPs for it | 04:18 |
| *** jml has joined #tripleo | 04:21 | |
| *** matsuhashi has joined #tripleo | 04:21 | |
| *** rakesh_hs2 has joined #tripleo | 04:28 | |
| *** matsuhashi has quit IRC | 04:36 | |
| *** matsuhashi has joined #tripleo | 04:36 | |
| *** matsuhashi has quit IRC | 04:39 | |
| *** eghobo has quit IRC | 04:39 | |
| *** matsuhashi has joined #tripleo | 04:39 | |
| *** eghobo has joined #tripleo | 04:40 | |
| tchaypo | okay, I'm on the seed vm in hp1 | 04:50 |
| tchaypo | which is not where I want to be | 04:51 |
| *** ramishra has quit IRC | 05:00 | |
| *** ramishra has joined #tripleo | 05:00 | |
| *** ramishra has quit IRC | 05:03 | |
| *** ramishra has joined #tripleo | 05:03 | |
| *** jml has quit IRC | 05:09 | |
| *** matsuhashi has quit IRC | 05:11 | |
| *** matsuhashi has joined #tripleo | 05:11 | |
| * tchaypo hunts for "os-assert-users" stuff | 05:15 | |
| *** matsuhashi has quit IRC | 05:16 | |
| *** jml has joined #tripleo | 05:16 | |
| *** matsuhashi has joined #tripleo | 05:17 | |
| *** lazy_prince has joined #tripleo | 05:17 | |
| *** matsuhashi has quit IRC | 05:24 | |
| *** matsuhashi has joined #tripleo | 05:24 | |
| *** rdopieralski has joined #tripleo | 05:40 | |
| *** akuznetsov has joined #tripleo | 05:43 | |
| *** dshulyak_ has joined #tripleo | 05:50 | |
| *** e0ne has joined #tripleo | 05:51 | |
| *** rcarrillocruz has joined #tripleo | 05:51 | |
| *** rcarrill` has quit IRC | 05:53 | |
| *** sseago_ has quit IRC | 05:59 | |
| tchaypo | okay, I've spent most of today chasing this bug and I still can't reproduce. I'm going to drop it now and arrange pycon travel and get some reviews done | 06:00 |
| *** e0ne has quit IRC | 06:01 | |
| *** e0ne has joined #tripleo | 06:01 | |
| *** bmahalakshmi has joined #tripleo | 06:04 | |
| *** wendar has quit IRC | 06:04 | |
| *** wendar_ has joined #tripleo | 06:04 | |
| *** e0ne has quit IRC | 06:05 | |
| *** jprovazn has joined #tripleo | 06:14 | |
| *** dshulyak_ has quit IRC | 06:16 | |
| *** jml has quit IRC | 06:19 | |
| *** pcrews has quit IRC | 06:19 | |
| *** openstackgerrit has quit IRC | 06:19 | |
| *** jml has joined #tripleo | 06:23 | |
| *** pcrews has joined #tripleo | 06:23 | |
| *** openstackgerrit has joined #tripleo | 06:23 | |
| *** jtomasek has joined #tripleo | 06:36 | |
| *** jcoufal has joined #tripleo | 06:37 | |
| jprovazn | Hi, so heat convergence engine and observer would be used for failed service recover on node level? e.g. haproxy service fails -> a notify agent sends a message -> the notification is catched by heat -> heat restarts whole instance (=VM/machine) | 06:45 |
| jprovazn | ? | 06:45 |
| *** ramishra has quit IRC | 06:48 | |
| *** ramishra has joined #tripleo | 06:51 | |
| tchaypo | Ng: are you around? | 06:54 |
| *** mrunge has joined #tripleo | 06:55 | |
| *** hewbrocca has joined #tripleo | 07:02 | |
| openstackgerrit | Michael Chapman proposed a change to openstack/tripleo-incubator: Ensure add-apt-reopsitory is available https://review.openstack.org/100794 | 07:04 |
| openstackgerrit | Michael Chapman proposed a change to openstack/tripleo-incubator: Ensure add-apt-repository is available https://review.openstack.org/100794 | 07:05 |
| *** matsuhashi has quit IRC | 07:08 | |
| *** matsuhas_ has joined #tripleo | 07:08 | |
| *** eguz has joined #tripleo | 07:08 | |
| jprovazn | lifeless: you still around? | 07:10 |
| *** eguz has quit IRC | 07:11 | |
| *** jcoufal has quit IRC | 07:12 | |
| *** eghobo has quit IRC | 07:12 | |
| *** fabbione has joined #tripleo | 07:12 | |
| *** jcoufal has joined #tripleo | 07:13 | |
| fabbione | lifeless: morning man | 07:13 |
| fabbione | lifeless: you happen to be around by any chance? | 07:13 |
| *** bmahalakshmi has quit IRC | 07:13 | |
| lifeless | fabbione: hi | 07:13 |
| lifeless | jprovazn: hi | 07:13 |
| lifeless | fabbione: welcome to this little corner of the 'net | 07:13 |
| fabbione | lifeless: hey.. so I am sitting here in Brno with some of the O^3 guys | 07:13 |
| fabbione | lifeless: and they were trying to explain to me some of the HA requirements you guys have for under/over cloud (or whatever is called in your little corner of the net) | 07:14 |
| fabbione | lifeless: thx :) | 07:14 |
| lifeless | jprovazn: convergence, perhaps, but more likely we'll have some layering to make dynamic to have that | 07:14 |
| fabbione | lifeless: so basically the thing is that you seem to have some kind of ideas on how it should all glue together | 07:14 |
| fabbione | lifeless: that from my HA experience seems a bit off. now that said, it could be due to a language barrier, or even guys not knowing HA concepts etc.. (aka I am not frigging blaming anybody :P) | 07:15 |
| fabbione | lifeless: do you think we can arrange something like a google hangout or a skype so that we can sync a bit on come stuff? | 07:16 |
| fabbione | lifeless: i'd like to understand how you guys are planning to provide HighAvailablity and hopefully I can provide you with some suggestions after doing HA for so long | 07:16 |
| lifeless | fabbione: cool | 07:16 |
| lifeless | fabbione: turns out today I managed to come down with some stomach thing | 07:16 |
| fabbione | lifeless: not today no.. next week? | 07:17 |
| lifeless | fabbione: so I'm feeling a bit blergh | 07:17 |
| lifeless | fabbione: what tz are you typically in ? | 07:17 |
| fabbione | lifeless: EU | 07:17 |
| lifeless | next week I'm in PST | 07:17 |
| fabbione | whenever fits... | 07:17 |
| lifeless | I do however have a meeting in 45m I can't escape | 07:17 |
| *** bmahalakshmi has joined #tripleo | 07:17 | |
| fabbione | i was trying to be at the summit in RDU but flight tickets are just prohibitive | 07:17 |
| lifeless | tell you what, gimme 10m and I'll be online in hangouts | 07:17 |
| fabbione | lifeless: 09:17 < fabbione> lifeless: not today no.. next week? | 07:18 |
| fabbione | i cna't do it now :) i have a meeting coming up | 07:18 |
| *** ramishra has quit IRC | 07:18 | |
| fabbione | i just wanted to be on your radar basically :D | 07:18 |
| lifeless | fabbione: ah | 07:18 |
| lifeless | ok | 07:18 |
| fabbione | so PST are you going be in meetings all week? | 07:18 |
| lifeless | so yeah, around this time is fine for me except next week i'm in our mothership talking with the enterprise folk... about ha and stuff | 07:18 |
| lifeless | I should have time, but I am not sure just yet | 07:19 |
| fabbione | yeah.. that would be a late afternoon meeting for me | 07:19 |
| fabbione | that can easily be arranged | 07:19 |
| lifeless | cool | 07:19 |
| fabbione | and it might even be better if you have other HA folks on your team | 07:19 |
| lifeless | so lets ping here and tee-up adhoc | 07:19 |
| fabbione | totally | 07:19 |
| fabbione | should we aim for Tuesday? | 07:19 |
| fabbione | just to have a target date/time | 07:19 |
| lifeless | fabbione: can you tell me what stuff is concerning - like is it the service monitor impl(we don't have one specced yet), or the use of galera/virtual ips/haproxy ? | 07:20 |
| Ng | tchaypo: Only very slightly | 07:20 |
| Ng | Hey fabbione :) | 07:20 |
| fabbione | so there are 9h difference.. i can 20:00 that would be 11am your time | 07:20 |
| lifeless | fabbione: let me do some tz math to figure out whats up that week | 07:20 |
| fabbione | Ng: yo! | 07:20 |
| fabbione | lifeless: i think the major problem is that i was not able to understand what was discussed previously | 07:21 |
| fabbione | lifeless: so start with a background/brainstorm so i get the picture of what you guys have in your mind | 07:21 |
| fabbione | lifeless: and then perhaps pitch in some ideas from the HA world to avoid repeating the same mistkaes of many other projects | 07:21 |
| lifeless | fabbione: ok sure | 07:22 |
| fabbione | lifeless: I was asked to basically sanity check HA.. but it's hard to do it without proper background :) | 07:22 |
| lifeless | ack | 07:23 |
| openstackgerrit | Michael Chapman proposed a change to openstack/tripleo-incubator: Ensure add-apt-repository is available https://review.openstack.org/100794 | 07:27 |
| *** athomas has joined #tripleo | 07:27 | |
| lifeless | tchaypo: there is no hp1 atm | 07:39 |
| lifeless | tchaypo: since per meeting this morning, bugs. | 07:39 |
| openstackgerrit | Ramakrishnan G proposed a change to openstack/diskimage-builder: Add virtual media boot support in ramdisk element. https://review.openstack.org/97143 | 07:43 |
| *** e0ne has joined #tripleo | 07:48 | |
| tchaypo | lifeless: well poop. What about hp2? | 07:52 |
| lifeless | tchaypo: if we can't bring up hp1, what makes you think the brand new region is ready ? :) | 08:00 |
| *** e0ne has quit IRC | 08:01 | |
| *** e0ne has joined #tripleo | 08:03 | |
| *** jistr has joined #tripleo | 08:03 | |
| *** yamahata has quit IRC | 08:05 | |
| *** yamahata has joined #tripleo | 08:05 | |
| *** regebro has quit IRC | 08:08 | |
| *** andreaf_ has quit IRC | 08:11 | |
| *** jml has quit IRC | 08:12 | |
| *** jml has joined #tripleo | 08:16 | |
| *** jml has joined #tripleo | 08:16 | |
| *** ramishra has joined #tripleo | 08:18 | |
| *** lucasagomes has joined #tripleo | 08:28 | |
| *** regebro has joined #tripleo | 08:37 | |
| *** IvanBerezovskiy has joined #tripleo | 08:41 | |
| *** jp_at_hp has joined #tripleo | 08:48 | |
| *** giulivo has joined #tripleo | 08:50 | |
| *** pelix has joined #tripleo | 08:50 | |
| *** dtantsur|afk is now known as dtantsur | 08:56 | |
| *** andreaf has joined #tripleo | 08:57 | |
| *** hewbrocca has quit IRC | 08:57 | |
| *** andreaf_ has joined #tripleo | 08:58 | |
| tchaypo | Blind optimism | 08:59 |
| *** untriaged-bot has joined #tripleo | 09:01 | |
| untriaged-bot | Untriaged bugs so far: | 09:01 |
| untriaged-bot | https://bugs.launchpad.net/diskimage-builder/+bug/1329746 | 09:01 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1329238 | 09:01 |
| uvirtbot | Launchpad bug 1329746 in diskimage-builder "disk-image-create should allow sending compat flags to qemu-img" [Undecided,In progress] | 09:01 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1331337 | 09:01 |
| uvirtbot | Launchpad bug 1329238 in tripleo "OVS isn't persisting mac addresses on OVS bridges" [Undecided,In progress] | 09:01 |
| *** untriaged-bot has quit IRC | 09:01 | |
| uvirtbot | Launchpad bug 1331337 in tripleo "install_dependencies fails if python-software-properties is not installed." [Undecided,New] | 09:01 |
| *** matsuhas_ has quit IRC | 09:01 | |
| *** andreaf has quit IRC | 09:01 | |
| *** matsuhashi has joined #tripleo | 09:09 | |
| *** martyntaylor has joined #tripleo | 09:09 | |
| *** markmc has joined #tripleo | 09:09 | |
| openstackgerrit | Michael Chapman proposed a change to openstack/tripleo-incubator: Ensure add-apt-repository is available https://review.openstack.org/100794 | 09:12 |
| *** ramishra has quit IRC | 09:16 | |
| openstackgerrit | xu-haiwei proposed a change to openstack/tripleo-incubator: Do not set NeutronControlPlaneID when building image only https://review.openstack.org/100828 | 09:17 |
| *** mkerrin has quit IRC | 09:22 | |
| *** loki184 has joined #tripleo | 09:22 | |
| fabbione | /save/ | 09:25 |
| fabbione | ops | 09:25 |
| ohadlevy | fabbione: panic :) | 09:31 |
| *** hewbrocca has joined #tripleo | 09:32 | |
| fabbione | ohadlevy: ehehe | 09:36 |
| *** matsuhashi has quit IRC | 09:38 | |
| fabbione | 7win 2 | 09:41 |
| *** fabbione has quit IRC | 09:46 | |
| *** fabbione has joined #tripleo | 09:47 | |
| *** fabbione has quit IRC | 09:48 | |
| *** fabbione has joined #tripleo | 09:49 | |
| *** ifarkas_ has quit IRC | 09:51 | |
| *** ccrouch1 has joined #tripleo | 09:53 | |
| *** ccrouch has quit IRC | 09:54 | |
| *** fandi has joined #tripleo | 09:55 | |
| *** rakesh_hs2 has quit IRC | 09:56 | |
| *** ramishra has joined #tripleo | 09:57 | |
| *** rakesh_hs2 has joined #tripleo | 09:57 | |
| *** jcoufal has quit IRC | 09:59 | |
| *** matsuhashi has joined #tripleo | 10:01 | |
| *** matsuhashi has quit IRC | 10:06 | |
| *** giulivo is now known as gfidente | 10:07 | |
| *** matsuhashi has joined #tripleo | 10:08 | |
| *** ramishra has quit IRC | 10:09 | |
| *** akrivoka has joined #tripleo | 10:13 | |
| *** rlandy has joined #tripleo | 10:13 | |
| *** ifarkas has joined #tripleo | 10:15 | |
| *** matsuhashi has quit IRC | 10:17 | |
| *** matsuhashi has joined #tripleo | 10:18 | |
| *** ifarkas has quit IRC | 10:26 | |
| *** mkerrin has joined #tripleo | 10:27 | |
| *** ifarkas has joined #tripleo | 10:29 | |
| *** yamahata_ has joined #tripleo | 10:31 | |
| *** fandi has quit IRC | 10:38 | |
| *** gcha has joined #tripleo | 10:38 | |
| openstackgerrit | Stuart McLaren proposed a change to openstack/tripleo-heat-templates: Add nova_compute_ips to overcloud-source.yaml https://review.openstack.org/100848 | 10:39 |
| *** jcoufal has joined #tripleo | 10:43 | |
| martyntaylor | root | 10:44 |
| martyntaylor | wrong window :P | 10:44 |
| tchaypo | it would have been more fun if it was the next line | 10:56 |
| *** morazi has joined #tripleo | 10:58 | |
| *** yamahata has quit IRC | 10:59 | |
| openstackgerrit | Stuart McLaren proposed a change to openstack/tripleo-image-elements: Basic stunnel firewall configuration https://review.openstack.org/100853 | 11:00 |
| *** bmahalakshmi has quit IRC | 11:02 | |
| *** ramishra has joined #tripleo | 11:10 | |
| *** morazi has quit IRC | 11:10 | |
| *** Haomeng has joined #tripleo | 11:13 | |
| *** ramishra has quit IRC | 11:14 | |
| *** dshulyak has quit IRC | 11:14 | |
| *** e0ne has quit IRC | 11:15 | |
| *** Haomeng has quit IRC | 11:16 | |
| *** bogdando has quit IRC | 11:16 | |
| *** rpodolyaka has quit IRC | 11:16 | |
| *** regebro is now known as regebro|lunch | 11:24 | |
| *** e0ne has joined #tripleo | 11:25 | |
| *** bogdando has joined #tripleo | 11:26 | |
| *** dshulyak has joined #tripleo | 11:26 | |
| *** rpodolyaka has joined #tripleo | 11:27 | |
| *** d0ugal has quit IRC | 11:33 | |
| *** d0ugal has joined #tripleo | 11:34 | |
| *** martyntaylor has quit IRC | 11:35 | |
| *** martyntaylor has joined #tripleo | 11:42 | |
| *** CaptTofu_ has joined #tripleo | 11:47 | |
| *** matsuhashi has quit IRC | 11:48 | |
| *** sseago has joined #tripleo | 11:50 | |
| *** matsuhashi has joined #tripleo | 11:51 | |
| *** yolanda has joined #tripleo | 11:53 | |
| yolanda | hi, i'm having troubles in the check of a change, getting these errors all the time: http://logs.openstack.org/24/99924/6/check-tripleo/check-tripleo-undercloud-precise/c993d36, http://logs.openstack.org/24/99924/6/check-tripleo/check-tripleo-overcloud-f20/b3c6306 | 11:53 |
| *** matsuhashi has quit IRC | 11:56 | |
| *** matsuhashi has joined #tripleo | 11:56 | |
| *** dprince has joined #tripleo | 11:57 | |
| *** weshay has joined #tripleo | 11:58 | |
| *** lucasagomes is now known as lucas-hungry | 11:59 | |
| *** matsuhashi has quit IRC | 12:01 | |
| *** jistr is now known as jistr|english | 12:02 | |
| *** bogdando has quit IRC | 12:02 | |
| *** bogdando has joined #tripleo | 12:03 | |
| *** e0ne_ has joined #tripleo | 12:03 | |
| *** tzumainn has joined #tripleo | 12:05 | |
| *** e0ne has quit IRC | 12:05 | |
| *** noslzzp has joined #tripleo | 12:06 | |
| *** jistr|mobi has joined #tripleo | 12:07 | |
| *** dshulyak has quit IRC | 12:08 | |
| *** rpodolyaka has quit IRC | 12:08 | |
| *** e0ne_ has quit IRC | 12:08 | |
| *** bogdando has quit IRC | 12:09 | |
| *** ramishra has joined #tripleo | 12:11 | |
| *** yamahata has joined #tripleo | 12:11 | |
| *** e0ne has joined #tripleo | 12:11 | |
| d0ugal | tzumainn: add_role on Deployment Plan.. it calls add_provider_resource on master_template and environment_file | 12:12 |
| *** dshulyak has joined #tripleo | 12:13 | |
| *** bogdando has joined #tripleo | 12:13 | |
| d0ugal | tzumainn: Oh, before I continue, I clarified the naming slightly Store and Model - model being an instance and store being the collection. | 12:13 |
| *** rpodolyaka has joined #tripleo | 12:13 | |
| d0ugal | tzumainn: So, then, the issue is, that DeploymentPlanStore needs to access it's own driver, master template's driver, environment files driver. | 12:14 |
| d0ugal | tzumainn: and over those three we have two drivers, so we can't just use the same one for all easily. | 12:15 |
| d0ugal | tzumainn: so wiring all of this up is getting me into a pickle | 12:15 |
| *** ramishra has quit IRC | 12:15 | |
| *** loki184 has quit IRC | 12:16 | |
| *** fandi has joined #tripleo | 12:17 | |
| openstackgerrit | Nicholas Randon proposed a change to openstack/tripleo-incubator: Improve readability of long JQ expression https://review.openstack.org/99681 | 12:19 |
| openstackgerrit | Nicholas Randon proposed a change to openstack/tripleo-incubator: Wait for os-collect-config to complete on the seed https://review.openstack.org/96515 | 12:19 |
| openstackgerrit | Nicholas Randon proposed a change to openstack/tripleo-incubator: Clean-up os-apply-config lines in devtest_seed.sh. https://review.openstack.org/99645 | 12:20 |
| tzumainn | d0ugal, hm, interesting | 12:23 |
| tzumainn | well, if DeploymentPlan, Template, and EnvironmentFile are all represented by their own model | 12:24 |
| tzumainn | then couldn't they each take care of their own single driver? | 12:24 |
| *** jdob has joined #tripleo | 12:27 | |
| d0ugal | tzumainn: Just getting you an example, I wrote some code for this. | 12:27 |
| d0ugal | tzumainn: https://etherpad.openstack.org/p/15bBoHOMny | 12:29 |
| d0ugal | Quick example way at the bottom. | 12:29 |
| *** morazi has joined #tripleo | 12:32 | |
| *** yfaykya has quit IRC | 12:34 | |
| *** yfaykya has joined #tripleo | 12:34 | |
| *** CaptTofu_ has quit IRC | 12:36 | |
| *** andreaf_ has quit IRC | 12:37 | |
| *** andreaf_ has joined #tripleo | 12:37 | |
| *** CaptTofu_ has joined #tripleo | 12:38 | |
| *** yfaykya has quit IRC | 12:38 | |
| tzumainn | d0ugal, wouldn't it be DeploymentPlanStore.create(<parameters for deployment plan>) | 12:39 |
| tzumainn | and within that, you'd ask TemplateStore to create a template, etc? | 12:39 |
| *** andreaf_ is now known as andreaf | 12:40 | |
| *** yfaykya has joined #tripleo | 12:40 | |
| d0ugal | tzumainn: like this? DeploymentPlanStore.create(..., master_template=template_store.create(), environment_file=env_store.create()) | 12:41 |
| d0ugal | ('...' would be name, description etc.) | 12:42 |
| tzumainn | d0ugal, something like that, yeah, except... | 12:43 |
| tzumainn | now that I think of it, when the DeploymentPlanStore is initialized, we could just set its template store and env_file store aloing with the plan_store | 12:43 |
| d0ugal | tzumainn: plan_store = DeploymentPlanStore(driver=sql_driver, environment_store=env_store, template_store=template_store)? | 12:45 |
| *** yfaykya has quit IRC | 12:45 | |
| d0ugal | Starts to feel a bit ugly :) | 12:45 |
| d0ugal | I might bug jdob for ideas | 12:45 |
| tzumainn | hm, maybe - but we're only initializing each store once, right? | 12:45 |
| d0ugal | True | 12:45 |
| d0ugal | It would probably be fine. | 12:46 |
| jdob | why need a driver if both sub-stores are specified? | 12:46 |
| tzumainn | you mean a driver for the deployment plan? | 12:46 |
| d0ugal | jdob: deployment plans need to be stored too | 12:46 |
| *** lucas-hungry is now known as lucasagomes | 12:46 | |
| *** yfaykya has joined #tripleo | 12:46 | |
| jdob | what's stored about them besides the env and template? the name and UUID? | 12:47 |
| d0ugal | description | 12:47 |
| d0ugal | and updated date | 12:47 |
| jdob | gotcha. description is the only field we could actually shove in the template, but i see your point | 12:47 |
| *** yfaykya has quit IRC | 12:51 | |
| *** julim has joined #tripleo | 12:51 | |
| *** fandi has quit IRC | 12:52 | |
| *** yfaykya has joined #tripleo | 12:53 | |
| tzumainn | jdob, hey, question just to make sure I understand correctly | 12:53 |
| tzumainn | when we add a provider resource to an environment file's resource_registry | 12:54 |
| *** rakesh_hs2 has quit IRC | 12:54 | |
| tzumainn | do we also have to add all the templates that the provider resource is dependent on? | 12:54 |
| jdob | no | 12:55 |
| tzumainn | lsmola__, ^ | 12:55 |
| jdob | see shadower's spec, there's an example in there of the compute template referring to the compute-config one | 12:55 |
| tzumainn | jdob, okay, that's what I thought | 12:55 |
| *** yfaykya has quit IRC | 12:58 | |
| *** yfaykya has joined #tripleo | 12:59 | |
| openstackgerrit | Dan Prince proposed a change to openstack/tripleo-image-elements: Some testing for os-is-bootstrap-host... https://review.openstack.org/100898 | 13:00 |
| lsmola__ | tzumainn: jdob well you have to list all the provider resources in resource registry | 13:01 |
| jdob | covered by when he said <tzumainn> when we add a provider resource to an environment file's resource_registry :D | 13:01 |
| *** yfaykya has quit IRC | 13:03 | |
| lsmola__ | tzumainn: jdob the shadowers example uses get_file, so no provider resource | 13:03 |
| lsmola__ | tzumainn: jdob which we can't use AFAIK when working with API | 13:04 |
| dprince | CI update: all undercloud jobs are now failing today. https://bugs.launchpad.net/tripleo/+bug/1331445 | 13:04 |
| uvirtbot | Launchpad bug 1331445 in tripleo "Refusing to bootstrap mysql cluster until role is known. (all Undercloud jobs fail)" [Critical,Triaged] | 13:04 |
| dprince | I'm running a test patch to see what os-is-bootstrap-host is getting from Heat metadata now | 13:04 |
| *** yfaykya has joined #tripleo | 13:05 | |
| dprince | I ran this manually on Fedora and it seemed to work fine. I'm wondering if something has changed that would require us to rebuild the testenv nodes? | 13:05 |
| shadower | lsmola__: we can. See steve baker's and mine comments | 13:05 |
| tzumainn | shadower, any chance you could clarify the apparent confusion ^ ? | 13:06 |
| tzumainn | oh, there you are | 13:06 |
| shadower | okay | 13:06 |
| tzumainn | sorry, I see that you're already on it : ) | 13:06 |
| *** regebro|lunch is now known as regebro | 13:06 | |
| shadower | so first: there are multiple ways to split templates off. One is with environment files, another is with get_file, yet another is with something like merge.py-lite | 13:07 |
| *** yamahata has quit IRC | 13:07 | |
| *** yamahata has joined #tripleo | 13:07 | |
| shadower | with get_file: heat client doesn't really do any template processing but it looks at the get-file instances and builds a dictionary of filename/contents | 13:07 |
| lsmola__ | shadower: so we will use get_file instead provider resources? | 13:08 |
| shadower | this dictionary is then passed to Heat API alongside with all the other stuff (the template contenst, environment file, parameters, etc.) | 13:08 |
| lsmola__ | shadower: last time I remember, we wanted to use provider resource for everything | 13:08 |
| shadower | so we could do the same thing ourselves -- build the map and pass it to Heat API | 13:08 |
| lsmola__ | shadower: actually there is files paramter, and it is used also for provider resoruces | 13:08 |
| shadower | lsmola__: that's a tiny detail. It can be either | 13:09 |
| lsmola__ | shadower: I mean you pass the file contect this way, always | 13:09 |
| shadower | get_file is slightly less verbose | 13:09 |
| *** yfaykya has quit IRC | 13:09 | |
| shadower | but we can go provide resources all the way if we decide to (this is all in the spec) | 13:09 |
| shadower | there's a lot of different ways how we can partition the heat templates. I can't list them all so I just provided one example (focusing on the main points -- what are roles and the main template going to look like) and mentioned other options in the text | 13:10 |
| lsmola__ | shadower: right, so we should be able to deal with both get_file, and provider resource dependencies of the template | 13:11 |
| *** yfaykya has joined #tripleo | 13:11 | |
| shadower | yeah | 13:11 |
| lsmola__ | tzumainn: ^ | 13:11 |
| tzumainn | well, there are two separate questions, I think | 13:13 |
| lsmola__ | tzumainn: btw. in the get_file: compute-config.yaml, the compute-config.yaml is just index for us, you can send any file content under this | 13:13 |
| lsmola__ | tzumainn: so it's similar to provider resoruces | 13:13 |
| shadower | yea | 13:13 |
| lsmola__ | tzumainn: you just don't need to list it in resource registry | 13:13 |
| tzumainn | so if we do add a provider resource to an environment file, do we actually have to also register the templates it depends on? | 13:14 |
| tzumainn | and b) is there a way to get the dependent templates for a provider resource | 13:14 |
| tzumainn | right? | 13:14 |
| *** yfaykya has quit IRC | 13:15 | |
| shadower | tzumainn: register where? | 13:17 |
| tzumainn | er, sorry, in the resource_registry in the environment file | 13:17 |
| *** yfaykya has joined #tripleo | 13:17 | |
| shadower | tzumainn: that depends how the templates are structured. A resource type can either be something like AAA::BBBB::CCCCC or a URL (e.g. file:///opt/stack/tht/compute.yaml or https://example.com/tht/compute.yaml) | 13:19 |
| shadower | putting URLs directly into the type field is discouraged but allowed | 13:19 |
| shadower | tzumainn: so the type-as-a-URL templates don't have to exist in the resource_registry | 13:19 |
| tzumainn | but the former does? | 13:20 |
| shadower | tzumainn: yeah | 13:20 |
| shadower | otherwise Heat doesn't know how to map the type to a template | 13:20 |
| tzumainn | so to take your compute.yaml example - if we register that as a provider resource in an environment file, we also have to register compute-config.yaml, correct? | 13:21 |
| *** yfaykya has quit IRC | 13:22 | |
| shadower | tzumainn: well that depends on how is compute.yaml using compute-config.yaml. If it's using a config resource with the template URL as its type then no. If you're using get_file than again, no. But if you want to assign a custom type to it (e.g. OS::TripleO::ComputeConfig) then yes | 13:22 |
| shadower | if you're using merge.py or an equivalent to mangle both templates into one, then again, no need for any resource_registry stuff | 13:22 |
| tzumainn | shadower, when you say assign a custom type to it, do you mean assign a custom type to it within compute.yaml? | 13:23 |
| shadower | tzumainn: exactly | 13:23 |
| *** yfaykya has joined #tripleo | 13:23 | |
| shadower | just like the master template would use a custom type for the role | 13:23 |
| *** jprovazn has quit IRC | 13:24 | |
| shadower | does that make things clearer? | 13:24 |
| tzumainn | shadower, definitely - thanks! | 13:24 |
| tzumainn | I love you | 13:24 |
| shadower | cool | 13:24 |
| shadower | tzumainn: I love you too | 13:24 |
| shadower | ♥ | 13:24 |
| * pbelanyi turns away | 13:25 | |
| shadower | pbelanyi: you look so cute when you blush | 13:25 |
| *** jesusaurus has quit IRC | 13:27 | |
| *** yfaykya has quit IRC | 13:27 | |
| shadower | jdob: posted a bunch of replies to your comments on the remove-merge.py spec | 13:28 |
| tzumainn | jdob, btw ^^ clarified a lot for me | 13:28 |
| shadower | jdob: I've another version ready but want to give you an opportunity to reply before pushin git | 13:28 |
| *** yfaykya has joined #tripleo | 13:29 | |
| *** CaptTofu_ has quit IRC | 13:30 | |
| *** mestery has joined #tripleo | 13:32 | |
| *** yfaykya has quit IRC | 13:34 | |
| *** jrist has joined #tripleo | 13:35 | |
| *** yfaykya has joined #tripleo | 13:35 | |
| jdob | shadower: I saw they were posted but haven't digested them yet, will do it shortly | 13:35 |
| shadower | jdob: no rush, just wanted to make sure you're aware of them | 13:36 |
| *** jesusaurus has joined #tripleo | 13:38 | |
| *** CaptTofu_ has joined #tripleo | 13:39 | |
| *** yfaykya has quit IRC | 13:40 | |
| *** yfaykya has joined #tripleo | 13:42 | |
| *** yolanda has quit IRC | 13:42 | |
| *** jistr|mobi has quit IRC | 13:42 | |
| *** jistr|english is now known as jistr | 13:42 | |
| *** jistr|mobi has joined #tripleo | 13:43 | |
| *** mestery has quit IRC | 13:44 | |
| *** yfaykya has quit IRC | 13:46 | |
| *** yfaykya has joined #tripleo | 13:48 | |
| *** ramishra has joined #tripleo | 13:51 | |
| *** spzala has joined #tripleo | 13:52 | |
| *** yfaykya has quit IRC | 13:52 | |
| *** sballe has joined #tripleo | 13:54 | |
| *** yfaykya has joined #tripleo | 13:54 | |
| *** ramishra has quit IRC | 13:56 | |
| *** yfaykya has quit IRC | 13:58 | |
| *** jistr|mobi has quit IRC | 13:58 | |
| howleyt | Hi, I have a question on https://review.openstack.org/#/c/83324 which is a patch for keystone/cert generation. It suggests using o-a-c to get the files in place on the uc/oc. How do I go about doing that. | 13:58 |
| *** jistr has quit IRC | 13:59 | |
| *** yfaykya has joined #tripleo | 14:00 | |
| *** jistr has joined #tripleo | 14:01 | |
| *** edmund has joined #tripleo | 14:02 | |
| *** nosnos has quit IRC | 14:03 | |
| *** yfaykya has quit IRC | 14:05 | |
| *** yfaykya has joined #tripleo | 14:06 | |
| *** yfaykya has quit IRC | 14:10 | |
| *** yfaykya has joined #tripleo | 14:12 | |
| *** devananda has quit IRC | 14:14 | |
| *** rcarrillocruz has quit IRC | 14:14 | |
| *** devananda has joined #tripleo | 14:15 | |
| *** mestery has joined #tripleo | 14:16 | |
| *** yfaykya has quit IRC | 14:17 | |
| *** yfaykya has joined #tripleo | 14:18 | |
| *** rcarrillocruz has joined #tripleo | 14:20 | |
| *** yfaykya has quit IRC | 14:23 | |
| *** yfaykya has joined #tripleo | 14:24 | |
| *** yfaykya has quit IRC | 14:29 | |
| *** rwsu has joined #tripleo | 14:29 | |
| *** yfaykya has joined #tripleo | 14:30 | |
| *** ramishra has joined #tripleo | 14:32 | |
| *** yfaykya has quit IRC | 14:35 | |
| *** yfaykya has joined #tripleo | 14:37 | |
| *** yfaykya has quit IRC | 14:41 | |
| *** yfaykya has joined #tripleo | 14:43 | |
| *** jistr has quit IRC | 14:44 | |
| *** yfaykya has quit IRC | 14:47 | |
| *** jistr has joined #tripleo | 14:48 | |
| *** rameshg87 has joined #tripleo | 14:48 | |
| *** yfaykya has joined #tripleo | 14:49 | |
| *** martyntaylor1 has joined #tripleo | 14:50 | |
| *** martyntaylor has quit IRC | 14:52 | |
| *** yfaykya has quit IRC | 14:53 | |
| *** ramishra has quit IRC | 14:54 | |
| *** yfaykya has joined #tripleo | 14:55 | |
| openstackgerrit | Richard Su proposed a change to openstack/tripleo-image-elements: WIP: SELinux file context updates https://review.openstack.org/99242 | 14:55 |
| *** martyntaylor1 has quit IRC | 14:56 | |
| *** yfaykya has quit IRC | 14:59 | |
| *** untriaged-bot has joined #tripleo | 15:00 | |
| untriaged-bot | Untriaged bugs so far: | 15:00 |
| untriaged-bot | https://bugs.launchpad.net/diskimage-builder/+bug/1329746 | 15:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1329238 | 15:00 |
| uvirtbot | Launchpad bug 1329746 in diskimage-builder "disk-image-create should allow sending compat flags to qemu-img" [Undecided,In progress] | 15:00 |
| uvirtbot | Launchpad bug 1329238 in tripleo "OVS isn't persisting mac addresses on OVS bridges" [Undecided,In progress] | 15:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1331337 | 15:00 |
| uvirtbot | Launchpad bug 1331337 in tripleo "install_dependencies fails if python-software-properties is not installed." [Undecided,In progress] | 15:00 |
| *** untriaged-bot has quit IRC | 15:00 | |
| *** yfaykya has joined #tripleo | 15:01 | |
| *** TravT has joined #tripleo | 15:05 | |
| *** yfaykya has quit IRC | 15:06 | |
| *** yfaykya has joined #tripleo | 15:07 | |
| *** gothicmindfood has quit IRC | 15:09 | |
| rameshg87 | hello tripleo reviewers .. | 15:11 |
| rameshg87 | please have a look at reviews: https://review.openstack.org/#/c/97301/ https://review.openstack.org/#/c/97143/ | 15:12 |
| *** yfaykya has quit IRC | 15:12 | |
| rameshg87 | they have been waiting for some time :-( | 15:12 |
| TheJulia | The first one the build failed. | 15:13 |
| TheJulia | Same with the second one. | 15:13 |
| *** yamahata has quit IRC | 15:13 | |
| *** yfaykya has joined #tripleo | 15:13 | |
| *** rdopieralski has quit IRC | 15:14 | |
| *** jtomasek has quit IRC | 15:16 | |
| *** regebro has quit IRC | 15:17 | |
| *** dprince has quit IRC | 15:17 | |
| *** yfaykya has quit IRC | 15:18 | |
| *** yfaykya has joined #tripleo | 15:19 | |
| *** regebro has joined #tripleo | 15:20 | |
| *** IvanBerezovskiy has left #tripleo | 15:21 | |
| rameshg87 | TheJulia: i couldn't get the error message in those tempest tests. since jenkins gave a +1, i thought it was non-voting | 15:22 |
| *** athomas has quit IRC | 15:23 | |
| rameshg87 | Thejulia, i think it is not related to my changes | 15:23 |
| *** shardy is now known as shardy_afk | 15:23 | |
| *** yfaykya has quit IRC | 15:24 | |
| *** blamar has joined #tripleo | 15:25 | |
| *** yfaykya has joined #tripleo | 15:26 | |
| *** hewbrocca has quit IRC | 15:27 | |
| *** yfaykya has quit IRC | 15:30 | |
| *** yfaykya has joined #tripleo | 15:32 | |
| *** yfaykya has quit IRC | 15:36 | |
| *** yfaykya has joined #tripleo | 15:38 | |
| TheJulia | rameshg87: I honestly don't know what to tell you because I'm fairly new to this project. It might be some sort of timing issue. Interestingly enough check-tripleo-undercloud-percise passed on a second go-around without any changes. | 15:39 |
| *** gothicmindfood has joined #tripleo | 15:40 | |
| rameshg87 | TheJulia, thanks..i think i can give a "recheck no bug" to see if it gets resolved. i didn't take any action on it because Jenkins gave a +1 and thought other tests were non-voting | 15:40 |
| d0ugal | jdob: Do you have a preference between these? https://etherpad.openstack.org/p/15bBoHOMny | 15:41 |
| d0ugal | jdob: (or an alternative is welcome too) | 15:41 |
| *** yfaykya has quit IRC | 15:42 | |
| *** rameshg87 has left #tripleo | 15:43 | |
| *** yfaykya has joined #tripleo | 15:44 | |
| *** dtantsur is now known as dtantsur|afk | 15:44 | |
| openstackgerrit | Stuart McLaren proposed a change to openstack/tripleo-heat-templates: Add nova_compute_ips to overcloud-source.yaml https://review.openstack.org/100848 | 15:45 |
| openstackgerrit | Stuart McLaren proposed a change to openstack/tripleo-image-elements: Basic stunnel firewall configuration https://review.openstack.org/100853 | 15:46 |
| *** yfaykya has quit IRC | 15:48 | |
| *** yfaykya has joined #tripleo | 15:50 | |
| *** eghobo has joined #tripleo | 15:52 | |
| *** sballe_ has joined #tripleo | 15:53 | |
| *** yfaykya has quit IRC | 15:54 | |
| *** sballe has quit IRC | 15:56 | |
| *** yfaykya has joined #tripleo | 15:56 | |
| d0ugal | jdob: gah, nevermind, ignore that, its not that simple. This is turning into a mess. | 15:59 |
| *** spzala has quit IRC | 16:00 | |
| *** spzala has joined #tripleo | 16:00 | |
| *** yfaykya has quit IRC | 16:01 | |
| *** jcoufal has quit IRC | 16:02 | |
| *** yfaykya has joined #tripleo | 16:02 | |
| *** yfaykya has quit IRC | 16:07 | |
| *** yfaykya has joined #tripleo | 16:08 | |
| *** yfaykya has quit IRC | 16:13 | |
| *** yfaykya has joined #tripleo | 16:15 | |
| *** yjiang5_away is now known as yjiang5 | 16:15 | |
| *** eghobo has quit IRC | 16:18 | |
| *** eghobo has joined #tripleo | 16:18 | |
| *** yfaykya has quit IRC | 16:19 | |
| *** lazy_prince has quit IRC | 16:20 | |
| *** yfaykya has joined #tripleo | 16:21 | |
| *** dprince has joined #tripleo | 16:22 | |
| dprince | SpamapS: I *think* we might have a heat regresssion from yesterday | 16:23 |
| dprince | SpamapS: https://bugs.launchpad.net/tripleo/+bug/1331445 | 16:23 |
| uvirtbot | Launchpad bug 1331445 in tripleo "Refusing to bootstrap mysql cluster until role is known. (all Undercloud jobs fail)" [Critical,Triaged] | 16:23 |
| dprince | SpamapS: any ideas on which one of those I might should try reverting first? | 16:24 |
| *** yfaykya has quit IRC | 16:25 | |
| *** yfaykya has joined #tripleo | 16:27 | |
| *** wendar_ is now known as wendar | 16:31 | |
| *** yfaykya has quit IRC | 16:31 | |
| *** e0ne_ has joined #tripleo | 16:31 | |
| *** jml has quit IRC | 16:31 | |
| *** yfaykya has joined #tripleo | 16:33 | |
| *** e0ne has quit IRC | 16:33 | |
| *** jml has joined #tripleo | 16:34 | |
| dprince | SpamapS: Load resources using Resource.load_all_from_stack() looks suspicious | 16:37 |
| *** yfaykya has quit IRC | 16:38 | |
| *** yfaykya has joined #tripleo | 16:39 | |
| mordred | are you guys managing r1 and r2 as regions in a single cloud? or as two completely separate clouds? | 16:39 |
| dprince | mordred: for now two separate clouds | 16:40 |
| dprince | mordred: long term ideally a single cloud w/ regions... | 16:40 |
| mordred | yah. that'll be neat | 16:40 |
| *** yfaykya has quit IRC | 16:44 | |
| *** yfaykya has joined #tripleo | 16:45 | |
| *** e0ne_ is now known as e0ne | 16:46 | |
| openstackgerrit | Dan Prince proposed a change to openstack-infra/tripleo-ci: temprevert Heat commit for load_all_from_stack... https://review.openstack.org/100955 | 16:46 |
| *** yfaykya has quit IRC | 16:49 | |
| *** martyntaylor has joined #tripleo | 16:50 | |
| *** yfaykya has joined #tripleo | 16:51 | |
| *** ramishra has joined #tripleo | 16:55 | |
| *** regebro is now known as regebro|afk | 16:55 | |
| *** yfaykya has quit IRC | 16:56 | |
| *** yfaykya has joined #tripleo | 16:58 | |
| *** e0ne_ has joined #tripleo | 16:59 | |
| *** ramishra has quit IRC | 16:59 | |
| openstackgerrit | Michael Kerrin proposed a change to openstack/tripleo-image-elements: Use Percona Packages for Ubuntu and Debian https://review.openstack.org/90134 | 17:01 |
| *** e0ne has quit IRC | 17:01 | |
| openstackgerrit | Michael Kerrin proposed a change to openstack/tripleo-image-elements: Don't reset MySQL database from image https://review.openstack.org/98978 | 17:01 |
| openstackgerrit | Michael Kerrin proposed a change to openstack/tripleo-image-elements: MySQL upstart script setup https://review.openstack.org/99964 | 17:01 |
| *** yfaykya has quit IRC | 17:02 | |
| *** jang has joined #tripleo | 17:02 | |
| *** yfaykya has joined #tripleo | 17:04 | |
| *** yfaykya has quit IRC | 17:08 | |
| *** yfaykya has joined #tripleo | 17:10 | |
| *** e0ne_ has quit IRC | 17:10 | |
| *** e0ne_ has joined #tripleo | 17:11 | |
| *** lazy_prince has joined #tripleo | 17:12 | |
| *** e0ne__ has joined #tripleo | 17:13 | |
| *** mrunge has quit IRC | 17:13 | |
| *** yfaykya has quit IRC | 17:14 | |
| *** dshulyak_ has joined #tripleo | 17:15 | |
| *** yfaykya has joined #tripleo | 17:16 | |
| openstackgerrit | Michael Tupitsyn proposed a change to openstack/os-cloud-config: Do not create admin user if it exists already https://review.openstack.org/100719 | 17:16 |
| *** e0ne_ has quit IRC | 17:16 | |
| *** e0ne__ has quit IRC | 17:17 | |
| *** sballe__ has joined #tripleo | 17:19 | |
| *** saurabhs has joined #tripleo | 17:19 | |
| *** yfaykya has quit IRC | 17:20 | |
| *** jprovazn has joined #tripleo | 17:21 | |
| *** yfaykya has joined #tripleo | 17:22 | |
| *** sballe_ has quit IRC | 17:23 | |
| *** jdob has quit IRC | 17:24 | |
| *** jdob has joined #tripleo | 17:24 | |
| openstackgerrit | James Slagle proposed a change to openstack/tripleo-specs: Add a local datasource collector to occ https://review.openstack.org/100965 | 17:24 |
| openstackgerrit | James Slagle proposed a change to openstack/tripleo-specs: Add a local datasource collector to occ https://review.openstack.org/100965 | 17:25 |
| *** lucasagomes is now known as lucas-dinner | 17:26 | |
| *** yfaykya has quit IRC | 17:27 | |
| *** pcrews has quit IRC | 17:27 | |
| *** jp_at_hp has quit IRC | 17:28 | |
| *** yfaykya has joined #tripleo | 17:28 | |
| *** pcrews has joined #tripleo | 17:29 | |
| *** dshulyak_ has quit IRC | 17:31 | |
| *** dshulyak_ has joined #tripleo | 17:31 | |
| *** bmahalakshmi has joined #tripleo | 17:33 | |
| *** yfaykya has quit IRC | 17:33 | |
| *** jp_at_hp has joined #tripleo | 17:34 | |
| openstackgerrit | James Slagle proposed a change to openstack/tripleo-specs: Configurable /mnt/state https://review.openstack.org/94876 | 17:34 |
| *** yfaykya has joined #tripleo | 17:34 | |
| slagle | lifeless: can you review https://review.openstack.org/#/c/94876/ again when you're able? | 17:35 |
| slagle | any other cores and interested folks too :). | 17:36 |
| *** dshulyak_ has quit IRC | 17:37 | |
| *** yfaykya has quit IRC | 17:39 | |
| *** jdob has quit IRC | 17:39 | |
| *** jdob has joined #tripleo | 17:39 | |
| *** dshulyak_ has joined #tripleo | 17:40 | |
| *** yfaykya has joined #tripleo | 17:40 | |
| *** nati_ueno has joined #tripleo | 17:41 | |
| *** yfaykya has quit IRC | 17:45 | |
| *** yfaykya has joined #tripleo | 17:47 | |
| *** akrivoka has quit IRC | 17:49 | |
| *** yfaykya has quit IRC | 17:52 | |
| *** yfaykya has joined #tripleo | 17:53 | |
| *** nati_ueno has quit IRC | 17:53 | |
| *** dshulyak_ has quit IRC | 17:53 | |
| *** nati_ueno has joined #tripleo | 17:54 | |
| *** jp_at_hp has quit IRC | 17:54 | |
| *** e0ne has joined #tripleo | 17:55 | |
| *** e0ne has quit IRC | 17:56 | |
| *** ramishra has joined #tripleo | 17:56 | |
| *** e0ne has joined #tripleo | 17:56 | |
| *** yfaykya has quit IRC | 17:57 | |
| *** akuznetsov has quit IRC | 17:58 | |
| *** yfaykya has joined #tripleo | 17:59 | |
| vinsh | jprovazn, do you have any working examples on how to use: https://review.openstack.org/#/c/94609/3 | 17:59 |
| vinsh | ? | 18:00 |
| *** ramishra has quit IRC | 18:00 | |
| vinsh | I see that the certs are defined as a string in the template from: https://review.openstack.org/#/c/94591/1 | 18:00 |
| *** e0ne has quit IRC | 18:01 | |
| vinsh | But.. HOW does this get the same cert on all nodes? | 18:01 |
| *** yfaykya has quit IRC | 18:03 | |
| vinsh | "If keys/certs are not found in heat metadata, empty files are generated" | 18:03 |
| vinsh | how do you add the certs into the metadata? | 18:03 |
| jprovazn | vinsh, Hi, https://review.openstack.org/#/c/95101/ | 18:04 |
| vinsh | OH! | 18:04 |
| vinsh | wow, thank you :) | 18:05 |
| vinsh | Its interesting that I had sort of hacked/designed similar on my own. | 18:05 |
| *** yfaykya has joined #tripleo | 18:05 | |
| jprovazn | vinsh, you are welcome :) | 18:06 |
| vinsh | ah-hah.. stuff is being linked to blueprints finally.. helpful. :) | 18:08 |
| *** rcarrillocruz has quit IRC | 18:08 | |
| jprovazn | yep :) | 18:08 |
| *** rcarrillocruz has joined #tripleo | 18:08 | |
| *** yfaykya has quit IRC | 18:09 | |
| greghaynes | jprovazn: have you made the -env.json files permissions 0600 patch? | 18:10 |
| greghaynes | if not, ill push that up real quick | 18:10 |
| *** dtantsur|afk is now known as dtantsur | 18:11 | |
| *** yfaykya has joined #tripleo | 18:11 | |
| jprovazn | greghaynes, I didn't get to real work today, if you can send a patch for it, it would be great, I'll address SpamapS's mktemp comment in a few | 18:12 |
| *** jtomasek has joined #tripleo | 18:14 | |
| *** yfaykya has quit IRC | 18:16 | |
| *** yfaykya has joined #tripleo | 18:17 | |
| *** mrunge has joined #tripleo | 18:21 | |
| greghaynes | slagle: dprince can I get a re-+2 on https://review.openstack.org/#/c/95014/ (had to rebase) | 18:21 |
| *** yfaykya has quit IRC | 18:21 | |
| dprince | greghaynes: done | 18:22 |
| greghaynes | tyty | 18:22 |
| *** akuznetsov has joined #tripleo | 18:22 | |
| openstackgerrit | A change was merged to openstack/tripleo-incubator: Add greghaynes to the cd admins https://review.openstack.org/95014 | 18:22 |
| greghaynes | ohey, a +a | 18:22 |
| * greghaynes thought ptl had to | 18:23 | |
| dprince | greghaynes: oh, did I break a rule? | 18:23 |
| *** ramishra has joined #tripleo | 18:23 | |
| *** yfaykya has joined #tripleo | 18:23 | |
| dprince | greghaynes: just don't do anything crazy until lifeless acks it then | 18:24 |
| greghaynes | oh, says only cd-admins should approve | 18:24 |
| greghaynes | but yes | 18:24 |
| *** e0ne has joined #tripleo | 18:24 | |
| *** ramishra has quit IRC | 18:27 | |
| *** yfaykya has quit IRC | 18:28 | |
| *** yfaykya has joined #tripleo | 18:29 | |
| *** dshulyak_ has joined #tripleo | 18:30 | |
| *** akuznetsov has quit IRC | 18:31 | |
| *** yfaykya has quit IRC | 18:34 | |
| *** yfaykya has joined #tripleo | 18:36 | |
| *** pelix has quit IRC | 18:39 | |
| *** yfaykya has quit IRC | 18:40 | |
| *** yfaykya has joined #tripleo | 18:42 | |
| slagle | the rule is you need a supermajority | 18:43 |
| *** jtomasek has quit IRC | 18:43 | |
| slagle | i added my +2 in a comment :) | 18:44 |
| greghaynes | haha, ty | 18:45 |
| *** spzala has quit IRC | 18:46 | |
| *** yfaykya has quit IRC | 18:46 | |
| *** yfaykya has joined #tripleo | 18:48 | |
| *** dshulyak_ has quit IRC | 18:48 | |
| greghaynes | dont throw any parties in the rh1 cloud? | 18:50 |
| TheJulia | heh | 18:50 |
| greghaynes | oops, wat responding to the dprince scrollback of dont do anything crazy :p | 18:50 |
| openstackgerrit | Gregory Haynes proposed a change to openstack/tripleo-incubator: Require and set mode 0600 on heat env caches https://review.openstack.org/100994 | 18:51 |
| greghaynes | jprovazn: ^ | 18:51 |
| *** yfaykya has quit IRC | 18:52 | |
| *** bmahalakshmi has quit IRC | 18:54 | |
| *** yfaykya has joined #tripleo | 18:54 | |
| jprovazn | greghaynes, thanks, will look soon | 18:54 |
| *** jistr has quit IRC | 18:55 | |
| *** blamar has quit IRC | 18:57 | |
| greghaynes | dprince: for debugging of the os-is-bootstrap host fail, I wonder if we should add the output of os-collectl-config --print to our final log sweep? | 18:58 |
| greghaynes | would make it so you dont have to https://review.openstack.org/#/c/100898 | 18:58 |
| *** yfaykya has quit IRC | 18:58 | |
| *** rlandy has quit IRC | 18:59 | |
| *** yfaykya has joined #tripleo | 19:00 | |
| *** TravT has quit IRC | 19:01 | |
| SpamapS | o/ | 19:03 |
| *** yfaykya has quit IRC | 19:05 | |
| openstackgerrit | Gregory Haynes proposed a change to openstack-infra/tripleo-ci: Store os-collect-config --print in host_info log https://review.openstack.org/100997 | 19:05 |
| *** yfaykya has joined #tripleo | 19:06 | |
| dprince | greghaynes: the metadata is just missing | 19:06 |
| *** openstackgerrit has quit IRC | 19:06 | |
| dprince | greghaynes: I'm seeing this in the nova API logs Failed to get metadata for ip | 19:07 |
| *** dshulyak_ has joined #tripleo | 19:07 | |
| greghaynes | :/ | 19:07 |
| dprince | greghaynes: I updated the ticket a few minutes ago with that. So I don't think this is related to os-is-bootstrap specifically. It just happens to be the first thing to fail because of it | 19:07 |
| *** openstackgerrit_ has joined #tripleo | 19:07 | |
| greghaynes | yep, it is | 19:08 |
| *** openstackgerrit_ is now known as openstackgerrit | 19:08 | |
| *** mrunge has quit IRC | 19:08 | |
| *** spzala has joined #tripleo | 19:09 | |
| *** jtomasek has joined #tripleo | 19:10 | |
| *** e0ne has quit IRC | 19:11 | |
| *** yfaykya has quit IRC | 19:11 | |
| *** yfaykya has joined #tripleo | 19:12 | |
| dprince | greghaynes: this fixes it https://review.openstack.org/#/c/100955/ | 19:14 |
| dprince | greghaynes: we could send that and get undercloud CI working while we have a closer look at heat | 19:15 |
| greghaynes | ahoy | 19:15 |
| greghaynes | SpamapS: ^ | 19:15 |
| dprince | greghaynes: I may also throw up a revert | 19:15 |
| dprince | greghaynes: but that may take another day to get through the gate, who knows | 19:16 |
| *** e0ne has joined #tripleo | 19:16 | |
| *** yfaykya has quit IRC | 19:17 | |
| *** markmc has quit IRC | 19:17 | |
| *** julim has quit IRC | 19:18 | |
| * SpamapS reads issues | 19:18 | |
| *** yfaykya has joined #tripleo | 19:18 | |
| vinsh | jprovazn, I gave this a try: https://review.openstack.org/#/c/94609 | 19:18 |
| SpamapS | hm | 19:19 |
| vinsh | seems to fail with this: | 19:19 |
| vinsh | dib-run-parts Wed Jun 18 19:17:00 UTC 2014 Running /opt/stack/os-config-refresh/configure.d/10-keystone-state | 19:19 |
| vinsh | + touch /mnt/state/etc/keystone/ssl/private/cakey.pem | 19:19 |
| vinsh | touch: cannot touch '/mnt/state/etc/keystone/ssl/private/cakey.pem': No such file or directory | 19:19 |
| SpamapS | Jun 18 17:32:15 localhost nova-api[3535]: 2014-06-18 17:32:15.333 3836 ERROR nova.api.metadata.handler [-] Failed to get metadata for ip: 172.19.0.4 | 19:19 |
| vinsh | jprovazn, cause that directory hasn't been created yet. | 19:19 |
| dprince | SpamapS: I now think that was unrelated | 19:19 |
| SpamapS | Ok | 19:19 |
| SpamapS | it is | 19:19 |
| SpamapS | that is nova's metadata | 19:19 |
| dprince | SpamapS: I was trying things :) | 19:19 |
| SpamapS | dprince: so what _is_ the issue? | 19:19 |
| SpamapS | or we don't know, we're just putting CI back on track? | 19:20 |
| SpamapS | dprince: I will gladly fast track a revert. the gate is in good health right now. | 19:20 |
| dprince | SpamapS: both. I'm certain that reverting that heat commit fixes it now | 19:20 |
| SpamapS | 2 hours or so | 19:20 |
| dprince | SpamapS: okay, let me through up a vague ticket w/ what I know and then push the revert against that | 19:20 |
| SpamapS | dprince: submit that revert.. I'll +2 and find you another core. | 19:20 |
| dprince | SpamapS: ack, thanks man | 19:21 |
| SpamapS | dprince: and we'll use check-experimental before we unrevert | 19:21 |
| dprince | SpamapS: right, note that you'll only see undercloud passing though, we still have an overcloud issue that needs fixing | 19:21 |
| lifeless | greghaynes: dprince: ola | 19:21 |
| greghaynes | hihi | 19:21 |
| SpamapS | dprince: do we know what that issue is? | 19:21 |
| * dprince misses derek much already | 19:21 | |
| dprince | SpamapS: no, was going to look at it today, but we hit this! | 19:22 |
| lifeless | dprince: it doesn't require PTL adding, it does hwoever want a supermajority | 19:22 |
| dprince | lifeless: ack, slagle said the same | 19:23 |
| *** yfaykya has quit IRC | 19:23 | |
| * dprince takes hand slap for breaking rules | 19:23 | |
| lifeless | sso tat would have wanted 7; I see 5 on the review. | 19:23 |
| lifeless | slagle: were you going to be +2? | 19:23 |
| slagle | lifeless: yes. i added my posthumous +2 in a comment :) | 19:24 |
| *** e0ne has quit IRC | 19:24 | |
| *** ramishra has joined #tripleo | 19:24 | |
| *** yfaykya has joined #tripleo | 19:25 | |
| *** jang1 has joined #tripleo | 19:26 | |
| SlickNik | SpamapS: Wanted to follow up with you about https://bugs.launchpad.net/diskimage-builder/+bug/1329068 | 19:26 |
| uvirtbot | Launchpad bug 1329068 in trove-integration "The trove guest-image is unable to publish the guest-config via cloud-init" [Critical,In progress] | 19:26 |
| SlickNik | SpamapS: When you have a moment. | 19:26 |
| dprince | SpamapS: https://review.openstack.org/101001 | 19:27 |
| dprince | SpamapS: ^^ that is the heat revert we need... | 19:27 |
| * dprince goes to run check experimental on it | 19:27 | |
| SpamapS | dprince: ugh | 19:28 |
| SpamapS | dprince: why did you open a new bug report? | 19:28 |
| dprince | SpamapS: sorry, I thought this would be cleaner I guess | 19:28 |
| SpamapS | dprince: this is specifically why launchpad tracks against multiple projects. :-P | 19:28 |
| *** ramishra has quit IRC | 19:29 | |
| SpamapS | dprince: just splits the horizon. We can deal. | 19:29 |
| dprince | SpamapS: sure, I've done it both ways. next time we can just add the proj I suppose | 19:29 |
| *** yfaykya has quit IRC | 19:30 | |
| SpamapS | dprince: I'm biased as I'm a long time launchpad user.. but it is really the only reason launchpad is awesome for bug tracking. | 19:30 |
| SpamapS | everything else is "meh" or "ew" | 19:30 |
| SpamapS | but that is like "oh nice only one number to reference everywhere" | 19:30 |
| SpamapS | anyway I don't want to harp on it | 19:30 |
| SpamapS | just.. let's do that not just next time, but all the time. :) | 19:30 |
| *** spzala has quit IRC | 19:30 | |
| *** yfaykya has joined #tripleo | 19:31 | |
| *** spzala has joined #tripleo | 19:31 | |
| *** spzala has quit IRC | 19:32 | |
| dprince | SpamapS: okay, so if you can push through this issue I'll go back to thinking about the other overcloud failures | 19:32 |
| *** spzala has joined #tripleo | 19:32 | |
| SpamapS | dprince: cool. Can you be more specific about "Heat specific metadata" ? We'll merge the revert, but the bug needs to stay open and we need to investigate so we can get a fixed patch up and a test. | 19:33 |
| dprince | SpamapS: everything is just missing. We hit this in a script called is os-is-bootstrap-host first. It tries to run os-apply-config to get a couple items | 19:34 |
| dprince | SpamapS: bootstrap_host.bootstrap_nodeid and bootstrap_host.nodeid | 19:35 |
| dprince | SpamapS: upon further investigation it looks like pretty much everything is missing | 19:35 |
| *** yfaykya has quit IRC | 19:35 | |
| SpamapS | dprince: https://review.openstack.org/#/c/101001/ -1'd.. needs a tiny bit of supporting reasoning in the commit message. | 19:35 |
| greghaynes | yea, that one is definitely the first thing to assert that some value is in metadata | 19:35 |
| *** e0ne has joined #tripleo | 19:35 | |
| dprince | SpamapS: how would you say this consisely? :) | 19:36 |
| SpamapS | ok so we need a test that cfn api actually works | 19:36 |
| *** e0ne has quit IRC | 19:36 | |
| * dprince updates the commit message | 19:36 | |
| vinsh | quick peek.. anyone seen this at end of devtest_overcloud : http://paste.openstack.org/show/84424/ image exits in glance.. but nova doesn't see it? | 19:37 |
| *** yfaykya has joined #tripleo | 19:37 | |
| *** e0ne has joined #tripleo | 19:37 | |
| greghaynes | vinsh: fun! | 19:37 |
| dprince | SpamapS: updated... | 19:37 |
| vinsh | greghaynes, Loads! | 19:37 |
| greghaynes | vinsh: this is with multiple controllers, yes? | 19:38 |
| greghaynes | probably the most fun part of this is going to be checking the logs | 19:38 |
| greghaynes | :) | 19:38 |
| vinsh | yes | 19:38 |
| vinsh | Ive got controllers for my controllers at this point. | 19:38 |
| greghaynes | we might want to consider some kind of centralised logging nowish... | 19:39 |
| SpamapS | dprince: cool thanks and +2d | 19:39 |
| SpamapS | +2d dude | 19:39 |
| SpamapS | rhymes | 19:39 |
| vinsh | greghaynes, imma look at where nova thinks its looking. | 19:40 |
| *** e0ne has quit IRC | 19:40 | |
| vinsh | after I eat a burrito. | 19:40 |
| greghaynes | vinsh: yes. I just mean youre going to have to find which nova you actually hit | 19:40 |
| greghaynes | SpamapS: http://i.imgur.com/CbnYHlh.gif | 19:40 |
| *** yfaykya has quit IRC | 19:41 | |
| greghaynes | wow these heat convergence specs have multiplied | 19:42 |
| * SpamapS wins | 19:42 | |
| *** blamar has joined #tripleo | 19:43 | |
| *** yfaykya has joined #tripleo | 19:43 | |
| *** jtomasek has quit IRC | 19:44 | |
| *** TravT has joined #tripleo | 19:45 | |
| *** jtomasek has joined #tripleo | 19:46 | |
| dprince | SpamapS: the new ticket is cleaner, old ticket marked as dup, and now everyone is happy | 19:46 |
| SpamapS | dprince: can I get an example os-collect-config.log ? | 19:46 |
| SpamapS | dprince: mmkay | 19:47 |
| SpamapS | cleaner just means all that delicious red herring meat is thrown away | 19:47 |
| dprince | SpamapS: http://logs.openstack.org/19/100719/2/check-tripleo/check-tripleo-undercloud-precise/e534283/logs/undercloud-undercloud-c7n5mfpzzqju_logs/os-collect-config.txt.gz | 19:47 |
| dprince | SpamapS: any of the failing undercloud job logs would have similar | 19:47 |
| *** yfaykya has quit IRC | 19:48 | |
| * dprince just picked one | 19:48 | |
| *** hrushig_ has joined #tripleo | 19:48 | |
| hrushig_ | we are seeing issues in our labs during pxe boot of baremetal nodes -- either UC or OC Error: /etc/iscsi/initiatorname.iscsi does not contain a valid Initiat ^[[25;1H^[[25;1H^[[22;1H^[[0;25;37;40mError: /etc/iscsi/initiatorname.iscsi does not contain a valid InitiatorName. ^[[23;1H^[[0;25;37;40mThe iSCSI driver has not been correctly installed and cannot start. ^[[24;1H^[[0;25;37;40m | 19:49 |
| hrushig_ | ^[[25;1H^[[25;1H^[[25;1H^[[1;1H^[[0;25;37;40m[^[[0;25;32;40m ok ^[[0;25;37;40m] Setting kernel variables ...done. ^[[2;1H^[[0;25;37;40m[ 7.285481] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready ^[[3;1H^[[0;25;37;40m[ 8.195095] init: udev-fallback-graphics main process (1241) terminated with ^[[4;1H^[[0;25;37;40mstatus 1 569] iTCO_wdt: Intel TCO WatchDog Timer Driver | 19:49 |
| hrushig_ | v1.10 ^[[25;1H^[[25;1H^[[4;1H^[[0;25;37;40mstatus 1 ^[[5;1H^[[0;25;37;40m[ 8.289281] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready ^[[6;1H^[[0;25;37;40m[....] Starting iSCSI initiator service: iscsid/etc/init.d/open-iscsi: 72: /etc/^[[7;1H^[[0;25;37;40minit.d/open-iscsi: csi 00:01: Found new BMC (man_id: 0x00000b, prod_id: 0x2100, ^[[25;1H^[[25;1H^[ | 19:49 |
| *** yfaykya has joined #tripleo | 19:49 | |
| hrushig_ | [25;1H^[[7;1H^[[0;25;37;40minit.d/open-iscsi: cannot create /etc/iscsi/initiatorname.iscsi: Read-only file ^[[8;1H^[[0;25;37;40msystem ^[[9;1H^[[0;25;37;40m/etc/init.d/open-iscsi: 73: /etc/init.d/open-iscsi: cannot create /etc/iscsi/ini^[[10;1H^[[0;25;37;40mtiatorname.iscsi: Read-only fiLink Up ^[[25;1H^[[25;1H^[[10;1H^[[0;25;37;40mtiatorname.i | 19:49 |
| hrushig_ | scsi: Read-only file system | 19:49 |
| hrushig_ | 19:49 | |
| hrushig_ | . | 19:49 |
| hrushig_ | Tom Hancock added a comment - 12/Jun/14 09:08 AM UTC | 19:49 |
| hrushig_ | To clarify - the dhcp-all-interfaces error comes from the undercloud image itself, so the | 19:49 |
| nealph | lifeless:ping | 19:49 |
| hrushig_ | parted reset in t | 19:49 |
| *** e0ne has joined #tripleo | 19:50 | |
| *** e0ne has quit IRC | 19:50 | |
| hrushig_ | removing a partition info and tried again... same error | 19:50 |
| *** e0ne has joined #tripleo | 19:51 | |
| lifeless | nealph: hi | 19:51 |
| *** julim has joined #tripleo | 19:51 | |
| lifeless | dprince: did we figure out the issue from yesterdays CI blocker? | 19:52 |
| dprince | lifeless: not yet. just getting back to that now | 19:53 |
| lifeless | hah wheeeeee | 19:53 |
| lifeless | when it goes south, it goes deep south | 19:53 |
| dprince | lifeless: I'll have to drop off for a bit soon but I can look at it more this evening | 19:53 |
| *** yfaykya has quit IRC | 19:53 | |
| jprovazn | vinsh, https://review.openstack.org/#/c/94609 - dires are created in elements/keystone/os-refresh-config/configure.d/10-keystone-state, but in keystone-pki the script should be renamed to something like 11-keystone-pki-perms so it doesn't overwrites the configure script from keystone element | 19:55 |
| jprovazn | vinsh, thanks for the comment | 19:55 |
| lifeless | dprince: cool; did we get tchaypo's bastion access sorted? | 19:55 |
| *** e0ne has quit IRC | 19:55 | |
| *** yfaykya has joined #tripleo | 19:55 | |
| lifeless | dprince: I couldn't poke at it - I don't know my password (vs ssh key) and can't reset it without that, and can't do sudo without it. (Are we meant to ahve sudo on the bastion?) | 19:55 |
| dprince | lifeless: not yet, been a busy day | 19:55 |
| lifeless | dprince: ack | 19:55 |
| vinsh | jprovazn, Welcome. | 19:57 |
| *** yfaykya has quit IRC | 19:59 | |
| *** e0ne has joined #tripleo | 20:01 | |
| *** yfaykya has joined #tripleo | 20:01 | |
| nealph | lifeless:circling back to https://review.openstack.org/#/c/96599/3 | 20:04 |
| *** blamar has quit IRC | 20:05 | |
| *** e0ne has quit IRC | 20:06 | |
| nealph | Can you let me know your thoughts? My sense is that this fits well and (in part) addressses a known bug... | 20:06 |
| *** yfaykya has quit IRC | 20:06 | |
| *** yfaykya has joined #tripleo | 20:07 | |
| lifeless | nealph: its a local machine configuration issue right? | 20:09 |
| lifeless | as in machine A can have config A and B config B and nothing knows anything about it ? | 20:09 |
| lifeless | nealph: I'm entirely pro seeing the bug fixed, but this isn't a cluster orchestration layer issue, its within the machine, there's a single right value, the upstream default is wrong | 20:11 |
| lifeless | nealph: making it part of the UI for cluster deployment increases the learning curve and surface area to (AFAICT) no value | 20:11 |
| lifeless | nealph: so I'm proposing the fix be: | 20:11 |
| nealph | essentially it's a config option that should be available (and populated) for all machines deploying ceilometer. the upstream default is...well...nothing. | 20:11 |
| *** yfaykya has quit IRC | 20:12 | |
| lifeless | - file a bug upstream (production defaults) | 20:12 |
| lifeless | - change the config file template to have whatever default the bug says we should have, with a reference to the bug beside it. | 20:12 |
| lifeless | oops, family - be back soon | 20:13 |
| nealph | np...will paste some links in a moment | 20:13 |
| *** yfaykya has joined #tripleo | 20:14 | |
| *** blamar has joined #tripleo | 20:14 | |
| *** rcarrill` has joined #tripleo | 20:14 | |
| *** rcarrillocruz has quit IRC | 20:16 | |
| jprovazn | greghaynes, can you please check comments on https://review.openstack.org/#/c/98709/ ? | 20:16 |
| greghaynes | ah, right | 20:17 |
| greghaynes | :( | 20:17 |
| nealph | perhaps my issue is not linking the changes together....https://review.openstack.org/#/c/96599/ is the template change, https://review.openstack.org/#/c/96597/ + https://review.openstack.org/#/c/96527/ updates it into ceilometer conf. | 20:17 |
| greghaynes | that is a pretty silly reason we have to push the work out of the templating layer | 20:18 |
| greghaynes | +2'd though | 20:18 |
| *** yfaykya has quit IRC | 20:18 | |
| nealph | https://bugs.launchpad.net/python-keystoneclient/+bug/1180998 is the upstream...I noted in a comment but can call out more obviously. | 20:18 |
| uvirtbot | Launchpad bug 1180998 in python-keystoneclient "Keystoneclient middleware is trying to create the signing_dir in the AuthProtocol initialization even though keystone is configured to work with UUID token which is incorrect" [Medium,Triaged] | 20:18 |
| jprovazn | greghaynes, thanks | 20:19 |
| *** yfaykya has joined #tripleo | 20:20 | |
| *** e0ne has joined #tripleo | 20:20 | |
| lifeless | nealph: so I don't understand - does it create a single dir in /tmp per request ? | 20:21 |
| nealph | Yes, that's the bug, exactly. | 20:21 |
| lifeless | so that bug being medium seems wrong :) | 20:22 |
| lifeless | but | 20:22 |
| lifeless | thats beside the point - nothing here invalidates the route I mentioned for fixing this | 20:23 |
| lifeless | a) ensure there is a bug | 20:23 |
| lifeless | b) change the config template to have the right value and reference the bug there | 20:23 |
| lifeless | its not a cluster configuration problem | 20:23 |
| lifeless | there's no call for it to be configurable | 20:23 |
| *** yfaykya has quit IRC | 20:25 | |
| *** ramishra has joined #tripleo | 20:25 | |
| *** yfaykya has joined #tripleo | 20:26 | |
| *** blamar has quit IRC | 20:28 | |
| *** ramishra has quit IRC | 20:29 | |
| *** dshulyak_ has quit IRC | 20:30 | |
| *** yfaykya has quit IRC | 20:31 | |
| *** lazy_prince has quit IRC | 20:31 | |
| *** yfaykya has joined #tripleo | 20:32 | |
| *** julim has quit IRC | 20:34 | |
| nealph | agreed on priority mismatch. :) There is definitely a bug...we see this impacting our cloud intances in prod. As for configurability, you seem to be saying that deployer doesn't need to specify something different between node A and node B.... | 20:35 |
| nealph | but that's exactly what upstream ceilometer allows for...my thought was to match that. No? | 20:35 |
| *** yfaykya has quit IRC | 20:36 | |
| vinsh | jprovazn, bingo "dib-run-parts Wed Jun 18 20:34:22 UTC 2014 11-keystone-pki-perms completed | 20:37 |
| vinsh | 11-keystone-pki-perms " | 20:37 |
| morganfainberg | lifeless oh thats an icky bug | 20:38 |
| morganfainberg | per request?! | 20:38 |
| *** yfaykya has joined #tripleo | 20:38 | |
| *** edmund has quit IRC | 20:38 | |
| nealph | morganfainberg:potentially...I've not looked at the keystone code in a couple of weeks. | 20:39 |
| morganfainberg | nealph, ok let me take a look at that. | 20:39 |
| jprovazn | vinsh, cool | 20:40 |
| *** edmund has joined #tripleo | 20:40 | |
| morganfainberg | nealph, ok so keystone is configured to use UUID but middleware is still trying to create the signing dir cache location | 20:41 |
| *** jprovazn has quit IRC | 20:43 | |
| *** edunham has joined #tripleo | 20:43 | |
| morganfainberg | nealph, it the issue that each thread that auth_token ends up in maeks a new signing dir? | 20:43 |
| *** yfaykya has quit IRC | 20:43 | |
| morganfainberg | nealph, i'm not clear on the issue you're talking about here. | 20:43 |
| *** yfaykya has joined #tripleo | 20:44 | |
| nealph | yes, essentially the impact over time is to kill the node with tmp dirs...keystone client takes an optional arguement KEYSTONE_SIGNING_DIR that, if set with correct perms, provides a target for the signing action. | 20:44 |
| nealph | morganfainberg:^^^^ | 20:45 |
| morganfainberg | but it's not per-request | 20:45 |
| morganfainberg | it's per thread we initialize auth_token middleware in | 20:45 |
| morganfainberg | per initializeation that is | 20:45 |
| morganfainberg | because i didn't see anything in __call__ that looked like it would be per-request. it's a slow leak, but a rather nasty one regardless | 20:45 |
| nealph | well, ceilometer initializes at each polling cycle for the agent..not sure about how many threads for the api. | 20:46 |
| nealph | by default that's a polling action every minute, I think... | 20:47 |
| * nealph tries to remember the default polling interval | 20:47 | |
| nealph | so it's a slow death, but a death nonetheless. :) | 20:47 |
| morganfainberg | nealph, there is only one place in the code we do mkdtmp | 20:49 |
| morganfainberg | and that is when we initialize the auth_token middleware object, not handling a request | 20:49 |
| *** yfaykya has quit IRC | 20:49 | |
| morganfainberg | nealph, so in theory you could have 1 tmpdir per auth_token middleware object initialized (should be at most 1 per thread) | 20:50 |
| *** yfaykya has joined #tripleo | 20:50 | |
| *** jp_at_hp has joined #tripleo | 20:52 | |
| nealph | one per keystone client object, yes. | 20:53 |
| nealph | it becomes an issue. | 20:53 |
| morganfainberg | do you have information on how fast the directories are created? | 20:53 |
| morganfainberg | i want to make sure i have a good understanding of what is going on with this so it can be resolved appropriately. | 20:54 |
| morganfainberg | inc. possible reprioritisation. | 20:54 |
| *** yfaykya has quit IRC | 20:55 | |
| nealph | I don't have stats on hand....this is a known production issue for us so we have mitigated by specifying KEYSTONE_SIGNING_DIR. They would frown on me reverting that fix to get stats. :) | 20:56 |
| *** blamar has joined #tripleo | 20:56 | |
| *** yfaykya has joined #tripleo | 20:56 | |
| morganfainberg | nealph, if you have some basic stats on how fast it is occuring that would be great, doesn't need to be in depth | 20:56 |
| morganfainberg | nealph, i'll keep looking to see what we can do. | 20:57 |
| nealph | morganfainberg:thx...though my current focus is to get this workaround into the tripleo template. | 20:59 |
| *** untriaged-bot has joined #tripleo | 21:00 | |
| untriaged-bot | Untriaged bugs so far: | 21:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1329238 | 21:00 |
| uvirtbot | Launchpad bug 1329238 in tripleo "OVS isn't persisting mac addresses on OVS bridges" [Undecided,In progress] | 21:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1331337 | 21:00 |
| uvirtbot | Launchpad bug 1331337 in tripleo "install_dependencies fails if python-software-properties is not installed." [Undecided,In progress] | 21:00 |
| *** untriaged-bot has quit IRC | 21:00 | |
| *** yamahata has joined #tripleo | 21:00 | |
| nealph | morganfainberg:that's not to sound short-sighted, just realistinc...a proper fix may be a ways off. | 21:00 |
| morganfainberg | nealph, ++ | 21:00 |
| *** yfaykya has quit IRC | 21:01 | |
| lifeless | nealph: sorry, we seem to be failing to communicate | 21:01 |
| lifeless | nealph: The TEMPLATE IS NOT WHERE THE FIX HAPPENS. | 21:01 |
| lifeless | nealph: I have suggested multiple times how to do the workaround in the tripleo context. | 21:02 |
| lifeless | nealph: I really want to help you do it that way. | 21:02 |
| lifeless | nealph: but you need to let go of the idea of this being a heat template change. Its not. | 21:02 |
| lifeless | nealph: its a tripleo-image-elements change that is needed. | 21:02 |
| *** yfaykya has joined #tripleo | 21:03 | |
| lifeless | nealph: sorry if I sound grumpy (and I'm sure I do) but I've run out of alternative ways of saying it. | 21:04 |
| *** jdob has quit IRC | 21:04 | |
| NobodyCam | off the wall question. is there a way to slow down heat? | 21:04 |
| lifeless | NobodyCam: run it under qemu ? | 21:04 |
| NobodyCam | lol.. | 21:04 |
| lifeless | NobodyCam: can I ask why ? | 21:04 |
| NobodyCam | lifeless: I am working with the es guys to see whats going on in there env | 21:04 |
| NobodyCam | and while scripted runs are failing seem to be able to manually nova boot the images from the undercloud | 21:05 |
| NobodyCam | s/faling seem/faling. I seem/ | 21:05 |
| *** dtantsur is now known as dtantsur|afk | 21:07 | |
| *** yfaykya has quit IRC | 21:07 | |
| bnemec | From 192.0.2.46 icmp_seq=1 Destination Host Unreachable <-- Well hello, gate bug. So nice of you to finally show up in my local devtest run. | 21:08 |
| *** yfaykya has joined #tripleo | 21:09 | |
| vinsh | greghaynes, btw.. in multi controller setup.. I randomly hit this often: | 21:10 |
| vinsh | ==> /var/log/upstart/os-collect-config.log <== | 21:10 |
| vinsh | [ALERT] 168/210830 (20789) : Starting proxy heat_cfn: cannot bind socket | 21:10 |
| vinsh | failed! | 21:10 |
| vinsh | looks like heat-cfn is fighting for the same bind as haproxy | 21:10 |
| nealph | lifeless:no, no, I get your frustration...not trying to sound obstinate here...just trying to understand and adjust my approach. I really am open, I promise. Lemme parse back through and give it more thought. | 21:11 |
| vinsh | nealph, have you sent in your union dues? :) | 21:12 |
| *** yfaykya has quit IRC | 21:13 | |
| vinsh | SpamapS, so, in my heat.conf I see for the heat_api_cfn section.. that bind 0.0.0.0 is commented out. | 21:14 |
| lifeless | nealph: try this - grep 'signing_dir' tripleo_image_elements -r | 21:14 |
| lifeless | nealph: in the upstream tree, not your helion tree | 21:15 |
| *** yfaykya has joined #tripleo | 21:15 | |
| vinsh | SpamapS, BUT. heat-api still binds there... ignoring the real ip it should bind to | 21:15 |
| *** jtomasek has quit IRC | 21:16 | |
| *** yamahata has quit IRC | 21:18 | |
| vinsh | ah, its cause its not defined for all sections of heat.conf.. so it assumes 0.0.0.0 | 21:18 |
| *** yamahata has joined #tripleo | 21:18 | |
| *** yfaykya has quit IRC | 21:19 | |
| *** dprince has quit IRC | 21:19 | |
| *** dprince has joined #tripleo | 21:20 | |
| *** martyntaylor has quit IRC | 21:20 | |
| *** yfaykya has joined #tripleo | 21:21 | |
| SpamapS | vinsh: all sections? | 21:23 |
| SpamapS | vinsh: if there's a bug please report and I'll bump it to high priority. | 21:23 |
| vinsh | Yeah, so there are sections like "heat_api_cloudwatch" and "heat_api" | 21:23 |
| vinsh | and it looks like you have to have "bind" set in each of them | 21:23 |
| vinsh | if its not set.. that sub-service just binds to 0.0.0.0 | 21:24 |
| SpamapS | vinsh: well I suppose that makes sense.. they're all independently configurable for a reason. | 21:24 |
| vinsh | I'll propose a patch to the element.. ok? | 21:24 |
| lifeless | vinsh: patch incoming ? | 21:24 |
| lifeless | vinsh: +1 | 21:24 |
| vinsh | k. | 21:24 |
| lifeless | I think we can go with the same parameter for now | 21:24 |
| lifeless | just pushing it down to all the sections | 21:25 |
| vinsh | it exists in all sections in the image-element but is commented out in all but one. I changed the one I had on-node and all is well. | 21:25 |
| *** yfaykya has quit IRC | 21:25 | |
| *** weshay has quit IRC | 21:25 | |
| *** ramishra has joined #tripleo | 21:25 | |
| vinsh | hmm looks like https://review.openstack.org/#/c/61376/5/elements/heat/os-config-applier/etc/heat/heat.conf might need refactoring after. | 21:26 |
| vinsh | i'll propose an update to that one actually.. as its the go-forward | 21:27 |
| vinsh | make sense? | 21:27 |
| *** yfaykya has joined #tripleo | 21:27 | |
| greghaynes | you dont basically want that for each section? | 21:27 |
| vinsh | I do want that for each section | 21:27 |
| greghaynes | ok, SGTM | 21:27 |
| *** ramishra has quit IRC | 21:30 | |
| SpamapS | vinsh: you could probably propose a change to Heat to have the single default section bind_host be the default for all of the others. | 21:31 |
| SpamapS | given the name of the section.. it kind of makes sense. ;) | 21:31 |
| *** yfaykya has quit IRC | 21:32 | |
| vinsh | hmmmm | 21:32 |
| *** dkehn_ has joined #tripleo | 21:33 | |
| vinsh | which one is the defaults now? | 21:33 |
| *** yfaykya has joined #tripleo | 21:33 | |
| vinsh | ohhh | 21:33 |
| *** dkehnx has quit IRC | 21:34 | |
| *** dkehn has quit IRC | 21:34 | |
| vinsh | oh.. while I am in my element repo.. time to turn neutron logging to "shut the hell up!!" | 21:35 |
| *** dkehn has joined #tripleo | 21:36 | |
| bnemec | So, weirdness in my failing devtest run, possibly related to the overcloud instance ping failure... | 21:37 |
| bnemec | My controller is assigned 192.0.2.23: | 21:38 |
| *** yfaykya has quit IRC | 21:38 | |
| bnemec | | ae50138a-7f68-4b59-aa40-1195528366e3 | overcloud-controller0-fquwsldeflig | ACTIVE | - | Running | ctlplane=192.0.2.23 | | 21:38 |
| bnemec | But my compute node is configured to talk to 192.0.2.22: | 21:39 |
| bnemec | [root@overcloud-novacompute0-3zhc2khrj6fv nova]# cat nova.conf | grep auth_host | 21:39 |
| bnemec | auth_host = 192.0.2.22 | 21:39 |
| devananda | hi! do ya'll plan to maintain a stable/XXX branch of diskimage-builder at any point? | 21:39 |
| bnemec | Yet it seems to be accessible at that address too. | 21:39 |
| *** yfaykya has joined #tripleo | 21:39 | |
| bnemec | Which I don't understand. | 21:39 |
| devananda | or just use pip package releases to version it? | 21:39 |
| bnemec | Oh, this is virtual IP stuff. | 21:41 |
| greghaynes | ahoy, a lot of that stuff did merge recently... I wonder if thats related | 21:42 |
| greghaynes | like maybe a race condition where we arent waiting long enough for keepalived to bring up the vip? | 21:43 |
| *** yfaykya has quit IRC | 21:44 | |
| greghaynes | oh, no, were pinging the floating IP which is timing out | 21:44 |
| bnemec | Yeah, but I can't ping that port even from inside the network namespace. | 21:45 |
| bnemec | Just trying to figure out what went wrong. | 21:45 |
| greghaynes | vnc console? | 21:45 |
| greghaynes | You reproduced locally? | 21:45 |
| *** yfaykya has joined #tripleo | 21:45 | |
| lifeless | bnemec: reproduced \o/ | 21:46 |
| *** jp_at_hp has quit IRC | 21:47 | |
| bnemec | I think so. I hit the "From 192.0.2.46 icmp_seq=1 Destination Host Unreachable" error in my last devtest run. | 21:47 |
| nealph | lifeless:okay, I see your point wrt neutron and nova conf...I'll adapt my change in https://review.openstack.org/#/c/96597/ to deploy the option without connecting it to a template value. Takeaway here being: if it's not an option to be differentiated betwen the nodes in the cluster, then image-elements is the right place to fix it. | 21:47 |
| *** andreaf has quit IRC | 21:48 | |
| bnemec | Crud, we don't include stackuser in the user image? | 21:48 |
| *** morazi has quit IRC | 21:48 | |
| lifeless | nealph: generally yes - if we need the ability to control it from outside, then t-h-t becomes relevant | 21:48 |
| lifeless | nealph: the key thing here is that there is no need to vary this at all, ever. | 21:48 |
| lifeless | nealph: there *is* a right answer | 21:49 |
| vinsh | SpamapS, I think this change goes better along with the other haproxy/stunnel changes. I will just move it further up in that one to the defaults section. | 21:50 |
| *** yfaykya has quit IRC | 21:50 | |
| *** jang has quit IRC | 21:50 | |
| * nealph might just be guilty of over-thinking a fix. :) | 21:51 | |
| *** yfaykya has joined #tripleo | 21:52 | |
| openstackgerrit | Adam Vinsh proposed a change to openstack/tripleo-image-elements: Update openstack services to listen on stunnel connect port https://review.openstack.org/61376 | 21:53 |
| vinsh | SpamapS, ^^ | 21:53 |
| greghaynes | vinsh: did you just move it up into defaults? | 21:54 |
| vinsh | ya | 21:54 |
| greghaynes | SpamapS seemed unsure if that would actually work, but +1 if it does | 21:54 |
| vinsh | it will only work if haproxy or haproxy+stunnel is used.. which is really the only place it matters | 21:55 |
| vinsh | I have a stack up on the way now. will have to see :) oops submitted before confirmed :) | 21:55 |
| greghaynes | Yes, i mean whether heat respects that value in the defaults section | 21:55 |
| vinsh | Respect my heautority! | 21:56 |
| *** yfaykya has quit IRC | 21:56 | |
| vinsh | I had thought what he was saying was "use the defaults section cause thats what its for" | 21:56 |
| greghaynes | *shrug* easy to test, and if it doesnt respect that then we should make heat respect your heautority | 21:57 |
| vinsh | ya. | 21:57 |
| *** yfaykya has joined #tripleo | 21:58 | |
| bnemec | [ 110.758345] cloud-init[434]: ci-info: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!Route info failed!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! | 21:58 |
| bnemec | No message with that many !'s can possibly be a good thing | 21:58 |
| vinsh | its got flair. | 21:59 |
| bnemec | Rest of the user instance console-log, for anyone interested: http://paste.openstack.org/show/84433/ | 22:00 |
| greghaynes | ISE | 22:00 |
| greghaynes | there it goes | 22:00 |
| *** dprince has quit IRC | 22:02 | |
| *** yfaykya has quit IRC | 22:03 | |
| lifeless | bnemec: | 22:03 |
| lifeless | [ 110.734631] cloud-init[434]: ci-info: | eth0: | False | . | . | . | | 22:03 |
| *** yfaykya has joined #tripleo | 22:04 | |
| bnemec | lifeless: Yeah, I'm not sure that's abnormal though. I just booted an instance on my local OpenStack and it says the same thing, but it got route information fine. | 22:05 |
| bnemec | http://paste.openstack.org/show/84434/ | 22:05 |
| *** noslzzp has quit IRC | 22:05 | |
| *** blamar has quit IRC | 22:05 | |
| lifeless | bnemec: its hugely abnormal | 22:05 |
| lifeless | bnemec: but it might be abnormal-in-fedora vs abnormal-due-to-this-failure | 22:07 |
| lifeless | oh yay ipmitool segfaulting. yay. | 22:07 |
| bnemec | Yeah, it certainly doesn't seem right. | 22:08 |
| *** yfaykya has quit IRC | 22:08 | |
| *** yfaykya has joined #tripleo | 22:10 | |
| * bnemec stares blankly at the ovs-vsctl output | 22:10 | |
| lifeless | so clearly it came up | 22:11 |
| lifeless | this smells of a race to me | 22:11 |
| lifeless | and we know we have one | 22:11 |
| lifeless | since we don't have the nova callback wired up properly yet | 22:11 |
| lifeless | whats the bug # we're dealingw ith here ? | 22:13 |
| openstackgerrit | lifeless proposed a change to openstack/tripleo-image-elements: Longer timeout for VIF plugging. https://review.openstack.org/101040 | 22:14 |
| openstackgerrit | Phil Neal proposed a change to openstack/tripleo-image-elements: Add Ceilometer conf option for auth signing directory https://review.openstack.org/96597 | 22:14 |
| bnemec | lifeless: https://bugs.launchpad.net/tripleo/+bug/1292105 | 22:14 |
| uvirtbot | Launchpad bug 1292105 in tripleo "CI failed pinging overcloud instance" [Critical,Triaged] | 22:14 |
| *** yfaykya has quit IRC | 22:15 | |
| openstackgerrit | lifeless proposed a change to openstack/tripleo-image-elements: Longer timeout for VIF plugging. https://review.openstack.org/101040 | 22:15 |
| *** yfaykya has joined #tripleo | 22:16 | |
| *** dkehn_ is now known as dkehnx | 22:17 | |
| *** yfaykya has quit IRC | 22:20 | |
| *** yfaykya has joined #tripleo | 22:22 | |
| *** e0ne has quit IRC | 22:25 | |
| SpamapS | jogo: hey, I heard tell that you made a fake-nova-backed tempesty test thing at one point. ? | 22:25 |
| *** e0ne has joined #tripleo | 22:26 | |
| *** ramishra has joined #tripleo | 22:26 | |
| *** chuckC has quit IRC | 22:26 | |
| jogo | the large-ops test runs with fake nova backend | 22:27 |
| *** yfaykya has quit IRC | 22:27 | |
| *** yfaykya has joined #tripleo | 22:28 | |
| *** yamahata has quit IRC | 22:28 | |
| SpamapS | jogo: ah but I left out a word.. "heat" | 22:29 |
| *** e0ne has quit IRC | 22:29 | |
| *** e0ne has joined #tripleo | 22:30 | |
| *** ramishra has quit IRC | 22:31 | |
| *** yfaykya has quit IRC | 22:33 | |
| *** e0ne has quit IRC | 22:33 | |
| *** yfaykya has joined #tripleo | 22:34 | |
| *** edmund has quit IRC | 22:35 | |
| jogo | oh yeah | 22:35 |
| jogo | I have something somewhere | 22:35 |
| jogo | now that the gate is working that is getting back to the top of my work stack | 22:35 |
| jogo | but in short I see two options for a first pass | 22:36 |
| jogo | 1) test that tickles where heat breaks now (and why we cannot gate on heat tests today) | 22:36 |
| jogo | 2) add a large-ops test that uses heat to do the same thing it does today (boot 50 instances at once) | 22:36 |
| jogo | I have just been playing with templates so far and thats it | 22:37 |
| jogo | so nothing fancy | 22:37 |
| jogo | SpamapS: unrelated question | 22:37 |
| jogo | nova's instance metadata API | 22:37 |
| jogo | does heat use it? | 22:37 |
| jogo | servers/UUID/meta | 22:37 |
| *** e0ne has joined #tripleo | 22:37 | |
| *** yfaykya has quit IRC | 22:39 | |
| SpamapS | jogo: the "why we can't gate on heat slow tests today" has been solved | 22:39 |
| SpamapS | jogo: there was an overlapping IP range | 22:39 |
| jogo | oh cool so we are moving to gating heat in the near future? | 22:40 |
| SpamapS | jogo: slowly. We need to see it pass reliably for a while first. | 22:40 |
| SpamapS | jogo: but the last week has been better I'm told | 22:40 |
| SpamapS | jogo: instance metadata, I don't think Heat uses it directly, but it is exposed | 22:40 |
| *** yfaykya has joined #tripleo | 22:41 | |
| jogo | as to the first part: awesome, so converting a large-ops test into a heat template would be nice | 22:41 |
| jogo | for a large-ops heat job I was under the impression that there were two parts -- making the template and booting it. faking the metadata service stuff | 22:42 |
| jogo | SpamapS: for instance metadata, i ask because it allows someone to set arbitrary key value pairs and have those appear on the nova metadata service inside the instance | 22:42 |
| *** e0ne has quit IRC | 22:43 | |
| jogo | SpamapS: and that sounds like something that heat would do | 22:43 |
| jogo | or rather find useful | 22:43 |
| jogo | (not saying you *should* use it though) | 22:43 |
| *** e0ne has joined #tripleo | 22:44 | |
| SpamapS | jogo: yes I believe you can explicitly set them with one of the properties of OS::Nova::Server | 22:45 |
| *** yfaykya has quit IRC | 22:45 | |
| SpamapS | jogo: http://docs.openstack.org/developer/heat/template_guide/openstack.html#OS::Nova::Server | 22:46 |
| jogo | what does heat or tripleo use to get things like that into the instnace? heat metadata server? | 22:46 |
| *** yfaykya has joined #tripleo | 22:47 | |
| *** e0ne has quit IRC | 22:48 | |
| *** chuckC has joined #tripleo | 22:49 | |
| *** yfaykya has quit IRC | 22:51 | |
| *** mestery has quit IRC | 22:51 | |
| SpamapS | jogo: EC2 metadata or configdrive | 22:51 |
| SpamapS | jogo: Heat itself will just call the Nova API | 22:52 |
| SpamapS | jogo: in-instance tools are left to fend for themselves. | 22:52 |
| *** sballe__ has quit IRC | 22:52 | |
| lifeless | jogo: SpamapS: we also get stuff from heat | 22:52 |
| lifeless | we use the ec2 metadata/config drive to bootstrap | 22:53 |
| lifeless | most things are in deployments now | 22:53 |
| *** yfaykya has joined #tripleo | 22:53 | |
| lifeless | which is heat | 22:53 |
| jogo | deployments? | 22:55 |
| lifeless | OS::Heat::StructuredDeployment | 22:56 |
| lifeless | e.g. ../tripleo-heat-templates/overcloud-source.yaml line 292 | 22:56 |
| *** yfaykya has quit IRC | 22:57 | |
| *** yfaykya has joined #tripleo | 22:59 | |
| vinsh | who knows much about "neutron ovs_db" ... as in.. when is it initialized? | 22:59 |
| jogo | so I think I am asking the wrong question | 23:00 |
| jogo | what metadata service do things like os_collect_config pull from? | 23:01 |
| jogo | heat? | 23:01 |
| lifeless | ec2 / config_drive / heat cfn / heat native | 23:01 |
| lifeless | all at once | 23:01 |
| jogo | so from ec2 you use user-data? | 23:02 |
| lifeless | SpamapS: ^ | 23:02 |
| jogo | so if I want to send some arbitrary key value into an instance how does that happen | 23:03 |
| *** yfaykya has quit IRC | 23:03 | |
| jogo | I am trying to better understand how folks usually do that | 23:03 |
| jogo | as there are many ways to do it | 23:03 |
| jogo | including nova instance.metadata | 23:03 |
| jogo | but those aren't in the ec2 metadata | 23:04 |
| lifeless | we would recommend folk doing stuff with heat use the structureddeployment | 23:04 |
| jogo | and heat gets the data in via ? | 23:04 |
| lifeless | unless the k:v is needed for instance bootstrapping | 23:04 |
| lifeless | heat makes that available via the heat cfn / native APIs | 23:04 |
| *** yfaykya has joined #tripleo | 23:05 | |
| jogo | ahh | 23:05 |
| jogo | right cloud formations | 23:05 |
| lifeless | hmm, all jobs failing | 23:06 |
| lifeless | SpamapS: is that revert for heat +A'd ? | 23:07 |
| lifeless | SpamapS: if so can we ask infra to promote it please | 23:08 |
| lifeless | SpamapS: if not we should land 100955 | 23:08 |
| *** saurabhs has left #tripleo | 23:08 | |
| *** yfaykya has quit IRC | 23:10 | |
| *** yfaykya has joined #tripleo | 23:11 | |
| *** gfidente has quit IRC | 23:14 | |
| jogo | lifeless SpamapS: thanks for the clarification, the answer is heat serves the metadata is the prefered method | 23:14 |
| *** yfaykya has quit IRC | 23:16 | |
| *** yfaykya has joined #tripleo | 23:17 | |
| *** BadCub01 has joined #tripleo | 23:17 | |
| openstackgerrit | A change was merged to openstack/tripleo-specs: TripleO on OpenStack https://review.openstack.org/92642 | 23:18 |
| openstackgerrit | Michael Tupitsyn proposed a change to openstack/os-cloud-config: Do not create admin user if it exists already https://review.openstack.org/100719 | 23:19 |
| *** BadCub01 has quit IRC | 23:19 | |
| *** CaptTofu_ has quit IRC | 23:21 | |
| *** CaptTofu_ has joined #tripleo | 23:22 | |
| *** yfaykya has quit IRC | 23:22 | |
| *** yfaykya has joined #tripleo | 23:23 | |
| *** mordred has quit IRC | 23:24 | |
| openstackgerrit | Michael Tupitsyn proposed a change to openstack/os-cloud-config: Do not create admin user if it exists already https://review.openstack.org/100719 | 23:26 |
| *** CaptTofu_ has quit IRC | 23:26 | |
| vinsh | greghaynes, SpamapS looks like my patch to the ssl ports https://review.openstack.org/#/c/61376/ worked: http://paste.openstack.org/show/84437/ | 23:27 |
| vinsh | testing shows goodness. | 23:27 |
| *** mordred has joined #tripleo | 23:27 | |
| *** ramishra has joined #tripleo | 23:27 | |
| vinsh | maybe not for heat-engine though. whatever its doing. whoknows. | 23:27 |
| vinsh | no errors from it. | 23:27 |
| *** yfaykya has quit IRC | 23:28 | |
| *** CaptTofu_ has joined #tripleo | 23:29 | |
| *** yfaykya has joined #tripleo | 23:30 | |
| *** mordred has quit IRC | 23:30 | |
| *** mordred has joined #tripleo | 23:30 | |
| *** ramishra has quit IRC | 23:31 | |
| *** yfaykya has quit IRC | 23:34 | |
| *** yfaykya has joined #tripleo | 23:36 | |
| *** yfaykya has quit IRC | 23:40 | |
| SpamapS | vinsh: heat-engine doesn't listen | 23:41 |
| *** yfaykya has joined #tripleo | 23:42 | |
| vinsh | k | 23:44 |
| *** yfaykya has quit IRC | 23:46 | |
| lifeless | vinsh: heat-engine is RPC only | 23:47 |
| *** yfaykya has joined #tripleo | 23:48 | |
| greghaynes | vinsh: mmm goodness | 23:51 |
| greghaynes | looks like the tests were not so happy | 23:51 |
| greghaynes | oh what, trust checksum mismatch | 23:51 |
| vinsh | now I got this neutron port-list giving me nothing back for overcloud vm | 23:51 |
| vinsh | trying to figure out whos not talking to who on that one. | 23:52 |
| greghaynes | http://logs.openstack.org/76/61376/6/check-tripleo/check-tripleo-ironic-seed-precise/52ef13c/console.html#_2014-06-18_22_03_03_399 | 23:52 |
| lifeless | SpamapS: ping | 23:52 |
| lifeless | SpamapS: did you see my q above? I want to unblock tripleo | 23:52 |
| *** yfaykya has quit IRC | 23:52 | |
| vinsh | greghaynes, woah. | 23:52 |
| greghaynes | did they push out new images but not update shas yet? | 23:52 |
| greghaynes | looks like we didnt cache hit on the image... and I dont think we cache shas | 23:53 |
| greghaynes | yea, cache miss on shas | 23:53 |
| *** lucas-dinner has quit IRC | 23:53 | |
| * greghaynes shakes head at cloud-images.ubuntu.com | 23:53 | |
| *** yfaykya has joined #tripleo | 23:54 | |
| greghaynes | vinsh: did a recheck no bug on it | 23:54 |
| lifeless | greghaynes: pointless cycles until my q for SpamapS is sorted | 23:55 |
| greghaynes | oh | 23:55 |
| *** lifeless changes topic to "CI DOWN, HEAT regression and overcloud failing | Using OpenStack to deploy OpenStack;meetings Tuesday 1900//0700 UTC in #openstack-meeting-alt" | 23:55 | |
| greghaynes | lifeless: the heat fix was +A'd | 23:55 |
| greghaynes | er revert | 23:55 |
| greghaynes | lifeless: dont think its merged yet though | 23:56 |
| lifeless | greghaynes: how far down the queue is it ? | 23:56 |
| lifeless | we either need to ask for it to be promoted - its a regression fix - or land the workaround | 23:56 |
| lifeless | or we'll be down for another day | 23:56 |
| greghaynes | lifeless: looks like its merged | 23:58 |
| *** yfaykya has quit IRC | 23:58 | |
| greghaynes | https://review.openstack.org/#/c/101001/ | 23:58 |
| lifeless | retrying https://review.openstack.org/#/c/101040/ then | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
