| *** ooolpbot has joined #tripleo | 00:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 00:10 |
|---|---|---|
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1638350 | 00:10 |
| *** ooolpbot has quit IRC | 00:10 | |
| openstack | Launchpad bug 1638350 in tripleo "pingtest failing on OVB jobs to create Cinder volume and Nova server" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 00:10 |
| *** jerrygb has joined #tripleo | 00:17 | |
| *** jerrygb_ has joined #tripleo | 00:20 | |
| *** jerrygb has quit IRC | 00:23 | |
| EmilienM | ayoung: so I guess I'll have to take over your fernet work in tripleo right? | 00:31 |
| *** limao has joined #tripleo | 00:31 | |
| *** limao has quit IRC | 00:32 | |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-tripleo: Create heat user in keystone profile https://review.openstack.org/392288 | 00:32 |
| EmilienM | mwhahaha: with Depends-On the revert ^ | 00:32 |
| *** limao has joined #tripleo | 00:32 | |
| *** cwolferh has joined #tripleo | 00:36 | |
| ayoung | EmilienM, it will happen faster if you do it, but if it is breaking things, I can work on it. We were not planning on Fernet as part of the installer until Ocata came out. Its in the plan, just was not yet a firedrill | 00:37 |
| ayoung | is it a firedrill now? | 00:37 |
| *** dmacpher has joined #tripleo | 00:37 | |
| *** jerrygb_ has quit IRC | 00:52 | |
| *** jkilpatr has quit IRC | 00:55 | |
| *** rlandy has quit IRC | 00:58 | |
| *** dsneddon is now known as dsneddon_ | 01:00 | |
| *** dsneddon has joined #tripleo | 01:04 | |
| *** morazi has joined #tripleo | 01:04 | |
| *** ooolpbot has joined #tripleo | 01:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1638350 | 01:10 |
| *** ooolpbot has quit IRC | 01:10 | |
| openstack | Launchpad bug 1638350 in tripleo "pingtest failing on OVB jobs to create Cinder volume and Nova server" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 01:10 |
| *** bana_k has quit IRC | 01:12 | |
| *** [1]cdearborn has quit IRC | 01:17 | |
| ggillies | is anyone around who knows how user supplied environment files work with the new TripleO Newton mistral powered overcloud deploy workflow? | 01:22 |
| ggillies | anyway lodged a (hopefully valid) bug https://bugs.launchpad.net/tripleo/+bug/1638428 | 01:35 |
| openstack | Launchpad bug 1638428 in tripleo "Symlinks to environment files don't work when using --environment-directory" [Undecided,New] | 01:35 |
| *** maticue has quit IRC | 01:36 | |
| *** jerrygb has joined #tripleo | 02:05 | |
| *** ooolpbot has joined #tripleo | 02:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1638350 | 02:10 |
| openstack | Launchpad bug 1638350 in tripleo "pingtest failing on OVB jobs to create Cinder volume and Nova server" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 02:10 |
| *** ooolpbot has quit IRC | 02:10 | |
| *** ebalduf has quit IRC | 02:11 | |
| *** fultonj has quit IRC | 02:21 | |
| *** bana_k has joined #tripleo | 02:26 | |
| *** dhill_ has quit IRC | 02:42 | |
| *** dhill__ has joined #tripleo | 02:44 | |
| *** ramishra_ has joined #tripleo | 02:50 | |
| *** ramishra has quit IRC | 02:51 | |
| *** ebalduf has joined #tripleo | 03:05 | |
| *** bana_k has quit IRC | 03:08 | |
| *** ooolpbot has joined #tripleo | 03:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1638350 | 03:10 |
| *** ooolpbot has quit IRC | 03:10 | |
| openstack | Launchpad bug 1638350 in tripleo "pingtest failing on OVB jobs to create Cinder volume and Nova server" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 03:10 |
| openstackgerrit | RedHat RDO CI proposed openstack/tripleo-heat-templates: GATE TEST, please ignore https://review.openstack.org/365449 | 03:30 |
| *** cwolferh has quit IRC | 03:33 | |
| *** lifeless has quit IRC | 03:41 | |
| *** lifeless has joined #tripleo | 03:42 | |
| *** ebalduf has quit IRC | 03:49 | |
| *** flepied has joined #tripleo | 03:54 | |
| *** morazi has quit IRC | 03:55 | |
| *** absubram has joined #tripleo | 04:04 | |
| *** absubram_ has joined #tripleo | 04:05 | |
| *** absubram has quit IRC | 04:08 | |
| *** absubram_ is now known as absubram | 04:08 | |
| *** coolsvap has joined #tripleo | 04:09 | |
| *** ooolpbot has joined #tripleo | 04:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1638350 | 04:10 |
| openstack | Launchpad bug 1638350 in tripleo "pingtest failing on OVB jobs to create Cinder volume and Nova server" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 04:10 |
| *** ooolpbot has quit IRC | 04:10 | |
| *** jerrygb has quit IRC | 04:19 | |
| *** Vijayendra has joined #tripleo | 04:22 | |
| *** jerrygb has joined #tripleo | 04:24 | |
| *** chlong has quit IRC | 04:24 | |
| *** chandankumar has joined #tripleo | 04:34 | |
| *** absubram has quit IRC | 04:37 | |
| *** limao has quit IRC | 04:44 | |
| *** tzumainn has quit IRC | 05:02 | |
| *** jerrygb has quit IRC | 05:03 | |
| *** limao has joined #tripleo | 05:04 | |
| *** jerrygb has joined #tripleo | 05:06 | |
| *** ooolpbot has joined #tripleo | 05:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1638350 | 05:10 |
| openstack | Launchpad bug 1638350 in tripleo "pingtest failing on OVB jobs to create Cinder volume and Nova server" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 05:10 |
| *** ooolpbot has quit IRC | 05:10 | |
| *** dsariel has joined #tripleo | 05:35 | |
| *** saneax-_-|AFK is now known as saneax | 05:37 | |
| *** oshvartz has quit IRC | 05:39 | |
| *** dsariel has quit IRC | 05:47 | |
| *** pkovar has joined #tripleo | 05:48 | |
| *** jerrygb has quit IRC | 05:53 | |
| *** skramaja has joined #tripleo | 05:54 | |
| *** dmacpher has quit IRC | 05:55 | |
| *** jerrygb has joined #tripleo | 05:57 | |
| *** cwolferh has joined #tripleo | 05:59 | |
| *** dsneddon has quit IRC | 05:59 | |
| *** bkopilov has joined #tripleo | 06:04 | |
| *** ooolpbot has joined #tripleo | 06:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1638350 | 06:10 |
| openstack | Launchpad bug 1638350 in tripleo "pingtest failing on OVB jobs to create Cinder volume and Nova server" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 06:10 |
| *** ooolpbot has quit IRC | 06:10 | |
| openstackgerrit | Noam Angel proposed openstack/diskimage-builder: redhat-common add option to select networking service NetworkManager/network https://review.openstack.org/392170 | 06:12 |
| openstackgerrit | Noam Angel proposed openstack/diskimage-builder: simplify ARCH param for rhel/centos param can be x86_64 and amd64 https://review.openstack.org/392176 | 06:15 |
| *** dsariel has joined #tripleo | 06:17 | |
| *** jerrygb has quit IRC | 06:22 | |
| *** jerrygb has joined #tripleo | 06:27 | |
| *** dsariel has quit IRC | 06:30 | |
| *** ramishra_ has quit IRC | 06:34 | |
| *** ramishra has joined #tripleo | 06:34 | |
| *** ramishra has quit IRC | 06:35 | |
| *** ramishra has joined #tripleo | 06:35 | |
| *** dsariel has joined #tripleo | 06:36 | |
| *** limao has quit IRC | 06:36 | |
| *** limao has joined #tripleo | 06:38 | |
| *** abregman has joined #tripleo | 06:41 | |
| *** dsariel has quit IRC | 06:46 | |
| *** limao has quit IRC | 06:48 | |
| *** rcernin has joined #tripleo | 06:50 | |
| *** jbadiapa has joined #tripleo | 06:55 | |
| *** lmiccini has joined #tripleo | 06:58 | |
| *** ramishra has quit IRC | 06:59 | |
| *** ramishra has joined #tripleo | 06:59 | |
| *** ramishra has quit IRC | 06:59 | |
| *** ramishra has joined #tripleo | 07:03 | |
| *** jerrygb has quit IRC | 07:06 | |
| *** rasca has joined #tripleo | 07:07 | |
| *** ooolpbot has joined #tripleo | 07:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1638350 | 07:10 |
| *** ooolpbot has quit IRC | 07:10 | |
| openstack | Launchpad bug 1638350 in tripleo "pingtest failing on OVB jobs to create Cinder volume and Nova server" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 07:10 |
| *** jerrygb has joined #tripleo | 07:10 | |
| bandini | mornin' | 07:11 |
| *** ohamada has joined #tripleo | 07:11 | |
| *** oshvartz has joined #tripleo | 07:12 | |
| *** tesseract has joined #tripleo | 07:17 | |
| *** tesseract is now known as Guest48712 | 07:17 | |
| *** limao has joined #tripleo | 07:24 | |
| *** aufi has joined #tripleo | 07:25 | |
| *** limao_ has joined #tripleo | 07:27 | |
| *** limao has quit IRC | 07:28 | |
| *** limao has joined #tripleo | 07:29 | |
| *** ealcaniz has joined #tripleo | 07:29 | |
| *** fragatina has joined #tripleo | 07:30 | |
| *** tremble has joined #tripleo | 07:31 | |
| *** limao_ has quit IRC | 07:31 | |
| *** limao has quit IRC | 07:32 | |
| *** limao_ has joined #tripleo | 07:32 | |
| *** dsariel has joined #tripleo | 07:34 | |
| *** liverpooler has joined #tripleo | 07:36 | |
| *** fragatina has quit IRC | 07:39 | |
| *** fragatina has joined #tripleo | 07:39 | |
| openstackgerrit | Noam Angel proposed openstack/diskimage-builder: add element cloud-init-allow-password-authentication https://review.openstack.org/391765 | 07:40 |
| *** bana_k has joined #tripleo | 07:41 | |
| *** jerrygb has quit IRC | 07:42 | |
| *** hjensas has quit IRC | 07:44 | |
| *** panda|Zz is now known as panda | 07:45 | |
| *** jerrygb has joined #tripleo | 07:46 | |
| *** openstackgerrit has quit IRC | 07:48 | |
| *** openstackgerrit has joined #tripleo | 07:48 | |
| *** jerrygb_ has joined #tripleo | 07:50 | |
| *** ccamacho has joined #tripleo | 07:51 | |
| ccamacho | morning guys! | 07:51 |
| *** jerrygb has quit IRC | 07:52 | |
| *** hjensas has joined #tripleo | 07:52 | |
| openstackgerrit | Noam Angel proposed openstack/diskimage-builder: add option to configure cloud-init to allow password authentication https://review.openstack.org/391765 | 07:52 |
| *** jerrygb has joined #tripleo | 07:56 | |
| *** hjensas has quit IRC | 07:58 | |
| *** jerrygb_ has quit IRC | 07:58 | |
| *** zoli_gone-proxy is now known as zoliXXL | 07:59 | |
| *** hjensas has joined #tripleo | 07:59 | |
| *** zoliXXL is now known as zoli|wfh | 07:59 | |
| *** hjensas has quit IRC | 07:59 | |
| *** jprovazn has joined #tripleo | 08:03 | |
| *** limao has joined #tripleo | 08:04 | |
| *** yamahata has quit IRC | 08:07 | |
| *** jerrygb has quit IRC | 08:08 | |
| *** pcaruana has joined #tripleo | 08:09 | |
| *** ooolpbot has joined #tripleo | 08:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1638350 | 08:10 |
| openstack | Launchpad bug 1638350 in tripleo "pingtest failing on OVB jobs to create Cinder volume and Nova server" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 08:10 |
| *** ooolpbot has quit IRC | 08:10 | |
| *** zoli|wfh is now known as zoli_gone-proxy | 08:12 | |
| *** jerrygb has joined #tripleo | 08:13 | |
| *** bana_k has quit IRC | 08:14 | |
| *** jlinkes has joined #tripleo | 08:17 | |
| *** ebarrera has joined #tripleo | 08:17 | |
| *** dciabrin has joined #tripleo | 08:19 | |
| *** zoli_gone-proxy is now known as zoliXXL | 08:20 | |
| openstackgerrit | Florian Fuchs proposed openstack/tripleo-ui: Use mistral action to create new containers https://review.openstack.org/392189 | 08:26 |
| *** jerrygb_ has joined #tripleo | 08:27 | |
| *** athomas has joined #tripleo | 08:28 | |
| *** jerrygb has quit IRC | 08:29 | |
| *** shardy has joined #tripleo | 08:30 | |
| *** hewbrocca_afk is now known as hewbrocca | 08:33 | |
| *** sshnaidm has quit IRC | 08:35 | |
| *** jpena|off is now known as jpena | 08:37 | |
| *** fzdarsky|afk has joined #tripleo | 08:37 | |
| *** mcornea has joined #tripleo | 08:38 | |
| *** sshnaidm has joined #tripleo | 08:39 | |
| *** florianf has joined #tripleo | 08:39 | |
| *** abregman has quit IRC | 08:40 | |
| *** fzdarsky|afk has quit IRC | 08:42 | |
| *** hjensas has joined #tripleo | 08:43 | |
| *** hjensas has joined #tripleo | 08:43 | |
| *** gfidente|afk is now known as gfidente | 08:44 | |
| *** abregman has joined #tripleo | 08:44 | |
| *** jaosorior has joined #tripleo | 08:45 | |
| *** abregman has quit IRC | 08:45 | |
| *** jpich has joined #tripleo | 08:47 | |
| *** abregman has joined #tripleo | 08:49 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: WIP TLS everywhere job https://review.openstack.org/391738 | 08:51 |
| *** lucas-afk is now known as lucasagomes | 08:51 | |
| *** abregman is now known as abregman|mtg | 08:52 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Enable TLS in the internal network for Cinder API https://review.openstack.org/392090 | 08:53 |
| jaosorior | jistr: hey dude, can we merge this blueprint https://review.openstack.org/#/c/282307/ ? | 08:54 |
| *** tosky has joined #tripleo | 08:54 | |
| jaosorior | or shardy ^^ | 08:55 |
| jaosorior | or marios ^^ | 08:55 |
| *** chem has joined #tripleo | 08:56 | |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-specs: Composable Service Upgrades https://review.openstack.org/392116 | 08:57 |
| *** chem has quit IRC | 08:59 | |
| *** chem has joined #tripleo | 08:59 | |
| *** limao has quit IRC | 09:00 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Add missing Barbican endpoint from tls-everywhere environment https://review.openstack.org/392473 | 09:01 |
| openstackgerrit | Julie Pichon proposed openstack/tripleo-ui: Include nested parameters in service parameters https://review.openstack.org/392474 | 09:01 |
| *** fzdarsky has joined #tripleo | 09:03 | |
| *** paramite has joined #tripleo | 09:05 | |
| jtomasek | florianf: hi, could you please check this one again, I had to do a slight update to it https://review.openstack.org/#/c/391855/ | 09:05 |
| jtomasek | jpich: do you plan to create backports also for other patches in GUI? | 09:07 |
| jpich | jtomasek: I saw jrist's update on the bug for that one since I was following it, are you aware of other must-have? | 09:08 |
| florianf | jtomasek: yep | 09:08 |
| *** jerrygb_ has quit IRC | 09:08 | |
| *** limao has joined #tripleo | 09:08 | |
| *** ooolpbot has joined #tripleo | 09:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1638350 | 09:10 |
| *** ooolpbot has quit IRC | 09:10 | |
| openstack | Launchpad bug 1638350 in tripleo "pingtest failing on OVB jobs to create Cinder volume and Nova server" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 09:10 |
| jtomasek | jpich: I think all of the recent patches are bugfixes which should get backported | 09:10 |
| jtomasek | jpich: I am going to create backports, no problem | 09:10 |
| jpich | jtomasek: Ok, great!! | 09:11 |
| *** jerrygb has joined #tripleo | 09:12 | |
| jtomasek | florianf: seems like jenkins is being weird :/ funny it always happens on the day we need to get stuff merged | 09:13 |
| jaosorior | ccamacho: thanks dude | 09:13 |
| ccamacho | jaosorior np man :) | 09:13 |
| *** jaosorior has quit IRC | 09:14 | |
| *** jaosorior has joined #tripleo | 09:14 | |
| jpich | jtomasek: I could use some advice on how to debug the test failure on the backport I just proposed... http://logs.openstack.org/74/392474/1/check/gate-tripleo-ui-nodejs4-npm-run-test/12b1fe3/console.html#_2016-11-02_09_07_33_998433 or is that the weird stuff you're talking about? | 09:14 |
| jtomasek | jpich: it is the same one that florianf's patch is hitting... there must be some problem at the gate | 09:15 |
| jpich | jtomasek: Also not sure if you saw but the talk now is of cutting the new stable release in 2 weeks together with Ocata-1, rather than tomorrow? | 09:15 |
| jtomasek | tests pass locally for me for those patches | 09:15 |
| openstackgerrit | Merged openstack/tripleo-specs: Internal TLS using certmonger https://review.openstack.org/282307 | 09:15 |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-quickstart: Fix tempest runner in devmode https://review.openstack.org/391312 | 09:15 |
| jaosorior | marios: by the way, the ovb-ha gate is broken :/ | 09:16 |
| jtomasek | jpich: oh, so things have changed from what was said yesterday on the tripleo meeting? | 09:16 |
| jpich | jtomasek: Huh that's I understood from yesterday's meeting | 09:20 |
| jpich | jtomasek: but I wasn't in the best of form and may have misunderstood so ignore what I said unless someone else talks about it later :-) | 09:20 |
| *** abregman_ has joined #tripleo | 09:21 | |
| *** dtantsur|afk is now known as dtantsur | 09:21 | |
| ccamacho | jaosorior, dude just found a nit in the TLS spec... | 09:22 |
| jaosorior | ccamacho: could fix it up | 09:24 |
| *** abregman_ has quit IRC | 09:24 | |
| jaosorior | ccamacho: even if it's already merged, I can send a fix for it | 09:24 |
| jpich | jtomasek: The meeting notes sent to the list seem to confirm what I understood, newton-2 the week of Nov 14th | 09:24 |
| *** abregman|mtg has quit IRC | 09:24 | |
| *** derekh has joined #tripleo | 09:26 | |
| *** abregman has joined #tripleo | 09:26 | |
| *** milan has quit IRC | 09:26 | |
| jtomasek | jpich: ok so there are going to be 2 releases, see 14:34:18 at http://eavesdrop.openstack.org/meetings/tripleo/2016/tripleo.2016-11-01-14.00.log.html | 09:27 |
| jpich | jtomasek: Yeah I'm not reading it like you (to me the discussion after obsoletes that statement) but it's fine, getting backports in is good either way :) | 09:28 |
| jtomasek | ok, we'll see:) | 09:29 |
| florianf | jtomasek, jpich: I'm also not sure how to read that, tbh. | 09:29 |
| marios | jaosorior: yah seen a couple of things with failed pingtest today | 09:32 |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-specs: Fix nit in document https://review.openstack.org/392496 | 09:32 |
| ccamacho | jaosorior mind to quickly check this https://review.openstack.org/#/c/392496 ? | 09:33 |
| jtomasek | florianf, jpich: ok I've managed to reproduce the failing tests locally, investigating further | 09:35 |
| florianf | jtomasek:tests run fine for me. what did you do to make them fail? | 09:38 |
| openstackgerrit | Merged openstack/tripleo-specs: Fix nit in document https://review.openstack.org/392496 | 09:38 |
| jtomasek | florianf: use node v 4.6.1 | 09:38 |
| florianf | jtomasek: oh | 09:38 |
| jtomasek | florianf: hm, wait, maybe that was not it, I've removed node modules, cleared npm cache and re run npm install | 09:39 |
| jtomasek | florianf: can you try doing the same? | 09:40 |
| jtomasek | florianf: but keep the original node modules around please:) | 09:40 |
| jtomasek | florianf: hm no, lets compare npm list first | 09:40 |
| jtomasek | florianf: unless you already deleted it:) | 09:41 |
| florianf | jtomasek: I did :-) installing, but using latest node | 09:41 |
| jtomasek | ok | 09:41 |
| florianf | I'll keep it around when using 4.6.1 | 09:41 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: WIP TLS everywhere job https://review.openstack.org/391738 | 09:44 |
| florianf | jtomasek: OK, I'm getting the errors that are in the jenkings logs | 09:44 |
| florianf | jtomasek: using node v6.9.0 | 09:44 |
| jtomasek | florianf: ok, some minor version update of some package we use had to cause that | 09:44 |
| jtomasek | I suspect jasmine-immutable-matchers | 09:44 |
| florianf | jtomasek: exactly | 09:45 |
| florianf | jtomasek: toEqualImmutable is undefined | 09:45 |
| jtomasek | florianf: that is it, in package.json it is defined as ^1.0.1 and needs to be ~1.0.1 | 09:46 |
| jtomasek | florianf: because 1.1.1 probably changes the api | 09:46 |
| jtomasek | testing it now | 09:46 |
| florianf | testing... | 09:46 |
| EmilienM | hello | 09:46 |
| jaosorior | sshnaidm: is there a straightforward way that I can add a host to /etc/hosts for all the nodes in the cluster before they run puppet? | 09:46 |
| jtomasek | EmilienM: hi, so there are going to be 2 releases? one this week and another at ocata-1 end? | 09:47 |
| EmilienM | jtomasek: not sure yet | 09:48 |
| sshnaidm | jaosorior, I don't know such ways, sorry | 09:48 |
| EmilienM | during the meeting we said we would release newton-2 in same time as ocata-1 | 09:48 |
| *** jaosorior has quit IRC | 09:48 | |
| florianf | jtomasek: works now with the changed dependency version | 09:49 |
| *** jaosorior has joined #tripleo | 09:49 | |
| EmilienM | sshnaidm, panda: FYI we are still debugging CI but we have a fix | 09:49 |
| EmilienM | https://review.openstack.org/#/c/392288/ | 09:49 |
| jtomasek | florianf: yep, I am going to send a patch | 09:49 |
| jaosorior | EmilienM: that fix seems to have failed the ha gate though, is that expected? | 09:50 |
| sshnaidm | EmilienM, great, thanks | 09:50 |
| EmilienM | jaosorior: right but I don't see why. Do you ? http://logs.openstack.org/88/392288/4/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/8cb3936/console.html#_2016-11-02_02_24_03_515747 | 09:50 |
| jaosorior | EmilienM: yeah, I don't either | 09:51 |
| jaosorior | EmilienM: I looked at it earlier in the morning | 09:51 |
| jaosorior | and couldn't come up with the actual issue... | 09:51 |
| EmilienM | shardy: do you see why this pingtest would fail? | 09:51 |
| EmilienM | jaosorior: well, one thing is sure, we need https://review.openstack.org/#/c/392288/ | 09:51 |
| jaosorior | EmilienM: I +2ed it... one thing is that some resources are IN_PROGRESS... might be that it timed out? | 09:52 |
| EmilienM | jaosorior: well, no. All is Complete | 09:52 |
| EmilienM | http://logs.openstack.org/88/392288/4/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/8cb3936/console.html#_2016-11-02_02_23_32_395293 | 09:52 |
| EmilienM | even: Stack CREATE completed successfully | 09:52 |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui: Pin jasmine-immutable-matchers version to ~1.0.1 https://review.openstack.org/392510 | 09:52 |
| jaosorior | EmilienM: so, that works now but I guess it's a temporal fix, right? We still need to pass those to t-h-t, cause if we deploy keystone on a separate node than heat, that will break | 09:53 |
| jtomasek | florianf: ^^ | 09:53 |
| jaosorior | EmilienM: if you check a bit above the log you passed me, there are several resources that are in CREATE_IN_PROGRESS | 09:53 |
| florianf | jtomasek: don't we have to run shrinkwrap as well? | 09:53 |
| EmilienM | jaosorior: also, we had to revert https://review.openstack.org/#/q/Id02ab0bab8d7bd5c68e35408f72f24505e262306,n,z | 09:53 |
| jtomasek | florianf: I ran it and it did not introduce any changes | 09:53 |
| *** jerrygb has quit IRC | 09:54 | |
| jaosorior | EmilienM: yeah... that's a complicated one | 09:54 |
| florianf | jtomasek: ok | 09:54 |
| sshnaidm | EmilienM, do we have opened bug about this issuee? | 09:54 |
| florianf | jtomasek: for some reason, shrinkwrap break for me. but that's probably local | 09:54 |
| EmilienM | yes the bug in alert | 09:55 |
| EmilienM | https://launchpad.net/bugs/1638350 | 09:55 |
| openstack | Launchpad bug 1638350 in tripleo "pingtest failing on OVB jobs to create Cinder volume and Nova server" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 09:55 |
| jtomasek | florianf: youre probably hitting npm v2 vs v3 problem | 09:55 |
| florianf | jtomasek: So I should use 2? | 09:55 |
| jtomasek | florianf: if you have npm v3 the shrinkwrap format is different | 09:55 |
| jtomasek | yes | 09:55 |
| * jtomasek starts to really consider yarn | 09:56 | |
| *** dprince has joined #tripleo | 09:56 | |
| EmilienM | jaosorior: I see stack complete in heat engine logs http://logs.openstack.org/88/392288/4/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/8cb3936/logs/overcloud-controller-0/var/log/heat/heat-engine.txt.gz#_2016-11-02_02_22_49_880 | 09:56 |
| shardy | EmilienM: http://logs.openstack.org/88/392288/4/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/8cb3936/console.html#_2016-11-02_02_22_27_410273 | 09:56 |
| shardy | I think the create worked but the tripleo.sh script timed out | 09:56 |
| jtomasek | florianf: I use nvm (node version manager) and it was very helpful for runing various node/npm versions | 09:57 |
| jaosorior | EmilienM: ok, so the log was misleading. I rechecked it anyway, hopefully it's just a timeout and it passes in the next run. | 09:57 |
| jaosorior | EmilienM: when you get some time, can you check out these ones? https://review.openstack.org/#/q/topic:barbican-integration+status:open | 09:57 |
| florianf | jtomasek: thanks for the tip | 09:58 |
| EmilienM | well, I would rather try to fix tripleo.sh instead of ignoring the actual issue | 09:58 |
| EmilienM | shardy: the 300s timeout, right? | 09:58 |
| *** jerrygb has joined #tripleo | 09:58 | |
| shardy | yeah I don't see where that timeout is defined | 09:58 |
| shardy | but AFAICS it's just bad luck where the timout was hit very close to the time when the stack create completed | 09:59 |
| shardy | there aren't any errors in the heat logs | 09:59 |
| *** thrash|g0ne is now known as thrash | 09:59 | |
| shardy | https://github.com/openstack-infra/tripleo-ci/blob/master/scripts/tripleo.sh#L757 | 09:59 |
| EmilienM | right | 10:00 |
| shardy | probably that tripleo wait_for | 10:00 |
| EmilienM | should we increase it a little? | 10:00 |
| shardy | we could try that | 10:00 |
| EmilienM | I'm going to check in logstash how many times we did hit this | 10:00 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: TEST: DONT RECHECK: periodic jobs https://review.openstack.org/359215 | 10:01 |
| *** limao has quit IRC | 10:01 | |
| EmilienM | shardy: logstash says that we reached this timeout starting from October 26th | 10:03 |
| EmilienM | on 91 jobs | 10:03 |
| *** snecklifter has joined #tripleo | 10:03 | |
| snecklifter | is there an easy way to change the swift mount point in tripleo? | 10:04 |
| snecklifter | so for example we want to use local storage but a separate array | 10:05 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Ensure heat-domain hiera is in nodes that contain keystone https://review.openstack.org/392519 | 10:06 |
| jaosorior | EmilienM: that should address the issue I pointed out in the patch ^^ | 10:07 |
| *** jaosorior is now known as jaosorior_lunch | 10:08 | |
| EmilienM | jaosorior_lunch: a commit message would have been great too, but thanks for the quick patch | 10:08 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: Increase pingtest timeout by 1 minute https://review.openstack.org/392521 | 10:10 |
| *** ooolpbot has joined #tripleo | 10:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1638350 | 10:10 |
| *** ooolpbot has quit IRC | 10:10 | |
| openstack | Launchpad bug 1638350 in tripleo "pingtest failing on OVB jobs to create Cinder volume and Nova server" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 10:10 |
| EmilienM | shardy: ^ timeout | 10:10 |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-tripleo: Create heat user in keystone profile https://review.openstack.org/392288 | 10:10 |
| *** milan has joined #tripleo | 10:10 | |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-specs: Composable Service Upgrades https://review.openstack.org/392116 | 10:12 |
| EmilienM | marios: thx for your work on ^ :) | 10:16 |
| openstackgerrit | Merged openstack/tripleo-ui: Pin jasmine-immutable-matchers version to ~1.0.1 https://review.openstack.org/392510 | 10:16 |
| *** lazy_prince has quit IRC | 10:17 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-specs: Single process Heat driven Undercloud installer https://review.openstack.org/352113 | 10:19 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Re-add NFS backend for Glance https://review.openstack.org/392128 | 10:19 |
| marios | EmilienM: thanks for running with it so quickly will try and keep it updated ... doesn't have to have all the details (still some things to clarify) but we can land it once it describes the direction we are taking 'enough' :) | 10:19 |
| marios | EmilienM: i wish i had more time to play with the prototyping at this point :( | 10:20 |
| *** mburned is now known as mburned_out | 10:20 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-specs: Remove duplicated Newton entry https://review.openstack.org/392525 | 10:20 |
| EmilienM | marios: no worries | 10:21 |
| *** nyechiel has joined #tripleo | 10:21 | |
| *** lazy_prince has joined #tripleo | 10:25 | |
| *** b00tcat has joined #tripleo | 10:31 | |
| b00tcat | good morning - someone please give me workflow on this one https://review.openstack.org/#/c/383057/ as I have more patches coming up ^^ | 10:31 |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui: Include nested parameters in service parameters https://review.openstack.org/392474 | 10:31 |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui: Pin jasmine-immutable-matchers version to ~1.0.1 https://review.openstack.org/392534 | 10:31 |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui: Make sample file use %(tenant_id)s macro https://review.openstack.org/392535 | 10:31 |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui: Fix deployment success output https://review.openstack.org/392536 | 10:31 |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui: Improve parameters loading https://review.openstack.org/392537 | 10:31 |
| EmilienM | b00tcat: we are currently having CI problems, under control. Nothing will merge until we sort things out | 10:32 |
| jtomasek | florianf: you'll probably need to rebase this one https://review.openstack.org/#/c/392189/ | 10:32 |
| *** numans has joined #tripleo | 10:32 | |
| florianf | jtomasek: yeah, that's what I'm currently doing :-) | 10:33 |
| openstackgerrit | Florian Fuchs proposed openstack/tripleo-ui: Use mistral action to create new containers https://review.openstack.org/392189 | 10:34 |
| florianf | jtomasek: should we backport this? It's not a bug atm, but rather a precaution | 10:35 |
| *** mgould|afk is now known as mgould | 10:36 | |
| florianf | jtomasek: although the more we backport right now, the less we have to worry about conflicts later... | 10:36 |
| jtomasek | florianf: If it is something that might cause bug in newton, then we should probably backport it | 10:37 |
| florianf | jtomasek: It won't unless there are changes to the way swift containers are created in tripleo-common | 10:38 |
| jtomasek | florianf: I doubt there is going to be such change in newton codebase. I think we're fine with not backporting it | 10:39 |
| florianf | jtomasek: fine for me | 10:39 |
| jtomasek | florianf: ok | 10:39 |
| openstackgerrit | Merged openstack/tripleo-ui: Parameters listing updates https://review.openstack.org/391855 | 10:41 |
| openstackgerrit | Ihar Hrachyshka proposed openstack/tripleo-heat-templates: Revert "Adjust MTU to compensate for VLAN tag issue" https://review.openstack.org/392542 | 10:43 |
| openstackgerrit | Tomas Sedovic proposed openstack/tripleo-common: Fix the validation ssh keys workflow https://review.openstack.org/391093 | 10:47 |
| *** [1]cdearborn has joined #tripleo | 10:49 | |
| openstackgerrit | Tomas Sedovic proposed openstack/tripleo-heat-templates: Add an optional extra node admin ssh key parameter https://review.openstack.org/390854 | 10:52 |
| *** mburned_out is now known as mburned | 10:52 | |
| panda | EmilienM: pretty sure the timeout is because of the redis bug. | 10:54 |
| openstackgerrit | Tomas Sedovic proposed openstack/tripleo-heat-templates: Add an optional extra node admin ssh key parameter https://review.openstack.org/390854 | 10:55 |
| EmilienM | panda: have you seen it in logs? | 10:56 |
| panda | EmilienM: yes, http://logs.openstack.org/88/392288/4/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/8cb3936/logs/overcloud-controller-0/var/log/audit/audit.txt.gz search for "denied" | 10:58 |
| EmilienM | panda: so it was because of Gnocchi failed to start? | 11:05 |
| *** akrivoka has joined #tripleo | 11:06 | |
| panda | EmilienM: no, let me recap redis fails -> gnocchi-metricd retries continuously -> gnocchi-metricd takes 50% of CPU -> everything slows down -> tenant-stack creation succeds 1 minute after the timeout -> failure | 11:07 |
| openstackgerrit | Merged openstack/tripleo-ui: Pin jasmine-immutable-matchers version to ~1.0.1 https://review.openstack.org/392534 | 11:07 |
| openstackgerrit | Merged openstack/tripleo-ui: Make sample file use %(tenant_id)s macro https://review.openstack.org/392535 | 11:07 |
| openstackgerrit | Merged openstack/tripleo-ui: Include nested parameters in service parameters https://review.openstack.org/392474 | 11:07 |
| openstackgerrit | Merged openstack/tripleo-ui: Fix deployment success output https://review.openstack.org/392536 | 11:07 |
| EmilienM | panda: have we found why redis fails? | 11:08 |
| panda | EmilienM: yes, https://github.com/ClusterLabs/resource-agents/pull/872 | 11:08 |
| *** hjensas has quit IRC | 11:08 | |
| *** ooolpbot has joined #tripleo | 11:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1638350 | 11:10 |
| *** ooolpbot has quit IRC | 11:10 | |
| openstack | Launchpad bug 1638350 in tripleo "pingtest failing on OVB jobs to create Cinder volume and Nova server" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 11:10 |
| jpich | jtomasek, florianf: FYI https://bugs.launchpad.net/tripleo/+bug/1638523 - I think not knowing how to properly set the parameters is likely to be a common issue, I saw it happen in Cardiff as well during the hackfest | 11:10 |
| openstack | Launchpad bug 1638523 in tripleo "Invalid JSON in parameters breaks deployments/parameters screen" [High,Triaged] | 11:10 |
| jpich | jtomasek, florianf: ...It's still not clear to me how to use stuff like ExtraConfig either. Would either of you happen to know of examples I could look into? | 11:11 |
| shardy | panda: nice work, thanks for the detailed analysis! :) | 11:11 |
| EmilienM | panda: nice :) | 11:11 |
| EmilienM | panda: have you checked how we can package this thing? | 11:12 |
| shardy | this isn't the first time we've hit issues due to gnocchi-metricd - it'd be good to get better validations in place so we can declare the overcloud failed instead of getting indirect failures like this | 11:12 |
| panda | EmilienM: asking on #rdo | 11:13 |
| openstackgerrit | Derek Higgins proposed openstack/diskimage-builder: dhcp-all-interfaces: support Centos/RHEL 6 https://review.openstack.org/392555 | 11:14 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Do not reference CephBase from CephExternal service https://review.openstack.org/392558 | 11:16 |
| jtomasek | jpich: yeah, if the parameter type is set correctly, we could provide a client side validation, although, I think the parameters should be validated server side too. tripleo-common needs to make sure the parameter value is saved in correctly and return validation error otherwise | 11:16 |
| jpich | jtomasek: That sounds good, tough the JSON part of the validation probably falls on us? | 11:17 |
| jtomasek | jpich: regarding ExtraConfig and similar, the parameter description should provide enough information so the user is able to know how to use it. If it does not, then it is tripleo-heat-templates bug | 11:18 |
| jpich | jtomasek: Ok | 11:18 |
| panda | EmilienM: maybe we should raise the timeout for now | 11:18 |
| panda | s/raise/increase | 11:19 |
| EmilienM | panda: right | 11:19 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-puppet-elements: Move nova.conf truncation into separate element https://review.openstack.org/392559 | 11:19 |
| EmilienM | panda: I added 1 min | 11:19 |
| jtomasek | jpich: we should be able to validate json client side, but tripleo-common should probably validate it through heat, and decline to save it if it fails | 11:19 |
| panda | EmilienM: I too like to live dangerously | 11:20 |
| jpich | jtomasek: Ok... I'll investigate a bit and open a second bug then. I assume this is all done via the overcloud parameter things? | 11:21 |
| *** jaosorior_lunch is now known as jaosorior | 11:21 | |
| *** zoliXXL is now known as zoli|lunch | 11:22 | |
| jtomasek | jpich: yes, I think | 11:23 |
| jpich | jtomasek: Great, thank you for all the information! | 11:23 |
| openstackgerrit | Merged openstack/tripleo-quickstart: Fix documentation for accessing overcloud nodes https://review.openstack.org/392220 | 11:24 |
| *** bfournie has quit IRC | 11:26 | |
| *** bfournie has joined #tripleo | 11:26 | |
| *** jkilpatr has joined #tripleo | 11:27 | |
| florianf | jpich, jtomasek: If we validate json clientside, how do we know which params are supposed to contain json values? | 11:31 |
| jtomasek | florianf: by parameter Type | 11:31 |
| florianf | jtomasek: ok, that makes sense, thanks | 11:32 |
| jtomasek | florianf: http://docs.openstack.org/developer/heat/template_guide/hot_spec.html#parameters-section | 11:32 |
| florianf | jtomasek: great, thx | 11:32 |
| jaosorior | sshnaidm: do you know if somewhere in tripleo-ci we open some ports for services? | 11:33 |
| jaosorior | sshnaidm: I'm having this issue http://logs.openstack.org/38/391738/16/experimental-tripleo/gate-tripleo-ci-centos-7-ovb-fakeha-caserver/d9a0d06/console.html#_2016-11-02_10_36_31_606589 | 11:33 |
| *** hjensas has joined #tripleo | 11:35 | |
| thrash | EmilienM: we good to merge this? https://review.openstack.org/#/c/390590/ | 11:36 |
| sshnaidm | jaosorior, I'm not sure they are closed anywhere | 11:36 |
| EmilienM | thrash: yes! I'll let you approve it | 11:37 |
| sshnaidm | jaosorior, we can look at netstat in postci.txt and iptables there | 11:37 |
| jaosorior | sshnaidm: lets do that | 11:37 |
| sshnaidm | jaosorior, but I doubt that it's problem with ports | 11:37 |
| jaosorior | sshnaidm: or it might be that I have to open them in the node I just brought up | 11:37 |
| EmilienM | thrash: thx | 11:37 |
| sshnaidm | jaosorior, it may be also service not running, isn't it? | 11:38 |
| jaosorior | sshnaidm: that might be the case, though it would be pretty strange, since that would mean that the heat-template would have failed. | 11:39 |
| d0ugal | Does anyone have a tripleo-overcloud-passwords file handy I can see? | 11:39 |
| sshnaidm | jaosorior, did it work locally? | 11:40 |
| jaosorior | sshnaidm: yes | 11:40 |
| * d0ugal realises that sounds like a not very subtle fishing attempt | 11:40 | |
| jaosorior | sshnaidm: though it's a little bit modified than what the intern originally tested... gotta test it out again | 11:40 |
| shadower | d0ugal: give me a sec | 11:40 |
| jpich | d0ugal: I have one from a newton deployment, does that work? | 11:40 |
| jaosorior | shardy: is there anything extra that I have to do in this template to get exit codes to count as failures for the stack https://review.openstack.org/#/c/391738/16/heat-templates/freeipa_server.yaml ? | 11:41 |
| d0ugal | jpich: sure | 11:41 |
| d0ugal | shadower: jpich might beat you :) | 11:41 |
| *** rhallisey has joined #tripleo | 11:41 | |
| shadower | d0ugal: http://paste.openstack.org/show/587628/ | 11:41 |
| shadower | :-) | 11:41 |
| jpich | d0ugal: http://paste.openstack.org/show/587629/ | 11:41 |
| d0ugal | jpich: I don't understand why you have one in a newton deploy... | 11:41 |
| jpich | d0ugal: I don't beat people | 11:41 |
| d0ugal | Maybe I am missing something. | 11:41 |
| shadower | jpich: phew! | 11:41 |
| d0ugal | jpich, shadower - thanks! | 11:42 |
| EmilienM | dtantsur: do you need a new release of puppet-ironic? | 11:42 |
| EmilienM | dtantsur: nevermind, i'll do it anyway. | 11:42 |
| jpich | d0ugal: I guess the CLI creates it anyway, just with a different source for the passwords? This is a downstream env | 11:43 |
| sshnaidm | jaosorior, are these ports outgoing..? | 11:43 |
| jpich | d0ugal: Actually I have no idea so don't mind me. All I can say is this is Newton :-) | 11:43 |
| d0ugal | jpich: I can't find the code that creates it now. | 11:44 |
| dtantsur | ok :) | 11:44 |
| shardy | jaosorior: No, heat just looks at the return code from bash | 11:44 |
| shardy | jaosorior: perhaps you need pipefail? | 11:45 |
| d0ugal | jpich: I thought it was removed in https://github.com/openstack/python-tripleoclient/commit/1969ccff64f89b04aa4d653a14fccfaa8b31f345 | 11:45 |
| jaosorior | shardy: how can I just print the output of that softwareconfig as the output of the stack? | 11:46 |
| jaosorior | sshnaidm: yes... I...think O_o... so the service in that node should be listening on those ports | 11:47 |
| d0ugal | jpich, shadower - I assume you both have a fairly old env? | 11:47 |
| openstackgerrit | Merged openstack/puppet-tripleo: NFS mounting for Glance file backend https://review.openstack.org/392131 | 11:47 |
| jtomasek | florianf: https://review.openstack.org/#/c/392537/ is the last backport pending review | 11:47 |
| jpich | d0ugal: The backport merged on Oct 25, probably wasn't in the package I used | 11:47 |
| d0ugal | jpich: Yeah, that is probably the case | 11:47 |
| shadower | d0ugal: 2-3 weeks I think | 11:47 |
| shardy | jaosorior: sorry I don't follow - you want the freeipa_server.yaml stack to fail if the SoftwareConfig fails? | 11:47 |
| sshnaidm | jaosorior, then problems will be on ipa node if at all, because we don't block outgoing from undercloud | 11:47 |
| d0ugal | jpich: okay, that's good - it means I maybe do understand what is going on still | 11:47 |
| d0ugal | shadower: cool, that would make sense. | 11:47 |
| jaosorior | shardy: yes | 11:48 |
| shardy | jaosorior: there's no error path from cloud-init, so you'll need to use a SoftwareDeployment I think | 11:48 |
| jaosorior | shardy: ok | 11:48 |
| jpich | d0ugal: That file was created on Oct 28 so I think it matches for my env too. Onwards!! | 11:48 |
| sshnaidm | jaosorior, but problem is that we don't collect any stats from ipa node | 11:49 |
| sshnaidm | jaosorior, I'd recommend to use den env to debug it | 11:49 |
| sshnaidm | jaosorior, s/den/dev/ | 11:49 |
| *** maeca1 has joined #tripleo | 11:50 | |
| *** pkovar has quit IRC | 11:51 | |
| sshnaidm | jaosorior, share you public key, I'll try to set it for you | 11:52 |
| *** zoliXXL has joined #tripleo | 11:53 | |
| sshnaidm | jaosorior, but it'd be anyway good idea to include forcing opening these ports in heat template | 11:53 |
| jaosorior | sshnaidm: sent | 11:53 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: WIP TLS everywhere job https://review.openstack.org/391738 | 11:57 |
| *** jcoufal has joined #tripleo | 11:59 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: WIP TLS everywhere job https://review.openstack.org/391738 | 12:00 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-specs: Composable HA services https://review.openstack.org/378035 | 12:01 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: WIP TLS everywhere job https://review.openstack.org/391738 | 12:01 |
| openstackgerrit | Lucas Alvares Gomes proposed openstack/tripleo-quickstart: Force the use of python version 2.7 https://review.openstack.org/368013 | 12:01 |
| EmilienM | can someone quickly review this specs bug? https://review.openstack.org/#/c/392525/ | 12:02 |
| openstackgerrit | lilintan proposed openstack/tripleo-quickstart: Drop *openstack/common* in flake8 exclude list https://review.openstack.org/373874 | 12:03 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: WIP TLS everywhere job https://review.openstack.org/391738 | 12:03 |
| jaosorior | EmilienM: should I merge it? | 12:03 |
| EmilienM | jaosorior: yeah go ahead | 12:03 |
| *** egafford has joined #tripleo | 12:03 | |
| d0ugal | I assume when Ocata is released we would require Mitaka users to upgrade to Newton first - is that correct? | 12:04 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: WIP TLS everywhere job https://review.openstack.org/391738 | 12:04 |
| d0ugal | i.e. we wont ever support Mitaka to Ocata directly. | 12:04 |
| d0ugal | marios: ^ | 12:05 |
| EmilienM | shadower, jaosorior: thx guys | 12:05 |
| shadower | no worries | 12:05 |
| EmilienM | d0ugal: right | 12:05 |
| d0ugal | EmilienM: thanks | 12:05 |
| EmilienM | upgrades work from n-1 to n and I'm not aware about another way. | 12:05 |
| *** lucasagomes is now known as lucas-hungry | 12:05 | |
| marios | d0ugal: on a call atm will have a look later | 12:05 |
| d0ugal | cool, that I what I figured but wanted to check. | 12:06 |
| d0ugal | marios: no worries, EmilienM answered :) | 12:06 |
| marios | d0ugal: but yes, we go one at a time | 12:06 |
| *** jayg|g0n3 is now known as jayg | 12:06 | |
| *** morazi has joined #tripleo | 12:06 | |
| openstackgerrit | Merged openstack/tripleo-specs: Remove duplicated Newton entry https://review.openstack.org/392525 | 12:07 |
| *** ooolpbot has joined #tripleo | 12:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1638350 | 12:10 |
| openstack | Launchpad bug 1638350 in tripleo "pingtest failing on OVB jobs to create Cinder volume and Nova server" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 12:10 |
| *** ooolpbot has quit IRC | 12:10 | |
| openstackgerrit | Honza Pokorny proposed openstack/tripleo-ui: Redirect user to login page when token expires https://review.openstack.org/392589 | 12:10 |
| *** adarazs is now known as adarazs_lunch | 12:10 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: WIP TLS everywhere job https://review.openstack.org/391738 | 12:10 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-specs: Composable HA services https://review.openstack.org/378035 | 12:11 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-specs: oooq as the tool set for tripleo CI https://review.openstack.org/386250 | 12:11 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-specs: Unified log and status capture https://review.openstack.org/391951 | 12:12 |
| *** morazi has quit IRC | 12:12 | |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Fix password handling for users upgrading from Mitaka https://review.openstack.org/392593 | 12:12 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Fix password handling for users upgrading from Mitaka https://review.openstack.org/392593 | 12:13 |
| d0ugal | jpich: https://review.openstack.org/#/c/392593 - if you get a moment to see if that makes sense. | 12:17 |
| jpich | d0ugal: Will look! | 12:18 |
| openstackgerrit | Ana Krivokapic proposed openstack/tripleo-docs: Replace mention of 'liberty' in directory path with '*' https://review.openstack.org/392597 | 12:18 |
| *** maticue has joined #tripleo | 12:20 | |
| openstackgerrit | Honza Pokorny proposed openstack/tripleo-ui: Remove unused TripleO API service https://review.openstack.org/392602 | 12:24 |
| *** rbrady has joined #tripleo | 12:24 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: WIP TLS everywhere job https://review.openstack.org/391738 | 12:24 |
| *** pkovar has joined #tripleo | 12:24 | |
| *** morazi has joined #tripleo | 12:25 | |
| *** cdearborn has joined #tripleo | 12:28 | |
| openstackgerrit | Merged openstack/python-tripleoclient: Use workflow for overcloud node delete https://review.openstack.org/390590 | 12:30 |
| *** fultonj has joined #tripleo | 12:31 | |
| *** liverpooler has quit IRC | 12:31 | |
| *** saneax is now known as saneax-_-|AFK | 12:32 | |
| *** ohamada has quit IRC | 12:32 | |
| *** liverpooler has joined #tripleo | 12:33 | |
| *** ohamada has joined #tripleo | 12:33 | |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Fix password handling for users upgrading from Mitaka https://review.openstack.org/392593 | 12:35 |
| jtomasek | honza: it should point to the sourcemap, are you trying to debug the same problem as jenkins fails on in your patch? | 12:37 |
| honza | jtomasek: yes | 12:37 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Fix password handling for users upgrading from Mitaka https://review.openstack.org/392593 | 12:39 |
| *** zoli|lunch is now known as zoli|wfh | 12:40 | |
| honza | jtomasek: the location is this src/webpack.tests.js:43857 --- i don't think there is a sourcemap for that file? | 12:41 |
| * jtomasek applies that patch | 12:41 | |
| openstackgerrit | Dan Prince proposed openstack/tripleo-puppet-elements: Disable old hiera.yaml o-a-c script https://review.openstack.org/392615 | 12:41 |
| *** jerrygb has quit IRC | 12:42 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: WIP: Full quickstart gate run on OVB https://review.openstack.org/381094 | 12:42 |
| *** yolanda has quit IRC | 12:44 | |
| *** yolanda has joined #tripleo | 12:44 | |
| *** [1]cdearborn has quit IRC | 12:44 | |
| EmilienM | mwhahaha: sounds like with your patch we still hit a 504 issue with cinder-api http://logs.openstack.org/88/392288/5/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/d219830/console.html#_2016-11-02_12_16_57_080321 | 12:45 |
| *** sshnaidm is now known as sshnaidm|afk | 12:45 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-specs: oooq as the tool set for tripleo CI https://review.openstack.org/386250 | 12:45 |
| *** jerrygb has joined #tripleo | 12:45 | |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Hiera optimization: use a new hiera hook https://review.openstack.org/384757 | 12:45 |
| EmilienM | bandini, panda: what is ETA to have RA rpm updated? | 12:45 |
| panda | EmilienM: I don't know, I'm testing a build preview from internal brew now. | 12:46 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-specs: update the index for ocata https://review.openstack.org/388208 | 12:47 |
| EmilienM | panda: so it sounds like even if we increase the timeout and fix heat, we have a 504 in cinder | 12:48 |
| *** lblanchard has joined #tripleo | 12:48 | |
| EmilienM | slagle: ^ fyi about our problem from yesterday | 12:48 |
| EmilienM | panda: ouch, sounds like cinder-api is not started on some controllers | 12:49 |
| EmilienM | see in http://logs.openstack.org/88/392288/5/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/d219830/logs/overcloud-controller-1/var/log/cinder/ | 12:49 |
| EmilienM | there is no cinder-api.log | 12:50 |
| *** jeckersb is now known as jeckersb_gone | 12:50 | |
| EmilienM | same for controller002 | 12:50 |
| panda | I don't know if theyt depend on redis even indirectly | 12:50 |
| EmilienM | they don't | 12:51 |
| pradk | marios, so yurii +1'ed the upgrade patch confirming the patch helps .. there is still an issue with gnochci-upgrade when there are no metrics, but thats a gnocchi issue not tripleo | 12:52 |
| bandini | EmilienM: I gave panda a test build to speed things up a bit. I will ping the folks that release this stuff to understand how far away we are from a 7.2.z and 7.3 release (it is usually not the fastest of things, so we're better off discussing either injecting a fixed rpm in the chain or setting selinux in permissive for the time being) | 12:53 |
| panda | EmilienM: ok, I seen galera and cinder failing in my live environment, looking at that too, hope this is not too much compromised and give realistic situation | 12:54 |
| EmilienM | jaosorior: when you enabled wsgi for cinder-api was it stable? | 12:54 |
| marios | pradk: ack thanks for update (still on the same call you're on :)) hopefully it can land today/tomorrow and we get into stable | 12:55 |
| EmilienM | panda: cinder-api should start with httpd in wsgi | 12:55 |
| *** adarazs_lunch is now known as adarazs | 12:55 | |
| marios | pradk: can you make it a little more explicit there what yuri's +1 means :) (I ran this, it works!) | 12:56 |
| marios | pradk: to entice other reviewers :) | 12:56 |
| openstackgerrit | Lukas Bezdicka proposed openstack/tripleo-heat-templates: Ensure we update ceph and composable nodes https://review.openstack.org/392260 | 12:56 |
| pradk | marios, i requested yurii to add a comment | 12:56 |
| marios | pradk: thanks | 12:56 |
| *** tzumainn has joined #tripleo | 12:56 | |
| dtantsur | folks, at which stage are workbooks loaded in mistral? is just replacing a file in /usr/share/tripleo-common enough? | 12:57 |
| dtantsur | d0ugal, maybe you know ^^^? | 12:57 |
| jaosorior | EmilienM: I had tried it locally, and at least in CI it seemed quite stable. I'm actually checking just that at the moment, seems that all three controllers have the httpd conf for cinder there, so investigating still | 12:57 |
| panda | EmilienM: this is what I see in pcs status for cinder | 12:57 |
| panda | * openstack-cinder-volume_monitor_60000 on overcloud-controller-0 'not running' (7): call=69, status=complete, exitreason='none', | 12:57 |
| panda | last-rc-change='Wed Nov 2 00:17:35 2016', queued=0ms, exec=0ms | 12:57 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-specs: Instance HA Specification https://review.openstack.org/389103 | 12:57 |
| dtantsur | hmm, I guess I should use mistral workbook-update | 12:58 |
| jaosorior | :qa | 12:58 |
| panda | :eng | 12:59 |
| *** abregman has quit IRC | 12:59 | |
| *** ccamacho is now known as ccamacho|lunch | 12:59 | |
| EmilienM | panda: your problem looks different | 13:00 |
| EmilienM | panda: in CI, the problem is that cinder api doesn't start on controller1 & 2 (but starts well on controller0) | 13:00 |
| EmilienM | which is why pingtest fails | 13:00 |
| EmilienM | HAproxy fails to reach cinder api (which is weird because HAproxy should know cinder-api is down on the 2 nodes) | 13:01 |
| jaosorior | EmilienM: in puppet, in a class declaration, what's taken into account? a value coming from hiera or a parameter explicitly given in the class? | 13:02 |
| EmilienM | jaosorior: priority is the param given to a class, and then, hiera | 13:03 |
| *** snecklifter has quit IRC | 13:03 | |
| jaosorior | EmilienM: ok, then I'll try to revert the cinder change that way | 13:03 |
| EmilienM | wait | 13:03 |
| EmilienM | I see cinder running with netstat: | 13:03 |
| EmilienM | tcp 0 0 172.17.0.13:8776 0.0.0.0:* LISTEN 2340/httpd | 13:03 |
| *** snecklifter has joined #tripleo | 13:03 | |
| *** zoliXXL has quit IRC | 13:04 | |
| jaosorior | wha... | 13:04 |
| panda | EmilienM: ok, I'll respin the live environment as soon as I analyze these other failures, and see what happens to cinder-api. I don't even understand how all periodic jobs yesterday were green. | 13:04 |
| *** zoliXXL has joined #tripleo | 13:05 | |
| florianf | jtomasek: review done... | 13:05 |
| jaosorior | EmilienM: actually, in controller-1 haproxy detects cinder API to be up at some point: Nov 02 13:54:55 overcloud-controller-1 haproxy[30543]: Server cinder/overcloud-controller-1.internalapi.localdomain is UP, reason: Layer4 check passed, check duration: 0ms. 3 active and 0 backup servers online. | 13:06 |
| *** abregman has joined #tripleo | 13:07 | |
| *** Goneri has joined #tripleo | 13:07 | |
| EmilienM | jaosorior: right but something during pingtest see cinder as down | 13:07 |
| EmilienM | jaosorior: maybe in heat logs we can see it, let me look | 13:07 |
| *** kjw3 has joined #tripleo | 13:08 | |
| EmilienM | oh look that http://logs.openstack.org/88/392288/5/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/d219830/logs/overcloud-controller-2/var/log/heat/heat-engine.txt.gz#_2016-11-02_12_15_13_510 | 13:08 |
| jaosorior | EmilienM: yep, cinder is not responding for some reason | 13:09 |
| EmilienM | so maybe cinder API didn't reply on time | 13:09 |
| EmilienM | jaosorior: but why don't we have cinder-api logs on other controller? | 13:09 |
| jaosorior | EmilienM: that makes no sense to me | 13:09 |
| EmilienM | I looked at old ovb-ha jobs and they had cinder-api.log | 13:10 |
| *** ooolpbot has joined #tripleo | 13:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1638350 | 13:10 |
| openstack | Launchpad bug 1638350 in tripleo "pingtest failing on OVB jobs to create Cinder volume and Nova server" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 13:10 |
| *** ooolpbot has quit IRC | 13:10 | |
| *** sudipto has joined #tripleo | 13:10 | |
| *** sudipto_ has joined #tripleo | 13:10 | |
| *** cylopez has joined #tripleo | 13:11 | |
| EmilienM | I'm doing a recheck on https://review.openstack.org/#/c/392288/ to see if our problem is consistent | 13:11 |
| jaosorior | EmilienM: seems to have happened here too https://review.openstack.org/#/c/392519/ | 13:12 |
| openstackgerrit | Dmitry Tantsur proposed openstack/tripleo-common: Do not try "manage" actions on nodes that are not in "enroll" state https://review.openstack.org/392179 | 13:12 |
| EmilienM | it might be a random performance issue caused by Gnocchi + Redis problem that panda fixed | 13:13 |
| EmilienM | that's why we need to have this fix in RPM asap, bandini | 13:13 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Run cinder over eventlet again https://review.openstack.org/392647 | 13:14 |
| jaosorior | EmilienM: gonna try that out ^^ | 13:14 |
| EmilienM | jaosorior: right, look http://logs.openstack.org/19/392519/1/check-tripleo/gate-tripleo-ci-centos-7-ovb-nonha/728a389/console.html#_2016-11-02_12_10_20_900983 | 13:15 |
| EmilienM | jaosorior: even on nonha | 13:15 |
| jaosorior | what the hell | 13:15 |
| EmilienM | jaosorior: please Depends-On https://review.openstack.org/#/c/392288/ | 13:15 |
| EmilienM | we need mwhahaha's change in any case. | 13:16 |
| EmilienM | jaosorior: well no, use git rebase | 13:16 |
| *** [1]cdearborn has joined #tripleo | 13:16 | |
| tosky | ccamacho|lunch: why invalid for https://bugs.launchpad.net/tripleo/+bug/1560098 ? | 13:16 |
| openstack | Launchpad bug 1560098 in tripleo "The list of enabled Sahara plugins should be configurable" [Low,Invalid] - Assigned to Carlos Camacho (ccamacho) | 13:16 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Run cinder over eventlet again https://review.openstack.org/392647 | 13:16 |
| jaosorior | EmilienM: sure | 13:16 |
| *** florianf has quit IRC | 13:16 | |
| jtomasek | honza: the problem must be in a way how dispatch is passed to that constructor. when I remove that, it works | 13:17 |
| jtomasek | honza: I need to go afk for a while, I'll investigate further | 13:17 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Run cinder over eventlet again https://review.openstack.org/392647 | 13:17 |
| jaosorior | EmilienM: done ^^ | 13:18 |
| *** coolsvap has quit IRC | 13:18 | |
| EmilienM | jaosorior: excellent. Let's see | 13:18 |
| EmilienM | in the meantime, we might need to find a workaround in tripleo-ci | 13:18 |
| jaosorior | EmilienM: though I really hope that it gets fixed with the fix from panda and bandini | 13:18 |
| EmilienM | panda: how can we get your change in tripleo-ci until it's packaged? | 13:18 |
| *** sshnaidm|afk is now known as sshnaidm | 13:19 | |
| panda | EmilienM: *IF* we can access the package from CI env, we could inject it to overcloud image with a hacky virt-customize, before it gets downloaded. | 13:20 |
| openstackgerrit | Dmitry Tantsur proposed openstack/python-tripleoclient: Do not try to "provide" updated nodes not in "manageable" state https://review.openstack.org/392653 | 13:20 |
| panda | EmilienM: but right now is in internal brew only. | 13:20 |
| panda | s/downloaded/uploaded | 13:20 |
| *** lucas-hungry is now known as lucasagomes | 13:22 | |
| panda | bandini: any chance to get this package somewhere out as a preview ? | 13:22 |
| panda | anyway, I tested the package, it works, redis is up | 13:23 |
| bandini | panda: let me find out | 13:23 |
| jaosorior | sshnaidm: you were right, the deployment had actually failed, it was now that I changed it to run over a software deployment instead of cloud-init that it showed up | 13:26 |
| sshnaidm | jaosorior, ok, great | 13:26 |
| bandini | panda: just checked with Oyvind. we can use that build for CI until we get it released proper | 13:26 |
| jaosorior | sshnaidm: so the deal is that I try to install some packages and that fails | 13:26 |
| *** jkilpatr has quit IRC | 13:27 | |
| jaosorior | sshnaidm: http://logs.openstack.org/38/391738/23/experimental-tripleo/gate-tripleo-ci-centos-7-ovb-fakeha-caserver/e388c8c/console.html#_2016-11-02_13_17_00_952400 | 13:27 |
| sshnaidm | jaosorior, do you know why? | 13:27 |
| panda | bandini: we need to find a place where to put it ... maybe hardcode in tripleo-ci at this point .. | 13:27 |
| *** florianf has joined #tripleo | 13:28 | |
| bandini | panda: yeah maybe we can stick it up somewhere on an rdo folder so it has decent infrastrcture? (as opposed to say people.rh.com or people.fedoraproject.org) | 13:28 |
| *** jeckersb_gone is now known as jeckersb | 13:28 | |
| sshnaidm | jaosorior, Could not resolve host: mirror.regionone.tripleo-test-cloud-rh1.openstack.org | 13:29 |
| sshnaidm | jaosorior, what are dns there? | 13:29 |
| jaosorior | sshnaidm: just using the defaults | 13:29 |
| jaosorior | sshnaidm: should be the overcloud-full image | 13:29 |
| *** absubram has joined #tripleo | 13:30 | |
| sshnaidm | jaosorior, can you check on dev env? | 13:30 |
| *** cdearborn has quit IRC | 13:30 | |
| *** hjensas has quit IRC | 13:31 | |
| jaosorior | sshnaidm: can't log into that node :/ | 13:31 |
| sshnaidm | jaosorior, can you include "default" ssh keys into template? | 13:32 |
| jaosorior | will do | 13:32 |
| sshnaidm | jaosorior, try ssh centos@192.0.2.15 | 13:33 |
| jaosorior | sshnaidm: interesting, it already has the default one | 13:33 |
| sshnaidm | jaosorior, it should be "centos" user | 13:33 |
| jaosorior | sshnaidm: only nameserver is nameserver 192.0.2.5 | 13:34 |
| sshnaidm | jaosorior, yep | 13:34 |
| sshnaidm | jaosorior, so it won't resolve anything | 13:34 |
| honza | jtomasek: thanks for the pointer, i'll have a closer look | 13:35 |
| *** jkilpatr has joined #tripleo | 13:38 | |
| jaosorior | sshnaidm: is that 10.1.8.10 constant? or where can I fetch that? Or is the only way to grep that resolv.conf? | 13:39 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: WIP TLS everywhere job https://review.openstack.org/391738 | 13:40 |
| sshnaidm | jaosorior, why not 8.8.8.8 ? | 13:40 |
| jaosorior | sshnaidm: added those there ^^ | 13:40 |
| *** jpena is now known as jpena|lunch | 13:40 | |
| *** rodrigods has quit IRC | 13:43 | |
| *** rodrigods has joined #tripleo | 13:43 | |
| *** jkilpatr has quit IRC | 13:45 | |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-heat-templates: Add Sahara plugins list as a configurable parameter https://review.openstack.org/392670 | 13:47 |
| *** ccamacho|lunch is now known as ccamacho | 13:48 | |
| ccamacho | tosky you are right man | 13:48 |
| ccamacho | I thought it was fixed long time ago | 13:48 |
| jaosorior | shardy: isn't the password passed already via t-h-t? | 13:48 |
| ccamacho | there is the fix | 13:49 |
| tosky | ccamacho: I took the liberty of moving to New again (maybe Confirmed then?) | 13:49 |
| ccamacho | tosky sure man np | 13:49 |
| tosky | ooooh, the patch! | 13:49 |
| * tosky cries for joy | 13:50 | |
| ccamacho | tosky for for not pushing this before man, really sorry | 13:50 |
| ccamacho | **sorry | 13:50 |
| shardy | jaosorior: Yes, but what if someone sometime accidentally removes the tht hieradata key? | 13:50 |
| openstackgerrit | Alex Schultz proposed openstack/puppet-tripleo: Create heat user in keystone profile https://review.openstack.org/392288 | 13:50 |
| shardy | we've been bitten by exactly that sort of problem before, which is why I removed all the _unset_ defaults | 13:51 |
| shardy | passwords should not ever be defaulted IMHO | 13:51 |
| tosky | ccamacho: it was probably late for newton anyway, maybe, but now I see the light for Ocata | 13:51 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Run cinder over eventlet again https://review.openstack.org/392647 | 13:51 |
| EmilienM | jaosorior: thx | 13:52 |
| *** jaosorior has quit IRC | 13:53 | |
| *** jkilpatr has joined #tripleo | 13:57 | |
| *** yamahata has joined #tripleo | 13:59 | |
| *** abregman_ has joined #tripleo | 14:06 | |
| *** abregman_ has quit IRC | 14:08 | |
| *** abregman_ has joined #tripleo | 14:08 | |
| *** abregman has quit IRC | 14:09 | |
| *** abregman_ has quit IRC | 14:09 | |
| *** ooolpbot has joined #tripleo | 14:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1638350 | 14:10 |
| *** ooolpbot has quit IRC | 14:10 | |
| openstack | Launchpad bug 1638350 in tripleo "pingtest failing on OVB jobs to create Cinder volume and Nova server" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 14:10 |
| *** abregman has joined #tripleo | 14:10 | |
| *** jeckersb is now known as jeckersb_gone | 14:15 | |
| *** michapma_alt has joined #tripleo | 14:19 | |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates: Fixup the start of swift services https://review.openstack.org/392680 | 14:25 |
| d0ugal | dtantsur: Sorry I didn't reply before - did you manage to update the mistral workbooks? | 14:25 |
| dtantsur | d0ugal, yep, thanks | 14:26 |
| *** ealcaniz has quit IRC | 14:26 | |
| openstackgerrit | Honza Pokorny proposed openstack/tripleo-ui: Redirect user to login page when token expires https://review.openstack.org/392589 | 14:27 |
| *** dhill__ has quit IRC | 14:28 | |
| *** dhill_ has joined #tripleo | 14:28 | |
| weshay | ping Slower_ re: containerized compute steps | 14:29 |
| *** akshai has joined #tripleo | 14:34 | |
| *** rlandy has joined #tripleo | 14:35 | |
| *** jeckersb_gone is now known as jeckersb | 14:37 | |
| *** jaosorior has joined #tripleo | 14:41 | |
| honza | jtomasek: are you saying that "dispatch" is somehow "null" when passed in? | 14:42 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: WIP TLS everywhere job https://review.openstack.org/391738 | 14:43 |
| jaosorior | sshnaidm: seems I'm still having mirror errors, even after adding 8.8.8.8 ^^ | 14:44 |
| sshnaidm | jaosorior, do you have a link? | 14:45 |
| *** jprovazn has quit IRC | 14:45 | |
| jaosorior | sshnaidm: http://logs.openstack.org/38/391738/23/experimental-tripleo/gate-tripleo-ci-centos-7-ovb-fakeha-caserver/e388c8c/console.html | 14:45 |
| *** thrash is now known as thrash|dr | 14:45 | |
| sshnaidm | jaosorior, I'd add "cat /etc/resolv.conf" to see what is really configured there | 14:47 |
| sshnaidm | jaosorior, you can try on dev env again, btw | 14:47 |
| jaosorior | sshnaidm: can you pass again how to log in there? | 14:47 |
| panda | EmilienM: what should my change depend on ? | 14:48 |
| sshnaidm | jaosorior, but you may want to restart it from scratch | 14:48 |
| sshnaidm | jaosorior, https://etherpad.openstack.org/p/tripleo-ci-devenvs | 14:49 |
| jaosorior | sshnaidm: well, I could just fetch the latest changes and run that | 14:49 |
| sshnaidm | jaosorior, just run this script: https://raw.githubusercontent.com/sshnaidm/various/master/devenv/ovb_rh1.sh | 14:49 |
| jtomasek | honza: googling that issue gives hints that it has something to do with how class works in es6, and for some reason passing dispatch breaks it | 14:49 |
| akrivoka | bnemec: regarding the removal of keystone endpoint creation from the tripleo client... I was going off of the conversations at the ui session at the summit, as well as the upstream meeting a couple of weeks ago - both suggested this should be removed. the code in question is both dead and broken. if it really can't be removed, should we fix it then? if people try to execute this code path it will break. | 14:49 |
| bnemec | akrivoka: It's not dead, it still gets run if you're using older templates. And how is it broken? | 14:51 |
| jaosorior | sshnaidm: uh... not sure where to do so, tried doing it in the environment you passed me and it fails | 14:51 |
| *** jpena|lunch is now known as jpena | 14:51 | |
| openstackgerrit | Pierre Blanc proposed openstack/os-net-config: Add support for ethtool_opts option https://review.openstack.org/392697 | 14:52 |
| panda | jaosorior: you know what the depends-on should be add a workaround to the chain for ovb-ha ? | 14:52 |
| sshnaidm | jaosorior, no, you need to run it on your machine | 14:52 |
| sshnaidm | jaosorior, and you need credentials, see priv | 14:52 |
| jaosorior | panda: I don't, would need more context on what you're asking | 14:52 |
| akrivoka | bnemec: e.g. https://github.com/openstack/python-tripleoclient/blob/master/tripleoclient/v1/overcloud_deploy.py#L514 and https://github.com/openstack/python-tripleoclient/blob/master/tripleoclient/utils.py#L440 | 14:53 |
| jaosorior | sshnaidm: I didn't get any priv message about credentials :/ | 14:53 |
| *** thrash|dr has quit IRC | 14:53 | |
| *** pabelanger has quit IRC | 14:54 | |
| panda | jaosorior: to unblock ovb-ha gate we need at least a fix for heat, and then I wanted to add my workaround to the redis bug, I want to depend-on the heat-fix but I don't know what's its ID | 14:54 |
| *** tzumainn has quit IRC | 14:54 | |
| *** dmanchad has quit IRC | 14:54 | |
| *** tzumainn has joined #tripleo | 14:54 | |
| *** morazi has quit IRC | 14:55 | |
| jaosorior | panda: bnemec might know | 14:55 |
| *** sai has quit IRC | 14:55 | |
| *** ealcaniz has joined #tripleo | 14:55 | |
| *** chandankumar has quit IRC | 14:55 | |
| panda | or maybe mwhahaha | 14:55 |
| bnemec | akrivoka: That's only broken if you're using a new enough version of the undercloud to not need to run post-config anyway. | 14:55 |
| *** dmanchad has joined #tripleo | 14:55 | |
| honza | jtomasek: it looks like the problem is with importing the actions? | 14:56 |
| *** abregman_ has joined #tripleo | 14:56 | |
| bnemec | The point of leaving the occ post-config is compatibility with older versions. | 14:56 |
| *** chandankumar has joined #tripleo | 14:57 | |
| honza | jtomasek: i'll google around more after lunch, thanks for looking | 14:58 |
| *** sai has joined #tripleo | 14:58 | |
| jtomasek | honza: so, I have a solution... | 14:58 |
| honza | jtomasek: i'm all ears | 14:58 |
| *** abregman has quit IRC | 14:58 | |
| jtomasek | honza: in the BaseHttpRequestErrorHandler you can import store (as we do in index.js) | 14:59 |
| *** jcoufal has quit IRC | 14:59 | |
| jtomasek | honza: and then just run store.dispatch(...) in logout() | 14:59 |
| *** thrash has joined #tripleo | 14:59 | |
| *** thrash has quit IRC | 14:59 | |
| *** thrash has joined #tripleo | 14:59 | |
| honza | jtomasek: man | 14:59 |
| *** morazi has joined #tripleo | 14:59 | |
| jtomasek | honza: with that you don't even need to pass dispatch in all the actions | 14:59 |
| *** pabelanger has joined #tripleo | 14:59 | |
| *** jcoufal has joined #tripleo | 15:00 | |
| honza | jtomasek: awesomeness | 15:00 |
| *** tosky has quit IRC | 15:00 | |
| jtomasek | honza: hah, it is not entirely recommended (as described in this reply https://github.com/reactjs/redux/issues/916#issuecomment-149190441) | 15:01 |
| jtomasek | honza: but we already do it and until we don't use our code server side, it is fine | 15:02 |
| jtomasek | honza: we'll get rid of that by moving things into middlewares but that needs further investigation, so this solution is imho quite nice | 15:03 |
| *** fultonj_ has joined #tripleo | 15:03 | |
| mwhahaha | panda: https://review.openstack.org/#/c/392288/ heat fix | 15:04 |
| panda | mwhahaha: thanks | 15:05 |
| jtomasek | akrivoka: good catch | 15:05 |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci: Workaround: Set selinux to permissive to workaround bug LP#1637961 https://review.openstack.org/392703 | 15:05 |
| *** rhallisey has quit IRC | 15:07 | |
| *** rhallisey has joined #tripleo | 15:08 | |
| openstackgerrit | ayoung proposed openstack/tripleo-heat-templates: Unset Keystone public_endpoint https://review.openstack.org/386080 | 15:08 |
| openstackgerrit | Merged openstack/tripleo-ui: Improve parameters loading https://review.openstack.org/392537 | 15:09 |
| *** ooolpbot has joined #tripleo | 15:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1638350 | 15:10 |
| openstack | Launchpad bug 1638350 in tripleo "pingtest failing on OVB jobs to create Cinder volume and Nova server" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 15:10 |
| *** ooolpbot has quit IRC | 15:10 | |
| shardy | mwhahaha: thanks for the heat fix, I tested locally and can confirm the domain stuff look good now | 15:10 |
| mwhahaha | shardy: sounds good | 15:11 |
| shardy | I guess we should wire in something via CI that uses that, I'll see if we can add a resource to the pingtest stack which fails when that's misconfigured | 15:11 |
| EmilienM | should we merge it? | 15:11 |
| *** oshvartz has quit IRC | 15:12 | |
| openstackgerrit | Merged openstack/tripleo-ui: Remove unused TripleO API service https://review.openstack.org/392602 | 15:13 |
| *** a_martin has joined #tripleo | 15:18 | |
| *** a_martin has left #tripleo | 15:18 | |
| shadower | EmilienM: do you know if/how we can remove the stable/newton branch in tripleo-validations? I don't think we want it yet (it's not part of the coordinated release) and apparently downstream packaging uses it instead of master | 15:19 |
| shadower | which is not what I'd like right now | 15:19 |
| *** tosky has joined #tripleo | 15:20 | |
| shardy | shadower: how can the downstream release not consume the stable branches? | 15:20 |
| shadower | shardy: well they do. But I'm not sure we want to do that for tripleo-validations just yet | 15:21 |
| EmilienM | shardy: we can't | 15:21 |
| EmilienM | err | 15:21 |
| EmilienM | shadower: ^ | 15:21 |
| shadower | oh okay | 15:21 |
| * shadower goes to write the backports | 15:21 | |
| shardy | shadower: yeah it's been released as a tripleo deliverable, and thus must follow the normal stable branch policy | 15:22 |
| shadower | I see. Fair enough | 15:23 |
| *** Vijayendra has quit IRC | 15:24 | |
| *** percevalbot has joined #tripleo | 15:25 | |
| *** rbowen has quit IRC | 15:27 | |
| *** rbowen has joined #tripleo | 15:27 | |
| *** jcoufal has quit IRC | 15:28 | |
| *** eggmaster has joined #tripleo | 15:29 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/diskimage-builder: Updated from global requirements https://review.openstack.org/392721 | 15:31 |
| *** jprovazn has joined #tripleo | 15:32 | |
| *** jcoufal has joined #tripleo | 15:34 | |
| *** dmacpher has joined #tripleo | 15:40 | |
| *** chandankumar has quit IRC | 15:40 | |
| *** Vijayendra has joined #tripleo | 15:40 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/python-tripleoclient: Updated from global requirements https://review.openstack.org/389945 | 15:40 |
| *** abregman_ has quit IRC | 15:41 | |
| jaosorior | bnemec: hey dude, got a bit of time to help me out? | 15:42 |
| *** saneax-_-|AFK is now known as saneax | 15:43 | |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Spec for adding diskimage_builder module in-chroot https://review.openstack.org/336664 | 15:43 |
| *** ebalduf has joined #tripleo | 15:45 | |
| bnemec | jaosorior: I'm in a call right now. Will let you know when I'm free. | 15:50 |
| *** percevalbot has quit IRC | 15:52 | |
| shadower | incoming patch bomb :-( | 15:53 |
| openstackgerrit | Tomas Sedovic proposed openstack/tripleo-validations: Validation to check the undercloud processes https://review.openstack.org/392847 | 15:53 |
| openstackgerrit | Tomas Sedovic proposed openstack/tripleo-validations: Validation to check ulimits on controller https://review.openstack.org/392848 | 15:53 |
| openstackgerrit | Tomas Sedovic proposed openstack/tripleo-validations: Validate ntp is running and synchronised https://review.openstack.org/392849 | 15:53 |
| openstackgerrit | Tomas Sedovic proposed openstack/tripleo-validations: Validate pacemaker status https://review.openstack.org/392850 | 15:53 |
| openstackgerrit | Tomas Sedovic proposed openstack/tripleo-validations: Validate the open-files-limit value in mysql https://review.openstack.org/392851 | 15:53 |
| openstackgerrit | Tomas Sedovic proposed openstack/tripleo-validations: Validate rabbitmq limits https://review.openstack.org/392852 | 15:53 |
| openstackgerrit | Tomas Sedovic proposed openstack/tripleo-validations: Remove `become: true` from the inventory script https://review.openstack.org/392853 | 15:53 |
| openstackgerrit | Tomas Sedovic proposed openstack/tripleo-validations: Generate documentation for validations https://review.openstack.org/392854 | 15:53 |
| openstackgerrit | Tomas Sedovic proposed openstack/tripleo-validations: Make the ini-based validations more robust https://review.openstack.org/392855 | 15:53 |
| openstackgerrit | Tomas Sedovic proposed openstack/tripleo-validations: Add network gateway validation https://review.openstack.org/392856 | 15:53 |
| openstackgerrit | Tomas Sedovic proposed openstack/tripleo-validations: Updated from global requirements https://review.openstack.org/392857 | 15:53 |
| openstackgerrit | Tomas Sedovic proposed openstack/tripleo-validations: Validate rogue DHCP servers https://review.openstack.org/392858 | 15:53 |
| openstackgerrit | Tomas Sedovic proposed openstack/tripleo-validations: Add a validation-specific Ansible output callback https://review.openstack.org/392859 | 15:53 |
| openstackgerrit | Tomas Sedovic proposed openstack/tripleo-validations: Add the NoopFirewallDriver validation https://review.openstack.org/392860 | 15:53 |
| openstackgerrit | Tomas Sedovic proposed openstack/tripleo-validations: Validate haproxy configuration values https://review.openstack.org/392861 | 15:53 |
| openstackgerrit | Tomas Sedovic proposed openstack/tripleo-validations: Validation to verify connectivity to overcloud https://review.openstack.org/392862 | 15:53 |
| openstackgerrit | Tomas Sedovic proposed openstack/tripleo-validations: Validation to detect if undercloud disks use 512e https://review.openstack.org/392863 | 15:53 |
| openstackgerrit | Tomas Sedovic proposed openstack/tripleo-validations: Validate the IP range for the control plane https://review.openstack.org/392864 | 15:53 |
| jaosorior | shadower: ouch!! | 15:53 |
| dtantsur | WOW, that's a dump | 15:54 |
| shadower | jaosorior: yea I didn't realise I had to cherrypick these to stable | 15:54 |
| * shadower is an ideot | 15:54 | |
| shadower | who can't even type | 15:54 |
| dtantsur | LOL this is post-summit week, everything is normal | 15:54 |
| jpich | d0ugal: Testing the passwords patch now... I wonder if we should save them back into the environment for clarity/avoid further problems down the line? | 15:54 |
| *** percevalbot has joined #tripleo | 15:54 | |
| shadower | jaosorior: so yea if you have time to review ~20 validation cherry-picks, I'd very much appreciate it | 15:55 |
| shadower | mandre: ^ any chance you could have a look at these, too? | 15:55 |
| jaosorior | shadower: add me as a reviewer | 15:55 |
| shadower | it's all just clean cherry-pics | 15:56 |
| shadower | jaosorior: thanks, will do | 15:56 |
| *** dmacpher is now known as dmacpher-lunch | 15:56 | |
| *** dmacpher-lunch is now known as dmacpher-afk | 15:56 | |
| jaosorior | EmilienM: so it seems that getting cinder back to eventlet passed the pingtest. We can merge that and I'll try to dig into why it's failing with httpd. | 15:56 |
| jaosorior | EmilienM: I'm quite stumped... as the ovb gate was quite green in every patch I did for getting cinder to httpd | 15:57 |
| jtomasek | shadower: since you're core, it takes only another core to automatically +2 and verified them (iirc the backports rule correctly) | 15:57 |
| shadower | cool | 15:58 |
| dtantsur | backports proposed by stable cores require only one +2 | 15:58 |
| dtantsur | if stable cores are the same as regular cores for tripleo, this holds | 15:58 |
| * dtantsur is a stable liaison for ironic, so he has to remember some rules.. | 15:58 | |
| *** bana_k has joined #tripleo | 15:58 | |
| shardy | shadower: do all of those combine to fix bug #1637273 ? | 16:00 |
| openstack | bug 1637273 in tripleo "The mysql-open-files-limit validation fails" [High,Fix released] https://launchpad.net/bugs/1637273 - Assigned to Tomas Sedovic (tsedovic) | 16:00 |
| EmilienM | jaosorior: interesting... | 16:00 |
| EmilienM | jaosorior: can you do recheck to see if it's consistent? | 16:00 |
| jaosorior | EmilienM: will do | 16:00 |
| shadower | shardy: they don't. I'm not sure why it set that topic. | 16:00 |
| shadower | shardy: my local branch name is different | 16:00 |
| *** snecklifter has quit IRC | 16:01 | |
| d0ugal | jpich: it will be saved to the mistral environment, just as other Heat parameters are. | 16:01 |
| *** [1]cdearborn has quit IRC | 16:02 | |
| jaosorior | panda: what's up with your fix for redis? how's that going? | 16:03 |
| EmilienM | jaosorior: did we switch cinder into wsgi in apache? | 16:03 |
| jaosorior | EmilienM: huh? | 16:03 |
| EmilienM | soerry | 16:03 |
| EmilienM | in newton | 16:03 |
| EmilienM | only master right? | 16:04 |
| *** ohamada has quit IRC | 16:04 | |
| jaosorior | EmilienM: no, only ocata | 16:04 |
| EmilienM | excellent | 16:04 |
| EmilienM | so we need to land this puppet-tripleo patch that Alex did | 16:04 |
| panda | jaosorior: the fix is in a package, it's working but will not be included in less than 10 days, my workaround instead is not wokrking at all ... I'm not sure how to deal with it with dib | 16:04 |
| jaosorior | EmilienM: any idea how to help panda with that? | 16:04 |
| EmilienM | jaosorior: wait, I'm going to remove the depends-on the tirpleo-ci patch that increase the timeout | 16:04 |
| EmilienM | panda: 10 days??? | 16:05 |
| EmilienM | to update an rpm? | 16:05 |
| openstackgerrit | Merged openstack/diskimage-builder: Make xenial the ubuntu default https://review.openstack.org/334136 | 16:05 |
| *** ohamada has joined #tripleo | 16:05 | |
| panda | EmilienM: it's RHEL base ... | 16:05 |
| *** pcaruana has quit IRC | 16:06 | |
| *** aufi has quit IRC | 16:07 | |
| d0ugal | Anyone got a moment to review https://review.openstack.org/#/c/391373/ ? | 16:08 |
| d0ugal | Seeing the timeout being hit on another review | 16:08 |
| panda | slagle: I'm trying to set selinux to permissive in overcloud images, looks like this https://review.openstack.org/392703 is failing .. can you provide any hints on how to do it properly ? | 16:08 |
| *** limao has joined #tripleo | 16:08 | |
| *** rcernin has quit IRC | 16:09 | |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-tripleo: Create heat user in keystone profile https://review.openstack.org/392288 | 16:10 |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart: Add roles-gate playbook run to OVB ci-script https://review.openstack.org/392873 | 16:10 |
| *** ooolpbot has joined #tripleo | 16:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1638350 | 16:10 |
| *** ooolpbot has quit IRC | 16:10 | |
| openstack | Launchpad bug 1638350 in tripleo "pingtest failing on OVB jobs to create Cinder volume and Nova server" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 16:10 |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-tripleo: Run cinder over eventlet again https://review.openstack.org/392647 | 16:10 |
| EmilienM | mwhahaha, jaosorior^ I removed the depends-on | 16:10 |
| EmilienM | so we can land Alex fix, backport it into stable/newton | 16:10 |
| openstackgerrit | Merged openstack/tripleo-validations: Validation to check the undercloud processes https://review.openstack.org/392847 | 16:10 |
| jaosorior | EmilienM: works for me | 16:10 |
| EmilienM | and then merge jaosorior fix so our CI is working again | 16:10 |
| openstackgerrit | Merged openstack/tripleo-validations: Validation to check ulimits on controller https://review.openstack.org/392848 | 16:11 |
| EmilienM | in the meantime we try to find a solution for the redis thing | 16:11 |
| jaosorior | EmilienM: I'm not happy about getting cinder off httpd... and honestly the fact that it over eventlet works makes no sense to me... but alright... I'll try to get it working on a further iteration | 16:11 |
| EmilienM | jaosorior: I'm not happy either :( just try to find solutions here | 16:12 |
| openstackgerrit | Merged openstack/tripleo-validations: Validate ntp is running and synchronised https://review.openstack.org/392849 | 16:14 |
| openstackgerrit | Merged openstack/tripleo-validations: Validate pacemaker status https://review.openstack.org/392850 | 16:15 |
| openstackgerrit | Merged openstack/tripleo-validations: Validate the open-files-limit value in mysql https://review.openstack.org/392851 | 16:15 |
| openstackgerrit | Merged openstack/tripleo-validations: Validate rabbitmq limits https://review.openstack.org/392852 | 16:15 |
| openstackgerrit | Merged openstack/tripleo-validations: Remove `become: true` from the inventory script https://review.openstack.org/392853 | 16:15 |
| openstackgerrit | Merged openstack/tripleo-validations: Generate documentation for validations https://review.openstack.org/392854 | 16:15 |
| jaosorior | EmilienM: lets merge this if it solves the issue and tomorrow I start working on getting cinder running over httpd again. If have time to help out I would appreciate it. | 16:15 |
| openstackgerrit | Merged openstack/tripleo-validations: Make the ini-based validations more robust https://review.openstack.org/392855 | 16:16 |
| openstackgerrit | Merged openstack/tripleo-validations: Add network gateway validation https://review.openstack.org/392856 | 16:16 |
| bnemec | jaosorior: I'm free now. What's up? | 16:16 |
| shadower | jaosorior: thanks a lot! | 16:16 |
| EmilienM | bnemec: we figured that a lot of not all CI jobs running on ovb were failing on pingtest to create volume1 | 16:17 |
| EmilienM | bnemec: by investigating, I saw cinder-api running on one controller only, but haproxy not aware about this downtime on other controllers | 16:17 |
| openstackgerrit | Dougal Matthews proposed openstack/instack-undercloud: Increase the timeout for the default plan creation https://review.openstack.org/391373 | 16:18 |
| *** dsneddon_ is now known as dsneddon | 16:19 | |
| *** anton is now known as aarapov | 16:19 | |
| *** aarapov is now known as anton | 16:21 | |
| *** bana_k has quit IRC | 16:21 | |
| *** anton is now known as aarapov | 16:21 | |
| *** aarapov is now known as anton | 16:21 | |
| *** bana_k has joined #tripleo | 16:22 | |
| *** Vijayendra has quit IRC | 16:23 | |
| gfidente | EmilienM jaosorior why cinder-api would be running only on one controller though? | 16:23 |
| jaosorior | gfidente: no idea dude | 16:23 |
| gfidente | jaosorior that seems to be the actual issue though | 16:24 |
| jaosorior | gfidente: so, recently we switched to running cinder over httpd | 16:24 |
| gfidente | pcmk is only managing cinder-volume | 16:24 |
| jaosorior | not sure if that's related... and how the hell is that an issue, since I tried it locally and it worked | 16:24 |
| jaosorior | exactly | 16:24 |
| jaosorior | makes no sense to me | 16:24 |
| gfidente | so cinder-api should be running on all nodes | 16:24 |
| jaosorior | gfidente: but running cinder over eventlet made the gate pass | 16:25 |
| jaosorior | I'm more leaning towards the fix that panda is doing, that might be the real issue | 16:25 |
| EmilienM | maybe it's a performance problem also | 16:25 |
| jaosorior | buuut, to be honest, I have no clue what's up | 16:25 |
| EmilienM | gnocchi eating CPU because of redis | 16:25 |
| jaosorior | EmilienM: which might make sense, since gnocchi is also running over httpd | 16:25 |
| EmilienM | if anyone wants to debug, here are the logs: http://logs.openstack.org/88/392288/6/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/d9ad9e9/ | 16:26 |
| panda | why all gates to the selinux-permissive patch are failing with this http://logs.openstack.org/03/392703/1/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/583d3f3/console.html#_2016-11-02_15_48_18_332283. It shouldn't be related ... | 16:26 |
| EmilienM | they are logs of a patch that contains the fix for the heat problem | 16:26 |
| *** jkilpatr has quit IRC | 16:26 | |
| EmilienM | so what remains now is the cinder problem | 16:26 |
| jaosorior | panda: pass the patch | 16:26 |
| EmilienM | jaosorior: not gnocchi-statsd | 16:26 |
| EmilienM | which was the process who eats cpu iirc | 16:26 |
| jaosorior | panda: I saw the same issue in another unrelated patch | 16:26 |
| panda | jaosorior: https://review.openstack.org/392703 any hints to set selinux permissive in oevercloud images are appreciated | 16:27 |
| panda | not sure I'm doing this properly | 16:27 |
| * EmilienM brb lunch | 16:27 | |
| *** dsariel has quit IRC | 16:27 | |
| jaosorior | panda: not sure how to do it either | 16:27 |
| *** Vijayendra has joined #tripleo | 16:27 | |
| jaosorior | I gotta go :/ | 16:28 |
| *** jaosorior has quit IRC | 16:28 | |
| bnemec | panda: Just include selinux-permissive in the image. You should be able to use https://github.com/openstack/python-tripleoclient/blob/master/tripleoclient/v1/overcloud_image.py#L320 to do that. | 16:28 |
| panda | oh, I'm alone now, with all things crumbling around me | 16:28 |
| panda | and bnemec comes to the rescue! | 16:28 |
| *** mcornea has quit IRC | 16:29 | |
| panda | bnemec: ok, I'll do it, do you think this could also work ? https://review.openstack.org/392703 | 16:29 |
| openstackgerrit | Brent Eagles proposed openstack/tripleo-heat-templates: Add more documentation around configuring DVR https://review.openstack.org/392874 | 16:30 |
| panda | bnemec: ok so you say it's easier if I just export OVERCLOUD_FULL_DIB_EXTRA_ARGS=selinux-permissive before building the image, right ? | 16:32 |
| *** hewbrocca is now known as hewbrocca_afk | 16:32 | |
| *** Vijayendra has quit IRC | 16:33 | |
| bnemec | panda: I think that's slightly less hacky than copying dib files around. It's still kind of ugly doing it in tripleo.sh though since it means people following the docs won't get the workaround. | 16:34 |
| *** Guest48712 has quit IRC | 16:34 | |
| *** ealcaniz has quit IRC | 16:34 | |
| *** michapma_alt has quit IRC | 16:35 | |
| panda | bnemec: step 1: unblock CI, step 2: help rest of the world :) changing patch now, any other suggestion appreciated | 16:35 |
| *** numans has quit IRC | 16:36 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci: Workaround: Set selinux to permissive to workaround bug LP#1637961 https://review.openstack.org/392703 | 16:37 |
| *** jkilpatr has joined #tripleo | 16:39 | |
| *** ayoung has quit IRC | 16:40 | |
| *** ebarrera has quit IRC | 16:40 | |
| *** cylopez has quit IRC | 16:40 | |
| *** nyechiel has quit IRC | 16:44 | |
| *** jbadiapa has quit IRC | 16:45 | |
| *** jpena is now known as jpena|brb | 16:45 | |
| *** jbadiapa has joined #tripleo | 16:46 | |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Fix password handling for users upgrading from Mitaka https://review.openstack.org/392593 | 16:46 |
| *** artom has quit IRC | 16:48 | |
| *** mhenkel has joined #tripleo | 16:50 | |
| *** pblaho has quit IRC | 16:52 | |
| *** artom has joined #tripleo | 16:54 | |
| *** tremble has quit IRC | 16:56 | |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Fix password handling for users upgrading from Mitaka https://review.openstack.org/392593 | 16:56 |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart: Add roles-gate playbook run to OVB ci-script https://review.openstack.org/392873 | 16:56 |
| chem | d0ugal: I was about to ping you for this issue :) | 16:57 |
| *** milan has quit IRC | 16:57 | |
| openstackgerrit | Alex Schultz proposed openstack/puppet-tripleo: Create heat user in keystone profile https://review.openstack.org/392288 | 16:57 |
| *** bana_k has quit IRC | 16:57 | |
| d0ugal | chem: hah :) | 16:57 |
| d0ugal | chem: the patch is super hacky :( | 16:58 |
| chem | d0ugal: I can test it and gives it a try, I'll report to the review. | 16:58 |
| d0ugal | chem: Great, thanks. | 16:58 |
| *** imranh has joined #tripleo | 16:59 | |
| *** sshnaidm is now known as sshnaidm|afk | 17:01 | |
| *** ealcaniz has joined #tripleo | 17:04 | |
| openstackgerrit | Dan Sneddon proposed openstack/tripleo-specs: Spec: tripleo-routed-networks-deployment - Routed Spine and Leaf https://review.openstack.org/377088 | 17:06 |
| *** limao has quit IRC | 17:06 | |
| *** dtantsur is now known as dtantsur|afk | 17:06 | |
| *** ooolpbot has joined #tripleo | 17:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1638350 | 17:10 |
| *** ooolpbot has quit IRC | 17:10 | |
| openstack | Launchpad bug 1638350 in tripleo "pingtest failing on OVB jobs to create Cinder volume and Nova server" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 17:10 |
| *** jaosorior has joined #tripleo | 17:13 | |
| thrash | d0ugal: does that patch address this? AttributeError: 'module' object has no attribute 'get_overcloud_passwords' | 17:13 |
| thrash | d0ugal: I'm seeing that at the end of a deployment. | 17:14 |
| d0ugal | thrash: no, I've not seen that error. | 17:17 |
| openstackgerrit | Dan Sneddon proposed openstack/tripleo-specs: Spec: undercloud-ntp-server - Add NTP services to Undercloud https://review.openstack.org/350729 | 17:17 |
| d0ugal | thrash: do you have the traceback? | 17:17 |
| openstackgerrit | Dan Sneddon proposed openstack/tripleo-specs: Spec: undercloud-ntp-server - Add NTP services to Undercloud https://review.openstack.org/350729 | 17:18 |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-tripleo: Run cinder over eventlet again https://review.openstack.org/392647 | 17:21 |
| openstackgerrit | Tomas Sedovic proposed openstack/tripleo-validations: Validation to verify connectivity to overcloud https://review.openstack.org/392862 | 17:22 |
| openstackgerrit | Tomas Sedovic proposed openstack/tripleo-validations: Validation to detect if undercloud disks use 512e https://review.openstack.org/392863 | 17:22 |
| openstackgerrit | Tomas Sedovic proposed openstack/tripleo-validations: Add the NoopFirewallDriver validation https://review.openstack.org/392860 | 17:22 |
| openstackgerrit | Tomas Sedovic proposed openstack/tripleo-validations: Validate haproxy configuration values https://review.openstack.org/392861 | 17:22 |
| openstackgerrit | Tomas Sedovic proposed openstack/tripleo-validations: Validate rogue DHCP servers https://review.openstack.org/392858 | 17:22 |
| openstackgerrit | Tomas Sedovic proposed openstack/tripleo-validations: Add a validation-specific Ansible output callback https://review.openstack.org/392859 | 17:22 |
| openstackgerrit | Tomas Sedovic proposed openstack/tripleo-validations: Updated from global requirements https://review.openstack.org/392857 | 17:22 |
| openstackgerrit | Tomas Sedovic proposed openstack/tripleo-validations: Fix the mysql-open-files-limit validation https://review.openstack.org/392869 | 17:22 |
| openstackgerrit | Tomas Sedovic proposed openstack/tripleo-validations: Fix the ctlplane-ip-range validation https://review.openstack.org/392868 | 17:22 |
| openstackgerrit | Tomas Sedovic proposed openstack/tripleo-validations: Validate the IP range for the control plane https://review.openstack.org/392864 | 17:22 |
| openstackgerrit | Tomas Sedovic proposed openstack/tripleo-validations: Use custom CA cert with tripleo-ansible-inventory https://review.openstack.org/392867 | 17:22 |
| shadower | I'm so sorry | 17:22 |
| *** dhill_ has quit IRC | 17:22 | |
| *** jlinkes has quit IRC | 17:23 | |
| *** florianf has quit IRC | 17:23 | |
| d0ugal | shadower: stop working so hard! | 17:24 |
| shadower | haha | 17:25 |
| * shadower is about to leave | 17:25 | |
| d0ugal | shadower: oh, it is all backports? Then I am less impressed :P | 17:25 |
| shadower | d0ugal: yeah. I didn't realise we had to backport them but we do | 17:25 |
| shadower | all boring work | 17:25 |
| shadower | and finicky just enough so it can't be automated fully | 17:26 |
| *** jpena|brb is now known as jpena | 17:28 | |
| mwhahaha | EmilienM: yea the new memcache package results in memcache service failures with tripleo as well, http://logs.openstack.org/74/392874/1/check/gate-tripleo-ci-centos-7-nonha-multinode/183a7a8/logs/syslog.txt.gz#_Nov_02_16_48_10 | 17:32 |
| *** zoli|wfh is now known as zoli|gone | 17:32 | |
| *** lucasagomes is now known as lucas-afk | 17:32 | |
| EmilienM | mwhahaha: can't we revert it? where is package? in rdo? | 17:32 |
| EmilienM | dmsimard: ^ | 17:32 |
| dmsimard | EmilienM: it's in buildlogs -testing | 17:32 |
| dmsimard | EmilienM: it's that cdn crap again, not sure to what extent we can and can't revert | 17:33 |
| EmilienM | dmsimard: if we fix it using mwhahaha's code, we need to add a new module in OPM | 17:33 |
| dmsimard | yes | 17:33 |
| mwhahaha | well i can do it without a new module | 17:33 |
| dmsimard | need to notify jayg | 17:33 |
| mwhahaha | we don't use the limits module elsewhere in triple o (but we should) | 17:33 |
| thrash | d0ugal: I think it might be a mismatch between my tripleoclient and tripleo-common | 17:33 |
| mwhahaha | can always just do it the old fashioned way with augeas or file_line :D | 17:33 |
| d0ugal | thrash: ah, that might make sense. | 17:34 |
| dmsimard | EmilienM: another temporary compromise would be to pin the version of memcached on your end to the previous one (memcached-1.4.25-1.el7.x86_64) | 17:34 |
| thrash | d0ugal: http://paste.openstack.org/show/587679/ | 17:34 |
| *** chem has quit IRC | 17:34 | |
| EmilienM | dmsimard: yes if we can do that asap that would be great | 17:35 |
| d0ugal | thrash: https://github.com/openstack/python-tripleoclient/blob/master/tripleoclient/workflows/parameters.py#L26 | 17:35 |
| d0ugal | thrash: that is what you are missing for some reason. | 17:35 |
| dmsimard | EmilienM: pinning is client side, though, not on our end | 17:35 |
| *** [1]cdearborn has joined #tripleo | 17:36 | |
| dmsimard | talking with Haikel in #rdo about it.. | 17:36 |
| *** shardy has quit IRC | 17:36 | |
| *** milan has joined #tripleo | 17:37 | |
| *** cylopez has joined #tripleo | 17:38 | |
| d0ugal | Sigh, I am seeing this tons now: http://logs.openstack.org/93/392593/6/check/gate-tripleo-ci-centos-7-nonha-multinode/33770da/console.html#_2016-11-02_17_27_10_078673 | 17:39 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: nova: add missing vnc console port in firewall https://review.openstack.org/392898 | 17:39 |
| d0ugal | thrash: Can you take a look at this? https://review.openstack.org/#/c/391373/ | 17:40 |
| d0ugal | thrash: I am seeing quite a few CI failures due to timeouts, and one failure locally in my dev env. It just seems to be really slow sometimes. | 17:40 |
| pradk | can i get some reviews on https://review.openstack.org/#/c/392287/ please | 17:40 |
| *** fragatina has quit IRC | 17:40 | |
| *** saneax is now known as saneax-_-|AFK | 17:41 | |
| *** ccamacho is now known as ccamacho|out | 17:42 | |
| *** lmiccini has quit IRC | 17:43 | |
| *** bana_k has joined #tripleo | 17:44 | |
| thrash | d0ugal: I think I know what happened... I was messing with the composable undercloud stuff ad I think I had an older version of the client. | 17:44 |
| thrash | my bad. :D | 17:44 |
| EmilienM | mwhahaha: even multinode jobs are broken now right? | 17:44 |
| d0ugal | thrash: hah, at least that should be easy to resolve :) | 17:44 |
| mwhahaha | not sure, if no memcache is available does that break the world? | 17:44 |
| EmilienM | yes? | 17:45 |
| EmilienM | we use memcache for swift at least | 17:45 |
| EmilienM | on the undercloud | 17:45 |
| EmilienM | can swift work without memcache? | 17:45 |
| openstackgerrit | Alex Schultz proposed openstack/puppet-tripleo: Configure memcache file limits https://review.openstack.org/392901 | 17:45 |
| EmilienM | mwhahaha: we also need to fix instack-undercloud I think | 17:45 |
| mwhahaha | k | 17:45 |
| d0ugal | EmilienM: are you seeing an error creating the default plan? | 17:46 |
| EmilienM | mwhahaha: nice hack btw | 17:46 |
| EmilienM | d0ugal: it might be related | 17:46 |
| EmilienM | d0ugal: mistral requires swift that require memcache that is broken iiuc | 17:46 |
| EmilienM | I think that all CI is currently broken now | 17:46 |
| d0ugal | EmilienM: This passed recently: https://review.openstack.org/#/c/391373/2 | 17:46 |
| openstackgerrit | Honza Pokorny proposed openstack/tripleo-ui: Redirect user to login page when token expires https://review.openstack.org/392589 | 17:47 |
| EmilienM | d0ugal: let me check your error | 17:47 |
| *** jpich has quit IRC | 17:47 | |
| *** panda is now known as panda|bbl | 17:47 | |
| EmilienM | Nov 2 17:13:01 centos-7-2-node-osic-cloud1-s3500-5160825 systemd: [/usr/lib/systemd/system/memcached.service:42] Unknown lvalue 'MemoryDenyWriteExecute' in section 'Service' | 17:48 |
| EmilienM | on the undercloud http://logs.openstack.org/93/392593/6/check/gate-tripleo-ci-centos-7-nonha-multinode/33770da/logs/var/log/messages | 17:48 |
| EmilienM | mwhahaha: is that the symptom? ^ | 17:48 |
| d0ugal | EmilienM: oh, I had missed that. | 17:48 |
| mwhahaha | EmilienM: Nov 2 17:11:57 centos-7-2-node-osic-cloud1-s3500-5160825 memcached: failed to set rlimit for open files. Try starting as root or requesting smaller maxconns value. | 17:48 |
| *** thrash is now known as thrash|biab | 17:48 | |
| EmilienM | d0ugal: so yes, don't run recheck now | 17:49 |
| d0ugal | EmilienM: okay :) | 17:49 |
| *** athomas has quit IRC | 17:50 | |
| *** athomas has joined #tripleo | 17:51 | |
| EmilienM | mwhahaha: can I submit the patch for instack? | 17:52 |
| EmilienM | mwhahaha: I have an idea | 17:52 |
| mwhahaha | EmilienM: sure i was just doing it but feel free to if you want something else | 17:52 |
| *** cylopez has quit IRC | 17:53 | |
| *** noslzzp has quit IRC | 17:53 | |
| *** cylopez has joined #tripleo | 17:53 | |
| *** nyechiel has joined #tripleo | 17:53 | |
| *** noslzzp has joined #tripleo | 17:55 | |
| *** noslzzp has quit IRC | 17:55 | |
| *** derekh has quit IRC | 17:55 | |
| openstackgerrit | Emilien Macchi proposed openstack/instack-undercloud: memcache: use composable service https://review.openstack.org/392909 | 17:56 |
| EmilienM | mwhahaha: ok that's how I see it for longterm | 17:56 |
| EmilienM | but I realize we have a chicken and egg problem | 17:56 |
| *** noslzzp has joined #tripleo | 17:56 | |
| mwhahaha | we should probably just drop step from that role | 17:56 |
| EmilienM | I think none of both patches will pass CI as they fix undercloud and another overcloud | 17:57 |
| EmilienM | if you see what I mean | 17:57 |
| mwhahaha | yea | 17:57 |
| EmilienM | but, I have an idea :P | 17:57 |
| EmilienM | put your code in somewhere loaded by undercloud | 17:57 |
| mwhahaha | k let me post my temporary patch that we can revert later as part of the use the role thing | 17:57 |
| EmilienM | mwhahaha: go ahead | 17:58 |
| EmilienM | mwhahaha: we still need to find a way to send a patch that fix both undercloud & overcloud. My idea was to put this code in ::tripleo::profile::base::kernel for example, as we load it in undercloud AND overcloud | 17:58 |
| EmilienM | we could immediatly revert this thing once we have it in puppet-tripleo | 17:59 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Fix password handling for users upgrading from Mitaka https://review.openstack.org/392593 | 17:59 |
| EmilienM | (in memcached role) | 17:59 |
| EmilienM | dmsimard: does it break newton CI too? | 17:59 |
| dmsimard | EmilienM: if you use stable repos, no | 17:59 |
| dmsimard | EmilienM: the change landed in -testing, not -stable | 18:00 |
| EmilienM | ok | 18:00 |
| dmsimard | however, it will probably eventually be pushed to -stable is my guess, it addresses 3 CVEs | 18:00 |
| mwhahaha | EmilienM: could just swap out the saz-memcache module for one that patches max_connections to 1024 as a temp solution | 18:00 |
| EmilienM | mwhahaha: yes, with hiera? | 18:00 |
| mwhahaha | maybe? | 18:01 |
| EmilienM | mwhahaha: even that, you need to do it for both undercloud & overcloud in same time or CI jobs won't pass all | 18:01 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Fix password handling for users upgrading from Mitaka https://review.openstack.org/392593 | 18:02 |
| mwhahaha | thats why i said swap out the memcache module itself, or does that get loaded in two seperate ways | 18:02 |
| openstackgerrit | Alex Schultz proposed openstack/instack-undercloud: Configure memcache file limits https://review.openstack.org/392914 | 18:02 |
| EmilienM | mwhahaha: oh you mean stop using the module? | 18:03 |
| * EmilienM googling swapout | 18:03 | |
| mwhahaha | EmilienM: a fork of it that | 18:03 |
| mwhahaha | yea so we create a fork of the memcache module, patch in max_connections=1024 and inject that into the system | 18:04 |
| EmilienM | mhh | 18:04 |
| * mwhahaha has no idea how to do inject part for tripleo | 18:04 | |
| EmilienM | a sec | 18:04 |
| EmilienM | I don't get it, why don't we set memcached::max_connections: 1024 in hiera? | 18:05 |
| mwhahaha | unless you can do that in two places for under and overcloud | 18:05 |
| EmilienM | yes we could | 18:05 |
| mwhahaha | that works just fine then | 18:06 |
| mwhahaha | i just don't know how to do that part :D | 18:06 |
| EmilienM | this is going to be very very dirty | 18:06 |
| EmilienM | but it will work i think | 18:06 |
| EmilienM | mwhahaha: the end-goal is to use tripleo memcached profile that configure limits, right? | 18:07 |
| mwhahaha | yea | 18:07 |
| EmilienM | ok. Let me try something (very dirty again) | 18:07 |
| *** ooolpbot has joined #tripleo | 18:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1638350 | 18:10 |
| *** ooolpbot has quit IRC | 18:10 | |
| openstack | Launchpad bug 1638350 in tripleo "pingtest failing on OVB jobs to create Cinder volume and Nova server" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 18:10 |
| *** athomas has quit IRC | 18:11 | |
| openstackgerrit | Thiago da Silva proposed openstack/tripleo-heat-templates: set url_base option in static web middleware https://review.openstack.org/392918 | 18:11 |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-tripleo: Configure memcache file limits https://review.openstack.org/392901 | 18:11 |
| EmilienM | mwhahaha: first step ^ | 18:12 |
| EmilienM | oh it won't work, a sec | 18:12 |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-tripleo: Configure memcache file limits https://review.openstack.org/392901 | 18:13 |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-tripleo: Remove memcached workaround in kernel manifest https://review.openstack.org/392920 | 18:14 |
| EmilienM | mwhahaha: done^ | 18:14 |
| EmilienM | wdyt? | 18:15 |
| mwhahaha | it might work, i'm trying to see if there are further issues since it's not working in p-o-i | 18:16 |
| mwhahaha | gimme a min | 18:16 |
| EmilienM | ok | 18:16 |
| mwhahaha | might need to drop the relationship unless memcached is included everywhere | 18:17 |
| EmilienM | mwhahaha: in poi you mean? | 18:17 |
| sshnaidm|afk | have anyone seen such error? is it known issue? "RuntimeError: Timed out creating the default Deployment Plan" in undercloud install | 18:17 |
| mwhahaha | EmilienM: no in your hack | 18:17 |
| EmilienM | sshnaidm|afk: all CI is broken now | 18:17 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart: Document Third Party CI and devmode.sh https://review.openstack.org/360007 | 18:17 |
| *** openstackgerrit has quit IRC | 18:18 | |
| *** flepied has quit IRC | 18:18 | |
| EmilienM | sshnaidm|afk: mistral doesn't work because swift doesn't work because memcached doesn't start | 18:18 |
| EmilienM | sshnaidm|afk: we're trying to fix it now, if you read us | 18:18 |
| *** openstackgerrit has joined #tripleo | 18:18 | |
| sshnaidm|afk | EmilienM, thanks, I'll read from beginning | 18:18 |
| *** sshnaidm|afk is now known as sshnaidm | 18:18 | |
| *** jkilpatr has quit IRC | 18:19 | |
| EmilienM | mwhahaha: in kernel role? | 18:19 |
| EmilienM | mwhahaha: put it in the if? | 18:19 |
| mwhahaha | EmilienM: yea just a thought on first glance, i'd have to think about it some more | 18:19 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Fix password handling for users upgrading from Mitaka https://review.openstack.org/392593 | 18:20 |
| EmilienM | k | 18:20 |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-tripleo: Remove memcached workaround in kernel manifest https://review.openstack.org/392920 | 18:21 |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-tripleo: Configure memcache file limits https://review.openstack.org/392901 | 18:21 |
| EmilienM | let's try again | 18:21 |
| *** mburned is now known as mburned_out | 18:23 | |
| *** mburned_out is now known as mburned | 18:23 | |
| EmilienM | mwhahaha: adding ci & alert tags to https://bugs.launchpad.net/tripleo/+bug/1638646 | 18:27 |
| openstack | Launchpad bug 1638646 in tripleo "memcache package update breaks memcache starting due to rlimits issue" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 18:27 |
| EmilienM | so they appear here and people know we are broken | 18:27 |
| *** mgould is now known as mgould|afk | 18:30 | |
| tosky | is there some ntp-related setting in tripleo that can be reused amongst the deployed components? | 18:31 |
| *** jkilpatr has joined #tripleo | 18:32 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci: Workaround: Set selinux to permissive to workaround bug LP#1637961 https://review.openstack.org/392703 | 18:39 |
| *** jpena is now known as jpena|off | 18:39 | |
| bkero | Were the puppetlabs-ntp problems fixed by the current pinning in rdo.yml and the source installation fixed by https://review.openstack.org/#/c/389250/ ? | 18:43 |
| bkero | Or is this planned on being fixed by upgrading to puppet4 for tripleo? | 18:44 |
| bkero | EmilienM: ^ | 18:44 |
| EmilienM | bkero: it was fixed by https://review.rdoproject.org/r/#/c/3345/ | 18:45 |
| EmilienM | why? | 18:45 |
| bkero | EmilienM: Just going through this etherpad. It isn't crossed off. https://etherpad.openstack.org/p/tripleo-ci-status | 18:46 |
| *** nyechiel has quit IRC | 18:50 | |
| bkero | EmilienM: fixes it for install-from-package, but doesn't this still need to land? https://review.openstack.org/#/c/389250/ | 18:50 |
| EmilienM | bkero: no, we install opm from rpms, not from source | 18:50 |
| EmilienM | since we have delorean in place | 18:51 |
| bkero | So dib install modules from source is still broken, but that doesn't affect our CI? Okay | 18:51 |
| *** sudipto has quit IRC | 18:52 | |
| *** sudipto_ has quit IRC | 18:52 | |
| bkero | weshay was saying there are outstanding critical issues that need to be addressed. Is that memcache, or something else? | 18:52 |
| *** pkovar has quit IRC | 18:54 | |
| * bkero assumes it's that and relocates to #rdo | 18:54 | |
| EmilienM | bkero: https://bugs.launchpad.net/tripleo/+bugs?field.tag=alert | 18:55 |
| bkero | EmilienM: thanks | 18:55 |
| EmilienM | yw | 18:56 |
| panda|bbl | my workaround fails for a completely unrelated problem ... :( now I'm testing without the depends on the heat fix | 18:56 |
| *** tosky has quit IRC | 18:57 | |
| EmilienM | panda|bbl: have you seen memcached is broken? | 18:58 |
| panda|bbl | EmilienM: yes, is that the problem that make default deployment plan timing out ? | 19:00 |
| EmilienM | panda|bbl: yeah, mistral plan fails because swift fails to start because memcached is broken | 19:02 |
| EmilienM | it's a long story :P | 19:02 |
| *** paramite has quit IRC | 19:02 | |
| *** chandankumar has joined #tripleo | 19:03 | |
| panda|bbl | ... we really need to stop updating the packages in the gates .. | 19:05 |
| EmilienM | no, we should gate their updates ;-) | 19:06 |
| EmilienM | stop updates is bad, memcached had CVE fixes | 19:06 |
| mwhahaha | to be honest we need to do a better job ensuring services are running and are OK in our CI as well | 19:07 |
| mwhahaha | i was thinking about that earlier since we had issues with keystone yesterday because of the fernet tokens, and that cinder-api thing | 19:07 |
| mwhahaha | we don't have good visibility of understanding if all the expected services are up and running on the expected hosts | 19:08 |
| EmilienM | right. We need service validation | 19:08 |
| EmilienM | mwhahaha: did you see https://review.openstack.org/#/c/342202/ ? | 19:08 |
| *** akshai has quit IRC | 19:08 | |
| mwhahaha | EmilienM: interesting | 19:09 |
| EmilienM | mwhahaha: so we can do "composable service validation" just with puppet | 19:10 |
| *** milan has quit IRC | 19:10 | |
| EmilienM | catalog would fail if validation didn't work | 19:10 |
| *** ooolpbot has joined #tripleo | 19:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1638350 | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1638646 | 19:10 |
| *** ooolpbot has quit IRC | 19:10 | |
| openstack | Launchpad bug 1638350 in tripleo "pingtest failing on OVB jobs to create Cinder volume and Nova server" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 19:10 |
| openstack | Launchpad bug 1638646 in tripleo "memcache package update breaks memcache starting due to rlimits issue" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 19:10 |
| EmilienM | and it would tell you in logs what doesn't work | 19:10 |
| *** bfournie has quit IRC | 19:10 | |
| EmilienM | mwhahaha: there is a spec also https://review.openstack.org/#/c/372336/ | 19:10 |
| mwhahaha | EmilienM: yea, we'd probably want to create a set of classes for such things so we can just include them. since each service probably needs its own bunch of commands for validation | 19:10 |
| *** Goneri has quit IRC | 19:10 | |
| EmilienM | mwhahaha: we haven't found agreement on the tool | 19:11 |
| bkero | It's almost as if the puppet modules for these services have their own tests already implemented :) | 19:11 |
| EmilienM | mwhahaha: because some folks want it consummable outside puppet run | 19:11 |
| mwhahaha | tool or not, collecting the set of commands to validate services would be a start | 19:11 |
| EmilienM | right | 19:11 |
| EmilienM | memcached is updated | 19:11 |
| EmilienM | let's do recheck | 19:11 |
| bkero | \o/ | 19:11 |
| EmilienM | on a single patch now | 19:11 |
| EmilienM | to see if it works | 19:12 |
| EmilienM | mwhahaha: I did recheck on your puppet-tripleo 392288 patch for heat | 19:12 |
| mwhahaha | k | 19:12 |
| *** bfournie has joined #tripleo | 19:13 | |
| *** mburned is now known as mburned_out | 19:17 | |
| *** mburned_out is now known as mburned | 19:17 | |
| *** bfournie has quit IRC | 19:18 | |
| *** bana_k has quit IRC | 19:18 | |
| *** bana_k has joined #tripleo | 19:20 | |
| *** jerrygb has quit IRC | 19:21 | |
| *** jerrygb has joined #tripleo | 19:22 | |
| *** ealcaniz has quit IRC | 19:22 | |
| bkero | EmilienM++ | 19:23 |
| bkero | mwhahaha++ | 19:23 |
| bkero | The puppet-tripleo CI seems to be passing, just waiting for multinode now, and the OVB. | 19:25 |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci: Workaround: Set selinux to permissive to workaround bug LP#1637961 https://review.openstack.org/392703 | 19:25 |
| *** jerrygb has quit IRC | 19:26 | |
| openstackgerrit | James Slagle proposed openstack/python-tripleoclient: Fix nodes count check on stack-update with custom roles https://review.openstack.org/392313 | 19:30 |
| *** dhill_ has joined #tripleo | 19:33 | |
| weshay | panda|bbl, this change is for the ping test? https://review.openstack.org/#/c/392703/4/scripts/tripleo.sh | 19:33 |
| *** ayoung has joined #tripleo | 19:34 | |
| *** akrivoka has quit IRC | 19:34 | |
| *** nyechiel has joined #tripleo | 19:36 | |
| *** liverpooler has quit IRC | 19:37 | |
| panda|bbl | weshay: yes | 19:40 |
| *** jerrygb has joined #tripleo | 19:42 | |
| *** rcernin has joined #tripleo | 19:45 | |
| weshay | panda|bbl, this recreates in devmode :) | 19:46 |
| *** oshvartz has joined #tripleo | 19:48 | |
| *** bfournie has joined #tripleo | 19:52 | |
| *** jeckersb is now known as jeckersb_gone | 19:56 | |
| *** ealcaniz has joined #tripleo | 19:57 | |
| *** openstack has joined #tripleo | 19:58 | |
| *** chandankumar has quit IRC | 19:59 | |
| *** yolanda has quit IRC | 19:59 | |
| openstackgerrit | James Slagle proposed openstack/python-tripleoclient: Fix nodes count check on stack-update with custom roles https://review.openstack.org/392313 | 20:04 |
| *** Goneri has joined #tripleo | 20:04 | |
| openstackgerrit | Pierre Blanc proposed openstack/os-net-config: Add support for ethtool_opts option https://review.openstack.org/392697 | 20:04 |
| *** dprince has quit IRC | 20:08 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/python-tripleoclient: Updated from global requirements https://review.openstack.org/375125 | 20:09 |
| *** ooolpbot has joined #tripleo | 20:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1638350 | 20:10 |
| openstack | Launchpad bug 1638350 in tripleo "pingtest failing on OVB jobs to create Cinder volume and Nova server" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1638646 | 20:10 |
| *** ooolpbot has quit IRC | 20:10 | |
| openstack | Launchpad bug 1638646 in tripleo "memcache package update breaks memcache starting due to rlimits issue" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 20:10 |
| *** nyechiel has quit IRC | 20:13 | |
| *** panda|bbl is now known as panda | 20:15 | |
| *** jtomasek has quit IRC | 20:17 | |
| openstackgerrit | Pierre Blanc proposed openstack/os-net-config: Add support for ethtool_opts option https://review.openstack.org/392697 | 20:17 |
| *** openstackgerrit has quit IRC | 20:18 | |
| *** openstackgerrit has joined #tripleo | 20:18 | |
| openstackgerrit | James Slagle proposed openstack/python-tripleoclient: Fix nodes count check on stack-update with custom roles https://review.openstack.org/392313 | 20:19 |
| *** jtomasek has joined #tripleo | 20:21 | |
| *** akshai has joined #tripleo | 20:23 | |
| *** thrash|biab is now known as thrash | 20:25 | |
| *** nyechiel has joined #tripleo | 20:26 | |
| *** akshai_ has joined #tripleo | 20:27 | |
| panda | still getting the plan timeout :( | 20:29 |
| *** akshai has quit IRC | 20:29 | |
| *** jkilpatr has quit IRC | 20:30 | |
| *** ealcaniz has quit IRC | 20:30 | |
| *** jayg is now known as jayg|g0n3 | 20:30 | |
| openstackgerrit | Merged openstack/tripleo-quickstart: Add ci-script for upgrade on baremetal https://review.openstack.org/391309 | 20:32 |
| *** dsariel has joined #tripleo | 20:34 | |
| *** mburned is now known as mburned_out | 20:35 | |
| *** nyechiel has quit IRC | 20:38 | |
| *** jeckersb_gone is now known as jeckersb | 20:44 | |
| *** jkilpatr has joined #tripleo | 20:50 | |
| weshay | panda, let's chat here | 20:52 |
| weshay | panda, this is related and causing ping test to fail as well.. Launchpad bug 1638646 in tripleo "memcache package update breaks memcache starting due to rlimits issue" [ | 20:52 |
| openstack | Launchpad bug 1638646 in tripleo "memcache package update breaks memcache starting due to rlimits issue" [Critical,In progress] https://launchpad.net/bugs/1638646 - Assigned to Emilien Macchi (emilienm) | 20:52 |
| panda | weshay: this is actually causing the process to fail during undercloud install | 20:55 |
| weshay | hrm.. I can't log in w/ heat-admin to the controller or compute | 20:55 |
| panda | I think it's easier to chat ... | 20:56 |
| weshay | on bluejeans? | 20:56 |
| panda | weshay: yep | 20:57 |
| weshay | https://bluejeans.com/u/whayutin/ | 20:57 |
| *** akshai_ has quit IRC | 20:57 | |
| panda | weshay: http://logs.openstack.org/03/392703/4/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/defbe53/console.html#_2016-11-02_20_05_50_515239 | 20:58 |
| *** gfidente has quit IRC | 21:00 | |
| panda | weshay: https://bugs.launchpad.net/tripleo/+bug/1637961 | 21:02 |
| openstack | Launchpad bug 1637961 in tripleo "periodic HA master job pingtest times out" [High,Confirmed] - Assigned to Gabriele Cerami (gcerami) | 21:02 |
| *** ipsecguy_ has joined #tripleo | 21:02 | |
| *** dciabrin has quit IRC | 21:03 | |
| *** akshai has joined #tripleo | 21:03 | |
| *** jtomasek has quit IRC | 21:04 | |
| *** dciabrin has joined #tripleo | 21:05 | |
| openstackgerrit | Merged openstack/diskimage-builder: Updated from global requirements https://review.openstack.org/392721 | 21:06 |
| *** ipsecguy has quit IRC | 21:06 | |
| *** lblanchard has quit IRC | 21:06 | |
| *** ooolpbot has joined #tripleo | 21:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1638350 | 21:10 |
| openstack | Launchpad bug 1638350 in tripleo "pingtest failing on OVB jobs to create Cinder volume and Nova server" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1638646 | 21:10 |
| *** ooolpbot has quit IRC | 21:10 | |
| openstack | Launchpad bug 1638646 in tripleo "memcache package update breaks memcache starting due to rlimits issue" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 21:10 |
| panda | EmilienM: are you seeing gates passing anywhere ? | 21:11 |
| weshay | panda, https://review.openstack.org/#/c/392914/ | 21:12 |
| weshay | panda, http://logs.openstack.org/14/392914/1/check-tripleo/gate-tripleo-ci-centos-7-ovb-nonha/cc65f16/console.html | 21:12 |
| EmilienM | panda: nowhere | 21:13 |
| *** morazi has quit IRC | 21:13 | |
| openstackgerrit | Jon Schlueter proposed openstack/tripleo-common: Fix the validation ssh keys workflow https://review.openstack.org/392971 | 21:15 |
| *** bfournie has quit IRC | 21:19 | |
| panda | EmilienM: ok, escalating the issue. | 21:21 |
| EmilienM | I'm wondering if something works today? | 21:21 |
| EmilienM | did anything had a successful thing today? everything fails :( | 21:21 |
| mwhahaha | EmilienM: https://review.openstack.org/#/c/392288/ at least it passed multinode | 21:24 |
| mwhahaha | so i think the good package should be available now | 21:24 |
| panda | mwhahaha: lucky you/ | 21:24 |
| weshay | panda, https://etherpad.openstack.org/p/delorean_master_current_issues | 21:24 |
| EmilienM | mwhahaha: ok so I'm doing recheck on https://review.openstack.org/#/c/392647/ | 21:24 |
| weshay | issue # 88 | 21:24 |
| EmilienM | and see if it pass | 21:24 |
| weshay | panda, https://gist.github.com/dmsimard/174971182870d03394f3f2223cf59412 | 21:25 |
| EmilienM | slagle: are we ok to land https://review.openstack.org/#/c/392288/ (as we know it fix the problem with heat) and consider https://review.openstack.org/#/c/392647/ as a temporary workaround to bring back ovb jobs (it's failing now but because of the memcached issue we fixed) | 21:26 |
| *** cylopez has quit IRC | 21:27 | |
| weshay | panda, https://cbs.centos.org/koji/buildinfo?buildID=13784 | 21:28 |
| weshay | that package is getting pushed into delorean-deps? | 21:29 |
| mwhahaha | it's part of delorean-ocata-testing | 21:30 |
| weshay | k.. which hopefully finds it's way into delorean deps | 21:31 |
| *** maticue has quit IRC | 21:31 | |
| mwhahaha | i don't think memcached itself is in the deps | 21:31 |
| * mwhahaha shrugs | 21:31 | |
| weshay | or I guess delorean.. is what you are saying. not deps :) | 21:32 |
| openstackgerrit | Ryan Brady proposed openstack/tripleo-common: Sets defaults in swift connection related to retries https://review.openstack.org/389124 | 21:33 |
| dmsimard | mwhahaha: delorean-deps == -testing (buildlogs), and yes, that's where the memcached package is. | 21:36 |
| dmsimard | mwhahaha: https://trunk.rdoproject.org/centos7-master/delorean-deps.repo --> http://buildlogs.centos.org/centos/7/cloud/x86_64/openstack-ocata/ | 21:37 |
| weshay | dmsimard, nice test script :) | 21:37 |
| jidar | hey guys, is anything done in newer builds to limit the ibdata1 from eatting up (in my case 70gigs) and causing undercloud install's to fail due to timeout of the db-sync? | 21:37 |
| dmsimard | weshay: bah, Haikel needed a way to reproduce the issue | 21:37 |
| dmsimard | jidar: 70GB ? It sounds like an issue around tokens/instances not being pruned from the database | 21:38 |
| *** jcoufal has quit IRC | 21:38 | |
| jidar | could be, that's typically not setup as a cron job on install | 21:38 |
| dmsimard | jidar: OpenStack's databases are typically really small | 21:39 |
| EmilienM | mwhahaha: I'm approving https://review.openstack.org/#/c/392288/ to unblock one of our problems | 21:39 |
| dmsimard | jidar: I'm not familiar around if TripleO sets up the necessary crons -- I would hope so but maybe someone else can chime in | 21:39 |
| jidar | dmsimard: they are, in my case the keystone is 392k | 21:39 |
| jidar | but ibdata1 is 73gigs | 21:39 |
| dmsimard | I can't even imagine how the database grew to that size. Could Ceilometer or some other telemetry project store it's data on MySQL ? | 21:40 |
| jidar | doubt it, it's been running for about a year like this | 21:41 |
| dmsimard | If it's that large we're probably not setting innodb_file_per_table :( | 21:41 |
| jidar | heh, I was googling around and found that as a potential solution (after I drop databases) | 21:42 |
| jidar | I might as well just nuke this undercloud | 21:42 |
| dmsimard | yeah, it's been a while since I had a problem like that -- I think the gist of it is that emptying the database won't recover the space used on disk unless you do something like an optimize table on everything and then restart mysql | 21:43 |
| jidar | anybody know off hand if I drop all the databases and re-run a undercloud install will it create them properly? | 21:43 |
| jidar | dmsimard: yep, that's what I gather as well | 21:43 |
| jidar | it's likely so large because I used this box as a lab and did around 1000 overcloud deployments while testing osp7 (from the week it was released) | 21:44 |
| jidar | without the pruning of heat templates from the db | 21:44 |
| *** akshai has quit IRC | 21:46 | |
| *** akshai has joined #tripleo | 21:46 | |
| *** maticue has joined #tripleo | 21:48 | |
| *** kjw3 has quit IRC | 21:50 | |
| panda | I put up a review for setting file_per_table a while ago, but it was left rotting since | 21:53 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Configure civetweb bind address from hostname https://review.openstack.org/390569 | 21:55 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Configure civetweb bind address from hostname https://review.openstack.org/390569 | 21:56 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart: add doc regarding how to console into the libvirt guests https://review.openstack.org/392983 | 21:57 |
| *** morazi has joined #tripleo | 22:01 | |
| EmilienM | panda: so https://review.openstack.org/#/c/392288/ is about to merge | 22:05 |
| EmilienM | panda: https://review.openstack.org/#/c/392647/6 will probably help us in short term | 22:05 |
| EmilienM | but we need to find why cinder in wsgi is failing | 22:05 |
| EmilienM | if it's perf, etc. and how we can address it quickly | 22:06 |
| EmilienM | as the ra thing won't be packaged until 10 days apparently | 22:06 |
| panda | EmilienM: I have this to workaround https://review.openstack.org/392703 | 22:07 |
| EmilienM | panda: have you seen bnemec's comment? | 22:08 |
| panda | EmilienM: yes :( I'll revert to my previous method | 22:08 |
| openstackgerrit | Markos Chandras proposed openstack/diskimage-builder: elements: zypper-minimal: Mount common pseudo filesystems https://review.openstack.org/392986 | 22:08 |
| *** oshvartz has quit IRC | 22:08 | |
| bnemec | panda: Hold on. | 22:08 |
| *** rbrady is now known as rbrady-afk | 22:09 | |
| bnemec | I just left another comment with an alternate method that worked for me locally. | 22:09 |
| jidar | so it doesn't look like any packages provide data inside mariadb, so I have to imagine it's safe to nuke the database and re-do the undercloud install | 22:09 |
| panda | bnemec: thanks, pushing | 22:10 |
| *** ooolpbot has joined #tripleo | 22:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1638350 | 22:10 |
| openstack | Launchpad bug 1638350 in tripleo "pingtest failing on OVB jobs to create Cinder volume and Nova server" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1638646 | 22:10 |
| *** ooolpbot has quit IRC | 22:10 | |
| openstack | Launchpad bug 1638646 in tripleo "memcache package update breaks memcache starting due to rlimits issue" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 22:10 |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci: Workaround: Set selinux to permissive to workaround bug LP#1637961 https://review.openstack.org/392703 | 22:11 |
| mwhahaha | EmilienM: should we close our bug 1638646 since the package was updated? | 22:11 |
| openstack | bug 1638646 in tripleo "memcache package update breaks memcache starting due to rlimits issue" [Critical,In progress] https://launchpad.net/bugs/1638646 - Assigned to Emilien Macchi (emilienm) | 22:11 |
| panda | mwhahaha: I'd like to see it passing first ... | 22:11 |
| mwhahaha | panda: it did? | 22:12 |
| mwhahaha | https://review.openstack.org/#/c/392288/ multinode passed | 22:12 |
| EmilienM | mwhahaha: yes please | 22:12 |
| EmilienM | yeah it's a problem solved | 22:12 |
| EmilienM | well, with a workaround | 22:12 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-docs: Uses Ceph/Hammer repos for Liberty and Mitaka only https://review.openstack.org/392987 | 22:14 |
| openstackgerrit | Merged openstack/puppet-tripleo: Create heat user in keystone profile https://review.openstack.org/392288 | 22:26 |
| openstackgerrit | Pradeep Kilambi proposed openstack/instack-undercloud: Add gnocchi support on undercloud https://review.openstack.org/392992 | 22:26 |
| *** flaper87 has quit IRC | 22:29 | |
| EmilienM | mwhahaha: I'm doing the backport | 22:29 |
| mwhahaha | EmilienM: k i was just looking at it, conflicts :( | 22:29 |
| mwhahaha | just the rabbit_port causing problems | 22:29 |
| EmilienM | mwhahaha: no worries | 22:29 |
| EmilienM | mwhahaha: it shows that something has not been backported though | 22:30 |
| mwhahaha | https://github.com/openstack/puppet-tripleo/commit/597194a4199b34e1389b26f080ae1fefccbca924 | 22:31 |
| mwhahaha | interesting considering that patch had tripleo/rc3 as a topic | 22:32 |
| EmilienM | mwhahaha: https://github.com/openstack/puppet-tripleo/commit/e5f0042e47eb4c2b77348dcec0d3d2730e5d9841 no? | 22:32 |
| mwhahaha | no | 22:32 |
| mwhahaha | rabbit_port not rabbit_hosts | 22:32 |
| EmilienM | mhh ok | 22:33 |
| EmilienM | anyway I also try to backport both and merge conflict | 22:33 |
| mwhahaha | https://review.openstack.org/#/c/386761/ | 22:33 |
| mwhahaha | i think we can get away with backporting both right? or is stable/newton all hosed | 22:34 |
| EmilienM | mwhahaha: yes we can backport both | 22:34 |
| EmilienM | I had merge conflict for all the patches mentionned ^ | 22:34 |
| EmilienM | you wanna try? :D | 22:35 |
| mwhahaha | sure | 22:35 |
| mwhahaha | oh nice that one conflicts too | 22:35 |
| * mwhahaha sighs | 22:35 | |
| EmilienM | lol | 22:36 |
| EmilienM | I guess we missed another backport | 22:36 |
| *** maeca1 has quit IRC | 22:36 | |
| EmilienM | mwhahaha: sounds like https://github.com/openstack/puppet-tripleo/commit/ad4e331917a2da9f2fd8215369736896348244fd | 22:37 |
| mwhahaha | no that's in there | 22:38 |
| EmilienM | right | 22:38 |
| mwhahaha | i think git is lost because of the spaces | 22:38 |
| mwhahaha | it's probably cause it's out of order or soemthing | 22:38 |
| mwhahaha | bah the tests too | 22:39 |
| *** pradk has quit IRC | 22:41 | |
| *** Goneri has quit IRC | 22:42 | |
| *** jprovazn has quit IRC | 22:43 | |
| mwhahaha | man the internal tls stuff is in keystone too. guess i'll have to workaround that one | 22:45 |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-tripleo: Make service profiles default to rabbitmq_node_ips https://review.openstack.org/392994 | 22:45 |
| EmilienM | this rebase is a mess | 22:47 |
| openstackgerrit | gyani pillala proposed openstack/tripleo-heat-templates: VSA Storage HP LeftHand Driver Integration code with tripleo Director template files https://review.openstack.org/392995 | 22:47 |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-tripleo: Add port to rabbitmq node ip list https://review.openstack.org/392996 | 22:47 |
| EmilienM | mwhahaha: damn, yeah, tls stuff can be backported, it's a feature i think | 22:48 |
| mwhahaha | EmilienM: i don't think you need 392994 that looks like a bad cp or something | 22:48 |
| EmilienM | damn yeah | 22:48 |
| mwhahaha | this rabbit hole is getting very dark, the tls one ends up touching haproxy which off course conflicts | 22:49 |
| EmilienM | mwhahaha: ok I have merge conflict | 22:50 |
| *** fultonj has quit IRC | 22:50 | |
| EmilienM | I need to leave for a few hours now, get dinner, etc | 22:50 |
| mwhahaha | i've got the rabbit port one and i'll just cp the heat one | 22:50 |
| *** ccamacho|out has quit IRC | 22:50 | |
| EmilienM | ok | 22:51 |
| mwhahaha | if someone wants to touch the tls later, that's their problem. i don't think it should get pulled back to newton | 22:51 |
| EmilienM | thanks | 22:51 |
| EmilienM | I'll catchup later | 22:51 |
| EmilienM | mwhahaha: ++ I agree | 22:51 |
| EmilienM | it's a feature | 22:51 |
| EmilienM | mwhahaha: feel free to backport your patch, I'll approve get it merged asap | 22:51 |
| leifmadsen | so... I'm trying to deploy a small overcloud (baremetal, no IPMI support; just in my local lab). I'm using the fake_pxe driver, and introspection has finished successfully. However, when I go to do an 'openstack overcloud deploy --templates -e deploy.yaml' (deploy.yaml just contains the parameter_defaults for one controller, one compute), I get this message: 2016-11-02 21:54:16Z | 22:53 |
| leifmadsen | [overcloud.Controller.0.Controller]: CREATE_FAILED ResourceInError: resources.Controller: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500" | 22:53 |
| leifmadsen | 22:53 | |
| leifmadsen | I'm curious how best to debug why it doesn't think there are any valid hosts found. I've set the profile on the nodes to a controller and a compute, but i suspect it thinks the flavor can't apply to the specs that introspection found maybe? | 22:53 |
| openstackgerrit | gyani pillala proposed openstack/tripleo-heat-templates: Heat templates supporting VSA HP Stroage Backend with LeftHand Driver https://review.openstack.org/392998 | 22:54 |
| *** ipsecguy_ has quit IRC | 22:54 | |
| *** ipsecguy has joined #tripleo | 22:55 | |
| mwhahaha | leifmadsen: usually that's a miss match between the baremetal flavor and what you have available from the introspection | 22:58 |
| leifmadsen | mwhahaha: yea, that's what I figured, but I'm not seeing something obvious I guess when looking at the default flavor and the baremetal values | 22:58 |
| openstackgerrit | Alex Schultz proposed openstack/puppet-tripleo: Add port to rabbitmq node ip list https://review.openstack.org/392996 | 22:59 |
| openstackgerrit | Alex Schultz proposed openstack/puppet-tripleo: Create heat user in keystone profile https://review.openstack.org/393000 | 22:59 |
| leifmadsen | default flavor values seems to be 4G of RAM, 40G disk, and 1CPU... BM have 16G RAM, 110G disk, and 4CPU | 22:59 |
| *** panda is now known as panda|Zz | 23:01 | |
| mwhahaha | leifmadsen: http://tripleo.org/advanced_deployment/profile_matching.html perhaps that might have some useful information | 23:03 |
| *** david-lyle has quit IRC | 23:04 | |
| leifmadsen | mwhahaha: I think I figured it out | 23:07 |
| leifmadsen | I think I assigned capabilities profile:controller | 23:08 |
| leifmadsen | not control | 23:08 |
| mwhahaha | ah that might do it | 23:08 |
| *** absubram has quit IRC | 23:09 | |
| *** ooolpbot has joined #tripleo | 23:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1638350 | 23:10 |
| openstack | Launchpad bug 1638350 in tripleo "pingtest failing on OVB jobs to create Cinder volume and Nova server" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 23:10 |
| *** ooolpbot has quit IRC | 23:10 | |
| *** jaosorior has quit IRC | 23:10 | |
| *** owalsh has quit IRC | 23:14 | |
| *** owalsh has joined #tripleo | 23:15 | |
| leifmadsen | mwhahaha: I knew it was something stupid/obvious :) | 23:15 |
| leifmadsen | just ripped the environment down, spinning back up, hoping that's all it was | 23:16 |
| *** dmacpher-afk has quit IRC | 23:19 | |
| *** mhenkel has quit IRC | 23:21 | |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Configure civetweb bind address from hostname https://review.openstack.org/390569 | 23:22 |
| *** david-lyle has joined #tripleo | 23:30 | |
| *** rcernin has quit IRC | 23:31 | |
| *** dmacpher-afk has joined #tripleo | 23:33 | |
| *** pmannidi_ has joined #tripleo | 23:36 | |
| *** pmannidi has quit IRC | 23:38 | |
| slagle | EmilienM: so should we recheck this one now? https://review.openstack.org/#/c/392647/ | 23:44 |
| slagle | i did; not sure if that is the last patch needed to fix CI | 23:45 |
| dsneddon | EmilienM, I set some of the work items on this blueprint to POSTPONED. Do you think I can mark it as completed, then submit a new blueprint? https://blueprints.launchpad.net/tripleo/+spec/tripleo-ipv6 | 23:53 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!