| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: POC: WIP: oooq undercloud install https://review.openstack.org/358919 | 00:05 |
|---|---|---|
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: WIP: DONT MERGE Testin OOOQ job https://review.openstack.org/359146 | 00:07 |
| *** [1]cdearborn has quit IRC | 00:07 | |
| *** [1]cdearborn has joined #tripleo | 00:08 | |
| *** ooolpbot has joined #tripleo | 00:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 00:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1616144 | 00:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1616219 | 00:10 |
| *** ooolpbot has quit IRC | 00:10 | |
| openstack | Launchpad bug 1616144 in tripleo "tripleo-ci: nonha jobs failing with Unable to establish connection to https://192.0.2.2:13004/v1/a90407df1e7f4f80a38a1b1671ced2ff/stacks/overcloud/f9f6f712-8e89-4ea9-a34b-6084dc74b5c1" [Critical,Triaged] | 00:10 |
| openstack | Launchpad bug 1616219 in tripleo "latest puppetlabs_spec_helper now requires ruby 2.2" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 00:10 |
| *** bfournie has joined #tripleo | 00:25 | |
| *** limao has joined #tripleo | 00:41 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: Revert "Pin puppetlabs_spec_helper to 1.1.1" https://review.openstack.org/359495 | 00:59 |
| *** ooolpbot has joined #tripleo | 01:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1616144 | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1616219 | 01:10 |
| *** ooolpbot has quit IRC | 01:10 | |
| openstack | Launchpad bug 1616144 in tripleo "tripleo-ci: nonha jobs failing with Unable to establish connection to https://192.0.2.2:13004/v1/a90407df1e7f4f80a38a1b1671ced2ff/stacks/overcloud/f9f6f712-8e89-4ea9-a34b-6084dc74b5c1" [Critical,Triaged] | 01:10 |
| openstack | Launchpad bug 1616219 in tripleo "latest puppetlabs_spec_helper now requires ruby 2.2" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 01:10 |
| *** nyechiel has joined #tripleo | 01:13 | |
| *** thrash is now known as thrash|g0ne | 01:15 | |
| *** bana_k has quit IRC | 01:18 | |
| *** chlong has joined #tripleo | 01:20 | |
| *** nyechiel has quit IRC | 01:26 | |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Move RabbitMQ settings out of controller.yaml https://review.openstack.org/356640 | 01:29 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Move MySQL settings out of puppet/controller.yaml https://review.openstack.org/356505 | 01:29 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Move Keepalived/HAproxy settings out of controller https://review.openstack.org/357344 | 01:29 |
| *** bana_k has joined #tripleo | 01:36 | |
| *** eggmaster has quit IRC | 01:40 | |
| openstackgerrit | Emilien Macchi proposed openstack/instack-undercloud: Revert "Pin puppetlabs_spec_helper to 1.1.1" https://review.openstack.org/359535 | 01:49 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: Revert "Pin puppetlabs_spec_helper to 1.1.1" https://review.openstack.org/359536 | 01:49 |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-tripleo: Revert "Pin puppetlabs_spec_helper to 1.1.1" https://review.openstack.org/359537 | 01:50 |
| *** saneax is now known as saneax-_-|AFK | 01:51 | |
| *** mlupton has quit IRC | 01:52 | |
| *** limao has quit IRC | 01:53 | |
| *** bana_k has quit IRC | 01:55 | |
| *** yamahata has quit IRC | 01:57 | |
| *** saneax-_-|AFK is now known as saneax | 01:58 | |
| *** limao has joined #tripleo | 02:09 | |
| *** ooolpbot has joined #tripleo | 02:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1616144 | 02:10 |
| *** ooolpbot has quit IRC | 02:10 | |
| openstack | Launchpad bug 1616144 in tripleo "tripleo-ci: nonha jobs failing with Unable to establish connection to https://192.0.2.2:13004/v1/a90407df1e7f4f80a38a1b1671ced2ff/stacks/overcloud/f9f6f712-8e89-4ea9-a34b-6084dc74b5c1" [Critical,Triaged] | 02:10 |
| *** [1]cdearborn has quit IRC | 02:21 | |
| *** limao has quit IRC | 02:24 | |
| *** saneax is now known as saneax-_-|AFK | 02:24 | |
| *** jdob has quit IRC | 02:31 | |
| *** ebalduf has quit IRC | 02:49 | |
| *** Ryjedo_ has joined #tripleo | 02:50 | |
| *** Ryjedo has quit IRC | 02:51 | |
| *** Ryjedo_ is now known as Ryjedo | 02:51 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: Update authtoken parameters to match recent changes https://review.openstack.org/359401 | 02:59 |
| openstackgerrit | Emilien Macchi proposed openstack/instack-undercloud: Update authtoken parameters to match recent changes https://review.openstack.org/359400 | 02:59 |
| *** ooolpbot has joined #tripleo | 03:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1616144 | 03:10 |
| *** ooolpbot has quit IRC | 03:10 | |
| openstack | Launchpad bug 1616144 in tripleo "tripleo-ci: nonha jobs failing with Unable to establish connection to https://192.0.2.2:13004/v1/a90407df1e7f4f80a38a1b1671ced2ff/stacks/overcloud/f9f6f712-8e89-4ea9-a34b-6084dc74b5c1" [Critical,Triaged] | 03:10 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: WIP - Implement undercloud upgrade job - Mitaka -> Newton https://review.openstack.org/346995 | 03:22 |
| *** rajinir has quit IRC | 03:25 | |
| *** penick has quit IRC | 03:32 | |
| *** padkrish has joined #tripleo | 04:00 | |
| *** ooolpbot has joined #tripleo | 04:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1616144 | 04:10 |
| *** ooolpbot has quit IRC | 04:10 | |
| openstack | Launchpad bug 1616144 in tripleo "tripleo-ci: nonha jobs failing with Unable to establish connection to https://192.0.2.2:13004/v1/a90407df1e7f4f80a38a1b1671ced2ff/stacks/overcloud/f9f6f712-8e89-4ea9-a34b-6084dc74b5c1" [Critical,Triaged] | 04:10 |
| *** tzumainn has quit IRC | 04:12 | |
| *** jraju has joined #tripleo | 04:14 | |
| *** jraju has quit IRC | 04:15 | |
| openstackgerrit | Lars Kellogg-Stedman proposed openstack/tripleo-heat-templates: Add fluentd client service https://review.openstack.org/353506 | 04:17 |
| openstackgerrit | Lars Kellogg-Stedman proposed openstack/puppet-tripleo: updates to fluentd support https://review.openstack.org/353507 | 04:17 |
| *** eggmaster has joined #tripleo | 04:20 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Revert "Pin puppetlabs_spec_helper to 1.1.1" https://review.openstack.org/359495 | 04:22 |
| *** mhenkel has joined #tripleo | 04:24 | |
| *** bana_k has joined #tripleo | 04:27 | |
| *** limao has joined #tripleo | 04:28 | |
| *** limao has quit IRC | 04:33 | |
| openstackgerrit | Merged openstack/instack-undercloud: Revert "Pin puppetlabs_spec_helper to 1.1.1" https://review.openstack.org/359535 | 04:38 |
| openstackgerrit | Merged openstack/puppet-tripleo: Revert "Pin puppetlabs_spec_helper to 1.1.1" https://review.openstack.org/359537 | 04:51 |
| *** jaosorior has joined #tripleo | 05:07 | |
| *** jaosorior has quit IRC | 05:09 | |
| *** ooolpbot has joined #tripleo | 05:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1616144 | 05:10 |
| *** ooolpbot has quit IRC | 05:10 | |
| openstack | Launchpad bug 1616144 in tripleo "tripleo-ci: nonha jobs failing with Unable to establish connection to https://192.0.2.2:13004/v1/a90407df1e7f4f80a38a1b1671ced2ff/stacks/overcloud/f9f6f712-8e89-4ea9-a34b-6084dc74b5c1" [Critical,Triaged] | 05:10 |
| *** jaosorior has joined #tripleo | 05:10 | |
| *** noslzzp_ has joined #tripleo | 05:15 | |
| *** noslzzp has quit IRC | 05:17 | |
| *** TSCHAK has quit IRC | 05:19 | |
| *** jstir_____ has quit IRC | 05:28 | |
| *** noslzzp_ has quit IRC | 05:30 | |
| *** limao has joined #tripleo | 05:30 | |
| *** noslzzp has joined #tripleo | 05:33 | |
| *** mpsairam has quit IRC | 05:39 | |
| *** TSCHAK has joined #tripleo | 05:40 | |
| *** coolsvap has joined #tripleo | 05:49 | |
| *** aufi has joined #tripleo | 06:02 | |
| *** ooolpbot has joined #tripleo | 06:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1616144 | 06:10 |
| openstack | Launchpad bug 1616144 in tripleo "tripleo-ci: nonha jobs failing with Unable to establish connection to https://192.0.2.2:13004/v1/a90407df1e7f4f80a38a1b1671ced2ff/stacks/overcloud/f9f6f712-8e89-4ea9-a34b-6084dc74b5c1" [Critical,Triaged] | 06:10 |
| *** ooolpbot has quit IRC | 06:10 | |
| *** r-mibu has quit IRC | 06:12 | |
| *** bana_k has quit IRC | 06:14 | |
| openstackgerrit | Saravanan KR proposed openstack/os-net-config: Supporting numbered nics for DPDK Port and DPDK Bond https://review.openstack.org/359610 | 06:14 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Create entries for overcloud VIPs in /etc/hosts https://review.openstack.org/357765 | 06:17 |
| openstackgerrit | Saravanan KR proposed openstack/os-net-config: Supporting numbered nics for DPDK Port and DPDK Bond https://review.openstack.org/359610 | 06:17 |
| *** hjensas has quit IRC | 06:17 | |
| *** mpsairam has joined #tripleo | 06:19 | |
| openstackgerrit | Saravanan KR proposed openstack/os-net-config: Supporting numbered nics for DPDK Port and DPDK Bond https://review.openstack.org/359610 | 06:19 |
| *** tobias_fiberdata has quit IRC | 06:24 | |
| *** rcernin has joined #tripleo | 06:24 | |
| *** jprovazn has joined #tripleo | 06:24 | |
| *** padkrish has quit IRC | 06:26 | |
| *** tobias_fiberdata has joined #tripleo | 06:27 | |
| *** ccamacho has joined #tripleo | 06:32 | |
| *** r-mibu has joined #tripleo | 06:32 | |
| *** ccamacho has quit IRC | 06:32 | |
| *** ccamacho has joined #tripleo | 06:36 | |
| *** leanderthal|afk has quit IRC | 06:36 | |
| openstackgerrit | Michele Baldessari proposed openstack/tripleo-heat-templates: WIP DO NOT MERGE - Enable the HA deployment by default https://review.openstack.org/359060 | 06:39 |
| *** rain has joined #tripleo | 06:40 | |
| *** rain is now known as leanderthal | 06:40 | |
| *** leanderthal is now known as leanderthal|afk | 06:40 | |
| shadower | jaosorior, marios: hey guys, any chance you could merge this? https://review.openstack.org/#/c/348305/ It's a script for discovering the nodes to run validations against | 06:40 |
| shadower | we've been using this for a year now, just porting it over to the new repo | 06:41 |
| shadower | happy to answer any questions | 06:42 |
| *** bootsha has joined #tripleo | 06:44 | |
| *** fragatina has quit IRC | 06:44 | |
| *** rasca has joined #tripleo | 06:45 | |
| jaosorior | shadower: seems to me that it'll only work with keystone v2 | 06:45 |
| *** fragatina has joined #tripleo | 06:46 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: WIP: DONT MERGE Testin OOOQ job https://review.openstack.org/359146 | 06:47 |
| shadower | jaosorior: ah so we should add a v3 support? Tbh we've only tested it with whatever gets installed by default | 06:50 |
| *** fragatina has quit IRC | 06:51 | |
| jaosorior | shadower: thought that was the goal. To start using v3. That script even attempts to replace the auth-url for v3, yet it keeps giving v2 parameters | 06:53 |
| openstackgerrit | Christian Schwede proposed openstack/tripleo-quickstart: Add objectstorage flavor and extra blockdevices option https://review.openstack.org/359630 | 06:55 |
| shadower | jaosorior: I'm not very familiar with the state of v2 & v3. So should this support both or just v3? I haven't actually written the code, but mandre is busy so I'm taking it over | 06:56 |
| *** tesseract- has joined #tripleo | 06:56 | |
| *** masco has joined #tripleo | 06:59 | |
| *** saneax-_-|AFK is now known as saneax | 06:59 | |
| *** mcornea has joined #tripleo | 06:59 | |
| *** hjensas has joined #tripleo | 07:01 | |
| *** hjensas has quit IRC | 07:01 | |
| *** hjensas has joined #tripleo | 07:01 | |
| jaosorior | shadower: alright, lets do this. If you and mandre have been testing this. Lets merge it (if it works for you guys) and afterwards we can fix the keystone issues. | 07:02 |
| *** tremble has joined #tripleo | 07:02 | |
| *** tremble has joined #tripleo | 07:02 | |
| shadower | jaosorior: all righ, thanks. We've been using this daily for a year (including on newton/master) | 07:02 |
| *** coolsvap is now known as coolsvap_ | 07:09 | |
| *** ooolpbot has joined #tripleo | 07:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1616144 | 07:10 |
| openstack | Launchpad bug 1616144 in tripleo "tripleo-ci: nonha jobs failing with Unable to establish connection to https://192.0.2.2:13004/v1/a90407df1e7f4f80a38a1b1671ced2ff/stacks/overcloud/f9f6f712-8e89-4ea9-a34b-6084dc74b5c1" [Critical,Triaged] | 07:10 |
| *** ooolpbot has quit IRC | 07:10 | |
| jaosorior | shadower: seems you gotta update that requirements.txt | 07:11 |
| shadower | jaosorior: yep, I'm on it | 07:11 |
| *** florianf has joined #tripleo | 07:11 | |
| *** pgadiya has joined #tripleo | 07:15 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Register enabled services via global config https://review.openstack.org/359639 | 07:17 |
| openstackgerrit | Tomas Sedovic proposed openstack/tripleo-validations: Add a dynamic ansible inventory script https://review.openstack.org/348305 | 07:19 |
| *** numans has joined #tripleo | 07:19 | |
| *** ioggstream has joined #tripleo | 07:22 | |
| *** ohamada has joined #tripleo | 07:25 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Register enabled services via global config https://review.openstack.org/359639 | 07:28 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Conditionally expose haproxy endpoints https://review.openstack.org/359645 | 07:28 |
| *** jpich has joined #tripleo | 07:32 | |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui: Fix JSON parsing error when execution does not contain output https://review.openstack.org/359648 | 07:32 |
| openstackgerrit | Merged openstack/tripleo-validations: Add a dynamic ansible inventory script https://review.openstack.org/348305 | 07:35 |
| *** gfidente has joined #tripleo | 07:35 | |
| *** athomas has joined #tripleo | 07:37 | |
| *** hewbrocca-afk is now known as hewbrocca | 07:40 | |
| *** anshul has joined #tripleo | 07:41 | |
| hewbrocca | Good morning every one | 07:41 |
| hewbrocca | everyone | 07:41 |
| hewbrocca | všem | 07:41 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Add hyperconverged-ceph environment to include CephOSD on computes https://review.openstack.org/338113 | 07:44 |
| *** ifarkas_afk is now known as ifarkas | 07:47 | |
| openstackgerrit | Julie Pichon proposed openstack/tripleo-common: Add Mistral action & workflow for root device configuration https://review.openstack.org/355947 | 07:49 |
| openstackgerrit | Julie Pichon proposed openstack/tripleo-common: Add boot option argument to the Node Registration Mistral workflow https://review.openstack.org/348875 | 07:49 |
| openstackgerrit | Julie Pichon proposed openstack/tripleo-common: Add Mistral action for boot configuration https://review.openstack.org/354174 | 07:49 |
| *** abregman has joined #tripleo | 07:51 | |
| *** TSCHAK has quit IRC | 07:54 | |
| jaosorior | gfidente: can we merge this? https://review.openstack.org/#/c/359153/1 :D | 07:58 |
| jaosorior | gfidente: all the commits that depend on that one are all green like this https://review.openstack.org/#/c/332641/15 | 07:58 |
| hewbrocca | merge early, merge often | 07:59 |
| ccamacho | o/ | 08:00 |
| gfidente | jaosorior, yeah in fact I think we can merge the other you pointed and the one above too | 08:02 |
| *** abregman has quit IRC | 08:02 | |
| gfidente | https://review.openstack.org/#/c/350396/ | 08:02 |
| gfidente | even the zaqar one | 08:04 |
| jaosorior | gfidente: zaqar and the refactor one won't merge until the inspector SSL one merges | 08:04 |
| jaosorior | gfidente: they're in a chain | 08:04 |
| gfidente | yes but I'm setting w+1 so they all go in | 08:04 |
| gfidente | when that one does | 08:04 |
| gfidente | and I approved the inspector one too | 08:04 |
| jaosorior | did you? :O | 08:05 |
| gfidente | it's https://review.openstack.org/#/c/332641/15 which misses a vote but I'd happily skip it :) | 08:05 |
| jaosorior | gfidente: well, bnemec and dprince had a +1 for it. It was just the mistral issues that were blocking | 08:05 |
| jaosorior | I mean +2 | 08:05 |
| gfidente | right so let's just merge all 4 | 08:05 |
| jaosorior | aww yeah | 08:06 |
| gfidente | you know there has been an earthquake here tonight | 08:08 |
| *** derekh has joined #tripleo | 08:08 | |
| gfidente | woke up scared :( | 08:08 |
| hewbrocca | get those patches merged before the judgement day! | 08:08 |
| jaosorior | gfidente: dude, apparently in middle Italy the earthquake was pretty bad | 08:09 |
| hewbrocca | seriously, gfidente you and yours OK? | 08:09 |
| gfidente | yeah no damages here | 08:10 |
| ccamacho | gfidente o.O sorry man :( | 08:10 |
| *** ooolpbot has joined #tripleo | 08:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1616144 | 08:10 |
| *** ooolpbot has quit IRC | 08:10 | |
| openstack | Launchpad bug 1616144 in tripleo "tripleo-ci: nonha jobs failing with Unable to establish connection to https://192.0.2.2:13004/v1/a90407df1e7f4f80a38a1b1671ced2ff/stacks/overcloud/f9f6f712-8e89-4ea9-a34b-6084dc74b5c1" [Critical,Triaged] | 08:10 |
| gfidente | it's just that it happened during the night so you realize while you can't do much as by when you're awake and leaving the place it's done already | 08:10 |
| jaosorior | damn dude, that sucks | 08:11 |
| gfidente | yeah scary feeling :( | 08:11 |
| jaosorior | I can imagine | 08:11 |
| hewbrocca | File that under "Things we can't do a damn thing about" | 08:11 |
| jaosorior | good thing not much happened where you're at | 08:11 |
| *** lucas-afk is now known as lucasagomes | 08:12 | |
| ccamacho | gfidente Its good to hear that you are OK man. Do you guys know if anyone else its working from Italy? | 08:12 |
| gfidente | looking at the situation seems like in more central areas things are much worse | 08:13 |
| *** abregman has joined #tripleo | 08:13 | |
| gfidente | but not in the city | 08:13 |
| *** abregman has quit IRC | 08:14 | |
| jaosorior | ccamacho: bandini, but IIRC he's in the north | 08:16 |
| ccamacho | bandini ping! | 08:16 |
| ccamacho | ^ dude are you OK? | 08:17 |
| bandini | jaosorior, ccamacho: I am fine thanks. I am way far from where it happened | 08:18 |
| ccamacho | good to hear man :) | 08:19 |
| *** dbecker has quit IRC | 08:19 | |
| *** abregman has joined #tripleo | 08:19 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Configure keystone endpoints in service profile https://review.openstack.org/359680 | 08:20 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Move service/endpoint configuration to keystone profile https://review.openstack.org/359681 | 08:21 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Configure keystone endpoints in service profile https://review.openstack.org/359680 | 08:22 |
| *** dbecker has joined #tripleo | 08:22 | |
| *** zoli_gone-proxy is now known as zoliXXL | 08:24 | |
| gfidente | fwiw we have an office in rome too but I've seen a map somewhere saying on richter scale intensity went quickly to 4 within few kms, which is much less dangerous | 08:26 |
| *** numans has quit IRC | 08:27 | |
| *** yamahata has joined #tripleo | 08:27 | |
| *** abregman has quit IRC | 08:29 | |
| *** abregman has joined #tripleo | 08:29 | |
| *** pcaruana has joined #tripleo | 08:29 | |
| hewbrocca | gfidente: you are very lucky marios is not here | 08:30 |
| gfidente | wait give me some hints, is it something I wrote in the previous line or earlier? | 08:30 |
| *** zoliXXL is now known as zoli_gone-proxy | 08:37 | |
| *** stendulker has joined #tripleo | 08:38 | |
| hewbrocca | gfidente: he would certainly blame you for the earthquaqe | 08:38 |
| hewbrocca | quake | 08:38 |
| gfidente | I actually texted him yesterday | 08:39 |
| gfidente | but it was earlier in the evening | 08:40 |
| gfidente | but I was honest and I told him I was about to add bugs for when he'll be back | 08:40 |
| *** zoli_gone-proxy is now known as zoliXXL | 08:42 | |
| Jokke_ | morning gfidente \o | 08:43 |
| Jokke_ | and hewbrocca | 08:43 |
| ccamacho | gfidente++ I can also add some bugs if needed :P | 08:43 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Register enabled services via global config https://review.openstack.org/359639 | 08:44 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Move service/endpoint configuration to keystone profile https://review.openstack.org/359681 | 08:44 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Conditionally expose haproxy endpoints https://review.openstack.org/359645 | 08:44 |
| *** electrofelix has joined #tripleo | 08:45 | |
| gfidente | ccamacho, <3 | 08:45 |
| ccamacho | ajaajaj | 08:45 |
| gfidente | Jokke_, morning | 08:49 |
| *** yamahata has quit IRC | 08:50 | |
| gfidente | Jokke_, the manila/cephfs I think needs more mangling in puppet | 08:52 |
| *** milan has quit IRC | 08:52 | |
| Jokke_ | gfidente: what did you find? | 08:55 |
| gfidente | Jokke_, basically the problem is that hieradata only binds to classes, not defined types, so you'll have to get and pass them to ::backend::cephfsnative as it's happening for ::backend::generic | 08:56 |
| hewbrocca | excellent | 08:56 |
| hewbrocca | gfidente: ccamacho if you do create bugs, make sure you assign them to marios | 08:57 |
| ccamacho | sure, np :) | 08:57 |
| gfidente | Jokke_, like it is from line 102 in manila.pp | 08:58 |
| gfidente | it's not nice | 08:58 |
| gfidente | but the purpose for defined types is that you can declare them multiple times | 08:58 |
| Jokke_ | gfidente: you mean in the puppet-tripleo? | 08:59 |
| gfidente | Jokke_, puppet-tripleo yes sorry, pacemaker/manila.pp | 08:59 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Move MySQL settings out of puppet/controller.yaml https://review.openstack.org/356505 | 08:59 |
| gfidente | they don't work well with hiera, I am unsure what are the cases where a defined type work better than a class given class title can be changed, we can ask puppet experts | 09:00 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Move RabbitMQ settings out of controller.yaml https://review.openstack.org/356640 | 09:01 |
| Jokke_ | gfidente: ok, cool. I'm fine with that. Mind to put comment with -1 to the review | 09:01 |
| Jokke_ | gfidente: it has +2 already from Dan so no-one merges it while I try to fix it | 09:02 |
| gfidente | Jokke_, yeah I noticed we can also ping dprince when he is online | 09:03 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Create entries for overcloud VIPs in /etc/hosts https://review.openstack.org/357765 | 09:04 |
| *** paramite has joined #tripleo | 09:05 | |
| *** ooolpbot has joined #tripleo | 09:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1616144 | 09:10 |
| *** ooolpbot has quit IRC | 09:10 | |
| openstack | Launchpad bug 1616144 in tripleo "tripleo-ci: nonha jobs failing with Unable to establish connection to https://192.0.2.2:13004/v1/a90407df1e7f4f80a38a1b1671ced2ff/stacks/overcloud/f9f6f712-8e89-4ea9-a34b-6084dc74b5c1" [Critical,Triaged] | 09:10 |
| *** abregman has quit IRC | 09:12 | |
| *** yolanda has quit IRC | 09:16 | |
| openstackgerrit | Bernard Cafarelli proposed openstack/instack-undercloud: Fix puppet-stack-config lvm devices filter https://review.openstack.org/359724 | 09:17 |
| openstackgerrit | Merged openstack/tripleo-quickstart: Proper rST formatting of README.rst https://review.openstack.org/357307 | 09:18 |
| jaosorior | gfidente: can you check this commit out again? https://review.openstack.org/#/c/356416/8 | 09:19 |
| *** yolanda has joined #tripleo | 09:19 | |
| *** jaosorior is now known as jaosorior_lunch | 09:21 | |
| *** limao has quit IRC | 09:22 | |
| *** abregman has joined #tripleo | 09:26 | |
| *** akrivoka has joined #tripleo | 09:26 | |
| *** sshnaidm|afk is now known as sshnaidm | 09:27 | |
| *** yolanda has quit IRC | 09:28 | |
| *** numans has joined #tripleo | 09:29 | |
| *** yolanda has joined #tripleo | 09:29 | |
| openstackgerrit | Merged openstack/tripleo-quickstart: Convert the full documentation to rST https://review.openstack.org/357248 | 09:30 |
| openstackgerrit | Merged openstack/instack-undercloud: Mistral db-populate depends on inspector's endpoint https://review.openstack.org/359153 | 09:33 |
| openstackgerrit | Michele Baldessari proposed openstack/tripleo-heat-templates: Availability monitoring agents support https://review.openstack.org/254788 | 09:33 |
| openstackgerrit | Merged openstack/instack-undercloud: Enable SSL for ironic inspector https://review.openstack.org/332641 | 09:33 |
| openstackgerrit | Merged openstack/instack-undercloud: Refactor endpoint env variable population https://review.openstack.org/350396 | 09:33 |
| openstackgerrit | Merged openstack/instack-undercloud: Enable TLS for Zaqar API https://review.openstack.org/351576 | 09:34 |
| gfidente | jaosorior_lunch, "let there be merges" | 09:35 |
| *** bootsha has quit IRC | 09:40 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: WIP: DONT MERGE Testin OOOQ job https://review.openstack.org/359146 | 09:43 |
| *** milan has joined #tripleo | 09:44 | |
| openstackgerrit | Attila Darazs proposed openstack/tripleo-quickstart: Update configuration docs https://review.openstack.org/356700 | 09:46 |
| *** numans has quit IRC | 09:46 | |
| *** numans has joined #tripleo | 09:47 | |
| *** tosky has joined #tripleo | 09:50 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Don't trigger mariadb upgrade dump/restore when not needed https://review.openstack.org/359218 | 09:55 |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: DO NOT MERGE - test https://review.openstack.org/359759 | 09:59 |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-heat-templates: Composable Mistral services https://review.openstack.org/323436 | 09:59 |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-heat-templates: Do not merge, checking composable mistral. https://review.openstack.org/352947 | 10:01 |
| *** dtantsur has joined #tripleo | 10:07 | |
| openstackgerrit | Erno Kuvaja proposed openstack/puppet-tripleo: Add Manila CephFS backend to manila class https://review.openstack.org/354047 | 10:08 |
| dtantsur | morning/afternoon folks! my IRC bouncer has been down since yesterday evening, so please repeat any pings. | 10:08 |
| openstackgerrit | Erno Kuvaja proposed openstack/tripleo-heat-templates: Add integration with Manila CephFS Native driver https://review.openstack.org/358525 | 10:09 |
| *** ooolpbot has joined #tripleo | 10:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1616144 | 10:10 |
| *** ooolpbot has quit IRC | 10:10 | |
| openstack | Launchpad bug 1616144 in tripleo "tripleo-ci: nonha jobs failing with Unable to establish connection to https://192.0.2.2:13004/v1/a90407df1e7f4f80a38a1b1671ced2ff/stacks/overcloud/f9f6f712-8e89-4ea9-a34b-6084dc74b5c1" [Critical,Triaged] | 10:10 |
| Jokke_ | gfidente: PS 5 out ... hopefully I got you right there | 10:11 |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui: Mistral Actions error handling https://review.openstack.org/359764 | 10:13 |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: Re-enable use of cached images https://review.openstack.org/359765 | 10:14 |
| *** pkovar has joined #tripleo | 10:18 | |
| *** bootsha has joined #tripleo | 10:20 | |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: DO NOT MERGE - test https://review.openstack.org/359759 | 10:21 |
| shadower | EmilienM: ping (question about packaging tripleo-validations) | 10:28 |
| *** jaosorior_lunch is now known as jaosorior | 10:30 | |
| *** rodrigods has quit IRC | 10:34 | |
| *** rodrigods has joined #tripleo | 10:34 | |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-heat-templates: Fix swap file template for Overcloud nodes https://review.openstack.org/359780 | 10:38 |
| jaosorior | ccamacho: that's strange that it needs sudo. Thought heat would run the softwareconfig commands as root | 10:39 |
| ccamacho | jaosorior, mmm let me re-run it again just to double check. | 10:39 |
| *** florianf has quit IRC | 10:42 | |
| slagle | derekh: hey, i'm trying to figure out why we only 25-30 jobs running in zuul, yet when I look on the overcloud, I see ~ 70 instances ACTIVE | 10:42 |
| derekh | slagle: weird, taking a look | 10:42 |
| *** nyechiel has joined #tripleo | 10:44 | |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Update the 'overcloud deploy --templates' command to use Mistral https://review.openstack.org/342261 | 10:49 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Add 'openstack overcloud plan create' command https://review.openstack.org/359784 | 10:49 |
| *** athomas has quit IRC | 10:49 | |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Add 'openstack overcloud plan create' command https://review.openstack.org/359784 | 10:51 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Update the 'overcloud deploy --templates' command to use Mistral https://review.openstack.org/342261 | 10:51 |
| slagle | derekh: i wasnt sure if the delay was in create-env, but I dont see high load on the controller | 10:51 |
| slagle | and the te-broker didnt look like it was doing much either | 10:51 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Move out of roles any Ceph setting https://review.openstack.org/356657 | 10:52 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/instack-undercloud: Enable SSL for ironic inspector https://review.openstack.org/359787 | 10:52 |
| derekh | slagle: what ever is going on its before the te-broker, zuul would still run 75 jobs even if the te-broker was busted | 10:54 |
| *** athomas has joined #tripleo | 10:54 | |
| *** florianf has joined #tripleo | 10:56 | |
| slagle | pabelanger: any idea why zuul only shows ~ 28 jobs in progress for tripleo? | 10:56 |
| slagle | pabelanger: we have ~70 instances in ACTIVE when I check the cloud | 10:56 |
| slagle | is there some other delay before nodepool starts a job running on an instance? | 10:57 |
| derekh | slagle: pabelanger a bunch of the ACTIVE instances don't have a floating IP , we may be a problem getting a FIP | 10:58 |
| *** mburned is now known as mburned_out | 10:58 | |
| *** d0ugal has quit IRC | 10:59 | |
| *** d0ugal has joined #tripleo | 11:00 | |
| hewbrocca | Turns out this operating an OpenStack Cloud thing is harder than it's cracked up to be | 11:01 |
| slagle | grafana thinks we have a bunch of nodes in building | 11:09 |
| slagle | while they are in fact active | 11:10 |
| *** ooolpbot has joined #tripleo | 11:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1616144 | 11:10 |
| *** ooolpbot has quit IRC | 11:10 | |
| openstack | Launchpad bug 1616144 in tripleo "tripleo-ci: nonha jobs failing with Unable to establish connection to https://192.0.2.2:13004/v1/a90407df1e7f4f80a38a1b1671ced2ff/stacks/overcloud/f9f6f712-8e89-4ea9-a34b-6084dc74b5c1" [Critical,Triaged] | 11:10 |
| *** mburned_out is now known as mburned | 11:11 | |
| openstackgerrit | Tomas Sedovic proposed openstack/tripleo-validations: Validate rabbitmq limits https://review.openstack.org/353970 | 11:11 |
| derekh | slagle: that might be because its waiting on them to have a FIP | 11:11 |
| openstackgerrit | Attila Darazs proposed openstack/tripleo-quickstart: WIP: Fix release image URLs to point to buildlogs https://review.openstack.org/359796 | 11:12 |
| slagle | derekh: could be, but the fip's create very quickly | 11:12 |
| derekh | slagle: yes but 30ish of the ACTIVE instnace don't have a FIP | 11:13 |
| jtomasek_ | d0ugal: who can workflow +1 this one? https://review.openstack.org/#/c/348537/ | 11:13 |
| *** jtomasek_ is now known as jtomasek | 11:13 | |
| *** rhallisey has joined #tripleo | 11:14 | |
| *** ramishra has quit IRC | 11:20 | |
| *** jprovazn has quit IRC | 11:21 | |
| derekh | slagle: I can't see any problem, need somebody to look at the nodepool logs when they're up | 11:21 |
| *** ramishra has joined #tripleo | 11:22 | |
| derekh | slagle: btw do you know the password for that openstackzuul user? | 11:22 |
| derekh | slagle: we should make sure non of its resources were left behind after we switched nodpool back to use the origional user | 11:23 |
| openstackgerrit | mathieu bultel proposed openstack/tripleo-heat-templates: Do not use --timeout for the upgrade init command https://review.openstack.org/359802 | 11:23 |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-heat-templates: Fix check for MariaDB upgrade manual switch off https://review.openstack.org/359803 | 11:24 |
| *** thrash|g0ne is now known as thrash | 11:25 | |
| openstackgerrit | mathieu bultel proposed openstack/tripleo-heat-templates: Do not use --timeout for the upgrade init command https://review.openstack.org/359802 | 11:25 |
| slagle | derekh: yes, they're in openstackzuulrc on the uc | 11:26 |
| slagle | derekh: there are a few instances left behind | 11:26 |
| slagle | i was going to ask pabelanger about deleting them today | 11:26 |
| derekh | slagle: ok | 11:27 |
| d0ugal | jtomasek: I can, but I see I am too late :) | 11:28 |
| slagle | derekh: i assigned a bunch of fip's to the ACTIVE instances | 11:29 |
| slagle | will check if that changes anything | 11:29 |
| *** hjensas has quit IRC | 11:29 | |
| derekh | ack | 11:29 |
| *** bootsha has quit IRC | 11:37 | |
| *** bkopilov has quit IRC | 11:40 | |
| *** bootsha has joined #tripleo | 11:40 | |
| *** ohamada_ has joined #tripleo | 11:41 | |
| *** ohamada has quit IRC | 11:41 | |
| *** jprovazn has joined #tripleo | 11:41 | |
| *** bootsha has quit IRC | 11:43 | |
| *** masco has quit IRC | 11:45 | |
| *** stendulker has quit IRC | 11:45 | |
| *** pkovar has quit IRC | 11:47 | |
| *** ioggstream has quit IRC | 11:47 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/tripleo-docs: [WIP] Documentation for installing and using Ironic in overcloud https://review.openstack.org/354016 | 11:48 |
| openstackgerrit | Dmitry Tantsur proposed openstack/tripleo-docs: [WIP] Document hybrid virtual and bare metal overcloud https://review.openstack.org/359240 | 11:48 |
| *** ohamada_ has quit IRC | 11:48 | |
| *** ohamada_ has joined #tripleo | 11:49 | |
| *** pkovar has joined #tripleo | 11:51 | |
| *** hjensas has joined #tripleo | 11:51 | |
| *** jaosorior has quit IRC | 11:51 | |
| *** jaosorior has joined #tripleo | 11:52 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Register enabled services via global config https://review.openstack.org/359639 | 11:53 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Move service/endpoint configuration to keystone profile https://review.openstack.org/359681 | 11:53 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Conditionally expose haproxy endpoints https://review.openstack.org/359645 | 11:53 |
| *** sshnaidm is now known as sshnaidm|afk | 11:54 | |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Add 'openstack overcloud plan create' command https://review.openstack.org/359784 | 11:55 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Update the 'overcloud deploy --templates' command to use Mistral https://review.openstack.org/342261 | 11:55 |
| *** ohamada_ has quit IRC | 11:57 | |
| *** ohamada_ has joined #tripleo | 11:57 | |
| *** lucasagomes is now known as lucas-hungry | 12:01 | |
| EmilienM | good morning | 12:01 |
| *** jpena is now known as jpena|lunch | 12:01 | |
| *** bkopilov has joined #tripleo | 12:01 | |
| *** morazi has joined #tripleo | 12:03 | |
| ccamacho | morning! | 12:04 |
| jtomasek | d0ugal: so the custom environment which cli adds, it is added to the plan - it is in swift, right? | 12:04 |
| d0ugal | jtomasek: Yeah | 12:04 |
| jaosorior | EmilienM: hey dude, how's it going? | 12:04 |
| EmilienM | jaosorior: very good and you? | 12:04 |
| jtomasek | d0ugal: ok, what is it's path? | 12:05 |
| EmilienM | jaosorior: we promoted puppet CI last night and the spec_helper thing is fixed in tripleo | 12:05 |
| openstackgerrit | Erno Kuvaja proposed openstack/puppet-tripleo: Add Manila CephFS backend to manila class https://review.openstack.org/354047 | 12:05 |
| derekh | ack | 12:05 |
| jaosorior | EmilienM: all good. Got enough coffee in my system :P | 12:05 |
| jaosorior | EmilienM: excellent! | 12:05 |
| d0ugal | jtomasek: user-environment.yaml | 12:05 |
| jaosorior | saw your pinning commits | 12:05 |
| d0ugal | jtomasek: I don't know if we want to change this or not | 12:05 |
| jaosorior | jistr: hey dude, can you check this one out? https://review.openstack.org/#/c/353037/ you had reviewed the puppet part | 12:05 |
| d0ugal | jtomasek: I got a bit confused and then honestly forgot about it | 12:06 |
| d0ugal | jtomasek: I'll speak with shardy about it tomorrow hopefully | 12:06 |
| jistr | jaosorior: ah yea looking, didn't get to it yesterday | 12:06 |
| jtomasek | d0ugal: so the deal with this one https://review.openstack.org/#/c/355598/1/tripleo_common/actions/heat_capabilities.py is: | 12:06 |
| jtomasek | d0ugal: the main requirement is to list also environments which are not included in capabilities map, so for example user-environment.yaml can be also listed | 12:07 |
| jtomasek | the patch does not do entirely that... it adds environments which are in mistral environment if they are not in the capabilities-map | 12:07 |
| jtomasek | d0ugal: that is just half of the work | 12:07 |
| jtomasek | d0ugal: I am going to add this to the review | 12:08 |
| jtomasek | d0ugal: what we need to make it fully functional is know the path where to search for those environments, because it is not possible to distinguish environment from ordinary template currently | 12:08 |
| jaosorior | EmilienM: Hey dude, for the keystone endpoint parts and the composability of haproxy. I ended up coming up with an extra output for the service profiles in t-h-t. It merges a so called "global_settings" map in all-nodes-config. What do you think? https://review.openstack.org/#/c/359639/ | 12:08 |
| EmilienM | jaosorior: great idea! let me a few time to catch-up and start my review time :) | 12:09 |
| EmilienM | jaosorior: it's on my list | 12:09 |
| jtomasek | d0ugal: so I'd favor if the user-environment had path of 'environments/user-environment.yaml' | 12:09 |
| jtomasek | and the patch would list all environments from that directory | 12:09 |
| *** ooolpbot has joined #tripleo | 12:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1616144 | 12:10 |
| *** ooolpbot has quit IRC | 12:10 | |
| openstack | Launchpad bug 1616144 in tripleo "tripleo-ci: nonha jobs failing with Unable to establish connection to https://192.0.2.2:13004/v1/a90407df1e7f4f80a38a1b1671ced2ff/stacks/overcloud/f9f6f712-8e89-4ea9-a34b-6084dc74b5c1" [Critical,Triaged] | 12:10 |
| d0ugal | jtomasek: Nesting it in a directory makes it harder | 12:10 |
| *** egafford has joined #tripleo | 12:10 | |
| d0ugal | jtomasek: because then I need to make all the relative paths go up one level ../ | 12:10 |
| jtomasek | d0ugal: I see | 12:10 |
| gfidente | jistr, pls https://review.openstack.org/#/c/350841/ | 12:10 |
| d0ugal | jtomasek: or that is why I changed it to the top level, but I can try again at some point | 12:10 |
| jtomasek | d0ugal: then the patch needs to explicitly add that single environment to the list | 12:10 |
| d0ugal | jtomasek: currently drowing in tasks tho :/ | 12:10 |
| d0ugal | jtomasek: Makes sense. | 12:11 |
| jtomasek | d0ugal: ok, thanks | 12:11 |
| jaosorior | gfidente: hey dude, I'm reading your commit | 12:12 |
| *** jdob has joined #tripleo | 12:12 | |
| jaosorior | can you explain this part to me a bit? https://review.openstack.org/#/c/350841/6/puppet/controller.yaml | 12:12 |
| gfidente | jaosorior, hey thanks | 12:12 |
| jaosorior | it's a bit cryptic | 12:12 |
| *** dprince has joined #tripleo | 12:12 | |
| jaosorior | mostly cause I lack understanding of swift I guess | 12:12 |
| *** ccamacho is now known as ccamacho|lunch | 12:12 | |
| *** ioggstream has joined #tripleo | 12:13 | |
| gfidente | jaosorior, so it's a bit confusing indeed but basically %PORT% is going to be replaced in puppet, and it has always been so | 12:13 |
| gfidente | jaosorior, though we could only use the local disk /d1 | 12:13 |
| gfidente | now if you give a hash {sdb: {bytes_size: 2048}} | 12:14 |
| gfidente | it'll join r1z1-IP:%PORT%/d1 with r1z1-IP:%PORT%/sdb | 12:14 |
| gfidente | in a list | 12:14 |
| jaosorior | gfidente: ah! I think I get it now. So then now it's gonna be a list, first with /d1, and that will have appended what comes from SwiftRawDisks | 12:14 |
| jaosorior | right | 12:14 |
| jaosorior | ok, that makes sense | 12:14 |
| gfidente | the syntax is just what swift uses | 12:15 |
| *** bfournie has quit IRC | 12:15 | |
| jaosorior | gfidente: yeah, like I mentioned it was mostly difficult cause I lack some swift understanding. But it makes sense. | 12:15 |
| pabelanger | slagle: derekh: yes, this is a bug in shade and FIPs, we believe the issue is fixed and just need to schedule a nodepool reboot | 12:15 |
| gfidente | jaosorior, then swift::storage::disks will consume SwiftRawDisks map and format them | 12:16 |
| *** hrybacki is now known as hrybacki|appt | 12:17 | |
| jistr | gfidente: got some questions :) how come we have the SwiftRawDisks on controller directly? Shouldn't that be handled by composable services nowadays? | 12:18 |
| *** pradk has joined #tripleo | 12:18 | |
| gfidente | jistr, yes we still generate the devices list in role yaml | 12:18 |
| gfidente | but it's WIP, maybe I can actually move that part into service yaml first and adds this on top? | 12:18 |
| jistr | gfidente: i'm fine with either, was just trying to understand what's going on :) | 12:19 |
| gfidente | jistr, but I see your point, let's do things the other way around | 12:19 |
| gfidente | first move it out of there | 12:19 |
| dtantsur | hmm, folks, any quick ideas why the same haproxy configuration would work in non-HA but won't do anything in HA? | 12:20 |
| *** coolsvap_ has quit IRC | 12:21 | |
| jaosorior | dtantsur: got a patch? | 12:22 |
| dtantsur | jaosorior, still https://review.openstack.org/356539 | 12:22 |
| *** aqkhan__ has joined #tripleo | 12:23 | |
| jistr | gfidente: 2nd Q :) how come it's a hash and not a list? What gets generated when passed into the 'r1z1-IP:%PORT%/DEVICE' template? You have an example of {sdb: {}} there, so the for_each will iterate over key/value pairs, and key is sdb, value is {} in that example. How does that work with the `repeat/template` there? | 12:23 |
| jaosorior | dtantsur: is there an issue in CI? Or in your local testing? | 12:23 |
| dtantsur | jaosorior, with non-HA configuration, HAProxy correctly handles Ironic; with HA configuration it just does not listen on the port | 12:23 |
| dtantsur | jaosorior, locally | 12:23 |
| openstackgerrit | Julie Pichon proposed openstack/tripleo-common: Better namespace the list of TripleO actions https://review.openstack.org/357682 | 12:23 |
| gfidente | jistr, it's a hash because swift::storage::disk takes a hash and can set different options on each disk | 12:23 |
| jaosorior | dtantsur: got a machine that's available somewhere? I could take a look | 12:24 |
| dtantsur | jaosorior, right, ironic is not in /etc/haproxy/haproxy.cfg Oo do we configure HA case so differently? | 12:24 |
| gfidente | regarding repeat / for_each behaviour with hashes, I actually pushed a change in heat to do that: https://review.openstack.org/#/c/350822/ | 12:24 |
| dtantsur | jaosorior, I'll have to figure out sharing quickstart machines :) | 12:24 |
| gfidente | jistr, it's one of the depends-on | 12:25 |
| jaosorior | dtantsur: Adding "tripleo::haproxy::ironic: true" as you did in puppet/services/haproxy.pp should be enough | 12:25 |
| jistr | gfidente: oh it iterates over *keys only*... hmm ok... | 12:25 |
| jaosorior | dtantsur: it seems to me like this is an issue unrelated to that patch | 12:25 |
| dtantsur | jaosorior, ok, lemme double-check everything about my environment first | 12:25 |
| gfidente | jistr, yeah | 12:25 |
| jistr | gfidente: hm then i think the t-h-t patch is ok, but tbh i'd -1 your heat patch :D | 12:26 |
| jaosorior | dtantsur: I assume you're trying to use templates from somewhere that you cloned in git, right? | 12:26 |
| gfidente | jistr++ thanks :0 | 12:26 |
| dtantsur | jaosorior, correct: | 12:26 |
| dtantsur | openstack overcloud deploy --templates /home/stack/tripleo-heat-templates/ --compute-flavor compute --control-flavor control --control-scale 3 --ntp-server pool.ntp.org --libvirt-type qemu -e /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml -e /home/stack/tripleo-heat-templates/environments/services/ironic.yaml -e /home/stack/config.yaml | 12:26 |
| *** aqkhan_ has quit IRC | 12:27 | |
| openstackgerrit | Julie Pichon proposed openstack/tripleo-common: Better namespace the list of TripleO actions https://review.openstack.org/357682 | 12:27 |
| jaosorior | dtantsur: get rid of those /usr/share/ parts | 12:27 |
| *** yolanda has quit IRC | 12:27 | |
| dtantsur | ah damn | 12:27 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates: Fix Flavor get_param for Block Storage. https://review.openstack.org/359844 | 12:27 |
| jaosorior | dtantsur: you gotta include everything from templates in your home directory | 12:27 |
| dtantsur | yeah, that's fair | 12:27 |
| gfidente | jistr, though if you look at the heat code, it was actually already behaving that way | 12:27 |
| jaosorior | dtantsur: the ones in /usr/share/ will include the ones in /usr/share | 12:27 |
| gfidente | except for the enforced validation | 12:27 |
| jaosorior | dtantsur: that's why you don't see the changes. | 12:27 |
| dtantsur | thanks jaosorior, you saved my day! I guess I have to tear down the whole thing, right? | 12:28 |
| *** julim has joined #tripleo | 12:28 | |
| *** trozet has quit IRC | 12:28 | |
| jaosorior | dtantsur: you could try just updating it. However, that's a 50/50 I think. But however you see fit | 12:28 |
| dtantsur | ack, I'll go the safe and slow path :) | 12:29 |
| *** yolanda has joined #tripleo | 12:29 | |
| gfidente | jistr, so the change is just overriding the validation to make it pass | 12:29 |
| jaosorior | gfidente: by the way, taht puppet-tripleo patch is being tested in CI here https://review.openstack.org/#/c/357791/ and should only run in master nonha | 12:30 |
| jistr | gfidente: yea i see... still i think when iterating over a map, it should yield k/v pairs or be disabled... i can't recall a programming language where iterating a map would yield something else.. | 12:30 |
| gfidente | jaosorior, you mean https://review.openstack.org/#/c/353037/ ? | 12:31 |
| jaosorior | gfidente: that's the usage in t-h-t. Which also points to the puppet-tripleo patch https://review.openstack.org/#/c/353021/ | 12:31 |
| gfidente | jistr, it's what is happening in itertools.product(*list) | 12:32 |
| openstackgerrit | Christian Schwede proposed openstack/tripleo-quickstart: Add objectstorage flavor and extra blockdevices option https://review.openstack.org/359630 | 12:34 |
| gfidente | jistr, btw let me cleanup the role templates from the rings stuff | 12:34 |
| gfidente | jistr, so the change is contained in service template | 12:35 |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui: Mistral Actions error handling https://review.openstack.org/359764 | 12:35 |
| jistr | gfidente: ok, just thinking that still nothing asserts that itertools.product(*list) is the right thing to implement a for_each function with :) | 12:36 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: Convert overcloud.yaml to support jinja2 templating https://review.openstack.org/315679 | 12:36 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: Move AllNodesDeployments into jinja template loop https://review.openstack.org/337267 | 12:36 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: Move AllNodesValidationDeployments into jinja template loop https://review.openstack.org/337587 | 12:36 |
| gfidente | jistr, yeah I see what you mean with yielding k,v tuples | 12:37 |
| gfidente | AND for_each | 12:38 |
| openstackgerrit | Attila Darazs proposed openstack/tripleo-quickstart: Handle image location based on CI_ENV and REL_TYPE https://review.openstack.org/359796 | 12:39 |
| *** jeckersb is now known as jeckersb_gone | 12:39 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Create entries for overcloud VIPs in /etc/hosts https://review.openstack.org/357765 | 12:40 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Enable usage of FQDNs for the endpoints https://review.openstack.org/356416 | 12:40 |
| chem | I've got a new bug for M/N upgrade, more related to N, but I got during upgrade so .. here it is : https://bugs.launchpad.net/tripleo/+bug/1616453 . It's again a very simple one :) | 12:41 |
| openstack | Launchpad bug 1616453 in tripleo "Deploying with cinder-block-storage fails." [Undecided,In progress] - Assigned to Sofer Athlan-Guyot (sofer-athlan-guyot) | 12:41 |
| *** trozet has joined #tripleo | 12:43 | |
| *** bootsha has joined #tripleo | 12:43 | |
| jaosorior | EmilienM: if you have time can you check this out? https://review.openstack.org/#/c/356416/ | 12:47 |
| openstackgerrit | Justin Kilpatrick proposed openstack/tripleo-quickstart: Add documentation for add-on roles https://review.openstack.org/346733 | 12:47 |
| *** bfournie has joined #tripleo | 12:48 | |
| *** bootsha has quit IRC | 12:48 | |
| EmilienM | jaosorior: sure | 12:48 |
| EmilienM | chem: I'm trying to see how the cinder/flavor thing could happen | 12:49 |
| EmilienM | chem: dprince did that code long time ago and it worked until now? https://review.openstack.org/#/c/148357/ | 12:49 |
| EmilienM | jaosorior: it looks cool, have you tried update? | 12:50 |
| EmilienM | jaosorior: I'm running "check experimental" | 12:51 |
| chem | EmilienM: well I guess no one is using block storage node | 12:51 |
| *** pradk has quit IRC | 12:51 | |
| *** sshnaidm|afk is now known as sshnaidm | 12:52 | |
| chem | EmilienM: look at the flavor get_param in {swift,controler,ceph}-storage-puppet.yaml | 12:52 |
| *** lblanchard has joined #tripleo | 12:52 | |
| EmilienM | chem: right | 12:53 |
| jaosorior | EmilienM: so for that specific commit update won't be an issue, as the functionality for the endpoint map doesn't change much. And actually fixes an issue in the endpoint map, where CLOUDNAME would only expose the fqdn of the external network which wasn't gonna work for internal endpoints | 12:53 |
| chem | EmilienM: and applying the patch I proposed get me out of this error. | 12:53 |
| *** rlandy has joined #tripleo | 12:56 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: WIP: DONT MERGE Testin OOOQ job https://review.openstack.org/359146 | 12:58 |
| *** tzumainn has joined #tripleo | 12:58 | |
| *** lucas-hungry is now known as lucasagomes | 12:59 | |
| jistr | gfidente: i +2'd the swift patch. I think we can land it and do the refactoring afterwards if you don't want to reshuffle the world. | 13:00 |
| *** cdearborn has joined #tripleo | 13:00 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: Implement non-ovb overcloud update job - Newton -> Newton https://review.openstack.org/351330 | 13:01 |
| jistr | gfidente: your choice | 13:01 |
| gfidente | chem, EmilienM the problem was introduced very recently by I2ea673d9bc52967f9b7c25555059b964abf66966 | 13:01 |
| gfidente | never released | 13:01 |
| gfidente | so I think it's worth mentioning this is yet another bug which I am *not* the author of! ;) | 13:02 |
| EmilienM | lol | 13:03 |
| gfidente | but there isn't marios so it wouldn't make sense | 13:03 |
| gfidente | I didn't even review that one guys, just saying ... | 13:04 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Migrate to using osc-lib https://review.openstack.org/335460 | 13:05 |
| *** jpena|lunch is now known as jpena | 13:06 | |
| *** dprince has quit IRC | 13:08 | |
| *** pgadiya has quit IRC | 13:09 | |
| *** dprince has joined #tripleo | 13:09 | |
| *** ooolpbot has joined #tripleo | 13:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1616144 | 13:10 |
| *** ooolpbot has quit IRC | 13:10 | |
| openstack | Launchpad bug 1616144 in tripleo "tripleo-ci: nonha jobs failing with Unable to establish connection to https://192.0.2.2:13004/v1/a90407df1e7f4f80a38a1b1671ced2ff/stacks/overcloud/f9f6f712-8e89-4ea9-a34b-6084dc74b5c1" [Critical,Triaged] | 13:10 |
| dtantsur | jaosorior, thanks, now I got ironic working | 13:11 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates: M/N upgrade fail to restart nova-scheduler. https://review.openstack.org/358006 | 13:13 |
| gfidente | so I wanted to leave this to you guys | 13:13 |
| gfidente | https://www.youtube.com/watch?v=cVMnVWZwK8Y | 13:13 |
| gfidente | in true tripleo spirit | 13:13 |
| gfidente | though without marios probably only jistr will understand | 13:13 |
| *** bootsha has joined #tripleo | 13:14 | |
| openstackgerrit | Emilien Macchi proposed openstack/instack-undercloud: Update authtoken parameters to match recent changes https://review.openstack.org/359400 | 13:14 |
| jistr | radio tripleo | 13:14 |
| openstackgerrit | Lars Kellogg-Stedman proposed openstack/tripleo-heat-templates: Add fluentd client service https://review.openstack.org/353506 | 13:15 |
| gfidente | jistr, exactly | 13:15 |
| openstackgerrit | Emilien Macchi proposed openstack/instack-undercloud: Update authtoken parameters to match recent changes https://review.openstack.org/359400 | 13:16 |
| *** asbishop has joined #tripleo | 13:16 | |
| jistr | gfidente: we should put such songs as intermezzos in the calls | 13:17 |
| jistr | gfidente: or better yet, have the calls be the songs, and have meeting talk as intermezzos between the songs | 13:17 |
| gfidente | ahahaha | 13:17 |
| openstackgerrit | Emilien Macchi proposed openstack/instack-undercloud: Update authtoken parameters to match recent changes https://review.openstack.org/359400 | 13:19 |
| chem | gfidente: EmilienM thanks for the quick review :) | 13:20 |
| gfidente | chem, ty for fixing it :) | 13:20 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: Update authtoken parameters to match recent changes https://review.openstack.org/359401 | 13:21 |
| chem | gfidente: well, that's kinda the fun part of testing upgrade, so np. | 13:21 |
| EmilienM | pabelanger: when you have a minute, could you look https://review.openstack.org/#/c/356675/ please? it's a start of tripleo scenario jobs | 13:23 |
| social | EmilienM: I really hope it's close to final form this time https://review.openstack.org/#/c/354125 | 13:24 |
| slagle | derekh: well we got a lot more jobs running now by manually assigning floating ips | 13:25 |
| *** pgadiya has joined #tripleo | 13:25 | |
| slagle | i'm not sure what is supposed to do that automatically and why it takes so long | 13:25 |
| hewbrocca | slagle: wooo | 13:25 |
| pabelanger | slagle: manually? | 13:25 |
| derekh | slagle: cool, nodepool should have done it, I'm surprised it worked tbh | 13:26 |
| pabelanger | ya, we need to restart nodepool to pick up shade fixes | 13:26 |
| jaosorior | dtantsur: nice! | 13:26 |
| *** openstackgerrit has quit IRC | 13:26 | |
| *** openstackgerrit has joined #tripleo | 13:26 | |
| pabelanger | derekh: is there any ipv6 where tripleo-test-cloud-rh1 is? | 13:27 |
| slagle | pabelanger: yes i assigned them manually | 13:28 |
| derekh | pabelanger: I don't know, zoliXXL ^ any idea? | 13:28 |
| slagle | pabelanger: instnaces were just sitting there in ACTIVE with no floating ip | 13:28 |
| slagle | and we only had 28 jobs running | 13:28 |
| slagle | with a bunch of instances doing nothing | 13:28 |
| slagle | will have to chcek what happens after the current batch of jobs finish | 13:29 |
| zoliXXL | pabelanger, don't know, I would have to ask around | 13:29 |
| slagle | right now we have all floating ips assigned (75) to ACTIVE instances | 13:29 |
| slagle | and the in progress jobs has gone up from 32 to 48 | 13:30 |
| slagle | still dont know why it's not going higher than that | 13:30 |
| *** maeca1 has joined #tripleo | 13:32 | |
| zoliXXL | slagle - there should be IPv6 connectivity in PHX | 13:32 |
| *** jeckersb_gone is now known as jeckersb | 13:33 | |
| slagle | ok? | 13:33 |
| slagle | what is that related to? | 13:33 |
| zoliXXL | slagle, sorry - that was for pabelanger | 13:33 |
| slagle | oh ok... | 13:33 |
| slagle | thought it was something about the floating ip problem | 13:34 |
| pabelanger | well, it is. If we have ipv6, we don't need FIPs any more | 13:34 |
| pabelanger | we're down to 2 clouds today using FIPs, bluebox and tripleo-test-cloud-rh1 | 13:34 |
| pabelanger | osic-cloud1 was FIPs, but we just migrated to IPv6 only | 13:34 |
| slagle | how do we migrate? | 13:35 |
| zoliXXL | pabelanger, check the document (link sent on other server) | 13:35 |
| * zoliXXL hopes it helps | 13:35 | |
| pabelanger | if the infrastructure for ipv6 is there, we should consider adding it to the cloud | 13:35 |
| slagle | EmilienM: bnemec jistr dprince : is there any topic you'd like to cover for the DD tomorrow? | 13:36 |
| slagle | EmilienM: bnemec jistr dprince : if not, i will cancel again | 13:36 |
| slagle | or bandini perhaps? ^ | 13:37 |
| EmilienM | slagle: not on my side, releases are burning time | 13:37 |
| EmilienM | and I guess it's same problem for others | 13:37 |
| slagle | yea it is for me | 13:37 |
| * jistr passes too | 13:38 | |
| dprince | slagle: I think we are pretty busy right now in general | 13:39 |
| openstackgerrit | Merged openstack/tripleo-common: Change output for GetCapabilitiesAction https://review.openstack.org/348537 | 13:39 |
| dprince | I'd cancel for the N3 push | 13:39 |
| *** zoliXXL is now known as zoli|trng | 13:39 | |
| slagle | ack, wfm | 13:39 |
| *** trozet has quit IRC | 13:40 | |
| bandini | slagle: same for me | 13:40 |
| hewbrocca | hmmm | 13:43 |
| slagle | pabelanger: what changes do we need to make for ipv6? is it just getting an IP range, defining the network, and launching the instances with that network? | 13:44 |
| pabelanger | slagle: more or less. You'd need to create a new IPV6 network in neutron, which all the specific data. We could still use ipv4 for the endpoint API access, I think we do that for osic-cloud1. | 13:46 |
| hewbrocca | that's good, I think we would have to redeploy to get IPV6 APIs | 13:46 |
| pabelanger | your private networks could still be ipv4 too | 13:46 |
| hewbrocca | we could do that if we really needed to... | 13:46 |
| pabelanger | So, once new network is in place, confirming launching a node can pull down the ipv6 address, we'd update nodepool to switch networks | 13:47 |
| pabelanger | We need a dual stack for the AFS mirror, since AFS only support ipv4 | 13:48 |
| pabelanger | but jobs can use ipv6 to connect to apache | 13:48 |
| pabelanger | we can just launch a replacement mirror then rotate out the original via DNS | 13:49 |
| pabelanger | did that yesterday with osic-cloud1 | 13:49 |
| slagle | zoli|trng: do we already have an ipv6 range we can use? | 13:54 |
| slagle | or do we need to request one from someone? | 13:54 |
| *** Yarboa has joined #tripleo | 13:54 | |
| *** Yarboa has left #tripleo | 13:55 | |
| *** florianf has quit IRC | 13:55 | |
| *** pradk has joined #tripleo | 13:59 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: WIP: DONT MERGE Testin OOOQ job https://review.openstack.org/359146 | 14:02 |
| *** maeca1 has quit IRC | 14:02 | |
| d0ugal | thrash: Does the failure on PS3 make any sense to you? https://review.openstack.org/#/c/359784 | 14:04 |
| openstackgerrit | Ryan Brady proposed openstack/tripleo-common: Fixes missing action entry point https://review.openstack.org/359907 | 14:04 |
| d0ugal | thrash: I assume I will get it in PS4 | 14:04 |
| d0ugal | rbrady: Thanks :) | 14:05 |
| *** maeca1 has joined #tripleo | 14:05 | |
| d0ugal | rbrady: I spotted that but then forgot about it | 14:05 |
| openstackgerrit | Robin Cernin proposed openstack/tripleo-validations: Validate RabbitMQ HA policies in clustered environment https://review.openstack.org/358430 | 14:06 |
| rbrady | d0ugal: no worries. I'm in review / cleanup mode now | 14:06 |
| rbrady | d0ugal: I'm trying to keep our tripleo-mistral-api etherpad up to date. if you know of any other workflow related patches, please add them | 14:07 |
| *** yamahata has joined #tripleo | 14:07 | |
| d0ugal | rbrady: I'll check in a bit | 14:08 |
| openstackgerrit | Merged openstack/puppet-tripleo: Add manifests to inject and trust CA certificates https://review.openstack.org/353021 | 14:08 |
| *** florianf has joined #tripleo | 14:09 | |
| *** maeca1 has quit IRC | 14:10 | |
| *** ooolpbot has joined #tripleo | 14:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1616144 | 14:10 |
| *** ooolpbot has quit IRC | 14:10 | |
| openstack | Launchpad bug 1616144 in tripleo "tripleo-ci: nonha jobs failing with Unable to establish connection to https://192.0.2.2:13004/v1/a90407df1e7f4f80a38a1b1671ced2ff/stacks/overcloud/f9f6f712-8e89-4ea9-a34b-6084dc74b5c1" [Critical,Triaged] | 14:10 |
| thrash | d0ugal: looking... | 14:11 |
| *** limao has joined #tripleo | 14:14 | |
| dtantsur | folks, what's the reason for stack update to fail with http://paste.openstack.org/show/563010/ ? | 14:16 |
| thrash | d0ugal: first glance it doesn't make sense. | 14:16 |
| thrash | d0ugal: but let me dig further. | 14:17 |
| dtantsur | I only updated a few puppet variables in ControllerExtraConfig... | 14:17 |
| d0ugal | thrash: :) glad it isn't just me (it works when I tested manually, maybe I should just wait for the PS4 build) | 14:17 |
| dtantsur | I wonder why it even starts to update computes.... | 14:17 |
| *** limao_ has joined #tripleo | 14:17 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Remove support for legacy rh1 https://review.openstack.org/347918 | 14:18 |
| *** limao has quit IRC | 14:18 | |
| dtantsur | in case someone wonders why I'm bashing THT, this is an example... | 14:19 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Use delorean's exit code to decide on retry https://review.openstack.org/348790 | 14:19 |
| sshnaidm | can somebody explain why 15-seconds task in Ironic could take about 36 seconds? http://paste.openstack.org/show/563012/ | 14:20 |
| *** limao has joined #tripleo | 14:20 | |
| dtantsur | sshnaidm, s/ironic/ipmitool/ I guess | 14:21 |
| *** dtrainor has quit IRC | 14:21 | |
| *** dtrainor has joined #tripleo | 14:22 | |
| sshnaidm | dtantsur, I'm not sur | 14:22 |
| sshnaidm | e | 14:22 |
| sshnaidm | dtantsur, ipmitool runs with -R 3 -N 5 - 3*5 = 15 seconds, right? | 14:23 |
| dtantsur | well, provided that it works as expected | 14:23 |
| sshnaidm | dtantsur, maybe ironic scheduled it 20 seconds later | 14:23 |
| dtantsur | I doubt oslo.concurrency has problems calculating the elapsed time | 14:23 |
| *** limao_ has quit IRC | 14:23 | |
| dtantsur | I don't see how ironic is related at all | 14:23 |
| sshnaidm | dtantsur, what service runs ipmitool? | 14:24 |
| dtantsur | ironic-conductor | 14:24 |
| sshnaidm | ok, so could be ironic-conductor related? | 14:24 |
| zoli|trng | slagle, IMO we have to request it from Network team | 14:25 |
| dtantsur | or systemd related :) or linux related. but chances are high that ipmitool just runs longer than it should | 14:25 |
| dtantsur | sshnaidm, 'man ipmitool' suggests the issue: -R 3 means 3 retries, -N 5 means 5 seconds between retries, but nothing specify how long a retry lasts | 14:26 |
| *** ebalduf has joined #tripleo | 14:26 | |
| sshnaidm | dtantsur, does conductor lock host when waiting from ipmitool? | 14:27 |
| dtantsur | sshnaidm, depends on the exact command. starting with liberty or mitaka (don't remember exactly), this particular command is usually run without locks | 14:28 |
| dtantsur | sshnaidm, we log every time we acquire/release lock, so you can grep for these messages | 14:28 |
| sshnaidm | dtantsur, can you take a look? http://paste.openstack.org/show/563014/ | 14:30 |
| openstackgerrit | Emilien Macchi proposed openstack/instack-undercloud: Update authtoken parameters to match recent changes https://review.openstack.org/359400 | 14:30 |
| slagle | zoli|trng: can you put in the request? | 14:30 |
| sshnaidm | dtantsur, I see there 36 sec of lock, does it mean it was locked 36 seconds? | 14:30 |
| zoli|trng | slagle, OK, will do | 14:30 |
| slagle | thanks | 14:30 |
| * zoli|trng expects "happy faces" of the Network folks :) | 14:30 | |
| *** ccamacho|lunch is now known as ccamacho | 14:30 | |
| sshnaidm | dtantsur, because I got exception: NodeLocked: Node ... is locked by host localhost.localdomain, please retry after the current operation is completed | 14:31 |
| *** limao_ has joined #tripleo | 14:31 | |
| *** ayoung has joined #tripleo | 14:31 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: Update authtoken parameters to match recent changes https://review.openstack.org/359401 | 14:32 |
| ayoung | EmilienM, how do I translate this syntax for the Heat template that will become the Hiera data: apache::mod { 'mod_authn_mellon': } | 14:32 |
| openstackgerrit | Michele Baldessari proposed openstack/instack-undercloud: galera and rabbit do not need to be accessible over the network https://review.openstack.org/352484 | 14:32 |
| sshnaidm | and timeout is about 25 seconds | 14:32 |
| ayoung | I've tried about 10 variations. Keep getting syntax/parsing errors | 14:33 |
| EmilienM | ayoung: you can't do it in hiera, you need to do it in puppet-tripleo, in a profile | 14:33 |
| ayoung | EmilienM, ok. I'll find it | 14:33 |
| *** limao has quit IRC | 14:33 | |
| EmilienM | ayoung: probably on keystone profile | 14:33 |
| dtantsur | sshnaidm, it's not impossible. then probably this particular operation was with lock | 14:34 |
| *** bootsha has quit IRC | 14:34 | |
| jaosorior | ayoung: the file you're looking for is this https://github.com/openstack/puppet-tripleo/blob/master/manifests/profile/base/keystone.pp | 14:35 |
| ayoung | jaosorior, but all the modules are in the hiera data | 14:35 |
| ayoung | /usr/share/openstack-tripleo-heat-templates/puppet/controller.yaml | 14:35 |
| *** limao has joined #tripleo | 14:35 | |
| *** limao_ has quit IRC | 14:36 | |
| *** dsariel has joined #tripleo | 14:36 | |
| jaosorior | ayoung: so, you put the parameters to the puppet modules via hieradata. But if you need to include a specific class, you do that in the puppet manifests | 14:36 |
| ayoung | is it just an include | 14:36 |
| sshnaidm | dtantsur, do yo know where is timeout for task is defined? | 14:36 |
| jaosorior | ayoung: yes, just the includes or resource/class definitions | 14:36 |
| jaosorior | ayoung: all the other parameters go through t-h-t as hieradata | 14:37 |
| dtantsur | sshnaidm, for which task? you mean where these -R and -N come from? | 14:37 |
| ayoung | jaosorior, so this needs to happen before the HTTPD server comes up. | 14:37 |
| sshnaidm | dtantsur, no, I mean for all these "power status" tasks | 14:37 |
| d0ugal | Anyone understand this failure? It is beyond me. http://logs.openstack.org/84/359784/4/check/check-osc-plugins/8a2e6f5/console.html | 14:37 |
| jaosorior | ayoung: if you have ordering dependencies that needs to go in the puppet manifests. I can help with that | 14:38 |
| dtantsur | sshnaidm, wait a second, there is a timeout, but what are you trying to achieve? | 14:38 |
| jaosorior | ayoung: mind explaining a bit more what's going on with that include then? | 14:38 |
| sshnaidm | dtantsur, I try to understand how to prevent exceptions of " NodeLocked: Node ... is locked", for example increasing timeout | 14:38 |
| dtantsur | sshnaidm, the timeout for this operation is usually set on the client side | 14:39 |
| ayoung | jaosorior, OK, here is the deal: The code generated by the SAML2 registration is going to inject mod_auth_mellon diectives into the HTTPD config. GOing to distribute those files as a tarbal using Heat artifacts. But the module needs to be loaed before the artifacts are expanded and the HTTPD server comes up, or it will error out | 14:39 |
| *** bootsha has joined #tripleo | 14:39 | |
| ayoung | the files in the tarball should not conflict with anything managed by Puppet, but will create new Routes in the web server. | 14:40 |
| dtantsur | sshnaidm, there are ironic options node_locked_retry_attempts and node_locked_retry_interval which you can try increasing though | 14:40 |
| dtantsur | sshnaidm, but the root cause is probably slow hardware, and in this case it will make the whole thing slower | 14:40 |
| ayoung | jaosorior, so all I am trying to do here is install the mod_auth_mellon RPM and add it to the enabled modules in the HTTPD before the server comes up the first time | 14:41 |
| dtantsur | folks, is there a way to run puppet with modifying a few hieradata variables on a controller? | 14:41 |
| dtantsur | * rerun - after the deployment | 14:41 |
| jaosorior | ayoung: kinda makes sense (I still don't fully understand the tarball part, but that's fine for now) how will we tell the overcloud that it should run that? | 14:41 |
| hewbrocca | dtantsur: that's what a stack update is supposed to do | 14:41 |
| *** milan has quit IRC | 14:41 | |
| ayoung | jaosorior, http://adam.younglogic.com/2016/08/rbac-policy-update-tripleo/ is an example of the tarball thing | 14:42 |
| dtantsur | hewbrocca, do we have any documentation on that? my first attempt failed with a vague error, the second attempt seems stuck | 14:42 |
| jaosorior | ayoung: meaning, are you planning on using a flag that tells the keystone manifest that federation is enabled? | 14:42 |
| *** limao has quit IRC | 14:42 | |
| ayoung | . ./stackrc; ./upload-swift-artifacts openstack-policy.tar.gz | 14:42 |
| dtantsur | (unless updating 2 things in a config file really takes > 20 minutes) | 14:42 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Deploy minimal services in multinode job https://review.openstack.org/355097 | 14:42 |
| ayoung | jaosorior, yeah, I have that working already | 14:42 |
| *** limao has joined #tripleo | 14:42 | |
| ayoung | jaosorior, ah, you mean to trigger whether or not to enable the module? | 14:43 |
| jaosorior | ayoung: yes | 14:43 |
| ayoung | jaosorior, probably eventually. I was just trying to make it work first | 14:43 |
| *** bootsha has quit IRC | 14:43 | |
| ayoung | it would have to be a flag specific to SAML, and we would need another for OIDC | 14:44 |
| thrash | d0ugal: running check_osc_commands.py manually works. | 14:44 |
| ayoung | actually I think gilles has something like that working... | 14:44 |
| rbrady | jtomasek: when I pushed the parameters action patch, it was missing an entry point for the ResetParametersAction. Here's a patch to fix https://review.openstack.org/#/c/359907/ | 14:44 |
| ayoung | he was looking into OIDC | 14:44 |
| sshnaidm | dtantsur, http://paste.openstack.org/show/563017/ I see api errors here about this, but who is "client" here? | 14:44 |
| dtantsur | sshnaidm, there are two timeouts: one short on ironic-conductor side, one longer on ironicclient side (as you see it retries the request) | 14:45 |
| *** limao_ has joined #tripleo | 14:45 | |
| dtantsur | sshnaidm, you can increase either or both; increasing server-side timeouts might have less predictable results | 14:45 |
| d0ugal | thrash: okay, I guess I'll report a bug. I didn't think to try that! | 14:45 |
| openstackgerrit | Lars Kellogg-Stedman proposed openstack/puppet-tripleo: updates to fluentd support https://review.openstack.org/353507 | 14:46 |
| openstackgerrit | Lars Kellogg-Stedman proposed openstack/tripleo-heat-templates: Add fluentd client service https://review.openstack.org/353506 | 14:46 |
| dtantsur | hewbrocca, I guess I'll revert to recommending ssh-ing into controllers and fixing stuff there... | 14:46 |
| sshnaidm | dtantsur, I'd like to increase clients ones, because I see conductor succeeds 10 seconds later, so if we'd wait another 10 secs.. | 14:46 |
| jaosorior | ayoung: so what about updates? if HTTPD is already running, I guess this comes in, gets installed, and it's a matter of restarting HTTPD, right? | 14:46 |
| jaosorior | ayoung: just thinking about the implementation | 14:46 |
| hewbrocca | hmmph | 14:47 |
| hewbrocca | dtantsur: That seems too long | 14:47 |
| hewbrocca | Unfortunately puppet is not fast | 14:47 |
| dtantsur | sshnaidm, these parameters get passed to ironicclient in its constructor, so it depends on how and where it gets created | 14:47 |
| hewbrocca | jistr: any idea why dtantsur stack update dies? | 14:47 |
| ayoung | jaosorior, yes, but that is OK. Say they want to add OIDC, and iw as not there to start, you just need to have the module enabled in the config prior to restarting the server. Adding the directives and adding the module can happen at the same time | 14:48 |
| *** rajinir has joined #tripleo | 14:48 | |
| jaosorior | ayoung: ok, let me finish up a patch and I'll help you out in a bit. | 14:48 |
| ayoung | jaosorior, thanks | 14:49 |
| *** limao has quit IRC | 14:49 | |
| jistr | dtantsur: can you paste `heat resource-list -n3 overcloud | grep -vi complete` ? | 14:49 |
| ayoung | jaosorior, we'll have a third form, too. We will want mod_auth_gssapi and mod_lookup_identity | 14:50 |
| *** akrivoka has quit IRC | 14:50 | |
| ayoung | the tarball will have files like https://github.com/admiyo/rippowam/blob/master/roles/packstack/templates/keystone-federation-ipsilon.conf.j2#L25 in it, and you can seethe mellon directtives | 14:51 |
| sshnaidm | dtantsur, eh.. that's going to be complicated.. | 14:51 |
| *** akrivoka has joined #tripleo | 14:51 | |
| dtantsur | jistr, http://paste.openstack.org/show/563082/ note that it's my second update, the first one failed with http://paste.openstack.org/show/563010/ | 14:51 |
| jaosorior | ayoung: I'm done, lets do this. you wanna discuss this in a call or is IRC fine? | 14:52 |
| *** jeckersb is now known as jeckersb_gone | 14:53 | |
| openstackgerrit | Justin Kilpatrick proposed openstack/tripleo-quickstart: Add documentation for add-on roles https://review.openstack.org/346733 | 14:54 |
| *** abregman has quit IRC | 14:54 | |
| *** limao has joined #tripleo | 14:55 | |
| *** jobewan has joined #tripleo | 14:55 | |
| *** jobewan is now known as jobewan_away | 14:55 | |
| jistr | dtantsur: oof too much Heat... it's nothing i've ever seen before... one could perhaps try to dive in through the heat graph with a series of `resource-list` and `resource-show` commands to decipher what went wrong for the individual failed resources / substacks, but e.g. the NotFound_Remote looks quite weird indeed. I think our Heat folks might be useful for disecting that one perhaps. | 14:55 |
| *** jobewan_away is now known as jobewan | 14:56 | |
| *** limao_ has quit IRC | 14:56 | |
| *** jeckersb_gone is now known as jeckersb | 14:56 | |
| jaosorior | ayoung: ?? | 14:57 |
| ayoung | jaosorior, so, for now, let's just figure out how to hardwire in the modules | 14:57 |
| ayoung | we'll make it configurABLE as a second stp | 14:57 |
| ayoung | step | 14:57 |
| ayoung | nah, here is better | 14:57 |
| jaosorior | ayoung: sure | 14:57 |
| jaosorior | ayoung: this is not useful for you, right? https://github.com/puppetlabs/puppetlabs-apache/blob/master/manifests/mod/auth_mellon.pp | 14:58 |
| ayoung | jaosorior, that should work | 14:59 |
| ayoung | I did n't think I needed an explicit module for it, though | 14:59 |
| ayoung | and that is yet another dependency to pull in, if it is not already shipped | 15:00 |
| *** jistr is now known as jistr|mtg | 15:00 | |
| *** hrybacki|appt is now known as hrybacki | 15:00 | |
| jaosorior | ayoung: it includes ::apache::mod { 'auth_mellon': } and you had written it needed 'mod_authn_mellon' | 15:00 |
| jaosorior | was that a typo? or is it something else? | 15:01 |
| ayoung | typo | 15:01 |
| jaosorior | ok | 15:01 |
| ayoung | that is the right one, and it looks like it is in openstack-puppet | 15:01 |
| ayoung | $ rpmquery -f /usr/share/openstack-puppet/modules/apache/manifests/mod/auth_mellon.pp | 15:01 |
| ayoung | puppet-apache-1.10.0-0.20160818185445.d14e1a8.el7.centos | 15:01 |
| ayoung | er puppet apache | 15:02 |
| ayoung | let me see if it has oidc, too | 15:02 |
| jaosorior | ayoung: we are using puppet-apache, since that's how keystone (and other modules) run over httpd | 15:03 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Move local bind host out of compute.yaml https://review.openstack.org/359940 | 15:03 |
| dprince | jaosorior: hi, did we land your patch to add the VIPs to the endpoint map yet? | 15:03 |
| dprince | jaosorior: yours or gfidente's? | 15:04 |
| * dprince looks | 15:04 | |
| jaosorior | dprince: we didn't, it's still up for review | 15:04 |
| dprince | jaosorior: okay, I need it to get rid of NovaPublicIP in the patch above (https://review.openstack.org/359940) | 15:04 |
| dprince | EmilienM: ^^ | 15:04 |
| ayoung | jaosorior, OK, so I should be able to add in a line for that in /usr/share/openstack-tripleo-heat-templates/puppet/controller.yaml | 15:05 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Optionally include mod auth mellon for apache https://review.openstack.org/359942 | 15:05 |
| hewbrocca | zaneb: around? could you have a look at dtantsur issue? | 15:05 |
| jaosorior | ayoung: that's how the puppet part could look like ^^ | 15:05 |
| hewbrocca | I think it would be worth knowing what is going wrong | 15:05 |
| hewbrocca | or therve ^^^ | 15:05 |
| dtantsur | hewbrocca, zaneb, never mind, I have to kill the environment; too many urgent things to figure out | 15:06 |
| jaosorior | dprince: I think you can already do that with some stuff that gfidente added | 15:06 |
| jaosorior | dprince: let me dig that out | 15:06 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Add 'openstack overcloud plan create' command https://review.openstack.org/359784 | 15:07 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Update the 'overcloud deploy --templates' command to use Mistral https://review.openstack.org/342261 | 15:07 |
| ayoung | jaosorior, OK, let me test that | 15:08 |
| jaosorior | dprince: maybe something like this could help? https://review.openstack.org/#/c/318344/12/puppet/services/cinder-api.yaml where instead of NovaApiNetwork you would use PublicNetwork | 15:08 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Move Keepalived/HAproxy settings out of controller https://review.openstack.org/357344 | 15:08 |
| dprince | jaosorior: I'm already using that. | 15:09 |
| dprince | jaosorior: that is for local bind IPs | 15:09 |
| *** abregman has joined #tripleo | 15:09 | |
| jaosorior | ayoung: you need to pass auth_mellon_enabled: true via t-h-t, and that's it. | 15:09 |
| jaosorior | dprince: I see, so yeah, then you need my parts | 15:09 |
| dprince | jaosorior: we spoke about exposing VIPs to the endpoint map vs adding a new parameter | 15:09 |
| dprince | jaosorior: right, which patch? | 15:09 |
| jaosorior | dprince: which is bundled in this patch https://review.openstack.org/#/c/357765/17 | 15:10 |
| *** ooolpbot has joined #tripleo | 15:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1616144 | 15:10 |
| *** ooolpbot has quit IRC | 15:10 | |
| openstack | Launchpad bug 1616144 in tripleo "tripleo-ci: nonha jobs failing with Unable to establish connection to https://192.0.2.2:13004/v1/a90407df1e7f4f80a38a1b1671ced2ff/stacks/overcloud/f9f6f712-8e89-4ea9-a34b-6084dc74b5c1" [Critical,Triaged] | 15:10 |
| EmilienM | dprince: currently in meeting, will look later | 15:10 |
| jaosorior | dprince: exactly. that one is adding it to the endpoint map. | 15:10 |
| jaosorior | dprince: you think it's fine to add the netipmap like that to the output of endpoint_map? or should it be a parameter then? gfidente's patch for the servicenetmap already merged, so that's a parameter already | 15:14 |
| *** zoli|trng is now known as zoli|gone | 15:18 | |
| *** zoli|gone is now known as zoli_gone-proxy | 15:19 | |
| *** pgadiya has quit IRC | 15:20 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Fix Flavor get_param for Block Storage. https://review.openstack.org/359844 | 15:22 |
| *** yamahata has quit IRC | 15:23 | |
| zaneb | dtantsur: it's likely that updating 2 things in a config file really does take 20 minutes | 15:24 |
| *** jobewan is now known as jobewan_away | 15:24 | |
| *** jobewan_away has quit IRC | 15:25 | |
| dtantsur | oh | 15:25 |
| *** david-lyle_ has joined #tripleo | 15:25 | |
| *** yamahata has joined #tripleo | 15:25 | |
| larsks | I am seeing the gate-tripleo-ci-centos-7-ovb-ha job fail w/ "Could not evaluate: rs.add() failed to add host to replicaset tripleo: replSetReconfig command must be sent to the current replica set primary."... | 15:25 |
| dtantsur | zaneb, do you know what could cause http://paste.openstack.org/show/563010/ ? | 15:26 |
| *** jobewan has joined #tripleo | 15:26 | |
| *** jobewan is now known as jobewan_away | 15:26 | |
| jaosorior | dprince: by the way, we needed a way to have configuration that's global for the overcloud, such as what services are running. This commit introduces it by having a global_settings output in the service profiles and saving that in all-nodes-config https://review.openstack.org/#/c/359639/4 | 15:26 |
| *** david-lyle_ has quit IRC | 15:26 | |
| *** jobewan_away has quit IRC | 15:26 | |
| jaosorior | dprince: the usage of that is for haproxy to know what services it should open the endpoint for, and for keystone to create the endpoints | 15:26 |
| jaosorior | dprince: if you could check it out I would appreciate it | 15:26 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Populate HAProxy endpoint ports from EndpointMap https://review.openstack.org/355366 | 15:27 |
| zaneb | dtantsur: it looks like the Software config with id 719873dc-c177-4fd3-813c-de1683a7bfa7 was not found ;) | 15:27 |
| dtantsur | weeelll.. where did it go? :) | 15:27 |
| *** saneax is now known as saneax-_-|AFK | 15:27 | |
| zaneb | but yeah, I'm not sure how that could come to be deleted | 15:27 |
| *** fragatina has joined #tripleo | 15:31 | |
| *** numans has quit IRC | 15:31 | |
| *** fragatin_ has joined #tripleo | 15:32 | |
| *** aufi has quit IRC | 15:35 | |
| *** dsariel has quit IRC | 15:35 | |
| *** fragatina has quit IRC | 15:35 | |
| larsks | I have a tripleo architecture section: when deploying w/ pacemaker, are the files in puppet/services/pacemaker/... used *in addition to* or *instead of* the corresponding files in puppet/services/? | 15:37 |
| hewbrocca | larsks: bandini might know | 15:38 |
| larsks | hewbrocca: thanks. or maybe one of these folks ^^^ who have been chatting here recently, I hope :) | 15:38 |
| openstackgerrit | Dmitry Tantsur proposed openstack/tripleo-docs: [WIP] Documentation for installing and using Ironic in overcloud https://review.openstack.org/354016 | 15:38 |
| bandini | larsks: instead of | 15:38 |
| larsks | bandini: darn. Thanks! | 15:38 |
| * larsks wanders off to configure All The Other Things. | 15:39 | |
| bandini | larsks: most pacemaker profiles include the base profile and then do other things | 15:39 |
| openstackgerrit | Brent Eagles proposed openstack/tripleo-heat-templates: Set NeutronL3HA to false when deploying DVR https://review.openstack.org/359969 | 15:39 |
| bandini | larsks: my turn :) do we have proxy support in the pipeline for oooq? | 15:40 |
| bandini | for downloading images I mean | 15:40 |
| *** jobewan has joined #tripleo | 15:41 | |
| *** tremble has quit IRC | 15:42 | |
| dtantsur | derekh, do you think we could just merge https://review.openstack.org/356539 ? these bits are not in the gate anyway | 15:42 |
| *** bana_k has joined #tripleo | 15:42 | |
| *** trozet has joined #tripleo | 15:43 | |
| openstackgerrit | Brent Eagles proposed openstack/tripleo-heat-templates: Set NeutronL3HA to false when deploying DVR https://review.openstack.org/359969 | 15:43 |
| larsks | bandini: what sort of proxy support are you asking about? | 15:43 |
| jrist | hey guys just wanted to say thanks for all of the reviews | 15:44 |
| bandini | larsks: when we do a curl to fetch the rdo artifacts I would love to use a proxy, because I am a cheap bastard and my bandwidth is crap ;) | 15:44 |
| larsks | bandini: unfortunately, the bandwidth at the *other* end is also crap (and possibly rate limited right now?), so a proxy might not be much help. But you should open an issue about that if you haven't already, because I'm going to forget all about it due to tripleo eating my brain this week. | 15:45 |
| hewbrocca | mmm brain | 15:46 |
| *** dsariel has joined #tripleo | 15:47 | |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates: Clean up old functions https://review.openstack.org/359979 | 15:48 |
| bandini | larsks: alright, thanks ;) | 15:49 |
| *** penick has joined #tripleo | 15:53 | |
| *** abregman has quit IRC | 15:53 | |
| jaosorior | dtantsur: I tried the documentation you mentioned for building the IPA image | 15:57 |
| jaosorior | dtantsur: buuuut it keeps failing :( | 15:57 |
| *** penick has quit IRC | 15:58 | |
| ayoung | jaosorior, so...it does not look like the mellon RPM installs with that change | 15:59 |
| ayoung | I made the change on the undercloud, so maybe that is not sufficient? | 16:00 |
| ayoung | DO I need to make it in the image? | 16:00 |
| jaosorior | ayoung: the change in the undercloud? what do you mean? | 16:00 |
| dtantsur | jaosorior, the image building? or the deployment in the end? | 16:00 |
| jaosorior | dtantsur: image building | 16:00 |
| jaosorior | ayoung: so the stuff on the undercloud is independent of the stuff on the overcloud. | 16:00 |
| jaosorior | ayoung: to test it you either need to upload that from swift to the overcloud, or rebuild the images | 16:01 |
| jaosorior | dtantsur: I consistently get: Failed to build image "ironic-python-agent" | 16:01 |
| jaosorior | dtantsur: and this is suspicious No package python-hardware-detect available. | 16:02 |
| dtantsur | jaosorior, I really wonder which repositories it tries to use.. | 16:02 |
| jaosorior | dtantsur: no idea | 16:03 |
| jaosorior | :( | 16:03 |
| openstackgerrit | Merged openstack/tripleo-quickstart: Move unique socket generation into quickstart.sh https://review.openstack.org/357900 | 16:05 |
| *** jbadiapa has quit IRC | 16:06 | |
| *** jubapa has quit IRC | 16:06 | |
| jaosorior | ayoung: if you need to rebuild the images you can use this http://tripleo.org/developer/in_progress_review.html | 16:06 |
| jaosorior | ayoung: and then this for the overcloud-full image http://tripleo.org/post_deployment/build_single_image.html | 16:07 |
| jaosorior | ayoung: or remove overcloud-full.qcow2 and then ./tripleo-ci/scripts/tripleo.sh --overcloud-images | 16:08 |
| openstackgerrit | Dmitry Tantsur proposed openstack/tripleo-docs: [WIP] Documentation for installing and using Ironic in overcloud https://review.openstack.org/354016 | 16:08 |
| *** ifarkas is now known as ifarkas_afk | 16:08 | |
| openstackgerrit | Florian Fuchs proposed openstack/tripleo-ui: Use Mistral API for plan deletion https://review.openstack.org/356458 | 16:08 |
| jaosorior | ayoung: that will also rebuild the image you need. But yeah, need to define those exports first. | 16:08 |
| *** jistr|mtg is now known as jistr | 16:08 | |
| *** ooolpbot has joined #tripleo | 16:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1616144 | 16:10 |
| *** ooolpbot has quit IRC | 16:10 | |
| openstack | Launchpad bug 1616144 in tripleo "tripleo-ci: nonha jobs failing with Unable to establish connection to https://192.0.2.2:13004/v1/a90407df1e7f4f80a38a1b1671ced2ff/stacks/overcloud/f9f6f712-8e89-4ea9-a34b-6084dc74b5c1" [Critical,Triaged] | 16:10 |
| *** yamahata has quit IRC | 16:11 | |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart: Add options to ci-script to manage images in OVB https://review.openstack.org/355476 | 16:12 |
| *** mcornea has quit IRC | 16:16 | |
| *** limao has quit IRC | 16:17 | |
| *** hewbrocca is now known as hewbrocca-afk | 16:17 | |
| *** limao has joined #tripleo | 16:18 | |
| ayoung | jaosorior, we need a better way of customizing the puppet manifests. There should be a local directory of files that gets applied on top of the rpms managed files for anything that needs to be customized on the images. | 16:20 |
| *** athomas has quit IRC | 16:21 | |
| *** fragatin_ has quit IRC | 16:21 | |
| jaosorior | ayoung: so this is what shardy does http://hardysteven.blogspot.fi/2016/08/tripleo-deploy-artifacts-and-puppet.html | 16:21 |
| *** dsneddon has quit IRC | 16:23 | |
| *** bana_k has quit IRC | 16:23 | |
| *** limao has quit IRC | 16:23 | |
| openstackgerrit | Lars Kellogg-Stedman proposed openstack/puppet-tripleo: updates to fluentd support https://review.openstack.org/353507 | 16:24 |
| *** florianf has quit IRC | 16:27 | |
| jaosorior | ayoung: I actually modify the image directly (which is not a very clean thing to do) but it's pretty fast compared to rebuilding the whole thing. | 16:27 |
| *** dsneddon has joined #tripleo | 16:27 | |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates: WIP: Ceilometer Wsgi Mitaka->Newton upgrades https://review.openstack.org/360004 | 16:28 |
| *** jpich has quit IRC | 16:29 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: Update authtoken parameters to match recent changes https://review.openstack.org/359401 | 16:29 |
| ayoung | jaosorior, I keep getting Failed: | 16:29 |
| ayoung | haproxy.x86_64 0:1.5.14-3.el7 | 16:29 |
| *** rcernin has quit IRC | 16:29 | |
| ayoung | I have the original files, still. So, yeah, miodifying the image would be good here | 16:29 |
| openstackgerrit | Emilien Macchi proposed openstack/instack-undercloud: Update authtoken parameters to match recent changes https://review.openstack.org/359400 | 16:31 |
| *** yolanda has quit IRC | 16:32 | |
| *** rasca has quit IRC | 16:33 | |
| *** tesseract- has quit IRC | 16:35 | |
| openstackgerrit | Attila Darazs proposed openstack/tripleo-quickstart: WIP: Document Third Party CI and developer usage https://review.openstack.org/360007 | 16:35 |
| *** nyechiel has quit IRC | 16:36 | |
| jaosorior | ayoung: http://jaormx.github.io/2016/modifying-overcloud-images/ this is what I wrote some time ago. But I'm not sure if it's up-to-date. You might need to /opt/stack/puppet-modules/ with /etc/puppet/modules . I'm not sure | 16:38 |
| ayoung | jaosorior, that looks like exactly what I need. Thanks | 16:40 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Move network bind IPs out of compute.yaml https://review.openstack.org/359940 | 16:41 |
| *** derekh has quit IRC | 16:45 | |
| *** anshul has quit IRC | 16:46 | |
| *** jaosorior has quit IRC | 16:47 | |
| openstackgerrit | Attila Darazs proposed openstack/tripleo-quickstart: Handle image location based on CI_ENV and REL_TYPE https://review.openstack.org/359796 | 16:47 |
| openstackgerrit | Attila Darazs proposed openstack/tripleo-quickstart: Handle image location based on CI_ENV and REL_TYPE https://review.openstack.org/359796 | 16:48 |
| *** yamahata has joined #tripleo | 16:52 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: WIP - Implement undercloud upgrade job - Mitaka -> Newton https://review.openstack.org/346995 | 16:52 |
| *** ohamada_ has quit IRC | 16:52 | |
| EmilienM | it seems like tripleo updates job is broken, http://logs.openstack.org/30/351330/10/experimental/gate-tripleo-ci-centos-7-nonha-multinode-updates-nv/e5bc760/console.html#_2016-08-24_14_40_38_806984 | 16:54 |
| EmilienM | UPDATE_FAILED NotFound_Remote: resources[0]: Software config with id 91df76f0-ec0d-4d6c-8ad9-d27ff6a17634 not found | 16:54 |
| EmilienM | have you seen that before? | 16:54 |
| *** dtantsur is now known as dtantsur|afk | 16:55 | |
| * EmilienM filing a bug | 16:56 | |
| *** skramaja has quit IRC | 16:57 | |
| *** pkovar has quit IRC | 17:03 | |
| EmilienM | ftr, the bug is reported here: https://bugs.launchpad.net/heat/+bug/1616550 | 17:04 |
| openstack | Launchpad bug 1616550 in tripleo "Overcloud update is broken: UPDATE_FAILED NotFound_Remote: resources[0]: Software config with id not found" [High,Confirmed] | 17:04 |
| *** gfidente is now known as gfidente|afk | 17:09 | |
| *** ooolpbot has joined #tripleo | 17:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1616144 | 17:10 |
| *** ooolpbot has quit IRC | 17:10 | |
| openstack | Launchpad bug 1616144 in tripleo "tripleo-ci: nonha jobs failing with Unable to establish connection to https://192.0.2.2:13004/v1/a90407df1e7f4f80a38a1b1671ced2ff/stacks/overcloud/f9f6f712-8e89-4ea9-a34b-6084dc74b5c1" [Critical,Triaged] | 17:10 |
| *** jdob has quit IRC | 17:10 | |
| *** jcoufal has joined #tripleo | 17:12 | |
| *** yolanda has joined #tripleo | 17:14 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: Implement non-ovb overcloud update job - Newton -> Newton https://review.openstack.org/351330 | 17:15 |
| *** ioggstream has quit IRC | 17:16 | |
| *** dsariel has quit IRC | 17:17 | |
| *** bana_k has joined #tripleo | 17:18 | |
| dprince | slagle: http://paste.openstack.org/show/563104/ | 17:21 |
| openstackgerrit | Merged openstack/tripleo-ui: Fix JSON parsing error when execution does not contain output https://review.openstack.org/359648 | 17:21 |
| dprince | slagle: didn't you just fix a keystone related Ironic issue? | 17:21 |
| dprince | slagle: I'm hitting a new one now | 17:21 |
| dprince | keystone V3 wins again! | 17:22 |
| *** sshnaidm is now known as sshnaidm|afk | 17:23 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: Implement non-ovb overcloud update job - Newton -> Newton https://review.openstack.org/351330 | 17:24 |
| *** rcernin has joined #tripleo | 17:24 | |
| rook | gfidente lucasagomes do we have a way to have OOO prepare a NVMe ssd for Ceph? | 17:25 |
| rook | the WWN is blank | 17:25 |
| rook | i thought a had a bug on this | 17:26 |
| openstackgerrit | Lars Kellogg-Stedman proposed openstack/puppet-tripleo: updates to fluentd support https://review.openstack.org/353507 | 17:27 |
| lucasagomes | rook, for ceph ? You mean ironic to pick that specific device for deploying the image? | 17:27 |
| lucasagomes | rook, if so, the root device hints may be a bit limited at the moment | 17:28 |
| lucasagomes | I'm currently working on things like: https://review.openstack.org/#/c/348953/ | 17:28 |
| lucasagomes | that might help a bit | 17:28 |
| *** electrofelix has quit IRC | 17:29 | |
| rook | https://gist.github.com/jtaleric/11dd93bcad1d8199f74b65a97d4f9ad1 lucasagomes | 17:29 |
| slagle | dprince: that's not the issue i was seeing (which is fixed) | 17:29 |
| slagle | hopefully | 17:29 |
| rook | this is what we see w/ NVMe & OOO | 17:29 |
| rook | lucasagomes: this is Newton | 17:30 |
| rook | lucasagomes: maybe it doesn't matter | 17:30 |
| rook | maybe I can just tell the journal to use the /dev/nvme0n1 | 17:30 |
| lucasagomes | rook, yeah, if that name is persistent +1 | 17:31 |
| *** lucasagomes is now known as lucas-dinner | 17:35 | |
| slagle | pabelanger: we have some instances still hanging around from the switch over to the zuul user last week that we had to revert | 17:35 |
| slagle | pabelanger: can I delete those? | 17:35 |
| pabelanger | slagle: yes | 17:35 |
| slagle | ok | 17:35 |
| pabelanger | I also need to test the shared network patch again | 17:35 |
| rook | slagle: happen to know how i pass the pg_num when deploying ceph? | 17:45 |
| slagle | rook: no | 17:45 |
| EmilienM | rook: with puppet-ceph hieradata? | 17:46 |
| rook | ye | 17:46 |
| EmilienM | probably in the tht ceph service | 17:46 |
| rook | yes* | 17:46 |
| slagle | try grepping for pg_num in tripleo-heat-templates | 17:46 |
| rook | slagle: what I am doing now... | 17:47 |
| EmilienM | no, in puppet-ceph | 17:47 |
| EmilienM | in puppet ceph, we have osd_pool_default_pg_num | 17:47 |
| rook | right | 17:47 |
| EmilienM | and also pg_num for each pool | 17:47 |
| rook | let me ask a silly question, why isn't this calculated for the user | 17:47 |
| rook | it seems like it could be. | 17:47 |
| EmilienM | I see ceph-mon tht service has it | 17:47 |
| EmilienM | so CephPools is an hash where you can specify pg_num | 17:48 |
| EmilienM | rook: how could it be calculated? | 17:48 |
| *** tosky has quit IRC | 17:49 | |
| zaneb | EmilienM, dtantsur|afk, hewbrocca-afk: found the problem: https://bugs.launchpad.net/tripleo/+bug/1616550/comments/3 | 17:50 |
| openstack | Launchpad bug 1616550 in tripleo "Overcloud update is broken: UPDATE_FAILED NotFound_Remote: resources[0]: Software config with id not found" [High,Confirmed] | 17:50 |
| *** jpena is now known as jpena|away | 17:50 | |
| colonwq | EmilienM, Let me look it up. It is a base-2 math problem. | 17:50 |
| rook | (Target PGs per OSD) * (OSD #) * (%Data) / (size) -- from the Ceph page | 17:50 |
| EmilienM | indeed, it could be calculated | 17:51 |
| EmilienM | zaneb: thanks! | 17:51 |
| colonwq | rook, EmilienM I was just about to past that. | 17:51 |
| pabelanger | Who is Gabriele Cerami IRC nick again> | 17:52 |
| pabelanger | need to better understand https://review.openstack.org/#/c/354098/ | 17:52 |
| pabelanger | as adding a new pipeline is not in the cards | 17:52 |
| EmilienM | pabelanger: he used to be "panda" | 17:52 |
| EmilienM | but offline now, maybe afk or pto | 17:53 |
| rook | anyhoo, EmilienM would i just do : | 17:53 |
| rook | ceph::pool::pgp_num: | 17:53 |
| rook | value: | 17:54 |
| rook | for the time being? | 17:54 |
| rook | under ExtraConfig: | 17:54 |
| EmilienM | override CephPools hash is better | 17:54 |
| rook | EmilienM: that is over my head - what do you mean override CephPools | 17:54 |
| rook | i don't see that referenced in ceph-puppet | 17:55 |
| rook | https://github.com/openstack/tripleo-heat-templates/blob/6c537d2005b8cb5b25f63722376fca08471987ac/puppet/services/ceph-mon.yaml#L43 ok | 17:56 |
| *** nyechiel has joined #tripleo | 17:56 | |
| pabelanger | EmilienM: thanks, do you know the context of 354098? | 17:57 |
| EmilienM | pabelanger: not at all | 17:57 |
| EmilienM | pabelanger: maybe ask weshay | 17:57 |
| colonwq | rook, try this in your enviornment. ceph::profile::params::osd_pool_default_pg_num: NEW_NUMBER | 17:58 |
| pabelanger | EmilienM: thanks | 17:58 |
| rook | colonwq: under extraconfig? | 17:59 |
| rook | colonwq: weird, my ceph-mon.yaml is different... | 17:59 |
| rook | err | 17:59 |
| colonwq | You could put it in your ceph.yaml. I started using a signle file for my RDO/RHSOP install so I have mutated a bit | 18:00 |
| *** nyechiel has quit IRC | 18:01 | |
| rook | ok, i am using the storage-environment.yaml | 18:02 |
| rook | where all my osds are defined as well | 18:03 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Move glance/heat hiera settings to services https://review.openstack.org/360029 | 18:04 |
| colonwq | rook, that should work for you. | 18:06 |
| openstackgerrit | Martin André proposed openstack/tripleo-common: Allow running validation against different plans https://review.openstack.org/318194 | 18:07 |
| openstackgerrit | Martin André proposed openstack/tripleo-common: Workflows to run validations https://review.openstack.org/353899 | 18:07 |
| openstackgerrit | Martin André proposed openstack/tripleo-common: Workflows to load validations https://review.openstack.org/353898 | 18:07 |
| openstackgerrit | Martin André proposed openstack/tripleo-common: Disable retry files for ansible validations https://review.openstack.org/329039 | 18:07 |
| openstackgerrit | Martin André proposed openstack/tripleo-common: Workflow to setup validations https://review.openstack.org/313632 | 18:07 |
| mandre | dprince: ^ | 18:08 |
| *** ooolpbot has joined #tripleo | 18:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1616144 | 18:10 |
| *** ooolpbot has quit IRC | 18:10 | |
| openstack | Launchpad bug 1616144 in tripleo "tripleo-ci: nonha jobs failing with Unable to establish connection to https://192.0.2.2:13004/v1/a90407df1e7f4f80a38a1b1671ced2ff/stacks/overcloud/f9f6f712-8e89-4ea9-a34b-6084dc74b5c1" [Critical,Triaged] | 18:10 |
| mandre | dprince: I'm now storing the keys in a mistral workflow and writing the identity file to disk when running an validation (it gets removed once ansible finishes running) | 18:10 |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates: Set the default gnocchi driver to swift https://review.openstack.org/354385 | 18:13 |
| *** jobewan has quit IRC | 18:19 | |
| *** rhallisey has quit IRC | 18:20 | |
| *** marcusvrn_ has joined #tripleo | 18:20 | |
| *** jdob has joined #tripleo | 18:21 | |
| dprince | mandre: ack, looking | 18:23 |
| *** jdob1 has joined #tripleo | 18:23 | |
| *** jdob has quit IRC | 18:23 | |
| *** mhenkel has quit IRC | 18:25 | |
| weshay | k | 18:31 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: Implement scenario001 CI job https://review.openstack.org/360039 | 18:32 |
| EmilienM | slagle: ^ when you have spare time, please review | 18:33 |
| slagle | pabelanger: so we are basically back in the same situation with the ACTIVE node just sitting idle in the cloud | 18:33 |
| slagle | pabelanger: is there any way to look at nodepool logs to see if something is going on? | 18:33 |
| slagle | grafana says we have 32 instances in Building. i thought that corresponding to the nova build state, but clearly it must not | 18:34 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: Test new scenario001 job https://review.openstack.org/360040 | 18:35 |
| *** myoung is now known as myoung|pto | 18:35 | |
| ansiwen | EmilienM: Hi, I will need you help to get the tempest running on puppet-ec2api. I need to create ec2 credentials like the CI of ec2api is doing here: https://github.com/openstack/ec2-api/blob/master/devstack/create_config#L51 | 18:36 |
| ansiwen | EmilienM: chem recommendet implementing a provider for that. But I wanted to ask you first, what you suggest, and if you might be able to point me to an example. | 18:37 |
| ansiwen | EmilienM: I'm in Toronto atm, btw. ;-) | 18:38 |
| pabelanger | slagle: Yes, we know what the problem and fixes are. We are just waiting to schedule a nodepool restart | 18:39 |
| slagle | pabelanger: ok, cool | 18:39 |
| pabelanger | slagle: until that happens, nodes will stick around for 1 hour, before being deleted, and relaunched | 18:39 |
| *** karthiks is now known as karthiks_afk | 18:42 | |
| openstackgerrit | Ben Swartzlander proposed openstack/tripleo-heat-templates: Add integration with NetApp Manila driver https://review.openstack.org/354019 | 18:43 |
| *** jobewan has joined #tripleo | 18:43 | |
| *** anshul has joined #tripleo | 18:45 | |
| *** fragatina has joined #tripleo | 18:46 | |
| vkmc | hey all, I was wondering, why in the tripleo-heat-templates every option for every driver has a default value and it's copied in 2 different files? e.g. https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/extraconfig/pre_deploy/controller/cinder-netapp.yaml and https://github.com/openstack/tripleo-heat-templates/blob/master/environments/cinder-netapp-config.yaml | 18:48 |
| ayoung | https://review.openstack.org/#/c/359942/1 EmilienM this failed Lint, saying module was reference by rlative name. Should it be ::apache::mod::auth_mellon instead? | 18:50 |
| ayoung | https://review.openstack.org/#/c/359942/1/manifests/profile/base/keystone.pp | 18:50 |
| EmilienM | ayoung: include ::classname | 18:51 |
| EmilienM | :: | 18:51 |
| ayoung | EmilienM, Just to confirm: instead of apache:: it should be ::apache:: | 18:51 |
| ayoung | I thnk you are saying "yes" | 18:51 |
| EmilienM | yes | 18:51 |
| ayoung | THanks | 18:51 |
| dprince | mandre: +2 on your patch. | 18:52 |
| dprince | mandre: nice work on this | 18:52 |
| paramite | marios, dprince, slagle: guys, I think it is obvious, that CI is failing on https://review.openstack.org/#/c/254788/ randomly and the patch is OK, can we get it merged now when Jenkins gave it +1? | 18:54 |
| EmilienM | paramite: nice work! | 18:55 |
| openstackgerrit | Robin Cernin proposed openstack/tripleo-validations: Validate RabbitMQ HA policies in clustered environment https://review.openstack.org/358430 | 18:55 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Move Keepalived/HAproxy settings out of controller https://review.openstack.org/357344 | 18:55 |
| paramite | EmilienM, I'm awful with sensing sarcasm, so not sure if you are joking right now or you are serious :) | 18:56 |
| EmilienM | paramite: I'm very serious | 18:57 |
| EmilienM | paramite: I dropped a question inline, but otherwise it lgtm | 18:57 |
| paramite | ack\ | 18:57 |
| trozet | EmilienM, bnemec: do you mind reviewing this bug fix? https://review.openstack.org/#/c/355987/ | 18:58 |
| *** jcoufal has quit IRC | 18:59 | |
| paramite | gah merge failed :( ... ok, rebase once again | 18:59 |
| EmilienM | trozet: +2 | 18:59 |
| trozet | EmilienM: thanks! | 19:00 |
| dprince | paramite: that patch is in merge conflict? | 19:04 |
| dprince | paramite: oh, you got that too | 19:04 |
| openstackgerrit | Brent Eagles proposed openstack/tripleo-heat-templates: Make OVS ARP responder feature configurable https://review.openstack.org/360053 | 19:04 |
| paramite | yeah, I'm working on it, please stay tuned | 19:04 |
| paramite | dprince, ^ | 19:05 |
| dprince | paramite: yep, it looks reasonablly close to me | 19:05 |
| *** Goneri has joined #tripleo | 19:05 | |
| *** ccamacho has quit IRC | 19:07 | |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Move glance/heat hiera settings to services https://review.openstack.org/360029 | 19:07 |
| openstackgerrit | Martin Mágr proposed openstack/tripleo-heat-templates: Availability monitoring agents support https://review.openstack.org/254788 | 19:07 |
| bswartz | hey tripleo guys -- what is your FFE process? if something doesn't merge by the end of this week, would it make sense to request an FFE before or after the end of the week? | 19:09 |
| dprince | bswartz: probably just wait. shardy will be back next week and I'd really like to pile it on right when he returns | 19:10 |
| *** ooolpbot has joined #tripleo | 19:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1616144 | 19:10 |
| *** ooolpbot has quit IRC | 19:10 | |
| openstack | Launchpad bug 1616144 in tripleo "tripleo-ci: nonha jobs failing with Unable to establish connection to https://192.0.2.2:13004/v1/a90407df1e7f4f80a38a1b1671ced2ff/stacks/overcloud/f9f6f712-8e89-4ea9-a34b-6084dc74b5c1" [Critical,Triaged] | 19:10 |
| bswartz | dprince: is the process similar to other OS projects though? Just send the request to the ML and the core reviewer team approves/denies it? | 19:10 |
| dprince | bswartz: yeah, a similar process | 19:11 |
| dprince | bswartz: so... seriously you could send it sooner or later I think | 19:11 |
| bswartz | okay just confirming -- I'm a bit nervous about this patch merging | 19:11 |
| *** rhallisey has joined #tripleo | 19:14 | |
| *** asbishop has quit IRC | 19:16 | |
| *** asbishop has joined #tripleo | 19:16 | |
| openstackgerrit | Tim Rozet proposed openstack/tripleo-heat-templates: Fixes Neutron metadata parameters in THT https://review.openstack.org/355987 | 19:25 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Move Swift hiera settings into composable services https://review.openstack.org/360058 | 19:27 |
| dprince | EmilienM: swift above it made sense to do both controller.yaml and swift-storage.yaml at the same time | 19:27 |
| dprince | EmilienM: also glance and heat are here: https://review.openstack.org/#/c/360029/ | 19:27 |
| EmilienM | dprince: excellent | 19:28 |
| dprince | EmilienM: trying to chunk services in 2's or 3's to keep it minimal patches | 19:28 |
| dprince | Swift was a bit large though due to the extra swift-base.yaml | 19:28 |
| EmilienM | yeah | 19:29 |
| *** Goneri has quit IRC | 19:30 | |
| openstackgerrit | Lars Kellogg-Stedman proposed openstack/puppet-tripleo: updates to fluentd support https://review.openstack.org/353507 | 19:30 |
| ayoung | EmilienM, sorry to bother you again...but it looks like the upstream puppet module assumes that the binary code is already installed. For CentOS, mod_auth_mellon is a separate RPM. Is that an upstream bug? And, do we havea way to work around those? | 19:39 |
| EmilienM | ayoung: use ensure_packages | 19:40 |
| ayoung | EmilienM, thanks | 19:40 |
| EmilienM | https://github.com/puppetlabs/puppetlabs-stdlib#ensure_packages | 19:41 |
| EmilienM | ensure_packages(['mellon'], {'ensure' => 'present'}) | 19:41 |
| EmilienM | in the if | 19:41 |
| EmilienM | and do: Package['melon'] -> Package['httpd'] | 19:41 |
| EmilienM | so you don't have orchestration problems | 19:42 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Mv keystone, mongo, manila out of controller.yaml https://review.openstack.org/360060 | 19:42 |
| *** akrzos_ has joined #tripleo | 19:42 | |
| *** Goneri has joined #tripleo | 19:43 | |
| akrzos_ | ping I a requirement to run the external network on my computes but when i deployed i saw control plane ips given to my vlan for external network on computes | 19:43 |
| akrzos_ | have* | 19:43 |
| *** jprovazn has quit IRC | 19:43 | |
| ayoung | ensure_packages(['mod_auth_mellon'], {'ensure' => 'present'}) | 19:43 |
| EmilienM | ayoung: yes and also the -> thing, and you're good! | 19:44 |
| ayoung | Package['mod_auth_mellon'] -> Package['httpd'] | 19:44 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart: set up quickstart to replace instack-virt-setup https://review.openstack.org/358089 | 19:44 |
| ayoung | ok, gonna try it | 19:44 |
| EmilienM | ayoung: it should do the job | 19:44 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Fix use of snmp service https://review.openstack.org/360062 | 19:44 |
| rook | slagle i wonder if we should figure out what isn't letting go of fds w/ mariadb... I am hitting this with Newton now (undercloud): ERROR: Remote error: DBError (pymysql.err.InternalError) (23, u"Out of resources when opening file '/tmp/#sql_4ab3_0.MAD' (Errcode: 24)") | 19:45 |
| ayoung | EmilienM, we've def got memory leaks. I'm getting OOM errors after a few redeploys. | 19:45 |
| EmilienM | ayoung: add swap | 19:46 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Fix use of snmp service https://review.openstack.org/360062 | 19:46 |
| ayoung | EmilienM, that is a bandaid. I want to know what is gobbling up the memory. | 19:46 |
| rook | yeah... :( | 19:46 |
| EmilienM | ayoung: i know | 19:46 |
| rook | OOM has been a PITA with the releases so far. | 19:46 |
| slagle | rook: what are the open files limits for the mysql process? | 19:47 |
| *** Goneri has quit IRC | 19:48 | |
| rook | Max open files 1024 16384 files *soft | 19:48 |
| ayoung | http://paste.openstack.org/show/563123/ | 19:48 |
| ayoung | lets see what happens if I restart heat | 19:49 |
| rook | it will releaes that memory ayoung | 19:49 |
| rook | but the next heat command you run, it will gobble it back up | 19:49 |
| ayoung | EmilienM, well...it frees up enough memory to let me run guestfish.... | 19:50 |
| ayoung | rook, yeah, I was seeing it in MySQL. I wonder if there are certain queries that are exploding in Joins or something | 19:51 |
| openstackgerrit | Dmitry Tantsur proposed openstack/tripleo-heat-templates: Ironic: add missing haproxy and firewall configuration https://review.openstack.org/356539 | 19:51 |
| slagle | rook: in my master environment it has 16384 for both soft and hard limit | 19:52 |
| slagle | rook: do you have LimitNOFile=16384 in /etc/systemd/system/mariadb.service.d/limits.conf? | 19:52 |
| slagle | LimitNOFILE | 19:53 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Fix use of snmp service https://review.openstack.org/360062 | 19:57 |
| ayoung | EmilienM, is there anything like tox I can run prior to posting an update for openstack/puppet-tripleo | 19:58 |
| ayoung | get the puppet lint setup? | 19:58 |
| EmilienM | ayoung: http://docs.openstack.org/developer/puppet-openstack-guide/testing.html#unit-tests | 19:58 |
| EmilienM | ah for lint | 19:58 |
| EmilienM | just run bundle exec rake lint | 19:59 |
| ayoung | puppet-lint manifests/profile/base/keystone.pp seems to work, too, and it caught the relative path error | 19:59 |
| *** akrivoka has quit IRC | 20:00 | |
| openstackgerrit | ayoung proposed openstack/puppet-tripleo: Optionally include mod auth mellon for apache https://review.openstack.org/359942 | 20:01 |
| openstackgerrit | Christian Schwede proposed openstack/tripleo-heat-templates: Add system-uuid based hostname entries https://review.openstack.org/358643 | 20:03 |
| *** dprince has quit IRC | 20:09 | |
| *** ooolpbot has joined #tripleo | 20:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1616144 | 20:10 |
| *** ooolpbot has quit IRC | 20:10 | |
| openstack | Launchpad bug 1616144 in tripleo "tripleo-ci: nonha jobs failing with Unable to establish connection to https://192.0.2.2:13004/v1/a90407df1e7f4f80a38a1b1671ced2ff/stacks/overcloud/f9f6f712-8e89-4ea9-a34b-6084dc74b5c1" [Critical,Triaged] | 20:10 |
| *** Goneri has joined #tripleo | 20:16 | |
| openstackgerrit | Merged openstack/os-net-config: Enhance and fix NIC mapping feature https://review.openstack.org/354839 | 20:23 |
| rook | slagle: yup | 20:25 |
| rook | slage - this is the downstream | 20:25 |
| rook | if that matters. | 20:25 |
| *** chem has quit IRC | 20:25 | |
| rook | slagle: another question... how can i determine why a overcloud delete is failing? | 20:27 |
| ayoung | EmilienM, hmmmm. RPM was not installed, but the mellon specific code was executed in httpd. | 20:28 |
| EmilienM | ayoung: do you have the boolean enabled?, | 20:28 |
| *** anshul has quit IRC | 20:29 | |
| ayoung | EmilienM, I did this by hand, so I removed the boolean, but I think I did it in stage 3... | 20:29 |
| ayoung | going to extract it out | 20:29 |
| ayoung | if $step >= 4 or ( $step >= 3 and $sync_db ) { | 20:29 |
| ayoung | that does not look right...too late, I think | 20:29 |
| rook | 2016-08-24 20:29:51.896 128879 INFO heat.engine.stack [req-09cbd4ca-2a19-4490-8d3d-76c7c8efd30a - - - - -] Stack DELETE FAILED (overcloud-Controller-ugijdcwsqtnd-1-jsutr26ykthq): Resource DELETE failed: resources.UpdateConfig: Engine went down during stack DELETE | 20:30 |
| rook | slagle: ^ weird. | 20:30 |
| ayoung | rook, OOM | 20:30 |
| rook | gwad i hope not. | 20:30 |
| rook | this host has like 256GB of ram | 20:30 |
| ayoung | rook that is how it failed for me last time | 20:30 |
| ayoung | look in journalctl | 20:31 |
| slagle | rook: it tells you right there | 20:31 |
| slagle | Engine went down | 20:31 |
| rook | yes slagle what would cause the heat engine to go down | 20:31 |
| rook | fyi it is UP according to systemctl | 20:31 |
| slagle | systemctl stop openstack-heat-engine | 20:31 |
| slagle | that would cause it ^ :) | 20:31 |
| rook | haha | 20:31 |
| rook | well, without stating the obvious. | 20:32 |
| slagle | you're kind of asking the obvious :) | 20:32 |
| slagle | it says engine went down. is it up? if so, was it up the whole time during the delete? is there anything in the logs? | 20:32 |
| rook | yup, it has been up the entire time. | 20:34 |
| rook | https://gist.github.com/jtaleric/f0dcf3108ec9609e3264eed5c1822fa4 slagle | 20:36 |
| slagle | would have to check the heat-engine log | 20:36 |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart: Add options to ci-script to manage images in OVB https://review.openstack.org/355476 | 20:37 |
| slagle | rook: ok, the error doesnt mean anything more to me than what it says | 20:38 |
| rook | https://bugzilla.redhat.com/show_bug.cgi?id=1234607#c3 wonder if that would help me | 20:38 |
| openstack | bugzilla.redhat.com bug 1234607 in openstack-heat "There needs to be a force delete option for Heat stacks" [High,Closed: notabug] - Assigned to zbitter | 20:38 |
| *** limao has joined #tripleo | 20:39 | |
| slagle | zaneb: therve : any thoughts on https://gist.github.com/jtaleric/f0dcf3108ec9609e3264eed5c1822fa4 ? | 20:40 |
| slagle | how can you tell what caused the engine to do down assuming it was oom'd or manually stopped | 20:40 |
| slagle | assuming it *wasn't* | 20:40 |
| rook | whelp that worked. | 20:41 |
| * rook waits for something else to blow up now | 20:41 | |
| rook | if someone wants the abandonded-overcloud.json i have it.. | 20:41 |
| *** bootsha has joined #tripleo | 20:42 | |
| zaneb | slagle: ask systemd? it's almost certainly the OOM killer | 20:42 |
| rook | slage - now the db issue, not sure what the difference is here, but RHEL72, downstream product, i have a 1024 for soft limit. | 20:42 |
| rook | zaneb if this was a virt undercloud maybe. | 20:42 |
| *** morazi has quit IRC | 20:43 | |
| *** limao has quit IRC | 20:43 | |
| rook | zaneb this is a baremetal undercloud, I don't see any references to oom or weighted in the logs... | 20:43 |
| *** nyechiel has joined #tripleo | 20:45 | |
| ayoung | nothing in journalctl? | 20:45 |
| rook | negative ayoung :/ | 20:45 |
| rook | this isn't my first dance with OOM... that SOB has trolled me hard in the past years. | 20:45 |
| ayoung | rook, anything in the log around 2016-08-24 20:35:03.495 | 20:46 |
| zaneb | rook: that message is only recorded when a new heat-engine starts up. and it should only appear for stacks that were IN_PROGRESS and locked by an engine worker that is no longer reachable over RPC | 20:46 |
| rook | zaneb: this deployment fell apart due to mariadb running out of fd | 20:47 |
| ayoung | maybe the delete action improperly reported the enginer as down when it should have been unreachable ? | 20:47 |
| rook | could that cause this weird state. | 20:47 |
| ayoung | rook, AH | 20:47 |
| ayoung | rook, yeah, when I saw the error, it was Maria OOMing | 20:47 |
| ayoung | so the Engine might hang trying to talk to Maria, or somehow report a 500 or something | 20:48 |
| ayoung | well, the RPC equivalent. | 20:48 |
| rook | yeah, mariadb didn't cause OOM, but running out of file descriptors did occure, which made the deployment fail | 20:48 |
| ayoung | rook, is it possible that there are too many workers for something out there? We've seen that elsewhere | 20:48 |
| ayoung | each worker starts up its own connection pool and pretty soon Maria has been exhauster. Poor Mary | 20:49 |
| * ayoung still not getting RPM installed on Overcloud dagnabit | 20:49 | |
| *** ioggstream has joined #tripleo | 20:49 | |
| rook | possibly, but we have a soft file limit to 1024 -- slage's has 16384, however mine doesn't.... | 20:49 |
| rook | this isn't the first time i have seen this issue, but slage didn't | 20:50 |
| rook | so, there must be something more to it! | 20:50 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: Implement scenario001 CI job https://review.openstack.org/360039 | 20:51 |
| *** Goneri has quit IRC | 20:52 | |
| ayoung | EmilienM, so /etc/puppet/modules/tripleo/manifests/profile/base/keystone.pp has the line ensure_packages(['mod_auth_mellon'], {'ensure' => 'present'}) but the rpm is not on the system | 20:53 |
| slagle | rook: anything in the journalctl output for mariadb or mariadb log about the file limit? | 20:54 |
| ayoung | I was able to run sudo yum install mod_auth_mellon so it looks like dns/resolve is OK. | 20:54 |
| *** jdob1 has quit IRC | 20:55 | |
| openstackgerrit | Christian Schwede proposed openstack/tripleo-heat-templates: Add system-uuid based hostname entries https://review.openstack.org/358643 | 20:56 |
| zaneb | rook: it's possible that one of the workers exited due to the DB exception | 20:56 |
| *** morazi has joined #tripleo | 20:56 | |
| zaneb | rook: is there an actual problem here? | 20:56 |
| zaneb | afaict we ran out of file descriptors, which is a non-survivable error | 20:57 |
| zaneb | Heat actually put the stuff that was IN_PROGRESS into the FAILED state, which is a *huge* victory | 20:57 |
| zaneb | and running the delete again will clean everything up | 20:57 |
| rook | let me start at the beginning as it seems it is being confused on how I got to this point | 20:57 |
| zaneb | I'd call that an unqualified success | 20:58 |
| rook | I did a deployment, it failed due to running out of FDs | 20:58 |
| rook | following? | 20:58 |
| zaneb | yep | 20:58 |
| rook | I tried to delete the said failed deployment after restarting all services... | 20:58 |
| rook | this failed | 20:58 |
| rook | and I saw the message about heat-engine going away, hwoever the FD issue was resolved. | 20:58 |
| rook | however* | 20:58 |
| rook | so, the FD issue and the heat delete issue aren't related. | 20:59 |
| rook | I just did another heat stack-delete and it succeeded w/o issue (after the abandon) | 20:59 |
| zaneb | abandon? | 20:59 |
| beagles | this is me wishing we could add "queryable" priority tags to reviews | 21:00 |
| rook | https://bugzilla.redhat.com/show_bug.cgi?id=1234607#c3 zaneb | 21:01 |
| openstack | bugzilla.redhat.com bug 1234607 in openstack-heat "There needs to be a force delete option for Heat stacks" [High,Closed: notabug] - Assigned to zbitter | 21:01 |
| zaneb | rook: did you retry the delete before doing the abandon? | 21:02 |
| rook | multiple times zaneb | 21:03 |
| rook | each resulting in the same issue i posted above (the traceback) | 21:03 |
| zaneb | ok | 21:03 |
| zaneb | rook: https://bugs.launchpad.net/heat/+bug/1570576 seems like it *might* be related | 21:03 |
| openstack | Launchpad bug 1570576 in heat "After updating a stack stuck IN_PROGRESS, resources will be permanently stuck IN_PROGRESS" [High,Triaged] | 21:03 |
| *** nyechiel has quit IRC | 21:03 | |
| zaneb | related: https://bugzilla.redhat.com/show_bug.cgi?id=1320771#c8 | 21:04 |
| openstack | bugzilla.redhat.com bug 1320771 in openstack-heat "Heat resources left IN_PROGRESS after engine restart during update" [High,New] - Assigned to therve | 21:04 |
| zaneb | note comment #11 where someone reports they ran into this issue due to running out of sql connections | 21:04 |
| zaneb | the surprising thing though is that your status message says "Engine went down during stack DELETE", but the problem that might have caused the engine to go down occurred before any deletes had been attempted | 21:06 |
| *** lblanchard has quit IRC | 21:08 | |
| *** ooolpbot has joined #tripleo | 21:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1616144 | 21:10 |
| *** ooolpbot has quit IRC | 21:10 | |
| openstack | Launchpad bug 1616144 in tripleo "tripleo-ci: nonha jobs failing with Unable to establish connection to https://192.0.2.2:13004/v1/a90407df1e7f4f80a38a1b1671ced2ff/stacks/overcloud/f9f6f712-8e89-4ea9-a34b-6084dc74b5c1" [Critical,Triaged] | 21:10 |
| *** maeca1 has joined #tripleo | 21:10 | |
| *** bootsha has quit IRC | 21:13 | |
| *** asbishop has quit IRC | 21:14 | |
| *** maeca1 has quit IRC | 21:21 | |
| *** rhallisey has quit IRC | 21:23 | |
| *** mhenkel has joined #tripleo | 21:29 | |
| *** jeckersb is now known as jeckersb_gone | 21:31 | |
| *** mburned is now known as mburned_out | 21:32 | |
| *** paramite has quit IRC | 21:34 | |
| *** dsariel has joined #tripleo | 21:38 | |
| openstackgerrit | Giulio Fidente proposed openstack/python-tripleoclient: Do not use selinux-permissive for the CentOS image https://review.openstack.org/360097 | 21:42 |
| *** bootsha has joined #tripleo | 21:43 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Permit usage of raw disks as backends for Swift storage https://review.openstack.org/350841 | 21:43 |
| *** ioggstream has quit IRC | 21:44 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Fix check for MariaDB upgrade manual switch off https://review.openstack.org/359803 | 21:45 |
| openstackgerrit | Merged openstack/tripleo-specs: Add 'bug tagging' policy https://review.openstack.org/352852 | 21:45 |
| *** bootsha has quit IRC | 21:52 | |
| *** weshay is now known as weshay_bbiab | 21:53 | |
| *** noslzzp has quit IRC | 22:02 | |
| beagles | Can the ptb abandon this https://review.openstack.org/#/c/245935/ at your leisure. No longer seems relevant. | 22:04 |
| openstackgerrit | Robin Cernin proposed openstack/tripleo-validations: Validation stonith device exists in OpenStack Platform HA cluster https://review.openstack.org/360102 | 22:07 |
| *** cdearborn has quit IRC | 22:07 | |
| openstackgerrit | Robin Cernin proposed openstack/tripleo-validations: Validation stonith device exists in OpenStack Platform HA cluster https://review.openstack.org/360102 | 22:10 |
| *** ooolpbot has joined #tripleo | 22:11 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 22:11 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1616144 | 22:11 |
| *** ooolpbot has quit IRC | 22:11 | |
| openstack | Launchpad bug 1616144 in tripleo "tripleo-ci: nonha jobs failing with Unable to establish connection to https://192.0.2.2:13004/v1/a90407df1e7f4f80a38a1b1671ced2ff/stacks/overcloud/f9f6f712-8e89-4ea9-a34b-6084dc74b5c1" [Critical,Triaged] | 22:11 |
| *** dsariel has quit IRC | 22:12 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Fixes Neutron metadata parameters in THT https://review.openstack.org/355987 | 22:26 |
| *** dsneddon has quit IRC | 22:29 | |
| *** dsneddon has joined #tripleo | 22:30 | |
| *** Ryjedo has quit IRC | 22:31 | |
| *** dsneddon has quit IRC | 22:32 | |
| *** jrist has quit IRC | 22:33 | |
| *** yamahata has quit IRC | 22:36 | |
| *** dsneddon has joined #tripleo | 22:36 | |
| *** yamahata has joined #tripleo | 22:38 | |
| *** bfournie has quit IRC | 22:42 | |
| *** jrist has joined #tripleo | 22:45 | |
| *** jubapa has joined #tripleo | 22:48 | |
| *** jbadiapa has joined #tripleo | 22:48 | |
| *** rcernin has quit IRC | 22:53 | |
| *** marcusvrn_ has quit IRC | 22:54 | |
| *** rlandy is now known as rlandy|bbl | 22:55 | |
| *** zigo has quit IRC | 22:56 | |
| *** zigo has joined #tripleo | 22:57 | |
| *** jbadiapa has quit IRC | 23:03 | |
| *** jubapa has quit IRC | 23:04 | |
| *** pradk has quit IRC | 23:06 | |
| *** ooolpbot has joined #tripleo | 23:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1616144 | 23:10 |
| *** ooolpbot has quit IRC | 23:10 | |
| openstack | Launchpad bug 1616144 in tripleo "tripleo-ci: nonha jobs failing with Unable to establish connection to https://192.0.2.2:13004/v1/a90407df1e7f4f80a38a1b1671ced2ff/stacks/overcloud/f9f6f712-8e89-4ea9-a34b-6084dc74b5c1" [Critical,Triaged] | 23:10 |
| *** jobewan has quit IRC | 23:15 | |
| *** apetrich has quit IRC | 23:18 | |
| *** apetrich has joined #tripleo | 23:18 | |
| *** Ryjedo has joined #tripleo | 23:20 | |
| *** Ryjedo has quit IRC | 23:20 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Add sshnaidm to tripleo-cd-admins https://review.openstack.org/357217 | 23:29 |
| *** jcoufal has joined #tripleo | 23:36 | |
| *** jcoufal has quit IRC | 23:44 | |
| *** yolanda has quit IRC | 23:48 | |
| *** dmacpher has quit IRC | 23:51 | |
| openstackgerrit | Lars Kellogg-Stedman proposed openstack/tripleo-heat-templates: Add fluentd client service https://review.openstack.org/353506 | 23:52 |
| rook | slagle: you still around? After running a systemctl daemon-reload, then systemctl restart mariadb; i see the limit is correct. | 23:55 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!