| *** zaneb has joined #tripleo | 00:32 | |
| *** dbecker has joined #tripleo | 00:32 | |
| *** dhill_ has joined #tripleo | 00:40 | |
| *** dhill__ has quit IRC | 00:41 | |
| *** dhill_ has quit IRC | 00:52 | |
| *** dhill_ has joined #tripleo | 00:53 | |
| *** dhill_ has joined #tripleo | 00:53 | |
| *** dbecker has quit IRC | 00:53 | |
| *** bana_k has joined #tripleo | 00:56 | |
| openstackgerrit | Honza Pokorny proposed openstack/tripleo-ui: Add central logging https://review.openstack.org/376060 | 01:00 |
|---|---|---|
| baseball | I'm trying to use tripleo quickstart... When I run it I am getting an error after a few seconds saying "bootstrap failed" and it tells me to run quickstart.sh --install-deps or quickstart.sh --clean | 01:36 |
| baseball | Both of those do not change anything | 01:36 |
| *** dmacpher-afk has quit IRC | 01:53 | |
| *** bana_k has quit IRC | 01:59 | |
| *** baseball has quit IRC | 02:36 | |
| *** baseball has joined #tripleo | 02:49 | |
| *** dmacpher has joined #tripleo | 02:59 | |
| *** dmacpher has quit IRC | 03:00 | |
| *** david-lyle has quit IRC | 03:04 | |
| *** rwsu has joined #tripleo | 03:24 | |
| *** dmacpher has joined #tripleo | 03:44 | |
| *** baseball has quit IRC | 04:01 | |
| *** TSCHAK has quit IRC | 04:11 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/python-tripleoclient: Updated from global requirements https://review.openstack.org/375993 | 04:27 |
| *** limao has joined #tripleo | 04:55 | |
| *** coolsvap has joined #tripleo | 04:57 | |
| *** StevenK_ is now known as StevenK | 04:58 | |
| *** dsariel has joined #tripleo | 04:59 | |
| *** florianf has joined #tripleo | 05:12 | |
| *** xuao has joined #tripleo | 05:12 | |
| *** jidar_ has joined #tripleo | 05:13 | |
| *** igorbelikov_ has joined #tripleo | 05:13 | |
| *** NobodyCam_ has joined #tripleo | 05:13 | |
| *** crinkle_ has joined #tripleo | 05:13 | |
| *** george_goh|afk has joined #tripleo | 05:15 | |
| *** rdopiera_ has joined #tripleo | 05:16 | |
| *** cmyster_ has joined #tripleo | 05:16 | |
| *** pleia2_ has joined #tripleo | 05:16 | |
| *** Hazelesque_ has joined #tripleo | 05:16 | |
| *** matbu_ has joined #tripleo | 05:16 | |
| *** lynxman_ has joined #tripleo | 05:17 | |
| *** rickflare2 has joined #tripleo | 05:17 | |
| *** spredzy_ has joined #tripleo | 05:17 | |
| *** masco has joined #tripleo | 05:18 | |
| *** zoliXXLz has joined #tripleo | 05:19 | |
| *** stevebak` has joined #tripleo | 05:19 | |
| *** jayg|g0n3 has joined #tripleo | 05:19 | |
| *** another_larsks has joined #tripleo | 05:19 | |
| *** tbarron_ has joined #tripleo | 05:19 | |
| *** weshay_ has joined #tripleo | 05:20 | |
| *** etp_ has joined #tripleo | 05:20 | |
| *** sirushti_ has joined #tripleo | 05:20 | |
| *** saneaxL has joined #tripleo | 05:20 | |
| *** cmyster has quit IRC | 05:20 | |
| *** rcrit has quit IRC | 05:21 | |
| *** igorbelikov has quit IRC | 05:21 | |
| *** lynxman has quit IRC | 05:21 | |
| *** NobodyCam has quit IRC | 05:21 | |
| *** ayoung has quit IRC | 05:21 | |
| *** saneax-_-|AFK has quit IRC | 05:21 | |
| *** spredzy has quit IRC | 05:21 | |
| *** etp has quit IRC | 05:21 | |
| *** jayg|g0n` has quit IRC | 05:21 | |
| *** vkmc has quit IRC | 05:21 | |
| *** tbarron has quit IRC | 05:21 | |
| *** sirushti has quit IRC | 05:21 | |
| *** Hazelesque has quit IRC | 05:21 | |
| *** weshay has quit IRC | 05:21 | |
| *** larsks has quit IRC | 05:21 | |
| *** sshnaidm has quit IRC | 05:21 | |
| *** dsneddon has quit IRC | 05:21 | |
| *** zoli_gone-proxy has quit IRC | 05:21 | |
| *** rdopiera has quit IRC | 05:21 | |
| *** hogepodge has quit IRC | 05:21 | |
| *** jidar has quit IRC | 05:21 | |
| *** stevebaker has quit IRC | 05:21 | |
| *** crinkle has quit IRC | 05:21 | |
| *** george_goh has quit IRC | 05:21 | |
| *** pleia2 has quit IRC | 05:21 | |
| *** rickflare has quit IRC | 05:21 | |
| *** SpamapS has quit IRC | 05:21 | |
| *** matbu has quit IRC | 05:21 | |
| *** spredzy_ is now known as spredzy | 05:21 | |
| *** rickflare2 is now known as rickflare | 05:21 | |
| *** tbarron_ is now known as tbarron | 05:21 | |
| *** etp_ is now known as etp | 05:21 | |
| *** lynxman_ is now known as lynxman | 05:21 | |
| *** sirushti_ is now known as sirushti | 05:21 | |
| *** igorbelikov_ is now known as igorbelikov | 05:22 | |
| *** NobodyCam_ is now known as NobodyCam | 05:22 | |
| *** vkmc has joined #tripleo | 05:23 | |
| *** ayoung has joined #tripleo | 05:27 | |
| *** dsneddon has joined #tripleo | 05:27 | |
| *** SpamapS has joined #tripleo | 05:27 | |
| *** sshnaidm has joined #tripleo | 05:28 | |
| *** rcrit has joined #tripleo | 05:29 | |
| *** rcernin has joined #tripleo | 05:30 | |
| *** pgadiya has joined #tripleo | 05:32 | |
| *** ramishra has joined #tripleo | 05:48 | |
| *** matbu_ is now known as matbu | 05:55 | |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui: DeploymentDetail component https://review.openstack.org/376189 | 05:59 |
| *** ccamacho has joined #tripleo | 06:08 | |
| *** mcornea has joined #tripleo | 06:16 | |
| *** yolanda has quit IRC | 06:18 | |
| ccamacho | Good morning guys! | 06:19 |
| *** rasca has joined #tripleo | 06:19 | |
| *** bana_k has joined #tripleo | 06:22 | |
| *** jidar_ is now known as jidar | 06:24 | |
| *** jprovazn has joined #tripleo | 06:28 | |
| *** cmyster_ is now known as cmyster | 06:32 | |
| *** cmyster has joined #tripleo | 06:32 | |
| *** Guest64565 is now known as leanderthal | 06:33 | |
| bandini | morning *, ccamacho | 06:35 |
| ccamacho | bandini o/ moooorning! | 06:35 |
| *** liverpooler has joined #tripleo | 06:37 | |
| *** jbadiapa has joined #tripleo | 06:45 | |
| *** pcaruana has joined #tripleo | 06:45 | |
| *** dmacpher has quit IRC | 06:51 | |
| matbu | o/ | 06:57 |
| bandini | morning matbu | 06:58 |
| *** flepied1 has quit IRC | 07:03 | |
| *** flepied has joined #tripleo | 07:04 | |
| openstackgerrit | zhongshengping proposed openstack/tripleo-common: Use fnmatch from oslo.utils https://review.openstack.org/376230 | 07:06 |
| zoliXXLz | good morning | 07:10 |
| *** ebarrera has joined #tripleo | 07:11 | |
| *** flepied has quit IRC | 07:11 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: WIP: Multinode periodic temepest with no skips https://review.openstack.org/376237 | 07:12 |
| *** zoliXXLz is now known as zoli|brb | 07:13 | |
| *** hjensas has joined #tripleo | 07:14 | |
| *** hjensas has quit IRC | 07:14 | |
| *** hjensas has joined #tripleo | 07:14 | |
| *** tremble has joined #tripleo | 07:18 | |
| *** jpena|off is now known as jpena | 07:21 | |
| *** bana_k has quit IRC | 07:22 | |
| *** jtomasek has quit IRC | 07:22 | |
| *** b00tcat has joined #tripleo | 07:24 | |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: Implement overcloud upgrade job - Mitaka -> Newton https://review.openstack.org/323750 | 07:25 |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-heat-templates: Add metricd workers support in gnocchi https://review.openstack.org/374704 | 07:28 |
| *** akuznetsov has joined #tripleo | 07:29 | |
| *** jaosorior has joined #tripleo | 07:29 | |
| *** aufi has joined #tripleo | 07:38 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Bind MySQL address to hostname appropriate to its network https://review.openstack.org/375523 | 07:38 |
| *** zoli|brb is now known as zoli | 07:41 | |
| *** zoli is now known as zoliXXL | 07:41 | |
| *** jpich has joined #tripleo | 07:42 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Add parameters to run nova over httpd https://review.openstack.org/376255 | 07:52 |
| *** bana_k has joined #tripleo | 07:55 | |
| openstackgerrit | Julie Pichon proposed openstack/python-tripleoclient: Stop plan creation when container exists https://review.openstack.org/376264 | 07:58 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Deploy nova over Apache httpd https://review.openstack.org/376267 | 08:00 |
| *** athomas has joined #tripleo | 08:01 | |
| *** ohamada has joined #tripleo | 08:06 | |
| matbu | jistr: hi man can you review this one plz https://review.openstack.org/374892 | 08:09 |
| matbu | jistr: at least for removing the -2 | 08:10 |
| jaosorior | matbu: well, the -2 is from gfidente. He probably you gotta ping him | 08:11 |
| *** abehl has joined #tripleo | 08:11 | |
| matbu | jaosorior: k, he seems to ne be on the chan | 08:14 |
| matbu | will wait | 08:14 |
| *** bana_k has quit IRC | 08:14 | |
| openstackgerrit | Merged openstack/puppet-tripleo: Fix Swift manifest for Puppet4 deployment https://review.openstack.org/374339 | 08:17 |
| *** bvandenh has joined #tripleo | 08:17 | |
| *** gfidente has joined #tripleo | 08:18 | |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: Implement overcloud upgrade job - Mitaka -> Newton https://review.openstack.org/323750 | 08:18 |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: Implement overcloud upgrade job - Mitaka -> Newton https://review.openstack.org/323750 | 08:19 |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: Implement overcloud upgrade job - Mitaka -> Newton https://review.openstack.org/323750 | 08:19 |
| bandini | matbu: iirc jistr is on PTO this week | 08:19 |
| matbu | bandini: k thx | 08:23 |
| *** numans has joined #tripleo | 08:27 | |
| openstackgerrit | Alejandro Andreu proposed openstack/tripleo-puppet-elements: Midonet element now complies with version 5.2 https://review.openstack.org/376291 | 08:32 |
| b00tcat | ^ I just cherry picked this for stable/mitaka too | 08:32 |
| jaosorior | b00tcat: is there a bug report for it? | 08:33 |
| *** fzdarsky has joined #tripleo | 08:34 | |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common: Use kwargs to pass in data and error to Mistral Result https://review.openstack.org/375348 | 08:34 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Add the workflow name to the starting workflow message https://review.openstack.org/375331 | 08:34 |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common: Remove the unused service_host arg from node registration https://review.openstack.org/326036 | 08:34 |
| *** yolanda has joined #tripleo | 08:35 | |
| *** morazi has joined #tripleo | 08:39 | |
| *** Guest84596 has quit IRC | 08:40 | |
| *** Guest84596 has joined #tripleo | 08:40 | |
| *** Guest84596 is now known as assassin | 08:40 | |
| openstackgerrit | Alfredo Moralejo proposed openstack/instack-undercloud: Increase timeout for Deployment Plan creation https://review.openstack.org/376297 | 08:40 |
| b00tcat | jaosorior: nope - but this element wouldn't work as it is right now so I modified it | 08:41 |
| *** derekh has joined #tripleo | 08:43 | |
| *** florianf has quit IRC | 08:45 | |
| *** florianf has joined #tripleo | 08:46 | |
| *** masco is now known as masco|lunch | 08:55 | |
| *** flepied has joined #tripleo | 08:57 | |
| matbu | gfidente: hello can you review this again plz: https://review.openstack.org/374892 | 09:01 |
| *** nijaba has quit IRC | 09:06 | |
| *** mhenkel has joined #tripleo | 09:09 | |
| mhenkel | Hi All. Trying a tripleO deployment on OSP8 and deployment gets stuck on 2016-09-26 01:57:08 [ControllerDeployment]: CREATE_IN_PROGRESS state changed | 09:10 |
| *** nijaba has joined #tripleo | 09:11 | |
| *** nijaba has joined #tripleo | 09:11 | |
| mhenkel | on the overcloud controller I see the following messages: | 09:11 |
| mhenkel | Sep 25 22:10:17 localhost os-collect-config: 2016-09-25 22:10:17.154 3310 WARNING os_collect_config.ec2 [-] ('Connection aborted.', error(113, 'No route to host')) | 09:11 |
| mhenkel | Sep 25 22:10:17 localhost os-collect-config: 2016-09-25 22:10:17.154 3310 WARNING os-collect-config [-] Source [ec2] Unavailable. | 09:11 |
| mhenkel | Sep 25 22:10:17 localhost os-collect-config: 2016-09-25 22:10:17.590 3310 WARNING os_collect_config.local [-] /var/lib/os-collect-config/local-data not found. Skipping | 09:11 |
| mhenkel | Sep 25 22:10:17 localhost os-collect-config: 2016-09-25 22:10:17.590 3310 WARNING os_collect_config.local [-] No local metadata found (['/var/lib/os-collect-config/local-data']) | 09:11 |
| mhenkel | not sure if that is relevant | 09:12 |
| mhenkel | any idea on how to debug? | 09:12 |
| sshnaidm | derekh, hi | 09:12 |
| sshnaidm | derekh, can you please take a look in your time why I can't login to te-broker? | 09:13 |
| *** Guest94479 is now known as beagles | 09:13 | |
| derekh | sshnaidm: yup, on rh1 yes? | 09:13 |
| sshnaidm | derekh, yeah | 09:13 |
| *** chem has joined #tripleo | 09:15 | |
| derekh | sshnaidm: your key is there, are you using the correct login name and fowarding your key? http://paste.openstack.org/show/582917/ | 09:16 |
| jistr | matbu, bandini: yup i'm on PTO :) +2'd though. The removal of gfidente -2 has to be done by him i think. | 09:17 |
| gfidente | hey, checking | 09:18 |
| matbu | jistr: ack thx, enjoy pto :) | 09:19 |
| gfidente | merging | 09:19 |
| bandini | jistr: enjoy PTO ;) | 09:19 |
| jistr | thanks :) | 09:20 |
| *** jistr has quit IRC | 09:20 | |
| *** florianf has quit IRC | 09:20 | |
| *** chem has quit IRC | 09:21 | |
| *** chem has joined #tripleo | 09:22 | |
| b00tcat | after several overcloud redeployments, I am now getting this in my undercloud: CREATE_FAILED ResourceInError: resources.Controller: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500" | 09:23 |
| b00tcat | is there an easy way to fix it? I mean without redeploying the undercloud | 09:23 |
| b00tcat | the red hat docs have a solution but I don't have a subscription :-( | 09:24 |
| *** florianf has joined #tripleo | 09:24 | |
| gfidente | maybe someone likes me enough to merge https://review.openstack.org/#/c/373998/ ? :) | 09:25 |
| *** tosky has joined #tripleo | 09:28 | |
| sshnaidm | panda, rabbitmq on overcloud controller? | 09:28 |
| panda | sshnaidm: fails to start on controller 1 and 2 | 09:29 |
| sshnaidm | panda, journalctl? | 09:29 |
| panda | sshnaidm: cpu load is at 17 ... | 09:29 |
| panda | sshnaidm: no memory related errors, some errors on haproxy regarding services down | 09:30 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: get_param calls with multiple arguments need brackets around them https://review.openstack.org/375576 | 09:31 |
| sshnaidm | panda, haproxy msgs is ok | 09:32 |
| panda | sshnaidm: yep, they tend to fade after a while | 09:33 |
| sshnaidm | panda, do you have an access to controller itself? | 09:33 |
| panda | sshnaidm: yes | 09:33 |
| sshnaidm | panda, was mysql started? /var/log/rabbitmq/ may contain hints also | 09:34 |
| *** pgadiya has quit IRC | 09:35 | |
| panda | sshnaidm: ... deployment ended with CREATE_COMPLETE ... but it's lying | 09:35 |
| sshnaidm | panda, source overcloudrc and try nova list | 09:36 |
| sshnaidm | panda, is it with quickstart? | 09:36 |
| panda | sshnaidm: ERROR (ConnectFailure): Unable to establish connection to http://[2001:db8:fd00:1000::19]:5000/v2.0/tokens | 09:36 |
| panda | sshnaidm: yes | 09:36 |
| sshnaidm | panda, I saw a few times quickstart hanging when deployment has finished successfully | 09:38 |
| bandini | b00tcat: http://tripleo.org/troubleshooting/troubleshooting.html#no-valid-host-found-error | 09:38 |
| panda | sshnaidm: rabbitmq logs report successful connections until they died | 09:38 |
| panda | sshnaidm: deployment finished now ... but it's not really working ... | 09:39 |
| panda | sshnaidm: I'l launch build on CI once again, maybe there it works | 09:39 |
| *** flaper87 has joined #tripleo | 09:39 | |
| *** flaper87 has joined #tripleo | 09:39 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci: Add IPv6 network configuration for ipv6 job types https://review.openstack.org/363674 | 09:40 |
| panda | sshnaidm: and then I'm free to analyse what's happening from quickstart setup. | 09:41 |
| panda | sshnaidm: maybe I'm just using the wrong environment files .. DEPLOY_ARGS in tripleo-ci contains everything and more | 09:41 |
| sshnaidm | panda, yeah, it's a mess | 09:42 |
| *** panda is now known as panda|afk | 09:44 | |
| *** Steve__ has joined #tripleo | 09:45 | |
| b00tcat | thanks bandini | 09:45 |
| Steve__ | Hey guys, does anyone have any guidence around using tripleo and puppet-ceph to control crush maps? | 09:46 |
| *** pgadiya has joined #tripleo | 09:48 | |
| *** dbecker has joined #tripleo | 09:57 | |
| openstackgerrit | Nam Nguyen Hoai proposed openstack/python-tripleoclient: Using assertIsNone() instead of assertEqual(None) https://review.openstack.org/376345 | 09:57 |
| *** zoliXXL is now known as zoli|lunch | 10:03 | |
| gfidente | Steve__, I think we can't do that yet | 10:07 |
| Steve__ | cheers gfidente, guess it will have to be injected after a deploy | 10:09 |
| Steve__ | sad times... | 10:09 |
| gfidente | so you'd have to pass ceph::profile::params::osd_crush_update_on_start: false | 10:11 |
| gfidente | via ExtraConfig | 10:11 |
| gfidente | and edit crush after the deployment yes | 10:11 |
| gfidente | question for you though, what would you consider "sufficient" support for crushmap control in tripleo? | 10:12 |
| Steve__ | thanks, im right in assuming that ceph::profile::params::osd_crush_update_on_start: basicly says dont update crush after a deploy, is that correct | 10:12 |
| gfidente | Steve__, yes | 10:12 |
| gfidente | it basically sets the value of global/osd_crush_update_on_start in ceph.coinf | 10:12 |
| Steve__ | gfidente, i guess for *me* the ability to specify certain hosts, to be in certain roots, the ability to create rulesets to point to cetain root's. To allow support for OpenStack multiple backends. | 10:13 |
| Steve__ | *cinder backends | 10:13 |
| Steve__ | if that makes sense | 10:13 |
| gfidente | multiple cinder backends as in, different pools in the same ceph cluster? | 10:14 |
| Steve__ | correct | 10:14 |
| gfidente | others mentioned interest in: different pool in different ceph clusters as well | 10:14 |
| gfidente | I think coolsvap was looking into that ^^ | 10:15 |
| gfidente | though, different pools in same ceph cluster seems an easier target | 10:15 |
| *** akuznetsov has quit IRC | 10:15 | |
| *** panda|afk is now known as panda | 10:15 | |
| Steve__ | diffrent pools in diffrent clusters would be intresting but pools in the same cluster would be great | 10:16 |
| panda | anyone able to read a rabbit crash report ? | 10:16 |
| coolsvap | gfidente: i dont remember signing up for something like this | 10:17 |
| openstackgerrit | Michele Baldessari proposed openstack/puppet-tripleo: WIP DO NOT MERGE Initial Composable HA https://review.openstack.org/362389 | 10:17 |
| *** limao has quit IRC | 10:22 | |
| *** masco|lunch is now known as masco | 10:24 | |
| *** zoli|lunch is now known as zoliXXL | 10:25 | |
| *** jtomasek has joined #tripleo | 10:27 | |
| *** xuao has quit IRC | 10:30 | |
| *** tremble has quit IRC | 10:31 | |
| *** flepied has quit IRC | 10:31 | |
| gfidente | coolsvap, wrong nick, that was colonwq ;) | 10:32 |
| coolsvap | gfidente: thanks for the confirmation | 10:32 |
| *** tremble has joined #tripleo | 10:33 | |
| colonwq | gfidente, Steve__ I am catching up on the scroll back. | 10:35 |
| mcornea | anyone has some hints on how to debug Filter TripleOCapabilitiesFilter returned 0 hosts ? | 10:36 |
| jaosorior | gfidente, mandre, mcornea hey guys, can you check these two commits out? https://review.openstack.org/#/c/375523/ https://review.openstack.org/#/c/375519/ | 10:37 |
| *** dbecker has quit IRC | 10:37 | |
| colonwq | Steve__, You should be able to configure cinder to talk to multiple pools from the same Ceph cluster (one pool per backend). However, tripleo (t-h-t and puppet-tripleo) do not support this at the this time. They support one of each cinder backend type. | 10:39 |
| Steve__ | colonwq, thanks. | 10:41 |
| *** sshnaidm is now known as sshnaidm|afk | 10:41 | |
| Steve__ | pretty much what i had worked out from reading, but just wanted to doublecheck | 10:41 |
| colonwq | Steve__, I made a blog post on how to configure cinder to work with two ceph clusters. You can reuse the same process but only need one ceph config file and one use anme. | 10:43 |
| *** rodrigods has quit IRC | 10:43 | |
| *** rodrigods has joined #tripleo | 10:43 | |
| colonwq | Steve__, It is all manual but it is along the correct path. http://keithschincke.blogspot.com/2016/09/integrating-openstack-mitaka-cinder.html | 10:44 |
| Steve__ | thanks ill take a read | 10:45 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Enable TLS in the internal network for keystone https://review.openstack.org/327029 | 10:47 |
| *** amoralej has joined #tripleo | 10:48 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Enable TLS in the internal network for keystone https://review.openstack.org/327029 | 10:48 |
| *** tremble has quit IRC | 10:50 | |
| *** tremble has joined #tripleo | 10:52 | |
| *** tremble has quit IRC | 10:52 | |
| *** tremble has joined #tripleo | 10:52 | |
| *** rhallisey has joined #tripleo | 10:55 | |
| *** milan has joined #tripleo | 11:01 | |
| *** tremble has quit IRC | 11:02 | |
| jaosorior | marios: hey dude, if you have time can you check these out? https://review.openstack.org/#/c/375523/ https://review.openstack.org/#/c/375519/ | 11:03 |
| *** tremble has joined #tripleo | 11:04 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Remove double definition of config_settings key in keystone https://review.openstack.org/376403 | 11:11 |
| *** adarazs is now known as adarazs_afk | 11:16 | |
| *** pkovar has joined #tripleo | 11:28 | |
| *** tremble has quit IRC | 11:28 | |
| *** tremble has joined #tripleo | 11:30 | |
| *** tremble has quit IRC | 11:30 | |
| *** tremble has joined #tripleo | 11:30 | |
| *** jpena is now known as jpena|lunch | 11:34 | |
| *** onovy has quit IRC | 11:38 | |
| *** onovy has joined #tripleo | 11:38 | |
| jaosorior | who can I ask regarding validation errors when deploying the overcloud? | 11:38 |
| jaosorior | tried to add a nested stack and got something like this: | 11:39 |
| jaosorior | Failed to validate: : resources.ControllerServiceChain: : Failed to validate nested template: Failed to validate: resources[43]: Failed to validate: resources.ApacheServiceBase: Failed to validate: resources.ApacheTLS: The Resource Type (OS::TripleO::Services::ApacheTLS) could not be found | 11:39 |
| jaosorior | not sure where to start debugging that | 11:39 |
| EmilienM | hello | 11:41 |
| *** amoralej is now known as amoralej|lunch | 11:43 | |
| *** tremble has quit IRC | 11:43 | |
| *** tremble has joined #tripleo | 11:44 | |
| jaosorior | EmilienM: hey dude, can you check these two CRs out? https://review.openstack.org/#/c/375523/ https://review.openstack.org/#/c/375519/ | 11:44 |
| EmilienM | jaosorior: sure | 11:45 |
| *** jtomasek has quit IRC | 11:48 | |
| jaosorior | anybody around that has worked in validations that could help me out? | 11:48 |
| Jokke__ | Can we get these two in, pretty please? https://review.openstack.org/#/c/374961/ https://review.openstack.org/#/c/358525/ | 11:50 |
| EmilienM | jaosorior: can you look https://review.openstack.org/#/q/topic:tripleo/rc2 please? | 11:51 |
| *** dprince has joined #tripleo | 11:51 | |
| *** sshnaidm|afk is now known as sshnaidm | 11:52 | |
| jaosorior | EmilienM: will do as soon as SSO from review.openstack.org starts working again for me X_x | 11:52 |
| jaosorior | for some reason now it's not letting me log in | 11:52 |
| jaosorior | what the hell | 11:52 |
| EmilienM | launchpad is down | 11:53 |
| EmilienM | do you have gertty? | 11:53 |
| jaosorior | EmilienM: no :( | 11:53 |
| *** dbecker has joined #tripleo | 11:53 | |
| jaosorior | dprince: hey, could you check these commits out? https://review.openstack.org/#/c/375523/ https://review.openstack.org/#/c/375519/ | 11:54 |
| panda | sshnaidm: mmhh ... it's actually working ... I had to add the correct ipv6 on the undercloud | 11:54 |
| panda | sshnaidm: now nova list shows empty table | 11:54 |
| *** masco has quit IRC | 11:55 | |
| panda | sshnaidm: stil don't understand the errors on rabbitmq | 11:55 |
| sshnaidm | panda, does rabbitmq run or died? | 11:55 |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: Implement overcloud upgrade job - Mitaka -> Newton https://review.openstack.org/323750 | 11:55 |
| panda | sshnaidm: maybe overcloud is working but the clustering is not | 11:55 |
| panda | sshnaidm: it's running only on controller-0 | 11:55 |
| panda | sshnaidm: it failed on the other nodes | 11:56 |
| panda | sshnaidm: I'll wait to see at it goes in OVB env, but even if it succeds I'll check if all the rabbitmq process are running on the controllers | 11:57 |
| sshnaidm | panda, maybe it's worth to check clustering | 11:57 |
| sshnaidm | panda, did you try the dev env from my mail? | 11:57 |
| panda | sshnaidm: not yet. If last build fails, I'll use the env, so I can check what is wrong. | 11:58 |
| *** tremble has quit IRC | 11:58 | |
| sshnaidm | slagle, ping me when you're around please | 11:59 |
| *** numans has quit IRC | 11:59 | |
| *** tremble has joined #tripleo | 12:00 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Enable TLS in the internal network for keystone https://review.openstack.org/327029 | 12:02 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Fetch internal certificates for HAProxy based on network https://review.openstack.org/366548 | 12:02 |
| openstackgerrit | Ryan Hallisey proposed openstack/tripleo-heat-templates: Add a heat output for RoleNodes https://review.openstack.org/327822 | 12:02 |
| *** adarazs_afk is now known as adarazs | 12:04 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Enable TLS in the internal network for keystone https://review.openstack.org/327029 | 12:05 |
| sshnaidm | panda, btw, I don't see any memory errors with ha today, so I hope it's over | 12:05 |
| *** jayg|g0n3 is now known as jayg | 12:09 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: WIP: Multinode periodic tempest with no skips https://review.openstack.org/376237 | 12:10 |
| EmilienM | dprince: should I abandon https://review.openstack.org/#/c/373500/ ? | 12:16 |
| panda | beekneemech: do you still have you ipv6 env online ? I didn't increase the ram on my local tests, but I waited more for the deploy to fail. It actually deployed, but took a lot of time. WHe I check the overcloud, I see that rabbitmq failed in two nodes, but nodes ping and nova list on the overcloud return at least an empty table. CI builds are still timing out. Can you check if your test env have rabbitmq | 12:20 |
| panda | failing on two of the three controllers ? | 12:20 |
| *** shadower has joined #tripleo | 12:21 | |
| jaosorior | EmilienM: hey dude, one-liner here https://review.openstack.org/#/c/376403/1/puppet/services/keystone.yaml :D | 12:21 |
| EmilienM | jaosorior: done | 12:21 |
| jaosorior | EmilienM: launchpad is online again. I'll check the CR you asked | 12:22 |
| EmilienM | cool | 12:23 |
| panda | sshnaidm: CI build failed, I'm going to use the env now to confront local result given by quickstart. | 12:23 |
| sshnaidm | panda, ok, tell me how it goes and if there something wrong | 12:23 |
| social | rhallisey: are you doing network isolation? I'm hitting ntrollers | 12:23 |
| social | grr paste | 12:24 |
| social | rhallisey: https://bugs.launchpad.net/tripleo/+bug/1604927 | 12:24 |
| openstack | Launchpad bug 1604927 in tripleo "Deployment times out due to nova-conductor never starting" [Critical,Triaged] | 12:24 |
| dprince | jaosorior: hi, should we remove line 76 here https://review.openstack.org/#/c/375523/4 | 12:24 |
| rhallisey | social, no just HA deploys | 12:24 |
| rhallisey | I run into ram limits | 12:24 |
| social | rhallisey: do you use extraconfig to add swap? | 12:26 |
| jaosorior | dprince: nope. That is meant to bind the wsrep provider. I did not touch that part cause it's more restrictive than the mysql configuration itself (it does take hostnames.. but has a 60 character limit, and our multinode configuration exceeds that apparently) | 12:26 |
| rhallisey | social, no, show I include swap.j2? | 12:27 |
| rhallisey | should* | 12:27 |
| panda | beekneemech: sshnaidm : oh, I can see on host_info.txt on CI build that rabbitmq failed there too. So at least I was able to reproduce the error with quickstart. Now to understand what's happening ... | 12:27 |
| jaosorior | dprince: I can fix it to have a name that makes more sense. But lets do that in another patch | 12:28 |
| * rhallisey checks the docs | 12:29 | |
| *** pabelanger has joined #tripleo | 12:29 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/python-tripleoclient: Updated from global requirements https://review.openstack.org/375993 | 12:30 |
| pabelanger | EmilienM: do you mind having somebody fix the nonha-multinod jobs? There is some issues with timeouts. For example: | 12:30 |
| pabelanger | 60mins to timeout SSH: http://logs.openstack.org/34/376334/1/check/gate-tripleo-ci-centos-7-nonha-multinode/8817160/console.html#_2016-09-26_10_58_21_962810 | 12:30 |
| pabelanger | 90mins to timeout failed job: http://logs.openstack.org/34/376334/1/check/gate-tripleo-ci-centos-7-nonha-multinode/8817160/console.html#_2016-09-26_12_23_53_862381 | 12:31 |
| pabelanger | that took 2.5hours to fall it looks like | 12:31 |
| pabelanger | when it might have been able to fail in 30mins? | 12:31 |
| gfidente | help merging https://review.openstack.org/#/c/373998/ ? :) | 12:31 |
| *** ohamada has quit IRC | 12:32 | |
| *** ohamada has joined #tripleo | 12:32 | |
| gfidente | thanks EmilienM | 12:32 |
| EmilienM | pabelanger: if I look on http://tripleo.org/cistatus.html it doesn't happen much | 12:32 |
| EmilienM | pabelanger: one time | 12:33 |
| EmilienM | pabelanger: I'm looking | 12:34 |
| pabelanger | EmilienM: ya, it is likely an edge case | 12:34 |
| *** ccamacho is now known as ccamacho|lunch | 12:35 | |
| EmilienM | pabelanger: I found 2 timeout and both were on rax-ord | 12:35 |
| EmilienM | and a third one | 12:36 |
| *** zoliXXL is now known as zoli|brb | 12:36 | |
| EmilienM | oh another on rax-dfw | 12:36 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Add pameter for gmcast.listen_addr configuration https://review.openstack.org/376443 | 12:37 |
| pabelanger | http://logs.openstack.org/46/376246/1/check/gate-tripleo-ci-centos-7-nonha-multinode/f468e9c/console.html#_2016-09-26_09_04_43_142199 is another | 12:37 |
| jaosorior | dprince: so here I'm passing a better name for that https://review.openstack.org/376443 so now we can pass it that way via t-h-t. | 12:38 |
| EmilienM | pabelanger: it's rax-ord | 12:38 |
| pabelanger | 2 failures in gate right now | 12:40 |
| pabelanger | both mistral | 12:40 |
| *** jprovazn has quit IRC | 12:40 | |
| pabelanger | 1 ord, 1 dfw | 12:40 |
| *** cylopez has joined #tripleo | 12:40 | |
| EmilienM | pabelanger: other clouds look fine | 12:41 |
| pabelanger | Failed connect to trunk.rdoproject.org:80; Connection timed out | 12:42 |
| EmilienM | ah, so it's probably something we need to report on #rdo, jpena, etc | 12:43 |
| dprince | jaosorior: ack, thanks | 12:43 |
| *** jpena|lunch is now known as jpena | 12:43 | |
| pabelanger | EmilienM: but the failure in 376334 likely exposes an issue in tripleo-ci, where an error is raised, but cannot notify the exit handler | 12:44 |
| dprince | EmilienM: we should either fix or remove that file. I'm fine removing if you prefer | 12:45 |
| EmilienM | dprince: +1 to remove it as we don't use it now. | 12:45 |
| EmilienM | dprince: if we want swap, we can recreate the file later | 12:45 |
| dprince | EmilienM: I've +2'd. Either way works I think. | 12:46 |
| EmilienM | dprince: I agree | 12:46 |
| *** florianf has quit IRC | 12:46 | |
| EmilienM | dprince: can you also look https://review.openstack.org/#/c/367324/ ? | 12:46 |
| EmilienM | dprince: the jobname should be short this time | 12:47 |
| *** trown is now known as trown|brb | 12:47 | |
| openstackgerrit | Brad P. Crochet proposed openstack/puppet-tripleo: Add zaqar profiles https://review.openstack.org/331681 | 12:48 |
| dprince | jaosorior: +2 on all your work, thanks man | 12:49 |
| jaosorior | dprince: thanks for checking it out | 12:51 |
| *** dmsimard2 is now known as dmsimard | 12:51 | |
| *** Goneri has joined #tripleo | 12:53 | |
| openstackgerrit | Emilien Macchi proposed openstack/python-tripleoclient: Keystone credentials needs to be set with the overcloud password https://review.openstack.org/376449 | 12:55 |
| EmilienM | matbu, gfidente, backport ^ | 12:56 |
| *** david-lyle has joined #tripleo | 12:56 | |
| jaosorior | gfidente: soon | 12:57 |
| shadower | jpich: regarding the i18n blueprient: should I just edit the whiteboard? | 12:57 |
| jpich | shadower: Yes please! | 12:57 |
| *** trown|brb is now known as trown | 12:57 | |
| jpich | shadower: Thanks for giving it some thought :) | 12:57 |
| gfidente | jaosorior, ack | 12:58 |
| EmilienM | jpich: hi! should we backport https://review.openstack.org/#/c/373336/ ? | 12:58 |
| *** amoralej|lunch is now known as amoralej | 12:58 | |
| *** florianf has joined #tripleo | 12:58 | |
| EmilienM | dprince: fyi ^ https://review.openstack.org/#/c/373336/ backport? | 13:00 |
| jpich | EmilienM: I'm not familiar with that area of the code but it sounds like things are breaking so I would say yes? Is the author online? | 13:00 |
| EmilienM | sshnaidm: ^ | 13:00 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Use prameter name to configure gmcast_listen_addr https://review.openstack.org/376457 | 13:01 |
| sshnaidm | EmilienM, I don't think it's mine | 13:01 |
| jaosorior | gfidente: ^ | 13:01 |
| EmilienM | sshnaidm: wrong nickname, I wanted to ping slagle | 13:01 |
| EmilienM | sshnaidm: sorry :) | 13:01 |
| sshnaidm | EmilienM, :) | 13:01 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Use parameter name to configure gmcast_listen_addr https://review.openstack.org/376457 | 13:02 |
| shadower | jpich: no problem. Unfortunately, I don't have any good answers regarding validations :-( | 13:02 |
| jpich | shadower: That'll be still a starting point somehow! :) | 13:02 |
| openstackgerrit | Emilien Macchi proposed openstack/python-tripleoclient: Remove the reference to overcloud-without-mergepy file https://review.openstack.org/376458 | 13:02 |
| *** fultonj has joined #tripleo | 13:03 | |
| *** zoli|brb is now known as zoliXXL | 13:03 | |
| *** numans has joined #tripleo | 13:14 | |
| *** myoung|alt is now known as myoung | 13:15 | |
| *** limao has joined #tripleo | 13:15 | |
| *** rlandy has joined #tripleo | 13:18 | |
| *** trozet__ is now known as trozet | 13:18 | |
| EmilienM | pabelanger: i see the same timeout in single node puppet jobs | 13:18 |
| *** jaosorior has quit IRC | 13:19 | |
| *** jaosorior has joined #tripleo | 13:20 | |
| *** lblanchard has joined #tripleo | 13:20 | |
| openstackgerrit | Paul Belanger proposed openstack/python-tripleoclient: Add ssl development headers to bindep https://review.openstack.org/376469 | 13:22 |
| pabelanger | EmilienM: okay, so something is going on then | 13:22 |
| shadower | jpich: done | 13:25 |
| jpich | shadower: That is very helpful, thanks! | 13:27 |
| shadower | my pleasure! | 13:27 |
| jpich | I might RST-ify that whiteboard to make it more readable in the future... :) | 13:28 |
| openstackgerrit | Merged openstack/puppet-tripleo: Move inclusion of ::manila::db::mysql in manila/api profile https://review.openstack.org/374961 | 13:33 |
| *** skramaja_ has joined #tripleo | 13:34 | |
| *** skramaja has quit IRC | 13:36 | |
| *** liverpooler has quit IRC | 13:36 | |
| *** xuao has joined #tripleo | 13:36 | |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates: Fix ignore warning on ceph major upgrade. https://review.openstack.org/376509 | 13:37 |
| *** jeckersb_gone is now known as jeckersb | 13:37 | |
| thrash | EmilienM: scenario 003 passed with mistral https://review.openstack.org/#/c/323436/ \o/ | 13:38 |
| EmilienM | thrash: woot | 13:40 |
| EmilienM | nice job here | 13:40 |
| openstackgerrit | Brad P. Crochet proposed openstack-infra/tripleo-ci: Only ask for the overcloud-full image https://review.openstack.org/363592 | 13:41 |
| *** limao_ has joined #tripleo | 13:42 | |
| *** rcarrillocruz has joined #tripleo | 13:44 | |
| *** akshai has joined #tripleo | 13:44 | |
| *** skramaja has joined #tripleo | 13:45 | |
| *** limao has quit IRC | 13:45 | |
| *** skramaja_ has quit IRC | 13:46 | |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-docs: Switch docs to use tripleo-build-images https://review.openstack.org/337669 | 13:50 |
| *** ccamacho|lunch is now known as ccamacho | 13:50 | |
| openstackgerrit | Merged openstack/python-tripleoclient: Stop plan creation when container exists https://review.openstack.org/376264 | 13:52 |
| *** cinerama` is now known as cinerama | 13:52 | |
| openstackgerrit | Merged openstack/tripleo-puppet-elements: Install ceph-selinux in overcloud-{controller,cephstorage} images https://review.openstack.org/373998 | 13:52 |
| openstackgerrit | Merged openstack/puppet-tripleo: Make mysql bind-address configurable https://review.openstack.org/375519 | 13:53 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Use parameter name to configure gmcast_listen_addr https://review.openstack.org/376457 | 13:53 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Bind MySQL address to hostname appropriate to its network https://review.openstack.org/375523 | 13:53 |
| *** jprovazn has joined #tripleo | 13:54 | |
| b00tcat | can someone have a look at https://review.openstack.org/#/c/376291/ ? :-) | 13:54 |
| EmilienM | as a reminder, please focus on https://launchpad.net/tripleo/+milestone/newton-rc2 -- we'll cut RC2 before Thursday end of day and branch stable/newton by this time | 13:58 |
| *** hjensas has quit IRC | 13:58 | |
| *** xuao has quit IRC | 14:00 | |
| *** numan_ has joined #tripleo | 14:00 | |
| *** numans has quit IRC | 14:01 | |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Add the workflow name to the starting workflow message https://review.openstack.org/375331 | 14:02 |
| panda | EmilienM: do we have a rabbitmq expert available ? https://bugs.launchpad.net/tripleo/+bug/1627729 | 14:03 |
| openstack | Launchpad bug 1627729 in tripleo "Rabbitmq fails to start on two nodes on HA IPv6 configuration" [Undecided,Confirmed] | 14:03 |
| *** Jokke__ is now known as Jokke_ | 14:04 | |
| *** florianf has quit IRC | 14:06 | |
| *** florianf has joined #tripleo | 14:07 | |
| *** florianf has quit IRC | 14:07 | |
| *** florianf has joined #tripleo | 14:08 | |
| *** saneaxL is now known as saneax-_-|AFK | 14:11 | |
| *** absubram has joined #tripleo | 14:14 | |
| *** florianf has quit IRC | 14:15 | |
| *** florianf has joined #tripleo | 14:16 | |
| *** zoliXXL is now known as zoli|brb | 14:19 | |
| jaosorior | Jokke_: hey dude, when you have time can you check this out? https://review.openstack.org/#/c/375357/ | 14:21 |
| *** rajinir has joined #tripleo | 14:23 | |
| *** mgould has joined #tripleo | 14:23 | |
| mgould | hi everyone! | 14:23 |
| mgould | could someone please review https://review.openstack.org/#/c/371568/ ? Trivial patch, has one +2, passing CI | 14:24 |
| *** beekneemech is now known as bnemec | 14:25 | |
| Jokke_ | jaosorior: looking | 14:26 |
| *** rbowen has quit IRC | 14:28 | |
| *** dhill__ has joined #tripleo | 14:30 | |
| *** dhill_ has quit IRC | 14:30 | |
| *** numan_ has quit IRC | 14:30 | |
| *** rbowen has joined #tripleo | 14:31 | |
| EmilienM | jaosorior: can you look gfidente's patch please? https://review.openstack.org/#/c/375404/ | 14:31 |
| EmilienM | and also bandini's patch https://review.openstack.org/#/c/375977/ | 14:31 |
| EmilienM | and this backport https://review.openstack.org/#/c/376449/ | 14:31 |
| *** akshai has quit IRC | 14:33 | |
| jaosorior | EmilienM: was this a clean backport or were there any conflicts? https://review.openstack.org/#/c/376449/ | 14:33 |
| EmilienM | jaosorior: clean | 14:34 |
| EmilienM | jaosorior: if it's not clean I put it in the commit message usually | 14:34 |
| *** akshai has joined #tripleo | 14:34 | |
| EmilienM | damn launchpad down again | 14:34 |
| *** akshai has quit IRC | 14:35 | |
| bandini | it was timing out for most of Sunday as well :/ | 14:36 |
| *** fultonj_ has joined #tripleo | 14:37 | |
| *** fultonj has quit IRC | 14:37 | |
| *** fultonj_ has quit IRC | 14:37 | |
| *** fultonj has joined #tripleo | 14:37 | |
| *** zoli|brb is now known as zoli | 14:38 | |
| *** zoli is now known as zoliXXL | 14:39 | |
| dhill__ | lol | 14:39 |
| dhill__ | launchpadd- | 14:39 |
| dhill__ | launchpad-- | 14:39 |
| trown | hmm... no launchpad makes my comment on https://review.openstack.org/#/c/376297/1 a bit cruel | 14:46 |
| EmilienM | trown: lol | 14:46 |
| EmilienM | indeed :P | 14:46 |
| *** akshai has joined #tripleo | 14:49 | |
| amoralej | trown, lp seems to be down... i'll fill a bug for https://review.openstack.org/#/c/376297 when is back | 14:49 |
| *** bnemec has quit IRC | 14:49 | |
| trown | amoralej: thanks, I was looking to file one immediately after putting that comment up, and found the same | 14:50 |
| amoralej | no prob, i'll be checking... | 14:50 |
| *** florianf has quit IRC | 14:51 | |
| trown | thanks for tracking that down, was just starting to look into it, and checked the etherpad to find out it was already figured out | 14:51 |
| trown | pretty awesome | 14:51 |
| *** mcornea has quit IRC | 14:51 | |
| jaosorior | gfidente, dprince can you guys check this one out? https://review.openstack.org/#/c/376457/ | 14:51 |
| *** dmsimard has quit IRC | 14:52 | |
| *** mcornea has joined #tripleo | 14:52 | |
| EmilienM | jaosorior, gfidente, dprince: FYI I would be a bit careful about what we land this week and try to keep focus on RC2 | 14:52 |
| EmilienM | I see some changes in Galera config, please be careful again | 14:53 |
| EmilienM | we are releasing RC2 this week and start branching | 14:53 |
| jaosorior | EmilienM: well, it just takes a parameter into use (that was defaulting to use hieradata) so it really does take the same value as before in the end | 14:53 |
| jaosorior | but yeah, I agree with the "gotta be careful" statement | 14:53 |
| sshnaidm | panda, did it go well with dev env? | 14:57 |
| *** hogepodge has joined #tripleo | 14:58 | |
| rhallisey | social, went oom on the update | 14:58 |
| *** bnemec has joined #tripleo | 14:58 | |
| rhallisey | 8g was not enough | 14:58 |
| social | rhallisey: enable swap | 14:58 |
| rhallisey | going to this time | 14:59 |
| *** akshai has quit IRC | 15:00 | |
| *** akshai has joined #tripleo | 15:02 | |
| social | rhallisey: I have this ugly thing https://paste.fedoraproject.org/436057/47490208 | 15:02 |
| rhallisey | social, perfect thank you | 15:02 |
| * rhallisey grows his treasure chest of scripts | 15:02 | |
| *** akuznetsov has joined #tripleo | 15:02 | |
| *** florianf has joined #tripleo | 15:05 | |
| *** leanderthal is now known as leanderthal|afk | 15:05 | |
| jaosorior | Jokke_: can you raise a bug on us using a deprecated API version of glance? | 15:05 |
| Jokke_ | jaosorior: sure ... against what 'though? | 15:06 |
| jaosorior | I think it's a #tripleo issue | 15:06 |
| jaosorior | so | 15:06 |
| Jokke_ | just against tripleo or some specific other part? | 15:06 |
| jaosorior | just on the tripleo launchpad | 15:06 |
| Jokke_ | sure | 15:06 |
| jaosorior | Jokke_: lets leave it generic for now | 15:06 |
| jaosorior | Jokke_: anyway, thanks for checking out that issue. Hopefully migrating to glance v2 does the trick. | 15:07 |
| Jokke_ | jaosorior: it should as none of that data will go via headers anymore | 15:07 |
| jaosorior | Jokke_: is there anything else to do? For instance, will that work with existing deployments? | 15:08 |
| Jokke_ | it'll be in the request body as json | 15:08 |
| jaosorior | excellent! | 15:08 |
| Jokke_ | Nova (and Cinder I think) are at least defaulting to v2 in Newton | 15:09 |
| Jokke_ | Horizon apparently uses some mix of v1 and v2 depending what has been enabled | 15:09 |
| Jokke_ | so the main consumers are there. That's why we finally got to deprecate the horrible v1 | 15:09 |
| Jokke_ | jaosorior: what comes to your initial issue leading to finding these. In short term I think safest bet is to put terminating proxy in front of glance | 15:11 |
| *** ebarrera has quit IRC | 15:11 | |
| jaosorior | Jokke_: how come? | 15:11 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/instack-undercloud: Stop pinning Glance API version on stackrc https://review.openstack.org/376566 | 15:12 |
| jaosorior | Jokke_: lets see how that goes ^^ | 15:12 |
| * bnemec guesses not well | 15:12 | |
| jaosorior | bnemec: ?? | 15:12 |
| bnemec | I seem to recall that we had problems using glance v2. | 15:13 |
| jaosorior | bnemec: when? | 15:13 |
| bnemec | Although for stackrc we might be okay because we don't call glance using stackrc anyway. | 15:13 |
| jaosorior | bnemec: in newton? or was that in another release? | 15:13 |
| bnemec | Whenever glance v2 became a thing. | 15:14 |
| bnemec | I think it was more for the ping test though because we were passing an option that v2 didn't support. | 15:14 |
| bnemec | On the undercloud we don't actually use stackrc to talk to glance much. | 15:14 |
| bnemec | Anyway, we'll see what CI says. | 15:14 |
| jaosorior | yep | 15:15 |
| openstackgerrit | Flavio Percoco proposed openstack/tripleo-specs: Containerize TripleO blueprint https://review.openstack.org/223182 | 15:15 |
| bnemec | On the plus side, CI should be much faster now that https://review.openstack.org/#/c/375073/ has merged and is in the cached images. | 15:15 |
| bnemec | At least I'm assuming it's in the cached images based on the job runtimes the past couple of days. | 15:15 |
| trown | bnemec: oh, that is what was adding so much time to my tests using the overcloud from tripleo-ci as an undercloud | 15:16 |
| trown | I couldn't figure out why the image was taking so long to boot | 15:17 |
| bnemec | trown: Yeah, it just started in the last couple of weeks for some reason. | 15:17 |
| bnemec | I'm not sure why because from what I've found Centos has been defaulting to the 5 minute timeout for a lot longer than that. | 15:17 |
| Jokke_ | I'd assume there's at least bunch of "is_public=True"s that needs to be changed to "visibility=public" | 15:17 |
| Jokke_ | ;) | 15:17 |
| openstackgerrit | Flavio Percoco proposed openstack/tripleo-specs: Containerize TripleO blueprint https://review.openstack.org/223182 | 15:17 |
| *** dmsimard has joined #tripleo | 15:18 | |
| Jokke_ | 'visibility="public"' | 15:19 |
| jaosorior | Jokke_: that is correct. | 15:19 |
| jaosorior | we are using is_public | 15:19 |
| Jokke_ | jaosorior: I know as we're using v1 API. That's one of the really needed migrations when going from v1 to v2 | 15:19 |
| Jokke_ | v1 does not support visibility and in v2 it's not boolean. Due to the fact that OS community made the decision not to ever remove anything from the APIs we could not make that transition to v2 either as we would have been stuck with it forever | 15:21 |
| *** kjw3 has quit IRC | 15:21 | |
| openstackgerrit | Isaac Beckman proposed openstack/diskimage-builder: Enable ssh password authentication https://review.openstack.org/376572 | 15:25 |
| *** myoung is now known as myoung|afk | 15:25 | |
| mah | Hi all, I have a problem when I deployed OPNFV-apex which is based on TripleO.. that I can not do any openstack commands for overcloud at the undercloud, while it works fine if I am at the overcloud.. knowing that the connectivity is fine betwen the nodes | 15:27 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: WIP: Multinode periodic tempest with no skips https://review.openstack.org/376237 | 15:27 |
| *** toure has quit IRC | 15:27 | |
| *** openstackstatus has joined #tripleo | 15:28 | |
| *** ChanServ sets mode: +v openstackstatus | 15:28 | |
| *** toure has joined #tripleo | 15:28 | |
| *** mcornea has quit IRC | 15:29 | |
| *** akrzos has quit IRC | 15:30 | |
| *** dtrainor has joined #tripleo | 15:32 | |
| *** akrzos has joined #tripleo | 15:32 | |
| *** aufi has quit IRC | 15:34 | |
| EmilienM | mah: unfortunatly, opnfv is a fork of TripleO, it makes debugging harder for TripleO Community as we don't test it against upstream gate nor nobody from our group is working on it. | 15:35 |
| EmilienM | maybe if you give more logs and details we can help | 15:35 |
| EmilienM | matbu: are you still working on https://review.openstack.org/#/c/370069/ ? | 15:36 |
| *** ccamacho is now known as ccamacho|away | 15:36 | |
| *** dbecker has quit IRC | 15:37 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Generate internal TLS hieradata for apache services https://review.openstack.org/366075 | 15:38 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Add flag for internal TLS https://review.openstack.org/365942 | 15:38 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Add HAProxy TLS handled by certmonger as composable service https://review.openstack.org/356430 | 15:38 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Add option to specify Certmonger CA https://review.openstack.org/368643 | 15:38 |
| matbu | EmilienM: i think i will abandon it | 15:39 |
| matbu | bandini: did you need to apply this fix for the init step ? https://review.openstack.org/#/c/370069/ | 15:39 |
| *** athomas has quit IRC | 15:39 | |
| bandini | matbu: nope, it is weird I have never seen that issue happen lately | 15:41 |
| matbu | bandini: yes me too, so i'll abandon the change and close the LP | 15:41 |
| Jokke_ | EmilienM: thanks ... you saved me being politically incorrect on gerrit ;) | 15:41 |
| bandini | matbu: ack | 15:42 |
| Jokke_ | noticed that recheck before hitting post | 15:42 |
| trown | EmilienM: with no ETA on launchpad coming back, and https://review.openstack.org/#/c/376297/ being both pretty trivial and blocking RDO CI, can we get by with no bug? | 15:42 |
| matbu | EmilienM: i'll do that probably tomorow morning, i just want to make sure we don't need anymore | 15:43 |
| EmilienM | trown: yes let's go4it | 15:43 |
| *** morazi has quit IRC | 15:43 | |
| *** dsariel has quit IRC | 15:43 | |
| -openstackstatus- NOTICE: Earlier job failures for "zuul-cloner: error: too few arguments" should now be solved, and can safely be rechecked | 15:44 | |
| *** akshai has quit IRC | 15:44 | |
| *** dprince has quit IRC | 15:47 | |
| -openstackstatus- NOTICE: Launchpad SSO is offline, preventing login to https://review.openstack.org/, https://wiki.openstack.org/ and many other sites; no ETA has been provided by the LP admin team | 15:50 | |
| *** ChanServ changes topic to "Launchpad SSO is offline, preventing login to https://review.openstack.org/, https://wiki.openstack.org/ and many other sites; no ETA has been provided by the LP admin team" | 15:50 | |
| bnemec | Outstanding | 15:50 |
| *** shardy has joined #tripleo | 15:51 | |
| trown | really good timing :) | 15:51 |
| panda | you can all go home now. | 15:51 |
| panda | wait ... | 15:51 |
| *** rcernin has quit IRC | 15:52 | |
| *** zoliXXL is now known as zoli|gone | 15:54 | |
| *** panda is now known as panda|bbl | 15:54 | |
| *** pgadiya has quit IRC | 15:54 | |
| *** zoli|gone is now known as zoli_gone-proxy | 15:54 | |
| jaosorior | mwhahaha: hey, could you take a look at this patch? https://review.openstack.org/#/c/366548/ it's WIP since that work is meant for Ocata, that's why I -1W it. But would really use some input there. | 15:55 |
| mwhahaha | Sure | 15:55 |
| *** tzumainn has joined #tripleo | 15:55 | |
| *** akshai has joined #tripleo | 15:57 | |
| trown | ah launchpad is back | 16:00 |
| jaosorior | mwhahaha: the t-h-t patch is this https://review.openstack.org/#/c/356430/ | 16:02 |
| *** lucasagomes is now known as lucas-hungry | 16:03 | |
| openstackgerrit | Miles Gould proposed openstack/tripleo-docs: WIP: document process for deploying on UEFI nodes https://review.openstack.org/376612 | 16:04 |
| mah | The problem occurs at the post install configuration ... it stucks at the neutron external network configuration, I knew that the problem was related to undercloud and overcloud ... making authentication in requesting tokens .. all commands stucks at this point | 16:05 |
| EmilienM | jpich: are we keeping https://bugs.launchpad.net/tripleo/+bug/1626621 for rc2? | 16:06 |
| openstack | Launchpad bug 1626621 in tripleo "Remove invalid Deprecation warning for bulk introspection" [Medium,In progress] - Assigned to Julie Pichon (jpichon) | 16:06 |
| EmilienM | mandre: do you know if there is progress on https://bugs.launchpad.net/tripleo/+bug/1626481 ? | 16:07 |
| openstack | Launchpad bug 1626481 in tripleo "Normalize and refactor Deployment data in App State" [Medium,In progress] - Assigned to Jiri Tomasek (jtomasek) | 16:07 |
| *** dprince has joined #tripleo | 16:07 | |
| jpich | EmilienM: There is some disagreement in the review comments. I'd rather get rid of the warning and reintroduce it the Ocata client, but I won't fight too hard for it, it's not the *worst* user experience either... | 16:07 |
| EmilienM | jpich: ok fair enough. If you feel like we won't make it on time, please defer it to ocata-1 | 16:08 |
| jpich | EmilienM: Ok! rc2 is tomorrow? Thursday? | 16:09 |
| EmilienM | jpich: Thursday | 16:09 |
| jpich | EmilienM: Understood, thank you | 16:09 |
| EmilienM | chem: any progress on https://bugs.launchpad.net/tripleo/+bug/1626592 ? | 16:10 |
| openstack | Launchpad bug 1626592 in tripleo "M/N upgrade, statsd doesn't restart." [High,In progress] - Assigned to Sofer Athlan-Guyot (sofer-athlan-guyot) | 16:10 |
| gfidente | chem, I commented on https://review.openstack.org/#/c/376509/ | 16:10 |
| gfidente | if you agree and update it I will +2 again | 16:10 |
| social | rhallisey: mError: /Stage[main]/Tripleo::Profile::Base::Rabbitmq/Rabbitmq_policy[ha-all@/]: Could not evaluate: Command is still failing after 180 seconds expired! | 16:11 |
| social | rhallisey: what about you? | 16:11 |
| rhallisey | I've had that in the past | 16:11 |
| *** ChanServ changes topic to "TripleO : http://tripleo.org/ | https://wiki.openstack.org/wiki/TripleO | Meetings On Tuesdays at 14:00 UTC in #openstack-meeting-alt" | 16:12 | |
| -openstackstatus- NOTICE: Launchpad SSO logins are confirmed working correctly again | 16:12 | |
| rhallisey | it didn't reliably reproduce because something new would show up | 16:12 |
| *** david-lyle has quit IRC | 16:13 | |
| *** david-lyle has joined #tripleo | 16:14 | |
| EmilienM | apetrich: any progress on https://bugs.launchpad.net/tripleo/+bug/1623673 ? | 16:15 |
| openstack | Launchpad bug 1623673 in tripleo "mixed deployment fails resources.ControllerServicesBaseDeployment_Step2 with status code: 6 " [Critical,Triaged] - Assigned to Adriano Petrich (apetrich) | 16:15 |
| *** limao_ has quit IRC | 16:16 | |
| *** limao has joined #tripleo | 16:16 | |
| cmyster | 8 | 16:16 |
| EmilienM | panda|bbl: I renamed the bug report https://bugs.launchpad.net/tripleo/+bug/1605363 | 16:20 |
| openstack | Launchpad bug 1605363 in tripleo "TripleO CI job timeouts with IPv6 scenario" [Critical,In progress] - Assigned to Gabriele Cerami (gcerami) | 16:20 |
| EmilienM | and defer it to ocata | 16:20 |
| *** limao has quit IRC | 16:20 | |
| EmilienM | if your patch is for RC2 please use tripleo/rc2 gerrit topic | 16:21 |
| mah | The problem occurs at the post install configuration ... it stucks at the neutron external network configuration, I knew that the problem was related to undercloud and overcloud ... making authentication in requesting tokens .. all commands stucks at this point | 16:22 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Fix ignore warning on ceph major upgrade. https://review.openstack.org/376509 | 16:23 |
| apetrich | EmilienM, not really. Still debugging it. all that I know is that ControllerServicesBaseDeployment_Step2 fails with return code 6 it depends on ControllerLoadBalancerDeployment_Step1 that finishes fine. | 16:23 |
| EmilienM | matbu: when you have a moment, please update https://bugs.launchpad.net/tripleo/+bug/1620696 | 16:24 |
| openstack | Launchpad bug 1620696 in tripleo "M/N upgrades UPDATE_FAILED .enabled_services.list_join: Incorrect arguments to "list_join" should be: "list_join" : [ " ", [ "str1", "str2"]]" [Critical,In progress] - Assigned to mbu (mat-bultel) | 16:24 |
| jaosorior | mah: the undercloud should have access to the ctlplane network. Before newton, post-config was done from the undercloud to the overcloud; and that's probably what you're seeing failing. | 16:24 |
| shadower | dsneddon: ping | 16:24 |
| dsneddon | shadower, Hello | 16:25 |
| jaosorior | mah: by default the keystone admin network is working on the ctlplane network. | 16:25 |
| mah | yes exactly | 16:25 |
| mah | I am using mitaka | 16:25 |
| shadower | dsneddon: heya. Are you aware of the situation where a DHCP server on a pacemaker-managed network can cause trouble? | 16:25 |
| rhallisey | social, just had an update work | 16:26 |
| dsneddon | shadower, What do you mean by "pacemaker-managed network"? | 16:26 |
| *** bana_k has joined #tripleo | 16:26 | |
| shadower | dsneddon: so, I'm a little fuzzy on this. But it goes something like this I think: we have nodes managed by pacemaker. And if those nodes get a dhcp response pacemaker doesn't know about, it may think the node is down and can fence it | 16:27 |
| *** ebarrera has joined #tripleo | 16:27 | |
| gfidente | b00tcat, not sure if you're the author but seems like there is a typo in https://review.openstack.org/#/c/376291/1 ? | 16:28 |
| dsneddon | shadower, What is getting the DHCP response from the rogue server, the node or the BMC? | 16:28 |
| *** pcaruana has quit IRC | 16:29 | |
| dsneddon | shadower, Because in most of our labs the BMC is on a separate VLAN, and nobody should be hosting a DHCP server there. If you are talking about the node, then you can't use network isolation when there are DHCP servers on any of your overcloud networks. Only the Undercloud can serve DHCP. | 16:30 |
| shadower | dsneddon: what's a BMC? Sorry, not my area of expectise but I'm trynig to learn this | 16:31 |
| *** enb201 has joined #tripleo | 16:31 | |
| dsneddon | shadower, BMC == DRAC or iLO or IPMI controller. | 16:32 |
| shadower | dsneddon: basically, at some point I was asked to write a validation to detect rogue DHCP servers because they may badly interact with pacemaker. | 16:32 |
| shadower | ah right | 16:32 |
| shadower | dsneddon: and so maybe more generally -- in which situations would a DHCP server be problematic? | 16:32 |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder: Create runtime-ssh-host-keys element https://review.openstack.org/374366 | 16:32 |
| dsneddon | shadower, A DHCP server would be problematic on any network used for an OpenStack deployment. | 16:33 |
| bnemec | Our nodes shouldn't be DHCP'ing except on initial deployment, should they? | 16:33 |
| bnemec | We statically configure everything the first time os-net-config runs. | 16:33 |
| dsneddon | shadower, The reason is the dhcp-all-interfaces script which will enable DHCP on all interfaces as a failsafe if network isolation config doesn't work. The intention is for the control plane to be the only interface which gets DHCP response, but since we don't know which interface that is at that stage, we enable DHCP on all interfaces. | 16:34 |
| dsneddon | shadower, So obviously if you have a DHCP server on one of your overcloud networks, that can actually cause problems with connectivity back to the Undercloud. | 16:34 |
| shadower | dsneddon: right | 16:34 |
| shadower | dsneddon: is there a way I could tell which networks to check for? I assume they're passed to the Heat templates | 16:35 |
| dsneddon | shadower, Do you know which network the DHCP IP is appearing on? | 16:36 |
| *** abehl has quit IRC | 16:36 | |
| shadower | dsneddon: this isn't a problem I'm seeing right now. But I'd like to write a validation that checks for DHCP responses on networks where that could cause problems | 16:37 |
| dsneddon | bnemec, You are right that since we configure static interfaces with os-net-config this isn't always a problem. It depends on whether the DHCP server is attached via a trunked VLAN or on the native VLAN. | 16:37 |
| leifmadsen | lhinds: o/ | 16:38 |
| dsneddon | shadower, OK, well, you can't do that on the Undercloud, the best place to do that is in ironic-python-agent, that is where we run LLDP collection to get all the physical networking data (soon that data will be stored in the ironic DB). | 16:38 |
| shadower | dsneddon: ah okay. Why can't this happen on the undercloud? | 16:39 |
| shadower | because not all networks are necessarily reachable from the UC? | 16:39 |
| dsneddon | shadower, Because the Undercloud isn't attached to all overcloud networks, only the provisioning and sometimes external (although the external can be reached through routing). | 16:40 |
| shadower | dsneddon: right, thanks. | 16:40 |
| dsneddon | shadower, Although we could write a check for DHCP servers on the provisioning net during undercloud installation. | 16:41 |
| shadower | yeah | 16:41 |
| dsneddon | shadower, BTW, can you please update Bob Fournier with your progress. He is working on compiling a list of future validations and slowly implementing them. Here is the full list, feel free to comment: https://docs.google.com/a/redhat.com/document/d/1A5rZ4lZCbv9ZvDCcOKCPFqIs9hgVN8BW3Wr3k4zIU04/edit?usp=sharing | 16:41 |
| shadower | dsneddon: yeah, will do | 16:42 |
| shadower | thanks | 16:42 |
| dsneddon | shadower, If you haven't worked with Bob Fournier (bfournie), he is a newish software engineer with loads of networking expertise. | 16:42 |
| dsneddon | shadower, He is helping to make the UI work with networking, and making general improvements. | 16:42 |
| *** dmacpher has joined #tripleo | 16:43 | |
| shadower | dsneddon: we've crossed paths. I think he's sent me that doc some time ago but I got sidetracked with other stuff | 16:43 |
| dsneddon | shadower, I'm really glad someone is working on this, don't be shy about bugging me if you need help. | 16:43 |
| shadower | dsneddon: thanks, I'll do that. Networking is not my strong suite (but I'd like to fix that) | 16:44 |
| shadower | dsneddon: oh so, how could I detect the provisioning network/interface? | 16:45 |
| *** shardy has quit IRC | 16:45 | |
| bnemec | panda|bbl: I reproduced the rabbitmq failure locally. The strange thing is that rabbit appears to be running. | 16:45 |
| bnemec | I'm wondering if this is another stupid rabbit hostname thing. | 16:45 |
| panda|bbl | bnemec: even on controllers 1 and 2 ? | 16:45 |
| shadower | ironic.conf? | 16:45 |
| *** panda|bbl is now known as panda | 16:45 | |
| panda | bnemec: I see the logs is full of crash reports | 16:46 |
| bnemec | panda: pcs status says it's stopped everywhere: Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] | 16:46 |
| panda | bnemec: I have this Clone Set: rabbitmq-clone [rabbitmq] | 16:46 |
| panda | Started: [ overcloud-controller-0 ] | 16:46 |
| panda | Stopped: [ overcloud-controller-1 overcloud-controller-2 ] | 16:46 |
| *** ohamada has quit IRC | 16:47 | |
| bnemec | I guess that's not surprising. This is apparently a racy bug. | 16:47 |
| panda | bnemec: this is what happens also on CI job http://logs.openstack.org/74/363674/27/experimental-tripleo/gate-tripleo-ci-centos-7-ovb-ha-ipv6/74c9d65/logs/overcloud-controller-0/var/log/host_info.txt.gz | 16:47 |
| bnemec | Lots of Error: badarg's in my startup_log. | 16:47 |
| dsneddon | shadower, There are a bunch of "dhcp ping" type scripts out there. I can come up with one if you have trouble finding one. | 16:48 |
| panda | bnemec: if you check the sasl log, lots of crash reports | 16:48 |
| *** enb201 has quit IRC | 16:49 | |
| shadower | dsneddon: I've cooked something up using scapy, but I'd appreciate a second pair of eyes (and also not sure if we can actually distribute it becaues scapy is GPL) | 16:49 |
| shadower | dsneddon: here's what I've got: https://review.openstack.org/#/c/353973/2/validations/files/rogue_dhcp.py | 16:49 |
| openstackgerrit | Dan Prince proposed openstack/puppet-tripleo: Move db syncs into mysql base role https://review.openstack.org/376633 | 16:49 |
| *** dprince has quit IRC | 16:50 | |
| dsneddon | shadower, Yeah, I would stay away from scapy. I can help you do it with raw sockets. | 16:50 |
| jaosorior | mwhahaha: hey, I answered on https://review.openstack.org/#/c/366548 thanks a lot for checking it out. just one thing is that <service>_network is generated automatically by tripleo-heat-templates. And so should be the hash that contains the cert info. Do you still think I should sanity check that? | 16:50 |
| shadower | dsneddon: yeah, I'd appreciate that. | 16:50 |
| mwhahaha | trust but verify | 16:50 |
| *** Steve__ has quit IRC | 16:50 | |
| mwhahaha | jaosorior: so yea it's better to have a failure message with what happened then some terse puppet error | 16:50 |
| jaosorior | mwhahaha: alright. I'll push the sanity-check of that tomorrow. Thanks | 16:51 |
| *** jpena is now known as jpena|off | 16:51 | |
| *** akshai has quit IRC | 16:52 | |
| *** akshai has joined #tripleo | 16:53 | |
| *** derekh has quit IRC | 16:53 | |
| *** pleia2_ is now known as pleia2 | 16:53 | |
| panda | bnemec: anything I could look at to help ? I tried to follow the traceback in rabbit crash report, but it's not easy ... the function that crashes is "run(M,F,A)". helpful :( | 16:55 |
| *** lucas-hungry is now known as lucasagomes | 16:57 | |
| bnemec | I just _love_ debugging big json dumps. | 16:57 |
| bnemec | /sarcasm | 16:57 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Enable TLS in the internal network for keystone https://review.openstack.org/327029 | 16:58 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Fetch internal certificates for HAProxy based on network https://review.openstack.org/366548 | 16:58 |
| jaosorior | mwhahaha: well, did a quick one now. What do you think? ^^ | 16:58 |
| bnemec | panda: I pinged one of our rabbit people. Maybe he'll have some idea what's going on. | 16:58 |
| openstackgerrit | Julie Pichon proposed openstack/python-tripleoclient: Remove deprecation warning for bulk introspection https://review.openstack.org/376639 | 16:58 |
| *** akshai has quit IRC | 16:58 | |
| mwhahaha | jaosorior: yea something like that but I think you need to either use verify_string() or just !$service_network to handle undef | 16:59 |
| panda | bnemec: ok | 16:59 |
| jaosorior | mwhahaha: verify_string? | 17:00 |
| *** akshai has joined #tripleo | 17:00 | |
| jaosorior | I don't find reference to that function | 17:00 |
| mwhahaha | validate_string | 17:00 |
| mwhahaha | sorry | 17:00 |
| mwhahaha | brain not functioning yet | 17:00 |
| *** mgould has left #tripleo | 17:00 | |
| mwhahaha | actually validate_string doesn't fail for undef either | 17:01 |
| mwhahaha | so it would be better to just use !$service_network | 17:01 |
| *** bana_k has quit IRC | 17:01 | |
| jaosorior | ok | 17:01 |
| mwhahaha | since it's likely that it could be undef due to the hiera default value of undef being used in the base haproxy class | 17:02 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Enable TLS in the internal network for keystone https://review.openstack.org/327029 | 17:02 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Fetch internal certificates for HAProxy based on network https://review.openstack.org/366548 | 17:02 |
| *** trown is now known as trown|lunch | 17:03 | |
| bnemec | mah: I'm not sure what would cause that. We don't do any external network configuration at deploy time in TripleO. | 17:03 |
| bnemec | mah: Although make sure you don't have a proxy set. I recall that causing issues talking to keystone in the past. | 17:03 |
| mwhahaha | jaosorior: looks good | 17:04 |
| jaosorior | mwhahaha: thanks for the input | 17:05 |
| *** fzdarsky is now known as fzdarsky|afk | 17:06 | |
| *** akuznetsov has quit IRC | 17:06 | |
| *** dprince has joined #tripleo | 17:07 | |
| *** jaosorior has quit IRC | 17:07 | |
| mah | It occurs at the post install configuration | 17:07 |
| dprince | EmilienM: there is nothing risky about creating databases locally on the MySQL server I think | 17:08 |
| dprince | EmilienM: the actual db_syncs can occur wherever... | 17:08 |
| *** jpich has quit IRC | 17:10 | |
| *** fzdarsky|afk has quit IRC | 17:11 | |
| dprince | EmilienM: in my patch I'm only changing where the database gets created (moving it to the local DB). I'm not changing where the db_syncs actually occur for any of the services | 17:11 |
| dprince | EmilienM: I think my commit message confused you. I will fix it. But please review the code | 17:12 |
| openstackgerrit | Dan Prince proposed openstack/puppet-tripleo: Move db creation into mysql base role https://review.openstack.org/376633 | 17:12 |
| EmilienM | dprince: sure I'm probably wrong, let me review again | 17:13 |
| bnemec | mah: Can you curl the keystone endpoint? Something like "curl http://10.0.0.9:5000/v2.0" | 17:14 |
| bnemec | With whatever your overcloud public endpoint acutally is, of course. | 17:14 |
| dprince | EmilienM: There is an associated t-h-t patch that will follow this. But if you don't like what I'm doing let me know and I'll stop that work | 17:15 |
| EmilienM | dprince: I found a problem, let me comment | 17:16 |
| mah | it is reachable | 17:16 |
| EmilienM | dprince: done | 17:17 |
| dprince | EmilienM: ack, so you buy the approach the? | 17:17 |
| dprince | EmilienM: then | 17:17 |
| dprince | EmilienM: problems are easy to fix.... :) | 17:17 |
| bnemec | mah: Do plain keystone commands work? | 17:18 |
| *** cdearborn has joined #tripleo | 17:18 | |
| EmilienM | dprince: yes | 17:18 |
| EmilienM | dprince: I like it | 17:18 |
| mah | lets try | 17:21 |
| mah | no : DEBUG:keystoneclient.auth.identity.v2:Making authentication request to http://192.168.162.13:5000/v2.0/tokens INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): 192.168.162.13 | 17:22 |
| openstackgerrit | Dan Prince proposed openstack/puppet-tripleo: Move db syncs into mysql base role https://review.openstack.org/376633 | 17:22 |
| mah | then stucks after that | 17:23 |
| *** bana_k has joined #tripleo | 17:23 | |
| EmilienM | mah: do you have mysql running? | 17:24 |
| mah | yes | 17:24 |
| bnemec | mah: Can you curl that tokens address? | 17:24 |
| mah | I can curl yes | 17:24 |
| mah | but it gave it can found it | 17:25 |
| mah | 404 error | 17:25 |
| bnemec | That would be "no". :-) | 17:26 |
| *** pkovar has quit IRC | 17:26 | |
| bnemec | Oh, actually I get the same thing locally. | 17:27 |
| bnemec | Weird. | 17:27 |
| mah | yes | 17:28 |
| panda | ok, from where does /usr/lib/lib/ocf/resource.d/heartbeat/rabbitmq-cluster come from ? no package perovides it ... | 17:28 |
| *** jprovazn has quit IRC | 17:30 | |
| * beagles notes that service_config_settings stuff is pretty wild | 17:32 | |
| *** tosky has quit IRC | 17:35 | |
| *** flepied has joined #tripleo | 17:37 | |
| bnemec | mah: Can you paste the full --debug output of the hanging command? | 17:39 |
| EmilienM | panda: have you tried to query rpm? | 17:41 |
| EmilienM | let me find the package again, I remember jistr contributing on it | 17:42 |
| *** egafford has joined #tripleo | 17:42 | |
| *** milan is now known as milan|afk | 17:43 | |
| EmilienM | panda: https://github.com/ClusterLabs/resource-agents | 17:43 |
| EmilienM | so the code is here: https://github.com/ClusterLabs/resource-agents/blob/master/heartbeat/rabbitmq-cluster | 17:43 |
| EmilienM | and the package now | 17:43 |
| *** dtantsur is now known as dtantsur|afk | 17:44 | |
| EmilienM | panda: probably resource-agents | 17:45 |
| gfidente | panda, resource-agents yes | 17:45 |
| gfidente | panda, https://github.com/ClusterLabs/resource-agents | 17:45 |
| *** akshai has quit IRC | 17:45 | |
| EmilienM | gfidente: thx for confirming | 17:46 |
| *** lblanchard has quit IRC | 17:46 | |
| gfidente | panda, though you should see it owned by the package | 17:46 |
| gfidente | looks like you have /lib/ twice in the path? | 17:46 |
| panda | gfidente: just because I've written the missin path by hand .. | 17:47 |
| gfidente | EmilienM, ah didn't notice you pointed out the url as well a few lines above | 17:48 |
| gfidente | I know, unbelievable | 17:48 |
| panda | I'm also asking on another channel but it seems that the package is not installed. The file is there but the package is not. Anyway, we're trying to track down an error in rabbitmq on controllers 1 and 2 | 17:51 |
| panda | gfidente: ^ | 17:52 |
| *** flepied has quit IRC | 17:54 | |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Move db::mysql into service_config_settings https://review.openstack.org/376984 | 17:55 |
| *** panda is now known as panda|dinner | 17:55 | |
| *** amoralej is now known as amoralej|off | 17:58 | |
| EmilienM | gfidente: can you review https://review.openstack.org/#/c/373500/ please ? | 18:00 |
| EmilienM | dprince: would you be able to test https://review.openstack.org/#/c/367324/ in the tripleo ci dashboard? | 18:00 |
| openstackgerrit | Honza Pokorny proposed openstack/tripleo-common: Support node untagging https://review.openstack.org/372628 | 18:10 |
| *** trown|lunch is now known as trown | 18:10 | |
| openstackgerrit | Honza Pokorny proposed openstack/tripleo-common: Support node untagging https://review.openstack.org/372628 | 18:16 |
| *** mburned_` is now known as mburned_out | 18:17 | |
| *** rook_ is now known as rook | 18:19 | |
| *** onovy has quit IRC | 18:28 | |
| *** panda|dinner is now known as panda | 18:29 | |
| dtrainor | oslo_config.cfg.DuplicateOptError: duplicate option: undercloud_zaqar_password | 18:32 |
| dtrainor | anyone seen that one before? | 18:32 |
| *** akshai has joined #tripleo | 18:34 | |
| *** cylopez has quit IRC | 18:36 | |
| *** akshai has quit IRC | 18:37 | |
| *** akshai has joined #tripleo | 18:38 | |
| dtrainor | oh i see. | 18:38 |
| *** enb201 has joined #tripleo | 18:40 | |
| *** rasca has quit IRC | 18:41 | |
| openstackgerrit | Sarath Kumar Sankaran Kutty proposed openstack/os-net-config: Add extra_options attribute to NFVSwitch https://review.openstack.org/375763 | 18:42 |
| *** akshai has quit IRC | 18:43 | |
| dprince | EmilienM: You can run the CI dashboard report locally. It should generate HTML that is viewable for a local machine to see how wide it is | 18:46 |
| *** dprince has quit IRC | 18:52 | |
| *** ccamacho|away has quit IRC | 18:52 | |
| *** dprince has joined #tripleo | 18:53 | |
| *** lblanchard has joined #tripleo | 18:54 | |
| openstackgerrit | John Trowbridge proposed openstack-infra/tripleo-ci: Move ENV var setting to a common function https://review.openstack.org/377035 | 18:59 |
| *** enb201 has quit IRC | 19:01 | |
| *** enb2011 has joined #tripleo | 19:01 | |
| bkero | trown: got a spare space at the end of line 405 :) | 19:03 |
| bkero | gerrit's highlighting it in red for me | 19:03 |
| openstackgerrit | Tom Barron proposed openstack/tripleo-common: DO NOT MERGE - Test setting SWIFT_INTERNAL_URL https://review.openstack.org/377045 | 19:05 |
| trown | bkero: k, need to setup my virthost with vim plugin that fixes that automagically | 19:05 |
| bkero | I didn't want to be "that guy" that -1s for whitespace :) | 19:06 |
| trown | bkero: working on what it would take to make it possible to do "CI_JOB=ha tripleo.sh --virt-setup" and end up with a local virtual deployment capable of reproducing that job from tripleo-ci | 19:07 |
| bkero | trown: I'd love to be able to reproduce those on my local virthost. | 19:07 |
| trown | bkero: also got tabs in there :( yuck, thats what I get for submitting directly from the virthost where I was testing it | 19:13 |
| bkero | yeah | 19:20 |
| *** trown is now known as trown|brb | 19:28 | |
| *** trown|brb is now known as trown | 19:31 | |
| *** enb2011 has quit IRC | 19:31 | |
| *** enb201 has joined #tripleo | 19:35 | |
| *** eggmaster has joined #tripleo | 19:39 | |
| *** onovy has joined #tripleo | 19:40 | |
| *** dsariel has joined #tripleo | 19:43 | |
| *** crinkle_ is now known as crinkle | 19:45 | |
| *** florianf has quit IRC | 19:51 | |
| *** enb201 has quit IRC | 20:03 | |
| *** dsariel has quit IRC | 20:04 | |
| openstackgerrit | John Trowbridge proposed openstack-infra/tripleo-ci: Move ENV var setting to a common function https://review.openstack.org/377035 | 20:04 |
| *** trozet has quit IRC | 20:08 | |
| *** ebarrera has quit IRC | 20:12 | |
| *** lucasagomes is now known as lucas-afk | 20:21 | |
| *** akshai has joined #tripleo | 20:21 | |
| *** flepied has joined #tripleo | 20:21 | |
| *** enb201 has joined #tripleo | 20:22 | |
| *** akshai has quit IRC | 20:22 | |
| *** jayg is now known as jayg|g0n3 | 20:22 | |
| openstackgerrit | Dan Sneddon proposed openstack/tripleo-specs: Spec: tripleo-routed-networks-deployment - Routed Spine and Leaf https://review.openstack.org/377085 | 20:32 |
| *** egafford has quit IRC | 20:34 | |
| openstackgerrit | Lokesh Jain proposed openstack/tripleo-heat-templates: Updates for configuring Nuage in Mitaka release https://review.openstack.org/372749 | 20:34 |
| *** egafford has joined #tripleo | 20:34 | |
| *** trown is now known as trown|outtypewww | 20:38 | |
| openstackgerrit | Merged openstack/python-tripleoclient: Keystone credentials needs to be set with the overcloud password https://review.openstack.org/376449 | 20:38 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates: Update gnocchi database during M/N upgrade. https://review.openstack.org/374884 | 20:42 |
| openstackgerrit | Dan Sneddon proposed openstack/tripleo-specs: Spec: tripleo-routed-networks-deployment - Routed Spine and Leaf https://review.openstack.org/377088 | 20:42 |
| openstackgerrit | Merged openstack/puppet-tripleo: Add pameter for gmcast.listen_addr configuration https://review.openstack.org/376443 | 20:42 |
| *** dsneddon is now known as dsneddon_lunch | 20:44 | |
| *** egafford has quit IRC | 20:47 | |
| *** mah has quit IRC | 20:48 | |
| tbarron | Deploy of manila with everything fresh and the outstanding tripleo/rc2 patch https://review.openstack.org/#/c/358525/ configures manila as expected and starts both systemd and pcs controlled services as expected but fails with | 20:50 |
| tbarron | Error: Puppet::Type::Keystone_user_role::ProviderOpenstack: Could not find project with name [services] and domain [Default] | 20:50 |
| tbarron | Error: /Stage[main]/Manila::Keystone::Auth/Keystone::Resource::Service_identity[manilav2]/Keystone_user_role[manilav2@services]: Could not evaluate: undefined method `[]' for nil:NilClass | 20:50 |
| tbarron | Anyone recognize these Keystone Service_identity errors (same for manilav1 btw) ?? | 20:51 |
| tbarron | I'm wondering whether I'm missing an outstanding patch for OOO/keystone or whether this is likely a manila-specific issue. | 20:51 |
| *** ebalduf has joined #tripleo | 20:52 | |
| *** Goneri has quit IRC | 20:53 | |
| *** absubram has quit IRC | 20:54 | |
| tbarron | ^^ when I say "everything fresh", fresh undercloud and packages this morning, fresh git tree for THT, and and puppet-tripleo deployed via upload-puppet-modules | 20:54 |
| enb201 | I am trying to deploy to deploy TripleO Quickstart... | 21:02 |
| enb201 | I see the follow error.. | 21:02 |
| enb201 | PLAY RECAP ********************************************************************* | 21:02 |
| enb201 | localhost : ok=17 changed=8 unreachable=0 failed=0 | 21:02 |
| enb201 | tripleo : ok=105 changed=37 unreachable=0 failed=0 | 21:02 |
| enb201 | undercloud : ok=7 changed=4 unreachable=1 failed=0 | 21:02 |
| enb201 | The undercloud is accessible. | 21:03 |
| enb201 | $ ssh -F /home/jamie/.quickstart/ssh.config.ansible undercloud | 21:03 |
| enb201 | Warning: Permanently added 'tripleo,169.47.5.183' (ECDSA) to the list of known hosts. | 21:03 |
| enb201 | Warning: Permanently added 'undercloud' (ECDSA) to the list of known hosts. | 21:03 |
| enb201 | Last login: Mon Sep 26 20:57:00 2016 from gateway | 21:03 |
| enb201 | [stack@undercloud ~]$ | 21:03 |
| enb201 | TASK [tripleo/undercloud : Install the undercloud] ***************************** | 21:03 |
| enb201 | task path: /home/jamie/tripleo-quickstart/roles/tripleo/undercloud/tasks/install-undercloud.yml:15 | 21:03 |
| enb201 | Monday 26 September 2016 16:38:26 -0400 (0:00:06.152) 0:10:35.891 ****** | 21:03 |
| enb201 | fatal: [undercloud]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh.", "unreachable": true} | 21:03 |
| bkero | enb201: I've had luck adding ssh_args="-F /home/jamie/.quickstart/ssh.config.ansible" to ansible.cfg | 21:05 |
| *** dtrainor has quit IRC | 21:05 | |
| enb201 | thanks | 21:10 |
| *** chem has quit IRC | 21:12 | |
| *** ebalduf has quit IRC | 21:13 | |
| *** apetrich has quit IRC | 21:13 | |
| *** apetrich has joined #tripleo | 21:14 | |
| *** chem has joined #tripleo | 21:14 | |
| *** dprince has quit IRC | 21:18 | |
| pabelanger | EmilienM: are you tracking the tripleo job timeout issue any place? Here is another that is taking the full job timeout to report failures: http://logs.openstack.org/39/376639/1/check/gate-tripleo-ci-centos-7-nonha-multinode-updates-nv/c13e65d/console.html#_2016-09-26_19_54_00_564947 | 21:22 |
| *** akshai has joined #tripleo | 21:24 | |
| *** ayoung has quit IRC | 21:26 | |
| *** akshai has quit IRC | 21:29 | |
| *** dtrainor has joined #tripleo | 21:29 | |
| openstackgerrit | Ryan Hallisey proposed openstack/tripleo-specs: Containerize TripleO blueprint https://review.openstack.org/223182 | 21:30 |
| *** akshai has joined #tripleo | 21:30 | |
| *** akshai has quit IRC | 21:31 | |
| *** akshai has joined #tripleo | 21:32 | |
| *** akshai has quit IRC | 21:32 | |
| EmilienM | pabelanger: i'm not, I'm supposed to be on PTO this afternoon :) | 21:33 |
| EmilienM | pabelanger: but let me check if I can build a logstash query | 21:33 |
| EmilienM | we have an etherpad for queries https://etherpad.openstack.org/p/tripleo-ci-logstash-queries | 21:34 |
| *** r-mibu has quit IRC | 21:40 | |
| *** r-mibu has joined #tripleo | 21:40 | |
| openstackgerrit | Ben Nemec proposed openstack/tripleo-heat-templates: Fix predictable placement indexing https://review.openstack.org/377110 | 21:43 |
| *** social has quit IRC | 21:50 | |
| *** b00tcat has quit IRC | 21:50 | |
| EmilienM | pabelanger: yes I see it in graphs, trying to debug | 21:58 |
| EmilienM | the bad news, I see it on all cloud providers | 22:01 |
| EmilienM | I added the query at the end of https://etherpad.openstack.org/p/tripleo-ci-logstash-queries | 22:03 |
| openstackgerrit | Dan Sneddon proposed openstack/tripleo-specs: Spec: tripleo-routed-networks-deployment - Routed Spine and Leaf https://review.openstack.org/377088 | 22:04 |
| EmilienM | correction: it's only on rax cloud | 22:07 |
| *** trozet has joined #tripleo | 22:11 | |
| *** cdearborn has quit IRC | 22:13 | |
| *** ayoung has joined #tripleo | 22:14 | |
| *** social has joined #tripleo | 22:16 | |
| *** yolanda has quit IRC | 22:17 | |
| *** yolanda has joined #tripleo | 22:18 | |
| *** spredzy has quit IRC | 22:39 | |
| *** lblanchard has quit IRC | 22:41 | |
| *** spredzy has joined #tripleo | 22:41 | |
| *** social has quit IRC | 22:47 | |
| *** lucas-afk has quit IRC | 23:02 | |
| *** lucasagomes has joined #tripleo | 23:05 | |
| *** mhenkel has quit IRC | 23:10 | |
| *** fultonj has quit IRC | 23:12 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!