| *** apetrich has joined #tripleo | 00:00 | |
| *** mbound has quit IRC | 00:00 | |
| *** yamahata has quit IRC | 00:14 | |
| *** saneax is now known as saneax_AFK | 00:15 | |
| *** rcernin has joined #tripleo | 00:17 | |
| *** Lokesh_Jain has quit IRC | 00:17 | |
| *** rcernin has quit IRC | 00:22 | |
| *** panda has quit IRC | 00:24 | |
| *** panda has joined #tripleo | 00:25 | |
| *** apetrich has quit IRC | 00:29 | |
| *** apetrich has joined #tripleo | 00:30 | |
| *** TSCHAK_ has quit IRC | 00:31 | |
| *** TSCHAK has joined #tripleo | 00:34 | |
| openstackgerrit | Merged openstack/puppet-tripleo: add plumgrid neutron profile https://review.openstack.org/317259 | 00:35 |
|---|---|---|
| *** rcernin has joined #tripleo | 00:47 | |
| *** cwolferh has quit IRC | 00:50 | |
| *** rcernin has quit IRC | 00:53 | |
| *** weshay_mtg has quit IRC | 00:55 | |
| *** numans has joined #tripleo | 00:56 | |
| *** MaxPC has joined #tripleo | 00:56 | |
| *** saneax_AFK is now known as saneax | 01:00 | |
| *** mbound has joined #tripleo | 01:01 | |
| *** rcernin has joined #tripleo | 01:05 | |
| *** mbound has quit IRC | 01:06 | |
| *** weshay_mtg has joined #tripleo | 01:07 | |
| *** rcernin has quit IRC | 01:10 | |
| *** cwolferh has joined #tripleo | 01:11 | |
| *** MaxPC has quit IRC | 01:22 | |
| *** xinwu has quit IRC | 01:34 | |
| *** links has joined #tripleo | 01:54 | |
| *** jrist has quit IRC | 01:56 | |
| *** saneax is now known as saneax_AFK | 01:56 | |
| *** jrist has joined #tripleo | 02:02 | |
| *** coolsvap has joined #tripleo | 02:18 | |
| *** weshay_mtg has quit IRC | 02:28 | |
| *** r-mibu has quit IRC | 02:37 | |
| *** tzumainn has quit IRC | 02:37 | |
| *** julim has joined #tripleo | 02:38 | |
| *** lblanchard has quit IRC | 02:38 | |
| *** cmyster has quit IRC | 02:47 | |
| *** r-mibu has joined #tripleo | 02:47 | |
| *** apetrich has quit IRC | 02:56 | |
| *** apetrich has joined #tripleo | 02:56 | |
| *** links has quit IRC | 02:56 | |
| *** ramishra has joined #tripleo | 03:02 | |
| openstackgerrit | Numan Siddique proposed openstack/tripleo-puppet-elements: FOR TESTING ONLY... PLZ DONT MERGE https://review.openstack.org/328839 | 03:04 |
| *** apetrich has quit IRC | 03:10 | |
| *** apetrich has joined #tripleo | 03:10 | |
| *** saneax_AFK is now known as saneax | 03:14 | |
| *** fragatina has quit IRC | 03:22 | |
| *** morazi has quit IRC | 03:35 | |
| *** ramishra has quit IRC | 03:38 | |
| *** ramishra has joined #tripleo | 03:57 | |
| *** cllewellyn_ has joined #tripleo | 04:02 | |
| *** ramishra has quit IRC | 04:03 | |
| *** xinwu has joined #tripleo | 04:03 | |
| *** julim has quit IRC | 04:04 | |
| *** links has joined #tripleo | 04:05 | |
| *** ramishra has joined #tripleo | 04:13 | |
| *** apetrich has quit IRC | 04:19 | |
| *** apetrich has joined #tripleo | 04:20 | |
| *** panda has quit IRC | 04:24 | |
| *** panda has joined #tripleo | 04:25 | |
| *** apetrich has quit IRC | 04:25 | |
| *** apetrich has joined #tripleo | 04:26 | |
| *** cllewellyn__ has joined #tripleo | 04:27 | |
| *** skramaja has quit IRC | 04:29 | |
| *** oshvartz has quit IRC | 04:30 | |
| *** masco has joined #tripleo | 04:36 | |
| *** skramaja has joined #tripleo | 04:45 | |
| *** ramishra has quit IRC | 04:50 | |
| *** ramishra has joined #tripleo | 04:50 | |
| *** fragatina has joined #tripleo | 04:52 | |
| *** fragatina has quit IRC | 04:53 | |
| *** fragatina has joined #tripleo | 04:53 | |
| *** ramishra has quit IRC | 04:54 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Clear up "already provided" message https://review.openstack.org/290968 | 04:58 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Convert element_dependencies to logging https://review.openstack.org/328072 | 04:58 |
| *** ramishra has joined #tripleo | 05:00 | |
| *** jaosorior has joined #tripleo | 05:04 | |
| *** olap has quit IRC | 05:05 | |
| *** dixiaoli has quit IRC | 05:13 | |
| *** cllewellyn_ has quit IRC | 05:18 | |
| *** cllewellyn__ has quit IRC | 05:18 | |
| *** ramishra has quit IRC | 05:18 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Updates ControlPlaneSubnetCidr to be a string https://review.openstack.org/316233 | 05:18 |
| bandini | matbu: that is good news ;) | 05:18 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: profile/base/nova: declare nova class properly https://review.openstack.org/328347 | 05:19 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Remove usage of ::nova class in THT https://review.openstack.org/325983 | 05:19 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Composable Neutron Plumgrid plugin https://review.openstack.org/327307 | 05:20 |
| *** cllewellyn__ has joined #tripleo | 05:20 | |
| *** cllewellyn_ has joined #tripleo | 05:21 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Drop extraconfig for neutron-plumgrid.yaml https://review.openstack.org/327318 | 05:22 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Add fact to get the fqdn for a host in the different networks https://review.openstack.org/329299 | 05:23 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Drop galera_bootstrapped fact https://review.openstack.org/328979 | 05:24 |
| *** leanderthal|afk is now known as leanderthal | 05:30 | |
| *** apetrich has quit IRC | 05:44 | |
| *** apetrich has joined #tripleo | 05:44 | |
| *** ramishra has joined #tripleo | 05:45 | |
| *** ramishra has quit IRC | 05:48 | |
| *** tremble has quit IRC | 05:49 | |
| *** apetrich has quit IRC | 05:50 | |
| *** apetrich has joined #tripleo | 05:50 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Split Heat pacemaker roles into separate services https://review.openstack.org/327708 | 05:51 |
| *** yamahata has joined #tripleo | 05:52 | |
| *** oshvartz has joined #tripleo | 05:55 | |
| *** mbound has joined #tripleo | 05:55 | |
| *** apetrich has quit IRC | 05:56 | |
| *** coolsvap has quit IRC | 05:57 | |
| *** apetrich has joined #tripleo | 05:58 | |
| *** ramishra has joined #tripleo | 05:58 | |
| *** rlandy has quit IRC | 05:58 | |
| *** numans has quit IRC | 06:00 | |
| *** mbound has quit IRC | 06:00 | |
| *** cllewellyn_ has quit IRC | 06:00 | |
| *** cllewellyn__ has quit IRC | 06:00 | |
| *** ramishra has quit IRC | 06:03 | |
| *** saneax is now known as saneax_AFK | 06:04 | |
| *** coolsvap has joined #tripleo | 06:10 | |
| *** yolanda has joined #tripleo | 06:13 | |
| *** yolanda_ has joined #tripleo | 06:13 | |
| *** yolanda_ has quit IRC | 06:14 | |
| *** itamarl has joined #tripleo | 06:15 | |
| *** olap has joined #tripleo | 06:16 | |
| *** ramishra has joined #tripleo | 06:16 | |
| *** openstackgerrit has quit IRC | 06:18 | |
| *** openstackgerrit has joined #tripleo | 06:18 | |
| *** xinwu has quit IRC | 06:21 | |
| *** rook has quit IRC | 06:22 | |
| *** anshul has joined #tripleo | 06:23 | |
| *** anshul is now known as Guest34058 | 06:24 | |
| *** numans has joined #tripleo | 06:24 | |
| *** rcernin has joined #tripleo | 06:29 | |
| *** apetrich has quit IRC | 06:34 | |
| *** apetrich has joined #tripleo | 06:34 | |
| openstackgerrit | Martin André proposed openstack/tripleo-common: Allow running validation against different plans https://review.openstack.org/318194 | 06:46 |
| openstackgerrit | Martin André proposed openstack/tripleo-common: Disable retry files for ansible validations https://review.openstack.org/329039 | 06:46 |
| openstackgerrit | Martin André proposed openstack/tripleo-common: Validations actions and workbook https://review.openstack.org/313632 | 06:46 |
| *** aufi has joined #tripleo | 06:47 | |
| *** ifarkas has joined #tripleo | 06:49 | |
| *** rook has joined #tripleo | 06:49 | |
| *** athomas has joined #tripleo | 06:56 | |
| *** jprovazn has joined #tripleo | 06:57 | |
| openstackgerrit | Michele Baldessari proposed openstack/tripleo-heat-templates: Add redis constraint to aodh upgrade manifest https://review.openstack.org/329655 | 06:59 |
| *** cllewellyn__ has joined #tripleo | 07:01 | |
| *** cllewellyn_ has joined #tripleo | 07:01 | |
| *** tremble has joined #tripleo | 07:02 | |
| *** cwolferh has quit IRC | 07:02 | |
| *** tesseract has joined #tripleo | 07:03 | |
| *** fzdarsky has joined #tripleo | 07:03 | |
| *** rcernin has quit IRC | 07:04 | |
| *** rcernin has joined #tripleo | 07:04 | |
| *** florianf has joined #tripleo | 07:06 | |
| *** dtrainor has quit IRC | 07:09 | |
| tobias_fiberdata | the neutron-server service is timing out after reboot with the latest release | 07:11 |
| tobias_fiberdata | is this a known issue? | 07:12 |
| tobias_fiberdata | it's possible to start it afterwards though | 07:12 |
| *** saneax_AFK is now known as saneax | 07:13 | |
| matbu | tobias_fiberdata: reboot of the controller ? | 07:14 |
| matbu | tobias_fiberdata: on which release ? master ? | 07:14 |
| tobias_fiberdata | uhm, the tripleO server | 07:14 |
| tobias_fiberdata | i was not clear i believe. the latest mitaka based tripleO | 07:15 |
| *** milan has quit IRC | 07:17 | |
| *** ebarrera has joined #tripleo | 07:17 | |
| matbu | tobias_fiberdata: k, and what do you mean by tripleo server ? undercloud or overcloud ? | 07:19 |
| tobias_fiberdata | undercloud | 07:19 |
| tobias_fiberdata | i can priv you the logoutput | 07:19 |
| matbu | tobias_fiberdata: i experiment something for CI purpose, and i notice that the overcloud controller, when rebooting, sometimes the neutron-server is down | 07:20 |
| tobias_fiberdata | could put on verbose and debug if you want more details | 07:20 |
| tobias_fiberdata | could it be something similar in this case? but this is undercloud though | 07:20 |
| matbu | tobias_fiberdata: for the UC i never seen it before, but idk if mean of us try to reboot the nodes :) | 07:21 |
| matbu | tobias_fiberdata: yep maybe | 07:21 |
| matbu | tobias_fiberdata: could you fill a bug on launchpad ? | 07:21 |
| tobias_fiberdata | yea sure i could. I'll give myself some more details with verbose and debug | 07:22 |
| matbu | tobias_fiberdata: k thx | 07:22 |
| *** dtrainor has joined #tripleo | 07:22 | |
| *** shardy has joined #tripleo | 07:24 | |
| *** jpena|off is now known as jpena | 07:26 | |
| openstackgerrit | yolanda.robla proposed openstack/tripleo-quickstart: Allow to specify templates path on overcloud deployment https://review.openstack.org/329556 | 07:27 |
| tobias_fiberdata | matbu, ah well, i'll try to do it as fast as i can though. gotta prio our openstackdeployment first of all. Seems like Dell R610 is not very nice to me. | 07:31 |
| *** hjensas__ has joined #tripleo | 07:31 | |
| *** openstackgerrit has quit IRC | 07:33 | |
| *** openstackgerrit has joined #tripleo | 07:33 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Enable TLS in the internal network for keystone https://review.openstack.org/327029 | 07:34 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Enable TLS in the internal network for heat https://review.openstack.org/327069 | 07:34 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Enable TLS in the internal network for glance API and registry https://review.openstack.org/327473 | 07:35 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Enable TLS in the internal network for RabbitMQ https://review.openstack.org/327482 | 07:35 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Enable TLS in the internal network for cinder-api https://review.openstack.org/328859 | 07:35 |
| *** cmyster has joined #tripleo | 07:36 | |
| *** pino|work_ has joined #tripleo | 07:37 | |
| *** dsariel has joined #tripleo | 07:37 | |
| *** links has quit IRC | 07:39 | |
| *** pino|work has quit IRC | 07:40 | |
| *** shardy has quit IRC | 07:43 | |
| *** jpich has joined #tripleo | 07:45 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Clear up "already provided" message https://review.openstack.org/290968 | 07:45 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Convert element_dependencies to logging https://review.openstack.org/328072 | 07:45 |
| *** pino|work_ is now known as pino|work | 07:45 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Enable TLS in the internal network for keystone https://review.openstack.org/327029 | 07:46 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Enable TLS in the internal network for heat https://review.openstack.org/327069 | 07:46 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Enable TLS in the internal network for glance API and registry https://review.openstack.org/327473 | 07:46 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Enable TLS in the internal network for RabbitMQ https://review.openstack.org/327482 | 07:46 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Enable TLS in the internal network for cinder-api https://review.openstack.org/328859 | 07:46 |
| *** cmyster has quit IRC | 07:48 | |
| *** dtantsur|afk is now known as dtantsur | 07:50 | |
| *** dtrainor has quit IRC | 07:52 | |
| *** jaosorior is now known as jaosorior_brb | 07:53 | |
| *** jtomasek_ has joined #tripleo | 07:53 | |
| *** links has joined #tripleo | 07:55 | |
| *** ccamacho has joined #tripleo | 07:56 | |
| *** milan has joined #tripleo | 08:06 | |
| openstackgerrit | Carlos Camacho proposed openstack/puppet-tripleo: Composable roles within services - NTP https://review.openstack.org/310725 | 08:06 |
| *** Guest34058 has quit IRC | 08:07 | |
| *** dtrainor has joined #tripleo | 08:08 | |
| *** dbecker has quit IRC | 08:09 | |
| *** jtomasek_ has quit IRC | 08:10 | |
| *** shardy has joined #tripleo | 08:10 | |
| *** dbecker has joined #tripleo | 08:10 | |
| *** zoli_gone-proxy is now known as zoliXXL | 08:11 | |
| *** abehl has joined #tripleo | 08:12 | |
| *** liverpooler has joined #tripleo | 08:15 | |
| *** ohamada has joined #tripleo | 08:16 | |
| jaosorior_brb | upgrades gate seems to be broken in master :/ | 08:17 |
| *** dmk0202 has joined #tripleo | 08:22 | |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/puppet-tripleo: WIP: integration of the new puppet pacemaker. https://review.openstack.org/309069 | 08:22 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates: WIP: integration of the new puppet pacemaker. https://review.openstack.org/302409 | 08:24 |
| *** panda has quit IRC | 08:24 | |
| *** panda has joined #tripleo | 08:25 | |
| *** olap has quit IRC | 08:27 | |
| *** cllewellyn__ has quit IRC | 08:27 | |
| *** cllewellyn_ has quit IRC | 08:27 | |
| *** apetrich has quit IRC | 08:28 | |
| *** olap has joined #tripleo | 08:28 | |
| *** stendulker has joined #tripleo | 08:28 | |
| *** apetrich has joined #tripleo | 08:30 | |
| *** abehl has quit IRC | 08:32 | |
| *** abehl has joined #tripleo | 08:33 | |
| *** paramite has joined #tripleo | 08:34 | |
| *** abehl has quit IRC | 08:34 | |
| *** abehl has joined #tripleo | 08:34 | |
| openstackgerrit | Carlos Camacho proposed openstack/puppet-tripleo: Composable roles within services - NTP https://review.openstack.org/310725 | 08:35 |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-heat-templates: Composable roles within services - NTP https://review.openstack.org/310421 | 08:36 |
| dtantsur | morning folks! a second +2 is needed on the ironic-in-overcloud spec https://review.openstack.org/320995 please | 08:38 |
| *** zoliXXL is now known as zoli|brb | 08:39 | |
| *** jaosorior_brb has quit IRC | 08:40 | |
| *** jaosorior_brb has joined #tripleo | 08:41 | |
| *** cllewellyn__ has joined #tripleo | 08:41 | |
| *** cllewellyn_ has joined #tripleo | 08:41 | |
| *** jaosorior_brb is now known as jaosorior | 08:41 | |
| jaosorior | ccamacho hey dude, seems to me like the upgrades gate is broken, have you noticed? | 08:42 |
| ccamacho | upgrades in Master? | 08:42 |
| jaosorior | yes | 08:42 |
| ccamacho | jaosorior ^ | 08:42 |
| jaosorior | I recheck a bunch of commits in the morning | 08:43 |
| jaosorior | and not a single one of them has passed upgrades | 08:43 |
| *** cllewellyn__ has quit IRC | 08:43 | |
| ccamacho | mmmm yesterday was fine, but were landed a lot of patches.. | 08:43 |
| ccamacho | letme check | 08:43 |
| jaosorior | resources.ControllerNodesPostDeployment: resources.ControllerPostPuppet: resources.ControllerPostPuppetRestartDeployment: Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 1 | 08:43 |
| *** cllewellyn__ has joined #tripleo | 08:44 | |
| jaosorior | shardy: Any tips on how to debug something like that? ^^ | 08:44 |
| ccamacho | dead by timeout :$ | 08:44 |
| dtantsur | jaosorior, once you have a second of time, could you please look again at the documentation patch https://review.openstack.org/#/c/322776/ ? | 08:44 |
| *** cllewellyn__ has quit IRC | 08:44 | |
| *** cllewellyn__ has joined #tripleo | 08:44 | |
| shardy | jaosorior: get the ID of the failing SoftwareDeployment, then run heat deployment-show <id> | 08:45 |
| shardy | the stderr should give some clues | 08:45 |
| *** derekh has joined #tripleo | 08:45 | |
| *** cllewellyn__ has quit IRC | 08:45 | |
| ccamacho | jaosorior, can you post the patch link? or is from your local env? | 08:46 |
| jaosorior | check any recent patch's upgrade job | 08:46 |
| *** cllewellyn__ has joined #tripleo | 08:46 | |
| jaosorior | ccamacho: for instance http://logs.openstack.org/04/329504/1/check-tripleo/gate-tripleo-ci-centos-7-upgrades/2a9a0fc/ | 08:47 |
| openstackgerrit | Merged openstack/tripleo-docs: Rework nodes registration and configuration https://review.openstack.org/322776 | 08:47 |
| dtantsur | shardy, hi! a kind request to review the ironic-in-overcloud spec https://review.openstack.org/255792 please. we're getting some good progress with the patches already, would be nice to have the spec landed | 08:47 |
| *** cllewellyn__ has quit IRC | 08:47 | |
| dtantsur | meh, wrong link | 08:47 |
| dtantsur | shardy, the correct link: https://review.openstack.org/320995 | 08:47 |
| *** cllewellyn__ has joined #tripleo | 08:47 | |
| jaosorior | ccamacho: This has the same issue http://logs.openstack.org/18/329718/1/check-tripleo/gate-tripleo-ci-centos-7-upgrades/cb4b9c1/ | 08:48 |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-heat-templates: Composable roles within services - NTP https://review.openstack.org/310421 | 08:48 |
| *** cllewellyn__ has quit IRC | 08:48 | |
| jaosorior | damn, so if ugprades doesn't fail with the overcloud deploy timeout, it seems to fail with the Controller PostPuppetRestartDeployment error | 08:49 |
| ccamacho | checking logs | 08:49 |
| *** cllewellyn__ has joined #tripleo | 08:49 | |
| openstackgerrit | Carlos Camacho proposed openstack/puppet-tripleo: Composable roles within services - NTP https://review.openstack.org/310725 | 08:50 |
| *** apetrich has quit IRC | 08:50 | |
| *** zoli|brb is now known as zoli | 08:50 | |
| *** cllewellyn__ has quit IRC | 08:51 | |
| shardy | dtantsur: sure, will do | 08:51 |
| dtantsur | thnx! | 08:51 |
| *** mbound has joined #tripleo | 08:52 | |
| *** cllewellyn__ has joined #tripleo | 08:52 | |
| *** apetrich has joined #tripleo | 08:53 | |
| *** mcornea has joined #tripleo | 08:53 | |
| *** cllewellyn__ has quit IRC | 08:54 | |
| *** cllewellyn__ has joined #tripleo | 08:54 | |
| *** cllewellyn__ has quit IRC | 08:56 | |
| *** cmyster has joined #tripleo | 08:56 | |
| *** ramishra has quit IRC | 08:56 | |
| *** cllewellyn__ has joined #tripleo | 08:56 | |
| *** jaosorior has quit IRC | 08:57 | |
| *** cllewellyn__ has quit IRC | 08:58 | |
| *** cllewellyn__ has joined #tripleo | 08:59 | |
| *** electrofelix has joined #tripleo | 08:59 | |
| ccamacho | I will deploy any job to get the error, from the CI not getting any useful. | 09:00 |
| *** ramishra has joined #tripleo | 09:02 | |
| *** fzdarsky has quit IRC | 09:02 | |
| *** fzdarsky has joined #tripleo | 09:04 | |
| *** cllewellyn__ has quit IRC | 09:06 | |
| *** cllewellyn_ has quit IRC | 09:06 | |
| *** cllewellyn__ has joined #tripleo | 09:06 | |
| *** cllewellyn_ has joined #tripleo | 09:06 | |
| *** jtomasek_ has joined #tripleo | 09:07 | |
| *** cllewellyn__ has quit IRC | 09:08 | |
| *** cllewellyn__ has joined #tripleo | 09:08 | |
| *** cllewellyn_ has quit IRC | 09:08 | |
| *** cllewellyn_ has joined #tripleo | 09:08 | |
| *** cllewellyn_ has quit IRC | 09:09 | |
| *** cllewellyn_ has joined #tripleo | 09:09 | |
| *** mgould|afk is now known as mgould | 09:10 | |
| *** cllewellyn_ has quit IRC | 09:11 | |
| *** cllewellyn_ has joined #tripleo | 09:12 | |
| *** jtomasek_ has quit IRC | 09:13 | |
| *** sambetts|afk is now known as sambetts | 09:16 | |
| jistr | heya folks, do we still manage endpoints via os-cloud-config or did the endpoint management via Puppet make it in? | 09:17 |
| * jistr can't find it in puppet but keeps looking | 09:18 | |
| *** jaosorior has joined #tripleo | 09:21 | |
| chem`` | ccamacho: I think I figure out the problem | 09:21 |
| ccamacho | chem``, with upgrades? | 09:22 |
| chem`` | ccamacho: yeap | 09:22 |
| ccamacho | tell me :) | 09:22 |
| ccamacho | im deploying all jobs to check them until now they are running.. | 09:22 |
| chem`` | ccamacho: looking at the log, it seems that openstack-nova-scheduler, openstack-cinder-volume, nova-api and nova-conductor fail to restart | 09:23 |
| chem`` | ccamacho: after the upgrade script say that the cluster is instable for too long and abort | 09:24 |
| ccamacho | woow.. | 09:24 |
| chem`` | ccamacho: I think this is due to the removal of the openstack-core constraint on the conductor resource | 09:24 |
| ccamacho | let me see if I can reproduce it locally | 09:25 |
| ccamacho | THe good thing is that we have some clues | 09:25 |
| *** apetrich has quit IRC | 09:25 | |
| chem`` | ccamacho: you can see that hapening at the end of the 3.2M log/message file in the controler-0 logs | 09:26 |
| *** mikelk has joined #tripleo | 09:26 | |
| chem`` | ccamacho: parsing 3.2M file in firefox is a joy :) I need to upgrade to a 64GB laptop. | 09:26 |
| *** akrivoka has joined #tripleo | 09:26 | |
| bandini | marios: can I tickle your brain for an upgrade issue I am seeing? | 09:26 |
| chem`` | ccamacho: ... or download the file ... | 09:26 |
| *** apetrich has joined #tripleo | 09:28 | |
| chem`` | ccamacho: this is the file http://logs.openstack.org/18/329718/1/check-tripleo/gate-tripleo-ci-centos-7-upgrades/cb4b9c1/logs/overcloud-controller-0/var/log/messages | 09:28 |
| ccamacho | chem``: I usually do http://paste.openstack.org/show/516179/ as is really hard to see the logs in the browsre | 09:28 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Enable TLS in the internal network for keystone https://review.openstack.org/327029 | 09:29 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Enable TLS in the internal network for heat https://review.openstack.org/327069 | 09:29 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Enable TLS in the internal network for glance API and registry https://review.openstack.org/327473 | 09:29 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Enable TLS in the internal network for RabbitMQ https://review.openstack.org/327482 | 09:29 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Enable TLS in the internal network for cinder-api https://review.openstack.org/328859 | 09:29 |
| ccamacho | hey bandini, marios morning!!!: chem`` is giving us some clues about the upgrades error | 09:29 |
| matbu | chem``: ccamacho you are talking about the compute upgrade ? | 09:32 |
| ccamacho | matbu, :) nope upgrade job in CIç | 09:33 |
| matbu | ccamacho: CI job for minor upgrade? | 09:33 |
| ccamacho | marios, matbu, bandini, sorry guys mixing upgrades, is the minor upgrade job in CI | 09:34 |
| *** apetrich has quit IRC | 09:34 | |
| matbu | ccamacho: I was talking about major liberty to mitaka upgrade, the compute failed because it failed with nova-scheduler | 09:34 |
| matbu | ack :) | 09:34 |
| *** ohamada_ has joined #tripleo | 09:34 | |
| *** ohamada has quit IRC | 09:34 | |
| bandini | matbu: so my status today is that aodh and keystone work correctly. I am having a very odd issue with the major-pacemaker-upgrade step. the first step completes but the second step is never started | 09:35 |
| *** cllewellyn__ has quit IRC | 09:35 | |
| chem`` | ccamacho: thanks for the snippet by the way | 09:35 |
| bandini | matbu: so I have not yet reached the compute issue ;) | 09:35 |
| bandini | ccamacho: in what situations is the minor upgrade ci job broken? it worked for me this morning | 09:36 |
| *** cllewellyn__ has joined #tripleo | 09:36 | |
| ccamacho | chem`` np | 09:36 |
| ccamacho | bandini, not sure, there are lot of jobs in CI getting errors in the upgrades job | 09:37 |
| matbu | bandini: yep , the same for me | 09:37 |
| matbu | bandini: the aodh and keystone works | 09:37 |
| *** cllewellyn__ has quit IRC | 09:37 | |
| ccamacho | so right now im locally deploying all jobs from master to check it locally | 09:37 |
| matbu | bandini: but the step2 never start ? (hang forever) | 09:37 |
| bandini | matbu: EXACTLY | 09:37 |
| matbu | bandini: cool :) i'm not crazy | 09:38 |
| *** apetrich has joined #tripleo | 09:38 | |
| bandini | matbu: it is as if step1 completes, cluster is down. but step2 never starts | 09:38 |
| matbu | bandini: nothing happen, | 09:38 |
| hewbrocca | bandini: the jury is still out on whether you're crazy | 09:38 |
| bandini | matbu: right | 09:38 |
| bandini | hewbrocca: oh no, it has very well decided on that ;) | 09:38 |
| matbu | bandini: yep, i checked eveyr thing, i don't see what heat is waiting for | 09:38 |
| bandini | matbu: ok, shall we collect some infos/data on the etherpad ? | 09:39 |
| *** cllewellyn__ has joined #tripleo | 09:39 | |
| matbu | bandini: and when i executed the step2 script, it works correctly | 09:39 |
| bandini | ah, good to know | 09:39 |
| hewbrocca | os-collect-config running on the nodes? | 09:39 |
| *** cllewellyn__ has quit IRC | 09:39 | |
| matbu | bandini: yep if you want | 09:39 |
| ccamacho | matbu bandini, about that never end timeout, if you are testing locally can you connect to the vms using virt-manager to see the console? In liberty sometimes the deployment hangs and the reason is that the mvs are not booting up. just saying... | 09:39 |
| *** cllewellyn__ has joined #tripleo | 09:39 | |
| matbu | ccamacho: nop it during an upgrade steps | 09:40 |
| bandini | ccamacho: the vms are up and running, it is just heat from the undercloud that seems to be stuck | 09:40 |
| bandini | hewbrocca: os-collect-config is running, it seems heat is not telling it to do the second step | 09:40 |
| matbu | bandini: i was thinking of a network issue, some of a VIP that heat wants to reach, but which has been disable by the cluster down | 09:40 |
| matbu | hewbrocca: bandini yep if you trigger os-collect-config manually eveyr thing is fine, the previous step is ended correctly | 09:41 |
| bandini | matbu: that might be a good lead actually | 09:42 |
| jistr | bandini: re AODH and keystone working correctly -- we don't have the AODH endpoints tough yet, right? Just sent a suggestion to pradk how to solve that a while ago. | 09:43 |
| hewbrocca | Man, we really, really need to replace this whole os-collect-config nonsense with a nice push/pull thing like Zaqar | 09:44 |
| bandini | jistr: that is correct. while not ideal, I feel it is a bit of a minor issue (i.e. well the aodh endpoints won't be around until convergence step runs). but yeah worth fixing | 09:44 |
| bandini | jistr: I have got bigger fish to fry at the moment :D | 09:44 |
| jistr | bandini: yea makes sense :D | 09:44 |
| *** athomas has quit IRC | 09:45 | |
| matbu | bandini: jistr btw i wonder if those two additionnals steps (aodh / keystone) could be add to the major upgrade controller step | 09:45 |
| jistr | bandini: though the endopints wouldn't be created on convergence either | 09:45 |
| matbu | to avoid a 5 steps upgrade overcloud | 09:45 |
| jistr | currently we don't use Puppet to create endpoints AFAIK, and os-cloud-config only runs on stack create | 09:45 |
| bandini | jistr: ah, I did not know that. That is definitely a bigger problem | 09:45 |
| *** cllewellyn__ has quit IRC | 09:46 | |
| *** cllewellyn_ has quit IRC | 09:46 | |
| chem`` | ccamacho: so the final error is 'ERROR: cluster remained unstable for more than 1800 seconds, exiting.' from os-collect-config and this from the pacemaker engine http://paste.fedoraproject.org/379421/46598391/ | 09:46 |
| chem`` | ccamacho: I'm looking at other logs to see how they look | 09:46 |
| *** cllewellyn__ has joined #tripleo | 09:46 | |
| *** cllewellyn_ has joined #tripleo | 09:46 | |
| hewbrocca | Arrgh I thought we had the puppet endpoint creation, at least on trunk | 09:47 |
| jistr | matbu: yea it could reduce the number of steps, but on the other hand we wanted them separate on purpose i think, to have them separately testable too, and have a smaller failure domain (to avoid "i've attempted to execute this blob of 3 invasive operations on my cloud, and the blob failed, what state is my cloud in now?") | 09:47 |
| * hewbrocca so tired of os-cloud-config | 09:47 | |
| *** cllewellyn__ has quit IRC | 09:47 | |
| *** cllewellyn__ has joined #tripleo | 09:47 | |
| *** florianf has quit IRC | 09:48 | |
| jistr | it is proposed but not merged yet, probably needs an amendment wrt composability | 09:48 |
| matbu | jistr: hm yep true | 09:48 |
| shardy | Yeah, we should figure out how to land that though, it'll make the composable endpoints much easier I think | 09:48 |
| chem`` | ccamacho: well ignore the paste, it's the same on a working one | 09:49 |
| marios | bandini: hey man, sorry was getting some foods.. gimme couple mins and reading back | 09:49 |
| *** trumpetnl has joined #tripleo | 09:49 | |
| *** athomas has joined #tripleo | 09:50 | |
| chem`` | ccamacho: oups sorry wrong file, the paste is still legite ... | 09:50 |
| ccamacho | chem`` ack, for me both ha nonha ran without errors, now runing a minor upgrade | 09:50 |
| bandini | marios: I am trying to describe the issue matbu and I are seeing here: https://etherpad.openstack.org/p/tripleo-liberty-mitaka-upgrades | 09:51 |
| *** cinerama has quit IRC | 09:51 | |
| marios | bandini: ack thanks (there is a lot of text there)... are you sure there ar eno errors after controller_pacemaker_1.sh | 09:54 |
| marios | bandini: things like the cluster timing out for example after things are stopped? or even not setting in time after being started? | 09:54 |
| bandini | marios: I know sorry lots of text :) I believe there are no errors. the cluster is fully stopped and the crudini operations took place | 09:54 |
| matbu | marios: yep the step1 is really done | 09:55 |
| bandini | and the packages are updated | 09:55 |
| *** cinerama has joined #tripleo | 09:55 | |
| matbu | marios: you can stop/start the cluster manually, it'sworks fine | 09:55 |
| bandini | if you look at line 69-72 you see that Step2 is never triggered on the controller | 09:55 |
| bandini | yet heat on the undercloud shows it as CREATE_IN_PROGRESS | 09:55 |
| marios | matbu: bandini so the cluster stays stopped? | 09:56 |
| hewbrocca | stevebaker: ^^^ this sounds weirdly familiar to me | 09:56 |
| bandini | marios: correct. cluster is down. step1 completed successfully. step2 is never started so we all hang there | 09:57 |
| matbu | marios: yes | 09:57 |
| bandini | I have reproduced this on three different systems | 09:58 |
| bandini | so it is not a race or something | 09:58 |
| matbu | marios: heat show step2 in progress, but the script _2.sh is never start | 09:58 |
| bandini | exactly, script _2.sh does not even exist in /var/lib/heat-config/heat-config-scripts/ | 09:58 |
| bandini | it really looks like heat is in some la-la-la land here | 09:59 |
| matbu | bandini: lol yep | 09:59 |
| bandini | matbu: I started seeing this today but I was more focused on the aodh/keystone steps. Have you seen this behaviour from day 1? | 10:00 |
| marios | bandini: is swift-* started on controllers? (cluster is stopped right) | 10:00 |
| shardy | bandini: https://paste.fedoraproject.org/379427/98481714/ | 10:00 |
| shardy | try that - it shows how to grab the server metadata | 10:00 |
| shardy | then you can grep that and check heat actually exposed the step2 config via the deployment | 10:00 |
| matbu | bandini: yep, i have seen it for a long time | 10:00 |
| chem`` | ccamacho: on your plateform is the openstack-nova-consoleauth service enabled ? | 10:01 |
| shardy | bandini: that will bisect the problem to heat vs something in the node (or network) | 10:01 |
| marios | bandini: matbu fwiw my additions at pacemaker_common_functions.sh adds a lot of debugging, wondering if it would help here. | 10:01 |
| marios | bandini: still think there may be an error, timeout for something to stop possibly. but is strange if all the crudini are also set (so 1.sh really did complete) | 10:01 |
| bandini | shardy: thanks will try! | 10:02 |
| matbu | marios: yep, and i think the blockstorage is done also | 10:02 |
| *** florianf has joined #tripleo | 10:02 | |
| matbu | shardy: will try to, /me deploying a new env | 10:02 |
| marios | bandini: matbu (I mean at https://review.openstack.org/#/c/321027/13/extraconfig/tasks/pacemaker_common_functions.sh ) | 10:02 |
| bandini | marios: yes crudini did run, because I had to add another one crudini line due to a change in keystone paste.ini files | 10:03 |
| ccamacho | chem`` not enabled by default | 10:03 |
| chem`` | ccamacho: so it's not managed by systemd.. weird | 10:04 |
| bandini | marios: swift is all down btw | 10:04 |
| marios | bandini: ok thx was wondering if it was just missing the bootstrap for some reason @ https://github.com/openstack/tripleo-heat-templates/blob/bcd726f1242d78169e6a5687e998473c1043c622/extraconfig/tasks/major_upgrade_controller_pacemaker_2.sh#L9 and then just started swift | 10:04 |
| bandini | marios: nope that is fine. step_2 script does not exist on the controllers yet so it cannot have run | 10:05 |
| bandini | matbu: you said that running os-collect-config by hand triggers things and it all works, correct? | 10:05 |
| matbu | bandini: nop, i try os-collect-config in debug mode, to see what goes wrong | 10:06 |
| matbu | bandini: but every thing was fine | 10:06 |
| bandini | matbu: so if you run it by hand does it run Step2 of the upgrade or not? | 10:06 |
| matbu | bandini: but i execute the _2.sh script manually on the controller | 10:06 |
| matbu | bandini: nop it didn't run the step2 | 10:07 |
| bandini | got it | 10:07 |
| matbu | afair | 10:07 |
| bandini | I can try, does it need any special parameters? | 10:07 |
| matbu | but i mean, it's not an issue with the step2 script itself, cause, you can run it manually and the cluster will start, the vip too and so on... | 10:08 |
| bandini | matbu: fully agreed | 10:08 |
| matbu | bandini: for running the script ? or os-collect-config ? | 10:08 |
| bandini | we need to understand why step2 is not triggered on the controllers | 10:08 |
| bandini | matbu: yes if I were to rerun os-collect-config manually, how would I do that? just run the binary or are there special parameters | 10:09 |
| matbu | just do sudo service os-collect-config stop | 10:09 |
| matbu | sudo os-collect-config --force --one-time --debug | 10:09 |
| matbu | bandini: ^ | 10:09 |
| bandini | matbu: ack, trying now | 10:09 |
| bandini | thenI will follow shardy's tips | 10:09 |
| matbu | yep me too, upgrading UC atm | 10:10 |
| matbu | bandini: but i happy you hit that too, cause i was wondering if it was only an issue with my env | 10:11 |
| matbu | i'm* | 10:11 |
| bandini | matbu: ack, I confirm that it talks to heat but no Step2 in sight | 10:12 |
| bandini | matbu: indeed it's good to have common issues :) | 10:12 |
| matbu | hehe yep | 10:12 |
| matbu | marios: do you remember the review that Dan paste about mistral during the composable upgrade meeting ? | 10:14 |
| bandini | shardy: the upgrade timed out, so I guess that is why I get empty strings from your commands? https://paste.fedoraproject.org/379433/14659857/ | 10:16 |
| bandini | I assume I need to run those commands while heat is still trying | 10:16 |
| marios | matbu: which one, remote execution one? | 10:16 |
| *** karthiks has quit IRC | 10:17 | |
| matbu | marios: i don't remeber exactly, he pasted it in the bj chat as an example on how to use mistral | 10:17 |
| yolanda | hi shardy , i'm having some issues with https://review.openstack.org/#/c/299643 change, the upload-puppet-modules script. I'm hitting that problem with slash removal | 10:17 |
| marios | matbu: bandini sure, sec | 10:18 |
| marios | err sry bandini | 10:18 |
| marios | https://etherpad.openstack.org/p/tripleo-remote-execution matbu | 10:18 |
| yolanda | also when i tried to upload just tripleo package, it may be some problem with my paths, becuse it failed when not having tripleo puppet module updated. I had to use the approach to move to /etc/puppet/modules, and upload the whole directory | 10:18 |
| matbu | marios: thx man | 10:18 |
| marios | matbu: maybe it was this one (https://review.openstack.org/#/c/313957/ ) but see the etherpad | 10:18 |
| marios | ack | 10:18 |
| shardy | bandini: No, you need to use resource-metadata on the OS::Nova::Server resource, not OS::Heat::SoftwareDeployment | 10:19 |
| shardy | it should work even after a timeout | 10:19 |
| * bandini whistles innocently | 10:19 | |
| *** apetrich has quit IRC | 10:19 | |
| shardy | yolanda: Hi, perhaps we need some more fixes re the slash removal, but it works fine for me just specifying the local directory | 10:20 |
| shardy | e.g upload-puppet-modules -d puppet_modules | 10:20 |
| yolanda | mm, i was using absolute directory | 10:21 |
| shardy | where ./puppet_modules exists and contains e.g a "tripleo" directory which is a copy of the puppet-tripleo module | 10:21 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Clear up "already provided" message https://review.openstack.org/290968 | 10:21 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Convert element_dependencies to logging https://review.openstack.org/328072 | 10:21 |
| shardy | yolanda: yeah, perhaps that is still broken, a relative path without any slashes should work | 10:21 |
| *** apetrich has joined #tripleo | 10:22 | |
| shardy | if you wanted to fix the script for absolute paths I'm pretty sure dprince would be fine with you pushing a fix to the patch | 10:22 |
| *** karthiks has joined #tripleo | 10:22 | |
| yolanda | shardy i'll retry with relative path to confirm | 10:22 |
| yolanda | also if you can add some clarification for the change? it has a -1 due to that issue | 10:23 |
| bandini | shardy: https://paste.fedoraproject.org/379441/98624814/ | 10:25 |
| bandini | marios, matbu: ^ | 10:25 |
| bandini | not entirely sure how to interpret that yet | 10:25 |
| yolanda | hi, when deploying tripleo composable roles, i got that error... http://paste.openstack.org/show/516195/ | 10:26 |
| yolanda | is that a known problem? | 10:26 |
| shardy | yolanda: https://paste.fedoraproject.org/379442/98634114/ shows it working fine with relative paths | 10:26 |
| yolanda | shardy, thx. Knowing that i need to pass a relative path is enough to me. Going to do a try with that to confirm from my side | 10:28 |
| yolanda | shardy, also, are you familiar with that error i pasted? that's only failure i see when testing composable roles | 10:28 |
| matbu | bandini: marios is not that ControllerAllNodesValidationDeployment which trying to check the status of the ips | 10:29 |
| matbu | but the VIP are down | 10:29 |
| shardy | yolanda: the first thing to check is pull the latest puppet-ceilometer and add it to puppet_modules (named "ceilometer") | 10:29 |
| marios | matbu: the vip are brought down at https://github.com/openstack/tripleo-heat-templates/blob/bcd726f1242d78169e6a5687e998473c1043c622/extraconfig/tasks/major_upgrade_controller_pacemaker_1.sh#L29 | 10:29 |
| matbu | marios: yep | 10:29 |
| shardy | I have an updated ceilometer module there, IIRC it may have been to fix that issue | 10:29 |
| bandini | ok but does ControllerAllNodesValidationDeployment check for VIPs? that would make little sense to me | 10:30 |
| matbu | marios: maybe the Allnodevalidation steps, is trying to check if all the ip is reachable | 10:30 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Add websocket utils module https://review.openstack.org/322611 | 10:32 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Use Mistral for baremetal registration https://review.openstack.org/322612 | 10:32 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: WIP Use Mistral for baremetal introspection https://review.openstack.org/327780 | 10:32 |
| yolanda | shardy, ok, going to try that, also using that relative path approach | 10:32 |
| yolanda | i guess it should be the common workflow | 10:32 |
| * matbu brb lunch time, it would be easier after lunch :D | 10:33 | |
| bandini | matbu: the way I read overcloud.yaml it checks only for the non VIPs ip addresses | 10:33 |
| bandini | matbu: enjoy ;) | 10:33 |
| *** jefrite has joined #tripleo | 10:34 | |
| dtantsur | wow | 10:41 |
| dtantsur | I mean WOW!! | 10:41 |
| dtantsur | I got a successful pass of my ironic-in-overcloud patch \o/ | 10:41 |
| dtantsur | ifarkas, ^^^ | 10:41 |
| dtantsur | look, ironic running on the controller: http://logs.openstack.org/28/316128/21/check-tripleo/gate-tripleo-ci-centos-7-ha/941866f/logs/overcloud-controller-0/var/log/ironic/ | 10:42 |
| * dtantsur celebrates | 10:42 | |
| ifarkas | \o/ | 10:43 |
| ifarkas | congrats dtantsur! | 10:43 |
| dtantsur | folks, please review https://review.openstack.org/#/c/319297/ it seems to be working | 10:44 |
| openstackgerrit | Dmitry Tantsur proposed openstack/tripleo-heat-templates: Basic support for deploying Ironic in overcloud https://review.openstack.org/316128 | 10:45 |
| openstackgerrit | Dmitry Tantsur proposed openstack/tripleo-heat-templates: DO NOT MERGE: testing ironic https://review.openstack.org/329872 | 10:45 |
| dtantsur | ifarkas, cleaned up the patches ^^^ please take a look as well | 10:45 |
| ifarkas | will do | 10:46 |
| *** apetrich has quit IRC | 10:50 | |
| *** panda has quit IRC | 10:50 | |
| *** apetrich has joined #tripleo | 10:50 | |
| jaosorior | dtantsur: got the commit for t-h-t? | 10:50 |
| *** zoli is now known as zoli|lunch | 10:50 | |
| *** weshay has joined #tripleo | 10:51 | |
| jaosorior | dtantsur: ironic-api doesn't need the database? | 10:51 |
| *** olap has quit IRC | 10:51 | |
| dtantsur | jaosorior, tht is https://review.openstack.org/316128 ironic-api accesses the database as of now | 10:51 |
| *** olap has joined #tripleo | 10:52 | |
| * dtantsur checks | 10:53 | |
| jaosorior | dtantsur: Commented here | 10:53 |
| jaosorior | https://review.openstack.org/#/c/319297/5 | 10:53 |
| dtantsur | jaosorior, hmm, so maybe it makes sense to move the database creation to the base ironic.pp, right? | 10:54 |
| *** panda has joined #tripleo | 10:54 | |
| jaosorior | dtantsur: that may be the case. Would need to check the other services and see if they do something like that | 10:55 |
| jaosorior | but it does seem to me that it's wrong that there is no trace of database creation on the api profile | 10:56 |
| jaosorior | dtantsur: On the other hand, the mysql related values are on the ironic-base template in t-h-t. So I guess it does make sense to move that | 10:57 |
| dtantsur | jaosorior, will do. could you please review the remaining parts, so that I can update them at once? | 10:58 |
| jaosorior | dtantsur: I gave a read to the t-h-t and the puppet parts. It looks pretty good from my side. | 10:59 |
| dtantsur | jaosorior, environments/ironic-generic-config.yaml is for people to include to enable ironic (it's optional). will comment. thanks! | 10:59 |
| jaosorior | commented on both | 10:59 |
| jaosorior | is it optional? | 10:59 |
| dtantsur | jaosorior, yes | 11:00 |
| jaosorior | dtantsur: Should those resources be set as OS::Heat::None here then? https://review.openstack.org/#/c/316128/22/overcloud-resource-registry-puppet.yaml | 11:00 |
| dtantsur | jaosorior, dunno, maybe? I don't understand this bit, sorry :) | 11:01 |
| *** pradk has quit IRC | 11:01 | |
| dtantsur | jaosorior, isn't e.g. sahara optional as well? | 11:01 |
| openstackgerrit | Marios Andreou proposed openstack/instack-undercloud: Overcloud is not able to deploy with the default 4GB of RAM using instack-undercloud https://review.openstack.org/329874 | 11:01 |
| openstackgerrit | Dmitry Tantsur proposed openstack/puppet-tripleo: Add base ironic profiles https://review.openstack.org/319297 | 11:03 |
| jaosorior | dtantsur: It should be. However, it needs to be added to the ControllerServices list parameter to be taken into use | 11:03 |
| jaosorior | so not sure how that file that I commented on will actually be used | 11:04 |
| ccamacho | chem`` reproduced locally http://paste.openstack.org/show/516225/ "ERROR: cluster remained unstable for more than 1800 seconds, exiting" the minor upgrades job is failing also locally | 11:04 |
| dtantsur | jaosorior, probably -e /path/to/environment? | 11:04 |
| chem`` | ccamacho: great news! So do you see why the nova-consoleauth service is not restarting ? | 11:04 |
| dtantsur | jaosorior, in the same fashion as network isolation | 11:05 |
| openstackgerrit | Dmitry Tantsur proposed openstack/tripleo-heat-templates: Basic support for deploying Ironic in overcloud https://review.openstack.org/316128 | 11:05 |
| openstackgerrit | Dmitry Tantsur proposed openstack/tripleo-heat-templates: DO NOT MERGE: testing ironic https://review.openstack.org/329872 | 11:05 |
| dtantsur | updated ^^^ | 11:06 |
| *** cllewellyn_ has quit IRC | 11:06 | |
| *** cllewellyn__ has quit IRC | 11:06 | |
| jaosorior | dtantsur: yeah, I understand how it can be added; but not the effect that it will actually have. For example -> OS::Tripleo::Services::IronicApi: is already being set as puppet/services/ironic-api.yaml in the base resource registry | 11:06 |
| dtantsur | jaosorior, I don't see a contradiction, sorry... | 11:07 |
| jaosorior | so it seems to me taht doing -e environments/ironic-config.yaml is a no-op | 11:07 |
| openstackgerrit | John Trowbridge proposed openstack-infra/tripleo-ci: [NO MERGY] Test a fake periodic job https://review.openstack.org/229789 | 11:08 |
| dtantsur | jaosorior, well, I'll check it again | 11:08 |
| shardy | jaosorior: are you saying the problem is there's no way to append to the ControllerServices parameter in an environment file? | 11:08 |
| jaosorior | dtantsur: So what I mean that the values that are being set here https://review.openstack.org/#/c/316128/23/environments/ironic-config.yaml is the value that this already has https://review.openstack.org/#/c/316128/23/overcloud-resource-registry-puppet.yaml | 11:08 |
| jaosorior | unless I'm misunderstanding something | 11:09 |
| shardy | jaosorior: you're right, it won't do anything | 11:09 |
| dtantsur | jaosorior, yeah, I've checked the other files, I think the environment can be dropped.. I'm not sure how a user requests Ironic (Sahara etc) to be deployed though | 11:09 |
| shardy | what is needed is a way to add OS::Tripleo::Services::IronicApi to ControllerServices, but for now we'll have to document copying the entire default list and adding it | 11:10 |
| jaosorior | dtantsur: well, I guess they manually set the value for the controller services list | 11:10 |
| jaosorior | shardy: yeah, there is no trivial way of just adding services | 11:10 |
| jaosorior | shardy: Do you know what the status of OS::Heat::value (or however it's called) is? | 11:10 |
| dtantsur | that's fine with me, thanks :) | 11:11 |
| shardy | I've been looking into ways we could add a "merge" feature to heat environments so that you could e.g to -e ironic-config.yaml and have it add to ControllerServices just by defining ControllerServices with values to be appended | 11:11 |
| jaosorior | that or yaql could help maybe? | 11:11 |
| openstackgerrit | Dmitry Tantsur proposed openstack/tripleo-heat-templates: Basic support for deploying Ironic in overcloud https://review.openstack.org/316128 | 11:11 |
| openstackgerrit | Dmitry Tantsur proposed openstack/tripleo-heat-templates: DO NOT MERGE: testing ironic https://review.openstack.org/329872 | 11:11 |
| dtantsur | dropped the file for now ^^^ | 11:11 |
| shardy | jaosorior: yaql could help if we wanted to have say an ControllerExtraServices param and join it (actually, list_join could do that..) | 11:11 |
| ccamacho | chem``: do you know if there is a bug for this? | 11:12 |
| shardy | but then you still can't declare values for that more than once | 11:12 |
| shardy | we need to either support client-side appending of values in tripleo-common, or add a feature to heat | 11:12 |
| chem`` | ccamacho: not that I'm aware of | 11:12 |
| shardy | I prefer the latter, going to post a spec | 11:12 |
| shardy | for now we'll have to just document specifying the entire list | 11:12 |
| shardy | which to be fair isn't that hard | 11:12 |
| jaosorior | yeah, seems like the only solution for now | 11:13 |
| jaosorior | it isn't that hard. But not very user-friendly either | 11:13 |
| chem`` | ccamacho: I can start one on launchpad, so that we can put our finding there | 11:13 |
| ccamacho | chem`` neat! ill post comments | 11:13 |
| shardy | jaosorior: well, it'd be pretty trivial to have either the UI or a wizard in the CLI prompt and askk the user which services they want | 11:13 |
| shardy | then the interface to t-h-t remains clean, we just require the list output from those answers | 11:14 |
| *** dprince has joined #tripleo | 11:14 | |
| shardy | anyway, something we can think about for sure | 11:14 |
| dtantsur | thanks for clarification shardy. please see the updated patches | 11:14 |
| shardy | dtantsur: np, will do | 11:15 |
| jaosorior | shardy: true, well, the documentation for how to add services to the list could go on ccamacho's tutorial. | 11:15 |
| jaosorior | ccamacho: How's your tutorial patch going by the way? | 11:16 |
| *** ccamacho is now known as ccamacho|lunch | 11:16 | |
| ccamacho|lunch | jaosorior, I think is going well just add comments and Ill put more information there :) | 11:16 |
| *** stendulker has quit IRC | 11:16 | |
| jaosorior | ccamacho|lunch can you roll that link? | 11:16 |
| ccamacho|lunch | sure | 11:17 |
| ccamacho|lunch | https://review.openstack.org/#/c/311512/24 | 11:17 |
| dprince | derekh, bnemec: so switch the IP to .224? | 11:17 |
| derekh | dbecker: yup | 11:17 |
| dbecker | derekh, ack | 11:17 |
| derekh | dprince: yup, dbecker sorry wrong person | 11:18 |
| dbecker | derekh, :-) | 11:18 |
| *** thrash|g0ne is now known as thrash | 11:19 | |
| dprince | derekh: the IP is updated. Will have to wait for the TTL to expire | 11:21 |
| derekh | dprince: cool beans, thanks | 11:23 |
| *** hewbrocca is now known as hewbrocca-afk | 11:25 | |
| *** hewbrocca-afk is now known as hewbrocca | 11:25 | |
| chem`` | ccamacho|lunch: https://bugs.launchpad.net/tripleo/+bug/1592776 | 11:25 |
| openstack | Launchpad bug 1592776 in tripleo "Ha upgrade jobs failing with "cluster remained unstable for more than 1800 seconds"" [Undecided,New] | 11:25 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/puppet-tripleo: WIP: integration of the new puppet pacemaker. https://review.openstack.org/309069 | 11:27 |
| *** adarazs is now known as adarazs_lunch | 11:33 | |
| *** pkovar has joined #tripleo | 11:33 | |
| *** cllewellyn__ has joined #tripleo | 11:35 | |
| *** cllewellyn_ has joined #tripleo | 11:36 | |
| *** cllewellyn_ has quit IRC | 11:37 | |
| *** bvandenh has quit IRC | 11:37 | |
| *** cllewellyn_ has joined #tripleo | 11:37 | |
| *** rasca has quit IRC | 11:38 | |
| *** bvandenh has joined #tripleo | 11:47 | |
| *** paramite has quit IRC | 11:50 | |
| *** rhallisey has joined #tripleo | 11:57 | |
| *** bfournie has quit IRC | 11:58 | |
| *** fzdarsky has quit IRC | 11:58 | |
| *** MaxPC has joined #tripleo | 11:58 | |
| *** jcoufal has joined #tripleo | 11:59 | |
| *** morazi has joined #tripleo | 12:00 | |
| *** jpena is now known as jpena|lunch | 12:01 | |
| *** jayg|g0n3 is now known as jayg | 12:02 | |
| *** rasca has joined #tripleo | 12:04 | |
| mgould | hi everyone | 12:08 |
| mgould | could someone please review https://review.openstack.org/#/c/321118/ ? Trivial patch, already has one +2, passing CI apart from one failure on a broken (and now disabled) gate | 12:08 |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-puppet-elements: Add mistral packages to controller image https://review.openstack.org/329504 | 12:09 |
| *** ccamacho|lunch is now known as ccamacho | 12:10 | |
| bandini | shardy: here is the output for controller-0 https://paste.fedoraproject.org/379478/65992672/, is there anything in particular that should catch attention? | 12:14 |
| *** mbound has quit IRC | 12:15 | |
| *** mbound has joined #tripleo | 12:15 | |
| *** ramishra has quit IRC | 12:16 | |
| openstackgerrit | Merged openstack/instack-undercloud: Add option to enable introspection of UEFI nodes https://review.openstack.org/321118 | 12:16 |
| *** adarazs_lunch is now known as adarazs | 12:17 | |
| *** paramite has joined #tripleo | 12:17 | |
| ccamacho | chem`` im back, yesterday landed https://review.openstack.org/#/c/326118/7/puppet/services/pacemaker/nova-consoleauth.yaml and by default is disabled | 12:18 |
| *** ramishra has joined #tripleo | 12:18 | |
| EmilienM | hello | 12:21 |
| *** fultonj has quit IRC | 12:22 | |
| hewbrocca | EmilienM: bonjour et bienvenue | 12:23 |
| *** trown|outtypewww is now known as trown | 12:23 | |
| *** fultonj has joined #tripleo | 12:23 | |
| mgould | hi EmilienM | 12:23 |
| trown | dtantsur: looks like tripleo got a promote on master last night, I think you were waiting on that for updated IPA? | 12:25 |
| *** Goneri has joined #tripleo | 12:26 | |
| dtantsur | trown, not IPA, but our ironic-on-overcloud work. it now passed, thanks | 12:26 |
| jaosorior | EmilienM: Just so you know, TripleO upgrades gate is broken. So if you have a patch that fails that, no need for rechecks | 12:26 |
| EmilienM | I just figured | 12:26 |
| EmilienM | why is it failing? | 12:26 |
| trown | hmm... wonder if that is related to the promote | 12:27 |
| jaosorior | bandini, ccamacho and chem`` are looking into it | 12:27 |
| trown | derekh: we promote solely based on ha+nonha ya? | 12:27 |
| ccamacho | Hey EmilienM here is the error https://bugs.launchpad.net/tripleo/+bug/1592776 | 12:28 |
| openstack | Launchpad bug 1592776 in tripleo "Ha upgrade jobs failing with "cluster remained unstable for more than 1800 seconds"" [Undecided,New] | 12:28 |
| EmilienM | shardy: so I half-figured why upgrade job is broken on liberty | 12:28 |
| trown | mgould: looking | 12:28 |
| EmilienM | ccamacho: thx | 12:28 |
| EmilienM | shardy: i think stable/liberty is missing some ipv6 backports, because it started to fail when we enable ipv6 onupgrade job, on March 30th | 12:29 |
| *** rlandy has joined #tripleo | 12:29 | |
| EmilienM | shardy: I'm doing local testing today and maybe we can sort this out but it should be a big deal | 12:29 |
| ccamacho | EmilienM Im starting to crawl into controller logs.. But after having lunch is much harder, need more coffee.. | 12:29 |
| trown | mgould: is there a follow-up backport of the undercloud.conf regeneration? | 12:30 |
| mgould | trown: yes, one moment | 12:30 |
| * mgould thanks jaosorior for the review | 12:30 | |
| EmilienM | ccamacho, dprince, dprince, thrash, jaosorior: composable standup? | 12:30 |
| mgould | trown: https://review.openstack.org/#/c/324553/ | 12:31 |
| ccamacho | yeahp | 12:31 |
| mgould | also passing all non-broken gates | 12:31 |
| ccamacho | joining | 12:31 |
| mgould | there are liberty backports too, but they're still failing CI | 12:31 |
| *** bfournie has joined #tripleo | 12:33 | |
| *** zoli|lunch is now known as zoli | 12:37 | |
| *** zoli is now known as zoliXXL | 12:37 | |
| openstackgerrit | Merged openstack/instack-undercloud: Fix inspection_enable_uefi description https://review.openstack.org/324553 | 12:37 |
| *** apetrich has quit IRC | 12:40 | |
| *** mbound has quit IRC | 12:40 | |
| shardy | EmilienM: thanks for the update, good that we understand the root-cause now :) | 12:41 |
| *** apetrich has joined #tripleo | 12:42 | |
| *** fzdarsky has joined #tripleo | 12:42 | |
| *** tbonds has quit IRC | 12:47 | |
| *** jprovazn has quit IRC | 12:48 | |
| *** itamarl has quit IRC | 12:58 | |
| *** fzdarsky has quit IRC | 12:58 | |
| *** tzumainn has joined #tripleo | 12:58 | |
| *** julim has joined #tripleo | 12:59 | |
| *** rbrady has joined #tripleo | 12:59 | |
| jaosorior | EmilienM: Hey dude, I'm looking into adding a custom fact to get different fqdn's depending on the network; Would this be appropriate for that? https://review.openstack.org/#/c/329299/ | 13:00 |
| *** dprince has quit IRC | 13:02 | |
| EmilienM | jaosorior: looking dude | 13:02 |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates: Enable Manila integration - as a composable controller service https://review.openstack.org/188137 | 13:02 |
| *** ibravo has joined #tripleo | 13:02 | |
| EmilienM | jaosorior: this is awesome! | 13:02 |
| EmilienM | jaosorior: I like the idea! | 13:02 |
| jaosorior | EmilienM: thanks! | 13:04 |
| *** jcoufal has quit IRC | 13:04 | |
| *** noslzzp has joined #tripleo | 13:05 | |
| openstackgerrit | Harry Rybacki proposed openstack/tripleo-quickstart: [WIP] Add scale to roles gate https://review.openstack.org/329542 | 13:05 |
| *** hewbrocca is now known as hewbrocca-afk | 13:06 | |
| *** hewbrocca-afk is now known as hewbrocca | 13:06 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Add redis constraint to aodh upgrade manifest https://review.openstack.org/329655 | 13:07 |
| openstackgerrit | Bob Callaway proposed openstack/tripleo-heat-templates: Enable Neutron LBaaS Integration https://review.openstack.org/313933 | 13:08 |
| *** jpena|lunch is now known as jpena | 13:10 | |
| EmilienM | bnemec: hey, my stuff in tripleo-ci for liberty does not seem to work http://logs.openstack.org/64/329664/4/check-tripleo/gate-tripleo-ci-centos-7-upgrades/67ee596/console.html#_2016-06-15_02_11_01_561 | 13:12 |
| EmilienM | bnemec: can you look https://review.openstack.org/329663 again please? | 13:12 |
| jaosorior | marios: Hey dude, regarding https://review.openstack.org/#/c/327029/10/manifests/profile/base/keystone.pp I have kept putting tls_cert_refresh_command's default as something else than undef, because I can't set the default explicitly in the parameter definition. It needs the "include ::apache::params" to come first. Else it won't find the service_name in the resource catalog | 13:13 |
| marios | jaosorior: ack thanks for clarifications , i am in a call right now. will likely revisit later/tomorrow | 13:14 |
| jaosorior | sure thing | 13:14 |
| *** rcernin has quit IRC | 13:14 | |
| jaosorior | marios: Thanks for taking a look at it dude | 13:14 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: DONT MERGE TESTING https://review.openstack.org/316436 | 13:14 |
| openstackgerrit | Brad P. Crochet proposed openstack/puppet-tripleo: Add Mistral profiles https://review.openstack.org/323431 | 13:15 |
| *** tbonds has joined #tripleo | 13:16 | |
| *** lblanchard has joined #tripleo | 13:17 | |
| EmilienM | marios: damn, stable/mitaka upgrade job also red? | 13:17 |
| EmilienM | master + stable/mitaka? | 13:17 |
| jaosorior | EmilienM: What? It shouldn't | 13:18 |
| EmilienM | jaosorior, marios, shardy, slagle, trown, bnemec: please do not approve patches until we sort things out for upgrade job - I noticed some patches landed this morning without all green jobs | 13:19 |
| EmilienM | marios: https://review.openstack.org/329874 | 13:19 |
| jaosorior | EmilienM: that is the random error where it times out in the "overcloud deploy" | 13:19 |
| jaosorior | not sure if it's a nova issue or ironic... but it happens very randomly | 13:20 |
| Ng | ccamacho: I never left here! :D | 13:20 |
| marios | EmilienM: ack don't think i did /usually avoid that | 13:20 |
| *** ramishra has quit IRC | 13:20 | |
| ccamacho | Ng hey! :) | 13:20 |
| Ng | ccamacho: thanks for the offer, I'm sure I'll be taking you up on it | 13:21 |
| EmilienM | I'm looking when it broke exactly | 13:21 |
| EmilienM | it broke yesterday between 4pm and 11pm on my TZ | 13:22 |
| EmilienM | I'm comparing packaging now | 13:22 |
| EmilienM | that's the list of package diff: https://www.diffchecker.com/bnubbtry (left job that passed upgrade and right job that failed, a few hours after) | 13:24 |
| hewbrocca | Ng: welcome! | 13:24 |
| EmilienM | a lot of puppet modules updates | 13:24 |
| *** bfournie has quit IRC | 13:25 | |
| *** jprovazn has joined #tripleo | 13:27 | |
| thrash | EmilienM: that looks more like it's just a different order than necessarily updates | 13:27 |
| *** rcernin has joined #tripleo | 13:28 | |
| thrash | EmilienM: odd tho... they don't even seem to *be* on the left (passing) | 13:28 |
| EmilienM | this diff is actually better: https://www.diffchecker.com/b59ji2sc | 13:28 |
| *** akshai has joined #tripleo | 13:28 | |
| thrash | that seems much more sane. :) | 13:29 |
| EmilienM | I sorted packages | 13:29 |
| EmilienM | sorry yeah | 13:29 |
| thrash | EmilienM: order different? Passing on right now? or left? | 13:30 |
| EmilienM | thrash: left is packages from a job that passed upgrade | 13:30 |
| EmilienM | thrash: right is a failing job | 13:30 |
| thrash | because otherwise, openstack-puppet-modules took a huge rollback. :) | 13:30 |
| thrash | openstack-puppet-modules-8.1.1-0.20160609150428.ab63b38.el7.centos.noarch -> openstack-puppet-modules-8.0.0-0.20160520142355.6a3e8bf.el7.centos.noarch | 13:31 |
| thrash | that's left to right | 13:31 |
| thrash | line 420 | 13:31 |
| ccamacho | lot of packages... | 13:31 |
| thrash | And puppet modules now coming from packages. | 13:31 |
| thrash | that's the two things I see. | 13:31 |
| panda | Looking at overcloud I see that puppet modules are symlinked to /etc/puppet/modules. What part of tripleo is creating those symlinks ? | 13:32 |
| EmilienM | thrash: there is a problem | 13:32 |
| EmilienM | thrash: puppet openstack modules version is 8.0 | 13:32 |
| EmilienM | it should be 9.0 no? | 13:32 |
| EmilienM | jayg: ^ | 13:32 |
| EmilienM | thrash: yeah this regression looks weird | 13:33 |
| jayg | 8 is mitaka for opm, what is the question? | 13:33 |
| EmilienM | jayg: we're investigating why upgrade job is broken in tripleo | 13:33 |
| EmilienM | jayg: something between yesterday 4pm and 11pm (our TZ) broke us | 13:34 |
| EmilienM | jayg: https://www.diffchecker.com/b59ji2sc | 13:34 |
| EmilienM | jayg: on your left, packages of a job that passed upgrade and on your right, packages from a job that failed upgrade job | 13:34 |
| jayg | I didn't tag anything in rdo yesterday, only did downstream build | 13:34 |
| * jayg looks | 13:34 | |
| EmilienM | why do we have openstack-puppet-modules-8.0.0-0.20160520142355.6a3e8bf.el7.centos.noarch on recent jobs? | 13:35 |
| thrash | EmilienM: and why would all of the puppet modules start coming from packages instead of source? | 13:35 |
| EmilienM | I have no idea | 13:36 |
| jayg | yeah, that is weird that the newer side shows older opm | 13:36 |
| EmilienM | let's confirm on other jobs | 13:37 |
| *** dtrainor has quit IRC | 13:37 | |
| EmilienM | yeah I confirm | 13:37 |
| EmilienM | on another (failing) very recent job: openstack-puppet-modules-8.0.0-0.20160520142355.6a3e8bf.el7.centos.noarch | 13:37 |
| trown | EmilienM: I wonder if it is promote of current-tripleo | 13:37 |
| *** dtrainor has joined #tripleo | 13:37 | |
| trown | EmilienM: it happened this morning | 13:37 |
| EmilienM | it's installing Mitaka | 13:37 |
| trown | derekh: promote does not check upgrades job? | 13:38 |
| dtantsur | EmilienM, hey, did you have a chance to see my response on https://review.openstack.org/#/c/319297/ ? I've just checked and the connection is set correctly in the resulting ironic.conf | 13:38 |
| EmilienM | trown: no, it started to fail between 4pm and 11 pm last night | 13:38 |
| trown | EmilienM: oh, promote happened this morning | 13:38 |
| EmilienM | trown: but maybe it's related... | 13:38 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart: use the ansible-role-tripleo-inventory to override native inventory https://review.openstack.org/329938 | 13:38 |
| openstackgerrit | Merged openstack/tripleo-ui: Register nodes new workflow https://review.openstack.org/323665 | 13:38 |
| derekh | trown: correct, https://review.openstack.org/#/c/315075/2/scripts/mirror-server/mirror-server.pp | 13:38 |
| EmilienM | dtantsur: will look when our CI is back | 13:38 |
| *** coolsvap has quit IRC | 13:39 | |
| jaosorior | EmilienM: the column of packages on the left, is it coming from a passing upgrades job? | 13:39 |
| EmilienM | derekh: any idea of what happens? | 13:39 |
| *** jcoufal has joined #tripleo | 13:39 | |
| trown | derekh: hmm that feels optimistic | 13:39 |
| dtantsur | EmilienM, sure, no hurry. just letting you know that it seems to work as expected | 13:39 |
| ccamacho | jaosorior yeahp | 13:39 |
| EmilienM | jaosorior: like I said, left is green, right is red | 13:39 |
| jaosorior | dafuq | 13:39 |
| derekh | EmilienM: what happen on what? I havn't been following along, /me reads back | 13:39 |
| trown | I guess upgrades code path is not really dependent on anything external to tripleo that the other jobs are though | 13:39 |
| EmilienM | derekh: start at XX:33:49 | 13:40 |
| *** mbound has joined #tripleo | 13:41 | |
| *** jcoufal_ has joined #tripleo | 13:41 | |
| EmilienM | trown: can we compare packages before/after promotion? | 13:42 |
| trown | EmilienM: ya we can find the previous hash from https://trunk.rdoproject.org/centos7/promote-current-tripleo.log and compare the versions.csv in each | 13:44 |
| *** jcoufal has quit IRC | 13:45 | |
| trown | EmilienM: https://trunk.rdoproject.org/centos7/39/b4/39b44bf2ee28cc21ce92e5cd694cd82a4ad7ac8f_6bf0c01f/versions.csv vs https://trunk.rdoproject.org/centos7/db/aa/dbaa9e6db36181e1ec6d1c00b086fc6fb45e90e2_6686315c/versions.csv | 13:45 |
| *** mbound has quit IRC | 13:46 | |
| EmilienM | trown: opm is on same hash | 13:47 |
| trown | EmilienM: but puppet modules are not http://chunk.io/f/e8484db8ae1e4fbd82f90f7000a42f1b | 13:47 |
| *** rodrigods has quit IRC | 13:47 | |
| *** rodrigods has joined #tripleo | 13:48 | |
| trown | there are not many packages that DID NOT change | 13:48 |
| derekh | EmilienM: looking into it now, will shout if I find anything | 13:49 |
| EmilienM | thx | 13:49 |
| EmilienM | i'm quite sure this puppet regression makes the upgrade failing | 13:50 |
| EmilienM | we can stop investigating pacemaker & things | 13:50 |
| EmilienM | chem``: ^ | 13:50 |
| *** pradk has joined #tripleo | 13:51 | |
| jaosorior | Now I'm seeing some errors related to the installation of tripleo-common | 13:51 |
| jaosorior | Error: Execution of '/bin/yum -d 0 -e 0 -y list tripleo-common' returned 1: Error: No matching Packages to list | 13:51 |
| jaosorior | Error: /Stage[main]/Main/Package[tripleo-common]/ensure: change from absent to present failed: Execution of '/bin/yum -d 0 -e 0 -y list tripleo-common' returned 1: Error: No matching Packages to list | 13:52 |
| EmilienM | trown: thi sis the last patch that worked: https://review.openstack.org/#/c/312420/ | 13:52 |
| jaosorior | I've seen that in a couple of patches in the past hour | 13:52 |
| EmilienM | it was almost 4pm | 13:52 |
| jaosorior | EmilienM: That's stable/mitaka; shouldn't we be looking for CRs for master? | 13:52 |
| *** dtrainor has quit IRC | 13:54 | |
| *** rcernin has quit IRC | 13:54 | |
| EmilienM | jaosorior: indeed | 13:54 |
| *** dtrainor has joined #tripleo | 13:54 | |
| EmilienM | jaosorior: the most recent patch I have is https://review.openstack.org/#/c/328361/ | 13:55 |
| EmilienM | it was in the morning | 13:55 |
| EmilienM | doing diff again | 13:55 |
| *** links has quit IRC | 13:56 | |
| EmilienM | jaosorior: https://www.diffchecker.com/f9hkwqmj | 13:59 |
| *** dprince has joined #tripleo | 13:59 | |
| EmilienM | this is diff between https://review.openstack.org/#/c/324541/ and https://review.openstack.org/#/c/328361/ | 14:00 |
| jaosorior | now that looks like a more reasonable search | 14:00 |
| EmilienM | yeah | 14:01 |
| jaosorior | only thing that merged in os-net-config was this https://review.openstack.org/#/c/291384/2 | 14:01 |
| jaosorior | which is only adding debug statements | 14:01 |
| *** ibravo2 has joined #tripleo | 14:01 | |
| *** egafford has joined #tripleo | 14:01 | |
| EmilienM | yeah | 14:02 |
| jaosorior | so it must be something from t-h-t | 14:02 |
| *** cdearborn has joined #tripleo | 14:02 | |
| jaosorior | we now have to fix the undercloud | 14:03 |
| jaosorior | which is now broken it seems | 14:03 |
| jaosorior | so now officially all the gate is red | 14:03 |
| jaosorior | crapo | 14:03 |
| jaosorior | derekh, any idea what might be causing tripleo-common not to be found like I posted above? ^^ | 14:04 |
| EmilienM | jaosorior: ask on #rdo | 14:04 |
| EmilienM | maybe it's a repo thing | 14:04 |
| *** ibravo has quit IRC | 14:05 | |
| ccamacho | Im deploying from https://review.openstack.org/#/q/project:openstack/tripleo-heat-templates+status:merged the patches from the morning to see when fails.. time consuming task.. | 14:05 |
| derekh | jaosorior: no idea off the top of my head, will take a look in a few minutes, tracking down the other error first | 14:06 |
| *** tbonds has quit IRC | 14:06 | |
| jaosorior | EmilienM, derekh: they said that that package got renamed not too long ago | 14:07 |
| jaosorior | to openstack-tripleo-common | 14:07 |
| *** tbonds has joined #tripleo | 14:07 | |
| derekh | jaosorior: that would do it | 14:08 |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-quickstart: Prepare tempest if running it https://review.openstack.org/329082 | 14:08 |
| jaosorior | gonna change the name in instack-undercloud | 14:08 |
| openstackgerrit | Emilien Macchi proposed openstack/instack-undercloud: Update tripleo-common package name https://review.openstack.org/329961 | 14:08 |
| EmilienM | jaosorior: ^ | 14:08 |
| *** olap has quit IRC | 14:08 | |
| *** rcernin has joined #tripleo | 14:09 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/instack-undercloud: Use renamed tripleo-common package https://review.openstack.org/329962 | 14:09 |
| jaosorior | EmilienM: ok, let me abandon my change | 14:09 |
| EmilienM | jaosorior: sorry man | 14:10 |
| jaosorior | haha no worries | 14:10 |
| *** olap has joined #tripleo | 14:10 | |
| jaosorior | the point is to get it fixed; not who fixes it | 14:10 |
| jaosorior | EmilienM: +2ed your change | 14:10 |
| EmilienM | trown: when was last promotion before the latest? | 14:10 |
| EmilienM | package was renamed 14 days ago | 14:10 |
| EmilienM | how did we miss it? | 14:11 |
| jaosorior | that's an excellent question | 14:11 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/puppet-tripleo: WIP: integration of the new puppet pacemaker. https://review.openstack.org/309069 | 14:12 |
| EmilienM | like apevec said, there is a Provides, so I don't understand | 14:13 |
| EmilienM | ah indeed puppet3 does not support well virtual packages | 14:13 |
| EmilienM | so we need this undercloud patch | 14:13 |
| *** masco has quit IRC | 14:13 | |
| jaosorior | alright, makes sense then | 14:14 |
| EmilienM | I'm not sure it's going to fix upgrade job though | 14:14 |
| jaosorior | it won't | 14:15 |
| jaosorior | that seems to be another issue | 14:15 |
| *** fzdarsky has joined #tripleo | 14:17 | |
| jaosorior | pradk: upgrades gate is broken. recheck won't help | 14:18 |
| *** rcernin has quit IRC | 14:18 | |
| ccamacho | jaosorior this patch https://review.openstack.org/#/c/327307/ | 14:18 |
| *** trown is now known as trown|mtg | 14:18 | |
| ccamacho | mmm im deploying the previous one | 14:18 |
| pradk | jaosorior, good to know, thx | 14:19 |
| jaosorior | ccamacho: it had passed at some point. I can try doing a revert for that though | 14:19 |
| ccamacho | jaosorior, no wait until I have the prev one deployed.. | 14:19 |
| jaosorior | ccamacho: ?? | 14:20 |
| jaosorior | ah | 14:20 |
| jaosorior | you're testing locally | 14:20 |
| jaosorior | alright | 14:20 |
| ccamacho | yeahp | 14:20 |
| *** hjensas__ has quit IRC | 14:22 | |
| *** jrist has quit IRC | 14:23 | |
| *** zoliXXL is now known as zoli|mtg | 14:24 | |
| *** dprince has quit IRC | 14:25 | |
| shardy | slagle: Hey, any thoughts on how we might decommission https://github.com/agroup/ ? | 14:26 |
| shardy | I encountered some folks referring to the old instack* stuff in there recently | 14:26 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart: use the ansible-role-tripleo-inventory to override native inventory https://review.openstack.org/329938 | 14:26 |
| *** rcernin has joined #tripleo | 14:30 | |
| derekh | shardy: slagle I see a "Delete this organization" button | 14:32 |
| *** apetrich has quit IRC | 14:34 | |
| *** jefrite has quit IRC | 14:34 | |
| *** apetrich has joined #tripleo | 14:36 | |
| *** jrist has joined #tripleo | 14:36 | |
| *** jrist has joined #tripleo | 14:36 | |
| openstackgerrit | John Trowbridge proposed openstack/tripleo-quickstart: use environmental variables for ansible ssh configuration https://review.openstack.org/329124 | 14:37 |
| *** bfournie has joined #tripleo | 14:43 | |
| *** trumpetnl has quit IRC | 14:44 | |
| openstackgerrit | Brad P. Crochet proposed openstack/python-tripleoclient: Add Mistral password to deployment https://review.openstack.org/329987 | 14:49 |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-heat-templates: Composable Mistral services https://review.openstack.org/323436 | 14:50 |
| *** panda has quit IRC | 14:50 | |
| *** panda has joined #tripleo | 14:50 | |
| *** numans has quit IRC | 14:55 | |
| *** yamahata has quit IRC | 14:59 | |
| *** yamahata has joined #tripleo | 14:59 | |
| *** cllewellyn_ has quit IRC | 15:01 | |
| *** cllewellyn__ has quit IRC | 15:01 | |
| *** zoli|mtg is now known as zoli | 15:03 | |
| openstackgerrit | Adriano Petrich proposed openstack/tripleo-quickstart: inject debug options on the under/overcloud images https://review.openstack.org/329999 | 15:03 |
| *** hewbrocca is now known as hewbrocca-afk | 15:05 | |
| EmilienM | jaosorior: back | 15:05 |
| EmilienM | jaosorior: so it seems my patch helps, it's passing more than 50 min | 15:06 |
| EmilienM | so I guess it's deploying overcloud now | 15:06 |
| *** cllewellyn_ has joined #tripleo | 15:06 | |
| *** cllewellyn__ has joined #tripleo | 15:06 | |
| *** hewbrocca-afk is now known as hewbrocca | 15:06 | |
| EmilienM | ccamacho, jaosorior, derekh: any update on upgrade job failure? | 15:06 |
| ccamacho | EmilienM im finishing to run the minor upgrade in https://review.openstack.org/#/c/328361/ @ Jun 14 7:17 PM as passed/merged the next one, failed/merged https://review.openstack.org/#/c/327318/ @ Jun 15 7:22 AM <-Networking related | 15:08 |
| ccamacho | if my test pass might be related to THT | 15:08 |
| ccamacho | if not the problem happened between Jun 14 7:17 PM and Jun 15 7:22 AM | 15:08 |
| *** tobias_fiberdata has quit IRC | 15:09 | |
| *** numans has joined #tripleo | 15:09 | |
| ccamacho | im basically based in this patches list https://review.openstack.org/#/q/project:openstack/tripleo-heat-templates+status:merged | 15:09 |
| jaosorior | ccamacho: You da man | 15:10 |
| chem`` | EmilienM: ccamacho jaosorior derekh I've got an error I've never had before: http://logs.openstack.org/09/302409/43/check-tripleo/gate-tripleo-ci-centos-7-ha/8aca348/console.html#_2016-06-15_15_03_52_967 | 15:10 |
| jaosorior | EmilienM: Just waiting for your patch to pass CI so I can merge it | 15:10 |
| chem`` | EmilienM: does it looks related to your patch or I just recheck ? | 15:11 |
| jaosorior | chem`` that error is being fixed | 15:11 |
| jaosorior | by EmilienM's patch | 15:11 |
| chem`` | jaosorior: oki, so this is the tripleo package name stuff | 15:11 |
| EmilienM | chem``: http://logs.openstack.org/09/302409/43/check-tripleo/gate-tripleo-ci-centos-7-ha/8aca348/logs/undercloud/var/log/undercloud_install.txt.gz#_2016-06-15_14_51_58_000 | 15:11 |
| marios | jistr: sorry forgot to say, didn't get round to revisit the docs patches... bnemec thanks for comments there will do, i'll have another pass tomorrow | 15:12 |
| EmilienM | chem``: yes it is | 15:12 |
| derekh | EmilienM: I havn't find anything in the logs yet that seems relevant | 15:12 |
| EmilienM | :( | 15:12 |
| jistr | marios: sure thing | 15:12 |
| *** bfournie has quit IRC | 15:16 | |
| EmilienM | the good news is upgrade job still working on stable/mitaka | 15:20 |
| EmilienM | so it's really something in master | 15:20 |
| *** ebarrera has quit IRC | 15:21 | |
| *** hjensas__ has joined #tripleo | 15:22 | |
| jistr | bandini, marios: to have the thing complete and behaving nice, one probably needs all three of these patches https://github.com/openstack/tripleo-common/commits/master/undercloud_heat_plugins/server_update_allowed.py | 15:22 |
| marios | jistr: ack nice thanks | 15:23 |
| ccamacho | jaosorior, EmilienM, chem``, derekh got a different issue deploying the minor upgrade in https://review.openstack.org/#/c/328361/ (with the depends puppet-tripleo) , damn http://paste.openstack.org/show/516289/ | 15:23 |
| *** ifarkas has quit IRC | 15:23 | |
| EmilienM | ccamacho: logs of nova compute? | 15:24 |
| *** aufi has quit IRC | 15:24 | |
| EmilienM | why doesn't it start? | 15:24 |
| *** leanderthal is now known as leanderthal|afk | 15:24 | |
| ccamacho | logging in there | 15:24 |
| bandini | jistr: + this that marios mentioned right? https://review.openstack.org/#/c/283832 | 15:25 |
| bandini | so 4 patches in total | 15:25 |
| openstackgerrit | Sanjay Upadhyay proposed openstack/tripleo-specs: new spec: tripleo-sriov https://review.openstack.org/313872 | 15:26 |
| marios | bandini: no should be those three in total | 15:26 |
| chem`` | ccamacho: failing restart is the problem we've seen this morning | 15:27 |
| marios | bandini: you can get to the reviews like gerrit_url="http://review.openstack.org/#q,$1,n,z" where $1 is the change id from those commits | 15:27 |
| bandini | marios: right, I am blind | 15:27 |
| ccamacho | yeahp | 15:27 |
| ccamacho | EmilienM, http://paste.openstack.org/show/516290/ | 15:27 |
| EmilienM | mhh | 15:28 |
| d0ugal | My undercloud install has stopped at: 2016-06-15 15:24:32 - Notice: /Stage[main]/Nova::Cert/Nova::Generic_service[cert]/Service[nova-cert]/ensure: ensure changed 'stopped' to 'running' | 15:28 |
| EmilienM | it sounds like rabbitmq is down or something? | 15:28 |
| d0ugal | any one else hitting this? | 15:28 |
| EmilienM | ccamacho: it's only during upgrade? | 15:28 |
| EmilienM | ccamacho: or also during deployment | 15:28 |
| ccamacho | yeahp | 15:28 |
| ccamacho | the deployment went fine | 15:28 |
| ccamacho | only in the upgrade | 15:28 |
| EmilienM | ok | 15:28 |
| EmilienM | that"s super interesting | 15:28 |
| EmilienM | ccamacho: rabbitmq status? up? | 15:29 |
| EmilienM | ccamacho: can you also paste nova.conf please? | 15:29 |
| ccamacho | yeahp wait a min | 15:29 |
| openstackgerrit | Brad P. Crochet proposed openstack/python-tripleoclient: Add Mistral password to deployment https://review.openstack.org/329987 | 15:29 |
| openstackgerrit | Brad P. Crochet proposed openstack/puppet-tripleo: Add Mistral profiles https://review.openstack.org/323431 | 15:31 |
| *** coolsvap has joined #tripleo | 15:31 | |
| EmilienM | oh wait | 15:31 |
| *** apetrich has quit IRC | 15:31 | |
| EmilienM | I need to see your nova.conf | 15:32 |
| EmilienM | I can also look at jobs | 15:32 |
| EmilienM | mhh, no | 15:32 |
| ccamacho | EmilienM http://paste.openstack.org/show/516295/ | 15:33 |
| ccamacho | rabbit on the controller is running, let me check the config | 15:33 |
| *** apetrich has joined #tripleo | 15:34 | |
| EmilienM | let me check something | 15:34 |
| ccamacho | This is wrong.. auth_url=http://192.0.2.21:35357/v3 right ? | 15:34 |
| *** bfournie has joined #tripleo | 15:34 | |
| *** shardy has quit IRC | 15:35 | |
| *** dsariel has quit IRC | 15:35 | |
| EmilienM | on the compute, we have rabbit_userid=guest | 15:35 |
| EmilienM | not on the controller | 15:35 |
| hewbrocca | O NOES not the damn rabbit password again | 15:36 |
| *** saneax is now known as saneax_AFK | 15:37 | |
| *** olap has quit IRC | 15:37 | |
| EmilienM | ok I might have something | 15:37 |
| *** mcornea has quit IRC | 15:40 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: compute: align rabbitmq configuration with nova-base service https://review.openstack.org/330022 | 15:41 |
| EmilienM | ccamacho: ^ I'm not sure it will fix the problem | 15:41 |
| EmilienM | but it's at least a good cleanup | 15:41 |
| EmilienM | ccamacho: were you able to start nova compute by hand? | 15:42 |
| *** tobias_fiberdata has joined #tripleo | 15:44 | |
| *** mikelk has quit IRC | 15:45 | |
| jaosorior | EmilienM: it makes sense as a cleanup | 15:45 |
| EmilienM | jaosorior: yeah but I figured that nova.conf rabbit things are diff on controler/compute | 15:47 |
| EmilienM | that's not good ^ | 15:47 |
| ccamacho | EmilienM dead.. http://paste.openstack.org/show/516306/ | 15:47 |
| *** zoli is now known as zoli|gone | 15:48 | |
| EmilienM | ccamacho: wait | 15:49 |
| EmilienM | don't touch anything | 15:49 |
| ccamacho | EmilienM ack | 15:49 |
| *** oshvartz has quit IRC | 15:50 | |
| EmilienM | ccamacho: can I ssh maybe | 15:50 |
| EmilienM | ? | 15:50 |
| ccamacho | sure | 15:50 |
| *** pcaruana has quit IRC | 15:50 | |
| *** xinwu has joined #tripleo | 15:50 | |
| EmilienM | ccamacho: how is controller going? nova conductor for example, can you restart it? | 15:51 |
| *** jrist has quit IRC | 15:52 | |
| EmilienM | I want to see if rabbit is only unavailable for compute service or for everything | 15:52 |
| jaosorior | EmilienM: Well, they should be different, I guess | 15:52 |
| *** trown|mtg is now known as trown | 15:52 | |
| *** dmk0202 has quit IRC | 15:52 | |
| EmilienM | jaosorior: what different? | 15:52 |
| jaosorior | now that I think about it, this is gonna end up being good for security; they should have different credentials | 15:52 |
| jaosorior | controllers and computes | 15:52 |
| *** rcernin has quit IRC | 15:53 | |
| EmilienM | jaosorior: we talked about it during summit with ayoung | 15:53 |
| EmilienM | let me find this issue first | 15:53 |
| jaosorior | yep | 15:53 |
| EmilienM | this is not related | 15:53 |
| ayoung | I'll; join the convo in a sec...in another right now | 15:54 |
| jaosorior | ayoung: No convo about that yet. First we gotta debug something wrong in the upgrades gate | 15:55 |
| ayoung | k | 15:55 |
| EmilienM | ok same for nova conductor | 15:55 |
| EmilienM | so it's not only compute | 15:55 |
| *** tesseract has quit IRC | 15:55 | |
| EmilienM | same for cinder schedule as an example | 15:58 |
| EmilienM | so something is wrong with credentials in general | 15:58 |
| *** zoli|gone is now known as zoli_gone-proxy | 15:58 | |
| *** ohamada_ has quit IRC | 15:59 | |
| ccamacho | EmilienM so the creds are messed up? | 15:59 |
| EmilienM | maybe, let me find why | 15:59 |
| *** noslzzp has quit IRC | 15:59 | |
| *** xinwu has quit IRC | 16:00 | |
| EmilienM | ccamacho: do you have a nova.conf pre upgrade by any chance? | 16:00 |
| *** pkovar has quit IRC | 16:01 | |
| EmilienM | I'm wondering if credentials were updated during update | 16:01 |
| *** jcoufal has joined #tripleo | 16:01 | |
| EmilienM | ccamacho: we should try again by 1) deploying overcloud 2) backup nova.conf on controller/compute nodes 3) run update 4) compare config files | 16:02 |
| EmilienM | can we try that? | 16:02 |
| EmilienM | I suspect a change during the update that breaks services | 16:03 |
| *** jcoufal_ has quit IRC | 16:03 | |
| EmilienM | wait, credentials are good | 16:03 |
| EmilienM | you can see them in /etc/rabbitmq/rabbitmq.config | 16:04 |
| ccamacho | EmilienM nope :( but | 16:04 |
| openstackgerrit | John Trowbridge proposed openstack/tripleo-quickstart: Use quickstart.sh to manage venv in all ci-scripts https://review.openstack.org/330040 | 16:04 |
| ccamacho | just installed the pacemaker env, and then minor upgrade | 16:04 |
| ccamacho | the only thing i did there | 16:05 |
| EmilienM | ccamacho: trying to restart rabbit | 16:05 |
| *** athomas has quit IRC | 16:05 | |
| *** numans has quit IRC | 16:07 | |
| *** ramishra has joined #tripleo | 16:07 | |
| *** ramishra has quit IRC | 16:07 | |
| EmilienM | ccamacho: it works | 16:07 |
| EmilienM | I did one thing: pcs resource restart rabbitmq | 16:08 |
| EmilienM | so I'm not sure why but rabbitmq needs to be restarted during the update | 16:08 |
| ccamacho | without changing anything ?¿ | 16:08 |
| openstackgerrit | Jakub Libosvar proposed openstack/tripleo-heat-templates: Rename Neutron database name https://review.openstack.org/330042 | 16:08 |
| ccamacho | mmm | 16:08 |
| EmilienM | nope | 16:08 |
| EmilienM | and I'm not sure it's related to our CI issue | 16:08 |
| EmilienM | ccamacho: is it? | 16:08 |
| *** krotscheck is now known as krotscheck_dcm | 16:09 | |
| *** tremble has quit IRC | 16:09 | |
| ccamacho | not sure, it should worked.. I will re-deploy it.. but it should worked.. as It passed CI... | 16:10 |
| *** [1]cdearborn has joined #tripleo | 16:10 | |
| dtrainor | I have a failed deployment. I show CREATE_FAILED for Compute and Controller with 'heat stack-list --show-nested -f stack_status=CREATE_FAILED', but no failures in 'heat resource-list foo'. Looking at the resource details via resource-show doesn't give me any clues either. Where else can I look for information? | 16:10 |
| *** tobias_fiberdata has quit IRC | 16:11 | |
| EmilienM | ccamacho: so | 16:12 |
| EmilienM | ccamacho: if we compare with our CI failures | 16:12 |
| EmilienM | in CI we also have rabbit issues, or? let me verify | 16:13 |
| *** ramishra has joined #tripleo | 16:13 | |
| EmilienM | http://logs.openstack.org/14/329714/1/check-tripleo/gate-tripleo-ci-centos-7-upgrades/dec4380/logs/overcloud-novacompute-0/var/log/nova/nova-compute.txt.gz#_2016-06-15_07_56_50_312 | 16:14 |
| ccamacho | EmilienM, if you deploy master and execute the minor upgrade, you will reproduce the error as is in the bug ticket, I tried to deploy a prev commit which passed CI to see if the error is related to THT but the rabbit issue hit me in the face.. | 16:14 |
| *** tobias_fiberdata has joined #tripleo | 16:15 | |
| *** milan has quit IRC | 16:16 | |
| EmilienM | it sounds failing during ControllerPostPuppetRestartDeployment | 16:16 |
| hewbrocca | silly wabbit | 16:16 |
| ccamacho | Emilien, in the env you logged I was launching https://review.openstack.org/#/c/328361/ which passed | 16:16 |
| EmilienM | I think it fails during extraconfig/tasks/pacemaker_resource_restart.sh | 16:17 |
| EmilienM | when it restart rabbit | 16:17 |
| EmilienM | everything in logs point to rabbit | 16:17 |
| EmilienM | jistr: you still around? | 16:18 |
| hewbrocca | EmilienM: He just left :( | 16:18 |
| derekh | slagle: bnemec been waiting for DNS to update so I can rerecord the rh2 deployment, will be doing it later tonight | 16:18 |
| derekh | slagle: bnemec gonna try and condense down to a 15 minute video to post somewhere | 16:19 |
| EmilienM | launchpad 1567385 | 16:19 |
| openstack | Launchpad bug 1567385 in tripleo "Minor update always triggered on first stack-deploy after major upgrade" [High,Fix released] https://launchpad.net/bugs/1567385 - Assigned to Jiří Stránský (jistr) | 16:19 |
| *** pkovar has joined #tripleo | 16:19 | |
| hewbrocca | Oh no, is it that one? | 16:19 |
| EmilienM | no | 16:20 |
| EmilienM | launchpad 1567384 | 16:20 |
| openstack | Launchpad bug 1567384 in tripleo "Services not restarted on stack-update - config changes can go unapplied" [High,Fix released] https://launchpad.net/bugs/1567384 - Assigned to Jiří Stránský (jistr) | 16:20 |
| ccamacho | derekh let me know when published to link it to the tripleo channel I have created (https://www.youtube.com/channel/UCNGDxZGwUELpgaBoLvABsTA) | 16:20 |
| EmilienM | ok that might be that one | 16:20 |
| *** jaosorior has quit IRC | 16:20 | |
| derekh | ccamacho: will do | 16:20 |
| EmilienM | ccamacho: ok I know where is the issue but can't find why, I'm going to add debug in script and kick off CI jobs | 16:21 |
| openstackgerrit | Dan Radez proposed openstack/tripleo-heat-templates: Adding Congress Support https://review.openstack.org/330050 | 16:21 |
| derekh | EmilienM: gotta run, sorry wasn't any help | 16:21 |
| ccamacho | EmilienM nice! What's the problem then? | 16:22 |
| EmilienM | we got it covered | 16:22 |
| EmilienM | just please don't merge anything until we fix this | 16:22 |
| derekh | ack | 16:22 |
| *** derekh has quit IRC | 16:22 | |
| EmilienM | ccamacho: the pcs resource restart rabbit | 16:22 |
| EmilienM | ccamacho: maybe it fails | 16:22 |
| ccamacho | ack | 16:22 |
| EmilienM | it causes to cloud to go down | 16:22 |
| EmilienM | ccamacho: let me some time, I continue to read logs | 16:24 |
| ccamacho | too late :( | 16:24 |
| EmilienM | ccamacho: not on your setup | 16:25 |
| EmilienM | ccamacho: you can break your setup | 16:25 |
| *** cdearborn has quit IRC | 16:25 | |
| ccamacho | EmilienM :) sure then :) | 16:25 |
| ccamacho | Anyway your keys will remain in the undercloud without problems.. | 16:26 |
| *** pkovar has quit IRC | 16:28 | |
| *** numans has joined #tripleo | 16:30 | |
| *** dprince has joined #tripleo | 16:31 | |
| *** tobias_fiberdata has quit IRC | 16:34 | |
| *** jpena is now known as jpena|off | 16:35 | |
| *** cwolferh has joined #tripleo | 16:36 | |
| *** trown is now known as trown|lunch | 16:39 | |
| *** mgould is now known as mgould|afk | 16:40 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: DO NOT MERFGE - debug why upgrade job fails https://review.openstack.org/330069 | 16:42 |
| EmilienM | ccamacho: ^ | 16:42 |
| *** pkovar has joined #tripleo | 16:42 | |
| EmilienM | let's see how it goes | 16:42 |
| ccamacho | EmilienM ack | 16:42 |
| *** dmacpher is now known as dmacpher-afk | 16:43 | |
| *** mbound has joined #tripleo | 16:43 | |
| EmilienM | any core around please look https://review.openstack.org/#/c/329961/ and see if we can land it | 16:43 |
| EmilienM | I think yes | 16:43 |
| EmilienM | now I see some HA jobs failing too | 16:44 |
| EmilienM | http://logs.openstack.org/61/329961/1/check-tripleo/gate-tripleo-ci-centos-7-ha/20f4a7a/logs/postci.txt.gz#_2016-06-15_15_42_27_000 | 16:44 |
| *** cdearborn has joined #tripleo | 16:44 | |
| * EmilienM brb lunch | 16:46 | |
| *** yolanda has quit IRC | 16:46 | |
| *** xinwu has joined #tripleo | 16:49 | |
| *** [2]cdearborn has joined #tripleo | 16:50 | |
| *** cllewellyn_ has quit IRC | 16:53 | |
| *** cllewellyn__ has quit IRC | 16:53 | |
| *** apetrich has quit IRC | 16:53 | |
| *** oshvartz has joined #tripleo | 16:56 | |
| openstackgerrit | Mike Burns proposed openstack/tripleo-common: update removed undercloud-package-install https://review.openstack.org/330084 | 16:56 |
| *** milan has joined #tripleo | 16:56 | |
| *** yamahata has quit IRC | 16:56 | |
| *** apetrich has joined #tripleo | 16:58 | |
| *** [1]cdearborn has quit IRC | 16:58 | |
| EmilienM | trown|lunch: I know it's bad but I think we can land https://review.openstack.org/#/c/329961/ as it fix undercloud | 16:58 |
| EmilienM | and ha/upgade failures are not related I think | 16:59 |
| EmilienM | but yeah it's bad | 16:59 |
| openstackgerrit | Lars Kellogg-Stedman proposed openstack/tripleo-quickstart: make --requirements cumulative https://review.openstack.org/330086 | 16:59 |
| * EmilienM afk lunch | 16:59 | |
| *** NobodyCam has quit IRC | 16:59 | |
| *** igorbelikov has quit IRC | 16:59 | |
| *** dtantsur is now known as dtantsur|afk | 17:00 | |
| openstackgerrit | Dan Prince proposed openstack/tripleo-common: Add RegisterNodesAction action https://review.openstack.org/319587 | 17:00 |
| *** NobodyCam has joined #tripleo | 17:00 | |
| *** igorbelikov has joined #tripleo | 17:01 | |
| openstackgerrit | Lars Kellogg-Stedman proposed openstack/tripleo-quickstart: Use quickstart.sh to manage venv in all ci-scripts https://review.openstack.org/330040 | 17:01 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-common: Add baremetal workflows https://review.openstack.org/300200 | 17:05 |
| *** cdearborn has quit IRC | 17:06 | |
| *** cdearborn has joined #tripleo | 17:07 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart: move release var from positional to an argument https://review.openstack.org/330091 | 17:09 |
| openstackgerrit | Brad P. Crochet proposed openstack/puppet-tripleo: Add Mistral profiles https://review.openstack.org/323431 | 17:09 |
| *** pkovar has quit IRC | 17:10 | |
| *** coolsvap has quit IRC | 17:11 | |
| *** numans has quit IRC | 17:13 | |
| *** bswartz has quit IRC | 17:13 | |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-heat-templates: Composable Mistral services https://review.openstack.org/323436 | 17:14 |
| *** fzdarsky is now known as fzdarsky|afk | 17:14 | |
| *** yamahata has joined #tripleo | 17:15 | |
| openstackgerrit | Pradeep Kilambi proposed openstack/python-tripleoclient: Fix keystone init https://review.openstack.org/330096 | 17:19 |
| *** [2]cdearborn has quit IRC | 17:21 | |
| *** pcaruana has joined #tripleo | 17:21 | |
| *** ramishra has quit IRC | 17:25 | |
| *** noslzzp has joined #tripleo | 17:29 | |
| *** fragatina has quit IRC | 17:29 | |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui: Nodes Introspection new workflow https://review.openstack.org/330115 | 17:34 |
| *** electrofelix has quit IRC | 17:34 | |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui: Nodes Introspection new workflow https://review.openstack.org/330115 | 17:34 |
| openstackgerrit | Merged openstack/tripleo-common: Add --json-output option to tripleo-build-images https://review.openstack.org/327830 | 17:40 |
| openstackgerrit | Brad P. Crochet proposed openstack/puppet-tripleo: Add Mistral profiles https://review.openstack.org/323431 | 17:44 |
| openstackgerrit | Merged openstack/instack-undercloud: Update tripleo-common package name https://review.openstack.org/329961 | 17:45 |
| *** akshai has quit IRC | 17:47 | |
| *** akshai has joined #tripleo | 17:47 | |
| ccamacho | EmilienM jaosorior, just to update you, I have redeployed a passing submission not affected by the timeout issue (The one with the rabbit issue i just re-deployed it again https://review.openstack.org/#/c/328361/) and had failed with the same error (1800 secs timeout when minor upgrade), so I dont think is related to THT or puppet-tripleo. It might be a package breaking the deployment? | 17:48 |
| ccamacho | Emilien I will leave the the deployment in that state just in case you want to log in and see the environment | 17:49 |
| EmilienM | back from lunch | 17:49 |
| EmilienM | ccamacho: mhh ok | 17:49 |
| ccamacho | in this case rabbit restarted without issues but then the timeout issue, this is the current state of the overcloud deployment http://paste.openstack.org/show/516324/ | 17:51 |
| *** trown|lunch is now known as trown | 17:51 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: DO NOT MERFGE - debug why upgrade job fails https://review.openstack.org/330069 | 17:58 |
| EmilienM | ccamacho: really I have no idea what's going on | 17:59 |
| openstackgerrit | Pradeep Kilambi proposed openstack/python-tripleoclient: Run post deploy config on force https://review.openstack.org/330096 | 17:59 |
| *** bswartz has joined #tripleo | 18:04 | |
| openstackgerrit | Pradeep Kilambi proposed openstack/python-tripleoclient: Run post deploy config on force https://review.openstack.org/330096 | 18:04 |
| trown | EmilienM: I am pretty confused how we got to this mess with tripleo-common vs openstack-tripleo-common | 18:05 |
| trown | EmilienM: as you said that rename happened two weeks ago | 18:05 |
| *** mbound has quit IRC | 18:06 | |
| EmilienM | trown: me too | 18:08 |
| EmilienM | so many CI issues this week | 18:08 |
| openstackgerrit | Harry Rybacki proposed openstack/tripleo-quickstart: [WIP] Add scale to roles gate https://review.openstack.org/329542 | 18:08 |
| *** ccamacho is now known as ccamacho|out | 18:08 | |
| *** fragatina has joined #tripleo | 18:08 | |
| *** akshai_ has joined #tripleo | 18:10 | |
| *** akshai has quit IRC | 18:13 | |
| openstackgerrit | John Trowbridge proposed openstack/tripleo-quickstart: Use quickstart.sh to manage venv in all ci-scripts https://review.openstack.org/330040 | 18:14 |
| *** cwolferh has quit IRC | 18:15 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: controller/cinder: set auth_uri with version-less endpoint https://review.openstack.org/330129 | 18:15 |
| EmilienM | bnemec: thx | 18:17 |
| openstackgerrit | Harry Rybacki proposed openstack/tripleo-quickstart: [WIP] Add scale to roles gate https://review.openstack.org/329542 | 18:17 |
| *** akshai_ has quit IRC | 18:17 | |
| *** akshai has joined #tripleo | 18:17 | |
| dprince | trown: how long as it been since we promoted? | 18:19 |
| dprince | EmilienM: what did you want help w/? | 18:20 |
| trown | dprince: there was a promote this morning | 18:20 |
| dprince | trown: so is that why we are broken perhaps? | 18:21 |
| dprince | trown: perhaps the timing of that along with some other packaging change got us broken? | 18:21 |
| EmilienM | I think it broke before | 18:21 |
| EmilienM | it seems like yesterday | 18:21 |
| trown | dprince: ya, that was my first thought, but 1) tripleo-common is in our includepkgs so it should not get affected by promote and 2) tripleo-ci is doing the promote via periodic job, so how did periodic job pass | 18:22 |
| EmilienM | it does not seem related to promotion | 18:23 |
| EmilienM | trown: wait, does promotion run upgrade job right? | 18:23 |
| *** chem``` has joined #tripleo | 18:23 | |
| dprince | EmilienM: I think it runs all 3 (upgrade job included) | 18:24 |
| EmilienM | http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-upgrades/3040af8/console.html | 18:24 |
| EmilienM | failure | 18:24 |
| trown | EmilienM: dprince, but upgrade job does not vote | 18:24 |
| trown | also, I think https://github.com/openstack-infra/tripleo-ci/blob/master/scripts/tripleo.sh#L233 may have got us | 18:24 |
| EmilienM | were are the promotion logs ? | 18:24 |
| trown | tripleo-common in there and not openstack-tripleo-common | 18:24 |
| dprince | trown: so this could be an issue | 18:24 |
| EmilienM | where* | 18:25 |
| EmilienM | I want to check if upgrade job passed the promotion | 18:25 |
| *** chem`` has quit IRC | 18:25 | |
| trown | ya only ha and nonha are checked for promote https://github.com/openstack-infra/tripleo-ci/blob/master/scripts/mirror-server/mirror-server.pp#L51 | 18:25 |
| trown | not sure why we did that | 18:25 |
| EmilienM | damn | 18:26 |
| openstackgerrit | Lars Kellogg-Stedman proposed openstack/tripleo-quickstart: make --requirements cumulative https://review.openstack.org/330086 | 18:26 |
| EmilienM | we need to fix that | 18:26 |
| *** cwolferh has joined #tripleo | 18:26 | |
| trown | I dont know how to find logs from the "real" periodic job, but I have not seen upgrades passing on the fake one | 18:26 |
| EmilienM | trown: the promotion upgrade job failed for the same reason | 18:27 |
| EmilienM | http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-upgrades/3040af8/console.html#_2016-06-15_11_33_36_896 | 18:27 |
| EmilienM | ControllerPostPuppetRestartDeployment error | 18:27 |
| EmilienM | hopefully my patch https://review.openstack.org/#/c/330069/ can help to debug | 18:27 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: promote: add upgrade job part of voting https://review.openstack.org/330139 | 18:29 |
| EmilienM | trown: is it good? ^ | 18:29 |
| trown | EmilienM: I think we need to understand why it was left out originally... I am doubtful it was simply an oversight | 18:30 |
| EmilienM | well, what I understand now is that we allowed a promotion that was not passing our CI jobs | 18:30 |
| trown | EmilienM: dprince, wdyt of merging https://review.openstack.org/#/c/329961/ without upgrades job, since upgrades job is known broken, and that at least fixes the other two | 18:30 |
| EmilienM | trown: we landed it | 18:30 |
| trown | oh. great :) | 18:31 |
| EmilienM | yeah | 18:31 |
| EmilienM | this one was safe to land | 18:31 |
| *** jpich has quit IRC | 18:32 | |
| *** sambetts is now known as sambetts|afk | 18:32 | |
| *** hjensas__ has quit IRC | 18:33 | |
| openstackgerrit | Harry Rybacki proposed openstack/tripleo-quickstart: [WIP] Add full-deploy-with-scale script https://review.openstack.org/330146 | 18:34 |
| *** egafford has quit IRC | 18:35 | |
| openstackgerrit | John Trowbridge proposed openstack-infra/tripleo-ci: Update current repo setup includepkgs https://review.openstack.org/330148 | 18:37 |
| trown | EmilienM: I think ^ fixes the bit that broke the undercloud on promote | 18:38 |
| *** akshai has quit IRC | 18:39 | |
| EmilienM | trown: nice | 18:39 |
| EmilienM | trown: +2 | 18:39 |
| *** akshai has joined #tripleo | 18:39 | |
| EmilienM | trown: maybe related to upgrade job failure? (not sure) | 18:40 |
| openstackgerrit | Harry Rybacki proposed openstack/tripleo-quickstart: [WIP] Add full-deploy-with-scale script https://review.openstack.org/330146 | 18:41 |
| trown | EmilienM: not sure, probably not | 18:41 |
| *** jaosorior has joined #tripleo | 18:44 | |
| jaosorior | EmilienM: Still around? | 18:47 |
| EmilienM | yes | 18:47 |
| jaosorior | how's the upgrades job debugging going? | 18:48 |
| jaosorior | any news about that? | 18:48 |
| EmilienM | jaosorior: not much | 18:48 |
| EmilienM | jaosorior: we figured that promotion job didn't run upgrade (it will in future) | 18:48 |
| EmilienM | we also merged the tripleo-common package thing | 18:48 |
| openstackgerrit | Harry Rybacki proposed openstack/tripleo-quickstart: [WIP] Add full-deploy-with-scale script https://review.openstack.org/330146 | 18:48 |
| jaosorior | EmilienM: yeah, +2ed that | 18:48 |
| EmilienM | but really nothing else | 18:48 |
| jaosorior | the promotion | 18:48 |
| EmilienM | jaosorior: waiting on https://review.openstack.org/#/c/330069/ | 18:48 |
| jaosorior | got a patch with logs of the upgrades failure? | 18:48 |
| EmilienM | so we can have more debug on where it fails | 18:49 |
| jaosorior | ok | 18:49 |
| EmilienM | jaosorior: it's a ControllerPostPuppetRestartDeployment error | 18:49 |
| EmilienM | the bash script that restart resurces fail | 18:49 |
| jaosorior | yeah, that's up to where I figured out | 18:49 |
| dprince | trown: do we think the non-ha and ha jobs will pass with just 330148? | 18:49 |
| dprince | trown: just wondering if if we should consider going ahead and sending it? | 18:50 |
| *** apetrich has quit IRC | 18:50 | |
| *** panda has quit IRC | 18:50 | |
| *** panda has joined #tripleo | 18:50 | |
| jaosorior | EmilienM: got some logs from a run that had failed with it | 18:50 |
| jaosorior | I'm not sure if it's a red herring | 18:51 |
| trown | dprince: kind of depends if something we landed in tripleo-common in the last 3 days is broken | 18:51 |
| jaosorior | but there's a bunch of "client unexpectedly closed TCP connection" in the end of the puppet logs | 18:51 |
| trown | dprince: I dont think that patch should be a requirement for ha and non-ha to pass though | 18:51 |
| trown | dprince: and I do not have much hope that patch will fix the upgrades job | 18:52 |
| *** apetrich has joined #tripleo | 18:52 | |
| jaosorior | other than that I haven't noticed much :/ | 18:53 |
| dprince | trown: the recent ha and non-ha jobs I'm looking at all fail with Execution of '/bin/yum -d 0 -e 0 -y list tripleo-common' returned 1: Error: No matching Packages to list | 18:58 |
| EmilienM | dprince: yeah we fixed it | 18:58 |
| trown | dprince: that should be fixed by instack-undercloud patch | 18:58 |
| jaosorior | dprince: that was fixed already with a commit from EmilienM | 18:58 |
| EmilienM | https://review.openstack.org/329961 | 18:59 |
| *** akrivoka has quit IRC | 18:59 | |
| dprince | I saw that, just got confused. | 18:59 |
| dprince | okay, so ha and non-ha should be fine then... | 18:59 |
| EmilienM | yes only upgrade is failing | 19:00 |
| jaosorior | yeah | 19:00 |
| jaosorior | EmilienM: Seen this? http://paste.openstack.org/show/516336/ | 19:04 |
| EmilienM | yes | 19:04 |
| jaosorior | aw | 19:04 |
| jaosorior | damn | 19:04 |
| jaosorior | alright | 19:04 |
| EmilienM | but thanks | 19:05 |
| EmilienM | it's really the problem | 19:05 |
| EmilienM | cluster dies during upgrade | 19:05 |
| *** mbound has joined #tripleo | 19:06 | |
| dprince | jaosorior, EmilienM that error message comes from our pacemaker_common_functions.sh | 19:06 |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-quickstart: Create playbook for running ansible tempest role https://review.openstack.org/330164 | 19:06 |
| dprince | so it could just be we are hitting that timeout now | 19:06 |
| EmilienM | dprince: see https://review.openstack.org/#/c/330069/ | 19:07 |
| EmilienM | I'm trying to debug it | 19:07 |
| EmilienM | we thought it was rabbitmq | 19:07 |
| jaosorior | dprince: EmilienM is waaay ahead of us O_O | 19:07 |
| EmilienM | why would we get a timeout? | 19:07 |
| EmilienM | jaosorior: I spent my day on it | 19:07 |
| jaosorior | I still thought it was rabbitmq | 19:08 |
| dprince | EmilienM: perhaps because something (anything) is taking longer... | 19:08 |
| jaosorior | it's very weirdly closing connections (from the logs) | 19:08 |
| EmilienM | we hit this timeout 100% of time | 19:08 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/tripleo-common: Updated from global requirements https://review.openstack.org/323090 | 19:09 |
| *** egafford has joined #tripleo | 19:09 | |
| dprince | EmilienM: the resource is openstack-core | 19:10 |
| dprince | so that could be a lot of things right? | 19:10 |
| *** mbound has quit IRC | 19:11 | |
| EmilienM | dprince: yeah I checked in logs, afict it was only rabbitmq that crashed | 19:12 |
| openstackgerrit | Lars Kellogg-Stedman proposed openstack/tripleo-quickstart: return global control of force_cached_image https://review.openstack.org/330166 | 19:19 |
| *** dprince has quit IRC | 19:21 | |
| *** apetrich has quit IRC | 19:22 | |
| *** karthiks has quit IRC | 19:24 | |
| *** skramaja has quit IRC | 19:24 | |
| *** apetrich has joined #tripleo | 19:24 | |
| trown | larsks: do you have strong feelings about bumping the default stopping point of quickstart to post undercloud install? ie just after running `openstack undercloud install` | 19:30 |
| larsks | trown: that seems reasonable to me. | 19:32 |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Move hook generation in to python https://review.openstack.org/271139 | 19:32 |
| larsks | trown: but not the post-install? | 19:32 |
| bandini | marios, jistr: I think I know why heat decides to ignore Step2 in the major upgrade. The yum upgrade -y -q to mitaka at the end of Step 1, breaks the process somehow. Not sure why yet, but if I comment the yum update, Step2 takes place | 19:32 |
| *** openstackgerrit has quit IRC | 19:33 | |
| trown | larsks: k, it increases the time of our "quick" gates a bit, but I think the user experience is a bit better | 19:33 |
| *** openstackgerrit has joined #tripleo | 19:33 | |
| larsks | trown: right, but i was asking, should we also run the post-install (e.g., stop after the tripleo/undercloud role is complete)? | 19:33 |
| *** akshai_ has joined #tripleo | 19:33 | |
| larsks | In particular, that makes sure your network is set up correctly. | 19:34 |
| trown | larsks: ya, there are quite a few things people can do after `openstack undercloud install` but before running deploy... though I guess that is what skip tags are for | 19:35 |
| *** akshai has quit IRC | 19:35 | |
| trown | fwiw, shardy would like to stop just before someone would run `openstack overcloud deploy` https://bugs.launchpad.net/tripleo-quickstart/+bug/1569477 | 19:35 |
| openstack | Launchpad bug 1569477 in tripleo-quickstart 0.1 "Undercloud install should be automated by default" [High,Confirmed] | 19:35 |
| *** bfournie1 has joined #tripleo | 19:35 | |
| trown | I guess if we are changing it we could go for the full change... | 19:36 |
| *** karthiks has joined #tripleo | 19:36 | |
| *** bfournie has quit IRC | 19:36 | |
| *** skramaja has joined #tripleo | 19:36 | |
| *** egafford1 has joined #tripleo | 19:39 | |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-heat-templates: Composable Mistral services https://review.openstack.org/323436 | 19:40 |
| openstackgerrit | Brad P. Crochet proposed openstack/puppet-tripleo: Add Mistral profiles https://review.openstack.org/323431 | 19:40 |
| trown | larsks: now that we have the ability to run arbitrary playbooks with quickstart.sh, I am less convinced that tags are even worth the effort | 19:41 |
| trown | could just have different playbooks for different flows | 19:41 |
| *** egafford has quit IRC | 19:41 | |
| larsks | trown: it may be worthwhile to maintain some sort of big switches (e.g., "do not install undercloud", "do not deploy overcloud", "do not validate") maybe. | 19:42 |
| trown | the '*-scripts' tags are still nice | 19:42 |
| larsks | Or at least some way to control that via the quickstart.sh script. Maybe we just include multiple playbooks or something... | 19:43 |
| *** dsariel has joined #tripleo | 19:43 | |
| *** egafford1 is now known as egafford | 19:44 | |
| *** cllewellyn_ has joined #tripleo | 19:54 | |
| *** cllewellyn__ has joined #tripleo | 19:54 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: Enable libvirt as a micro-service https://review.openstack.org/329718 | 19:55 |
| *** jprovazn has quit IRC | 19:56 | |
| *** jaosorior has quit IRC | 19:57 | |
| openstackgerrit | John Trowbridge proposed openstack/tripleo-quickstart: Move default stopping point to just before overcloud deploy https://review.openstack.org/330176 | 19:57 |
| trown | panda: ^ we should rebase your stuff on that I think | 19:57 |
| trown | panda: specifically the ironic config for qemu://session | 19:58 |
| *** jcoufal_ has joined #tripleo | 19:59 | |
| panda | trown: before or after it's merged ? | 19:59 |
| *** krotscheck_dcm is now known as krotscheck | 20:00 | |
| trown | panda: suppose it doesn't matter | 20:00 |
| openstackgerrit | Harry Rybacki proposed openstack/tripleo-quickstart: [WIP] Add scale to roles gate https://review.openstack.org/329542 | 20:00 |
| *** jcoufal has quit IRC | 20:02 | |
| EmilienM | trown, bnemec: you want me to update commit message? or can we land it like it? | 20:02 |
| bnemec | EmilienM: If it passed CI, just fix the commit message and then land it. | 20:03 |
| trown | EmilienM: I think we should just update commit just before merging | 20:03 |
| bnemec | No need to wait for another CI run on a commit message change. | 20:03 |
| EmilienM | k | 20:03 |
| trown | yep | 20:03 |
| EmilienM | I don't think CI test this code (or does it?) | 20:04 |
| *** cllewellyn_ has quit IRC | 20:04 | |
| *** cllewellyn__ has quit IRC | 20:04 | |
| bnemec | Probably not. | 20:04 |
| bnemec | In fact, you might want to check with derek that once it's merged it gets applied to the actual CI env. | 20:04 |
| EmilienM | yep | 20:04 |
| trown | ya, I think there is no automation to do the puppet apply | 20:05 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: promote: add upgrade job part of voting https://review.openstack.org/330139 | 20:05 |
| EmilienM | killing useless job | 20:05 |
| EmilienM | sending an email to derek | 20:05 |
| EmilienM | feel free to +2 again | 20:05 |
| EmilienM | I'll let him +A | 20:05 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: promote: add upgrade job part of voting https://review.openstack.org/330139 | 20:05 |
| trown | oh whoops | 20:05 |
| trown | merged :) | 20:06 |
| *** toure has joined #tripleo | 20:06 | |
| EmilienM | lol | 20:06 |
| EmilienM | thanks trown ! | 20:06 |
| EmilienM | so fast | 20:06 |
| *** fzdarsky|afk has quit IRC | 20:07 | |
| *** dprince has joined #tripleo | 20:07 | |
| *** karts has joined #tripleo | 20:07 | |
| *** krsacme has joined #tripleo | 20:07 | |
| EmilienM | trown: no worries, I emailed him and he'll figure | 20:07 |
| trown | cool | 20:08 |
| *** karthiks has quit IRC | 20:11 | |
| *** skramaja has quit IRC | 20:11 | |
| *** toure is now known as toure|biab | 20:19 | |
| *** MaxPC has quit IRC | 20:27 | |
| EmilienM | trown: ok my patch to debug finished CI | 20:31 |
| EmilienM | I'm currently digging into http://logs.openstack.org/69/330069/2/check-tripleo/gate-tripleo-ci-centos-7-upgrades/aecb52e/logs/overcloud-controller-0/var/log/messages | 20:31 |
| EmilienM | it failed before trying to stop rabbit | 20:31 |
| EmilienM | see Jun 15 19:35:35 localhost systemd: Unit openstack-ceilometer-collector.service entered failed state. | 20:31 |
| EmilienM | dprince: ^ | 20:32 |
| EmilienM | it failed earlier than you said in the review | 20:32 |
| EmilienM | I don't see the "pacemaker is about to restart rabbit" | 20:32 |
| *** egafford has quit IRC | 20:32 | |
| EmilienM | I see nothing special in http://logs.openstack.org/69/330069/2/check-tripleo/gate-tripleo-ci-centos-7-upgrades/aecb52e/logs/overcloud-controller-0/var/log/ceilometer/collector.txt.gz | 20:33 |
| EmilienM | Jun 15 19:52:56 localhost pengine[11455]: warning: Processing failed op start for ip-fd00.fd00.fd00.3000..18 on overcloud-controller-0: unknown error (1) | 20:35 |
| EmilienM | we didn't have it on previous jobs ^ | 20:36 |
| EmilienM | we really need a pacemaker guru | 20:38 |
| EmilienM | let's file a bug | 20:39 |
| EmilienM | trown: do we have a bug alraedy for it ^ | 20:39 |
| *** julim has quit IRC | 20:39 | |
| trown | EmilienM: not that I am aware of | 20:39 |
| EmilienM | kk | 20:39 |
| EmilienM | trown: https://bugs.launchpad.net/tripleo/+bug/1592776 | 20:40 |
| EmilienM | it's not only HA job | 20:40 |
| openstack | Launchpad bug 1592776 in tripleo "Ha upgrade jobs failing with "cluster remained unstable for more than 1800 seconds"" [Undecided,New] | 20:40 |
| openstackgerrit | Gabriele Cerami proposed openstack/tripleo-quickstart: Update downloaded images to latest delorean repos https://review.openstack.org/327898 | 20:41 |
| openstackgerrit | Gabriele Cerami proposed openstack/tripleo-quickstart: Move ironic config to post install https://review.openstack.org/328300 | 20:41 |
| *** noslzzp has quit IRC | 20:50 | |
| dprince | EmilienM: ack, I was looking at a different patch I think | 21:03 |
| *** bfournie1 has quit IRC | 21:04 | |
| *** ooolpbot has joined #tripleo | 21:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1592776 | 21:10 |
| *** ooolpbot has quit IRC | 21:10 | |
| openstack | Launchpad bug 1592776 in tripleo "upgrade jobs failing with "cluster remained unstable for more than 1800 seconds"" [Critical,Confirmed] | 21:10 |
| *** trozet has quit IRC | 21:13 | |
| *** dprince has quit IRC | 21:13 | |
| *** trozet has joined #tripleo | 21:14 | |
| *** trown is now known as trown|outtypewww | 21:15 | |
| *** cmyster has quit IRC | 21:15 | |
| *** cmyster has joined #tripleo | 21:15 | |
| ayoung | EmilienM, jmiu and Ozz helped me figure out the problem from yesterday. I was updating the HA Controller template, but deploying non HA. | 21:15 |
| ayoung | Got it working now | 21:16 |
| EmilienM | cool | 21:16 |
| ayoung | EmilienM, I'm even more dangerous than I was before | 21:16 |
| ayoung | http://adam.younglogic.com/2016/06/custom-overcloud-deploys/ | 21:16 |
| ayoung | EmilienM, I need to go play Dad for a while, but tomorrow, lets confer about V3 Keystone everywhere... | 21:17 |
| *** rhallisey has quit IRC | 21:17 | |
| EmilienM | ayoung: enjoy :) | 21:18 |
| *** jayg is now known as jayg|g0n3 | 21:24 | |
| *** lblanchard has quit IRC | 21:28 | |
| *** ccamacho|out has quit IRC | 21:33 | |
| *** myoung is now known as myoung|afk | 21:35 | |
| *** cdearborn has quit IRC | 21:47 | |
| *** weshay has quit IRC | 22:00 | |
| *** openstackgerrit has quit IRC | 22:02 | |
| *** yamahata has quit IRC | 22:04 | |
| *** openstackgerrit has joined #tripleo | 22:05 | |
| *** ibravo2 has quit IRC | 22:08 | |
| *** paramite has quit IRC | 22:08 | |
| *** jcoufal_ has quit IRC | 22:09 | |
| *** ooolpbot has joined #tripleo | 22:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1592776 | 22:10 |
| *** ooolpbot has quit IRC | 22:10 | |
| openstack | Launchpad bug 1592776 in tripleo "upgrade jobs failing with "cluster remained unstable for more than 1800 seconds"" [Critical,Confirmed] | 22:10 |
| *** mbound has joined #tripleo | 22:10 | |
| *** egafford has joined #tripleo | 22:20 | |
| *** rlandy has quit IRC | 22:24 | |
| *** abehl has quit IRC | 22:27 | |
| *** yamahata has joined #tripleo | 22:27 | |
| openstackgerrit | Ben Nemec proposed openstack/tripleo-heat-templates: Enable firewall by default on the overcloud https://review.openstack.org/321833 | 22:27 |
| openstackgerrit | Ben Nemec proposed openstack/tripleo-heat-templates: Allow pcsd port in firewall https://review.openstack.org/330249 | 22:27 |
| *** myoung|afk has quit IRC | 22:29 | |
| *** jcoufal has joined #tripleo | 22:55 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!