| *** dhill_ has quit IRC | 00:04 | |
| *** dhill_ has joined #tripleo | 00:05 | |
| *** dsneddon has quit IRC | 00:06 | |
| *** ooolpbot has joined #tripleo | 00:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 00:10 |
|---|---|---|
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649742 | 00:10 |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged] | 00:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1650005 | 00:10 |
| *** ooolpbot has quit IRC | 00:10 | |
| openstack | Launchpad bug 1650005 in tripleo "Multinode jobs failing on libvirt issues" [Critical,Triaged] | 00:10 |
| *** morazi has quit IRC | 00:14 | |
| *** ipsecguy has quit IRC | 00:35 | |
| *** ipsecguy has joined #tripleo | 00:36 | |
| *** cdearborn has joined #tripleo | 00:37 | |
| *** limao has joined #tripleo | 00:54 | |
| *** fragatina has joined #tripleo | 00:55 | |
| *** ooolpbot has joined #tripleo | 01:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649742 | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1650005 | 01:10 |
| *** ooolpbot has quit IRC | 01:10 | |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged] | 01:10 |
| openstack | Launchpad bug 1650005 in tripleo "Multinode jobs failing on libvirt issues" [Critical,Triaged] | 01:10 |
| *** yamahata_ has joined #tripleo | 01:24 | |
| *** dr_gogeta86 has quit IRC | 01:35 | |
| *** bana_k has quit IRC | 01:38 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Default libvirt_cpu_mode to none when using qemu https://review.openstack.org/410359 | 01:44 |
| *** tzumainn has quit IRC | 01:45 | |
| *** pleia2 has left #tripleo | 01:59 | |
| *** jkilpatr has quit IRC | 02:08 | |
| *** ooolpbot has joined #tripleo | 02:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649742 | 02:10 |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged] | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1650005 | 02:10 |
| *** ooolpbot has quit IRC | 02:10 | |
| openstack | Launchpad bug 1650005 in tripleo "Multinode jobs failing on libvirt issues" [Critical,Triaged] | 02:10 |
| *** ansmith has joined #tripleo | 02:17 | |
| *** jeckersb_gone is now known as jeckersb | 02:20 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: Remove swapfile from undercloud https://review.openstack.org/410323 | 02:21 |
| *** cwolferh has quit IRC | 02:23 | |
| *** fzdarsky_ has joined #tripleo | 02:25 | |
| *** Goneri has quit IRC | 02:26 | |
| *** fzdarsky|afk has quit IRC | 02:29 | |
| *** ctayal has quit IRC | 02:32 | |
| *** jeckersb is now known as jeckersb_gone | 02:43 | |
| *** ansmith has quit IRC | 02:51 | |
| *** bkopilov has quit IRC | 03:07 | |
| *** ooolpbot has joined #tripleo | 03:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649742 | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1650005 | 03:10 |
| *** ooolpbot has quit IRC | 03:10 | |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged] | 03:10 |
| openstack | Launchpad bug 1650005 in tripleo "Multinode jobs failing on libvirt issues" [Critical,Triaged] | 03:10 |
| *** cwolferh has joined #tripleo | 03:14 | |
| *** egafford has quit IRC | 03:26 | |
| openstackgerrit | Lars Kellogg-Stedman proposed openstack/puppet-tripleo: add support for collectd https://review.openstack.org/411047 | 03:28 |
| openstackgerrit | Lars Kellogg-Stedman proposed openstack/tripleo-heat-templates: add collectd composable service https://review.openstack.org/411048 | 03:28 |
| *** cwolferh has quit IRC | 03:42 | |
| *** nyechiel has joined #tripleo | 03:47 | |
| *** ctayal has joined #tripleo | 04:04 | |
| *** links has joined #tripleo | 04:06 | |
| *** ccamacho has quit IRC | 04:09 | |
| *** ooolpbot has joined #tripleo | 04:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649742 | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1650005 | 04:10 |
| *** ooolpbot has quit IRC | 04:10 | |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged] | 04:10 |
| openstack | Launchpad bug 1650005 in tripleo "Multinode jobs failing on libvirt issues" [Critical,Triaged] | 04:10 |
| *** dmacpher is now known as dmacpher-afk | 04:12 | |
| *** bkopilov has joined #tripleo | 04:25 | |
| *** cwolferh has joined #tripleo | 04:29 | |
| openstackgerrit | Lars Kellogg-Stedman proposed openstack/tripleo-heat-templates: add collectd composable service https://review.openstack.org/411048 | 04:36 |
| *** saneax-_-|AFK is now known as saneax | 04:46 | |
| openstackgerrit | Jason E. Rist proposed openstack/tripleo-ui: Adds ids namespaced with `plans_` to artifacts on plans page https://review.openstack.org/411102 | 04:52 |
| openstackgerrit | Jason E. Rist proposed openstack/tripleo-ui: Adds an id for the Login button for test automation https://review.openstack.org/409378 | 04:53 |
| *** ctayal has quit IRC | 04:58 | |
| *** ooolpbot has joined #tripleo | 05:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649742 | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1650005 | 05:10 |
| *** ooolpbot has quit IRC | 05:10 | |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged] | 05:10 |
| openstack | Launchpad bug 1650005 in tripleo "Multinode jobs failing on libvirt issues" [Critical,Triaged] | 05:10 |
| *** saneax is now known as saneax-_-|AFK | 05:12 | |
| *** dhill_ has quit IRC | 05:14 | |
| *** saneax-_-|AFK is now known as saneax | 05:17 | |
| *** nyechiel has quit IRC | 05:24 | |
| *** ctayal has joined #tripleo | 05:27 | |
| *** dmacpher-afk is now known as dmacpher | 05:28 | |
| *** oshvartz has quit IRC | 05:39 | |
| *** ctayal_ has joined #tripleo | 05:53 | |
| *** fragatina has quit IRC | 05:55 | |
| *** ctayal has quit IRC | 05:57 | |
| *** masco has joined #tripleo | 05:57 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Deployed server: switch to apply-config hook https://review.openstack.org/396171 | 05:59 |
| *** udesale has joined #tripleo | 06:00 | |
| *** ooolpbot has joined #tripleo | 06:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649742 | 06:10 |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged] | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1650005 | 06:10 |
| *** ooolpbot has quit IRC | 06:10 | |
| openstack | Launchpad bug 1650005 in tripleo "Multinode jobs failing on libvirt issues" [Critical,Triaged] | 06:10 |
| *** abregman has joined #tripleo | 06:10 | |
| *** coolsvap has joined #tripleo | 06:10 | |
| *** jaosorior has joined #tripleo | 06:16 | |
| *** ctayal_ has quit IRC | 06:20 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Recreate initramfs within loopback image https://review.openstack.org/411130 | 06:22 |
| *** ealcaniz has joined #tripleo | 06:25 | |
| *** ctayal has joined #tripleo | 06:28 | |
| *** lmiccini has joined #tripleo | 06:29 | |
| *** ctayal_ has joined #tripleo | 06:39 | |
| *** ctayal has quit IRC | 06:41 | |
| *** rajinir has quit IRC | 06:46 | |
| *** ctayal_ has quit IRC | 06:48 | |
| *** pgadiya has joined #tripleo | 06:49 | |
| openstackgerrit | Jason E. Rist proposed openstack/tripleo-ui: Adds ids namespaced with `plans_` to artifacts on plans page https://review.openstack.org/411102 | 06:57 |
| *** yprokule has joined #tripleo | 06:59 | |
| *** iranzo has joined #tripleo | 07:00 | |
| *** oshvartz has joined #tripleo | 07:01 | |
| *** ctayal has joined #tripleo | 07:09 | |
| *** ooolpbot has joined #tripleo | 07:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649742 | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1650005 | 07:10 |
| *** ooolpbot has quit IRC | 07:10 | |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged] | 07:10 |
| openstack | Launchpad bug 1650005 in tripleo "Multinode jobs failing on libvirt issues" [Critical,Triaged] | 07:10 |
| *** rcernin has joined #tripleo | 07:13 | |
| *** ctayal has quit IRC | 07:16 | |
| *** ealcaniz has quit IRC | 07:16 | |
| *** yamahata_ has quit IRC | 07:19 | |
| *** ctayal has joined #tripleo | 07:21 | |
| *** ramishra has quit IRC | 07:24 | |
| *** ramishra has joined #tripleo | 07:26 | |
| *** jaosorior has quit IRC | 07:37 | |
| *** jaosorior has joined #tripleo | 07:37 | |
| *** milan has joined #tripleo | 07:44 | |
| openstackgerrit | Michele Baldessari proposed openstack/puppet-tripleo: Composable HA https://review.openstack.org/362389 | 07:45 |
| *** bana_k has joined #tripleo | 07:48 | |
| jaosorior | d0ugal: ping | 07:52 |
| *** jaosorior has quit IRC | 07:55 | |
| *** abehl has joined #tripleo | 07:56 | |
| *** Vijayendra has quit IRC | 07:57 | |
| *** abehl has quit IRC | 07:58 | |
| *** abehl has joined #tripleo | 07:58 | |
| *** gfidente has joined #tripleo | 07:58 | |
| *** gfidente has quit IRC | 07:58 | |
| *** gfidente has joined #tripleo | 07:58 | |
| *** bana_k has quit IRC | 08:02 | |
| *** anton has quit IRC | 08:05 | |
| bandini | morning | 08:08 |
| *** jlinkes has joined #tripleo | 08:09 | |
| *** ooolpbot has joined #tripleo | 08:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649742 | 08:10 |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged] | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1650005 | 08:10 |
| *** ooolpbot has quit IRC | 08:10 | |
| openstack | Launchpad bug 1650005 in tripleo "Multinode jobs failing on libvirt issues" [Critical,Triaged] | 08:10 |
| *** apetrich has quit IRC | 08:11 | |
| *** b00tcat has quit IRC | 08:12 | |
| *** b00tcat has joined #tripleo | 08:15 | |
| *** ccamacho has joined #tripleo | 08:15 | |
| *** jbadiapa has quit IRC | 08:16 | |
| *** shardy_afk is now known as shardy | 08:17 | |
| openstackgerrit | Merged openstack/tripleo-common: Add FreeIPA enrollment environment generator https://review.openstack.org/409165 | 08:18 |
| *** dciabrin has joined #tripleo | 08:18 | |
| *** aufi has joined #tripleo | 08:18 | |
| *** anton has joined #tripleo | 08:19 | |
| *** florianf has joined #tripleo | 08:20 | |
| *** ctayal has quit IRC | 08:25 | |
| openstackgerrit | Carlos Camacho proposed openstack/puppet-tripleo: Use transport_url for rabbitmq connection parameters in heat https://review.openstack.org/410333 | 08:28 |
| openstackgerrit | Flavio Percoco proposed openstack/tripleo-heat-templates: WIP: Containerized Glance https://review.openstack.org/400870 | 08:28 |
| openstackgerrit | Flavio Percoco proposed openstack/tripleo-heat-templates: WIP: Allow to disable glance registry in tripleo https://review.openstack.org/408471 | 08:28 |
| *** jaosorior has joined #tripleo | 08:30 | |
| *** chem has quit IRC | 08:32 | |
| *** fragatina has joined #tripleo | 08:33 | |
| *** mcornea has joined #tripleo | 08:35 | |
| *** leanderthal has joined #tripleo | 08:37 | |
| jaosorior | bandini: hey, could you check this out https://review.openstack.org/#/c/410722/ ? | 08:37 |
| *** pcaruana has joined #tripleo | 08:37 | |
| *** dsariel has quit IRC | 08:37 | |
| *** fragatina has quit IRC | 08:37 | |
| *** ohamada has joined #tripleo | 08:40 | |
| bandini | jaosorior: ack, yeah it was on my list for today | 08:41 |
| *** imranh has quit IRC | 08:42 | |
| *** jpena|off is now known as jpena | 08:43 | |
| *** imranh has joined #tripleo | 08:43 | |
| *** ealcaniz has joined #tripleo | 08:44 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Nothing to see here https://review.openstack.org/111011 | 08:48 |
| *** zoli|gone is now known as zoli|wfh | 08:48 | |
| *** amoralej|off is now known as amoralej | 08:51 | |
| *** milan has quit IRC | 08:53 | |
| *** jpich has joined #tripleo | 08:55 | |
| *** jpena is now known as jpena|off | 08:55 | |
| *** yprokule_ has joined #tripleo | 08:55 | |
| *** jpena|off is now known as jpena | 08:56 | |
| *** ohamada has quit IRC | 08:56 | |
| *** ohamada has joined #tripleo | 08:56 | |
| *** chem has joined #tripleo | 08:57 | |
| *** yprokule has quit IRC | 08:57 | |
| *** yprokule_ is now known as yprokule | 08:57 | |
| *** chem has quit IRC | 08:59 | |
| *** chem has joined #tripleo | 08:59 | |
| *** panda|zZ is now known as panda | 09:01 | |
| openstackgerrit | Merged openstack/tripleo-ui: Add dependencies for UI internationalisation https://review.openstack.org/401205 | 09:02 |
| jaosorior | shardy: are you around? | 09:02 |
| *** athomas has joined #tripleo | 09:04 | |
| openstackgerrit | Merged openstack/tripleo-ui: Adds basic internationalization support https://review.openstack.org/399626 | 09:05 |
| *** ealcaniz has quit IRC | 09:06 | |
| *** kodokuu has joined #tripleo | 09:07 | |
| skramaja | shardy: did you get a chance to look at the proposal - http://lists.openstack.org/pipermail/openstack-dev/2016-December/108819.html | 09:07 |
| *** ooolpbot has joined #tripleo | 09:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649742 | 09:10 |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged] | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1650005 | 09:10 |
| *** ooolpbot has quit IRC | 09:10 | |
| openstack | Launchpad bug 1650005 in tripleo "Multinode jobs failing on libvirt issues" [Critical,Triaged] | 09:10 |
| *** tremble has joined #tripleo | 09:12 | |
| *** tremble has quit IRC | 09:12 | |
| *** tremble has joined #tripleo | 09:12 | |
| *** flepied has quit IRC | 09:16 | |
| shardy | jaosorior: Hi, yup I'm around | 09:16 |
| shardy | skramaja: sorry, I missed that - why do we need a different method to make these changes instead of puppet? | 09:18 |
| shardy | I'm not in favor of introducing a new tool which also modifies the system config - we've already got puppet managing some kernel configuration in t-h-t | 09:18 |
| shardy | the issue we can't really solve without help from ironic is how to avoid a reboot | 09:19 |
| shardy | skramaja: is it possible to configure the DPDK interfaces with os-net-config, but not bring them up until the reboot happens? | 09:19 |
| skramaja | kernel arg changes for dpdk is mandatory before os-net-config binds it.. so the current way of doing is setting the kernel args in first-boot.yaml.. and rebooting.. | 09:19 |
| skramaja | shardy: os-net-config does a ifup - so basically we should avoid ifup for DPDK | 09:20 |
| shardy | skramaja: Yeah, so I guess we have two options - modify os-net-config to allow configuring but not ifup-ing the interfaces | 09:21 |
| shardy | or add a configuration step before os-net-config runs | 09:21 |
| shardy | we don't need a new tool to do the second though, it's already possible via t-h-t | 09:21 |
| shardy | skramaja: did you see that we now run os-net-config via a script? | 09:21 |
| skramaja | shardy: you meant the 20-os-net-config script? | 09:22 |
| shardy | skramaja: no, we don't use that anymore, we run it via a script in t-h-t | 09:22 |
| shardy | which makes it much easier to modify or run an alternative script | 09:22 |
| shardy | not sure if that helps tho, as you'd have to reboot during the script | 09:23 |
| shardy | skramaja: to clarify, what is the objection to the current firstboot script approach? | 09:23 |
| *** apetrich has joined #tripleo | 09:23 | |
| shardy | that seems to be a good time in the deployment cycle to do the reboot, e.g as early as possible | 09:23 |
| d0ugal | Where are we with CI today? is there anything I can do to help? | 09:23 |
| shardy | so it doesn't seem so bad, given that the ironic ramdisk won't do this for us atm | 09:23 |
| skramaja | shardy: there are other scenarios like "tune-d" which also needs reboot.. | 09:24 |
| owalsh | skramaja: can't we configure tuned in the firstboot script? | 09:24 |
| skramaja | shardy: if we put everything on first-boot, which we are doing now as stopgap, | 09:24 |
| shardy | skramaja: Ok, but what's the reason that's viewed as a stopgap? | 09:25 |
| *** ealcaniz has joined #tripleo | 09:25 | |
| shardy | is it because there's no error path when the script doesn't work? | 09:25 |
| shardy | owalsh: yeah exactly, that would work OK I think | 09:25 |
| skramaja | shardy: yes and no clear steps sepration.. | 09:26 |
| skramaja | its like dumping everything into firstboot.. | 09:26 |
| shardy | skramaja: well, it's not "everything", it's some specific actions which need to happen, uh, on firstboot | 09:26 |
| owalsh | skramaja: I agree it feels a bit wrong... | 09:27 |
| skramaja | shardy: :) | 09:27 |
| owalsh | skramaja: I'm looking at dropping a yaml file and running a python script | 09:27 |
| owalsh | skramaja: ... sometihng I can unit test etc... | 09:28 |
| shardy | Like, I get that it's not *ideal* - the perfect solution would be for the deploy ramdisk to configure everything before the image even boots | 09:28 |
| shardy | but we know the ironic folks don't want that | 09:28 |
| shardy | so we're stuck with doing something early in the first boot, then rebooting | 09:28 |
| shardy | cloud-init is a not-bad way to do that IMO | 09:28 |
| shardy | we could reinvent another way to do the same config slightly later, but I'm just not clear why | 09:28 |
| *** dtantsur|afk is now known as dtantsur | 09:29 | |
| owalsh | number of people telling me the reboot is very costly | 09:29 |
| skramaja | ok shardy | 09:29 |
| shardy | owalsh: Yeah, but there's nothing we can do about that unless we convice the ironice community to make these changes during deployment of the image | 09:29 |
| shardy | or, perhaps provide some hooks in the ipa ramdisk that allow it | 09:30 |
| shardy | or, we could run a patched version of the ipa ramdisk | 09:30 |
| shardy | none of the reboot problems are solved by introducing a new agent that runs the same script we run via cloud-init tho | 09:30 |
| shardy | skramaja: there are some other ways to move the config out of cloud-init | 09:31 |
| * skramaja listening | 09:31 | |
| shardy | if you want to run this via t-h-t, we could have a conditional deployment, which reboots only when some previous deployment did something | 09:31 |
| shardy | using heat conditionals | 09:31 |
| shardy | or a provider template that defaults to OS::Heat::None | 09:31 |
| shardy | skramaja: let me write a quick prototype, give me a few mins | 09:32 |
| *** jbadiapa has joined #tripleo | 09:32 | |
| skramaja | thanks shardy | 09:32 |
| owalsh | ditto | 09:32 |
| skramaja | owalsh: what do you mean by - "I'm looking at dropping a yaml file and running a python script", not clear.. | 09:33 |
| *** akrivoka has joined #tripleo | 09:33 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci: Postci tests trial and error https://review.openstack.org/411189 | 09:34 |
| owalsh | skramaja: 3rd example here http://docs.openstack.org/developer/heat/template_guide/software_deployment.html#software-config-resources | 09:35 |
| owalsh | skramaja: set yaml in boot_config file, then run a python script that read the yaml and applies config | 09:36 |
| owalsh | skramaja: better than templating params in scripts IMO :-) | 09:37 |
| skramaja | got it owalsh.. | 09:37 |
| *** shinobu__ has joined #tripleo | 09:38 | |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Remove dead mocks and dead code https://review.openstack.org/409130 | 09:40 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Remove the keystone_pki cert generation from tripleoclient https://review.openstack.org/409198 | 09:40 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Use the overcloudrc generated in a Mistral action https://review.openstack.org/398975 | 09:40 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Remove the unused get_config_value function https://review.openstack.org/409255 | 09:40 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Simplify Fetching the passwords from Mistral https://review.openstack.org/409791 | 09:40 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: [WIP] Respond to multiple messages from workflow executions https://review.openstack.org/406178 | 09:40 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Remove the unused delete image function https://review.openstack.org/409258 | 09:40 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Remove the unused constant RESOURCE_REGISTRY_NAME https://review.openstack.org/409256 | 09:40 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Remove two unused Exception subclasses https://review.openstack.org/409257 | 09:40 |
| *** fzdarsky_ is now known as fzdarsky | 09:42 | |
| *** kodokuu has quit IRC | 09:43 | |
| *** kodokuu has joined #tripleo | 09:43 | |
| *** lucas-afk is now known as lucasagomes | 09:49 | |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common: Remove the unused constant OBJECT_META_KEY_PREFIX https://review.openstack.org/411198 | 09:52 |
| *** dsariel has joined #tripleo | 09:52 | |
| *** flepied has joined #tripleo | 09:55 | |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common: Remove unused exception ContainerAlreadyExistsError https://review.openstack.org/411200 | 09:56 |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common: Remove unused exception HeatValidationFailedError https://review.openstack.org/411201 | 09:56 |
| * d0ugal looks for other stuff to delete | 09:56 | |
| *** milan has joined #tripleo | 09:58 | |
| *** paramite has joined #tripleo | 10:03 | |
| *** yprokule_ has joined #tripleo | 10:03 | |
| skramaja | shardy: another case with first-boot maintanence is, if there is a cluster with DPDK and SR-IOV or other feature, which needs different kernel args per role, it will be hard to maintain in first-boot script.. | 10:04 |
| *** yprokule has quit IRC | 10:04 | |
| *** yprokule_ is now known as yprokule | 10:04 | |
| skramaja | shardy: as of now we are comparing host name and taking decision in the first-boot script. | 10:05 |
| *** ooolpbot has joined #tripleo | 10:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649742 | 10:10 |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged] | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1650005 | 10:10 |
| *** ooolpbot has quit IRC | 10:10 | |
| openstack | Launchpad bug 1650005 in tripleo "Multinode jobs failing on libvirt issues" [Critical,Triaged] | 10:10 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Add pre-network hook and example showing config-then-reboot https://review.openstack.org/411204 | 10:11 |
| shardy | skramaja: ack, see if ^^ would suit you better | 10:12 |
| shardy | it enables per-role pre-network config, including an example that does a reboot | 10:12 |
| shardy | not yet tested, feedback welcome | 10:12 |
| skramaja | looking at it shardy.. | 10:12 |
| skramaja | owalsh: ^ | 10:13 |
| *** hewbrocca_afk is now known as hewbrocca | 10:13 | |
| openstackgerrit | Gabriele Cerami proposed openstack/tripleo-quickstart-extras: Add overcloud delete role https://review.openstack.org/402228 | 10:17 |
| *** openstackgerrit has quit IRC | 10:18 | |
| * owalsh looking | 10:19 | |
| Ng | what's the rough workflow for adding a new action in tripleo-common and having it show up in mistral? Do I have to rebuild or reboot the undercloud, or can I poke things so it notices the new stuff? | 10:19 |
| shardy | Ng: you have to install a new tripleo-common with the action, then run mistral-db-manage populate | 10:20 |
| Ng | shardy: aha, that is the magic command I hadn't spotted. thanks! | 10:20 |
| shardy | Ng: that populate normally happens via the "openstack undercloud install" | 10:20 |
| shardy | so you can re-run that if you like but it takes a while | 10:20 |
| Ng | shardy: I'm using quickstart, so my undercloud image comes pre-built | 10:21 |
| Ng | (I think it can build images, but I've never looked at that part) | 10:21 |
| d0ugal | Ng: I don't follow this exactly, but this is one way to do everything needed. https://github.com/openstack/tripleo-common#action-development | 10:21 |
| shardy | Ng: you can still run openstack undercloud install | 10:21 |
| d0ugal | We should probably update that just to show the needed commands, rather than the opinionated install method | 10:21 |
| shardy | it's designed to be idempotent, so you can run it as many times as you like (in theory) | 10:21 |
| Ng | interesting | 10:21 |
| Ng | thanks :) | 10:21 |
| *** nyechiel has joined #tripleo | 10:23 | |
| therve | d0ugal, Isn't it too early for spring cleanup? :) | 10:24 |
| d0ugal | therve: hah, I think this is long overdue from spring this year ;) | 10:24 |
| therve | Heh | 10:24 |
| d0ugal | therve: also I recently discovered https://pypi.python.org/pypi/vulture | 10:25 |
| d0ugal | It gives lots of false positives, but gives a good starting point. | 10:26 |
| therve | That's pretty cool, thanks for the pointer | 10:26 |
| *** openstackgerrit has joined #tripleo | 10:28 | |
| openstackgerrit | Andrey Shestakov proposed openstack/diskimage-builder: Fix dhcp-all-interfaces for ubuntu-minimal xenial https://review.openstack.org/407725 | 10:28 |
| apetrich | marios, about your comment here https://review.openstack.org/#/c/407972/5/tripleo_common/constants.py I was thinking about using CONTAINER_NAME_PATTERN do you think it is clear enough? | 10:28 |
| marios | apetrich: sure don't mean to bikeshed on that just something more explicit that sounds fine jpich ^^ | 10:29 |
| d0ugal | apetrich: I'd go with "plan" rather than "container" | 10:29 |
| d0ugal | I know we use container in various places when we mean plan, but really we should move away from that IMO | 10:29 |
| apetrich | d0ugal, aye | 10:29 |
| jpich | I'd prefer to use plan as well | 10:30 |
| *** nmathew has joined #tripleo | 10:31 | |
| *** fultonj has quit IRC | 10:31 | |
| shardy | Ng: Oh, I just remembered another useful thing when working with actions is entry_point_inspector | 10:31 |
| Ng | oh? | 10:31 |
| shardy | the centos package appears broken due to deps re pyparsing, but if you pip install you can do | 10:31 |
| *** fultonj has joined #tripleo | 10:32 | |
| shardy | epi group show mistral.actions | grep tripleo | 10:32 |
| shardy | I found it useful to ensure my actions were updated to what I expected before testing | 10:32 |
| d0ugal | shardy: neat, I didn't know about that one. | 10:32 |
| Ng | nice | 10:32 |
| shardy | https://github.com/openstack/instack-undercloud/blob/master/elements/undercloud-install/os-refresh-config/post-configure.d/98-undercloud-setup#L101 | 10:33 |
| shardy | And if you're hacking on workflows you might want to cut/paste from that script | 10:33 |
| shardy | I had a little script which refreshed the actions and workflows which made life a little easier | 10:33 |
| therve | shardy, Is it better than the mistral list actions CLI? | 10:34 |
| shardy | therve: well it's useful to figure out if a missing action wasn't correctly installed, or if it silently failed during the db populate | 10:34 |
| d0ugal | therve: epi? I guess it is just quicker - you can verify before you run the populate command and restart mistral. | 10:35 |
| shardy | therve: I experienced both problems at various times | 10:35 |
| d0ugal | I keep meaning to improve the populate command, it should handle the faulures but it could give us more useful feedback. | 10:35 |
| apetrich | PLAN_NAME_PATTERN gets the line just under 79 chars | 10:35 |
| therve | shardy, Makes sense, thanks | 10:35 |
| shardy | d0ugal: when I was testing it just failed but continued | 10:35 |
| shardy | maybe that's fixed now, I think we discussed it before :) | 10:36 |
| jaosorior | shardy: hey, so for the TLS everywhere work, certmonger requires the nodes' hostnames and services principals in order to request the appropriate certificates, this is already happening and it's good. However, these entries need to exist in the CA beforehand; what I was doing is to pre-create all the entries with a script, and once having that (and an OTP that the CA gives you), trigger the overcloud | 10:36 |
| jaosorior | deployment with the OTP, so stuff would work... The issue is that pre-creating these entries is very tedious as deployments can get quite big and we need an entry per subjectAltName that will be used in the certs (so we also need to populate the per-network hostnames with their respective service principals). Now, for another use-case (having identities per nove host) aside from this one we came up with a | 10:36 |
| jaosorior | solution based on Nova's vendordata plugin structure. This "plugin" is a service that basically detects when a server is created, gets the host name, and creates the entry in the CA, and removes this entry when the server is deleted. Now, for the TripleO case, we do need the subjectAltNames, which we could pass through nova metadata. Basically we could pass the relevant service name with the name of the | 10:36 |
| jaosorior | network it's listening on, and the vendordata service could create the relevant entry from this data. Would you be fine if I introduce a hook for the nova metadata in the TripleO servers that reads an entry from the service profiles, and from that generates said list? | 10:36 |
| *** derekh has joined #tripleo | 10:36 | |
| d0ugal | shardy: I do sort-of remember this. Maybe I even opened a bug. I shall check. | 10:36 |
| shardy | pastebomb ;) | 10:36 |
| d0ugal | woah | 10:36 |
| jaosorior | wrote it on the spot, no pasting :P | 10:36 |
| *** tobias_fiberdata has quit IRC | 10:36 | |
| jaosorior | weechat just separated it | 10:36 |
| jaosorior | sorry for the long text though | 10:37 |
| d0ugal | jaosorior: IIRC, 512 is the maximum char limit in IRC :/ | 10:37 |
| *** limao has quit IRC | 10:37 | |
| shardy | jaosorior: so, to clarify, the question is can we add a list of things to the nova server metadata map, for consumption by the Nova vendordata plugin? | 10:39 |
| shardy | https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/controller-role.yaml#L160 | 10:40 |
| shardy | so we'd merge some additional data in with ServerMetadata here? | 10:40 |
| jaosorior | yeah | 10:40 |
| shardy | Sounds OK to me | 10:40 |
| jaosorior | yeahgreat | 10:41 |
| *** b00tcat has quit IRC | 10:41 | |
| jaosorior | I' | 10:41 |
| openstackgerrit | Adriano Petrich proposed openstack/tripleo-common: Validate the plan name before running https://review.openstack.org/407972 | 10:41 |
| jaosorior | I'll come up with a POC in a bit to show it better | 10:41 |
| skramaja | thanks shardy for the prototye.. i will start working on it to incorporate kernel args and tuned changes.. will poke you, if hit with any hurdels.. | 10:41 |
| *** bvandenh has joined #tripleo | 10:41 | |
| *** b00tcat has joined #tripleo | 10:43 | |
| *** tobias_fiberdata has joined #tripleo | 10:50 | |
| *** tosky has joined #tripleo | 10:52 | |
| *** jlinkes has quit IRC | 10:53 | |
| *** chem has quit IRC | 10:54 | |
| *** chem has joined #tripleo | 10:54 | |
| *** Vijayendra has joined #tripleo | 10:55 | |
| * owalsh doing the same, thanks shardy | 10:55 | |
| openstackgerrit | Honza Pokorny proposed openstack/tripleo-ui: Remove SharedWorker in favor of cookies https://review.openstack.org/408184 | 11:05 |
| *** jlinkes has joined #tripleo | 11:08 | |
| *** ooolpbot has joined #tripleo | 11:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649742 | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1650005 | 11:10 |
| *** ooolpbot has quit IRC | 11:10 | |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged] | 11:10 |
| openstack | Launchpad bug 1650005 in tripleo "Multinode jobs failing on libvirt issues" [Critical,Triaged] | 11:10 |
| *** ealcaniz has quit IRC | 11:11 | |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Add pre-network hook and example showing config-then-reboot https://review.openstack.org/411204 | 11:12 |
| shardy | owalsh, skramaja: np - let me know if it works OK - added the stop of occ in ^^ | 11:13 |
| b00tcat | EmilienM: I remember you commented a while ago on my patches that we needed the puppet packages in RDO, and I'm not that :-) but out of curiosity, what if we hosted them in the MidoNet repos? would that also be an option? | 11:19 |
| *** ealcaniz has joined #tripleo | 11:20 | |
| *** cdearborn has quit IRC | 11:20 | |
| openstackgerrit | Dan Prince proposed openstack-infra/tripleo-ci: Disable the ceilo swift pipeline for multinode https://review.openstack.org/409483 | 11:31 |
| shardy | b00tcat: I think the main reason they would be better in RDO is we then get automatic delorean builds, which can then be consumed in CI | 11:32 |
| *** dtantsur is now known as dtantsur|brb | 11:32 | |
| shardy | b00tcat: if there's any future plan to get midonet integration tested in CI (would be a good idea?), I expect that will be useful | 11:32 |
| *** panda is now known as panda|lunch | 11:33 | |
| b00tcat | shardy: right, thanks :-) I'm atm trying to get stuff packaged at RDO | 11:33 |
| b00tcat | shardy: although I must admit that I still didn't really understand when the RDO packages get pulled, as I thought they'd be in the base image already | 11:34 |
| shardy | b00tcat: once a repo is packaged and in the RDO repos, it gets built nearly every commit | 11:35 |
| shardy | https://trunk.rdoproject.org/ | 11:36 |
| shardy | b00tcat: however, we've recently removed all vendor packages from the base image, ref https://review.openstack.org/#/c/409809/ | 11:37 |
| b00tcat | shardy: thanks | 11:37 |
| shardy | the reason is none of them are tested in CI, and they broke us several times | 11:37 |
| b00tcat | shardy: that makes sense :D | 11:37 |
| *** shardy is now known as shardy_lunch | 11:39 | |
| *** adarazs is now known as adarazs_lunch | 11:39 | |
| *** pkovar has joined #tripleo | 11:41 | |
| *** rbowen has joined #tripleo | 11:43 | |
| *** apetrich has quit IRC | 11:44 | |
| jistr | weshay: i think i managed to pin down the "running `virsh list` destroys the dev env" issue -- https://bugs.launchpad.net/tripleo-quickstart/+bug/1650238 | 11:44 |
| openstack | Launchpad bug 1650238 in tripleo-quickstart "Systemd freezing execution in VMs on virsh list" [Low,New] | 11:44 |
| hewbrocca | jistr: are you using fedora as your virthost? | 11:45 |
| jistr | hewbrocca: no, quickstart will refuse running on fedora | 11:45 |
| hewbrocca | pmyers was having some problems with that, had to switch to centos | 11:45 |
| hewbrocca | ahh, OK | 11:45 |
| jistr | centos 7 | 11:45 |
| hewbrocca | adarazs_lunch: around ^^^ ? | 11:45 |
| jistr | i think it's likely that the root cause itself is in libvirt/qemu/kvm | 11:45 |
| jistr | not sure what can we do about it in quickstart | 11:46 |
| hewbrocca | huh yeah | 11:46 |
| jistr | (perhaps qemu:///system instead of qemu:///session, but that might have some other drawbacks) | 11:46 |
| hewbrocca | libvirt should not segfault, no matter what args you pass it | 11:46 |
| jistr | yea | 11:46 |
| hewbrocca | what's "-i" do to sudo? | 11:46 |
| openstackgerrit | Julie Pichon proposed openstack-infra/tripleo-ci: Add jpich to Planet https://review.openstack.org/411255 | 11:46 |
| jistr | hewbrocca: runs a login shell and executes the command there IIRC | 11:47 |
| jistr | as opposed to executing the target command directly | 11:47 |
| *** panda|lunch is now known as panda | 11:48 | |
| jistr | trown: FYI when you're around ^^ | 11:48 |
| jistr | well at least we know what commands *not* to run :) | 11:48 |
| EmilienM | hello | 11:48 |
| * jistr restarts OOOQ and relocates | 11:48 | |
| hewbrocca | jistr: oh, I gotcha | 11:50 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Remove white space between print () in plan_management.py https://review.openstack.org/410262 | 11:50 |
| *** bkopilov has quit IRC | 11:50 | |
| hewbrocca | why on earth would that make any difference? | 11:50 |
| tosky | uhm, the environment variables overwritten | 11:51 |
| EmilienM | tripleo ocata-2 has been released | 11:54 |
| *** jpena is now known as jpena|lunch | 11:54 | |
| *** fzdarsky is now known as fzdarsky|lunch | 11:55 | |
| *** jkilpatr has joined #tripleo | 11:57 | |
| *** ealcaniz has quit IRC | 11:59 | |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates: Adds a step0 for pre upgrade-init checks https://review.openstack.org/408631 | 11:59 |
| EmilienM | derekh: have you seen bnemec's thought on https://bugs.launchpad.net/tripleo/+bug/1649742 ? | 12:02 |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged] | 12:02 |
| derekh | EmilienM: nope, looking | 12:02 |
| derekh | EmilienM: ya, I came to a similar conclusion this morning, and ran a ci job with a tcpdump of the traffic to gearman http://logs.openstack.org/11/111011/82/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/4c53429/logs/dump.tcp | 12:04 |
| *** rbowen has quit IRC | 12:04 | |
| *** david-lyle has quit IRC | 12:05 | |
| *** david-lyle has joined #tripleo | 12:05 | |
| *** rbowen has joined #tripleo | 12:06 | |
| *** Baggypants12000 has joined #tripleo | 12:07 | |
| openstackgerrit | Numan Siddique proposed openstack/tripleo-heat-templates: OVN plugin configuration fixes https://review.openstack.org/397674 | 12:07 |
| EmilienM | derekh: do you have an idea? I see nothing special in the tcpdump | 12:07 |
| marios | shardy_lunch: matbu ccamacho hi guys I left a comment at https://review.openstack.org/#/c/408631/3 grateful for any thoughts if you get a chance - we still on for our call in 2.5 hours (ish)? | 12:08 |
| marios | chem: ^ | 12:08 |
| ccamacho | marios sure, Ill check it. | 12:09 |
| chem | marios: yep | 12:10 |
| marios | chem: jistr i updated the ovs upgrade one last night :) got rid of the ceph repetition with your suggestions https://review.openstack.org/#/c/399708/ thanks | 12:10 |
| *** ooolpbot has joined #tripleo | 12:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649742 | 12:10 |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged] | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1650005 | 12:10 |
| *** ooolpbot has quit IRC | 12:10 | |
| openstack | Launchpad bug 1650005 in tripleo "Multinode jobs failing on libvirt issues" [Critical,Triaged] | 12:10 |
| *** rbowen has quit IRC | 12:11 | |
| EmilienM | matbu, shardy_lunch: what is the plan for https://review.openstack.org/#/c/403397 ? Where this code is going to live? | 12:12 |
| derekh | EmilienM: the only think I think I see is the testenv-client finishing the CI job here @ 10:48:05.58650 | 12:12 |
| derekh | EmilienM: then pretty much ignoring all traffic from the geard server after that | 12:13 |
| EmilienM | derekh: could it be a firewall rule? | 12:13 |
| EmilienM | also I'm curious why it only happens on ha & update scenarios | 12:13 |
| *** cdearborn has joined #tripleo | 12:14 | |
| openstackgerrit | Numan Siddique proposed openstack/tripleo-heat-templates: OVN plugin configuration fixes https://review.openstack.org/397674 | 12:15 |
| derekh | EmilienM: Its possible, but I didn't think the firewall was running on those underclouds | 12:15 |
| *** masco is now known as masco_afk | 12:15 | |
| EmilienM | derekh: even if it would be, the rules would be the same as non-ha scenario | 12:16 |
| derekh | EmilienM: yup | 12:16 |
| derekh | EmilienM: I've recheck this, we can ssh to it in a while to see if we can see anything https://review.openstack.org/#/c/363630 | 12:16 |
| openstackgerrit | Numan Siddique proposed openstack/tripleo-heat-templates: OVN bridge mappings for tripleo https://review.openstack.org/402293 | 12:18 |
| EmilienM | matbu: I was thinking at starting the work to move it to a repo and package it soon, time it kind of running out quickly in this cycle | 12:18 |
| *** udesale has quit IRC | 12:19 | |
| matbu | EmilienM: we will discusss it this afternoon | 12:21 |
| matbu | EmilienM: but yes it could be useful, is there a doc somewhere on how should that be packaged ? (i can take care of it) | 12:25 |
| *** lucasagomes is now known as lucas-hungry | 12:26 | |
| EmilienM | matbu: ask pabelanger | 12:28 |
| EmilienM | matbu: he did that already I think | 12:28 |
| *** owalsh is now known as owalsh-afk | 12:28 | |
| *** skramaja has quit IRC | 12:28 | |
| *** skramaja_ has joined #tripleo | 12:28 | |
| *** dprince has joined #tripleo | 12:29 | |
| *** pkovar has quit IRC | 12:30 | |
| *** masco_afk is now known as masco | 12:36 | |
| *** abregman is now known as abregman|afk | 12:37 | |
| *** kodokuu has quit IRC | 12:38 | |
| EmilienM | bnemec: https://bugs.launchpad.net/tripleo/+bug/1650005 I think we can close it, it seems like it works fine now | 12:42 |
| openstack | Launchpad bug 1650005 in tripleo "Multinode jobs failing on libvirt issues" [Critical,Triaged] | 12:42 |
| *** nmathew has quit IRC | 12:42 | |
| * EmilienM removing the alert and closing it | 12:43 | |
| *** jpena|lunch is now known as jpena | 12:43 | |
| gfidente | EmilienM this guy is blocked by CI from a few days | 12:44 |
| gfidente | https://review.openstack.org/402385 | 12:44 |
| *** pradk has joined #tripleo | 12:44 | |
| gfidente | there are 3 submissions in tripleo which depend on it | 12:44 |
| EmilienM | gfidente: man, everything is blocked right now | 12:44 |
| gfidente | have any idea if/how we can push it a bit further? | 12:44 |
| *** bfournie has quit IRC | 12:44 | |
| EmilienM | gfidente: the qemu thing broke puppet and tripleo CI all week | 12:44 |
| EmilienM | we merged almost nothing | 12:45 |
| gfidente | yeah I was hitting that with quickstart locally | 12:45 |
| *** pradk has quit IRC | 12:45 | |
| gfidente | my only working workaround to date | 12:45 |
| gfidente | has been to downgrade libvirtd | 12:45 |
| EmilienM | gfidente: see https://review.openstack.org/#/c/411178/ - that's the fix for master CI | 12:45 |
| openstackgerrit | Merged openstack/instack: Add changes for dib feature/v2 branch https://review.openstack.org/408456 | 12:45 |
| EmilienM | gfidente: but stable/newton, stable/mitaka and stable/hammer is still broken now, we're working on it. any help is welcome | 12:45 |
| openstackgerrit | Merged openstack/instack-undercloud: Set Ironic cleaning network to ctlplane https://review.openstack.org/406197 | 12:46 |
| gfidente | EmilienM but is it the same issue for all the releases? | 12:47 |
| gfidente | or am I missing something? | 12:47 |
| EmilienM | kind of | 12:47 |
| openstackgerrit | Merged openstack/instack-undercloud: Optional Cinder support for undercloud https://review.openstack.org/405435 | 12:47 |
| EmilienM | gfidente: there are multiple emails about that on openstack-dev | 12:47 |
| *** rhallisey has joined #tripleo | 12:47 | |
| EmilienM | [rdo-list] Known issue with CentOS 7.3 and qemu-kvm(-ev) 2.6.0 | 12:48 |
| EmilienM | gfidente: that ^ | 12:48 |
| gfidente | yeah I replied to that thread too | 12:48 |
| EmilienM | gfidente: the poi patch merged, do recheck now | 12:48 |
| gfidente | I am not sure we want to push jewel for the mitaka relase | 12:48 |
| gfidente | we shipped hewel in the overcloud images | 12:49 |
| gfidente | *hammer | 12:49 |
| EmilienM | gfidente: like I said, there is no way we push jewel in mitaka branch | 12:49 |
| gfidente | yeah | 12:49 |
| EmilienM | gfidente: I -2 the idea and I'll keep my position | 12:49 |
| gfidente | yeah I was saying the same thing | 12:49 |
| *** tobias-fiberdata has joined #tripleo | 12:50 | |
| EmilienM | gfidente: I did recheck on your puppet-ceph patch though I don't know why it failed, I'm looking at logs | 12:51 |
| *** chlong has joined #tripleo | 12:51 | |
| gfidente | EmilienM same issue with nodes not starting in nova | 12:51 |
| EmilienM | sounds like a timeout | 12:51 |
| adarazs_lunch | jistr, hewbrocca: thanks for the bug against quickstart, let me know how that deployment goes! | 12:51 |
| *** adarazs_lunch is now known as adarazs | 12:51 | |
| EmilienM | gfidente: it doesn't seem related to your patch :/ | 12:52 |
| gfidente | yeah I mean it's the known issue with libvirt unable to start the guests | 12:52 |
| gfidente | so the overcloud fails | 12:52 |
| EmilienM | gfidente: are you sure? | 12:53 |
| *** tobias_fiberdata has quit IRC | 12:53 | |
| EmilienM | it's multinode job, there is no nova here | 12:53 |
| *** jprovazn has joined #tripleo | 12:54 | |
| gfidente | EmilienM pingtest | 12:57 |
| EmilienM | gfidente: no, overcloud failed before pingtest | 12:57 |
| * EmilienM brb | 12:57 | |
| *** cdearborn has quit IRC | 12:58 | |
| jistr | adarazs: it always goes well, as long as one doesn't try to `virsh list` the wrong way :D | 13:00 |
| adarazs | jistr: yeah, what I usually do is "su - stack" completely, which is equivalent to using sudo with `-i` so I wouldn't have discovered this :) | 13:02 |
| gfidente | EmilienM I see both scenarios failing on pingtest | 13:03 |
| jistr | marios: re ovs upgrade -- looks good, thanks :) | 13:04 |
| jistr | +2'd | 13:04 |
| marios | thanks jistr | 13:05 |
| *** pkovar has joined #tripleo | 13:06 | |
| *** skramaja_ has quit IRC | 13:06 | |
| *** skramaja has joined #tripleo | 13:06 | |
| *** shardy_lunch is now known as shardy | 13:06 | |
| EmilienM | gfidente: do you have logs? | 13:07 |
| *** ooolpbot has joined #tripleo | 13:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649742 | 13:10 |
| *** ooolpbot has quit IRC | 13:10 | |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged] | 13:10 |
| *** amoralej is now known as amoralej|lunch | 13:10 | |
| *** pgadiya has quit IRC | 13:11 | |
| openstackgerrit | mathieu bultel proposed openstack/tripleo-quickstart-extras: add composable services deployment support for oooq https://review.openstack.org/410826 | 13:11 |
| *** jeckersb_gone is now known as jeckersb | 13:12 | |
| *** oshvartz has quit IRC | 13:14 | |
| *** dmacpher has quit IRC | 13:15 | |
| *** dmacpher has joined #tripleo | 13:15 | |
| *** fzdarsky|lunch is now known as fzdarsky | 13:15 | |
| *** abregman|afk is now known as abregman | 13:16 | |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Run upgrade steps before post-deploy config https://review.openstack.org/409857 | 13:19 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Move UpgradeInitCommand to role templates https://review.openstack.org/411310 | 13:19 |
| *** apetrich has joined #tripleo | 13:20 | |
| *** cdearborn has joined #tripleo | 13:22 | |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-quickstart-extras: Repositories setup role https://review.openstack.org/408162 | 13:22 |
| *** saneax is now known as saneax-_-|AFK | 13:23 | |
| *** bfournie has joined #tripleo | 13:24 | |
| *** kodokuu has joined #tripleo | 13:24 | |
| *** jprovazn has quit IRC | 13:25 | |
| shardy | matbu: Hey, I've got the upgrade working locally now - I've pushed the tht patches, but a few changes are needed to the tripleo-ci patch too, are you OK if I just push them? | 13:25 |
| *** zoli|wfh is now known as zoli|lunch | 13:26 | |
| shardy | I'm just testing again trying to combine the upgrade and converge, then I can refactor my local changes into your patch, if you're OK with that | 13:26 |
| *** jayg|g0n3 is now known as jayg | 13:26 | |
| matbu | shardy: yep sure | 13:27 |
| matbu | shardy: where is the tht review ? | 13:27 |
| shardy | https://review.openstack.org/#/c/411310/ | 13:27 |
| *** [1]cdearborn has joined #tripleo | 13:27 | |
| matbu | shardy: i have implemented the base for making the same test in quickstart yesterday | 13:27 |
| shardy | matbu: ^^ - I then need to change UpgradeInitCommand to handle disabling the old hiera element | 13:28 |
| shardy | as well as installing the new hiera heat-config hook | 13:28 |
| matbu | shardy: it should be easy then to reused the same workflow in both (tripleo-ci and quickstart) | 13:28 |
| shardy | other than that, it works pretty well | 13:28 |
| matbu | nice thanks | 13:28 |
| *** lucas-hungry is now known as lucasagomes | 13:29 | |
| shardy | matbu: nice re quickstart - I was thinking we'd need that pretty soon | 13:29 |
| *** links has quit IRC | 13:31 | |
| *** ayoung_dadmode is now known as ayoung | 13:32 | |
| *** rbowen has joined #tripleo | 13:32 | |
| shardy | matbu: did you have any preference re the git vs rpm comment in https://review.openstack.org/#/c/404831/31/scripts/deploy.sh ? | 13:33 |
| shardy | locally I've been unpacking a newton tht RPM instead of cloning the git tree, but either should work | 13:34 |
| matbu | shardy: i think git is fine but maybe rpm is more robust ? | 13:35 |
| shardy | matbu: Yeah it might be, IIRC we had spurious errors cloning from git URLs in the past, but my main motivation for RPM was to enable overlaying the tripleo-ci repo in future, then we can potentially make Depends-On work | 13:36 |
| *** bkopilov has joined #tripleo | 13:36 | |
| shardy | Hopefully we won't need Depends-On for the stable tht version very often tho | 13:36 |
| matbu | shardy: yep right | 13:37 |
| *** sudipto has joined #tripleo | 13:37 | |
| *** sudipto_ has joined #tripleo | 13:37 | |
| shardy | I'll push the RPM version then you can compare and see if you're happy with it | 13:37 |
| *** limao has joined #tripleo | 13:37 | |
| dprince | shardy: hi, so I'm trying to get rid of my use of noauth middleware in Heat | 13:40 |
| dprince | shardy: https://review.openstack.org/#/c/347505/ | 13:40 |
| jaosorior | shardy: a bit of yaql help. Any idea why what's wrong with this one http://paste.openstack.org/show/592480/ ? It's just trying to generate a list with the values of the maps | 13:41 |
| dprince | shardy: since I've gone and mocked out a full on keystone it should be possible I think | 13:41 |
| dprince | shardy: any ideas on how to work past this... I hit it when I disable noauth: http://paste.openstack.org/show/592479/ | 13:41 |
| dprince | shardy: I guess I just need to beef up my service catalog fake? | 13:41 |
| *** cdearborn has quit IRC | 13:41 | |
| dprince | shardy: I'd like it just simply not to query it though, which it doesn't seem to w/ noauth | 13:42 |
| shardy | dprince: Are you overriding the default signal_transport? Looks like it's trying to use CFN, which is the default | 13:43 |
| dprince | shardy: I use Heat API directly, no CFN | 13:43 |
| *** dhill_ has joined #tripleo | 13:44 | |
| dprince | shardy: which is why this on surprises me | 13:44 |
| *** ansmith has joined #tripleo | 13:44 | |
| shardy | dprince: Yeah but *Deployment resources default to signalling via the CFN API, for historical reasons | 13:44 |
| shardy | http://docs.openstack.org/developer/heat/template_guide/openstack.html#OS::Heat::SoftwareDeployment-prop-signal_transport | 13:44 |
| shardy | dprince: it can be globally configured in the heat.conf, sec | 13:44 |
| dprince | shardy: default_deployment_signal_transport = HEAT_SIGNAL | 13:44 |
| dprince | shardy: I set that | 13:44 |
| shardy | Hrm, then I guess there's a bug where it's still trying to find the cfn endpoint and failing | 13:45 |
| dprince | shardy: okay, well at least we agree this looks suspicios :) | 13:45 |
| * shardy finds the code | 13:45 | |
| shardy | dprince: Yeah, it's some heat validation getting in the way - we try to ensure the services used by resources are in the catalog, a bit too eagerly in this case it seems | 13:46 |
| dprince | shardy: FWIW, noauth seems to avoid this codepath entirely https://review.openstack.org/#/c/347505/. The patch has been posted since mid-summer though and no love as of yet | 13:47 |
| dprince | shardy: so I figured I'd try without it... because it should be feasable now that I've got a keystone fake | 13:47 |
| shardy | dprince: Ok, lets see if we can chase for some review love on https://review.openstack.org/#/c/347505 then | 13:49 |
| *** zoli|lunch is now known as zoli|wfh | 13:50 | |
| dprince | shardy: ack, thanks | 13:51 |
| shardy | jaosorior: try $.data.principaldata.select([$.service, $.network]) | 13:53 |
| jaosorior | shardy: that worked, thanks | 13:54 |
| *** kodokuu has left #tripleo | 13:59 | |
| openstackgerrit | Julie Pichon proposed openstack/tripleo-ui: Disallow underscores in plan names https://review.openstack.org/411330 | 13:59 |
| jaosorior | shardy: is it possible to do string interpolation in yaql | 14:06 |
| jaosorior | ? | 14:06 |
| jaosorior | something like: $.data.principaldata.select("$.service/$.network") | 14:07 |
| shardy | jaosorior: not sure tbh - I'd check the tests in the yaql tree for anything similar | 14:08 |
| *** amoralej|lunch is now known as amoralej | 14:09 | |
| *** ooolpbot has joined #tripleo | 14:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649742 | 14:10 |
| *** ooolpbot has quit IRC | 14:10 | |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged] | 14:10 |
| openstackgerrit | Raoul Scarazzini proposed openstack/tripleo-quickstart-extras: Baremetal undecloud role playbook fixes https://review.openstack.org/411336 | 14:11 |
| *** mhenkel has joined #tripleo | 14:11 | |
| *** derekh is now known as derekh_brb | 14:12 | |
| jaosorior | shardy: found it :D | 14:14 |
| jaosorior | there's a concat and a format function | 14:14 |
| shardy | jaosorior: ah, nice | 14:15 |
| openstackgerrit | Keith Schincke proposed openstack/puppet-tripleo: Add support for not using admin_token in Ceph/RGW https://review.openstack.org/405567 | 14:17 |
| gfidente | adarazs if I use localhost as VIRTHOST | 14:17 |
| gfidente | am I supposed to create the stack user myself? | 14:17 |
| *** tzumainn has joined #tripleo | 14:17 | |
| adarazs | gfidente: it should work if execute quickstart as another user (not root, not 'stack') and use "127.0.0.2" instead of localhost. ansible is a bit finicky when using a local connection. | 14:18 |
| adarazs | 127.0.0.2 goes around that. :) | 14:19 |
| gfidente | adarazs right but the user itself | 14:19 |
| gfidente | 'stack' | 14:19 |
| gfidente | should I create it anyway? | 14:19 |
| adarazs | gfidente: it will be created. | 14:19 |
| gfidente | it seems to be trying to ssh on baremetal with that user | 14:19 |
| gfidente | but the user isn't there | 14:19 |
| *** chlong has quit IRC | 14:20 | |
| adarazs | gfidente: https://github.com/openstack/tripleo-quickstart/blob/master/playbooks/provision.yml#L15 -> https://github.com/openstack/tripleo-quickstart/blob/master/roles/provision/remote/tasks/main.yml#L36 | 14:21 |
| *** Vijayendra has quit IRC | 14:21 | |
| adarazs | gfidente: this is where the user creation happens. you have to have ssh access to root@$VIRTHOST to make this work | 14:22 |
| *** chlong has joined #tripleo | 14:22 | |
| adarazs | as we initially ssh as root to create the non_root_user (stack) | 14:22 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Add hook to generate metadata from service profiles https://review.openstack.org/411339 | 14:23 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Add metadata settings for needed kerberos principals https://review.openstack.org/411340 | 14:23 |
| gfidente | adarazs yeah I see that if virthost matches localhost it uses provision/local | 14:23 |
| gfidente | which doesn't create the stack user it seems, right? | 14:24 |
| *** morazi has joined #tripleo | 14:24 | |
| adarazs | gfidente: yeah, I think we have a couple of extra cases if you run quickstart with localhost as virthost, but it's poorly tested. I know trown is using "127.0.0.2" to get around all the ansible limitations and run it locally | 14:24 |
| marios | shardy matbu chem ccamacho mcornea tripleo upgrades squad call in 5 minutes @ https://etherpad.openstack.org/p/tripleo-upgrades-squad (dial-in info on the pad) | 14:25 |
| *** ealcaniz has joined #tripleo | 14:25 | |
| ccamacho | marios yeahp :) | 14:25 |
| adarazs | gfidente: try that. | 14:25 |
| mcornea | marios: thanks, joining | 14:25 |
| gfidente | adarazs ack thanks | 14:25 |
| *** egafford has joined #tripleo | 14:26 | |
| *** chlong has quit IRC | 14:26 | |
| *** chlong has joined #tripleo | 14:27 | |
| *** lblanchard has joined #tripleo | 14:30 | |
| *** Goneri has joined #tripleo | 14:31 | |
| *** paramite has quit IRC | 14:31 | |
| EmilienM | chem: I'm not coming but you might want to talk about https://etherpad.openstack.org/p/tripleo-upgrade-jobs | 14:31 |
| *** morazi has quit IRC | 14:31 | |
| *** jcoufal has joined #tripleo | 14:31 | |
| chem | EmilienM: I'm there already. Thanks for the pingback | 14:31 |
| marios | shardy: are you able to join? | 14:33 |
| *** pkovar has quit IRC | 14:33 | |
| *** oshvartz has joined #tripleo | 14:35 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart-extras: Replace hardcoded stack user by ansible_user https://review.openstack.org/404800 | 14:36 |
| shardy | marios: ah, thanks, joining | 14:38 |
| *** ealcaniz has quit IRC | 14:39 | |
| openstackgerrit | Steven Hardy proposed openstack-infra/tripleo-ci: Implement major upgrade for Newton to Ocata https://review.openstack.org/404831 | 14:42 |
| openstackgerrit | Steven Hardy proposed openstack-infra/tripleo-ci: Fix REPO_PREFIX handling of ceph repos https://review.openstack.org/409814 | 14:42 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Force kill testenv-client https://review.openstack.org/411353 | 14:43 |
| derekh_brb | EmilienM: bnemec ^^ a really nasty hack, but it might give us some breathing room to properly debug the problem with the te-client hanging | 14:43 |
| openstackgerrit | John Fulton proposed openstack/tripleo-specs: Integrate TripleO with Tendrl for External Storage Deployment/Management https://review.openstack.org/387631 | 14:44 |
| EmilienM | derekh_brb: nice :D | 14:44 |
| derekh_brb | EmilienM: bnemec I'm not even sure this will cause things to clean up properly on the te-broker, we'll have to check that if we are going to merge it | 14:44 |
| *** derekh_brb is now known as derekh | 14:44 | |
| *** morazi has joined #tripleo | 14:44 | |
| EmilienM | ok | 14:45 |
| *** ealcaniz has joined #tripleo | 14:47 | |
| *** ealcaniz is now known as ealcaniz|Orange- | 14:47 | |
| dprince | shardy: https://review.rdoproject.org/r/4094 Add element to disable old hiera element | 14:48 |
| *** dsariel has quit IRC | 14:48 | |
| dprince | shardy: I think that should handle it. A bit odd perhaps, and I could move it into the heat-templates tree if that makes more sense to you | 14:48 |
| dprince | shardy: I think perhaps eventually we'd want an element that removes anything in libexec/os-refresh-config that isn't in a proper RPM package. That would solve the problem globally | 14:49 |
| EmilienM | bnemec: hi! do you think we'll make progress on https://blueprints.launchpad.net/tripleo/+spec/undercloud-upgrade during ocata-3 ? | 14:50 |
| openstackgerrit | John Fulton proposed openstack/tripleo-specs: Integrate TripleO with Tendrl for External Storage Deployment/Management https://review.openstack.org/387631 | 14:53 |
| *** athomas has quit IRC | 14:53 | |
| EmilienM | shardy: do you think we can close https://blueprints.launchpad.net/tripleo/+spec/allow-password-modification which is duplicated with https://bugs.launchpad.net/tripleo/+bug/1611704 ? | 14:53 |
| openstack | Launchpad bug 1611704 in tripleo "User defined passwords should be easier to define" [High,In progress] - Assigned to Saravanan KR (skramaja) | 14:53 |
| *** ealcaniz|Orange- has quit IRC | 14:53 | |
| *** pkovar has joined #tripleo | 14:53 | |
| openstackgerrit | Brad P. Crochet proposed openstack/python-tripleoclient: Migrate overcloud update to a mistral workflow https://review.openstack.org/381351 | 14:55 |
| *** b00tcat has quit IRC | 14:55 | |
| *** b00tcat has joined #tripleo | 14:56 | |
| *** udesale has joined #tripleo | 14:57 | |
| *** links has joined #tripleo | 15:01 | |
| *** athomas has joined #tripleo | 15:01 | |
| shardy | EmilienM: Yes I think we can consider it a bug and track it via https://bugs.launchpad.net/tripleo/+bug/1611704 | 15:03 |
| openstack | Launchpad bug 1611704 in tripleo "User defined passwords should be easier to define" [High,In progress] - Assigned to Saravanan KR (skramaja) | 15:03 |
| *** Goneri has quit IRC | 15:03 | |
| *** chlong has quit IRC | 15:03 | |
| shardy | dprince: ack, thanks, and +100 on removing all the non-package-owned-things ;) | 15:03 |
| EmilienM | shardy: excellent | 15:05 |
| EmilienM | dtantsur|brb, lucasagomes: what progress do we have on https://blueprints.launchpad.net/tripleo/+spec/switch-to-virtualbmc ? Still on track for ocata? | 15:05 |
| *** pblaho has quit IRC | 15:06 | |
| *** pradk has joined #tripleo | 15:06 | |
| lucasagomes | EmilienM, hi there, yes... the package for virtualbmc is all sorted. It's missing the work for tripleo-quickstart, my patch got a +2 but was -1'd yesterday | 15:06 |
| lucasagomes | so I'm going to address the comments for it today | 15:06 |
| lucasagomes | EmilienM, https://review.openstack.org/#/c/399704/8 | 15:06 |
| EmilienM | lucasagomes: why tripleo-quickstart? | 15:06 |
| lucasagomes | EmilienM, cause that's the tool we use for development of tripleo ? | 15:07 |
| lucasagomes | (ssh driver being a test/develop driver | 15:07 |
| EmilienM | lucasagomes: do we test that in tripleo-ci? | 15:07 |
| lucasagomes | )* | 15:07 |
| EmilienM | lucasagomes: we haven't switched all jobs to oooq now, sshnaidm|afk is working on htat | 15:07 |
| *** sshnaidm|afk is now known as sshnaidm | 15:08 | |
| lucasagomes | EmilienM, right, so the remaining should also be updated. I will take a look at the tripleo-ci | 15:08 |
| jistr | weshay, adarazs, trown: folks how do you test changes to quickstart? do you re-run the whole thing from scratch every time? I'm used to making playbooks more-or-less idempotent and re-running them, but it doesn't seem like that's the best approach with OOOQ... are there some tricks you do to make the run partial or something? | 15:08 |
| EmilienM | lucasagomes: ok. I consider it still on track then | 15:10 |
| *** ooolpbot has joined #tripleo | 15:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649742 | 15:10 |
| *** ooolpbot has quit IRC | 15:10 | |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged] | 15:10 |
| lucasagomes | EmilienM, right on | 15:10 |
| trown | jistr: there are kind of two types of changes, and I approach them differently | 15:10 |
| *** numans has quit IRC | 15:10 | |
| trown | jistr: If I am changing the virt-setup stuff I just re-run from scratch each time only to the point undercloud is installed | 15:11 |
| *** tosky has quit IRC | 15:11 | |
| trown | jistr: If I am changing something that happens after the undercloud is installed, I usually just hack directly on the scripts on the undercloud until I get it right, then convert that back to the jinja templates in quickstart and re-run from scratch to confirm my changes and submit them | 15:12 |
| dprince | shardy: using something like this (with conventions) I think I could get rid of neutron for our network isolation support https://review.openstack.org/#/c/409523/1/deployed-server/deployed-neutron-port.yaml | 15:12 |
| dprince | shardy: I'm using that for the t-h-t undercloud installer now and it works really nicely for 'named' Neutron ports | 15:13 |
| *** numans has joined #tripleo | 15:14 | |
| dprince | undercloud neutron... how do you like them apples | 15:14 |
| derekh | Networking people, does this look normal? same tcp connection multiple processes http://paste.openstack.org/show/592499/ | 15:14 |
| *** Goneri has joined #tripleo | 15:15 | |
| trown | jistr: we do have tags on the tasks for doing partial runs, but that interface is a bit fragile given it is not CI'd at all... we are pretty resource constrained in what we can CI (as in number of concurrent jobs allowed in centosci) or it would make sense to have some jobs confirming some other subsets of tags working | 15:15 |
| jistr | trown: ok, thanks. How do i do a partial run, e.g. to the point undercloud is installed? I guess with --tags or --skip-tags? | 15:15 |
| jistr | trown: haha thanks | 15:15 |
| shardy | dprince: Nice, yeah that's the sort of thing I imagined we might use to replace neutron, e.g just a big map of ips | 15:16 |
| shardy | dprince: you could also combine it with the network composition stuff I started in https://review.openstack.org/#/c/409921/ so that you auto-generated the last octet e.g based on a combination of the network cidr and the node index | 15:17 |
| dprince | shardy: my implementation looks like this ATM http://paste.openstack.org/show/592500/ | 15:18 |
| trown | jistr: actually with default tags and quickstart-extras.yml playbook you should get all of the templated scripts written out, but nothing run after the undercloud install | 15:18 |
| dprince | shardy: ack on the NW composition stuff | 15:18 |
| trown | jistr: hmm... nope... I am wrong on that | 15:19 |
| shardy | dprince: nice, yeah I guess we could handle explicit mappings and index based ones and neither would actually need neutron | 15:19 |
| trown | jistr: that is a bug since moving to quickstart-extras... there are roles not appropriately tagged in quickstart-extras.yml | 15:19 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Move UpgradeInitCommand to role templates https://review.openstack.org/411310 | 15:20 |
| dprince | shardy: it is working really great for the Undercloud. I'm not fully able to use overcloud.yaml. So we won't need a separate undercloud.yaml that I was prototyping before... | 15:20 |
| trown | jistr: I will make a launchpad for that and submit a patch | 15:21 |
| EmilienM | dtrainor: hey! are we still on track for https://blueprints.launchpad.net/tripleo/+spec/proxy-undercloud-api-services ? | 15:21 |
| jistr | trown: perfect, thanks a lot :) if we're not aiming for idempotency, then being able to do partial runs of particular parts would help a lot i think | 15:21 |
| dtrainor | hi, EmilienM. tl;dr: yes. just in the last 48 hours in the testing i've done, i've discovered that this work might not be suited for haproxy, instead apache, since apache is responsible for serving the ui. | 15:23 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Add ZaqarApiNetwork to the service net map https://review.openstack.org/411379 | 15:24 |
| EmilienM | dtrainor: end of ocata-3 is end of january. Are we going to finish the work by then? | 15:25 |
| openstackgerrit | Raoul Scarazzini proposed openstack/tripleo-quickstart-extras: Baremetal undecloud role playbook fixes https://review.openstack.org/411336 | 15:25 |
| trown | jistr: idempotency is actually a goal, we just need more CI to get there though. being able to run quickstart jobs on OVB and nodepool will help alot though | 15:25 |
| trown | jistr: actually overcloud-deploy role has the tags in the main task list, so the include of it without a tag is actually fine... I still think there is some issue with the steps between undercloud install and overcloud-deploy but if you are working on THT/puppet-tripleo stuff running with default tags and quickstart-extras.yml playbook should get you where you want | 15:28 |
| trown | https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/overcloud-deploy/tasks/main.yml | 15:28 |
| *** liverpooler has joined #tripleo | 15:28 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: DONT REVIEW: test https://review.openstack.org/410470 | 15:28 |
| * trown reviews ^ | 15:28 | |
| panda | derekh: I seems normal, different processes sharing the same socket, especially when you fork and the file descriptors gets duplicated. | 15:29 |
| derekh | panda: ok | 15:29 |
| jistr | trown: cool, thanks for the info. Btw the way i used to approach idempotency was to do just very easy checks, which, in case of executing larger operations in one swoop, would err towards the side of not running the larger chunk at all. E.g. "if i see keystone service running on the undercloud VM, then i will not run undercoud install, b/c it was most likely already run". I didn't go checking for all | 15:30 |
| jistr | services in detail etc. Perhaps a bit subjective approach to ansible idempotency, but seemed to work well for me :) | 15:31 |
| dtrainor | EmilienM, i am aware, and yep. We're discussing some concerns we've had about some defaults that we'll need for UI in order to accomplish this. We'll have more answers shortly. | 15:33 |
| trown | jistr: ya that makes sense, there are lots of places in quickstart where there are checks like that, but to me ensuring idempotency requires CI testing it | 15:34 |
| EmilienM | dtrainor: ok, still on track then. Thanks | 15:37 |
| trown | jistr: another trick is to run with "--teardown none --retain-inventory", that will keep quickstart from tearing down the VMs and inventory before running... that is at least how you would uncover where we have bugs with idempotency :) | 15:38 |
| *** owalsh-afk is now known as owalsj | 15:38 | |
| *** owalsj is now known as owalsh | 15:38 | |
| trown | jistr: default is set to simulate starting from scratch because that is the only thing that is CI'd | 15:38 |
| dtrainor | yessir, thanks for bringing it up. | 15:38 |
| jistr | trown: re "--teardown none --retain-inventory" -- nice, i'll definitely use it at some point. Thanks a bunch :) | 15:40 |
| trown | jistr: just to close loop on tags thing... only one role doesn't have tags, and it is only templating some files so shouldn't intefere, so I am not going to put a bug up for it | 15:40 |
| jistr | ack, sounds fine | 15:41 |
| *** liverpooler has quit IRC | 15:41 | |
| *** liverpooler has joined #tripleo | 15:41 | |
| trown | jistr: but I am unconvinced the tags approach was good in the first place, so maybe we can rethink that interface at somepoint | 15:41 |
| trown | probably just having more example playbooks that are indvidually CI'd with better docs on constructing and using custom playbooks would be better than tags | 15:43 |
| weshay | sshnaidm, think we've figured out why the container validate failed | 15:43 |
| sshnaidm | weshay, great, why? | 15:43 |
| weshay | sshnaidm, https://review.openstack.org/#/c/395880/10..11/docker/firstboot/start_docker_agents.sh | 15:44 |
| weshay | the containers are missing a hosts file | 15:44 |
| weshay | sshnaidm, so we're pretty dam close to full success again | 15:44 |
| weshay | think we're clear to move forward and getting it upstream | 15:44 |
| *** oshvartz has quit IRC | 15:44 | |
| weshay | I see some -1's but I'd almost rather *not* change anything at this point | 15:45 |
| weshay | trown, maybe you can weigh in | 15:45 |
| jistr | trown: yea, or perhaps having the checks on playbook level rather than tasks level would make it more DRY and visible (tags on tasks level need to be specified in many places, so it's easier to miss it somewhere) | 15:45 |
| jistr | trown: https://github.com/jistr/kubelunch/blob/4153e4f834eef874991b7fb7fba71cf8af724df6/playbooks/kubelunch.yml#L6-L17 | 15:45 |
| sshnaidm | weshay, ok, then just ping me when it's merged, I'm almost ready to put it in upstream.. | 15:46 |
| derekh | panda: sshnaidm EmilienM: looking back at the last ha job to pass, it was on centos 7.2 , the first job to have the timout error is centos 7.3 | 15:47 |
| *** saibarspeis has quit IRC | 15:47 | |
| jistr | trown: also previously i did "stopping points" but those 1) assume you can always safely run from the beginning 2) they "purposefully failed" the ansible playbook at some point, so they gave a false appearance of failure (including return code). So it's kinda dirty and definitely not good for use in CI for example. I probably wouldn't go that way again. | 15:47 |
| derekh | that doesn't narrow much down, probably means it could be any number of things..... | 15:47 |
| sshnaidm | derekh, that's bad.. seems like some things works there different | 15:47 |
| derekh | could be | 15:48 |
| *** masco has quit IRC | 15:49 | |
| trown | jistr: ya in quickstart we have a mix of tags, example playbooks, and when variables to control execution... the worst of all worlds :) | 15:49 |
| EmilienM | derekh: mhh, we have fixed puppet-nova to configure nova correctly with the new qemu | 15:49 |
| jistr | haha | 15:49 |
| EmilienM | derekh: do you have logs? | 15:50 |
| trown | jistr: personally I think playbooks are best, just declare exactly what you want to run and run it... but there is some trade-off in user-friendliness to non-ansible users | 15:50 |
| derekh | EmilienM: http://logs.openstack.org/97/409697/1/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/5ac4a16/ | 15:51 |
| trown | though the user-friendliness of tags has not been all that either | 15:52 |
| EmilienM | derekh: I don't see qemu bug here | 15:52 |
| derekh | EmilienM: that was the last HA job to pass, it was on 7.2 | 15:53 |
| EmilienM | derekh: ah ok | 15:53 |
| *** jlinkes has quit IRC | 15:55 | |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-quickstart-extras: Look for success message in logs when check status https://review.openstack.org/405392 | 15:56 |
| jistr | trown: yea... personally i don't like tags too much for controlling what roles to execute. It's a bit too verbose / error prone, maybe they work better for more granular, perhaps role-spanning selection ("only generate configs but don't start any services" etc.). The when variables feel a bit better for role selection. And yea i agree custom playbook being cleaner but harder for folks wihout ansible | 15:59 |
| jistr | experience (especially if the playbook is a bit larger/elaborate). | 15:59 |
| trown | ya, when is nice too because it can go right into the yaml config or on the command line, where as tags are command line only | 16:01 |
| *** Goneri has quit IRC | 16:01 | |
| *** trown is now known as trown|mtg | 16:01 | |
| *** abregman has quit IRC | 16:01 | |
| *** links has quit IRC | 16:04 | |
| *** udesale has quit IRC | 16:05 | |
| *** jbadiapa has quit IRC | 16:05 | |
| *** pkovar has quit IRC | 16:07 | |
| openstackgerrit | Michele Baldessari proposed openstack/puppet-pacemaker: Relax requires when creating resources on remote nodes https://review.openstack.org/408303 | 16:08 |
| openstackgerrit | Michele Baldessari proposed openstack/puppet-tripleo: Initial pacemaker remote profile support https://review.openstack.org/400967 | 16:09 |
| *** ooolpbot has joined #tripleo | 16:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649742 | 16:10 |
| *** ooolpbot has quit IRC | 16:10 | |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged] | 16:10 |
| *** jmelvin has joined #tripleo | 16:11 | |
| *** pblaho has joined #tripleo | 16:11 | |
| *** Goneri has joined #tripleo | 16:14 | |
| jmelvin | ping triple0, can someone check this pastebin please. i'm having trouple installing Mitaka undercloud: https://paste.fedoraproject.org/506942/81835814/ | 16:14 |
| *** tremble has quit IRC | 16:14 | |
| EmilienM | jaosorior: is it something we still target for ocata-3 ? https://blueprints.launchpad.net/tripleo/+spec/novajoin | 16:15 |
| *** ayoung has quit IRC | 16:17 | |
| shardy | jmelvin: hi, heat::engine::auth_encryption_key looks wrong, it should be 32 characters long | 16:18 |
| jmelvin | shardy, right, but i'm wondering why that would happen? | 16:18 |
| shardy | jmelvin: I assume undercloud_heat_encryption_key isn't set to "16" in your undercloud.conf :) | 16:19 |
| *** alee has joined #tripleo | 16:19 | |
| jaosorior | EmilienM: so, we are still targetting it for ocata-3 | 16:19 |
| jaosorior | EmilienM: the service is already accepted in the big tent, and we're polishing some installation details | 16:20 |
| jmelvin | shardy, they have set: undercloud_heat_encryption_key = 16 | 16:20 |
| shardy | jmelvin: Ok, that's the problem then :) | 16:20 |
| jaosorior | EmilienM: but we do need to finish up the puppet modules in order to be able to take use of it in tripleo | 16:20 |
| jmelvin | shardy, change it to 32? | 16:20 |
| jaosorior | EmilienM: cause at the moment the TLS work is REALLY tedious to set up, and this will ease a lot of the pain from it. | 16:20 |
| shardy | jmelvin: just unset it then we'll create a random 32 character string | 16:21 |
| jmelvin | shardy, so comment it out right? | 16:21 |
| shardy | https://github.com/openstack/instack-undercloud/blob/master/instack_undercloud/undercloud.py#L343 | 16:21 |
| shardy | jmelvin: yes | 16:21 |
| EmilienM | jaosorior: I understand you're polishing details. But are we going to merge everything before end of january? | 16:21 |
| jmelvin | shardy, thank you sir , i will give that a try | 16:21 |
| EmilienM | jaosorior: I still see a lot of dependencies here | 16:22 |
| EmilienM | and it seems very experimental to me now | 16:22 |
| EmilienM | jaosorior: can you give me list of blockers/wip in the blueprint whiteboard? | 16:22 |
| jaosorior | EmilienM: will do | 16:22 |
| EmilienM | jaosorior: thanks. I'll see if we keep it for ocata or postpone it. | 16:23 |
| *** almondjoy has joined #tripleo | 16:23 | |
| jaosorior | EmilienM: we're working very hard to keep it in schedule, without this the TLS-everywhere work is too hard to set up to consider it production-ready | 16:23 |
| alee | EmilienM, I've been doing installs from tripleo quickstart with the patches I have for awhile now. | 16:24 |
| EmilienM | jaosorior: I know you're working hard :-) you don't need to convaince me | 16:24 |
| *** jkilpatr_ has joined #tripleo | 16:24 | |
| EmilienM | alee: I know it can work on your local dev environment. My concern is to make it work in production and productized in tripleo on time | 16:24 |
| alee | EmilienM, there are a few dependencies - but we can enumerate them | 16:24 |
| EmilienM | alee: please do in the whiteboard. | 16:24 |
| EmilienM | alee: again, schedule is very short and we have 44 blueprints postponed from ocata-1 :) | 16:25 |
| alee | EmilienM, understood -- just keep in mind as Oz said, TLS is very tedious to set up without novajoin | 16:26 |
| EmilienM | alee: that's not the discussion here, I know the feature it's important | 16:26 |
| EmilienM | alee: I have no magical power to extend the ocata cycle. It's short and that's it | 16:26 |
| alee | EmilienM, but we'll enumerate the dependencies so you can see whats needed | 16:26 |
| EmilienM | alee: just keep in mind that we won't push for last-minutes features after ocata-3 | 16:27 |
| *** jkilpatr has quit IRC | 16:27 | |
| EmilienM | we tried that during newton and some people here can tell it was not a good idea :-) | 16:27 |
| alee | EmilienM, I think we're close -- but I'll let you make that determination | 16:28 |
| alee | I understand things are tight based on holidays etc. | 16:28 |
| EmilienM | shardy: I see some overlap between https://review.openstack.org/#/c/372336/ and https://review.openstack.org/#/c/393281/ - wdyt? | 16:28 |
| EmilienM | alee: we don't have the same definition of "close" :-) | 16:29 |
| alee | :) | 16:29 |
| EmilienM | alee: as long as I don't see all patches merged and CI testing this feature, it's not close to me, it's in good shape | 16:29 |
| shardy | EmilienM: They are different I think - one is about doing validation between each configuration step, the other is about triggering the tripleo-validations ansible things via mistral | 16:30 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: TEST: Enable undercloud ssl in ha job https://review.openstack.org/411404 | 16:30 |
| *** jaosorior has quit IRC | 16:30 | |
| EmilienM | shardy: yeah but I thought we could use the same tooling, and avoid duplication here | 16:30 |
| alee | EmilienM, ok - we'll detail where we are - what patches need to merge etc. and go from there. | 16:31 |
| EmilienM | cool thx | 16:31 |
| *** jaosorior has joined #tripleo | 16:31 | |
| *** pkovar has joined #tripleo | 16:31 | |
| shardy | EmilienM: Yeah in some future architecture perhaps we could, e.g it's likely all the validations will be ansible, just triggered differently | 16:31 |
| shardy | but right now we don't configure things via mistral, so they can't be driven the same way | 16:32 |
| shardy | EmilienM: I'll give some thought to how we might combine them in future, but for now I'd say keep them separate | 16:33 |
| EmilienM | shardy: lgtm | 16:33 |
| *** rcernin has quit IRC | 16:34 | |
| EmilienM | trozet: any progress on congress & tacker? I don't see much updates in the blueprint | 16:34 |
| EmilienM | trown|mtg: I see https://review.openstack.org/#/c/330050/ for congress but nothing for tacker. Are we still on track for ocata-3? should we postpone the two or one into pike? | 16:35 |
| EmilienM | trozet: ^ | 16:35 |
| EmilienM | shardy: I have some doubts about progress on https://blueprints.launchpad.net/tripleo/+spec/override-network-name | 16:38 |
| *** jaosorior has quit IRC | 16:39 | |
| derekh | EmilienM: sshnaidm panda bnemec: see my most recent comment on https://bugs.launchpad.net/tripleo/+bug/1649742 , that about as much info as I've managed to get so far today | 16:39 |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged] | 16:39 |
| *** jaosorior has joined #tripleo | 16:40 | |
| bnemec | derekh: Oh, so this is 7.3 biting us again? | 16:40 |
| EmilienM | it's unclear to me why we only hit that on the ha & uodate jobs | 16:40 |
| *** ccamacho is now known as ccamacho|out | 16:41 | |
| derekh | bnemec: possibly | 16:41 |
| bnemec | EmilienM: I have a theory about that. | 16:41 |
| bnemec | I was looking at the firewall rules in host_info, and the nonha job has a lot more because it enables undercloud ssl. | 16:41 |
| derekh | we pass some timeout on the HA job maybe? | 16:41 |
| bnemec | So I pushed https://review.openstack.org/411404 to test it. | 16:41 |
| *** pcaruana has quit IRC | 16:42 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Add hook to generate metadata from service profiles https://review.openstack.org/411339 | 16:42 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Add metadata settings for needed kerberos principals https://review.openstack.org/411340 | 16:42 |
| jaosorior | bnemec: won't that fail? since ha uses net-iso while non-ha doesn't. So there will be openssl errors | 16:43 |
| bnemec | jaosorior: It's only the undercloud. That should work either way. | 16:43 |
| jaosorior | ah, right | 16:44 |
| jaosorior | yeah, then it should be fine | 16:44 |
| panda | and anyway, this is only to check if testeenv client gets control again | 16:44 |
| bnemec | Also true, although this would be a nice workaround that would unblock ovb ci until we can figure out what broke. | 16:45 |
| openstackgerrit | Pradeep Kilambi proposed openstack/instack-undercloud: Disable legacy ceilometer api https://review.openstack.org/409160 | 16:45 |
| openstackgerrit | Pradeep Kilambi proposed openstack/instack-undercloud: Add gnocchi support on undercloud https://review.openstack.org/392992 | 16:45 |
| EmilienM | akrivoka: how are we doing with https://blueprints.launchpad.net/tripleo/+spec/gui-plan-import-export ? | 16:46 |
| EmilienM | akrivoka: I think this one needs to be postponed to pike-1 right? | 16:46 |
| *** mcornea has quit IRC | 16:47 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Add metadata settings for needed kerberos principals https://review.openstack.org/411340 | 16:47 |
| *** leanderthal is now known as leanderthal|afk | 16:48 | |
| *** hewbrocca is now known as hewbrocca_afk | 16:49 | |
| *** b00tcat has quit IRC | 16:49 | |
| *** bvandenh has quit IRC | 16:49 | |
| *** jpich has quit IRC | 16:51 | |
| bnemec | I'm still seeing a lot of multinode jobs failing on deploy timeouts like this: http://logs.openstack.org/67/405567/19/check/gate-tripleo-ci-centos-7-nonha-multinode/4c9e57b/console.html#_2016-12-15_16_29_29_960830 | 16:51 |
| *** fragatina has joined #tripleo | 16:53 | |
| *** fzdarsky is now known as fzdarsky|afk | 16:53 | |
| *** b00tcat has joined #tripleo | 16:54 | |
| EmilienM | bnemec: me too, and I failed to find why | 16:54 |
| *** rcernin has joined #tripleo | 16:54 | |
| EmilienM | bfournie: are we still on track for https://blueprints.launchpad.net/tripleo/+spec/tripleo-lldp-validation ? | 16:55 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Nothing to see here https://review.openstack.org/111011 | 16:55 |
| pradk | could someone please review this again https://review.openstack.org/#/c/409160/2 .. just rebased (already had 2 +2's) | 16:57 |
| *** fragatina has quit IRC | 16:57 | |
| *** abehl has quit IRC | 16:58 | |
| bfournie | EmilienM: I'd like to push this pike. The lldp plugin and cli dependencies are slower getting through review and some discussion came up in UI meetings two weeks ago in which lldp reporting may be used for building selection of networks for templates instead of validation, so there may need to be some tweaking done here | 16:59 |
| *** limao has quit IRC | 17:00 | |
| EmilienM | bfournie: ack | 17:01 |
| *** ohamada has quit IRC | 17:01 | |
| *** bvandenh has joined #tripleo | 17:04 | |
| therve | bnemec, EmilienM: looks like a connectivity issue on the node: http://logs.openstack.org/67/405567/19/check/gate-tripleo-ci-centos-7-nonha-multinode/4c9e57b/logs/subnode-2/var/log/messages | 17:05 |
| *** rbowen has left #tripleo | 17:06 | |
| *** jd__ has quit IRC | 17:08 | |
| *** jd__ has joined #tripleo | 17:09 | |
| *** ooolpbot has joined #tripleo | 17:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649742 | 17:10 |
| *** ooolpbot has quit IRC | 17:10 | |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged] | 17:10 |
| openstackgerrit | Brent Eagles proposed openstack/puppet-tripleo: Add base profile for Octavia services https://review.openstack.org/351898 | 17:11 |
| openstackgerrit | Brent Eagles proposed openstack/puppet-tripleo: Add initial profiles for rest of Octavia services https://review.openstack.org/411422 | 17:11 |
| *** yprokule has quit IRC | 17:12 | |
| *** zoli|wfh is now known as zoli|gone | 17:14 | |
| *** jkilpatr_ has quit IRC | 17:15 | |
| *** aufi has quit IRC | 17:17 | |
| *** rhallisey has quit IRC | 17:19 | |
| *** rbrady is now known as rbrady-afk | 17:19 | |
| *** jpena is now known as jpena|away | 17:21 | |
| *** [1]cdearborn has quit IRC | 17:21 | |
| *** trown|mtg is now known as trown | 17:22 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Noop the updates and ha jobs https://review.openstack.org/411424 | 17:23 |
| bnemec | derekh: EmilienM: ^We've got a 5 hour queue on ovb jobs right now, which sucks for testing the patches intended to fix the problem. | 17:24 |
| bnemec | I'm thinking we run them selectively on the patches related to fixing the problem and fail fast for everything else. | 17:25 |
| EmilienM | bnemec: can we kill them all and resend our patch that try to fix it? | 17:25 |
| bnemec | EmilienM: I'm not sure of a safe way to kill the running jobs. | 17:25 |
| *** spredzy has quit IRC | 17:26 | |
| bnemec | Even killing the zuul job doesn't actually free up the testenv, so we'd probably just start hitting our quota limits. | 17:26 |
| derekh | bnemec: your thinking we exit 1 at the top of toci_gate_test.sh ? | 17:28 |
| *** lblanchard1 has joined #tripleo | 17:28 | |
| bnemec | derekh: Yeah, that's basically what the patch does. | 17:28 |
| *** jkilpatr has joined #tripleo | 17:28 | |
| derekh | bnemec: opp, /me didn't see the patch | 17:28 |
| bnemec | So we never start the testenv-client and it never hangs. | 17:28 |
| derekh | bnemec: lgtm | 17:29 |
| *** lblanchard has quit IRC | 17:30 | |
| derekh | Could iptables6 be effecting ipv4 connections ? it fails to start on the failing jobs http://paste.openstack.org/show/592524/ | 17:31 |
| derekh | it started fine on the last job I found that passed | 17:31 |
| derekh | bnemec: EmilienM panda sshnaidm ^ | 17:31 |
| * derekh has gotta run | 17:32 | |
| *** derekh has quit IRC | 17:33 | |
| sshnaidm | derekh, it fails consistently last days, also as ip4tables too | 17:33 |
| *** jbadiapa has joined #tripleo | 17:33 | |
| bnemec | That seems concerning. | 17:33 |
| sshnaidm | but ip4tables is restarted later and works, | 17:34 |
| panda | mmmh | 17:34 |
| sshnaidm | but I didn't check ip6tables though.. | 17:34 |
| panda | I don't think ip6tables is going to affect ipv4 connection, but if ip6tables fails, maybe iptables has issues too | 17:34 |
| *** yamahata has joined #tripleo | 17:35 | |
| *** arxcruz has quit IRC | 17:35 | |
| bnemec | It's so weird that deployments work though. | 17:35 |
| bnemec | If the firewall had failed closed it should be breaking everything in the environment. | 17:36 |
| *** dtantsur|brb is now known as dtantsur | 17:36 | |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder: Recreate initramfs within loopback image https://review.openstack.org/411130 | 17:37 |
| *** jpena|away is now known as jpena|off | 17:38 | |
| *** thrash is now known as thrash|biab | 17:40 | |
| sshnaidm | there are job without ip6tables failure, but still killed, you can check on status page in gate-tripleo-ci-centos-7-ovb-nonha section: http://status-tripleoci.rhcloud.com/ | 17:40 |
| sshnaidm | I meant gate-tripleo-ci-centos-7-ovb-ha section | 17:41 |
| *** trown is now known as trown|lunch | 17:43 | |
| *** panda is now known as panda|bbl | 17:44 | |
| *** jaosorior has quit IRC | 17:44 | |
| *** milan has quit IRC | 17:45 | |
| *** [1]cdearborn has joined #tripleo | 17:47 | |
| shardy | EmilienM: Yeah you can defer https://blueprints.launchpad.net/tripleo/+spec/override-network-name to pike - it's probably not that much work, but I can't be sure I'll find time to work on it | 17:48 |
| shardy | I guess if the patches are ready we can move it back to ocata-3, but I don't think it's high priority at this point | 17:49 |
| EmilienM | shardy: ack | 17:50 |
| openstackgerrit | Steven Hardy proposed openstack-infra/tripleo-ci: Implement major upgrade for Newton to Ocata https://review.openstack.org/404831 | 17:54 |
| *** lucasagomes is now known as lucas-afk | 17:54 | |
| *** lmiccini has quit IRC | 17:55 | |
| dtantsur | folks, do we even enroll nodes in the -undercloud job? | 17:56 |
| EmilienM | dtantsur: YOU§ | 17:57 |
| EmilienM | I have questions about your blueprints, stay here :P | 17:57 |
| openstackgerrit | Lars Kellogg-Stedman proposed openstack/tripleo-puppet-elements: opstools: ensure that collectd is available on overcloud images https://review.openstack.org/409192 | 17:57 |
| *** florianf has quit IRC | 17:58 | |
| * dtantsur sloooowly walks away | 17:58 | |
| * EmilienM runs and closes the door with a key and eats the key | 17:58 | |
| EmilienM | https://blueprints.launchpad.net/tripleo/+spec/raid-workflow | 17:58 |
| EmilienM | https://blueprints.launchpad.net/tripleo/+spec/undercloud-boot-from-volume | 17:59 |
| EmilienM | are we on track for ocata-3? or move it to pike-1? | 17:59 |
| dtantsur | EmilienM, we're still on track with both. RAID one is close to completion, I just need a bit more time. Volume one still looks good enough with Cinder undercloud patch merged today. | 18:00 |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Create ubuntu/fedora test for pip-and-virtualenv https://review.openstack.org/407769 | 18:00 |
| EmilienM | dtantsur: excellent sir | 18:00 |
| * EmilienM opens the door magically | 18:00 | |
| dtantsur | EmilienM, my question about -undercloud job was actually related to https://blueprints.launchpad.net/tripleo/+spec/re-enable-cleaning. It seems done code-wise, but I need some victim job to enable cleaning there. | 18:00 |
| dtantsur | I don't want to bring more potential for transient failures to ha and nonha jobs though | 18:01 |
| *** ctayal has joined #tripleo | 18:01 | |
| dtantsur | so I thought if we enroll nodes in the undercloud job, it can be a candidate... but I suspect we don't | 18:01 |
| *** ctayal has quit IRC | 18:01 | |
| *** ctayal has joined #tripleo | 18:02 | |
| EmilienM | dtantsur: can you set a priority to both please? | 18:02 |
| * EmilienM brb lunch | 18:02 | |
| *** fragatina has joined #tripleo | 18:02 | |
| dtantsur | EmilienM, for me, volume is high, raid and cleaning are low. I don't have ACL to change priorities in tripleo. | 18:02 |
| EmilienM | dtantsur: let me fix that | 18:03 |
| *** arxcruz has joined #tripleo | 18:03 | |
| dtantsur | thanks | 18:03 |
| *** fragatina has quit IRC | 18:03 | |
| EmilienM | dtantsur: weird, I don't see why it's blocking | 18:04 |
| EmilienM | I'll look after my lunch break | 18:04 |
| *** fragatina has joined #tripleo | 18:04 | |
| dtantsur | maybe I'm not on some team.. dunno | 18:04 |
| *** jbadiapa has quit IRC | 18:04 | |
| *** rhallisey has joined #tripleo | 18:04 | |
| *** dtantsur is now known as dtantsur|afk | 18:05 | |
| *** pkovar has quit IRC | 18:07 | |
| *** pkovar has joined #tripleo | 18:08 | |
| *** pkovar has quit IRC | 18:08 | |
| *** bvandenh has quit IRC | 18:08 | |
| shardy | /quit/quit | 18:09 |
| shardy | oops | 18:09 |
| mwhahaha | :o | 18:09 |
| shardy | have a good evening all | 18:09 |
| bnemec | dtantsur|afk: We don't enroll nodes in the undercloud job. It's a standalone vm, so there aren't any nodes for it to manage. | 18:09 |
| bnemec | o/ shardy | 18:09 |
| *** shardy has quit IRC | 18:10 | |
| *** ooolpbot has joined #tripleo | 18:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649742 | 18:10 |
| *** ooolpbot has quit IRC | 18:10 | |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged] | 18:10 |
| bnemec | Although we could conceivably set up a pyghmi fakebmc on the node and then we could register something at least. | 18:10 |
| *** athomas has quit IRC | 18:12 | |
| bnemec | EmilienM: It would be good to merge https://review.openstack.org/#/c/411424/ asap so we can start clearing the queue. | 18:13 |
| *** chlong has joined #tripleo | 18:14 | |
| EmilienM | bnemec: done | 18:14 |
| bnemec | EmilienM: Thanks | 18:14 |
| * bnemec crosses his fingers for the multinode gate jobs | 18:15 | |
| mwhahaha | EmilienM: do we have an alternative way of rechecking just the ovb jobs? the ML mail about recheck impact made me wonder how many of those rechecks were because ovb jobs were flakey but resulted in a full recheck | 18:21 |
| openstackgerrit | Leif Madsen proposed openstack/tripleo-quickstart: Allow basic install to work on Fedora https://review.openstack.org/411442 | 18:22 |
| *** rasca has quit IRC | 18:22 | |
| *** amoralej is now known as amoralej|off | 18:27 | |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates: Deployed server bootstrap via Heat https://review.openstack.org/411445 | 18:29 |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates: Deployed server bootstrap via Heat https://review.openstack.org/411445 | 18:31 |
| EmilienM | mwhahaha: we don't until now | 18:31 |
| mwhahaha | should probably have one | 18:31 |
| mwhahaha | especially if we are not merging stuff if ovb is red | 18:32 |
| EmilienM | mwhahaha: we might need to investigate in project-config. Since tripleo ovb jobs are in a different pipeline it might be possible, let me look | 18:35 |
| EmilienM | mwhahaha: yeah I think we can | 18:36 |
| EmilienM | mwhahaha: let me propose it | 18:36 |
| mwhahaha | We should be able to | 18:37 |
| *** rlandy has joined #tripleo | 18:37 | |
| *** rbrady-afk is now known as rbrady | 18:37 | |
| openstackgerrit | Leif Madsen proposed openstack/tripleo-quickstart: Allow basic install to work on Fedora https://review.openstack.org/411442 | 18:39 |
| *** bana_k has joined #tripleo | 18:39 | |
| EmilienM | mwhahaha: https://review.openstack.org/411450 | 18:40 |
| *** trown|lunch is now known as trown | 18:44 | |
| *** gfidente has quit IRC | 18:44 | |
| *** thrash|biab is now known as thrash | 18:44 | |
| openstackgerrit | Tim Rozet proposed openstack/puppet-tripleo: Corrects default for glance API service https://review.openstack.org/411459 | 18:52 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Re-enable graphite metrics https://review.openstack.org/411460 | 18:52 |
| *** dprince has quit IRC | 18:57 | |
| *** dprince has joined #tripleo | 18:58 | |
| openstackgerrit | Dan Prince proposed openstack/python-tripleoclient: Deploy the undercloud with Heat https://review.openstack.org/351351 | 18:59 |
| *** rlandy is now known as rlandy|mtg | 19:00 | |
| *** dsneddon has joined #tripleo | 19:01 | |
| ansiwen | my build still fails because puppet-ec2api is not installed. it is included rdo.yml, but it is not added to the PROJECTS variable. So where do I have to add it so it is installed? See here: http://logs.openstack.org/67/406167/2/check-tripleo/gate-tripleo-ci-centos-7-ovb-nonha/a070ca6/console.html#_2016-12-15_02_47_48_093616 | 19:03 |
| bnemec | ansiwen: You may need to add it to this list: http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/tripleo.yaml#n159 | 19:09 |
| *** ooolpbot has joined #tripleo | 19:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649742 | 19:10 |
| *** ooolpbot has quit IRC | 19:10 | |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged] | 19:10 |
| bnemec | Although I've never been entirely clear on how that works. | 19:10 |
| openstackgerrit | Merged openstack/diskimage-builder: Recreate initramfs within loopback image https://review.openstack.org/411130 | 19:10 |
| *** fragatina has quit IRC | 19:11 | |
| ansiwen | bnemec: ok, thanks... but that's weird. I mean, when the undercloud is installed, there must be some mechanism independent of the CI infra to install the required puppet packages. so is this mechanism not used in CI? I'm confused... | 19:13 |
| *** ctayal has quit IRC | 19:14 | |
| ansiwen | bnemec: where does this list of packages come from? see: http://logs.openstack.org/67/406167/2/check-tripleo/gate-tripleo-ci-centos-7-ovb-nonha/a070ca6/console.html#_2016-12-15_03_07_05_966151 | 19:14 |
| *** chem has quit IRC | 19:14 | |
| sshnaidm | bnemec, it worked: https://review.openstack.org/#/c/411353 | 19:15 |
| bnemec | ansiwen: Ah, then it probably needs to be added to the opm dep list: https://review.rdoproject.org/r/gitweb?p=openstack/openstack-puppet-modules-distgit.git;a=blob;f=openstack-puppet-modules.spec;h=4df642ca2734d20fa7f98a77a81ef882f266036b;hb=HEAD | 19:16 |
| *** [1]cdearborn has quit IRC | 19:17 | |
| ansiwen | bnemec: aha! thanks! | 19:17 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart: make quickstart-extras-requirements.txt a default requirements file https://review.openstack.org/410757 | 19:17 |
| *** dprince has quit IRC | 19:18 | |
| *** [1]cdearborn has joined #tripleo | 19:18 | |
| bnemec | sshnaidm: Have we figured out if that's going to break testenv cleanup? | 19:19 |
| *** dprince has joined #tripleo | 19:20 | |
| trozet | EmilienM: how long do I have till Ocata-3? | 19:22 |
| trozet | dsneddon: hi | 19:27 |
| weshay | EmilienM, ping.. there are periodic versions of the puppet jobs right? e.g. gate-puppet-openstack-integration-4-scenario001-tempest-centos-7/ /me looking for logs | 19:27 |
| *** ctayal has joined #tripleo | 19:27 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart: make quickstart-extras-requirements.txt a default requirements file https://review.openstack.org/410757 | 19:29 |
| *** ctayal has quit IRC | 19:33 | |
| openstackgerrit | Ben Kero proposed openstack/tripleo-quickstart-extras: Add generic undercloud setup role https://review.openstack.org/406324 | 19:33 |
| *** rbrady is now known as rbrady-afk | 19:34 | |
| *** sudipto_ has quit IRC | 19:36 | |
| *** sudipto has quit IRC | 19:36 | |
| *** dprince has quit IRC | 19:38 | |
| *** ctayal has joined #tripleo | 19:39 | |
| *** dsariel has joined #tripleo | 19:39 | |
| *** sudipto has joined #tripleo | 19:41 | |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates: Deployed server bootstrap via Heat https://review.openstack.org/411445 | 19:42 |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates: Set rabbitmq package_provider to yum https://review.openstack.org/411471 | 19:42 |
| openstackgerrit | Pradeep Kilambi proposed openstack/instack-undercloud: Add gnocchi support on undercloud https://review.openstack.org/392992 | 19:45 |
| openstackgerrit | Pradeep Kilambi proposed openstack/instack-undercloud: Disable legacy ceilometer api https://review.openstack.org/409160 | 19:46 |
| *** sudipto has quit IRC | 19:49 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart: make quickstart-extras-requirements.txt a default requirements file https://review.openstack.org/410757 | 19:51 |
| *** nyechiel has quit IRC | 19:53 | |
| *** nyechiel has joined #tripleo | 19:53 | |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Add undercloud roles data and environment https://review.openstack.org/352037 | 19:55 |
| *** rlandy|mtg is now known as rlandy | 19:56 | |
| *** dsneddon has quit IRC | 20:00 | |
| *** dsneddon has joined #tripleo | 20:01 | |
| dsneddon | trozet, Hello | 20:01 |
| trozet | dsneddon: hey, so I filed a bug a little while ago to change the default glance backend to be file rather than swift | 20:02 |
| dsneddon | trozet, OK, got the bug link? | 20:02 |
| EmilienM | trozet: https://releases.openstack.org/ocata/schedule.html | 20:03 |
| EmilienM | weshay: periodic jobs for puppet? no | 20:03 |
| EmilienM | weshay: we have one, for updating our constraints though | 20:03 |
| trozet | dsneddon: but now I am trying to figure out how this will work for HA, because it would need to create NFS by default and I only see a way to mount to an NFS with puppet-tripleo/manifests/glance/nfs_mount.pp | 20:03 |
| trozet | dsneddon: https://bugs.launchpad.net/tripleo/+bug/1650348 | 20:04 |
| openstack | Launchpad bug 1650348 in tripleo "Glance API service defaults to Swift storage" [Undecided,In progress] - Assigned to Tim Rozet (trozet) | 20:04 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart: make quickstart-extras-requirements.txt a default requirements file https://review.openstack.org/410757 | 20:04 |
| trozet | EmilienM: radez was going to do the Congress part and I was going to work on tacker. I think I can get tacker done by ocata-3, just need to get started on it | 20:04 |
| EmilienM | trozet: well that's the problem | 20:05 |
| EmilienM | trozet: blueprints not started by end of ocata-2 won't like be done by ocata-3. Have you seen the short schedule? | 20:05 |
| trozet | EmilienM: we already implemented for our THT fork in mitaka, I just need to port it to a composable service | 20:05 |
| EmilienM | trozet: I haven't seen any progress until now on tacker until now | 20:05 |
| trown | EmilienM: is it a known issue that multinode job passes in check then fails in gate? or am I just really unlucky on https://review.openstack.org/#/c/409809/ | 20:06 |
| EmilienM | I have serious doubts of finishing tacker and congress on time for end of january tbh | 20:06 |
| trown | EmilienM: that patch has passed in check only to fail in gate 5 times now | 20:06 |
| EmilienM | trown: :( | 20:06 |
| trozet | EmilienM: well whats wrong with leaving it for now, and then update it later...like beginning of Jan? | 20:07 |
| EmilienM | trozet: have you seen my email about release status on openstack-dev? | 20:07 |
| *** panda|bbl is now known as panda | 20:08 | |
| EmilienM | trozet: we won't push features at the end of january anymore, which means doing congress and tacker in less than one month is unrealistic, specially regarding holidays, etc | 20:08 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart: make quickstart-extras-requirements.txt a default requirements file https://review.openstack.org/410757 | 20:08 |
| trozet | EmilienM: OK | 20:08 |
| EmilienM | weshay: what's your specific question for? what is the goal? | 20:08 |
| trozet | EmilienM: then I guess there is no choice but to move it? | 20:09 |
| EmilienM | trozet: well, I'm asking you so you have the choice | 20:09 |
| EmilienM | trozet: I just want to let you know that if by end of january tack and watcher are not merged and fully tested in CI, any patch will be -2 until we open Pike development. | 20:09 |
| *** ooolpbot has joined #tripleo | 20:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649742 | 20:10 |
| *** ooolpbot has quit IRC | 20:10 | |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged] | 20:10 |
| weshay | EmilienM, to get the set of puppet and packstack upstream tests running on along w/ the tripleo-periodic jobs on the same hash | 20:10 |
| trozet | EmilienM: OK, I will see if I can get it working before the break and post patches | 20:10 |
| weshay | meh.. maybe it's just best done in the set of rdo jobs we have | 20:10 |
| EmilienM | weshay: we don't have upstream periodic jobs for promotions. Ask dmsimard, we have RDO jobs in centos CI | 20:11 |
| trozet | dsneddon: is there anything that will configure each controller as an NFS server so you can use glance HA with NFS default out of the box? | 20:11 |
| weshay | ya.. k | 20:11 |
| dsneddon | trozet, We don't have that today. | 20:11 |
| dsneddon | trozet, I mean, there actually probably are Puppet and Ansible scripts to do that, but they aren't integrated into RDO | 20:12 |
| dsneddon | trozet, er, tripleo, I mean | 20:12 |
| trozet | dsneddon: so then that means that file backend cannot be the default for HA glance api | 20:12 |
| trown | EmilienM: wrt your convo with lucas about virtualbmc work (just saw in scrollback looking for something else) it is only relevant for virt-setup | 20:12 |
| trown | EmilienM: and the only CI we have for virt-setup is quickstart | 20:13 |
| EmilienM | trozet: we have 47 blueprints for Ocata 3 and we finished 1 blueprint in ocata-2. Trust me, it's not easy to manage | 20:13 |
| dsneddon | trozet, But that sounds like a good case for a composable service. | 20:13 |
| trozet | dsneddon: so then what if default is file for nonHA and Swift is default for HA, or we incorporate the missing puppet like you say to create NFS | 20:13 |
| *** kjw3 has joined #tripleo | 20:13 | |
| EmilienM | trown: oh, ok... thanks | 20:13 |
| trown | EmilienM: it replaces pxe_ssh | 20:13 |
| trozet | dsneddon: so that is my dilemma...if a user does not include swift in his composable role with glance API, glance wont work currently | 20:14 |
| trozet | dsneddon: i feel like that is wrong, and the default should be file (the glance default), and not rely on another composable service being included by default | 20:15 |
| trozet | dsneddon: what do you think? | 20:15 |
| dsneddon | trozet, Yes, I agree with you, that's a good default model. | 20:16 |
| *** derekh has joined #tripleo | 20:17 | |
| trozet | dsneddon: so we can do that with a simple change for nonHA obviously, but HA we run into the problem where no NFS is created by default. So how do we solve that for now? | 20:17 |
| derekh | bnemec: EmilienM sshnaidm panda its not nice but if we want it this appears to have worked https://review.openstack.org/#/c/411353 | 20:17 |
| derekh | the testenv was clean up 20 minutes after the testenv client got killed http://paste.openstack.org/show/592541/ | 20:18 |
| EmilienM | derekh, bnemec: should we cancel https://review.openstack.org/#/c/411424/ ? | 20:18 |
| EmilienM | derekh: why 20 min? | 20:18 |
| dsneddon | trozet, Well, I think "for now" is probably requiring Swift with HA, but "real soon" could be an NFS composable service. | 20:18 |
| derekh | on a train at the moment with patching internet, just popping in to let ye know so ye can decide | 20:19 |
| trozet | dsneddon: OK that makes sense to me | 20:19 |
| EmilienM | derekh: i've +2, I'll let bnemec voting (approving one and block other) | 20:19 |
| EmilienM | derekh: I'm also afk for an errand now | 20:19 |
| derekh | EmilienM: let me see if if one of the many timeouts we have in the code | 20:19 |
| dsneddon | trozet, I haven't tried it, but does the Cinder service like RBD as a backend? | 20:20 |
| openstackgerrit | Lars Kellogg-Stedman proposed openstack/puppet-tripleo: add support for collectd https://review.openstack.org/411047 | 20:20 |
| trozet | dsneddon: so then my first thought is in environment/puppet-pacemaker.yaml, set parameter default for GlanceBackend to be swift, but then that doesnt work if someone passes storage_environment.yaml to say use ceph. I don't know which parameter default it will take | 20:20 |
| bnemec | derekh: EmilienM: Yeah, I'm still a little concerned that we may end up leaving testenvs sitting around after jobs complete. | 20:20 |
| bnemec | I think they eventually get cleaned up, but if they sit there for an extra hour we're going to have quota problems. | 20:20 |
| EmilienM | bnemec: yeah :( | 20:21 |
| derekh | bnemec: its certainly possible, I'm happy to hold onto the patch until we run out of other options | 20:21 |
| weshay | EmilienM, container ci is passing the ping test.. w/ https://review.openstack.org/#/c/395880 | 20:21 |
| trozet | dsneddon: we tested Cinder with hyperconverged ceph, seems to work OK | 20:22 |
| openstackgerrit | Lars Kellogg-Stedman proposed openstack/tripleo-heat-templates: add collectd composable service https://review.openstack.org/411048 | 20:22 |
| trozet | dsneddon: but you remind me of another funny thing. In the ceph-base.yaml, the heat parameter GlanceBackend is also defined, and it is defaulted to swift, which also makes no sense | 20:22 |
| EmilienM | weshay: where do you see it? | 20:22 |
| weshay | pm | 20:22 |
| *** eggmaste` has joined #tripleo | 20:23 | |
| *** eggmaster has quit IRC | 20:24 | |
| dsneddon | trozet, Here's how inheritance works: first, everything in overcloud-resource-registry.yaml is evaluated, to set the initial resource definitions. Then, overcloud.j2.yaml sets all the default parameters, which are available to any subsequent included resources. Then, files included on the command line get evaluated, and any parameters set in overcloud.j2.yaml are overridden. If you include multiple files on the deployment command-l | 20:24 |
| dsneddon | ine, then the last one included wins. | 20:24 |
| *** tzumainn has quit IRC | 20:24 | |
| *** tzumainn has joined #tripleo | 20:25 | |
| dsneddon | trozet, So if you set a default parameter in an environment file, that overrides the puppet-pacemaker.yaml. | 20:25 |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci: Postci tests trial and error https://review.openstack.org/411189 | 20:25 |
| trozet | dsneddon: but that is only the environment file is specified after the -e puppet-pacemaker.yaml, right? | 20:25 |
| dsneddon | trozet, But, if you set a top-level parameter (not a parameter default), then subsequent parameter_defaults on the same property won't override the top-level parameter. | 20:25 |
| dsneddon | trozet, Right, you want to include puppet-pacemaker.yaml first, so that environment files specified after can override those params. | 20:26 |
| trozet | dsneddon: yeah but GlanceBackend isnt a global overcloud parameter, so cant use parameter: | 20:26 |
| trozet | dsneddon: yeah so we don't want order to matter...so it doesnt seem liek the solution is to set the parameter default in puppet-pacemaker.yaml | 20:27 |
| *** ayoung has joined #tripleo | 20:27 | |
| trozet | dsneddon: there is a pacemaker/glance-api.yaml, but doesnt look like it is used anymore: https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/services/pacemaker/glance-api.yaml | 20:28 |
| dsneddon | trozet, What you may need to do is set a top-level parameter, not parameter_default, so it won't be overridden. | 20:29 |
| trozet | dsneddon: i guess i could fix that to override the parameter | 20:29 |
| trozet | dsneddon: right, that would work too | 20:30 |
| dsneddon | trozet, What's the exact parameter(s)? | 20:31 |
| *** shinobu|afk has joined #tripleo | 20:31 | |
| trozet | dsneddon: so create the top level parameter, override the parameter_default in puppet-pacemaker, then change env files to override parameter instead of parameter_default | 20:31 |
| trozet | dsneddon: GlanceBackend | 20:31 |
| trozet | dsneddon: i think your solution is the best one | 20:32 |
| derekh | bnemec: EmilienM sshnaidm the geard server has keepalives set with the following params tcp_keepidle=7200, tcp_keepintvl=75, tcp_keepcnt=9 | 20:32 |
| openstackgerrit | Ben Kero proposed openstack/tripleo-quickstart-extras: overcloud-deploy: add variable for hypervisor_wait https://review.openstack.org/404943 | 20:32 |
| derekh | so after 2hrs of a idle tcp connection, it sends up to 9 keepalives every 75 seconds | 20:33 |
| *** shinobu__ has quit IRC | 20:33 | |
| derekh | thats 131 minues, | 20:34 |
| derekh | the te got cleaned up almost exactly 131 minutes after it was created | 20:34 |
| dsneddon | trozet, I don't see that set in puppet-pacemaker.yaml. | 20:35 |
| trozet | dsneddon: it's not. I'm adding it | 20:35 |
| derekh | so as best I can see if force killed testenvs will be cleaned up 131 minutes after they were created | 20:35 |
| dsneddon | trozet, Ah, of course. | 20:35 |
| dsneddon | trozet, Anyway, you want to set it as a parameter_default in puppet-pacemaker.yaml if you want to be able to override it later. | 20:36 |
| trozet | dsneddon: exactly. thanks. I will look into figuring out how to add NFS as a composable service | 20:36 |
| *** kbyrne has quit IRC | 20:36 | |
| derekh | at least that what the evidence would suggest* | 20:36 |
| *** flepied has quit IRC | 20:36 | |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras: overcloud-deploy: add variable for hypervisor_wait https://review.openstack.org/404943 | 20:37 |
| dsneddon | trozet, I'm happy to help if you have questions about the composable service, although shardy is probably the subject matter expert. | 20:37 |
| trozet | dsneddon: i think i've got the hang of that now after doing the ODL one | 20:38 |
| *** shinobu|afk has quit IRC | 20:38 | |
| *** derekh has quit IRC | 20:41 | |
| trozet | dsneddon: so one thing, if I define a top-level parameter, don't I need to pass it to the role's properties in overcloud.j2.yaml? | 20:49 |
| trozet | dsneddon: here I mean https://github.com/openstack/tripleo-heat-templates/blob/master/overcloud.j2.yaml#L329 | 20:50 |
| dsneddon | trozet, Yeah, that's what I mean by a top-level parameter. | 20:51 |
| *** weshay is now known as weshay_bbiab | 20:51 | |
| dsneddon | trozet, Although you can define parameters in environment files, but those will actually be considered separate and limited in scope to the nested stack. | 20:51 |
| trozet | dsneddon: so if I put it there as a property for hte dynamic roles J2 stuff, if a role does not include glance-api (with the local GlanceBackend param) will that fail? | 20:51 |
| trown | larsks: bug for the ironic issue https://bugs.launchpad.net/tripleo/+bug/1650374 | 20:52 |
| openstack | Launchpad bug 1650374 in tripleo "[instack-undercloud] package upgrade should happen before service configuration" [Critical,Triaged] | 20:52 |
| larsks | trown, thanks! | 20:53 |
| *** flepied has joined #tripleo | 20:54 | |
| akrivoka | EmilienM: yes, most likely postpone to pike | 20:55 |
| trown | larsks: oddly we have dependencies explicitly setup in the other direction, seemingly intentionally, https://github.com/openstack/instack-undercloud/blob/master/elements/puppet-stack-config/puppet-stack-config.pp#L44 | 20:56 |
| larsks | trown, are those relevant? For this fix we would want a notification dependency on the ironic-conductor service, right? Or on all 'openstack' services. Or something like that. Right? | 20:58 |
| trown | larsks: hmm ya, that is making sure openstack packages are upgraded first? | 21:02 |
| larsks | Yeah. | 21:02 |
| dsneddon | trozet, I am pretty sure that if you don't include the Glance role, that property won't matter, since the parameter will never be referenced. | 21:03 |
| dsneddon | trozet, Just like we set all sorts of parameters for network isolation that get ignored if you don't include network-isolation.yaml. | 21:04 |
| trown | EmilienM: is there a way to reference all openstack services in puppet... seems they are not tagged with openstack | 21:04 |
| mwhahaha | trown: no there is not | 21:04 |
| mwhahaha | trown: re: 1650374, shouldn't that get fixed with dependency for the ironic package? rather than updating all the packages before openstack? | 21:05 |
| trown | mwhahaha: how so? (does seem like potential packaging issue just not sure what the fix would be) | 21:06 |
| *** Goneri has quit IRC | 21:07 | |
| mwhahaha | trown: not sure but it seems like an issue in some dependency where it's not properly ensureing a min version or something | 21:07 |
| trown | mwhahaha: oh, ya it is not a dependency issue in that sense... simply restarting the service fixes it... but it may be a systemd service file issue | 21:08 |
| mwhahaha | trown: that error didn't seem to be ironic specific but rather some python crypto | 21:08 |
| trown | I see ironic-conductor has "Restart=on-failure" and other openstack services look to have "Restart=always" | 21:08 |
| larsks | trown, that just impacts what systemd will do when the service exits. | 21:09 |
| trown | larsks: ah ok... found examples of both now actually too | 21:09 |
| *** egafford has quit IRC | 21:10 | |
| *** ooolpbot has joined #tripleo | 21:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649742 | 21:10 |
| *** ooolpbot has quit IRC | 21:10 | |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged] | 21:10 |
| mwhahaha | trown: my thought on why it was a packaging issue is because it sounds like either it's relying on some other dependency that's not properly being pulled in when ironic gets installed. the bug was for initial install right? | 21:10 |
| larsks | trown, also, is this actually a packaging problem? That is, should the ironic package either (a) require a min. version of the appropriate python module? Does conductor work *before* the upgrade? | 21:10 |
| trown | mwhahaha: no, that trace happens when importing nodes | 21:10 |
| larsks | Or (b) should the package take care of the restart? That gets tricky, I guess. | 21:10 |
| *** mhenkel has quit IRC | 21:11 | |
| trown | mwhahaha: so ironic-conductor is happily running but with some outdated stuff in memory | 21:11 |
| *** mhenkel has joined #tripleo | 21:11 | |
| mwhahaha | trown: yea that's what i meant by the packaging thing because it shouldn't be, it should have had everything it was going to need before. so it's some package getting udpated breaks it because its a soft dependency or something and that package is not properly backwards compatible | 21:12 |
| trozet | dsneddon: ok cool | 21:13 |
| mwhahaha | trown: but this was a fresh install on a 7.2 image with a 7.3 packages, then as part of the install we do a package update right? just trying to understand how the system was put together | 21:13 |
| *** jayg is now known as jayg|g0n3 | 21:13 | |
| trown | mwhahaha: ya, it is a quickstart image built on 7.2, but the base repos for CentOS are now 7.3, so pretty much every base package gets updated when doing the undercloud install | 21:14 |
| trown | mwhahaha: once we promote a 7.3 based image the issue will go away (until the next time it happens) | 21:14 |
| trown | mwhahaha: a similar issue cropped up again with ironic-conductor when dowstream imported newton: https://bugzilla.redhat.com/show_bug.cgi?id=1391602 ... so it does seem specific to ironic-conductor | 21:16 |
| openstack | bugzilla.redhat.com bug 1391602 in openstack-ironic "fail: openstack baremetal import --json instackenv.json, Exception registering nodes: No valid host was found. Reason: No conductor service registered which supports driver..." [High,Closed: worksforme] - Assigned to lmartins | 21:16 |
| trown | in that case though ironic-conductor was in a failed state post undercloud install | 21:16 |
| mwhahaha | trown: so that error is probably coming from python2-cryptography which i guess was 1.2.1 in 7.2 and is now 1.3.1 in 7.3 | 21:17 |
| trown | in this case it is running fine... just unable to actually handle pxe_ssh | 21:17 |
| trown | mwhahaha: indeed that is one of the packages that changed | 21:17 |
| mwhahaha | trown: it appears that 1.3.1 is not properly backwards compatible with 1.2.1 | 21:18 |
| * mwhahaha shrugs | 21:18 | |
| trown | trozet: is this what you were seeing too (that made you hack out the upgrade from Apex CI) https://bugs.launchpad.net/tripleo/+bug/1650374 | 21:18 |
| openstack | Launchpad bug 1650374 in tripleo "[instack-undercloud] package upgrade should happen before service configuration" [Critical,Triaged] | 21:18 |
| mwhahaha | other than doing a yum update prior to doing anything, not sure we should be accounting for this in tripleo | 21:18 |
| trozet | trown: it is the same error | 21:19 |
| trown | mwhahaha: wdyt of just adding a dependency on Service<| tag == 'ironic-service' |> -> Exec<| title == 'package-upgrade' |> | 21:19 |
| trozet | trown: although the reason for stopping package upgrade is bigger than just that error | 21:19 |
| *** panda is now known as panda|Zz | 21:20 | |
| trozet | trown: i really dont think yum update should be in the install at all | 21:20 |
| trown | trozet: ya, I agree (hence making it optional) :) | 21:20 |
| mwhahaha | trown: i think that's just going to end causing more problems | 21:20 |
| openstackgerrit | Brent Eagles proposed openstack/puppet-tripleo: Add base profile for Octavia services https://review.openstack.org/351898 | 21:20 |
| trozet | trown: yeah but it shoudl be off by default. If you produce some software that you validate with CI that it works, then when a user uses it, his software changes compared to what you validated against... | 21:20 |
| mwhahaha | this really seems like the base image needs to be the same point release as everything else | 21:20 |
| openstackgerrit | Brent Eagles proposed openstack/puppet-tripleo: Add initial profiles for rest of Octavia services https://review.openstack.org/411422 | 21:21 |
| mwhahaha | i would maybe we should add a check for that | 21:21 |
| mwhahaha | but we'll come up with 50 million work arounds every time a new one is releases | 21:21 |
| mwhahaha | it might just be better to say sorry you need to upgrade your node to A.B before installing | 21:22 |
| mwhahaha | and leave it at that | 21:22 |
| trown | mwhahaha: well it is tricky with the upgrade in instack-undercloud... because the base image can be 7.2, but as soon as you run upgrade it is 7.3 | 21:22 |
| mwhahaha | trown: that's what i'm talking about, that shouldn't happen | 21:22 |
| trown | ah... so run yum upgrade manually before running `openstack undercloud install` | 21:22 |
| mwhahaha | yea | 21:22 |
| trown | that seems reasonable to document actually | 21:23 |
| mwhahaha | we should add a check in the instack-undercloud install to make sure we're at least A.B or something | 21:23 |
| trown | though then the upgrade in instack-undercloud is a bit pointless | 21:23 |
| mwhahaha | not necessarily | 21:24 |
| trown | and without that, we wouldnt have this situation to begin with :P | 21:24 |
| mwhahaha | so maybe i'm confused, was this a fresh install or an upgrade | 21:24 |
| mwhahaha | I'm getting lost in what is being 'upgraded' | 21:24 |
| mwhahaha | i'm saying the OS version on a new install should be at least a min version of something | 21:24 |
| *** dsneddon has quit IRC | 21:25 | |
| mwhahaha | so the tripleo repo should say, this was built against 7.3 for example. min version to install on is 7.3 | 21:25 |
| mwhahaha | we do not allow install to run on 7.2, please upgrade to 7.3 first | 21:25 |
| mwhahaha | then install | 21:25 |
| trown | ya it is a fresh install... but on an image with packages pre-installed | 21:25 |
| mwhahaha | i really don't like the images with package pre-installed for this specific reason but that's for another day | 21:26 |
| trown | well, ya we are kind of getting the worst of both worlds by having some packages getting updated and some static | 21:26 |
| mwhahaha | for the undercloud, i think it should just be a blank os image | 21:27 |
| mwhahaha | the overcloud, it might make sense | 21:27 |
| trown | but ideally we validate some content and it doesnt change so that others could reproduce it or use it for tripleo based projects ... ie OPNFV Apex | 21:27 |
| trown | historically, there was not a single test day in RDO that was successful before we adopted the image approach, so I think validated content has been a big win... the upgrade during undercloud install does mess that up a bit though | 21:28 |
| mwhahaha | maybe we need to shuffle the order of things, might it make more sense to config repos, yum update, then install? | 21:29 |
| mwhahaha | it seems like we config repos, install, then yum update | 21:29 |
| trown | anywho... I think adding an upgrade in quickstart in the case an upgrade in the undercloud install will do it anyways will fix the current issue | 21:30 |
| mwhahaha | or is ironic-inspector already running before the installation and since it doesn't get anything new it's never restarted | 21:30 |
| trown | hmm... not sure... I dont think it should be running, but the alternative doesnt make a ton of sense either | 21:31 |
| trown | gotta run, will put some more in the bug later | 21:31 |
| trown | trozet: going to see yurtseven :) | 21:31 |
| mwhahaha | i'll take a look in a bit if i get some free cycles | 21:31 |
| trozet | mwhahaha: I think actually you want the opposite. You don't want a blank OS image with packages installed later | 21:31 |
| mwhahaha | trozet: why's that? | 21:32 |
| openstackgerrit | Steve Baker proposed openstack/tripleo-heat-templates: set -e in run-os-net-config.sh https://review.openstack.org/411509 | 21:32 |
| *** trown is now known as trown|outtypewww | 21:32 | |
| trozet | mwhahaha: that way you have control over a piece of undercloud software you know works | 21:32 |
| mwhahaha | trozet: shouldn't that be a static repo that has been validated? | 21:32 |
| mwhahaha | trozet: shouldn't the specific rdo repo be that? | 21:33 |
| trozet | mwhahaha: that is another way, but when you do yum update * in your install... | 21:33 |
| mwhahaha | trozet: So when I said yum update I meant base OS, not necessarily the RDO stuff as well | 21:33 |
| trozet | trown: oh is he "unbanned" tonight? | 21:33 |
| openstackgerrit | Steve Baker proposed openstack/tripleo-heat-templates: Don't run yum_update.sh inside docker https://review.openstack.org/411511 | 21:34 |
| mwhahaha | trozet: so you end up with a specific OS release -> specific RDO packages (which should be validated on the OS release) | 21:34 |
| trozet | mwhahaha: yeah but OOO/RDO cant control what gets updated in centos repos | 21:35 |
| trozet | mwhahaha: so then if centos repo like this last bug updates a package that we depend on, and it isnt compatible... | 21:35 |
| trozet | mwhahaha: when we deploy and overcloud image, we dont do a yum update on the image and connect to the internet right? | 21:36 |
| mwhahaha | trozet: yea makes sense. I think the issue here is that python2-cryptography is being updated later but that ironic relies on it is never restarted when it is updated | 21:36 |
| trozet | deploy an* | 21:36 |
| mwhahaha | trozet: depends on how you're doing yum update, could be offline repos or mirrors. the point was more to be able to have consistent artifacts for OS/RDO as part of the install since we're getting a partial update of some stuff at some point | 21:37 |
| trozet | mwhahaha: yeah I just think that if trown produces an undercloud qcow2 and overcloud qcow2, they should contain the same software when i deploy it 3 weeks from the day it was validated and passed CI | 21:38 |
| trozet | mwhahaha: otherwise the validation with CI really means nothing | 21:38 |
| mwhahaha | trozet: but that means that image is tied to a specific point in time OS dependency repo | 21:39 |
| mwhahaha | trozet: which is not the case in either scenario | 21:39 |
| trozet | mwhahaha: what do you mean it isnt hte case? | 21:40 |
| mwhahaha | centos 7.2 image got centos 7.3 after the fact | 21:40 |
| trozet | mwhahaha: right | 21:40 |
| trozet | mwhahaha: i'm saying thats how it should be, not how it is | 21:40 |
| mwhahaha | trozet: yea i'm agreeing :D | 21:41 |
| trozet | mwhahaha: oh, cool :) | 21:41 |
| mwhahaha | trozet: that's what i was kinda pointing at if you start with a known good base OS, when you configure the openstack repo for install, you get all the packages you need | 21:41 |
| mwhahaha | but that's not what happens since we're also relying on some of the base OS repo packages | 21:41 |
| mwhahaha | and in this case, python2-cryptography changed between 7.2 and 7.3 | 21:42 |
| mwhahaha | so it would be nice if the openstack repo/artifacts/whatever you want to call it knew what the min version of the base os was required for it | 21:42 |
| mwhahaha | giving you something similar to what we'd have in an image build | 21:43 |
| trozet | mwhahaha: oh you mean if you dont use the pre-build image? | 21:43 |
| mwhahaha | and even the prebuilt image, you would be able to know that the following update has to occur prior to installing openstack | 21:43 |
| *** jkilpatr has quit IRC | 21:44 | |
| mwhahaha | so if you had a 7.1 image, you go to run the install and it says you must update to 7.3 as this version has only been validated against 7.3 | 21:44 |
| trozet | mwhahaha: that woudln't happen because the image would have been built with 7.3 | 21:44 |
| trozet | mwhahaha: if you have the image, then nothing needs to be updated it should just work | 21:45 |
| mwhahaha | trozet: thats if your image had everything installed upfront which i don't think to be the case | 21:45 |
| trozet | mwhahaha: but if you just install undercloud on your host, and you need to pull packages, then it should install specific packages you need, and those should resolve dependencies, not just yum update all I think | 21:45 |
| *** dsneddon has joined #tripleo | 21:46 | |
| trozet | mwhahaha: it does, we install with no internet | 21:46 |
| *** fragatina has joined #tripleo | 21:46 | |
| *** iranzo has quit IRC | 21:46 | |
| mwhahaha | trozet: right but i'm not referring to that use case as that's a specific super controlled case | 21:46 |
| *** shinobu_ has quit IRC | 21:47 | |
| mwhahaha | trozet: i'm thinking more how I interact with the undercloud where I don't have an image for it. I have a basic centos VM that I then configure repos and install. which to me would be more like what someone might do at company XYZ looking to evaluate or start up a new cloud install | 21:48 |
| mwhahaha | trozet: There are folks who don't want tripleo doing the image build/install part which is where this specific issue will come up more often | 21:49 |
| bnemec | EmilienM: Good news! I think we can just turn on undercloud ssl to unblock the ha and updates jobs: http://logs.openstack.org/04/411404/1/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/b4765f8/console.html | 21:49 |
| *** almondjoy has quit IRC | 21:49 | |
| trozet | mwhahaha: yeah so in that case where you just have your own VM and want to install undercloud, it should install specific packages and not do yum update. To your point, if it tried to install some package that relied on the kernel updating, then it should warn you or something like you said about min reqs | 21:52 |
| *** shinobu_ has joined #tripleo | 21:52 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Enable undercloud ssl on all OVB jobs https://review.openstack.org/411514 | 21:57 |
| *** akrivoka has quit IRC | 21:59 | |
| *** fultonj has quit IRC | 21:59 | |
| *** jkilpatr has joined #tripleo | 22:01 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: DONT REVIEW: test https://review.openstack.org/410470 | 22:01 |
| *** ansmith has quit IRC | 22:01 | |
| sshnaidm | bnemec, any idea why ssl helps..? | 22:05 |
| bnemec | sshnaidm: The undercloud ssl config opens a bunch more ports in the firewall. That's my best guess at this point. | 22:06 |
| bnemec | It also explains why that didn't fix the mitaka jobs. Those had a much simpler undercloud ssl config that didn't open extra ports (IIRC). | 22:06 |
| sshnaidm | bnemec, maybe it's worth to look at dropped packets in syslog, I've seen a few iptables logs there, maybe some of them from geard server..? | 22:07 |
| sshnaidm | will look now.. | 22:08 |
| bnemec | I need to reboot for a system update. | 22:09 |
| bnemec | brb | 22:09 |
| *** ooolpbot has joined #tripleo | 22:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649742 | 22:10 |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged] | 22:10 |
| *** ooolpbot has quit IRC | 22:10 | |
| *** ayoung has quit IRC | 22:10 | |
| *** bnemec has quit IRC | 22:10 | |
| *** dsneddon has quit IRC | 22:15 | |
| *** bnemec has joined #tripleo | 22:16 | |
| *** lblanchard1 has quit IRC | 22:17 | |
| openstackgerrit | Steve Baker proposed openstack/tripleo-heat-templates: Use the local collector to bootstrap deployed servers https://review.openstack.org/405032 | 22:18 |
| openstackgerrit | Steve Baker proposed openstack/tripleo-heat-templates: docker: expose agent mounts as template param https://review.openstack.org/409579 | 22:18 |
| larsks | In THT master, KeyStoneFernetKey0 (and 1) are now required parameters? | 22:20 |
| EmilienM | bnemec: oh wow | 22:21 |
| EmilienM | bnemec: should we block https://review.openstack.org/#/c/411424/ ? | 22:21 |
| EmilienM | larsks: yes but generated by tripleoclient | 22:22 |
| EmilienM | you don't need to specify them | 22:22 |
| larsks | EmilienM, I guess that means I need to update my tripleoclient... | 22:22 |
| EmilienM | larsks: for sure :) | 22:22 |
| bnemec | EmilienM: Probably. Once we get the workaround merged, most of the jobs will go back to finishing in a reasonable time and the queue should clear up by itself. | 22:22 |
| EmilienM | bnemec: sounds good. This week was terrible. | 22:23 |
| bnemec | EmilienM: Of course it was. I just got back from PTO. It's like a cosmic rule or something. ;-) | 22:24 |
| EmilienM | bnemec: i know the feeling :D | 22:24 |
| *** liverpooler has quit IRC | 22:25 | |
| *** dsneddon has joined #tripleo | 22:26 | |
| *** egafford has joined #tripleo | 22:31 | |
| *** liverpooler has joined #tripleo | 22:31 | |
| *** jcoufal has quit IRC | 22:34 | |
| *** egafford has quit IRC | 22:35 | |
| *** rhallisey has quit IRC | 22:35 | |
| *** dsavineau has left #tripleo | 22:35 | |
| *** sai is now known as sai-out | 22:36 | |
| *** saneax-_-|AFK is now known as saneax | 22:40 | |
| *** bfournie has quit IRC | 22:47 | |
| *** liverpooler has quit IRC | 22:47 | |
| *** saneax is now known as saneax-_-|AFK | 22:48 | |
| Ng | so I got some great hints earlier for making mistral notice new actions, but I'm also trying to hack on python-tripleoclient and even after blowing away openstackclient and tripleoclient and re-installing them both with setup.py, it's not seeing my added commands (which are in setup.cfg) | 22:50 |
| Ng | I'm sure I'm missing something simple and obvious | 22:50 |
| *** myoung is now known as myoung|afk | 22:50 | |
| *** nyechiel has quit IRC | 22:51 | |
| *** morazi has quit IRC | 22:53 | |
| *** jmelvin has quit IRC | 22:54 | |
| *** rlandy is now known as rlandy|bbl | 22:55 | |
| *** egafford has joined #tripleo | 22:55 | |
| *** flepied has quit IRC | 23:00 | |
| *** saneax-_-|AFK is now known as saneax | 23:03 | |
| *** ctayal has quit IRC | 23:05 | |
| *** ctayal has joined #tripleo | 23:06 | |
| *** ooolpbot has joined #tripleo | 23:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649742 | 23:10 |
| *** ooolpbot has quit IRC | 23:10 | |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha and ovb-updates" [Critical,Triaged] | 23:10 |
| EmilienM | bnemec: how do you want to proceed? | 23:15 |
| EmilienM | we merge your stuff and revert the noop later? | 23:15 |
| EmilienM | your stuff == ssl/undercloud | 23:16 |
| bnemec | The noop actually failed the gate anyway, so it hasn't gone in. I would propose we just merge the undercloud ssl instead. | 23:17 |
| bnemec | EmilienM: ^ | 23:17 |
| *** egafford has quit IRC | 23:18 | |
| EmilienM | bnemec: I agree | 23:25 |
| EmilienM | bnemec: have you seen it passing CI? | 23:25 |
| bnemec | EmilienM: It's passed all the voting multinode jobs, and looks like the updates job is going to timeout on the same kind of hang we're seeing elsewhere. | 23:26 |
| bnemec | And the same basic patch got the ha job working in my test commit. | 23:27 |
| bnemec | Also, the ovb jobs are completely hosed right now so it can't really make it any worse. :-) | 23:27 |
| EmilienM | bnemec: even if we +A now,i'm not sure it will move in gate directly | 23:27 |
| bnemec | EmilienM: Yeah, we'll have to wait for the multinode updates job to timeout first, but it should go in after that. | 23:28 |
| EmilienM | bnemec: I'm ok with that, really | 23:28 |
| *** ayoung has joined #tripleo | 23:29 | |
| *** ctayal has quit IRC | 23:32 | |
| EmilienM | bnemec: I need to leave for a couple of hours - again feel free to approve it if you see it passing. Otherwise I'll do it later in the evening. Maybe we can unblock CI for tomorrow, who know :) | 23:32 |
| bnemec | EmilienM: I already pulled the trigger. I left a big comment explaining my reasoning. :-) | 23:32 |
| *** ctayal has joined #tripleo | 23:33 | |
| *** pradk has quit IRC | 23:38 | |
| *** tzumainn has quit IRC | 23:50 | |
| *** [1]cdearborn has quit IRC | 23:53 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!