| *** gbarros has joined #tripleo | 00:05 | |
| *** rlandy|rover is now known as rlandy|rover|bbl | 00:06 | |
| *** ooolpbot has joined #tripleo | 00:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 00:10 |
|---|---|---|
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731063 | 00:10 |
| openstack | Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] | 00:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1734134 | 00:10 |
| *** ooolpbot has quit IRC | 00:10 | |
| openstack | Launchpad bug 1734134 in tripleo "Pike periodic promotion job multinode-1ctlr-featureset016 fail with error running docker 'gnocchi_db_sync' - rados.Rados.connect PermissionDeniedError: error connecting to the cluster" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 00:10 |
| *** bfournie has joined #tripleo | 00:20 | |
| openstackgerrit | K Jonathan Harker proposed openstack/diskimage-builder master: zypper: fix package removal https://review.openstack.org/517658 | 00:20 |
| openstackgerrit | K Jonathan Harker proposed openstack/diskimage-builder master: Install dpkg on yum-based systems in testing https://review.openstack.org/523251 | 00:21 |
| *** bfournie has quit IRC | 00:21 | |
| *** gbarros has quit IRC | 00:23 | |
| *** etingof has quit IRC | 00:25 | |
| *** etingof has joined #tripleo | 00:26 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: Run ovb-ha with minimal services https://review.openstack.org/522310 | 00:33 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: Run ovb-ha with minimal services https://review.openstack.org/522310 | 00:33 |
| *** jkilpatr has quit IRC | 00:36 | |
| *** gbarros has joined #tripleo | 00:36 | |
| *** absubram has joined #tripleo | 00:43 | |
| *** jkilpatr has joined #tripleo | 00:44 | |
| *** ipsecguy has quit IRC | 00:45 | |
| openstackgerrit | Clark Boylan proposed openstack/diskimage-builder master: Install dpkg on yum-based systems in testing https://review.openstack.org/523251 | 00:50 |
| *** jkilpatr has quit IRC | 00:50 | |
| *** gbarros has quit IRC | 00:52 | |
| *** jkilpatr has joined #tripleo | 01:02 | |
| *** ipsecguy has joined #tripleo | 01:05 | |
| *** jkilpatr has quit IRC | 01:09 | |
| *** ooolpbot has joined #tripleo | 01:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731063 | 01:10 |
| openstack | Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1734134 | 01:10 |
| *** ooolpbot has quit IRC | 01:10 | |
| openstack | Launchpad bug 1734134 in tripleo "Pike periodic promotion job multinode-1ctlr-featureset016 fail with error running docker 'gnocchi_db_sync' - rados.Rados.connect PermissionDeniedError: error connecting to the cluster" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 01:10 |
| *** jkilpatr has joined #tripleo | 01:10 | |
| *** liverpooler has joined #tripleo | 01:11 | |
| *** jlabarre has quit IRC | 01:29 | |
| *** jkilpatr has quit IRC | 01:38 | |
| *** tbonds has joined #tripleo | 01:41 | |
| *** absubram has quit IRC | 01:45 | |
| *** fragatin_ is now known as fragatina | 01:46 | |
| *** yolanda has quit IRC | 01:54 | |
| *** Goneri has quit IRC | 01:57 | |
| *** bfournie has joined #tripleo | 01:59 | |
| *** dmacpher has joined #tripleo | 02:05 | |
| *** ooolpbot has joined #tripleo | 02:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731063 | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1734134 | 02:10 |
| *** ooolpbot has quit IRC | 02:10 | |
| openstack | Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] | 02:10 |
| openstack | Launchpad bug 1734134 in tripleo "Pike periodic promotion job multinode-1ctlr-featureset016 fail with error running docker 'gnocchi_db_sync' - rados.Rados.connect PermissionDeniedError: error connecting to the cluster" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 02:10 |
| *** yolanda has joined #tripleo | 02:18 | |
| *** yolanda has quit IRC | 02:24 | |
| openstackgerrit | Merged openstack/paunch master: zuul: change OVB job layout https://review.openstack.org/522583 | 02:36 |
| openstackgerrit | Emilien Macchi proposed openstack/paunch stable/pike: zuul: change OVB job layout https://review.openstack.org/523280 | 02:39 |
| *** yolanda has joined #tripleo | 02:41 | |
| *** fzdarsky_ has joined #tripleo | 02:41 | |
| *** tbonds has quit IRC | 02:43 | |
| *** fzdarsky|afk has quit IRC | 02:43 | |
| *** ramishra has joined #tripleo | 02:45 | |
| *** yamahata has quit IRC | 02:53 | |
| *** rlandy|rover|bbl is now known as rlandy|rover | 02:57 | |
| *** fragatina has quit IRC | 02:59 | |
| *** fragatina has joined #tripleo | 03:00 | |
| *** fragatina has quit IRC | 03:04 | |
| *** ooolpbot has joined #tripleo | 03:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731063 | 03:10 |
| openstack | Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1734134 | 03:10 |
| *** ooolpbot has quit IRC | 03:10 | |
| openstack | Launchpad bug 1734134 in tripleo "Pike periodic promotion job multinode-1ctlr-featureset016 fail with error running docker 'gnocchi_db_sync' - rados.Rados.connect PermissionDeniedError: error connecting to the cluster" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 03:10 |
| *** kbyrne has quit IRC | 03:10 | |
| *** kbyrne has joined #tripleo | 03:11 | |
| openstackgerrit | Ian Wienand proposed openstack/python-tripleoclient master: Use qemu-img in bindep https://review.openstack.org/523285 | 03:15 |
| *** myoung is now known as myoung|afk | 03:32 | |
| *** links has joined #tripleo | 03:39 | |
| *** rlandy|rover has quit IRC | 03:41 | |
| EmilienM | ianw: you might want to see https://review.openstack.org/523167 | 03:45 |
| EmilienM | ianw: I'm fine with your fix if that works | 03:45 |
| ianw | EmilienM: I believe it will, since the func tests aren't installing that | 03:45 |
| ianw | sorry, we removed that to make it *easier* for tripleo to not have to strip repos | 03:46 |
| EmilienM | ianw: wfm,I'll abandon my patch once yours works | 03:46 |
| ianw | we do need those -ev packages to run devstack on centos, for example. but devstack goes ahead and sets up the repos | 03:46 |
| *** psahoo has joined #tripleo | 03:48 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: ci: add ovb-ha.yaml https://review.openstack.org/522306 | 03:55 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: Run ovb-ha with minimal services https://review.openstack.org/522310 | 03:58 |
| *** psahoo is now known as psahoo|bf | 04:02 | |
| *** gkadam has quit IRC | 04:07 | |
| *** psahoo|bf is now known as psahoo | 04:08 | |
| *** ykarel has joined #tripleo | 04:09 | |
| *** ooolpbot has joined #tripleo | 04:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731063 | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1734134 | 04:10 |
| *** ooolpbot has quit IRC | 04:10 | |
| openstack | Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] | 04:10 |
| openstack | Launchpad bug 1734134 in tripleo "Pike periodic promotion job multinode-1ctlr-featureset016 fail with error running docker 'gnocchi_db_sync' - rados.Rados.connect PermissionDeniedError: error connecting to the cluster" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 04:10 |
| *** ratailor has joined #tripleo | 04:10 | |
| *** pdeore has joined #tripleo | 04:13 | |
| *** udesale has joined #tripleo | 04:14 | |
| *** mdnadeem has joined #tripleo | 04:15 | |
| *** dpawar has joined #tripleo | 04:16 | |
| *** dpawar has quit IRC | 04:23 | |
| *** dbecker has quit IRC | 04:25 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Use EPEL for debootstrap on centos https://review.openstack.org/523251 | 04:26 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: Introduce fs035, ovb-ha-ipv6 https://review.openstack.org/522615 | 04:28 |
| openstackgerrit | Merged openstack/instack-undercloud master: Do not set dhcp_domain in Nova from overcloud_domain_name https://review.openstack.org/520152 | 04:28 |
| openstackgerrit | zenghui.shi proposed openstack/tripleo-heat-templates master: Add PTP composable service https://review.openstack.org/491317 | 04:28 |
| *** dmacpher has quit IRC | 04:29 | |
| *** liverpooler has quit IRC | 04:36 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: Run ovb-ha-ipv6 in check-tripleo https://review.openstack.org/523297 | 04:36 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: Run ovb-ha-ipv6 in check-tripleo https://review.openstack.org/523298 | 04:37 |
| *** psachin has joined #tripleo | 04:38 | |
| *** dbecker has joined #tripleo | 04:38 | |
| *** dmacpher has joined #tripleo | 04:42 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/newton: ceilometer: set event dispatcher to Gnocchi by default https://review.openstack.org/521132 | 04:49 |
| *** pgadiya has joined #tripleo | 04:51 | |
| *** pgadiya has quit IRC | 04:52 | |
| *** janki has joined #tripleo | 05:07 | |
| *** ooolpbot has joined #tripleo | 05:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731063 | 05:10 |
| openstack | Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1734134 | 05:10 |
| *** ooolpbot has quit IRC | 05:10 | |
| openstack | Launchpad bug 1734134 in tripleo "Pike periodic promotion job multinode-1ctlr-featureset016 fail with error running docker 'gnocchi_db_sync' - rados.Rados.connect PermissionDeniedError: error connecting to the cluster" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 05:10 |
| *** threestrands has quit IRC | 05:10 | |
| *** threestrands has joined #tripleo | 05:10 | |
| *** threestrands has quit IRC | 05:12 | |
| *** threestrands has joined #tripleo | 05:12 | |
| *** pgadiya has joined #tripleo | 05:13 | |
| *** shreshtha has joined #tripleo | 05:15 | |
| *** shreshtha is now known as shreshtha-afh | 05:16 | |
| *** shreshtha-afh is now known as shreshtha-wfh | 05:16 | |
| *** akane_ has joined #tripleo | 05:18 | |
| *** akane has joined #tripleo | 05:18 | |
| *** iranzo has joined #tripleo | 05:21 | |
| *** brault has quit IRC | 05:21 | |
| *** stendulker has joined #tripleo | 05:21 | |
| *** brault has joined #tripleo | 05:22 | |
| *** dpawar has joined #tripleo | 05:22 | |
| *** dmacpher has quit IRC | 05:26 | |
| Tengu | hello there! | 05:36 |
| *** skramaja has joined #tripleo | 05:39 | |
| *** abregman has joined #tripleo | 05:43 | |
| *** dmacpher has joined #tripleo | 05:43 | |
| *** dsariel has joined #tripleo | 05:44 | |
| *** jfrancoa has joined #tripleo | 05:47 | |
| *** cshastri has joined #tripleo | 05:53 | |
| *** jaganathan has joined #tripleo | 05:57 | |
| *** pdeore has quit IRC | 06:00 | |
| *** fragatina has joined #tripleo | 06:00 | |
| *** masco has joined #tripleo | 06:01 | |
| *** fragatina has quit IRC | 06:05 | |
| *** pcaruana has joined #tripleo | 06:06 | |
| *** pcaruana has quit IRC | 06:06 | |
| jaosorior | good morning! | 06:06 |
| *** ooolpbot has joined #tripleo | 06:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731063 | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1734134 | 06:10 |
| *** ooolpbot has quit IRC | 06:10 | |
| openstack | Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] | 06:10 |
| openstack | Launchpad bug 1734134 in tripleo "Pike periodic promotion job multinode-1ctlr-featureset016 fail with error running docker 'gnocchi_db_sync' - rados.Rados.connect PermissionDeniedError: error connecting to the cluster" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 06:10 |
| *** akane_ has quit IRC | 06:13 | |
| *** akane has quit IRC | 06:13 | |
| *** karthiks has joined #tripleo | 06:14 | |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-heat-templates stable/pike: Missing cherry-picking stop condition for swift services. https://review.openstack.org/522753 | 06:19 |
| *** fragatina has joined #tripleo | 06:21 | |
| *** absubram has joined #tripleo | 06:22 | |
| *** dmacpher has quit IRC | 06:26 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Use EPEL for debootstrap on centos https://review.openstack.org/523251 | 06:31 |
| *** absubram has quit IRC | 06:31 | |
| *** akane has joined #tripleo | 06:31 | |
| *** akane_ has joined #tripleo | 06:31 | |
| *** gfidente has joined #tripleo | 06:32 | |
| *** gfidente has quit IRC | 06:32 | |
| *** gfidente has joined #tripleo | 06:32 | |
| *** fragatina has quit IRC | 06:34 | |
| Tengu | how are you jaosorior ? today, I'm working at home \o/ | 06:34 |
| *** fragatina has joined #tripleo | 06:34 | |
| *** absubram has joined #tripleo | 06:37 | |
| *** fragatina has quit IRC | 06:38 | |
| *** nguyentrihai has quit IRC | 06:39 | |
| *** bkopilov has joined #tripleo | 06:42 | |
| jaosorior | Tengu: I'm alright. Slept like shit. But nothing coffee won't fix. | 06:42 |
| *** fragatina has joined #tripleo | 06:43 | |
| *** nguyentrihai has joined #tripleo | 06:45 | |
| *** jtomasek has joined #tripleo | 06:48 | |
| *** jtomasek has quit IRC | 06:48 | |
| *** jtomasek has joined #tripleo | 06:49 | |
| Tengu | hehe | 06:55 |
| Tengu | oh, btw. tea time. | 06:55 |
| zshi | morning all! | 06:57 |
| Tengu | hello zshi | 06:57 |
| zshi | hey Tengu | 06:58 |
| openstackgerrit | Attila Darazs proposed openstack/tripleo-quickstart-extras master: GATE CHECK for quickstart-extras https://review.openstack.org/472607 | 07:00 |
| ratailor | mandre, jaosorior, jistr EmilienM can you pl review, https://code.engineering.redhat.com/gerrit/#/q/owner:owalsh%2540redhat.com+status:open | 07:01 |
| Tengu | jbadiapa: I propose I abandon my EPEL wrapper and you update your review in order to use the "manage_repo => false" param for collectd class. That way everyone is happy, and we (openstack) still don't get EPEL repositories activated. We'll see if other puppet mods require epel class - if this is the case, we might consider my patch or something similar (poke gfidente ;)) | 07:03 |
| *** threestrands has quit IRC | 07:04 | |
| *** fragatina has quit IRC | 07:06 | |
| *** fragatina has joined #tripleo | 07:07 | |
| *** dmacpher has joined #tripleo | 07:09 | |
| *** marios has joined #tripleo | 07:09 | |
| openstackgerrit | Jose Luis Franco proposed openstack-infra/tripleo-ci master: WIP: Upgrade UC and OC using tripleo-upgrade role https://review.openstack.org/515643 | 07:09 |
| *** ooolpbot has joined #tripleo | 07:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731063 | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1734134 | 07:10 |
| *** ooolpbot has quit IRC | 07:10 | |
| openstack | Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] | 07:10 |
| openstack | Launchpad bug 1734134 in tripleo "Pike periodic promotion job multinode-1ctlr-featureset016 fail with error running docker 'gnocchi_db_sync' - rados.Rados.connect PermissionDeniedError: error connecting to the cluster" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 07:10 |
| jbadiapa | Tengu, ack thanks for all the help | 07:10 |
| *** yamahata has joined #tripleo | 07:10 | |
| *** fragatina has quit IRC | 07:11 | |
| Tengu | jbadiapa: no problem. thank you for your patience :) | 07:11 |
| *** spectr has joined #tripleo | 07:18 | |
| *** spectr has quit IRC | 07:21 | |
| *** oidgar has joined #tripleo | 07:23 | |
| *** moshele has joined #tripleo | 07:23 | |
| openstackgerrit | Jaganathan Palanisamy proposed openstack/tripleo-common master: Memory channels parameter is not derivable https://review.openstack.org/523315 | 07:24 |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui stable/pike: Change plan files whitelist when creating plan https://review.openstack.org/523316 | 07:26 |
| Tengu | jbadiapa: do you want me to patch your patch and test it locally, as my unit test infra is working on my laptop? :) | 07:27 |
| zshi | hi, how to disable package update when building overcloud-full images? | 07:27 |
| zshi | want to build a newton centos7.3 overcloud-image without upgrading to 7.4 | 07:28 |
| Tengu | hmm, not sure you can do that... and why do you want to get a not up-to-date image? | 07:29 |
| zshi | want to build a initial env to test fast forward upgrade. | 07:29 |
| jbadiapa | Tengu, Let me update my modifications | 07:30 |
| Tengu | makes sens. hmm. you'll probably need to dig in the overcloud building script though | 07:30 |
| Tengu | jbadiapa: ok ;) | 07:30 |
| zshi | I used centos7.3 as base image, it always be updated to 7.4 when building | 07:30 |
| zshi | worth trying :-) | 07:31 |
| *** ykarel is now known as ykarel|lunch | 07:34 | |
| marios | matbu: fixup commit? https://review.openstack.org/#/c/522217/3 and i revote | 07:34 |
| matbu | marios: yep merged conflict | 07:35 |
| matbu | thx | 07:35 |
| *** shreshtha-wfh has quit IRC | 07:35 | |
| marios | matbu: yeah i mean it should be the commit not the chid | 07:35 |
| matbu | marios: ha you mean the cherry pick id | 07:37 |
| matbu | yep right | 07:37 |
| *** agurenko has joined #tripleo | 07:38 | |
| *** fragatina has joined #tripleo | 07:39 | |
| *** oidgar has quit IRC | 07:41 | |
| *** yprokule has joined #tripleo | 07:41 | |
| *** gfidente has quit IRC | 07:45 | |
| *** cylopez has joined #tripleo | 07:47 | |
| marios | matbu: please https://review.openstack.org/#/c/522540/ when you next can | 07:47 |
| *** rcernin has quit IRC | 07:53 | |
| *** rasca has joined #tripleo | 07:53 | |
| *** japestinho has quit IRC | 07:55 | |
| *** d0ugal has joined #tripleo | 07:55 | |
| *** fzdarsky_ is now known as fzdarsky | 07:57 | |
| openstackgerrit | Juan Badia Payno proposed openstack/puppet-tripleo master: Disabled epel on collectd and added unit test https://review.openstack.org/521629 | 07:57 |
| jbadiapa | Tengu ^^^ | 07:58 |
| Tengu | cool, I abandon the other one then :) | 07:58 |
| *** paramite has joined #tripleo | 08:01 | |
| *** ebarrera has joined #tripleo | 08:05 | |
| Tengu | jbadiapa: epel wrapper abandonned. | 08:08 |
| *** shardy has joined #tripleo | 08:08 | |
| *** ratailor has quit IRC | 08:09 | |
| *** aufi has joined #tripleo | 08:10 | |
| *** ooolpbot has joined #tripleo | 08:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731063 | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1734134 | 08:10 |
| *** ooolpbot has quit IRC | 08:10 | |
| openstack | Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] | 08:10 |
| openstack | Launchpad bug 1734134 in tripleo "Pike periodic promotion job multinode-1ctlr-featureset016 fail with error running docker 'gnocchi_db_sync' - rados.Rados.connect PermissionDeniedError: error connecting to the cluster" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 08:10 |
| *** mrunge has quit IRC | 08:10 | |
| *** gkadam has joined #tripleo | 08:13 | |
| openstackgerrit | Rabi Mishra proposed openstack/python-tripleoclient master: Add timeout argument for scaledown/node_delete operation https://review.openstack.org/523326 | 08:14 |
| *** mrunge has joined #tripleo | 08:15 | |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common master: Handle error in the deploy_on_server workflow https://review.openstack.org/523332 | 08:17 |
| *** bogdando has joined #tripleo | 08:19 | |
| *** amoralej|off is now known as amoralej | 08:21 | |
| *** florianf has joined #tripleo | 08:22 | |
| *** ebarrera has quit IRC | 08:22 | |
| *** ebarrera has joined #tripleo | 08:23 | |
| *** pcaruana has joined #tripleo | 08:28 | |
| *** mcornea has joined #tripleo | 08:29 | |
| *** ykarel|lunch is now known as ykarel | 08:40 | |
| *** ffiore has joined #tripleo | 08:40 | |
| shardy | therve: Hey, good morning - wanted to chat about Zaqar/Mongodb when you have time, as IIRC you worked on switching the backend to swift for the undercloud? | 08:43 |
| therve | shardy, Yep | 08:44 |
| shardy | therve: Hi! So there's a thread on rdo-dev about retiring Mongo, and someone asked isn't TripleO still using Mongo | 08:44 |
| shardy | so I started looking, and it seems that is still the default backend for overcloud Zaqar | 08:44 |
| shardy | I'm wondering if we should switch that to Swift, but if we do then how would we handle upgrades etc | 08:45 |
| openstackgerrit | Rabi Mishra proposed openstack/python-tripleoclient master: Add timeout argument for scaledown/node_delete operation https://review.openstack.org/523326 | 08:45 |
| therve | So the plan in the overcloud was to use redis | 08:45 |
| therve | I didn't have time to properly test it, but there is support for it now | 08:46 |
| openstackgerrit | Flavio Percoco proposed openstack/tripleo-heat-templates master: Deploy OpenShift using OOO on the overcloud https://review.openstack.org/494470 | 08:46 |
| therve | Zaqar in the overcloud was never really supported, so I would be inclined to not care about upgrade | 08:46 |
| *** ccamacho has joined #tripleo | 08:46 | |
| shardy | Hmm, Ok - I don't see it marked as such, but I guess we can document it in a release note | 08:47 |
| shardy | therve: do you happen to know if anyone is already working on the switch to redis? | 08:47 |
| therve | shardy, Me :) | 08:47 |
| shardy | therve: aha! :) | 08:48 |
| *** ratailor has joined #tripleo | 08:48 | |
| shardy | therve: would you like to reply to the rdo-dev thread to clarify the status then? :) | 08:48 |
| therve | Woo rdo-dev. Another list I'm not on | 08:48 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-quickstart-extras master: Dump t-h-t env files used for overcloud deployment https://review.openstack.org/523349 | 08:48 |
| shardy | mhenkel: it also looks like ./contrail/roles_data_contrail.yaml references Mongo, but it's been disabled by default for a while, so would be good to confirm if you're actually using it? | 08:49 |
| shardy | therve: hehe, I can reply if you prefer | 08:49 |
| bogdando | o/ folks, PTAL https://review.openstack.org/522859 | 08:51 |
| bogdando | looks like we shall not mess with the host netns from containers? | 08:51 |
| bogdando | the bug reporter confirms it fixes his issue reported | 08:51 |
| *** holser has joined #tripleo | 08:52 | |
| bogdando | networking team, please ^^ | 08:52 |
| therve | shardy, I'll do it, thanks for the fwd | 08:52 |
| shardy | therve: np, thanks! | 08:52 |
| saneax | bogdando, as it is, without https://review.openstack.org/522859 host netns is broken | 08:52 |
| therve | shardy, Do you think we should switch the default? | 08:52 |
| shardy | therve: well if RDO are removing the package I don't think we really have any choice? | 08:53 |
| therve | shardy, Is it tested anywhere? | 08:53 |
| shardy | therve: Probably not, ./environments/scenario002-multinode-containers.yaml tests Zaqar but with ZaqarMessageStore: 'swift' | 08:54 |
| *** holser has quit IRC | 08:54 | |
| shardy | So hopefully we can switch the default and make that job test it with Redis? | 08:54 |
| *** holser has joined #tripleo | 08:55 | |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-heat-templates stable/pike: Missing cherry-picking stop condition for swift services. https://review.openstack.org/522753 | 08:56 |
| therve | We could yeah | 08:57 |
| *** shreshtha has joined #tripleo | 08:59 | |
| *** dparkes has quit IRC | 09:00 | |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates master: Select first node as bootstrap node not using name https://review.openstack.org/513450 | 09:01 |
| *** oidgar has joined #tripleo | 09:03 | |
| *** nyechiel has joined #tripleo | 09:06 | |
| *** jpich has joined #tripleo | 09:07 | |
| *** dparkes has joined #tripleo | 09:07 | |
| *** jpena|off is now known as jpena | 09:09 | |
| *** ooolpbot has joined #tripleo | 09:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731063 | 09:10 |
| openstack | Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1734134 | 09:10 |
| *** ooolpbot has quit IRC | 09:10 | |
| openstack | Launchpad bug 1734134 in tripleo "Pike periodic promotion job multinode-1ctlr-featureset016 fail with error running docker 'gnocchi_db_sync' - rados.Rados.connect PermissionDeniedError: error connecting to the cluster" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 09:10 |
| *** suuuper has joined #tripleo | 09:13 | |
| jpich | d0ugal: Thank you for looking into that YAQL error! I'mma apply your patch and launch another deployment, see if I can get more useful output for debugging now :) | 09:13 |
| openstackgerrit | Jaganathan Palanisamy proposed openstack/tripleo-heat-templates master: Memory channels parameter default value https://review.openstack.org/523358 | 09:13 |
| *** gfidente has joined #tripleo | 09:14 | |
| d0ugal | jpich: great. I have not had a chance to test it yet - I am not quite sure how to use the workflow I modified. | 09:15 |
| d0ugal | jpich: I was just about to try, but you might beat me to it. | 09:15 |
| jpich | d0ugal: I have a deployment I know will fail again right at my fingertips yeah :) You can try the path when it works?? heh | 09:17 |
| *** oidgar has quit IRC | 09:18 | |
| *** lucas-afk is now known as lucasagomes | 09:19 | |
| d0ugal | jpich: I am just about to update it | 09:19 |
| d0ugal | I think I spotted an error | 09:20 |
| jpich | Ack | 09:20 |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common master: Handle error in the deploy_on_server workflow https://review.openstack.org/523332 | 09:21 |
| *** shreshtha has quit IRC | 09:21 | |
| d0ugal | jpich: ^ | 09:21 |
| jpich | Ta! | 09:21 |
| *** dbecker has quit IRC | 09:21 | |
| d0ugal | jpich: probably wasn't a problem, but we use empty strings over nulls everywhere else for some reason | 09:21 |
| *** shreshtha has joined #tripleo | 09:21 | |
| *** Tripleo-User has joined #tripleo | 09:21 | |
| *** shreshtha has quit IRC | 09:21 | |
| *** shreshtha has joined #tripleo | 09:22 | |
| Tripleo-User | have people tested containerized HA overcloud ? | 09:22 |
| openstackgerrit | Merged openstack/python-tripleoclient stable/pike: Fix static-inventory option for minor update https://review.openstack.org/522217 | 09:23 |
| *** dbecker has joined #tripleo | 09:23 | |
| *** pgadiya has joined #tripleo | 09:24 | |
| *** moshele has quit IRC | 09:24 | |
| d0ugal | jpich: hrm, I might have just spotted another issue with the original workflow. | 09:24 |
| d0ugal | jpich: did this ever work? | 09:24 |
| *** moshele has joined #tripleo | 09:24 | |
| *** absubram has quit IRC | 09:24 | |
| *** pgadiya has quit IRC | 09:24 | |
| jpich | d0ugal: I'm guessing it must have? Unless I'm not activating the right environments for deploying with Ceph... I'm fairly sure the error I'm seeing is related | 09:25 |
| *** moshele has quit IRC | 09:25 | |
| *** moshele_ has joined #tripleo | 09:26 | |
| *** moshele_ has quit IRC | 09:26 | |
| *** moshele has joined #tripleo | 09:26 | |
| *** aufi has quit IRC | 09:26 | |
| d0ugal | jpich: okay, I might just not be digging deep enough. Looking at the action I can't see where deploy_stdout and deploy_stderr come from. | 09:26 |
| d0ugal | jpich: unless they are in the file in swift it downloads. | 09:26 |
| d0ugal | jpich: I would be curious to see the traceback in the executor log if you have it. | 09:27 |
| jpich | d0ugal: It's super long, I can give you access to the machine? | 09:27 |
| d0ugal | jpich: sure | 09:27 |
| d0ugal | jpich: https://github.com/d0ugal.keys | 09:28 |
| shardy | jaosorior or skramaja: Hey could one of you pls approve https://review.openstack.org/#/c/521927/ when you get a moment? | 09:28 |
| shardy | https://review.openstack.org/#/c/521928 is the next in that series, also passing CI and ready for review | 09:29 |
| shardy | would be nice to avoid rebasing those again | 09:29 |
| jaosorior | shardy: will that work? | 09:30 |
| jaosorior | shardy: my concern is line 11 | 09:30 |
| *** hewbrocca_afk is now known as hewbrocca | 09:30 | |
| jaosorior | shardy: I did not +A it cause I started having doubts about that, and was gonna check on that later (eventually I forgot about it) | 09:31 |
| shardy | jaosorior: feel free to test, but FWIW I've been testing that for a week or more locally and it works fine | 09:32 |
| jaosorior | shardy: oh, so it worked without enabled_roles? | 09:32 |
| shardy | basically we set enabled_roles to roles in the case where all roles disable upgrade | 09:32 |
| jaosorior | oh right | 09:33 |
| *** japestinho has joined #tripleo | 09:33 | |
| shardy | but unconditionally set is_upgrade to false | 09:33 |
| jaosorior | and there can't be an empty list in roles either now that I think about it | 09:33 |
| jaosorior | yeah, that makes sense | 09:33 |
| shardy | jaosorior: yeah, that enabled_roles[0] blew up which is what I needed to fix | 09:33 |
| shardy | otherwise a compute only stack where the Compute role has upgrades disabled won't deploy | 09:33 |
| shardy | open to other suggestions on how to handle it tho :) | 09:34 |
| shardy | I suspect that whole j2 block could be simplified but I went for the simplest fix to unblock my testing | 09:34 |
| jaosorior | shardy: 2ed | 09:34 |
| shardy | jaosorior: thanks! | 09:34 |
| jaosorior | shardy: also +2ed the hostsentry patch | 09:35 |
| *** etingof has quit IRC | 09:38 | |
| *** shreshtha has quit IRC | 09:38 | |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-heat-templates stable/pike: Add validation task in docker services [Octavia] https://review.openstack.org/517266 | 09:39 |
| *** aufi has joined #tripleo | 09:41 | |
| *** phpcodemonkey has joined #tripleo | 09:41 | |
| skramaja | shardy: done! | 09:44 |
| *** shreshtha has joined #tripleo | 09:44 | |
| *** shreshtha has quit IRC | 09:44 | |
| *** shreshtha has joined #tripleo | 09:44 | |
| *** derekh has joined #tripleo | 09:46 | |
| *** openstackgerrit has quit IRC | 09:48 | |
| shardy | skramaja: thanks! :) | 09:51 |
| *** shreshtha has quit IRC | 09:53 | |
| *** etingof has joined #tripleo | 09:54 | |
| *** openstackgerrit has joined #tripleo | 09:54 | |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-common stable/pike: Handle 'false' in when statements for ansible upgrade_tasks https://review.openstack.org/522540 | 09:54 |
| *** shreshtha has joined #tripleo | 09:54 | |
| *** pblaho has joined #tripleo | 09:55 | |
| *** rcernin has joined #tripleo | 09:55 | |
| openstackgerrit | Jaganathan Palanisamy proposed openstack/tripleo-heat-templates master: Memory channels parameter default value https://review.openstack.org/523358 | 09:58 |
| *** aufi has quit IRC | 10:03 | |
| *** pblaho has quit IRC | 10:03 | |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common master: Log the error from OrchestrationDeployAction https://review.openstack.org/523372 | 10:06 |
| openstackgerrit | mathieu bultel proposed openstack/python-tripleoclient stable/pike: Consume a zaqar queue for update to poll ansible result https://review.openstack.org/522218 | 10:06 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-quickstart-extras master: Dump t-h-t env files used for overcloud deployment https://review.openstack.org/523349 | 10:06 |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common master: Log the error from OrchestrationDeployAction https://review.openstack.org/523372 | 10:07 |
| *** yamahata has quit IRC | 10:07 | |
| d0ugal | jpich: ^ | 10:07 |
| d0ugal | jpich: having that would have probably saved us 40 mins or so | 10:08 |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common master: Log the error from OrchestrationDeployAction https://review.openstack.org/523372 | 10:08 |
| *** gfidente has quit IRC | 10:09 | |
| jpich | d0ugal: Gah! Thanks :) | 10:09 |
| *** cylopez has quit IRC | 10:09 | |
| *** ooolpbot has joined #tripleo | 10:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731063 | 10:10 |
| openstack | Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1734134 | 10:10 |
| *** ooolpbot has quit IRC | 10:10 | |
| openstack | Launchpad bug 1734134 in tripleo "Pike periodic promotion job multinode-1ctlr-featureset016 fail with error running docker 'gnocchi_db_sync' - rados.Rados.connect PermissionDeniedError: error connecting to the cluster" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 10:10 |
| *** pblaho has joined #tripleo | 10:12 | |
| *** gfidente has joined #tripleo | 10:12 | |
| openstackgerrit | Giulio Fidente proposed openstack/instack-undercloud stable/pike: Do not set dhcp_domain in Nova from overcloud_domain_name https://review.openstack.org/523375 | 10:13 |
| *** aufi has joined #tripleo | 10:15 | |
| openstackgerrit | Christian Schwede proposed openstack/tripleo-heat-templates master: Ensure rsyncd PID file is removed during overcloud updates https://review.openstack.org/523134 | 10:15 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-quickstart-extras master: Dump t-h-t env files used for overcloud deployment https://review.openstack.org/523349 | 10:16 |
| *** gfidente has quit IRC | 10:17 | |
| openstackgerrit | Jaganathan Palanisamy proposed openstack/tripleo-heat-templates master: Memory channels parameter default value https://review.openstack.org/523358 | 10:19 |
| *** gfidente has joined #tripleo | 10:21 | |
| *** gfidente has quit IRC | 10:21 | |
| *** gfidente has joined #tripleo | 10:21 | |
| *** pkovar has joined #tripleo | 10:24 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Remove hardcoded docker image names https://review.openstack.org/518094 | 10:24 |
| *** cylopez has joined #tripleo | 10:28 | |
| *** tosky has joined #tripleo | 10:31 | |
| *** salmankhan has joined #tripleo | 10:32 | |
| bogdando | folks PTAL https://review.openstack.org/#/c/523349 . I think those dumped env files complement the incomplete question 'which is the overcloud deploy command you're using?" The command basically does not help much as we need contents as well :) | 10:33 |
| bogdando | and get contents of templates auto-generated, provided as args, scattered around, is not a trivial | 10:34 |
| bogdando | getting* | 10:34 |
| skramaja | shardy: could you take a look at https://review.openstack.org/#/c/500475/, it had your +2 ealier but no changes after that. | 10:40 |
| shardy | skramaja: ack will do | 10:43 |
| *** wojdec has joined #tripleo | 10:43 | |
| shardy | bogdando: would it be better to instead just export the plan and save the tarball in the CI jobs? | 10:45 |
| shardy | then we see the contents that's actually deployed after any CLI logic | 10:45 |
| bogdando | shardy: I'm not sure it simplifies just looking into a single log file | 10:45 |
| shardy | bogdando: well catting every template adds a large amount of extra text | 10:46 |
| *** nyechiel has quit IRC | 10:46 | |
| shardy | personally I'd perhaps prefer having a plan directory in the logs, but we can go with the consensis | 10:46 |
| shardy | consensus I mean | 10:46 |
| bogdando | sometimes it's nice to have both command and data in a one place, just to do in browser ctrl+F | 10:46 |
| *** wojdec has left #tripleo | 10:46 | |
| shardy | bogdando: yeah but this won't help when the files are j2 rendered | 10:47 |
| bogdando | shardy: also, I'm not sure all things quickstart renders make it into the plan | 10:47 |
| bogdando | hm, shardy why? It provides the end data used for the command as is | 10:48 |
| *** salmankhan has quit IRC | 10:48 | |
| shardy | bogdando: have you seen the --no-cleanup option to the deploy command? | 10:48 |
| bogdando | shardy: not really :o | 10:48 |
| shardy | bogdando: some environment files are j2 rendered, so they don't exist on the filesystem, we render them during plan creation | 10:49 |
| shardy | bogdando: the plan does contain a merged environment, but that's a bug | 10:49 |
| shardy | bogdando: not blocking, this can go in if folks find it useful for CI | 10:50 |
| bogdando | shardy: right. Also I'm not sure with the 'all things quickstart renders make it into the plan' | 10:51 |
| openstackgerrit | Merged openstack/tripleo-validations master: Add env var for custom ssh user https://review.openstack.org/522874 | 10:52 |
| shardy | bogdando: yeah that's what I was trying to fix with https://review.openstack.org/#/c/448209/ but ran into problems with the tripleo-common part | 10:52 |
| shardy | bogdando: as a first step we could just save all the user environments to the plan, then the data you need would be present | 10:52 |
| *** shreshtha_ has joined #tripleo | 10:52 | |
| shardy | and a second step would be removing the merged user-environment.yaml and switching to heat server side merging | 10:52 |
| *** shreshtha_ has quit IRC | 10:54 | |
| *** shreshtha_ has joined #tripleo | 10:54 | |
| *** udesale has quit IRC | 10:55 | |
| *** shreshtha has quit IRC | 10:55 | |
| bogdando | shardy: right. Honestly look too complicated to me :) | 10:56 |
| shardy | bogdando: why? | 10:56 |
| bogdando | I'm trying to act as a user, being told please give us the command to deploy, knowing not much of the underhood | 10:56 |
| shardy | bogdando: I guess that's why I'm asking some questions, if you want to improve the operator experience, quickstart doesn't help | 10:57 |
| shardy | like, we could add some flag to tripleoclient I guess, as --debug is too noisy | 10:57 |
| bogdando | So he could open the autogenerated docs or a deployment log and get anything needed from there... But yeah, those rendered configs really spoil the thing | 10:57 |
| shardy | or, we could disable the client logging for --debug because it's too noisy, but log the templates in that mode | 10:57 |
| shardy | bogdando: just trying to share some ideas, like I said not blocking if your patch is a quick fix for CI debugging | 10:58 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-quickstart-extras master: Collect most of var/lib/heat-config directory. https://review.openstack.org/523092 | 11:02 |
| bogdando | shardy: so the plan is, IIUC: add that quick debugging, then try to accomplish that first step with https://review.openstack.org/#/c/448209/, then propose a supporting debug trigger for logging all the env files from the client, and finally, undo the quick debugging patch I provided? | 11:04 |
| bogdando | and w/o that quick debug patch we just can't be sure we have rendered files in the plan, right? | 11:06 |
| *** jkilpatr has joined #tripleo | 11:06 | |
| openstackgerrit | Juan Badia Payno proposed openstack/puppet-tripleo master: Disabled epel on collectd and added unit test https://review.openstack.org/521629 | 11:06 |
| *** fzdarsky is now known as fzdarsky|lunch | 11:07 | |
| *** morazi has quit IRC | 11:07 | |
| openstackgerrit | Athlan-Guyot sofer proposed openstack-infra/tripleo-ci master: Make sure we collect var/lib/heat-config directory. https://review.openstack.org/523388 | 11:08 |
| *** ooolpbot has joined #tripleo | 11:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731063 | 11:10 |
| openstack | Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1734134 | 11:10 |
| *** ooolpbot has quit IRC | 11:10 | |
| openstack | Launchpad bug 1734134 in tripleo "Pike periodic promotion job multinode-1ctlr-featureset016 fail with error running docker 'gnocchi_db_sync' - rados.Rados.connect PermissionDeniedError: error connecting to the cluster" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 11:10 |
| shardy | bogdando: well that's one way to do it, but my point was that the client already has access to all the environment files, including those that are j2 rendered | 11:10 |
| shardy | so an alternative would be to add a flag to tripleoclient now that dumps all the files, not only those that aren't j2 rendered | 11:11 |
| shardy | If we also saved the plan in CI, I think we'd have all the information to debug template related issues | 11:11 |
| bogdando | shardy: good point, thank you. Now I get it :) | 11:17 |
| *** salmankhan has joined #tripleo | 11:26 | |
| *** jkilpatr has quit IRC | 11:27 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Allow empty list of enabled_roles https://review.openstack.org/521927 | 11:29 |
| *** adarazs|ruck is now known as adarazs|ruck|lnc | 11:30 | |
| *** stendulker has quit IRC | 11:30 | |
| openstackgerrit | Luke Hinds proposed openstack/tripleo-heat-templates master: Implements management of `/etc/login.defs` https://review.openstack.org/457985 | 11:31 |
| chem | jfrancoa: hey do you have in mind the recheck line for re-checking only "RedHat RDO CI" (in https://review.openstack.org/#/c/523092/) | 11:32 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient master: Remove the overcloudrc.v3 file https://review.openstack.org/521941 | 11:33 |
| openstackgerrit | Jaganathan Palanisamy proposed openstack/tripleo-common master: SRIOV derive parameters workflows https://review.openstack.org/522265 | 11:34 |
| amoralej | periodic jobs are failing in master with heat validation error, i've reported in https://bugs.launchpad.net/tripleo/+bug/1734871 | 11:36 |
| openstack | Launchpad bug 1734871 in tripleo "overcloud deployment fails on mistral action DeployStackAction" [Undecided,Triaged] | 11:36 |
| amoralej | shardy, ^ i'm not sure who is the best one to take care | 11:36 |
| *** jkilpatr has joined #tripleo | 11:38 | |
| *** mdnadeem has quit IRC | 11:38 | |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-quickstart-extras master: Always use overcloudrc, it is now v3 by default https://review.openstack.org/523393 | 11:39 |
| *** salmankhan has quit IRC | 11:39 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Add parameter ExtraHostFileEntries https://review.openstack.org/521928 | 11:40 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient master: Remove the overcloudrc.v3 file https://review.openstack.org/521941 | 11:41 |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-quickstart-extras master: Always use overcloudrc, it is now v3 by default https://review.openstack.org/523393 | 11:41 |
| shardy | amoralej: I suspect it's a keyston regression, as looking at recent commits some validation of trusts role names was added | 11:43 |
| shardy | https://github.com/openstack/keystone/commits/master | 11:43 |
| shardy | jaosorior: Hey do you know who we have working on keystone atm? | 11:43 |
| shardy | https://github.com/openstack/keystone/commit/f8e79ab50775bcf5964c7547297577d0a3b82519 | 11:43 |
| jaosorior | shardy: hrybacki, raildo and jdennis | 11:43 |
| shardy | that has a test case with "member" but not "_member_", let me see if that fails | 11:44 |
| jaosorior | shardy: raildo is the one with the closest timezone, he's in Brazil. | 11:44 |
| jaosorior | shardy: else we gotta wait for jdennis and hrybacki who are in the states | 11:44 |
| *** deadnull has joined #tripleo | 11:46 | |
| *** deadnull has quit IRC | 11:46 | |
| chem | jfrancoa: I've made a review to collect /var/lib/heat-config and added the rdo ci in it https://review.openstack.org/#/c/523388/ so that we may better understand the issue with upgrade. | 11:47 |
| shardy | jaosorior: ack thanks, I'll see if there's an obvious fix otherwise we can wait for those folks to wake up | 11:47 |
| shardy | amoralej: I'll investigate a bit more then comment on the bug | 11:47 |
| amoralej | ok, thanks shardy | 11:48 |
| *** mhenkel has quit IRC | 11:50 | |
| *** gfidente has quit IRC | 11:50 | |
| bogdando | jistr, shardy: what is the place we define the inner playbook exec command? I'd add -v there | 11:53 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates master: Remove hiera hook transition from the upgradeinitcommand. https://review.openstack.org/509375 | 11:53 |
| *** mhenkel has joined #tripleo | 11:54 | |
| *** mdnadeem has joined #tripleo | 11:55 | |
| bogdando | jistr, shardy: is it tripleo_common/actions/ansible.py ? | 11:56 |
| shardy | jistr: Hey is the kubespray install still working for you? I'm running latest tht & kubespray and have some issues, tried with and without the firewall and things time out waiting for the apiserver to be running | 11:56 |
| shardy | flaper87: ^^ FYI not working for me locally atm | 11:56 |
| shardy | Oh wait | 11:57 |
| shardy | nvm I think I saw the reason, haproxy is still running on the host | 11:57 |
| bogdando | oh, nvm, I think it's in the templates directly | 11:58 |
| *** ratailor has quit IRC | 11:59 | |
| *** shardy is now known as shardy_afk | 11:59 | |
| *** aufi has quit IRC | 12:00 | |
| *** morazi has joined #tripleo | 12:00 | |
| *** bkopilov has quit IRC | 12:01 | |
| *** psachin has quit IRC | 12:01 | |
| openstackgerrit | Merged openstack/python-tripleoclient master: Mount a tmpfs filesystem for heat tmpfiles https://review.openstack.org/508558 | 12:03 |
| *** pblaho has quit IRC | 12:03 | |
| *** thrash|g0ne is now known as thrash | 12:03 | |
| *** d0ugal has quit IRC | 12:04 | |
| jistr | shardy_afk, bogdando, flaper87: FWIW this fix worked https://review.openstack.org/#/c/523136/ | 12:06 |
| jistr | i'm trying to fix for the current issue in the CI job (due to missing git) | 12:06 |
| *** jkilpatr has quit IRC | 12:06 | |
| *** jkilpatr has joined #tripleo | 12:07 | |
| jistr | also we have another problem -- the job was green even though Ansible failed (not Kubespray, the outer Ansible deployment playbook failed, and the job was still green) | 12:07 |
| jistr | after i push a fix for the git issue i'll see if i can spot where the problem is with that ^ | 12:07 |
| *** d0ugal has joined #tripleo | 12:08 | |
| *** ooolpbot has joined #tripleo | 12:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731063 | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1734134 | 12:10 |
| *** ooolpbot has quit IRC | 12:10 | |
| openstack | Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] | 12:10 |
| openstack | Launchpad bug 1734134 in tripleo "Pike periodic promotion job multinode-1ctlr-featureset016 fail with error running docker 'gnocchi_db_sync' - rados.Rados.connect PermissionDeniedError: error connecting to the cluster" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 12:10 |
| *** raildo has joined #tripleo | 12:10 | |
| *** ansmith has quit IRC | 12:11 | |
| flaper87 | jistr: looking | 12:18 |
| flaper87 | shardy_afk: does the dns work too? | 12:18 |
| flaper87 | jistr: +A | 12:19 |
| *** pblaho has joined #tripleo | 12:20 | |
| jistr | flaper87: thanks | 12:20 |
| *** udesale has joined #tripleo | 12:23 | |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-heat-templates master: Download Kubespray instead of git clone https://review.openstack.org/523397 | 12:24 |
| flaper87 | jistr: ^ why is that better than git-clone? (out of curiosity) | 12:25 |
| openstackgerrit | mathieu bultel proposed openstack/tripleo-common stable/pike: Add exclude list to not override user data https://review.openstack.org/523398 | 12:25 |
| jistr | flaper87: git isn't installed in CI, so the job failed on trying to do git clone. I thought to just install git too in that "to be removed" branch of code that clones Kubespray, but we don't have sudo rights on undercloud from external_deploy_tasks. | 12:26 |
| flaper87 | jistr: a-ha, ok! nah, download is prob better | 12:27 |
| flaper87 | :) | 12:27 |
| jistr | tested locally, wfm, pending the experimental job results to see what's next in CI | 12:27 |
| *** aufi has joined #tripleo | 12:27 | |
| jistr | it will at least fail on the puppet step 2 due to puppetlabs-firewall which is fixed in repo but not yet in current-tripleo-ci RPMs | 12:28 |
| jistr | if it would get *that* far, that would be awesome | 12:28 |
| openstackgerrit | Lee Yarwood proposed openstack/tripleo-heat-templates master: WIP ffu: Add fast-forward upgrade outputs to RoleConfig https://review.openstack.org/499221 | 12:28 |
| openstackgerrit | Lee Yarwood proposed openstack/tripleo-heat-templates master: WIP ffu: tripleo-packages repo management https://review.openstack.org/515429 | 12:28 |
| openstackgerrit | Lee Yarwood proposed openstack/tripleo-heat-templates master: DNM ffu: Workaround missing repo control when moving to Queens https://review.openstack.org/518719 | 12:28 |
| openstackgerrit | Lee Yarwood proposed openstack/tripleo-heat-templates master: WIP ffu: Introduce Keystone fast-forward upgrade tasks https://review.openstack.org/514621 | 12:28 |
| openstackgerrit | Lee Yarwood proposed openstack/tripleo-heat-templates master: WIP ffu: Introduce Neutron fast-forward upgrade tasks https://review.openstack.org/521543 | 12:28 |
| openstackgerrit | Lee Yarwood proposed openstack/tripleo-heat-templates master: WIP ffu: Introduce Glance fast-forward upgrade tasks https://review.openstack.org/521544 | 12:28 |
| openstackgerrit | Lee Yarwood proposed openstack/tripleo-heat-templates master: WIP ffu: Introduce Cinder fast-forward upgrade tasks https://review.openstack.org/521545 | 12:28 |
| openstackgerrit | Lee Yarwood proposed openstack/tripleo-heat-templates master: WIP ffu: Introduce Nova fast-forward upgrade tasks https://review.openstack.org/522921 | 12:28 |
| openstackgerrit | Lee Yarwood proposed openstack/tripleo-heat-templates master: WIP ffu: Allow FASTFORWARDUPGRADE as a StackUpdateType https://review.openstack.org/522547 | 12:28 |
| openstackgerrit | Lee Yarwood proposed openstack/tripleo-heat-templates master: WIP ffu: Add fast-forward-upgrade env https://review.openstack.org/522203 | 12:28 |
| openstackgerrit | Lee Yarwood proposed openstack/tripleo-heat-templates master: WIP Add missing become to upgrade_steps_playbook https://review.openstack.org/522922 | 12:28 |
| openstackgerrit | Lee Yarwood proposed openstack/tripleo-heat-templates master: WIP Add missing become to Host prep steps play https://review.openstack.org/522923 | 12:28 |
| openstackgerrit | Lee Yarwood proposed openstack/tripleo-heat-templates master: WIP ffu: Introduce Pacemaker fast-forward upgrade tasks https://review.openstack.org/523399 | 12:28 |
| openstackgerrit | Lee Yarwood proposed openstack/tripleo-heat-templates master: DNM ffu: ignore_errors during upgrade_tasks https://review.openstack.org/523400 | 12:28 |
| *** salmankhan has joined #tripleo | 12:29 | |
| *** ramishra has quit IRC | 12:33 | |
| jistr | shardy_afk: re your earlier question, latest kubespray master is working for me, just finished a deployment | 12:33 |
| flaper87 | jistr: roger, lemme know and I'll review it | 12:33 |
| *** gfidente has joined #tripleo | 12:35 | |
| *** gfidente has quit IRC | 12:35 | |
| *** gfidente has joined #tripleo | 12:35 | |
| jistr | flaper87: re patches needed, it's just the one you already +A'd, the git/curl switcheroo, and master puppetlabs-firewall for this https://github.com/puppetlabs/puppetlabs-firewall/commit/3a9a1f7b6ef5ed5065d97ef02f69b37ca35a7110 | 12:36 |
| *** psahoo has quit IRC | 12:40 | |
| openstackgerrit | Brad P. Crochet proposed openstack/python-tripleoclient master: Display Horizon URL at the end of a deployment https://review.openstack.org/518030 | 12:40 |
| jfrancoa | chem: thanks, I also added a dependency on https://review.openstack.org/#/c/517266/, to see if we can get the logs from a O->P CI job run | 12:42 |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-common master: Validate roles data and network data https://review.openstack.org/508567 | 12:43 |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-common master: Add a Get Networks workflow https://review.openstack.org/509419 | 12:43 |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-common master: Add UpdateNetworks workflow https://review.openstack.org/513463 | 12:43 |
| bogdando | jistr did it work with ext LB and VIP? | 12:44 |
| jistr | bogdando: i wasn't testing w/ that, ATM i'm trying to go the shortest path to successful Kubespray CI job | 12:45 |
| bogdando | jistr: I'm still trying atm | 12:45 |
| jistr | bogdando, shardy_afk, flaper87: btw if we keep having some issues with LB and we want to test the APBs ASAP, i think we could just point the APBs to 1st controller IP for the time being, and work on both in parallel | 12:46 |
| *** jfrancoa has quit IRC | 12:47 | |
| flaper87 | jistr: yup yup | 12:47 |
| *** jfrancoa has joined #tripleo | 12:49 | |
| *** jtomasek_ has joined #tripleo | 12:49 | |
| bogdando | jistr: there is a case when it configures access enpoint with the 1st master | 12:49 |
| bogdando | so yeah, to unlbock apbs might be ok | 12:49 |
| bogdando | jistr: https://github.com/kubernetes-incubator/kubespray/blob/master/docs/ha-mode.md see at the bottom | 12:50 |
| bogdando | No ext/int LB | 12:50 |
| openstackgerrit | Martin André proposed openstack/tripleo-common master: Prevent apache from listening on default ports https://review.openstack.org/523404 | 12:50 |
| bogdando | that's our case, but not sure what it gets for all-in-one case | 12:51 |
| *** ffiore has quit IRC | 12:51 | |
| mandre | jaosorior: ^^ | 12:51 |
| *** adarazs|ruck|lnc is now known as adarazs|ruck | 12:51 | |
| *** jtomasek has quit IRC | 12:52 | |
| *** atoth has joined #tripleo | 12:53 | |
| jistr | bogdando: but that's the internal access endpoint, within the cluster, no? I think how we configure external access from undercloud is out of Kubespray sphere of influence. | 12:54 |
| bogdando | jistr: yes | 12:54 |
| jistr | we were using the 1st master mode internally too earlier, but now we have the localhost LB | 12:54 |
| jistr | ack | 12:54 |
| bogdando | jistr: but when access_ip is a floating IP, it may work as well | 12:55 |
| *** fzdarsky|lunch is now known as fzdarsky | 12:55 | |
| bogdando | just with a potential fix for all-in-one, to not set it to localhost:port | 12:55 |
| *** abishop has joined #tripleo | 12:55 | |
| *** nyechiel has joined #tripleo | 12:56 | |
| bogdando | jistr: I mean, we can try with no external LB and no localhost LB, using access_ip = floating ip | 12:56 |
| jistr | bogdando: right, but i wonder if that's better than external/vip LB && localhost LB at the same time... | 12:57 |
| *** shardy_afk is now known as shardy | 12:57 | |
| jistr | maybe it's fewer parts to break... | 12:57 |
| shardy | bogdando: what do you mean by floating IP in this case? | 12:57 |
| jistr | but will increase traffic on the host which owns the VIP | 12:58 |
| shardy | I thought we'd need the VIP for that | 12:58 |
| bogdando | shardy: some publicly routed IP | 12:58 |
| bogdando | if you want to have not only internal access | 12:58 |
| bogdando | jistr: that case is not supported :o | 12:58 |
| bogdando | using both modes at once | 12:58 |
| shardy | bogdando: well we do only want internal access, so we can deploy OpenStack services on the internal network without exposing k8s to the tenants | 12:59 |
| shardy | jistr: but yeah I'm fine with just selecting the controller IP in a single node job to make progress | 12:59 |
| shardy | Honestly I thought making kubespray work with an external LB would be easier than it turned out to be | 12:59 |
| bogdando | shardy, jistr: so then let's try with "no external LB and no localhost LB", using 1st master ip=<some-sane-value> in inventory | 13:00 |
| bogdando | shardy: untested layout, is only the best effort :) | 13:00 |
| jistr | bogdando: what do you mean by not supported though? I don't think Kubespray needs to do anything extra. | 13:00 |
| shardy | well I think we can stick with the current localhost LB with a single controller | 13:00 |
| shardy | e.g what we already landed | 13:00 |
| jistr | IIUC we can keep access_ip to be the internal LB | 13:00 |
| bogdando | jistr: please look into the table, there is no case for Ext LB && localhost LB | 13:01 |
| shardy | bogdando: I think we agree it should be one or the other | 13:01 |
| shardy | bogdando: but we're still missing a working example of how to configure an external LB | 13:01 |
| jistr | bogdando: right well from Kubespray's point of view, we'd be using only "localhost LB"... it does not have to care or know how we approach the API from undercloud | 13:01 |
| *** amoralej is now known as amoralej|lunch | 13:02 | |
| *** dtantsur|afk is now known as dtantsur | 13:02 | |
| *** jpena is now known as jpena|lunch | 13:02 | |
| bogdando | shardy: ext LB and ext VIP is TBD, for sure. My comments were wrt fast-unblock APBs | 13:02 |
| *** cmyster has quit IRC | 13:03 | |
| shardy | bogdando: yeah I think we have a way forward which isn't ideal but will allow us to continue | 13:04 |
| jaosorior | raildo: around? | 13:04 |
| raildo | jaosorior, yep | 13:04 |
| shardy | it's just a bit frustrating that this is documented for kubespray but seems in pretty rough shape | 13:05 |
| jaosorior | shardy: ^^ | 13:05 |
| shardy | raildo: Hi! So there's an issue in the periodic promotions, which I think is related to keystone rejecting the _member_ role when creating a trust | 13:05 |
| jistr | export KUBE_API=$(openstack server list | grep controller-0 | grep -Eo '192\.168\.24\.[0-9]+') | 13:05 |
| jistr | curl -k https://$KUBE_API:6443 | 13:05 |
| shardy | raildo: I see there have been some recent changes to keystone in this regard, so wanted to get someone to take a look | 13:06 |
| jistr | shardy, bogdando, flaper87: ^ this works already so we don't really need anything special | 13:06 |
| jistr | (access to kubernetes from undercloud) | 13:06 |
| raildo | shardy, sure, can you send me a link for that issue? | 13:06 |
| shardy | raildo: https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset002-master-upload/5c3aa6c/undercloud/var/log/heat/heat_api.log.txt.gz#_2017-11-28_09_58_08_490 | 13:06 |
| shardy | raildo: ield 'roles/1/name': u'_member_' does not match '^[a-zA-Z0-9-]+$' | 13:06 |
| *** mdnadeem has quit IRC | 13:06 | |
| *** liverpooler has joined #tripleo | 13:06 | |
| *** jlabarre has joined #tripleo | 13:06 | |
| shardy | raildo: I looked at the recent commits and it looks like there were changes to the roles validation recently, so perhaps this case got missed? | 13:06 |
| shardy | we're still using the historical _member_role in some cases | 13:07 |
| shardy | raildo: https://github.com/openstack/keystone/commit/f8e79ab50775bcf5964c7547297577d0a3b82519 | 13:07 |
| shardy | that has a test case for "member" but not "_member_" | 13:07 |
| shardy | I was going to add that and run the tests but not got to it yet | 13:07 |
| sshnaidm | jistr, hi | 13:08 |
| jistr | sshnaidm: hi | 13:08 |
| *** rhallisey has joined #tripleo | 13:09 | |
| shardy | jistr: Personally I'd prefer to derive it from the dynamic inventory e.g using https://review.openstack.org/#/c/517051/ | 13:09 |
| shardy | jistr: but yeah whatever works for a first pass I guess | 13:10 |
| *** ooolpbot has joined #tripleo | 13:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731063 | 13:10 |
| openstack | Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1734134 | 13:10 |
| *** ooolpbot has quit IRC | 13:10 | |
| openstack | Launchpad bug 1734134 in tripleo "Pike periodic promotion job multinode-1ctlr-featureset016 fail with error running docker 'gnocchi_db_sync' - rados.Rados.connect PermissionDeniedError: error connecting to the cluster" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 13:10 |
| sshnaidm | jistr, wanted to ask you about https://bugs.launchpad.net/tripleo/+bug/1734778 but I see jfrancoa have already handled this, nm, ignore it :) | 13:10 |
| openstack | Launchpad bug 1728917 in tripleo "duplicate for #1734778 Upgrades CI jobs in master not performing upgrade" [High,Fix released] - Assigned to Carlos Camacho (ccamacho) | 13:10 |
| *** ebarrera has quit IRC | 13:10 | |
| raildo | shardy, yeah, but that change is specific for dealing with trusts, I'm not sure if that fits with our case here | 13:10 |
| jistr | shardy: right i didn't mean to actually fetch the IP that way, i was just trying to illustrate that we don't need to switch the Kubespray LB mode to get a working access from UC | 13:10 |
| flaper87 | shardy: jistr if we can get it from the inventory already, then I'd vote for that | 13:10 |
| *** dprince has joined #tripleo | 13:11 | |
| *** suuuper1 has joined #tripleo | 13:11 | |
| jfrancoa | sshnaidm: yes, chem and I are trying to debug what's wrong with it. It seems the overcloud packages are not being updated, and so it can't find the puppet base::docker library | 13:11 |
| sshnaidm | jfrancoa, I see, thanks for handing this | 13:12 |
| jfrancoa | sshnaidm: no problem | 13:12 |
| *** cmyster has joined #tripleo | 13:12 | |
| *** cmyster has quit IRC | 13:12 | |
| *** cmyster has joined #tripleo | 13:12 | |
| chem | jfrancoa: sshnaidm I'm starting a new test job with more debug info using https://review.openstack.org/#/c/523388/ | 13:12 |
| *** suuuper has quit IRC | 13:13 | |
| *** ebarrera has joined #tripleo | 13:13 | |
| raildo | shardy, for role names we use that schema validation: https://github.com/openstack/keystone/blob/master/keystone/common/validation/parameter_types.py#L25 | 13:13 |
| shardy | raildo: yes heat creates a trust, which I think is where it's failing | 13:14 |
| raildo | shardy, ah, so that's makes sense | 13:14 |
| sshnaidm | chem, please see my comment there | 13:14 |
| *** lucasagomes is now known as lucas-hungry | 13:14 | |
| shardy | raildo: any chance we can relax that validation a bit to special-case the _member_ role? | 13:14 |
| shardy | raildo: this is a promotion blocker for us atm | 13:14 |
| raildo | shardy, that's probably the best way to do that, I'll talk with the keystone folks and send a patch asap | 13:15 |
| shardy | https://bugs.launchpad.net/tripleo/+bug/1734871 | 13:15 |
| openstack | Launchpad bug 1734871 in tripleo "overcloud deployment fails on mistral action DeployStackAction" [Critical,Triaged] | 13:15 |
| shardy | raildo: Ok thanks, would you be able to add keystone to that bug and get it assigned? | 13:15 |
| raildo | shardy, sure | 13:16 |
| *** links has quit IRC | 13:16 | |
| *** yolanda has quit IRC | 13:16 | |
| *** thrash is now known as thrash|biab | 13:16 | |
| shardy | raildo: thanks! :) | 13:17 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack-infra/tripleo-ci master: Make sure we collect var/lib/heat-config directory. https://review.openstack.org/523388 | 13:18 |
| jaosorior | adarazs|ruck, sshnaidm: Could you check this out https://review.openstack.org/#/c/521591/ ? | 13:18 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-quickstart-extras master: Collect most of var/lib/heat-config directory. https://review.openstack.org/523092 | 13:19 |
| chem | sshnaidm: done, thanks. | 13:19 |
| *** trown|outtypewww is now known as trown | 13:20 | |
| adarazs|ruck | jaosorior: +2 | 13:20 |
| jaosorior | adarazs|ruck: thanks | 13:20 |
| *** pchavva has joined #tripleo | 13:21 | |
| *** yolanda has joined #tripleo | 13:21 | |
| *** mdnadeem has joined #tripleo | 13:22 | |
| sshnaidm | jaosorior, +w | 13:22 |
| *** ffiore has joined #tripleo | 13:22 | |
| jaosorior | shardy: could you check this out https://review.openstack.org/#/c/521731/ ? | 13:22 |
| jaosorior | sshnaidm: thanks :D | 13:22 |
| *** pchavva has quit IRC | 13:27 | |
| *** bkopilov has joined #tripleo | 13:28 | |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-common stable/pike: [DO NOT MERGE] Testing upgrade. https://review.openstack.org/523408 | 13:29 |
| *** dmacpher is now known as dmacpher-afk | 13:30 | |
| chem | sshnaidm: will the patch from tripleo-ci be taken with the depends-on on the above review ? | 13:30 |
| shardy | jaosorior: ack will do | 13:31 |
| *** aputtur_ has quit IRC | 13:31 | |
| sshnaidm | chem, yeah, of course | 13:31 |
| *** skramaja has quit IRC | 13:32 | |
| *** rlandy has joined #tripleo | 13:33 | |
| *** rlandy is now known as rlandy|rover | 13:35 | |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-common stable/pike: [DO NOT MERGE] Testing upgrade. https://review.openstack.org/523408 | 13:36 |
| chem | sshnaidm: ack, was wondering because there are not in the same branch | 13:37 |
| chem | sshnaidm: (master vs stable/pike) | 13:37 |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: Remove BASE var from vxlan_networking.sh https://review.openstack.org/509391 | 13:38 |
| sshnaidm | chem, tripleo-ci is branchless, so I hope it should be ok | 13:38 |
| chem | sshnaidm: ack, will have a definitive answer in a few hours :) | 13:39 |
| *** jaganathan has quit IRC | 13:40 | |
| *** ansmith has joined #tripleo | 13:40 | |
| *** fultonj has joined #tripleo | 13:41 | |
| raildo | shardy, https://review.openstack.org/#/c/523415/1 | 13:42 |
| *** pchavva has joined #tripleo | 13:42 | |
| dtantsur | hey folks! can I get a 2nd +2 on https://review.openstack.org/#/c/519300/ please? it's quite important to move forward with ironic stuff in the undercloud | 13:43 |
| openstackgerrit | Keith Schincke proposed openstack/tripleo-common master: Add support for ceph_rbdmirror_ansible_vars from tht https://review.openstack.org/520658 | 13:43 |
| *** ramishra has joined #tripleo | 13:44 | |
| *** chem has quit IRC | 13:48 | |
| *** fultonj has quit IRC | 13:48 | |
| *** chem has joined #tripleo | 13:51 | |
| openstackgerrit | Merged openstack/tripleo-common stable/pike: Handle 'false' in when statements for ansible upgrade_tasks https://review.openstack.org/522540 | 13:53 |
| *** ykarel is now known as ykarel|away | 13:54 | |
| jistr | bogdando, shardy, flaper87: the git/curl patch is good, CI went further, but apparently the external_deploy_tasks are testing previously unexercised code paths in config-download... looking furhter and hoping i'll have more fixes shortly | 13:54 |
| EmilienM | hello | 13:57 |
| mwhahaha | reminder, meeting in 3 mins https://etherpad.openstack.org/p/tripleo-meeting-items | 13:57 |
| mwhahaha | #startmeeting tripleo | 14:00 |
| mwhahaha | #topic agenda | 14:00 |
| mwhahaha | * Review past action items | 14:00 |
| mwhahaha | * One off agenda items | 14:00 |
| mwhahaha | * Squad status | 14:00 |
| mwhahaha | * Bugs & Blueprints | 14:00 |
| openstack | Meeting started Tue Nov 28 14:00:13 2017 UTC and is due to finish in 60 minutes. The chair is mwhahaha. Information about MeetBot at http://wiki.debian.org/MeetBot. | 14:00 |
| mwhahaha | * Projects releases or stable backports | 14:00 |
| mwhahaha | * Specs | 14:00 |
| mwhahaha | * open discussion | 14:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 14:00 |
| mwhahaha | Anyone can use the #link, #action and #info commands, not just the moderatorǃ | 14:00 |
| mwhahaha | Hi everyone! who is around today? | 14:00 |
| *** openstack changes topic to " (Meeting topic: tripleo)" | 14:00 | |
| openstack | The meeting name has been set to 'tripleo' | 14:00 |
| *** openstack changes topic to "agenda (Meeting topic: tripleo)" | 14:00 | |
| d0ugal | Hello! | 14:00 |
| beagles | o/ | 14:00 |
| arxcruz | o/ | 14:00 |
| *** yamahata has joined #tripleo | 14:00 | |
| abishop | o/ | 14:00 |
| trown | o/ | 14:00 |
| ccamacho | hey folks | 14:00 |
| slagle | hi | 14:00 |
| gfidente | o/ | 14:00 |
| *** mmethot has joined #tripleo | 14:00 | |
| jfrancoa | o/ | 14:00 |
| EmilienM | o/ | 14:01 |
| lyarwood | o/ | 14:01 |
| marios | o/ | 14:01 |
| chem | o/ | 14:01 |
| shardy | o/ | 14:01 |
| jpich | o/ | 14:02 |
| *** salmankhan has quit IRC | 14:02 | |
| atoth | o/ | 14:02 |
| adarazs|ruck | o/ | 14:02 |
| openstackgerrit | Merged openstack/tripleo-ui stable/pike: Change plan files whitelist when creating plan https://review.openstack.org/523316 | 14:02 |
| *** pradk has joined #tripleo | 14:03 | |
| mwhahaha | ok lets do this | 14:03 |
| mwhahaha | #topic review past action items | 14:03 |
| *** openstack changes topic to "review past action items (Meeting topic: tripleo)" | 14:03 | |
| mwhahaha | none | 14:03 |
| mwhahaha | moving on to the agenda | 14:03 |
| mwhahaha | #topic one off agenda items | 14:04 |
| mwhahaha | #link https://etherpad.openstack.org/p/tripleo-meeting-items | 14:04 |
| *** openstack changes topic to "one off agenda items (Meeting topic: tripleo)" | 14:04 | |
| mwhahaha | (gfidente) how to discern containerized vs non-containerized services within the templates? | 14:04 |
| gfidente | yeah I was trying to understand if we have a way to do that from the templates right now? | 14:04 |
| *** jd_ has left #tripleo | 14:04 | |
| beagles | I could use that ability as well | 14:04 |
| mwhahaha | seems like maybe a setting we could do in docker.yaml? | 14:05 |
| mwhahaha | if one isn't already there | 14:05 |
| marios | mwhahaha: well i think it would have to be per service? | 14:05 |
| *** salmankhan has joined #tripleo | 14:05 | |
| mwhahaha | depends on what you're looking for | 14:05 |
| mwhahaha | but yea perhaps there as well | 14:05 |
| marios | mwhahaha: like a hiera we set for 'servicename_is_docker' or something | 14:06 |
| marios | gfidente: but what do you have in mind/ have a spec? bug? | 14:06 |
| *** ykarel|away has quit IRC | 14:06 | |
| gfidente | marios no I hit frequently a pattern where I need to know what other services are deployed | 14:06 |
| gfidente | for example, to build the list of pools to create | 14:06 |
| gfidente | to grant permissions on a key | 14:06 |
| *** lucas-hungry is now known as lucasagomes | 14:07 | |
| jaosorior | o/ | 14:07 |
| marios | gfidente: well the service list you can get from the role data but the if service is containerized is not there if that is what you want specifically | 14:07 |
| gfidente | I think I should approach this differently and emit from every service enabled a pool to be created | 14:07 |
| marios | afaik at least | 14:07 |
| mandre | o/ | 14:07 |
| *** thrash|biab is now known as thrash | 14:07 | |
| *** jpena|lunch is now known as jpena|off | 14:08 | |
| gfidente | regarding containerized vs non-containerized , that seems just another special case where we need to grant permissions on a file only if the targer service is not containerized | 14:08 |
| *** amoralej|lunch is now known as amoralej | 14:08 | |
| gfidente | I was mostly trying to understand if there were ideas on how this could have been approached | 14:09 |
| *** gbarros has joined #tripleo | 14:09 | |
| *** amoralej is now known as amoralej|off | 14:10 | |
| *** ooolpbot has joined #tripleo | 14:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731063 | 14:10 |
| openstack | Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1734134 | 14:10 |
| *** ooolpbot has quit IRC | 14:10 | |
| openstack | Launchpad bug 1734134 in tripleo "Pike periodic promotion job multinode-1ctlr-featureset016 fail with error running docker 'gnocchi_db_sync' - rados.Rados.connect PermissionDeniedError: error connecting to the cluster" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 14:10 |
| gfidente | anyway looks like the answer is, not right now | 14:10 |
| gfidente | I think we can move on | 14:10 |
| mwhahaha | ok thanks | 14:10 |
| gfidente | emitting hieradata | 14:10 |
| gfidente | only works with puppet | 14:11 |
| gfidente | maybe we can append ansible vars into a playbook in the future, not sure | 14:11 |
| mwhahaha | no it works beyond that | 14:11 |
| mwhahaha | because you can query hieradata externally | 14:11 |
| marios | we are still applying puppet and the hiera can still be quieried | 14:11 |
| gfidente | yeah so that probably is fine if we move the logic in the playbook vs heat | 14:12 |
| gfidente | (heat templates) | 14:12 |
| mwhahaha | well you need to set the fact that it is containerized vs not in the THT | 14:13 |
| mwhahaha | then how that information gets consumed in the deployment can live in ansible/puppet | 14:13 |
| jtomasek_ | o/ | 14:13 |
| *** pcaruana has quit IRC | 14:13 | |
| mwhahaha | it seems that writing out a hash in hiera of the containerized services that can be queried might be the best way at the moment | 14:13 |
| openstackgerrit | Merged openstack/python-tripleoclient master: Deploy with --config-download even when some role has count 0 https://review.openstack.org/523136 | 14:13 |
| matbu | o/ | 14:14 |
| mwhahaha | anyway moving on | 14:14 |
| mwhahaha | #topic Squad status | 14:15 |
| mwhahaha | ci | 14:15 |
| mwhahaha | #link https://etherpad.openstack.org/p/tripleo-ci-squad-meeting | 14:15 |
| mwhahaha | upgrade | 14:15 |
| mwhahaha | #link https://etherpad.openstack.org/p/tripleo-upgrade-squad-status | 14:15 |
| mwhahaha | containers | 14:15 |
| mwhahaha | #link https://etherpad.openstack.org/p/tripleo-containers-squad-status | 14:15 |
| *** openstack changes topic to "Squad status (Meeting topic: tripleo)" | 14:15 | |
| mwhahaha | integration | 14:15 |
| mwhahaha | #link https://etherpad.openstack.org/p/tripleo-integration-squad-status | 14:15 |
| mwhahaha | ui/cli | 14:15 |
| mwhahaha | #link https://etherpad.openstack.org/p/tripleo-ui-cli-squad-status | 14:15 |
| mwhahaha | validations | 14:15 |
| mwhahaha | #link https://etherpad.openstack.org/p/tripleo-validations-squad-status | 14:15 |
| mwhahaha | networking | 14:15 |
| mwhahaha | #link https://etherpad.openstack.org/p/tripleo-networking-squad-status | 14:15 |
| mwhahaha | workflows | 14:15 |
| mwhahaha | #link https://etherpad.openstack.org/p/tripleo-workflows-squad-status | 14:15 |
| mwhahaha | i see folks updating their status now :D | 14:15 |
| *** masco has quit IRC | 14:15 | |
| *** jmelvin has joined #tripleo | 14:16 | |
| mwhahaha | any other status related items that folks want to raise attention on? | 14:16 |
| *** pcaruana has joined #tripleo | 14:17 | |
| *** Goneri has joined #tripleo | 14:18 | |
| mwhahaha | sounds like nope | 14:19 |
| mwhahaha | #topic bugs & blueprints | 14:19 |
| mwhahaha | #link https://launchpad.net/tripleo/+milestone/queens-2 | 14:19 |
| mwhahaha | For Queens we currently have 71 (+1) blueprints and about 541 (+19) open bugs. 256 queens-2 and 285 queens-3. | 14:19 |
| *** openstack changes topic to "bugs & blueprints (Meeting topic: tripleo)" | 14:19 | |
| mwhahaha | so a reminder, queens-2 ends next week | 14:19 |
| mwhahaha | please get your blueprints updated with their current status | 14:20 |
| mwhahaha | remember we want features merged by the end of queens-2 | 14:20 |
| mwhahaha | considering we have ~541 open bugs, we shouldn't be continuing to add features in queens-3 | 14:20 |
| EmilienM | what did we say about CI changes? | 14:20 |
| mwhahaha | which ci changes | 14:21 |
| EmilienM | I'm worried about what kind of change in CI can we make after m-2 | 14:21 |
| openstackgerrit | Keith Schincke proposed openstack/tripleo-heat-templates master: Add ceph-rbdmirror ansible container service https://review.openstack.org/520244 | 14:21 |
| *** yolanda has quit IRC | 14:21 | |
| mwhahaha | EmilienM: so if it's an addition, i think those are fine. I'm not sure which types of changes you're planning. are you talking about the ovb stuff? | 14:22 |
| EmilienM | no, the ovb stuff will be fine by end of next week. | 14:22 |
| EmilienM | I'm interested by the scenarios and undercloud-container | 14:22 |
| mwhahaha | you can start planning them but I'm not sure we should switch to that | 14:22 |
| EmilienM | weshay and Slower have some WIP but I'm afraid we won't make the m-2 schedule | 14:22 |
| mwhahaha | i still wanted the undercloud container jobs by m2 | 14:23 |
| * mwhahaha pokes dprince, weshay & Slower | 14:23 | |
| EmilienM | I propose to re-discuss when the work has been done | 14:23 |
| EmilienM | we can use 2 weeks and observe stability numbers | 14:23 |
| EmilienM | and take a decision afterward | 14:23 |
| mwhahaha | sure | 14:23 |
| mwhahaha | I think for things that are close to being done by m2 i'd be ok letting slip ~2weeks | 14:24 |
| mwhahaha | but that's it | 14:24 |
| EmilienM | I agree | 14:24 |
| mwhahaha | so as a reminder for folks who have open reviews for blueprints and features, get your status updated and be able to report how close to being done | 14:24 |
| EmilienM | it's a trade-off if we want to release on time and on good conditions | 14:24 |
| dprince | mwhahaha: I think we are close | 14:24 |
| mwhahaha | dprince: sounds good, if you need reviews plz ping us | 14:25 |
| mwhahaha | any other bugs/bluepritn items? | 14:26 |
| *** lblanchard has joined #tripleo | 14:26 | |
| jkilpatr | when a overcloud is being deployed if a node gets stuck in wait-call-back is there a retry on that now? | 14:27 |
| mwhahaha | jkilpatr: depends | 14:27 |
| mwhahaha | #topic projects releases or stable backports | 14:27 |
| *** openstack changes topic to "projects releases or stable backports (Meeting topic: tripleo)" | 14:27 | |
| mwhahaha | queens-2 next week | 14:27 |
| mwhahaha | any backports that need attention? | 14:27 |
| shardy | https://review.openstack.org/#/c/522803/ needs a review please, and I'd like to land/backport https://review.openstack.org/#/c/513450/ as that's a regression for pike | 14:28 |
| *** rbrady-afk is now known as rbrady | 14:28 | |
| flaper87 | bogdando: shardy jistr http://logs.openstack.org/51/521951/17/check/ansible-role-k8s-keystone-kubernetes-centos/8511f86/job-output.txt.gz#_2017-11-28_13_15_58_711025 T_T | 14:29 |
| *** janki has quit IRC | 14:29 | |
| flaper87 | this is the iptable rules for that job: http://logs.openstack.org/51/521951/17/check/ansible-role-k8s-keystone-kubernetes-centos/8511f86/primary/logs/iptables.txt.gz | 14:30 |
| EmilienM | shardy: ack | 14:30 |
| flaper87 | no idea why dns doesn't work there | 14:30 |
| bogdando | flaper87: dns dns dns | 14:30 |
| mwhahaha | any other backport items? | 14:30 |
| bogdando | again dns | 14:30 |
| flaper87 | interestingly enough, 2 of the dns containers run http://logs.openstack.org/51/521951/17/check/ansible-role-k8s-keystone-kubernetes-centos/8511f86/primary/logs/k8s-describe-all.txt.gz | 14:30 |
| flaper87 | bogdando: yeah, figured as much but not sure why it doesn't work :( | 14:31 |
| mwhahaha | flaper87: we're in a meeting | 14:31 |
| mwhahaha | moving on to specs | 14:31 |
| mwhahaha | #topic specs | 14:31 |
| mwhahaha | #link https://review.openstack.org/#/q/project:openstack/tripleo-specs+status:open | 14:31 |
| *** openstack changes topic to "specs (Meeting topic: tripleo)" | 14:31 | |
| flaper87 | mwhahaha: oh man, so sorry :( | 14:31 |
| mwhahaha | so reminder, we will be freezing the specs next week | 14:31 |
| mwhahaha | please review open specs | 14:31 |
| mwhahaha | we don't have that many so it should be trivial | 14:32 |
| mwhahaha | the important one for jaosorior is https://review.openstack.org/521727 | 14:32 |
| *** lblanchard has quit IRC | 14:32 | |
| mwhahaha | please take a second to review the ipsec spec | 14:32 |
| mwhahaha | any other spec related topics? | 14:33 |
| EmilienM | ipsec for queens? | 14:33 |
| EmilienM | hum | 14:33 |
| mwhahaha | that's the hope, but we'll see | 14:34 |
| EmilienM | like the spec was pushed 8 days again, (in the middle of QUeens) | 14:34 |
| EmilienM | s/again/ago/ | 14:34 |
| EmilienM | I thought we were doing better at planning | 14:34 |
| EmilienM | to me, a spec added in the middle of a cycle is for the next cycle | 14:34 |
| EmilienM | our experience should help to make a better planning | 14:34 |
| shardy | Hey I've been working on some interface tweaks to enable multiple compute-only stacks | 14:34 |
| *** yolanda has joined #tripleo | 14:35 | |
| shardy | it's been possible for a while, but I'm trying to make it easier - wasn't planning a spec but can do one if folks want one? | 14:35 |
| mwhahaha | EmilienM: i agree but it is was it is | 14:35 |
| shardy | it's not really a feature, more an improved interface I think | 14:36 |
| EmilienM | shardy: specs aren't required all the times, it's just a good way to collaborate on planning and design | 14:36 |
| mwhahaha | shardy: it would probably be beneficial to have one. is the thought to support cells or something? | 14:36 |
| shardy | mwhahaha: Initially it's just to enable potentially easier scaling where you don't want to update a single heat stack with 2000 nodes in it | 14:36 |
| shardy | mwhahaha: but yeah could be a stack per cell or something in future | 14:36 |
| shardy | also it's for folks that want to scale out without touching the controlplane | 14:36 |
| *** holser has quit IRC | 14:37 | |
| *** cshastri has quit IRC | 14:37 | |
| mwhahaha | yea it wouldn't hurt to write out these use cases in a spec | 14:37 |
| shardy | mwhahaha: ack OK I'll push one today | 14:37 |
| mwhahaha | i'm also wondering how much of a deal that is as we switch to ansible driven deployment | 14:37 |
| shardy | mwhahaha: yeah that's kind of a step further, e.g deploy the controlplane w/heat then just use ansible to configure the computes | 14:38 |
| bogdando | flaper87: where is this test located? | 14:38 |
| shardy | but we'd still need a tool to generate the inventory in that case | 14:38 |
| mwhahaha | yea | 14:38 |
| shardy | mwhahaha: for now I'm making use of the dynamic-inventory, just so we can work out how to decouple things | 14:38 |
| mwhahaha | sounds good | 14:39 |
| shardy | https://review.openstack.org/#/q/topic:compute_only_stack2+(status:open+OR+status:merged) has the first steps anyway | 14:39 |
| mwhahaha | moving on to open discussion since we're basically doing that now :D | 14:40 |
| mwhahaha | #topic open discussion | 14:40 |
| *** openstack changes topic to "open discussion (Meeting topic: tripleo)" | 14:40 | |
| shardy | hehe yeah sorry about that :D | 14:40 |
| *** jmelvin is now known as jmelvin|bomgar | 14:40 | |
| EmilienM | we have 2 CI alerts | 14:40 |
| mwhahaha | so yea wouldn't hurt for a spec on that. would also help us track the work | 14:40 |
| EmilienM | and one of them is here for long time: https://bugs.launchpad.net/tripleo/+bug/1731063 | 14:41 |
| openstack | Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] | 14:41 |
| slagle | shardy: do you think scaling will still be an issue when driving everythng with config-download? | 14:41 |
| arxcruz | EmilienM: dalvarez is working on that | 14:41 |
| EmilienM | arxcruz: so why he's not assigned? | 14:41 |
| slagle | shardy: e.g., is the scaling issue due to the # of resources | 14:41 |
| EmilienM | when you work on a bug, put your name on it so we know something happens | 14:41 |
| arxcruz | EmilienM: he just joined this morning on that, I'll check with him | 14:41 |
| arxcruz | EmilienM: ok | 14:41 |
| shardy | slagle: yeah probably less of an issue, but I'm still not sure we'd want really huge environments deployed with a common stack for controlplane and compute nodes? | 14:42 |
| dalvarez | EmilienM, arxcruz i was asked to take a look by lpeer just in case i could see something | 14:42 |
| shardy | so this is an attempt to give some more options | 14:42 |
| EmilienM | arxcruz: it's not obvious. 8 days without any comment and no assignment | 14:42 |
| EmilienM | dalvarez: please assign yourself to the bug | 14:42 |
| openstackgerrit | Dmitry Tantsur proposed openstack/instack-undercloud master: Enable the ansible deploy interface out of box https://review.openstack.org/522568 | 14:42 |
| *** cylopez has quit IRC | 14:42 | |
| slagle | shardy: true. but i'd almost rather see us work on more native ironic and ansible interfaces to enalbe that, instead of building more reliance on Heat | 14:43 |
| slagle | shardy: just deploy some nodes with ironic, use ansible to configure them | 14:43 |
| shardy | slagle: yes we could do that, but atm we'd still need to generate the playbooks and inventory | 14:44 |
| slagle | as opposed to multiple stacks, which is going to cause issues with a lot of baked in assumptions everywhere | 14:44 |
| dtantsur | a (shameless) highlight of something that aligns well with the idea of an ansible deploy: https://blueprints.launchpad.net/tripleo/+spec/ansible-deploy | 14:44 |
| shardy | slagle: I think do do pure ansible we'd have to do some more work, e.g refactor all the heat-config things into pure ansible, and write a tool that converts all composable service templates into ansible roles | 14:44 |
| EmilienM | dalvarez: what's your launchpad ID? | 14:45 |
| dalvarez | EmilienM, not sure if i have to be the asignee anyways but i'll do it | 14:45 |
| shardy | which would be cool, but a step beyond what I was attempting | 14:45 |
| dalvarez | EmilienM, done | 14:45 |
| slagle | shardy: i don't think we'd have to take it that far. we could make what gets generated with config-download configurable via ansible directly | 14:45 |
| EmilienM | dalvarez: cool | 14:45 |
| *** gbarros has quit IRC | 14:46 | |
| slagle | shardy: or perhaps this is an opportunity to integrate with apb directly | 14:46 |
| slagle | to use those native roles | 14:46 |
| shardy | slagle: ack, yeah open to ideas but I was looking for ways to help us scale in the Queens timeframe | 14:46 |
| shardy | e.g before we move to the roles flaper87 has been working on | 14:46 |
| shardy | as atm those expect k8s etc | 14:47 |
| slagle | shardy: right, for queens this could be difficult. i'm just adverse to adding new deps on Heat around multiple stacks, etc. | 14:47 |
| slagle | as that becomes more difficult to move away from in the future | 14:48 |
| slagle | but yea for Queens, not sure what options there really are. | 14:48 |
| shardy | slagle: sure, I'm not really saying we have to use heat, only that we could deploy the controlplane with no computes, then work out what data is needed to configure the computes via ansible | 14:48 |
| mwhahaha | shardy: given that m2 is next week, i don't think we should target this for queens. i'd rather see a spec and conversations on how to approach it in rocky | 14:48 |
| shardy | slagle: config download provides a nice starting point for that, but definitely more we can do there | 14:48 |
| shardy | mwhahaha: well lets have the discussion and see where it goes I guess | 14:49 |
| mwhahaha | sure | 14:49 |
| mwhahaha | anyway any other topics? | 14:49 |
| dtantsur | another small highlight please | 14:49 |
| *** holser has joined #tripleo | 14:49 | |
| dtantsur | we're moving away from classic drivers in ironic | 14:49 |
| dtantsur | so I'm putting a lot of patches as part of https://bugs.launchpad.net/tripleo/+bug/1690185 | 14:50 |
| openstack | Launchpad bug 1690185 in tripleo "[RFE] Deprecate classic drivers" [High,In progress] - Assigned to Dmitry Tantsur (divius) | 14:50 |
| dtantsur | I'd appreciate attention to them to avoid a rush later on, when deprecation warnings start to pop up | 14:50 |
| *** aputtur_ has joined #tripleo | 14:50 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/pike: Ensure os-net-config conditional for upgrade doesn’t fail. https://review.openstack.org/523073 | 14:50 |
| dtantsur | (or when we actually pull the trigger next cycle (?)) | 14:50 |
| *** rbowen has joined #tripleo | 14:50 | |
| dtantsur | thanks | 14:50 |
| EmilienM | you mean, you need reviews? | 14:50 |
| dtantsur | yep, I need reviews | 14:51 |
| dtantsur | it's a lot of small patches to ~ all OoO projects | 14:51 |
| *** salmankhan has quit IRC | 14:51 | |
| EmilienM | dtantsur: we like to create etherpads in this kind of situation | 14:51 |
| *** brault has quit IRC | 14:51 | |
| EmilienM | dtantsur: so people can follow the WiP | 14:52 |
| dtantsur | good point, I'll get you one | 14:52 |
| EmilienM | dtantsur: send it to ML, people will help | 14:52 |
| rdopiera | in fact, we like to create etherpads in every situation | 14:52 |
| slagle | dtantsur: is all the work done for undercloud deploy and undercloud install? | 14:52 |
| dtantsur | slagle: I'm on "undercloud install" stage currently | 14:52 |
| slagle | dtantsur: i was looking at https://review.openstack.org/#/c/519300/ earlier after you pasted it | 14:52 |
| dtantsur | also some THT patches are up as well | 14:52 |
| slagle | and was wondering if you shouldn't just be making your effort focused on undercloud deploy | 14:53 |
| dtantsur | slagle: if you promise me that people switch to it in Queens in production ;) | 14:53 |
| slagle | i can't promise :) | 14:53 |
| slagle | i think it has to be in undercloud deploy though | 14:53 |
| dtantsur | I'm going to do both | 14:53 |
| slagle | ok | 14:53 |
| dtantsur | especially since it also affects ironic in the overcloud | 14:54 |
| dtantsur | ETOOIRONIC | 14:54 |
| dtantsur | I'm waiting for undercloud install bit to merge to cargo-cult it to undercloud deploy | 14:54 |
| dprince | yeah, do it in both so we have parity maybe | 14:54 |
| *** brault has joined #tripleo | 14:55 | |
| mwhahaha | ok i've got to take a kid to school, any other notable topics? | 14:55 |
| EmilienM | mwhahaha: close it, thanks | 14:55 |
| EmilienM | we can continue here | 14:56 |
| mwhahaha | thanks everyone | 14:56 |
| mwhahaha | #endmeeting | 14:56 |
| *** openstack changes topic to "CI status: GREENish, see alerts and read ML. scenario001/003 are non-voting :( | http://tripleo.org/ | https://docs.openstack.org/tripleo-docs/latest/" | 14:56 | |
| openstack | Meeting ended Tue Nov 28 14:56:19 2017 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 14:56 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/tripleo/2017/tripleo.2017-11-28-14.00.html | 14:56 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/tripleo/2017/tripleo.2017-11-28-14.00.txt | 14:56 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/tripleo/2017/tripleo.2017-11-28-14.00.log.html | 14:56 |
| *** akane has quit IRC | 14:59 | |
| *** akane_ has quit IRC | 14:59 | |
| openstackgerrit | Attila Darazs proposed openstack/tripleo-quickstart-extras master: GATE CHECK for quickstart-extras https://review.openstack.org/472607 | 15:00 |
| *** suuuper1 has quit IRC | 15:02 | |
| sshnaidm | EmilienM, don't we have scenario00*-multinode-upgrades jobs anymore? | 15:02 |
| EmilienM | sshnaidm: we should not | 15:03 |
| EmilienM | sshnaidm: and if we have let me know | 15:03 |
| sshnaidm | EmilienM, ack | 15:03 |
| *** udesale has quit IRC | 15:03 | |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-heat-templates master: Stop creating kubectl binary on undercloud https://review.openstack.org/523435 | 15:05 |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-heat-templates master: Pass private key file from parent Ansible to Kubespray https://review.openstack.org/523436 | 15:05 |
| sshnaidm | EmilienM, and "multinode-upgrades" as well? | 15:06 |
| *** abregman has quit IRC | 15:06 | |
| EmilienM | sshnaidm: overcloud upgrade jobs should run in RDO CI from what I understood | 15:06 |
| *** myoung|afk is now known as myoung | 15:07 | |
| EmilienM | sshnaidm: if you still see them, let me know | 15:07 |
| *** bfournie has quit IRC | 15:07 | |
| *** mrch has quit IRC | 15:07 | |
| sshnaidm | EmilienM, yeah, you're right, they're on rdo cloud now.. | 15:08 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: Introduce fs035, ovb-ha-ipv6 https://review.openstack.org/522615 | 15:08 |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-heat-templates master: Add validation task in docker services [Horizon] https://review.openstack.org/523438 | 15:08 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: Create ovb-ha-ipv6 and deprecate fs024 job https://review.openstack.org/522618 | 15:09 |
| *** ooolpbot has joined #tripleo | 15:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731063 | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1734134 | 15:10 |
| *** ooolpbot has quit IRC | 15:10 | |
| openstack | Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] - Assigned to Daniel Alvarez (dalvarezs) | 15:10 |
| openstack | Launchpad bug 1734134 in tripleo "Pike periodic promotion job multinode-1ctlr-featureset016 fail with error running docker 'gnocchi_db_sync' - rados.Rados.connect PermissionDeniedError: error connecting to the cluster" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 15:10 |
| openstackgerrit | Ben Nemec proposed openstack/tripleo-heat-templates master: Swap the order of stdout and stderr in debug output https://review.openstack.org/522405 | 15:11 |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-heat-templates stable/pike: Add validation task in docker services [Octavia] https://review.openstack.org/517266 | 15:12 |
| *** mdnadeem has quit IRC | 15:16 | |
| *** moshele has quit IRC | 15:16 | |
| *** ramishra has quit IRC | 15:17 | |
| openstackgerrit | Merged openstack/python-tripleoclient stable/pike: Consume a zaqar queue for update to poll ansible result https://review.openstack.org/522218 | 15:18 |
| *** ramishra has joined #tripleo | 15:19 | |
| sshnaidm | EmilienM, do you have a patch that runs ipv6 with ssl? | 15:20 |
| *** artom has quit IRC | 15:21 | |
| EmilienM | sshnaidm: https://review.openstack.org/#/c/522618/ | 15:23 |
| EmilienM | sshnaidm: and it fails. I need help | 15:23 |
| EmilienM | sshnaidm: can you please review https://review.openstack.org/#/c/522615/ ? | 15:24 |
| dtantsur | EmilienM, mwhahaha, https://etherpad.openstack.org/p/tripleo-switch-to-hardware-types it's better than I thought, but some work still to be done | 15:24 |
| EmilienM | sshnaidm: I'm unsure about the parameters | 15:24 |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Add novajoin to collect-logs.yml list https://review.openstack.org/521591 | 15:24 |
| openstackgerrit | Christian Schwede proposed openstack/tripleo-heat-templates stable/pike: Mount /var/cache/swift across containers https://review.openstack.org/523440 | 15:24 |
| sshnaidm | EmilienM, I suspect I know why, we need PublicVirtualFixedIPs there, and overcloud_public_vip6.. | 15:24 |
| *** artom has joined #tripleo | 15:25 | |
| sshnaidm | EmilienM, will look | 15:25 |
| jaosorior | sshnaidm: that would be this commit https://review.openstack.org/#/c/522714/ | 15:25 |
| *** gbarros has joined #tripleo | 15:25 | |
| sshnaidm | jaosorior, exactly, great! | 15:26 |
| EmilienM | yeah we need https://review.openstack.org/#/c/522714/ | 15:26 |
| *** d0ugal has quit IRC | 15:28 | |
| *** gbarros has quit IRC | 15:28 | |
| *** janki has joined #tripleo | 15:28 | |
| sshnaidm | jaosorior, but we need to define overcloud_public_vip6 anywhere, does it have a default? | 15:28 |
| *** gbarros has joined #tripleo | 15:29 | |
| sshnaidm | jaosorior, like here: https://github.com/openstack/tripleo-quickstart-extras/blob/8977df65d3413ea1d87b5f2dac76e7ed04811bc7/roles/overcloud-ssl/defaults/main.yml#L5 | 15:29 |
| openstackgerrit | Julie Pichon proposed openstack/tripleo-docs master: Fix typo in tunnel example (UI deployments) https://review.openstack.org/523442 | 15:29 |
| jfrancoa | hey folks, I've got some patches cherry-picked to stable/pike pending to be merged [validation-steps for upgrades], could you help reviewing them please? https://etherpad.openstack.org/p/validation-steps-pike | 15:30 |
| jfrancoa | marios: chem: matbu: social: ^^^ | 15:31 |
| jaosorior | sshnaidm: mayhaps :/ I reaaally gotta go though. Feel free to change it. Else I'll check it out tomorrow. | 15:31 |
| sshnaidm | jaosorior, sure, thanks | 15:31 |
| marios | jfrancoa: ack will give them another pass but might be tomorrow morning | 15:32 |
| jfrancoa | marios: thanks a lot man | 15:32 |
| jistr | flaper87: didn't test with dns yet, trying now but it doesn't seem to be working "out of the box" on the nodes | 15:33 |
| *** ramishra has quit IRC | 15:34 | |
| jistr | flaper87: i'm not very familiar with it, trying to look up now if it should be working from the cluster as well or just from inside the pods | 15:35 |
| *** gbarros has quit IRC | 15:38 | |
| trozet | jaosorior: hi, would you midn looking at https://review.openstack.org/#/c/515576/ I noticed you did the weboscket stuff for zaqar | 15:38 |
| *** gbarros has joined #tripleo | 15:39 | |
| jfrancoa | sshnaidm: has ther been any changes in lates upstream pike repo? the upgrades job is failing with http://mirror.dfw.rax.openstack.org:8080/rdo/centos7-pike/deps/latest/repodata/repomd.xml: [Errno 14] HTTP Error 404 - Not Found | 15:39 |
| jfrancoa | sshnaidm: https://logs.rdoproject.org/40/522540/1/openstack-check/gate-tripleo-ci-centos-7-containers-multinode-upgrades-pike/Z0a1c011e0d67427883dc8ed276b9572c/undercloud/home/jenkins/failed_upgrade.log.txt.gz | 15:40 |
| *** bfournie has joined #tripleo | 15:40 | |
| *** trown is now known as trown|brb | 15:40 | |
| sshnaidm | jfrancoa, hmm.. this is wrong link, where is it from? | 15:41 |
| jfrancoa | sshnaidm: it's from this patch https://review.openstack.org/#/c/522540/1 | 15:42 |
| *** d0ugal has joined #tripleo | 15:42 | |
| sshnaidm | jfrancoa, but where is this URL generated/set? | 15:43 |
| sshnaidm | jfrancoa, there is no such repo URL (and wasn't iirc) | 15:43 |
| *** yamahata__ has joined #tripleo | 15:44 | |
| jfrancoa | sshnaidm: oh, sorry, It's generated in this template https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/overcloud-upgrade/templates/overcloud-repo-tripleo-ci.yaml.j2 | 15:45 |
| *** isq_ has quit IRC | 15:45 | |
| flaper87 | jistr: lemme know your findings | 15:45 |
| sshnaidm | jfrancoa, oh, this famous file.. | 15:46 |
| jfrancoa | sshnaidm: to fill in the UpgradeInitiCommand, and this is the one being used in the job which failed: https://logs.rdoproject.org/40/522540/1/openstack-check/gate-tripleo-ci-centos-7-containers-multinode-upgrades-pike/Z0a1c011e0d67427883dc8ed276b9572c/undercloud/home/jenkins/overcloud-repo.yaml.txt.gz | 15:46 |
| flaper87 | I kinda want to use the dns! | 15:46 |
| jfrancoa | sshnaidm: yes...we need to get rid of it, I know... | 15:46 |
| jfrancoa | sshnaidm: but until we do, we need to have some upgrades job running | 15:46 |
| *** holser has quit IRC | 15:47 | |
| *** shreshtha_ has quit IRC | 15:49 | |
| *** holser has joined #tripleo | 15:49 | |
| sshnaidm | jfrancoa, this url works for queens only, maybe you need to add pike here: https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/overcloud-upgrade/templates/overcloud-repo-tripleo-ci.yaml.j2#L49 | 15:49 |
| sshnaidm | jfrancoa, and https://buildlogs.centos.org/centos/7/cloud/x86_64/openstack-pike/ exists, so should be ok.. | 15:50 |
| *** trown|brb is now known as trown | 15:51 | |
| *** gbarros has quit IRC | 15:52 | |
| jfrancoa | sshnaidm: cool, I will change it and try it. Thanks a lot | 15:52 |
| sshnaidm | jfrancoa, sure, np | 15:53 |
| *** gbarros has joined #tripleo | 15:53 | |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-heat-templates master: Set coe_ facts for APBs https://review.openstack.org/523451 | 16:01 |
| *** haint has joined #tripleo | 16:01 | |
| openstackgerrit | Martin André proposed openstack/tripleo-common master: Prevent apache from listening on default ports https://review.openstack.org/523404 | 16:03 |
| *** nyechiel has quit IRC | 16:04 | |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: Clone repos with zuul changes applied https://review.openstack.org/522052 | 16:04 |
| *** nguyentrihai has quit IRC | 16:05 | |
| *** marios has quit IRC | 16:06 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/ocata: Set host name explicitly for telemetry https://review.openstack.org/521943 | 16:07 |
| *** Tripleo-User has quit IRC | 16:08 | |
| *** iranzo has quit IRC | 16:09 | |
| *** iranzo has joined #tripleo | 16:09 | |
| *** iranzo has quit IRC | 16:09 | |
| *** iranzo has joined #tripleo | 16:09 | |
| jbadiapa | jaosorior, are you around? | 16:10 |
| *** ooolpbot has joined #tripleo | 16:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731063 | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1734134 | 16:10 |
| *** ooolpbot has quit IRC | 16:10 | |
| openstack | Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] - Assigned to Daniel Alvarez (dalvarezs) | 16:10 |
| openstack | Launchpad bug 1734134 in tripleo "Pike periodic promotion job multinode-1ctlr-featureset016 fail with error running docker 'gnocchi_db_sync' - rados.Rados.connect PermissionDeniedError: error connecting to the cluster" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 16:10 |
| *** gvrangan has joined #tripleo | 16:12 | |
| *** crushil has joined #tripleo | 16:13 | |
| *** dtrainor has quit IRC | 16:14 | |
| openstackgerrit | Brad P. Crochet proposed openstack/python-tripleoclient master: Display Horizon URL at the end of a deployment https://review.openstack.org/518030 | 16:21 |
| ccamacho | hey shardy around? | 16:22 |
| ccamacho | sorry for bugging you | 16:22 |
| ccamacho | related to https://bugs.launchpad.net/tripleo/+bug/1734706 | 16:22 |
| openstack | Launchpad bug 1728917 in tripleo "duplicate for #1734706 Upgrades CI jobs in master not performing upgrade" [High,Fix released] - Assigned to Carlos Camacho (ccamacho) | 16:22 |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/pike: Start sequence at 1 for deploy steps playbook https://review.openstack.org/522803 | 16:23 |
| ccamacho | mm the heat validation issue "ERROR: The template version is invalid: "heat_template_version: pike". "heat_template_version" should be one of: ... " is reproducible from older branches | 16:23 |
| ccamacho | so, in heat how can I check that the validation is working properly?? | 16:23 |
| ccamacho | jfrancoa^ | 16:23 |
| ccamacho | pradk ^ you might have some clue here?? | 16:24 |
| *** iranzo has quit IRC | 16:26 | |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-specs master: Add split-controlplane spec https://review.openstack.org/523459 | 16:27 |
| *** gbarros has quit IRC | 16:28 | |
| shardy | slagle, mwhahaha: ^^ quick first draft, feedback/help welcome | 16:29 |
| shardy | ccamacho: what branch are you using, obviously one reason for that error is using e.g pike template version on an older branch | 16:30 |
| shardy | openstack orchestration template version list tells you the versions available | 16:30 |
| mwhahaha | shardy: ccamacho: the job didn't actually run the upgrade process | 16:31 |
| mwhahaha | shardy: ccamacho: so the queens THT gets used against a pike cloud | 16:31 |
| shardy | Ok that explains it then | 16:31 |
| ccamacho | yeahp I have a pike undercloud and when executing the overcloud pingtest it fails with pike templates, same as the bug but in different branches. in ci queens fails using queens templates | 16:31 |
| mwhahaha | ccamacho: see the bug I marked that one as a dupe of | 16:31 |
| *** gkadam_ has joined #tripleo | 16:31 | |
| mwhahaha | ccamacho: the problem is that the master upgrade job doesn't actually run the upgrade | 16:31 |
| mwhahaha | ccamacho: https://bugs.launchpad.net/tripleo/+bug/1728917 | 16:33 |
| openstack | Launchpad bug 1728917 in tripleo "Upgrades CI jobs in master not performing upgrade" [High,Fix released] - Assigned to Carlos Camacho (ccamacho) | 16:33 |
| mwhahaha | quickstart isn't actually running any of the upgrade processes, it just loads teh containers and that's it | 16:33 |
| *** cmyster has left #tripleo | 16:33 | |
| *** gkadam has quit IRC | 16:34 | |
| ccamacho | mwhahaha oki thanks for the clarification | 16:34 |
| *** yprokule has quit IRC | 16:34 | |
| *** aufi has quit IRC | 16:36 | |
| raildo | shardy, that patch was merged, I hope that unblock you guys soon :) sorry about that anyway | 16:36 |
| EmilienM | sshnaidm: any comment so far on ipv6 things? | 16:37 |
| *** lblanchard has joined #tripleo | 16:37 | |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart-extras master: Add ui_validate_simple to the logs collected https://review.openstack.org/523461 | 16:38 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: ci: add ovb-ha.yaml https://review.openstack.org/522306 | 16:42 |
| EmilienM | mwhahaha: is that better^ ? | 16:43 |
| *** agurenko has quit IRC | 16:43 | |
| mwhahaha | EmilienM: i think so :D though not sure why we specify that and in docker-ha.yaml | 16:43 |
| *** jmelvin|bomgar has quit IRC | 16:43 | |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient master: Remove the overcloudrc.v3 file https://review.openstack.org/521941 | 16:43 |
| EmilienM | mwhahaha: not sure we load docker-ha.yaml | 16:43 |
| mwhahaha | EmilienM: We do | 16:44 |
| mwhahaha | EmilienM: i checked the dependent job | 16:44 |
| EmilienM | mwhahaha: I guess for now it doesn't hurt, I'll eventually clean that up later | 16:44 |
| *** oidgar has joined #tripleo | 16:45 | |
| EmilienM | mwhahaha: or I could remove it? | 16:45 |
| mwhahaha | EmilienM: for the record, i hate using parameter_defaults to manage services | 16:45 |
| mwhahaha | EmilienM: I think it should be removed if it's already defined in docker-ha.yaml, we shouldn't dupe it | 16:46 |
| EmilienM | mwhahaha: ok | 16:46 |
| *** ffiore has quit IRC | 16:46 | |
| EmilienM | OS::TripleO::Tasks::ControllerPreConfig is set to OS::Heat::None | 16:47 |
| EmilienM | and not extraconfig/tasks/pre_puppet_pacemaker.yaml | 16:47 |
| EmilienM | bandini: ^ is it a big deal? | 16:47 |
| EmilienM | in docker-ha.yaml | 16:47 |
| *** ed_b has quit IRC | 16:48 | |
| *** ed_b has joined #tripleo | 16:49 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: ci: add ovb-ha.yaml https://review.openstack.org/522306 | 16:49 |
| EmilienM | mwhahaha: the only reason why I created ci/environments/ovb-ha.yaml is because by default we install too much services and I want a very minimum ovb scenario | 16:50 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Select first node as bootstrap node not using name https://review.openstack.org/513450 | 16:50 |
| mwhahaha | EmilienM: i know but the correct way to do that is construct appropriate roles. i know why we did it, i just disagree with it | 16:50 |
| openstackgerrit | Merged openstack/tripleo-common stable/pike: config: Always add step conditional first for upgrade_tasks https://review.openstack.org/522831 | 16:50 |
| *** ed_b has quit IRC | 16:52 | |
| *** atoth has quit IRC | 16:54 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: Run ovb-ha with minimal services https://review.openstack.org/522310 | 16:54 |
| *** hewbrocca is now known as hewbrocca_afk | 16:55 | |
| *** atoth has joined #tripleo | 16:58 | |
| EmilienM | mwhahaha: would you see one role per scenario then ? | 17:00 |
| *** trown is now known as trown|lunch | 17:00 | |
| mwhahaha | EmilienM: the whole point of the role files is so that people can reuse what we test. when we cheat and override them via parameter_defaults, others can't reuse it. so if this is a valid scenario that folks might actually deploy then a role file is the correct place to define that list of services | 17:01 |
| mwhahaha | EmilienM: I'm not -2 on it, i just don't like this practice as we aren't actually exercising what we expect customers to do | 17:01 |
| EmilienM | mwhahaha: that's something we really do, I agree | 17:01 |
| EmilienM | mwhahaha: I agree, I could work on that in the next steps | 17:02 |
| mwhahaha | EmilienM: CI is a bit where in this aspect because we are so resource constrainted | 17:02 |
| mwhahaha | EmilienM: so we end up faking it this way but for OVB we're less so than multinode | 17:02 |
| EmilienM | mwhahaha: I guess I could re-do https://review.openstack.org/522306 and create 2 roles, ControllerMini and ComputeMini | 17:04 |
| mwhahaha | EmilienM: i'd just create a tech-debt bug because we also don't support role data generation in quickstart | 17:04 |
| mwhahaha | EmilienM: so tehre's more work that needs to be done | 17:04 |
| *** suuuper has joined #tripleo | 17:07 | |
| *** moshele has joined #tripleo | 17:08 | |
| *** Lokesh_Jain__ has joined #tripleo | 17:08 | |
| EmilienM | mwhahaha: I'm creating the RFE now | 17:09 |
| *** ooolpbot has joined #tripleo | 17:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731063 | 17:10 |
| openstack | Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] - Assigned to Daniel Alvarez (dalvarezs) | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1734134 | 17:10 |
| *** ooolpbot has quit IRC | 17:10 | |
| openstack | Launchpad bug 1734134 in tripleo "Pike periodic promotion job multinode-1ctlr-featureset016 fail with error running docker 'gnocchi_db_sync' - rados.Rados.connect PermissionDeniedError: error connecting to the cluster" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 17:10 |
| *** sai_ has joined #tripleo | 17:10 | |
| *** gvrangan has quit IRC | 17:11 | |
| chem | rasca: hey, maybe you know, how do I check the status of the RDO ci for https://review.openstack.org/#/c/523408/ ? | 17:12 |
| chem | rasca: it's been hours and still no output is why I'm trying to find out. | 17:12 |
| *** moshele has quit IRC | 17:12 | |
| *** ccamacho has quit IRC | 17:12 | |
| chem | jfrancoa: yeah, depressing rigth :) | 17:12 |
| chem | jfrancoa: is the tag recheck experimental rdo ... ? | 17:13 |
| jfrancoa | chem: hahaha yes, I went to check it twice :-) | 17:13 |
| *** dougbtv_ has joined #tripleo | 17:13 | |
| openstackgerrit | Merged openstack/paunch stable/pike: zuul: change OVB job layout https://review.openstack.org/523280 | 17:13 |
| *** nyechiel has joined #tripleo | 17:13 | |
| EmilienM | mwhahaha: https://bugs.launchpad.net/tripleo/+bug/1734947 could you please review it ? | 17:13 |
| openstack | Launchpad bug 1734947 in tripleo "RFE: support role data generation in quickstart" [Medium,Triaged] | 17:13 |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-quickstart-extras master: WIP: Fix SSL certs creation for ipv6 https://review.openstack.org/523477 | 17:13 |
| sshnaidm | EmilienM, ^^ | 17:13 |
| jfrancoa | chem: and still no RDO ci results..I think it depends on the repositories you are modifying (it's a guess from my experience) | 17:14 |
| EmilienM | sshnaidm: looking | 17:14 |
| sshnaidm | EmilienM, oops, fixing.. | 17:14 |
| *** agurenko has joined #tripleo | 17:15 | |
| *** jmelvin|bomgar has joined #tripleo | 17:15 | |
| EmilienM | sshnaidm: don't you want to squash with jaosorior's patch? | 17:15 |
| *** crushil has quit IRC | 17:15 | |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-quickstart-extras master: WIP: Fix SSL certs creation for ipv6 https://review.openstack.org/523477 | 17:15 |
| sshnaidm | EmilienM, mm..it's a little bit different approach, let's see if it works firstly | 17:16 |
| *** crushil has joined #tripleo | 17:16 | |
| *** gkadam_ has quit IRC | 17:16 | |
| EmilienM | sshnaidm: in that case, please rebase on top of jaosorior's patch so I can easily use depends-on | 17:16 |
| *** gvrangan has joined #tripleo | 17:16 | |
| EmilienM | sshnaidm: can you please take a look at https://review.openstack.org/#/c/522615/11/config/general_config/featureset035.yml ? | 17:16 |
| sshnaidm | EmilienM, can you maybe to duplicate your patch with depending on that? | 17:17 |
| sshnaidm | EmilienM, yeah, will look | 17:17 |
| *** gvrangan has quit IRC | 17:17 | |
| *** gvrangan has joined #tripleo | 17:18 | |
| *** zshi has quit IRC | 17:20 | |
| EmilienM | sshnaidm: duplicate what? | 17:20 |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart-extras master: Add ui_validate_simple to the logs collected https://review.openstack.org/523461 | 17:20 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart-extras master: WIP: Fix SSL certs creation for ipv6 https://review.openstack.org/523477 | 17:21 |
| EmilienM | sshnaidm: ^ rebased | 17:21 |
| EmilienM | so I can depends-on | 17:21 |
| *** crushil has quit IRC | 17:22 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: Introduce fs035, ovb-ha-ipv6 https://review.openstack.org/522615 | 17:22 |
| sshnaidm | EmilienM, mwhahaha I'm not sure 522714 will work.. | 17:22 |
| sshnaidm | I'd wait to merge it | 17:22 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: Create ovb-ha-ipv6 and deprecate fs024 job https://review.openstack.org/522618 | 17:22 |
| *** gvrangan has quit IRC | 17:22 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: Create ovb-ha-ipv6 and deprecate fs024 job https://review.openstack.org/522618 | 17:22 |
| mwhahaha | sshnaidm: ok i'll remove the +A | 17:22 |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-quickstart-extras master: Add pike as release using old deps repo. https://review.openstack.org/523480 | 17:22 |
| EmilienM | sshnaidm: ^ I updated the Depends-On | 17:23 |
| EmilienM | to Depends-On your patch, which I rebased on top of Juan's patch | 17:23 |
| sshnaidm | EmilienM, so you will have two changes together? | 17:24 |
| *** gbarros has joined #tripleo | 17:25 | |
| *** jfrancoa has quit IRC | 17:27 | |
| EmilienM | sshnaidm: these are the changes: https://review.openstack.org/#/q/topic:fs035+(status:open+OR+status:merged) | 17:31 |
| EmilienM | sshnaidm: I added the t-q-e to the topic | 17:31 |
| sshnaidm | EmilienM, I mean that my change and Juans are not compatible | 17:32 |
| *** zshi has joined #tripleo | 17:32 | |
| EmilienM | sshnaidm: oh ok, I can rebase yours on master and only test yours, how does it sound? | 17:32 |
| sshnaidm | EmilienM, yeah, exactly | 17:32 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart-extras master: WIP: Fix SSL certs creation for ipv6 https://review.openstack.org/523477 | 17:32 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: Create ovb-ha-ipv6 and deprecate fs024 job https://review.openstack.org/522618 | 17:33 |
| EmilienM | sshnaidm: done | 17:33 |
| sshnaidm | EmilienM, great, let's see.. | 17:33 |
| *** dougbtv_ has quit IRC | 17:33 | |
| *** shardy has quit IRC | 17:37 | |
| *** florianf is now known as florianf|afk | 17:37 | |
| *** gbarros has quit IRC | 17:41 | |
| jlabarre | on a tripleo-quickstart installation, how do I see the information on the tripleo VMs with virsh? I wanted to get a "dumpxml" on them, but virsh list (under root OR stack) isn't seeing them | 17:41 |
| openstackgerrit | Sergii Golovatiuk proposed openstack-infra/tripleo-ci master: Introduce TRIPLEO_HEAT_TEMPLATES_ROOT https://review.openstack.org/523488 | 17:42 |
| *** fragatina has quit IRC | 17:43 | |
| *** thrash is now known as thrash|biab | 17:44 | |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart-extras master: Add ui_validate_simple to the logs collected https://review.openstack.org/523461 | 17:44 |
| *** gbarros has joined #tripleo | 17:44 | |
| *** yamahata has quit IRC | 17:47 | |
| *** dtantsur is now known as dtantsur|afk | 17:50 | |
| *** holser has quit IRC | 17:51 | |
| *** pcaruana has quit IRC | 17:53 | |
| *** holser has joined #tripleo | 17:53 | |
| *** holser has quit IRC | 17:54 | |
| *** moshele has joined #tripleo | 17:55 | |
| *** ebarrera has quit IRC | 17:56 | |
| *** bogdando has quit IRC | 17:56 | |
| *** pblaho has quit IRC | 17:56 | |
| *** sshnaidm is now known as sshnaidm|off | 17:59 | |
| openstackgerrit | Justin Kilpatrick proposed openstack/tripleo-specs master: rsyslog remote logging https://review.openstack.org/523493 | 18:00 |
| *** trown|lunch is now known as trown | 18:01 | |
| *** moshele has quit IRC | 18:01 | |
| *** tosky has quit IRC | 18:03 | |
| *** gfidente has quit IRC | 18:04 | |
| *** derekh has quit IRC | 18:04 | |
| *** thrash|biab is now known as thrash | 18:05 | |
| *** thrash is now known as thrash|biab | 18:06 | |
| *** nyechiel has quit IRC | 18:08 | |
| *** gvrangan has joined #tripleo | 18:08 | |
| openstackgerrit | Keith Schincke proposed openstack/tripleo-common master: Add support for ceph_rbdmirror_ansible_vars from tht https://review.openstack.org/520658 | 18:10 |
| *** ooolpbot has joined #tripleo | 18:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731063 | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1734134 | 18:10 |
| *** ooolpbot has quit IRC | 18:10 | |
| openstack | Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] - Assigned to Daniel Alvarez (dalvarezs) | 18:10 |
| openstack | Launchpad bug 1734134 in tripleo "Pike periodic promotion job multinode-1ctlr-featureset016 fail with error running docker 'gnocchi_db_sync' - rados.Rados.connect PermissionDeniedError: error connecting to the cluster" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 18:10 |
| *** gvrangan has quit IRC | 18:10 | |
| *** almondjoy has joined #tripleo | 18:10 | |
| mwhahaha | dprince: how can I specify a specific starting order of containers within a step? ie neutron server before metadata agent | 18:10 |
| *** crushil has joined #tripleo | 18:11 | |
| *** gbarros has quit IRC | 18:11 | |
| *** mcornea has quit IRC | 18:12 | |
| dprince | http://git.openstack.org/cgit/openstack/tripleo-heat-templates/tree/docker/services/keystone.yaml#n158 | 18:12 |
| dprince | mwhahaha: ^ | 18:12 |
| *** crushil_ has joined #tripleo | 18:12 | |
| mwhahaha | dprince: so i could do service as start_order: 1 and all the agents as start_order: 2? and it should work? | 18:13 |
| openstackgerrit | Numan Siddique proposed openstack/tripleo-quickstart master: Enable OVN HA profile in fset28 https://review.openstack.org/499570 | 18:15 |
| mwhahaha | or does that not work across service files | 18:15 |
| dprince | mwhahaha: yep, should be that easy | 18:15 |
| * mwhahaha gives it a shot | 18:15 | |
| dprince | mwhahaha: start_order only guarantees on each host | 18:15 |
| dprince | mwhahaha: step is cluster wide | 18:15 |
| mwhahaha | dprince: thats fine | 18:15 |
| mwhahaha | dprince: we lost service start order relationships that we used to have with puppet | 18:16 |
| mwhahaha | that we just got for free | 18:16 |
| *** dsariel has quit IRC | 18:16 | |
| dprince | mwhahaha: yep, we did the best we could. Our mechanism is simple but it can work | 18:17 |
| *** crushil has quit IRC | 18:17 | |
| mwhahaha | yup it seems to work for the most part | 18:17 |
| *** dtrainor has joined #tripleo | 18:17 | |
| *** dtrainor has quit IRC | 18:18 | |
| *** dtrainor has joined #tripleo | 18:18 | |
| *** gvrangan has joined #tripleo | 18:19 | |
| openstackgerrit | Martin André proposed openstack/tripleo-common master: Prevent apache from listening on default ports https://review.openstack.org/523404 | 18:19 |
| *** fragatina has joined #tripleo | 18:19 | |
| *** suuuper has quit IRC | 18:20 | |
| *** gbarros has joined #tripleo | 18:21 | |
| *** yamahata has joined #tripleo | 18:22 | |
| *** lucasagomes is now known as lucas-afk | 18:23 | |
| *** liverpooler has quit IRC | 18:25 | |
| *** gvrangan has quit IRC | 18:27 | |
| *** liverpooler has joined #tripleo | 18:28 | |
| *** suuuper has joined #tripleo | 18:28 | |
| *** dhill_ has quit IRC | 18:33 | |
| *** gvrangan has joined #tripleo | 18:33 | |
| *** moshele has joined #tripleo | 18:35 | |
| *** dhill_ has joined #tripleo | 18:39 | |
| *** gvrangan has quit IRC | 18:42 | |
| *** abregman has joined #tripleo | 18:42 | |
| *** nyechiel has joined #tripleo | 18:45 | |
| *** dpawar has quit IRC | 18:45 | |
| *** dtrainor has quit IRC | 18:47 | |
| openstackgerrit | mathieu bultel proposed openstack/python-tripleoclient master: Catch zaqar exception when no message to claim https://review.openstack.org/523500 | 18:48 |
| *** gvrangan has joined #tripleo | 18:48 | |
| dmsimard | mwhahaha, EmilienM: not sure who this would be interesting to but you would probably know... it turns out that cliff (the backend CLI python module for openstackclient -- and tripleoclient) has a sphinx extension to automatically document things: https://docs.openstack.org/cliff/latest/user/sphinxext.html | 18:49 |
| thrash|biab | dmsimard: that is interesting. | 18:50 |
| *** thrash|biab is now known as thrash | 18:51 | |
| dmsimard | thrash: yeah, my understanding is that it's what they use to generate https://docs.openstack.org/python-openstackclient/latest/cli/man/openstack.html for example | 18:51 |
| dmsimard | Who doesn't like free and up to date docs with no maintenance ? | 18:52 |
| mwhahaha | the guy who ends up having to support that free doc generation configuration :D | 18:52 |
| mwhahaha | is not free, just slightly less expensive | 18:53 |
| dmsimard | mwhahaha: it doesn't seem that bad tbh | 18:53 |
| * mwhahaha points to all the reno & docs failures due to deps changing | 18:53 | |
| *** gvrangan has quit IRC | 18:53 | |
| mwhahaha | but yea, would be interesting to see what it spits out | 18:54 |
| dmsimard | mwhahaha: hey because of reno you could publish your puppet modules to pypi | 18:59 |
| *** dtrainor has joined #tripleo | 18:59 | |
| mwhahaha | ಠ_ಠ| 18:59 |
| *** moshele has quit IRC | 19:01 | |
| dmsimard | gem install puppet; pip install puppet-nova; puppet apply "::nova" | 19:02 |
| *** gvrangan has joined #tripleo | 19:03 | |
| *** gvrangan has quit IRC | 19:03 | |
| mwhahaha | on a side note, it appears we are running the neutron-ovs agent in step4 instead of step5 now | 19:03 |
| *** suuuper has quit IRC | 19:04 | |
| *** gvrangan has joined #tripleo | 19:04 | |
| *** janki has quit IRC | 19:05 | |
| *** gvrangan has quit IRC | 19:08 | |
| *** ooolpbot has joined #tripleo | 19:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731063 | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1734134 | 19:10 |
| *** ooolpbot has quit IRC | 19:10 | |
| openstack | Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] - Assigned to Daniel Alvarez (dalvarezs) | 19:10 |
| openstack | Launchpad bug 1734134 in tripleo "Pike periodic promotion job multinode-1ctlr-featureset016 fail with error running docker 'gnocchi_db_sync' - rados.Rados.connect PermissionDeniedError: error connecting to the cluster" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 19:10 |
| *** oidgar has quit IRC | 19:20 | |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-heat-templates master: Fix neutron agent start order https://review.openstack.org/523508 | 19:20 |
| *** nyechiel has quit IRC | 19:20 | |
| *** dpawar_ has joined #tripleo | 19:24 | |
| *** tosky has joined #tripleo | 19:29 | |
| openstackgerrit | Brad P. Crochet proposed openstack/python-tripleoclient master: [WIP] Use cliff autodoc generation https://review.openstack.org/523510 | 19:29 |
| thrash | dmsimard: mwhahaha ^^^ | 19:30 |
| thrash | I couldn't resist. :) | 19:30 |
| mwhahaha | fine by me :D | 19:30 |
| * mwhahaha waits for the output | 19:30 | |
| dmsimard | thrash: sweet, that looked easy enough | 19:30 |
| *** Goneri has quit IRC | 19:31 | |
| thrash | dmsimard: thanks for the pointer to it | 19:31 |
| openstackgerrit | Ian Main proposed openstack/python-tripleoclient master: Generate undercloud-passwords.conf and fix output dir. https://review.openstack.org/523511 | 19:33 |
| *** pkovar has quit IRC | 19:34 | |
| openstackgerrit | Ian Main proposed openstack/tripleo-quickstart master: DNM: Update undercloud install options for containers https://review.openstack.org/517445 | 19:35 |
| *** links has joined #tripleo | 19:38 | |
| *** links has quit IRC | 19:42 | |
| beagles | mwhahaha, is running the ovs agent in step 4 breaking stuff? | 19:43 |
| mwhahaha | beagles: see related bug, we've had issues in the past in multinode deployments where it caused problems | 19:43 |
| beagles | mwhahaha, we've been discussing this because step 5 causes problem with octavia deployment and we haven't yet worked out why the OVS agent would be affected | 19:44 |
| mwhahaha | beagles: essentially it's an service start ordering regression from the containerization | 19:44 |
| mwhahaha | beagles: what problems do you see in octavia? | 19:44 |
| dmsimard | thrash: there's no job that generates the docs? :/ | 19:45 |
| beagles | mwhahaha, yeah, we are aware because octavia works with containers and not baremetal. This is because the workflows in step 5 run before the services are run. The octavia deployment creates neutron ports which will fail if a properly configured OVS agent isn't in place. In containers we are (or were) okay because the ovs agent was already running | 19:46 |
| mwhahaha | beagles: sounds like we need a step6 :D | 19:47 |
| beagles | mwhahaha, we sorted that out last week and are looking at moving to external deployment tasks which we can trigger after step 5 | 19:47 |
| beagles | mwhahaha, it's a bit late in the cycle but if we gotta we gotta | 19:47 |
| mwhahaha | beagles: so i'm not completely sure if we need to move it to step5 or if just making sure neutron was started is sufficient but I was airing on the side of it used to be like this for 2+ cycles | 19:47 |
| mwhahaha | beagles: but i do think we have issues with service start order after poking around stable/ocata scenario001 vs pike/master scenario001 | 19:48 |
| mwhahaha | beagles: so if you want to check out the related bugs and give feedback that'd be OK | 19:49 |
| beagles | mwhahaha, yeah - in some respects it seems more likely that starting the l3 and dhcp agents before the ovs agent is problematic - but really shouldn't be a problem unless there are networks/ports already created | 19:49 |
| mwhahaha | beagles: https://github.com/openstack/puppet-tripleo/commit/bb63f514d22ea82d17947a5972b4da16e66b5a36#diff-1342e6e1493d4b3510ef492cc372e875 was the original change | 19:49 |
| beagles | mwhahaha, yup | 19:49 |
| beagles | mwhahaha, I wish I knew for sure .. | 19:49 |
| mwhahaha | but it's good to know that the ld/dhcp agents might have problems. so perhaps we should line it up neutron-server -> ovs -> other agents | 19:50 |
| beagles | mwhahaha, the safest bet is for me to buckle down and get this converted to external deployment steps like *right now* | 19:50 |
| mwhahaha | yea | 19:50 |
| mwhahaha | beagles: we do see message timesouts today on initial startup, http://logs.openstack.org/56/519756/5/check/legacy-tripleo-ci-centos-7-scenario003-multinode-oooq-container/e2bd239/logs/subnode-2/var/log/containers/neutron/neutron-openvswitch-agent.log.txt.gz#_2017-11-18_02_56_23_150 | 19:51 |
| mwhahaha | beagles: so this is kinda why i was adjusting the service startup because they all start at the same time which may be causing problems | 19:51 |
| mwhahaha | actually the agents get started first | 19:52 |
| mwhahaha | before neutron-server | 19:52 |
| beagles | really? | 19:52 |
| mwhahaha | yea | 19:52 |
| beagles | that's probably wrong :) | 19:52 |
| mwhahaha | right so my patch should fix that part :D | 19:52 |
| beagles | although it shouldn't "break" anything it just seems "unright" :) | 19:53 |
| mwhahaha | that ordering was handled in puppet-tripleo and kinda got lost | 19:53 |
| mwhahaha | my experiance is that "shouldn't" in openstack means: yea it's not going to work :D | 19:53 |
| beagles | yeah, R.I.P. puppet service lifecycle management | 19:53 |
| beagles | heheh, true | 19:53 |
| openstackgerrit | David Moreau Simard proposed openstack/python-tripleoclient master: Add jobs to build and publish python-tripleoclient docs https://review.openstack.org/523516 | 19:54 |
| dmsimard | thrash: ^ just by curiosity | 19:54 |
| dmsimard | it's rebased on top of your patch | 19:54 |
| dmsimard | if it works you can rebase on top of mine instead or something | 19:54 |
| mwhahaha | speaking of lost things, bandini where did the rabbitmq logs go in container land | 19:55 |
| mwhahaha | also mariadb logs | 19:56 |
| thrash | dmsimard: ack. I was workng on that. lol | 19:59 |
| *** rbrady is now known as rbrady-afk | 20:00 | |
| *** oidgar has joined #tripleo | 20:07 | |
| *** ooolpbot has joined #tripleo | 20:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731063 | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1734134 | 20:10 |
| *** ooolpbot has quit IRC | 20:10 | |
| openstack | Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] - Assigned to Daniel Alvarez (dalvarezs) | 20:10 |
| openstack | Launchpad bug 1734134 in tripleo "Pike periodic promotion job multinode-1ctlr-featureset016 fail with error running docker 'gnocchi_db_sync' - rados.Rados.connect PermissionDeniedError: error connecting to the cluster" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 20:10 |
| dmsimard | thrash, mwhahaha: sweet! http://logs.openstack.org/16/523516/1/check/build-openstack-sphinx-docs/81c9420/html/ | 20:13 |
| thrash | dmsimard: eazy peezy | 20:13 |
| mwhahaha | nice | 20:14 |
| dmsimard | I mean, at first glance, some of the help/docstrings could use some improvement | 20:14 |
| dmsimard | but certainly beats updating that by hand | 20:14 |
| thrash | dmsimard: definitely | 20:14 |
| dmsimard | so I'll actually abandon my patch | 20:14 |
| dmsimard | because that patch needs to go in project-config instead | 20:14 |
| dmsimard | and you can do depends-on it | 20:14 |
| openstackgerrit | Brad P. Crochet proposed openstack/python-tripleoclient master: [WIP] Use cliff autodoc generation https://review.openstack.org/523510 | 20:14 |
| mwhahaha | k | 20:15 |
| thrash | dmsimard: ack | 20:15 |
| dmsimard | thrash: https://review.openstack.org/#/c/523523/ | 20:16 |
| *** etingof has quit IRC | 20:18 | |
| *** Goneri has joined #tripleo | 20:21 | |
| *** raildo has quit IRC | 20:21 | |
| *** pcaruana has joined #tripleo | 20:21 | |
| EmilienM | pradk: any news on fixing gnocchi? | 20:21 |
| EmilienM | https://bugs.launchpad.net/tripleo/+bug/1734134 | 20:21 |
| openstack | Launchpad bug 1734134 in tripleo "Pike periodic promotion job multinode-1ctlr-featureset016 fail with error running docker 'gnocchi_db_sync' - rados.Rados.connect PermissionDeniedError: error connecting to the cluster" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 20:21 |
| pradk | EmilienM, whats wrong | 20:21 |
| pradk | EmilienM, thought gfidente was looking at it? this is a regressions from ceph change. I asked if we could just revert it, but dint get a reply | 20:22 |
| EmilienM | pradk: we need to find a solution now | 20:22 |
| EmilienM | pradk: what patch is it a regression again? | 20:22 |
| pradk | https://review.openstack.org/#/c/508975/20/docker/services/gnocchi-api.yaml | 20:23 |
| pradk | EmilienM, ^^ this one | 20:23 |
| EmilienM | is the fix this patch? | 20:23 |
| EmilienM | https://review.openstack.org/#/c/522628/ | 20:23 |
| EmilienM | this patch sounds like fragile | 20:24 |
| EmilienM | we mix puppet and ansible for the same things | 20:24 |
| EmilienM | mwhahaha: ^ see my thoughts on that | 20:24 |
| pradk | hmm gnocchi doesnt have a dedicated uid.. so this wont work i think | 20:24 |
| EmilienM | john and gfidente are offline now | 20:24 |
| *** dpawar_ has quit IRC | 20:24 | |
| openstackgerrit | Brad P. Crochet proposed openstack/python-tripleoclient master: [WIP] Use cliff autodoc generation https://review.openstack.org/523510 | 20:24 |
| * mwhahaha doesn't like mixing puppet and ansible either | 20:24 | |
| mwhahaha | cause this is the stuff we get | 20:24 |
| EmilienM | if we revert https://review.openstack.org/#/c/508975 what happens? | 20:25 |
| *** atoth has quit IRC | 20:25 | |
| pradk | EmilienM, gfidente has this https://review.openstack.org/#/c/522630/ .. but its still failing with same error i think | 20:25 |
| EmilienM | ok no revert I guess | 20:26 |
| pradk | http://logs.openstack.org/30/522630/1/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/cbae486/logs/subnode-2/var/log/containers/gnocchi/gnocchi-upgrade.log.txt.gz#_2017-11-23_20_01_48_401 | 20:26 |
| pradk | yea same :/ | 20:26 |
| mwhahaha | the problem is that it's also broken stable/pike now | 20:26 |
| mwhahaha | https://review.openstack.org/#/c/522024/ | 20:26 |
| openstackgerrit | Merged openstack/tripleo-common master: Add exclude list to not override user data https://review.openstack.org/522657 | 20:27 |
| pradk | yea .. can we just revert this and see if it fixes the job? | 20:27 |
| EmilienM | so what's the plan? | 20:27 |
| EmilienM | pradk: it's a security patch, we can't revert that imho | 20:27 |
| mwhahaha | can we just revert teh gnocchi aspect of it? | 20:28 |
| pradk | hmm k | 20:28 |
| openstackgerrit | mathieu bultel proposed openstack/tripleo-common stable/pike: Add exclude list to not override user data https://review.openstack.org/523398 | 20:28 |
| EmilienM | mwhahaha: so we let gnocchi with this security issue? | 20:29 |
| pradk | mwhahaha, isnt that still a security issue? | 20:29 |
| mwhahaha | yea it's still a problem but to get us to a point where we can really fix it | 20:29 |
| pradk | do we have a reproducer somewhere? may be we can try few perms and see | 20:29 |
| mwhahaha | and is gnocchi commonly deployed? | 20:29 |
| dmsimard | what's the problem here, a mismatch of uid/gid ? | 20:30 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-heat-templates master: DMN: testing scenarios https://review.openstack.org/519756 | 20:30 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-heat-templates master: DMN: testing scenarios https://review.openstack.org/519756 | 20:30 |
| mwhahaha | dmsimard: yea they locked down the ceph keys and gnocchi can't auth to ceph to do the db upgrade | 20:31 |
| mwhahaha | cause the perms are bad or something | 20:31 |
| mwhahaha | it's not clear from the error logs what is actually wrong | 20:31 |
| dmsimard | I know that Kolla pre-creates users/groups with specific IDs in advance | 20:31 |
| dmsimard | but I don't know how you would upgrade from "not that" to "that" | 20:32 |
| dmsimard | https://github.com/openstack/kolla/blob/8e11ab68d9f306592c7638e148acddfdb91807d4/kolla/common/config.py#L667 | 20:32 |
| mwhahaha | http://logs.openstack.org/24/522024/1/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/5fad4b6/logs/subnode-2/var/log/containers/gnocchi/gnocchi-upgrade.log.txt.gz#_2017-11-21_23_34_51_123 | 20:32 |
| *** jpich has quit IRC | 20:33 | |
| mwhahaha | pradk: shouldn't this be a blocker if it was backported to stable/pike? are we not hitting this in testing elsewhere? | 20:33 |
| pradk | mwhahaha, i think it did not get picked up downstream as we only pull in blocker fixes | 20:34 |
| openstackgerrit | Merged openstack/python-tripleoclient master: Fix for timeouts on scale down https://review.openstack.org/522863 | 20:34 |
| openstackgerrit | Merged openstack/paunch master: Update and replace http with https for doc links https://review.openstack.org/485589 | 20:34 |
| mwhahaha | pradk: pretty sure this got pulled in somewhere | 20:34 |
| * mwhahaha checks | 20:34 | |
| mwhahaha | maybe not yet | 20:35 |
| mwhahaha | maybe for the next import | 20:35 |
| * EmilienM miam miam quick break | 20:35 | |
| *** crushil_ has quit IRC | 20:38 | |
| openstackgerrit | Alex Schultz proposed openstack/python-tripleoclient stable/pike: Fix for timeouts on scale down https://review.openstack.org/523532 | 20:39 |
| mwhahaha | rbrady-afk: -^ | 20:39 |
| *** agurenko has quit IRC | 20:39 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/newton: Set host name explicitly for telemetry https://review.openstack.org/521948 | 20:39 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-heat-templates master: Fix neutron agent start order https://review.openstack.org/523508 | 20:44 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-heat-templates master: DMN: testing scenarios https://review.openstack.org/519756 | 20:44 |
| pradk | mwhahaha, i gave reldel guys heads up to skip this change if they import. | 20:44 |
| mwhahaha | k | 20:44 |
| *** aputtur_ has quit IRC | 20:44 | |
| pradk | do we enable selinux in our jobs? | 20:44 |
| mwhahaha | no | 20:45 |
| pradk | ok | 20:45 |
| pradk | i might be readin this wrong.. but from traceback.. dont you think rados is throwing the error and not gnocchi | 20:46 |
| pradk | File "/usr/lib/python2.7/site-packages/gnocchi/storage/common/ceph.py", line 68, in create_rados_connection | 20:46 |
| pradk | conn.connect() | 20:46 |
| pradk | File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) | 20:46 |
| pradk | PermissionDeniedError: error connecting to the cluster | 20:46 |
| pradk | gnocchi is just creating the connection | 20:46 |
| mwhahaha | right that error leaves something to be desired | 20:46 |
| pradk | the perdenied is not from gnocchi\ | 20:46 |
| pradk | permission | 20:46 |
| mwhahaha | since it's coming from rados.pyx | 20:46 |
| mwhahaha | so i don't know if that's bad creds or it's defaulting to something bad because it cant' access /etc/ceph/ceph.conf | 20:47 |
| * EmilienM back | 20:57 | |
| EmilienM | I couldn't reproduce it in my rdocloud, it failed before | 20:57 |
| *** dprince has quit IRC | 20:57 | |
| EmilienM | due to som ehostname/fqdn thing gfidente told me | 20:57 |
| EmilienM | mwhahaha: have you tried to reproduce scenario001-container? | 20:58 |
| mwhahaha | no | 20:58 |
| *** oidgar has quit IRC | 21:01 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: Introduce fs035, ovb-ha-ipv6 https://review.openstack.org/522615 | 21:03 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: Create ovb-ha-ipv6 and deprecate fs024 job https://review.openstack.org/522618 | 21:03 |
| *** moshele has joined #tripleo | 21:04 | |
| *** thrash is now known as thrash|biab | 21:06 | |
| mwhahaha | EmilienM: i'm trying to run scenario001 at the moment, so we'll see what happens | 21:07 |
| EmilienM | mwhahaha: in rdocloud? | 21:07 |
| EmilienM | mwhahaha: it failed for me, step2 of the mistral ceph-ansible workflow | 21:08 |
| EmilienM | 2 times in a row | 21:08 |
| mwhahaha | no a different way | 21:08 |
| * mwhahaha opts for the follow the documentation | 21:08 | |
| mwhahaha | we'll see if the environment files we use really are consumable by end users :D | 21:09 |
| *** morazi has quit IRC | 21:09 | |
| EmilienM | :-O | 21:09 |
| *** ooolpbot has joined #tripleo | 21:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731063 | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1734134 | 21:10 |
| *** ooolpbot has quit IRC | 21:10 | |
| openstack | Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] - Assigned to Daniel Alvarez (dalvarezs) | 21:10 |
| openstack | Launchpad bug 1734134 in tripleo "Pike periodic promotion job multinode-1ctlr-featureset016 fail with error running docker 'gnocchi_db_sync' - rados.Rados.connect PermissionDeniedError: error connecting to the cluster" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 21:10 |
| *** gbarros has quit IRC | 21:12 | |
| openstackgerrit | Ade Lee proposed openstack/puppet-tripleo master: Add multiple backends for barbican https://review.openstack.org/523538 | 21:16 |
| pradk | mwhahaha, so the only key rings i see are http://logs.openstack.org/75/508975/20/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/2250893/logs/subnode-2/etc/ceph/ | 21:18 |
| openstackgerrit | Ade Lee proposed openstack/tripleo-heat-templates master: Add multiple secret store backends for barbican https://review.openstack.org/523539 | 21:19 |
| pradk | i guess openstack keyring is used for gnocchi.. but not sure if code translates to that | 21:19 |
| pradk | template: /etc/ceph/ceph.client.USER.keyring | 21:19 |
| pradk | assume USER here would be openstack | 21:19 |
| pradk | or should there be a separate key for each service? | 21:20 |
| pradk | if its common key, setting owner to gnocchi:gnocchi would break other services? | 21:20 |
| *** abregman has quit IRC | 21:20 | |
| *** gbarros has joined #tripleo | 21:21 | |
| *** pcaruana has quit IRC | 21:21 | |
| mwhahaha | yea by default it would be using the openstack one | 21:22 |
| openstackgerrit | Toure Dunnon proposed openstack/tripleo-common master: Migrate the ansible actions from tripleo-common to mistral-extra. https://review.openstack.org/523540 | 21:24 |
| pradk | hm so if we bind mont /etc/ceph into gnocchi container and set owner to gnocchi:gnocchi .. i guess rados wont be able to read it ? | 21:25 |
| *** dsneddon has joined #tripleo | 21:25 | |
| pradk | donno how other services are using ceph | 21:26 |
| pradk | may be they dont use rados gateway like gnocchi does? | 21:26 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Remove httpd log dir for glance-api https://review.openstack.org/516231 | 21:28 |
| EmilienM | pradk: be careful with wording, rados gatewal isn't librados | 21:30 |
| EmilienM | gateway* | 21:30 |
| EmilienM | in scenario001, we don't have rgw | 21:30 |
| EmilienM | librados requires keyring + ceph.conf | 21:31 |
| pradk | right | 21:31 |
| EmilienM | rados gateway is an object storage api, comparable to swift | 21:31 |
| pradk | i meant librados | 21:31 |
| EmilienM | ok | 21:31 |
| EmilienM | the permission issue can be couple of things : | 21:31 |
| EmilienM | wrong permission on keyring file | 21:31 |
| EmilienM | or wrong permission from the user in the pool | 21:32 |
| *** rcernin has quit IRC | 21:33 | |
| *** liverpooler has quit IRC | 21:37 | |
| *** liverpooler has joined #tripleo | 21:38 | |
| *** ansmith has quit IRC | 21:39 | |
| *** aputtur_ has joined #tripleo | 21:39 | |
| openstackgerrit | Alex Schultz proposed openstack/python-tripleoclient stable/pike: Fix for timeouts on scale down https://review.openstack.org/523532 | 21:40 |
| *** pchavva has quit IRC | 21:42 | |
| *** toure is now known as toure_biab | 21:42 | |
| *** openstackstatus has quit IRC | 21:42 | |
| *** lblanchard has quit IRC | 21:42 | |
| *** openstack has joined #tripleo | 21:43 | |
| *** ChanServ sets mode: +o openstack | 21:43 | |
| *** openstackstatus has joined #tripleo | 21:45 | |
| *** ChanServ sets mode: +v openstackstatus | 21:45 | |
| *** moshele has quit IRC | 21:48 | |
| *** hamzy has quit IRC | 21:50 | |
| *** fzdarsky is now known as fzdarsky|afk | 21:51 | |
| *** morazi has joined #tripleo | 21:52 | |
| *** trown is now known as trown|outtypewww | 22:01 | |
| *** rbowen has quit IRC | 22:02 | |
| *** ooolpbot has joined #tripleo | 22:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731063 | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1734134 | 22:10 |
| *** ooolpbot has quit IRC | 22:10 | |
| openstack | Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] - Assigned to Daniel Alvarez (dalvarezs) | 22:10 |
| openstack | Launchpad bug 1734134 in tripleo "Pike periodic promotion job multinode-1ctlr-featureset016 fail with error running docker 'gnocchi_db_sync' - rados.Rados.connect PermissionDeniedError: error connecting to the cluster" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 22:10 |
| openstackgerrit | Tony Breeds proposed openstack/tripleo-heat-templates master: Add ComputeAlt role and environment https://review.openstack.org/523547 | 22:12 |
| *** jlabarre has quit IRC | 22:14 | |
| *** abishop has quit IRC | 22:17 | |
| *** gbarros has quit IRC | 22:19 | |
| *** rcernin has joined #tripleo | 22:23 | |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci master: Fix sub_nodes and sub_nodes_private with multiple nodes https://review.openstack.org/523226 | 22:25 |
| *** rhallisey has quit IRC | 22:26 | |
| *** moshele has joined #tripleo | 22:28 | |
| *** zshi has quit IRC | 22:29 | |
| *** jtomasek_ has quit IRC | 22:29 | |
| openstackgerrit | Matt Young proposed openstack/tripleo-quickstart master: collect logs: remote --ansible-debug https://review.openstack.org/523552 | 22:41 |
| openstackgerrit | Matt Young proposed openstack/tripleo-quickstart master: collect logs: remove --ansible-debug https://review.openstack.org/523552 | 22:41 |
| *** ansmith has joined #tripleo | 22:47 | |
| *** pmannidi has joined #tripleo | 22:51 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Swap the order of stdout and stderr in debug output https://review.openstack.org/522405 | 22:53 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Download Kubespray instead of git clone https://review.openstack.org/523397 | 22:57 |
| *** vpickard is now known as vpickard_ | 22:57 | |
| *** bfournie has quit IRC | 22:57 | |
| openstackgerrit | Moshe Levi proposed openstack/tripleo-heat-templates master: Adds environment file for ODL OVS Hardware Offload https://review.openstack.org/518715 | 22:58 |
| *** aputtur_ has quit IRC | 22:58 | |
| mwhahaha | EmilienM: so you'll be happy to know that the starting order of the neutron services doesn't fix our scenarios | 23:00 |
| openstackgerrit | Attila Darazs proposed openstack/tripleo-quickstart-extras master: GATE CHECK for quickstart-extras https://review.openstack.org/472607 | 23:00 |
| * mwhahaha cries | 23:00 | |
| mwhahaha | http://logs.openstack.org/56/519756/11/check/tripleo-ci-centos-7-scenario003-multinode-oooq-container/92c0255/job-output.txt.gz#_2017-11-28_22_50_15_030617 | 23:00 |
| *** jmelvin|bomgar has quit IRC | 23:03 | |
| mwhahaha | tonyb: which sshd? the OS::Tripleo::Services::Sshd? | 23:04 |
| *** aputtur_ has joined #tripleo | 23:04 | |
| EmilienM | mwhahaha: :( | 23:07 |
| mwhahaha | but it does fix teh initial rpc error we see | 23:07 |
| EmilienM | mwhahaha: but we need that anyway I think | 23:07 |
| mwhahaha | which is nice | 23:07 |
| mwhahaha | i bet we have another ovs starting thing somewhere | 23:07 |
| mwhahaha | cause it's still throwing a not connected exception | 23:07 |
| openstackgerrit | Tony Breeds proposed openstack/tripleo-heat-templates master: Add ComputeAlt role and environment https://review.openstack.org/523547 | 23:07 |
| mwhahaha | http://logs.openstack.org/56/519756/11/check/tripleo-ci-centos-7-scenario003-multinode-oooq-container/92c0255/logs/subnode-2/var/log/containers/neutron/neutron-l3-agent.log.txt.gz#_2017-11-28_22_43_17_075 | 23:08 |
| EmilienM | mwhahaha: we need to check timestamps | 23:08 |
| tonyb | mwhahaha: I may have gone a little nuts with that ^^ version | 23:08 |
| EmilienM | make sure neutron server actually starts before the agents | 23:08 |
| EmilienM | and ovs starts before all | 23:08 |
| EmilienM | I'm checking it now | 23:08 |
| mwhahaha | tonyb: ok i'll take a look. i think you can leave sshd | 23:08 |
| tonyb | mwhahaha: okay I'll back that out | 23:10 |
| *** ooolpbot has joined #tripleo | 23:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731063 | 23:10 |
| openstack | Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] - Assigned to Daniel Alvarez (dalvarezs) | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1734134 | 23:10 |
| *** ooolpbot has quit IRC | 23:10 | |
| openstack | Launchpad bug 1734134 in tripleo "Pike periodic promotion job multinode-1ctlr-featureset016 fail with error running docker 'gnocchi_db_sync' - rados.Rados.connect PermissionDeniedError: error connecting to the cluster" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 23:10 |
| openstackgerrit | Tony Breeds proposed openstack/tripleo-heat-templates master: Add ComputeAlt role and environment https://review.openstack.org/523547 | 23:10 |
| mwhahaha | tonyb: you missed it in the role file anyway :D just the environment file | 23:10 |
| EmilienM | hum so | 23:13 |
| tonyb | mwhahaha: ssssh (pun intended) you weren't s'posed to notice ;P | 23:13 |
| mwhahaha | :D | 23:13 |
| EmilienM | http://logs.openstack.org/56/519756/11/check/tripleo-ci-centos-7-scenario003-multinode-oooq-container/92c0255/logs/subnode-2/var/log/containers/neutron/neutron-server.log.txt.gz#_2017-11-28_22_38_15_952 | 23:13 |
| EmilienM | neutron-server is ready at 22:38:15 | 23:13 |
| EmilienM | the agents start right after (a few seconds after) | 23:14 |
| mwhahaha | which is how it used to do it | 23:14 |
| EmilienM | ovs started way before it | 23:14 |
| mwhahaha | i looked at an ocata | 23:14 |
| EmilienM | so your patch works | 23:14 |
| mwhahaha | patch | 23:14 |
| mwhahaha | yea | 23:14 |
| moshele | @mwhahaha: hi regarding your comment on https://review.openstack.org/#/c/520041/ I created a different fix in https://review.openstack.org/#/c/521232/. is that better ? | 23:14 |
| *** almondjoy has quit IRC | 23:15 | |
| mwhahaha | moshele: yea that one is less dangerous | 23:15 |
| * mwhahaha prepares to melt zuul | 23:16 | |
| moshele | @mwhahaha: ok cool | 23:16 |
| * mwhahaha pokes EmilienM with https://review.openstack.org/#/c/521232/ for a second opinion | 23:17 | |
| EmilienM | mwhahaha: we need metadata proxy logs to continue the debug | 23:17 |
| EmilienM | mwhahaha: afik they are in /var/lib/neutron | 23:17 |
| EmilienM | you know what I mean? | 23:17 |
| EmilienM | sometimes they show useful stuff when ssh doesn't work | 23:18 |
| mwhahaha | EmilienM: do we not capture them by default? | 23:18 |
| EmilienM | let me check | 23:18 |
| mwhahaha | if it's in a container we might | 23:18 |
| EmilienM | http://logs.openstack.org/56/519756/11/check/tripleo-ci-centos-7-scenario003-multinode-oooq-container/92c0255/logs/subnode-2/var/lib/ | 23:18 |
| EmilienM | ahh | 23:18 |
| EmilienM | maybe then ok | 23:18 |
| mwhahaha | no | 23:19 |
| EmilienM | let me check | 23:19 |
| mwhahaha | we only capture unbound | 23:19 |
| mwhahaha | that was a change recently | 23:19 |
| mwhahaha | so we'd need to add that to the list | 23:19 |
| mwhahaha | but good question if they are exposed at all under teh containers | 23:19 |
| EmilienM | I don't find thel | 23:19 |
| moshele | beagles: hi have you had a chance to test https://review.openstack.org/#/c/507100/ in your SR-IOV env? | 23:20 |
| *** etingof has joined #tripleo | 23:21 | |
| EmilienM | mwhahaha: the thing is they have a volatile folder name | 23:22 |
| EmilienM | with uuid iiurc | 23:22 |
| EmilienM | iirc* | 23:22 |
| mwhahaha | but is it /var/lib/neutron/? | 23:22 |
| EmilienM | yes | 23:23 |
| mwhahaha | can we depends-on a way to get them? | 23:23 |
| *** aputtur_ has quit IRC | 23:23 | |
| EmilienM | mwhahaha: I'm poking at it now | 23:24 |
| openstackgerrit | James Slagle proposed openstack/tripleo-docs master: Document deployment with Ansible and config-download https://review.openstack.org/523603 | 23:27 |
| slagle | EmilienM: there's a start to some docs ^ | 23:28 |
| EmilienM | mwhahaha: in fact I'm not sure about the logdir - I'm investigating | 23:30 |
| EmilienM | slagle: awesome, I'll take a look asap | 23:30 |
| mwhahaha | k | 23:30 |
| EmilienM | slagle: wow that looks cool in a first look | 23:31 |
| EmilienM | even a picture | 23:31 |
| *** tosky has quit IRC | 23:31 | |
| *** tosky has joined #tripleo | 23:34 | |
| EmilienM | mwhahaha: have you seen http://logs.openstack.org/56/519756/11/check/tripleo-ci-centos-7-scenario003-multinode-oooq-container/92c0255/logs/subnode-2/var/log/extra/docker/containers/neutron_l3_agent/log/neutron/neutron-l3-agent.log.txt.gz#_2017-11-28_22_43_17_913? | 23:36 |
| EmilienM | I guess yes | 23:36 |
| mwhahaha | yea that's what i was refering to | 23:36 |
| mwhahaha | it goes along with http://logs.openstack.org/56/519756/11/check/tripleo-ci-centos-7-scenario003-multinode-oooq-container/92c0255/logs/subnode-2/var/log/extra/docker/containers/neutron_l3_agent/log/neutron/neutron-l3-agent.log.txt.gz#_2017-11-28_22_43_17_075 | 23:37 |
| mwhahaha | which is what i originally theorized was the problem but i have no idea wtf those services are | 23:37 |
| * mwhahaha gets all hand-wavey | 23:37 | |
| EmilienM | I'm calling Ihar | 23:37 |
| EmilienM | I'm trying to see if we're alone to have this problem | 23:38 |
| *** tosky has quit IRC | 23:38 | |
| EmilienM | I have a logstash query handy: | 23:38 |
| mwhahaha | he originally looked at it and said we needed to wait for Terry | 23:38 |
| EmilienM | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%20%5C%22NotConnectedError%3A%20Cannot%20commit%20transaction%20AddPortCommand%5C%22 | 23:38 |
| EmilienM | who is Terry? | 23:39 |
| mwhahaha | i don't know but he was on pto until this week | 23:39 |
| EmilienM | I'm not waiting for anything | 23:39 |
| EmilienM | our gate is unstable | 23:39 |
| EmilienM | we don't have time to wait | 23:39 |
| EmilienM | so logstash tells me it started on Nov 23th | 23:39 |
| mwhahaha | well it existed before that | 23:40 |
| EmilienM | probably | 23:40 |
| *** ihrachys has joined #tripleo | 23:40 | |
| ihrachys | heya | 23:40 |
| EmilienM | we need to look at what we bumped in rdoinfo regarding neutron deps | 23:40 |
| EmilienM | ihrachys: hi. Debugging http://logs.openstack.org/56/519756/11/check/tripleo-ci-centos-7-scenario003-multinode-oooq-container/92c0255/logs/subnode-2/var/log/extra/docker/containers/neutron_l3_agent/log/neutron/neutron-l3-agent.log.txt.gz#_2017-11-28_22_43_17_913 | 23:40 |
| EmilienM | it's not making any progress :( | 23:40 |
| ihrachys | is it smth new? | 23:40 |
| EmilienM | no | 23:41 |
| ihrachys | there was ovsdbapp release | 23:41 |
| ihrachys | have you tried to revert that one? | 23:41 |
| EmilienM | not afik | 23:41 |
| EmilienM | but i'm going to | 23:41 |
| ihrachys | ok that would be my first guess | 23:41 |
| ihrachys | otherwiseguy aka Terry should be pulled into it | 23:41 |
| ihrachys | is he on the issue? | 23:41 |
| ihrachys | if not, we may get him | 23:41 |
| EmilienM | ihrachys: he's not AFIK | 23:42 |
| EmilienM | ihrachys: do we use openstack/neutron-dynamic-routing in tripleo? | 23:42 |
| EmilienM | I'm not familiar anymore with neutron bits :( | 23:42 |
| ihrachys | where is the issue tracked? | 23:42 |
| EmilienM | https://bugs.launchpad.net/tripleo/+bug/1731063 | 23:42 |
| openstack | Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] - Assigned to Daniel Alvarez (dalvarezs) | 23:42 |
| ihrachys | EmilienM, I don't think we even have a package for that? | 23:42 |
| EmilienM | ok | 23:42 |
| ihrachys | oh seems we have now https://github.com/rdo-packages/neutron-dynamic-routing-distgit/commits/rpm-master | 23:43 |
| ihrachys | but no one from redhat neutron team looked at it afaiu | 23:43 |
| ihrachys | actually, the repo is empty, see: https://github.com/rdo-packages/neutron-dynamic-routing-distgit | 23:43 |
| EmilienM | https://review.rdoproject.org/r/#/c/10555/ | 23:44 |
| EmilienM | this is the bump | 23:44 |
| ihrachys | EmilienM, how is https://review.openstack.org/#/c/523508 related? | 23:44 |
| mwhahaha | we were starting all the agents at teh same time as the service itself | 23:44 |
| mwhahaha | which was a regression from previous releases | 23:44 |
| EmilienM | ihrachys: we're exploring options here | 23:44 |
| EmilienM | and alex fixed a huge regression | 23:45 |
| mwhahaha | and it introduced some spurious rpc errors | 23:45 |
| mwhahaha | so that clears those up | 23:45 |
| EmilienM | so ovsdbapp 0.7.0 is good? | 23:45 |
| mwhahaha | the problem is we don't know, 0.4.0 is good | 23:45 |
| *** bfournie has joined #tripleo | 23:45 | |
| mwhahaha | cause that's what pike has | 23:45 |
| mwhahaha | but sometime between pike and now, we're getting these problems | 23:45 |
| EmilienM | https://github.com/openstack/ovsdbapp/releases | 23:45 |
| EmilienM | they did a bunch of tags lately | 23:45 |
| mwhahaha | and since we aren't getting regular promotions it's hard to tell | 23:46 |
| ihrachys | EmilienM, well the hell I know :) Terry would be the guy to answer. and dig actually,. | 23:46 |
| mwhahaha | is he on east coast time? | 23:46 |
| mwhahaha | or europe? | 23:46 |
| EmilienM | ihrachys: well, is he here? | 23:46 |
| EmilienM | because I never saw him looking at that bug unless I missed something | 23:46 |
| EmilienM | and we need help now, like serious help | 23:46 |
| *** bfournie has quit IRC | 23:47 | |
| ihrachys | EmilienM, define here. probably not in the channel? | 23:47 |
| EmilienM | ihrachys: here, on this bug, on this channel, here | 23:47 |
| EmilienM | like we're deploying your software and it's broken | 23:47 |
| ihrachys | I am not going to solve it right now. but let's drop an email to Terry and maybe Brent and Daniel and the neutron team about the issue and hopefully have some more attention till tomorrow. | 23:47 |
| EmilienM | so by here I mean, HERE IN THE REAL WORLD | 23:47 |
| ihrachys | Daniel is assigned to the bug, so I would ask him where we are | 23:47 |
| EmilienM | not in devstack | 23:47 |
| EmilienM | we haven't promoted RDO CI in 20 days | 23:47 |
| EmilienM | we disabled voting on some scenarios | 23:48 |
| EmilienM | it makes our gate unstable | 23:48 |
| EmilienM | so yeah a bit of commitment is warmly welcome | 23:48 |
| *** bfournie has joined #tripleo | 23:48 | |
| ihrachys | ok I get the seriousness of the issue. in theory, Brian and Miguel should track those issues and make them progress. if that didn't happen, I am sorry and I will try to pull the strings right now. | 23:48 |
| EmilienM | thank you very much | 23:48 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: ci: add ovb-ha.yaml https://review.openstack.org/522306 | 23:49 |
| EmilienM | I won't propose any revert in rdoinfo now, as we don't know exactly which tag we should have | 23:49 |
| EmilienM | mwhahaha: my logstash query sucks, I can't find tripleo job errors | 23:50 |
| EmilienM | I'm trying to see since when *exactly* we hit this problem | 23:50 |
| mwhahaha | on our last promotion we only had like 1 | 23:50 |
| mwhahaha | so basically 60 days of no promotion, promotion, 20 days of no promotion | 23:51 |
| mwhahaha | it won't do you any good to go looking at that | 23:51 |
| mwhahaha | so hopefully we can get the next promotion by queens-m3 | 23:52 |
| mwhahaha | fortunately i have a cat to hug | 23:52 |
| EmilienM | well even in devstack I'm looking at it | 23:52 |
| mwhahaha | he keeps me company | 23:52 |
| EmilienM | I have nothing to hug | 23:52 |
| EmilienM | hance my anger | 23:52 |
| mwhahaha | i'll buy you one of these http://www.mydailysales.com/wp-content/uploads/2013/10/201310181022407185493.jpg | 23:52 |
| mwhahaha | they look awesome | 23:52 |
| EmilienM | it sounds like openstack/neutron-dynamic-routing gate hits this error as well | 23:53 |
| EmilienM | I'm trying to see theerror | 23:53 |
| EmilienM | mwhahaha: lol | 23:53 |
| EmilienM | http://logs.openstack.org/17/522717/2/check/legacy-neutron-dynamic-routing-dsvm-tempest-scenario-ipv4/88ebbb0/logs/screen-q-dhcp.txt?level=ERROR#_Nov_24_09_44_35_501572 | 23:53 |
| EmilienM | i'm wondering what version of ovsdbapp devstack gate use | 23:54 |
| mwhahaha | i wish i knew more about the interface wiring up of the vms for the metadata fetches | 23:54 |
| mwhahaha | cause it seems like it's that but i don't know when/how things get magically connected | 23:55 |
| EmilienM | ovsdbapp==0.8.0 | 23:55 |
| EmilienM | so the failure is on ovsdbapp==0.8.0 | 23:55 |
| EmilienM | now let's see neutron's gate | 23:55 |
| openstackgerrit | Merged openstack/tripleo-quickstart master: Run ovb-ha with minimal services https://review.openstack.org/522310 | 23:56 |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/ocata: Enable os-collect-config service https://review.openstack.org/523245 | 23:56 |
| ihrachys | EmilienM, mwhahaha I see daniel was actively working on trello card no? | 23:57 |
| mwhahaha | as of this morning | 23:57 |
| ihrachys | ack | 23:57 |
| EmilienM | http://logs.openstack.org/33/523133/2/gate/legacy-tempest-dsvm-neutron-full/974fbf7/logs/ | 23:57 |
| *** thrash|biab is now known as thrash | 23:57 | |
| EmilienM | can someone tell me which version of ovsdbapp it's running | 23:57 |
| EmilienM | it's not in pip-freeze file or in dpkg | 23:57 |
| *** thrash is now known as thrash|g0ne | 23:58 | |
| *** pradk_ has joined #tripleo | 23:59 | |
| ihrachys | EmilienM, 2017-11-28 19:10:22.437 9739 INFO neutron.common.config [-] /usr/local/bin/neutron-openvswitch-agent version 10.0.5.dev23 | 23:59 |
| ihrachys | isn't it some old release? | 23:59 |
| EmilienM | ihrachys: also, how come neutron's gate doesn't have any centos7 job in the gate? a bit sad no? | 23:59 |
| ihrachys | it wasn't using ovsdbapp before pike | 23:59 |
| EmilienM | ok I grabbed a wrong link | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
