| *** dhill_ has quit IRC | 00:04 | |
| *** ooolpbot has joined #tripleo | 00:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 00:10 |
|---|---|---|
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731063 | 00:10 |
| openstack | Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] | 00:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1733345 | 00:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1734134 | 00:10 |
| *** ooolpbot has quit IRC | 00:10 | |
| openstack | Launchpad bug 1733345 in tripleo "Master promotion: error creating the default Deployment Plan overcloud" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 00:10 |
| openstack | Launchpad bug 1734134 in tripleo "Pike periodic promotion job multinode-1ctlr-featureset016 fail with error running docker 'gnocchi_db_sync' - rados.Rados.connect PermissionDeniedError: error connecting to the cluster" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 00:10 |
| *** dmacpher has quit IRC | 00:11 | |
| *** jkilpatr has quit IRC | 00:22 | |
| *** bobh has joined #tripleo | 00:23 | |
| *** bobh has quit IRC | 00:28 | |
| *** fpan has quit IRC | 00:34 | |
| *** bobh has joined #tripleo | 00:34 | |
| *** fpan has joined #tripleo | 00:35 | |
| *** bobh has quit IRC | 00:38 | |
| *** tosky has quit IRC | 00:47 | |
| EmilienM | amoralej|off: you need to remove these jobs from the promotion pipeline. We only want containers jobs now | 00:50 |
| *** dmacpher has joined #tripleo | 01:09 | |
| *** etingof has quit IRC | 01:09 | |
| *** ooolpbot has joined #tripleo | 01:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731063 | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1733345 | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1734134 | 01:10 |
| *** ooolpbot has quit IRC | 01:10 | |
| openstack | Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] | 01:10 |
| openstack | Launchpad bug 1733345 in tripleo "Master promotion: error creating the default Deployment Plan overcloud" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 01:10 |
| openstack | Launchpad bug 1734134 in tripleo "Pike periodic promotion job multinode-1ctlr-featureset016 fail with error running docker 'gnocchi_db_sync' - rados.Rados.connect PermissionDeniedError: error connecting to the cluster" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 01:10 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates stable/newton: RHEL/Update: replace wc by yum to check updates https://review.openstack.org/522396 | 01:18 |
| *** bobh has joined #tripleo | 01:22 | |
| *** etingof has joined #tripleo | 01:23 | |
| *** dougbtv_ has quit IRC | 01:29 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: ci: add ovb-ha.yaml https://review.openstack.org/522306 | 01:30 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: Run ovb-ha with minimal services https://review.openstack.org/522310 | 01:32 |
| EmilienM | amoralej|off: non-container scenarios were dropped from gate a few weeks ago, they shouldn't run in the periodic pipeline, sorry it was miscommunicated, although we discussed about it on the ML. Let me know if I can help | 01:42 |
| *** gbarros has joined #tripleo | 01:46 | |
| *** bobh has left #tripleo | 01:48 | |
| *** japestinho has joined #tripleo | 01:55 | |
| *** pchavva has quit IRC | 02:03 | |
| *** ramishra has joined #tripleo | 02:09 | |
| *** ooolpbot has joined #tripleo | 02:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731063 | 02:10 |
| openstack | Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1733345 | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1734134 | 02:10 |
| *** ooolpbot has quit IRC | 02:10 | |
| openstack | Launchpad bug 1733345 in tripleo "Master promotion: error creating the default Deployment Plan overcloud" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 02:10 |
| openstack | Launchpad bug 1734134 in tripleo "Pike periodic promotion job multinode-1ctlr-featureset016 fail with error running docker 'gnocchi_db_sync' - rados.Rados.connect PermissionDeniedError: error connecting to the cluster" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 02:10 |
| *** gbarros has quit IRC | 02:22 | |
| *** dougbtv_ has joined #tripleo | 02:23 | |
| *** gbarros has joined #tripleo | 02:32 | |
| *** fragatin_ has quit IRC | 02:38 | |
| *** fragatina has joined #tripleo | 02:39 | |
| *** threestrands_ has joined #tripleo | 02:49 | |
| *** moshele has joined #tripleo | 02:49 | |
| *** gbarros has quit IRC | 02:49 | |
| *** threestrands_ has quit IRC | 02:50 | |
| *** threestrands has quit IRC | 02:50 | |
| *** threestrands_ has joined #tripleo | 02:50 | |
| *** atoth_ has quit IRC | 03:00 | |
| *** moshele has quit IRC | 03:02 | |
| *** ooolpbot has joined #tripleo | 03:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731063 | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1733345 | 03:10 |
| openstack | Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1734134 | 03:10 |
| *** ooolpbot has quit IRC | 03:10 | |
| openstack | Launchpad bug 1733345 in tripleo "Master promotion: error creating the default Deployment Plan overcloud" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 03:10 |
| openstack | Launchpad bug 1734134 in tripleo "Pike periodic promotion job multinode-1ctlr-featureset016 fail with error running docker 'gnocchi_db_sync' - rados.Rados.connect PermissionDeniedError: error connecting to the cluster" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 03:10 |
| *** newmember has quit IRC | 03:18 | |
| *** newmember has joined #tripleo | 03:19 | |
| *** psahoo has joined #tripleo | 03:40 | |
| *** zshi has quit IRC | 03:46 | |
| *** zshi has joined #tripleo | 03:47 | |
| *** mdnadeem has joined #tripleo | 03:49 | |
| *** psachin has joined #tripleo | 03:50 | |
| *** daidv has joined #tripleo | 04:00 | |
| *** ooolpbot has joined #tripleo | 04:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731063 | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1733345 | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1734134 | 04:10 |
| *** ooolpbot has quit IRC | 04:10 | |
| openstack | Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] | 04:10 |
| openstack | Launchpad bug 1733345 in tripleo "Master promotion: error creating the default Deployment Plan overcloud" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 04:10 |
| openstack | Launchpad bug 1734134 in tripleo "Pike periodic promotion job multinode-1ctlr-featureset016 fail with error running docker 'gnocchi_db_sync' - rados.Rados.connect PermissionDeniedError: error connecting to the cluster" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 04:10 |
| *** stendulker has joined #tripleo | 04:18 | |
| *** Dinesh_Bhor has joined #tripleo | 04:21 | |
| *** ykarel|afk has joined #tripleo | 04:24 | |
| *** links has joined #tripleo | 04:26 | |
| *** anilvenkata has joined #tripleo | 04:28 | |
| *** michchap has quit IRC | 04:34 | |
| *** udesale has joined #tripleo | 04:53 | |
| *** skramaja has joined #tripleo | 04:53 | |
| *** dmacpher has quit IRC | 04:59 | |
| *** iranzo has joined #tripleo | 05:07 | |
| *** ooolpbot has joined #tripleo | 05:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731063 | 05:10 |
| openstack | Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1733345 | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1734134 | 05:10 |
| *** ooolpbot has quit IRC | 05:10 | |
| openstack | Launchpad bug 1733345 in tripleo "Master promotion: error creating the default Deployment Plan overcloud" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 05:10 |
| openstack | Launchpad bug 1734134 in tripleo "Pike periodic promotion job multinode-1ctlr-featureset016 fail with error running docker 'gnocchi_db_sync' - rados.Rados.connect PermissionDeniedError: error connecting to the cluster" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 05:10 |
| *** pdeore has joined #tripleo | 05:11 | |
| *** fpan has quit IRC | 05:16 | |
| *** fpan has joined #tripleo | 05:16 | |
| *** jaosorior has joined #tripleo | 05:19 | |
| *** ratailor has joined #tripleo | 05:30 | |
| *** stendulker_ has joined #tripleo | 05:32 | |
| *** stendulker has quit IRC | 05:36 | |
| *** dpawar has joined #tripleo | 05:52 | |
| *** udesale__ has joined #tripleo | 05:52 | |
| *** udesale has quit IRC | 05:53 | |
| *** udesale has joined #tripleo | 05:55 | |
| *** udesale__ has quit IRC | 05:57 | |
| Tengu | hello there :) | 05:59 |
| jaosorior | good morning | 06:00 |
| Tengu | how are you jaosorior ? | 06:00 |
| jaosorior | Tengu: quite alright; working on some ansible bits. you? | 06:01 |
| Tengu | quite good as well. a but cold due to the home->office journey, but it's ok. | 06:02 |
| *** jaganathan has joined #tripleo | 06:02 | |
| *** ooolpbot has joined #tripleo | 06:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731063 | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1733345 | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1734134 | 06:10 |
| *** ooolpbot has quit IRC | 06:10 | |
| openstack | Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] | 06:10 |
| openstack | Launchpad bug 1733345 in tripleo "Master promotion: error creating the default Deployment Plan overcloud" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 06:10 |
| openstack | Launchpad bug 1734134 in tripleo "Pike periodic promotion job multinode-1ctlr-featureset016 fail with error running docker 'gnocchi_db_sync' - rados.Rados.connect PermissionDeniedError: error connecting to the cluster" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 06:10 |
| *** janki has joined #tripleo | 06:15 | |
| *** janki has quit IRC | 06:16 | |
| *** janki has joined #tripleo | 06:16 | |
| *** dsariel has joined #tripleo | 06:18 | |
| *** akane has joined #tripleo | 06:20 | |
| *** akane_ has joined #tripleo | 06:20 | |
| *** fragatina has quit IRC | 06:21 | |
| *** jbadiapa has joined #tripleo | 06:30 | |
| *** gkadam has joined #tripleo | 06:33 | |
| *** d0ugal has quit IRC | 06:35 | |
| *** masco has joined #tripleo | 06:36 | |
| *** spectr has joined #tripleo | 06:44 | |
| *** moshele has joined #tripleo | 06:45 | |
| *** abregman|afk has quit IRC | 06:45 | |
| *** rcernin has quit IRC | 06:47 | |
| *** d0ugal has joined #tripleo | 06:49 | |
| *** gkadam has quit IRC | 06:49 | |
| *** rcernin has joined #tripleo | 06:49 | |
| *** spectr has quit IRC | 06:55 | |
| *** moshele has quit IRC | 06:56 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: WIP: POC: logs collection from native zuulv3 job https://review.openstack.org/522554 | 07:04 |
| *** stendulker_ has quit IRC | 07:05 | |
| *** oidgar has joined #tripleo | 07:08 | |
| *** marios has joined #tripleo | 07:08 | |
| *** ooolpbot has joined #tripleo | 07:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731063 | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1733345 | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1734134 | 07:10 |
| *** ooolpbot has quit IRC | 07:10 | |
| openstack | Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] | 07:10 |
| openstack | Launchpad bug 1733345 in tripleo "Master promotion: error creating the default Deployment Plan overcloud" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 07:10 |
| openstack | Launchpad bug 1734134 in tripleo "Pike periodic promotion job multinode-1ctlr-featureset016 fail with error running docker 'gnocchi_db_sync' - rados.Rados.connect PermissionDeniedError: error connecting to the cluster" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 07:10 |
| *** jfrancoa has joined #tripleo | 07:12 | |
| *** etingof has quit IRC | 07:21 | |
| *** florianf has joined #tripleo | 07:21 | |
| *** moshele has joined #tripleo | 07:25 | |
| *** oidgar has quit IRC | 07:26 | |
| *** ykarel|afk is now known as ykarel | 07:27 | |
| *** aufi has joined #tripleo | 07:28 | |
| *** rcernin has quit IRC | 07:28 | |
| *** ccamacho has quit IRC | 07:32 | |
| *** ccamacho has joined #tripleo | 07:32 | |
| *** holser has joined #tripleo | 07:33 | |
| *** etingof has joined #tripleo | 07:34 | |
| openstackgerrit | Martin André proposed openstack/tripleo-heat-templates stable/pike: Add DeployIdentifier to Nova discover hosts container https://review.openstack.org/523045 | 07:36 |
| *** ykarel is now known as ykarel|lunch | 07:37 | |
| *** nyechiel has joined #tripleo | 07:46 | |
| *** jtomasek has joined #tripleo | 07:47 | |
| *** stendulker_ has joined #tripleo | 07:48 | |
| Tengu | can some people with +2 review/workflow have a look on that review? https://review.openstack.org/#/c/522720/ it's requested for another one, and thus locks it :). Thanks! | 07:48 |
| *** stendulker_ has quit IRC | 07:53 | |
| *** stendulker_ has joined #tripleo | 07:53 | |
| *** stendulker_ has quit IRC | 07:54 | |
| *** yprokule has joined #tripleo | 07:54 | |
| *** stendulker_ has joined #tripleo | 07:55 | |
| *** stendulker_ has quit IRC | 07:55 | |
| *** stendulker has joined #tripleo | 07:55 | |
| *** matbu has quit IRC | 08:04 | |
| *** matbu__ has joined #tripleo | 08:04 | |
| *** matbu__ is now known as matbu | 08:04 | |
| *** nyechiel has quit IRC | 08:05 | |
| *** abregman has joined #tripleo | 08:06 | |
| *** stendulker has quit IRC | 08:08 | |
| *** stendulker has joined #tripleo | 08:08 | |
| *** dparkes has joined #tripleo | 08:09 | |
| *** ooolpbot has joined #tripleo | 08:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731063 | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1733345 | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1734134 | 08:10 |
| *** ooolpbot has quit IRC | 08:10 | |
| openstack | Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] | 08:10 |
| openstack | Launchpad bug 1733345 in tripleo "Master promotion: error creating the default Deployment Plan overcloud" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 08:10 |
| openstack | Launchpad bug 1734134 in tripleo "Pike periodic promotion job multinode-1ctlr-featureset016 fail with error running docker 'gnocchi_db_sync' - rados.Rados.connect PermissionDeniedError: error connecting to the cluster" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 08:10 |
| *** ffiore has joined #tripleo | 08:10 | |
| *** flaper87 has quit IRC | 08:11 | |
| *** pcaruana has joined #tripleo | 08:11 | |
| *** flaper87 has joined #tripleo | 08:13 | |
| openstackgerrit | zenghui.shi proposed openstack/tripleo-heat-templates master: Add PTP composable service https://review.openstack.org/491317 | 08:17 |
| *** agurenko has joined #tripleo | 08:18 | |
| *** ebarrera has joined #tripleo | 08:19 | |
| *** abregman has quit IRC | 08:20 | |
| *** cylopez has joined #tripleo | 08:23 | |
| *** athomas has joined #tripleo | 08:24 | |
| *** etingof has quit IRC | 08:25 | |
| *** michchap has joined #tripleo | 08:26 | |
| *** oidgar has joined #tripleo | 08:27 | |
| *** amoralej|off is now known as amoralej | 08:29 | |
| *** gkadam has joined #tripleo | 08:31 | |
| *** ykarel|lunch is now known as ykarel | 08:31 | |
| *** abregman has joined #tripleo | 08:37 | |
| *** jpena|off is now known as jpena | 08:38 | |
| *** etingof has joined #tripleo | 08:39 | |
| *** ffiore has quit IRC | 08:40 | |
| openstackgerrit | zenghui.shi proposed openstack/puppet-tripleo master: Add PTP service https://review.openstack.org/491324 | 08:43 |
| *** fzdarsky has joined #tripleo | 08:43 | |
| *** paramite has joined #tripleo | 08:43 | |
| *** jaganathan has quit IRC | 08:48 | |
| *** ffiore has joined #tripleo | 08:50 | |
| *** jaganathan has joined #tripleo | 08:50 | |
| *** ukalifon has joined #tripleo | 08:51 | |
| *** gfidente has joined #tripleo | 08:52 | |
| *** gfidente has quit IRC | 08:52 | |
| *** gfidente has joined #tripleo | 08:52 | |
| *** bogdando has joined #tripleo | 08:54 | |
| *** jpich has joined #tripleo | 08:56 | |
| moshele | skramaja: hi can you re-approve this https://review.openstack.org/#/c/520042/ | 08:56 |
| d0ugal | moshele: done | 08:59 |
| skramaja | thanks d0ugal | 09:01 |
| *** holser has quit IRC | 09:02 | |
| Tengu | hello d0ugal ! care to have a look at that review? https://review.openstack.org/#/c/522720/ thank you :) | 09:02 |
| *** tzumainn has joined #tripleo | 09:03 | |
| d0ugal | Tengu: sorry, I'm not really qualified to review puppet. | 09:04 |
| Tengu | d0ugal: oh, no problem then :) | 09:04 |
| moshele | d0ugal: thanks | 09:04 |
| Tengu | d0ugal: what's your field? so that I'll avoid poking you for nothing :) | 09:05 |
| *** etingof has quit IRC | 09:05 | |
| d0ugal | Tengu: I work on Mistral and tripleo-common mostly. | 09:05 |
| Tengu | d0ugal: ok! so deploy team. | 09:06 |
| d0ugal | Tengu: workflows :) | 09:06 |
| Tengu | :) | 09:06 |
| *** ooolpbot has joined #tripleo | 09:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731063 | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1733345 | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1734134 | 09:10 |
| *** ooolpbot has quit IRC | 09:10 | |
| openstack | Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] | 09:10 |
| openstack | Launchpad bug 1733345 in tripleo "Master promotion: error creating the default Deployment Plan overcloud" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 09:10 |
| openstack | Launchpad bug 1734134 in tripleo "Pike periodic promotion job multinode-1ctlr-featureset016 fail with error running docker 'gnocchi_db_sync' - rados.Rados.connect PermissionDeniedError: error connecting to the cluster" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 09:10 |
| *** jchhatbar has joined #tripleo | 09:12 | |
| *** janki has quit IRC | 09:13 | |
| *** holser has joined #tripleo | 09:15 | |
| *** etingof has joined #tripleo | 09:20 | |
| *** mcornea has joined #tripleo | 09:21 | |
| openstackgerrit | Martin André proposed openstack/tripleo-heat-templates master: Remove hardcoded docker image names https://review.openstack.org/518094 | 09:23 |
| openstackgerrit | Martin André proposed openstack/tripleo-heat-templates master: Remove hardcoded skydive container image https://review.openstack.org/518095 | 09:23 |
| *** ffiore has quit IRC | 09:25 | |
| *** ffiore has joined #tripleo | 09:25 | |
| *** oidgar has quit IRC | 09:27 | |
| *** derekh has joined #tripleo | 09:28 | |
| *** nyechiel has joined #tripleo | 09:30 | |
| *** ffiore has quit IRC | 09:31 | |
| *** dbecker has joined #tripleo | 09:34 | |
| gfidente | hey guys, a relatively urgent pike cherry pick https://review.openstack.org/#/c/522512/ | 09:35 |
| *** spectr has joined #tripleo | 09:37 | |
| Tengu | gfidente: can't do more than +1… can you have a look at https://review.openstack.org/#/c/522720/ in the meanwhile? :) | 09:38 |
| gfidente | Tengu I think we don't want to use epel though | 09:39 |
| Tengu | gfidente: it's disabled. but the class is requested | 09:39 |
| Tengu | at least for recent collectd puppet modules | 09:39 |
| gfidente | Tengu so in other modules we use tags | 09:40 |
| gfidente | to skip anything that tries to pull in dependencies | 09:40 |
| gfidente | is this possibly a fix you can put in the collectd module? | 09:40 |
| Tengu | well, that basically means you won't be able to take recent collectd module, and be stuck with a +660 patch older version. | 09:40 |
| Tengu | not a really good deal imho. | 09:40 |
| Tengu | gfidente: I doubt it. the way puppet-collectd is done can't really avoid the epel thingy. at least they let free the activation or not of the epel sources. | 09:41 |
| Tengu | the collectd module has a "require ::epel" for redhat family. | 09:42 |
| Tengu | gfidente: https://github.com/voxpupuli/puppet-collectd/blob/master/manifests/repo/redhat.pp#L13 | 09:42 |
| openstackgerrit | Martin André proposed openstack/tripleo-heat-templates master: Remove httpd log dir for glance-api https://review.openstack.org/516231 | 09:42 |
| Tengu | gfidente: my patch allows to get a recent collectd module with bug fixes and more module supporte while avoiding epel being enabled by default. Guess it's all for the best. | 09:43 |
| bogdando | jistr: hi. perchance do you know why https://review.openstack.org/#/c/516894/ applies inner playbooks w/o become? that fails kubesprat install for my test env | 09:43 |
| gfidente | Tengu looks like we can just pass manage_repo: false ? | 09:45 |
| gfidente | and that will avoid inclusion of repo.pp completely | 09:45 |
| openstackgerrit | Saravanan KR proposed openstack/tripleo-heat-templates master: Add a tag to all the role specific parameters https://review.openstack.org/517231 | 09:45 |
| gfidente | Tengu yeah but what I am saying is that it doesn't seem to be necessary to approach it that way | 09:45 |
| gfidente | Tengu I think if we pass collectd::params::manage_repo: false | 09:45 |
| Tengu | gfidente: might be, but epel might be requested if I can push a small project to tripleo, in order to get certbot. | 09:46 |
| gfidente | it just won't try to include repo.pp and as such won't try to configure the epel repo either | 09:46 |
| Tengu | gfidente: more over, epel is required in order to get a few collectd modules. if ppl want them on their system, it's maybe better to at least put some fences to epel. | 09:46 |
| *** akrivoka has joined #tripleo | 09:46 | |
| openstackgerrit | Merged openstack/tripleo-docs master: Update upgrade documentation with the Ceph related steps https://review.openstack.org/511635 | 09:47 |
| Tengu | gfidente: for instance, the rabbitmq module comes from epel… might interest some ppl to get stats on that service ;). | 09:47 |
| openstackgerrit | Alfredo Moralejo proposed openstack/tripleo-heat-templates master: Revert "CI: remove useless templates" https://review.openstack.org/523071 | 09:47 |
| jistr | bogdando: become is in inventory now https://github.com/openstack/tripleo-heat-templates/blob/master/extraconfig/services/kubernetes-master.yaml#L107 | 09:48 |
| jistr | bogdando: to avoid using `become` on undercloud too | 09:48 |
| jistr | bogdando: the patch you linked hasn't been rebased yet | 09:48 |
| bogdando | jistr: right, so I hope a rebase fixed this for me | 09:48 |
| bogdando | just asked :) | 09:48 |
| bogdando | btw, jistr, flaper87 it's great that inner ansible log can be tailed live and looks ok, thanks for fixing logging! :) | 09:50 |
| openstackgerrit | Martin André proposed openstack/tripleo-quickstart-extras master: Switch to tripleomaster registry by default https://review.openstack.org/519593 | 09:50 |
| *** hjensas has quit IRC | 09:51 | |
| *** hjensas has joined #tripleo | 09:51 | |
| *** ffiore has joined #tripleo | 09:51 | |
| openstackgerrit | Saravanan KR proposed openstack/tripleo-heat-templates master: Remove unused pre_network configuration https://review.openstack.org/517582 | 09:52 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates stable/pike: Ensure os-net-config conditional for upgrade doesn’t fail. https://review.openstack.org/523073 | 09:52 |
| *** shardy has joined #tripleo | 09:53 | |
| skramaja | could someone review these patches - https://review.openstack.org/#/c/517857/ and https://review.openstack.org/#/c/520342/ | 09:54 |
| Tengu | gfidente: answered. if still not convinced, I'll abandon that patch and check with jbadiapa for the "manage_repo => false" part. both views are right I think. | 09:55 |
| gfidente | skramaja I think you could make those into a single patch | 10:03 |
| gfidente | but I wonder if it isn't better to add some code | 10:03 |
| gfidente | to validate the contents of those roles | 10:03 |
| ratailor | https://review.openstack.org/#/c/522845/ | 10:04 |
| ratailor | https://review.openstack.org/#/c/522847/ | 10:04 |
| ratailor | https://review.openstack.org/#/c/522843/ | 10:04 |
| ratailor | https://review.openstack.org/#/c/522848/ | 10:04 |
| ratailor | https://review.openstack.org/#/c/522846/ | 10:04 |
| ratailor | mwhahaha, shardy jaosorior EmilienM flaper87 gfidente Can you please review these ^^, these are GA blocker patches. | 10:04 |
| openstackgerrit | Martin André proposed openstack/tripleo-heat-templates stable/pike: Containerize cinder services under pacemaker in CI https://review.openstack.org/518856 | 10:06 |
| openstackgerrit | Martin André proposed openstack/tripleo-heat-templates stable/pike: Reintroduce containerized Cinder https://review.openstack.org/523076 | 10:06 |
| skramaja | gfidente: i was earlier single but dpdk shoudl be backported to pike where as sriov was introduced in queens only. thats why seprated. | 10:06 |
| skramaja | *it | 10:06 |
| gfidente | okay | 10:06 |
| gfidente | can you add just some code to validate the contents | 10:06 |
| gfidente | ? | 10:06 |
| skramaja | you mean via yaml-validate.py? gfidente | 10:07 |
| *** abregman has quit IRC | 10:08 | |
| *** owalsh is now known as owalsh_afk | 10:08 | |
| gfidente | skramaja yep | 10:08 |
| *** dtantsur|afk is now known as dtantsur | 10:08 | |
| skramaja | gfidente: sure, i will look in to it, but lets not block dpdk, its already delayed. i will add a follow-up to add validation as like hci roles.. | 10:09 |
| gfidente | skramaja ok so please answer in the review which will go in queens cycle only | 10:09 |
| gfidente | I'll +2 the one for pike, then we validate both in the queens submission? | 10:09 |
| *** abregman has joined #tripleo | 10:10 | |
| *** ooolpbot has joined #tripleo | 10:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731063 | 10:10 |
| openstack | Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1734134 | 10:10 |
| *** ooolpbot has quit IRC | 10:10 | |
| openstack | Launchpad bug 1734134 in tripleo "Pike periodic promotion job multinode-1ctlr-featureset016 fail with error running docker 'gnocchi_db_sync' - rados.Rados.connect PermissionDeniedError: error connecting to the cluster" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 10:10 |
| *** ramishra has quit IRC | 10:10 | |
| skramaja | gfidente: sure, i could do that. | 10:11 |
| *** ramishra has joined #tripleo | 10:12 | |
| *** dsariel has quit IRC | 10:13 | |
| gfidente | skramaja ack works for me | 10:14 |
| chem | matbu: ccamacho: hey, I think https://review.openstack.org/#/c/522577/ is ready for prime time | 10:18 |
| *** ramishra has quit IRC | 10:19 | |
| *** ramishra has joined #tripleo | 10:20 | |
| *** spectr has joined #tripleo | 10:22 | |
| *** spectr has quit IRC | 10:23 | |
| *** gfidente has quit IRC | 10:25 | |
| *** gfidente has joined #tripleo | 10:25 | |
| *** gfidente has quit IRC | 10:25 | |
| *** gfidente has joined #tripleo | 10:25 | |
| *** suuuper has joined #tripleo | 10:26 | |
| *** dparkes has quit IRC | 10:27 | |
| *** nyechiel has quit IRC | 10:29 | |
| *** holser has quit IRC | 10:42 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/puppet-tripleo master: ironic: support enabling staging drivers https://review.openstack.org/523082 | 10:45 |
| *** holser has joined #tripleo | 10:45 | |
| *** holser has quit IRC | 10:46 | |
| *** holser has joined #tripleo | 10:46 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/tripleo-heat-templates master: Enable ironic staging drivers in the undercloud https://review.openstack.org/523084 | 10:52 |
| openstackgerrit | Dmitry Tantsur proposed openstack/puppet-tripleo master: ironic: add support for the ansible deploy method https://review.openstack.org/523085 | 10:53 |
| *** pcaruana has quit IRC | 10:55 | |
| *** tvignaud has quit IRC | 10:56 | |
| *** pcaruana has joined #tripleo | 10:59 | |
| *** salmankhan has joined #tripleo | 10:59 | |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-quickstart master: Enable set_ctlplane_dns in ipa.yml environment https://review.openstack.org/523086 | 11:01 |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-quickstart-extras master: Introduce a variable to allow setting DNS on ctlplane network https://review.openstack.org/523087 | 11:01 |
| ratailor | https://review.openstack.org/#/c/522845/ | 11:01 |
| ratailor | https://review.openstack.org/#/c/522847/ | 11:01 |
| ratailor | https://review.openstack.org/#/c/522843/ | 11:01 |
| ratailor | https://review.openstack.org/#/c/522848/ | 11:01 |
| ratailor | https://review.openstack.org/#/c/522846/ | 11:01 |
| ratailor | jistr,mandre Can you please review these ^^ | 11:01 |
| *** tvignaud has joined #tripleo | 11:02 | |
| *** ianw has quit IRC | 11:03 | |
| *** ianw_ has joined #tripleo | 11:03 | |
| *** SlickNik has quit IRC | 11:03 | |
| openstackgerrit | Luke Hinds proposed openstack/tripleo-heat-templates master: Implements management of `/etc/login.defs` https://review.openstack.org/457985 | 11:03 |
| *** ianw_ is now known as ianw | 11:04 | |
| *** SlickNik has joined #tripleo | 11:04 | |
| *** sri_ has joined #tripleo | 11:06 | |
| *** udesale has quit IRC | 11:07 | |
| *** stendulker has quit IRC | 11:09 | |
| *** ooolpbot has joined #tripleo | 11:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731063 | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1734134 | 11:10 |
| openstack | Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] | 11:10 |
| *** ooolpbot has quit IRC | 11:10 | |
| openstack | Launchpad bug 1734134 in tripleo "Pike periodic promotion job multinode-1ctlr-featureset016 fail with error running docker 'gnocchi_db_sync' - rados.Rados.connect PermissionDeniedError: error connecting to the cluster" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 11:10 |
| *** salmankhan has quit IRC | 11:10 | |
| *** rfolco has joined #tripleo | 11:11 | |
| *** ratailor has quit IRC | 11:16 | |
| *** ukalifon has quit IRC | 11:17 | |
| *** hewbrocca_afk is now known as hewbrocca | 11:20 | |
| *** ccamacho is now known as ccamacho|doc | 11:22 | |
| *** pdeore has quit IRC | 11:26 | |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-quickstart-extras master: Collect var/lib/heat-config directory. https://review.openstack.org/523092 | 11:26 |
| *** dsariel has joined #tripleo | 11:29 | |
| *** rodolof has joined #tripleo | 11:34 | |
| *** spectr has joined #tripleo | 11:35 | |
| openstackgerrit | Michele Baldessari proposed openstack/puppet-tripleo master: WIP DNR Instance HA POC https://review.openstack.org/506922 | 11:37 |
| lyarwood | chem: any idea where https://review.openstack.org/#/c/521060/ is stuck? | 11:37 |
| openstackgerrit | Dmitry Tantsur proposed openstack/tripleo-heat-templates master: undercloud: support the standard set of hardware types and interface https://review.openstack.org/523093 | 11:37 |
| * lyarwood doesn't see it on http://zuulv3.openstack.org/ | 11:38 | |
| *** threestrands_ has quit IRC | 11:42 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/tripleo-heat-templates master: undercloud: enable ansible deploy method by default https://review.openstack.org/523094 | 11:43 |
| *** nyechiel has joined #tripleo | 11:44 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/tripleo-heat-templates master: Enable ironic staging drivers in the undercloud https://review.openstack.org/523084 | 11:45 |
| *** atoth has joined #tripleo | 11:49 | |
| *** patrickeast has quit IRC | 11:53 | |
| *** hrybacki has quit IRC | 11:54 | |
| *** hrybacki has joined #tripleo | 11:55 | |
| *** patrickeast has joined #tripleo | 11:55 | |
| *** tosky has joined #tripleo | 11:59 | |
| *** fzdarsky is now known as fzdarsky|afk | 12:00 | |
| *** jaganathan has quit IRC | 12:02 | |
| *** raildo has joined #tripleo | 12:05 | |
| *** pkovar has joined #tripleo | 12:06 | |
| *** ooolpbot has joined #tripleo | 12:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731063 | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1734134 | 12:10 |
| *** ooolpbot has quit IRC | 12:10 | |
| openstack | Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] | 12:10 |
| openstack | Launchpad bug 1734134 in tripleo "Pike periodic promotion job multinode-1ctlr-featureset016 fail with error running docker 'gnocchi_db_sync' - rados.Rados.connect PermissionDeniedError: error connecting to the cluster" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 12:10 |
| *** jkilpatr has joined #tripleo | 12:13 | |
| *** adarazs|ruck is now known as adarazs|ruck|lnc | 12:14 | |
| *** jkilpatr has quit IRC | 12:14 | |
| *** jkilpatr has joined #tripleo | 12:14 | |
| *** oidgar has joined #tripleo | 12:15 | |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui master: Change plan files whitelist when creating plan https://review.openstack.org/523100 | 12:15 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Fix applying pci whitelist in nova compute https://review.openstack.org/520042 | 12:16 |
| jtomasek | honza, florianf: could you please check ^^ so we can backport it to pike | 12:16 |
| jaosorior | shardy: I saw that we're passing the net_ip_map to the hieradata (since fairly recently) what is that being used for? | 12:20 |
| *** fzdarsky|afk is now known as fzdarsky | 12:20 | |
| therve | lyarwood, Parent is not merged | 12:21 |
| *** vpickard_ is now known as vpickard | 12:22 | |
| *** oidgar has quit IRC | 12:22 | |
| lyarwood | therve: \o/ I'm blind, thanks! | 12:23 |
| *** rodolof has quit IRC | 12:23 | |
| *** rodolof has joined #tripleo | 12:24 | |
| openstackgerrit | Merged openstack/puppet-tripleo master: Fix typo in ganesha VIP order constraint https://review.openstack.org/522956 | 12:24 |
| *** pkovar has quit IRC | 12:24 | |
| *** pkovar has joined #tripleo | 12:26 | |
| florianf | jtomasek: yup | 12:27 |
| *** udesale has joined #tripleo | 12:29 | |
| *** links has quit IRC | 12:29 | |
| *** psahoo has quit IRC | 12:33 | |
| *** ccamacho|doc is now known as ccamacho | 12:33 | |
| *** ffiore has quit IRC | 12:35 | |
| *** oidgar has joined #tripleo | 12:35 | |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-quickstart-extras master: Multinode provision role https://review.openstack.org/516795 | 12:37 |
| sshnaidm | adarazs|ruck|lnc, ^^ | 12:37 |
| *** aputtur_ has quit IRC | 12:38 | |
| *** amoralej is now known as amoralej|lunch | 12:42 | |
| *** links has joined #tripleo | 12:42 | |
| openstackgerrit | Saravanan KR proposed openstack/tripleo-heat-templates master: Add NovaMigrationTarget service to SR-IOV Compute role https://review.openstack.org/520342 | 12:45 |
| skramaja | gfidente: ^ - fyi | 12:45 |
| skramaja | gfidente: this is queens specific change, where as https://review.openstack.org/#/c/517857/ (DPDK) is needed for stable/pike too. | 12:46 |
| jpich | Any tips for debugging "server misbehaving" (DNS?) errors during deployment, when trying to download images from the docker registry? "docker pull" seems to work when I try it on random overcloud nodes, after the deployment failed | 12:48 |
| *** bfournie has quit IRC | 12:50 | |
| *** bfournie has joined #tripleo | 12:50 | |
| adarazs|ruck|lnc | sshnaidm: awesome, thanks for the fix. | 12:53 |
| *** adarazs|ruck|lnc is now known as adarazs|ruck | 12:53 | |
| *** bfournie has quit IRC | 12:54 | |
| *** jpena is now known as jpena|lunch | 12:55 | |
| *** lennyb__ has joined #tripleo | 12:56 | |
| *** mburned_out is now known as mburned | 12:59 | |
| *** jlabarre has joined #tripleo | 13:00 | |
| *** dtantsur is now known as dtantsur|bbl | 13:03 | |
| *** owalsh_afk is now known as owalsh | 13:03 | |
| *** lebauce_ has quit IRC | 13:04 | |
| *** dprince has joined #tripleo | 13:05 | |
| *** trown|outtypewww is now known as trown | 13:05 | |
| *** amoralej|lunch is now known as amoralej | 13:06 | |
| *** ffiore has joined #tripleo | 13:06 | |
| *** abishop has joined #tripleo | 13:07 | |
| *** aputtur_ has joined #tripleo | 13:09 | |
| *** ooolpbot has joined #tripleo | 13:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731063 | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1734134 | 13:10 |
| *** ooolpbot has quit IRC | 13:10 | |
| openstack | Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] | 13:10 |
| openstack | Launchpad bug 1734134 in tripleo "Pike periodic promotion job multinode-1ctlr-featureset016 fail with error running docker 'gnocchi_db_sync' - rados.Rados.connect PermissionDeniedError: error connecting to the cluster" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 13:10 |
| *** jkilpatr has quit IRC | 13:11 | |
| *** rhallisey has joined #tripleo | 13:11 | |
| jtomasek | apetrich: I've added one more comment to https://review.openstack.org/#/c/511249/8 do you see that change as a problem? | 13:12 |
| openstackgerrit | Michele Baldessari proposed openstack/puppet-tripleo master: WIP DNR Instance HA POC https://review.openstack.org/506922 | 13:12 |
| jaosorior | jistr: commented on your patch | 13:13 |
| *** ramishra has quit IRC | 13:14 | |
| *** stendulker has joined #tripleo | 13:14 | |
| *** ramishra has joined #tripleo | 13:15 | |
| *** mdnadeem has quit IRC | 13:15 | |
| *** dougbtv has joined #tripleo | 13:17 | |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-docs master: Fill in upgrades CI developer doc https://review.openstack.org/520636 | 13:17 |
| *** lebauce has joined #tripleo | 13:19 | |
| *** ykarel has quit IRC | 13:19 | |
| therve | EmilienM, http://logs.openstack.org/03/523103/1/check/puppet-openstack-integration-4-scenario002-tempest-centos-7/155a3cb/logs/puppet.txt.gz#_2017-11-27_13_04_26 | 13:20 |
| *** ramishra has quit IRC | 13:20 | |
| *** bfournie has joined #tripleo | 13:21 | |
| *** ansmith has joined #tripleo | 13:24 | |
| therve | EmilienM, There is something weird going on, it seems to be doing migrations twice? | 13:25 |
| mwhahaha | therve: if it errors, we retry | 13:26 |
| therve | mwhahaha, It doesn't seem to error though | 13:27 |
| *** skramaja has quit IRC | 13:27 | |
| mwhahaha | therve: i don't see the execution a second time | 13:27 |
| gfidente | adarazs|ruck not sure if you managed to reproduce the issue we're seeing with the promotion job? | 13:27 |
| adarazs|ruck | gfidente: no, my deployment failed on Friday, I'm trying it again. | 13:27 |
| therve | mwhahaha, See here http://logs.openstack.org/03/523103/1/check/puppet-openstack-integration-4-scenario002-tempest-centos-7/155a3cb/logs/nova/nova-manage.txt.gz#_2017-11-27_13_02_11_334 | 13:27 |
| therve | Goes back to 215 after being on 373 | 13:28 |
| gfidente | adarazs|ruck not sure if I can build the images somehow? | 13:28 |
| gfidente | adarazs|ruck my problem is that it tried to download images that did not exist | 13:28 |
| *** morazi has joined #tripleo | 13:28 | |
| mwhahaha | therve: that doesn't seem to be a 2nd execution because the req number is the same | 13:28 |
| therve | mwhahaha, Well, it doesn't seem to but it looks like one :) | 13:29 |
| mwhahaha | therve: and is also only 102 ms after 372-373 | 13:29 |
| mwhahaha | so i'd blame nova-manage | 13:29 |
| therve | Yeah trying to see if it's somewhere else | 13:29 |
| *** jkilpatr has joined #tripleo | 13:29 | |
| *** morazi has quit IRC | 13:30 | |
| *** morazi has joined #tripleo | 13:31 | |
| *** lblanchard has joined #tripleo | 13:31 | |
| *** pradk has quit IRC | 13:31 | |
| *** rlandy has joined #tripleo | 13:31 | |
| therve | mwhahaha, Yeah looks like it's doing it twice on normal jobs as well... | 13:32 |
| therve | Probably something like placement db | 13:33 |
| *** rbowen has joined #tripleo | 13:33 | |
| *** rlandy is now known as rlandy|rover | 13:34 | |
| *** masco has quit IRC | 13:35 | |
| *** brian83 has joined #tripleo | 13:36 | |
| *** daidv has quit IRC | 13:38 | |
| jistr | jaosorior, bogdando: i thought of making it unconditional too but i wasn't sure if it could have some negative effects on all the other networking variants... | 13:40 |
| *** pchavva has joined #tripleo | 13:40 | |
| shardy | jaosorior: sorry missed your ping, it's to enable per-node override of bind IPs for spine/leaf with composable networks | 13:43 |
| shardy | https://review.openstack.org/#/c/514707/ has more details | 13:43 |
| *** pchavva1 has joined #tripleo | 13:44 | |
| jaosorior | shardy: makes sense. Thanks | 13:44 |
| *** eck`gone is now known as eck` | 13:45 | |
| therve | mwhahaha, Should I change the default timeout at the very least? | 13:45 |
| mwhahaha | therve: no | 13:46 |
| mwhahaha | therve: this is a perf problem and should be addressed rather than upping the timeout | 13:46 |
| therve | mwhahaha, Well, it's a known perf problem, and there is no fix for it. | 13:46 |
| mwhahaha | therve: the default tiemout is 300 seconds | 13:46 |
| therve | I know | 13:46 |
| openstackgerrit | Luke Hinds proposed openstack/tripleo-heat-templates master: Implements AIDE Intrusion Detection System https://review.openstack.org/483028 | 13:47 |
| mwhahaha | and the answer we always get is that it should only take like 10 seconds | 13:47 |
| mwhahaha | so upping the timeout isn't fixing anything | 13:47 |
| mwhahaha | we already increase it to 600 in the low memory environment file | 13:47 |
| mwhahaha | therve: https://github.com/openstack/tripleo-heat-templates/blob/master/environments/low-memory-usage.yaml#L20 | 13:48 |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-quickstart-extras master: Set DNS on ctlplane network https://review.openstack.org/523087 | 13:48 |
| mwhahaha | sorry 900 | 13:48 |
| mwhahaha | so that's even worse | 13:48 |
| jistr | bogdando, jaosorior: trying as suggested ^ | 13:48 |
| therve | mwhahaha, Well it's not set there at least | 13:48 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates master: WIP Enable loadbalancer for kubernetes masters https://review.openstack.org/516894 | 13:48 |
| *** jpena|lunch is now known as jpena | 13:49 | |
| mwhahaha | therve: oh this is a p-o-i test | 13:49 |
| mwhahaha | therve: no we should raise this with infra | 13:49 |
| therve | mwhahaha, So remember that you raised this issue with zzzeek some months ago | 13:50 |
| bogdando | jistr: ack. I think that makes sense, rather then adding yet another deployment forking control flag | 13:50 |
| therve | And that seems to come from having poor performing VMs | 13:50 |
| bogdando | there are too many | 13:50 |
| therve | wrt to disk cache | 13:51 |
| mwhahaha | therve: no that was a different thing on dev envs. if we're seeing this on infra nodes we should docuemnt this so they can go raise with with the appropriate cloud providers | 13:51 |
| mwhahaha | therve: speaking of which that was on a vexxhost node | 13:52 |
| therve | Well we saw it in customer envs too. We just raised the heat timeout recently because of that | 13:52 |
| *** trozet has quit IRC | 13:52 | |
| mwhahaha | 300 seconds is already excessive so in this case no, this is not the solution for this one | 13:52 |
| therve | Well there is having a solution and having a workaround | 13:53 |
| mwhahaha | we already have a work around | 13:53 |
| therve | Just wasting test resources in the mean time is not a great idea either | 13:53 |
| mwhahaha | we set it to 300 seconds | 13:53 |
| *** trozet has joined #tripleo | 13:54 | |
| mwhahaha | mnaser: fyi http://logs.openstack.org/03/523103/1/check/puppet-openstack-integration-4-scenario002-tempest-centos-7/155a3cb/logs/nova/nova-manage.txt.gz#_2017-11-27_13_02_11_334 nova db sync timeout on a vexxhost node. usually means io issues | 13:54 |
| mwhahaha | therve: the ubuntu job version finished in 30 seconds, http://logs.openstack.org/03/523103/1/check/puppet-openstack-integration-4-scenario002-tempest-ubuntu-xenial/3528293/logs/nova/nova-manage.txt.gz | 13:56 |
| therve | mwhahaha, FWIW I raised https://bugs.launchpad.net/nova/+bug/1734698 | 13:56 |
| openstack | Launchpad bug 1734698 in OpenStack Compute (nova) "Squash database patches" [Undecided,New] | 13:56 |
| therve | mwhahaha, So your point is that we don't have a functional env anyway? | 13:58 |
| *** apetrich has quit IRC | 13:58 | |
| mwhahaha | therve: something like that, that test node was probably going to timeout on something else if it actually succeeded | 13:58 |
| *** apetrich has joined #tripleo | 13:59 | |
| therve | Fair | 13:59 |
| therve | mwhahaha, Should that error be blocking though? We seem to be proceeding with the run, until to hit a 500 on nova later on | 13:59 |
| mwhahaha | therve: the way puppet works it will continue on anything not explicilty defined as a requirement | 14:00 |
| therve | Right, should it be a requirement? | 14:00 |
| mwhahaha | therve: so when it comes to api requests, we don't know where it's running (it may not be local) | 14:00 |
| mwhahaha | so there is not a specific service requirement | 14:00 |
| *** karthiks has quit IRC | 14:01 | |
| mwhahaha | there's probably some improvement to be done using spaceship operators or something but requires time/efforts that folks aren't lining up to do. | 14:02 |
| * mwhahaha goes afk for a bit | 14:02 | |
| *** liverpooler has joined #tripleo | 14:05 | |
| *** jmelvin has joined #tripleo | 14:07 | |
| *** nyechiel has quit IRC | 14:09 | |
| *** ooolpbot has joined #tripleo | 14:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731063 | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1734134 | 14:10 |
| *** ooolpbot has quit IRC | 14:10 | |
| openstack | Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] | 14:10 |
| openstack | Launchpad bug 1734134 in tripleo "Pike periodic promotion job multinode-1ctlr-featureset016 fail with error running docker 'gnocchi_db_sync' - rados.Rados.connect PermissionDeniedError: error connecting to the cluster" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 14:10 |
| *** gbarros has joined #tripleo | 14:10 | |
| openstackgerrit | Andy Smith proposed openstack/puppet-tripleo master: Support both rabbitmq and oslo.messaging service nodes https://review.openstack.org/522406 | 14:12 |
| *** hjensas has quit IRC | 14:13 | |
| *** hjensas has joined #tripleo | 14:14 | |
| *** hjensas has quit IRC | 14:14 | |
| *** hjensas has joined #tripleo | 14:14 | |
| flaper87 | jistr: did you have problems getting kube-dns working with kubespray? I remember you dealing with some firewall rules | 14:15 |
| flaper87 | jistr: bogdando http://logs.openstack.org/51/521951/8/check/ansible-role-k8s-keystone-kubernetes-centos/7664471/primary/logs/containers/kube-dns-cf9d8c47-z69tr_kube-system_kubedns-c447681dec79f4b3f725153ce1171e990847ac9aeec51bc44d1d21bb9f5a22f6.txt.gz | 14:15 |
| flaper87 | jistr: bogdando shouldn't it be 6443? | 14:16 |
| jistr | we have 6443 yea | 14:17 |
| bogdando | flaper87: yes, it should be 6443 | 14:17 |
| jistr | and re networking yes i had to add the rules which are in kubernetes-master.yaml and -worker.yaml | 14:17 |
| flaper87 | mmh, thought kubespray would set those rules in iptables | 14:17 |
| openstackgerrit | Luke Hinds proposed openstack/tripleo-heat-templates master: Implements management of `/etc/login.defs` https://review.openstack.org/457985 | 14:17 |
| *** dhill_ has joined #tripleo | 14:19 | |
| rook | shardy: hey - ever seen where overcloud delete results in : | 14:19 |
| rook | ERROR: Remote error: InvalidToken | 14:19 |
| rook | [u' | 14:19 |
| *** gbarros has quit IRC | 14:19 | |
| rook | I see a fernet token error in the logs... any chance this is something easy to fix? | 14:19 |
| rook | this is after re-issuing a undercloud install | 14:19 |
| *** mrch has joined #tripleo | 14:19 | |
| rook | https://gist.github.com/jtaleric/56e4e894a1378df89212050dddd4602d shardy | 14:20 |
| shardy | rook: Hi! No sorry not seen that myself, looking | 14:20 |
| rook | shardy: running queries like heat stack-list works just fine | 14:21 |
| jistr | EmilienM, slagle: have you seen something like this earlier in config download CI? http://logs.openstack.org/88/522488/2/experimental/tripleo-ci-centos-7-scenario006-multinode-oooq/4c9be22/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2017-11-23_12_39_29 | 14:21 |
| rook | just deleting seems to cause things to fail | 14:21 |
| jistr | EmilienM, slagle: stack goes to CREATE_COMPLETE, client prints Deploying overcloud configuration and then exits 1 without further explanation... | 14:22 |
| shardy | rook: my guess is the heat encryption key changed and it now can't decrypt some DB contents | 14:22 |
| shardy | rook: do you have any backups to check that? | 14:22 |
| rook | shardy: nope. | 14:22 |
| rook | so, running undercloud install isn't as non-disruptive as i thought :) | 14:23 |
| shardy | rook: Ok, well it'll be hard to be sure, you could copy heat.conf noting auth_encryption_key then re-run the undercloud install | 14:23 |
| shardy | rook: well, it should be, if this key changes then it's a major bug | 14:23 |
| rook | shardy yes, this is a problem for sure, as it changed some rabbit passwords too | 14:24 |
| *** dtantsur|bbl is now known as dtantsur | 14:24 | |
| rook | i wonder if I can change keystone to uuid to get around this lol | 14:24 |
| shardy | rook: Ok, that should never happen | 14:24 |
| sshnaidm | mwhahaha, EmilienM, adarazs|ruck take a look please at subunit generation in jobs: https://review.openstack.org/#/c/522955/ | 14:24 |
| *** dpawar has quit IRC | 14:24 | |
| shardy | rook: if it's what I think, then it's not a keystone token error | 14:24 |
| rook | oh? | 14:24 |
| shardy | looking at the log you pasted the error is coming from heat's DB abstraction | 14:24 |
| rook | interseting | 14:25 |
| rook | ok | 14:25 |
| rook | i just saw fernet.py and thought keystone | 14:25 |
| shardy | /usr/lib/python2.7/site-packages/heat/common/crypt.py | 14:25 |
| rook | without considering there could be other crypto | 14:25 |
| rook | and ignoring the path | 14:25 |
| shardy | rook: was the undercloud upgraded (between major versions)? | 14:25 |
| rook | nope | 14:25 |
| rook | shardy: literally running undercloud install on the same version.... | 14:26 |
| rook | shardy: I switched some tht's to be non-containerzied... | 14:26 |
| rook | so I wanted to revert back to containerized... | 14:26 |
| shardy | rook: Ok, to clarify - this is a normal baremetal undercloud, not the experimental containerized one? | 14:26 |
| rook | right | 14:27 |
| rook | bm | 14:27 |
| shardy | I only ask because of the comment about tht | 14:27 |
| rook | shardy: i wanted to deploy a non-contizerized overcloud. | 14:27 |
| rook | hence the THT change. | 14:27 |
| rook | yeah | 14:27 |
| rook | sorry... | 14:27 |
| *** ramishra has joined #tripleo | 14:27 | |
| shardy | rook: ack, yeah I just wasn't clear how that was related to the undercloud install being re-run | 14:27 |
| *** pradk has joined #tripleo | 14:28 | |
| shardy | e.g if some packages got updated etc | 14:28 |
| rook | hm | 14:28 |
| slagle | jistr: i've not seen that. is it reproducable on that patch? | 14:28 |
| jistr | EmilienM, slagle: nevermind i think i see what the problem is, i'll try to fix it. I've been staring at it for 30 minutes and of course i realize what the problem is when i ping somone :D | 14:28 |
| jistr | slagle: yes reproducible both on k8s and openshift patches | 14:28 |
| slagle | jistr: in general, i think we need to fix the client to be more verbose on errors | 14:29 |
| slagle | jistr: if you get an error, you pretty much have to rerun with --debug to see what actually happened | 14:29 |
| jistr | slagle: yes i'll post that as well | 14:29 |
| jistr | slagle: the 'ctlplane' hidden in the output is a message of KeyError i believe. Will fix both. | 14:29 |
| jistr | we need to at least print the exception name | 14:30 |
| rook | shardy I wish I reported this a while ago, but I have been hitting this many times with Pike (downstream for sure). | 14:30 |
| jistr | slagle: hm the lack of verbosity might be problem of openstackclient actually | 14:32 |
| *** links has quit IRC | 14:32 | |
| slagle | jistr: well, i've noticed all of our log.info's only get shown if you run with --verbose | 14:33 |
| shardy | rook: Ok, we'll need a bug please - I'm just trying to see if I can reproduce, but I upgrade my undercloud nearly every day and I've never seen this, so I'll be quite surprised if we're breaking that key every update | 14:33 |
| slagle | jistr: which I don't think is what we've actually intended | 14:34 |
| *** zbitter has joined #tripleo | 14:34 | |
| rook | shardy: this was the error i saw in all the nova services on the UC | 14:34 |
| rook | 2017-11-27 08:11:46.799 37641 ERROR nova AccessRefused: (0, 0): (403) ACCESS_REFUSED - Login was refused using authentication mechanism AMQPLAIN. For details see the broker logfile. | 14:34 |
| rook | it was rabbitmq -- the way I got around this was uninstalling rabbitmq-server and deleting /var/lib/rabbit/* | 14:35 |
| rook | shardy: the assumption would be if I reissue a openstack undercloud install, this should fix heat, right? | 14:36 |
| *** akrivoka has quit IRC | 14:36 | |
| openstackgerrit | Christian Schwede proposed openstack/tripleo-heat-templates master: Ensure rsyncd PID file is removed during overcloud updates https://review.openstack.org/523134 | 14:36 |
| *** shardy is now known as shardy_afk | 14:37 | |
| openstackgerrit | Jiri Stransky proposed openstack/python-tripleoclient master: Deploy with --config-download even when some role has count 0 https://review.openstack.org/523136 | 14:42 |
| *** udesale has quit IRC | 14:42 | |
| *** noslzzp has joined #tripleo | 14:43 | |
| jistr | slagle: ^ i think that should help, will add a depends-on to kubespray CI test to prove it | 14:44 |
| *** rmascena has joined #tripleo | 14:45 | |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-heat-templates master: [NO MERGE] Test Kubespray CI https://review.openstack.org/522488 | 14:46 |
| *** raildo has quit IRC | 14:48 | |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-quickstart-extras master: DNM: test container upd with multiprocess https://review.openstack.org/515372 | 14:49 |
| *** pblaho has joined #tripleo | 14:50 | |
| EmilienM | hello | 14:51 |
| EmilienM | jistr: nice catch | 14:53 |
| EmilienM | therve: can I still help? | 14:53 |
| jistr | thx | 14:53 |
| jistr | can't find where to fix the verbosity though, i may just report it for now | 14:53 |
| therve | EmilienM, Maybe on https://review.openstack.org/#/c/523071/ | 14:54 |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-quickstart-extras master: DNM: test container upd with multiprocess https://review.openstack.org/515372 | 14:54 |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-quickstart-extras master: DNM: test container updates https://review.openstack.org/515372 | 14:57 |
| *** salmankhan has joined #tripleo | 14:57 | |
| *** suuuper has quit IRC | 14:57 | |
| EmilienM | therve: we won't approve this patch | 14:57 |
| *** suuuper has joined #tripleo | 14:57 | |
| EmilienM | therve: these jobs need to be removed from the periodic pipeline | 14:57 |
| EmilienM | therve: we don't want them anymore | 14:58 |
| therve | EmilienM, Yeah there are references everywhere though | 14:58 |
| therve | EmilienM, Even in tht jobs | 14:58 |
| jistr | d0ugal, jpich: fyi i've reported https://bugs.launchpad.net/tripleo/+bug/1734704 -- not high priority but a nice to have for UX if it's achievable. I looked for a while but can't see where to dive in the code to fix this. I guess it may be influenced by openstackclient too, not just tripleoclient (?) | 14:58 |
| openstack | Launchpad bug 1734704 in tripleo "Failures don't produce enough information" [Medium,Triaged] | 14:58 |
| *** zbitter is now known as zaneb | 14:58 | |
| therve | It should probably be the other way around | 14:58 |
| d0ugal | jistr: yeah, that is a openstackclient thing. | 14:59 |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates master: Allow to override manage polling param https://review.openstack.org/523141 | 14:59 |
| *** yamahata has quit IRC | 14:59 | |
| *** yamahata has joined #tripleo | 15:00 | |
| d0ugal | jistr: I am pretty sure it just prints out str(exception) | 15:00 |
| adarazs|ruck | gfidente: woo, I managed to reproduce the fs016 error \o/ | 15:00 |
| d0ugal | jistr: i.e. http://paste.openstack.org/show/627492/ | 15:00 |
| gfidente | adarazs|ruck ! thanks ! | 15:00 |
| jistr | yea :/ having at least the exception name there too might be nice... | 15:00 |
| adarazs|ruck | gfidente: is this your public key? https://github.com/gfidente.keys | 15:00 |
| gfidente | adarazs|ruck yes | 15:00 |
| jpich | d0ugal: It should be overridable at the specific client level, I think. Not sure which command was failing exactly though | 15:01 |
| jistr | jpich: it was `overcloud deploy` | 15:01 |
| bogdando | folks PTAL https://review.openstack.org/#/c/522874/ | 15:02 |
| * jistr updated the bug desc to mention that | 15:02 | |
| jpich | jistr: Right, I guess we might want to catch when calling the config_download function so we can get a more detailed description when there's an issue... I do agree with the sentiment that the default behaviour isn't the most ideal, too, though :) | 15:03 |
| *** fragatina has joined #tripleo | 15:03 | |
| *** holser has quit IRC | 15:04 | |
| *** hamzy has quit IRC | 15:06 | |
| therve | EmilienM, What about https://github.com/openstack/tripleo-heat-templates/blob/master/zuul.d/layout.yaml#L60 ? | 15:06 |
| mwhahaha | EmilienM: fyi tripleo-buildimage-overcloud-full-centos-7failure (non-voting) is in the gate for python-tripleoclient | 15:07 |
| *** ooolpbot has joined #tripleo | 15:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 15:10 |
| *** cylopez has quit IRC | 15:10 | |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731063 | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1734134 | 15:10 |
| *** ooolpbot has quit IRC | 15:10 | |
| openstack | Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] | 15:10 |
| openstack | Launchpad bug 1734134 in tripleo "Pike periodic promotion job multinode-1ctlr-featureset016 fail with error running docker 'gnocchi_db_sync' - rados.Rados.connect PermissionDeniedError: error connecting to the cluster" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 15:10 |
| *** gbarros has joined #tripleo | 15:10 | |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates master: Remove Cinder UID from CephX keyrings' ACLs https://review.openstack.org/522628 | 15:11 |
| *** fragatin_ has joined #tripleo | 15:12 | |
| *** bnemec has quit IRC | 15:13 | |
| *** fragatina has quit IRC | 15:13 | |
| EmilienM | therve: I'll update that this week | 15:13 |
| *** bogdando has quit IRC | 15:14 | |
| therve | Yeah I'm still not convinced by the process | 15:14 |
| EmilienM | mwhahaha: I think we want tripleo-buildimage-overcloud-full-centos-7 voting, right? | 15:14 |
| mwhahaha | EmilienM: i would think so, except it seems to be failing | 15:15 |
| EmilienM | ok | 15:15 |
| EmilienM | mwhahaha: any log? | 15:15 |
| mwhahaha | EmilienM: http://logs.openstack.org/46/522146/1/gate/tripleo-buildimage-overcloud-full-centos-7/fe9a1ed/job-output.txt.gz#_2017-11-27_11_20_50_788262 | 15:15 |
| mwhahaha | missing qemu-img-ev | 15:15 |
| mwhahaha | so bad repo config or something | 15:16 |
| EmilienM | it's bindep | 15:16 |
| mwhahaha | yea bad repo config from upstream | 15:16 |
| *** toure_biab is now known as toure | 15:16 | |
| mwhahaha | missing the centos ev repo or whatewver | 15:17 |
| EmilienM | mwhahaha: I'm pinging paul | 15:17 |
| *** agurenko has quit IRC | 15:17 | |
| EmilienM | in the meantime, let's disable voting - nothing we can do immediately I guess | 15:18 |
| mwhahaha | well it is currently non-voting, we just need to pull it from the gate | 15:19 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: Remove tripleo-buildimage-overcloud-full-centos-7 from gate https://review.openstack.org/523146 | 15:20 |
| EmilienM | mwhahaha: ^ | 15:20 |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-quickstart-extras master: Set DNS on ctlplane network https://review.openstack.org/523087 | 15:22 |
| *** marrusl has joined #tripleo | 15:23 | |
| openstackgerrit | Merged openstack/tripleo-common master: Finish deprecation of pull_source https://review.openstack.org/516138 | 15:25 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Ensure os-net-config conditional for upgrade doesn’t fail. https://review.openstack.org/522577 | 15:25 |
| openstackgerrit | Merged openstack/tripleo-common stable/pike: Generate password for libvirt TLS SCRAM auth https://review.openstack.org/522846 | 15:25 |
| openstackgerrit | Merged openstack/puppet-tripleo stable/pike: Configure libvirt SASL SCRAM-SHA1 when TLS is enabled https://review.openstack.org/522847 | 15:25 |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/pike: Re-enable libvirt TLS with SCRAM SHA-1 auth https://review.openstack.org/522848 | 15:25 |
| openstackgerrit | Merged openstack/tripleo-puppet-elements stable/pike: Include required packages for libvirt SASL auth https://review.openstack.org/522843 | 15:25 |
| openstackgerrit | Merged openstack/python-tripleoclient stable/pike: Allow the user to run ansible update on all nodes https://review.openstack.org/522146 | 15:25 |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: Switch to RDO Queens baseurl in overcloud upgrade https://review.openstack.org/517280 | 15:25 |
| *** rmascena has quit IRC | 15:25 | |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-quickstart-extras master: Set DNS on ctlplane network https://review.openstack.org/523087 | 15:26 |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: Include 'queens' as target version for upgrade. https://review.openstack.org/516647 | 15:32 |
| therve | EmilienM, so as we tracked this morning there is nothing in rdo referencing those envs | 15:33 |
| *** fultonj has joined #tripleo | 15:33 | |
| therve | It seems to come from quickstart | 15:33 |
| *** salmankhan has quit IRC | 15:34 | |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates master: Add name property where missing https://review.openstack.org/521992 | 15:34 |
| EmilienM | therve: what's the problem? | 15:34 |
| therve | EmilienM, Just that I'm not sure how to fix it | 15:34 |
| EmilienM | therve: ok now I remember why I didn't update THT zuul layout | 15:35 |
| *** oidgar has quit IRC | 15:35 | |
| EmilienM | therve: last week I found an issue with zuul, which checkouts the layout from master for stable branches jobs | 15:35 |
| therve | EmilienM, Should we just updated https://github.com/openstack/tripleo-quickstart/blob/master/config/general_config/featureset006.yml#L14 ? | 15:35 |
| EmilienM | therve: and I was afraid to it that again if I remove the baremetal jobs from tht/master | 15:35 |
| therve | OK | 15:36 |
| *** aufi has quit IRC | 15:36 | |
| EmilienM | therve: we should not run featureset006 in Queens and beyond | 15:36 |
| therve | Ah, ok | 15:36 |
| therve | amoralej, ^^^ | 15:37 |
| *** aufi has joined #tripleo | 15:37 | |
| amoralej | i'll send a review to remove all non-containers jobs | 15:37 |
| EmilienM | only for queens | 15:38 |
| EmilienM | adarazs|ruck: I'm still open to rename the ovb-ha job (back to our discussion from last week) | 15:39 |
| therve | Yeah it's blocking promotion right now | 15:39 |
| EmilienM | adarazs|ruck: we can do it at any time, and it should not block the things I'm doing now with ipv6 | 15:40 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient master: Add each command to the history file https://review.openstack.org/521859 | 15:40 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient master: If there is an exception, always print the full traceback. https://review.openstack.org/523151 | 15:40 |
| d0ugal | jistr, jpich: What about something like https://review.openstack.org/523151 (NOT YET TESTED!) | 15:40 |
| adarazs|ruck | EmilienM: yeah, okay. It's really not a big deal. My concern is/was easy reproduction of upstream jobs + easy way to related the featuresets, but we can do that other ways and have sexy names for all the upstream jobs. ;) | 15:41 |
| EmilienM | adarazs|ruck: I've asked around and it seems like your proposal is good | 15:41 |
| EmilienM | adarazs|ruck: if we can get someone working on the renaming | 15:42 |
| jpich | d0ugal: Would be pretty cool if we can get everything fixed in just one place yeah! I wonder about the full traceback though, IMO if you want that you run the command with --debug. Just a bit more info, like the exception type would help for "regular" errors... I think | 15:43 |
| d0ugal | jpich: --debug is waaaay too noisy IMO. | 15:43 |
| d0ugal | jpich: if I ever get an error or see a report for one I ask for the --debug output but only care about the traceback. | 15:44 |
| EmilienM | adarazs|ruck: I guess I was a fan of short name because *i* know what the job does, but it's not the case for everyone | 15:44 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates stable/pike: Ensure os-net-config conditional for upgrade doesn’t fail. https://review.openstack.org/523073 | 15:44 |
| adarazs|ruck | EmilienM: :) | 15:44 |
| adarazs|ruck | EmilienM: yeah, the problem with those names is that they are a bit misleading/they really don't tell the full story. so we opted for the featureset names because they suck, but at least they don't give you the idea that you know what's happening. :) | 15:45 |
| *** pabelanger has joined #tripleo | 15:45 | |
| *** dpawar has joined #tripleo | 15:47 | |
| EmilienM | pabelanger: fyi, https://review.openstack.org/#/c/523146/ | 15:48 |
| *** dpawar has quit IRC | 15:48 | |
| EmilienM | pabelanger: fyi_bis: tripleoclient gate is failing to deploy qemu-img-ev from bindep | 15:48 |
| jpich | d0ugal: I guess it might work well enough because we catch simple errors from the workflows and display them manually... I'm worried we'll end up with super long tracebacks for small errors like "plan overlcoud doesn't exist", which wouldn't be the best user experience either | 15:48 |
| openstackgerrit | Ryan Brady proposed openstack/python-tripleoclient master: Fix for timeouts on scale down https://review.openstack.org/522863 | 15:48 |
| EmilienM | pabelanger: maybe related to the repo change you mentioned last week | 15:48 |
| d0ugal | jpich: Right, I just commented on my review with something along those lines. | 15:49 |
| chem | matbu: ccamacho marios: help on fast backport merging https://review.openstack.org/#/c/522833/ :) | 15:49 |
| chem | matbu: marios and this one https://review.openstack.org/#/c/522833/ | 15:50 |
| marios | chem: ack will add to review list | 15:50 |
| *** hamzy has joined #tripleo | 15:50 | |
| matbu | chem: ack | 15:51 |
| *** salmankhan has joined #tripleo | 15:51 | |
| EmilienM | jaosorior: any luck with ipv6 + ssl? | 15:52 |
| jistr | jpich, d0ugal: e.g. Puppet has separate --debug and --trace options, maybe we could do that? --debug could also imply --trace | 15:52 |
| pabelanger | EmilienM: you likey need centos-release-openstack-ocata.rpm, or setup the repos manually. We removed it last week from centos DIB | 15:53 |
| EmilienM | pabelanger: ok so I can add centos-release-openstack-pike for ex | 15:54 |
| *** akrivoka has joined #tripleo | 15:54 | |
| *** bnemec has joined #tripleo | 15:54 | |
| *** jlabarre has quit IRC | 15:54 | |
| EmilienM | pabelanger: can I specify an in bindep? maybe you can send a patch | 15:54 |
| jaosorior | EmilienM: haven't had time to look deeper into it. but from what I looked into on Friday. We need support from the CI folks to know where that PublicVirtualFixedIps is coming from, and remove that, to use the one we're setting up. | 15:55 |
| jistr | jpich, d0ugal: but if there's an easy way to distinguish between the "expected exceptions" vs. "not expected exceptions" and decide for stack trace based on that, then that sounds good too | 15:55 |
| *** ramishra has quit IRC | 15:56 | |
| jpich | Expected exceptions :-) Yeah it does sounds nice, if it's feasible without too much awkwardness | 15:56 |
| EmilienM | jaosorior: ok I'll dig if they don't help | 15:56 |
| openstackgerrit | James Slagle proposed openstack/tripleo-quickstart-extras master: Fix loop in config-download.yaml.j2 template https://review.openstack.org/523155 | 15:56 |
| EmilienM | jaosorior: have you asked for help? | 15:56 |
| jaosorior | EmilienM: I did on friday. not today. I spaced out, sorry about that. | 15:57 |
| *** jlabarre has joined #tripleo | 15:57 | |
| *** rodolof has quit IRC | 15:58 | |
| pabelanger | EmilienM: no, you'll need to setup the repos before running bindep | 15:59 |
| openstackgerrit | mathieu bultel proposed openstack/python-tripleoclient stable/pike: Fix static-inventory option for minor update https://review.openstack.org/522217 | 15:59 |
| jpich | My deployment fails with just "resources.WorkflowTasks_Step2_Execution: ERROR" when I try to deploy with Ceph (that's pretty much the entire output of stack failures list), any tip for where I could look to get more information? | 15:59 |
| pabelanger | EmilienM: sadlly, bindep knows nothing about repos, it expects everything to be setup before it has been called | 15:59 |
| *** akane_ has quit IRC | 16:01 | |
| *** akane has quit IRC | 16:01 | |
| *** dpawar has joined #tripleo | 16:01 | |
| EmilienM | pabelanger: is it called before the pre.yaml playbooks? | 16:02 |
| *** ffiore has quit IRC | 16:02 | |
| d0ugal | jistr: --trace could make sense, but I guess that would need to be proposed to python-openstackclient. | 16:02 |
| *** jchhatbar has quit IRC | 16:03 | |
| *** ebarrera has quit IRC | 16:05 | |
| pabelanger | EmilienM: not sure, but sounds right. you could add it to pre_tasks for the bindep playbook | 16:05 |
| *** moshele has quit IRC | 16:06 | |
| *** shardy_afk is now known as shardy | 16:07 | |
| jistr | d0ugal: ah i see. I still forget we're not in control of the framework / top-level options. | 16:09 |
| *** ffiore has joined #tripleo | 16:09 | |
| *** gbarros has quit IRC | 16:10 | |
| *** ooolpbot has joined #tripleo | 16:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731063 | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1734134 | 16:10 |
| *** ooolpbot has quit IRC | 16:10 | |
| openstack | Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] | 16:10 |
| openstack | Launchpad bug 1734134 in tripleo "Pike periodic promotion job multinode-1ctlr-featureset016 fail with error running docker 'gnocchi_db_sync' - rados.Rados.connect PermissionDeniedError: error connecting to the cluster" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 16:10 |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/pike: Polish swift containers logging configuration https://review.openstack.org/521501 | 16:12 |
| jistr | jpich: in CI i used to go to mistral engine / executor logs, but there might be a better way in an interactive environment. I think there would be a failed workflow & action execution where hopefully the ansible logs could be fetched. | 16:14 |
| jpich | jistr: I'll start there, thanks! | 16:14 |
| *** fultonj has quit IRC | 16:14 | |
| *** dougbtv_ has quit IRC | 16:15 | |
| EmilienM | sshnaidm: why did you do recheck on https://review.openstack.org/#/c/522310/ ? | 16:16 |
| EmilienM | sshnaidm: it's failing here: http://logs.openstack.org/10/522310/9/check-tripleo/tripleo-ci-centos-7-ovb-ha-oooq/af061eb/logs/overcloud-controller-0/var/log/containers/neutron/neutron-server.log.txt.gz#_2017-11-27_04_17_19_919 | 16:16 |
| EmilienM | sshnaidm: please no random recheck | 16:16 |
| *** gkadam has quit IRC | 16:17 | |
| EmilienM | jaosorior: I'm now investigating the classic ovb-ha on ipv4, it's also failing on ssl, http://logs.openstack.org/10/522310/9/check-tripleo/tripleo-ci-centos-7-ovb-ha-oooq/af061eb/logs/overcloud-controller-0/var/log/containers/neutron/neutron-server.log.txt.gz#_2017-11-27_04_17_19_919 | 16:18 |
| EmilienM | jaosorior: any idea where it could come from? | 16:18 |
| *** akane has joined #tripleo | 16:19 | |
| *** akane_ has joined #tripleo | 16:19 | |
| *** akane has quit IRC | 16:20 | |
| *** akane_ has quit IRC | 16:20 | |
| EmilienM | jaosorior: I'm wondering if it's because I missed OS::TripleO::Services::CACerts in https://review.openstack.org/#/c/522306/9/ci/environments/ovb-ha.yaml | 16:20 |
| *** gkadam has joined #tripleo | 16:21 | |
| *** apetrich has quit IRC | 16:22 | |
| *** dparkes has joined #tripleo | 16:22 | |
| jaosorior | EmilienM: so, that's surely is an issue. yes. | 16:23 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: ci: add ovb-ha.yaml https://review.openstack.org/522306 | 16:23 |
| EmilienM | jaosorior: thanks for confirming | 16:23 |
| *** apetrich has joined #tripleo | 16:24 | |
| *** gbarros has joined #tripleo | 16:24 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: Run ovb-ha with minimal services https://review.openstack.org/522310 | 16:24 |
| *** gbarros has quit IRC | 16:25 | |
| *** Goneri has joined #tripleo | 16:27 | |
| *** suuuper has quit IRC | 16:28 | |
| openstackgerrit | Merged openstack/tripleo-docs master: ironic: rework enrollment with the recent changes https://review.openstack.org/510867 | 16:29 |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: Adding the possibility to pass the log url to tempestmail https://review.openstack.org/518327 | 16:30 |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Setting default log url for tempestmail https://review.openstack.org/518723 | 16:30 |
| EmilienM | jaosorior: hey found it I think | 16:31 |
| *** gfidente has quit IRC | 16:31 | |
| EmilienM | jaosorior: https://review.openstack.org/#/c/522714/2/roles/overcloud-prep-config/tasks/main.yml | 16:31 |
| EmilienM | jaosorior: look the "when" statement | 16:32 |
| EmilienM | when: ssl_overcloud|bool and undercloud_type != "baremetal" and overcloud_ipv6|bool | 16:32 |
| *** raildo has joined #tripleo | 16:32 | |
| EmilienM | jaosorior: undercloud_type is a baremetal in OVB jobs | 16:32 |
| *** marrusl has quit IRC | 16:32 | |
| EmilienM | jaosorior: so PublicVirtualFixedIPs is never configured like we want | 16:32 |
| openstackgerrit | James Slagle proposed openstack/tripleo-quickstart-extras master: Fix loop in config-download.yaml.j2 template https://review.openstack.org/523155 | 16:33 |
| jaosorior | EmilienM: isn't it? http://logs.openstack.org/18/522618/5/check-tripleo/tripleo-ci-centos-7-ovb-ha-oooq-ipv6/56df109/logs/undercloud/home/zuul/network-environment.yaml.txt.gz | 16:33 |
| jaosorior | EmilienM: was pretty sure that that specific code was setting up the PublicVirtualFixedIps value there ^^ | 16:33 |
| EmilienM | oh ok | 16:34 |
| jaosorior | now... why is it being skipped for "baremetal"? | 16:34 |
| jaosorior | that's kinda strange | 16:34 |
| *** gfidente has joined #tripleo | 16:35 | |
| *** gfidente has quit IRC | 16:35 | |
| *** gfidente has joined #tripleo | 16:35 | |
| EmilienM | jaosorior: are we sure the problem comes from this parameter? | 16:37 |
| *** bogdando has joined #tripleo | 16:38 | |
| jaosorior | EmilienM: I think that's the issue. unless we use some network configuration that ignores that parameter. But that shouldn't be the case for network isolation. | 16:38 |
| jaosorior | EmilienM: also, this doesn't seem to be an issue with IPv4 :/ | 16:39 |
| sshnaidm | EmilienM, I saw this, it wasn't random recheck | 16:39 |
| sshnaidm | EmilienM, I don't do random rechecks usually | 16:39 |
| EmilienM | sshnaidm: good | 16:39 |
| EmilienM | jaosorior: "Certificate did not match expected hostname: 2001:db8:fd00:1000::16" | 16:39 |
| EmilienM | jaosorior: something in this script: http://logs.openstack.org/18/522618/5/check-tripleo/tripleo-ci-centos-7-ovb-ha-oooq-ipv6/56df109/logs/undercloud/home/zuul/overcloud-create-ssl-cert.sh.txt.gz | 16:40 |
| bnemec | Note that PublicVirtualFixedIps was hard-coded in the ipv4 templates in ci: http://git.openstack.org/cgit/openstack-infra/tripleo-ci/tree/test-environments/network-templates/network-environment.yaml#n16 | 16:41 |
| *** stendulker has quit IRC | 16:41 | |
| jaosorior | EmilienM: so that's what I was looking at. If you notice the network-environment.yaml that I passed, it requested the ::14 address. That script that generates the cert also requests a cert for the ::14 address... so... what I haven't figured out is, where does the ::16 come from? | 16:42 |
| EmilienM | bnemec: I don't think we use that file anymore but unsure | 16:42 |
| bnemec | EmilienM: Right, but I'm pretty sure it got copied verbatim to tht. | 16:43 |
| EmilienM | jaosorior: where did you see ::16 ? | 16:43 |
| openstackgerrit | Tim Rozet proposed openstack/puppet-tripleo master: Enables websocket based port status for OpenDaylight https://review.openstack.org/515576 | 16:43 |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates master: WIP: Convert tags to when statements for Q major upgrade workflow https://review.openstack.org/510902 | 16:43 |
| jaosorior | EmilienM: in the actual error message. | 16:43 |
| EmilienM | jaosorior: oh nevermind | 16:43 |
| EmilienM | yeah | 16:43 |
| EmilienM | bnemec: we're debugging http://logs.openstack.org/18/522618/5/check-tripleo/tripleo-ci-centos-7-ovb-ha-oooq-ipv6/56df109/logs/undercloud/home/zuul/overcloud_deploy_post.log.txt.gz#_2017-11-24_08_43_24 | 16:43 |
| EmilienM | jaosorior: true, sorry | 16:44 |
| *** jpena is now known as jpena|brb | 16:44 | |
| bnemec | EmilienM: Yeah, I'm just saying there are probably assumptions in the ci ssl stuff that the public vip is going to be set by the templates. | 16:45 |
| bnemec | But that's not the case for the ipv6 templates. | 16:45 |
| jaosorior | bnemec: it should be though http://logs.openstack.org/18/522618/5/check-tripleo/tripleo-ci-centos-7-ovb-ha-oooq-ipv6/56df109/logs/undercloud/home/zuul/network-environment.yaml.txt.gz | 16:45 |
| EmilienM | jaosorior: see http://logs.openstack.org/18/522618/5/check-tripleo/tripleo-ci-centos-7-ovb-ha-oooq-ipv6/56df109/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2017-11-24_08_42_58 | 16:45 |
| *** marios has quit IRC | 16:45 | |
| EmilienM | Overcloud Endpoint: https://[2001:db8:fd00:1000::16]:13000/ | 16:45 |
| bnemec | jaosorior: Are you sure that file is actually getting used? | 16:45 |
| jaosorior | EmilienM: right, that's the ::16 address that I have no idea where it comes from. | 16:46 |
| jaosorior | fucking shit | 16:47 |
| jaosorior | well that was pointless | 16:47 |
| *** akane has joined #tripleo | 16:47 | |
| EmilienM | -e /opt/stack/new/tripleo-ci/test-environments/ipv6-network-templates/network-environment.yaml | 16:47 |
| jaosorior | bnemec, EmilienM: I was sure it was being used. it isn't http://logs.openstack.org/18/522618/5/check-tripleo/tripleo-ci-centos-7-ovb-ha-oooq-ipv6/56df109/logs/undercloud/home/zuul/overcloud-deploy.sh | 16:47 |
| *** akane_ has joined #tripleo | 16:47 | |
| EmilienM | I'm wondering if /opt/stack/new/tripleo-ci/test-environments/ipv6-network-templates/network-environment.yaml is the ony we would like | 16:47 |
| bnemec | Yeah | 16:47 |
| bnemec | Note that the public vip _shouldn't_ be set in network-environment IMHO. | 16:48 |
| EmilienM | see https://review.openstack.org/#/c/522615/4/config/general_config/featureset035.yml | 16:48 |
| jaosorior | bnemec: agree | 16:48 |
| bnemec | That was a change that got in before I noticed. | 16:48 |
| bnemec | The main reason being that network-environment.yaml is a generated file, so if someone makes changes via the UI and re-generates it you'll lose that value. | 16:48 |
| jaosorior | bnemec: which is why I did the change for quickstart to generate that file. I wonder why it is mostly using ipv4 addresses and not being used though | 16:49 |
| jaosorior | * why I did the change in the quickstart-generated file | 16:49 |
| bnemec | That I can't answer off the top of my head. :-) | 16:50 |
| jaosorior | adarazs|ruck, sshnaidm any idea why the ipv6 job isn't using the quickstart-generated network-environment.yaml file? http://logs.openstack.org/18/522618/5/check-tripleo/tripleo-ci-centos-7-ovb-ha-oooq-ipv6/56df109/logs/undercloud/home/zuul/overcloud-deploy.sh | 16:51 |
| EmilienM | bnemec: I agree, I think it's a misunderstanding from the folks who wrote this ansible task | 16:51 |
| EmilienM | jaosorior: see https://review.openstack.org/#/c/522615/4/config/general_config/featureset035.yml | 16:51 |
| *** gfidente has quit IRC | 16:51 | |
| EmilienM | -e {{ tripleo_root }}/tripleo-ci/test-environments/ipv6-network-templates/network-environment.yaml | 16:51 |
| EmilienM | (I re-used the lines from featureset001, ovb-ha) | 16:52 |
| EmilienM | which is on ipv4 | 16:52 |
| sshnaidm | jaosorior, if it doesn't have network-isolation, it doesn't use network-environments.yaml | 16:52 |
| *** gfidente has joined #tripleo | 16:52 | |
| *** gfidente has quit IRC | 16:52 | |
| *** gfidente has joined #tripleo | 16:52 | |
| sshnaidm | EmilienM, ^^ | 16:52 |
| jaosorior | oh, right | 16:52 |
| EmilienM | so we never used the generated file :( | 16:52 |
| sshnaidm | jaosorior, that's techdebt in oooq, rlandy|rover is working on this | 16:52 |
| jaosorior | ok | 16:52 |
| EmilienM | oh that's https://review.openstack.org/#/c/521664/ | 16:53 |
| sshnaidm | yep | 16:53 |
| rlandy|rover | jaosorior: sshnaidm: yep - https://review.openstack.org/#/c/521664/ is on my plate to work on today | 16:53 |
| EmilienM | rlandy|rover: could you please take a look at my question? https://review.openstack.org/#/c/522615/4/config/general_config/featureset035.yml | 16:55 |
| adarazs|ruck | gfidente: FYI please add reviews for stable/pike too: https://bugs.launchpad.net/tripleo/+bug/1734134/comments/6 | 16:55 |
| openstack | Launchpad bug 1734134 in tripleo "Pike periodic promotion job multinode-1ctlr-featureset016 fail with error running docker 'gnocchi_db_sync' - rados.Rados.connect PermissionDeniedError: error connecting to the cluster" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 16:55 |
| *** gbarros has joined #tripleo | 16:55 | |
| adarazs|ruck | gfidente: or you have to get them merged to master first? | 16:56 |
| EmilienM | rlandy|rover: I guess I'll just want for your patch to land :D | 16:56 |
| gfidente | adarazs|ruck ideally master first | 16:57 |
| adarazs|ruck | okay. | 16:57 |
| adarazs|ruck | I hope we'll get some master job runs today. | 16:57 |
| *** rhallisey has quit IRC | 16:57 | |
| *** rhallisey has joined #tripleo | 16:58 | |
| rlandy|rover | EmilienM: responded in the patch | 16:58 |
| rlandy|rover | you should not require my changes - most of which are needed when there is no network isolation | 16:58 |
| *** moshele has joined #tripleo | 16:58 | |
| *** toure is now known as toure_biab | 16:59 | |
| *** salmankhan has quit IRC | 16:59 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: WIP: POC: logs collection from native zuulv3 job https://review.openstack.org/522554 | 17:00 |
| lyarwood | hopefully a stupid question but is docker-puppet.py responsible for moving configs under /var/lib/config-data/puppet-generated/? This doesn't appear to happen for the iscsid service on computes and is causing my deployments (as part of FFU) to fail. | 17:01 |
| EmilienM | rlandy|rover: ok, I see, thank you! | 17:01 |
| sshnaidm | rlandy|rover, left comment too | 17:02 |
| *** gfidente has quit IRC | 17:03 | |
| *** gfidente has joined #tripleo | 17:04 | |
| *** gfidente has quit IRC | 17:04 | |
| *** gfidente has joined #tripleo | 17:04 | |
| *** lebauce has quit IRC | 17:04 | |
| bogdando | jistr, shardy: do you know how to w/a https://bugs.launchpad.net/tripleo/+bug/1733837 ? | 17:04 |
| openstack | Launchpad bug 1733837 in tripleo "Compute only stack fail to create" [Medium,In progress] - Assigned to Steven Hardy (shardy) | 17:04 |
| *** holser has joined #tripleo | 17:04 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: tripleo-buildimage: deploy repo in pre-run https://review.openstack.org/523167 | 17:04 |
| EmilienM | pabelanger: ^ | 17:05 |
| openstackgerrit | Emilien Macchi proposed openstack/python-tripleoclient master: CI test - DNM https://review.openstack.org/523168 | 17:05 |
| EmilienM | sshnaidm, rlandy|rover : thanks for your comments, it makes sense to me now! | 17:08 |
| *** pcaruana has quit IRC | 17:09 | |
| lyarwood | owalsh: ^ re docker-puppet moving things under /var/lib/config-data/puppet-generated/ , I'm still stuck trying to work out why the iscsid configs aren't ending up in there for computes, it wasn't a missing yum update after all. | 17:10 |
| *** akane_ has quit IRC | 17:10 | |
| *** akane has quit IRC | 17:10 | |
| *** ooolpbot has joined #tripleo | 17:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731063 | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1734134 | 17:10 |
| *** ooolpbot has quit IRC | 17:10 | |
| openstack | Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] | 17:10 |
| openstack | Launchpad bug 1734134 in tripleo "Pike periodic promotion job multinode-1ctlr-featureset016 fail with error running docker 'gnocchi_db_sync' - rados.Rados.connect PermissionDeniedError: error connecting to the cluster" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 17:10 |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/pike: Stop the object-expirer service before removing swift-proxy https://review.openstack.org/521917 | 17:10 |
| openstackgerrit | Merged openstack/tripleo-common master: Handle 'false' in when statements for ansible upgrade_tasks https://review.openstack.org/522539 | 17:10 |
| openstackgerrit | Merged openstack/tripleo-common master: config: Always add step conditional first for upgrade_tasks https://review.openstack.org/521060 | 17:10 |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Set ceph_use_fqdn for rdocloud https://review.openstack.org/519744 | 17:10 |
| openstackgerrit | Merged openstack/python-tripleoclient stable/pike: Move back to docker-site.yaml for ceph ansible update https://review.openstack.org/522891 | 17:10 |
| *** psachin has quit IRC | 17:11 | |
| *** akane has joined #tripleo | 17:11 | |
| *** akane_ has joined #tripleo | 17:11 | |
| *** ffiore has quit IRC | 17:12 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: Introduce fs035, ovb-ha-ipv6 https://review.openstack.org/522615 | 17:12 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: Introduce fs035, ovb-ha-ipv6 https://review.openstack.org/522615 | 17:12 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: Create ovb-ha-ipv6 and deprecate fs024 job https://review.openstack.org/522618 | 17:12 |
| *** marrusl has joined #tripleo | 17:14 | |
| openstackgerrit | Emilien Macchi proposed openstack/python-tripleoclient master: CI test - DNM https://review.openstack.org/523168 | 17:14 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: tripleo-buildimage: deploy repo in pre-run https://review.openstack.org/523167 | 17:14 |
| openstackgerrit | Emilien Macchi proposed openstack/python-tripleoclient master: CI test DNM https://review.openstack.org/523168 | 17:15 |
| *** akane_ has quit IRC | 17:16 | |
| *** akane has quit IRC | 17:16 | |
| owalsh | lyarwood: didn't you say the resource registry was still using the puppet service? | 17:16 |
| *** gkadam has quit IRC | 17:16 | |
| gfidente | adarazs|ruck so regarding the pike/master divergence | 17:16 |
| gfidente | aren't scenario jobs using images built from master branch? | 17:17 |
| EmilienM | pabelanger: we can't merge anything in stable/ocata puppet-tripleo right now, can you please help? | 17:17 |
| lyarwood | owalsh: that was my mistake, I was actually using /home/stack/tripleo-heat-templates/environments/docker.yaml | 17:17 |
| lyarwood | owalsh: I can see the config container started on the compute for iscsid but the files for it never end up under /var/lib/config-data/puppet-generated/ as they do on the controller | 17:18 |
| *** dpawar has quit IRC | 17:18 | |
| pabelanger | EmilienM: I'd recommend reaching out to jeblair today. We'll need his expertise to help unblock | 17:18 |
| lyarwood | owalsh: I *think* /var/lib/docker-puppet/docker-puppet.sh should move them there, just looking for confirmation | 17:19 |
| *** aufi has quit IRC | 17:20 | |
| owalsh | lyarwood: could it be the rolling upgrade mechanism excluding the computes from the upgrade? | 17:20 |
| lyarwood | owalsh: nope, I'm running the deploy_steps_playbook.yaml playbook so it might be an issue with that | 17:21 |
| openstackgerrit | Andy Smith proposed openstack/puppet-tripleo master: WIP Support separate oslo.messaging services for RPC and Notifications https://review.openstack.org/510684 | 17:21 |
| bogdando | jistr, shardy: wrt the bug, removing 'disable_upgrade_deployment: true' looks like a working w/a | 17:22 |
| owalsh | lyarwood: possible that /etc/iscsi/.initiator_reset is being mapped from the host to the config container, so puppet does nothing | 17:24 |
| *** moshele has quit IRC | 17:25 | |
| *** jpena|brb is now known as jpena | 17:25 | |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: Multinode provision role https://review.openstack.org/516795 | 17:26 |
| lyarwood | owalsh: \o/ | 17:26 |
| lyarwood | owalsh: yeah that's it | 17:26 |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates master: Add S3 storage driver as an option and its config https://review.openstack.org/521093 | 17:27 |
| owalsh | lyarwood: could you raise an LP for that, think it will be an issue with regular upgrades too | 17:27 |
| lyarwood | owalsh: ack thanks again :) | 17:27 |
| owalsh | lyarwood: np | 17:28 |
| *** lebauce_ has joined #tripleo | 17:29 | |
| *** rwsu has joined #tripleo | 17:32 | |
| *** dsariel has quit IRC | 17:33 | |
| openstackgerrit | Merged openstack/tripleo-ui master: Change plan files whitelist when creating plan https://review.openstack.org/523100 | 17:33 |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: Make ovb-manage-stack role usable for upstream reproducers https://review.openstack.org/518765 | 17:35 |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/pike: Make CephPools type json instead of list https://review.openstack.org/522512 | 17:35 |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: ovb: run ovb-containers-oooq only on pike https://review.openstack.org/522579 | 17:35 |
| openstackgerrit | Merged openstack/tripleo-quickstart master: scenario001: stop running pingtest https://review.openstack.org/522590 | 17:35 |
| *** jfrancoa has quit IRC | 17:39 | |
| *** iranzo has quit IRC | 17:41 | |
| *** suuuper has joined #tripleo | 17:42 | |
| *** toure_biab is now known as toure | 17:44 | |
| openstackgerrit | Merged openstack/tripleo-puppet-elements master: Add libreswan (IPSEC management) package to overcloud images https://review.openstack.org/521728 | 17:45 |
| EmilienM | jaosorior: I'm not sure why we need https://review.openstack.org/#/c/521728/ since we moved to containers | 17:46 |
| mwhahaha | EmilienM: the tunnel isn't in a container | 17:50 |
| mwhahaha | network config is on host | 17:51 |
| jaosorior | I could containerize the userspace utility and make it very painful to manage the tunnels :D | 17:51 |
| EmilienM | oh ok | 17:51 |
| mwhahaha | jaosorior: plz do and make sure to assign all bugs to EmilienM | 17:51 |
| * mwhahaha puts EmilienM into a container | 17:51 | |
| EmilienM | lol | 17:51 |
| EmilienM | only if I'm controlled by k8s | 17:52 |
| *** anilvenkata has quit IRC | 17:59 | |
| *** derekh has quit IRC | 18:02 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/pike: Fix wrong logic for triggering os-net-config special handling. https://review.openstack.org/522833 | 18:06 |
| *** ebarrera has joined #tripleo | 18:07 | |
| *** ooolpbot has joined #tripleo | 18:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731063 | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1734134 | 18:10 |
| *** ooolpbot has quit IRC | 18:10 | |
| openstack | Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] | 18:10 |
| openstack | Launchpad bug 1734134 in tripleo "Pike periodic promotion job multinode-1ctlr-featureset016 fail with error running docker 'gnocchi_db_sync' - rados.Rados.connect PermissionDeniedError: error connecting to the cluster" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 18:10 |
| *** fragatin_ has quit IRC | 18:11 | |
| *** yprokule has quit IRC | 18:17 | |
| *** gfidente has quit IRC | 18:17 | |
| *** moshele has joined #tripleo | 18:18 | |
| EmilienM | mwhahaha: https://review.openstack.org/#/c/523167/ fix the job and we can make it voting later | 18:19 |
| *** holser has quit IRC | 18:19 | |
| *** pradk_ has joined #tripleo | 18:20 | |
| *** gfidente has joined #tripleo | 18:20 | |
| *** gfidente has quit IRC | 18:20 | |
| *** gfidente has joined #tripleo | 18:20 | |
| mwhahaha | EmilienM: so that'll break in 6 months when ocata eols? | 18:20 |
| EmilienM | mwhahaha: yeah I need to change the version probably | 18:20 |
| *** jpena is now known as jpena|off | 18:20 | |
| EmilienM | mwhahaha: or making it the same as the branch | 18:20 |
| EmilienM | mwhahaha: I can do that | 18:20 |
| mwhahaha | EmilienM: k | 18:21 |
| *** holser has joined #tripleo | 18:21 | |
| *** dparkes has quit IRC | 18:21 | |
| *** bkopilov has quit IRC | 18:21 | |
| *** pradk has quit IRC | 18:22 | |
| *** fzdarsky is now known as fzdarsky|afk | 18:23 | |
| *** akrivoka has quit IRC | 18:24 | |
| *** tosky has quit IRC | 18:26 | |
| *** trown is now known as trown|lunch | 18:29 | |
| *** hewbrocca is now known as hewbrocca_afk | 18:30 | |
| *** gfidente has quit IRC | 18:31 | |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: Skip tempest test test_implied_roles_create_check_show_delete https://review.openstack.org/522479 | 18:31 |
| *** salmankhan has joined #tripleo | 18:33 | |
| *** gfidente has joined #tripleo | 18:34 | |
| *** gfidente has quit IRC | 18:34 | |
| *** gfidente has joined #tripleo | 18:34 | |
| *** abregman has quit IRC | 18:38 | |
| *** gkadam has joined #tripleo | 18:39 | |
| openstackgerrit | Merged openstack/puppet-tripleo master: Change haproxy check to tcpka for ec2_api https://review.openstack.org/517655 | 18:39 |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/ocata: Add support for Dell EMC VMAX ISCSI Backend https://review.openstack.org/518352 | 18:39 |
| *** newmember has quit IRC | 18:43 | |
| openstackgerrit | Lee Yarwood proposed openstack/tripleo-common stable/pike: config: Always add step conditional first for upgrade_tasks https://review.openstack.org/522831 | 18:46 |
| *** fragatina has joined #tripleo | 18:47 | |
| *** fragatin_ has joined #tripleo | 18:48 | |
| *** pkovar has quit IRC | 18:49 | |
| *** fragatina has quit IRC | 18:51 | |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: Allow pre-installed DLRN https://review.openstack.org/499117 | 18:53 |
| *** moshele has quit IRC | 18:54 | |
| *** moshele has joined #tripleo | 18:57 | |
| *** suuuper has quit IRC | 18:57 | |
| *** aputtur_ has quit IRC | 18:59 | |
| *** ccamacho has quit IRC | 19:02 | |
| *** jpich has quit IRC | 19:04 | |
| *** aputtur_ has joined #tripleo | 19:04 | |
| *** dpawar has joined #tripleo | 19:06 | |
| *** holser has quit IRC | 19:08 | |
| *** ooolpbot has joined #tripleo | 19:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731063 | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1734134 | 19:10 |
| *** ooolpbot has quit IRC | 19:10 | |
| openstack | Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] | 19:10 |
| openstack | Launchpad bug 1734134 in tripleo "Pike periodic promotion job multinode-1ctlr-featureset016 fail with error running docker 'gnocchi_db_sync' - rados.Rados.connect PermissionDeniedError: error connecting to the cluster" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 19:10 |
| *** etingof has quit IRC | 19:11 | |
| *** mmethot has quit IRC | 19:12 | |
| *** dpawar has quit IRC | 19:12 | |
| *** dpawar has joined #tripleo | 19:12 | |
| *** dtantsur is now known as dtantsur|afk | 19:14 | |
| *** pblaho has quit IRC | 19:15 | |
| *** pcaruana has joined #tripleo | 19:16 | |
| *** bogdando has quit IRC | 19:19 | |
| *** prad__ has joined #tripleo | 19:20 | |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-common master: Remove yum-axelget with overrides in tripleo_kolla_template_overrides https://review.openstack.org/523198 | 19:22 |
| *** pradk_ has quit IRC | 19:23 | |
| *** dpawar has quit IRC | 19:25 | |
| *** tosky has joined #tripleo | 19:27 | |
| *** moshele has quit IRC | 19:39 | |
| *** etingof has joined #tripleo | 19:41 | |
| *** newmember has joined #tripleo | 19:45 | |
| EmilienM | rlandy|rover: I might use your help | 19:45 |
| *** rbrady is now known as rbrady-afk | 19:45 | |
| EmilienM | rlandy|rover: when I'm using the network-environment.yaml that is generated by quickstart, it fails when I'm deploying with ipv6: http://logs.openstack.org/18/522618/6/check-tripleo/tripleo-ci-centos-7-ovb-ha-oooq-ipv6/ad1bf46/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2017-11-27_19_21_13 | 19:46 |
| EmilienM | the network environment is here: http://logs.openstack.org/18/522618/6/check-tripleo/tripleo-ci-centos-7-ovb-ha-oooq-ipv6/ad1bf46/logs/undercloud/home/zuul/network-environment.yaml.txt.gz | 19:46 |
| rlandy|rover | EmilienM: ok - looking | 19:46 |
| EmilienM | the external network is ipv4, I guess it's expected in our CI | 19:46 |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: Add role to setup any centos vm to look like nodepool vm https://review.openstack.org/517411 | 19:46 |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: Add various functionality to nodepool role https://review.openstack.org/518560 | 19:46 |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Create pre-generated fake subunit file https://review.openstack.org/522955 | 19:46 |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/newton: Allow for update after RHEL registration https://review.openstack.org/522033 | 19:46 |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/newton: RHEL/Update: replace wc by yum to check updates https://review.openstack.org/522396 | 19:46 |
| *** trown|lunch is now known as trown | 19:47 | |
| *** artom has quit IRC | 19:48 | |
| *** artom has joined #tripleo | 19:49 | |
| hamzy | should I be able to run tox locally in order to check new code that I am writing? I see a couple of errors: http://paste.openstack.org/show/627509/ | 19:50 |
| rlandy|rover | EmilienM: usually, yes, we expect ipv4 ... I guess that is why the file originally used was hard coded ... https://github.com/openstack-infra/tripleo-ci/blob/master/test-environments/ipv6-network-templates/network-environment.yaml | 19:50 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: Run ovb-ha with minimal services https://review.openstack.org/522310 | 19:50 |
| *** newmember has quit IRC | 19:51 | |
| EmilienM | rlandy|rover: oh I see :-) | 19:51 |
| rlandy|rover | EmilienM: do you want to use the values in https://github.com/openstack-infra/tripleo-ci/blob/master/test-environments/ipv6-network-templates/network-environment.yaml? | 19:51 |
| EmilienM | rlandy|rover: in a weird way, I found a limitation in quickstart | 19:51 |
| rlandy|rover | if so, you can just override the variables ... like | 19:51 |
| EmilienM | rlandy|rover: I guess I want that yeah | 19:52 |
| *** dsariel has joined #tripleo | 19:52 | |
| rlandy|rover | EmilienM: getting (I just need to remember where I did a similar thing) | 19:52 |
| EmilienM | rlandy|rover: ok I'll wait, thanks for your help! | 19:52 |
| *** jmelvin has quit IRC | 19:53 | |
| *** salmankhan has quit IRC | 19:54 | |
| rlandy|rover | EmilienM: see https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/overcloud-prep-config/defaults/main.yml#L14 os overridden by https://github.com/openstack/tripleo-quickstart-extras/blob/master/config/general_config/ovb-minimal-pacemaker-public-bond.yml#L22 ... you will also want the resource registry piece .. https://github.com/openstack/tripleo-quickstart-extras/blob/master/config/general_config/ovb-min | 19:54 |
| rlandy|rover | imal-pacemaker-public-bond.yml#L19 | 19:54 |
| jrist | eyeballs plz https://review.openstack.org/#/c/507969/ | 19:54 |
| *** amoralej is now known as amoralej|off | 19:55 | |
| *** atoth has quit IRC | 19:56 | |
| *** holser has joined #tripleo | 19:57 | |
| rlandy|rover | EmilienM: so you will build up the network-environment file in {{working_dir}} to match what you had hard-coded - if that makes sense. | 19:58 |
| rlandy|rover | https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/overcloud-deploy/tasks/create-scripts.yml#L46 ... https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/overcloud-deploy/templates/resource-registry-nic-configs.yaml.j2 | 19:59 |
| EmilienM | rlandy|rover: I'm not sure if it's the best thing to hardcode that in the featureset | 20:02 |
| EmilienM | rlandy|rover: but I'll give it a try | 20:02 |
| *** aputtur_ has quit IRC | 20:02 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: Introduce fs035, ovb-ha-ipv6 https://review.openstack.org/522615 | 20:03 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: Introduce fs035, ovb-ha-ipv6 https://review.openstack.org/522615 | 20:04 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: Create ovb-ha-ipv6 and deprecate fs024 job https://review.openstack.org/522618 | 20:04 |
| EmilienM | rlandy|rover: if you can review https://review.openstack.org/#/c/522615/6..7 please | 20:04 |
| rlandy|rover | EmilienM: I agree certain values should not be hardcoded ... | 20:04 |
| mwhahaha | rlandy|rover: https://bugs.launchpad.net/tripleo/+bug/1733672 is still a problem right? | 20:05 |
| openstack | Launchpad bug 1733672 in tripleo "tripleo-ci-centos-7-scenario001-multinode-oooq-container is failing on deploying the overcloud - master release only" [High,Triaged] - Assigned to Ronelle Landy (rlandy) | 20:05 |
| rlandy|rover | will comment there | 20:05 |
| EmilienM | rlandy|rover: and give feedback on the review | 20:05 |
| EmilienM | thanks | 20:05 |
| rlandy|rover | mwhahaha: unfortunately yes ... maybe Pradeep has more info | 20:06 |
| * rlandy|rover checks the test runs for the last few days ... | 20:07 | |
| mwhahaha | rlandy|rover: ok i'll see if i can find something. i'm bumping that one up to critical because it's failing pretty hard for several days now | 20:07 |
| *** morazi has quit IRC | 20:07 | |
| mwhahaha | rlandy|rover: something landed on 11/21 that broke it | 20:07 |
| mwhahaha | was probably https://review.openstack.org/#/c/521646/ | 20:08 |
| openstackgerrit | Michael Bayer proposed openstack/tripleo-quickstart master: Don't hardcode .quickstart since this is configurable https://review.openstack.org/523215 | 20:09 |
| mwhahaha | so something is returning 2 when it shouldn't be but only for scenario001 | 20:10 |
| *** ooolpbot has joined #tripleo | 20:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731063 | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1734134 | 20:10 |
| *** ooolpbot has quit IRC | 20:10 | |
| openstack | Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] | 20:10 |
| openstack | Launchpad bug 1734134 in tripleo "Pike periodic promotion job multinode-1ctlr-featureset016 fail with error running docker 'gnocchi_db_sync' - rados.Rados.connect PermissionDeniedError: error connecting to the cluster" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 20:10 |
| mwhahaha | this is why we don't non-vote jobs :/ | 20:10 |
| EmilienM | yup :( | 20:10 |
| rlandy|rover | mwhahaha: ack to moving that bug to critical - I marked it 'high' originally due to the non-voting status :( | 20:11 |
| mwhahaha | which was fine but we need to get scenario001 back voting and this seems to be a bigger problem | 20:12 |
| EmilienM | mwhahaha: I can't tell when the last time scenario001 was passing | 20:12 |
| mwhahaha | EmilienM: 11/21 | 20:12 |
| *** paramite has quit IRC | 20:12 | |
| mwhahaha | EmilienM: the failures point to a 2 being returned which would make https://review.openstack.org/#/c/521646/ the likely suspect | 20:12 |
| EmilienM | something failing at ControllerDeployment_Step4 | 20:13 |
| EmilienM | mwhahaha: indeed | 20:13 |
| EmilienM | mwhahaha: this one? http://logs.openstack.org/76/515576/6/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/8c4e0d7/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2017-11-27_18_39_07 | 20:13 |
| mwhahaha | yea | 20:13 |
| mwhahaha | so either docker-puppet or paunch is returning a 2 | 20:14 |
| EmilienM | okl et me see | 20:14 |
| mwhahaha | but i think bnemec looked at it last week because he was hitting it locally but i don't know if he figured it out | 20:14 |
| bnemec | So it turned out that ansible-playbook returns 2 on failure. | 20:14 |
| bnemec | Which is super fun. | 20:14 |
| mwhahaha | nice | 20:15 |
| mwhahaha | so that's probably what is happening here but i don't know if we capture those logs | 20:15 |
| bnemec | I had a legit error in my puppet run that was obscured by some bad error reporting. | 20:15 |
| bnemec | I tried to fix it in https://review.openstack.org/#/c/522405 but that's actually not working so I need to try again. | 20:15 |
| mwhahaha | so step4 in scenario001 probably has an ansible failure for some unknown reason | 20:15 |
| EmilienM | yeah I don't see anything in http://logs.openstack.org/76/515576/6/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/8c4e0d7/logs/subnode-2/var/log/journal.txt.gz | 20:16 |
| EmilienM | usually you see ansible logs | 20:16 |
| mwhahaha | does anyone know if paunch logs somewhere | 20:17 |
| mwhahaha | same for docker-puppet.py | 20:17 |
| *** prad__ has quit IRC | 20:18 | |
| EmilienM | dprince: ^ | 20:18 |
| EmilienM | I haven't found any logs with a failure | 20:21 |
| rlandy|rover | EmilienM: left a comment on https://review.openstack.org/#/c/522615/ | 20:22 |
| *** moshele has joined #tripleo | 20:22 | |
| EmilienM | rlandy|rover: ok, thanks. I'll see how that works first and then clean that up | 20:24 |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Remove tripleo-buildimage-overcloud-full-centos-7 from gate https://review.openstack.org/523146 | 20:24 |
| mwhahaha | ah ha | 20:24 |
| mwhahaha | postci.txt to the rescue | 20:24 |
| EmilienM | mwhahaha: found it | 20:25 |
| rlandy|rover | EmilienM: sure - let's see if that even gets you a clean deploy - then we can decide where to put the variables. I suspect we may need other changes | 20:25 |
| EmilienM | Unable to find image '192.168.24.1:8787/tripleomaster/centos-binary-ceilometer-ipmi:3b85715c6488a692fcdf4e4c6a589d77c60eb5a6_a3792c8d' locally\ | 20:25 |
| EmilienM | http://logs.openstack.org/76/515576/6/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/8c4e0d7/logs/subnode-2/var/log/journal.txt.gz#_Nov_27_18_38_56 | 20:25 |
| EmilienM | mwhahaha: ^ | 20:25 |
| *** jmelvin has joined #tripleo | 20:26 | |
| EmilienM | rlandy|rover: have you already deployed ssl + ipv6? | 20:26 |
| mwhahaha | EmilienM: hmm it looks like it found it tho | 20:26 |
| *** rbowen_ has joined #tripleo | 20:26 | |
| mwhahaha | EmilienM: that's what it always says when it first goes to pull it down | 20:26 |
| mwhahaha | EmilienM: go lok in postci | 20:27 |
| mwhahaha | EmilienM: it's failing on gnocchi_db_sync | 20:27 |
| *** aputtur_ has joined #tripleo | 20:27 | |
| *** rbowen_ has quit IRC | 20:27 | |
| mwhahaha | http://logs.openstack.org/31/522231/1/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/013cc53/logs/postci.txt.gz | 20:27 |
| mwhahaha | Error running ['docker', 'run', '--name', 'gnocchi_db_sync', '--label', 'config_id=tripleo_step4' | 20:27 |
| *** rbowen_ has joined #tripleo | 20:27 | |
| EmilienM | hey isn't the bug gfidente fixed? | 20:27 |
| mwhahaha | what bug | 20:27 |
| EmilienM | https://bugs.launchpad.net/tripleo/+bug/1709630 | 20:28 |
| openstack | Launchpad bug 1709630 in tripleo "docker run gnocchi_db_sync is failing in gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container" [Critical,Fix released] - Assigned to Giulio Fidente (gfidente) | 20:28 |
| *** rbowen has quit IRC | 20:29 | |
| EmilienM | nevermind, it's old | 20:29 |
| mwhahaha | well it's back :D | 20:29 |
| mwhahaha | it may have always been failing and we didn't see it until we fixed the |failed | 20:29 |
| rlandy|rover | EmilienM: no - I have not worked much with ipv6 - I think sshnaidm may have | 20:30 |
| mwhahaha | EmilienM: the rados.Rados.Connect thing i think is another bug | 20:31 |
| mwhahaha | http://logs.openstack.org/31/522231/1/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/013cc53/logs/subnode-2/var/log/containers/gnocchi/gnocchi-upgrade.log.txt.gz | 20:31 |
| EmilienM | oh!! | 20:31 |
| EmilienM | it's not RGW | 20:31 |
| *** yamahata has quit IRC | 20:32 | |
| EmilienM | I remember this bug now | 20:32 |
| EmilienM | I had it long time ago | 20:32 |
| EmilienM | it happens when Ceph isn't ready when you run the gnocchi command | 20:32 |
| EmilienM | it's an orchestration thing | 20:32 |
| mwhahaha | of course it is | 20:32 |
| EmilienM | nice catch | 20:32 |
| *** threestrands_ has joined #tripleo | 20:32 | |
| *** threestrands_ has quit IRC | 20:32 | |
| *** threestrands_ has joined #tripleo | 20:32 | |
| mwhahaha | so it's step4, shouldn't ceph be up? | 20:33 |
| EmilienM | let me check | 20:33 |
| *** threestrands_ has quit IRC | 20:33 | |
| *** threestrands_ has joined #tripleo | 20:34 | |
| *** threestrands has joined #tripleo | 20:34 | |
| *** threestrands has quit IRC | 20:34 | |
| *** threestrands has joined #tripleo | 20:34 | |
| EmilienM | ceph should be ready at step2 | 20:35 |
| EmilienM | iiuc | 20:35 |
| mwhahaha | i poked pradk since he's hiding | 20:35 |
| EmilienM | gfidente: ^ around? | 20:36 |
| mwhahaha | also postci.txt in yaml is sooooo much more readable | 20:36 |
| *** mcornea has quit IRC | 20:36 | |
| mwhahaha | https://bugs.launchpad.net/tripleo/+bug/1734134 | 20:37 |
| openstack | Launchpad bug 1734134 in tripleo "Pike periodic promotion job multinode-1ctlr-featureset016 fail with error running docker 'gnocchi_db_sync' - rados.Rados.connect PermissionDeniedError: error connecting to the cluster" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 20:37 |
| mwhahaha | is dupe | 20:37 |
| EmilienM | yeah i confirm WorkflowTasks_Step2 | 20:37 |
| EmilienM | so I wonder why gnocchi thinks ceph isn't ready | 20:37 |
| EmilienM | PermissionDeniedError: error connecting to the cluster | 20:37 |
| mwhahaha | it's a perm thing | 20:37 |
| EmilienM | maybe on the keyring | 20:37 |
| EmilienM | file is /etc/ceph/ceph.client.openstack.keyring | 20:38 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates master: Select first node as bootstrap node not using name https://review.openstack.org/513450 | 20:39 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates master: Select first node as bootstrap node not using name https://review.openstack.org/513450 | 20:41 |
| EmilienM | mwhahaha: even with https://review.openstack.org/#/c/508975/20/docker/services/gnocchi-api.yaml | 20:41 |
| EmilienM | http://logs.openstack.org/30/522630/1/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/cbae486/logs/subnode-2/var/log/containers/gnocchi/gnocchi-upgrade.log.txt.gz | 20:42 |
| EmilienM | the fix doesn't work | 20:42 |
| EmilienM | mwhahaha: maybe the keyring is good and it's within ceph | 20:42 |
| EmilienM | mwhahaha: where are ceph logs? | 20:43 |
| mwhahaha | probably missing | 20:43 |
| mwhahaha | that was a known issue i think | 20:43 |
| EmilienM | here: http://logs.openstack.org/30/522630/1/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/cbae486/logs/subnode-2/var/log/extra/docker/containers/ceph-mon-centos-7-rax-iad-0001039318/log/ceph/ | 20:43 |
| EmilienM | ouch | 20:43 |
| EmilienM | hard to find | 20:43 |
| mwhahaha | so alledgedly it's up ~15mins prior | 20:44 |
| EmilienM | 2017-11-23 19:45:00.358696 mon.centos-7-rax-iad-0001039318 mon.0 192.168.24.3:6789/0 81 : cluster [INF] Cluster is now healthy | 20:44 |
| *** threestrands_ has quit IRC | 20:44 | |
| EmilienM | and PermissionDeniedError: error connecting to the cluster happens 15min later yeah | 20:45 |
| mwhahaha | how do you specify the connection info for ceph cluster | 20:45 |
| EmilienM | mwhahaha: /etc/ceph/ceph.conf | 20:45 |
| mwhahaha | 2017-11-23 20:01:47,729 [12] DEBUG gnocchi.service: storage.coordination_url = **** | 20:45 |
| EmilienM | mwhahaha: it finds the ceph-mons | 20:45 |
| mwhahaha | oh | 20:45 |
| mwhahaha | hmm | 20:45 |
| EmilienM | and use /etc/ceph/ceph.client.openstack.keyring for auth | 20:45 |
| EmilienM | and use ceph_pool to list objects I guess | 20:45 |
| EmilienM | I know sileht wrote that code - sileht are you still up? | 20:46 |
| mwhahaha | http://logs.openstack.org/31/522231/1/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/013cc53/logs/subnode-2/var/log/config-data/gnocchi/etc/ | 20:46 |
| mwhahaha | no etc/ceph | 20:46 |
| EmilienM | ahah you nailed it I guess | 20:46 |
| EmilienM | unless gnocchi-upgrade is run outside of the container | 20:47 |
| mwhahaha | no it isn't | 20:47 |
| mwhahaha | Error running ['docker', 'run', '--name', 'gnocchi_db_sync', '--label', 'config_id=tripleo_step4' | 20:47 |
| EmilienM | I swear I saw /etc/ceph in the containers before | 20:48 |
| EmilienM | how does it work for nova cinder and glance then | 20:48 |
| mwhahaha | magic evidently | 20:49 |
| EmilienM | let me investigate | 20:49 |
| mwhahaha | we may just not be copying them | 20:49 |
| mwhahaha | https://github.com/openstack/tripleo-heat-templates/blob/master/docker/services/gnocchi-api.yaml#L96 | 20:49 |
| *** dhill_ has quit IRC | 20:50 | |
| *** morazi has joined #tripleo | 20:50 | |
| EmilienM | yeah exactly | 20:50 |
| *** pradk has joined #tripleo | 20:50 | |
| openstackgerrit | James Slagle proposed openstack/tripleo-common master: Add reproduce_command option https://review.openstack.org/522003 | 20:50 |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci master: Fix sub_nodes and sub_nodes_private with multiple nodes https://review.openstack.org/523226 | 20:55 |
| EmilienM | mwhahaha: I think I found why | 20:56 |
| EmilienM | we execute /usr/bin/bootstrap_host_exec gnocchi_api su gnocchi -s /bin/bash -c '/usr/bin/gnocchi-upgrade --sacks-number=SACK_NUM' | 20:56 |
| EmilienM | but I'm not sure we actually do that at step 4 | 20:56 |
| EmilienM | we might do that before all steps during bootstrap | 20:56 |
| EmilienM | err no | 20:57 |
| EmilienM | it's in the step4 block... nervermind | 20:57 |
| EmilienM | I'm deploying, so trying to reproduce | 20:57 |
| *** rbowen has joined #tripleo | 20:57 | |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: Add help links to /var/lib/mistral https://review.openstack.org/522231 | 20:58 |
| *** rbowen_ has quit IRC | 20:59 | |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates master: Don't assume single sub_node in nic config https://review.openstack.org/523227 | 21:03 |
| *** holser has quit IRC | 21:03 | |
| *** yamahata has joined #tripleo | 21:04 | |
| *** dsariel has quit IRC | 21:07 | |
| *** dhill_ has joined #tripleo | 21:09 | |
| *** ooolpbot has joined #tripleo | 21:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731063 | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1734134 | 21:10 |
| *** ooolpbot has quit IRC | 21:10 | |
| openstack | Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] | 21:10 |
| openstack | Launchpad bug 1734134 in tripleo "Pike periodic promotion job multinode-1ctlr-featureset016 fail with error running docker 'gnocchi_db_sync' - rados.Rados.connect PermissionDeniedError: error connecting to the cluster" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 21:10 |
| *** liverpooler has quit IRC | 21:11 | |
| *** gbarros has quit IRC | 21:12 | |
| *** gbarros has joined #tripleo | 21:15 | |
| gfidente | pradk EmilienM I was unable to reproduce using tripleo-master-ci | 21:17 |
| gfidente | it seems to be happening only in promotion | 21:17 |
| gfidente | but adarazs|ruck reproduced it | 21:17 |
| mwhahaha | gfidente: i pulled that from the ci status | 21:17 |
| EmilienM | it doesn't happen on promotions only | 21:17 |
| EmilienM | we pull it from the gate | 21:17 |
| mwhahaha | of course we haven't had a master promotion in 19 days | 21:18 |
| mwhahaha | so there is that | 21:18 |
| gfidente | ok well give me 20mins | 21:18 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: tripleo-buildimage: deploy repo in pre-run https://review.openstack.org/523167 | 21:19 |
| *** jtomasek has quit IRC | 21:19 | |
| EmilienM | dprince, Slower: I see undercloud-container passing now, on https://review.openstack.org/#/c/523168/ - should we make the job voting? | 21:21 |
| dprince | EmilienM: lets do it | 21:22 |
| *** rasca has quit IRC | 21:22 | |
| *** gbarros has quit IRC | 21:22 | |
| dprince | EmilienM: but, I think the job will go away once we finish Slower's work to use the containerized undercloud everywhere | 21:22 |
| dprince | but in the meantime lets do it | 21:22 |
| *** moshele has quit IRC | 21:22 | |
| EmilienM | dprince: you mean in the scenarios? I didn't know Slower was working on that | 21:23 |
| Slower | EmilienM: yeah.. lemme find the patches.. | 21:23 |
| dprince | EmilienM: no, you are doing the scenarios :) | 21:23 |
| EmilienM | ah | 21:23 |
| *** rhallisey has quit IRC | 21:23 | |
| *** gbarros has joined #tripleo | 21:24 | |
| dprince | EmilienM: Slower has been working with weshay on swapping in the undercloud for existing instack jobs | 21:24 |
| EmilienM | ok | 21:24 |
| EmilienM | Slower: I checked the undercloud-containers job and I don't see any sanity check running | 21:24 |
| Slower | https://review.openstack.org/#/c/517444/ https://review.openstack.org/#/c/517445/ and https://review.openstack.org/#/c/518118 | 21:24 |
| Slower | 518118 is the real test there.. depending on the other 2 | 21:25 |
| Slower | EmilienM: we have 3 issues from that, the TLS patch isn't functional: https://review.openstack.org/#/c/517079/ | 21:25 |
| EmilienM | oh ok, yeah the "quickstartization of undercloud-containers" :-) | 21:25 |
| Slower | EmilienM: we're not outputing undercloud_passwords.conf | 21:26 |
| EmilienM | ok | 21:26 |
| Slower | EmilienM: and some scenarios do a second run of the underclould to check for idempotency which fails | 21:26 |
| *** ebarrera has quit IRC | 21:26 | |
| Slower | EmilienM: dprince and I were going to have a chat about this in a bit.. | 21:27 |
| Slower | if you want to join us.. | 21:27 |
| *** hamzy has quit IRC | 21:27 | |
| EmilienM | Slower: I'm going to have lunch now, but I'll look updates on the patches | 21:28 |
| Slower | EmilienM: ok sounds good | 21:28 |
| EmilienM | Slower: it gives me time to finish what I started with OVB | 21:28 |
| EmilienM | Slower: I hope I can start playing with scenarios by next week or so | 21:29 |
| Slower | EmilienM: Alex wants to see this working by M2.. so time is tight.. fyi | 21:31 |
| EmilienM | Slower: we should stop sleeping | 21:33 |
| EmilienM | or make Alex sleeping | 21:33 |
| * mwhahaha shakes his fist and tells people to get back to work | 21:33 | |
| Slower | haha | 21:34 |
| *** shardy has quit IRC | 21:36 | |
| mwhahaha | slagle: https://review.openstack.org/#/c/522397 what's the impact of the skip identifier on that? does it completely break what they are relying on for that bugfix? | 21:37 |
| *** pcaruana has quit IRC | 21:41 | |
| *** ansmith has quit IRC | 21:42 | |
| *** hamzy has joined #tripleo | 21:44 | |
| gfidente | EmilienM mwhahaha pradk so gnocchi-upgrade starts at step4, I don't think it's a problem with the ordering | 21:44 |
| mwhahaha | yea we see ceph up 15mins prior | 21:44 |
| gfidente | do you guys have a live env where we can check the permissions? | 21:45 |
| mwhahaha | no | 21:45 |
| gfidente | one thing I don't get, if we're seeing it at gate, does it mean it's intermittent? | 21:45 |
| mwhahaha | emilien was working on reproducing it, but basically scenario001 is failing with it constantly | 21:45 |
| mwhahaha | gfidente: we switched scenario001 to non-voting so no one is paying attention | 21:45 |
| mwhahaha | gfidente: http://cistatus.tripleo.org/ reports it's been full of fail since 11-21 | 21:46 |
| mwhahaha | when we merged in the change that stopped ignoring 2 as a success | 21:46 |
| *** raildo has quit IRC | 21:47 | |
| mwhahaha | gfidente: it does appear that it was working on 11-20 http://logs.openstack.org/91/520091/1/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/d3e51cc/logs/subnode-2/var/log/containers/gnocchi/gnocchi-upgrade.log.txt.gz | 21:49 |
| mwhahaha | that failed for a different reason | 21:49 |
| EmilienM | yeah I'm deploying the CI job in my env, I can give SSH access when it fails | 21:49 |
| gfidente | so assuming it was introduced by https://review.openstack.org/#/c/522024 | 21:50 |
| gfidente | I wonder if the other services are working | 21:50 |
| gfidente | and why it passed CI anyway | 21:50 |
| mwhahaha | gfidente: it didn't, scenario001 was non-voting | 21:51 |
| gfidente | mwhahaha oh and that is the only one using ceph/gnocchi? | 21:51 |
| mwhahaha | http://logs.openstack.org/24/522024/1/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/5fad4b6/logs/subnode-2/var/log/containers/gnocchi/gnocchi-upgrade.log.txt.gz | 21:51 |
| mwhahaha | yea | 21:51 |
| mwhahaha | which is why i've been so irritated that no one is helping look at that vm connectivity thing | 21:51 |
| gfidente | right okay | 21:52 |
| mwhahaha | cause this is now a 2nd problem that we have to deal with | 21:52 |
| mwhahaha | cause non-voting | 21:52 |
| gfidente | so I guess we merge https://review.openstack.org/#/c/522535/ | 21:52 |
| gfidente | just to unblock | 21:52 |
| mwhahaha | wrong patch? | 21:52 |
| sshnaidm | any thoughts why this error appears in container-multinode? ERROR: The template version is invalid: "heat_template_version: queens". "heat_template_version" should be one of: 2013-05-23, 2014-10-16, 2015-04-30, 2015-10-15, 2016-04-08, 2016-10-14, 2017-02-24, 2017-09-01, newton, ocata, pike | 21:52 |
| sshnaidm | pingtest stack: https://logs.rdoproject.org/91/509391/2/openstack-check/gate-tripleo-ci-centos-7-containers-multinode-upgrades-master/Z3271cea6ed654fa3b61910cdf83eb2da/undercloud/home/jenkins/overcloud_validate.log.txt.gz#_2017-11-27_15_08_08 | 21:52 |
| mwhahaha | sshnaidm: you don't have queens heat | 21:52 |
| gfidente | yeah this https://review.openstack.org/#/c/522630/ | 21:52 |
| *** pchavva has quit IRC | 21:53 | |
| sshnaidm | and it's very strange.. | 21:53 |
| mwhahaha | gfidente: that's failing scenario001 as well. will need to check that | 21:53 |
| gfidente | yeah I am looking at the logs | 21:53 |
| mwhahaha | same error | 21:54 |
| mwhahaha | http://logs.openstack.org/30/522630/1/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/cbae486/logs/subnode-2/var/log/containers/gnocchi/gnocchi-upgrade.log.txt.gz | 21:54 |
| gfidente | mwhahaha it actually failed with same error | 21:55 |
| *** gbarros has quit IRC | 21:55 | |
| mwhahaha | sshnaidm: that's not installing rdo packages for teh undercloud? https://logs.rdoproject.org/91/509391/2/openstack-check/gate-tripleo-ci-centos-7-containers-multinode-upgrades-master/Z3271cea6ed654fa3b61910cdf83eb2da/undercloud/etc/yum.repos.d/ | 21:56 |
| *** trown is now known as trown|outtypewww | 21:56 | |
| *** apetrich has quit IRC | 21:57 | |
| sshnaidm | mwhahaha, looking... on subnode-2 it's pike.. | 21:57 |
| *** apetrich has joined #tripleo | 21:57 | |
| mwhahaha | sshnaidm: but you'd have to upgrade the undercloud first cause that's where the template processing occurs | 21:57 |
| mwhahaha | sshnaidm: so undercloud has to be queens | 21:57 |
| mwhahaha | sshnaidm: if it is queens, then it's a question for the heat folks as to if they support the queens template version yet | 21:58 |
| mwhahaha | sshnaidm: which if they don't then we need to revert our change to use it | 21:59 |
| mwhahaha | https://review.openstack.org/#/c/500451/ | 21:59 |
| sshnaidm | thrash|g0ne, | 22:00 |
| sshnaidm | oops, therve , are you around? | 22:00 |
| gfidente | EmilienM how are you adding gnocchi to the deployment? | 22:01 |
| gfidente | I mean, to reproduce the issue | 22:01 |
| mwhahaha | he's supposed to be reproducing scenario001 so i assume yes | 22:03 |
| *** lblanchard has quit IRC | 22:04 | |
| mwhahaha | sshnaidm: oh so i think i figured it out | 22:04 |
| mwhahaha | sshnaidm: we're running ping test from queens on a pike cloud prior to the upgrade | 22:05 |
| mwhahaha | sshnaidm: https://review.openstack.org/#/c/500451/11/ci/pingtests/tenantvm_floatingip.yaml | 22:06 |
| sshnaidm | mwhahaha, we run after: https://logs.rdoproject.org/91/509391/2/openstack-check/gate-tripleo-ci-centos-7-containers-multinode-upgrades-master/Z3271cea6ed654fa3b61910cdf83eb2da/console.txt.gz#_2017-11-27_15_07_44_069 | 22:06 |
| sshnaidm | mwhahaha, we run queens pingtest on pike overcloud | 22:06 |
| *** rbowen has quit IRC | 22:06 | |
| sshnaidm | mwhahaha, that's the problem, right? | 22:06 |
| mwhahaha | sshnaidm: rpm list indicates that subnode-2 is still on pike | 22:06 |
| sshnaidm | mwhahaha, yep, exactly.. | 22:07 |
| mwhahaha | the overcloud upgrade didn't run? | 22:07 |
| mwhahaha | sshnaidm: all it did was upload teh containers https://logs.rdoproject.org/91/509391/2/openstack-check/gate-tripleo-ci-centos-7-containers-multinode-upgrades-master/Z3271cea6ed654fa3b61910cdf83eb2da/undercloud/home/jenkins/overcloud-upgrade.sh | 22:08 |
| mwhahaha | sshnaidm: the upgrade was never run | 22:09 |
| *** ooolpbot has joined #tripleo | 22:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731063 | 22:10 |
| openstack | Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1734134 | 22:10 |
| *** ooolpbot has quit IRC | 22:10 | |
| openstack | Launchpad bug 1734134 in tripleo "Pike periodic promotion job multinode-1ctlr-featureset016 fail with error running docker 'gnocchi_db_sync' - rados.Rados.connect PermissionDeniedError: error connecting to the cluster" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 22:10 |
| sshnaidm | mwhahaha, isn't it enough: openstack overcloud container image prepare --images-file /home/jenkins/overcloud_upgrade_containers.yaml ..... ? | 22:10 |
| mwhahaha | sshnaidm: lol no | 22:10 |
| openstackgerrit | K Jonathan Harker proposed openstack/diskimage-builder master: zypper: fix package removal https://review.openstack.org/517658 | 22:11 |
| sshnaidm | need somebody from upgrades, what does happen with this job at all | 22:11 |
| EmilienM | gfidente: yes | 22:11 |
| EmilienM | gfidente: I can reproduce exactly a CI scenario | 22:11 |
| EmilienM | gfidente: I'm using traas | 22:12 |
| *** brian83 has quit IRC | 22:12 | |
| mwhahaha | sshnaidm: https://docs.openstack.org/tripleo-docs/latest/install/post_deployment/upgrade.html#upgrading-the-overcloud-to-ocata-or-pike | 22:12 |
| EmilienM | gfidente: it's still deploying now, let me check where is it | 22:12 |
| EmilienM | gfidente: step0, let's wait a bit more | 22:13 |
| sshnaidm | mwhahaha, weird, so this job never upgraded actually? | 22:14 |
| mwhahaha | sshnaidm: most likely, the upgrade jobs have been a WIP in for some time | 22:14 |
| sshnaidm | why do we run it then.. | 22:14 |
| mwhahaha | because it used to work for older releases | 22:15 |
| EmilienM | resources.WorkflowTasks_Step2_Execution: ERROR | 22:15 |
| EmilienM | ok let's see now | 22:15 |
| *** dprince has quit IRC | 22:16 | |
| mwhahaha | sshnaidm: file a bug and poke the upgrades team. seems like we're missing the actual upgrade steps | 22:16 |
| *** abishop has quit IRC | 22:16 | |
| EmilienM | ok so ceph is failing to bootstrap | 22:17 |
| EmilienM | in my env | 22:17 |
| EmilienM | http://paste.openstack.org/show/627514/ | 22:17 |
| EmilienM | gfidente, mwhahaha ^ | 22:18 |
| EmilienM | gfidente: I'll send tmate access in private | 22:18 |
| sshnaidm | mwhahaha, it's even 3 problems there.. no repos on undercloud, no upgrades, failing pingtest | 22:20 |
| mwhahaha | sshnaidm: well some how the undercloud did get queens packages so i have no idea without looking further at the job definition | 22:20 |
| *** d0ugal has quit IRC | 22:21 | |
| EmilienM | gfidente: I'll let you the server | 22:21 |
| EmilienM | gfidente: not sure where to start | 22:21 |
| openstackgerrit | zenghui.shi proposed openstack/tripleo-heat-templates master: Add PTP composable service https://review.openstack.org/491317 | 22:22 |
| openstackgerrit | Merged openstack/python-tripleoclient master: Output the location of the overcloud rc file https://review.openstack.org/521844 | 22:22 |
| sshnaidm | mwhahaha, that's how: https://logs.rdoproject.org/91/509391/2/openstack-check/gate-tripleo-ci-centos-7-containers-multinode-upgrades-master/Z3271cea6ed654fa3b61910cdf83eb2da/undercloud/home/jenkins/upgrade-undercloud-repo.sh.txt.gz | 22:23 |
| mwhahaha | ah it got removed before we captured the stuff | 22:25 |
| sshnaidm | mwhahaha, it doesn't upgrade undercloud, just removes the repos: https://logs.rdoproject.org/91/509391/2/openstack-check/gate-tripleo-ci-centos-7-containers-multinode-upgrades-master/Z3271cea6ed654fa3b61910cdf83eb2da/console.txt.gz#_2017-11-27_15_02_02_819 | 22:27 |
| sshnaidm | that's this awful old upgrades role.. | 22:27 |
| mwhahaha | sshnaidm: yea right it was already on queens so it didn't really need to do both the undercloud upgrade and the overcloud upgrade | 22:27 |
| mwhahaha | sshnaidm: we support installing N-1 from the current version so i assume that's what is used | 22:27 |
| openstackgerrit | James Slagle proposed openstack/tripleo-common master: Handle ansible-playbook failure for config-download https://review.openstack.org/523237 | 22:28 |
| *** dhill_ has quit IRC | 22:28 | |
| sshnaidm | mwhahaha, so what should it to upgrade? overcloud from pike to master? | 22:29 |
| mwhahaha | sshnaidm: yea | 22:29 |
| sshnaidm | mwhahaha, and it started to fail because we changes heat version in pingtest templates to "queens", right? | 22:30 |
| mwhahaha | sshnaidm: yea | 22:30 |
| mwhahaha | sshnaidm: which uncovered the fact that we're not actually upgrading anything | 22:30 |
| sshnaidm | mwhahaha, yep | 22:31 |
| *** dhill_ has joined #tripleo | 22:33 | |
| sshnaidm | fyi https://bugs.launchpad.net/tripleo/+bug/1734778 | 22:34 |
| openstack | Launchpad bug 1734778 in tripleo "CI: Container multinode upgrades job does not upgrade overcloud from pike to queens/master" [Critical,Triaged] | 22:34 |
| *** rcernin has joined #tripleo | 22:34 | |
| sshnaidm | matbu, jrist please FYI ^^ | 22:34 |
| EmilienM | rlandy|rover: 2017-11-27 21:32:12 | Could not fetch contents for file:///home/zuul/nic-configs/compute.yaml | 22:35 |
| EmilienM | rlandy|rover: re: ipv6 thing http://logs.openstack.org/18/522618/7/check-tripleo/tripleo-ci-centos-7-ovb-ha-oooq-ipv6/9cdc15f/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2017-11-27_21_32_12 | 22:35 |
| EmilienM | rlandy|rover: something with use_resource_registry_nic_configs | 22:36 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates stable/pike: Set the default CephAnsiblePlaybook to use into the env files https://review.openstack.org/523240 | 22:39 |
| EmilienM | rlandy|rover: use_resource_registry_nic_configs is not documented anywhere | 22:40 |
| rlandy|rover | EmilienM: I need to check if copied the custom nic-configs in this job | 22:40 |
| rlandy|rover | EmilienM: my fault - I should have doc'ed that | 22:40 |
| EmilienM | rlandy|rover: do I really need use_resource_registry_nic_configs now? | 22:40 |
| rlandy|rover | only f you need nice configs that are in your local dir | 22:41 |
| rlandy|rover | looking at the ipv6 original file referenced nic-configs | 22:41 |
| *** dparkes has joined #tripleo | 22:42 | |
| mwhahaha | slagle: would you mind lifting your -1 workflow on https://review.openstack.org/#/c/454341/ ? | 22:45 |
| EmilienM | sshnaidm: you might want to ping jistr, not jrist :-) | 22:45 |
| jrist | EmilienM: already pointed him that way | 22:45 |
| *** florianf has quit IRC | 22:47 | |
| *** jmelvin has quit IRC | 22:48 | |
| slagle | mwhahaha: ok. but that doesn't work | 22:49 |
| *** tvignaud has quit IRC | 22:49 | |
| mwhahaha | slagle: which part? the json bit in ocata? | 22:49 |
| *** tvignaud has joined #tripleo | 22:50 | |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-heat-templates stable/ocata: Use the local collector to bootstrap deployed servers https://review.openstack.org/454341 | 22:51 |
| *** salmankhan has joined #tripleo | 22:51 | |
| *** ansmith has joined #tripleo | 22:51 | |
| slagle | mwhahaha: see my comment on the patch | 22:51 |
| openstackgerrit | Merged openstack/instack-undercloud master: Updated from global requirements https://review.openstack.org/520409 | 22:52 |
| openstackgerrit | Merged openstack/puppet-tripleo master: Set disable:true for oci-machine-register https://review.openstack.org/522388 | 22:52 |
| openstackgerrit | Merged openstack/python-tripleoclient stable/pike: Mock creation of ~/.tripleo directory https://review.openstack.org/522188 | 22:52 |
| slagle | mwhahaha: you will probably need to rewrite /etc/os-collect-config.conf completely | 22:52 |
| mwhahaha | slagle: so how did we handle it for queens? | 22:52 |
| EmilienM | rlandy|rover: I'm confused by your comment, what should I do? | 22:52 |
| *** hamzy has quit IRC | 22:55 | |
| slagle | mwhahaha: there's nothing to handle | 22:56 |
| EmilienM | rlandy|rover: I'm going to try ipv6 single-nic-vlans for now | 22:57 |
| EmilienM | rlandy|rover: and make it more complex in the future, I'll need your help | 22:57 |
| slagle | mwhahaha: the issue is that you just can't switch to use the local collector, as the previous iteration of the script disabled using the local collector | 22:57 |
| slagle | mwhahaha: when the switch was made, it was not done in an upgrade safe way, for whatever reason | 22:58 |
| mwhahaha | slagle: oh you mean if you ran an update on an existing cloud you can't then go back and switch it | 22:58 |
| slagle | hence you can't just backport if the previous release you're backporing *to* used the previous iteration, and expect the upgrade job to work | 22:58 |
| slagle | not without more work | 22:58 |
| slagle | which is why i abandoned it, since no one seemed to care about upgrades jobs at the time | 22:59 |
| *** ebarrera has joined #tripleo | 22:59 | |
| mwhahaha | (or currently evidently) | 22:59 |
| openstackgerrit | Attila Darazs proposed openstack/tripleo-quickstart-extras master: GATE CHECK for quickstart-extras https://review.openstack.org/472607 | 23:00 |
| slagle | yea, well | 23:00 |
| slagle | i abandoned it in more ways than one | 23:00 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: Introduce fs035, ovb-ha-ipv6 https://review.openstack.org/522615 | 23:00 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: Create ovb-ha-ipv6 and deprecate fs024 job https://review.openstack.org/522618 | 23:01 |
| *** ebarrera has quit IRC | 23:03 | |
| mwhahaha | i guess i'll just backport the part i actually need | 23:04 |
| mwhahaha | which is the systemctl enable | 23:04 |
| *** ipsecguy has quit IRC | 23:04 | |
| *** hamzy has joined #tripleo | 23:08 | |
| *** tzumainn has quit IRC | 23:08 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Undercloud: Add router for IPv6 ctlplane subnet https://review.openstack.org/516992 | 23:08 |
| *** ooolpbot has joined #tripleo | 23:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731063 | 23:10 |
| openstack | Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1734134 | 23:10 |
| *** ooolpbot has quit IRC | 23:10 | |
| openstack | Launchpad bug 1734134 in tripleo "Pike periodic promotion job multinode-1ctlr-featureset016 fail with error running docker 'gnocchi_db_sync' - rados.Rados.connect PermissionDeniedError: error connecting to the cluster" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 23:10 |
| *** salmankhan has quit IRC | 23:12 | |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-heat-templates stable/ocata: Enable os-collect-config service https://review.openstack.org/523245 | 23:14 |
| *** gfidente has quit IRC | 23:16 | |
| *** bfournie has quit IRC | 23:26 | |
| *** pradk has quit IRC | 23:29 | |
| openstackgerrit | Ian Main proposed openstack/python-tripleoclient master: Mount a tmpfs filesystem for heat tmpfiles https://review.openstack.org/508558 | 23:30 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Set the default CephAnsiblePlaybook to use into the env files https://review.openstack.org/522535 | 23:49 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Mount /var/cache/swift across containers https://review.openstack.org/522715 | 23:49 |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: Restore tags inside the main playbook https://review.openstack.org/473491 | 23:51 |
| *** ipsecguy has joined #tripleo | 23:56 | |
| *** tosky has quit IRC | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!