| *** bfournie has joined #tripleo | 00:03 | |
| *** limao has joined #tripleo | 00:03 | |
| *** bfournie has quit IRC | 00:04 | |
| *** bfournie has joined #tripleo | 00:04 | |
| *** itlinux has quit IRC | 00:06 | |
| *** ooolpbot has joined #tripleo | 00:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 00:10 |
|---|---|---|
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709066 | 00:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709327 | 00:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709630 | 00:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710013 | 00:10 |
| openstack | Launchpad bug 1709066 in tripleo "[CI] Upgrade job down: AnsibleUndefinedVariable: 'docker_registry_namespace' is undefined" [Critical,In progress] - Assigned to Jiří Stránský (jistr) | 00:10 |
| *** ooolpbot has quit IRC | 00:10 | |
| openstack | Launchpad bug 1709327 in tripleo "CI: extremely long times of overcloud deploy in multinode jobs" [Critical,Triaged] | 00:10 |
| openstack | Launchpad bug 1709630 in tripleo "docker run gnocchi_db_sync is failing in gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 00:10 |
| openstack | Launchpad bug 1710013 in tripleo "KeyError: 'bmc_prefix' ovb version in oooq is no longer working" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 00:10 |
| *** jlabarre has quit IRC | 00:12 | |
| *** jlabarre has joined #tripleo | 00:15 | |
| *** k2so is now known as sdake | 00:16 | |
| *** funzo has joined #tripleo | 00:17 | |
| pabelanger | incoming 2 merges | 00:18 |
| *** dparkes has quit IRC | 00:18 | |
| *** ecerquei__ has joined #tripleo | 00:18 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Stop trying to build networking-bagpipe with DLRN https://review.openstack.org/492339 | 00:20 |
| openstackgerrit | Merged openstack/tripleo-quickstart master: Disable existing repositories in tripleo-ci https://review.openstack.org/492289 | 00:21 |
| pabelanger | nice 2 down | 00:22 |
| pabelanger | 1 left | 00:22 |
| *** funzo has quit IRC | 00:22 | |
| *** gvrangan has quit IRC | 00:35 | |
| *** psahoo has joined #tripleo | 00:39 | |
| *** psahoo has quit IRC | 01:04 | |
| *** itlinux has joined #tripleo | 01:04 | |
| *** gvrangan has joined #tripleo | 01:08 | |
| *** ooolpbot has joined #tripleo | 01:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709066 | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709327 | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709630 | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710013 | 01:10 |
| openstack | Launchpad bug 1709066 in tripleo "[CI] Upgrade job down: AnsibleUndefinedVariable: 'docker_registry_namespace' is undefined" [Critical,In progress] - Assigned to Jiří Stránský (jistr) | 01:10 |
| *** ooolpbot has quit IRC | 01:10 | |
| openstack | Launchpad bug 1709327 in tripleo "CI: extremely long times of overcloud deploy in multinode jobs" [Critical,Triaged] | 01:10 |
| openstack | Launchpad bug 1709630 in tripleo "docker run gnocchi_db_sync is failing in gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 01:10 |
| openstack | Launchpad bug 1710013 in tripleo "KeyError: 'bmc_prefix' ovb version in oooq is no longer working" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 01:10 |
| itlinux | what's the list of valid recourses ... is not a valid keyword inside a resource definition', | 01:16 |
| *** psahoo has joined #tripleo | 01:16 | |
| itlinux | I need to copy a cert for my ldap over the servers.. | 01:16 |
| itlinux | thanks! | 01:16 |
| *** funzo has joined #tripleo | 01:18 | |
| *** chem has quit IRC | 01:21 | |
| *** funzo has quit IRC | 01:23 | |
| *** jpena|off is now known as jpena | 01:30 | |
| *** yamahata has quit IRC | 01:31 | |
| *** jpena is now known as jpena|off | 01:38 | |
| *** atheurer has quit IRC | 01:41 | |
| *** ramishra has joined #tripleo | 01:53 | |
| *** gvrangan has quit IRC | 01:55 | |
| *** atheurer has joined #tripleo | 01:56 | |
| *** eck` is now known as eck`gone | 02:09 | |
| *** ooolpbot has joined #tripleo | 02:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709066 | 02:10 |
| openstack | Launchpad bug 1709066 in tripleo "[CI] Upgrade job down: AnsibleUndefinedVariable: 'docker_registry_namespace' is undefined" [Critical,In progress] - Assigned to Jiří Stránský (jistr) | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709327 | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709630 | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710013 | 02:10 |
| *** ooolpbot has quit IRC | 02:10 | |
| openstack | Launchpad bug 1709327 in tripleo "CI: extremely long times of overcloud deploy in multinode jobs" [Critical,Triaged] | 02:10 |
| openstack | Launchpad bug 1709630 in tripleo "docker run gnocchi_db_sync is failing in gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 02:10 |
| openstack | Launchpad bug 1710013 in tripleo "KeyError: 'bmc_prefix' ovb version in oooq is no longer working" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 02:10 |
| *** ecerquei_ has joined #tripleo | 02:11 | |
| *** cinerama` has joined #tripleo | 02:13 | |
| *** dougbtv_ has joined #tripleo | 02:15 | |
| *** cinerama has quit IRC | 02:18 | |
| *** ecerquei__ has quit IRC | 02:18 | |
| *** dougbtv has quit IRC | 02:18 | |
| *** jistr has quit IRC | 02:18 | |
| *** dciabrin has quit IRC | 02:18 | |
| *** ramishra has quit IRC | 02:18 | |
| *** radez has quit IRC | 02:18 | |
| *** dsariel has quit IRC | 02:18 | |
| *** funzo has joined #tripleo | 02:19 | |
| *** zshi_laptop has joined #tripleo | 02:19 | |
| *** social_ has joined #tripleo | 02:20 | |
| *** openstackgerrit has quit IRC | 02:21 | |
| *** lifeless_ has joined #tripleo | 02:21 | |
| *** bandini has quit IRC | 02:22 | |
| *** jistr has joined #tripleo | 02:23 | |
| *** tzumainn has joined #tripleo | 02:23 | |
| *** bandini has joined #tripleo | 02:23 | |
| *** funzo has quit IRC | 02:24 | |
| *** radez has joined #tripleo | 02:25 | |
| *** gbarros has quit IRC | 02:26 | |
| *** ramishra has joined #tripleo | 02:27 | |
| *** atheurer has quit IRC | 02:28 | |
| *** lifeless has quit IRC | 02:28 | |
| *** social has quit IRC | 02:28 | |
| *** yamahata has joined #tripleo | 02:28 | |
| *** dciabrin has joined #tripleo | 02:29 | |
| *** bandini has quit IRC | 02:31 | |
| *** openstackgerrit has joined #tripleo | 02:31 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/instack-undercloud master: Updated from global requirements https://review.openstack.org/492768 | 02:31 |
| *** gbarros has joined #tripleo | 02:31 | |
| *** bandini has joined #tripleo | 02:33 | |
| *** ykarel_ has joined #tripleo | 02:42 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/tripleo-common master: Updated from global requirements https://review.openstack.org/487398 | 02:42 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/tripleo-heat-templates master: Updated from global requirements https://review.openstack.org/488148 | 02:42 |
| *** ykarel__ has joined #tripleo | 02:51 | |
| *** ykarel_ has quit IRC | 02:55 | |
| *** shreshtha_ has quit IRC | 02:57 | |
| *** rbrady has quit IRC | 03:02 | |
| *** limao has quit IRC | 03:05 | |
| *** ooolpbot has joined #tripleo | 03:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709066 | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709327 | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709630 | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710013 | 03:10 |
| openstack | Launchpad bug 1709066 in tripleo "[CI] Upgrade job down: AnsibleUndefinedVariable: 'docker_registry_namespace' is undefined" [Critical,In progress] - Assigned to Jiří Stránský (jistr) | 03:10 |
| *** ooolpbot has quit IRC | 03:10 | |
| openstack | Launchpad bug 1709327 in tripleo "CI: extremely long times of overcloud deploy in multinode jobs" [Critical,Triaged] | 03:10 |
| openstack | Launchpad bug 1709630 in tripleo "docker run gnocchi_db_sync is failing in gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 03:10 |
| openstack | Launchpad bug 1710013 in tripleo "KeyError: 'bmc_prefix' ovb version in oooq is no longer working" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 03:10 |
| *** ykarel__ has quit IRC | 03:17 | |
| openstackgerrit | Alex Schultz proposed openstack-infra/tripleo-ci master: Switch to use local buildlogs mirror https://review.openstack.org/492786 | 03:18 |
| mwhahaha | pabelanger: fyi, i found the spot in tripleo-ci -^ | 03:18 |
| *** ykarel__ has joined #tripleo | 03:20 | |
| *** funzo has joined #tripleo | 03:21 | |
| *** tzumainn has quit IRC | 03:22 | |
| *** tzumainn has joined #tripleo | 03:22 | |
| itlinux | hi guyes I enabled LDAP and created a group and a project I know I am on that group.. but I get this error You are not authorized for any projects or domains. | 03:24 |
| itlinux | when I try to login | 03:24 |
| itlinux | any tips.. | 03:24 |
| mwhahaha | itlinux: you could try this thread, http://lists.openstack.org/pipermail/openstack/2016-August/017194.html | 03:26 |
| *** funzo has quit IRC | 03:26 | |
| itlinux | thanks man! | 03:26 |
| *** gvrangan has joined #tripleo | 03:26 | |
| *** kinton has joined #tripleo | 03:29 | |
| *** ykarel__ has quit IRC | 03:35 | |
| *** limao has joined #tripleo | 03:36 | |
| *** gvrangan has quit IRC | 03:40 | |
| *** udesale has joined #tripleo | 03:41 | |
| *** psachin has joined #tripleo | 03:41 | |
| *** limao has quit IRC | 03:42 | |
| kinton | hello all | 03:42 |
| *** psachin has quit IRC | 03:43 | |
| *** itlinux_ has joined #tripleo | 03:43 | |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-quickstart-extras master: Make container varaibles defined for upgrades https://review.openstack.org/491463 | 03:47 |
| *** psachin has joined #tripleo | 03:51 | |
| *** gbarros has quit IRC | 03:53 | |
| pabelanger | mwhahaha: nice, will look in the morning | 03:58 |
| *** ykarel__ has joined #tripleo | 04:03 | |
| *** itlinux_ has quit IRC | 04:03 | |
| jaosorior | mwhahaha: you around? can you remove the -W from my patches? | 04:04 |
| mwhahaha | jaosorior: i am looking at them | 04:04 |
| jaosorior | mwhahaha: thanks | 04:04 |
| mwhahaha | i've removed the ones that fit our rc1 criteria | 04:04 |
| mwhahaha | ie have bugs | 04:04 |
| mwhahaha | i'm going to send a note | 04:05 |
| jaosorior | mwhahaha: thought I had an FFE for the tls everywhere in containers work | 04:05 |
| mwhahaha | then you should reference a ffe bp in the commit message :)( | 04:05 |
| mwhahaha | the lack of attribution to bp or bug makes it hard to know if we should be allowing it | 04:06 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo master: Modify resource dependencies of certmonger_user resources https://review.openstack.org/490791 | 04:06 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Move HAProxy's public TLS logic from controller to service template https://review.openstack.org/491446 | 04:07 |
| jaosorior | mwhahaha: well, I surely didn't think it would take a week for some of them to merge :) | 04:08 |
| mwhahaha | yea me neither :D | 04:08 |
| mwhahaha | but we do need to be better about this | 04:08 |
| mwhahaha | too many patches w/o proper references to why we need them | 04:08 |
| *** ooolpbot has joined #tripleo | 04:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709066 | 04:10 |
| openstack | Launchpad bug 1709066 in tripleo "[CI] Upgrade job down: AnsibleUndefinedVariable: 'docker_registry_namespace' is undefined" [Critical,In progress] - Assigned to Alex Schultz (alex-schultz) | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709327 | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709630 | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710013 | 04:10 |
| *** ooolpbot has quit IRC | 04:10 | |
| openstack | Launchpad bug 1709327 in tripleo "CI: extremely long times of overcloud deploy in multinode jobs" [Critical,Triaged] | 04:10 |
| openstack | Launchpad bug 1709630 in tripleo "docker run gnocchi_db_sync is failing in gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 04:10 |
| openstack | Launchpad bug 1710013 in tripleo "KeyError: 'bmc_prefix' ovb version in oooq is no longer working" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 04:10 |
| jaosorior | mwhahaha: if I merely changed the commit message, should I add the +A again? | 04:10 |
| mwhahaha | yea | 04:11 |
| jaosorior | mwhahaha: how's your containerized undercloud debugging skills? | 04:13 |
| mwhahaha | questionable :D | 04:13 |
| mwhahaha | what's up? | 04:13 |
| jaosorior | mwhahaha: any idea what went wrong here http://logs.openstack.org/85/489885/6/check/gate-tripleo-ci-centos-7-undercloud-containers/f0e5436/ ? It's the patch that tries to get nova to run over httpd again in containers (it already is running over it in the non-containerized deployment) | 04:14 |
| *** itlinux_ has joined #tripleo | 04:16 | |
| mwhahaha | jaosorior: so it looks like 8774 isn't up | 04:16 |
| jaosorior | mwhahaha: oh wait, I found something in the journal-text.txt log http://logs.openstack.org/85/489885/6/check/gate-tripleo-ci-centos-7-undercloud-containers/f0e5436/logs/var/log/journal-text.txt.gz | 04:17 |
| jaosorior | mwhahaha: seems to be errors applying the apache manifests | 04:17 |
| *** hamzy has joined #tripleo | 04:18 | |
| jaosorior | mwhahaha: any idea what sets these parameters for the undercloud https://github.com/openstack/tripleo-heat-templates/blob/master/docker/services/nova-api.yaml#L7 ? They seem to work for the overcloud :/ | 04:18 |
| jaosorior | mwhahaha: which was this commit https://review.openstack.org/#/c/490385/ | 04:19 |
| mwhahaha | yea not sure | 04:19 |
| *** itlinux_ has quit IRC | 04:19 | |
| *** itlinux_ has joined #tripleo | 04:19 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Make containerized nova-api run with httpd https://review.openstack.org/475366 | 04:20 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Enable TLS for nova api and placement containers https://review.openstack.org/489885 | 04:20 |
| jaosorior | mwhahaha: uhm...I have the feeling that I need promotion to happen to get that working | 04:22 |
| *** funzo has joined #tripleo | 04:22 | |
| jaosorior | might be that it gets that environment from a package or something of the sort | 04:22 |
| mwhahaha | it's possible | 04:22 |
| jaosorior | gonna hit denver? | 04:22 |
| *** itlinux_ has quit IRC | 04:23 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo master: HAProxy: Set listen options for internal services too https://review.openstack.org/491832 | 04:23 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo master: Create separate resource for HAProxy horizon endpoint https://review.openstack.org/491437 | 04:24 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo master: Enable TLS in the internal network for horizon https://review.openstack.org/489593 | 04:24 |
| *** shreshtha has joined #tripleo | 04:26 | |
| *** funzo has quit IRC | 04:26 | |
| *** shreshtha has quit IRC | 04:28 | |
| mwhahaha | jaosorior: i live in denver? :D | 04:28 |
| * mwhahaha hits it daily | 04:28 | |
| jaosorior | lol | 04:29 |
| jaosorior | mwhahaha: I actually didn't know | 04:29 |
| *** shreshtha has joined #tripleo | 04:29 | |
| jaosorior | just got my flight :D | 04:30 |
| mwhahaha | nice | 04:30 |
| *** dsariel has joined #tripleo | 04:31 | |
| *** ykarel__ has quit IRC | 04:38 | |
| * mwhahaha wanders off | 04:39 | |
| jaosorior | mwhahaha: have a good one | 04:39 |
| *** ykarel__ has joined #tripleo | 04:43 | |
| *** gvrangan has joined #tripleo | 04:45 | |
| *** dbecker has joined #tripleo | 04:46 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Make containerized nova-api run with httpd https://review.openstack.org/475366 | 04:59 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Enable TLS for nova api and placement containers https://review.openstack.org/489885 | 05:00 |
| *** jaganathan has joined #tripleo | 05:02 | |
| *** ratailor has joined #tripleo | 05:06 | |
| *** pdeore has joined #tripleo | 05:06 | |
| *** ooolpbot has joined #tripleo | 05:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709066 | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709327 | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709630 | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710013 | 05:10 |
| openstack | Launchpad bug 1709066 in tripleo "[CI] Upgrade job down: AnsibleUndefinedVariable: 'docker_registry_namespace' is undefined" [Critical,In progress] - Assigned to Alex Schultz (alex-schultz) | 05:10 |
| *** ooolpbot has quit IRC | 05:10 | |
| openstack | Launchpad bug 1709327 in tripleo "CI: extremely long times of overcloud deploy in multinode jobs" [Critical,Triaged] | 05:10 |
| openstack | Launchpad bug 1709630 in tripleo "docker run gnocchi_db_sync is failing in gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 05:10 |
| openstack | Launchpad bug 1710013 in tripleo "KeyError: 'bmc_prefix' ovb version in oooq is no longer working" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 05:10 |
| *** jfrancoa has joined #tripleo | 05:19 | |
| *** dpawar has joined #tripleo | 05:21 | |
| *** dbecker has quit IRC | 05:21 | |
| *** funzo has joined #tripleo | 05:23 | |
| *** funzo has quit IRC | 05:23 | |
| *** funzo has joined #tripleo | 05:23 | |
| *** limao has joined #tripleo | 05:24 | |
| *** marios has joined #tripleo | 05:27 | |
| *** funzo has quit IRC | 05:27 | |
| *** yprokule has joined #tripleo | 05:32 | |
| *** garyk has joined #tripleo | 05:35 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Enable TLS configuration for containerized Galera https://review.openstack.org/489963 | 05:43 |
| *** ykarel__ has quit IRC | 05:43 | |
| *** ykarel has joined #tripleo | 05:48 | |
| *** nyechiel has joined #tripleo | 05:53 | |
| *** itlinux has quit IRC | 05:55 | |
| *** itlinux has joined #tripleo | 05:57 | |
| *** gfidente has joined #tripleo | 06:05 | |
| *** ooolpbot has joined #tripleo | 06:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709066 | 06:10 |
| openstack | Launchpad bug 1709066 in tripleo "[CI] Upgrade job down: AnsibleUndefinedVariable: 'docker_registry_namespace' is undefined" [Critical,In progress] - Assigned to Alex Schultz (alex-schultz) | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709327 | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709630 | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710013 | 06:10 |
| *** ooolpbot has quit IRC | 06:10 | |
| openstack | Launchpad bug 1709327 in tripleo "CI: extremely long times of overcloud deploy in multinode jobs" [Critical,Triaged] | 06:10 |
| openstack | Launchpad bug 1709630 in tripleo "docker run gnocchi_db_sync is failing in gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 06:10 |
| openstack | Launchpad bug 1710013 in tripleo "KeyError: 'bmc_prefix' ovb version in oooq is no longer working" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 06:10 |
| *** lvdombrkr has joined #tripleo | 06:15 | |
| *** janki has joined #tripleo | 06:16 | |
| *** tesseract has joined #tripleo | 06:16 | |
| *** masco has joined #tripleo | 06:19 | |
| *** skramaja has joined #tripleo | 06:20 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [DNM] create intermediate LVMNode object, refactor slightly https://review.openstack.org/492812 | 06:20 |
| *** rcernin has joined #tripleo | 06:23 | |
| *** funzo has joined #tripleo | 06:23 | |
| *** ramishra has quit IRC | 06:26 | |
| jaosorior | dciabrin: let me know when you're around | 06:26 |
| jaosorior | or jistr :O | 06:26 |
| *** jlabarre has quit IRC | 06:27 | |
| *** jlabarre has joined #tripleo | 06:27 | |
| *** ramishra has joined #tripleo | 06:27 | |
| *** funzo has quit IRC | 06:28 | |
| *** itlinux has quit IRC | 06:28 | |
| *** jprovazn has joined #tripleo | 06:30 | |
| *** anshul has joined #tripleo | 06:34 | |
| *** nyechiel has quit IRC | 06:35 | |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates master: Correct gnocchi-upgrade command quotes https://review.openstack.org/491826 | 06:37 |
| dciabrin | jaosorior, hey o/ | 06:38 |
| jaosorior | gfidente: commented. | 06:39 |
| jaosorior | dciabrin: yooo, how's it going? | 06:39 |
| gfidente | jaosorior hey there, morning | 06:39 |
| dciabrin | jaosorior, well that's friday :) | 06:40 |
| jaosorior | gfidente: morning dude, how's life | 06:40 |
| gfidente | true the comment | 06:40 |
| gfidente | I believe we can use the flags which set the backend type | 06:40 |
| gfidente | to mount it optionally yes | 06:40 |
| jaosorior | gfidente: that would be great :D | 06:41 |
| gfidente | only I'd need to do that in all container | 06:41 |
| gfidente | :P | 06:41 |
| jaosorior | dciabrin: I'm gonna need some help with the uid/permissions issue :/ right now I'm doing nova, but I'll let you know when I switch back to doing haproxy | 06:41 |
| *** ykarel has quit IRC | 06:41 | |
| jaosorior | gfidente: oh crap :O | 06:42 |
| gfidente | jaosorior it's fine | 06:42 |
| gfidente | can I do that for all in a separate submission? | 06:42 |
| *** ykarel has joined #tripleo | 06:42 | |
| jaosorior | gfidente: sure! no biggie. But reference a bug or a blueprint | 06:42 |
| gfidente | yeah you want to open a bug? | 06:43 |
| gfidente | or I do it? | 06:43 |
| jaosorior | can you do it? | 06:43 |
| gfidente | yeahok | 06:43 |
| dciabrin | jaosorior, ack any time | 06:45 |
| jaosorior | dciabrin: also, when mysql is not running in HA, does it still read /etc/my.cnf.d/galera.cnf? | 06:47 |
| dciabrin | jaosorior, good question . i think it does due to it being compiled with galera support. let me double check | 06:48 |
| *** ramishra has quit IRC | 06:51 | |
| *** bogdando has joined #tripleo | 06:51 | |
| *** ramishra has joined #tripleo | 06:53 | |
| *** ccamacho has joined #tripleo | 06:53 | |
| *** cshastri has joined #tripleo | 06:56 | |
| *** ebarrera has joined #tripleo | 06:56 | |
| *** amoralej|off is now known as amoralej | 06:57 | |
| *** nyechiel has joined #tripleo | 06:57 | |
| *** jlinkes has joined #tripleo | 06:58 | |
| *** jaosorior has quit IRC | 07:03 | |
| *** pgadiya has joined #tripleo | 07:03 | |
| *** nyechiel has quit IRC | 07:07 | |
| *** ooolpbot has joined #tripleo | 07:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709066 | 07:10 |
| openstack | Launchpad bug 1709066 in tripleo "[CI] Upgrade job down: AnsibleUndefinedVariable: 'docker_registry_namespace' is undefined" [Critical,In progress] - Assigned to Alex Schultz (alex-schultz) | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709327 | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709630 | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710013 | 07:10 |
| *** ooolpbot has quit IRC | 07:10 | |
| openstack | Launchpad bug 1709327 in tripleo "CI: extremely long times of overcloud deploy in multinode jobs" [Critical,Triaged] | 07:10 |
| openstack | Launchpad bug 1709630 in tripleo "docker run gnocchi_db_sync is failing in gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 07:10 |
| openstack | Launchpad bug 1710013 in tripleo "KeyError: 'bmc_prefix' ovb version in oooq is no longer working" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 07:10 |
| *** jaosorior has joined #tripleo | 07:15 | |
| bogdando | folks, let's merge this please https://review.openstack.org/#/c/492463/ | 07:22 |
| bogdando | do we have elastic-recheck cores around? | 07:22 |
| jaosorior | jistr, bandini: Could you check this out https://review.openstack.org/#/c/489885/ ? | 07:22 |
| jaosorior | sorry, this one https://review.openstack.org/#/c/475366/ | 07:23 |
| *** Vijayendra has quit IRC | 07:23 | |
| *** funzo has joined #tripleo | 07:24 | |
| jaosorior | bogdando: ^^ | 07:24 |
| jaosorior | bogdando: I suggest you ask in #openstack-infra | 07:25 |
| bogdando | jaosorior: tls has FFE for pike, right jaosorior? | 07:26 |
| jaosorior | bogdando: yep | 07:26 |
| bogdando | jaosorior: I really like the easy way to switch things onto wsgi | 07:26 |
| bogdando | and after that, they say tripleo is complex? naah | 07:27 |
| jaosorior | bogdando: haha well, the magic is done by puppet mostly | 07:27 |
| *** funzo has quit IRC | 07:29 | |
| jaosorior | bogdando: the change is needed. that's what the undercloud-containers job uses | 07:29 |
| bogdando | yeah, but puppet has been around for decades, no one can say it's complex. so beat it! :D | 07:30 |
| jaosorior | bogdando: https://github.com/openstack-infra/tripleo-ci/blob/master/scripts/tripleo.sh#L622 | 07:30 |
| jaosorior | bogdando: changing that environment file was the only way I could get the undercloud-containers job passing | 07:31 |
| bogdando | jaosorior: I see | 07:31 |
| bogdando | ok then | 07:31 |
| bogdando | jaosorior: do we need an update to the containers prepare template of the tripleoclient? | 07:31 |
| bogdando | to s/nova-base/nova-api | 07:32 |
| bogdando | it seems like that we have to track changes in both places | 07:32 |
| bogdando | unless we removed the former | 07:32 |
| bogdando | https://review.openstack.org/#/c/487613/ | 07:32 |
| *** tzumainn has quit IRC | 07:36 | |
| jaosorior | bogdando: thought it was in tripleo-common. That change is done | 07:36 |
| bogdando | jaosorior: thanks for double check! | 07:36 |
| jaosorior | bogdando: check the Depends-On | 07:36 |
| bogdando | jaosorior: I missed the depends-on, sorry. It's yet early morning to me | 07:37 |
| jaosorior | why not +2? LD | 07:38 |
| jaosorior | :D | 07:38 |
| *** rwsu has joined #tripleo | 07:40 | |
| bogdando | jaosorior: I should +2 containers related changes only :) | 07:42 |
| bogdando | folks, PTAL https://review.openstack.org/#/c/492178/ | 07:42 |
| jaosorior | bogdando: that's only for containers | 07:42 |
| jaosorior | bogdando: nova is already running over httpd in non-containerized deployments | 07:42 |
| bogdando | I'm using this reworked script to get the jobs logs and analyze for errorish patterns. then check them against elastic recheck knowledge base, works like a charm! | 07:43 |
| bogdando | and here is a helper script https://github.com/bogdando/fuel-log-parse#examples-for-tripleo-ci-openstack-infra-logs explained, helps to get the patterns to analyse, from the downloaded logs. You should try it. This way we can fix the unclassified failures http://status.openstack.org/elastic-recheck/data/others.html | 07:47 |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder master: [DNM] create intermediate LVMNode object, refactor slightly https://review.openstack.org/492812 | 07:51 |
| *** ffiore has joined #tripleo | 07:52 | |
| bogdando | folks, where leaves the project-config repo for rdo distgits? I want to add a new distgit repo | 07:54 |
| bogdando | lives* | 07:54 |
| bogdando | is it https://review.rdoproject.org/r/#/admin/projects/config ? | 07:55 |
| *** chem has joined #tripleo | 07:55 | |
| *** daidv has joined #tripleo | 07:57 | |
| *** dsariel_ has joined #tripleo | 07:57 | |
| *** karthiks has joined #tripleo | 07:57 | |
| *** dsariel has quit IRC | 07:57 | |
| *** rwsu has quit IRC | 07:58 | |
| *** rwsu has joined #tripleo | 08:00 | |
| *** stendulker has joined #tripleo | 08:01 | |
| *** openstackgerrit has quit IRC | 08:02 | |
| *** paramite has joined #tripleo | 08:04 | |
| jaosorior | jistr: hey dude, you around? | 08:07 |
| jaosorior | we need your wisdom | 08:08 |
| jaosorior | n stuff | 08:08 |
| *** ooolpbot has joined #tripleo | 08:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709066 | 08:10 |
| openstack | Launchpad bug 1709066 in tripleo "[CI] Upgrade job down: AnsibleUndefinedVariable: 'docker_registry_namespace' is undefined" [Critical,In progress] - Assigned to Alex Schultz (alex-schultz) | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709327 | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709630 | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710013 | 08:10 |
| *** ooolpbot has quit IRC | 08:10 | |
| openstack | Launchpad bug 1709327 in tripleo "CI: extremely long times of overcloud deploy in multinode jobs" [Critical,Triaged] | 08:10 |
| openstack | Launchpad bug 1709630 in tripleo "docker run gnocchi_db_sync is failing in gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 08:10 |
| openstack | Launchpad bug 1710013 in tripleo "KeyError: 'bmc_prefix' ovb version in oooq is no longer working" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 08:10 |
| chem | marios: he, there is an opportunity to +1W https://review.openstack.org/#/c/483293/ if you have 10s | 08:10 |
| *** cylopez has joined #tripleo | 08:10 | |
| *** snecklifter has joined #tripleo | 08:11 | |
| *** mcornea has joined #tripleo | 08:13 | |
| *** derekh has joined #tripleo | 08:14 | |
| marios | chem: o/ yeah i noticed it was the next one in the chain | 08:16 |
| marios | chem: should we wait for shardy? i mean he didn't update it in few days wondering if there was something else or you already talked to him? | 08:17 |
| skramaja | mysql is failing duing init tasks on controller step2 - http://paste.openstack.org/show/618134/, its is a fresh deploy, any clue? | 08:19 |
| *** milan has joined #tripleo | 08:20 | |
| *** shardy has joined #tripleo | 08:21 | |
| *** funzo has joined #tripleo | 08:25 | |
| flaper87 | bogdando: ++ | 08:26 |
| flaper87 | thanks for working on the package | 08:26 |
| shardy | Morning all | 08:27 |
| *** openstackgerrit has joined #tripleo | 08:27 | |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-heat-templates master: Containerized swift-proxy upgrade fix [WIP] https://review.openstack.org/492869 | 08:27 |
| shardy | Anyone know the status re the gate? cistatus looks better today, should we update $topic? | 08:27 |
| jaosorior | shardy: I think mwhahaha sent a mail about it. | 08:28 |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-heat-templates master: Containerized swift upgrade fix [WIP] https://review.openstack.org/492869 | 08:29 |
| *** funzo has quit IRC | 08:29 | |
| *** dparkes has joined #tripleo | 08:29 | |
| shardy | jaosorior: aha, missed that, thanks | 08:33 |
| skramaja | shardy: mysql is failing duing init tasks on controller step2 - http://paste.openstack.org/show/618134/, there is no mysql container running. | 08:33 |
| shardy | jaosorior: were you and jistr discussing issues with mysql yesterday? I thought I saw something in the scrollback but didn't follow closely | 08:33 |
| jaosorior | shardy: which mysql issues? | 08:34 |
| shardy | skramaja is having issues which sound like a possible race, we're trying to do the mysql init tasks, but the mariadb container isn't running yet | 08:34 |
| skramaja | shardy: 9ac035007193 10.60.19.51:5000/rhosp12/openstack-mariadb-docker:latest "kolla_start" 21 hours ago Created mysql | 08:34 |
| *** jlabarre has quit IRC | 08:34 | |
| shardy | I just wnndered if anyone else has seen similar, I've not, but to be fair I've been working on composable networking this week not containers | 08:34 |
| jaosorior | shardy: I haven't seen something similar. dciabrin and me are debugging some issue where the /etc/my.cnf.d/tripleo.cnf file is not being used by the bootstrap nodes, so it's not ran in the db syncs | 08:35 |
| jaosorior | * bootstrap containers | 08:36 |
| shardy | jaosorior: ah, Ok that must be what I was thinking of | 08:36 |
| jaosorior | shardy: and it TLS everywhere environments it's an issue, because that sets the SSL flags for the connection (which is mandatory, else it fails) | 08:36 |
| jaosorior | so we're fixing that | 08:36 |
| dciabrin | shardy, hmm maybe there was a failure during the mysql bootstrap, normally the db_sync tasks are only run at step3, and mysql bootstraps at step2, so there shouldn't be any race here | 08:36 |
| jaosorior | or well, dciabrin fixed that mostly. I just tagged along :P | 08:36 |
| dciabrin | haha | 08:36 |
| *** jlabarre has joined #tripleo | 08:37 | |
| *** rwsu has quit IRC | 08:37 | |
| shardy | dciabrin: ack, Ok well skramaja pasted http://paste.openstack.org/show/618134/ - we may need more info to see if it's related, but the issue seems to be we try to do the docker-puppet initialization without validating the mariadb container is running | 08:39 |
| shardy | which I know is a general problem, but it makes the failure a bit less clear | 08:39 |
| shardy | skramaja: are you running custom roles, or just 3 controllers? | 08:39 |
| skramaja | shardy: 1 controller and 1 computeovsdpdk | 08:39 |
| dciabrin | skramaja, can you maybe apply that patch to enable puppet debug logs and redeploy? we'll have a full trace at what's going on for all docker-puppet steps | 08:42 |
| gfidente | jaosorior was reviewing the code | 08:42 |
| gfidente | for public tls against rgw | 08:42 |
| jistr | jaosorior: reviewed the patches. Looking at the failed upgrade job there too | 08:42 |
| skramaja | dciabrin: yes, which patch are you referring to? | 08:43 |
| jistr | we must have merged more upgrade breaking patches, it fails even earlier than before | 08:43 |
| gfidente | jaosorior assuming ceph-ansible binds rgw on the same backend socket, I doesn't look like we need changes in haproxy for tls, am I reading it correctly? | 08:43 |
| *** jtomasek has joined #tripleo | 08:44 | |
| dciabrin | skramaja, damned me... http://paste.openstack.org/show/618137/ . better if I paste it :) | 08:45 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Internal TLS support for mongodb container https://review.openstack.org/492878 | 08:47 |
| *** ykarel has quit IRC | 08:47 | |
| *** ykarel has joined #tripleo | 08:48 | |
| jaosorior | dciabrin: ^^ | 08:48 |
| skramaja | dciabrin: thanks. i will try it. | 08:49 |
| *** jtomasek has quit IRC | 08:49 | |
| *** lucas-afk is now known as lucasagomes | 08:49 | |
| *** rwsu has joined #tripleo | 08:49 | |
| jaosorior | jistr: seems to fail in the "create custom t-h-t script" step | 08:50 |
| jistr | yea | 08:50 |
| skramaja | dciabrin: shardy this is the actual mysql container failure log - http://paste.openstack.org/show/618140/, though the actual container does not have any logs and these logs are not much userful. i will try the patach. | 08:50 |
| jaosorior | jistr: 'deploy_composable_scenario' is undefined" seems like a quickstart issue | 08:51 |
| jistr | jaosorior: yea definitely | 08:51 |
| bogdando | jistr, oidgar: I confirm I can see that "Started Mistral Workflow tripleo.plan_management.v1.create_deployment_plan -> Error rendering template network/network.network.yaml : 'network' is undefined" error with the tripleo-common and tripleoclient installed from master (with pip). I'd love to know how to debug the root cause down... Does it come from my quickstart ansible inventory/t-h-t templates data perhaps? I have no idea how to debug | 08:52 |
| bogdando | this :/ | 08:52 |
| jistr | looking for recent changes... | 08:52 |
| bogdando | may be I have a bad mistral container image deployed UC | 08:52 |
| bogdando | whatever | 08:52 |
| jaosorior | jistr: seems that that variable should be defined in the "featureset" files | 08:53 |
| jaosorior | jistr: what featureset file is gate-tripleo-ci-centos-7-containers-multinode-upgrades-nv using ? | 08:53 |
| *** dtantsur|afk is now known as dtantsur | 08:54 | |
| shardy | bogdando: sounds like you're still running old mistral actions from tripleo-common - are you sure the mistral containers are latest? | 08:54 |
| jistr | jaosorior: yea i'll add https://github.com/openstack/tripleo-quickstart/commit/14e5779a53ff29cf4045c490a08b028535e2f62e for the rest of the scenario featuresets | 08:54 |
| jaosorior | jistr: also, another issue is that the variable is being used in roles/overcloud-prep-config/templates/overcloud-custom-tht-script.sh.j2 while the default is being defined in the overcloud-deploy role | 08:55 |
| jaosorior | jistr: I think the fix would be to move the variable default definition from overcloud-deploy, to common | 08:55 |
| bogdando | shardy: I've redeployed UC a few days ago | 08:56 |
| jistr | bogdando: oidgar succeeded with fixing that issue on his env yesterday by updating tripleo-common and refreshing using these commands https://github.com/openstack/tripleo-common#action-development | 08:56 |
| bogdando | jistr: thanks! I'll try that | 08:57 |
| *** jaganathan has quit IRC | 08:57 | |
| *** jaganathan has joined #tripleo | 08:58 | |
| jistr | bogdando: oh you're using *containerized* undercloud, sorry i need to focus more :) | 08:58 |
| bogdando | yeah, I'll try to adapt the commands... | 08:58 |
| shardy | bogdando: ack, well it sounds like it might be related to https://github.com/openstack/tripleo-common/commit/44a02bbed0ba57d9932355f69d57a88b39c38373 but it could be some other issue | 08:58 |
| shardy | we render the j2 files via process-templates for undercloud deploy though don't we? | 08:59 |
| shardy | so perhaps it's a different issue | 08:59 |
| *** salmankhan has joined #tripleo | 08:59 | |
| * shardy hasn't tested containerized undercloud for a while | 08:59 | |
| jistr | it's very likely that it is related -- missing the code that feeds the network variable to rendering the j2. And that code is in tripleo-common | 09:00 |
| jaosorior | shardy: by the way, this patch breaks TLS everywhere https://review.openstack.org/#/c/400407/ please lets not merge it :( | 09:00 |
| jistr | bogdando: so if latest containers don't have it, you may need to pull source and run the refresh within the mistral containers | 09:00 |
| bogdando | jistr: got it. I better off revert | 09:01 |
| shardy | jaosorior: ack, any ideas how we fix that? | 09:01 |
| shardy | where is the dependency on the metadata server, can we just use the config drive instead? | 09:01 |
| jaosorior | shardy: we would need to talk to rcrit when he logs in. But basically we use nova metadata to pass the relevant info about which service principals will be created (to request a certificate for those) and then pass in an OTP | 09:02 |
| jaosorior | so we're heavily dependent on the metadata service for now | 09:02 |
| shardy | jaosorior: understood, I just wondered where the code is - stevebaker already fixed a couple of places to make them just use the config drive instead | 09:03 |
| jaosorior | shardy: which plces? | 09:03 |
| *** dsariel_ has quit IRC | 09:04 | |
| shardy | jaosorior: https://review.openstack.org/#/c/487230/1/firstboot/userdata_example.yaml | 09:05 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates master: WIP Tag the ha containers with 'pcmklatest' at deploy time https://review.openstack.org/491705 | 09:05 |
| shardy | it should be pretty simple to move away from the metadata service, but I'm not clear where it's accessed | 09:05 |
| shardy | maybe you, rcrit and stevebaker can chat about it later | 09:06 |
| *** zshi_laptop has quit IRC | 09:06 | |
| jaosorior | shardy: https://raw.githubusercontent.com/openstack/novajoin/master/files/cloud-config-novajoin.json | 09:06 |
| *** gvrangan has quit IRC | 09:07 | |
| jaosorior | basically we are using a dynamic vendordata plugin. So I'm not sure if this will work :/ | 09:08 |
| jaosorior | our plugin (novajoin) gets a notification when a node is created, calls freeipa, generates a host entry, and it then passes an OTP (one time password), that will be fetched via the metadata. | 09:09 |
| jaosorior | not entirely sure if that's possible with config-drive | 09:09 |
| *** ooolpbot has joined #tripleo | 09:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709066 | 09:10 |
| openstack | Launchpad bug 1709066 in tripleo "[CI] Upgrade job down: AnsibleUndefinedVariable: 'docker_registry_namespace' is undefined" [Critical,In progress] - Assigned to Alex Schultz (alex-schultz) | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709327 | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709630 | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710013 | 09:10 |
| *** ooolpbot has quit IRC | 09:10 | |
| openstack | Launchpad bug 1709327 in tripleo "CI: extremely long times of overcloud deploy in multinode jobs" [Critical,Triaged] | 09:10 |
| openstack | Launchpad bug 1709630 in tripleo "docker run gnocchi_db_sync is failing in gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 09:10 |
| openstack | Launchpad bug 1710013 in tripleo "KeyError: 'bmc_prefix' ovb version in oooq is no longer working" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 09:10 |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-quickstart master: Add missing deploy_composable_scenario in featuresets https://review.openstack.org/492915 | 09:11 |
| *** psahoo has quit IRC | 09:11 | |
| *** ykarel is now known as ykarel|lunch | 09:12 | |
| *** gvrangan has joined #tripleo | 09:15 | |
| *** kinton has quit IRC | 09:16 | |
| *** psahoo has joined #tripleo | 09:16 | |
| shardy | jaosorior: ack, sounds like it needs more discussion | 09:18 |
| jaosorior | jistr: what does deploy_composable_scenario actually do? | 09:18 |
| shardy | jaosorior: maybe we need to add a boolean to undercloud.conf so it can be easily re-enabled when needed | 09:18 |
| jaosorior | shardy: haha well, it hasn't been disabled yet :D | 09:19 |
| shardy | jaosorior: sure, my point is we don't always need it, so it could at least be made optional | 09:19 |
| jistr | jaosorior: i think something that should be replaced with `composable_scenario != null` ;) | 09:19 |
| jaosorior | but yeah, I'm in for a boolean, would REALLY it to be deployed by default though. deploying TLS everywhere is not that simple, and I wouldn't wanna make it even harder for deployers. | 09:19 |
| jaosorior | shardy: ^^ | 09:19 |
| *** sshnaidm|afk is now known as sshnaidm|off | 09:19 | |
| shardy | jaosorior: sure, I'm not going to argue over defaults, just nice to keep chipping away at the memory footprint of the undercloud ;) | 09:20 |
| *** zshi_laptop has joined #tripleo | 09:20 | |
| *** psahoo has quit IRC | 09:22 | |
| *** jprovazn has quit IRC | 09:22 | |
| *** ebarrera has quit IRC | 09:23 | |
| *** funzo has joined #tripleo | 09:26 | |
| *** limao has quit IRC | 09:26 | |
| dciabrin | jaosorior, I confirm that when mysql is started via systemctl (i.e like in non-HA), it still reads all files from /etc/my.cnf.d | 09:28 |
| *** ebarrera has joined #tripleo | 09:28 | |
| jistr | bogdando: replied at https://review.openstack.org/#/c/492613 -- i think that patch should be fine as the change is backwards compatible. I'll fix the other one. Thanks for the reviews! | 09:29 |
| *** funzo has quit IRC | 09:31 | |
| gfidente | jistr do I read correctly containers step_3 are started after puppet step3 finished? | 09:31 |
| bogdando | jistr, shardy: hm, could this https://review.openstack.org/#/c/491886/1 be the fix for the networks undefined issue?.. | 09:31 |
| jistr | gfidente: yes | 09:32 |
| bogdando | ah, likely it isn't, I do not use networks isolation | 09:32 |
| jistr | bogdando: this is the fix for network undefined: https://github.com/openstack/tripleo-common/commit/e48a73cb4ef2d40cfdd322e7cc19f615aa4ad1c7 | 09:33 |
| jaosorior | dciabrin: thanks! | 09:33 |
| chem | dciabrin: bandini he, now that we have those shinny pcmklatest, do you have somewhere the step used to untag/tag/ image and restart the container somewhere ? | 09:36 |
| *** gvrangan has quit IRC | 09:36 | |
| openstackgerrit | Jiri Stransky proposed openstack/puppet-tripleo master: Allow configuring multiple insecure registries https://review.openstack.org/492612 | 09:36 |
| gfidente | jistr given you seem free | 09:40 |
| gfidente | I think I spotted the problem blocking scenario001 | 09:41 |
| gfidente | https://review.openstack.org/#/c/491826 | 09:41 |
| *** dparkes has quit IRC | 09:41 | |
| gfidente | but it fails as if it couldn't read the config file anyway | 09:41 |
| gfidente | http://logs.openstack.org/26/491826/4/check/gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container/e22a6f9/logs/subnode-2/var/log/containers/gnocchi/gnocchi-upgrade.log.txt.gz | 09:41 |
| gfidente | you have suggestions? | 09:41 |
| *** rwsu has quit IRC | 09:44 | |
| gfidente | jistr I remember you mentioning we don't have /kolla/ at that stage? | 09:49 |
| *** garyk1 has joined #tripleo | 09:49 | |
| *** garyk has quit IRC | 09:49 | |
| gfidente | jistr note in that job it's puppet-ceph creating the ceph.conf file | 09:50 |
| *** jfrancoa has quit IRC | 09:50 | |
| *** jfrancoa has joined #tripleo | 09:52 | |
| *** ratailor has quit IRC | 09:54 | |
| *** rwsu has joined #tripleo | 09:56 | |
| *** salmankhan has quit IRC | 09:59 | |
| *** salmankhan has joined #tripleo | 09:59 | |
| *** daidv has quit IRC | 10:00 | |
| *** pgadiya has quit IRC | 10:01 | |
| *** sshnaidm|off has quit IRC | 10:02 | |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates master: Remove ipv6 specific network templates https://review.openstack.org/492550 | 10:05 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates master: Convert network templates to be rendered via j2 https://review.openstack.org/492218 | 10:05 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates master: WIP: Adds pacemaker update_tasks for Pike minor update workflow https://review.openstack.org/486567 | 10:06 |
| chem | marios: bandini: hi, I would like to rebase https://review.openstack.org/#/c/486567/ on top of https://review.openstack.org/#/c/491705/ somehow | 10:08 |
| chem | marios: bandini: do you mind ? | 10:08 |
| chem | marios bandini: or best we land https://review.openstack.org/#/c/491705/ ? | 10:09 |
| *** ooolpbot has joined #tripleo | 10:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709066 | 10:10 |
| openstack | Launchpad bug 1709066 in tripleo "[CI] Upgrade job down: AnsibleUndefinedVariable: 'docker_registry_namespace' is undefined" [Critical,In progress] - Assigned to Alex Schultz (alex-schultz) | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709327 | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709630 | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710013 | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710112 | 10:10 |
| *** ooolpbot has quit IRC | 10:10 | |
| *** jaganathan has quit IRC | 10:10 | |
| openstack | Launchpad bug 1709327 in tripleo "CI: extremely long times of overcloud deploy in multinode jobs" [Critical,Triaged] | 10:10 |
| openstack | Launchpad bug 1709630 in tripleo "docker run gnocchi_db_sync is failing in gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 10:10 |
| openstack | Launchpad bug 1710013 in tripleo "KeyError: 'bmc_prefix' ovb version in oooq is no longer working" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 10:10 |
| openstack | Launchpad bug 1710112 in tripleo "Upgrade failing with AnsibleUndefinedVariable: 'deploy_composable_scenario' is undefined" [Critical,In progress] - Assigned to Jiří Stránský (jistr) | 10:10 |
| *** ratailor has joined #tripleo | 10:13 | |
| *** jprovazn has joined #tripleo | 10:14 | |
| *** ansmith has quit IRC | 10:15 | |
| *** ansmith_ has quit IRC | 10:15 | |
| jprovazn | gfidente: hi | 10:16 |
| gfidente | jprovazn hey | 10:16 |
| jprovazn | gfidente: can you take a quick look at https://review.openstack.org/#/c/492589/ - it should make https://review.openstack.org/#/c/482496/ happy (and then also https://review.openstack.org/#/c/482501/ will be happy eventually) | 10:17 |
| skramaja | dciabrin: shardy step 2 is now completed, the problem happens with the definiiton of "TimeZone" parameter. docker run is failing with it. though haven't looked at the root cause, will file a bug for now. | 10:17 |
| *** pgadiya has joined #tripleo | 10:18 | |
| gfidente | jprovazn I think I understand but why would we be creating the key twice? | 10:19 |
| gfidente | jprovazn I mean why can't we just move that into the puppet/mds profile? | 10:19 |
| gfidente | ah external ceph with puppet | 10:20 |
| gfidente | ? | 10:20 |
| jprovazn | gfidente: don't forget about "external ceph" scenario where mds is not used | 10:20 |
| jprovazn | gfidente: yes | 10:20 |
| gfidente | right but then it's the admin creating the keys | 10:20 |
| *** gkadam has joined #tripleo | 10:20 | |
| gfidente | like for the other services | 10:20 |
| jprovazn | gfidente: you mean then admin is supposed to create keys manually? | 10:21 |
| gfidente | in the ceph cluster yes | 10:21 |
| gfidente | I think what we might need to do is https://review.openstack.org/#/c/482496/6/puppet/services/ceph-base.yaml into ceph-external.yaml too? | 10:21 |
| jprovazn | gfidente: in ceph cluster yes - of course, this is more about client side (inject part is removed in a follow up patch) | 10:22 |
| *** psachin has quit IRC | 10:22 | |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-heat-templates master: Correct gnocchi-upgrade command quotes https://review.openstack.org/491826 | 10:22 |
| gfidente | problem with this is it won't work with ansible | 10:22 |
| jprovazn | gfidente: let me send you depends-on patch, sec | 10:23 |
| gfidente | I would try to remove it if we can | 10:23 |
| jprovazn | gfidente: I think once this one is merged too, it should work with and w/o ceph-ansiblehttps://review.openstack.org/#/c/482501/ | 10:23 |
| gfidente | from the manila profile | 10:23 |
| gfidente | right because of the check on the file itself | 10:24 |
| gfidente | jprovazn so you want to keep it there | 10:24 |
| jprovazn | gfidente: well, I want it there as long as puppet-ceph is kept | 10:25 |
| gfidente | ok I think if it helps ci | 10:25 |
| gfidente | we can merge it first and iterate later? | 10:25 |
| derekh | EmilienM: when you get a chance can you look back at the patch to add ironic to overcloud CI https://review.openstack.org/#/c/485261/ | 10:25 |
| jprovazn | gfidente: these 3 patches should assure that manila will be set with all variants - internal/external ceph, ansible/puppet deployment | 10:25 |
| jprovazn | gfidente: I don't think there is other option to satisfy all deployment options tripleo supports | 10:26 |
| *** funzo has joined #tripleo | 10:26 | |
| gfidente | jprovazn so I was thinking we could deploy the manila keyring from ceph-external | 10:27 |
| gfidente | to remove it from the manila profile | 10:27 |
| *** jaganathan has joined #tripleo | 10:27 | |
| jprovazn | gfidente: we could, but how idoes it play with old puppet-ceph internal deployment? | 10:27 |
| gfidente | again this would be pretty much to replicate the situation with ceph-ansible | 10:27 |
| gfidente | jprovazn internal deployments don't use ceph-external and the keyring is created by -base | 10:28 |
| *** ansmith has joined #tripleo | 10:28 | |
| *** ansmith_ has joined #tripleo | 10:29 | |
| jprovazn | gfidente: so are you prposing to move ceph creation to both ceph-external and ceph-mds? | 10:29 |
| gfidente | ceph-base you did it already in https://review.openstack.org/#/c/479426/ | 10:29 |
| jprovazn | gfidente: right, and https://review.openstack.org/#/c/492589/ is to make CI happy with ^ | 10:30 |
| gfidente | sorry wrong link | 10:30 |
| gfidente | right one : https://review.openstack.org/#/c/482496/6/puppet/services/ceph-base.yaml | 10:30 |
| gfidente | so I am saying if you add those bits into ceph-external.yaml | 10:30 |
| gfidente | too | 10:31 |
| *** funzo has quit IRC | 10:31 | |
| gfidente | we don't need https://review.openstack.org/#/c/482496/6/puppet/services/ceph-base.yaml | 10:31 |
| gfidente | sorry again wrong link | 10:31 |
| gfidente | we don't need https://review.openstack.org/#/c/492589/ | 10:31 |
| gfidente | and you can remove ceph::key entirely from the manila profile | 10:31 |
| jprovazn | gfidente: true, yea, that might work too | 10:32 |
| gfidente | but I am not sure how to make that all pass CI | 10:32 |
| gfidente | if not by using the define you pushed anyway | 10:32 |
| jprovazn | gfidente: not sure if having ceph-key creation on 2 places in templates is better than having shared code for both cases honestly, but I can do that | 10:32 |
| gfidente | SO | 10:32 |
| gfidente | I guess we can try merging what we have and iterate? | 10:33 |
| jprovazn | gfidente: I think defined check is needed for your variant too | 10:33 |
| jprovazn | gfidente: yep | 10:33 |
| gfidente | well in my variant we should be able to remove that code from manila.pp | 10:33 |
| gfidente | but the define is good for CI as we move key creation from one profile to the other anyway | 10:34 |
| jprovazn | gfidente: yes, it wouldn't be in manila.pp | 10:34 |
| *** sshnaidm has joined #tripleo | 10:34 | |
| *** sshnaidm is now known as sshnaidm|off | 10:34 | |
| gfidente | ok so let's try to merge what you have | 10:34 |
| jprovazn | ok | 10:34 |
| gfidente | and update with green CI | 10:34 |
| marios | chem: i don't mind either way whatever is easier | 10:35 |
| *** psachin has joined #tripleo | 10:36 | |
| chem | marios: easier is to have bandini patch merged :) | 10:36 |
| marios | chem: yeah but is it ready/tested enough so might not be so easy | 10:36 |
| marios | chem: why not rebase, don't they already share most of the parent reviews ? /me checking | 10:36 |
| *** iranzo has joined #tripleo | 10:36 | |
| marios | chem: ah no bandini one doesn't have shardy reviews as parent at all | 10:37 |
| chem | marios: nope | 10:37 |
| chem | marios: will rebase all this later, in the middle of something else | 10:38 |
| *** ykarel|lunch is now known as ykarel | 10:38 | |
| shardy | marios: hey I was thinking next time we rebase that series we should probably add a bug reference - do you know if one was raised to track this work through feature freeze, or should I raise one now? | 10:39 |
| *** limao has joined #tripleo | 10:40 | |
| marios | shardy: not aware of one - yeah i think we are all usign the etherpad for tracking so it would be useful. don't think we need one "because upgrades" i mean ffe | 10:40 |
| chem | shardy: it should be in the update rfe I guess | 10:41 |
| shardy | There is https://bugs.launchpad.net/tripleo/+bug/1664650 but yeah I was thinking more a catch-all "minor updates w/containers and pacemaker not working" to make it explicit | 10:41 |
| openstack | Launchpad bug 1664650 in tripleo "Minor updates cause control-plane downtime" [High,Triaged] | 10:41 |
| shardy | https://bugs.launchpad.net/tripleo/+bug/1674767 is related too I think | 10:41 |
| openstack | Launchpad bug 1674767 in tripleo "Minor update workflow doesn't work for custom roles" [High,Triaged] | 10:42 |
| *** jaganathan has quit IRC | 10:42 | |
| marios | shardy: well can we perhaps rename one and mark the rest clones rather than file anew one | 10:42 |
| *** ykarel_ has joined #tripleo | 10:42 | |
| marios | shardy: i mean if it is already confusing | 10:42 |
| shardy | marios: yeah, I'll rename that second one then | 10:42 |
| shardy | as the rework we're doing e.g update_tasks etc should make updates for custom roles possible too | 10:43 |
| marios | shardy: cool thanks. slightly related in that it uses the same tht series, have you seen https://bugs.launchpad.net/tripleo/+bug/1708115 esp comment #8 by any chance | 10:44 |
| openstack | Launchpad bug 1708115 in tripleo "Ensure non-controller are usable after upgrade and before converge." [Critical,Triaged] | 10:44 |
| openstackgerrit | Jan Provaznik proposed openstack/tripleo-heat-templates master: Let mds create manila key and fs https://review.openstack.org/482496 | 10:44 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Fix metadata_settings in containerized mongodb https://review.openstack.org/492014 | 10:45 |
| marios | shardy: idea is to add the 'when' from the tags when the config is written, so we don't break the existing upgrade_tasks | 10:45 |
| chandankumar | arxcruz: hello | 10:45 |
| shardy | marios: ack yeah sounds reasonable | 10:45 |
| shardy | marios: ideally I'd like to move away from that function that writes the playbook in tripleoclient, but we'll need to switch from tags to variables in upgrade_tasks | 10:46 |
| chandankumar | arxcruz: https://review.openstack.org/#/c/492525/ can we get same patch for heat also? | 10:46 |
| *** ykarel has quit IRC | 10:46 | |
| shardy | maybe it's safest to wait until the next release to do that, but I think it's the cleanest end solution? | 10:46 |
| *** akrivoka has joined #tripleo | 10:46 | |
| openstackgerrit | Bogdan Dobrelya proposed openstack-infra/tripleo-ci master: Rework the getthelogs helper script for wget recursive https://review.openstack.org/492178 | 10:46 |
| marios | shardy: yeah. i mean i was trying to come up with a way to make it work without having to consider all the upgrade_tasks too so late | 10:46 |
| shardy | happy to see we're incrementally moving towards making the scripts a wrapper for the playbook runs tho :) | 10:46 |
| shardy | marios: yeah - tbh I was tempted to just change them all and document it as a release note | 10:47 |
| shardy | but it's probably safer to wait I guess - this seems like a reasonable compromise | 10:47 |
| marios | shardy: yeah. i mean this is also not the best interface :) https://review.openstack.org/#/c/490847/4/scripts/upgrade-non-controller.sh but if nothing else is consistent :) | 10:47 |
| marios | shardy: ie. you'd still use that to upgrade your 'non controller' (wherver disable_upgrade_deployment). | 10:47 |
| marios | shardy: assuming it works, we can consider removing the tripleo_upgrade_node.sh alltogether (i.e. the file we deliver during the composable steps) | 10:48 |
| *** iranzo has quit IRC | 10:48 | |
| shardy | marios: yup that would be good, thanks for digging into it | 10:48 |
| shardy | maybe I can look at starting a patch to switch away from tags, even if we don't land it until after we branch | 10:49 |
| openstackgerrit | Jan Provaznik proposed openstack/puppet-tripleo master: Do not create fs and server side key from manila https://review.openstack.org/482501 | 10:50 |
| marios | shardy: ack mostly wanted to run the idea of https://review.openstack.org/#/c/491749/ for adding the whens for now | 10:50 |
| marios | shardy: since we discussed, it is the 'safest' today | 10:50 |
| marios | shardy: s/we/as we | 10:50 |
| shardy | marios: one question - how does this iterate over each step/tag? | 10:50 |
| marios | shardy: the loop variable same as update tasks | 10:50 |
| shardy | that was why I had to use variables for update_tasks - ansible can't iterate over task include changing the tags | 10:51 |
| jprovazn | gfidente: I updated https://review.openstack.org/#/c/482496/ and https://review.openstack.org/#/c/482501/ - this should match your proposal, right? | 10:51 |
| marios | shardy: i mean here https://review.openstack.org/#/c/490848/2/common/deploy-steps.j2 | 10:51 |
| shardy | will that work if we don't add conditionals to every task to replace the tags? | 10:51 |
| * shardy is probably missing something and needs to look closer, sec | 10:51 | |
| *** jkilpatr has quit IRC | 10:52 | |
| marios | shardy: that one adds an upgrade_steps_playbook by including the upgrades_steps.yaml which are writting after you do the config download | 10:53 |
| gfidente | jprovazn yep | 10:53 |
| gfidente | thanks :D | 10:53 |
| gfidente | fingers crossed | 10:53 |
| jprovazn | gfidente: cool, let's see how green it is :) | 10:53 |
| jprovazn | thanks | 10:53 |
| shardy | Yeah but I'm not clear how passing step as the loop var to upgrade_steps_tasks.yaml helps when the conditional in the tasks is a tag, not a when step == N | 10:53 |
| shardy | do we have something that rewrites that somewhere? | 10:53 |
| marios | shardy: the 'when' should be added into upgrade_tasks at the time of download usign https://review.openstack.org/#/c/491749/ | 10:54 |
| shardy | marios: Oh I see | 10:55 |
| shardy | hmm | 10:55 |
| shardy | Ok, kinda ugly, let me think about it | 10:55 |
| marios | shardy: /me confused not sure if we have chicken/egg situation :) | 10:55 |
| marios | shardy: i think it should be ok, i.e. we write the when from the tags at the time the role/upgrade_tasks.yaml is written out | 10:55 |
| *** pkovar has joined #tripleo | 10:55 | |
| shardy | marios: basically we need to swith t-h-t from tags to when I think, but this looks like an OK band-aid if we decide that's too risky | 10:56 |
| *** rwsu has quit IRC | 10:56 | |
| shardy | :) | 10:56 |
| marios | shardy: right, the whole point of this was to *avoid* touching the existing upgrade_tasks | 10:56 |
| shardy | marios: yup, I'm just thinking about how we do change them, e.g even if we wait is there any way to avoid a flag-day conversion | 10:57 |
| shardy | I'm not sure there is | 10:57 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates master: WIP: Adds pacemaker update_tasks for Pike minor update workflow https://review.openstack.org/486567 | 10:57 |
| shardy | maybe we can make SkipUpgradeConfigTags still work by extracting the step variable, but I'm not sure we can support both models concurrently in t-h-t | 10:57 |
| shardy | like, if we can't avoid a one-off conversion, it might be cleaner to just do it | 10:58 |
| shardy | but open to discussion on that :) | 10:58 |
| * shardy goes to consider options over lunch | 10:59 | |
| *** stendulker has quit IRC | 10:59 | |
| *** shardy is now known as shardy_lunch | 10:59 | |
| jistr | gfidente: re ceph conf file not being read -- should be solved by PS 5 here right? https://review.openstack.org/#/c/491826 | 11:01 |
| gfidente | jistr no I am not sure | 11:01 |
| gfidente | kolla copy is dealing with trailing / | 11:01 |
| gfidente | I thought I posted a comment about that | 11:02 |
| jistr | gfidente: it's doing a python glob, so the star could make a difference. lemme double check | 11:02 |
| gfidente | yes I saw the glob | 11:02 |
| gfidente | but | 11:02 |
| *** limao_ has joined #tripleo | 11:03 | |
| gfidente | https://github.com/openstack/kolla/blob/master/docker/base/set_configs.py#L86 | 11:03 |
| gfidente | all other containers rely on that | 11:03 |
| gfidente | (for /etc/ceph) | 11:03 |
| *** limao has quit IRC | 11:03 | |
| gfidente | so I just don't think the * is the issue | 11:03 |
| jistr | gfidente: http://paste.openstack.org/show/618157/ | 11:05 |
| jistr | gfidente: it does make a difference though | 11:05 |
| jistr | at least in what the glob returns | 11:06 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates master: WIP: Adds pacemaker update_tasks for Pike minor update workflow https://review.openstack.org/486567 | 11:06 |
| jistr | gfidente: ok but maybe it results in the same thing in the end, when considering how kolla processes it (not 100% sure...) | 11:06 |
| gfidente | jistr so in the other containers we use / not /* | 11:07 |
| jistr | ack | 11:07 |
| gfidente | I have a local env with the above patch applied | 11:07 |
| gfidente | without the * | 11:07 |
| gfidente | at step 3 | 11:07 |
| gfidente | hope to see it failg | 11:07 |
| *** rwsu has joined #tripleo | 11:08 | |
| *** ratailor has quit IRC | 11:08 | |
| *** limao_ has quit IRC | 11:09 | |
| *** limao has joined #tripleo | 11:09 | |
| jistr | gfidente: with docker-puppet.py we execute puppet as if it was step 6 https://github.com/openstack/tripleo-heat-templates/blob/master/docker/docker-puppet.py#L255 | 11:10 |
| *** ooolpbot has joined #tripleo | 11:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709066 | 11:10 |
| openstack | Launchpad bug 1709066 in tripleo "[CI] Upgrade job down: AnsibleUndefinedVariable: 'docker_registry_namespace' is undefined" [Critical,In progress] - Assigned to Alex Schultz (alex-schultz) | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709327 | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709630 | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710013 | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710112 | 11:10 |
| *** ooolpbot has quit IRC | 11:10 | |
| openstack | Launchpad bug 1709327 in tripleo "CI: extremely long times of overcloud deploy in multinode jobs" [Critical,Triaged] | 11:10 |
| openstack | Launchpad bug 1709630 in tripleo "docker run gnocchi_db_sync is failing in gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 11:10 |
| openstack | Launchpad bug 1710013 in tripleo "KeyError: 'bmc_prefix' ovb version in oooq is no longer working" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 11:10 |
| openstack | Launchpad bug 1710112 in tripleo "Upgrade failing with AnsibleUndefinedVariable: 'deploy_composable_scenario' is undefined" [Critical,In progress] - Assigned to Jiří Stránský (jistr) | 11:10 |
| gfidente | jistr oh | 11:10 |
| jistr | gfidente: so if ceph is `step >= 2`, it should be picked up | 11:10 |
| gfidente | yeah | 11:10 |
| jistr | https://github.com/openstack/puppet-tripleo/blob/master/manifests/profile/base/ceph.pp#L50 | 11:10 |
| gfidente | my deployment just went into step 4 | 11:11 |
| gfidente | http://paste.openstack.org/show/618159/ | 11:12 |
| *** jkilpatr has joined #tripleo | 11:12 | |
| gfidente | and yet I have that error in the gnocchi-upgrade.log file | 11:12 |
| gfidente | so I am confused now | 11:13 |
| gfidente | I don't know why it moved into step4 | 11:13 |
| gfidente | and I am also confused by why it failed to read the file which instead seems to be there | 11:13 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates master: WIP: Adds pacemaker update_tasks for Pike minor update workflow https://review.openstack.org/486567 | 11:15 |
| *** pdeore has quit IRC | 11:16 | |
| *** pdeore has joined #tripleo | 11:16 | |
| *** ansmith_ has quit IRC | 11:16 | |
| *** ansmith has quit IRC | 11:17 | |
| *** zshi_laptop has quit IRC | 11:18 | |
| *** udesale has quit IRC | 11:18 | |
| *** shreshtha has quit IRC | 11:19 | |
| openstackgerrit | Merged openstack/tripleo-quickstart master: fix FreeIPA dns server issue https://review.openstack.org/491888 | 11:19 |
| *** shreshtha has joined #tripleo | 11:19 | |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-heat-templates master: Containerized swift upgrade fix [WIP] https://review.openstack.org/492869 | 11:20 |
| jistr | gfidente, jfrancoa: got it! see https://review.openstack.org/#/c/491826/5/docker/services/gnocchi-api.yaml | 11:20 |
| gfidente | you know things | 11:21 |
| *** pdeore has quit IRC | 11:21 | |
| gfidente | but it was also obvious looking at the lines above that one | 11:21 |
| gfidente | will you amend it? | 11:21 |
| *** jkilpatr has quit IRC | 11:24 | |
| *** limao has quit IRC | 11:24 | |
| *** jprovazn has quit IRC | 11:24 | |
| *** jfrancoa has quit IRC | 11:24 | |
| *** garyk1 has quit IRC | 11:24 | |
| *** ebarrera has quit IRC | 11:24 | |
| *** shardy_lunch has quit IRC | 11:24 | |
| *** cshastri has quit IRC | 11:24 | |
| *** bogdando has quit IRC | 11:24 | |
| *** yamahata has quit IRC | 11:24 | |
| jaosorior | jistr: any idea who else we can ask for reviews for this https://review.openstack.org/#/c/492915/ ? | 11:26 |
| jaosorior | would sure be good to get that fixed :D | 11:26 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Keep dynamic network creation backward compatible. https://review.openstack.org/491510 | 11:26 |
| *** ratailor has joined #tripleo | 11:27 | |
| *** funzo has joined #tripleo | 11:27 | |
| *** bogdando has joined #tripleo | 11:27 | |
| *** bandini has quit IRC | 11:28 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Set virsh secret with an init step when using Ceph https://review.openstack.org/492082 | 11:28 |
| *** bandini has joined #tripleo | 11:29 | |
| *** limao has joined #tripleo | 11:30 | |
| *** shardy_lunch has joined #tripleo | 11:30 | |
| *** yamahata has joined #tripleo | 11:31 | |
| *** funzo has quit IRC | 11:32 | |
| *** jprovazn has joined #tripleo | 11:32 | |
| jaosorior | bandini: can you revisit https://review.openstack.org/#/c/489963/ ? | 11:33 |
| *** shardy_lunch is now known as shardy | 11:33 | |
| *** sayani0_0 has joined #tripleo | 11:34 | |
| *** brault has quit IRC | 11:35 | |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-heat-templates master: Correct gnocchi-upgrade command quotes https://review.openstack.org/491826 | 11:35 |
| openstackgerrit | Damien Ciabrini proposed openstack/tripleo-heat-templates master: Bind mount tripleo.cnf in transient bootstrap containers https://review.openstack.org/492963 | 11:36 |
| jistr | gfidente: i also removed the asterisk again. Not sure what difference it makes (it would probably behave the same in the end?) but at least we'll be consistent with the rest of templates. | 11:36 |
| gfidente | consistent yes | 11:36 |
| lvdombrkr | hello guys, which is triple0 stable version righ now, which to deploy? | 11:37 |
| jaosorior | dciabrin: hey! now that that bit is working, lets pull the mongodb patches and try those out | 11:38 |
| dciabrin | jaosorior, absolutely. let's tackle gnocchi as well | 11:38 |
| jaosorior | dciabrin: gnocchi is being worked on, I think | 11:39 |
| *** dougbtv_ is now known as dougbtv | 11:39 | |
| jaosorior | dciabrin: https://review.openstack.org/491826 | 11:39 |
| dciabrin | jaosorior, ah ok | 11:39 |
| *** mdbooth has quit IRC | 11:39 | |
| jaosorior | we could try out the patch though | 11:40 |
| jaosorior | dciabrin: ^ | 11:40 |
| dciabrin | jaosorior, let's try out both patches, I'm feeling lucky today :) | 11:40 |
| jaosorior | haha like a boss | 11:41 |
| *** mdbooth has joined #tripleo | 11:42 | |
| *** rwsu has quit IRC | 11:42 | |
| jistr | jaosorior: i added shardy and adarazs_mtg to the review :) It's blocked on the gnocchi-upgrade bug though :D | 11:43 |
| jistr | jaosorior: i'll add a depends-on to get the CI running in parallel | 11:43 |
| jaosorior | uok | 11:44 |
| jaosorior | jistr: we're trying the gnocchi patch in a bit | 11:44 |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-quickstart master: Add missing deploy_composable_scenario in featuresets https://review.openstack.org/492915 | 11:44 |
| gfidente | though I am still unsure why locally the deployment went into step4 despite the command failin | 11:45 |
| gfidente | the initial submission which broke the job also passed CI somehow | 11:45 |
| *** shreshtha has quit IRC | 11:45 | |
| *** liverpooler has joined #tripleo | 11:45 | |
| gfidente | jistr is that the detached: false | 11:45 |
| jistr | gfidente: when you do `docker ps -a`, does the container have exit code other than 0? | 11:46 |
| gfidente | no it's 0 | 11:47 |
| jistr | gfidente: `detached: false` should execute the container "synchronously" in paunch, and wait for it to finish before proceeding, and AFAIK it should also fail the deployment if the container fails | 11:47 |
| jistr | gfidente: ok so maybe it's a bug in gnocchi-upgrade then? returning 0 even though it failed | 11:47 |
| gfidente | so I see the upgrade command existing with 1 | 11:47 |
| jistr | huh | 11:47 |
| jistr | so bootstrap exec bug? | 11:47 |
| * jistr looks | 11:47 | |
| gfidente | but in CI it is doing what it is expected I suppose | 11:48 |
| jistr | hmm bootstrap looks ok https://github.com/openstack/tripleo-common/blob/master/scripts/bootstrap_host_exec#L16 | 11:49 |
| lvdombrkr | hello guys, which is triple0 stable version righ now, which to deploy? | 11:50 |
| *** cshastri has joined #tripleo | 11:51 | |
| *** garyk has joined #tripleo | 11:51 | |
| jistr | lvdombrkr: hi, latest stable is Ocata | 11:51 |
| *** jkilpatr has joined #tripleo | 11:51 | |
| lvdombrkr | jistr: is this guide i can follow up? https://docs.openstack.org/tripleo-docs/latest/install/environments/environments.html | 11:52 |
| *** jfrancoa has joined #tripleo | 11:52 | |
| *** ebarrera has joined #tripleo | 11:52 | |
| *** jprovazn has quit IRC | 11:53 | |
| *** lucasagomes is now known as lucas-hungry | 11:54 | |
| jistr | lvdombrkr: yup i think that should work. I'm not aware how often those docs are regenerated, but it looks exactly the same as the ones on http://tripleo.org/ | 11:54 |
| jistr | so those are the main tripleo docs | 11:54 |
| arxcruz | chandankumar: what you mean? | 11:54 |
| dtantsur | mwhahaha, EmilienM, can we move on with https://review.openstack.org/#/c/490026/ and the next patches, if the CI is fine please? | 11:55 |
| lvdombrkr | jistr:thank you! | 11:55 |
| dtantsur | I cannot promise what happens on upgrade, if we don't | 11:55 |
| weshay | trown|outtypewww, https://review.openstack.org/#/c/492915/ | 12:01 |
| openstackgerrit | Merged openstack/tripleo-common master: Add OVN images to the overcloud containers https://review.openstack.org/489875 | 12:02 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/tripleo-common master: Updated from global requirements https://review.openstack.org/487398 | 12:02 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/tripleo-heat-templates master: Updated from global requirements https://review.openstack.org/488148 | 12:02 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Move HAProxy's public TLS logic from controller to service template https://review.openstack.org/491446 | 12:02 |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: Remove --subunit --no-pretty from tempest run https://review.openstack.org/491170 | 12:02 |
| openstackgerrit | Merged openstack/puppet-tripleo master: Modify resource dependencies of certmonger_user resources https://review.openstack.org/490791 | 12:03 |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Switch to use local buildlogs mirror https://review.openstack.org/492786 | 12:03 |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates master: Don't unregister on system/resource delete https://review.openstack.org/492970 | 12:06 |
| *** jprovazn has joined #tripleo | 12:09 | |
| *** jprovazn is now known as jprovazn_mtg | 12:09 | |
| *** rhallisey has joined #tripleo | 12:10 | |
| *** jprovazn_mtg is now known as jprovazn | 12:10 | |
| *** ooolpbot has joined #tripleo | 12:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709066 | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709327 | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709630 | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710013 | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710112 | 12:10 |
| *** ooolpbot has quit IRC | 12:10 | |
| openstack | Launchpad bug 1709066 in tripleo "[CI] Upgrade job down: AnsibleUndefinedVariable: 'docker_registry_namespace' is undefined" [Critical,In progress] - Assigned to Alex Schultz (alex-schultz) | 12:10 |
| openstack | Launchpad bug 1709327 in tripleo "CI: extremely long times of overcloud deploy in multinode jobs" [Critical,Triaged] | 12:10 |
| openstack | Launchpad bug 1709630 in tripleo "docker run gnocchi_db_sync is failing in gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 12:10 |
| openstack | Launchpad bug 1710013 in tripleo "KeyError: 'bmc_prefix' ovb version in oooq is no longer working" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 12:10 |
| openstack | Launchpad bug 1710112 in tripleo "Upgrade failing with AnsibleUndefinedVariable: 'deploy_composable_scenario' is undefined" [Critical,In progress] - Assigned to Jiří Stránský (jistr) | 12:10 |
| lvdombrkr | is there posibility to controller node and compute node install in one physical server? | 12:12 |
| *** ratailor has quit IRC | 12:13 | |
| lvdombrkr | not for production env. of course | 12:13 |
| jaosorior | yep | 12:13 |
| jaosorior | you can create an all-in-one role | 12:13 |
| *** trown|outtypewww is now known as trown | 12:14 | |
| lvdombrkr | jaosorior: thank you.. | 12:14 |
| trown | weshay: the composable_roles and deploy_composable_scenario keys control different things | 12:14 |
| *** tzumainn has joined #tripleo | 12:15 | |
| trown | weshay: I found all of those variables quite obtuse while trying to make the 3nodes job though | 12:15 |
| *** pkovar has quit IRC | 12:15 | |
| *** pgadiya has quit IRC | 12:16 | |
| *** sshnaidm|off has quit IRC | 12:17 | |
| weshay | trown++ | 12:18 |
| weshay | I think ben messed that up a bit imho | 12:18 |
| weshay | composable deployments were conflated w/ multinode nodepool and scenarios | 12:19 |
| *** jprovazn has quit IRC | 12:19 | |
| *** mwhahaha has quit IRC | 12:19 | |
| *** jprovazn has joined #tripleo | 12:19 | |
| trown | weshay: ya, one of the reasons I didnt go for full generic multinode right off is because that interface requires a big refactor, but it could be an early queens thing to go after | 12:20 |
| *** mwhahaha has joined #tripleo | 12:20 | |
| weshay | trown, ya.. +100 | 12:20 |
| *** bandini has quit IRC | 12:21 | |
| *** tzumainn has quit IRC | 12:21 | |
| *** cdearborn has joined #tripleo | 12:21 | |
| mrch | hey guys, do you support openstack-murano installation automation? | 12:22 |
| *** mandre_mtg has quit IRC | 12:22 | |
| shardy | mrch: No I don't think so, the current supported services can be found here: | 12:23 |
| shardy | https://github.com/openstack/tripleo-heat-templates/tree/master/puppet/services | 12:23 |
| shardy | mrch: it's a plugin model, so support could be added, but I'm not aware of anyone working on it atm | 12:23 |
| *** bandini has joined #tripleo | 12:23 | |
| *** mandre has joined #tripleo | 12:24 | |
| weshay | mwhahaha, can we prioritize this patch as well https://review.openstack.org/#/c/492315/ | 12:27 |
| *** morazi has joined #tripleo | 12:28 | |
| *** bfournie has quit IRC | 12:29 | |
| *** bfournie has joined #tripleo | 12:29 | |
| *** catintheroof has joined #tripleo | 12:30 | |
| *** bfournie has quit IRC | 12:33 | |
| *** pkovar has joined #tripleo | 12:34 | |
| *** jmelvin has joined #tripleo | 12:36 | |
| *** gkadam has quit IRC | 12:37 | |
| *** abishop has joined #tripleo | 12:39 | |
| *** jtomasek has joined #tripleo | 12:39 | |
| *** ansmith_ has joined #tripleo | 12:40 | |
| *** dbecker has joined #tripleo | 12:41 | |
| openstackgerrit | Merged openstack/puppet-tripleo master: Do not include manila ceph key resource twice https://review.openstack.org/492589 | 12:41 |
| *** ansmith has joined #tripleo | 12:42 | |
| mwhahaha | weshay: that one isn't passing ci so i don't htink it's the fix | 12:43 |
| mwhahaha | weshay: i rechecked the other patch to correct it and let me see if that one actually passed ci | 12:43 |
| mwhahaha | weshay: https://review.openstack.org/#/c/491826/ | 12:44 |
| *** pradk has joined #tripleo | 12:44 | |
| *** jlabarre has quit IRC | 12:45 | |
| weshay | mwhahaha, k.. thanks I wasn't sure if the last run w/ that revert gnocchi patch ran w/ all of your patches | 12:46 |
| mwhahaha | weshay: it looks like ci is much better today so i've rechecked that one and we'll see which passes CI first | 12:47 |
| *** pblaho has quit IRC | 12:47 | |
| *** mwhahaha changes topic to "CI Status: GREENish | "you can't land a Cessna with workarounds, so don't try in Quickstart" - TripleO : http://tripleo.org/ | https://wiki.openstack.org/wiki/TripleO | Meetings On Tuesdays at 14:00 UTC here." | 12:47 | |
| weshay | ya.. the zuul queue looks more sane to me | 12:48 |
| weshay | mwhahaha++ | 12:48 |
| weshay | pabelanger++ | 12:48 |
| pabelanger | mwhahaha: I've enqueue and promoted 491878 once again | 12:48 |
| *** jlabarre has joined #tripleo | 12:49 | |
| pabelanger | it failed on a contain job last night | 12:49 |
| pabelanger | container* | 12:49 |
| mwhahaha | pabelanger: ok, we fixed up the packaging itself so it's not as critical anymore | 12:50 |
| pabelanger | Oh, ya. That works too | 12:50 |
| *** ansmith_ has quit IRC | 12:52 | |
| *** ansmith has quit IRC | 12:53 | |
| *** jpena|off is now known as jpena | 12:54 | |
| pradk | jistr, gfidente, thx for patches on gnocchi upgrade issue while i was out. you guys rock! | 12:55 |
| *** eck`gone is now known as eck` | 12:56 | |
| *** dbecker has quit IRC | 12:57 | |
| *** jcoufal has joined #tripleo | 12:58 | |
| *** janki has quit IRC | 12:58 | |
| gfidente | pradk wait | 12:58 |
| *** gbarros has joined #tripleo | 12:59 | |
| gfidente | pradk if it doesn't pass ci yet you're booked | 12:59 |
| gfidente | LS | 12:59 |
| jaosorior | jistr: is this https://github.com/openstack/tripleo-heat-templates/blob/master/docker/services/nova-api.yaml#L87 a file within the container or on the host? | 12:59 |
| pradk | gfidente, lol sure, i'm back now | 13:00 |
| pabelanger | mwhahaha: http://logs.openstack.org/15/492915/1/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/58d54bc/console.html is an issue | 13:00 |
| pabelanger | mwhahaha: we run yum install before we update repo files to use infra AFS mirrors | 13:01 |
| pabelanger | so there is a chance we can have a failure like above | 13:01 |
| *** zshi_laptop has joined #tripleo | 13:02 | |
| pradk | jaosorior, did anyone reach out to you on telemetry having trouble reaching keystone when tls is enabled? | 13:03 |
| jaosorior | pradk: I don't recall, what's up? | 13:03 |
| pradk | jaosorior, i don't think there is a bug yet.. but someone pinged me couple of days ago asking for help . Basically with tls enabled deploy was failing | 13:04 |
| *** lucas-hungry is now known as lucasagomes | 13:05 | |
| pradk | jaosorior, i suggested he ping you as well .. i guess he dint, so lets assume all is well :) | 13:05 |
| jaosorior | pradk: what's the error log? | 13:05 |
| *** bfournie has joined #tripleo | 13:06 | |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates master: Remove ipv6 specific network templates https://review.openstack.org/492550 | 13:06 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates master: Convert network templates to be rendered via j2 https://review.openstack.org/492218 | 13:06 |
| pradk | jaosorior, basically ceilometer can't reach keystone endpoint when tls is enabled .. i don't have the pastern handy, as i was chatting with him on my mobile. | 13:06 |
| *** MVenesio has joined #tripleo | 13:07 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: TLS everywhere: Configure CA for mongodb https://review.openstack.org/492992 | 13:07 |
| jaosorior | dciabrin: ^^ | 13:07 |
| *** lblanchard has joined #tripleo | 13:08 | |
| dciabrin | jaosorior, updating... | 13:08 |
| *** ansmith has joined #tripleo | 13:09 | |
| jaosorior | pradk: uhm... it might be several things. Either a timing issue (ceilometer trying to reach keystone before httpd is started), or something restarting httpd, or ceilometer hardcoding http:// | 13:09 |
| *** ansmith_ has joined #tripleo | 13:09 | |
| *** ooolpbot has joined #tripleo | 13:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709066 | 13:10 |
| openstack | Launchpad bug 1709066 in tripleo "[CI] Upgrade job down: AnsibleUndefinedVariable: 'docker_registry_namespace' is undefined" [Critical,In progress] - Assigned to Alex Schultz (alex-schultz) | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709327 | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709630 | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710013 | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710112 | 13:10 |
| *** ooolpbot has quit IRC | 13:10 | |
| openstack | Launchpad bug 1709327 in tripleo "CI: extremely long times of overcloud deploy in multinode jobs" [Critical,Triaged] | 13:10 |
| openstack | Launchpad bug 1709630 in tripleo "docker run gnocchi_db_sync is failing in gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 13:10 |
| openstack | Launchpad bug 1710013 in tripleo "KeyError: 'bmc_prefix' ovb version in oooq is no longer working" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 13:10 |
| openstack | Launchpad bug 1710112 in tripleo "Upgrade failing with AnsibleUndefinedVariable: 'deploy_composable_scenario' is undefined" [Critical,In progress] - Assigned to Jiří Stránský (jistr) | 13:10 |
| jaosorior | pradk: but yeah, would need the logs | 13:10 |
| *** leitan has joined #tripleo | 13:12 | |
| *** rlandy has joined #tripleo | 13:12 | |
| *** skramaja has quit IRC | 13:13 | |
| *** MVenesio has quit IRC | 13:13 | |
| *** MVenesio has joined #tripleo | 13:13 | |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder master: Add kpartx as a requirement to build images https://review.openstack.org/493004 | 13:16 |
| marios | shardy: http://paste.openstack.org/raw/618169/ | 13:17 |
| marios | shardy: it is cool stuff :) | 13:17 |
| marios | shardy: the missign piece for me is how are passing role_name ... will that be directly to the invocation of ansible-playbook (I commented here https://review.openstack.org/#/c/486104/13/common/deploy-steps.j2) | 13:17 |
| *** oanson has quit IRC | 13:19 | |
| *** oanson has joined #tripleo | 13:19 | |
| *** jlabarre has quit IRC | 13:21 | |
| *** jlabarre has joined #tripleo | 13:23 | |
| *** dprince has joined #tripleo | 13:24 | |
| *** ansmith_ has quit IRC | 13:24 | |
| *** ansmith has quit IRC | 13:25 | |
| * mwhahaha is enjoying all the Verified+2 emails | 13:25 | |
| shardy | marios: nice! role_name is set already in the dynamic inventory | 13:26 |
| shardy | marios: so you can just do e.g --limit Controller | 13:26 |
| shardy | or --limit overcloud-controller-0 | 13:27 |
| *** sshnaidm has joined #tripleo | 13:27 | |
| shardy | or just run for all nodes | 13:27 |
| shardy | and the role_name will be set correctly on each node | 13:27 |
| shardy | marios: when run via heat we set the role_name as an input to the SoftwareDeployment, so it basically works the same | 13:27 |
| pabelanger | mwhahaha: much nicer with working infrastructure :) | 13:28 |
| pabelanger | lets see what fails in the gate today | 13:28 |
| shardy | https://github.com/openstack/tripleo-validations/blob/master/tripleo_validations/inventory.py#L159 | 13:28 |
| shardy | marios: ^^ FYI | 13:28 |
| *** funzo has joined #tripleo | 13:29 | |
| mwhahaha | pabelanger: yea i noticed a few [256]s again but that was before my final tripleo-ci patch merged so we'll see if those keep popping up | 13:29 |
| *** udesale has joined #tripleo | 13:30 | |
| pabelanger | mwhahaha: ya, those [256]s should be much smaller now. Keeping an eye on elastic-recheck to be sure | 13:30 |
| *** garyk has quit IRC | 13:30 | |
| marios | shardy: cool i didn't check the inventory will look | 13:31 |
| marios | shardy: thanks very much | 13:31 |
| shardy | marios: np, good to see this is starting to come together :) | 13:31 |
| *** pradk has quit IRC | 13:33 | |
| *** funzo has quit IRC | 13:33 | |
| *** amoralej is now known as amoralej|lunch | 13:35 | |
| *** agopi|away has joined #tripleo | 13:38 | |
| *** jprovazn has quit IRC | 13:41 | |
| *** jprovazn has joined #tripleo | 13:41 | |
| *** weshay is now known as weshay_doc | 13:43 | |
| shardy | marios, jistr, mwhahaha: Hey if the gate is looking better should we consider sending some of these remaining patches related to enabling minor updates? | 13:47 |
| shardy | https://review.openstack.org/#/q/topic:add_update_tasks | 13:47 |
| *** ansmith has joined #tripleo | 13:47 | |
| shardy | the top 4/5 reviews look pretty green and ready AFAICS | 13:47 |
| *** ansmith_ has joined #tripleo | 13:47 | |
| shardy | we discussed adding a bug reference, but I'm loath to do another rebase just to add it if we know updates is a critical blocker and they're otherwise ready to land | 13:48 |
| *** jpena is now known as jpena|off | 13:48 | |
| jistr | yea +1 | 13:48 |
| shardy | https://review.openstack.org/#/c/483293/ is the top of the series | 13:48 |
| *** zshi_laptop has quit IRC | 13:48 | |
| jistr | we probably shouldn't wait on upgrade job as its fix is still blocked by sc001 not passing | 13:48 |
| shardy | jistr: Ok, I was wondering about that | 13:49 |
| chem | marios: shardy jistr do you know which review are adding to paunch the possibility to detect new docker image and install them ? | 13:49 |
| jistr | chem: just a bug so far i guess https://bugs.launchpad.net/tripleo/+bug/1708967 | 13:51 |
| openstack | Launchpad bug 1708967 in tripleo "prepare command needs to discover the versioned tag from the latest tag" [High,Triaged] - Assigned to Steve Baker (steve-stevebaker) | 13:51 |
| shardy | chem: yeah, until quite recently the assumption was that the tag (and thus the json input to paunch) would change | 13:51 |
| shardy | sounds like we're going to handle that via the prepare command so paunch doesn't need to care about pulling and comparing versions of otherwise identical images | 13:52 |
| mwhahaha | shardy: yea let me slowly add them to the gate and keep an eye on them today | 13:54 |
| shardy | mwhahaha: ack thanks that would be great | 13:54 |
| *** zshi_laptop has joined #tripleo | 13:54 | |
| *** gcerami|Zz is now known as gcerami | 13:55 | |
| chem | shardy: jistr ack, thanks for the pointer and the explanation. So for now we don't exactly know how update of non-pacemaker container will work, but we assume that it will be done with paunch at the config step. | 13:55 |
| shardy | mwhahaha: Oh, I just realized the top patch Depends-On https://review.openstack.org/#/c/491463/ | 13:56 |
| shardy | I thought that fix had landed | 13:56 |
| mwhahaha | k i'm keeping an eye on that one too | 13:56 |
| jistr | chem: it should work already, provided that you give it a different tag than it had | 13:56 |
| * mwhahaha was reviewing the patches | 13:56 | |
| shardy | mwhahaha: ack, Ok sounds good | 13:56 |
| jistr | chem: so on paunch level, all should already work | 13:56 |
| chem | jistr: ack, just wanted to make sure, thanks. | 13:56 |
| jistr | we just need a image prepare command that will make sure that we always feed different tags, that's the part which isn't implemented yet | 13:57 |
| *** trown is now known as trown|brb | 13:57 | |
| shardy | Yeah, the thing which AFAIK paunch can't do is automatically figure out there's a new "latest" | 13:57 |
| shardy | which might be nice, but it'd also be nice to easily see exactly what version is running | 13:57 |
| shardy | sounds like that's the approach we're going after with the prepare bug | 13:57 |
| openstackgerrit | Attila Darazs proposed openstack/tripleo-quickstart-extras master: GATE CHECK for quickstart-extras https://review.openstack.org/472607 | 14:00 |
| *** shreshtha has joined #tripleo | 14:00 | |
| jistr | right | 14:00 |
| openstackgerrit | Damien Ciabrini proposed openstack/tripleo-heat-templates master: Internal TLS support for mongodb container https://review.openstack.org/492878 | 14:00 |
| jistr | so if you wanted to test updates right now, it's possible but a bit convoluted | 14:01 |
| *** zshi_laptop has quit IRC | 14:01 | |
| openstackgerrit | Damien Ciabrini proposed openstack/tripleo-heat-templates master: Internal TLS support for mongodb container https://review.openstack.org/492878 | 14:03 |
| jistr | actually... maybe not so much. Initially i thought that you'd have to re-tag things manually, but there might be a simpler variant. Right now we could be updating between :latest from local registry and :latest from dockerhub, back and forth as many times as needed, without any local tag magic. | 14:03 |
| jistr | change of image URL will have similar effect as change of tag | 14:03 |
| jistr | chem: ^ | 14:04 |
| jistr | (just talking about services managed by paunch) | 14:04 |
| openstackgerrit | Matt Young proposed openstack/tripleo-quickstart-extras master: Display output from machine provisioning script https://review.openstack.org/493023 | 14:04 |
| *** sayani0_0 has quit IRC | 14:06 | |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates master: Convert compute-role.yaml to role.role.j2.yaml https://review.openstack.org/482623 | 14:06 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates master: Convert controller-role.yaml to role.role.j2.yaml https://review.openstack.org/480272 | 14:06 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates master: Convert objectstorage-role.yaml to role.role.j2.yaml https://review.openstack.org/482624 | 14:06 |
| *** pchavva has joined #tripleo | 14:07 | |
| *** yprokule has quit IRC | 14:08 | |
| *** ooolpbot has joined #tripleo | 14:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709066 | 14:10 |
| openstack | Launchpad bug 1709066 in tripleo "[CI] Upgrade job down: AnsibleUndefinedVariable: 'docker_registry_namespace' is undefined" [Critical,In progress] - Assigned to Alex Schultz (alex-schultz) | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709327 | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709630 | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710013 | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710112 | 14:10 |
| *** ooolpbot has quit IRC | 14:10 | |
| openstack | Launchpad bug 1709327 in tripleo "CI: extremely long times of overcloud deploy in multinode jobs" [Critical,Triaged] | 14:10 |
| openstack | Launchpad bug 1709630 in tripleo "docker run gnocchi_db_sync is failing in gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 14:10 |
| mwhahaha | hmm i think scenario001 containers is broken | 14:10 |
| openstack | Launchpad bug 1710013 in tripleo "KeyError: 'bmc_prefix' ovb version in oooq is no longer working" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 14:10 |
| openstack | Launchpad bug 1710112 in tripleo "Upgrade failing with AnsibleUndefinedVariable: 'deploy_composable_scenario' is undefined" [Critical,In progress] - Assigned to Jiří Stránský (jistr) | 14:10 |
| jistr | mwhahaha: thanks for recheck at https://review.openstack.org/#/c/491463/, was just looking at the failure | 14:10 |
| jistr | deployed server bootstrap fail, seems unrelated enough and hopefully intermittent | 14:11 |
| mwhahaha | jistr: yea looked be ok | 14:11 |
| mwhahaha | jistr: do you know what's going wrong on the scenario001 container job for https://review.openstack.org/#/c/491826/ | 14:11 |
| fultonj | jschlueter: i don't understanad your comment in https://review.openstack.org/#/c/492303/1/docker/services/ceph-ansible/ceph-base.yaml | 14:12 |
| mwhahaha | jistr: i've seen that on two different rechecks now so i'm wondering if that scenario is broken but i can't find the failure | 14:12 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates master: Convert objectstorage-role.yaml to role.role.j2.yaml https://review.openstack.org/482624 | 14:12 |
| *** pradk has joined #tripleo | 14:12 | |
| fultonj | yes, chmod != setfacl, but do you see some sort of problem or are you just stating what it does? | 14:12 |
| jschlueter | fultonj: 644 is user read/write group read, world read ... acl entries listed only adds read only for several other users | 14:13 |
| fultonj | that is a true statement | 14:13 |
| jschlueter | fultonj: is this a secret file or just a config? | 14:13 |
| jschlueter | and what is default permissions for that file? | 14:13 |
| fultonj | jschlueter: it is the mode of the keyring | 14:13 |
| jschlueter | fultonj: is it expected that the keyring can be read by anyone with access to the filesyste,? | 14:14 |
| fultonj | jschlueter: in all prior versions we set the same permission to this file | 14:14 |
| pradk | gfidente, dont we create the metrics pool when ceph is deployed automatically? https://review.openstack.org/#/c/491826/ | 14:14 |
| fultonj | jschlueter: the key cannot be read the openstack containers otherwise | 14:14 |
| pradk | gfidente, is this a side effect of containerizing ceph? | 14:14 |
| jschlueter | fultonj: ok just asking if it was the same thing | 14:14 |
| fultonj | the key must be read by the containers so they may connect to ce3ph | 14:14 |
| jaosorior | pradk: well, ceph isn't deployed by default, right? | 14:15 |
| chem | jistr: hum the lastest hub/latest local would only work upstream, so I guess we just assume that we will get the new tags from the improved prepare command so that paunch can work its magic. | 14:15 |
| fultonj | i wanted to setfacl per user to be more restrictive but there were two issues | 14:15 |
| pradk | jaosorior, in scenario001 it is | 14:15 |
| jschlueter | fultonj: was just asking as it wasn't stated that it should be restrictive or not | 14:15 |
| fultonj | 1. setfacl's are not passed up via kolla_config | 14:15 |
| pabelanger | remote: https://review.openstack.org/493024 DNM - Testing ruby gem reverse proxy cache | 14:15 |
| pabelanger | mwhahaha: ^ | 14:15 |
| fultonj | 2. the uids are not consistent in kolla images as they are in the rpm packages | 14:15 |
| fultonj | jschlueter: not a problem | 14:15 |
| mwhahaha | pabelanger: cool | 14:15 |
| fultonj | jschlueter: just wasn't sure if that was a question | 14:16 |
| gfidente | jschlueter so acls were an attempt to make this more secure than it was | 14:16 |
| gfidente | let me find the code | 14:16 |
| *** trown|brb is now known as trown | 14:17 | |
| pradk | jaosorior, gfidente, obviously its working in non containerized job so ceph logic to create pools is kicking in scenario001 jobs | 14:17 |
| *** sshnaidm is now known as sshnaidm|off | 14:17 | |
| pradk | but not in container ones | 14:17 |
| gfidente | pradk pools are created in step4 correct | 14:18 |
| gfidente | pradk but I think jistr said we run puppet always as if it was step6 | 14:18 |
| openstackgerrit | Julia Kreger proposed openstack/diskimage-builder master: Add netbase to ensure /etc/protocols is placed for debian https://review.openstack.org/490656 | 14:18 |
| *** jtomasek has quit IRC | 14:19 | |
| *** jprovazn has quit IRC | 14:19 | |
| pradk | gfidente, we run db sync in step 3 in container templates | 14:20 |
| *** amoralej|lunch is now known as amoralej | 14:20 | |
| *** salmankhan has quit IRC | 14:21 | |
| chandankumar | mwhahaha: hello | 14:21 |
| mwhahaha | chandankumar: hey | 14:21 |
| pradk | gfidente, so seems like a ordering issue .. we should either move this out to > step4 or ensure ceph pools are created earlier | 14:21 |
| chandankumar | mwhahaha: do we use identity v2 api by default in tripleo currently or v3 api? | 14:22 |
| pradk | but all db syncs are in step 3.. so i'm a bit skeptical to change that | 14:22 |
| mwhahaha | chandankumar: i want to say 3 but i know we still have issues with mistral not supporting v3 so i'm not sure if we just switch to v3 for the undercloud only | 14:22 |
| *** bogdando has quit IRC | 14:22 | |
| chandankumar | mwhahaha: what about overcloud? | 14:23 |
| mwhahaha | chandankumar: that's what i mean | 14:23 |
| mwhahaha | chandankumar: looks like overcloud is v2 by default | 14:23 |
| mwhahaha | because we still haven't been able to fully switch | 14:24 |
| chandankumar | mwhahaha: but it is deprecated since mitaka | 14:24 |
| * mwhahaha blames mistral | 14:24 | |
| mwhahaha | we configure it and we provide an overcloudrc.v3 | 14:24 |
| mwhahaha | but not every service supported it | 14:24 |
| *** psachin has quit IRC | 14:24 | |
| mwhahaha | we were working on getting mistral switched over but have been runing into some issues and being so late in the cycle i'm not sure we want to continue that work right now | 14:24 |
| chandankumar | mwhahaha: so basically openstack core services works fine with v3 in tripleo | 14:25 |
| mwhahaha | chandankumar: yes that is my understanding | 14:25 |
| pradk | gfidente, looking at https://github.com/openstack/tripleo-heat-templates/blob/master/docker/services/ceph-ansible/ceph-base.yaml .. only step i see here is step2 | 14:26 |
| pradk | gfidente, where is the step 4 coming in for ceph pools? | 14:26 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates master: Convert blockstorage-role.yaml to role.role.j2.yaml https://review.openstack.org/482625 | 14:28 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates master: Convert cephstorage-role.yaml to role.role.j2.yaml https://review.openstack.org/482626 | 14:28 |
| jaosorior | mwhahaha: I thought so too, but apparently we're deploying it by default in containerized environments. | 14:29 |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder master: Increase timeout for removal https://review.openstack.org/493026 | 14:29 |
| jaosorior | mwhahaha: and while trying the TLS bits with mongodb, we uncovered that issue. | 14:29 |
| vkmc | hey guys, I'm seeing one of the containers fail due to puppet... can the missing dependencies would be the cause for this? http://paste.openstack.org/show/618190/ | 14:29 |
| jaosorior | mwhahaha: the tls commit being this one https://review.openstack.org/#/c/492878/ | 14:29 |
| *** funzo has joined #tripleo | 14:29 | |
| vkmc | not sure if I need to change the puppet manifest or update images | 14:30 |
| mwhahaha | jaosorior: hmm ok | 14:30 |
| jaosorior | mwhahaha: might wanna remove it from here https://github.com/openstack/tripleo-heat-templates/blob/master/environments/docker.yaml#L38 | 14:31 |
| mwhahaha | jaosorior: unles there's a reason to keep deploying it, yea that's probably best | 14:31 |
| mwhahaha | pradk: didn't we remove mongodb from the regular (non containerized) deployment? | 14:31 |
| jaosorior | vkmc: mayhaps. Is there anything else in the logs? | 14:32 |
| gfidente | pradk right with ceph-ansible there won't be any issue | 14:32 |
| pradk | mwhahaha, yea i think so | 14:32 |
| gfidente | the issue is with puppet-ceph only | 14:32 |
| gfidente | but it used to be step 4 | 14:32 |
| mwhahaha | pradk: we appear to sill be doing it for containers | 14:32 |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: Exclude networking-bagpipe from dlrn https://review.openstack.org/491878 | 14:32 |
| vkmc | jaosorior, not really... I can paste the full output but those are the only suspicious lines | 14:32 |
| *** fultonj has quit IRC | 14:32 | |
| gfidente | pradk can you tentatively push a change for puppet-tripleo to move pools in step3 | 14:32 |
| gfidente | and depend the CI fix on it to see if it passes? | 14:32 |
| pradk | mwhahaha, we dont use mongodb anymore.. so if its coming up in containers its not used, unless zaqar is still using it | 14:33 |
| jaosorior | vkmc: lets get the whole output. Lets see if there's anything else there | 14:33 |
| openstackgerrit | Andreas Karis proposed openstack/tripleo-heat-templates master: Case insentitive MAC address matching in OsNetConfigMappings https://review.openstack.org/492244 | 14:33 |
| vkmc | jaosorior, sure thing, just a second | 14:33 |
| gfidente | pradk that'd be manifests/profile/base/ceph/mon.pp | 14:33 |
| *** jtomasek has joined #tripleo | 14:33 | |
| pradk | gfidente, hmm but this is happening before puppet kicks in | 14:33 |
| *** fultonj has joined #tripleo | 14:33 | |
| mwhahaha | jaosorior: https://review.openstack.org/#/c/466265/ you removed it :D | 14:34 |
| gfidente | pradk? | 14:34 |
| pradk | gfidente, i think* .. i thought we dont rely in puppet tripleo steps anymore | 14:34 |
| openstackgerrit | Andreas Karis proposed openstack/tripleo-heat-templates master: Case insentitive MAC address matching in OsNetConfigMappings https://review.openstack.org/492244 | 14:34 |
| mwhahaha | jaosorior: sounds like we need that for containers | 14:34 |
| pabelanger | mwhahaha: okay, .gemrc updates works | 14:34 |
| pabelanger | going to add that to configure_mirror.sh script | 14:34 |
| mwhahaha | pabelanger: cool | 14:34 |
| *** itlinux has joined #tripleo | 14:34 | |
| *** funzo has quit IRC | 14:34 | |
| jaosorior | mwhahaha: right, I didn't really know if there was a reason why it was still deployed in containerized environments | 14:34 |
| pabelanger | mwhahaha: for puppet-openstack-integration, you'll need to create it for /root/.gemrc because you gem install as root | 14:34 |
| mwhahaha | jaosorior: looks like a it was missed in the conversion to containers | 14:34 |
| chandankumar | mwhahaha: thanks :-) | 14:35 |
| pradk | gfidente, may be my understanding is wrong here .. jistr can you confirm. Does puppet-tripleo steps matter in container deploy? | 14:35 |
| gfidente | jistr ^^ see my last comment in https://review.openstack.org/#/c/491826/ | 14:35 |
| *** rwsu has joined #tripleo | 14:36 | |
| pradk | may be i should rephrase the question .. at what point if puppet-tripleo kicks in doesa it happen? we run gnocchi-upgrade in step 4 in containre templates | 14:36 |
| pradk | if puppet is running in step6. to create pools, thats too late | 14:36 |
| pradk | s/step 4/ 3 | 14:36 |
| *** udesale has quit IRC | 14:37 | |
| pradk | gfidente, good question on why we dint see this before.. thats because the sack initilization during upgrades is new in osp12 | 14:38 |
| pradk | gfidente, we expect the storage to be available and configured when upgrade runs.. that dep dint exist before | 14:38 |
| vkmc | jaosorior, full log | 14:39 |
| vkmc | https://paste.fedoraproject.org/paste/EZF5722FUVXxsBuoeZjPfA | 14:39 |
| vkmc | let me stripe the color codes | 14:39 |
| gfidente | pradk great so | 14:39 |
| shardy | pradk: the step in puppet-tripleo does matter if the puppet code is doing initialization e.g it runs via docker-puppet here: | 14:39 |
| gfidente | I would tentatively push a change for puppet-tripleo | 14:39 |
| shardy | https://github.com/openstack/tripleo-heat-templates/blob/master/docker/deploy-steps-playbook.yaml#L77 | 14:39 |
| gfidente | and move pools into step3 | 14:39 |
| gfidente | depend the db_sync on it and check if it passes | 14:39 |
| pradk | shardy, thx for confirming | 14:40 |
| pradk | gfidente, hmm but db sync runs in docker template | 14:40 |
| pradk | gfidente, which is whats failing afaict | 14:40 |
| pradk | https://github.com/openstack/tripleo-heat-templates/blob/master/docker/services/gnocchi-api.yaml#L119 | 14:41 |
| vkmc | jaosorior, https://paste.fedoraproject.org/paste/9CxuE81hjd-yyZmhfxGzyw | 14:41 |
| vkmc | much better | 14:41 |
| pradk | gfidente, ^^ its not run via puppet, we explicitly run this in container in step 4 | 14:41 |
| gfidente | pradk no pool creation happens in puppet | 14:42 |
| gfidente | https://github.com/openstack/puppet-tripleo/blob/master/manifests/profile/base/ceph/mon.pp#L47 | 14:42 |
| jistr | gfidente: ack but pradk said they need the storage available too? (that means OSDs?) | 14:42 |
| pradk | right, so when you say depend db_sync on it | 14:42 |
| pradk | so lets back for a second.. so how is the non container deploy working fine.. we dont have this issue there, which means the ordering there is fine. | 14:44 |
| jaosorior | vkmc: the logs seem incomplete :/ | 14:44 |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-quickstart-extras master: Switch to overcloudrc.v3 for running tempest https://review.openstack.org/493030 | 14:45 |
| pradk | jistr, gfidente, thats because puppet-tripleo runs the db::sync in step 4 for gnocchi https://github.com/openstack/puppet-tripleo/blob/master/manifests/profile/base/gnocchi/api.pp#L93 | 14:45 |
| chandankumar | arxcruz: mwhahaha let see how much it breaks https://review.openstack.org/#/c/493030/ | 14:45 |
| chandankumar | arxcruz: mwhahaha i am not sure about mistral condition part | 14:46 |
| pradk | jistr, in this case the failure is in step 3 in docker template not the puppet | 14:46 |
| pradk | gfidente, jistr, so should we move the db sync to step 4 in gnocchi docker template instead? | 14:46 |
| *** akrivoka has quit IRC | 14:46 | |
| gfidente | fwiw, we can create pools at anytime | 14:47 |
| gfidente | but they refuse writes until osds are available | 14:47 |
| gfidente | which means step 4 | 14:47 |
| pradk | hmm ok, so currently https://github.com/openstack/tripleo-heat-templates/blob/master/docker/services/gnocchi-api.yaml#L104-L121 is where it runs | 14:48 |
| *** bogdando has joined #tripleo | 14:48 | |
| pradk | at which point pools are not available | 14:48 |
| pradk | for my understanding, where is puppet kicking in .. before or after this step? | 14:48 |
| *** iranzo has joined #tripleo | 14:48 | |
| *** iranzo has joined #tripleo | 14:48 | |
| *** anshul has quit IRC | 14:49 | |
| jaosorior | mwhahaha: the depends on was needed cause that commit was failing on a scenario that's failing with the gnocchi bug | 14:50 |
| mwhahaha | jaosorior: right but that patch isn't going to land anytime soon | 14:51 |
| *** atheurer has joined #tripleo | 14:51 | |
| mwhahaha | jaosorior: it would be better to switch to the revert now | 14:51 |
| jaosorior | mwhahaha: I agree | 14:51 |
| pradk | gfidente, we do need ceph to be able to accept writes | 14:53 |
| pradk | at this stage | 14:53 |
| pradk | gfidente, we can try moving to step 3, but i think it will fail with some error | 14:54 |
| pradk | pool create that is | 14:54 |
| *** iranzo has quit IRC | 14:55 | |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: pin the ovb git repo to a working version https://review.openstack.org/492686 | 14:55 |
| *** ramishra has quit IRC | 14:55 | |
| *** rcernin has quit IRC | 14:57 | |
| pabelanger | 2 merges in a row, cool | 14:57 |
| shardy | \o/ | 14:58 |
| *** akrivoka has joined #tripleo | 14:59 | |
| *** jfrancoa has quit IRC | 14:59 | |
| gfidente | pradk yeah they won't accept writes | 15:00 |
| gfidente | because osds are setup in 3 | 15:01 |
| gfidente | I mean | 15:01 |
| gfidente | they could be created in 3 and they will be writable in 4 | 15:01 |
| gfidente | but then you'd want to move db init in 4 anyway | 15:01 |
| gfidente | so sounds like we just do that? | 15:02 |
| jistr | vkmc, jaosorior: hmm i don't think the logs are incomplete (e.g. "Applied catalog in ... seconds") but they don't show any error.. | 15:02 |
| pradk | hmm yea seems like it .. i dont know what implications if any will it have | 15:02 |
| pradk | jistr, you see any issues if i move the db sync to step 4 and api to step 5? in https://github.com/openstack/tripleo-heat-templates/blob/master/docker/services/gnocchi-api.yaml#L104-L121 | 15:03 |
| pradk | jistr, rest of the services run sycn in step 3 | 15:03 |
| jistr | vkmc: does the container show up in `docker ps -a` with nonzero exit status? | 15:04 |
| vkmc | jistr, it does yes... I'm pair debugging with jaosorior to see if we can reach any conclusion | 15:04 |
| ansiwen | shardy: do you remember that random mysql error while deploying overcloud? did you find the issue for that? | 15:04 |
| jistr | vkmc, jaosorior: ack | 15:05 |
| *** cshastri has quit IRC | 15:05 | |
| *** ykarel_ has quit IRC | 15:05 | |
| *** jlabarre has quit IRC | 15:06 | |
| *** jprovazn has joined #tripleo | 15:08 | |
| *** jprovazn has quit IRC | 15:08 | |
| *** jlabarre has joined #tripleo | 15:08 | |
| *** ebarrera has quit IRC | 15:09 | |
| *** ebarrera has joined #tripleo | 15:09 | |
| *** bswartz has joined #tripleo | 15:09 | |
| *** ramishra has joined #tripleo | 15:09 | |
| *** ebarrera has quit IRC | 15:10 | |
| *** ooolpbot has joined #tripleo | 15:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709066 | 15:10 |
| openstack | Launchpad bug 1709066 in tripleo "[CI] Upgrade job down: AnsibleUndefinedVariable: 'docker_registry_namespace' is undefined" [Critical,In progress] - Assigned to Alex Schultz (alex-schultz) | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709327 | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709630 | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710112 | 15:10 |
| *** ooolpbot has quit IRC | 15:10 | |
| openstack | Launchpad bug 1709327 in tripleo "CI: extremely long times of overcloud deploy in multinode jobs" [Critical,Triaged] | 15:10 |
| openstack | Launchpad bug 1709630 in tripleo "docker run gnocchi_db_sync is failing in gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 15:10 |
| openstack | Launchpad bug 1710112 in tripleo "Upgrade failing with AnsibleUndefinedVariable: 'deploy_composable_scenario' is undefined" [Critical,In progress] - Assigned to Jiří Stránský (jistr) | 15:10 |
| *** mrch has quit IRC | 15:10 | |
| gfidente | pradk go go go | 15:12 |
| gfidente | let CI tell us | 15:12 |
| jistr | pradk, gfidente: i hope there wouldn't be issues with that. Early start is needed for services which are used by other services or init tasks (DBs, keystone). Hopefully gnocchi is enough of a "leaf service" that it wouldn't matter. | 15:13 |
| pradk | it will be needed by ceilo-upgrade .. :/ | 15:14 |
| gfidente | good news is | 15:14 |
| gfidente | with ceph-ansible | 15:14 |
| gfidente | ceph inluding pools is completed in step2 | 15:14 |
| pradk | k lemme try a patch and see | 15:14 |
| gfidente | because orchestration happens outside | 15:14 |
| gfidente | so it should be a non-issue in the future | 15:14 |
| pradk | when does ceph-ansible kick in? | 15:14 |
| gfidente | (in which future?) | 15:14 |
| gfidente | pike! | 15:14 |
| gfidente | except not in CI yet | 15:14 |
| gfidente | blocked on a bunch of things | 15:14 |
| pradk | ok | 15:14 |
| gfidente | so maybe in the change | 15:14 |
| gfidente | use a TODO | 15:15 |
| gfidente | to remember we move it back? | 15:15 |
| gfidente | no sorry no | 15:15 |
| gfidente | puppet-ceph remains around for deprecation | 15:15 |
| gfidente | forget about it | 15:15 |
| gfidente | it will have to work with it in pike | 15:15 |
| mwhahaha | looks like the the revert failed | 15:15 |
| *** dprince has quit IRC | 15:17 | |
| *** jlabarre has quit IRC | 15:18 | |
| pradk | mwhahaha, i think this might be due to tls, but we dint have logs to debug | 15:20 |
| *** anshul has joined #tripleo | 15:20 | |
| pradk | donno if its the same issue.. but some services were having issues finding keystone | 15:20 |
| *** jlabarre has joined #tripleo | 15:22 | |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates master: Correct gnocchi-upgrade command quotes https://review.openstack.org/491826 | 15:22 |
| pradk | mwhahaha, hmm http://logs.openstack.org/15/492315/4/check/gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container/9c754cd/logs/subnode-2/var/log/containers/ceilometer/ceilometer-upgrade.log.txt.gz | 15:22 |
| *** ffiore has quit IRC | 15:22 | |
| pradk | 503 | 15:23 |
| mwhahaha | is this done via ansible or puppet? | 15:23 |
| mwhahaha | ie do we need retries? | 15:23 |
| mwhahaha | puppet has retries, not sure if the ansible version of that does | 15:23 |
| *** karthiks has quit IRC | 15:23 | |
| *** cylopez has quit IRC | 15:23 | |
| openstackgerrit | James Slagle proposed openstack/tripleo-validations master: Configurable ansible_ssh_user for tripleo-ansible-inventory https://review.openstack.org/493046 | 15:24 |
| pradk | mwhahaha, yea the docker template doesnt have retries https://github.com/openstack/tripleo-heat-templates/blob/master/docker/services/ceilometer-agent-central.yaml#L118 | 15:24 |
| *** shardy has quit IRC | 15:24 | |
| mwhahaha | meh | 15:24 |
| *** jtomasek has quit IRC | 15:25 | |
| *** jtomasek has joined #tripleo | 15:26 | |
| pradk | this is the piece of the architecture i dont fully understand, we have puppet doing db sync and we have ansible, why dont we just let puppet do it always .. | 15:26 |
| mwhahaha | the problem is it's an exec | 15:26 |
| jaosorior | have a good weekend folks! | 15:26 |
| mwhahaha | so we could if the exec is tagged | 15:27 |
| *** anshul has quit IRC | 15:27 | |
| *** jaosorior has quit IRC | 15:27 | |
| mwhahaha | i'm not a fan of this whole thing because we're duplicating (and running into problems when they diverge) much of the deployment actions | 15:27 |
| jistr | vkmc: hehe thanks for karma :) what was the issue in the end? | 15:28 |
| *** jtomasek has quit IRC | 15:28 | |
| jistr | (or solution) | 15:28 |
| pradk | gfidente, jistr, i moved the steps around.. lets see https://review.openstack.org/#/c/491826/ | 15:28 |
| vkmc | jistr, thanks you for helping me :) | 15:28 |
| vkmc | jistr, I'm pointing to the wrong puppet manifest in pacemaker/manila_share.yaml | 15:28 |
| vkmc | so... commiting changes now and redeploying the overcloud | 15:28 |
| vkmc | hopefully that should do it | 15:29 |
| jistr | vkmc: ack, thanks | 15:29 |
| *** cdearborn_ has joined #tripleo | 15:30 | |
| *** funzo has joined #tripleo | 15:30 | |
| *** funzo has quit IRC | 15:34 | |
| openstackgerrit | Victoria Martinez de la Cruz proposed openstack/tripleo-heat-templates master: Containerize Manila Share for HA https://review.openstack.org/482680 | 15:35 |
| *** jlinkes has quit IRC | 15:35 | |
| *** jlabarre has quit IRC | 15:40 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/instack-undercloud master: Set all nodes resource class to "baremetal" on undercloud install https://review.openstack.org/490851 | 15:40 |
| *** ramishra has quit IRC | 15:41 | |
| *** jlabarre has joined #tripleo | 15:43 | |
| *** cdearborn has quit IRC | 15:45 | |
| *** dpawar has quit IRC | 15:46 | |
| *** itlinux has quit IRC | 15:46 | |
| *** marios has quit IRC | 15:49 | |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-quickstart master: Add missing deploy_composable_scenario in featuresets https://review.openstack.org/492915 | 15:51 |
| fultonj | jistr: fyi, regarding https://bugs.launchpad.net/tripleo/+bug/1709689 | 15:56 |
| openstack | Launchpad bug 1709689 in tripleo "kolla rabbitmq container setup should not try to delete /etc/hosts" [Medium,Triaged] - Assigned to Jiří Stránský (jistr) | 15:56 |
| fultonj | jistr: i didn't see the issue in my logs from last night | 15:56 |
| fultonj | my deploy failed w/ http://paste.openstack.org/show/618183 as usual but it' | 15:57 |
| fultonj | it's not clear to me why i can't get a good docker-puppet.py run with the newest quickstart builds | 15:57 |
| *** salmankhan has joined #tripleo | 15:57 | |
| fultonj | i guess i'll keep investigating | 15:58 |
| fultonj | basically, i haven't been able to stand up an OC without hitting http://paste.openstack.org/show/618183 all week and i've been rebuilding my UC form oooq daily | 15:58 |
| fultonj | but on investigating the docker-puppet.py run, it's not always the same issue | 15:59 |
| jistr | so it didn't fail on the /etc/hosts but it failed on sth else? | 15:59 |
| fultonj | jistr: yes | 15:59 |
| fultonj | jistr: i can attach my var log messages | 15:59 |
| fultonj | to the same bug | 15:59 |
| fultonj | i had puppet-debug on this time | 16:00 |
| fultonj | DockerPuppetDebug: true | 16:00 |
| jistr | ok would be good to post them somewhere, not sure though if the same bug in case we're not hitting the /etc/hosts issue | 16:00 |
| fultonj | i can open a new bug | 16:00 |
| jistr | yea that would be better i think | 16:01 |
| fultonj | ok | 16:01 |
| openstackgerrit | Merged openstack/tripleo-ui master: Imported Translations from Zanata https://review.openstack.org/492441 | 16:04 |
| *** cdearborn_ has quit IRC | 16:07 | |
| *** dparkes has joined #tripleo | 16:08 | |
| *** ooolpbot has joined #tripleo | 16:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709066 | 16:10 |
| openstack | Launchpad bug 1709066 in tripleo "[CI] Upgrade job down: AnsibleUndefinedVariable: 'docker_registry_namespace' is undefined" [Critical,In progress] - Assigned to Alex Schultz (alex-schultz) | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709327 | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709630 | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710112 | 16:10 |
| openstack | Launchpad bug 1709327 in tripleo "CI: extremely long times of overcloud deploy in multinode jobs" [Critical,Triaged] | 16:10 |
| *** ooolpbot has quit IRC | 16:10 | |
| openstack | Launchpad bug 1709630 in tripleo "docker run gnocchi_db_sync is failing in gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 16:10 |
| openstack | Launchpad bug 1710112 in tripleo "Upgrade failing with AnsibleUndefinedVariable: 'deploy_composable_scenario' is undefined" [Critical,In progress] - Assigned to Jiří Stránský (jistr) | 16:10 |
| *** apetrich has quit IRC | 16:11 | |
| *** snecklifter has quit IRC | 16:13 | |
| mwhahaha | pabelanger: thoughts on http://logs.openstack.org/74/489874/6/check/gate-tripleo-ci-centos-7-scenario002-multinode-upgrades/b230f9b/console.html#_2017-08-11_13_56_41_887202 | 16:13 |
| *** apetrich has joined #tripleo | 16:13 | |
| pradk | mwhahaha, is there a way to do retry in exec .. this should be a heat thing ? | 16:15 |
| mwhahaha | pradk: i have no idea | 16:15 |
| mwhahaha | i don't know if heat has a retry mechanism for this | 16:16 |
| *** salmankhan has quit IRC | 16:17 | |
| pradk | mwhahaha, or perhaps i could move this to a later step | 16:17 |
| *** salmankhan has joined #tripleo | 16:17 | |
| mwhahaha | pradk: it doesn't matter unless you can be sure that httpd won't be restarted within that step | 16:17 |
| pradk | right | 16:17 |
| mwhahaha | pradk: we might be able to add the exec into puppet_config | 16:17 |
| EmilienM | hello | 16:18 |
| pradk | hmm | 16:18 |
| mwhahaha | pradk: actually no it won't work because the stuff isn't setup by then | 16:19 |
| mwhahaha | pradk: this is a deficiency of the docker config stuff because we're losing the retries | 16:19 |
| * mwhahaha is unsure how to proceed | 16:19 | |
| *** trown is now known as trown|lunch | 16:19 | |
| mwhahaha | pradk: we really need retries within the ceilometer-upgrade command | 16:20 |
| pradk | yea i wish instead of duplicating this, we just reused puppet | 16:20 |
| *** dtantsur is now known as dtantsur|afk | 16:20 | |
| mwhahaha | pradk: you could cheat and just exec puppet | 16:20 |
| mwhahaha | puppet -e 'include ::whatever::class' | 16:20 |
| mwhahaha | FACTER_step=5 puppet -e 'include ::tripleo::profile::base::cilometer::upgrade' | 16:21 |
| * mwhahaha feels dirty for sugesting that | 16:21 | |
| mwhahaha | unless puppet isn't actually in the container (which is possible) | 16:23 |
| pradk | hmm right another dependency to pull in | 16:23 |
| mwhahaha | we should have fixed ceilometer-upgrade rather than work around it in puppet | 16:24 |
| *** mcornea has quit IRC | 16:24 | |
| pradk | yea i agree .. lemme see if its possible, meanwhile to unblock us we need something quick | 16:24 |
| pradk | i'm surprised we dint run into this until now | 16:25 |
| pradk | if its intermittent may be we try a few rechecks and get this revert in? | 16:26 |
| mwhahaha | i don't know what's worse a broken gate or an intermittent gate | 16:26 |
| mwhahaha | i don't think it's a good idea to try and force it in | 16:26 |
| mwhahaha | or we'll just end up with stuck gate queue again | 16:26 |
| pabelanger | mwhahaha: that is possible a RAX networking issue | 16:26 |
| pradk | understand | 16:26 |
| pabelanger | mwhahaha: I'll look at apache logs | 16:26 |
| *** bogdando has quit IRC | 16:27 | |
| *** rlandy is now known as rlandy|brb | 16:27 | |
| *** itlinux has joined #tripleo | 16:33 | |
| *** derekh has quit IRC | 16:35 | |
| *** psachin has joined #tripleo | 16:37 | |
| *** jlabarre has quit IRC | 16:40 | |
| *** tesseract has quit IRC | 16:41 | |
| *** lucasagomes is now known as lucas-afk | 16:41 | |
| *** jlabarre has joined #tripleo | 16:41 | |
| *** sshnaidm|off has quit IRC | 16:42 | |
| *** dparkes has quit IRC | 16:43 | |
| *** dprince has joined #tripleo | 16:45 | |
| *** milan has quit IRC | 16:48 | |
| jidar | I've seen people suggesting to keep the puppet stuff entirely external to the container and just mapping in /opt/{puppet-whatever} into the container for execution | 16:48 |
| *** sshnaidm|off has joined #tripleo | 16:55 | |
| *** akrivoka has quit IRC | 16:56 | |
| *** rlandy|brb is now known as rlandy | 17:00 | |
| *** amoralej is now known as amoralej|off | 17:04 | |
| *** ansmith has quit IRC | 17:04 | |
| *** ansmith_ has quit IRC | 17:04 | |
| *** iranzo has joined #tripleo | 17:05 | |
| *** iranzo has joined #tripleo | 17:05 | |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates master: Don't unregister on system/resource delete https://review.openstack.org/492970 | 17:08 |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates master: Fix Heat condition for RHEL registration yum update https://review.openstack.org/492632 | 17:08 |
| *** psachin has quit IRC | 17:08 | |
| *** ansmith has joined #tripleo | 17:09 | |
| *** anshul has joined #tripleo | 17:09 | |
| *** ooolpbot has joined #tripleo | 17:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709066 | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709327 | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709630 | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710112 | 17:10 |
| openstack | Launchpad bug 1709066 in tripleo "[CI] Upgrade job down: AnsibleUndefinedVariable: 'docker_registry_namespace' is undefined" [Critical,In progress] - Assigned to Alex Schultz (alex-schultz) | 17:10 |
| *** ooolpbot has quit IRC | 17:10 | |
| openstack | Launchpad bug 1709327 in tripleo "CI: extremely long times of overcloud deploy in multinode jobs" [Critical,Triaged] | 17:10 |
| openstack | Launchpad bug 1709630 in tripleo "docker run gnocchi_db_sync is failing in gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 17:10 |
| openstack | Launchpad bug 1710112 in tripleo "Upgrade failing with AnsibleUndefinedVariable: 'deploy_composable_scenario' is undefined" [Critical,In progress] - Assigned to Jiří Stránský (jistr) | 17:10 |
| pabelanger | mwhahaha: http://logs.openstack.org/33/481233/3/gate/gate-tripleo-ci-centos-7-scenario004-multinode-oooq-puppet/09a3368/ looks like a real issue with mirror | 17:12 |
| pabelanger | will look shortly | 17:12 |
| gfidente | pradk did the CI pass? | 17:15 |
| *** vkhanna has joined #tripleo | 17:16 | |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates master: Consolidate puppet/docker deployments with one deploy steps workflow https://review.openstack.org/483293 | 17:25 |
| *** anshul has quit IRC | 17:25 | |
| *** iranzo has quit IRC | 17:29 | |
| *** funzo has joined #tripleo | 17:32 | |
| vkmc | jistr, I'm not entirely sure if latest changes are correct for this one https://review.openstack.org/#/c/482680/8..9/docker/services/pacemaker/manila-share.yaml | 17:34 |
| *** pchavva has quit IRC | 17:35 | |
| *** vkhanna1 has joined #tripleo | 17:35 | |
| *** funzo has quit IRC | 17:36 | |
| *** pkovar has quit IRC | 17:37 | |
| vkmc | in L46, I've seen other services pointing to the non-pacemaker template | 17:38 |
| *** vkhanna has quit IRC | 17:38 | |
| *** vkhanna1 is now known as vkhanna | 17:38 | |
| pradk | gfidente, i got further | 17:38 |
| *** masco has quit IRC | 17:38 | |
| gfidente | pradk cool | 17:38 |
| pradk | gfidente, not its failing with same error as the revert | 17:38 |
| pradk | now* | 17:38 |
| gfidente | ahahaha | 17:38 |
| *** garyk has joined #tripleo | 17:38 | |
| pradk | gfidente, we need retries during ceilometer-upgrade .. so the step change did fix the ceph issue | 17:39 |
| mwhahaha | time to bust out the lame for loop? :D | 17:39 |
| gfidente | pradk hey hey hey | 17:39 |
| gfidente | let's be clear | 17:40 |
| pradk | gnocchi-ceph issue :) | 17:40 |
| gfidente | ahahaha | 17:40 |
| pradk | lol | 17:40 |
| gfidente | I was joking | 17:40 |
| gfidente | obviously | 17:40 |
| gfidente | just in case | 17:40 |
| pradk | sure | 17:40 |
| *** dpawar has joined #tripleo | 17:40 | |
| pradk | if we can get the loop thing to fix this.. we can just merge this without revert i guess | 17:41 |
| pradk | as both are at the same point now | 17:41 |
| *** dpawar has quit IRC | 17:41 | |
| vkmc | dciabrin, you worked with Cinder HA so maybe you can help me out with this | 17:41 |
| pradk | but i have no idea how to retry in heat .. may be we do what mwhahaha suggested and use puppet exec trick | 17:41 |
| *** dpawar has joined #tripleo | 17:42 | |
| gfidente | pradk what is the command that fails? | 17:43 |
| pradk | gfidente, ceilo-upgrade http://logs.openstack.org/26/491826/7/check/gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container/83da5f9/logs/postci.txt.gz | 17:43 |
| pradk | with 503 http://logs.openstack.org/26/491826/7/check/gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container/83da5f9/logs/subnode-2/var/log/containers/ceilometer/ceilometer-upgrade.log.txt.gz | 17:43 |
| pradk | we had similar/same issue in non-containerized | 17:44 |
| pradk | which was fixed with a retry loop | 17:44 |
| gfidente | and where is the retry loop for that? | 17:44 |
| pradk | but since docker config doesnt use the puppet stuff.. we need to figure out how to | 17:44 |
| gfidente | ah ok I see what you mean | 17:44 |
| pradk | that would be: | 17:44 |
| gfidente | make it a bash retry | 17:44 |
| gfidente | in the command: section | 17:44 |
| pradk | https://github.com/openstack/tripleo-heat-templates/blob/master/docker/services/ceilometer-agent-central.yaml#L118 | 17:44 |
| pradk | yea | 17:45 |
| gfidente | seems easiest option | 17:45 |
| pradk | bash retry.. hmm yea i guess thats better than puppet exec | 17:45 |
| pradk | mwhahaha, wdyt ^^ ? | 17:45 |
| gfidente | for n in 1..10; do command || sleep 3s; done | 17:45 |
| gfidente | I think it's our only option right now | 17:45 |
| gfidente | as it won't kill and restart the container | 17:46 |
| *** garyk has quit IRC | 17:47 | |
| *** trown|lunch is now known as trown | 17:51 | |
| *** dbecker has joined #tripleo | 17:52 | |
| gfidente | pradk tty next week! | 17:52 |
| gfidente | thanks! | 17:52 |
| pradk | have a good weekend | 17:53 |
| *** yamahata has quit IRC | 17:53 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: openstack-heat-templates: fix deprecation path https://review.openstack.org/490038 | 17:55 |
| mwhahaha | Yea a bash loop is probably bettet | 17:57 |
| openstackgerrit | John Trowbridge proposed openstack/tripleo-quickstart-extras master: Add --tag parameter to image prepare command https://review.openstack.org/492456 | 18:00 |
| *** dpawar has quit IRC | 18:00 | |
| openstackgerrit | John Trowbridge proposed openstack-infra/tripleo-ci master: WIP: containers periodic test https://review.openstack.org/475747 | 18:01 |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates master: Correct gnocchi-upgrade command quotes https://review.openstack.org/491826 | 18:04 |
| *** gbarros has quit IRC | 18:05 | |
| pradk | mwhahaha, gfidente, ^^ lets see if this helps ceilo-upgarde | 18:06 |
| gfidente | voted, leaving for today though | 18:06 |
| gfidente | thanks pradk | 18:07 |
| *** gfidente has quit IRC | 18:07 | |
| *** ooolpbot has joined #tripleo | 18:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709066 | 18:10 |
| openstack | Launchpad bug 1709066 in tripleo "[CI] Upgrade job down: AnsibleUndefinedVariable: 'docker_registry_namespace' is undefined" [Critical,In progress] - Assigned to Alex Schultz (alex-schultz) | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709327 | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709630 | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710112 | 18:10 |
| *** ooolpbot has quit IRC | 18:10 | |
| openstack | Launchpad bug 1709327 in tripleo "CI: extremely long times of overcloud deploy in multinode jobs" [Critical,Triaged] | 18:10 |
| openstack | Launchpad bug 1709630 in tripleo "docker run gnocchi_db_sync is failing in gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 18:10 |
| openstack | Launchpad bug 1710112 in tripleo "Upgrade failing with AnsibleUndefinedVariable: 'deploy_composable_scenario' is undefined" [Critical,In progress] - Assigned to Jiří Stránský (jistr) | 18:10 |
| *** leitan has quit IRC | 18:12 | |
| *** dobson has quit IRC | 18:23 | |
| *** yamahata has joined #tripleo | 18:25 | |
| *** dobson has joined #tripleo | 18:26 | |
| mwhahaha | pradk: commented | 18:26 |
| *** dbecker has quit IRC | 18:26 | |
| *** chem has quit IRC | 18:28 | |
| *** funzo has joined #tripleo | 18:32 | |
| *** sshnaidm|off has quit IRC | 18:37 | |
| *** funzo has quit IRC | 18:37 | |
| openstackgerrit | Merged openstack/instack-undercloud master: Remove docker --iptables=false on the undercloud https://review.openstack.org/491824 | 18:44 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/instack stable/pike: Updated from global requirements https://review.openstack.org/493131 | 18:58 |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates master: Correct gnocchi-upgrade command quotes https://review.openstack.org/491826 | 19:07 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Add script to create tripleo-admin on deployed servers https://review.openstack.org/490458 | 19:07 |
| openstackgerrit | Dan Sneddon proposed openstack/tripleo-heat-templates master: Remove ipv6 specific network templates https://review.openstack.org/492550 | 19:08 |
| *** ooolpbot has joined #tripleo | 19:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709066 | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709327 | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709630 | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710112 | 19:10 |
| openstack | Launchpad bug 1709066 in tripleo "[CI] Upgrade job down: AnsibleUndefinedVariable: 'docker_registry_namespace' is undefined" [Critical,In progress] - Assigned to Alex Schultz (alex-schultz) | 19:10 |
| *** ooolpbot has quit IRC | 19:10 | |
| openstack | Launchpad bug 1709327 in tripleo "CI: extremely long times of overcloud deploy in multinode jobs" [Critical,Triaged] | 19:10 |
| openstack | Launchpad bug 1709630 in tripleo "docker run gnocchi_db_sync is failing in gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 19:10 |
| openstack | Launchpad bug 1710112 in tripleo "Upgrade failing with AnsibleUndefinedVariable: 'deploy_composable_scenario' is undefined" [Critical,In progress] - Assigned to Jiří Stránský (jistr) | 19:10 |
| *** leitan has joined #tripleo | 19:10 | |
| pabelanger | mwhahaha: merging is spot on right now. Current issues are limited to rax-ord, we're replacing the mirror with a larger flavor think we are getting networking throttled and we are 100% for proxy cache. | 19:10 |
| fultonj | thanks for improving ci | 19:12 |
| *** morazi has quit IRC | 19:12 | |
| *** salmankhan has quit IRC | 19:19 | |
| *** jlabarre has quit IRC | 19:26 | |
| *** jlabarre has joined #tripleo | 19:26 | |
| *** leitan_ has joined #tripleo | 19:31 | |
| *** leitan has quit IRC | 19:31 | |
| *** limao has quit IRC | 19:32 | |
| *** limao has joined #tripleo | 19:32 | |
| *** leitan_ has quit IRC | 19:33 | |
| *** funzo has joined #tripleo | 19:33 | |
| *** jcoufal has quit IRC | 19:33 | |
| *** limao has quit IRC | 19:37 | |
| *** lblanchard has quit IRC | 19:37 | |
| *** funzo has quit IRC | 19:38 | |
| *** jlabarre has quit IRC | 19:44 | |
| *** jlabarre has joined #tripleo | 19:45 | |
| *** jkilpatr has quit IRC | 19:49 | |
| *** leitan has joined #tripleo | 19:50 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/ocata: Use number for KeystoneCronTokenFlushMaxDelay instead of string https://review.openstack.org/492023 | 19:53 |
| *** dougbtv_ has joined #tripleo | 19:56 | |
| *** jlabarre has quit IRC | 19:56 | |
| *** jlabarre has joined #tripleo | 19:56 | |
| *** rwsu has quit IRC | 19:57 | |
| *** dougbtv has quit IRC | 19:59 | |
| *** leitan_ has joined #tripleo | 20:00 | |
| *** sshnaidm|off has joined #tripleo | 20:02 | |
| *** leitan has quit IRC | 20:03 | |
| *** bandini has quit IRC | 20:04 | |
| *** bandini has joined #tripleo | 20:06 | |
| *** ansmith has quit IRC | 20:09 | |
| *** ooolpbot has joined #tripleo | 20:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709066 | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709327 | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709630 | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710112 | 20:10 |
| openstack | Launchpad bug 1709066 in tripleo "[CI] Upgrade job down: AnsibleUndefinedVariable: 'docker_registry_namespace' is undefined" [Critical,In progress] - Assigned to Alex Schultz (alex-schultz) | 20:10 |
| *** ooolpbot has quit IRC | 20:10 | |
| openstack | Launchpad bug 1709327 in tripleo "CI: extremely long times of overcloud deploy in multinode jobs" [Critical,Triaged] | 20:10 |
| openstack | Launchpad bug 1709630 in tripleo "docker run gnocchi_db_sync is failing in gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 20:10 |
| openstack | Launchpad bug 1710112 in tripleo "Upgrade failing with AnsibleUndefinedVariable: 'deploy_composable_scenario' is undefined" [Critical,In progress] - Assigned to Jiří Stránský (jistr) | 20:10 |
| *** MVenesio has quit IRC | 20:14 | |
| *** slagle has quit IRC | 20:15 | |
| *** catintheroof has quit IRC | 20:15 | |
| *** dparkes has joined #tripleo | 20:20 | |
| *** oidgar has quit IRC | 20:20 | |
| *** morazi has joined #tripleo | 20:24 | |
| *** jkilpatr has joined #tripleo | 20:25 | |
| *** jlabarre has quit IRC | 20:28 | |
| *** jlabarre has joined #tripleo | 20:28 | |
| openstackgerrit | Ben Nemec proposed openstack/tripleo-heat-templates master: Use ssl-enabled keystone admin endpoint by default https://review.openstack.org/426348 | 20:32 |
| mwhahaha | pabelanger: citycloud is still showing some problems as well, http://logs.openstack.org/33/481233/3/gate/gate-tripleo-ci-centos-7-scenario004-multinode-oooq-puppet/09a3368/logs/undercloud/home/jenkins/undercloud_install.log.txt.gz#_2017-08-11_16_43_19 | 20:32 |
| pabelanger | mwhahaha: ya, I did see that, have to look at apache logs still. It would be helpful is we could see the output from yum attempt in puppet, wonder if DNS issue or just 404 | 20:33 |
| *** funzo has joined #tripleo | 20:34 | |
| mwhahaha | pabelanger: that's the output, not sure if yum logs anything else | 20:34 |
| pabelanger | mwhahaha: ya, I think puppet is absorbing all the other bits about the failure. No issue, just a little more work to figure out why | 20:35 |
| pabelanger | I _think_ it is an IO issue on mirror | 20:35 |
| *** ccamacho has quit IRC | 20:35 | |
| mwhahaha | either way failures have been way less at least | 20:36 |
| pabelanger | but not at bad as rax-ord | 20:36 |
| *** jlabarre has quit IRC | 20:36 | |
| pabelanger | or, we are getting throttled | 20:36 |
| pabelanger | Yup, much better now | 20:36 |
| mwhahaha | rax-ord was always over subscribed | 20:36 |
| *** salmankhan has joined #tripleo | 20:37 | |
| *** funzo has quit IRC | 20:38 | |
| *** weshay_doc is now known as weshay | 20:39 | |
| *** itlinux has quit IRC | 20:46 | |
| *** leitan_ has quit IRC | 20:49 | |
| openstackgerrit | Ben Nemec proposed openstack/tripleo-docs master: Drop stable branch section from earlier than Newton https://review.openstack.org/493232 | 20:51 |
| *** cylopez has joined #tripleo | 20:51 | |
| openstackgerrit | Ben Nemec proposed openstack/tripleo-docs master: Drop stable branch section from earlier than Newton https://review.openstack.org/493232 | 20:52 |
| bnemec | Found another one. | 20:53 |
| *** dsavineau has quit IRC | 20:54 | |
| *** rhallisey has quit IRC | 20:54 | |
| *** salmankhan has quit IRC | 20:55 | |
| *** gcerami has quit IRC | 20:56 | |
| *** fultonj has quit IRC | 20:57 | |
| *** abishop has quit IRC | 21:00 | |
| *** bfournie has quit IRC | 21:01 | |
| *** jistr is now known as jistr|off | 21:02 | |
| *** dougbtv__ has joined #tripleo | 21:02 | |
| *** dougbtv_ has quit IRC | 21:06 | |
| *** trown is now known as trown|outtypewww | 21:07 | |
| *** dprince has quit IRC | 21:09 | |
| *** ooolpbot has joined #tripleo | 21:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709066 | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709327 | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709630 | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710112 | 21:10 |
| openstack | Launchpad bug 1709066 in tripleo "[CI] Upgrade job down: AnsibleUndefinedVariable: 'docker_registry_namespace' is undefined" [Critical,In progress] - Assigned to Alex Schultz (alex-schultz) | 21:10 |
| *** ooolpbot has quit IRC | 21:10 | |
| openstack | Launchpad bug 1709327 in tripleo "CI: extremely long times of overcloud deploy in multinode jobs" [Critical,Triaged] | 21:10 |
| openstack | Launchpad bug 1709630 in tripleo "docker run gnocchi_db_sync is failing in gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 21:10 |
| openstack | Launchpad bug 1710112 in tripleo "Upgrade failing with AnsibleUndefinedVariable: 'deploy_composable_scenario' is undefined" [Critical,In progress] - Assigned to Jiří Stránský (jistr) | 21:10 |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart master: Change devmode to deploy containerized services by default https://review.openstack.org/472412 | 21:11 |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: Make container varaibles defined for upgrades https://review.openstack.org/491463 | 21:20 |
| openstackgerrit | Merged openstack/tripleo-common stable/newton: Add skip_deploy_identifier https://review.openstack.org/489813 | 21:20 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: TLS everywhere: Configure CA for mongodb https://review.openstack.org/492992 | 21:21 |
| *** ansmith has joined #tripleo | 21:24 | |
| *** atoth has quit IRC | 21:25 | |
| *** MVenesio has joined #tripleo | 21:33 | |
| *** funzo has joined #tripleo | 21:35 | |
| openstackgerrit | Ben Nemec proposed openstack/tripleo-heat-templates master: Add ability to generate an environment index https://review.openstack.org/491925 | 21:35 |
| openstackgerrit | Ben Nemec proposed openstack/tripleo-heat-templates master: Begin adding environments with all params for a service https://review.openstack.org/475924 | 21:35 |
| openstackgerrit | Ben Nemec proposed openstack/tripleo-heat-templates master: Add ServiceData to hidden params https://review.openstack.org/493244 | 21:35 |
| openstackgerrit | Ben Nemec proposed openstack/tripleo-heat-templates master: Update generated ssl environments https://review.openstack.org/493245 | 21:35 |
| openstackgerrit | Ben Nemec proposed openstack/tripleo-heat-templates master: Convert enable-internal-tls.yaml to be generated https://review.openstack.org/493246 | 21:35 |
| *** dsneddon has joined #tripleo | 21:38 | |
| *** funzo has quit IRC | 21:40 | |
| openstackgerrit | Attila Darazs proposed openstack/tripleo-quickstart-extras master: GATE CHECK for quickstart-extras https://review.openstack.org/472607 | 22:00 |
| *** shreshtha_ has joined #tripleo | 22:01 | |
| *** shreshtha has quit IRC | 22:03 | |
| *** agopi|away has quit IRC | 22:05 | |
| pabelanger | good day for gate pipeline, didn't see anything too crazy fail. And things that did, are being worked on in openstack-infra | 22:09 |
| *** ooolpbot has joined #tripleo | 22:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709327 | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709630 | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710112 | 22:10 |
| *** ooolpbot has quit IRC | 22:10 | |
| openstack | Launchpad bug 1709327 in tripleo "CI: extremely long times of overcloud deploy in multinode jobs" [Critical,Triaged] | 22:10 |
| openstack | Launchpad bug 1709630 in tripleo "docker run gnocchi_db_sync is failing in gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 22:10 |
| openstack | Launchpad bug 1710112 in tripleo "Upgrade failing with AnsibleUndefinedVariable: 'deploy_composable_scenario' is undefined" [Critical,In progress] - Assigned to Jiří Stránský (jistr) | 22:10 |
| *** jmelvin has quit IRC | 22:11 | |
| *** gbarros has joined #tripleo | 22:12 | |
| openstackgerrit | Merged openstack/tripleo-docs master: Fix some grammar and add a few notes to docs. https://review.openstack.org/490585 | 22:12 |
| *** MVenesio_ has joined #tripleo | 22:18 | |
| *** MVenesio has quit IRC | 22:22 | |
| pabelanger | MERGE ALL THE THINGS | 22:23 |
| mwhahaha | working on it | 22:25 |
| mwhahaha | if we could get the puppet integration jobs to stop timing out | 22:25 |
| mwhahaha | then we could merge everything :D | 22:25 |
| mwhahaha | or https://review.openstack.org/#/c/492544/ which ever | 22:26 |
| *** ecerquei_ has quit IRC | 22:28 | |
| *** ecerquei has joined #tripleo | 22:28 | |
| *** ecerquei has quit IRC | 22:29 | |
| *** leitan has joined #tripleo | 22:30 | |
| *** funzo has joined #tripleo | 22:36 | |
| *** rlandy has quit IRC | 22:38 | |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-heat-templates stable/newton: Adds SSH Banner text into sshd_config https://review.openstack.org/492152 | 22:40 |
| *** funzo has quit IRC | 22:40 | |
| pabelanger | mwhahaha: +3 | 22:40 |
| pabelanger | mwhahaha: what is puppet integration timing out on? | 22:40 |
| mwhahaha | processing | 22:41 |
| mwhahaha | it dies mid tempest run | 22:41 |
| mwhahaha | so it's just running slowly and unfortunately we don't get logs when it times out | 22:41 |
| mwhahaha | i think it's just the slow clouds because it's not centos specific, we get it happening on xenial tests as well | 22:42 |
| *** kbyrne has joined #tripleo | 22:44 | |
| pabelanger | wonder if we are spending time downloading some non-cache things | 22:45 |
| pabelanger | next week we should update it to use gemrc file | 22:46 |
| mwhahaha | we've been using the mirrors downstream for a long time | 22:46 |
| pabelanger | and see if that saves some time off | 22:46 |
| mwhahaha | p-o-i uses packaged puppet | 22:46 |
| mwhahaha | so when we get some long run logs we can take a look | 22:46 |
| pabelanger | k, no puppet modules outside of openstack? | 22:46 |
| mwhahaha | we are pulling modules from github | 22:46 |
| pabelanger | okay, should check that too | 22:47 |
| mwhahaha | but puppet itself is packaged | 22:47 |
| pabelanger | ya | 22:47 |
| mwhahaha | i've seen github timeouts but what's taking a long time is the puppet run itself | 22:47 |
| pabelanger | if on infracloud, hitting github could explain some slowness, we had networking issues there this week | 22:47 |
| pabelanger | Ya | 22:47 |
| pabelanger | we did disable citycloud-lon1 today | 22:47 |
| pabelanger | so, that will help | 22:47 |
| pabelanger | infracloud continues to be some what of an issue | 22:48 |
| pabelanger | I'll look into that next week and see if there is a pattern | 22:48 |
| mwhahaha | yea infracloud or cityclound-lon1 are usually the ones that it times out on | 22:48 |
| pabelanger | we can also turn on elastic-recheck for puppet too I think | 22:48 |
| mwhahaha | yea | 22:48 |
| *** MVenesio_ has quit IRC | 22:51 | |
| *** toure is now known as toure|gone | 23:10 | |
| *** ooolpbot has joined #tripleo | 23:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709327 | 23:10 |
| openstack | Launchpad bug 1709327 in tripleo "CI: extremely long times of overcloud deploy in multinode jobs" [Critical,Triaged] | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1709630 | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1710112 | 23:10 |
| *** ooolpbot has quit IRC | 23:10 | |
| openstack | Launchpad bug 1709630 in tripleo "docker run gnocchi_db_sync is failing in gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container" [Critical,In progress] - Assigned to Giulio Fidente (gfidente) | 23:10 |
| openstack | Launchpad bug 1710112 in tripleo "Upgrade failing with AnsibleUndefinedVariable: 'deploy_composable_scenario' is undefined" [Critical,In progress] - Assigned to Jiří Stránský (jistr) | 23:10 |
| *** dparkes has quit IRC | 23:26 | |
| *** limao has joined #tripleo | 23:34 | |
| *** funzo has joined #tripleo | 23:36 | |
| *** bfournie has joined #tripleo | 23:39 | |
| *** limao has quit IRC | 23:40 | |
| *** bfournie has quit IRC | 23:40 | |
| *** funzo has quit IRC | 23:41 | |
| *** zzzeek has quit IRC | 23:43 | |
| *** zzzeek has joined #tripleo | 23:44 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!