| *** dparkes has quit IRC | 00:00 | |
| *** fragatina has quit IRC | 00:04 | |
| *** tbonds has quit IRC | 00:06 | |
| *** fragatina has joined #tripleo | 00:06 | |
| *** ooolpbot has joined #tripleo | 00:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 00:10 |
|---|---|---|
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757556 | 00:10 |
| *** ooolpbot has quit IRC | 00:10 | |
| openstack | Launchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged] | 00:10 |
| *** fragatina has quit IRC | 00:10 | |
| *** rlandy is now known as rlandy|bbl | 00:11 | |
| *** liverpooler has joined #tripleo | 00:11 | |
| *** dmacpher has quit IRC | 00:13 | |
| *** openstack has joined #tripleo | 00:14 | |
| *** ChanServ sets mode: +o openstack | 00:14 | |
| openstackgerrit | Marius Cornea proposed openstack/tripleo-upgrade master: WIP: Add support for per role FFU upgrade https://review.openstack.org/556687 | 00:19 |
| *** gvrangan has joined #tripleo | 00:20 | |
| openstackgerrit | Sai Ram Peesapati proposed openstack/tripleo-heat-templates master: Resource name correction https://review.openstack.org/557101 | 00:21 |
| openstackgerrit | Marius Cornea proposed openstack/tripleo-upgrade master: WIP: Add support for per role FFU upgrade https://review.openstack.org/556687 | 00:21 |
| *** abishop has quit IRC | 00:30 | |
| *** pmannidi has joined #tripleo | 00:33 | |
| *** yamahata has joined #tripleo | 00:38 | |
| *** fragatina has joined #tripleo | 00:39 | |
| *** fragatina has quit IRC | 00:43 | |
| *** yamahata has quit IRC | 00:49 | |
| *** dmacpher has joined #tripleo | 01:07 | |
| *** itlinux has joined #tripleo | 01:07 | |
| *** alee has joined #tripleo | 01:09 | |
| *** ooolpbot has joined #tripleo | 01:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757556 | 01:10 |
| openstack | Launchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged] | 01:10 |
| *** ooolpbot has quit IRC | 01:10 | |
| *** alee_ has quit IRC | 01:10 | |
| *** psahoo has joined #tripleo | 01:11 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Increase time for log collection in OVB https://review.openstack.org/556884 | 01:11 |
| *** gyankum has joined #tripleo | 01:12 | |
| mwhahaha | EmilienM: weshay: so i think the httpd count might account for the higher load that was previously mentioned for baremetal vs containers. we'll see what the tempest bits come back with | 01:13 |
| *** gvrangan has quit IRC | 01:13 | |
| *** itlinux has quit IRC | 01:15 | |
| *** PagliaccisCloud has quit IRC | 01:19 | |
| *** PagliaccisCloud has joined #tripleo | 01:20 | |
| *** cshastri has joined #tripleo | 01:22 | |
| *** gvrangan has joined #tripleo | 01:26 | |
| *** tbonds has joined #tripleo | 01:32 | |
| *** gvrangan has quit IRC | 01:35 | |
| *** gvrangan has joined #tripleo | 01:44 | |
| *** gvrangan has quit IRC | 01:48 | |
| *** atoth has quit IRC | 01:54 | |
| openstackgerrit | Merged openstack/os-net-config stable/queens: Don't restart ivs/nfvswitch in os-net-config https://review.openstack.org/556998 | 02:01 |
| *** ykarel|away has joined #tripleo | 02:02 | |
| *** cshastri has quit IRC | 02:08 | |
| *** ooolpbot has joined #tripleo | 02:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757556 | 02:10 |
| openstack | Launchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged] | 02:10 |
| *** ooolpbot has quit IRC | 02:10 | |
| *** ramishra has joined #tripleo | 02:11 | |
| *** psachin has joined #tripleo | 02:12 | |
| *** masco has joined #tripleo | 02:14 | |
| *** ykarel|away has quit IRC | 02:14 | |
| *** ykarel|away has joined #tripleo | 02:16 | |
| *** cshastri has joined #tripleo | 02:18 | |
| *** skramaja has joined #tripleo | 02:31 | |
| *** ykarel|away has quit IRC | 02:34 | |
| *** ccamacho|PTO has quit IRC | 02:38 | |
| *** jaganathan has quit IRC | 02:41 | |
| *** jaganathan has joined #tripleo | 02:42 | |
| weshay | mwhahaha, geez my controller load average: 22.79, 22.60, 22.46 | 02:42 |
| weshay | httpd and openvswitch | 02:42 |
| weshay | w/ https://review.openstack.org/#/c/557073/ | 02:43 |
| *** fragatina has joined #tripleo | 02:44 | |
| *** fragatin_ has joined #tripleo | 02:46 | |
| *** fragatina has quit IRC | 02:48 | |
| *** anilvenkata has quit IRC | 02:50 | |
| *** rbrady has joined #tripleo | 02:51 | |
| openstackgerrit | wes hayutin proposed openstack-infra/tripleo-ci master: featureset20 requires larger instances https://review.openstack.org/557015 | 02:54 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart master: DO NOT MERGE https://review.openstack.org/557116 | 02:56 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart-extras master: allow full tempest fs020 more resources https://review.openstack.org/556965 | 02:59 |
| *** dpawar has joined #tripleo | 03:03 | |
| *** dpawar has quit IRC | 03:03 | |
| *** dpawar has joined #tripleo | 03:04 | |
| *** ooolpbot has joined #tripleo | 03:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757556 | 03:10 |
| *** ooolpbot has quit IRC | 03:10 | |
| openstack | Launchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged] | 03:10 |
| *** ccamacho has joined #tripleo | 03:11 | |
| *** itlinux has joined #tripleo | 03:18 | |
| openstackgerrit | Vincent S. Cojot proposed openstack/tripleo-heat-templates master: Santize shell syntax a bit https://review.openstack.org/557133 | 03:23 |
| *** janki has joined #tripleo | 03:32 | |
| openstackgerrit | Marius Cornea proposed openstack/tripleo-upgrade master: WIP: Add support for per role FFU upgrade https://review.openstack.org/556687 | 03:37 |
| *** cshastri has quit IRC | 03:45 | |
| *** cshastri has joined #tripleo | 03:45 | |
| *** kbyrne has quit IRC | 03:51 | |
| *** jaosorior has joined #tripleo | 03:56 | |
| *** kbyrne has joined #tripleo | 03:56 | |
| *** udesale has joined #tripleo | 03:57 | |
| *** sanjayu has quit IRC | 04:06 | |
| *** sshnaidm_ has joined #tripleo | 04:08 | |
| *** sshnaidm has quit IRC | 04:08 | |
| *** pdeore has joined #tripleo | 04:09 | |
| *** pdeore has quit IRC | 04:09 | |
| *** ooolpbot has joined #tripleo | 04:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757556 | 04:10 |
| *** ooolpbot has quit IRC | 04:10 | |
| openstack | Launchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged] | 04:10 |
| *** ykarel|away has joined #tripleo | 04:15 | |
| *** shreshtha has joined #tripleo | 04:22 | |
| jaganathan | d0ugal, hi, when i execute wrokflow separately working fine after updating the workbook. but seems invoking old workbook when using deploy command. | 04:24 |
| jaganathan | d0ugal, any idea? | 04:24 |
| *** psachin has quit IRC | 04:25 | |
| *** masco has quit IRC | 04:26 | |
| *** sshnaidm_ has quit IRC | 04:32 | |
| *** cshastri has quit IRC | 04:39 | |
| *** psachin has joined #tripleo | 04:43 | |
| *** gkadam has joined #tripleo | 04:44 | |
| openstackgerrit | Sai Ram Peesapati proposed openstack/tripleo-heat-templates master: Add Nuage Docker Service https://review.openstack.org/557149 | 04:44 |
| *** links has joined #tripleo | 04:46 | |
| *** sshnaidm_ has joined #tripleo | 04:48 | |
| *** ratailor has joined #tripleo | 04:48 | |
| *** ratailor_ has joined #tripleo | 04:52 | |
| *** cshastri has joined #tripleo | 04:54 | |
| *** cshastri has joined #tripleo | 04:55 | |
| *** ratailor has quit IRC | 04:57 | |
| *** dpawar_ has joined #tripleo | 04:57 | |
| *** dpawar_ has quit IRC | 04:58 | |
| *** udesale has quit IRC | 04:58 | |
| *** udesale has joined #tripleo | 04:59 | |
| *** anilvenkata has joined #tripleo | 05:00 | |
| *** dpawar has quit IRC | 05:00 | |
| *** dmacpher_ has joined #tripleo | 05:02 | |
| *** masco has joined #tripleo | 05:02 | |
| *** dmacpher has quit IRC | 05:04 | |
| *** gkadam has quit IRC | 05:07 | |
| *** sshnaidm_ has quit IRC | 05:09 | |
| *** ooolpbot has joined #tripleo | 05:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757556 | 05:10 |
| openstack | Launchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged] | 05:10 |
| *** ooolpbot has quit IRC | 05:10 | |
| *** pdeore has joined #tripleo | 05:15 | |
| *** agurenko has joined #tripleo | 05:17 | |
| *** dpawar has joined #tripleo | 05:18 | |
| *** pgadiya has joined #tripleo | 05:19 | |
| *** moshele has joined #tripleo | 05:22 | |
| *** sshnaidm has joined #tripleo | 05:24 | |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: Add option to specify the timeout for reproducer script https://review.openstack.org/555251 | 05:26 |
| *** dmacpher__ has joined #tripleo | 05:32 | |
| *** dmacpher_ has quit IRC | 05:36 | |
| *** jfrancoa has joined #tripleo | 05:36 | |
| *** karthiks has quit IRC | 05:40 | |
| *** rlandy|bbl has quit IRC | 05:43 | |
| *** jtomasek has joined #tripleo | 05:43 | |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-quickstart master: Update overcloud-prep-containers script name. https://review.openstack.org/556860 | 05:48 |
| *** avivgta has quit IRC | 05:48 | |
| itlinux | hello all, in a push for the overcloud, does the cinder-nic template gets to be overwritten by the controller or by the compute? I have setup the compute with two bonds. Thanks for the advice | 05:52 |
| openstackgerrit | zenghui.shi proposed openstack/tripleo-heat-templates master: Add condition to ovs run during upgrade. https://review.openstack.org/553926 | 05:52 |
| *** gfidente has joined #tripleo | 05:54 | |
| *** gfidente has quit IRC | 05:54 | |
| *** gfidente has joined #tripleo | 05:54 | |
| *** marios has joined #tripleo | 05:57 | |
| *** cshastri has quit IRC | 05:57 | |
| *** masco has quit IRC | 06:00 | |
| *** dpawar has quit IRC | 06:01 | |
| *** dpawar has joined #tripleo | 06:01 | |
| *** karthiks has joined #tripleo | 06:03 | |
| *** khyr0n has joined #tripleo | 06:05 | |
| *** dpawar has quit IRC | 06:05 | |
| openstackgerrit | zenghui.shi proposed openstack/tripleo-heat-templates master: Ensure openvswitch user exist before package update https://review.openstack.org/557155 | 06:08 |
| *** cshastri has joined #tripleo | 06:09 | |
| *** ooolpbot has joined #tripleo | 06:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757556 | 06:10 |
| *** ooolpbot has quit IRC | 06:10 | |
| openstack | Launchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged] | 06:10 |
| *** moshele has quit IRC | 06:14 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/python-tripleoclient master: Remove repeated code from TLS related bits in undercloud_config https://review.openstack.org/557159 | 06:17 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/python-tripleoclient master: Enable TLS by default in the containerized undercloud https://review.openstack.org/557160 | 06:17 |
| openstackgerrit | Jose Luis Franco proposed openstack-infra/tripleo-ci master: Add multinode-overcloud-update playbook to run list https://review.openstack.org/547058 | 06:22 |
| *** gfidente has quit IRC | 06:24 | |
| *** florianf has joined #tripleo | 06:24 | |
| *** dpawar has joined #tripleo | 06:26 | |
| *** udesale_ has joined #tripleo | 06:28 | |
| *** jaosorior has quit IRC | 06:29 | |
| *** ykarel_ has joined #tripleo | 06:30 | |
| *** udesale has quit IRC | 06:30 | |
| *** udesale__ has joined #tripleo | 06:30 | |
| *** yprokule has joined #tripleo | 06:30 | |
| *** dbecker has quit IRC | 06:32 | |
| *** jtomasek has quit IRC | 06:32 | |
| *** ykarel|away has quit IRC | 06:32 | |
| *** udesale_ has quit IRC | 06:33 | |
| *** pcaruana has joined #tripleo | 06:38 | |
| openstackgerrit | Dan Macpherson proposed openstack/tripleo-docs master: WIP: Add FFU docs https://review.openstack.org/549892 | 06:39 |
| *** ccamacho has quit IRC | 06:41 | |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: Switch to tempest run cli for running tempest tests https://review.openstack.org/556014 | 06:43 |
| *** ihrachys has quit IRC | 06:43 | |
| *** karthiks has quit IRC | 06:43 | |
| *** dbecker has joined #tripleo | 06:45 | |
| *** ccamacho has joined #tripleo | 06:59 | |
| *** karthiks has joined #tripleo | 07:02 | |
| *** ykarel__ has joined #tripleo | 07:03 | |
| openstackgerrit | zenghui.shi proposed openstack/tripleo-heat-templates stable/newton: Ensure openvswitch user exist before package update https://review.openstack.org/557164 | 07:04 |
| *** aufi has joined #tripleo | 07:06 | |
| *** ykarel_ has quit IRC | 07:06 | |
| *** jaosorior has joined #tripleo | 07:10 | |
| *** ooolpbot has joined #tripleo | 07:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757556 | 07:10 |
| *** ooolpbot has quit IRC | 07:10 | |
| openstack | Launchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged] | 07:10 |
| jaosorior | bandini: could you check this out https://review.openstack.org/#/c/554836/ | 07:11 |
| *** avivgta has joined #tripleo | 07:12 | |
| *** ykarel__ is now known as ykarel | 07:13 | |
| *** rcernin has quit IRC | 07:15 | |
| *** ebarrera has joined #tripleo | 07:15 | |
| *** ebarrera has quit IRC | 07:19 | |
| *** ebarrera has joined #tripleo | 07:19 | |
| *** tzumainn has quit IRC | 07:22 | |
| *** mdnadeem has joined #tripleo | 07:22 | |
| *** tosky has joined #tripleo | 07:23 | |
| Tengu | hello! small question: does the undercloud machine hostname impact the overcloud heat stack? I.e. if I decide to change the undercloud hostname, does it change anything for tripleo? | 07:24 |
| *** tesseract has joined #tripleo | 07:25 | |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-quickstart-extras master: Create import-role role. https://review.openstack.org/553492 | 07:30 |
| *** cylopez has joined #tripleo | 07:33 | |
| *** dmacpher__ has quit IRC | 07:33 | |
| *** cylopez1 has joined #tripleo | 07:34 | |
| *** gfidente has joined #tripleo | 07:36 | |
| *** gfidente has quit IRC | 07:36 | |
| *** gfidente has joined #tripleo | 07:36 | |
| *** holser__ has joined #tripleo | 07:37 | |
| *** cylopez has quit IRC | 07:37 | |
| *** shardy has joined #tripleo | 07:38 | |
| *** bogdando has joined #tripleo | 07:39 | |
| *** athomas has joined #tripleo | 07:43 | |
| *** udesale__ is now known as udesale | 07:43 | |
| *** skramaja is now known as skramaja|lunch | 07:46 | |
| *** mdnadeem has quit IRC | 07:51 | |
| *** cylopez1 has left #tripleo | 07:53 | |
| openstackgerrit | Attila Darazs proposed openstack/tripleo-quickstart-extras master: GATE CHECK for quickstart-extras https://review.openstack.org/472607 | 07:55 |
| *** lucas-afk is now known as lucasagomes | 07:56 | |
| *** mrunge has quit IRC | 07:58 | |
| *** mrunge has joined #tripleo | 08:01 | |
| *** pmannidi has quit IRC | 08:01 | |
| *** arxcruz|rover is now known as arxcruz|off | 08:03 | |
| *** jpich has joined #tripleo | 08:04 | |
| *** ktibi has joined #tripleo | 08:05 | |
| bogdando | o/ https://bugs.launchpad.net/tripleo/+bug/1759507 is it something new, or do we know a bz already? | 08:07 |
| openstack | Launchpad bug 1759507 in tripleo "containerized keystone intermittently fails undercloud: grpc: the connection is unavailable" [High,Triaged] | 08:07 |
| *** derekh has joined #tripleo | 08:08 | |
| *** ooolpbot has joined #tripleo | 08:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757556 | 08:10 |
| *** ooolpbot has quit IRC | 08:10 | |
| openstack | Launchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged] | 08:10 |
| *** mdnadeem has joined #tripleo | 08:11 | |
| *** gkadam has joined #tripleo | 08:19 | |
| *** akrivoka has joined #tripleo | 08:21 | |
| *** akane_ has joined #tripleo | 08:23 | |
| *** ccamacho has quit IRC | 08:23 | |
| *** rcernin has joined #tripleo | 08:24 | |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates master: WIP: Migrate Ceph deployment to external_deploy_task https://review.openstack.org/546966 | 08:36 |
| *** dpawar has quit IRC | 08:40 | |
| *** rcernin has quit IRC | 08:42 | |
| *** masco has joined #tripleo | 08:43 | |
| *** anande has joined #tripleo | 08:45 | |
| *** ccamacho has joined #tripleo | 08:46 | |
| *** chem|eod is now known as chem | 08:49 | |
| *** anande has quit IRC | 08:53 | |
| *** anande_ has joined #tripleo | 08:53 | |
| *** ccamacho has quit IRC | 08:56 | |
| *** anande_ is now known as anande | 08:56 | |
| openstackgerrit | Bogdan Dobrelya proposed openstack/python-tripleoclient master: Hard link http boot contents for Ironic https://review.openstack.org/556516 | 08:56 |
| *** udesale has quit IRC | 08:58 | |
| bogdando | folks, PTAL https://review.openstack.org/#/q/topic:containers-default+(status:open+OR+status:merged) | 09:00 |
| *** anande is now known as anande|afk | 09:01 | |
| jaosorior | bogdando: is the undercloud-upgrades job working? | 09:04 |
| bogdando | jaosorior: AFAIK, it should be. holser__^^ | 09:05 |
| bogdando | though not into the containerized UC yet | 09:05 |
| jaosorior | bogdando: could you check this out https://review.openstack.org/#/c/557159/1 ? | 09:05 |
| *** udesale has joined #tripleo | 09:05 | |
| bogdando | jaosorior, holser__: http://logs.openstack.org/59/557159/1/check/tripleo-ci-centos-7-undercloud-upgrades/e4a1cf3/logs/var/log/undercloud_upgrade.txt.gz | 09:09 |
| *** chem has quit IRC | 09:09 | |
| *** dpawar has joined #tripleo | 09:09 | |
| bogdando | Sergii knows something of those online migrations fixes | 09:09 |
| *** ooolpbot has joined #tripleo | 09:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757556 | 09:10 |
| openstack | Launchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged] | 09:10 |
| *** ooolpbot has quit IRC | 09:10 | |
| *** salmankhan has joined #tripleo | 09:10 | |
| *** chem has joined #tripleo | 09:10 | |
| holser__ | jaosorior, bogdando - it's a known bug I am producing a fix | 09:10 |
| *** shardy has quit IRC | 09:10 | |
| jaosorior | holser__: thanks | 09:11 |
| holser__ | jaosorior, give me 5 min | 09:11 |
| holser__ | I am writing commit message | 09:11 |
| holser__ | of fix | 09:11 |
| jaosorior | bogdando: anyway, would appreciate the reviews for these https://review.openstack.org/#/q/status:open+project:openstack/python-tripleoclient+branch:master+topic:public-tls-default | 09:11 |
| *** shardy has joined #tripleo | 09:12 | |
| bogdando | jaosorior: this brings undercloud security into new level! W/o heat installer and templates integration, would never be a thing. Well done, really. | 09:14 |
| *** skramaja|lunch is now known as skramaja | 09:14 | |
| bogdando | jaosorior: is this tested by undercloud-containers job?.. | 09:15 |
| jaosorior | bogdando: that's what I need to figure out next. | 09:15 |
| bogdando | :) | 09:15 |
| jaosorior | bogdando: any idea what fs configures that? | 09:15 |
| bogdando | for ovb, this enables TLS for UC AFAICT https://review.openstack.org/#/c/542556/85/config/general_config/featureset001.yml | 09:17 |
| bogdando | I can see no heat params adjusted, so is it defaults in t-h-t? | 09:17 |
| bogdando | let me check for that undercloud containers job as well | 09:17 |
| jaosorior | bogdando: that's awesome | 09:17 |
| openstackgerrit | Sergii Golovatiuk proposed openstack/instack-undercloud master: Update puppet-nova before running nova-manage https://review.openstack.org/557261 | 09:18 |
| holser__ | jaosorior - https://review.openstack.org/557261 | 09:18 |
| bogdando | jaosorior: note, ovb is not yet switched to UC containers (WIP) | 09:18 |
| *** waleedm has joined #tripleo | 09:18 | |
| *** akane_ has quit IRC | 09:20 | |
| jaosorior | bogdando: I wonder why we're trying to use undercloud deploy instead of using undercloud install --user-heat | 09:21 |
| jaosorior | that way it reads undercloud.conf, and it's a more straightforward way to upgrade to using containers, since we would then expect users to deploy with the same command. | 09:21 |
| jaosorior | it would then be a matter of changing --use-heat to default to true | 09:23 |
| holser__ | jaosorior EmilienM is working on that - so answer is WIP | 09:23 |
| jaosorior | ok | 09:23 |
| jaosorior | there probably is a reason, I was just wondering why | 09:23 |
| holser__ | time & priorities... | 09:24 |
| holser__ | but everything you say makes total sense to me | 09:24 |
| bogdando | jaosorior, holser__, EmilienM: is working on integrating --use-heat for instack upgrades | 09:25 |
| bogdando | jaosorior: any ideas how to figure out TLS deployment from the CI logs?.. | 09:25 |
| bogdando | via tht params passed prolly? | 09:26 |
| bogdando | jaosorior: http://git.openstack.org/cgit/openstack/tripleo-quickstart/tree/config/general_config/featureset027.yml here is the fs for that undercloud-containers job btw | 09:26 |
| bogdando | we need it undercloud_generate_service_certificate: true? | 09:27 |
| jaosorior | bogdando: yes | 09:28 |
| openstackgerrit | Michele Baldessari proposed openstack/puppet-pacemaker master: Fix up error code path https://review.openstack.org/557273 | 09:28 |
| bogdando | jaosorior: commented | 09:29 |
| bogdando | we prolly want only OVB fs001 testing it | 09:29 |
| *** kopecmartin has joined #tripleo | 09:30 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-quickstart master: Test containerized undercloud with TLS https://review.openstack.org/557276 | 09:31 |
| jaosorior | bogdando: lets give it a try ^^ | 09:31 |
| *** salmankhan has quit IRC | 09:32 | |
| *** anande|afk has quit IRC | 09:32 | |
| *** salmankhan has joined #tripleo | 09:35 | |
| *** itlinux has quit IRC | 09:36 | |
| *** dmacpher has joined #tripleo | 09:41 | |
| *** anande|afk has joined #tripleo | 09:43 | |
| *** anande|afk is now known as anande | 09:43 | |
| *** dtantsur|afk is now known as dtantsur | 09:46 | |
| openstackgerrit | Merged openstack/tripleo-common master: Refactor setting default CA https://review.openstack.org/554835 | 09:49 |
| openstackgerrit | Merged openstack/tripleo-quickstart master: Add ff-upg to featureset table https://review.openstack.org/555372 | 09:49 |
| *** anande has quit IRC | 09:50 | |
| *** anande has joined #tripleo | 09:50 | |
| *** ykarel is now known as ykarel|afk | 09:53 | |
| *** dpawar has quit IRC | 09:54 | |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-heat-templates master: Add purge to Nova cleanup cron. https://review.openstack.org/551966 | 09:55 |
| jaosorior | bogdando: could you check this one out as well https://review.openstack.org/#/c/554836/ ? | 09:55 |
| *** liverpooler has quit IRC | 09:59 | |
| *** ratailor_ is now known as rtailor | 10:00 | |
| *** anande has quit IRC | 10:00 | |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-quickstart master: Fix Fast Forward featureset documentation. https://review.openstack.org/556801 | 10:00 |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-quickstart master: Add missing Featureset037 documentation. https://review.openstack.org/556814 | 10:00 |
| *** anande has joined #tripleo | 10:00 | |
| *** adarazs_lunch is now known as adarazs | 10:03 | |
| *** panda|off is now known as panda | 10:04 | |
| *** jaganathan has quit IRC | 10:09 | |
| *** ykarel|afk is now known as ykarel | 10:09 | |
| *** ooolpbot has joined #tripleo | 10:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757556 | 10:10 |
| openstack | Launchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged] | 10:10 |
| *** ooolpbot has quit IRC | 10:10 | |
| *** dpawar has joined #tripleo | 10:11 | |
| *** dmacpher has quit IRC | 10:20 | |
| openstackgerrit | Jose Luis Franco proposed openstack-infra/tripleo-ci master: Default to docker.io instead of trunk.registry.rdoproject.org https://review.openstack.org/545873 | 10:21 |
| openstackgerrit | Jose Luis Franco proposed openstack-infra/tripleo-ci master: DNM: test P->Q Upgrade in experimental. https://review.openstack.org/545477 | 10:21 |
| *** cshastri has quit IRC | 10:28 | |
| *** sshnaidm has quit IRC | 10:29 | |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates master: Migrate Ceph deployment to external_deploy_task https://review.openstack.org/546966 | 10:31 |
| jaosorior | slagle: Hey, I took a quick look at the patch you posted for populating the undercloud's /etc/hosts, but haven't figured out what's wrong there. Have you? | 10:36 |
| *** moshele has joined #tripleo | 10:36 | |
| *** ansmith has quit IRC | 10:37 | |
| *** sshnaidm has joined #tripleo | 10:38 | |
| apetrich | yolanda, ping. Do you have a second to talk about https://bugs.launchpad.net/tripleo/+bug/1731413 | 10:41 |
| openstack | Launchpad bug 1731413 in tripleo "mistral db-sync error when upgrading undercloud from newton to ocata" [High,Triaged] - Assigned to Adriano Petrich (apetrich) | 10:41 |
| *** dpawar has quit IRC | 10:45 | |
| *** udesale_ has joined #tripleo | 10:46 | |
| *** khyr0n has quit IRC | 10:48 | |
| *** udesale has quit IRC | 10:49 | |
| *** dpawar has joined #tripleo | 10:49 | |
| *** atoth has joined #tripleo | 10:52 | |
| *** udesale_ has quit IRC | 10:52 | |
| *** sshnaidm has quit IRC | 10:55 | |
| *** psahoo has quit IRC | 10:56 | |
| *** gyankum has quit IRC | 10:56 | |
| *** dpawar has quit IRC | 10:59 | |
| *** ykarel_ has joined #tripleo | 11:03 | |
| *** mdnadeem_ has joined #tripleo | 11:03 | |
| *** pdeore_ has joined #tripleo | 11:03 | |
| *** gkadam_ has joined #tripleo | 11:03 | |
| *** pgadiya_ has joined #tripleo | 11:03 | |
| *** shreshtha_ has joined #tripleo | 11:03 | |
| *** ratailor_ has joined #tripleo | 11:03 | |
| *** chkumar246 has joined #tripleo | 11:03 | |
| *** pdeore has quit IRC | 11:04 | |
| *** rtailor has quit IRC | 11:04 | |
| *** mdnadeem has quit IRC | 11:04 | |
| *** ykarel has quit IRC | 11:04 | |
| *** links has quit IRC | 11:04 | |
| *** chandankumar has quit IRC | 11:04 | |
| *** pgadiya has quit IRC | 11:04 | |
| *** gkadam__ has joined #tripleo | 11:04 | |
| *** shreshtha has quit IRC | 11:04 | |
| *** links has joined #tripleo | 11:05 | |
| *** shrjoshi has joined #tripleo | 11:05 | |
| *** ratailor__ has joined #tripleo | 11:05 | |
| *** ykarel__ has joined #tripleo | 11:05 | |
| *** pdeore__ has joined #tripleo | 11:06 | |
| openstackgerrit | yogananth subramanian proposed openstack/tripleo-common master: [WIP] Build OVS system container https://review.openstack.org/557300 | 11:06 |
| *** gkadam has quit IRC | 11:07 | |
| *** sshnaidm has joined #tripleo | 11:07 | |
| *** shreshtha_ has quit IRC | 11:08 | |
| *** gkadam_ has quit IRC | 11:08 | |
| *** mdnadeem_ has quit IRC | 11:08 | |
| *** chkumar246 has quit IRC | 11:08 | |
| *** ratailor_ has quit IRC | 11:08 | |
| *** ykarel_ has quit IRC | 11:08 | |
| *** pgadiya_ has quit IRC | 11:08 | |
| *** pdeore_ has quit IRC | 11:08 | |
| *** jtomasek has joined #tripleo | 11:10 | |
| *** ooolpbot has joined #tripleo | 11:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757556 | 11:10 |
| *** ooolpbot has quit IRC | 11:10 | |
| openstack | Launchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged] | 11:10 |
| *** dpawar has joined #tripleo | 11:14 | |
| openstackgerrit | Rafael Folco proposed openstack/tripleo-quickstart-extras master: [tempestmail] Add instruction to run unit tests https://review.openstack.org/530504 | 11:14 |
| *** pkovar has joined #tripleo | 11:14 | |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates master: Migrate Ceph deployment to external_deploy_task https://review.openstack.org/546966 | 11:18 |
| *** panda is now known as panda|lunch | 11:18 | |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates master: Migrate Ceph deployment to external_deploy_task https://review.openstack.org/546966 | 11:19 |
| *** thrash|g0ne is now known as thrash | 11:19 | |
| *** chkumar246 has joined #tripleo | 11:20 | |
| openstackgerrit | Martin André proposed openstack/tripleo-docs master: Document toggling debug for containerized services https://review.openstack.org/557305 | 11:20 |
| *** mdnadeem_ has joined #tripleo | 11:21 | |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates stable/newton: Work around packaging issue in iptables-services. https://review.openstack.org/554647 | 11:22 |
| *** pkovar has quit IRC | 11:24 | |
| *** shardy is now known as shardy_lunch | 11:24 | |
| *** pkovar has joined #tripleo | 11:27 | |
| *** jtomasek has quit IRC | 11:28 | |
| *** dtantsur is now known as dtantsur|brb | 11:31 | |
| *** rfolco|off is now known as rfolco|ruck | 11:34 | |
| janki | Hi. How can I debug puppet hiera issue in deployment? I have a "log_mechanism" variable in THT. Default is file. I am changing it to "console" via env file. on overcloud node, hiera says the value is console but puppet manifest says "file" and the service is configured according to the manifest. | 11:34 |
| janki | mandre, marios shardy_lunch jaosorior ^ any idea? | 11:34 |
| *** pdeore__ has quit IRC | 11:36 | |
| *** shrjoshi has quit IRC | 11:38 | |
| jaosorior | janki: got links to the code? Which manifest and which variable in t-h-t? | 11:39 |
| jistr | marios / matbu / chem: has anyone tested upgrade converge recently? does it work for you? i'm getting just this: | 11:40 |
| jistr | Waiting for messages on queue 'tripleo' with no timeout. | 11:40 |
| jistr | socket is already closed. | 11:40 |
| jistr | and then nothing happens... | 11:40 |
| jistr | i'll dive in, just wondering if anyone hit this already so that i don't do duplicate work | 11:40 |
| jaosorior | jistr: I recently had the same issue (on update). I ended up redeploying the whole thing :/ | 11:41 |
| janki | jaosorior, https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/services/opendaylight-api.yaml#L69 I am changing log mechanism at this by https://github.com/openstack/tripleo-heat-templates/blob/master/environments/services-docker/neutron-opendaylight.yaml#L16 | 11:41 |
| marios | jistr: i had that at some point but there was a missing patch (lemme see if i can find) | 11:41 |
| marios | jistr: but generally converge hasn't passed for me yet. matbu is looking into it (i mean 'final bits for converge' ) | 11:41 |
| jaosorior | janki: ok, what about the manifest that sets that? | 11:41 |
| janki | jaosorior, puppet will then configure the logging mechanism to console (the code is merged and working) and ODL will log to console. However, after the deployment ODL still logs to file becasue the manifest on controller says it is file | 11:42 |
| marios | jistr: maybe this https://review.openstack.org/#/c/554237/ | 11:42 |
| janki | jaosorior, and wierd thing is this whole thing works fine in non-ha deployment. ODL logs to console. but in HA, ODL Logs to file despite being directed to log to consolde | 11:42 |
| *** leanderthal has joined #tripleo | 11:43 | |
| marios | jistr: (from my notes: still hanging ping matbu missing https://review.openstack.org/554237 & add to readme & rerun further converge fails for http://pastebin.test.redhat.com/567490 ) fwiw :) | 11:43 |
| janki | jaosorior, on overcloud controller nodes, hiers has the value of console | 11:43 |
| marios | jistr: but if i recall yeah it was just hanging w no output | 11:43 |
| openstackgerrit | Lukas Bezdicka proposed openstack/tripleo-heat-templates stable/queens: FFU: Fix gnocchi FFU tasks https://review.openstack.org/557309 | 11:43 |
| marios | jistr: and the stack never went to update in progress | 11:43 |
| jaosorior | janki: understood. Can you pass me the code that configures that in puppet? | 11:43 |
| jistr | marios: yup that sounds like my issue, thanks a lot! | 11:43 |
| jaosorior | janki: I saw the hiera setup in t-h-t. Just need to see the puppet bits. | 11:43 |
| *** dsariel has quit IRC | 11:43 | |
| marios | jistr: glad useful np | 11:44 |
| *** gkadam__ has quit IRC | 11:44 | |
| janki | jaosorior, https://github.com/opendaylight/integration-packaging-puppet-opendaylight/commit/cc8f1728228fbac279d91aab5837096907a5e828 | 11:44 |
| openstackgerrit | binhong.hua proposed openstack/tripleo-quickstart-extras master: move extra_tht_config_args to overcloud_deploy https://review.openstack.org/557310 | 11:44 |
| fultonj | gfidente: fyi, i'm looking at your updates to https://review.openstack.org/#/c/546966 | 11:45 |
| gfidente | fultonj believe it or not, it works | 11:45 |
| fultonj | awesome | 11:46 |
| gfidente | I am sure there are bugs, but ... ps17 stands up things | 11:46 |
| fultonj | i'll let you know how it works for me, and we'll see what ci says | 11:46 |
| gfidente | jistr ^^ maybe you want to review https://review.openstack.org/#/c/546966 | 11:46 |
| *** moshele has quit IRC | 11:47 | |
| gfidente | jistr maybe you actually don't want to! | 11:47 |
| gfidente | jistr still | 11:47 |
| gfidente | pls :D | 11:47 |
| jistr | :D sure thing i'll review | 11:48 |
| *** abishop has joined #tripleo | 11:48 | |
| bandini | therve: are you going to do the backport dance to stable/queens for https://review.openstack.org/521204 ? | 11:48 |
| *** ykarel__ is now known as ykarel | 11:49 | |
| *** anilvenkata has quit IRC | 11:50 | |
| *** anilvenkata has joined #tripleo | 11:50 | |
| *** raildo has joined #tripleo | 11:51 | |
| *** quiquell is now known as quiquell|lunch | 11:53 | |
| *** ktibi has quit IRC | 11:54 | |
| *** anande has quit IRC | 11:54 | |
| *** morazi has joined #tripleo | 11:54 | |
| *** lucasagomes is now known as lucas-hungry | 11:55 | |
| janki | jaosorior, any idea? | 11:56 |
| jaosorior | janki: I don't find where the ::opendaylight::config class is called | 11:56 |
| *** jcoufal has joined #tripleo | 11:56 | |
| EmilienM | hi | 11:57 |
| jaosorior | janki: oh wait, found it | 11:57 |
| jaosorior | EmilienM: uh... you might be able to help janki better, she's having puppet-related issues. | 11:57 |
| EmilienM | jaosorior: i'm in meeting now | 11:58 |
| *** jlabarre has joined #tripleo | 11:58 | |
| janki | jaosorior, ohk. | 11:58 |
| jaosorior | janki: haven't figured out the issue yet | 11:58 |
| jaosorior | janki: I'll give it another try after the security squad meeting (which starts in 2 min) | 11:58 |
| janki | jaosorior, the code and implementation is good as non-HA deployment has correct configs and works fine. issue is in HA deployment only. sure. please let me know if you find anything | 11:59 |
| jaosorior | janki: will do | 11:59 |
| jaosorior | how the hell can it be snowing again.. get it together Finland! | 11:59 |
| owalsh | jaosorior, lhinds: do you need me for anything at the security squad meeting? need to step out for a few minutes | 12:00 |
| moguimar | jaosorior: meeting time? | 12:00 |
| jaosorior | lhinds, alee, moguimar around? | 12:00 |
| moguimar | o/ | 12:00 |
| therve | bandini, Yep | 12:00 |
| jaosorior | owalsh: no worries. There will be logs of the meeting. Just gonna point out the current progress of things. | 12:00 |
| owalsh | jaosorior: ack, thanks, I'll review the logs right after | 12:01 |
| jaosorior | Lets wait a bit to see if more folks join | 12:01 |
| raildo | o/ | 12:01 |
| *** holser__ has quit IRC | 12:01 | |
| openstackgerrit | mathieu bultel proposed openstack/tripleo-heat-templates stable/queens: FFU: Fix gnocchi FFU tasks https://review.openstack.org/557309 | 12:01 |
| *** holser__ has joined #tripleo | 12:01 | |
| *** tcw has quit IRC | 12:02 | |
| jaosorior | Hey raildo, how's it going? | 12:02 |
| *** udesale has joined #tripleo | 12:02 | |
| raildo | hey jaosorior, all good, what about you? | 12:02 |
| jaosorior | all good here. Was hoping for spring to come... but it seems it was a false alarm | 12:03 |
| moguimar | haha | 12:03 |
| jaosorior | #startmeeting TripleO Security Squad | 12:03 |
| openstack | Meeting started Wed Mar 28 12:03:45 2018 UTC and is due to finish in 60 minutes. The chair is jaosorior. Information about MeetBot at http://wiki.debian.org/MeetBot. | 12:03 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 12:03 |
| *** openstack changes topic to " (Meeting topic: TripleO Security Squad)" | 12:03 | |
| openstack | The meeting name has been set to 'tripleo_security_squad' | 12:03 |
| jaosorior | Alright folks, lets start | 12:04 |
| jaosorior | #topic Work progress update | 12:04 |
| *** openstack changes topic to "Work progress update (Meeting topic: TripleO Security Squad)" | 12:04 | |
| jaosorior | Out of all the topics, the main effort lately has been in two areas: * TLS by default * Secret Management | 12:05 |
| raildo | #link https://etherpad.openstack.org/p/tripleo-security-squad | 12:05 |
| jaosorior | thanks raildo | 12:05 |
| jaosorior | #topic Public TLS by default | 12:05 |
| *** openstack changes topic to "Public TLS by default (Meeting topic: TripleO Security Squad)" | 12:05 | |
| jaosorior | So, there has been a bunch of progress on this side. | 12:05 |
| jaosorior | TLS is now the default in the classic undercloud, and mistral will automatically push the CA certificate to the overcloud nodes. | 12:06 |
| jaosorior | I also have put up patches for enabling TLS by default for the containerized undercloud | 12:06 |
| jaosorior | namely | 12:06 |
| jaosorior | #link https://review.openstack.org/557159 | 12:06 |
| jaosorior | and | 12:06 |
| jaosorior | #link https://review.openstack.org/557160 | 12:06 |
| jaosorior | And I actually got TLS (public) for the overcloud working. | 12:07 |
| jaosorior | But currently I'm facing some package related issues that prevent CI on that job from running properly | 12:07 |
| jaosorior | #link https://review.openstack.org/#/c/554926/ | 12:08 |
| jaosorior | That one ^^ enables TLS by default for the overcloud. But is currently failing because I introduce a new script to tripleo-common, thus packaging fails. | 12:08 |
| jaosorior | #link https://review.openstack.org/#/c/554926/ | 12:08 |
| jaosorior | that one ^^ is the commit to the spec file in RDO, that would enable that script. But I haven't gotten that commit to work yet | 12:09 |
| jaosorior | So any help on that side is welcome | 12:09 |
| *** dprince has joined #tripleo | 12:09 | |
| jaosorior | The rest of the reviews that have been put up are on the etherpad. | 12:09 |
| jaosorior | Any questions/feedback? | 12:09 |
| *** ooolpbot has joined #tripleo | 12:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757556 | 12:10 |
| *** ooolpbot has quit IRC | 12:10 | |
| openstack | Launchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged] | 12:10 |
| bogdando | folks PTAL https://review.openstack.org/#/c/553427/ all green | 12:10 |
| raildo | for now, I believe that we just need to go deeper in those patches and have that reviewed soon | 12:10 |
| jaosorior | that would be appreciated | 12:11 |
| openstackgerrit | Gael Chamoulaud proposed openstack/tripleo-validations stable/queens: Fix overcloud services connectivity validation https://review.openstack.org/557315 | 12:11 |
| jaosorior | so please, if you have some time, reviews are always welcome | 12:11 |
| *** ansmith has joined #tripleo | 12:11 | |
| jaosorior | Moving on | 12:12 |
| jaosorior | #topic Secret Management audit | 12:12 |
| *** openstack changes topic to "Secret Management audit (Meeting topic: TripleO Security Squad)" | 12:12 | |
| jaosorior | alee and me went through a deployment and started listing out the files and places where we have sensitive data | 12:13 |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-quickstart-extras master: Refactored validate-tempest role for undercloud and containers https://review.openstack.org/551441 | 12:13 |
| jaosorior | #link https://etherpad.openstack.org/p/tripleo-audit-secrets | 12:13 |
| jaosorior | fortunately, it seems that we're well covered in basic unix permissions for those files. | 12:14 |
| jaosorior | the goal of that is, once we have identified everything, we'll go through those secrets and come up with strategies on how to secure them | 12:14 |
| jaosorior | we're already throwing around ideas on how to do that, but we'll need to meet up again to formalize some of those approaches, and discuss them further. | 12:14 |
| jaosorior | So, if folks have time to review the etherpad I posted above, please check it out; if we missed anything feel free to add that there. | 12:15 |
| jaosorior | We could probably use the same Etherpad to post some ideas on how to lock those places down | 12:16 |
| jaosorior | Any questions/feedback? | 12:16 |
| *** moshele has joined #tripleo | 12:18 | |
| raildo | jaosorior, it more fore curiosity, is there anyway to guarantee that we are covering all the secrets in this document? | 12:18 |
| raildo | jaosorior, I mean, I'm supposing that you guys listed those secrets bases in your previous knowledge in where the secrets are, right? | 12:19 |
| *** gkadam has joined #tripleo | 12:19 | |
| jaosorior | raildo: we also did a bunch of grep magic around the nodes :D but of course we might have missed something | 12:19 |
| raildo | jaosorior, ok, got it | 12:20 |
| jaosorior | raildo: this is why it's important to have more folks review that list, and if someone knows of something we missed, it should be posted there. | 12:20 |
| jaosorior | and once we secure it, it'll be a matter of documenting the approach and writing up some best-practices. | 12:20 |
| raildo | jaosorior, yeak, makes sense. thanks! | 12:20 |
| jaosorior | in k8s, nobody stops you from writing out some passwords in your templates; but of course it's prefered to use k8s secrets instead. | 12:21 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/python-tripleoclient master: Hard link http boot contents for Ironic https://review.openstack.org/556516 | 12:21 |
| jaosorior | Anyway | 12:22 |
| jaosorior | that's all from my side | 12:22 |
| *** eck` is now known as eck`gone | 12:22 | |
| jaosorior | #topic Any other business | 12:22 |
| *** openstack changes topic to "Any other business (Meeting topic: TripleO Security Squad)" | 12:22 | |
| jaosorior | Does someone have something else that you would like to bring up to the meeting? | 12:22 |
| *** ramishra has quit IRC | 12:22 | |
| *** moshele has quit IRC | 12:23 | |
| holser__ | bnemec - Do we really need https://github.com/openstack/instack-undercloud/blob/master/instack_undercloud/undercloud.py#L2362-L2366 | 12:24 |
| moguimar | nothing here | 12:24 |
| *** pradk has quit IRC | 12:24 | |
| jaosorior | Alright | 12:24 |
| holser__ | I guess it was special case for N>M upgrade | 12:24 |
| jaosorior | Remember that if you're interested in a topic and would like some help getting started out, you're welcome to reach out to me or lhinds and we'll help you out getting started | 12:24 |
| jaosorior | Thanks for attending folks! | 12:25 |
| jaosorior | #endmeeting | 12:25 |
| *** openstack changes topic to "Welcome to Rocky. CI status - Promotions: RED; check/gate: GREEN; RDO CI jobs: Questionable | http://tripleo.org/ | https://docs.openstack.org/tripleo-docs/latest/" | 12:25 | |
| openstack | Meeting ended Wed Mar 28 12:25:12 2018 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 12:25 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/tripleo_security_squad/2018/tripleo_security_squad.2018-03-28-12.03.html | 12:25 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/tripleo_security_squad/2018/tripleo_security_squad.2018-03-28-12.03.txt | 12:25 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/tripleo_security_squad/2018/tripleo_security_squad.2018-03-28-12.03.log.html | 12:25 |
| *** ramishra has joined #tripleo | 12:25 | |
| *** masco has quit IRC | 12:28 | |
| jaosorior | bandini: around? | 12:31 |
| rook | I wonder if my deployment didn't pull the patch for the apache worker change | 12:32 |
| rook | i set ZUUL_CHANGES="openstack/tripleo-heat-templates:master:refs/changes/73/557073/1", but i don't see a reduction | 12:33 |
| jaosorior | jistr, marios: Do the update_tasks also get run on upgrades? | 12:33 |
| janki | EmilienM, hey, are you free now? | 12:34 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates stable/queens: Fix fluentd upgrade tasks during FFU. https://review.openstack.org/554831 | 12:34 |
| *** quiquell|lunch is now known as quiquell | 12:34 | |
| *** pchavva has joined #tripleo | 12:36 | |
| openstackgerrit | Merged openstack/tripleo-common master: Read bytes for default CA https://review.openstack.org/554836 | 12:36 |
| hjensas | Hey, I have a heat templating question. I would like to get_attr the hostname in a puppet service template. Ref: http://paste.openstack.org/show/716297/ | 12:36 |
| bandini | jaosorior: sort of | 12:36 |
| *** rlandy has joined #tripleo | 12:37 | |
| jaosorior | bandini: you had passed me an example from rabbitmq that would be relevant in the SSL issue. | 12:37 |
| jaosorior | I lost it :( | 12:37 |
| jaosorior | could you pass it again? | 12:37 |
| bogdando | folks, https://review.openstack.org/#/c/553427/ please | 12:37 |
| bogdando | please review https://review.openstack.org/#/q/topic:containers-default+(status:open+OR+status:merged) | 12:39 |
| rook | I wonder if my deployment didn't pull the patch for the apache worker change ( weshay ) | 12:39 |
| marios | jaosorior: no only the upgrade_tasks | 12:39 |
| rook | i set ZUUL_CHANGES="openstack/tripleo-heat-templates:master:refs/changes/73/557073/1", but i don't see a reduction | 12:39 |
| bandini | jaosorior: ah sure | 12:39 |
| marios | jaosorior: (that is the upgrade_tasks_playbook and the deploy_steps_playbook ) | 12:39 |
| therve | shardy_lunch, https://review.openstack.org/#/c/530948/ isn't used yet in tripleo, right? | 12:40 |
| *** masco has joined #tripleo | 12:40 | |
| jaosorior | marios: that's good to know. | 12:40 |
| bandini | jaosorior: https://github.com/openstack/tripleo-heat-templates/blob/master/docker/services/pacemaker/rabbitmq.yaml#L309 for example | 12:40 |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-common master: Log ansible and ansible-playbook command. https://review.openstack.org/557320 | 12:40 |
| jaosorior | bandini: thanks | 12:40 |
| rook | https://snapshot.raintank.io/dashboard/snapshot/07Z6jz5SjyHBAw1feiOTxR5prpIur4ft?orgId=2 ( weshay EmilienM ) we can see from yesterday to today # of processes/threads | 12:41 |
| jaosorior | bandini, marios: So... if I would like some tasks to run both on upgrade and update... I would need to replicate the code then? | 12:41 |
| bandini | jaosorior: hohum good question. dciabrin ^ | 12:42 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-quickstart-extras master: Revert "undercloud: workaround for masquerading network in CI/OVB" https://review.openstack.org/557322 | 12:42 |
| openstackgerrit | Jose Luis Franco proposed openstack-infra/tripleo-ci master: Add multinode-overcloud-update playbook to run list https://review.openstack.org/547058 | 12:42 |
| *** chkumar246 is now known as chandankumar | 12:43 | |
| EmilienM | rook: with alex's patch? | 12:44 |
| *** dsariel has joined #tripleo | 12:44 | |
| rook | EmilienM: I am wondering if I need to do more than set the ZUUL_CHANGES prior to running oooq -- i pinged weshay about it. | 12:44 |
| rook | I set that env var -- but I am doubting it is setup right, i can jump into the container and double check | 12:44 |
| weshay | rook, /me checks my env.. I did the same thing | 12:46 |
| rook | ack | 12:46 |
| *** sshnaidm is now known as sshnaidm|afk | 12:46 | |
| marios | jaosorior: no we do someincludes | 12:47 |
| marios | bandini: already | 12:47 |
| marios | jaosorior: sec let me get you pointer | 12:47 |
| weshay | rook, on your undercloud do you have /opt/gating_repo/openstack-tripleo-heat-templates-8.0.0-0.20180327225321.bfadb63.el7.centos.noarch.rpm | 12:47 |
| EmilienM | rook: you can easily check with ps and # of httpd processes | 12:47 |
| marios | jaosorior: by includes i mean yaml & referencing & and * | 12:47 |
| EmilienM | janki: what's up? | 12:47 |
| rook | EmilienM: that is what the graph i shared shows | 12:47 |
| rook | EmilienM: which doesn't show much of a difference | 12:47 |
| rook | weshay: ack let me check | 12:47 |
| marios | jaosorior: e.g. here https://github.com/openstack/tripleo-heat-templates/blob/4ed5eae601ac591696f87db5b7c57eef97bce95c/docker/services/pacemaker/rabbitmq.yaml#L345 | 12:48 |
| EmilienM | rook: because I tested mwhahaha's patch and I could reduce the number of process by 3 | 12:48 |
| bandini | marios: ah right. damien mentioned that too me once | 12:48 |
| rook | Nope weshay | 12:48 |
| rook | doesn't exist | 12:48 |
| marios | jaosorior: it is including this EATE_COMPLETE state changed | 12:48 |
| *** mcornea has joined #tripleo | 12:48 | |
| marios | jaosorior: this https://github.com/openstack/tripleo-heat-templates/blob/4ed5eae601ac591696f87db5b7c57eef97bce95c/docker/services/pacemaker/rabbitmq.yaml#L235 | 12:48 |
| marios | bandini: yeah i added it first for the upgrade_tasks, trying to re-use what we were already doing for the pike minor update | 12:49 |
| marios | bandini: i.e. container to container | 12:49 |
| bandini | right | 12:49 |
| marios | bandini: wasn't sure it would work but it did :) | 12:49 |
| bandini | eheh w00t | 12:49 |
| janki | EmilienM, I have a parameter in ODL THT will logs to file. I am chaging it to "console" via THT env file. ODL Puppet module changes are implemented and works fine. In non-HA deployment, ODL Logs to console but in HA, it logs to file even though specified to log to console via THT env file. hieradata on controller node has value set to console but the manifest at /etc/puppet/ has it set to file. I dont know why this is happening anf where to look further | 12:50 |
| weshay | rook, http://pastebin.test.redhat.com/569681 | 12:50 |
| *** salmankhan has quit IRC | 12:50 | |
| jaosorior | marios: thanks! | 12:51 |
| rook | ok weshay let me nuke this and try again | 12:51 |
| * marios nods jaosorior | 12:52 | |
| jaosorior | bandini: got time for a quick call? | 12:52 |
| *** lblanchard has joined #tripleo | 12:53 | |
| *** thrash is now known as thrash|biab | 12:53 | |
| rook | beagles: so close! | 12:54 |
| rook | Step3 | 12:54 |
| *** lucas-hungry is now known as lucasagomes | 12:55 | |
| *** panda|lunch is now known as panda | 12:56 | |
| EmilienM | janki: show me the code in THT that does this param | 12:57 |
| janki | EmilienM, https://github.com/openstack/tripleo-heat-templates/blob/master/environments/services-docker/neutron-opendaylight.yaml#L16, https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/services/opendaylight-api.yaml#L69 https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/services/opendaylight-api.yaml#L147 | 12:58 |
| openstackgerrit | Marius Cornea proposed openstack/tripleo-upgrade master: WIP: Add support for per role FFU upgrade https://review.openstack.org/556687 | 12:59 |
| janki | EmilienM, puupet-odl changes https://github.com/opendaylight/integration-packaging-puppet-opendaylight/commit/cc8f1728228fbac279d91aab5837096907a5e828 | 12:59 |
| *** salmankhan has joined #tripleo | 12:59 | |
| janki | EmilienM, I would like to emphasis on the point that in non-HA, ODL logs to console (as is specified and desired). In HA, it logs to file even though specified to log to console. | 13:00 |
| EmilienM | janki: have you checked that the version of the puppet module deployed in your env is the right one and contains the right code? | 13:00 |
| *** trown|brb is now known as trown | 13:00 | |
| skramaja | EmilienM: could you take a look at https://review.openstack.org/#/c/554846/? | 13:01 |
| EmilienM | skramaja: sure | 13:01 |
| skramaja | thanks. | 13:01 |
| janki | EmilienM, yes version is correct. infact, hiera opendaylight::log_mechanism on controller node returns "console". so I am guessing correct value is being passed from THT to puppet. | 13:02 |
| bandini | jaosorior: sure | 13:02 |
| trown | chem: is ffu supposed to be from newton->queens? | 13:03 |
| *** eck`gone is now known as eck` | 13:03 | |
| chem | trown: hi, yes | 13:04 |
| janki | EmilienM, but /etc/puppet/modules/opendaylight has the log_mechanism value set to "file". I think something is wrong here | 13:04 |
| trown | chem: so skips 2 releases? | 13:04 |
| *** aputtur_ has joined #tripleo | 13:04 | |
| chem | trown: yes | 13:04 |
| trown | ok | 13:05 |
| *** Goneri has joined #tripleo | 13:06 | |
| EmilienM | janki: it's very hard to tell when you can't reproduce easily | 13:06 |
| trown | chem: so for ffu job we deploy newton bits using queens heat templates | 13:06 |
| trown | chem: is that right? | 13:06 |
| EmilienM | janki: and when there is no CI job in place | 13:06 |
| openstackgerrit | Merged openstack/tripleo-common stable/newton: Fix overcloud node delete after an upgrade https://review.openstack.org/546487 | 13:07 |
| openstackgerrit | Merged openstack/tripleo-common stable/ocata: Fix overcloud node delete after an upgrade https://review.openstack.org/546486 | 13:07 |
| weshay | EmilienM, so in the latest run fs001 passed, but 35 failed | 13:07 |
| EmilienM | janki: so if I summarize, hiera returns the right value on the controller but it's not applied in the ODL config? | 13:07 |
| janki | EmilienM, I am currently facing it in the system. Would you like to access it? | 13:07 |
| EmilienM | weshay: :( reason? | 13:07 |
| weshay | checking | 13:07 |
| janki | EmilienM, yes. correct | 13:07 |
| chem | trown: hum... it's a mixed version so we deploy newton overcloud with checked out newton templates in ~/tripleo-heat-template from a queens undercloud | 13:08 |
| janki | EmilienM, and this is happening only in HA deployment | 13:08 |
| weshay | EmilienM, same reason fs001 failed the last time https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master/25885fd/tempest.html.gz | 13:08 |
| weshay | ssh timeout | 13:08 |
| EmilienM | weshay: with new ovs? | 13:09 |
| *** jmelvin has joined #tripleo | 13:09 | |
| weshay | EmilienM, that package is in dlrn-deps so they all will have it, but me looks | 13:09 |
| weshay | EmilienM, openvswitch-2.9.0-3.el7.x86_64 | 13:10 |
| *** ooolpbot has joined #tripleo | 13:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757556 | 13:10 |
| *** ooolpbot has quit IRC | 13:10 | |
| openstack | Launchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged] | 13:10 |
| *** anilvenkata has quit IRC | 13:10 | |
| *** tzumainn has joined #tripleo | 13:10 | |
| *** toure|gone is now known as toure | 13:10 | |
| *** alee has quit IRC | 13:10 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: Lower the number of apache servers https://review.openstack.org/557073 | 13:11 |
| trown | chem: ah i see that now, so we have to backport anything we want to use if ffu to newton (like keystone only) | 13:11 |
| fultonj | Has anyone seen docker.io/tripleomaster/centos-binary-cinder-volume fail to standup because it is trying to delete /etc/hosts? | 13:11 |
| EmilienM | mwhahaha: ^ added bug reference and also with new ovs so we can see if numbers are better | 13:11 |
| chem | trown: yeap | 13:11 |
| fultonj | http://paste.openstack.org/show/716299/ | 13:11 |
| mwhahaha | K | 13:12 |
| *** ykarel_ has joined #tripleo | 13:12 | |
| *** ykarel has quit IRC | 13:12 | |
| *** ykarel__ has joined #tripleo | 13:14 | |
| *** dtantsur|brb is now known as dtantsur | 13:15 | |
| janki | EmilienM, any suggestion on where to look? | 13:16 |
| jistr | gfidente: reviewed. Just a nit + a few things to consider but overall i think it's very nice | 13:16 |
| EmilienM | janki: not really, is the config file modified by other things at all during the deployment? | 13:16 |
| *** ykarel_ has quit IRC | 13:17 | |
| janki | EmilienM, no. there is no other file changing the param. and if there had been any, hiera wont have shown "console" right?? | 13:17 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates stable/newton: Work around packaging issue in iptables-services. https://review.openstack.org/554647 | 13:20 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-upgrade stable/queens: WIP: Add support for per role FFU upgrade https://review.openstack.org/557328 | 13:22 |
| chem | mcornea: backported ^ to queens to have a test run | 13:23 |
| EmilienM | janki: I guess I'm asking if the config file changed because some other parameters were changed too | 13:23 |
| EmilienM | janki: i'm wondering if the config file is touched at all | 13:23 |
| *** lblanchard has quit IRC | 13:24 | |
| mcornea | chem: cool, one note about that - I saw that it triggers regular upgrade job while the patches touches only ffu part in tripleo-upgrade, can we limit the jobs based on the portion of code that is changed? | 13:24 |
| chem | mcornea: we can't until software factory has moved to zuulv3 | 13:25 |
| mcornea | chem: ack | 13:25 |
| *** masco has quit IRC | 13:25 | |
| openstackgerrit | Sergii Golovatiuk proposed openstack/instack-undercloud master: Remove nova-manage db from pre_upgrade https://review.openstack.org/557330 | 13:25 |
| *** aputtur_ has quit IRC | 13:26 | |
| janki | EmilienM, I dont think this parameter is dependent on any other parameter. How to confirm it though? | 13:26 |
| EmilienM | janki: ok let me try again: what other parameters in the config files are controlled by tripleo? | 13:27 |
| *** psachin has quit IRC | 13:30 | |
| janki | EmilienM, https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/services/opendaylight-api.yaml#L61, SNAT mechanism. I guess I know where you are going with this. I need to check snat value in puppet manifest if that is changing | 13:30 |
| EmilienM | janki: yes, check it | 13:31 |
| EmilienM | and see if they are applied. | 13:31 |
| *** mmethot has quit IRC | 13:31 | |
| janki | EmilienM, well. I just started a stack update. I will check once this is over and get back to you | 13:31 |
| *** aputtur_ has joined #tripleo | 13:31 | |
| *** ykarel__ is now known as ykarel | 13:31 | |
| *** pradk has joined #tripleo | 13:32 | |
| dtantsur | hi folks, an easy patch for review, if you have a minute: https://review.openstack.org/#/c/555336/ | 13:32 |
| *** udesale has quit IRC | 13:32 | |
| *** thrash|biab is now known as thrash | 13:34 | |
| *** tcw has joined #tripleo | 13:35 | |
| skramaja | i see the RDO ci is failing most of the time. is it mandatory to get RDO ci verified before pushing patches? done few re-tries for https://review.openstack.org/#/c/555166/, but still failing.. | 13:35 |
| skramaja | EmilienM: mwhahaha ^ | 13:36 |
| *** mmethot has joined #tripleo | 13:37 | |
| chandankumar | weshay: EmilienM tosky arxcruz|off seomthing happened on containers undercloud with tempest http://logs.openstack.org/42/551442/7/check/tripleo-ci-centos-7-undercloud-containers/1310e2f/logs/undercloud/home/zuul/tempest_output.log.txt.gz#_2018-03-28_13_13_29 | 13:38 |
| weshay | someone ate the cirros image eh | 13:38 |
| tosky | it wasn't me! | 13:39 |
| EmilienM | skramaja: +2 | 13:39 |
| *** masco has joined #tripleo | 13:39 | |
| skramaja | thanks. i will give few more rechecks :) | 13:39 |
| tosky | I don't even remember who was supposed to pre-cache the image | 13:40 |
| chandankumar | weshay: tosky the good thing is that it has started running | 13:40 |
| chandankumar | few more tweaks we are back on track | 13:41 |
| *** liverpooler has joined #tripleo | 13:41 | |
| *** haleyb has quit IRC | 13:41 | |
| EmilienM | skramaja: no need | 13:42 |
| skramaja | ack. | 13:43 |
| mwhahaha | chandankumar: it's cause the image isn't in the container, it's on the host. would need to get mounted | 13:43 |
| chandankumar | mwhahaha: yup one more thing, we need to create tempest related resources outside containers | 13:44 |
| *** skramaja has quit IRC | 13:45 | |
| *** akane_ has joined #tripleo | 13:46 | |
| *** akane_ has quit IRC | 13:46 | |
| mwhahaha | owalsh: seeing failures on upgrades for the undercloud in the nova client, http://logs.openstack.org/59/557159/1/check/tripleo-ci-centos-7-undercloud-upgrades/c63ec8a/job-output.txt.gz#_2018-03-28_10_46_04_206023 | 13:46 |
| *** aputtur_ has quit IRC | 13:50 | |
| jaosorior | mwhahaha: holser__ was working on that ^^ | 13:51 |
| mwhahaha | yea | 13:51 |
| *** ratailor_ has joined #tripleo | 13:58 | |
| *** ratailor__ has quit IRC | 14:01 | |
| bnemec | holser__: I think we probably do, but you might want to talk to owalsh about that. | 14:04 |
| *** udesale has joined #tripleo | 14:05 | |
| *** udesale has quit IRC | 14:05 | |
| *** udesale has joined #tripleo | 14:05 | |
| *** jrist has quit IRC | 14:07 | |
| *** dsariel has quit IRC | 14:09 | |
| *** dsariel has joined #tripleo | 14:09 | |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: Add missing dash in repo-setup task. https://review.openstack.org/556886 | 14:09 |
| openstackgerrit | Merged openstack/tripleo-quickstart master: Update overcloud-prep-containers script name. https://review.openstack.org/556860 | 14:09 |
| *** ooolpbot has joined #tripleo | 14:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757556 | 14:10 |
| *** ooolpbot has quit IRC | 14:10 | |
| openstack | Launchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged] | 14:10 |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates master: get-occ-config.sh: fix handling multiple nodes in a role https://review.openstack.org/557346 | 14:10 |
| *** ratailor_ has quit IRC | 14:11 | |
| slagle | holser__: fyi^^ | 14:11 |
| openstackgerrit | Merged openstack/tripleo-common master: Fix missing permissions on Swift rebalance playbook https://review.openstack.org/556928 | 14:12 |
| holser__ | slagle - thanks | 14:13 |
| holser__ | I understand why you changed it | 14:14 |
| jaosorior | bandini: pacemaker restart didn't work | 14:17 |
| *** florianf has quit IRC | 14:17 | |
| openstackgerrit | Rafal Szmigiel proposed openstack/tripleo-heat-templates stable/pike: Adding missing space in octavia-worker.yaml causing Pike deployment fail when Octavia and Fluentd are enabled. https://review.openstack.org/557347 | 14:18 |
| bandini | jaosorior: can you drop me and dciabrin a mail with the exact change and command you ran and result? | 14:18 |
| bandini | I'm doing 5 things at the same time and am easily confused right now | 14:19 |
| *** ykarel has quit IRC | 14:20 | |
| bogdando | mwhahaha: chandankumar: so volumes here https://review.openstack.org/#/c/548325/19/docker/services/tempest.yaml@49 should prolly include, at the very least, /opt/cache:/opt/cache IIUC | 14:20 |
| mwhahaha | bogdando: chandankumar: yea but it probably should be configirable. this is one of the problems with using a container for a cli :/ | 14:21 |
| bogdando | ...via puppet-tripleo :) | 14:22 |
| mwhahaha | for containers it'd be THT | 14:22 |
| mwhahaha | probably need a CacheDir option for tempest | 14:22 |
| bogdando | I understood you wrong then, so let's add some TempestVolumes param there | 14:23 |
| mwhahaha | yea | 14:24 |
| mwhahaha | so we need to pass in where we're going to pass in cached content | 14:24 |
| *** florianf has joined #tripleo | 14:24 | |
| *** mcornea has quit IRC | 14:25 | |
| *** dougbtv has quit IRC | 14:25 | |
| *** masco has quit IRC | 14:25 | |
| *** waleedm has quit IRC | 14:27 | |
| openstackgerrit | Rafal Szmigiel proposed openstack/tripleo-heat-templates stable/ocata: Adding missing space in octavia-worker.yaml causing Ocata deployment fail when Octavia and Fluentd are enabled. https://review.openstack.org/557353 | 14:32 |
| *** moshele has joined #tripleo | 14:33 | |
| *** jrist has joined #tripleo | 14:33 | |
| *** jrist has joined #tripleo | 14:33 | |
| chandankumar | mwhahaha: bogdando I need to mount files only if something is available | 14:33 |
| chandankumar | mwhahaha: bogdando in my first attempt, I want to get things working then we can improve the thingsa | 14:34 |
| *** mcornea has joined #tripleo | 14:34 | |
| *** kopecmartin has quit IRC | 14:34 | |
| mwhahaha | chandankumar: if you're launching the container by hand (and specifying -v options) then you could do that | 14:34 |
| chandankumar | mwhahaha: yup I will update the patchset | 14:35 |
| *** ramishra has quit IRC | 14:38 | |
| *** moshele has quit IRC | 14:40 | |
| *** etingof has quit IRC | 14:41 | |
| *** moshele has joined #tripleo | 14:43 | |
| *** holser__ has quit IRC | 14:44 | |
| openstackgerrit | Rafael Folco proposed openstack/tripleo-quickstart-extras master: Temporarily add tempest failures to skip list https://review.openstack.org/557354 | 14:44 |
| *** moshele has quit IRC | 14:45 | |
| *** ykarel has joined #tripleo | 14:46 | |
| *** holser__ has joined #tripleo | 14:50 | |
| openstackgerrit | Steven Hardy proposed openstack/python-tripleoclient master: Don't exit on a RUNNING message for deploy workflow https://review.openstack.org/557356 | 14:50 |
| *** udesale has quit IRC | 14:50 | |
| therve | Who can I bother for quickstart-extra reviews? I have dumb patches that have sitting for a while | 14:51 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-common master: Standardise Workflow messaging and optionally persist messages https://review.openstack.org/425060 | 14:52 |
| jaosorior | therve: I can help reviewing (tomorrow) | 14:52 |
| therve | jaosorior, Thanks I'll bother you then :) | 14:52 |
| shardy_lunch | mandre: ^^ updated the workflow patch, it doesn't handle the plan state correctly for tripleo.deployment.v1.deploy_plan yet, but my assumption is we'll be moving to the synchronous config_download version as discussed yesterday | 14:53 |
| *** shardy_lunch is now known as shardy | 14:53 | |
| *** zoli is now known as zoli|gone | 14:54 | |
| *** zoli|gone is now known as zoli | 14:54 | |
| shardy | therve: Hey I can help wrt reviews, will check your open reviews now | 14:55 |
| therve | shardy, https://review.openstack.org/#/q/owner:therve%2540redhat.com+status:open+project:openstack/tripleo-quickstart-extras | 14:55 |
| therve | https://review.openstack.org/#/c/536385/ if you like deleting code too :) | 14:56 |
| thrash | florianf: Is it possible to override the variables in a validation when run from the ui? | 14:57 |
| *** sshnaidm|afk is now known as sshnaidm | 14:57 | |
| thrash | florianf: or from the cli for that matter... | 14:58 |
| florianf | thrash: nope, not currently. IIRC all the ui hands over to mistral is the plan name. are you asking in general or something specific? | 14:58 |
| thrash | florianf: well, both in general, and specifically to migrate the pre-deploy validations to tripleo-validations. | 14:58 |
| *** sai_p has joined #tripleo | 14:59 | |
| *** udesale has joined #tripleo | 14:59 | |
| *** gfidente has quit IRC | 14:59 | |
| sai_p | Hello can someone review this https://review.openstack.org/#/c/546476/2 ? Thank you | 14:59 |
| sai_p | It;s been open for a long time | 14:59 |
| thrash | florianf: for instance, the check-boot-images validation that I pushed up... There is no way to change the default kernel and ramdisk name. It's not something that is likely going to differ, but it is possible. | 15:00 |
| thrash | *differ from the default | 15:00 |
| florianf | thrash: from the command line, if you use ansible-playbook, you could use the --extra-vars option | 15:00 |
| thrash | florianf: I understand that part. But users won't be calling them directly to do so. | 15:01 |
| florianf | thrash: ok, i understand. is this something that would be detectable from the plan files? | 15:01 |
| florianf | thrash: or parameters? | 15:01 |
| *** gfidente has joined #tripleo | 15:01 | |
| *** gfidente has quit IRC | 15:01 | |
| *** gfidente has joined #tripleo | 15:01 | |
| florianf | thrash: or glance maybe? | 15:01 |
| thrash | florianf: Right now, I don't think so. I think they were strictly cli params. | 15:01 |
| thrash | florianf: we might want to use a tag on the glance images instead of relying on the name. | 15:01 |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-quickstart-extras master: Refactored validate-tempest role for undercloud and containers https://review.openstack.org/551441 | 15:02 |
| bogdando | dsneddon: hi, PTAL https://review.openstack.org/#/c/553427/ | 15:02 |
| florianf | thrash: and then get the kernel and ramdisk names based on the tag? | 15:02 |
| mandre | shardy: that was a long lunch, even according to French standards :) | 15:02 |
| thrash | florianf: at that point, the names would probably not matter. We just want to see that one and only one image has the tag. | 15:02 |
| florianf | thrash: that way you wouldn't need to add parameters, but the validation could just look them up from glance | 15:02 |
| shardy | mandre: hehe yeah I forgot to update my nick (honest! :) | 15:02 |
| florianf | thrash: oh right, even better. | 15:02 |
| *** links has quit IRC | 15:03 | |
| thrash | florianf: ok... Let me see what it would take to change that. | 15:03 |
| florianf | thrash: sounds great | 15:03 |
| *** udesale has quit IRC | 15:03 | |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-quickstart-extras master: Refactored validate-tempest role for undercloud and containers https://review.openstack.org/551441 | 15:04 |
| mandre | shardy: I'll review your changes, on my side I've been working on the next patch in the series to run the enable_ssh_admin workflow prior to config_download | 15:04 |
| thrash | rbrady,apetrich,jtomasek workflows squad meeting | 15:04 |
| shardy | mandre: ack sounds good, then we can rebase the series and test together | 15:04 |
| shardy | mandre: I think I noticed an issue with how tripleoclient handles SUCCESS messages but the workflow changes should be ready for review I think | 15:05 |
| mandre | shardy: alright, I'll give it a try locally | 15:05 |
| shardy | currently we break on any SUCCESS message, but we're getting them from the nested workflows, so we actually break on tripleo.swift_rings_backup.v1.create_swift_rings_backup_container_plan not tripleo.deployment.v1.deploy_plan | 15:05 |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-quickstart master: Run tempest on fs027 using containerized tempest https://review.openstack.org/551442 | 15:06 |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-quickstart master: Run tempest on undercloud on fs003 https://review.openstack.org/554522 | 15:06 |
| bandini | jaosorior: https://review.openstack.org/#/c/556772/ got a min? | 15:09 |
| *** pcaruana has quit IRC | 15:09 | |
| *** ooolpbot has joined #tripleo | 15:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757556 | 15:10 |
| *** ooolpbot has quit IRC | 15:10 | |
| openstack | Launchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged] | 15:10 |
| *** agurenko has quit IRC | 15:11 | |
| bnemec | hjensas: Just to wrap up my undercloud installation issues from yesterday, it turned out to be a problem related to my proxy. | 15:11 |
| bnemec | Apparently you now need to no_proxy the admin vip too. | 15:11 |
| *** agurenko has joined #tripleo | 15:11 | |
| mwhahaha | boo proxies | 15:11 |
| *** gyankum has joined #tripleo | 15:11 | |
| hjensas | bnemec: good to hear you where able to figure it out. | 15:12 |
| *** chlong has joined #tripleo | 15:14 | |
| bnemec | hjensas: Thanks for your help with my investigation. It was very helpful to have someone else check my results. | 15:14 |
| *** yprokule has quit IRC | 15:15 | |
| bnemec | mwhahaha: Yeah, unfortunately some of our users have to use proxies. :-/ | 15:15 |
| * bnemec does it so squid can cache packages | 15:15 | |
| mwhahaha | yea i know, it royally messes with things :/ | 15:16 |
| bnemec | Ooh, I probably have a blog post to update. | 15:16 |
| bnemec | I think I had one about using squid with TripleO. | 15:16 |
| mwhahaha | bnemec: if you have a specific blog post around it let me know, i think this might be something we should document better | 15:16 |
| shardy | somewhat relatedly, now containers are the default, can we modify overcloud-full to remove all the packages? | 15:17 |
| shardy | e.g the openstack stuff, they're all still in there and I'm guessing CI will run faster with a smaller image | 15:17 |
| mwhahaha | shardy: not until we figure out the provision bits | 15:18 |
| mwhahaha | shardy: it's more dependent on ironic & ovs and stuff | 15:18 |
| shardy | mwhahaha: I'm not saying move to a generic centos image, just lets remove all the packages for things we now deploy in containers | 15:18 |
| mwhahaha | we ran into problem where we still are using things from the image build (ie openvswitch) | 15:18 |
| mwhahaha | shardy: i think dprince had a test patch | 15:19 |
| mwhahaha | with that | 15:19 |
| shardy | mwhahaha: Ok, well something to work towards I guess, it seems kind of a waste to have a huge glance image and also a bunch of huge container images | 15:19 |
| mwhahaha | shardy: https://review.openstack.org/#/c/521172/ | 15:19 |
| mwhahaha | yes | 15:20 |
| openstackgerrit | Merged openstack/tripleo-common master: Make some tweaks to swift logging https://review.openstack.org/498930 | 15:20 |
| *** ukalifon has quit IRC | 15:21 | |
| bnemec | mwhahaha: http://blog.nemebean.com/content/taking-advantage-squid-tripleo-development | 15:21 |
| *** agopi|out has joined #tripleo | 15:21 | |
| EmilienM | mwhahaha, slagle: we have OVB job almost ready to deploy a containerized undercloud; what do you think if we move forward now with https://review.openstack.org/#/c/542906/ so we also have a multinode job. So by rocky-m1, we would have one OVB and one multinode with the containerized u/c. | 15:23 |
| EmilienM | note: the OVB job is here: https://review.openstack.org/#/c/542556/ (waiting on a promotion to have latest openstackclient | 15:23 |
| *** morazi has quit IRC | 15:24 | |
| openstackgerrit | Steven Hardy proposed openstack/python-tripleoclient master: Don't exit on a RUNNING message for deploy workflow https://review.openstack.org/557356 | 15:25 |
| *** jfrancoa has quit IRC | 15:25 | |
| openstackgerrit | Jiri Stransky proposed openstack/python-tripleoclient master: Keep plan environment on upgrade converge https://review.openstack.org/557371 | 15:27 |
| *** aufi has quit IRC | 15:29 | |
| *** leanderthal has quit IRC | 15:29 | |
| *** etingof has joined #tripleo | 15:30 | |
| jistr | chem, marios, matbu ^ here's a possible fix for the converge issue. I just broke my env by not having it :D so need to revert a bit and retest. | 15:30 |
| jistr | i got the classic resources.Networks: Resource DELETE failed: Conflict: resources.TenantNetwork.resources.TenantSubnet: Unable to complete operation on subnet 0850a855-48bd-4a9d-ba9e-8488c2618214: One or more ports have an IP allocation from this subnet. | 15:30 |
| *** morazi has joined #tripleo | 15:30 | |
| marios | jistr: thanks, will check it out tomorrow | 15:30 |
| jistr | ack thanks | 15:30 |
| *** avivgta has quit IRC | 15:31 | |
| openstackgerrit | Christian Schwede proposed openstack/tripleo-common stable/queens: Fix missing permissions on Swift rebalance playbook https://review.openstack.org/557372 | 15:31 |
| *** pradk has quit IRC | 15:32 | |
| jistr | marios: btw thanks a lot for the patch link earlier, indeed bumping tripleo-common to latest fixed the issue | 15:32 |
| marios | jistr: cool thanks for letting me know, i just passed on info from matbu anyway | 15:33 |
| marios | jistr: have a good evening o// | 15:33 |
| jistr | marios: thanks you too | 15:34 |
| slagle | EmilienM: undercloud-containers has been voting for a while hasn't it? | 15:41 |
| slagle | EmilienM: if so, i'm ok now with moving containers-multinode if you feel it's stable | 15:41 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Do not create NetworkVlanID is the value is not defined https://review.openstack.org/554872 | 15:41 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Remove the lacp bond mode constraints https://review.openstack.org/554846 | 15:41 |
| EmilienM | slagle: yes it's voting for some weeks now and yes, I think the fs010 is stable now, as I'm doing recheck almost every day and keep passing well. OVB isn't stable yet but for other reasons that are under investigation and not related to containerized u/c | 15:42 |
| janki | EmilienM, Hey. update - snat value is changing in the ODL config. | 15:42 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: Test zuul layout for RDO CI https://review.openstack.org/557380 | 15:48 |
| EmilienM | janki: weird, so it's really this log parameter that doesn't work. | 15:49 |
| janki | EmilienM, yes. and only in HA | 15:49 |
| openstackgerrit | Merged openstack/instack-undercloud stable/queens: Use the new dnsmasq PXE filter in ironic-inspector https://review.openstack.org/556677 | 15:54 |
| openstackgerrit | Merged openstack/python-tripleoclient master: Updated from global requirements https://review.openstack.org/557082 | 15:54 |
| *** itlinux has joined #tripleo | 15:54 | |
| *** dpawar has quit IRC | 15:55 | |
| openstackgerrit | Attila Darazs proposed openstack/tripleo-quickstart-extras master: GATE CHECK for quickstart-extras https://review.openstack.org/472607 | 15:55 |
| *** fpan has joined #tripleo | 15:56 | |
| *** etingof has quit IRC | 16:01 | |
| ykarel | mwhahaha, i remember now the patch for fs20 containerized was required for queens, | 16:01 |
| mwhahaha | ykarel: yea we merged the cherry-pick for the default thing later. but still shouldn't have been required | 16:02 |
| mwhahaha | https://review.openstack.org/#/c/550965/ | 16:02 |
| mwhahaha | that merged 2 weeks ago | 16:02 |
| ykarel | okk if cherry-picked than not required | 16:02 |
| mwhahaha | weshay: to answer you question, yes that included neutron | 16:02 |
| weshay | k.. and the logs are going to kill us here | 16:03 |
| weshay | pinche logs | 16:03 |
| *** florianf has quit IRC | 16:03 | |
| mwhahaha | so i wonder if we're having more of a penalty of the neutron services under containerization because they are heavy on the process launching (ie neutron-rootwrap) | 16:04 |
| mwhahaha | weshay: we could try running an fs020 on baremetal and see if it's timing out | 16:04 |
| mwhahaha | ie -e environments/baremetal-services.yaml | 16:04 |
| ykarel | yes that could be as rootwrap was taking time | 16:04 |
| mwhahaha | for giggles | 16:04 |
| *** fragatin_ has quit IRC | 16:05 | |
| weshay | mwhahaha, sure.. I can line that up | 16:06 |
| EmilienM | janki: are you using the same environment file that load the parameter right? | 16:06 |
| mwhahaha | weshay: or we could just try uncontainerizing neutron | 16:06 |
| janki | EmilienM, yes | 16:06 |
| *** ebarrera has quit IRC | 16:07 | |
| *** agurenko has quit IRC | 16:07 | |
| *** bogdando has quit IRC | 16:07 | |
| weshay | mwhahaha, let's create a patch that uncontainerizes neutron and run fs20 via a test patch | 16:07 |
| janki | EmilienM, and I have re-re-rechecked all spellings everywhere | 16:08 |
| *** florianf has joined #tripleo | 16:08 | |
| openstackgerrit | Ronelle Landy proposed openstack-infra/tripleo-ci master: Define overcloud_docker_registry_namespace for rdocloud https://review.openstack.org/557035 | 16:08 |
| mwhahaha | weshay: k let me try that. what's the bug we're using for all this stuff? | 16:08 |
| *** marios has quit IRC | 16:08 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: CI test - never merge https://review.openstack.org/557423 | 16:09 |
| *** ooolpbot has joined #tripleo | 16:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757556 | 16:10 |
| openstack | Launchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged] | 16:10 |
| *** ooolpbot has quit IRC | 16:10 | |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-heat-templates master: DNM: Test uncontainerizing neutron services https://review.openstack.org/557424 | 16:11 |
| mwhahaha | weshay: -^ | 16:11 |
| *** lucasagomes is now known as lucas-afk | 16:13 | |
| mcornea | chem: looks like we caught a valid ffu bug in https://logs.rdoproject.org/28/557328/1/openstack-check/gate-tripleo-ci-centos-7-multinode-1ctlr-featureset036-oc-ffu-queens-nv/Zfd50b33301c347f3ba3813f3e824bfdf/undercloud/home/jenkins/ffu_upgrade_steps_playbook.log.txt.gz | 16:13 |
| *** dparkes has joined #tripleo | 16:17 | |
| ykarel | mwhahaha, will this run fs20? | 16:17 |
| weshay | ykarel, no | 16:17 |
| mwhahaha | i'm attempting a local reproducer with it | 16:17 |
| mcornea | chem: now I am wondering where the fluentd service comes from | 16:17 |
| weshay | we have to go make that happen now.. I don't think fs20 is a check job | 16:17 |
| weshay | I'm going to cheat | 16:17 |
| mwhahaha | it takes too long to be a check job | 16:18 |
| *** holser__ has quit IRC | 16:18 | |
| chem | mcornea: yeah, I've got a valid patch for this | 16:18 |
| chem | mcornea: currently merging https://review.openstack.org/#/c/554831/ | 16:19 |
| chem | mcornea: but yeah your question is still valid | 16:19 |
| openstackgerrit | Christian Schwede proposed openstack/tripleo-common master: Fix parameter indentation on Swift rebalance playbook https://review.openstack.org/557425 | 16:20 |
| weshay | ykarel, ah.. I think we have it in check experimental for rdo sf | 16:21 |
| mcornea | chem: I see there's no ci/environments/scenario001-multinode.yaml in https://github.com/openstack/tripleo-heat-templates/tree/stable/queens/ci/environments | 16:21 |
| mcornea | chem: only environments/scenario001-multinode-containers.yaml which does contain the fluentd service | 16:21 |
| ykarel | weshay, if its there we should run it | 16:22 |
| rook | weshay if it is easier for me to use someones RDO cloud account I can play there... | 16:22 |
| chem | mcornea: yeah it's automatically converted at some point from non-container to container | 16:22 |
| rook | vs trying to reproduce this on my hypervisor. | 16:22 |
| chem | mcornea: and yeah must have the fluentd service in dest but not in source (which may be valide use case) | 16:23 |
| weshay | bah.. it was removed | 16:23 |
| weshay | https://review.rdoproject.org/jenkins/job/gate-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-master/ | 16:23 |
| chem | mcornea: and consequently (if my memory about the ansible tasks playbook is correct) would lead to have a lot of upgrade tasks rewriten | 16:23 |
| mcornea | chem: yeah but fluentd is an optional service and we shouldn't enable optional services during ffu | 16:24 |
| *** ihrachys has joined #tripleo | 16:25 | |
| mcornea | chem: unless it's a requirement as it's the case with panko which is enabled by default | 16:25 |
| *** florianf has quit IRC | 16:26 | |
| *** morazi has quit IRC | 16:26 | |
| chem | mcornea: it was faster to fix fluentd | 16:26 |
| weshay | ykarel, fyi https://review.rdoproject.org/r/13126 | 16:26 |
| chem | "fix" | 16:26 |
| ykarel | weshay, ack | 16:27 |
| chem | mcornea: so new services can appear during upgrade though, making those upgrade task flawed | 16:27 |
| chem | mcornea: so even if not required for ffu it was the right thing to do | 16:27 |
| *** haleyb has joined #tripleo | 16:27 | |
| mcornea | chem: yeah, just thinking out loud why I didn't see this in my test | 16:28 |
| weshay | rook, this box is killing me.. virt-customize works fine by hand | 16:28 |
| weshay | rook, can we reboot this thing | 16:29 |
| *** jpich has quit IRC | 16:29 | |
| hjensas | shardy: Any pointers on this: http://paste.openstack.org/show/716297/ ? Can I get the hostname somehow on L43? | 16:33 |
| hjensas | shardy: or, if there is a better|different approach ... | 16:33 |
| weshay | mwhahaha, ok.. man.. we're queued up | 16:35 |
| mwhahaha | k | 16:35 |
| weshay | mwhahaha, https://review.rdoproject.org/zuul/ | 16:35 |
| weshay | ykarel, you see it ^ | 16:35 |
| weshay | rook, where you at? | 16:36 |
| ykarel | weshay, Good | 16:36 |
| openstackgerrit | Merged openstack/tripleo-common master: Fails to get DPDK feature status. https://review.openstack.org/556887 | 16:36 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Added the missing attributes for roles_data of ovs-dpdk-agent docker service https://review.openstack.org/555166 | 16:36 |
| rook | weshay: kick it | 16:37 |
| weshay | rook, k.. rebooting | 16:37 |
| *** khrystop_ has joined #tripleo | 16:38 | |
| *** ihrachys_ has joined #tripleo | 16:38 | |
| *** khrystoph has quit IRC | 16:38 | |
| *** dsariel has quit IRC | 16:39 | |
| *** morazi has joined #tripleo | 16:39 | |
| *** fragatina has joined #tripleo | 16:40 | |
| *** yamahata has joined #tripleo | 16:40 | |
| *** thrash is now known as thrash|biab | 16:40 | |
| *** ihrachys has quit IRC | 16:41 | |
| itlinux | hello guys. has anyone created a role for NOC? can anyone suggest what's best on doing that... I have some old info where I had to modify the policy.json but that was for ocata.. and my understanding is that the RBAC now is going to be changing | 16:41 |
| *** dparkes has quit IRC | 16:42 | |
| *** dparkes has joined #tripleo | 16:43 | |
| shardy | hjensas: Hmm no the service/*.yaml templates are instantiated once per role, not once per node, so in the heat template there's no way to access the hostname - probably the way I'd do it is pass the entire map into puppet-tripleo and look up the hostname there at runtime? | 16:44 |
| shardy | hjensas: the other approach would be to set the entire map as individual keys in config_settings (with some prefix to avoid colliding names) then use hiera interpolation to look up the hostname | 16:45 |
| *** trown is now known as trown|lunch | 16:46 | |
| shardy | hjensas: e.g similar to what we do in https://review.openstack.org/#/c/526692/ to look up net_ip_map entries for a specific network | 16:47 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: CI test - never merge https://review.openstack.org/557423 | 16:47 |
| *** alee has joined #tripleo | 16:47 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: Test zuul layout for RDO-CI https://review.openstack.org/557380 | 16:48 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: remove OS::TripleO::Services::CephClient from CephAll https://review.openstack.org/553416 | 16:49 |
| *** dsariel has joined #tripleo | 16:51 | |
| hjensas | shardy: thanks, I'll look into those suggestions and see if I can come up with something. | 16:51 |
| *** rbrady is now known as rbrady-afk | 16:51 | |
| *** chlong has quit IRC | 16:51 | |
| EmilienM | weshay: I would appreciate some feedback from the CI squad (at least one ack) https://review.openstack.org/#/c/542906/ | 16:52 |
| EmilienM | janki: sorry I was away a bit. I'll keep looking and let you know if I find something. | 16:53 |
| *** trozet has quit IRC | 16:53 | |
| weshay | ack | 16:54 |
| weshay | EmilienM, this is looking awesome man! We should really get ara in the project our logs have gotten worse due to ansible :) | 16:56 |
| EmilienM | weshay: for the steps run by config-download you mean? | 16:56 |
| EmilienM | yes it would be cool for our CI environment | 16:57 |
| weshay | EmilienM, ya for the new undercloud and config-download.. ya | 16:57 |
| EmilienM | I remember we talked about it, let me pull the strings | 16:57 |
| weshay | EmilienM, unformatted ansible logs SUCK | 16:57 |
| *** pkovar has quit IRC | 16:57 | |
| weshay | ansible logs generally suck anyway | 16:57 |
| EmilienM | https://etherpad.openstack.org/p/tripleo-config-download | 16:58 |
| EmilienM | ARA integration. Pending packaging: https://bugzilla.redhat.com/show_bug.cgi?id=1426193 | 16:58 |
| openstack | bugzilla.redhat.com bug 1426193 in Package Review "Review Request: ara - Ansible Run Analysis, Record and visualize Ansible Playbook runs" [Unspecified,On_qa] - Assigned to misc | 16:58 |
| weshay | dmsimard, get on that ;) | 16:58 |
| EmilienM | looks like it's on qa by number80 | 16:58 |
| dmsimard | The package is already in rawhide | 16:58 |
| dmsimard | it's approved and everything | 16:59 |
| number80 | yeah, it needs to be closed, usually, it happens when update is pushed to stable | 16:59 |
| *** tosky has quit IRC | 16:59 | |
| weshay | EmilienM, do we want featureset010 to work only for master atm? | 17:00 |
| weshay | this looks awesome, so that's my only question | 17:00 |
| weshay | EmilienM, I thought chem was using fs010 in some upgrade job.. I'll have to look | 17:00 |
| dmsimard | weshay, EmilienM: I'll try and get the ball rolling for ara on centos, no ETA though because Ansible 2.5 broke some things | 17:01 |
| dmsimard | oh and btw, 2.5 should probably be landing in centos extras sooner rather than later so I hope we're ready :/ | 17:01 |
| EmilienM | weshay: for now, only fs010 and very soon 001 (ovb-ha) | 17:01 |
| weshay | rook, this box is fooooked | 17:02 |
| chem | EmilienM: hum pike upgrade is fs011 not fs 010 | 17:02 |
| weshay | rook, check it out | 17:02 |
| rook | weshay: I have tried.. not sure what tmux you are in | 17:02 |
| rook | ugh | 17:02 |
| weshay | rook, there should only be one session | 17:02 |
| rook | yeah, I am in it | 17:02 |
| rook | lol | 17:02 |
| rook | but you aren't | 17:02 |
| weshay | wha? | 17:03 |
| rook | [root@c08-h09-r930 ~]# tmux ls | 17:03 |
| rook | 0: 1 windows (created Wed Mar 28 16:46:39 2018) [203x64] | 17:03 |
| *** dsariel has quit IRC | 17:04 | |
| weshay | rook, oh shit it worked | 17:04 |
| weshay | hot dam | 17:04 |
| weshay | we're in business | 17:04 |
| weshay | rook, run "yum update libvirt" w/o -y | 17:04 |
| weshay | rook, there are package dep errors | 17:05 |
| rook | are we on the same host? | 17:05 |
| rook | lol | 17:05 |
| rook | [root@c08-h09-r930 tripleo-quickstart]# yum update libvirt | 17:05 |
| rook | Loaded plugins: product-id, search-disabled-repos, subscription-manager | 17:05 |
| rook | This system is not registered with an entitlement server. You can use subscription-manager to register. | 17:05 |
| rook | No packages marked for update | 17:05 |
| weshay | 10.12.70.15 | 17:05 |
| rook | yarp | 17:06 |
| *** derekh has quit IRC | 17:07 | |
| *** thrash|biab is now known as thrash | 17:08 | |
| janki | EmilienM, sure. thanks and have a nice day | 17:08 |
| *** janki has quit IRC | 17:09 | |
| *** ooolpbot has joined #tripleo | 17:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757556 | 17:10 |
| openstack | Launchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged] | 17:10 |
| *** ooolpbot has quit IRC | 17:10 | |
| *** dtantsur is now known as dtantsur|afk | 17:10 | |
| EmilienM | chem: ack, then we're good | 17:10 |
| EmilienM | weshay: nice comment I'll fix hat up | 17:11 |
| EmilienM | that* | 17:11 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Add purge to Nova cleanup cron. https://review.openstack.org/551966 | 17:13 |
| weshay | EmilienM, ya.. I'm not like that mwhahaha guy.. +2'ing everything I see | 17:15 |
| * weshay runs | 17:15 | |
| EmilienM | lol | 17:15 |
| mwhahaha | someone has to actually look at code | 17:15 |
| mwhahaha | it doesn't merge itself | 17:15 |
| sdoran | EmilienM: Did some digging on Ansible packaging. Our docs were out of date since there has been a good bit of repo shuffling. The short answer is Upstream Ansible will be in EPEL and releases.ansible.com, and RHEL has the Ansible Engine repo. No more extras. | 17:16 |
| sdoran | I updated the docs, but we'll have to either use EPEL or releases.ansible.com to get Ansible for CentOS. The last Ansible in CentOS Extras was 2.4.2 and there won't be any more release in the repo. | 17:17 |
| EmilienM | sdoran: that looks cool | 17:17 |
| EmilienM | (easier to upgrade I imagine) | 17:17 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Use the same namespace for all docker registries https://review.openstack.org/557444 | 17:17 |
| sdoran | releases.ansible.com is the fastest way to get new release since things tend to take a while to get through EPEL Testing then into EPEL. | 17:17 |
| sdoran | Plus it has the least amount of extra baggage. | 17:18 |
| EmilienM | ok | 17:18 |
| *** dsariel has joined #tripleo | 17:19 | |
| sdoran | The long term plan on the RHEL side is for the Ansible Engine repo to be available to every Red Hat sub. And they have a repo for each major release, e.g., `rhel-7-server-ansible-2.4-rpms`, making it easy to pin versions. | 17:20 |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates master: FFU: Add support for panko service upgrade https://review.openstack.org/533764 | 17:22 |
| sdoran | I'll have to get with slagle and see about how we go about managing repos for CentOS since Ansible won't be "magically available" anymore. | 17:22 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: Deploy container-multinode (fs010) with a containerized undercloud https://review.openstack.org/542906 | 17:23 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates stable/queens: CI test on queens https://review.openstack.org/557446 | 17:24 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates stable/pike: CI test on queens https://review.openstack.org/557447 | 17:24 |
| *** Goneri has quit IRC | 17:25 | |
| slagle | sdoran: EmilienM : i'm guessing that might mean we want to package ansible in the rdo deps repos then. | 17:26 |
| slagle | sdoran: EmilienM : b/c we do not want to be in the business of enabling EPEL | 17:26 |
| openstackgerrit | Merged openstack/puppet-pacemaker master: Fix up error code path https://review.openstack.org/557273 | 17:27 |
| openstackgerrit | Merged openstack/puppet-pacemaker master: Allow meta_params and op_params on resource::ip https://review.openstack.org/553206 | 17:27 |
| *** mdnadeem_ has quit IRC | 17:28 | |
| *** moshele has joined #tripleo | 17:30 | |
| EmilienM | slagle: exactly, since mitaka and now, it has never been a plan to use EPEL in OpenStack, so indeed if Ansible is taken from EPEL we'll have to duplicate the effort and package it in RDO (and probably gate on every change that is made); if it's taken from releases.ansible.com then we might need consider pulling it directly but again we need to find a way to not break our CI every single day and have proper | 17:31 |
| EmilienM | testing | 17:31 |
| mwhahaha | we have testing? | 17:33 |
| *** fpan has quit IRC | 17:33 | |
| slagle | we tests for tests and the former always fail. | 17:34 |
| mwhahaha | we should probably align with how we plan on distributing ansible downstream as well | 17:34 |
| mwhahaha | so we don't do something in RDO that won't actually replicate what we do elsewhere and vice versa | 17:34 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates stable/ocata: CI test on queens https://review.openstack.org/557451 | 17:35 |
| mwhahaha | it seems like it's similar to the ceph bits | 17:35 |
| mwhahaha | where we need to rely on an external part | 17:35 |
| *** fragatina has quit IRC | 17:35 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates stable/newton: CI test on queens https://review.openstack.org/557452 | 17:35 |
| mwhahaha | maybe we could just automate a version bump somewhere? | 17:35 |
| *** shardy has quit IRC | 17:35 | |
| *** tesseract has quit IRC | 17:35 | |
| mwhahaha | they do appear to have a preview and anightly set of rpms | 17:35 |
| EmilienM | mwhahaha: what would be cool imho is 1) import the package in RDO 2) do automatic version bumps 3) for each bump, we run CI jobs in RDO CI before (with some tripleo scenarios) | 17:36 |
| EmilienM | 1) probably means to import ansible into delorean | 17:37 |
| *** dparkes has quit IRC | 17:38 | |
| *** yamahata has quit IRC | 17:38 | |
| *** Goneri has joined #tripleo | 17:40 | |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates master: FFU: Add support for panko service upgrade https://review.openstack.org/533764 | 17:41 |
| *** trozet has joined #tripleo | 17:45 | |
| *** fpan has joined #tripleo | 17:48 | |
| openstackgerrit | Rafael Folco proposed openstack/tripleo-quickstart-extras master: Collect dstat on overcloud nodes https://review.openstack.org/557455 | 17:48 |
| EmilienM | slagle, mwhahaha, sdoran : I'll keep the discussion on the ML re: ansible packaging, please keep posted, it would be cool to move things forward during Rocky! | 17:48 |
| *** moshele has quit IRC | 17:49 | |
| *** khyr0n has joined #tripleo | 17:51 | |
| *** psachin has joined #tripleo | 17:53 | |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: Use openstack commands in overcloud-deploy.sh https://review.openstack.org/529347 | 17:54 |
| *** gkadam has quit IRC | 17:54 | |
| openstackgerrit | Marius Cornea proposed openstack/tripleo-upgrade master: WIP: Add support for per role FFU upgrade https://review.openstack.org/556687 | 17:54 |
| *** gfidente has quit IRC | 17:55 | |
| slagle | EmilienM: what discussion on the ML? where is it? | 17:55 |
| *** salmankhan has quit IRC | 17:55 | |
| *** trown|lunch is now known as trown | 17:56 | |
| EmilienM | slagle: there is none now, but I plan to kick it off today | 17:56 |
| slagle | oh. sorry, was confused | 17:56 |
| EmilienM | np | 17:57 |
| EmilienM | if we could have a standard path because ansible isn't alone here, like we said there is OVS, Ceph and many other things | 17:57 |
| *** khyr0n has quit IRC | 17:58 | |
| *** panda is now known as panda|off | 17:59 | |
| openstackgerrit | Marius Cornea proposed openstack/tripleo-upgrade master: WIP: Add support for per role FFU upgrade https://review.openstack.org/556687 | 18:00 |
| *** itlinux has quit IRC | 18:00 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: CI test - never merge https://review.openstack.org/557423 | 18:03 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: Test zuul layout for RDO-CI https://review.openstack.org/557380 | 18:03 |
| openstackgerrit | Marius Cornea proposed openstack/tripleo-upgrade master: WIP: Add support for per role FFU upgrade https://review.openstack.org/556687 | 18:04 |
| *** lvdombrkr has joined #tripleo | 18:05 | |
| *** dsariel has quit IRC | 18:06 | |
| openstackgerrit | Marius Cornea proposed openstack/tripleo-upgrade master: WIP: Add support for per role FFU upgrade https://review.openstack.org/556687 | 18:07 |
| openstackgerrit | Marius Cornea proposed openstack/tripleo-upgrade master: WIP: Add support for per role FFU upgrade https://review.openstack.org/556687 | 18:09 |
| *** ooolpbot has joined #tripleo | 18:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757556 | 18:10 |
| *** ooolpbot has quit IRC | 18:10 | |
| openstack | Launchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged] | 18:10 |
| *** liverpooler has quit IRC | 18:11 | |
| openstackgerrit | Harald Jensås proposed openstack/puppet-tripleo master: Ironic Inspector - disjoint ip range(s) for HA https://review.openstack.org/557459 | 18:12 |
| *** pblaho has quit IRC | 18:12 | |
| *** mcornea has quit IRC | 18:14 | |
| *** itlinux has joined #tripleo | 18:15 | |
| *** yamahata has joined #tripleo | 18:16 | |
| *** psachin has quit IRC | 18:16 | |
| weshay | rook, can you join my bluejeans | 18:17 |
| *** athomas has quit IRC | 18:17 | |
| weshay | talking perf tools | 18:17 |
| openstackgerrit | Marius Cornea proposed openstack/tripleo-upgrade master: WIP: Add support for per role FFU upgrade https://review.openstack.org/556687 | 18:18 |
| rook | link to join the bj party | 18:18 |
| sshnaidm | EmilienM, is there an easy way to run service in overcloud nodes at start of overcloud deployment? | 18:20 |
| EmilienM | sshnaidm: write the service in tripleo-heat-templates/docker/services/ | 18:20 |
| EmilienM | what service is it? | 18:20 |
| *** abishop_ has joined #tripleo | 18:20 | |
| mwhahaha | EmilienM: he probably wants to run dstat or something | 18:21 |
| sshnaidm | EmilienM, dstat for example.. not sure if it should run in docker, for measuring host performance | 18:21 |
| mwhahaha | so it's not exactly a service | 18:21 |
| mwhahaha | it'd be more of a host prep or pre-deploy thing | 18:21 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: CI test - never merge https://review.openstack.org/557460 | 18:21 |
| rook | weshay: I have a interview in 10 minutes | 18:21 |
| mwhahaha | you could either hack it into the image modification part for CI | 18:21 |
| rook | shoot me the link if ya want me there homie. | 18:21 |
| mwhahaha | rook: good luck on your new job? :D | 18:22 |
| rook | thanks mwhahaha | 18:22 |
| *** lvdombrkr has quit IRC | 18:22 | |
| weshay | rook, bah.. k.. don't worry bout it | 18:22 |
| rook | weshay what? too long of a convo | 18:22 |
| rook | I like perf tools | 18:22 |
| rook | lets do this | 18:22 |
| weshay | ok | 18:22 |
| mwhahaha | i like to listen about perf tools, can i join | 18:22 |
| EmilienM | sshnaidm: is it just a CI thing? | 18:22 |
| weshay | https://bluejeans.com/u/whayutin/ | 18:23 |
| *** abishop has quit IRC | 18:23 | |
| sshnaidm | EmilienM, hmm.. I think it could be useful for any ops too, but currently for CI, yes | 18:24 |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-quickstart-extras master: Refactored validate-tempest role for undercloud and containers https://review.openstack.org/551441 | 18:24 |
| EmilienM | sshnaidm: if it's useful outside of ci then create a service in THT | 18:24 |
| ykarel | mwhahaha, weshay tempest run results on local setup with non containerized neutron services: https://review.rdoproject.org/paste/show/93/ | 18:24 |
| ykarel | And i checked l3 agent logs and it was taking around 3 seconds for doing router operations | 18:25 |
| sshnaidm | EmilienM, where do we have docs/guidelines how to do it? | 18:26 |
| EmilienM | sshnaidm: you probably want to look at existing services in extraconfig/services/ | 18:26 |
| EmilienM | and just create a service that runs host_prep_tasks | 18:27 |
| EmilienM | it's ansible driven | 18:27 |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-quickstart master: Run tempest on fs027 using containerized tempest https://review.openstack.org/551442 | 18:27 |
| *** fragatina has joined #tripleo | 18:27 | |
| *** tbonds has quit IRC | 18:28 | |
| openstackgerrit | Harald Jensås proposed openstack/tripleo-heat-templates master: Ironic Inspector - disjoint ip range(s) for HA https://review.openstack.org/557465 | 18:29 |
| *** mcornea has joined #tripleo | 18:30 | |
| openstackgerrit | Tim Rozet proposed openstack/puppet-tripleo master: Removes neutron ownership of certs https://review.openstack.org/557469 | 18:30 |
| openstackgerrit | Marius Cornea proposed openstack/tripleo-upgrade master: WIP: Add support for per role FFU upgrade https://review.openstack.org/556687 | 18:30 |
| openstackgerrit | Tim Rozet proposed openstack/tripleo-heat-templates master: Fixes Neutron certificate and key permissions https://review.openstack.org/556673 | 18:33 |
| weshay | ykarel++ | 18:34 |
| weshay | ykarel, nice man | 18:34 |
| mwhahaha | so it sounds like we could maybe try with neutron containerized but l3 agent not? | 18:34 |
| *** morazi has quit IRC | 18:34 | |
| mwhahaha | and see if it's a similar number | 18:34 |
| ykarel | mwhahaha, need some extra work for that | 18:35 |
| ykarel | just non containerized l3 failed for me | 18:35 |
| mwhahaha | orly | 18:35 |
| *** rbrady-afk is now known as rbrady | 18:35 | |
| ykarel | mwhahaha, https://review.rdoproject.org/paste/show/94/ | 18:35 |
| mwhahaha | oh that's interesting | 18:36 |
| mwhahaha | beagles: -^ | 18:36 |
| *** gyankum has quit IRC | 18:36 | |
| *** morazi has joined #tripleo | 18:36 | |
| *** gyankum has joined #tripleo | 18:40 | |
| itlinux | hello all, I am getting close to production, and I wonder if anyone has the policy.json to adopt a role with read only for our NOC! | 18:41 |
| mwhahaha | weshay: atop is the other app i've used in the past for recording process interactions over time, https://www.atoptool.nl/ we might think of that instead of dstat if there's a package available | 18:44 |
| mwhahaha | weshay: that being said I think configuring collectd on the overcloud nodes to send to the undercloud might make more sense for the tempest runs | 18:44 |
| weshay | aye | 18:45 |
| *** lblanchard has joined #tripleo | 18:47 | |
| *** lblanchard has quit IRC | 18:47 | |
| weshay | mwhahaha, do you want access to ykarel deployment w/o containerized neutron? | 18:47 |
| weshay | ack re: collectd | 18:47 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: CI test - DNM https://review.openstack.org/557474 | 18:48 |
| *** gyankum has quit IRC | 18:48 | |
| mwhahaha | weshay: no i'm ok i'm spinning mine up still | 18:48 |
| weshay | k | 18:48 |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-heat-templates master: WIP: running dstat in overcloud nodes https://review.openstack.org/557476 | 18:50 |
| sshnaidm | EmilienM, something like that? ^^ | 18:50 |
| EmilienM | looking | 18:51 |
| EmilienM | sshnaidm: yeah and you can even live test it by adding the service to the ci environment | 18:52 |
| EmilienM | sshnaidm: you'll need to add the service in the catalog, grep for OS::TripleO::Services::Rhsm and add OS::TripleO::Services::Dstat maybe | 18:52 |
| *** moshele has joined #tripleo | 18:54 | |
| *** moshele has quit IRC | 18:56 | |
| sshnaidm | EmilienM, is catalog here (in environments directory)? https://github.com/openstack/tripleo-heat-templates/blob/6f2204fc245dfebeff986f877daa55c7c8ff8ad1/environments/rhsm.yaml | 18:57 |
| openstackgerrit | Carlos Goncalves proposed openstack/tripleo-common master: Move health_manager cfg section to octavia_worker https://review.openstack.org/557483 | 18:59 |
| EmilienM | sshnaidm: no, you'll have to patch overcloud-resource-registry-puppet.j2.yaml and roles_data.yaml probably, look for existing services | 18:59 |
| EmilienM | sshnaidm: also create an environments/dstat.yaml probably | 19:00 |
| *** sanjayu has joined #tripleo | 19:02 | |
| *** gkadam has joined #tripleo | 19:06 | |
| sshnaidm | EmilienM, "rhsm" also appears in all files in roles/*.yaml, should I add it there too? | 19:07 |
| EmilienM | sshnaidm: you can add it on the roles you think the service should be running | 19:08 |
| EmilienM | mwhahaha might have thoughts on that | 19:09 |
| *** ooolpbot has joined #tripleo | 19:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757556 | 19:10 |
| openstack | Launchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged] | 19:10 |
| *** ooolpbot has quit IRC | 19:10 | |
| mwhahaha | ehhh i'm not sure this is the best thing to configure as a service | 19:10 |
| mwhahaha | i think i'd rather spend time to get collectd configured correctly | 19:10 |
| *** atoth has quit IRC | 19:10 | |
| *** yamahata has quit IRC | 19:11 | |
| *** yamahata has joined #tripleo | 19:11 | |
| slagle | weshay: is there a place to see ovb jobs here? http://cistatus.tripleo.org/ | 19:13 |
| slagle | ones from the check queue | 19:13 |
| sshnaidm | slagle, http://cistatus.tripleo.org/promotion/ | 19:13 |
| sshnaidm | slagle, and look for "RDO cloud OVB jobs" | 19:14 |
| sshnaidm | mwhahaha, I suspect it will require big puppet config, right? | 19:14 |
| mwhahaha | sshnaidm: which will? collectd? we supposedly ship that already | 19:15 |
| EmilienM | slagle, sdoran: I just the email re: packaging/ansible https://lists.rdoproject.org/pipermail/dev/2018-March/008645.html | 19:15 |
| sdoran | <ack> | 19:15 |
| sshnaidm | mwhahaha, what needs to be done to run it then? | 19:15 |
| mwhahaha | sshnaidm: ¯\_(ツ)_/¯ | 19:16 |
| mwhahaha | the key would be getting the collector going on the undercloud. that i don't think we currently do | 19:16 |
| slagle | sshnaidm: thanks! | 19:16 |
| mwhahaha | https://github.com/openstack/tripleo-heat-templates/blob/master/environments/collectd-environment.yaml | 19:16 |
| sshnaidm | mwhahaha, sorry, I don't understand you | 19:17 |
| sshnaidm | mwhahaha, we want to run performance tool on overcloud when it's being deployed | 19:18 |
| mwhahaha | i'm aware at you what you're doing | 19:18 |
| sshnaidm | especially in OVB case | 19:18 |
| mwhahaha | i'm saying we already ship a tool that could be used | 19:18 |
| mwhahaha | and dstat doesn't give us decent enough data to actually do anything with | 19:18 |
| mwhahaha | it's ok for the basics but doesn't really give us info | 19:19 |
| EmilienM | have we reached out MetMon team? | 19:19 |
| mwhahaha | my personal preference would be atop (but we don't have it in rdo), collectd we already ship so we'll miss some of the early io but i think it can be setup in like step1 | 19:20 |
| mwhahaha | the issue with collectd is that it ships the info to a central server so we have to do that configuration on the undercloud (or somewhere) | 19:20 |
| mwhahaha | also it would be a really good idea to actually test the stuff we ship (novel concept) | 19:21 |
| rook | I agree mwhahaha -- I need to look at the shipping version of collectd we have now... | 19:22 |
| rook | do we have a graphite/carbon server we can send data to right now? | 19:23 |
| mwhahaha | i think rdocloud has one | 19:23 |
| EmilienM | mwhahaha: I updated https://storyboard.openstack.org/#!/story/2001546 for tripleo and assigned to me and set In Progress | 19:25 |
| sshnaidm | mwhahaha, what does atop provide that dstat can't provide? | 19:25 |
| *** pblaho has joined #tripleo | 19:25 | |
| mwhahaha | sshnaidm: a file that can be used to replay the events | 19:25 |
| EmilienM | imho, we should reach out the metmon folks via ML, they own these features... | 19:26 |
| mwhahaha | so i can pull the atop file down, load it up and get a whole bunch more detail than dstat provides | 19:26 |
| mwhahaha | process/network/io/etc | 19:26 |
| mwhahaha | at a configurable interval | 19:26 |
| mwhahaha | so i could go to 5 mins into the deploy and see exactly what was using cpu | 19:26 |
| mwhahaha | and step forward in chunks | 19:26 |
| mwhahaha | https://www.atoptool.nl/ | 19:27 |
| openstackgerrit | Harald Jensås proposed openstack/puppet-tripleo master: Ironic Inspector - disjoint ip range(s) for HA https://review.openstack.org/557459 | 19:27 |
| mwhahaha | anyway, might as well reuse the collectd stuff we already have | 19:27 |
| *** dprince has quit IRC | 19:27 | |
| sshnaidm | mwhahaha, I don't see a problem to run atop instead of dstat in my patch | 19:28 |
| mwhahaha | atop isn't available via rdo | 19:28 |
| mwhahaha | i think it's in epel | 19:28 |
| mwhahaha | and we don't want that | 19:28 |
| sshnaidm | mwhahaha, didn't you say want it? I'm confused | 19:29 |
| mwhahaha | we don't want epel | 19:29 |
| mwhahaha | https://centos.pkgs.org/7/epel-x86_64/atop-2.3.0-8.el7.x86_64.rpm.html | 19:30 |
| *** tbonds has joined #tripleo | 19:30 | |
| sshnaidm | well, we can check with #rdo if it's possible to ship it, but if it's not there it couldn't be used right now | 19:31 |
| mwhahaha | rook: there are other outputs than graphite right? | 19:31 |
| *** waleedm has joined #tripleo | 19:31 | |
| rook | mwhahaha: our stack is, collectd -> graphite -> Grafana | 19:32 |
| rook | well collectd -> graphite -> carbon -> graphite -> grafana | 19:32 |
| rook | to be explicit. | 19:32 |
| mwhahaha | if we want to do it itteratively, can we do something else beside graphite for consumption | 19:32 |
| mwhahaha | because realistically we just want to output to the logs/ folder somehow | 19:32 |
| mwhahaha | not necessarily to a central place in the beginning | 19:33 |
| rook | I have never used it in that fashion, so I don't know. | 19:34 |
| rook | I don't see why you would want to do that. | 19:34 |
| sshnaidm | weshay, well, configuring collectd in tht seems like something requires a big deep dive and N days of effort, not sure it worth it, I can continue with dstat, at least we'll have something in CI for now | 19:34 |
| rook | well, we have collectd + ansible playbooks to configure it today... | 19:35 |
| rook | I literally just ran the playbook against my OOOQ cloud | 19:35 |
| rook | yeah, if you want to reinvent like we did, days.. for sure. | 19:35 |
| weshay | sshnaidm, please leave it for now | 19:36 |
| weshay | sshnaidm, focus on upgrades | 19:36 |
| mwhahaha | so interesting note, neutron-rootwrap isn't high on the cpu list with neutron not containerized | 19:37 |
| * mwhahaha wonders | 19:37 | |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-heat-templates master: WIP: running dstat in overcloud nodes https://review.openstack.org/557476 | 19:38 |
| rook | https://snapshot.raintank.io/dashboard/snapshot/t9Y1DJU0l7eluWCDhPS2cCx2B3qUxWID?orgId=2 EmilienM mwhahaha , weshay hooked me up with a working patch of your work. We can see less http processes -- i mean, still a ton, but less . Tempest is running now. I will run some performance tests against neutron once it completes. | 19:38 |
| EmilienM | mwhahaha: you have numbers? link? | 19:38 |
| mwhahaha | no i don't have numbers cause it never shows up in any of our stuff | 19:39 |
| rook | mwhahaha what about the agent? | 19:39 |
| mwhahaha | i'm watching the tempest run | 19:39 |
| rook | ovs-agent | 19:39 |
| mwhahaha | neutron-server is high | 19:39 |
| mwhahaha | but that's it | 19:39 |
| mwhahaha | and i assume that's always been high | 19:39 |
| mwhahaha | so i bet it's the rootwrap stuff in containers takes a huge perf hit | 19:40 |
| openstackgerrit | Harald Jensås proposed openstack/puppet-tripleo master: Ironic Inspector - disjoint ip range(s) for HA https://review.openstack.org/557459 | 19:42 |
| EmilienM | on another topic I don't think we use mox3 in tripleo, I checked tripleo-common, tripleoclient and os-*-config | 19:43 |
| EmilienM | bnemec: any thoughts? ^ I think i'm going to close the goal for tripleo | 19:43 |
| rook | hm mwhahaha i wonder if we try switching the rootwrap mode? | 19:43 |
| rook | https://specs.openstack.org/openstack/neutron-specs/specs/kilo/rootwrap-daemon-mode.html | 19:43 |
| *** asbishop has joined #tripleo | 19:44 | |
| mwhahaha | rook: i think we're using daemon already | 19:44 |
| mwhahaha | rook: or maybe we werent' for ovs | 19:44 |
| mwhahaha | i see daemons | 19:44 |
| *** abishop_ has quit IRC | 19:46 | |
| openstackgerrit | Emilien Macchi proposed openstack/os-net-config master: Remove mox from test-requirements.txt https://review.openstack.org/557493 | 19:46 |
| rook | i wonder if ajo could comment on the rootwrap stuff | 19:47 |
| ajo | Rook which stuff? :) | 19:48 |
| rook | ajo rootwrap in containers bad... rootwrap outof containers, not bad. | 19:48 |
| rook | high CPU utilization in containers. | 19:48 |
| *** yamahata has quit IRC | 19:48 | |
| ajo | Hmmm | 19:50 |
| ajo | Why is that? | 19:50 |
| mwhahaha | i believe that is the question | 19:50 |
| ajo | I noticed high cpu utilization in the ovs agent. | 19:50 |
| ajo | Calling epoll like crazy | 19:50 |
| rook | me too | 19:50 |
| mwhahaha | the ovs agent uses rootwrap right? is it not using it in a daemon mode? | 19:51 |
| eck` | ajo: interesting i've been looking at epoll stuff today... | 19:52 |
| eck` | just submitted... https://review.openstack.org/#/c/557487/ | 19:52 |
| eck` | which is similar to a thing i fixed a while ago in oslo.messaging | 19:53 |
| ajo | Hmmm let me read | 19:53 |
| eck` | where it spams epoll | 19:53 |
| eck` | (that patch is for mistral) | 19:53 |
| eck` | but we probably have similar issues elsewhere | 19:53 |
| ajo | mwhahaha: by default uses root daemon afaik | 19:53 |
| mwhahaha | k | 19:53 |
| eck` | right now i'm looking at ironic, it seems to have a similar problem | 19:53 |
| rook | yeah, i don't see anything int he config stating other wise | 19:53 |
| rook | so it must be the default, daemon | 19:53 |
| eck` | i am just tired of my poor cpus burning to pieces for no good reason, that is why i am looking into this | 19:53 |
| eck` | i have reached my breaking point | 19:53 |
| *** ledo_ has joined #tripleo | 19:55 | |
| rook | eck`: which ironic service? | 19:55 |
| *** waleedm has quit IRC | 19:56 | |
| eck` | rook: both conductor and inspector | 19:56 |
| eck` | i found one instance of my known issue that i am submitting a fix for in literally 30 seconds... | 19:57 |
| eck` | but it doesn't seem to help the symptom | 19:57 |
| eck` | so something else is going on | 19:57 |
| ajo | eck`: probably related. I also suspected of event let too | 19:57 |
| *** ykarel is now known as ykarel|away | 19:57 | |
| eck` | (but my patch would still be an improvement so i'll throw it up) | 19:57 |
| eck` | eventlet is the devil | 19:57 |
| ajo | I can't find my bugreport | 19:57 |
| ajo | Where was it... | 19:57 |
| bnemec | EmilienM: The only one I was aware of was os-net-config, and I believe we already merged a patch to remove it from there. | 19:57 |
| bnemec | That makes sense. Most of our python code was written after the switch to mock. | 19:58 |
| bnemec | We weren't even using mox in os-net-config, just the stubout module of mox. | 19:58 |
| rook | well ajo mwhahaha i take that back... looking at neutron currently the ovs-agent isn't pegging the CPU.. just neutron-server | 19:58 |
| eck` | hm ironic uses threading.Lock | 19:58 |
| eck` | which probably has the same problem under the cover | 19:58 |
| eck` | because it ultimately uses a Condition | 19:58 |
| *** tosky has joined #tripleo | 19:58 | |
| EmilienM | bnemec: good news! | 20:01 |
| *** ansmith has quit IRC | 20:03 | |
| *** tosky has quit IRC | 20:05 | |
| *** ykarel|away has quit IRC | 20:06 | |
| eck` | nevermind i have that backwards, conditions use locks | 20:07 |
| *** tosky has joined #tripleo | 20:09 | |
| eck` | neutron at a glance appears to only use threading.Lock as well... | 20:10 |
| * eck` wonders if eventlet is doing something silly with that as well | 20:10 | |
| *** ooolpbot has joined #tripleo | 20:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757556 | 20:10 |
| *** ooolpbot has quit IRC | 20:10 | |
| openstack | Launchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged] | 20:10 |
| *** fpan has quit IRC | 20:10 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart-extras master: Collect dstat on overcloud nodes https://review.openstack.org/557455 | 20:10 |
| *** moshele has joined #tripleo | 20:11 | |
| ajo | eck`: saving your review to the future me, :) | 20:11 |
| EmilienM | bnemec: why did we abandon https://review.openstack.org/#/c/532354/ ? | 20:11 |
| ajo | It's probably related to what I saw | 20:11 |
| EmilienM | I should probably ask Sean | 20:11 |
| openstackgerrit | Harald Jensås proposed openstack/puppet-tripleo master: Ironic Inspector - disjoint ip range(s) for HA https://review.openstack.org/557459 | 20:12 |
| bnemec | EmilienM: Because at the time it was still in use. | 20:12 |
| *** gkadam has quit IRC | 20:12 | |
| eck` | ajo: fyi https://review.openstack.org/#/c/557500/ is the ironic version i just posted | 20:13 |
| eck` | same thing basically | 20:13 |
| bnemec | EmilienM: I could have sworn I reviewed a patch that removed it, but I don't see it so maybe I'm nuts. :-/ | 20:13 |
| eck` | threading.Event i know for a fact is evil | 20:13 |
| eck` | so those ones are easy to just patch out | 20:13 |
| ajo | Thanks eck` | 20:13 |
| ajo | Eventlet is evil, I agree | 20:13 |
| bnemec | EmilienM: Oh, I looked but didn't review: https://review.openstack.org/#/c/550988/ | 20:13 |
| ajo | So is the python GIL :P | 20:14 |
| rook | lol | 20:14 |
| eck` | and i see that eventlet patches out _allocate_lock, so i am suspicious of that | 20:14 |
| eck` | hate. | 20:16 |
| mwhahaha | weshay: so i still got a timeout on tempest in my fs020 recreate locally | 20:21 |
| mwhahaha | weshay: with uncontainerized neutron | 20:21 |
| *** tbonds has quit IRC | 20:21 | |
| weshay | mwhahaha, well.. this one didn't time out :) https://review.rdoproject.org/jenkins/job/gate-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-master/9/console | 20:23 |
| mwhahaha | weshay: maybe recreate script has different timeout values? | 20:23 |
| * mwhahaha shrugs | 20:23 | |
| mwhahaha | interesting | 20:23 |
| weshay | mwhahaha, https://review.openstack.org/#/c/555251/ | 20:24 |
| weshay | oh. .that merged | 20:24 |
| mwhahaha | oh well i didn't specify -t | 20:24 |
| mwhahaha | so yea that's the problem | 20:24 |
| mwhahaha | i need a -t 300 | 20:24 |
| mwhahaha | so yea | 20:24 |
| weshay | I love yatin | 20:25 |
| mwhahaha | i'll try containerizing somethings next | 20:25 |
| weshay | I express my love publically | 20:25 |
| * rook is waiting for his tempest to fail | 20:26 | |
| rook | the thought is that it is something with HA | 20:27 |
| rook | I wonder if spawning keepalived | 20:27 |
| rook | is some of the issue? | 20:27 |
| *** karthiks has quit IRC | 20:27 | |
| * rook waits to see | 20:27 | |
| *** paramite has joined #tripleo | 20:28 | |
| *** morazi has quit IRC | 20:32 | |
| EmilienM | bnemec: thank you | 20:35 |
| *** eck` is now known as eck`gone | 20:38 | |
| *** karthiks has joined #tripleo | 20:40 | |
| *** dparkes has joined #tripleo | 20:40 | |
| *** pchavva has quit IRC | 20:41 | |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-quickstart master: Test containerized undercloud with TLS https://review.openstack.org/557276 | 20:42 |
| weshay | hrm... ok mwhahaha shiat.. same results https://logs.rdoproject.org/24/557424/1/openstack-experimental/gate-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-master/Z5a2df7deb1104f2399ad2daeb489de22/tempest.html.gz | 20:44 |
| weshay | made no diff | 20:44 |
| *** morazi has joined #tripleo | 20:44 | |
| weshay | rfolco|ruck, you have the tempest bugs logged? | 20:44 |
| mwhahaha | weshay: so thsoe seem legit failures | 20:44 |
| *** moshele has quit IRC | 20:44 | |
| mwhahaha | weshay: but it actually succeeded in 3:45 | 20:45 |
| mwhahaha | that's what i thought wasn't currently happening | 20:45 |
| rfolco|ruck | weshay, https://bugs.launchpad.net/tripleo/+bug/1759583 | 20:45 |
| openstack | Launchpad bug 1759583 in tripleo "fs020 full tempest failures" [High,Triaged] | 20:45 |
| weshay | rfolco|ruck, ok.. let's work on that bug | 20:45 |
| weshay | rfolco|ruck, join in a bluejeans | 20:45 |
| rfolco|ruck | weshay, https://review.openstack.org/#/c/557354/ | 20:46 |
| *** tosky has quit IRC | 20:50 | |
| *** raildo has quit IRC | 20:51 | |
| mwhahaha | is it supposed to be concurrency 3 or 2 | 20:52 |
| openstackgerrit | Harald Jensås proposed openstack/tripleo-heat-templates master: Ironic Inspector - disjoint ip range(s) for HA https://review.openstack.org/557465 | 20:52 |
| mwhahaha | for tempest | 20:52 |
| *** dsariel has joined #tripleo | 20:52 | |
| rook | weshay: mwhahaha the task that has been in question is tempest.scenario.test_network_basic_ops.TestNetworkBasicOps | 20:52 |
| rook | mwhahaha: i thought i heard 2 | 20:52 |
| mwhahaha | my recreate has 3 for some reason | 20:53 |
| mwhahaha | rook: so that's the one we get ssh timeouts for in ovb and other CI jobs | 20:53 |
| mwhahaha | in fs020, we're getting job timeouts trying to run tempest | 20:53 |
| mwhahaha | but there are some failing tests now | 20:53 |
| mwhahaha | looks like a 503 | 20:54 |
| mwhahaha | so i wonder if we maybe need to tune some haproxy configs or something | 20:54 |
| *** tosky has joined #tripleo | 20:54 | |
| rook | 503 to the client? | 20:54 |
| mwhahaha | yea | 20:54 |
| mwhahaha | https://logs.rdoproject.org/24/557424/1/openstack-experimental/gate-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-master/Z5a2df7deb1104f2399ad2daeb489de22/tempest.html.gz | 20:54 |
| mwhahaha | that's w/o containerized neutron | 20:55 |
| mwhahaha | so it runs and completes in a decent time | 20:55 |
| mwhahaha | but we have fails in some api compute items | 20:55 |
| mwhahaha | i'm trying to just rerun only one of those failed test groups to see if i can reproduce the problem | 20:55 |
| rook | i wonder if there is anything in the keystone log? | 20:55 |
| rook | ack | 20:56 |
| mwhahaha | 503 can come from anywhere :D though it's more likely to come from haproxy i think | 20:56 |
| rook | mwhahaha typically i have seen 503s due to not getting a token | 20:56 |
| rook | but, yeah, anywhere. | 20:56 |
| rook | :/ | 20:56 |
| *** yamahata has joined #tripleo | 20:58 | |
| mwhahaha | of course 503s tend not to get logged correctly anyway | 20:58 |
| mwhahaha | i wished we logged haproxy too | 20:59 |
| rook | i thought I had a patch for that | 21:01 |
| rook | i am sure i did | 21:01 |
| mwhahaha | and it probably got lost in containers | 21:01 |
| mwhahaha | cause redoing the work is always fun | 21:01 |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: Remove compatibility cruft from overcloud-validate https://review.openstack.org/537327 | 21:02 |
| openstackgerrit | Merged openstack/tripleo-common master: Remove PackageUpdateManager https://review.openstack.org/536385 | 21:02 |
| mwhahaha | so yea if i rerun only those failed test groups it works fine | 21:02 |
| mwhahaha | so there's a concurrency thing going on somewhere | 21:02 |
| rook | https://review.openstack.org/#/c/497524/ | 21:02 |
| rook | shit | 21:02 |
| mwhahaha | yea you're logging, but it doesn't look like it's getting outputted (or captured) | 21:03 |
| *** yamahata has quit IRC | 21:03 | |
| mwhahaha | owalsh: nova metadata is http protocol right? | 21:05 |
| rook | check the swift log | 21:05 |
| rook | lol | 21:05 |
| mwhahaha | pffft swift | 21:05 |
| owalsh | mwhahaha: you mean vs https? | 21:06 |
| mwhahaha | owalsh: i mean vs tcp | 21:07 |
| mwhahaha | owalsh: https://logs.rdoproject.org/24/557424/1/openstack-experimental/gate-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-master/Z5a2df7deb1104f2399ad2daeb489de22/overcloud-controller-0/var/log/extra/docker/containers/haproxy/stdout.log.txt.gz | 21:07 |
| mwhahaha | owalsh: it's missing mode http | 21:07 |
| mwhahaha | so is panko | 21:07 |
| rook | i wish i was kiding | 21:07 |
| rook | but i think haproxy for some odd reason, logs to the swift logs. | 21:07 |
| rook | i also think they are in /var/log/messages | 21:07 |
| owalsh | mwhahaha: yea, it's http | 21:07 |
| mwhahaha | rook: oh it's the faciliity mapping | 21:07 |
| mwhahaha | owalsh: ok we're missing the mode http in the haproxy config for it | 21:08 |
| *** moshele has joined #tripleo | 21:08 | |
| *** trown is now known as trown|outtypewww | 21:08 | |
| *** paramite has quit IRC | 21:08 | |
| mwhahaha | rook: ok well no 503 from the swift log | 21:09 |
| mwhahaha | but i don't see haproxy in there | 21:09 |
| mwhahaha | https://logs.rdoproject.org/24/557424/1/openstack-experimental/gate-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-master/Z5a2df7deb1104f2399ad2daeb489de22/overcloud-controller-0/var/log/swift/swift.log.txt.gz | 21:10 |
| mwhahaha | did you mean that log? | 21:10 |
| *** ooolpbot has joined #tripleo | 21:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757556 | 21:10 |
| *** ooolpbot has quit IRC | 21:10 | |
| openstack | Launchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged] | 21:10 |
| *** gunix has quit IRC | 21:10 | |
| owalsh | mwhahaha: ok, got an LP for it? I'll take a look tomorrow if I can | 21:12 |
| mwhahaha | owalsh: i've got it. thanks for the info | 21:12 |
| mwhahaha | i'll add you to the lp | 21:12 |
| mwhahaha | er review | 21:12 |
| owalsh | ack, thanks | 21:12 |
| openstackgerrit | Alex Schultz proposed openstack/puppet-tripleo master: Fix nova metadata and panko mode https://review.openstack.org/557511 | 21:13 |
| *** gunix has joined #tripleo | 21:13 | |
| *** asbishop has quit IRC | 21:13 | |
| openstackgerrit | Michele Baldessari proposed openstack/puppet-pacemaker master: WIP DNR initial sketches for proper resource change support https://review.openstack.org/557512 | 21:13 |
| *** itlinux has quit IRC | 21:14 | |
| mwhahaha | hmm these failures are getting a 503 in the compute client | 21:16 |
| * mwhahaha goes looking in the right service logs | 21:16 | |
| mwhahaha | maybe we need more nova blaming | 21:16 |
| owalsh | mwhahaha: I expect ec2api metadata needs mode http too | 21:19 |
| mwhahaha | owalsh: so i want to say it doesn't cause it's rcp | 21:19 |
| mwhahaha | rpc | 21:19 |
| mwhahaha | so mode http screws with it | 21:19 |
| mwhahaha | but that might just be ec2api | 21:19 |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/queens: Fix fluentd upgrade tasks during FFU. https://review.openstack.org/554831 | 21:19 |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: pass the fine input to subunit2junitxml https://review.openstack.org/551364 | 21:19 |
| openstackgerrit | Merged openstack/instack-undercloud stable/queens: Add configuration for the Nova proxy endpoint https://review.openstack.org/556396 | 21:19 |
| openstackgerrit | Merged openstack/instack-undercloud stable/queens: Add Nova UI endpoints to the endpoint list https://review.openstack.org/556398 | 21:19 |
| openstackgerrit | Merged openstack/instack-undercloud stable/queens: Fix next_hop for metadata service host route on local_subnet https://review.openstack.org/556675 | 21:19 |
| openstackgerrit | Rafael Folco proposed openstack/tripleo-quickstart-extras master: Temporarily add tempest failures to skip list https://review.openstack.org/557354 | 21:19 |
| openstackgerrit | Merged openstack/instack-undercloud master: Updated from global requirements https://review.openstack.org/555672 | 21:19 |
| openstackgerrit | Merged openstack/instack-undercloud master: add lower-constraints job https://review.openstack.org/555566 | 21:19 |
| openstackgerrit | Merged openstack/instack-undercloud master: Fix help string for subnets option https://review.openstack.org/554895 | 21:19 |
| openstackgerrit | Merged openstack/puppet-tripleo master: Add fact to get array of nic alias name https://review.openstack.org/553119 | 21:19 |
| owalsh | yea, ec2 metadata is just a hacked nova metadata AFAIK | 21:20 |
| owalsh | mwhahaha: ^^^ | 21:20 |
| mwhahaha | k | 21:20 |
| *** rlandy is now known as rlandy|brb | 21:22 | |
| mwhahaha | bandini or jaosorior: why do we not have any haproxy logs anywhere? did we not properly configure syslog for it under containers | 21:24 |
| *** yamahata has joined #tripleo | 21:24 | |
| *** moshele has quit IRC | 21:26 | |
| *** rfolco|ruck is now known as rfolco|off | 21:26 | |
| mwhahaha | cause of course it's in journal.txt | 21:26 |
| mwhahaha | Mar 28 19:03:40 overcloud-controller-0 haproxy[32495]: 192.168.24.13:49754 [28/Mar/2018:19:03:40.880] nova_osapi nova_osapi/<NOSRV> 0/-1/-1/-1/0 503 212 - - SC-- 17/0/0/0/0 0/0 "GET /v2.1/servers/detail?all_tenant | 21:27 |
| mwhahaha | s=True HTTP/1.1" | 21:27 |
| *** ansmith has joined #tripleo | 21:27 | |
| mwhahaha | oh i found it | 21:29 |
| mwhahaha | weshay: nova_osapi crashes or something | 21:29 |
| *** alee has quit IRC | 21:29 | |
| mwhahaha | https://logs.rdoproject.org/24/557424/1/openstack-experimental/gate-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-master/Z5a2df7deb1104f2399ad2daeb489de22/overcloud-controller-0/var/log/journal.txt.gz#_Mar_28_18_36_01 | 21:30 |
| openstackgerrit | Merged openstack/puppet-tripleo master: Enable networking-mlnx ml2 drivers https://review.openstack.org/550505 | 21:30 |
| *** Goneri has quit IRC | 21:32 | |
| *** jlabarre has quit IRC | 21:33 | |
| openstackgerrit | Steve Baker proposed openstack/tripleo-common master: Document functions for container prepare https://review.openstack.org/554707 | 21:35 |
| openstackgerrit | Steve Baker proposed openstack/tripleo-common master: Represent image prepare defaults in new heat param format https://review.openstack.org/554708 | 21:35 |
| openstackgerrit | Steve Baker proposed openstack/tripleo-common master: Move build_service_filter to kolla_builder from tripleoclient https://review.openstack.org/554738 | 21:35 |
| openstackgerrit | Steve Baker proposed openstack/tripleo-common master: Perform multiple container image prepares and merge result https://review.openstack.org/554739 | 21:35 |
| openstackgerrit | Steve Baker proposed openstack/tripleo-common master: Break out function get_undercloud_registry https://review.openstack.org/557519 | 21:35 |
| openstackgerrit | Steve Baker proposed openstack/tripleo-common master: WIP repurpose action container_images_prepare https://review.openstack.org/557520 | 21:35 |
| sai_p | Can someone please give workflow to these reviews https://review.openstack.org/#/c/546476/ https://review.openstack.org/#/c/557101/ ? | 21:38 |
| sai_p | Thank you | 21:38 |
| mwhahaha | sai_p: do you know if nuage is replacing metadata_ip with metadata_host? i think that's the new one in nova | 21:39 |
| rook | I see the haproxy logs mwhahaha | 21:39 |
| mwhahaha | rook: i found them eventually | 21:39 |
| rook | Mar 28 18:23:51 overcloud-controller-0 haproxy[32399]: 192.168.24.1:50366 [28/Mar/2018:18:23:50.993] nova_osapi nova_osapi/overcloud-controller-0.internalapi.localdomain 0/0/0/124/124 200 1797 - - ---- 82/0/0/0/0 0/0 "GET /v2.1/servers/d93be453-7a5c-41f4-893c-73a78468a9a9 HTTP/1.1" | 21:39 |
| * mwhahaha wishes we output them in the haproxy container instead of journal.txt | 21:39 | |
| rook | mwhahaha: ack | 21:39 |
| openstackgerrit | Harald Jensås proposed openstack/instack-undercloud stable/queens: Fix help string for subnets option https://review.openstack.org/557521 | 21:40 |
| rook | mwhahaha yeah me too -- if you have a example how other services do this, i don't mind fixing it. | 21:43 |
| mwhahaha | well it's cause we're using syslog | 21:43 |
| rook | I think we can update the haproxy conf to log to a specific log. | 21:44 |
| rook | oh | 21:44 |
| mwhahaha | other services get outputed to stdout | 21:44 |
| rook | well | 21:44 |
| rook | FML | 21:44 |
| mwhahaha | yea something like that | 21:44 |
| rook | maybe we can just scrape the logs as a post task to create a haproxy log? | 21:44 |
| mwhahaha | at least we are logging :D | 21:44 |
| rook | your welcome? | 21:44 |
| rook | lol | 21:44 |
| mwhahaha | or drop a rsyslog config | 21:44 |
| mwhahaha | rook: you know we actually had an option to enable it | 21:44 |
| *** ledo_ has quit IRC | 21:44 | |
| mwhahaha | that we could have just set to true | 21:44 |
| * mwhahaha failed on that review | 21:45 | |
| mwhahaha | https://github.com/openstack/puppet-tripleo/blob/master/manifests/haproxy.pp#L52 | 21:45 |
| rook | well shitfire. | 21:47 |
| *** akrivoka has quit IRC | 21:47 | |
| mwhahaha | oh well :D | 21:47 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: Deploy container-multinode (fs010) with a containerized undercloud https://review.openstack.org/542906 | 21:48 |
| *** pickle has quit IRC | 21:48 | |
| *** rlandy|brb is now known as rlandy | 21:48 | |
| rook | mwhahaha yeah, i looked back at my patch, i enabled http log and tcplog | 21:48 |
| mwhahaha | yea it says if you set it globally it'll fall back as needed | 21:49 |
| mwhahaha | eh that whole thing needs a rewrite anyway | 21:49 |
| EmilienM | sorry if I missed some context; but have we checked the load of nova-api and nova-api-metadata? | 21:49 |
| rook | EmilienM: https://snapshot.raintank.io/dashboard/snapshot/5fvHrtMq15duO2iQyFS1U7LQ5rq10Y3x?orgId=2 | 21:51 |
| EmilienM | looks like nova_api_wsgi is hungry | 21:52 |
| *** pblaho has quit IRC | 21:53 | |
| -openstackstatus- NOTICE: the zuul web dashboard will experience a short downtime as we roll out some changes - no job execution should be affected | 21:53 | |
| EmilienM | mwhahaha: what are we waiting to land your patch? | 21:54 |
| EmilienM | aka https://review.openstack.org/#/c/557073/ | 21:54 |
| EmilienM | I just noticed all ovb jobs became red on this one | 21:55 |
| mwhahaha | EmilienM: yea it's not super pressing | 21:55 |
| EmilienM | well | 21:55 |
| EmilienM | reducing httpd instances can only help no? | 21:55 |
| mwhahaha | evidently no :D | 21:55 |
| *** jmelvin has quit IRC | 21:55 | |
| EmilienM | sshnaidm: thanks for your comment on https://review.rdoproject.org/r/#/c/13130/ - can you merge it? | 21:55 |
| mwhahaha | i think it'll help but i don't think now is the time to merge that one | 21:56 |
| EmilienM | sshnaidm: oh you say "files:" won't work at all? | 21:56 |
| sshnaidm | EmilienM, yes, "files:" doesn't work | 21:56 |
| EmilienM | where is paul | 21:57 |
| *** social has quit IRC | 21:58 | |
| EmilienM | sshnaidm: I think I know how to make it work | 21:59 |
| *** pabelanger has joined #tripleo | 22:03 | |
| pabelanger | hello | 22:03 |
| EmilienM | pabelanger: hey, sshnaidm has a comment on https://review.rdoproject.org/r/#/c/13130/1/zuul/upstream.yaml about the version of zuul and usage of files: | 22:04 |
| *** yamahata has quit IRC | 22:04 | |
| EmilienM | pabelanger: can you check the syntax and tell if it's supposed to work? I checked old version of project config with examples | 22:04 |
| EmilienM | and files: was used in zuul 2.5 in the project layout | 22:05 |
| openstackgerrit | Alex Schultz proposed openstack/puppet-tripleo master: Fix mode for openstack services https://review.openstack.org/557511 | 22:05 |
| pabelanger | you should be able to see the config output that zuul test generates to validate zuul loaded the regex on that job | 22:05 |
| *** tbonds has joined #tripleo | 22:06 | |
| pabelanger | but regex is supported on files | 22:06 |
| EmilienM | pabelanger: files is supported in zuul v2.5? | 22:06 |
| EmilienM | sshnaidm: ^ fyi | 22:06 |
| pabelanger | yes | 22:07 |
| EmilienM | ok | 22:07 |
| EmilienM | I'll investigate | 22:07 |
| pabelanger | http://git.zuul-ci.org/cgit/zuul/tree/doc/source/zuul.rst?h=2.6.0#n752 | 22:07 |
| pabelanger | that is an example from 2.5 docs | 22:08 |
| pabelanger | well, docs, not example | 22:08 |
| EmilienM | oh doc | 22:08 |
| EmilienM | good | 22:08 |
| pabelanger | http://git.zuul-ci.org/cgit/zuul/tree/tests/fixtures/layout.yaml?h=2.6.0#n113 | 22:09 |
| pabelanger | for unit test example | 22:09 |
| pabelanger | also, I should have used https urls :) | 22:09 |
| *** ooolpbot has joined #tripleo | 22:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757556 | 22:10 |
| *** ooolpbot has quit IRC | 22:10 | |
| openstack | Launchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged] | 22:10 |
| EmilienM | pabelanger: like jim said on #zuul it might be an ordering thing | 22:10 |
| EmilienM | pabelanger: the layout below might cancel what I did above | 22:10 |
| sai_p | mwhahaha: Currently, Nuage is not using metadata_host and not sure will be used in future based on TLS value | 22:11 |
| pabelanger | EmilienM: oh, yes. that is like it. Last rule wins IIRC | 22:11 |
| mwhahaha | sai_p: ok | 22:11 |
| *** Goneri has joined #tripleo | 22:12 | |
| *** alee has joined #tripleo | 22:14 | |
| mwhahaha | owalsh: tomorrow can you take a peek at https://bugs.launchpad.net/tripleo/+bug/1759583 seems nova-api is timing out | 22:14 |
| openstack | Launchpad bug 1759583 in tripleo "UnexpectedResponseCode: Unexpected response code received from api.compute tempest tests" [High,Triaged] | 22:14 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart-extras master: Temporarily add tempest failures to skip list https://review.openstack.org/557354 | 22:15 |
| owalsh | mwhahaha: smells like networking looking at the traceback... | 22:19 |
| mwhahaha | owalsh: it's likely, but any hints as to what part of networking would be nice :D since i can't find that message anywhere | 22:19 |
| *** morazi has quit IRC | 22:24 | |
| *** fpan has joined #tripleo | 22:25 | |
| *** rcernin has joined #tripleo | 22:28 | |
| openstackgerrit | Merged openstack/os-net-config master: Remove mox usage https://review.openstack.org/550988 | 22:29 |
| *** fpan has quit IRC | 22:32 | |
| EmilienM | mwhahaha: fyi both community goals for rocky are done now | 22:34 |
| mwhahaha | k | 22:34 |
| EmilienM | (sb updated) | 22:34 |
| mwhahaha | stevebaker: i'm getting an error when trying to do an image prepate http://paste.openstack.org/show/716658/ thoughts? | 22:41 |
| stevebaker | mwhahaha: looking | 22:42 |
| mwhahaha | it's not consistent on the image, next run it's the barbican-worker | 22:43 |
| stevebaker | mwhahaha: could docker.io be flaking on ping API calls? | 22:43 |
| mwhahaha | it's possible i guess | 22:44 |
| mwhahaha | do we retry that? | 22:44 |
| rook | EmilienM: just read back - were you referring to my graphs? | 22:46 |
| EmilienM | rook: yes | 22:46 |
| rook | re: nova_api_wsgi being hungry | 22:46 |
| rook | EmilienM ah, 10% isn't bad | 22:46 |
| stevebaker | mwhahaha: my immediate emotional reaction is if a REST API can't respond to ping calls then I give up | 22:46 |
| stevebaker | :) | 22:46 |
| mwhahaha | how anti-cloud of you | 22:46 |
| rook | When compared to Neutron 50-70% CPU util | 22:46 |
| stevebaker | mwhahaha: where are you calling this from? an rdocloud node? | 22:47 |
| mwhahaha | stevebaker: no rh2 | 22:47 |
| mwhahaha | must be something going on over there | 22:47 |
| * mwhahaha gives up for the day | 22:47 | |
| mwhahaha | stevebaker: i'll figure it out next week i guess | 22:47 |
| *** ansiwen[q] has joined #tripleo | 22:47 | |
| stevebaker | I've not managed to reproduce yet, will try from a few other places | 22:48 |
| *** pblaho has joined #tripleo | 22:48 | |
| *** pblaho has quit IRC | 23:00 | |
| EmilienM | dsneddon: hey, if you have time this week, please review https://review.openstack.org/#/c/553427/ - it would help for the efforts with containerized undercloud and masquerading | 23:01 |
| dsneddon | EmilienM, I am doing that this moment. | 23:02 |
| dsneddon | EmilienM, I have a question. how do you expect people to set masquerading for routed networks with this patch? Would they create an environment file that overrides environments/services/masquerade-networks.yaml? I wonder if we should have a comment that explains that the hash is <source>:[<destination>, <destination>, ...] ? | 23:03 |
| dsneddon | EmilienM, I was thinking of adding a comment and then merging. | 23:03 |
| *** fpan has joined #tripleo | 23:04 | |
| EmilienM | dsneddon: see the default in https://review.openstack.org/#/c/553427/9/puppet/services/masquerade-networks.yaml | 23:04 |
| EmilienM | default: {'192.168.24.0/24': ['192.168.24.0/24', '192.168.25.0/24']} | 23:05 |
| *** tbonds has quit IRC | 23:05 | |
| EmilienM | which comes from hjensas's input on how he feed it in tripleoclient | 23:05 |
| EmilienM | dsneddon: I have to go now but I'll check your answer later, thanks for your review! | 23:09 |
| dsneddon | EmilienM, Thanks | 23:09 |
| *** ooolpbot has joined #tripleo | 23:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757556 | 23:10 |
| *** ooolpbot has quit IRC | 23:10 | |
| openstack | Launchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged] | 23:10 |
| EmilienM | also if it's just a comment / or doc, I'm happy to push on top of this patch | 23:10 |
| *** rlandy has quit IRC | 23:10 | |
| dsneddon | EmilienM, Good point, I'll merge and figure out the details of documentation with hjensas | 23:10 |
| EmilienM | ack thanks | 23:11 |
| * EmilienM out | 23:11 | |
| *** tosky has quit IRC | 23:16 | |
| openstackgerrit | Merged openstack/puppet-tripleo master: Removes neutron ownership of certs https://review.openstack.org/557469 | 23:21 |
| *** Goneri has quit IRC | 23:22 | |
| openstackgerrit | Marius Cornea proposed openstack/tripleo-upgrade master: Add support for per role FFU upgrade https://review.openstack.org/556687 | 23:23 |
| *** mcornea has quit IRC | 23:34 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart-extras master: Collect dstat on overcloud nodes https://review.openstack.org/557455 | 23:48 |
| openstackgerrit | Attila Darazs proposed openstack/tripleo-quickstart-extras master: GATE CHECK for quickstart-extras https://review.openstack.org/472607 | 23:55 |
| *** fpan has quit IRC | 23:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!