| *** ooolpbot has joined #tripleo | 00:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 00:10 |
|---|---|---|
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674770 | 00:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1676156 | 00:10 |
| *** ooolpbot has quit IRC | 00:10 | |
| openstack | Launchpad bug 1674770 in tripleo "Update timeout too long in CI" [Critical,Triaged] | 00:10 |
| openstack | Launchpad bug 1676156 in tripleo "CI: exit codes and results of shell tasks are ignored in quickstart" [Critical,Triaged] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm) | 00:10 |
| *** lblanchard has joined #tripleo | 00:24 | |
| *** jkilpatr has quit IRC | 00:31 | |
| *** jkilpatr has joined #tripleo | 00:31 | |
| pabelanger | EmilienM: please re-open: https://bugs.launchpad.net/tripleo/+bug/1674681 | 00:38 |
| openstack | Launchpad bug 1674681 in tripleo "buildlogs.centos.org CDN issues" [Critical,Fix released] - Assigned to Paul Belanger (pabelanger) | 00:38 |
| pabelanger | I am going to focus on that in the morning | 00:38 |
| pabelanger | elastic recheck is also added: http://status.openstack.org/elastic-recheck/index.html | 00:38 |
| EmilienM | pabelanger: ok | 00:46 |
| EmilienM | done | 00:46 |
| EmilienM | I'm afk now, good night | 00:46 |
| *** limao has joined #tripleo | 00:48 | |
| *** limao has quit IRC | 01:00 | |
| *** dmacpher has quit IRC | 01:00 | |
| *** limao has joined #tripleo | 01:00 | |
| *** limao has quit IRC | 01:05 | |
| *** ooolpbot has joined #tripleo | 01:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674770 | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1676156 | 01:10 |
| *** ooolpbot has quit IRC | 01:10 | |
| openstack | Launchpad bug 1674770 in tripleo "Update timeout too long in CI" [Critical,Triaged] | 01:10 |
| openstack | Launchpad bug 1676156 in tripleo "CI: exit codes and results of shell tasks are ignored in quickstart" [Critical,Triaged] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm) | 01:10 |
| *** lblanchard has quit IRC | 01:46 | |
| *** tobias-fiberdata has joined #tripleo | 01:47 | |
| *** tobias_fiberdata has quit IRC | 01:49 | |
| *** dixiaoli has joined #tripleo | 01:53 | |
| *** ooolpbot has joined #tripleo | 02:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674770 | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1676156 | 02:10 |
| *** ooolpbot has quit IRC | 02:10 | |
| openstack | Launchpad bug 1674770 in tripleo "Update timeout too long in CI" [Critical,Triaged] | 02:10 |
| openstack | Launchpad bug 1676156 in tripleo "CI: exit codes and results of shell tasks are ignored in quickstart" [Critical,Triaged] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm) | 02:10 |
| EmilienM | https://bugs.launchpad.net/tripleo/+bug/1676250 | 02:18 |
| openstack | Launchpad bug 1676250 in tripleo "Promotion to OpenStack trunk fails (timeout at step3)" [Critical,Triaged] | 02:18 |
| *** dixiaoli has quit IRC | 02:21 | |
| *** saibarspeis has joined #tripleo | 02:29 | |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: Stop uploading puppet artifacts to the overcloud nodes https://review.openstack.org/423190 | 02:31 |
| *** tobias-fiberdata has quit IRC | 02:36 | |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: Stop adding FW rules for docker registry https://review.openstack.org/446024 | 02:37 |
| *** dixiaoli has joined #tripleo | 02:39 | |
| *** cdearborn has quit IRC | 02:59 | |
| *** dixiaoli has quit IRC | 03:00 | |
| *** dixiaoli has joined #tripleo | 03:03 | |
| *** links has joined #tripleo | 03:09 | |
| *** ooolpbot has joined #tripleo | 03:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674770 | 03:10 |
| openstack | Launchpad bug 1674770 in tripleo "Update timeout too long in CI" [Critical,Triaged] | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1676156 | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1676250 | 03:10 |
| *** ooolpbot has quit IRC | 03:10 | |
| openstack | Launchpad bug 1676156 in tripleo "CI: exit codes and results of shell tasks are ignored in quickstart" [Critical,Triaged] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm) | 03:10 |
| openstack | Launchpad bug 1676250 in tripleo "Promotion to OpenStack trunk fails (timeout at step3)" [Critical,Triaged] | 03:10 |
| *** saibarspeis has quit IRC | 03:17 | |
| *** tobias_fiberdata has joined #tripleo | 03:20 | |
| *** saibarsp_ has joined #tripleo | 03:28 | |
| *** psahoo has joined #tripleo | 03:35 | |
| *** dparkes has quit IRC | 03:44 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/ocata: etcd: secure EtcdInitialClusterToken parameter https://review.openstack.org/446517 | 03:47 |
| *** tobias-fiberdata has joined #tripleo | 03:47 | |
| *** tobias_fiberdata has quit IRC | 03:51 | |
| *** dixiaoli has quit IRC | 03:52 | |
| *** dmacpher has joined #tripleo | 03:55 | |
| *** tobias-fiberdata has quit IRC | 03:56 | |
| *** mdnadeem has joined #tripleo | 03:57 | |
| *** saibarsp_ has quit IRC | 04:03 | |
| *** udesale has joined #tripleo | 04:09 | |
| *** ooolpbot has joined #tripleo | 04:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674770 | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1676156 | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1676250 | 04:10 |
| *** ooolpbot has quit IRC | 04:10 | |
| openstack | Launchpad bug 1674770 in tripleo "Update timeout too long in CI" [Critical,Triaged] | 04:10 |
| openstack | Launchpad bug 1676156 in tripleo "CI: exit codes and results of shell tasks are ignored in quickstart" [Critical,Triaged] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm) | 04:10 |
| openstack | Launchpad bug 1676250 in tripleo "Promotion to OpenStack trunk fails (timeout at step3)" [Critical,Triaged] | 04:10 |
| *** gkadam has joined #tripleo | 04:15 | |
| *** gbarros has quit IRC | 04:17 | |
| *** udesale has quit IRC | 04:17 | |
| *** udesale has joined #tripleo | 04:18 | |
| *** radeks has joined #tripleo | 04:28 | |
| *** anton has quit IRC | 04:31 | |
| *** ramishra has joined #tripleo | 04:34 | |
| *** Vijayendra has joined #tripleo | 04:42 | |
| *** dsneddon has quit IRC | 04:46 | |
| *** dixiaoli has joined #tripleo | 04:48 | |
| *** dixiaoli has quit IRC | 04:49 | |
| *** dixiaoli has joined #tripleo | 04:50 | |
| *** dsneddon has joined #tripleo | 04:51 | |
| *** dixiaoli has quit IRC | 04:52 | |
| *** Vijayendra has quit IRC | 04:53 | |
| *** dmacpher has quit IRC | 04:56 | |
| *** Vijayendra has joined #tripleo | 05:01 | |
| *** japestinho has quit IRC | 05:10 | |
| *** ooolpbot has joined #tripleo | 05:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674770 | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1676156 | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1676250 | 05:10 |
| *** ooolpbot has quit IRC | 05:10 | |
| openstack | Launchpad bug 1674770 in tripleo "Update timeout too long in CI" [Critical,Triaged] | 05:10 |
| openstack | Launchpad bug 1676156 in tripleo "CI: exit codes and results of shell tasks are ignored in quickstart" [Critical,Triaged] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm) | 05:10 |
| openstack | Launchpad bug 1676250 in tripleo "Promotion to OpenStack trunk fails (timeout at step3)" [Critical,Triaged] | 05:10 |
| *** pgadiya has joined #tripleo | 05:10 | |
| *** prateek has joined #tripleo | 05:12 | |
| openstackgerrit | Peng Liu proposed openstack/puppet-tripleo master: Add l2 gateway Neutron service plugin profile https://review.openstack.org/444050 | 05:14 |
| *** ratailor has joined #tripleo | 05:23 | |
| *** rcernin has joined #tripleo | 05:26 | |
| *** dmacpher has joined #tripleo | 05:29 | |
| *** dixiaoli has joined #tripleo | 05:29 | |
| *** lmiccini has joined #tripleo | 05:38 | |
| *** pgadiya has quit IRC | 05:41 | |
| *** rwsu has quit IRC | 05:47 | |
| *** skramaja has joined #tripleo | 05:48 | |
| *** iranzo has joined #tripleo | 05:48 | |
| *** dmacpher has quit IRC | 05:55 | |
| openstackgerrit | Saravanan KR proposed openstack/tripleo-heat-templates master: Modify pci_passthrough heira value as string https://review.openstack.org/448600 | 06:01 |
| *** dsariel has joined #tripleo | 06:01 | |
| openstackgerrit | Christian Schwede proposed openstack/tripleo-heat-templates stable/ocata: Fix usage of CinderNfsServers https://review.openstack.org/450085 | 06:04 |
| *** florianf has joined #tripleo | 06:05 | |
| *** ooolpbot has joined #tripleo | 06:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674770 | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1676156 | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1676250 | 06:10 |
| *** ooolpbot has quit IRC | 06:10 | |
| openstack | Launchpad bug 1674770 in tripleo "Update timeout too long in CI" [Critical,Triaged] | 06:10 |
| openstack | Launchpad bug 1676156 in tripleo "CI: exit codes and results of shell tasks are ignored in quickstart" [Critical,Triaged] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm) | 06:10 |
| openstack | Launchpad bug 1676250 in tripleo "Promotion to OpenStack trunk fails (timeout at step3)" [Critical,Triaged] | 06:10 |
| *** jaosorior has joined #tripleo | 06:13 | |
| *** aufi has joined #tripleo | 06:13 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo master: Ensure directory exists for certificates for httpd https://review.openstack.org/449536 | 06:14 |
| *** masco has joined #tripleo | 06:24 | |
| *** jprovazn has joined #tripleo | 06:26 | |
| apetrich | Morning | 06:28 |
| *** dparkes has joined #tripleo | 06:34 | |
| *** dsariel has quit IRC | 06:36 | |
| openstackgerrit | Saravanan KR proposed openstack/tripleo-heat-templates master: WIP: Add role specific information to the service template https://review.openstack.org/437956 | 06:36 |
| *** cylopez has joined #tripleo | 06:37 | |
| *** bkopilov_ has joined #tripleo | 06:39 | |
| *** ealcaniz has joined #tripleo | 06:40 | |
| *** pmannidi has quit IRC | 06:41 | |
| *** suuuper has joined #tripleo | 06:44 | |
| *** dparkes has quit IRC | 06:50 | |
| *** jlinkes has joined #tripleo | 06:51 | |
| *** pmannidi has joined #tripleo | 06:54 | |
| *** jaosorior has quit IRC | 06:58 | |
| *** yprokule has joined #tripleo | 06:59 | |
| *** ratailor has quit IRC | 06:59 | |
| *** ratailor has joined #tripleo | 07:03 | |
| *** ooolpbot has joined #tripleo | 07:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674770 | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1676156 | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1676250 | 07:10 |
| *** ooolpbot has quit IRC | 07:10 | |
| openstack | Launchpad bug 1674770 in tripleo "Update timeout too long in CI" [Critical,Triaged] | 07:10 |
| openstack | Launchpad bug 1676156 in tripleo "CI: exit codes and results of shell tasks are ignored in quickstart" [Critical,Triaged] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm) | 07:10 |
| openstack | Launchpad bug 1676250 in tripleo "Promotion to OpenStack trunk fails (timeout at step3)" [Critical,Triaged] | 07:10 |
| *** dsariel has joined #tripleo | 07:13 | |
| *** jaosorior has joined #tripleo | 07:18 | |
| *** pcaruana has joined #tripleo | 07:20 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: docker/keystone: Bind mount entire fernet keys repository https://review.openstack.org/446473 | 07:23 |
| *** chem has joined #tripleo | 07:23 | |
| *** flepied has quit IRC | 07:24 | |
| *** fragatina has quit IRC | 07:26 | |
| *** fragatina has joined #tripleo | 07:27 | |
| *** leanderthal|afk is now known as leanderthal | 07:27 | |
| jaosorior | mandre: was it so that you had tested this in your deployment? https://review.openstack.org/#/c/446473/ | 07:27 |
| *** mcornea has joined #tripleo | 07:28 | |
| mandre | jaosorior: yes, I think that's one I tested locally | 07:30 |
| *** japestinho has joined #tripleo | 07:32 | |
| jaosorior | jistr: could you check this out https://review.openstack.org/#/c/446473/ ? mandre had tried it out locally IIRC | 07:33 |
| *** jpena|off is now known as jpena | 07:34 | |
| *** tesseract has joined #tripleo | 07:44 | |
| *** jpena is now known as jpena|off | 07:47 | |
| *** agurenko has joined #tripleo | 07:50 | |
| *** mdnadeem has quit IRC | 07:51 | |
| *** pmannidi has quit IRC | 07:52 | |
| *** jpena|off is now known as jpena | 07:54 | |
| *** amoralej|off is now known as amoralej | 07:55 | |
| *** abehl has joined #tripleo | 07:55 | |
| *** flepied has joined #tripleo | 07:56 | |
| *** dixiaoli has quit IRC | 07:59 | |
| *** zzzeek has quit IRC | 08:00 | |
| openstackgerrit | Attila Darazs proposed openstack/tripleo-quickstart-extras master: GATE CHECK https://review.openstack.org/430277 | 08:00 |
| *** zzzeek has joined #tripleo | 08:01 | |
| *** pkovar has joined #tripleo | 08:04 | |
| *** dixiaoli has joined #tripleo | 08:05 | |
| saneax | GM folks, I see that there was a work already done to get monitoring tools deployed with tripleo, however in the documentation side, I do not find much. IS there a document which one can refer when building a tool which could needs to be integrated with sensu? | 08:06 |
| *** jpich has joined #tripleo | 08:07 | |
| shardy | saneax: I don't think there is right now, unfortunately | 08:07 |
| *** dixiaoli has quit IRC | 08:07 | |
| shardy | saneax: we should probably raise a bug and ask the folks who worked on that integration to contribute some docs, or at least update this README | 08:08 |
| shardy | https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/services/README.rst | 08:08 |
| *** dixiaoli has joined #tripleo | 08:08 | |
| *** dixiaoli has quit IRC | 08:08 | |
| shardy | saneax: I'd suggest chatting with larsks later when he comes online, as IIRC he wrote many of the patches related to this | 08:09 |
| *** karimb has joined #tripleo | 08:09 | |
| *** ooolpbot has joined #tripleo | 08:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674770 | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1676156 | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1676250 | 08:10 |
| *** ooolpbot has quit IRC | 08:10 | |
| openstack | Launchpad bug 1674770 in tripleo "Update timeout too long in CI" [Critical,Triaged] | 08:10 |
| openstack | Launchpad bug 1676156 in tripleo "CI: exit codes and results of shell tasks are ignored in quickstart" [Critical,Triaged] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm) | 08:10 |
| openstack | Launchpad bug 1676250 in tripleo "Promotion to OpenStack trunk fails (timeout at step3)" [Critical,Triaged] | 08:10 |
| *** gfidente has joined #tripleo | 08:16 | |
| *** gfidente has quit IRC | 08:16 | |
| *** gfidente has joined #tripleo | 08:16 | |
| *** athomas has joined #tripleo | 08:16 | |
| *** dixiaoli has joined #tripleo | 08:16 | |
| mandre | jaosorior: the keystone image with mod_ssl I uploaded to the registy last week was failing in a strange way, I had to revert it | 08:18 |
| mandre | jaosorior: were you able to get it to work locally? | 08:19 |
| jaosorior | mandre: I didn't get that far. Had another heat-related issue which I'm trying to debug at the moment. | 08:22 |
| jaosorior | mandre: what was the error with the iamge? | 08:22 |
| jaosorior | *image | 08:22 |
| *** dparkes has joined #tripleo | 08:22 | |
| *** zoli|gone is now known as zoli|wfh | 08:26 | |
| *** ealcaniz has quit IRC | 08:28 | |
| *** stendulker has joined #tripleo | 08:29 | |
| mandre | jaosorior: I don't remember exactly what was the error I got, but here is the bug that was reported downstream https://bugzilla.redhat.com/show_bug.cgi?id=1435757 | 08:31 |
| openstack | bugzilla.redhat.com bug 1435757 in openstack-containers "[openstack containers] HA deployment with containers on overcloud fails during ControllerContainersDeployment_Step3 ." [High,New] - Assigned to m.andre | 08:31 |
| jaosorior | damn | 08:31 |
| mandre | jaosorior: I just kicked off another deployment | 08:31 |
| mandre | we'll see | 08:31 |
| *** lucas-afk is now known as lucasagomes | 08:32 | |
| *** openstackgerrit has quit IRC | 08:33 | |
| *** derekh has joined #tripleo | 08:37 | |
| *** openstackgerrit has joined #tripleo | 08:43 | |
| openstackgerrit | Gael Chamoulaud proposed openstack/tripleo-quickstart master: Add blank newline at the end of file https://review.openstack.org/449035 | 08:43 |
| *** ckyriakidou has joined #tripleo | 08:43 | |
| openstackgerrit | Gael Chamoulaud proposed openstack/tripleo-quickstart-extras master: Add blank newline at the end of file https://review.openstack.org/449028 | 08:43 |
| openstackgerrit | Gael Chamoulaud proposed openstack/tripleo-quickstart-extras master: Fix ansible-lint.sh to check playbooks https://review.openstack.org/446525 | 08:44 |
| openstackgerrit | Gael Chamoulaud proposed openstack/tripleo-quickstart-extras master: [overcloud-prep-images] Update deprecated openstack command https://review.openstack.org/440518 | 08:44 |
| *** akrivoka has joined #tripleo | 08:44 | |
| *** ccamacho has joined #tripleo | 08:44 | |
| openstackgerrit | Gael Chamoulaud proposed openstack/tripleo-quickstart-extras master: [overcloud-deploy] Fix hosts file generation https://review.openstack.org/438625 | 08:45 |
| *** akrivoka has quit IRC | 08:46 | |
| openstackgerrit | Gael Chamoulaud proposed openstack/tripleo-quickstart master: Implement fact caching https://review.openstack.org/448478 | 08:47 |
| openstackgerrit | Gael Chamoulaud proposed openstack/tripleo-quickstart master: Enable ansible pipelining https://review.openstack.org/446582 | 08:47 |
| openstackgerrit | Gael Chamoulaud proposed openstack/tripleo-quickstart master: Add ability to run tripleo-validations tests https://review.openstack.org/403731 | 08:47 |
| openstackgerrit | Gael Chamoulaud proposed openstack/tripleo-quickstart-extras master: Add new role for tripleo-validations https://review.openstack.org/403576 | 08:47 |
| *** akrivoka has joined #tripleo | 08:49 | |
| *** salmankhan has joined #tripleo | 08:49 | |
| *** dparkes has quit IRC | 08:52 | |
| amoralej | sshnaidm, in status-tripleoci.rhcloud.com there is not info for periodic jobs since 21st, are they disabled? | 08:57 |
| openstackgerrit | Numan Siddique proposed openstack/puppet-tripleo master: Pacemaker support for OVN DB servers https://review.openstack.org/372274 | 08:58 |
| openstackgerrit | Gael Chamoulaud proposed openstack/tripleo-quickstart-extras master: Add new role for tripleo-validations https://review.openstack.org/403576 | 08:58 |
| mandre | jaosorior: the keystone container complains about an invalid SSLPassPhraseDialog command, and ends up in a restarting loop | 08:58 |
| mandre | jaosorior: http://paste.openstack.org/show/604269/ | 08:58 |
| openstackgerrit | Gael Chamoulaud proposed openstack/tripleo-quickstart master: Add ability to run tripleo-validations tests https://review.openstack.org/403731 | 08:59 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/tripleo-ui stable/ocata: Imported Translations from Zanata https://review.openstack.org/449988 | 08:59 |
| jaosorior | mandre: wasn't that the issue we talked about in IRC? That you tried to enable TLS but were missing the CA bits. | 09:01 |
| mandre | jaosorior: it doesn't seem to be the case with the deploy-centos job at https://review.openstack.org/#/c/446911/, strange | 09:01 |
| jaosorior | mandre: are you seeing that SSLPassPhraseDialog in regular deployments with that image? | 09:02 |
| openstackgerrit | Ana Krivokapic proposed openstack/tripleo-common master: Add plan export action and workflow https://review.openstack.org/422789 | 09:02 |
| mandre | jaosorior: same error indeed, and yes, regular deployment | 09:02 |
| jaosorior | what the hell | 09:02 |
| jaosorior | mandre: let me know if you can reproduce it | 09:03 |
| jaosorior | shardy: when one does a get_param on a resource that's OS::Heat::None; what does it output? | 09:03 |
| shardy | jaosorior: assuming you mean get_attr, it returns None/null | 09:04 |
| shardy | as implied by the name ;) | 09:04 |
| jaosorior | shardy: it was get_attr, right | 09:04 |
| jaosorior | shardy: how would one output None or null explicitly in a heat template? | 09:04 |
| shardy | jaosorior: just output a null yaml value? | 09:05 |
| jaosorior | shardy: oh; that makes sense | 09:05 |
| jaosorior | shardy: thanks | 09:05 |
| *** apetrich has quit IRC | 09:06 | |
| jaosorior | shardy: for the TLS-everywhere bits I usually have nested resources for outputting extra config_settings and metadata_settings. I was thinking of getting rid of those and replacing it with conditionals; with the idea that that would be less memory intensive than having an extra stack. What do you think? | 09:06 |
| *** limao has joined #tripleo | 09:07 | |
| *** bogdando has joined #tripleo | 09:09 | |
| *** dparkes has joined #tripleo | 09:09 | |
| *** ooolpbot has joined #tripleo | 09:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674770 | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1676156 | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1676250 | 09:10 |
| *** ooolpbot has quit IRC | 09:10 | |
| openstack | Launchpad bug 1674770 in tripleo "Update timeout too long in CI" [Critical,Triaged] | 09:10 |
| openstack | Launchpad bug 1676156 in tripleo "CI: exit codes and results of shell tasks are ignored in quickstart" [Critical,Triaged] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm) | 09:10 |
| openstack | Launchpad bug 1676250 in tripleo "Promotion to OpenStack trunk fails (timeout at step3)" [Critical,Triaged] | 09:10 |
| *** arxcruz has quit IRC | 09:11 | |
| *** limao has quit IRC | 09:11 | |
| *** limao has joined #tripleo | 09:13 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Rabbitmq: Use conditional instead of nested stack for TLS-specific bits https://review.openstack.org/450135 | 09:14 |
| jaosorior | shardy: something like this ^^ | 09:14 |
| sshnaidm | amoralej, periodic jobs run, although there is problem with status page right now | 09:15 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: DO NOT MERGE: Testing conditional instead of nested stack for rabbitmq https://review.openstack.org/450137 | 09:15 |
| *** nyechiel has joined #tripleo | 09:16 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: DO NOT MERGE: Testing conditional instead of nested stack for rabbitmq https://review.openstack.org/450137 | 09:16 |
| *** ckyriakidou has quit IRC | 09:17 | |
| *** dixiaoli has quit IRC | 09:18 | |
| *** hewbrocca_afk is now known as hewbrocca | 09:19 | |
| *** dixiaoli has joined #tripleo | 09:19 | |
| *** arxcruz has joined #tripleo | 09:22 | |
| *** yamahata has quit IRC | 09:26 | |
| *** udesale__ has joined #tripleo | 09:28 | |
| *** udesale has quit IRC | 09:28 | |
| *** ckyriakidou has joined #tripleo | 09:29 | |
| *** bogdando has quit IRC | 09:30 | |
| *** limao_ has joined #tripleo | 09:31 | |
| *** limao_ has quit IRC | 09:31 | |
| *** bogdando has joined #tripleo | 09:31 | |
| *** limao_ has joined #tripleo | 09:32 | |
| *** limao has quit IRC | 09:34 | |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates stable/ocata: N->O Upgrade, make sure all nova placement parameter properly set. https://review.openstack.org/450142 | 09:36 |
| *** udesale__ has quit IRC | 09:36 | |
| *** udesale has joined #tripleo | 09:36 | |
| openstackgerrit | Luke Hinds proposed openstack/puppet-tripleo master: SSHD Service extensions https://review.openstack.org/443113 | 09:37 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates master: N->O upgrade, blanks ipv6 rules before activating it. https://review.openstack.org/449613 | 09:38 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates stable/ocata: N->O upgrade, blanks ipv6 rules before activating it. https://review.openstack.org/450144 | 09:39 |
| *** tosky has joined #tripleo | 09:45 | |
| jaosorior | shardy: also, have you seen something like this http://logs.openstack.org/70/449570/2/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/496aca8/console.html#_2017-03-27_08_10_12_757655 ? I was trying to test TLS-everywhere and trying to make it work with containers; but I stumbled upon that when enabling it in keystone. | 09:45 |
| openstackgerrit | Merged openstack/tripleo-docs master: Small fixups for the TLS everywhere documentation https://review.openstack.org/449672 | 09:49 |
| *** limao_ has quit IRC | 09:53 | |
| *** limao has joined #tripleo | 09:54 | |
| shardy | jaosorior: Yes there's a stack nesting limit in heat.conf which is evidently being exceeded | 09:54 |
| jaosorior | shardy: I didn't know of the stack nesting limit :/ | 09:55 |
| shardy | we should probably change the message to say that instead of "Recursion" | 09:55 |
| shardy | IMO we probably need to look at the tht approach if we're getting to 7, but that's the reason for the error | 09:55 |
| jaosorior | shardy: so. I'm getting that when ebaling the TLS-everywhere bits with containers | 09:56 |
| pliu | Anyone can help to give workflow to https://review.openstack.org/#/c/444050/ and https://review.openstack.org/#/c/447429/? much appreciate. | 09:57 |
| shardy | gate-tripleo-ci-centos-7-multinode-upgrades-nvSUCCESS in 11m 58s | 09:58 |
| shardy | that can't be right... | 09:58 |
| b00tcat | Hi, is the port "OS::TripleO::Controller::Ports::RedisVipPort" mandatory in the Controller role? I've seen it in many network environment files (if not all) | 09:59 |
| *** paramite has joined #tripleo | 09:59 | |
| jaosorior | shardy: that's because in a usual TLS deployment. Lets say for keystone: puppet/services/keystone.yaml has a nested stack for apache-base, which has a nested stack for the TLS bits (called apacheTLS). In containers, there's docker/services/keystone.yaml which then has a nested stack for keystone-base (which is puppet/services/keystone.yaml.... So this is where it comes from | 09:59 |
| shardy | Oh I see you've got an exit 0 so those tests are disabled | 09:59 |
| jaosorior | shardy: right. sorry I didn't mention that. It's just a test patch that runs on the ha job. | 10:00 |
| jaosorior | shardy: and skips the rest | 10:00 |
| jaosorior | that's why I have it with a -2 | 10:00 |
| shardy | jaosorior: yeah, so short term the solution is to increase the limit, but we perhaps want to look at ways to flatten if possible | 10:01 |
| shardy | jaosorior: Ok cool, just looked a the elapsed time and was like *what*?! :) | 10:01 |
| jaosorior | shardy: ok; I'll attempt to flatten it then. | 10:02 |
| jaosorior | shardy: This is what I came up with for the rabbitmq TLS bits. It uses conditionals instead of the nested stack https://review.openstack.org/#/c/450135/ | 10:03 |
| shardy | jaosorior: ah yeah that's a nice approach | 10:04 |
| jaosorior | ok; if that one passes I'll move the rest of the TLS bits to use that then | 10:05 |
| jaosorior | thanks for checking it out | 10:05 |
| shardy | sounds good, np | 10:05 |
| *** dixiaoli has quit IRC | 10:06 | |
| saneax | thanks shardy, I will raise the bug as well to track this | 10:08 |
| pliu | shardy, may I ask if you could give workflow to https://review.openstack.org/#/c/444050/ and https://review.openstack.org/#/c/447429/? | 10:08 |
| *** ooolpbot has joined #tripleo | 10:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674770 | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1676156 | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1676250 | 10:10 |
| *** ooolpbot has quit IRC | 10:10 | |
| openstack | Launchpad bug 1674770 in tripleo "Update timeout too long in CI" [Critical,Triaged] | 10:10 |
| openstack | Launchpad bug 1676156 in tripleo "CI: exit codes and results of shell tasks are ignored in quickstart" [Critical,Triaged] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm) | 10:10 |
| openstack | Launchpad bug 1676250 in tripleo "Promotion to OpenStack trunk fails (timeout at step3)" [Critical,Triaged] | 10:10 |
| jaosorior | mandre: do you need help debugging the mod_ssl issue? | 10:10 |
| *** dixiaoli has joined #tripleo | 10:11 | |
| chem | matbu apetrich lbezdick mcornea marios yprokule ccamacho rbartal amit_u: hi, I made a mistake sending the invite. 2:30 would be better for EST people, so if it's ok with mcornea and ccamacho, I would like to move the appointement for bugtriage to 2:30, ie, half an hour latter | 10:12 |
| mcornea | chem: it's ok for me | 10:13 |
| ccamacho | chem sure thing | 10:13 |
| chem | oki, thanks | 10:13 |
| marios | chem: ack thanks wfm | 10:13 |
| shardy | pliu: I approved the puppet-tripleo one, we can approve the t-h-t one after it merges | 10:13 |
| pliu | shardy, thanks a lot. | 10:14 |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-ui master: Add favicons webpack plugin https://review.openstack.org/450151 | 10:15 |
| *** limao has quit IRC | 10:15 | |
| *** limao has joined #tripleo | 10:16 | |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-ui master: Add favicon icons https://review.openstack.org/420111 | 10:22 |
| openstackgerrit | Saravanan KR proposed openstack/tripleo-heat-templates master: Modify pci_passthrough heira value as string https://review.openstack.org/448600 | 10:24 |
| *** milan has joined #tripleo | 10:29 | |
| openstackgerrit | Alfredo Moralejo proposed openstack/tripleo-quickstart-extras master: Use new register nodes commands for newton and newer https://review.openstack.org/449160 | 10:29 |
| *** fzdarsky has joined #tripleo | 10:30 | |
| *** udesale has quit IRC | 10:31 | |
| bogdando | oooq folks, could you please merge https://review.openstack.org/#/c/448294/ ? | 10:32 |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-ui master: Add favicons webpack plugin https://review.openstack.org/450151 | 10:32 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Rabbitmq: Use conditional instead of nested stack for TLS-specific bits https://review.openstack.org/450135 | 10:33 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Apache: Use conditional instead of nested stack for TLS-specific bits https://review.openstack.org/450154 | 10:33 |
| *** yatinkarel has quit IRC | 10:34 | |
| *** panda|pto is now known as panda | 10:35 | |
| *** limao has quit IRC | 10:37 | |
| *** limao has joined #tripleo | 10:37 | |
| *** deadnull has quit IRC | 10:37 | |
| *** dixiaoli has quit IRC | 10:37 | |
| *** dixiaoli has joined #tripleo | 10:38 | |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-quickstart-extras master: Write openstack bash completion for undercloud https://review.openstack.org/448606 | 10:39 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-quickstart master: Define libvirt pool as a common meta role https://review.openstack.org/448543 | 10:40 |
| bogdando | PTAL https://review.openstack.org/#/c/449552/ | 10:41 |
| bogdando | two more, enabling the wrapper containers for tripleo https://review.openstack.org/#/c/447409/ https://review.openstack.org/#/c/447000/. A minor thing that allow folks on arbitrary OS types to run oooq from a centos7 container | 10:43 |
| *** fzdarsky has quit IRC | 10:43 | |
| sshnaidm | fyi https://bugs.launchpad.net/tripleo/+bug/1676369 | 10:44 |
| openstack | Launchpad bug 1676369 in tripleo "quickstart: Required an ansible lint rule which ensures we use pipefail with pipes in shell module" [High,Triaged] | 10:44 |
| *** yatinkarel has joined #tripleo | 10:47 | |
| *** nmathew has joined #tripleo | 10:50 | |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-quickstart-extras master: Ensure working dirs to exist https://review.openstack.org/448991 | 10:51 |
| sshnaidm | adarazs, also take a look in your time please: https://review.openstack.org/#/c/449562/ | 10:51 |
| sshnaidm | panda, ^ | 10:52 |
| *** ealcaniz has joined #tripleo | 10:52 | |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: Include gate.repo in image building if exists https://review.openstack.org/445951 | 10:52 |
| *** yatinkarel has quit IRC | 10:53 | |
| cmyster | chem: I thought we wanted the triagers to set a time between themselves? | 10:56 |
| cmyster | chem: also, we got an answer for https://bugzilla.redhat.com/show_bug.cgi?id=1432571 | 10:57 |
| openstack | bugzilla.redhat.com bug 1432571 in rhosp-director "[OSP10] openstack overcloud upgrade stuck after updating controller node and fails" [High,New] - Assigned to sathlang | 10:57 |
| *** florianf has quit IRC | 10:57 | |
| *** limao has quit IRC | 10:58 | |
| *** limao has joined #tripleo | 10:59 | |
| openstackgerrit | Oliver Walsh proposed openstack/tripleo-heat-templates master: WIP: SSH known_hosts config https://review.openstack.org/449660 | 10:59 |
| *** lucasagomes is now known as lucas-hungry | 11:00 | |
| *** ccamacho is now known as ccamacho|lunch | 11:02 | |
| *** yatinkarel has joined #tripleo | 11:05 | |
| *** dixiaoli has quit IRC | 11:06 | |
| *** apetrich has joined #tripleo | 11:07 | |
| *** ansmith has quit IRC | 11:08 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: MySQL: Use conditional instead of nested stack for TLS-specific bits https://review.openstack.org/450168 | 11:09 |
| *** ooolpbot has joined #tripleo | 11:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674770 | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1676156 | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1676250 | 11:10 |
| *** ooolpbot has quit IRC | 11:10 | |
| openstack | Launchpad bug 1674770 in tripleo "Update timeout too long in CI" [Critical,Triaged] | 11:10 |
| openstack | Launchpad bug 1676156 in tripleo "CI: exit codes and results of shell tasks are ignored in quickstart" [Critical,Triaged] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm) | 11:10 |
| openstack | Launchpad bug 1676250 in tripleo "Promotion to OpenStack trunk fails (timeout at step3)" [Critical,Triaged] | 11:10 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: DO NOT MERGE: Testing conditional instead of nested stack for apache https://review.openstack.org/450169 | 11:10 |
| *** nmathew has quit IRC | 11:12 | |
| *** thrash|g0ne is now known as thrash | 11:13 | |
| *** adarazs is now known as adarazs_lunch | 11:14 | |
| *** florianf has joined #tripleo | 11:16 | |
| openstackgerrit | Luke Hinds proposed openstack/tripleo-specs master: blueprint for TripleO management of AIDE https://review.openstack.org/437872 | 11:18 |
| *** limao has quit IRC | 11:19 | |
| openstackgerrit | Luke Hinds proposed openstack/tripleo-specs master: blueprint for TripleO management of AIDE https://review.openstack.org/437872 | 11:20 |
| *** limao has joined #tripleo | 11:20 | |
| *** yatinkarel has quit IRC | 11:24 | |
| *** dixiaoli_ has joined #tripleo | 11:25 | |
| *** stendulker has quit IRC | 11:26 | |
| *** eck`gone is now known as eck` | 11:28 | |
| jaosorior | EmilienM: hey dude, last week we talked about this patch https://review.openstack.org/#/c/449536/3/manifests/certmonger/httpd.pp where I'm trying to ensure that those directories exist and contain the right seilnux tags. That resource is called several times. And even though I'm using ensure_resource I still get "duplicate resource declaration" any ideas? | 11:28 |
| *** florianf has quit IRC | 11:31 | |
| *** florianf has joined #tripleo | 11:31 | |
| *** yatinkarel has joined #tripleo | 11:36 | |
| *** snecklifter has quit IRC | 11:37 | |
| mandre | jaosorior: so it looks like we have an ssl.conf but we're never loading the ssl module | 11:37 |
| jaosorior | mandre: thought that would be part of ssl.conf | 11:38 |
| jaosorior | mandre: also, that should have been done by the puppet module | 11:38 |
| mandre | jaosorior: I would expect to see an ssl.load file in /etc/httpd/conf.modules.d | 11:39 |
| *** limao_ has joined #tripleo | 11:41 | |
| *** limao has quit IRC | 11:42 | |
| jaosorior | mandre: let me check how it's done in puppetlabs-apache | 11:42 |
| *** bfournie has quit IRC | 11:44 | |
| jaosorior | mandre: ok. So the issue is that we only load the SSL module when we enable TLS. | 11:44 |
| *** dixiaoli_ has quit IRC | 11:44 | |
| jaosorior | mandre: Seems to me that the correct thing to do is to remove the ssl.conf entirely. Since that's managed by puppet anyway | 11:45 |
| jaosorior | mandre: now, I', not entirely sure if we should do that in our template files, or in the dockerfile in kolla | 11:46 |
| mandre | jaosorior: aha! we have some extra files in the container http://paste.openstack.org/show/604287/ | 11:46 |
| jaosorior | mandre: right, and those are usually purged by puppet | 11:47 |
| *** dixiaoli has joined #tripleo | 11:48 | |
| mandre | jaosorior: right... and we're copying the files while we should be bind mounting the directory | 11:49 |
| EmilienM | jaosorior: did the resources has same properties everytime it's called? | 11:49 |
| jaosorior | EmilienM: yes | 11:49 |
| openstackgerrit | Arx Cruz proposed openstack/tripleo-quickstart-extras master: Altering static home directory to working_dir var https://review.openstack.org/450177 | 11:50 |
| *** morazi has joined #tripleo | 11:50 | |
| jaosorior | EmilienM: if you check the patch, the resource parameters are even strings there, they don't even come from variables. The names should have been the same as well. | 11:50 |
| EmilienM | jaosorior: ok, I'll look today | 11:50 |
| jaosorior | EmilienM: the namevar which in this case is the file path is the only thing coming from a variable | 11:50 |
| jaosorior | mandre: yet, the ssl.load won't be copied because we do < ...::ssl: false > if TLS everywhere is not enabled. | 11:51 |
| jaosorior | mandre: anyway, should we remove ssl.conf in either kolla or the tripleo templates. What do you think? | 11:53 |
| openstackgerrit | OpenStack Release Bot proposed openstack/tripleo-validations stable/ocata: Update .gitreview for stable/ocata https://review.openstack.org/450178 | 11:55 |
| openstackgerrit | OpenStack Release Bot proposed openstack/tripleo-validations stable/ocata: Update UPPER_CONSTRAINTS_FILE for stable/ocata https://review.openstack.org/450179 | 11:55 |
| openstackgerrit | OpenStack Release Bot proposed openstack/tripleo-validations master: Update reno for stable/ocata https://review.openstack.org/450180 | 11:55 |
| mandre | jaosorior: I think the easiest is simply to bind mount the /etc/httpd directory in the keystone container | 11:56 |
| *** jayg|g0n3 is now known as jayg | 11:56 | |
| jaosorior | mandre: I'm not sure if we'll gain much from that. ssl.conf will still blow up. | 11:57 |
| mandre | jaosorior: nah, because we won't have an ssl.conf in that case | 11:59 |
| *** abishop has joined #tripleo | 11:59 | |
| mandre | jaosorior: it's absent from /var/lib/config-data/keystone/etc/httpd/conf.d on the host (it was properly deleted by puppet) | 11:59 |
| jaosorior | mandre: if we don't have ssl.conf. Then I'm confused on what's causing the blowup. Since nothing should be trying to load mod_ssl-related bits | 12:00 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-docs master: Basic structure of TripleO Deployment Guide https://review.openstack.org/449684 | 12:00 |
| EmilienM | mandre, jrist, florianf: https://review.openstack.org/#/c/449678/ merged - the branch has been created! | 12:02 |
| *** limao_ has quit IRC | 12:02 | |
| florianf | EmilienM: Thanks a lot! | 12:02 |
| EmilienM | mandre, jrist, florianf: can you please check why https://review.openstack.org/#/c/450178/ is failing? | 12:03 |
| *** lucas-hungry is now known as lucasagomes | 12:03 | |
| *** limao has joined #tripleo | 12:03 | |
| florianf | EmilienM: I'll have a look | 12:03 |
| EmilienM | thanks | 12:03 |
| EmilienM | sshnaidm: hey good afternoon! do we have any clue on https://bugs.launchpad.net/tripleo/+bug/1676250 yet? | 12:04 |
| openstack | Launchpad bug 1676250 in tripleo "Promotion to OpenStack trunk fails (timeout at step3)" [Critical,Triaged] | 12:04 |
| mandre | jaosorior: we do have ssl.conf in the container because we're copying the files from /var/lib/config-data/keystone/etc/httpd/conf.d to the container while we should bind mount the directory instead | 12:04 |
| sshnaidm | EmilienM, hey, still not, trying to reproduce | 12:05 |
| openstackgerrit | Arx Cruz proposed openstack/tripleo-quickstart-extras master: Altering static home directory to working_dir var https://review.openstack.org/450177 | 12:05 |
| sshnaidm | EmilienM, looking at logs, but no success yet.. | 12:05 |
| mandre | jaosorior: I think this may fix our issue https://review.openstack.org/#/c/447676/ | 12:06 |
| openstackgerrit | Merged openstack/tripleo-validations master: Update reno for stable/ocata https://review.openstack.org/450180 | 12:06 |
| mandre | jaosorior: we'll need to do it for all our containers | 12:06 |
| jaosorior | mandre: I don't see it doing much for /etc/httpd | 12:06 |
| jaosorior | mandre: instead it only addresses /var/www | 12:07 |
| jaosorior | oh wait | 12:07 |
| openstackgerrit | Dan Radez proposed openstack/tripleo-heat-templates stable/ocata: Setting keystone region for congress https://review.openstack.org/450182 | 12:07 |
| jaosorior | mandre: I missed it | 12:07 |
| jaosorior | right | 12:07 |
| jaosorior | it goes for the whole /etc/httpd instead of just /etc/httpd/conf.modules.d | 12:07 |
| jaosorior | mandre: I think it'll work | 12:07 |
| *** yatinkarel has quit IRC | 12:08 | |
| *** dougbtv has joined #tripleo | 12:08 | |
| *** ooolpbot has joined #tripleo | 12:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674770 | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1676156 | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1676250 | 12:10 |
| *** ooolpbot has quit IRC | 12:10 | |
| openstack | Launchpad bug 1674770 in tripleo "Update timeout too long in CI" [Critical,Triaged] | 12:10 |
| *** trown|outtypewww is now known as trown | 12:10 | |
| openstack | Launchpad bug 1676156 in tripleo "CI: exit codes and results of shell tasks are ignored in quickstart" [Critical,Triaged] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm) | 12:10 |
| *** dprince has joined #tripleo | 12:10 | |
| openstack | Launchpad bug 1676250 in tripleo "Promotion to OpenStack trunk fails (timeout at step3)" [Critical,Triaged] | 12:10 |
| mandre | jaosorior: just kicked off a deployment | 12:10 |
| *** ratailor has quit IRC | 12:15 | |
| *** apetrich has quit IRC | 12:15 | |
| *** amoralej is now known as amoralej|lunch | 12:16 | |
| *** ccamacho|lunch is now known as ccamacho | 12:16 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/instack master: Updated from global requirements https://review.openstack.org/431951 | 12:17 |
| *** dixiaoli has quit IRC | 12:18 | |
| *** bfournie has joined #tripleo | 12:20 | |
| *** yatinkarel has joined #tripleo | 12:20 | |
| *** masco has quit IRC | 12:22 | |
| *** limao has quit IRC | 12:23 | |
| *** limao has joined #tripleo | 12:24 | |
| Ng | EmilienM: picking up from last week, https://review.openstack.org/#/c/442497/ seems to be in a slightly odd state where one voting job failed, but jenkins still has a +1 (and said failed update was the ovb-updates job, which I think only failed because of 1674770) - any chance we can merge it? :D | 12:24 |
| *** gkadam has quit IRC | 12:25 | |
| EmilienM | Ng: why ovb-updates is failing? | 12:26 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/tripleo-heat-templates master: Updated from global requirements https://review.openstack.org/440124 | 12:26 |
| Ng | EmilienM: the console log ends with the UPDATE_COMPLETE completing successfully, but then the jenkins script tills it a minute later for a timeout, afaics | 12:26 |
| EmilienM | ok | 12:27 |
| openstackgerrit | Luke Hinds proposed openstack/puppet-tripleo master: SSHD Service extensions https://review.openstack.org/443113 | 12:27 |
| *** vpickard_ is now known as vpickard | 12:27 | |
| shardy | d0ugal: Hey could you check https://review.openstack.org/#/c/446045 please? | 12:28 |
| *** yatinkarel has quit IRC | 12:28 | |
| openstackgerrit | Luke Hinds proposed openstack/puppet-tripleo master: SSHD Service extensions https://review.openstack.org/443113 | 12:28 |
| shardy | d0ugal: I'm about to rebase that series, and it'd be nice to land that one vs run it through CI again | 12:29 |
| *** dsariel has quit IRC | 12:30 | |
| shardy | ^^ and any other folks who like reviewing tripleoclient code :) | 12:30 |
| d0ugal | shardy: sure, looking | 12:30 |
| lhinds | whats the best way to remove a package, ensure => 'absent', or is there already a list that can be added to that iterates over pkgs to be removed? | 12:30 |
| shardy | lhinds: the package provide is disabled by default, would it be better to remove it from the image? | 12:31 |
| shardy | sorry package provider | 12:31 |
| d0ugal | shardy: is CI working now? I've not looked since first thing, but the gate was failing for almost everything | 12:31 |
| lhinds | shardy: its the telnet client, which no one uses, but gets us a compliance tick. I know you can add pkgs in DIB, not seen anything to remove. Do you know of where thats done? | 12:32 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-quickstart-extras master: undercloud/heat: Set workers correctly for httpd https://review.openstack.org/450191 | 12:32 |
| *** lmiccini has quit IRC | 12:33 | |
| *** jmelvin has joined #tripleo | 12:33 | |
| EmilienM | I'm seeing network issue with ipv6 on rh1, http://logs.openstack.org/27/446927/1/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/d602f1f/console.html#_2017-03-27_10_09_20_185265 and http://logs.openstack.org/27/446927/1/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/d602f1f/console.html#_2017-03-27_10_18_39_088665 | 12:34 |
| EmilienM | beekneemech, sshnaidm, panda : we have ipv6 link on rh1, right? | 12:35 |
| *** dmacpher has joined #tripleo | 12:36 | |
| shardy | lhinds: I think package-installs can do it https://docs.openstack.org/developer/diskimage-builder/elements/package-installs/README.html | 12:37 |
| panda | EmilienM: no idea, but everything works when dns returns IPv4 address for the git repo. | 12:37 |
| lhinds | thx shardy , this looks like the ticket | 12:37 |
| shardy | lhinds: we seem to be using the old/deprecated format, but e.g see tripleo-puppet-elements/elements/overcloud-controller/install.d/package-installs-overcloud-controller | 12:37 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Pick dynamically the first node for stack validation https://review.openstack.org/446887 | 12:38 |
| *** nenad has joined #tripleo | 12:38 | |
| *** jpena is now known as jpena|lunch | 12:39 | |
| openstackgerrit | Merged openstack/tripleo-ui stable/ocata: Imported Translations from Zanata https://review.openstack.org/449988 | 12:39 |
| *** links has quit IRC | 12:40 | |
| *** ansmith has joined #tripleo | 12:40 | |
| sshnaidm | EmilienM, I think we don't have | 12:41 |
| openstackgerrit | Arx Cruz proposed openstack/tripleo-quickstart-extras master: Send mail tool https://review.openstack.org/423340 | 12:41 |
| *** Goneri has joined #tripleo | 12:41 | |
| *** psahoo has quit IRC | 12:42 | |
| *** yatinkarel has joined #tripleo | 12:42 | |
| *** rlandy has joined #tripleo | 12:43 | |
| *** dsariel has joined #tripleo | 12:43 | |
| openstackgerrit | Merged openstack/puppet-tripleo master: Add l2 gateway Neutron service plugin profile https://review.openstack.org/444050 | 12:44 |
| openstackgerrit | Merged openstack/tripleo-common master: Rename 'uploads' key to 'container_images' https://review.openstack.org/447323 | 12:45 |
| *** limao_ has joined #tripleo | 12:45 | |
| *** lmiccini has joined #tripleo | 12:45 | |
| openstackgerrit | mathieu bultel proposed openstack/tripleo-quickstart-extras master: Allow complex upgrade deployment for N to O https://review.openstack.org/439598 | 12:46 |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: Add pipefail to each command that piped with timestamp https://review.openstack.org/450023 | 12:46 |
| *** adarazs_lunch is now known as adarazs | 12:47 | |
| EmilienM | sshnaidm: I'm wondering why the centos-ceph-jewel repo is deployed with ipv6 | 12:47 |
| *** limao has quit IRC | 12:47 | |
| sshnaidm | EmilienM, why not? | 12:48 |
| *** thrash is now known as thrash|biab | 12:48 | |
| EmilienM | sshnaidm: in any case, we should use the openstack mirrors, we have jewel packaging in AFS | 12:48 |
| EmilienM | puppet ci is using it | 12:48 |
| EmilienM | pabelanger: are you working on this one^ ? (just to make sure I don't start looking at it) | 12:49 |
| weshay | adarazs, sshnaidm, matbu anyone know why dell-ironic ci is listed on this review of tq? https://review.openstack.org/#/c/410831/ | 12:50 |
| *** chlong has joined #tripleo | 12:50 | |
| *** michapma_dsk has joined #tripleo | 12:51 | |
| openstackgerrit | Chris Jones proposed openstack/tripleo-common master: Extend testing for GenerateFencingParamatersAction. https://review.openstack.org/450199 | 12:52 |
| adarazs | weshay: I have no clue why it's called like that, or what that account is. | 12:52 |
| jaosorior | weshay: http://lists.openstack.org/pipermail/openstack-infra/2017-March/005261.html | 12:52 |
| adarazs | weshay: what are those gate jobs? | 12:52 |
| jaosorior | adarazs: ^^ | 12:52 |
| adarazs | oh cool, thanks jaosorior :) | 12:52 |
| EmilienM | pabelanger: please read and review https://bugs.launchpad.net/tripleo/+bug/1676421 | 12:52 |
| openstack | Launchpad bug 1676421 in tripleo "TripleO CI should deploy Ceph packages from OpenStack AFS mirrors" [High,Triaged] | 12:52 |
| adarazs | one mystery solved. | 12:53 |
| weshay | thanks | 12:53 |
| *** apetrich has joined #tripleo | 12:58 | |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates master: Swift auth url should use a suffix https://review.openstack.org/448077 | 13:00 |
| openstackgerrit | Pradeep Kilambi proposed openstack/puppet-tripleo master: Re-run gnocchi and ceilometer upgrade in step 5 https://review.openstack.org/447599 | 13:00 |
| *** amoralej|lunch is now known as amoralej | 13:00 | |
| *** jpena|lunch is now known as jpena | 13:02 | |
| *** apetrich has quit IRC | 13:03 | |
| *** jcoufal has joined #tripleo | 13:05 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: docker/keystone: Bind mount entire fernet keys repository https://review.openstack.org/446473 | 13:06 |
| *** ckyriakidou has quit IRC | 13:09 | |
| *** ooolpbot has joined #tripleo | 13:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674770 | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1676156 | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1676250 | 13:10 |
| *** ooolpbot has quit IRC | 13:10 | |
| openstack | Launchpad bug 1674770 in tripleo "Update timeout too long in CI" [Critical,Triaged] | 13:10 |
| openstack | Launchpad bug 1676156 in tripleo "CI: exit codes and results of shell tasks are ignored in quickstart" [Critical,Triaged] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm) | 13:10 |
| openstack | Launchpad bug 1676250 in tripleo "Promotion to OpenStack trunk fails (timeout at step3)" [Critical,Triaged] | 13:10 |
| *** cdearborn has joined #tripleo | 13:12 | |
| EmilienM | sshnaidm: 1674770 and 1676156 should still be open? | 13:15 |
| *** apetrich has joined #tripleo | 13:15 | |
| *** gbarros has joined #tripleo | 13:16 | |
| *** garyk has joined #tripleo | 13:17 | |
| garyk | Wonder if someone can help me. I am using RHOS 10 director and trying to create overcloud. Here something is not working. Can someone please maybe take alook and let me know what I am doing wrong - https://paste.fedoraproject.org/paste/zKOYF0zpkuCrQyn9Dj7BBl5M1UNdIGYhyRLivL9gydE= | 13:18 |
| openstackgerrit | Pradeep Kilambi proposed openstack/instack-undercloud stable/ocata: Set OS_AUTH_TYPE on undercloud stackrc https://review.openstack.org/450213 | 13:18 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Run nova-api hosts discovery after nova-compute start https://review.openstack.org/448575 | 13:19 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-quickstart-extras master: Add option to enable/disable TripleO UI in the undercloud https://review.openstack.org/450214 | 13:19 |
| *** toure|gone is now known as toure | 13:19 | |
| openstackgerrit | Julien Danjou proposed openstack/tripleo-common stable/ocata: overcloudrc: set OS_AUTH_TYPE https://review.openstack.org/450215 | 13:20 |
| *** liverpooler has joined #tripleo | 13:21 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart-extras master: Use openstack deploy to install the undercloud https://review.openstack.org/419040 | 13:22 |
| *** tzumainn has joined #tripleo | 13:23 | |
| *** ckyriakidou has joined #tripleo | 13:24 | |
| *** maeca1 has joined #tripleo | 13:25 | |
| *** thrash|biab is now known as thrash | 13:27 | |
| *** michapma_dsk has quit IRC | 13:28 | |
| pabelanger | EmilienM: yes! comment left | 13:30 |
| *** rbrady-afk is now known as rbrady | 13:30 | |
| *** jprovazn has quit IRC | 13:31 | |
| jrist | florianf: any ideas why it is failing? | 13:33 |
| honza | big shoutout to jpich for always correcting my mistakes, thanks for keeping me honest | 13:33 |
| florianf | jrist: my google research hints to a race condition. ;-) rechecks seem to confirm that. | 13:34 |
| jpich | Heh, any time?? You're most welcome :) | 13:34 |
| jrist | florianf: hmm | 13:36 |
| florianf | jrist: this one: https://github.com/pypa/setuptools/issues/951 | 13:36 |
| *** rsquared has quit IRC | 13:36 | |
| jrist | *boggle* | 13:37 |
| jrist | https://github.com/pypa/pip/pull/4294 ? | 13:37 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/instack-undercloud master: Add missing project name for novajoin https://review.openstack.org/450221 | 13:37 |
| *** dsariel has quit IRC | 13:41 | |
| jrist | florianf: ha, at the bottom of that is a bunch of openstack commits referring to that issue | 13:41 |
| florianf | jrist: yep | 13:42 |
| openstackgerrit | Saravanan KR proposed openstack/tripleo-quickstart-extras master: Add the option '--long' to stack failures list https://review.openstack.org/450230 | 13:44 |
| garyk | EmilienM: this is what I get - Failed to validate power driver interface for node 691058f2-71f0-4b01-b9c3-99f35dfa26c5. Error: SSH connection cannot be established: Failed to establish SSH connection to host 192.168.126.254. | 13:44 |
| garyk | so something is incorrect with my confi. | 13:44 |
| *** garyk has quit IRC | 13:46 | |
| slagle | does anyone know about the "Dell Ironic CI" reporting on some tripleo patches? | 13:46 |
| jaosorior | slagle: http://lists.openstack.org/pipermail/openstack-infra/2017-March/005261.html | 13:47 |
| slagle | jaosorior: ah, thx. figured i'd missed something :) | 13:48 |
| openstackgerrit | Florian Fuchs proposed openstack/tripleo-validations master: [WIP]Show all roles/services in inventory https://review.openstack.org/450233 | 13:48 |
| *** lblanchard has joined #tripleo | 13:49 | |
| *** limao_ has quit IRC | 13:51 | |
| *** limao has joined #tripleo | 13:51 | |
| openstackgerrit | Saravanan KR proposed openstack/tripleo-heat-templates master: Modify pci_passthrough heira value as string https://review.openstack.org/448600 | 13:51 |
| *** rbrady has quit IRC | 13:53 | |
| *** jprovazn has joined #tripleo | 13:59 | |
| openstackgerrit | yolanda.robla proposed openstack/tripleo-image-elements master: Add overcloud-secure-block-device element https://review.openstack.org/449122 | 13:59 |
| *** gkadam has joined #tripleo | 14:00 | |
| *** nmathew has joined #tripleo | 14:00 | |
| openstackgerrit | Merged openstack/python-tripleoclient master: Make fencing action parameter optional. https://review.openstack.org/442497 | 14:00 |
| *** morazi has quit IRC | 14:01 | |
| *** nmathew has quit IRC | 14:01 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: DO NOT MERGE: Testing conditional instead of nested stack for MySQL https://review.openstack.org/450246 | 14:03 |
| *** limao_ has joined #tripleo | 14:06 | |
| openstackgerrit | Chris Jones proposed openstack/python-tripleoclient stable/ocata: Make fencing action parameter optional. https://review.openstack.org/450251 | 14:08 |
| *** nyechiel has quit IRC | 14:08 | |
| *** limao has quit IRC | 14:09 | |
| Ng | apetrich: hey, got a sec? | 14:09 |
| *** chlong has quit IRC | 14:09 | |
| *** ooolpbot has joined #tripleo | 14:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674770 | 14:10 |
| openstack | Launchpad bug 1674770 in tripleo "Update timeout too long in CI" [Critical,Triaged] | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1676156 | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1676250 | 14:10 |
| *** ooolpbot has quit IRC | 14:10 | |
| openstack | Launchpad bug 1676156 in tripleo "CI: exit codes and results of shell tasks are ignored in quickstart" [Critical,Triaged] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm) | 14:10 |
| openstack | Launchpad bug 1676250 in tripleo "Promotion to OpenStack trunk fails (timeout at step3)" [Critical,Triaged] | 14:10 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart-extras master: Send mail tool https://review.openstack.org/423340 | 14:10 |
| *** nyechiel has joined #tripleo | 14:10 | |
| *** ramishra has quit IRC | 14:11 | |
| openstackgerrit | mathieu bultel proposed openstack/tripleo-quickstart master: Add composable deployment config files and mixed release https://review.openstack.org/410831 | 14:12 |
| *** skramaja has quit IRC | 14:14 | |
| jrist | akrivoka, jtomasek - can we get some reviews on https://review.openstack.org/#/c/448229/ ? | 14:14 |
| sshnaidm | EmilienM, fix for https://bugs.launchpad.net/tripleo/+bug/1676156 was merged | 14:15 |
| openstack | Launchpad bug 1676156 in tripleo "CI: exit codes and results of shell tasks are ignored in quickstart" [Critical,Triaged] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm) | 14:15 |
| apetrich | Ng, I do now | 14:15 |
| Ng | apetrich: I wanted to see if my comments on https://review.openstack.org/#/c/445570/ change your review? :) | 14:15 |
| EmilienM | sshnaidm: can we close it? | 14:16 |
| *** morazi has joined #tripleo | 14:17 | |
| apetrich | Ng, sure do. thanks for that | 14:17 |
| honza | jrist: thanks for hustling | 14:17 |
| sshnaidm | EmilienM, it should close automatically now I think | 14:17 |
| Ng | apetrich: thanks! :) | 14:17 |
| jrist | honza: small patch, but fixes a thing for automation | 14:17 |
| EmilienM | sshnaidm: it didn't. Was it only https://review.openstack.org/450023 ? | 14:18 |
| jaosorior | shardy: could you check this out https://review.openstack.org/#/c/450154/1 ? | 14:18 |
| openstackgerrit | Steven Hardy proposed openstack/python-tripleoclient master: Move clients into class constructor https://review.openstack.org/449633 | 14:19 |
| openstackgerrit | Steven Hardy proposed openstack/python-tripleoclient master: Don't track added_files in deploy environment processing https://review.openstack.org/446045 | 14:19 |
| openstackgerrit | Steven Hardy proposed openstack/python-tripleoclient master: Add breakpoint cleanup env during plan processing https://review.openstack.org/450262 | 14:19 |
| openstackgerrit | Steven Hardy proposed openstack/python-tripleoclient master: Validate NtpServers during plan processing https://review.openstack.org/450263 | 14:19 |
| openstackgerrit | Steven Hardy proposed openstack/python-tripleoclient master: Write user parameters environment to swift https://review.openstack.org/450264 | 14:19 |
| sshnaidm | EmilienM, yes, only this one. If it doesn't close today, I'll close it. | 14:20 |
| EmilienM | sshnaidm: done :) | 14:21 |
| *** dsariel has joined #tripleo | 14:23 | |
| jaosorior | shardy: the one I sent you is on top of this one https://review.openstack.org/#/c/450135/ | 14:25 |
| *** eglynn has joined #tripleo | 14:25 | |
| *** ccamacho is now known as ccamacho|brb | 14:25 | |
| jpich | honza: Btw if your earlier shout-out was about the i18n bugs, no worries at all. Until one knows about it it's not obvious how to handle them. I'm grateful to see someone else also triaging/dealing/helping with these bugs so thank you :) | 14:26 |
| *** abishop_ has joined #tripleo | 14:26 | |
| *** rsquared has joined #tripleo | 14:26 | |
| honza | jpich: It was triggered by the recheck nonsense I was doing. But you always do this kind of stuff. | 14:26 |
| eglynn | dumb question about https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/services/nova-placement.yaml#L125-L133 | 14:26 |
| eglynn | I'm wondering about the attempt to stop nova-placement at an earlier upgrade step than the installation of the relevant package | 14:26 |
| eglynn | i.e. stop in step2, install the package in step3 | 14:27 |
| jaosorior | shardy: thanks | 14:27 |
| eglynn | not being au fait with how upgrade_tasks working, that sequencing seems a tad illogical | 14:27 |
| shardy | eglynn: Hey, it's to enable re-running an upgrade when you've got beyond the yum install | 14:27 |
| jpich | honza: Hah! I've done the same thing so many times, I find same-repo dependencies very difficult to notice in the Gerrit UI. They used to be more obvious | 14:28 |
| *** abishop has quit IRC | 14:28 | |
| shardy | eglynn: you're right, it's not really needed for new services when they're not yet installed or running | 14:28 |
| shardy | eglynn: in this particular case removing it wouldn't make much difference anyway, as a bunch of other services also stop httpd | 14:29 |
| eglynn | shardy: a-ha, got it, thanks! ... the context is that we're scratching our heads over how nova-placement-api.log gets owned by root in the upgrade case only (but not in a fresh install) | 14:29 |
| *** rbrady has joined #tripleo | 14:30 | |
| *** athomas has quit IRC | 14:32 | |
| *** zoli|wfh is now known as zoli|afk | 14:32 | |
| EmilienM | mwhahaha: do you see a risk of dup res here? https://review.openstack.org/#/c/449536/3/manifests/certmonger/httpd.pp | 14:32 |
| *** gkadam has quit IRC | 14:32 | |
| mwhahaha | EmilienM: no because ensure resource | 14:32 |
| *** trozet has quit IRC | 14:32 | |
| EmilienM | mwhahaha: well, jaosorior says he has the issue | 14:33 |
| EmilienM | sshnaidm: I see a lot of that: http://logs.openstack.org/36/449536/3/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/9b4c723/console.html#_2017-03-27_06_34_12_639851 | 14:33 |
| EmilienM | sshnaidm: I filed a bug: https://bugs.launchpad.net/tripleo/+bug/1676421 | 14:33 |
| openstack | Launchpad bug 1676421 in tripleo "TripleO CI should deploy Ceph packages from OpenStack AFS mirrors" [High,Triaged] | 14:33 |
| EmilienM | sshnaidm: i'm going to work on it on high prio | 14:33 |
| sshnaidm | EmilienM, hmm.. that's something new | 14:33 |
| mwhahaha | EmilienM: interesting | 14:34 |
| yolanda | hi, question. Do you have any idea on which element/task is installing the kexec-tools package on the overcloud nodes? | 14:34 |
| akrivoka | jrist: done | 14:34 |
| yolanda | i see it installed but cannot find anything that directly brings it, also no dependencies for it | 14:34 |
| jaosorior | mwhahaha: here's the log http://logs.openstack.org/48/446348/5/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha-oooq-nv/44331c7/console.html#_2017-03-27_07_33_22_291124 | 14:34 |
| jrist | akrivoka: \o/ | 14:34 |
| EmilienM | sshnaidm: yes and sounds critical, we hit it quite a lot. Let me get numbers | 14:35 |
| mwhahaha | jaosorior: hmm i guess ensure resource won't work like that | 14:35 |
| mwhahaha | jaosorior: i would recommend splitting out that management into a seperate class that you can use unique on the cert list to setup | 14:35 |
| sshnaidm | EmilienM, seems like we need to set preference for ipv4 over ipv6 | 14:37 |
| jaosorior | mwhahaha: is that the only way? That's a bit inconvenient since the path comes from a hash that defines the specs for the cert. This is where it's called https://github.com/openstack/puppet-tripleo/blob/master/manifests/profile/base/certmonger_user.pp#L63 | 14:37 |
| mwhahaha | jaosorior: we should just take the entire hash and do it on that | 14:37 |
| EmilienM | sshnaidm: that would be a short fix. Do you know how we can do? the right fix would be to use AFS mirrors | 14:37 |
| shardy | eglynn: Hmm, that's a puzzling one - I can't see anything in upgrade_tasks or puppet-nova which would cause that | 14:38 |
| sshnaidm | EmilienM, should be 'precedence ::ffff:0:0/96 100' in /etc/gai.conf | 14:38 |
| mwhahaha | jaosorior: like write something to setup the folders, alternatively is there not some system path that we can use that already has the proper selinux context? | 14:38 |
| shardy | eglynn: I'd be tempted to modify the image to add an auditd rule that logs any relevant changes to /var/log/nova, then compare the deploy vs upgrade cases | 14:38 |
| *** athomas has joined #tripleo | 14:39 | |
| eglynn | shardy: a-ha, cool ... good suggestion | 14:39 |
| pabelanger | sshnaidm: EmilienM: I would first switch to AFS mirror for ceph before preferring ipv4 over ipv6. | 14:39 |
| jaosorior | mwhahaha: so; currently I'm using a path that already has the right context. But for containers I need to create a directory per-service, and we'll bind mount that to the container. This is because we don't want to bind-mount the current directory, since that will contain the certs/keys for the rest of the services. | 14:39 |
| EmilienM | pabelanger: yes, I'm working on that today | 14:40 |
| pabelanger | also, don't prefer networks in jobs. If there are problems, ask -infra | 14:40 |
| mwhahaha | jaosorior: sounds like we need to be solving the container case elsewhere | 14:40 |
| pabelanger | EmilienM: sshnaidm: please include me on any networking issues, even fixes. I want to make sure tripleo has this working properly moving forward | 14:41 |
| EmilienM | pabelanger: yes | 14:41 |
| pabelanger | I am head deep into the gem mirror right now | 14:41 |
| jaosorior | mwhahaha: I don't see how that conclusion came up | 14:41 |
| pabelanger | only 300GB of gems | 14:41 |
| mwhahaha | jaosorior: which conclusion? that we need to solve the container case elsewhere? or how it's being done in containers? | 14:41 |
| jaosorior | mwhahaha: it's not being done | 14:42 |
| sshnaidm | EmilienM, pabelanger are afs mirrors available only from openstack infra? | 14:42 |
| jaosorior | that's what I'm trying to implement | 14:42 |
| pabelanger | sshnaidm: yes, each cloud region with have a local mirror | 14:42 |
| pabelanger | eg: http://mirror.regionone.infracloud-vanilla.openstack.org/ | 14:43 |
| sshnaidm | pabelanger, so if I reproduce the job locally it won't work for me? | 14:43 |
| mwhahaha | jaosorior: maybe for containers it needs to be copied in with the right contexts a opposed to having puppet try and create the folders with the right context | 14:43 |
| *** trozet has joined #tripleo | 14:43 | |
| pabelanger | sshnaidm: it should, ideally you'd use actually upstream mirrors for that | 14:44 |
| openstackgerrit | Andy Smith proposed openstack/tripleo-heat-templates master: Qpid dispatch router composable role https://review.openstack.org/423326 | 14:44 |
| jaosorior | mwhahaha: well, the issue is that the current folder where the certs/keys are contains files for all the services. By instead creating a directory per-service, I can then bind-mount that to the container, and not have the issue of those files being in all of them unnecessarily | 14:45 |
| *** jpena is now known as jpena|brb | 14:45 | |
| *** trown is now known as trown|brb | 14:45 | |
| *** dsariel has quit IRC | 14:45 | |
| mwhahaha | jaosorior: why not a refreshonly restorecon after all the files have been created? | 14:46 |
| mwhahaha | rather than trying to have puppet manage the contexts | 14:46 |
| adarazs | rlandy, trown|brb: do you mind if I move over the remaining bugs from tripleo-quickstart launchpad to tripleo? | 14:46 |
| sshnaidm | pabelanger, EmilienM, fyi, we configure repos in oooq jobs here: https://github.com/openstack/tripleo-quickstart/tree/59012fdb35dd221444516c8752e4df84ab91504a/config/release/tripleo-ci | 14:47 |
| sshnaidm | so we just install package centos-release-ceph-jewel from extras of centos: https://github.com/openstack/tripleo-quickstart/blob/59012fdb35dd221444516c8752e4df84ab91504a/config/release/tripleo-ci/master.yml#L56-L58 | 14:47 |
| EmilienM | sshnaidm: it would be great to add something in the roadmap (please create a ticket for that maybe): use tripleo-repos (that beekneemech wrote) in oooq | 14:48 |
| *** dsariel has joined #tripleo | 14:48 | |
| EmilienM | sshnaidm: yes, we'll need to stop installing this package | 14:48 |
| EmilienM | sshnaidm: and handle the repo manually (like we do in Puppet CI), so we can configure the baseurl ourselves | 14:48 |
| sshnaidm | EmilienM, we have pretty sophisticated repo-setup role for this | 14:48 |
| jaosorior | mwhahaha: Uhm... I guess that's an option | 14:48 |
| EmilienM | sshnaidm: ok | 14:49 |
| *** trown|brb is now known as trown | 14:49 | |
| pabelanger | sshnaidm: EmilienM: yes, I am also going to talk with dmsimard to see how we can mirror DLRN. I need to know which repos are important | 14:49 |
| trown | adarazs: by all means :), I moved all the ones we have triagedxc | 14:49 |
| sshnaidm | EmilienM, can you link to how you do it in puppet CI? | 14:49 |
| EmilienM | sshnaidm: on my list of urgent things to do, investigate the timouts in ovb jobs and also the promotion jobs | 14:49 |
| sshnaidm | EmilienM, yeah, me too.. | 14:50 |
| EmilienM | sshnaidm: https://github.com/openstack/puppet-openstack-integration/blob/master/manifests/repos.pp#L56 - https://github.com/openstack/puppet-openstack-integration/blob/master/run_tests.sh#L39-L44 | 14:50 |
| EmilienM | sshnaidm: if you want, I can work on the timeout things and you on the repo (if you know better than me the repo-setup module) | 14:50 |
| EmilienM | but I can also do both | 14:51 |
| sshnaidm | EmilienM, I'll handle the repo anyway | 14:51 |
| EmilienM | sshnaidm: ok, if you have any progress on this topic, please update the launchpad report, I'll work on this topic later in my day. I'm dealing with internal things this morning | 14:52 |
| sshnaidm | EmilienM, all right | 14:52 |
| *** udesale has joined #tripleo | 14:54 | |
| *** dsariel has quit IRC | 14:55 | |
| *** dsariel has joined #tripleo | 14:55 | |
| *** garyk has joined #tripleo | 14:56 | |
| *** yamahata has joined #tripleo | 14:56 | |
| ansiwen | EmilienM: any idea why ec2api.conf doesn't have the correct settings, although in hiera the values are correctly set? | 14:57 |
| garyk | EmilienM: quick question - i have director install on machine A, I want to run Overcloud in machine B. In ironic.conf I set libvirt_uri = qemu+ssh://root@192.168.126.254/system (is that correct). Anything else needs to be set? | 14:57 |
| mwhahaha | ansiwen: example? | 14:57 |
| *** Goneri has quit IRC | 14:57 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: README.rst: add documentation for the quickstart transition scripts https://review.openstack.org/450281 | 14:57 |
| ansiwen | mwhahaha: [root@controller-0 ~]# hiera ec2api::keystone::authtoken::auth_uri | 14:58 |
| ansiwen | http://172.17.1.13:5000/v2.0 | 14:58 |
| ansiwen | mwhahaha: but in /etc/ec2api/ec2api.conf the config is commented out | 14:58 |
| EmilienM | garyk: I'm not sure why you need to modify this file manually? what kind of setup do you run? | 14:59 |
| mwhahaha | ansiwen: so meaning it's not getting set, is ec2api::keystone::authtoken getting properly included? | 14:59 |
| ansiwen | mwhahaha: which is why nothing works, btw. ;-) | 14:59 |
| ansiwen | mwhahaha: let me check | 14:59 |
| garyk | i have director installed in a VM. In another VM i would like to run the Overcloud. Where the Overcloud is running virtualized by Libvirt. If you want I can draw a diagram | 15:00 |
| mwhahaha | ansiwen: it's not automatically included in ec2api::api (unlike every other m odule) | 15:00 |
| mwhahaha | ansiwen: so it's probably missing from the puppet-tripleo class | 15:00 |
| ansiwen | mwhahaha: so, but how can CI pass then? | 15:01 |
| mwhahaha | ansiwen: we don't actually test it? :D | 15:01 |
| mwhahaha | happy monday! | 15:01 |
| ansiwen | mwhahaha: that's pretty embarrising, that means maybe it never worked? | 15:01 |
| mwhahaha | ansiwen: i'm not aware of any tripleo test for it | 15:01 |
| mwhahaha | ansiwen: in upstream we probably include it in p-o-i | 15:01 |
| * mwhahaha goes looking | 15:01 | |
| ansiwen | mwhahaha: but I spent weeks to include the tempest runs, so that means that is worth nothing for the actual tripleo-ci? | 15:02 |
| *** flepied has quit IRC | 15:02 | |
| mwhahaha | ansiwen: https://github.com/openstack/puppet-openstack-integration/blob/master/manifests/ec2api.pp#L28-L32 | 15:02 |
| mwhahaha | ansiwen: we test it in p-o-i but we also set it up correctly | 15:02 |
| *** morazi_ has joined #tripleo | 15:02 | |
| ansiwen | mwhahaha: yes, I wrote that... | 15:02 |
| mwhahaha | ansiwen: we don't have something like https://github.com/openstack/puppet-nova/blob/master/manifests/api.pp#L300 in ec2api | 15:03 |
| *** agurenko has quit IRC | 15:03 | |
| ansiwen | mwhahaha: but I also wrote the other file... I expected that "scenario-002" test, where I included it, does some smoke test at least? | 15:03 |
| *** prateek has quit IRC | 15:03 | |
| mwhahaha | ansiwen: triplo scenario002? | 15:03 |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/newton: Set manila default_share_type config option https://review.openstack.org/423405 | 15:03 |
| *** ealcaniz has quit IRC | 15:03 | |
| ansiwen | mwhahaha: yes | 15:05 |
| mwhahaha | ansiwen: no idea | 15:05 |
| *** jpena|brb is now known as jpena | 15:05 | |
| mwhahaha | ansiwen: but we need ec2api::keystone::authtoken needs to be added in https://github.com/openstack/puppet-tripleo/blob/master/manifests/profile/base/nova/ec2api.pp#L29-L33 | 15:05 |
| *** morazi has quit IRC | 15:05 | |
| ansiwen | mwhahaha: yes, but I would also like to include a test, how do other modules test this? so scenario-002 only tests that the deployment doesn't fail, but not that the service is actually running= | 15:06 |
| ansiwen | ? | 15:06 |
| ansiwen | mwhahaha: btw: thanks a lot! | 15:06 |
| mwhahaha | ansiwen: not sure i'll have to look and get back to you on that (got a meeting at the moment) | 15:06 |
| mwhahaha | i'm not that familar with the tripleo scenario jobs so i don't know how the tempest configs happen | 15:07 |
| *** rcernin has quit IRC | 15:07 | |
| ansiwen | mwhahaha: there is only something like a "ping test" whatever that means | 15:07 |
| mwhahaha | ansiwen: yea that's possible | 15:07 |
| ansiwen | shardy, EmilienM: any comment on that? how does tripleo-ci scenarios actually tests that the service is correctly configured? | 15:08 |
| EmilienM | ansiwen: we use the pingtest, but for ec2api we don't have any test I thin | 15:09 |
| *** ooolpbot has joined #tripleo | 15:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674770 | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1676250 | 15:10 |
| *** ooolpbot has quit IRC | 15:10 | |
| openstack | Launchpad bug 1674770 in tripleo "Update timeout too long in CI" [Critical,Triaged] | 15:10 |
| openstack | Launchpad bug 1676250 in tripleo "Promotion to OpenStack trunk fails (timeout at step3)" [Critical,Triaged] | 15:10 |
| *** milan has quit IRC | 15:10 | |
| shardy | ansiwen: yeah there are some gaps, it depends on what support exists in heat, as we create a heat stack to exercise the services that are deployed | 15:10 |
| *** cylopez has left #tripleo | 15:12 | |
| ansiwen | shardy: so it seems, although puppet-ec2api is tempest tested in puppet-ci, it was not tested in the tripleo-ci, although I included it in the scenario-002. so did I do something wrong, or is that just not tested? | 15:12 |
| *** Goneri has joined #tripleo | 15:13 | |
| shardy | ansiwen: well including it just proves it deploys, you have to add something to the pingtest template to actually exercise the deployed service | 15:13 |
| shardy | ansiwen: so in this particular case, it's just not tested | 15:13 |
| shardy | https://github.com/openstack/tripleo-heat-templates/blob/master/ci/pingtests/scenario002-multinode.yaml | 15:13 |
| shardy | in most other cases, we're able to do basic functional testing by adding something to the pingtest template (which creates various resources, depending on the scenario) | 15:14 |
| *** limao_ has quit IRC | 15:17 | |
| *** limao has joined #tripleo | 15:18 | |
| openstackgerrit | Alex Schultz proposed openstack/instack-undercloud stable/ocata: Disable VIP validation when UI is enabled https://review.openstack.org/450285 | 15:18 |
| *** flepied has joined #tripleo | 15:19 | |
| *** limao has quit IRC | 15:22 | |
| *** dparkes has quit IRC | 15:22 | |
| *** chlong has joined #tripleo | 15:25 | |
| *** rbrady is now known as rbrady-afk | 15:27 | |
| ansiwen | mwhahaha: I also don't include ::ec2api::db in puppet-tripleo, but this is correctly set... | 15:28 |
| mwhahaha | ansiwen: i think that gets pulled in via ::ec2api | 15:28 |
| openstackgerrit | Oliver Walsh proposed openstack/tripleo-heat-templates master: WIP: SSH known_hosts config https://review.openstack.org/449660 | 15:29 |
| mwhahaha | ansiwen: https://github.com/openstack/puppet-ec2api/blob/master/manifests/init.pp#L39 | 15:30 |
| ansiwen | mwhahaha: I see... that's confusing... | 15:30 |
| mwhahaha | ansiwen: we should update ec2api to include ec2api::keystone::authtoken if it's required for it to work | 15:31 |
| *** zoli|afk is now known as zoli|wfh | 15:32 | |
| ansiwen | what about ::ec2api::db::mysql and ::ec2api::keystone::auth? | 15:32 |
| ansiwen | mwhahaha: | 15:32 |
| pradk | can i get some reviews on https://review.openstack.org/#/c/447599/ ... this is a test blocker | 15:32 |
| mwhahaha | ansiwen: no those are specific to the user so those shouldn'tbe included | 15:33 |
| mwhahaha | ansiwen: we should just include ::ec2api::keystone::authtoken in ::ec2api::api but the fix for now would be add it to the profile | 15:33 |
| openstackgerrit | Sven Anderson proposed openstack/puppet-tripleo master: Add missing include of ::ec2api::keystone::authtoken https://review.openstack.org/450294 | 15:36 |
| ansiwen | mwhahaha: ^^^ | 15:36 |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-quickstart master: Add featureset011 - upgrade from BM to containerized https://review.openstack.org/450295 | 15:37 |
| mwhahaha | ansiwen: looks good | 15:37 |
| *** leanderthal is now known as leanderthal|afk | 15:38 | |
| *** dsariel has quit IRC | 15:39 | |
| openstackgerrit | Sven Anderson proposed openstack/puppet-tripleo stable/ocata: Add missing include of ::ec2api::keystone::authtoken https://review.openstack.org/450299 | 15:40 |
| ansiwen | mwhahaha: ^^^ | 15:40 |
| openstackgerrit | Oliver Walsh proposed openstack/tripleo-heat-templates master: WIP: SSH known_hosts config https://review.openstack.org/449660 | 15:40 |
| openstackgerrit | Pradeep Kilambi proposed openstack/puppet-tripleo stable/ocata: Re-run gnocchi and ceilometer upgrade in step 5 https://review.openstack.org/447735 | 15:45 |
| weshay | dprince, fyi https://review.openstack.org/#/c/450281/ | 15:45 |
| dprince | weshay: cool. Thanks | 15:46 |
| ansiwen | mwhahaha: do you have an idea, how I could quickly test that in a deployed environment? like patching puppet-tripleo and do a puppet apply? | 15:49 |
| ansiwen | mwhahaha: never done that before, therefor my dumb question | 15:50 |
| mwhahaha | ansiwen: puppet apply -e 'include ::ec2api::keystone::authtoken' | 15:50 |
| mwhahaha | ansiwen: should update the config if you want to see if that 'fixes it' | 15:50 |
| *** udesale has quit IRC | 15:50 | |
| mwhahaha | ansiwen: you'd have to restart the service manually tho | 15:51 |
| mwhahaha | ansiwen: or you could do puppet apply -e "class { '::tripleo::profile::base::nova::ec2api': step=> 4 }" after patching puppet-tripleo | 15:52 |
| rdopiera | maybe someone can help me here: I'm trying to get Horizon to run with tripleo docker stuff. I have it mostly working, the only problem is that apache insists on running on port 80, which conflicts with the apache running on the host system and possibly in the keystone docker -- I tried to make it run on a different port by adding apache::port: 8080 | 15:52 |
| rdopiera | in the role data outputs, but it seems that puppet ignores that | 15:52 |
| openstackgerrit | Martin André proposed openstack/tripleo-heat-templates master: Remove kolla_config copy from keystone service. https://review.openstack.org/447676 | 15:53 |
| rdopiera | which is quaint, because it doesn't ignore other apache::* settings | 15:53 |
| rdopiera | anybody has any idea what could be going on? | 15:53 |
| *** dsariel has joined #tripleo | 15:54 | |
| ansiwen | mwhahaha: dumb question, I do that on the controller-node, not on the undercloud, right? | 15:54 |
| jaosorior | mandre: you forgot to rebase from master | 15:54 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci master: Force deploy commands to timeout https://review.openstack.org/450302 | 15:54 |
| *** iranzo has quit IRC | 15:54 | |
| openstackgerrit | Radomir Dopieralski proposed openstack/tripleo-heat-templates master: WIP: Containerize Horizon https://review.openstack.org/450303 | 15:55 |
| mandre | jaosorior: I removed it on purpose, see we're bind mounting the entire /etc/keystone directory | 15:55 |
| jaosorior | uhm.. | 15:56 |
| jaosorior | mandre: ok, re-scored | 15:57 |
| mandre | jaosorior: you wanted a more granular approach? | 15:57 |
| *** paramite has quit IRC | 15:57 | |
| *** dsariel has quit IRC | 15:58 | |
| *** dmarlin has joined #tripleo | 15:58 | |
| jaosorior | mandre: naaah, that's fine for me | 15:59 |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/ocata: Install openstack-selinux for deployed-server https://review.openstack.org/450025 | 15:59 |
| openstackgerrit | Attila Darazs proposed openstack/tripleo-quickstart-extras master: GATE CHECK https://review.openstack.org/430277 | 16:00 |
| *** lmiccini has quit IRC | 16:00 | |
| *** milan has joined #tripleo | 16:03 | |
| EmilienM | sshnaidm: Moving to post; patches have been merged in master and backported in stable/ocata upstream. Now waiting for a new downstream build that would include the fix. | 16:03 |
| EmilienM | oops, wrong copy paste | 16:03 |
| EmilienM | sshnaidm: http://logs.openstack.org/87/448287/4/check-tripleo/gate-tripleo-ci-centos-7-ovb-updates/b917870/console.html#_2017-03-27_14_14_23_544181 | 16:03 |
| EmilienM | look this one, it failed after a stack created successfuly | 16:04 |
| *** rsquared has left #tripleo | 16:04 | |
| sshnaidm | EmilienM, hmm.. may it be tripleoclient that reports error? | 16:04 |
| openstackgerrit | Saravanan KR proposed openstack/tripleo-common master: Add an action to fetch and flatten the heat resource tree and parameters https://review.openstack.org/450021 | 16:04 |
| jaosorior | EmilienM: there's a strange 'headers' line after the overcloud stack creation success | 16:04 |
| jaosorior | sshnaidm, EmilienM might be that it's hiding some error. | 16:05 |
| sshnaidm | yeah, seems like truncated output | 16:05 |
| EmilienM | jaosorior: yeah | 16:05 |
| *** udesale has joined #tripleo | 16:06 | |
| EmilienM | jaosorior: sounds like related to the WebsocketClient part of oooclient | 16:06 |
| sshnaidm | EmilienM, jaosorior in my local reproducing I had about 10 minutes between "CREATE_COMPLETE Stack CREATE completed successfully" and actual end of task | 16:06 |
| EmilienM | d0ugal: ^ any idea? | 16:07 |
| * d0ugal reads | 16:07 | |
| d0ugal | EmilienM: that is probably a KeyError in tripleoclient, but I am not sure where/why | 16:09 |
| *** jpich has quit IRC | 16:09 | |
| *** ooolpbot has joined #tripleo | 16:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674770 | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1676250 | 16:10 |
| *** ooolpbot has quit IRC | 16:10 | |
| openstack | Launchpad bug 1674770 in tripleo "Update timeout too long in CI" [Critical,Triaged] | 16:10 |
| openstack | Launchpad bug 1676250 in tripleo "Promotion to OpenStack trunk fails (timeout at step3)" [Critical,Triaged] | 16:10 |
| sshnaidm | d0ugal, can we see the full output anywhere? | 16:10 |
| d0ugal | sshnaidm: only if the command is run with --debug | 16:10 |
| EmilienM | can someone review the basic structure of tripleo deployment guide? it has +1 from Doc PTL. Thanks! | 16:11 |
| d0ugal | Without that python-openstackclient prints the string representation of errors, which for a keyerror is just the key name :( | 16:11 |
| *** dsariel has joined #tripleo | 16:11 | |
| d0ugal | sshnaidm: but grepping tripleoclient I don't see any code that looks for a headers key - so maybe it is a malformed message? | 16:12 |
| jaosorior | EmilienM: pass the link | 16:12 |
| *** cinerama has joined #tripleo | 16:12 | |
| jaosorior | d0ugal: what about tripleoclient/utils.py? | 16:12 |
| sshnaidm | d0ugal, yeah, most likely | 16:12 |
| EmilienM | jaosorior: lol yeah sorry | 16:12 |
| d0ugal | jaosorior: what about it? :) | 16:12 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-quickstart-extras master: Use openstack deploy to install the undercloud https://review.openstack.org/419040 | 16:12 |
| EmilienM | jaosorior: https://review.openstack.org/#/c/449684/ | 16:12 |
| ansiwen | mwhahaha: I'm still worried about the ::ec2api::api::keystone_ec2_tokens_url value... it's set here https://github.com/openstack/puppet-openstack-integration/blob/master/manifests/ec2api.pp#L34 | 16:13 |
| mwhahaha | ansiwen: then that needs to be handled either in puppet-tripleo or THT | 16:13 |
| ansiwen | mwhahaha: but not in the THT configs | 16:13 |
| jaosorior | d0ugal: I mean plugin.py https://github.com/openstack/python-tripleoclient/blob/master/tripleoclient/plugin.py#L107-L124 | 16:13 |
| jaosorior | not utils | 16:13 |
| mwhahaha | ansiwen: so since that's not being set in THT, you should handle it in the ec2api profile in puppet-tripleo | 16:14 |
| d0ugal | I think we should add a new arg to tripleoclient commands that if given prints the exception if there is any - we can then use this in CI. --print-tracebacks - sound like a good idea? | 16:14 |
| jaosorior | d0ugal: I dig | 16:14 |
| d0ugal | --debug outputs tons of information, so I don't find it that useful - usually I just want the traceback. | 16:14 |
| jaosorior | d0ugal: though I think it should be by default. | 16:14 |
| *** nyechiel has quit IRC | 16:14 | |
| jaosorior | d0ugal: it would help folks debug or send reports | 16:14 |
| sshnaidm | d0ugal, so the full command will be "openstack overcloud deploy --debug ..."? | 16:14 |
| d0ugal | sshnaidm: yup, the output will be huge :) | 16:15 |
| hjensas | EmilienM: What is policy around deprecation of options in undercloud.conf? instack-undercloud? (Let me know if you are the wrong guy to ask...) It regarding https://review.openstack.org/#/c/437544/ . Do I have to keep options available for one cycle like with puppet-modules? | 16:15 |
| sshnaidm | d0ugal, it's infra's problems :) | 16:15 |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder master: WIP: Add lvm management to diskimage-builder https://review.openstack.org/444403 | 16:15 |
| d0ugal | jaosorior: yeah, I guess --debug works for most projects, but we do far more in each command. | 16:15 |
| jaosorior | d0ugal: right. But I mean, the --print-tracebacks should be default. | 16:16 |
| ansiwen | mwhahaha: well, maybe it should be set in THT? | 16:16 |
| mwhahaha | ansiwen: ideally, yes | 16:16 |
| *** dsariel has quit IRC | 16:16 | |
| EmilienM | hjensas: yes we need to deprecate it, send a warning in logs if used, handle the transition for the user and write a release note. | 16:16 |
| d0ugal | jaosorior: yeah, that is probably also true. It isn't great for users, but better than "headers" :) | 16:16 |
| jaosorior | haha yep | 16:16 |
| d0ugal | jaosorior: Not sure how I missed that bit in plugin.py - that'll probably be it, so that means it is a bad message. Never seen that before | 16:17 |
| akrivoka | shardy: EmilienM: this patch had +2 from both of you before I had to rebase it to handle trivial merge conflict, would you mind re-adding your reviews so we can merged it? https://review.openstack.org/#/c/422789/ | 16:17 |
| jaosorior | d0ugal: you really don't want to get a bug report with the only thing in the description is 'headers' | 16:17 |
| d0ugal | A bad message, but still valid JSON. | 16:17 |
| shardy | akrivoka: ack looking | 16:17 |
| EmilienM | akrivoka: why the container job is failing? | 16:17 |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-quickstart-extras master: DONT REVIEW: run deploy in debug mode https://review.openstack.org/450327 | 16:17 |
| EmilienM | mandre: I thought the container job was now fixed, isn't? | 16:17 |
| d0ugal | jaosorior: yeah, so I'll turn this into a "real" patch then. https://review.openstack.org/#/c/449583/ | 16:17 |
| jaosorior | d0ugal: that would be cool | 16:18 |
| hjensas | EmilienM: ok, got it. No shortcuts. :) | 16:18 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: DONT REVIEW: test consistent repo with debug https://review.openstack.org/447514 | 16:22 |
| sshnaidm | EmilienM, maybe will help ^^ | 16:22 |
| EmilienM | sshnaidm: definitly, thanks | 16:24 |
| EmilienM | sshnaidm: I started to work on the ceph mirror fyi. Is it ok? | 16:24 |
| sshnaidm | EmilienM, I'm working on implementing it in repo-setup for oooq jobs | 16:24 |
| sshnaidm | EmilienM, do you work on it in tripleo.sh? | 16:25 |
| EmilienM | sshnaidm: ok cool. All I did until now is to fix Puppet CI to use this mirror, so nothing related to TripleO yet | 16:25 |
| sshnaidm | EmilienM, ok | 16:25 |
| *** abehl has quit IRC | 16:27 | |
| *** pkovar1 has joined #tripleo | 16:27 | |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-quickstart-extras master: Use openstack deploy to install the undercloud https://review.openstack.org/419040 | 16:28 |
| mandre | EmilienM: a couple of patches that reduce the total time of the container job landed today | 16:28 |
| shardy | d0ugal: Hey are you OK to approve https://review.openstack.org/#/c/446045 and https://review.openstack.org/#/c/449633/ now they passed CI? | 16:28 |
| akrivoka | EmilienM: not sure why it failed now, it passed on a previous recheck I did earlier today | 16:28 |
| EmilienM | akrivoka: we can probably merge it then | 16:29 |
| akrivoka | EmilienM: yes please :) | 16:29 |
| EmilienM | done | 16:29 |
| akrivoka | thank you :) | 16:29 |
| bogdando | folks, once I reworked custom t-h-t for undercloud deploy https://review.openstack.org/#/c/419040/29..30/roles/undercloud-deploy/tasks/create-scripts.yml could you please help me to define the custom *minimal* deployment environment ? Like only keystone and db/mq? | 16:29 |
| bogdando | I'm quite new to those -e env foo things... | 16:30 |
| EmilienM | akrivoka: sorry, we have a bunch of CI issues lately | 16:30 |
| bogdando | mandre: ^ ^ here by a chance? | 16:30 |
| mandre | EmilienM akrivoka: this is a new thing, though... | 16:30 |
| mandre | http://logs.openstack.org/89/422789/12/check-tripleo/gate-tripleo-ci-centos-7-ovb-containers-oooq-nv/9590a3c/logs/oooq/undercloud/home/jenkins/overcloud_deploy.log.txt.gz#_2017-03-27_15_26_20 | 16:30 |
| *** dsariel has joined #tripleo | 16:31 | |
| mandre | akrivoka: not related to your patch | 16:31 |
| EmilienM | mandre: ok, so we can land it? | 16:31 |
| *** pkovar has quit IRC | 16:31 | |
| mandre | EmilienM: yes | 16:31 |
| shardy | bogdando: hey check https://github.com/openstack/tripleo-heat-templates/blob/master/roles_data_undercloud.yaml and http://hardysteven.blogspot.co.uk/2016/08/tripleo-composable-services-101.html | 16:32 |
| shardy | bogdando: I think you can just redefine the list of services in that special roles_data file? | 16:32 |
| EmilienM | pabelanger: a baby step https://review.openstack.org/450331 | 16:33 |
| shardy | http://hardysteven.blogspot.co.uk/2016/10/tripleo-composablecustom-roles.html may also be of interest to see how the roles_data is consumed | 16:33 |
| shardy | bogdando: ^^ | 16:33 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/tripleo-heat-templates master: Updated from global requirements https://review.openstack.org/440124 | 16:34 |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart master: Download overcloud_release rpm for mixed upgrade https://review.openstack.org/449349 | 16:37 |
| bogdando | shardy: thank you, does this cover undercloud deploy as well? | 16:37 |
| *** rcernin has joined #tripleo | 16:37 | |
| shardy | bogdando: for the heat-driven undercloud installer yes (which I think is what you're dealing with?) | 16:37 |
| pabelanger | EmilienM: nice | 16:37 |
| bogdando | shardy: yupp | 16:37 |
| shardy | bogdando: cool, in that case the overcloud and undercloud process in t-h-t is essentially the same I think | 16:38 |
| *** dsariel has quit IRC | 16:38 | |
| *** Goneri has quit IRC | 16:38 | |
| *** aufi has quit IRC | 16:39 | |
| mandre | bogdando: what shardy said. For the undercloud you'll need to enable some services, like ironic, mistral and zaqar | 16:39 |
| mandre | bogdando: take this as an example https://github.com/dprince/undercloud_containers/blob/master/doit.sh#L137-L145 | 16:39 |
| bogdando | mandre: that's clear. I don't deploy overcloud, so I don't need them perhaps | 16:39 |
| EmilienM | pabelanger: sad your mirrors aren't HTTPS | 16:40 |
| pabelanger | EmilienM: we just need a cert, wonder if we have a wildcard | 16:41 |
| *** jaosorior has quit IRC | 16:41 | |
| pabelanger | I can look into that | 16:41 |
| EmilienM | pabelanger: nothing urgent, just if we can it to our list of things :D | 16:42 |
| dprince | mandre: this one would help me push a new round of containers for tripleoupstream https://review.openstack.org/#/c/450274/ | 16:42 |
| pabelanger | EmilienM: ya, lets get using http first, then we can add a cert :) | 16:42 |
| *** rwsu has joined #tripleo | 16:48 | |
| *** derekh has quit IRC | 16:49 | |
| *** flepied has quit IRC | 16:52 | |
| *** yprokule has quit IRC | 16:52 | |
| mandre | dprince: left a comment | 16:53 |
| *** udesale has quit IRC | 16:54 | |
| mandre | bogdando: indeed, you can try providing an environment file with just keystone and mariadb services | 16:55 |
| bogdando | mandre: what about -e /usr/share/openstack-tripleo-heat-templates/environments/docker.yaml? | 16:56 |
| dprince | mandre: commented back. As far as I can tell that moved it into the 'source' section | 16:57 |
| bogdando | mandre: it seems like a base thing I should keep in place? | 16:58 |
| *** salmankhan has quit IRC | 16:58 | |
| *** tesseract has quit IRC | 16:59 | |
| *** dsariel has joined #tripleo | 17:00 | |
| *** maeca1 has quit IRC | 17:01 | |
| EmilienM | sshnaidm: do we have a bug report for this kind of timeout? http://logs.openstack.org/51/431951/6/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/08b6153/console.html#_2017-03-27_14_19_02_647908 | 17:03 |
| dprince | mandre: ack on the bad commit message. Updated. It has been broken since the 19th | 17:03 |
| EmilienM | sshnaidm: and there is no collected log, so impossible to debug this one :/ | 17:03 |
| dprince | mandre: nobody upstream is using binary Ironic Kolla containers I guess?! | 17:03 |
| *** saibarspeis has joined #tripleo | 17:04 | |
| sshnaidm | EmilienM, I have an old patch for saving logs in timeouted jobs: https://review.openstack.org/#/c/410470 | 17:04 |
| sshnaidm | EmilienM, but it's solved already for oooq jobs, so we have logs there | 17:05 |
| EmilienM | ok | 17:05 |
| *** trown is now known as trown|compassing | 17:05 | |
| EmilienM | slagle, pabelanger: we have some cases where subnode fails to be deployed (or is deployed but unreachable) http://logs.openstack.org/63/450263/1/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/fe6f204/logs/undercloud/home/jenkins/overcloud_deploy.log.txt.gz#_2017-03-27_16_14_37 | 17:06 |
| *** skramaja has joined #tripleo | 17:07 | |
| slagle | EmilienM: i think it's more of the case of why is the validation trying to ping 15.184.64.1 | 17:08 |
| openstackgerrit | Diana Clarke proposed openstack/tripleo-docs master: typo: centos7-oacata -> centos7-ocata https://review.openstack.org/450342 | 17:08 |
| *** jlinkes has quit IRC | 17:08 | |
| pabelanger | EmilienM: slagle: yes, that is the gateway it looks like | 17:08 |
| *** gfidente is now known as gfidente|afk | 17:08 | |
| *** rcernin has quit IRC | 17:08 | |
| slagle | EmilienM: that doesnt seem like an IP we'd configure from a range we normally use in CI (192.168.xx) | 17:08 |
| pabelanger | which might not respond to ICMP | 17:08 |
| rdopiera | flaper87: I fugured out why horizon won't start in that docker container -- apache wants to run on port 80, which is already taken, and otherwise not available for non-root users anyways | 17:09 |
| *** ooolpbot has joined #tripleo | 17:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674770 | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1676250 | 17:10 |
| *** ooolpbot has quit IRC | 17:10 | |
| openstack | Launchpad bug 1674770 in tripleo "Update timeout too long in CI" [Critical,Triaged] | 17:10 |
| openstack | Launchpad bug 1676250 in tripleo "Promotion to OpenStack trunk fails (timeout at step3)" [Critical,Triaged] | 17:10 |
| *** pcaruana has quit IRC | 17:12 | |
| EmilienM | sshnaidm: ok, +2 but 'recheck' to make sure this time it pass better | 17:13 |
| EmilienM | slagle: could we file a bug on this one maybe? | 17:13 |
| *** hewbrocca is now known as hewbrocca_afk | 17:16 | |
| *** jpena is now known as jpena|off | 17:16 | |
| sshnaidm | EmilienM, woops, seems like my patch didn't help much :( will work on it further | 17:17 |
| openstackgerrit | Saravanan KR proposed openstack/tripleo-common master: Add an action to fetch and flatten the heat resource tree and parameters https://review.openstack.org/450021 | 17:17 |
| skramaja | jtomasek: shardy ^ | 17:17 |
| skramaja | jtomasek: shardy i tried to match what is being done in UI to flatten the heat resource tree. ptal. | 17:18 |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder master: WIP: Add lvm management to diskimage-builder https://review.openstack.org/444403 | 17:20 |
| slagle | EmilienM: it is the default gw for the network interface configured by the host cloud | 17:20 |
| slagle | EmilienM: when did this error start? or is this just a one off? | 17:20 |
| *** thrash is now known as thrash|biab | 17:20 | |
| slagle | EmilienM: we do actually ping all default gw's in the validation, so it's expected. but i've never seen this error before, so i dont know if something changed with how the networking is configured | 17:21 |
| *** chlong has quit IRC | 17:22 | |
| *** saibarspeis has quit IRC | 17:22 | |
| weshay | bkero, have you seen this before? http://logs.openstack.org/81/450281/1/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/d309d56/logs/undercloud/home/jenkins/overcloud_validate.log.txt.gz | 17:23 |
| slagle | EmilienM: i'm actually quite confused why this job has 2 subnodes | 17:24 |
| bkero | weshay: that usually means the overcloud failed to deploy | 17:24 |
| EmilienM | slagle: I think it's one off | 17:24 |
| weshay | bkero, that's what I thought | 17:24 |
| slagle | EmilienM: http://logs.openstack.org/63/450263/1/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/fe6f204/logs/ | 17:24 |
| slagle | subnode-1 and subnode-2 | 17:24 |
| EmilienM | subnode-2 is actually subnode-1 | 17:24 |
| weshay | http://logs.openstack.org/81/450281/1/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/d309d56/logs/undercloud/home/jenkins/overcloud_deploy.log.txt.gz | 17:25 |
| EmilienM | iirc it's a quickstart thing | 17:25 |
| EmilienM | weshay: can you confirm ^? | 17:25 |
| weshay | ugh.. first I'm hearing of that one | 17:25 |
| slagle | EmilienM: are you saying the 2nd subnode is actually used as the first? | 17:25 |
| * weshay looks | 17:25 | |
| slagle | then what is the first doing? | 17:25 |
| *** nenad has quit IRC | 17:26 | |
| EmilienM | I don't recall the details but iirc log collection takes logs from the second node and put it in subnode1 | 17:26 |
| slagle | is it just cattle grazing in the field | 17:26 |
| bkero | slagle: IIRC some legacy tripleo-ci scripts index the primary node as subnode-1 and the subnode as subnode-2 | 17:26 |
| EmilienM | adarazs, sshnaidm ^ please help here :) | 17:26 |
| openstackgerrit | Saravanan KR proposed openstack/tripleo-heat-templates master: WIP: Add role specific information to the service template https://review.openstack.org/437956 | 17:26 |
| slagle | bkero: what scripts? i've never seen this before | 17:26 |
| bkero | Whereas quickstart lists the subnodes starting at subnode-1 | 17:26 |
| slagle | quickstart shouldn't be assigning subnode indexes at all. it's all setup under /etc/nodepool already | 17:27 |
| *** amoralej is now known as amoralej|off | 17:27 | |
| sshnaidm | slagle, subnode-2 is created by infra scripts iirc | 17:28 |
| bkero | slagle: I'm guessing scripts/common_functions.sh | 17:28 |
| bkero | line 247 | 17:28 |
| sshnaidm | slagle, subnode-1 is created by quickstart logs role | 17:28 |
| EmilienM | why can't we use subnode-2 to avoid confusion? | 17:29 |
| slagle | sshnaidm: and is that the correct node index as indicated by nodepool? or does quickstart just pick "1" | 17:29 |
| bkero | slagle: Some things are set up under /etc/nodepool. There aren't indexed identifiers for each subnode though. | 17:29 |
| sshnaidm | slagle, quickstart just starts from 1 | 17:29 |
| EmilienM | sshnaidm: what if we start from 2 (to avoid confusion with what we had before); any opinion? | 17:30 |
| sshnaidm | slagle, bkero I'm working on removing this "subnode-2" from oooq jobs, just didn't realize how to do it yet | 17:30 |
| *** garyk has quit IRC | 17:31 | |
| bkero | sshnaidm: does tripleo-ci/scripts/common_function.sh:247 help? | 17:31 |
| sshnaidm | EmilienM, then it will overwrite our directory I think, it's not good | 17:31 |
| openstackgerrit | Saravanan KR proposed openstack/tripleo-heat-templates master: WIP: Add role specific information to the service template https://review.openstack.org/437956 | 17:31 |
| bkero | slagle: Here are the files available to identify from nodepool: https://docs.openstack.org/infra/nodepool/scripts.html | 17:31 |
| slagle | sshnaidm: bkero : the reason we have subnode-2 is b/c it matches what is done across every other openstack infra multinode job | 17:31 |
| *** pkovar1 has quit IRC | 17:31 | |
| *** arxcruz has quit IRC | 17:31 | |
| openstackgerrit | Merged openstack/instack-undercloud master: Explicitly configure credentials used by ironic to access inspector and service catalog https://review.openstack.org/446981 | 17:31 |
| sshnaidm | bkero, it's postci function, I don't think we still use it in quickstart jobs | 17:31 |
| slagle | if you look at a devstack multinode job for example | 17:31 |
| sshnaidm | slagle, change from 1 to 2 will be super-easy, but it will override our logs, it's what I don't want | 17:32 |
| sshnaidm | slagle, we need to find a way to get rid of this subnode-2 before | 17:33 |
| jtomasek | skramaja: thanks, I'll try to review it tomorrow | 17:33 |
| EmilienM | sshnaidm: i'm fine to keep using subnode 1 as long as 1) we remove subnode2 asap to avoid confusion 2) we inform our users that subnode 1 is now the overcloud log collection | 17:34 |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-quickstart master: WIP: count subnodes from 2 https://review.openstack.org/450351 | 17:36 |
| *** Goneri has joined #tripleo | 17:36 | |
| *** nyechiel has joined #tripleo | 17:36 | |
| sshnaidm | actually let's check, maybe I'm wrong | 17:36 |
| sshnaidm | slagle, EmilienM bkero ^^ | 17:36 |
| skramaja | jtomasek: thanks. | 17:36 |
| *** chlong has joined #tripleo | 17:37 | |
| *** milan has quit IRC | 17:38 | |
| *** milan has joined #tripleo | 17:39 | |
| EmilienM | sshnaidm: cool, let's see how it works. | 17:40 |
| * EmilienM afk lunch | 17:40 | |
| *** sshnaidm is now known as sshnaidm|afk | 17:40 | |
| *** garyk has joined #tripleo | 17:41 | |
| *** flepied has joined #tripleo | 17:41 | |
| *** ckyriakidou has quit IRC | 17:42 | |
| bkero | sshnaidm|afk, slagle: I'm hearing that it's actually the IP of the subnode on the vxlan bridge, and doesn't actually have to do with the hostname or role at all. | 17:42 |
| *** shardy has quit IRC | 17:44 | |
| bkero | If that's really the case though, it should be subnode-3 since the primary node is 192.168.24.2 | 17:45 |
| *** lucasagomes is now known as lucas-afk | 17:45 | |
| *** tosky has quit IRC | 17:45 | |
| *** ccamacho|brb is now known as ccamacho | 17:46 | |
| *** nyechiel has quit IRC | 17:46 | |
| *** thrash|biab is now known as thrash | 17:46 | |
| *** garyk has quit IRC | 17:46 | |
| *** dsariel has quit IRC | 17:48 | |
| *** zoli|wfh is now known as zoli|gone | 17:48 | |
| *** fragatina has quit IRC | 17:49 | |
| *** panda is now known as panda|bbl | 17:51 | |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-quickstart-extras master: Use openstack deploy to install the undercloud https://review.openstack.org/419040 | 17:57 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates master: Remove 'Controller' role references from overcloud.j2.yaml https://review.openstack.org/450380 | 17:58 |
| *** garyk has joined #tripleo | 17:59 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Stop deployment before jobs end https://review.openstack.org/450381 | 18:00 |
| bkero | sshnaidm|afk: Speaking with infra folks, I think zuul's ansible inventory naming is what is using 'subnode-2' and collecting | 18:04 |
| *** sshnaidm|afk is now known as sshnaidm | 18:04 | |
| sshnaidm | bkero, yeah, the question is how to avoid it.. | 18:05 |
| *** milan has quit IRC | 18:08 | |
| *** gbarros has quit IRC | 18:09 | |
| *** ooolpbot has joined #tripleo | 18:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674770 | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1676250 | 18:10 |
| *** ooolpbot has quit IRC | 18:10 | |
| openstack | Launchpad bug 1674770 in tripleo "Update timeout too long in CI" [Critical,Triaged] | 18:10 |
| openstack | Launchpad bug 1676250 in tripleo "Promotion to OpenStack trunk fails (timeout at step3)" [Critical,Triaged] | 18:10 |
| *** milan has joined #tripleo | 18:10 | |
| bkero | sshnaidm: I don't know if that will be possible | 18:10 |
| *** rbrady-afk is now known as rbrady | 18:10 | |
| openstackgerrit | Honza Pokorny proposed openstack/tripleo-specs master: Add network configuration spec https://review.openstack.org/396383 | 18:10 |
| *** suuuper has quit IRC | 18:13 | |
| *** bogdando has quit IRC | 18:13 | |
| bkero | sshnaidm: openstack-infra/zuul/zuul/launcher/ansiblelaunchserver.py | 18:16 |
| *** arxcruz has joined #tripleo | 18:16 | |
| bkero | line 1008 | 18:16 |
| ansiwen | EmilienM: where does the hiera data that starts with heat::? I have a value "heat::keystone_ec2_uri": "http://172.17.1.13:5000/v2.0/ec2tokens", and wonder where it is set | 18:17 |
| ansiwen | *come from | 18:17 |
| *** garyk has quit IRC | 18:18 | |
| dprince | stevebaker: isn't cloudwatch deprecated? https://review.openstack.org/#/c/443095/5 | 18:22 |
| openstackgerrit | Saravanan KR proposed openstack/tripleo-common master: Workflow to select nodes matching a profile/role https://review.openstack.org/441054 | 18:24 |
| *** jprovazn has quit IRC | 18:26 | |
| skramaja | EmilienM: i have replied for https://review.openstack.org/#/c/449530/2/manifests/profile/base/neutron/ovs.pp@34, ptal | 18:27 |
| *** alop has joined #tripleo | 18:27 | |
| *** fzdarsky has joined #tripleo | 18:29 | |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-quickstart-extras master: Clone projects if they aren't cloned by ZUUL https://review.openstack.org/449562 | 18:29 |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-quickstart-extras master: DONT REVIEW: testing build role https://review.openstack.org/449985 | 18:29 |
| *** d0ugal has quit IRC | 18:31 | |
| *** Goneri has quit IRC | 18:34 | |
| *** yamahata has quit IRC | 18:36 | |
| *** paramite has joined #tripleo | 18:36 | |
| EmilienM | ansiwen: http://codesearch.openstack.org/?q=heat%3A%3Akeystone_ec2_uri&i=nope&files=&repos= | 18:37 |
| *** fragatina has joined #tripleo | 18:37 | |
| *** arxcruz has quit IRC | 18:39 | |
| *** milan has quit IRC | 18:39 | |
| *** milan has joined #tripleo | 18:40 | |
| *** maeca1 has joined #tripleo | 18:40 | |
| pabelanger | EmilienM: how are things merging today for gate? haven't been watching much | 18:42 |
| *** dcritch has joined #tripleo | 18:43 | |
| bkero | Hopefully the stable branches are stable | 18:44 |
| *** salmankhan has joined #tripleo | 18:45 | |
| *** milan has quit IRC | 18:46 | |
| *** bswartz has quit IRC | 18:47 | |
| *** d0ugal has joined #tripleo | 18:47 | |
| *** bswartz has joined #tripleo | 18:48 | |
| mwhahaha | stab...stable? what is that? | 18:49 |
| *** salmankhan has quit IRC | 18:49 | |
| *** paramite has quit IRC | 18:52 | |
| *** trown|compassing is now known as trown | 18:53 | |
| *** lblanchard has quit IRC | 18:54 | |
| mandre | dprince: do you have an idea of what could cause http://logs.openstack.org/89/422789/12/check-tripleo/gate-tripleo-ci-centos-7-ovb-containers-oooq-nv/9590a3c/logs/oooq/undercloud/home/jenkins/overcloud_deploy.log.txt.gz#_2017-03-27_15_26_20? | 18:54 |
| *** lblanchard has joined #tripleo | 18:54 | |
| *** fzdarsky has quit IRC | 18:55 | |
| mandre | we've started seing this failure recently | 18:55 |
| *** maeca1 has quit IRC | 18:56 | |
| EmilienM | pabelanger: http://tripleo.org/cistatus.html | 18:56 |
| EmilienM | and https://status-tripleoci.rhcloud.com/ | 18:56 |
| pabelanger | http://status.openstack.org/elastic-recheck/data/others.html | 18:57 |
| EmilienM | I don't think it is stable now | 18:57 |
| pabelanger | gate-tripleo-ci-centos-7-nonha-multinode-oooq (70) | 18:57 |
| EmilienM | 57.3% is not stable | 18:57 |
| pabelanger | down from 75 currently | 18:57 |
| dprince | mandre: looks like the root partition of the instack host is out of space? | 18:57 |
| dprince | mandre: how big is our image compared to the normal overcloud-full.qcow2? | 18:57 |
| EmilienM | gate-tripleo-ci-centos-7-nonha-multinode-oooq is 50% not that good | 18:58 |
| dprince | mandre: Before deploying, Ironic creates a raw image in the 'master_images' directory. This appears to be failing due to it running out of space | 18:58 |
| dprince | mandre: perhaps we were already close to the limit, and whatever is in the containers job image is larger in size, thus causing a failure | 18:59 |
| EmilienM | dprince, beekneemech: do we have metrics on ovb jobs & the time per overcloud step deploymentr? | 19:00 |
| *** beekneemech is now known as bnemec | 19:00 | |
| EmilienM | we're debugging why promotion fails to happen (timeout at step5) | 19:00 |
| bnemec | EmilienM: Not per step. I stopped collecting the heat resource times because some of them vary per job and bloated the graphite data. | 19:01 |
| dprince | EmilienM: I don't think I ever implented per-step metrics | 19:01 |
| EmilienM | ok | 19:01 |
| dprince | EmilienM: could be done... but would be a bit tricky I think | 19:01 |
| bnemec | We could whitelist the ones that don't change though. | 19:01 |
| bnemec | I've considered doing it, but since quickstart doesn't report to graphite at all I haven't been that motivated to mess with it further. | 19:01 |
| *** yamahata has joined #tripleo | 19:01 | |
| bnemec | EmilienM: If you just want single job metrics they're dumped out in postci though. | 19:01 |
| bnemec | That would give you an idea of what is normal. | 19:02 |
| EmilienM | mwhahaha: the logs I showed yo you are from this weekend I think. In the meantime, I see promotion job failing on bootstraping ironic nodes | 19:02 |
| EmilienM | bnemec: ah, right! indeed, thanks | 19:02 |
| EmilienM | so here's the fresh error we have in promotion job: http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-ovb-ha/ad539b7/console.html#_2017-03-27_08_16_02_158084 | 19:02 |
| mwhahaha | ah | 19:03 |
| EmilienM | http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-ovb-ha/ad539b7/logs/undercloud/var/log/nova/nova-compute.txt.gz#_2017-03-27_08_15_37_909 | 19:03 |
| bnemec | ControllerServiceChain 1406.0 | 19:03 |
| bnemec | Seriously? | 19:03 |
| EmilienM | InstanceDeployFailure: Failed to provision instance fb0f6da4-e2a9-44b7-a3cb-9545e37849b8: Timeout reached while waiting for callback | 19:03 |
| sshnaidm | EmilienM, it's because of http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-ovb-ha/ad539b7/logs/postci.txt.gz#_2017-03-27_08_31_19_000 | 19:04 |
| sshnaidm | EmilienM, but it's not for all other jobs | 19:04 |
| EmilienM | sshnaidm: yeah, sounds like we have multiple problems, trying to document them | 19:05 |
| openstackgerrit | Ben Kero proposed openstack/tripleo-quickstart-extras master: Refactor the toci-vxlan-networking script to be more correct https://review.openstack.org/446934 | 19:07 |
| *** rlandy is now known as rlandy|brb | 19:09 | |
| *** mcornea has quit IRC | 19:09 | |
| *** ooolpbot has joined #tripleo | 19:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674770 | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1676250 | 19:10 |
| *** ooolpbot has quit IRC | 19:10 | |
| openstack | Launchpad bug 1674770 in tripleo "Update timeout too long in CI" [Critical,Triaged] | 19:10 |
| openstack | Launchpad bug 1676250 in tripleo "Promotion to OpenStack trunk fails (timeout at step5)" [Critical,Triaged] | 19:10 |
| EmilienM | here's another timeout, but different: http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-ovb-updates/ce65e45/console.html#_2017-03-27_08_26_23_334496 | 19:10 |
| EmilienM | it happens at step 5 | 19:10 |
| *** jkilpatr has quit IRC | 19:11 | |
| EmilienM | (all these logs are from promotion jobs) | 19:11 |
| EmilienM | same for nonha: http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-ovb-nonha/1c5d57e/console.html#_2017-03-27_08_38_09_859482 - this time at PostDeploySteps | 19:12 |
| EmilienM | so it sounds like something in OpenStack makes us timeouting | 19:12 |
| ansiwen | EmilienM: sorry, my question was misleading, I mean more like: why is it set for heat and mistral, where is that used? maybe old stuff that can be removed? | 19:13 |
| *** gbarros has joined #tripleo | 19:14 | |
| *** panda|bbl is now known as panda | 19:14 | |
| *** florianf has quit IRC | 19:15 | |
| *** mhenkel_ has joined #tripleo | 19:16 | |
| mhenkel_ | hey All. Quick question: is there an easy way to retrieve the hostname of a node in a triple0 heat template? | 19:17 |
| mhenkel_ | something like: $hname: {get_param: Hostname} | 19:17 |
| *** florianf has joined #tripleo | 19:18 | |
| EmilienM | sshnaidm: why don't we have update_image(): IMAGE=/opt/stack/new/tripleo-ci/overcloud-full.qcow2 bits in the periodic job for promotion? | 19:22 |
| sshnaidm | bkero, slagle seems like subnode-2 logs are not overridden, but just complemented http://logs.openstack.org/51/450351/1/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/85cbaa9/logs/ | 19:22 |
| EmilienM | sshnaidm: good news | 19:22 |
| *** jkilpatr has joined #tripleo | 19:22 | |
| bkero | sshnaidm: cool | 19:22 |
| sshnaidm | EmilienM, why to update it? It's built from fresh repos | 19:22 |
| bkero | sshnaidm: So a question regarding tripleo-ci/scripts/tripleo.sh ovs_vxlan_setup | 19:23 |
| bkero | You chose offset 2 to begin, which makes the primary node start as 192.168.24.2 | 19:23 |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-quickstart master: Count subnodes from 2 https://review.openstack.org/450351 | 19:23 |
| EmilienM | sshnaidm: I'm reading logs and try to compare classic nonha that works today and the periodic one that fails to promote | 19:23 |
| bkero | Was there any reason for this? It makes the subnode indexing incorrect | 19:23 |
| sshnaidm | EmilienM, we don't update them because we already use repos from host when building them | 19:24 |
| bkero | According to infra, it should be (primary-1, subnode-2, subnode-3, subnode-4, etc). primary-1 is implicit though, because that would move logs around and change the path that devs expected logs to be at. | 19:24 |
| *** skramaja has quit IRC | 19:24 | |
| sshnaidm | bkero, sorry, I'm not sure I understand.. | 19:25 |
| *** arxcruz has joined #tripleo | 19:25 | |
| bkero | sshnaidm: this command here: https://git.openstack.org/cgit/openstack-infra/tripleo-ci/tree/scripts/tripleo.sh#n1317 | 19:25 |
| bkero | ovs_vxlan_bridge $PUB_BRIDGE_NAME $primary_node "True" 2 192.168.24 24 $sub_nodes | 19:25 |
| bkero | The '2' as arg[4] is the offset, which means it will start the primary node bridge interface as 192.168.24.2, instead of 192.168.24.1. | 19:26 |
| *** jlinkes has joined #tripleo | 19:26 | |
| bkero | That means the first subnode will be 192.168.24.3 | 19:26 |
| EmilienM | bnemec: yeah, ControllerServiceChain 1418.0 is terrible | 19:26 |
| bkero | That '3' or '2' is supposed to match 'subnode-2' in log collection | 19:26 |
| EmilienM | bnemec: do you know how to get more details on which resource took the most of time? | 19:27 |
| sshnaidm | bkero, sorry, I don't follow - how the inventory names related to IPs? | 19:27 |
| *** mwhahaha has quit IRC | 19:27 | |
| bkero | sshnaidm: by the number | 19:28 |
| *** radeks has quit IRC | 19:29 | |
| bkero | sshnaidm: The inventory name format is: "$ROLE-${IP_LAST_OCTET}" | 19:29 |
| bkero | Or, is supposed to be. | 19:29 |
| *** mwhahaha has joined #tripleo | 19:29 | |
| bnemec | EmilienM: You might need a bigger nested depth here: http://git.openstack.org/cgit/openstack-infra/tripleo-ci/tree/scripts/get_host_info.sh#n43 | 19:29 |
| openstackgerrit | Ben Kero proposed openstack/tripleo-quickstart-extras master: Refactor the toci-vxlan-networking script to be more correct https://review.openstack.org/446934 | 19:30 |
| bnemec | EmilienM: Although I'm not sure how much that will help. If the sub-resources are just numbers it's going to be hard to figure out which is which. | 19:30 |
| sshnaidm | bkero, subnode-{{ item.0 + 2 }} is inventory name, but IP will stay the same | 19:30 |
| sshnaidm | bkero, subnode-2 is just a name in hosts file, it shouldn't affect anything | 19:30 |
| sshnaidm | bkero, or I miss something..? | 19:30 |
| bkero | sshnaidm: It's also in zuul's inventory, which is collecting the logs | 19:31 |
| sshnaidm | bkero, ok, zuul inventory is limited to zuul tasks only | 19:31 |
| bkero | sshnaidm: I ran 'git blame' on tripleo.sh and found your name with the 'ovs_vxlan_bridge' call. I'm saying that our use of offset '2' to that call makes the IPs not line up to the inventory names like they are supposed to. | 19:31 |
| bkero | and I was wondering why it was done that way | 19:32 |
| sshnaidm | bkero, lemme check.. | 19:32 |
| *** eglynn has quit IRC | 19:34 | |
| sshnaidm | bkero, oh, I see, I changed there ip range only to 192.168.24 | 19:34 |
| sshnaidm | bkero, I think slagle should know more about offset | 19:35 |
| sshnaidm | bkero, but I still don't see where the problem is | 19:35 |
| EmilienM | ok something happens before step 1 that takes 50 min instead of 9 min | 19:35 |
| bkero | sshnaidm: The problem is that the offset of '2' instead of '1' causes the IPs not to match the names like they are expected to. | 19:36 |
| bkero | It's not causing an error right now, but it could potentially. | 19:37 |
| sshnaidm | bkero, why are they expected? | 19:37 |
| bkero | sshnaidm: devstack-gate scripts and zuul use them for things like log collection, host-ready scripts (multinode), and some other zuul tasks. | 19:37 |
| bkero | It's how zuul's ansible inventory is built, and that'll certainly be a problem with the zuulv3 transition | 19:38 |
| EmilienM | zaneb or therve (late for you): I would need your help asap to debug TripleO Ci promotion (we haven't promoted in the last 2 weeks) and I suspect this is Heat related | 19:38 |
| sshnaidm | bkero, but we don't change their hostnames or IPs, our inventory is completely internal stuff | 19:38 |
| EmilienM | zaneb, therve: my first suspicion is Heat being slower than before for some reasons | 19:39 |
| zaneb | EmilienM: o/ | 19:39 |
| EmilienM | zaneb: let me show you links : | 19:39 |
| sshnaidm | bkero, subnode-2 will stay the same in /etc/whatever | 19:39 |
| bkero | sshnaidm: Yes. I'm just saying it's technically incorrect in that it breaks upstream convention. It's not creating a problem right now though. | 19:39 |
| bkero | It could create a problem though. | 19:40 |
| sshnaidm | bkero, 192.168.24 is our internal stuff too, for infra it will be still 10.0.0... | 19:40 |
| EmilienM | # Without package promotion (OpenStack 2 weeks old) : from overcloud deploy to start step 1: ~10 min http://logs.openstack.org/10/425710/7/check-tripleo/gate-tripleo-ci-centos-7-ovb-nonha/204ef02/console.html#_2017-03-27_16_06_34_628054 | 19:40 |
| EmilienM | # With package promotion (from current OpenStack): from overcloud deploy to start step1: ~ 50 min http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-ovb-nonha/1c5d57e/console.html#_2017-03-27_08_08_53_275348 | 19:40 |
| sshnaidm | bkero, I'm curios too why offset is 2 though, let's ask slagle: why is it 2? https://github.com/openstack-infra/tripleo-ci/blob/f8ee060fdb68ef8a542295322943da3adcad68f8/scripts/tripleo.sh#L1317-L1317 | 19:40 |
| EmilienM | zaneb: problem reproducted on all 3 jobs that deploy with promoted repos, which means it's pretty consistent | 19:41 |
| zaneb | that's definitely too long | 19:41 |
| zaneb | can't imagine what might have caused that though | 19:41 |
| EmilienM | zaneb: see http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-ovb-nonha/1c5d57e/logs/postci.txt.gz#_2017-03-27_08_46_41_000 | 19:42 |
| EmilienM | ControllerServiceChain 1418.0 | 19:42 |
| *** maeca1 has joined #tripleo | 19:42 | |
| slagle | sshnaidm: it's 2 b/c .1 is used for local_ip in undercloud.conf by default and I wanted to keep that | 19:42 |
| zaneb | EmilienM: do you have the exact Heat commit IDs from before & after handy? | 19:42 |
| slagle | sshnaidm: why? | 19:42 |
| *** eglynn has joined #tripleo | 19:42 | |
| slagle | sshnaidm: so if you're going to have .1 on br-ctlplane, you need a different IP address on br-ex | 19:43 |
| *** maeca1 has quit IRC | 19:43 | |
| EmilienM | so in other words: ControllerServiceChain 103.0 without promotion and ControllerServiceChain 1418 with promotion | 19:43 |
| sshnaidm | slagle, so the next IP will be 2 or 3? | 19:43 |
| EmilienM | zaneb: 2 weeks ago and now, it's A LOT of commits | 19:43 |
| *** rlandy|brb is now known as rlandy | 19:43 | |
| slagle | sshnaidm: what do you mean by next IP? | 19:44 |
| bnemec | EmilienM: We need a dlrn-bisect command. | 19:44 |
| EmilienM | I need to look at the last 14 days and see when this thing started | 19:44 |
| bnemec | You could track down any bug in OpenStack to the commit that caused it, no matter the project. :-) | 19:44 |
| slagle | sshnaidm: i think the 2 means that ovs_vxlan_bridge will start at .2. but you could go look at that code to confirm | 19:44 |
| openstackgerrit | Ben Kero proposed openstack/tripleo-quickstart-extras master: Refactor the toci-vxlan-networking script to be more correct https://review.openstack.org/446934 | 19:45 |
| bkero | sshnaidm: It does do that, yes. | 19:45 |
| sshnaidm | slagle, uh, I see | 19:45 |
| EmilienM | the number of patches merged in Heat since then is not crazy, we could also look here | 19:45 |
| bkero | sshnaidm: We're wondering why it starts at .2 though, since the inventory names start at 1 | 19:45 |
| bkero | slagle: ^ | 19:45 |
| EmilienM | the version that worked well: openstack-heat-api-8.0.0-0.20170309072146.990f484.el7.centos.noarch | 19:46 |
| openstackgerrit | Sven Anderson proposed openstack/tripleo-heat-templates master: Add missing ec2api::api::keystone_ec2_tokens_url config https://review.openstack.org/450434 | 19:46 |
| slagle | bkero: what is an inventory name | 19:46 |
| bkero | I asked infra about this a bit ago -- The inventory name convention (that defined hostname in logs for example) is "$ROLE-$IP_LAST_OCTET". So that means primary node is primary-1, subnode-2, subnode-3, etc | 19:46 |
| bkero | slagle: In zuul's ansible inventory in this case | 19:46 |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart master: Download overcloud_release rpm for mixed upgrade https://review.openstack.org/449349 | 19:46 |
| *** rhallisey has quit IRC | 19:46 | |
| openstackgerrit | Sven Anderson proposed openstack/tripleo-heat-templates stable/ocata: Add missing ec2api::api::keystone_ec2_tokens_url config https://review.openstack.org/450435 | 19:47 |
| slagle | bkero: apparently not, b/c subnode-2 is the dir that gets created during log creation, but the IP address is .3 for that subnode | 19:47 |
| EmilienM | zaneb: could it be https://github.com/openstack/heat/commit/4a500125b350b46dee0d3c9f01c3cac7223d9c80 ? (note: we don't use converge in tripleo CI afik) | 19:47 |
| slagle | bkero: i'm pretty sure nothing in tripleo-ci creates that subnode-2 dir | 19:47 |
| zaneb | EmilienM: yeah, convergence is disabled so it can't be that | 19:47 |
| bkero | slagle: That's my point. The IP address for that is 3, but it should be 2. | 19:47 |
| slagle | bkero: no. it shouldn't. the name of the dir should be subnode-3 | 19:48 |
| bkero | Err, no | 19:48 |
| slagle | bkero: the assumption in the inventory name sounds like the problem to me | 19:48 |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder master: Have simple-init try to enable network.service https://review.openstack.org/450436 | 19:48 |
| slagle | bkero: the function supports an offset | 19:48 |
| slagle | bkero: we use it. i don't create an inventory name based on the offset | 19:49 |
| bkero | Yes. Using that offset for anything other than '1' means that it doesn't match up with zuul's inventory and log collection though. | 19:49 |
| EmilienM | zaneb: ok I haven't found any commit in Heat. Now I'm going to find out when this thing started. | 19:49 |
| *** v1k0d3n has quit IRC | 19:49 | |
| EmilienM | zaneb: in the meantime, if you find anything useful in the links I posted (heat logs, etc) | 19:49 |
| bkero | Since the subnode would be .3, even though zuul's log collection has it as subnode-2 | 19:49 |
| slagle | bkero: doesnt match what? | 19:49 |
| slagle | bkero: it is matched, everything is in subnode-2 | 19:50 |
| bkero | The 2 from '-2' comes from what zuul expects the last octet of the IP to be. | 19:50 |
| zaneb | EmilienM: ok | 19:50 |
| bkero | It doesn't actually have that hardcoded, but that is the assumption | 19:50 |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder master: Have simple-init try to start network.service https://review.openstack.org/450436 | 19:50 |
| *** v1k0d3n has joined #tripleo | 19:51 | |
| slagle | bkero: sounds like a bad assumption, or offset shouldnt be supported i guess | 19:51 |
| bkero | slagle: I'm wondering why the offset of '2' was used | 19:51 |
| *** jlinkes has quit IRC | 19:51 | |
| *** florianf has quit IRC | 19:51 | |
| slagle | bkero: which i just explained | 19:51 |
| slagle | bkero: sounds like the name is just a label really, since it obviously isn't assuming the IP is actually .2 anywhere | 19:52 |
| slagle | bkero: otherwise, it would be connecting to the wrong node | 19:52 |
| bkero | slagle: It's not creating a problem now, and it is just a label. It could be an incorrect label once we start using more N>2 jobs though | 19:53 |
| bkero | Or an unclear label | 19:53 |
| slagle | we already have n>2 jobs | 19:53 |
| bkero | Correct, although I don't know how much or how many people have been digging into logs attempting to understand things though. | 19:53 |
| slagle | bkero: what was unclear to me was why we had subnode-1 and subnode-2 dirs | 19:54 |
| bkero | slagle: subnode-1 was behavior inside quickstart, sshnaidm submitted a patch to change that to -2 | 19:54 |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder master: Have simple-init try to enable network.service https://review.openstack.org/450436 | 19:54 |
| slagle | bkero: b/c the multinode non-ha was only 1 subnode before the quickstart transition | 19:54 |
| slagle | bkero: so subnode-1 is for .1? that would be the primary node though | 19:55 |
| bkero | slagle: No, subnode-1 was for .3, which was the subnode | 19:55 |
| bkero | My concern would be, on a 3-node job for example, the job would have /logs for the primary node, /logs/subnode-2 for the subnode 192.168.24.3, /logs/subnode-3 for the subnode 192.168.24.4, and that seems odd. | 19:56 |
| bkero | and could create confusion | 19:56 |
| slagle | ok, that's exactly what we have today, and i've not heard of any confusion | 19:57 |
| mandre | dprince: idk, that's odd, we're using the normal overcloud-full.qcow2 | 19:57 |
| bkero | I don't know if many others here besides you have had to debug 3node jobs though | 19:58 |
| mandre | dprince: but you're right, the disk may be filled by our docker images as we add more services | 19:58 |
| dprince | mandre: okay, just my initial take on it. Could be I'm missing something | 19:58 |
| bkero | and with only one subnode, it wouldn't create confusion. More than one might be a problem though. | 19:58 |
| *** florianf has joined #tripleo | 19:58 | |
| dprince | mandre: exactly, it could be all related to the registry | 19:58 |
| slagle | bkero: others have debugged 3 node jobs | 19:58 |
| slagle | bkero: sorry, just speaking from evidence here. | 19:59 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Sort ResourceGroup resource list https://review.openstack.org/435121 | 19:59 |
| slagle | bkero: anything can always be changed though | 19:59 |
| mandre | dprince: are you pushing new images to the docker hub? | 19:59 |
| dprince | mandre: yes | 19:59 |
| dprince | mandre: we are over 2 weeks old | 20:00 |
| mandre | dprince: so you may be fixing my issue in the process :) | 20:00 |
| slagle | bkero: the subnode-$ip convention is new to me, i don't think anyone in tripleo was anticipating that | 20:00 |
| dprince | mandre: I may, but I could be breaking somethign else | 20:00 |
| slagle | bkero: clearly offset can't be used if that assumption is baked into the zuul inventory and you want that assumption to be correct | 20:00 |
| mandre | dprince: since i suspect the images were larger than necessary because we built them in several batches and they had different base images | 20:01 |
| slagle | bkero: so you'll have to find a way to not use offset | 20:01 |
| bkero | slagle: I didn't understand why it would begin with subnode-2 either, so I asked infra. That's what they told me. | 20:01 |
| bkero | It's also assuming devstack-gate's use of ovs_vxlan_bridge, not our source/calling | 20:01 |
| dprince | mandre: I need this fix too https://review.openstack.org/#/c/450380/ | 20:01 |
| *** dsariel has joined #tripleo | 20:01 | |
| slagle | bkero: nothing says we have to use the function at all | 20:02 |
| sshnaidm | bnemec, do yo know maybe why is it configured so? https://github.com/openstack-infra/tripleo-ci/blob/f8ee060fdb68ef8a542295322943da3adcad68f8/toci_gate_test-orig.sh#L14-L15 | 20:03 |
| sshnaidm | bnemec, rh1 cloud, rh2.env..? | 20:03 |
| slagle | bkero: i think it's fair game to have the jobs use os-net-config instead. or, write something entirely new | 20:03 |
| bkero | slagle: That's true. I the assumption exists that if one is performing a multinode job, that one will have a network between them though. | 20:03 |
| bkero | So something analogous would exist | 20:03 |
| *** jlinkes has joined #tripleo | 20:04 | |
| slagle | bkero: yes, if you want to route a private cidr around the nodes, you'd need something similar | 20:04 |
| bnemec | sshnaidm: I'm betting it was a sed mistake when we moved clouds. | 20:04 |
| bnemec | sshnaidm: Since the mirror/proxy/etc addresses are all the same it works anyway. | 20:04 |
| bnemec | I think that's the only thing we actually use from that file. | 20:04 |
| EmilienM | zaneb: ok I found something interesting | 20:04 |
| bkero | slagle: I'm mostly thinking into a future of zuulv3 and inheriting the inventory that already includes these hosts with certain labels | 20:05 |
| zaneb | EmilienM: that's good because I've found nothing :) | 20:05 |
| EmilienM | zaneb: the promotion job on March 17th failed but the ControllerServiceChain was not slow | 20:05 |
| sshnaidm | bnemec, and PUBLIC_IP_NET is not used I suppose.. | 20:05 |
| EmilienM | ControllerServiceChain 151.0 | 20:05 |
| EmilienM | so, it means something broke us between 17 & 25 | 20:06 |
| bnemec | sshnaidm: No, that's just for deploying the cloud in the first place. | 20:06 |
| slagle | bkero: i think the some positive iterative improvements in this area would be welcome | 20:06 |
| EmilienM | I keep looking | 20:06 |
| slagle | bkero: there is no "requirement" to copy what's been done | 20:06 |
| slagle | bkero: e.g., i dont know of anyone asking for that | 20:06 |
| slagle | in fact, the opposite | 20:06 |
| sshnaidm | bnemec, and where is NODEPOOL_CLOUD taken from..? | 20:07 |
| bkero | slagle: I'll keep that in mind and think about better ways to implement this then. | 20:07 |
| bnemec | sshnaidm: I would assume that's passed in from nodepool | 20:08 |
| bkero | Thank you for sating my curiosity | 20:08 |
| *** akrivoka has quit IRC | 20:08 | |
| slagle | sure, np | 20:08 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Fix typo in cloud number https://review.openstack.org/450442 | 20:09 |
| sshnaidm | bnemec, yeah, it's in /etc/nodepool/provider | 20:10 |
| *** ooolpbot has joined #tripleo | 20:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674770 | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1676250 | 20:10 |
| *** ooolpbot has quit IRC | 20:10 | |
| openstack | Launchpad bug 1674770 in tripleo "Update timeout too long in CI" [Critical,Triaged] | 20:10 |
| openstack | Launchpad bug 1676250 in tripleo "Promotion to OpenStack trunk fails (timeout at step5)" [Critical,Triaged] | 20:10 |
| openstackgerrit | Ben Nemec proposed openstack/python-tripleoclient master: Call undercloud install function directly https://review.openstack.org/431145 | 20:10 |
| bnemec | I started trying to rebase ^ this morning | 20:10 |
| bnemec | Six hours later I actually got it done. | 20:11 |
| bnemec | (it was a pretty trivial rebase too) | 20:11 |
| openstackgerrit | Ben Kero proposed openstack/tripleo-quickstart-extras master: Refactor the toci-vxlan-networking script to be more correct https://review.openstack.org/446934 | 20:11 |
| zaneb | EmilienM: I can't see any commits that even look suspicious in that time frame | 20:11 |
| EmilienM | zaneb: yeah, not easy. And in heat logs? nothing wrong? | 20:11 |
| zaneb | EmilienM: will check those next | 20:12 |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates stable/ocata: Sort ResourceGroup resource list https://review.openstack.org/450444 | 20:14 |
| EmilienM | sshnaidm: would it be possible to run tripleo-ci on a specific hash + a specific commit in Nova applied? | 20:18 |
| sshnaidm | EmilienM, with quickstart? | 20:19 |
| EmilienM | sshnaidm: I found out that the timeout issue that we hit in promotion job started on March 18th, but in the same time as the Nova regression if you remember | 20:19 |
| EmilienM | sshnaidm: which makes impossible or very hard to find which day it really broke us (timeout) | 20:19 |
| sshnaidm | EmilienM, yeah, right | 20:19 |
| EmilienM | so if we manage to reproduce what we had on March 18, until 24 included with the nova patch applied | 20:19 |
| EmilienM | we would find out which day what broke us | 20:19 |
| EmilienM | sshnaidm: oooq or whatever. It has to be OVB though | 20:20 |
| openstackgerrit | Tim Rozet proposed openstack/tripleo-heat-templates master: Fixes port binding controller for OpenDaylight https://review.openstack.org/448831 | 20:20 |
| sshnaidm | EmilienM, by specific hash you mean delorean hash, right? | 20:20 |
| EmilienM | sshnaidm: yes | 20:20 |
| sshnaidm | EmilienM, yeah, sure | 20:20 |
| EmilienM | sshnaidm: but on top of this has, I want to build a specific version of nova | 20:20 |
| EmilienM | this hash* | 20:20 |
| sshnaidm | EmilienM, about nova I should look... | 20:21 |
| EmilienM | and I want to do it for the 7 days so we can find which day it broke | 20:21 |
| *** toure is now known as toure|afk | 20:23 | |
| *** jlinkes has quit IRC | 20:24 | |
| EmilienM | zaneb: could it be related to https://github.com/openstack/instack-undercloud/commit/9f23fbda47bf4e12c22744fb9a3cf784619c7f1a ? | 20:25 |
| EmilienM | is ControllerServiceChain using zaqar somehow? | 20:26 |
| zaneb | no idea. anything is possible | 20:26 |
| zaneb | I think we're using zaqar for signalling in tripleo? | 20:26 |
| EmilienM | yes | 20:27 |
| *** jlinkes has joined #tripleo | 20:27 | |
| *** salmankhan has joined #tripleo | 20:27 | |
| *** jayg is now known as jayg|g0n3 | 20:27 | |
| sshnaidm | EmilienM, I see last successful consistent delorean hash was in 15.03 | 20:28 |
| EmilienM | sshnaidm: what does it mean? | 20:29 |
| *** Goneri has joined #tripleo | 20:29 | |
| EmilienM | sshnaidm: I see last successful periodic job is on March 17th | 20:29 |
| sshnaidm | EmilienM, ok, so using this hash from 17.03 | 20:31 |
| sshnaidm | EmilienM, and which commit of nova? | 20:31 |
| zaneb | EmilienM: so from the logs it looks like for multiple things (but not everything) in overcloud-ControllerServiceChain it takes exactly 2 minutes to start the next thing after the previous thing was completed | 20:32 |
| EmilienM | sshnaidm: gimme asec | 20:33 |
| zaneb | EmilienM: whereas when it worked, the whole thing completed in 2 minutes | 20:33 |
| zaneb | (roughly) | 20:33 |
| EmilienM | sshnaidm: https://review.openstack.org/#/c/448098/ | 20:34 |
| openstackgerrit | Harry Rybacki proposed openstack/tripleo-quickstart-extras master: [WIP] Add role to deploy FreeIPA https://review.openstack.org/436198 | 20:34 |
| *** maeca1 has joined #tripleo | 20:35 | |
| *** liverpooler has quit IRC | 20:35 | |
| *** maeca1 has quit IRC | 20:35 | |
| openstackgerrit | Sven Anderson proposed openstack/tripleo-heat-templates master: Add missing ec2api::api::keystone_ec2_tokens_url config https://review.openstack.org/450434 | 20:35 |
| sshnaidm | EmilienM, so this one https://github.com/openstack/nova/commit/fe8415060ca452990d7019a03eaaa4b92aadfe8b | 20:36 |
| EmilienM | sshnaidm: exactly | 20:36 |
| openstackgerrit | Sven Anderson proposed openstack/tripleo-heat-templates stable/ocata: Add missing ec2api::api::keystone_ec2_tokens_url config https://review.openstack.org/450435 | 20:36 |
| openstackgerrit | Sven Anderson proposed openstack/puppet-tripleo master: Add missing include of ::ec2api::keystone::authtoken https://review.openstack.org/450294 | 20:37 |
| openstackgerrit | Sven Anderson proposed openstack/puppet-tripleo stable/ocata: Add missing include of ::ec2api::keystone::authtoken https://review.openstack.org/450299 | 20:38 |
| *** mhenkel_ has quit IRC | 20:38 | |
| *** mhenkel_ has joined #tripleo | 20:39 | |
| EmilienM | ansiwen: repeated here ^ but please don't backport things before it gets merged in master. You don't know if the code will change or not. | 20:39 |
| EmilienM | which is the case, you made a typo | 20:39 |
| ansiwen | ok | 20:40 |
| *** mhenkel_ has quit IRC | 20:43 | |
| EmilienM | sshnaidm: in the meantime I also want to try a revert of https://github.com/openstack/instack-undercloud/commit/30f2d9a3c32d25919f1ececb31523a748a115e5f | 20:44 |
| EmilienM | sshnaidm: on top of your patch https://review.openstack.org/#/c/447514/ | 20:44 |
| openstackgerrit | Florian Fuchs proposed openstack/tripleo-validations master: [WIP]Show all roles/services in inventory https://review.openstack.org/450233 | 20:45 |
| *** jcoufal has quit IRC | 20:48 | |
| *** ansmith has quit IRC | 20:49 | |
| *** ckyriakidou has joined #tripleo | 20:50 | |
| *** dprince has quit IRC | 20:58 | |
| *** gbarros has quit IRC | 20:59 | |
| *** rhallisey has joined #tripleo | 20:59 | |
| *** trown is now known as trown|outtypewww | 21:01 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: DONT MERGE: testing consistent repo with nova commit https://review.openstack.org/450455 | 21:02 |
| sshnaidm | EmilienM, I hope that will work ^^ | 21:02 |
| *** gfidente|afk has quit IRC | 21:02 | |
| *** mhenkel_ has joined #tripleo | 21:03 | |
| EmilienM | sshnaidm: funky | 21:03 |
| EmilienM | sshnaidm: we'll see | 21:03 |
| EmilienM | sshnaidm: thanks for trying | 21:03 |
| openstackgerrit | Sven Anderson proposed openstack/tripleo-heat-templates master: Add missing ec2api::api::keystone_ec2_tokens_url config https://review.openstack.org/450434 | 21:03 |
| *** lblanchard has quit IRC | 21:06 | |
| *** ooolpbot has joined #tripleo | 21:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674770 | 21:10 |
| openstack | Launchpad bug 1674770 in tripleo "Update timeout too long in CI" [Critical,Triaged] | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1676250 | 21:10 |
| *** ooolpbot has quit IRC | 21:10 | |
| openstack | Launchpad bug 1676250 in tripleo "Promotion jobs fail on timeout, ControllerServiceChain takes too long" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 21:10 |
| *** ckyriakidou has quit IRC | 21:11 | |
| *** florianf has quit IRC | 21:11 | |
| *** ccamacho has quit IRC | 21:12 | |
| *** fragatin_ has joined #tripleo | 21:13 | |
| *** fragatina has quit IRC | 21:16 | |
| *** abishop_ is now known as abishop | 21:16 | |
| *** jobewan has joined #tripleo | 21:16 | |
| *** panda is now known as panda|zZ | 21:17 | |
| *** salmankhan has quit IRC | 21:19 | |
| *** abishop has quit IRC | 21:22 | |
| *** rbrady is now known as rbrady-afk | 21:27 | |
| *** fragatin_ has quit IRC | 21:29 | |
| *** fragatina has joined #tripleo | 21:29 | |
| *** jlinkes has quit IRC | 21:32 | |
| openstackgerrit | Emilien Macchi proposed openstack/python-tripleoclient master: Stop using os-cloud-config https://review.openstack.org/450465 | 21:40 |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-quickstart master: WIP: Use AFS mirrors for centos hammer and jewel repos https://review.openstack.org/450466 | 21:42 |
| EmilienM | slagle: https://review.openstack.org/#/c/450465/ http://my1.fr/files/emilien-right-now.jpg | 21:42 |
| *** sshnaidm is now known as sshnaidm|off | 21:43 | |
| slagle | EmilienM: like i know. +2 though, that's what ci is for | 21:45 |
| pabelanger | EmilienM: noticed the gate just reset for tripleo, have a log of the failure handy? | 21:46 |
| *** jmelvin has quit IRC | 21:49 | |
| *** dparkes has joined #tripleo | 21:49 | |
| openstackgerrit | Ben Kero proposed openstack-infra/tripleo-ci master: Revert second repo-setup run for overcloud nodes https://review.openstack.org/447142 | 21:50 |
| bkero | weshay: I figured out the link you sent me earlier | 21:50 |
| bkero | weshay: Well, maybe. The overcloud-deploy-post tag wasn't being included. | 21:51 |
| openstackgerrit | Ben Kero proposed openstack-infra/tripleo-ci master: Revert second repo-setup run for overcloud nodes https://review.openstack.org/447142 | 21:52 |
| *** bfournie has quit IRC | 21:52 | |
| EmilienM | pabelanger: no | 21:55 |
| EmilienM | pabelanger: see the WIP by sshnaidm|off, https://review.openstack.org/#/c/450466/1 | 21:56 |
| EmilienM | pabelanger: for logs, maybe bnemec can provide it though | 21:56 |
| pabelanger | k | 21:57 |
| pabelanger | if you want to send it over, I can look | 21:57 |
| pabelanger | EmilienM: network issues should be under control now | 21:57 |
| pabelanger | so, trying to see if jobs are still failing because of that | 21:57 |
| bnemec | What do we need logs for? | 21:58 |
| pabelanger | I haven't see a failure all day according to logstash.o.o | 21:58 |
| pabelanger | bnemec: I want to see why the gate pipeline for tripleo change queue reset | 21:58 |
| pabelanger | its almost at 6hrs | 21:58 |
| pabelanger | want to see if it is network still or something else | 21:59 |
| EmilienM | adarazs, trown|outtypewww: thx for moving bugs in lp | 22:02 |
| *** chlong has quit IRC | 22:03 | |
| bnemec | Oh, I don't have any particular insight into the gate queue. | 22:03 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci master: Force deploy commands to timeout https://review.openstack.org/450302 | 22:06 |
| *** jkilpatr has quit IRC | 22:07 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/tripleo-heat-templates master: Updated from global requirements https://review.openstack.org/440124 | 22:08 |
| *** cdearborn has quit IRC | 22:09 | |
| *** ooolpbot has joined #tripleo | 22:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674770 | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1676250 | 22:10 |
| *** ooolpbot has quit IRC | 22:10 | |
| openstack | Launchpad bug 1674770 in tripleo "Update timeout too long in CI" [Critical,Triaged] | 22:10 |
| openstack | Launchpad bug 1676250 in tripleo "Promotion jobs fail on timeout, ControllerServiceChain takes too long" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 22:10 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci master: Use shorter timeout for update step https://review.openstack.org/448778 | 22:11 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci master: Force deploy commands to timeout https://review.openstack.org/450302 | 22:11 |
| *** karimb has quit IRC | 22:21 | |
| therve | EmilienM, There is something going on with zaqar in the promo job | 22:21 |
| therve | "Script timed out before returning headers: zaqar-server" in the httpd logs | 22:21 |
| therve | "IOError: request data read error" in zaqar logs | 22:21 |
| *** bfournie has joined #tripleo | 22:25 | |
| *** gkadam has joined #tripleo | 22:26 | |
| EmilienM | therve: yeah? I looked at the Ci jobs when we made the transition and I saw 0 problem. | 22:27 |
| EmilienM | Maybe something changed in the meantime | 22:27 |
| therve | EmilienM, https://github.com/openstack/python-tripleoclient/commit/36b6b09fb307399458a9bfadef497cbcae35f3c4 | 22:27 |
| therve | The time seems to be around that | 22:28 |
| therve | We can try to revert that one too for mitigation | 22:28 |
| EmilienM | http://logs.openstack.org/20/394420/5/check-tripleo/gate-tripleo-ci-centos-7-ovb-nonha/4c2619c/console.html#_2017-03-16_01_39_39_356357 | 22:29 |
| EmilienM | looking at the Ci job that ran on your patch, I don't see any problem | 22:29 |
| EmilienM | the first phase took 10min | 22:29 |
| EmilienM | sshnaidm|off: not sure it worked: http://logs.openstack.org/55/450455/1/check/gate-tripleo-ci-centos-7-undercloud-oooq/e5ec94a/logs/rpm-qa.txt.gz | 22:30 |
| EmilienM | sshnaidm|off: openstack-nova-api-15.0.0-0.20170311221824.0f29bad.el7.centos.noarch is too old I think | 22:30 |
| therve | So yeah maybe the wsgi setup is broken then | 22:31 |
| EmilienM | therve: ok, I'll give it a try. It's blocking promotion jobs. | 22:31 |
| sshnaidm|off | EmilienM, it should be in non-oooq jobs (ovb) | 22:33 |
| EmilienM | sshnaidm|off: oh, ok | 22:33 |
| *** jkilpatr has joined #tripleo | 22:35 | |
| *** Goneri has quit IRC | 22:37 | |
| therve | EmilienM, https://review.openstack.org/#/c/444671/ should have improved things, but maybe it didn't :/ | 22:37 |
| zaneb | EmilienM: I can't find anything in the Heat logs to suggest why it's slow | 22:38 |
| *** cwolferh has quit IRC | 22:40 | |
| EmilienM | therve: where did you see the IOError: request data read error" in zaqar logs ? | 22:40 |
| therve | EmilienM, http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-ovb-nonha/1c5d57e/logs/undercloud.tar.xz I believe | 22:40 |
| EmilienM | ah indeed | 22:41 |
| EmilienM | let me look timestamps | 22:41 |
| openstackgerrit | Pradeep Kilambi proposed openstack/puppet-tripleo master: Restrict mongodb memory usage https://review.openstack.org/419090 | 22:42 |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates master: Containerize Redis service https://review.openstack.org/442151 | 22:43 |
| EmilienM | therve: yeah, it makes sense | 22:43 |
| EmilienM | we didn't have this error a few weeks ago (on March 17th at least) | 22:43 |
| *** mhenkel_ has quit IRC | 22:44 | |
| EmilienM | probably because we didn't use zaqar for getting stack events | 22:44 |
| therve | Yeah that would increase the number of messages by a lot | 22:46 |
| EmilienM | therve: the second question is why so much bad events | 22:47 |
| EmilienM | therve: I'm going to try a revert just for testing | 22:47 |
| openstackgerrit | Emilien Macchi proposed openstack/python-tripleoclient master: Revert "Use a Zaqar queue to get stack events" https://review.openstack.org/450481 | 22:48 |
| *** dmarlin has quit IRC | 22:48 | |
| EmilienM | sshnaidm|off: can i push a patch on top of https://review.openstack.org/#/c/447514/ that depends-on the revert ^ ? | 22:49 |
| sshnaidm|off | EmilienM, yep | 22:50 |
| therve | EmilienM, There is no bad events. Just something weird happening in the http layer | 22:51 |
| EmilienM | ok | 22:51 |
| *** morazi_ has quit IRC | 22:52 | |
| *** cwolferh has joined #tripleo | 22:52 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: DNM - test rever zaqar / oooclient https://review.openstack.org/450483 | 22:53 |
| * therve off | 22:55 | |
| *** jobewan has quit IRC | 22:58 | |
| *** cwolferh has quit IRC | 23:00 | |
| *** mhenkel_ has joined #tripleo | 23:02 | |
| *** dsariel has quit IRC | 23:05 | |
| *** dparkes has quit IRC | 23:06 | |
| *** ooolpbot has joined #tripleo | 23:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1674770 | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1676250 | 23:10 |
| *** ooolpbot has quit IRC | 23:10 | |
| openstack | Launchpad bug 1674770 in tripleo "Update timeout too long in CI" [Critical,Triaged] | 23:10 |
| openstack | Launchpad bug 1676250 in tripleo "Promotion jobs fail on timeout, ControllerServiceChain takes too long" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 23:10 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Have simple-init try to enable network.service https://review.openstack.org/450436 | 23:19 |
| openstackgerrit | Dan Sneddon proposed openstack/tripleo-specs master: blueprint tripleo-routed-networks-deployment https://review.openstack.org/421009 | 23:23 |
| *** thrash is now known as thrash|g0ne | 23:23 | |
| openstackgerrit | Honza Pokorny proposed openstack/tripleo-specs master: Add gui logging spec https://review.openstack.org/395138 | 23:25 |
| honza | EmilienM: both of my specs should now merge cleanly | 23:26 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Have simple-init try to enable network.service https://review.openstack.org/450436 | 23:26 |
| EmilienM | honza: cool, I'll review them this week, thx again | 23:29 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Have simple-init enable network.service https://review.openstack.org/450436 | 23:29 |
| honza | EmilienM: Writing specs is hard work, and I'm not very good at it. Hope it's useful. | 23:29 |
| EmilienM | honza: it's hard but helpful to understand what you're doing and so far your specs have been excellent imho | 23:32 |
| *** pmannidi has joined #tripleo | 23:36 | |
| *** alop has quit IRC | 23:39 | |
| *** fragatin_ has joined #tripleo | 23:55 | |
| *** fragatina has quit IRC | 23:58 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
