| *** ctayal has quit IRC | 00:00 | |
| *** morazi has quit IRC | 00:04 | |
| *** ctayal has joined #tripleo | 00:04 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Allow package-installs to parse DIB_PYTHON_VERSION https://review.openstack.org/408293 | 00:04 |
|---|---|---|
| openstackgerrit | Merged openstack/diskimage-builder: Update sysctl-write-value to do conflict checking https://review.openstack.org/406413 | 00:04 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Move pip-and-virtualenv source install to 10- https://review.openstack.org/408277 | 00:07 |
| *** ansmith has joined #tripleo | 00:07 | |
| *** mhenkel has quit IRC | 00:08 | |
| *** ooolpbot has joined #tripleo | 00:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 00:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649615 | 00:10 |
| *** ooolpbot has quit IRC | 00:10 | |
| openstack | Launchpad bug 1649615 in tripleo "Undercloud running out of disk space in CI" [Critical,Fix committed] | 00:10 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Disable centos6 testing https://review.openstack.org/408936 | 00:11 |
| EmilienM | bnemec: any change with the postci thing? I'll report a bug otherwise | 00:11 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: DONT REVIEW: test https://review.openstack.org/410470 | 00:14 |
| *** ctayal has quit IRC | 00:16 | |
| *** dsneddon has joined #tripleo | 00:20 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Add install-types as pip-and-virtualenv dep https://review.openstack.org/408248 | 00:21 |
| *** dmacpher-afk has quit IRC | 00:21 | |
| *** rcernin has quit IRC | 00:22 | |
| *** ctayal has joined #tripleo | 00:24 | |
| *** dsneddon has quit IRC | 00:25 | |
| openstackgerrit | Merged openstack/diskimage-builder: Move pip-and-virtualenv source install to 10- https://review.openstack.org/408277 | 00:36 |
| *** dsneddon has joined #tripleo | 00:49 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Allow package-installs to parse DIB_PYTHON_VERSION https://review.openstack.org/408293 | 00:52 |
| *** limao has joined #tripleo | 00:55 | |
| openstackgerrit | Merged openstack/diskimage-builder: Add install-types as pip-and-virtualenv dep https://review.openstack.org/408248 | 00:57 |
| *** jeckersb_gone is now known as jeckersb | 01:01 | |
| *** dsneddon has quit IRC | 01:03 | |
| *** dmacpher has joined #tripleo | 01:05 | |
| *** ctayal has quit IRC | 01:06 | |
| *** dsneddon has joined #tripleo | 01:08 | |
| *** dsneddon has quit IRC | 01:10 | |
| *** ooolpbot has joined #tripleo | 01:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649615 | 01:10 |
| *** ooolpbot has quit IRC | 01:10 | |
| openstack | Launchpad bug 1649615 in tripleo "Undercloud running out of disk space in CI" [Critical,Fix committed] | 01:10 |
| *** dsneddon_ has joined #tripleo | 01:11 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Allow package-installs to parse DIB_PYTHON_VERSION https://review.openstack.org/408293 | 01:13 |
| openstackgerrit | Merged openstack/diskimage-builder: Disable centos6 testing https://review.openstack.org/408936 | 01:14 |
| EmilienM | I filed https://bugs.launchpad.net/tripleo/+bug/1649743 | 01:19 |
| openstack | Launchpad bug 1649743 in tripleo "postci timeouts on ovb-ha" [Critical,Triaged] | 01:19 |
| *** jkilpatr_ has quit IRC | 01:22 | |
| *** saneax is now known as saneax-_-|AFK | 01:35 | |
| *** cwolferh has quit IRC | 01:37 | |
| *** rbrady has quit IRC | 01:47 | |
| *** ctayal has joined #tripleo | 01:59 | |
| *** ctayal_ has joined #tripleo | 02:05 | |
| *** ctayal has quit IRC | 02:05 | |
| *** ooolpbot has joined #tripleo | 02:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649615 | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649742 | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649743 | 02:10 |
| *** ooolpbot has quit IRC | 02:10 | |
| openstack | Launchpad bug 1649615 in tripleo "Undercloud running out of disk space in CI" [Critical,Fix committed] | 02:10 |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha" [Critical,Triaged] | 02:10 |
| openstack | Launchpad bug 1649743 in tripleo "postci timeouts on ovb-ha" [Critical,Triaged] | 02:10 |
| *** rbrady has joined #tripleo | 02:10 | |
| *** ansmith has quit IRC | 02:12 | |
| *** another_larsks has quit IRC | 02:14 | |
| *** larsks has joined #tripleo | 02:18 | |
| *** fragatin_ has joined #tripleo | 02:23 | |
| openstackgerrit | Merged openstack/diskimage-builder: Allow package-installs to parse DIB_PYTHON_VERSION https://review.openstack.org/408293 | 02:23 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart: move the undercloud deploy role to quickstart-extras for composability https://review.openstack.org/410427 | 02:24 |
| *** fragatina has quit IRC | 02:27 | |
| *** fragatin_ has quit IRC | 02:28 | |
| *** lblanchard has quit IRC | 02:35 | |
| *** thrash is now known as thrash|g0ne | 02:44 | |
| *** mrunge has quit IRC | 02:49 | |
| *** mrunge has joined #tripleo | 02:54 | |
| *** cwolferh has joined #tripleo | 03:00 | |
| *** dsneddon has joined #tripleo | 03:07 | |
| *** dsneddo__ has joined #tripleo | 03:08 | |
| *** dsneddon_ has quit IRC | 03:09 | |
| *** ooolpbot has joined #tripleo | 03:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649615 | 03:10 |
| openstack | Launchpad bug 1649615 in tripleo "Undercloud running out of disk space in CI" [Critical,Fix committed] | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649742 | 03:10 |
| *** ooolpbot has quit IRC | 03:10 | |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha" [Critical,Triaged] | 03:10 |
| *** dsneddon has quit IRC | 03:11 | |
| *** dsneddo__ has quit IRC | 03:18 | |
| *** EmilienM has quit IRC | 03:19 | |
| *** EmilienM has joined #tripleo | 03:20 | |
| *** dsneddon has joined #tripleo | 03:20 | |
| *** yamahata_ has quit IRC | 03:24 | |
| *** coolsvap has joined #tripleo | 03:27 | |
| openstackgerrit | RedHat RDO CI proposed openstack/tripleo-heat-templates: GATE TEST, please ignore https://review.openstack.org/360618 | 03:30 |
| openstackgerrit | RedHat RDO CI proposed openstack/tripleo-heat-templates: GATE TEST, please ignore https://review.openstack.org/365449 | 03:31 |
| *** prateek has joined #tripleo | 03:49 | |
| *** dsneddon_ has joined #tripleo | 03:56 | |
| *** dsneddon has quit IRC | 03:56 | |
| *** pgadiya has joined #tripleo | 04:00 | |
| *** rbrady has quit IRC | 04:02 | |
| *** ooolpbot has joined #tripleo | 04:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649615 | 04:10 |
| openstack | Launchpad bug 1649615 in tripleo "Undercloud running out of disk space in CI" [Critical,Fix committed] | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649742 | 04:10 |
| *** ooolpbot has quit IRC | 04:10 | |
| *** dsneddon_ has quit IRC | 04:10 | |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha" [Critical,Triaged] | 04:10 |
| *** dsneddon has joined #tripleo | 04:10 | |
| *** udesale has joined #tripleo | 04:14 | |
| *** sudipto has joined #tripleo | 04:18 | |
| *** sudipto_ has joined #tripleo | 04:18 | |
| *** dsneddon has quit IRC | 04:25 | |
| *** prateek has quit IRC | 04:31 | |
| *** chlong has joined #tripleo | 04:44 | |
| *** Vijayendra has joined #tripleo | 04:46 | |
| *** sudipto_ has quit IRC | 04:56 | |
| *** sudipto has quit IRC | 04:56 | |
| *** links has joined #tripleo | 04:57 | |
| *** dsneddon has joined #tripleo | 05:00 | |
| *** rlandy has quit IRC | 05:02 | |
| *** prateek has joined #tripleo | 05:09 | |
| *** prateek has quit IRC | 05:09 | |
| *** cwolferh has quit IRC | 05:10 | |
| *** ooolpbot has joined #tripleo | 05:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649615 | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649742 | 05:10 |
| *** ooolpbot has quit IRC | 05:10 | |
| openstack | Launchpad bug 1649615 in tripleo "Undercloud running out of disk space in CI" [Critical,Fix committed] | 05:10 |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha" [Critical,Triaged] | 05:10 |
| *** pradk has quit IRC | 05:12 | |
| *** dsneddon has quit IRC | 05:12 | |
| *** masco has joined #tripleo | 05:12 | |
| *** dsneddon has joined #tripleo | 05:13 | |
| *** prateek has joined #tripleo | 05:14 | |
| *** dsneddon has quit IRC | 05:19 | |
| *** dsneddon has joined #tripleo | 05:19 | |
| *** ctayal_ has quit IRC | 05:22 | |
| *** yamahata_ has joined #tripleo | 05:28 | |
| *** dsneddon_ has joined #tripleo | 05:38 | |
| *** sudipto has joined #tripleo | 05:39 | |
| *** sudipto_ has joined #tripleo | 05:39 | |
| *** dsneddon has quit IRC | 05:40 | |
| *** rajinir has quit IRC | 05:46 | |
| *** jaosorior has joined #tripleo | 05:50 | |
| *** cwolferh has joined #tripleo | 05:55 | |
| *** fragatina has joined #tripleo | 06:03 | |
| *** dsneddon has joined #tripleo | 06:08 | |
| *** fragatina has quit IRC | 06:08 | |
| *** dsneddon_ has quit IRC | 06:10 | |
| *** ooolpbot has joined #tripleo | 06:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649615 | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649742 | 06:10 |
| *** ooolpbot has quit IRC | 06:10 | |
| openstack | Launchpad bug 1649615 in tripleo "Undercloud running out of disk space in CI" [Critical,Fix committed] | 06:10 |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha" [Critical,Triaged] | 06:10 |
| *** pmannidi_ has quit IRC | 06:10 | |
| *** pmannidi has joined #tripleo | 06:11 | |
| *** shinobu_ has joined #tripleo | 06:11 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Introduce role-specific nova-server-metadata https://review.openstack.org/410545 | 06:11 |
| *** ctayal has joined #tripleo | 06:13 | |
| shinobu_ | can we create several pools with a set of specific ceph osd nodes? | 06:13 |
| *** abregman has joined #tripleo | 06:13 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Fix pip-and-virtualenv to work with python3 https://review.openstack.org/407723 | 06:14 |
| *** pkovar has joined #tripleo | 06:15 | |
| *** dsneddon_ has joined #tripleo | 06:17 | |
| *** dsneddo__ has joined #tripleo | 06:19 | |
| *** dsneddon has quit IRC | 06:21 | |
| *** dsneddon_ has quit IRC | 06:23 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Remove files from undercloud to save space https://review.openstack.org/410131 | 06:28 |
| *** ealcaniz has joined #tripleo | 06:31 | |
| *** lmiccini has joined #tripleo | 06:39 | |
| *** afazekas has quit IRC | 06:40 | |
| *** afazekas has joined #tripleo | 06:42 | |
| *** flepied has quit IRC | 06:44 | |
| *** chandankumar|off is now known as chandankumar | 06:48 | |
| *** ealcaniz has quit IRC | 06:51 | |
| *** dmacpher has quit IRC | 06:52 | |
| *** saneax-_-|AFK is now known as saneax | 06:55 | |
| *** abehl has joined #tripleo | 06:55 | |
| *** oshvartz has quit IRC | 06:57 | |
| *** hjensas has quit IRC | 06:57 | |
| *** yprokule has joined #tripleo | 07:01 | |
| shinobu_ | how can we create crushmap in detail using undercloud deployment? | 07:02 |
| *** lmiccini has quit IRC | 07:02 | |
| *** iranzo has joined #tripleo | 07:03 | |
| *** iranzo has joined #tripleo | 07:03 | |
| *** lmiccini has joined #tripleo | 07:03 | |
| jaosorior | shinobu_: it's quite early for a los of people at the moment, you might want to wait a couple of hours for more folks to log in | 07:03 |
| shinobu_ | jaosorior: thanks for your res. so for complex scenario, we have to deploy ceph cluster manually? | 07:04 |
| shinobu_ | jaosorior: i'm looking into ceph-cluster-config.yaml but no finding | 07:04 |
| jaosorior | shinobu_: I don't do any work with ceph, so I can't answer that question | 07:05 |
| jaosorior | shinobu_: that's why I recommended you wait | 07:05 |
| shinobu_ | jaosorior: ah, ack | 07:05 |
| shinobu_ | jaosorior: additionaly i'm pretty worried about overrinding configs by os-collect-config ;) | 07:06 |
| *** tzumainn has quit IRC | 07:06 | |
| jaosorior | shinobu_: overriding them how? | 07:08 |
| *** ooolpbot has joined #tripleo | 07:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649615 | 07:10 |
| openstack | Launchpad bug 1649615 in tripleo "Undercloud running out of disk space in CI" [Critical,Fix committed] | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649742 | 07:10 |
| *** ooolpbot has quit IRC | 07:10 | |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha" [Critical,Triaged] | 07:10 |
| openstackgerrit | Merged openstack/diskimage-builder: Catch errors in DIB_INIT_SYSTEM export https://review.openstack.org/400542 | 07:14 |
| *** mhenkel has joined #tripleo | 07:14 | |
| *** dsneddo__ has quit IRC | 07:15 | |
| openstackgerrit | Merged openstack/diskimage-builder: Fix pip-and-virtualenv to work with python3 https://review.openstack.org/407723 | 07:18 |
| *** Vijayendra has quit IRC | 07:18 | |
| *** florianf has joined #tripleo | 07:19 | |
| *** Vijayendra has joined #tripleo | 07:23 | |
| *** jprovazn has joined #tripleo | 07:24 | |
| *** ramishra has quit IRC | 07:25 | |
| *** ctayal has quit IRC | 07:25 | |
| *** ramishra has joined #tripleo | 07:26 | |
| *** tobias_fiberdata has quit IRC | 07:30 | |
| *** tobias_fiberdata has joined #tripleo | 07:35 | |
| *** yamahata_ has quit IRC | 07:37 | |
| *** mcornea has joined #tripleo | 07:38 | |
| *** dciabrin has quit IRC | 07:41 | |
| *** ctayal has joined #tripleo | 07:41 | |
| *** rasca has joined #tripleo | 07:45 | |
| *** oshvartz has joined #tripleo | 07:46 | |
| shinobu_ | jaosorior: os-collect-config calls *-refresh-* (calls *-refresh-*) if metadata is changed? | 07:54 |
| d0ugal | so, what is the status of CI? | 07:55 |
| *** mhenkel has quit IRC | 07:55 | |
| *** rcernin has joined #tripleo | 07:56 | |
| *** ealcaniz has joined #tripleo | 07:56 | |
| *** rcernin has quit IRC | 07:56 | |
| *** rcernin has joined #tripleo | 07:57 | |
| jaosorior | d0ugal: flaky | 07:58 |
| d0ugal | :) | 07:58 |
| jaosorior | d0ugal: more broken than not though | 07:58 |
| d0ugal | so, operating normally then? :P | 07:58 |
| jaosorior | shinobu_: uh.... it should, if heat metadata changes. | 07:59 |
| *** yamahata_ has joined #tripleo | 07:59 | |
| jaosorior | d0ugal: uhm.... lets say it's limping | 07:59 |
| *** pcaruana has joined #tripleo | 08:00 | |
| d0ugal | lol | 08:00 |
| *** aufi has joined #tripleo | 08:01 | |
| *** ealcaniz has quit IRC | 08:03 | |
| *** fzdarsky has joined #tripleo | 08:04 | |
| *** hjensas has joined #tripleo | 08:06 | |
| *** shardy has joined #tripleo | 08:07 | |
| *** Vijayendra has quit IRC | 08:08 | |
| *** ooolpbot has joined #tripleo | 08:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649615 | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649742 | 08:10 |
| *** ooolpbot has quit IRC | 08:10 | |
| openstack | Launchpad bug 1649615 in tripleo "Undercloud running out of disk space in CI" [Critical,Fix committed] | 08:10 |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha" [Critical,Triaged] | 08:10 |
| *** ccamacho has joined #tripleo | 08:10 | |
| *** bvandenh has quit IRC | 08:13 | |
| *** jlinkes has joined #tripleo | 08:14 | |
| *** jaosorior has quit IRC | 08:16 | |
| *** jaosorior has joined #tripleo | 08:16 | |
| *** flepied has joined #tripleo | 08:17 | |
| *** b00tcat has joined #tripleo | 08:24 | |
| *** tremble has joined #tripleo | 08:26 | |
| *** tremble has joined #tripleo | 08:26 | |
| *** jpich has joined #tripleo | 08:27 | |
| *** kodokuu has joined #tripleo | 08:27 | |
| *** mhenkel has joined #tripleo | 08:27 | |
| *** dciabrin has joined #tripleo | 08:28 | |
| jaosorior | bandini: hey dude | 08:30 |
| ccamacho | morning folks! | 08:30 |
| jaosorior | ccamacho: hey dude, how's it going? | 08:31 |
| yolanda | hi, good morning. How is status of tripleo-ci, still failing? i have an error on http://logs.openstack.org/56/408456/1/check/gate-tripleo-ci-centos-7-nonha-multinode/a2bf880/console.html | 08:32 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Enable SECURE_PROXY_SSL_HEADER option for horizon https://review.openstack.org/408992 | 08:32 |
| ccamacho | mwhahaha morning dude, one thing, if we want to start using os_transport_url in pupet-tripleo mmm we need to change the parameters in THT as they are all numbers.. im running locally some tests to change them https://review.openstack.org/#/c/410395/ | 08:32 |
| jaosorior | yolanda: most of the time it's failing :( | 08:33 |
| yolanda | is there some known problem? | 08:33 |
| ccamacho | jaosorior http://funnymemes.in/memes/Wednesday_Meme.jpg | 08:34 |
| jaosorior | hahahaha | 08:35 |
| jaosorior | yolanda: I'm honestly lost in why stuff is failing :/ | 08:35 |
| jaosorior | it just stacked up | 08:35 |
| ccamacho | xD | 08:35 |
| *** jpena|off is now known as jpena | 08:37 | |
| jaosorior | ccamacho: can you check this out https://review.openstack.org/#/c/409165/ ? It would unblock my work on the TLS-everywhere gate | 08:37 |
| ccamacho | jaosorior sure man | 08:37 |
| ccamacho | jaosorior I checked it 2/3 days ago I didnt +2 +A because CI wasnt working and it wont pass the gates. Im just about to +2+A when passing CI, mind to remind me to do it? | 08:39 |
| jaosorior | sure | 08:40 |
| jaosorior | ccamacho: I'm not too worried about that patch not passing CI... cause it's not used at all in CI. And I can't use it either because I need that to merge, then I need to merge a patch in RDO to add the script to the package spec | 08:41 |
| jaosorior | and then I'll finally be able to use it in CI | 08:41 |
| ccamacho | jaosorior, mmm just +2+A but it still needs to recheck to be merged right? | 08:43 |
| jaosorior | ccamacho: ovb is non-voting | 08:43 |
| ccamacho | gate-tripleo-ci-centos-7-ovb-ha is failing but does not say (non-voting) | 08:44 |
| jaosorior | ccamacho: that's cause it's third-party. It can't vote even if we wanted to | 08:44 |
| jaosorior | :/ | 08:44 |
| ccamacho | aaaaaaaaaaaa I was waiting for it.. ack | 08:45 |
| *** ctayal has quit IRC | 08:45 | |
| jaosorior | ccamacho: what's up dude, gonna get some vacations? | 08:45 |
| ccamacho | jaosorior I did last week, but working at the end in other stuff... My thesis is (Killing me slowly).. Almost done.. | 08:47 |
| jaosorior | damn dude | 08:47 |
| jaosorior | when are you done with that? | 08:47 |
| *** openstackgerrit has quit IRC | 08:48 | |
| *** jpena is now known as jpena|off | 08:49 | |
| ccamacho | jaosorior I have a hard deadline for next February.. Or Ill have to change the "Real decreto" to be able to read the Phd dissertation on 2019 :/ | 08:49 |
| ccamacho | so its now or never! | 08:49 |
| jaosorior | daaaaamn | 08:49 |
| jaosorior | dude, good luck! | 08:49 |
| ccamacho | jaosorior, but Im hoping to finish it this year... Im just fixing nits, and last week running the last simulations,... | 08:51 |
| * ccamacho grabbing some coffee | 08:51 | |
| matbu | ccamacho: what is the subject of your thesis ? | 08:53 |
| jaosorior | ccamacho: when you're done you could change your IRC handle to dr_camacho :P | 08:54 |
| yolanda | ccamacho, so for tests to pass, we wait until it's solved? | 08:55 |
| ccamacho | matbu its about software product lines, so we are formally defining what a SPL and defining equivalence relations between the semantics that we defined. https://github.com/ccamacho/papers/blob/master/1-s2.0-S0950584913001262-main.pdf and https://github.com/ccamacho/papers/blob/master/1-s2.0-S2352220815000917-main.pdf and what im finishing is an extesion to use probabilistic analysis to detect errors in the models. | 08:56 |
| *** amoralej|off is now known as amoralej | 08:56 | |
| ccamacho | jaosorior more like ccamacho_with_life | 08:57 |
| *** kodokuu has quit IRC | 08:58 | |
| *** yolanda has quit IRC | 09:01 | |
| *** dtantsur|afk is now known as dtantsur | 09:03 | |
| marios | ccamacho: hang in there man :) you will do it and it will be great. | 09:04 |
| *** yolanda has joined #tripleo | 09:05 | |
| dtantsur | morning folks! is it worth rechecking the multinode job right now? | 09:07 |
| ccamacho | marios, naaa this is like mm its an actual pathology... "Impostor syndrome" but thanks for the support :P | 09:07 |
| *** mhenkel has quit IRC | 09:07 | |
| *** leanderthal|afk has quit IRC | 09:07 | |
| ccamacho | dtantsur there are some jobs passing http://tripleo.org/cistatus.html | 09:08 |
| marios | ccamacho: erm not sure what you mean but srsly just keep going and at some point it will be done! sounds like you're prety close | 09:08 |
| marios | anyone else have problem with review.rdoproject.org right now? | 09:09 |
| dtantsur | cool, will try | 09:09 |
| *** ooolpbot has joined #tripleo | 09:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649615 | 09:10 |
| openstack | Launchpad bug 1649615 in tripleo "Undercloud running out of disk space in CI" [Critical,Fix committed] | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649742 | 09:10 |
| *** ooolpbot has quit IRC | 09:10 | |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha" [Critical,Triaged] | 09:10 |
| *** flepied has quit IRC | 09:15 | |
| *** ohamada has joined #tripleo | 09:15 | |
| bandini | hi jaosorior marios ccamacho | 09:15 |
| ccamacho | bandini o/ | 09:16 |
| marios | o/ bandini | 09:16 |
| marios | how are the alps this morning then | 09:16 |
| bandini | marios: cold and no snow ;) | 09:17 |
| jaosorior | bandini: lol, I think I figured it out :P | 09:17 |
| bandini | jaosorior: cool ;) | 09:18 |
| bandini | jaosorior: actually I have a question you might help me with ;) | 09:18 |
| *** ealcaniz has joined #tripleo | 09:19 | |
| *** bogdando has joined #tripleo | 09:19 | |
| jaosorior | sure! | 09:19 |
| *** ealcaniz is now known as ealcaniz|Orange- | 09:19 | |
| jaosorior | what's up? | 09:19 |
| bandini | jaosorior: https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/services/database/redis-base.yaml#L40. If I look at a deployed overcloud I correctly see the ip address of the host on the redis network. Yet that get_param call should return a network name, not an ip address? What am I missing? | 09:19 |
| bandini | dciabrin: ^ | 09:19 |
| Ng | shardy: huh, so I ran a deploy right before going to bed last night with the scheduler hints parameter and it doesn't seem to have worked. I previously messed up the hint and it completely exploded, so I think it was at least paying attention to the file | 09:19 |
| shardy | Ng: so you tagged the nodes in ironic, then added corresponding scheduler hints in an environment file, and what happened exactly? | 09:21 |
| jaosorior | bandini: we replace the network names with the relevant IP address | 09:21 |
| jaosorior | bandini: let me find where that happens | 09:21 |
| Ng | shardy: oh, actually that's a good point, I didn't tag them, I was trying to make the scheduler key off their name (since quickstart has already created them with appropriate/predictable names): http://paste.fedoraproject.org/506079/48170726/ | 09:21 |
| bandini | jaosorior: ah I see | 09:22 |
| shardy | Ng: Ah, yeah, by default quickstart doesn't tag them with their index, only e.g "control" | 09:22 |
| shardy | so you'll need to modify the profile to be explicit about e.g control-0 or whatever you put in the hints | 09:22 |
| Ng | ok, thanks | 09:22 |
| shardy | Ng: again, an option to do that automagically via quickstart would be good | 09:23 |
| shardy | http://docs.openstack.org/developer/tripleo-docs/advanced_deployment/profile_matching.html#manual-profile-tagging | 09:23 |
| Ng | shardy: agreed, I'll try and get that done after I've figured out this feature :) | 09:23 |
| *** bvandenh has joined #tripleo | 09:23 | |
| shardy | Ng: cool, perhaps condider raising a quickstart bug so we can keep track of the RFEs :) | 09:23 |
| bandini | shardy, Ng: I think quickstart should do predictable placing by default. It expands our testing and I feel it is less confusing | 09:23 |
| bandini | yep | 09:23 |
| Ng | bandini: +1 | 09:24 |
| *** gfidente has joined #tripleo | 09:24 | |
| shardy | bandini: Yeah, perhaps - personally I kinda like having a bucket of generic baremetal boxes, as I don't always test the same mix of controllers vs $other_roles | 09:24 |
| shardy | which quickstart currently doesn't support at all | 09:24 |
| shardy | but I think the predictable placement as a default probably does make sense | 09:24 |
| shardy | so everything lines up | 09:24 |
| bandini | yeah somehow I feel that the default makes a bit more sense to be predictable | 09:25 |
| bandini | Ng: care to open a bug about the default and we discuss it there? | 09:25 |
| *** openstackgerrit has joined #tripleo | 09:25 | |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: Implement major upgrade for Newton to Ocata https://review.openstack.org/404831 | 09:25 |
| shardy | yeah, we should have quickstart drop a ooq_heat_env.yaml on the undercloud along with the expected heat config | 09:26 |
| Ng | bandini: https://bugs.launchpad.net/tripleo-quickstart/+bug/1649813 | 09:26 |
| openstack | Launchpad bug 1649813 in tripleo-quickstart "Use predictable placement by default" [Undecided,New] | 09:26 |
| bandini | Ng: sweet, thanks! | 09:26 |
| jaosorior | bandini: https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/role.role.j2.yaml#L369 | 09:27 |
| bandini | jaosorior: thanks, looking | 09:27 |
| *** lucas-afk is now known as lucasagomes | 09:27 | |
| jaosorior | bandini: that will look for all the values that match a network name in the net_ip_map and will replace it for the relevant IP | 09:27 |
| gfidente | jaosorior funny same question going on again and again :0 | 09:28 |
| gfidente | so I didn't have great success with oooq yet | 09:28 |
| jaosorior | gfidente: do you think we should do it in services.yaml instead? https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/services/services.yaml would be easier to find | 09:29 |
| *** shinobu__ has joined #tripleo | 09:29 | |
| gfidente | jaosorior in services I don't think we have the local list of ips to match? | 09:30 |
| *** chem has quit IRC | 09:30 | |
| jaosorior | gfidente: dammit, you're right | 09:30 |
| gfidente | local as in the local node list of ips | 09:30 |
| jaosorior | yeah, that's true | 09:30 |
| *** chem has joined #tripleo | 09:30 | |
| jaosorior | forgot about that | 09:30 |
| *** derekh has joined #tripleo | 09:32 | |
| *** athomas has joined #tripleo | 09:35 | |
| *** dbecker has joined #tripleo | 09:36 | |
| *** yamahata_ has quit IRC | 09:37 | |
| *** nyechiel has joined #tripleo | 09:37 | |
| cmyster | question: just using the default network configurations of multiple nics + netowrk isolation templates renders the deployment successful but the controllers are inable to reach outside the environment (for instance if I wanted to do, say, yum update) is this something to be expected? | 09:39 |
| cmyster | unable* | 09:39 |
| *** milan has joined #tripleo | 09:39 | |
| openstackgerrit | Markos Chandras proposed openstack/diskimage-builder: elements: dib-python: Add python2 as the default version for openSUSE https://review.openstack.org/410639 | 09:40 |
| *** flepied has joined #tripleo | 09:40 | |
| *** karthiks has quit IRC | 09:48 | |
| jistr | folks, anyone seen virtual setup VMs become unreachable when using quickstart? https://paste.fedoraproject.org/506087/48170885/raw/ | 09:48 |
| jistr | strangely enough, it seems that the trigger could be running `virsh list` as root | 09:48 |
| *** imranh has quit IRC | 09:48 | |
| *** rain has joined #tripleo | 09:49 | |
| *** rain is now known as leanderthal | 09:49 | |
| *** mhenkel has joined #tripleo | 09:50 | |
| *** imranh has joined #tripleo | 09:50 | |
| bandini | jaosorior: https://paste.fedoraproject.org/506089/14817090/ ideas why this one would not get a proper replacement? | 09:51 |
| *** zoli|gone is now known as zoli | 09:52 | |
| gfidente | bandini because it is not a key in the yaml file | 09:53 |
| gfidente | it's an item of the list making up for the value of another key | 09:53 |
| gfidente | you'll have to use a sad trick I think, where you do | 09:53 |
| jaosorior | gfidente: it needs to be a value, no? | 09:54 |
| jaosorior | gfidente: and this is a value, however, it's part of a list, so it won't get replaced, since that replacement is not recursive | 09:54 |
| *** mhenkel has quit IRC | 09:54 | |
| gfidente | jaosorior yeah map_merge is checking for values | 09:54 |
| gfidente | but I don't think it goes into lists | 09:54 |
| *** mhenkel has joined #tripleo | 09:54 | |
| *** paramite has joined #tripleo | 09:55 | |
| bandini | oh so the map_merge is what is messing with me | 09:55 |
| gfidente | map_replace actually that is | 09:55 |
| gfidente | bandini I think you could try something sad like | 09:56 |
| gfidente | fake_key_name: {get_param: [ServiceNetMap, MysqlNetwork]} | 09:56 |
| * bandini looks closely | 09:56 | |
| *** imranh has quit IRC | 09:56 | |
| gfidente | and then hiera("fake_key_name") as value in the list | 09:56 |
| bandini | I see | 09:57 |
| gfidente | there was another place where we had to do this | 09:57 |
| bandini | HA! I was totally looking for other network replacements under map_merge ;) | 09:57 |
| gfidente | bandini don't we have already | 09:57 |
| gfidente | mysql_bind_address ? | 09:57 |
| * bandini looks | 09:57 | |
| bandini | I could not find one right away | 09:58 |
| gfidente | wait that is the bind address though, not useful | 09:58 |
| gfidente | you want the vip | 09:58 |
| gfidente | but anyway, it does the fake_key trick | 09:58 |
| jaosorior | bandini: what do you need the local IP that nova is binding on or the vip? | 09:59 |
| bandini | jaosorior: I need the local IP nova is binding to | 10:00 |
| jaosorior | bandini: you could use gfidente's trick | 10:00 |
| jaosorior | bandini: or, could you use an fqdn? | 10:00 |
| bandini | basically this is a bunch of work dciabrin is doing to fix vip failover | 10:00 |
| bandini | dciabrin: can we use an fqdn? | 10:00 |
| bandini | jaosorior: ack, thanks | 10:00 |
| jaosorior | bandini: if you can use an fqdn you can do it like this https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/services/nova-api.yaml#L123 | 10:01 |
| *** hewbrocca_afk is now known as hewbrocca | 10:02 | |
| bandini | jaosorior: yeah we saw that one. I am discussing with dciabrin the fqdn vs ip thing | 10:02 |
| d0ugal | Does anyone know what happened to the py34 checks? | 10:03 |
| d0ugal | They seem to have gone. i.e. in https://review.openstack.org/#/c/410262/ | 10:03 |
| jaosorior | d0ugal: well, they've been replaced by py35 checks it seems | 10:03 |
| d0ugal | jaosorior: but that has been there for a while and is still non-voting :/ | 10:04 |
| jaosorior | d0ugal: I could try to make them voting | 10:04 |
| jaosorior | d0ugal: or do you explicitly want py34 checks? | 10:04 |
| d0ugal | Maybe that is already happening? I'm not sure | 10:04 |
| d0ugal | jaosorior: I don't - just trying to understand | 10:04 |
| jaosorior | oh, ok | 10:04 |
| d0ugal | They also got removed from Mistral | 10:04 |
| jaosorior | ah | 10:04 |
| jaosorior | d0ugal: might want to ask in #openstack-infra | 10:05 |
| d0ugal | Yeah, I guess so | 10:05 |
| d0ugal | thanks | 10:05 |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: Implement major upgrade for Newton to Ocata https://review.openstack.org/404831 | 10:05 |
| *** jpena|off is now known as jpena | 10:05 | |
| *** kodokuu has joined #tripleo | 10:05 | |
| jprovazn | gfidente, hi, I think ceph mds perms will need more tuning - now it fails with http://paste.openstack.org/show/592318/ and service is in failed state, if I set "allow *" to mon/mds/osd then it starts | 10:06 |
| jprovazn | gfidente, some "initilization" permission is probably missing - if I then reset perms back to original it started too | 10:07 |
| *** dtantsur is now known as dtantsur|brb | 10:09 | |
| *** ooolpbot has joined #tripleo | 10:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649615 | 10:10 |
| openstack | Launchpad bug 1649615 in tripleo "Undercloud running out of disk space in CI" [Critical,Fix committed] | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649742 | 10:10 |
| *** ooolpbot has quit IRC | 10:10 | |
| shinobu__ | jprovazn: how about crushmap? can we customize it using template? e.g., make pools with specific osd nodes, or specific osds in osd nodes | 10:10 |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha" [Critical,Triaged] | 10:10 |
| gfidente | shinobu_ no we can't | 10:11 |
| *** Vijayendra has joined #tripleo | 10:12 | |
| gfidente | jprovazn apparently profile mds is not enough then | 10:13 |
| *** imranh has joined #tripleo | 10:14 | |
| gfidente | jprovazn which means that the docs are probably wrong http://docs.ceph.com/docs/master/rados/configuration/auth-config-ref/#daemon-keyrings | 10:14 |
| *** nyechiel_ has joined #tripleo | 10:14 | |
| shinobu__ | gfidente: so we must create crushmap manually (meaning that build ceph cluster manually) after deployment of overcloud if we want to create very customize pools? | 10:15 |
| jprovazn | gfidente, we set this: | 10:15 |
| jprovazn | caps: [mds] allow * | 10:15 |
| jprovazn | caps: [mon] allow profile mds | 10:15 |
| jprovazn | caps: [osd] allow rwx pool=cephfs, allow rwx pool=cephfs_metadata | 10:15 |
| jprovazn | gfidente, it seems also osd differs, should we try: osd 'allow rwx' | 10:15 |
| gfidente | shinobu_ yeah | 10:16 |
| gfidente | jprovazn ack in patchset was rwx | 10:16 |
| gfidente | I'd probably go just back to that | 10:16 |
| gfidente | *patchset 8 | 10:16 |
| *** nyechiel has quit IRC | 10:18 | |
| jprovazn | gfidente, ack, re-deploying with rwx for osd | 10:19 |
| gfidente | for mon | 10:19 |
| gfidente | I think the problem is for mon caps right? | 10:20 |
| panda|Zz | derekh: do you know where is the command "ts" that adds timestamps , and why the command here https://github.com/openstack-infra/tripleo-ci/blob/master/toci_instack_ovb.sh#L50 ends with a empty pipe ? | 10:21 |
| *** panda|Zz is now known as panda | 10:21 | |
| jprovazn | gfidente, well, I'm not sure which of them is incorrect | 10:22 |
| jprovazn | gfidente, mon is same as in the doc you linked, osd differs | 10:22 |
| matbu | panda: i think it comes with moreutils pkg (idk if it's the question :)) | 10:23 |
| gfidente | jprovazn I think we're doing rwx, just only for those two pools | 10:24 |
| jprovazn | gfidente, yes, it seems it's the only difference in compare to the doc | 10:25 |
| derekh | panda: there is a ts command in the moreutils package, is that the one we are using? if not I'd look for a function in devstack-gate somewhere | 10:25 |
| jprovazn | gfidente, I'll ping when OC is re-deployed | 10:26 |
| derekh | panda: you mean this pipe "ts '%Y-%m-%d %H:%M:%S.000 |"? | 10:27 |
| panda | derekh: yes | 10:28 |
| derekh | panda: thats not a pipe, its part of the string ts is outputing | 10:28 |
| derekh | panda: run this and see what I mean | 10:28 |
| derekh | find /etc/sysconfig/ | ts '%Y-%m-%d %H:%M:%S.000 |||' | 10:28 |
| *** limao has quit IRC | 10:28 | |
| *** limao has joined #tripleo | 10:29 | |
| * panda facepalms | 10:29 | |
| panda | derekh: ok I see it now, thanks | 10:29 |
| derekh | panda: no prob | 10:29 |
| *** limao has quit IRC | 10:30 | |
| *** leanderthal has quit IRC | 10:30 | |
| *** limao has joined #tripleo | 10:30 | |
| *** limao has quit IRC | 10:35 | |
| gfidente | do you guys know if we can change the virt_type in quickstart | 10:37 |
| gfidente | from qemu to kvm? | 10:37 |
| *** links has quit IRC | 10:38 | |
| *** links has joined #tripleo | 10:38 | |
| shinobu__ | gfidente: ty ;) | 10:38 |
| jaosorior | gfidente: yeah | 10:40 |
| jaosorior | gfidente: you can change it by setting the libvirt_args: variable | 10:40 |
| *** saibarspeis has joined #tripleo | 10:40 | |
| jaosorior | gfidente: the default is -> libvirt_args: "--libvirt-type qemu" | 10:42 |
| gfidente | shinobu_ hehe sorry but I think we never got a clear picture of how the user should provide the input for the crushmap customization | 10:43 |
| gfidente | shinobu_ meantime you can disable the autocreation of the crushmap passing ceph::profile::params::osd_crush_update_on_start: false | 10:44 |
| gfidente | via ExtraConfig | 10:44 |
| gfidente | but if you have ideas on what the user input should be, I think it'd be nice to discuss that on openstack-dev | 10:44 |
| gfidente | and should you try that, I'd be happy if you could add me on CC :) | 10:44 |
| gfidente | jaosorior++ thanks | 10:45 |
| gfidente | jaosorior why does it default to qemu btw? | 10:45 |
| gfidente | aren't the baremetalbrbm provisioned on kvm capabled hosts? | 10:46 |
| jaosorior | adarazs: ^^ | 10:46 |
| shinobu__ | gfidente: ack. thaks for your input! | 10:47 |
| adarazs | jaosorior, gfidente: I thought we don't default to qemu, let me check. | 10:47 |
| *** _milan_ has joined #tripleo | 10:47 | |
| *** milan has quit IRC | 10:48 | |
| jaosorior | adarazs: https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/overcloud-deploy/defaults/main.yml#L27 | 10:48 |
| panda | adarazs: jaosorior: roles/overcloud-deploy/tasks/pre-deploy.yml | 10:49 |
| panda | in -extras | 10:49 |
| panda | we use kvm on RedHat distros | 10:49 |
| adarazs | hrm, interesting. we set up nested virt correctly though. | 10:49 |
| adarazs | panda: and do you know why we don't do that on centos? | 10:50 |
| gfidente | right I was going to say that I run on CentOS | 10:50 |
| adarazs | though it's not super important, we're not really running many things in the overcloud :) | 10:50 |
| gfidente | but sounds like we could very well default to kvm | 10:50 |
| openstackgerrit | Carlos Camacho proposed openstack-infra/tripleo-ci: Test do not merge https://review.openstack.org/410685 | 10:50 |
| gfidente | panda adarazs what is a scenario where the baremetalbrbm guests should use qemu? | 10:50 |
| adarazs | gfidente: I assume we override that for baremetal deployments, but I'm not sure. | 10:52 |
| panda | it was changed for this bug https://bugzilla.redhat.com/show_bug.cgi?id=1382389 | 10:53 |
| openstack | bugzilla.redhat.com bug 1382389 in qemu-kvm-rhev "domain not booting fully when launched via HA configuration w/ qemu" [Medium,New] - Assigned to bdas | 10:53 |
| adarazs | gfidente: baremetal is not the focus of stock default quickstart, so the defaults are what good for our testing on virthosts. | 10:53 |
| panda | why it is qemu by default, I do not know. | 10:53 |
| gfidente | adarazs yeah but in virthosts | 10:54 |
| gfidente | the baremetalbrbm are running on kvm capable hosts | 10:54 |
| *** tosky has joined #tripleo | 10:55 | |
| panda | gfidente: libvirt-type is only used during overcloud deploy | 10:55 |
| gfidente | panda ok so the problem I have is that the baremetalbrbm guests | 10:56 |
| gfidente | are using virt_type qemu | 10:56 |
| shardy | yeah so kvm will only work if you've configured nested virt. Historically that hasn't really worked on CentOS but maybe the bugs are fixed now | 10:56 |
| gfidente | shardy why would we need nested virt? | 10:56 |
| bandini | to test spawning a vm? | 10:57 |
| gfidente | I think it's the overcloud guests which eventually would need that | 10:57 |
| panda | gfidente: baremetal vms should use kvm , where do you see qemu used ? | 10:57 |
| gfidente | right but the baremetalbrbm guests are the overcloud hosts | 10:57 |
| shardy | gfidente: I think we're talking about two different things - I was referring to the overcloud virt config, and I think you're referring to the quickstart virsh generated nodes | 10:57 |
| gfidente | shardy yes | 10:57 |
| panda | gfidente: the variable libvirt_domain_type sets the domain type of the baremetal vms, and its default is kvm | 10:58 |
| openstackgerrit | Carlos Camacho proposed openstack/puppet-tripleo: Use transport_url for heat https://review.openstack.org/410333 | 10:59 |
| gfidente | panda wait I see an email sent from dmsimard | 10:59 |
| gfidente | about a problem in qemu-kvm-em | 10:59 |
| gfidente | in version 2.6.0 | 10:59 |
| gfidente | the email was to rdo-list | 11:00 |
| panda | oh, yeah the CentOS 7.3 "big" issue | 11:00 |
| *** udesale has quit IRC | 11:05 | |
| openstackgerrit | Jan Provaznik proposed openstack/puppet-tripleo: Set ceph key when using manila ceph backend https://review.openstack.org/405999 | 11:07 |
| gfidente | panda so I think I don't get why it should be failing like that in oooq if the guests are configured to use hvm | 11:08 |
| gfidente | :( | 11:08 |
| *** Vijayendra has quit IRC | 11:08 | |
| panda | gfidente: what is failing ? | 11:08 |
| gfidente | spawning the undercloud vm | 11:09 |
| *** hjensas has quit IRC | 11:09 | |
| panda | gfidente: do you have error/logs ? | 11:09 |
| openstackgerrit | Jan Provaznik proposed openstack/tripleo-heat-templates: Set manila cephfs backend if ceph is deployed https://review.openstack.org/405627 | 11:09 |
| gfidente | qemu-kvm: CPU feature arat not found | 11:09 |
| *** sudipto_ has quit IRC | 11:09 | |
| *** sudipto has quit IRC | 11:09 | |
| *** ooolpbot has joined #tripleo | 11:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649615 | 11:10 |
| openstack | Launchpad bug 1649615 in tripleo "Undercloud running out of disk space in CI" [Critical,Fix committed] | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649742 | 11:10 |
| *** ooolpbot has quit IRC | 11:10 | |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha" [Critical,Triaged] | 11:10 |
| panda | gfidente: can you check the dumpxml for the undercloud vm ? | 11:11 |
| gfidente | panda yeah cpu mode is host-model | 11:11 |
| gfidente | type is hvm | 11:11 |
| panda | gfidente: than the bug is worse than we imagined | 11:12 |
| gfidente | panda yeah I see +arat in the cmdline | 11:13 |
| panda | gfidente: check domain type, not only type arch | 11:13 |
| gfidente | panda yeah that is kvm | 11:14 |
| panda | dmsimard: looks like the combination kvm + host-model is also affected by the libvirt bug | 11:14 |
| *** akrivoka has joined #tripleo | 11:15 | |
| openstackgerrit | Jan Provaznik proposed openstack/tripleo-common: Generate a CephX for the new Ceph MDS service https://review.openstack.org/402390 | 11:17 |
| openstackgerrit | Jan Provaznik proposed openstack/tripleo-common: Generate a CephX for manila ceph client service https://review.openstack.org/410695 | 11:17 |
| gfidente | panda I've downgraded qemu for now and retrying | 11:17 |
| panda | gfidente: ok, patching quickstart to use cpu mode='none' for undercloud | 11:18 |
| gfidente | well then I think we'll hit same issue when undercloud nova tries to boot overcloud nodes? | 11:18 |
| openstackgerrit | Julie Pichon proposed openstack/python-tripleoclient: Return proper exit code on plan deletion failure https://review.openstack.org/410697 | 11:19 |
| *** jaosorior has quit IRC | 11:19 | |
| *** prateek_ has joined #tripleo | 11:21 | |
| *** prateek has quit IRC | 11:21 | |
| panda | gfidente: we'll have to override it in undercloud conf also then (don't know if it's possible) | 11:22 |
| *** zoli is now known as zoli|brb | 11:27 | |
| *** dtantsur|brb is now known as dtantsur | 11:28 | |
| gfidente | panda right, so downgrading qemu didn't help, I think the problem is in the use of +arat in libvirt and not in qemu? | 11:29 |
| *** mhenkel has quit IRC | 11:30 | |
| *** jaosorior has joined #tripleo | 11:30 | |
| panda | gfidente: until now , we thought it was a problem with the qemu version shipped with the updated package, but even rdo propmotion failed with the same error | 11:31 |
| gfidente | sghall we move to rdo? | 11:32 |
| *** hjensas has joined #tripleo | 11:33 | |
| *** Vijayendra has joined #tripleo | 11:43 | |
| *** zoli|brb is now known as zoli | 11:43 | |
| *** rhallisey has joined #tripleo | 11:44 | |
| panda | what's the hiera override for the libvirt type in nova conf ? | 11:45 |
| shardy | panda: https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/services/nova-libvirt.yaml#L51 | 11:48 |
| gfidente | panda try compute::libvirt::libvirt_virt_type | 11:48 |
| *** pkovar has quit IRC | 11:48 | |
| shardy | panda: you shouldn't need to explicitly override it tho, use --libvirt-type on the CLI or NovaComputeLibvirtType in an environment file | 11:48 |
| panda | sorry, I meant for the cpu mode | 11:49 |
| *** fzdarsky is now known as fzdarsky|lunch | 11:51 | |
| shardy | https://github.com/openstack/puppet-nova/blob/master/manifests/compute/libvirt.pp#L24 | 11:51 |
| shardy | panda: ^^ you can see the various options there | 11:51 |
| panda | shardy: thanks. | 11:52 |
| openstackgerrit | yolanda.robla proposed openstack/tripleo-quickstart: Allow to consume external overcloud images https://review.openstack.org/404734 | 11:55 |
| openstackgerrit | Attila Darazs proposed openstack/tripleo-quickstart: Fetch images in a standalone role https://review.openstack.org/408760 | 11:55 |
| *** kodokuu has quit IRC | 11:55 | |
| *** adarazs is now known as adarazs_lunch | 11:56 | |
| *** fpan has quit IRC | 11:56 | |
| openstackgerrit | Gabriele Cerami proposed openstack/tripleo-quickstart: Workarounds to bug https://bugzilla.redhat.com/show_bug.cgi?id=1371617 https://review.openstack.org/410718 | 11:56 |
| openstack | bugzilla.redhat.com bug 1371617 in libvirt "Libvirt passes unsupported "arat" flag to QEMU when using host-model" [High,Assigned] - Assigned to jdenemar | 11:56 |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-common: Implement stack update as mistral actions https://review.openstack.org/379516 | 11:59 |
| *** fpan has joined #tripleo | 12:03 | |
| *** sudipto has joined #tripleo | 12:06 | |
| *** sudipto_ has joined #tripleo | 12:06 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Pass hieradata for internal TLS for RabbitMQ https://review.openstack.org/409143 | 12:07 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Set rabbitmq's port and IP via the config file and not the env file https://review.openstack.org/410722 | 12:07 |
| *** jkilpatr has joined #tripleo | 12:08 | |
| EmilienM | hello | 12:09 |
| *** ooolpbot has joined #tripleo | 12:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649615 | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649742 | 12:10 |
| *** ooolpbot has quit IRC | 12:10 | |
| openstack | Launchpad bug 1649615 in tripleo "Undercloud running out of disk space in CI" [Critical,Fix committed] | 12:10 |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha" [Critical,Triaged] | 12:10 |
| jaosorior | EmilienM: hey dude, how's it going? | 12:10 |
| *** ealcaniz|Orange- has quit IRC | 12:10 | |
| EmilienM | jaosorior: CI is still broken :( | 12:10 |
| jaosorior | EmilienM: it is dude :/ | 12:10 |
| EmilienM | is anyone looking at it? | 12:10 |
| sshnaidm | derekh, hi | 12:11 |
| *** athomas has quit IRC | 12:13 | |
| panda | EmilienM: I'm trying but doesn't seem to be easy. | 12:14 |
| *** prateek_ has quit IRC | 12:14 | |
| *** jkilpatr has quit IRC | 12:15 | |
| *** lucasagomes is now known as lucas-hungry | 12:17 | |
| *** dtantsur is now known as dtantsur|brb | 12:21 | |
| derekh | sshnaidm: whats up? | 12:21 |
| *** pkovar has joined #tripleo | 12:22 | |
| sshnaidm | derekh, do you have maybe any ideas why postci function can hang forever? | 12:23 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: DONT REVIEW: test https://review.openstack.org/410470 | 12:23 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Enable TLS in the internal network for RabbitMQ https://review.openstack.org/327482 | 12:24 |
| derekh | sshnaidm: nope, but if you recheck this, you can ssh to the node when the ci job is nearly finished to see what its doing https://review.openstack.org/#/c/363630/ | 12:24 |
| panda | sshnaidm: derekh EmilienM I don't think it's postci, it returns correclty 0, I'm afraid it's something related to testenv-client | 12:24 |
| matbu | shardy: thx for the review, in the latest patch, i've added the function for setting up the delorean repo on the OC nodes (move out from bootstrap_nodes function to a dedicated one) | 12:24 |
| shardy | matbu: np - I'm still testing locally, not quite got the full multinode test working yet, but I'm doing end-to-end upgrade tests based on your patch | 12:26 |
| sshnaidm | derekh, panda when I use devenv it always happens, right now I ran there toci_gate_test.sh and it's hung on postci. Although it didn't help much. | 12:26 |
| *** florianf has quit IRC | 12:27 | |
| *** athomas has joined #tripleo | 12:27 | |
| sshnaidm | derekh, panda you can try to look, it's in screen session: ssh -ttt centos@66.187.229.211 -A "ssh -ttt jenkins@192.168.100.154 'cd /opt/stack/new/tripleo-ci; bash'" | 12:27 |
| *** shardy is now known as shardy_lunch | 12:27 | |
| *** pgadiya has quit IRC | 12:28 | |
| derekh | sshnaidm: looking | 12:29 |
| *** jkilpatr has joined #tripleo | 12:29 | |
| *** florianf has joined #tripleo | 12:31 | |
| sshnaidm | derekh, panda maybe it's because this trash: https://github.com/openstack-infra/tripleo-ci/blob/master/scripts/common_functions.sh#L232-L234 | 12:31 |
| sshnaidm | but it returns 0 as panda said.. | 12:32 |
| panda | sshnaidm: exactly, postci is completed when the deploy hangs | 12:32 |
| sshnaidm | panda, derekh is there reason we use "return" and not "exit" ? | 12:33 |
| sshnaidm | it looks like it doesn't leave shell.. | 12:34 |
| panda | sshnaidm: because it's a function | 12:35 |
| derekh | sshnaidm: panda can I kill dstat and the python http server to see if one of them is holding the process open | 12:36 |
| sshnaidm | derekh, sure | 12:36 |
| panda | derekh: go | 12:37 |
| sshnaidm | derekh, dstat? | 12:37 |
| derekh | sshnaidm: yup, dstat | 12:37 |
| derekh | jenkins 1501 1 4 08:01 pts/2 00:10:59 /usr/bin/python /usr/bin/dstat -tcmndrylpg --top-cpu-adv --top-io-adv --nocolor | 12:37 |
| derekh | sshnaidm: thats all I killed | 12:37 |
| sshnaidm | derekh, yeah, I see.. why the hell it's hung | 12:38 |
| panda | yeah, dstat log is one that is still written when the job times out | 12:38 |
| panda | who's supposed to kill it ? | 12:38 |
| derekh | sshnaidm: no idea, and why has it only started now and why on only the ha job, so many questions | 12:39 |
| yolanda | agh, so my tests passed check correctly, then failed on gate. Cannot make it land :( | 12:39 |
| derekh | panda: I don't think we ever had to kill it | 12:39 |
| sshnaidm | derekh, panda let's kill it in postci? | 12:39 |
| *** Lord_Phoenix has joined #tripleo | 12:40 | |
| *** pgadiya has joined #tripleo | 12:40 | |
| *** dsariel has quit IRC | 12:40 | |
| derekh | sshnaidm: or maybe running it with disown might work, not sure | 12:40 |
| sshnaidm | yeah, I'll try now without disown | 12:42 |
| derekh | hmm, we already have disown in there | 12:43 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: DONT REVIEW: test https://review.openstack.org/410470 | 12:43 |
| sshnaidm | derekh, where? | 12:43 |
| derekh | sshnaidm: I think the disown command after we start dstat is supposed to prevent this hanging from happening | 12:44 |
| sshnaidm | derekh, uh, I thought it may be a cause for problem | 12:45 |
| *** rbrady has joined #tripleo | 12:45 | |
| *** rbrady has quit IRC | 12:45 | |
| *** rbrady has joined #tripleo | 12:45 | |
| derekh | sshnaidm: I'm not 100% sure of anything except that its probably relevant | 12:48 |
| *** sudipto_ has quit IRC | 12:51 | |
| *** sudipto has quit IRC | 12:51 | |
| *** trown|outtypewww is now known as trown | 12:51 | |
| *** amoralej is now known as amoralej|lunch | 12:55 | |
| *** masco has quit IRC | 12:57 | |
| *** dtantsur|brb is now known as dtantsur | 12:59 | |
| *** pgadiya has quit IRC | 13:02 | |
| *** pradk has joined #tripleo | 13:02 | |
| *** adarazs_lunch is now known as adarazs | 13:02 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: Kill dstat to prevent hanging of job https://review.openstack.org/410738 | 13:04 |
| sshnaidm | derekh, panda just a straight forward workaround to try ^^ | 13:04 |
| panda | sshnaidm: why did you remove the destroy_vms line ? | 13:05 |
| sshnaidm | panda, it's obsolete | 13:05 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart-extras: add the containers prep role to the quickstart-extras https://review.openstack.org/400983 | 13:05 |
| panda | sshnaidm: by what ? | 13:06 |
| sshnaidm | panda, it was required before we used ovb | 13:07 |
| *** lucas-hungry is now known as lucasagomes | 13:07 | |
| sshnaidm | panda, there is no such function anymore "destroy_vms" | 13:08 |
| panda | sshnaidm: I see | 13:08 |
| *** jayg|g0n3 is now known as jayg | 13:09 | |
| *** fzdarsky|lunch is now known as fzdarsky | 13:09 | |
| *** ooolpbot has joined #tripleo | 13:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649615 | 13:10 |
| openstack | Launchpad bug 1649615 in tripleo "Undercloud running out of disk space in CI" [Critical,Fix committed] | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649742 | 13:10 |
| *** ooolpbot has quit IRC | 13:10 | |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha" [Critical,Triaged] | 13:10 |
| *** bfournie has quit IRC | 13:12 | |
| *** leanderthal has joined #tripleo | 13:12 | |
| *** kodokuu has joined #tripleo | 13:13 | |
| *** jeckersb is now known as jeckersb_gone | 13:14 | |
| sshnaidm | derekh, EmilienM you can vote for another patch that frees some space: https://review.openstack.org/#/c/410323/ | 13:15 |
| *** shardy_lunch is now known as shardy | 13:16 | |
| *** ealcaniz has joined #tripleo | 13:16 | |
| EmilienM | sshnaidm: approved | 13:16 |
| sshnaidm | EmilienM, I think we can remove alert from https://bugs.launchpad.net/tripleo/+bug/1649615, leaving it only in https://bugs.launchpad.net/tripleo/+bug/1649742 | 13:16 |
| openstack | Launchpad bug 1649615 in tripleo "Undercloud running out of disk space in CI" [Critical,Fix committed] | 13:16 |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha" [Critical,Triaged] | 13:16 |
| *** abregman is now known as abregman|mtg | 13:18 | |
| EmilienM | sshnaidm: please do | 13:18 |
| sshnaidm | done | 13:18 |
| *** dbecker has quit IRC | 13:19 | |
| *** egafford has joined #tripleo | 13:23 | |
| openstackgerrit | Chris Jones proposed openstack/tripleo-common: [WIP] Add tooling for obtaining Fencing data automatically. https://review.openstack.org/408601 | 13:25 |
| *** dmacpher has joined #tripleo | 13:26 | |
| sshnaidm | Disowned process hangs terminal during logout/exit https://lists.gnu.org/archive/html/bug-bash/2007-12/msg00006.html | 13:26 |
| *** fultonj has joined #tripleo | 13:28 | |
| EmilienM | sshnaidm: https://review.openstack.org/#/c/410738/1/scripts/common_functions.sh | 13:28 |
| EmilienM | why do you stop calling destroy_vms? | 13:28 |
| sshnaidm | EmilienM, it doesn't exist anymore | 13:28 |
| EmilienM | sshnaidm: +2 | 13:29 |
| sshnaidm | EmilienM, just cleaning junk, it was worth to mention it in commit msg though.. | 13:29 |
| *** arxcruz is now known as arxcruz|brb | 13:30 | |
| *** chlong has quit IRC | 13:32 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart: make quickstart-extras-requirements.txt a default requirements file https://review.openstack.org/410757 | 13:32 |
| shardy | matbu, marios: Hey FYI I found an issue with the upgrade workflow - we have to run UpgradeInitCommand in the role templates, or we can't install the new hiera hook before it runs (e.g in ControllerDeployment) | 13:32 |
| shardy | that's why the CI job is timing out I think (my local tests get stuck) | 13:33 |
| shardy | working on a patch now | 13:33 |
| *** sshnaidm is now known as sshnaidm|afk | 13:33 | |
| *** limao has joined #tripleo | 13:34 | |
| *** ohamada_ has joined #tripleo | 13:36 | |
| *** ohamada has quit IRC | 13:36 | |
| *** dsariel has joined #tripleo | 13:36 | |
| *** ansmith has joined #tripleo | 13:40 | |
| marios | shardy: o/ cool - might be why we are seeing local testing timing out (though today it failed for a different reason for me.. .forgot i reset the env, so the nodes didn't have the python-heat-agent-ansible ... :D so i eventually noticed the "Skipping group ansible with no hook script None" | 13:40 |
| marios | shardy: but yeah last time when i had done the setup properly it was hanging for me and eventually timed out | 13:41 |
| shardy | marios: Yeah I didn't see it before because I was testing master->master upgrades, now I'm using newton images it hangs | 13:41 |
| shardy | fix in-progress | 13:41 |
| shardy | basically we need to install the repos, and the hiera/ansible heat config hooks earlier | 13:42 |
| shardy | the order we have now would work OK if it was only the ansible hook | 13:42 |
| matbu | yep same for me on my test | 13:43 |
| shardy | Actually I think I did test with newton images before too, but it must have been before the patch switching to the hiera hook landed in tht | 13:43 |
| matbu | but it's bloked on networkDeployment | 13:43 |
| matbu | s/bloked/blocked/ | 13:43 |
| EmilienM | jaosorior: please hold on on +A patches | 13:43 |
| shardy | matbu: I think you'll find NetworkDeployment is the last resource which worked, it's actually blocked IN_PROGRESS on ControllerDeployment | 13:43 |
| EmilienM | jaosorior: our CI is still broken | 13:43 |
| EmilienM | jaosorior: and we don't want to merge patches that don't pass all ovb jobs | 13:44 |
| matbu | shardy: i did the test before, it was working | 13:44 |
| *** abregman|mtg is now known as abregman | 13:44 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart: make quickstart-extras-requirements.txt a default requirements file https://review.openstack.org/410757 | 13:44 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart: move the undercloud deploy role to quickstart-extras for composability https://review.openstack.org/410427 | 13:44 |
| matbu | (hmm at least iirc...) | 13:44 |
| EmilienM | all reviewers: CI is still broken now, please do not merge patches until you see all ovb jobs passing http://tripleo.org/cistatus.html | 13:44 |
| jpich | ack | 13:45 |
| *** rlandy has joined #tripleo | 13:45 | |
| shardy | matbu: unless the hooks are already installed in the image (not the case for newton), this should have been broken since https://review.openstack.org/#/c/384757/ landed | 13:46 |
| ccamacho | "matbu: shardy: i did the test before, it was working" Also for me... | 13:47 |
| *** bfournie has joined #tripleo | 13:47 | |
| shardy | anyway, good we have consistent results, working on a fix now | 13:47 |
| *** jpena is now known as jpena|lunch | 13:47 | |
| matbu | shardy: a fix in tht ? | 13:48 |
| shardy | matbu: yup | 13:48 |
| shardy | ccamacho: I guess either that was a couple of weeks ago (before the referenced patch landed), or you weren't using newton images | 13:49 |
| *** Goneri has joined #tripleo | 13:50 | |
| ccamacho | shardy It worked after merging the ansible upgrades patch, deploying first a newton overcloud and then running the major upgrade. The upgrade hangs and the only error I was able to see was related to rabbitmq | 13:52 |
| ccamacho | some rabbitmq parameters are deprecated and we are not configuring them correctly in master (transport_url) | 13:52 |
| *** pkovar has quit IRC | 13:52 | |
| ccamacho | Im tesing some submission to fix that | 13:52 |
| ccamacho | /tesing/testing/ | 13:52 |
| ccamacho | I didnt followed https://review.openstack.org/#/c/384757/ so im not sure if it started to fail just after it landed | 13:53 |
| openstackgerrit | Luke Hinds proposed openstack/puppet-tripleo: Adds ability to populate SSH Banner text https://review.openstack.org/408637 | 13:53 |
| shardy | ccamacho: Yeah https://review.openstack.org/#/c/393448/ landed on the same day as https://review.openstack.org/#/c/384757/ so nobody spotted it testing locally | 13:54 |
| *** Goneri has quit IRC | 13:55 | |
| *** Goneri has joined #tripleo | 13:55 | |
| *** limao_ has joined #tripleo | 13:58 | |
| *** limao has quit IRC | 13:58 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart: make quickstart-extras-requirements.txt a default requirements file https://review.openstack.org/410757 | 13:59 |
| *** hrybacki is now known as hrybacki|mtg | 14:01 | |
| *** pkovar has joined #tripleo | 14:02 | |
| *** abregman is now known as abregman|mtg | 14:02 | |
| *** fragatina has joined #tripleo | 14:04 | |
| *** ayoung has quit IRC | 14:04 | |
| *** morazi has joined #tripleo | 14:05 | |
| *** amoralej|lunch is now known as amoralej | 14:07 | |
| openstackgerrit | Carlos Camacho proposed openstack/puppet-tripleo: Use transport_url for heat https://review.openstack.org/410333 | 14:07 |
| marios | shardy: /win 6 | 14:08 |
| marios | oops sorry shardy | 14:08 |
| * shardy suspects his irssi tabs aren't aligned with marios ;) | 14:08 | |
| *** fragatina has quit IRC | 14:09 | |
| *** liverpooler has joined #tripleo | 14:09 | |
| *** ooolpbot has joined #tripleo | 14:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649742 | 14:10 |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha" [Critical,Triaged] | 14:10 |
| *** ooolpbot has quit IRC | 14:10 | |
| *** liverpooler has quit IRC | 14:10 | |
| *** liverpooler has joined #tripleo | 14:10 | |
| ccamacho | mwhahaha quick question, when using os_transport_url() in the puppet-tripleo patches using it, we are getting this error Error: Evaluation Error: Error while evaluating a Function Call, undefined method `map' for \"overcloud-controller-0.internalapi.localdomain\":String at /etc/puppet/modules/tripleo/manifests/profile/base/heat.pp:82:28 on node overcloud-controller-0.localdomain\u001b[0m\n | 14:10 |
| ccamacho | does it sound like a known issue? | 14:10 |
| mwhahaha | That sounds like an undefined var | 14:12 |
| *** morazi has quit IRC | 14:13 | |
| mwhahaha | ccamacho: whats on line 82 of the heat class? | 14:13 |
| ccamacho | mwhahaha this call http://paste.openstack.org/show/592350/ | 14:14 |
| *** leanderthal has quit IRC | 14:14 | |
| ccamacho | I even hardwired the values... but still getting the same error | 14:15 |
| mwhahaha | Is hosts not an array? | 14:15 |
| ccamacho | yeahp its an array | 14:15 |
| mwhahaha | Yea not sure because in the function we only map the hosts and the query param | 14:16 |
| mwhahaha | It says it's a string | 14:17 |
| mwhahaha | (your error) | 14:17 |
| ccamacho | If I i.e. build the url like "${messaging_driver}://${messaging_username}:${messaging_password}@${messaging_hosts}:${messaging_port}/", the result is like rabbit://guest:guest@[ips:here]:5673/ so the values are correct, hosts is an array | 14:17 |
| mwhahaha | Not that is not correct | 14:17 |
| mwhahaha | You can't build that strikes g if messaging hosts is an array | 14:18 |
| mwhahaha | s/strikes g/string | 14:18 |
| mwhahaha | Stupid phone autocorrect | 14:18 |
| ccamacho | yeahp I know, just wanted to write something to the config file and check if the hosts were an array, in my local env is $messaging_hosts = any2array(hiera('rabbitmq_node_names', undef)), | 14:19 |
| mwhahaha | \"overcloud-controller-0.internalapi.localdomain\":String | 14:19 |
| ccamacho | mm let me give you the last error | 14:20 |
| mwhahaha | My internet is dead so I'm on phone, ill look more if I can get my Internets back | 14:20 |
| *** rain has joined #tripleo | 14:20 | |
| *** lblanchard has joined #tripleo | 14:21 | |
| ccamacho | mwhahaha thanks for the hints, ill keep checking the parameters. and will pasta the error when finished the deployment | 14:24 |
| *** dprince has joined #tripleo | 14:24 | |
| *** morazi has joined #tripleo | 14:26 | |
| openstackgerrit | Michele Baldessari proposed openstack/puppet-tripleo: Initial pacemaker remote profile support https://review.openstack.org/400967 | 14:26 |
| *** jcoufal has joined #tripleo | 14:27 | |
| *** yamahata_ has joined #tripleo | 14:28 | |
| *** jeckersb_gone is now known as jeckersb | 14:29 | |
| *** liverpooler has quit IRC | 14:32 | |
| *** liverpooler has joined #tripleo | 14:32 | |
| *** udesale has joined #tripleo | 14:47 | |
| *** dbecker has joined #tripleo | 14:49 | |
| b00tcat | hi, are the OVB test passing now? I remember they where broken not that long ago | 14:50 |
| *** rain has quit IRC | 14:52 | |
| openstackgerrit | Saverio Proto proposed openstack/diskimage-builder: Support sysv init system used by Debian Wheezy https://review.openstack.org/409866 | 14:53 |
| *** jpena|lunch is now known as jpena | 14:54 | |
| jpich | florianf: FWIW ocata-2 seems to be tomorrow, not Friday!! | 14:55 |
| slagle | hmm, changing the undercloud's ip address horribly breaks swift | 14:56 |
| slagle | the old ip is embedded in the ring files | 14:57 |
| *** links has quit IRC | 14:57 | |
| florianf | jpich: you're right! Looks like I need to nag people for reviews on that one: https://review.openstack.org/#/c/399626/ | 14:58 |
| florianf | ^ akrivoka, honza, jrist for instance... ;-) | 14:58 |
| jrist | florianf: it will be reviewed today | 15:00 |
| lucasagomes | trown, <all> if you have some time later on mind taking a look at https://review.openstack.org/#/c/399704/ ? I've fixed the stuff we discussed the last time about the privilege model | 15:00 |
| jpich | florianf: Yeah it'd be great to get it in O-2 - and if there are nits remaining we can fix them over the next milestone... I'll try and get to it today or tomorrow morning as well!! | 15:00 |
| lucasagomes | cheers :-) | 15:00 |
| florianf | jrist, jpich: Thanks a ton! | 15:01 |
| *** hrybacki|mtg is now known as hrybacki | 15:02 | |
| shardy | slagle: Yeah that was discussed in a bug recently, sec let me find it | 15:02 |
| *** lblanchard has quit IRC | 15:03 | |
| *** Lord_Phoenix has quit IRC | 15:03 | |
| *** tzumainn has joined #tripleo | 15:03 | |
| shardy | slagle: https://bugs.launchpad.net/tripleo/+bug/1645267 and https://bugs.launchpad.net/tripleo/+bug/1646450 | 15:04 |
| openstack | Launchpad bug 1645267 in tripleo "Undercloud upgrade breaks due to IP range change" [High,Triaged] | 15:04 |
| openstack | Launchpad bug 1645267 in tripleo "duplicate for #1646450 Undercloud upgrade breaks due to IP range change" [High,Triaged] | 15:04 |
| shardy | slagle: basically I think we need a check which prevents the undercloud upgrade running with a mismatched range? | 15:04 |
| *** dmarlin is now known as dmarlin_ | 15:07 | |
| d0ugal | jpich: did you open a bug about the deprecation of CLI commands? | 15:07 |
| jpich | d0ugal: I did! https://bugs.launchpad.net/tripleo/+bug/1649541 | 15:08 |
| openstack | Launchpad bug 1649541 in tripleo "Deprecate baremetal import/introspect/configure commands" [Medium,Triaged] - Assigned to Julie Pichon (jpichon) | 15:08 |
| d0ugal | jpich: thanks | 15:08 |
| *** lblanchard has joined #tripleo | 15:08 | |
| *** ooolpbot has joined #tripleo | 15:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649742 | 15:10 |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha" [Critical,Triaged] | 15:10 |
| *** ooolpbot has quit IRC | 15:10 | |
| EmilienM | sshnaidm|afk: https://review.openstack.org/#/c/410738/ is failing :-/ | 15:12 |
| EmilienM | jpich: I hate releasing on Friday :-) | 15:13 |
| *** chem has quit IRC | 15:13 | |
| jpich | EmilienM: Makes total sense to me :) | 15:13 |
| *** chem has joined #tripleo | 15:13 | |
| panda | sshnaidm|afk: maybe you should use sigkill | 15:13 |
| *** sshnaidm|afk is now known as sshnaidm | 15:13 | |
| slagle | shardy: could be, or instructions on how to rebuild the ring files | 15:14 |
| slagle | assuming it's possible | 15:14 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart: make quickstart-extras-requirements.txt a default requirements file https://review.openstack.org/410757 | 15:14 |
| EmilienM | panda: sounds like it fails during the overcloud | 15:14 |
| shardy | slagle: Yeah but won't allowing the range change to work break all deployed overclouds? | 15:15 |
| shardy | e.g they'll be pointing to swift on the old ctlplane range for metadata? | 15:15 |
| shardy | I feel like maybe failing with a helpful message of what to do next might be best | 15:16 |
| sshnaidm | panda, it's weird, sigterm should have worked.. will try, but the problem could be not with dstat :( | 15:16 |
| panda | EmilienM: we're also seeing the postci hang again, where do you see the overcloud failure ? | 15:17 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: Kill dstat to prevent hanging of job https://review.openstack.org/410738 | 15:17 |
| *** _milan_ is now known as milan | 15:18 | |
| sshnaidm | EmilienM, multinode? | 15:19 |
| *** chlong has joined #tripleo | 15:20 | |
| *** tosky has quit IRC | 15:22 | |
| *** milan is now known as [milan] | 15:23 | |
| *** jefrite_ has quit IRC | 15:30 | |
| *** ohamada__ has joined #tripleo | 15:33 | |
| *** ohamada_ has quit IRC | 15:33 | |
| openstackgerrit | Merged openstack/diskimage-builder: elements: dib-python: Add python2 as the default version for openSUSE https://review.openstack.org/410639 | 15:33 |
| *** fragatina has joined #tripleo | 15:37 | |
| *** fragatina has quit IRC | 15:37 | |
| *** fragatina has joined #tripleo | 15:38 | |
| *** arxcruz|brb is now known as arxcruz | 15:39 | |
| *** saneax is now known as saneax-_-|AFK | 15:41 | |
| *** [milan] is now known as miLan | 15:43 | |
| *** miLan is now known as iMilan | 15:44 | |
| *** iMilan is now known as ]milan[ | 15:45 | |
| *** ]milan[ is now known as milan | 15:46 | |
| *** NobodyCam is now known as WeAreAllMiLan | 15:46 | |
| *** udesale has quit IRC | 15:47 | |
| *** WeAreAllMiLan is now known as NobodyCam | 15:48 | |
| *** jkilpatr_ has joined #tripleo | 15:50 | |
| *** jkilpatr has quit IRC | 15:52 | |
| *** trown is now known as trown|outtypewww | 15:52 | |
| *** milan is now known as iMPoster | 15:52 | |
| *** iMPoster is now known as milan | 15:52 | |
| openstackgerrit | Flavio Percoco proposed openstack/tripleo-heat-templates: Use the local collector to bootstrap deployed servers https://review.openstack.org/405032 | 15:56 |
| openstackgerrit | Flavio Percoco proposed openstack/tripleo-heat-templates: Introduce role-specific NodeUserData, use for docker https://review.openstack.org/408374 | 15:56 |
| openstackgerrit | Flavio Percoco proposed openstack/tripleo-heat-templates: Use overcloud-full instead of atomic-image https://review.openstack.org/395880 | 15:56 |
| openstackgerrit | Flavio Percoco proposed openstack/tripleo-heat-templates: docker: expose agent mounts as template param https://review.openstack.org/409579 | 15:56 |
| *** ayoung has joined #tripleo | 15:57 | |
| openstackgerrit | mathieu bultel proposed openstack/tripleo-quickstart-extras: add composable services deployment support for oooq https://review.openstack.org/410826 | 15:58 |
| *** fragatina has quit IRC | 15:58 | |
| openstackgerrit | mathieu bultel proposed openstack/tripleo-quickstart: Add keystone only config file for composable roles deployment https://review.openstack.org/410831 | 16:06 |
| openstackgerrit | mathieu bultel proposed openstack/tripleo-quickstart-extras: add composable services deployment support for oooq https://review.openstack.org/410826 | 16:07 |
| openstackgerrit | mathieu bultel proposed openstack/tripleo-quickstart-extras: add composable services deployment support for oooq https://review.openstack.org/410826 | 16:08 |
| jpich | florianf: So, about my Top Secret Plan To Improve The Onboarding Docs | 16:08 |
| jpich | florianf: Actually I only have 2 items on that to-do list at the moment :) I'd like to clean up a bit http://tripleo.org/introduction/components.html for the bits I know are wrong or outdated, and maybe add a few more explanations so it's more useful to people who really just arrived there from the docs intro | 16:09 |
| jpich | florianf: For example it wasn't particularly clear to me that the tripleo heat templates were only relevant/describing the overcloud, when I started | 16:09 |
| *** ooolpbot has joined #tripleo | 16:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649742 | 16:10 |
| *** ooolpbot has quit IRC | 16:10 | |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha" [Critical,Triaged] | 16:10 |
| *** ealcaniz has quit IRC | 16:10 | |
| jpich | florianf: Then I don't think the bit about needing to update specs in the RDO github in order to get upstream patches with new dependencies is documented anywhere (?) (not in the upstream docs at least? Maybe on RDO somewhere?) and this confused me a lot the first time I hit that so, hopefully that'd fit under the dev docs on tripleo.org as well | 16:10 |
| jpich | florianf: If you had other ideas or pain points it'd be cool to expand, though :) | 16:11 |
| openstackgerrit | mathieu bultel proposed openstack/tripleo-quickstart-extras: add composable services deployment support for oooq https://review.openstack.org/410826 | 16:12 |
| jpich | florianf: I've also been pestering honza with writing developer docs for the UI but I know there's been lots of other stuff going on (the hands-on presentation from last week could be a good basis though :-)) | 16:12 |
| florianf | jpich: that's what I was gonna say: the ui docs definitely could use some love as well. | 16:13 |
| *** kodokuu has quit IRC | 16:14 | |
| *** rcernin has quit IRC | 16:14 | |
| jpich | florianf: Feel free to start on these docs :-) I'll be happy to review and learn from it, though I can't do much more yet | 16:14 |
| *** dmarlin has joined #tripleo | 16:15 | |
| jpich | florianf: tripleo-common has some docs in-tree ( https://github.com/openstack/tripleo-common/tree/master/doc/source ) so maybe we can do something similar (...although maybe that brings us into issues with the infra mainly using python tooling for docs?) | 16:15 |
| florianf | jpich: Also: the developer docs don't yet mention quickstart... | 16:15 |
| florianf | jpich: although that's probably going to end up in the environment setup section of the docs anyway (I guess...?) | 16:16 |
| jpich | florianf: Good point, I thought maybe it was documented elsewhere but I'm not seeing it on http://docs.openstack.org/developer/openstack-projects.html either | 16:18 |
| *** Goneri has quit IRC | 16:18 | |
| jpich | There is in-tree documentation though: https://github.com/openstack/tripleo-quickstart/tree/master/doc/source | 16:18 |
| jpich | Maybe it'd be good to link to it more prominently, also do the same for the tripleo-common ones | 16:19 |
| jpich | florianf: You know what, I might file all these as documentation bugs over the next few days, since the list has now grown beyond 2 tasks :-) | 16:19 |
| *** jaosorior has quit IRC | 16:20 | |
| *** jaosorior has joined #tripleo | 16:21 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart: config for containerized-compute https://review.openstack.org/393348 | 16:24 |
| *** dsariel has quit IRC | 16:24 | |
| *** Goneri has joined #tripleo | 16:25 | |
| *** aufi has quit IRC | 16:28 | |
| florianf | jpich: big +1 on filing bugs! | 16:28 |
| jpich | florianf: :) I'll try and do that tomorrow | 16:29 |
| jpich | florianf: I'm looking at your awesome i18n patch right now, it looks much simpler than I expected/feared! | 16:30 |
| *** zoli is now known as zoli|afk-kiosk | 16:30 | |
| jpich | florianf: Will the use of localStorage conflict with the on-going work at https://review.openstack.org/#/c/408184/ , or are these two completely different concepts? | 16:30 |
| openstackgerrit | Chris Jones proposed openstack/puppet-pacemaker: [WIP] Add fence_ironic support. https://review.openstack.org/390396 | 16:31 |
| openstackgerrit | mathieu bultel proposed openstack/tripleo-quickstart-extras: add composable services deployment support for oooq https://review.openstack.org/410826 | 16:31 |
| shardy | dprince: Hey, can you give me a summary of the steps needed to switch an old image over to use the new hiera hook? | 16:33 |
| shardy | dprince: just installing the heat-config-agent-hiera hook doesn't seem to be enough, and IIRC we've got to disable some element scripts? | 16:33 |
| sshnaidm | bnemec, dstat was hanging job when it ran locally on dev env, so we tried to see if it's a problem on CI too.. maybe it's not | 16:33 |
| bnemec | sshnaidm: Interesting. I wonder if it's some sort of bad interaction with the parallel log collection wait: http://git.openstack.org/cgit/openstack-infra/tripleo-ci/tree/scripts/common_functions.sh#n212 | 16:36 |
| florianf | jpich: No, it doesn't interfere with the potential switch to a cookie. localStorage is just a way to store domain-related stuff in the browser. We don't use it for the token, because that would be a little hairy security-wise. But we can still use it for a simple language setting. | 16:36 |
| shardy | dprince: Ah, I suspect https://review.openstack.org/#/c/392615/ is what I'm missing | 16:36 |
| jpich | florianf: Excellent, thank you for the clarification :) | 16:37 |
| *** dsariel has joined #tripleo | 16:40 | |
| sshnaidm | bnemec, if this patch won't help, so it's not dstats fault at all | 16:41 |
| sshnaidm | bnemec, I wonder if there are any latest changes in infra that could break us | 16:41 |
| sshnaidm | because I'm not aware of last changes in tripleo-ci that could cause this.. | 16:42 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: TEST: List active jobs in postci https://review.openstack.org/410855 | 16:43 |
| bnemec | sshnaidm: ^ might be helpful too | 16:43 |
| *** abregman|mtg is now known as abregman | 16:44 | |
| bnemec | I will say I've noticed in the past that there is an unusually long time between when postci completes and when console.log resumes logging. | 16:44 |
| *** abregman has quit IRC | 16:44 | |
| *** chlong has quit IRC | 16:45 | |
| bnemec | For example, on a passing nonha job, postci completes at 20:38:42, but console.log doesn't resume until 20:40:42. | 16:45 |
| bnemec | http://logs.openstack.org/30/409130/7/check-tripleo/gate-tripleo-ci-centos-7-ovb-nonha/f2a09bb/console.html | 16:45 |
| bnemec | sshnaidm: ^ | 16:45 |
| *** rcernin has joined #tripleo | 16:46 | |
| sshnaidm | bnemec, yep, weird | 16:48 |
| *** jaosorior has quit IRC | 16:48 | |
| *** jaosorior has joined #tripleo | 16:48 | |
| sshnaidm | bnemec, it was already problem with that before, exactly the same, but I think it just passed over and nobody cared too much :) | 16:48 |
| sshnaidm | https://bugs.launchpad.net/tripleo/+bug/1583676 | 16:50 |
| openstack | Launchpad bug 1583676 in tripleo "Ci: postci functions sometimes freezes" [High,Fix released] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm) | 16:50 |
| *** ctayal has joined #tripleo | 16:54 | |
| *** bvandenh has quit IRC | 16:57 | |
| *** fragatina has joined #tripleo | 16:57 | |
| *** chlong has joined #tripleo | 16:58 | |
| *** dsneddon has joined #tripleo | 16:59 | |
| openstackgerrit | mathieu bultel proposed openstack/tripleo-quickstart-extras: Add no-cleanup arg for overcloud deployment https://review.openstack.org/407059 | 16:59 |
| *** limao_ has quit IRC | 16:59 | |
| *** yprokule has quit IRC | 17:00 | |
| *** rajinir has joined #tripleo | 17:00 | |
| rlandy | dprince: I am running a bond-with-vlans deployment that has been failing overcloud deployment "Could not fetch contents for file:///home/scripts/run-os-net-config.sh". I use modified nic-config files copied to the /home/stack directory. | 17:04 |
| *** jlinkes has quit IRC | 17:05 | |
| rlandy | dprince: looking at the changes from https://github.com/openstack/tripleo-heat-templates/commit/2985cd9a3a04acfe069c063c65ebf487a1413388 | 17:05 |
| larsks | I have a deployment that got as far as "[overcloud.Controller.0.ControllerDeployment]: CREATE_IN_PROGRESS state changed"...and then seems to have stopped. There are no errors on the controller, other than the failure of os-collect-config to reach the metadata service @ 169.254.169.254. What should I be looking at? | 17:06 |
| shardy | larsks: you probably need to update your image - there was a dependency on a new hiera hook added recently | 17:07 |
| larsks | shardy, thanks. I will try that. | 17:07 |
| *** tremble has quit IRC | 17:08 | |
| shardy | larsks: I'm working on the steps to switch over to the new hook during upgrade, but that process isn't yet well proven, so rebuilding or pulling new overcloud-full images is probably easiest | 17:08 |
| larsks | I'm using OSP10 images; should the latest rhosp-director-images be sufficient? | 17:08 |
| *** dhill_ has quit IRC | 17:08 | |
| shardy | larsks: Oh, I assumed you were running tripleo builds - the change I mentioned should not exist in newton/OSP10 builds | 17:09 |
| larsks | shardy, darn :) | 17:09 |
| *** ooolpbot has joined #tripleo | 17:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649742 | 17:10 |
| *** ooolpbot has quit IRC | 17:10 | |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha" [Critical,Triaged] | 17:10 |
| shardy | larsks: https://review.openstack.org/#/c/384757/ is the change, and there are some related patches changing the image | 17:10 |
| openstackgerrit | Gabriele Cerami proposed openstack/tripleo-quickstart-extras: Add overcloud delete role https://review.openstack.org/402228 | 17:10 |
| shardy | larsks: you would see what you describe if you tried deploying trunk t-h-t with newton images (that's exactly what I've been debugging actually ;) | 17:10 |
| larsks | Well, I will probably be trying that shortly, so good to know... | 17:11 |
| shardy | larsks: I guess my next step would be looking at the network config, perhaps the failure to poll the metatadata service is a clue | 17:12 |
| *** shardy is now known as shardy_afk | 17:13 | |
| *** bvandenh has joined #tripleo | 17:13 | |
| panda | sshnaidm: crossing fingers | 17:14 |
| *** dhill_ has joined #tripleo | 17:14 | |
| sshnaidm | panda, :( | 17:15 |
| *** dsneddon_ has joined #tripleo | 17:15 | |
| openstackgerrit | mathieu bultel proposed openstack/tripleo-quickstart: Add keystone only config file for composable roles deployment https://review.openstack.org/410831 | 17:16 |
| sshnaidm | panda, let's wait for bnemec patch with jobs listing.. | 17:16 |
| EmilienM | so it sounds like to me we have 2 CI problems | 17:17 |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates: Make the openvswitch 2.4->2.5 upgrade more robust https://review.openstack.org/399708 | 17:17 |
| EmilienM | 1) https://bugs.launchpad.net/tripleo/+bug/1649742 | 17:17 |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha" [Critical,Triaged] | 17:17 |
| EmilienM | and 2) multinode job failing sometimes http://logs.openstack.org/38/410738/2/check/gate-tripleo-ci-centos-7-nonha-multinode/00f4c89/console.html#_2016-12-14_16_30_30_398990 | 17:17 |
| marios | jistr: chem i updated https://review.openstack.org/399708 no repetition not even once \o/ thanks guys | 17:17 |
| EmilienM | it sounds like the boot from volume doesn't work | 17:18 |
| EmilienM | and it could be related to the qemu thing we're talking about every day | 17:18 |
| *** jpich has quit IRC | 17:18 | |
| EmilienM | dmsimard: http://logs.openstack.org/38/410738/2/check/gate-tripleo-ci-centos-7-nonha-multinode/00f4c89/logs/subnode-2/var/log/nova/nova-compute.txt.gz#_2016-12-14_16_30_02_244 | 17:18 |
| EmilienM | is is the qemu thing? | 17:18 |
| dmsimard | looking | 17:19 |
| *** fragatina has quit IRC | 17:19 | |
| panda | EmilienM: the boot volume thing was caused by libvirt crashes | 17:19 |
| bnemec | EmilienM: That's a libvirt segfault: http://logs.openstack.org/38/410738/2/check/gate-tripleo-ci-centos-7-nonha-multinode/00f4c89/logs/subnode-2/var/log/messages | 17:19 |
| EmilienM | do we know why it crash? | 17:19 |
| dmsimard | EmilienM: qemu-kvm-common-ev-2.3.0-31.0.el7_2.21.1.x86_64 -- probably not good if underlying system has 7.3 packages | 17:19 |
| panda | EmilienM: http://logs.openstack.org/38/410738/2/check/gate-tripleo-ci-centos-7-nonha-multinode/00f4c89/logs/subnode-2/var/log/messages , search for "segfault" | 17:20 |
| EmilienM | dmsimard: so we need https://review.openstack.org/#/c/410359/ right? | 17:20 |
| dmsimard | EmilienM: I'm not sure about the issues regarding 2.3 on 7.3, amoralej might be more familiar with them | 17:20 |
| dmsimard | I know about the 2.6 issue on 7.3 and that's what the review you linked is about | 17:21 |
| dmsimard | sgordon commented on the 2.3 thing somewhere | 17:21 |
| EmilienM | let's try if it works again in your patch, I'm removing the useless depends-on | 17:21 |
| EmilienM | amoralej: can you also look please? | 17:21 |
| * amoralej checking | 17:21 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: Default libvirt_cpu_mode to none when using qemu https://review.openstack.org/410359 | 17:21 |
| EmilienM | dmsimard: ^ removing depends-on and seeing if it pass multinode job | 17:22 |
| EmilienM | dmsimard: it might be something we need | 17:22 |
| EmilienM | dmsimard: hopefully ovb jobs are not voting so we can fix problem 1) | 17:22 |
| dmsimard | EmilienM: see this mail from sgordon http://lists.openstack.org/pipermail/openstack-dev/2016-December/108898.html | 17:22 |
| amoralej | but, as we are fixing it in puppet-nova, is this required in tripleo-ci ? | 17:22 |
| EmilienM | amoralej: yes, tripleo jobs are voting in puppet CI | 17:23 |
| EmilienM | amoralej: so we need to fix tripleo ci first | 17:23 |
| amoralej | arggg | 17:23 |
| *** dsneddon_ has quit IRC | 17:23 | |
| panda | sshnaidm: ya, seems to be stuck again | 17:23 |
| *** jkilpatr_ has quit IRC | 17:24 | |
| *** fragatina has joined #tripleo | 17:25 | |
| EmilienM | dmsimard: do you see the "CPU feature arat not found" thing in logs? | 17:26 |
| dmsimard | EmilienM: no crap I linked the wrong thing | 17:27 |
| EmilienM | amoralej: wait, sorry I said wrong | 17:28 |
| EmilienM | amoralej: jpena's patch would actually help without patching tripleo-ci, sorry I confused myself | 17:28 |
| amoralej | that was my guess EmilienM | 17:28 |
| EmilienM | yeah sorry again... | 17:28 |
| EmilienM | mwhahaha: you ok to land https://review.openstack.org/#/c/410619/ ? | 17:28 |
| EmilienM | amoralej: the reason is that we run ovb jobs in puppet-nova, not multinode, so it's not voting | 17:29 |
| mwhahaha | sec | 17:29 |
| *** lucasagomes is now known as lucas-afk | 17:30 | |
| amoralej | EmilienM, i don't want to make you waste time, but i don't full understand that, i need to learn more about how tripleo jobs run in upstream gates | 17:31 |
| EmilienM | amoralej: ok no worries, let me explain a little bit : | 17:32 |
| dprince | rlandy: is your bond-with-vlans out of tree? | 17:33 |
| dprince | rlandy: if so the path the the run-os-net-config.sh script needs to be fully qualified I think | 17:34 |
| EmilienM | amoralej: brb lunch | 17:35 |
| *** mcornea has quit IRC | 17:36 | |
| *** jkilpatr_ has joined #tripleo | 17:37 | |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-quickstart-extras: Repositories setup role https://review.openstack.org/408162 | 17:37 |
| *** mcornea has joined #tripleo | 17:38 | |
| *** panda is now known as panda|bbl | 17:38 | |
| *** lmiccini has quit IRC | 17:40 | |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-quickstart-extras: Repositories setup role https://review.openstack.org/408162 | 17:41 |
| *** rbowen has left #tripleo | 17:41 | |
| *** rain has joined #tripleo | 17:41 | |
| *** rain is now known as leanderthal | 17:42 | |
| *** yamahata_ has quit IRC | 17:42 | |
| rlandy | dprince: ah ok - I see it's in the nic-configs as well - which are copied locally. - retrying deploy | 17:43 |
| panda|bbl | sshnaidm: bnemec: I see the worlddump log file on the logs of the hang jobs, that should mean that everything on our side exits correctly and it's devstack that hangs | 17:47 |
| panda|bbl | sshnaidm: bnemec . bah nevermind, worlddump is ran after the timeout | 17:48 |
| panda|bbl | derekh: this morning you just killed dstat ? | 17:50 |
| panda|bbl | sshnaidm: another test we can make is to comment dstat line and see what happens. | 17:51 |
| derekh | panda|bbl: yup | 17:51 |
| *** hewbrocca is now known as hewbrocca_afk | 17:51 | |
| larsks | My overcloud hosts are unable to contact the metadata service @ 169.254.169.254. There seem to be two problems. The first is that we are using REDIRECT rules in the nat/PREROUTING chain, but the metadata service on the undercloud isn't listening on localhost, so that results in a connection refused. (the other problem is that the redirect rule is matching the wrong interface, but that might be not be a tripleo problem). | 17:51 |
| *** ctayal has quit IRC | 17:51 | |
| sshnaidm | panda|bbl, yeah, it's possible | 17:51 |
| derekh | panda|bbl: sshnaidm ye could also try killing the simplehttpserver | 17:51 |
| larsks | Should the metadata service be listening on localhost explicitly, or should it be bound to 0.0.0.0 instead of specific ips? | 17:51 |
| sshnaidm | derekh, agree | 17:53 |
| sshnaidm | let's do a massacre | 17:53 |
| *** jaosorior has quit IRC | 17:54 | |
| *** tosky has joined #tripleo | 17:54 | |
| panda|bbl | sshnaidm: yeah, enough with the pleasantries, let the guns do the talk now. | 17:54 |
| *** dtantsur is now known as dtantsur|afk | 17:56 | |
| bnemec | Kill em all and let the kernel sort them out in /dev/null? | 17:58 |
| bnemec | ;-) | 17:59 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: DONT REVIEW: test https://review.openstack.org/410470 | 17:59 |
| *** mcornea has quit IRC | 18:00 | |
| *** florianf has quit IRC | 18:02 | |
| *** flepied has quit IRC | 18:03 | |
| *** athomas has quit IRC | 18:05 | |
| *** dsariel has quit IRC | 18:05 | |
| *** derekh has quit IRC | 18:09 | |
| *** sudipto has joined #tripleo | 18:09 | |
| *** sudipto_ has joined #tripleo | 18:09 | |
| *** ooolpbot has joined #tripleo | 18:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649742 | 18:10 |
| *** ooolpbot has quit IRC | 18:10 | |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha" [Critical,Triaged] | 18:10 |
| *** iranzo has quit IRC | 18:10 | |
| *** milan has quit IRC | 18:16 | |
| sshnaidm | dstat was killed, it didn't help http://logs.openstack.org/38/410738/2/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/c71a34e/logs/postci.txt.gz#_2016-12-14_17_13_35_000 | 18:16 |
| *** jkilpatr_ has quit IRC | 18:20 | |
| *** david-lyle has quit IRC | 18:21 | |
| *** yamahata_ has joined #tripleo | 18:22 | |
| *** jkilpatr has joined #tripleo | 18:22 | |
| *** thrash|g0ne is now known as thrash | 18:24 | |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart-extras: Use full path to run-os-net-config.sh in locally copied nic-configs https://review.openstack.org/410908 | 18:28 |
| *** david-lyle has joined #tripleo | 18:30 | |
| *** jpena is now known as jpena|off | 18:30 | |
| *** ctayal has joined #tripleo | 18:31 | |
| *** ohamada__ has quit IRC | 18:32 | |
| *** dsneddon_ has joined #tripleo | 18:34 | |
| *** sudipto_ has quit IRC | 18:35 | |
| *** sudipto has quit IRC | 18:35 | |
| *** pkovar has quit IRC | 18:35 | |
| *** dsneddon_ has quit IRC | 18:38 | |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart-extras: Use full path to run-os-net-config.sh in locally copied nic-configs https://review.openstack.org/410908 | 18:41 |
| *** rbrady has quit IRC | 18:43 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: TEST: Drop redirects and pipes from postci call https://review.openstack.org/410915 | 18:45 |
| bnemec | sshnaidm: panda|bbl: ^ Just in case we're somehow losing useful output | 18:46 |
| *** jprovazn has quit IRC | 18:48 | |
| sshnaidm | bnemec, ok, let's see | 18:48 |
| *** fpan has quit IRC | 18:49 | |
| sshnaidm | bnemec, do you know why we run postci with "exit value"? I don't see it's used anywhere | 18:49 |
| *** abehl has quit IRC | 18:51 | |
| bnemec | sshnaidm: It was being used for metrics: http://git.openstack.org/cgit/openstack-infra/tripleo-ci/tree/scripts/common_functions.sh#n229 | 18:52 |
| bnemec | When we start logging those again (whether it's graphite or some other system) we'll still want to be able to do that. | 18:53 |
| sshnaidm | bnemec, don't we log it now? | 18:53 |
| bnemec | sshnaidm: No, the graphite server disappeared on us and we haven't set up a new one. | 18:53 |
| *** fultonj has quit IRC | 18:57 | |
| *** rbrady has joined #tripleo | 18:58 | |
| *** rbrady has joined #tripleo | 18:58 | |
| *** leanderthal has quit IRC | 19:00 | |
| *** tosky has quit IRC | 19:00 | |
| *** fultonj has joined #tripleo | 19:03 | |
| *** cdearborn has joined #tripleo | 19:04 | |
| *** ooolpbot has joined #tripleo | 19:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649742 | 19:10 |
| *** ooolpbot has quit IRC | 19:10 | |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha" [Critical,Triaged] | 19:10 |
| *** liverpooler has quit IRC | 19:10 | |
| *** amoralej is now known as amoralej|off | 19:11 | |
| *** fzdarsky is now known as fzdarsky|afk | 19:11 | |
| *** liverpooler has joined #tripleo | 19:12 | |
| openstackgerrit | Martin Mágr proposed openstack/tripleo-specs: Performance monitoring spec https://review.openstack.org/410924 | 19:15 |
| *** panda|bbl is now known as panda | 19:17 | |
| *** coolsvap has quit IRC | 19:19 | |
| *** ctayal_ has joined #tripleo | 19:25 | |
| *** ctayal has quit IRC | 19:27 | |
| *** ctayal has joined #tripleo | 19:29 | |
| *** fultonj has quit IRC | 19:30 | |
| *** ctayal_ has quit IRC | 19:32 | |
| *** flepied has joined #tripleo | 19:33 | |
| *** jkilpatr has quit IRC | 19:34 | |
| sshnaidm | EmilienM, can you take a look in your time please: https://review.openstack.org/#/c/401282/ | 19:35 |
| *** fragatina has quit IRC | 19:35 | |
| *** fragatina has joined #tripleo | 19:36 | |
| *** fragatina has quit IRC | 19:38 | |
| *** nyechiel_ has quit IRC | 19:46 | |
| *** jkilpatr has joined #tripleo | 19:53 | |
| EmilienM | sshnaidm: sure thing | 19:55 |
| *** cdearborn has quit IRC | 19:57 | |
| *** ayoung has quit IRC | 19:59 | |
| *** fultonj has joined #tripleo | 20:00 | |
| panda | sshnaidm: about https://review.openstack.org/#/c/401282/ this is how infra told me to handle this | 20:00 |
| panda | sshnaidm: https://review.openstack.org/#/c/403776/ | 20:00 |
| *** akrivoka has quit IRC | 20:01 | |
| panda | sshnaidm: the discussion on why is here https://review.openstack.org/402123 | 20:01 |
| sshnaidm | panda, I don't get how it will help to depends-on | 20:03 |
| *** dsariel has joined #tripleo | 20:04 | |
| sshnaidm | panda, why not to use infras way to do it.. | 20:05 |
| panda | sshnaidm: after you download extras in infra, it needs to be installed anyway in the virtual env | 20:06 |
| sshnaidm | panda, yes, and I hope to add possibility to install reqs from directory to quickstart for this | 20:07 |
| panda | sshnaidm: if you want to install the version downloaded with the infra, you need to know where it is | 20:07 |
| sshnaidm | panda, it's always where is all repos are located, "/opt/stack/new/.." | 20:07 |
| panda | sshnaidm: and change the requirements file accordingly | 20:08 |
| sshnaidm | panda, yep, one more option | 20:08 |
| sshnaidm | panda, but infra will download for you everything and will apply all patches you need on this repo | 20:08 |
| sshnaidm | panda, and you'll have already patched repo with all dependent-on in it, just use it | 20:09 |
| sshnaidm | panda, it's as I understand the current process | 20:10 |
| *** ooolpbot has joined #tripleo | 20:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649742 | 20:10 |
| *** ooolpbot has quit IRC | 20:10 | |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha" [Critical,Triaged] | 20:10 |
| panda | sshnaidm: using zuul-cloner is the same thing as using infra. Anyway the problem is that you need a different requirements file for local use and for upstream use | 20:11 |
| panda | sshnaidm: this is what I tried to do here https://review.openstack.org/402091 | 20:12 |
| *** cdearborn has joined #tripleo | 20:14 | |
| sshnaidm | panda, infra doesn't only clone repo with zuul_cloner, it also applies all patches in zuul_changes iirc, that's the most important part. All you need it's just support for using repo from directory, it's not different requirements file | 20:15 |
| panda | sshnaidm: what you mean by repo from directory ? | 20:16 |
| *** Goneri has quit IRC | 20:16 | |
| sshnaidm | panda, I'd prefer option in quickstart.sh like "--extras-directory" | 20:16 |
| sshnaidm | quickstart.sh --extras-directory file:///opt/stack/new/tripleo-quickstart-extras | 20:17 |
| sshnaidm | and installing req from there | 20:17 |
| *** bvandenh has quit IRC | 20:19 | |
| EmilienM | sshnaidm: I'm approving https://review.openstack.org/#/c/410359/ | 20:20 |
| EmilienM | sshnaidm: and I hope it will help to have https://review.openstack.org/#/c/410738/ passing | 20:20 |
| EmilienM | sshnaidm: i'm rebasing it | 20:20 |
| panda | sshnaidm: zuul cloner takes care of the ZUUL_CHANGES and installs all the patches. using an --extras-directory option isn't much different than using a different requirements file. The only difference with the approach infra suggested is that you don't need to specify any particular option to make things work upstream, it automatically detects that it's running in upstream and behaves accordingly | 20:20 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: Kill dstat to prevent hanging of job https://review.openstack.org/410738 | 20:20 |
| sshnaidm | EmilienM, sure, thanks | 20:21 |
| EmilienM | sshnaidm: i'm not sure it will work but let's try that. I'm trying to deploy OVB in my env now | 20:21 |
| sshnaidm | EmilienM, although I'm not sure this patch solves postci problem | 20:21 |
| sshnaidm | EmilienM, ok | 20:22 |
| panda | what's the correlation between qemu and dstat ? :blink: | 20:22 |
| sshnaidm | EmilienM, locally OVB works, tried it a few times today | 20:22 |
| sshnaidm | panda, just for approve | 20:22 |
| bnemec | At this point it wouldn't shock me if the qemu change somehow fixed the postci hang too. | 20:24 |
| bnemec | It's that bizarre. :-) | 20:24 |
| *** rhallisey has quit IRC | 20:24 | |
| panda | bnemec: it's a weird bug, it requires weird fixes | 20:24 |
| EmilienM | panda, sshnaidm: I'm rebasing because I saw pingtest failing on the qemu problem | 20:26 |
| *** ctayal has quit IRC | 20:26 | |
| EmilienM | the boot from volume was failing in the ovb-ha job because of some libvirt connection error | 20:26 |
| panda | EmilienM: ah, the segfault | 20:27 |
| sshnaidm | panda, not sure about zuul_cloner, need to check. This addition in quickstart is not required if infra does it. BTW, you clone it to /opt/.git, which we delete in our job: https://github.com/openstack-infra/tripleo-ci/blob/master/toci_gate_test.sh#L19 | 20:27 |
| openstackgerrit | Alex Schultz proposed openstack/os-cloud-config: Skip 1.13.1 version of docutils https://review.openstack.org/410944 | 20:28 |
| EmilienM | liberty has been EOL in tripleo, yeah! | 20:29 |
| sshnaidm | need to remove it from status page.. | 20:30 |
| *** zoli|afk-kiosk is now known as zoli|gone | 20:31 | |
| *** gfidente has quit IRC | 20:32 | |
| *** fultonj has quit IRC | 20:33 | |
| *** pcaruana has quit IRC | 20:33 | |
| openstackgerrit | Alex Schultz proposed openstack/os-cloud-config: Skip 0.13.1 version of docutils https://review.openstack.org/410944 | 20:36 |
| *** shinobu__ has quit IRC | 20:37 | |
| *** StevenK has quit IRC | 20:38 | |
| *** rbowen has joined #tripleo | 20:38 | |
| *** StevenK has joined #tripleo | 20:38 | |
| *** abehl has joined #tripleo | 20:45 | |
| *** ayoung has joined #tripleo | 20:46 | |
| *** ayoung has quit IRC | 20:49 | |
| openstackgerrit | Peter Stachowski proposed openstack/diskimage-builder: Pip install as 10- incompatible with 05-heat-cfntools https://review.openstack.org/410951 | 20:50 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Use hostname -s instead of hostnamectl --transient https://review.openstack.org/410952 | 20:51 |
| *** fultonj has joined #tripleo | 20:52 | |
| *** dprince has quit IRC | 20:57 | |
| EmilienM | bnemec: thanks for https://bugs.launchpad.net/tripleo/+bug/1650005 -- I was also wondering if we already had a bug report | 21:01 |
| openstack | Launchpad bug 1650005 in tripleo "Multinode jobs failing on libvirt issues" [Critical,Triaged] | 21:01 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart-extras: add the containers prep role to the quickstart-extras https://review.openstack.org/400983 | 21:03 |
| *** ipsecguy has quit IRC | 21:06 | |
| *** jkilpatr has quit IRC | 21:07 | |
| sshnaidm | bnemec, so all jobs are done? http://logs.openstack.org/55/410855/1/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/c890755/logs/postci.txt.gz#_2016-12-14_18_37_15_000 | 21:07 |
| openstackgerrit | Gabriele Cerami proposed openstack/tripleo-quickstart: Update list of packages to include from delorean-current for devmode https://review.openstack.org/410955 | 21:08 |
| *** ooolpbot has joined #tripleo | 21:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649742 | 21:10 |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha" [Critical,Triaged] | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1650005 | 21:10 |
| *** ooolpbot has quit IRC | 21:10 | |
| openstack | Launchpad bug 1650005 in tripleo "Multinode jobs failing on libvirt issues" [Critical,Triaged] | 21:10 |
| bnemec | EmilienM: Yeah, I figured we should have an alert for it so at least people will know why their jobs keep failing. | 21:13 |
| EmilienM | yeah | 21:13 |
| bnemec | sshnaidm: AFAICT, yes. I don't see anything wrong with the output there. | 21:13 |
| bnemec | Which doesn't get us any closer to a solution, unfortunately. | 21:14 |
| sshnaidm | bnemec, maybe it's worth to check testenv-client as panda suggested.. but no idea why can it fail in ha particularly | 21:17 |
| bnemec | sshnaidm: Hmm, this might be relevant: | 21:18 |
| bnemec | [1] Running python -m SimpleHTTPServer 8766 > $WORKSPACE/logs/yum_mirror.log 2> $WORKSPACE/logs/yum_mirror_error.log & (wd: /opt/stack/new/delorean/data/repos) | 21:18 |
| bnemec | I'm seeing that in the jobs output on the followup patch with no redirects. | 21:18 |
| panda | derek suggested to kill the server | 21:19 |
| panda | to check testenv-client we'd have to add some more logging messages ... | 21:19 |
| sshnaidm | bnemec, I killed it here (I hope), didn't help: http://logs.openstack.org/70/410470/4/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/4988df1/logs/postci.txt.gz#_2016-12-14_19_49_40_000 | 21:20 |
| *** rbrady has quit IRC | 21:21 | |
| *** jkilpatr has joined #tripleo | 21:22 | |
| *** ipsecguy has joined #tripleo | 21:28 | |
| *** liverpooler has quit IRC | 21:30 | |
| *** rbrady has joined #tripleo | 21:34 | |
| *** liverpooler has joined #tripleo | 21:35 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: TEST: extra testenv-client logging https://review.openstack.org/410965 | 21:36 |
| *** Goneri has joined #tripleo | 21:38 | |
| *** ipsecguy has quit IRC | 21:39 | |
| *** liverpooler has quit IRC | 21:40 | |
| *** milan has joined #tripleo | 21:43 | |
| *** dciabrin has quit IRC | 21:43 | |
| *** cdearborn has quit IRC | 21:43 | |
| *** bana_k has joined #tripleo | 21:47 | |
| *** cdearborn has joined #tripleo | 21:49 | |
| *** sshnaidm is now known as sshnaidm|afk | 21:49 | |
| *** shinobu_ has quit IRC | 21:49 | |
| *** shinobu_ has joined #tripleo | 21:52 | |
| *** ipsecguy has joined #tripleo | 21:54 | |
| openstackgerrit | Dan Prince proposed openstack/tripleo-common: Don't require mistralclient for password gen https://review.openstack.org/410970 | 21:55 |
| *** jayg is now known as jayg|g0n3 | 21:57 | |
| *** ansmith has quit IRC | 21:58 | |
| openstackgerrit | Dan Prince proposed openstack/python-tripleoclient: Deploy the undercloud with Heat https://review.openstack.org/351351 | 22:03 |
| *** jcoufal has quit IRC | 22:05 | |
| *** lblanchard has quit IRC | 22:10 | |
| *** ooolpbot has joined #tripleo | 22:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649742 | 22:10 |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha" [Critical,Triaged] | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1650005 | 22:10 |
| *** ooolpbot has quit IRC | 22:10 | |
| openstack | Launchpad bug 1650005 in tripleo "Multinode jobs failing on libvirt issues" [Critical,Triaged] | 22:10 |
| *** ayoung has joined #tripleo | 22:11 | |
| *** rbowen has quit IRC | 22:11 | |
| mwhahaha | ccamacho: i think i figured out that map error | 22:12 |
| ccamacho | hey mwhahaha | 22:12 |
| ccamacho | the array with the host? | 22:13 |
| ccamacho | s | 22:13 |
| mwhahaha | i think it's normalize_ip_for_uri switching an array to a string | 22:13 |
| mwhahaha | i'm testing now | 22:13 |
| mwhahaha | https://github.com/openstack/puppet-openstacklib/blob/master/lib/puppet/parser/functions/normalize_ip_for_uri.rb#L34 | 22:13 |
| mwhahaha | https://github.com/openstack/puppet-openstacklib/blob/master/lib/puppet/parser/functions/os_transport_url.rb#L115 | 22:14 |
| *** cdearborn has quit IRC | 22:14 | |
| ccamacho | mwhahaha ack, I launched a test a few hour ago, let me check if it failed or passed | 22:19 |
| mwhahaha | i was able to repro the issue just in unit tests in openstacklib | 22:19 |
| EmilienM | 44 blueprints and 147 bugs targeted | 22:21 |
| EmilienM | ocata-3 | 22:21 |
| EmilienM | I hope you guys don't take PTO for christmas | 22:21 |
| EmilienM | also stop sleeping | 22:21 |
| jidar | hah | 22:21 |
| ccamacho | EmilienM who likes to sleep man... | 22:22 |
| panda | EmilienM: can you place an order for a day of 48 hours to management too ? | 22:22 |
| EmilienM | sleeping is cheating | 22:22 |
| mwhahaha | ccamacho: https://bugs.launchpad.net/puppet-openstacklib/+bug/1650042 and https://bugs.launchpad.net/puppet-openstacklib/+bug/1650042 | 22:22 |
| openstack | Launchpad bug 1650042 in puppet-openstacklib "os_transport_url fails when an array of size 1 is passed in as the hosts param" [Medium,Confirmed] - Assigned to Alex Schultz (alex-schultz) | 22:22 |
| openstackgerrit | Carlos Camacho proposed openstack/puppet-tripleo: Use transport_url for rabbitmq connection parameters in heat https://review.openstack.org/410333 | 22:24 |
| ccamacho | mwhahaha yeah thats the thing, I ran a workaround to see if other issue was unblocked.. | 22:24 |
| EmilienM | panda: I'm watching https://review.openstack.org/#/c/410738/ in the gate | 22:24 |
| ccamacho | but didnt had luck there | 22:24 |
| mwhahaha | er https://review.openstack.org/410979 | 22:24 |
| EmilienM | panda: it has been 10 min it's stuck after pingtest | 22:24 |
| EmilienM | panda: it sounds like postci returned 0 | 22:25 |
| ccamacho | mwhahah in the mean time any2array($messaging_hosts)[0] and use host instead | 22:25 |
| *** jeckersb is now known as jeckersb_gone | 22:25 | |
| panda | EmilienM: as usual, postci completes correctly, it's something *after* that hangs, but there's nothing after. after that the control should be returned to testenv-client | 22:26 |
| EmilienM | and why HA job all the time | 22:27 |
| EmilienM | the nonha seems pretty stable now | 22:27 |
| panda | EmilienM: I don't know, we killed all the killable there, at this point the next step is to check what testenv-client is doing, if the control is effectively returned to it, and if it hangs in any of its operations | 22:29 |
| EmilienM | I saw bnemec is experimenting something, is it related? | 22:30 |
| bnemec | Yeah, I pushed a testenv-client patch with a bunch of extra logging: https://review.openstack.org/#/c/410965/ | 22:30 |
| panda | oh | 22:31 |
| bnemec | The fact that this is ha-only is truly bizarre though. | 22:32 |
| EmilienM | http://tripleo.org/cistatus.html | 22:32 |
| EmilienM | numbers talk | 22:32 |
| *** ctayal has joined #tripleo | 22:34 | |
| *** paramite has quit IRC | 22:34 | |
| *** panda is now known as panda|zZ | 22:35 | |
| panda|zZ | and while I'm all talking people are getting things done | 22:36 |
| shinobu_ | if we were to use ceph::profile::params::osd_* via ExtraConfig, basically can we set all osd_* params in ceph.conf ? | 22:39 |
| *** cdearborn has joined #tripleo | 22:40 | |
| EmilienM | shinobu_: yes but that's the the proper way to do | 22:41 |
| EmilienM | shinobu_: a good thing would be to patch THT to add the missing parameters | 22:41 |
| *** pradk has quit IRC | 22:42 | |
| panda|zZ | oh and if someone could approve this so I could study a bit more https://review.openstack.org/410157 https://review.openstack.org/410156 .. thanks | 22:43 |
| *** ayoung is now known as ayoung_dadmode | 22:43 | |
| shinobu_ | EmilienM: thanks. patch in ceph.yaml underneath hieradata then *.pp underneath manifests? | 22:43 |
| EmilienM | shinobu_: yeah | 22:44 |
| shinobu_ | EmilienM: got it! | 22:44 |
| openstackgerrit | Keith Schincke proposed openstack/tripleo-heat-templates: Add support for not using admin_token in Ceph/RGW https://review.openstack.org/405625 | 22:45 |
| *** flepied has quit IRC | 22:49 | |
| *** flepied has joined #tripleo | 22:51 | |
| *** abehl has quit IRC | 22:56 | |
| *** cdearborn has quit IRC | 23:01 | |
| *** rcernin has quit IRC | 23:05 | |
| *** bfournie has quit IRC | 23:06 | |
| *** rlandy has quit IRC | 23:09 | |
| *** ooolpbot has joined #tripleo | 23:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1649742 | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1650005 | 23:10 |
| *** ooolpbot has quit IRC | 23:10 | |
| openstack | Launchpad bug 1649742 in tripleo "postci timeouts on ovb-ha" [Critical,Triaged] | 23:10 |
| openstack | Launchpad bug 1650005 in tripleo "Multinode jobs failing on libvirt issues" [Critical,Triaged] | 23:10 |
| *** trown|outtypewww has quit IRC | 23:12 | |
| *** trown has joined #tripleo | 23:14 | |
| *** migi has quit IRC | 23:17 | |
| *** migi has joined #tripleo | 23:18 | |
| *** yamahata_ has quit IRC | 23:21 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: TEST: Disable net-iso in ha jobs https://review.openstack.org/410998 | 23:35 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: TEST: Disable pacemaker in the updates job https://review.openstack.org/410999 | 23:35 |
| *** chlong has quit IRC | 23:37 | |
| *** bfournie has joined #tripleo | 23:40 | |
| openstackgerrit | Merged openstack/diskimage-builder: Pip install as 10- incompatible with 05-heat-cfntools https://review.openstack.org/410951 | 23:42 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: TEST: Disable pacemaker in the updates job https://review.openstack.org/410999 | 23:44 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: TEST: extra testenv-client logging https://review.openstack.org/410965 | 23:44 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: TEST: Disable net-iso in ha jobs https://review.openstack.org/410998 | 23:44 |
| *** milan has quit IRC | 23:47 | |
| *** dmarlin has quit IRC | 23:58 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
