| *** itlinux has quit IRC | 00:07 | |
| *** ooolpbot has joined #tripleo | 00:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 00:10 |
|---|---|---|
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1724328 | 00:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731032 | 00:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731063 | 00:10 |
| *** ooolpbot has quit IRC | 00:10 | |
| openstack | Launchpad bug 1724328 in tripleo "Netwon to Ocata upgrade failure because of ceilometer-upgrade" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel) | 00:10 |
| openstack | Launchpad bug 1731032 in tripleo "CI: Deployment fails on controller timeout with task creation in heat timing out" [Critical,Triaged] | 00:10 |
| *** pmannidi_2 has joined #tripleo | 00:10 | |
| openstack | Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] | 00:10 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart-extras master: Don't try to start docker for the UC https://review.openstack.org/519441 | 00:14 |
| mwhahaha | EmilienM: after fixing larks's magical code, it kinda still works http://people.redhat.com/aschultz/foo.svg https://github.com/mwhahaha/dotstack | 00:19 |
| * mwhahaha wanders off for now | 00:20 | |
| weshay | hrm.. not good | 00:20 |
| weshay | I think we're seeing the same ssh failure that we see in 001 in 003 | 00:20 |
| weshay | https://review.openstack.org/520574 | 00:20 |
| weshay | http://logs.openstack.org/06/517906/9/gate/legacy-tripleo-ci-centos-7-scenario003-multinode-oooq-container/4e21544/logs/undercloud/home/zuul/tempest_output.log.txt.gz | 00:20 |
| mwhahaha | weshay: yea | 00:20 |
| mwhahaha | weshay: hopefully https://review.openstack.org/#/c/520280/ will help shed some light on the subject | 00:21 |
| weshay | rlandy|rover|afk, fyi.. I updated the query as we discussed earlier | 00:21 |
| weshay | arxcruz, ^ | 00:21 |
| mwhahaha | weshay: since it was a different job, the old query didn't seem to work with it | 00:22 |
| mwhahaha | sorry different tempest test | 00:22 |
| *** dxiri_ has joined #tripleo | 00:22 | |
| weshay | I ran it locally w/ rlandy|rover|afk it picked up 003 as failing 1% of the time | 00:22 |
| mwhahaha | also 004 just failed with it | 00:22 |
| weshay | oh f me | 00:22 |
| *** rlandy|rover|afk is now known as rlandy|rover | 00:23 | |
| weshay | we should talk less often | 00:23 |
| *** bfournie has joined #tripleo | 00:23 | |
| rlandy|rover | weshay: thank you - will review | 00:23 |
| mwhahaha | http://logs.openstack.org/25/519725/4/gate/legacy-tripleo-ci-centos-7-scenario004-multinode-oooq-puppet/d35082b/logs/undercloud/home/zuul/tempest_output.log.txt.gz#_2017-11-17_00_15_12 | 00:23 |
| mwhahaha | though it might be different | 00:24 |
| mwhahaha | haven't looked | 00:24 |
| mwhahaha | that was the top of the gate | 00:24 |
| *** bfournie has quit IRC | 00:24 | |
| weshay | weird.. I didn't get that in my feed | 00:24 |
| mwhahaha | it's still running | 00:24 |
| * mwhahaha uses the status page to spot them early | 00:25 | |
| weshay | bah see it | 00:25 |
| mwhahaha | http://zuulv3.openstack.org/ | 00:25 |
| *** bfournie has joined #tripleo | 00:25 | |
| weshay | ya.. I do a little bit of both | 00:25 |
| weshay | mwhahaha, hrm.. may want to shoot it all in the head to get https://review.openstack.org/#/c/520280/6/puppet/services/neutron-metadata.yaml in | 00:26 |
| *** dxiri has quit IRC | 00:26 | |
| mwhahaha | i gotta go pickup the kid, maybe later tonight | 00:26 |
| weshay | kk no worries | 00:26 |
| mwhahaha | i'll see what it looks like in 3 hours. we probably won't have merged anything anyway | 00:26 |
| weshay | thanks mwhahaha | 00:26 |
| mwhahaha | :( | 00:26 |
| EmilienM | back online | 00:29 |
| *** daidv has quit IRC | 00:30 | |
| EmilienM | mwhahaha: do you remember the vxlan setup error when deploying traas in rdo cloud? | 00:30 |
| EmilienM | I have it now, I can't figure what is it again | 00:30 |
| *** daidv has joined #tripleo | 00:31 | |
| EmilienM | http://paste.openstack.org/show/626564/ | 00:31 |
| EmilienM | ovs not started on overcloud node, and I can successfully start it | 00:33 |
| EmilienM | I think it was related to openvswitch-2.8 | 00:33 |
| EmilienM | ok found it, ovs 2.8 update doesn't restart ovs service | 00:37 |
| *** gbarros has quit IRC | 00:39 | |
| *** jlabarre has quit IRC | 00:55 | |
| *** liverpooler has joined #tripleo | 00:57 | |
| *** dxiri_ has quit IRC | 01:06 | |
| *** dxiri has joined #tripleo | 01:07 | |
| *** ooolpbot has joined #tripleo | 01:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1724328 | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731032 | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731063 | 01:10 |
| *** ooolpbot has quit IRC | 01:10 | |
| openstack | Launchpad bug 1724328 in tripleo "Netwon to Ocata upgrade failure because of ceilometer-upgrade" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel) | 01:10 |
| openstack | Launchpad bug 1731032 in tripleo "CI: Deployment fails on controller timeout with task creation in heat timing out" [Critical,Triaged] | 01:10 |
| openstack | Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] | 01:10 |
| *** dxiri has quit IRC | 01:11 | |
| *** dsneddon has quit IRC | 01:11 | |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates master: Check DeployedServerPortMap https://review.openstack.org/520816 | 01:15 |
| *** ipsecguy has quit IRC | 01:37 | |
| *** chlong_ has joined #tripleo | 01:38 | |
| *** almondjoy has quit IRC | 01:41 | |
| *** chlong has quit IRC | 01:41 | |
| *** cshastri has joined #tripleo | 01:49 | |
| *** dxiri has joined #tripleo | 01:54 | |
| *** dxiri has joined #tripleo | 01:54 | |
| *** dxiri has joined #tripleo | 01:54 | |
| *** ipsecguy has joined #tripleo | 01:56 | |
| *** dxiri has quit IRC | 01:59 | |
| *** chlong__ has joined #tripleo | 01:59 | |
| *** chlong_ has quit IRC | 02:03 | |
| *** ooolpbot has joined #tripleo | 02:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1724328 | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731032 | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731063 | 02:10 |
| *** ooolpbot has quit IRC | 02:10 | |
| openstack | Launchpad bug 1724328 in tripleo "Netwon to Ocata upgrade failure because of ceilometer-upgrade" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel) | 02:10 |
| openstack | Launchpad bug 1731032 in tripleo "CI: Deployment fails on controller timeout with task creation in heat timing out" [Critical,Triaged] | 02:10 |
| openstack | Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] | 02:10 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart-extras master: Don't try to start docker for the UC https://review.openstack.org/519441 | 02:11 |
| *** morazi has joined #tripleo | 02:15 | |
| *** zshi has quit IRC | 02:17 | |
| *** fragatina has quit IRC | 02:30 | |
| *** toure is now known as toure_biab | 02:31 | |
| *** zshi has joined #tripleo | 02:31 | |
| *** ramishra has joined #tripleo | 02:39 | |
| *** yamahata has joined #tripleo | 02:51 | |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: Fix tempest html generation file https://review.openstack.org/517906 | 02:52 |
| *** fzdarsky_ has joined #tripleo | 02:52 | |
| *** itlinux has joined #tripleo | 02:53 | |
| *** fzdarsky has quit IRC | 02:55 | |
| openstackgerrit | Remo Mattei proposed openstack/tripleo-docs master: Adding custom image steps by adding an example of dependency repo. https://review.openstack.org/519456 | 02:56 |
| openstackgerrit | Remo Mattei proposed openstack/tripleo-docs master: Adding custom image steps by adding an example of dependency repo. https://review.openstack.org/519456 | 02:58 |
| mwhahaha | looks like the releasenotes job is broken | 03:00 |
| mwhahaha | https://bugs.launchpad.net/tripleo/+bug/1732815 | 03:02 |
| openstack | Launchpad bug 1732815 in tripleo "CI: releasenotes job broken with Could not import extension oslosphinx (exception: No module named oslosphinx) " [Critical,Triaged] | 03:02 |
| *** links has joined #tripleo | 03:03 | |
| *** stendulker has joined #tripleo | 03:05 | |
| mwhahaha | also different failure but also releasenotes, https://bugs.launchpad.net/tripleo/+bug/1732816 | 03:07 |
| openstack | Launchpad bug 1732816 in tripleo "CI: releasenotes job fails due to ValueError: Unknown reference 'origin/stable/ocata'" [Critical,Triaged] - Assigned to Christian Inzinger (ci) | 03:07 |
| mwhahaha | not sure if completely blocking so i'm not adding alert for now | 03:08 |
| *** ooolpbot has joined #tripleo | 03:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1724328 | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731032 | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731063 | 03:10 |
| *** ooolpbot has quit IRC | 03:10 | |
| openstack | Launchpad bug 1724328 in tripleo "Netwon to Ocata upgrade failure because of ceilometer-upgrade" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel) | 03:10 |
| openstack | Launchpad bug 1731032 in tripleo "CI: Deployment fails on controller timeout with task creation in heat timing out" [Critical,Triaged] | 03:10 |
| openstack | Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] | 03:10 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart-extras master: Don't try to start docker for the UC https://review.openstack.org/519441 | 03:18 |
| *** tzumainn has quit IRC | 03:19 | |
| *** fragatina has joined #tripleo | 03:21 | |
| *** fragatina has quit IRC | 03:25 | |
| *** jrist has quit IRC | 03:29 | |
| *** psachin has joined #tripleo | 03:31 | |
| *** itlinux has quit IRC | 03:36 | |
| *** psahoo has joined #tripleo | 03:42 | |
| *** itlinux has joined #tripleo | 03:45 | |
| *** jrist has joined #tripleo | 03:53 | |
| itlinux | hello has anyone setup regions or cells with OOO? | 03:54 |
| *** morazi has quit IRC | 03:55 | |
| openstackgerrit | Merged openstack/tripleo-common master: Log Ansible actions to ansible.log https://review.openstack.org/514783 | 03:58 |
| openstackgerrit | Merged openstack/tripleo-common master: Add work_dir for some actions https://review.openstack.org/514784 | 03:59 |
| *** gbarros has joined #tripleo | 04:02 | |
| *** gbarros has quit IRC | 04:05 | |
| *** ooolpbot has joined #tripleo | 04:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1724328 | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731032 | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731063 | 04:10 |
| openstack | Launchpad bug 1724328 in tripleo "Netwon to Ocata upgrade failure because of ceilometer-upgrade" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel) | 04:10 |
| *** ooolpbot has quit IRC | 04:10 | |
| openstack | Launchpad bug 1731032 in tripleo "CI: Deployment fails on controller timeout with task creation in heat timing out" [Critical,Triaged] | 04:10 |
| openstack | Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] | 04:10 |
| openstackgerrit | Merged openstack/python-tripleoclient master: Add --config-download https://review.openstack.org/514701 | 04:11 |
| *** shreshtha_ has joined #tripleo | 04:18 | |
| *** dbecker has quit IRC | 04:23 | |
| openstackgerrit | Merged openstack/python-tripleoclient master: Add --output-dir https://review.openstack.org/514789 | 04:23 |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/pike: Add missing host_prep_tasks for log dirs creation https://review.openstack.org/520568 | 04:23 |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/pike: Add missing host_prep_tasks for memcached https://review.openstack.org/520569 | 04:23 |
| openstackgerrit | Merged openstack/tripleo-common master: Truncate the zaqar message to 512 kbytes https://review.openstack.org/518328 | 04:24 |
| *** itlinux has quit IRC | 04:24 | |
| *** ykarel|away has joined #tripleo | 04:25 | |
| *** shreshtha_ has quit IRC | 04:30 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart-extras master: Don't try to start docker for the UC https://review.openstack.org/519441 | 04:31 |
| weshay | mwhahaha, scen003 is dying | 04:32 |
| *** itlinux has joined #tripleo | 04:33 | |
| *** itlinux has quit IRC | 04:33 | |
| *** aditya_r has joined #tripleo | 04:37 | |
| *** dbecker has joined #tripleo | 04:38 | |
| EmilienM | weshay: how that | 04:38 |
| EmilienM | weshay: timeouts and stuff? | 04:38 |
| weshay | tempest | 04:38 |
| EmilienM | damn | 04:38 |
| EmilienM | ssh thing? | 04:38 |
| weshay | yup | 04:38 |
| EmilienM | damn | 04:38 |
| EmilienM | we should make it non voting... | 04:38 |
| EmilienM | that's why we need in-tree layout asap | 04:38 |
| EmilienM | so we can control these things immediatly | 04:39 |
| *** shreshtha has joined #tripleo | 04:39 | |
| weshay | http://logs.openstack.org/57/520057/1/gate/legacy-tripleo-ci-centos-7-scenario003-multinode-oooq-container/8c18a46/logs/undercloud/home/zuul/tempest/tempest.log.txt.gz#_2017-11-17_03_55_51_565 | 04:39 |
| EmilienM | if it's still failing on our morning, let's disable voting on 003 as well | 04:39 |
| EmilienM | so we can merge stuffs | 04:39 |
| EmilienM | carefully I guess | 04:39 |
| weshay | EmilienM, and merge that debug patch on neutron metadata service | 04:40 |
| EmilienM | yea | 04:40 |
| EmilienM | do we have debug btw? let me look | 04:40 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: DNM: Testing debug neutron metadata https://review.openstack.org/519756 | 04:41 |
| EmilienM | it didn't fail | 04:41 |
| EmilienM | trying again ^ | 04:41 |
| EmilienM | I'm out | 04:41 |
| EmilienM | hopefully we can merge the tripleo-ci patch with zuul layout | 04:41 |
| EmilienM | so we can quickly disable 003 voting tomorrow | 04:41 |
| weshay | aight | 04:41 |
| EmilienM | otherwise I'll ping infra first thing on the morning | 04:41 |
| EmilienM | and disable it in project config | 04:41 |
| EmilienM | no big deal | 04:41 |
| EmilienM | ttyl | 04:41 |
| *** daidv has quit IRC | 04:42 | |
| *** daidv has joined #tripleo | 04:42 | |
| *** pmannidi has quit IRC | 04:48 | |
| *** itlinux has joined #tripleo | 04:49 | |
| *** udesale has joined #tripleo | 04:50 | |
| *** dpawar has joined #tripleo | 04:53 | |
| *** pgadiya has joined #tripleo | 05:05 | |
| *** ooolpbot has joined #tripleo | 05:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1724328 | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731032 | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731063 | 05:10 |
| *** ooolpbot has quit IRC | 05:10 | |
| openstack | Launchpad bug 1724328 in tripleo "Netwon to Ocata upgrade failure because of ceilometer-upgrade" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel) | 05:10 |
| openstack | Launchpad bug 1731032 in tripleo "CI: Deployment fails on controller timeout with task creation in heat timing out" [Critical,Triaged] | 05:10 |
| openstack | Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] | 05:10 |
| *** abregman|pto is now known as abregman|afk | 05:12 | |
| *** jaosorior has joined #tripleo | 05:12 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates stable/pike: Enable httpd to request certificates for the "external" network https://review.openstack.org/520893 | 05:12 |
| *** fragatina has joined #tripleo | 05:12 | |
| *** fragatina has quit IRC | 05:14 | |
| *** fragatina has joined #tripleo | 05:15 | |
| *** janki has joined #tripleo | 05:22 | |
| *** itlinux has quit IRC | 05:35 | |
| *** skramaja has joined #tripleo | 05:36 | |
| *** ratailor has joined #tripleo | 05:36 | |
| *** ratailor has quit IRC | 05:36 | |
| *** ratailor has joined #tripleo | 05:36 | |
| *** itlinux has joined #tripleo | 05:38 | |
| *** itlinux has quit IRC | 05:51 | |
| jaosorior | flaper87: hey, saw you were testing out CI for the ansible roles. If you get some time, can you get me an overview of how that works? Also, have you gotten around packaging the roles for RDO? | 05:51 |
| flaper87 | jaosorior: yes and no! Yes I'll happily walk you through it and no I haven't gotten around packaging these roles. I think the packaging will probably start early next year. | 05:52 |
| flaper87 | jaosorior: do you need the packages now? | 05:52 |
| flaper87 | jaosorior: do you want to have a quick call later today? | 05:53 |
| jaosorior | sure, lets do that | 05:53 |
| jaosorior | flaper87: so, I ask because of two things: 1: I want to start trying to stuff out. 2. I need to do the same for the ipsec role (not related to k8s) | 05:53 |
| flaper87 | jaosorior: coolio, shoot me an invite, pref between 10am and 12pm (since we are both in Europe) | 05:53 |
| jaosorior | flaper87: what time is it there? | 05:53 |
| jaosorior | it's almost 8am here. | 05:54 |
| flaper87 | jaosorior: I think we are both in... dafuq? | 05:54 |
| flaper87 | you are 1h ahead of me? | 05:54 |
| jaosorior | flaper87: Finland :D | 05:54 |
| jaosorior | yeah, one hour difference | 05:54 |
| flaper87 | I thought you started your day as early as I did | 05:54 |
| jaosorior | and holy shit you're up early | 05:54 |
| flaper87 | ;) | 05:54 |
| jaosorior | at what time do you start? | 05:54 |
| flaper87 | 6am, or earlier | 05:55 |
| flaper87 | but I'm weird, I just don't sleep much | 05:55 |
| flaper87 | :D | 05:55 |
| jaosorior | flaper87: haha nah, I take my time having breakfast and only start my workday when my gf goes off to work. | 05:55 |
| flaper87 | jaosorior: makes total sense :D | 05:55 |
| jaosorior | then I also take my time preparing my coffee :P | 05:55 |
| flaper87 | jaosorior: ok ok, so, 10am my time? | 05:55 |
| flaper87 | jaosorior: I prepare coffee as soon as I get my feet out of my bed | 05:56 |
| jaosorior | lets do that | 05:56 |
| flaper87 | ok, I start my day at 6:10 'cause I take 10mins to prepare coffee | 05:56 |
| flaper87 | :P | 05:56 |
| jaosorior | haha same here! | 05:56 |
| jaosorior | (regarding the coffee preparation) | 05:56 |
| jaosorior | anyway, 10 am it is (11 here I guess) | 05:56 |
| flaper87 | lol | 05:56 |
| *** marios has joined #tripleo | 05:56 | |
| flaper87 | ttyl then, going to finish something | 05:57 |
| jaosorior | enjoy | 05:57 |
| flaper87 | jaosorior: but seriously, shoot me an invite. I don't mean to be all formal and stuff but I'll forget if I don't have something buzzing my brain (and I sometimes ignore IRC) | 05:59 |
| bandini | morning | 06:00 |
| *** psachin_ has joined #tripleo | 06:00 | |
| jaosorior | flaper87: got the invite? | 06:01 |
| jaosorior | bandini: yoooo | 06:01 |
| *** psachin has quit IRC | 06:01 | |
| openstackgerrit | Merged openstack/tripleo-common stable/pike: Fail ceph-ansible workflow if there is stale puppet-ceph hieradata https://review.openstack.org/520577 | 06:02 |
| bandini | jaosorior: yo! ;) | 06:02 |
| flaper87 | bandini: sssh | 06:03 |
| flaper87 | jaosorior: yes | 06:03 |
| bandini | flaper87: you old fart! | 06:03 |
| flaper87 | bandini: *cough* last time I checked, you were older than me *cough* | 06:04 |
| EmilienM | quick reminder (before I go to bed) please do not approve patches today | 06:05 |
| * flaper87 ties his hands to not start an innappropriate conversation on public channels | 06:05 | |
| EmilienM | sorry for that but we're trying to migrate to zuul v3 layout and need the patches to land | 06:05 |
| flaper87 | EmilienM: roger that, read your email now, gtfo | 06:05 |
| EmilienM | lol | 06:05 |
| EmilienM | flaper87: thx for your update | 06:05 |
| EmilienM | very useful | 06:05 |
| bandini | EmilienM: ack! | 06:05 |
| * EmilienM kidding. So useless. | 06:05 | |
| bandini | lol | 06:05 |
| EmilienM | oh wait it's a public channel | 06:05 |
| flaper87 | EmilienM: no problem, as said, I commit to sending more useless electronic paper updates | 06:05 |
| EmilienM | no it's awesome! thx flavio | 06:05 |
| bandini | flaper87: I don't know what you're talking about | 06:05 |
| EmilienM | flaper87: I go to bed now, in case you cared | 06:06 |
| flaper87 | EmilienM: I don't but hey, sleep tight and dream about failing CI jobs | 06:06 |
| EmilienM | can you sing me a song before? | 06:06 |
| flaper87 | EmilienM: there was a time, when there was CI, then EmilienM touched it and it was all broken | 06:06 |
| flaper87 | EmilienM: do you like that song? | 06:06 |
| EmilienM | flaper87, bandini: hey btw, how is world cup selections going? ahahah. Good night? | 06:07 |
| *** rlandy|rover has quit IRC | 06:07 | |
| bandini | EmilienM: ahahahah | 06:07 |
| EmilienM | k, thx, bye | 06:07 |
| flaper87 | EmilienM: got the wrong guy, dude. I couldn't care less about the world cup | 06:07 |
| flaper87 | :P | 06:07 |
| bandini | same here, although it was hilarious to watch people freak out about it :) | 06:07 |
| flaper87 | bandini: seriously | 06:07 |
| bandini | 0 fucks given | 06:08 |
| flaper87 | FIRST TIME AFTER 60+ YEARS! ARRGH, suicide, shame hats on, etc | 06:08 |
| flaper87 | and I was like eating gelato in front of them... in late november | 06:08 |
| bandini | ahaha yes! | 06:08 |
| flaper87 | anyway, stop distracting me, gotta finish something | 06:08 |
| bandini | dude, it's Beverdi. there is nothing to do but, start drinking in the morning early on | 06:09 |
| flaper87 | bandini: what do you think the Rum bottle on the table is for? | 06:10 |
| *** ooolpbot has joined #tripleo | 06:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1724328 | 06:10 |
| openstack | Launchpad bug 1724328 in tripleo "Netwon to Ocata upgrade failure because of ceilometer-upgrade" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel) | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731032 | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731063 | 06:10 |
| *** ooolpbot has quit IRC | 06:10 | |
| openstack | Launchpad bug 1731032 in tripleo "CI: Deployment fails on controller timeout with task creation in heat timing out" [Critical,Triaged] | 06:10 |
| openstack | Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] | 06:10 |
| *** akane_ has joined #tripleo | 06:11 | |
| *** akane has joined #tripleo | 06:11 | |
| *** jprovazn has joined #tripleo | 06:24 | |
| *** jprovazn has quit IRC | 06:25 | |
| openstackgerrit | Michele Baldessari proposed openstack/puppet-tripleo master: Make sure rabbitmq is fully up before creating any rabbitmq resources https://review.openstack.org/520119 | 06:28 |
| *** pmannidi_2 has quit IRC | 06:47 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/instack-undercloud master: Updated from global requirements https://review.openstack.org/520409 | 06:49 |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: update the tripleo-ui when condition https://review.openstack.org/514757 | 06:50 |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: add a very simple tripleo-ui validation https://review.openstack.org/514794 | 06:50 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/instack-undercloud master: Updated from global requirements https://review.openstack.org/520409 | 06:51 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/tripleo-ui stable/pike: Imported Translations from Zanata https://review.openstack.org/520935 | 06:54 |
| *** suuuper has joined #tripleo | 06:54 | |
| *** aditya_r has quit IRC | 06:56 | |
| openstackgerrit | Attila Darazs proposed openstack/tripleo-quickstart-extras master: GATE CHECK for quickstart-extras https://review.openstack.org/472607 | 07:00 |
| *** pcaruana has joined #tripleo | 07:04 | |
| *** dparkes has joined #tripleo | 07:06 | |
| *** ykarel|away is now known as ykarel|lunch | 07:06 | |
| *** ooolpbot has joined #tripleo | 07:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1724328 | 07:10 |
| openstack | Launchpad bug 1724328 in tripleo "Netwon to Ocata upgrade failure because of ceilometer-upgrade" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel) | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731032 | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731063 | 07:10 |
| *** ooolpbot has quit IRC | 07:10 | |
| openstack | Launchpad bug 1731032 in tripleo "CI: Deployment fails on controller timeout with task creation in heat timing out" [Critical,Triaged] | 07:10 |
| openstack | Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] | 07:10 |
| *** rcernin has quit IRC | 07:18 | |
| *** psachin_ has quit IRC | 07:20 | |
| *** jfrancoa has joined #tripleo | 07:21 | |
| openstackgerrit | Merged openstack/tripleo-validations master: Add per-network bind ips to inventory https://review.openstack.org/517051 | 07:24 |
| *** psachin has joined #tripleo | 07:33 | |
| *** psachin has quit IRC | 07:38 | |
| *** psachin has joined #tripleo | 07:39 | |
| *** florianf has joined #tripleo | 07:53 | |
| *** hjensas has quit IRC | 07:55 | |
| jaosorior | bandini: if you have some time can you check this out https://github.com/ClusterLabs/resource-agents/pull/1058 | 07:57 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/instack-undercloud master: Updated from global requirements https://review.openstack.org/520409 | 08:01 |
| jaosorior | dciabrin_: hey, could you take a look at this? https://github.com/ClusterLabs/resource-agents/pull/1058 | 08:06 |
| cschwede | Good morning! Can someone please review https://review.openstack.org/#/c/516436/ ? It's a critical bug, might be even a blocker in some cases. LP for this is: https://bugs.launchpad.net/tripleo/+bug/1732670 | 08:08 |
| openstack | Launchpad bug 1732670 in tripleo "Containerized deployment fails when using SwiftRawDisk" [Medium,In progress] - Assigned to Christian Schwede (cschwede) | 08:08 |
| *** chem`` has joined #tripleo | 08:10 | |
| *** ooolpbot has joined #tripleo | 08:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1724328 | 08:10 |
| openstack | Launchpad bug 1724328 in tripleo "Netwon to Ocata upgrade failure because of ceilometer-upgrade" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel) | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731032 | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731063 | 08:10 |
| *** ooolpbot has quit IRC | 08:10 | |
| openstack | Launchpad bug 1731032 in tripleo "CI: Deployment fails on controller timeout with task creation in heat timing out" [Critical,Triaged] | 08:10 |
| openstack | Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] | 08:10 |
| *** pgadiya has quit IRC | 08:10 | |
| *** chem` has quit IRC | 08:12 | |
| *** ffiore has joined #tripleo | 08:14 | |
| *** dpawar has quit IRC | 08:15 | |
| *** ebarrera has joined #tripleo | 08:16 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/pike: Allow the configuration of image_member_quota from THT https://review.openstack.org/519309 | 08:22 |
| *** bogdando has joined #tripleo | 08:23 | |
| *** amoralej|off is now known as amoralej | 08:27 | |
| *** pgadiya has joined #tripleo | 08:27 | |
| *** chem`` has quit IRC | 08:28 | |
| flaper87 | jistr: bogdando fwiw, I'm rebasing my openshift patch | 08:32 |
| flaper87 | shardy ^ | 08:32 |
| jaosorior | yay | 08:33 |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Import TripleO legacy zuul jobs https://review.openstack.org/514793 | 08:33 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Drop step_config as top level docker requirement https://review.openstack.org/442716 | 08:33 |
| chandankumar | flaper87: hello May i pm? | 08:33 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Dynamically generate FQDN hiera entries https://review.openstack.org/520082 | 08:33 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Debug configuration for neutron metadata agent https://review.openstack.org/520280 | 08:33 |
| flaper87 | chandankumar: yeah, sure | 08:35 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Add rsyslog-sidecar resource and configuration https://review.openstack.org/519361 | 08:39 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Add option for HAProxy (non-HA) container to log to stdout/stderr https://review.openstack.org/519362 | 08:39 |
| *** pgadiya has quit IRC | 08:40 | |
| *** dparkes has quit IRC | 08:43 | |
| openstackgerrit | Flavio Percoco proposed openstack/tripleo-heat-templates master: Deploy OpenShift using OOO on the overcloud https://review.openstack.org/494470 | 08:46 |
| flaper87 | jaosorior: did a tiny change to our call, hope it works for ya' | 08:47 |
| *** dparkes has joined #tripleo | 08:48 | |
| *** cylopez has joined #tripleo | 08:48 | |
| *** jpena|off is now known as jpena | 08:49 | |
| jaosorior | flaper87: still doesn't reflect in the calendar | 08:49 |
| *** ccamacho has joined #tripleo | 08:49 | |
| *** ffiore has quit IRC | 08:49 | |
| *** ffiore has joined #tripleo | 08:50 | |
| *** jpich has joined #tripleo | 08:51 | |
| jaosorior | flaper87: what change? | 08:51 |
| *** pgadiya has joined #tripleo | 08:52 | |
| *** dparkes has quit IRC | 08:56 | |
| *** cylopez has quit IRC | 08:56 | |
| *** dpawar has joined #tripleo | 08:56 | |
| flaper87 | jaosorior: mde it 30 mins starting at 10:30 | 08:58 |
| flaper87 | does that work? | 08:58 |
| flaper87 | (starting in 30mins) | 08:59 |
| jaosorior | flaper87: sure. I gotta go at 10:50. But I think 20 min is just fine. | 08:59 |
| flaper87 | ++ | 08:59 |
| *** hjensas has joined #tripleo | 09:02 | |
| openstackgerrit | mathieu bultel proposed openstack/tripleo-quickstart-extras master: Add test_mtu_sized_frames to the skip list during investigation https://review.openstack.org/519770 | 09:06 |
| *** cylopez has joined #tripleo | 09:10 | |
| chandankumar | d0ugal: feel free to package tripleo-common tempest plugin for RDO, here is the reference review https://review.rdoproject.org/r/#/q/topic:add-zaqar-tempest-plugin | 09:10 |
| *** ooolpbot has joined #tripleo | 09:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1724328 | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731032 | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731063 | 09:10 |
| *** ooolpbot has quit IRC | 09:10 | |
| openstack | Launchpad bug 1724328 in tripleo "Netwon to Ocata upgrade failure because of ceilometer-upgrade" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel) | 09:10 |
| openstack | Launchpad bug 1731032 in tripleo "CI: Deployment fails on controller timeout with task creation in heat timing out" [Critical,Triaged] | 09:10 |
| openstack | Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] | 09:10 |
| *** dbecker has quit IRC | 09:10 | |
| *** shardy has joined #tripleo | 09:11 | |
| Tengu | jaosorior: "small" question: how does the public endpoint VIP works with a multi-controller containerized infra? I guess it's attached to the node itself - do the service containers have any idea of the VIP location? | 09:12 |
| Tengu | jaosorior: asking for certodia, as it detects if it's running on the master by just checking if the VIP is attached to the node, I'm pretty sure this can't work with containers. | 09:13 |
| *** dparkes has joined #tripleo | 09:15 | |
| *** dparkes has quit IRC | 09:16 | |
| jaosorior | Tengu: the containers use net=host | 09:17 |
| Tengu | ah, cool | 09:17 |
| jaosorior | Tengu: so that bit didn't change. | 09:17 |
| Tengu | so they know the node IPs | 09:17 |
| jaosorior | yep | 09:17 |
| Tengu | so certodia will work with some mounted volume for the HAProxy socket | 09:17 |
| Tengu | there's just the restart of haproxy in order to reload the certificate. | 09:18 |
| Tengu | that might be worked a bit I guess. | 09:18 |
| Tengu | but basically, it will at least renew the certificate correctly. hmm. another volume should be provided in order to get the "horizon" apache vhost root directory, aka /var/www in the non-container env. | 09:19 |
| Tengu | as it's based on the webroot validation. | 09:19 |
| openstackgerrit | mathieu bultel proposed openstack/tripleo-heat-templates master: Add validation task in docker services [Aodh] https://review.openstack.org/491411 | 09:20 |
| *** derekh has joined #tripleo | 09:20 | |
| *** marios has quit IRC | 09:23 | |
| *** dparkes has joined #tripleo | 09:24 | |
| *** dparkes has quit IRC | 09:26 | |
| *** gkadam has joined #tripleo | 09:32 | |
| *** lucas-afk is now known as lucasagomes | 09:33 | |
| *** mrunge_ has joined #tripleo | 09:34 | |
| *** mrunge_ has quit IRC | 09:35 | |
| *** ykarel|lunch is now known as ykarel | 09:38 | |
| jaosorior | Tengu: so the restart/reload will work nicely without containers. But with containers is still a bit problematic | 09:43 |
| jaosorior | I had the same issue with the CRL | 09:43 |
| *** holser has joined #tripleo | 09:43 | |
| jaosorior | though it might work, as dciabrin_ made the container use the systemd wrapper. But I haven't tried it out. | 09:43 |
| jaosorior | brb, lunch | 09:43 |
| *** dparkes has joined #tripleo | 09:46 | |
| *** dparkes has quit IRC | 09:46 | |
| *** dparkes has joined #tripleo | 09:46 | |
| *** chem`` has joined #tripleo | 09:47 | |
| *** holser has quit IRC | 09:48 | |
| *** holser has joined #tripleo | 09:49 | |
| openstackgerrit | Merged openstack/puppet-pacemaker master: zuul: run ovb-ha-oooq https://review.openstack.org/520278 | 09:50 |
| *** chem`` is now known as chem | 09:52 | |
| *** chem has quit IRC | 09:53 | |
| *** chem has joined #tripleo | 09:53 | |
| openstackgerrit | Merged openstack/tripleo-ui stable/pike: Imported Translations from Zanata https://review.openstack.org/520935 | 09:55 |
| Tengu | jaosorior: I can cehck how we did that in our case. I think we bypass the systemd stuff all the way in order to restart the haproxy from a sidekick (in a rancher env). that same one has the haproxy configuration in fact. | 09:56 |
| Tengu | jaosorior: that way, we have two containers: one with the service, and another one that hold the configuration and a reload script - there's also a third one for the metrics. That way, the service is never down, there's a call we can make that mimic the "graceful" of apache. | 09:57 |
| Tengu | jaosorior: both containers (config + service) are basically the same (same base), volumes are cross-mounted, meaning we only have to update the configuration container in case of haproxy config update - and the service is gracefully restarted. no downtime :). | 09:58 |
| Tengu | same can be done for the certificate reload (well, and config update, in fact). though it might be a bit more complicated. | 09:59 |
| shardy | Small docs fix needs another reviewer if anyone has a moment: https://review.openstack.org/#/c/520146/ | 10:02 |
| *** mrch has joined #tripleo | 10:05 | |
| openstackgerrit | Steven Hardy proposed openstack/puppet-tripleo master: Add kubernetes API to haproxy LB configuration https://review.openstack.org/516407 | 10:07 |
| openstackgerrit | Juan Badia Payno proposed openstack/tripleo-heat-templates master: logging: merge fluentd-client and fluentd-base https://review.openstack.org/501457 | 10:09 |
| *** ooolpbot has joined #tripleo | 10:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1724328 | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731032 | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731063 | 10:10 |
| *** ooolpbot has quit IRC | 10:10 | |
| openstack | Launchpad bug 1724328 in tripleo "Netwon to Ocata upgrade failure because of ceilometer-upgrade" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel) | 10:10 |
| openstack | Launchpad bug 1731032 in tripleo "CI: Deployment fails on controller timeout with task creation in heat timing out" [Critical,Triaged] | 10:10 |
| openstack | Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] | 10:10 |
| Tengu | shardy: can't do more than a +1 :/. sorry | 10:15 |
| shardy | Tengu: np :) | 10:16 |
| *** marios has joined #tripleo | 10:23 | |
| flaper87 | shardy: btw, I rebased my openshift patch | 10:24 |
| flaper87 | it took me like 10 mins to do so | 10:24 |
| flaper87 | will keep an eye on it and move it forward | 10:24 |
| *** fragatin_ has joined #tripleo | 10:25 | |
| shardy | flaper87: ack thanks - I'm still getting the haproxy/vip stuff working via kubespray but then will try the openshift patch | 10:25 |
| flaper87 | shardy: awesome, I'll let you know when it's in a state for you to try it. don't want to make you waste time | 10:26 |
| shardy | flaper87: ack sounds good, thanks! | 10:26 |
| *** daidv has quit IRC | 10:26 | |
| *** fragatina has quit IRC | 10:27 | |
| flaper87 | jaosorior: question re keystone-fernet keys | 10:28 |
| flaper87 | how do we manage those in tht? | 10:28 |
| flaper87 | Do we generate them once for all of the controller nodes? | 10:29 |
| flaper87 | or do we generate them on every node? | 10:29 |
| *** cshastri has quit IRC | 10:31 | |
| lyarwood | marios: seeing the following error running the upgrade tasks directly from my ffu playbook, can you remind me where we are adding the - step|int == x line for these? | 10:35 |
| lyarwood | marios: http://paste.openstack.org/show/626595/ | 10:36 |
| openstackgerrit | Julie Pichon proposed openstack/python-tripleoclient master: Fix truncated error message https://review.openstack.org/521057 | 10:37 |
| marios | lyarwood: looking | 10:38 |
| *** dpawar has quit IRC | 10:39 | |
| shardy | flaper87: AFAICS we treat the keys the same as passwords, e.g they're generated in tripleo-common then passed in via parameters, which means they're common to all nodes | 10:39 |
| shardy | (unless we do something magic in puppet) | 10:39 |
| flaper87 | shardy: gotcha, that's what I thought | 10:40 |
| openstackgerrit | Julie Pichon proposed openstack/python-tripleoclient master: Fix truncated error message https://review.openstack.org/521057 | 10:40 |
| marios | lyarwood: yeah we are doing them in tripleo-common config download for P leme get you a link | 10:40 |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common master: Add zaqarclient to TripleOAction https://review.openstack.org/516347 | 10:40 |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common master: Stream the Ansible output to a Zaqar queue https://review.openstack.org/521058 | 10:40 |
| *** athomas has quit IRC | 10:41 | |
| lyarwood | marios: cool thanks | 10:41 |
| marios | lyarwood: but that doesn't look like it failed for the int cast? https://github.com/openstack/tripleo-common/blob/a8caa24f1b84af1ae2fde7946e543b23175c7343/tripleo_common/utils/config.py#L141 | 10:41 |
| lyarwood | marios: it's the ordering of the conditionals, for this step os_net_config_need_upgrade hasn't been created yet | 10:42 |
| lyarwood | marios: so the step check needs to go above | 10:42 |
| lyarwood | marios: the indentation is also off | 10:42 |
| *** shreshtha has quit IRC | 10:44 | |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common master: Stream the Ansible output to a Zaqar queue https://review.openstack.org/521058 | 10:45 |
| *** paramite has quit IRC | 10:48 | |
| *** akane has quit IRC | 10:52 | |
| *** akane_ has quit IRC | 10:52 | |
| *** salmankhan has joined #tripleo | 10:53 | |
| *** dbecker has joined #tripleo | 10:54 | |
| ansiwen | my change depending on a python-tempestconf change is not using the new python-tempestconf | 10:54 |
| ansiwen | http://logs.openstack.org/39/515139/3/check/legacy-tripleo-ci-centos-7-scenario002-multinode-oooq-container/cbeb36f/job-output.txt.gz#_2017-11-16_17_12_25_403063 | 10:55 |
| chandankumar | ansiwen: because these jobs uses rpm | 10:55 |
| ansiwen | chandankumar: but it pulls the changed version, if I understand the logs correctly | 10:56 |
| *** udesale has quit IRC | 10:56 | |
| ansiwen | chandankumar: or is that log line that I just sent unrelated? | 10:56 |
| ansiwen | chandankumar: you can see before, that the changes are pulled at least | 10:57 |
| *** dtantsur|afk is now known as dtantsur | 10:57 | |
| chandankumar | ansiwen: seeing first time | 10:57 |
| *** salmankhan has quit IRC | 10:57 | |
| ansiwen | chandankumar: how else can I test python-tempestconf changes? manually? | 10:57 |
| chandankumar | ansiwen: can you add tempest_format: venv in https://review.openstack.org/#/c/515139 this review? | 10:59 |
| *** shreshtha has joined #tripleo | 11:02 | |
| openstackgerrit | Lee Yarwood proposed openstack/tripleo-common master: config: Always add step conditional first for upgrade_tasks https://review.openstack.org/521060 | 11:02 |
| shardy | jistr: Hey question re the kube*localhost kubespray params, did you test without those, as I need hypercube to bind to the internal API ip not :::6443 or we get a conflict with binding to that port on the vip | 11:04 |
| shardy | I removed them and still see hyperkube bound to all interfaces | 11:04 |
| shardy | trying to figure out the vars needed but wanted to see if you'd already debugged similar | 11:04 |
| *** ykarel is now known as ykarel|afk | 11:06 | |
| *** salmankhan has joined #tripleo | 11:06 | |
| ansiwen | chandankumar: I don't understand, can you explain for stupid people? | 11:09 |
| *** pgadiya has quit IRC | 11:09 | |
| *** ooolpbot has joined #tripleo | 11:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1724328 | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731032 | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731063 | 11:10 |
| *** ooolpbot has quit IRC | 11:10 | |
| openstack | Launchpad bug 1724328 in tripleo "Netwon to Ocata upgrade failure because of ceilometer-upgrade" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel) | 11:10 |
| openstackgerrit | Lee Yarwood proposed openstack/tripleo-common master: config: Always add step conditional first for upgrade_tasks https://review.openstack.org/521060 | 11:10 |
| openstack | Launchpad bug 1731032 in tripleo "CI: Deployment fails on controller timeout with task creation in heat timing out" [Critical,Triaged] | 11:10 |
| openstack | Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] | 11:10 |
| *** pgadiya has joined #tripleo | 11:10 | |
| *** dpawar has joined #tripleo | 11:15 | |
| *** cdearborn has joined #tripleo | 11:15 | |
| *** stendulker has quit IRC | 11:15 | |
| *** dpawar has quit IRC | 11:18 | |
| *** dpawar has joined #tripleo | 11:22 | |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-quickstart master: TEST DON'T MERGE - Enabling EC2-API Tempest tests. https://review.openstack.org/515139 | 11:22 |
| chandankumar | ansiwen: ^^ it might help | 11:22 |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common master: Stream the Ansible output to a Zaqar queue https://review.openstack.org/521058 | 11:23 |
| dpawar | in a containerized overcloud deployment, if i want to enable debug on nova_compute container so that i can get debug log of nova, how can i do that ? | 11:24 |
| dpawar | shardy: ^^ | 11:24 |
| *** ykarel|afk is now known as ykarel | 11:26 | |
| yolanda | hi, question... i'm doing some test on upgrades and i hit https://bugs.launchpad.net/tripleo/+bug/1710685 | 11:27 |
| openstack | Launchpad bug 1710685 in tripleo "Ugly output when registering nodes" [High,Triaged] | 11:27 |
| yolanda | i say it went to fix released, then on triaged | 11:27 |
| shardy | dpawar: the same as pre-containers, e.g set either Debug: true or NovaDebug: true in an environment file, then re-run the deploy command | 11:27 |
| yolanda | what's the status of it, is that a known bug? | 11:27 |
| shardy | dpawar: that should regenerate the nova config with debug enabled and restart the service container | 11:27 |
| shardy | dpawar: you could probably also manually hack the generated config under /var/lib/config-data and restart the service container manually, but obviously any subsequent deploy via heat will overwrite that | 11:29 |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common master: Stream the Ansible output to a Zaqar queue https://review.openstack.org/521058 | 11:30 |
| dpawar | shardy: in the second option, with that be a persistent change ? i mean across restarts of containers | 11:31 |
| shardy | dpawar: no, as I said the next time you do any kind of update your changes will be erased | 11:33 |
| shardy | so the heat param approach is probably best, even though it'll take slightly longer to apply | 11:33 |
| *** shreshtha has quit IRC | 11:35 | |
| dpawar | right | 11:35 |
| *** pgadiya has quit IRC | 11:36 | |
| *** pradk has quit IRC | 11:39 | |
| openstackgerrit | Julie Pichon proposed openstack/python-tripleoclient master: Mock creation of ~/.tripleo directory https://review.openstack.org/521065 | 11:41 |
| dpawar | shardy: also we add veritas entry in nova.conf in nova_compute, eg enabled_filters. is there any way to do that ? like for block-storage/cinder we already have hooks for adding veritas entries in cinder.conf, in pre-containers veritas installer used to add this entry | 11:42 |
| Tengu | duh. I think certodia is almost finished. Just have to finish the "custodia" library, and create a bunch of tests for the libs… once it's done, let's encrypt certificate will be supported in a really nice way for openstack public endpoints \o/ | 11:43 |
| shardy | dpawar: there is a NovaSchedulerDefaultFilters parameter, which should work exactly the same as pre-containers | 11:43 |
| shardy | dpawar: because we still use puppet to generate the config files, all the pre-containers parameters and hiera overrides should still work exactly the same as before | 11:44 |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common master: Stream the Ansible output to a Zaqar queue https://review.openstack.org/521058 | 11:53 |
| *** ratailor has quit IRC | 11:54 | |
| dpawar | shardy: one more - veritas binaries reside on host. where as since now nova volume service is now containerized, 1. veritas extension of nova driver (opensourced) expects a veritas_cli to be present inside container. 2. veritas_cli needs to communicate services running on host (outside container). is there any way to achieve that ? | 11:54 |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common master: Stream the Ansible output to a Zaqar queue https://review.openstack.org/521058 | 11:54 |
| dpawar | i mean how to achieve 1 and then 2 ? | 11:54 |
| *** hjensas has quit IRC | 11:55 | |
| *** yamahata has quit IRC | 11:55 | |
| *** rhallisey has joined #tripleo | 12:00 | |
| shardy | dpawar: if the tools are packaged and open source we can probably add them to the nova volume container image, if not you may need to rebuild the container image to add those tools | 12:01 |
| shardy | dpawar: The normal pattern for bootstrapping tasks isn't to run them inside the service container, but instead we launch temporary containers (by default using the same image) that run either puppet or CLI commands | 12:02 |
| shardy | dpawar: what services on the host does the tool connect to? | 12:02 |
| *** raildo has joined #tripleo | 12:05 | |
| *** tvignaud has quit IRC | 12:05 | |
| *** tvignaud has joined #tripleo | 12:05 | |
| janki | Hi. Can I get some reviews on https://review.openstack.org/#/c/520573/ | 12:08 |
| dpawar | shardy: 1. tool is not opensourced. the nova driver is opensourced. pre-containers because the veritas_cli was present on host (/usr/bin/) and nova driver was also no host hence there was no issue. 2. the vertas_cli communicates with rabbitmq server and requires certain python libraries | 12:08 |
| *** ramishra has quit IRC | 12:08 | |
| *** dbecker has quit IRC | 12:08 | |
| jaosorior | flaper87: once for all the controller nodes. | 12:09 |
| jaosorior | flaper87: so we generate those via tripleo-common; and then inject them to all the controllers. They have to be the same. | 12:09 |
| dpawar | veritas_cli requires mq libraries as well as veritas libraries | 12:09 |
| shardy | dpawar: ack, it's kind of weird to have an open source drive which depends on a proprietary tool IMO | 12:10 |
| shardy | driver | 12:10 |
| *** ooolpbot has joined #tripleo | 12:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1724328 | 12:10 |
| openstack | Launchpad bug 1724328 in tripleo "Netwon to Ocata upgrade failure because of ceilometer-upgrade" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel) | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731032 | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731063 | 12:10 |
| *** ooolpbot has quit IRC | 12:10 | |
| openstack | Launchpad bug 1731032 in tripleo "CI: Deployment fails on controller timeout with task creation in heat timing out" [Critical,Triaged] | 12:10 |
| openstack | Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] | 12:10 |
| jaosorior | shardy: hey, if you get some time could you take another look at this https://review.openstack.org/#/c/520346/ ? and of course, if you think we should do another approach, I would really appreciate the feedback. | 12:10 |
| shardy | dpawar: would it be possible to build a container image with only the veritas cli and it's deps? | 12:10 |
| *** psahoo has quit IRC | 12:10 | |
| shardy | jaosorior: ack will do | 12:11 |
| *** janki has quit IRC | 12:11 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Set host name explicitly for telemetry https://review.openstack.org/509199 | 12:11 |
| *** dtantsur is now known as dtantsur|brb | 12:12 | |
| dpawar | shardy: do you mean, take existing nova compute image and rebuilt it with required cli and deps and use it at the time of overcloud deploy? will that be acceptable ? | 12:13 |
| *** udesale has joined #tripleo | 12:13 | |
| shardy | dpawar: for upstream, no, but we'd have to have a discussion about the distribution/support model for downstream builds | 12:14 |
| ansiwen | chandankumar: I see, thanks! | 12:14 |
| shardy | dpawar: thats's why I wondered if it's possible to decouple the driver part from the CLI part | 12:14 |
| shardy | dpawar: is the CLI used for setup/bootstrapping or does the driver actually call it? | 12:14 |
| dpawar | shardy: drives actually calls CLI while connecting to volume | 12:15 |
| dpawar | shardy: AFAIK, every third party driver calls CLI for this | 12:16 |
| shardy | dpawar: Ok, I guess rebuilding the nova image is the only option then - the TripleO interfaces enable that, you'd just pass a different DockerCinderVolumeImage or whatever to consume the modified image | 12:17 |
| shardy | obviously maintaining that image forever is a maintenance burden tho | 12:17 |
| shardy | dpawar: how did you install the proprietary bits pre-containers, virt-customize of the overcloud-full image? | 12:18 |
| dpawar | shardy: what if we patch and commit the running container ? | 12:18 |
| shardy | dpawar: thanks kind of a container anti-pattern, and we don't have any interfaces that enable it atm | 12:19 |
| shardy | mandre: ^^ any thoughts on this, does kolla have any existing patterns to handle vendor plugins? | 12:20 |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates master: Check DeployedServerPortMap https://review.openstack.org/520816 | 12:21 |
| mandre | shardy, dpawar: I need to take kids to school, will be back in a snap | 12:21 |
| shardy | mandre: ack, thanks | 12:22 |
| *** shardy is now known as shardy_lunch | 12:22 | |
| dpawar | shardy: for install of prorietory bits pre-containers we used overcloud deploy command that calls veritas_puppet modules which do install and configure veritas hyperscale. | 12:23 |
| dpawar | shardy_lunch: for install of prorietory bits pre-containers we used overcloud deploy command that calls veritas_puppet modules which do install and configure veritas hyperscale. | 12:23 |
| *** suuuper has quit IRC | 12:24 | |
| *** dpawar has quit IRC | 12:30 | |
| *** lucasagomes is now known as lucas-hungry | 12:33 | |
| *** abishop has joined #tripleo | 12:36 | |
| *** social has joined #tripleo | 12:38 | |
| *** apetrich has quit IRC | 12:41 | |
| openstackgerrit | Wojciech Dec proposed openstack/puppet-tripleo master: Adding manifest for Cisco VTS ML2 mechanism driver configuration https://review.openstack.org/510645 | 12:42 |
| *** salmankhan has quit IRC | 12:45 | |
| *** nguyentrihai has joined #tripleo | 12:45 | |
| *** jlabarre has joined #tripleo | 12:46 | |
| *** akane has joined #tripleo | 12:47 | |
| *** akane_ has joined #tripleo | 12:47 | |
| *** thrash|g0ne is now known as thrash | 12:47 | |
| *** dpawar has joined #tripleo | 12:47 | |
| *** jpena is now known as jpena|lunch | 12:50 | |
| *** salmankhan has joined #tripleo | 12:51 | |
| *** pradk has joined #tripleo | 12:55 | |
| *** salmankhan has quit IRC | 12:56 | |
| *** salmankhan has joined #tripleo | 12:57 | |
| *** akane has quit IRC | 12:59 | |
| *** akane_ has quit IRC | 12:59 | |
| *** ramishra has joined #tripleo | 13:00 | |
| *** salmankhan has quit IRC | 13:01 | |
| *** psachin has quit IRC | 13:01 | |
| mandre | dpawar: so generally, if you need to include 3rd party packages to a kolla image, you do this by using the kolla customization mechanism to build customized images | 13:03 |
| mandre | dpawar: https://docs.openstack.org/kolla/latest/admin/image-building.html#dockerfile-customisation | 13:03 |
| *** dtantsur|brb is now known as dtantsur | 13:03 | |
| jaosorior | flaper87, jistr: At what point in the deployment does the external_deploy_tasks stuff gets ran? | 13:03 |
| jaosorior | is it part of the regular steps? | 13:03 |
| jaosorior | flaper87, jistr: We're looking, with jkilpatr to use that in order to deploy a containerized rsyslog. | 13:04 |
| jaosorior | with ansible. | 13:04 |
| mandre | dpawar: however, in the tripleo/osp case were we build the base images, you'll need to add a new layer to the existing images | 13:05 |
| mandre | dpawar: https://docs.openstack.org/tripleo-docs/latest/install/containers_deployment/3rd_party.html | 13:05 |
| *** ffiore has quit IRC | 13:05 | |
| openstackgerrit | Oliver Walsh proposed openstack/puppet-tripleo master: Configure libvirt SASL SCRAM-SHA1 when TLS is enabled https://review.openstack.org/520253 | 13:05 |
| *** cshastri has joined #tripleo | 13:05 | |
| *** skramaja has quit IRC | 13:09 | |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates stable/pike: Set host name explicitly for telemetry https://review.openstack.org/521082 | 13:09 |
| openstackgerrit | Oliver Walsh proposed openstack/tripleo-heat-templates master: Re-enable libvirt TLS with SCRAM SHA-1 auth https://review.openstack.org/520254 | 13:09 |
| *** ooolpbot has joined #tripleo | 13:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1724328 | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731032 | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731063 | 13:10 |
| *** ooolpbot has quit IRC | 13:10 | |
| openstack | Launchpad bug 1724328 in tripleo "Netwon to Ocata upgrade failure because of ceilometer-upgrade" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel) | 13:10 |
| openstack | Launchpad bug 1731032 in tripleo "CI: Deployment fails on controller timeout with task creation in heat timing out" [Critical,Triaged] | 13:10 |
| openstack | Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] | 13:10 |
| *** dprince has joined #tripleo | 13:10 | |
| *** atoth has quit IRC | 13:10 | |
| *** ecerquei has joined #tripleo | 13:13 | |
| *** leitan has joined #tripleo | 13:15 | |
| *** dhill_ has quit IRC | 13:15 | |
| *** trown|outtypewww is now known as trown | 13:17 | |
| *** ramishra has quit IRC | 13:17 | |
| *** chlong__ has quit IRC | 13:18 | |
| *** atoth has joined #tripleo | 13:18 | |
| *** shardy_lunch is now known as shardy | 13:18 | |
| *** akane has joined #tripleo | 13:19 | |
| *** akane_ has joined #tripleo | 13:19 | |
| *** links has quit IRC | 13:19 | |
| *** holser has quit IRC | 13:19 | |
| *** toure_biab is now known as toure | 13:20 | |
| *** lblanchard has joined #tripleo | 13:21 | |
| jaosorior | jkilpatr: but yeah, more or less it would look very similar to this: https://github.com/openstack/tripleo-heat-templates/blob/master/extraconfig/services/kubernetes-master.yaml#L58 | 13:22 |
| *** ramishra has joined #tripleo | 13:22 | |
| openstackgerrit | Flavio Percoco proposed openstack/tripleo-quickstart master: Add featureset33 for scenario009 https://review.openstack.org/494418 | 13:22 |
| *** holser has joined #tripleo | 13:22 | |
| dpawar | mandre: Thanks! will go through these docs and get back to you if requred. | 13:22 |
| dpawar | shardy: Thanks! | 13:22 |
| openstackgerrit | Flavio Percoco proposed openstack/tripleo-heat-templates master: Deploy OpenShift using OOO on the overcloud https://review.openstack.org/494470 | 13:23 |
| jaosorior | jkilpatr: except that we would let tripleo handle the docker container creation, like this https://github.com/openstack/tripleo-heat-templates/blob/master/docker/services/collectd.yaml#L91 | 13:23 |
| *** holser has quit IRC | 13:24 | |
| *** holser has joined #tripleo | 13:25 | |
| mandre | dpawar: anytime! | 13:26 |
| *** rlandy has joined #tripleo | 13:29 | |
| *** rlandy is now known as rlandy|rover | 13:29 | |
| *** janki has joined #tripleo | 13:30 | |
| *** lucas-hungry is now known as lucasagomes | 13:30 | |
| *** apetrich has joined #tripleo | 13:31 | |
| *** athomas has joined #tripleo | 13:34 | |
| *** ffiore has joined #tripleo | 13:34 | |
| jaosorior | shardy: could you revisit this https://review.openstack.org/#/c/520333/ ? | 13:38 |
| *** eck` is now known as eck`gone | 13:39 | |
| shardy | jaosorior: done | 13:39 |
| jaosorior | shardy: thanks | 13:40 |
| *** cshastri has quit IRC | 13:42 | |
| *** morazi has joined #tripleo | 13:45 | |
| *** aditya_r has joined #tripleo | 13:46 | |
| *** salmankhan has joined #tripleo | 13:46 | |
| *** jpena|lunch is now known as jpena | 13:52 | |
| *** artom has quit IRC | 13:58 | |
| *** jmelvin has joined #tripleo | 14:02 | |
| *** dtantsur is now known as dtantsur|brb | 14:03 | |
| *** jkilpatr has quit IRC | 14:03 | |
| *** zshi has quit IRC | 14:06 | |
| openstackgerrit | Dan Prince proposed openstack/tripleo-quickstart-extras master: Don't try to start docker for the UC https://review.openstack.org/519441 | 14:08 |
| *** lblanchard has quit IRC | 14:09 | |
| *** dbecker has joined #tripleo | 14:10 | |
| *** ooolpbot has joined #tripleo | 14:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1724328 | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731032 | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731063 | 14:10 |
| *** ooolpbot has quit IRC | 14:10 | |
| openstack | Launchpad bug 1724328 in tripleo "Netwon to Ocata upgrade failure because of ceilometer-upgrade" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel) | 14:10 |
| openstack | Launchpad bug 1731032 in tripleo "CI: Deployment fails on controller timeout with task creation in heat timing out" [Critical,Triaged] | 14:10 |
| openstack | Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] | 14:10 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates master: docker: Add unit tests on service_name https://review.openstack.org/442755 | 14:11 |
| *** almondjoy has joined #tripleo | 14:11 | |
| *** beagles has joined #tripleo | 14:13 | |
| *** chandankumar is now known as chkumar|somewher | 14:13 | |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common master: Stream the Ansible output to a Zaqar queue https://review.openstack.org/521058 | 14:14 |
| *** tzumainn has joined #tripleo | 14:14 | |
| *** ykarel is now known as ykarel|away | 14:14 | |
| *** jkilpatr has joined #tripleo | 14:16 | |
| openstackgerrit | mathieu bultel proposed openstack/tripleo-common master: Update ceph-ansible playbook path in parameter default for update https://review.openstack.org/515442 | 14:16 |
| pradk | can i request some review on this backport https://review.openstack.org/#/c/521082/ | 14:17 |
| *** rfolco is now known as rfolco_lunch | 14:17 | |
| *** salmankhan has quit IRC | 14:18 | |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common master: Stream the Ansible output to a Zaqar queue https://review.openstack.org/521058 | 14:18 |
| EmilienM | hi | 14:19 |
| *** salmankhan has joined #tripleo | 14:20 | |
| openstackgerrit | mathieu bultel proposed openstack/tripleo-common master: Stream the Ansible output to a Zaqar queue https://review.openstack.org/521058 | 14:20 |
| *** ykarel|away has quit IRC | 14:20 | |
| jaosorior | matbu_: could you check this out https://review.openstack.org/#/c/520333/ ? | 14:23 |
| matbu_ | jaosorior: sure | 14:25 |
| *** dxiri has joined #tripleo | 14:25 | |
| *** eck`gone is now known as eck` | 14:25 | |
| chem | matbu_: when using master-tripleo-ci as a release I've got an consistant problem with openstack-ceilometer-api package during undercloud install is that known ? | 14:26 |
| *** gbarros has joined #tripleo | 14:27 | |
| *** salmankhan has quit IRC | 14:28 | |
| EmilienM | mwhahaha: gate is ok'ish | 14:29 |
| EmilienM | mwhahaha: we could start landing the zuul v3 things | 14:29 |
| EmilienM | wdyt? | 14:29 |
| EmilienM | it's 5h now | 14:29 |
| d0ugal | matbu_: IIUC doing the queue post after the loop will send the full stdout in one message when the process finishes. | 14:29 |
| matbu_ | chem: hmm i wasn't aware of | 14:29 |
| matbu_ | d0ugal: yep but is it a pb ? | 14:29 |
| matbu_ | d0ugal: if the message is too big maybe | 14:30 |
| d0ugal | matbu_: but I thought the point was to have realtime output while the process was still running? | 14:30 |
| matbu_ | d0ugal: yep true, so sorry about the change, anyway the post() wasn't correct | 14:31 |
| *** udesale has quit IRC | 14:31 | |
| matbu_ | d0ugal: but i will still working on it | 14:31 |
| matbu_ | folks upgrade squad mtg | 14:31 |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common master: Stream the Ansible output to a Zaqar queue https://review.openstack.org/521058 | 14:32 |
| d0ugal | matbu_: np, I just fixed an issue I spotted ^ but I'll leave you to work on it now | 14:32 |
| d0ugal | matbu_: let me know if I can help. | 14:32 |
| *** dxiri has quit IRC | 14:33 | |
| *** ramishra has quit IRC | 14:34 | |
| *** anilvenkata has quit IRC | 14:34 | |
| *** ramishra has joined #tripleo | 14:35 | |
| weshay | pradk, thanks for jumping into that rdo-cloud escalation | 14:36 |
| weshay | pradk++ | 14:36 |
| *** karthiks has quit IRC | 14:36 | |
| jaosorior | chem: do you know how to fix that part? | 14:38 |
| jaosorior | chem: of the VIP generation? | 14:38 |
| *** chem is now known as chem_mtg | 14:42 | |
| chem_mtg | jaosorior: in the link [2] there is a example on how to do that | 14:42 |
| chem_mtg | jaosorior: basically you check if compat_name is defined or else fall back to name | 14:43 |
| chem_mtg | jaosorior: makes sense ? | 14:43 |
| jaosorior | kinda | 14:44 |
| jaosorior | ish | 14:44 |
| mwhahaha | EmilienM: I want to see a few of these things merge first. also check to see if there are any important patches we're missing | 14:47 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Dynamically generate VIP hiera entries https://review.openstack.org/520333 | 14:47 |
| jaosorior | chem_mtg: is this better https://review.openstack.org/#/c/520333/3/puppet/all-nodes-config.j2.yaml ? | 14:47 |
| *** salmankhan has joined #tripleo | 14:48 | |
| openstackgerrit | Ryan Brady proposed openstack/tripleo-common master: Validate roles data and network data https://review.openstack.org/508567 | 14:49 |
| openstackgerrit | Ryan Brady proposed openstack/tripleo-common master: Add a Get Networks workflow https://review.openstack.org/509419 | 14:49 |
| openstackgerrit | Ryan Brady proposed openstack/tripleo-common master: Add UpdateNetworks workflow https://review.openstack.org/513463 | 14:49 |
| openstackgerrit | Ryan Brady proposed openstack/tripleo-common master: Adds Workflow to List Available Roles https://review.openstack.org/509832 | 14:49 |
| openstackgerrit | Ryan Brady proposed openstack/tripleo-common master: [WIP] Adds UpdateRoles Workflow https://review.openstack.org/515045 | 14:49 |
| EmilienM | mwhahaha: ok | 14:51 |
| EmilienM | matbu_: fyi, we asked to not approve patches on the ML | 14:51 |
| EmilienM | chem_mtg: same ^ | 14:52 |
| trown | itching to use those new powers :P | 14:52 |
| trown | congrats chem_mtg and mwhahaha | 14:52 |
| chem_mtg | EmilienM: ack | 14:52 |
| trown | err | 14:52 |
| trown | matbu_ | 14:52 |
| mwhahaha | i see how it is | 14:52 |
| chem_mtg | jaosorior: looking again | 14:53 |
| EmilienM | trown: "with great powers come..." I forgot the rest :P | 14:53 |
| mwhahaha | in the mean time, it looks like ovs problems http://logs.openstack.org/70/519770/3/gate/legacy-tripleo-ci-centos-7-scenario003-multinode-oooq-container/1db4d85/logs/subnode-2/var/log/containers/neutron/neutron-l3-agent.log.txt.gz#_2017-11-17_07_48_14_011 | 14:53 |
| EmilienM | mwhahaha: OVS 2.8? | 14:53 |
| mwhahaha | no idea | 14:53 |
| EmilienM | numans: are you around? | 14:53 |
| EmilienM | mwhahaha: is that your debug patch ^? | 14:53 |
| mwhahaha | yes 2.8.1 | 14:54 |
| mwhahaha | no that was the test_mtu_size_frames patch https://review.openstack.org/#/c/519770/ | 14:54 |
| EmilienM | ok | 14:54 |
| mwhahaha | but that's the 2nd one i've seen fail on scenario003 with errors in there | 14:54 |
| EmilienM | mwhahaha: can you confirm the same neutron-l3 error in scenario001 in previous days | 14:55 |
| EmilienM | ? | 14:55 |
| mwhahaha | it is not | 14:55 |
| EmilienM | :( | 14:55 |
| mwhahaha | well same general failure | 14:55 |
| mwhahaha | but i think different cause | 14:55 |
| mwhahaha | metadata agent does have a connection problem to rabbitmq | 14:55 |
| mwhahaha | but it's ealier than the test | 14:55 |
| mwhahaha | so not sure if related | 14:55 |
| *** akrivoka has joined #tripleo | 14:56 | |
| mwhahaha | here's the other ovs one http://logs.openstack.org/56/519756/4/check/legacy-tripleo-ci-centos-7-scenario003-multinode-oooq-container/1b81416/logs/subnode-2/var/log/containers/neutron/neutron-l3-agent.log.txt.gz#_2017-11-17_06_18_16_923 | 14:56 |
| mwhahaha | that was my DMN test patch of my debug flag | 14:56 |
| *** dxiri has joined #tripleo | 14:57 | |
| mwhahaha | both had http://logs.openstack.org/56/519756/4/check/legacy-tripleo-ci-centos-7-scenario003-multinode-oooq-container/1b81416/logs/subnode-2/var/log/containers/neutron/neutron-metadata-agent.log.txt.gz#_2017-11-17_06_15_10_695 | 14:57 |
| mwhahaha | which is also concerning | 14:57 |
| *** ratailor has joined #tripleo | 14:58 | |
| chem_mtg | jaosorior: I've just checked the newton and there was a change only with *InternalNetwork*, not with *InternalApiNetMap* so my comment was irrelevant | 14:58 |
| jaosorior | chem_mtg: oh | 14:58 |
| *** yamahata has joined #tripleo | 14:58 | |
| jaosorior | chem_mtg: I'll revert to the previous version then | 14:58 |
| chem_mtg | jaosorior: sorry about that, so yeap revert. | 14:59 |
| *** psachin has joined #tripleo | 14:59 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Dynamically generate VIP hiera entries https://review.openstack.org/520333 | 15:00 |
| mwhahaha | EmilienM: speaking of broken things, did you see my note about the release notes jobs being broken last night https://bugs.launchpad.net/tripleo/+bug/1732815 | 15:00 |
| openstack | Launchpad bug 1732815 in tripleo "CI: releasenotes job broken with Could not import extension oslosphinx (exception: No module named oslosphinx) " [Critical,Triaged] | 15:00 |
| jaosorior | chem_mtg: done ^^ | 15:00 |
| *** janki has quit IRC | 15:00 | |
| mwhahaha | EmilienM: and https://bugs.launchpad.net/tripleo/+bug/1732816 | 15:00 |
| openstack | Launchpad bug 1732816 in tripleo "CI: releasenotes job fails due to ValueError: Unknown reference 'origin/stable/ocata'" [Critical,Triaged] | 15:00 |
| EmilienM | I didn't | 15:01 |
| EmilienM | mwhahaha: have you talked with Doug? | 15:01 |
| *** chlong__ has joined #tripleo | 15:01 | |
| EmilienM | I don't see why it fails now | 15:01 |
| mwhahaha | not yet | 15:01 |
| EmilienM | maybe something changed outside of our control | 15:02 |
| *** dtantsur|brb is now known as dtantsur | 15:02 | |
| *** artom has joined #tripleo | 15:04 | |
| ratailor | https://review.openstack.org/#/c/520273/ | 15:04 |
| ratailor | https://review.openstack.org/#/c/520274/ | 15:04 |
| ratailor | EmilienM, mwhahaha could you please set workflow for these ^^ patches, these needed to be merged before OSP12 GA. | 15:04 |
| EmilienM | ratailor: see mailing-list, we don't approve patches today. We'll probably approve it over the weekend. | 15:05 |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates master: Add S3 storage driver as an option and its config https://review.openstack.org/521093 | 15:05 |
| ratailor | EmilienM, sure, no problem. | 15:05 |
| openstackgerrit | Beth Elwell proposed openstack/tripleo-heat-templates stable/pike: Disabled fwaas plugin for horizon https://review.openstack.org/521099 | 15:08 |
| openstackgerrit | Dmitry Tantsur proposed openstack/instack-undercloud master: [DO NOT MERGE] Just testing, nothing to see here https://review.openstack.org/521100 | 15:09 |
| *** artom_ has joined #tripleo | 15:09 | |
| openstackgerrit | Steven Hardy proposed openstack/puppet-tripleo master: Add kubernetes API to haproxy LB configuration https://review.openstack.org/516407 | 15:10 |
| *** ooolpbot has joined #tripleo | 15:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1724328 | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731032 | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731063 | 15:10 |
| *** ooolpbot has quit IRC | 15:10 | |
| openstack | Launchpad bug 1724328 in tripleo "Netwon to Ocata upgrade failure because of ceilometer-upgrade" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel) | 15:10 |
| EmilienM | mwhahaha: interesting to see only scenarios that are unstable | 15:10 |
| openstack | Launchpad bug 1731032 in tripleo "CI: Deployment fails on controller timeout with task creation in heat timing out" [Critical,Triaged] | 15:10 |
| openstack | Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] | 15:10 |
| EmilienM | mwhahaha: and not other classic multinode jobs | 15:10 |
| *** dbecker has quit IRC | 15:10 | |
| weshay | EmilienM, yup | 15:10 |
| mwhahaha | EmilienM: do classic run tempest? | 15:10 |
| *** dbecker has joined #tripleo | 15:10 | |
| EmilienM | mwhahaha: yep, afik | 15:10 |
| EmilienM | unless it changed | 15:10 |
| EmilienM | mwhahaha: I think gate is about to reset | 15:11 |
| EmilienM | mwhahaha: see first job on top, http://zuulv3.openstack.org/static/stream.html?uuid=cae58e56b98c4cab83a9effe287e0f1b&logfile=console.log | 15:11 |
| *** udesale has joined #tripleo | 15:11 | |
| mwhahaha | meh | 15:11 |
| mwhahaha | i'm approving the v3 patches to get them in now | 15:11 |
| EmilienM | k | 15:11 |
| weshay | mwhahaha, yes.. it runs tempest http://logs.openstack.org/84/514784/10/gate/legacy-tripleo-ci-centos-7-containers-multinode/617922a/logs/stackviz/#/stdin | 15:12 |
| *** udesale has quit IRC | 15:12 | |
| *** artom has quit IRC | 15:12 | |
| weshay | http://logs.openstack.org/84/514784/10/gate/legacy-tripleo-ci-centos-7-containers-multinode/617922a/logs/stackviz/#/stdin/timeline?test=tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern.test_volume_boot_pattern | 15:12 |
| openstackgerrit | Alan Bishop proposed openstack/tripleo-heat-templates stable/pike: Update Iscsid puppet_config to reset host's IQN https://review.openstack.org/521102 | 15:13 |
| mwhahaha | EmilienM: i hope the project-config patch doesn't hang on https://review.openstack.org/#/q/I34113054a2f9ca4c9aa7e7b1ad3bb8dcc76523f6 since one of those is abanonded | 15:14 |
| mwhahaha | I hope gerrit isn't going to be dumb | 15:14 |
| mwhahaha | anyway all the v3 patches are queued | 15:14 |
| EmilienM | mwhahaha: we're talking about the reno errors on #openstack-meeting | 15:14 |
| EmilienM | mwhahaha: ok, I'll remove one depends-on and make infra re-approve it | 15:15 |
| *** chem has joined #tripleo | 15:15 | |
| mwhahaha | EmilienM: no let's just wait | 15:15 |
| mwhahaha | i think it should be ok | 15:15 |
| EmilienM | I don't think so | 15:15 |
| EmilienM | in my experience, all depends-on need to be merged | 15:16 |
| *** chem_mtg has quit IRC | 15:16 | |
| *** rmascena has joined #tripleo | 15:16 | |
| weshay | arxcruz, are we pushing https://review.openstack.org/#/c/518723/ along? I'd like to see some progress on the tempest emails. Do you need help or to discuss? | 15:16 |
| arxcruz | weshay: yes, I'll update the patch | 15:17 |
| openstackgerrit | Martin André proposed openstack/tripleo-heat-templates master: Bind mount /etc/fstab to generate glance-api config https://review.openstack.org/521103 | 15:18 |
| *** aditya_ra has joined #tripleo | 15:18 | |
| jaosorior | mandre: commented ^^ | 15:18 |
| *** raildo has quit IRC | 15:19 | |
| *** dxiri has quit IRC | 15:19 | |
| *** jaosorior has quit IRC | 15:19 | |
| *** aditya_r has quit IRC | 15:21 | |
| *** bobh has joined #tripleo | 15:22 | |
| mwhahaha | EmilienM: so going back to the classic vs scenarios. It does seem like it's a resource problem | 15:24 |
| mwhahaha | weshay: did we ever fix the missing dstat? | 15:24 |
| *** aditya_ra has quit IRC | 15:24 | |
| weshay | mwhahaha, aye.. rlandy did | 15:25 |
| openstackgerrit | Oliver Walsh proposed openstack/puppet-tripleo master: Configure libvirt SASL SCRAM-SHA1 when TLS is enabled https://review.openstack.org/520253 | 15:25 |
| *** ramishra has quit IRC | 15:28 | |
| openstackgerrit | Ronelle Landy proposed openstack-infra/tripleo-ci master: Add MTU adjustment parameters to multinode settings https://review.openstack.org/521112 | 15:30 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart-extras master: dns from libvirthost should be off for ovb https://review.openstack.org/521113 | 15:30 |
| *** ratailor has quit IRC | 15:31 | |
| *** itlinux has joined #tripleo | 15:31 | |
| rlandy|rover | mwhahaha: https://review.openstack.org/#/c/519725/ - dstat fix | 15:35 |
| mwhahaha | rlandy|rover: k i'll recheck it after the v3 stuff lands. thanks | 15:35 |
| rlandy|rover | mwhahaha: thanks | 15:36 |
| *** mrch has quit IRC | 15:36 | |
| *** ramishra has joined #tripleo | 15:38 | |
| *** derekh has quit IRC | 15:40 | |
| *** artom_ is now known as artom | 15:40 | |
| openstackgerrit | Beth Elwell proposed openstack/tripleo-heat-templates master: Disabled fwaas plugin for horizon https://review.openstack.org/521116 | 15:41 |
| *** itlinux has quit IRC | 15:42 | |
| *** ramishra has quit IRC | 15:43 | |
| *** itlinux has joined #tripleo | 15:44 | |
| *** ramishra has joined #tripleo | 15:45 | |
| jpich | akrivoka: Noticed your vote on https://review.openstack.org/#/c/518619/ - it's been over a week since your tripleo-validations core nominations with loadsa +1 and no objections, IMO it's fine to start using your +2 powers :) | 15:48 |
| *** cylopez has quit IRC | 15:48 | |
| akrivoka | jpich: I have no +2 power on validations yet | 15:49 |
| jpich | akrivoka: Oh, really! I thought you had generic TripleO +2 powers with the usual "use your judgement for where to use etc", my bad. Hm... Ok let's find out how to fix this | 15:50 |
| akrivoka | jpich: I actually have no core status anywhere on tripleo (I have technically on tripleo-ui, just because I created the repo/gerrit, but I'm not supposed to use it) | 15:52 |
| akrivoka | jpich: thanks for looking into this! | 15:52 |
| mwhahaha | akrivoka: jpich: fixed | 15:53 |
| akrivoka | mwhahaha: jpich: works, thanks! | 15:53 |
| jpich | mwhahaha: Thank you! What steps does this involve? Adding someone to the tripleo-core group on gerrit, or are there other steps as well? | 15:53 |
| mwhahaha | jpich: just that | 15:53 |
| jpich | Great, good to know | 15:53 |
| mwhahaha | please don't approve anything at the moment | 15:53 |
| mwhahaha | see ML as we're trying to switch to zuul v3 | 15:54 |
| akrivoka | ack | 15:54 |
| mwhahaha | if you need something merged, let me know and i'll add it to the list to approve after the v3 stuff lands | 15:54 |
| mwhahaha | trying to make this as quick and painless as possible :D | 15:54 |
| akrivoka | jpich: I amended my vote on the above patch, but it still needs a second +2 | 15:54 |
| ccamacho | hey mwhahaha I remove the +A in a backport to pike https://review.openstack.org/#/c/519680 added you to the review, can you +A it when is the correct time? | 15:54 |
| mwhahaha | ccamacho: sure | 15:55 |
| ccamacho | thanks man | 15:55 |
| ccamacho | beagles ^ | 15:55 |
| *** thrash is now known as thrash|biab | 15:55 | |
| jpich | akrivoka: Yep, but it's one step closer now :-) Cheers! And welcome/congrats \o/ | 15:55 |
| beagles | ccamacho, ack | 15:55 |
| akrivoka | jpich: yaay! | 15:55 |
| akrivoka | thanks :) | 15:55 |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-quickstart master: Set correct docker registry namespace. https://review.openstack.org/521121 | 15:57 |
| *** artom_ has joined #tripleo | 16:00 | |
| *** artom has quit IRC | 16:04 | |
| *** artom__ has joined #tripleo | 16:06 | |
| *** artom_ has quit IRC | 16:09 | |
| *** ooolpbot has joined #tripleo | 16:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1724328 | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731032 | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731063 | 16:10 |
| *** ooolpbot has quit IRC | 16:10 | |
| openstack | Launchpad bug 1724328 in tripleo "Netwon to Ocata upgrade failure because of ceilometer-upgrade" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel) | 16:10 |
| openstack | Launchpad bug 1731032 in tripleo "CI: Deployment fails on controller timeout with task creation in heat timing out" [Critical,Triaged] | 16:10 |
| openstack | Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] | 16:10 |
| rlandy|rover | jfrancoa: FYI regarding your review above, we decided to redo all the defaults on docker registry: https://review.openstack.org/#/c/520639/ and ... https://review.openstack.org/#/c/520201/ | 16:12 |
| jfrancoa | rlandy|rover: thanks for the info, I observed that upgrades jobs were failing because they could not find the container image. But if it's already implemented I will abandon the patch | 16:13 |
| *** ffiore has quit IRC | 16:18 | |
| *** akane_ has quit IRC | 16:19 | |
| *** akane has quit IRC | 16:19 | |
| *** ccamacho has quit IRC | 16:23 | |
| *** dpawar has quit IRC | 16:23 | |
| *** ffiore has joined #tripleo | 16:23 | |
| *** thrash|biab is now known as thrash | 16:26 | |
| *** f13o has joined #tripleo | 16:28 | |
| *** artom__ is now known as artom | 16:29 | |
| *** bogdando has quit IRC | 16:31 | |
| *** dxiri has joined #tripleo | 16:33 | |
| *** dxiri has joined #tripleo | 16:33 | |
| *** shardy has quit IRC | 16:38 | |
| *** jistr has quit IRC | 16:39 | |
| *** shardy has joined #tripleo | 16:39 | |
| openstackgerrit | Oliver Walsh proposed openstack/puppet-tripleo master: Configure libvirt SASL SCRAM-SHA1 when TLS is enabled https://review.openstack.org/520253 | 16:39 |
| *** ebarrera has quit IRC | 16:40 | |
| *** gkadam has quit IRC | 16:41 | |
| *** gkadam has joined #tripleo | 16:41 | |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart master: Specify docker.io defaults for upstream config release files. https://review.openstack.org/520639 | 16:43 |
| *** jistr has joined #tripleo | 16:43 | |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart-extras master: Update the Docker registry defaults to rdoproject https://review.openstack.org/520201 | 16:44 |
| *** yamahata has quit IRC | 16:47 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart-extras master: Don't try to start docker for the UC https://review.openstack.org/519441 | 16:48 |
| *** lucasagomes is now known as lucas-afk | 16:48 | |
| *** holser has quit IRC | 16:52 | |
| *** holser has joined #tripleo | 16:53 | |
| *** f13o has quit IRC | 17:01 | |
| *** psachin has quit IRC | 17:02 | |
| *** holser has quit IRC | 17:03 | |
| *** holser has joined #tripleo | 17:03 | |
| openstackgerrit | Julien Danjou proposed openstack/tripleo-heat-templates stable/newton: ceilometer: set event dispatcher to Gnocchi by default https://review.openstack.org/521132 | 17:07 |
| *** gkadam has quit IRC | 17:08 | |
| *** ooolpbot has joined #tripleo | 17:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1724328 | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731032 | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731063 | 17:10 |
| *** ooolpbot has quit IRC | 17:10 | |
| openstack | Launchpad bug 1724328 in tripleo "Netwon to Ocata upgrade failure because of ceilometer-upgrade" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel) | 17:10 |
| openstack | Launchpad bug 1731032 in tripleo "CI: Deployment fails on controller timeout with task creation in heat timing out" [Critical,Triaged] | 17:10 |
| openstack | Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] | 17:10 |
| *** marios has quit IRC | 17:12 | |
| openstackgerrit | Julien Danjou proposed openstack/tripleo-heat-templates stable/newton: ceilometer: set event dispatcher to Gnocchi by default https://review.openstack.org/521132 | 17:12 |
| *** ffiore has quit IRC | 17:13 | |
| *** ykarel|away has joined #tripleo | 17:19 | |
| EmilienM | weshay: have you found the file in tripleo-ci? i don't see the patch yet | 17:20 |
| EmilienM | weshay: now the tripleo-ci patch has merged, I can re-add the jobs in projects like mistral, etc | 17:21 |
| EmilienM | if you see what I mean | 17:21 |
| EmilienM | the job / template layout is merged | 17:21 |
| * mwhahaha waits for the grass to grow | 17:21 | |
| weshay | EmilienM, you had it set correctly in tripleo-ci | 17:21 |
| weshay | I shouldn't need to change anything | 17:21 |
| EmilienM | weshay: you're welcome. | 17:22 |
| openstackgerrit | Merged openstack/tripleo-common stable/pike: Truncate the zaqar message to 512 kbytes https://review.openstack.org/520571 | 17:22 |
| weshay | EmilienM, You are welcome AMERICA.. thank you | 17:22 |
| EmilienM | :P | 17:22 |
| *** dougbtv_ has joined #tripleo | 17:22 | |
| *** shardy has quit IRC | 17:23 | |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-quickstart-extras master: Add -n to lsof collection https://review.openstack.org/521136 | 17:23 |
| *** shardy has joined #tripleo | 17:24 | |
| *** shardy has quit IRC | 17:25 | |
| *** rfolco_lunch is now known as rfolco | 17:25 | |
| *** dbecker has quit IRC | 17:26 | |
| *** holser has quit IRC | 17:29 | |
| openstackgerrit | Oliver Walsh proposed openstack/tripleo-heat-templates master: WIP: Mount etc/my.cnf.d/tripleo.cnf in all db_sync containers https://review.openstack.org/521139 | 17:30 |
| *** dparkes has quit IRC | 17:30 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Add validation task in docker services [Mixed-3] https://review.openstack.org/505587 | 17:32 |
| dprince | weshay: added you as coauthor on https://review.openstack.org/#/c/519441/. With your latest commit msg update do you think we are ready to land it? | 17:37 |
| *** brault has quit IRC | 17:39 | |
| openstackgerrit | Julie Pichon proposed openstack/python-tripleoclient master: Add mock to avoid writing missing files to disk https://review.openstack.org/521140 | 17:40 |
| *** sri__ has quit IRC | 17:41 | |
| *** jfrancoa has quit IRC | 17:41 | |
| EmilienM | sshnaidm|off: I sent an email and ask to not approve patches | 17:45 |
| openstackgerrit | Ronelle Landy proposed openstack-infra/tripleo-ci master: Add MTU adjustment parameters to multinode settings https://review.openstack.org/521112 | 17:45 |
| EmilienM | sshnaidm|off: in fact, it's gchamoul who did a recheck... | 17:45 |
| EmilienM | gchamoul: please no recheck, like requested on tripleo ML | 17:45 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-quickstart-extras master: Use openstack commands for postci https://review.openstack.org/521146 | 17:46 |
| * mwhahaha optimizes | 17:46 | |
| EmilienM | pradk: please no +A, like sent on ML | 17:46 |
| *** dtantsur is now known as dtantsur|afk | 17:48 | |
| *** jpich has quit IRC | 17:48 | |
| weshay | dprince, yes I want to recheck it w/ the job definition that EmilienM has in tripleo-ci now | 17:49 |
| weshay | but I think we're good.. | 17:49 |
| weshay | dprince, the interesting thing was that it was not the duplicate entries in /etc/hosts | 17:49 |
| dprince | weshay: it was the way hosts was configured for multinode jobs? | 17:50 |
| weshay | \/etc/hosts had to be completely cleared in the same we clear for the overcloud deployments upstream | 17:50 |
| dprince | weshay: okay. The latest run looks to be failing though. Is that expected? | 17:50 |
| *** dougbtv_ has quit IRC | 17:51 | |
| dprince | you removed the Depends-On for the CI patch I think? | 17:51 |
| weshay | hrm.. k will check in a sec | 17:51 |
| *** ykarel|away has quit IRC | 17:51 | |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates master: docker: Add unit tests on service_name https://review.openstack.org/442755 | 17:54 |
| *** gbarros has quit IRC | 18:02 | |
| *** cylopez has joined #tripleo | 18:02 | |
| EmilienM | numans: can we move scenario007-container to gate and stop experimental? | 18:03 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Add validation task in docker services [Sahara] https://review.openstack.org/502039 | 18:06 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Add validation task in docker services [Ironic] https://review.openstack.org/502037 | 18:06 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Add validation task in docker services [Neutron] https://review.openstack.org/489957 | 18:07 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Add validation task in docker services [Aodh] https://review.openstack.org/491411 | 18:07 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: Move scenario007-container to the gate https://review.openstack.org/521149 | 18:08 |
| *** ooolpbot has joined #tripleo | 18:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1724328 | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731032 | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731063 | 18:10 |
| *** ooolpbot has quit IRC | 18:10 | |
| openstack | Launchpad bug 1724328 in tripleo "Netwon to Ocata upgrade failure because of ceilometer-upgrade" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel) | 18:10 |
| openstack | Launchpad bug 1731032 in tripleo "CI: Deployment fails on controller timeout with task creation in heat timing out" [Critical,Triaged] | 18:10 |
| openstack | Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] | 18:10 |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-common master: Validate roles data and network data https://review.openstack.org/508567 | 18:11 |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-common master: Add a Get Networks workflow https://review.openstack.org/509419 | 18:11 |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-common master: Add UpdateNetworks workflow https://review.openstack.org/513463 | 18:11 |
| *** jpena is now known as jpena|off | 18:14 | |
| *** ramishra has quit IRC | 18:20 | |
| EmilienM | for these who are curious how simple it is to add tripleo CI jobs outside of tripleo projects, see https://review.openstack.org/#/q/topic:tripleo/scenario007 | 18:21 |
| *** gbarros has joined #tripleo | 18:24 | |
| *** trown is now known as trown|lunch | 18:29 | |
| *** leitan has quit IRC | 18:30 | |
| *** athomas has quit IRC | 18:31 | |
| *** leitan has joined #tripleo | 18:38 | |
| EmilienM | arxcruz: thx for restoring the tempest html file, it works now! | 18:53 |
| *** dhill_ has joined #tripleo | 18:56 | |
| *** hjensas has joined #tripleo | 18:59 | |
| *** hjensas has quit IRC | 18:59 | |
| *** hjensas has joined #tripleo | 18:59 | |
| weshay | dprince, the undercloud-containers job is failing because it's again deploying the overcloud | 18:59 |
| weshay | I blame EmilienM :) | 18:59 |
| *** dsneddon has joined #tripleo | 19:00 | |
| weshay | so not sure if we have everything in place yet for zuul | 19:00 |
| dprince | weshay: would re-adding the Depends-On fix it again? | 19:00 |
| fultonj | EmilienM, regarding topic:tripleo/scenario007 I just want to confirm my understanding.... | 19:00 |
| weshay | dprince, ya.. I abandoned the patch, so we may need to turn that back on | 19:00 |
| fultonj | In 521149, looks like scenario007 already exists and you're moving it from tripleo experimental to regluar tripleo check and gate jobs. | 19:00 |
| fultonj | In 521152, looks like ovn already uses zuul so you're adding tripleo's scenario007 to it | 19:00 |
| fultonj | right? | 19:00 |
| EmilienM | fultonj: that's 100% exact | 19:01 |
| fultonj | ok, cool | 19:01 |
| EmilienM | weshay: what did I do again :P | 19:01 |
| weshay | dprince, restored patch | 19:01 |
| EmilienM | fultonj: and as you can see the benefit of having that in our repos | 19:02 |
| EmilienM | fultonj: we get control of everything quite easily | 19:03 |
| EmilienM | fultonj: also, projects themselves can decide to disable voting when they want | 19:03 |
| fultonj | yeah, it's nice | 19:03 |
| openstackgerrit | Ryan Brady proposed openstack/tripleo-common master: Adds Workflow to List Available Roles https://review.openstack.org/509832 | 19:03 |
| openstackgerrit | Ryan Brady proposed openstack/tripleo-common master: [WIP] Adds UpdateRoles Workflow https://review.openstack.org/515045 | 19:03 |
| *** salmankhan has quit IRC | 19:04 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart-extras master: Don't try to start docker for the UC https://review.openstack.org/519441 | 19:04 |
| fultonj | now if only other projects used zuul.... | 19:04 |
| weshay | dprince, ok.. it's back | 19:04 |
| *** chlong__ has quit IRC | 19:04 | |
| dprince | weshay: ack, thanks | 19:04 |
| EmilienM | fultonj: they do | 19:04 |
| weshay | bbiab | 19:04 |
| EmilienM | fultonj: everyone does | 19:04 |
| EmilienM | fultonj: you mean outside of openstack? | 19:05 |
| fultonj | yeah | 19:05 |
| *** thrash is now known as thrash|mtg | 19:05 | |
| EmilienM | ah.. | 19:05 |
| EmilienM | weshay: I'm re-adding tripleo jobs to projects (telemetry, etc) like they were before, but with zuul v3 layout. So no lost in the transition | 19:08 |
| *** ooolpbot has joined #tripleo | 19:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1724328 | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731032 | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731063 | 19:10 |
| *** ooolpbot has quit IRC | 19:10 | |
| openstack | Launchpad bug 1724328 in tripleo "Netwon to Ocata upgrade failure because of ceilometer-upgrade" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel) | 19:10 |
| openstack | Launchpad bug 1731032 in tripleo "CI: Deployment fails on controller timeout with task creation in heat timing out" [Critical,Triaged] | 19:10 |
| openstack | Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] | 19:10 |
| *** akane has joined #tripleo | 19:15 | |
| *** akane_ has joined #tripleo | 19:15 | |
| *** akane_ has quit IRC | 19:17 | |
| *** akane has quit IRC | 19:17 | |
| EmilienM | dprince: do we still actually need t-p-e? | 19:18 |
| dprince | EmilienM: sorry, t-p-e? | 19:19 |
| EmilienM | dprince: tripleo-puppet-elements | 19:19 |
| EmilienM | in a containerized overcloud, I'm unsure where t-p-e is useful | 19:19 |
| EmilienM | but I probably missed something | 19:19 |
| dprince | there are still some baremetal dependencies there. Pacemaker | 19:19 |
| dprince | I've been testing a leaner version though | 19:20 |
| dprince | EmilienM: sec and I'll so you what worked for me | 19:20 |
| EmilienM | ok | 19:20 |
| *** chlong__ has joined #tripleo | 19:20 | |
| EmilienM | I'm trying to find out which CI jobs do we actually need to run for t-p-e, any thoughts on multinode jobs? | 19:21 |
| dprince | feel free to try it | 19:21 |
| EmilienM | try what? | 19:22 |
| *** fragatin_ has quit IRC | 19:22 | |
| openstackgerrit | Dan Prince proposed openstack/tripleo-puppet-elements master: Try removing OS packages from tripleo-puppet-elements https://review.openstack.org/521172 | 19:25 |
| dprince | EmilienM: ^^ that has worked for me locally. Can probably clean out a bunch more | 19:25 |
| dprince | EmilienM: the pacemaker and networking stuff needs to stay though | 19:26 |
| EmilienM | oh cool | 19:26 |
| EmilienM | yeah indeed | 19:26 |
| dprince | made my image a *lot* smaller and thus speeds up larger deployments just a bit | 19:26 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-puppet-elements master: Migrate instack-undercloud to zuul v3 jobs https://review.openstack.org/521174 | 19:28 |
| EmilienM | dprince: yeah | 19:28 |
| EmilienM | dprince: same question for tripleo-image-elements - do we use it in multinode jobs? i believe not | 19:30 |
| *** itlinux has quit IRC | 19:31 | |
| dprince | EmilienM: probably not. os-net-config's element is the only peice that is there for the legacy os-net-config Yaml format | 19:31 |
| EmilienM | ok | 19:32 |
| dprince | EmilienM: if the Yaml has been updated should be okay to drop it. | 19:32 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-image-elements master: Migrate instack-undercloud to zuul v3 jobs https://review.openstack.org/521175 | 19:33 |
| dsneddon | dprince, EmilienM: I know I've run across a number of upgrade scenarios where the YAMLs weren't updated. I'm fine with dropping it, since it was technically deprecated for more than one cycle. | 19:33 |
| *** itlinux has joined #tripleo | 19:33 | |
| dprince | EmilienM, dsneddon: https://review.openstack.org/#/c/470017/ | 19:34 |
| *** amoralej is now known as amoralej|off | 19:34 | |
| dprince | per the last comment from bnemec-pto we were just waiting on CI | 19:34 |
| *** chlong__ has quit IRC | 19:36 | |
| EmilienM | dprince: sounds like we're blocked by newton support | 19:37 |
| dsneddon | dprince, Thanks, do you want me to submit a patch to tripleo-docs, since we still document the old element config? https://docs.openstack.org/tripleo-docs/latest/install/advanced_deployment/network_isolation.html | 19:37 |
| EmilienM | https://review.openstack.org/#/c/483064/1 | 19:37 |
| dprince | dsneddon: sure, that would be cool man | 19:38 |
| EmilienM | dprince: maybe we should make the switch and maintain newton nic-configs in tripleo-ci | 19:38 |
| EmilienM | so we can move forward | 19:38 |
| dprince | EmilienM: works for me | 19:38 |
| EmilienM | I'll check with bnemec-pto when he's back | 19:38 |
| dsneddon | I'll write up the docs now so I don't forget | 19:38 |
| EmilienM | dsneddon++ | 19:39 |
| EmilienM | slagle: any thoughts on making tripleo-ci-centos-7-3nodes-multinode voting? | 19:41 |
| EmilienM | also maybe rename to have -oooq in the name | 19:41 |
| EmilienM | slagle: the job is fast and stable | 19:41 |
| openstackgerrit | Emilien Macchi proposed openstack/paunch master: Migrate instack-undercloud to zuul v3 jobs https://review.openstack.org/521177 | 19:45 |
| *** dprince has quit IRC | 19:48 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-validations master: Migrate instack-undercloud to zuul v3 jobs https://review.openstack.org/521179 | 19:48 |
| *** dtrainor has quit IRC | 19:48 | |
| *** chlong__ has joined #tripleo | 19:49 | |
| *** itlinux has quit IRC | 19:50 | |
| EmilienM | can someone approve https://review.openstack.org/#/c/520689/ please | 19:50 |
| *** leitan has quit IRC | 19:51 | |
| *** trown|lunch is now known as trown | 19:51 | |
| *** vfarias has joined #tripleo | 19:52 | |
| *** fragatina has joined #tripleo | 19:54 | |
| *** itlinux has joined #tripleo | 19:54 | |
| *** haint has joined #tripleo | 19:55 | |
| *** holser has joined #tripleo | 19:56 | |
| openstackgerrit | Merged openstack/instack-undercloud master: Migrate instack-undercloud to zuul v3 jobs https://review.openstack.org/520264 | 19:56 |
| openstackgerrit | Merged openstack/tripleo-common master: Migrate tripleo-common to zuul v3 jobs https://review.openstack.org/520238 | 19:56 |
| openstackgerrit | Merged openstack/python-tripleoclient master: Migrate python-tripleoclient to zuul v3 jobs https://review.openstack.org/520241 | 19:56 |
| *** cylopez has quit IRC | 19:57 | |
| *** dtrainor has joined #tripleo | 19:57 | |
| *** nguyentrihai has quit IRC | 19:58 | |
| slagle | EmilienM: maybe not a good idea right now, due to resource concerns raised by infra at the past ptg | 19:59 |
| EmilienM | mwhahaha: please do not backport these patches ^ | 19:59 |
| mwhahaha | EmilienM: hadn't planned on it | 19:59 |
| EmilienM | slagle: i'll ask them again, last time I checked, resources were fine | 19:59 |
| slagle | EmilienM: until we're well past that i don't think i would add new voting jobs, especially 3 nodes | 19:59 |
| slagle | EmilienM: ok | 19:59 |
| EmilienM | mwhahaha: good. I'll take care of that this weekend. | 19:59 |
| mwhahaha | well i'd rather drop some of the other ones and use 3 node instead | 20:00 |
| mwhahaha | since that's more of a realistic deployment scenario | 20:00 |
| mwhahaha | it runs better anyway | 20:00 |
| EmilienM | you mean, consolidating services? | 20:00 |
| EmilienM | it's complicated to manage upgrades, from previous tags, etc... | 20:00 |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates master: Add tls roles for undercloud https://review.openstack.org/517079 | 20:07 |
| openstackgerrit | Emilien Macchi proposed openstack/os-apply-config master: Migrate instack-undercloud to zuul v3 jobs https://review.openstack.org/521181 | 20:08 |
| openstackgerrit | Emilien Macchi proposed openstack/os-net-config master: Migrate instack-undercloud to zuul v3 jobs https://review.openstack.org/521182 | 20:09 |
| mwhahaha | EmilienM: did you miss the os-* projects? | 20:09 |
| openstackgerrit | Emilien Macchi proposed openstack/os-collect-config master: Migrate instack-undercloud to zuul v3 jobs https://review.openstack.org/521183 | 20:09 |
| mwhahaha | EmilienM: also your patch commit message is wrong | 20:09 |
| *** ooolpbot has joined #tripleo | 20:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1724328 | 20:10 |
| openstack | Launchpad bug 1724328 in tripleo "Netwon to Ocata upgrade failure because of ceilometer-upgrade" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel) | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731032 | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731063 | 20:10 |
| *** ooolpbot has quit IRC | 20:10 | |
| openstack | Launchpad bug 1731032 in tripleo "CI: Deployment fails on controller timeout with task creation in heat timing out" [Critical,Triaged] | 20:10 |
| openstack | Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] | 20:10 |
| openstackgerrit | Emilien Macchi proposed openstack/os-refresh-config master: Migrate instack-undercloud to zuul v3 jobs https://review.openstack.org/521184 | 20:10 |
| EmilienM | mwhahaha: damn | 20:10 |
| openstackgerrit | Emilien Macchi proposed openstack/os-refresh-config master: Migrate os-refresh-config to zuul v3 jobs https://review.openstack.org/521184 | 20:11 |
| *** rmascena has quit IRC | 20:11 | |
| openstackgerrit | Emilien Macchi proposed openstack/os-collect-config master: Migrate os-collect-config to zuul v3 jobs https://review.openstack.org/521183 | 20:11 |
| openstackgerrit | Emilien Macchi proposed openstack/os-net-config master: Migrate os-net-config to zuul v3 jobs https://review.openstack.org/521182 | 20:11 |
| openstackgerrit | Emilien Macchi proposed openstack/os-apply-config master: Migrate os-apply-config to zuul v3 jobs https://review.openstack.org/521181 | 20:12 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-validations master: Migrate tripleo-validations to zuul v3 jobs https://review.openstack.org/521179 | 20:12 |
| openstackgerrit | Emilien Macchi proposed openstack/paunch master: Migrate paunch to zuul v3 jobs https://review.openstack.org/521177 | 20:13 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-image-elements master: Migrate t-i-e to zuul v3 jobs https://review.openstack.org/521175 | 20:13 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-puppet-elements master: Migrate t-p-e to zuul v3 jobs https://review.openstack.org/521174 | 20:13 |
| *** gbarros has quit IRC | 20:15 | |
| *** bobh has quit IRC | 20:17 | |
| mwhahaha | weshay, EmilienM: we don't have a bug for teh scenario003 failures yet do we? | 20:17 |
| mwhahaha | cause it is different | 20:18 |
| mwhahaha | so i'll create one for those | 20:18 |
| EmilienM | mwhahaha: not afik | 20:18 |
| *** rfolco has quit IRC | 20:18 | |
| *** thrash|mtg is now known as thrash|biab | 20:22 | |
| *** eck` is now known as eck`gone | 20:27 | |
| *** ipsecguy_ has joined #tripleo | 20:27 | |
| weshay | mwhahaha, it's failing on a different tempest test, but is it a different root cause? | 20:28 |
| mwhahaha | maybe? | 20:29 |
| *** morazi has quit IRC | 20:29 | |
| mwhahaha | it seems like different reasons but not sure | 20:29 |
| mwhahaha | i'm trying to take a look now | 20:29 |
| *** ipsecguy has quit IRC | 20:30 | |
| weshay | mwhahaha, if you see something different please let me know, I'd like to see it for my own neducation | 20:33 |
| mwhahaha | i'm kinda concerned about the dockerd-current messages about something becoming an orphan | 20:33 |
| mwhahaha | http://logs.openstack.org/36/521136/1/check/legacy-tripleo-ci-centos-7-scenario003-multinode-oooq-container/a6708d9/logs/subnode-2/var/log/journal.txt.gz#_Nov_17_18_54_17 | 20:33 |
| *** cylopez has joined #tripleo | 20:34 | |
| mwhahaha | and ofcourse docker is not set to debug on the subnode | 20:35 |
| weshay | mwhahaha, hrm.. | 20:35 |
| weshay | http://logs.openstack.org/36/521136/1/check/legacy-tripleo-ci-centos-7-scenario003-multinode-oooq-container/a6708d9/logs/subnode-2/var/log/extra/errors.txt.gz | 20:35 |
| mwhahaha | weshay: do we do any docker log capture stuff? or status of docker ? | 20:36 |
| weshay | no time stamps but look for NotConnectedError: Cannot commit transaction | 20:36 |
| weshay | mwhahaha, ya.. sec | 20:36 |
| mwhahaha | so tempest didn't start running until 18:56 | 20:36 |
| mwhahaha | 18:56:34 to be specific, so anything prior to that is probably not something valid | 20:37 |
| mwhahaha | however teh ovs connection stuff might be problemmattic | 20:37 |
| mwhahaha | which is what is saw ealier | 20:37 |
| mwhahaha | cause that is right when tempest is running | 20:37 |
| mwhahaha | oh silly glareclient | 20:38 |
| mwhahaha | but not related to this | 20:39 |
| mwhahaha | i think the scenario003 problems are related to the ovs failures | 20:39 |
| mwhahaha | let me see if those show up on the scenario001 failures | 20:39 |
| weshay | Timeout in RPC method get_active_networks_info. Waiting for 28 seconds before next attempt. If the server is not down, consider increasing the rpc_response_timeout option as Neutron server(s) may be overloaded and unable to respond quickly enough.: MessagingTimeout: Timed out waiting for a reply to message ID f40afd12b16c4c6abfa0ea604cb6ff61 | 20:40 |
| mwhahaha | weshay: are you talking about http://logs.openstack.org/36/521136/1/check/legacy-tripleo-ci-centos-7-scenario003-multinode-oooq-container/a6708d9/logs/subnode-2/var/log/containers/neutron/neutron-metadata-agent.log.txt.gz#_2017-11-17_18_54_58_565 | 20:40 |
| mwhahaha | ? | 20:40 |
| mwhahaha | that was while the deployment was occuring | 20:40 |
| mwhahaha | prior to tempest | 20:40 |
| mwhahaha | http://logs.openstack.org/36/521136/1/check/legacy-tripleo-ci-centos-7-scenario003-multinode-oooq-container/a6708d9/logs/subnode-2/var/log/containers/neutron/neutron-l3-agent.log.txt.gz#_2017-11-17_18_57_44_523 | 20:41 |
| weshay | I see it after 18:56 | 20:41 |
| mwhahaha | that however, is different | 20:41 |
| mwhahaha | where? | 20:41 |
| *** fultonj has quit IRC | 20:41 | |
| *** dprince has joined #tripleo | 20:41 | |
| weshay | bah.. sorry the errors.txt has the is scrambled a bit | 20:43 |
| mwhahaha | yea | 20:43 |
| *** itlinux has quit IRC | 20:43 | |
| mwhahaha | it's better to go look at the source | 20:43 |
| mwhahaha | i think the scenario003 problem is the ovs errors | 20:43 |
| mwhahaha | that being said the scenario001 logs that i have were all openvswitch-2.7.2 | 20:45 |
| mwhahaha | so i wonder if we have a new problem and 001 is fine now | 20:45 |
| mwhahaha | need to find a more recent 001 failure | 20:45 |
| weshay | k.. /me gets | 20:46 |
| weshay | rlandy|rover, ^ | 20:46 |
| weshay | here is the lasted scen001 failure on tempest http://logs.openstack.org/81/520581/1/check/legacy-tripleo-ci-centos-7-scenario001-multinode-oooq-container/128fccc/ | 20:47 |
| mwhahaha | ok that one is openvswitch-2.8.1-2.1fc28.el7.x86_64 | 20:47 |
| mwhahaha | http://logs.openstack.org/81/520581/1/check/legacy-tripleo-ci-centos-7-scenario001-multinode-oooq-container/128fccc/logs/subnode-2/var/log/containers/neutron/neutron-l3-agent.log.txt.gz#_2017-11-16_15_11_32_810 | 20:48 |
| mwhahaha | also issues talking to ovs | 20:48 |
| mwhahaha | http://logs.openstack.org/81/520581/1/check/legacy-tripleo-ci-centos-7-scenario001-multinode-oooq-container/128fccc/logs/subnode-2/var/log/containers/neutron/neutron-metadata-agent.log.txt.gz#_2017-11-16_15_21_24_094 metadata service was working | 20:48 |
| mwhahaha | see there's always that messagingtimeout because i think it gets started prior to the neutron service | 20:48 |
| mwhahaha | i don't think we have good ordering on container startup, but i'm not sure | 20:49 |
| mwhahaha | we used to with puppet because you could tell the agents to wait, but i'm not sure we have a similar concept with containers anymore | 20:49 |
| mwhahaha | dprince: do container starts have ordering associated with them still? ie agent waits until server is started | 20:50 |
| dprince | mwhahaha: there are 2 types of ordering | 20:50 |
| dprince | mwhahaha: the "stepwise" ording of the higher deployment workflow (1-5), where 1 is for the LB, 2 database/rabbit, etc. | 20:51 |
| dprince | mwhahaha: then there is a start_order which can be used to control container startup within each step | 20:51 |
| mwhahaha | ok so there's still that concept, cool | 20:52 |
| dprince | mwhahaha: start order is applied via paunch | 20:52 |
| mwhahaha | i'll have to remember start_order | 20:52 |
| dprince | mwhahaha: yep, we already rely on it for a few things | 20:52 |
| mwhahaha | might need to check if the neutron agents containers are properly waiting for neutron server | 20:52 |
| mwhahaha | weshay: http://logs.openstack.org/79/516979/2/gate/legacy-tripleo-ci-centos-7-scenario001-multinode-oooq-container/13841da/logs/subnode-2/var/log/containers/neutron/neutron-l3-agent.log.txt.gz#_2017-11-08_21_13_17_387 looks like the ovs backend error was there with the older ovs package as well | 20:53 |
| dprince | mwhahaha: had a nice chat with stevebaker yesterday about some potential improvements to paunch to help us catch (fail-fast) containers in restart, and or unhealthy | 20:53 |
| mwhahaha | dprince: are we tracking when docker containers are crashing? I was wondering if this error was a crashing container http://logs.openstack.org/36/521136/1/check/legacy-tripleo-ci-centos-7-scenario003-multinode-oooq-container/a6708d9/logs/subnode-2/var/log/journal.txt.gz#_Nov_17_18_54_17 | 20:54 |
| mwhahaha | containerd: 6c5eb757d223c6711a2ab61dd25e0023bda57dff8ca275154ad00a1372158a7c:e80037679a625f34dfa3e10ae1adb8111fc3dc0fcf2ff21c28748645d9142567 (pid 74940) has become an orphan, killing it | 20:54 |
| dprince | mwhahaha: no, we start it and go | 20:54 |
| dprince | mwhahaha: having paunch detect the failure with a 'status' command or something and then wiring that into the higher level workflow between each step would be useful here | 20:55 |
| dprince | mwhahaha: we aren't catching this at this point | 20:55 |
| mwhahaha | ok sounds like something we'd probably like to wire into CI as well to see if containers aren't fully OK at teh end of the run | 20:56 |
| *** jmelvin has quit IRC | 20:58 | |
| *** holser has quit IRC | 21:00 | |
| *** vfarias has quit IRC | 21:01 | |
| *** holser has joined #tripleo | 21:01 | |
| weshay | mwhahaha, are these errors just startup issues as http://logs.openstack.org/79/516979/2/gate/legacy-tripleo-ci-centos-7-scenario001-multinode-oooq-container/13841da/logs/subnode-2/var/log/containers/neutron/neutron-l3-agent.log.txt.gz#_2017-11-08_21_13_30_985 | 21:01 |
| weshay | shows the deivce eventually comes up | 21:02 |
| *** ianw has quit IRC | 21:02 | |
| mwhahaha | weshay: i don't think so | 21:02 |
| mwhahaha | but i don't know, i'm not neutron guy | 21:02 |
| mwhahaha | because it started 5 mins before | 21:02 |
| mwhahaha | so the only think i could think of is that it's some perf related issue since it's so flaky | 21:02 |
| weshay | just sayin.. I see | 21:02 |
| weshay | record=qg-8d5f417f-66). Not connected | 21:02 |
| weshay | and then finally | 21:03 |
| weshay | Device qg-8d5f417f-66 already exists | 21:03 |
| *** Hazelesque_ is now known as Hazelesque | 21:03 | |
| weshay | mwhahaha, EmilienM rlandy|rover are we going to move 003 to non-voting? | 21:04 |
| mwhahaha | of course i bet we don't have l3 agent debugging enabled | 21:04 |
| * mwhahaha sighs and prepares a patch | 21:04 | |
| weshay | heh | 21:04 |
| * mwhahaha throws things at beagles | 21:04 | |
| weshay | mwhahaha, it's a tripleo bug man.. it's ALL YOUR FAULT | 21:05 |
| mwhahaha | it was like this when i got here! | 21:05 |
| weshay | lolz | 21:05 |
| *** dparkes has joined #tripleo | 21:06 | |
| * mwhahaha debugs all the things | 21:07 | |
| weshay | dprince, think we're back to green | 21:07 |
| dprince | weshay: boom :) | 21:08 |
| *** ianw has joined #tripleo | 21:09 | |
| *** ooolpbot has joined #tripleo | 21:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1724328 | 21:10 |
| openstack | Launchpad bug 1724328 in tripleo "Netwon to Ocata upgrade failure because of ceilometer-upgrade" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel) | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731032 | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731063 | 21:10 |
| *** ooolpbot has quit IRC | 21:10 | |
| openstack | Launchpad bug 1731032 in tripleo "CI: Deployment fails on controller timeout with task creation in heat timing out" [Critical,Triaged] | 21:10 |
| openstack | Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] | 21:10 |
| *** holser has quit IRC | 21:13 | |
| *** eck`gone is now known as eck` | 21:14 | |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-heat-templates master: Add debug to neutron agents https://review.openstack.org/521191 | 21:14 |
| mwhahaha | debug all the things -^ | 21:15 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-heat-templates master: DNM: Testing debug neutron metadata https://review.openstack.org/519756 | 21:15 |
| mwhahaha | weshay: so to answer your question, yea we may have to disable voting on scenario003 if this continues | 21:16 |
| weshay | rlandy|rover, ^ | 21:16 |
| rlandy|rover | weshay: ack - with containers and w/o - both are unstable | 21:17 |
| weshay | mwhahaha, doesn't look like a promotion would help either | 21:17 |
| mwhahaha | no it's not something that anyone seems to be aware of | 21:18 |
| weshay | although | 21:18 |
| weshay | scen003 only started acting flaky in the promotion jobs on the 16th | 21:18 |
| weshay | that just may be luck though | 21:18 |
| mwhahaha | well it might have been flaky before that but we had told people to not aprove anything last week :D | 21:19 |
| mwhahaha | and 001 was far more noticable | 21:19 |
| weshay | http://38.145.34.234/ | 21:19 |
| weshay | master | 21:19 |
| mwhahaha | weshay: btw with our reduction of log that we collect, it seems that collect-logs is down to <3 mins | 21:19 |
| weshay | fs16 and fs18 | 21:19 |
| weshay | mwhahaha, k.. good.. I doc'd the diff I saw in the bug | 21:20 |
| mwhahaha | i've got a patch that might drop it another 30secs | 21:20 |
| * weshay looks | 21:20 | |
| mwhahaha | http://logs.openstack.org/46/521146/1/check/legacy-tripleo-ci-centos-7-scenario002-multinode-oooq-puppet/73977e4/logs/quickstart_collect_logs.txt.gz | 21:20 |
| mwhahaha | 2:32 | 21:21 |
| mwhahaha | https://review.openstack.org/#/c/521146/ the openstack command is unfortunately really heavy on startup | 21:21 |
| mwhahaha | also https://review.openstack.org/#/c/521136/ will help as well | 21:21 |
| weshay | EOC, cool | 21:22 |
| mwhahaha | well EOF was already taken so i figured EOC for end of command :D | 21:22 |
| *** dtrainor has quit IRC | 21:23 | |
| *** rlandy|rover has quit IRC | 21:24 | |
| weshay | this is how we know nothing merges into elastic-recheck http://status.openstack.org/elastic-recheck/ | 21:25 |
| weshay | the bug w/ the highest fail rate has 1 fail in 24hr | 21:25 |
| mwhahaha | dat 43 hour indexing delay | 21:25 |
| openstackgerrit | Alan Bishop proposed openstack/puppet-tripleo master: Fix use of deprecated "api_class" key manager option https://review.openstack.org/521195 | 21:25 |
| openstackgerrit | Dan Sneddon proposed openstack/tripleo-docs master: Update docs to reflect updates to os-net-config mechanism https://review.openstack.org/521197 | 21:30 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-quickstart-extras master: Use openstack commands for postci https://review.openstack.org/521146 | 21:32 |
| *** thrash|biab is now known as thrash | 21:39 | |
| *** pradk has quit IRC | 21:39 | |
| *** myoung is now known as myoung|bbl | 21:42 | |
| thrash | mwhahaha: made a suggestion on those commands... wdyt? | 21:43 |
| mwhahaha | thrash: that might be an improvement | 21:44 |
| thrash | mwhahaha: just a thought... I know I use it a lot when I'm just working in general. | 21:44 |
| mwhahaha | unfortunately i'm not sure every one of those commands supports it | 21:45 |
| mwhahaha | but we can give it a shot | 21:45 |
| mwhahaha | cause the formatter is on a per client basis | 21:45 |
| mwhahaha | so the mistral one might, but nova might not | 21:45 |
| thrash | mwhahaha: I thought with osc the formatter was in osc, not the clients. | 21:45 |
| mwhahaha | it's not returning when you do openstack --help | 21:46 |
| mwhahaha | maybe though | 21:46 |
| thrash | hmmm | 21:46 |
| mwhahaha | i think we can just append it to the command | 21:46 |
| mwhahaha | i'll give it a shot when my undercloud finishes building | 21:46 |
| *** florianf has quit IRC | 21:48 | |
| mwhahaha | yea you can just append it | 21:48 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-quickstart-extras master: Use openstack commands for postci https://review.openstack.org/521146 | 21:49 |
| *** trown is now known as trown|outtypewww | 21:51 | |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-heat-templates master: Add debug to neutron agents https://review.openstack.org/521191 | 21:55 |
| *** jtomasek has quit IRC | 22:02 | |
| *** artom has quit IRC | 22:03 | |
| *** fragatina has quit IRC | 22:03 | |
| *** Goneri has joined #tripleo | 22:04 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart-extras master: Fix the undercloud-containers job https://review.openstack.org/519441 | 22:04 |
| *** fragatina has joined #tripleo | 22:05 | |
| *** abishop has quit IRC | 22:09 | |
| *** ooolpbot has joined #tripleo | 22:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1724328 | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731032 | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731063 | 22:10 |
| *** ooolpbot has quit IRC | 22:10 | |
| openstack | Launchpad bug 1724328 in tripleo "Netwon to Ocata upgrade failure because of ceilometer-upgrade" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel) | 22:10 |
| openstack | Launchpad bug 1731032 in tripleo "CI: Deployment fails on controller timeout with task creation in heat timing out" [Critical,Triaged] | 22:10 |
| openstack | Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] | 22:10 |
| openstackgerrit | Alex Schultz proposed openstack-infra/tripleo-ci master: Move scenario003 to non-voting https://review.openstack.org/521205 | 22:16 |
| mwhahaha | weshay, EmilienM -^ fyi | 22:16 |
| * mwhahaha sends a note to the ML | 22:17 | |
| *** fultonj has joined #tripleo | 22:21 | |
| zaneb | mwhahaha: hopefully this will do the trick on bug 1731032: https://review.openstack.org/#/c/521204/ | 22:22 |
| openstack | bug 1731032 in OpenStack Heat "CI: Deployment fails on controller timeout with task creation in heat timing out" [Critical,In progress] https://launchpad.net/bugs/1731032 - Assigned to Zane Bitter (zaneb) | 22:22 |
| mwhahaha | zaneb: cool thanks | 22:23 |
| *** Goneri has quit IRC | 22:24 | |
| *** gbarros has joined #tripleo | 22:30 | |
| *** dprince has quit IRC | 22:33 | |
| *** fultonj has quit IRC | 22:34 | |
| *** Guest78 has joined #tripleo | 22:35 | |
| *** cdearborn has quit IRC | 22:39 | |
| *** Guest78 has quit IRC | 22:42 | |
| openstackgerrit | mathieu bultel proposed openstack/tripleo-common master: Stream the Ansible output to a Zaqar queue https://review.openstack.org/521058 | 22:48 |
| *** lblanchard has joined #tripleo | 22:56 | |
| *** akrivoka has quit IRC | 22:58 | |
| *** chlong__ has quit IRC | 23:00 | |
| * mwhahaha wonders if the top change in the queue is just going to time out and ruin everything | 23:04 | |
| *** ooolpbot has joined #tripleo | 23:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1724328 | 23:10 |
| openstack | Launchpad bug 1724328 in tripleo "Netwon to Ocata upgrade failure because of ceilometer-upgrade" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel) | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731032 | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1731063 | 23:10 |
| *** ooolpbot has quit IRC | 23:10 | |
| openstack | Launchpad bug 1731032 in OpenStack Heat "CI: Deployment fails on controller timeout with task creation in heat timing out" [Critical,In progress] - Assigned to Zane Bitter (zaneb) | 23:10 |
| openstack | Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] | 23:10 |
| *** gbarros has quit IRC | 23:53 | |
| *** pcaruana has quit IRC | 23:54 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
