| openstackgerrit | Damien Ciabrini proposed openstack/tripleo-common master: DNR undercloud/overcloud with mariadb 10.3.10 and galera 3.23 https://review.openstack.org/617355 | 00:06 |
|---|---|---|
| openstackgerrit | Dan Radez proposed openstack/tripleo-common stable/queens: Install python2-networking-ansible in neutron-server https://review.openstack.org/621044 | 00:07 |
| *** ade_lee has quit IRC | 00:09 | |
| *** wolverineav has quit IRC | 00:09 | |
| *** flaper87 has quit IRC | 00:09 | |
| *** ooolpbot has joined #tripleo | 00:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 00:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1805870 | 00:10 |
| *** ooolpbot has quit IRC | 00:10 | |
| openstack | Launchpad bug 1805870 in tripleo "mistral unit tests failing in dlrn build of tripleo-common" [Critical,Triaged] - Assigned to Adriano Petrich (apetrich) | 00:10 |
| *** wolverineav has joined #tripleo | 00:10 | |
| *** wolverineav has quit IRC | 00:12 | |
| *** vinaykns has quit IRC | 00:13 | |
| *** wolverineav has joined #tripleo | 00:13 | |
| *** flaper87 has joined #tripleo | 00:14 | |
| *** threestrands has joined #tripleo | 00:18 | |
| *** ansmith has joined #tripleo | 00:19 | |
| *** tosky has quit IRC | 00:27 | |
| *** rh-jelabarre has joined #tripleo | 00:41 | |
| *** aedc has quit IRC | 00:49 | |
| *** bfournie has joined #tripleo | 01:02 | |
| *** hamzy_ has joined #tripleo | 01:07 | |
| *** ooolpbot has joined #tripleo | 01:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1805870 | 01:10 |
| *** ooolpbot has quit IRC | 01:10 | |
| openstack | Launchpad bug 1805870 in tripleo "mistral unit tests failing in dlrn build of tripleo-common" [Critical,Triaged] - Assigned to Adriano Petrich (apetrich) | 01:10 |
| openstackgerrit | Merged openstack/tripleo-common stable/rocky: Generating fencing for capitalized MACs https://review.openstack.org/619725 | 01:14 |
| *** ansmith has quit IRC | 01:19 | |
| *** boazel_ has joined #tripleo | 01:19 | |
| *** boazel has quit IRC | 01:22 | |
| openstackgerrit | Daniel Farrell proposed openstack/tripleo-heat-templates master: Use new ODL diagstatus heath check REST URL https://review.openstack.org/621052 | 01:34 |
| *** wolverineav has quit IRC | 01:38 | |
| *** ooolpbot has joined #tripleo | 02:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1805870 | 02:10 |
| openstack | Launchpad bug 1805870 in tripleo "mistral unit tests failing in dlrn build of tripleo-common" [Critical,Triaged] - Assigned to Adriano Petrich (apetrich) | 02:10 |
| *** ooolpbot has quit IRC | 02:10 | |
| *** karthiks has quit IRC | 02:17 | |
| *** mrsoul has quit IRC | 02:22 | |
| *** mschuppert has quit IRC | 02:22 | |
| *** mcornea has quit IRC | 02:23 | |
| *** eck` is now known as eck`gone | 02:24 | |
| *** psachin has joined #tripleo | 02:40 | |
| mwhahaha | we had a timeout on undercloud isntall setting up mistral, http://logs.openstack.org/78/620978/1/gate/tripleo-ci-centos-7-undercloud-oooq/191bbd9/logs/undercloud/home/zuul/undercloud_install.log.txt.gz#_2018-11-30_01_02_58 | 03:07 |
| *** ooolpbot has joined #tripleo | 03:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1805870 | 03:10 |
| openstack | Launchpad bug 1805870 in tripleo "mistral unit tests failing in dlrn build of tripleo-common" [Critical,Triaged] - Assigned to Adriano Petrich (apetrich) | 03:10 |
| *** ooolpbot has quit IRC | 03:10 | |
| *** apetrich has quit IRC | 03:15 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Revert "Make tripleo-buildimage-overcloud-full-centos-7 non-voting" https://review.openstack.org/620201 | 03:18 |
| *** ykarel|away has joined #tripleo | 03:22 | |
| *** rlandy has quit IRC | 03:31 | |
| *** psachin has quit IRC | 03:43 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Add missing ws separator between words https://review.openstack.org/619169 | 03:53 |
| *** udesale has joined #tripleo | 04:01 | |
| *** janki has joined #tripleo | 04:03 | |
| *** ooolpbot has joined #tripleo | 04:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1805870 | 04:10 |
| *** ooolpbot has quit IRC | 04:10 | |
| openstack | Launchpad bug 1805870 in tripleo "mistral unit tests failing in dlrn build of tripleo-common" [Critical,Triaged] - Assigned to Adriano Petrich (apetrich) | 04:10 |
| openstackgerrit | yatin proposed openstack/tripleo-quickstart-extras master: Fix regression of collect-logs package listing https://review.openstack.org/620927 | 04:16 |
| *** ramishra has joined #tripleo | 04:29 | |
| *** ratailor has joined #tripleo | 04:50 | |
| *** wolverineav has joined #tripleo | 04:52 | |
| *** ooolpbot has joined #tripleo | 05:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1805870 | 05:10 |
| openstack | Launchpad bug 1805870 in tripleo "mistral unit tests failing in dlrn build of tripleo-common" [Critical,Triaged] - Assigned to Adriano Petrich (apetrich) | 05:10 |
| *** ooolpbot has quit IRC | 05:10 | |
| mwhahaha | marios_|ruck, ssbarnea|rover, weshay: fyi i just realized -> https://bugs.launchpad.net/tripleo/+bug/1805968 Probably need to fix that before all the scenario standalone jobs get turned on :D | 05:14 |
| openstack | Launchpad bug 1805968 in tripleo "standalone job does not run yum update during the container image prepare" [Critical,Triaged] | 05:14 |
| *** wolverineav has quit IRC | 05:15 | |
| openstackgerrit | Merged openstack/tripleo-docs master: Add BIOS settings doc https://review.openstack.org/577343 | 05:16 |
| *** threestrands_ has joined #tripleo | 05:17 | |
| *** threestrands has quit IRC | 05:20 | |
| *** ykarel|away has quit IRC | 05:24 | |
| jaosorior | stevebaker: around? | 05:29 |
| jaosorior | mwhahaha: any progress on the gate issues? | 05:32 |
| mwhahaha | jaosorior: some of my patches for scenarios non voting are in the gate now. It looks like queens upgrades are broken but that's not new | 05:34 |
| jaosorior | mwhahaha: is the broken queens upgrade affecting the gate too? | 05:34 |
| mwhahaha | jaosorior: other than that, we've at least reduced the check backlog by clearing our gate | 05:34 |
| mwhahaha | I don't think so because I don't think anything has a +1 zuul since ipgrades have been broken for weeks | 05:35 |
| *** pcaruana has quit IRC | 05:35 | |
| jaosorior | mwhahaha: is there a bug for that? | 05:36 |
| mwhahaha | Mostly we need to address the time outs but we'll need to take a look at which jobs are having a issues after the scenario patches land | 05:36 |
| mwhahaha | I think so, it's not an alert through (should be) | 05:36 |
| mwhahaha | I've seen it before but I don't have it handy | 05:36 |
| mwhahaha | I think it's failing on the missing policy.json from heat | 05:37 |
| jaosorior | understood, I'll try to poke people to see if anybody knows what bug it is. And add an alert to it. | 05:37 |
| *** hakhande_ has joined #tripleo | 05:37 | |
| mwhahaha | I recall wanting to look into it previously but I always get side tracked | 05:37 |
| jaosorior | mwhahaha: missing policy.json? | 05:37 |
| jaosorior | heat using isn't using policy in code yet? | 05:38 |
| mwhahaha | I think it is but something changed so the undercloud update fails on it | 05:38 |
| jaosorior | OK | 05:38 |
| *** ykarel|away has joined #tripleo | 05:39 | |
| mwhahaha | https://bugs.launchpad.net/tripleo/+bug/1796639 | 05:39 |
| openstack | Launchpad bug 1796639 in tripleo "[stable/queens] The tripleo-ci-centos-7-undercloud-upgrades is failing on queens for No such file or directory: '/etc/heat/policy.json" [Critical,Triaged] | 05:39 |
| *** ykarel|away is now known as ykarel | 05:39 | |
| jaosorior | mwhahaha: thanks, added the alert | 05:40 |
| * mwhahaha wanders off to bed. Back in ~9 hours | 05:41 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/queens: Switch scenarios to non-voting https://review.openstack.org/620978 | 05:41 |
| jaosorior | mwhahaha: have a good one | 05:41 |
| openstackgerrit | Rabi Mishra proposed openstack/tripleo-heat-templates master: WIP set uuid facter explicitly https://review.openstack.org/621072 | 05:53 |
| ramishra | jaosorior: Hey! Are you ok with https://review.openstack.org/#/c/614457/ ? shardy has reviewed it already. | 05:56 |
| jaosorior | ramishra: I have the same question as Steven here: https://review.openstack.org/#/c/614457/10/roles/Standalone.yaml | 05:58 |
| jaosorior | shouldn't we only need the tenant network? | 05:58 |
| jaosorior | given that ctlplane is created by default | 05:58 |
| jaosorior | same here https://review.openstack.org/#/c/614457/10/roles/Standalone.yaml | 05:59 |
| ramishra | jaosorior: Is that question shardy asked? We do need all networks and create the ports as the ServiceNetMap has all those networks specified | 05:59 |
| jaosorior | ramishra: "Ack Ok - perhaps we could optimize this in future to remove the other networks, e.g we probably only need the ctlplane, External and Tenant networks?" | 05:59 |
| ramishra | jaosorior: Nope that would not work unless we override ServiceNetMap | 06:00 |
| jaosorior | I thought we already did | 06:00 |
| jaosorior | but we don't | 06:01 |
| jaosorior | got it | 06:01 |
| jaosorior | that could be left for another iteration | 06:01 |
| jaosorior | I think we should though | 06:01 |
| ramishra | jaosorior: No the earlier solution was not accpetable | 06:01 |
| jaosorior | ramishra: why wasn't it? | 06:01 |
| ramishra | jaosorior: I mean the one I proposed earlier, then the one shardy suggested is possibly a hack and would break in the future | 06:02 |
| ramishra | It's all there in the discussion in the review | 06:02 |
| jaosorior | ah yeah | 06:02 |
| jaosorior | this one https://review.openstack.org/#/c/615363/2/roles/standalone/templates/standalone_config.yaml.j2 | 06:02 |
| jaosorior | I actually like that solution better x_x that's the interfaces that we created for such a thing | 06:02 |
| jaosorior | ramishra: but anyway, the patches you pointed me to is fine. I'll +2 it for now but won't merge it given the situation in the gate. | 06:03 |
| ramishra | the issue is, once we overide ServiceNetMap, we've to override everything in the map again if we want to change anything | 06:03 |
| ramishra | that's a valid concern from alex and steve | 06:04 |
| ramishra | so, we don't change anything for undercloud/standalone as there is no network isolation for them atm | 06:04 |
| ramishra | we can revisit the way to do it later | 06:05 |
| jaosorior | ramishra: well... that's what we ask deployers to do if they wanna change the network. So, if that's not an acceptable way of changing them, then we need to come up with a better way. | 06:05 |
| jaosorior | ramishra: but anyway, I did +2 your patch. | 06:05 |
| ramishra | yeah opinions differ, but we need a common way forward;) | 06:05 |
| *** ooolpbot has joined #tripleo | 06:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1796639 | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1805870 | 06:10 |
| *** ooolpbot has quit IRC | 06:10 | |
| openstack | Launchpad bug 1796639 in tripleo "[stable/queens] The tripleo-ci-centos-7-undercloud-upgrades is failing on queens for No such file or directory: '/etc/heat/policy.json" [Critical,Triaged] | 06:10 |
| openstack | Launchpad bug 1805870 in tripleo "mistral unit tests failing in dlrn build of tripleo-common" [Critical,Triaged] - Assigned to Adriano Petrich (apetrich) | 06:10 |
| ykarel | jaosorior, ovb jobs are broken after https://review.openstack.org/#/c/619519/1 | 06:10 |
| ykarel | marios_|ruck, ssbarnea|rover ^^ | 06:10 |
| jaosorior | ykarel: are you gonna submit a patch to fix it? | 06:11 |
| ykarel | jaosorior, okk will submit, need bug? | 06:12 |
| jaosorior | ykarel: or should I give it a try? | 06:12 |
| ykarel | jaosorior, ok u can fix, just need to remove the debug task | 06:12 |
| jaosorior | ykarel: a bug would be nice. So we can alert on it and folks can keep track of that. | 06:12 |
| ykarel | jaosorior, okk submitting bug first | 06:13 |
| ykarel | jaosorior, https://bugs.launchpad.net/tripleo/+bug/1805972 | 06:15 |
| openstack | Launchpad bug 1805972 in tripleo "OVB jobs failing with: "registered_overcloud_nodes.stdout": "VARIABLE IS NOT DEFINED!"" [Undecided,New] | 06:15 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-quickstart master: Reintroduce get-overcloud-nodes tasks in tripleo-inventory role https://review.openstack.org/621075 | 06:16 |
| jaosorior | ykarel: ^^ | 06:16 |
| openstackgerrit | Kartikeya Jain proposed openstack/diskimage-builder master: Adding support for SLES 15 in element 'sles' https://review.openstack.org/619186 | 06:17 |
| ykarel | jaosorior, +1 | 06:19 |
| ykarel | jaosorior, also added promotion-blocker tag as it's blocking promotion of all realeases | 06:20 |
| jaosorior | ykarel: are you sure that's the issue? | 06:20 |
| jaosorior | I mean, it looks like it'll break things | 06:21 |
| jaosorior | however, the error seems different | 06:21 |
| jaosorior | https://ci.centos.org/job/tripleo-quickstart-gate-master-delorean-quick-basic/7943/console | 06:21 |
| jaosorior | ykarel: or do you have another log? | 06:21 |
| ykarel | jaosorior, see failed jobs: https://trunk.rdoproject.org/api-centos-rocky/api/civotes_detail.html?commit_hash=e77cead2ef9f3669b9dbd65229f4c01e28d13615&distro_hash=689771707fb5a44fbeb78a001092af22266690cf | 06:21 |
| jaosorior | ykarel: got it, thanks | 06:22 |
| ykarel | jaosorior, the link u shared have some other issue | 06:22 |
| ykarel | some syntax problem | 06:22 |
| jaosorior | ykarel: that was a failure in the patch that introduced this bug | 06:23 |
| ykarel | jaosorior, looks like that job is already broken, let me check in some other patch | 06:23 |
| ykarel | jaosorior, so if u see the comments in https://review.openstack.org/#/c/619519/, u can find that job passed sometimes on same patchset | 06:26 |
| jaosorior | strange | 06:26 |
| ykarel | or see https://ci.centos.org/job/tripleo-quickstart-gate-master-delorean-quick-basic/ and job 619519 | 06:27 |
| *** jfrancoa has joined #tripleo | 06:31 | |
| *** flaper87 has quit IRC | 06:32 | |
| jaosorior | mandre: let me know when you're around | 06:34 |
| *** ksambor has joined #tripleo | 06:38 | |
| *** radeks has joined #tripleo | 06:49 | |
| *** radeks_ has joined #tripleo | 06:50 | |
| *** radeks has quit IRC | 06:53 | |
| jaosorior | hjensas: is this still an issue https://bugs.launchpad.net/tripleo/+bug/1800166 ? | 06:55 |
| openstack | Launchpad bug 1800166 in tripleo "tripleo-container-image-prepare :: unrecognized arguments: --log-file" [High,Triaged] | 06:55 |
| Tengu | hello there :) | 06:55 |
| *** mmagr has joined #tripleo | 07:02 | |
| *** quiquell|off is now known as quiquell | 07:03 | |
| jaosorior | mandre: submitted this https://bugs.launchpad.net/tripleo/+bug/1805977 | 07:03 |
| openstack | Launchpad bug 1805977 in tripleo "tripleo image prepare command can't run certain for the overcloud" [High,Triaged] | 07:03 |
| *** mschuppert has joined #tripleo | 07:06 | |
| openstackgerrit | Michele Baldessari proposed openstack/puppet-pacemaker master: Remove unneeded spec testing file https://review.openstack.org/619892 | 07:09 |
| openstackgerrit | Cédric Jeanneret proposed openstack/tripleo-heat-templates master: Ensure we get dedicated logging file for HAProxy https://review.openstack.org/620601 | 07:09 |
| openstackgerrit | Marios Andreou proposed openstack-infra/tripleo-ci master: WIP: Add tripleo-ci-centos-7-scenario004-standalone job https://review.openstack.org/619520 | 07:09 |
| *** ooolpbot has joined #tripleo | 07:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1796639 | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1805870 | 07:10 |
| *** ooolpbot has quit IRC | 07:10 | |
| openstack | Launchpad bug 1796639 in tripleo "[stable/queens] The tripleo-ci-centos-7-undercloud-upgrades is failing on queens for No such file or directory: '/etc/heat/policy.json" [Critical,Triaged] | 07:10 |
| openstack | Launchpad bug 1805870 in tripleo "mistral unit tests failing in dlrn build of tripleo-common" [Critical,Triaged] - Assigned to Adriano Petrich (apetrich) | 07:10 |
| openstackgerrit | Cédric Jeanneret proposed openstack/tripleo-heat-templates master: Ensure we get dedicated logging file for HAProxy https://review.openstack.org/620601 | 07:10 |
| *** kopecmartin|off has quit IRC | 07:10 | |
| *** hamzy__ has joined #tripleo | 07:11 | |
| *** hamzy_ has quit IRC | 07:12 | |
| *** rbrady has quit IRC | 07:12 | |
| *** kopecmartin has joined #tripleo | 07:13 | |
| *** threestrands_ has quit IRC | 07:13 | |
| *** apetrich has joined #tripleo | 07:16 | |
| *** skramaja has joined #tripleo | 07:19 | |
| *** pcaruana has joined #tripleo | 07:22 | |
| *** radeks_ has quit IRC | 07:26 | |
| *** radeks_ has joined #tripleo | 07:26 | |
| openstackgerrit | Cédric Jeanneret proposed openstack/tripleo-heat-templates master: Ensure we actually pass parameters to parent template https://review.openstack.org/621105 | 07:27 |
| *** cylopez has joined #tripleo | 07:30 | |
| *** cylopez has left #tripleo | 07:30 | |
| *** slaweq has joined #tripleo | 07:35 | |
| hjensas | jaosorior: no, we had new container images promoted since that so the specific issue is gone now. (The fact that we can end up with RPM repos promoted and then several days of delay before we promote containers is still a problem. But let's close that bug. :) ) | 07:35 |
| *** radeks__ has joined #tripleo | 07:35 | |
| *** radeks_ has quit IRC | 07:35 | |
| marios | reviews please if you can spare them https://review.openstack.org/#/c/619504/ we are trying to merge the env files so testing scenario standalones is easier thanks | 07:36 |
| jaosorior | marios: can't approve things yet | 07:36 |
| *** florianf|afk is now known as florianf | 07:37 | |
| marios | jaosorior: ack i thought we were clear now at least i saw patches restored by alex? | 07:37 |
| Tengu | some are in the gate, but apparently it's still really slow. | 07:37 |
| marios | jaosorior: thanks i will hold then but if you vote i'll -1 it | 07:37 |
| openstackgerrit | Filippo Inzaghi proposed openstack/python-tripleoclient master: Add Python 3.6 classifier to setup.cfg https://review.openstack.org/621112 | 07:37 |
| marios | Tengu: zuul queue looks much better grafana reckons 7 hours | 07:38 |
| marios | Tengu: but yeah still long | 07:38 |
| Tengu | yeah. was something like 40 yesterday. | 07:38 |
| Tengu | we might need some new power. lately it's going crazy | 07:39 |
| openstackgerrit | Merged openstack/tripleo-docs master: [Documentation] Specify the name of the irc server. https://review.openstack.org/620887 | 07:39 |
| openstackgerrit | Mike Fedosin proposed openstack/tripleo-heat-templates master: Allow to skip docker reconfiguration during stack update https://review.openstack.org/620621 | 07:40 |
| *** cylopez has joined #tripleo | 07:41 | |
| cylopez | jaosorior: hello | 07:41 |
| jaosorior | cylopez: sup | 07:41 |
| cylopez | jaosorior: I'm look about https://review.openstack.org/#/c/620876/, I have 2 ubunutu bionic in failure, I have no idea why | 07:42 |
| jaosorior | cylopez: the error seems unrelated | 07:43 |
| cylopez | jaosorior: oh great | 07:43 |
| jaosorior | so, might be that the ubuntu gate is broken | 07:43 |
| jaosorior | gotta talk to the folks at puppet-openstack :/ | 07:43 |
| cylopez | jaosorior: okok, is a blocker to get it merge ? | 07:43 |
| *** ykarel is now known as ykarel|lunch | 07:43 | |
| jaosorior | yes | 07:44 |
| cylopez | damned | 07:44 |
| cylopez | jaosorior: I should ask for a "recheck" in few days so ? | 07:44 |
| jaosorior | probably | 07:44 |
| cylopez | jaosorior: thx and have a nice friday | 07:45 |
| jaosorior | probablythanks! | 07:46 |
| Tengu | :][ | 07:46 |
| *** cfontain has joined #tripleo | 07:48 | |
| hjensas | cylopez: jaosorior: I saw bandini push a patch to revert to the previous version of ubuntu in puppet-pacemaker yesterday ... | 07:49 |
| cylopez | hjensas: good to know, thx | 07:50 |
| * bandini looks | 07:56 | |
| bandini | cylopez, hjensas: my bionic failures were very specific. Fixed via https://review.openstack.org/#/c/619892/ fwiw | 07:57 |
| openstackgerrit | RedHat RDO CI proposed openstack/tripleo-heat-templates stable/queens: GATE CHECK for TripleO https://review.openstack.org/567224 | 08:00 |
| *** jpena|off is now known as jpena | 08:01 | |
| *** akrivoka has joined #tripleo | 08:03 | |
| *** jtomasek has joined #tripleo | 08:06 | |
| *** rcernin has quit IRC | 08:06 | |
| *** ooolpbot has joined #tripleo | 08:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1796639 | 08:10 |
| openstack | Launchpad bug 1796639 in tripleo "[stable/queens] The tripleo-ci-centos-7-undercloud-upgrades is failing on queens for No such file or directory: '/etc/heat/policy.json" [Critical,Triaged] | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1805870 | 08:10 |
| *** ooolpbot has quit IRC | 08:10 | |
| openstack | Launchpad bug 1805870 in tripleo "mistral unit tests failing in dlrn build of tripleo-common" [Critical,Triaged] - Assigned to Adriano Petrich (apetrich) | 08:10 |
| openstackgerrit | Harald Jensås proposed openstack/tripleo-heat-templates master: Add MTU to neutron networks and nic-config templates https://review.openstack.org/599593 | 08:10 |
| openstackgerrit | Harald Jensås proposed openstack/tripleo-heat-templates master: Use mtu of Tenant network to control neutron global_physnet_mtu https://review.openstack.org/612533 | 08:10 |
| openstackgerrit | Natal Ngétal proposed openstack/tripleo-quickstart master: [Configuration] Remove mitaka file. https://review.openstack.org/617670 | 08:17 |
| therve | apetrich: I think https://bugs.launchpad.net/tripleo/+bug/1805870, the failures are legit, and not in master | 08:19 |
| openstack | Launchpad bug 1805870 in tripleo "mistral unit tests failing in dlrn build of tripleo-common" [Critical,Triaged] - Assigned to Adriano Petrich (apetrich) | 08:19 |
| therve | We expect CI to be broken so often that we don't recognize legit failures anymore :D | 08:21 |
| Tengu | roohhh | 08:22 |
| Tengu | that's nasty ;) | 08:22 |
| *** shardy has joined #tripleo | 08:23 | |
| openstackgerrit | Thomas Herve proposed openstack/python-tripleoclient stable/rocky: Undercloud: Disable persisting VIPs in /etc/hosts https://review.openstack.org/620652 | 08:24 |
| *** shardy has quit IRC | 08:24 | |
| *** shardy has joined #tripleo | 08:24 | |
| *** amoralej|off is now known as amoralej | 08:24 | |
| openstackgerrit | Natal Ngétal proposed openstack/tripleo-docs master: [Documentation] Replace newton by rocky. https://review.openstack.org/620910 | 08:26 |
| Hobbestigrou | hi | 08:29 |
| Hobbestigrou | someone can review this patch please https://review.openstack.org/#/c/620910/ ? | 08:30 |
| openstackgerrit | Marios Andreou proposed openstack-infra/tripleo-ci master: WIP: Add new tripleo-ci-centos-7-scenario001-standalone job https://review.openstack.org/619508 | 08:33 |
| Tengu | hmm. that said, I found "current-tripleo" to be more stable than "current". current-tripleo seems to point to the changes that actually passed the CI. | 08:33 |
| marios | Tengu: current is just master, or at least latest build from master. current-tripleo is things that have passed upstream promotion http://logs.openstack.org/87/616187/1/check/openstack-tox-docs/164c631/html/contributor/stages-overview.html | 08:36 |
| marios | Tengu: so your finding is correct | 08:37 |
| marios | :) | 08:37 |
| Tengu | :) | 08:37 |
| Tengu | I updated my lab in order to use current-tripleo as master was broken. | 08:37 |
| Tengu | now I can play :) | 08:37 |
| *** ykarel|lunch is now known as ykarel | 08:38 | |
| *** xek has joined #tripleo | 08:38 | |
| *** tosky has joined #tripleo | 08:42 | |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-common master: Fix environments storting https://review.openstack.org/621122 | 08:43 |
| *** ccamacho has joined #tripleo | 08:46 | |
| *** salmankhan has joined #tripleo | 08:50 | |
| jtomasek | apetrich: ^ feel free to take that over:) | 08:51 |
| *** jpich has joined #tripleo | 08:53 | |
| *** salmankhan has quit IRC | 08:56 | |
| *** holser_ has joined #tripleo | 09:05 | |
| *** ooolpbot has joined #tripleo | 09:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1796639 | 09:10 |
| *** ooolpbot has quit IRC | 09:10 | |
| openstack | Launchpad bug 1796639 in tripleo "[stable/queens] The tripleo-ci-centos-7-undercloud-upgrades is failing on queens for No such file or directory: '/etc/heat/policy.json" [Critical,Triaged] | 09:10 |
| cylopez | hjensas: hello have time for a bug on os-net-config | 09:10 |
| *** jaganathan__ has left #tripleo | 09:10 | |
| *** bogdando has joined #tripleo | 09:12 | |
| hjensas | cylopez: yes, I can plan it for the next sprint. | 09:13 |
| *** dsneddon has quit IRC | 09:19 | |
| apetrich | therve, my bad. I will check that. It was late and tested only master | 09:22 |
| therve | apetrich: No worries :) | 09:22 |
| d0ugal | therve: the failures are not in master? where are they? | 09:29 |
| therve | d0ugal: patches? | 09:29 |
| d0ugal | oh | 09:29 |
| d0ugal | so we have a bug about errors in patches in gerrit? | 09:29 |
| therve | Yep | 09:29 |
| d0ugal | haha, that is confusing. | 09:30 |
| openstackgerrit | Thomas Herve proposed openstack/tripleo-common master: Set heatclient exceptions to verbose https://review.openstack.org/619039 | 09:31 |
| apetrich | therve, I have a fix for that patch, mind if I submit? | 09:32 |
| *** derekh has joined #tripleo | 09:33 | |
| apetrich | therve, nevermind. Just saw your fix | 09:33 |
| * apetrich is still slow today. more coffee needed | 09:34 | |
| *** gfidente has joined #tripleo | 09:40 | |
| *** yprokule has joined #tripleo | 09:43 | |
| openstackgerrit | Thomas Herve proposed openstack/tripleo-common master: Set heatclient exceptions to verbose https://review.openstack.org/619039 | 09:43 |
| *** hberaud has joined #tripleo | 09:45 | |
| Hobbestigrou | please someone can review this patch https://review.openstack.org/#/c/611596/ ? | 09:49 |
| *** aedc has joined #tripleo | 09:51 | |
| bogdando | folks, WDYT of cronie in containers and https://bugzilla.redhat.com/show_bug.cgi?id=1654659 | 09:52 |
| openstack | bugzilla.redhat.com bug 1654659 in cronie "Optimize cronie RPM spec for systemd requirements when installing it in containers" [Unspecified,New] - Assigned to tmraz | 09:52 |
| bogdando | we have it installed in many services, how actually it works? | 09:52 |
| bogdando | would it work w/o systemd in containers? | 09:52 |
| bogdando | mwhahaha, jistr: any ideas? | 09:53 |
| * jistr no idea on this topic | 09:53 | |
| bogdando | owalsh: perchance? | 09:55 |
| bogdando | https://bugs.launchpad.net/tripleo/+bug/1701254 was the original issue for that | 09:56 |
| openstack | Launchpad bug 1701254 in tripleo "Containerize crond" [High,Fix released] - Assigned to Dan Prince (dan-prince) | 09:56 |
| bogdando | I hope that works https://bugzilla.redhat.com/show_bug.cgi?id=1654659#c6 | 10:01 |
| openstack | bugzilla.redhat.com bug 1654659 in cronie "Optimize cronie RPM spec for systemd requirements when installing it in containers" [Unspecified,New] - Assigned to tmraz | 10:01 |
| openstackgerrit | Saravanan KR proposed openstack/tripleo-heat-templates master: Process the templates even if j2_excludes file is not present https://review.openstack.org/621129 | 10:02 |
| bogdando | folks, PTAL https://review.openstack.org/#/c/620281/ | 10:05 |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-quickstart-extras master: Add export ANSIBLE_LIBRARY to linting so we can find modules https://review.openstack.org/620556 | 10:08 |
| *** ooolpbot has joined #tripleo | 10:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1796639 | 10:10 |
| *** ooolpbot has quit IRC | 10:10 | |
| openstack | Launchpad bug 1796639 in tripleo "[stable/queens] The tripleo-ci-centos-7-undercloud-upgrades is failing on queens for No such file or directory: '/etc/heat/policy.json" [Critical,Triaged] | 10:10 |
| openstackgerrit | Saravanan KR proposed openstack/tripleo-heat-templates master: Deprecate duplicate NFV environment files https://review.openstack.org/621133 | 10:18 |
| openstackgerrit | Rabi Mishra proposed openstack/tripleo-heat-templates master: Set facter variable 'uuid' explicitly in docker-puppet.py https://review.openstack.org/621072 | 10:19 |
| *** akrivoka has quit IRC | 10:26 | |
| *** akrivoka has joined #tripleo | 10:27 | |
| *** slaweq has quit IRC | 10:32 | |
| *** jfrancoa has quit IRC | 10:34 | |
| *** dsneddon has joined #tripleo | 10:37 | |
| *** moguimar has joined #tripleo | 10:40 | |
| *** ramishra has quit IRC | 10:43 | |
| *** ramishra has joined #tripleo | 10:43 | |
| *** pcaruana has quit IRC | 10:44 | |
| therve | hjensas: Hi, around? Wanted to chat about https://review.openstack.org/#/c/599593/ | 10:48 |
| hjensas | therve: yes, I'm here. | 10:49 |
| therve | hjensas: I'd like to reduce the amount of duplication in there | 10:49 |
| therve | hjensas: Is that required to define the parameters everywhere? | 10:49 |
| *** pcaruana has joined #tripleo | 10:50 | |
| hjensas | therve: yeas, since we pass the parameter to the child stack we need to define the parameters. Otherwize Heat complain that we pass it an "unknown". It's like 'ls --foo /tmp' says "unrecognized option foo". | 10:51 |
| hjensas | therve: it's a pita. So if there is a way to get around it I'm all ears. | 10:52 |
| therve | Let's try to understand | 10:52 |
| therve | hjensas: Where are those values actually used? | 10:52 |
| *** jfrancoa has joined #tripleo | 10:53 | |
| hjensas | therve: in all the templates under network/config/ | 10:53 |
| hjensas | therve: and here network/network.j2 | 10:54 |
| therve | OK | 10:54 |
| therve | hjensas: Like when you build MinViableMtu? | 10:54 |
| hjensas | network/ports/port.j2 etc. So basically anything that is'nt static stuff used for CI. | 10:55 |
| therve | hjensas: Wouldn't it be possible to pass a hash map instead of individual names? | 10:56 |
| therve | mtus: {"ctlplane": 1500, "storage": 1500}, etc | 10:56 |
| hjensas | therve: so the biggest reason for the duplication is in https://review.openstack.org/#/c/599593/18/puppet/role.role.j2.yaml | 10:58 |
| therve | Right you pass explicitely the names | 10:58 |
| hjensas | therve: a hash map would work, but it would diverge from what we do for other things, like {{network.name}}IpSubnet, {{network.name}}InterfaceRoutes. | 10:59 |
| *** udesale has quit IRC | 10:59 | |
| therve | hjensas: Aren't those horrible to use too though? | 10:59 |
| *** dtantsur|mtg is now known as dtantsur|afk | 11:00 | |
| therve | Like ci/common/net-config-multinode-os-net-config.yaml has 28 parameters, it uses 3 of them | 11:01 |
| hjensas | therve: yeah, it's kind of crazy. | 11:02 |
| *** iurygregory is now known as iury|lunch | 11:03 | |
| hjensas | therve: but, for an actual user it's https://review.openstack.org/#/c/599593/18/network_data.yaml that need to change. Then nic-config templates can be rendered based on roles data. (as documented in https://review.openstack.org/#/c/596188/7/doc/source/install/advanced_deployment/network_isolation.rst) | 11:04 |
| therve | hjensas: Sure. It still has an impact though | 11:05 |
| therve | First it's basically impossible to review | 11:05 |
| therve | There are 30 interesting lines hidden in 1400 | 11:05 |
| *** hberaud is now known as hberaud|lunch | 11:05 | |
| therve | Then it has a performance impact on Heat | 11:05 |
| therve | We keep duplicating structure that we don't need, but Heat spends useless cycles processing those | 11:06 |
| therve | hjensas: That the very least, the parameters section should be jinja'ed | 11:07 |
| therve | It's the very same for every single config, right? | 11:07 |
| openstackgerrit | Florian Fuchs proposed openstack/tripleo-validations master: Add a lookup plugin to get mistral workflow results https://review.openstack.org/575121 | 11:07 |
| hjensas | therve: yes, I totally get your point. I like the idea of using a hash map instead. | 11:07 |
| therve | It's not perfect because we lose some validation | 11:08 |
| therve | But I think it works here | 11:08 |
| hjensas | therve: I did'nt think adding jinja to the static CI content would be an option. But that would help loads when developing this stuff. | 11:09 |
| owalsh | bogdando: hey, what was the Q? | 11:09 |
| therve | hjensas: Yeah it's possible we can't do that currently | 11:09 |
| hjensas | therve: I wonder how we could introduce a hash map and still be compatible with old templates. | 11:09 |
| *** rfolco is now known as rfolco_doctor | 11:09 | |
| therve | hjensas: Well, we can for introducing that one at least | 11:10 |
| *** morazi has quit IRC | 11:10 | |
| *** ooolpbot has joined #tripleo | 11:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1796639 | 11:10 |
| openstack | Launchpad bug 1796639 in tripleo "[stable/queens] The tripleo-ci-centos-7-undercloud-upgrades is failing on queens for No such file or directory: '/etc/heat/policy.json" [Critical,Triaged] | 11:10 |
| *** ooolpbot has quit IRC | 11:10 | |
| therve | Now for removing the existing ones, it's worth testing | 11:10 |
| therve | I can also look at changing heat so it some cases it doesn't complain when you pass an unknown parameter | 11:12 |
| hjensas | therve: Could we make heat accept missing parameters if they are not referenced by any resource? | 11:12 |
| therve | hjensas: Not sure I get that one | 11:12 |
| therve | You mean accept parameters that aren't defined? | 11:13 |
| *** dmacpher has joined #tripleo | 11:13 | |
| hjensas | therve: I.e we pass parameter 'foo' and bar to the template, but only 'foo' is defined in parameters, and 'bar' is neveren referenced in the resources section. i.e no {get_param: bar} | 11:14 |
| therve | hjensas: Right. Yeah that's worth investigating. | 11:14 |
| therve | It removes a source of validation, but I don't know how useful it is in practice | 11:14 |
| hjensas | therve: then again, validation wize it's kind of good to error on that situation. | 11:15 |
| therve | hjensas: Well, what matters is to fail when you do get_param on something not defined | 11:15 |
| owalsh | bogdando: looks like you've figured it out anyway - rpm deps pulling in systemd, yea? | 11:16 |
| therve | hjensas: Passing something not used... It's bad when there is a typo, and you have a default on you parameter | 11:16 |
| bogdando | owalsh: yea | 11:16 |
| hjensas | therve: yes, those things would be painful to troubleshoot. | 11:16 |
| therve | hjensas: OK, thanks for the dicussion, some food for thoughts :) | 11:17 |
| * therve lunch | 11:17 | |
| hjensas | therve: it would be cool if we could use a hash map, and then put custom constraint in each parameter in the template to ensure what is required is defined in the map. | 11:18 |
| hjensas | therve: that way we would'nt have to duplicate, but still we could raise validation error when appropriate. | 11:19 |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-quickstart-extras master: Add export ANSIBLE_LIBRARY to linting so we can find modules https://review.openstack.org/620556 | 11:20 |
| *** ramishra has quit IRC | 11:20 | |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-quickstart-extras master: Add export ANSIBLE_LIBRARY to linting so we can find modules https://review.openstack.org/620556 | 11:22 |
| *** ramishra has joined #tripleo | 11:26 | |
| tosky | bogdando: about the RDO patches, wouldn't it be possible to use the condition already used in the spec file? %if 0%(?fedora} || 0%{?rhel} > 7 | 11:31 |
| *** raildo has joined #tripleo | 11:32 | |
| *** holser_ is now known as holser|lunch | 11:33 | |
| *** iury|lunch is now known as iurygregory | 11:36 | |
| Hobbestigrou | hi | 11:39 |
| *** cfontain has quit IRC | 11:39 | |
| Hobbestigrou | https://bugs.launchpad.net/tripleo/+bug/1652227/comments/4 | 11:39 |
| openstack | Launchpad bug 1652227 in tripleo "[tripleo-incubator] "tox -e docs" fail" [Low,Triaged] | 11:39 |
| Hobbestigrou | quick question | 11:40 |
| Hobbestigrou | i can change the status to won't fix ? | 11:40 |
| openstackgerrit | Quique Llorente proposed openstack/tripleo-quickstart-extras master: Install correct python version packages at tempest https://review.openstack.org/618721 | 11:47 |
| openstackgerrit | Quique Llorente proposed openstack-infra/tripleo-ci master: Enable tempest for fedora28 standalone https://review.openstack.org/618056 | 11:49 |
| *** slaweq has joined #tripleo | 11:51 | |
| *** pbandark has joined #tripleo | 11:54 | |
| *** hberaud|lunch is now known as hberaud | 12:04 | |
| openstackgerrit | Janki Chhatbar proposed openstack/tripleo-heat-templates master: Don't mount data folder https://review.openstack.org/620053 | 12:10 |
| *** ooolpbot has joined #tripleo | 12:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1796639 | 12:10 |
| *** ooolpbot has quit IRC | 12:10 | |
| openstack | Launchpad bug 1796639 in tripleo "[stable/queens] The tripleo-ci-centos-7-undercloud-upgrades is failing on queens for No such file or directory: '/etc/heat/policy.json" [Critical,Triaged] | 12:10 |
| openstackgerrit | Rabi Mishra proposed openstack/tripleo-heat-templates master: Set facter variable 'uuid' explicitly in docker-puppet.py https://review.openstack.org/621072 | 12:10 |
| *** dprince has joined #tripleo | 12:11 | |
| *** pcaruana has quit IRC | 12:11 | |
| openstackgerrit | Gael Chamoulaud proposed openstack/tripleo-validations master: Avoid 'non-zero return code' msg when ceph-ansible isn't installed https://review.openstack.org/621148 | 12:12 |
| openstackgerrit | Kamil Sambor proposed openstack/tripleo-heat-templates master: Check and enable openvswitch package https://review.openstack.org/619031 | 12:13 |
| dprince | bogdando: Hi, I'm having a lot of trouble following the logic of all of our patches. Would a spec that describes explicitly what you plan on doing make sense? | 12:15 |
| ssbarnea|rover | marios: what is http://logs.openstack.org/57/620457/3/check/heat-functional-orig-mysql-lbaasv2/9ed4f0e/logs/devstacklog.txt.gz ? | 12:15 |
| dprince | bogdando: in light of the recent mailing list discussions I'm not sure why you would still have this posted: https://review.openstack.org/#/c/620281/ | 12:16 |
| openstackgerrit | Dan Radez proposed openstack/puppet-tripleo stable/queens: Add manifest for networking_ansible ML2 plugin https://review.openstack.org/621149 | 12:17 |
| ssbarnea|rover | marios: ^ that one is on one fixing tickets, so unless we sort it it, nothing will merge. | 12:17 |
| ssbarnea|rover | ohh, damn looks like flakiness. | 12:18 |
| ssbarnea|rover | i would put a query on it but I don't know what string to look for. | 12:18 |
| *** slaweq has quit IRC | 12:24 | |
| *** udesale has joined #tripleo | 12:26 | |
| fultonj | marios: how's the scen001 multi to standalone conversion going? | 12:27 |
| fultonj | marios: i think we were going to talk about that tuesday, but ran out of meeting time but wondered if there was anything you wanted input on from gfidente and me | 12:28 |
| marios | fultonj: o/ we are almost there. latest issue we had was module path for the linting fixed with Depends-On: https://review.openstack.org/620556 # t-q-e | 12:28 |
| marios | fultonj: the deployment is at least passing so we are almost at a green job | 12:29 |
| marios | fultonj: gfidente sorry about the call, i was hoping it would be more useful to you | 12:29 |
| marios | fultonj: mostly was interested in any input with the featureset_override we were discussing, as users of the job | 12:29 |
| fultonj | marios: np, usually the call is. | 12:29 |
| gfidente | marios I am not super expert of that anyway | 12:30 |
| fultonj | oh, i think i like featureset_override | 12:30 |
| fultonj | though, i'm not sure i understand all of the implications | 12:30 |
| marios | gfidente: fultonj can you folks sanity check the env files please scen1 /4 wrt ceph if you haven't already thanks https://review.openstack.org/#/c/619508/ https://review.openstack.org/619520 (depends on has the tht env file) | 12:33 |
| marios | gfidente: fultonj we quiquell and i agreed to start merging the envs at least | 12:34 |
| *** rfolco_doctor is now known as rfolco | 12:34 | |
| fultonj | marios: i'm looking for the logs collected from the new scen001 | 12:35 |
| marios | gfidente: fultonj scen1 is voted already (but check if you see nits please and 4 still needs vote @ https://review.openstack.org/#/c/619504/ ) | 12:35 |
| marios | fultonj: thanks | 12:35 |
| marios | fultonj: so they are both currently waiting for check after i updated the depends-on (the path thing) | 12:36 |
| marios | fultonj: i mean for latest run. but lemme find you a 'good' one sec | 12:36 |
| marios | fultonj: here is one with deployment passing http://logs.openstack.org/08/619508/7/check/tripleo-ci-centos-7-scenario001-standalone/08c9ed2/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz | 12:38 |
| fultonj | marios: tks | 12:38 |
| marios | fultonj: thanks for checking. might be worth looking later to se the latest run once it happens if u have time thank you | 12:39 |
| *** holser|lunch is now known as holser_ | 12:39 | |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates master: Add DockerPuppetMountHostPuppet parameter https://review.openstack.org/550848 | 12:40 |
| *** jpena is now known as jpena|lunch | 12:41 | |
| *** xek has quit IRC | 12:46 | |
| openstackgerrit | Kamil Sambor proposed openstack/tripleo-heat-templates master: Check and enable openvswitch package https://review.openstack.org/619031 | 12:46 |
| *** ansmith has joined #tripleo | 12:46 | |
| *** xek has joined #tripleo | 12:47 | |
| *** hakhande_ has quit IRC | 12:47 | |
| *** morazi has joined #tripleo | 12:49 | |
| *** iurygregory is now known as iurygregory|mtg | 12:59 | |
| *** zul has joined #tripleo | 13:06 | |
| *** vpickard_ is now known as vpickard | 13:07 | |
| *** salmankhan has joined #tripleo | 13:09 | |
| *** ooolpbot has joined #tripleo | 13:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1796639 | 13:10 |
| *** ooolpbot has quit IRC | 13:10 | |
| openstack | Launchpad bug 1796639 in tripleo "[stable/queens] The tripleo-ci-centos-7-undercloud-upgrades is failing on queens for No such file or directory: '/etc/heat/policy.json" [Critical,Triaged] | 13:10 |
| *** ratailor has quit IRC | 13:11 | |
| *** abishop has joined #tripleo | 13:13 | |
| *** trown|outtypewww has quit IRC | 13:17 | |
| *** trown|brb has joined #tripleo | 13:18 | |
| openstackgerrit | Quique Llorente proposed openstack/tripleo-quickstart-extras master: Install correct python version packages at tempest https://review.openstack.org/618721 | 13:18 |
| openstackgerrit | Cédric Jeanneret proposed openstack/paunch master: Correct error output https://review.openstack.org/621157 | 13:18 |
| openstackgerrit | Hervé Beraud proposed openstack/tripleo-heat-templates stable/rocky: Remove unnecessary logs due to misconfiguration from pacemaker https://review.openstack.org/621159 | 13:25 |
| *** iurygregory|mtg is now known as iurygregory | 13:29 | |
| cylopez | jaosorior: it's green now https://review.openstack.org/#/c/620876/ | 13:30 |
| cylopez | :-) | 13:30 |
| *** skramaja has quit IRC | 13:33 | |
| openstackgerrit | Hervé Beraud proposed openstack/tripleo-heat-templates stable/queens: Remove unnecessary logs due to misconfiguration from pacemaker https://review.openstack.org/613054 | 13:35 |
| *** rlandy has joined #tripleo | 13:36 | |
| *** jpena|lunch is now known as jpena | 13:40 | |
| pbandark | hi.. on undercloud node, after introspecting the nodes, when I execute `openstack hypervisor list` command, I can see undercloud IP for all available nodes in "Host IP" column. Is it expected? https://pastebin.com/XxXm6nDH | 13:43 |
| *** vinaykns has joined #tripleo | 13:45 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Add an element to configure iBFT network interfaces https://review.openstack.org/391787 | 13:46 |
| *** mmagr has quit IRC | 13:46 | |
| *** udesale has quit IRC | 13:48 | |
| jaosorior | marios: at a quick glance, what do you think of this docs commit https://review.openstack.org/#/c/620921/ ? | 13:49 |
| *** udesale has joined #tripleo | 13:49 | |
| *** dmacpher has quit IRC | 13:51 | |
| *** eck`gone is now known as eck` | 13:52 | |
| openstackgerrit | Rabi Mishra proposed openstack/tripleo-heat-templates master: Set facter variable 'uuid' explicitly in docker-puppet.py https://review.openstack.org/621072 | 13:53 |
| *** slaweq has joined #tripleo | 13:56 | |
| *** jcoufal has joined #tripleo | 13:57 | |
| *** EmilienM is now known as EvilienM | 13:58 | |
| Tengu | bandini: hmm, you didn't set a BJ session for the meeting about the podman/pcmk rpms, did you? | 14:01 |
| bandini | Tengu: I added it later :) | 14:04 |
| *** amoralej is now known as amoralej|lunch | 14:04 | |
| bandini | Tengu: https://redhat.bluejeans.com/4884650097/?src=meet_now | 14:05 |
| *** tzumainn has joined #tripleo | 14:05 | |
| apetrich | jtomasek, just to make sure I'm doing the correct thing. I'm looking at this https://review.openstack.org/#/c/621122/ | 14:07 |
| dprince | bogdando: Trying to refine the idea here (constructively) but I'm having a hard time following some of your comments https://review.openstack.org/#/c/620909. Can you explain exactly which part you are -2'ing | 14:07 |
| *** ksambor has quit IRC | 14:07 | |
| *** ykarel is now known as ykarel|away | 14:07 | |
| dprince | bogdando: fwiw, I think a spec is a better place to refine an idea sometimes than the mailing list | 14:08 |
| bogdando | dprince: the blocker is "a no go as is wrt having puppet executed 1 time centrally..." | 14:08 |
| fultonj | gfidente: when you get a chance would you please reivew https://review.openstack.org/#/q/topic:bug/1800232 ? | 14:08 |
| bogdando | which is expanded in comments | 14:08 |
| bogdando | see from hjensas , dprince | 14:09 |
| bogdando | There is a bunch of bootstrap-host based coordination tasks like creating things with openstack services puppet providers and doing db-sync et al tasks. So the approach need to be updated like once for configs plus once for bootstrap tasks of each a service. | 14:09 |
| bogdando | while the former should be done at undercloud, 1 time onlym indeed, the latter would require 1 per service run for control plane | 14:10 |
| dprince | bogdando: I'm aware of that. I created boostrap-host-exec | 14:10 |
| bogdando | at least 1 run | 14:10 |
| *** ooolpbot has joined #tripleo | 14:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1796639 | 14:10 |
| *** ooolpbot has quit IRC | 14:10 | |
| openstack | Launchpad bug 1796639 in tripleo "[stable/queens] The tripleo-ci-centos-7-undercloud-upgrades is failing on queens for No such file or directory: '/etc/heat/policy.json" [Critical,Triaged] | 14:10 |
| marios | jaosorior: in a call will check in a bit | 14:10 |
| dprince | bogdando: I think we can solve that problem differently perhaps. | 14:10 |
| bogdando | not sure the spec explains that clearly, so I set a blocker flag | 14:10 |
| dprince | bogdando: so you are -2 because the spec needs to refine the idea more then? A bit harsh I think | 14:11 |
| marios | jaosorior: makes sense but want to look more closely/not during a call :) | 14:11 |
| marios | jaosorior: will add for next reviews | 14:11 |
| bogdando | well, also as it looked like a no go to change what we have | 14:11 |
| bogdando | as we'd still need to run puppet many times | 14:11 |
| jtomasek | apetrich: yes | 14:11 |
| *** hjensas has quit IRC | 14:12 | |
| dprince | bogdando: the idea is to minimize where puppet has to run. Even if we ran it still on the bootstrap nodes that would be a lesser offense. And I think even that could be improved too in time | 14:12 |
| dprince | bogdando: step at a time... but in the right direction | 14:12 |
| bogdando | ok, if we can solve this, I'll remove the blocker, just update the spec please then | 14:13 |
| bogdando | I thought we cant, rbh | 14:13 |
| *** toure|biab is now known as toure | 14:13 | |
| bogdando | just wondering if would have to run it on computes still | 14:13 |
| bogdando | e.g. for DVR/OVN | 14:13 |
| bogdando | then we have a blocker again | 14:14 |
| dprince | bogdando: I was trying to propose a solutio to your problem on the list... some of these details can be worked out I think | 14:15 |
| dprince | bogdando: etcd is perhaps my favorite solution, but would likely involve a larger development effort | 14:15 |
| jaosorior | marios: thanks | 14:16 |
| dprince | bogdando: this was sort of a hybrid, a step towards optimizing our configs without breaking what we support today | 14:16 |
| beagles | bogdando, dprince: probably an issue wherever there are role specific parameters | 14:16 |
| dprince | Not breaking upgrades, and supporting all the config options has always been the tricky part of our config management tooling | 14:17 |
| beagles | (i.e. on where puppet config stuff needs to run) | 14:17 |
| dprince | beagles: perhaps a role specific "overlay" could be implemented | 14:17 |
| beagles | interesting idea | 14:18 |
| dprince | beagles: I think we probably want to move towards the concept of pods/services and away from roles a bit in doing this. | 14:18 |
| beagles | dprince, ah right | 14:19 |
| dprince | beagles: so perhaps we'd actually make more unique services which configured things slightly differently | 14:19 |
| bogdando | dprince: running puppet multiple times is rather an orthogonal problem to having it in each layer | 14:19 |
| bogdando | but they a causal related indeed ) | 14:19 |
| bogdando | in each container* via base layer | 14:19 |
| dprince | bogdando: As it is today where it exists in the base layer is the best place for it. I've been convinced of that for quite a while. Do note when we implemented docker-puppet.py we tried putting in in a few places in the base made the most sense | 14:21 |
| bogdando | dprince: I'd rather agree with that jistr evaluates | 14:21 |
| bogdando | a) detach systemd off those, b) move puppet and manifests off the base to upper | 14:22 |
| cschwede | Hi! Can someone please review + approve this tiny fix: https://review.openstack.org/#/c/620650/ ? That would be great! | 14:22 |
| dprince | bogdando: these problems are related... because how you generate configs infuences where the tooling needs to be in the container layers | 14:22 |
| *** ksambor has joined #tripleo | 14:22 | |
| bogdando | this solves all of the security concerns by the price of not much traffic increased | 14:22 |
| dprince | bogdando: I didn't read jistr's comments the same way. I think you misunderstood something | 14:22 |
| bogdando | but that is only needed if we would not have that spec implemented that you're proposing, dprince, ofc | 14:22 |
| bogdando | otherwise, a golden config image stored only at UC is a way to go | 14:23 |
| bogdando | btw, my latest comment was for exatly that | 14:23 |
| bogdando | coult you please clarify that also in the spec? | 14:24 |
| *** janki has left #tripleo | 14:24 | |
| *** mcornea has joined #tripleo | 14:24 | |
| *** janki has joined #tripleo | 14:24 | |
| jistr | bogdando, dprince: yea i was advocating *against* doing the layer restructuring because of the excessive bandwidth/space costs. FWIW i didn't get to read dprince's spec yet so i don't have a complete picture of the landscape right now, will try to catch up though. | 14:24 |
| dprince | bogdando: moving puppet out of the base image isn't something we should do until we have a better mechanism for generating config files. If you intend to do this you misunderstood something | 14:24 |
| bogdando | dprince: why? he evaluated and declined, while agree with the former only :) | 14:24 |
| dprince | bogdando: my spec, was an idea which would enable us to move things around | 14:25 |
| jaosorior | :qa | 14:25 |
| bogdando | I agreed with an approach is feasible, had we detached systemd | 14:25 |
| bogdando | dprince: moving puppet out of the base image is that we can do right now | 14:26 |
| dprince | bogdando: minimizing dependencies in the base image is great. But approach it as a packaging issue for now. Not a layering issue I think | 14:26 |
| bogdando | and change nothing but the runtime vs config tag | 14:26 |
| bogdando | but the size / traffic impact would be high | 14:26 |
| bogdando | if we removed systemd , it'd much better | 14:26 |
| bogdando | and feasible IMO | 14:26 |
| dprince | bogdando: if you do it you will actually cost more storage/network traffic. I don't even need to run any tests to tell you that | 14:26 |
| dprince | bogdando: it defeats your purposes | 14:26 |
| *** rbrady has joined #tripleo | 14:26 | |
| bogdando | dprince: not that much if systemd goes away | 14:27 |
| bogdando | a decent price for improved security | 14:27 |
| dprince | bogdando: "improved security". you keep going back to that. We don't have numbers for this yet do we? | 14:27 |
| jistr | dprince, bogdando: yea that was also one of my points in the first e-mail i sent to the thread. How serious/real is the security concern here. | 14:28 |
| beagles | cschwede, reviewed but we approvals are being held back until the gate gets in proper shape ( see channel status ) | 14:28 |
| cschwede | beagles: thx a lot! | 14:28 |
| dprince | bogdando: while I logically like the idea of moving puppet out of the base image, there is a storage/network cost to doing so. We know that. It isn't a guess | 14:29 |
| bogdando | dprince, jistr: I have no numbers neither do I know from where to get those, sorry. And I think the last a couple of releases, we have architecture choices dictated not really any numbers driven | 14:30 |
| dprince | bogdando: the cost is actually quite high. Several gigs of data I think | 14:30 |
| bogdando | security concerns is never real until a user is hit with it, then it becomes very serios | 14:30 |
| *** janki has quit IRC | 14:30 | |
| bogdando | that is why they created the defences in depth principle, to have things improved w/o getting any numbers | 14:31 |
| dprince | bogdando: we can make that decision I guess. If the team wants to have "logically correct" containers we can do that today. I don't think it is a good idea however. Leaving PUppet in the base image is a small cost for the network/storage benefits we know that exist | 14:31 |
| bogdando | dprince: sadly, systemd won't be decoupled until fedora29 I think | 14:33 |
| dprince | bogdando: what we can do, is refactor how are config files are generated. Consider things future things like etcd (again), and perhaps even smaller things like adjusting how/where we run puppet. Doing this will allow us to restructure the container layers to be more logical | 14:33 |
| bogdando | so we'd have to postpone | 14:33 |
| bogdando | but would be nice to do something with the spec clarified for all of the aforementioned places | 14:33 |
| bogdando | and give it a try with a golden image | 14:33 |
| *** ykarel|away has quit IRC | 14:33 | |
| bogdando | if the latter is never supposed to be pushed overcloud and/or used for running anything | 14:33 |
| dprince | bogdando: you can try the golden image today I think by setting all of the Docker*Image params to the same image. It might just work | 14:34 |
| jistr | dprince, bogdando: one thing i'm thinking from this conversation -- no improvement on this front can be realistically made until we decouple systemd from puppet. And after we do so, isn't the main security concern solved by that, even without the restructuring? I mean how dangerous is a puppet in a container, if i have bash in that container too? They won't give you much differing capabilities, will they? | 14:34 |
| dprince | jistr: I think the systemd decoupling problem stands on its own. We should do it regardless | 14:35 |
| bogdando | dprince: but trying a golden image buys nothing w/o a required change to remove config bits off runtime images | 14:35 |
| bogdando | and we could see that doing only either of it is rather a counterproductive | 14:36 |
| bogdando | we'd need both | 14:36 |
| dprince | jistr: but yes, solving that problem I think does resolve a good bit of the potential security concerns | 14:36 |
| jistr | dprince, bogdando: yea. So IMO let's focus on fixing the packaging, dropping systemd. And then we can follow up whether it's still worth to address this further, no? | 14:36 |
| dprince | bogdando: we've covered many different topics and I feel like you keep switching back and forth on them | 14:37 |
| dprince | bogdando: the reason the all-in-one (golden) image was suggested was a means to minimize network/storage traffic to your Edge nodes. That was a single topic, unrelated to other onces | 14:37 |
| dprince | ones | 14:37 |
| bogdando | dprince: yes, that's what the spec positions for, and I'm totally for it, if we could avoid run puppets for compute roles | 14:37 |
| jistr | bogdando: WDYT about focusing on the systemd drop for now? I think that adds a ton of value for all the possible follow up scenarios. | 14:38 |
| dprince | jistr: I'm in agreement with the systemd packaging fixes. +2 on all that. | 14:38 |
| bogdando | jistr: so we'd have to remove bash, ruby, puppet and it would be much better ;) | 14:38 |
| bogdando | I had to fix containers for the missing perl, (perl!) to awk | 14:39 |
| dprince | bogdando: you can't remove Puppet until we address the other issues. | 14:39 |
| bogdando | and now we want to keep python and ruby there :D | 14:39 |
| bogdando | please don't | 14:39 |
| jistr | if we drop python we need to drop openstack too :D | 14:40 |
| bogdando | jistr: so my take is, yes, it still would be a security issue even w/o systemd | 14:40 |
| bogdando | but I better off stop eating jaosorior's cake now :D | 14:40 |
| jaosorior | bogdando: security is for everyone, not just the security team ;) | 14:41 |
| bogdando | jistr: yes, I have some patches in work for decoupling systemd | 14:41 |
| Tengu | a step by step approach is always the best. first the obvious things, then dig deeper in the meat. | 14:42 |
| jistr | bogdando: cool :) Personally i think dropping systemd from all the images is where we get the most value per amount of work/trouble. | 14:42 |
| bogdando | dprince: I consider all-in-one *config* image only, for UC only. And I'm against using it for far edges as a silver bullet for *runtime* | 14:42 |
| bogdando | if we'd still need that image to config control plane, that's ok | 14:43 |
| bogdando | not for runtime please | 14:43 |
| dprince | bogdando: it was a suggestion based on your LP ticket. Use it or not | 14:43 |
| dprince | bogdando: given the architecture today, my guess is that gives you the smallest network traffic and storage requirements to your edge nodes. | 14:43 |
| bogdando | and that is totally related to removing any of the config bits off other containers, dprince | 14:43 |
| bogdando | jistr: http://lists.openstack.org/pipermail/openstack-discuss/2018-November/000393.html | 14:44 |
| dprince | bogdando: depending on the solution they might be related things. But there is an order of operations. By choosing to move PUppet out of the base image first it would actually make many of the problems (storage/network) much worse for edge clouds I think | 14:45 |
| openstackgerrit | Merged openstack/puppet-pacemaker master: Remove unneeded spec testing file https://review.openstack.org/619892 | 14:45 |
| dprince | bogdando: all that because some of us *think* a puppet/ruby security update might be a problem. This isn't something we should do right now | 14:46 |
| bogdando | dprince: yes, we can't make this to puppet first. 1) decouple systemd, 2) the spec of yours with a golden image and etcd, 3) remove all config bits off runtime containers images | 14:46 |
| bogdando | that's how I see it, jistr^^ | 14:47 |
| bogdando | but there is also a discussible 1.5) | 14:47 |
| bogdando | :) | 14:47 |
| bogdando | which is move puppet and modules on top of the layers | 14:47 |
| bogdando | and improve security right now | 14:47 |
| dprince | bogdando: your last statement is close. (I almost agree with it). THe bit about the golden image however is an optional thing for the spec. But we can perhaps clarify that elsewhere | 14:48 |
| bogdando | with a very little overhead of a 80MB layer | 14:48 |
| dprince | bogdando: golden image to me is an optimization that would be opt-in perhaps. Not always useful.... but perhaps useful in edge clouds | 14:48 |
| bogdando | dprince: we're understanding it differently still, dprince | 14:50 |
| bogdando | I'm against using it for edge, runtime | 14:50 |
| bogdando | and I'm failing to see how we could remove all config-time bits off containers w/o that *config* only a golden image | 14:50 |
| bogdando | not an opt-in to me, but a mandatory part of the spec to add it there | 14:51 |
| jistr | bogdando: re (2) and (3), i still need to read the dprince's spec. I'm +2 on (1). For now i'm -1 on the discussible (1.5). Regarding (1.5) i think we've discussed it long enough, there's no agreement, and we can't do it anyway until (1) happens. So let's drop discussion about (1.5) for now and focus on (1) which we seem to all agree with. My 2 cents :) | 14:51 |
| *** brault has quit IRC | 14:51 | |
| *** ykarel|away has joined #tripleo | 14:52 | |
| *** ykarel|away is now known as ykarel | 14:52 | |
| bogdando | the all-in-one runtime vs config in the spec scope is the most that concerns me, jistr | 14:53 |
| bogdando | please do not attempt to push an idea for golden image for a runtime folks | 14:53 |
| bogdando | a no go | 14:53 |
| dprince | bogdando: I'm not pushing golden images, it was simply an idea. But do note that my spec does not require it | 14:55 |
| bogdando | dprince: could you elaborate? | 14:56 |
| bogdando | 1) we have to run puppet for a CP that fits all those db manage | 14:56 |
| bogdando | 2) we aiming to remove all config bits off images | 14:56 |
| dprince | jistr: agree that we have spent way to much time on this. the reason I keep talking about is there are still patches posted which are relayering some of our containers.... some with +2's on them | 14:56 |
| bogdando | 3) so how would we do that w/o a config-time golden image? | 14:56 |
| bogdando | please explain in the spec, dprince | 14:57 |
| * jistr looks for the patches | 14:57 | |
| bogdando | it's very confusing to get it otherwise, sorry | 14:57 |
| dprince | bogdando: yes, this is why I created the spec. Lets clarify it there | 14:58 |
| bogdando | ok, great. thanks folks | 14:58 |
| jistr | we shouldn't have +2s on the relayering right now, given there's no general agreement | 14:58 |
| bogdando | we have nothing left WIP for relayering jistr | 14:58 |
| bogdando | what are those +2s you're talking about? | 14:59 |
| dprince | bogdando: sorry, the kolla image from this morning. Did you remove that? | 14:59 |
| bogdando | yes | 14:59 |
| bogdando | and that was a leftover | 14:59 |
| bogdando | the patch was about another thing, changing FROM, dprince | 14:59 |
| jistr | ok sounds like we're ok for now then :) | 14:59 |
| bogdando | and I still think we should not use openstack-base for tripleoclient image | 15:00 |
| bogdando | but that's unrelated to puppet | 15:00 |
| bogdando | dprince: ^^ | 15:00 |
| dprince | bogdando: that is perhaps another thing, but we can debate that separately. The Puppet was what confused me | 15:00 |
| bogdando | my point is if it works only from the openstack-base, we have dependencies issues for trilpeoclient | 15:01 |
| bogdando | as it should be working jsut fine off the centos base | 15:01 |
| bogdando | and install anything it wants via depencencies | 15:01 |
| bogdando | and not pull in the openstack-base world | 15:01 |
| dprince | bogdando: that may be, but why not share a base image when you can? | 15:01 |
| bogdando | ok, I'll revive the patch then, but removing puppet leftovers | 15:01 |
| dprince | bogdando: again, this goes towards layering. | 15:01 |
| bogdando | dprince: well, to end up with a smaller image? | 15:02 |
| bogdando | tripleo deps << openstack world | 15:02 |
| bogdando | (much less than) | 15:02 |
| bogdando | dprince: yeah, lightweight containers, think of it like that, not just layering :) | 15:05 |
| bogdando | a huge topic indeed | 15:05 |
| *** ooolpbot has joined #tripleo | 15:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1796639 | 15:10 |
| *** ooolpbot has quit IRC | 15:10 | |
| openstack | Launchpad bug 1796639 in tripleo "[stable/queens] The tripleo-ci-centos-7-undercloud-upgrades is failing on queens for No such file or directory: '/etc/heat/policy.json" [Critical,Triaged] | 15:10 |
| mwhahaha | ykarel: do you have a patch for a partial revert of https://review.openstack.org/#/c/619519/1/roles/tripleo-inventory/tasks/inventory.yml or do you want us to revert that patch | 15:10 |
| ykarel | mwhahaha, already have a patch https://review.openstack.org/#/c/621075/ | 15:11 |
| mwhahaha | k | 15:11 |
| ykarel | can't get why the bug is not in alert? | 15:12 |
| weshay | mwhahaha, we can't have nice things | 15:12 |
| mwhahaha | that's ok, i don't like nice things anyway | 15:13 |
| weshay | lolz | 15:13 |
| *** chandan_kumar is now known as chkumar|off | 15:14 | |
| mwhahaha | weshay: so given that i cleared out the gate yesterday and it's already back 11+ hours, what failed in the gate | 15:15 |
| *** jcoufal has quit IRC | 15:16 | |
| ssbarnea|rover | mwhahaha: regarding https://review.openstack.org/#/q/status:open++topic:scenarios-nv -- the two queens branch are failing due to https://bugs.launchpad.net/tripleo/+bug/1796639 -- what can we do about it? | 15:16 |
| openstack | Launchpad bug 1796639 in tripleo "[stable/queens] The tripleo-ci-centos-7-undercloud-upgrades is failing on queens for No such file or directory: '/etc/heat/policy.json" [Critical,Triaged] | 15:16 |
| mwhahaha | ssbarnea|rover: fix https://bugs.launchpad.net/tripleo/+bug/1796639 ? | 15:16 |
| weshay | mwhahaha, still is ovh bs1 | 15:16 |
| * mwhahaha enjoys stating the obvious | 15:16 | |
| mwhahaha | i mean that bug has been open for 2 months now | 15:16 |
| weshay | http://dashboard-ci.tripleo.org/d/cEEjGFFmz/cockpit?orgId=1&panelId=61&fullscreen | 15:16 |
| mwhahaha | perhaps we should pay attnetion to it | 15:16 |
| mwhahaha | let me go spin up a reproducer i guess | 15:17 |
| mwhahaha | i think i started to previously and got side tracked with the rest of the world falling apart | 15:17 |
| weshay | hrm.. couple real issues | 15:17 |
| weshay | marios, http://logs.openstack.org/68/619468/3/gate/tripleo-ci-centos-7-standalone/d418ae2/logs/stackviz/#/testrepository.subunit/test-details/tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_hotplug_nic | 15:18 |
| openstackgerrit | Nicolas Hicher proposed openstack-infra/tripleo-ci master: testenv-worker: add keystone v3 support https://review.openstack.org/604470 | 15:18 |
| weshay | that failed in the gate.. let's add it to the skip list, w/ a bug for chandan and arx to look at | 15:18 |
| ssbarnea|rover | mwhahaha: my current nickname suffix does not allow me to *fix* things,... | 15:19 |
| weshay | chem, matbu marios /me looking at http://logs.openstack.org/92/616592/7/gate/tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates/8147d4d/job-output.txt.gz#_2018-11-30_09_48_55_958592 | 15:19 |
| weshay | haven't seen that error before | 15:19 |
| mwhahaha | ssbarnea|rover: :D i'm going to tackle that one today | 15:19 |
| ssbarnea|rover | mwhahaha: thank you, I will assign it to you and look for the next one. | 15:20 |
| weshay | marios, bug #2 http://logs.openstack.org/92/616592/7/gate/tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates/8147d4d/logs/undercloud/home/zuul/overcloud_update_data_migration.log.txt.gz#_2018-11-30_09_48_55 | 15:20 |
| chem | weshay: in the middle of something will have a look later | 15:20 |
| weshay | mwhahaha, two real bugs, rest are timeouts | 15:20 |
| weshay | chem++ | 15:20 |
| weshay | thanks | 15:20 |
| weshay | I have to step out and get my stupid car fixed... | 15:21 |
| mwhahaha | k | 15:21 |
| weshay | marios, ssbarnea|rover can we get those two issues written up please and one skip list for basicops issue | 15:21 |
| ssbarnea|rover | added to https://review.rdoproject.org/etherpad/p/ruckrover-sprint23 | 15:23 |
| openstackgerrit | Michele Baldessari proposed openstack/puppet-pacemaker master: Match node properties more strictly https://review.openstack.org/620892 | 15:25 |
| marios | weshay: ack | 15:25 |
| openstackgerrit | Nicolas Hicher proposed openstack-infra/tripleo-ci master: vexxhost: ensure use private ip for te-broker https://review.openstack.org/621182 | 15:25 |
| *** hberaud is now known as hberaud|afk | 15:26 | |
| *** dsneddon has quit IRC | 15:28 | |
| *** mmagr has joined #tripleo | 15:29 | |
| *** salmankhan has quit IRC | 15:29 | |
| *** Vorrtex has joined #tripleo | 15:30 | |
| *** dsneddon has joined #tripleo | 15:31 | |
| *** bnemec is now known as beekneemech | 15:34 | |
| jaosorior | alright! off now. Have a good weekend everyone! | 15:36 |
| *** ramishra has quit IRC | 15:38 | |
| *** hberaud|afk is now known as hberaud | 15:38 | |
| *** amoralej|lunch is now known as amoralej | 15:39 | |
| *** quiquell is now known as quiquell|off | 15:39 | |
| openstackgerrit | Sergii Golovatiuk proposed openstack/python-tripleoclient master: Stop undercloud upgrade if deployment tasks fail https://review.openstack.org/620905 | 15:42 |
| *** dansmith is now known as SteelyDan | 15:43 | |
| openstackgerrit | Bogdan Dobrelya proposed openstack/paunch master: Add missing default-runtime support bits https://review.openstack.org/608632 | 15:44 |
| chem | bandini: EvilienM ah, yes, fedora28 ... no upgrade there | 15:46 |
| marios | ssbarnea|rover: weshay so it hink those two are the same wes i mean 17:19 < weshay> chem, matbu marios /me looking at | 15:47 |
| marios | http://logs.openstack.org/92/616592/7/gate/tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates/8147d4d/job-output.txt.gz#_2018-11-30_09_48_55_958592 | 15:47 |
| marios | 17:20 < weshay> marios, bug #2 | 15:47 |
| marios | http://logs.openstack.org/92/616592/7/gate/tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates/8147d4d/logs/undercloud/home/zuul/overcloud_update_data_migration.log.txt.gz#_2018-11-30_09_48_55 | 15:47 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/paunch master: Add missing default-runtime support bits https://review.openstack.org/608632 | 15:47 |
| marios | ssbarnea|rover: weshay i filed https://bugs.launchpad.net/tripleo/+bug/1806068 | 15:47 |
| openstack | Launchpad bug 1806068 in tripleo "centos-7-scenario000-multinode-oooq-container-updates job failing overcloud_online_data_migration.sh.txt.gz with "u'Error creating static inventory"" [Undecided,Triaged] - Assigned to Marios Andreou (marios-b) | 15:47 |
| marios | weshay: nm i see the testops one looking now | 15:49 |
| marios | ssbarnea|rover: did you dig there already? | 15:49 |
| *** thrash is now known as thrash|biab | 15:51 | |
| *** ade_lee has joined #tripleo | 15:53 | |
| *** ksambor has quit IRC | 15:57 | |
| openstackgerrit | John Fulton proposed openstack/tripleo-common master: Add python-notario package to mistral-executor container https://review.openstack.org/621193 | 15:57 |
| *** mmagr has quit IRC | 16:02 | |
| *** hjensas has joined #tripleo | 16:04 | |
| *** artom is now known as temka | 16:08 | |
| *** ooolpbot has joined #tripleo | 16:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1796639 | 16:10 |
| *** ooolpbot has quit IRC | 16:10 | |
| openstack | Launchpad bug 1796639 in tripleo "[stable/queens] The tripleo-ci-centos-7-undercloud-upgrades is failing on queens for No such file or directory: '/etc/heat/policy.json" [Critical,Triaged] - Assigned to Alex Schultz (alex-schultz) | 16:10 |
| *** rbrady has quit IRC | 16:11 | |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-common master: Increase ansible gather_timeout to 30secs for config-download https://review.openstack.org/621207 | 16:15 |
| openstackgerrit | Merged openstack/tripleo-common stable/rocky: Select message IDs properly when publishing to swift https://review.openstack.org/619249 | 16:20 |
| *** yprokule has quit IRC | 16:23 | |
| *** pcaruana has joined #tripleo | 16:26 | |
| * mwhahaha flips tables | 16:26 | |
| mwhahaha | ssbarnea|rover, weshay: so the reproducer for the queens upgrade job fails on setting up the queens repos | 16:26 |
| ssbarnea|rover | rlandy: ^^ | 16:26 |
| mwhahaha | echo 'Failed to parse dlrn hash' | 16:27 |
| *** thrash|biab is now known as thrash | 16:27 | |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-quickstart master: Add test_black_regex for fs052 standalone for failing tempest https://review.openstack.org/621221 | 16:28 |
| mwhahaha | cause the curl fails for https://mirror.regionone.rdo-cloud.rdoproject.org:8080/rdo/centos7-queens/d7d80c59146ce502ae82ad9ee93ab6409a3353f2_15fc9723/delorean.repo | 16:28 |
| marios | weshay: ssbarnea|rover https://review.openstack.org/621221 for https://bugs.launchpad.net/tripleo/+bug/1806075 just adding skip for now as requested | 16:28 |
| openstack | Launchpad bug 1806075 in tripleo "centos-7-standalone job failing tempest TestNetworkBasicOps" [Undecided,Triaged] - Assigned to Marios Andreou (marios-b) | 16:28 |
| mwhahaha | curl: (35) SSL received a record that exceeded the maximum permissible length. | 16:28 |
| mwhahaha | dat ssl | 16:28 |
| *** brault has joined #tripleo | 16:28 | |
| fultonj | marios: do you have a few minutes to give me a hand with https://review.openstack.org/#/c/617368 ? | 16:29 |
| rlandy | ssbarnea|rover: probably pretty old | 16:29 |
| marios | fultonj: can we please do it on monday is a bit late here and i still have some pending tasks before i can go | 16:30 |
| fultonj | marios: certainly | 16:30 |
| marios | thanks fultonj | 16:30 |
| fultonj | thank you | 16:30 |
| marios | no thank you | 16:30 |
| marios | :D | 16:31 |
| marios | k going | 16:31 |
| mwhahaha | rlandy: fyi https://bugs.launchpad.net/tripleo/+bug/1806077 | 16:32 |
| openstack | Launchpad bug 1806077 in tripleo "reproducer queens undercloud upgrade job fails trying to setup the queens repo" [High,Triaged] | 16:32 |
| *** holser_ has quit IRC | 16:32 | |
| rlandy | mwhahaha: thanks for logging that - will review | 16:33 |
| *** ccamacho has quit IRC | 16:35 | |
| *** ccamacho has joined #tripleo | 16:35 | |
| mwhahaha | rlandy: might be because the tripleo dlrn hash is bad or some reason. i'll keep poking at it because i need to reproduce this thing | 16:35 |
| *** xek has quit IRC | 16:36 | |
| marios_|ruck | fultonj: successful scen1 deploy from today if you're interested http://logs.openstack.org/08/619508/10/check/tripleo-ci-centos-7-scenario001-standalone/d3d5873/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz | 16:36 |
| rlandy | not necessarily | 16:37 |
| marios_|ruck | fultonj: (from earlier) | 16:37 |
| fultonj | marios_|ruck: cool | 16:37 |
| *** bogdando has quit IRC | 16:37 | |
| mwhahaha | rlandy: well for whatever reason it was bad for me. anyway i replaced it with current-tripleo for now and will continue | 16:37 |
| rlandy | I'm trying that reproducer | 16:38 |
| mwhahaha | rlandy: http://logs.openstack.org/75/620975/1/check/tripleo-ci-centos-7-undercloud-upgrades/6d3151c/logs/reproducer-quickstart.sh is the specific reproducer i was working with (not sure if i provided the right link) | 16:39 |
| rlandy | yeah - it was in the bug | 16:39 |
| rlandy | ssbarnea|rover: ^^ trying it out | 16:40 |
| mwhahaha | of course it worked fine for me (the upgrade process) | 16:41 |
| mwhahaha | so in CI we aren't actually updating anything | 16:42 |
| mwhahaha | http://logs.openstack.org/75/620975/1/check/tripleo-ci-centos-7-undercloud-upgrades/6d3151c/logs/undercloud/home/zuul/undercloud_upgrade.log.txt.gz | 16:42 |
| mwhahaha | that seems wrong | 16:42 |
| ssbarnea|rover | mwhahaha: you must use SSL | 16:42 |
| mwhahaha | ssbarnea|rover: i did, the issue was a bad hash. either way the job doesn't seem to be doing what it's supposed to | 16:42 |
| mwhahaha | ssbarnea|rover: the undercloud upgrade is supposed to update some packages (and they aren't) | 16:43 |
| mwhahaha | i wonder if the repo config is bad in the job | 16:43 |
| mwhahaha | http://logs.openstack.org/75/620975/1/check/tripleo-ci-centos-7-undercloud-upgrades/6d3151c/logs/undercloud/home/zuul/undercloud_upgrade.sh.txt.gz | 16:43 |
| mwhahaha | tripleoclient is supposed to be updated, but did not happen | 16:43 |
| ssbarnea|rover | mwhahaha: 100% this web server is incorectly configured. | 16:43 |
| ssbarnea|rover | evem my browser refused to load that url | 16:43 |
| mwhahaha | ssbarnea|rover: it's limited to rdo cloud addresses | 16:44 |
| ssbarnea|rover | mwhahaha: i suspect HSTS policy forbits you from using HTTP. | 16:48 |
| *** udesale has quit IRC | 16:49 | |
| mwhahaha | meh i got the process working with out that, still i think it's a job config problem because we didn't actually end up udpating any packages between pike/queens | 16:49 |
| mwhahaha | which is weird | 16:49 |
| *** RaisdbyRobots has joined #tripleo | 16:51 | |
| rlandy | doesn't seem like reproducer issue | 16:52 |
| ssbarnea|rover | mwhahaha: run: curl -s -D- https://rdoproject.org/ | grep Strict | 16:53 |
| ssbarnea|rover | you will understand why it fails. now the funny part is that my own curl is not affected by this, only the browser but i suspect a different version of curl may behave different | 16:54 |
| *** abishop has quit IRC | 16:54 | |
| ssbarnea|rover | the current setting of rdoproect.org enforces use of SSL (HTTPS) for all subdomains. | 16:55 |
| *** trown|brb is now known as regain | 16:55 | |
| *** regain is now known as trown | 16:55 | |
| *** abishop has joined #tripleo | 16:55 | |
| *** trown is now known as trown|lunch | 16:56 | |
| ssbarnea|rover | ... and that is cached for 1Y. | 16:56 |
| *** rnoriega- has quit IRC | 16:56 | |
| *** rnoriega has joined #tripleo | 16:58 | |
| *** jpena is now known as jpena|off | 16:59 | |
| *** RaisdbyRobots has quit IRC | 16:59 | |
| *** jpich has quit IRC | 17:00 | |
| *** iurygregory has quit IRC | 17:00 | |
| *** tongl has joined #tripleo | 17:03 | |
| *** pcaruana has quit IRC | 17:04 | |
| openstackgerrit | Mike Fedosin proposed openstack/ansible-role-container-registry master: Allow to skip docker reconfiguration https://review.openstack.org/621241 | 17:05 |
| weshay | ssbarnea|rover, marios_|ruck fyi.. when skipping a tempest test.. that info goes here https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/validate-tempest/vars/tempest_skip_master.yml | 17:06 |
| weshay | when you skip due to a bug | 17:06 |
| *** ooolpbot has joined #tripleo | 17:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1796639 | 17:10 |
| *** ooolpbot has quit IRC | 17:10 | |
| openstack | Launchpad bug 1796639 in tripleo "[stable/queens] The tripleo-ci-centos-7-undercloud-upgrades is failing on queens for No such file or directory: '/etc/heat/policy.json" [Critical,Triaged] - Assigned to Alex Schultz (alex-schultz) | 17:10 |
| marios_|ruck | weshay: ack | 17:13 |
| weshay | marios_|ruck, the why.. being.. we have a job that runs all the skip tests so we know when to take it back out of the skiplist and we always have a bug associated | 17:14 |
| marios | weshay: makes sense | 17:17 |
| ssbarnea|rover | weshay: this approach requires two merges from ci (us): add workaround, remove workaround. we could avoid this by publishing the list of ignored tests somewhere and having a periodic job for updating it based on the bug status on LP. When the bug is closed the skip is *no* longer performed. | 17:17 |
| weshay | ssbarnea|rover, we have that | 17:17 |
| ssbarnea|rover | cool | 17:17 |
| weshay | ssbarnea|rover, talk to arx if you want details | 17:18 |
| ssbarnea|rover | weshay: too busy right now, trying to fix his other problem and I think I narrowed it down: read https://bugs.launchpad.net/tripleo/+bug/1806077 | 17:20 |
| openstack | Launchpad bug 1806077 in tripleo "reproducer queens undercloud upgrade job fails trying to setup the queens repo" [High,Triaged] | 17:20 |
| weshay | ssbarnea|rover, ok.. queens is not huge priority atm | 17:20 |
| ssbarnea|rover | defaulting to http:// something does not play well with SSL. I doubt is queens related | 17:21 |
| *** hjensas has quit IRC | 17:21 | |
| ssbarnea|rover | not sure why I seen this only on queens because this kind of issue is related to infrastructure configuration. | 17:21 |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-quickstart-extras master: Add TestNetworkBasicOps.test_hotplug_nic to master skip list https://review.openstack.org/621248 | 17:21 |
| marios | weshay: ssbarnea|rover ^ | 17:21 |
| *** ykarel is now known as ykarel|away | 17:22 | |
| *** shardy has quit IRC | 17:22 | |
| weshay | thanks marios have a good weekend | 17:23 |
| marios | u 2 | 17:23 |
| *** moguimar has quit IRC | 17:25 | |
| weshay | ssbarnea|rover, anything I can help w/? | 17:25 |
| ssbarnea|rover | weshay: just want to tell me if my comments on https://bugs.launchpad.net/tripleo/+bug/1806077 makes sense. it should be easy to swich config for rdo proxy to ssl to avoid it. | 17:26 |
| openstack | Launchpad bug 1806077 in tripleo "reproducer queens undercloud upgrade job fails trying to setup the queens repo" [High,Triaged] | 17:26 |
| *** kopecmartin is now known as kopecmartin|off | 17:26 | |
| weshay | woot.. full pass on 3rd party :) https://review.openstack.org/#/c/621075/ | 17:30 |
| weshay | ssbarnea|rover, check that out :)) | 17:31 |
| weshay | you probably haven't seen that since you joined | 17:31 |
| rascasoft | hey folks, are there specific limitations for the things you can include in the standalone deployment? https://docs.openstack.org/tripleo-docs/latest/install/containers_deployment/standalone.html | 17:31 |
| ssbarnea|rover | weshay: my mistake sent you wrong link, SSL received a record that exceeded the maximum permissible | 17:31 |
| weshay | rascasoft, bah.. get out of here | 17:31 |
| weshay | who is this guy | 17:31 |
| rascasoft | I'm trying to include octavia via the usual -e option, but I'm miserably failing | 17:31 |
| openstackgerrit | Goutham Pacha Ravi proposed openstack/tripleo-heat-templates master: Switch Manila API to httpd and support TLS https://review.openstack.org/594566 | 17:32 |
| ssbarnea|rover | https://bugs.launchpad.net/tripleo/+bug/1806077 | 17:32 |
| openstack | Launchpad bug 1806077 in tripleo "reproducer queens undercloud upgrade job fails trying to setup the queens repo" [High,Triaged] | 17:32 |
| rascasoft | weshay, you'll never get rid of me | 17:32 |
| weshay | NOOOOOOOOOOOOOOOOo | 17:32 |
| mwhahaha | rascasoft: we've never tried octavia, so not sure what limitations would be | 17:32 |
| mwhahaha | rascasoft: best take that up with the networking folks | 17:32 |
| mwhahaha | rodrigods: though if the octavia install requires mistral/swift/zaqar, none of that exists in standalone | 17:33 |
| rascasoft | mwhahaha, well at the moment I'm failing with a specific error, is this worth a bug or is too specific? | 17:33 |
| mwhahaha | rodrigods: er sorry wrong person | 17:33 |
| mwhahaha | rascasoft: probably? it's likely you're hitting some assumptions in the octavia deployment that needs to be handled specially for standalone | 17:33 |
| rascasoft | mwhahaha, ok, I'd go with a bug so everyone can be happy | 17:34 |
| weshay | rascasoft, so.. here's an example if a different set of services and params https://review.openstack.org/#/c/616592/7/ci/environments/scenario001-standalone.yaml | 17:34 |
| mwhahaha | rascasoft: best ask cgoncalves or beagles about octavia+standalone. not sure if they've tried it | 17:34 |
| beagles | rascasoft, almost certain we havent tried it | 17:35 |
| rascasoft | weshay, oh nice, maybe it can be useful, thanks for still loving me | 17:35 |
| *** weshay is now known as he_hates_me | 17:35 | |
| beagles | rascasoft, a bug would be very welcome and we'll look at it asap | 17:35 |
| * he_hates_me loves the xfl | 17:35 | |
| rascasoft | beagles, what are your feelings, it might work if I include all the dependencies? Do we want bugs on this topic or it's too experimental? | 17:35 |
| rascasoft | beagles, oh ok | 17:36 |
| mwhahaha | he_hates_me: too obscure | 17:36 |
| he_hates_me | probably | 17:36 |
| *** he_hates_me is now known as weshay | 17:36 | |
| * mwhahaha gets the reference tho | 17:36 | |
| beagles | mwhahaha, do we have any standalone jobs that involve deployments using external_deploy_tasks yet? | 17:36 |
| mwhahaha | beagles: ceph | 17:37 |
| weshay | probably should kick/ban rascasoft | 17:37 |
| beagles | mwhahaha, okay so that mechanism is probably fine at least | 17:37 |
| mwhahaha | beagles: the external deploy tasks work fine, it's any assumptions on mistral/swift that don't exist | 17:37 |
| mwhahaha | beagles: so ceph has a param that can be used in the case of standalone for local file stuff | 17:38 |
| rascasoft | weshay, then I'll change my name, and then who knows, maybe I already did that and I am rascasoft and mwhahaha at the same time | 17:38 |
| mwhahaha | beagles: example is https://review.openstack.org/#/c/616592/7/ci/environments/scenario001-standalone.yaml@42 | 17:38 |
| rascasoft | weshay, Invasion of the Body Snatchers | 17:38 |
| mwhahaha | beagles: so it depends on what the octavia installer needs and if any of those external service dependencies can be replaced with something static | 17:38 |
| beagles | mwhahaha, ack | 17:41 |
| rlandy | mwhahaha: I could reproduce the error | 17:41 |
| rlandy | 2018-11-30 17:39:14 | + echo 'Failed to parse dlrn hash' | 17:41 |
| rlandy | 2018-11-30 17:39:14 | Failed to parse dlrn hash | 17:41 |
| *** hberaud is now known as hberaud|gone | 17:45 | |
| *** brault has quit IRC | 17:46 | |
| openstackgerrit | Merged openstack/ansible-role-redhat-subscription master: Set up repos when using portal with activation key https://review.openstack.org/620937 | 17:48 |
| mwhahaha | if ya'll could stop approving things that would be lovely | 17:49 |
| *** openstackgerrit has quit IRC | 17:51 | |
| *** amoralej is now known as amoralej|off | 17:53 | |
| *** hberaud|gone is now known as hberaud | 17:54 | |
| apetrich | mwhahaha, btw that bug was a real error but for a patch https://review.openstack.org/#/c/619039/ | 17:57 |
| mwhahaha | apetrich: yea i saw | 17:57 |
| *** hberaud is now known as hberaud|gone | 17:58 | |
| *** derekh has quit IRC | 18:00 | |
| mwhahaha | well i can't reproduce the queens upgrade errror | 18:02 |
| mwhahaha | seems to be ci specific | 18:02 |
| mwhahaha | (great) | 18:02 |
| *** gfidente has quit IRC | 18:04 | |
| *** cylopez has quit IRC | 18:04 | |
| mwhahaha | oh damn it | 18:06 |
| mwhahaha | it's install-built-repo | 18:06 |
| tongl | @mwhahaha: https://review.openstack.org/#/c/618796/, https://review.openstack.org/#/c/618798/, https://review.openstack.org/#/c/605056/ have been reviewed by Emilien and passed zull check. Do you think I need to ask Emilien for workflow +1 or you can do it? | 18:06 |
| mwhahaha | tongl: i can but the CI gate is backed up so we should not be merging anything non-critical at the moment | 18:07 |
| tongl | We are waiting for these patches to be merged so that we can package them downstream into RHOSP. | 18:08 |
| tongl | These are the missing pieces for RHOSP integration with NSX neutron plugin. | 18:09 |
| * mwhahaha shrugs | 18:09 | |
| mwhahaha | even if i do approve them, they likely won't be merged | 18:09 |
| mwhahaha | so if you're waiting, i would be proactively pulling them down because they are unlikely to land upstream in the near future | 18:09 |
| *** ooolpbot has joined #tripleo | 18:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1796639 | 18:10 |
| openstack | Launchpad bug 1796639 in tripleo "[stable/queens] The tripleo-ci-centos-7-undercloud-upgrades is failing on queens for No such file or directory: '/etc/heat/policy.json" [Critical,Triaged] - Assigned to Alex Schultz (alex-schultz) | 18:10 |
| *** ooolpbot has quit IRC | 18:10 | |
| tongl | okay, do you have any idea when they can be landed upstream? | 18:10 |
| mwhahaha | when we can actually merged patches with out everything timing out | 18:11 |
| *** jfrancoa has quit IRC | 18:16 | |
| *** ykarel|away has quit IRC | 18:17 | |
| *** wolverineav has joined #tripleo | 18:17 | |
| *** openstackgerrit has joined #tripleo | 18:26 | |
| openstackgerrit | Marius Cornea proposed openstack/tripleo-heat-templates master: Use InternalApi network for openshift_master_cluster_hostname https://review.openstack.org/621256 | 18:26 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-quickstart-extras master: Add ability to disable yum update with gating repo https://review.openstack.org/621259 | 18:47 |
| openstackgerrit | Marius Cornea proposed openstack/puppet-tripleo master: Set balance to source for openshift_master endpoints https://review.openstack.org/621260 | 18:49 |
| openstackgerrit | Marius Cornea proposed openstack/tripleo-heat-templates master: Set balance to source for openshift_master endpoints https://review.openstack.org/621261 | 18:51 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-quickstart-extras master: Add ability to disable yum update with gating repo https://review.openstack.org/621259 | 18:53 |
| mwhahaha | weshay, marios, ssbarnea|rover -^ should fix the upgrade job(s) | 18:53 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-common stable/queens: Switch scenarios to non-voting https://review.openstack.org/620975 | 18:55 |
| ssbarnea|rover | mwhahaha: cool. i think that your SSL problem with rdoproject.com mirror will be also sorted till monday. | 18:56 |
| openstackgerrit | Marius Cornea proposed openstack/tripleo-heat-templates master: Set balance to source for openshift_router endpoints https://review.openstack.org/621261 | 18:57 |
| ssbarnea|rover | mwhahaha: tbh I am not sure if disabling yum update is the best idea because we will no longer use latest code, meaning missing fixes or bugs. | 18:58 |
| mwhahaha | ssbarnea|rover: not true | 18:58 |
| mwhahaha | ssbarnea|rover: we do a yum update as part of the upgrade itself | 18:58 |
| mwhahaha | ssbarnea|rover: so this is prematurely yum updating packages | 18:58 |
| mwhahaha | this only affects upgrades | 18:58 |
| mwhahaha | so prior to the upgrade we need to swap out the repos, but we do updates as part of the upgrade execution | 18:59 |
| mwhahaha | for the undercloud that includes a yum update of python-tripleoclient followed by an actual yum update in the upgrade process | 18:59 |
| mwhahaha | for the overcloud that is also true | 18:59 |
| ssbarnea|rover | mwhahaha: ahh, in this case it should be ok. | 18:59 |
| mwhahaha | yes | 18:59 |
| mwhahaha | thats why it's still true by default | 18:59 |
| mwhahaha | it only affects upgrades when we swap out the newer repos | 18:59 |
| mwhahaha | it should just add the gating repo in and not update | 18:59 |
| mwhahaha | because that gets handled by the subsequent upgrade processes | 19:00 |
| ssbarnea|rover | makes sense. we skip it because we do it later anyway with upgrades, saving some time which is key for upgrades. | 19:00 |
| mwhahaha | so what was happening is in the undercloud update was that it wa getting the newer heat packages that removed the policy.json w/o a service restart | 19:00 |
| mwhahaha | causing the error | 19:00 |
| mwhahaha | very obscure and annoying :D | 19:01 |
| openstackgerrit | Alex Schultz proposed openstack/puppet-tripleo stable/queens: Switch scenarios to non-voting https://review.openstack.org/620976 | 19:01 |
| *** wolverineav has quit IRC | 19:02 | |
| *** wolverineav has joined #tripleo | 19:06 | |
| *** ooolpbot has joined #tripleo | 19:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1796639 | 19:10 |
| openstack | Launchpad bug 1796639 in tripleo "[stable/queens] The tripleo-ci-centos-7-undercloud-upgrades is failing on queens for No such file or directory: '/etc/heat/policy.json" [Critical,In progress] - Assigned to Alex Schultz (alex-schultz) | 19:10 |
| *** ooolpbot has quit IRC | 19:10 | |
| *** morazi has quit IRC | 19:11 | |
| dprince | mwhahaha: hi, is the gate okay again? | 19:20 |
| mwhahaha | no | 19:20 |
| dprince | :/ | 19:20 |
| mwhahaha | pretty much | 19:20 |
| mwhahaha | we need https://review.openstack.org/#/c/621248/ at least | 19:21 |
| dprince | and a weekend | 19:21 |
| mwhahaha | yes | 19:21 |
| openstackgerrit | Natal Ngétal proposed openstack/tripleo-docs master: [Documentation] Add a note for tripleo container. https://review.openstack.org/621274 | 19:23 |
| *** wolverineav has quit IRC | 19:37 | |
| *** wolverineav has joined #tripleo | 19:41 | |
| *** wolverineav has quit IRC | 19:42 | |
| *** wolverineav has joined #tripleo | 19:42 | |
| *** wolverineav has quit IRC | 19:42 | |
| *** wolverineav has joined #tripleo | 19:43 | |
| *** wolverineav has quit IRC | 19:43 | |
| *** wolverineav has joined #tripleo | 19:43 | |
| *** wolverineav has quit IRC | 19:44 | |
| *** wolverineav has joined #tripleo | 19:45 | |
| rascasoft | mwhahaha, weshay, beagles, https://bugs.launchpad.net/tripleo/+bug/1806113 | 19:46 |
| openstack | Launchpad bug 1806113 in tripleo "[tripleo] Standalone setup is unable to deploy Octavia" [Undecided,Triaged] | 19:46 |
| rascasoft | environment is accessible if you need it ;) | 19:46 |
| mwhahaha | oh yea it doesn't work because it's trying to get the default nova keypair | 19:47 |
| mwhahaha | which doesn't exist | 19:47 |
| mwhahaha | so there needs to be a way to specify that outside of that | 19:47 |
| mwhahaha | there is no compute service | 19:47 |
| *** wolverineav has quit IRC | 19:52 | |
| *** wolverineav has joined #tripleo | 19:53 | |
| weshay | rascasoft, you can't just open bugs and quit.. | 19:54 |
| *** hjensas has joined #tripleo | 19:58 | |
| *** wolverineav has quit IRC | 19:59 | |
| *** rh-jelabarre has quit IRC | 20:03 | |
| *** rh-jelabarre has joined #tripleo | 20:05 | |
| *** morazi has joined #tripleo | 20:05 | |
| openstackgerrit | Merged openstack/tripleo-quickstart master: Revert "master/hack: Switch to CBS instead of buildlogs" https://review.openstack.org/619625 | 20:09 |
| weshay | hooooray | 20:09 |
| *** ooolpbot has joined #tripleo | 20:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1796639 | 20:10 |
| openstack | Launchpad bug 1796639 in tripleo "[stable/queens] The tripleo-ci-centos-7-undercloud-upgrades is failing on queens for No such file or directory: '/etc/heat/policy.json" [Critical,In progress] - Assigned to Alex Schultz (alex-schultz) | 20:10 |
| *** ooolpbot has quit IRC | 20:10 | |
| *** jistr has quit IRC | 20:11 | |
| *** jistr has joined #tripleo | 20:13 | |
| *** radeks__ has quit IRC | 20:14 | |
| *** xek has joined #tripleo | 20:19 | |
| *** vpickard is now known as vpickard_ | 20:28 | |
| openstackgerrit | Merged openstack/puppet-pacemaker master: Match node properties more strictly https://review.openstack.org/620892 | 20:32 |
| *** radeks has joined #tripleo | 20:32 | |
| *** wolverineav has joined #tripleo | 20:33 | |
| *** slaweq has quit IRC | 20:34 | |
| *** matbu has quit IRC | 20:38 | |
| *** matbu has joined #tripleo | 20:46 | |
| *** openstackgerrit has quit IRC | 20:50 | |
| *** itlinux has joined #tripleo | 20:51 | |
| *** itlinux_ has joined #tripleo | 20:58 | |
| *** itlinux has quit IRC | 20:59 | |
| *** dprince has quit IRC | 21:04 | |
| *** raildo has quit IRC | 21:05 | |
| *** rlandy has quit IRC | 21:08 | |
| *** wolverineav has quit IRC | 21:08 | |
| *** ooolpbot has joined #tripleo | 21:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1796639 | 21:10 |
| openstack | Launchpad bug 1796639 in tripleo "[stable/queens] The tripleo-ci-centos-7-undercloud-upgrades is failing on queens for No such file or directory: '/etc/heat/policy.json" [Critical,In progress] - Assigned to Alex Schultz (alex-schultz) | 21:10 |
| *** ooolpbot has quit IRC | 21:10 | |
| weshay | mwhahaha, I'll have to find it, but we're supposed to deal w/ using the gating repo only where appropriate https://review.openstack.org/#/c/621259/ | 21:10 |
| weshay | re: that review | 21:10 |
| mwhahaha | weshay: what do you mean | 21:10 |
| weshay | so if the patch is on rocky for example.. the gating repo would be skipped in the upgrade phase/playbook where the release is set to master | 21:11 |
| weshay | if the patch is on master | 21:11 |
| weshay | for instance than the gating repo would be enabled on master | 21:11 |
| mwhahaha | ok i can join the ib_gating_repo_update with ib_gating_repo_enabled | 21:11 |
| mwhahaha | either way we need to *not* always run the yum update on the upgrade jobs | 21:12 |
| weshay | mwhahaha, I'm not saying to do anything as much as I'm just thinking | 21:12 |
| mwhahaha | that's fine, what you say makes sense | 21:12 |
| weshay | it was complicated and I need to poke at this for a while | 21:12 |
| weshay | mwhahaha, /me wondering though | 21:12 |
| weshay | if your intent is disable any patches from landing on the env. during the upgrade/update portion | 21:12 |
| mwhahaha | no | 21:12 |
| mwhahaha | it is not | 21:12 |
| mwhahaha | it is so that the yum update -y does not happen after install before upgrade | 21:13 |
| mwhahaha | because the upgrade itself will pull in the appropriate updated packages (From the gating repo) at upgrade time | 21:13 |
| *** openstackgerrit has joined #tripleo | 21:13 | |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-quickstart-extras master: Add ability to disable yum update with gating repo https://review.openstack.org/621259 | 21:13 |
| weshay | OH | 21:13 |
| mwhahaha | that's what's breaking the queens upgrade | 21:13 |
| mwhahaha | because that role is updating all the packages | 21:13 |
| mwhahaha | so the upgrade process breaks for the udnercloud | 21:14 |
| *** Vorrtex has quit IRC | 21:14 | |
| weshay | k k | 21:14 |
| weshay | that makes sense | 21:14 |
| mwhahaha | the queens heat drops policy.json but the yum update doesn't restart the service | 21:14 |
| mwhahaha | so sadness ensues | 21:14 |
| mwhahaha | it also explains why if you follow the normal procedure it always work | 21:14 |
| weshay | k | 21:14 |
| mwhahaha | it only fails in CI because of this | 21:14 |
| weshay | thanks for explaining that | 21:14 |
| weshay | right | 21:14 |
| openstackgerrit | Dan Sneddon proposed openstack/os-net-config master: Fix incorrect application of 'ip' commands for bridges https://review.openstack.org/621291 | 21:20 |
| *** radeks has quit IRC | 21:21 | |
| *** toure is now known as toure|gone | 21:26 | |
| *** salmankhan has joined #tripleo | 21:31 | |
| *** vinaykns has quit IRC | 21:34 | |
| *** chem has quit IRC | 21:34 | |
| *** wolverineav has joined #tripleo | 21:35 | |
| *** salmankhan has quit IRC | 21:36 | |
| *** abishop has quit IRC | 21:49 | |
| *** dpeacock has quit IRC | 21:52 | |
| *** dpeacock has joined #tripleo | 21:55 | |
| *** wolverineav has quit IRC | 21:59 | |
| *** wolverineav has joined #tripleo | 22:04 | |
| openstackgerrit | Harald Jensås proposed openstack/os-net-config master: Fix - restart VLAN interface on underlying device change https://review.openstack.org/621298 | 22:06 |
| *** ooolpbot has joined #tripleo | 22:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1796639 | 22:10 |
| openstack | Launchpad bug 1796639 in tripleo "[stable/queens] The tripleo-ci-centos-7-undercloud-upgrades is failing on queens for No such file or directory: '/etc/heat/policy.json" [Critical,In progress] - Assigned to Alex Schultz (alex-schultz) | 22:10 |
| *** ooolpbot has quit IRC | 22:10 | |
| *** salmankhan has joined #tripleo | 22:11 | |
| *** wolverineav has quit IRC | 22:12 | |
| *** itlinux_ has quit IRC | 22:13 | |
| *** wolverineav has joined #tripleo | 22:13 | |
| *** wolverineav has quit IRC | 22:18 | |
| *** wolverineav has joined #tripleo | 22:19 | |
| *** mcornea has quit IRC | 22:20 | |
| mwhahaha | bandini: jfyi https://review.rdoproject.org/r/#/c/17630/ | 22:26 |
| *** wolverineav has quit IRC | 22:28 | |
| *** wolverineav has joined #tripleo | 22:29 | |
| openstackgerrit | Harald Jensås proposed openstack/os-net-config master: Fix - restart VLAN interface on underlying device change https://review.openstack.org/621298 | 22:34 |
| openstackgerrit | Harald Jensås proposed openstack/os-net-config master: Fix - restart VLAN interface on underlying device change https://review.openstack.org/621298 | 22:36 |
| *** slaweq has joined #tripleo | 22:36 | |
| *** xek has quit IRC | 22:37 | |
| *** slaweq has quit IRC | 22:43 | |
| *** apetrich has quit IRC | 22:47 | |
| *** slaweq has joined #tripleo | 22:50 | |
| *** slaweq has quit IRC | 22:52 | |
| *** wolverineav has quit IRC | 22:54 | |
| *** slaweq has joined #tripleo | 22:54 | |
| *** wolverineav has joined #tripleo | 22:55 | |
| *** cfontain has joined #tripleo | 22:58 | |
| *** wolverineav has quit IRC | 23:00 | |
| *** wolverineav has joined #tripleo | 23:00 | |
| *** apetrich has joined #tripleo | 23:02 | |
| *** itlinux has joined #tripleo | 23:07 | |
| *** ooolpbot has joined #tripleo | 23:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1796639 | 23:10 |
| openstack | Launchpad bug 1796639 in tripleo "[stable/queens] The tripleo-ci-centos-7-undercloud-upgrades is failing on queens for No such file or directory: '/etc/heat/policy.json" [Critical,In progress] - Assigned to Alex Schultz (alex-schultz) | 23:10 |
| *** ooolpbot has quit IRC | 23:10 | |
| *** slaweq has quit IRC | 23:22 | |
| *** wolverineav has quit IRC | 23:42 | |
| *** tosky has quit IRC | 23:42 | |
| *** wolverineav has joined #tripleo | 23:45 | |
| *** wolverineav has quit IRC | 23:48 | |
| *** wolverineav has joined #tripleo | 23:49 | |
| openstackgerrit | Jill Rouleau proposed openstack/paunch master: Add systemd healthchecks to podman https://review.openstack.org/620372 | 23:50 |
| *** salmankhan has quit IRC | 23:51 | |
| *** slaweq has joined #tripleo | 23:54 | |
| *** pbandark has quit IRC | 23:55 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!