| *** aakarsh has joined #oooq | 00:11 | |
| *** Vorrtex has joined #oooq | 00:32 | |
| *** rnoriega has quit IRC | 00:40 | |
| *** radez has quit IRC | 00:40 | |
| *** amoralej|off has quit IRC | 00:40 | |
| *** rnoriega has joined #oooq | 00:40 | |
| *** jpena|off has quit IRC | 00:40 | |
| *** jpena|off has joined #oooq | 00:48 | |
| *** Vorrtex has quit IRC | 00:54 | |
| *** Vorrtex has joined #oooq | 00:58 | |
| *** Goneri has quit IRC | 01:13 | |
| *** dalvarez has quit IRC | 01:54 | |
| *** radez has joined #oooq | 01:57 | |
| *** Vorrtex has quit IRC | 02:15 | |
| *** gkadam has quit IRC | 02:38 | |
| *** gkadam has joined #oooq | 02:39 | |
| *** ykarel|away has joined #oooq | 03:25 | |
| *** gkadam has quit IRC | 04:01 | |
| *** gkadam has joined #oooq | 04:01 | |
| *** ykarel|away has quit IRC | 04:08 | |
| *** ykarel|away has joined #oooq | 04:11 | |
| *** udesale has joined #oooq | 04:16 | |
| *** tcw has quit IRC | 04:29 | |
| *** tcw has joined #oooq | 04:29 | |
| *** ykarel|away has quit IRC | 04:32 | |
| *** ykarel|away has joined #oooq | 04:50 | |
| *** ykarel|away is now known as ykarel | 04:50 | |
| *** raukadah is now known as chandankumar | 05:01 | |
| *** gkadam has quit IRC | 05:05 | |
| *** saneax has joined #oooq | 05:07 | |
| *** ratailor has joined #oooq | 05:12 | |
| *** saneax has quit IRC | 05:15 | |
| *** jfrancoa has joined #oooq | 05:28 | |
| *** kopecmartin|off is now known as kopecmartin | 05:32 | |
| *** jfrancoa has quit IRC | 05:34 | |
| *** marios has joined #oooq | 05:41 | |
| *** jfrancoa has joined #oooq | 05:48 | |
| *** ksambor has joined #oooq | 05:58 | |
| *** skramaja has joined #oooq | 06:35 | |
| *** tesseract has joined #oooq | 07:14 | |
| *** dmellado has quit IRC | 07:17 | |
| *** bogdando has joined #oooq | 07:17 | |
| *** ksambor has quit IRC | 07:18 | |
| *** ksambor_ has joined #oooq | 07:18 | |
| *** saneax has joined #oooq | 07:19 | |
| *** dmellado has joined #oooq | 07:29 | |
| *** amoralej has joined #oooq | 07:31 | |
| *** dtantsur|afk is now known as dtantsur | 07:34 | |
| *** gkadam has joined #oooq | 07:42 | |
| *** gkadam has quit IRC | 07:50 | |
| *** ykarel is now known as ykarel|lunch | 07:51 | |
| *** jpena|off is now known as jpena | 07:56 | |
| *** ccamacho has joined #oooq | 07:58 | |
| chandankumar | marios: Hello | 08:02 |
|---|---|---|
| chandankumar | marios: I need some help here http://logs.openstack.org/68/664568/2/check/openstack-tox-py27/d23b190/job-output.txt.gz#_2019-06-11_18_27_45_784781 | 08:02 |
| chandankumar | marios: https://review.opendev.org/#/c/664568/ | 08:03 |
| chandankumar | marios: I am not sure why it is coming | 08:03 |
| marios | chandankumar: s/TripleoAction/TripleOAction | 08:09 |
| *** chem has joined #oooq | 08:09 | |
| chandankumar | marios: oh no, thanks :-) | 08:10 |
| marios | chandankumar: yeah ... sometimes just needs fresh eyes ;) | 08:10 |
| *** holser_ has joined #oooq | 08:33 | |
| *** derekh has joined #oooq | 08:37 | |
| *** holser_ has quit IRC | 08:41 | |
| *** holser_ has joined #oooq | 08:42 | |
| zbr|rover | chandankumar: are you aware of other ans27 issues? | 08:48 |
| zbr|rover | that fix at the gate for 7.5h and still counting... | 08:49 |
| *** jaosorior has quit IRC | 08:51 | |
| arxcruz | sshnaidm https://review.rdoproject.org/r/#/c/21099/ just update with comments :) | 08:58 |
| *** blueDuckRandom has joined #oooq | 08:59 | |
| chandankumar | zbr|rover: nope | 09:01 |
| *** blueDuckRandomSt has quit IRC | 09:01 | |
| chandankumar | zbr|rover: will I take over this patch https://review.opendev.org/#/c/661994 python3 os_temepst work? | 09:08 |
| zbr|rover | chandankumar: please do, i am bit overloaded now. | 09:10 |
| chandankumar | zbr|rover: thanks, will take care of that | 09:10 |
| *** ykarel|lunch is now known as ykarel | 09:37 | |
| *** gkadam has joined #oooq | 09:53 | |
| *** gkadam has quit IRC | 09:53 | |
| zbr|rover | sshnaidm: marios chandankumar : can you help me rereview molecule https://review.opendev.org/#/c/651772/ ? made few changes yesterday to make it work with the new openstack-tox-molecule job template. | 09:54 |
| marios | zbr|rover: voted on 971 earlier will add that to next reviews | 09:55 |
| zbr|rover | marios: thanks. i am getting closer in getting these merged. | 09:58 |
| sshnaidm | zbr|rover, will do | 10:01 |
| zbr|rover | marios: luke also made a very useful comment in yesterday meeting, about adding [molecule] tests first to places that are likely to break. the criteria for adding a test should be likeness to break or if it covers code that is about to be changed. adding tests for the sake of them would be a waste of resources. | 10:08 |
| *** blueDuckRandom has quit IRC | 10:32 | |
| chandankumar | zbr|rover: is overcloud deploy broken on fs001 https://logs.rdoproject.org/68/664568/5/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/04e4ee8/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2019-06-12_09_42_55 ? | 10:49 |
| chandankumar | sshnaidm: do we collect the files from ~/.config folder in openstack deployment? | 10:51 |
| sshnaidm | chandankumar, nope https://github.com/openstack/tripleo-ci/blob/master/toci-quickstart/config/collect-logs.yml#L9 | 10:53 |
| *** jaosorior has joined #oooq | 10:53 | |
| chandankumar | sshnaidm: ack, added here https://review.opendev.org/#/c/664857/ | 10:59 |
| sshnaidm | chandankumar, commented | 11:01 |
| *** ykarel is now known as ykarel|afk | 11:03 | |
| *** ykarel|afk has quit IRC | 11:08 | |
| *** tosky has joined #oooq | 11:14 | |
| *** jaosorior has quit IRC | 11:15 | |
| *** jaosorior has joined #oooq | 11:28 | |
| *** blueDuckRandom has joined #oooq | 11:30 | |
| *** amoralej is now known as amoralej|lunch | 11:34 | |
| *** jpena is now known as jpena|lunch | 11:35 | |
| *** ratailor has quit IRC | 11:41 | |
| sshnaidm | chandankumar, one more | 11:44 |
| *** ykarel|afk has joined #oooq | 11:46 | |
| *** ykarel|afk has quit IRC | 11:50 | |
| *** ykarel has joined #oooq | 11:51 | |
| *** tosky has quit IRC | 11:51 | |
| *** tosky has joined #oooq | 11:51 | |
| marios | chandankumar: rfolco sshnaidm still ok to talk in few mins? (bluj on invite) thanks | 11:53 |
| chandankumar | marios: yes | 11:53 |
| sshnaidm | marios, ya | 11:54 |
| *** rlandy has joined #oooq | 11:55 | |
| rfolco | marios, o/ | 11:57 |
| rlandy | arxcruz: chandankumar: related to card rocky escalation card ... can you review https://review.opendev.org/#/c/663596/? thanks | 12:01 |
| arxcruz | rlandy so the test will not be executed at least for now, because we are not setting this on tempestconf and there isn't default values | 12:03 |
| arxcruz | rlandy correct? | 12:03 |
| rlandy | arxcruz: yes, to fix the card, the initial review was a skip list | 12:04 |
| arxcruz | ok | 12:04 |
| rlandy | this review was attached to the card so I'm just checking | 12:04 |
| *** rlandy is now known as rlandy|ruck | 12:05 | |
| *** sanjayu_ has joined #oooq | 12:05 | |
| weshay | sshnaidm I'm hearing vexxhost was approved yesterday | 12:06 |
| weshay | folks.. we're probably moving off of rdo-cloud for 3rd party | 12:07 |
| sshnaidm | weshay, no shit?! | 12:07 |
| zbr|rover | :D | 12:07 |
| weshay | hopefully we can flex back to rdo-cloud if needed | 12:07 |
| sshnaidm | weshay, tbh rdo cloud behaved ok recently.. like afraid that we leave him | 12:08 |
| *** saneax has quit IRC | 12:08 | |
| weshay | sshnaidm ya.. flexing to both clouds for some time should be possible, not 100% sure yet... this is all news to me | 12:08 |
| rlandy|ruck | weshay: do we have an admin contact there? | 12:10 |
| rlandy|ruck | although everyone hates rdocloud, the admins are awesome | 12:10 |
| *** udesale has quit IRC | 12:10 | |
| *** udesale has joined #oooq | 12:11 | |
| weshay | rlandy|ruck we have a very tight relationship w/ the CEO mnaser, you'll see him on #tripleo and #openstack-ansible | 12:11 |
| weshay | rlandy|ruck notes.. vexxhost is not net new.. we run against vexhost in upstream jobs, and the test jobs that sshnaidm put up | 12:12 |
| rlandy|ruck | good then I hope the CEO cleans up failed stacks | 12:12 |
| weshay | https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-rocky-vexxhost | 12:13 |
| weshay | master is a concern, could be devs really breaking the build https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master-vexxhost | 12:13 |
| *** amoralej|lunch is now known as amoralej | 12:21 | |
| ykarel | weshay, master vexxhost failures doesn't seem dev related, considering those jobs running with no real dev changes + same run rdo cloud ovb passing(checked some of last runs) | 12:25 |
| ykarel | i saw WIP in commit message, so asked | 12:25 |
| weshay | ykarel some were, see the undercloud failures some were not | 12:25 |
| ykarel | sorry wrong channel | 12:25 |
| ykarel | weshay, yes i just saw few failures: 1 failed with ntp sync, and other two timed out | 12:26 |
| weshay | ykarel noting ovb breaks because devs merge patches even when 3rd party fails | 12:26 |
| ykarel | weshay, yes agree with that, my point was mainly related to some of vexxhost fails and there equivalent rdo-cloud runs pass | 12:27 |
| ykarel | example PS14 result in https://review.opendev.org/#/c/614633/ | 12:28 |
| weshay | ya.. we need investigate it for sure.. the relative stability of rocky is reassuring but yes.. I'm also not 100% what the mirror config is for vexx rdo | 12:28 |
| weshay | sshnaidm rlandy|ruck was ovb in vexx using vexx mirrors or rdo-cloud mirrors | 12:29 |
| weshay | ? | 12:29 |
| sshnaidm | weshay, I don't think so, need to check | 12:29 |
| rlandy|ruck | weshay: vexxhost mirrors hence the bug you logged last week re: broken mirror | 12:29 |
| ykarel | yes it's worth investigating vexxhost failures before completely switching to it, or better to run parallel in both cloud for some time to isolate infra related issues | 12:32 |
| *** jpena|lunch is now known as jpena | 12:33 | |
| ykarel | rlandy|ruck, in case u have not checked yet, in master seems there is new blocker(new tempest tests failing) apart from dns tempest tests | 12:33 |
| ykarel | rlandy|ruck, fs020 runs in last periodic run:- https://trunk-primary.rdoproject.org/api-centos-master-uc/api/civotes_detail.html?commit_hash=77febec7848d2d031954619b6ce442c9723d0043&distro_hash=38e20448476702921862f74696c9866f62117f9c | 12:34 |
| ykarel | one is with skip list and one without, both have a new test failing | 12:34 |
| weshay | ykarel chandankumar who updated the skip list for master tempest dns issues? | 12:36 |
| ykarel | weshay, ksambor_ | 12:36 |
| rlandy|ruck | ykarel: thanks - I had not checked the new failure yet - will log. We are still waiting for the initial skip list review to clear queues | 12:36 |
| rlandy|ruck | weshay: https://review.opendev.org/#/c/664233/ | 12:36 |
| ykarel | rlandy|ruck, ack | 12:37 |
| rlandy|ruck | neutron_tempest_plugin.scenario.admin.test_floatingip.FloatingIpTestCasesAdmin10100Detail | 12:38 |
| rlandy|ruck | test_two_vms_fips[id-6bba729b-3fb6-494b-9e1e-82bbd89a1045] | 12:38 |
| rlandy|ruck | ykarel: ^^ you mean this test? | 12:38 |
| ykarel | rlandy|ruck, yes ^^ | 12:38 |
| rlandy|ruck | fine logging - thanks for the heads up | 12:38 |
| ykarel | rlandy|ruck, and is possibly caused with https://review.opendev.org/#/c/660741/ | 12:38 |
| rlandy|ruck | will add | 12:39 |
| *** jaosorior has quit IRC | 12:41 | |
| chandankumar | marios: https://code.engineering.redhat.com/gerrit/#/c/173155/ I think we need to update the commit message for this one | 12:43 |
| chandankumar | if it works | 12:43 |
| marios | chandankumar: what do you mean what's missing there (or the WIP DNM bit) | 12:43 |
| marios | chandankumar: cos that is needed for now :D | 12:43 |
| marios | i mean just posted it and there is ceph and ops-tools missing still | 12:44 |
| chandankumar | marios: ok | 12:44 |
| rlandy|ruck | ykarel: logged ... https://bugs.launchpad.net/tripleo/+bug/1832537 | 12:47 |
| openstack | Launchpad bug 1832537 in tripleo "neutron_tempest_plugin.scenario.admin.test_floatingip.FloatingIpTestCasesAdmin is failing on master promotions" [High,Triaged] | 12:47 |
| weshay | rlandy|ruck https://review.rdoproject.org/etherpad/p/tripleoci-perf-sync-06-2019 | 12:48 |
| ykarel | rlandy|ruck, ack btw why not Critical? iirc only Critical ones get escalated | 12:49 |
| rlandy|ruck | updated | 12:49 |
| ykarel | ack | 12:50 |
| *** ykarel is now known as ykarel|afk | 12:52 | |
| *** zbr|rover is now known as zbr|flow | 12:53 | |
| weshay | rlandy|ruck can you paste in job links | 12:53 |
| weshay | rlandy|ruck and reviews | 12:53 |
| rlandy|ruck | ack | 12:53 |
| rlandy|ruck | https://logs.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-stein/f4dc304/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz | 12:54 |
| rlandy|ruck | think we may have a real problem with stein as well | 12:54 |
| chandankumar | rlandy|ruck: dns skip tests merged right now | 12:56 |
| chandankumar | we will get a master promotion soon | 12:56 |
| rlandy|ruck | chandankumar: great - thanks - we have a new problem though https://bugs.launchpad.net/tripleo/+bug/1832537 | 12:56 |
| openstack | Launchpad bug 1832537 in tripleo "neutron_tempest_plugin.scenario.admin.test_floatingip.FloatingIpTestCasesAdmin is failing on master promotions" [Critical,Triaged] | 12:56 |
| weshay | rlandy|ruck on mute | 13:02 |
| *** rlandy|ruck is now known as rlandy|ruck|mtg | 13:04 | |
| arxcruz | chandankumar featureset021 job still using validate-tempest correct ? | 13:06 |
| chandankumar | arxcruz: yes | 13:06 |
| arxcruz | ok | 13:06 |
| chandankumar | all multinode jobs are using validate-tempest | 13:07 |
| *** jtomasek_ has joined #oooq | 13:07 | |
| *** jtomasek has quit IRC | 13:08 | |
| *** skramaja has quit IRC | 13:13 | |
| weshay | rlandy|ruck|mtg I have to drop, I'll sync w/ you shortly | 13:32 |
| *** jaosorior has joined #oooq | 13:32 | |
| weshay | tcw hey | 13:33 |
| tcw | hey | 13:33 |
| *** vinaykns has joined #oooq | 13:38 | |
| *** rlandy|ruck|mtg is now known as rlandy|ruck | 13:39 | |
| *** Goneri has joined #oooq | 13:45 | |
| weshay | panda you here today? | 13:50 |
| *** jaosorior has quit IRC | 13:50 | |
| panda | weshay: yes | 13:51 |
| weshay | panda when you are ready for 1-1 let's get started | 13:53 |
| weshay | I'm a few min early | 13:53 |
| panda | weshay: I'm already there. | 13:54 |
| jfrancoa | rlandy|ruck: o/, are you aware of any launchpad with the tripleo-ci-centos7-scenario10-standalone? it's constantly timeouting and blocking us from merging a patch... https://review.opendev.org/#/c/664225/ but I doubt the time out is related to the patch itself as it didn't timeout during Zuul check | 13:58 |
| rlandy|ruck | jfrancoa: looking | 13:59 |
| jfrancoa | rlandy|ruck: thanks | 13:59 |
| rlandy|ruck | http://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-scenario010-standalone | 14:01 |
| rlandy|ruck | yeah - so I see the timeout with stable/stein | 14:02 |
| rlandy|ruck | jfrancoa: ^^ can you access the link above | 14:02 |
| marios | wdyt folks http://logs.openstack.org/02/663702/3/check/openstack-tox-docs/8f84144/html/ci/ruck_rover_primer.html review please at https://review.opendev.org/#/c/663702/ when you have some time thank you! | 14:03 |
| rlandy|ruck | the timeouts are all on the patch but does not necessarily mean related | 14:03 |
| jfrancoa | rlandy|ruck: mmm...so it does look to be related | 14:04 |
| rlandy|ruck | jfrancoa: not totally ... | 14:04 |
| rlandy|ruck | tripleo-ci-centos-7-scenario010-standaloneopenstack/tripleo-heat-templatesstable/steingate661946,150952019-06-12T03:06:47SUCCESS | 14:04 |
| *** jaosorior has joined #oooq | 14:04 | |
| rlandy|ruck | ^^ that is a success on stable/stein | 14:04 |
| rlandy|ruck | in gate | 14:04 |
| rlandy|ruck | we don't have a lot of data here compared to other scenario tests | 14:05 |
| jfrancoa | rlandy|ruck: right...I had seen that the last 4 timeouts appeared in our patch, but there are other patches in which the job timeouted | 14:05 |
| rlandy|ruck | jfrancoa: yep - I am not ruling out any issue being on our side | 14:06 |
| rlandy|ruck | just observation | 14:06 |
| rlandy|ruck | http://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-scenario010-standalone&branch=stable%2Fstein | 14:07 |
| rlandy|ruck | this is not a test with a good record on stable/stein | 14:08 |
| *** gkadam has joined #oooq | 14:08 | |
| *** gkadam has quit IRC | 14:08 | |
| rlandy|ruck | jfrancoa: so the only job I see passing with stable/stein is https://review.opendev.org/#/c/661946/1/ci/environments/scenario010-standalone.yaml | 14:09 |
| rlandy|ruck | could it be that a rebase would help? | 14:10 |
| rlandy|ruck | give you that change to get the test to passing | 14:10 |
| jfrancoa | rlandy|ruck: yes...I was seeing that one right now..and it happens to take too long to install ceph-ansible http://logs.openstack.org/25/664225/2/gate/tripleo-ci-centos-7-scenario010-standalone/62625aa/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz#_2019-06-11_07_59_54 | 14:12 |
| rlandy|ruck | jfrancoa: I can run a test project job to execute just that test if it helps | 14:13 |
| rlandy|ruck | but ut might be quicker to just rebase, recheck | 14:13 |
| rlandy|ruck | it | 14:13 |
| rlandy|ruck | really this test is not on our radar screen - which is bad | 14:14 |
| jfrancoa | rlandy|ruck: thanks, but the issue seems to be here http://logs.openstack.org/25/664225/2/gate/tripleo-ci-centos-7-scenario010-standalone/62625aa/logs/undercloud/home/zuul/undercloud-ansible-HmElX0/ceph-ansible/ceph_ansible_command.log.txt.gz#_2019-06-11_08_01_28_284 , maybe I should just open a lp and ping Ceph folks | 14:14 |
| jfrancoa | rlandy|ruck: thanks a lot for your help, it was very useful! | 14:15 |
| rlandy|ruck | jfrancoa: sure - pls log the lp - I will track it | 14:15 |
| rlandy|ruck | also we were not tracking this test - I am adding it to sove | 14:15 |
| rlandy|ruck | sova | 14:15 |
| rlandy|ruck | it appears we trigger it infrequently | 14:16 |
| rlandy|ruck | jfrancoa: lastly - if you are desperate to get the patch through, there is the non-voting option - although that is somewhat cheating :) | 14:17 |
| rlandy|ruck | and weshay would need to approve | 14:17 |
| jfrancoa | rlandy|ruck: well, it's quite important to merge it asap downstream, but let's try to fix the issue first...if it gets too hard to solve we'll go for the cheating path :-) | 14:18 |
| jfrancoa | rlandy|ruck: thanks again | 14:19 |
| *** jtomasek_ has quit IRC | 14:20 | |
| *** jtomasek_ has joined #oooq | 14:22 | |
| jfrancoa | rlandy|ruck: https://bugs.launchpad.net/tripleo/+bug/1832597 | 14:24 |
| openstack | Launchpad bug 1832597 in tripleo "tripleo-ci-centos-7-scenario010-standalone time outs in gate checks" [High,Triaged] - Assigned to John Fulton (jfulton-org) | 14:24 |
| rlandy|ruck | jfrancoa: thanks | 14:25 |
| rlandy|ruck | will keep watch on the status | 14:25 |
| derekh | panda: I'm still trying to sort out scenario012 and redeployed a new env today, it now seems to download ovn container images to the undercloud, did you fix something as this is part of what I was trying to figure out in my previous env | 14:29 |
| rlandy|ruck | ykarel|afk: seen this failure on fs001 stein? https://logs.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-stein/f4dc304/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2019-06-12_11_43_49 | 14:30 |
| panda | derekh: no, I did not change anything there | 14:33 |
| derekh | panda: ok | 14:34 |
| *** ykarel|afk is now known as ykarel | 14:40 | |
| rlandy|ruck | chandankumar: hey ... | 14:41 |
| rlandy|ruck | chandankumar: we see this failure in gates ... | 14:41 |
| rlandy|ruck | http://logs.openstack.org/37/662337/3/gate/tripleo-ci-centos-7-standalone/0d724c0/job-output.txt.gz#_2019-06-12_07_33_51_769006 | 14:41 |
| rlandy|ruck | I figure it's really not your fault | 14:41 |
| rlandy|ruck | network connection | 14:41 |
| rlandy|ruck | but is there anything we can do to help it along? | 14:41 |
| ykarel | rlandy|ruck, yes it's known already uses address 172.18.0.80 | 14:44 |
| ykarel | happens randomly | 14:44 |
| rlandy|ruck | ykarel: working with kforde to get rod of stuck stacks today | 14:44 |
| rlandy|ruck | there since 05/17 | 14:45 |
| rlandy|ruck | I think that is killing us | 14:45 |
| ykarel | rlandy|ruck, ack | 14:45 |
| apetrich | weshay, can I expense watermelons? Asking for a friend (that is me) | 14:48 |
| marios | apetrich: wagyu beef? :D | 14:48 |
| marios | apetrich: i hear it goes well with watermelon ;) :D | 14:49 |
| apetrich | marios, only if in tartar form. it is way too hot | 14:49 |
| apetrich | marios, also one step at a time. :) otherwise weshay might notice | 14:49 |
| marios | apetrich: honestly man call me a heathen, and i *have* tried it more than once but i can't bring myself to love tartar | 14:50 |
| marios | now, a chargrilled ribeye on the other hand... me hungry | 14:50 |
| *** blueDuckRandomSt has joined #oooq | 14:51 | |
| apetrich | marios, I don't eat mayo or like ketchup. So it is a bit hard to find a good one :) | 14:51 |
| apetrich | marios, picanha here is almost the same price as ground meat. I think any meat that I'm having for months is picanha :) grilled, minced, cubed | 14:52 |
| *** rfolco has quit IRC | 14:52 | |
| apetrich | you name it. I'm using picanha for that :) | 14:52 |
| rlandy|ruck | weshay: 1-on-1? | 14:52 |
| ykarel | jfrancoa, yes you need to rebase on https://review.opendev.org/#/c/661946 | 14:52 |
| ykarel | that timeout is seen issue in other scenarios as well earlier and was worked around by disabling ceph v1 | 14:53 |
| weshay | rlandy|ruck give me 5 min please | 14:53 |
| *** blueDuckRandom has quit IRC | 14:53 | |
| rlandy|ruck | ykarel: already suggested the rebase :) | 14:54 |
| ykarel | rlandy|ruck, okk cool | 14:54 |
| marios | apetrich: thanks for making me hungry ! :D | 14:55 |
| apetrich | marios, glad to be of service :) | 14:56 |
| jfrancoa | ykarel: great, thanks a lot for the info. I'll do it right away | 14:56 |
| jfrancoa | ykarel: could I include that patch as the solution for https://bugs.launchpad.net/tripleo/+bug/1832597 ? | 14:58 |
| openstack | Launchpad bug 1832597 in tripleo "tripleo-ci-centos-7-scenario010-standalone time outs in gate checks" [High,Triaged] - Assigned to Francesco Pantano (fmount) | 14:58 |
| *** sanjayu_ has quit IRC | 14:58 | |
| ykarel | jfrancoa, yes | 14:58 |
| weshay | rlandy|ruck will be about 20 | 14:58 |
| rlandy|ruck | weshay: no rush | 14:58 |
| *** marios has quit IRC | 15:02 | |
| zbr|flow | does anyone know why i do no find a change in upstream zuul 4h after it received wf? https://review.opendev.org/#/c/652971/ | 15:03 |
| zbr|flow | apparently zuul does not plan to run it, and if I do a recheck i guess it will run the check again. | 15:04 |
| *** marios has joined #oooq | 15:04 | |
| rlandy|ruck | it will | 15:04 |
| *** aakarsh has quit IRC | 15:04 | |
| *** jaosorior has quit IRC | 15:06 | |
| *** jfrancoa has quit IRC | 15:07 | |
| *** Vorrtex has joined #oooq | 15:09 | |
| weshay | chandankumar do you have any ex-colleagues that are interested in working at rh that you would recommend? | 15:11 |
| weshay | rlandy|ruck ready when you are | 15:16 |
| rlandy|ruck | weshay: joining | 15:17 |
| *** rlandy|ruck is now known as rlandy|ruck|mtg | 15:17 | |
| zbr|flow | weshay: i do happen to know one ex-collegue that I would recommend but he is based in London... | 15:20 |
| weshay | rlandy|ruck|mtg http://codesearch.openstack.org/?q=tripleo-build-containers-jobs&i=nope&files=&repos= | 15:24 |
| zbr|flow | @oooq: just learnt a new trick. to force zuul to add change to the queue if it missed it (due to unmerged deps), remove and readd workflow to it. | 15:26 |
| * marios almost out | 15:33 | |
| zbr|flow | weshay: or marios : give this a kick: https://review.opendev.org/#/c/651772/ | 15:35 |
| zbr|flow | you already reviewed it. | 15:35 |
| *** bogdando has quit IRC | 15:36 | |
| marios | zbr|flow: in the morning | 15:37 |
| *** marios has quit IRC | 15:39 | |
| *** Vorrtex has quit IRC | 15:59 | |
| *** ykarel is now known as ykarel|away | 16:03 | |
| *** Goneri has quit IRC | 16:05 | |
| *** tosky has quit IRC | 16:10 | |
| *** rlandy|ruck|mtg is now known as rlandy|ruck | 16:12 | |
| *** kopecmartin is now known as kopecmartin|off | 16:16 | |
| weshay | arxcruz do you have a test job that will trigger fs021 on this https://review.opendev.org/#/q/topic:featureset021-whitelist+(status:open+OR+status:merged) ? | 16:16 |
| *** Goneri has joined #oooq | 16:18 | |
| *** Goneri has quit IRC | 16:24 | |
| *** holser_ is now known as holser|eod | 16:35 | |
| *** Goneri has joined #oooq | 16:36 | |
| *** dtantsur is now known as dtantsur|afk | 16:45 | |
| *** holser|eod has quit IRC | 16:47 | |
| *** derekh has quit IRC | 16:50 | |
| *** blueDuckRandom has joined #oooq | 16:51 | |
| *** blueDuckRandomSt has quit IRC | 16:53 | |
| *** aakarsh has joined #oooq | 16:54 | |
| *** amoralej is now known as amoralej|off | 16:58 | |
| *** rfolco has joined #oooq | 17:02 | |
| *** tosky has joined #oooq | 17:05 | |
| chandankumar | weshay: need to check but for which posts? | 17:12 |
| weshay | chandankumar either ci or tempest | 17:13 |
| *** jpena is now known as jpena|off | 17:13 | |
| *** ykarel|away has quit IRC | 17:24 | |
| weshay | chandankumar rfolco sshnaidm rlandy|ruck https://review.opendev.org/#/c/663702/ | 17:27 |
| *** holser_ has joined #oooq | 17:37 | |
| rlandy|ruck | sshnaidm: which sova branch do I use to add a check job? tripleo-ci-centos-7-scenario010-standalone | 17:40 |
| weshay | sshnaidm can you please review https://review.rdoproject.org/r/#/c/21075/ | 17:42 |
| weshay | shall we merge https://review.opendev.org/#/c/649965 ? | 17:44 |
| chandankumar | rlandy|ruck: https://review.opendev.org/#/c/664857/ please have a look when free | 17:47 |
| weshay | also need eyes on https://review.rdoproject.org/r/#/c/21099/ | 17:48 |
| chandankumar | do we have a bug for tracking overcloud timedout in rdo third party jin? | 17:48 |
| weshay | chandankumar example? | 17:49 |
| chandankumar | weshay: https://logs.rdoproject.org/24/639324/26/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039/c6bf5ea/job-output.txt.gz#_2019-06-12_12_15_03_099579 | 17:49 |
| weshay | ya.. 39 is just not stable :( | 17:49 |
| weshay | also non-voting atm | 17:49 |
| weshay | chandankumar it runs for 3 hours, 23 minutes when passing | 17:50 |
| weshay | what is the timeout set to? | 17:50 |
| weshay | we need to give that job a little more time because of the ipa setup | 17:50 |
| chandankumar | 12600 | 17:50 |
| weshay | chandankumar it's also hitting retry limits | 17:51 |
| weshay | only a couple timeouts | 17:51 |
| chandankumar | weshay: ok, will check this job tomorrow then | 17:52 |
| weshay | chandankumar would be interesting to run that job against vexx | 17:52 |
| weshay | see if that has better results | 17:52 |
| rlandy|ruck | chandankumar: +2'ed https://review.opendev.org/#/c/664857/5 - we we look at putting the bm logs upstream, will have to check this as we to make sure we don't leak access to hardware | 17:53 |
| rlandy|ruck | atm it's fine | 17:53 |
| chandankumar | rlandy|ruck: it is the content of stackrc and overclourc | 17:53 |
| chandankumar | which we collect alreadt | 17:54 |
| chandankumar | *already | 17:54 |
| rlandy|ruck | suppose that is true | 17:54 |
| rlandy|ruck | we do expose stackrc | 17:54 |
| chandankumar | https://logs.rdoproject.org/57/664857/5/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/dd7cf2e/logs/undercloud/home/zuul/stackrc.gz and https://logs.rdoproject.org/57/664857/5/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/dd7cf2e/logs/undercloud/home/zuul/overcloudrc.gz | 17:56 |
| rlandy|ruck | weshay: is this correct for the containers build jobs? https://review.opendev.org/#/c/664996 | 18:17 |
| *** udesale has quit IRC | 18:20 | |
| rlandy|ruck | on second thoughts - since tripleo-ci is not branched, maybe should have templates per release | 18:21 |
| *** Goneri has quit IRC | 18:26 | |
| *** holser_ has quit IRC | 18:40 | |
| sshnaidm | rlandy|ruck, if tripleo-ci-centos-7-scenario010-standalone is in upstream, then for master branch and gates (if it gates) | 18:41 |
| sshnaidm | rlandy|ruck, commented https://review.rdoproject.org/r/#/c/21075/ | 18:46 |
| rlandy|ruck | sshnaidm: thanks | 18:51 |
| rlandy|ruck | sova reviews coming up | 18:52 |
| *** tesseract has quit IRC | 18:54 | |
| weshay | rlandy|ruck ya.. that looks right to me https://review.opendev.org/#/c/664996/2/zuul.d/layout.yaml | 19:01 |
| rlandy|ruck | weshay: on second thoughts - I think we should create the per-release templates | 19:01 |
| rlandy|ruck | in case we add more tests | 19:02 |
| rlandy|ruck | saves us updating across repos | 19:02 |
| rlandy|ruck | like ... | 19:02 |
| weshay | rlandy|ruck although it didn't trigger the jobs | 19:02 |
| weshay | perhaps we need to add zuul files to the file list | 19:02 |
| weshay | rlandy|ruck ya.. that is fine too.. to create templates | 19:03 |
| rlandy|ruck | there is no file list | 19:03 |
| rlandy|ruck | better | 19:03 |
| weshay | rlandy|ruck http://zuul.openstack.org/status filter on the review | 19:03 |
| weshay | 664996 | 19:03 |
| weshay | build jobs did not trigger | 19:03 |
| rlandy|ruck | weird because there is no file list | 19:04 |
| rlandy|ruck | and the change is stable/stein | 19:05 |
| rlandy|ruck | weshay: k - let's go the template route anyways | 19:05 |
| rlandy|ruck | will redo this review with templates | 19:05 |
| weshay | chandankumar rfolco the rhui w/ rhel8 content is working in rdo-cloud | 19:12 |
| weshay | rlandy|ruck can I download the rhel8 image that is used in internal sf? | 19:13 |
| *** Goneri has joined #oooq | 19:14 | |
| rfolco | weshay, do you know why my vars in the job definition are not visible from tqe playbooks ? release has the right content... but docker_* vars are just defined when I manually inject a set_fact here.... | 19:14 |
| rfolco | https://sf.hosted.upshift.rdu2.redhat.com/logs/28/172928/8/check/tripleo-ci-rhel-7-standalone-rocky/0838911/job-output.txt.gz#_2019-06-12_14_16_11_168436 | 19:14 |
| rfolco | release = rocky, cool | 19:14 |
| rlandy|ruck | weshay: the fake one we construct? | 19:15 |
| weshay | rlandy|ruck ya | 19:15 |
| rfolco | but docker_registry_host is docker.io, while it should be rdo | 19:15 |
| rlandy|ruck | I am nit sure how you would do that | 19:15 |
| rlandy|ruck | if glance can let you | 19:15 |
| rfolco | vars are defined here... https://code.engineering.redhat.com/gerrit/#/c/172928/8/zuul.d/standalone-jobs.yaml | 19:16 |
| rlandy|ruck | glance image-download | 19:16 |
| weshay | rlandy|ruck ya.. I might need you to paste me creds | 19:16 |
| weshay | ya.. so I can do that | 19:16 |
| rlandy|ruck | weshay: that's going to cost you | 19:16 |
| weshay | rfolco not sure what you are asking me yet | 19:17 |
| * rlandy|ruck has to look | 19:17 | |
| weshay | you mean.. "docker_registry_namespace": "tripleorocky" | 19:17 |
| weshay | the ones you are setting | 19:17 |
| rfolco | docker_registry_host... yeah | 19:17 |
| weshay | in that patch | 19:17 |
| rfolco | when tqe playbook runs... | 19:17 |
| weshay | rfolco tqe is not in play in container builds | 19:17 |
| rfolco | https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/roles/standalone/tasks/main.yml#L37 | 19:17 |
| rfolco | weshay, this is standalone | 19:18 |
| rfolco | standalone job | 19:18 |
| weshay | oh oh | 19:18 |
| rfolco | then I injected a set_fact in tqe playbook | 19:18 |
| rfolco | https://sf.hosted.upshift.rdu2.redhat.com/logs/28/172928/8/check/tripleo-ci-rhel-7-standalone-rocky/0838911/job-output.txt.gz#_2019-06-12_14_16_11_684445 | 19:18 |
| weshay | rfolco jump on blue | 19:18 |
| rfolco | after that it gets my vars | 19:19 |
| rlandy|ruck | weshay: sending you creds in email | 19:19 |
| weshay | thanks | 19:19 |
| rlandy|ruck | that's from nodepool image - let me check | 19:20 |
| rlandy|ruck | weshay: you want image called cloud-rhel-8- | 19:26 |
| rlandy|ruck | but there are two of them | 19:26 |
| *** tosky has quit IRC | 19:26 | |
| rlandy|ruck | weshay: glance image-download --file cloud-8.qcow2 c87acc00-4def-4abd-a30e-7131e673eded | 19:32 |
| rlandy|ruck | I don't see the equivalent openstack command | 19:33 |
| rlandy|ruck | hmmm .... no containers build jobs either here | 19:34 |
| *** dtrainor has joined #oooq | 19:43 | |
| *** Goneri has quit IRC | 19:45 | |
| weshay | rlandy|ruck do you know where onfig/release/tripleo-ci/RedHat-7/rocky.yml is? | 20:00 |
| weshay | panda ^ | 20:00 |
| weshay | rfolco https://code.engineering.redhat.com/gerrit/#/c/171845/4 | 20:04 |
| weshay | rlandy|ruck do you have a moment | 20:05 |
| weshay | rlandy|ruck nvrmind | 20:08 |
| weshay | panda nvrmind | 20:08 |
| rlandy|ruck | weshay: no such file | 20:22 |
| weshay | rlandy|ruck ya.. I found how that happened | 20:22 |
| rlandy|ruck | hack | 20:23 |
| *** rf0lc0 has joined #oooq | 20:33 | |
| rlandy|ruck | weshay: neither of the two reviews triggered the containers build jobs ... because of tripleo-build-containers-base-with-files | 20:33 |
| rlandy|ruck | https://github.com/openstack/tripleo-ci/blob/e99487b28aedfeb0ad771adcb936b5e32fcd4584/zuul.d/base.yaml#L217 | 20:34 |
| rlandy|ruck | I'd need to force a change to say files * | 20:34 |
| *** rfolco has quit IRC | 20:36 | |
| rlandy|ruck | tripleo-build-containers-centos-7-steinqueued | 20:38 |
| rlandy|ruck | tripleo-build-containers-centos-7-buildah-stein | 20:38 |
| rlandy|ruck | there we go | 20:38 |
| *** aakarsh has quit IRC | 20:44 | |
| *** holser_ has joined #oooq | 20:48 | |
| rlandy|ruck | good - stein will promote again | 21:06 |
| *** rf0lc0 has quit IRC | 21:09 | |
| *** ksambor_ has quit IRC | 21:10 | |
| rlandy|ruck | sshnaidm: https://github.com/sshnaidm/sova/pull/67 | 21:25 |
| *** holser_ has quit IRC | 21:49 | |
| rlandy|ruck | weshay: ping | 21:51 |
| weshay | rlandy|ruck ahoy | 21:52 |
| rlandy|ruck | can you access rrcockpit? | 21:52 |
| rlandy|ruck | looks to me like it's lost connection | 21:52 |
| weshay | ah crud | 21:53 |
| rlandy|ruck | weshay: ^^ you see what I mean? | 21:53 |
| weshay | I can reach it.. but the values are not coming up | 21:53 |
| * weshay blames alex | 21:53 | |
| rlandy|ruck | alex? | 21:53 |
| rlandy|ruck | how? | 21:53 |
| rlandy|ruck | weshay: same here - accessible - but no values | 21:54 |
| rlandy|ruck | how did it die? | 21:54 |
| weshay | some of the data is up | 21:54 |
| rlandy|ruck | rdocloud upstream data gone | 21:54 |
| rlandy|ruck | bad network gateway | 21:55 |
| weshay | well.. just kind of kidding but he and I just ran a query on check jobs | 21:55 |
| rlandy|ruck | how would that impact anything? | 21:55 |
| weshay | rlandy|ruck want to tmate into the prod server? | 21:55 |
| rlandy|ruck | sure | 21:55 |
| weshay | going to blue too | 21:55 |
| *** aakarsh has joined #oooq | 22:24 | |
| *** rlandy|ruck is now known as rlandy|ruck|bbl | 22:39 | |
| *** holser_ has joined #oooq | 22:49 | |
| arxcruz | weshay i don't have yet, but i'll create a dummy patch tomorrow :) or today in a few hours hehehe | 22:50 |
| weshay | arxcruz thanks man | 22:51 |
| weshay | rlandy|ruck|bbl got it back... restarted docker; docker-compose down; docker-compose up | 22:51 |
| weshay | took a few minutes for the data to return | 22:51 |
| weshay | rlandy|ruck|bbl got it back... restarted docker; docker-compose down; docker-compose up -d | 22:51 |
| *** vinaykns has quit IRC | 23:13 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!