| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039 @ (2 more messages) | 01:02 |
|---|---|---|
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039 @ (2 more messages) | 03:02 |
| *** ykarel|away has joined #oooq | 03:18 | |
| *** ykarel|away is now known as ykarel | 03:20 | |
| *** udesale has joined #oooq | 03:51 | |
| *** udesale has quit IRC | 04:21 | |
| *** udesale has joined #oooq | 04:22 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039 @ (2 more messages) | 05:03 |
| *** skramaja has joined #oooq | 05:20 | |
| *** ratailor has joined #oooq | 05:28 | |
| *** saneax has joined #oooq | 05:48 | |
| *** jtomasek has joined #oooq | 06:15 | |
| *** chandankumar has quit IRC | 06:39 | |
| *** chandankumar has joined #oooq | 06:40 | |
| *** quiquell|off is now known as quiquell | 06:48 | |
| *** ykarel has quit IRC | 06:57 | |
| *** ykarel has joined #oooq | 06:58 | |
| *** jfrancoa has joined #oooq | 06:59 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039 @ (2 more messages) | 07:03 |
| *** jfrancoa has quit IRC | 07:05 | |
| *** jfrancoa has joined #oooq | 07:21 | |
| *** udesale has quit IRC | 07:27 | |
| *** udesale has joined #oooq | 07:27 | |
| *** ykarel is now known as ykarel|lunch | 07:45 | |
| *** holser_ has joined #oooq | 07:49 | |
| *** kopecmartin|off is now known as kopecmartin | 08:06 | |
| *** ykarel_ has joined #oooq | 08:09 | |
| *** ykarel|lunch has quit IRC | 08:11 | |
| *** ykarel_ is now known as ykarel | 08:11 | |
| quiquell | panda|ruck: ping | 08:20 |
| *** dtantsur|afk is now known as dtantsur | 08:21 | |
| arxcruz | ykarel: thanks for you response on https://bugs.launchpad.net/tripleo/+bug/1817154 | 08:21 |
| openstack | Launchpad bug 1817154 in tripleo " DuplicateOptError: duplicate option: barbican" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz) | 08:21 |
| arxcruz | now I am wondering, why this bug is marked as critical if it's not promotion blocker | 08:21 |
| arxcruz | also, I wasn't aware of the vitrage error | 08:22 |
| arxcruz | panda|ruck: rfolco|rover ^ | 08:22 |
| *** arxcruz sets mode: +v panda|ruck | 08:22 | |
| *** arxcruz sets mode: +v rfolco|rover | 08:22 | |
| ykarel | arxcruz, only the bug "title" is misleading, bug is there and is blocking promotion and hence promotion blocker | 08:23 |
| ykarel | description was correct | 08:23 |
| quiquell | marios: o/ | 08:23 |
| quiquell | marios: as expected the no_logs has nothing to do with the error at RDO registry login | 08:23 |
| quiquell | marios: Let's try a regenerated password just in case https://review.rdoproject.org/r/18971 | 08:23 |
| arxcruz | ykarel: so, there are 2 issues actually | 08:23 |
| ykarel | arxcruz, yes | 08:24 |
| marios | quiquell: ok cool thanks for looking into | 08:24 |
| arxcruz | one is the barbican, and the object has no attribute dest | 08:24 |
| marios | quiquell:++ | 08:24 |
| arxcruz | ykarel: now I understood :) | 08:24 |
| quiquell | marios: wait.. this is not the review | 08:24 |
| ykarel | ack | 08:24 |
| marios | quiquell: maybe we should have a call and coordinate a bit in a bit with zbr | 08:24 |
| arxcruz | quiquell++ | 08:24 |
| hubbot1 | arxcruz: quiquell's karma is now 19 | 08:24 |
| marios | quiquell: zbr about stories 650 651 652 | 08:24 |
| marios | arxcruz: urgh https://review.openstack.org/#/c/638740/ | 08:25 |
| *** amoralej|off is now known as amoralej | 08:26 | |
| marios | quiquell: https://review.rdoproject.org/r/#/c/18971/ is that right? | 08:29 |
| quiquell | marios: now it is I forgot to add the secret | 08:29 |
| quiquell | ykarel, arxcruz: Didn't OVN guys have a fix to the barbican error ? | 08:29 |
| quiquell | ykarel, amoralej: can we merge this https://review.rdoproject.org/r/18971 ? | 08:30 |
| quiquell | biab | 08:31 |
| *** quiquell is now known as quiquell|brb | 08:31 | |
| ykarel | quiquell, for OVN /me don't know who was working on it | 08:31 |
| marios | quiquell|brb: ack thanks checking in sec | 08:32 |
| quiquell|brb | ykarel: there were like two/three reviews for fixes | 08:32 |
| ykarel | quiquell|brb, re. secret, why new secret? why old secret working in other container build job? | 08:32 |
| quiquell|brb | ykarel: don't know man, I have just test the same job at local zuul and everyting is working fine | 08:32 |
| quiquell|brb | ykarel: only different stuff is the secrets | 08:32 |
| quiquell|brb | ykarel: Can we merge this review and just to test it ? | 08:33 |
| quiquell|brb | ykarel: if it's not working we can revert it | 08:33 |
| ykarel | quiquell|brb, no issue with merge, just trying to understand the situation | 08:33 |
| quiquell|brb | ykarel: is just new secret and used by new job not involve at current container build | 08:33 |
| quiquell|brb | ykarel: more or less dibugging what's up with this secret and the error | 08:33 |
| quiquell|brb | debugging | 08:33 |
| quiquell|brb | OK go back in a few | 08:34 |
| ykarel | quiquell|brb, re. barbican can you share the reviews with arxcruz so he can check if that fixes the barbican duplicate key error | 08:34 |
| quiquell|brb | Let me find it | 08:34 |
| ykarel | or may be u are talking about some other barbican error | 08:34 |
| quiquell|brb | maybe | 08:34 |
| *** zbr|ssbarnea has joined #oooq | 08:34 | |
| ykarel | quiquell|brb, /me looking at https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-centos-7-master-containers-build-push/09b78f7/job-output.txt.gz | 08:36 |
| ykarel | can you point where is the error exactly | 08:36 |
| ykarel | Login to RDO registr? | 08:36 |
| ykarel | and with local zuul egistry access was successful, which secret you used there | 08:37 |
| quiquell|brb | ykarel: yep RDO registry | 08:39 |
| quiquell|brb | ykarel: local zuul works just fine | 08:39 |
| quiquell|brb | ykarel: same parent job same stuff in the playbook | 08:39 |
| ykarel | so which secret you used there | 08:39 |
| quiquell|brb | ykarel: encrypted with encrypt_secret.py --tenant rdoproject.org https://softwarefactory-project.io/zuul/ config | 08:40 |
| quiquell|brb | piping echo -n [rdo registry password] | 08:41 |
| ykarel | ack | 08:41 |
| quiquell|brb | ykarel: difference is, old password is used by bash script, don't know if maybe it's preocessing the password differently | 08:41 |
| quiquell|brb | ykarel: new playbook is using ansible docker login module | 08:41 |
| quiquell|brb | ykarel: Let's just merge to check we can revert that | 08:42 |
| ykarel | but old secret should have also worked, so it's something other than token, probably ^^ | 08:42 |
| ykarel | okk | 08:42 |
| quiquell|brb | ykarel: yep probably, is failing at least we can discard | 08:42 |
| * quiquell|brb hurry up and goes away | 08:42 | |
| ykarel | quiquell|brb, ack | 08:45 |
| *** ccamacho has joined #oooq | 08:45 | |
| ykarel | arxcruz, jobs passing after vitrage_tempest_plugin fix | 08:45 |
| ykarel | example:- https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-singlenode-featureset027-master/3486611/ | 08:46 |
| arxcruz | ykarel: did you submit the patch on vitrage upstream ? | 08:47 |
| ykarel | arxcruz, i added patch to vitrage-tempest-plugin package | 08:47 |
| arxcruz | ykarel: not upstream right ? | 08:47 |
| ykarel | arxcruz, now it's also fixed upstream | 08:47 |
| arxcruz | ykarel: ok, cool | 08:47 |
| arxcruz | ykarel++ | 08:47 |
| hubbot1 | arxcruz: ykarel's karma is now 9 | 08:47 |
| arxcruz | ykarel: i'll check the barbican error then | 08:48 |
| *** jpena|off is now known as jpena | 08:48 | |
| ykarel | arxcruz, i reverted RDO patch, mentioning upstream patch merged :- https://review.rdoproject.org/r/#/c/18970/ | 08:48 |
| ykarel | arxcruz, ack | 08:48 |
| *** chem has joined #oooq | 08:51 | |
| *** tosky has joined #oooq | 08:55 | |
| *** bogdando has joined #oooq | 09:00 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039 @ (2 more messages) | 09:03 |
| *** quiquell|brb is now known as quiquell | 09:07 | |
| quiquell | marios, zbr|ssbarnea: We have the new password in place | 09:09 |
| quiquell | will have to wait until next periodic to see how it works | 09:10 |
| *** holser_ has quit IRC | 09:11 | |
| zbr|ssbarnea | not sure about which password are you talking | 09:11 |
| *** chem` has joined #oooq | 09:11 | |
| quiquell | zbr|ssbarnea: issue with RDO registry and docker login at new containers build job | 09:12 |
| *** zbr|ssbarnea is now known as zbr|out | 09:13 | |
| *** chem has quit IRC | 09:15 | |
| *** chem` has quit IRC | 09:17 | |
| marios | zbr|out: quiquell o/ | 09:27 |
| marios | zbr|out: quiquell lets have a quick sync about the container build tasks f28/centos stories 650 651 652 | 09:27 |
| quiquell | marios: ack | 09:28 |
| marios | quiquell: i'll send invite don't know how logn zbr|out is out, lets say in 3 hours just before scrum or something? | 09:28 |
| marios | otherwise we can do it sooner if he's back | 09:28 |
| marios | don't know | 09:28 |
| quiquell | marios: I will have to leave in the middle of the scrum though | 09:28 |
| zbr|out | marios: lets do it now, send bj. | 09:29 |
| quiquell | akc | 09:29 |
| marios | zbr|out: quiquell cool yes | 09:29 |
| marios | https://redhat.bluejeans.com/7661925373/ | 09:29 |
| marios | kpomomg | 09:29 |
| marios | joining even :) | 09:29 |
| marios | panda|ruck: rfolco|rover fyi ^ we are talking about stories 650 651 652 | 09:29 |
| *** derekh has joined #oooq | 09:30 | |
| arxcruz | quiquell: do they have? can you pass me the review? | 09:32 |
| quiquell | arxcruz: nah was other stuff | 09:32 |
| marios | https://review.openstack.org/#/c/636160/ quiquell here | 09:38 |
| *** ccamacho has quit IRC | 09:38 | |
| *** holser_ has joined #oooq | 09:50 | |
| marios | zbr|out: quiquell 25 mins in reminder :D | 09:54 |
| panda|ruck | hello monday | 09:56 |
| quiquell | panda|ruck: hello there | 09:56 |
| *** holser_ has quit IRC | 09:57 | |
| *** holser_ has joined #oooq | 09:58 | |
| panda|ruck | what did I miss ? | 10:04 |
| *** sshnaidm|off is now known as sshnaidm | 10:05 | |
| panda|ruck | I see barbican is still failing | 10:05 |
| panda|ruck | arxcruz: anything I can do to help ? | 10:07 |
| *** ykarel_ has joined #oooq | 10:12 | |
| *** ccamacho has joined #oooq | 10:13 | |
| *** ykarel has quit IRC | 10:14 | |
| quiquell | marios: https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/playbooks/tripleo-ci-periodic-base/containers-build.yaml#L17 | 10:20 |
| marios | https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/zuul.d/jobs.yaml#L494 | 10:20 |
| arxcruz | panda|ruck: well, there's a workaround in place at https://review.openstack.org/#/c/638740/ | 10:20 |
| arxcruz | panda|ruck: and this is the proper fix https://review.openstack.org/#/c/639049/ | 10:20 |
| *** chem has joined #oooq | 10:21 | |
| arxcruz | since it's on tempest side, need review from them, not sure if we will have it merged soon | 10:21 |
| *** ykarel_ is now known as ykarel | 10:21 | |
| panda|ruck | arxcruz: wow | 10:24 |
| panda|ruck | arxcruz: how's a redirection making the test work ? | 10:26 |
| arxcruz | panda|ruck: https://tree.taiga.io/project/tripleo-ci-board/task/784?kanban-status=1447276 | 10:27 |
| arxcruz | panda|ruck: basically, it fails due the duplicate option, and exit | 10:27 |
| tosky | arxcruz: are you sure that's the right fix? I think that the octavia plugin stepped a line | 10:27 |
| arxcruz | tosky: well, honestly, i don't know | 10:28 |
| arxcruz | tosky: an option is move this test to tempest itself | 10:28 |
| arxcruz | but eventually we will hit this again if one plugin depends on another | 10:28 |
| arxcruz | tosky: I'm open to sugestions | 10:29 |
| arxcruz | sugestions/options | 10:29 |
| tosky | arxcruz: talk with barbican people | 10:30 |
| tosky | arxcruz: also, did you see the last comment by ykarel in https://review.openstack.org/#/c/638740/ ? | 10:30 |
| tosky | which points to https://bugs.launchpad.net/tripleo/+bug/1817154/comments/5 | 10:31 |
| openstack | Launchpad bug 1817154 in tripleo " DuplicateOptError: duplicate option: barbican" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz) | 10:31 |
| arxcruz | tosky: yes, because are two failures | 10:32 |
| arxcruz | one the duplication error on barbican | 10:32 |
| arxcruz | and the other is the mistyped on vitrage | 10:32 |
| arxcruz | ykarel: ^ | 10:33 |
| panda|ruck | arxcruz: so is the error on barbican fatal too ? | 10:33 |
| tosky | according ykarel, not for promotions | 10:33 |
| tosky | if I read that comment correctly | 10:33 |
| ykarel | tosky, yes right, that barbican error is not fatal, it returns 0 exit code | 10:34 |
| ykarel | the other vitrage error is fixed now | 10:34 |
| panda|ruck | ykarel: makes sense otherwise we would not see that error after the traceback | 10:34 |
| ykarel | yes | 10:35 |
| arxcruz | tosky: yes, not for promotion | 10:35 |
| ykarel | but good to fix those error directly which causes it, not by doing tempest init > /dev/null | 10:35 |
| ykarel | which may hide some real issues | 10:35 |
| arxcruz | up to you guys, the right patch is already on tempest side, and ykarel and tosky are on review | 10:35 |
| arxcruz | :) | 10:36 |
| arxcruz | I also don't like workarounds | 10:36 |
| ykarel | arxcruz, /me not right persion for tempest review :) good to involve tempest guys | 10:36 |
| tosky | arxcruz: what I would do is ping both gmann and the barbican people | 10:36 |
| panda|ruck | I like workarounds when we don't know when the fixx will merge | 10:36 |
| tosky | because I really think that the check there was put for some reasons | 10:36 |
| panda|ruck | barbican is a perfect name for a fallout tribe. | 10:37 |
| *** jaosorior has joined #oooq | 10:37 | |
| panda|ruck | (tell me why) I don't like Mooondays. | 10:38 |
| arxcruz | panda|ruck: why? | 10:39 |
| arxcruz | tosky: what's the official tempest channel? u know ? | 10:40 |
| tosky | arxcruz: #openstack-qa as usual | 10:40 |
| tosky | but gmann moved timezone, he may not be around yet | 10:40 |
| *** ccamacho has quit IRC | 10:43 | |
| *** ccamacho has joined #oooq | 10:43 | |
| *** ccamacho has quit IRC | 10:45 | |
| quiquell | marios: https://github.com/openstack/tripleo-quickstart/blob/master/config/release/tripleo-ci/CentOS-7/promotion-testing-hash-master.yml | 10:47 |
| panda|ruck | did you guys change the registry password ? | 10:48 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039 @ (2 more messages) | 11:03 |
| quiquell | sshnaidm: ping | 11:05 |
| sshnaidm | quiquell, pong | 11:06 |
| quiquell | sshnaidm: do you have time ? | 11:06 |
| sshnaidm | quiquell, yep | 11:06 |
| quiquell | sshnaidm: I am looking at this BM task https://tree.taiga.io/project/tripleo-ci-board/task/782?kanban-status=1447274 | 11:06 |
| quiquell | sshnaidm: I was thinking that maybe as an alternative to nodepool static provider | 11:06 |
| quiquell | sshnaidm: maybe we can use openstack provider if we use ironic with those BM machiens ? | 11:07 |
| quiquell | sshnaidm: Or it does not make sense at all ? | 11:07 |
| sshnaidm | quiquell, as I understand (and it can be wrong) we use a vm as nodepool node, and then just connect to BMs by ssh. | 11:08 |
| sshnaidm | quiquell, not sure how openstack provider will help here.. | 11:09 |
| quiquell | sshnaidm: Ack I have totally misunderstand the thing | 11:09 |
| sshnaidm | quiquell, but I may be also wrong, need to talk with rlandy about it | 11:10 |
| quiquell | sshnaidm: The idea is to use those nodes as the subnodes and the nodepool as the undercloud is that it ? | 11:11 |
| quiquell | sshnaidm: Then I don't get the relation to static driver | 11:11 |
| sshnaidm | quiquell, not sure, I thought we run undercloud on BM too | 11:11 |
| quiquell | sshnaidm: we can use openstack provider to get this VM | 11:11 |
| quiquell | sshnaidm: Will ask her too | 11:11 |
| sshnaidm | quiquell, actually vm works as "jenkins slave", just for connecting to BMs, but openstack should be BM based only, including undercloud | 11:12 |
| *** ccamacho has joined #oooq | 11:12 | |
| sshnaidm | quiquell, unlike OVB where we have undercloud installed on nodepool node | 11:13 |
| *** ccamacho has quit IRC | 11:13 | |
| quiquell | sshnaidm: is like the zuul-executor | 11:13 |
| *** ccamacho has joined #oooq | 11:14 | |
| quiquell | sshnaidm: and we cannot connect those BMs to nodepool useing openstack provider ? | 11:14 |
| sshnaidm | quiquell, on phase2 it was jenkins and it's slaves (or "workers" as politically correct now), and we need to use nodepool nodes instead of this slaves | 11:14 |
| quiquell | so we don't need the VM ? | 11:15 |
| sshnaidm | quiquell, nodepool can work with clouds or same host only afaik, it's not aware of baremetal | 11:15 |
| quiquell | sshnaidm: well if we deploy an opesntack with ironic it will work too | 11:15 |
| sshnaidm | quiquell, openstack, aws, openshift, kubernetes, that's it | 11:15 |
| sshnaidm | quiquell, I'm not sure, it needs an openstack cloud with nova to bring up VMs | 11:16 |
| sshnaidm | quiquell, it doesn't know about anything else, ironic, heat, whatever.. | 11:16 |
| sshnaidm | quiquell, it might be an idea for plugin for nodepool - to start/destroy BMs with ironic, but it's not there | 11:17 |
| quiquell | sshnaidm: humm there is no nova -> ironic stuff ? | 11:17 |
| sshnaidm | quiquell, no | 11:17 |
| quiquell | sshnaidm: so it's explicit ironic :-/ | 11:17 |
| quiquell | ack | 11:17 |
| *** udesale has quit IRC | 11:18 | |
| sshnaidm | quiquell, maybe dtantsur or derekh have ideas, if it's possible to make ironic work with nodepool anyhow | 11:19 |
| quiquell | sshnaidm: going to ask | 11:19 |
| sshnaidm | but afaik it will require a separate plugin | 11:19 |
| marios | quiquell: reminder when you get a chance add stuff in https://tree.taiga.io/project/tripleo-ci-board/task/712 or file a new task under 652 | 11:22 |
| marios | quiquell: *story 652 | 11:22 |
| quiquell | marios: done | 11:25 |
| dtantsur | sshnaidm: I don't know much about nodepool, but I don't see why it wouldn't be possible | 11:26 |
| quiquell | dtantsur: current nodepool openstack provider only knows about nova not ironic | 11:26 |
| sshnaidm | dtantsur, can ironic set up BMs and then report their IPs? | 11:27 |
| sshnaidm | quiquell, if so, we can use static host driver ^^ | 11:27 |
| *** ccamacho has quit IRC | 11:28 | |
| sshnaidm | well, theoretically | 11:28 |
| quiquell | sshnaidm: but where you set upt the ironic setup ? | 11:29 |
| quiquell | sshnaidm: we will have to generate nodepool config | 11:29 |
| sshnaidm | quiquell, yeah, exactly, that's why "theoretically" :D | 11:29 |
| chandankumar | arxcruz: this https://review.openstack.org/#/c/639049/ will go in plugin side | 11:29 |
| chandankumar | not tempest side | 11:30 |
| arxcruz | chandankumar: I don't agree, why barbican should add it on his side because octavia resolves to use their config? | 11:31 |
| arxcruz | ¯\_(ツ)_/¯ | 11:31 |
| arxcruz | but whatever the majority decides | 11:31 |
| chandankumar | let me try that | 11:32 |
| quiquell | sshnaidm: tristanC is saying flavor is enough to run BM with ironic and nodepool #zuul | 11:35 |
| dtantsur | sshnaidm: you can use ironic behind nova (like in ooo) or you can run it separately (e.g. using metalsmith) | 11:37 |
| dtantsur | depends on your case | 11:38 |
| quiquell | dtantsur: that's what I mean before | 11:38 |
| quiquell | dtantsur: like nodepool->nova->ironic | 11:38 |
| arxcruz | chandankumar: https://review.openstack.org/#/c/638502/ | 11:40 |
| arxcruz | rfolco|rover: did this yesterday, although is abandoned | 11:40 |
| arxcruz | lets see what gman will say | 11:40 |
| chandankumar | ok | 11:41 |
| arxcruz | i'm tending to agree with tosky and octavia crossed a line | 11:42 |
| *** saneax has quit IRC | 11:42 | |
| *** saneax has joined #oooq | 11:43 | |
| quiquell | dtantsur: like this ? https://docs.openstack.org/ironic/pike/install/configure-nova-flavors.html | 11:43 |
| sshnaidm | quiquell, so we need to have openstack installed somewhere for that | 11:49 |
| sshnaidm | quiquell, and BMs registered in ironic with specific nova flavor afaiu | 11:50 |
| sshnaidm | quiquell, and then using the regular openstack driver it will create BMs like nodepool nodes | 11:50 |
| quiquell | sshnaidm: yep, for sure we have an openstack downstream | 11:50 |
| quiquell | sshnaidm: with ironic support | 11:51 |
| quiquell | sshnaidm: so then we can use BM as normal nodepool nodes | 11:51 |
| quiquell | will talk with rlandy about it | 11:51 |
| quiquell | But looks like doable | 11:51 |
| sshnaidm | quiquell, well, but we need to bring a few nodes to install OS on them, and we'll get something like "multinode" | 11:51 |
| sshnaidm | quiquell, and we need to *provision* these nodes with ironic - it's part of test | 11:52 |
| sshnaidm | quiquell, not to have them already provisioned | 11:52 |
| dtantsur | quiquell: yep, but don't use the "pike" link unless you actually use Pike. the latest is https://docs.openstack.org/ironic/latest/install/configure-nova-flavors.html | 11:53 |
| quiquell | dtantsur: yep sorry | 11:53 |
| sshnaidm | quiquell, it looks like same problem of nodepool to support OVB | 11:53 |
| *** ratailor has quit IRC | 11:54 | |
| quiquell | sshnaidm: so part of the test is what nodepool already does ? | 11:54 |
| quiquell | sshnaidm: I mean starting the node with the correct iso install there ? | 11:54 |
| sshnaidm | quiquell, part of test - to see if ooo provisions BMs with ironic correctly, from nodepool we'll get already provisioned BM with some centos on it | 11:55 |
| quiquell | sshnaidm: yep that's multinode I see | 11:55 |
| sshnaidm | quiquell, yeah, exactly why we use OVB and not multinode | 11:55 |
| quiquell | we want to exercise introspect suff in the tests that's it ? | 11:56 |
| sshnaidm | quiquell, the whole provision process | 11:56 |
| sshnaidm | quiquell, like pxe, loading ipa image, inspection, loading os image, etc, etc | 11:57 |
| quiquell | I see | 11:57 |
| quiquell | sshnaidm: so this card is only about undercloud | 11:59 |
| quiquell | sshnaidm: maybe for undercloud real hardware this is all good | 11:59 |
| *** ccamacho has joined #oooq | 11:59 | |
| sshnaidm | quiquell, yeah, it's about connecting from nodepool node *vm* to future undercloud *BM* | 12:00 |
| sshnaidm | quiquell, and using secrets seems like straight-through approach here | 12:00 |
| quiquell | ok ok | 12:00 |
| sshnaidm | quiquell, but I don't know how undercloud itself is provisioned.. | 12:01 |
| quiquell | Yep secrets is the way | 12:01 |
| quiquell | sshnaidm: at OVB is part of the heat template we generate ? | 12:01 |
| *** panda|ruck is now known as panda|ruck|lunch | 12:02 | |
| sshnaidm | quiquell, well, we have 2 ways in ovb - one with heat template, as we used in rhos2, the other is to use nodepool node as we have now in 3d party | 12:02 |
| sshnaidm | quiquell, but you can't use heat with BM | 12:02 |
| sshnaidm | quiquell, something has to provision this bm | 12:03 |
| sshnaidm | quiquell, it needs to have OS already and SSH enabled there | 12:03 |
| sshnaidm | quiquell, with this rhos-jenkins key | 12:03 |
| quiquell | sshnaidm: maybe it make sense to have the undercloud BM hardware in the nodepool using flavors | 12:04 |
| quiquell | sshnaidm: or we test all the BM tripleo stuff at undercloud too ? | 12:04 |
| sshnaidm | quiquell, so we'll have it kind of OVB way now.. but I don't like this, because it pollutes host with all this stuff of nodepool | 12:04 |
| quiquell | sshnaidm: well we can reduce all the pollution we want diciding what to upt at diskimage builder | 12:05 |
| quiquell | sshnaidm: well I don't know if DIB will work I suppose we have to use cloud image | 12:06 |
| sshnaidm | quiquell, disk image builder - for vms, not BMs | 12:06 |
| quiquell | sshnaidm: so how were we doing the undercloud provisioning at phase2 ? | 12:06 |
| quiquell | sshnaidm: pre install tripleo ? :-) | 12:07 |
| sshnaidm | quiquell, yeah, that's the question to ask rlandy or weshay how we provision undercloud | 12:07 |
| quiquell | sshnaidm: ack will do thanks! | 12:07 |
| sshnaidm | quiquell, and how we clean all BMs from previous installations | 12:08 |
| quiquell | sshnaidm: there is another task for that I think | 12:08 |
| quiquell | A no it's about jobs | 12:09 |
| quiquell | sshnaidm: well we are injecting images using PXE so it will overwite stuff there ? | 12:09 |
| sshnaidm | quiquell, seems like so, yeah | 12:12 |
| sshnaidm | quiquell, at least with overcloud nodes | 12:12 |
| quiquell | ok | 12:13 |
| *** quiquell is now known as quiquell|lunch | 12:13 | |
| *** saneax has quit IRC | 12:17 | |
| rfolco|rover | arxcruz, did you find a fix for barbican dup config ? | 12:27 |
| arxcruz | rfolco|rover: i found not only one, but two! | 12:28 |
| arxcruz | rfolco|rover: ops, maybe 3 | 12:28 |
| arxcruz | rfolco|rover: 1 solution: add the try catch on tempest | 12:28 |
| *** quiquell|lunch is now known as quiquell | 12:28 | |
| arxcruz | 2 solution: your patch | 12:28 |
| arxcruz | 3 solution: make octavia guys remove the barbican call | 12:29 |
| rfolco|rover | my patch is abandoned. Should I restore it? Need to fix it, coz except cfg.DuplicateOptError: does not exist | 12:30 |
| rfolco|rover | arxcruz, do you have any ideas on what could put in line 36 ? https://review.openstack.org/#/c/638502/2/barbican_tempest_plugin/plugin.py | 12:31 |
| *** jpena is now known as jpena|lunch | 12:31 | |
| arxcruz | rfolco|rover: from oslo_config.cfg import DuplicateOptError | 12:32 |
| rfolco|rover | arxcruz, ok will restore the change then | 12:33 |
| rfolco|rover | thanks | 12:33 |
| bogdando | o/ PTAL https://review.openstack.org/639078 | 12:34 |
| quiquell | marios: I am going to add a task to track the password issue there is a lot of stuff already in the current tasks | 12:40 |
| *** udesale has joined #oooq | 12:42 | |
| *** fultonj has joined #oooq | 12:42 | |
| ykarel | rfolco|rover, panda|ruck|lunch is fs021 overcloud deployment failures already known? i see same failures since 19th Feb | 12:44 |
| ykarel | rfolco|rover, panda|ruck|lunch https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset021-master | 12:44 |
| rfolco|rover | ykarel, looking for opened bugs | 12:44 |
| marios | quiquell: ok cool | 12:44 |
| ykarel | rfolco|rover, ack | 12:44 |
| marios | quiquell: can you also remove in the other task | 12:45 |
| marios | quiquell: i mean wherever you already added stuff | 12:45 |
| quiquell | marios: cleanup and new task ack | 12:46 |
| quiquell | marios: btw, we need a change at config project to be able to run check job at https://review.openstack.org/#/c/636160 | 12:47 |
| quiquell | marios: running at RDO I mean | 12:47 |
| marios | zbr|out: ^^ | 12:47 |
| quiquell | marios: preparing the review | 12:47 |
| marios | quiquell: k add comment on gerrit i guess for now? | 12:47 |
| marios | quiquell: thanks | 12:47 |
| quiquell | marios: we need to merge this tripleo-build-containers-jobs is config project | 12:50 |
| rfolco|rover | ykarel, looks like this job is broken for a while | 12:52 |
| rfolco|rover | quiquell, you know about https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset021-master ? | 12:53 |
| arxcruz | rfolco|rover: fs021 is supposed to be broken | 12:54 |
| rfolco|rover | arxcruz, reason? | 12:54 |
| arxcruz | rfolco|rover: because it's running tempest without skip list | 12:54 |
| arxcruz | as a baseline for fs020 | 12:54 |
| quiquell | rfolco|rover: fs021 is not part of the creteria | 12:55 |
| quiquell | promotion criteria | 12:55 |
| marios | quiquell: ack tidy up and we discuss in 1 hour? me working on the ci repos thing so i can present it a bit too posted https://review.rdoproject.org/r/#/c/18975/ fyi updating the others from https://tree.taiga.io/project/tripleo-ci-board/task/773 momentarily | 12:56 |
| ykarel | rfolco|rover, quiquell arxcruz it's failing at overcloud deploy, that's not expected | 12:56 |
| ykarel | only tempest tests may fail | 12:56 |
| rfolco|rover | ykarel, I'll open a bug for it, high prio, not critical | 12:59 |
| ykarel | rfolco|rover, i would say it's critical also as the job is not doing it's purpose | 13:00 |
| ykarel | you can lower the priority for investigation at your end, once other priority task are finished you can pick it | 13:01 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci- (2 more messages) | 13:03 |
| quiquell | marios: done | 13:03 |
| quiquell | marios: nah we cannot run the job templates upstream at RDO :-( | 13:05 |
| quiquell | marios: we have to redefine them | 13:05 |
| quiquell | marios: we need to merge https://review.rdoproject.org/r/#/c/18913/ first | 13:10 |
| quiquell | marios: to be able to run tripleo-build-containers-fedora-28 as third party | 13:10 |
| weshay | panda|ruck|lunch rfolco|rover has there been any chatter about check jobs failing w/o any good reason | 13:14 |
| *** rlandy has joined #oooq | 13:15 | |
| weshay | arxcruz chandankumar what's the update on this https://bugs.launchpad.net/tripleo/+bug/1817154/ ? | 13:15 |
| openstack | Launchpad bug 1817154 in tripleo " DuplicateOptError: duplicate option: barbican" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz) | 13:15 |
| weshay | please update the trello cix | 13:15 |
| chandankumar | weshay: arxcruz just proposed the right fix | 13:16 |
| arxcruz | weshay: https://tree.taiga.io/project/tripleo-ci-board/task/784 | 13:16 |
| weshay | nice | 13:16 |
| arxcruz | weshay: i'll update the trello | 13:16 |
| chandankumar | weshay: https://review.openstack.org/639083 | 13:16 |
| quiquell | sshnaidm: What's the issue here ? https://review.rdoproject.org/r/#/c/18913/15/zuul.d/projects.yaml@9 | 13:17 |
| quiquell | sshnaidm: It's not bad to run jobs if we change the definition | 13:17 |
| quiquell | rlandy: o/ | 13:17 |
| weshay | arxcruz++ | 13:17 |
| hubbot1 | weshay: arxcruz's karma is now 13 | 13:17 |
| rfolco|rover | weshay, I am working on the opened bugs, do we have a bug for check jobs ? | 13:17 |
| rlandy | quiquell: hi | 13:18 |
| sshnaidm | quiquell, we don't do it for all these jobs we have there, why to do it for this one? You can always to run it in test project | 13:18 |
| quiquell | rlandy: was taking a look at BM stuff we have this sprint | 13:18 |
| quiquell | rlandy: Found that we can use nodepool openstack provider using flavors to connect to BM | 13:19 |
| quiquell | rlandy: don't know if this can help to provision undercloud | 13:19 |
| weshay | rfolco|rover you have any updates on fs037? https://trello.com/c/3P5xXu5s/888-cixlp1817331tripleociproa-periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-rocky-job-failing-on-update-step | 13:19 |
| sshnaidm | quiquell, I don't see any additional value there tbh | 13:19 |
| quiquell | sshnaidm: well I suppose you have the result in the review if you change the definition | 13:20 |
| rfolco|rover | weshay, I think I used wrong var, updated to required-projects.override-checkout and waiting for tests | 13:20 |
| rlandy | quiquell: "Found that we can use nodepool openstack provider using flavors to connect to BM" - yep, with the project secret, do you have more on that? | 13:20 |
| quiquell | rlandy: maybe I have to learn a little on BM do you have some blue time for me ? | 13:20 |
| rlandy | quiquell: yes - going to talk about it at the meeting | 13:21 |
| weshay | rfolco|rover thanks | 13:21 |
| rlandy | quiquell:we can blue after that if you want more info | 13:21 |
| quiquell | rlandy: have to leave earlier today, in fact I have to leav in the middle of scrum | 13:21 |
| quiquell | weshay: ^ | 13:21 |
| weshay | chandankumar you have a merge conflict on https://review.rdoproject.org/r/#/c/18917/ | 13:22 |
| rlandy | quiquell: k - we can chat now for a bit if you like | 13:22 |
| quiquell | rlandy: ack thanks let me connect to your blue | 13:22 |
| sshnaidm | quiquell, and when you change a name, you need to change a project.yaml too.. Well, I won't block it, but it won't help with anything there I think | 13:22 |
| quiquell | sshnaidm: ack, maybe we can do different reviews we need the project-template though | 13:22 |
| weshay | chandankumar can I move this https://logs.rdoproject.org/openstack-periodic-24hr/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-queens/e5b1de7/ to done re: https://trello.com/c/Fbub5Wix/880-cixlp1816414tripleociproa-queensfs017-telemetry-tempest-plugin-tests-failed-in-multinode-promoti | 13:24 |
| weshay | on-pipeline | 13:24 |
| chandankumar | weshay: https://review.rdoproject.org/r/#/c/18955/ and https://review.rdoproject.org/r/#/c/18957/ have done the job | 13:24 |
| chandankumar | weshay: or better move it to preventive action | 13:24 |
| weshay | chandankumar I did so | 13:25 |
| weshay | thank you | 13:25 |
| weshay | chandankumar++ | 13:25 |
| hubbot1 | weshay: chandankumar's karma is now 8 | 13:25 |
| *** panda|ruck|lunch is now known as panda|ruck | 13:26 | |
| weshay | panda|ruck|lunch you have a mtg in 4 min :) | 13:27 |
| ykarel | chandankumar, weshay but even without those patches jobs were passing, so probable RCA is missing for fs017 queens | 13:27 |
| weshay | ykarel ya.. I need to sync w/ you and a few other people this week | 13:27 |
| *** jpena|lunch is now known as jpena | 13:28 | |
| weshay | I think we have a few of these | 13:28 |
| ykarel | weshay, ack | 13:28 |
| ykarel | weshay, chandankumar https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-queens | 13:31 |
| chandankumar | ykarel: weshay there is a backport to queens gnocchi branch | 13:31 |
| ykarel | weshay, chandankumar mentioned patches in trello merged on 22nd, but this jobs is passing before htat | 13:31 |
| ykarel | only 1 failure | 13:32 |
| ykarel | as per https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-queens | 13:32 |
| chandankumar | https://github.com/gnocchixyz/gnocchi/commit/a81ea38f746018f3e58cd1974daefb80cb48a19c -> got merged, then it started passing | 13:32 |
| chandankumar | after that silhet cleaned some tests in telemetry tempest plugin which is needed also for queens | 13:33 |
| ykarel | chandankumar, why it started failing? | 13:33 |
| ykarel | and failed only once | 13:33 |
| *** amoralej is now known as amoralej|lunch | 13:36 | |
| chandankumar | ykarel: there were two issues one with heat stack delete and another was ip address failing may be somehting changed on that | 13:37 |
| panda|ruck | rfolco|rover: https://review.rdoproject.org/r/#/c/18956 override-checkout is a var in the job definition, not in var | 13:37 |
| *** udesale has quit IRC | 13:37 | |
| rfolco|rover | panda|ruck, sh*t | 13:38 |
| rfolco|rover | panda|ruck, thanks man | 13:38 |
| *** udesale has joined #oooq | 13:38 | |
| ykarel | rlandy, weshay can you revisit https://review.openstack.org/#/c/638438/ , this will reduce gate resets | 13:38 |
| panda|ruck | weshay: https://trello.com/c/3P5xXu5s/888-cixlp1817331tripleociproa-periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-rocky-job-failing-on-update-step affects all the stable branches, not just rocky. master is not affected | 13:38 |
| weshay | panda|ruck rfolco|rover any updates on https://bugs.launchpad.net/tripleo/+bug/1817370 | 13:38 |
| openstack | Launchpad bug 1817370 in tripleo "disk-image-create fails in ovb jobs intermittently" [Critical,New] | 13:38 |
| ykarel | weshay, rlandy gate job failed http://logs.openstack.org/21/639021/1/gate/tripleo-ci-centos-7-undercloud-containers/72026b3/job-output.txt.gz | 13:39 |
| ykarel | POST_FAILURES | 13:39 |
| chandankumar | ykarel: and the patch got merged on 20 itself then the job started passing, why it showed up, I have not investigated | 13:39 |
| weshay | ykarel you don't need to alert us to those | 13:39 |
| chandankumar | ykarel: i checked with silhet backporting it will fix the issue | 13:39 |
| Tengu | o_O how could something failing pep8 test get merged in python-tripleoclient ?! | 13:40 |
| ykarel | weshay, sorry, didn't get it | 13:40 |
| rfolco|rover | weshay, will take a look on this one in a bit | 13:40 |
| weshay | ykarel just asking if you are trying to alert us to something specific or that we've had a job that went into post_failure randomly | 13:41 |
| ykarel | weshay, nope because of that | 13:41 |
| ykarel | weshay, you two were the ones who had +2 and now random gate resets makes critical fixes to get merge | 13:42 |
| ykarel | and the review lying for long, just need attention so i pinged you guys | 13:42 |
| weshay | ykarel k k.. no worries.. you are doing a good thing | 13:43 |
| weshay | panda|ruck and rfolco|rover should be looking for trends and issues in http://dashboard-ci.tripleo.org/d/cEEjGFFmz/cockpit?orgId=1&panelId=61&fullscreen | 13:43 |
| weshay | ykarel I guess I'll I'm saying is that don't feel obligated to report every hiccup :) | 13:43 |
| * weshay looking at post_failures | 13:44 | |
| ykarel | weshay, /me do it only when it's get critical or didn't get noticed for long | 13:44 |
| weshay | ykarel k k | 13:44 |
| weshay | ykarel you know what you are doing.. not sure why I'm trying to suggest something :) | 13:45 |
| *** saneax has joined #oooq | 13:45 | |
| weshay | ykarel http://logs.openstack.org/21/639021/1/gate/tripleo-ci-centos-7-undercloud-containers/72026b3/job-output.txt.gz is this a timeout? | 13:45 |
| ykarel | weshay, yes because logs are collected twice | 13:45 |
| ykarel | and the patch i mentioned fixes it | 13:45 |
| panda|ruck | I actually poll the zuul queue in real time .. | 13:46 |
| *** skramaja_ has joined #oooq | 13:46 | |
| *** skramaja has quit IRC | 13:46 | |
| ykarel | chandankumar, ack, so it's still not clear what caused it | 13:47 |
| ykarel | but good it's fixe | 13:48 |
| weshay | ykarel++ | 13:48 |
| hubbot1 | weshay: ykarel's karma is now 10 | 13:48 |
| weshay | voted | 13:48 |
| ykarel | weshay, Thanks | 13:48 |
| weshay | panda|ruck can you please add a storyless task to figure how to prevent or block https://trello.com/c/DfQAjJrJ/819-cixlp1802971tripleociproa-tempest-volumebootpattern-and-basicops-running-concurrently-causing-timeouts | 13:52 |
| panda|ruck | weshay: ok | 13:52 |
| *** saneax has quit IRC | 13:55 | |
| *** skramaja_ has quit IRC | 13:57 | |
| weshay | quiquell sshnaidm did I get this right? https://review.rdoproject.org/r/#/c/18964/ | 14:00 |
| * chandankumar will miss scrum today | 14:00 | |
| weshay | np | 14:02 |
| weshay | rlandy mtg | 14:02 |
| panda|ruck | chandankumar: and scrum will miss you | 14:02 |
| quiquell | weshay: Looks ok, if the ansible module is returning empty list on transient errors | 14:02 |
| weshay | aye | 14:02 |
| *** ykarel_ has joined #oooq | 14:04 | |
| *** amoralej|lunch is now known as amoralej | 14:05 | |
| *** ykarel has quit IRC | 14:07 | |
| *** vinaykns has joined #oooq | 14:11 | |
| quiquell | jpena, ykarel_, amoralej: missing piece for to test secrets https://review.rdoproject.org/r/#/c/18980/ | 14:14 |
| quiquell | can we merge ? | 14:14 |
| ykarel_ | quiquell, +2 | 14:19 |
| weshay | sshnaidm can you vote on this change https://review.rdoproject.org/r/#/c/18964/ | 14:22 |
| *** zul has joined #oooq | 14:32 | |
| panda|ruck | rfolco|rover: https://review.rdoproject.org/r/#/c/18956/3/zuul.d/multinode-jobs.yaml option is just override-checkout, without the required-projects. part | 14:46 |
| panda|ruck | marios: do you want to chat after this call about the promotion job and the gate ? | 14:49 |
| *** ykarel_ is now known as ykarel|away | 14:54 | |
| marios | panda|ruck: well yes but i want to finish something else first for the tripleo-ci -repos and then setup tmate for weshay | 14:56 |
| marios | panda|ruck: can we do it after th call like half hour | 14:56 |
| panda|ruck | marios: up to you | 14:58 |
| marios | panda|ruck: can we say in 1 hour or too late for you? | 14:58 |
| rlandy | weshay: pls ping me some time today/this afternoon - to go through BM stuff. | 14:58 |
| weshay | rlandy k | 14:59 |
| panda|ruck | marios: ok for me | 14:59 |
| marios | panda|ruck: cool do you want to send invite and c rlandy and anyone else interested | 15:00 |
| marios | panda|ruck: tell me to fk of if u busy i send it | 15:00 |
| rlandy | panda|ruck: weshay: what happened to scenario009 multinode? not on this sprint? | 15:01 |
| *** udesale has quit IRC | 15:02 | |
| panda|ruck | rlandy: it became ruck/rover task | 15:02 |
| rlandy | panda|ruck; ack - ok | 15:03 |
| panda|ruck | rlandy: in the end, we are trying to get a job green, and that what ruck/rover usually do | 15:03 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci- (2 more messages) | 15:03 |
| rlandy | let me know if you need help there | 15:03 |
| panda|ruck | marios: invite sent, hope you like the title | 15:04 |
| marios | panda|ruck: o_O | 15:05 |
| marios | thanks! :) | 15:05 |
| marios | weshay: no tmate for beaker? centos? | 15:09 |
| marios | weshay: i added https://github.com/weshayutin.keys to authorized_keys | 15:10 |
| *** ykarel|away has quit IRC | 15:11 | |
| marios | weshay: giving you info in pvt so we don't spam | 15:11 |
| * panda|ruck has the mos eisley cantina theme stuck on his head | 15:24 | |
| weshay | panda|ruck can you add a brief agenda | 15:39 |
| *** agopi has quit IRC | 15:40 | |
| marios | weshay: added note under issue 2 @ | 15:45 |
| marios | ### UPDATE | 15:45 |
| marios | https://tree.taiga.io/project/tripleo-ci-board/task/765?kanban-status=1447275 | 15:46 |
| marios | weshay: here | 15:46 |
| weshay | ah thanks | 15:46 |
| rfolco|rover | panda|ruck, hmmm override-checkout is confusing, https://zuul-ci.org/docs/zuul/user/config.html?highlight=secret#attr-job.override-checkout | 15:47 |
| rfolco|rover | panda|ruck, See also the project-specific job.required-projects.override-checkout attribute to apply this behavior to a subset of a job’s projects. | 15:48 |
| rfolco|rover | this ^ makes me believe you're right. It applies to all projects. | 15:48 |
| *** chandankumar is now known as raukadah | 15:51 | |
| panda|ruck | rfolco|rover: yep | 15:56 |
| marios | panda|ruck: no bluejeans | 15:57 |
| marios | on the calendar event | 15:57 |
| rfolco|rover | panda|ruck, I beg Your forgiveness Your Majesty Panda, Lord of CI | 15:57 |
| weshay | rfolco|rover panda|ruck fyi.. the pass rate on fs001 is at 25% | 15:58 |
| rfolco|rover | weshay, new reproducer is good thing for https://bugs.launchpad.net/tripleo/+bug/1817370 ? | 15:59 |
| openstack | Launchpad bug 1817370 in tripleo "disk-image-create fails in ovb jobs intermittently" [Critical,Triaged] | 15:59 |
| weshay | rfolco|rover yes | 15:59 |
| marios | panda|ruck: bluejeans number please | 15:59 |
| panda|ruck | marios: in the meeting | 15:59 |
| marios | thanks | 16:00 |
| weshay | marios https://bluejeans.com/3492508669 | 16:00 |
| marios | weshay: thanks | 16:02 |
| weshay | rfolco|rover panda|ruck introspection is failing in rdo jobs | 16:19 |
| * weshay opens a bug | 16:19 | |
| panda|ruck | weshay: rfolco|rover we also have tripleo-ci-centos-7-undercloud-containers consistently post-failing in gates | 16:22 |
| rfolco|rover | I guess these two above are more urgent than the one I am working on https://bugs.launchpad.net/tripleo/+bug/1817370 | 16:23 |
| openstack | Launchpad bug 1817370 in tripleo "disk-image-create fails in ovb jobs intermittently" [Critical,Triaged] | 16:23 |
| panda|ruck | timeout problem | 16:23 |
| rfolco|rover | weshay, panda|ruck: sort_by_priority(bugs) | 16:23 |
| panda|ruck | rfolco|rover: the one you're woking on has been escalated and it's a promotion blocker for the updates job | 16:24 |
| weshay | https://bugs.launchpad.net/tripleo/+bug/1817598 | 16:24 |
| openstack | Launchpad bug 1817598 in tripleo "introspection ( prepare images ) failing in 3rd party ovb jobs fs001/35" [Critical,Triaged] | 16:24 |
| panda|ruck | in the disk-image-create bug is very difficult to understand what's the root cause unless you're familiar with all the layers there are there | 16:24 |
| weshay | rfolco|rover panda|ruck first thing is to check in w/ rhos-ops | 16:25 |
| weshay | and determine if rdo cloud is really back from the outtage | 16:25 |
| weshay | we have no failed stacks in infra atm | 16:26 |
| *** agopi has joined #oooq | 16:28 | |
| weshay | panda|ruck rfolco|rover who is picking up the interaction w/ ops? | 16:30 |
| rfolco|rover | weshay, is there a name to ask or broadcast ? | 16:31 |
| weshay | kforde alderman | 16:31 |
| rfolco|rover | k | 16:31 |
| weshay | panda|ruck it would be a really good idea to ensure introspection errors are captured by sova http://cistatus.tripleo.org/ | 16:33 |
| panda|ruck | weshay: rfolco|rover was filing a bug for the gates , I'll join the conversation in rhos-ops too | 16:33 |
| weshay | panda|ruck now that we have the bmc log | 16:33 |
| weshay | we can search for ocket.error: [Errno 99] Cannot assign requested address | 16:34 |
| weshay | panda|ruck want me to create a storyless task? | 16:35 |
| panda|ruck | weshay: no more specific error ? | 16:35 |
| weshay | sshnaidm do you know if ovb 2.0 handles "Cannot assign requested address" is a better way? | 16:35 |
| weshay | panda|ruck I think that's all you need for now | 16:36 |
| weshay | in sova | 16:36 |
| *** fultonj has quit IRC | 16:39 | |
| weshay | rfolco|rover need help? | 16:39 |
| *** jfrancoa has quit IRC | 16:41 | |
| weshay | panda|ruck I need to see a pull request by EOD | 16:43 |
| *** raukadah is now known as chandankumar | 16:44 | |
| panda|ruck | weshay: it's going to be difficult. This is a codebase I never worked on, and we are trying to look for a pattern in a new file that has no fixed name (contains the id number of the stack) | 16:47 |
| *** kopecmartin is now known as kopecmartin|off | 16:48 | |
| weshay | panda|ruck get on my lue | 16:49 |
| weshay | blue | 16:49 |
| weshay | I'll help you | 16:49 |
| marios | i believe that's a monday | 16:49 |
| marios | ttyl folks have a great day | 16:49 |
| weshay | thanks marios | 16:49 |
| panda|ruck | weshay https://bugs.launchpad.net/tripleo/+bug/1817602 | 16:55 |
| openstack | Launchpad bug 1817602 in tripleo "excessive errors in upgrade jobs causing logstash issues" [Critical,Triaged] | 16:55 |
| weshay | rfolco|rover join my blue | 16:57 |
| rfolco|rover | k | 16:58 |
| weshay | panda|ruck https://logs.rdoproject.org/98/604298/246/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/e9f347b/job-output.txt.gz#_2019-02-25_09_59_59_621141 | 16:59 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci- (2 more messages) | 17:03 |
| weshay | https://logs.rdoproject.org/86/638286/3/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/a7e1c2f/logs/undercloud/home/zuul/overcloud_image_build.log.txt.gz#_2019-02-22_19_16_55 | 17:05 |
| weshay | CalledProcessError: Command '['disk-image-create' | 17:05 |
| weshay | sshnaidm rlandy fyi.. two sova pull requests coming shortly | 17:05 |
| rlandy | k | 17:06 |
| *** bogdando has quit IRC | 17:06 | |
| weshay | rfolco|rover panda|ruck https://github.com/sshnaidm/sova/blob/master/tripleoci/config.py#L237-L244 | 17:12 |
| weshay | overcloud_image_build.log | 17:13 |
| panda|ruck | weshay: https://review.rdoproject.org/r/18987 | 17:13 |
| sshnaidm | weshay, ? | 17:13 |
| sshnaidm | weshay, it should be in /logs/undercloud/var/log/extra/logstash.txt.gz | 17:14 |
| weshay | panda|ruck rfolco|rover https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/collect-logs/defaults/main.yml#L169 | 17:14 |
| weshay | sshnaidm thanks | 17:14 |
| weshay | rfolco|rover "CalledProcessError: Command '['disk-image-create'" | 17:15 |
| sshnaidm | weshay, https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/collect-logs/defaults/main.yml#L169 | 17:16 |
| weshay | sshnaidm rlandy https://review.rdoproject.org/r/#/c/18987/ | 17:16 |
| weshay | sshnaidm the ovb jobs send data to logstash too? | 17:21 |
| weshay | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22CalledProcessError%3A%20Command%20'%5B'disk-image-create%5C%22%3A%20%5C%22failed%5C%22%5C%22%20AND%20build_name%3A*tripleo-ci-*%20AND%20tags%3Aconsole%20AND%20voting%3A1%20AND%20build_status%3AFAILURE | 17:22 |
| *** dtantsur is now known as dtantsur|afk | 17:23 | |
| sshnaidm | weshay, hmm.. | 17:24 |
| sshnaidm | weshay, we have a different log server in rdo ci, need to ask | 17:25 |
| weshay | sshnaidm ya.. no biggie | 17:25 |
| weshay | sshnaidm if you can hang for 5 min, hopefully we'll have two pull requests coming | 17:26 |
| sshnaidm | weshay, sure | 17:26 |
| *** fultonj has joined #oooq | 17:27 | |
| *** fultonj has quit IRC | 17:27 | |
| *** fultonj has joined #oooq | 17:28 | |
| panda|ruck | weshay: https://github.com/sshnaidm/sova/pull/57 | 17:30 |
| panda|ruck | sshnaidm: ^ | 17:31 |
| weshay | sshnaidm hold up.. 1 correction | 17:32 |
| *** chandankumar is now known as raukadah | 17:32 | |
| weshay | sshnaidm ok.. this looks ok to me now .. https://github.com/sshnaidm/sova/pull/57 | 17:35 |
| sshnaidm | panda|ruck, commented on https://review.rdoproject.org/r/#/c/18987/ | 17:37 |
| panda|ruck | weshay: sshnaidm thanks https://review.rdoproject.org/r/18987 updated | 17:38 |
| sshnaidm | panda|ruck, +w | 17:43 |
| sshnaidm | panda|ruck, wrt https://github.com/sshnaidm/sova/pull/57 - please do the same for promotion branch | 17:44 |
| derekh | sshnaidm: sorry missed this earlier, did you get an answer? <sshnaidm> quiquell, maybe dtantsur or derekh have ideas, if it's possible to make ironic work with nodepool anyhow | 17:45 |
| sshnaidm | derekh, yeah, we talked with dtantsur|afk | 17:45 |
| derekh | sshnaidm: ack, sorry for the delay... | 17:46 |
| sshnaidm | derekh, np | 17:46 |
| panda|ruck | sshnaidm: ok | 17:49 |
| weshay | sshnaidm http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2019-02-25.log.html#t2019-02-25T16:07:14 | 17:50 |
| *** holser_ has quit IRC | 17:50 | |
| weshay | sshnaidm https://review.openstack.org/#/c/639165/2 | 17:50 |
| weshay | sshnaidm ok.. last thing https://github.com/sshnaidm/sova/pull/58#pullrequestreview-207541877 | 17:54 |
| weshay | lgtm | 17:54 |
| *** holser_ has joined #oooq | 17:55 | |
| sshnaidm | weshay, wait | 17:59 |
| weshay | sshnaidm aye what's up | 17:59 |
| sshnaidm | weshay, the discussion there is about errors.txt, not logstash.txt, right? | 17:59 |
| sshnaidm | weshay, then these logstash settings are not related | 18:00 |
| *** derekh has quit IRC | 18:01 | |
| sshnaidm | weshay, need to add here check for max size: https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/collect-logs/tasks/collect.yml#L331 | 18:02 |
| weshay | panda|ruck https://review.openstack.org/#/c/638438/ | 18:02 |
| sshnaidm | what the heck is going on in these jobs to have half giga errors.. | 18:02 |
| weshay | sshnaidm /me looks | 18:03 |
| weshay | sshnaidm +1 on a check for size | 18:03 |
| weshay | what would be the max size? | 18:03 |
| *** trown is now known as trown|lunch | 18:04 | |
| panda|ruck | sshnaidm: for promo https://github.com/sshnaidm/sova/pull/59 | 18:08 |
| *** jpena is now known as jpena|off | 18:11 | |
| sshnaidm | weshay, I think the problem is with upgrade jobs only, because seems like we run collect logs twice - before and after upgrade, need to check it.. | 18:27 |
| sshnaidm | weshay, in result we collect errors from "errors.txt" file itself and have a lot of duplicate logs in containers, something catastrophic is going on there | 18:28 |
| sshnaidm | weshay, did something change there recently? seems like it didn't work that way before | 18:28 |
| weshay | aye I agree | 18:28 |
| weshay | sshnaidm we can revert my patch if needed | 18:28 |
| weshay | ur's hasn't landed yet | 18:28 |
| sshnaidm | weshay, which patch? | 18:28 |
| weshay | ur patch to fix collect log.. from running twice | 18:29 |
| sshnaidm | weshay, my patch is not related to this I think | 18:29 |
| sshnaidm | weshay, or maybe yes..hmm | 18:29 |
| weshay | so in the short term.. land ur patch.. + mine to remove upgrade/update from logstash.. reevaluate from there | 18:30 |
| weshay | that's what I'm thinking anyway | 18:30 |
| sshnaidm | weshay, well, my patch is actually preventing from collect logs to run twice | 18:30 |
| weshay | heh.. ok.. needs to merge though | 18:31 |
| weshay | let's sync tomorrow on it? | 18:31 |
| sshnaidm | weshay, but I think in upgrade playbooks we run collect logs twice anyway, let me check this | 18:31 |
| weshay | oh.. | 18:31 |
| weshay | fak | 18:31 |
| *** amoralej is now known as amoralej|off | 18:33 | |
| weshay | rlandy is possible to download the clouds.yaml from your tenant | 18:42 |
| weshay | or is that something we always have to recreate | 18:43 |
| rlandy | weshay: you create it | 18:43 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci- (2 more messages) | 19:03 |
| weshay | rfolco|rover https://review.rdoproject.org/r/#/c/18964/ | 19:03 |
| *** sshnaidm is now known as sshnaidm|afk | 19:11 | |
| *** yolanda has joined #oooq | 19:15 | |
| *** trown|lunch is now known as trown | 19:19 | |
| rlandy | rfolco|rover: weshay: should we w+1 on https://review.openstack.org/#/c/637212/? | 19:22 |
| *** brault has joined #oooq | 19:33 | |
| weshay | rlandy can you join my blue? | 19:56 |
| weshay | we can chat bm in a few | 19:57 |
| rlandy | weshay: ack | 19:57 |
| *** brault has quit IRC | 20:24 | |
| rlandy | sshnaidm|afk: do you still have the command sto import the images into your tenant? | 20:34 |
| sshnaidm|afk | rlandy, you mean to accept or to share? | 20:37 |
| rlandy | sshnaidm|afk: to accept | 20:37 |
| rlandy | sshnaidm|afk: rfolco|rover needs them | 20:37 |
| rlandy | I forgot those aren't just available | 20:37 |
| sshnaidm|afk | rlandy, openstack --os-cloud rdo-cloud image set --accept ${IMAGE_ID} | 20:37 |
| rlandy | sshnaidm|afk: cool - thanks | 20:38 |
| rlandy | rfolco|rover: weshay: ^^ | 20:38 |
| rlandy | getting image ids, sec | 20:38 |
| *** jtomasek has quit IRC | 20:39 | |
| *** ccamacho has quit IRC | 20:39 | |
| *** jtomasek has joined #oooq | 20:40 | |
| rlandy | | 8a2bfe94-fc8d-4d5e-a7df-15b33c2d1dfe | openstack-infra-centos-7 | active | | 20:41 |
| rlandy | | 66087b30-07d0-413c-be83-56433379ffa8 | openstack-infra-fedora-28 | active | | 20:41 |
| rlandy | sorry | 20:41 |
| rlandy | | 6a6d23d7-65e7-43ea-9307-71b2e17d2ead | upstream-cloudinit-centos-7 | active | | 20:41 |
| rlandy | | 88006cd1-d089-4bd3-b70a-8cbc7eb32f63 | upstream-cloudinit-fedora-28 | active | | 20:41 |
| *** jtomasek has quit IRC | 20:42 | |
| *** jtomasek has joined #oooq | 20:43 | |
| weshay | sshnaidm|afk are the images shared w/ the team? | 20:46 |
| sshnaidm|afk | weshay, should be | 20:46 |
| rlandy | rfolco|rover can't accept | 20:46 |
| weshay | sshnaidm|afk sorry to hit you up so late.. it works for me .. but not rfolco | 20:46 |
| rfolco|rover | I'm a picky person | 20:47 |
| sshnaidm|afk | what's his tenant id? | 20:47 |
| rfolco|rover | rfolco | 20:47 |
| sshnaidm|afk | rfolco|rover, ID, not name | 20:49 |
| weshay | sshnaidm|afk export OS_TENANT_ID=7314597d33d44a34873b50a738710b07 | 20:49 |
| rlandy | TENANT_ID=7314597d33d44a34873b50a738710b07 | 20:49 |
| sshnaidm|afk | rfolco|rover, openstack --os-cloud rdo-cloud image set --accept 6a6d23d7-65e7-43ea-9307-71b2e17d2ead | 20:51 |
| sshnaidm|afk | rfolco|rover, openstack --os-cloud rdo-cloud image set --accept 88006cd1-d089-4bd3-b70a-8cbc7eb32f63 | 20:51 |
| rfolco|rover | sshnaidm|afk, what did change? the same command did not work before :) | 20:52 |
| sshnaidm|afk | ¯\_(ツ)_/¯ | 20:53 |
| *** agopi has quit IRC | 20:58 | |
| *** jrist has quit IRC | 20:59 | |
| *** agopi has joined #oooq | 20:59 | |
| *** weshay has quit IRC | 20:59 | |
| *** weshay has joined #oooq | 21:00 | |
| *** jrist has joined #oooq | 21:03 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci- (2 more messages) | 21:03 |
| *** agopi_ has joined #oooq | 21:07 | |
| *** jrist has quit IRC | 21:09 | |
| *** jpena|off has quit IRC | 21:09 | |
| *** jtomasek has quit IRC | 21:09 | |
| *** jpena|off has joined #oooq | 21:10 | |
| *** agopi has quit IRC | 21:10 | |
| *** jrist has joined #oooq | 21:10 | |
| *** agopi__ has joined #oooq | 21:13 | |
| *** agopi_ has quit IRC | 21:16 | |
| *** agopi__ is now known as agopi | 21:16 | |
| *** trown is now known as trown|outtypewww | 22:00 | |
| *** agopi has quit IRC | 22:00 | |
| *** holser_ has quit IRC | 22:06 | |
| *** agopi has joined #oooq | 22:19 | |
| weshay | rlandy rfolco|rover fyi.. a pass at the docs again https://review.openstack.org/639208 | 22:21 |
| weshay | https://review.rdoproject.org/r/#/c/18990/ updated requirements | 22:22 |
| *** sshnaidm|afk has quit IRC | 22:24 | |
| * rlandy looks | 22:25 | |
| weshay | rlandy rfolco|rover add the docker group if missing https://review.openstack.org/639212 | 22:35 |
| rlandy | weshay: pls see question about removing from clouda.yaml https://review.openstack.org/#/c/639208 | 22:35 |
| rlandy | clouds.yaml | 22:35 |
| * weshay looks | 22:35 | |
| weshay | rlandy I don't think you need those other bits | 22:36 |
| weshay | oh shoot | 22:36 |
| * rlandy is not sure | 22:36 | |
| weshay | values network | 22:36 |
| weshay | hrm | 22:36 |
| rlandy | weshay: also one comment in https://review.openstack.org/#/c/639212/1 | 22:37 |
| weshay | rlandy ya.. you don't need it | 22:37 |
| weshay | rlandy although I'll add it back | 22:37 |
| *** sshnaidm has joined #oooq | 22:39 | |
| weshay | rlandy k.. thanks for the comments.. updated | 22:41 |
| * rlandy looks | 22:42 | |
| rlandy | revoted | 22:43 |
| rlandy | weshay: responded to https://review.openstack.org/#/c/639212/ but it's not a major deal | 22:45 |
| rlandy | should still work | 22:45 |
| rlandy | whatever I +2'ed it because it won't break | 22:47 |
| rlandy | logically, I don't see what other case there is but it's fine | 22:47 |
| weshay | let me think overnight if I can damage a box by adding the group | 22:47 |
| weshay | rlandy can you vote on this for rfolco|rover https://review.rdoproject.org/r/#/c/18956/ | 22:48 |
| weshay | rlandy the job worked w/ his patch https://logs.rdoproject.org/56/18956/5/check/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-rocky/7dd67dd/ | 22:49 |
| weshay | rfolco++ | 22:49 |
| rlandy | rfolco|rover: weshay: question on review https://review.rdoproject.org/r/#/c/18956/ | 22:52 |
| rlandy | rfolco|rover: weshay: afaict from other jobs, branch_override stays | 22:53 |
| * weshay checks to see what's up w/ the queens job | 22:58 | |
| weshay | rlandy thanks for the review | 23:00 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci- (2 more messages) | 23:03 |
| *** vinaykns has quit IRC | 23:54 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!