*** rascasoft has quit IRC | 00:02 | |
*** tosky has quit IRC | 00:46 | |
hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-undercloud-containers, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ (3 more messages) | 00:48 |
---|---|---|
hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-undercloud-containers, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ (3 more messages) | 02:48 |
*** apetrich has quit IRC | 03:15 | |
*** saneax has joined #oooq | 03:19 | |
*** gkadam-bmgr has joined #oooq | 03:34 | |
*** skramaja has joined #oooq | 03:35 | |
*** rlandy|bbl is now known as rlandy | 03:49 | |
*** ykarel|away has joined #oooq | 03:54 | |
*** udesale has joined #oooq | 03:55 | |
*** udesale has quit IRC | 03:56 | |
*** udesale has joined #oooq | 04:28 | |
*** ykarel|away is now known as ykarel | 04:34 | |
hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-undercloud-containers, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ (3 more messages) | 04:48 |
*** ykarel has quit IRC | 05:59 | |
*** chandankumar is now known as chkumar|ruck | 06:14 | |
*** ykarel has joined #oooq | 06:16 | |
quiquell|off | chkumar|ruck: o/ | 06:17 |
chkumar|ruck | quiquell|off: \o/ | 06:17 |
chkumar|ruck | quiquell|off: want to ruck or rover? | 06:17 |
quiquell|off | I can Rover but will ruck too | 06:18 |
chkumar|ruck | quiquell|off: http://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-undercloud-containers is in pretty bad state | 06:19 |
chkumar|ruck | already an alert posted | 06:19 |
quiquell|off | Have to leave now going back in a few | 06:19 |
chkumar|ruck | sure | 06:19 |
chkumar|ruck | ssbarnea: arxcruz let me know when you are back on irc, we have do a handover | 06:20 |
*** quiquell|off is now known as quiquell_rover | 06:20 | |
hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-undercloud-containers, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ (3 more messages) | 06:48 |
quiquell_rover | chkumar|ruck: I see a lot of noop failures | 06:53 |
quiquell_rover | chkumar|ruck: also a lot of timeout | 06:54 |
chkumar|ruck | quiquell_rover: I have not went over noop failure | 06:55 |
chkumar|ruck | quiquell_rover: you start with noop failures, I will go over promotion jobs | 06:55 |
quiquell_rover | chkumar|ruck: ack | 06:57 |
*** quiquell_rover is now known as quiquell|rover | 07:07 | |
*** quiquell|rover has quit IRC | 07:07 | |
*** quiquell has joined #oooq | 07:07 | |
quiquell | chkumar|ruck: do we have the etherpad ? | 07:08 |
*** quiquell is now known as quiquell|rover | 07:09 | |
*** jfrancoa has joined #oooq | 07:09 | |
chkumar|ruck | quiquell|rover: nope | 07:10 |
chkumar|ruck | quiquell|rover: feel free to create one | 07:10 |
quiquell|rover | chkumar|ruck: I mean the previous one | 07:11 |
quiquell|rover | chkumar|ruck: Want you to try the new reproducer ? | 07:13 |
quiquell|rover | chkumar|ruck: I am going to use it for the undercloud job stuff | 07:13 |
*** ccamacho has joined #oooq | 07:15 | |
chkumar|ruck | quiquell|rover: for this one https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-fedora-28-master-containers-build/12515ad/logs/undercloud/home/zuul/undercloud_install.log.txt.gz#_2019-01-31_05_22_44 | 07:16 |
chkumar|ruck | quiquell|rover: https://review.openstack.org/#/c/634109/ | 07:16 |
chkumar|ruck | quiquell|rover: yes I want to use the new reproducer | 07:17 |
quiquell|rover | sshnaidm|afk: what do we doo with https://review.openstack.org/#/c/633444/ ? | 07:21 |
quiquell|rover | marios: ^ ? | 07:21 |
marios | quiquell|rover: o/ | 07:21 |
marios | quiquell|rover: looking https://review.openstack.org/#/c/633444/ ? | 07:21 |
marios | quiquell|rover: is virt-resize log already collected i asked yesterday | 07:22 |
marios | quiquell|rover: is it this? https://github.com/openstack-infra/tripleo-ci/blob/44a129efdaec1fd05acf5d91ecae6cafab7989b8/toci-quickstart/config/collect-logs.yml#L104 | 07:25 |
marios | no var/log/libvirt probably | 07:25 |
marios | but ok so its likely there then | 07:25 |
quiquell|rover | marios: reproducer is not using toci | 07:25 |
marios | quiquell|rover: so then | 07:25 |
quiquell|rover | marios: that's why we collect it again there | 07:25 |
marios | where/how please | 07:25 |
quiquell|rover | http://logs.rdoproject.org/58/18558/28/check/tripleo-ci-reproducer-fedora-28-libvirt/fc70dcc/tripleo-ci-reproducer/virt-customize.log | 07:26 |
marios | quiquell|rover: oh you mean you just store them in the root | 07:27 |
marios | my question was _where_ are those logs are they in the /logs directory | 07:27 |
marios | quiquell|rover: so the answer is we don't use collect logs we just write them to the root logs dir | 07:28 |
quiquell|rover | marios: yep since we generate different stuff | 07:28 |
quiquell|rover | marios: This is the reproducer role change to Depends-On to the this https://review.rdoproject.org/r/#/c/18558 | 07:28 |
marios | wasup with undercloud_containers we have bug? | 07:30 |
marios | timeout | 07:30 |
quiquell|rover | marios: yep looking inito it | 07:30 |
quiquell|rover | Something related with puppet | 07:30 |
marios | quiquell|rover: ack so holding +A | 07:30 |
*** kopecmartin|off is now known as kopecmartin | 07:30 | |
quiquell|rover | marios: ack | 07:30 |
quiquell|rover | have to drop for a few | 07:31 |
quiquell|rover | Sorry was testing the away stuff from weechat | 07:32 |
quiquell|rover | marios: chan you send me a message to see if weechat answer it ? | 07:33 |
marios | quiquell|rover: no | 07:34 |
marios | (you mean pvt or here?) | 07:34 |
marios | quiquell|rover: ^ | 07:34 |
marios | quiquell|rover: tell | 07:34 |
marios | quiquell|rover: me | 07:34 |
marios | quiquell|rover: when | 07:35 |
marios | quiquell|rover: to | 07:35 |
marios | quiquell|rover: s | 07:35 |
marios | quiquell|rover: t | 07:35 |
marios | quiquell|rover: o | 07:35 |
marios | quiquell|rover: p | 07:35 |
marios | quiquell|rover: . | 07:35 |
chkumar|ruck | quiquell|rover: |o| -> message received by weechat | 07:35 |
quiquell|rover | Humm weechat not answering that's no good | 07:36 |
quiquell|rover | ok now I have to really drop for a few | 07:36 |
*** holser_ has joined #oooq | 07:39 | |
ykarel | quiquell|rover, chkumar|ruck there is some issue in promotions | 07:44 |
ykarel | /var/lib/docker/tmp/GetImageBlob143622093: no space left on device"} | 07:44 |
ykarel | failed: [localhost] (item=zaqar) => {"changed": false, "item": "zaqar", "msg": "Error pulling trunk.registry.rdoproject.org/tripleorocky/centos-binary-zaqar - code: None message: open /var/lib/docker/tmp/GetImageBlob868334417: no space left on device"} | 07:44 |
ykarel | can u check, master is not promoting since yesterday | 07:44 |
ykarel | seen ^^ in http://38.145.34.55/rocky.log, you can check other releases too | 07:45 |
*** quiquell|rover is now known as quique|rover|brb | 07:47 | |
marios | wdyt folks moving ci into new section https://review.openstack.org/#/c/633771 http://logs.openstack.org/71/633771/2/check/openstack-tox-docs/266ee8b/html/ | 07:52 |
marios | quique|rover|brb: ssbarnea arxcruz panda|off weshay ^ | 07:53 |
quique|rover|brb | marios: I am ok with it will check | 07:54 |
* quique|rover|brb is away: brb | 07:55 | |
marios | quique|rover|brb: thanks. no new content just rearrange index | 07:55 |
*** quique|rover|brb is now known as quiquell|rover | 07:55 | |
quiquell|rover | marios: send me a message now | 07:55 |
marios | quiquell|rover: no | 07:56 |
marios | quiquell|rover: i | 07:56 |
marios | quiquell|rover: will | 07:56 |
marios | quiquell|rover: not | 07:56 |
marios | quiquell|rover: never! | 07:56 |
quiquell|rover | Argg ok | 07:56 |
*** quiquell|rover is now known as quique|rover|brb | 07:56 | |
quique|rover|brb | Go back in a few | 07:56 |
*** quique|rover|brb is now known as quique|rover|bbl | 07:56 | |
* quique|rover|bbl is back (gone 00:04:03) | 07:59 | |
* quique|rover|bbl is away: bbl | 07:59 | |
*** rascasoft has joined #oooq | 08:01 | |
*** udesale has quit IRC | 08:14 | |
*** udesale has joined #oooq | 08:15 | |
*** quique|rover|bbl is now known as quiquell|rover | 08:16 | |
* quiquell|rover is back (gone 00:16:43) | 08:16 | |
*** apetrich has joined #oooq | 08:17 | |
chkumar|ruck | ykarel: checking | 08:19 |
*** ykarel is now known as ykarel|lunch | 08:21 | |
marios | quiquell|rover: am i wrong about the keys https://review.openstack.org/#/c/631067/40 | 08:23 |
marios | sshnaidm|afk: ^ | 08:23 |
quiquell|rover | chkumar|ruck: Running undercloud-containers locally don't have the timeouts it's doing tempest now | 08:26 |
quiquell|rover | chkumar|ruck: want to setup reproducer ? | 08:26 |
chkumar|ruck | quiquell|rover: yes | 08:27 |
chkumar|ruck | quiquell|rover: pass me the docs I will start | 08:27 |
quiquell|rover | chkumar|ruck: https://github.com/rdo-infra/ansible-role-tripleo-ci-reproducer/blob/master/README.md | 08:28 |
quiquell|rover | chkumar|ruck: clone the role | 08:28 |
quiquell|rover | marios: key stuff have just merge, so we can put that in the documentation | 08:29 |
quiquell|rover | chkumar|ruck: also rlandy is creating the reproducer script for it here | 08:31 |
quiquell|rover | chkumar|ruck: https://review.openstack.org/#/c/631067 | 08:31 |
quiquell|rover | chkumar|ruck: so you can run the reproducer script from the review for the affected job | 08:31 |
*** tosky has joined #oooq | 08:34 | |
marios | quiquell|rover: right but more than that, rlandy may need to make her role template configurable (is j2) to take into account the user key i want to use | 08:35 |
ykarel|lunch | quiquell|rover, undercloud-containers should have been fixed with:- https://review.openstack.org/#/c/633944/ | 08:35 |
ykarel|lunch | it looks good after that patch merged: http://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-undercloud-containers&branch=master | 08:35 |
quiquell|rover | ykarel|lunch: Yep working now at the zuul reproducer | 08:35 |
marios | quiquell|rover: here https://review.openstack.org/#/c/631067/40/roles/create-zuul-based-reproducer/templates/launcher-playbook.yaml.j2@83 | 08:35 |
quiquell|rover | ykarel|lunch: thanks | 08:36 |
quiquell|rover | ykarel|lunch: let's wait for results at noop change https://review.openstack.org/#/c/560445/ | 08:36 |
ykarel|lunch | it should be the same :) | 08:36 |
ykarel|lunch | quiquell|rover, can u check the promoter issue | 08:36 |
quiquell|rover | ykarel|lunch: we are into that | 08:37 |
ykarel|lunch | okk, thanks | 08:37 |
* ykarel|lunch going for lunch now | 08:37 | |
quiquell|rover | marios: yep that's what I mean | 08:37 |
quiquell|rover | marios: adding --key stuff to bash jinja template and pass the vars to the role | 08:38 |
quiquell|rover | marios: at least the flexibiligy is in the role don't know what they want there | 08:38 |
marios | quiquell|rover: k | 08:39 |
*** jpena|off is now known as jpena | 08:46 | |
*** bogdando has joined #oooq | 08:46 | |
hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-undercloud-containers, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ (3 more messages) | 08:48 |
*** weshay has quit IRC | 08:51 | |
*** dsneddon has quit IRC | 08:51 | |
*** dsneddon has joined #oooq | 08:55 | |
chkumar|ruck | quiquell|rover: I am putting custom script to install docker and create clouds.yaml file if openrc file is there for reproducer | 09:08 |
*** ykarel|lunch is now known as ykarel | 09:09 | |
quiquell|rover | chkumar|ruck: run pre.yaml from the role | 09:10 |
quiquell|rover | chkumar|ruck: it install docke and such | 09:10 |
chkumar|ruck | quiquell|rover: ok | 09:10 |
quiquell|rover | chkumar|ruck: openrc is not enough nodepool needs clouds.yaml | 09:10 |
quiquell|rover | chkumar|ruck: you can test rlandys stuff too | 09:10 |
quiquell|rover | chkumar|ruck: it do all the requirements | 09:10 |
*** dtantsur|afk is now known as dtantsur | 09:21 | |
arxcruz | quiquell|rover: chkumar|ruck let me know if you guys want to sync | 09:22 |
quiquell|rover | arxcruz: where is the old etherpad ? | 09:23 |
arxcruz | quiquell|rover: https://review.rdoproject.org/etherpad/p/ruckrover-sprint24 | 09:23 |
*** sshnaidm|afk is now known as sshnaidm | 09:25 | |
chkumar|ruck | quiquell|rover: new one created? | 09:31 |
quiquell|rover | chkumar|ruck: #tripleo | 09:31 |
chkumar|ruck | panda|off: arxcruz with respect to this bug https://bugs.launchpad.net/tripleo/+bug/1813913 -> https://review.openstack.org/634109 | 09:42 |
openstack | Launchpad bug 1813913 in tripleo "periodic-tripleo-fedora-28-master-containers-build failing with unauthorized access to rdo registry" [Wishlist,Triaged] - Assigned to Gabriele Cerami (gcerami) | 09:42 |
arxcruz | panda|off: just let it go | 09:42 |
arxcruz | chkumar|ruck: just let it go :D | 09:43 |
arxcruz | chkumar|ruck: panda|off is handling it, and it's nothing we should care right now | 09:43 |
chkumar|ruck | arxcruz: ack | 09:44 |
panda|off | chkumar|ruck: I downgraded it to wishlist, I hoped it was enough of a signal. We'll be working on this in this sprint | 09:46 |
*** panda|off is now known as panda | 09:46 | |
*** derekh has joined #oooq | 09:48 | |
chkumar|ruck | quiquell|rover: for ruck rover handover sending an invite to meet in another 30 mins? | 09:52 |
arxcruz | chkumar|ruck: which bj ? | 09:58 |
chkumar|ruck | arxcruz: will pass the bj link here | 09:59 |
quiquell|rover | chkumar|ruck: 30 sure, you send it ? | 09:59 |
chkumar|ruck | quiquell|rover: invite sent | 10:00 |
quiquell|rover | chkumar|ruck: ack | 10:02 |
quiquell|rover | chkumar|ruck: going to add stuff to grafana to show disk space at promoter | 10:03 |
chkumar|ruck | quiquell|rover: undercloud timedout gone | 10:03 |
quiquell|rover | chkumar|ruck: cool | 10:04 |
quiquell|rover | chkumar|ruck: can you take a look at the stuff I have copied to the etherpad ? | 10:05 |
quiquell|rover | arxcruz: can you complete the new etherpad with pending stuff ? | 10:05 |
chkumar|ruck | quiquell|rover: both bugs updated | 10:05 |
arxcruz | quiquell|rover: which new etherpad ? | 10:05 |
quiquell|rover | ykarel, chkumar|ruck I am going to stop promoter | 10:05 |
chkumar|ruck | arxcruz: https://review.rdoproject.org/etherpad/p/ruckrover-unisprint5 | 10:06 |
arxcruz | quiquell|rover: chkumar|ruck done | 10:12 |
quiquell|rover | arxcruz: thanks | 10:13 |
sshnaidm | ssbarnea, before setting -1 you'd better to read a bug in commit message and to ask submitter if you have questions | 10:14 |
quiquell|rover | ssbarnea: we don't want to introduce big refactors there https://review.openstack.org/#/c/633444/ | 10:15 |
ssbarnea | sshnaidm: i read the bug and it did not contain any clear reasoning. | 10:15 |
quiquell|rover | ssbarnea: We want to conditionally set force_tcg | 10:16 |
sshnaidm | ssbarnea, then need to read again | 10:16 |
quiquell|rover | ssbarnea: with ansible and environment is not easy that's why I used a template | 10:16 |
ssbarnea | quiquell|rover: what prevents you from using environment: ... to do that? | 10:16 |
quiquell|rover | ssbarnea: Try to set conditionally at environment var at ansible | 10:16 |
sshnaidm | quiquell|rover, there is a lot of "&&" now in the patch | 10:16 |
quiquell|rover | sshnaidm: then you tell me omit is not working there | 10:16 |
quiquell|rover | ssbarnea: ^ | 10:17 |
sshnaidm | ssbarnea, no way in ansible to omit environment variable | 10:17 |
quiquell|rover | sshnaidm: you mean too much ? | 10:17 |
ssbarnea | quiquell|rover: the var does not accept a false value? | 10:17 |
quiquell|rover | ssbarnea: no | 10:17 |
quiquell|rover | ssbarnea: so please ask before -1 | 10:17 |
quiquell|rover | ssbarnea: people is not dumb or stupid by default | 10:17 |
sshnaidm | quiquell|rover, yeah :) seems like replace didn't work much | 10:17 |
ssbarnea | quiquell|rover: you can still do "export inside the shell script". and avoid creating temporary sh files that do this. there are still alternatives. | 10:18 |
quiquell|rover | ssbarnea: like that better but didn't want to repeat the ENV vars agian and again | 10:18 |
quiquell|rover | ssbarnea: so better put them at one file and use it at all scripts | 10:18 |
quiquell|rover | ssbarnea: also reuse the conditional block | 10:19 |
ssbarnea | quiquell|rover: you could still use jinja to avid repeading yourself: {{ setup_vars }} and define them in a single place. | 10:19 |
quiquell|rover | ssbarnea: then you complate the shell: foobar stuff | 10:19 |
quiquell|rover | s/complete/complicate/ | 10:19 |
sshnaidm | ssbarnea, no, duplication of exports in the shell is completely antipattern, external script is better solution | 10:19 |
quiquell|rover | ssbarnea: ^ | 10:20 |
chkumar|ruck | quiquell|rover: there are 4 jobs got time out in master promotion due to undercloud install testing here https://review.rdoproject.org/r/#/c/13943/ | 10:20 |
quiquell|rover | ssbarnea: Man is quite clear | 10:20 |
ssbarnea | sshnaidm: ok, in this case make the entire resize logic a single shell script. that would be nice and clean and easy to test/debug. | 10:20 |
sshnaidm | quiquell|rover, commented | 10:20 |
quiquell|rover | ssbarnea: is not only resize | 10:20 |
sshnaidm | ssbarnea, no, it won't be visible | 10:20 |
quiquell|rover | ssbarnea: is all all little snippets with virt-resize and virt-customize there is no common between them | 10:21 |
quiquell|rover | sshnaidm: ack | 10:21 |
sshnaidm | ssbarnea, and making a thousand scripts doesn't make sense | 10:21 |
ssbarnea | quiquell|rover: are these tasks needing shared-vars inside the same file/role? do they run on different conditions? if not really they are likely to be easier to implement in a single shell script. | 10:22 |
quiquell|rover | ssbarnea: I am not going to put libvirt_nodepool.yaml into a bash script | 10:23 |
quiquell|rover | ssbarnea: that would be anohter task | 10:23 |
quiquell|rover | ssbarnea: maybe you are right | 10:23 |
quiquell|rover | ssbarnea: But I am nog going to do it | 10:23 |
quiquell|rover | ssbarnea: there is a lot of stuff to refactor at libvirt role, so are going to find a lot of fun there | 10:24 |
ssbarnea | well, there are exact 10 tasks there, with only one condition on once, to me looks like amost perfect candidate to write as single bash, faster. i would do it if others like the ideas. | 10:24 |
quiquell|rover | ssbarnea: be prepare | 10:24 |
panda | what reviews are you discussing ? https://review.openstack.org/633444 ? | 10:25 |
quiquell|rover | ssbarnea: well is a mix of bash and python, but now we have libvirt CI so you can propose a patch | 10:26 |
quiquell|rover | panda: yep | 10:26 |
panda | only that one ? | 10:26 |
quiquell|rover | panda: yep | 10:27 |
ssbarnea | quiquell|rover: as I mentioned at the end of the message, feel free to ignore my opinion on that. i may be biased because I got some bad experiences with code going like this in infrared, where debugging an error on that area was a nightmare. as least I am glad that our team does not hate bash :D | 10:27 |
*** apetrich has quit IRC | 10:27 | |
chkumar|ruck | arxcruz: ssbarnea quiquell|rover https://redhat.bluejeans.com/1571313919/?src=meet_now ruck rover handover link bj | 10:28 |
quiquell|rover | ssbarnea: ack, sorry about my manners too, have to much at my brains now | 10:28 |
quiquell|rover | ssbarnea: want to merge last pieces of repro, pleas help me with that | 10:28 |
sshnaidm | quiquell|rover, commented on previous patch: https://review.openstack.org/#/c/633444/21/roles/libvirt/setup/overcloud/templates/libguestfs-env.sh.j2 | 10:29 |
chkumar|ruck | arxcruz: sshnaidm quiquell|rover syncup time | 10:30 |
panda | an alternative solution would be to move that template with the export to a default/main.yaml dict variable virt_environment: , then set ENVIRONMENT: {{ virt_environment }} | 10:30 |
sshnaidm | chkumar|ruck, you mean ssbarnea ? | 10:31 |
chkumar|ruck | sshnaidm: yes | 10:31 |
quiquell|rover | chkumar|ruck: going there | 10:31 |
quiquell|rover | panda: that is good | 10:31 |
quiquell|rover | chkumar|ruck: where is the blue | 10:32 |
chkumar|ruck | quiquell|rover: ssbarnea https://redhat.bluejeans.com/1571313919/?src=meet_now | 10:33 |
*** rnoriega has quit IRC | 10:35 | |
chkumar|ruck | ssbarnea: can you join? | 10:35 |
chkumar|ruck | ssbarnea: https://redhat.bluejeans.com/1571313919/?src=meet_now | 10:35 |
*** rnoriega has joined #oooq | 10:36 | |
arxcruz | https://review.rdoproject.org/etherpad/p/ruckrover-unisprint5 | 10:38 |
*** apetrich has joined #oooq | 10:41 | |
arxcruz | chkumar|ruck: http://dashboard-ci.tripleo.org/d/cEEjGFFmz/cockpit?orgId=1 | 10:45 |
hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-undercloud-containers, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo- (3 more messages) | 10:48 |
arxcruz | i whish our meetings be like this | 10:48 |
arxcruz | less than 15 minutes | 10:48 |
arxcruz | lol | 10:48 |
chkumar|ruck | sshnaidm: panda https://review.openstack.org/#/c/634109/ this one is good to go | 10:49 |
chkumar|ruck | do I need to fire a test job in rdo ci? | 10:49 |
quiquell|rover | chkumar|ruck: this is all f28-stanadlone jobs exercise that ? | 10:50 |
chkumar|ruck | quiquell|rover: yes | 10:50 |
quiquell|rover | chkumar|ruck: I don't see standalone scenarios there | 10:50 |
quiquell|rover | marios: Do you know why we don't have stdln scenarios here ? https://review.openstack.org/#/c/634109 | 10:51 |
sshnaidm | chkumar|ruck, commented | 10:52 |
quiquell|rover | sshnaidm: for python3 we need it to specify binary or it will be UTF | 10:52 |
chkumar|ruck | quiquell|rover: https://github.com/openstack-infra/tripleo-ci/blob/master/zuul.d/layout.yaml#L9 is missing from python-tripleoclcinet | 10:52 |
quiquell|rover | chkumar|ruck: can you add it to your review ? | 10:53 |
quiquell|rover | so we check ? | 10:53 |
quiquell|rover | ahh wait no | 10:53 |
quiquell|rover | we need f28 | 10:53 |
chkumar|ruck | yes | 10:53 |
quiquell|rover | f28 is not passing | 10:53 |
quiquell|rover | but deploy is all good ? | 10:54 |
quiquell|rover | have fail at tempest | 10:54 |
chkumar|ruck | let me check temepst stuff | 10:55 |
marios | quiquell|rover: not reall. i mean we should add at least for the tripleoclient/v1/tripleo_config/deploy/upgrade classes right | 10:56 |
marios | which are the standalone ones being run in ci | 10:56 |
marios | quiquell|rover: you doing or should i ? | 10:56 |
quiquell|rover | marios: I have mix stuff we just needed f28 | 10:57 |
marios | quiquell|rover: wut | 10:57 |
quiquell|rover | marios: but you can put a review for the layout of python-client and addd scnearios | 10:57 |
marios | quiquell|rover: ok will do | 10:57 |
quiquell|rover | marios: why only run stdl scenarios at deploy/upgrade ? | 10:57 |
chkumar|ruck | marios: is there scenario for f28? | 10:57 |
chkumar|ruck | *standalone scenario | 10:58 |
marios | quiquell|rover: cos thats where the standalone commands live in client | 10:58 |
marios | quiquell|rover: we can add whatever there but i think those are a good start | 10:58 |
marios | quiquell|rover: like the files starting 'tripleo in tripleoclient/v1 | 10:58 |
quiquell|rover | marios: we don't exercise this https://review.openstack.org/#/c/634109 with standalone ? | 10:58 |
marios | quiquell|rover: yeah we do | 10:59 |
marios | quiquell|rover: that is what i mean tripleo_deploy.opy | 10:59 |
marios | quiquell|rover: see here https://github.com/openstack/python-tripleoclient/tree/master/tripleoclient/v1 | 10:59 |
marios | quiquell|rover: all the client commands live in these files | 10:59 |
marios | quiquell|rover: the standalone ones are in the files that have 'tripleo' in the start | 10:59 |
quiquell|rover | marios: ok we are in the same page | 11:00 |
marios | quiquell|rover: cool | 11:00 |
ykarel | quiquell|rover, ack | 11:00 |
quiquell|rover | marios: so good that we add stld scenarios there | 11:00 |
quiquell|rover | marios: you put the review ? | 11:00 |
marios | quiquell|rover: yes good call. sure i can do that i thought we already established it. but if you prefer to go ahead | 11:00 |
marios | quiquell|rover: :D | 11:00 |
marios | man this is exhausting | 11:00 |
marios | ! | 11:00 |
quiquell|rover | marios: Let me put that, you have already did a lot of work there | 11:01 |
quiquell|rover | marios: thanks | 11:01 |
marios | quiquell|rover: ok no problem man just let me know if you don't get to it or need review | 11:01 |
marios | quiquell|rover: is pretty easy now we have the template | 11:01 |
marios | quiquell|rover: just change the standalonein tripleo-ci and include the template in client | 11:02 |
marios | done | 11:02 |
chkumar|ruck | sshnaidm: done | 11:03 |
*** rfolco has joined #oooq | 11:05 | |
chkumar|ruck | let me update the commit msg also for rb also | 11:06 |
ssbarnea | quiquell|rover: sshnaidm: we can easily merge dictionaries, see https://gist.github.com/ssbarnea/923bd01ad44662a73e69985e1aecaf77 | 11:20 |
*** udesale has quit IRC | 11:21 | |
ssbarnea | there are few other similar ways to achieve the same | 11:21 |
sshnaidm | ssbarnea, there are a few even more complicated and non obvious solutions, I'm sure | 11:24 |
ssbarnea | quiquell|rover: i even updated it to define environment at block level, so you would not have to redefine environment on each task! | 11:24 |
ssbarnea | environment is inhereted! | 11:24 |
ssbarnea | you can define it at task, block or playbook level. avoiding the need to repeat yourself. | 11:25 |
ssbarnea | ansible is quite nice, when you don't encounter a bug. | 11:25 |
sshnaidm | quiquell|rover, I see you've already deployed disk.conf on promoter? | 11:28 |
quiquell|rover | sshnaidm: but we can remove and wait for infra update | 11:34 |
sshnaidm | quiquell|rover, nm, just need to check it goes w/o errors | 11:34 |
quiquell|rover | ssbarnea: sshnaidm is in charge now of those reviews I am will do my ruck/rovering part now | 11:34 |
quiquell|rover | sshnaidm: ack now is good moment promotions are screw | 11:35 |
*** holser_ is now known as holser|burger | 11:46 | |
chkumar|ruck | quiquell|rover: I am looking at this one https://logs.rdoproject.org/00/627500/81/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-pike-branch/49a6d64/logs/undercloud/home/zuul/overcloud_validate.log.txt.gz#_2019-01-31_10_12_52 | 12:01 |
chkumar|ruck | sshnaidm: quiquell|rover panda can we merge this https://review.openstack.org/#/c/627500/ ? | 12:02 |
chkumar|ruck | rdo-pick failure unrelated to this patch | 12:02 |
quiquell|rover | chkumar|ruck: is software-factory review merged ? | 12:03 |
chkumar|ruck | quiquell|rover: yes yesterday | 12:03 |
chkumar|ruck | quiquell|rover: rdo-check jobs are running | 12:03 |
chkumar|ruck | all passed except pike one | 12:03 |
quiquell|rover | chkumar|ruck: pike one I am open LP | 12:04 |
quiquell|rover | chkumar|ruck: Can be related to cirros version | 12:04 |
chkumar|ruck | yes | 12:04 |
quiquell|rover | https://review.openstack.org/#/c/633214/ | 12:04 |
quiquell|rover | chkumar|ruck: https://bugs.launchpad.net/tripleo/+bug/1814090 | 12:05 |
openstack | Launchpad bug 1814090 in tripleo "Validate the overcloud failing at pike OVB" [High,Triaged] - Assigned to Quique Llorente (quiquell) | 12:05 |
sshnaidm | chkumar|ruck, quiquell|rover seems like we need to add ports cleanup to our ovb cleanup script | 12:18 |
chkumar|ruck | sshnaidm: I think i can add that | 12:19 |
sshnaidm | chkumar|ruck, we need to remove ports in DOWN, better comparing their age - more thatn 5 hours from creation date, like we do with stacks | 12:19 |
quiquell|rover | sshnaidm: ok make sense, weird it didn't affect us before is related to the new te-broker stuff ? | 12:20 |
sshnaidm | quiquell|rover, kind of, yeah | 12:20 |
sshnaidm | quiquell|rover, it took time for this ports to grow, also outages affect it | 12:20 |
quiquell|rover | sshnaidm: yep not best moment, chkumar|ruck you take care ? | 12:21 |
quiquell|rover | chkumar|ruck: will continue looking for issues | 12:21 |
chkumar|ruck | quiquell|rover: yes | 12:21 |
sshnaidm | chkumar|ruck, quiquell|rover I'm deleting them right now, to free some IPs | 12:21 |
quiquell|rover | sshnaidm: ack thanks | 12:22 |
chkumar|ruck | quiquell|rover: till then can we merge os_tempest last bits | 12:22 |
quiquell|rover | chkumar|ruck: yep pass me the review will do | 12:24 |
chkumar|ruck | quiquell|rover: https://review.openstack.org/627500 | 12:24 |
sshnaidm | chkumar|ruck, quiquell|rover wrt https://review.openstack.org/#/c/634213/1 - afaik cirros image presents on nodepool image already and we should take it from there, not to download | 12:25 |
sshnaidm | chkumar|ruck, quiquell|rover the url was mostly for reproducer or manual runs with quickstart.sh | 12:25 |
quiquell|rover | sshnaidm: looks it's use by validate overcloud at pike | 12:26 |
sshnaidm | can we check that we don't download it, but take from /opt/cache/files/ ? | 12:26 |
quiquell|rover | sshnaidm: http://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/tree/roles/validate-simple/templates/overcloud-validate.sh.j2 | 12:26 |
quiquell|rover | sshnaidm: so just ust it as a fallback if it's not at /opt/cache/files ? | 12:26 |
sshnaidm | quiquell|rover, yep | 12:27 |
quiquell|rover | sshnaidm: will do | 12:27 |
sshnaidm | quiquell|rover, and it should be there in CI | 12:27 |
*** jpena is now known as jpena|lunch | 12:27 | |
quiquell|rover | sshnaidm: fallback is already in place | 12:28 |
ssbarnea | sshnaidm: can we please merge https://review.rdoproject.org/r/#/c/18518/3 ? | 12:28 |
*** udesale has joined #oooq | 12:28 | |
sshnaidm | ssbarnea, looking | 12:28 |
sshnaidm | ssbarnea, +w | 12:30 |
ssbarnea | sshnaidm: thanks. | 12:30 |
quiquell|rover | sshnaidm: fallback already in place https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/validate-simple/templates/overcloud-validate.sh.j2#L64 | 12:31 |
quiquell|rover | ssbarnea: so cirros image is not in the nodepool image I suppose | 12:31 |
sshnaidm | quiquell|rover, maybe they changed the path? | 12:33 |
quiquell|rover | sshnaidm: going to reproduce | 12:33 |
sshnaidm | quiquell|rover, is it on ovb jobs only? | 12:34 |
quiquell|rover | damn is ovb | 12:34 |
quiquell|rover | well I just need to see the nodepool image | 12:34 |
quiquell|rover | sshnaidm: yep ovb | 12:34 |
quiquell|rover | sshnaidm: what images do we use for OVB ? | 12:34 |
sshnaidm | quiquell|rover, the same, from rdo | 12:34 |
quiquell|rover | sshnaidm: upstream-fedora-28 ? | 12:35 |
sshnaidm | quiquell|rover, do you have a log? | 12:35 |
quiquell|rover | centos-7 I mean | 12:35 |
sshnaidm | quiquell|rover, yeah, centos, but different label | 12:35 |
sshnaidm | I don't know why different label, it should be the same as for standalone/multinode | 12:35 |
quiquell|rover | sshnaidm: what label ?= | 12:36 |
sshnaidm | quiquell|rover, tripleo-ovb-centos-7 | 12:36 |
quiquell|rover | damn the CIX | 12:37 |
quiquell|rover | arxcruz: how is the CIX now ? daily or what ? | 12:37 |
arxcruz | quiquell|rover: mondays and wednesdays, it's on rhos team meetings calendar as CI Escalation Status | DFG:ALL | 12:38 |
quiquell|rover | arxcruz: nice thanks | 12:38 |
quiquell|rover | sshnaidm: I see cirros image at /opt/cache/files | 12:42 |
quiquell|rover | nodepool image | 12:42 |
sshnaidm | quiquell|rover, great | 12:42 |
sshnaidm | quiquell|rover, so, do you have a log from job? I'd like to see why it downloads.. | 12:42 |
sshnaidm | quiquell|rover, which version btw? | 12:43 |
quiquell|rover | sshnaidm: http://logs.rdoproject.org/45/560445/234/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-pike-branch/d250d6f/logs/undercloud/home/zuul/overcloud_validate.log.txt.gz | 12:43 |
quiquell|rover | sshnaidm: testing the bash snipet works fine | 12:44 |
quiquell|rover | sshnaidm: image have to be different | 12:44 |
sshnaidm | quiquell|rover, so it should be /opt/cache/files/cirros-0.3.6-x86_64-uec.tar.gz | 12:45 |
quiquell|rover | sshnaidm: yep | 12:46 |
quiquell|rover | upstream-infra-centos-7 have it | 12:46 |
quiquell|rover | sshnaidm: what the different with OVB images ? | 12:46 |
quiquell|rover | sshnaidm: this is the new OVB stuff ? ovb-manage-stack | 12:48 |
hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, (2 more messages) | 12:48 |
quiquell|rover | sshnaidm: we are using plain CentOS-7-x86_64-GenericCloud | 12:51 |
quiquell|rover | sshnaidm: for OVB nodes | 12:51 |
quiquell|rover | sshnaidm: http://logs.rdoproject.org/45/560445/234/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-pike-branch/d250d6f/logs/undercloud/home/zuul/workspace/ovb/env-99378-base.yaml.txt.gz | 12:51 |
quiquell|rover | sshnaidm: is that right ? | 12:51 |
sshnaidm | quiquell|rover, plain image is for overcloud nodes | 12:52 |
sshnaidm | quiquell|rover, for undercloud node it's nodepool image | 12:52 |
sshnaidm | quiquell|rover, it should be as regular nodepool image that we use in upstream | 12:52 |
quiquell|rover | sshnaidm: the validation is done to the overcloud at undercloud ahh ok | 12:53 |
quiquell|rover | sshnaidm: Can I reproduce that ? | 12:54 |
sshnaidm | quiquell|rover, I think so | 12:54 |
*** rlandy has joined #oooq | 12:54 | |
quiquell|rover | sshnaidm: I will need to put my cloud as secrets or something ? | 12:54 |
*** weshay has joined #oooq | 12:54 | |
sshnaidm | quiquell|rover, http://paste.openstack.org/show/744304/ | 12:55 |
sshnaidm | quiquell|rover, just copy all needed vars from tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001 and add cloud config | 12:56 |
quiquell|rover | sshnaidm: nodeset is tripleo-ovb-centos-7 | 12:56 |
weshay | quiquell|rover, chkumar|ruck https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-fedora-28-ovb-1ctlr_1comp-featureset002-master-upload%09 | 12:57 |
sshnaidm | quiquell|rover, yes | 12:57 |
weshay | if that job doesn't work.. we can't promote :) | 12:57 |
sshnaidm | quiquell|rover, nodeset is already set in rdo config | 12:57 |
weshay | brb | 12:57 |
quiquell|rover | sshnaidm: I mean what images poiunt too this nodset | 12:57 |
quiquell|rover | weshay: ack will look | 12:57 |
quiquell|rover | weshay: skipped means dependant fail | 12:58 |
quiquell|rover | weshay: upload job has not run at all | 12:58 |
sshnaidm | quiquell|rover, nodeset: tripleo-ovb-centos-7-primary | 12:58 |
*** zbr has joined #oooq | 12:58 | |
sshnaidm | quiquell|rover, actually it's the same label: https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/nodesets.yaml#L68-L83 | 13:00 |
quiquell|rover | sshnaidm: wher ewas the nodepool config for software factory ? | 13:01 |
sshnaidm | quiquell|rover, nodepool/rdo-cloud.yaml | 13:02 |
sshnaidm | labels: | 13:03 |
sshnaidm | - name: tripleo-ovb-centos-7 | 13:03 |
sshnaidm | min-ram: 8192 | 13:03 |
sshnaidm | flavor-name: 'ci.m1.nodepool' | 13:03 |
sshnaidm | diskimage: upstream-centos-7 | 13:03 |
*** jjoyce has quit IRC | 13:03 | |
sshnaidm | quiquell|rover, exactly as for multinode and others | 13:03 |
quiquell|rover | sshnaidm: wait I have just mount upstream-infra | 13:03 |
quiquell|rover | sshnaidm: let me test exactly that | 13:04 |
*** jjoyce has joined #oooq | 13:04 | |
weshay | quiquell|rover, chkumar|ruck sorry I linked the f28 job | 13:04 |
weshay | meant https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-master-upload%09 | 13:04 |
quiquell|rover | weshay: they have fix the timeout | 13:04 |
weshay | chkumar|ruck++ | 13:05 |
weshay | https://review.rdoproject.org/r/#/c/13943/ | 13:05 |
hubbot1 | weshay: chkumar|ruck's karma is now 5 | 13:05 |
quiquell|rover | weshay: last run is green so could be fixed now | 13:05 |
ssbarnea | sshnaidm: i am in process of trying a nick swap from ssbarnea to "zbr", to avoid confusions in our channels. | 13:05 |
chkumar|ruck | weshay: it is related to undercloud install timedout | 13:05 |
chkumar|ruck | weshay: fixed in the morning | 13:05 |
weshay | ah cool | 13:05 |
quiquell|rover | weshay: promoter without diskpace we have clean up but we have to monitor | 13:05 |
sshnaidm | ssbarnea, zbr? | 13:05 |
chkumar|ruck | weshay: we got retry limit due to max of ports down which sshnaidm just cleaned up | 13:06 |
marios | chkumar|ruck: o/ so do i understand correctly that the master skip list always wins for the tempest tests like this https://github.com/openstack/tripleo-quickstart-extras/blob/b15450d98698814652db1cd9c00da510a11ec184/roles/validate-tempest/vars/tempest_skip_master.yml#L56 even if i have the plugin added in my job like https://review.openstack.org/#/c/633919/2/zuul.d/standalone-jobs.yaml ? | 13:06 |
weshay | marios, ya.. | 13:06 |
weshay | skip list always wins | 13:06 |
marios | k | 13:06 |
marios | then we have to check | 13:06 |
marios | like e.g. barbican and the standalone job. the definition has it but the results don't" http://logs.openstack.org/98/604298/205/check/tripleo-ci-centos-7-standalone/9c9a45f/logs/undercloud/home/zuul/tempest.log.txt.gz https://github.com/openstack-infra/tripleo-ci/blob/44a129efdaec1fd05acf5d91ecae6cafab7989b8/zuul.d/standalone-jobs.yaml#L389 | 13:06 |
marios | cos its in the skip list | 13:06 |
marios | i came across it just now for designate added a depends on for the skip list to check https://review.openstack.org/#/c/633919/ (https://review.openstack.org/634223 ) | 13:07 |
ssbarnea | sshnaidm: yeah. (my family name sounds like this and we do not have any other nick starting with z, should be easy pick). | 13:07 |
weshay | arxcruz, since you are not ruck/rover anymore.. can you add a storyless task to update the skip lists | 13:07 |
panda | quiquell|rover: how did you clean up ? | 13:07 |
weshay | on sprint 5 | 13:07 |
quiquell|rover | panda: docker rmi | 13:08 |
ssbarnea | sshnaidm: i wonder what is the best way to let people know about such a change, as they know to look for "ssbarnea" (more worried about other channels), any idea? | 13:08 |
sshnaidm | ssbarnea, rename to zbr-ssbarnea | 13:09 |
marios | quiquell|rover: did you already post tripleo client standalone or should i | 13:09 |
panda | quiquell|rover: mmhh I see only 1 image and 27G in the docker dir. | 13:09 |
marios | quiquell|rover: or you planning on doing it/have time to do it | 13:09 |
*** zbr is now known as help | 13:09 | |
chkumar|ruck | quiquell|rover: weshay https://review.openstack.org/#/c/630925/ we need this one for manila tests failure | 13:09 |
*** help is now known as Guest57406 | 13:10 | |
arxcruz | weshay: ack | 13:10 |
*** Guest57406 is now known as zbr|ssbarnea | 13:10 | |
quiquell|rover | marios: damn I forgot man | 13:11 |
quiquell|rover | panda: humm maybe I miss the containers ? but we are not starting image up there | 13:11 |
quiquell|rover | panda: what can be those 27gb ? | 13:11 |
marios | wdyt folks moving ci into new section https://review.openstack.org/#/c/633771 http://logs.openstack.org/71/633771/3/check/openstack-tox-docs/0ea6f8e/html/ thanks. no new content just rearranges sections in the index and adds standalone ci index @ /ci/index.html | 13:11 |
quiquell|rover | panda: intermediate layers or somethig ? | 13:11 |
marios | quiquell|rover: ok should i do it or you still gona wana? | 13:11 |
panda | quiquell|rover: directmapping messing | 13:11 |
weshay | quiquell|rover, sshnaidm rlandy so.. the name resolution is not working completely on libvirt runs | 13:12 |
panda | quiquell|rover: and we have not time right now to test overlay2 storage | 13:12 |
marios | ssbarnea: did my update address your concern in ps://review.openstack.org/#/c/633771 please revote when you have time thanks | 13:12 |
sshnaidm | weshay, what do you have in /etc/resolv.conf? | 13:12 |
quiquell|rover | panda: if we are not bad with promotions maybe is not bad time to test overlay2 | 13:12 |
weshay | sshnaidm, so we can hit yum mirrors etc | 13:12 |
quiquell|rover | weshay: do we need promotions right now ? | 13:12 |
weshay | but ntp servers fail | 13:12 |
sshnaidm | weshay, external ntp servers are not accessible from rh network, btw | 13:13 |
sshnaidm | weshay, you need to use only rh ntp in rh net | 13:13 |
weshay | quiquell|rover, it's getting old, but we have a couple days until we're red | 13:13 |
weshay | sshnaidm, no shit | 13:13 |
sshnaidm | we had this problem with old reproducer too | 13:13 |
ssbarnea | marios: yes, thanks. voted. | 13:13 |
weshay | sshnaidm, I'm talking outside, we can't hit the public ntp servers | 13:13 |
sshnaidm | weshay, why? | 13:14 |
weshay | only on libvirt | 13:14 |
quiquell|rover | weshay: has passed mypending repro stuff to sshnaidm so I focus at ruck/rover | 13:14 |
weshay | quiquell|rover, k | 13:14 |
marios | ssbarnea: thanks | 13:14 |
quiquell|rover | panda: can we test overlay2 in two days ? | 13:14 |
marios | quiquell|rover: ok, so should i do it then ? :) /me firing up vim | 13:14 |
*** gkadam-bmgr is now known as gkadam | 13:15 | |
quiquell|rover | marios: Will do don't worry your merge it and look at it like the lord of standalone CI | 13:15 |
*** zbr|ssbarnea is now known as zbr|ruck | 13:16 | |
*** zbr|ruck is now known as zbr|rover | 13:16 | |
weshay | sshnaidm, have you run a full deployment w/ libvirt? | 13:16 |
panda | quiquell|rover: maybe less, but you need to attach another volume to the server, format it as overlay2, switch docker dir, and leave it there for a while. I can't certainly be worse than this situation, devicemapper performances are bad, and space is not freed correctly | 13:16 |
*** zbr|rover is now known as zbr|ssbarnea | 13:16 | |
quiquell|rover | panda: also put that stuff at infra code | 13:16 |
marios | quiquell|rover: ack /me puts down vim | 13:16 |
weshay | sshnaidm, maybe we need to inject more data into unbound? | 13:16 |
panda | quiquell|rover: yes, after you tested it ... | 13:16 |
quiquell|rover | panda: I think we have the volume already | 13:16 |
quiquell|rover | panda: I see diffent volume for docke rimages | 13:17 |
*** ssbarnea has quit IRC | 13:17 | |
rlandy | weshay: it's because we are defaulting to 8.8.8.8 | 13:17 |
weshay | https://review.openstack.org/#/c/630925/ | 13:17 |
weshay | rlandy, we're not | 13:17 |
panda | quiquell|rover: yes, that saved you this morning, from having to regreate the entire server | 13:17 |
*** zbr has joined #oooq | 13:17 | |
weshay | rlandy, it's 127.0.0.1 | 13:17 |
quiquell|rover | rlandy, weshay : Humm sshnaidm have a review for nameserver | 13:17 |
*** zbr|ssbarnea has quit IRC | 13:17 | |
weshay | it's unbound | 13:17 |
* weshay looks | 13:17 | |
panda | quiquell|rover: when happened before, there was no way to interact with the server and remove containers. | 13:17 |
sshnaidm | quiquell|rover, weshay yeah, for a year.. | 13:18 |
weshay | lol :) | 13:18 |
*** zbr is now known as zbr|ssbarnea | 13:18 | |
weshay | sshnaidm, point us at :) | 13:18 |
sshnaidm | weshay, https://review.openstack.org/#/c/565839/ | 13:18 |
*** zbr|ssbarnea has quit IRC | 13:19 | |
*** ykarel has quit IRC | 13:19 | |
quiquell|rover | sshnaidm: at least now we have libvirt CI | 13:19 |
*** holser|burger is now known as holser_ | 13:19 | |
*** ykarel has joined #oooq | 13:19 | |
quiquell|rover | marios: https://review.openstack.org/634225 | 13:19 |
sshnaidm | quiquell|rover, yeah, just need to stabilize it with force_tcg | 13:20 |
weshay | zuul@subnode-1 ~]$ cat /etc/resolv.conf | 13:20 |
weshay | ; generated by /usr/sbin/dhclient-script | 13:20 |
weshay | nameserver 127.0.0.1 | 13:20 |
weshay | [zuul@subnode-1 ~]$ | 13:20 |
quiquell|rover | sshnaidm: we are near, I think it just take lik 5 minutes or so | 13:20 |
weshay | quiquell|rover, kind of | 13:20 |
*** zbr|ssbarnea has joined #oooq | 13:20 | |
*** gkadam has quit IRC | 13:20 | |
sshnaidm | quiquell|rover, renewed a patch: https://review.openstack.org/#/c/633444/ | 13:20 |
*** agopi is now known as agopi|brb | 13:21 | |
*** zbr|ssbarnea has quit IRC | 13:21 | |
marios | quiquell|rover: ack also needs tripleo-ci update | 13:21 |
marios | quiquell|rover: for the files: | 13:21 |
weshay | can we configure unbound to forward? | 13:22 |
weshay | yup | 13:22 |
weshay | unbound sucks | 13:24 |
*** zbr|ssbarnea has joined #oooq | 13:25 | |
*** agopi|brb has quit IRC | 13:26 | |
weshay | sshnaidm, jump on my tmate | 13:26 |
quiquell|rover | marios: what files do we need for python-tripleoclient ? | 13:27 |
quiquell|rover | marios: maybe irrelevant-files ? | 13:28 |
*** jpena|lunch is now known as jpena | 13:31 | |
weshay | rlandy, is your libvirt deployment have 8.8.8.8? | 13:32 |
*** ykarel is now known as ykarel|away | 13:32 | |
zbr|ssbarnea | weshay: zbr|ssbarnea :) | 13:33 |
marios | quiquell|rover: tripleoclient/v1/tripleo_(deploy|config|upgrade) | 13:36 |
marios | and the heat launcher stuff | 13:36 |
quiquell|rover | marios: ack | 13:36 |
*** ykarel|away has quit IRC | 13:38 | |
marios | brb | 13:38 |
quiquell|rover | marios: https://review.openstack.org/634225 | 13:44 |
quiquell|rover | panda: going to test overlay2 | 13:45 |
quiquell|rover | panda: I am following this https://docs.docker.com/storage/storagedriver/overlayfs-driver/#configure-docker-with-the-overlay-or-overlay2-storage-driver | 13:50 |
marios | quiquell|rover: ack will check in bit | 13:51 |
panda | quiquell|rover: yep | 13:51 |
quiquell|rover | panda: do I have to format it ? | 13:51 |
panda | quiquell|rover: is the promopter script stopped ? | 13:51 |
panda | quiquell|rover: yes, xfs currently does not have the d_type flag set to true | 13:51 |
quiquell|rover | panda: yep | 13:52 |
quiquell|rover | panda: also docker is stoped | 13:52 |
chkumar|ruck | weshay: can we move it to done https://tree.taiga.io/project/tripleo-ci-board/us/554 ? | 13:52 |
chkumar|ruck | final patch is in gates | 13:52 |
chkumar|ruck | sshnaidm: one question, I am able to get a list of port having down status but to compare for 5hours will I look at port show updated_at time? | 13:53 |
quiquell|rover | panda: | 13:53 |
quiquell|rover | I see fstype=1 | 13:53 |
panda | chkumar|ruck: usually we move to done when it's merged | 13:53 |
quiquell|rover | panda: https://paste.fedoraproject.org/paste/DTw14~L4ukqEcMgZF-fEHw | 13:54 |
chkumar|ruck | panda: yes, once it merged, we will move it to done may be by tomorrow | 13:54 |
quiquell|rover | panda: checking this https://docs.docker.com/storage/storagedriver/overlayfs-driver/ | 13:55 |
sshnaidm | chkumar|ruck, seems like that | 13:56 |
panda | quiquell|rover: why am I blind ? | 13:56 |
panda | quiquell|rover: it is, then it's even easier | 13:56 |
panda | quiquell|rover: well | 13:56 |
panda | quiquell|rover: you'll have to reformat anyway | 13:56 |
quiquell|rover | I see fstype=1 | 13:56 |
chkumar|ruck | sshnaidm: ok then | 13:56 |
quiquell|rover | panda: I have to reformat | 13:57 |
panda | quiquell|rover: because the dangling free space, it's not enough to just rm -rf * | 13:57 |
panda | quiquell|rover: or at least was never enough for me | 13:57 |
panda | quiquell|rover: you can try to see if a rm -rf in /var/lib/docker frees space | 13:57 |
chkumar|ruck | quiquell|rover: there we are using docker to push the images na? can we switch to buildah directly? | 13:58 |
quiquell|rover | panda: wait we have same volume for / and docker | 13:58 |
quiquell|rover | I was wrong | 13:58 |
panda | chkumar|ruck: in the promoter server we are just pullin, tagging and pushing | 13:58 |
panda | quiquell|rover: no, it's a different volume | 13:58 |
panda | quiquell|rover: /dev/vdb1 | 13:58 |
quiquell|rover | vda I am blind now | 13:59 |
quiquell|rover | :-) | 13:59 |
panda | quiquell|rover: wait | 13:59 |
panda | quiquell|rover: the promoter was not stopped | 13:59 |
rlandy | weshay: ; generated by /usr/sbin/dhclient-script | 13:59 |
rlandy | nameserver 192.168.122.1 | 13:59 |
rlandy | nameserver 8.8.8.8 | 13:59 |
panda | quiquell|rover: master is promoting | 13:59 |
panda | quiquell|rover: ETOOMANYPROMOTIONS | 14:00 |
quiquell|rover | jarlll | 14:00 |
quiquell|rover | let me kill stuff | 14:00 |
panda | chkumar|ruck: and we can't switch the container build to buildah right now, not supported by kolla | 14:00 |
chkumar|ruck | panda: ok, got it :-) | 14:01 |
*** agopi|brb has joined #oooq | 14:01 | |
quiquell|rover | panda: ok now | 14:02 |
quiquell|rover | damn channels | 14:02 |
quiquell|rover | panda: mkfs.xfs -n fstype=1 -f /dev/vdb1 | 14:02 |
*** agopi|brb is now known as agopi | 14:02 | |
quiquell|rover | is that it ? | 14:02 |
panda | chkumar|ruck: but a partial refactor of the build process is in scope for this sprint | 14:02 |
panda | quiquell|rover: at this point I think fstype=1 is the default | 14:03 |
panda | quiquell|rover: I never had to specify it the first time | 14:03 |
rlandy | weshay: unbound is run in zuul itself | 14:03 |
rlandy | my run never gets there | 14:03 |
rlandy | after our setup | 14:03 |
weshay | quiquell|rover, chkumar|ruck http://rhos-release.virt.bos.redhat.com:3030/rhosp | 14:06 |
quiquell|rover | panda: Storage Driver: overlay2 <- let's start promoter and see what we have | 14:10 |
panda | \o/ | 14:10 |
panda | maybe now we can even promote two release at a time | 14:10 |
panda | devicemapper was making everything explode | 14:11 |
chkumar|ruck | weshay: let's hope for tomorrow having a master promotion | 14:13 |
*** ykarel|away has joined #oooq | 14:15 | |
*** ykarel|away is now known as ykarel | 14:15 | |
chkumar|ruck | quiquell|rover: I am logging out, see ya tomorrow | 14:15 |
*** chkumar|ruck is now known as chkumar|out | 14:15 | |
quiquell|rover | chkumar|out: see you | 14:15 |
quiquell|rover | weshay: damn my vpn is not working | 14:16 |
quiquell|rover | weshay: promoter up with overlay2 | 14:22 |
weshay | k cool | 14:24 |
quiquell|rover | weshay: periodic-standalone-featureset052 critearia also match f28 isn't it ? | 14:25 |
quiquell|rover | panda: ^? | 14:26 |
*** skramaja has quit IRC | 14:26 | |
*** vinaykns has joined #oooq | 14:27 | |
panda | quiquell|rover: it should be changed to the full name of the job | 14:31 |
panda | quiquell|rover: I think it's already changed | 14:31 |
panda | quiquell|rover: should be already changed | 14:31 |
quiquell|rover | panda: I know but question is is the f28 is part of the promotion | 14:31 |
quiquell|rover | panda: looks like | 14:31 |
panda | quiquell|rover: yes it is | 14:32 |
quiquell|rover | panda: ok | 14:32 |
*** zul has joined #oooq | 14:32 | |
quiquell|rover | panda: So we need chkumar|out fix for f28 ? | 14:32 |
quiquell|rover | panda: Unauthorized for url: https://trunk.registry.rdoproject.org/v2/tripleomasterF28/centos-binary-cron/manifests/tripleo-ci-testing | 14:33 |
panda | quiquell|rover: I know, we're going to fix it in this sprint | 14:33 |
zbr|ssbarnea | quiquell|rover: sshnaidm : https://review.rdoproject.org/r/#/c/18649/ when you can, tx. | 14:33 |
quiquell|rover | panda: so we cannot change that from the criteria | 14:34 |
*** dtrainor has quit IRC | 14:34 | |
quiquell|rover | panda: I think I am going to harcode something in the promoter | 14:34 |
quiquell|rover | panda: to skip f28 for now ? | 14:34 |
sshnaidm | zbr|ssbarnea, who does really care about jjb now?? | 14:35 |
zbr|ssbarnea | sshnaidm: i don't but i suspect there are few jobs already used there, right? | 14:36 |
sshnaidm | zbr|ssbarnea, it's really hard to find something more neglected and not important than jjb now | 14:38 |
quiquell|rover | panda: do you have the review about fixing criteria around ? | 14:38 |
zbr|ssbarnea | sshnaidm: this change is not about jjb, is about getting it out of tox-linters. | 14:38 |
*** dtrainor has joined #oooq | 14:38 | |
sshnaidm | zbr|ssbarnea, that changes everything | 14:39 |
zbr|ssbarnea | sshnaidm: I do not want to run jjb when I lint my changes to ci-config. i guess it remained there for historical reasons. | 14:39 |
zbr|ssbarnea | sshnaidm: if you tell me that jjb is no longer in use and we can remove all jjb stuff from ci-config, I will be happy to do that. | 14:42 |
quiquell|rover | sshnaidm, panda: overlay2 https://review.rdoproject.org/r/18654 | 14:44 |
hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, (2 more messages) | 14:48 |
*** saneax has quit IRC | 15:12 | |
chkumar|out | quiquell|rover: weshay https://logs.rdoproject.org/43/13943/73/check/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-master-upload/9b8c324/logs/undercloud/home/zuul/overcloud_prep_images.log.txt.gz#_2019-01-31_14_20_47 | 15:18 |
chkumar|out | recent failure on fs02 | 15:18 |
sshnaidm | quiquell|rover, works perfect: https://review.rdoproject.org/r/#/c/18558/ | 15:19 |
quiquell|rover | sshnaidm: cool cool, let's go for it them | 15:19 |
quiquell|rover | chkumar|out: ack will open a LP | 15:19 |
sshnaidm | quiquell|rover, I think we need a more explaining commit message here: https://review.rdoproject.org/r/#/c/18654/ | 15:20 |
chkumar|out | sshnaidm: panda https://review.openstack.org/#/c/634109/ anything more needed on this | 15:21 |
panda | chkumar|out: python-tripleoclient core votes ... we are not allowed. | 15:22 |
chkumar|out | panda: thanks! pinged tripleo core thanks :-) | 15:28 |
chkumar|out | weshay: regarding fs20/fs21 test comparison script, where we will keep that script? | 15:30 |
quiquell|rover | chkumar|out: has fail a testproject | 15:33 |
quiquell|rover | chkumar|out: not in periodic | 15:33 |
quiquell|rover | chkumar|out: https://softwarefactory-project.io/zuul/t/rdoproject.org/builds?job_name=periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-master-upload | 15:33 |
chkumar|out | quiquell|rover: not sure tomorrow morning it will affect promotion jon | 15:33 |
quiquell|rover | Ok drop now have paste all in the ether | 15:33 |
chkumar|out | *job | 15:33 |
quiquell|rover | chkumar|out: depend on what they are testing in the review | 15:33 |
quiquell|rover | chkumar|out: ahh silly me this is you testing it ? | 15:34 |
quiquell|rover | chkumar|out: it's not timeout weird... | 15:34 |
quiquell|rover | chkumar|out: going to open the LP but not mark it at promotion-blocker | 15:35 |
quiquell|rover | chkumar|out: tomorrow we see | 15:35 |
chkumar|out | quiquell|rover: sure /me out now :-) | 15:36 |
quiquell|rover | chkumar|out: yep me too soon | 15:36 |
panda | zbr|ssbarnea: muteeeee | 15:37 |
quiquell|rover | chkumar|out: feels transitory | 15:37 |
panda | zbr|ssbarnea: muuuuuuuute | 15:37 |
panda | zbr|ssbarnea: thaaaaaanks | 15:37 |
*** chkumar|out is now known as invincible | 15:37 | |
*** trown|outtypewww is now known as trown | 15:38 | |
panda | arxcruz: muuuute | 15:38 |
panda | I still hear mac noises :( | 15:39 |
quiquell|rover | sshnaidm: Can you look at this OVB issue https://bugs.launchpad.net/tripleo/+bug/1814130 ? | 15:40 |
openstack | Launchpad bug 1814130 in tripleo "OVB fs002 job failing at IPMI power status" [High,Confirmed] - Assigned to Quique Llorente (quiquell) | 15:40 |
quiquell|rover | sshnaidm: thanks | 15:40 |
quiquell|rover | drop now | 15:40 |
quiquell|rover | weshay: everything is in the etherpad https://review.rdoproject.org/etherpad/p/ruckrover-unisprint5 | 15:40 |
*** quiquell|rover is now known as quique|rover|off | 15:40 | |
*** udesale has quit IRC | 15:40 | |
*** ccamacho has quit IRC | 15:41 | |
*** ccamacho has joined #oooq | 15:50 | |
ykarel | quique|rover|off, master promotion still failing | 16:03 |
ykarel | weshay, panda can u check ^^, multiple attempts failed with different reasons | 16:04 |
*** saneax has joined #oooq | 16:04 | |
panda | marios: zbr|ssbarnea meeting tomorrow at 10 UTC for cross pollination ? | 16:04 |
zbr|ssbarnea | panda: sure, please send calendar invite (mention your BJ id) | 16:05 |
marios | panda: ack | 16:06 |
sshnaidm | panda, can you vote please? https://review.openstack.org/#/c/633444/ | 16:06 |
kopecmartin | weshay, panda how can I add user stories to the sprint? i can't find any big "Add to the sprint" button :/ | 16:07 |
panda | kopecmartin: are you coming from an epic ? | 16:07 |
panda | kopecmartin: I mean the user story is associated to an epic ? | 16:07 |
kopecmartin | panda, yes | 16:07 |
panda | kopecmartin: ok, go to the epic, and add it with the big + button at the bottom | 16:08 |
rfolco | weshay, can I make us #653 a task a of #650 ? | 16:11 |
panda | ykarel: looking | 16:11 |
kopecmartin | panda, i see only "create a relationship with" window, nothing which is sprint related | 16:14 |
panda | kopecmartin: which one is the epic ? | 16:14 |
kopecmartin | panda, https://tree.taiga.io/project/tripleo-ci-board/epic/657 | 16:15 |
weshay | rfolco, ya | 16:18 |
panda | kopecmartin: below just before the omment there should be a line called "related user stories" with a big + on the other side of the line | 16:18 |
kopecmartin | panda, there is, but i can set there only a relationship of a new or existing user story with the epic, no option for adding it to the sprint | 16:19 |
panda | kopecmartin: ah you already have the us ? | 16:20 |
kopecmartin | panda, yes :) | 16:20 |
panda | kopecmartin: 2 minutes bj, it's quicker | 16:21 |
kopecmartin | panda, sure https://bluejeans.com/3988558640 | 16:21 |
marios | weshay: you like it? (tripleo docs) moving ci into its own section https://review.openstack.org/#/c/633771 http://logs.openstack.org/71/633771/3/check/openstack-tox-docs/0ea6f8e/html/ thanks. no new content just rearranges sections in the index and adds standalone ci index @ /ci/index.html | 16:22 |
* weshay looks | 16:23 | |
marios | rlandy: you've written a lot of the stuff there wdyt? ^ | 16:23 |
weshay | ah nice | 16:23 |
weshay | thanks marios | 16:23 |
marios | imo the info was lost in the contributor guide | 16:23 |
rlandy | marios: +1 | 16:23 |
marios | and gives us a nicer place to add stuff/fixup our docs in general | 16:23 |
marios | (like the new reproducer stuff) | 16:23 |
marios | k | 16:23 |
marios | thanks | 16:23 |
*** quique|rover|off is now known as quiquell|rover | 16:24 | |
quiquell|rover | ykarel: looking | 16:24 |
rlandy | marios: one comment | 16:25 |
rlandy | not sure we want to add the old reproducer stuff | 16:25 |
quiquell|rover | ykarel: we have clean the promoter and switches to overlay2 | 16:25 |
marios | rlandy: sure but lets make it a followup so its clean this is just moving things in the index | 16:26 |
marios | rlandy: except we can't really remove the old stuff alltogether until we have the new stuff in place for docs. alternatively, whichever review adds the new reproducer docs should remove the old stuff? wdyt | 16:27 |
marios | panda: weshay rfolco i think we may need another task for standalones - for tracking fixes. e.g. the thing i mentioned earlier wrt the tempest weshay - as discovered today scen 3 is broken because we weren't running designate and when we do it fails http://logs.openstack.org/19/633919/2/check/tripleo-ci-centos-7-scenario003-standalone/326536d/logs/undercloud/home/zuul/tempest.log.txt.gz#_2019-01-31_14_15_19 https://review.openstack.org/#/c/634223 | 16:31 |
ykarel | quiquell|rover, so cleaning not done correctly? | 16:31 |
ykarel | why switches to overlay? | 16:32 |
quiquell|rover | ykarel: it was could be the overlay2 | 16:32 |
marios | panda: weshay rfolco also other etc issues around the standalone have come up. latest today that we wanted to expand coverage to also include python-tripleoclient (quique posted https://review.openstack.org/#/c/634225 https://review.openstack.org/#/c/634231 ) | 16:32 |
quiquell|rover | ykarel: after cleaning still 28gb were there, panda looked at it | 16:32 |
marios | weshay: panda rfolco so an 'existing jobs maintenance & fixes' story , maybe storyless task | 16:33 |
quiquell|rover | ykarel: looks like it's related to old overlay fs | 16:33 |
weshay | marios, so I think that is ruck/rover work | 16:33 |
panda | ykarel: performance and bugs. | 16:33 |
weshay | if we can pass off to quiquell|rover and Chandan.. I think it's fine | 16:33 |
marios | weshay: like 'fix designate scenario3' ... ack. i mean, ok once it is in promotions then it belongs to ruck/rover ? | 16:33 |
ykarel | quiquell|rover, ok so issue not clear yet | 16:33 |
rlandy | marios: fine by me | 16:33 |
marios | weshay: ok. i was calling that one out in particular cos its work needs doing to fix designate as the current specific example | 16:34 |
weshay | marios, I think we're calling scen1-4 done by the sprint team | 16:34 |
marios | weshay: ack ok thanks | 16:34 |
weshay | where f28 container builds bugs would be tracked by the team because it's def in progress | 16:34 |
quiquell|rover | weshay, marios: same we did with f28 stuff | 16:34 |
ykarel | panda, ok, so issue is still there, right | 16:34 |
weshay | quiquell|rover, what do you want to do w/ my mariadb update to the cockpit? | 16:35 |
weshay | investigate json datasource first? | 16:35 |
marios | time to go do other stuff with other people | 16:35 |
marios | i mean , you folks are awesome and all but | 16:35 |
quiquell|rover | Wesh | 16:35 |
quiquell|rover | weshay: let's try if we can do i | 16:35 |
weshay | k | 16:35 |
quiquell|rover | weshay: without mariadb | 16:36 |
panda | ykarel: quiquell|rover: besides space, there was anything else on the promoter logs ? | 16:36 |
quiquell|rover | weshay: proxy python server that read LP and grafana connect to it | 16:36 |
weshay | quiquell|rover, imho it may be nice to have a regular sql db as a datasource merged for future work as well | 16:36 |
weshay | quiquell|rover, k | 16:36 |
quiquell|rover | panda: no that I saw | 16:36 |
panda | quiquell|rover: before you moved to overlay2, there was a master promotion running, did you get if it completed after ? | 16:37 |
weshay | arxcruz, can you hand off the stacks issue to quiquell|rover as well? | 16:37 |
quiquell|rover | weshay: could be for failed pronoters jobs and noop is good thing too | 16:38 |
arxcruz | weshay: done already | 16:38 |
quiquell|rover | weshay: every snapshot view graph | 16:38 |
ykarel | panda, ack, what i saw was space, registry tls error, and now a new error, and master not promoted yet | 16:38 |
weshay | ya.. I found the straight query to mariadb from graphana powerful | 16:39 |
quiquell|rover | panda: don't know just started the thing and went away | 16:39 |
quiquell|rover | weshay: ack the we go to mariadb for LP | 16:39 |
quiquell|rover | weshay: be sure records are limited and don't grow | 16:39 |
quiquell|rover | weshay: we just one a snapshot of LPs | 16:40 |
weshay | ok.. so for lp I'll leave it to your expertise, but I do have plans to track more $things via mariadb | 16:40 |
quiquell|rover | Like truncte/insert | 16:40 |
weshay | quiquell|rover, ya.. bugs as sql keys makes sense to me | 16:40 |
quiquell|rover | Ack | 16:40 |
weshay | and we'll have a good record over time | 16:40 |
weshay | we could potentially add columns for reporter and maybe who fixed the bug too.. the second would be harder | 16:41 |
weshay | ykarel, quiquell|rover master current-tripleo should not promote now should it? | 16:42 |
* weshay see's scenario004 failing in the latest run | 16:42 | |
ykarel | weshay, 29th hash | 16:42 |
ykarel | hash from 29th jan | 16:42 |
ykarel | it's not promoting since 29 due to one or other issue in promoter | 16:43 |
weshay | ykarel, ah k.. cool | 16:43 |
quiquell|rover | weshay: we are affected by f28 | 16:43 |
*** fmount has joined #oooq | 16:43 | |
quiquell|rover | Is part of the criteria | 16:43 |
weshay | oh... k.. I still think f28 needs to be promoted by hand.. maybe I'm wrong | 16:43 |
weshay | panda, would know | 16:43 |
weshay | and it's not the end of the world if they are slightly diff hashes tbh | 16:43 |
weshay | so +1 on getting master to promote | 16:44 |
quiquell|rover | weshay: issue is criteria is toci jobtype | 16:45 |
quiquell|rover | So we cannot skip f28 | 16:45 |
weshay | wait.. why? | 16:45 |
weshay | blue? | 16:45 |
quiquell|rover | Give a few and we blue | 16:45 |
weshay | https://bluejeans.com/4113567798 k | 16:46 |
quiquell|rover | F28 shares same criteria as centoa in standalone | 16:46 |
quiquell|rover | Same toci_jobtype | 16:46 |
panda | weshay: quiquell|rover f28 needs to be promoted by hand, but can be criteria for centos7 promotion | 16:46 |
panda | I'll join too | 16:47 |
quiquell|rover | panda: but we cannot skip it from promoter | 16:47 |
weshay | quiquell|rover, why? | 16:47 |
quiquell|rover | Give me 15 min | 16:47 |
fmount | Hi there, I'm running quickstart to make a deployment with controller>1 (-N ~/.quickstart/config/nodes/3ctrl_*) and I see ntp_server parameter is missing | 16:47 |
quiquell|rover | F28 and c7 share same toci_jobtype | 16:47 |
panda | quiquell|rover: it should be fixed, let me look | 16:48 |
fmount | weshay: I see you as owner of some commits on the node config files | 16:48 |
panda | quiquell|rover: at the configurations | 16:48 |
hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, (2 more messages) | 16:48 |
quiquell|rover | panda: did you have a review to use job names? | 16:49 |
fmount | I propose a fix for that kind of configuration to make ansible set that fact correctly | 16:49 |
*** bogdando has quit IRC | 16:50 | |
weshay | fmount, the param should not be in nodes config.. but what's the issue? | 16:50 |
weshay | fmount, http://codesearch.openstack.org/?q=ntp_server&i=nope&files=&repos=tripleo-quickstart,tripleo-quickstart-extras | 16:50 |
fmount | I think that param is featureset independent | 16:51 |
weshay | correct | 16:51 |
fmount | https://bugs.launchpad.net/tripleo/+bug/1814116 | 16:51 |
openstack | Launchpad bug 1814116 in tripleo "overcloud deploy blocked. Can't deploy overcloud due to missing NTP parameter in the overcloud-deploy.sh script" [Low,In progress] - Assigned to Francesco Pantano (fmount) | 16:51 |
panda | quiquell|rover: it wasnt' merged :( dammit | 16:52 |
panda | quiquell|rover: https://review.rdoproject.org/api/17780 | 16:52 |
fmount | it is more related to the overcloud configuration you will need to deploy | 16:52 |
quiquell|rover | panda: will work that our | 16:53 |
quiquell|rover | Is a blocker now | 16:54 |
*** vinaykns has quit IRC | 16:54 | |
rlandy | weshay; quiquell|rover: you guys have some conflicting review comments :) | 16:54 |
panda | quiquell|rover: I think we are ready to merge it | 16:55 |
panda | quiquell|rover: we were just missing a vote, and it was lost in other tasks | 16:55 |
*** vinaykns has joined #oooq | 16:56 | |
quiquell|rover | panda: do the rebase I will merge | 16:57 |
quiquell|rover | rlandy: my only concern is restoring primary group of user | 16:57 |
*** ykarel is now known as ykarel|away | 16:57 | |
rlandy | quiquell|rover: right - so I removed all the docker work | 16:57 |
quiquell|rover | Other stuff not very important | 16:57 |
weshay | fmount, do you have a copy of the log? | 16:57 |
rlandy | quiquell|rover: you guys also conflict on deps install | 16:58 |
fmount | yes, it you take a look here: | 16:58 |
fmount | https://bugs.launchpad.net/tripleo/+bug/1814116 | 16:58 |
openstack | Launchpad bug 1814116 in tripleo "overcloud deploy blocked. Can't deploy overcloud due to missing NTP parameter in the overcloud-deploy.sh script" [Low,In progress] - Assigned to Francesco Pantano (fmount) | 16:58 |
rlandy | I actually agree with you about using quickstart deps install | 16:58 |
fmount | you can find the link to the log | 16:58 |
weshay | 'Specify --ntp-server as parameter or NtpServer in ' | 16:58 |
weshay | InvalidConfiguration: Specify --ntp-server as parameter or NtpServer in environments when using multiple controllers (with HA). | 16:58 |
weshay | Specify --ntp-server as parameter or NtpServer in environments when using multiple controllers (with HA). | 16:58 |
weshay | + status_code=1 | 16:58 |
quiquell|rover | rlandy: we have to find a way to CI from clean imagen and put as much a a possible at pre.yaml | 16:59 |
fmount | there is a task that set the --ntp-server {{ ntp_server }} if ntp_server is defined | 16:59 |
weshay | fmount, you using rdo-cloud? | 16:59 |
panda | quiquell|rover: rebased | 16:59 |
rlandy | quiquell|rover: trying to meet all comments - but hard when they conflict :) | 17:00 |
weshay | fmount, ? | 17:00 |
quiquell|rover | rlandy: let me look at it again | 17:00 |
quiquell|rover | Pandas +w | 17:01 |
rlandy | also weshay had a review to add more deps to be installed | 17:01 |
rlandy | quiquell|rover: ^^ | 17:01 |
rlandy | in pre | 17:01 |
rlandy | which is fine | 17:01 |
weshay | ya.. that was python3 | 17:01 |
weshay | though | 17:01 |
quiquell|rover | rlandy: this is all good | 17:01 |
weshay | I abandoned | 17:01 |
quiquell|rover | Let put stuff at pre.yaml so we CI it | 17:01 |
weshay | rlandy, quiquell|rover imho.. I'm tenative about duplicating --install-deps | 17:01 |
weshay | quiquell|rover, nack | 17:02 |
quiquell|rover | weshay: do it | 17:02 |
weshay | pre.yaml I think is too late | 17:02 |
rlandy | weshay: I agree with quiquell|rover that creating a dep on quickstart is the wromg why to go | 17:02 |
quiquell|rover | weshay: depends if we call pre.yaml in bash no | 17:02 |
weshay | happy to discuss.. but we're going to maintain two ways to setup for ansible? | 17:03 |
weshay | 3 actually? | 17:03 |
weshay | quickstart.sh, old repro, new repro | 17:03 |
rlandy | weshay: what was wrong with https://review.rdoproject.org/r/#/c/18545/7/playbooks/tripleo-ci-reproducer/pre.yaml? | 17:03 |
rlandy | weshay: I am just trying to address all comments - but they are conflicting here | 17:03 |
quiquell|rover | weshay: are we going to install ansible? | 17:04 |
*** kopecmartin is now known as kopecmartin|off | 17:04 | |
weshay | one it duplicates, and two it's 3-5 min in the run | 17:04 |
rlandy | quiquell|rover: ack | 17:04 |
rlandy | we have to install ansible | 17:04 |
weshay | quiquell|rover, ansible libselinux and netaddr need to be installed up front | 17:04 |
rlandy | in shell script | 17:04 |
rlandy | netaddr not | 17:04 |
rlandy | that should be taken care of | 17:04 |
weshay | rlandy, I think we'll have to | 17:04 |
rlandy | not an dircet ansible requirement | 17:04 |
weshay | but it's a quickstart.sh bug | 17:05 |
rlandy | the other two are | 17:05 |
weshay | we also have the option of tearing out the install-deps feature in quickstart.sh into it's own script | 17:05 |
rlandy | we need some consensus shere | 17:05 |
rlandy | here | 17:05 |
weshay | and having both scripts call it | 17:05 |
quiquell|rover | weshay: if we install them at pre.yall and call it in the bash is good | 17:06 |
fmount | weshay: no I didn't use it, my point was more generic, no environment related, but I understand the design choice on put it there | 17:06 |
rlandy | you guys also disagree on docker | 17:06 |
quiquell|rover | Let me check | 17:06 |
rlandy | group stuff | 17:06 |
weshay | rlandy, on newgrp | 17:06 |
weshay | you mean right? | 17:06 |
rlandy | should be bj for 5 | 17:06 |
weshay | I've been sitting in my blue | 17:06 |
rlandy | I'd like to get comments addressed and ready for monday merge | 17:06 |
quiquell|rover | weshay: I am ok with newgrp but we have to restore the primary group after | 17:06 |
rlandy | joining weshay sitting on blue | 17:07 |
weshay | quiquell|rover, it doesn't replace man | 17:07 |
weshay | it adds | 17:07 |
quiquell|rover | It replace | 17:07 |
quiquell|rover | Let me show you | 17:07 |
weshay | I've checked man | 17:07 |
quiquell|rover | Need 5 min for blue | 17:07 |
weshay | works for me | 17:07 |
quiquell|rover | Humm weird | 17:07 |
weshay | quiquell|rover, join my blue dude | 17:08 |
quiquell|rover | I have too and docker was used to create files | 17:08 |
quiquell|rover | Joining | 17:11 |
quiquell|rover | weshay: paste the blue link | 17:13 |
quiquell|rover | please | 17:13 |
*** ccamacho has quit IRC | 17:13 | |
panda | weshay: quiquell|rover : rlandy install script https://review.openstack.org/613696 | 17:19 |
weshay | vinaykns, here | 17:28 |
*** dsneddon has quit IRC | 17:29 | |
sshnaidm | weshay, can you vote please? https://review.openstack.org/#/c/633444/ | 17:30 |
weshay | [weshayutin@localhost ~]$ groups | 17:32 |
weshay | weshayutin wheel libvirt docker | 17:32 |
weshay | [weshayutin@localhost ~]$ | 17:32 |
weshay | sshnaidm, can you join my blue | 17:36 |
vinaykns | hey weshay: looks puppet is taking forever to configure on the host | 17:39 |
vinaykns | on UC | 17:40 |
vinaykns | weshay: it got stuck at this point http://pastebin.test.redhat.com/705349 | 17:41 |
*** dsneddon has joined #oooq | 17:45 | |
vinaykns | weshay: http://pastebin.test.redhat.com/705352 | 17:45 |
weshay | https://github.com/rdo-infra/ansible-role-tripleo-ci-reproducer/blob/master/playbooks/tripleo-ci-reproducer/pre.yaml#L37 | 17:46 |
*** ccamacho has joined #oooq | 17:53 | |
*** ykarel|away has quit IRC | 17:59 | |
*** derekh has quit IRC | 18:00 | |
*** ykarel|away has joined #oooq | 18:00 | |
*** saneax has quit IRC | 18:05 | |
*** jpena is now known as jpena|off | 18:08 | |
*** ykarel|away has quit IRC | 18:13 | |
*** holser_ has quit IRC | 18:17 | |
*** trown is now known as trown|lunch | 18:29 | |
*** dtantsur is now known as dtantsur|afk | 18:39 | |
*** quiquell|rover is now known as quique|rover|off | 18:41 | |
*** panda is now known as panda|off | 18:41 | |
hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, (2 more messages) | 18:48 |
weshay | vinaykns, can you tmate? | 18:54 |
vinaykns | sure | 18:54 |
weshay | how does one execute bindep? | 18:54 |
vinaykns | but I changed the ntp server | 18:56 |
weshay | vinaykns, ya.. that looked good :) | 18:59 |
*** vinaykns has quit IRC | 19:22 | |
sshnaidm | rlandy, fails for me because doesn't find docker-compose.. weird, because I have it | 19:24 |
*** fmount has quit IRC | 19:32 | |
*** sshnaidm is now known as sshnaidm|off | 19:33 | |
*** fmount has joined #oooq | 19:33 | |
*** trown|lunch is now known as trown | 19:43 | |
*** dsneddon has quit IRC | 19:46 | |
rlandy | wierd | 19:47 |
rlandy | installed how? | 19:47 |
rlandy | as what user? | 19:47 |
*** holser_ has joined #oooq | 19:55 | |
*** weshay has quit IRC | 19:56 | |
*** rnoriega has quit IRC | 19:57 | |
*** rnoriega has joined #oooq | 19:57 | |
*** weshay has joined #oooq | 20:01 | |
*** dsneddon has joined #oooq | 20:02 | |
*** vinaykns has joined #oooq | 20:10 | |
*** jfrancoa has quit IRC | 20:13 | |
*** holser_ has quit IRC | 20:21 | |
*** rfolco has quit IRC | 20:40 | |
hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024, tripleo-ci-centos-7-ovb-1ctlr_1comp- (2 more messages) | 20:48 |
rlandy | weshay: sshnaidm|off: not sure we want to ditch pre altogether | 20:57 |
rlandy | we will still need some stuff | 20:57 |
weshay | ya.. I wanted to go through each of the 7 or 8 sections there | 20:58 |
rlandy | like creating docker config if it does not exist | 20:58 |
rlandy | weshay: I am going through that all now | 20:58 |
rlandy | which steps can go where | 20:58 |
rlandy | weshay; can show you what I have - if you're not too tired of talking about this | 20:58 |
* weshay wonders if the config is created by the rpm | 20:59 | |
weshay | however I guess since we're virtenv'ing it | 20:59 |
weshay | ya.. we may have to leave that | 21:00 |
weshay | that would require sudo | 21:00 |
rlandy | oh | 21:00 |
rlandy | I guess I should install | 21:00 |
rlandy | we need to add the group if it is not there | 21:01 |
weshay | rlandy, it's not created by the rpm | 21:01 |
rlandy | the install does not do that | 21:01 |
rlandy | the file is created via rpm | 21:02 |
rlandy | the group is not adde | 21:02 |
weshay | \/etc/docker/daemon.json is not created by the rpm | 21:02 |
rlandy | name: Create docker configuration file | 21:02 |
weshay | https://github.com/rdo-infra/ansible-role-tripleo-ci-reproducer/blob/master/playbooks/tripleo-ci-reproducer/pre.yaml#L88 | 21:03 |
rlandy | ^^ overwrites it instead of adding to it | 21:03 |
weshay | it's not added by the rpm | 21:03 |
rlandy | it is | 21:03 |
weshay | rpm -q --whatprovides /etc/docker/daemon.jso | 21:03 |
rlandy | that just overwrites it | 21:03 |
rlandy | [zuul@subnode-1 ~]$ ls /etc/docker/daemon.json | 21:03 |
rlandy | [zuul@subnode-1 ~]$ sudo cat /etc/docker/daemon.json | 21:03 |
rlandy | {} | 21:03 |
weshay | rlandy, sorry but you are incorrect | 21:03 |
weshay | maybe by the service | 21:03 |
* weshay checks | 21:03 | |
weshay | run that command | 21:04 |
weshay | rlandy, nope | 21:04 |
rlandy | [zuul@subnode-1 ~]$ rpm -q --whatprovides /etc/docker/daemon.json | 21:04 |
rlandy | docker-1.13.1-88.git07f3374.el7.centos.x86_64 | 21:04 |
weshay | how do you have a higher version of docker | 21:05 |
weshay | f28? | 21:05 |
rlandy | centos | 21:05 |
weshay | rlandy, ok.. on fedora it's not created | 21:05 |
weshay | :) | 21:05 |
rlandy | oh how fantastic | 21:05 |
rlandy | also | 21:06 |
rlandy | comparing the two deps sets | 21:06 |
rlandy | we are off | 21:06 |
weshay | I think we were planning on using the python module anyway | 21:06 |
rlandy | to install docker? | 21:06 |
weshay | https://github.com/rdo-infra/ansible-role-tripleo-ci-reproducer/blob/master/playbooks/tripleo-ci-reproducer/pre.yaml#L42 | 21:07 |
rlandy | that is fine - will remove rpm | 21:07 |
rlandy | so ... | 21:08 |
rlandy | using some of pre could work | 21:08 |
rlandy | if we skipped the dangerous parts | 21:08 |
rlandy | we will need the config | 21:08 |
weshay | ya | 21:08 |
rlandy | and the service start and enable | 21:08 |
rlandy | and the pip install | 21:08 |
weshay | just remove anything that requires sudo | 21:08 |
rlandy | (for the moment) | 21:09 |
weshay | everything else should be idempotent | 21:09 |
rlandy | ha | 21:09 |
rlandy | A LOT requires sudo | 21:09 |
rlandy | everything we need | 21:09 |
rlandy | it's all become true | 21:09 |
weshay | rlandy, let's chat | 21:10 |
rlandy | weshay: can you give me 5? need some tea | 21:10 |
weshay | ah of course for tea | 21:11 |
*** holser_ has joined #oooq | 21:12 | |
rlandy | weshay: k - joining your blue | 21:14 |
*** trown is now known as trown|outtypewww | 22:02 | |
rlandy | weshay: we can set a var when we run pre from launcher playbook | 22:09 |
rlandy | something like in_launcher | 22:10 |
weshay | rlandy, ok.. but it probably isn't required | 22:10 |
weshay | if we check first | 22:10 |
weshay | however I could see people still complaining | 22:10 |
*** ccamacho has quit IRC | 22:10 | |
weshay | so do as you see fit | 22:10 |
rlandy | idk | 22:10 |
weshay | if a user has $user in group docker + packages installed | 22:11 |
weshay | I would expect pre.yaml to run through w/o passwordless sudo | 22:11 |
rlandy | well ... | 22:12 |
* rlandy thinks about users | 22:12 | |
rlandy | gerrit user | 22:12 |
rlandy | rdo gerrit user | 22:12 |
rlandy | current user | 22:12 |
rlandy | we let people define gerrit user | 22:13 |
rlandy | should pre run with that user? | 22:13 |
*** agopi has quit IRC | 22:16 | |
weshay | rlandy, I can run as $USER whayutin to gerrit user weshayutin | 22:20 |
rlandy | weshay: ack | 22:20 |
rlandy | I need to add the ability to change all users in the script | 22:20 |
rlandy | user_pri_key: "id_rsa" | 22:21 |
rlandy | user_pub_key: "{{ user_pri_key }}.pub" | 22:21 |
rlandy | ssh_path: "{{ ansible_user_dir }}/.ssh" | 22:21 |
rlandy | upstream_gerrit_user: "{{ ansible_user }}" | 22:21 |
rlandy | upstream_gerrit_key: "{{ user_pri_key }}" | 22:21 |
rlandy | rdo_gerrit_user: "{{ ansible_user }}" | 22:21 |
rlandy | rdo_gerrit_key: "{{ user_pri_key }}" | 22:21 |
*** holser_ has quit IRC | 22:21 | |
hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset022, tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph- (2 more messages) | 22:49 |
*** holser_ has joined #oooq | 22:58 | |
*** agopi has joined #oooq | 22:59 | |
weshay | rlandy, ok.. got it working w/ tq.. using https://review.openstack.org/#/c/634329/ | 23:06 |
weshay | rlandy, going to add a bindep to the role.. I think | 23:07 |
weshay | and source install-deps.sh from your script | 23:07 |
rlandy | weshay: cool - tqe review updated | 23:07 |
weshay | # install required deps for a python virtual environment | 23:07 |
weshay | install_deps | 23:07 |
weshay | # setup the virtual environment | 23:07 |
weshay | install_virtual_env | 23:07 |
weshay | # activate the virtual environment | 23:07 |
weshay | . $OPT_WORKDIR/bin/activate | 23:07 |
rlandy | weshay: adding reproducer role changes now | 23:07 |
weshay | # continue package installs with bindep | 23:07 |
weshay | install_package_deps_via_bindep | 23:07 |
weshay | and those are steps to install the virtenv and packages | 23:07 |
weshay | so that sould work consistently across projects | 23:08 |
*** rascasoft has quit IRC | 23:10 | |
*** holser_ has quit IRC | 23:27 | |
*** vinaykns has quit IRC | 23:28 | |
rlandy | weshay: https://review.rdoproject.org/r/#/c/18664/1/playbooks/tripleo-ci-reproducer/pre.yaml | 23:30 |
rlandy | ^^ review with config changes - will add checks to that | 23:30 |
weshay | so the Create docker configuration file | 23:31 |
weshay | will fail the Sagi case? | 23:31 |
weshay | do you need to check the contents of the file first w/o sudo? | 23:32 |
weshay | and only run the blockinfile if it doesn't exist right? | 23:32 |
weshay | if the correct contents don't exist | 23:32 |
weshay | make it idempotent | 23:32 |
*** rascasoft has joined #oooq | 23:54 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!