Thursday, 2019-01-31

*** rascasoft has quit IRC00:02
*** tosky has quit IRC00:46
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-undercloud-containers, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-scenario010-multinode-oooq-container @  (3 more messages)00:48
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-undercloud-containers, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-scenario010-multinode-oooq-container @  (3 more messages)02:48
*** apetrich has quit IRC03:15
*** saneax has joined #oooq03:19
*** gkadam-bmgr has joined #oooq03:34
*** skramaja has joined #oooq03:35
*** rlandy|bbl is now known as rlandy03:49
*** ykarel|away has joined #oooq03:54
*** udesale has joined #oooq03:55
*** udesale has quit IRC03:56
*** udesale has joined #oooq04:28
*** ykarel|away is now known as ykarel04:34
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-undercloud-containers, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-scenario010-multinode-oooq-container @  (3 more messages)04:48
*** ykarel has quit IRC05:59
*** chandankumar is now known as chkumar|ruck06:14
*** ykarel has joined #oooq06:16
quiquell|offchkumar|ruck: o/06:17
chkumar|ruckquiquell|off: \o/06:17
chkumar|ruckquiquell|off: want to ruck or rover?06:17
quiquell|offI can Rover but will ruck too06:18
chkumar|ruckquiquell|off: http://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-undercloud-containers is in pretty bad state06:19
chkumar|ruckalready an alert posted06:19
quiquell|offHave to leave now going back in a few06:19
chkumar|rucksure06:19
chkumar|ruckssbarnea: arxcruz let me know when you are back on irc, we have do a handover06:20
*** quiquell|off is now known as quiquell_rover06:20
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-undercloud-containers, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-scenario010-multinode-oooq-container @  (3 more messages)06:48
quiquell_roverchkumar|ruck: I see a lot of noop failures06:53
quiquell_roverchkumar|ruck: also a lot of timeout06:54
chkumar|ruckquiquell_rover: I have not went over noop failure06:55
chkumar|ruckquiquell_rover: you start with noop failures, I will go over promotion jobs06:55
quiquell_roverchkumar|ruck: ack06:57
*** quiquell_rover is now known as quiquell|rover07:07
*** quiquell|rover has quit IRC07:07
*** quiquell has joined #oooq07:07
quiquellchkumar|ruck: do we have the etherpad ?07:08
*** quiquell is now known as quiquell|rover07:09
*** jfrancoa has joined #oooq07:09
chkumar|ruckquiquell|rover: nope07:10
chkumar|ruckquiquell|rover: feel free to create one07:10
quiquell|roverchkumar|ruck: I mean the previous one07:11
quiquell|roverchkumar|ruck: Want you to try the new reproducer ?07:13
quiquell|roverchkumar|ruck: I am going to use it for the undercloud job stuff07:13
*** ccamacho has joined #oooq07:15
chkumar|ruckquiquell|rover: for this one https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-fedora-28-master-containers-build/12515ad/logs/undercloud/home/zuul/undercloud_install.log.txt.gz#_2019-01-31_05_22_4407:16
chkumar|ruckquiquell|rover: https://review.openstack.org/#/c/634109/07:16
chkumar|ruckquiquell|rover: yes I want to use the new reproducer07:17
quiquell|roversshnaidm|afk: what do we doo with https://review.openstack.org/#/c/633444/ ?07:21
quiquell|rovermarios: ^ ?07:21
mariosquiquell|rover: o/07:21
mariosquiquell|rover: looking https://review.openstack.org/#/c/633444/ ?07:21
mariosquiquell|rover: is virt-resize log already collected i asked yesterday07:22
mariosquiquell|rover: is it this? https://github.com/openstack-infra/tripleo-ci/blob/44a129efdaec1fd05acf5d91ecae6cafab7989b8/toci-quickstart/config/collect-logs.yml#L10407:25
mariosno var/log/libvirt probably07:25
mariosbut ok so its likely there then07:25
quiquell|rovermarios: reproducer is not using toci07:25
mariosquiquell|rover: so then07:25
quiquell|rovermarios: that's why we collect it again there07:25
marioswhere/how please07:25
quiquell|roverhttp://logs.rdoproject.org/58/18558/28/check/tripleo-ci-reproducer-fedora-28-libvirt/fc70dcc/tripleo-ci-reproducer/virt-customize.log07:26
mariosquiquell|rover: oh you mean you just store them in the root07:27
mariosmy question was _where_ are those logs are they in the /logs directory07:27
mariosquiquell|rover: so the answer is we don't use collect logs we just write them to the root logs dir07:28
quiquell|rovermarios: yep since we generate different stuff07:28
quiquell|rovermarios: This is the reproducer role change to Depends-On to the this https://review.rdoproject.org/r/#/c/1855807:28
marioswasup with undercloud_containers we have bug?07:30
mariostimeout07:30
quiquell|rovermarios: yep looking inito it07:30
quiquell|roverSomething related with puppet07:30
mariosquiquell|rover: ack so holding +A07:30
*** kopecmartin|off is now known as kopecmartin07:30
quiquell|rovermarios: ack07:30
quiquell|roverhave to drop for a few07:31
quiquell|roverSorry was testing the away stuff from weechat07:32
quiquell|rovermarios: chan you send me a message to see if weechat answer it ?07:33
mariosquiquell|rover: no07:34
marios(you mean pvt or here?)07:34
mariosquiquell|rover: ^07:34
mariosquiquell|rover: tell07:34
mariosquiquell|rover: me07:34
mariosquiquell|rover: when07:35
mariosquiquell|rover: to07:35
mariosquiquell|rover: s07:35
mariosquiquell|rover: t07:35
mariosquiquell|rover: o07:35
mariosquiquell|rover: p07:35
mariosquiquell|rover: .07:35
chkumar|ruckquiquell|rover: |o| -> message received by weechat07:35
quiquell|roverHumm weechat not answering that's no good07:36
quiquell|roverok now I have to really drop for a few07:36
*** holser_ has joined #oooq07:39
ykarelquiquell|rover, chkumar|ruck there is some issue in promotions07:44
ykarel /var/lib/docker/tmp/GetImageBlob143622093: no space left on device"}07:44
ykarelfailed: [localhost] (item=zaqar) => {"changed": false, "item": "zaqar", "msg": "Error pulling trunk.registry.rdoproject.org/tripleorocky/centos-binary-zaqar - code: None message: open /var/lib/docker/tmp/GetImageBlob868334417: no space left on device"}07:44
ykarelcan u check, master is not promoting since yesterday07:44
ykarelseen ^^ in http://38.145.34.55/rocky.log, you can check other releases too07:45
*** quiquell|rover is now known as quique|rover|brb07:47
marioswdyt folks moving ci into new section https://review.openstack.org/#/c/633771 http://logs.openstack.org/71/633771/2/check/openstack-tox-docs/266ee8b/html/07:52
mariosquique|rover|brb: ssbarnea arxcruz panda|off weshay ^07:53
quique|rover|brbmarios: I am ok with it will check07:54
* quique|rover|brb is away: brb07:55
mariosquique|rover|brb: thanks. no new content just rearrange index07:55
*** quique|rover|brb is now known as quiquell|rover07:55
quiquell|rovermarios: send me a message now07:55
mariosquiquell|rover: no07:56
mariosquiquell|rover: i07:56
mariosquiquell|rover: will07:56
mariosquiquell|rover: not07:56
mariosquiquell|rover: never!07:56
quiquell|roverArgg ok07:56
*** quiquell|rover is now known as quique|rover|brb07:56
quique|rover|brbGo back in a few07:56
*** quique|rover|brb is now known as quique|rover|bbl07:56
* quique|rover|bbl is back (gone 00:04:03)07:59
* quique|rover|bbl is away: bbl07:59
*** rascasoft has joined #oooq08:01
*** udesale has quit IRC08:14
*** udesale has joined #oooq08:15
*** quique|rover|bbl is now known as quiquell|rover08:16
* quiquell|rover is back (gone 00:16:43)08:16
*** apetrich has joined #oooq08:17
chkumar|ruckykarel: checking08:19
*** ykarel is now known as ykarel|lunch08:21
mariosquiquell|rover: am i wrong about the keys https://review.openstack.org/#/c/631067/4008:23
mariossshnaidm|afk: ^08:23
quiquell|roverchkumar|ruck: Running undercloud-containers locally don't have the timeouts it's doing tempest now08:26
quiquell|roverchkumar|ruck: want to setup reproducer ?08:26
chkumar|ruckquiquell|rover: yes08:27
chkumar|ruckquiquell|rover: pass me the docs I will start08:27
quiquell|roverchkumar|ruck: https://github.com/rdo-infra/ansible-role-tripleo-ci-reproducer/blob/master/README.md08:28
quiquell|roverchkumar|ruck: clone the role08:28
quiquell|rovermarios: key stuff have just merge, so we can put that in the documentation08:29
quiquell|roverchkumar|ruck: also rlandy is creating the reproducer script for it here08:31
quiquell|roverchkumar|ruck: https://review.openstack.org/#/c/63106708:31
quiquell|roverchkumar|ruck: so you can run the reproducer script from the review for the affected job08:31
*** tosky has joined #oooq08:34
mariosquiquell|rover: right but more than that, rlandy may need to make her role template configurable (is j2) to take into account the user key i want to use08:35
ykarel|lunchquiquell|rover, undercloud-containers should have been fixed with:- https://review.openstack.org/#/c/633944/08:35
ykarel|lunchit looks good after that patch merged: http://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-undercloud-containers&branch=master08:35
quiquell|roverykarel|lunch: Yep working now at the zuul reproducer08:35
mariosquiquell|rover: here https://review.openstack.org/#/c/631067/40/roles/create-zuul-based-reproducer/templates/launcher-playbook.yaml.j2@8308:35
quiquell|roverykarel|lunch: thanks08:36
quiquell|roverykarel|lunch: let's wait for results at noop change https://review.openstack.org/#/c/560445/08:36
ykarel|lunchit should be the same :)08:36
ykarel|lunchquiquell|rover, can u check the promoter issue08:36
quiquell|roverykarel|lunch: we are into that08:37
ykarel|lunchokk, thanks08:37
* ykarel|lunch going for lunch now08:37
quiquell|rovermarios: yep that's what I mean08:37
quiquell|rovermarios: adding --key stuff to bash jinja template and pass the vars to the role08:38
quiquell|rovermarios: at least the flexibiligy is in the role don't know what they want there08:38
mariosquiquell|rover: k08:39
*** jpena|off is now known as jpena08:46
*** bogdando has joined #oooq08:46
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-undercloud-containers, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-scenario010-multinode-oooq-container @  (3 more messages)08:48
*** weshay has quit IRC08:51
*** dsneddon has quit IRC08:51
*** dsneddon has joined #oooq08:55
chkumar|ruckquiquell|rover: I am putting custom script to install docker and create clouds.yaml file if openrc file is there for reproducer09:08
*** ykarel|lunch is now known as ykarel09:09
quiquell|roverchkumar|ruck: run pre.yaml from the role09:10
quiquell|roverchkumar|ruck: it install docke and such09:10
chkumar|ruckquiquell|rover: ok09:10
quiquell|roverchkumar|ruck: openrc is not enough nodepool needs clouds.yaml09:10
quiquell|roverchkumar|ruck: you can test rlandys stuff too09:10
quiquell|roverchkumar|ruck: it do all the requirements09:10
*** dtantsur|afk is now known as dtantsur09:21
arxcruzquiquell|rover: chkumar|ruck let me know if you guys want to sync09:22
quiquell|roverarxcruz: where is the old etherpad ?09:23
arxcruzquiquell|rover: https://review.rdoproject.org/etherpad/p/ruckrover-sprint2409:23
*** sshnaidm|afk is now known as sshnaidm09:25
chkumar|ruckquiquell|rover: new one created?09:31
quiquell|roverchkumar|ruck: #tripleo09:31
chkumar|ruckpanda|off: arxcruz with respect to this bug https://bugs.launchpad.net/tripleo/+bug/1813913 -> https://review.openstack.org/63410909:42
openstackLaunchpad bug 1813913 in tripleo "periodic-tripleo-fedora-28-master-containers-build failing with unauthorized access to rdo registry" [Wishlist,Triaged] - Assigned to Gabriele Cerami (gcerami)09:42
arxcruzpanda|off: just let it go09:42
arxcruzchkumar|ruck: just let it go :D09:43
arxcruzchkumar|ruck: panda|off is handling it, and it's nothing we should care right now09:43
chkumar|ruckarxcruz: ack09:44
panda|offchkumar|ruck: I downgraded it to wishlist, I hoped it was enough of a signal. We'll be working on this in this sprint09:46
*** panda|off is now known as panda09:46
*** derekh has joined #oooq09:48
chkumar|ruckquiquell|rover: for ruck rover handover sending an invite to meet in another 30 mins?09:52
arxcruzchkumar|ruck: which bj ?09:58
chkumar|ruckarxcruz: will pass the bj link here09:59
quiquell|roverchkumar|ruck: 30 sure, you send it ?09:59
chkumar|ruckquiquell|rover: invite sent10:00
quiquell|roverchkumar|ruck: ack10:02
quiquell|roverchkumar|ruck: going to add stuff to grafana to show disk space at promoter10:03
chkumar|ruckquiquell|rover: undercloud timedout gone10:03
quiquell|roverchkumar|ruck: cool10:04
quiquell|roverchkumar|ruck: can you take a look at the stuff I have copied to the etherpad ?10:05
quiquell|roverarxcruz: can you complete the new etherpad with pending stuff ?10:05
chkumar|ruckquiquell|rover: both bugs updated10:05
arxcruzquiquell|rover: which new etherpad ?10:05
quiquell|roverykarel, chkumar|ruck I am going to stop promoter10:05
chkumar|ruckarxcruz: https://review.rdoproject.org/etherpad/p/ruckrover-unisprint510:06
arxcruzquiquell|rover: chkumar|ruck done10:12
quiquell|roverarxcruz: thanks10:13
sshnaidmssbarnea, before setting -1 you'd better to read a bug in commit message and to ask submitter if you have questions10:14
quiquell|roverssbarnea: we don't want to introduce big refactors there https://review.openstack.org/#/c/633444/10:15
ssbarneasshnaidm: i read the bug and it did not contain any clear reasoning.10:15
quiquell|roverssbarnea: We want to conditionally set force_tcg10:16
sshnaidmssbarnea, then need to read again10:16
quiquell|roverssbarnea: with ansible and environment is not easy that's why I used a template10:16
ssbarneaquiquell|rover: what prevents you from using environment: ... to do that?10:16
quiquell|roverssbarnea: Try to set conditionally at environment var at ansible10:16
sshnaidmquiquell|rover, there is a lot of "&&" now in the patch10:16
quiquell|roversshnaidm: then you tell me omit is not working there10:16
quiquell|roverssbarnea: ^10:17
sshnaidmssbarnea, no way in ansible to omit environment variable10:17
quiquell|roversshnaidm: you mean too much ?10:17
ssbarneaquiquell|rover: the var does not accept a false value?10:17
quiquell|roverssbarnea: no10:17
quiquell|roverssbarnea: so please ask before -110:17
quiquell|roverssbarnea: people is not dumb or stupid by default10:17
sshnaidmquiquell|rover, yeah :) seems like replace didn't work much10:17
ssbarneaquiquell|rover: you can still do "export inside the shell script". and avoid creating temporary sh files that do this. there are still alternatives.10:18
quiquell|roverssbarnea: like that better but didn't want to repeat the ENV vars agian and again10:18
quiquell|roverssbarnea: so better put them at one file and use it at all scripts10:18
quiquell|roverssbarnea: also reuse the conditional block10:19
ssbarneaquiquell|rover: you could still use jinja to avid repeading yourself:  {{ setup_vars }} and define them in a single place.10:19
quiquell|roverssbarnea: then you complate the shell: foobar stuff10:19
quiquell|rovers/complete/complicate/10:19
sshnaidmssbarnea, no, duplication of exports in the shell is completely antipattern, external script is better solution10:19
quiquell|roverssbarnea: ^10:20
chkumar|ruckquiquell|rover: there are 4 jobs got time out in master promotion due to undercloud install testing here https://review.rdoproject.org/r/#/c/13943/10:20
quiquell|roverssbarnea: Man is quite clear10:20
ssbarneasshnaidm: ok, in this case make the entire resize logic a single shell script. that would be nice and clean and easy to test/debug.10:20
sshnaidmquiquell|rover, commented10:20
quiquell|roverssbarnea: is not only resize10:20
sshnaidmssbarnea, no, it won't be visible10:20
quiquell|roverssbarnea: is all all little snippets with virt-resize and virt-customize there is no common between them10:21
quiquell|roversshnaidm: ack10:21
sshnaidmssbarnea, and making a thousand scripts doesn't make sense10:21
ssbarneaquiquell|rover: are these tasks needing shared-vars inside the same file/role? do they run on different conditions? if not really they are likely to be easier to implement in a single shell script.10:22
quiquell|roverssbarnea: I am not going to put libvirt_nodepool.yaml into a bash script10:23
quiquell|roverssbarnea: that would be anohter task10:23
quiquell|roverssbarnea: maybe you are right10:23
quiquell|roverssbarnea: But I am nog going to do it10:23
quiquell|roverssbarnea: there is a lot of stuff to refactor at libvirt role, so are going to find a lot of fun there10:24
ssbarneawell, there are exact 10 tasks there, with only one condition on once, to me looks like amost perfect candidate to write as single bash, faster. i would do it if others like the ideas.10:24
quiquell|roverssbarnea: be prepare10:24
pandawhat reviews are you  discussing ? https://review.openstack.org/633444 ?10:25
quiquell|roverssbarnea: well is a mix of bash and python, but now we have libvirt CI so you can propose a patch10:26
quiquell|roverpanda: yep10:26
pandaonly that one ?10:26
quiquell|roverpanda: yep10:27
ssbarneaquiquell|rover: as I mentioned at the end of the message, feel free to ignore my opinion on that. i may be biased because I got some bad experiences with code going like this in infrared, where debugging an error on that area was a nightmare. as least I am glad that our team does not hate bash :D10:27
*** apetrich has quit IRC10:27
chkumar|ruckarxcruz: ssbarnea quiquell|rover https://redhat.bluejeans.com/1571313919/?src=meet_now ruck rover handover link bj10:28
quiquell|roverssbarnea: ack, sorry about my manners too, have to much at my brains now10:28
quiquell|roverssbarnea: want to merge last pieces of repro, pleas help me with that10:28
sshnaidmquiquell|rover, commented on previous patch: https://review.openstack.org/#/c/633444/21/roles/libvirt/setup/overcloud/templates/libguestfs-env.sh.j210:29
chkumar|ruckarxcruz: sshnaidm quiquell|rover syncup time10:30
pandaan alternative solution would be to move that template with the export to a default/main.yaml dict variable virt_environment: , then set ENVIRONMENT: {{ virt_environment }}10:30
sshnaidmchkumar|ruck, you mean ssbarnea ?10:31
chkumar|rucksshnaidm: yes10:31
quiquell|roverchkumar|ruck: going there10:31
quiquell|roverpanda: that is good10:31
quiquell|roverchkumar|ruck: where is the blue10:32
chkumar|ruckquiquell|rover: ssbarnea https://redhat.bluejeans.com/1571313919/?src=meet_now10:33
*** rnoriega has quit IRC10:35
chkumar|ruckssbarnea: can you join?10:35
chkumar|ruckssbarnea: https://redhat.bluejeans.com/1571313919/?src=meet_now10:35
*** rnoriega has joined #oooq10:36
arxcruzhttps://review.rdoproject.org/etherpad/p/ruckrover-unisprint510:38
*** apetrich has joined #oooq10:41
arxcruzchkumar|ruck: http://dashboard-ci.tripleo.org/d/cEEjGFFmz/cockpit?orgId=110:45
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-undercloud-containers, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo- (3 more messages)10:48
arxcruzi whish our meetings be like this10:48
arxcruzless than 15 minutes10:48
arxcruzlol10:48
chkumar|rucksshnaidm: panda https://review.openstack.org/#/c/634109/ this one is good to go10:49
chkumar|ruckdo I need to fire a test job in rdo ci?10:49
quiquell|roverchkumar|ruck: this is all f28-stanadlone jobs exercise that ?10:50
chkumar|ruckquiquell|rover: yes10:50
quiquell|roverchkumar|ruck: I don't see standalone scenarios there10:50
quiquell|rovermarios: Do you know why we don't have stdln scenarios here ? https://review.openstack.org/#/c/63410910:51
sshnaidmchkumar|ruck, commented10:52
quiquell|roversshnaidm: for python3 we need it to specify binary or it will be UTF10:52
chkumar|ruckquiquell|rover: https://github.com/openstack-infra/tripleo-ci/blob/master/zuul.d/layout.yaml#L9 is missing from python-tripleoclcinet10:52
quiquell|roverchkumar|ruck: can you add it to your review ?10:53
quiquell|roverso we check ?10:53
quiquell|roverahh wait no10:53
quiquell|roverwe need f2810:53
chkumar|ruckyes10:53
quiquell|roverf28 is not passing10:53
quiquell|roverbut deploy is all good ?10:54
quiquell|roverhave fail at tempest10:54
chkumar|rucklet me check temepst stuff10:55
mariosquiquell|rover: not reall. i mean we should add at least for the tripleoclient/v1/tripleo_config/deploy/upgrade classes right10:56
marioswhich are the standalone ones being run in ci10:56
mariosquiquell|rover: you doing or should i ?10:56
quiquell|rovermarios: I have mix stuff we just needed f2810:57
mariosquiquell|rover: wut10:57
quiquell|rovermarios: but you can put a review for the layout of python-client and addd scnearios10:57
mariosquiquell|rover: ok will do10:57
quiquell|rovermarios: why only run stdl scenarios at deploy/upgrade ?10:57
chkumar|ruckmarios: is there scenario for f28?10:57
chkumar|ruck*standalone scenario10:58
mariosquiquell|rover: cos thats where the standalone commands live in client10:58
mariosquiquell|rover: we can add whatever there but i think those are a good start10:58
mariosquiquell|rover: like the files starting 'tripleo in tripleoclient/v110:58
quiquell|rovermarios: we don't exercise this https://review.openstack.org/#/c/634109 with standalone ?10:58
mariosquiquell|rover: yeah we do10:59
mariosquiquell|rover: that is what i mean tripleo_deploy.opy10:59
mariosquiquell|rover: see here https://github.com/openstack/python-tripleoclient/tree/master/tripleoclient/v110:59
mariosquiquell|rover: all the client commands live in these files10:59
mariosquiquell|rover: the standalone ones are in the files that have 'tripleo' in the start10:59
quiquell|rovermarios: ok we are in the same page11:00
mariosquiquell|rover: cool11:00
ykarelquiquell|rover, ack11:00
quiquell|rovermarios: so good that we add stld scenarios there11:00
quiquell|rovermarios: you put the review ?11:00
mariosquiquell|rover: yes good call. sure i can do that i thought we already established it. but if you prefer to go ahead11:00
mariosquiquell|rover: :D11:00
mariosman this is exhausting11:00
marios!11:00
quiquell|rovermarios: Let me put that, you have already did a lot of work there11:01
quiquell|rovermarios: thanks11:01
mariosquiquell|rover: ok no problem man just let me know if you don't get to it or need review11:01
mariosquiquell|rover: is pretty easy now we have the template11:01
mariosquiquell|rover: just change the standalonein tripleo-ci and include the template in client11:02
mariosdone11:02
chkumar|rucksshnaidm: done11:03
*** rfolco has joined #oooq11:05
chkumar|rucklet me update the commit msg also for rb also11:06
ssbarneaquiquell|rover: sshnaidm: we can easily merge dictionaries, see https://gist.github.com/ssbarnea/923bd01ad44662a73e69985e1aecaf7711:20
*** udesale has quit IRC11:21
ssbarneathere are few other similar ways to achieve the same11:21
sshnaidmssbarnea, there are a few even more complicated and non obvious solutions, I'm sure11:24
ssbarneaquiquell|rover: i even updated it to define environment at block level, so you would not have to redefine environment on each task!11:24
ssbarneaenvironment is inhereted!11:24
ssbarneayou can define it at task, block or playbook level. avoiding the need to repeat yourself.11:25
ssbarneaansible is quite nice, when you don't encounter a bug.11:25
sshnaidmquiquell|rover, I see you've already deployed disk.conf on promoter?11:28
quiquell|roversshnaidm: but we can remove and wait for infra update11:34
sshnaidmquiquell|rover, nm, just need to check it goes w/o errors11:34
quiquell|roverssbarnea: sshnaidm is in charge now of those reviews I am will do my ruck/rovering part now11:34
quiquell|roversshnaidm: ack now is good moment promotions are screw11:35
*** holser_ is now known as holser|burger11:46
chkumar|ruckquiquell|rover: I am looking at this one https://logs.rdoproject.org/00/627500/81/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-pike-branch/49a6d64/logs/undercloud/home/zuul/overcloud_validate.log.txt.gz#_2019-01-31_10_12_5212:01
chkumar|rucksshnaidm: quiquell|rover panda can we merge this https://review.openstack.org/#/c/627500/ ?12:02
chkumar|ruckrdo-pick failure unrelated to this patch12:02
quiquell|roverchkumar|ruck: is software-factory review merged ?12:03
chkumar|ruckquiquell|rover: yes yesterday12:03
chkumar|ruckquiquell|rover: rdo-check jobs are running12:03
chkumar|ruckall passed except pike one12:03
quiquell|roverchkumar|ruck: pike one I am open LP12:04
quiquell|roverchkumar|ruck: Can be related to cirros version12:04
chkumar|ruckyes12:04
quiquell|roverhttps://review.openstack.org/#/c/633214/12:04
quiquell|roverchkumar|ruck: https://bugs.launchpad.net/tripleo/+bug/181409012:05
openstackLaunchpad bug 1814090 in tripleo "Validate the overcloud failing at pike OVB" [High,Triaged] - Assigned to Quique Llorente (quiquell)12:05
sshnaidmchkumar|ruck, quiquell|rover seems like we need to add ports cleanup to our ovb cleanup script12:18
chkumar|rucksshnaidm: I think i can add that12:19
sshnaidmchkumar|ruck, we need to remove ports in DOWN, better comparing their age - more thatn 5 hours from creation date, like we do with stacks12:19
quiquell|roversshnaidm: ok make sense, weird it didn't affect us before is related to the new te-broker stuff ?12:20
sshnaidmquiquell|rover, kind of, yeah12:20
sshnaidmquiquell|rover, it took time for this ports to grow, also outages affect it12:20
quiquell|roversshnaidm: yep not best moment, chkumar|ruck you take care ?12:21
quiquell|roverchkumar|ruck: will continue looking for issues12:21
chkumar|ruckquiquell|rover: yes12:21
sshnaidmchkumar|ruck, quiquell|rover I'm deleting them right now, to free some IPs12:21
quiquell|roversshnaidm: ack thanks12:22
chkumar|ruckquiquell|rover: till then can we merge os_tempest last bits12:22
quiquell|roverchkumar|ruck: yep pass me the review will do12:24
chkumar|ruckquiquell|rover: https://review.openstack.org/62750012:24
sshnaidmchkumar|ruck, quiquell|rover wrt https://review.openstack.org/#/c/634213/1 - afaik cirros image presents on nodepool image already and we should take it from there, not to download12:25
sshnaidmchkumar|ruck, quiquell|rover the url was mostly for reproducer or manual runs with quickstart.sh12:25
quiquell|roversshnaidm: looks it's use by validate overcloud at pike12:26
sshnaidmcan we check that we don't download it, but take from /opt/cache/files/ ?12:26
quiquell|roversshnaidm: http://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/tree/roles/validate-simple/templates/overcloud-validate.sh.j212:26
quiquell|roversshnaidm: so just ust it as a fallback if it's not at /opt/cache/files ?12:26
sshnaidmquiquell|rover, yep12:27
quiquell|roversshnaidm: will do12:27
sshnaidmquiquell|rover, and it should be there in CI12:27
*** jpena is now known as jpena|lunch12:27
quiquell|roversshnaidm: fallback is already in place12:28
ssbarneasshnaidm: can we please merge https://review.rdoproject.org/r/#/c/18518/3 ?12:28
*** udesale has joined #oooq12:28
sshnaidmssbarnea, looking12:28
sshnaidmssbarnea, +w12:30
ssbarneasshnaidm: thanks.12:30
quiquell|roversshnaidm: fallback already in place https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/validate-simple/templates/overcloud-validate.sh.j2#L6412:31
quiquell|roverssbarnea: so cirros image is not in the nodepool image I suppose12:31
sshnaidmquiquell|rover, maybe they changed the path?12:33
quiquell|roversshnaidm: going to reproduce12:33
sshnaidmquiquell|rover, is it on ovb jobs only?12:34
quiquell|rover damn is ovb12:34
quiquell|roverwell I just need to see the nodepool image12:34
quiquell|roversshnaidm: yep ovb12:34
quiquell|roversshnaidm: what images do we use for OVB ?12:34
sshnaidmquiquell|rover, the same, from rdo12:34
quiquell|roversshnaidm: upstream-fedora-28 ?12:35
sshnaidmquiquell|rover, do you have a log?12:35
quiquell|rovercentos-7 I mean12:35
sshnaidmquiquell|rover, yeah, centos, but different label12:35
sshnaidmI don't know why different label, it should be the same as for standalone/multinode12:35
quiquell|roversshnaidm: what label ?=12:36
sshnaidmquiquell|rover, tripleo-ovb-centos-712:36
quiquell|roverdamn the CIX12:37
quiquell|roverarxcruz: how is the CIX now ? daily or what ?12:37
arxcruzquiquell|rover: mondays and wednesdays, it's on rhos team meetings calendar as CI Escalation Status | DFG:ALL12:38
quiquell|roverarxcruz: nice thanks12:38
quiquell|roversshnaidm: I see cirros image at /opt/cache/files12:42
quiquell|rovernodepool image12:42
sshnaidmquiquell|rover, great12:42
sshnaidmquiquell|rover, so, do you have a log from job? I'd like to see why it downloads..12:42
sshnaidmquiquell|rover, which version btw?12:43
quiquell|roversshnaidm: http://logs.rdoproject.org/45/560445/234/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-pike-branch/d250d6f/logs/undercloud/home/zuul/overcloud_validate.log.txt.gz12:43
quiquell|roversshnaidm: testing the bash snipet works fine12:44
quiquell|roversshnaidm: image have to be different12:44
sshnaidmquiquell|rover, so it should be /opt/cache/files/cirros-0.3.6-x86_64-uec.tar.gz12:45
quiquell|roversshnaidm: yep12:46
quiquell|roverupstream-infra-centos-7 have it12:46
quiquell|roversshnaidm: what the different with OVB images ?12:46
quiquell|roversshnaidm: this is the new OVB stuff ? ovb-manage-stack12:48
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001,  (2 more messages)12:48
quiquell|roversshnaidm: we are using plain CentOS-7-x86_64-GenericCloud12:51
quiquell|roversshnaidm: for OVB nodes12:51
quiquell|roversshnaidm: http://logs.rdoproject.org/45/560445/234/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-pike-branch/d250d6f/logs/undercloud/home/zuul/workspace/ovb/env-99378-base.yaml.txt.gz12:51
quiquell|roversshnaidm: is that right ?12:51
sshnaidmquiquell|rover, plain image is for overcloud nodes12:52
sshnaidmquiquell|rover, for undercloud node it's nodepool image12:52
sshnaidmquiquell|rover, it should be as regular nodepool image that we use in upstream12:52
quiquell|roversshnaidm: the validation is done to the overcloud at undercloud ahh ok12:53
quiquell|roversshnaidm: Can I reproduce that ?12:54
sshnaidmquiquell|rover, I think so12:54
*** rlandy has joined #oooq12:54
quiquell|roversshnaidm: I will need to put my cloud as secrets or something ?12:54
*** weshay has joined #oooq12:54
sshnaidmquiquell|rover, http://paste.openstack.org/show/744304/12:55
sshnaidmquiquell|rover, just copy all needed vars from tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001 and add cloud config12:56
quiquell|roversshnaidm: nodeset is tripleo-ovb-centos-712:56
weshayquiquell|rover, chkumar|ruck https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-fedora-28-ovb-1ctlr_1comp-featureset002-master-upload%0912:57
sshnaidmquiquell|rover, yes12:57
weshayif that job doesn't work.. we can't promote :)12:57
sshnaidmquiquell|rover, nodeset is already set in rdo config12:57
weshaybrb12:57
quiquell|roversshnaidm: I mean what images poiunt too this nodset12:57
quiquell|roverweshay: ack will look12:57
quiquell|roverweshay: skipped means dependant fail12:58
quiquell|roverweshay: upload job has not run at all12:58
sshnaidmquiquell|rover, nodeset: tripleo-ovb-centos-7-primary12:58
*** zbr has joined #oooq12:58
sshnaidmquiquell|rover, actually it's the same label: https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/nodesets.yaml#L68-L8313:00
quiquell|roversshnaidm: wher ewas the nodepool config for software factory ?13:01
sshnaidmquiquell|rover, nodepool/rdo-cloud.yaml13:02
sshnaidmlabels:13:03
sshnaidm          - name: tripleo-ovb-centos-713:03
sshnaidm            min-ram: 819213:03
sshnaidm            flavor-name: 'ci.m1.nodepool'13:03
sshnaidm            diskimage: upstream-centos-713:03
*** jjoyce has quit IRC13:03
sshnaidmquiquell|rover, exactly as for multinode and others13:03
quiquell|roversshnaidm: wait I have just mount upstream-infra13:03
quiquell|roversshnaidm: let me test exactly that13:04
*** jjoyce has joined #oooq13:04
weshayquiquell|rover, chkumar|ruck sorry I linked the f28 job13:04
weshaymeant https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-master-upload%0913:04
quiquell|roverweshay: they have fix the timeout13:04
weshaychkumar|ruck++13:05
weshayhttps://review.rdoproject.org/r/#/c/13943/13:05
hubbot1weshay: chkumar|ruck's karma is now 513:05
quiquell|roverweshay: last run is green so could be fixed now13:05
ssbarneasshnaidm: i am in process of trying a nick swap from ssbarnea to "zbr", to avoid confusions in our channels.13:05
chkumar|ruckweshay: it is related to undercloud install timedout13:05
chkumar|ruckweshay: fixed in the morning13:05
weshayah cool13:05
quiquell|roverweshay: promoter without diskpace we have clean up but we have to monitor13:05
sshnaidmssbarnea, zbr?13:05
chkumar|ruckweshay: we got retry limit due to max of ports down which sshnaidm just cleaned up13:06
marioschkumar|ruck: o/ so do i understand correctly that the master skip list always wins for the tempest tests like this https://github.com/openstack/tripleo-quickstart-extras/blob/b15450d98698814652db1cd9c00da510a11ec184/roles/validate-tempest/vars/tempest_skip_master.yml#L56 even if i have the plugin added in my job like https://review.openstack.org/#/c/633919/2/zuul.d/standalone-jobs.yaml ?13:06
weshaymarios, ya..13:06
weshayskip list always wins13:06
mariosk13:06
mariosthen we have to check13:06
marioslike e.g. barbican and the standalone job. the definition has it but the results don't"  http://logs.openstack.org/98/604298/205/check/tripleo-ci-centos-7-standalone/9c9a45f/logs/undercloud/home/zuul/tempest.log.txt.gz https://github.com/openstack-infra/tripleo-ci/blob/44a129efdaec1fd05acf5d91ecae6cafab7989b8/zuul.d/standalone-jobs.yaml#L38913:06
marioscos its in the skip list13:06
mariosi came across it just now for designate added a depends on for the skip list to check https://review.openstack.org/#/c/633919/ (https://review.openstack.org/634223 )13:07
ssbarneasshnaidm: yeah. (my family name sounds like this and we do not have any other nick starting with z, should be easy pick).13:07
weshayarxcruz, since you are not ruck/rover anymore.. can you add a storyless task to update the skip lists13:07
pandaquiquell|rover: how did you clean up ?13:07
weshayon sprint 513:07
quiquell|roverpanda: docker rmi13:08
ssbarneasshnaidm: i wonder what is the best way to let people know about such a change, as they know to look for "ssbarnea" (more worried about other channels), any idea?13:08
sshnaidmssbarnea, rename to zbr-ssbarnea13:09
mariosquiquell|rover: did you already post tripleo client standalone or should i13:09
pandaquiquell|rover: mmhh I see only 1 image and 27G in the docker dir.13:09
mariosquiquell|rover: or you planning on doing it/have time to do it13:09
*** zbr is now known as help13:09
chkumar|ruckquiquell|rover: weshay https://review.openstack.org/#/c/630925/ we need this one for manila tests failure13:09
*** help is now known as Guest5740613:10
arxcruzweshay: ack13:10
*** Guest57406 is now known as zbr|ssbarnea13:10
quiquell|rovermarios: damn I forgot man13:11
quiquell|roverpanda: humm maybe I miss the containers ? but we are not starting image up there13:11
quiquell|roverpanda: what can be those 27gb ?13:11
marios wdyt folks moving ci into new section https://review.openstack.org/#/c/633771 http://logs.openstack.org/71/633771/3/check/openstack-tox-docs/0ea6f8e/html/ thanks. no new content just rearranges sections in the index and adds standalone ci index @ /ci/index.html13:11
quiquell|roverpanda: intermediate layers or somethig ?13:11
mariosquiquell|rover: ok should i do it or you still gona wana?13:11
pandaquiquell|rover: directmapping messing13:11
weshayquiquell|rover, sshnaidm rlandy so.. the name resolution is not working completely on libvirt runs13:12
pandaquiquell|rover: and we have not time right now to test overlay2 storage13:12
mariosssbarnea: did my update address your concern in ps://review.openstack.org/#/c/633771 please revote when you have time thanks13:12
sshnaidmweshay, what do you have in /etc/resolv.conf?13:12
quiquell|roverpanda: if we are not bad with promotions maybe is not bad time to test overlay213:12
weshaysshnaidm, so we can hit yum mirrors etc13:12
quiquell|roverweshay: do we need promotions right now ?13:12
weshaybut ntp servers fail13:12
sshnaidmweshay, external ntp servers are not accessible from rh network, btw13:13
sshnaidmweshay, you need to use only rh ntp in rh net13:13
weshayquiquell|rover, it's getting old, but we have a couple days until we're red13:13
weshaysshnaidm, no shit13:13
sshnaidmwe had this problem with old reproducer too13:13
ssbarneamarios: yes, thanks. voted.13:13
weshaysshnaidm, I'm talking outside, we can't hit the public ntp servers13:13
sshnaidmweshay, why?13:14
weshayonly on libvirt13:14
quiquell|roverweshay: has passed mypending repro stuff to sshnaidm so I focus at ruck/rover13:14
weshayquiquell|rover, k13:14
mariosssbarnea: thanks13:14
quiquell|roverpanda: can we test overlay2 in two days ?13:14
mariosquiquell|rover: ok, so should i do it then ? :) /me firing up vim13:14
*** gkadam-bmgr is now known as gkadam13:15
quiquell|rovermarios: Will do don't worry your merge it and look at it like the lord of standalone CI13:15
*** zbr|ssbarnea is now known as zbr|ruck13:16
*** zbr|ruck is now known as zbr|rover13:16
weshaysshnaidm, have you run a full deployment w/ libvirt?13:16
pandaquiquell|rover: maybe less, but you need to attach another volume to the server, format it as overlay2, switch docker dir, and leave it there for a while. I can't certainly be worse than this situation, devicemapper performances are bad, and space is not freed correctly13:16
*** zbr|rover is now known as zbr|ssbarnea13:16
quiquell|roverpanda: also put that stuff at infra code13:16
mariosquiquell|rover: ack /me puts down vim13:16
weshaysshnaidm, maybe we need to inject more data into unbound?13:16
pandaquiquell|rover: yes, after you tested it ...13:16
quiquell|roverpanda: I think we have the volume already13:16
quiquell|roverpanda: I see diffent volume for docke rimages13:17
*** ssbarnea has quit IRC13:17
rlandyweshay: it's because we are defaulting to 8.8.8.813:17
weshayhttps://review.openstack.org/#/c/630925/13:17
weshayrlandy, we're not13:17
pandaquiquell|rover: yes, that saved you this morning, from having to regreate the entire server13:17
*** zbr has joined #oooq13:17
weshayrlandy, it's 127.0.0.113:17
quiquell|roverrlandy, weshay : Humm sshnaidm have a review for nameserver13:17
*** zbr|ssbarnea has quit IRC13:17
weshayit's unbound13:17
* weshay looks13:17
pandaquiquell|rover: when happened before, there was no way to interact with the server and remove containers.13:17
sshnaidmquiquell|rover, weshay yeah, for a year..13:18
weshaylol :)13:18
*** zbr is now known as zbr|ssbarnea13:18
weshaysshnaidm, point us at  :)13:18
sshnaidmweshay, https://review.openstack.org/#/c/565839/13:18
*** zbr|ssbarnea has quit IRC13:19
*** ykarel has quit IRC13:19
quiquell|roversshnaidm: at least now we have libvirt CI13:19
*** holser|burger is now known as holser_13:19
*** ykarel has joined #oooq13:19
quiquell|rovermarios: https://review.openstack.org/63422513:19
sshnaidmquiquell|rover, yeah, just need to stabilize it with force_tcg13:20
weshayzuul@subnode-1 ~]$ cat /etc/resolv.conf13:20
weshay; generated by /usr/sbin/dhclient-script13:20
weshaynameserver 127.0.0.113:20
weshay[zuul@subnode-1 ~]$13:20
quiquell|roversshnaidm: we are near, I think it just take lik 5 minutes or so13:20
weshayquiquell|rover, kind of13:20
*** zbr|ssbarnea has joined #oooq13:20
*** gkadam has quit IRC13:20
sshnaidmquiquell|rover, renewed a patch: https://review.openstack.org/#/c/633444/13:20
*** agopi is now known as agopi|brb13:21
*** zbr|ssbarnea has quit IRC13:21
mariosquiquell|rover: ack also needs tripleo-ci update13:21
mariosquiquell|rover: for the files:13:21
weshaycan we configure unbound to forward?13:22
weshayyup13:22
weshayunbound sucks13:24
*** zbr|ssbarnea has joined #oooq13:25
*** agopi|brb has quit IRC13:26
weshaysshnaidm, jump on my tmate13:26
quiquell|rovermarios: what files do we need for python-tripleoclient ?13:27
quiquell|rovermarios: maybe irrelevant-files ?13:28
*** jpena|lunch is now known as jpena13:31
weshayrlandy, is your libvirt deployment have 8.8.8.8?13:32
*** ykarel is now known as ykarel|away13:32
zbr|ssbarneaweshay: zbr|ssbarnea :)13:33
mariosquiquell|rover: tripleoclient/v1/tripleo_(deploy|config|upgrade)13:36
mariosand the heat launcher stuff13:36
quiquell|rovermarios: ack13:36
*** ykarel|away has quit IRC13:38
mariosbrb13:38
quiquell|rovermarios: https://review.openstack.org/63422513:44
quiquell|roverpanda: going to test overlay213:45
quiquell|roverpanda: I am following this https://docs.docker.com/storage/storagedriver/overlayfs-driver/#configure-docker-with-the-overlay-or-overlay2-storage-driver13:50
mariosquiquell|rover: ack will check in bit13:51
pandaquiquell|rover: yep13:51
quiquell|roverpanda: do I have to format it ?13:51
pandaquiquell|rover: is the promopter script stopped ?13:51
pandaquiquell|rover: yes, xfs currently does not have the d_type flag set to true13:51
quiquell|roverpanda: yep13:52
quiquell|roverpanda: also docker is stoped13:52
chkumar|ruckweshay: can we move it to done https://tree.taiga.io/project/tripleo-ci-board/us/554 ?13:52
chkumar|ruckfinal patch is in gates13:52
chkumar|rucksshnaidm: one question, I am able to get a list of port having down status but to compare for 5hours will I look at port show updated_at time?13:53
quiquell|roverpanda:13:53
quiquell|roverI see fstype=113:53
pandachkumar|ruck: usually we move to done when it's merged13:53
quiquell|roverpanda: https://paste.fedoraproject.org/paste/DTw14~L4ukqEcMgZF-fEHw13:54
chkumar|ruckpanda: yes, once it merged, we will move it to done may be by tomorrow13:54
quiquell|roverpanda: checking this https://docs.docker.com/storage/storagedriver/overlayfs-driver/13:55
sshnaidmchkumar|ruck, seems like that13:56
pandaquiquell|rover: why am I blind ?13:56
pandaquiquell|rover: it is, then it's even easier13:56
pandaquiquell|rover: well13:56
pandaquiquell|rover: you'll have to reformat anyway13:56
quiquell|roverI see fstype=113:56
chkumar|rucksshnaidm: ok then13:56
quiquell|roverpanda: I have to reformat13:57
pandaquiquell|rover: because the dangling free space, it's not enough to just rm -rf *13:57
pandaquiquell|rover: or at least was never enough for me13:57
pandaquiquell|rover: you can try to see if a rm -rf in /var/lib/docker frees space13:57
chkumar|ruckquiquell|rover: there we are using docker to push the images na? can we switch to buildah directly?13:58
quiquell|roverpanda: wait we have same volume for / and docker13:58
quiquell|roverI was wrong13:58
pandachkumar|ruck: in the promoter server we are just pullin, tagging and pushing13:58
pandaquiquell|rover: no, it's a different volume13:58
pandaquiquell|rover: /dev/vdb113:58
quiquell|rovervda I am blind now13:59
quiquell|rover:-)13:59
pandaquiquell|rover: wait13:59
pandaquiquell|rover: the promoter was not stopped13:59
rlandyweshay: ; generated by /usr/sbin/dhclient-script13:59
rlandynameserver 192.168.122.113:59
rlandynameserver 8.8.8.813:59
pandaquiquell|rover: master is promoting13:59
pandaquiquell|rover: ETOOMANYPROMOTIONS14:00
quiquell|roverjarlll14:00
quiquell|roverlet me kill stuff14:00
pandachkumar|ruck: and we can't switch the container build to buildah right now, not supported by kolla14:00
chkumar|ruckpanda: ok, got it :-)14:01
*** agopi|brb has joined #oooq14:01
quiquell|roverpanda: ok now14:02
quiquell|roverdamn channels14:02
quiquell|roverpanda: mkfs.xfs -n fstype=1 -f /dev/vdb114:02
*** agopi|brb is now known as agopi14:02
quiquell|roveris that it ?14:02
pandachkumar|ruck: but a partial refactor of the build process is in scope for this sprint14:02
pandaquiquell|rover: at this point I think fstype=1 is the default14:03
pandaquiquell|rover: I never had to specify it the first time14:03
rlandyweshay: unbound is run in zuul itself14:03
rlandymy run never gets there14:03
rlandyafter our setup14:03
weshayquiquell|rover, chkumar|ruck http://rhos-release.virt.bos.redhat.com:3030/rhosp14:06
quiquell|roverpanda: Storage Driver: overlay2 <- let's start promoter and see what we have14:10
panda\o/14:10
pandamaybe now we can even promote two release at a time14:10
pandadevicemapper was making everything explode14:11
chkumar|ruckweshay: let's hope for tomorrow having a master promotion14:13
*** ykarel|away has joined #oooq14:15
*** ykarel|away is now known as ykarel14:15
chkumar|ruckquiquell|rover: I am logging out, see ya tomorrow14:15
*** chkumar|ruck is now known as chkumar|out14:15
quiquell|roverchkumar|out: see you14:15
quiquell|roverweshay: damn my vpn is not working14:16
quiquell|roverweshay: promoter up with overlay214:22
weshayk cool14:24
quiquell|roverweshay: periodic-standalone-featureset052 critearia also match f28 isn't it ?14:25
quiquell|roverpanda: ^?14:26
*** skramaja has quit IRC14:26
*** vinaykns has joined #oooq14:27
pandaquiquell|rover: it should be changed to the full name of the job14:31
pandaquiquell|rover: I think it's already changed14:31
pandaquiquell|rover: should be already changed14:31
quiquell|roverpanda: I know but question is is the f28 is part of the promotion14:31
quiquell|roverpanda: looks like14:31
pandaquiquell|rover: yes it is14:32
quiquell|roverpanda: ok14:32
*** zul has joined #oooq14:32
quiquell|roverpanda: So we need chkumar|out fix for f28 ?14:32
quiquell|roverpanda: Unauthorized for url: https://trunk.registry.rdoproject.org/v2/tripleomasterF28/centos-binary-cron/manifests/tripleo-ci-testing14:33
pandaquiquell|rover: I know, we're going to fix it in this sprint14:33
zbr|ssbarneaquiquell|rover: sshnaidm : https://review.rdoproject.org/r/#/c/18649/ when you can, tx.14:33
quiquell|roverpanda: so we cannot change that from the criteria14:34
*** dtrainor has quit IRC14:34
quiquell|roverpanda: I think I am going to harcode something in the promoter14:34
quiquell|roverpanda: to skip f28 for now ?14:34
sshnaidmzbr|ssbarnea, who does really care about jjb now??14:35
zbr|ssbarneasshnaidm: i don't but i suspect there are few jobs already used there, right?14:36
sshnaidmzbr|ssbarnea, it's really hard to find something more neglected and not important than jjb now14:38
quiquell|roverpanda: do you have the review about fixing criteria around ?14:38
zbr|ssbarneasshnaidm: this change is not about jjb, is about getting it out of tox-linters.14:38
*** dtrainor has joined #oooq14:38
sshnaidmzbr|ssbarnea, that changes everything14:39
zbr|ssbarneasshnaidm: I do not want to run jjb when I lint my changes to ci-config. i guess it remained there for historical reasons.14:39
zbr|ssbarneasshnaidm: if you tell me that jjb is no longer in use and we can remove all jjb stuff from ci-config, I will be happy to do that.14:42
quiquell|roversshnaidm, panda: overlay2 https://review.rdoproject.org/r/1865414:44
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001,  (2 more messages)14:48
*** saneax has quit IRC15:12
chkumar|outquiquell|rover: weshay https://logs.rdoproject.org/43/13943/73/check/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-master-upload/9b8c324/logs/undercloud/home/zuul/overcloud_prep_images.log.txt.gz#_2019-01-31_14_20_4715:18
chkumar|outrecent failure on fs0215:18
sshnaidmquiquell|rover, works perfect: https://review.rdoproject.org/r/#/c/18558/15:19
quiquell|roversshnaidm: cool cool, let's go for it them15:19
quiquell|roverchkumar|out: ack will open a LP15:19
sshnaidmquiquell|rover, I think we need a more explaining commit message here: https://review.rdoproject.org/r/#/c/18654/15:20
chkumar|outsshnaidm: panda https://review.openstack.org/#/c/634109/ anything more needed on this15:21
pandachkumar|out: python-tripleoclient core votes ... we are not allowed.15:22
chkumar|outpanda: thanks! pinged tripleo core thanks :-)15:28
chkumar|outweshay: regarding fs20/fs21 test comparison script, where we will keep that script?15:30
quiquell|roverchkumar|out: has fail a testproject15:33
quiquell|roverchkumar|out: not in periodic15:33
quiquell|roverchkumar|out: https://softwarefactory-project.io/zuul/t/rdoproject.org/builds?job_name=periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-master-upload15:33
chkumar|outquiquell|rover: not sure tomorrow morning it will affect promotion jon15:33
quiquell|roverOk drop now have paste all in the ether15:33
chkumar|out*job15:33
quiquell|roverchkumar|out: depend on what they are testing in the review15:33
quiquell|roverchkumar|out: ahh silly me this is you testing it ?15:34
quiquell|roverchkumar|out: it's not timeout weird...15:34
quiquell|roverchkumar|out: going to open the LP but not mark it at promotion-blocker15:35
quiquell|roverchkumar|out: tomorrow we see15:35
chkumar|outquiquell|rover: sure /me out now :-)15:36
quiquell|roverchkumar|out: yep me too soon15:36
pandazbr|ssbarnea: muteeeee15:37
quiquell|roverchkumar|out: feels transitory15:37
pandazbr|ssbarnea: muuuuuuuute15:37
pandazbr|ssbarnea: thaaaaaanks15:37
*** chkumar|out is now known as invincible15:37
*** trown|outtypewww is now known as trown15:38
pandaarxcruz: muuuute15:38
pandaI still hear mac noises :(15:39
quiquell|roversshnaidm: Can you look at this OVB issue https://bugs.launchpad.net/tripleo/+bug/1814130 ?15:40
openstackLaunchpad bug 1814130 in tripleo "OVB fs002 job failing at IPMI power status" [High,Confirmed] - Assigned to Quique Llorente (quiquell)15:40
quiquell|roversshnaidm: thanks15:40
quiquell|roverdrop now15:40
quiquell|roverweshay: everything is in the etherpad https://review.rdoproject.org/etherpad/p/ruckrover-unisprint515:40
*** quiquell|rover is now known as quique|rover|off15:40
*** udesale has quit IRC15:40
*** ccamacho has quit IRC15:41
*** ccamacho has joined #oooq15:50
ykarelquique|rover|off, master promotion still failing16:03
ykarelweshay, panda  can u check ^^, multiple attempts failed with different reasons16:04
*** saneax has joined #oooq16:04
pandamarios: zbr|ssbarnea meeting tomorrow at 10 UTC for cross pollination ?16:04
zbr|ssbarneapanda: sure, please send calendar invite (mention your BJ id)16:05
mariospanda: ack16:06
sshnaidmpanda, can you vote please? https://review.openstack.org/#/c/633444/16:06
kopecmartinweshay, panda how can I add user stories to the sprint? i can't find any big "Add to the sprint" button :/16:07
pandakopecmartin: are you coming from an epic ?16:07
pandakopecmartin: I mean the user story is associated to an epic ?16:07
kopecmartinpanda, yes16:07
pandakopecmartin: ok, go to the epic, and add it with the big + button at the bottom16:08
rfolcoweshay, can I make us #653 a task a of #650 ?16:11
pandaykarel: looking16:11
kopecmartinpanda, i see only "create a relationship with" window, nothing which is sprint related16:14
pandakopecmartin: which one is the epic ?16:14
kopecmartinpanda, https://tree.taiga.io/project/tripleo-ci-board/epic/65716:15
weshayrfolco, ya16:18
pandakopecmartin: below just before the omment there should be a line called "related user stories" with a big + on the other side of the line16:18
kopecmartinpanda, there is, but i can set there only a relationship of a new or existing user story with the epic, no option for adding it to the sprint16:19
pandakopecmartin: ah you already have the us ?16:20
kopecmartinpanda, yes :)16:20
pandakopecmartin: 2 minutes bj, it's quicker16:21
kopecmartinpanda, sure https://bluejeans.com/398855864016:21
mariosweshay: you like it? (tripleo docs) moving ci into its own section https://review.openstack.org/#/c/633771 http://logs.openstack.org/71/633771/3/check/openstack-tox-docs/0ea6f8e/html/ thanks. no new content just rearranges sections in the index and adds standalone ci index @ /ci/index.html16:22
* weshay looks16:23
mariosrlandy: you've written a lot of the stuff there wdyt? ^16:23
weshayah nice16:23
weshaythanks marios16:23
mariosimo the info was lost in the contributor guide16:23
rlandymarios: +116:23
mariosand gives us a nicer place to add stuff/fixup our docs in general16:23
marios(like the new reproducer stuff)16:23
mariosk16:23
mariosthanks16:23
*** quique|rover|off is now known as quiquell|rover16:24
quiquell|roverykarel: looking16:24
rlandymarios: one comment16:25
rlandynot sure we want to add the old reproducer stuff16:25
quiquell|roverykarel: we have clean the promoter and switches to overlay216:25
mariosrlandy: sure but lets make it a followup so its clean this is just moving things in the index16:26
mariosrlandy: except we can't really remove the old stuff alltogether until we have the new stuff in place for docs. alternatively, whichever review adds the new reproducer docs should remove the old stuff? wdyt16:27
mariospanda: weshay rfolco i think we may need another task for standalones - for tracking fixes. e.g. the thing i mentioned earlier wrt the tempest weshay - as discovered today scen 3 is broken because we weren't running designate and when we do it fails http://logs.openstack.org/19/633919/2/check/tripleo-ci-centos-7-scenario003-standalone/326536d/logs/undercloud/home/zuul/tempest.log.txt.gz#_2019-01-31_14_15_19 https://review.openstack.org/#/c/63422316:31
ykarelquiquell|rover, so cleaning not done correctly?16:31
ykarelwhy switches to overlay?16:32
quiquell|roverykarel: it was could be the overlay216:32
mariospanda: weshay rfolco also other etc issues around the standalone have come up. latest today that we wanted to expand coverage to also include python-tripleoclient (quique posted https://review.openstack.org/#/c/634225 https://review.openstack.org/#/c/634231 )16:32
quiquell|roverykarel: after cleaning still 28gb were there, panda looked at it16:32
mariosweshay: panda rfolco so an 'existing jobs maintenance & fixes' story , maybe storyless task16:33
quiquell|roverykarel: looks like it's related to old overlay fs16:33
weshaymarios, so I think that is ruck/rover work16:33
pandaykarel: performance and bugs.16:33
weshayif we can pass off to quiquell|rover and Chandan.. I think it's fine16:33
mariosweshay: like 'fix designate scenario3' ... ack. i mean, ok once it is in promotions then it belongs to ruck/rover ?16:33
ykarelquiquell|rover, ok so issue not clear yet16:33
rlandymarios: fine by me16:33
mariosweshay: ok. i was calling that one out in particular cos its work needs doing to fix designate as the current specific example16:34
weshaymarios, I think we're calling scen1-4 done by the sprint team16:34
mariosweshay: ack ok thanks16:34
weshaywhere f28 container builds bugs would be tracked by the team because it's def in progress16:34
quiquell|roverweshay, marios: same we did with f28 stuff16:34
ykarelpanda, ok, so issue is still there, right16:34
weshayquiquell|rover, what do you want to do w/ my mariadb update to the cockpit?16:35
weshayinvestigate json datasource first?16:35
mariostime to go do other stuff with other people16:35
mariosi mean , you folks are awesome and all but16:35
quiquell|roverWesh16:35
quiquell|roverweshay: let's try if we can do i16:35
weshayk16:35
quiquell|roverweshay: without mariadb16:36
pandaykarel: quiquell|rover: besides space, there was anything else on the promoter logs ?16:36
quiquell|roverweshay: proxy python server that read LP and grafana connect to it16:36
weshayquiquell|rover, imho it may be nice to have a regular sql db as a datasource merged for future work as well16:36
weshayquiquell|rover, k16:36
quiquell|roverpanda: no that I saw16:36
pandaquiquell|rover: before you moved to overlay2, there was a master promotion running, did you get if it completed after ?16:37
weshayarxcruz, can you hand off the stacks issue to quiquell|rover as well?16:37
quiquell|roverweshay: could be for failed pronoters jobs and noop is good thing too16:38
arxcruzweshay: done already16:38
quiquell|roverweshay: every snapshot view graph16:38
ykarelpanda, ack, what i saw was space, registry tls error, and now a new error, and master not promoted yet16:38
weshayya.. I found the straight query to mariadb from graphana powerful16:39
quiquell|roverpanda: don't know just started the thing and went away16:39
quiquell|roverweshay: ack the we go to mariadb for LP16:39
quiquell|roverweshay: be sure records are limited and don't grow16:39
quiquell|roverweshay: we just one a snapshot of LPs16:40
weshayok.. so for lp I'll leave it to your expertise, but I do have plans to track more $things via mariadb16:40
quiquell|roverLike truncte/insert16:40
weshayquiquell|rover, ya.. bugs as sql keys makes sense to me16:40
quiquell|roverAck16:40
weshayand we'll have  a good record over time16:40
weshaywe could potentially add columns for reporter and maybe who fixed the bug too.. the second would be harder16:41
weshayykarel, quiquell|rover master current-tripleo should not promote now should it?16:42
* weshay see's scenario004 failing in the latest run16:42
ykarelweshay, 29th hash16:42
ykarelhash from 29th jan16:42
ykarelit's not promoting since 29 due to one or other issue in promoter16:43
weshayykarel, ah k.. cool16:43
quiquell|roverweshay: we are affected by f2816:43
*** fmount has joined #oooq16:43
quiquell|roverIs part of the criteria16:43
weshayoh... k.. I still think f28 needs to be promoted by hand.. maybe I'm wrong16:43
weshaypanda, would know16:43
weshayand it's not the end of the world if they are slightly diff hashes tbh16:43
weshayso +1 on getting master to promote16:44
quiquell|roverweshay: issue is criteria is toci jobtype16:45
quiquell|roverSo we cannot skip f2816:45
weshaywait.. why?16:45
weshayblue?16:45
quiquell|roverGive a few and we blue16:45
weshayhttps://bluejeans.com/4113567798 k16:46
quiquell|roverF28 shares same criteria as centoa in standalone16:46
quiquell|roverSame toci_jobtype16:46
pandaweshay: quiquell|rover f28 needs to be promoted by hand, but can be criteria for centos7 promotion16:46
pandaI'll join too16:47
quiquell|roverpanda: but we cannot skip it from promoter16:47
weshayquiquell|rover, why?16:47
quiquell|roverGive me 15 min16:47
fmountHi there, I'm running quickstart to make a deployment with controller>1 (-N ~/.quickstart/config/nodes/3ctrl_*) and I see ntp_server parameter is missing16:47
quiquell|roverF28 and c7 share same toci_jobtype16:47
pandaquiquell|rover: it should be fixed, let me look16:48
fmountweshay: I see you as owner of some commits on the node config files16:48
pandaquiquell|rover: at the configurations16:48
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001,  (2 more messages)16:48
quiquell|roverpanda: did you have a review to use job names?16:49
fmountI propose a fix for that kind of configuration to make ansible set that fact correctly16:49
*** bogdando has quit IRC16:50
weshayfmount, the param should not be in nodes config.. but what's the issue?16:50
weshayfmount, http://codesearch.openstack.org/?q=ntp_server&i=nope&files=&repos=tripleo-quickstart,tripleo-quickstart-extras16:50
fmountI think that param is featureset independent16:51
weshaycorrect16:51
fmounthttps://bugs.launchpad.net/tripleo/+bug/181411616:51
openstackLaunchpad bug 1814116 in tripleo "overcloud deploy blocked. Can't deploy overcloud due to missing NTP parameter in the overcloud-deploy.sh script" [Low,In progress] - Assigned to Francesco Pantano (fmount)16:51
pandaquiquell|rover: it wasnt' merged :( dammit16:52
pandaquiquell|rover: https://review.rdoproject.org/api/1778016:52
fmountit is more related to the overcloud configuration you will need to deploy16:52
quiquell|roverpanda: will work that our16:53
quiquell|roverIs a blocker now16:54
*** vinaykns has quit IRC16:54
rlandyweshay; quiquell|rover: you guys have some conflicting review comments :)16:54
pandaquiquell|rover: I think we are ready to merge it16:55
pandaquiquell|rover: we were just missing a vote, and it was lost in other tasks16:55
*** vinaykns has joined #oooq16:56
quiquell|roverpanda: do the rebase I will merge16:57
quiquell|roverrlandy: my only concern is restoring primary group of user16:57
*** ykarel is now known as ykarel|away16:57
rlandyquiquell|rover: right - so I removed all the docker work16:57
quiquell|roverOther stuff not very important16:57
weshayfmount, do you have a copy of the log?16:57
rlandyquiquell|rover: you guys also conflict on deps install16:58
fmountyes, it you take a look here:16:58
fmounthttps://bugs.launchpad.net/tripleo/+bug/181411616:58
openstackLaunchpad bug 1814116 in tripleo "overcloud deploy blocked. Can't deploy overcloud due to missing NTP parameter in the overcloud-deploy.sh script" [Low,In progress] - Assigned to Francesco Pantano (fmount)16:58
rlandyI actually agree with you about using quickstart deps install16:58
fmountyou can find the link to the log16:58
weshay    'Specify --ntp-server as parameter or NtpServer in '16:58
weshayInvalidConfiguration: Specify --ntp-server as parameter or NtpServer in environments when using multiple controllers (with HA).16:58
weshaySpecify --ntp-server as parameter or NtpServer in environments when using multiple controllers (with HA).16:58
weshay+ status_code=116:58
quiquell|roverrlandy: we have to find a way to CI from clean imagen and put as much a a possible at pre.yaml16:59
fmountthere is a task that set the --ntp-server {{ ntp_server }} if ntp_server is defined16:59
weshayfmount, you using rdo-cloud?16:59
pandaquiquell|rover: rebased16:59
rlandyquiquell|rover: trying to meet all comments - but hard when they conflict :)17:00
weshayfmount, ?17:00
quiquell|roverrlandy: let me look at it again17:00
quiquell|roverPandas +w17:01
rlandyalso weshay had a review to add more deps to be installed17:01
rlandyquiquell|rover: ^^17:01
rlandyin pre17:01
rlandywhich is fine17:01
weshayya.. that was python317:01
weshaythough17:01
quiquell|roverrlandy: this is all good17:01
weshayI abandoned17:01
quiquell|roverLet put stuff at pre.yaml so we CI it17:01
weshayrlandy, quiquell|rover imho.. I'm tenative about duplicating --install-deps17:01
weshayquiquell|rover, nack17:02
quiquell|roverweshay: do it17:02
weshaypre.yaml I think is too late17:02
rlandyweshay: I agree with quiquell|rover that creating a dep on quickstart is the wromg why to go17:02
quiquell|roverweshay: depends if we call pre.yaml in bash no17:02
weshayhappy to discuss.. but we're going to maintain two ways to setup for ansible?17:03
weshay3 actually?17:03
weshayquickstart.sh, old repro, new repro17:03
rlandyweshay: what was wrong with https://review.rdoproject.org/r/#/c/18545/7/playbooks/tripleo-ci-reproducer/pre.yaml?17:03
rlandyweshay: I am just trying to address all comments - but they are conflicting here17:03
quiquell|roverweshay: are we going to install ansible?17:04
*** kopecmartin is now known as kopecmartin|off17:04
weshayone it duplicates, and two it's 3-5 min in the run17:04
rlandyquiquell|rover: ack17:04
rlandywe have to install ansible17:04
weshayquiquell|rover, ansible libselinux and netaddr need to be installed up front17:04
rlandyin shell script17:04
rlandynetaddr not17:04
rlandythat should be taken care of17:04
weshayrlandy, I think we'll have to17:04
rlandynot an dircet ansible requirement17:04
weshaybut it's a quickstart.sh bug17:05
rlandythe other two are17:05
weshaywe also have the option of tearing out the install-deps feature in quickstart.sh into it's own script17:05
rlandywe need some consensus shere17:05
rlandyhere17:05
weshayand having both scripts call it17:05
quiquell|roverweshay: if we install them at pre.yall and call it in the bash is good17:06
fmountweshay: no I didn't use it, my point was more generic, no environment related, but I understand the design choice on put it there17:06
rlandyyou guys also disagree on docker17:06
quiquell|roverLet me check17:06
rlandygroup stuff17:06
weshayrlandy, on newgrp17:06
weshayyou mean right?17:06
rlandyshould be bj for 517:06
weshayI've been sitting in my blue17:06
rlandyI'd like to get comments addressed and ready for monday merge17:06
quiquell|roverweshay: I am ok with newgrp but we have to restore the primary group after17:06
rlandyjoining weshay sitting on blue17:07
weshayquiquell|rover, it doesn't replace man17:07
weshayit adds17:07
quiquell|roverIt replace17:07
quiquell|roverLet me show you17:07
weshayI've checked man17:07
quiquell|roverNeed 5 min for blue17:07
weshayworks for me17:07
quiquell|roverHumm weird17:07
weshayquiquell|rover, join my blue dude17:08
quiquell|roverI have too and docker was used to create files17:08
quiquell|roverJoining17:11
quiquell|roverweshay: paste the blue link17:13
quiquell|roverplease17:13
*** ccamacho has quit IRC17:13
pandaweshay: quiquell|rover : rlandy install script https://review.openstack.org/61369617:19
weshayvinaykns, here17:28
*** dsneddon has quit IRC17:29
sshnaidmweshay, can you vote please? https://review.openstack.org/#/c/633444/17:30
weshay[weshayutin@localhost ~]$ groups17:32
weshayweshayutin wheel libvirt docker17:32
weshay[weshayutin@localhost ~]$17:32
weshaysshnaidm, can you join my blue17:36
vinayknshey weshay: looks puppet is taking forever to configure on the host17:39
vinayknson UC17:40
vinayknsweshay: it got stuck at this point http://pastebin.test.redhat.com/70534917:41
*** dsneddon has joined #oooq17:45
vinayknsweshay: http://pastebin.test.redhat.com/70535217:45
weshayhttps://github.com/rdo-infra/ansible-role-tripleo-ci-reproducer/blob/master/playbooks/tripleo-ci-reproducer/pre.yaml#L3717:46
*** ccamacho has joined #oooq17:53
*** ykarel|away has quit IRC17:59
*** derekh has quit IRC18:00
*** ykarel|away has joined #oooq18:00
*** saneax has quit IRC18:05
*** jpena is now known as jpena|off18:08
*** ykarel|away has quit IRC18:13
*** holser_ has quit IRC18:17
*** trown is now known as trown|lunch18:29
*** dtantsur is now known as dtantsur|afk18:39
*** quiquell|rover is now known as quique|rover|off18:41
*** panda is now known as panda|off18:41
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001,  (2 more messages)18:48
weshayvinaykns, can you tmate?18:54
vinayknssure18:54
weshayhow does one execute bindep?18:54
vinayknsbut I changed the ntp server18:56
weshayvinaykns, ya.. that looked good :)18:59
*** vinaykns has quit IRC19:22
sshnaidmrlandy, fails for me because doesn't find docker-compose.. weird, because I have it19:24
*** fmount has quit IRC19:32
*** sshnaidm is now known as sshnaidm|off19:33
*** fmount has joined #oooq19:33
*** trown|lunch is now known as trown19:43
*** dsneddon has quit IRC19:46
rlandywierd19:47
rlandyinstalled how?19:47
rlandyas what user?19:47
*** holser_ has joined #oooq19:55
*** weshay has quit IRC19:56
*** rnoriega has quit IRC19:57
*** rnoriega has joined #oooq19:57
*** weshay has joined #oooq20:01
*** dsneddon has joined #oooq20:02
*** vinaykns has joined #oooq20:10
*** jfrancoa has quit IRC20:13
*** holser_ has quit IRC20:21
*** rfolco has quit IRC20:40
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024, tripleo-ci-centos-7-ovb-1ctlr_1comp- (2 more messages)20:48
rlandyweshay: sshnaidm|off: not sure we want to ditch pre altogether20:57
rlandywe will still need some stuff20:57
weshayya.. I wanted to go through each of the 7 or 8 sections there20:58
rlandylike creating docker config if it does not exist20:58
rlandyweshay: I am going through that all now20:58
rlandywhich steps can go where20:58
rlandyweshay; can show you what I have - if you're not too tired of talking about this20:58
* weshay wonders if the config is created by the rpm20:59
weshayhowever I guess since we're virtenv'ing it20:59
weshayya.. we may have to leave that21:00
weshaythat would require sudo21:00
rlandyoh21:00
rlandyI guess I should install21:00
rlandywe need to add the group if it is not there21:01
weshayrlandy, it's not created by the rpm21:01
rlandythe install does not do that21:01
rlandythe file is created via rpm21:02
rlandythe group is not adde21:02
weshay\/etc/docker/daemon.json  is not created by the rpm21:02
rlandyname: Create docker configuration file21:02
weshayhttps://github.com/rdo-infra/ansible-role-tripleo-ci-reproducer/blob/master/playbooks/tripleo-ci-reproducer/pre.yaml#L8821:03
rlandy^^ overwrites it instead of adding to it21:03
weshayit's not added by the rpm21:03
rlandyit is21:03
weshayrpm -q --whatprovides /etc/docker/daemon.jso21:03
rlandythat just overwrites it21:03
rlandy[zuul@subnode-1 ~]$ ls /etc/docker/daemon.json21:03
rlandy[zuul@subnode-1 ~]$ sudo cat /etc/docker/daemon.json21:03
rlandy{}21:03
weshayrlandy, sorry but you are incorrect21:03
weshaymaybe by the service21:03
* weshay checks21:03
weshayrun that command21:04
weshayrlandy, nope21:04
rlandy[zuul@subnode-1 ~]$  rpm -q --whatprovides /etc/docker/daemon.json21:04
rlandydocker-1.13.1-88.git07f3374.el7.centos.x86_6421:04
weshayhow do you have a higher version of docker21:05
weshayf28?21:05
rlandycentos21:05
weshayrlandy, ok.. on fedora it's not created21:05
weshay:)21:05
rlandyoh how fantastic21:05
rlandyalso21:06
rlandycomparing the two deps sets21:06
rlandywe are off21:06
weshayI think we were planning on using the python module anyway21:06
rlandyto install docker?21:06
weshayhttps://github.com/rdo-infra/ansible-role-tripleo-ci-reproducer/blob/master/playbooks/tripleo-ci-reproducer/pre.yaml#L4221:07
rlandythat is fine - will remove rpm21:07
rlandyso ...21:08
rlandyusing some of pre could work21:08
rlandyif we skipped the dangerous parts21:08
rlandywe will need the config21:08
weshayya21:08
rlandyand the service start and enable21:08
rlandyand the pip install21:08
weshayjust remove anything that requires sudo21:08
rlandy(for the moment)21:09
weshayeverything else should be idempotent21:09
rlandyha21:09
rlandyA LOT requires sudo21:09
rlandyeverything we need21:09
rlandyit's all become true21:09
weshayrlandy, let's chat21:10
rlandyweshay: can you give me 5? need some tea21:10
weshayah of course for tea21:11
*** holser_ has joined #oooq21:12
rlandyweshay: k - joining your blue21:14
*** trown is now known as trown|outtypewww22:02
rlandyweshay: we can set a var when we run pre from launcher playbook22:09
rlandysomething like in_launcher22:10
weshayrlandy, ok.. but it probably isn't required22:10
weshayif we check first22:10
weshayhowever I could see people still complaining22:10
*** ccamacho has quit IRC22:10
weshayso do as you see fit22:10
rlandyidk22:10
weshayif a user has $user in group docker + packages installed22:11
weshayI would expect pre.yaml to run through w/o passwordless sudo22:11
rlandywell ...22:12
* rlandy thinks about users22:12
rlandygerrit user22:12
rlandyrdo gerrit user22:12
rlandycurrent user22:12
rlandywe let people define gerrit user22:13
rlandyshould pre run with that user?22:13
*** agopi has quit IRC22:16
weshayrlandy, I can run as $USER whayutin to gerrit user weshayutin22:20
rlandyweshay: ack22:20
rlandyI need to add the ability to change all users in the script22:20
rlandy    user_pri_key: "id_rsa"22:21
rlandy    user_pub_key: "{{ user_pri_key }}.pub"22:21
rlandy    ssh_path: "{{ ansible_user_dir }}/.ssh"22:21
rlandy    upstream_gerrit_user: "{{ ansible_user }}"22:21
rlandy    upstream_gerrit_key: "{{ user_pri_key }}"22:21
rlandy    rdo_gerrit_user: "{{ ansible_user }}"22:21
rlandy    rdo_gerrit_key: "{{ user_pri_key }}"22:21
*** holser_ has quit IRC22:21
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset022, tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph- (2 more messages)22:49
*** holser_ has joined #oooq22:58
*** agopi has joined #oooq22:59
weshayrlandy, ok.. got it working w/ tq.. using https://review.openstack.org/#/c/634329/23:06
weshayrlandy, going to add a bindep to the role.. I think23:07
weshayand source install-deps.sh from your script23:07
rlandyweshay: cool - tqe review updated23:07
weshay# install required deps for a python virtual environment23:07
weshay    install_deps23:07
weshay    # setup the virtual environment23:07
weshay    install_virtual_env23:07
weshay    # activate the virtual environment23:07
weshay    . $OPT_WORKDIR/bin/activate23:07
rlandyweshay: adding reproducer role changes now23:07
weshay    # continue package installs with bindep23:07
weshay    install_package_deps_via_bindep23:07
weshayand those are steps to install the virtenv and packages23:07
weshayso that sould work consistently across projects23:08
*** rascasoft has quit IRC23:10
*** holser_ has quit IRC23:27
*** vinaykns has quit IRC23:28
rlandyweshay: https://review.rdoproject.org/r/#/c/18664/1/playbooks/tripleo-ci-reproducer/pre.yaml23:30
rlandy^^ review with config changes - will add checks to that23:30
weshayso the Create docker configuration file23:31
weshaywill fail the Sagi case?23:31
weshaydo you need to check the contents of the file first w/o sudo?23:32
weshayand only run the blockinfile if it doesn't exist right?23:32
weshayif the correct contents don't exist23:32
weshaymake it idempotent23:32
*** rascasoft has joined #oooq23:54

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!