Wednesday, 2018-12-19

« Tuesday, 2018-12-18 Index Thursday, 2018-12-20 »
*** brault has joined #oooq00:03
*** brault has quit IRC00:07
*** rlandy has quit IRC00:18
*** quiquell|off has quit IRC00:29
*** weshay_pto has quit IRC02:18
*** apetrich has quit IRC03:15
*** brault has joined #oooq04:05
*** brault has quit IRC04:09
*** chkumar|out is now known as chandankumar04:12
*** udesale has joined #oooq04:14
*** ykarel has joined #oooq04:22
*** ykarel has quit IRC05:24
*** saneax has joined #oooq05:31
*** ykarel has joined #oooq05:36
*** udesale has quit IRC05:45
*** saneax has quit IRC05:48
*** udesale has joined #oooq05:49
*** saneax has joined #oooq05:50
*** saneax has quit IRC05:51
*** saneax has joined #oooq05:53
*** saneax has quit IRC05:55
*** ratailor has joined #oooq06:04
*** ratailor has quit IRC06:04
*** ratailor has joined #oooq06:05
*** ratailor has quit IRC06:06
*** ratailor has joined #oooq06:06
*** brault has joined #oooq07:12
*** quiquell has joined #oooq07:13
*** brault has quit IRC07:14
quiquellmarios: o/07:19
quiquellmarios: workflowed .gitignore stuff, it's useful07:19
*** ratailor has quit IRC07:20
chandankumarmarios: quiquell https://review.openstack.org/#/c/612377/ please have a look at this review :-)07:20
marios_|ruckthanks quiquell was just doing a clearup07:20
*** ratailor has joined #oooq07:20
marios_|ruckand either merge or abandon by friday :)07:20
marios_|ruckack chandankumar07:20
*** ratailor has quit IRC07:20
marios_|ruckquiquell: scen2 in gate again07:20
*** ratailor has joined #oooq07:20
marios_|ruckpreparing another goat07:21
*** ratailor has joined #oooq07:21
quiquellmarios_|ruck: Will it be for this year the christmas present ?07:21
marios_|ruckhello ratailor07:21
quiquell:-(07:21
marios_|ruckquiquell: no i am gonna sacrifice to zuul07:21
marios_|ruckfor scen207:21
ratailormarios_|ruck, hello07:21
marios_|rucko/ hey07:21
marios_|ruck:)07:21
quiquellmarios_|ruck: I mean scen2 as present no the goat07:21
marios_|rucki though you were away you nick is join/drop ratailor07:21
quiquellmarios_|ruck: on little kitty dies at every recheck now also a goat07:22
ratailormarios_|ruck, I was online, but somehow couldn't join all channels. That's why restarted hexchat.07:22
ratailormarios_|ruck, now everything is settled. :)07:23
marios_|ruckack np sorry for noise :)07:23
marios_|ruckratailor: ^07:23
ratailornp .. :)07:23
marios_|ruckratailor: might be you don't have registered nick or something sometimes it kicks me too07:23
ratailormarios_|ruck, yea. it happened first time with me.07:24
*** holser_ has joined #oooq07:30
*** ykarel is now known as ykarel|lunch07:34
*** ccamacho has joined #oooq07:40
*** saneax has joined #oooq07:49
*** tosky has joined #oooq08:02
*** jfrancoa has joined #oooq08:07
*** gkadam has joined #oooq08:11
*** jtomasek has joined #oooq08:12
*** amoralej|off is now known as amoralej08:29
*** ykarel|lunch is now known as ykarel08:37
*** brault has joined #oooq08:40
*** brault has quit IRC08:41
*** brault has joined #oooq08:41
*** kopecmartin|off is now known as kopecmartin08:45
*** jpena|off is now known as jpena08:47
*** skramaja has joined #oooq09:10
*** brault has quit IRC09:24
*** chem has joined #oooq09:28
*** derekh has joined #oooq09:37
*** brault has joined #oooq09:45
*** ssbarnea|rover has quit IRC10:34
*** ssbarnea|rover has joined #oooq10:35
*** dsneddon has quit IRC10:37
ssbarnea|rovermarios: can you put a vote on https://review.openstack.org/#/c/626000/ ?10:43
marios_|ruckssbarnea|rover: k10:44
ssbarnea|rovermarios: thanks. do you know whom to ping to get the requirements review from merged https://review.openstack.org/#/c/626000/ ?10:54
marios_|ruckssbarnea|rover: no10:54
marios_|ruckmaybe ask in tripleo someone will know10:55
ssbarnea|roverwe were lucky with tripleo-qs as we do not have a check-requirements job, but other projects do have.10:55
ssbarnea|roververy quiet, everyone is probably shopping :D10:55
ssbarnea|rovermarios_|ruck: FYI, I will be on PTO traveling from tomorrow till Monday inclusive, no computer.11:01
quiquellor drunk11:01
marios_|ruckssbarnea|rover: thanks11:02
*** ccamacho has quit IRC11:05
*** dsneddon has joined #oooq11:07
*** udesale has quit IRC11:13
*** ccamacho has joined #oooq11:14
*** holser_ is now known as holser|lunch11:16
marios_|ruckreviews please rfolco panda sshnaidm|off if you have time thanks https://review.openstack.org/#/c/604768/911:30
marios_|ruckquiquell: i respectfully disagree ^11:31
quiquellmarios_|ruck: ack, just remove the standalone-upgrade from the review, we can do that at another review11:32
quiquellmarios_|ruck: I think you were right11:32
quiquellmarios_|ruck: Extra mile is not a -1Âthere11:32
marios_|ruckquiquell: i replied please see review. "why? this is fine to merge as is? what is broken here please be specific./"11:33
quiquellmarios_|ruck: answered11:35
marios_|ruckquiquell: ok, still unrelated change. what is broken with THIS review please?11:36
quiquellmarios_|ruck: "build" tag is not there11:36
marios_|ruckquiquell: ?11:37
*** jtomasek has quit IRC11:37
quiquellmarios_|ruck: best fix to remove "Âtags" all together from thestandalone-upgrade job11:37
quiquellmarios_|ruck: but you can clean the job and keep it as the previous code11:37
quiquellmarios_|ruck: And we fix the job at another review so we merge this11:37
marios_|ruckquiquell: oh you mean how it was in the previous version that i had and argued with you about, until i updated it to this version?11:37
quiquellmarios_|ruck: yep, sorry about that11:37
quiquellmarios_|ruck: Looks like changing the job need some clarifications and all11:38
marios_|ruckquiquell: still don't see what is broken on this review though you didn't answer the question11:38
marios_|ruckquiquell: what specifically about this review are you -1 about? you are saying tags but that is different review, not here11:38
marios_|ruckquiquell: what do you mean by 13:36 < quiquell> marios_|ruck: "build" tag is not there11:38
marios_|ruckplease quiquell ?11:38
quiquellmarios_|ruck: Maybe question is, overwriting "tags" from base job is not a bad thing now that we have extr_tags ?11:39
quiquellmarios_|ruck: look at the base job11:39
quiquellmarios_|ruck: base jobs have tag "build, stanadlone"Â but atyour review (and I think in the base) it only has "standalone"11:40
quiquellmarios_|ruck: but that was different in the original job definition11:40
marios_|ruckquiquell: please comment on the review if you want me to fix something thanks11:40
marios_|ruckquiquell: point to the thing you want me to fix11:41
marios_|ruckssbarnea|rover: re prog call today it overlaps the sprint end/retro11:41
marios_|ruckssbarnea|rover: for status i updated, just added the rocky promotions stuff11:42
pandapoint to the ceiling, point to the door, point to the window, point to the door.11:42
quiquellmarios_|ruck: ack, done11:43
marios_|ruckquiquell: ok. though i disagree it is broken. it is not broken11:44
*** jtomasek has joined #oooq11:45
marios_|ruckquiquell: i am the first person to receive feedback on reviews and the first person to fix things. but this is unreasonable IMO11:45
quiquellmarios_|ruck: ack, maybe the -1 there does not make sense since it was working, and was an extra mile11:45
quiquellmarios_|ruck: the stuff I ask for11:46
marios_|ruckquiquell: so it is broken now?11:48
marios_|ruckquiquell: i don't understand why you are being so steadfast about -1 this patch...11:49
quiquellmarios_|ruck: what I don't know if why it was not broken before it was not including "build" tag11:49
quiquellmarios_|ruck: ack, let's merge like this, will put a review to test using base tags11:51
marios_|ruckssbarnea|rover: master still blocked on that zaqar issue but sounds like panda has unblocked it earlier lets see11:53
marios_|ruckssbarnea|rover: ( https://bugs.launchpad.net/tripleo/+bug/1808349 )11:53
openstackLaunchpad bug 1808349 in tripleo "periodic-...-1ctlr-featureset010-master periodic-...-multinode-1ctlr-featureset037-updates-master jobs failing "Not found image: ... tripleomaster/centos-binary-zaqar" " [High,In progress] - Assigned to Emilien Macchi (emilienm)11:53
marios_|ruckwe didn't promote master for like 2 weeks...11:54
ykarelmarios_|ruck, master has more issues, os-ken and ovb, is ovb cleared now?11:55
ykarelfor os-ken i know it's not cleared yet,11:55
marios_|ruckykarel: ah yeah forgot about those :/11:55
marios_|ruckhttps://review.rdoproject.org/r/#/c/17852/ ykarel still ongoing but i don't have visibility into that one11:57
ykarelmarios_|ruck, that must be cleared by today11:57
ykarelwe are on it11:57
ykareland what about ovb, is the issue cleared there? i have not checked those today11:58
*** rfolco has quit IRC11:59
marios_|ruckykarel: not too bad on check at http://cistatus.tripleo.org/ but not a lot of green on the periodic yet11:59
*** rf0lc0 has joined #oooq11:59
ykarelmarios_|ruck, ack looks good today12:02
ykarellooks like someone have fixed it12:02
marios_|ruckykarel: i looked a bit yesterday when you pointed at it but they were all 'testenv-client' ... was wondering if related to the rdo cloud outage the previous night maybe12:03
marios_|ruckykarel: otherwise i am not aware of specific issues that were fixed there12:03
ykarelmarios_|ruck, ack12:05
*** jtomasek has quit IRC12:11
quiquellpanda: f28 upstream changes on the gates if all is ok they will be merge in hour and half12:20
*** ratailor has quit IRC12:23
panda\o/12:24
* panda bursts into dance12:24
quiquellpanda: without tempest12:25
marios_|ruckrf0lc0: gona miss the first part of retro have call conflict (program call)12:27
rf0lc0marios_|ruck, np12:27
*** jpena is now known as jpena|lunch12:29
quiquellssbarnea|rover: Do you have a minute to ask about molecule ?12:30
* panda bursts into dance a little less12:32
ssbarnea|roverquiquell: sure but in 15min12:32
quiquellssbarnea|rover: damn we have the meeting now12:33
ssbarnea|roverok12:33
ssbarnea|rovernow.12:33
ssbarnea|roverdo we have a bj ir here?12:33
quiquellssbarnea|rover: In half an hour we have12:34
quiquellssbarnea|rover: We can talk tomorrow12:34
ssbarnea|roverquiquell: we cannot, i will be on PTO12:34
quiquellssbarnea|rover: ack, np, let's talk when you are back it is not urgent12:35
rf0lc0quiquell, ssbarnea|rover marios_|ruck panda sshnaidm|off chandankumar arxcruz|next_yr.... add your retro cards please https://trello.com/b/0VFswmht/rdo-infra-retrospective?menu=filter&filter=label:Sprint2312:35
pandarf0lc0: do you know when sprint 3 will start ?12:47
chandankumarpanda: rf0lc0 it might be helpful for above question https://docs.google.com/spreadsheets/d/14df0Yvf7IPDjJy6dns9UDTpos7iEOs7PWtrAJV-kLHc/edit?usp=drive_web&ouid=10451958229787168574412:48
ssbarnea|roverpanda: sorry for the spam of ideas, but there were logged in rock/rover pad12:49
pandachandankumar: I think that was an example that used automated completion12:50
pandachandankumar: I doubt we'll have a sprint during company shut down12:51
pandammhh12:51
pandaupdated 2 days ago12:51
rf0lc0panda, will double check this info12:52
*** jtomasek has joined #oooq12:56
*** weshay has joined #oooq13:00
*** jpena|lunch is now known as jpena13:01
*** rlandy has joined #oooq13:03
*** ykarel is now known as ykarel|away13:04
*** ykarel|away has quit IRC13:12
*** jtomasek has quit IRC13:13
*** trown|outtypewww is now known as trown13:13
*** jtomasek has joined #oooq13:13
*** holser|lunch is now known as holser_13:13
ssbarnea|roverhttps://review.openstack.org/#/c/625896/ -- confirmed fix for loading qs ansible.cfg (permission fix)13:18
chandankumarssbarnea|rover: regarding infrared tempest plugin implementation https://github.com/openstack/openstack-ansible-os_tempest/blob/master/tests/os_tempest-overrides.yml#L20 and https://github.com/openstack/openstack-ansible-os_tempest/blob/master/vars/redhat-7.yml#L27 anything we can improve there13:23
*** ykarel|away has joined #oooq13:34
*** saneax has quit IRC13:42
*** ykarel|away is now known as ykarel14:00
quiquellpanda: the less needed of the refivew for f28 has merge and the most needed has fail :-(14:05
quiquellpanda: dose of reality there14:05
pandaquiquell: move to experimental!14:08
quiquellpanda: I am going to move to experimental myself14:08
*** skramaja has quit IRC14:13
*** ssbarnea|rover has quit IRC14:40
*** ssbarnea has joined #oooq14:40
marios_|ruckssbarnea|rover come back ! we decided you will be assisting weshay this sprint. thanks14:41
marios_|ruckpanda proposed i second it14:42
pandassbarnea: your PTO is being canceled14:43
marios_|ruckssbarnea: and your stock options14:43
pandassbarnea: and you'll have to go to the office too14:43
marios_|ruckssbarnea: (sorry about that)14:44
ssbarneayeah, for the next 4 days I will be remotee in Malaga.14:45
pandassbarnea: why is there a RH office in malaga ?14:45
marios_|ruckssbarnea: no sorry you have to cancel it you are now rover14:45
marios_|ruckagain14:45
*** udesale has joined #oooq14:50
marios_|ruckhttps://redhat.bluejeans.com/7661925373/14:56
marios_|ruckrlandy: ssbarnea ^ quick sync/?14:56
rlandyweshay: ^^ I'm joining you on ruck/rover ... do you have  time to meet to information transfer14:56
rlandymarios_|ruck: thanks - joining14:56
marios_|ruckhttps://review.rdoproject.org/etherpad/p/ruckrover-sprint2314:57
marios_|ruckrlandy: ^14:57
weshayrlandy, cool.. thanks for volunteering15:03
rlandyweshay: https://review.rdoproject.org/etherpad/p/ruckrover-sprint2415:08
*** rlandy is now known as rlandy|rover15:08
ssbarneachandankumar: i want to contribute the jinja2 validator to ansible-lint, i guess you have nothing against.15:14
rlandy|roverssbarnea: which patch are we watching for lint failures?15:18
ssbarnearlandy|rover: this needs to merge asap https://review.openstack.org/#/c/626000/15:19
rlandy|roverssbarnea: any idea who has core here?15:20
rlandy|roverI'd be happy to push the merge - not sure with whom15:21
ssbarnearlandy|rover: i did ping few people on #openstack-requirements but not visible results yet on the patch.15:22
* marios_|ruck out see you tomorrow folks bai15:22
*** marios_|ruck has quit IRC15:22
marios never liked that guy ^15:22
ssbarnearlandy|rover: there is something weird about requirements-check job - complains on tripleo-upgrades abut ansible not being listed on openstack/requirements ... which is really interesting....15:28
rlandy|roveron phone15:28
ssbarneais like someone added this job without even running it once on this repository because it does already have ansible listed on requirements.txt ...wtg?!15:28
quiquellOk dropping now, whoever goes on PTO enjoy !!!15:29
*** quiquell is now known as quiquell|off15:29
ssbarnealets take this to #tripleo15:29
*** ccamacho has quit IRC15:35
weshayrlandy|rover, https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic15:38
rlandy|roverssbarnea: sorry - back - looking15:41
chandankumarssbarnea: sure go ahead15:42
*** brault has quit IRC15:48
rlandy|roverssbarnea: we have mass timeouts and failures in promotion :(15:49
ssbarneawhat kind of timeouts, post ones?15:49
rlandy|roverhmmm ... looking through15:52
*** ccamacho has joined #oooq16:01
rlandy|roverssbarnea:  | RUN END RESULT_TIMED_OUT: [untrusted : git.openstack.org/openstack-infra/tripleo-ci/playbooks/tripleo-ci/run-v3.yaml@master]16:06
rlandy|roverso not post16:06
rlandy|roverhttps://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/74a0e5d/job-output.txt.gz#_2018-12-19_14_37_13_85060916:06
rlandy|roverthat one familiar to you?16:06
rlandy|roverhttps://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/74a0e5d/logs/undercloud/home/zuul/overcloud_prep_images.log.txt.gz16:07
ssbarnearlandy|rover: nope but ini theory it could be generated by the same bug that affects post.16:07
rlandy|roverback with this16:07
ssbarneai think that any task that could take more than 30min should only be run with ansible async.16:08
rlandy|roverironic failure16:11
*** ykarel is now known as ykarel|away16:19
rlandy|roverknown ironic issue - again16:25
rlandy|roverboth 001 and 002 fail on it16:26
rlandy|roverchandankumar: is this a known tempest failure? https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset019-master/3328b0a/logs/undercloud/home/zuul/tempest.log.txt.gz#_2018-12-19_13_00_5916:31
* rlandy|rover catches up16:31
chandankumarrlandy|rover: reason is here https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset019-master/3328b0a/logs/subnode-2/var/log/extra/errors.txt.gz#_2018-12-19_13_00_50_38316:33
chandankumarrlandy|rover: look for this id 20bea8ec-3403-4801-b0cd-abfde78b4a4816:33
chandankumarrlandy|rover: https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset019-master/3328b0a/logs/subnode-2/var/log/containers/neutron/server.log.txt.gz#_2018-12-19_13_00_45_90616:35
*** udesale has quit IRC16:35
rlandy|roverchandankumar: thanks - I am just trying to discover what we do and do not already know about16:35
rlandy|roverand map that to what was in the etherpad from last sprint16:35
chandankumarrlandy|rover: anything fails look for ids16:36
chandankumarrlandy|rover: then go to https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset019-master/3328b0a/logs/undercloud/var/log/extra/errors.txt.gz16:36
chandankumarrlandy|rover: if multinode then go to https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset019-master/3328b0a/logs/subnode-2/var/log/extra/errors.txt.gz16:37
rlandy|roverok16:37
chandankumarrlandy|rover: both generally take you to the right place where wrong has happened16:37
rlandy|roverweshay: ^^ I am going through the master promotion errors. if you already know about these failures/have done this. let me know16:39
rlandy|roverleaving notes on etherpad16:39
*** jtomasek has quit IRC16:39
ssbarnearlandy|rover: weshay : the fix for not triple skipping loading of ansible.cfg https://review.openstack.org/#/c/625896/ - passed CI, tested output, no more warnings.16:41
weshayssbarnea, hey.. qq16:42
weshayre: the du post task16:42
weshaydo you have a second to fill me in on that?16:42
ssbarneaweshay: yep, du was *not* the cause.16:42
weshayphew16:43
* weshay feels better16:43
weshayI know that failed oddly from time to time.. but should never fail the job16:43
rlandy|roverthanks16:43
ssbarneanow we think it was zuul, and the patch will merge in few minutes, is waiting the gate: https://review.openstack.org/#/c/626138/16:44
ssbarneaweshay: still, the (wasted time) around collect logs should go in vain, I ended up with two reviews to improve collect logs, which should go in. (speed, verbosity and maintenance)16:45
weshayssbarnea, do you really want to hardcode /home/zuul?16:46
ssbarneaweshay: look at previous code, i am not making things worse than before. i am against hardcoding stuff, but also against fixing unrelated problems in a change.16:47
weshayah k16:47
*** dsneddon has quit IRC16:47
weshayk.. /me looks through this16:47
*** chandankumar is now known as chkumar|vanished16:50
weshayssbarnea, ok.. minor request on that review.. basically just start a patch to address quique's comments on top of this patch and we'll get the this one merged16:53
weshayjust don't want to loose the thread of his comments because they are good16:53
sshnaidm|offweshay, I shoot it: https://review.rdoproject.org/r/#/c/17895/16:55
weshaywow.. so that is live nowish?16:56
* weshay looks at jobs16:56
rlandy|rover2018-12-19 02:58:05 | 2018-12-19 02:57:21Z [overcloud.Compute]: CREATE_FAILED  CREATE aborted (Task create from ResourceGroup "Compute" Stack "overcloud" [62f20031-64f8-43d1-81b5-b4d26878ac45] Timed out)16:57
rlandy|roverdeployment timeout now16:57
*** kopecmartin is now known as kopecmartin|off16:57
weshayrlandy|rover, you seeing that review?16:59
* weshay watching a new job 16:59
rlandy|roverweshay: that review?16:59
weshayhttps://review.rdoproject.org/r/#/c/17895/116:59
weshayovb-manage.. ya.. this looks live17:00
weshaynice sshnaidm|off /me watching17:00
weshayprint ovb args17:00
rlandy|roverlive17:00
weshaycreate stack17:00
*** dsneddon has joined #oooq17:00
weshayrlandy|rover, watching https://review.rdoproject.org/zuul/stream/e638a314d8524c7db9dce14078f2d480?logfile=console.log17:00
rlandy|rovercreated and +2 and merged w/o review?17:01
weshay2018-12-19 17:00:52.123304 | primary |   os_password: '******'17:01
weshayrlandy|rover, that joker sshnaidm|off self merged it :)17:01
rlandy|roverweshay: I see that17:01
* weshay helps my daughter . .sec17:01
weshayrlandy|rover, so knowing how the stacks are cleaned up is important here17:03
weshayI'll have to read through taiga17:03
rlandy|roveryeah - watching jobs17:03
weshayhrm.. also want to confirm I don't see the te-broker getting hit17:04
rlandy|roverweshay:just major change to merge without saying anything17:04
rlandy|roveror asking for review17:04
rlandy|roverall while being out for the day17:05
* rlandy|rover checks tenant17:05
weshayrlandy|rover, seems to be working, let's hold off on judgement until we can chat w/ Sagi..17:08
weshayrlandy|rover, are you able to ssh to the te-broker?17:08
rlandy|roverI am still trying to connect to the tennat in rdocloud17:08
weshayhttp://38.145.33.166/testenv-worker.log17:09
weshaynow I got it17:09
rlandy|rovergateway timeout17:10
rlandy|roverweshay: https://review.rdoproject.org/zuul/stream/ab70f41978dc4046a9169a61932b5db4?logfile=console.log17:12
weshayrlandy|rover, hrm.. should we blue?17:13
rlandy|roverI have no idea what is happening here17:13
rlandy|roversure17:13
weshayin my blue17:14
weshayhttps://review.rdoproject.org/zuul/stream/e638a314d8524c7db9dce14078f2d480?logfile=console.log17:14
weshaysshnaidm|off, you available?17:18
weshaysshnaidm|off, there is something up w/ regards to stack delete17:18
sshnaidm|offweshay, yeah17:19
sshnaidm|offweshay, what's that?17:19
weshayhttps://bluejeans.com/u/whayutin/17:19
weshayrlandy|rover, | 58c88572-7955-4c77-8966-89cf2cccacab | baremetal_53733 | DELETE_FAILED      | 2018-12-19T17:04:09Z | 2018-12-19T17:12:53Z |17:20
sshnaidm|offweshay, link?17:20
weshaysshnaidm|off, https://review.rdoproject.org/zuul/stream/ab70f41978dc4046a9169a61932b5db4?logfile=console.log17:21
weshayhttps://review.rdoproject.org/zuul/stream/ab70f41978dc4046a9169a61932b5db4?logfile=console.log17:22
rlandy|roverTriggered by: https://review.openstack.org/62519117:22
weshaysshnaidm|off, http://paste.openstack.org/show/737744/17:24
*** derekh has quit IRC17:44
*** gkadam has quit IRC17:46
*** jpena is now known as jpena|off17:54
*** trown is now known as trown|lunch17:56
*** rascasoft has joined #oooq18:02
*** rascasoft_ has joined #oooq18:03
*** rascasoft_ has quit IRC18:03
ssbarneaweshay: rlandy|rover : i need a bit of help debugging virt-resize failure with reproducer: see https://seashells.io/v/482J9yNQ18:13
ssbarneaguess what: if I run the same command manually, it works. in fact the undercloud-resized.qcow2 is created but supermin fails with this askward error when run from ansible.18:15
ssbarneaall files owned by root, repro run as root too.18:15
*** holser_ is now known as holser|eod18:26
ykarel|awayssbarnea, looks like issue with SUPERMIN variables18:30
ykarel|awaylooks like u r trying libvirt nodepool one18:30
ykarel|awayif yes try without these supermin variables: http://git.openstack.org/cgit/openstack/tripleo-quickstart/tree/playbooks/libvirt-nodepool.yml18:30
* ykarel|away leaving, too late, can catch up tomorrow if i got something wrong18:32
*** saneax has joined #oooq18:33
*** amoralej is now known as amoralej|off18:39
*** ykarel|away has quit IRC18:43
*** saneax has quit IRC19:01
*** trown|lunch is now known as trown19:06
weshayssbarnea, run the customize once by hand19:10
weshayfor some untold reason that fixes it19:10
weshaysec.. relocating19:10
weshayrf0lc0, all the jobs depend on the container builds now19:13
weshaysshnaidm|off, rlandy|rover stacks look clean19:14
rlandy|rovergood19:14
weshayssbarnea, are you hitting that w/ https://review.openstack.org/#/c/625648/ ?19:22
rf0lc0weshay, rlandy|rover panda https://review.rdoproject.org/r/#/q/status:open+branch:master+topic:f28_promotion19:23
*** sshnaidm|off has quit IRC19:24
rf0lc0I could be stealing. I could be killing. But I am just begging your review.19:24
rf0lc0panda, https://dilbert.com/strip/1996-02-2819:25
rf0lc0old but gold19:25
weshayrlandy|rover, https://review.openstack.org/#/c/625896/19:27
*** sshnaidm has joined #oooq19:27
*** gouthamr_ is now known as gouthamr19:38
rlandy|roverUnable to freeze job graph: Decryption failed. - weord19:38
rlandy|roverweird19:38
weshayssbarnea, did you are marios see anything related to an undercloud install error..19:39
weshay    "2018-12-19 15:46:51,387 INFO: 23626 -- Finished processing puppet configs for heat_api_cfn",19:39
weshay    "2018-12-19 15:46:51,388 ERROR: 23618 -- ERROR configuring heat_api",19:39
weshay    "2018-12-19 15:46:51,390 ERROR: 23618 -- ERROR configuring iscsid"19:39
*** brault has joined #oooq19:47
*** brault has quit IRC19:51
rlandy|roverhmmm19:52
weshayrlandy|rover, any idea where the initial container config get's logged?20:11
rlandy|roverweshay: initial container config?20:16
weshayya20:17
rlandy|roverlooks like jobs are failing the retry limit20:17
weshaywhen launched but mounting the config files from puppet20:17
rlandy|roveron the undercloud20:17
weshaythe retry limit on the ovb stack?20:17
rlandy|roverhmmm.. I know where that is in the lcal reproducer20:18
weshayrlandy|rover, are you talking about ovb w/ regards to the retry limit?20:19
weshay2018-12-19 17:32:31.397983 | primary | heatclient.exc.HTTPBadRequest: ERROR: Request limit exceeded: You have reached the maximum stacks per tenant, 100. Please delete some stacks.20:19
weshayrlandy|rover, I think we have to let things settle a bit more20:19
rlandy|roveryeah .. I just need to think where it is20:19
weshayk20:20
rlandy|roverit's on the provider20:20
rlandy|roverwhy is it still overrunning quota?20:20
weshayrlandy|rover, it's not20:21
weshaythose jobs launched 3hrs ago20:21
weshayrlandy|rover, let's let it settle a bit more20:21
weshayopenstack stack list | wc -l                                                       master ✱ ⚡ thinkdoe ⌚ 13:19:5620:21
weshay6820:21
weshayrlandy|rover, let's revaluate ovb tomorrow morning20:22
rlandy|roverweshay: retry limit was just changed ... https://review.rdoproject.org/r/#/c/17941/1/roles/ovb-manage/tasks/ovb-create-stack.yml20:27
weshay+120:28
rlandy|roveryou want me to up that?20:28
rlandy|roverweshay: ^^20:28
rlandy|roverweshay: also - wrt your question on initial container config - can you explain more - undercloud?20:30
weshaysec. working w/ Emilien20:31
weshayrlandy|rover, https://bugs.launchpad.net/tripleo/+bug/180354420:31
openstackLaunchpad bug 1803544 in tripleo "unable to find user root: no matching entries in passwd file" [Critical,Triaged]20:31
weshayhttps://github.com/containers/libpod/pull/197820:31
ssbarneaweshay: replied to your comment https://review.openstack.org/625997 -- review quickly because without fixing the linting nothing will merge to tripleo-upgrades.20:32
rlandy|roverttps://github.com/openstack/tripleo-heat-templates/blob/master/common/deploy-steps-tasks.yaml#L41920:36
*** weshay is now known as weshay|ruck20:37
rlandy|roverweshay|ruck: ^^ are you looking for the above?20:38
ssbarneaweshay|ruck: do we all get tshirts with "recheck" for xmas?20:38
weshay|ruckrlandy|rover, ya.. I think that is right20:39
weshay|ruckinitially looking20:39
rlandy|roverssbarnea: I want a 'containers linux' shirt like you have20:40
rlandy|roverremote workers get no swag :(20:40
weshay|ruckrlandy|rover, so.. the gate is unstable because of the podman rpm20:41
weshay|ruckthere are fatal errors on the previous and current version20:42
weshay|ruckand they are both in the gate atm20:42
weshay|ruckjust to screw w/ us20:42
rlandy|rovercharming20:42
rlandy|roverok - so are we getting the disable patch?20:42
ssbarneaweshay|ruck: rlandy|rover : to make tempest sendmail failures traceable: https://review.openstack.org/#/c/625915/20:42
weshay|ruckrlandy|rover, http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22ERROR%20configuring%20iscsid%5C%22%20AND%20tags:%5C%22console%5C%22%20AND%20voting:120:42
weshay|ruckssbarnea, I hope that is not failing jobs ya?20:43
weshay|ruckk.. ignore_errors: true20:43
weshay|ruckssbarnea, there is so much that is fucked about CI upstream20:43
ssbarneaweshay|ruck: nope, not failing because task has ignore. the issue was that it failed in various ways and it was not possible to make a query to trace this task failure. now we have a clear message to look for.20:44
weshay|ruckmostly the shift of packages on the daily20:44
weshay|ruckor even at the same time20:44
ssbarneai think that CR message should explain it.20:44
rlandy|roverwant me to w+ that?20:44
weshay|ruckssbarnea, rlandy|rover although it may never happen.. /me dreams of a stable set of yum repos like internal builds have20:44
rlandy|roverinternal is not as stable as you dream20:45
ssbarneaweshay|ruck: no they are not stable there.20:45
weshay|ruckssbarnea, I know I'm talking about something completely different20:45
weshay|ruckssbarnea, what moves in a puddle?20:45
weshay|ruckbaseos?20:45
rlandy|roverweshay|ruck: what you install it on20:45
weshay|ruckrlandy|rover, ?20:46
rlandy|roverdownstream does fail20:46
rlandy|roverdeps20:46
weshay|ruckrlandy|rover, ya.. but the puddle always has the same set of rpms20:46
weshay|ruckminus rhel20:46
rlandy|rovercorrect underlying fails20:47
weshay|ruckwe litterlly have two different versions of podman causing different bugs in the gate at the same time20:47
rlandy|roveralthough downstream does simplify being on on rhel20:47
rlandy|roverhere we have multiple possible issues20:47
ssbarneathis looks bit funny:  ERROR: Ignoring Errors20:48
weshay|ruckheh20:48
weshay|ruckssbarnea, when do you sleep?20:49
rlandy|roverweshay|ruck: so we're heading up 77 stacks in create_complete20:49
weshay|ruckrlandy|rover, I read that as a good thing ya?20:50
rlandy|roverI guess so20:50
weshay|ruckor do you think we have old?20:50
*** jfrancoa has quit IRC20:50
rlandy|roveroldest is only up 3 hours20:50
rlandy|rover2018-12-19T17:00:48Z20:50
rlandy|rover[rlandy@rlandy workspace]$ date -u20:50
rlandy|roverWed Dec 19 20:50:50 UTC 201820:50
weshay|ruckrlandy|rover, ya.. I think we may have stabilized,20:50
weshay|rucklet's ignore it until tomorrow20:51
rlandy|roveralmost 420:51
rlandy|roverweshay|ruck: k - keeping watch on that20:51
ssbarneaweshay|ruck: i am going offline now, i have a dead horse to beat in RDR2 before going to sleep.20:52
weshay|ruckssbarnea, thanks for the help man20:52
weshay|ruckssbarnea++20:52
weshay|ruckhttps://review.openstack.org/#/c/624420/20:53
weshay|ruckYFI20:53
weshay|ruckFYI20:53
rlandy|rovergood - should help us20:55
weshay|ruckrlandy|rover, why doesn't this job create a reproducer? https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master/958a074/logs/20:58
rlandy|roverhttps://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master/958a074/job-output.txt.gz#_2018-12-19_14_28_06_78170020:59
rlandy|roverfailure in collect logs20:59
*** ssbarnea has quit IRC21:00
weshay|ruckk21:00
rlandy|roverweshay|ruck: here is the error ...21:01
rlandy|roverhttps://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master/958a074/logs/quickstart_collect_logs.log21:01
rlandy|roverno line number sorry21:02
rlandy|roverAnsibleError: An unhandled exception occurred while templating '{{ (lookup('file', artcl_collect_dir + '/zuul_console.json') | from_json) }}'. Error was a <class 'ansible.errors.AnsibleError'>, original message: An unhandled exception occurred while running the lookup plugin 'file'. Error was a <class 'ansible.errors.AnsibleError'>, original message: could not locate file in lookup: /home/zuul/workspace/logs/zuul_console.js21:02
rlandy|roveron21:02
*** tosky has quit IRC21:02
rlandy|roverhmmm21:02
rlandy|rovercan't be just this job21:03
rlandy|rovermust have been like that for a while, no?21:03
weshay|ruckrlandy|rover, don't know21:07
weshay|ruckthere is soooo much whacky stuff going on21:07
rlandy|roverweshay|ruck: I am looking into that21:08
rlandy|roverwe can divide and conquer here21:08
weshay|ruckaye..21:10
* weshay|ruck running old reproducer on fs00121:10
weshay|ruckto test the image prep21:10
sshnaidmweshay|ruck, maybe te-broker limited stacks count too, need to check..21:11
weshay|rucksshnaidm, ya.. it's looking fairly good atm21:11
weshay|ruckrf0lc0, I need the agenda for tomorrows meeting21:12
weshay|ruckrlandy|rover, k.. fs001 runnig21:14
weshay|ruckrunning21:14
rlandy|roversshnaidm: my new reproducer is failing with ovb - will check with you tommorrow21:15
rlandy|rovermaybe something updated21:15
weshay|ruckrlandy|rover, hrm.. reproducer fails on +(/opt/stack/tripleo-ci/toci_gate_test.sh:274): sudo pip install gear21:16
sshnaidmrlandy|rover, yeah, it's expected21:16
sshnaidmrlandy|rover, actually with second thought - no, not expected21:16
weshay|ruckguess we need put the peddle to the metal21:16
sshnaidmrlandy|rover, reproducer uses the ovb-manage-stack role, nothing should change there21:17
weshay|ruckdang.. maybe the whole squad should focus on the reproducer21:17
weshay|ruckthis sprint21:17
weshay|ruckgoing into 2019 w/o a working tool sounds bad21:17
rf0lc0weshay|ruck, k, will create the agenda first hour tomorrow morning. The board panda will create21:18
rlandy|roverweshay|ruck: will see if I can get new reproducer working with sshnaidm tomorrow21:18
* rlandy|rover is looking at why we don;t even have a reproducer created in logs21:18
rlandy|roverbigger problem21:19
*** brault has joined #oooq21:19
weshay|ruckrlandy|rover, ah.. got it fixed21:21
weshay|ruckold stuff running, I'll put up a patch21:21
rlandy|rovercool21:21
rlandy|roverstacks are down to 6421:21
rlandy|roversshnaidm: ^^ looks good so far21:21
sshnaidm\o/21:21
weshay|ruckrlandy|rover, sshnaidm btw.. old reproducer is calling testenv-client21:21
weshay|ruckI didn't think it did that21:22
rlandy|rover could not locate file in lookup: /home/zuul/workspace/logs/zuul_console.json21:22
sshnaidmweshay|ruck, it shouldn't21:22
rlandy|roverour csreen scrape can probably go21:22
weshay|ruckSuccessfully installed extras-1.0.0 gear-0.12.0 lockfile-0.12.2 pbr-5.1.1 python-daemon-2.2.021:22
weshay|ruck+(/opt/stack/tripleo-ci/toci_gate_test.sh:279): NETISO_ENV=multi-nic21:22
weshay|ruck+(/opt/stack/tripleo-ci/toci_gate_test.sh:282): ./testenv-client -b 192.168.100.250:4730 -t 18000 --envsize 4 --ucinstance 2aae4cbb-2e75-4a7a-b4c8-0f3f6c6ab94c --net-iso multi-nic -- ./toci_quickstart.sh21:22
weshay|ruck+(/opt/stack/tripleo-ci/toci_gate_test.sh:276): sleep 120021:22
rlandy|roverscreen21:22
sshnaidmweshay|ruck, ok, I changed conditions there, will check tomorrow21:22
weshay|rucksshnaidm, ha.. faker21:23
weshay|ruck:)21:23
rlandy|roverwow - no reproducer file for a while21:23
rlandy|roverand nobody noticed21:23
*** brault has quit IRC21:23
* rlandy|rover fixes21:23
sshnaidmmaybe because of https://review.openstack.org/#/c/618653/21:24
sshnaidmrlandy|rover, I thought it's a plan :D21:24
weshay|ruckhttps://review.openstack.org/62639721:28
weshay|rucksshnaidm, you fixing htat?21:29
weshay|ruckthat?21:29
rlandy|roverno21:29
rlandy|roverit's failing looking for console logs21:31
sshnaidmweshay|ruck, tomorrow21:32
weshay|ruckrlandy|rover, does21:39
weshay|ruckelif [ "$ENVIRONMENT" = "ovb" ] ; then21:39
weshay|ruck    # We only support multi-nic at the moment21:39
weshay|ruck    NETISO_ENV="multi-nic"21:39
weshay|ruck    ./toci_quickstart.sh21:39
weshay|rucktrigger a ovb job?21:40
weshay|ruckah .. I guess so21:40
rlandy|rovershould do21:40
weshay|ruckrlandy|rover, k.. reproducer is running now21:44
rlandy|rovergood21:44
weshay|ruckI cut out some stuff in toci_gate_test21:44
weshay|rucksshnaidm, ^21:45
weshay|ruckI think all we need is21:45
weshay|ruckif [ "$ENVIRONMENT" = "ovb" ] ; then21:45
weshay|ruck    # We only support multi-nic at the moment21:45
weshay|ruck    NETISO_ENV="multi-nic"21:45
weshay|ruck    ./toci_quickstart.sh21:45
weshay|ruckelse21:45
weshay|ruck    # Copy nodepool keys to current user21:45
weshay|ruck    sudo cp /etc/nodepool/id_rsa* $HOME/.ssh/21:45
sshnaidmweshay|ruck, I tried to leave possibility to run with te-broker and not to change anything in tripleo-ci21:46
*** trown is now known as trown|outtypewww21:47
weshay|rucksshnaidm, meh.. why go back21:47
weshay|rucksshnaidm, ovb is questionable atm anyway21:47
weshay|rucksshnaidm, let's just kill kill kill21:47
weshay|rucksshnaidm, at least in the toci_gate_test.sh21:48
weshay|ruckthat only runs in reproducer now21:48
sshnaidmweshay|ruck, ok21:48
weshay|ruckugh http://paste.openstack.org/show/737762/21:49
weshay|ruckrlandy|rover, ^21:49
rlandy|roverweshay|ruck: any objections to getting rid of the zuul_variables in the old reproducer21:49
rlandy|roverwe are not going that route anyways21:49
rlandy|roverand it's failing now21:49
rlandy|roverwhere is that paste from?21:50
weshay|ruckrlandy|rover, my local box21:50
rlandy|roverk - let me make this fix and I'll try it21:51
weshay|ruckI'm concerned we have to reproducer at all atm21:51
weshay|ruckand concerned is the nice term21:51
weshay|ruckrlandy|rover, ah sorry..21:54
weshay|ruckrlandy|rover, fetch images is failing because it's trying to pull an unpromoted hash21:54
weshay|ruckI have to hack that21:54
rlandy|roverweshay|ruck: https://review.openstack.org/626400 Remove reproducer lines added to get zuul related info21:56
rlandy|roverlet's see if the reproducer creates in that review21:56
rlandy|roverwhat's next to tackle?22:03
rlandy|roverweshay|ruck: ^^?22:03
weshay|ruckrlandy|rover, so for ovb create, we probably want to sed the dlrn_hash in the image url to current-tripleo22:04
weshay|ruckbugs bugs bugs22:04
weshay|ruckman.. we need ci on the repro22:04
rlandy|roverthis should not have changed though22:06
rlandy|roveronly if that hash does not exist22:06
rlandy|roverright?22:06
weshay|ruckya.. that sounds right22:09
rlandy|roverso really this is a note to self when rewriting the reproducer22:10
weshay|ruckrlandy|rover, sshnaidm https://tree.taiga.io/project/morucci-software-factory/issue/221722:39
rlandy|roverweshay|ruck: something for this sprint?22:41
weshay|ruckit's not ours.. it's prod-chain-infra22:41
rlandy|roverinitially the 100 stacks was calulated22:41
rlandy|roverweshay|ruck: we determined our estimated resources22:41
weshay|ruckrlandy|rover, sshnaidm they are going to recalculate the stack quota and nodepool settings to appropriately use the available resources22:41
rlandy|roverk - understand22:42
rlandy|roverwe did try estimate that originally with jpena22:42
weshay|ruckya.. but the resources has also been changed22:42
weshay|ruckand nobody knows how this whole thing works22:42
weshay|ruckalan had no idea there was a quota on stacks22:43
rlandy|rover"and nobody knows how this whole thing works"22:44
rlandy|roverquote of the year22:44
* rlandy|rover wants to put that on our status22:44
rlandy|roverreproducer is back ...22:45
rlandy|roverhttps://logs.rdoproject.org/00/626400/1/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/9491ed8/logs/reproducer-quickstart.sh22:45
weshay|ruckthanks22:45
rlandy|roveryou can vote22:49
weshay|ruckrlandy|rover, https://review.rdoproject.org/zuul/builds22:52
weshay|ruckstarting to see things work22:53
rlandy|roverso 053 is a mixed result22:54
rlandy|roverthe stacks are stable22:55
rlandy|roverperiodic kicked23:06
rlandy|rover2018-12-19 19:17:45,686 - testenv-worker-8814 - INFO - Getting new job... - was the last record on te-broker23:09
rlandy|roverso that is good23:09
*** rascasoft has quit IRC23:12
rlandy|roverweshay|ruck: got to go my other volunteer job - will be back in about 3 hours23:24
*** rlandy|rover is now known as rlandy|rover|bbl23:24
*** rascasoft has joined #oooq23:48
*** rascasoft has quit IRC23:54
*** tosky has joined #oooq23:56
« Tuesday, 2018-12-18 Index Thursday, 2018-12-20 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!