Tuesday, 2018-09-18

*** jaganathan has quit IRC01:03
hubbot1FAILING CHECK JOBS on stable/pike: tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/602248, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens, tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/567224, stable/rocky: tripleo-ci- (2 more messages)01:46
*** rlandy|bbl is now known as rlandy02:23
*** saneax has joined #oooq02:43
*** apetrich has quit IRC02:52
*** ykarel|away has joined #oooq03:01
*** ykarel|away has quit IRC03:07
*** rlandy has quit IRC03:42
hubbot1FAILING CHECK JOBS on stable/pike: tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/602248, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-3nodes-multinode, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens, tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/567224,  (2 more messages)03:46
*** rlandy has joined #oooq03:50
*** rlandy has quit IRC03:53
*** udesale has joined #oooq03:53
*** ykarel|away has joined #oooq04:08
*** ykarel|away has quit IRC04:34
*** ykarel|away has joined #oooq04:53
*** jaosorior has joined #oooq04:54
*** skramaja has joined #oooq04:55
*** ykarel|away is now known as ykarel04:55
*** jaganathan_ has joined #oooq05:17
*** ratailor has joined #oooq05:23
*** quique|rover|off is now known as quiquell|rover05:31
*** jtomasek has joined #oooq05:35
*** holser_ has joined #oooq05:45
hubbot1FAILING CHECK JOBS on stable/pike: tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/602248, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-3nodes-multinode, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens, tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/567224,  (2 more messages)05:46
*** jfrancoa has joined #oooq06:06
*** jtomasek has quit IRC06:09
*** jtomasek has joined #oooq06:13
*** jtomasek has quit IRC06:35
*** quiquell|rover is now known as quique|rover|brb06:36
*** jtomasek has joined #oooq06:46
ssbarneamorning!06:59
ssbarneachkumar|ruck: quique|rover|brb -  https://review.openstack.org/#/c/588587/ waits for a +W07:00
*** jtomasek has quit IRC07:01
*** quique|rover|brb is now known as quiquell|rover07:02
*** saneax has quit IRC07:13
*** ykarel is now known as ykarel|lunch07:31
*** gkadam has joined #oooq07:39
jaosoriorcould someone checkt his out https://review.openstack.org/#/c/570841/5 ? would be good to merge it07:43
hubbot1FAILING CHECK JOBS on stable/pike: tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/602248, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-3nodes-multinode, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens, tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/567224,  (2 more messages)07:46
marioso/ jaosorior done recheck?07:48
*** bogdando has joined #oooq07:50
*** agopi has quit IRC07:54
jaosoriormarios: rechecked it now.07:56
*** kopecmartin has joined #oooq08:00
mariosjaosorior: thanks08:02
*** tosky has joined #oooq08:16
*** ykarel|lunch is now known as ykarel08:25
*** saneax has joined #oooq08:27
*** chem has joined #oooq08:41
*** saneax has quit IRC08:50
*** saneax has joined #oooq08:52
ssbarneadoes anyone knows what ETA should we expect for removal of the legacy- jobs? (raw estimate).09:08
ssbarneai am asking because I want to know if it would worth fixing an issue or not.09:09
quiquell|roverssbarnea: Need to ask rlandy and rfolco, they are deep into that09:19
quiquell|roverssbarnea: But it will take time to reparent the jobs and remove legacy ones,09:19
ssbarneaas in months i guess09:20
quiquell|roverssbarnea: something like that09:22
*** kopecmartin has quit IRC09:26
quiquell|roverchkumar|ruck: For master promotion we have the undercloud upgrade, telemetry and ceph issues that's it ?09:28
chkumar|ruckquiquell|rover: yes09:29
chkumar|ruckquiquell|rover: in recent job some neutron tests also failed09:29
quiquell|roverchkumar|ruck: but not periodic ones ?09:29
chkumar|ruckquiquell|rover: fs020 telemetry issue impaced fs020 periodic jobs09:30
quiquell|roverchkumar|ruck: fs020 is out of criteria at master, but not at rocky09:30
chkumar|ruckquiquell|rover: for fs020 rocky connection timed out happen, I am still diggin what wnet wrong there09:31
quiquell|roverchkumar|ruck: rocky is telemetry and ceph ?09:31
chkumar|ruckduring overcloud deploy09:31
chkumar|ruckquiquell|rover: for rocky fs019 blocked due to ceph issue, fs01/fs035 due to ironic conductor issue09:32
chkumar|ruckquiquell|rover: for rocky it is only ceph09:33
quiquell|roverchkumar|ruck: I see fs01/fs035 passing09:33
quiquell|roverchkumar|ruck: at rocky09:33
chkumar|ruckquiquell|rover: ah both is passing right now09:35
quiquell|roverchkumar|ruck: yep09:35
quiquell|roverWe have to being impacted by some infra issue09:36
quiquell|roverall those ironic issues09:36
*** dtantsur|afk is now known as dtantsur09:36
*** saneax has quit IRC09:39
*** saneax has joined #oooq09:39
*** saneax has quit IRC09:40
*** saneax has joined #oooq09:41
quiquell|roverchkumar|ruck: this is not a periodic jobs https://bugs.launchpad.net/tripleo/+bug/179307309:45
openstackLaunchpad bug 1793073 in tripleo "[queens] fs01 periodic job failed with Stderr: u'/usr/bin/ironic-inspector-rootwrap: Unauthorized command: systemctl start openstack-ironic-inspector-dnsmasq.service (no filter matched)" [Critical,Triaged] - Assigned to Quique Llorente (quiquell)09:45
quiquell|roverchkumar|ruck: queens fs001 periodic jobs are passing09:45
chkumar|ruckquiquell|rover: oh sorry it was from noop job09:46
hubbot1FAILING CHECK JOBS on stable/pike: tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/602248, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-3nodes-multinode, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-queens, tripleo-ci-centos-7-scenario004 (2 more messages)09:46
quiquell|roverchkumar|ruck: Going to remove promotion blocker ok ?09:46
chkumar|ruckquiquell|rover: yes, it is from this review https://review.openstack.org/#/c/56722409:47
chkumar|ruckquiquell|rover: archived the card the from CIX board also09:49
quiquell|roverchkumar|ruck: ok09:50
quiquell|roverchkumar|ruck: Now it's a potential blocker09:50
chkumar|ruckquiquell|rover: dtantsur is looking into that09:50
*** ykarel is now known as ykarel|away09:50
quiquell|roverchkumar|ruck: ok09:50
*** ykarel|away is now known as ykarel|afk09:51
*** kopecmartin has joined #oooq10:02
*** gkadam has quit IRC10:02
*** gkadam has joined #oooq10:04
*** quiquell|rover has quit IRC10:08
*** quiquell has joined #oooq10:09
quiquellchkumar|ruck: we have issues with te-broker after the restart last week10:19
quiquellchkumar|ruck: it's overlapping ip addresses10:19
chkumar|ruckquiquell: new ip for tbroker was https://github.com/openstack-infra/tripleo-ci/commit/51548c763579ae770c4f9d3bd08f9367b963844b10:20
chkumar|ruckis this one overlapping ?10:20
*** gkadam has quit IRC10:21
quiquellchkumar|ruck: nope i mean bmc openstack instances10:21
quiquellOVB start a BMC instance to act as IPMI using virtualbmc10:21
quiquellbut looks like openstack is assigning dup ip address to those instances...10:22
chkumar|ruckoh10:23
pandaouch10:30
pandathis is not te-broker issue10:30
pandathis should not happen in the nodepool tenant10:30
*** kopecmartin has quit IRC10:31
chkumar|ruckquiquell: I need to check how te-broker work10:32
quiquellchkumar|ruck: looks like something is taking 192.168.100.18910:33
quiquellchkumar|ruck: I think all the runs with this IP fails10:33
chkumar|ruckquiquell: how to check that it is checking that particular ip?10:38
quiquellchkumar|ruck: looking for the IP here http://38.145.33.166/testenv-worker.log10:40
quiquellchkumar|ruck: I find the execution number10:40
quiquellchkumar|ruck: then with the execution number I can search here10:40
quiquellhttp://38.145.33.166/console-logs/10:41
quiquellfor bmc-10:41
pandardocloud is donw for me ...10:44
panda504 Gateway Time-out10:44
chkumar|ruckpanda: working for me10:44
pandachkumar|ruck: you can access https://phx2.cloud.rdoproject.org/dashboard/project/ ?10:45
chkumar|ruckpanda: yup10:45
quiquellpanda: semaphore looks good https://semaphore.cloud.upshift.engineering.redhat.com/10:47
*** udesale has quit IRC10:47
pandammhh stil can't access it :(10:48
chkumar|ruckbrb10:56
*** gkadam has joined #oooq10:57
*** ykarel|afk is now known as ykarel11:00
*** kopecmartin has joined #oooq11:14
*** ratailor has quit IRC11:34
*** ratailor has joined #oooq11:34
*** ratailor has quit IRC11:34
*** gkadam has quit IRC11:39
*** gkadam has joined #oooq11:40
hubbot1FAILING CHECK JOBS on stable/pike: tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/602248, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-3nodes-multinode, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-queens, tripleo-ci-centos-7-scenario004 (2 more messages)11:46
quiquellchkumar|ruck: let's update CIX stuff11:55
*** saneax has quit IRC11:56
*** saneax has joined #oooq11:56
*** agopi has joined #oooq11:57
chkumar|ruckquiquell: we have a queens promotion but scenario 004 is still timing out12:01
quiquellchkumar|ruck: scenario004 is featurset01912:02
quiquellCIX12:02
chkumar|ruckquiquell: yes12:03
chkumar|ruckweshay: https://review.rdoproject.org/etherpad/p/ruckrover-sprint1912:05
*** gkadam has quit IRC12:06
*** jtomasek has joined #oooq12:16
weshayanyone want/need a 1-1?12:22
weshaytoday12:22
*** kopecmartin has quit IRC12:22
pandanow or never ?12:23
weshaypanda, in about a hour12:25
pandaweshay: I'm good, we have one tomorrow.12:25
*** trown|outtypewww is now known as trown12:26
chkumar|ruckquiquell: weshay I am heading out, I need some sleep, see ya tomorrow12:31
weshaychkumar|ruck, thanks brotha12:31
quiquellchkumar|ruck: see ya, rest12:31
*** chkumar|ruck is now known as chkumar|off12:32
quiquellweshay: So we don't need the fix for rocky->master ?12:32
weshayquiquell, the upgrade job?12:32
weshaywe do12:32
quiquellthis https://review.openstack.org/#/c/590774/12:33
weshayquiquell, there were a few related errors and there was too much getting blocked12:34
quiquellweshay: ack12:34
*** dtantsur is now known as dtantsur|brb12:40
*** rlandy has joined #oooq12:42
rlandypanda: weshay: pls revote/w https://review.openstack.org/#/c/602445 thank you12:45
rlandyquiquell: I'm rechecking https://review.openstack.org/#/c/589448 - zuul -112:53
quiquellrlandy: ack, maybe there is something broken there, we are reviving it too much12:54
rlandyquiquell: rfolco: have we done any testing with ^^ and scenario tests12:54
rlandythe ones I missed last time?12:54
*** kopecmartin has joined #oooq12:54
rlandytripleo-ci-centos-7-containerized-undercloud-upgradesFAILURE in 56m 27s12:54
rlandy^^ gone none-voting12:54
rlandywon't fail now12:55
rfolcorlandy, yes if you look at previous ps, zuul +1 with scenario tests green (voting ones)12:56
quiquellrlandy: We run scenarios at tripleo-ci changes now12:56
rlandythe ocata oens12:57
rlandyones12:57
rlandy that I broke12:57
rlandyw/o containers12:57
rlandyrfolco knows about them12:57
quiquellweshay: Cannot add you to reviews at review.o.o12:57
weshaylp :(12:57
rfolcoquiquell, gmail not redhats12:57
weshayquiquell, I'm sick of my personal email getting killed12:58
quiquellrfolco: gmail not there anymore12:58
* weshay switching to whayutin@redhat.com12:58
quiquellweshay: ack12:58
rfolcooh ok :)12:58
rlandytripleo-ci-centos-7-scenario001-multinode-oooq12:58
rlandyrfolco: ^^ those tests are ok?12:59
rlandyif so, I am +212:59
rfolcorlandy, https://review.openstack.org/#/c/589448 ?12:59
rfolcowhich one12:59
rlandyrfolco: my change broke the ocata tests13:00
rlandy▼ tripleo-ci-centos-7-scenario001-multinode-oooq13:00
rlandy▼ tripleo-ci-centos-7-scenario002-multinode-oooq13:00
rlandy▼ tripleo-ci-centos-7-scenario003-multinode-oooq13:00
rlandy▼ tripleo-ci-centos-7-scenario004-multinode-oooq13:00
rlandythe  non-containerized ones13:00
rfolcorlandy, I rechecked https://review.openstack.org/#/c/589448 due to container-upgrades job, which is now nv...13:03
rfolcorlandy, can you pls elaborate a bit more on your question?13:04
*** saneax has quit IRC13:04
*** saneax has joined #oooq13:04
rlandyrfolco: you did the right thing13:05
rlandyI was going to do the same13:05
rlandyone section of tests we don;t kick with a change is the ocata only tests13:05
rlandythose were the ones I broke when the vars I was using were not carried through13:05
* rlandy gets13:06
quiquellweshay, rlandy, panda: To unblock master, rocky https://review.openstack.org/#/c/60332313:07
quiquellmarios: ^13:08
*** jtomasek has quit IRC13:08
weshayquiquell++13:08
hubbot1weshay: quiquell's karma is now 413:08
quiquellDon't need karam need reviewing :-P13:09
rfolcoquiquell++13:10
hubbot1rfolco: quiquell's karma is now 513:10
rfolcoquiquell++13:10
hubbot1rfolco: quiquell's karma is now 613:10
weshayquiquell, lolz13:10
weshayI am13:10
quiquellrfolco++13:10
hubbot1quiquell: rfolco's karma is now 313:10
quiquellteam work !13:10
*** jtomasek has joined #oooq13:10
rlandyrfolco: https://bugs.launchpad.net/tripleo/+bug/179244013:11
openstackLaunchpad bug 1792440 in tripleo "Chage to use subnodes in place of /etc/nodepool/* files breaks scenario tests" [Critical,Fix released] - Assigned to Ronelle Landy (rlandy)13:11
rfolcorlandy, ok but we reverted already13:11
rfolcoisn't it?13:11
rlandyrfolco: yes we revert that change - I ma asking if we tested quiquell's patch with those tests13:12
rlandyso we don;t break them when we merge that patch13:12
rlandythey are not kicked with a change13:12
quiquellweshay, rlandy: Going to update the ceph tqe change to remove Depends-On they are not needed13:12
quiquellOk now https://review.openstack.org/#/c/603323/13:13
quiquellsorry my brain hurts now13:13
rfolcorlandy, the problem is: I am not getting reviews over the last 2 sprints... https://review.openstack.org/#/c/602425/ adds scenario113:14
*** quiquell is now known as quique|rover|lch13:14
rfolcorlandy, I remember we talked about adding scenario to config/* changes in tripleo-ci... let me know if I misunderstood you13:18
rlandyrfolco: correct - I +2'ed that13:18
weshayrlandy, point me at your osp-14 again when you have a sec13:19
rlandybut it still would not run here13:19
rlandyweshay: the rhos-14 job? config file? which link?13:19
weshayjob13:19
rlandyhttps://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/tripleo-quickstart-rhos-14-passed-p1-baremetal-hp_dl360_envE-single_nic_vlans/13:19
rlandysee the extra-params in the config13:19
rlandyrfolco: marios picked up the error13:20
rlandyso ocata must have run somewhere13:20
weshaythanks13:21
rfolcorlandy, I am still floating on ocata jobs sorry13:21
rlandyfloating?13:22
rlandyrfolco: what does that mean :) ^^13:23
rfolcohmm maybe a not very known slang for saying "I have no idea what you talking about"13:24
rfolcorlandy, ^13:24
*** agopi has quit IRC13:24
*** zul has joined #oooq13:27
rlandymaybe I am just not hip enough to understand the latest slang13:28
rlandybasically those jobs fried us before13:28
rlandyI would prefer to not have that repeat13:28
*** vinaykns has joined #oooq13:30
rlandyrfolco: the gate job on ocata kicks those tests13:34
rlandyhttps://review.openstack.org/#/c/56429113:34
rlandyrfolco: we would know as soon as those jobs run13:35
rfolcorlandy, I tested it in patchset3 of https://review.openstack.org/#/c/602425/13:36
rfolcorlandy, see zuul results for ps3, it ran scenario000113:36
rlandyok - let's live dangerously13:36
rfolcorlandy, ah... one thing13:36
rfolcotripleo-ci-centos-7-scenario001-multinode-oooq-container13:36
*** skramaja has quit IRC13:37
rfolconon-containerized did not run :(13:37
rlandyI want non-containerized13:37
rlandythey run differently13:37
rfolcolet me check then, the filter included oooq13:37
rlandyit won't kick unless you have an ocata change13:39
*** quique|rover|lch is now known as quiquell|rover13:40
weshayreminder ooo mtg in 20min13:41
weshayI'm going to invite folks to chat irc/blue after the mtg13:42
rfolcorlandy, found why... branch filter13:43
rfolcohttps://github.com/openstack-infra/tripleo-ci/blob/master/zuul.d/multinode-jobs.yaml#L16913:43
rfolcorlandy, do we want to include master ? sign here ______________________13:43
hubbot1FAILING CHECK JOBS on stable/pike: tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/602248, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-3nodes-multinode, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-queens, tripleo-ci-centos-7-scenario004 (2 more messages)13:46
*** agopi has joined #oooq13:46
quiquell|roverweshay: ruck/rover need to be at any of those ?13:46
*** apetrich has joined #oooq13:48
*** vinaykns has quit IRC13:49
rlandyrfolco: I am not sure we will even be able to check it as is13:49
rlandymaybe we just let it go13:49
rfolcorlandy, need to fix https://review.openstack.org/#/c/602425/ if you want to run scn001 for tq/config* on master13:50
weshayquiquell|rover, sorry?13:51
weshayquiquell|rover, no13:51
quiquell|roverweshay: ack13:51
*** vinaykns has joined #oooq13:52
rlandyyou change is fine13:53
rlandybut can be added to13:53
rfolcorlandy, k13:53
weshayrlandy, are you back on thrs?13:55
rlandyweshay: yes  I don't plan to fast for two days :)13:55
weshayI'm going for it ;))13:55
rlandyrfolco: if you have a change that will kick ocata - we can test there - otherwise, we'll just try and see what happens14:00
rfolcorlandy, will do it, no cost14:03
*** jschlueter has joined #oooq14:03
ssbarneaweshay: have a look at https://review.openstack.org/#/c/563849/ when you have bit of time, it may worth to give some input sooner than later, if you think it matters.14:06
rlandyrfolco: thanks - would prefer not to have to do another revert14:06
rfolcorlandy, for the love of gods of cloud computing and patron saint of intermittent jobs... no14:07
weshayssbarnea, we have 3rd party ci14:07
weshayssbarnea, are you comfortable jumping into reviews on the sprint or do you want some fresh work?14:08
*** dtantsur|brb is now known as dtantsur14:09
mariosweshay: rlandy rfolco panda fwiw and as I declared I would fyi i put my notes at http://mariosandreou.com/tripleo.xml (it should show up at http://tripleo.org/planet.html at some point or something changed)14:10
ssbarneaweshay:  today i did only reviews, but I am can do whatever you think is more important. PM me when you have few minutes.14:12
rlandymarios: cool - thanks14:12
*** ykarel is now known as ykarel|away14:16
*** jfrancoa has quit IRC14:17
*** quiquell|rover is now known as quique|rover|off14:17
rfolcorlandy, tq is not branched so there is no way tq/config/* change in stable/ocata to trigger scn001-oooq (non-containerized) as is....14:20
*** jfrancoa has joined #oooq14:20
weshayarxcruz, rlandy :)14:21
* weshay looks at you guys 14:21
pandarfolco: branchful jobs ?14:23
rlandyrfolco: we'd need a change on ocata branch of tht14:23
rfolcopanda, https://github.com/openstack-infra/tripleo-ci/blob/master/zuul.d/multinode-jobs.yaml#L16914:24
rlandylike the gate does14:24
rfolcorlandy, a change on tht is not a change to tq/config/*, which was the request14:25
rfolcoI got the following request: "I want to trigger scn001 against changes to tq/config*"14:26
rlandythat is correct14:26
rlandybut we need the ocata jobs14:26
rlandywhich may not be possible14:26
rlandythe containerized ones kick anyways14:26
rfolcoyeah, true, only containerized one is kicked14:27
rlandyfine14:27
rlandywe'll see it soon enough14:27
*** jfrancoa has quit IRC14:27
rlandyweshay: ^^ why arxcruz and me? missed the reference. from the tripleo meeting?14:28
weshayrlandy, arxcruz there is an opportunity to potentially package and containerize a thing14:29
weshaywhich is good experience for tempest work14:29
weshaycitellus14:29
rlandyagree14:29
rlandythanks14:29
*** ykarel|away has quit IRC14:36
weshayquique|rover|off, https://review.openstack.org/#/c/603367/14:38
weshaydoh14:38
*** bogdando has quit IRC14:48
*** hamzy has quit IRC14:56
*** hamzy has joined #oooq14:58
*** zul has quit IRC15:08
*** zul has joined #oooq15:18
weshayrlandy, https://review.openstack.org/#/c/603422/15:32
pandahttps://trello.com/c/JN4UIjWv/854-replace-zuulchanges-handling-in-build-test-packages-role15:41
weshaypanda, https://review.openstack.org/#/c/603277/15:43
hubbot1FAILING CHECK JOBS on stable/pike: tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/602248, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-3nodes-multinode, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-queens, tripleo-ci-centos-7-scenario004 (2 more messages)15:46
*** saneax has quit IRC15:51
*** hamzy has quit IRC16:01
*** gkadam has joined #oooq16:02
*** ykarel|away has joined #oooq16:10
weshayrfolco, rlandy panda https://review.rdoproject.org/r/1630516:11
rlandyjob definition16:13
*** rlandy is now known as rlandy|brb16:14
*** holser_ has quit IRC16:23
*** dtantsur is now known as dtantsur|afk16:28
rfolcorlandy|brb, weshay panda ssbarnea marios https://review.openstack.org/589448 "its now or never -- presley, elvis."16:36
pandanever.16:36
rfolco:'-(16:36
pandawe need to test the reproducer ...16:36
ssbarnearfolco: i wanted to say the same. i was checking the two extra verification changes pointed by wes.16:37
rfolcothen... "love me tender..."16:37
weshaymarios, https://review.rdoproject.org/r/1630616:38
*** rlandy|brb is now known as rlandy16:38
rlandyrfolco: https://review.openstack.org/#/c/603422/ has not run yet16:40
rlandyI'll +2 when it has16:40
weshayrlandy, rfolco how far into a job do we need to see for ocata?16:42
weshayhttp://zuul.openstack.org/stream.html?uuid=a44e36b031784428bcbcf1d763719b13&logfile=console.log16:42
rlandyone job run16:42
rlandyany scenario test16:42
weshayrlandy, right.. but the whole way?16:42
rlandyweshay: so really it's a matter of it the vars set carry16:42
weshayrlandy, even if we break ocata it's not going to be an emergency16:42
weshayk16:42
rlandywe can +2 it now16:43
rlandyand hope for the best16:43
weshayrlandy, rfolco http://lists.openstack.org/pipermail/openstack-dev/2018-September/134671.html16:43
weshayrfolco, rlandy going to capitalize on the opportunity16:44
weshay+2 workflowed16:44
rlandyok16:45
weshaywe'll keep eyes on it16:45
rlandyhttps://review.openstack.org/#/c/602445/16:45
rlandypassed gates16:45
rlandywaiting for merge16:45
weshaypanda, rlandy fyi https://review.openstack.org/#/c/600641/16:56
weshayI need to fix the syntax16:56
weshaybut it's a good idea /me thinks16:56
*** dsneddon has joined #oooq17:00
*** kopecmartin has quit IRC17:01
weshaypanda, I don17:05
*** vinaykns has quit IRC17:05
weshayI don't think you've thought this one through https://review.openstack.org/#/c/603277/4/roles/build-test-packages/tasks/main.yml17:05
weshaypanda, what does the logic on a single task look like17:05
weshaycan you mock that for me?17:06
weshayI don't think it's possible17:06
*** gkadam has quit IRC17:07
weshaypanda, hrm.. https://review.rdoproject.org/r/#/c/16305/1/playbooks/legacy/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-rocky/run.yaml17:09
weshaypanda, can you get on blue w/ me?17:09
*** dsneddon has quit IRC17:10
pandaweshay: never heard of with_list before, and I didn't find in the loops section.. still not sure what's the deifference with with_items17:10
pandaweshay: ok17:10
weshaypanda, with_list is a thing17:10
weshay:)17:10
weshaypanda, I tested this out in https://github.com/weshayutin/westest_ansible17:10
rlandypanda: weshay: rfolco: we have a merge on https://review.openstack.org/#/c/602445/17:11
weshaypanda, I also don't understand your comments in rdo-project17:11
weshayrlandy, nice17:11
rlandypanda: weshay: rfolco: going to ask for infra if they can reconfigure (and if we can watch)17:11
weshayrlandy, oh ya.. we had a merge party in #tripleo17:11
weshayrlandy++17:11
hubbot1weshay: rlandy's karma is now 2117:11
weshayloop me in please17:12
rlandywe did indeed17:12
weshaypanda, I'm trying to add check jobs17:12
weshaynot periodic17:12
pandaweshay: rlandy for how long will we have these legacy definitions17:16
panda?17:16
pandain rdo17:16
weshaypanda, hopefully not long17:16
weshaylike a few more days17:16
rlandywhen we can reparent successfully.17:16
rlandywe can get rid of them17:16
pandaweshay: then you can ignore my comments. It's just useless moaning.17:17
rlandyweshay: you are still missing the jobs definitions, no?17:18
weshayya.. looking at that now17:18
weshaypanda, and the other review?17:19
weshaypanda, https://github.com/weshayutin/westest_ansible/blob/master/playbooks/test.yml#L1417:25
weshaypanda, https://github.com/weshayutin/westest_ansible/blob/master/vars/vars.yml#L1417:25
rlandyweshay: are you following the conversation on #sf-dfg?17:25
*** dsneddon has joined #oooq17:32
ssbarnearlandy: just curios about #sf-dfg channel, last message appears from Sunday and posted by a bot. Am I in a wrong room? ... there are 51 others there.17:35
rlandyssbarnea: on RHAT on freenode17:35
ssbarneayep, some as me but I see no traffic, maybe the bridge is wedged, again.17:36
ssbarnearlandy: weshay : please help me merge https://review.openstack.org/#/c/597581/ which is part of preventive measures from a bugfix made two weeks ago. That is the proof that I should have not split it in two, splitting caused a lot of extra wasted time. When the bug was breaking a gate, people were more "inclined" to make it happen ;)17:41
hubbot1FAILING CHECK JOBS on stable/pike: tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/602248, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-3nodes-multinode, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-queens, tripleo-ci-centos-7-scenario004 (2 more messages)17:46
*** trown is now known as trown|lunch18:00
*** panda is now known as panda|dinner18:01
rlandyssbarnea: sorry too late - looks like you got the reviews you needed18:07
*** ykarel|away has quit IRC18:10
*** trown|lunch is now known as trown18:55
agopirlandy, weshay can you please review+merge this https://code.engineering.redhat.com/gerrit/#/c/149351/19:00
rlandydone19:02
agopithanks rlandy ssbarnea19:02
*** hamzy has joined #oooq19:04
agopiokay weird rlandy weshay19:05
agopino jjb-push build triggered?19:05
agopilast one was triggered 11 days back on 9/719:05
agopihttps://code.engineering.redhat.com/gerrit/#/c/149941/ and https://code.engineering.redhat.com/gerrit/#/c/150055/19:06
agopiwere merged but it didn't trigger jjb-push job?19:06
agopilike it did with19:06
agopihttps://code.engineering.redhat.com/gerrit/#/c/149622/19:06
rlandyhmm ... don;t know19:09
rlandycomparing19:09
agopinot sure how it is related but browbeat-ci last triggered on sept 11 and then the next patchset was submitted on 9/14 however none of the patchsets triggered browbeat-ci :/19:10
agopiugh19:10
rlandylet's try run that push manuall19:14
agopii think the problem is with the gerrit servers19:15
agopirlandy, probably related to it is that the executors went down on sept 11 or 12 https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/user/agopi/my-views/view/Browbeat_view/job/browbeat-quickstart-rhos-13-baremetal-mixed/2/console19:21
gouthamrhi folks, a fix for stable/pike jobs (https://review.openstack.org/#/c/601522/) is timing out on the gate, and rechecks aren't helping19:27
gouthamrthe full suite of tempest integration tests are being run, and the timeout seems to be around 45 mins19:28
*** panda|dinner is now known as panda19:30
rlandyagopi: updated correcetly now?19:39
rlandytripleo-ci-centos-7-scenario004-multinode-oooq-containerTIMED_OUT in 3h 24m 17s19:40
rlandy^^ is the only timeout that would fail your job19:40
agopinope rlandy, the jobs still aren't triggering :/ opening a ticket and will add you to watch list.19:40
rlandyagopi: I did a manual push19:41
agopiI saw that, the problem seems to be with gerrit trigger plugin or the server added to jenkins19:41
agopiand that is why19:41
rlandyyes - but I want to know if the push updated19:42
agopineither browbeat-ci triggered nor the jjb push job triggered19:42
agopiit did update rlandy19:42
agopithanks19:42
rlandyok - so you are not blocked19:42
rlandywe have a workaround for noe19:42
agopiI am still blocked as we're not able to test any change submitted to browbeat :/19:43
hubbot1FAILING CHECK JOBS on stable/pike: tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/602248, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-3nodes-multinode, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-queens, tripleo-ci-centos-7-scenario004 (2 more messages)19:46
rlandyagopi: is there an error you see? can you paste?19:51
* rlandy checks gerrit setup19:51
agopii can't see any error rlandy the jobs just aren't getting triggered19:52
agopihttps://review.openstack.org/603482 was submitted couple of mins back19:52
agopibut https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/user/agopi/my-views/view/Browbeat_view/job/browbeat-quickstart-gerrit-pike-baremetal-CI/19:52
agopino build has been triggered since sept 1119:52
agopiopened a ticket just now rlandy19:55
agopihttps://redhat.service-now.com/pnt?id=ticket&table=x_redha_pnt_devops_table&sys_id=c328dfb2db246740a16700b5ca96198619:55
rlandyhttps://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/user/agopi/my-views/view/Browbeat_view/job/browbeat-quickstart-gerrit-pike-baremetal-CI/19:57
rlandy^6 doe snot trigger off gerrit changes though?19:57
rlandydoes not19:57
agopiit does rlandy19:57
rlandyhow - there are no changes listed19:57
agopihttps://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/user/agopi/my-views/view/Browbeat_view/job/browbeat-quickstart-gerrit-pike-baremetal-CI/594/19:57
rlandyno gerrit trigger in the job19:57
agopiyeah i've had te same question19:58
agopibut same goes for jjb-push job19:58
agopiit doesnt have anything related to gerrit trigger in job19:58
agopihttps://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/tripleo-environments-jjb-push/19:58
rlandygit ssh://rdo-ci-downstream@review.openstack.org:29418/openstack/browbeat.git19:58
rlandyoh19:59
*** gouthamr has quit IRC20:00
agopiyeah is myoung on pto, rlandy ?20:00
rlandyagopi: yeah myoung is on pto until he officially leaves20:01
rlandyhe won't be working20:01
*** dmellado has quit IRC20:01
*** hubbot1 has quit IRC20:01
agopioh okay do you know of anyone else who has an idea of how these jobs were getting triggered through gerrit events?20:01
*** gouthamr has joined #oooq20:03
rlandyagopi: https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/user/agopi/my-views/view/Browbeat_view/job/browbeat-quickstart-gerrit-queens-baremetal-CI/ is running20:07
rlandyStarted by Naginator after the failure of build #36720:08
agopii just triggered it manually rlandy20:08
agopii wanted to see if after the build it would at least post the vote20:08
rlandywell, would change to master start a job on a pike branch?20:09
rlandyStarted by Naginator after the failure of build #36720:09
agopiyep it would start a job on both pike and queens rlandy20:10
agopiit probably says that because i hit retry20:10
agopihttps://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/user/agopi/my-views/view/Browbeat_view/job/browbeat-quickstart-gerrit-pike-baremetal-CI/20:11
agopisomething mustve hapened at the time build 596 was triggered20:11
rlandythese triggers are not like the ones I set up20:11
agopibecasuse look at the time20:11
agopiit sasys jan 1 197020:11
rlandywhen I worked with triggers, you would see it in the change that triggered the job20:11
weshaythat's when the lord created unix20:11
agopiyes weshay, looks like a reboot occurred20:12
agopiand the executors also went down the same time20:12
agopiand rlandy do you know what mustve changed ?20:12
weshaycoming into this w/o much context20:12
weshayon jenkins?20:13
rlandyso am I20:13
rlandyon jenkins20:13
agopiyes weshay jenkins20:13
* weshay looks20:13
rlandybut these triggers are different from the ones we used to use20:13
*** cgoncalves is now known as cgoncalves|pto20:13
*** abishop has joined #oooq20:14
weshayagopi, when did you detect a change?20:14
weshayhttps://code.engineering.redhat.com/gerrit/gitweb?p=tripleo-environments.git;a=log20:14
agopibetween 9/7 and 9/1220:14
agopiit wasn't introduced through these commits i believe20:15
weshayagopi, no changes in git20:15
agopiyep nothing through git i believe more like some sort of shutdown mustve happened20:15
agopifirst thing I saw that was wonky was this https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/user/agopi/my-views/view/Browbeat_view/job/browbeat-quickstart-rhos-13-baremetal-mixed/2/console20:15
rlandyI checked the git logs20:16
rlandynothing much went into jobs20:16
weshayagopi, that is the slave disconnecting20:16
agopiyep and it got connected in a couple of mins20:16
agopiaround same time20:16
agopihttps://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/user/agopi/my-views/view/Browbeat_view/job/browbeat-quickstart-gerrit-pike-baremetal-CI/20:16
agopithis occured20:17
weshayagopi, ok.. that may be your lab20:17
weshayyour slaves20:17
weshayhave you gone through all that?20:17
weshayyour networking20:17
rlandyweshay: our jenkins push didn;t happen either though20:17
agopii spoke to kambiz and by the time he looked into it, it was working fine so we didn't investigate much20:17
rlandyon changes to jobs/tripleo-quickstart20:17
agopibut i don't think it was just our lab issues20:18
agopibecause the gerrit trigger stopped working20:18
agopiyes like rlandy mentioned20:18
weshayrlandy, which change?20:18
weshayhttps://code.engineering.redhat.com/gerrit/gitweb?p=tripleo-environments.git;a=commit;h=6702f1f6bb59fd31c62431958598804faddd92a920:18
agopihttps://code.engineering.redhat.com/gerrit/#/c/149941/20:18
agopilook at this, no jjb-push got triggered20:18
rlandythat should trigger jjob=push20:18
weshayagopi, no20:19
weshaythat should NOT trigger a push20:19
weshayagopi, it only updated settings20:19
abishophey, question for the group, what's the best way to get a non-bleeding edge deployment of rocky?20:19
abishopI tried "--release tripleo-ci/consistent-rocky" but it fails to find "consistent" images on docker.io20:19
agopiin that case weshay netiher did this trigger the push20:20
agopihttps://code.engineering.redhat.com/gerrit/#/c/149351/20:20
rlandyfyi all - rdocloud outage20:20
weshayagopi, rlandy it did https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/tripleo-environments-jjb-push/98/20:21
weshayagopi, it did not log back to gerrit20:21
rlandyI started that manually20:21
rlandyStarted by user Ronelle20:21
weshayrlandy, bah20:21
weshay:)20:21
weshay00:00:00.006 Started by user Ronelle20:22
agopii can submit another patchset and see if it triggers after merge20:22
rlandyI wanted to see if the update would work20:22
rlandygo4it20:22
rlandylet's watch20:22
agopican we merge this https://code.engineering.redhat.com/gerrit/#/c/146159/3/jenkins/jobs/tripleo-quickstart/tripleo-quickstart-browbeat.yml20:23
agopinothing important in those jobs20:23
weshayrlandy, I don't see the gerrit plugin20:23
agopiif it got merged, it should trigger right?20:23
weshaynot w/o the pluggin20:24
weshayinstalling and restarting20:24
weshayso something changed for sure20:24
agopiokay cool, lets see if it fixes everything20:25
rlandyJenkins is going to shut down?20:25
weshayrlandy, the gerrit plugin was TOAST20:25
agopistill weird that eevrything was fine but just gerrit plugin went awol lol20:25
weshayagopi, yes20:26
agopiis it gonna shut down lol20:33
agopiits been saying that for a while now20:33
agopiweshay, is it for waiting for the builds that are currently executing to shut down?20:34
weshayyes20:34
rlandyhttps://review.rdoproject.org/zuul/status.html - no connection :(20:35
agopii can stop my builds weshay if needed.20:35
weshayagopi, sure man20:35
agopii can't ugh20:36
agopiit is not letting me stop lol20:36
agopino option.20:36
weshayagopi, can't stop..20:41
weshaywon't stop20:41
*** panda has quit IRC20:41
weshaywe're in it for the long run20:41
agopiokay i had only kicked off a perfci build for rhos-13 like 2-3 hours bakc so it'll take another 10 hours to finish20:43
agopiso tomorrow morning we'll see the restart fixed everything.20:44
*** hamzy has quit IRC20:47
*** weshay is now known as weshay_pto20:53
*** abishop has quit IRC21:00
*** trown is now known as trown|outtypewww21:06
*** chem has quit IRC21:11
*** gouthamr_ has joined #oooq21:18
*** dmellado has joined #oooq21:20
*** gouthamr has quit IRC21:26
*** hubbot1 has joined #oooq21:28
*** rlandy has quit IRC21:31
*** agopi has quit IRC21:32
weshay_ptoWOOT https://review.openstack.org/#/c/589448/21:37
hubbot1FAILING CHECK JOBS on stable/pike: tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/602248, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-3nodes-multinode, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-queens, tripleo-ci-centos-7-scenario004 (2 more messages)21:46
*** sshnaidm|ptg has joined #oooq22:00
*** sshnaidm|ptg is now known as sshnaidm|pto22:00
*** agopi has joined #oooq22:05
*** toure is now known as toure|gpne22:22
*** toure|gpne is now known as toure|gone22:22
*** jaosorior_ has joined #oooq22:43
*** jaosorior has quit IRC22:45
*** sshnaidm|pto has quit IRC22:59
*** tosky has quit IRC23:18
*** abishop has joined #oooq23:20
hubbot1FAILING CHECK JOBS on stable/pike: tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/602248, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-3nodes-multinode, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-queens, tripleo-ci-centos-7-scenario004 (2 more messages)23:46

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!