Thursday, 2019-04-04

*** dtrainor_ has quit IRC00:02
*** sshnaidm_ has quit IRC00:25
*** sshnaidm has joined #oooq00:29
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035,  (3 more messages)00:31
*** dsneddon has quit IRC01:06
*** dtrainor_ has joined #oooq01:27
*** dtrainor_ has quit IRC01:29
*** dtrainor_ has joined #oooq01:30
*** dtrainor_ has quit IRC01:34
*** dsneddon has joined #oooq01:37
*** dsneddon has quit IRC01:43
*** dsneddon has joined #oooq02:14
*** dsneddon has quit IRC02:20
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035,  (3 more messages)02:31
*** rlandy|bbl is now known as rlandy02:59
*** vinaykns has quit IRC03:18
*** rlandy has quit IRC03:47
*** dsneddon has joined #oooq04:00
*** ykarel has joined #oooq04:03
*** udesale has joined #oooq04:09
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035,  (3 more messages)04:31
jaosoriorquiquell|off: not cutting an RC yet, but here's a minor release for tripleo-common https://review.openstack.org/64987504:51
*** dsneddon has quit IRC05:06
*** ratailor has joined #oooq05:12
*** ykarel is now known as ykarel|afk05:19
*** jdennis has quit IRC05:20
*** ccamacho has quit IRC05:22
*** ykarel|afk has quit IRC05:23
*** marios has joined #oooq05:25
*** jdennis has joined #oooq05:35
*** ykarel|afk has joined #oooq05:42
*** ykarel|afk is now known as ykarel05:43
*** quiquell|off is now known as quiquell05:46
*** quiquell is now known as quiquell|rover05:57
quiquell|roversshnaidm: I am going to add stein to cockpit-ci as part of rr, ok ?05:59
*** kopecmartin|off is now known as kopecmartin06:14
*** holser_ has joined #oooq06:18
*** quiquell|rover is now known as quiquell|brb06:23
*** jfrancoa has joined #oooq06:25
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @  (2 more messages)06:31
*** rascasoft has quit IRC06:44
*** dsneddon has joined #oooq07:03
*** quiquell|brb is now known as quiquell|rover07:08
*** ccamacho has joined #oooq07:11
*** dsneddon has quit IRC07:16
ykarelquiquell|rover, hi07:17
quiquell|roverykarel: o/07:18
ykarelquiquell|rover, do you have some card about phase2 removal, i see some of your patches but no context there07:18
quiquell|roverykarel: ther is not I just try to cleanup as I find stuff07:19
ykarelquiquell|rover, you know the reason for removal?07:19
quiquell|roverykarel: baremetal is going to run at internal software factory zuul now07:20
quiquell|roverykarel: and added as promotion criteria for current-tripleo07:20
ykarelquiquell|rover, okk, Thanks for info07:20
quiquell|roverykarel: so like moving phase2 to current-tripleo07:20
ykarelquiquell|rover, okk got it07:20
ykarelbut anyway always good to add reasoning in commit messages so others know07:21
*** tosky has joined #oooq07:26
*** rascasoft has joined #oooq07:30
*** rascasoft has quit IRC07:30
*** apetrich has joined #oooq07:37
*** jpena|off is now known as jpena07:43
*** dsneddon has joined #oooq07:43
*** dsneddon has quit IRC07:48
*** ykarel is now known as ykarel|lunch07:48
*** amoralej|off is now known as amoralej07:53
*** bogdando has joined #oooq08:04
*** panda has joined #oooq08:09
*** marios has quit IRC08:24
*** dtantsur|afk is now known as dtantsur08:25
*** gkadam has joined #oooq08:27
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @  (2 more messages)08:31
chandankumarquiquell|rover: needs +w on this https://review.openstack.org/#/c/648111/08:35
chandankumarthanks!08:35
quiquell|roverchandankumar: done08:38
chandankumarquiquell|rover: thanks!08:38
*** dsneddon has joined #oooq08:44
*** ykarel|lunch is now known as ykarel08:45
*** dsneddon has quit IRC08:49
*** jaosorior has quit IRC08:54
*** ykarel is now known as ykarel|away09:00
*** dsneddon has joined #oooq09:03
*** skramaja has joined #oooq09:03
*** ykarel|away has quit IRC09:05
*** dsneddon has quit IRC09:08
zbrquiquell|rover: clearly the promotion testing hash files are out of sync with the release files.09:11
quiquell|roverzbr: yep we merge a not finishes refactoring that was bad decision09:14
quiquell|roverzbr: fix and then refactor full09:14
quiquell|roverzbr: or full refactor and fix09:14
quiquell|roverzbr: we are in the middle now09:14
*** marios has joined #oooq09:14
quiquell|roverzbr: I prefer to fix first so you know later if the refactor is not working09:14
zbri will upload something soon and ask you to look at it.09:14
zbri would not call this refactor, more of get-in-sync09:15
quiquell|roverack09:16
zbrquiquell|rover: just to be sure, the problems are likely to be in promotion testing files, right?09:18
quiquell|roverzbr: yep09:21
zbr.... i reall hate comments with ? in code, questions are for review, not for merged code. :D09:22
*** rascasoft has joined #oooq09:29
zbrquiquell|rover: part of the problem, images are not distro-aware yet.09:30
zbrsee https://images.rdoproject.org/centos7/master/rdo_trunk/current-tripleo-rdo-internal/09:30
quiquell|roverzbr: there is a centos7 in the URL this is not enough ?09:31
zbrbut https://images.rdoproject.org/fedora28/ is empty,09:31
quiquell|roverzbr: we have just creatd just from yesterday09:31
quiquell|roverzbr: the idea is to push images there09:31
quiquell|roverzbr: is not empty09:32
zbrok, so is valid to start configuring the new urls09:32
quiquell|roverzbr: it has images already check it out09:32
quiquell|roverzbr: ahh no09:32
quiquell|rover:-/09:32
quiquell|roverzbr: they should09:32
zbrif you used symlinks they dissapear when target is removed09:32
zbrweb server will not display invalid symlinks09:33
*** ykarel|away has joined #oooq09:33
*** ykarel|away is now known as ykarel09:33
zbri would like to repair the overcloud_image_url and ipa_image_url variables but we need to seed something there to make it work.09:36
zbrquiquell|rover: have a look at https://github.com/openstack/tripleo-quickstart/blob/master/config/release/tripleo-ci/CentOS-7/master.yml#L26-L2709:45
zbrthese URLs do not exist on master and stein. aparently we do have a folder current-tripleo-rdo but not one current-tripleo09:46
zbron older releases we have both09:46
zbrbut this means that the config is invalid, and I need to know what we need to put there.09:47
sshnaidmquiquell|rover, what is the "item" here? https://review.rdoproject.org/r/#/c/19915/6/files/playbooks/roles/create_project/tasks/main.yaml10:00
quiquell|roversshnaidm: https://review.rdoproject.org/r/#/c/19915/6/files/playbooks/setup-gerrit.yaml10:01
sshnaidmquiquell|rover, yeah, but what is "item"?10:02
quiquell|roversshnaidm: ansible "with_items" element10:02
quiquell|roversshnaidm: elements from "branches"10:02
sshnaidmquiquell|rover, where is with_items? which line?10:02
quiquell|rover4710:02
quiquell|roverwith_items: "{{ branches }}"10:03
*** dtantsur is now known as dtantsur|brb10:07
sshnaidmquiquell|rover, thanks, +w10:28
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @  (2 more messages)10:32
quiquell|roversshnaidm: reproducer standalone issue is now with identation10:32
quiquell|roversshnaidm: runcmd has bad identation10:32
sshnaidmquiquell|rover, ack, will look10:33
quiquell|roversshnaidm: I am trying to fix it10:33
quiquell|rovermarios, panda: Do we merge the build-test-package fix ? https://review.openstack.org/#/c/636588/10:35
*** derekh has joined #oooq10:35
mariosquiquell|rover: i need to review that added for next10:36
quiquell|roverok10:36
mariosquiquell|rover: is it somethign urgent10:36
quiquell|rovermarios: just to remove an alert we have regarding this10:36
quiquell|rovermarios: not really urgent10:36
quiquell|roversshnaidm: argg is driving me crazy trying to fix the indent issue10:39
quiquell|roversshnaidm: can you try it ?10:39
mariosquiquell|rover: btw just spotted that wrt the release wiki i pointed you at yesterday https://docs.openstack.org/tripleo-docs/latest/developer/release.html in docs10:39
quiquell|rovermarios: jaosorio is making a minor release for us10:40
quiquell|rovermarios: of tripleo-common10:40
mariosquiquell|rover: k10:41
quiquell|rovermarios: https://review.openstack.org/64987510:41
*** holser_ is now known as holser|burger10:47
quiquell|roversshnaidm: https://bugs.launchpad.net/tripleo/+bug/182315810:47
openstackLaunchpad bug 1823158 in tripleo "Bad runcmd identation at zuul reproducer openstack nodepool " [High,In progress] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm)10:47
zbrquiquell|rover: how confident are you about how valid are our 56 release config files in general?10:50
*** jaosorior has joined #oooq10:52
*** ratailor has quit IRC10:54
*** udesale has quit IRC10:54
*** ratailor has joined #oooq10:56
*** jpena is now known as jpena|lunch10:58
*** jaosorior has quit IRC10:59
sshnaidmquiquell|rover, yeah, will fix11:01
*** jaosorior has joined #oooq11:01
*** panda has quit IRC11:02
zbrquiquell|rover nad sshnaidm please have a look at https://review.openstack.org/#/c/649965/ and tell me what you think -- see output of openstack-tox-py3511:02
quiquell|roverzbr: job failed11:03
quiquell|roverzbr: not a bad idea, don't know how extensive can be the number of checks11:05
zbrquiquell|rover: read the log, it failed because 17/56 release files contained invalid URLs11:05
zbrnot because that change was bad, because it undercovered bugs in our config.s11:05
zbrthe test made is minimal, but enough to identify a class of errors https://review.openstack.org/#/c/649965/3/test/validate-release-config.yml11:06
*** zbr has quit IRC11:14
quiquell|roverchandankumar, arxcruz: Do you know what's the tempest issue here ? http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset021-master/212d87e/logs/tempest.html.gz11:20
quiquell|roverMax retries exceeded with url: /v3/auth/tokens11:20
arxcruzquiquell|rover: some big shit happened11:21
arxcruzit seems the keystone service is down11:21
arxcruzor at least the endpoint11:21
*** zbr has joined #oooq11:31
arxcruzweshay|ruck: quiquell|rover marios sshnaidm https://bugs.launchpad.net/tripleo/+bug/1821690/comments/1111:33
openstackLaunchpad bug 1821690 in tripleo "periodic-ovb-1ctlr_1comp-featureset020-master fails tempest test_auto_allocate_network tempest.api.compute.images.test_images" [High,Triaged] - Assigned to Arx Cruz (arxcruz)11:33
arxcruzso, i found the root cause of the failure, but there are more failures, please ask in the CIX call to have a neutron specialist taking a look at it11:34
mariosarxcruz: ack thanks sounds like we need to ping neutron team11:34
mariosis one for ruck/rover i guess11:34
mariosarxcruz: probably worth adding a note on the escalation card11:34
mariosarxcruz: ah you did already cool11:35
mariosarxcruz: that card is most likely how we get network folks to looks11:35
arxcruzmarios: come on man, I'm a pro :P11:36
mariosarxcruz: showoff!11:39
*** panda has joined #oooq11:39
*** panda is now known as panda|lunch11:39
*** quiquell|rover is now known as quiquell|rover|e11:40
arxcruzmarios: https://bugs.launchpad.net/tripleo/+bug/1818860 I am reopening because it's green because we added the test to skip list, but I haven 't found the root cause11:40
openstackLaunchpad bug 1818860 in tripleo "tempest.api.network.test_ports.PortsIpV6TestJSON.test_update_port_with_two_security_groups_and_extra_attributes failing with SecurityGroupNotFound" [Critical,Fix released] - Assigned to Arx Cruz (arxcruz)11:40
*** quiquell|rover|e is now known as quique|rover|eat11:40
arxcruzquique|rover|eat: oh wait, this is featureset02111:42
*** ykarel has quit IRC11:45
*** panda|lunch has quit IRC11:45
*** ykarel has joined #oooq11:46
*** ratailor has quit IRC11:49
*** dtantsur|brb is now known as dtantsur11:57
quique|rover|eatsshnaidm, marios: Add stein promotions to cockpit https://review.rdoproject.org/r/1995511:57
*** rlandy has joined #oooq11:58
*** trown|outtypewww is now known as trown12:01
rlandyquique|rover|eat: hello12:02
quique|rover|eatrlandy: updated notes ath etherpad12:03
rlandyquique|rover|eat: got temp kicked off ruck/rover12:04
quique|rover|eatrlandy: lucky you :-)12:04
quique|rover|eatrlandy: going back for lunch :-)12:04
rlandyquique|rover|eat: it will come back to haunt me12:04
quique|rover|eatrlandy: stein promotions at cockpit https://review.rdoproject.org/r/#/c/19955/12:06
rlandyk - was just moving stuff around at escalation board12:07
rlandythere were two failures yesterday - will see if they repeat12:07
quique|rover|eatrlandy: cinder container is not found is fixed12:09
quique|rover|eatrlandy: we should have correct stein promotions12:09
weshay|ruckquique|rover|eat I I see 0 gate failures and nothing in the gate..  and don't see jobs blocking...   afaict.. we're in good shape12:12
*** quique|rover|eat is now known as quiquell|rover12:13
quiquell|roverweshay|ruck: yep12:13
quiquell|roverweshay|ruck: we can use this time to fix last things from stein fedora/centos promotion pipeline12:14
quiquell|roverweshay|ruck: btw today I have to leave earlier to see another school12:14
weshay|ruckk np12:14
quiquell|roverweshay|ruck: have updatd the etherpad https://etherpad.openstack.org/p/ruckroversprint812:15
weshay|ruckquiquell|rover is your devel env still up re: cockpit and stein?12:15
quiquell|roverweshay|ruck: yep12:15
weshay|ruckquiquell|rover k.. add a screenshot12:16
weshay|ruck:P12:16
quiquell|roverok12:16
quiquell|roverweshay|ruck: added https://review.rdoproject.org/r/#/c/19955/12:18
weshay|ruckquiquell|rover any idea why? https://review.openstack.org/#/c/649705/ http://logs.openstack.org/05/649705/1/check/tripleo-ci-centos-7-standalone-upgrade/39d1173/job-output.txt.gz#_2019-04-03_20_36_51_55015112:21
*** jpena|lunch is now known as jpena12:22
weshay|ruckchandankumar need a few12:24
quiquell|roverweshay|ruck: is not legit ?12:24
chandankumarweshay|ruck: yes, ?12:24
rlandyquiquell|rover: weshay|ruck: we're missing current-tripleo image again12:25
quiquell|roverweshay|ruck: print that registry to see if some service is down12:25
quiquell|roverrlandy: container  ?12:25
chandankumarweshay|ruck: regarding this patch https://review.openstack.org/#/c/649705/ tengu is looking into that12:26
rlandyquiquell|rover: there seems to be a hitch where the images are not linked12:26
rlandygetting on promotion server12:26
rlandyquiquell|rover: we didn't promote master?12:27
weshay|ruckquiquell|rover ya12:28
rlandylooks like not since yesterday12:28
weshay|ruckbut check the log on standalone upgrade12:28
weshay|ruckit's not working12:28
weshay|ruckfor some f.. reason I can figure out12:28
weshay|ruckworks locally for me.. in isolation12:28
weshay|ruckI wonder if it's ansible VERSION12:28
weshay|ruckugh12:28
weshay|ruckquiquell|rover dang it.. added in 2.712:29
* Tengu still scratching his head12:29
chandankumarweshay|ruck: 1:1 time12:30
quiquell|roverweshay|ruck: ovn12:31
quiquell|roverweshay|ruck: http://logs.openstack.org/05/649705/1/check/tripleo-ci-centos-7-standalone-upgrade/39d1173/logs/undercloud/var/log/extra/failed_services.txt.gz12:31
*** jaosorior has quit IRC12:31
rlandyquiquell|rover: added images back = creating a bug12:31
quiquell|rovertripleo_ovn_northd_healthcheck.service failed.12:31
*** jaosorior has joined #oooq12:31
quiquell|roverrlandy: what do you mean ?12:31
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci- (2 more messages)12:32
weshay|ruckchandankumar /me joins12:32
quiquell|roverTengu: this rings any bell ?12:32
quiquell|roverTengu: http://logs.openstack.org/05/649705/1/check/tripleo-ci-centos-7-standalone-upgrade/39d1173/logs/undercloud/var/log/extra/failed_services.txt.gz12:32
Tenguquiquell|rover: hmmm nope. not related to my recent changes about the silencing of the healtchecks at least.12:33
Tengudidn't hit lsof.12:33
quiquell|roverTengu: could be too much load ? http://logs.openstack.org/05/649705/1/check/tripleo-ci-centos-7-standalone-upgrade/39d1173/logs/undercloud/var/log/containers/openvswitch/ovn-northd.log.txt.gz12:34
Tenguinvalid syntax 'FA-16-3E-33-1D-8E' in addresses column12:35
Tenguquiquell|rover: so apparently it's more related to syntax than load.12:36
Tengumaybe something changed lately in ovn-northd regarding MAC syntax?12:36
quiquell|roverTengu: yep loosk like thanks12:37
quiquell|roverTengu: going to dig there12:37
Tengu:) that's just error message reading though ;)12:37
* Tengu goes back on his tests in order to understand what's going on with the failed services.12:37
quiquell|roverTengu: isn't it the failed services you are looking for ?12:38
Tenguquiquell|rover: weshay|ruck apparently HL the "changed: false"12:38
weshay|ruckTengu naw.. ansible version is fucking us12:39
weshay|ruckI'll put up a new patch12:39
Tenguweshay|ruck: pffrrtt12:39
Tenguversions. who cares about consistency now?12:39
* Tengu runs away12:39
quiquell|roverweshay|ruck: ansible version ?12:40
weshay|ruckya.. the fail_msg is 2.712:42
quiquell|roverahh ok12:43
quiquell|roverjfrancoa: did you open a LP about standalone-upgrade failure ?12:43
quiquell|roverjaosorior: looks like ovn_northd service is failing12:43
jaosoriorquiquell|rover: which jobs? do we have any names that could help with that?12:44
quiquell|roverjaosorior: dalvarez I think12:44
jfrancoaquiquell|rover: yes, this one https://bugs.launchpad.net/tripleo/+bug/182304212:44
openstackLaunchpad bug 1823042 in tripleo "Upgrading from rocky to stein/master does not verify the neutron mechanism correctly" [High,New] - Assigned to Kamil Sambor (ksambor)12:44
quiquell|roverjaosorior: tripleo-ci-centos-7-standalone-upgrade12:45
jaosoriorquiquell|rover: have you tried poking him/12:45
quiquell|roverjaosorior: not yet12:45
jfrancoaquiquell|rover: jaosorior he's aware of it. I think ksambor is working on the lp, or at least he told me yesterday to assign it to him12:45
quiquell|roverjfrancoa: is looking at the ovn-northd issue ?12:45
jaosoriorjfrancoa: ack12:46
quiquell|roverahh yes I see12:46
quiquell|roverthanks12:47
*** holser|burger is now known as holser_12:47
quiquell|roverrlandy: I though imges have to be at stein/master/rdo_trunk/current-tripleo12:49
quiquell|roverwait master12:49
quiquell|rovernothing12:49
*** zul has joined #oooq12:51
quiquell|roverrlandy: I mean centos12:55
quiquell|roverrlandy: we are using this now I think https://images.rdoproject.org/centos7/master/rdo_trunk/12:56
quiquell|roverrlandy: well not at all release file though12:56
*** panda has joined #oooq13:00
weshay|ruckquiquell|rover it's a soft link but yes13:00
quiquell|roverthat's right13:01
quiquell|roverreal deal is the other13:01
sshnaidmquiquell|rover, is it ok? https://review.rdoproject.org/r/#/c/19954/ worked for me13:02
quiquell|roversshnaidm: still deploying13:02
sshnaidmquiquell|rover, ok13:02
weshay|ruckrlandy please join13:02
rfolcoping planning meeting13:02
rfolcomarios, quiquell, sshnaidm, weshay, panda, rlandy, arxcruz, mwhahaha, rfolco, chkumar, ssbarnea, kopecmartin13:03
weshay|ruckzbr please join the call13:03
rlandyjoining13:03
quiquell|roversshnaidm: is at tempest now, so all good13:03
quiquell|roverlet's merge it13:03
quiquell|roversshnaidm: would be nice to have some openstack CI for reproducer13:04
quiquell|roversshnaidm: workflowed13:04
*** jaosorior has quit IRC13:04
zbrcalendar bit messed...13:04
sshnaidmquiquell|rover, that's definitely..13:04
*** jaosorior has joined #oooq13:04
*** udesale has joined #oooq13:05
amoralejquiquell|rover, so promotion pipeline for stein is working? or is something pending?13:09
quiquell|roveramoralej: I think everything has being sorted out13:10
amoralejquiquell|rover, thanks that was my understanding13:10
quiquell|roveramoralej: although looks like images are disappearing ttps://review.openstack.org/64998613:10
quiquell|roveramoralej: I mean https://bugs.launchpad.net/tripleo/+bug/182317313:11
openstackLaunchpad bug 1823173 in tripleo "images.rdoproject.org/master/rdo_trunk/current-tripleo images are missing" [Critical,Triaged] - Assigned to Quique Llorente (quiquell)13:11
quiquell|roveramoralej: let's give it a try for some days to see how it works13:11
quiquell|roveramoralej: thanks for all the help13:11
amoralejquiquell|rover, glad to help13:11
weshay|ruckhttps://tree.taiga.io/project/tripleo-ci-board/epic/80613:11
mariosthanks weshay|ruck13:12
quiquell|roverhave to drop now13:12
quiquell|roverrlandy, sshnaidm, marios, weshay|ruck: branchful reproducer last piece https://review.rdoproject.org/r/#/c/19915/6/defaults/main.yaml13:13
*** quiquell|rover is now known as quiquell|off13:13
rlandyquiquell|off: re; images - we need to updated release files then13:19
chandankumarhttps://github.com/chkumar246/ansible-role-collect-logs collected repo13:20
weshay|ruckapetrich may I move your 1-1 a bit13:20
apetrichweshay|ruck, kein problem13:20
apetrichweshay|ruck, I'm here just ping when ready13:20
*** skramaja has quit IRC13:24
*** vinaykns has joined #oooq13:49
weshay|ruckapetrich is 3pm utc too late?13:52
apetrichweshay|ruck, damn DST is that in one hour or two?13:53
weshay|rucklolz.. in little over one hour13:53
*** mjturek has joined #oooq13:53
*** altlogbot_1 has quit IRC13:54
apetrichweshay|ruck, that's fine. deal13:54
*** udesale has quit IRC14:06
*** udesale has joined #oooq14:07
*** udesale has quit IRC14:11
*** udesale has joined #oooq14:11
*** holser_ is now known as holser|meeting14:24
zbrweshay|ruck: look at https://review.openstack.org/#/c/649965/ and check result of tripleo-ci-validations job.14:25
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci- (2 more messages)14:32
*** chem has joined #oooq14:36
*** Vorrtex has joined #oooq14:52
*** jaosorior has quit IRC15:01
apetrichweshay|ruck, need more time before the 1:1?15:01
weshay|ruckI'm in15:02
weshay|ruckapetrich ^15:03
*** d0ugal has quit IRC15:05
*** d0ugal has joined #oooq15:05
*** jfrancoa has quit IRC15:06
zbrhttp://logs.openstack.org/65/649965/12/check/tripleo-ci-validations/9a6b109/tox/reports.html15:07
*** dsneddon has joined #oooq15:07
*** holser|meeting is now known as holser_15:08
*** aakarsh has quit IRC15:08
*** dsneddon has quit IRC15:09
*** dsneddon has joined #oooq15:11
*** d0ugal has quit IRC15:14
*** d0ugal has joined #oooq15:14
*** jfrancoa has joined #oooq15:22
*** aakarsh has joined #oooq15:32
*** udesale has quit IRC15:32
*** kopecmartin is now known as kopecmartin|off15:41
rlandyzbr: https://bugs.launchpad.net/tripleo/+bug/182317315:44
openstackLaunchpad bug 1823173 in tripleo "images.rdoproject.org/master/rdo_trunk/current-tripleo images are missing" [Critical,Triaged] - Assigned to Quique Llorente (quiquell)15:44
zbrrlandy: panda https://review.rdoproject.org/r/#/c/19540/15:46
weshay|ruckquiquell|off rlandy again ^15:48
weshay|ruck?15:48
weshay|ruckimages https://images.rdoproject.org/master/rdo_trunk/current-tripleo/15:48
weshay|ruckah no15:48
rlandyagain?15:49
rlandyweshay|ruck: we are still discussing the missing images15:50
rlandydefinite bug here15:50
ykarelweshay|ruck, stein phase1 failed as it expected current-tripleo-rdo to exist, which doesn't exist15:52
*** sshnaidm_ has joined #oooq15:52
ykarelweshay|ruck, can you get it fixed and also add to branching doc so it can be prepared in starting15:52
weshay|ruckykarel phase 1 failed where? promoting?15:52
ykarelweshay|ruck, parent job15:53
weshay|ruckthe jobs themselves use current-tripleo and then promote current-tripleo-rdo15:53
* weshay|ruck looks15:53
ykarelweshay|ruck, https://ci.centos.org/job/rdo-promote-get-hash-stein-current-tripleo/1/consoleText15:53
mariosttyl15:53
*** sshnaidm has quit IRC15:54
*** marios has quit IRC15:54
weshay|rucksec ykarel /me fixes something15:54
ykarelweshay|ruck, ack15:54
weshay|ruckykarel it didn't fail https://ci.centos.org/job/rdo-promote-get-hash-stein-current-tripleo/1/console15:55
weshay|ruckthe tripleo job did though15:56
* weshay|ruck looks15:56
ykarelweshay|ruck, right but it's dependent tripleo job failed because it needs artifact created by ^^15:56
ykarelweshay|ruck, /tmp/jenkins8568890531950177181.sh: line 27: [: ==: unary operator expected, u seen this?15:57
*** jfrancoa has quit IRC15:57
weshay|ruck15:29:55 /home/jenkins/workspace/tripleo-quickstart-promote-stein-current-tripleo-delorean-minimal/tripleo-quickstart/ci-scripts/full-deploy.sh: line 36: LOCATION: unbound variable15:58
ykarelyes ^^ is created by the parent job15:58
weshay|rucklocation is created by the parent job?15:59
weshay|ruckdon't think so15:59
* weshay|ruck revies.. noting ykarel is probably right15:59
ykarelweshay|ruck, yes https://github.com/rdo-infra/ci-config/blob/master/jenkins/jobs/scripts/centos-stein-current-tripleo.sh#L1315:59
* weshay|ruck just catching up15:59
weshay|ruckappreciates ykarel patience15:59
rlandyzbr: https://bugs.launchpad.net/tripleo/+bug/182317315:59
openstackLaunchpad bug 1823173 in tripleo "images.rdoproject.org/master/rdo_trunk/current-tripleo images are missing" [Critical,Triaged] - Assigned to Quique Llorente (quiquell)15:59
zbrrlandy: panda https://review.rdoproject.org/r/#/c/19966/16:00
ykarelweshay|ruck, sorry https://github.com/rdo-infra/ci-config/blob/master/jenkins/jobs/scripts/promote-get-hash.sh#L1316:00
weshay|ruckykarel k.. k.. so couple things.. I think when branching.. the note to self is.. to not just create current-tripleo hash, images, and containers... probably we should do both current-tripleo and current-tripleo rdo16:01
ykarelweshay|ruck, yes16:01
ykarelgood to add to branching doc16:01
ykarelso not miss in future16:01
weshay|ruckykarel lastely.. I'm think we probably don't want rdo phase 1 to promote stein at this time...16:01
weshay|ruckbecause it's really not stein yet16:01
weshay|rucktripleo has not branched16:01
ykarelweshay|ruck, it will eventually branch16:02
ykarelso current master is stein for tripleo16:02
weshay|ruckykarel you guys are not planning on "releasing" stein from this or future stein promotions w/o tripleo signaling it's done branching right?16:02
*** amoralej is now known as amoralej|off16:02
weshay|ruckdoes that make sense?16:02
ykarelweshay|ruck, we don't release stein based on dlrn builds16:02
weshay|ruckok16:03
ykarelwe release the tagged builds: CBS16:03
weshay|ruckykarel which release files are used for that16:03
weshay|ruckyou guys still use anything under config/release ?16:03
weshay|ruckoutside of tripleo, rdo phase 116:03
weshay|ruckjust curious16:03
ykarelweshay|ruck, no, tripleo are not tested with CBS builds16:04
weshay|ruckk k16:04
ykarelas we don't have containers yet16:04
weshay|ruckykarel so I'll go create the dlrn link for current-tripleo-rdo to match current-tripleo16:04
ykarelweshay|ruck, no16:04
weshay|ruckafaict that is all that is required here16:04
weshay|ruck?16:04
ykarelweshay|ruck, create current-tripleo-rdo matching previous-current-tripleo16:04
weshay|ruckah.. or else the job won't run16:05
weshay|ruckright16:05
ykarelotherwise promotion job will not run16:05
ykarelyes right16:05
weshay|ruckk16:05
weshay|ruckya16:05
ykareluntil next promotion16:05
weshay|ruckykarel however we are very close to a tripleo promotion of sten16:05
weshay|ruckstein16:05
weshay|ruckwe could just sit on this16:05
weshay|ruckand allow the chain to work through it's cycle16:05
ykarelokk work for me16:06
weshay|ruckykarel is there any reason you need this now?16:06
weshay|rucktakes a while for my coffee to kick in.. /me is tired today16:06
*** bogdando has quit IRC16:06
ykarelweshay|ruck, just looking for jobs green as we will be updating rdo dashboard16:06
weshay|ruckmeh..16:06
ykareland in near future we will plan test day rdo16:06
ykarelso before that want all green16:06
ykarelweshay|ruck, so just create a current-tripleo-rdo link and then we can hopefully get job running16:07
ykareltake your time16:07
* ykarel leaving16:07
weshay|ruckk. have a good night16:07
ykarelu too16:07
*** ykarel is now known as ykarel|away16:07
ykarel|awayweshay|ruck, ya just remember we need promotion16:08
ykarel|awayas we need current-tripleo-rdo for packstack/poi stein jobs16:08
weshay|ruckk16:08
*** jfrancoa has joined #oooq16:12
*** jfrancoa has quit IRC16:20
rlandyweshay|ruck: hey - can you join bj again for a minute?16:25
rlandywe have a possible fix for promoter images16:26
weshay|rucksure16:26
weshay|ruckwhich?16:26
pandaweshay|ruck: impromptu meeting in rfolco's room16:26
rlandyhttps://bluejeans.com/587845809716:26
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci- (2 more messages)16:32
*** yolanda has quit IRC16:33
*** tosky has quit IRC16:43
zbrweshay|ruck: https://review.openstack.org/#/c/649965/16:51
*** ccamacho has quit IRC16:58
*** panda is now known as panda|off16:59
zbrweshay|ruck: have you found the report http://logs.openstack.org/65/649965/12/check/tripleo-ci-validations/9a6b109/tox/reports.html ?17:00
*** dtantsur is now known as dtantsur|afk17:00
zbrcheck it and add comments on the review regarding how to fix the 9 errors found.17:00
weshay|ruckI like the report for sure17:02
weshay|ruckzbr where is the review w/ the fix?17:02
*** jpena is now known as jpena|off17:08
*** holser_ has quit IRC17:10
*** gkadam has quit IRC17:11
*** dsneddon has quit IRC17:12
*** panda|off has quit IRC17:14
*** dsneddon has joined #oooq17:15
*** derekh has quit IRC17:18
*** dsneddon has quit IRC17:21
zbrweshay|ruck: do you have a min to explain on bj?17:25
weshay|ruckzbr I have to roll to an appointment now.. let's do it tomorrow :)17:25
weshay|ruckrlandy bb in about 1.5 hrs17:26
weshay|ruckthanks17:26
zbrsure17:26
rlandyweshay|ruck: ack17:27
*** rlandy is now known as rlandy|ruck17:27
*** dsneddon has joined #oooq17:30
*** dsneddon has quit IRC17:40
*** dsneddon has joined #oooq17:40
*** dsneddon has quit IRC17:46
*** dsneddon has joined #oooq17:47
*** sshnaidm_ is now known as sshnaidm|off17:50
*** derekh has joined #oooq18:09
*** derekh has quit IRC18:10
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci- (2 more messages)18:32
rfolcorlandy|ruck, apparently osp14 job failed at prepare image container phase, you know what is goin on there or needs investigation ?19:26
rfolcohttps://sf.hosted.upshift.rdu2.redhat.com/logs/74/166374/9/check/tripleo-ci-rhel-7-standalone-rhos-14/5451ad0/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz#_2019-04-02_18_49_2519:26
rlandy|ruckneeds investigation19:26
rfolcoI am just clarifying points to add more info to the stories19:27
rlandy|ruckthat is where the work should start19:27
rfolcook19:27
rfolconice!19:27
vinayknsrlandy|ruck: I'm trying to deploy quickstart, but I'm hitting http://pastebin.test.redhat.com/74990519:31
vinayknsI'm not sure why it isn't reacheable19:31
rlandy|ruckSSH Error: data could not be sent to remote host "localhost". Make sure this host can be reached over ssh19:32
rlandy|ruckvinaykns: very hard to tell from the paste19:33
*** tosky has joined #oooq19:33
rlandy|ruckcheck if you can manually ssh with the keys available19:33
vinayknsyea, this is what I could get from the error logs19:34
vinayknsyea I could login into the undercloud with the keys I have19:35
rlandy|ruckrfolco: editing https://tree.taiga.io/project/tripleo-ci-board/us/951?milestone=226267 to add details19:48
rfolcorlandy|ruck, cool19:49
rlandy|ruckrfolco: some of that review can merge - I guess19:50
rfolcorlandy|ruck, one task is to split mergable stuff from that?19:51
rlandy|ruckrfolco: I'll see what can be done19:51
rlandy|ruckthe nodeset etc.19:51
rfolcok19:52
weshay|ruckrlandy|ruck back19:55
rlandy|ruckweshay|ruck: k- missed nothing expect an rdocloud backup19:55
weshay|ruckbackup?19:55
rlandy|ruckweshay|ruck: node failures in promotion19:55
weshay|ruckoh19:56
rlandy|rucksuggestion to submit failing node and stack19:56
rlandy|ruckscripts to run more frequently19:56
* rlandy|ruck puts in review19:56
weshay|ruckk19:56
rlandy|ruckweshay|ruck: also watching fs03919:57
rlandy|ruckhas a  lot of failures19:57
weshay|ruckrlandy|ruck ya... that's I think still on sagi19:57
rlandy|ruckweshay|ruck: some are the usual introspection ones19:57
rlandy|ruckupsetting19:58
weshay|ruckconnecting to keystone?19:58
rlandy|ruckno19:58
rlandy|rucknode note reaching right state19:58
weshay|ruckrlandy|ruck oh crap20:01
rlandy|ruckscenario002?20:01
weshay|ruckya.. rdo cloud is not registring in cockpit20:01
weshay|ruckat around 12:00 utc20:02
rlandy|rucknot registering?20:02
rlandy|ruckweshay|ruck: how so?20:02
weshay|ruckthe monitoring went down20:03
rlandy|ruckoh stacks20:03
rlandy|ruckweshay|ruck: can you show me where to fix that?20:03
weshay|ruckit may be rdo20:03
weshay|ruckbut I'm not 100% sure.. looking into it20:03
weshay|ruckseems responsive and working to me20:04
rlandy|ruckI nuked instances in error20:05
rlandy|ruckmaybe I deleted something that monitors?20:05
*** chem has quit IRC20:08
rlandy|ruckweshay|ruck: rdocloud is up for me20:10
weshay|ruckaye.. agree20:10
weshay|ruckrlandy|ruck so I see where it potentially is running and how but not sure if it's still accurate20:10
weshay|ruckhttps://github.com/rdo-infra/ci-config/blob/master/ci-scripts/infra-setup/roles/rrcockpit/files/telegraf/rdocloud.py#L920:10
weshay|ruckI think it may now run an internal node20:10
rlandy|ruckweshay|ruck: I hope the te-broker was not in error I didn;t delete it :(20:11
weshay|ruckrlandy|ruck naw.. nothing is on the te-broker20:12
rlandy|ruckthe comments says it is20:12
rlandy|ruck0b820839-86a7-4f03-ae92-9178ef71449b | te-broker                                               | ACTIVE  | private=192.168.100.250, 38.145.33.16620:12
rlandy|ruckok - I didn't kill it20:12
weshay|ruckI'm on the box20:12
rlandy|ruckthese mass delete scripts give me high blood pressure20:12
weshay|ruckah ya found it20:13
weshay|ruckrlandy|ruck heh20:13
weshay|ruckur fine20:13
rlandy|ruckbreathe ... ok what is the issue so I know for next time?20:13
*** dsneddon has quit IRC20:14
rlandy|ruckweshay|ruck: ^^?20:16
*** dsneddon has joined #oooq20:17
weshay|rucksshnaidm|off is rdo-cloud monitoring still done on the te-broker?20:17
weshay|ruck[root@te-broker tebroker]# ls -ltra influxdb_stats20:17
weshay|ruck-rw-r--r--. 1 root root 386 Apr  4 20:15 influxdb_stats20:17
weshay|ruck[root@te-broker tebroker]# pwd20:17
weshay|ruckvia cron in root?20:17
weshay|ruckrlandy|ruck I can show you what I see if you want20:18
rlandy|ruckpls20:18
weshay|ruckhttps://bluejeans.com/u/whayutin/20:19
*** dsneddon has quit IRC20:31
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp- (2 more messages)20:32
*** dsneddon has joined #oooq20:32
*** Vorrtex has quit IRC20:46
*** sshnaidm|off has quit IRC20:55
*** aakarsh has quit IRC20:58
*** sshnaidm|off has joined #oooq20:58
*** radez has quit IRC20:58
*** jrist has quit IRC20:59
*** jrist has joined #oooq21:01
*** dalvarez has quit IRC21:01
*** amoralej|off has quit IRC21:01
*** aakarsh has joined #oooq21:02
rlandy|ruckhttps://review.rdoproject.org/r/1997621:06
rlandy|ruckweshay|ruck: ^^21:06
*** mjturek has quit IRC21:08
rlandy|ruckweshay|ruck: watch just died - running to get it fxed - back in a bit21:20
*** rlandy|ruck is now known as rlandy|ruck|brb21:21
weshay|rucknp21:21
*** vinaykns has quit IRC21:27
*** rlandy|ruck|brb is now known as rlandy|ruck21:39
weshay|ruckrlandy|ruck when you have a sec.. can you workflow this for me https://review.openstack.org/#/c/649705/21:54
rlandy|ruckweshay|ruck: done21:55
weshay|ruckthanks21:55
*** rascasoft has quit IRC22:05
*** rascasoft has joined #oooq22:06
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp- (2 more messages)22:32
*** ykarel|away has quit IRC22:33
*** ykarel has joined #oooq22:36
*** rlandy|ruck is now known as rlandy23:03
*** dsneddon has quit IRC23:20
*** dsneddon has joined #oooq23:22
*** dsneddon has quit IRC23:27
*** dsneddon has joined #oooq23:30
*** dsneddon has quit IRC23:40
*** tosky has quit IRC23:43
*** dsneddon has joined #oooq23:47
*** dsneddon has quit IRC23:52
*** dsneddon has joined #oooq23:53
*** dtrainor has joined #oooq23:56
*** dsneddon has quit IRC23:57

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!