Friday, 2019-05-17

*** gkadam has joined #oooq01:16
*** dsneddon has quit IRC01:21
*** dsneddon has joined #oooq01:23
*** gkadam has quit IRC01:40
*** apetrich has quit IRC01:58
*** ykarel|away has joined #oooq02:29
*** dsneddon has quit IRC03:20
*** dsneddon has joined #oooq03:21
*** gkadam has joined #oooq03:23
*** gkadam has quit IRC03:23
*** dsneddon has quit IRC03:25
*** dsneddon has joined #oooq03:47
*** dsneddon has quit IRC03:55
*** ratailor has joined #oooq04:01
*** ykarel|away has quit IRC04:12
*** ykarel|away has joined #oooq04:26
*** dsneddon has joined #oooq04:32
*** udesale has joined #oooq04:34
*** dsneddon has quit IRC04:38
*** ykarel|away is now known as ykarel04:39
*** dsneddon has joined #oooq04:39
*** yolanda_ has quit IRC04:41
*** yolanda_ has joined #oooq04:42
*** dsneddon has quit IRC04:44
*** saneax has joined #oooq04:45
*** aakarsh has quit IRC04:59
*** marios has joined #oooq05:11
*** marios_|ruck has joined #oooq05:11
*** dsneddon has joined #oooq05:40
ykarelmarios_|ruck, remember ovb issue i mentioned yesterday?06:11
ykarelit's affecting not just rocky06:12
ykarelit's affecting master,stein too06:12
ykareli added more info in ruckrover etherpad, please check06:12
ykarelas i didn't find a bug for it06:12
marios_|ruckykarel: ack will check and file it thanks06:20
ykarelack06:21
*** ykarel is now known as ykarel|afk06:41
*** tesseract has joined #oooq07:08
*** dsneddon has quit IRC07:12
*** ykarel|afk is now known as ykarel07:16
*** jaosorior has quit IRC07:19
*** amoralej|off is now known as amoralej07:22
*** dsneddon has joined #oooq07:24
*** ccamacho has joined #oooq07:27
*** jtomasek has joined #oooq07:28
*** dsneddon has quit IRC07:29
*** dtantsur|afk is now known as dtantsur07:31
*** kopecmartin|off is now known as kopecmartin07:36
*** holser_ has joined #oooq07:51
*** bogdando has joined #oooq07:53
*** jpena|off is now known as jpena07:59
*** ykarel is now known as ykarel|lunch08:00
marios_|ruckpanda https://review.rdoproject.org/r/#/c/20785/08:11
ykarel|lunchwhy duplicate ^^ https://review.rdoproject.org/r/#/c/20721/08:14
* ykarel|lunch leaving08:14
marios_|ruckykarel|lunch: well obviously i didn't realise there was a duplicate. thanks for pointing it out. looking :D08:14
*** tosky has joined #oooq08:15
*** dsneddon has joined #oooq08:21
*** dsneddon has quit IRC08:26
*** ccamacho has quit IRC08:27
*** derekh has joined #oooq08:32
*** ykarel|lunch is now known as ykarel08:50
*** ccamacho has joined #oooq08:59
*** panda|rover|off is now known as panda|rover09:10
panda|rovermarios_|ruck: how's going ?09:14
marios_|ruckpanda|rover: O/09:15
marios_|ruckpanda|rover: ok ... nothing major so far - known stuff on ether09:15
marios_|ruckpanda|rover: mainly ongoing promotion blockers (one new one via ykarel today)09:16
marios_|ruck but looks like fix on the way09:16
ykarelgetting the fixes will take time, good to be proactive in the fixes i think09:17
ykarelas multiple releases are affected09:17
*** dsneddon has joined #oooq09:23
*** saneax has quit IRC09:33
marios_|ruckhmm problem is we need promotion to get that though right cos python-ironicclient https://review.opendev.org/#/c/659592/09:34
*** saneax has joined #oooq09:35
*** saneax has quit IRC09:42
ykarelmarios_|ruck, we need to get new release with the fixes09:52
ykareli tried to comment in etherpad, may be i didn't explained clearly09:52
ykareland getting new releases will take time,09:53
ykareltill then we can go back to working releases09:53
marios_|ruckykarel: you mean carry the patches from dmitry manually somehow?09:56
marios_|ruckykarel: oh you mean pin09:57
marios_|ruckykarel: on older release09:57
ykarelmarios_|ruck, no, so currently 2.7.1 and 2.5.2 are broken09:57
ykarelthere is already a patch for master openstack/requirements09:57
ykarelwe need to go back to 2.7.0 and 2.5.1 in rdoinfo09:57
ykareluntil new releases with the fixes are in place09:58
marios_|ruckykarel: ack ok thanks will do09:58
*** ykarel is now known as ykarel|afk10:03
marios_|ruckpanda|rover: ykarel|afk please check when you have time thanks https://review.rdoproject.org/r/#/c/20787/10:36
*** jaosorior has joined #oooq10:55
*** ykarel|afk is now known as ykarel10:56
*** jaosorior has quit IRC11:00
*** jaosorior has joined #oooq11:09
*** dsneddon has quit IRC11:16
* marios_|ruck food biab11:18
*** jpena is now known as jpena|lunch11:31
*** marios_|ruck has quit IRC11:41
*** marios has quit IRC11:42
*** jaosorior has quit IRC11:44
*** chem has quit IRC11:51
*** hamzy has quit IRC11:55
*** dsneddon has joined #oooq12:00
*** marios_|ruck has joined #oooq12:07
*** marios has joined #oooq12:08
*** dsneddon has quit IRC12:08
*** ccamacho has quit IRC12:15
marios_|ruckpanda|rover: o/ i see you're digging into https://bugs.launchpad.net/tripleo/+bug/182886012:22
openstackLaunchpad bug 1828860 in tripleo "queens, tempest.api.volume multiple failures" [Critical,Triaged]12:22
marios_|ruckpanda|rover: i was looking in logs earlier but didn't find much.. .couple red herrings like ovs not connecting or rabbit but if you check the service logs it eventually connects12:22
marios_|ruckpanda|rover: reading latest comments12:22
marios_|ruckpanda|rover: oh its a different bug sorry12:23
marios_|ruckpanda|rover: you're looking at https://bugs.launchpad.net/tripleo/+bug/1818060 - it isn't new though but it wasn't on the ether12:24
openstackLaunchpad bug 1818060 in tripleo "Nodes periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master fail to get usable IPs though os-net-config with Error, some other host (BE:E5:4F:B9:21:B0) already uses address" [Critical,In progress] - Assigned to Gabriele Cerami (gcerami)12:24
marios_|ruckpanda|rover: is it from previous sprint maybe12:24
panda|rovermarios_|ruck: yep, that's a long running bug taht is tainting our success rate12:24
marios_|ruckpanda|rover: ack thanks12:25
*** chem has joined #oooq12:26
*** udesale has quit IRC12:30
*** udesale has joined #oooq12:31
*** dsneddon has joined #oooq12:36
*** jpena|lunch is now known as jpena12:36
*** rlandy has joined #oooq12:40
kopecmartinarxcruz, i guess there's no meeting today, is it?12:51
*** ratailor has quit IRC12:54
*** ratailor has joined #oooq12:56
weshaykopecmartin nope, I canceled and sent a notice12:57
weshaykopecmartin arxcruz I would like to change fs20 to have to compute nodes12:58
weshays/to/two12:58
*** LaurentDumont has quit IRC12:59
*** amoralej is now known as amoralej|lunch13:00
arxcruzweshay: kopecmartin will we ?13:03
*** jaosorior has joined #oooq13:11
marios_|ruckarxcruz: kopecmartin chkumar|pto are you folks aware of that https://bugs.launchpad.net/tripleo/+bug/1828860 - queens promote blocker tempest - i think maybe rabbit so trying to sell it to pidone for a look for now but fyi you should be aware of that13:23
openstackLaunchpad bug 1828860 in tripleo "queens, tempest.api.volume multiple failures" [Critical,Triaged]13:24
arxcruzmarios_|ruck:no, i wasn't aware13:24
arxcruzi'll take a look13:24
arxcruzmarios_|ruck: oh, this one is pretty simple13:25
arxcruzthe service cinder-backup wasn't started13:25
arxcruz:)13:25
*** ccamacho has joined #oooq13:25
marios_|ruckarxcruz: oh cool you mean there is a known issue there?13:25
arxcruzmarios_|ruck:for a long time, so, it requires the (back in the old days without containers) the service cinder-backup13:25
rfolcoweshay, overcloud-agent element is missing in dib, do you know which package provides it ?13:27
weshayrfolco show me13:28
weshayrfolco in my tests it worked on the first try13:28
rfolcoweshay, I followed your steps and it failed13:28
weshayperhaps you have something off in your dib repos? typo etc?13:29
arxcruzweshay: 1-1?13:30
rfolcoweshay,13:30
rfolcoERROR: Element 'overcloud-agent' not found in '/usr/share/tripleo-puppet-elements:/usr/share/instack-undercloud:/usr/share/tripleo-image-elements:/usr/bin/../share/diskimage-builder/elements'13:30
rfolco# rpm -qa |grep elements13:31
rfolcoopenstack-tripleo-image-elements-0.9.6-11.el7ost.noarch13:31
rfolcoopenstack-tripleo-puppet-elements-0.0.1-6.el7ost.noarch13:31
weshayrfolco why are you using osp?13:34
rfolcoweshay, I followed steps from https://docs.google.com/document/d/1vicjVYZik1H7T4zqGpKElTHlYlq7V70t4vko_Nw2pBI/edit#13:35
weshaythere is some mistake, not sure where yet13:35
weshay[root@hp-dl320g8-10 ~]# rpm -qa | grep openstack-tripleo | grep elements13:36
weshayopenstack-tripleo-image-elements-9.1.1-0.20190420053043.aa75390.el7.noarch13:36
weshayopenstack-tripleo-puppet-elements-9.0.2-0.20190425202749.1ab58f2.el7.noarch13:36
weshayon my box13:36
rfolcoyour instructions say:13:37
rfolcoexport USE_DELOREAN_TRUNK=013:37
rfolco# export DELOREAN_TRUNK_REPO="https://trunk.rdoproject.org/centos7-rocky/current-tripleo/"13:37
rfolco# export DELOREAN_REPO_FILE="delorean.repo"13:37
rfolcoassuming these # are mistakes then13:37
weshaynope13:37
weshaywhat was your rhos-release invocation?13:38
rfolcorhos-release -x13:38
rfolcorhos-release -P rdotrunk-rocky -p current-tripleo13:38
weshayls /etc/yum.repos.d/13:38
rfolco# ls /etc/yum.repos.d/13:39
rfolcoredhat.repo                                     rhos-release-rdotrunk-rocky-deps.repo      rhos-release-rdotrunk-rocky.repo  rhos-release-rhel-7.6.repo13:39
rfolcorhos-release-rdotrunk-rocky-ceph-luminous.repo  rhos-release-rdotrunk-rocky-opstools.repo  rhos-release.repo13:39
weshayecho $DIB_YUM_REPO_CONF13:40
rfolcoempty13:40
rfolcolet me export again13:41
weshaythat's likely the issue13:41
weshayand the osp packages13:42
rfolco# export DIB_YUM_REPO_CONF="/etc/yum.repos.d/delorean-rocky-deps.repo /etc/yum.repos.d/rhos-release-rhel-7.6.repo /etc/yum.repos.d/rhos-release-rdotrunk-rocky-opstools.repo /etc/yum.repos.d/rhos-release-rdotrunk-rocky-ceph-luminous.repo"13:42
rfolcoERROR: Element 'overcloud-agent' not found in '/usr/share/tripleo-puppet-elements:/usr/share/instack-undercloud:/usr/share/tripleo-image-elements:/usr/bin/../share/diskimage-builder/elements'13:43
*** vinaykns has joined #oooq13:43
*** aakarsh has joined #oooq13:44
weshayrfolco we'll tmate in a minute13:44
rfolcok thx13:44
*** amoralej|lunch is now known as amoralej13:54
*** tcw has quit IRC13:57
rlandyweshay: when you have time this afternoon, https://review.opendev.org/659794 Replace dict combine in standalone containers-prepare-parameters13:57
rlandy^^ need to discuss multiple replacements13:57
rfolcoweshay, works fine now, need to update packages after rhos-release setup, since I came from rhel7.3 and upgraded to 7.613:58
ykarelmarios_|ruck, weshay, this should also be https://bugs.launchpad.net/tripleo/+bug/1828859 promotion blocker13:58
openstackLaunchpad bug 1828859 in tripleo "[queens] Unhandled error: NoOptionError: No option 'tempest_roles' in section: 'auth'" [Critical,Triaged]13:58
ykarelweshay, ^^ was i talking that day with eggs and you13:59
ykareli thought it's already moved to CIX13:59
rfolcoweshay, still failing, but I passed that point, will investigate, thx13:59
marios_|ruckykarel: looking14:07
weshayrfolco if you want to compare your setupo to mine14:08
weshayrfolco hp-dl320g8-10.lab.eng.brq.rerdhat.com14:08
rfolcoweshay, I think you used a modified version of rhos-release coz yours define delorean.repo and mine rhos-release.repo14:08
marios_|ruckykarel: are you seeing that today somewhere or you just noticed the bug?14:09
rfolcoweshay, now I got 2019-05-17 14:07:44.170 | rhos-release-rdotrunk-rocky.repo is not a valid yum repo configuration file.14:09
ykarelmarios_|ruck, it's continuosly failing from last couple of days14:09
marios_|ruckykarel: thx14:09
rfolcoweshay, will compare, thx14:09
ykarelmarios_|ruck, https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-queens14:09
ykarelsame time with queens fs020 and with tempestconf 2.2.014:10
marios_|ruckykarel: ack and https://github.com/rdo-infra/ci-config/blob/4fa8eda72d664733acbe9f2650405592ef68affb/ci-scripts/dlrnapi_promoter/config/CentOS-7/queens.ini#L2414:10
marios_|ruckykarel: thanks gonna comment and add into our tracking. this must be from previous ruck|rover14:10
ykarelyes14:10
*** Goneri has joined #oooq14:14
*** ykarel is now known as ykarel|afk14:16
*** tcw has joined #oooq14:19
rlandymarios_|ruck: panda|rover: sshnaidm|afk: thanks for reviews - updated https://review.opendev.org/#/c/648519/14:20
rlandypanda|rover: doing follow up patch for zuul vars14:20
rlandymay need some more scrutiny14:20
marios_|ruckrlandy: ack afaics v30..v31 removes extras-common14:22
marios_|ruckhttps://review.opendev.org/#/c/648519/30..31/roles/check-to-build-or-not-to-build/meta/main.yml14:22
marios_|ruckthat is the question14:22
marios_|ruckrlandy: revoted14:23
rlandymarios_|ruck: ack - per sshnaidm|afk's suggestion14:25
marios_|ruckrlandy: ack14:25
*** dsneddon has quit IRC14:26
panda|roverand I thought I was very original14:28
panda|rover2B || ~2B14:29
panda|rover= FF14:29
*** ksambor has quit IRC14:30
weshayrfolco join my blue and setup a tmate please14:31
rfolcoweshay, sec14:32
rfolcoweshay, ready, I don't think its necessary. I fixed it.14:34
rfolco2019-05-17 14:33:19.449 | Build completed successfully14:34
weshayrfolco let's chat anyway14:34
rfolcoweshay, ok14:34
weshayand tmate14:34
weshayrfolco press prefix + :14:37
weshaytmate display -p '#{tmate_ssh}'14:37
marios_|ruckweshay: did you still want to catch up about the rhel stuff today14:38
weshayrfolco14:40
weshay07:31 <rfolco>  openstack-tripleo-image-elements-0.9.6-11.el7ost.noarch14:40
weshay07:31 <rfolco>  openstack-tripleo-puppet-elements-0.0.1-6.el7ost.noarch14:40
*** jaosorior has quit IRC14:41
panda|roverarxcruz: ping14:41
panda|roverrlandy: is it still usefule to recheck https://code.engineering.redhat.com/gerrit/#/c/165815/ ?14:43
rlandypanda|rover: yeah - it's the only way to test baseos role14:44
rlandylats run master, rocky, stein were all failing undercloud deploy14:45
rlandyas were ovb jobs14:45
rlandyso I am hanging on it to restest14:45
rlandyretest14:45
panda|roverok14:45
* rlandy checks master status now14:45
rlandylooks like I can recheck master today14:46
rlandyand rocky - good14:47
*** dtantsur is now known as dtantsur|afk14:52
*** bogdando has quit IRC15:00
panda|roverarxcruz: ?15:04
weshayrfolco review.rdoproject.org/config/playbooks/tripleo-ci-periodic-base-upload/post.yaml@master]15:05
weshayhttp://logs.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-stein-upload/2b91b9e/job-output.txt.gz15:05
rlandypanda|rover" removing ZUUL_CHANGES - not as easy as it seems15:05
rlandydirect replacement?15:05
weshayrfolco https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/playbooks/tripleo-ci-periodic-base-upload/post.yaml15:09
panda|roverrlandy: from zuul-variables.yaml  there's zuul.items[x].project.short_name  that serems to have what you need15:10
weshaymarios_|ruck ok ready15:12
*** dsneddon has joined #oooq15:13
weshaymarios_|ruck https://bluejeans.com/u/whayutin/15:13
marios_|ruckweshay: ack gimme 2 misn joining15:14
*** ykarel|afk is now known as ykarel|away15:15
*** Vorrtex has joined #oooq15:17
*** ratailor has quit IRC15:17
*** dsneddon has quit IRC15:19
rlandyhttps://review.opendev.org/#/c/659822 - let's see what that does15:31
marios_|ruckweshay: https://sf.hosted.upshift.rdu2.redhat.com/logs/40/167540/23/check/tripleo-ci-rhel-8-standalone-rhos-15/6db5809/zuul-info/inventory.yaml15:33
rfolcoweshay, ping me when you are free15:36
rfolcomarios_|ruck, mutex_release(wes)15:36
rfolcono pressure :)15:37
arxcruzpanda|rover:pong sorry the delay15:37
arxcruzpanda|rover: i did not check yet the tempestmail doing it right now15:37
arxcruzit seems my credentials are not working15:38
panda|roverarxcruz: I have another question15:38
arxcruzpanda|rover: 4215:39
panda|roverarxcruz: perfect, thanks.15:39
arxcruz:D15:39
marios_|rucksprint 9 manual rhel7 image build https://tree.taiga.io/project/tripleo-ci-board/task/1051 weshay15:41
arxcruzpanda|rover: whats up15:41
panda|roverarxcruz: what SMTP are you using to send emails ?15:42
arxcruzpanda|rover: i'm using google api15:42
weshaymarios_|ruck https://docs.google.com/document/d/1vicjVYZik1H7T4zqGpKElTHlYlq7V70t4vko_Nw2pBI/edit#15:44
arxcruzpanda|rover:but it's gmail smtp15:45
*** dsneddon has joined #oooq15:46
panda|roverarxcruz: so, no open relays ? :)15:46
arxcruzpanda|rover: nope15:46
panda|roverarxcruz: bummer15:51
arxcruzsorry :(15:51
weshayrfolco ok.. available if you are15:57
* marios_|ruck shutdown sequence panda|rover fyi already did hand off to weshay 15:57
rfolcoweshay, joining15:57
*** ykarel|away has quit IRC15:58
*** rlandy is now known as rlandy|biab15:58
panda|rovermarios_|ruck: have a mice weekend15:59
panda|rovereek eek16:00
marios_|ruckpanda|rover: no mice !16:00
panda|rovermarios_|ruck: pika pika16:00
*** panda|rover is now known as panda|rover|off16:06
*** marios_|ruck has quit IRC16:08
*** marios has quit IRC16:08
*** holser_ has quit IRC16:08
*** tesseract has quit IRC16:17
*** tesseract has joined #oooq16:18
*** tcw has quit IRC16:21
*** rlandy|biab is now known as rlandy16:30
*** tcw has joined #oooq16:49
*** kopecmartin is now known as kopecmartin|off16:52
*** tesseract has quit IRC16:58
*** tesseract has joined #oooq16:59
weshayrfolco https://teams.fedoraproject.org/discover17:00
weshayrfolco https://teams.fedoraproject.org/project/new/import17:00
*** derekh has quit IRC17:02
*** amoralej is now known as amoralej|off17:10
*** udesale has quit IRC17:15
*** ykarel|away has joined #oooq17:17
*** tesseract has quit IRC17:17
*** jpena is now known as jpena|off17:22
rlandyweshay: hey - is deploying the overcloud w/o net-iso supported in master?17:29
rlandylooking at a 1ctrl_1compute deploy17:30
weshayrlandy should be17:30
rlandy   "ERROR:nova_wait_for_placement_service:Failed to get placement service endpoint!",17:31
rlandyjuts checking if networksetup is ok17:31
rlandyjust17:31
weshayrlandy there have been a lot of changes and breakages around nova placement17:32
weshaywhere are you seeing that?17:32
rlandymaybe it's not  perf specific17:32
rlandyhttps://sf.hosted.upshift.rdu2.redhat.com/logs/31/169131/8/check/tripleo-ci-base-baremetal-microcloud-ha-2/1137ccd/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2019-05-15_23_50_0117:32
rlandyweshay: ^^ perf/browbeat17:32
rlandybut there is not external network defined there17:32
rlandyso going to define that17:32
rlandyweshay: this is like the 'choose your own adventure' version of networking17:33
weshayrlandy is this on a current-tripleo or tripleo-ci-testing?17:34
rlandyhome/zuul/workspace/.quickstart/config/release/tripleo-ci/CentOS-7/master.yml17:35
rlandyso current-tripleo17:36
rlandydid that updated recently?17:36
rlandypromoted? this is a run from night of 05/1517:36
* rlandy updates network and tries again17:36
weshayno we have not promoted in a few days17:38
weshayrlandy any other ovb or bm hitting that?17:40
rlandyweshay: not that I see but17:40
rlandythis is 1ctrl 1comp deploy17:40
rlandyonly other I know if fs02017:40
rlandywhich runs on promotion only17:40
rlandyexternal net is dicing17:40
rlandytrying again with new definition17:41
rlandyotherwise will try tripleo-ci-testing17:41
weshayrlandy you'd be better off trying previousl-current-tripleo17:41
weshayprevious > tripleo-ci-testing17:42
rlandyweshay: k - if I have no luck with this run17:42
weshayrlandy can we get on that box?17:42
rlandyweshay: ack - browbeat17:42
rlandybut I just restaretd job17:42
weshay2019-05-15 23:50:01 |         "DEBUG:keystoneauth.session:Request returned failure status: 503",17:42
rlandywill wipe the undercloud17:42
weshayrlandy I think you have a keystone issue there17:43
rlandytls17:43
rlandyI think I have a key issue maybe17:43
rlandymy other thought17:43
rlandywill check when this run progresses17:44
weshayrlandy k.. hold the env next time17:45
rlandyack17:50
*** dsneddon has quit IRC17:51
*** dsneddon has joined #oooq17:57
*** tcw has quit IRC18:10
*** ykarel|away has quit IRC18:20
weshayrlandy do you have a minute?18:27
rlandyweshay: sure18:27
rlandyweshay: passing rhos-14 :)18:27
weshaynice18:27
rlandyweshay: you want me to review something?18:28
weshayrlandy no.. have some questions18:28
rlandybj?18:29
weshayya please18:29
rlandyjoining'18:29
*** Vorrtex has quit IRC18:52
weshayrlandy https://code.engineering.redhat.com/gerrit/#/c/167540/23/zuul.d/standalone-jobs.yaml18:53
*** hamzy has joined #oooq19:35
*** aakarsh has quit IRC20:27
*** vinaykns has quit IRC20:51
rlandyweshay: https://bugs.launchpad.net/tripleo/+bug/182810320:55
openstackLaunchpad bug 1828103 in tripleo "install-deps.sh fails on rhel-8 - fails to install python2-libselinux" [Undecided,Triaged]20:55
*** rlandy has quit IRC22:31
*** tosky has quit IRC22:42

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!