Friday, 2019-05-24

*** LaurentDumont has joined #oooq00:05
*** ykarel has joined #oooq01:11
*** ykarel has quit IRC01:15
*** LaurentDumont has quit IRC01:45
*** yolanda has quit IRC01:51
*** yolanda has joined #oooq01:54
*** ykarel has joined #oooq01:55
*** altlogbot_0 has quit IRC02:12
*** altlogbot_2 has joined #oooq02:12
*** ykarel has quit IRC02:15
*** dsneddon has quit IRC03:20
*** ykarel has joined #oooq03:26
*** dsneddon has joined #oooq03:45
*** ykarel has quit IRC04:08
*** aakarsh has joined #oooq04:09
*** ykarel has joined #oooq04:24
*** raukadah is now known as chandankumar04:59
*** dsneddon has quit IRC05:21
*** marios has joined #oooq05:29
*** marios_|ruck has joined #oooq05:29
*** quiquell has joined #oooq05:48
*** dsneddon has joined #oooq06:16
*** holser_ has joined #oooq06:20
*** dsneddon has quit IRC06:21
*** jfrancoa has joined #oooq06:29
chandankumarmarios_|ruck: hello06:47
chandankumarmarios_|ruck: we fixed queens member roles06:47
chandankumarjobs passing in promotion pipeline06:47
chandankumarfs020 queens afailures are again back06:47
marios_|ruckchandankumar: i saw there was some update on kopecmartin|afk patch?06:47
marios_|ruckchandankumar: i mean it wasn't merged and there was some issue like pep/tox fail06:48
* marios_|ruck checks again06:48
chandankumarmarios_|ruck: we have taken a different approach all toagther06:48
chandankumarmarios_|ruck: https://review.rdoproject.org/r/#/c/20909/06:48
*** yolanda has quit IRC06:49
marios_|ruckchandankumar: so we don't want that? https://review.opendev.org/#/c/66074806:49
chandankumarmarios_|ruck: we need that06:49
*** yolanda has joined #oooq06:49
chandankumarmarios_|ruck: but it will come with more polished, in a new tempestconf tag06:49
chandankumarwhich will take time06:49
chandankumarwe have cherry -picked first patchset in the spec file to unblock the ci06:50
chandankumarwhich is working in our jobs06:50
marios_|ruckchandankumar: so short term https://review.rdoproject.org/r/#/c/20909/ but longer term fix https://review.opendev.org/#/c/66074806:50
chandankumarmarios_|ruck: yes06:50
chandankumarmarios_|ruck: it will not cause any regression in downstream jobs also06:50
marios_|ruckchandankumar: ack thanks for update06:51
ykarelmarios_|ruck, hi06:53
marios_|ruckykarel: o/06:54
ykarelmarios_|ruck, can you check failed ovb stacks06:54
ykarelmarios_|ruck, and also collect some info before cleaning those, like port list06:54
marios_|ruckykarel: we have a ticket already at https://tickets.osci.io/rt/Ticket/Display.html?id=176806:54
marios_|ruckykarel: or we need a new one?06:54
marios_|ruckykarel: but didn't get chance to cycle back to that (weshay opened it last week).06:55
marios_|ruckykarel: or lots of fails today/yesterday/new thing?06:55
ykarelmarios_|ruck, how many stacks are there with ^^ ticket?06:55
ykarelfrom cockpit i saw 5 stack fails06:55
marios_|ruckykarel: ok, let me finish what am doing and will look into that06:55
chandankumarmarios_|ruck: I am looking with ykarel06:59
chandankumarthanks :-)06:59
*** ratailor has joined #oooq07:06
*** dsneddon has joined #oooq07:15
marios_|ruckykarel: chandankumar ack are you on a call?07:18
marios_|ruckykarel: there are 6 stacks in that ticket07:18
ykarelmarios_|ruck, ack, me and chandankumar sit in same office: Pune07:19
marios_|ruckykarel: yeah i know but sometimes folks work from home or don't directly sit next to each other :D07:20
*** dsneddon has quit IRC07:20
*** amoralej|off is now known as amoralej07:22
ykarelmarios_|ruck, hmm possible :)07:23
ykarelmarios_|ruck, i ran browbeat fs053 job with rally-openstack master and it passed:- https://review.opendev.org/#/c/661180/07:23
marios_|ruckykarel: chandankumar so did you collect logs is there something for me to do there07:23
marios_|ruckykarel: yah i'm sorry about that07:23
marios_|ruckhttps://bugs.launchpad.net/tripleo/+bug/182855907:23
openstackLaunchpad bug 1829030 in Rally "duplicate for #1828559 [rally 1.5.0] rally deployment create --fromenv creates wrong spec for openstack platform" [High,Fix released] - Assigned to Andrey Kurilin (andreykurilin)07:23
marios_|ruckykarel: i have been trying to find time to get to it all week07:24
marios_|ruckykarel: today :D :/07:24
marios_|ruckykarel: so their fix has been released looks like07:24
ykarelmarios_|ruck, np, you can pick whenever free, i just tried to narrow down it07:24
marios_|ruckykarel: is that what you meant07:24
*** jtomasek has joined #oooq07:25
ykarelmarios_|ruck, currently rally-openstack 1.4.0 is used which is failing07:26
ykareland 1.4.0 is last releae currently https://github.com/openstack/rally-openstack/releases07:26
marios_|ruckykarel: right i see you confirmed that th efix (from master) is ok though07:26
ykarelmarios_|ruck, yes, so we need a new release of rally-openstack07:27
ykareland update rally_openstack_version: <new release>07:27
*** apahim_ has joined #oooq07:27
marios_|ruckykarel: ack ... wondering if folco meant to file that against tripleo but filed it against rally by mistake (title is 'fs 053' tripleo specific) anyway07:28
ykarelhmm may be i didn't checked that07:28
*** apahim_ has quit IRC07:29
*** apahim_ has joined #oooq07:29
*** apahim has quit IRC07:30
*** dtantsur|afk is now known as dtantsur07:30
*** hrybacki has quit IRC07:33
chandankumarmarios_|ruck: https://logs.rdoproject.org/openstack-periodic-24hr/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-queens/7865943/logs/tempest.html.gz07:35
chandankumarqueens periodic passed fs01707:35
marios_|ruckchandankumar: nice ok so it just didn't report yet? i dont' see it in https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-queens07:37
*** hrybacki has joined #oooq07:39
chandankumarmarios_|ruck: once the periodic runs completed then it reports to zuul dashboard07:40
marios_|ruckchandankumar: ack so is ongoing you mean07:41
chandankumarsshnaidm: panda marios_|ruck dstat upstream deprecated replaced with new project https://www.redhat.com/en/blog/implementing-dstat-performance-co-pilot07:41
*** hrybacki has quit IRC07:44
*** hrybacki has joined #oooq07:45
*** jpena|off is now known as jpena07:50
*** apahim_ has quit IRC07:51
*** apahim_ has joined #oooq07:52
*** jtomasek has quit IRC08:02
*** hrybacki has quit IRC08:09
*** bogdando has joined #oooq08:09
*** hrybacki has joined #oooq08:09
*** tosky has joined #oooq08:10
*** apetrich has quit IRC08:12
*** apetrich has joined #oooq08:13
*** dsneddon has joined #oooq08:15
*** zbr has joined #oooq08:19
chandankumarzbr: Hello08:21
*** ykarel is now known as ykarel|lunch08:21
zbrchandankumar: hi.08:22
chandankumarzbr: https://docs.google.com/spreadsheets/d/1HMdPAiohKEfSeIosz0RzFBjdf4Qc9fZiuFIfjBzEQwc/edit#gid=1187359835 -> on resolved section08:22
chandankumartempest.api.compute.servers.test_server_actions.ServerActionsTestJSON -> related to this bug https://bugzilla.redhat.com/show_bug.cgi?id=171095508:22
openstackbugzilla.redhat.com bug 1710955 in openstack-tripleo "tenant_network_type is misconfigured as vxlan even though gre was chosen" [High,Closed: notabug] - Assigned to beagles08:23
*** jtomasek has joined #oooq08:24
chandankumarzbr: do you have logs for this test tempest.api.volume.test_volumes_clone_negative.VolumesCloneNegativeTest.test_create_from_volume_decreasing_size ?08:28
*** kopecmartin|afk is now known as kopecmartin08:29
*** bogdando has quit IRC08:33
*** bogdando has joined #oooq08:34
zbrchandankumar: nope08:36
zbrchandankumar: ready for a quick bj?08:37
chandankumarzbr: yes sir08:37
zbrhttps://bluejeans.com/265541792808:37
chandankumarzbr: http://pastebin.test.redhat.com/76604208:38
chandankumarzbr: https://tree.taiga.io/project/tripleo-ci-board/task/108008:42
*** tesseract has joined #oooq08:48
*** ccamacho has joined #oooq08:56
*** derekh has joined #oooq08:57
*** apahim has joined #oooq09:04
*** apahim_ has quit IRC09:07
marios_|ruckpanda: updated https://review.opendev.org/#/c/661030/3 whn you get a chance thanks - you're right i posted it in a rush before leaving, we only want the upgrade jobs there09:10
*** ykarel|lunch is now known as ykarel09:11
*** saneax has joined #oooq09:13
*** dsneddon has quit IRC09:18
*** apahim_ has joined #oooq09:20
*** apahim has quit IRC09:24
ykarelmarios_|ruck, chandankumar so only fs020 failed in queens promotion pipeline09:27
ykarelsome tempest test failed related to backup, can you guys promote queens skipping fs020 from promotion criteria09:28
ykarelpromotion is needed to clear upstream gate jobs09:28
marios_|ruckykarel: ack ... do you have pointer? i'll post the skip for fs020 sec09:29
ykarelmarios_|ruck, https://logs.rdoproject.org/openstack-periodic-24hr/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-queens/b5ffa14/logs/tempest.html.gz09:29
ykarelmarios_|ruck, instead of skip need to fix it09:30
ykarelissue is tests are running when backup service is not enabled09:30
ykarelthere must be already bug for it, as it was seeen few days back also09:30
marios_|ruckykarel: ok, i thought that's what you meant by "skipping fs020 from promotion criteria"09:30
*** holser_ is now known as holser|lunch09:30
marios_|ruckykarel: so we can promote queens w/out it today09:31
ykarelmarios_|ruck, yes i mean ^^09:31
ykarelby removing https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/dlrnapi_promoter/config/CentOS-7/queens.ini#L2709:32
marios_|ruckykarel: right ok so we're saying same09:33
marios_|ruckykarel: /me was confused when you said 12:30 < ykarel> marios_|ruck, instead of skip need to fix it09:33
ykarelmarios_|ruck, okk sorry for creating confusion09:34
marios_|ruckykarel: ack np ... obv we have to fix it but just making sure i understood what you said about the skip ok posting sec09:34
marios_|ruckand will file a bug for that tempest if we don't have one09:34
marios_|ruckchandankumar: are you aware of a bug for that fs20 issue already /me can't find one quickly09:37
marios_|ruckchandankumar: https://logs.rdoproject.org/openstack-periodic-24hr/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-queens/b5ffa14/logs/tempest.html.gz09:37
marios_|rucktempest.api.volume.admin.test_volumes_backup.VolumesBackupsAdminTest09:37
ykarelbackup = False is needed in Tempestconf09:41
ykarelunder volume-feature-enabled09:41
marios_|ruckykarel: ack ok but will file the bug so i can reference it in the skip so we don't forget to unskip09:42
ykarelmarios_|ruck, ack,09:43
chandankumarmarios_|ruck: sorry I was drafting ane email09:48
chandankumarzbr: replied the email osp-15 one,09:48
marios_|ruckchandankumar: np here https://bugs.launchpad.net/tripleo/+bug/183035109:48
openstackLaunchpad bug 1830351 in tripleo "periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-queens fails tempest run for tempest.api.volume.admin.test_volumes_backup test group" [Critical,Triaged]09:48
marios_|ruckchandankumar: if we have one already can close np09:48
chandankumarzbr: I have attached few infrared failures link09:48
chandankumarfrom few tests09:49
chandankumarwe can take a look if needed or we want to cleanup the skip tests for stein09:49
chandankumarzbr: feel free to add more info to the email anything missed09:49
marios_|ruckykarel: chandankumar https://review.rdoproject.org/r/#/c/20920/09:50
marios_|ruckTengu: did you and zbr talk (about molecule?) he's back :)09:52
zbrmarios_|ruck: not yet but there is a meeting today, in ~5h from how. -- still I can talk about it at any time.09:53
*** apahim_ has quit IRC09:53
marios_|ruckzbr: i just remembered Tengu was asking us about molecule few days back09:53
ykarelmarios_|ruck, +109:54
*** apahim_ has joined #oooq09:55
*** tesseract has quit IRC09:56
*** tesseract has joined #oooq09:59
*** tesseract has quit IRC10:00
*** tesseract has joined #oooq10:01
*** apahim_ has quit IRC10:10
*** apahim_ has joined #oooq10:16
chandankumarsshnaidm: Hello10:20
chandankumarsshnaidm: panda can we merge this https://review.rdoproject.org/r/#/c/20902/-> https://review.rdoproject.org/r/#/c/20898/ -> https://review.rdoproject.org/r/#/c/20919/10:21
chandankumarin sequence10:21
*** holser|lunch is now known as holser_10:34
*** panda has quit IRC10:38
chandankumarsshnaidm: marios_|ruck can we merge this https://review.rdoproject.org/r/#/c/20920/ if you have super powers10:42
marios_|ruckchandankumar: i do not10:43
*** panda has joined #oooq10:46
*** panda is now known as panda|rover10:48
chandankumarpanda|rover: please merge this patch https://review.rdoproject.org/r/#/c/20902/10:51
chandankumarpanda|rover: sorry this one https://review.rdoproject.org/r/#/c/20920/10:51
panda|roverchandankumar: can I review before merging it ?10:53
chandankumarpanda|rover: feel free to review it10:53
*** dsneddon has joined #oooq11:00
chandankumarpanda|rover: any example of checkout-override11:08
chandankumar?11:08
panda|roverchandankumar: sorry, override-checkout, you were using this before Sagi's comment11:10
*** tesseract has quit IRC11:10
chandankumarpanda|rover: ok, restoring my changes then11:11
panda|roverchandankumar: but you still have to answer his question11:11
chandankumarpanda|rover: commented back11:16
zbrpanda|rover: marios_|ruck : how are we supposed to test python3 code on rhel7 when default python3 is 3.4 which was dropped by pip in march?11:38
zbrI got: DEPRECATION: Python 3.4 support has been deprecated. pip 19.1 will be the last one supporting it. Please upgrade your Python as Python 3.4 won't be maintained after March 2019 (cf PEP 429).11:38
zbrwhile trying to run a "pip freeze"11:38
panda|roverzbr: and pip freeze doesn't work ?11:39
zbrnot sure, but returned nothing which was unexpected. going to test more.11:39
zbri think we do have a support issue, especially as we have lots of tox targets using basepython=python3 .... which happens to be 3.411:40
*** jpena is now known as jpena|lunch11:40
zbri can obviously switch to f28 machine to run them from there but...11:40
zbrpip is a never ending source of surprises11:41
panda|roverzbr: why are you teting python3 in rhel 711:42
zbrpanda|rover: because of https://github.com/openstack/tripleo-ci/blob/master/tox.ini#L2511:43
panda|roverzbr: I think you'll need to add a linters-legacy, and set python2 as base python there, then in all the centos/rhel7 jobs there you don't use linters, but linters-legacy11:45
zbrpanda|rover: i am working now on a fix for that issue, give me few minutes.11:45
zbrmolecule cannot be installed on python34 because it uses a version of ansible-lint which is not available on py34. nice surprise.11:46
panda|roverwe are obsolete11:46
zbrpanda|rover: speak for yourself, I still see myself 20s-ish ;)11:47
*** ratailor has quit IRC11:47
*** ratailor has joined #oooq11:48
panda|roverzbr: not even 20, you're just 16 + VAT .11:49
zbrpanda|rover: yep. btw, please wf https://review.opendev.org/#/c/655659/11:50
*** rfolco is now known as rfolco|brb11:51
chandankumararxcruz: Hello11:54
chandankumararxcruz: How we run check skip list script?11:54
arxcruzchandankumar: sorry ?11:55
chandankumararxcruz: https://github.com/openstack/tripleo-quickstart-extras/tree/master/roles/validate-tempest/files/check-skip-list11:56
arxcruzchandankumar:i don't remember11:57
chandankumartrying from commit message11:57
arxcruzcheck_skip.py --log-url http://log.o.o/tempest.log --skip-file11:57
arxcruzpath/to/skip/list11:57
chandankumararxcruz: perfect https://tree.taiga.io/project/tripleo-ci-board/task/1084?kanban-status=144727512:02
chandankumararxcruz: thanks!12:02
arxcruzchandankumar: the idea is to make it automated but yeah, sometimes i write stuff that actually works :D12:03
chandankumararxcruz: I am have a big idea12:03
chandankumartest matrix12:03
chandankumarwe can talk later some day12:03
*** dsneddon has quit IRC12:03
*** jaosorior has joined #oooq12:06
*** sshnaidm is now known as sshnaidm|afk12:06
chandankumararxcruz: kopecmartin please add the agenda here https://etherpad.openstack.org/p/tripleo-tempest-squad-meeting12:12
chandankumarfor tempest meeting12:12
arxcruzchandankumar: all my thoughts are in go to a biergarten :D12:13
Tengumarios_|ruck: zbr heya! I'm on PTO until Tuesday - I'll come back to you next week :).12:14
*** derekh has quit IRC12:15
marios_|ruckTengu: nice :D enjoy12:15
marios_|ruckits totally fine. we'll stay here and work12:15
Tengu:)12:15
Tengugetting married takes some time :312:15
marios_|ruckTengu: ah congrats!12:16
marios_|ruckany reviews need reviews?12:16
* Tengu start feeling some stress for tomorrow12:16
marios_|ruckTengu: tomorrow is the big day?12:16
Tengumarios_|ruck: hmmm nope, nothing really urgent12:16
Tenguyeah12:16
marios_|ruckTengu: just remember, if whisky doesn't solve all your problems then you're just not using enough of it!12:17
marios_|ruckTengu: you'll be fine - having some kind of party also?12:17
Tengumarios_|ruck: whisky is good :). We have plenty here (my wife does like it as well). and yeah, we throw some small party, mostly FOOD12:18
marios_|ruckpanda|rover: rfolco|brb chandankumar sshnaidm|afk need reviews?12:18
Tengubecause food is good12:18
chandankumarTengu: congrats :-)12:18
marios_|ruckTengu: nice :D have fun then12:18
Tenguchandankumar: thanks :)12:18
chandankumarTengu: try some heena on your hand :-)12:18
Tengumarios_|ruck: we'll try. just hoping we'll not get all the thunderstorm announced, we'll be on our motorcycles :D12:19
panda|rovereverybody is getting married12:19
Tenguchandankumar: not really a big fan, although some are really beautiful :). Not sure it would fit my hands though ;)12:19
panda|roverlike a virus12:19
Tenguit's spreading12:19
panda|roverones gets married, then everyone gets married.12:19
panda|roverone gets kids, all other get kids12:20
Tenguah, nah, no kids.12:20
Tenguwe already have a cat, and will probably get a dog soon. the kind of dog that can climb on a motorcycle :)12:20
panda|roverthat's what I was saying too says.12:20
panda|rovers/says//12:21
Tengu:D12:21
panda|rovereven the morning at the maternity ward. "I said no kids!"12:23
Tengunah. no kid, that's for sure. we're both on the same line regarding this topic :)12:23
Tenguanyway. have to go ;). see you next week folks!12:25
panda|rovernot fair. everyone should get his share of suffering12:25
Tengupanda|rover: I'm not crazy ;)12:25
panda|roverTengu: cheers.12:25
Tengu(well, not THAT crazy at least ;))12:26
chandankumarmarios_|ruck: arxcruz https://review.opendev.org/#/c/661265/12:28
chandankumarI am not sure what to put in commit message details here12:29
marios_|ruckchandankumar: nice12:29
marios_|ruckchandankumar: good to remove things there12:29
chandankumarmarios_|ruck: I will update other branches also in night12:30
*** rlandy has joined #oooq12:35
*** jpena|lunch is now known as jpena12:38
weshaychandankumar we don't need to track pike etc.. anymore12:47
chandankumarweshay: I am ignoring pike12:53
chandankumarwill update queens, rocky and master also12:53
weshayrock on12:53
chandankumarpanda|rover: marios_|ruck do we have a docs on retiring tripleo project?13:00
*** dsneddon has joined #oooq13:00
chandankumarcontext tripleo-ui email on openstack-discuss list13:00
panda|roverchandankumar: what kind of information are you looking for ?13:00
marios_|ruckchandankumar: yeah not aware of docs on that though13:00
chandankumarlike removeing fs from tripleo and jobs related to it13:01
panda|roverchandankumar: nope, you'll have to look at the code that uses it. I remember we were installing tripleo-ui only in some undercloud-only jobs13:02
chandankumarthey have merged a specical fs also last i remembered13:02
arxcruzchandankumar: we are waiting for u13:03
chandankumararxcruz: joined13:03
chandankumararxcruz: weshay kopecmartin https://etherpad.openstack.org/p/tripleo-tempest-squad-meeting13:04
rlandyweshay: are you still using envD?13:12
weshayrlandy no.. thank you for it13:14
*** rfolco|brb is now known as rfolco13:18
ykarelchandankumar, marios_|ruck queens promoted13:20
chandankumar\o/13:21
marios_|ruckfantastic thanks ykarel++ chandankumar++13:21
chandankumarplease someone sponsers beers lots13:21
chandankumarof13:21
arxcruzchandankumar: when you visit Berlin :)13:25
*** Goneri has joined #oooq13:29
*** ykarel is now known as ykarel|meeting13:29
*** quiquell has quit IRC13:31
zbrpanda|rover: marios_|ruck : fixed the py34 issue with https://review.opendev.org/#/c/661286/13:33
zbr... well, fix = configured tox to avoid using py34.13:33
*** Goneri has quit IRC13:36
panda|roverzbr: this means python3 test will continue to run on centos7/rhel7 ?13:37
zbrpanda|rover: yes, tested with .tox folder clean : tox will use the preference order mentioned in envlist.13:37
rfolcorlandy, hope I am doing it right this time: https://code.engineering.redhat.com/gerrit/17137513:38
zbrpanda|rover: i know is a relatively unknown/hidden feature of tox but it is supported by official maintainers, it was suggested to me few months ago.13:39
zbrpanda|rover: sidenote: if no py3* is installed, it will try to use py27, not really a big deal.13:39
*** ratailor has quit IRC13:40
rlandyrfolco: k - no way of checking from looking at the code13:40
rlandyrfolco: but we may as well merge and try since the old one was def wrong13:41
rfolcorlandy, I wish there was a way to check if decrypt works13:41
chandankumararxcruz: if someone sponsers13:41
rlandyrfolco: w+ - yo can try it when it merges13:41
rfolcorlandy, thx13:42
panda|roverzbr: I;m missing some info, how does a change and globbing in envlist determin the preference for basepython for first element in such list13:42
panda|rover?13:42
zbrpanda|rover: this is what I was trying to tell you: it does!13:43
panda|roverzbr: well you're removing all basepython, so I'll assume the default is python213:44
zbrpanda|rover: i bet tox is looking for the special list of interpreters in envlist and builds a preference list before starting to process the list.13:44
*** Goneri has joined #oooq13:45
panda|roverzbr: pics or it will not happen :)13:45
zbrpanda|rover: nope, basepython default is not python2, is default python of the current environment, which depends on the preference list.13:45
panda|roverzbr: I need proof13:48
panda|roverzbr: bring me evidence13:48
*** amoralej is now known as amoralej|lunch13:49
panda|roverzbr: put it on a transparent bag, and mark it "exhibit n.1"13:49
zbrthe code behind is even more complex as tox supports plugins to find alternative python versions (like pyenv).13:50
*** quiquell has joined #oooq13:58
chandankumarmarios_|ruck: time for honey badger call14:00
marios_|ruckchandankumar: yup there14:01
marios_|ruckchandankumar: thanks14:01
*** dsneddon has quit IRC14:03
arxcruzchandankumar: come to Berlin, and i'll pay you some beers :)14:09
rfolcorlandy, decryption apparently works, thank you!14:09
rfolcorlandy, let's see if rdo registry (push) will work...14:10
rlandyrfolco: \o/14:10
weshayzbr ur not on the call :P14:11
zbrweshay: yes, bit late.14:15
marios_|ruckpanda|rover: talk now?14:17
panda|rovermarios_|ruck: knock knock14:18
marios_|ruckpanda|rover: in 12 mins is also fine14:18
marios_|ruckpanda|rover: k joining14:18
marios_|ruckpanda|rover: whats your blue dont see on invite (number please?)14:18
panda|rovermarios_|ruck: we usually use ruck's room14:18
panda|rovermarios_|ruck: I'm in your now14:18
marios_|ruckpanda|rover: k joining14:18
weshayzbr k.. join my blue in a mintue.. /me  brb14:19
zbrweshay: re wrong python interpreter issue: fyi, ansible changed the way it detects python interpreter on remove machines, making it detect default python(3) on new systems, but they did this only on ansible2.8 and they will *not* backport this.14:19
zbrthis means that we may have either upgrade to ansible 2.8 or have to use workarounds of mentioning which interpreter to use on remove machines.14:19
*** jaosorior has quit IRC14:23
weshayzbr https://docs.google.com/document/d/1xm3LooXddibhyZncjsZSBw9ZxBqiVfRjXcS-II5tQHU/edit#heading=h.vqmiu6tkx7or14:23
weshayzbr https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/phase2-15_director-rhel-8.0-virthost-1cont_1comp-ipv4-vxlan-lvm/32/console14:24
*** quiquell has quit IRC14:24
rlandyweshay: marios_|ruck: rhos-15 seems different ... https://sf.hosted.upshift.rdu2.redhat.com/logs/19/170519/10/check/tripleo-ci-rhel-8-standalone-rhos-15/a17a591/logs/undercloud/home/zuul/containers-prepare-parameters.yaml.txt.gz14:25
rlandysee http://download.eng.bos.redhat.com/rcm-guest/puddles/OpenStack/15.0-RHEL-8/latest-RHOS_TRUNK-15-RHEL-8/container_image_prepare_parameter_defaults.yaml14:25
*** aakarsh has quit IRC14:26
*** quiquell has joined #oooq14:26
*** amoralej|lunch is now known as amoralej14:28
panda|rovermarios_|ruck: congratulation on resisting 27 minutes :)14:45
marios_|ruckpanda|rover: thanks, getting stronger !14:45
chandankumartime to log out, see ya on monday!14:48
chandankumarHappy weekend all :-)14:48
marios_|ruckhave a good w/e chandankumar14:48
*** ykarel|meeting is now known as ykarel14:48
*** chandankumar is now known as raukadah14:48
ykarelpanda|rover, hi14:48
ykarelpanda|rover, so remember i pushed a patch to collect: ip link output?14:49
ykarelpanda|rover, i got a reproducer, and mac address was not in ip link output, so mac address seems not in nodes itself14:50
ykarelpanda|rover, but while looking into it, i also thought about couple of things14:50
panda|roverykarel: another thing we can exclude14:50
ykarelpanda|rover, yes14:50
panda|roverykarel: you got a reproducer ?14:50
panda|roverykarel: did you signal the mac to kfode14:50
ykarelpanda|rover, by reproducer i meant on CI14:50
panda|roverykarel: how many times did you have to retry ?14:51
ykarelpanda|rover, 6 7 tries14:51
ykarellet me get exact number of jobs14:51
panda|roverykarel: we have to change strategy with the irc alert, we wont' be able to inject anything before we run overcloud deploy14:51
panda|roverykarel: so I'm trying to push for something that runs in tht before os-net-config, like NodeUserData14:52
panda|roverykarel: also, what was the mac address ?14:52
*** quiquell has quit IRC14:52
ykarelpanda|rover, so it got me 30+ jobs to get reproducer14:53
panda|roverykarel: wow ...14:53
panda|roverykarel: I have a patch taht in desperate cases just runs 10 ovb jobs at once14:53
ykarelpanda|rover, mac address: FA:FC:51:62:64:0E14:53
panda|roverso I could get with 3 retries14:53
ykarel(FA:FC:51:62:64:0E) already uses address 172.17.0.7414:53
ykarelpanda|rover, hmm it's random, best case u get in 1st try, worst case can be any thing :)14:54
panda|roveras usual, random, doesn't match any vendor14:54
ykarelpanda|rover, so getting back to things that i noticed which can be collected to narrow down further14:55
panda|rovermarios_|ruck: ^ maybe the manual path is not that bad, it just remains to understand how to correctly inject the irc notification14:55
ykarelpanda|rover, 1st so when the deploy fails on bmc node, good to collect port list for private network14:56
ykarelpanda|rover, and 2nd why we use private network which is shared across all stack, i think that made sense when te-broker was used14:57
ykarelor there is some other reason to use private network?14:57
panda|roverykarel: probably just legacy14:57
panda|roverykarel: why would we collect the ports ?14:58
panda|roverykarel: all the macs from neutron start with the basemac14:58
ykarelpanda|rover, /me talking about the first case14:58
ykarelwhere issue happens on bmc node14:58
ykarelwhich we have found in past can be related to stack cleanup timings14:59
panda|roverykarel: so you think bmc would remort a mac in the range of the legit neutron ports15:00
weshayzbr https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/phase2-15_director-rhel-8.0-virthost-1cont_1comp-ipv4-vxlan-lvm/32/artifact/15:00
ykarelpanda|rover, yes possible, because afair we hit the same issue at time of te broker and when ever identified it was related to ports consumed by stacks which were delete failed15:01
panda|roverok, so collect port to say "here there are still ports undeleted taht get in the way"15:02
panda|roverykarel: do you have time/woll to bj ? I have to ask you something15:02
ykarelpanda|rover, okk bj link?15:03
panda|roverykarel: 349250866915:03
ykarelin15:04
*** kopecmartin is now known as kopecmartin|off15:06
*** bogdando has quit IRC15:08
* marios_|ruck shutdown sequence 15:18
marios_|ruckweshay: panda|rover need something before i goo15:18
weshayhigh five 0/15:19
weshay?15:19
marios_|ruck\O15:19
panda|rovermarios_|ruck: nothing thanks15:19
*** marios has quit IRC15:23
*** marios_|ruck has quit IRC15:23
*** dsneddon has joined #oooq15:30
ykarelpanda|rover, fyi logs for job where it reproduced and ip link collected:- https://logs.rdoproject.org/22/660922/2/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-stein-branch/ae2e1f4/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz15:31
ykarelhttps://logs.rdoproject.org/22/660922/2/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-stein-branch/ae2e1f4/logs/overcloud-controller-1/var/log/extra/network.txt.gz15:31
panda|roverykarel: thanks, card updated.15:35
*** holser_ has quit IRC15:40
*** jfrancoa has quit IRC15:50
ykarelpanda|rover, ack15:56
*** ykarel is now known as ykarel|away15:56
*** dtantsur is now known as dtantsur|afk16:01
*** saneax has quit IRC16:06
*** ykarel|away has quit IRC16:07
rlandyweshay: do you have access to the osp container registry - to view the tags. I think we are getting the update containers naming wrong16:09
weshaylast time I tried I did not, may want to ask in #release-delivery16:11
rlandyasking16:15
*** brault has joined #oooq16:16
*** brault has quit IRC16:21
*** ccamacho has quit IRC16:30
zbrrlandy or weshay: https://review.opendev.org/#/c/661286/ tox fix16:31
*** dsneddon has quit IRC16:33
*** tcw has quit IRC16:40
*** quiquell has joined #oooq16:48
*** dsneddon has joined #oooq17:00
*** jpena is now known as jpena|off17:04
*** quiquell has quit IRC17:07
*** quiquell has joined #oooq17:08
*** quiquell has quit IRC17:12
*** ykarel|away has joined #oooq17:14
rfolcorlandy, weshay panda|rover zbr arxcruz raukadah: please review https://code.engineering.redhat.com/gerrit/171297 - need more eyes on it to make sure the push to rdo registry won't break anything (yes, review before I run it as check job)17:20
raukadahrfolco: will take a look, thanks :-)17:53
rlandyweshay: ok - is  modify_append_tag: valid for osp?18:38
rfolcorlandy, trying to find what is the ssh key used to upload images, the one that was used as input for this encrypted secret: https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/zuul.d/tripleo-rdo-base.yaml#L10519:06
rfolcorlandy, if you know pls send me in pvt19:06
rlandyI know there is an uploader user19:06
rlandypanda|rover: ^^ may know19:06
rfolcosecret is uploader_ssh_key19:07
rfolcoso I assume it uses a ssh key, not password19:07
rlandyyeah _ I would guess so19:09
rlandyif it's not in the same file we looked at yesterday I'm not sure19:09
rlandyrfolco:also check the google doc19:10
rlandythere is a key listed there19:10
rfolcorlandy, that one is for cloud log server19:11
rfolcoaaaah19:11
rfolcorlandy, sorry, found it19:11
rfolcouploader_ssh_priv_key19:12
rfolco:)19:12
rfolcothanks rlandy19:12
*** tosky has quit IRC19:16
*** amoralej is now known as amoralej|off19:24
weshayrlandy sorry was at lunch... rlandy modify_append_tag is fine, it just changes the tag after the update19:26
rlandyweshay: yeah - but look at what we land up with ...19:27
rlandyrhosp15/openstack-cinder-volume:20190523.1-updated-2019052408562519:28
rlandy"error pulling image \"192.168.24.1:8787/rhosp15/openstack-cinder-volume:20190523.1-updated-20190524085625\"19:28
weshayhrm19:29
weshaythe updated tags in other osp runs we've seen didn't impact the download19:29
rlandyI have bugged jjoyce all morning ... I can't access this brew thing19:29
weshayer.. pull19:29
rlandyweshay: notice that the defaul params changed ...19:30
rlandyhttps://sf.hosted.upshift.rdu2.redhat.com/logs/19/170519/10/check/tripleo-ci-rhel-8-standalone-rhos-15/a17a591/logs/undercloud/home/zuul/containers-prepare-parameters.yaml.txt.gz19:30
rlandythe openshift stuff went bye-bye19:30
rlandymaybe it's unrelated19:31
weshayya.. I think that's unrelated19:31
rlandyhttps://code.engineering.redhat.com/gerrit/#/c/170519/ show rhos-14 and rhos-15 latest runs19:31
rlandywith updated params19:31
rlandyand standalone role19:31
rlandyweshay: ok - will try that one again19:33
rlandywill see if it changes the result19:33
rlandyI think the params are right now19:33
rlandyper https://review.opendev.org/#/c/659794/11/roles/standalone/tasks/main.yml19:34
weshayrlandy my suggestion is to get on a deployment that is not ci19:35
weshayand deploy w/ those params19:35
weshayhttps://sf.hosted.upshift.rdu2.redhat.com/logs/40/167540/43/check/tripleo-ci-rhel-8-standalone-rhos-15/4990506/logs/undercloud/home/zuul/containers-prepare-parameters.yaml.txt.gz19:35
*** ykarel|away has quit IRC19:35
weshayand19:36
weshayhttps://sf.hosted.upshift.rdu2.redhat.com/logs/19/170519/10/check/tripleo-ci-rhel-8-standalone-rhos-15/a17a591/logs/undercloud/home/zuul/containers-prepare-parameters.yaml.txt.gz19:36
weshayare very different19:36
*** panda|rover has quit IRC19:36
weshayhrm.. 2019-05-23 13:19:49 | TASK [tripleo-container-tag : Tag 192.168.24.1:8787/rhosp15/openstack-mariadb:pcmklatest to latest 192.168.24.1:8787/rhosp15/openstack-mariadb:20190520.1 image] ***19:37
weshayrlandy perhaps we need to turn off the setting     modify_append_tag: "-updated-20190524085625"19:37
weshayuntil we get the whole container update working19:37
rlandyweshay: it's juts missing the openshift params19:37
rlandypossibly19:37
rlandythat was my next step if the current run fails19:38
weshayhttps://sf.hosted.upshift.rdu2.redhat.com/logs/19/170519/10/check/tripleo-ci-rhel-8-standalone-rhos-15/a17a591/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz#_2019-05-24_13_12_37_41519:38
weshaythe container update just dies...  &19:38
weshay^19:38
weshayso I'm not suprised the updated tag is not on the containers in the undercloud registry19:38
*** panda has joined #oooq19:40
weshayrlandy we should turn off all the container update bits in the config for osp19:42
rlandyweshay: ack19:43
rlandyjust extend the j219:43
rlandyit worked for rhos-14 interestingly enough19:45
rlandyunable to parse the registries configuration (/etc/containers/registries.conf): invalid URL '192.168.24.1:8787': parse 192.168.24.1:8787: first path segment in URL cannot contain colon19:46
rlandy^^ same error you emailed about yesterday19:47
weshayrlandy ya.. I emailed Steve about that, you were on the email I think19:49
weshayya19:49
rlandyk - we will see19:49
rlandymoving on19:49
rfolcoweshay, rlandy pls merge this one (secrets for upload img) >> https://code.engineering.redhat.com/gerrit/#/c/171408/19:53
weshayrfolco merged20:05
rfolcoweshay, thx20:05
*** tcw has joined #oooq20:14
rlandywow - really did myself in now wrt switching to centos virthost, rhel undercloud vm20:48
rlandyweshay: build_test_packages: true/false? for centos virthost, rhel undercloud vm?21:06
rlandyoh man - this is not going to work21:11
rlandykeep baseos to use on the virt host and donwload images on the undercloud21:11
*** holser_ has joined #oooq21:33
rlandyweshay: you around?21:42
rlandyoh my gosh - finally ....21:47
rlandycheck-tripleo-centos-7-stein-containers-buildopenstack/tripleo-ci-internal-jobsmasterperiodic-hourlymaster362019-05-24T17:01:09SUCCESS21:47
rlandyperiodic-tripleo-ci-centos-7-bm_envB-3ctlr_1comp-featureset001-steinopenstack/tripleo-ci-internal-jobsmasterperiodic-hourlymaster165192019-05-24T17:03:17SUCCESS21:47
rlandycheck-tripleo-centos-7-master-containers-build-pidoneopenstack/tripleo-ci-internal-jobsmasterperiodic-hourlymaster402019-05-24T17:01:18SUCCESS21:47
rlandyperiodic-tripleo-ci-centos-7-ha_lab_andromeda-baremetal_3ctlr_2comp-featureset001-masteropenstack/tripleo-ci-internal-jobsmasterperiodic-hourlymaster4302019-05-24T17:03:31FAILURE21:47
rlandycheck-tripleo-centos-7-master-containers-buildopenstack/tripleo-ci-internal-jobsmasterperiodic-hourlymaster412019-05-24T17:00:18SUCCESS21:47
rlandyperiodic-tripleo-ci-centos-7-bm_envA-3ctlr_1comp-featureset001-masteropenstack/tripleo-ci-internal-jobsmasterperiodic-hourlymaster163932019-05-24T17:02:21SUCCESS21:47
rlandybaremetal is green on rocky, stein, master21:47
*** rlandy has quit IRC22:16
*** holser_ has quit IRC22:56

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!