Wednesday, 2019-05-22

*** hamzy has joined #oooq00:10
*** altlogbot_3 has quit IRC00:10
*** Goneri has joined #oooq00:10
*** altlogbot_0 has joined #oooq00:11
*** saneax has joined #oooq01:30
*** Goneri has quit IRC01:31
*** Goneri has joined #oooq01:33
*** apetrich has quit IRC01:57
*** rlandy has joined #oooq02:06
rlandyweshay: hi  - I'm send marios an email updated before tomorrow ...02:07
rlandywill tell him still work to be done install-deps02:07
*** rlandy has quit IRC02:15
*** ykarel|away has joined #oooq02:24
*** Goneri has quit IRC03:05
*** saneax has quit IRC03:06
*** ykarel|away is now known as ykarel03:58
*** udesale has joined #oooq04:11
*** ykarel has quit IRC04:53
*** ykarel has joined #oooq05:11
*** marios_|ruck has joined #oooq05:17
*** marios has joined #oooq05:17
*** ratailor has joined #oooq05:20
*** ajo has quit IRC05:27
*** ajo has joined #oooq05:33
*** quiquell has joined #oooq05:37
*** raukadah is now known as chandankumar05:44
*** jfrancoa has joined #oooq05:55
*** saneax has joined #oooq06:05
*** kopecmartin|off is now known as kopecmartin06:06
*** ksambor_ is now known as ksambor06:08
marios_|ruckykarel: gitweb is back :)06:21
ykarelmarios_|ruck, yes it was back some days back06:22
marios_|ruckykarel: really? could have sworn it wasn't there yesterday06:22
ykarelmarios_|ruck, i didn't checked yesterday, but few days back i saw gitweb was working06:24
marios_|ruckykarel: ack. it *was* there on things that merged but not on reviews06:25
ykareli checked on unmerged changes only iirc06:26
ykarelbut may be u are right as u checked yesterday06:26
ykarelanyway good it's working06:26
marios_|ruckykarel: yeah its a bit different but still :D06:27
marios_|rucksmall joys in life06:28
* marios_|ruck cries06:28
*** dtantsur|afk is now known as dtantsur06:28
chandankumarykarel: fs030 looks weired06:32
ykarelchandankumar, how so, /me didn't checked06:33
marios_|ruckykarel: so seems the ansible revert fixed periodic containers build i see some green runs today06:37
ykarelmarios_|ruck, yes06:37
marios_|ruckykarel++ amoralej|off++06:37
marios_|ruckthanks06:37
ykarelit got green yesteardy run06:37
marios_|ruckykarel: yah was not rucking in the afternoon/evening was on the 15 stuff06:37
ykarelmarios_|ruck, ack06:38
marios_|ruckykarel: https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-centos-7-master-containers-build-push06:38
*** skramaja has joined #oooq06:38
ykarelyes06:39
marios_|ruckpanda: around? do you have 5 mins for sync on https://bugs.launchpad.net/tripleo/+bug/1818060 i think its the main master promotion blocker left right now?07:04
openstackLaunchpad bug 1818060 in tripleo "Nodes periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master fail to get usable IPs though os-net-config with Error, some other host (BE:E5:4F:B9:21:B0) already uses address" [Critical,In progress] - Assigned to Gabriele Cerami (gcerami)07:04
marios_|ruckpanda: asking you cos you said you were going to check it and i see comments there from you07:05
marios_|ruckchandankumar: looks like you're still working on https://bugs.launchpad.net/tripleo/+bug/1828859 thanks for looking into it afaics the only green runs are from your review but periodic still broken at https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-queens07:06
openstackLaunchpad bug 1828859 in tripleo "[queens] Unhandled error: NoOptionError: No option 'tempest_roles' in section: 'auth'" [Critical,Fix released] - Assigned to chandan kumar (chkumar246)07:06
*** amoralej|off is now known as amoralej07:15
*** tesseract has joined #oooq07:18
ykarelmarios_|ruck, there is new issue in master07:19
ykarelboth upstream and promotion in case u are not aware07:19
*** tosky has joined #oooq07:21
marios_|ruckykarel: looking07:21
ykarelmore context can be found in #tripleo history of todays and yesterday:- http://eavesdrop.openstack.org/irclogs/%23tripleo/%23tripleo.2019-05-21.log.html#t2019-05-21T18:11:5207:21
marios_|ruckykarel: oh i thought you meant just happened failing jobs07:22
ykarelthose jobs are failing since 21st morning07:23
ykarelsince puppet-nova patch merged,07:23
ykareli pushed revert after finding the issue07:23
*** holser_ has joined #oooq07:27
*** holser_ has quit IRC07:27
*** holser_ has joined #oooq07:28
chandankumarmarios_|ruck: I am looking at tempest tests of rhos1507:28
marios_|ruckchandankumar: ack k07:28
chandankumarverifying which one is fixed or not fixed07:28
marios_|ruckchandankumar: sorry, whenever you next have time (for the queens thing)07:28
ykarelyes please good to fix queens thing, eggs looking for downstream import this week07:29
chandankumarnp, I am on it07:29
marios_|ruckykarel: so sounds like instead of revert we will do alex patch(es)?07:31
ykarelmarios_|ruck, more work is needed, alex patch itself failing07:32
ykarelso we disucssed today morning regarding the revert07:32
ykarelsee my comment in the revert, dhcp_domain is needed by nova metadata07:32
ykarelso unless and until someone counters the substitue of dhcp_domain, revert should be good07:33
marios_|ruckykarel: ack. anyway we don't have bug right i think we need one for tracking this discussion probably07:34
ykarelmarios_|ruck, yes good to have bug07:34
ykarelmarios_|ruck, in the meanwhile to handle ovn agents mapping i pushed a tht patch to use fqdn_canonical, but there i need feedback from ovn guys if it passes07:34
marios_|ruckykarel: so its tempest fail?07:36
*** saneax has quit IRC07:36
ykarelmarios_|ruck, neutron port binding fails07:36
ykareland hence tempest test fails07:36
marios_|rucktest_network_basic_ops[compute,id-f323b3ba-82f8-4db7-8ea6-6a895869ec49,network,smoke]07:36
marios_|ruckykarel: ack yeah i was just expecting deploy fail in logs ack thanks07:37
*** apetrich has joined #oooq07:39
*** jpena|off is now known as jpena07:44
*** ykarel is now known as ykarel|lunch07:52
chandankumarmarios_|ruck: all the 24 failed tests are known08:05
chandankumarmarios_|ruck: http://pastebin.test.redhat.com/76525408:06
marios_|ruckchandankumar: which bit? :D08:06
marios_|ruckchandankumar: is that re the neutron port binding bug https://bugs.launchpad.net/tripleo/+bug/182999308:07
openstackLaunchpad bug 1829993 in tripleo "tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035 & featureset001 fail tempest.scenario.test_network_basic_ops " [Critical,Triaged]08:07
chandankumarmarios_|ruck: it is related to rhos-15 tempest tests failure08:09
chandankumarmarios_|ruck: which one I look first, queens or above featureset001 faoilure?08:11
marios_|ruckchandankumar: do whatever you're doing08:13
marios_|ruckchandankumar: i was just looking at some other tempest issue thats all08:13
marios_|ruckchandankumar: which isnt' a tempest issue tempest just catches it but anyway08:13
marios_|ruckchandankumar: ok cool good on those being known08:13
chandankumarmarios_|ruck: does reproducer works for queens job?08:15
*** bogdando has joined #oooq08:15
marios_|ruckchandankumar: i don't see why not but haven't tried (branches)08:16
chandankumarmarios_|ruck: I need to reproduce fs03008:16
*** ccamacho has joined #oooq08:23
pandathe new last released pipeline runs every 12 hours ?08:36
*** ykarel|lunch is now known as ykarel08:38
ykarelit runs thrice a days afair, but need to check08:40
ykarelyes it runs thrice:- '10 9,15,21 * * *'08:41
pandaykarel: ok I see08:41
pandaso it's running, ok, it will probably fail ...08:42
ykareli updated ur testing patch08:42
ykarellet me check if it finished08:42
pandaykarel: it failed08:42
ykarelpanda, ok, so same will happen in periodic run08:43
pandaand also I should test stein08:43
ykarelyes08:43
pandait failed on neutron db upgrade08:45
pandaat least it's a new error08:45
pandajfrancoa: we can't wait for chem to return, who's the next guy that can help me fix this ?08:45
jfrancoapanda: hey, I'd say me...but that didn't work out very well :-D08:47
marios_|ruckpanda: updated the program call doc see if you want to add something08:47
marios_|ruckpanda: are you folks talking about the standalone-upgrade job container tag?08:47
pandamarios_|ruck: can i add "WE'RE ALL GONNA DIE!"08:47
pandamarios_|ruck: yes08:47
marios_|ruckpanda: k do you want to jump on a call jfrancoa are you available?08:47
marios_|ruckpanda: did you se my ping earlier about https://bugs.launchpad.net/tripleo/+bug/181806008:48
openstackLaunchpad bug 1818060 in tripleo "Nodes periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master fail to get usable IPs though os-net-config with Error, some other host (BE:E5:4F:B9:21:B0) already uses address" [Critical,In progress] - Assigned to Gabriele Cerami (gcerami)08:48
jfrancoapanda: in the worst case, it's what I said, feel free to modify the playbook or the role, now that we have the stein job voting we'll know if the change breaks it08:48
pandaif I jump on a call I'll probably hurt it really bad08:48
marios_|ruckpanda: i wanted to talk to you about that since you were looking at it08:48
marios_|ruckpanda: ok so lets not blujeans if it isn't necessary08:48
marios_|ruckpanda: did i miss some discussion already? /me scrolling up08:49
jfrancoamarios_|ruck: sure, I'm available08:49
ykarelmarios_|ruck, so my tht patch passed to clear ovb failures: https://review.opendev.org/#/c/660564/08:49
ykarelwould ping ovn guys08:49
pandamarios_|ruck: let's chat 5 min it's simpler08:49
marios_|ruckpanda: ok bluejeans who is hosting08:49
pandamarios_|ruck: jfrancoa usually the ruck hosts08:50
marios_|ruckhttps://redhat.bluejeans.com/7661925373/08:50
*** panda is now known as panda|rover08:50
marios_|ruckjfrancoa:  ^08:50
*** derekh has joined #oooq08:51
panda|rovermarios_|ruck: jfrancoa  https://review.rdoproject.org/r/2086708:57
*** saneax has joined #oooq09:03
panda|rovermarios_|ruck: can you paste the docker.io bug here09:15
marios_|ruckhttps://bugs.launchpad.net/tripleo/+bug/182997609:16
openstackLaunchpad bug 1829976 in tripleo " periodic-tripleo-standalone-upgrade-stein fails upgrade fetching new containers "Not found image: ... cinder-api:tripleo-ci-testing" " [Critical,Triaged]09:16
panda|rovermarios_|ruck: thanks09:16
marios_|ruckpanda|rover: maybe you want to reword that one or file new one for the tags?09:16
marios_|ruckpanda|rover: up to you09:16
marios_|ruckpanda|rover: we can track both docker.io and tags issue there09:16
marios_|ruckpanda|rover: maybe less confusing09:17
panda|rovermarios_|ruck: I'll close this, I'll open a new one once we have the latest periodic results in ~1h09:19
marios_|ruckpanda|rover: ack going afk few mins get some food then will check the retag stuff for a bit09:20
panda|roverjfrancoa: container-prep is already run in standalone upgrade09:33
panda|roverjfrancoa: I'll try to check what happens if we are using container_build_id09:33
jfrancoapanda|rover: right, it's in the dependencies part of the role, isn't it?09:33
panda|roverjfrancoa: yes09:33
panda|roverjfrancoa:  https://review.opendev.org/660615 testing here https://review.rdoproject.org/r/2086709:42
jfrancoapanda|rover: good, let's see if that works better than the dlrn_hash_tag09:44
jfrancoamarios_|ruck: hey, is it safe to use the zuul-based reproducer-quickstart? last time I tried to use it wasn't very stable yet..09:52
marios_|ruckjfrancoa: mostly works but havent tried it in few days... hopefully if you hti issues they will be known ones09:53
jfrancoamarios_|ruck: ack, I'll give it a try. thanks!09:55
*** holser_ is now known as holser|lunch09:57
chandankumarwhy we define names of stuff in caps and small in openstack, it causes too much issues10:33
*** skramaja has quit IRC10:34
panda|roverchandankumar: ?10:34
chandankumarpanda|rover: https://review.opendev.org/#/c/660649/10:34
chandankumarpanda|rover: I scratched my head come with lots of workaround10:34
chandankumarthen I found that difference between Member and member10:35
chandankumarykarel: https://review.opendev.org/#/c/660649/10:35
marios_|ruckpanda|rover: did you file a new bug for the standalone-upgrade ?10:37
marios_|ruckpanda|rover: i mean vs https://bugs.launchpad.net/tripleo/+bug/182997610:37
openstackLaunchpad bug 1829976 in tripleo " periodic-tripleo-standalone-upgrade-stein fails upgrade fetching new containers "Not found image: ... cinder-api:tripleo-ci-testing" " [Critical,Fix released] - Assigned to Gabriele Cerami (gcerami)10:37
panda|rovermarios_|ruck: the periodic job did not run yet10:37
marios_|ruckpanda|rover: well.. we can be pretty sure the 'tripleo-ci-testing bit will fail10:37
panda|rovermarios_|ruck: I'm running a test with the containers build id in parallel10:37
panda|rovermarios_|ruck: I can open a generic bug "standalone ugrade does not use the correct container tag" but it would be not based on a error, just on observation10:38
marios_|ruckpanda|rover: ack np10:40
marios_|ruckpanda|rover: i used that one for now can update10:40
marios_|ruckpanda|rover: posted the retag for now in case we want to explore it more https://review.opendev.org/66066510:40
*** blueDuckRandom has joined #oooq10:56
panda|rovermarios_|ruck: jfrancoa https://review.rdoproject.org/r/20867  succeeded ... periodic job failed.11:01
panda|rovermarios_|ruck: jfrancoa I mean the periodi job in the periodic pipeline instead of the one in check11:01
panda|roverwaiting for the logs before opening bug11:08
marios_|ruckpanda|rover: ack11:09
panda|rovermarios_|ruck: https://logs.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-standalone-upgrade-stein/6336274/logs/undercloud/home/zuul/standalone_upgrade.log.txt.gz#_2019-05-22_10_55_4111:09
*** ykarel is now known as ykarel|afk11:12
panda|rovermarios_|ruck: https://bugs.launchpad.net/tripleo/+bug/183002511:16
openstackLaunchpad bug 1830025 in tripleo "periodic standalone upgrade job on stein fails to download images" [Critical,Fix committed] - Assigned to Gabriele Cerami (gcerami)11:16
*** panda|rover is now known as panda|rover|eat11:16
panda|rover|eatbbl11:16
arxcruzwhen will sorin come back ?11:16
chandankumarmarios_|ruck: Here is the tempest queens issue https://review.opendev.org/#/c/660649/11:17
marios_|ruckchandankumar: thanks! checking11:17
chandankumarmarios_|ruck: sorry, https://storyboard.openstack.org/#!/story/200575511:17
marios_|ruckarxcruz: the prophesies have spoken of his return11:17
marios_|ruckprophecies?11:17
arxcruzprophecy ?11:18
chandankumarI might need to revert previous tqe changes also11:18
chandankumarmarios_|ruck: I need to spend some time on osa side centos jobs are broken there11:28
chandankumarfrom a week or more11:28
marios_|ruckchandankumar: ack thx11:28
*** udesale has quit IRC11:30
*** udesale has joined #oooq11:31
*** jpena is now known as jpena|lunch11:31
*** sshnaidm|afk is now known as sshnaidm11:35
weshaychandankumar are you sure there patches out for all the tempest failures?11:35
weshayor that they are all reported11:36
weshayit could be the case, I just want to double check11:36
chandankumarweshay: bugs are open for all the bugs, expect one11:36
weshayI'll have to deploy a different setup11:36
chandankumarweshay: one is fixed by kopecmartin already and merged11:36
chandankumarI will cross check the bugs11:37
*** rfolco is now known as rfolco|dentist11:39
rfolco|dentistbrb11:39
kopecmartinchandankumar, https://bugs.launchpad.net/tripleo/+bug/1828859/comments/1011:39
openstackLaunchpad bug 1828859 in tripleo "[queens] Unhandled error: NoOptionError: No option 'tempest_roles' in section: 'auth'" [Critical,Fix released] - Assigned to chandan kumar (chkumar246)11:39
*** vkapalav has joined #oooq11:40
*** holser|lunch is now known as holser_11:41
weshaykopecmartin++11:45
weshaymarios_|ruck https://review.opendev.org/#/c/658745/11:45
weshayI think that fixes fs01/35 double check for me11:45
marios_|ruckweshay: ack they look green ... you mean for https://bugs.launchpad.net/tripleo/+bug/182999311:47
openstackLaunchpad bug 1829993 in tripleo "tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035 & featureset001 fail tempest.scenario.test_network_basic_ops " [Critical,Triaged]11:47
marios_|ruckweshay: looking at logs11:47
marios_|ruckweshay: program call ... updated status check11:47
weshaymarios_|ruck aye.. thanks... it's just stein .. not master :)11:48
* weshay updated11:48
weshaymarios_|ruck I'll attend11:48
marios_|ruckweshay: yeah i was confused by the branch i thought somehow it was a nuanced issue...11:48
weshayunless you want to11:48
marios_|ruckweshay: well i won't insist :D but i may join to listen in anyway for a bit11:49
*** skramaja has joined #oooq11:49
marios_|ruckweshay: you sure its just stein?11:50
marios_|ruckweshay: i see the issue in http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/e91828a/logs/tempest.html.gz11:50
marios_|ruckweshay: see bug links in description11:50
weshaymarios_|ruck the program call is stein only atm11:51
weshaythey don't want to hear about train :)11:51
marios_|ruckweshay: ah ok11:51
marios_|ruckweshay: kk11:51
marios_|ruckweshay: i thought you were talking about https://bugs.launchpad.net/tripleo/+bug/182999311:52
openstackLaunchpad bug 1829993 in tripleo "tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035 & featureset001 fail tempest.scenario.test_network_basic_ops " [Critical,Triaged]11:52
marios_|ruckweshay: cos the fix you pointing at it stein https://review.opendev.org/#/c/658745/11:52
chandankumarweshay: marios_|ruck more detailed report http://pastebin.test.redhat.com/765344 osp1511:54
marios_|ruckweshay: so it fixes it for stein but not master?11:54
marios_|rucka11:54
marios_|ruckchandankumar: thanks11:54
weshaychandankumar marios_|ruck if you guys are done looking at that one deployment11:54
weshaycare to try one that has open issues?11:55
weshayor would you like help and for me to show you what I did?11:55
marios_|ruckweshay: i wasn't looking there chandankumar was doing that while i was rucking in morning11:55
marios_|ruckweshay: do you want me to stop looking at osp15 and switch11:55
marios_|ruckweshay: we have to land 2 things there and decide where the job will run so not too much left waiting for rlandy to discuss some more11:56
weshaymarios_|ruck chandankumar I'll ping you guys in 20 and we'll chat and see what makes sense11:56
marios_|ruckweshay: k11:56
chandankumarweshay: we need to poke people on this https://trello.com/c/im5WRPw4/926-cixlp1821690tripleo-ci-tempest-testautoallocatenetwork-tempestapicomputeimagestestimages11:56
chandankumarit is in rhos-15 and cix also11:56
*** rlandy has joined #oooq11:57
chandankumarkopecmartin: if we send a review on tempestconf, it is not going to come easily right now11:57
weshaychandankumar yes.. that would be a good one11:57
chandankumarwe need to do a new release11:57
weshaychandankumar marios_|ruck if you guys could put up a patch to make fs020 two compute11:58
weshaythat would also be excellent11:58
chandankumarweshay: /me takes a look11:58
chandankumarmarios_|ruck: we need to recheck the https://review.opendev.org/#/c/643731/ rhel-8 dib changes11:59
rlandyarxcruz: hi ... thanks for your email on tempest failure on bm11:59
chandankumarnmargi updated hte parch11:59
rlandyarxcruz: I only ran that to try simulate the fs001 failure on ovb11:59
chandankumarI mean the downstream change11:59
arxcruzrlandy: np11:59
* chandankumar does11:59
rlandywhich  think got fixed?12:00
arxcruzi though it was some failure you needed my help12:00
chandankumarsorry12:00
chandankumarrlandy: marios_|ruck where we can test this change https://review.opendev.org/#/c/643731/ downstream on RHEL-8?12:00
*** apetrich has quit IRC12:01
marios_|ruckchandankumar: here https://code.engineering.redhat.com/gerrit/16763312:01
marios_|ruckchandankumar: but we're currently not using it in our job12:01
marios_|ruckchandankumar: we are using the fake one12:01
rlandymarios_|ruck: chandankumar: what would you need as the base to run on to the test?12:02
marios_|ruckrlandy: on the job definition we'd have to switch the nodeset to use that12:02
marios_|ruckrlandy: i.e. https://code.engineering.redhat.com/gerrit/#/c/167540/38/zuul.d/nodesets.yaml12:02
marios_|ruckrlandy: that what you mean?12:03
rlandymarios_|ruck: sshnaidm merged a nodeset for ovb12:04
marios_|ruckrlandy: do you want to sync re the 15 job12:04
rlandymarios_|ruck: sure12:04
marios_|ruckrlandy: k sec, dropping from program call weshay is there12:04
rlandymarios_|ruck: and if we added a build-images job to our job, it could run on the same node12:04
marios_|ruckrlandy: https://redhat.bluejeans.com/7661925373/12:04
rlandyovb build images12:04
rlandychandankumar: ^^12:05
marios_|ruckrlandy: ok12:05
marios_|ruckrlandy: lets wait12:05
marios_|ruckrlandy: till after program call12:05
rlandysshnaidm: are your fs001 jobs building images?12:05
marios_|ruckrlandy: weshay chandankumar we all tal kthen12:05
sshnaidmrlandy, yes12:05
rlandymarios_|ruck: k - ping me when ready12:05
marios_|ruckrlandy: me and weshay and chandankumar said we'd sync after the program call12:05
marios_|ruckrlandy: k12:05
*** panda|rover|eat is now known as panda|rover12:06
weshayrlandy you were right about https://review.opendev.org/#/c/660508/ and the -y12:06
weshay:)12:06
marios_|ruckrlandy: abandon https://review.opendev.org/#/c/659794 ?12:06
marios_|ruckweshay: abandon https://review.opendev.org/#/c/659879 ?12:07
weshaymarios_|ruck rlandy easy review hopefully.. but we need another job https://code.engineering.redhat.com/gerrit/#/c/170965/12:07
weshaymarios_|ruck we need https://review.opendev.org/#/c/659794 for osp-1412:08
marios_|rucksd]12:08
marios_|ruckweshay: ah it wasn't in depends-on and i thought we didn't need it12:08
weshayhttps://review.opendev.org/#/c/659879 abandoned12:09
lmiccinio/ rlandy weshay good morning! can anybody give https://code.engineering.redhat.com/gerrit/#/c/171040/ the final kick please?12:11
rlandyweshay: something is needed for rhos-14 - so keeping https://review.opendev.org/#/c/659794 for now12:11
rlandybut don't merge12:11
rlandylmiccini: hi - I merged https://code.engineering.redhat.com/gerrit/#/c/171040/12:12
rlandylmiccini: also - just fyi, we moved our periodic baremetal jobs to install on a base os and build images12:12
rlandylmiccini: iirc, you keep your own role there12:13
lmiccinirlandy: thank you. guess I'll need to dig into that12:13
rlandylmiccini: if you'd like to consider ... https://review.opendev.org/#/c/648519/12:14
rlandylmiccini: it also required some settings changes12:14
rlandywhich I can walk you through12:14
*** apetrich has joined #oooq12:14
rlandybut we have passing jobs for the first time in a long time now12:14
lmiccinirlandy: I think I have that 'to-build-or-not-to-build' but I am bypassing it for now, I am building rpms/containers and injecting that into the overcloud image12:16
lmiccinirlandy: as soon as I am done fixing the jenkins downstream pipelines let's have a chat if you have time :) (probably next week)12:16
rlandylmiccini: the difference is that the review above switches that from a playbook to a role so the decision can be made on the virthost12:16
rlandylmiccini: sure - as you have time12:16
*** ykarel|afk is now known as ykarel12:18
*** apetrich has quit IRC12:19
weshaychandankumar marios_|ruck k.. join my blue please12:23
marios_|ruckweshay:joining12:24
chandankumarmarios_|ruck: https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/validate-tempest/vars/tempest_skip_stein.yml12:31
*** jpena|lunch is now known as jpena12:33
*** apetrich has joined #oooq12:35
rlandywoohoo - finally - passing periodics ... periodic-tripleo-ci-centos-7-bm_envE-3ctlr_1comp-featureset001-rockyopenstack/tripleo-ci-internal-jobsmasterperiodic-hourlymaster113632019-05-22T07:03:19SUCCESS12:43
rlandysshnaidm: what happened with ovb introspection? still stuck?12:44
sshnaidmrlandy, yeah, need to look today12:45
rlandysshnaidm: can you hold the nodes?12:45
rlandysshnaidm: if yes, hold them and we can check IPMI12:46
sshnaidmrlandy, yeah, checking..12:46
jfrancoapanda|rover: so the fix seems to work (https://review.opendev.org/#/c/660615/) , right?  the standalone-upgrade-stein job passes and the test patch had both jobs passing12:47
*** amoralej is now known as amoralej|lunch12:51
panda|roverjfrancoa: that's when you come in and say: "this is passing, but it's not upgrading" or "it passes for real"12:52
panda|roverjfrancoa: also stein passes, everywhere, master fails in upstream12:52
*** tcw has joined #oooq12:52
panda|roverjfrancoa: "Command '['systemctl', 'stop', u'tripleo_nova_compute.service']' returned non-zero exit status 1"12:53
jfrancoapanda|rover: no, no...no negative comments...for what I checked it's configuring the right repos in deployment and upgrade, passing the right namespace (from tripleorocky to tripleostein), looks good man! kudos12:56
jfrancoapanda|rover++12:56
panda|roverjfrancoa: mind voting there ? so the seal of approval from upgrades is official ?13:00
jfrancoapanda|rover: sure, I was about to13:00
ykarelpanda|rover, so not fixing delorean_current part in standalone upgrade promotion jobs?13:05
ykarelme disconnected about the plan, may be u posted somewhere13:05
lmiccinirlandy: I have another one for you :( https://review.opendev.org/#/c/660555/13:05
weshaypanda|rover let's chat13:06
panda|roverykarel: party pooper :) I knew I missed something ...13:07
ykarelpanda|rover, okk seems /me looking wrong or u missed something13:07
ykarellooks like u ran without promotion release files13:08
panda|roverykarel: again ?13:08
ykareli diddn't see in Depends-on13:08
ykarelatleast in last PS13:08
ykarelpanda|rover, /me seeing https://review.rdoproject.org/r/#/c/20867/2..4//COMMIT_MSG13:09
rlandylmiccini:  https://review.opendev.org/#/c/660555/ looks ok to me  ... marios_|ruck, panda|rover ... pls review13:10
ykarelso ^^ reminds me of ocata breakages https://review.opendev.org/#/c/602171/13:13
panda|roverykarel: what's missing ?13:14
ykarelpanda|rover, depends-on on https://review.opendev.org/#/c/638756/13:14
ykarelto test periodic jobs by promotion release files13:15
panda|roverykarel: dammit I thought it was there13:16
panda|roverI was sure ... :)13:16
panda|rover:(13:16
ykarelpanda|rover, it was there u removed it :)13:16
ykarelmay be while pushing update from local13:16
ykarelpanda|rover, people generally miss adding that depends-on and this results in merging wrong or incomplete patches13:18
ykareliirc sshnaidm pushed to make that change permanent, not sure where it was stuck13:18
ykarela recent example is queens fix13:19
weshaymarios_|ruck chandankumar ugh.. so results are back phase2-15_director-rhel-8.0-virthost-3cont_2comp_3ceph-ipv4-geneve-ceph3.8 hr > 1.2 days3 hr 38 min9 of 1,323 failed (-132)13:21
weshaypanda|rover https://code.engineering.redhat.com/gerrit/#/c/167540/13:29
*** rfolco|dentist is now known as rfolco13:29
weshaypanda|rover https://review.opendev.org/#/c/660508/13:29
*** Goneri has joined #oooq13:29
weshaypanda|rover https://code.engineering.redhat.com/gerrit/#/c/167540/13:29
*** dtantsur is now known as dtantsur|brb13:29
weshaypanda|rover https://code.engineering.redhat.com/gerrit/#/c/170965/113:30
marios_|ruckweshay: looking13:30
marios_|ruckweshay: https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/ReleaseDelivery/view/OSP15/job/phase2-15_director-rhel-8.0-virthost-3cont_2comp_3ceph-ipv4-geneve-ceph/  that one? last failed is still 1 day ago?13:31
marios_|ruckweshay: ah latest test results... so from one hundred something down to 9 fails now?13:32
panda|roverykarel: now I see what happened, I ran "git review -D 20867" to get your updated review13:32
panda|roverykarel: and didn't check if it succeded13:32
weshaymarios_|ruck ya.. we probably should switch targets, but we're only 30min away from the mtg, and we can ask there13:32
ykarelpanda|rover, ack, isn't it -d13:32
panda|roverykarel: yep, that's why it failed13:33
ykarelpanda|rover, ohkk13:33
*** ratailor has quit IRC13:41
*** aakarsh has quit IRC13:42
weshayrlandy can you rereview https://review.opendev.org/#/c/660508/13:44
weshaylast patch to merge13:44
weshayrlandy arxcruz btw.. I think the ovb issues on fs01/35 were resolved by kamil yesterday13:46
arxcruzweshay: what was the problem? is it related to the email i sent ?13:46
rlandyweshay: was the bm issue the same as ovb?13:46
rlandyie: did the bm job reproduce the ovb error?13:46
marios_|ruckweshay: chandankumar is there no 'full link to logs' i mean is that it https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/ReleaseDelivery/view/OSP15/job/phase2-15_director-rhel-8.0-virthost-3cont_2comp_3ceph-ipv4-geneve-ceph/lastCompletedBuild/console13:47
marios_|ruckweshay: chandankumar i.e. console output? we cant check logs of nodes from there?13:47
weshayrlandy arxcruz https://review.opendev.org/#/c/658745/13:48
chandankumarmarios_|ruck: yes13:48
chandankumarmarios_|ruck: go down13:48
chandankumarmarios_|ruck: you will get the log link http://cougar11.scl.lab.tlv.redhat.com/phase2-15_director-rhel-8.0-virthost-3cont_2comp_3ceph-ipv4-geneve-ceph/17/13:48
rlandyweshay: marios_|ruck: https://review.opendev.org/#/c/660508/ - now the -y is gone from everywhere?13:48
marios_|ruckchandankumar: ack see it thanks13:48
marios_|ruckrlandy: yeah i commented there i wasn't sure why that was necessary?13:48
marios_|ruckrlandy: weshay did the -y cause issues ?13:49
rlandywell initially it was in two places13:49
weshaymarios_|ruck no.. but it was dupe -y13:49
weshayso we were passing yum -y -y foo13:49
rlandyI thought we would keep it in one13:49
weshaywhich works fine13:49
marios_|ruckweshay: i see13:49
rlandyoh it remained in package_manager13:49
rlandynvm13:50
rlandygot it13:50
rlandyweshay: k ... +2'ed https://review.opendev.org/#/c/66050813:50
rlandylet's try it when marios_|ruck is ready for the fallout13:50
* marios_|ruck hair and eyebrows on fire 13:50
marios_|ruckrlandy: lets go for it!13:51
rlandymarios_|ruck: k - here we go w+13:51
weshayrlandy marios_|ruck while ur in context https://code.engineering.redhat.com/gerrit/#/c/170965/113:51
*** amoralej|lunch is now known as amoralej13:52
marios_|ruckrlandy: no wait13:52
marios_|ruckrlandy: is that centos vs "CentOS" OK13:52
marios_|ruckrlandy: let me comment on the review13:52
weshayyes13:52
weshaymarios_|ruck cat /etc/os-release13:52
* rlandy breathes again13:53
marios_|ruckweshay: ok sorry rlandy ... me reads more carefully (where were they getting NAME from )13:53
weshay[root@localhost ~]# cat /etc/os-release13:53
weshayNAME="CentOS Linux"13:53
weshayVERSION="7 (Core)"13:53
weshayID="centos"13:53
marios_|ruckah they were using Name you are using ID13:53
marios_|ruckweshay: thanks13:53
weshayya.. ID seems to be there for scripting13:53
rlandyweshay: so ... https://code.engineering.redhat.com/gerrit/#/c/170965/113:54
weshayaye13:54
rlandyfirstly - you have extra space13:54
weshayah ya.. saw that /me fixes13:54
rlandysecondly, remind me pls13:54
rlandywhat are we doing here13:54
rlandythe needs_images throws me off13:54
weshayrlandy need_images is the downstream equiv to tripleo-ci-testing13:54
rlandyweshay: ok - so then we need to do some work on the job playbooks to get the right release file ...13:55
weshayrlandy so we'll have two jobs... for now.. 1 that uses passes_phase_1 and one that is basically testing untested builds13:55
* rlandy gets13:55
weshayyes13:55
weshayand zuul13:55
rlandyhttp://git.app.eng.bos.redhat.com/git/openstack/tripleo-ci-internal-jobs.git/tree/playbooks/configure-rhel.yaml#n2013:56
rlandyweshay: ^^13:57
rlandyso we'll need another case there13:57
rlandyor we can rethink the naming13:57
panda|roverjfrancoa: dang. we need to talk again.13:58
weshayrlandy release file would be needs_images/rhos-15.yml13:58
weshayrlandy that should work the same way as upstream13:58
weshayrlandy we just need to pass a different release to the job13:58
jfrancoapanda|rover: sure, bluejeans or irc?13:58
rlandy}}/{{ 'promotion-testing-hash-' if 'periodic' in zuul.pipeline else '' }}{{ release }}.yml13:58
rlandyweshay: ^^13:59
* rlandy is thinking about this13:59
weshayrlandy look at what we do upstream13:59
weshaymatch that13:59
weshayit's the same construct13:59
rlandytoci expects a pattern13:59
arxcruzweshay: https://review.rdoproject.org/r/#/c/20662/ is ready for review btw, although it's failing in the linters, and i have no idea where's the error, locally it also fail on master, so it's not this patch that is making linters fail13:59
panda|roverjfrancoa: mmhh, wait a secong13:59
marios_|ruckweshay: chandankumar honey badger council14:00
panda|roverykarel: http://logs.rdoproject.org/67/20867/5/check/periodic-tripleo-ci-centos-7-standalone-upgrade-stein/86df487/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz fails to pull the containers, but they re there, I checked, maybe a transient problem ?14:00
ykarelpanda|rover, looking14:01
*** aakarsh has joined #oooq14:02
marios_|ruckweshay: this might be why the numbers are better now14:03
ykarelpanda|rover, hmm skopeo inspect is working, but strange both master and stein failed14:04
panda|roverykarel: they were probably pulling the images at the same time ....14:05
panda|roverykarel: trying a recheck14:06
chandankumarweshay: marios_|ruck from geneve http://pastebin.test.redhat.com/76542414:06
ykarelpanda|rover, ack14:07
weshaychandankumar marios_|ruck sounds like we're off the hook for today... marios_|ruck continue to focus on the standalone osp-15 w/ rlandy14:11
marios_|ruckweshay: ack14:11
*** quiquell has quit IRC14:12
marios_|ruckrlandy: still in this call but we can sync in a bit if necessary. really we should discuss about where/when to run that job? otherwise i think the last 2 things were mergable at least14:13
rlandymarios_|ruck: ack when you are ready14:13
*** quiquell has joined #oooq14:21
*** bogdando has quit IRC14:21
*** bogdando has joined #oooq14:21
weshayarxcruz we can disable ansible-lint or try to find the fix for https://github.com/ansible/ansible/issues/990314:25
marios_|ruckrlandy: ok just finished call... say in 4 mins?14:26
marios_|ruckrlandy: or we can wait longer if you are middle sthing else14:26
weshayarxcruz oof .. .that is old14:26
weshaynvrmind14:26
rlandymarios_|ruck: working with sshnaidm but it's fine let's chat before you go offline14:26
arxcruzweshay: where's Sorin?14:27
chandankumarmarios_|ruck: rlandy this review is ok https://review.rdoproject.org/r/#/c/20884/ ?14:27
arxcruzthis is his area, even if i run it on master, I get failure14:27
weshayarxcruz tomorrow14:28
rlandychandankumar: zuul fails there14:28
arxcruzok, i'll check with him tomorrow14:28
marios_|ruckrlandy: is fine around for another hour14:28
weshayarxcruz ya.. he returns from the dessert tomorrow14:28
marios_|ruckrlandy: let me know when you finish14:28
arxcruzthe error doesn't shows where exactly is the problem14:28
rlandyk14:28
arxcruzweshay: anyway, besides the tox issue, the code is ready, my next step will be add the full job14:29
*** dtantsur|brb is now known as dtantsur14:29
arxcruzbut i want to do it in another patch, this one is already too big, i don't want to add more complexity14:29
weshayarxcruz chat w/ me for a sec14:29
arxcruzweshay: sure14:30
*** ykarel is now known as ykarel|afk14:31
marios_|ruckchandankumar: checking14:36
* chandankumar headed home14:37
weshaysshnaidm rlandy https://review.rdoproject.org/r/#/c/20861/14:38
rlandymarios_|ruck: k - time to meet now?14:38
marios_|ruckchandankumar: looks like zuul issue will comment14:38
marios_|ruckrlandy: ack14:39
marios_|ruckrlandy: https://redhat.bluejeans.com/7661925373/14:39
rlandyweshay: tox errors on that14:39
rlandymarios_|ruck: k- joining14:39
arxcruzweshay: rlandy found the issue upstream https://github.com/ansible/ansible-lint/issues/50614:44
*** udesale has quit IRC14:44
*** saneax has quit IRC14:44
*** udesale has joined #oooq14:44
*** saneax has joined #oooq14:44
*** holser__ has joined #oooq14:44
*** tosky has quit IRC14:45
*** tosky has joined #oooq14:46
*** holser_ has quit IRC14:46
*** quiquell has quit IRC14:53
rlandyweshay: this afternoon, will need someone else to look at the container-prepare-parameters. - if you have time. there is some spacing voodoo there14:59
*** Vorrtex has joined #oooq15:01
weshayarxcruz going to schedule a mtg w/ you me and sshnaidm to discuss the layout of the code we have for the reproducer and zuul-runner15:02
weshayrlandy sure.. happy to continue the streak15:02
*** skramaja has quit IRC15:03
*** ykarel|afk is now known as ykarel15:06
*** jtomasek has joined #oooq15:11
arxcruzweshay: okay, today? I'm heading to german class, i'll be back in 4 hours15:11
*** holser__ has quit IRC15:12
weshayarxcruz tomorrow15:12
*** holser_ has joined #oooq15:12
* marios_|ruck shutdown sequence 15:16
marios_|ruckweshay: so tomorrow ruck and phase 215:16
marios_|ruckweshay: when they have new runs by then15:17
marios_|ruckwill sync with chandankumar15:17
weshaymarios_|ruck hold on phase 2 until the osp-15mtg15:17
marios_|ruckweshay: ah ok15:17
weshaymarios_|ruck little bit of rucking, but mostly push on osp-15 standalone please15:17
marios_|ruckweshay: ack synced with rlandy waiting on the install-deps to merge then we merge the job15:17
weshaymarios_|ruck I can give you more context if you want.. basically what I was talking about on the osp-15 call today15:17
marios_|ruckweshay: rlandy noticed we don't have the stein equivalent like we do for OSP14/rocky15:18
weshaymarios_|ruck right.. so we need another job defined15:18
marios_|ruckweshay: maybe thats next15:18
marios_|ruckweshay: ah you mean standalone osp on their h/ware15:18
marios_|ruckjob15:18
weshayno15:18
weshayblue15:18
marios_|ruckgoing15:19
*** saneax has quit IRC15:21
panda|roverjfrancoa: jobs passed again an this time with the right repo https://review.rdoproject.org/r/2086715:28
panda|roverjfrancoa: can you take a look again ?15:28
jfrancoapanda|rover: sure, checking...15:28
sshnaidmrlandy, found what's problem with ovb, it's incorrect nodepool provider configuration15:28
rlandysshnaidm: sorry - was in meeting15:30
rlandyhappy you found it15:30
sshnaidmrlandy, can you join please #sf-ops15:30
sshnaidmrlandy, need to check that I don't break baremetal with a fix15:30
weshayrlandy marios_|ruck stein standalone on rhel8 is not a viable target because we do not have containers for it15:31
weshayalthough we now have a viable rhel-8 image15:31
weshaystill need containers15:32
rlandyweshay: we don't run rocky on rhel anything15:32
rlandywe run on centos15:32
rlandyeven downstream15:32
rlandystein should be the same, no?15:32
jfrancoapanda|rover: both jobs look fine, the stein and master one15:33
rlandyam I missing the point here?15:33
panda|roverjfrancoa: SHIP IT!15:33
jfrancoapanda|rover: go go go!15:34
*** aakarsh has quit IRC15:35
panda|roverweshay: https://review.opendev.org/660615 has a chance to unblock promotion15:35
rlandysshnaidm: just got disconnected internal ... https://github.com/softwarefactory-project/sf-config15:35
*** dalvarez has quit IRC15:35
*** amoralej has quit IRC15:35
*** radez has quit IRC15:35
sshnaidmrlandy, https://softwarefactory-project.io/cgit/config/tree/nodepool/rdo-cloud.yaml#n36215:36
sshnaidmrlandy, yeah, today it's keeping disconnecting for me too15:36
sshnaidmrlandy, that's how we define networks in rdo cloud - private with floating IPs15:36
*** jfrancoa has quit IRC15:36
sshnaidmrlandy, so let's try it15:37
rlandysshnaidm: yeah - we have no other way of knowing15:37
rlandyit's not an obvious config15:37
rlandysshnaidm: all we need for bm is for zuul to be able to connect to the underclou15:37
rlandysshnaidm: ping me when you merge, I'll kick a test bm job15:38
sshnaidmrlandy, can you merge it? vpn completely failed for me..15:38
* rlandy tries15:38
rlandysshnaidm: down for me as well15:39
rlandywon't resolve15:39
sshnaidmyeah, same thing15:39
rlandyok -let's give it a few minutes15:39
*** radez has joined #oooq15:40
*** vkapalav has quit IRC15:42
*** lmiccini has quit IRC15:43
*** lmiccini has joined #oooq15:44
*** holser_ has quit IRC15:47
*** holser_ has joined #oooq15:48
*** marios has quit IRC15:50
*** marios_|ruck has quit IRC15:50
*** holser__ has joined #oooq15:51
sshnaidmweshay, vpn fails for everyone15:51
*** fmount has quit IRC15:51
*** holser_ has quit IRC15:53
*** derekh has quit IRC15:59
*** ykarel is now known as ykarel|away16:01
*** weshay_ has joined #oooq16:02
weshay_rlandy,  hey16:02
weshay_the containers for stein would have to be rhel 8 based16:02
*** jpena is now known as jpena|off16:02
rlandyweshay: hi16:04
weshay_howdy16:04
rlandyweshay: why though?16:04
*** bogdando has quit IRC16:04
rlandynot rdo on rhel16:04
weshay_rlandy,  stein ships on rhel816:04
weshay_rlandy,  yes16:05
rlandyjust regular stein16:05
weshay_it should be on -816:05
weshay_doesn't really make for a good comparison16:05
weshay_between the osp job16:05
*** jtomasek has quit IRC16:05
weshay_centos-7 stein compared to rhel-8 osp-1516:05
rlandyweshay_: how about on fedora28?16:05
weshay_better to hold off16:05
weshay_rlandy,  meh..16:05
weshay_we have other work to do on osp-1516:06
rlandyweshay_: ack - ok16:06
weshay_rlandy, most importantly getting it running on need_images16:06
rlandywe were just talking about where those jobs run16:06
weshay_ya.. getting one non-voting check job would be a good step forward as well16:06
rlandyweshay_: I'm editing the projects file on the standalone job16:07
weshay_k16:07
rlandyweshay_: once https://review.opendev.org/#/c/660508/ merges we want to merge https://code.engineering.redhat.com/gerrit/#/c/167540/16:08
rlandyso the projects file has to be accurate16:08
weshay_rlandy,  we should figure out how to get that job going on need_images before the next big deployment failure in osp phase 116:08
rlandyweshay_: network is out atm so I can't access it16:08
weshay_heh16:08
weshay_weee16:08
rlandyweshay_: ack - as soon as I have access again16:09
rlandyjust spinning16:09
*** tesseract has quit IRC16:10
*** jbadiapa has quit IRC16:10
weshay_rfolco, any blockers on the images job?16:12
rlandyweshay_: in the mean time, to calify, for needs_images,16:14
rlandyclarify16:14
rlandywe want a new release file16:14
rlandya new job16:14
rlandytriggered how?16:14
weshay_rlandy, periodic trigger for now is fine16:14
*** tesseract has joined #oooq16:14
*** tesseract has quit IRC16:14
weshay_we'd want that to trigger often though.. at least through this osp-15 debacle16:15
*** ykarel|away has quit IRC16:15
rlandyweshay_: k - putting something together16:16
weshay_thanks16:16
*** tcw has quit IRC16:17
*** tcw has joined #oooq16:17
rfolcoweshay_, more or less. I am testing the playbooks in the rhel box. The build command is failing...16:17
weshay_rfolco,  can you pastebin the error on a public pastebin?16:18
*** Vorrtex has quit IRC16:18
sshnaidmrlandy, vpn seems to work again16:20
*** tcw has quit IRC16:20
*** tcw has joined #oooq16:20
rlandytrying16:20
rfolcoweshay_, I am re-executing now. rhos-release is creating delorean.repo (old way)... the rpm does not have the latest change looks like.16:21
rlandysshnaidm: still can't access the review  - can you?16:21
weshay_rfolco,  which rhose-release are you pulling?16:22
weshay_I can't get on the vpn either16:22
rlandyinternal irc just came back16:22
*** jpena|off is now known as jpena16:22
rlandygone again16:22
weshay_oooh I'm on16:23
*** amoralej has joined #oooq16:23
weshay_phx16:23
weshayping weshay_ get the f out of here16:23
sshnaidmrlandy, yeah, merged16:24
rlandyI'm on rdu216:24
*** weshay_ has quit IRC16:24
rfolcoweshay_, should be http://rhos-release.virt.bos.redhat.com/repos/rhos-release/rhos-release-latest.noarch.rpm, but configure-rhel playbook has another one... I am also fixing this part16:24
*** jbadiapa has joined #oooq16:24
rlandysshnaidm: ^^ still disconnected - https://code.engineering.redhat.com/gerrit/#/c/165815/ is a master bm test job16:25
*** kopecmartin is now known as kopecmartin|off16:25
rlandyok - finally back16:26
rlandylet's try this again16:26
weshayrfolco variables here look incomplete to me https://code.engineering.redhat.com/gerrit/#/c/170815/20/playbooks/build-image/templates/build-image.sh.j216:27
rlandyoh man - out again16:28
rfolcoweshay, yah I am tetsing it local now w/ DIB_YUM_REPO_CONF, just want to confirm what is really required there16:29
chandankumarweshay: do we use the box to try infrared manually tomorrow?16:31
weshaychandankumar not in your day... waiting on some infrared patches to land... will ask again in tomorrow's mtg what we should go after16:33
weshaychandankumar we could try for fun and profit16:33
weshayor just fun16:33
chandankumarweshay: let's drive the fun tomorrow16:33
chandankumarwith marios16:33
rlandysshnaidm: periodic-tripleo-ci-centos-7-bm_envD-3ctlr_1comp-featureset001-master kicked - will see what shakes out16:33
weshaysounds good to me16:33
chandankumaranyway tomorrow is judgement day in India16:34
weshaysshnaidm rfolco you guys are both building images in your respective jobs16:34
chandankumargovernment election results will come16:34
weshaysshnaidm rfolco by design16:34
weshaycan you two please share your work w/ each other16:34
chandankumarrlandy: I didnot get time to look at rdo logs stuff will work on that tomorrow16:35
*** Vorrtex has joined #oooq16:35
rfolcosshnaidm, https://code.engineering.redhat.com/gerrit/#/q/topic:rhel-rdo-rocky+(status:open+OR+status:merged)16:35
sshnaidmrlandy, now there is a problem, trying to solve on sf-ops16:35
rfolcosshnaidm, weshay this is my build-image job on rhel...16:36
*** udesale has quit IRC16:38
rlandychandankumar: no worries - I didn't get much time there either. I can also pick up tomorrow16:39
*** vinaykns has joined #oooq16:39
chandankumarsecret stuff is new, I will learn that16:39
rlandysshnaidm: on the upside, bm looks ok so far16:39
rlandychandankumar: I can show you how to to get the secret16:40
rlandyor sshnaidm also knows - if you want to start before I am on line16:40
chandankumarrlandy: yes, will poke sshnaidm tomorrow!16:41
rfolcoweshay, http://rhos-release.virt.bos.redhat.com/repos/rhos-release/rhos-release-latest.noarch.rpm is not the latest built rpm, where do I get it ?16:42
*** udesale has joined #oooq16:43
weshayrfolco #rhos-delivery16:46
rfolcoweshay, k thx16:46
chandankumarpune office people are celebrating for RHEL-816:46
chandankumarfor me nothing is there16:47
chandankumargoing together for dinner party16:47
chandankumarsshnaidm: how nodes are getting populated if this review https://review.opendev.org/#/c/660710/ gets merged?16:50
chandankumarthere are some node files where there is no default_name: overcloud-novacompute-116:50
chandankumar    hostnamemap_override: overcloud-novacompute-bar-116:51
chandankumarand what does topology does?16:51
sshnaidmchandankumar, should be fine, like here: https://github.com/openstack/tripleo-quickstart/blob/master/config/nodes/1ctlr_2comp_1supp.yml#L116:53
chandankumarso basically we have pre defined vms with that flavor named compute or controller16:54
chandankumarwhat does scale means?16:54
chandankumarrlandy: we have https://review.opendev.org/#/c/658262/ rhel8 support in progress for os_Temoest16:59
chandankumarI think we can test in rhos-15 job16:59
rlandychandankumar: ok - just include that depends-on? or is there more to it than that?17:00
rlandywe are using a mocked up rhel node17:01
rlandyif that would could for your testing17:01
chandankumarrlandy: yes, I will put a new patch from current one17:01
rlandychandankumar: https://code.engineering.redhat.com/gerrit/#/c/167540/ shoudl be ready to merge as  soon as https://review.opendev.org/#/c/660508/ merges17:02
*** dsneddon has joined #oooq17:06
*** udesale has quit IRC17:12
*** chandankumar is now known as raukadah17:14
*** jpena is now known as jpena|off17:17
*** jtomasek has joined #oooq17:27
rlandyweshay: I am building the rhos-15 needs_images job now17:43
rlandyshould that be named periodic-*17:43
rlandyreport to dlrn?17:43
rlandyor none of the above just use the correct release file17:44
weshayrlandy it does not have to report to dlrn at this time, not sure if it ever will17:45
weshayrlandy it certainly can be called periodic17:45
rlandyweshay: ok - coming up17:45
weshayrlandy rock on...17:45
weshayI can give you more context of how I see that job would play a role for us.. if you want it17:46
weshayrlandy can you retrigger fs001 on bm for me please17:54
weshayI thought we were out of the woods, but we're not17:54
rlandyweshay: sure - there is one running right now17:54
rlandyoh wait - it's periodic17:55
rlandywill retrigger17:55
weshayrlandy /me is looking for validation of whether or not tempest.scenario.test_network_basic_ops.TestNetworkBasicOps failures are real17:55
rlandyweshay: just one moment for me to submit rhos-15 need-images stuff17:55
rlandyweshay: pls review https://code.engineering.redhat.com/gerrit/171119 - not sure if this is what you intend17:58
rlandyalso note that the job will only get tripleo-ci-testing-hash if run in periodic pipeline17:59
rlandyI will edit the  playbook to test that next17:59
rlandyas the spacing will be off17:59
rlandyhttp://git.app.eng.bos.redhat.com/git/openstack/tripleo-ci-internal-jobs.git/tree/playbooks/configure-rhel.yaml#n3517:59
rlandy}}/{{ 'promotion-testing-hash-' if 'periodic' in zuul.pipeline else '' }}{{ release }}.yml18:00
*** amoralej is now known as amoralej|off18:00
weshayrlandy tripleo-ci-testing hash?18:00
rlandy^^ as currently set, the would come out to18:00
rlandypromotion-testing-hash-need-images/rhos-15.yaml18:00
weshaythat review looks good18:00
weshay      release_file: need_images/rhos-15.yml18:00
rlandyweshay: ^^ see what I am describing above?18:00
rlandypromotion-testing-hash-need-images/rhos-15.yaml18:00
rlandyin a periodic pipeline18:01
rlandythat is wrong I think18:01
rlandyanyways, let me kick bm again18:01
weshayrlandy that looks good to me :)18:01
weshay+218:01
weshayrlandy oh wait18:02
weshayneed_images/rhos-15 or need_images/rhos-15.yml18:02
weshayI think w/o the yaml18:02
* weshay checks18:02
rlandyarxcruz: are you still connected to rdoci-hp-01.v100.rdoci.lab.eng.rdu2.redhat.com?18:03
rlandyenvD18:04
rlandyweshay: bm kicked again .... but I see this virt-customize error in the previous job ...  fatal: unable to access 'https://opendev.org/openstack/tripleo-ci/': Encountered end of file18:06
rlandyhttps://sf.hosted.upshift.rdu2.redhat.com/logs/15/165815/53/check/periodic-tripleo-ci-centos-7-bm_envD-3ctlr_1comp-featureset001-master/dcd71e9/job-output.txt.gz#_2019-05-22_17_04_36_61865118:07
rlandyfixing rhos-1518:07
weshayrlandy ya.. 402fa89ab4460cc7fb62/work/src/code.engineering.redhat.com/tripleo-environments/config/release/need_images/rhos-15.yml.yml'18:08
weshayrlandy I have no idea if that job will actually pass btw.. depends on the rpms there18:09
weshaywill show it to the rhos-delivery folks when we are done18:09
*** Goneri has quit IRC18:11
rlandyupdated18:12
rlandyweshay: running now ... https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/stream/93e3d558ae04462b940b90d6818d3832?logfile=console.log18:20
rlandyneeds_images thing18:20
rlandyand your bm job18:21
*** jtomasek has quit IRC18:23
*** aakarsh has joined #oooq18:23
*** jtomasek_ has joined #oooq18:24
weshayrlandy will be  interesting to see what happens w/ containers here... as they  are not built yet for needs_images18:27
weshayuses older ones18:27
raukadahweshay: rlandy any idea about this issue https://sf.hosted.upshift.rdu2.redhat.com/logs/17/171117/1/check/tripleo-ci-rhel-8-standalone-rhos-15/6329fa4/job-output.txt.gz#_2019-05-22_13_55_00_43018518:27
*** tosky has quit IRC18:30
*** tosky has joined #oooq18:30
raukadahis it possible to use this job as a third party job against os_tempest?18:31
raukadahone blocker is rdo logs which I will get it fixed?18:31
raukadah*.18:31
rfolcoweshay, do you have 5 min to chat about https://code.engineering.redhat.com/gerrit/#/c/167729?18:32
weshayrfolco ya18:32
rfolcoweshay, your bj?18:32
rlandyraukadah: looking18:40
rlandyFailed to synchronize cache for repo 'rhos-release-extras', ignoring this repo18:42
raukadahrlandy: will i do a recheck?18:42
rlandyfeel like we hot this one before18:42
rlandyraukadah: you can type 'recheck'18:43
rlandyhttps://sf.hosted.upshift.rdu2.redhat.com/logs/17/171117/1/check/tripleo-ci-rhel-8-standalone-rhos-15/6329fa4/logs/undercloud/etc/yum.repos.d/rhos-release.repo.txt.gz18:44
rlandyraukadah: ^^ so that is the repo18:44
raukadahdone recheck, let's see how it goes!18:44
rlandyraukadah: we may need to inquire of #rhos-delivery people18:45
rlandyto check what is available18:45
*** sshnaidm is now known as sshnaidm|afk18:45
rlandyweshay: well the needs-images run is deploying standalone18:46
rlandyso it's using some containers18:46
weshayrfolco https://images.rdoproject.org/master/18:47
weshayhttps://images.rdoproject.org/rocky/rdo_trunk/18:47
weshayhttps://images.rdoproject.org/rocky/rdo_trunk/centos718:48
rfolcohttps://images.rdoproject.org/master/rdo_trunk/tripleo-ci-testing/18:48
weshayhttps://images.rdoproject.org/rocky/rdo_trunk/rhel718:48
*** holser__ is now known as holser-afk18:55
weshayrlandy need_images worked19:19
rlandyweshay: not sure it really worked - it passed ... but are the repos etc. correct?19:21
weshayrlandy I'm getting on w/ jjoyce to confirm that19:23
weshaythey are older than passed_phase_119:23
weshayhttps://sf.hosted.upshift.rdu2.redhat.com/logs/19/171119/2/check/periodic-tripleo-ci-rhel-8-standalone-need-images-rhos-15/93e3d55/logs/undercloud/etc/yum.repos.d/rhos-release-15-trunk.repo.txt.gz19:23
*** jtomasek_ has quit IRC19:24
*** jtomasek has joined #oooq19:24
*** quiquell has joined #oooq19:26
*** quiquell has quit IRC19:26
rlandyweshay: are the containers correct?     -   modify_append_tag: -updated-2019052214201019:32
*** Goneri has joined #oooq19:37
*** jtomasek has quit IRC19:42
*** jtomasek_ has joined #oooq19:43
weshayrlandy ok.. let's chat for a sec19:43
rlandyweshay: k  - joining19:43
arxcruzrlandy: i wasn't connected there...19:46
rlandyarxcruz: np - I was just reprovisioning so wanted to tell anyone who might still need that info19:46
arxcruzweshay: I found a workaround solution for the tox linters, i'll submit a patch tomorrow19:57
arxcruzweshay: Ansible-lint.............................................................Passed19:57
weshayarxcruz++19:58
*** jtomasek_ has quit IRC20:02
*** jtomasek has joined #oooq20:02
weshayrlandy http://www.mydailytutorials.com/ansible-replace-example/20:07
*** Vorrtex has quit IRC20:24
rfolcoweshay, success!! https://sf.hosted.upshift.rdu2.redhat.com/logs/16/170816/25/check/tripleo-buildimage-overcloud-rocky-rhel-7/9340e2d/logs/build-image.log.txt.gz20:42
weshaynice!20:43
weshayrfolco cool.. I think that logs everything we need too :)20:44
rfolco:)20:46
rlandyweshay: https://code.engineering.redhat.com/gerrit/#/c/167540 and https://code.engineering.redhat.com/gerrit/#/c/171119/ are rebased and should be mergeable when the upstream change is in20:47
rlandyweshay: ping ... the rhos-14 jobs are non-voting21:08
rlandyrhos-15 job should be non-voting as well?21:08
weshayrlandy in periodic?21:16
weshaydoesn't matter21:16
rlandyweshay: the rhos-15 job is also in check21:16
weshayrlandy it can vote on ci changes21:16
rlandyif we expect it to pass, it's ok21:16
weshayjust not tripleo changes yet21:16
rlandycorrect21:17
rlandyjust on this repo21:17
weshayya21:17
weshayit can vote on any ci or zuul change21:17
*** aakarsh has quit IRC21:30
weshayrlandy you can see the containers get updated http://logs.openstack.org/94/659394/1/check/tripleo-ci-centos-7-standalone/cb090dc/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz21:32
weshayvia https://github.com/openstack/ansible-role-tripleo-modify-image21:32
weshayI do not see that in the internal job21:32
rlandyhmmm - maybe missing a setting21:33
weshayhttp://logs.openstack.org/94/659394/1/check/tripleo-ci-centos-7-standalone/cb090dc/logs/undercloud/home/zuul/standalone_parameters.yaml.txt.gz21:33
weshaydon't se it yet21:33
weshaymodify_role: tripleo-modify-image21:34
weshay        modify_vars:21:34
weshay            tasks_from: yum_update.yml21:34
weshay            update_repo: gating-repo,delorean-current,quickstart-centos-ceph-nautilus21:34
weshaythat's it21:34
weshay     modify_role: tripleo-modify-image21:35
weshay        modify_vars:21:35
weshay            tasks_from: yum_update.yml21:35
weshay            update_repo: gating-repo,delorean-current,quickstart-centos-ceph-nautilus21:35
weshay            yum_repos_dir_path: /etc/yum.repos.d21:35
rlandylol21:35
weshayso the osp job has rdo repos listed :)21:35
rlandythe parameters file21:35
weshayback to your file :)21:35
weshaylolz21:35
rlandysomeone shoot me21:35
rlandybut it's not modified21:35
rlandyin rhos-1521:35
rlandyI have no idea where those settings should be21:36
rlandywe add them to the file21:36
rlandyso the placement is anyone's guess21:36
rlandythis is my latest try ...21:36
rlandyhttps://review.opendev.org/#/c/659794/5/roles/standalone/tasks/main.yml21:37
rlandyweshay: I think we will be spending some more quality time with that file21:37
*** Goneri has quit IRC21:43
weshayrlandy just change the repos21:44
weshayto21:44
weshayrhelosp-15.0-trunk21:45
weshayrhelosp-ceph-4.0-mon-pending21:46
weshaywould be the equiv21:46
weshayrlandy probably can make release vars for that21:46
*** jtomasek has quit IRC21:47
rlandyweshay: where?21:47
weshayrlandy want to chat?21:47
weshaylolz21:48
rlandyok21:48
weshayit's just quicker21:48
*** Goneri has joined #oooq21:53
*** rlandy is now known as rlandy|bbl21:59
*** Goneri has quit IRC22:32
*** tosky has quit IRC23:11
*** tosky has joined #oooq23:11
*** aakarsh has joined #oooq23:20
*** tosky has quit IRC23:34

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!