Thursday, 2019-05-23

*** dsneddon has quit IRC01:36
*** apetrich has quit IRC01:57
*** dsneddon has joined #oooq02:00
*** ykarel|away has joined #oooq02:20
*** irclogbot_1 has quit IRC02:26
*** irclogbot_0 has joined #oooq02:28
*** vinaykns has quit IRC02:45
*** dsneddon has quit IRC03:02
*** rlandy|bbl is now known as rlandy03:14
*** rlandy has quit IRC03:20
*** ykarel|away has quit IRC04:17
*** udesale has joined #oooq04:28
*** ykarel|away has joined #oooq04:32
*** dsneddon has joined #oooq04:58
*** ratailor has joined #oooq05:16
*** ykarel|away is now known as ykarel05:20
*** raukadah is now known as chandankumar05:25
*** quiquell has joined #oooq05:27
*** saneax has joined #oooq05:28
*** ccamacho has quit IRC05:29
*** zbr has joined #oooq05:34
*** marios_|ruck has joined #oooq05:38
*** marios has joined #oooq05:38
*** skramaja has joined #oooq05:43
*** dsneddon has quit IRC05:50
*** kopecmartin|off is now known as kopecmartin06:04
*** jfrancoa has joined #oooq06:05
marios_|ruckykarel: thank you https://review.opendev.org/#/c/660564/06:15
*** dsneddon has joined #oooq06:16
ykarelmarios_|ruck, o/06:17
ykarelmarios_|ruck, with ^^ ovb jobs are clear, but now seeing some random failures06:18
ykarelmarios_|ruck, and good news, stein being promoted06:18
marios_|ruckykarel: thanks gonna tidy up the bugs/etherpad  and see whats new06:18
marios_|ruckykarel: nice06:18
marios_|ruckykarel: finally06:18
marios_|rucksome good news06:18
*** dsneddon has quit IRC06:40
*** tesseract has joined #oooq06:41
marios_|ruckpanda|rover: i updated the description to be for both issues https://bugs.launchpad.net/tripleo/+bug/182997606:56
openstackLaunchpad bug 1829976 in tripleo " periodic-tripleo-standalone-upgrade-stein fails upgrade fetching new containers "Not found image: ... cinder-api:tripleo-ci-testing" " [Critical,Fix released] - Assigned to Gabriele Cerami (gcerami)06:56
marios_|ruckpanda|rover: and added reviews there06:57
marios_|ruck(adding)06:57
marios_|ruckchandankumar: i know you're busy (rdo) but can you sync with kopecmartin today if you haven't already did you see comment 11 https://bugs.launchpad.net/tripleo/+bug/1828859 - will we revert that https://review.opendev.org/#/c/660036/07:11
openstackLaunchpad bug 1828859 in tripleo "[queens] Unhandled error: NoOptionError: No option 'tempest_roles' in section: 'auth'" [Critical,Fix released] - Assigned to chandan kumar (chkumar246)07:11
marios_|ruckchandankumar: when you have time thanks07:11
marios_|ruckkopecmartin: are you looking into that sounded like you were looking at a fix https://bugs.launchpad.net/tripleo/+bug/1828859 from comment 1007:13
openstackLaunchpad bug 1828859 in tripleo "[queens] Unhandled error: NoOptionError: No option 'tempest_roles' in section: 'auth'" [Critical,Fix released] - Assigned to chandan kumar (chkumar246)07:13
chandankumarmarios_|ruck: testing here https://review.rdoproject.org/r/#/c/20899/07:14
kopecmartinmarios_|ruck, revert of that patch might not be needed at the end , we'll see, and the fix for the issue should be here https://review.opendev.org/#/c/660748/207:14
kopecmartini also commented in the LP07:14
marios_|ruckchandankumar: thanks07:14
marios_|ruckkopecmartin: can you ..07:14
*** fmount has joined #oooq07:14
marios_|ruckhah thanks07:15
* marios_|ruck refresh 07:15
marios_|ruckkopecmartin: you mean on https://bugs.launchpad.net/tripleo/+bug/1828859 ?I don't see it07:15
openstackLaunchpad bug 1828859 in tripleo "[queens] Unhandled error: NoOptionError: No option 'tempest_roles' in section: 'auth'" [Critical,Fix released] - Assigned to chandan kumar (chkumar246)07:15
kopecmartinmarios_|ruck, oh, I didn't comment directly .. I put there link to storyboard07:16
marios_|ruckkopecmartin: ack thanks07:16
marios_|ruckkopecmartin: chandankumar added comment 12 just now with links07:17
*** dalvarez has joined #oooq07:17
*** apetrich has joined #oooq07:19
*** rf0lc0 has joined #oooq07:23
zbrhello~07:25
marios_|ruckHE07:25
marios_|ruckHAS07:25
marios_|ruckRETURNED07:25
marios_|ruckrejoice07:25
marios_|ruck\o/07:25
marios_|ruckpraise be to zuul07:25
*** rfolco has quit IRC07:26
zbri returned yesterday, 22.5h hours door-to-door :p --- woke at 4am today so I suppose I will need some extra coffee today07:26
zbrwhat did I miss?07:26
marios_|ruckzbr: the usual. everything is on fire and there are 2 guys on each side dousing it with flamethrowers07:27
chandankumarzbr: Welcome back :-)07:28
*** tosky has joined #oooq07:32
*** jfrancoa has quit IRC07:35
*** ccamacho has joined #oooq07:35
*** ykarel is now known as ykarel|afk07:37
*** dsneddon has joined #oooq07:39
*** dsneddon has quit IRC07:44
*** jpena|off is now known as jpena07:45
chandankumarkopecmartin: it is affecting master jobs07:46
chandankumarkopecmartin: http://logs.openstack.org/65/660565/2/check/tripleo-ci-centos-7-undercloud-oooq/47f69d6/logs/undercloud/home/zuul/tempest.log.txt.gz#_2019-05-22_10_23_0307:47
*** amoralej|off is now known as amoralej07:47
*** apetrich has quit IRC07:48
*** holser_ has joined #oooq07:49
*** jtomasek has joined #oooq07:55
*** tosky has quit IRC07:55
*** tosky has joined #oooq07:55
*** apetrich has joined #oooq07:55
*** jtomasek has quit IRC07:57
*** apetrich has quit IRC07:59
*** jtomasek has joined #oooq07:59
*** apetrich has joined #oooq08:00
*** ykarel|afk is now known as ykarel08:03
*** jtomasek has quit IRC08:05
*** saneax has quit IRC08:07
*** saneax has joined #oooq08:08
*** jfrancoa has joined #oooq08:11
*** ykarel is now known as ykarel|lunch08:16
zbr... so much email to read.08:18
*** sanjayu_ has joined #oooq08:27
*** bogdando has joined #oooq08:29
*** saneax has quit IRC08:29
*** panda|rover has quit IRC08:40
*** dsneddon has joined #oooq08:40
*** panda has joined #oooq08:42
*** dsneddon has quit IRC08:46
*** derekh has joined #oooq08:46
arxcruzsshnaidm|afk: marios_|ruck quiquell https://review.rdoproject.org/r/#/c/20900/09:03
arxcruzfix the linters09:03
arxcruzzbr: ^09:03
arxcruzsince this is your fault :P09:03
marios_|ruckarxcruz: will check in bit09:04
arxcruzapetrich: the men will go to the biergarten this saturday, if you want :)09:04
arxcruzdtantsur: ^09:04
arxcruzhttp://www.jamesjune.de/09:04
chandankumarmarios_|ruck: panda is there some problem with rdo registry https://logs.rdoproject.org/32/20832/10/check/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-queens/116ae5b/logs/undercloud/home/zuul/overcloud_prep_containers.log.txt.gz ?09:06
*** panda is now known as panda|rover09:11
panda|roverchandankumar: I've seen reachability problems for the registry since yesterday09:12
chandankumarpanda|rover: will i recheck the patch?09:13
apetricharxcruz, can't do. Thanks for inviting!09:13
panda|roverchandankumar: I rechecked and it worked, but you may want to signal the thing to jpena09:13
marios_|ruckpanda|rover: chandankumar seen an undercloud issue earlier to do with container pull i put link in the etherpad.09:14
marios_|ruckpanda|rover: o/ so what do you think about comment #3 - anything else we might need hope that will cover it https://bugs.launchpad.net/tripleo/+bug/182997609:24
openstackLaunchpad bug 1829976 in tripleo " periodic-tripleo-standalone-upgrade-stein fails upgrade fetching new containers "Not found image: ... cinder-api:tripleo-ci-testing" " [Critical,Fix released] - Assigned to Gabriele Cerami (gcerami)09:24
marios_|ruckwait09:25
marios_|ruckhow did stein promote?09:25
marios_|rucksince periodic-tripleo-standalone-upgrade-stein is borked on that ^09:26
panda|rovermarios_|ruck: https://bugs.launchpad.net/tripleo/+bug/183002509:26
openstackLaunchpad bug 1830025 in tripleo "periodic standalone upgrade job on stein fails to download images" [Critical,Fix committed] - Assigned to Gabriele Cerami (gcerami)09:26
panda|rovermarios_|ruck: The fix merged yesterday  https://review.opendev.org/66061509:27
marios_|ruckpanda|rover: ok can you update it with stuff. its already on https://bugs.launchpad.net/tripleo/+bug/1829976 but whatever :D09:27
openstackLaunchpad bug 1829976 in tripleo " periodic-tripleo-standalone-upgrade-stein fails upgrade fetching new containers "Not found image: ... cinder-api:tripleo-ci-testing" " [Critical,Fix released] - Assigned to Gabriele Cerami (gcerami)09:27
marios_|ruckpanda|rover: ack well 9 hours ago09:29
marios_|ruckpanda|rover: ok cool so does that mean the periodic was green I don't see it https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-7-standalone-upgrade-stein09:29
marios_|ruckpanda|rover: 'that' being we had a stein promotion09:29
marios_|ruckpanda|rover: and also, we don't need the retag?09:30
marios_|ruckpanda|rover: if you want to comment there i can abandon09:30
panda|rovermarios_|ruck: no, because we ended up using the same thing as everything else: container_build_id09:30
marios_|ruckhttps://review.opendev.org/#/c/660665/ panda|rover09:30
marios_|ruckpanda|rover: fantastischen09:30
dtantsurarxcruz: maybe :) I don't know my plans for the weekend yet09:32
*** ykarel|lunch is now known as ykarel09:34
ykarelmarios_|ruck, panda|rover i reran stein-upgrade job09:38
ykarelthat's why it prmote09:38
ykareli rechecked panda's patch09:38
marios_|ruckykarel: cool thanks was confused by the lack of green standalone-upgrade in the periodic09:38
marios_|ruckykarel: thank you so it is not too good to be true09:38
marios_|ruckstein promoted09:38
marios_|ruck\o/09:38
marios_|ruckhoping master will too09:38
ykarelmarios_|ruck, for more clarity u can see https://trunk-primary.rdoproject.org/api-centos-stein/api/civotes_detail.html?commit_hash=e7e5a7fc1b915f285751418d2ad3f157cc2bd287&distro_hash=24ce03e4280eebd0fa16c7964365c71dc2c5894009:41
ykarelupgrade-stein job passed in check09:41
marios_|ruckthanks ykarel09:46
*** dsneddon has joined #oooq09:48
marios_|ruckpanda|rover: is that you re real time alert for bug 181806009:51
openstackbug 1818060 in tripleo "Nodes periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master fail to get usable IPs though os-net-config with Error, some other host (BE:E5:4F:B9:21:B0) already uses address" [Critical,In progress] https://launchpad.net/bugs/1818060 - Assigned to Gabriele Cerami (gcerami)09:51
marios_|ruckpanda|rover: moved your note on etherpad to under the bug fyi https://etherpad.openstack.org/p/ruckroversprint1009:52
*** dsneddon has quit IRC09:53
*** derekh has quit IRC09:54
chandankumarmarios_|ruck: I am updating the job for 2ctlr as discussed with ykarel09:54
*** derekh has joined #oooq09:54
marios_|ruckchandankumar: ack you mean you have updated the configs in other projects (probably you'll end up with big list of depends-on in the original patch )09:55
chandankumarmarios_|ruck: no, I am going to define a new patch09:55
chandankumarwhere we will define new jobs and then remove old the older one09:55
marios_|ruckchandankumar: and the other projects will use the new job?09:56
marios_|ruckchandankumar: i mean won't you still have to update those?09:56
marios_|ruckchandankumar: anyway ack i mean do it and i can review the patch will be clear there :D09:56
chandankumarmarios_|ruck: yes let me put the patch09:56
chandankumarmarios_|ruck: I still have one todo left09:56
chandankumarupdating skip list09:56
marios_|ruckchandankumar: ack09:57
*** skramaja_ has joined #oooq10:05
*** skramaja has quit IRC10:06
chandankumarmarios_|ruck: let's go this way https://review.rdoproject.org/r/#/c/20902/10:17
marios_|ruckchandankumar: ack added to reviews will check later10:26
blueDuckRandomso... templates/roles... how do they work exactly? where I can set the settings for the overcloud network, for example? any resource besides the docs that anyone can recommend to understand the process?10:31
*** holser_ is now known as holser|lunch10:33
panda|roverarxcruz: how much the mailing system from tempestmail is reusable ?10:46
arxcruzpanda|rover:what you mean ?10:46
panda|roverarxcruz: if I want an instance in rdocloud to send an email now, would it be simple  ?10:46
arxcruzpanda|rover: no, because requires manually go there and set the permissions with the oauth since it's using my account10:47
arxcruzfor security reasons10:47
chandankumarpanda|rover: https://logs.rdoproject.org/19/20619/4/check/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-master/4c9166c/job-output.txt.gz#_2019-05-23_10_43_30_73584010:52
chandankumarpanda|rover: do i need to make somewhere more changes to create the ovb topology?10:52
chandankumarsince https://review.opendev.org/#/c/660710/ is merged10:52
*** amoralej is now known as amoralej|lunch11:01
*** jpena is now known as jpena|lunch11:01
chandankumarpanda|rover: I think I got the answer https://review.rdoproject.org/r/#/c/20906/11:10
*** udesale has quit IRC11:13
*** udesale has joined #oooq11:13
*** skramaja has joined #oooq11:17
*** skramaja_ has quit IRC11:18
chandankumarmarios_|ruck: kopecmartin ykarel https://review.rdoproject.org/r/#/c/20832/ -> kopecmartin patch worked11:19
chandankumarhttps://logs.rdoproject.org/32/20832/10/check/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-queens/f6c553f/logs/undercloud/var/log/extra/rpm-list.txt.gz11:19
chandankumarpython2-tempestconf-2.2.1-0.20190523093149.f78e343.el7.noarch11:19
marios_|ruckchandankumar: thanks great11:27
panda|roversshnaidm|afk: do you know if it's possible to ssh into bmc nodes during an OVB run ?11:36
chandankumarmarios_|ruck: for os_tempest rhel-8 part, I need to disable rhos-release-extras repo where should I put that?11:36
chandankumarmarios_|ruck: https://code.engineering.redhat.com/gerrit/#/c/171117/311:37
sshnaidm|afkpanda|rover, I saw derekh doing this trick once, but not sure how11:37
*** jaosorior has joined #oooq11:38
panda|roversshnaidm|afk: I'll ask him, thanks.11:38
marios_|ruckchandankumar: maybe in the release config file see that for example https://code.engineering.redhat.com/gerrit/#/c/167407/11/config/release/rhos-15.yml11:38
derekhpanda|rover: lots might have changed since I last did it it but here is what I did11:38
derekhpanda|rover: ssh -A undercloud # to forward your ssh key to the undercloud node11:39
chandankumarsshnaidm|afk: Hello11:39
derekhpanda|rover: then ssh USER@<BMCIP>11:39
chandankumarsshnaidm|afk: please have a look at this one https://logs.rdoproject.org/19/20619/5/check/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-master/8d39154/job-output.txt.gz#_2019-05-23_11_22_35_19290011:39
chandankumarsshnaidm|afk: I added this https://review.rdoproject.org/r/#/c/20906/ also but not working11:40
derekhpanda|rover: iirc that bmc used to have the same public key on it that the undercloud used11:40
derekhpanda|rover: you'll also need to be using a ssh key agent11:41
panda|roverderekh: mmhh, so there's no easy way to add it to the ansible inventory in undercloud and do stuff rom there.11:41
panda|roverderekh: do you know what's the base image used in bmc node ? is it centos 7 ?11:42
sshnaidm|afkchandankumar, you made a right review, but you need to merge it to work11:42
sshnaidm|afkchandankumar, it won't work as a dependency11:42
*** sshnaidm|afk is now known as sshnaidm11:42
panda|roverderekh: because there's a ifup-eth script that behaves differently from centos7 classic11:42
derekhpanda|rover: it used to be centos7 unless somebody has changed it11:42
sshnaidmchandankumar, this is trusted repo11:42
chandankumarpanda|rover: marios_|ruck: sshnaidm|afk: https://review.rdoproject.org/r/#/c/20906/ let's merge it then11:42
chandankumarthanks:-)11:43
sshnaidmarxcruz, thanks for fixing linters, weird stuff..11:47
arxcruzsshnaidm: :)11:47
arxcruzsshnaidm: I blame zbr11:48
*** jaosorior has quit IRC11:49
*** dsneddon has joined #oooq11:50
marios_|ruckykarel: aware of any rdo registry issues?11:52
ykarelmarios_|ruck, yes, see #rhos-ops, seems related to load, jpena and kforde looking11:53
ykarelmarios_|ruck, have u seen any failure in last one hour?11:53
marios_|ruckykarel: ok thanks. well i went looking to see what breaks master promotion and all of them are for that11:53
marios_|ruckykarel: i'll probably file a bug for tracking it collecting logs still11:54
marios_|ruckykarel: nope in last hour though these are all the morning periodic runs like 2019-05-23 04:07:3411:54
ykarelmarios_|ruck, ack11:54
*** dsneddon has quit IRC11:55
*** panda|rover is now known as panda|rover|eat11:59
*** tosky has quit IRC12:01
*** tosky has joined #oooq12:03
*** tosky__ has joined #oooq12:08
*** tosky has quit IRC12:11
*** rf0lc0 is now known as rfolco12:18
*** aakarsh has quit IRC12:23
weshaypanda|rover|eat when ur back.. shall I waive the upgrade test in stein periodic based on http://logs.rdoproject.org/67/20867/5/check/periodic-tripleo-ci-centos-7-standalone-upgrade-stein/ce72c4e/ ?12:26
weshayoh already done12:27
marios_|ruckweshay: stein promoted thanks to ykarel12:28
weshaycool :)12:28
weshaythanks marios_|ruck ykarel!12:28
marios_|ruckweshay: but master fcucked on https://bugs.launchpad.net/tripleo/+bug/183019812:28
openstackLaunchpad bug 1830198 in tripleo "multiple rdo master promotion jobs failing to pull containers " [Critical,Triaged]12:28
marios_|ruckweshay: but that was this morning's run. sounds like that might be ok now. otherwise we are mostly clear on the master promotino except the intermittent ovb issue .... status updated in https://etherpad.openstack.org/p/ruckroversprint1012:29
*** rlandy has joined #oooq12:29
*** jpena|lunch is now known as jpena12:30
*** sanjayu_ has quit IRC12:30
*** amoralej|lunch is now known as amoralej12:30
chandankumarkopecmartin: I am going to cherry-pick this review https://review.opendev.org/#/c/660748/ since it is blocked on packstack12:30
chandankumarsince gates are going red12:31
chandankumarin tempestconf spec file12:31
kopecmartinchandankumar, i do interviews now. ... you can send a new patch set there if you want , to fix the thing you mentioned in the comment12:33
*** tosky has joined #oooq12:33
kopecmartinchandankumar, or I 'll have a look in 35 minutes12:33
chandankumarkopecmartin: let me unblock the gates12:33
chandankumarthen do the other stuff12:33
zbrweshay: hi! meeting?12:33
weshayzbr welcome back sir12:34
weshayzbr ya.. let's briefly chat12:34
* weshay will catch you up12:34
*** tosky__ has quit IRC12:36
weshayzbr https://bluejeans.com/411356779812:36
rlandyweshay: the bm job you asked for passed - undercloud is still available for testing12:36
weshayawesome12:37
weshayrlandy marios_|ruck can we prep this for merge https://code.engineering.redhat.com/gerrit/#/c/167540/12:37
rlandyweshay: marios_|ruck: afaik, it is ready for merge.12:38
rlandyI remove marios_|ruck line for comment12:38
rlandyremoved12:38
marios_|ruckrlandy: weshay chandankumar and i updated for the co-authored and i added the forgotten voting: false !12:38
marios_|ruckunless we want it voting?12:38
weshayrock.. done12:38
*** tosky__ has joined #oooq12:38
rlandymarios_|ruck: I commented - I'm fine with the non-voting12:39
marios_|ruckrlandy: yeah i think it should be at least for now .. the 14 one is too12:39
rlandyat least until we are sure it's stable12:39
*** tosky has quit IRC12:40
*** tosky__ is now known as tosky12:40
marios_|ruckweshay: can you check that when you get a few (we can discuss@scrum in 20 anyway ) https://tree.taiga.io/project/tripleo-ci-board/task/1081?kanban-status=144727412:43
marios_|ruckweshay: i think i misunderstood something... or is zuul even running there https://code.engineering.redhat.com/gerrit/#/q/project:openstack-tripleo-common12:44
*** ratailor has quit IRC12:47
*** ykarel is now known as ykarel|afk12:48
*** aakarsh has joined #oooq12:48
sshnaidmpanda|rover|eat, what was the context of bmc question? Did you see problems with introspection in ovb recently?12:58
rfolcopanda|rover|eat, please answer my question at https://tree.taiga.io/project/tripleo-ci-board/task/995?kanban-status=1538166 when you get a chance. Thanks.12:59
*** aakarsh has quit IRC13:00
*** panda|rover|eat is now known as panda|rover13:00
*** aakarsh has joined #oooq13:00
*** sanjayu_ has joined #oooq13:01
rfolcoping scrum marios, sshnaidm, weshay, panda, rlandy, arxcruz, rfolco, chkumar, zbr, kopecmartin13:01
weshayrlandy mtg13:02
rlandyweshay: https://code.engineering.redhat.com/gerrit/171249 Add updates repos for rhos-15 needs_images13:03
weshayah thanks13:04
*** Goneri has joined #oooq13:06
*** holser|lunch is now known as holser_13:07
*** ykarel|afk is now known as ykarel13:12
weshayrfolco refresh the page please13:18
chandankumarweshay: https://tree.taiga.io/project/tripleo-ci-board/task/1080?kanban-status=144727513:35
rlandymarios_|ruck: hi - so wrt needs_images job, not sure if weshay updated you13:37
panda|rovermarios_|ruck: should we setup an hand-off meeting on friday evening or Monday morning ?13:38
rlandyit was running but not updating the correct images13:38
marios_|ruckrlandy: he mentioned it but whats up13:38
rlandymarios_|ruck: https://code.engineering.redhat.com/gerrit/171249 - going to run the job again with that change to check13:39
rlandyif that works, the job can merge13:39
marios_|ruckpanda|rover: sure man lets setup one for friday afternoon before we leave then we can also meet as necesary i mean i'm not going anywhere i'll still be here monday :D13:39
marios_|ruckrlandy: ok let me know if you need something/votes etc13:39
rlandymarios_|ruck: it's tracked nowhere though13:39
rlandymarios_|ruck: the reason I am asking is - is this relevant to the card you have for tripleo-common?13:40
rlandymarios_|ruck: ie: what release file are you using?13:40
marios_|ruckrlandy: not sure maybe we can have a new task under https://tree.taiga.io/project/tripleo-ci-board/us/95213:40
rlandyif you are using the needs-iamges one, I will track under your card13:40
marios_|ruckrlandy: right now i'm using the job one but maybe i need to change to that one13:41
marios_|ruckrlandy: the job didn't even run its what i was saying we don't know whats up with zuul there13:41
marios_|ruckrlandy: but anyway sure, add it there if you think related for now13:41
marios_|ruckrlandy: we can move later if we want to13:41
rlandyk13:42
rlandymarios_|ruck: this whole work is growing organically - just trying to get some organization here13:42
marios_|ruckrlandy: like i would object to taiga organisation (!)13:43
rlandythere are lots of things I'd like to do in this world - not all are practical13:44
*** dsneddon has joined #oooq13:51
*** dsneddon has quit IRC13:56
*** quiquell has quit IRC13:58
marios_|ruckykarel: do you know why none of the periodics are running for master now... like in https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-centos-7-master-containers-build-push i see it ran at 0200 but not again today?13:59
marios_|ruckykarel: same in https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master13:59
marios_|ruckykarel: its the same 'load issue' sorry if you don't know i'll go rhos-ops13:59
ykarelmarios_|ruck, see review.rdo it's running13:59
ykarelmarios_|ruck, https://review.rdoproject.org/zuul/status13:59
ykarelcontainer build passed, other dependent one just kicked14:00
marios_|ruckykarel: i see it, but still... seems much fewer runs than usual? or i'm wrong?14:00
marios_|ruckykarel: like for yesterday i see multiple runs in the morning14:00
ykarelmarios_|ruck, it run thrice daily14:01
zbr@oooq older molecule review, apparently not outdated: https://review.opendev.org/#/c/652971/ -- thanks14:02
*** yolanda__ has joined #oooq14:02
marios_|ruckykarel: ah sorry i'm confusing with the 'check' in https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-centos-7-master-containers-build-push14:02
marios_|ruckykarel: thanks14:02
ykarelhttps://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-centos-7-master-containers-build-push&pipeline=openstack-periodic-master14:02
marios_|ruckykarel: yeah14:02
ykarelso today's first run failed due to registry, then i reran it14:03
weshayzbr https://bluejeans.com/132529721814:03
*** yolanda__ is now known as yolanda14:03
chandankumarmarios_|ruck: honey bangder call14:03
marios_|ruckykarel: there was also failed container-build-push14:03
marios_|ruckchandankumar: ah thanks!14:03
weshaymarios_|ruck panda|rover ^ please join for now.. even if we're not sure what is to be done14:03
weshaychandankumar ^14:04
ykarelmarios_|ruck, yes14:04
marios_|ruckyah sorry weshay14:04
marios_|ruckjoining14:04
marios_|ruckweshay: got carried away imagining a master promotion14:04
panda|roverzbr: so openstack-tox-py35 is the job testing with molecule ?14:04
ykarelmarios_|ruck, yes14:04
zbrpanda|rover: yeah.14:04
chandankumarweshay: I am working on backprt the patch which is on our plate14:05
marios_|ruckthanks ykarel lets see crossing fingers14:05
marios_|ruckmaster promote would be NICE14:05
ykarelyes hopefully it will promote today14:05
chandankumarhttps://review.opendev.org/#/c/659601/ this one14:06
*** skramaja has quit IRC14:08
panda|roverzbr: reviewd14:09
weshaymarios_|ruck panda|rover chandankumar arxcruz join me in my blue after this call14:09
weshayto summarize14:09
marios_|ruckweshay: k14:12
weshaymarios_|ruck chandankumar panda|rover zbr https://bluejeans.com/u/whayutin/14:14
chandankumarweshay: marios_|ruck https://bugzilla.redhat.com/show_bug.cgi?id=171005214:20
openstackbugzilla.redhat.com bug 1710052 in openstack-nova "[RHOS 15] Tempest test failure: test_list_extensions" [Unspecified,Closed: wontfix] - Assigned to alifshit14:20
chandankumarbug loink14:20
chandankumarweshay: marios_|ruck http://pastebin.test.redhat.com/765344 etherpad link14:25
chandankumarzbr:14:26
chandankumarzbr: http://pastebin.test.redhat.com/76534414:26
arxcruzweshay: now ?14:26
rlandychandankumar: hi - going to put some time into the log move today14:28
rlandyfinally14:28
rlandyif nothing else crops up wrt rhos-1514:28
weshayhttps://docs.google.com/document/d/1xm3LooXddibhyZncjsZSBw9ZxBqiVfRjXcS-II5tQHU/edit#14:33
*** mjturek1 has joined #oooq14:48
*** mjturek1 is now known as mjturek14:48
*** Vorrtex has joined #oooq14:52
*** kopecmartin is now known as kopecmartin|afk14:53
*** apahim has joined #oooq14:54
*** dsneddon has joined #oooq14:57
*** sanjayu_ has quit IRC14:58
sshnaidmrlandy, do we have introspection in bm job working in upshift?15:02
rlandysshnaidm: we have introspection workin on bm - but it's run from a bm undercloud to a bm overcloud - upshift has little to do with it15:02
rlandyyou can view ...15:03
sshnaidmrlandy, ack15:03
rlandysshnaidm: just did a test run for weshay ...15:03
rlandyhttps://sf.hosted.upshift.rdu2.redhat.com/logs/15/165815/55/check/periodic-tripleo-ci-centos-7-bm_envD-3ctlr_1comp-featureset001-master/f921566/15:04
rlandysshnaidm: ^^ but I am not sure that will be helpful to you15:04
rlandysshnaidm: what may help if introspection on bm for rhel15:04
rlandyI am taking down the rockt periodic job to use that env to test  rhel bm on15:04
rlandyrocky15:05
rlandythat will be somewhat more similar15:05
rlandyat least from an env perspective15:05
weshaymarios_|ruck, panda|rover in your time.. can you guys please update the tripleo-upgrade gerrit to launch the appropriate standalone-upgrade jobs there please15:05
* weshay watching https://review.opendev.org/#/c/661009/315:06
rlandyweshay: why did you recheck https://code.engineering.redhat.com/gerrit/#/c/171119/?15:06
rlandyit was just about to report results15:07
weshayrlandy the settings on the container update needed a change15:07
rlandyweshay: oh I see15:07
rlandyok15:07
marios_|ruckweshay: ack you mean add standalone upgrade jobs to tripleo-upgrade repo?15:12
marios_|ruckpanda|rover: are you doing or should i?15:12
chandankumarweshay: marios_|ruck zbr https://code.engineering.redhat.com/gerrit/#/c/171266/ patch backported15:14
chandankumarwill be available in next compose15:14
rlandysshnaidm: weshay: I am moving the log in our internal sf tenant to rdo log server. I need to create the secret there. Do you have the password for rdo log sever?15:15
marios_|ruckweshay: panda|rover added the full template https://review.opendev.org/66103015:15
* chandankumar logs out15:16
chandankumarsee ya tomorrow15:16
*** chandankumar is now known as raukadah15:17
sshnaidmrlandy, no, I see there is a ssh key for it: https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/zuul.d/secrets.yaml#L915:17
sshnaidmrlandy, maybe you need a ssh key? rdo folks can provide it15:17
rlandyjpena: ^^ can I get the ssh key for the rdo log server to add it tripleo-ci-internal tenant in internal sf?15:19
*** ccamacho has quit IRC15:22
* marios_|ruck shutdown sequence 15:27
*** marios_|ruck has quit IRC15:35
*** marios has quit IRC15:35
*** ykarel is now known as ykarel|away15:39
panda|roverweshay: isn't using the entire scenarios-full for tripleo-upgrade in https://review.opendev.org/#/c/661030/  an overkill ? how many of those jobs even include tripleo-upgrade ?15:41
jpenarlandy: why do you want to log to the external log server? The internal SF instance has its own log server, I think we should use that one15:41
rlandyjpena: weshay wants to logs of the jobs we are running to be public15:42
weshaypanda|rover meh..  I had the same thought, however they can join the fold like everyone else15:42
weshayjpena public logs please :)15:42
panda|roverjpena: one reason is so we can open launchpad bugs and reference the logs, right now the logs are in private urls and are off limits.15:43
jpenaok, understood15:43
weshaythanks15:43
*** hamzy has quit IRC15:44
*** ykarel|away has quit IRC15:46
raukadaharxcruz: https://review.opendev.org/#/c/660913/ can we merge this15:48
arxcruzraukadah: done15:49
raukadaharxcruz: thanks!15:50
jpenarlandy: I've found the key. Is there a safe place where I can copy it for you to use it?15:50
rlandyjpena: ack - sending you pvt15:51
panda|roverfacebook post15:51
rlandylol15:51
rlandyweshay: one more for you ... https://code.engineering.redhat.com/gerrit/#/c/171278/ - equivalent change for rhos-15 an drhos-14 release files15:56
*** ykarel|away has joined #oooq16:08
raukadahsshnaidm: will i remove the check jobs?16:09
*** bogdando has quit IRC16:10
sshnaidmraukadah, what do you mean?16:23
raukadahsshnaidm: regarding this comment https://review.rdoproject.org/r/2090216:24
raukadahsorry hopping on multiple channel16:24
sshnaidmraukadah, yeah, where do you want to run 020 jobs?16:26
raukadahsshnaidm: i just want to run in periodic16:26
raukadahnot in gates16:26
raukadahor check16:27
weshaysshnaidm ya.. only periodic16:27
weshayit's way too slow for anything else16:27
raukadahthe job was added but never used16:27
sshnaidmraukadah, ok, then you don't need all your tripleo-ci-centos-* jobs, only periodic-*16:27
raukadahsshnaidm: ack removing it16:27
raukadahweshay: panda|rover we have sorted out queen promotion issue16:33
raukadahwe will get the promotion tomorroqw16:33
weshaythanks16:34
rlandyraukadah: weshay: sshnaidm: first attempt at the log move ... https://code.engineering.redhat.com/gerrit/#/c/171284/16:40
rlandyraukadah: weshay: sshnaidm: posting the key in the google doc16:41
raukadahrlandy: I will take a look in the morning, thanks :-)16:42
raukadahit will open the doors for gating upstream jobs from upshift16:43
raukadahwe need to figure out how to send votes to upstream gerrit16:43
raukadah*projects16:43
rlandyraukadah: the linters just objected .. and we would need to merge to test this16:44
rlandybut let's start getting approval/reviews16:44
weshayrlandy looking at the code.. I think we need to remove16:45
weshay  modify_only_with_labels:16:45
weshay        - kolla_version16:45
weshayhttps://opendev.org/openstack/tripleo-common/src/branch/master/tripleo_common/image/kolla_builder.py#L32316:45
weshaynot 100% sure though16:45
*** zbr has quit IRC16:45
rlandyweshay: I have four diff reviews out with that line - which one are we talking about :)?16:47
weshayrlandy so the container update in osp-15 did not work16:47
weshayI suspect because it's not finding any osp containers w/ kolla labels16:47
weshayrlandy I could put up a DNM patch that removes that16:48
weshayand retriggers16:48
weshayto test the theory16:48
weshayif that's ok w/ you16:48
rlandyweshay: sure16:48
* rlandy does not own that file :)16:49
rlandyweshay: skip for any osp16:49
weshayhttps://code.engineering.redhat.com/gerrit/#/c/171119/16:50
weshayjust to test16:50
ykarel|awayif it's just about to check kolla_version label, u can check with skopeo inspect osp image and check for labels16:52
rlandyweshay: on the upside ...getting somewhere with 'replace' in that file ... https://code.engineering.redhat.com/gerrit/#/c/170519/916:53
rlandygot one more edit to make16:53
weshaynice16:54
*** dtantsur is now known as dtantsur|afk16:54
weshayrlandy can we add something there for16:55
weshay        modify_only_with_labels:16:55
weshay          - kolla_version16:55
weshayI think we want it to be undefined / removed from the params, but we'll find out I guess16:56
*** blueDuckRandom has quit IRC16:56
* ykarel|away out16:57
rlandyweshay:I'd pull it out into a separate step16:58
weshayaye16:58
rlandyweshay: add it only if osp_release not defined16:58
*** derekh has quit IRC17:00
*** ykarel|away has quit IRC17:02
*** udesale has quit IRC17:03
weshayaye17:06
*** tosky has quit IRC17:08
*** tosky has joined #oooq17:12
*** jpena is now known as jpena|off17:19
*** tosky has quit IRC17:22
*** amoralej is now known as amoralej|off17:23
*** tosky__ has joined #oooq17:23
*** ykarel has joined #oooq17:24
*** hamzy has joined #oooq17:28
*** hamzy has quit IRC17:40
*** hamzy has joined #oooq17:40
*** holser_ is now known as holser|eod17:44
*** hamzy_ has joined #oooq17:46
*** hamzy has quit IRC17:49
rfolcoweshay, rhel7 pipeline should ignore dlrn ?17:50
*** dsneddon has quit IRC18:06
*** holser|eod has quit IRC18:08
*** dsneddon has joined #oooq18:10
*** rlandy is now known as rlandy|brb18:16
*** hamzy__ has joined #oooq18:26
*** hamzy_ has quit IRC18:26
*** jfrancoa has quit IRC18:30
weshayrfolco ignore dlrn? not sure what you are asking me... you mean the voting?18:32
weshayya.. doesn't need to vote18:32
weshayI don't think it will anyway.. I'll raise it in the prod chain18:32
weshaybut certainly doesn't need to vote in the near future18:32
*** ykarel has quit IRC18:32
*** hamzy__ has quit IRC18:34
rfolcoweshay, no, not voting... reporting to dlrn...18:40
weshayya.. that's what I mean, doesn't hurt if they do .. but it's not needed at this time18:40
rfolcoweshay, it would require changes, additional work, just asking if its ok to ignore dlrn for now18:41
*** rlandy|brb is now known as rlandy18:42
weshayya.. not needed, don't see it being needed at this level18:42
*** hamzy has joined #oooq18:43
rfolcoweshay, ok, that simplifies the job, thanks. Need some review before actually running the job with push=true... https://code.engineering.redhat.com/gerrit/17129718:43
rfolcojust defined the job there... once get a +1, can test it18:44
rfolcohmmm I think need to merge secret first.18:45
*** hamzy has quit IRC18:47
*** hamzy has joined #oooq18:47
rlandyweshay: wrt image building on bm - sometimes the image build will fail and the pass when run again ... https://sf.hosted.upshift.rdu2.redhat.com/logs/periodic-hourly/code.engineering.redhat.com/openstack/tripleo-ci-internal-jobs/master/periodic-tripleo-ci-centos-7-bm_envA-3ctlr_1comp-featureset001-master/9bd2272/logs/undercloud/home/zuul/overcloud_image_build.log.txt.gz18:56
rlandyare we triggering too soon?18:56
weshayrlandy I think you hit an infra error 2019-05-23 16:15:52 | 2019-05-23 16:15:52.779 | http://mirror.centos.org/centos/7/rt/x86_64/repodata/repomd.xml: [Errno 14] curl#6 - "Could not resolve host: mirror.centos.org; Unknown error"18:57
rlandyweshay: yeah - but it is not uncommon18:57
weshayrlandy right.. I suspect there is an internal centos mirror18:58
* rlandy needs to get this into cockpit for tracking18:58
rlandyweshay: k, will deal with it after rhos is set up on bm18:58
weshayrlandy pm18:59
rlandyand this might finally be workable :) ... https://review.opendev.org/#/c/659794/19:02
rfolcoweshay, rlandy sshnaidm need to merge this quick one to continue testing. Cannot even test my container push patch without this one merged. https://code.engineering.redhat.com/gerrit/#/c/171299/19:02
rfolcoplease please19:03
rfolcoplease19:03
rfolcoI beg19:03
rlandyrfolco: relax - merging that19:03
rfolco\o/19:03
rfolcothanks :)19:04
*** hamzy has quit IRC19:05
*** tesseract has quit IRC19:05
rlandyweshay: did container update work w/o  kolla_version?19:17
weshayno.. see #tripleo19:19
rfolcorlandy, you know what I am missing here ? https://code.engineering.redhat.com/gerrit/#/c/171297/19:25
rfolcostill seeing  Decryption failed.19:25
rfolcoI thought just merging the secret in config was enough19:25
*** hamzy has joined #oooq19:26
rlandyrfolco: how did you formulate the secret?19:31
rfolcorlandy, I copied from rdo-infra/config19:31
rfolcohttps://github.com/rdo-infra/review.rdoproject.org-config/blob/ab585569489594c0070146bd2221133d01a9a419/zuul.d/secrets.yaml#L64719:32
rlandyrfolco:you need a secret encrypted for this tenant19:32
rfolcoisn't this an encrypted one ?19:33
rlandyrfolco: sending you pm19:33
rfolcok19:34
rlandyrfolco: https://www.softwarefactory-project.io/zuul-hands-on-part-5-job-secrets.html19:37
rfolcoaaaah19:38
rfolcoI think I got it19:38
rfolcoI'll have to type the password used by rdo registry to generate the secret19:39
rfolcorlandy, thx19:40
*** Goneri has quit IRC20:08
*** tosky__ is now known as tosky20:18
*** aakarsh has quit IRC20:36
*** Vorrtex has quit IRC20:50
*** LaurentDumont has joined #oooq21:02
rlandyweshay: when you have a moment. I think this 'containers-prepare-parameters replace' might actually be workable now https://review.opendev.org/#/c/659794/21:07
rlandypasses rhos-15 and 14 .. https://code.engineering.redhat.com/gerrit/#/c/170519/21:08
rlandyI know we still need the right repo updates21:08
rlandyand looks like it passes upstream centos7 and fedora2821:08
weshaycool21:09
rlandyNOT pushing to merge this - just for the team to review21:09
weshayrlandy k k21:09
weshayit looks good.. I'll try it in a bit21:09
weshaydissecting the reproducer atm21:10
weshaythe only thing I know for sure.. is that21:10
weshay              modify_only_with_labels:21:10
weshay              - kolla_version21:10
rlandyweshay: no rush - working on bm now21:10
weshayneeds to be in it's own task21:10
rlandynp - can easily change that21:11
weshaycool21:11
*** jtomasek has joined #oooq21:12
*** jtomasek has quit IRC21:25
rlandyadded some j2 - seeing if that works on https://review.opendev.org/#/c/659794/10/roles/standalone/tasks/main.yml21:30
rlandytesting https://code.engineering.redhat.com/gerrit/#/c/170519/1021:32
*** Goneri has joined #oooq21:38
*** Goneri has quit IRC22:08
*** panda|rover has quit IRC22:59
*** panda has joined #oooq22:59
*** tosky has quit IRC23:05
*** rlandy has quit IRC23:06
*** LaurentDumont has quit IRC23:56

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!