Wednesday, 2019-08-14

*** aakarsh has joined #oooq00:52
*** hamzy has joined #oooq01:17
*** saneax has quit IRC01:19
*** rfolco has quit IRC01:52
*** saneax has joined #oooq02:05
*** bhagyashris has joined #oooq02:43
*** ccamacho has joined #oooq03:06
*** sshnaidm|afk has quit IRC03:10
*** sshnaidm|afk has joined #oooq03:23
*** ykarel|away has joined #oooq03:38
*** gkadam has joined #oooq03:41
*** gkadam has quit IRC03:41
*** gkadam has joined #oooq03:43
*** udesale has joined #oooq03:54
*** gkadam has quit IRC04:00
*** ykarel|away is now known as ykarel04:11
*** dsneddon has quit IRC04:24
*** raukadah is now known as chkumar|ruck04:33
*** marios has joined #oooq05:02
*** ratailor has joined #oooq05:08
*** dsneddon has joined #oooq05:11
*** dsneddon has quit IRC05:17
*** ratailor has quit IRC05:20
*** dsneddon has joined #oooq05:21
*** dsneddon has quit IRC05:26
*** ykarel is now known as ykarel|afk05:26
*** jfrancoa has joined #oooq05:27
*** kopecmartin|off is now known as kopecmartin05:36
*** ratailor has joined #oooq05:46
*** dsneddon has joined #oooq05:58
*** dsneddon has quit IRC06:03
*** skramaja has joined #oooq06:18
*** jtomasek has joined #oooq06:21
* marios relocate biab06:25
*** marios has quit IRC06:26
*** ade_lee__ has quit IRC06:46
*** dsneddon has joined #oooq06:47
*** dsneddon has quit IRC06:51
*** dsneddon has joined #oooq07:01
jfrancoasshnaidm|afk: hey, good morning. I'm checking the logs from the rhel8 job and it seems to fail because it can't locate the container 2019-08-13 17:30:11 | tripleo_common.image.exception.ImageNotFoundException: Not found image: docker://trunk.registry.rdoproject.org/tripleomaster/rhel-binary-cinder-api:65570a311e1303d8084d12ee82868e6d0f411e8a_41250740 . Is it possible to access to the registry to check if the image is there?07:02
*** ccamacho has quit IRC07:10
*** tesseract has joined #oooq07:10
*** bogdando has joined #oooq07:17
*** amoralej|off is now known as amoralej07:18
*** ykarel|afk is now known as ykarel07:19
chkumar|ruckjfrancoa: hello07:21
chkumar|ruckjfrancoa: it is very old hash07:21
chkumar|ruckjfrancoa: it is one of the passed job https://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-standalone-master/69e55b5/logs/07:22
*** ykarel is now known as ykarel|afk07:33
*** marios has joined #oooq07:39
*** jpena|off is now known as jpena07:39
*** apetrich has joined #oooq07:45
*** ykarel|afk has quit IRC07:47
jfrancoachkumar|ruck: thanks a lot for the log, so in theory the job seems to be passing right?07:49
chkumar|ruckjfrancoa: it is passing in periodic not in master07:50
chkumar|ruckjfrancoa: we are waiting for the master promotion07:50
jfrancoachkumar|ruck: ahh ok..by the way, how do you trigger it? via "check-rdo" comment or something like that?07:50
chkumar|ruckjfrancoa: https://review.rdoproject.org/r/#/c/21649/34/zuul.d/projects.yaml07:52
chkumar|ruckjfrancoa: you can test it in rdo-jobs07:52
chkumar|ruckjfrancoa: by using check-rdo comment, it will rerun the rdo third party job07:52
jfrancoachkumar|ruck: but this job isn't still part of those third party jobs, right?07:53
jfrancoachkumar|ruck: by the way, so far we have only a standalone rhel8 job, right? are there plans to have a ovb multinode one (3 ctrlrs) ?07:53
chkumar|ruckjfrancoa: we have rhel8 standalone check job running in tripleo quickstart, extras and ci07:55
chkumar|ruckjfrancoa: ovb is in progress07:55
chkumar|ruckjfrancoa: marios is working on standalone scenario 1-4 https://review.rdoproject.org/r/#/c/21787/07:56
chkumar|ruckjfrancoa: for ovb fs01 https://review.rdoproject.org/r/#/c/21649/ sshnaidm|afk is testing here07:56
jfrancoachkumar|ruck: thanks a lot for the info. I was planning on working in a minor updates job, but it would make much more sense in an ovb scenario than in the standalone07:56
jfrancoachkumar|ruck: cool, that's useful. I'll have a look at the job definition to plan ahead how could we fit the tripleo-upgrade role to perform the update07:57
*** ykarel|afk has joined #oooq08:02
*** ykarel|afk is now known as ykarel08:04
*** jbadiapa has joined #oooq08:12
*** derekh has joined #oooq08:29
mariossshnaidm|afk: o/ thanks for checking replied at https://review.rdoproject.org/r/#/c/21787/4/zuul.d/standalone-jobs.yaml & see design https://tree.taiga.io/project/tripleo-ci-board/us/1228 re anchors and parenting check vs periodics08:32
*** dsneddon has quit IRC08:32
arxcruzi'll be a little bit afk, new computer arrives, you know... ;)08:46
mariosarxcruz: nice :D08:48
arxcruzyup, now i need to return the old one, that is a lot of things to do :@08:49
arxcruzI wanted to buy it just to not have the work to return back08:50
arxcruzanything that makes me move out of my house, it's a problem for me :/08:50
arxcruzyes, i am lazy08:50
pandaanyone up for https://review.rdoproject.org/r/21593 ?08:51
mariospanda: MAKE ME!08:58
marios(done) /me recaffeinate brb08:59
*** dsneddon has joined #oooq09:02
*** ykarel is now known as ykarel|afk09:02
*** jbadiapa has quit IRC09:03
pandamarios: thanks.09:08
*** dsneddon has quit IRC09:17
*** sshnaidm|afk is now known as sshnaidm09:23
chkumar|rucksshnaidm: marios https://review.opendev.org/#/c/676378/ for reducing the load on rdo log server09:25
sshnaidmmarios, hi, I read a story about anchors/parenting, but it's still not clear to me09:25
sshnaidmchkumar|ruck, +209:26
*** dsneddon has joined #oooq09:28
*** ykarel|afk is now known as ykarel09:29
mariossshnaidm: o/ did you read the "Check" part in design at https://tree.taiga.io/project/tripleo-ci-board/us/122809:30
mariossshnaidm: which part wasn't clear? "cannot use parenting because the periodics have an extra parent that the check jobs do not have"09:31
mariossshnaidm: that's the main part ^09:31
sshnaidmmarios, yeah, I don't understand why you don't inherit from centos standalone, like c7 sc001 -> rhel8 sc001, etc09:31
mariossshnaidm: lets be specific cos maybe i missed something. so here https://review.rdoproject.org/r/#/c/21787/5/zuul.d/standalone-jobs.yaml@279 periodic-tripleo-ci-rhel-8-scenario001-standalone-master09:32
mariossshnaidm: which job should that inherit from?09:32
mariossshnaidm: sorry wrong example... the issue is the check jobs09:32
mariossshnaidm: so here: https://review.rdoproject.org/r/#/c/21787/5/zuul.d/standalone-jobs.yaml@328 tripleo-ci-rhel-8-scenario001-standalone-rdo09:33
mariossshnaidm: which job should that inherit from ( tripleo-ci-rhel-8-scenario001-standalone-rdo )09:33
*** dsneddon has quit IRC09:33
sshnaidmmarios, I mean, tripleo-ci-rhel-8-scenario001-standalone-rdo can inherit from tripleo-ci-centos-7-scenario001-standalone-rdo09:40
sshnaidmmarios, sorry, from regular tripleo-ci-centos-7-scenario001 job09:40
sshnaidmnot from tripleo-ci-rhel-8-standalone-rdo09:41
mariossshnaidm: but what about the tripleo-ci-base-standalone-rdo that all the other rdo check jobs have? just ignore it and go directly upstream?09:41
mariossshnaidm: so you see the problem at least right? the rhel ones can use inheritance cos we have periodic scen1/2/34 to inherit from. However the check jobs can't because they have dfferent parent. does that make sense at least now?09:42
sshnaidmmarios, why not? when changing something in sa job upstream, you don't need to do it in rdo09:42
mariossshnaidm: which is why i use anchors. are you objecting to the use of anchors?09:42
sshnaidmmarios, I don't object to anchors, I object to duplicate definitions and duplicate maintenance09:43
mariossshnaidm: but then why don't we do that for all of them like all the centos7/rhel8 stuff shoudl all go directly upstream but it doesn't currently we have parent jobs in rdo they share and same for periodics with extra pre/post plays09:43
zbr|flumoning!09:43
mariossshnaidm: there is no duplicate definitions here though09:43
sshnaidmmarios, we didn't run sa rdo jobs in check pipeline in upstream ever09:44
sshnaidmmarios, so it's a new feature09:44
sshnaidmmarios, all previous sa jobs were in periodic pipeline only09:44
mariossshnaidm: but what about that though /me checking that pre https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/zuul.d/tripleo-rdo-base.yaml#L7109:45
mariossshnaidm: i.e. we miss that parent ^09:45
mariossshnaidm: hmm that looks important (all the nodepool/zuul stuff)09:46
mariossshnaidm: ?09:46
mariossshnaidm: besides that, i still don't get the objection? there is no duplicate definition here?09:46
sshnaidmmarios, seems like we have it here: https://github.com/openstack/tripleo-ci/blob/master/playbooks/openstack-zuul-jobs/legacy/pre.yaml#L109:47
sshnaidmmarios, if tempest_test_whitelist changes tomorrow for sc002, how many changes should we do?09:49
mariossshnaidm: as many as you have to do today09:49
mariossshnaidm: upstream for scen2 and rdo for centos09:49
mariossshnaidm: (the rhel ones either inherit from the centos, or use the anchors from it)09:49
marios12:46 < marios> sshnaidm: besides that, i still don't get the objection? there is no duplicate definition here?09:50
mariossshnaidm: i am not adding anything extra09:50
sshnaidmmarios, yeah, you're right, missed that09:50
sshnaidmthought we need to do more changes09:50
* marios was worried i missed something too :D 09:50
sshnaidmmarios, no, that's fine, still prefer to use inheritance instead of anchors though, but won't block on that09:51
mariossshnaidm: like i couldn't get what you're saying. yes it would be *nicer* to inherit form upstream but then a) its different to what we have for the rest of the rdo check jobs (and so could be confusing ) and we'd have to work out running that pre in upstream instead and then b) it doesn't buy us anything anyway cos we can at least use anchors09:51
mariossshnaidm: i wanted to use inheritance. i used inheritance for the periodics cos it was feasible but for the check we had that issue with different parent.09:52
mariossshnaidm: anchors are not ideal , but at least it means we only have to change 1 place09:52
sshnaidmmarios, why do we need check scenario jobs for rhel8 at all?09:53
sshnaidmmarios, going step back :D09:53
mariossshnaidm: yeah... that was on the card when i picked it up so i guess it came from PM... now i also thought about it.. i think it makes sense because for rhel we don't have upstream rhel8 scenario1/2/3/4 so at least lets have them in rdo check09:55
sshnaidmmarios, I can understand why we need periodic09:58
sshnaidmmarios, but I hardly can imagine that we run these jobs on check, yesterday we limited a regular rhel8 sa job to run on one repo only because rdo ci was overloaded09:58
mariossshnaidm: it allows us to catch things that will hurt rhel deployments09:59
sshnaidmmarios, but we don't have resources for that09:59
mariossshnaidm: that is the benefit ^ now yes there is a cost as we are increasing jobs09:59
mariossshnaidm: so then let's bring that to the discussion in scrum and either change the card to remove those or request resources09:59
sshnaidmmarios, you can still catch them every 4 hours09:59
mariossshnaidm: yeah but then you have to work out which patch it was09:59
sshnaidmmarios, yeah, but it's not a biggie to catch a patch during 4 hours, I think it may also be good for experimental jobs - on demand10:00
sshnaidmmarios, but I don't think anybody uses these experimental jobs10:00
mariossshnaidm: ack - so its a valid concern re resourcing... we need to at least bring attention to it. if you feel like it add a note on the card  https://tree.taiga.io/project/tripleo-ci-board/us/1228 otherwise we can discuss more once i post the change to layout in config repo in few mins10:01
sshnaidmmarios, anyway, back to patch - do you have some dummy patch with these jobs running?10:01
mariossshnaidm: not yet ... going to work on adding the layout and trying to get test runs today10:02
sshnaidmmarios, ok, let's see at least anchors syntax is right and does what it needs to do10:02
chkumar|rucksshnaidm: https://review.rdoproject.org/r/#/c/21831/ -> will fix rhel-8 check jobs10:02
mariossshnaidm: ack10:03
*** apetrich has quit IRC10:03
sshnaidmchkumar|ruck, great, thanks10:04
*** jfrancoa has quit IRC10:10
sshnaidmarxcruz, chkumar|ruck where do we build tripleomaster/centos-binary-tempest ?10:21
sshnaidmarxcruz, chkumar|ruck need to build it for rhel8 too10:22
chkumar|rucksshnaidm: https://logs.rdoproject.org/00/21800/8/check/periodic-tripleo-rhel-8-master-containers-build-push/324334c/logs/containers-successfully-built.log.txt.gz10:23
chkumar|rucksshnaidm: it is already built10:23
*** jfrancoa has joined #oooq10:23
chkumar|ruckjust bottom10:24
sshnaidmchkumar|ruck, seems like it's hardcoded somewhere: http://logs.rdoproject.org/49/21649/34/check/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-master/493ec30/logs/undercloud/home/zuul/tempest.log.txt.gz10:24
chkumar|rucksshnaidm: I know10:24
chkumar|rucksshnaidm: give me a min10:24
sshnaidmchkumar|ruck, it's tempest_container_namespace variable10:25
chkumar|rucksshnaidm: https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/validate-tempest/defaults/main.yml#L3410:26
chkumar|rucksshnaidm: I think we need to use ansible_distribution| lower will fix the issue?10:26
chkumar|rucklet me put a patch10:27
sshnaidmchkumar|ruck, why "lower"?10:28
chkumar|ruckansible_distribution returns RedHat10:28
chkumar|ruckcontainer name is in lower case10:29
chkumar|rucksorry10:31
chkumar|ruckit will not work10:31
chkumar|ruckrhel there10:31
*** bhagyashris has quit IRC10:38
mariospanda: ?11:04
*** udesale has quit IRC11:05
*** udesale has joined #oooq11:12
*** udesale has quit IRC11:16
chkumar|rucksshnaidm: boom11:20
chkumar|rucksshnaidm: http://38.145.34.55/redhat8_master.log11:20
sshnaidmchkumar|ruck, \o/11:20
chkumar|rucksshnaidm: and we have broken results11:28
chkumar|rucksshnaidm: https://review.rdoproject.org/r/#/c/21833/11:28
*** dsneddon has joined #oooq11:29
*** dsneddon has quit IRC11:34
*** jpena is now known as jpena|lunch11:35
*** skramaja has quit IRC11:36
sshnaidmchkumar|ruck, shouldn't we stop all fedora jobs and promotions?11:37
chkumar|rucksshnaidm: yes11:37
*** jaosorior has joined #oooq11:38
ykarelfedora jobs are already stopped11:39
chkumar|ruckwill remove it by evening11:39
*** gchamoul has joined #oooq11:58
*** dsneddon has joined #oooq11:59
*** ratailor has quit IRC12:02
*** dsneddon has quit IRC12:04
*** dsneddon has joined #oooq12:10
*** dsneddon has quit IRC12:16
*** rlandy has joined #oooq12:22
*** rlandy is now known as rlandy|rover12:22
rlandy|roverchkumar|ruck: hi12:25
rlandy|roverchkumar|ruck: when you have time, can we sync on the dlrn issue?12:25
*** rfolco has joined #oooq12:26
chkumar|ruckrlandy|rover: Hello12:30
chkumar|ruckrlandy|rover: yes sure12:31
rlandy|roverchkumar|ruck: bj?12:31
*** jpena|lunch is now known as jpena12:31
chkumar|ruckrlandy|rover: yes https://bluejeans.com/157131391912:32
*** dsneddon has joined #oooq12:49
*** apetrich has joined #oooq12:50
marioschkumar|ruck: rlandy|rover can you please update the program call document. there was no one on the call today apparently :/ ??12:52
rlandy|rovermarios: ack - will update and attend next week12:53
mariosrlandy|rover: chkumar|ruck i just got told off about it :/ ack thanks rlandy i'm bringing up promotions now to check12:53
mariosrlandy|rover: green promotion 13th12:54
mariosrlandy|rover: (stein is the thing we care about only but actually promotions looks great all the things green12:54
mariosscreen shot it !12:54
rlandy|rovermarios: k12:54
*** jaosorior has quit IRC12:56
*** dsneddon has quit IRC12:56
rlandy|rovermarios: doc updated12:58
rlandy|rovermarios: let us know if we missed anything12:59
chkumar|ruckmarios: sshnaidm panda https://review.opendev.org/#/c/676214/ can we merge it13:00
mariosack rlandy|rover thanks weshay_pto usually removes anything that isn't S but ack lgtm13:00
chkumar|ruckrlandy|rover: https://trello.com/c/67kkbRPR/1074-cixlp1835615tripleociproa-fs01master-error-stagemain-tripleoprofilepacemakerrabbitmqbundle-rabbitmqpolicyha-all-could-not-evalua13:01
*** dsneddon has joined #oooq13:02
*** vinaykns has joined #oooq13:02
chkumar|ruckrlandy|rover: https://trello.com/c/67kkbRPR/1074-cixlp1835615tripleociproa-fs01master-error-stagemain-tripleoprofilepacemakerrabbitmqbundle-rabbitmqpolicyha-all-could-not-evalua13:03
chkumar|rucksshnaidm: https://review.opendev.org/#/c/676378/ and https://review.opendev.org/#/c/675563/ are good to go13:05
pandazbr|flu: I don't understand why the file from task_from is not found in my registry login integration13:06
zbr|fluask on zuul, probably relative paths work bit different when with zuul, or is using another role with same name?13:07
rlandy|roverchkumar|ruck: W+ thosed13:07
chkumar|ruckthanks rlandy|rover :-)13:08
*** dsneddon has quit IRC13:08
*** amoralej is now known as amoralej|lunch13:08
*** ykarel is now known as ykarel|away13:15
*** amoralej|lunch is now known as amoralej13:28
chkumar|ruckrlandy|rover: https://review.opendev.org/#/c/676214 can I update this patch?13:34
rlandy|roverchkumar|ruck: sure - go4it13:35
chkumar|ruckcool13:35
rlandy|roverthe gate failures seem to be somewhat ok today13:35
rlandy|roverchkumar|ruck: initially I thought we shoudl do the update in a second patch13:35
rlandy|roverto get the timeout in - but it's ok13:36
*** amoralej is now known as amoralej|brb13:36
*** dsneddon has joined #oooq13:36
chkumar|ruckrlandy|rover: https://review.opendev.org/#/c/676386/ is also good to go13:36
chkumar|ruckfor rhel8 ovb13:36
rlandy|roverdone13:38
rlandy|roverchkumar|ruck: actually now that I think about it - https://review.opendev.org/#/c/676386/1/roles/validate-tempest/defaults/main.yml will fail fedora13:42
rlandy|roverI know we are getting of fedora13:42
rlandy|roverjust a thought13:43
chkumar|ruckrlandy|rover: yes, do we need to care about that?13:43
chkumar|ruckthere is just a standalone job13:43
chkumar|ruckwhere tempest container is not used13:43
chkumar|ruckso we are good there13:44
rlandy|roverchkumar|ruck: k - just checking13:44
rlandy|roverpost review - which is the wrong time - but whatever13:44
rlandy|roverchkumar|ruck: I think in most places we did this:13:44
rlandy|roverif rhel - > use rhel thing else -> use centos thing13:45
rlandy|rovercovers more cases13:45
rlandy|roverfor future ref13:45
chkumar|ruckyes you are correct13:47
*** aakarsh has quit IRC13:48
rfolcopanda, the design doc for staging env is the spec ?13:48
*** jaosorior has joined #oooq13:51
*** dsneddon has quit IRC13:52
sshnaidmrlandy|rover, marios please take a look: https://review.opendev.org/#/c/67625414:01
mariosack sshnaidm14:07
*** dsneddon has joined #oooq14:09
rlandy|roversshnaidm: done14:11
rlandy|roverchkumar|ruck: https://ci.centos.org/view/rdo/view/promotion-pipeline/job/rdo_trunk-promote-queens-current-tripleo/ - all green now :)14:11
rlandy|roverwill watch for next promotion14:12
*** dsneddon has quit IRC14:13
pandarlandy|rover: as far as I know, yes.14:15
pandarfolco: ^14:15
pandarlandy|rover: sorry14:15
rlandy|roverpanda: :)14:15
pandarfolco: until we transfer this somewhere else14:15
rfolcopanda, ok will review there then14:16
*** jaosorior has quit IRC14:16
*** amoralej|brb is now known as amoralej14:16
rlandy|roverhttps://review.opendev.org/#/c/676229/14:17
rlandy|roverthat came out better from testing14:17
rfolcopanda, I am looking at the push-containers.yaml file to see what are the necessary changes to make it testable with mock containers.... we have tasks like tag distro-binary-container...14:17
pandarfolco: that can be solved by cleverly naming the mock containers14:18
rfolcopanda, yup14:20
rlandy|roverchkumar|ruck: you're attending the escalation call or should I?14:22
*** aakarsh has joined #oooq14:22
chkumar|ruckrlandy|rover: I can take care of that14:22
rlandy|roverchkumar|ruck: cool - I'll go from next week14:22
chkumar|ruckrlandy|rover: sure14:23
*** Vorrtex has joined #oooq14:24
*** dsneddon has joined #oooq14:36
rlandy|roverchkumar|ruck: https://review.rdoproject.org/r/21841 Remove F28 from dlrn promoter files14:43
rlandy|roveradding rrcockpit stuff14:43
rlandy|roverrfolco: where was your last review to remove f28 jobs?14:43
rlandy|roverI think there is one left to merge14:44
chkumar|ruckrlandy|rover: cool, thanks :-)14:44
chkumar|ruckrlandy|rover: https://trello.com/c/ZvV5ul81/1029-cixlp1836046tripleociproa-tempestscenariotestnetworkbasicopstestnetworkbasicops-failing-on-queens14:45
chkumar|ruckfor this , we need to push some patches on tempest side14:45
chkumar|rucki mean in tq side14:46
*** dsneddon has quit IRC14:47
* rlandy|rover looks14:50
rlandy|roverchkumar|ruck: push patches that are ready or create them?14:50
rlandy|roveron the tq side14:52
chkumar|rucksshnaidm: https://review.opendev.org/#/c/676428/1/sigs.yaml14:53
rlandy|rovermarios: do you have +2 on review.rdoproject.org/config repo?14:53
mariosrlandy|rover: nop14:55
*** dsneddon has joined #oooq14:55
rlandy|rovermarios: I think it's time :)14:56
* marios hides14:56
marios;)14:56
rlandy|roverwill review that list later14:56
mariosthanks rlandy|rover14:56
*** ykarel|away has quit IRC14:56
*** dsneddon has quit IRC15:01
chkumar|ruckmarios: sshnaidm rlandy|rover https://review.opendev.org/#/c/676439/15:04
* chkumar|ruck logs out15:05
chkumar|rucksee ya15:05
*** chkumar|ruck is now known as raukadah15:05
rlandy|roverraukadah: hey a couple questions on https://review.opendev.org/#/c/676439/1 when you get back15:08
rlandy|roversshnaidm: wrt remove fedora promotions from rrcockpit, do you know if it's safe to just remove the displayed elements?15:09
rlandy|roverI had made an edit to the promoter config files15:10
rlandy|roverkind of a mystery which files it's actually using15:10
rlandy|roversshnaidm: also to remove a display element, I would have to rebuild all of ci-scripts/infra-setup/roles/rrcockpit/files/grafana/cockpit.dashboard.json?15:11
sshnaidmrlandy|rover, yeah, you need to deploy it in your machine, remove there elements and generate a new dashboard json file15:11
sshnaidmrlandy|rover, marios did it recently, he updated a readme file with details15:11
mariosrlandy|rover: see https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/infra-setup/roles/rrcockpit/README.md hope its useful15:14
rlandy|roversshnaidm: also if you ssh to the promoter server15:17
sshnaidmrlandy|rover, what do you mean?15:18
rlandy|roverif I read this correctly, it's picking up a conf file that is not being updatedd15:18
sshnaidmarxcruz, raukadah take a look at tempest failure on rhel8 please in your time: http://logs.rdoproject.org/49/21649/35/check/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-master/df63fb1/logs/undercloud/home/zuul/tempest.log.txt.gz15:19
rlandy|roversshnaidm: sent you tamte session15:19
sshnaidmarxcruz, raukadah what does that mean "FileNotFoundError: [Errno 2] No such file or directory: 'ping': 'ping'"15:19
*** ykarel|away has joined #oooq15:21
* marios almost out15:21
*** marios is now known as marios|out15:28
rlandy|roverhttps://review.rdoproject.org/r/#/c/21841/15:32
rlandy|roverpanda: sshnaidm: ^^ any objections to merging this?15:33
*** dsneddon has joined #oooq15:33
sshnaidmrlandy|rover, +w15:35
rlandy|roverthanks15:35
*** dsneddon has quit IRC15:39
sshnaidmrlandy|rover, do you remember if we have rhel qcow2 images updating and promotion?15:42
*** jfrancoa has quit IRC15:43
rlandy|roversshnaidm: hmmm ...15:43
* rlandy|rover checks15:43
rlandy|roverwe have the ipa and overcloud15:43
sshnaidmrlandy|rover, but we didn't promote with script?15:43
rlandy|roveryes15:43
sshnaidmrlandy|rover, yes - with script or without?15:43
rlandy|roverdepends which image you are talking about15:44
sshnaidmrlandy|rover, overcloud images15:44
rlandy|roversshnaidm: those get updated every promotion15:44
sshnaidmrlandy|rover, ok, I see the promote script is still not for rhel8..15:44
sshnaidmwas there any task to make promote images script for rhel8?15:45
rlandy|roverraukadah just added that - may it was just containers15:45
rlandy|roverI am checking the server15:45
rlandy|rover[centos@rcn-share www]$ cd  rcm-guest/images15:47
rlandy|rover[centos@rcn-share images]$ ls15:47
rlandy|roverredhat8  rhel8  rhel_base15:47
rlandy|roverlrwxrwxrwx.  1 centos centos   49 Aug 14 13:02 tripleo-ci-testing -> 6cd9e55d69e90ea73219abced4e9a3f2372b204e_10e135ca15:48
rlandy|roversshnaidm: ^^ only tripleo-ci-testing15:48
rlandy|roverso we are not promoting images to tripleo-ci15:48
sshnaidmrlandy|rover, yeah, that's right..15:49
rlandy|roverwe are uploading and soft linking15:49
rlandy|roverbut that's all15:49
sshnaidmrlandy|rover, working on promote script..15:49
rlandy|roversshnaidm++15:49
rlandy|roverhttp://38.145.34.55/redhat8_master.log15:50
rlandy|roversshnaidm: ^^ our first redaht8 promotion log15:50
rlandy|rover2019-08-14 13:24:01,101 11875 ERROR    promoter QCOW IMAGE UPLOAD FAILED LOGS BELOW:15:50
rlandy|rover2019-08-14 13:24:01,102 11875 ERROR    promoter ERROR: The promoted hash was not found: https://images.rdoproject.org/rhel8/master/rdo_trunk/5877cbf66372dba04d3a1b84016b83d4e06320bc_16fcf7dc15:50
rlandy|roverERROR: /home/centos/ci-config/ci-scripts/promote-images.sh failed with 3 exit code15:50
rlandy|rover^^ going to wrong server15:50
rlandy|rover2019-08-14 13:24:01,102 11875 ERROR    promoter ERROR: The promoted hash was not found: https://images.rdoproject.org/rhel8/master/rdo_trunk/5877cbf66372dba04d3a1b84016b83d4e06320bc_16fcf7dc15:51
*** bogdando has quit IRC15:54
*** tesseract has quit IRC15:56
rlandy|roverbiab lunch15:59
ykarel|awayohhk so rhel8 images are at 38.145.34.141/rcm-guest/images/rhel8/master/rdo_trunk16:02
ykarel|awaynot at images.rdoproject.org, sshnaidm right?16:02
rlandy|roverykarel|away: ack16:03
*** marios|out has quit IRC16:07
*** jpena is now known as jpena|off16:10
rlandy|roversshnaidm: did you kick the promoter?16:26
sshnaidmrlandy|rover, nope16:26
rlandy|roverhttp://38.145.34.55/redhat8_master.log16:26
rlandy|roverlooks like it's running now?16:26
rlandy|roverchecking server16:27
*** dsneddon has joined #oooq16:27
rlandy|roverentos/ci-config/ci-scripts/dlrnapi_promoter/config/CentOS-7/queens.ini16:28
rlandy|rovercentos   26088 26087  0 15:37 ?        00:00:00 python /home/centos/ci-config/ci-scripts/dlrnapi_promoter/dlrnapi_promoter.py /home/centos/ci-config/ci-scripts/dlrnapi_promoter/config/CentOS-7/queens.ini16:28
rlandy|rovercorrectly - queens is running16:29
raukadahrlandy|rover: I have added stnadalone and container build to the rhel-8 promotion criteria16:29
rlandy|roverraukadah: sshnaidm updated the image promoter script16:29
rlandy|roverto promote redhat images16:29
raukadahrlandy|rover: cool16:29
rlandy|roverI wanted to run the promoter for redhat8 manually to test16:30
rlandy|roverso right now  queens is running16:30
rlandy|roverok16:30
rlandy|roverbut the log looks like redhat8 is ALSO running16:30
rlandy|roverhttp://38.145.34.55/redhat8_master.log16:30
rlandy|roverthe ps -ef shows only queens process16:31
rlandy|roverfor promoter16:31
* rlandy|rover is confused16:31
rlandy|roverif I rekick redhat8, I don't want to disturb current upload16:31
rlandy|roverraukadah: maybe someone else kciked it manually or the log is stuck?16:31
rlandy|roverWed Aug 14 16:32:25 UTC 201916:32
rlandy|rover^^ time on promoter server16:32
*** dsneddon has quit IRC16:33
rlandy|roverchecking registry16:34
sshnaidmrlandy|rover, any dockers running there?16:34
sshnaidmrlandy|rover, let's wait a little..16:35
raukadahsshnaidm: above looks to be a container issue16:35
rlandy|roversshnaidm: ack https://console.registry.rdoproject.org/registry#/images/tripleomaster/rhel-binary-rabbitmq:5877cbf66372dba04d3a1b84016b83d4e06320bc_16fcf7dc16:35
rlandy|roverpushed a day ago16:35
sshnaidmrlandy|rover, that's fine16:35
sshnaidmrlandy|rover, well, really seems stuck.16:36
raukadahor may be something else16:36
rlandy|roverraukadah: maybe we should rekick?16:37
rlandy|roverthis si stuck16:39
raukadahrlandy|rover: nope, it is a rare ccase16:39
rlandy|roverthere is no active rhel8/redhat8 promoter process16:39
raukadahsorry I was taking about tempest failure16:39
rlandy|roverrare case :)16:39
rlandy|roverthis whole product is a rare case16:40
*** amoralej is now known as amoralej|off16:40
rlandy|rovernow it's promoting another hash  9450ef731a485c9dd5680c08d4d7ce849b35519016:40
rlandy|roverthat says two days ago16:41
rlandy|rovermaybe it's running through all the hashes it never tried to promote16:41
sshnaidmrlandy|rover, moving fast..16:41
rlandy|roversshnaidm: let's let it do it's thing for a while16:42
sshnaidmrlandy|rover, yeah, I think this is the logic of this script16:42
rlandy|rovercheck again in a bit16:42
rlandy|roverif it doesn't pick up your change, we can rekick then16:42
rlandy|roverthanks for that16:42
sshnaidmrlandy|rover, my change is on the server, so should pick it16:44
raukadahsshnaidm: https://github.com/openstack/tempest/blob/master/tempest/scenario/manager.py#L66616:44
raukadahsshnaidm: might be switching to subprocess.run fixes the issue16:45
raukadahwhich is a new thing in py3.516:45
sshnaidmraukadah, hmm..16:45
raukadahhttps://stackoverflow.com/questions/51886842/why-do-i-get-filenotfounderror-when-using-subprocess-popen16:45
sshnaidmraukadah, how is nobody hit this before16:45
raukadahsshnaidm: no idea, everything working in devstack does not works in tripleo16:46
sshnaidmraukadah, yeah, as usual :)16:46
raukadahhttps://github.com/home-assistant/home-assistant/issues/1230216:46
rlandy|rovermaybe move it off line 666 :)16:47
*** dsneddon has joined #oooq16:47
raukadahif we disable mtu stuff then might it will wor16:47
raukadah*work16:47
raukadahbut we need a bug for that16:50
sshnaidmraukadah, for samsung tv?16:50
raukadahsshnaidm: for above tempest failure16:50
rlandy|roverraukadah: and we're green ... http://rhos-release.virt.bos.redhat.com:3030/rhosp16:50
raukadahrlandy|rover: take a snap :-)16:51
sshnaidmimpressive!16:51
raukadahsshnaidm: I will prepare subprocess run fix tomorrow if it fixes the issue or not to check16:52
sshnaidmraukadah, ack, thanks16:52
sshnaidmraukadah, running it again today, let's see if it's reproducible16:53
*** dsneddon has quit IRC16:53
raukadahsure16:53
raukadahepel 8 is now available16:53
rlandy|roversent you the pic in email16:54
raukadahubi image + epel might solve few issues related to molecule16:54
raukadahrlandy|rover: awesome :-)16:54
*** kopecmartin is now known as kopecmartin|off16:54
sshnaidminteresting that there is no centos8 still, but epel is out16:55
rlandy|roverI know - I checked that16:55
rlandy|roverwhy epel w/o centos???16:55
sshnaidmrlandy|rover, for rhel users that don't pay? :)16:55
rlandy|roverwe shouldn't be encouraging that behavior16:56
raukadahsshnaidm: I am thinking of using rhel8 image for testing there16:58
raukadahin tempest16:58
raukadahall rhel no cirros16:59
sshnaidmraukadah, but it's huge relative to cirros16:59
sshnaidmraukadah, infra mirrors will fail on downloading it :D17:00
*** derekh has quit IRC17:00
raukadahsshnaidm: from rcm server also?17:00
sshnaidmraukadah, it's on rdo cloud.. even worth17:00
rlandy|roverhttps://review.opendev.org/#/c/676229/ looking better17:00
raukadahin ubi image, we donot have ping command also17:01
raukadahi have tried fedora cores on rdo-cloud17:01
raukadahit looks impressive17:01
raukadahi will share notes soon17:01
raukadahbut no python there as a feature17:01
sshnaidmrlandy|rover, if proxy servers don't update their delorean.repo we might have one from previous hash..17:02
rlandy|roversshnaidm: I think this is working https://review.opendev.org/#/c/676229/17:02
rlandy|roverbut we would have to update everywhere17:03
sshnaidmrlandy|rover, yeah, the problem can be when delorean.repo changes to new one17:03
sshnaidmrlandy|rover, I think that's why we didn't use proxies before in this step17:03
sshnaidmrlandy|rover, maybe it's fine.. but need to keep eye on that, in particular when promoting17:04
rlandy|roversshnaidm: I think we should merge the https://review.opendev.org/#/c/676214/17:04
rlandy|roverand see ifthe time helps at all17:04
rlandy|roverthen reconsider17:04
rlandy|roverI can update the dupliaction17:04
raukadahrlandy|rover: we also need to think about dlrn repo url hardcoded at multiple places17:04
raukadahwhich we were talking today17:04
sshnaidmraukadah, coreos looks nice, btw17:05
raukadahbelow is the steps17:05
sshnaidmrlandy|rover, ack, +2, not critical17:06
raukadah1. download image from here https://getfedora.org/en/coreos/ extract it to qcow217:06
raukadah2. https://coreos.com/os/docs/latest/booting-on-openstack.html upload it to glance17:06
raukadah3. download fcct binary from here https://github.com/coreos/fcct/releases and make it executbale17:07
raukadah4. generate ignition config https://docs.fedoraproject.org/en-US/fedora-coreos/getting-started/#_generating_ignition_configs17:07
raukadah5. https://coreos.com/os/docs/latest/booting-on-openstack.html - launch it17:08
raukadahand you are done17:08
sshnaidmraukadah++17:08
sshnaidmraukadah, need to play with it17:08
raukadahsshnaidm: thanks for my frind sinny kumari who works on that helping on that17:09
sshnaidmraukadah, but seems also pretty heavy, 424 MB17:09
raukadahhttps://sinny.io/2019/05/13/running-fedora-coreos-nightly-iso-and-qcow2-images-in-libvirt/17:09
zbr|fluit is a a hard-coreos17:09
raukadahthey are working on minimizing the size17:09
raukadahit has both docker as moby and podman17:10
sshnaidmraukadah, inside?17:10
raukadahyes17:10
sshnaidmthat explains its size..17:10
zbr|fluthat is cool, and useful.17:10
sshnaidmneed to add it to virtainers17:10
sshnaidmzbr|flu, don't we have molecule running on it yet?17:11
zbr|fluin fact molecule is tested on latest fedora, so i am ready to bet it will work.17:11
zbr|flunow depends what you want to test: using it as a test-platform for molecule, would be my guess, right?17:12
raukadahhttps://github.com/coreos/zincati is an interesting feature of coreos auto update of host17:12
zbr|fluin fact i was planning to start preparing major distros images prebuild with ansible support, so we can use them for testing easily.17:13
raukadahthere is no dnf there17:14
raukadahuse rpm-ostree17:14
zbr|fluinstalling python packages every time is a waste of time, i could reduce test execution to ⅓ by using prepared images instead of official ones. obviously that you would have to trust me that I do not put any bitcoin miner in them.17:14
sshnaidmis it fedora based or coreos based?17:14
sshnaidmI think "fedora" is just a brand name there17:14
raukadahcontainer linux replace packagwe manager with rpm-ostree17:15
sshnaidmzbr|flu, but usually you don't need python packages, maybe libselinux for rh based17:15
zbr|fluwithout python, you can only run "raw" commands with ansibe, pretty much useless.17:16
rlandy|roversshnaidm: adding followup patch with the parameters17:16
raukadahzbr|flu: use fedora coreos as a host use containers to do all kind of stuff17:17
raukadahmain aim is to keep host clean as much as possible17:17
sshnaidmzbr|flu, yeah, python is needed, but isn't it installed by default on every linux?17:17
zbr|flusshnaidm: nope, it is not. not even on rhel-8!17:17
sshnaidmzbr|flu, really?17:18
raukadahzbr|flu: https://review.opendev.org/#/c/676428/ you might be interested17:18
zbr|fluthanks17:20
zbr|flusshnaidm: yep. also there are lots of small bits that you will run into if you too not make an os ansible managable. for example: you forget to install rsync.17:22
zbr|fluthere is a very long list of of extra bits needed, and on top of that pile is the (in)famous libselinux :D17:22
zbr|flunow a practical question: what is the benefit of using fedora-coreos when I have two baremetals  (centos7 and fedora 30) which both have docker and podman installed?17:23
*** dsneddon has joined #oooq17:25
sshnaidmzbr|flu, using for what?17:25
zbr|fludevelopment, testing,....17:26
raukadahfedora-coreos baremetal support through pxe booting coming soon17:27
zbr|fluhappy to test, i already have a PXE environment setup, that how i provision my hosts (rhel/fedora/centos)17:27
raukadahall issues tracked here https://github.com/coreos/fedora-coreos-tracker17:28
sshnaidmI thought it's gonna be a "kubernetes distro", something light and quick17:28
sshnaidmbut >400 MB is not so light17:29
sshnaidmit replaces atomic afaik17:29
raukadahtrue!17:30
*** dsneddon has quit IRC17:30
zbr|flusshnaidm: what is the best way to approach sfci guys? i got a failure on their jobs at https://review.opendev.org/#/c/676464/17:33
sshnaidmzbr|flu, not sure, maybe #sf-ops ?17:34
zbr|fluwell posted 1h ago, no reply17:34
sshnaidmzbr|flu, it's on internal, on freenode it's #softwarefactory17:34
raukadahif sfci == softwarefactory there is a irc channel also #softwarefactory17:34
raukadahtristin will be available17:34
sshnaidmbut it's also doesn't look live17:34
sshnaidmAugust, nobody works :)17:34
raukadahzbr|flu: sshnaidm live pxe boot does not work for fedora coreos but pxe booting is supported17:53
*** dsneddon has joined #oooq18:09
*** rlandy|rover is now known as rlandy|rover|brb18:10
*** dsneddon has quit IRC18:14
*** rlandy|rover|brb is now known as rlandy|rover18:26
*** ykarel|away has quit IRC18:26
*** dsneddon has joined #oooq18:28
*** Vorrtex has quit IRC18:57
sshnaidmrlandy|rover, in your time please: https://review.opendev.org/#/c/676188/19:25
sshnaidmrlandy|rover, raukadah promoter SUCCESS promoting tripleo-ci-testing as current-tripleo19:29
sshnaidmfirst rhel8 promotion \o/ \o/19:30
raukadahsshnaidm: :-)19:45
raukadahit was a great team work :-)19:46
*** ankit08 has joined #oooq19:52
*** saneax has quit IRC20:07
rlandy|roverwoohoo finally20:23
rlandy|roversshnaidm: https://review.opendev.org/#/c/676188 - looks ok - w?20:24
rlandy|roverhttp://38.145.34.55/redhat8_master.log  finally stopped promoting20:25
rlandy|roversshnaidm: ^^20:25
rlandy|roverovb looks a lot better20:26
*** aakarsh has quit IRC20:27
sshnaidmrlandy|rover, yeah, not used to see it so green)20:28
rlandy|roverlol20:29
rlandy|rovermaybe the fewer servers is better20:29
rlandy|roveridk20:29
rlandy|roverluck of the draw maybe20:29
rlandy|roversshnaidm: I know it'd late for you but question about parameters in scripts/emit_releases_file/emit_releases_file.py and related tests20:30
*** saneax has joined #oooq20:31
rlandy|roveradding 2 parameters to the file is trivial20:31
rlandy|roverbut do they really need to be parameters20:31
rlandy|rovercould we just add them directly?20:31
rlandy|roverthe tests seem a but overdone20:32
rlandy|roverbit20:32
rlandy|roverwe can discuss tomorrow - if that works better20:32
sshnaidmrlandy|rover, let's talk tomorrow, marios also had some thoughts20:38
rlandy|roversshnaidm: ack - good night20:39
rlandy|roverand thanks for the help with image promotions20:40
*** vinaykns has quit IRC20:42
*** Vorrtex has joined #oooq20:45
*** rfolco has quit IRC20:51
*** jtomasek has quit IRC21:01
*** saneax has quit IRC21:10
*** Vorrtex has quit IRC21:31
*** rfolco has joined #oooq21:56
*** dsneddon has quit IRC21:56
*** rlandy|rover has quit IRC23:12
*** ankit08 has quit IRC23:18
*** ksambor has quit IRC23:23
*** ksambor has joined #oooq23:23
*** aakarsh has joined #oooq23:32
*** sshnaidm is now known as sshnaidm|afk23:38

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!