Wednesday, 2018-12-12

hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario010-multinode-oooq-container, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/560445, stable/ocata: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001 @ https://review.openstack.org/56429100:00
*** tosky has quit IRC00:01
*** dsneddon has quit IRC00:59
*** agopi has joined #oooq01:03
*** rlandy has quit IRC01:05
*** dsneddon has joined #oooq01:25
*** dsneddon has quit IRC01:30
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario010-multinode-oooq-container, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/560445, stable/ocata: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001 @ https://review.openstack.org/56429102:00
*** jrist has quit IRC02:25
*** ykarel|away has joined #oooq02:58
*** agopi has quit IRC02:59
*** ykarel|away has quit IRC03:05
*** agopi has joined #oooq03:06
*** apetrich has quit IRC03:15
*** dsneddon has joined #oooq03:26
*** dsneddon has quit IRC03:40
*** Tengu has quit IRC03:51
*** Tengu has joined #oooq03:58
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario010-multinode-oooq-container, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224,  (1 more message)04:00
*** ykarel|away has joined #oooq04:08
*** udesale has joined #oooq04:17
*** dsneddon has joined #oooq04:36
*** dsneddon has quit IRC04:41
*** ykarel|away has quit IRC05:09
*** brault has joined #oooq05:09
*** brault has quit IRC05:14
*** hubbot1 has quit IRC05:19
*** hubbot1 has joined #oooq05:19
*** ratailor has joined #oooq05:20
*** ykarel|away has joined #oooq05:25
*** ykarel|away is now known as ykarel05:26
*** sanjayu_ has joined #oooq05:51
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario010-multinode-oooq-container, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224,  (1 more message)06:01
*** dsneddon has joined #oooq06:31
*** dsneddon has quit IRC06:36
*** dsneddon has joined #oooq06:41
chandan_kumarquiquell|off: hello06:44
chandan_kumarquiquell|off: once this patch https://review.openstack.org/624411 merged feel free to recheck your patch06:45
*** quiquell|off is now known as quiquell07:13
quiquellchandan_kumar: o/ has recheck it, zuul is still failing07:13
chandan_kumarquiquell: there is mirror issue with packstack demo job07:14
*** matbu has quit IRC07:46
*** kopecmartin|off is now known as kopecmartin07:48
*** jfrancoa has joined #oooq07:50
*** matbu has joined #oooq07:51
*** vinaykns has joined #oooq07:55
*** apetrich has joined #oooq07:55
*** skramaja has joined #oooq07:57
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario010-multinode-oooq-container, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224,  (1 more message)08:01
quiquellpanda: Ping me when you are up08:02
*** jtomasek has joined #oooq08:05
*** amoralej|off is now known as amoralej08:12
*** sanjayu_ has quit IRC08:20
*** sanjayu_ has joined #oooq08:20
*** brault has joined #oooq08:23
*** brault has quit IRC08:24
*** brault has joined #oooq08:24
*** ccamacho has joined #oooq08:37
*** tosky has joined #oooq08:46
*** vinaykns has quit IRC09:16
*** dtantsur|afk is now known as dtantsur09:19
kopecmartinchandan_kumar, arxcruz when you're free, can one of you have a look on this please? https://tree.taiga.io/project/tripleo-ci-board/issue/51009:21
kopecmartinit won't take long09:21
*** holser_ has joined #oooq09:25
*** derekh has joined #oooq09:44
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario010-multinode-oooq-container, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224,  (1 more message)10:01
*** sshnaidm|afk is now known as sshnaidm10:05
*** gkadam has joined #oooq10:05
*** gkadam has quit IRC10:05
*** holser__ has joined #oooq10:13
*** holser__ has quit IRC10:14
*** holser_ has quit IRC10:14
*** holser_ has joined #oooq10:18
pandaquiquell: 'morning10:19
quiquellpanda: Have to leave now for a few minutes, if you have time, we can talk later on10:19
*** quiquell is now known as quiquell|brb10:19
pandaquiquell|brb: sure10:19
ssbarnea|rovermarios: got a bug in my du timeout patch, can you please recheck the new patch at https://review.openstack.org/#/c/624381/ ?10:26
ssbarnea|roverit appears that "-v" argument was not available on centos, too new.10:27
marios_|ruckssbarnea|rover: looking10:29
*** holser_ has quit IRC10:39
*** quiquell|brb is now known as quiquell10:40
*** dsneddon has quit IRC10:41
*** vinaykns has joined #oooq10:42
marios_|ruckssbarnea|rover: containers multinode seems to be in a bad state? all last 4 fails in gate10:43
marios_|ruck(grafana)10:43
*** holser_ has joined #oooq10:43
quiquellpanda: I am back do  you have a minute, just to get some context of fedora28 promotion pipeline10:47
ssbarnea|rovermarios_|ruck: i will have a look. i started to rebase failed CRs on the du one  as it seems that post_failure is quite common.10:48
ssbarnea|roverpanda: can this be abandoned now or not really? https://review.openstack.org/#/c/618780/10:49
marios_|ruckssbarnea|rover: added note on etherpad seems to be existing issues so far no new one will have a look again in a sec ther are a couple more hits for that job10:51
ssbarnea|rovermarios_|ruck: in this case is not docker bug , is podman bug. very confusing as docker has an almost identical bug itself.10:52
pandassbarnea|rover: I would like to change it a bit and make it permanent10:52
pandassbarnea|rover: It may be useful for the next time10:53
ssbarnea|roverpanda++10:53
hubbot1ssbarnea|rover: panda's karma is now 910:53
marios_|ruckssbarnea|rover: ack well i meant the one they said was 'internal they will fix' or sthing in the pull request10:53
marios_|ruckssbarnea|rover: change it if it isn't accurate in the ether10:53
pandassbarnea|rover: I'll remove the job configuration, I'll leave the jobs configured but inactive10:53
ssbarnea|roveri wanted to ask you the same, to make it generic "pre" release testing.10:53
pandaquiquell: sure10:53
pandaquiquell: ready10:53
pandaquiquell: where ?10:55
quiquellpanda: ack going to your blue10:55
*** dsneddon has joined #oooq10:55
quiquellpanda: https://bluejeans.com/789106523210:58
*** udesale has quit IRC11:02
*** udesale has joined #oooq11:03
chandan_kumarkopecmartin: verifying it11:16
*** rfolco has joined #oooq11:30
rfolcomarios, chandan_kumar: scen2 is not running tests in the whitelist11:33
marios_|ruckrfolco: ack (will check scen1/4 in bit)11:39
rfolcoI think this is more a tempest plugin problem11:42
rfolcoarxcruz, would you advise ? https://review.openstack.org/#/c/618709/ tempest plugin tests are not running in scen2 job11:44
marios_|ruckssbarnea|rover: added a new card for cix call later11:51
chandan_kumarrfolco: tempest.api.identity?11:51
chandan_kumarrfolco: as those tests are in global skip list that's why they are not runninbg11:52
marios_|ruckssbarnea|rover: for https://bugs.launchpad.net/tripleo/+bug/180797711:52
openstackLaunchpad bug 1807977 in tripleo " periodic-tripleo-centos-7-master-containers-build fails for mariadb:Error'd /bin/sh -c rm /etc/my.cnf.d/mariadb-server.cnf /etc/my.cnf.d/auth_gssapi.cnf HTTP Error 503 - Service Unavailable" [Undecided,Fix committed] - Assigned to Damien Ciabrini (dciabrin)11:52
rfolcochandan_kumar, barbican and telemetry11:53
rfolcohttps://review.openstack.org/#/c/618709/30/zuul.d/standalone-jobs.yaml11:53
rfolcohttp://logs.openstack.org/09/618709/30/check/tripleo-ci-centos-7-scenario002-standalone/cb5f284/logs/tempest.html.gz11:53
rfolcochandan_kumar, ^11:53
marios_|ruckssbarnea|rover: do yoy have pointer for the podman thing i added note in the etherpad remove with link please?11:54
chandan_kumarrfolco: chekcing11:54
ssbarnea|roverposted11:55
*** dtantsur is now known as dtantsur|brb11:56
arxcruzrfolco: checking11:59
chandan_kumarrfolco: encryption test is not in skip list12:00
chandan_kumarit shoudl run12:01
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario010-multinode-oooq-container, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224,  (1 more message)12:01
chandan_kumarrfolco: https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/validate-tempest/vars/tempest_skip_master.yml#L60 it is skipped there12:09
arxcruzrfolco: yeah, chandan_kumar is right, barbican tests are in the skip list12:09
chandan_kumarrfolco: need to send a patch on tqe side to eanble it12:10
arxcruzchandan_kumar: i believe we can remove it, since it is supported already in tripleo right ?12:10
chandan_kumararxcruz: on undercloud npt supported12:10
rfolcoarxcruz, chandan_kumar so... add barbican and remove the other from whitelist as it is in skip list ?12:27
* rfolco confused12:27
rfolcoregex, whitelist, skip list, global backlist, blacklist, black regex12:28
*** amoralej is now known as amoralej|lunch12:29
*** dsneddon has quit IRC12:30
rfolcochem, maybe we can antecipate this one? can you review please ? https://review.openstack.org/#/c/62303912:34
*** brault has quit IRC12:34
arxcruzrfolco: no, just add a depends on as per my comment it should work12:41
weshayssbarnea|rover, I'm happy to cover the mtg unless you want to do it12:51
*** vinaykns has quit IRC12:53
*** ratailor has quit IRC12:54
marios_|ruckssbarnea|rover: weshay call time13:01
ssbarnea|rovermarios_|ruck: joined, but I will let wes do the talk. listening it in the background.13:03
weshaymarios_|ruck, so tripleo-common should always be latest13:04
weshayit's tripleo13:04
* weshay looks at the job13:04
weshaymarios_|ruck, what I mean is .. patch in tripleo-common merges right.. 5 min later the promotion job kicks and should pick it up13:05
marios_|ruckweshay: yes but it has to be in latest built by rdo right? current/consistent13:06
marios_|ruckweshay: and when i looked newest build is from yesterday  so i don't know if there was some problem the fix we want merged last night13:06
*** trown|outtypewww is now known as trown13:07
weshaychandan_kumar, arxcruz we have 7 untriaged tempest bugs, sorry pita here13:08
arxcruzweshay: where ?13:08
weshayarxcruz, see the bug triage doc in the tempest google folder13:08
weshayarxcruz, bugzilla13:09
weshaymarios_|ruck, the job just kicked after it was merged13:10
weshaymarios_|ruck,13:11
weshay| * 092cddf2 - overrides for transitioning from mariadb 10.1 to 10.3 (2 days ago) <Damien Ciabrini>13:11
weshay* |   b5f6770b - Merge "Option to upload Octavia amphora img in RAW format" (24 hours ago) <Zuul>13:11
weshay|\ \13:11
weshayright.. we're looking for 092cddf213:11
marios_|ruckweshay: thanks where are you tracking?13:11
weshaymarios_|ruck, as soon as I get off this call I'll ping you because I want to run this by you as well13:12
weshay:)13:12
weshaygit log13:12
arxcruzweshay:  Can't use cf_internal_whiteboard as a field name.13:12
arxcruzi'm getting this13:12
marios_|ruckweshay: no i mean that the build for tripleo common kicked13:13
arxcruzweshay: nevermind13:13
weshayDec 12 11:41:32 Installed: openstack-tripleo-common-containers-10.1.1-0.20181211185305.97987c3.el7.noarch13:14
weshayhttps://logs.rdoproject.org/53/611653/3/openstack-check/tripleo-ci-centos-7-multinode-1ctlr-featureset010/4183cc4/logs/undercloud/var/log/yum.log.txt.gz13:14
weshaymarios_|ruck, for example13:14
weshaymarios_|ruck, 2018-12-11 18:51:402018-12-11 18:50:14openstack-tripleo-common13:15
weshayhttps://trunk.rdoproject.org/centos7-master/report.html13:16
weshaymarios_|ruck, see that?13:16
marios_|ruckweshay: yeah i added a comment on the card with that one13:16
marios_|ruckweshay: https://trello.com/c/O03boEBw/843-cixlp1807977tripleo-ciproamasterpromotion-blockerperiodic-tripleo-centos-7-master-containers-build-fails-kolla-build-mariadb-iss#comment-5c10fb42896c48894326988913:16
weshaymarios_|ruck, next promotion jobs should pick it up13:16
marios_|ruckack weshay13:17
weshaymarios_|ruck, re: pike the packages are not sane yet13:18
weshayci is good13:18
marios_|ruckweshay: cool was just wondering about the z4 /eol and if they need promote for puddles etc13:18
weshaymarios_|ruck, ya.. they have not imported in 191 days13:18
*** dtantsur|brb is now known as dtantsur13:20
weshaymarios_|ruck, so riddle me this batman.. https://github.com/openstack/tripleo-common/commits/master13:21
weshayMerge "overrides for transitioning from mariadb 10.1 to 10.3"13:21
weshayoverrides for transitioning from mariadb 10.1 to 10.3  …13:21
weshaymarios_|ruck, I forget why one sees that twice13:22
marios_|ruckweshay: wot why?13:22
marios_|ruckweshay: i mean it merged right?13:22
weshaywell one says it's commited, one says merge13:22
weshaywtf13:22
weshayI never understand that13:22
weshayssbarnea|rover, you are gerrit man.. do you know why that happens?13:23
*** skramaja has quit IRC13:23
sshnaidmweshay,  merged if there were commits between original commit and it merging time13:26
sshnaidmweshay, committed - it's merged to repo right after submitted w/o any commits in this time13:26
ssbarnea|roverweshay: sshnaidm is right: changes that were not on top of master and that were auto-rebased by zuul endup as "Merges", the other do get in directly.13:27
ssbarnea|roverthis can be changed in gerrit config, is called merge strategy or something like this.13:27
ssbarnea|roverit can be configured per project13:27
ssbarnea|roveri personally do not like the Merge prefix as it makes harder to read the git history.13:28
weshayOH13:28
weshaythe auto rebase13:28
weshaysshnaidm++13:28
weshayssbarnea|rover++13:28
hubbot1weshay: sshnaidm's karma is now 913:28
weshaythanks13:28
hubbot1weshay: ssbarnea|rover's karma is now 113:28
sshnaidmquiquell, hi, can you please give context about https://review.openstack.org/#/c/621517/ ?13:30
*** dsneddon has joined #oooq13:33
*** rlandy has joined #oooq13:33
rlandypanda: hello13:34
rlandypanda: need to chat with you for a few about the naming13:34
rlandywe decided to ditch it and just change the promotion criteria13:34
*** dsneddon has quit IRC13:37
weshaykopecmartin, chandan_kumar arxcruz there probably isn't a great time to have a sprint end / wrap up mtg this go13:44
weshayunless we do it today13:44
weshayI'm ok to work it just through taiga13:44
quiquellsshnaidm: standalone containers were not updated with delrean or gating repo RPMs13:44
weshayunless you guys feel like we should have a face to face mtg13:44
weshayquiquell, right13:44
quiquellsshnaidm: https://bugs.launchpad.net/tripleo/+bug/180596813:44
openstackLaunchpad bug 1805968 in tripleo "standalone job does not run yum update during the container image prepare" [Critical,In progress] - Assigned to Quique Llorente (quiquell)13:44
weshaythere is a bug there13:44
quiquellsshnaidm, weshay: In the review there is a Closes-Bug thingy13:45
quiquelltht changes are already merged13:45
sshnaidmquiquell, so this one is not ready for review?13:46
quiquellsshnaidm: Not yet, let me make it work first, I will move the taiga task to "in progress"13:46
*** amoralej|lunch is now known as amoralej13:46
quiquellAhh is at "in progress"13:46
sshnaidmquiquell, ack13:48
kopecmartinweshay, I'm fine with having it through taiga13:49
weshaychandan_kumar, kopecmartin why are your os_ansible tempest tasks in sprint issues?13:49
weshaywhy are they not a task from a user story?13:49
kopecmartinweshay, hmm, which one for example?13:50
weshaykopecmartin, ur's looks ok.. ci for openstack-tempest downstream13:50
weshaykopecmartin, chandan_kumar you guys have no work on the sprint outside of tasks13:51
weshaythis is odd13:51
* weshay looks at our epic / user stoires13:51
weshaystories13:51
weshaykopecmartin, chandan_kumar arxcruz /me looking into it13:53
weshayI think it may be a board thang13:53
*** dtantsur is now known as dtantsur|brb13:56
weshaykopecmartin, do you want doc of tempest updates on sprint 2?13:59
weshayare you working that?13:59
kopecmartinweshay, yes, the task number 489 Create basic doc structure14:00
kopecmartinit's done, waiting for review14:00
kopecmartinsorry, i forgot to pin it to the sprint14:00
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario010-multinode-oooq-container, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/560445, stable/ocata: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001 @ https://review.openstack.org/56429114:01
kopecmartinweshay, arxcruz, chandan_kumar btw , if we are working on the os_tempest role, we should get some powers to merge reviews, shouldn't we14:01
kopecmartin?14:01
kopecmartinIt can't take 5 days to get review on a simple doc change14:02
arxcruzkopecmartin: yup, it's a matter of reviews and show stuff there14:02
arxcruz:)14:02
weshaykopecmartin, arxcruz chandan_kumar now the board has the user stories from the epic where defined the mvp14:02
weshayand now I see your progress14:02
weshayhttps://tree.taiga.io/project/tripleo-ci-board/epic/41614:02
weshayepic ^14:02
*** holser_ is now known as holser|wanna_eat14:04
weshaykopecmartin, make sure if you pick up a task, to assign it to yourself :)14:05
weshayhttps://tree.taiga.io/project/tripleo-ci-board/task/489?kanban-status=144727514:05
* weshay updated14:05
weshayno worries14:05
kopecmartinweshay, sure, thanks14:05
quiquellpanda: for fedora28 make sense to updated the centos containers with changes ?14:08
quiquellpanda: I mean prep containers thing14:08
quiquellpanda: I am fixing it for centos14:09
pandaquiquell: is it some fix we can use with fedora28 containers in the future ?14:10
pandarlandy: ditched the renaming ?14:11
quiquellpanda: well making sure that tripleo-container-prep script works with python3 is good thing14:11
pandarlandy: or just postponed to a future sprint ? we still have to solve the 1:1 featureset mapping problem14:11
ykarelweshay, re. the tripleo-common patch(mariadb container-build fix) is not in consistent repo yet as tehre are some FTBFS:-  https://review.rdoproject.org/r/#/q/topic:rdo-FTBFS+status:open, so until cloudkitty, keystone build are fixed the tht patch will not be available for next promotion, just in case u missed14:11
quiquellpanda: about updating future fedora2x containers who knows what we will need14:11
ykarelmarios_|ruck, ^^ FYI14:12
pandaquiquell: I don't think we'll have to support versions other than 2814:12
pandaquiquell: but if this is a fix for python3, then we need it , yes14:12
weshayykarel, I think we pick it up from current14:12
ykarelweshay, nope, consistent14:12
weshayykarel, if it's tripleo we pick it up14:13
ykarelweshay, current is used in upstream ci for tripleo packages14:13
weshayright14:13
ykarelweshay, current is not used in promotion jobs14:13
weshayykarel, is there a reason why it should not be?14:13
weshayyou think? serious question14:13
ykarelweshay, afaik we promote repos, not packages14:14
ykarelso consistent repo is promoted to current-tripleo14:14
ykarelnot tripleo packages from current14:14
marios_|ruckykarel: thanks weshay .. ykarel i noticed fails in the master report from keystone and others (stopped scrolling down)14:14
marios_|ruckykarel: weshay since they all have to be green for consistent right?14:14
marios_|ruckykarel: so do we know why the keystone fails for example14:14
marios_|ruckweshay: ykarel ah you already linked thanks ykarel14:15
ykarelmarios_|ruck, u can look at for current failures which are blocking consistent promotion here: https://trunk.rdoproject.org/centos7-master/status_report.html14:15
ykareland u can check logs there14:15
marios_|ruckykarel: ack thanks yes that is what i meant by master report 16:14 < marios_|ruck> ykarel: thanks weshay .. ykarel i noticed fails in the master report from keystone and others (stopped scrolling14:15
ykareland in gerrit topic:rdo-FTBFS14:15
marios_|ruckykarel: i was looking for https://trello.com/c/O03boEBw/843-cixlp1807977tripleo-ciproamasterpromotion-blockerperiodic-tripleo-centos-7-master-containers-build-fails-kolla-build-mariadb-iss14:16
marios_|ruckykarel: bookmarking that topic didn't know of that thanks14:16
weshayykarel, marios_|ruck sec in 1-114:16
marios_|ruckssbarnea|rover: ^^14:16
marios_|ruckweshay: np ykarel pointing to some things that need to merge, before we can get a consistent build, which will include the tripleo-common fix https://review.rdoproject.org/r/#/q/topic:rdo-FTBFS+status:open14:17
rlandypanda: we solved that by just using names14:18
rlandypanda: the entire job rename to include arches can be postponed14:18
rlandypanda: so you want to chat about it?14:18
marios_|ruckykarel: quoted you on that trello card fyi ;)14:21
ykarelack14:23
rlandyquiquell: sshnaidm: weshay: pls review https://docs.google.com/document/d/1i1-_L2mx8_erETAVblchLnymC6-9h7A2x02begy99UQ/edit#14:26
rlandydoc for zuul-based reproducer workflows14:27
rlandyI have just put the basic workflow in there atm14:27
rlandywith headings for other sections like libvirt and OVB since we are not firmed up on those quite yet14:27
quiquellrlandy: thanks!14:28
rlandypanda: I included the changes needed to log the name in dlrn14:28
rlandypanda: ping me when you have time to go over it14:28
marios_|ruckcistatus.tripleo.org down for everyone or just me .com14:29
marios_|ruckhm just slow14:29
rlandyquiquell: sshnaidm: weshay: the info is nothing new - off the etherpad - just slightly more consumable14:29
rlandyfor meeting today14:29
weshayykarel, marios_|ruck so what's strange there is that the check jobs get the package14:30
quiquellrlandy: We are going to remove pipenv and user host stuff we don't have too much of dependencies I think14:30
sshnaidmrlandy, need to figure out what is with images.. before everybody starts upload the same images to their tenants :)14:30
weshaybut the promotion jobs do not14:30
weshaywhich is wrong14:30
rlandysshnaidm: absolutely14:31
rlandysshnaidm: I am not saying anyone should follow this yet14:31
rlandysshnaidm: just that weshay wanted us to start a doc  - so this is the start14:31
sshnaidmrlandy, ack14:32
rlandysshnaidm: feel free to comment strongly there14:32
weshaymarios_|ruck, ykarel the promotion jobs should have https://logs.rdoproject.org/53/611653/3/openstack-check/tripleo-ci-centos-7-multinode-1ctlr-featureset010/4183cc4/logs/undercloud/etc/yum.repos.d/delorean-current.repo.txt.gz14:32
pandarlandy: sure14:32
weshaythe delorean-current repo14:32
rlandyquiquell: ^^ as well - comment away14:32
weshayI thought they used to14:32
weshaymarios_|ruck, ykarel you guys have a minute?14:32
rlandyhappy to fix anything and here we have a trace of it14:32
weshaymarios_|ruck, ykarel https://bluejeans.com/u/whayutin/14:32
ykareljoining14:33
rlandysshnaidm: quiquell: also - feel free to edit the doc directly and skip the comments if that makes more sense14:33
rlandypanda: now?14:33
marios_|ruckweshay: sure sec grabbing headphones14:33
pandarlandy: ok14:33
rlandypanda: your bj?14:35
*** irclogbot_0 has quit IRC14:36
*** jrist has joined #oooq14:38
*** ykarel has quit IRC14:47
weshayhttps://github.com/openstack/tripleo-quickstart/blob/master/config/release/tripleo-ci/CentOS-7/promotion-testing-hash-master.yml14:48
pandarlandy: do not modify the dlrnapi_report.sh, I will pass the new name through TOCI_JOBTYPE, otherwise this will be another point of syncrhonization we need between the patches.14:51
*** irclogbot_0 has joined #oooq14:51
*** dciabrin has joined #oooq14:52
rlandypanda: do not change it at all - or do not change from what I have currently in the patch?14:53
pandarlandy: you're replacing TOCI_JOBTYPE with JOB_NAME, if you do that you can't merge your patch until I merge my patch. The dlrn_report.sh will still use TOCI_JOBTYPE, but I will pass the full name instead. When everything is in place we can eventually modify all the script to stop using TOCI_JOBTYPE14:54
*** ykarel has joined #oooq14:58
pandarlandy: https://review.rdoproject.org/r/1774415:04
rlandypanda: wrt dlrn-promoter.sh - still ok with adding both sets of lines?15:09
pandarlandy: I think it's ok. Until we merge my patch, the old names will still promote, the new name will do nothing because to success is reported with those names.15:12
pandaWhen we merge my patch, we'll promote with the old names, and the new lines should be idempotent and not promote anything again15:13
pandaAnd in the meantime, we gather testing informations on the new names15:14
pandamaybe we will pollute the logs a bit15:14
pandabecause we are running two master promotion commands in the same row every 10 minutes15:15
*** dsneddon has joined #oooq15:15
rlandypanda: https://review.rdoproject.org/r/#/c/17780/15:18
*** dsneddon has quit IRC15:19
rlandypanda: also - these two commits are ready for review and can merge:15:20
rlandyreview.rdoproject.org/r/#/c/17718 Add Fedora 28 periodic job for master15:20
rlandyreview.rdoproject.org/r/#/c/17719 Add Fedora-28 standalone job to the master promotion pipeline15:20
*** jrist has quit IRC15:21
*** holser|wanna_eat is now known as holser_15:22
*** dtantsur|brb is now known as dtantsur15:25
pandarlandy: question for those, but I can offer only my humble +1 there.15:26
pandarlandy: commented15:26
marios_|ruckssbarnea|rover: cix call15:30
* rlandy looks15:35
rlandypanda: you have a good question "question for all the other reviewers, do we want to create a hirerarchy instead of a suffix ? I have no strong preference."15:35
rlandyidk15:36
rlandylike centos15:36
rlandysubdir 7 subdir 815:36
rlandyweshay: want to weigh in?15:36
rlandyre: logs for promoter15:36
rlandyhttps://review.rdoproject.org/r/#/c/17780/5/ci-scripts/dlrnapi_promoter/config/CentOS-7/master.ini15:36
*** vinaykns has joined #oooq15:38
rlandyquiquell: sshnaidm; do you guys have +2/merge rights here ... review.rdoproject.org/r/#/c/17718 review.rdoproject.org/r/#/c/1771915:45
sshnaidmrlandy, yeah, will review15:45
rlandysshnaidm: hmm panda has a comment there15:46
rlandynot sure if we want to proceed15:46
rlandyactually I think we do - will reply15:46
sshnaidmrlandy, I don't understand why f28 job should use centos containers..15:48
quiquellsshnaidm: We don't have f28 yet15:49
quiquellsshnaidm: kolla does not support or deliver it, I think Alex is forking kolla just for that15:50
sshnaidmquiquell, so we'll change it to run f28 containers when it will be, we don't need 2 jobs15:50
quiquellsshnaidm: Yep, but still the plan is to use same DLRN hash, and that's tricky15:50
quiquellsshnaidm: not always the centos DLRN hash is at fedora DLRN15:50
quiquellsshnaidm: so some promotion are skipped until we find a 1-1 matching15:50
sshnaidmquiquell, I mean that in one moment will one job that run either f28 or c7 containers, but not both jobs in parallel15:51
sshnaidmquiquell, why is it not always? packages are not built in the same way?15:51
quiquellsshnaidm: well that I don't know, depends at what moment in python3 adaptation we are15:51
quiquellsshnaidm: Or how long do we want to support centos7 upstream ? don't know15:52
sshnaidmquiquell, it could be not the same when f28 package is not built, because of py3 deps for example.. but I don't think it's the case now15:53
pandathere will be a period in which we are still using the f28/centos job in centos pipeline, and we are testing the pure f28 job still15:53
pandahopefully will be  a short amount of time, but the pure f28 job will not work the moment we put it into configuration15:54
sshnaidmwhy do we need f28 job with c7 containers?15:54
quiquellsshnaidm: To fix f28 stuff not related to containers15:54
quiquellsshnaidm: We have already fix a lot of stuff so it's helping15:54
*** jrist has joined #oooq15:55
pandasshnaidm: this is what we are currently doing, f28 host/undercloud with the only available containers we have  : c715:55
pandaso we can move forward with f28 without having to wait for the containers, that nobody knows at this point when they will be available15:55
sshnaidmI'm aware of that, I'm asking why we need it when we'll have f28 containers15:55
pandait could take even another couple of months15:55
quiquellsshnaidm: we don't have f28 containers15:56
quiquellsshnaidm: or you mean at the moment we have them ?15:56
pandasshnaidm: define "when" we'll have f28 containers15:56
quiquellsshnaidm: At the moment we have them they are not going to be stable for sure15:56
sshnaidmquiquell, yeah, in the moment we'll have them15:56
pandasshnaidm: because I see a period in which we'll still have the mixed f28/c7 job, while we are testing the f28 pure job with f28 containers15:56
quiquellsshnaidm: A lot of false positive/negative will raise if we do the switch15:56
sshnaidmI don't see how mixed job will help there15:57
quiquellsshnaidm: They test the part that is already fixed af f28, so we know it's not broken, they are passing15:58
quiquellsshnaidm: at a pure f28 stuff you don't know if it's inside or outside the container15:58
quiquellsshnaidm: the issue15:58
quiquellsshnaidm: so until the containers are stable is not a bad thing to have still mixed, that's all out of my mind15:59
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario010-multinode-oooq-container, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/560445, stable/ocata: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001 @ https://review.openstack.org/56429116:01
*** sanjayu_ has quit IRC16:03
*** kopecmartin is now known as kopecmartin|off16:05
*** ykarel is now known as ykarel|away16:10
*** ykarel|away has quit IRC16:19
*** udesale has quit IRC16:19
*** rfolco is now known as rfolco_doctor16:28
*** jfrancoa has quit IRC16:46
*** ykarel has joined #oooq17:03
*** vinaykns has quit IRC17:04
*** quiquell is now known as quiquell|off17:06
*** dsneddon has joined #oooq17:16
*** jjoyce has quit IRC17:24
*** jjoyce has joined #oooq17:26
*** dtantsur is now known as dtantsur|afk17:26
*** vinaykns has joined #oooq17:29
*** dsneddon has quit IRC17:29
*** dsneddon has joined #oooq17:33
*** trown is now known as trown|lunch17:41
*** panda is now known as panda|off17:43
*** vinaykns has quit IRC17:49
weshayrlandy, want to 1-1?17:57
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario010-multinode-oooq-container, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/560445, stable/ocata: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001 @ https://review.openstack.org/56429118:01
rlandyweshay,: ack in your bj18:03
*** derekh has quit IRC18:03
weshayah sec otp18:04
weshayrlandy, ok joining18:07
*** holser_ is now known as holser|eod18:08
*** rfolco_doctor is now known as rfolco18:12
*** jrist has quit IRC18:33
*** trown|lunch is now known as trown18:50
*** holser|eod has quit IRC18:54
rfolcorlandy, panda|off weshay sshnaidm would you review please ? https://review.rdoproject.org/r/#/q/status:open+branch:master+topic:f28_promotion19:07
*** amoralej is now known as amoralej|off19:08
*** ykarel has quit IRC19:10
rfolcosshnaidm, this one too pls https://review.openstack.org/#/c/619337/19:14
sshnaidmrfolco, is it related to https://review.rdoproject.org/r/#/q/topic:f-28-pipeline ?19:15
rfolcono, it got a +w from alex but failed to merge19:15
rfolcothen several rounds of rechecks and got a +1 from zuul again19:15
sshnaidmrfolco, I mean previous topic19:20
sshnaidm https://review.rdoproject.org/r/#/q/status:open+branch:master+topic:f28_promotion19:20
weshayrlandy, can you paste me your google doc re: reproducer19:21
sshnaidmrlandy, any objections to merge this? https://review.rdoproject.org/r/#/c/17552/19:21
rlandymx8_erETAVblchLnymC6-9h7A2x02begy99UQ/edit#19:21
rfolcosshnaidm, right, last patch I asked for review is not related to f2819:21
rlandysshnaidm: go ahead19:21
rlandybetter than carrying around this huge review19:21
sshnaidmrlandy, do you mean "--poll" argument for deploy.py or "stack show"? https://review.rdoproject.org/r/#/c/17404/8/roles/ovb-manage/tasks/ovb-create-stack.yml19:23
rlandyfir deploy19:23
rlandyfor19:24
rlandysshnaidm: just a suggestion - not required19:30
rlandythat arg was not available when we first wrote the ovb role19:30
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario010-multinode-oooq-container, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/560445, stable/ocata: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001 @ https://review.openstack.org/56429120:01
*** vkmc has quit IRC20:03
rlandyweshay: do you have opinion on whether we need two diff job definitions https://review.rdoproject.org/r/#/c/17718/1/zuul.d/standalone-jobs.yaml?20:06
*** vkmc has joined #oooq20:07
* weshay looks20:08
weshaysshnaidm, the f28 job is bm f28 only, there are only centos7 containers atm20:09
weshayrlandy, /me checks something20:10
weshaysec20:10
weshayrlandy, panda|off the containers used for any job would be controlled by the release config https://github.com/openstack/tripleo-quickstart/blob/master/config/release/tripleo-ci/CentOS-7/master.yml#L2020:11
weshayhttps://github.com/openstack/tripleo-quickstart/blob/master/config/release/tripleo-ci/Fedora-28/master.yml#L2120:11
rlandyweshay: right so the job definition is the same - and stays as is20:11
weshayrlandy, actually no20:11
weshayrlandy, unless you can override the docker_registry_namespace20:12
weshayin zuul20:12
weshayso I think you need two jobs20:12
*** holser_ has joined #oooq20:12
weshaygoing to have two f28 jobs, one that runs w/ tripleomaster and one that runs w/ tripleomasterfedora20:13
weshayor something like that20:13
rlandydocker_registry_namespace override - would be a featureset_override?20:14
rlandyrfolco is not going to be happy of we expand that20:14
weshayrlandy, it could be fine if fs config is after release20:14
weshayya.. it's best to use a throw away fs config20:14
rfolcorlandy, your quota exceeded for overriding fs !!20:15
weshaylolz20:15
rfolcofinal warning20:15
rfolco[angry_face.gif]20:15
rlandyrfolco: just add that to the list of my other infractions20:16
rfolcoyeah I bought a white board here, I am counting peoples infractions20:16
* rlandy is juts thinking if that can be regular var20:16
rlandyor if it is required to be a featureset_override20:17
rfolcowhich var?20:17
rfolcoif it is in a fs file, makes sense, no?20:17
rlandyit's actually in release20:17
weshaytwo jobs, two fs files20:17
weshayrlandy,20:17
weshaythat's the easiest path20:17
weshayas long as the fs config is read in after release20:18
weshaywhich I'm not sure of20:18
weshaywish me luck20:18
weshayreproducer running20:18
rlandyweshay: a mixed release file makes the most sense in this case, no?20:18
panda|offdid I read override ?20:20
panda|off /nick -all panda|permanentlyoff20:20
weshayrlandy, it's derived via the os_dist from ansible20:20
rfolcololz panda wakes up from a nightmare hearing the word override20:21
rlandypanda: sorry I woke you20:21
* weshay didn't know rfolco and panda|off were such drama queens :)20:22
weshaywhat ever it is, it's temp O A R Y20:22
rfolcoaham20:22
rlandyoverride - is like saying voldermort20:22
weshaytemp O RARY20:22
weshaythat is20:22
weshayapparently20:23
* weshay fights back w/ BEATLEJUICE20:23
weshayBEATLEJUICE20:23
weshayBEATLEJUICE20:23
weshayBEATLEJUICE20:23
rfolco:)20:23
rlandyfine, ok, whatever, no override20:23
rlandyI'm sorry20:23
weshayI'm fine w/ it as long as it's temporary20:23
rfolcorlandy, you have to be careful with what you say20:23
weshaybut you can't have two diff releases20:23
rfolconeedless20:23
weshayrlandy, paste me your doc for the repro20:24
weshaywhen you have a sec after battling rfolco and panda|off20:24
weshay:)20:24
rlandyweshay: sorry - I thought I did ... https://docs.google.com/document/d/1i1-_L2mx8_erETAVblchLnymC6-9h7A2x02begy99UQ/edit#20:24
rlandythat's only the regular workflow,20:24
weshaythanks20:24
rlandyif you want dlrn and libvirt, I didn't move those to the doc yet20:24
rlandyin case we shared it with sf team, I didn't want to confuse the issue20:25
rlandythat inf is in the etherpad20:25
weshayok.. says20:29
weshaywaiting on at least one connection20:29
rlandydid you manage to install and set up???20:29
rfolcorlandy, is this before our discussion and new agreements of f28 pipeline ? https://review.rdoproject.org/r/#/c/17718/20:29
weshaymaybe20:29
weshaystill running20:29
weshaymy containers have not gone down20:29
weshayso that's good20:29
rfolcorlandy, this looks the same as mine20:29
rlandyweshay: oh ... neis gadol ...20:29
weshayha ya shem!!20:30
*** jrist has joined #oooq20:30
weshaysham? or poe?20:30
weshayPOE20:30
rlandywell in your case, poe - you are there20:30
weshaylolz20:30
rlandyfrom my perspective, sham20:30
* weshay goes and spins a draidel for good luck 20:30
weshayah shhot20:31
rlandyrfolco: https://review.rdoproject.org/r/#/c/17718/ is for the fixed pipeline20:31
weshay3 min and 7/8 are exited20:31
rfolcorlandy, f28 pure pipeline you mean?20:32
rlandyrfolco: I ma actually going to change that20:32
* rlandy just thinks about how best to do this20:33
rfolcorlandy, I am adding the f28 pure jobs on https://review.rdoproject.org/r/#/q/status:open++branch:master+topic:f28_promotion20:34
rlandyhttps://review.rdoproject.org/r/#/c/17718/1/zuul.d/standalone-jobs.yaml should be the mixed job20:37
* rlandy edits20:37
rlandyok people let's compare ...20:43
rlandyhttps://github.com/openstack/tripleo-quickstart/blob/master/config/release/tripleo-ci/Fedora-28/master.yml20:43
rlandyand20:43
rlandyhttps://github.com/openstack/tripleo-quickstart/blob/master/config/release/tripleo-ci/CentOS-7/promotion-testing-hash-master.yml20:43
rlandydocker_registry_host: trunk.registry.rdoproject.org20:44
rlandydocker_registry_namespace: tripleomaster20:44
weshaythey are both tripleomaster20:47
weshaywe'll need a net new namespace for fedora containers20:47
weshaytripleomaster assumes centos720:47
rlandyack20:49
* rlandy just commented on https://review.rdoproject.org/r/#/c/1773920:49
rlandyI think that needs a depends-on; a new release file20:50
rlandypromotion-testing-hash-master.yml20:50
rlandyin the fedora space20:50
rlandyrfolco: ^^ agree?20:51
weshayno luck w/ the reproducer :(20:51
rlandy:( ... what now?20:54
rlandyweshay: can I help?20:56
weshaymaybe.. I think it's having trouble connecting to docker20:57
weshayso we're missing something in the startup20:57
weshaysetup20:57
rlandywhat's your os - f28?20:58
weshayya.. fresh20:58
weshayoh..20:58
weshaylet me update20:58
weshay:)20:58
rlandyhmmm... I used docker-ce with f2720:58
rlandybut f28 should be good enough20:58
weshayya20:58
rlandymaybe rhel had good reason to kick docker out20:59
weshayrunning an update now20:59
weshay677 packages20:59
weshayso.. could be20:59
rlandyso ... back to docker_registry_namespace ... can we agree on a convention?20:59
rlandydocker_registry_host: trunk.registry.rdoproject.org20:59
rlandydocker_registry_namespace: tripleomaster20:59
rlandyand21:00
rlandydocker_registry_namespace: tripleomaster-fedora2821:00
rlandyall in favor???21:00
weshayrlandy, I'm not sure about hyphens21:01
rlandydocker_registry_namespace: tripleomasterF2821:01
rlandydocker_registry_namespace: tripleomasterFedora2821:01
weshaytripleomasterF28 tripleomasterCent8 tripleomasterRHEL821:02
weshay?21:02
rlandytripleomasterCentos821:02
weshaygoing from only one supported distro to 3+ is annoying21:03
weshay:)21:03
rlandyand we leave the default21:03
weshayya21:03
rlandyalso we need new release files21:03
weshaywe have them21:03
weshayF28?21:03
rlandyreally>21:03
rlandylook at the number of release files we have under https://github.com/openstack/tripleo-quickstart/tree/master/config/release/tripleo-ci/CentOS-721:04
weshayya21:04
rlandycompared to Fedora-2821:04
weshayright21:04
rlandythere is one master file21:04
weshaywe only support master21:04
rlandyno promotion stuff21:04
weshayright .. we need one more for promotion21:04
weshaynow21:04
rlandywe will need a promotion-testing-hash-master.yml21:04
weshayya21:05
weshayrlandy, 10 bucks says it's selinux21:16
rlandyha21:17
rlandycan you set permissive?21:17
weshaysure21:17
weshaydid, retrying21:17
weshayhrm.. I think I still have an issue w/ my ssh key though.. saw those errors fly by21:18
weshayrlandy, where do you want updates, to the etherpad or doc?21:18
rlandyweshay: I think the doc now21:20
rlandyalthough you can double update21:20
rlandyweshay: your ssh key? no passphrase21:37
rlandyright?21:37
weshayya..21:37
weshaybut there is still something off w/ it21:38
weshayit works w/ the upstream gerrit but not the local21:39
weshayreally odd21:39
weshayquique and I could not figure it out21:39
weshayand the fact I just recreated the issue on a fresh install on a diff laptop I think confirms it's the key21:39
weshayso downloading images etc.. creating new keys now21:39
weshayrfolco, can you bluejeans21:41
weshaylet's chat about these jobs21:41
weshayI think there is still confusion21:41
rlandyQUICKSTART_RELEASE="promotion-testing-hash-${QUICKSTART_RELEASE}"21:46
rfolcoweshay, ok21:46
rfolcoweshay, I am in21:47
rlandyweshay: rfolco: can I join - adding a F28 release file for promotion21:49
rfolcorlandy, please21:49
rfolcowes bj21:49
rfolcoweshay, https://review.rdoproject.org/r/#/c/1773821:51
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario010-multinode-oooq-container, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/560445, stable/ocata: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001 @ https://review.openstack.org/56429122:01
*** trown is now known as trown|outtypewww22:04
*** holser__ has joined #oooq22:07
*** holser_ has quit IRC22:09
*** holser__ has quit IRC22:10
*** jschluet has joined #oooq22:25
*** jschlueter has quit IRC22:29
*** gouthamr has quit IRC22:32
rfolcorlandy, weshay https://review.rdoproject.org/r/#/c/17738/4/ci-scripts/tripleo-upstream/build-containers-images.sh22:34
rfolcoin the containers build script I made it distro aware22:34
*** gouthamr has joined #oooq22:35
*** dtrainor has quit IRC23:09
rlandywhat is the default fedora mirror?23:10
*** ade_lee has quit IRC23:12
weshayrlandy, for promotion jobs23:14
weshay?23:14
weshayit's the rdo mirror23:14
weshayshould be the same as other jobs you find in centos23:14
rlandyweshay: yep _ I am trying to modify  release fil eto use all f2823:14
rlandy    # Otherwise, fallback to official mirrors provided by Fedora.23:15
rlandy    export NODEPOOL_FEDORA_MIRROR={{ lookup('env','NODEPOOL_FEDORA_MIRROR')|default('https://dl.fedoraproject.org/pub/fedora/linux/', true) }}23:15
rlandy    export NODEPOOL_BUILDLOGS_FEDORA_PROXY=///23:15
rlandy^^if there is an equivalent to BUILDLOGS for fedora23:15
weshayoh23:15
weshayhrm23:15
weshayhttps://dl.fedoraproject.org/pub/fedora/linux/23:16
weshayseems right23:16
rlandyI'm guessing on the rest23:20
rlandywill ask quiquell|off tomorrow - I think he did the master f28 release file23:21
*** rlandy is now known as rlandy|bbl23:28
*** dtrainor has joined #oooq23:31
*** weshay is now known as weshay_pto23:46

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!