Tuesday, 2019-03-19

hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039 @  (2 more messages)00:22
*** agopi__ has joined #oooq00:26
*** rascasoft has quit IRC00:31
*** rascasoft has joined #oooq00:33
*** holser_ has quit IRC00:36
*** rlandy has quit IRC00:46
*** rodrigods has joined #oooq01:42
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039 @  (2 more messages)02:22
*** apetrich has quit IRC03:14
*** dsneddon has quit IRC03:34
*** skramaja has joined #oooq03:38
*** raukadah is now known as chandankumar04:00
*** dsneddon has joined #oooq04:01
*** ykarel has joined #oooq04:05
*** dsneddon has quit IRC04:05
*** ykarel has quit IRC04:14
*** udesale has joined #oooq04:14
*** ykarel has joined #oooq04:17
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb- (3 more messages)04:22
*** dsneddon has joined #oooq04:39
*** dsneddon has quit IRC04:44
*** dsneddon has joined #oooq05:18
*** dsneddon has quit IRC05:23
*** udesale has quit IRC05:31
*** ykarel is now known as ykarel|afk05:32
*** saneax has joined #oooq05:46
*** ratailor has joined #oooq05:46
*** ykarel|afk is now known as ykarel05:55
*** dsneddon has joined #oooq05:57
*** dsneddon has quit IRC06:02
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb- (3 more messages)06:22
*** dsneddon has joined #oooq06:36
*** ccamacho has quit IRC06:37
*** dsneddon has quit IRC06:41
*** ykarel has quit IRC06:51
*** ykarel has joined #oooq06:51
*** jaosorior has quit IRC06:55
*** jaosorior has joined #oooq06:57
*** saneax has quit IRC07:06
*** saneax has joined #oooq07:07
*** dsneddon has joined #oooq07:13
*** dsneddon has quit IRC07:18
*** holser_ has joined #oooq07:19
*** tosky has joined #oooq07:25
*** apetrich has joined #oooq07:26
*** holser_ has quit IRC07:30
*** kopecmartin|off is now known as kopecmartin07:36
*** dsneddon has joined #oooq07:50
*** ksambor has quit IRC07:54
*** dsneddon has quit IRC07:55
*** jtomasek has joined #oooq08:01
*** udesale has joined #oooq08:09
*** ccamacho has joined #oooq08:14
*** dtantsur|afk is now known as dtantsur08:18
*** amoralej|off is now known as amoralej08:19
*** jfrancoa has joined #oooq08:19
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb- (3 more messages)08:23
*** dsneddon has joined #oooq08:23
*** holser_ has joined #oooq08:24
*** brault has joined #oooq08:24
*** dsneddon has quit IRC08:27
*** holser_ has quit IRC08:28
*** holser_ has joined #oooq08:42
arxcruzchandankumar: haha, sorry, you were faster than my comment, it was a backport, and I miss it... :)08:43
*** jpena|off is now known as jpena08:57
*** dsneddon has joined #oooq08:57
*** ykarel is now known as ykarel|lunch09:02
*** panda|pto is now known as panda09:04
*** dsneddon has quit IRC09:05
marios_|ruckme food biab09:12
*** jbadiapa has joined #oooq09:13
*** bogdando has joined #oooq09:18
*** holser_ has quit IRC09:20
chandankumarmarios_|ruck: \o/09:26
*** bogdando has quit IRC09:35
*** dsneddon has joined #oooq09:35
*** ykarel|lunch has quit IRC09:37
*** holser_ has joined #oooq09:37
*** bogdando has joined #oooq09:37
*** ykarel has joined #oooq09:37
*** dsneddon has quit IRC09:40
*** derekh has joined #oooq09:40
*** sshnaidm|afk is now known as sshnaidm|rover09:43
*** jbadiapa has quit IRC09:59
*** jbadiapa has joined #oooq10:12
*** dsneddon has joined #oooq10:12
*** dsneddon has quit IRC10:17
marios_|rucksup chandankumar10:17
chandankumarmarios_|ruck: one stupid in order to try new zuul workflow, do I need to use reproducer-quickstart.sh or reproducer-zuul-based-quickstart.sh ?10:19
arxcruzmarios_|ruck: gimme a +2, give me a +w give me a yay, code merged! https://review.openstack.org/#/c/643267/10:19
arxcruzchandankumar: you need the tarball and the reproducer-zuul-based-quickstart.sh10:19
marios_|ruckchandankumar: yeah the one with 'zuul' in the name don't know if the 'old' one even works anymore10:22
marios_|ruckarxcruz: will chekc10:22
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb- (3 more messages)10:23
arxcruzmarios_|ruck: chandankumar the old one works, except for fedora job10:23
marios_|ruckty arxcruz10:23
*** jbadiapa has quit IRC10:29
chandankumararxcruz: marios_|ruck thanks, trying that out10:32
*** jbadiapa has joined #oooq10:43
*** derekh has quit IRC10:45
*** derekh has joined #oooq10:48
*** dsneddon has joined #oooq10:52
*** udesale has quit IRC10:53
*** dsneddon has quit IRC10:58
zbrwf please https://review.openstack.org/#/c/640463/11:18
ykarelmarios_|ruck, sshnaidm|rover is the RDO phase1 master tripleo failure known?11:20
sshnaidm|roverykarel, tempest failure?11:21
marios_|ruckykarel: seems like there may still be an issue with tempest there? i saw note from weshay on https://etherpad.openstack.org/p/ruckroversprint711:21
marios_|ruckarxcruz: any update on that one? ^ bug 1819440 (note says wes gave you an env?_11:22
openstackbug 1819440 in tripleo "phase 1 failing on No such file or directory: '/home/stack/tempest'" [Critical,Triaged] https://launchpad.net/bugs/1819440 - Assigned to wes hayutin (weshayutin)11:22
marios_|ruckykarel: or are you seeing something else?11:22
arxcruzmarios_|ruck: wes didn't gave me the env yet11:22
sshnaidm|rovermarios_|ruck, well, it's not tempest11:22
arxcruzbut i'm trying to reproduce here11:22
arxcruzwith his info11:22
marios_|ruckthanks arxcruz sshnaidm|rover11:22
arxcruzright now is in TASK [overcloud-deploy : Deploy the overcloud]11:22
arxcruzfingers crossed11:22
ykarelmarios_|ruck, new issue11:23
ykarelnow it's failing before tempest11:23
sshnaidm|rovermarios_|ruck, "Error: unable to get cib"11:24
sshnaidm|rovermarios_|ruck, https://ci.centos.org/artifacts/rdo/jenkins-tripleo-quickstart-promote-master-current-tripleo-delorean-minimal-476/undercloud/home/stack/overcloud_deploy.log.gz11:24
sshnaidm|rovermarios_|ruck, it's not tempest11:24
marios_|rucksshnaidm|rover: ack we need new bug thanks11:24
marios_|rucksshnaidm|rover: (i'll check it and file in a bit)11:25
sshnaidm|roverI think I saw this "cib" error just recently, ykarel do you remember maybe?11:25
ykarelsshnaidm|rover, yes11:25
ykarelsshnaidm|rover, that was solved, but in this job it's different issue11:25
sshnaidm|roverI see, thanks11:25
marios_|rucksshnaidm|rover: ykarel there was an issue with master for cib but it was before promotion11:25
ykarelsshnaidm|rover, in this job issue is, pacemaker is disabled, but ovndb looking for pacemaker11:25
ykarelsshnaidm|rover, to me default is wrong11:26
ykarelbut good to ask ovn guys11:26
ykarelsshnaidm|rover, i mean https://github.com/openstack/tripleo-heat-templates/blob/master/overcloud-resource-registry-puppet.j2.yaml#L15811:26
sshnaidm|roverand who are ovn guys11:26
ykareland https://github.com/openstack/tripleo-heat-templates/blob/master/overcloud-resource-registry-puppet.j2.yaml#L17011:26
ykarelsshnaidm|rover, dalvarez ^^11:27
marios_|ruckykarel: sshnaidm|rover  maybe this is what you were thinking about https://bugs.launchpad.net/tripleo/+bug/1820280 (but closed after promotino)11:27
openstackLaunchpad bug 1820280 in tripleo "periodic ovb master jobs failing "Stderr: u'Cannot find device "br-ex" & error: Could not connect to cluster" [Undecided,Invalid] - Assigned to Marios Andreou (marios-b)11:27
dalvarezhi11:27
ykarelmarios_|ruck, ^^ seems different, but we had a recent bug which was solved by tripleo-common patch revert11:28
sshnaidm|roverdalvarez, hi, we have promotion job failure, seems like ovb requires pacemaker and we have it disabled in the job11:28
sshnaidm|roverdalvarez, s/ovb/ovn/11:29
*** ksambor has joined #oooq11:29
dalvarez13:28 <sshnaidm|rover> dalvarez, hi, we have promotion job failure, seems like ovb requires pacemaker and we have it disabled in the job11:29
dalvarez13:29 <sshnaidm|rover> dalvarez, s/ovb/ovn/11:29
dalvarezksambor: ^11:29
ykarelhmm i was looking for him also11:29
dalvarezsshnaidm|rover: ksambor can help there, but yeah ovn dbs require pacemaker usually, we have a service file for no pacemaker, not sure if we're using it in some job ksambor :?11:30
ykareli saw non pacemaker is used in some jobs11:30
ksamborsshnaidm|rover, where we have this job definition which is failing? also any logs?11:32
*** dsneddon has joined #oooq11:33
*** dsneddon has quit IRC11:37
sshnaidm|roverksambor, logs: https://ci.centos.org/artifacts/rdo/jenkins-tripleo-quickstart-promote-master-current-tripleo-delorean-minimal-476/11:40
sshnaidm|roverksambor, job config: https://github.com/openstack/tripleo-quickstart/blob/master/config/general_config/minimal.yml#L6111:41
sshnaidm|roverykarel, marios_|ruck panda do you know maybe what's reason not to use pacemaker in phase1 job..?11:41
ksamborsshnaidm|rover, thank you!11:41
marios_|rucksshnaidm|rover: no idea and wasn't around then/not really worked with those jobs before (bug at https://bugs.launchpad.net/tripleo/+bug/1820844 fyi)11:42
openstackLaunchpad bug 1820844 in tripleo "[rdo phase 1] failing tripleo-quickstart-promote-master-current-tripleo-delorean-minimal for no pacemaker" [Critical,Triaged]11:42
ykarelsshnaidm|rover, i just know: earlier we used to run two jobs: minimal and minimal_pacemaker, later on minimal_pacemaker was dropped from phase111:43
ykarelmay be you can find something in git history11:44
sshnaidm|roverseems like only weshay knows11:45
sshnaidm|roverykarel, where did you see that it's ovn's problem?11:46
ykarelsshnaidm|rover, see default resource registry11:46
sshnaidm|roverykarel, oh, right11:47
sshnaidm|roverI wonder which recent changes made it failing..11:48
ksamborsshnaidm|rover, we switch to ovn as a default11:49
ykarelmay be ovn default11:49
*** ratailor has quit IRC11:49
sshnaidm|roveroh, I see11:49
ykarelksambor, so https://github.com/openstack/tripleo-heat-templates/blob/master/overcloud-resource-registry-puppet.j2.yaml#L158 change was intentional or by mistake?11:56
ykarelwhy not set to docker/services/ovn-dbs.yaml?11:56
ykarelas pacemaker is disabled by default11:56
ksamborykarel, checking11:58
ykarelack11:58
*** amoralej is now known as amoralej|lunch12:01
ksamborykarel because ovn is by default in ha12:01
ykarelksambor, but pacemaker is not enabled by default12:01
*** vinaykns has joined #oooq12:02
ykarelhttps://github.com/openstack/tripleo-heat-templates/blob/master/overcloud-resource-registry-puppet.j2.yaml#L17012:03
sshnaidm|roverksambor, we have non-ha deployments as well12:04
*** skramaja_ has joined #oooq12:05
*** skramaja has quit IRC12:06
* ykarel need to go away for some time12:06
*** ykarel is now known as ykarel|afk12:06
*** panda is now known as panda|lunch12:07
*** ccamacho has quit IRC12:08
*** dsneddon has joined #oooq12:09
*** jbadiapa has quit IRC12:11
*** skramaja has joined #oooq12:12
*** skramaja_ has quit IRC12:12
*** dsneddon has quit IRC12:14
ksamborsshnaidm|rover, so I will prepare fix with OS::TripleO::Services::OVNDBs: docker/services/ovn-dbs.yaml as a default value, because our default is non ha for deployment12:16
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039 @ https://review.openstack.org/602248, master: tripleo-ci-centos-7 (2 more messages)12:23
weshayarxcruz you need to tmate in?12:23
*** agopi__ has quit IRC12:27
arxcruzweshay: yes12:35
*** jpena is now known as jpena|lunch12:36
*** rfolco has joined #oooq12:37
*** rlandy has joined #oooq12:38
*** trown|outtypewww is now known as trown12:38
weshayarxcruz ok.. bluejeans me bro12:39
arxcruzweshay: just a sec, find a place12:39
rlandysshnaidm|rover: hey - did the multiple ntp servers fix the OVB problems?12:41
sshnaidm|roverrlandy, should fix, but it's not constant problem12:42
sshnaidm|roverrlandy, it doesn't happen in all jobs12:42
sshnaidm|roverrlandy, could be seen only during the time12:43
rlandysshnaidm|rover: it didn't work for me- I had to edit the THT to force the restart12:43
rlandyputting in a patch now to do a formal test12:43
sshnaidm|roverrlandy, well, I see those ntp servers in config, so this part worked12:44
rlandybut bm is always a bit diff as it has no outside server access12:44
rlandysshnaidm|rover: yes  config gets changed12:44
sshnaidm|roverrlandy, in ovb it wasn't problem of restart12:44
rlandybit afaict the restart doesn't happen12:44
rlandyk - so we may have had two diff problems12:45
rlandylikely12:45
sshnaidm|roverrlandy, I see, but in ovb we restart, lemme find..12:45
*** vinaykns has quit IRC12:45
*** dsneddon has joined #oooq12:46
weshayarxcruz http://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/tree/roles/validate-tempest/tasks/tempest-results.yml#n3312:46
sshnaidm|rover2019-03-18 23:08:03 | RUNNING HANDLER [chrony : Restart chronyd] *************************************12:47
sshnaidm|rover2019-03-18 23:08:03 | Monday 18 March 2019  23:07:34 +0000 (0:00:00.492)       0:03:37.858 **********12:47
sshnaidm|rover2019-03-18 23:08:03 | changed: [overcloud-controller-2] => {12:47
sshnaidm|rover2019-03-18 23:08:03 |     "changed": true,12:47
sshnaidm|roverrlandy, ^^12:47
rlandysshnaidm|rover: check nova compute12:47
sshnaidm|roverrlandy, afaik "changed: true" shows it restarts12:47
rlandycontroller restarts12:47
rlandyanyways if this is working fr you, great12:48
sshnaidm|roverrlandy, hmm.. it doesn't run on compute at all12:48
rlandysshnaidm|rover: its does lower down in the in log12:49
arxcruzweshay: https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/validate-tempest/tasks/tempest-results.yml#L1812:49
rlandybut skips restart12:49
sshnaidm|roverrlandy, can you find it here? http://logs.rdoproject.org/78/644378/2/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035/a4c61f9/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz12:49
rlandysshnaidm|rover: ack -  search for the *second*  'Restart chronyd'12:51
*** dsneddon has quit IRC12:51
sshnaidm|roverrlandy, yeah, found it12:52
sshnaidm|roverrlandy, yeah, the same, it's skipped12:53
*** agopi__ has joined #oooq12:53
rlandysshnaidm|rover: I'm putting in a test patch - will let you know12:53
sshnaidm|roverrlandy, need to find what are role variables that passed there, it seems like misconfiguration12:53
rlandysshnaidm|rover: the service is started - not restarted12:53
rlandythe notify should restart12:54
rlandywe will see now12:54
sshnaidm|roverrlandy, "skip_reason": "Conditional result was False" - the only condition there is "when: chrony_manage_service|bool" https://github.com/openstack/ansible-role-chrony/blob/master/handlers/main.yml#L612:55
sshnaidm|roverrlandy, I don't see any other reason to skip it, chrony_manage_service is false for compute12:56
rlandysshnaidm|rover: right so https://github.com/openstack/tripleo-heat-templates/blob/1282bc27db6a036a49ef3dc2acccf71d3b0ba29b/deployment/timesync/chrony-baremetal-ansible.yaml#L12112:56
rlandywe can force that to true and force the state to restart12:56
rlandythat is what I am trying in the patch12:56
sshnaidm|roverrlandy, hmm.. I wonder where it's set for false12:58
rlandysshnaidm|rover: that is what we were trying to figure out yesterday12:59
rlandyalex is convinced it's environmental12:59
sshnaidm|roverrlandy, weird..12:59
rlandyeither way - I need a working job12:59
sshnaidm|roverit can't be environmental, we don't set chrony_manage_service anywhere13:00
rlandyI know :)13:00
rlandysshnaidm|rover: unrelated - fixed https://review.openstack.org/#/c/643140/ per your comment (which was correct - it should be 0)13:01
*** panda|lunch is now known as panda13:01
sshnaidm|roverrlandy, looking13:02
*** udesale has joined #oooq13:02
sshnaidm|roverrlandy, I bet it's something ansible related - with this handler13:02
rlandysshnaidm|rover: likely ... I don't think the logs are telling us the whole story13:03
rlandyhttp://paste.openstack.org/show/747980/ - is bogdan's paste from yesterday13:03
sshnaidm|roverrlandy, maybe it's because we don't run on compute when running on controllers.. why, btw?13:03
sshnaidm|roverbogdando, ^^13:04
sshnaidm|roverbogdando, why don't we run chrony role on both controllers and computes together?13:04
rlandyapparently the role is called five times13:04
sshnaidm|rovertripleo architecture sucks13:06
sshnaidm|rovercorrection: tripleo's lack of architecture sucks13:07
bogdandosshnaidm|rover: i think that just follows the playbooks engraved in those deploy-steps.j213:07
rlandylol13:07
*** skramaja has quit IRC13:07
bogdandothe way it applies host prep tasks by roles13:08
rlandysshnaidm|rover: now running bm again with https://review.openstack.org/#/c/644562/1/deployment/timesync/chrony-baremetal-ansible.yaml13:08
rlandy^^ we may not need both settings13:08
bogdandoI think computes historically always follow controllers for everything in deployment?..13:08
rlandybut that is what I used in my local test13:08
bogdandointrleaving of steps and misc tasks was never implemented to my knowledge13:09
*** amoralej|lunch is now known as amoralej13:10
sshnaidm|roverrlandy, maybe it's worth to put there a few debug messages with showing chrony_manage_service value_13:10
zbrrlandy: can you please wf https://review.rdoproject.org/r/#/c/19129/ ? (already 3x +2s)13:11
sshnaidm|roverrlandy, maybe in different patch13:11
bogdandoI'm pretty sure it's True, but we can try13:11
sshnaidm|roverbogdando, yeah, I suspect so..13:11
rlandyzbr: done13:12
zbrrlandy: thanks13:12
sshnaidm|roverbogdando, it seems like some "false" in role or its dependencies somewhere and glitch of ansible13:12
bogdandorlandy: we can fix that 5 times execution easily, just passing when step == in the host prep tasks13:12
bogdandonot sure that changes anything for the bug tho13:13
sshnaidm|rovermaybe it's problem of flush_handlers13:13
rlandysshnaidm|rover: yeah - job already started - but will add some debug to that13:14
zbrtrue/false with ansible can be bit tricky because their official documentation does not use "true" yaml booleans. (the true/false ones) but internally they convert strings to booleans. is you prob related to that?13:14
rlandyquestion is why does restart occur on controllers and not compute>13:15
*** agopi__ is now known as agopi13:15
rlandyif ansible was not reading the variables correctly, it would be consistent13:16
weshayarxcruz     - name: copy files from tempest container to home directory                                                                                                                               │······················13:17
weshay      command: "cp -Rf /var/lib/tempestdata/tempest {{ working_dir }}"                                                                                                                        â”‚······················13:17
weshay13:17
*** jbadiapa has joined #oooq13:18
sshnaidm|roverbogdando, which version of ansible is used?13:19
bogdando2.6.7 IIRC13:19
weshayarxcruz https://docs.openstack.org/tripleo-quickstart/latest/getting-started.html#a-step-by-step-deployment-with-playbooks13:19
*** ykarel|afk is now known as ykarel13:21
*** dsneddon has joined #oooq13:24
*** zul has quit IRC13:26
weshaysshnaidm|rover talking report-portal w/ arx13:27
weshayjump in13:27
weshayblue13:27
*** zul has joined #oooq13:29
*** dsneddon has quit IRC13:29
*** jpena|lunch is now known as jpena13:33
*** vinaykns has joined #oooq13:43
weshayarxcruz sshnaidm|rover https://seal51.qa.lab.tlv.redhat.com/ui/#rhel_osp13:45
arxcruzweshay: marios_|ruck sshnaidm|rover https://review.openstack.org/#/c/64457813:59
marios_|ruckack arxcruz thanks13:59
mariostripleo weekly time.now()14:02
*** dsneddon has joined #oooq14:04
*** dsneddon has quit IRC14:08
sshnaidm|roverweshay, marios https://review.rdoproject.org/r/#/c/19452/14:09
zbrchandankumar: please check https://review.openstack.org/#/c/644578/1 before someone wf it without looking at output.14:10
zbrarxcruz: ^^ this was for you :P14:11
arxcruzzbr: so... the explanation is already on the commit message, do i really need to change?14:11
sshnaidm|rovermarios, did you have a chance to look at queens 035 job?14:12
zbrarxcruz: commit message does not remain with the task, use of blame is not always convenient.14:12
zbrarxcruz: also, I am almost sure ansible will throw some warnings with current proposed code.14:13
arxcruzzbr: I use blame all the time to know who to blame :P14:14
arxcruzzbr: last time i complain about warning, the openstack infra told me: it's a warning, not an error :P14:15
zbrhaha! yeah, i suppose there is a big pot in purgatory full of ignored warnings.14:16
zbrone neat feature i wanted to share with @oooq is use of --reflink=auto when doing cp. For historical reasons cp does not use auto by default.14:18
zbrthis option can save a LOT of time/resources if the fs is CoW enabled, if not it does and old style copy.14:18
arxcruzzbr: done14:18
*** saneax has quit IRC14:19
*** saneax has joined #oooq14:19
zbrwarn goes under args.14:20
zbrthat the lat bit, i promise ;)14:20
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039 @ https://review.openstack.org/602248, master: tripleo-ci-centos-7 (2 more messages)14:23
*** saneax has quit IRC14:24
marios_|rucksshnaidm|rover: do you mean queens14:40
marios_|ruckhttps://bugs.launchpad.net/tripleo/+bug/182066714:41
openstackLaunchpad bug 1820667 in tripleo "[queens] periodic-centos-ovb-3ctlr_1comp-featureset035-queens fails overcloud deploy overcloud.AllNodesDeploySteps.ComputeDeployment_Step5.0:" [High,Triaged] - Assigned to Marios Andreou (marios-b)14:41
marios_|rucksshnaidm|rover: should be fixed by https://review.openstack.org/#/c/641977/114:42
marios_|rucksshnaidm|rover: if that is the one you meant14:42
marios_|ruck16:12 < sshnaidm|rover> marios, did you have a chance to look at queens 035 job?14:42
rlandyweshay: pls see the design section of https://tree.taiga.io/project/tripleo-ci-board/us/589?milestone=223049. Is this what you were expecting?14:43
*** dsneddon has joined #oooq14:43
sshnaidm|rovermarios_|ruck, yeah, got it14:43
*** dsneddon has quit IRC14:48
weshayrlandy added one line14:49
rlandyweshay: sure - I have not done work on cockpit - this afternoon could I get the speedy tour of making edits there?14:51
weshaysure!!!14:51
* weshay loves demo'ing that tool14:51
rlandyI think having this diff capability in the same env will be better than a standalone thing14:51
rlandydiff rpms, repos, playbooks etc.14:52
rlandywill have wider appeal than ovb vs bm14:52
arxcruzzbr: done, and no more changes!14:53
arxcruzi'll no longer add you for review :P14:54
zbrarxcruz: thanks, looks perfect now.15:00
*** saneax has joined #oooq15:03
*** dsneddon has joined #oooq15:22
*** dsneddon has quit IRC15:26
*** ykarel is now known as ykarel|away15:32
marios_|ruckweshay: so re that bug you pointed to just now we have this https://bugs.launchpad.net/tripleo/+bug/182057415:43
openstackLaunchpad bug 1820574 in tripleo "[rocky] periodic-rocky-containers-build fails for missing centos-binary-cron" [Critical,Triaged] - Assigned to Marios Andreou (marios-b)15:43
weshaymarios_|ruck the issue is the undercloud install on rocky.. w/ containers when the containers have not yet been built15:44
weshaymarios_|ruck so I discussed options w/ yatin15:44
* weshay looks at bug first15:45
*** dsneddon has joined #oooq15:45
weshaymarios_|ruck k.. so the tripleo-common patches that enable the stein container builds15:45
weshaycould be cherry-picked to rocky15:45
weshayOR15:45
weshayfor featureset023 we can hack the fs file15:46
weshayso that when the job is periodic15:46
weshaythe container_build_id is from current-tripleo15:46
weshayso that the undercloud can be installed15:47
weshaythen the containers could be built15:47
weshayfollowing?15:47
weshaysshnaidm|rover fyi ^15:47
marios_|ruckweshay: ish emilien is also talking in my ears its confusing15:47
* marios_|ruck reads back 15:47
marios_|ruckweshay: well lets cherrypick if that is possible/no issues15:48
marios_|ruckweshay: do the fs edit if we need to?15:48
weshaysure..15:48
weshayinternal is importing rocky the bit's we promoted15:48
weshayso we have plenty of time15:49
sshnaidm|roverweshay, should we build containers before undercloud installed or after?15:50
sshnaidm|roverweshay, seems logical to build them and to install undercloud with them15:50
weshaysshnaidm|rover in rocky we have a chicken and egg15:51
weshaythat IS the job that builds the containers15:51
*** hamzy has quit IRC15:55
zbrweshay: re out bindep talk, i updated https://review.openstack.org/#/c/639951/9 -- only when you have time.15:55
*** fmount is now known as fmount|afk16:00
marios_|ruckrfolco: you cant hear me?16:02
marios_|ruckpanda: canyou hear me16:02
rfolcomarios_|ruck, mor eor less16:02
rfolcoheard once16:02
marios_|ruckrfolco: so me and panda can talk16:02
marios_|ruckbut you don't hear us16:02
*** jaosorior has quit IRC16:02
rfolcoci community call now https://bluejeans.com/411356779816:03
rfolcomarios, quiquell, sshnaidm, weshay, panda, rlandy, arxcruz, mwhahaha, rfolco, chkumar, ssbarnea, kopecmartin16:03
rfolcostill having network issues on bj16:05
rfolcoanyone else?16:05
marios_|rucksshnaidm|rover: weshay wana join https://bluejeans.com/4113567798 ci community16:05
weshaymarios_|ruck I have another call16:05
marios_|ruckweshay: sorry you told me already16:05
sshnaidm|rovermarios_|ruck, I'm there..16:05
marios_|ruckweshay: ack16:05
marios_|rucksshnaidm|rover: ack16:05
sshnaidm|rovermarios_|ruck, https://review.rdoproject.org/r/#/c/19452/16:10
*** trown is now known as trown|lunch16:13
*** jbadiapa has quit IRC16:17
*** hamzy has joined #oooq16:18
marios_|rucksshnaidm|rover: ah its config so we cant see it until it merges16:19
sshnaidm|rovermarios_|ruck, yep, but we can run it in some patch after it's merged16:20
marios_|rucksshnaidm|rover: right16:20
sshnaidm|rovermarios_|ruck, just to check it16:20
marios_|ruckshbut16:20
marios_|rucksshnaidm|rover: btu16:20
marios_|rucksshnaidm|rover: i mean it is still couple days before we see its green and can switch to it16:20
marios_|rucksshnaidm|rover: so it all depends how much hassle those cherrypicks are16:20
sshnaidm|rovermarios_|ruck, why?16:20
marios_|ruckand if its worth the effot16:20
marios_|rucksshnaidm|rover: well it needs to merge and see if it has any issues16:21
sshnaidm|rovermarios_|ruck, it runs every 4 hours, why couple days?16:21
marios_|rucksshnaidm|rover: yeah ok16:21
marios_|rucksshnaidm|rover: still 'some time'16:21
marios_|rucksshnaidm|rover: lets see what happens once we merge it then?16:21
sshnaidm|rovermarios_|ruck, yes, and we can check it in patch right after merge16:21
marios_|rucksshnaidm|rover: and in meantime we can prepare for cherrypick if we need it16:21
marios_|rucklike there are unkown issues that will delay rocky push job16:22
sshnaidm|rovermarios_|ruck, yeah16:22
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039 @ https://review.openstack.org/602248, master: tripleo-ci-centos-7 (2 more messages)16:23
marios_|rucksshnaidm|rover: added note in the https://etherpad.openstack.org/p/ruckroversprint7 under https://bugs.launchpad.net/tripleo/+bug/182057416:23
openstackLaunchpad bug 1820574 in tripleo "[rocky] periodic-rocky-containers-build fails for missing centos-binary-cron" [Critical,Triaged] - Assigned to Marios Andreou (marios-b)16:23
marios_|ruckrlandy: weshay panda sshnaidm|rover so lets merge that https://review.rdoproject.org/r/#/c/19452/ and see how the periodic-tripleo-centos-7-rocky-containers-build-push does and consider that for the https://bugs.launchpad.net/tripleo/+bug/182057416:34
openstackLaunchpad bug 1820574 in tripleo "[rocky] periodic-rocky-containers-build fails for missing centos-binary-cron" [Critical,Triaged] - Assigned to Marios Andreou (marios-b)16:34
rlandymarios_|ruck: you're -1 on that patch16:36
sshnaidm|rovermarios_|ruck, so we merge it or not?16:36
marios_|ruckrlandy: yeah sorry16:37
marios_|rucksshnaidm|rover: i added coment just now16:37
marios_|ruckminor nit16:37
marios_|ruckneeds layout?16:37
marios_|ruckotherwise where will it run16:37
sshnaidm|rovermarios_|ruck, no, I'd like to test it before adding it to layout16:37
sshnaidm|rovermarios_|ruck, and it should be a different commit16:37
marios_|rucksshnaidm|rover: ah ok thanks then removing -116:37
marios_|rucksshnaidm|rover: ack16:38
marios_|ruckrlandy: thanks16:38
rlandymarios_|ruck: sshnaidm|rover: k - w+116:38
rlandyif we need to revert, we will16:38
marios_|ruckrlandy: thanks but it shouldnt even run as per our discussion just now its not even in the layout16:39
marios_|ruckrlandy: so should not break anything16:39
marios_|ruckt&c apply16:39
rlandymaster bm executing tempest16:39
rlandybig party16:39
rlandyweshay: ^^16:39
*** jbadiapa has joined #oooq16:39
marios_|ruckweshay: so what did you mean by 17:45 < weshay> marios_|ruck k.. so the tripleo-common patches that enable the stein container builds 17:45 < weshay> could be cherry-picked to rocky please16:40
weshaynice.. w/ your patch?.. still VERY COOL16:40
marios_|ruckweshay: which patches (looking to prepare incase the push job doesn't work out. )16:40
weshaymarios_|ruck ensuring the container build as we know it in stein works the same way in rocky16:41
weshayeither in check or promotoin16:41
weshaypromotion16:41
marios_|ruckweshay: i think thats what sshnaidm|rover is doing then16:41
marios_|ruckweshay: i mean he's proposed the job lets see how it runs for rocky16:41
marios_|ruckhttps://review.rdoproject.org/r/#/c/19452/16:41
marios_|ruckweshay: that what you mean ?16:41
sshnaidm|rovermarios_|ruck, https://review.rdoproject.org/r/#/c/19482/16:42
*** kopecmartin has quit IRC16:42
rlandyarxcruz: hi - still hitting the "| OSError: [Errno 2] No such file or directory: '/home/zuul/tempest'" error I asked about last time. Can you point me the fix?16:42
weshaysshnaidm|rover does that work in rocky?16:42
marios_|ruckweshay: or, what patches /features from stein do you have in mind for tripleo-common16:42
rlandyso I don't keep bugging you16:42
weshayis the code there in tripleo-common?16:42
weshayrlandy we fixe dit16:42
* weshay gets16:42
sshnaidm|roverweshay, let's see https://review.rdoproject.org/r/#/c/19482/16:42
weshayrlandy https://review.openstack.org/#/c/644578/316:42
marios_|ruckweshay: do you have time for sync now?16:42
weshaymarios_|ruck can you poke a tripleo-common16:42
weshaymarios_|ruck still on a call16:43
marios_|ruckweshay: k me and sshnaidm|rover already synced a bit, this rocky issue is the most pressing rocky is the only one that didn't promote yesterday16:43
marios_|ruckweshay: the etherpad has my latest info16:44
marios_|ruckweshay: i am poking at tripleo-common but can we narrow it a bit did you anticipate some particular thing there (was there a rework/fix of how containers are built in stein?)16:44
rlandythank you16:45
marios_|ruckweshay: ah buildah stuff16:45
marios_|ruckweshay: probably16:45
sshnaidm|roverpanda, is it still relevant? https://review.openstack.org/#/c/602456/16:46
pandasshnaidm|rover: if we are still using ZUUL_* bash environment variables, yes.16:48
*** udesale has quit IRC16:48
*** dsneddon has quit IRC16:48
sshnaidm|roverrlandy, can you vote please? https://review.openstack.org/#/c/644378/16:48
*** dtantsur is now known as dtantsur|afk16:56
*** jfrancoa has quit IRC17:01
marios_|rucksshnaidm|rover: fyi https://review.rdoproject.org/r/#/c/19484/17:05
* marios_|ruck time to go17:05
marios_|ruckstarting shutdown sequence17:06
arxcruzrlandy: https://review.openstack.org/#/c/644578/ sorry, i was on commute back to home17:06
rlandysshnaidm|rover: done17:16
rlandyarxcruz: thanks17:16
*** dsneddon has joined #oooq17:23
*** ykarel|away has quit IRC17:26
*** holser_ is now known as holser|run17:30
rlandysshnaidm|rover: this is the log from the passing bm overcloud deployment. It includes you change to report sources ... https://sf.hosted.upshift.rdu2.redhat.com/logs/25/165325/14/check/periodic-tripleo-ci-centos-7-baremetal-single_nic-3ctlr_1comp-featureset001-master/01828d8/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz17:32
rlandysearch for 'Ensure chronyd is running'17:32
*** trown|lunch is now known as trown17:43
*** fmount|afk is now known as fmount17:43
*** bogdando has quit IRC17:44
chandankumararxcruz: weshay https://review.openstack.org/#/c/644600/ with this change and martin doc change life will be easier for os_tempest17:53
rlandyweshay: marios_|ruck:could I get a second review here https://review.openstack.org/#/c/643140/?17:53
rlandyminimal change17:53
rlandyrequired for a depends on17:53
*** saneax has quit IRC17:54
weshayrlandy comment added17:56
rlandyweshay: https://review.openstack.org/#/c/643140/2..3/roles/common/defaults/main.yml17:57
rlandywas 1.- changed to 0.117:57
weshayrlandy no... .1 was change to .017:58
rlandyweshay: yep - had it as .1 - changed it to .017:59
rlandyleft comment - can change back per line above17:59
weshaywhy.. and why is .1 still there17:59
weshaywhat is the diff between the two vars?17:59
weshayundercloud_external_network_cidr17:59
weshayndercloud_external_network_cidr_ipv418:00
*** derekh has quit IRC18:00
rlandyweshay: see  https://review.openstack.org/#/c/642924/18:00
rlandyweshay: can't use ipv6 for masquerade18:00
weshayI'm not suggesting you should18:00
weshayYou have two vars for ipv418:00
weshayrlandy http://codesearch.openstack.org/?q=undercloud_external_network_cidr&i=nope&files=&repos=18:01
weshayhttp://codesearch.openstack.org/?q=external_network_cidr_ipv4&i=nope&files=&repos=18:01
rlandyweshay: the var undercloud_external_network_cidr_ipv4 is only ever referenced when ipv6 exists18:02
rlandyhttps://review.openstack.org/#/c/642924/7/roles/undercloud-deploy/templates/hieradata-overrides-t-h-t-undercloud.yaml.j218:03
rlandyif it's ipv4, that var is just ignored18:03
zbrmarios_|ruck: safe to wf, https://review.openstack.org/#/c/644378/ from sagi (multi ntp)18:03
*** chandankumar is now known as raukadah18:05
weshayrlandy this doesn't tell me it's ignored http://codesearch.openstack.org/?q=undercloud_external_network_cidr&i=nope&files=&repos=18:05
weshayrlandy the explanation of what you are doing in this patch is not good enough yet18:06
rlandywill update commit message18:06
weshayrlandy and why do we need two vars for ipv418:06
weshayrlandy undercloud_external_network_cidr: 10.0.0.0/2418:07
weshayrlandy http://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/tree/config/environments/ovb-common.yml#n1518:07
rlandyweshay: because when you are using ipv6, the  undercloud_external_network_cidr will resolve to an ipv6 value - which doesn't work for routing and masquerade18:07
rlandyit's the ipv4 value - when ipv6 is active18:07
weshayoooh18:08
weshayk.. make that clear in the code and the comment18:08
weshaybecause it's not at all clear atm18:08
zbrrfolco: can we agree to stick to one gerrit topic for all branching changes? so we can easily nav between them?18:14
zbrinitially I used "oooq/branching" but I am open to any other name as long we all try to use one.18:14
rfolcozbr, deal18:14
rfolcozbr, changing my patches18:14
zbrrfolco: thanks. doing this we can get a dasboard for free ;)18:15
rfolcoindeed, this should be a retro card btw :)18:15
*** jpena is now known as jpena|off18:19
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039 @ https://review.openstack.org/602248, master: tripleo-ci-centos-7 (2 more messages)18:23
rlandyweshay: code and message updated: https://review.openstack.org/#/c/643140/18:23
*** kopecmartin has joined #oooq18:34
*** amoralej is now known as amoralej|off18:34
rfolcozbr, I don't see your tasks in taiga. Please add a task for each of your user stories.18:45
*** jbadiapa has quit IRC18:50
weshayrlandy if you want a quick overview of the rrcockpit..19:25
weshayI have it going19:25
weshayrlandy /me just added https://review.rdoproject.org/r/#/c/19488/19:25
rlandyweshay: yes, pls .. I added cards for that user story19:26
zbrrlandy: another boring and simple one if you do not mind: https://review.rdoproject.org/r/#/c/19477/19:26
weshayzbr are those in use?19:27
rlandyzbr: w+1'ed19:27
zbrweshay: no idea, outside my scope to take care of that.19:27
weshayzbr let's chat for a hot sec19:28
zbrjoning19:28
rlandywow - this file is huge19:29
*** gouthamr_ has joined #oooq19:34
*** ksambor has quit IRC19:35
weshayzbrhttps://review.openstack.org/#/c/643963/1/config/release/tripleo-ci/CentOS-7/stein.yml19:38
weshayhttps://review.openstack.org/#/c/643963/1/config/release/tripleo-ci/CentOS-7/stein.yml19:38
*** gouthamr_ has quit IRC19:38
weshayrlandy ok.. I'm chatting w/ zbr will ping you in a few19:46
rlandyweshay: sure19:47
rfolcorlandy, envE is still safe to use ?19:52
*** trown is now known as trown|outtypewww20:14
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039 @ https://review.openstack.org/602248,  (2 more messages)20:23
*** tosky has quit IRC20:30
weshayrlandy k20:43
weshayready if it's not too late20:43
weshaywe can do it tomorrow20:44
weshayif you want20:44
rlandyweshay: no, now is good20:44
weshayk.. I'm in20:44
rlandyjoining20:45
*** hamzy has quit IRC20:46
*** agopi has quit IRC21:18
*** vinaykns has quit IRC21:45
*** agopi has joined #oooq22:10
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039 @ https://review.openstack.org/602248,  (2 more messages)22:23
*** rlandy has quit IRC23:13

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!