Wednesday, 2018-12-05

*** rlandy has quit IRC00:59
hubbot1FAILING CHECK JOBS on master: tripleo-ci-fedora-28-standalone @ https://review.openstack.org/560445, stable/ocata: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001 @ https://review.openstack.org/56429101:56
*** matbu has quit IRC02:32
*** matbu has joined #oooq02:32
*** apetrich has quit IRC03:15
*** ykarel|away has joined #oooq03:21
*** agopi has joined #oooq03:23
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035 @ https://review.openstack.org/604298, master: tripleo-quickstart-extras-gate-master-delorean-full-featureset052, tripleo-ci-centos-7-scenario009-multinode-oooq, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/560445, stable/ocata: tripleo-ci-centos-7-ovb-3ctlr_1comp- (1 more message)03:57
*** jaganathan has joined #oooq05:32
*** hubbot1 has quit IRC05:45
*** dmellado has quit IRC05:46
*** gouthamr has quit IRC05:46
*** hubbot1 has joined #oooq05:47
*** dmellado has joined #oooq05:48
*** gouthamr has joined #oooq05:53
*** dmellado has quit IRC05:55
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035 @ https://review.openstack.org/604298, master: tripleo-quickstart-extras-gate-master-delorean-full-featureset052, tripleo-ci-centos-7-scenario009-multinode-oooq, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/560445, stable/ocata: tripleo-ci-centos-7-ovb-3ctlr_1comp- (1 more message)05:57
*** dmellado has joined #oooq05:57
*** ratailor has joined #oooq06:00
*** dmellado has quit IRC06:02
*** hubbot1 has quit IRC06:06
*** hubbot1 has joined #oooq06:08
*** saneax has joined #oooq06:10
*** dmellado has joined #oooq06:14
*** gouthamr has quit IRC06:15
*** gouthamr has joined #oooq06:18
*** ykarel|away has quit IRC06:21
*** gouthamr has quit IRC06:29
*** gouthamr has joined #oooq06:32
*** ykarel|away has joined #oooq06:45
*** ykarel|away is now known as ykarel06:54
*** gouthamr has quit IRC06:58
*** gouthamr has joined #oooq07:07
*** jfrancoa has joined #oooq07:13
*** skramaja has joined #oooq07:15
*** quiquell|off is now known as quiquell07:17
*** saneax has quit IRC07:24
*** saneax has joined #oooq07:25
chandan_kumarquiquell: hello07:31
chandan_kumarquiquell: Can i edit your tempestconf patch?07:31
quiquellchandan_kumar: sure07:33
*** gouthamr has quit IRC07:40
ykarelssbarnea|rover, marios_|ruck is the following issue known: Transaction check error:07:42
ykarel  file /usr/lib/python2.7/site-packages/docker/__init__.py from install of python2-docker-3.3.0-1.el7.noarch conflicts with file from package python-docker-py-1:1.10.6-7.el7.noarch07:42
*** apetrich has joined #oooq07:42
ykarelseeing in RDO phase1 tripleo jobs07:42
*** gouthamr has joined #oooq07:42
*** rascasoft has joined #oooq07:43
chandan_kumarquiquell: https://review.openstack.org/#/c/622865/ -> py3 devstack job to check how may things are broken07:45
chandan_kumarso that I can fix all stuff in a single go07:45
marios_|ruckykarel: o/07:45
quiquellchandan_kumar: That's nice !!!07:46
marios_|ruckykarel: i saw it yesterday do you have a trace? lemme see if i find th enote07:46
marios_|ruckykarel: don't think we have a bug for it yet07:46
ykarelmarios_|ruck, https://ci.centos.org/job/tripleo-quickstart-promote-rocky-rdo_trunk-minimal/45/consoleText07:46
ykarelsame u can see in other phase1 tripleo jobs07:46
marios_|ruckykarel: yeah i saw it here http://logs.openstack.org/20/619520/11/check/tripleo-ci-centos-7-scenario004-standalone/b5bd260/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz#_2018-12-04_08_34_2507:46
marios_|ruckykarel: ack thanks07:46
ykarelmarios_|ruck, ack so it's same07:48
*** quiquell is now known as quiquell|brb07:48
ykareland should be and outcome of CentOS7.6 release07:48
quiquell|brbmarios_|ruck: all env files merged for standalone, and ANSIBLE_LIBRARY in the gates07:48
quiquell|brbs/in/at/07:48
marios_|ruckquiquell|brb: sweet07:49
marios_|ruckugh 18 hrs queue07:50
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035 @ https://review.openstack.org/604298, master: tripleo-quickstart-extras-gate-master-delorean-full-featureset052, tripleo-ci-centos-7-scenario009-multinode-oooq, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/560445, stable/ocata: tripleo-ci-centos-7-ovb-3ctlr_1comp- (1 more message)07:57
*** brault has joined #oooq07:57
*** ykarel is now known as ykarel|lunch08:00
*** dsneddon has quit IRC08:01
*** dsneddon has joined #oooq08:06
*** amoralej|off is now known as amoralej08:14
quiquell|brbssbarnea|rover: Have fixed merge at migrate ansible-lint to pre-commit https://review.openstack.org/61830608:14
quiquell|brbssbarnea|rover: Good thing there were stuff broken that get discovered I think08:15
*** bogdando has joined #oooq08:18
*** dsneddon has quit IRC08:22
*** quiquell|brb is now known as quiquell08:22
*** holser_ has joined #oooq08:41
*** dsneddon has joined #oooq08:47
*** tosky has joined #oooq08:50
*** gkadam has joined #oooq08:52
*** dsneddon has quit IRC08:52
*** ykarel|lunch is now known as ykarel09:00
*** brault has quit IRC09:03
*** brault has joined #oooq09:04
*** ccamacho has joined #oooq09:20
quiquellchandan_kumar: It's not python2 compatible http://logs.openstack.org/43/622343/1/check/python-tempestconf-tempest-devstack-admin/5cf3816/job-output.txt.gz#_2018-12-04_15_30_15_19052809:24
quiquellchandan_kumar: goint to try .encode('utf-8')09:24
quiquellchandan_kumar: going to add the py3 job to the review09:28
chandan_kumarquiquell: not worked09:28
chandan_kumarquiquell: http://logs.openstack.org/65/622865/1/check/python-tempestconf-tempest-devstack-py3/d80e3c6/job-output.txt.gz need to tweak few things there09:29
quiquellchandan_kumar: I am trying now with data.encode('utf-8')09:29
quiquellchandan_kumar: ack, I will test at my env with .encode09:29
chandan_kumarok09:30
chandan_kumarquiquell: how we are installing pip3 or pip based on python version in ansible playbook in f28?09:33
quiquellchandan_kumar: Depends on the playbook09:34
quiquellchandan_kumar: generally we are running ansible-playbook-3 so pip module will use pip309:34
quiquellchandan_kumar: You don't need to do anything special09:35
quiquellchandan_kumar: talking about quickstart jobs09:35
chandan_kumarquiquell: there are few small roles here https://github.com/openstack/python-tempestconf/blob/master/playbooks/python-tempestconf-tempest-devstack.yaml09:36
chandan_kumarquiquell: it is about devstack based upstream zuul jobs09:37
quiquellchandan_kumar: no idea how they execute ansible-playbook there09:37
quiquellchandan_kumar: at ourtripleo jobs we run ansible-playbook-309:37
chandan_kumarquiquell: trying to hunt down09:37
quiquellchandan_kumar: First you have to point out in the logs how it's run ?09:38
quiquellchandan_kumar: ack09:38
*** agopi is now known as agopi|brb09:40
ssbarnea|roverquiquell: marios : please revote on https://review.openstack.org/#/c/618306/ -- safe to workflow.09:41
quiquellssbarnea|rover: have you downgrade pre-commit version ?09:45
ssbarnea|roverupgrade09:46
ssbarnea|roverto latest09:46
ssbarnea|roverjust run "pre-commit autoupgrade" and it does everythign for you.09:46
quiquellssbarnea|rover: Ahh yes I was seeing patchset 6 ok ok09:47
quiquellssbarnea|rover: now it makes sense09:47
quiquellssbarnea|rover: +2, let's wait for CI09:47
quiquellssbarnea|rover: well quite the opposite let's wait for CI then +209:48
ssbarnea|roversure, is up to you. it would not go further if doesn't pass both checks/gates anyway.09:48
ssbarnea|roveron simple stuff i prefer to vote right away and hope it will pass, avoiding two other interactions with it.09:49
*** derekh has joined #oooq09:49
quiquellssbarnea|rover: the other linting review is already merged09:49
quiquellssbarnea|rover: ack09:49
ssbarnea|roveri know, this is why this one became so small09:49
*** dsneddon has joined #oooq09:50
ssbarnea|roverbtw, i want to make validate jinja2 script made by sshnaidm part of ansible-lint, i hope he does not mind me making a PR to ansible-lint with his code.09:52
ssbarnea|roverwe need to use this code in more places, and ansible-lint seems like a good place to host this feature, IMHO09:53
quiquellssbarnea|rover: talk with him first09:53
ssbarnea|roverquiquell: that's why i didn't do it yet.09:54
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario001-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, master: tripleo-quickstart-extras-gate-master-delorean-full-featureset052, tripleo-ci-centos-7-scenario009-multinode-oooq, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci- (1 more message)09:57
*** dsneddon has quit IRC09:59
*** quiquell is now known as quiquell|brb10:21
chandan_kumarquiquell|brb: sorry to say but the changes is not going to wokr10:22
chandan_kumari am working on setting pip executables to test it on python310:23
*** agopi|brb is now known as agopi10:29
*** dsneddon has joined #oooq10:40
*** chem has quit IRC10:43
*** chem has joined #oooq10:44
*** dsneddon has quit IRC10:45
*** dtantsur|afk is now known as dtantsur10:50
*** quiquell|brb is now known as quiquell10:52
quiquellchandan_kumar: ack10:52
chandan_kumarquiquell: https://review.openstack.org/#/c/622865/ need some help10:53
chandan_kumarquiquell: I am not sure it is the correct way10:53
quiquellchandan_kumar: first of all you have to run this thing on something like f2810:54
quiquellchandan_kumar: Don't know if centos-7 python3 support is going to be enough10:54
quiquellchandan_kumar: Ahh they are run at ubuntu-xenial10:57
*** dsneddon has joined #oooq11:01
chandan_kumarquiquell: yes let see how it goes11:03
*** chem has quit IRC11:06
*** chem has joined #oooq11:07
*** holser_ is now known as holser|lunch11:10
quiquellmarios_|ruck: I think we have a passing scenario002 job11:11
marios_|rucknice i saw 3 passing earlier too11:13
marios_|rucki have to find time to fix the tempset stuff on 1/4 today11:13
marios_|ruckbut i am very unlucky with those11:13
marios_|ruckquiquell: like https://url.corp.redhat.com/scen4-python-docker-fail-unrelated https://url.corp.redhat.com/scen1-package-fail11:13
marios_|ruckhttps://url.corp.redhat.com/scen1-ntp-fail-unrelated11:13
sshnaidmquiquell, can you please make a testing patch in extras repo for https://review.openstack.org/#/c/621562/ ?11:14
marios_|ruckam hitting all the things not even hittimg tempest yet quiquell11:14
marios_|ruckbut i need to fix it anyway11:14
quiquellsshnaidm: Forgot to paste it here, is this https://review.openstack.org/#/c/622244/11:14
quiquellsshnaidm: well now that we have working standalone sceanrios, let me fix the testing patch to run those11:15
quiquellmarios_|ruck: looks like ceph is trying to install already instdalled stuff at docker11:17
quiquellsshnaidm: Looks like something is wrong at packaging + ceph + ansible11:17
quiquells/ansible/standalone/11:17
quiquell1 and 4 will be the hardest to fix11:18
marios_|ruckquiquell: well that packaging thing has been seen elsewhere11:18
marios_|ruckquiquell: i mean in phase1 job sec11:18
marios_|ruckquiquell: ykarel pinged about it earlier and i filed a bub11:18
marios_|ruckbug11:18
marios_|ruck    https://bugs.launchpad.net/tripleo/+bug/1806853 python-dockery-py package conflict causing jobs to fail    "python-docker-py-1:1.10.6-7.el7.noarch conflicts with file from package python2-docker" @ https://url.corp.redhat.com/scen4-python-docker-fail-unrelated (yesterday) && just now ykarel irc11:18
openstackLaunchpad bug 1806853 in tripleo "python-dockery-py package conflict causing jobs to fail" [Undecided,Triaged] - Assigned to Marios Andreou (marios-b)11:18
marios_|ruckquiquell: going afk for a bit in a bit and back in an hour ish before program call11:19
marios_|ruckssbarnea|rover: ^11:19
marios_|ruckssbarnea|rover: updated the status as we discussed11:19
sshnaidmquiquell, btw, you removed all I added in https://review.rdoproject.org/r/#/c/17552/21..22 with last patchset11:19
quiquellsshnaidm: hate to work at same review, let me put that back sorry  :-(11:20
sshnaidmquiquell, you should have updated patchset locally first11:20
quiquellsshnaidm: Yep forbot...11:20
quiquellsshnaidm: Maybe we can merge and you add a new review with the RDO ?11:20
quiquellsshnaidm: And we merge that, whatever you prefer11:20
quiquellsshnaidm: holy sh.. alan pevec is talking about podman11:21
quiquellsshnaidm: Do we have to run this thing with podman ?11:21
sshnaidmquiquell, I've told you :D11:21
quiquellsshnaidm: you read the future... docker-compose + podman pufff11:21
sshnaidmquiquell, tell him that only after podman module for ansible11:22
quiquellsshnaidm: Can you retore the RDO ? I don't want to screw it twice11:23
quiquellsshnaidm: add RDO + rename "upstream" tenant to tripleo-ci-reproducer and we merge11:24
quiquellsshnaidm: Sounds good to you ?11:24
sshnaidmquiquell, can you elaborate about last changes?11:27
quiquellsshnaidm: I mean now that we run also RDO jobs with the same tenant11:28
quiquellsshnaidm: Having 'upstream' at tenant name does not make sense11:28
quiquellsshnaidm: Let's just call it 'tripleo-ci-reproducer' tenant11:28
quiquellsshnaidm: It's ok with you ?11:28
sshnaidmquiquell, of course11:29
quiquellsshnaidm: is the playbook working for you ? it get stuck at my system11:44
sshnaidmquiquell, I didn't try with last changes, but from my patchset it worked11:45
sshnaidmquiquell, why do we need all these roles in last patchset?11:46
sshnaidmquiquell, configure-mirrors ?11:46
quiquellsshnaidm: I have try to put there as much stuff as possible from openstack-infra/project-config base job11:47
quiquellsshnaidm: To reduce divergence11:47
quiquellsshnaidm: But I think we don't need any after merging the review about tqe requirements11:47
sshnaidmquiquell, idk.. maybe not all is required, I doubt if we need configure-mirrors for example11:48
quiquellsshnaidm: we can remove it, let's talk with rlandy first11:49
sshnaidmquiquell, try to run from my patchset and to see if it's stuck11:49
quiquellsshnaidm: For that I need the gerrit-stream merged11:50
quiquellsshnaidm: https://review.rdoproject.org/r/#/c/17677/11:50
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario001-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, master: tripleo-quickstart-extras-gate-master-delorean-full-featureset052, tripleo-ci-centos-7-scenario009-multinode-oooq, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci- (1 more message)11:57
sshnaidmquiquell, merged12:07
sshnaidmquiquell, need to figure out what to do with this stream permissions, we need it for everyone actually12:08
quiquellsshnaidm: yep... full access or groups12:11
quiquellsshnaidm: I suspect this is a software factory config12:11
quiquellsshnaidm: And that's why there is non of this upstream12:11
quiquellsshnaidm: Can you do a cleanup and run the playbook, I am doing it myself too to check that last version is working before merge it12:12
*** ratailor has quit IRC12:19
ykarelquiquell, can u please review https://review.openstack.org/#/c/618669/12:22
ykarelpanda ^^12:22
sshnaidmquiquell, yeah, will do now12:23
*** holser|lunch is now known as holser_12:25
quiquellykarel: thanks, checking12:27
quiquellykarel: workflowed I see the logs there12:29
ykarelquiquell, Thanks12:29
rfolcoquiquell, sshnaidm panda: may I get one more +2 for scen3 ? https://review.openstack.org/#/c/618761/12:29
quiquellrfolco: passing :-)12:31
quiquellrfolco: commented12:33
sshnaidmrfolco, commented12:34
quiquellsshnaidm: launching, standalone, multinode and OVB in the same review :-)12:36
sshnaidmquiquell, cool :)12:36
quiquellIf all of the three passes let's merge12:37
quiquellsshnaidm: Did it work for you the playbook does it get stuck ?12:37
quiquellsshnaidm: I see a "merge-check" pipeline12:39
quiquellsshnaidm: We have to exclude stuff from config12:40
sshnaidmquiquell, testing now the latest version12:40
sshnaidmquiquell, yeah, maybe12:40
quiquellsshnaidm: manual starting up is done with "docker-compose up reproducer"12:43
quiquellrfolco, marios_|ruck, sshnaidm, panda: We can also review the scenario002 job too https://review.openstack.org/#/c/618709/12:44
rfolcoquiquell, zuul ci did not run12:45
quiquellrfolco: Ahh yes I have the greasymonkey stuff, near finish12:46
sshnaidmmarios_|ruck, what was the final decision about settings in featureset and zuul config?12:47
marios_|rucksshnaidm: what do you mean? for standalone scenario we deicded to allow the featureset_override for env files12:51
marios_|rucksshnaidm: that?12:51
marios_|rucksshnaidm: you were on the cal for that though12:51
marios_|rucks/cal/call12:51
marios_|rucksshnaidm: have call in few mins so getting coffee lets catch up after?12:52
sshnaidmmarios_|ruck, no, I haven't been, that's why I'm asking12:55
sshnaidmmarios_|ruck, ok12:55
marios_|rucksshnaidm: ack we can have call after if u like12:57
sshnaidmmarios_|ruck, so we have only one fs file and overriding all the rest in zuul config, this was the resolution?12:58
sshnaidmquiquell, hmm.. something is wrong with this last patchset12:59
quiquellsshnaidm: what happend ?12:59
quiquellsshnaidm: is stuck ?12:59
sshnaidmquiquell, yes, and I don't see even this start page of zuul..13:00
sshnaidmquiquell, maybe we'll get rid of all this roles in base and try to add them one by one?13:01
quiquellsshnaidm: not related, they don't get executd at start-up13:01
rfolcosshnaidm, just double checking... this one I have to manual rebase, right? https://review.openstack.org/#/c/621907/13:02
sshnaidmquiquell, did they work in the job?13:02
quiquellsshnaidm: can you do a docker-compose logs --tail=10 -f scheduler ?13:02
quiquellsshnaidm: yep13:02
sshnaidmquiquell, I looked at all logs, everything looks fine13:02
quiquellsshnaidm: mine is stuck here scheduler_1     | 0046338ced9b63cccb04cd830e3ae8575f9f0ccbc8f0 refs/changes/27/454727/413:02
quiquellsshnaidm: maybe there are some infra issues13:02
sshnaidmrfolco, yeah, right13:02
*** ykarel is now known as ykarel|afk13:03
sshnaidmquiquell, mine is stuck on different patchset, but seems the same..13:03
sshnaidmscheduler_1     | 00465940e4302576580c59839fa975b1b5cfc5b4886c refs/changes/14/204314/113:04
sshnaidmquiquell, will try again the previous version13:04
quiquellsshnaidm: found that if you stop "reproducer" service it continues13:04
quiquellsshnaidm: wait13:04
sshnaidmquiquell, well, right13:04
quiquellsshnaidm: I think the readiness stuff is affecting startup13:04
sshnaidmquiquell, hmm.. interesting13:05
quiquellif you do a docker-compose stop reproducer13:05
quiquellit contineus13:05
sshnaidmquiquell, I just stopped playbook13:05
quiquellsshnaidm: this is similar as stop "reproducer"13:05
quiquellsshnaidm: Now it should be starting up13:05
quiquellsshnaidm: try to run again docker-compose up reproducer so it start to check readiness again13:06
sshnaidmquiquell, I think we need to get rid off reproducer docker13:07
sshnaidmquiquell, does it just checking web page? we don't need docker for that and definitely not zuul image13:08
quiquellsshnaidm: zuul image is the same13:08
quiquellsshnaidm: is about moving readiness to docker-compose13:08
quiquellsshnaidm: so you can start things up after manual change13:08
quiquellsshnaidm: and end at ready13:08
sshnaidmquiquell, it doesn't start anything, just checking web page13:09
quiquellsshnaidm: look at the depends_on in the docker-compose.yaml13:09
sshnaidmquiquell, I think it's overhead.. people can check it manually if they run with docker-compose13:09
*** ykarel|afk has quit IRC13:10
quiquellsshnaidm: I think is a nice thing to have13:11
quiquellsshnaidm: you change manuale etc_zuul and do docker-compose up reproducer13:11
sshnaidmquiquell, maybe, but it freezes the startup currently13:11
sshnaidmquiquell, you can just docker-compose up and be happy :) like I do it all the time13:12
quiquellsshnaidm: but that does not make sense, want to look a little on it13:12
quiquellsshnaidm: you can do that too now13:12
sshnaidmquiquell, yeah, I mean I don't need reproducer container for that13:13
quiquellsshnaidm: let me do some investigations13:13
quiquellDamn it was beautiful with docker-compose up reproducer and finish when ready :-(13:17
quiquellsshnaidm: ok going to put back the readiness in the main playbook :-((((13:18
quiquellsshnaidm: I have to leave earlier today and will go back next week13:19
sshnaidmquiquell, ack13:19
sshnaidmquiquell, no worries, we can fix it in next patches..13:20
sshnaidmquiquell, it's an amazing work, I can now create custom roles in test1 repo and use them in tripleo jobs13:21
quiquellsshnaidm: let me just add a pre readiness time13:21
*** amoralej is now known as amoralej|lunch13:23
*** udesale has joined #oooq13:26
quiquellsshnaidm: na not working,13:29
quiquellsshnaidm: ok have to leave, you can put back the readiness like it was before in the main playbook13:29
quiquellsshnaidm: and remove the reproducer stuff13:29
quiquellsshnaidm: well let me do it13:29
quiquellsshnaidm: now is like before readiness at the main playbook13:34
sshnaidmquiquell, ok13:34
*** ykarel|afk has joined #oooq13:35
*** rlandy has joined #oooq13:36
*** ykarel|afk is now known as ykarel13:36
rlandyquiquell: hello13:38
*** dtantsur is now known as dtantsur|brb13:39
quiquellrlandy: o/13:40
quiquellrlandy: I have to drop and go back next week13:40
rlandyquiquell: hey - go back next week? you mean you're going on pto for a few days?13:41
quiquellrlandy: bank holidays tomorrow and PTO on friday13:41
rlandyquiquell: k - fine - np ... I want to work on a few other things as well13:41
rlandyquiquell: I will flesh out the cards a bit more before tomorrow's scrum13:42
rlandyquiquell: once question for you though ... wrt dlrn stuff13:42
rlandyquiquell: maybe five minute bj?13:42
rlandywill be quicker13:42
rlandyI need to know where to store local files13:43
quiquellrlandy: sure, let me fix last stuff from repro review, looks like moving readiness to docker-compose is affecting zuul start up :-((((13:43
quiquellrlandy: ok all set, your room ?13:43
quiquellsshnaidm: now it's working as before13:43
rlandysure13:43
quiquellsshnaidm, rlandy: is ready to merge13:43
sshnaidmquiquell, great13:44
quiquellsshnaidm: Still I keep the taiga task about moving readines to docker-compose I think is a nice feature13:44
quiquellrlandy: https://redhat.bluejeans.com/u/ellorent/13:45
*** agopi has quit IRC13:45
rlandyjoining13:45
quiquellrlandy: I am there in the cold13:48
sshnaidmrfolco, I'm not sure we need to remove multinode job with merging standalone, did we decide something about it?13:48
rlandyquiquell: I joined your bj13:48
quiquellrlandy: ack13:48
rlandyquiquell: but you didn't ?13:49
sshnaidmrfolco, seems like logical to do it though13:49
quiquellrlandy: was at yours13:49
rlandyah there you are13:49
*** agopi has joined #oooq13:49
rfolcosshnaidm, I think this is reasonable to do... notice I am not removing its definition... this will be done in a separate patch (deprecate multinode scenarios)13:50
sshnaidmrfolco, but multinode was voting job13:53
sshnaidmrfolco, doesn't it mean that SA 003 job should vote too?13:53
chandan_kumartosky: Hello13:54
rfolcosshnaidm, 2 options: 1) keep multinode job as is for 2 weeks and if stable, remove 2) replace it w/ standalone voting now and take the risk13:54
chandan_kumartosky: is there a way to export devstack_localrc var use_python: true var here https://review.openstack.org/#/c/622865/2/roles/generate-tempestconf-file-cloud/tasks/main.yaml@5 ?13:55
rfolcosshnaidm, lets just add standalone and remove it later in a separate patch, I can split the removal into a separate one13:57
rfolcosshnaidm, sounds good ?13:57
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario001-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, master: tripleo-quickstart-extras-gate-master-delorean-full-featureset052, tripleo-ci-centos-7-scenario009-multinode-oooq, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci- (1 more message)13:57
sshnaidmrfolco, no guts, no glory13:57
sshnaidmrfolco, big chances are never small stakes13:58
* sshnaidm searching over english dictionary13:58
sshnaidmrfolco, it will load ci a bit more, running both multinode and standalone, but I think we can try it for a week14:00
rfolcosshnaidm, ok will split the patch14:01
weshay_mtgmarios_|ruck, ssbarnea|rover morning guys..14:01
marios_|ruckweshay_mtg: swear to god was about to ping you but saw _mtg and didnt we are in pvt with ssbarnea|rover14:01
marios_|ruckdo you want to sync call/can you?14:01
weshay_mtgI can't sync atm14:02
marios_|ruckack14:02
weshay_mtgbut can chat14:02
weshay_mtgirc14:02
marios_|ruckweshay_mtg: np did the call just now (status in doc). lucky on promotions but no master phase 1 yet14:02
marios_|ruckbut da zuul queue tho14:02
marios_|ruckso we need to reset it i think again14:02
weshay_mtgmeh.. master phase 1 is not critical14:02
marios_|ruckit was like 10ish hours this morning but now like 19 woo14:02
weshay_mtg:(14:02
weshay_mtg17hr I see14:03
weshay_mtg2014:03
marios_|ruckweshay_mtg: my screen says 18.9 hour maybe i need to refresh14:03
ssbarnea|roverit seems that we have a problem with chrony sync: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22fatal%3A%20%5Bundercloud%5D%3A%20FAILED!%20%3D%3E%20%7B%5C%22changed%5C%22%3A%20true%2C%20%5C%22cmd%5C%22%3A%20%5B%5C%22chronyc%5C%22%5C%2214:03
ssbarnea|roverand there is a fix in progress14:04
ssbarnea|roverhttps://review.openstack.org/#/c/621930/14:04
*** amoralej|lunch is now known as amoralej14:04
weshay_mtgovh bs1 still is a problem14:04
ssbarnea|rovermarios_|ruck: weshay_mtg : please workflow ^14:04
marios_|ruckssbarnea|rover: ack checking it now14:05
marios_|ruckssbarnea|rover: (i saw it yesterday https://url.corp.redhat.com/scen1-ntp-fail-unrelated )14:05
* weshay_mtg looks14:06
marios_|ruckssbarnea|rover: done14:06
ssbarnea|rovernow i hope the problem was related to number of servers and not the 20s waitsync time.14:06
*** sshnaidm has quit IRC14:06
marios_|ruckweshay_mtg: so me and ssbarnea|rover brought this up last week but we didnt get to it... what do when "lets reset zuul". does it mean going and abandon all the things? or is there some other magic voodoo thing we must incant.14:07
weshay_mtgmarios_|ruck, that's a discussion I have w/ Juan, Alex, or Emilien14:07
weshay_mtgusually they make the call and do the reset14:07
marios_|ruckweshay_mtg: ok then i summon a meeting of the council14:07
marios_|ruckplease14:07
weshay_mtg:)14:08
ssbarnea|roverjoing for lunch, brb in ~15min14:08
*** saneax has quit IRC14:09
quiquellok guys drop now, see you  next week14:11
*** weshay_mtg is now known as weshay_mtg_afk14:11
*** quiquell is now known as quiquell|off14:11
mariosquiquell|off: have a good one14:11
*** sshnaidm has joined #oooq14:14
rlandyrfolco: marios_|ruck: looking at the scenario cards, there is only the update one I see left (which rfolco is on)? Have time today to work on something. what should I take>14:19
rfolcorlandy, pleaaaase14:19
rfolcothe last 2 stories14:19
rfolcono time to work on those14:19
rfolcof28 containers and non-containers14:19
mariosrlandy: o/ looking14:19
*** saneax has joined #oooq14:20
marioswell what folco said if there are outstanding stories rlandy14:20
rfolcoif you can hlep with f28 promotion, weshay_mtg_afk will not cut my neck as he asked me ensure somebody will take it14:21
rfolcorlandy, ^14:21
rlandyrfolco: ok - so perhaps we can bj for two minutes?14:22
rfolcoindeed14:22
rfolcoone sec and join my blue14:22
jfrancoassbarnea|rover: marios_|ruck: hey folks, some jobs are failing during tempest run because one of the tests can't connect to "tempest-sendmail.tripleo.org" http://logs.openstack.org/53/620953/1/check/tripleo-ci-centos-7-undercloud-containers/b6dd075/job-output.txt.gz#_2018-12-05_01_06_29_797557 is there any launchpad bug opened for it?14:22
rfolcojfrancoa, yes, there is14:23
rfolcoone sec14:23
rfolcohttps://bugs.launchpad.net/tripleo/+bug/180669914:23
openstackLaunchpad bug 1806699 in tripleo "HTTPConnectionPool(host='tempest-sendmail.tripleo.org', port=8080): Max retries exceeded with url" [Undecided,Triaged]14:23
rlandyrfolco: joined14:23
jfrancoarfolco: thanks14:23
rfolcoyw14:24
*** skramaja has quit IRC14:24
rfolcojfrancoa, this has thousands of hits in elastic-recheck, we have to fix this asap, annoying bug14:24
rfolcoarxcruz is sick today, not sure he got any fixes14:25
jfrancoarfolco: and is it consistent? (just to avoid rechecking until it will be fixed)14:25
marios_|ruckjfrancoa: its a known one sec lemme find the bug14:25
jfrancoamarios_|ruck: thanks, rfolco already sent it to me14:25
rfolcojfrancoa, I think this is not your root cause... just noise14:25
rfolcoI might be wrong14:26
marios_|ruckjfrancoa: don't think it is consistent https://bugs.launchpad.net/tripleo/+bug/180669914:26
openstackLaunchpad bug 1806699 in tripleo "HTTPConnectionPool(host='tempest-sendmail.tripleo.org', port=8080): Max retries exceeded with url" [Undecided,Triaged]14:26
marios_|ruckjfrancoa: so try a recheck have you hit it many times?14:26
marios_|ruckjfrancoa: if so add pointers in the bug please?14:26
jfrancoamarios_|ruck: I observed it in https://review.openstack.org/#/c/620953/ , and there were two rechecks in a row, but the reason for those rechecks was probably the timing out on tripleo-ci-centos-7-containers-multinode14:28
jfrancoamarios_|ruck:  I'll try to recheck, let's see how it goes14:28
chandan_kumarweshay_mtg_afk: please have a look at this card https://tree.taiga.io/project/tripleo-ci-board/us/44114:28
jfrancoarfolco: marios_|ruck: thanks to both14:29
marios_|ruckthanks jfrancoa for checking14:29
marios_|ruckjfrancoa: ++14:29
arxcruzrfolco: i'll update14:30
arxcruzrfolco: fixed the service has stopped14:32
pandarfolco: ping me when you have 5 minutes14:33
* marios_|ruck back in 1514:38
rfolcopanda, I have 514:45
rfolcoblue?14:45
pandarfolco: red14:45
rfolcopanda, you're blue now14:46
pandarfolco: (means REaDy and alREaDy there14:46
rfolcothere? my blue?14:47
pandarfolco: yes14:47
pandaI'm blue, da bo debodeda14:47
*** zul has quit IRC14:57
*** sshnaidm has quit IRC14:57
*** quiquell|off has quit IRC14:59
*** dtantsur|brb is now known as dtantsur15:13
*** jaganathan has quit IRC15:13
*** sshnaidm has joined #oooq15:18
*** saneax has quit IRC15:21
*** zul has joined #oooq15:33
rlandysshnaidm: hi - just checking in - any progress with making the nodepool images public? if not, I'll ask kforde about it15:34
sshnaidmrlandy, not that I'm aware of it15:35
sshnaidmrlandy, although I opened a ticket15:35
*** sshnaidm is now known as sshnaidm|afk15:36
agopihi rlandy can you please merge this https://code.engineering.redhat.com/gerrit/#/c/154704/215:50
agopithanks :)15:50
rlandyagopi: done15:51
agopirlandy++15:52
hubbot1agopi: rlandy's karma is now 4015:52
agopithanks :)15:52
*** dsneddon has quit IRC15:55
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035 @ https://review.openstack.org/604298, master: tripleo-quickstart-extras-gate-master-delorean-full-featureset052, tripleo-ci-centos-7-scenario009-multinode-oooq, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/560445, stable/ocata: tripleo-ci-centos-7-ovb-3ctlr_1comp- (1 more message)15:57
*** sshnaidm|afk has quit IRC16:15
*** dsneddon has joined #oooq16:19
*** sshnaidm|afk has joined #oooq16:30
toskychandan_kumar: it should be in the variables exported by zuul: http://logs.openstack.org/65/622865/3/check/python-tempestconf-tempest-devstack-py3/b386fe1/zuul-info/inventory.yaml16:34
ssbarnea|roverrfolco: can you please change your vote on https://review.openstack.org/#/c/622210/ ? --- one of the top purposes of elastic-recheck is to identify infra issues.16:41
ssbarnea|rovermarios_|ruck: 2review: https://review.openstack.org/#/c/622210/ https://review.openstack.org/#/c/622352/16:42
rfolcossbarnea|rover, logstash url does not work for me.16:42
rfolcossbarnea|rover, I am ok with the query itself if you fix url16:43
marios_|ruckssbarnea|rover: ack but /me almost out16:44
marios_|rucklooking16:44
marios_|ruckssbarnea|rover: about that http connection one... wondering if we should make it more general like today i saw this https://url.corp.redhat.com/scen4-connectionpool-docker-WTF16:45
ssbarnea|roverrfolco: fixed! I was sure I did it last time, is correct now.16:46
marios_|ruckssbarnea|rover: commnted16:46
*** udesale has quit IRC16:46
*** udesale has joined #oooq16:47
ssbarnea|rovermarios_|ruck: we cannot treat generic http errors the same way as they are almost always caused by totally different causes. they need to be specific per system.16:47
marios_|ruckssbarnea|rover: ack16:47
marios_|ruckssbarnea|rover: k16:47
marios_|ruckhttp://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22%5Bprimary%5D%20Waiting%20for%20logger%5C%22%20AND%20tags:%5C%22console%5C%22 still gives me no hits16:47
marios_|ruckrfolco: ^16:47
rfolcolast 7days gave me 12 hits16:48
ssbarnea|rovermarios_|ruck: select 7 days and *WAIT*16:48
marios_|ruckssbarnea|rover: heh k16:48
marios_|ruckssbarnea|rover: but i'll save that joy for th emorning16:48
marios_|rucki have to go16:48
ssbarnea|roveralways select 7 days and wait. the URL never includes the timespan and timespan is always reset to 15mins.16:48
ssbarnea|roverwhich is pitty as most of the time we need opossite16:49
marios_|ruckssbarnea|rover: queue at 13 hrs now so at least stopped growing...16:49
*** bogdando has quit IRC16:54
rlandypanda: hi - is there anything you want to discuss re: user stories for reproducer work or are you ok with what we have done?16:56
rfolcomarios_|ruck, still there?17:03
marios_|ruck rfolco depends :D17:03
marios_|ruckrfolco: sup17:04
rfolcoquick question...17:04
rfolcohttps://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/standalone-upgrade/templates/standalone.sh.j2#L1-L917:04
marios_|ruckrfolco: yup17:04
rfolcothe equivalent to this for update would be ?17:04
marios_|ruckrfolco: i gues re-running the deploy17:04
marios_|ruckrfolco: so whatever the openstack tripleo deploy command is for the standalone?17:04
rfolcomarios_|ruck, cool, will try to ru-run the deploy script17:05
rfolcothanks marios_|ruck17:05
marios_|ruckrfolco: hmm hold on17:05
rfolco:-/17:05
marios_|ruckrfolco: just thinking a bit17:05
marios_|ruckrfolco: yeah so it depends what the job witll do... will you have a new openstack tripleo update command?17:07
marios_|ruckrfolco: i think there is no such thing because you just rerun the deployment17:07
marios_|ruckrfolco: otherwise you'll need to add the tripleo update command like17:07
rfolcomarios_|ruck, this is multinode update... http://logs.openstack.org/99/618899/3/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates/58e03c6/logs/undercloud/home/zuul/overcloud_update_run_Controller.log.txt.gz17:08
rfolcoso overcloud update ?17:08
marios_|ruckrfolco: right but that is running th eupdate workflow17:08
marios_|ruckrfolco: it is different17:08
marios_|ruckrfolco: you are talking about standalone update17:08
marios_|ruckwhich is just about exercising the update_tasks17:08
marios_|rucknot the minor update workflow17:08
rfolcomarios_|ruck, I created a copy of https://github.com/openstack/tripleo-quickstart-extras/tree/master/roles/standalone-upgrade17:09
marios_|ruckrfolco: so for standalone you want to invoke just the update tasks, in the same way that the standalone upgrade job is running just the upgrade tasks17:09
rfolcothen I wanted to change only the last part, calling the deploy script again17:09
marios_|ruckrfolco: can we talk about this tomorrow though?17:10
marios_|ruckrfolco: cos i have to go now17:10
rfolcomarios_|ruck, sure, thanks man17:10
*** rlandy is now known as rlandy|brb17:10
marios_|rucknp. basically yes do what chem does for the upgrade job but the caveat is you want the update tasks not the upgrade tasks17:10
marios_|ruckand ignore the current update job cos that is valid, but completely different to what you want to do17:11
*** dsneddon has quit IRC17:11
marios_|ruck(that tests the actual update workflow)17:11
rfolcounderstood17:11
rfolcowill use the standalone-upgrade as base and change to reflect the update tasks17:11
rfolcomaybe looking for update tags17:11
chemrfolco: marios_|ruck so don't go that route yet, we have to refactor the standalone upgrade role to just "merge" it in the standalone role as code duplication is massive17:12
chemrfolco: I think I've got my main escalation behind me so that I can move forward on this from tomorrow on17:13
rfolcochem, ok, so I'll just be here for ci support and zuul questions if any17:14
chemrfolco: thanks, I've that https://bugs.launchpad.net/tripleo/+bug/1805121 to track the merge process.17:15
openstackLaunchpad bug 1805121 in tripleo "Standalone upgrade and deployment duplicate a lot of code." [Medium,Triaged] - Assigned to Sofer Athlan-Guyot (sofer-athlan-guyot)17:15
marios_|ruckchem: rfolco ack will catchup tomorrow folks thanks17:16
chemmarios_|ruck: see ya17:16
rfolcochem, I see the duplication. 90% of standalone-upgrade can be a common role and the remaining 10% will be specific to update/upgrade17:16
*** radez is now known as radez_afk17:16
chemrfolco: yeah, and it would be easy to refactor into one,  Ithink I have some code preparing that somewhere.  I'll push tomorrow.17:17
marios_|ruckchem: makes sense re the duplication17:17
marios_|ruckchem: i remember we discussed at the time it was mostly copy/paste from the deploy17:18
marios_|ruckchem: so plan is to add conditionals for deploy/upgrade?17:18
marios_|ruckin the bash template with j2?17:18
pandarlandy|brb: you have time to chat ?17:18
marios_|ruckchem: but really, i am missing cardio right now so actually leaving please add me to reviews and rfolco we can discuss more tomorrow and scrum17:19
marios_|ruckbai17:19
chemmarios_|ruck: the idea I had was to add a parameter to the renderer like which takes "update" or "create" and adjust the j2 file accordingly17:19
rfolcochem, count on me for the refactoring and then for the update job, I want to help as I spent some time yesterday and today looking at this :)17:19
rfolcochem, thanks for working on this17:20
chemmarios_|ruck: and then add include tasks from main based on the featureset variable17:20
*** dsneddon has joined #oooq17:20
chemrfolco: well, this is useful for upgrade :)17:20
chem... and I love helping humanity out :)17:21
chemoki, time to go too.17:21
*** zul has quit IRC17:23
*** rlandy|brb is now known as rlandy17:42
rlandypanda: yes - in 5 mins?17:42
pandaok17:43
rlandypanda: k - ready to talk when you are17:48
ssbarnea|rovermarios_|ruck: can you please help me double check something?17:48
ssbarnea|roverhttps://github.com/openstack-infra/elastic-recheck/blob/master/queries/1721093.yaml -- i have the impression that this kind of queries are broken/unreliable.17:48
pandarlandy: my bj ?17:49
rlandysure joining17:49
rlandypanda: joined - can you hear me?17:50
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035 @ https://review.openstack.org/604298, master: tripleo-quickstart-extras-gate-master-delorean-full-featureset052, tripleo-ci-centos-7-scenario009-multinode-oooq, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/560445, stable/ocata: tripleo-ci-centos-7-ovb-3ctlr_1comp- (1 more message)17:57
*** derekh has quit IRC18:03
*** jfrancoa has quit IRC18:27
*** ccamacho has quit IRC18:40
*** ccamacho has joined #oooq18:50
*** gkadam has quit IRC18:57
*** amoralej is now known as amoralej|off18:58
*** ccamacho has quit IRC19:00
*** holser_ has quit IRC19:10
*** ykarel has quit IRC19:16
*** radez_afk is now known as radez19:46
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035 @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario009-multinode-oooq, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-standalone, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/560445, stable/ocata: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001 @  (1 more message)19:57
*** jaosorior has joined #oooq20:03
*** dtantsur is now known as dtantsur|afk20:06
*** jtomasek_ has quit IRC21:05
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035 @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario009-multinode-oooq, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-standalone, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/560445, stable/ocata: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001 @  (1 more message)21:57
*** udesale has quit IRC22:25
*** rlandy is now known as rlandy|bbl23:25
*** ade_lee has joined #oooq23:40
ade_leehey all - trying to do a oooq on master on rhel 7.6 -- and ran into this issue -- http://paste.openstack.org/show/736735/23:41
ade_leeany idea how to fix/ get around it?23:41
ade_leethat is master-tripleo-ci ..23:43
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario001-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario009-multinode-oooq, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-standalone, tripleo-ci-fedora-28-standalone @  (1 more message)23:57

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!