| *** sshnaidm|rover is now known as sshnaidm|afk | 00:56 | |
| hubbot | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-scenario001-multinode-oooq-container, tripleo-ci-centos-7-scenario002-multinode-oooq-container @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ (1 more message) | 00:58 |
|---|---|---|
| *** ykarel has joined #oooq | 02:35 | |
| *** skramaja has joined #oooq | 02:55 | |
| hubbot | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-scenario001-multinode-oooq-container, tripleo-ci-centos-7-scenario002-multinode-oooq-container @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ (1 more message) | 02:58 |
| *** agopi has quit IRC | 03:02 | |
| *** ykarel has quit IRC | 03:29 | |
| *** saneax has quit IRC | 03:34 | |
| *** udesale has joined #oooq | 03:44 | |
| *** ykarel has joined #oooq | 03:47 | |
| *** ykarel_ has joined #oooq | 04:04 | |
| *** ykarel has quit IRC | 04:04 | |
| *** ykarel_ has quit IRC | 04:12 | |
| *** ykarel_ has joined #oooq | 04:38 | |
| *** holser_ has joined #oooq | 04:45 | |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-scenario001-multinode-oooq-container, tripleo-ci-centos-7-scenario002-multinode-oooq-container, tripleo-ci- (1 more message) | 04:58 |
| *** ratailor has joined #oooq | 05:09 | |
| *** ccamacho has quit IRC | 05:14 | |
| *** bogdando has joined #oooq | 05:23 | |
| *** jbadiapa has quit IRC | 05:24 | |
| *** yolanda has joined #oooq | 05:30 | |
| *** quiquell|off is now known as quiquell | 05:31 | |
| *** ykarel_ is now known as ykarel | 05:43 | |
| *** holser_ has quit IRC | 05:50 | |
| chkumar|ruck | %gatestatus | 05:54 |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-scenario001-multinode-oooq-container, tripleo-ci-centos-7-scenario002-multinode-oooq-container, tripleo-ci- (1 more message) | 05:54 |
| *** ccamacho has joined #oooq | 06:19 | |
| *** saneax has joined #oooq | 06:26 | |
| *** sanjayu_ has joined #oooq | 06:30 | |
| *** saneax has quit IRC | 06:33 | |
| *** quiquell is now known as quiquell|bbl | 06:35 | |
| *** jbadiapa has joined #oooq | 06:43 | |
| *** matbu has joined #oooq | 06:49 | |
| *** jfrancoa has joined #oooq | 06:51 | |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-scenario001-multinode-oooq-container, tripleo-ci-centos-7-scenario002-multinode-oooq-container, legacy-tripleo-ci-centos-7 (1 more message) | 06:58 |
| *** gkadam has joined #oooq | 07:11 | |
| *** holser_ has joined #oooq | 07:12 | |
| *** sshnaidm|afk has quit IRC | 07:14 | |
| *** quiquell|bbl is now known as quiquell | 07:16 | |
| *** sshnaidm|afk has joined #oooq | 07:21 | |
| quiquell | sshnaidm|afk, chkumar|ruck: added zuulv3 RDO job builds to RR dashboard | 07:25 |
| chkumar|ruck | quiquell: cool, thanks :-) | 07:25 |
| quiquell | chkumar|ruck: Let me know if you are missing something | 07:26 |
| chkumar|ruck | quiquell: http://38.145.34.131:3000/d/2kHMNHvik/exploration?orgId=1 | 07:27 |
| chkumar|ruck | quiquell: I wanted to know how many a job failed in a day or two | 07:28 |
| chkumar|ruck | quiquell: under job frequency tab count is too much | 07:28 |
| quiquell | chkumar|ruck, let me remove the time range from tables (it was not suppose to be there) | 07:28 |
| quiquell | so you can play with global | 07:28 |
| chkumar|ruck | quiquell: one more improvement under job list result success should be green not RED | 07:29 |
| quiquell | chkumar|ruck: noted | 07:29 |
| quiquell | ok, reload again | 07:29 |
| quiquell | click on influxdb filter + | 07:29 |
| quiquell | select passed | 07:29 |
| quiquell | and value False | 07:29 |
| *** amoralej|off is now known as amoralej | 07:29 | |
| quiquell | then in the right corner "Last 7 days" if you click it you can select 3 days | 07:30 |
| quiquell | or whatever range you think of | 07:30 |
| quiquell | chkumar|ruck: http://38.145.34.131:3000/d/2kHMNHvik/exploration?orgId=1&from=now-3d&to=now&var-influxdb_filter=passed%7C%3D%7CFalse | 07:30 |
| quiquell | The url has the filter | 07:31 |
| chkumar|ruck | quiquell: yup now it looks good | 07:31 |
| chkumar|ruck | quiquell: thanks :-) | 07:31 |
| quiquell | chkumar|ruck: You can put the filter you want and the range you want | 07:31 |
| quiquell | If you click in the job name you go to the logs | 07:31 |
| quiquell | If you click in the Patch you go to the review | 07:31 |
| *** tosky has joined #oooq | 07:32 | |
| chkumar|ruck | that looks nice now | 07:32 |
| *** ratailor_ has joined #oooq | 07:33 | |
| *** ratailor has quit IRC | 07:35 | |
| chkumar|ruck | %gatestatus | 07:36 |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-scenario001-multinode-oooq-container, tripleo-ci-centos-7-scenario002-multinode-oooq-container, legacy-tripleo-ci- (1 more message) | 07:36 |
| chkumar|ruck | quiquell: we need one more fix http://38.145.34.131:3000/d/pgdr_WVmk/cockpit?orgId=1&from=now%2Fd&to=now%2Fd | 07:36 |
| chkumar|ruck | under last jobs tab, in jOb section, if we click on job name, it does not show log url | 07:37 |
| chkumar|ruck | the log url is getting appended with graphana url in firefox | 07:37 |
| quiquell | chkumar|ruck: ack | 07:40 |
| *** jaganathan has joined #oooq | 07:44 | |
| *** florianf has joined #oooq | 07:47 | |
| quiquell | chkumar|ruck: ok now I see the legacy- jobs at the dashboard | 08:05 |
| *** ykarel is now known as ykarel|lunch | 08:08 | |
| *** sshnaidm|afk is now known as sshnaidm|rover | 08:16 | |
| *** Goneri has joined #oooq | 08:17 | |
| quiquell | chkumar|ruck: Fixed | 08:22 |
| quiquell | chkumar|ruck: There is something bad at RDO zuuls.. with ovb jobs | 08:26 |
| chkumar|ruck | quiquell: thanks :-) | 08:29 |
| chkumar|ruck | sshnaidm|rover: I am filing a master bug to track the overcloud deploy timeout | 08:29 |
| sshnaidm|rover | chkumar|ruck, is it same kind of timeouts? | 08:30 |
| chkumar|ruck | sshnaidm|rover: https://review.rdoproject.org/etherpad/p/chkumar-ruck-rover-sprint16-notes | 08:30 |
| chkumar|ruck | check line: 18 to 25 | 08:30 |
| chkumar|ruck | sshnaidm|rover: till early morning scenario2 was failing | 08:31 |
| sshnaidm|rover | chkumar|ruck, in line 18 there is a job http://logs.openstack.org/45/560445/78/check/tripleo-ci-centos-7-scenario003-multinode-oooq-container/6195e6e/job-output.txt.gz#_2018-07-05_02_50_08_414152 | 08:33 |
| quiquell | sshnaidm|rover: Can you merge the RR monitor stuff ? | 08:33 |
| sshnaidm|rover | chkumar|ruck, it doesn't have overcloud timeout | 08:33 |
| sshnaidm|rover | chkumar|ruck, the job is just cut because it doesn't have time anymore, bc of long tasks: http://logs.openstack.org/45/560445/78/check/tripleo-ci-centos-7-scenario003-multinode-oooq-container/6195e6e/job-output.txt.gz#_2018-07-05_02_50_50_168231 | 08:34 |
| sshnaidm|rover | chkumar|ruck, so it's related to timeouts we have in general | 08:34 |
| sshnaidm|rover | chkumar|ruck, but overcloud itself passed well | 08:34 |
| chkumar|ruck | sshnaidm|rover: do we need a bug for that? | 08:34 |
| sshnaidm|rover | quiquell, yeah, in which order? | 08:35 |
| quiquell | sshnaidm|rover: Parenting order | 08:35 |
| sshnaidm|rover | chkumar|ruck, I think we have | 08:35 |
| quiquell | sshnaidm|rover: They are all done one after another as a big family | 08:35 |
| sshnaidm|rover | chkumar|ruck, it's a bug for general timeouts in job | 08:37 |
| sshnaidm|rover | chkumar|ruck, next line is the same: http://logs.openstack.org/45/560445/78/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates/04f7b4a/job-output.txt.gz#_2018-07-05_02_51_36_900718 | 08:38 |
| sshnaidm|rover | chkumar|ruck, and next too: http://logs.openstack.org/45/560445/78/check/tripleo-ci-centos-7-scenario002-multinode-oooq-container/2c3d93a/job-output.txt.gz#_2018-07-05_02_54_35_163029 | 08:39 |
| chkumar|ruck | sshnaidm|rover: we donot have a master bug to track timeout issue | 08:43 |
| quiquell | chkumar|ruck: I think we have | 08:44 |
| quiquell | https://bugs.launchpad.net/tripleo/+bug/1776796 | 08:44 |
| openstack | Launchpad bug 1776796 in tripleo "tripleo gate jobs timing out, duplicate containers pulls a possible cause" [Critical,Triaged] - Assigned to Quique Llorente (quiquell) | 08:44 |
| chkumar|ruck | quiquell: I am adding the findings there. | 08:46 |
| quiquell | chkumar|ruck: Yep is the place | 08:46 |
| sshnaidm|rover | quiquell, but I think I saw ara error in other places and it still worked, maybe just red herring | 08:48 |
| quiquell | sshnaidm|rover: Then the problem I have is the timeout | 08:48 |
| sshnaidm|rover | quiquell, it's timeout of collection logs | 08:48 |
| sshnaidm|rover | quiquell, I bet some infra host stuck | 08:49 |
| quiquell | Damn... | 08:49 |
| sshnaidm|rover | as usual | 08:49 |
| quiquell | But all the jobs have fail | 08:49 |
| sshnaidm|rover | :D | 08:49 |
| chkumar|ruck | sshnaidm|rover: quiquell after 11:00 jobs are coming greener | 08:49 |
| chkumar|ruck | in grafite | 08:49 |
| chkumar|ruck | sorry | 08:49 |
| chkumar|ruck | grafana | 08:49 |
| sshnaidm|rover | btw, there were github problems tonight, so please check if it's not that: https://status.github.com/messages | 08:50 |
| sshnaidm|rover | quiquell, maybe we can add this to grafana too ^^ :) | 08:50 |
| chkumar|ruck | sshnaidm|rover: yup | 08:51 |
| chkumar|ruck | sshnaidm|rover: periodic pike jobs got impacted due to github issues | 08:51 |
| quiquell | sshnaidm|rover: Good one | 08:51 |
| quiquell | sshnaidm|rover: Next to openstack infra issues | 08:52 |
| sshnaidm|rover | quiquell, and this too maybe: http://ds.iris.edu/seismon/eventlist/index.phtml :D | 08:56 |
| chkumar|ruck | sshnaidm|rover: http://logs.openstack.org/45/560445/78/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/a37efa0/logs/undercloud/var/lib/mistral/5654fc3f-9375-439a-b53e-25df469b290f/ansible.log.txt.gz | 08:56 |
| quiquell | sshnaidm|rover: ^ DNS ? have it at my mind | 08:56 |
| quiquell | sshnaidm|rover: hubbot server is a good place for the RR monitor ? | 08:56 |
| sshnaidm|rover | quiquell, yep | 08:57 |
| hubbot | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-scenario001-multinode-oooq-container, tripleo-ci-centos-7-scenario002-multinode-oooq-container, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates @ (1 more message) | 08:58 |
| quiquell | sshnaidm|rover: Going to gate the RR monitor scripts, with telegraf --test :-) | 08:59 |
| sshnaidm|rover | quiquell, cool! | 08:59 |
| quiquell | we need to convert that into infrastructure-as-code | 08:59 |
| chkumar|ruck | sshnaidm|rover: need some pointer on above job keystone init is not happening | 08:59 |
| *** ratailor_ has quit IRC | 09:02 | |
| *** ykarel|lunch is now known as ykarel | 09:08 | |
| chkumar|ruck | sshnaidm|rover: openstack check jobs are red due to github timepout issue | 09:09 |
| sshnaidm|rover | chkumar|ruck, when? | 09:10 |
| ykarel | chkumar|ruck, openstack-check/ or periodic pike? | 09:12 |
| ykarel | btw pike github failures were between 05:27 UTC and 05:32 UTC | 09:13 |
| chkumar|ruck | ykarel: https://review.rdoproject.org/zuul3/status.html | 09:15 |
| chkumar|ruck | ykarel: check running check jobs | 09:15 |
| chkumar|ruck | *openstack-check | 09:15 |
| chkumar|ruck | sshnaidm|rover: ^^ | 09:15 |
| * ykarel looks | 09:15 | |
| ykarel | chkumar|ruck, i can see different failures, but can't find github ones, can you share link? | 09:19 |
| chkumar|ruck | ykarel: http://logs.rdoproject.org/72/576772/3/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-ocata-branch/17c995c/job-output.txt.gz | 09:19 |
| chkumar|ruck | ykarel: others are due to overcloud deploy timing out | 09:20 |
| ykarel | chkumar|ruck, no different issues | 09:20 |
| chkumar|ruck | ykarel: one with vxlan issue | 09:21 |
| ykarel | hmm | 09:21 |
| ykarel | so timeout was at 6:45 UTC, and as per github status page it's operational since 7:10 UTC | 09:23 |
| ykarel | so we should not see the timeout issue now | 09:24 |
| chkumar|ruck | http://logs.rdoproject.org/63/578863/4/openstack-check/legacy-tripleo-ci-centos-7-container-to-container-upgrades-master/b8ca6cb/job-output.txt.gz | 09:24 |
| chkumar|ruck | ykarel: yes | 09:25 |
| chkumar|ruck | sshnaidm|rover: those were related to the patch itself | 09:33 |
| *** dtantsur|afk is now known as dtantsur | 09:33 | |
| *** Goneri has quit IRC | 09:40 | |
| *** Goneri has joined #oooq | 09:42 | |
| quiquell | sshnaidm|rover: seismon, good for summer and chand choosing destination | 09:47 |
| *** Goneri has quit IRC | 09:54 | |
| sshnaidm|rover | chkumar|ruck, last one seems like a problem with zuulv3 transition.. | 09:56 |
| sshnaidm|rover | chkumar|ruck, does it happen more? | 09:56 |
| *** Goneri has joined #oooq | 09:58 | |
| sshnaidm|rover | quiquell, need to fix log links in "last jobs" table | 09:58 |
| quiquell | sshnaidm|rover: It's RDO zuulv3 problem with OVB | 09:59 |
| quiquell | sshnaidm|rover: check this https://softwarefactory-project.io/zuul/api/tenant/rdoproject.org/builds?change=568602 | 09:59 |
| quiquell | job_url is wrong | 09:59 |
| sshnaidm|rover | quiquell, yeah, I see it happens when "node failure" | 10:00 |
| sshnaidm|rover | quiquell, actually there are no logs.. | 10:00 |
| quiquell | sshnaidm|rover: Yep, internal IRC is for this ghings ? or better open a ticket ? | 10:01 |
| sshnaidm|rover | quiquell, idk, maybe #rdo, but pabelanger is not here atm | 10:02 |
| sshnaidm|rover | quiquell, because he is doing the migration | 10:02 |
| quiquell | sshnaidm|rover: Already ask there | 10:02 |
| quiquell | sshnaidm|rover: They are super busy now | 10:02 |
| chkumar|ruck | sshnaidm|rover: nope | 10:02 |
| sshnaidm|rover | chkumar|ruck, then let's ignore it, we have enough problems.. | 10:03 |
| quiquell | sshnaidm|rover: You can filter them with the influxdb_filter | 10:03 |
| sshnaidm|rover | quiquell, I wish! :D | 10:04 |
| quiquell | Something like *ovb* and status failure | 10:04 |
| sshnaidm|rover | quiquell, I need ad-hoc vars in my brain | 10:04 |
| quiquell | sshnaidm|rover:Â Hehe | 10:04 |
| quiquell | sshnaidm|rover: Working in a review for that | 10:05 |
| chkumar|ruck | sshnaidm|rover: I am not able to set status to traiged and importance in the bug | 10:05 |
| chkumar|ruck | sshnaidm|rover: one more bug https://bugs.launchpad.net/tripleo/+bug/1780224 | 10:05 |
| openstack | Launchpad bug 1780224 in tripleo "ERROR configuring keystone_init_tasks in tripleo-ci-centos-7-scenario001-multinode-oooq-container job" [Undecided,New] | 10:05 |
| quiquell | sshnaidm|rover: Without ovb http://38.145.34.131:3000/d/pgdr_WVmk/cockpit?orgId=1&var-launchpad_tags=alert&var-promotion_names=current-tripleo&var-promotion_names=current-tripleo-rdo&var-promotion_names=current-tripleo-rdo-testing&var-releases=master&var-releases=queens&var-releases=pike&var-releases=ocata&var-influxdb_filter=job_name%7C!~%7C%2Fovb%2F | 10:06 |
| chkumar|ruck | sshnaidm|rover: it is not showing | 10:06 |
| sshnaidm|rover | chkumar|ruck, I see, seems like need to add you to group, lemme look | 10:06 |
| sshnaidm|rover | chkumar|ruck, add you here: https://launchpad.net/~tripleo | 10:08 |
| chkumar|ruck | sshnaidm|rover: joined the team | 10:12 |
| chkumar|ruck | sshnaidm|rover: this one https://bugs.launchpad.net/tripleo/+bug/1780183 is happening in another jobs also | 10:13 |
| openstack | Launchpad bug 1780183 in tripleo "Overcloud failed to Deploy in tripleo-ci-centos-7-nonha-multinode-oooq" [High,Triaged] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm) | 10:13 |
| *** jbadiapa has quit IRC | 10:15 | |
| *** bogdando has quit IRC | 10:16 | |
| sshnaidm|rover | chkumar|ruck, yep, looking in it | 10:16 |
| quiquell | panda: You there ? | 10:26 |
| amoralej | one more voe +2+w for https://review.openstack.org/#/c/579888/ ? | 10:27 |
| amoralej | i have a review that needs it | 10:27 |
| *** bogdando has joined #oooq | 10:45 | |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-scenario001-multinode-oooq-container, tripleo-ci-centos-7-scenario002-multinode-oooq-container, legacy-tripleo-ci-centos-7 (1 more message) | 10:58 |
| chkumar|ruck | sshnaidm|rover: https://review.openstack.org/#/c/579888/ | 11:00 |
| chkumar|ruck | need +w on this one | 11:00 |
| chkumar|ruck | sshnaidm|rover: for phase 2 queens promotions what is blocking | 11:02 |
| chkumar|ruck | where I can help? | 11:02 |
| *** amoralej is now known as amoralej|lunch | 11:04 | |
| sshnaidm|rover | quiquell, something is wrong with your patches parenting: https://review.rdoproject.org/r/#/c/14595/4 maybe just collect all json changes in one patch and scripts in other? | 11:20 |
| sshnaidm|rover | chkumar|ruck, I think you opened a bug for it yesterday, right? network problems in job | 11:21 |
| chkumar|ruck | sshnaidm|rover: https://bugs.launchpad.net/tripleo/+bug/1780091 | 11:21 |
| openstack | Launchpad bug 1780091 in tripleo "containerized undercloud deployment failed on periodic jobs" [Critical,Triaged] | 11:21 |
| chkumar|ruck | sshnaidm|rover: weshay was saying to check network configuration in this one | 11:22 |
| chkumar|ruck | sshnaidm|rover: but from last comment, someone has also hitted it | 11:22 |
| quiquell | sshnaidm|rover: will try to rebase the pile of sh... | 11:23 |
| quiquell | sshnaidm|rover: Fixed, try it now | 11:24 |
| quiquell | sshnaidm|rover: The last parent was already merged rebase was neede | 11:24 |
| chkumar|ruck | sshnaidm|rover: All bugs are listed here https://review.rdoproject.org/etherpad/p/chkumar-ruck-rover-sprint16-notes from line 10 -15 | 11:25 |
| sshnaidm|rover | chkumar|ruck, this etherpad is growing fast, better to have all bugs together in the top | 11:32 |
| *** sshnaidm|rover is now known as sshnaidm|rov|lnc | 11:33 | |
| panda | quiquell: now I am | 11:34 |
| quiquell | panda: I am ok now | 11:35 |
| quiquell | panda: sprint16 stuff, but let's talk in the meeting | 11:35 |
| quiquell | panda: Have add some stuff to the doc | 11:36 |
| *** skramaja_ has joined #oooq | 11:58 | |
| *** skramaja has quit IRC | 11:59 | |
| *** panda is now known as panda|lunch | 12:11 | |
| *** sshnaidm|rov|lnc is now known as sshnaidm|rover | 12:15 | |
| quiquell | sshnaidm|rover: Do you agree on moving RR monitoring to hubbot server ? | 12:16 |
| sshnaidm|rover | quiquell, moving what exactly? | 12:18 |
| quiquell | sshnaidm|rover: The stuff we have now telegraf + influxdb + grafana | 12:18 |
| quiquell | sshnaidm|rover: Later on delegate on infra | 12:19 |
| *** jbadiapa has joined #oooq | 12:20 | |
| *** trown|outtypewww is now known as trown | 12:20 | |
| sshnaidm|rover | quiquell, again problem: https://review.rdoproject.org/r/#/c/14610/2 | 12:21 |
| sshnaidm|rover | quiquell, I'd like to have a different host for that | 12:21 |
| *** rlandy has joined #oooq | 12:23 | |
| quiquell | sshnaidm|rover: rebased, repo is changing a lot | 12:25 |
| quiquell | sshnaidm|rover: try now | 12:27 |
| *** quiquell is now known as quiquell|lunch | 12:27 | |
| sshnaidm|rover | quiquell|lunch, ok, so 2 remains in conflict: https://review.rdoproject.org/r/#/q/owner:%22F%25C3%25A9lix+Enrique+Llorente+Pastora%22+status:open | 12:34 |
| quiquell|lunch | sshnaidm|rover: rebased | 12:38 |
| rlandy | panda|lunch: wanted to touch base about scenario007 and 008 tempest tests | 12:40 |
| sshnaidm|rover | chkumar|ruck, the board filtered out from ovb: http://38.145.34.131:3000/d/pgdr_WVmk/cockpit?orgId=1&var-launchpad_tags=alert&var-promotion_names=current-tripleo&var-promotion_names=current-tripleo-rdo&var-promotion_names=current-tripleo-rdo-testing&var-releases=master&var-releases=queens&var-releases=pike&var-releases=ocata&var-influxdb_filter=job_name%7C!~%7C%2Fovb%2F | 12:50 |
| sshnaidm|rover | chkumar|ruck, looks much better | 12:50 |
| sshnaidm|rover | quiquell|lunch, it's weird that "cloud !~ /rdo/" doesn't do it, I'd expect it filter out everything in rdo cloud.. | 12:51 |
| quiquell|lunch | sshnaidm|rover: there some builds with empty cloud, have to take a lok | 12:51 |
| *** quiquell|lunch is now known as quiquell | 12:52 | |
| quiquell | sshnaidm|rover: If you checkout the exploration there are some with cloud and region empty | 12:52 |
| sshnaidm|rover | quiquell, hmm, need to debug it | 12:52 |
| quiquell | We can filte by empty to get them | 12:53 |
| hubbot | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: legacy-tripleo-ci-centos-7-container-to-container-upgrades-master @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224 | 12:58 |
| panda|lunch | rlandy: after the planning , ok ? | 12:59 |
| rlandy | panda|lunch: ack | 12:59 |
| panda|lunch | rlandy: I created a specific card, and removed the migration from the rest of the code | 12:59 |
| *** panda|lunch is now known as panda | 12:59 | |
| rlandy | ok | 12:59 |
| sshnaidm|rover | panda, is the meeting now? | 13:00 |
| panda | sshnaidm|rover: yes | 13:01 |
| weshay | https://etherpad.openstack.org/p/tripleo-ci-squad-meeting | 13:02 |
| *** agopi has joined #oooq | 13:04 | |
| *** florianf has quit IRC | 13:12 | |
| ykarel | arxcruz, hi | 13:14 |
| arxcruz | ykarel: hey wanted to talk with you actually :) | 13:14 |
| arxcruz | ykarel: whats up ? | 13:14 |
| ykarel | arxcruz, i was looking at https://bugs.launchpad.net/tripleo/+bug/1779628 | 13:15 |
| openstack | Launchpad bug 1779628 in tripleo "Disable ssl validation in tempest on tripleo quickstart" [Medium,In progress] - Assigned to Arx Cruz (arxcruz) | 13:15 |
| arxcruz | ykarel: yes ? | 13:15 |
| ykarel | but looks like issue is happening with tempest container, | 13:15 |
| ykarel | shouldn't that be fixed | 13:15 |
| ykarel | without container it seems to pass | 13:15 |
| arxcruz | hmmmm | 13:15 |
| arxcruz | ykarel: i only saw this behavior on featureset035 and another one that I don't remember now | 13:16 |
| ykarel | and currently the patch is in gate | 13:16 |
| ykarel | arxcruz, i checked because fs035 on promotion job is not facing this issue | 13:16 |
| arxcruz | ykarel: oh, right... | 13:16 |
| ykarel | not sure why on check job tempest is running in container while in promotion job not | 13:17 |
| arxcruz | ykarel: that's because we are setting on tempestconf to false right now | 13:17 |
| arxcruz | ykarel: however, the default is true, so we are overwriting it | 13:17 |
| arxcruz | once this patch get merged, we will remove this from tempestconf | 13:17 |
| arxcruz | that was my agreement with tosky :) | 13:17 |
| tosky | yep | 13:17 |
| tosky | thanks :) | 13:18 |
| arxcruz | we shouldn't overwrite by default tempest options, unless is explicited required by the user | 13:18 |
| sshnaidm|rover | quiquell, all merged \o/ | 13:18 |
| * quiquell crying | 13:18 | |
| tosky | it would be interesting to know hy it does not work on a tripleo-quickstart deployment too, if some other configuration keys are needed | 13:18 |
| ykarel | arxcruz, ack. But why issues is not seen in fs035 promotion job | 13:18 |
| arxcruz | tosky: i need to investigate, basically we just need to point the ca file | 13:18 |
| tosky | oh, makes sense | 13:19 |
| arxcruz | ykarel: because tempestconf is overwriting it automagically | 13:19 |
| arxcruz | shit, now i don't know what i was doing... | 13:20 |
| ykarel | i still don't get :( | 13:20 |
| ykarel | how tempestconf is overriting in promotion job but not in check job, | 13:21 |
| ykarel | it should be same at both places | 13:21 |
| arxcruz | ykarel: can you point me the log from the promotion ? | 13:22 |
| arxcruz | i think i have an idea why | 13:22 |
| ykarel | arxcruz, https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master/a220a1c/undercloud/home/jenkins/tempest.log.txt.gz | 13:22 |
| ykarel | the only difference i can see is tempest_format | 13:23 |
| ykarel | but can be other as well | 13:23 |
| arxcruz | ykarel: hmmm so, found the problem | 13:27 |
| ykarel | and what's that? | 13:28 |
| arxcruz | ykarel: running in container, we loose the export 'PYTHONWARNINGS=ignore:Certificate has no, ignore:A true SSLContext object is not available' | 13:28 |
| arxcruz | while running locally, we don't loose | 13:28 |
| *** amoralej|lunch is now known as amoralej | 13:28 | |
| arxcruz | ykarel: https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master/a220a1c/undercloud/home/jenkins/tempest.log.txt.gz#_2018-07-05_10_00_39 | 13:28 |
| arxcruz | so containers will fail, because this is set outside the container | 13:29 |
| ykarel | arxcruz, so this needs to be fixed na | 13:30 |
| ykarel | as this would be faced when running ipv6 job with tempest container | 13:32 |
| arxcruz | ykarel: disable ssl will fix it | 13:35 |
| arxcruz | ykarel: ohhh, i see your point... | 13:37 |
| arxcruz | tosky: ykarel so, the problem is in tempestconf, we need to disable ssl directly on tempestconf because, it's failing due the fact tempestconf is being executed in the container, and don't have the PYTHONWARNINGS variable set | 13:38 |
| arxcruz | so when request tries to access https, it fails | 13:38 |
| arxcruz | outside container isn't affected be cause PYTHONWARNINGS is set properly | 13:39 |
| ykarel | arxcruz, so can't PYTHONWARNINGS be set in container to make both container and host behave the same | 13:40 |
| *** ykarel is now known as ykarel|away | 13:40 | |
| arxcruz | ykarel: probably yes, i don't know how, but it should be possible let me google it | 13:40 |
| *** skramaja_ has quit IRC | 13:43 | |
| *** ykarel|away has quit IRC | 13:45 | |
| *** florianf has joined #oooq | 13:46 | |
| *** quiquell is now known as quiquell|mtg | 13:53 | |
| *** agopi has quit IRC | 13:58 | |
| chkumar|ruck | sshnaidm|rover: looks cool | 14:00 |
| *** ykarel has joined #oooq | 14:08 | |
| ykarel | arxcruz, ok | 14:15 |
| *** agopi has joined #oooq | 14:25 | |
| *** agopi_ has joined #oooq | 14:27 | |
| *** agopi has quit IRC | 14:30 | |
| *** ccamacho1 has joined #oooq | 14:44 | |
| *** ccamacho1 has quit IRC | 14:44 | |
| *** ccamacho1 has joined #oooq | 14:45 | |
| *** ccamacho has quit IRC | 14:45 | |
| *** tcw1 has joined #oooq | 14:53 | |
| *** tcw has quit IRC | 14:57 | |
| *** tcw1 is now known as tcw | 14:57 | |
| hubbot | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: legacy-tripleo-ci-centos-7-container-to-container-upgrades-master @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224 | 14:58 |
| rfolco | sorry marios did not get what you said | 15:01 |
| *** ykarel is now known as ykarel|away | 15:03 | |
| arxcruz | ykarel|away: did you saw the patch? | 15:07 |
| ykarel|away | arxcruz, yes, looks okk, but want to see the job result | 15:08 |
| arxcruz | ykarel|away: i think it will fail, because the includes on ansible | 15:08 |
| ykarel|away | also good to add the update to bug as this is related | 15:08 |
| arxcruz | but we'll see | 15:08 |
| ykarel|away | okk | 15:09 |
| weshay | chkumar|ruck, ping | 15:12 |
| sshnaidm|rover | weshay, I think he finished for today | 15:20 |
| sshnaidm|rover | weshay, he sent mail with updates | 15:20 |
| panda | rlandy: sorry I was muted, I'll ping you for the scenarios | 15:25 |
| rlandy | panda: sure - whenever you are ready | 15:25 |
| rlandy | we need to close that out | 15:25 |
| *** sanjayu_ has quit IRC | 15:25 | |
| *** quiquell|mtg is now known as quiquell|off | 15:26 | |
| sshnaidm|rover | fyi, my patch is rebased on pandas patch: https://review.openstack.org/#/c/578456 | 15:27 |
| marios | rfolco: o/ sorry man bluejeans was over irssi window :) | 15:33 |
| panda | rlandy: we can chat now if you want | 15:43 |
| rlandy | panda: sure | 15:44 |
| panda | rlandy: I'm in my channel | 15:45 |
| *** agopi_ is now known as agopi | 15:45 | |
| chkumar|ruck | weshay: pong | 15:58 |
| weshay | chkumar|ruck, howdy... going to write up a card for you handle while ruck/rover | 16:00 |
| weshay | chkumar|ruck, we need a fs21 job that runs tempest w/o skip list.. arxcruz has that in progress | 16:00 |
| chkumar|ruck | weshay: sure | 16:00 |
| weshay | chkumar|ruck, but will also need to have the skip list cleaned out for the upstream releases | 16:00 |
| *** bogdando has quit IRC | 16:00 | |
| weshay | chkumar|ruck, make sense? | 16:00 |
| chkumar|ruck | weshay: yup make sense | 16:00 |
| weshay | chkumar|ruck, thanks chkumar|ruck | 16:01 |
| weshay | chkumar|ruck, don't worry about sending email status updates | 16:02 |
| weshay | chkumar|ruck, I appareciate.. but I think that is too much overhead | 16:02 |
| chkumar|ruck | weshay: sigi and I not synced today so sent email | 16:02 |
| weshay | k k | 16:02 |
| weshay | :) | 16:02 |
| weshay | just wanted you to know it's not required or my expectation for you to email updates on a daily basis | 16:03 |
| *** udesale has quit IRC | 16:04 | |
| *** yolanda_ has joined #oooq | 16:13 | |
| *** yolanda has quit IRC | 16:17 | |
| arxcruz | not my fault | 16:18 |
| arxcruz | :) | 16:18 |
| *** jfrancoa has quit IRC | 16:20 | |
| weshay | arxcruz, :) | 16:22 |
| arxcruz | weshay: fyi, i'm working on stackviz bug | 16:22 |
| arxcruz | ykarel|away: when you have some time, i would like to talk with you about packaging ;) | 16:23 |
| weshay | rlandy, can you reserve about 15minutes in your day to review https://review.openstack.org/#/c/565740/ w/ me | 16:26 |
| rlandy | weshay: sure - after I finish with panda | 16:26 |
| ykarel|away | arxcruz, sure, can do that tomorrow, just ping me :) | 16:26 |
| arxcruz | ykarel|away: yup | 16:26 |
| *** agopi is now known as agopi|lunch | 16:30 | |
| *** panda is now known as panda|off | 16:47 | |
| *** yolanda__ has joined #oooq | 16:53 | |
| *** ykarel|away has quit IRC | 16:54 | |
| *** yolanda_ has quit IRC | 16:56 | |
| hubbot | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: legacy-tripleo-ci-centos-7-container-to-container-upgrades-master @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224 | 16:58 |
| *** amoralej is now known as amoralej|off | 17:03 | |
| *** ccamacho1 has quit IRC | 17:03 | |
| *** bogdando has joined #oooq | 17:05 | |
| *** trown is now known as trown|lunch | 17:07 | |
| *** agopi|lunch is now known as agopi | 17:08 | |
| *** yolanda_ has joined #oooq | 17:12 | |
| *** yolanda__ has quit IRC | 17:15 | |
| *** rlandy is now known as rlandy|brb | 17:21 | |
| *** dtantsur is now known as dtantsur|afk | 17:26 | |
| *** bogdando has quit IRC | 17:35 | |
| *** rlandy|brb is now known as rlandy | 17:40 | |
| *** sshnaidm|rover has quit IRC | 17:43 | |
| *** holser_ has quit IRC | 17:43 | |
| *** sshnaidm|rover has joined #oooq | 17:52 | |
| weshay | rlandy, you have a sec? | 17:55 |
| *** trown|lunch is now known as trown | 17:59 | |
| rlandy | weshay; sure, ping me when you are want to do the review | 18:08 |
| weshay | rlandy, ready | 18:08 |
| rlandy | k - bj? | 18:08 |
| weshay | ya | 18:08 |
| *** tcw has quit IRC | 18:10 | |
| *** tcw has joined #oooq | 18:12 | |
| *** yolanda__ has joined #oooq | 18:13 | |
| *** florianf has quit IRC | 18:14 | |
| *** yolanda_ has quit IRC | 18:15 | |
| *** ccamacho has joined #oooq | 18:23 | |
| *** yolanda_ has joined #oooq | 18:35 | |
| *** yolanda__ has quit IRC | 18:39 | |
| *** yolanda__ has joined #oooq | 18:39 | |
| *** yolanda_ has quit IRC | 18:42 | |
| *** sshnaidm|rover is now known as sshnaidm|off | 18:46 | |
| *** ccamacho has quit IRC | 18:54 | |
| hubbot | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: legacy-tripleo-ci-centos-7-container-to-container-upgrades-master @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224 | 18:58 |
| rfolco | aren't we going to waste resources for undercloud jobs with this nodeset in our base job ? https://github.com/openstack-infra/tripleo-ci/blob/master/zuul.d/base.yaml#L19 | 19:18 |
| rfolco | I guess we need one base job for multinode and one for singlenode, just like devstack has | 19:19 |
| *** gkadam has quit IRC | 19:32 | |
| *** sanjay__u has joined #oooq | 19:56 | |
| weshay | rlandy, if this passes --extra-vars directly.. I think a file may work | 19:59 |
| weshay | https://review.openstack.org/#/c/546474/ | 19:59 |
| * rlandy looks | 20:00 | |
| weshay | rfolco, that was the plan all along wasn't it? | 20:00 |
| weshay | to have one base job for the singlenode undercloud jobs | 20:00 |
| weshay | and one for the multinode jobs | 20:00 |
| rfolco | weshay, I suspect nodepool is wasting one slave for undercloud singlenode jobs with two-node nodeset | 20:01 |
| rfolco | weshay, devstack has one base for multinode and one for singlenode, nodesets according to each situation | 20:01 |
| weshay | rfolco, 1. why suspect.. go check, 2. why are we using two node config for the undercloud jobs | 20:01 |
| weshay | rfolco, it's not how the old jobs work | 20:02 |
| weshay | FAK | 20:02 |
| weshay | rfolco, come one man | 20:02 |
| weshay | on | 20:02 |
| weshay | rfolco, the old parents for the tripleo jobs did that correctly | 20:02 |
| rfolco | weshay, there is no way to check this without looking at nodepool | 20:03 |
| weshay | hrm | 20:03 |
| *** Goneri has quit IRC | 20:03 | |
| weshay | rfolco, this is what it used to look like https://github.com/openstack-infra/tripleo-ci/blob/6e7955597158fa3738c1f5812888d50c2d8cf2d3/zuul.d/base.yaml | 20:04 |
| weshay | rfolco, open a bug and fix the nodes please | 20:05 |
| rfolco | weshay, exactly. The confusion was: we parent to multinode job for both, but we need abstract layer for each | 20:05 |
| rfolco | weshay, working on it | 20:05 |
| weshay | rfolco, I think this is really just tech debt | 20:06 |
| weshay | https://review.openstack.org/#/c/578432/ | 20:06 |
| weshay | rfolco, in that if we got multinode to work, single comes for free | 20:07 |
| weshay | however I am suprised the team did not get that done | 20:07 |
| rfolco | nodepool looks at nodeset and gives how many slaves you want, so for singlenode you have to ask just one | 20:08 |
| weshay | rfolco, https://review.openstack.org/#/c/578456/13/zuul.d/multinode-jobs.yaml | 20:11 |
| weshay | rfolco, you don't have to worry about it yet | 20:11 |
| rfolco | weshay, I don't get your point | 20:12 |
| weshay | rfolco, that change is not used yet | 20:12 |
| rfolco | weshay, needs to be fixed before we merge, yes | 20:13 |
| weshay | rfolco, you better vote on https://review.openstack.org/#/c/578456/ | 20:14 |
| rfolco | weshay, done | 20:16 |
| *** ccamacho has joined #oooq | 20:19 | |
| *** hubbot has quit IRC | 20:28 | |
| *** dmellado has quit IRC | 20:28 | |
| rlandy | rdo down? | 20:44 |
| rlandy | guess so | 20:44 |
| arxcruz | weshay: https://review.openstack.org/#/c/580361/ fix stackviz | 21:09 |
| arxcruz | weshay: https://review.openstack.org/#/c/580423/ swift jobs re-enabled on scenario002 | 21:15 |
| arxcruz | s/swift jobs/swift tests | 21:15 |
| weshay | rfolco, rlandy ya.. it just went down | 21:36 |
| weshay | thanks arxcruz | 21:36 |
| *** holser_ has joined #oooq | 21:36 | |
| rlandy | rfolco: thanks - saw saga on rhos-ops | 21:37 |
| rlandy | so much for debug :( | 21:37 |
| *** hubbot has joined #oooq | 21:38 | |
| weshay | arxcruz, I thought 21 would be easier to remember https://review.openstack.org/#/c/580480/ | 21:39 |
| arxcruz | weshay: haha, yes! :D\ | 21:40 |
| *** holser_ has quit IRC | 22:01 | |
| *** holser_ has joined #oooq | 22:06 | |
| *** holser_ has quit IRC | 22:40 | |
| hubbot | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: legacy-tripleo-ci-centos-7-container-to-container-upgrades-master @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224 | 22:59 |
| *** tosky has quit IRC | 23:01 | |
| *** yolanda_ has joined #oooq | 23:07 | |
| *** yolanda__ has quit IRC | 23:08 | |
| *** yolanda__ has joined #oooq | 23:11 | |
| *** yolanda_ has quit IRC | 23:13 | |
| *** rlandy has quit IRC | 23:25 | |
| *** yolanda_ has joined #oooq | 23:40 | |
| *** yolanda__ has quit IRC | 23:42 | |
| *** agopi is now known as agopi|off | 23:51 | |
| *** agopi|off has quit IRC | 23:56 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!