| *** rfolco has joined #oooq | 00:35 | |
| *** rfolco has quit IRC | 00:39 | |
| *** rfolco has joined #oooq | 00:48 | |
| *** rfolco has quit IRC | 00:53 | |
| *** d0ugal has quit IRC | 01:05 | |
| *** d0ugal has joined #oooq | 01:07 | |
| *** ysandeep has joined #oooq | 01:21 | |
| *** epoojad1 has joined #oooq | 04:39 | |
| *** holser has joined #oooq | 04:57 | |
| *** epoojad1 has quit IRC | 05:02 | |
| *** epoojad1 has joined #oooq | 05:03 | |
| *** bhagyashris has joined #oooq | 05:14 | |
| *** udesale has joined #oooq | 05:15 | |
| *** ykarel has joined #oooq | 05:22 | |
| *** holser has quit IRC | 05:33 | |
| *** skramaja has joined #oooq | 05:34 | |
| *** ratailor has joined #oooq | 05:42 | |
| *** marios has joined #oooq | 06:05 | |
| *** surpatil has joined #oooq | 06:23 | |
| *** soniya29 has joined #oooq | 06:26 | |
| *** epoojad1 has quit IRC | 06:43 | |
| *** epoojad1 has joined #oooq | 06:51 | |
| *** ykarel is now known as ykarel|lunch | 07:35 | |
| *** surpatil has quit IRC | 08:02 | |
| *** tosky has joined #oooq | 08:23 | |
| *** surpatil has joined #oooq | 08:24 | |
| *** ykarel|lunch is now known as ykarel | 08:27 | |
| *** SurajPatil has joined #oooq | 08:40 | |
| *** surpatil has quit IRC | 08:43 | |
| *** fmount has quit IRC | 08:52 | |
| *** fmount has joined #oooq | 08:55 | |
| *** ratailor has quit IRC | 09:01 | |
| *** ratailor has joined #oooq | 09:02 | |
| *** surpatil has joined #oooq | 09:10 | |
| *** SurajPatil has quit IRC | 09:12 | |
| *** ratailor_ has joined #oooq | 09:20 | |
| *** ratailor has quit IRC | 09:23 | |
| *** soniya29 has quit IRC | 09:27 | |
| *** bhagyashris has quit IRC | 09:27 | |
| *** bhagyashris has joined #oooq | 09:28 | |
| *** derekh has joined #oooq | 09:35 | |
| marios | errand biab | 09:57 |
|---|---|---|
| *** marios has quit IRC | 10:02 | |
| *** bhagyashris has quit IRC | 10:03 | |
| *** bhagyashris has joined #oooq | 10:03 | |
| *** SurajPatil has joined #oooq | 10:08 | |
| *** saneax has joined #oooq | 10:09 | |
| *** surpatil has quit IRC | 10:11 | |
| *** sanjayu_ has joined #oooq | 10:12 | |
| *** saneax has quit IRC | 10:15 | |
| *** surpatil has joined #oooq | 10:19 | |
| *** SurajPatil has quit IRC | 10:21 | |
| *** epoojad1 has quit IRC | 10:23 | |
| *** ysandeep has quit IRC | 10:53 | |
| *** fmount has quit IRC | 11:02 | |
| *** fmount- has joined #oooq | 11:03 | |
| *** surpatil has quit IRC | 11:05 | |
| *** udesale has quit IRC | 11:15 | |
| *** marios has joined #oooq | 11:19 | |
| *** fmount- is now known as fmount | 11:19 | |
| *** bhagyashris has quit IRC | 11:33 | |
| *** bhagyashris has joined #oooq | 11:34 | |
| *** sshnaidm|afk is now known as sshnaidm | 11:43 | |
| *** rfolco has joined #oooq | 11:57 | |
| *** ratailor_ has quit IRC | 12:00 | |
| *** fmount has quit IRC | 12:33 | |
| *** fmount has joined #oooq | 12:40 | |
| *** bhagyashris has quit IRC | 12:42 | |
| *** fmount has quit IRC | 12:47 | |
| *** fmount has joined #oooq | 12:50 | |
| *** rlandy has joined #oooq | 13:00 | |
| *** fmount has quit IRC | 13:02 | |
| rlandy | rfolco: no meeting? | 13:03 |
| rfolco | rlandy, I don't think so. No sprint. | 13:03 |
| rfolco | rlandy, Need you to merge a patch soon, if avail | 13:04 |
| rlandy | marios: hi | 13:05 |
| marios | rlandy: rfolco: o/ i was there but dropped | 13:06 |
| marios | was alone | 13:06 |
| rlandy | marios: I also joined and dropped | 13:06 |
| marios | rlandy: must have just missed you then | 13:06 |
| marios | rlandy: want to sync i did a quick round of ruck rover earlier | 13:06 |
| rlandy | marios: ack - I did some over the weekend we have legit fs020 issue | 13:07 |
| marios | rlandy: but not realy much to say | 13:07 |
| marios | rlandy: ah i see | 13:07 |
| rlandy | http://logs.rdoproject.org/62/21862/10/check/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-queens/3cbe8dc/logs/tempest.html.gz | 13:07 |
| marios | rlandy: cos queens/rocky both neeed fs20 for promote and i posted the testproject for it | 13:07 |
| marios | rlandy: master lots of failed and posted testproject failing master " https://review.rdoproject.org/r/24299 tesrproject missing rocky/queens fs20 https://review.rdoproject.org/r/24300 | 13:07 |
| marios | rlandy: stein promoting that is really stable lately | 13:07 |
| rlandy | marios: somehow rocky said it was 0 days | 13:08 |
| rlandy | so I didn;t worry about that | 13:08 |
| marios | rlandy: i was almost scared to check the promotions status this morning after promoter gone rogue on friday | 13:08 |
| marios | ;) | 13:08 |
| rlandy | but there a legit failures here | 13:08 |
| rlandy | it was fine after the restart | 13:08 |
| marios | hm so are we sure about fs 20 https://review.rdoproject.org/r/#/c/24300/ can be another sanity check anyway | 13:08 |
| rlandy | yes | 13:09 |
| marios | rlandy: i can dig more there if you like once it reports and file a bug ... maybe we can just skip for now to unblock | 13:09 |
| rlandy | reran over the weekend | 13:09 |
| marios | rlandy: i c | 13:09 |
| rlandy | yeah - I am going to log a bug and skip | 13:09 |
| marios | rlandy: ok cool then ping me to vote etc | 13:09 |
| rlandy | just checking if it's the same test on rocky and queens | 13:09 |
| marios | rlandy: latest-released train was running last i checked a while ago | 13:09 |
| rlandy | ok | 13:10 |
| rlandy | rekicked master 039 | 13:10 |
| marios | rlandy: i have the master jobs there https://review.rdoproject.org/r/24299 too fyi | 13:10 |
| marios | rlandy: i added those testproject to etherpad and tidied up to today | 13:10 |
| marios | ah i see you wrote there already | 13:10 |
| marios | me recoffee | 13:10 |
| rlandy | there are no logs on the rocky failure | 13:11 |
| *** ykarel has quit IRC | 13:13 | |
| *** surpatil has joined #oooq | 13:18 | |
| rlandy | 2019-12-23 13:13:16.376160 | TASK [remove-build-sshkey : Remove the build SSH key from all nodes] | 13:24 |
| rlandy | 2019-12-23 13:15:20.921820 | primary | ERROR | 13:24 |
| rlandy | 2019-12-23 13:15:20.922174 | primary | { | 13:24 |
| rlandy | 2019-12-23 13:15:20.922231 | primary | "msg": "Data could not be sent to remote host \"38.145.34.193\". Make sure this host can be reached over ssh: ssh: connect to host 38.145.34.193 port 22: Connection timed out\r\n", | 13:24 |
| rlandy | 2019-12-23 13:15:20.922277 | primary | "unreachable": true | 13:24 |
| *** surpatil has quit IRC | 13:24 | |
| rlandy | 2019-12-23 13:15:20.922317 | primary | } | 13:24 |
| rlandy | 2019-12-23 13:15:20.941441 | | 13:24 |
| rlandy | weird ^^ only hitting rhel nodes | 13:24 |
| rfolco | marios, rlandy: please https://review.rdoproject.org/r/#/c/24301/ | 13:27 |
| rlandy | ugh - we have node issues | 13:30 |
| rlandy | checking tenant | 13:30 |
| marios | rlandy: 503 or sthing else? | 13:31 |
| marios | rlandy: cos could just be overlapping pipes | 13:31 |
| marios | rfolco: looking sec | 13:31 |
| rlandy | marios: can find ips, can't find nodes etc. | 13:31 |
| rlandy | stacks are clean and not many of them | 13:32 |
| rlandy | we are dropped access to dlrn | 13:32 |
| rfolco | mjturek, ^ | 13:34 |
| rfolco | this explains why | 13:34 |
| *** fmount has joined #oooq | 13:36 | |
| marios | https://review.rdoproject.org/r/#/c/24301/ rfolco rlandy so merge? | 13:37 |
| rfolco | please marios | 13:37 |
| rfolco | all I can do is the manual test and i did | 13:37 |
| rfolco | this will fix component job and should not break any code path from normal/integration pipeline | 13:38 |
| rlandy | marios: not changing the skip list for rocky as we don't have any logs with the failures | 13:41 |
| rlandy | prepping the queens review | 13:42 |
| *** bogdando has joined #oooq | 13:44 | |
| marios | rlandy: ack ... master run was pretty bad at https://review.rdoproject.org/r/#/c/24299/ fails/retry_limit | 13:44 |
| *** epoojad1 has joined #oooq | 13:44 | |
| *** bogdando has quit IRC | 13:46 | |
| rfolco | marios, rlandy vpn working for you? I installed freeotp and scanned the qrcode from mojo page, it is generating code but not logging in. Any clues ? | 13:47 |
| rlandy | works for me | 13:48 |
| marios | rfolco: still on for me | 13:52 |
| rfolco | must be missing some step :( | 13:52 |
| *** epoojad1 has quit IRC | 13:52 | |
| marios | rfolco: what server u trying /me ams | 13:52 |
| marios | ams2 in fact | 13:53 |
| marios | rfolco: ping me pvt if you want details | 13:53 |
| rfolco | marios, tried sao paulo, brno and raleigh | 13:53 |
| rfolco | marios, yes please, pvt | 13:53 |
| rlandy | marios: we are not going to get any cloud/infra support today - will just do what we can | 13:58 |
| * rlandy needs to book india trip stuff | 13:59 | |
| marios | ack rlandy | 13:59 |
| marios | rlandy: yeah i got a reply from the visa inviation service application they will process in jan so fyi | 13:59 |
| rlandy | marios: yeah - you are way ahead of me - just trying to book flight and hotel | 14:00 |
| marios | rlandy: well i didn't book either of those yet started with the visa :D | 14:00 |
| marios | but did research flights | 14:00 |
| marios | rlandy: looks like i arrive sun and leave late fridy | 14:01 |
| marios | rlandy: but not booked that | 14:01 |
| rlandy | probably a shorter flight for you | 14:01 |
| marios | rlandy: well yeah compared to you for sure. still have like 2 connections though fwiw | 14:01 |
| rlandy | I have to leave saturday night to arrive sunday night | 14:01 |
| rlandy | I only have one connection - dircet flight from NYC to New Delhi | 14:02 |
| rlandy | 19 hours | 14:02 |
| rlandy | worse for weshay, he's west of me | 14:03 |
| migi | rlandy: hey | 14:11 |
| rlandy | migi: hi there | 14:12 |
| rlandy | happy holidays to you | 14:12 |
| migi | rlandy: you too ! | 14:12 |
| rlandy | thanks - what's up? | 14:12 |
| migi | rlandy: I have updated this small change, but now I wonder if nova change needs to be merged https://code.engineering.redhat.com/gerrit/#/c/187618/ | 14:12 |
| marios | did it work rfolco | 14:12 |
| rfolco | marios, the token or the patch | 14:13 |
| rfolco | none | 14:13 |
| rfolco | none | 14:13 |
| rfolco | none | 14:13 |
| rlandy | looking | 14:13 |
| migi | rlandy: also second question the downstream osp17 jobs should be based on the rhel8.1 / not rhel8.0 right ? | 14:13 |
| rlandy | migi: rhos-16 and rhos-17 jobs both should be rebased on 8.1 afaik | 14:14 |
| migi | rlandy: cool | 14:15 |
| rlandy | migi: https://code.engineering.redhat.com/gerrit/#/c/187618/2/zuul.d/projects.yaml ... I am not so familiar with the live migration test ... | 14:16 |
| rlandy | from what you have built, | 14:16 |
| rlandy | the standalone test would not run unless live-migration passed | 14:16 |
| rlandy | is that correct? | 14:16 |
| rlandy | also, should the live-migration test be dependent on the hash we pin? | 14:17 |
| rlandy | iiuc, it should | 14:17 |
| rlandy | otherwise we are not sure we are testing a consistent hash | 14:17 |
| rlandy | if so, | 14:17 |
| rlandy | the live migration test should be on the same spacing as the standalone test | 14:18 |
| migi | rlandy: you are right, so the live-migration and standalone should be in parallel but repo jobs should go into separate pipeline | 14:18 |
| rlandy | and use tripleo-ci-testing | 14:18 |
| rlandy | migi: can you define repo jobs? | 14:18 |
| migi | rlandy: unit/functional/pep8 | 14:19 |
| rlandy | migi: do they run against rpm? | 14:20 |
| migi | rlandy: which requires repository, so dark blue diagram: https://docs.google.com/drawings/d/1KcKh-WzRNrLAVF70w1aXtBDlaVp_TyVdJbxxd4sMSi4/edit | 14:20 |
| migi | rlandy: nope but they require rpm's | 14:20 |
| rlandy | then diff pipeline yes - if they are blue box | 14:20 |
| rlandy | this is strictly green box stuff | 14:20 |
| migi | yeah | 14:20 |
| migi | rlandy: do we want to make blue box passing info to green box? so if blue do not pass then green isn't passing or very separate pipeline | 14:21 |
| migi | rlandy: at the end both results should be taken into final passed/failed | 14:21 |
| rlandy | migi: as I understand, the green box runs when there is an rpm built | 14:23 |
| rlandy | ideally, you would like to stop the rpm build if the blue box test fail | 14:23 |
| migi | rlandy: correct and blue box runs to promote code before rpm built | 14:23 |
| rlandy | but that is not the case atm | 14:23 |
| rlandy | ^^ correct? | 14:23 |
| migi | very much | 14:24 |
| rlandy | ok so since you can't stop the rpm build on your tests | 14:24 |
| rlandy | and the only way to stop the promotion is through dlrn api reporting | 14:24 |
| rlandy | ie: if the rpm builds, the green box will run | 14:25 |
| rlandy | we can't stop thaht | 14:25 |
| rlandy | but we can stop the hash been promoted by adding your tests to the dlrn criteria for promotion | 14:25 |
| rlandy | and/or | 14:25 |
| migi | rlandy: we can't stop but we can add some final pass/fail | 14:25 |
| rlandy | rerun your tests in the green box | 14:25 |
| migi | yep | 14:25 |
| rlandy | correct | 14:25 |
| migi | rlandy: so I need to add them to promote component then | 14:25 |
| rlandy | the hash will be built | 14:26 |
| migi | similar to live-migration | 14:26 |
| rlandy | but not promoted | 14:26 |
| rlandy | not necessarily | 14:26 |
| rlandy | let's chat - I'll explain | 14:26 |
| migi | rlandy: aren't we chatting ? :) | 14:26 |
| rlandy | migi: k, I need to share resources - will post here | 14:27 |
| rlandy | https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/dlrnapi_promoter/config/CentOS-7/component/master.yaml | 14:27 |
| rlandy | example | 14:28 |
| rlandy | of upstream criteria | 14:28 |
| rlandy | just because we runs something in the pipeline does not mean it is part of the promotion crietria | 14:28 |
| rlandy | and just because something runs out side the pipeline doesn;t mean it can't be included in criteria | 14:28 |
| rlandy | we can build the promotion criteria so that they include the tests from your blue box | 14:29 |
| rlandy | as long as they report to the dlrn api with the same hash | 14:29 |
| rlandy | you don't need to duplicate the tests | 14:29 |
| rlandy | migi: does that make sense> | 14:29 |
| rlandy | ? | 14:30 |
| migi | rlandy: and where is downstream version of drnapi_promoter ? | 14:30 |
| migi | rlandy: in the file you gave there is periodic-tripleo-ci-centos-7-standalone-compute-master | 14:31 |
| rlandy | migi: I have not built it yet | 14:31 |
| migi | rlandy: and in downstream I am missing it | 14:31 |
| rlandy | but I can show you where the dlrn reports are | 14:31 |
| migi | rlandy: ok | 14:31 |
| migi | rlandy: so where the downstream standalone is now reporting ? | 14:32 |
| rlandy | migi: rfolco is building the promote job and he is still working on it | 14:32 |
| rlandy | migi: to the downstream dlrn | 14:32 |
| * rlandy gets | 14:32 | |
| migi | rlandy: a-h ok so not yet there | 14:32 |
| rlandy | migi: we can build the criteria file any time now | 14:33 |
| migi | rlandy: yep | 14:33 |
| migi | rlandy: pls show me the repo | 14:33 |
| rlandy | migi: https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?pipeline=openstack-component-compute&pipeline=openstack-component-keystone | 14:34 |
| rlandy | see ... | 14:34 |
| rlandy | dlrnapi --url http://10.0.76.142/api-redhat-osp17 --username tripleo-ci-internal repo-promote --commit-hash f8ac4007628e60f35236c127f3de35d3b9ea0346 --distro-hash c9bfa013c3a219d28abaea7b544163dd819e908e --promote-name tripleo-ci-testing | 14:34 |
| rlandy | dlrnapi --url http://10.0.76.142/api-redhat-osp17 | 14:34 |
| rlandy | ^^ downstream dlrn | 14:34 |
| rlandy | that promotes the hash to tripleo-ci-testing | 14:35 |
| rlandy | migi; your jobs will need ... | 14:35 |
| rlandy | 2019-12-23 03:47:14.670745 | TASK [dlrn-report : Report results to dlrn for the tested hash] | 14:36 |
| rlandy | 2019-12-23 03:47:19.711470 | Timeout exception waiting for the logger. Please check connectivity to [10.0.79.142:19885] | 14:36 |
| rlandy | 2019-12-23 03:47:30.200887 | primary | ok: Runtime: 0:00:14.934053 | 14:36 |
| *** skramaja has quit IRC | 14:36 | |
| rlandy | you can just include the following playbooks ... | 14:36 |
| migi | rlandy: see in ci-config | 14:37 |
| rlandy | http://git.app.eng.bos.redhat.com/git/openstack/tripleo-ci-internal-config.git/tree/zuul.d/dlrn-jobs.yaml#n35 | 14:37 |
| migi | rlandy: playbooks/dlrn/pre-dlrn.yaml | 14:38 |
| rlandy | the pre-run and post-run | 14:38 |
| migi | yep | 14:38 |
| migi | rlandy: and secrets as vars | 14:38 |
| migi | rlandy: secrets: | 14:38 |
| migi | - dlrnapi | 14:38 |
| rlandy | then we override the default ci-confg file | 14:38 |
| rlandy | to use the downstream one we define | 14:38 |
| rlandy | promotion runs independently in a separate pipeline | 14:39 |
| rlandy | and just read dlrn-api reports | 14:39 |
| migi | rlandy: ok now the tricky part, the hash of the downstream repo is not the one from promoted as it's rebased version so without really much thinking maybe you have an idea what's the best way to match those | 14:41 |
| migi | rlandy: the only one I have is to add extra logic to get one from spec file which currently is defined as: | 14:42 |
| migi | rlandy: %global dlrn_nvr nova-20.0.2.dev29-0.20191210093649.3c62efc | 14:42 |
| rlandy | migi: we pull the hash from http://10.0.76.142/rhel8-osp17/component/compute/ | 14:42 |
| rlandy | is that the hash you are expecting | 14:43 |
| migi | rlandy: no, something isn't correct | 14:47 |
| migi | in the downstream part, upstream is ok | 14:47 |
| rlandy | migi: can you point to what is wrong | 14:48 |
| rlandy | we use that has to add repo | 14:48 |
| rlandy | and update containers with that repo | 14:48 |
| migi | rlandy: maybe it's ok, but I think it may not be, let me check the component job | 14:48 |
| rlandy | ok | 14:49 |
| migi | rlandy: and repo is build from ? | 14:49 |
| migi | rlandy: I am unsure if component rpm's are build correctly | 14:50 |
| rlandy | https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-component-compute/code.engineering.redhat.com/openstack/tripleo-ci-internal-jobs/master/periodic-tripleo-ci-rhel-8-standalone-rhos-17-compute-rhos-16/a7a5304/logs/undercloud/etc/yum.repos.d/compute-component.repo.txt.gz | 14:51 |
| rlandy | migi: ^^ example | 14:51 |
| rlandy | and we got a legit ovb error | 14:57 |
| migi | rlandy: ok so it's correct I need to modify those jobs to get the Source0 | 14:57 |
| rlandy | migi: k - good to hear | 14:58 |
| migi | rlandy: yeah so far I was working out of git repo and now I need to add option to work directly from *spec file to pull and patch source | 14:58 |
| rlandy | k | 14:59 |
| migi | rlandy: but the downstream part for component test in standalone is correct | 14:59 |
| rlandy | :) | 14:59 |
| weshay | 0/ | 15:03 |
| rlandy | weshay: hi there | 15:03 |
| rlandy | thought you were on vaca | 15:03 |
| weshay | I'm around :) | 15:03 |
| rlandy | 2019-12-23 13:28:09 | Ansible timed out at 5705 seconds. | 15:06 |
| rlandy | shoot - we're back to that | 15:06 |
| weshay | fs01? | 15:06 |
| rlandy | weshay: yeah master is looking less that happy | 15:07 |
| rlandy | https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master | 15:07 |
| rlandy | 2019-12-23 11:05:03 | Ansible timed out at 5791 seconds. | 15:08 |
| rlandy | 2019-12-23 09:45:20 | ok: [overcloud-novacompute-0] => { | 15:08 |
| rlandy | 2019-12-23 11:05:03 | Overcloud configuration failed. | 15:08 |
| rlandy | 2019-12-23 11:05:03 | "censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", | 15:08 |
| weshay | rlandy, just in periodic? /me looking at https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001 | 15:08 |
| weshay | master is f.. | 15:08 |
| weshay | nvrmind | 15:08 |
| rlandy | weshay: no ... 2019-12-23 13:15:44 | Ansible timed out at 5665 seconds. | 15:09 |
| rlandy | in check as well | 15:09 |
| weshay | k.. sec helping folco | 15:09 |
| rlandy | 2019-12-23 12:02:14 | } | 15:09 |
| rlandy | 2019-12-23 13:15:41 | changed: [overcloud-controller-0] => (item=None) => { | 15:09 |
| rlandy | 2019-12-23 13:15:41 | "censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", | 15:09 |
| rlandy | somewhere we lose an hour | 15:10 |
| rlandy | weshay: no worries - just debugging out loud | 15:10 |
| rlandy | 2019-12-23 12:02:14 | TASK [Write kolla config json files] ******************************************* | 15:22 |
| rlandy | 2019-12-23 13:15:41 | changed: [overcloud-controller-0] => (item=None) => { | 15:23 |
| rlandy | ^^ that's the time waster | 15:23 |
| weshay | so this started on the 21st | 15:24 |
| weshay | rlandy, are the jobs you are looking at timing out on | 15:34 |
| weshay | 2019-12-23 13:15:44 | TASK [Run puppet host configuration for step 1] ******************************** | 15:34 |
| weshay | ? | 15:34 |
| rlandy | weshay: ack | 15:34 |
| rlandy | see the we spin there for an hour | 15:34 |
| rlandy | note the time | 15:35 |
| rlandy | http://logs.rdoproject.org/openstack-regular/opendev.org/openstack/tripleo-ci/master/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/215472a/logs/undercloud/var/log/extra/errors.txt.txt.gz | 15:36 |
| rlandy | possibly last error ^^ | 15:36 |
| rfolco | rlandy, marios weshay https://review.rdoproject.org/r/24307 | 15:39 |
| rlandy | Write kolla config json filestimeout is there sometimes | 15:44 |
| weshay | rlandy, what file? | 15:45 |
| rlandy | openvswitch-2.12.0-1.el7.x86_64 version is the same in passing and failing so it's not that | 15:45 |
| rlandy | just the deploy lo | 15:45 |
| rlandy | g | 15:45 |
| rfolco | marios, quick one please so I can leave it running while I grab food | 15:50 |
| rfolco | https://review.rdoproject.org/r/#/c/24307/ | 15:50 |
| marios | rfolco: sorry lookign just finished fighting with visa application system | 15:51 |
| rfolco | ah | 15:51 |
| rfolco | that is stressful | 15:51 |
| rfolco | np :) | 15:51 |
| rfolco | oops linters | 15:51 |
| weshay | rlandy, I think once the deployment tries to switch from root -> tripleo-admin it can't loging | 15:52 |
| weshay | log in on the compute | 15:53 |
| *** ykarel has joined #oooq | 15:53 | |
| rlandy | weshay: creating LP bug - sec | 15:53 |
| weshay | k | 15:53 |
| weshay | Dec 23 12:02:37 overcloud-novacompute-0 puppet-user[20225]: Puppet: 5.5.10 | 15:55 |
| weshay | Dec 23 12:02:37 overcloud-novacompute-0 sudo[20219]: pam_unix(sudo:session): session closed for user root | 15:55 |
| weshay | Dec 23 12:28:01 overcloud-novacompute-0 anacron[18273]: Job `cron.daily' started | 15:55 |
| weshay | http://logs.rdoproject.org/openstack-regular/opendev.org/openstack/tripleo-ci/master/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/215472a/logs/overcloud-novacompute-0/var/log/journal.txt.gz | 15:55 |
| weshay | maybe? Dec 23 11:42:57 overcloud-novacompute-0 ifup[12281]: /sbin/ifup: configuration for ovs-system not found. | 15:58 |
| weshay | nope | 15:58 |
| weshay | that's in a good job | 15:58 |
| rlandy | weshay: https://bugs.launchpad.net/tripleo/+bug/1857356 | 15:59 |
| openstack | Launchpad bug 1857356 in tripleo "Master fs001 centos 7 jobs are timing out on overcloud deploy - log hidden" [Critical,Triaged] | 15:59 |
| rlandy | now we have something to log to | 15:59 |
| weshay | man.. the journal on the undercloud is taking forever.. rlandy /me looking for what is happening at 12:02 | 16:03 |
| rlandy | weshay: somewhere we lose an hour | 16:04 |
| rlandy | checked version of openvswitch | 16:04 |
| rlandy | looks the same as passing job | 16:04 |
| rlandy | we need a diff of 12/21 to 12/23 | 16:04 |
| rlandy | periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-masteropenstack/tripleo-cimasteropenstack-periodic-mastermaster100942019-12-21T17:53:47FAILURE | 16:04 |
| rlandy | periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-masteropenstack/tripleo-cimasteropenstack-periodic-mastermaster93982019-12-21T02:03:00SUCCESS | 16:04 |
| rlandy | ^^ that diff | 16:04 |
| weshay | sshnaidm, do we have an internal cockpit? | 16:05 |
| sshnaidm | weshay, working on this | 16:05 |
| weshay | sshnaidm, cool thanks | 16:05 |
| *** sanjayu_ has quit IRC | 16:10 | |
| weshay | rlandy, chatting w/ migi.. delayed on debug | 16:16 |
| marios | weshay: i think we need to schedule a call asap in the new year to sync on the agenda | 16:18 |
| marios | weshay: so we can 'own' some part and work on the content | 16:18 |
| marios | rlandy: sshnaidm: ^ | 16:18 |
| rlandy | marios: ack | 16:18 |
| sshnaidm | marios, yeah | 16:19 |
| sshnaidm | I'll be back in 5 Jan | 16:20 |
| marios | sshnaidm: ack wfm should i just send invite now and we can update it based on everyone response | 16:21 |
| sshnaidm | marios, sound good | 16:21 |
| marios | sshnaidm: no 5th is sunday | 16:22 |
| marios | so 6th | 16:22 |
| rlandy | sshnaidm: hi - did you hear anything about the bus accident near the airport? It appears that one of the victims is a friend of mine. | 16:22 |
| sshnaidm | rlandy, yeah, saw in the news | 16:23 |
| sshnaidm | rlandy, omg | 16:23 |
| rlandy | sshnaidm: yeah - they just released her name | 16:23 |
| *** Goneri has quit IRC | 16:30 | |
| *** soniya29 has joined #oooq | 16:38 | |
| *** ykarel has quit IRC | 16:41 | |
| weshay | marios, aye.. I think looking at the calendar.. sprint planning on the 11th.. | 16:42 |
| weshay | not sure | 16:42 |
| *** Goneri has joined #oooq | 16:42 | |
| weshay | rfolco, and I should chat about that | 16:42 |
| rfolco | weshay, will reschedule to jan 2nd or jan 3rd | 16:43 |
| rfolco | weshay, waiting for your commands | 16:43 |
| weshay | rfolco, I don't think folks will be back that soon.. that was my mistake earlier | 16:43 |
| *** ykarel has joined #oooq | 16:43 | |
| *** soniya29 has quit IRC | 16:48 | |
| weshay | rfolco, ya.. let's chat about it for a sec.. on the phone w/ migi | 16:48 |
| weshay | atm | 16:48 |
| * marios out folks tty next monday | 16:48 | |
| marios | weshay: rlandy: sshnaidm: sent invite for 6th please respond and we can reschedule according to availability | 16:48 |
| rfolco | ok | 16:49 |
| sshnaidm | marios, wfm | 16:49 |
| *** soniya29 has joined #oooq | 16:50 | |
| weshay | rfolco, k ready | 16:50 |
| rlandy | marios: thanks - I'm in | 16:50 |
| weshay | rfolco, https://meet.google.com/hht-zokn-bog | 16:51 |
| *** derekh has quit IRC | 16:52 | |
| rlandy | marios: fyi ... https://bugs.launchpad.net/tripleo/+bug/1857356 added to promotion blocker | 16:52 |
| openstack | Launchpad bug 1857356 in tripleo "Master fs001 centos 7 jobs are timing out on overcloud deploy - log hidden" [Critical,Triaged] | 16:52 |
| rlandy | already failing in check though | 16:52 |
| rlandy | marios: also - looks like we have consistent failures on fs020 | 16:53 |
| rlandy | periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-rockytestprojectmastercheck24300,1159862019-12-23T11:32:54FAILURE | 16:53 |
| rlandy | periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-queenstestprojectmastercheck24300,1140422019-12-23T11:31:37FAILURE | 16:53 |
| rlandy | skip list and LP time | 16:55 |
| marios | rlandy: ack but /me leaving will hav to wait a few days :D enjoy your time out too rlandy | 17:05 |
| marios | have a good week all | 17:05 |
| *** marios is now known as marios|out | 17:05 | |
| rlandy | marios|out: bye - happy holidays | 17:05 |
| sshnaidm | marios|out, enjoy your holidays | 17:06 |
| marios|out | thanks you too folks | 17:06 |
| *** soniya29 has quit IRC | 17:09 | |
| *** ykarel is now known as ykarel|away | 17:10 | |
| weshay | rlandy, fs001 passed on all those pcs changes.. /me hit check-rdo on https://review.opendev.org/#/c/699318/1 | 17:11 |
| weshay | to try it again | 17:11 |
| *** marios|out has quit IRC | 17:15 | |
| rlandy | https://review.opendev.org/#/c/699318 - that does pass | 17:15 |
| rlandy | interesting | 17:15 |
| rlandy | anyways logging the queens/rocky tempest fails in the mean time | 17:17 |
| rfolco | weshay, rlandy pls re-vote/merge https://review.rdoproject.org/r/#/c/24307/ | 17:29 |
| rfolco | fixed linters and tested it locally | 17:29 |
| *** ykarel|away has quit IRC | 17:32 | |
| *** tosky has quit IRC | 17:42 | |
| *** ssbarnea has quit IRC | 17:46 | |
| rlandy | rfolco: will look in a bit - just filing tempest bugs | 17:48 |
| rfolco | rlandy, weshay did thx | 17:49 |
| rfolco | brb | 17:49 |
| *** rfolco is now known as rfolco|bbl | 17:49 | |
| sshnaidm | weshay, http://10.19.113.130/d/9DmvErfZz/cockpit?orgId=1 | 17:49 |
| weshay | sshnaidm, nice | 17:50 |
| weshay | thank you | 17:50 |
| weshay | sshnaidm, and that is setup w/ the same scripts in infra-setup/role/rrockpit? | 17:51 |
| weshay | sshnaidm, /me wondering how best to include something like internal sf zuul on the internal version | 17:51 |
| sshnaidm | weshay, right now it doesn't update automatically, it will require some work and change of roles | 17:51 |
| sshnaidm | weshay, maybe we can hold this scripts in some of internal repos | 17:52 |
| weshay | sshnaidm, should we maintain an internal version of the setup? | 17:52 |
| weshay | sshnaidm, aye | 17:52 |
| sshnaidm | weshay, if you set in hosts "10.19.113.130 internal-cockpit.triple.org" you can go directly to cockpit by http://internal-cockpit.triple.org | 17:57 |
| weshay | sshnaidm, k.. want me to setup a ddns entry? | 17:57 |
| sshnaidm | weshay, would be nice to manage this in some of internal dns | 17:57 |
| weshay | sshnaidm, sec.. I'll get it | 17:58 |
| rlandy | weshay: pls see https://review.opendev.org/700443 Add failing tempest test to skip list | 18:48 |
| rlandy | we have two new cards on cix now | 18:48 |
| weshay | rlandy, k | 18:50 |
| rlandy | 2019-12-23 18:13:50.613743 | RUN END RESULT_TIMED_OUT: [untrusted : opendev.org/openstack/tripleo-ci/playbooks/tripleo-buildcontainers/run.yaml@master] | 18:52 |
| weshay | hrm | 18:54 |
| *** ssbarnea has joined #oooq | 18:57 | |
| *** dtrainor has joined #oooq | 19:56 | |
| *** dtrainor has quit IRC | 21:06 | |
| *** rfolco|bbl has quit IRC | 21:23 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!