| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp- (3 more messages) | 00:10 |
|---|---|---|
| *** tosky has quit IRC | 00:18 | |
| *** rascasoft has joined #oooq | 00:32 | |
| *** trown is now known as trown|outtypewww | 00:39 | |
| *** rascasoft has quit IRC | 00:41 | |
| *** hamzy has joined #oooq | 01:11 | |
| *** rascasoft has joined #oooq | 01:28 | |
| *** rascasoft has quit IRC | 01:40 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp- (3 more messages) | 02:10 |
| *** rascasoft has joined #oooq | 02:16 | |
| *** rascasoft has quit IRC | 02:26 | |
| *** dsneddon has quit IRC | 02:44 | |
| *** rlandy has quit IRC | 02:49 | |
| *** dsneddon has joined #oooq | 03:09 | |
| *** apetrich has quit IRC | 03:15 | |
| *** dsneddon has quit IRC | 03:15 | |
| *** dsneddon has joined #oooq | 03:43 | |
| *** skramaja has joined #oooq | 03:48 | |
| *** skramaja has quit IRC | 03:53 | |
| *** dsneddon has quit IRC | 03:58 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039 @ https://review.openstack.org/602248, (3 more messages) | 04:10 |
| *** ykarel|away has joined #oooq | 04:47 | |
| *** ykarel|away is now known as ykarel | 04:47 | |
| *** dsneddon has joined #oooq | 04:51 | |
| *** dsneddon has quit IRC | 05:05 | |
| *** dsneddon has joined #oooq | 05:06 | |
| *** raukadah is now known as chandankumar | 05:09 | |
| *** ratailor has joined #oooq | 05:44 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039 @ https://review.openstack.org/602248, (3 more messages) | 06:10 |
| *** dsneddon has quit IRC | 06:29 | |
| *** saneax has joined #oooq | 06:42 | |
| *** quiquell|off is now known as quiquell | 07:01 | |
| *** dsneddon has joined #oooq | 07:01 | |
| *** dsneddon has quit IRC | 07:16 | |
| *** dsneddon has joined #oooq | 07:20 | |
| *** jfrancoa has joined #oooq | 07:21 | |
| *** kopecmartin|off is now known as kopecmartin | 07:23 | |
| *** saneax has quit IRC | 07:23 | |
| *** saneax has joined #oooq | 07:24 | |
| *** dsneddon has quit IRC | 07:25 | |
| *** rascasoft has joined #oooq | 07:30 | |
| *** jtomasek has joined #oooq | 07:37 | |
| *** gkadam has joined #oooq | 07:37 | |
| *** dsneddon has joined #oooq | 07:41 | |
| *** quiquell is now known as quiquell|brb | 07:54 | |
| *** dsneddon has quit IRC | 07:57 | |
| kopecmartin | chandankumar, arxcruz hi, when you have a moment, please, have a look https://review.openstack.org/#/c/625191/ | 08:10 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039 @ https://review.openstack.org/602248, (3 more messages) | 08:10 |
| *** apetrich has joined #oooq | 08:12 | |
| chandankumar | kopecmartin: done! | 08:14 |
| kopecmartin | chandankumar, thanks | 08:14 |
| *** amoralej|off is now known as amoralej | 08:14 | |
| chandankumar | kopecmartin: since now os_tempest playbook is merged now, I will take a look at standlone insallation and user stuff | 08:15 |
| kopecmartin | chandankumar, wait, what do you mean, I'm working on usage of os_tempest via ansible-playbook command | 08:16 |
| chandankumar | kopecmartin: ok you are looking into then good, I was about to start! | 08:17 |
| chandankumar | kopecmartin: I have to deal with few os_tempest deps and tempest users patches | 08:17 |
| kopecmartin | chandankumar, ok | 08:18 |
| chandankumar | kopecmartin: let me know if you face any issue | 08:18 |
| kopecmartin | sure | 08:18 |
| chandankumar | kopecmartin: one more thing if you need to install any plugin just do tempest_service_available_[service] = true | 08:18 |
| chandankumar | it will install the plugin for that | 08:18 |
| chandankumar | if source then it will clone from git | 08:19 |
| kopecmartin | yeah, i get it | 08:19 |
| chandankumar | if distro then from packages | 08:19 |
| *** ccamacho has joined #oooq | 08:19 | |
| chandankumar | cool! | 08:19 |
| *** quiquell|brb is now known as quiquell | 08:26 | |
| *** jfrancoa has quit IRC | 08:29 | |
| *** bogdando has joined #oooq | 08:42 | |
| *** tosky has joined #oooq | 08:42 | |
| *** jpena|off is now known as jpena | 08:48 | |
| *** dsneddon has joined #oooq | 08:55 | |
| *** holser_ has joined #oooq | 08:55 | |
| *** dsneddon has quit IRC | 09:00 | |
| *** panda|ruck|off is now known as panda|ruck | 09:05 | |
| panda|ruck | Tra-la-laaa | 09:06 |
| * panda|ruck captain underpands | 09:06 | |
| *** saneax has quit IRC | 09:08 | |
| *** saneax has joined #oooq | 09:09 | |
| panda|ruck | weeee all ovb jobs are failin | 09:09 |
| *** saneax has quit IRC | 09:10 | |
| *** saneax has joined #oooq | 09:10 | |
| *** ykarel is now known as ykarel|lunch | 09:27 | |
| *** dsneddon has joined #oooq | 09:32 | |
| panda|ruck | ykarel|lunch: was the python-hardware bug fixed ? | 09:32 |
| *** dtantsur|afk is now known as dtantsur | 09:37 | |
| *** dsneddon has quit IRC | 09:40 | |
| *** derekh has joined #oooq | 09:42 | |
| *** sshnaidm is now known as sshnaidm|off | 10:00 | |
| ykarel|lunch | panda|ruck, yes that is fixed | 10:03 |
| *** ykarel|lunch is now known as ykarel | 10:03 | |
| panda|ruck | ykarel: ok thanks | 10:05 |
| panda|ruck | weshay: rfolco|rover ykarel shutting down promoter-server for maintenance | 10:05 |
| ykarel | panda|ruck, ack | 10:06 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039 @ https://review.openstack.org/602248, (3 more messages) | 10:10 |
| *** saneax has quit IRC | 10:25 | |
| *** saneax has joined #oooq | 10:26 | |
| *** dsneddon has joined #oooq | 10:27 | |
| *** saneax has quit IRC | 10:30 | |
| *** dsneddon has quit IRC | 10:32 | |
| quiquell | panda|ruck: do you have desing time for me ? | 10:42 |
| panda|ruck | quiquell: I have always design time for everyone. except when I'm battling promotion issues | 10:43 |
| quiquell | panda|ruck: are you battling promotions ? | 10:43 |
| panda|ruck | quiquell: just finished. Updating the bug, then we can talk | 10:43 |
| quiquell | ack | 10:43 |
| panda|ruck | quiquell: ready | 10:49 |
| panda|ruck | quiquell: on your room | 10:49 |
| quiquell | going there | 10:50 |
| marios | https://review.openstack.org/#/c/638651/ needs more votes please when folks have time. part of https://tree.taiga.io/project/tripleo-ci-board/task/773 (tripleo-ci-testing for repos in containers build) | 10:51 |
| quiquell | marios: you can join | 10:52 |
| quiquell | marios: design BM trigger | 10:52 |
| marios | quiquell: sure gimme 2 mins joining | 10:52 |
| marios | quiquell: number/? | 10:52 |
| quiquell | wait cannot join my blue | 10:53 |
| quiquell | 7891065232 | 10:53 |
| sshnaidm|off | quiquell, panda|ruck marios please help merge logs size reducing: https://review.openstack.org/#/q/topic:newara | 11:04 |
| marios | sshnaidm|off: ack in a bit (in a call with panda|ruck now) | 11:06 |
| marios | https://review.openstack.org/#/c/639359/6 | 11:12 |
| marios | quiquell: ^ | 11:13 |
| marios | http://lists.openstack.org/pipermail/openstack-discuss/2019-February/003330.html | 11:13 |
| marios | quiquell: ^ | 11:13 |
| zbr | marios or panda|ruck: on bindep: https://review.openstack.org/#/c/639951/ - you are not core but votes may help getting the wf from those that can do it. | 11:16 |
| zbr | once we get this, we could avoid installing packages with ansible and use bindep | 11:17 |
| zbr | (in some places) | 11:17 |
| *** jfrancoa has joined #oooq | 11:17 | |
| chandankumar | zbr: need some help here https://review.openstack.org/#/c/640089/2/roles/validate-tempest/templates/run-tempest.sh.j2@88 I am not sure why jinja is not working | 11:27 |
| chandankumar | please have a look when free! | 11:28 |
| *** dsneddon has joined #oooq | 11:28 | |
| quiquell | panda|ruck: do you know why we use dlrn user "review_rdoproject_org" at reporting and "ciuser" at promotion ? | 11:31 |
| quiquell | humm nope same for promote :-/ | 11:32 |
| zbr | chandankumar: lgtm, what is the failure? do you have the generated file? | 11:34 |
| *** dsneddon has quit IRC | 11:34 | |
| panda|ruck | quiquell: we use ciuser at promotion | 11:34 |
| panda|ruck | quiquell: because review_rdoproject_org is a secret in zuul | 11:35 |
| chandankumar | zbr: 2019-03-01 06:10:14 | /home/zuul/tempest-setup.sh: line 159: KRB5_CLIENT_KTNAME: unbound variable | 11:35 |
| panda|ruck | quiquell: and we didn't probably want to use the same credentials for an external server | 11:35 |
| chandankumar | zbr: I am thinking adding two stuff there, first var is defined and then check the bool | 11:35 |
| quiquell | panda|ruck: where do we that password ? | 11:35 |
| chandankumar | zbr: what do you say? | 11:36 |
| quiquell | panda|ruck: I think the secret we have put at internal sf is for ciuser passwoerd :-/ | 11:36 |
| zbr | chandankumar: is not jinja issue, the env var is not defined | 11:36 |
| zbr | if it can be null do use. "${FOO:-}" to default its expension to empty string | 11:36 |
| zbr | instead of "${FOO}" | 11:36 |
| chandankumar | zbr: sure let me update | 11:37 |
| zbr | chandankumar: you jinja2 is good. btw read http://redsymbol.net/articles/unofficial-bash-strict-mode/ when you have time. | 11:38 |
| zbr | is one of my fav articles, very useful | 11:38 |
| chandankumar | zbr: sure, adding to the list | 11:38 |
| panda|ruck | zbr: I think ou'll need to add tests for https://review.openstack.org/639951 | 11:38 |
| panda|ruck | zbr: like in def test_detects_rhel(self): | 11:39 |
| panda|ruck | zbr: and you'll have to create fixtures | 11:39 |
| zbr | panda|ruck: already did, kinaof. Added f28 job which was missing in the past and updated the tests/bindep.txt file. not a full test. I hope no core asks for more on that. | 11:40 |
| panda|ruck | zbr: I wouln't count on it. | 11:40 |
| zbr | current bindep tests do not have something like this, even for other cases. i would rather prefer to alter the integration testing to check if right platforms are reported. | 11:41 |
| panda|ruck | zbr: you're adding a new platform: category and a new conditional, they'll probably want tests. at least check that verify a Contains("platform:dnf") when the version is right | 11:41 |
| quiquell | panda|ruck: I am going to create new user for internal sf it's also a good idea | 11:42 |
| marios | https://review.openstack.org/#/c/638651/ needs more votes please when folks have time. part of https://tree.taiga.io/project/tripleo-ci-board/task/773 (tripleo-ci-testing for repos in containers build) | 11:43 |
| zbr | adding tests for it would require a lot of mocking... the fact that is passing integration testing is good enough for me. anyway, lets see.. | 11:43 |
| panda|ruck | zbr: I don't think the integration testing is checking if bindep honours the new platform:yum atom at all | 11:46 |
| zbr | panda|ruck: true, i does only check that it does not choke. your point is valid, is just that I am trying to minimize the effort. me kinda lazy, just want to do minimal effort to get it done ;) | 11:49 |
| panda|ruck | zbr: is this review required ? | 11:50 |
| panda|ruck | zbr: for the US ? | 11:50 |
| zbr | arguable / non-blocking, it done we could get rid of stuff like https://review.openstack.org/#/c/636160/53/roles/tripleo-repos/defaults/main.yml | 11:52 |
| zbr | we do lots of hacks now to install alternative rpms, also affects new reproducer. so is in our interest to make bindep better. but is not directly blocking us form using (ugly) workarounds. | 11:53 |
| panda|ruck | zbr: ok, analyzing the amount of effort needed at this point, if they're going to ask you to add more tests, I would probably suggest to put this aside for this sprint, or work on it on the 20% | 11:53 |
| zbr | this is why i am trying to fix it without making any other work depending on it | 11:54 |
| panda|ruck | zbr: yes, but adding the test would probably double the workload on this patch | 11:54 |
| zbr | panda|ruck: totally agree. | 11:54 |
| panda|ruck | zbr: I reached the conclusion some time ago. For how easy a patch seems to take to write, it's the design and the testing that you should consider more time consuming | 11:56 |
| zbr | panda|ruck: my estimates are 5-10x more time than doing the fix locally. i count this most of the time, with one side note: every time i need something fix upstream i raise a bug (or pod fix), just to start the process, even if i never endup finishing it, at least someone else can continue it. | 12:00 |
| panda|ruck | quiquell: correction, until the end of the sprint, WE have 7 days, YOU have 5 | 12:00 |
| marios | quiquell: you can hear him smiling as he types that srsly man | 12:01 |
| panda|ruck | It's astonishly easy to forget thos details on time. | 12:01 |
| quiquell | marios: PTG is payback :-) | 12:01 |
| panda|ruck | astonishingly ? | 12:01 |
| panda|ruck | marios: a smile with 52 teeth | 12:02 |
| panda|ruck | marios: no wait, 43 and half until the end of the sprint | 12:02 |
| quiquell | panda|ruck, marios: To make DLRN API user configurable https://review.rdoproject.org/r/19064 | 12:03 |
| chandankumar | sshnaidm|off: I have updated the tls temepst taiga us feel free to take a look https://tree.taiga.io/project/tripleo-ci-board/us/670 when free, thanks! | 12:04 |
| quiquell | panda|ruck: linters are broken ? | 12:08 |
| marios | quiquell: ack added for later | 12:08 |
| panda|ruck | quiquell: where ? | 12:10 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-scenario002-multinode- (2 more messages) | 12:10 |
| panda|ruck | mmhh node failure | 12:10 |
| quiquell | yep | 12:11 |
| panda|ruck | quiquell: where's the graph in the cockpit that says how many staks are broken ? | 12:12 |
| panda|ruck | quiquell: http://dashboard-ci.tripleo.org/d/cEEjGFFmz/cockpit?orgId=1&panelId=175&fullscreen | 12:12 |
| panda|ruck | this one | 12:12 |
| quiquell | nope | 12:12 |
| panda|ruck | no ? | 12:12 |
| panda|ruck | I see 600 servers and 30 in error | 12:12 |
| quiquell | http://dashboard-ci.tripleo.org/d/cEEjGFFmz/cockpit?orgId=1&from=1551269576074&to=1551442376074&var-launchpad_tags=alert&var-promotion_names=current-tripleo&var-promotion_names=current-tripleo-rdo&var-promotion_names=current-tripleo-rdo-testing&var-releases=master&var-releases=rocky&var-releases=queens&var-releases=pike&panelId=231&fullscreen | 12:13 |
| quiquell | the problematic one | 12:13 |
| quiquell | shorter | 12:13 |
| quiquell | 200~http://dashboard-ci.tripleo.org/d/cEEjGFFmz/cockpit?orgId=1&panelId=231&fullscreen | 12:13 |
| panda|ruck | I don't see any particular explosion | 12:14 |
| quiquell | failed again | 12:15 |
| quiquell | well I think tox-liners doe not run at heat stacks | 12:15 |
| quiquell | also I think they have change at run at containers | 12:15 |
| panda|ruck | quiquell: better ask in rdo then ... wee need zuul logs .. | 12:16 |
| panda|ruck | weeeee | 12:16 |
| *** panda|ruck is now known as panda|ruck|lunch | 12:16 | |
| *** ykarel is now known as ykarel|afk | 12:18 | |
| *** dsneddon has joined #oooq | 12:19 | |
| *** dsneddon has quit IRC | 12:25 | |
| marios | panda|ruck|lunch: how do you automatically appear as reviewer in rdo config reviews. i just posted it but you're already on it?! https://review.rdoproject.org/r/#/c/19066 are you filed under 'default' somewhere? | 12:46 |
| marios | panda|ruck|lunch: quiquell ** we need this asap so we can test the actual push https://review.rdoproject.org/r/#/c/19065/ https://tree.taiga.io/project/tripleo-ci-board/task/817 (and then https://review.rdoproject.org/r/#/c/19066/ will be the last piece but we need this first) | 12:50 |
| marios | tox-linters tox-linters : NODE_FAILURE | 12:50 |
| quiquell | marios: Yep I am having same failure there | 12:51 |
| *** jpena is now known as jpena|lunch | 12:51 | |
| quiquell | marios: I think is related to move to runc | 12:51 |
| quiquell | of tox jobs | 12:51 |
| marios | weshay: panda|ruck|lunch: rfolco|rover fyi 2tasks under the centos container story https://tree.taiga.io/project/tripleo-ci-board/task/817 for testing the push (and fixing anything) then finally https://tree.taiga.io/project/tripleo-ci-board/task/818 switch to new job if we are brave ;) | 12:52 |
| marios | quiquell: thx | 12:52 |
| weshay | chandankumar who's bluejeans? | 12:58 |
| chandankumar | weshay: arxcruz kopecmartin https://redhat.bluejeans.com/1571313919/6145/?src=meet_now | 12:59 |
| chandankumar | weshay: mine! | 13:00 |
| *** dsneddon has joined #oooq | 13:01 | |
| weshay | chandankumar link? | 13:01 |
| chandankumar | weshay: https://redhat.bluejeans.com/1571313919/6145/?src=meet_now | 13:01 |
| chandankumar | arxcruz: we are waiting for you! | 13:02 |
| *** panda|ruck|lunch is now known as panda|ruck | 13:04 | |
| *** ratailor has quit IRC | 13:07 | |
| *** dsneddon has quit IRC | 13:10 | |
| weshay | arxcruz ping | 13:12 |
| arxcruz | weshay: sorry | 13:12 |
| *** trown|outtypewww is now known as trown | 13:16 | |
| chandankumar | weshay: arxcruz kopecmartin https://github.com/openstack/tripleo-quickstart-extras/blob/master/playbooks/tempest.yml | 13:16 |
| chandankumar | weshay: arxcruz kopecmartin https://github.com/openstack/openstack-ansible-tests/blob/master/test-vars.yml | 13:17 |
| zbr | marios: quiquell weshay : please revote on https://review.openstack.org/#/c/636160/ - f28 fixed last remark and passed zuul. | 13:18 |
| chandankumar | weshay: arxcruz kopecmartin https://github.com/openstack/openstack-ansible-os_tempest/blob/master/tests/os_tempest-overrides.yml | 13:18 |
| *** holser_ has quit IRC | 13:19 | |
| quiquell | zbr: I don't see the third party there | 13:20 |
| zbr | quiquell: go there and check manually, fedora passted. | 13:21 |
| zbr | is not my fault that rdo is very slow today | 13:21 |
| quiquell | you mean zuul ? | 13:22 |
| quiquell | yep checked :-) | 13:22 |
| quiquell | cool | 13:22 |
| quiquell | marios: going to workflow it | 13:23 |
| marios | zbr: ah you mean it didn't report yet but it passed? anyway +2 added note about waiting for rdo | 13:23 |
| quiquell | http://logs.rdoproject.org/60/636160/53/openstack-check/tripleo-build-containers-fedora-28/6c1718b/ | 13:23 |
| quiquell | ^ passed | 13:23 |
| *** holser_ has joined #oooq | 13:23 | |
| *** holser_ has quit IRC | 13:23 | |
| marios | quiquell: thanks | 13:23 |
| quiquell | zbr: +w | 13:23 |
| *** holser_ has joined #oooq | 13:24 | |
| zbr | thanks! | 13:24 |
| chandankumar | weshay: arxcruz kopecmartin http://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-standalone-os-tempest | 13:24 |
| marios | https://review.openstack.org/#/c/638651/ needs more votes please when folks have time. part of https://tree.taiga.io/project/tripleo-ci-board/task/773 (tripleo-ci-testing for repos in containers build) | 13:24 |
| quiquell | marios: can we workflow tripleo-repos ? | 13:28 |
| quiquell | panda|ruck: ^ Â? | 13:29 |
| panda|ruck | quiquell: yes, logic is working, deps is implicit, marios added also unit tests and they are working, so +2 from me | 13:32 |
| quiquell | panda|ruck: I mean tripleo-repos is like tht that we can review vote but not workflow ? | 13:33 |
| rfolco|rover | panda|ruck, you get 5 min? | 13:33 |
| quiquell | marios: +w | 13:33 |
| marios | quiquell: yeah | 13:34 |
| marios | quiquell: i think so | 13:34 |
| panda|ruck | quiquell: well, tripleo-repos now affects directly our jobs, I'd say we are forced to take part of the ownership, even only if we want to merge it to repo-setup and use only one method | 13:34 |
| marios | quiquell: i mean i think its done we tested it | 13:34 |
| panda|ruck | rfolco|rover: yes, I'm gonna get 5 minutes | 13:34 |
| *** rlandy has joined #oooq | 13:34 | |
| rfolco|rover | panda|ruck, can you bj my room pls? | 13:34 |
| marios | quiquell: it got the tripleo-ci-testing repo in the test job review.rdoproject.org/r/#/c/19000 | 13:34 |
| marios | thanks quiquell panda|ruck | 13:34 |
| quiquell | zbr, marios: what else can we merge ? | 13:35 |
| zbr | this stupid one https://review.openstack.org/#/c/640320/ | 13:36 |
| marios | quiquell: we also need this one https://review.rdoproject.org/r/19065 so we can test the push | 13:37 |
| marios | quiquell: ah thanks you already voted there sorry | 13:38 |
| quiquell | zbr: I don't see the tox ci there | 13:44 |
| quiquell | weshay, rlandy: for the containers-build-push cofing project -> https://review.rdoproject.org/r/#/c/19065/ | 13:45 |
| zbr | quiquell: that's because someone decided that tox.ini is on exclude pattern. you need to run it manually to see that it works. | 13:45 |
| quiquell | zbr: remove from exclude | 13:45 |
| quiquell | zbr: it really affects the tox jobs | 13:45 |
| quiquell | All of them | 13:46 |
| zbr | quiquell: it may upset others, i could try but not if others do not want it. | 13:47 |
| rlandy | quiquell: zbr: can +2 that but zuul would have to clear its -1 | 13:48 |
| quiquell | rlandy: damn... the tox NODE_FAILURE | 13:48 |
| quiquell | rlandy: btw, dlrn report not ready for QE at baremetal | 13:48 |
| quiquell | rlandy: password was failing | 13:48 |
| quiquell | rlandy: at test job | 13:48 |
| rlandy | quiquell: ok - I'll I can recreate it and check | 13:49 |
| quiquell | rlandy: is not that | 13:49 |
| quiquell | rlandy: the password is not the one fron ciuser | 13:49 |
| zbr | panda|ruck: ^ i think we have aproblem on rdo, with nodes, node failure with linters | 13:49 |
| quiquell | rlandy: is the user is review_rdoproject_org <- cannot find it | 13:50 |
| rlandy | quiquell; I took the one from centos | 13:50 |
| rlandy | quiquell: from the promoter server | 13:50 |
| rlandy | went to user centos | 13:50 |
| quiquell | rlandy: nope this is not the one | 13:50 |
| quiquell | rlandy: promoter pass is for user ciuser | 13:50 |
| panda|ruck | zbr: yes, quiquell asked in #rdo | 13:50 |
| panda|ruck | quiquell: any replay | 13:50 |
| quiquell | rlandy: but at RDO periodic jobs user is review_rdoproject_org | 13:50 |
| rlandy | quiquell: let me find the ciuser password sec | 13:51 |
| quiquell | rlandy: I have that one | 13:51 |
| quiquell | rlandy: but ciuser is not the user | 13:51 |
| * quiquell find the code | 13:51 | |
| zbr | quiquell: i have the impression that the only tox job running on tripleo-ci is the linters one. | 13:52 |
| *** ykarel|afk is now known as ykarel | 13:52 | |
| panda|ruck | zbr: heat stacks are currently high but not in error. | 13:52 |
| quiquell | panda|ruck: those tox failures are related to runc | 13:52 |
| quiquell | panda|ruck: I think heat stack are not involve there | 13:52 |
| panda|ruck | quiquell: runc ? | 13:53 |
| quiquell | panda|ruck: 0001702497-runc-centos-7-100-0000471906 | 13:54 |
| quiquell | this is the error | 13:54 |
| quiquell | https://paste.fedoraproject.org/paste/HxNqbp0bkGygMe-ukaKVeA | 13:54 |
| *** jpena|lunch is now known as jpena | 13:54 | |
| weshay | arxcruz https://review.openstack.org/#/c/639794/ | 13:54 |
| quiquell | rlandy: https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/ci-scripts/tripleo-upstream/dlrnapi_report.sh#L9 | 13:54 |
| panda|ruck | quiquell: today I don't understand when you are adding typos or writing seriously | 13:54 |
| quiquell | rlandy: so for periodics user is review_rdoproject_org | 13:54 |
| quiquell | rlandy: I am creating new user (tripleo-internal-ci) to report from BM | 13:54 |
| quiquell | rlandy: No one find the password for review_rdoproject_org though | 13:55 |
| rlandy | quiquell: lol | 13:55 |
| quiquell | yep | 13:55 |
| quiquell | :Ã-/ | 13:55 |
| rlandy | so I shouldn't bother looking for it | 13:55 |
| rlandy | how are reporting then? | 13:55 |
| quiquell | rlandy: totally I have put a "TODO" in the creds doc | 13:55 |
| rlandy | mystery | 13:56 |
| quiquell | rlandy: rdo secret is for review_rdoproject_org password | 13:56 |
| quiquell | rlandy: but internal factory secret is for ciuser password | 13:56 |
| quiquell | rlandy: at internal factory dlrn reporting was not working | 13:56 |
| quiquell | rlandy: I am taking a step buck and do noop job to test reporting | 13:56 |
| rlandy | quiquell; k - you can just recreate the secret for tripleo-ci-internal-config | 13:57 |
| quiquell | rlandy: but I don't have the password | 13:58 |
| rlandy | quiquell: ack - when you do | 13:58 |
| quiquell | sure sure | 13:58 |
| rlandy | quiquell: do yo have the secret create url for downstream? | 13:58 |
| quiquell | rlandy: yep no problem with that | 13:58 |
| * rlandy will send | 13:58 | |
| quiquell | know that stuff | 13:59 |
| quiquell | from reproducer work | 13:59 |
| quiquell | rlandy: I have put in place a change in the role to make user configurable -> https://review.rdoproject.org/r/#/c/19064/ | 13:59 |
| quiquell | rlandy: so we can use 'ciuser' now | 13:59 |
| quiquell | rlandy: or the new user jpena is creating | 13:59 |
| rlandy | quiquell: k - cool. we had a bunch of node failures last night so I couldn't test but previously, the job was not cleaning up so the old vm was still there | 14:01 |
| rlandy | might explain why we never connected to it | 14:01 |
| rlandy | retrying this morning | 14:01 |
| quiquell | rlandy: yap, before leave i show a lot of dirty stuff there | 14:02 |
| quiquell | rlandy: we need the cleanup | 14:02 |
| rlandy | quiquell: toci had skip-tags set | 14:02 |
| rlandy | there is a lot of voodoo there | 14:02 |
| rlandy | we undercover a lot of magic when we try to do something different | 14:02 |
| quiquell | hackgic | 14:03 |
| panda|ruck | quiquell: see what I meant ? | 14:03 |
| quiquell | panda|ruck: about what ? | 14:04 |
| panda|ruck | 13:54:36 +panda|ruck | quiquell: today I don't understand when you are adding typos or writing seriously | 14:05 |
| rlandy | quiquell: check it out ... https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/stream/b75753d1b8194b99bcb50a34e966b0ca?logfile=console.log | 14:05 |
| rlandy | we're installing undercloud | 14:06 |
| quiquell | panda|ruck: serious types I would say | 14:06 |
| quiquell | rlandy: \o/ !!! | 14:06 |
| quiquell | yey | 14:06 |
| quiquell | rlandy: another one, URL trigger is no good for us | 14:07 |
| rlandy | quiquell: we need a drink - both of us!! | 14:07 |
| quiquell | rlandy: we don't have the proper headers anywhere | 14:07 |
| weshay | quiquell rlandy NICE | 14:07 |
| weshay | so what was the deal w/ the key? | 14:07 |
| rlandy | weshay: keys were fine | 14:07 |
| quiquell | permissions ? | 14:07 |
| rlandy | weshay: it just wan't running clean up | 14:07 |
| rlandy | so old vm new keys | 14:07 |
| weshay | lolz | 14:07 |
| rlandy | no match | 14:07 |
| weshay | right | 14:07 |
| weshay | k | 14:07 |
| weshay | rlandy easy :) | 14:07 |
| quiquell | rlandy: maybe docker container with clean toci :-) | 14:08 |
| weshay | quiquell told you :) | 14:08 |
| quiquell | weshay: old doc | 14:08 |
| quiquell | dog | 14:08 |
| weshay | lolz | 14:08 |
| weshay | that's right.... young blookd | 14:08 |
| weshay | blood | 14:08 |
| rlandy | quiquell: bakc to trigger - no good? | 14:08 |
| rfolco|rover | quiquell, can you please join my bj to see if you have any clues on my centos reproducer error ? | 14:09 |
| rlandy | I think QE is using a URL trigger | 14:09 |
| panda|ruck | today is officailly typo day | 14:09 |
| quiquell | rlandy: yep we don't ahve ETag or Last-Modified headers | 14:09 |
| rlandy | panda|ruck: everyday is typo day | 14:09 |
| rfolco|rover | quiquell, my fedora one worked, I just cannot make the centos reproducer work | 14:09 |
| quiquell | rlandy: also if they fail is difficult to debug | 14:09 |
| rlandy | I bet | 14:09 |
| rlandy | connection etc. | 14:09 |
| rlandy | quiquell: what about the chained job idea? | 14:09 |
| rlandy | query dlrn | 14:10 |
| quiquell | rlandy: so going to do a POC on DLRN polling | 14:10 |
| *** holser_ has quit IRC | 14:10 | |
| quiquell | rlandy: yep going to do that | 14:10 |
| rlandy | quiquell,: awesome | 14:10 |
| quiquell | rfolco|rover: let's take a look | 14:10 |
| rlandy | quiquell: if this job passes, I ma going to merge the base job in config | 14:10 |
| rfolco|rover | quiquell, https://redhat.bluejeans.com/u/rfolco/ | 14:10 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-scenario002-multinode- (2 more messages) | 14:10 |
| quiquell | weshay: do you know where do we have the password for rewiew_rdoproject_org ? | 14:10 |
| rlandy | we are getting a lot of errors about having a base job in jobs | 14:11 |
| quiquell | rlandy, weshay: What was the state of reproducer and OVB ? | 14:16 |
| quiquell | we can run them out of the box ? | 14:16 |
| *** ykarel_ has joined #oooq | 14:16 | |
| rlandy | quiquell; ack - I added stuff to the script for ovb | 14:17 |
| rlandy | I think marios managed to do so | 14:17 |
| quiquell | marios: reproducer + ovb works out of the box ? | 14:18 |
| weshay | rlandy can you please merge https://review.rdoproject.org/r/#/c/19055/ | 14:18 |
| marios | quiquell: rlandy ? i don't think i tried that? | 14:18 |
| marios | oh | 14:18 |
| marios | you mean rdo cloud yah rlandy quiquell ? | 14:18 |
| rlandy | marios: oh - sorry | 14:18 |
| *** ykarel has quit IRC | 14:19 | |
| rlandy | weshay: done | 14:19 |
| weshay | rfolco|rover panda|ruck did sova ever end up show either the introspection or image build error? | 14:19 |
| marios | rlandy: quiquell i mean yeah few nits as documented in https://tree.taiga.io/project/tripleo-ci-board/task/765 | 14:20 |
| marios | rlandy: quiquell but yes got to a complete run yesterday | 14:20 |
| rfolco|rover | weshay, I checked yesterday, the last time it happened was before sshnaidm|off changing the sova production server... will check it again | 14:20 |
| weshay | rfolco|rover please add a link / log to prove it's working.. it's just good practice https://bugs.launchpad.net/tripleo/+bug/1817598 | 14:20 |
| openstack | Launchpad bug 1817598 in tripleo "introspection ( prepare images ) failing in 3rd party ovb jobs fs001/35" [Critical,Fix released] | 14:20 |
| quiquell | marios: ack didn't remember it was OVB thanks | 14:20 |
| *** holser_ has joined #oooq | 14:20 | |
| weshay | rfolco|rover k.. thanks | 14:20 |
| quiquell | rfolco|rover: ^ looks like it should work https://review.rdoproject.org/r/#/c/19064/ | 14:20 |
| rfolco|rover | ok | 14:20 |
| rlandy | marios: you're the best taiga user out there :) everything is so well doc'ed | 14:21 |
| quiquell | rfolco|rover: https://tree.taiga.io/project/tripleo-ci-board/task/765 | 14:21 |
| marios | rlandy: thanks. i am liking the markdown support there its pretty easy to make it look pretty | 14:22 |
| * rlandy needs to look into that | 14:22 | |
| *** dsneddon has joined #oooq | 14:23 | |
| *** vinaykns has joined #oooq | 14:23 | |
| bogdando | PTAL https://review.openstack.org/#/q/topic:ci_pipelines+(status:open+OR+status:merged) may be we should just give it a try and see? | 14:24 |
| *** dtrainor_ has quit IRC | 14:26 | |
| *** dtrainor has joined #oooq | 14:26 | |
| chandankumar | Happy weeekend guys, see ya on monday :-) | 14:26 |
| *** amoralej is now known as amoralej|lunch | 14:26 | |
| *** chandankumar is now known as raukadah | 14:26 | |
| marios | rlandy: quiquell welp we can't run the reproducer on a kvm guest then "libvirtError: invalid argument: could not find capabilities for arch=x86_64 domaintype=kvm" | 14:27 |
| marios | rlandy: quiquell (its just what beaker gave me, getting another box now) | 14:28 |
| marios | rlandy: quiquell yeah but that is silly, maybe use it for launching on ovb but not for libvirt o_O | 14:28 |
| *** dsneddon has quit IRC | 14:29 | |
| ykarel_ | panda|ruck, is the RDO phase1 master/rocky tripleo issue is known? any bug? | 14:29 |
| *** ykarel_ is now known as ykarel | 14:29 | |
| rlandy | marios: you should be able to run libvirt | 14:29 |
| marios | rlandy: on a kvm guest though? | 14:29 |
| panda|ruck | ykarel: I'm looking at it, but looks like even the collect logs is failing | 14:29 |
| panda|ruck | ykarel: fails during overcloud-prep-images | 14:30 |
| ykarel | panda|ruck, ack, okk so you are already on it | 14:30 |
| ykarel | hmm without logs it will difficult to find it | 14:30 |
| rlandy | marios: never tried that - on a beaker box directly - yes | 14:31 |
| panda|ruck | ykarel: https://ci.centos.org/job/tripleo-quickstart-promote-rocky-rdo_trunk-minimal/112/artifact/console.txt.gz | 14:31 |
| panda|ruck | ykarel: missing | 14:31 |
| panda|ruck | ykarel: I'm trying to understand where the collect logs fails | 14:31 |
| weshay | panda|ruck can we merge this if it's working an patch on top for improvements? | 14:31 |
| weshay | https://review.rdoproject.org/r/#/c/19048/2/ci-scripts/infra-cleanup/ovb-tenant-cleanup.sh | 14:31 |
| marios | rlandy: ack. yeah beaker just gave me a non vm box so will try that now. | 14:32 |
| * weshay see's 3 servers in error | 14:32 | |
| *** ccamacho has quit IRC | 14:33 | |
| ykarel | panda|ruck, /me looking | 14:33 |
| *** ccamacho has joined #oooq | 14:34 | |
| panda|ruck | weshay: yes, works | 14:34 |
| panda|ruck | weshay: jsut deleted those three servers in error | 14:34 |
| weshay | rock | 14:34 |
| panda|ruck | weshay: removing -1 | 14:34 |
| zbr | weshay: we hace some serious discussions related to yum/dnf/bindeo on infra now... pabelanger opened pandora box | 14:35 |
| weshay | ruh roh | 14:35 |
| weshay | marios you have 5 min? | 14:35 |
| marios | weshay: o/ | 14:35 |
| panda|ruck | zbr: you're a troublemaker :) | 14:35 |
| weshay | marios if you have 5min join my blue | 14:35 |
| zbr | it seems that official pkg manager in r8 is *yum*, not dnf, but that yum is a ~symlink for dnf. | 14:36 |
| marios | weshay: k joining | 14:36 |
| weshay | yes | 14:36 |
| zbr | ansible reports pkg_mgr as dnf which suited us. | 14:36 |
| weshay | zbr that is correct | 14:36 |
| panda|ruck | zbr: timebox the time you want to spend on this please ... | 14:36 |
| rlandy | weshay: quiquell: SUCCESS - https://sf.hosted.upshift.rdu2.redhat.com/logs/49/163849/39/check/periodic-tripleo-ci-centos-7-baremetal-3ctlr_1comp-featureset001-master/b75753d/ - with collect logs operational | 14:36 |
| panda|ruck | zbr: it could easily take the rest of the week | 14:36 |
| weshay | zbr does that mess up ansible or cli options? | 14:36 |
| ykarel | panda|ruck, so possibly logs are failing since https://review.openstack.org/#/c/631067/ merged | 14:37 |
| ykarel | before that i can see logs | 14:37 |
| ykarel | merged on 9th Feb | 14:37 |
| panda|ruck | ykarel: oh, so it's rlandy's fault | 14:37 |
| weshay | rlandy++ | 14:37 |
| hubbot1 | weshay: rlandy's karma is now 45 | 14:37 |
| weshay | rlandy++ | 14:37 |
| hubbot1 | weshay: rlandy's karma is now 46 | 14:37 |
| panda|ruck | :) | 14:38 |
| zbr | not messing ansible, but may mess us, and clearly is affecting the bindep change we wanted to make. | 14:38 |
| weshay | quiquell++ | 14:38 |
| hubbot1 | weshay: quiquell's karma is now 20 | 14:38 |
| ykarel | rfolco|rover, hi you remember discussion related to fs021 | 14:39 |
| ykarel | <rfolco|rover> ykarel, I'll open a bug for it, high prio, not critical | 14:39 |
| ykarel | ^^ one? | 14:39 |
| ykarel | i see job still failing | 14:39 |
| rlandy | panda|ruck: what did I do now? | 14:40 |
| rfolco|rover | ykarel, ok, panda|ruck don't we have a bug for fs20 ? | 14:40 |
| ykarel | 021? | 14:40 |
| rfolco|rover | oops | 14:40 |
| rfolco|rover | panda|ruck, fs021 | 14:40 |
| panda|ruck | fs20 ? | 14:41 |
| zbr | panda|ruck: that issue is far more important than you may think as it could ruin a huge amount of effort we invested in preparing for the *8, is not something we want to ignore. | 14:41 |
| panda|ruck | rlandy: the reproducer script creation broke the logs collection in ci.centos | 14:42 |
| panda|ruck | rlandy: working on it | 14:43 |
| panda|ruck | ykarel: yes, that's it | 14:43 |
| rlandy | panda|ruck: wow and nobody noticed for almost a month | 14:43 |
| panda|ruck | ykarel: that part runs and misses a lot of files because assumes zuul | 14:43 |
| ykarel | yup | 14:44 |
| panda|ruck | rlandy: yep, well, ci.centos jobs where passing at a reasonable rate | 14:44 |
| rlandy | panda|ruck: k - let me know if you need help - we should be able to skip if user is not zuul | 14:44 |
| panda|ruck | zbr: which issue ? | 14:44 |
| quiquell | rlandy: SUCCESS ? you kidding ? | 14:46 |
| quiquell | yey !!! | 14:46 |
| *** dsneddon has joined #oooq | 14:49 | |
| quiquell | rfolco|rover: https://operations.cee.redhat.com/quickvm | 14:49 |
| rlandy | quiquell: well undercloud only - but it's a start | 14:50 |
| quiquell | rlandy: good one, ok have to drop now | 14:50 |
| *** ccamacho has quit IRC | 14:51 | |
| *** quiquell is now known as quiquell|off | 14:51 | |
| rlandy | quiquell|off: have a good weekend | 14:52 |
| *** ccamacho has joined #oooq | 14:53 | |
| panda|ruck | rlandy: ykarel https://review.openstack.org/640393 | 14:53 |
| ykarel | panda|ruck, ack | 14:54 |
| panda|ruck | rlandy: ykarel in CI we are currently creating a zuul dict based on the zuul vars content. If we'll ever move quickstart to the executor itself, the zuul variable should be present anyway. | 14:55 |
| *** dsneddon has quit IRC | 14:56 | |
| rlandy | panda|ruck: yep, should be fine | 14:57 |
| ykarel | panda|ruck, ack | 14:57 |
| panda|ruck | weshay: we'll have to wait for https://review.openstack.org/640393 to have logs in ci.centos and undestand why it's not working. currently failing in overcloud-prep-images, but we have not logs to investigate | 14:59 |
| weshay | panda|ruck++ | 15:11 |
| hubbot1 | weshay: panda|ruck's karma is now 1 | 15:11 |
| *** dsneddon has joined #oooq | 15:16 | |
| *** dsneddon has quit IRC | 15:21 | |
| *** ratailor has joined #oooq | 15:23 | |
| weshay | rfolco|rover k | 15:24 |
| weshay | ready | 15:24 |
| weshay | zbr I need to speak w/ you in a bit | 15:24 |
| rfolco|rover | weshay, o/ | 15:24 |
| zbr | sure. | 15:24 |
| *** amoralej|lunch is now known as amoralej | 15:25 | |
| *** dsneddon has joined #oooq | 15:38 | |
| *** dsneddon has quit IRC | 15:46 | |
| weshay | panda|ruck sova is NOT fixed | 15:49 |
| weshay | https://logs.rdoproject.org/20/639620/2/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/6283a94/job-output.txt.gz | 15:50 |
| weshay | https://logs.rdoproject.org/20/639620/2/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/6283a94/logs/bmc-console.log | 15:50 |
| weshay | panda|ruck this is not working https://github.com/sshnaidm/sova/blob/master/tripleoci/data/patterns.yml#L570 | 15:50 |
| weshay | can you please take another look | 15:51 |
| weshay | panda|ruck I wonder if sova is pulling that log | 15:51 |
| rfolco|rover | weshay, this --> https://github.com/sshnaidm/sova/blob/master/tripleoci/data/patterns.yml#L529 | 15:52 |
| weshay | I would think this does it https://github.com/sshnaidm/sova/blob/master/tripleoci/config.py#L245 | 15:52 |
| *** ratailor has quit IRC | 15:54 | |
| *** dsneddon has joined #oooq | 15:54 | |
| panda|ruck | weshay: diving deeper ... | 15:57 |
| *** dsneddon has quit IRC | 15:59 | |
| weshay | thanks | 16:03 |
| *** ykarel is now known as ykarel|away | 16:03 | |
| *** holser_ has quit IRC | 16:03 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-scenario002-multinode- (2 more messages) | 16:10 |
| panda|ruck | 2019-03-01 16:13:33,576 - watchcat - DEBUG - utils.get_regular_file:237 - Get regular file /logs/bmc-console.log | 16:13 |
| panda|ruck | 2019-03-01 16:13:33,577 - watchcat - DEBUG - utils.get_regular_file:244 - File /cidata/82885c3/bmc-console.lo.gz was saved as 404 | 16:14 |
| panda|ruck | 2019-03-01 16:13:33,577 - watchcat - WARNING - analysis.analyze:105 - File /logs/bmc-console.log is not downloaded, skipping its patterns | 16:14 |
| panda|ruck | it fails to download it | 16:14 |
| *** dsneddon has joined #oooq | 16:15 | |
| panda|ruck | weshay: it tries to download tehe file assuming is a gzipped file, and there is an undocumented pattern to specify to get the plain file that we want ... trying to decode the syntax | 16:20 |
| *** jfrancoa has quit IRC | 16:21 | |
| weshay | panda|ruck ah nice | 16:23 |
| panda|ruck | weshay: nope, nothing I can do without modifying some code here, sova assumes it has to download a gzipped file | 16:23 |
| weshay | thank you | 16:23 |
| weshay | ah | 16:24 |
| weshay | :( | 16:24 |
| panda|ruck | weshay: it's hard-coding adding a .gz at the end | 16:24 |
| weshay | aye | 16:24 |
| panda|ruck | self.file_link).rstrip(".gz") + ".gz" | 16:24 |
| panda|ruck | so it strips the last char in the file, and adds .gx | 16:24 |
| panda|ruck | gz | 16:24 |
| panda|ruck | weshay: so you see 019-03-01 16:13:33,577 - watchcat - DEBUG - utils.get_regular_file:244 - File /cidata/82885c3/bmc-console.lo.gz w | 16:24 |
| panda|ruck | in the logs | 16:24 |
| panda|ruck | bmc-console.lo.gz | 16:24 |
| panda|ruck | weshay: one thing we could do could be to make the lgos gzipped from now one | 16:25 |
| panda|ruck | on* | 16:25 |
| rlandy | panda|ruck: weshay: psl review https://review.openstack.org/#/c/639824/ | 16:26 |
| *** gkadam has quit IRC | 16:27 | |
| rlandy | weshay: one more https://code.engineering.redhat.com/gerrit/#/c/164089/ | 16:27 |
| panda|ruck | weshay: in fact the function to download the file is tring to handle a lot of exceptions, but none works. So w may trying at source by just renaming the bmc-console.log to bmc-console.log.gz. | 16:28 |
| weshay | panda|ruck ya.. I had that thought | 16:30 |
| weshay | in collect-logs? | 16:30 |
| panda|ruck | weshay: yes, but we need to change sova too anyway, to change the file name there | 16:31 |
| panda|ruck | weshay: no, nevermind the last line | 16:31 |
| weshay | panda|ruck I can merge on sova | 16:31 |
| panda|ruck | weshay: oh yes, mind the list line indeed | 16:31 |
| weshay | panda|ruck ya | 16:31 |
| panda|ruck | weshay: ok, but sova will not be able to analyze the existing bmc_console logs | 16:32 |
| weshay | I think you can put up both changes and we can move forward on it | 16:32 |
| weshay | panda|ruck join my blue | 16:32 |
| weshay | let's knock this out | 16:32 |
| weshay | panda|ruck I think the bmc.log is created by the ovb post playbook? | 16:34 |
| weshay | but collect logs could rename it | 16:34 |
| marios | rlandy: fyi got the same on a non kvm box i put a note here for now https://tree.taiga.io/project/tripleo-ci-board/task/766 will revisit next week | 16:38 |
| marios | rlandy: so using --libvirt i get that http://pastebin.test.redhat.com/728999 - i mean makes sense on a kvm guest but this one (according to beaker anyway) is non virtualized. | 16:38 |
| rlandy | marios: ack - ok- thanks | 16:45 |
| weshay | rlandy ack to merge both | 16:45 |
| *** ykarel|away has quit IRC | 16:45 | |
| rlandy | thanks | 16:45 |
| *** ykarel|away has joined #oooq | 16:46 | |
| panda|ruck | weshay: https://review.rdoproject.org/r/19069 | 16:49 |
| *** irclogbot_0 has joined #oooq | 16:51 | |
| *** jtomasek has quit IRC | 16:52 | |
| weshay | panda|ruck https://github.com/sshnaidm/sova/blob/master/tripleoci/config.py#L238 | 16:54 |
| sshnaidm|off | weshay, what is the point to change bmc log name? it should be *.log | 17:04 |
| weshay | sshnaidm|off greetings day off :) | 17:04 |
| sshnaidm|off | weshay, it was an error in yaml, I fixed it today, should be fine in next hours | 17:04 |
| sshnaidm|off | weshay, I get sova pull requests quickly :) | 17:04 |
| * weshay looks at commits | 17:05 | |
| weshay | sshnaidm|off the bmc.log is not getting downloaded via sova | 17:05 |
| weshay | afawct | 17:05 |
| panda|ruck | sshnaidm|off: I saw this in sova container log 2019-03-01 16:13:33,577 - watchcat - DEBUG - utils.get_regular_file:244 - File /cidata/82885c3/bmc-console.lo.gz was saved as 404 | 17:05 |
| *** ykarel|away has quit IRC | 17:06 | |
| *** bogdando has quit IRC | 17:06 | |
| weshay | we're chatting in https://bluejeans.com/u/whayutin/ | 17:06 |
| weshay | sshnaidm|off I think we need sova fixed to handle .log files that are not zipped | 17:11 |
| sshnaidm|off | weshay, panda|ruck https://github.com/sshnaidm/sova/commit/a0e7555c47c3d14c9613725faa902e525c9713c6 | 17:11 |
| * weshay looks | 17:11 | |
| weshay | sshnaidm|off sanitize the name.. to always have .gz? | 17:14 |
| weshay | thanks! | 17:15 |
| *** kopecmartin is now known as kopecmartin|off | 17:15 | |
| weshay | zbr you still around? | 17:19 |
| zbr | yes, joining now. | 17:19 |
| *** panda|ruck is now known as panda|ruck|off | 17:21 | |
| sshnaidm|off | weshay, they're saved always gzipped to save disk space | 17:21 |
| zbr | https://review.openstack.org/#/c/636160/ | 17:25 |
| *** trown is now known as trown|lunch | 17:31 | |
| *** irclogbot_0 has quit IRC | 17:56 | |
| rfolco|rover | rlandy, reproducer -l on centos is retrieving "There is no node to run jobs", just checking if this is something you hit before... any clues or this would require some debugging/investigation ? | 17:59 |
| rlandy | rfolco|rover: no, where are you running? on a baremetal machine? | 18:00 |
| *** dtantsur is now known as dtantsur|afk | 18:00 | |
| rlandy | I don;t know how libvirt would work on a node of rdocloud | 18:00 |
| rlandy | I have only tested on a mini-dell | 18:01 |
| rlandy | rfolco|rover: do you have logs/tmate to share? | 18:01 |
| *** irclogbot_0 has joined #oooq | 18:02 | |
| rfolco|rover | rlandy, on a beaker box, just fresh centos | 18:02 |
| rfolco|rover | rlandy, I am not sure I should spend time on this | 18:02 |
| rlandy | rfolco|rover: did you hit the same issues as marios? | 18:02 |
| *** ykarel|away has joined #oooq | 18:03 | |
| * rfolco|rover not aware of marios issues | 18:03 | |
| rlandy | <marios> rlandy: fyi got the same on a non kvm box i put a note here for now https://tree.taiga.io/project/tripleo-ci-board/task/766 will revisit next week | 18:03 |
| rlandy | <marios> rlandy: so using --libvirt i get that http://pastebin.test.redhat.com/728999 - i mean makes sense on a kvm guest but this one (according to beaker anyway) is non virtuali | 18:03 |
| rlandy | rfolco|rover: ^^ | 18:03 |
| *** jpena is now known as jpena|off | 18:03 | |
| rlandy | rfolco|rover: if you can run ovb and standalone on rdocloud, you'rep robably good | 18:04 |
| rfolco|rover | rlandy, centos/ovb --> fail, centos/libvirt --> fail | 18:04 |
| rfolco|rover | rlandy, will move to f28 | 18:04 |
| rlandy | weird ... | 18:04 |
| *** jaosorior has quit IRC | 18:04 | |
| rlandy | rfolco|rover: k - let me know | 18:05 |
| rlandy | will look into it | 18:05 |
| rfolco|rover | rlandy, thx | 18:05 |
| *** irclogbot_0 has quit IRC | 18:05 | |
| weshay | rlandy I'll put in a check on the script to test virt capabilities if -l is called | 18:08 |
| weshay | rfolco|rover you got The error was: libvirtError: invalid argument: could not find capabilities for arch=x86_64 domaintype=kvm ? | 18:09 |
| rlandy | weshay: where did you test virt? bm box, right? | 18:09 |
| weshay | on your bm box? | 18:09 |
| rlandy | There is no node to run jobs | 18:09 |
| weshay | rlandy matt young's old laptop :) | 18:09 |
| rlandy | weshay: ^^ rfolco|rover"s error | 18:09 |
| rlandy | weshay: lol | 18:09 |
| weshay | ya.. I'll try it now | 18:09 |
| weshay | rlandy but a test up front to prevent folks from running -l on a vm would be good | 18:10 |
| weshay | I'll add that | 18:10 |
| *** irclogbot_0 has joined #oooq | 18:10 | |
| *** amoralej is now known as amoralej|off | 18:10 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-scenario002-multinode- (2 more messages) | 18:10 |
| rlandy | weshay: ack | 18:12 |
| weshay | rfolco|rover the only reason I can think of why you'd get that error is a missing kernel package | 18:17 |
| weshay | can you cat /proc/cpu and see if the flags are there? | 18:17 |
| rfolco|rover | weshay, flags? but this is not nested virt, is it? | 18:19 |
| weshay | cat /proc/cpuinfo | grep vmx | 18:20 |
| weshay | if that returns as a flag, you should not hit The error was: libvirtError: invalid argument: could not find capabilities for arch=x86_64 domaintype=kvm | 18:20 |
| weshay | if it's not there.. you need to enable it the bios | 18:21 |
| *** derekh has quit IRC | 18:22 | |
| rfolco|rover | afaik weshay this is required for nested virt | 18:22 |
| rfolco|rover | vmx I mean | 18:22 |
| rfolco|rover | I have the vms running look | 18:22 |
| rfolco|rover | [rfolco@rdo-ci-fx2-02-s5 ~]$ sudo virsh list --all | 18:22 |
| rfolco|rover | Id Name State | 18:22 |
| rfolco|rover | ---------------------------------------------------- | 18:22 |
| rfolco|rover | 1 subnode-0 running | 18:22 |
| rfolco|rover | 2 subnode-1 running | 18:22 |
| weshay | oh ok | 18:22 |
| weshay | so that's good | 18:22 |
| weshay | help me understand where the error was hit? | 18:22 |
| rfolco|rover | for some reason nodepool did not add to the pool or something? | 18:23 |
| weshay | rfolco|rover which job are you reproducing? | 18:23 |
| rfolco|rover | reproducing undercloud containers | 18:23 |
| rfolco|rover | http://logs.openstack.org/41/634241/1/check/tripleo-ci-centos-7-undercloud-containers/76e57c3/logs/ | 18:23 |
| weshay | and what was the error? | 18:23 |
| rfolco|rover | /var/tmp/REPRODUCER/roles/ansible-role-tripleo-ci-reproducer/tasks/start.yaml:57 fatal: [localhost]: FAILED! => {"changed": false, "msg": "There is no node to run jobs"} | 18:24 |
| weshay | rlandy unfortunatlely undercloud jobs use the multindoe nodeset | 18:24 |
| rfolco|rover | weshay, there is no node | 18:24 |
| weshay | rfolco|rover are there any containers that are down? | 18:24 |
| rfolco|rover | yes | 18:25 |
| rfolco|rover | zuul | 18:25 |
| rfolco|rover | lol | 18:25 |
| rfolco|rover | ce0512164585 rdoci/zuul:stable "/usr/bin/dumb-ini..." 36 minutes ago Exited (0) 36 minutes ago | 18:25 |
| rlandy | hmmm - shouldn't | 18:25 |
| weshay | that can be fine | 18:25 |
| weshay | while jobs are not running.. I've seen that go down and it comes back | 18:25 |
| * weshay tries that same job http://logs.openstack.org/41/634241/1/check/tripleo-ci-centos-7-undercloud-containers/76e57c3/logs/reproducer-zuul-based-quickstart.tar | 18:26 | |
| rlandy | weshay: https://github.com/openstack-infra/tripleo-ci/blob/master/zuul.d/undercloud-jobs.yaml#L22 undercloud job use the singlenode nodeset | 18:26 |
| rlandy | only libvirt always deploys two nodes | 18:26 |
| rlandy | the fault is in the libvirt template | 18:27 |
| weshay | e0734f1a469e rdoci/zuul:stable "/usr/bin/dumb-ini..." 45 hours ago Exited (0) 45 hours ago tripleo-ci-reproducer_gerritconfig_1 | 18:27 |
| weshay | for example | 18:27 |
| weshay | rlandy k | 18:27 |
| weshay | rfolco|rover trying now | 18:27 |
| rfolco|rover | weshay, good, I'm looking at the logs while my local f27-f28 upgrade goes in my local laptop env | 18:28 |
| rfolco|rover | hell docker-compose logs empty | 18:35 |
| rfolco|rover | doing a manual [rfolco@rdo-ci-fx2-02-s5 tripleo-ci-reproducer]$ docker-compose up -d | 18:36 |
| *** trown|lunch is now known as trown | 18:37 | |
| weshay | rfolco|rover bbiab but it's working for me | 18:45 |
| rfolco|rover | of course :( | 18:46 |
| rfolco|rover | weshay, fedora? | 18:46 |
| rfolco|rover | say yes say yes | 18:46 |
| *** ykarel|away has quit IRC | 18:48 | |
| rfolco|rover | weshay, magically after running docker-compose up and then re-running the reproducer, I was able to run the job | 19:05 |
| rfolco|rover | rlandy, ^ | 19:05 |
| rfolco|rover | http://rdo-ci-fx2-02-s5:9000/t/tripleo-ci-reproducer/status | 19:05 |
| rlandy | how nice | 19:05 |
| rfolco|rover | something happened with docker-compose I think | 19:06 |
| *** rfolco|rover has quit IRC | 19:21 | |
| *** tosky has quit IRC | 19:42 | |
| *** tosky has joined #oooq | 19:43 | |
| *** dmellado has quit IRC | 19:44 | |
| *** dmellado has joined #oooq | 19:45 | |
| *** irclogbot_0 has quit IRC | 19:50 | |
| *** rlandy is now known as rlandy|brb | 19:57 | |
| *** irclogbot_0 has joined #oooq | 20:01 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp- (3 more messages) | 20:10 |
| *** derekh has joined #oooq | 20:26 | |
| *** rfolco has joined #oooq | 20:37 | |
| *** rlandy|brb is now known as rlandy | 20:51 | |
| rlandy | rfolco: hoe's it going - need me to look at anything? | 21:16 |
| rfolco | rlandy, trying on laptop.... will see in a bit | 21:17 |
| rlandy | on your laptop - that is brave | 21:17 |
| weshay | back | 21:27 |
| weshay | the docker-compose.yaml was empty? | 21:28 |
| weshay | my undercloud failed to install | 21:29 |
| weshay | really not a fan of the containre's prepare in tripleo-common | 21:31 |
| rfolco | rlandy, where do you run your repro? vm ? | 21:32 |
| *** irclogbot_0 has quit IRC | 21:37 | |
| rlandy | rfolco: what do mean by my repo? | 21:37 |
| rlandy | oh my reproducer | 21:37 |
| rlandy | on my mini-dell | 21:37 |
| rlandy | so you have a log or error? | 21:38 |
| weshay | rfolco fyi /me made some comments | 21:38 |
| rfolco | weshay, ok, thanks. | 21:39 |
| weshay | rlandy fyi https://review.openstack.org/640538 | 21:43 |
| weshay | panda++ | 21:45 |
| weshay | sshnaidm++ | 21:45 |
| rlandy | looking | 21:46 |
| rlandy | there is such and env var as LIBVIRT_ERROR?? | 21:48 |
| rlandy | not defined anywhere | 21:48 |
| rlandy | weshay: ^^ | 21:48 |
| rlandy | oh I see what you did | 21:49 |
| rlandy | forget that | 21:49 |
| weshay | ? | 21:49 |
| weshay | oh | 21:49 |
| weshay | the || | 21:49 |
| rlandy | I see | 21:51 |
| rlandy | tested | 21:51 |
| rlandy | looks fine | 21:51 |
| * rlandy needs to read the whole line in order | 21:51 | |
| rlandy | it's not == true | 21:52 |
| weshay | aye | 21:54 |
| weshay | - pip conflicts: paramiko, requests, httpd2lib, etc. | 21:56 |
| weshay | ? | 21:56 |
| rfolco | if not too late, https://review.openstack.org/#/c/640538 | 21:57 |
| * weshay wonders if we should use a vm | 21:57 | |
| rfolco | weshay, ^ | 21:57 |
| weshay | virtenv | 21:57 |
| weshay | I mean | 21:57 |
| * weshay looks | 21:57 | |
| rfolco | weshay, prevent -l on vm... so just block if you run from a vm... a vm may or not have vmx | 21:59 |
| weshay | rfolco ya.. I think for now we can limit the scope of what we support.. I need to see if we can test non vmx/smx on a regular basis | 21:59 |
| rfolco | k | 22:00 |
| weshay | non virt capable boxes have to pretty old at this point I think | 22:01 |
| weshay | you know better than me on this one.. though.. I would rather have someone update the bios to allow vmx/smx | 22:01 |
| * weshay doesn't want to support the world.. want to get this working well | 22:02 | |
| weshay | rfolco were you able to resolve the pip conflicts? | 22:02 |
| rfolco | weshay, nah, I am moving to vm, f* my laptop | 22:03 |
| * weshay isn't 100% clear on why we shouldn't use a virtenv.. but I think there was one | 22:03 | |
| rfolco | weshay, my point on your patch is: IMHO you should just check if running on a vm, and bock -l. The way you're checking vmx is not what you need. Try finding something else to check instead of vmx. | 22:06 |
| weshay | if it's a host w/o vmx/smx I'm not sure we want to allow it | 22:07 |
| weshay | as well as not allowing -l from a guest | 22:08 |
| weshay | if that is sane | 22:08 |
| rfolco | so 2 conditions | 22:10 |
| *** rlandy has quit IRC | 22:10 | |
| rfolco | 1) block host if not kvm capable | 22:10 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp- (3 more messages) | 22:11 |
| rfolco | 2) block vm (even if nested virt capable) | 22:11 |
| rfolco | I was focused on #2 | 22:11 |
| weshay | ah k | 22:11 |
| weshay | thanks for clarifying that | 22:11 |
| weshay | I'll update it | 22:11 |
| rfolco | thats a command also... virt-host-validate | 22:11 |
| rfolco | much more complete | 22:11 |
| weshay | ugh.. can you make two pull requests w/o them being stacked? | 22:21 |
| weshay | github sucks. | 22:24 |
| weshay | confirmed.. it sucks | 22:34 |
| *** vinaykns has quit IRC | 22:36 | |
| weshay | rfolco fyi.. https://bugs.launchpad.net/tripleo/+bug/1818305 | 22:36 |
| openstack | Launchpad bug 1818305 in tripleo "overcloud-full image fails to build calling mkfs -t xfs " [Critical,Triaged] | 22:36 |
| weshay | two patches added to sova https://github.com/sshnaidm/sova/pull/64 | 22:36 |
| weshay | no action required | 22:37 |
| *** ajo_ has joined #oooq | 22:45 | |
| *** Tengu_ has joined #oooq | 22:47 | |
| weshay | rfolco just fyi.. undercloud install is failing twice on the repoducer at | 22:50 |
| weshay | 2019-03-01 22:31:52.356 27500 WARNING tripleoclient.v1.tripleo_deploy.Deploy [ ] TASK [Run tripleo-container-image-prepare logged to /var/log/tripleo-container-image-prepare.log] *** | 22:50 |
| weshay | which usually means the docker containers did not download | 22:50 |
| weshay | but looking at it.. | 22:51 |
| weshay | rfolco unless that patch really broke something | 22:52 |
| weshay | and it may have | 22:52 |
| *** Tengu has quit IRC | 22:52 | |
| *** vkmc has quit IRC | 22:52 | |
| *** ajo has quit IRC | 22:52 | |
| *** ajo_ is now known as ajo | 22:52 | |
| weshay | rfolco everythign failed on that patch | 22:53 |
| weshay | except amazing undercloud upgrades | 22:53 |
| *** vkmc has joined #oooq | 22:53 | |
| weshay | which makes me think the patch was not applied | 22:53 |
| weshay | rfolco you may want to try a recreate from a successful job | 22:53 |
| weshay | first | 22:53 |
| *** rascasoft has quit IRC | 23:07 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!