| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-quickstart-extras-gate-master-delorean-quick-basic, tripleo- (3 more messages) | 00:41 |
|---|---|---|
| *** ssbarnea has quit IRC | 00:52 | |
| *** ccamacho has quit IRC | 01:24 | |
| *** ccamacho has joined #oooq | 01:25 | |
| *** chem has quit IRC | 02:11 | |
| *** ykarel has joined #oooq | 02:34 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-quickstart-extras-gate-master-delorean-quick-basic, tripleo- (3 more messages) | 02:41 |
| *** udesale has joined #oooq | 03:47 | |
| *** huynq has joined #oooq | 04:11 | |
| ykarel | is this already known and worked upon: INFO:kolla.common.utils.crane:No package mod_xsendfile available | 04:39 |
| ykarel | master container build failing while building crane | 04:39 |
| ykarel | https://github.com/openstack/kolla/blob/master/docker/crane/Dockerfile.j2#L15 | 04:39 |
| ykarel | ruck rover ^^ | 04:40 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-quickstart-extras-gate-master-delorean-quick-basic, tripleo- (3 more messages) | 04:41 |
| *** saneax has joined #oooq | 05:09 | |
| *** ykarel has quit IRC | 05:21 | |
| *** ykarel has joined #oooq | 05:21 | |
| *** jaganathan has joined #oooq | 05:48 | |
| *** ykarel has quit IRC | 05:51 | |
| *** ratailor has joined #oooq | 05:58 | |
| *** saneax has quit IRC | 06:07 | |
| *** jfrancoa has joined #oooq | 06:35 | |
| *** quiquell|off is now known as quiquell | 06:37 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-quickstart-extras-gate-master-delorean-quick-basic, tripleo- (3 more messages) | 06:41 |
| *** chkumar|off is now known as chandankumar | 07:13 | |
| *** ccamacho has quit IRC | 07:15 | |
| *** ccamacho has joined #oooq | 07:15 | |
| chandankumar | nhicher: arxcruz https://review.openstack.org/#/c/614321/ will fix the issue | 07:17 |
| *** saneax has joined #oooq | 07:18 | |
| *** ykarel has joined #oooq | 07:42 | |
| *** ykarel has quit IRC | 07:49 | |
| *** ykarel has joined #oooq | 07:50 | |
| *** gkadam has joined #oooq | 07:54 | |
| *** ykarel has quit IRC | 07:56 | |
| *** ykarel has joined #oooq | 08:02 | |
| *** sshnaidm|off is now known as sshnaidm|ruck | 08:03 | |
| *** rascasoft has joined #oooq | 08:04 | |
| *** jaganathan has quit IRC | 08:08 | |
| *** holser_ has joined #oooq | 08:09 | |
| *** ykarel has quit IRC | 08:10 | |
| *** jtomasek has joined #oooq | 08:18 | |
| *** amoralej|off is now known as amoralej | 08:23 | |
| *** ykarel has joined #oooq | 08:27 | |
| *** ykarel has quit IRC | 08:28 | |
| *** ykarel has joined #oooq | 08:29 | |
| *** ykarel_ has joined #oooq | 08:31 | |
| *** ykarel has quit IRC | 08:34 | |
| *** ykarel has joined #oooq | 08:34 | |
| *** ykarel_ has quit IRC | 08:38 | |
| *** ykarel has quit IRC | 08:40 | |
| *** ssbarnea has joined #oooq | 08:40 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-quickstart-extras-gate-master-delorean-quick-basic, tripleo- (3 more messages) | 08:41 |
| ssbarnea | morning! i am wondering why nobody raise this issue... we have ~15 jobs run, lets say we have a 5% random failure rate.... have you computed the probability for the gate to pass? | 08:46 |
| ssbarnea | (1-0.05)^15 = 0.46 .... and this is for the gate itself. | 08:47 |
| ssbarnea | if we count both check and gate it would be like: 21% instead of 46% | 08:47 |
| ssbarnea | and this by assuming your change is "perfect". | 08:48 |
| ssbarnea | sshnaidm|ruck : if you can review https://review.openstack.org/#/c/613797/ now it would be great. check for last zuul rezults downvote due to one random flakiness. | 08:58 |
| *** panda|off is now known as panda | 09:01 | |
| *** d0ugal has joined #oooq | 09:02 | |
| *** chem has joined #oooq | 09:05 | |
| sshnaidm|ruck | ssbarnea, I still wait for answers for my comments | 09:12 |
| sshnaidm|ruck | ssbarnea, please, don't forge to reply to people issues, it's very important | 09:12 |
| sshnaidm|ruck | ssbarnea, I don't want now to list all patchsets to find what was fixed and what not, please respect your colleagues time | 09:13 |
| *** bogdando has joined #oooq | 09:21 | |
| sshnaidm|ruck | ssbarnea, commented on patch | 09:22 |
| sshnaidm|ruck | ssbarnea, I kindly remind you that it blocks promotion | 09:24 |
| sshnaidm|ruck | panda, wrt: https://review.rdoproject.org/r/#/c/17102 - I don't think it should be in journald. Firstly these files should be available, better by http, which is impossible in journald case. I'd like to link to them or send them to somebody w/o logging in to te-broker. | 09:36 |
| sshnaidm|ruck | panda, secondly, it's much faster and more convenient to lurk in one file then starting to grep journald. This logs are not logs from one service, these logs are from different machines, it's different | 09:37 |
| *** kopecmartin|off is now known as kopecmartin | 10:05 | |
| *** apetrich has quit IRC | 10:12 | |
| quiquell | panda, sshnaidm|ruck: Do you know if we have to run jobs at fedora28 with qemu or kvm ? | 10:13 |
| quiquell | right now they are running at qemu | 10:13 |
| sshnaidm|ruck | quiquell, hmm.. where do you see qemu? | 10:14 |
| quiquell | sshnaidm|ruck: http://logs.openstack.org/90/614090/1/gate/tripleo-ci-centos-7-containers-multinode/6dadb67/job-output.txt.gz | 10:14 |
| quiquell | sshnaidm|ruck: I think we are missing the kernel module for kvm there | 10:14 |
| quiquell | sshnaidm|ruck: I suppose it's better to use kvm ? | 10:14 |
| sshnaidm|ruck | quiquell, where exactly? | 10:15 |
| quiquell | wait didn't copy the pointer | 10:15 |
| quiquell | sshnaidm|ruck: http://logs.openstack.org/97/613297/10/check/tripleo-ci-fedora-28-standalone/2b37c5a/job-output.txt.gz#_2018-10-31_07_36_54_930751 | 10:16 |
| sshnaidm|ruck | quiquell, I think it's relevant for libvirt jobs only | 10:17 |
| sshnaidm|ruck | quiquell, we don't run vm on infra nodes | 10:17 |
| quiquell | sshnaidm|ruck: So this is not relevant | 10:18 |
| sshnaidm|ruck | quiquell, yeah, sure not on undercloud node | 10:18 |
| quiquell | sshnaidm|ruck: Nah is the same at centos standalone, thanks | 10:19 |
| rfolco|rover | quiquell, this is because it tries to load the kvm module and failed. The host has no support to it or kvm full accel mode is disabled in the host. | 10:19 |
| huynq | Hello! Did you get SSLError when import overcloud nodes? | 10:19 |
| huynq | http://paste.openstack.org/show/733674/ | 10:19 |
| quiquell | rfolco|rover: ack | 10:19 |
| sshnaidm|ruck | huynq, I didn't see it before, can you please check that time on machine is synced with ntp? | 10:20 |
| rfolco|rover | sshnaidm|ruck, you know if/how podman mirror is configured ? did you file any bug for the docker registry failures on gate ? | 10:24 |
| huynq | sshnaidm|ruck: I have just synced then do installation command again. It's still failed | 10:24 |
| rfolco|rover | sshnaidm|ruck, context - see failures recorded at https://review.rdoproject.org/etherpad/p/ruckrover-sprint21 | 10:25 |
| rfolco|rover | sshnaidm|ruck, and last night chat on openstack-infra | 10:26 |
| sshnaidm|ruck | rfolco|rover, are we talking about podman crash? | 10:26 |
| rfolco|rover | not sure if this is same issue | 10:27 |
| sshnaidm|ruck | rfolco|rover, logs..> | 10:27 |
| sshnaidm|ruck | ? | 10:27 |
| rfolco|rover | https://review.rdoproject.org/etherpad/p/ruckrover-sprint21 lines 21-29 | 10:27 |
| *** apetrich has joined #oooq | 10:27 | |
| rfolco|rover | sshnaidm|ruck, HTTPError: 503 Server Error... HTTP Error 502 - Bad Gateway... etc | 10:28 |
| sshnaidm|ruck | rfolco|rover, well, that's a lot of different errors | 10:29 |
| rfolco|rover | sshnaidm|ruck, it seems that podman mirror/proxy/cache whatever is not configured | 10:29 |
| sshnaidm|ruck | rfolco|rover, how do you see it? | 10:31 |
| rfolco|rover | sshnaidm|ruck, I don't. Per Paul's comments plus Alex's comments.... | 10:31 |
| rfolco|rover | sshnaidm|ruck, look... | 10:32 |
| rfolco|rover | <mwhahaha> clarkb: heads up, that job isn't using docker which is why that docker config file is not configured with a mirror. I will have to track down where the podman mirror config is tho | 10:32 |
| *** huynq has quit IRC | 10:37 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024 @ https://review.openstack.org/602248, master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci- (3 more messages) | 10:41 |
| *** dtantsur|afk is now known as dtantsur | 10:49 | |
| panda | sshnaidm|ruck: That change is about the console log deletion script and its output 1) log rotation is a system administration activity, I don't understand why you would want to expose that log to someone that has no access to the system, but systemd offers already a facility to expose logs via http with an integrated filtering system, configuring it it's not particularly difficult 2) journald was created | 10:54 |
| panda | specifically to filter and explore logs according to rules, why would I want to use grep above message level ? If you need a single file you can always journalctl -t tebroker-console-log | less . I don't see the advantage in creating these exception and distributing logs aroud the file system and handle them manually, or create our own scripts to handle them worse than journald could do for us. | 10:54 |
| sshnaidm|ruck | panda, ok, I understood it wrong, I thought it's about bmc logs | 10:59 |
| ssbarnea | panda sshnaidm|ruck quiquell : I propose to rename NEED_SUDO to NEED_PKG_MGR to better indicate that calling it is needed. I do not fancy panda proposal of NEED_PACKAGES because pip wheels are still packages, it does not say we need system packages. but "pkg_mgr" is already used with this meaning. Ok? | 10:59 |
| sshnaidm|ruck | ssbarnea, I don't think the problem is with name | 10:59 |
| ssbarnea | as we all know naming variables is one of the hardest programming challenges ;) | 10:59 |
| quiquell | ssbarnea, sshnaidm|ruck: also we cannot install python-virtualenv at fedora28 zuul nodes, they forbide it from dnf.conf | 11:00 |
| sshnaidm|ruck | quiquell, why so? do they preinstall it? | 11:01 |
| ssbarnea | quiquell it shoud not be a practical problem because there we do not expect to reach "sudo" part because all deps should already be installed. | 11:01 |
| quiquell | sshnaidm|ruck: yep, virtualenv, pip and setuptools are the one forbidden | 11:02 |
| quiquell | sshnaidm|ruck: We can override dnf.conf but pufff | 11:02 |
| quiquell | sshnaidm|ruck: That's why I did a patch so we don't install them if they are alaredy installed | 11:02 |
| ssbarnea | one of the reasons why we are attempting to install these at user level. | 11:02 |
| quiquell | sshnaidm|ruck, ssbarnea, panda: Some of the package has to be conditional and others don't | 11:03 |
| ssbarnea | quiquell I didn't see any failure to run this patch upstream in its current form. yeah, in the future we will improve the logic, but now we need to make it run. not to fix all corner cases, it could take many weeks to address all cases. | 11:05 |
| sshnaidm|ruck | quiquell, I'm fine with that | 11:05 |
| quiquell | ssbarnea: Agree just saying that it's not alwys true that we don't have stuff installed at CI upstream | 11:06 |
| ssbarnea | molecute testing passed on f26/f28/centos7 with pure os images, also passed upstream. | 11:06 |
| sshnaidm|ruck | ssbarnea, as it was said it's pretty urgent, if you think that should behave different from what it was before - you can submit a different followup patch and we can discuss it | 11:07 |
| quiquell | ssbarnea, sshnaidm|ruck, panda: there is nothing wrong functionally wrong here let's merge this to fix promotions | 11:07 |
| sshnaidm|ruck | ssbarnea, for now I'd like the current behavior to be fixed finally | 11:07 |
| sshnaidm|ruck | quiquell, I need installation to be out of condition, as all dev/nulls removed, I don't want to waste time on investigations what was the output there | 11:08 |
| quiquell | sshnaidm|ruck: so 1> /dev/null would be good enough ? | 11:08 |
| ssbarnea | i will remove stderr hiding part | 11:08 |
| quiquell | ssbarnea: cool | 11:08 |
| quiquell | ssbarnea, sshnaidm|ruck: About the list we can put them outside and just add sudo as prefix if needed | 11:09 |
| quiquell | ssbarnea: I will fix stuff for fedora28 if needed later on | 11:09 |
| quiquell | sshnaidm|ruck: ^ | 11:09 |
| sshnaidm|ruck | ssbarnea, quiquell you can use a special log file for that, I never felt this output hurts any way, let's not solve non-existing problems | 11:09 |
| panda | the output of taht command is either nothing or a version number | 11:10 |
| quiquell | panda: ack, there is no 'not command found' ? | 11:11 |
| panda | it's used for the return value | 11:11 |
| quiquell | panda: ack | 11:11 |
| panda | quiquell: no, we assume python is already installed | 11:11 |
| panda | quiquell: so taht command either returns nothing and exit 1 in python 2 or returns a version number uin python3 | 11:11 |
| sshnaidm|ruck | cool, I will know what version of pip there | 11:12 |
| panda | well sterr returns "ther is not module called virtualenv " in python2 | 11:12 |
| panda | quiquell: ssbarnea I need to talk to you | 11:14 |
| ssbarnea | sshnaidm|ruck best case (already present) https://seashells.io/v/p5NfQFju | 11:15 |
| *** jtomasek has quit IRC | 11:16 | |
| sshnaidm|ruck | ssbarnea, you need to remove condition also | 11:17 |
| ssbarnea | sshnaidm|ruck: i am against removing condition because it would break qs use when you do not have root access, it would be a clear regression. | 11:19 |
| ssbarnea | if you really want you can add NEED_SUDO=true on your .profile file :) | 11:19 |
| ssbarnea | i do like the fact that --install-deps could succeed without having to sudo, is an important feature. | 11:20 |
| ssbarnea | or you want to change the default value to "true"? which I hate but if we do it temporary until we fix the detection logic, it would be ok-ish. | 11:21 |
| nhicher | hello chandankumar, I added a depends-on on https://review.openstack.org/#/c/614321 for https://review.rdoproject.org/r/#/c/16566 | 11:22 |
| nhicher | chandankumar: also, I added a comment on https://review.rdoproject.org/r/#/c/16566/ | 11:22 |
| ssbarnea | this is the execution result when git was missing: https://seashells.io/v/pJzsxxq8 | 11:22 |
| panda | ssbarnea: what if libyaml is missing ? | 11:23 |
| panda | ssbarnea: or openssl-devel ? | 11:23 |
| ssbarnea | panda: this is not covered, but was not covered even before. | 11:24 |
| panda | ssbarnea: no, before it was just installed. | 11:24 |
| panda | ssbarnea: quiquell I need to know what's the status on fedora28 user stories and if we need to swarm | 11:25 |
| *** ratailor has quit IRC | 11:26 | |
| panda | sprint ends next week and we need to close the standalone | 11:26 |
| ssbarnea | panda: ok, I am makeing NEED_SUDO:=true by default and planning to make it false only when we get the verifications right. | 11:26 |
| quiquell | panda: standalone at upstream ci with the reviews in tasks | 11:27 |
| quiquell | panda: Found an issue with build-test-package I have a fix for it | 11:27 |
| quiquell | panda: but if we merge all the reviews then we are good | 11:28 |
| panda | ssbarnea: is that part important to reach the goal of having fedora28 + standalone in CI system ? | 11:28 |
| sshnaidm|ruck | ssbarnea, I don't see it as a feature at all and if you do - submit it as separate patch for team wide discussion | 11:28 |
| panda | quiquell: how much good we are ? | 11:28 |
| quiquell | panda: that parts fix regressions so it's even more important | 11:28 |
| panda | quiquell: all the reviews in fedora28 topic ? | 11:28 |
| quiquell | panda: yep, missing the build-test-packages one | 11:29 |
| sshnaidm|ruck | ssbarnea, please remove this "feature" completely, it's a bug, we don't have time for it now | 11:29 |
| quiquell | panda: have issues with software facotry | 11:29 |
| ssbarnea | updated, now sudo runs by default, feel free to check https://review.openstack.org/#/c/613797/ | 11:31 |
| panda | quiquell: how good we are ? I mean a full standalone f28 run ? | 11:32 |
| *** udesale has quit IRC | 11:32 | |
| panda | quiquell: ssbarnea how far we are from the full f28 standaonle run in CI ? | 11:34 |
| sshnaidm|ruck | ssbarnea, and commented again | 11:34 |
| sshnaidm|ruck | ssbarnea, please read again my comment about separate patch and team-wide discussion | 11:34 |
| quiquell | panda: yep full | 11:36 |
| ssbarnea | sshnaidm|ruck i don't find this pressure pointing as constructive, as in forcing me to remove a featured that was merged because you don't like it. | 11:36 |
| quiquell | panda: http://logs.openstack.org/97/613297/10/check/tripleo-ci-fedora-28-standalone/2b37c5a/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz#_2018-10-31_08_04_42 | 11:36 |
| quiquell | panda: we have a fix for the last line there "must have exactly one of create/read/write/append mode" | 11:37 |
| quiquell | panda: is one of the tasks | 11:37 |
| quiquell | panda: but we have to merge all the stuff there | 11:37 |
| panda | quiquell: ok so no need to swarm on cards, better to swarm on reviews. | 11:38 |
| panda | quiquell: I still see some cards on parked | 11:38 |
| panda | quiquell: too many cards in my opinion | 11:38 |
| panda | ssbarnea: let's have a chat, your bj ? | 11:38 |
| quiquell | panda: They can be for the next sprint | 11:38 |
| *** jtomasek has joined #oooq | 11:38 | |
| ssbarnea | https://bluejeans.com/2655417928 | 11:38 |
| quiquell | panda: just stuff I miss to have f28 in the pipeline with all the features | 11:39 |
| quiquell | panda: I can mark the parked ones with ones for the next sprint and bug ones | 11:39 |
| sshnaidm|ruck | ssbarnea, was merged a bug, not a feature, and this is urgent because it blocks now promotion | 11:54 |
| *** udesale has joined #oooq | 11:54 | |
| sshnaidm|ruck | ssbarnea, panda explained you above what's wrong about this functionality, I can't understand why you continue to argue and not fixing your bug | 11:54 |
| quiquell | panda, sshnaidm|ruck: after fixing rebase conflicts we need some reviewing here https://review.openstack.org/#/c/610491/ | 11:56 |
| quiquell | it's py3 at tqe | 11:56 |
| sshnaidm|ruck | quiquell, let's see CI jobs finish there | 11:58 |
| quiquell | sshnaidm|ruck: ack, thanks | 11:58 |
| *** jfrancoa has quit IRC | 11:58 | |
| *** jfrancoa has joined #oooq | 12:01 | |
| *** trown|outtypewww is now known as trown | 12:05 | |
| *** panda is now known as panda|lunch | 12:15 | |
| weshay | sshnaidm|ruck, rfolco|rover howdy | 12:23 |
| sshnaidm|ruck | weshay, hey | 12:23 |
| weshay | did podman get reverted? | 12:23 |
| ssbarnea | panda|lunch what can we do to get https://review.openstack.org/#/c/613920/ merged? somehow it seems impossible to pass even the checks! | 12:24 |
| rfolco|rover | sshnaidm|ruck, weshay: mirror bug for podman https://bugs.launchpad.net/tripleo/+bug/1800748 | 12:24 |
| openstack | Launchpad bug 1800748 in tripleo "undercloud-containers job does not configure a docker mirror when podman is the container cli" [High,Triaged] | 12:24 |
| weshay | hrm.. I see Emilien's comments | 12:24 |
| weshay | ah interesting | 12:25 |
| sshnaidm|ruck | ssbarnea, I will handle https://review.openstack.org/#/c/613797 | 12:25 |
| ssbarnea | sshnaidm|ruck in which way? i just posted a cleaned up version | 12:26 |
| jaosorior | weshay: security squad meeting on #tripleo. What's up? | 12:27 |
| sshnaidm|ruck | ssbarnea, it doesn't solve the problem, I'll handle it | 12:28 |
| weshay | ah sorry | 12:28 |
| Tengu | weshay: hey :). do you know if it's possible to push a common host_prep_tasks in t-h-t, like it's done for the step 1 tasks ? | 12:30 |
| Tengu | weshay: that should prevent any race condition with the config-data directory, although I'm not 100% sure it actually IS a race condition (apparently it's random right?) | 12:31 |
| jaosorior | Tengu: if that's needed for all the roles, we could just add that to common/deploy-step-tasks.yaml | 12:34 |
| Tengu | jaosorior: well, it's done already, but apparently it seems to create some issues :/. context: https://bugs.launchpad.net/tripleo/+bug/1800737 | 12:34 |
| openstack | Launchpad bug 1800737 in tripleo "relabel failed /var/lib/config-data: no such file or directory" [Critical,Triaged] | 12:34 |
| ssbarnea | does anyone node why scenario004 fails? i see something with 'ephemeral_device' is undefined" | 12:36 |
| rfolco|rover | ssbarnea, hmm I remember seeing this error before... searching | 12:37 |
| ssbarnea | rfolco|rover http://logs.openstack.org/20/613920/6/check/tripleo-ci-centos-7-scenario004-multinode-oooq-container/9b1e006/job-output.txt.gz | 12:37 |
| chandankumar | nhicher: updating a new review | 12:38 |
| *** rlandy has joined #oooq | 12:38 | |
| ssbarnea | rfolco|rover that was not the real error, is just a "warning". | 12:41 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024 @ https://review.openstack.org/602248, master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci- (3 more messages) | 12:41 |
| rfolco|rover | ssbarnea, something happened to ssh connection to primary node --> http://logs.openstack.org/20/613920/6/check/tripleo-ci-centos-7-scenario004-multinode-oooq-container/9b1e006/job-output.txt.gz#_2018-10-31_02_49_29_973113 | 12:41 |
| rfolco|rover | it worked on secondary though | 12:41 |
| rfolco|rover | ssbarnea, hard to tell what happened... ephemeral task in ara says: UNREACHABLE | 12:45 |
| weshay | panda|lunch, need your comments or ack on https://tree.taiga.io/project/tripleo-ci-board/epic/298 | 12:45 |
| rfolco|rover | ssbarnea, and after that, tasks executed only on secondary, which means primary is gone | 12:46 |
| rfolco|rover | ssbarnea, a recheck is your best friend now I guess | 12:46 |
| quiquell | rfolco|rover: We need some tshirts with 'recheck' on it and a cat or the like | 12:48 |
| ssbarnea | rfolco|rover please look at the build history of this CR https://review.openstack.org/#/c/613920/ -- and see the results. this one is critical to get in but we fail to even reach the gate part. already did more than 3 rechecks on it. | 12:48 |
| rfolco|rover | :) | 12:49 |
| rfolco|rover | ssbarnea, checking | 12:49 |
| *** sshnaidm|ruck is now known as sshnaidm|bbl | 12:50 | |
| *** quiquell is now known as quiquell|lunch | 12:51 | |
| rfolco|rover | ssbarnea, failed not always in the same job | 12:51 |
| rfolco|rover | tripleo-ci-centos-7-containers-multinode-queens TIMED_OUT.... then tripleo-ci-centos-7-undercloud-containers FAILURE.... then tripleo-ci-centos-7-scenario004-multinode-oooq-container POST_FAILURE... then tripleo-ci-centos-7-scenario004-multinode-oooq-container FAILURE again | 12:53 |
| weshay | rfolco|rover, sshnaidm|bbl fyi.. http://logstash.openstack.org/#/dashboard/file/logstash.json?query=build_status:%20FAILURE%20AND%20message:%20%5C%22panic:%20runtime%20error%5C%22 | 12:53 |
| jaosorior | weshay, panda|lunch: would it be possible to add me as a team member here https://tree.taiga.io/project/tripleo-ci-board/team so I can comment on epics and such? | 12:55 |
| *** gkadam has quit IRC | 12:57 | |
| panda|lunch | jaosorior: done and done | 12:57 |
| weshay | chandankumar, arxcruz kopecmartin | 13:01 |
| weshay | in my channel now please | 13:01 |
| arxcruz | weshay: you're not there | 13:02 |
| kopecmartin | weshay, sorry i can't, I'm officially on PTO, I'm at school, however I will be available in one hour | 13:02 |
| jaosorior | panda|lunch: thanks | 13:02 |
| *** panda|lunch is now known as panda | 13:02 | |
| *** gkadam has joined #oooq | 13:03 | |
| panda | jaosorior: invitation sent, tell me if you have problems. | 13:03 |
| panda | arxcruz: rfolco's channel in the invitation | 13:03 |
| weshay | kopecmartin, k | 13:03 |
| weshay | kopecmartin, ya.. see that on the cal, enjoy | 13:04 |
| panda | jaosorior: ok I see you. | 13:04 |
| chandankumar | arxcruz: https://bluejeans.com/5878458097/ | 13:04 |
| jaosorior | panda: worked. | 13:04 |
| jaosorior | thanks! | 13:04 |
| ssbarnea | rfolco|rover : do you know if we have special reason for not using any_errors_fatal to fail fast instead of continuing only with secondary one in case of failure? | 13:09 |
| rlandy | ssbarnea: bm master job is doing better now | 13:10 |
| *** agopi is now known as agopi|brb | 13:18 | |
| weshay | rfolco|rover, help Emilien w/ http://logstash.openstack.org/#/dashboard/file/logstash.json?query=build_status:%20FAILURE%20AND%20message:%20%5C%22panic:%20runtime%20error%5C%22 | 13:19 |
| rfolco|rover | ack weshay | 13:19 |
| ajo | folks | 13:21 |
| ajo | has anybody seen this Failed to create resource provider record in placement API for ....Got 500 ... | 13:21 |
| ajo | https://paste.fedoraproject.org/paste/zr3DMFrGixIdr6sq1h-FPQ | 13:21 |
| ajo | I can't create instances | 13:21 |
| ajo | neither in rdo/rocky or rdo/master | 13:22 |
| ajo | with ooq | 13:22 |
| *** agopi|brb has quit IRC | 13:24 | |
| rfolco|rover | ssbarnea, I don't quite understand it. It has failed on secondary node (undefined ephemeral error)... but the ephemeral task ran fine there on secondary... | 13:32 |
| ssbarnea | rfolco|rover : undefined variable is the message from "debug" module, not a real error. is just a confusing message, not a real error. | 13:33 |
| ssbarnea | if I would have time i would rewrite this code to avoid this message. | 13:33 |
| rfolco|rover | ssbarnea, I think ansible continues the rest of plays for unreachable, it would abort if any fatal error instead | 13:35 |
| rfolco|rover | you may know this better than I do professor sorin :) | 13:36 |
| ssbarnea | rfolco|rover: i don't have time to teach myself enough new tricks,.... but AFAIK without any_error_fatal, ansible will continue execution on remaining hosts. | 13:39 |
| ssbarnea | for log collection or similar tasks where you expect failures but where you just want a best-efforth attempt, you don't want to fail-fast, but for the rest, probably you want. | 13:39 |
| rfolco|rover | yep thats why it continues, its unreachable, not a failed task | 13:40 |
| rfolco|rover | ssbarnea, I did a quick search, to prevent ansible to continue on unreachable, set this in the playbook: max_fail_percentage: 0 | 13:44 |
| rfolco|rover | never used it though :) | 13:44 |
| rfolco|rover | ok back to gate nightmare | 13:45 |
| *** amoralej is now known as amoralej|lunch | 13:47 | |
| *** agopi has joined #oooq | 13:48 | |
| *** vinaykns has joined #oooq | 13:55 | |
| *** quiquell|lunch is now known as quiquell | 14:00 | |
| weshay | quiquell, you still blocked on https://tree.taiga.io/project/tripleo-ci-board/task/296?kanban-status=1447276 | 14:05 |
| weshay | quiquell, https://softwarefactory-project.io/r/#/c/14080/ | 14:05 |
| quiquell | weshay: it has just hit pypi | 14:08 |
| quiquell | weshay: unblocked one need another review to change versions of it | 14:08 |
| kopecmartin | weshay, if you wanted to discuss planning meeting, I'm available | 14:12 |
| quiquell | weshay: we need this now https://review.openstack.org/#/c/614516/ the pining of the rdopkg version | 14:13 |
| *** d0ugal has quit IRC | 14:14 | |
| weshay | panda, can we open up comment access on our taiga project? | 14:20 |
| weshay | panda, or does *everyone* need an id :( | 14:20 |
| panda | weshay: there's a generic user, even anonymous | 14:21 |
| panda | weshay: I can open comment to everyone | 14:21 |
| weshay | panda++ | 14:21 |
| hubbot1 | weshay: panda's karma is now 6 | 14:21 |
| panda | weshay: separately on taks, epics and user stories | 14:21 |
| panda | I hope it doesnt' expose us to spam | 14:21 |
| weshay | panda, probably fine to have comment access on any | 14:21 |
| weshay | panda, ya.. we'll see | 14:21 |
| *** d0ugal has joined #oooq | 14:27 | |
| *** d0ugal has quit IRC | 14:32 | |
| quiquell | marios: Has an idea about getting the variables from jobs | 14:40 |
| quiquell | marios: Can I add a PS to your wip ? | 14:40 |
| marios | quiquell: of course man but which one? | 14:41 |
| marios | i mean which wip you mean | 14:41 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024 @ https://review.openstack.org/602248, master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci- (3 more messages) | 14:41 |
| quiquell | https://review.openstack.org/#/c/613571 | 14:41 |
| marios | quiquell: sure but can we let the current one run firs tplease? | 14:41 |
| marios | quiquell: i'd like to see what happens | 14:41 |
| marios | quiquell: :) | 14:41 |
| marios | quiquell: then update? | 14:41 |
| quiquell | marios: Will prepare new review in the previous proto | 14:41 |
| marios | quiquell: or maybe add it as new patch ontop of it? | 14:42 |
| marios | quiquell: ack thx | 14:42 |
| marios | quiquell: the to_json is awful i doubt we'll keep it :/ like http://logs.openstack.org/71/613571/6/check/tripleo-ci-centos-7-standalone/fc1eed7/job-output.txt.gz#_2018-10-31_12_23_27_067632 | 14:42 |
| *** saneax has quit IRC | 14:42 | |
| marios | quiquell: rlandy i'm also playing with ara at https://review.openstack.org/#/c/613572/3/roles/collect-logs/tasks/publish.yml | 14:43 |
| rlandy | rascasoft: updated/rebased all patches - kicked the test job again for ha-utils - stuck right now on te-broker | 14:45 |
| rlandy | rascasoft: sorry - no more progress to report :( | 14:46 |
| rlandy | marios; are we done here: https://tree.taiga.io/project/tripleo-ci-board/issue/219?kanban-status=2027733? | 14:46 |
| quiquell | rlandy, marios: For the vars an idea https://review.openstack.org/614528 | 14:47 |
| quiquell | rlandy, marios: including the inventory.yml (don't know if it already exists) | 14:47 |
| quiquell | so we dump "{{ all.vars }}" into zuul variables yaml | 14:47 |
| *** quiquell is now known as quiquell|brb | 14:48 | |
| weshay | quiquell|brb, ssbarnea push push push on f28 | 14:53 |
| marios | quiquell|brb: ack checking (looks like it doesn't like you doing that maybe there is some option you need) | 14:53 |
| marios | adding commend | 14:53 |
| marios | t | 14:53 |
| weshay | panda, did you update tagia? | 14:54 |
| *** amoralej|lunch is now known as amoralej | 14:55 | |
| panda | weshay: partially | 14:55 |
| ssbarnea | weshay :( ... i am afraid that at this point 95% of speed is controlled by zuul. should I become religious? | 14:55 |
| *** d0ugal has joined #oooq | 14:55 | |
| weshay | ssbarnea, ya.. I understand for sure | 14:56 |
| weshay | ssbarnea, just make sure it's your #1 priority | 14:56 |
| weshay | that's all I can ask | 14:56 |
| *** quiquell|brb is now known as quiquell | 14:56 | |
| quiquell | weshay: We are almost there, success deployment at upstream CI is in the squash review | 14:57 |
| weshay | all in case you have not seen it | 14:58 |
| weshay | https://tree.taiga.io/project/tripleo-ci-board/wiki/notes-from-email | 14:58 |
| weshay | WHen I find something I think everyone should know.. I'm adding it to this wiki | 14:58 |
| weshay | on the daily | 14:58 |
| quiquell | Can we subscribe to it ? | 14:58 |
| weshay | don't know | 14:59 |
| panda | unfortunately not. THis will be a topic on retrospective. | 14:59 |
| *** dsneddon has quit IRC | 14:59 | |
| weshay | panda, logged out.. can't comment on https://tree.taiga.io/project/tripleo-ci-board/epic/298 | 15:00 |
| marios | thanks weshay | 15:00 |
| quiquell | weshay: So tripleo is eating all zuul ? | 15:00 |
| weshay | quiquell, 53% of the resources | 15:00 |
| weshay | quiquell, that's why need to go w/ standalone upstream | 15:01 |
| quiquell | ... /o\ | 15:01 |
| weshay | quiquell, we'll get there | 15:02 |
| rlandy | marios: not sure what you are saying in comment https://review.openstack.org/#/c/613678/5/toci-quickstart/config/collect-logs.yml | 15:02 |
| quiquell | weshay: for the f28 thing https://review.openstack.org/#/c/614516/ rdopkg version upgrade | 15:02 |
| marios | rlandy: nothing just pointing to it in the logs incase someone went looking as i did | 15:03 |
| quiquell | marios: I can copy it :-) | 15:04 |
| marios | quiquell: :) | 15:04 |
| quiquell | marios: trying shell cp it before :-P | 15:05 |
| ssbarnea | weshay: if you have few minutes, maybe you can help me understand something. | 15:07 |
| *** dsneddon has joined #oooq | 15:08 | |
| weshay | panda, trying to add cybertron | 15:08 |
| weshay | panda, do I need access to add people? | 15:08 |
| weshay | to our project? if so please grant it | 15:09 |
| quiquell | WHat's cybertron ? | 15:09 |
| weshay | quiquell, ben nemec | 15:09 |
| weshay | the guy who came up w/ ovb | 15:09 |
| panda | it's the planet of the transformers, duh | 15:09 |
| quiquell | weshay: Ahh it is a human being | 15:09 |
| weshay | ya | 15:09 |
| panda | weshay: you're already admin | 15:09 |
| weshay | ya | 15:09 |
| panda | weshay: anyway, I see this in the external users | 15:09 |
| panda | Note: by External User we mean any anonymous user not belonging to the Taiga platform, including search engines. Please use this role with care. | 15:09 |
| weshay | oh it's in admin | 15:09 |
| weshay | panda, sorry.. thanks | 15:10 |
| panda | it's probably lying then, and they mean registered users, not member of the project | 15:10 |
| *** dsneddon has quit IRC | 15:12 | |
| weshay | panda, ugh.. I don't seem to be able to add him | 15:13 |
| weshay | panda, even though he's in the system https://tree.taiga.io/profile/cybertron | 15:13 |
| weshay | wtf | 15:13 |
| panda | weshay: yeah, never worked for me, just use his email | 15:13 |
| *** ccamacho has quit IRC | 15:13 | |
| panda | if he's registered he'll be automatically added | 15:14 |
| rascasoft | rlandy, saw that, many thanks | 15:19 |
| weshay | arxcruz, did you want to 1-1? | 15:26 |
| arxcruz | weshay: sure | 15:26 |
| weshay | ok.. ready when you are | 15:26 |
| arxcruz | logging | 15:27 |
| rfolco|rover | sshnaidm|bbl, want to switch today? | 15:28 |
| arxcruz | weshay: ^ | 15:28 |
| ssbarnea | sshnaidm|bbl : it just failed, are you fixing it or should I do it? https://review.openstack.org/#/c/613797/ you did forget to run linting :D | 15:36 |
| weshay | chandankumar, arxcruz, rfolco|rover what's the story on https://bugs.launchpad.net/tripleo/+bug/1800742 | 15:37 |
| openstack | Launchpad bug 1800742 in tripleo "tempest.lib.exceptions.IdentityError: Got identity error, undercloud-containers" [Critical,Triaged] | 15:37 |
| *** quiquell is now known as quiquell|off | 15:38 | |
| ssbarnea | sshnaidm|bbl i fixed it. | 15:44 |
| *** kopecmartin is now known as kopecmartin|off | 15:49 | |
| marios | ansible.parsing.yaml.objects.AnsibleUnicode anyone have docs on this | 15:49 |
| ssbarnea | marios: bit out context | 15:52 |
| marios | ssbarnea: sorry talking on bluejeans with rlandy | 15:53 |
| weshay | arxcruz, help me understand one last thing | 15:54 |
| weshay | arxcruz, do we have a patch out there for https://bugs.launchpad.net/tripleo/+bug/1800742 | 15:54 |
| openstack | Launchpad bug 1800742 in tripleo "tempest.lib.exceptions.IdentityError: Got identity error, undercloud-containers" [Critical,Triaged] | 15:54 |
| arxcruz | weshay: checking | 15:54 |
| arxcruz | weshay: how many times this happen ? the error is like, the keystone service went down | 15:56 |
| arxcruz | 2018-10-30 21:38:09 | urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='192.168.24.2', port=13000): Read timed out. (read timeout=60) | 15:56 |
| *** agopi is now known as agopi|food | 16:04 | |
| weshay | arxcruz, you'd have to check elastic search | 16:05 |
| weshay | https://tree.taiga.io/project/tripleo-ci-board/wiki/notes-from-email | 16:08 |
| weshay | updated | 16:08 |
| *** dsneddon has joined #oooq | 16:15 | |
| *** jfrancoa has quit IRC | 16:17 | |
| weshay | panda, how did this get here? https://tree.taiga.io/project/tripleo-ci-board/task/297?kanban-status=1447275 | 16:19 |
| weshay | quiquell|off, ^ | 16:19 |
| *** gkadam has quit IRC | 16:20 | |
| panda | weshay: AFAIU he's writing everything that doesn't work on standalone f28 | 16:21 |
| weshay | panda, should be in parked right? | 16:21 |
| panda | weshay: maybe he's preparing a review for it, but this morning I asked more or less the same thing and he said that it could be done in the next sprint. | 16:22 |
| weshay | marios, rlandy any time available to help pick up #192 Iterate on fedora 28 standalone job to bring it to completion | 16:22 |
| weshay | 1points | 16:22 |
| panda | marios: rlandy weshay quiquell|off this morning said that the remaining task in the use story are low priority | 16:23 |
| weshay | panda, how is a promote job for f28 a low prioity? | 16:23 |
| panda | weshay: wel, lower tha having a complete standalone f28 run with oooq ? | 16:25 |
| weshay | need more info | 16:25 |
| weshay | complete? | 16:25 |
| weshay | what does that mean | 16:25 |
| panda | weshay: a job that completes deployment of standalone on fedora28 | 16:27 |
| chandankumar | weshay: blocked on undercloid install | 16:27 |
| weshay | panda, k ack and true for an upstream job | 16:28 |
| weshay | panda, once we have an upstream job working though.. next step is a promotion job | 16:28 |
| weshay | chandankumar, what context? | 16:28 |
| chandankumar | weshay: I looked at the undercloud install failing against one of my patch http://logs.openstack.org/56/605356/49/check/tripleo-ci-centos-7-undercloud-containers/8b654da/logs/undercloud/home/zuul/undercloud_install.log.txt.gz#_2018-10-31_13_53_01 so not lookfed today | 16:32 |
| rascasoft | rlandy, wow https://review.openstack.org/602734 gating jobs started! /me cries | 16:33 |
| chandankumar | and few bugs are on tripleo alert on undercloud install | 16:34 |
| marios | weshay: ack will check the board | 16:35 |
| marios | panda: ack | 16:36 |
| weshay | chandankumar, where is the podman for temepst work captured in taiga? | 16:36 |
| weshay | panda, rfolco|rover so some folks are doing it.. some are not but I find it helpful if folks post the taiga task in the commit message of reviews | 16:36 |
| weshay | do you guys? | 16:36 |
| weshay | retrospective idea for improvement maybe | 16:37 |
| panda | they started using US, I suggested the task. | 16:37 |
| rfolco|rover | weshay, not me. yet. | 16:37 |
| rlandy | weshay: hi re: picking up fedora 28 | 16:38 |
| rlandy | if marios and I do that, | 16:38 |
| chandankumar | weshay: https://tree.taiga.io/project/tripleo-ci-board/epic/101 I have not touched the review https://tree.taiga.io/project/tripleo-ci-board/us/102 | 16:38 |
| rlandy | reproducer work will be dead | 16:38 |
| chandankumar | weshay: emilienm asked me to take a look so updated the review today | 16:38 |
| weshay | rlandy, ok.. so the answer is no you guys can not pick it up | 16:38 |
| weshay | rlandy, marios that is fine | 16:39 |
| rlandy | weshay: decision for you and panda - but that is what marios and I are on atm | 16:39 |
| weshay | rlandy, k k | 16:39 |
| weshay | reproducer is life | 16:39 |
| weshay | it's not hard to create a job.. panda maybe you can assist ssbarnea | 16:39 |
| *** trown is now known as trown|lunch | 16:40 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024 @ https://review.openstack.org/602248, master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci- (3 more messages) | 16:41 |
| panda | weshay: yeah, but it's not just the job, is understanding where to put and reuse the results | 16:41 |
| weshay | panda, let's chat | 16:42 |
| weshay | https://bluejeans.com/u/whayutin/ | 16:42 |
| nhicher | chandankumar: FYI last run on vexxhost with your patch was a success https://logs.rdoproject.org/66/16566/10/check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master-vexxhost/38e686d | 16:50 |
| nhicher | weshay: ^ | 16:51 |
| *** chandankumar is now known as chkumar|off | 16:53 | |
| *** agopi|food is now known as agopi | 16:59 | |
| *** udesale has quit IRC | 17:03 | |
| weshay | nhicher, hell ya it was :)) | 17:04 |
| *** dsneddon has quit IRC | 17:10 | |
| weshay | nhicher, if that starts failing again.. PLEASE REPORT IT TO US | 17:14 |
| weshay | nhicher, ovb is totally down in rdo :) | 17:15 |
| weshay | rfolco|rover, sshnaidm|bbl ^ | 17:15 |
| nhicher | weshay: yes, I will monitor the job | 17:16 |
| weshay | nhicher, did you get in touch w sshnaidm|bbl about adding it to sova? | 17:17 |
| nhicher | weshay: yes, I've got discussion with sshnaidm|bbl last week but about -queens and -rocky jobs (these reviews are not merged, just used for depends-on), but -master for vexxhost is merged, should be possible to add it | 17:18 |
| weshay | rfolco|rover, https://review.rdoproject.org/r/#/c/17102/ ?? | 17:20 |
| *** dsneddon has joined #oooq | 17:21 | |
| weshay | rfolco|rover, bluejeans | 17:28 |
| rlandy | weshay: scheduled 1-1 today in 30 mins - do we need to meet again? | 17:31 |
| weshay | rlandy, only if you want to | 17:31 |
| rlandy | weshay: only if we need to discuss ironic or ovb or osp - otherwise not | 17:32 |
| rfolco|rover | weshay, I moved the bm logs removal to a separate patch | 17:32 |
| weshay | rfolco|rover, k.. please go out and get some reviews | 17:33 |
| *** bogdando has quit IRC | 17:33 | |
| *** holser_ has quit IRC | 17:33 | |
| weshay | ping people | 17:33 |
| rfolco|rover | weshay, it was a bit controversial about journald | 17:33 |
| weshay | I see | 17:33 |
| panda | rfolco|rover: the controversy was solved. | 17:34 |
| panda | rfolco|rover: though it was tracked there | 17:34 |
| panda | rfolco|rover: sshnaidm|bbl thought we were talking about the console log themselves, not just the deletion | 17:34 |
| rfolco|rover | panda, ok my friend charlie brown | 17:35 |
| panda | rfolco|rover: what do you expect the "size" there to do ? | 17:40 |
| rfolco|rover | panda, could you please elaborate ? | 17:41 |
| panda | rfolco|rover: size parameter rotates the log only if it's bigger than 100M. | 17:41 |
| panda | otherwise it doesnt' | 17:42 |
| rfolco|rover | panda, I don't think this is the case | 17:44 |
| rfolco|rover | ormally, logrotate is run as a daily cron job. It will not modify a log more than once in one day unless the criterion for that log is based on the log's size and logrotate is being run more than | 17:44 |
| rfolco|rover | once each day, or unless the -f or --force option is used. | 17:44 |
| rfolco|rover | panda, let me re-test and paste results there. No speculation. | 17:46 |
| panda | rfolco|rover: as I read it, it's rotate only once a day, and only if the log is > 100M otherwise bye | 17:49 |
| panda | rfolco|rover: logrotate is run by cron only daily | 17:49 |
| rfolco|rover | panda, this sentence is confusing. Let me test it myself and put an end to the mystery | 17:51 |
| panda | rfolco|rover: ok | 17:51 |
| *** fuzzball81 has quit IRC | 17:52 | |
| *** jjoyce has joined #oooq | 17:53 | |
| *** panda is now known as panda|off | 17:53 | |
| weshay | thanks all | 17:56 |
| *** trown|lunch is now known as trown | 18:04 | |
| *** holser_ has joined #oooq | 18:07 | |
| *** dtantsur is now known as dtantsur|afk | 18:17 | |
| *** jjoyce has quit IRC | 18:28 | |
| ssbarnea | does anyone knows why we need to pin to rdopkg version instead of mentioning minimal version? https://review.openstack.org/#/c/614516/2/roles/build-test-packages/tasks/main.yml | 18:29 |
| ssbarnea | to me this looks like recipe for more CR work to keep up | 18:30 |
| *** holser_ has quit IRC | 18:30 | |
| *** jjoyce has joined #oooq | 18:33 | |
| *** jjoyce is now known as fuzzball81 | 18:34 | |
| rfolco|rover | weshay, do not w+ pls | 18:39 |
| weshay | k | 18:39 |
| weshay | rfolco|rover, -1 yourself | 18:39 |
| weshay | if you want to indicate that | 18:40 |
| rfolco|rover | weshay, yeah I forgot | 18:40 |
| weshay | np | 18:40 |
| rfolco|rover | weshay, there is something else happening... | 18:41 |
| rfolco|rover | "/var/www/html/tebroker/testenv-worker.log" 2018-10-31-3:27:1 | 18:41 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024 @ https://review.openstack.org/602248, master: tripleo-ci-centos-7-ovb- (3 more messages) | 18:41 |
| rfolco|rover | it says the log was rotated today | 18:41 |
| rfolco|rover | :( | 18:41 |
| weshay | rfolco|rover, meh.. lower the size? | 18:42 |
| rfolco|rover | back to original problem | 18:42 |
| weshay | you mean it's working w/o any changes? | 18:42 |
| rfolco|rover | no, saying size won't fix the problem | 18:43 |
| rfolco|rover | its something else | 18:43 |
| rfolco|rover | cron runs | 18:43 |
| rfolco|rover | status of logrotate updates | 18:43 |
| rfolco|rover | but the f* file is not rotated | 18:43 |
| rfolco|rover | checking if any error on logs | 18:43 |
| *** sshnaidm|bbl is now known as sshnaidm|ruck | 18:57 | |
| weshay | nhicher, sshnaidm|ruck so using sova to compare ovb rdo-cloud and vexxhost jobs would be useful | 19:02 |
| weshay | sshnaidm|ruck, you've seen https://tree.taiga.io/project/tripleo-ci-board/epic/298 | 19:03 |
| weshay | ya? | 19:03 |
| sshnaidm|ruck | weshay, yeah, saw thits | 19:04 |
| sshnaidm|ruck | nhicher, what is job name now? | 19:04 |
| sshnaidm|ruck | weshay, this bullet points should be prioritized | 19:05 |
| weshay | sshnaidm|ruck, I'm just spec'ing out what would become user stories | 19:05 |
| sshnaidm|ruck | weshay, there some general improvements that hardly impacts ovb jobs | 19:05 |
| weshay | sshnaidm|ruck, well.. I'm happy to chat about this if you want | 19:06 |
| weshay | I am pitching it to a number of people, it's cross team etc | 19:06 |
| weshay | your input would be very helpful | 19:06 |
| sshnaidm|ruck | weshay, of course, let's chat | 19:07 |
| weshay | sshnaidm|ruck, k | 19:10 |
| * weshay goes to blue | 19:11 | |
| weshay | sshnaidm|ruck, https://bluejeans.com/u/whayutin/ | 19:13 |
| *** amoralej is now known as amoralej|off | 19:24 | |
| weshay | nhicher, you avail? https://bluejeans.com/u/whayutin/ | 19:26 |
| nhicher | sshnaidm|ruck: job is https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master-vexxhost | 19:35 |
| rlandy | weshay: hey | 19:41 |
| weshay | nhicher, hey | 19:42 |
| rlandy | weshay: do you know what creates job-output.txt | 19:42 |
| weshay | nhicher, hey.. jump on my bluejeans | 19:42 |
| weshay | rlandy, zuul | 19:42 |
| weshay | nhicher, https://bluejeans.com/u/whayutin/ | 19:42 |
| weshay | rlandy, maybe I'm missing the question | 19:42 |
| weshay | sshnaidm|ruck, rlandy s job-output.txt | 19:43 |
| weshay | sshnaidm|ruck, do you know? | 19:43 |
| rlandy | weshay: sshnaidm|ruck: I am looking for where that is created | 19:44 |
| rlandy | ie: where we gather to console log | 19:44 |
| rlandy | it's not an obvious step in collect logs | 19:44 |
| rlandy | weshay: sshnaidm|ruck: nvm - I think https://github.com/openstack-infra/zuul-jobs/blob/346dfe8d6874ab6f0e26109a52ca4664df106eed/roles/upload-logs/tasks/main.yaml#L41 | 19:48 |
| *** apetrich has quit IRC | 19:50 | |
| nhicher | weshay, sshnaidm|ruck https://softwarefactory-project.io/cgit/config/tree/nodepool/rdo-cloud.yaml#n239 | 19:56 |
| nhicher | https://openstack-virtual-baremetal.readthedocs.io/en/latest/host-cloud/prepare.html | 19:59 |
| weshay | nhicher, sshnaidm|ruck bacfaffbd2779fbf5c4da37345ca92cc | 20:00 |
| weshay | nhicher, sshnaidm|ruck bacfaffbd2779fbf5c4da37345ca92cc template | 20:00 |
| weshay | nhicher, sshnaidm|ruck bmc-base d712d9babb497b8c1644e51053f088d1 | 20:00 |
| nhicher | d712d9babb497b8c1644e51053f088d1 | 20:02 |
| *** apetrich has joined #oooq | 20:04 | |
| sshnaidm|ruck | weshay, nhicher https://review.rdoproject.org/r/#/c/17195/ | 20:05 |
| sshnaidm|ruck | weshay, nhicher https://review.rdoproject.org/r/#/c/17196/ | 20:13 |
| *** apetrich has quit IRC | 20:14 | |
| rfolco|rover | sshnaidm|ruck, perhaps we can merge https://review.rdoproject.org/r/#/c/17102/ as is and check tomorrow | 20:21 |
| rfolco|rover | if you run local, it works. If cron runs logrotate, it exists abnormally with [1] | 20:22 |
| rfolco|rover | ran out of ideas there. The only reason I found (theory) is selinux | 20:22 |
| *** trown is now known as trown|outtypewww | 20:29 | |
| ssbarnea | weshay : no sure how to describe it...https://review.openstack.org/#/c/613920/ | 20:30 |
| ssbarnea | i guess those 300 in sparta where screaming: this is recheck! | 20:31 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-multinode-1ctlr-featureset010 @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024 @ (3 more messages) | 20:41 |
| nhicher | sshnaidm|ruck, weshay: I disabled the crontab to run vexxhost job | 20:48 |
| nhicher | https://review.rdoproject.org/r/#/c/16566/ and https://review.rdoproject.org/r/#/c/17003/ are disabled | 20:49 |
| sshnaidm|ruck | nhicher, ack, I prepared patches: https://review.openstack.org/#/c/614632/ https://review.openstack.org/#/c/614633/ | 20:52 |
| sshnaidm|ruck | nhicher, hmm.. pretty short job: https://logs.rdoproject.org/32/614632/1/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-rocky-vexxhost/36ebcca/job-output.txt.gz | 20:58 |
| panda|off | rfolco|rover: it's definitely selinux | 21:05 |
| weshay | ssbarnea, not sure what you mean | 21:06 |
| weshay | ssbarnea, do you see the gate is red, no rechecks | 21:06 |
| weshay | ssbarnea, DO NOT APPROVE PATCHES | CI Status: RED | https://docs.openstack.org/tripleo-docs/latest/ | 21:06 |
| weshay | ssbarnea, patch that will fix the issue https://review.openstack.org/#/c/614537/ | 21:08 |
| weshay | sshnaidm|ruck, expect things to be red until that merges.. or depends-on that patch | 21:08 |
| weshay | and you should be much better | 21:08 |
| ssbarnea | we are all safe, i cannot approve any patch anyway :D | 21:14 |
| weshay | ssbarnea, ya.. that's not what I mean.. I mean when you recheck you are destined to fail | 21:27 |
| weshay | ssbarnea, failed jobs that is | 21:27 |
| *** chem has quit IRC | 21:28 | |
| weshay | ssbarnea, my notes.. https://tree.taiga.io/project/tripleo-ci-board/wiki/notes-from-email | 21:29 |
| weshay | ssbarnea, it's going to be my way of communicating w/ you guys | 21:29 |
| *** vinaykns has quit IRC | 21:45 | |
| *** apetrich has joined #oooq | 21:58 | |
| *** agopi is now known as agopi|brb | 22:04 | |
| *** agopi|brb has quit IRC | 22:08 | |
| *** rlandy is now known as rlandy|bbl | 22:26 | |
| ssbarnea | weshay : wiki page without watch option, a ticket would be easier to follow. | 22:32 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-multinode-1ctlr-featureset010 @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024 @ (3 more messages) | 22:41 |
| *** agopi|brb has joined #oooq | 22:44 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!