| *** rlandy|ruck is now known as rlandy|ruck|bbl | 00:11 | |
| weshay | rlandy|ruck|bbl I think that's default libvirt.. which deployment are you running? standard? | 00:16 |
|---|---|---|
| *** aakarsh|2 has joined #oooq | 00:36 | |
| *** dsneddon has quit IRC | 01:46 | |
| *** dsneddon has joined #oooq | 01:49 | |
| *** dsneddon has quit IRC | 01:54 | |
| *** dsneddon has joined #oooq | 01:59 | |
| *** dsneddon has quit IRC | 02:03 | |
| *** apetrich has quit IRC | 02:10 | |
| *** rlandy|ruck|bbl is now known as rlandy|ruck | 02:29 | |
| rlandy|ruck | weshay: thanks - got past that error | 02:29 |
| rlandy|ruck | got the job to start | 02:29 |
| rlandy|ruck | will debug more tomorrow | 02:29 |
| *** rlandy|ruck has quit IRC | 02:29 | |
| *** dsneddon has joined #oooq | 02:32 | |
| *** gkadam has joined #oooq | 03:51 | |
| *** gkadam has quit IRC | 03:51 | |
| *** jtomasek has quit IRC | 04:00 | |
| *** rfolco has quit IRC | 04:03 | |
| *** udesale has joined #oooq | 04:06 | |
| *** aakarsh|2 has quit IRC | 04:06 | |
| *** ratailor has joined #oooq | 04:20 | |
| *** raukadah is now known as chkumar|rover | 04:26 | |
| *** jtomasek has joined #oooq | 04:44 | |
| *** dsneddon has quit IRC | 04:45 | |
| *** skramaja has joined #oooq | 04:54 | |
| *** dsneddon has joined #oooq | 05:07 | |
| *** dsneddon has quit IRC | 05:15 | |
| chkumar|rover | sshnaidm: please merge this https://review.opendev.org/#/c/678622/ our centos7 container build is broken | 05:33 |
| chkumar|rover | because of reverts | 05:34 |
| *** sanjayu_ has joined #oooq | 05:42 | |
| *** dsneddon has joined #oooq | 05:47 | |
| *** ccamacho has quit IRC | 05:52 | |
| *** hamzy has quit IRC | 05:59 | |
| *** jfrancoa has joined #oooq | 06:05 | |
| *** jfrancoa has quit IRC | 06:09 | |
| *** dsneddon has quit IRC | 06:15 | |
| *** dsneddon has joined #oooq | 06:16 | |
| *** surpatil has joined #oooq | 06:22 | |
| *** jfrancoa has joined #oooq | 06:25 | |
| *** brault has joined #oooq | 06:38 | |
| *** dsneddon has quit IRC | 06:41 | |
| *** dsneddon has joined #oooq | 06:44 | |
| *** dsneddon has joined #oooq | 06:48 | |
| *** bogdando has joined #oooq | 07:14 | |
| *** dsneddon has quit IRC | 07:32 | |
| *** dtantsur|afk is now known as dtantsur | 07:37 | |
| *** jpena|off is now known as jpena | 07:40 | |
| *** surpatil has quit IRC | 07:44 | |
| sshnaidm | chkumar|rover, ack | 07:58 |
| *** dsneddon has joined #oooq | 08:01 | |
| *** panda has quit IRC | 08:02 | |
| *** panda has joined #oooq | 08:02 | |
| *** apetrich has joined #oooq | 08:22 | |
| *** ccamacho has joined #oooq | 08:47 | |
| *** derekh has joined #oooq | 08:50 | |
| *** surpatil has joined #oooq | 09:04 | |
| chem | chkumar|rover: hi, first time seeing that file /usr/share/ansible/roles/tripleo-hieradata/tasks/hieradata_vars.yaml conflicts between attempted installs of openstack-tripleo-common-11.1.1-0.20190826025903.29b7c8a.el7.noarch and tripleo-ansible-0.2.1-0.20190826144854.bf61a6f.el7.noarch | 09:12 |
| *** zbr has joined #oooq | 09:12 | |
| chkumar|rover | chem: https://review.opendev.org/#/c/673366/ and https://review.opendev.org/#/c/678622/ will fix the issue | 09:13 |
| chem | chkumar|rover: hum ... couldn't find the associated lp. are my lp search skills bad (certainly) or is there somewhere to look ? | 09:14 |
| zbr | hello! i am back. i wonder what I missed as apparently my connection dropped. | 09:15 |
| panda | zbr: so you don't know... | 09:15 |
| chkumar|rover | chem: https://bugs.launchpad.net/tripleo/+bug/1841405 | 09:17 |
| openstack | Launchpad bug 1841405 in tripleo "role 'dump_vars' not found leading to logs not getting collect in post" [Critical,Fix released] - Assigned to Kevin Carter (kevin-carter) | 09:17 |
| chem | chkumar|rover: oki, thanks | 09:20 |
| jfrancoa | chkumar|rover: hello, I took the freedom to update the depends-on patch here: https://review.rdoproject.org/r/#/c/21946/ | 09:27 |
| jfrancoa | chkumar|rover: I was debugging the issue in the reproducer environment rlandy lend me and I believe this patch should fix it: https://review.opendev.org/#/c/678767/ | 09:28 |
| chkumar|rover | jfrancoa: cool, thanks! so we need only one review | 09:30 |
| *** dtantsur is now known as dtantsur|bbl | 09:31 | |
| jfrancoa | chkumar|rover: I think so. I managed to run the overcloud update run passing --ssh-user tripleo-admin, it succeeded. So hopefully this will make it | 09:32 |
| chkumar|rover | jfrancoa: cool. | 09:32 |
| chkumar|rover | sshnaidm: Hello | 09:36 |
| chkumar|rover | sshnaidm: in order to remove pike jobs, do we need to remove current-tripleo-rdo aka rdo phase 1 jobs also https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/dlrnapi_promoter/config/CentOS-7/pike.ini#L31 ? | 09:36 |
| sshnaidm | chkumar|rover, yeah, we can remove them too from jenkins | 09:39 |
| chkumar|rover | sshnaidm: ok, preparing the patch | 09:39 |
| sshnaidm | chkumar|rover, but they don't run if no tripleoci promotion, so not urgent | 09:39 |
| chkumar|rover | ack! | 09:40 |
| sshnaidm | chkumar|rover, need also to remove everything from promoter about pike | 09:40 |
| chkumar|rover | sshnaidm: https://review.rdoproject.org/r/#/c/21961/ done here | 09:40 |
| sshnaidm | ack | 09:41 |
| *** sshnaidm is now known as sshnaidm|afk | 09:41 | |
| *** jaosorior has quit IRC | 09:43 | |
| *** matbu has joined #oooq | 09:45 | |
| *** apetrich has quit IRC | 09:55 | |
| *** apetrich has joined #oooq | 10:00 | |
| *** pierreprinetti has joined #oooq | 10:06 | |
| zbr | panda: is https://review.opendev.org/#/c/673481/ ready? | 10:09 |
| zbr | @oooq: i am focusing on reviews today, if you have any review where you need help, please ping me here with link. | 10:10 |
| jfrancoa | chkumar|rover: the job passed https://review.rdoproject.org/r/#/c/21946/ using https://review.opendev.org/678767, so I'll abandon https://review.opendev.org/#/c/678572/ and we'll try to merge the right fix | 10:51 |
| chkumar|rover | jfrancoa: cool, thanks! | 10:52 |
| chkumar|rover | jfrancoa++ | 10:53 |
| jfrancoa | chkumar|rover: no problem. happy to help | 10:53 |
| *** udesale has quit IRC | 11:02 | |
| *** tesseract has joined #oooq | 11:11 | |
| *** hamzy has joined #oooq | 11:18 | |
| *** jaosorior has joined #oooq | 11:20 | |
| *** jpena is now known as jpena|lunch | 11:30 | |
| *** sanjayu_ has quit IRC | 11:46 | |
| zbr | i see more and more activity on the pro-chaing-dfg document but not from our team. | 11:52 |
| *** sshnaidm|afk is now known as sshnaidm | 11:59 | |
| panda | zbr: TL;DR | 12:06 |
| *** dtantsur|bbl is now known as dtantsur | 12:08 | |
| *** ratailor_ has joined #oooq | 12:08 | |
| *** ratailor has quit IRC | 12:08 | |
| *** ratailor_ has quit IRC | 12:18 | |
| zbr | panda: can i update your https://review.opendev.org/#/c/673481/ to add the missing part from tox? | 12:19 |
| *** rfolco has joined #oooq | 12:22 | |
| panda | zbr: which missing part ? | 12:23 |
| panda | zbr: don't touch it right now, first I want to understand if oooo is ok with this solution | 12:23 |
| zbr | as long they see the -1 vote from rdo, they will not be positive about it. | 12:24 |
| zbr | i need to get in touch with cloudnull as he downvoted few changes around molecule, mainly because he used it with delegated and without tox. | 12:25 |
| *** rlandy has joined #oooq | 12:28 | |
| *** rlandy is now known as rlandy|ruck | 12:29 | |
| rlandy|ruck | weshay: chkumar|rover: anything we want to raise at tripleo meeting? | 12:29 |
| chkumar|rover | rlandy|ruck: nope | 12:30 |
| chkumar|rover | rlandy|ruck: just an update on pike job removal | 12:30 |
| chkumar|rover | we have all the patches up for the same | 12:30 |
| weshay | not as ruck/rover I think the sprint team had a few things... | 12:30 |
| rlandy|ruck | chkumar|rover: k - saw your jobs - will work on merge today | 12:30 |
| chkumar|rover | rlandy|ruck: https://etherpad.openstack.org/p/ruckroversprint14 line 26 | 12:30 |
| chkumar|rover | rlandy|ruck: let me know If I missed anything | 12:30 |
| chkumar|rover | rlandy|ruck: CI is also calm due to this https://bugs.launchpad.net/tripleo/+bug/1841564 | 12:31 |
| openstack | Launchpad bug 1841564 in tripleo "/usr/share/ansible/roles/tripleo-hieradata/tasks/hieradata_vars.yaml conflicts between attempted installs of tripleo-ansible and openstack-tripleo-common" [Critical,Confirmed] | 12:31 |
| rlandy|ruck | chkumar|rover: I'll check through those | 12:31 |
| weshay | rlandy|ruck chkumar|rover I think this job, is non-voting | 12:31 |
| weshay | http://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades&branch=stable%2Fstein | 12:31 |
| weshay | rlandy|ruck I think it should be voting.. | 12:31 |
| chkumar|rover | weshay: yes it is nv https://opendev.org/openstack/tripleo-ci/src/branch/master/zuul.d/multinode-jobs.yaml#L304 | 12:32 |
| rlandy|ruck | k - can update | 12:32 |
| weshay | chkumar|rover that could be something to mention in the mtg | 12:32 |
| *** jpena|lunch is now known as jpena | 12:32 | |
| weshay | chkumar|rover rlandy|ruck master should be nv, n-* should be voting | 12:33 |
| weshay | k | 12:33 |
| weshay | ? | 12:33 |
| rlandy|ruck | weshay: ack | 12:33 |
| chkumar|rover | http://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades | 12:33 |
| rlandy|ruck | weshay: the default is non-voting | 12:33 |
| chkumar|rover | looks pretty green | 12:33 |
| chkumar|rover | with few yellows | 12:33 |
| rlandy|ruck | chkumar|rover: weshay: patch in progress | 12:33 |
| weshay | rlandy|ruck right.. so the upgrade jobs.. for master should be nv, n-* should vote if stable | 12:33 |
| *** dsneddon has quit IRC | 12:36 | |
| *** jaosorior has quit IRC | 12:39 | |
| *** surpatil has quit IRC | 12:39 | |
| *** jaosorior has joined #oooq | 12:39 | |
| rlandy|ruck | hmmm - can we put j2 in job definitions? | 12:46 |
| zbr | rlandy|ruck: can you please wf https://review.opendev.org/#/c/678038/ ? | 12:52 |
| zbr | is required for s1-4 work, should have already being workflowed days ago. | 12:53 |
| zbr | rlandy|ruck: thanks! | 12:53 |
| zbr | what an useless UI experience on https://zuul.opendev.org/t/openstack/build/5bbd63bc123042d9a435cdf1dc06c0ef/console | 12:54 |
| zbr | i need to look at source code to figure-out the name of the log file | 12:55 |
| rlandy|ruck | weshay: https://review.opendev.org/#/c/678814/ - but I would prefer to add a j2 if condition there | 13:00 |
| rlandy|ruck | because that would save us changing the name | 13:00 |
| rlandy|ruck | not sure if that is possible | 13:00 |
| *** aakarsh|2 has joined #oooq | 13:01 | |
| rlandy|ruck | jfrancoa: hi - could you reach the nodes held for debug? | 13:02 |
| chkumar|rover | rlandy|ruck: queens upgrade issue now fixed | 13:02 |
| chkumar|rover | rlandy|ruck: https://review.opendev.org/#/c/678767/ | 13:02 |
| jfrancoa | rlandy|ruck: yep and as chkumar|rover said, it's fixed. thanks a lot for the nodes it was very helpful | 13:03 |
| *** dsneddon has joined #oooq | 13:03 | |
| rlandy|ruck | jfrancoa: k - going to let the admins know they can reclaim them | 13:03 |
| *** Goneri has joined #oooq | 13:05 | |
| *** yolanda has quit IRC | 13:06 | |
| *** yolanda__ has joined #oooq | 13:06 | |
| zbr | rfolco: apparently for s14 work we need to get rid of fluentd, and martin is still working on it https://review.opendev.org/#/c/668851/ | 13:06 |
| rlandy|ruck | jfrancoa: chkumar|rover: k - w+ https://review.opendev.org/#/c/678767/ | 13:06 |
| rfolco | zbr, ok, please keep chasing scen1-4 | 13:07 |
| rfolco | zbr, my python guru, do you understand future.wait ? | 13:08 |
| zbr | rfolco: not that kind of guru, yet. but if you point me to some code, i may have a chance of finding more. | 13:09 |
| *** jaosorior has quit IRC | 13:09 | |
| *** dsneddon has quit IRC | 13:09 | |
| rlandy|ruck | chkumar|rover: re: phase 1 master fix - is it worth rerunning phase 1 master or we would need the next promotion to pick that fix up? | 13:09 |
| rlandy|ruck | https://etherpad.openstack.org/p/ruckroversprint14 - line 38 | 13:09 |
| rfolco | zbr, I think something is wrong with this code - https://opendev.org/openstack/tripleo-common/commit/0be1be779a27d7bb3ba8f5469e391e4c72eee685 | 13:10 |
| chkumar|rover | rlandy|ruck: the promotion job for naster is running, let's get it finish, then it will automatically pick up | 13:10 |
| rfolco | zbr, its raising an system exception, mistakenly | 13:10 |
| rfolco | zbr, why? https://3d89b2f66ce8e968c7f7-8b938dd2076b97d235f21ad4df33ebf0.ssl.cf2.rackcdn.com/678058/27/check/tripleo-build-containers-centos-7-buildah/7508cee/logs/build.log.txt.gz | 13:10 |
| rlandy|ruck | chkumar|rover: it will kick phase 1 only of it passes | 13:11 |
| chkumar|rover | rlandy|ruck: ack, thanks! | 13:11 |
| rfolco | zbr, all containers are pushed to local registry, but the code future.wait is still raising exception | 13:11 |
| rlandy|ruck | k - if not then I'll manually kick it | 13:11 |
| rfolco | zbr, line 174... wondering if this is correct: return_when=futures.FIRST_EXCEPTION | 13:11 |
| weshay | FYI... mtgs most of the morning and afternoon :( | 13:12 |
| rfolco | weshay, ack | 13:13 |
| rlandy|ruck | https://review.rdoproject.org/r/#/c/21961/2/ci-scripts/dlrnapi_promoter/dlrn-promoter.sh - pike is gone an docat ais not :) | 13:17 |
| rlandy|ruck | and ocata | 13:17 |
| chkumar|rover | rlandy|ruck: ocata is still there? | 13:19 |
| rlandy|ruck | chkumar|rover: weshay: so https://review.opendev.org/#/c/678154/ has to merge first before we remove our pike jobs - or, as long a the eol tag is approved, we can go ahead? | 13:19 |
| rlandy|ruck | "CentOS-7/queens" "CentOS-7/ocata" "RedHat-8/master" | 13:20 |
| rlandy|ruck | yep | 13:20 |
| rlandy|ruck | fine though | 13:20 |
| chkumar|rover | arxcruz: please include this one https://review.opendev.org/#/c/678833/ aslo as a deps on fs01 os_tempest patch | 13:23 |
| weshay | rlandy|ruck I approve yes | 13:23 |
| chkumar|rover | it will be a precheck for the same | 13:23 |
| arxcruz | chkumar|rover: is that the problem ? | 13:24 |
| chkumar|rover | arxcruz: it will check early by doing a gateway ping | 13:25 |
| chkumar|rover | might be the issue, donot know | 13:25 |
| weshay | chkumar|rover rlandy|ruck please add some notes to https://docs.google.com/document/d/1LP23IwpCJLKCMe3wzDOvtqZZ79-eCxhxHVf7NOPeQbI/edit#heading=h.cyleyypknu4t | 13:25 |
| zbr | rfolco: i am looking at the code in i master and I have no idea how an empty set would endup evaluated as true!? | 13:26 |
| rfolco | zbr, there are some unit tests that I am starting to play with... to see if I catch the problem | 13:27 |
| zbr | have a look at https://github.com/openstack/tripleo-common/blob/9145769a7af55f4402e1ce30b542f7bd5e89e5e3/tripleo_common/image/builder/buildah.py#L188 | 13:27 |
| zbr | what gets printed is set([]) which on py27 means an empty set. | 13:28 |
| zbr | py3 prints it as set() but that is a detail, still evaluates the same. | 13:28 |
| rfolco | zbr, parsing your comments.... lets chat on community mtg if time permits | 13:29 |
| zbr | it would not be possible to reach the SystemError if not_done would be an empty set! | 13:29 |
| zbr | it has nothign to do with future.wait | 13:29 |
| zbr | but personally, I would replce FIRST_EXCEPTION and force it to run all tasks before returning. | 13:30 |
| rfolco | zbr, panda sshnaidm rlandy|ruck arxcruz: community mtg | 13:32 |
| *** udesale has joined #oooq | 13:32 | |
| chkumar|rover | weshay: rlandy|ruck updated the doc | 13:33 |
| chkumar|rover | feel free to comment | 13:33 |
| rlandy|ruck | thanks | 13:33 |
| weshay | lolz... in managers mtg.... ci creating documentation is coming up | 13:33 |
| weshay | :) | 13:33 |
| *** jaosorior has joined #oooq | 13:33 | |
| zbr | rfolco: sshnaidm please have a look at https://review.opendev.org/#/c/678838/ -- it should fix the issue with buildah SystemError | 13:35 |
| chkumar|rover | rlandy|ruck: container build job passed | 13:37 |
| rlandy|ruck | yep | 13:38 |
| *** dsneddon has joined #oooq | 13:42 | |
| rfolco | zbr, thx for the patch :) | 14:02 |
| * chkumar|rover headed home, will be back soon | 14:14 | |
| rlandy|ruck | chkumar|rover: pls see comment on https://review.rdoproject.org/r/#/c/21961/ | 14:16 |
| rlandy|ruck | I can add the cockpit stuff later | 14:16 |
| rlandy|ruck | your patch is fine for its content | 14:16 |
| *** Vorrtex has joined #oooq | 14:17 | |
| *** Vorrtex has quit IRC | 14:17 | |
| *** Vorrtex has joined #oooq | 14:19 | |
| *** sshnaidm_ has joined #oooq | 14:29 | |
| *** sshnaidm has quit IRC | 14:30 | |
| *** udesale has quit IRC | 14:32 | |
| *** sshnaidm__ has joined #oooq | 14:34 | |
| *** sshnaidm_ has quit IRC | 14:36 | |
| *** skramaja has quit IRC | 14:40 | |
| *** sshnaidm__ is now known as sshnaidm | 14:51 | |
| sshnaidm | weshay, https://review.opendev.org/#/c/678630/ - which readme? which context? | 14:51 |
| weshay | sshnaidm https://github.com/openstack/ansible-role-collect-logs/blob/master/README.rst don't assume everyone know what sova is, that's basically what I'm poking at | 14:53 |
| weshay | rlandy|ruck can you grep through the featuresets and make sure they all have podman for the appropriate releases please.. eg. https://review.opendev.org/#/c/678244/3/config/general_config/featureset037.yml | 14:55 |
| rlandy|ruck | weshay: ack | 14:56 |
| weshay | rlandy|ruck or we can make that a default | 14:56 |
| rlandy|ruck | let's see what the damage is | 14:56 |
| rlandy|ruck | ie: how the jobs are defined vs the fs | 14:56 |
| *** dsneddon has quit IRC | 14:58 | |
| *** dsneddon has joined #oooq | 15:04 | |
| *** bogdando has quit IRC | 15:09 | |
| *** dsneddon has quit IRC | 15:10 | |
| rlandy|ruck | weshay: will be much easier to add this setting to a common role than put it in every fs | 15:10 |
| chkumar|rover | rlandy|ruck: will remove rrcockpit and pike sova cleanup tomorrow | 15:12 |
| *** ksambor has quit IRC | 15:12 | |
| rlandy|ruck | chkumar|rover: it's fine - better to do it afterwards | 15:13 |
| weshay | rlandy|ruck agree | 15:13 |
| rlandy|ruck | it will remind us if we missed anything | 15:13 |
| weshay | rlandy|ruck so.. I would add to common, and remove from fs | 15:13 |
| rlandy|ruck | weshay: going to put it in common-extras | 15:13 |
| chkumar|rover | rlandy|ruck: weshay ack | 15:14 |
| rlandy|ruck | weshay: this is far from efficient : http://pastebin.test.redhat.com/792171 | 15:14 |
| * rlandy|ruck patches | 15:15 | |
| rlandy|ruck | [rlandy@localhost tripleo-quickstart]$ grep -r overcloud_container_cli | 15:15 |
| rlandy|ruck | config/general_config/featureset010.yml:overcloud_container_cli: podman | 15:15 |
| rlandy|ruck | config/general_config/pacemaker.yml:overcloud_container_cli: docker | 15:15 |
| rlandy|ruck | ^^ that is even worse | 15:15 |
| *** ccamacho has quit IRC | 15:23 | |
| chkumar|rover | rlandy|ruck: weshay see ya tomorrow, | 15:31 |
| *** chkumar|rover is now known as raukadah | 15:31 | |
| weshay | raukadah sent an email to you.. | 15:31 |
| weshay | hit me up w/ questions if you have them | 15:31 |
| raukadah | weshay: checking, will look into that tobiko stuff | 15:31 |
| weshay | raukadah thanks | 15:32 |
| raukadah | panda: sshnaidm: weshay https://review.rdoproject.org/r/#/q/topic:remove_pike+(status:open+OR+status:merged) time to say bye bye to pike | 15:33 |
| panda | so soon ? | 15:33 |
| raukadah | kind of | 15:34 |
| raukadah | panda: sshnaidm rfolco arxcruz https://review.rdoproject.org/r/#/q/topic:remove_pike+(status:open+OR+status:merged) please vote on this when free | 15:35 |
| raukadah | sorry this one https://lists.rdoproject.org/pipermail/dev/2019-August/009126.html | 15:35 |
| raukadah | zbr: can we make these check jobs to nv https://review.rdoproject.org/r/#/c/21787/ | 15:39 |
| raukadah | currently | 15:39 |
| *** dsneddon has joined #oooq | 15:39 | |
| *** jfrancoa has quit IRC | 15:47 | |
| *** brault has quit IRC | 15:50 | |
| *** sanjayu_ has joined #oooq | 15:51 | |
| *** chem` has joined #oooq | 15:51 | |
| *** chem has quit IRC | 15:52 | |
| *** sshnaidm is now known as sshnaidm|afk | 15:54 | |
| *** jpena is now known as jpena|off | 16:04 | |
| weshay | zbr fix this please https://review.rdoproject.org/r/#/c/21787/ | 16:15 |
| zbr | sure. | 16:16 |
| weshay | rlandy|ruck did you have luck w/ a repro or still need eyes? | 16:18 |
| rlandy|ruck | weshay: I got it to work until the multinode bridge | 16:19 |
| rlandy|ruck | I will try pick it up from there | 16:19 |
| rlandy|ruck | weshay: rdocloud never worked | 16:19 |
| weshay | wow.. 0 gate failures today :)) | 16:19 |
| rlandy|ruck | logger issue | 16:19 |
| weshay | I'm not familiar w/ the logger issue, but saw it being discussed | 16:19 |
| weshay | a few days ago | 16:19 |
| rlandy|ruck | started working with libvirt to avoid that | 16:19 |
| rlandy|ruck | weshay: you are familiar :) you logged the initial bug | 16:20 |
| weshay | hrm... k | 16:20 |
| weshay | OH? | 16:20 |
| weshay | lolz | 16:20 |
| weshay | ooh stein and master promoted too :) | 16:20 |
| rlandy|ruck | yesterday ack | 16:21 |
| rlandy|ruck | getting them to try promote today as well | 16:21 |
| rlandy|ruck | well, late last night | 16:21 |
| zbr | weshay: we need to wf https://review.opendev.org/678838 as soon is passing checks, (eta ~1h) -- is needed for s14 | 16:21 |
| weshay | zbr context w/ non-voting is so we can still get a +1 from 3rd party zuul | 16:22 |
| zbr | oops.just seen a commen,. need to check it. | 16:22 |
| weshay | closing unauth bug https://bugs.launchpad.net/tripleo/+bug/1839532 | 16:23 |
| openstack | Launchpad bug 1839532 in tripleo "tripleo gate jobs are failing to pull containers when running on ovh provider with "UNAUTHORIZED" error" [Critical,Fix released] | 16:23 |
| weshay | rlandy|ruck sorry to bug you.. I don't see the a logger bug in my list | 16:24 |
| weshay | help a brotha out | 16:24 |
| rlandy|ruck | weshay: getting | 16:26 |
| rlandy|ruck | weshay: https://bugs.launchpad.net/tripleo/+bug/1833465 | 16:26 |
| openstack | Launchpad bug 1833465 in tripleo "tripleo reproducer fails w/ "waiting on logger"" [Critical,Incomplete] | 16:26 |
| rlandy|ruck | ^^ that killed me yesterday | 16:26 |
| weshay | OHHh | 16:29 |
| weshay | that went away for awhile | 16:29 |
| weshay | I think that is a zuul bug | 16:29 |
| weshay | sshnaidm|afk we may need to update the repro containers | 16:29 |
| rlandy|ruck | it returned | 16:29 |
| weshay | rlandy|ruck well... maybe we just need to refresh | 16:29 |
| *** ksambor has joined #oooq | 16:29 | |
| rlandy|ruck | with a vengeance | 16:29 |
| *** ksambor has quit IRC | 16:30 | |
| weshay | ya.. it's terrible | 16:30 |
| rlandy|ruck | refresh what? | 16:30 |
| rlandy|ruck | I pick up the shared images | 16:30 |
| rlandy|ruck | what else can be refreshed? | 16:30 |
| weshay | rlandy|ruck the builds of the containers themselves | 16:30 |
| * weshay looks | 16:30 | |
| rlandy|ruck | weshay: I was running the reproducer to give the upgrades team a place t debug | 16:31 |
| rlandy|ruck | eventually, I just asked the admins to hold nodes for them | 16:31 |
| rlandy|ruck | so they are all set atm | 16:31 |
| weshay | k k | 16:32 |
| rlandy|ruck | no emergency - but I need to get this sorted | 16:32 |
| rlandy|ruck | so nobody else hits it | 16:32 |
| weshay | rlandy|ruck https://github.com/rdo-infra/ansible-role-tripleo-ci-reproducer/blob/master/defaults/main.yaml#L60 | 16:33 |
| weshay | rlandy|ruck I saw that bug in upstream or rdo zuul one time as well | 16:33 |
| weshay | zuul_image: rdoci/zuul:stable | 16:33 |
| weshay | zuul_scheduler_image: rdoci/zuul-scheduler:stable | 16:33 |
| weshay | zuul_web_image: rdoci/zuul-web:stable | 16:33 |
| weshay | zuul_executor_image: rdoci/zuul-executor:stable | 16:33 |
| weshay | zuul_merger_image: rdoci/zuul-merger:stable | 16:33 |
| weshay | zuul_fingergw_image: rdoci/zuul-fingergw:stable | 16:33 |
| weshay | ya.. so we're uploading zuul containers to rdoci namespace | 16:34 |
| weshay | I bet they've been updated and probably fixed | 16:34 |
| weshay | https://hub.docker.com/u/zuul | 16:34 |
| rlandy|ruck | weshay: there is a review attached to the bug | 16:34 |
| rlandy|ruck | it never merged | 16:34 |
| rlandy|ruck | <weshay> I bet they've been updated and probably fixed | 16:35 |
| rlandy|ruck | ^^ who is they? | 16:35 |
| weshay | in docker.io | 16:35 |
| weshay | because I think it was a zuul | 16:35 |
| rlandy|ruck | and then we would need to update what? | 16:35 |
| weshay | issue | 16:35 |
| weshay | the zuul code in the zuul containers | 16:36 |
| rlandy|ruck | why am I the only one finding it then? | 16:36 |
| weshay | it wasn't a reproducer issue | 16:36 |
| weshay | rlandy|ruck only one using it | 16:36 |
| weshay | no one is using any of the reproducers atm | 16:36 |
| rlandy|ruck | weshay: sshnaidm|afk tried it | 16:36 |
| rlandy|ruck | but maybe he has some modified env? | 16:36 |
| rlandy|ruck | also the vexxhost reproducer ci job keeps failing | 16:37 |
| weshay | ugh... | 16:37 |
| weshay | https://hub.docker.com/r/zuul/zuul/tags | 16:37 |
| weshay | they have one tag | 16:37 |
| rlandy|ruck | I just left it until we sort out vexxhost | 16:37 |
| weshay | geez | 16:37 |
| weshay | rlandy|ruck ya.. it's not on fire | 16:37 |
| rlandy|ruck | it was pretty embarrassing though to struggle so much to produce a test env | 16:37 |
| rlandy|ruck | weshay: suggested action? woudl not want to leave this for the next unsuspecting ruck | 16:38 |
| weshay | rlandy|ruck see if we can setup a job .. like the reproducer job.. that is pulling the latest zuul containers from docker.io | 16:40 |
| weshay | we we don't have to constantly wonder the state of those containers | 16:40 |
| weshay | rlandy|ruck so something that would override the defaults | 16:40 |
| rlandy|ruck | weshay: no so familiar with where the reproducer container settings are going down - few minutes - will see if I can set up job to override | 16:41 |
| *** dtantsur is now known as dtantsur|afk | 16:42 | |
| rlandy|ruck | lunch - back in a few | 16:45 |
| weshay | rlandy|ruck we could probably just have a perm. depends-on in a job | 16:55 |
| raukadah | https://twitter.com/thomasdcameron/status/1166386613305380864 | 16:59 |
| weshay | https://review.rdoproject.org/r/21964 | 17:01 |
| *** derekh has quit IRC | 17:02 | |
| *** pierreprinetti has quit IRC | 17:12 | |
| *** pierreprinetti has joined #oooq | 17:13 | |
| *** pierreprinetti has quit IRC | 17:15 | |
| *** jaosorior has quit IRC | 17:16 | |
| rlandy|ruck | weshay: thanks - watching | 17:20 |
| weshay | rlandy|ruck one mroe coming | 17:20 |
| rlandy|ruck | maybe can try that out in my env | 17:20 |
| rlandy|ruck | k - waiting | 17:21 |
| *** brault has joined #oooq | 17:23 | |
| weshay | rlandy|ruck something like that I think https://review.rdoproject.org/r/21965 | 17:23 |
| weshay | rlandy|ruck it may fail on a token | 17:24 |
| rlandy|ruck | hmmm | 17:25 |
| rlandy|ruck | let's see | 17:26 |
| rlandy|ruck | post failures there | 17:26 |
| rlandy|ruck | weshay: 2019-08-27 16:24:12.952375 | TASK [Run ansible playbook to collect logs] | 17:27 |
| rlandy|ruck | 2019-08-27 16:24:18.006183 | Timeout exception waiting for the logger. Please check connectivity to [38.145.35.133:19885] | 17:27 |
| rlandy|ruck | 2019-08-27 17:19:13.911826 | primary | ERROR | 17:27 |
| rlandy|ruck | ^^ on rdo jobs themselves now | 17:27 |
| weshay | ufn | 17:28 |
| weshay | fun | 17:28 |
| rlandy|ruck | tristanC is going t be happy when his rotation is over as well | 17:29 |
| rlandy|ruck | pinged | 17:29 |
| *** tesseract has quit IRC | 17:30 | |
| weshay | rlandy|ruck where did he drop the latest link to logreduce | 17:34 |
| * weshay checks email | 17:34 | |
| weshay | wanted to check that out | 17:34 |
| * rlandy|ruck gets | 17:34 | |
| rlandy|ruck | <tristanC> weshay: rlandy: i updated the logreduce filters to remove most of the false positive, here is the most recent report http://logs.rdoproject.org/67/678767/1/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/22b609d/report.html which looks pretty good | 17:34 |
| rlandy|ruck | that one> | 17:34 |
| weshay | 0.262 | 125904: 2019-08-27 12:01:58 | "InternalServerError: Internal Server Error (HTTP 500)", | 17:36 |
| weshay | 0.000 | 125905: 2019-08-27 12:01:58 | "DEBUG:keystoneauth.identity.v3.base:Making authentication request to http://172.17.0.109:5000/v3/auth/tokens", | 17:36 |
| rlandy|ruck | we see all sorts of logging problems | 17:37 |
| rlandy|ruck | when there is a lot of activity | 17:37 |
| weshay | ya.,.. I like trying to root cause the issue when I see these logs to see if it's really correct | 17:38 |
| rlandy|ruck | weshay: we dropped the max-servers once | 17:39 |
| rlandy|ruck | maybe we should drop that number again? | 17:39 |
| weshay | ya.. some working jobs is better than no working jobs | 17:39 |
| rlandy|ruck | ok - getting review | 17:39 |
| weshay | rlandy|ruck I've updated the program call doc | 17:54 |
| rlandy|ruck | weshay: thanks | 17:55 |
| rlandy|ruck | I will show up though | 17:55 |
| rlandy|ruck | checking doc | 17:56 |
| raukadah | weshay: can we add a summary on what we found on adding RHEL-8 gating? like the collectd package issue | 17:56 |
| weshay | raukadah where? | 17:57 |
| weshay | and why are you still here? | 17:57 |
| raukadah | my laptop is still open so lurking around on different channels | 17:58 |
| raukadah | weshay: to prod chain council | 17:58 |
| weshay | raukadah keeping some notes on what gains we've had from running rhel 8 is a smart idea | 17:59 |
| raukadah | weshay: I will drop an email tomorrow | 17:59 |
| weshay | I suspect we *may* want to hold off on a full communication of that until centos8 drops | 17:59 |
| *** fmount has quit IRC | 18:00 | |
| *** jaosorior has joined #oooq | 18:00 | |
| weshay | if things go relatively well w/ centos8 we can claim a big success | 18:00 |
| rlandy|ruck | max server changed merged | 18:00 |
| weshay | if they don't go well.. well.. don't know :) raukadah | 18:00 |
| raukadah | then it will be our experience | 18:00 |
| raukadah | !success == experience | 18:01 |
| openstack | raukadah: Error: "success" is not a valid command. | 18:01 |
| rlandy|ruck | hopefully next master run will be better | 18:01 |
| rlandy|ruck | weirdo-master-promote-packstack-scenario003 - still failing | 18:01 |
| rlandy|ruck | thought that wa sfuxed?? | 18:02 |
| rlandy|ruck | was fixed | 18:02 |
| raukadah | rlandy|ruck: one thing I realized today tempest network basic ops tests is too mucn annoying failing with common error ssh timeout | 18:02 |
| *** fmount has joined #oooq | 18:02 | |
| rlandy|ruck | raukadah: on rdocloud? | 18:02 |
| raukadah | yes | 18:02 |
| rlandy|ruck | there are lots of timeouts there | 18:02 |
| rlandy|ruck | I am not sure it's tempest's fault | 18:02 |
| rlandy|ruck | that just runs last | 18:02 |
| rlandy|ruck | and gets hit | 18:02 |
| rlandy|ruck | we dropped the server numbers again | 18:03 |
| rlandy|ruck | to lessen the load | 18:03 |
| raukadah | donot know may be some real bug hidden there | 18:03 |
| rlandy|ruck | yesterday we found that we kept overwhelming the log server | 18:03 |
| rlandy|ruck | that is true | 18:03 |
| rlandy|ruck | but it will still show | 18:03 |
| rlandy|ruck | I'm still running timeout comparisons | 18:04 |
| raukadah | this sprint we have stats, post_failure and 401 unauthorized access are our night maers | 18:04 |
| rlandy|ruck | I don't have a definitive culprit yet | 18:04 |
| rlandy|ruck | I followed up with caching change | 18:04 |
| rlandy|ruck | Emilien and Alex ditched that as a cause | 18:04 |
| rlandy|ruck | ack 401 did us in | 18:05 |
| rlandy|ruck | and may return | 18:05 |
| raukadah | yup, it was awesome , too much learning :-) | 18:05 |
| rlandy|ruck | rdocloud also has less attention than before | 18:05 |
| rlandy|ruck | focus now being on vexxhost | 18:05 |
| rlandy|ruck | so working within our resources may be our best bet | 18:05 |
| rlandy|ruck | temp | 18:05 |
| rlandy|ruck | raukadah: one more day for us :) | 18:06 |
| raukadah | weshay: are we planning to do our team meeting in india ? | 18:06 |
| raukadah | that would be fune :-) | 18:06 |
| raukadah | *fun | 18:07 |
| rlandy|ruck | want to host us all? | 18:07 |
| raukadah | rlandy|ruck: our boss weshay if he wishes :-) | 18:07 |
| weshay | raukadah first let me get everyone in the door, w/ a desk | 18:08 |
| weshay | raukadah get training started... | 18:08 |
| weshay | raukadah I'm planning on stealing the content from bootcamps | 18:08 |
| weshay | and yes.. bringing to Pune | 18:08 |
| weshay | raukadah NOTE: this is all in my head atm, I've only briefly talked about it w/ phil and others | 18:09 |
| raukadah | weshay: :-) | 18:09 |
| rlandy|ruck | it's like three days travel, right? | 18:10 |
| raukadah | rlandy|ruck: depends on where we are based on | 18:10 |
| raukadah | I think In india, we donot have visa issues | 18:10 |
| raukadah | weshay: rlandy|ruck zbr sshnaidm|afk panda arxcruz rfolco https://kubernetes.academy/ | 18:11 |
| rlandy|ruck | All U.S. citizens need a valid passport and valid Indian visa to enter and exit India for any purpose | 18:12 |
| rlandy|ruck | ^^ according to website | 18:12 |
| * raukadah loves this slide https://twitter.com/tobyhede/status/1166200411910365185 on k8s | 18:12 | |
| raukadah | rlandy|ruck: yes, | 18:15 |
| zbr | i agree, running my k8n cluster with kubespray was a PITA. | 18:15 |
| raukadah | if you are a foodie & travel, India is the best place, with more than 100 + food items and 28 + states with travel hard to complete in an year | 18:16 |
| raukadah | *in an | 18:16 |
| raukadah | *in a | 18:16 |
| rlandy|ruck | 13:57:34 TASK [Failure detected when testing packstack-scenario003] ********************* | 18:19 |
| rlandy|ruck | 13:57:34 task path: /home/jenkins/workspace/weirdo-master-promote-packstack-scenario003/weirdo/playbooks/packstack-scenario003.yml:35 | 18:19 |
| rlandy|ruck | still | 18:19 |
| raukadah | weshay: I am thinking to move collect-logs role to openstack-ansible-sig https://etherpad.openstack.org/p/ansible-sig once our integration work is done, but needs approval | 18:19 |
| weshay | raukadah++++ | 18:20 |
| raukadah | I learned one thing today give + take ~= collaboration | 18:22 |
| rlandy|ruck | https://review.opendev.org/#/q/topic:000-upgrades-voting+(status:open+OR+status:merged) | 18:32 |
| rlandy|ruck | ^^ don;t think I missed anything here | 18:32 |
| rlandy|ruck | but if anyone notices, pls comment/vote | 18:33 |
| rlandy|ruck | https://review.opendev.org/#/q/topic:move-cli-extras+(status:open+OR+status:merged) | 18:37 |
| rlandy|ruck | requires votes pls | 18:37 |
| *** ksambor has joined #oooq | 18:51 | |
| *** ksambor has quit IRC | 18:51 | |
| zbr | weshay: no invitation to Pune for me? I love Indian food | 18:55 |
| weshay | zbr perhaps :) | 18:55 |
| weshay | we'll be expecting people to lead training around tripleo.. | 18:56 |
| weshay | zbr perhaps we can have some training on molecule etc | 18:56 |
| weshay | when we get these folks in the builidng .. November 2019.. we'll start talking about it some more | 18:57 |
| zbr | that was implied assumption, i do not expect to go there for the food. | 18:58 |
| zbr | maybe I should change my tagline "making molecule presentations, for food" :D | 18:58 |
| weshay | zbr that would get me excited | 18:59 |
| zbr | i need to go now, already 8pm here. as an update, I just raised https://review.opendev.org/#/c/678938/ which is supposed to address the random errors with buildah containers (if it passed the check) | 19:00 |
| zbr | that is important because we have another change required by scenario 1-4 which was never merged because it was failing on buildah. | 19:01 |
| *** sanjayu_ has quit IRC | 19:06 | |
| rlandy|ruck | weshay: going back to testing reproducer patches | 19:07 |
| rlandy|ruck | which ones would I use locally? | 19:08 |
| weshay | containers? | 19:08 |
| rlandy|ruck | yep | 19:08 |
| rlandy|ruck | I would shut down my current deployment and start again | 19:09 |
| rlandy|ruck | it pulls from master though | 19:09 |
| weshay | ya.. try out changing the registry addr and namespace and tag | 19:09 |
| weshay | master? | 19:09 |
| weshay | no | 19:09 |
| weshay | the reproducer containers.. pull from rdo registry.. not related to openstack brnaches | 19:10 |
| rlandy|ruck | k - let's see | 19:12 |
| rlandy|ruck | testing on tenant | 19:51 |
| weshay | see you on tues :) | 20:11 |
| *** weshay is now known as weshay_pto | 20:11 | |
| *** Vorrtex has quit IRC | 20:15 | |
| rlandy|ruck | weshay_pto: next monday :) | 20:21 |
| rlandy|ruck | I'm out | 20:21 |
| rlandy|ruck | weshay_pto: ugh - still waiting for logger | 20:24 |
| rlandy|ruck | forget that | 20:24 |
| *** brault has quit IRC | 20:48 | |
| *** aakarsh|2 has quit IRC | 21:02 | |
| *** Goneri has quit IRC | 21:13 | |
| *** jtomasek has quit IRC | 21:29 | |
| *** aakarsh|2 has joined #oooq | 22:11 | |
| *** brault has joined #oooq | 22:49 | |
| *** brault has quit IRC | 22:53 | |
| *** rlandy|ruck has quit IRC | 23:57 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!