| *** holser has quit IRC | 00:21 | |
| *** Goneri has quit IRC | 00:47 | |
| *** apetrich has quit IRC | 02:10 | |
| *** ykarel|away has joined #oooq | 02:50 | |
| *** gkadam has joined #oooq | 03:54 | |
| *** gkadam has quit IRC | 03:54 | |
| *** ykarel|away has quit IRC | 04:06 | |
| *** ykarel|away has joined #oooq | 04:19 | |
| *** jtomasek has joined #oooq | 04:29 | |
| *** jtomasek has quit IRC | 04:40 | |
| *** raukadah is now known as chandankumar | 04:49 | |
| *** jtomasek has joined #oooq | 04:51 | |
| *** udesale has joined #oooq | 05:02 | |
| *** holser has joined #oooq | 05:08 | |
| *** ratailor has joined #oooq | 05:19 | |
| *** holser has quit IRC | 05:21 | |
| *** ccamacho has quit IRC | 05:32 | |
| *** marios has joined #oooq | 05:51 | |
| chandankumar | sshnaidm|afk: Hello | 05:53 |
|---|---|---|
| chandankumar | sshnaidm|afk: please try with this patch https://review.opendev.org/#/c/617522/ for ansible python interpreter | 05:54 |
| chandankumar | sshnaidm|afk: and here it is working https://logs.rdoproject.org/22/617522/4/openstack-check/tripleo-ci-rhel-8-standalone-rdo/c7c157c/logs/ara_oooq/ara-report/result/372675db-cb2d-4513-bd28-e49501fc5937/ | 05:57 |
| *** kopecmartin|off is now known as kopecmartion | 06:23 | |
| *** kopecmartion is now known as kopecmartin | 06:24 | |
| *** holser has joined #oooq | 06:25 | |
| chandankumar | panda|rover: morning | 06:25 |
| *** ccamacho has joined #oooq | 06:27 | |
| *** jfrancoa has joined #oooq | 06:35 | |
| *** surpatil has joined #oooq | 06:36 | |
| panda|rover | chandankumar: morning | 06:38 |
| surpatil | Good Morning | 06:41 |
| *** udesale has quit IRC | 06:56 | |
| *** sanjayu_ has quit IRC | 06:58 | |
| *** saneax has joined #oooq | 07:01 | |
| *** ykarel|away is now known as ykarel | 07:04 | |
| ykarel | panda|rover, are the ovb jobs failure in master periodic known? | 07:04 |
| *** bogdando has joined #oooq | 07:08 | |
| *** tesseract has joined #oooq | 07:15 | |
| *** tosky has joined #oooq | 07:23 | |
| *** jpena|off is now known as jpena | 07:33 | |
| chandankumar | panda|rover: I am updating this patch to address marios https://review.opendev.org/#/c/673481/ | 07:34 |
| chandankumar | 's comments | 07:34 |
| zbr | chandankumar: panda|rover : can we wf https://review.opendev.org/#/c/673481/ ? i am kinda tired of not seeing any login changes merged. | 07:34 |
| *** ykarel is now known as ykarel|lunch | 07:34 | |
| panda|rover | ykarel: probably not | 07:35 |
| panda|rover | ykarel|lunch: ^ | 07:35 |
| chandankumar | zbr: needs to address few comments | 07:35 |
| zbr | chandankumar: please do an ping, i want to clean these from our plate. | 07:35 |
| zbr | i can workflow on rdo side but they depend on stuff like this one | 07:36 |
| chandankumar | zbr: done | 07:38 |
| zbr | panda|rover: sshnaidm|afk : i also ned you help on https://review.rdoproject.org/r/#/c/21996/ which defaults to 2.8 for ALL rdo jobs. Do not merge but vote/comment, it will be jpena and tristan to workflow it at the right time. | 07:39 |
| zbr | nobody raised worries during yesterday rdo meeting, so I guess we will do it tomorrow (early enough to allow us to revert it needed) | 07:40 |
| chandankumar | panda|rover: can I break it https://review.rdoproject.org/r/#/c/21820/ into two patches, one to remove the job from container-registry and add it | 07:41 |
| chandankumar | so that third party should vanish and emilienm can merge it | 07:42 |
| chandankumar | ? | 07:42 |
| zbr | chandankumar: if you do it do it like this: create new change for removal, we do that one and after this werebase the original one, and voila, is "simplified" :D | 07:42 |
| chandankumar | *third party job | 07:42 |
| *** sshnaidm|afk is now known as sshnaidm|ruck | 07:43 | |
| sshnaidm|ruck | zbr, did you talk with jpena about setting 2.8 as default in rdo zuul? Maybe it's better than hardcoding it in config. When 2.9 arrives, we'll need to remove it | 07:45 |
| zbr | sshnaidm|ruck: yeah, talked with him and with openstack-infra, and pbelanger. | 07:45 |
| zbr | the way to change default ansible is in "base" jobs. | 07:45 |
| zbr | the one defined in rdo overrides the base one used upstream. | 07:46 |
| sshnaidm|ruck | zbr, isn't it in zuul config? | 07:46 |
| zbr | in fact is the only practical way to do it, based on pabelanger recommandation: he already did it long time ago on his ansible-network zuul. | 07:46 |
| chandankumar | zbr: panda|rover https://review.rdoproject.org/r/#/c/22097/ | 07:48 |
| zbr | sshnaidm|ruck: not really, afaik. but the question here: which jobs should we test now using depends-on? | 07:48 |
| sshnaidm|ruck | zbr, we don't have 2.7 set in our base job, so where is this set? | 07:49 |
| *** brault has joined #oooq | 07:50 | |
| zbr | i am sure there is a default on zuul, but we are better off controlling which version we use regardless which version is default in current zuul version, this allows them to upgrade zuul will more confidence. | 07:50 |
| zbr | i kinda like splitting zuul upgrades from ansible version upgrades. | 07:50 |
| zbr | it will happen upstream too but they are bit slower, also they do not have the same "business reasons" as us (rhel8/centos8) | 07:51 |
| chandankumar | sshnaidm|ruck: zbr please have a look at this hack https://review.opendev.org/#/c/617522/5/defaults/main.yml /me is looking for better solution so that it works for all distro | 07:55 |
| zbr | chandankumar: why ever doing this when you can do it directly withut hostbars? | 07:56 |
| panda|rover | oh yes remove seomthing so we can replace it with a equal job the patch after. | 07:57 |
| chandankumar | zbr: you mean just ansible_python.executable ? | 08:00 |
| zbr | chandankumar: YES | 08:00 |
| chandankumar | ansible_python_intepreter gives undefined var | 08:00 |
| chandankumar | ok updating the patch | 08:00 |
| * marios biab | 08:02 | |
| *** marios has quit IRC | 08:02 | |
| zbr | chandankumar: that is because you are using wrong variable, correct one is ansible_python.executable | 08:02 |
| zbr | the other one is the optional override config option | 08:03 |
| zbr | thus is why it can be undefined | 08:03 |
| zbr | you may need to run: setup setup: | 08:03 |
| zbr | gather_subset: min | 08:03 |
| zbr | which is the minimal gather facts needed to load python interpreter fact from remote host (aka detection) | 08:04 |
| chandankumar | ok | 08:08 |
| chandankumar | zbr: please merge this one https://review.rdoproject.org/r/#/c/22097/ | 08:08 |
| zbr | chandankumar: done. also please add the setup role, as I mention in last comment. | 08:10 |
| chandankumar | zbr: Dhanyabad :-) | 08:10 |
| *** brault has quit IRC | 08:13 | |
| zbr | chandankumar: did i tell you that I *HATE* how openstack ansible-* repos were configured? | 08:13 |
| chandankumar | zbr: nope | 08:14 |
| zbr | where user cannot even run tox normally. | 08:14 |
| zbr | not sure who did it, but the excuse is that they documented how you are supposed to test the code, including its linting. | 08:15 |
| zbr | because "we are reinventing the wheel here, every day" (probably a good tagline for openstack in general) | 08:15 |
| *** ykarel|lunch is now known as ykarel | 08:28 | |
| zbr | chandankumar: code still needs a lot of rework but i am updating it now. | 08:29 |
| chandankumar | ok | 08:31 |
| *** derekh has joined #oooq | 08:31 | |
| chandankumar | kopecmartin: Hello | 08:32 |
| chandankumar | kopecmartin: once it merges https://review.rdoproject.org/r/#/c/21879/, bot will generate a new review against tempestconf and reuse the same review for removing deps | 08:32 |
| kopecmartin | chandankumar: ok | 08:33 |
| ykarel | panda|rover, ack, i noticed all ovb jobs failed | 08:37 |
| *** dtantsur|afk is now known as dtantsur | 08:38 | |
| *** chem` has quit IRC | 08:42 | |
| zbr | chandankumar: please review https://review.opendev.org/#/c/617522/ | 08:42 |
| zbr | fixes several bugs | 08:42 |
| *** marios has joined #oooq | 08:55 | |
| chandankumar | zbr: check #openstack-ansible discussion | 09:00 |
| marios | panda|rover: trying the molecule check on the latest of https://review.rdoproject.org/r/#/c/21895/ right now | 10:10 |
| marios | converge looks like it hangs? does it take a whil | 10:10 |
| marios | while? | 10:10 |
| marios | panda|rover: ^ do you recall | 10:10 |
| zbr | marios: it does not hang, is due to pytest std collection. | 10:11 |
| zbr | locally it runs with -x but on CI it does not, side effect: you do not get output until it finishes. | 10:11 |
| marios | zbr: ack noting will add -x next time then :D | 10:12 |
| zbr | that is because pytest limitation: cannot do both: capturing std streams and displaying them. you need to pick one. | 10:12 |
| marios | thanks zbr | 10:12 |
| ykarel | panda|rover, so i found atleast https://review.opendev.org/#/c/656027/ is related to the failure, there may be other failures too | 10:12 |
| zbr | with tox I used a magic trick: {tty:-x} added, which mainly adds -x when there is a tty present. | 10:12 |
| zbr | marios: sorry option is named -s | 10:13 |
| zbr | marios: add this to your profile PYTEST_ADDOPTS='-s --color=yes --no-cov' | 10:14 |
| zbr | maybe even -x if you want to stop at first failure. | 10:14 |
| marios | zbr: ack noting | 10:15 |
| zbr | marios: my pleasure. glad you asked. | 10:16 |
| *** udesale has joined #oooq | 10:20 | |
| *** udesale has quit IRC | 10:28 | |
| *** udesale has joined #oooq | 10:29 | |
| *** soniya29 has joined #oooq | 10:32 | |
| panda|rover | marios: where ? | 10:37 |
| marios | panda|rover: where what? running it on the rdo vm you talked me into setting up the other day | 10:38 |
| marios | panda|rover: i think am ok for now thanks | 10:38 |
| panda|rover | marios: where does it hang ? | 10:38 |
| panda|rover | marios: ok | 10:38 |
| marios | panda|rover: when i did molecul check | 10:38 |
| marios | panda|rover: with test instead i am getting a fail and debugging it now | 10:39 |
| sshnaidm|ruck | folks, please take a look at https://review.opendev.org/#/c/678630/ | 10:41 |
| chandankumar | sshnaidm|ruck: I am little confused here https://review.opendev.org/#/c/678630/6/library/sova.py | 10:44 |
| chandankumar | in doc it mentions about podman container | 10:44 |
| sshnaidm|ruck | chandankumar, yeah, copy-paste, please leave a comment in a patch | 10:45 |
| zbr | in the absence of centos 7.7/8.0 it would worth mentioning that oraclelinux has 8.0 available https://hub.docker.com/_/oraclelinux :D | 10:52 |
| panda|rover | wow https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/logs_30/678630/6/check/tripleo-ci-centos-7-containers-multinode/cba2e3c/ | 11:09 |
| panda|rover | "payment required" | 11:09 |
| chandankumar | panda|rover: it was seen yesterday | 11:09 |
| chandankumar | i think it is fixed now | 11:09 |
| panda|rover | one hand on the keyboard, and one on the wallet | 11:11 |
| panda|rover | chandankumar: it's still there | 11:11 |
| *** jpena is now known as jpena|lunch | 11:21 | |
| chandankumar | panda|rover: is something wrong with check-queue today in upstream? | 11:26 |
| panda|rover | chandankumar: not that I see | 11:31 |
| chandankumar | it is more than 3-4 waiting for patch | 11:32 |
| panda|rover | chandankumar: link ? | 11:37 |
| chandankumar | panda|rover: http://zuul.openstack.org/status put patchno: 673481 | 11:39 |
| chandankumar | earlier all jobs were waiting | 11:39 |
| zbr | hahaha | 11:48 |
| panda|rover | ? | 11:50 |
| zbr | panda|rover: chandankumar any chance to get some supporting comments on https://bugs.centos.org/view.php?id=16389 ? | 11:52 |
| zbr | panda|rover: ^ when I read payment required. | 11:52 |
| panda|rover | are we really going to support python3 on centos7 ? | 11:54 |
| panda|rover | I think I'll kill myself | 11:54 |
| zbr | panda|rover: you are missing few bits: 7.7 is schedules to be releases in 3 days. 8.0 ... nobody knows. | 11:55 |
| zbr | we have jobs (even simple testing ones (tox) that do require python3, some even minimum pyhon3.6) | 11:56 |
| panda|rover | zbr: how many things we'll need to change to make centos7 nodes use python3 instead of python2 ? How many had coded conditionals ? | 11:56 |
| zbr | example docs generation, sphinx already dropped python2. | 11:56 |
| zbr | panda|rover: py36 is additional on c7.7 is not replacing python2. | 11:57 |
| zbr | mainly is like it was with epel, but now much easier to install. | 11:57 |
| zbr | it does not change default python | 11:57 |
| zbr | (that would be crazy) | 11:57 |
| panda|rover | zbr: yes, the question is; "are we going to suport 2, 3 or both for centos7?" | 11:57 |
| panda|rover | ykarel: I had a previous patch to test the CR repositories https://review.opendev.org/618780, probably need some updates on branches, but you think it should work ? | 11:58 |
| zbr | panda|rover: let me ask you something, maybe we can find an answer: default node on rdo for tox is centos7, without any python3. | 11:58 |
| zbr | i want/need to run some tox jobs (like tox-docs,...) with python3 minimum. | 11:59 |
| chandankumar | zbr: can you check from which repo libselinux comes? | 11:59 |
| zbr | what do I do? do I start switching jobs to fedora nodes? | 11:59 |
| zbr | libselinux needs to be build at the same time as libselinux, version needs to match. | 12:00 |
| zbr | at this moment is coming from nowhere. | 12:00 |
| ykarel | panda|rover, any reason to not use enable_cr_repo flag like it's done in fs021? | 12:00 |
| zbr | mainly if you want it, you need to get selinux sources and compile the python bindings yourself and install it. | 12:01 |
| chandankumar | zbr: https://cbs.centos.org/koji/buildinfo?buildID=14542 | 12:01 |
| chandankumar | it comes from different repo not in base | 12:02 |
| zbr | we are in a very bad spot with this because everyone avoid the pink elephant: selinux packager does not want to publish a python3 version of the rpm as this is "optional", is not an os component. python36 packages doesn't want either as they say, selinux is not part of python,... | 12:03 |
| chandankumar | or what about enabling epel8 and installing it from there? | 12:03 |
| panda|rover | ykarel: I have no idea what does that flag do | 12:03 |
| zbr | chandankumar: libxselinux depends on selinux and kerbel, is not portable. | 12:04 |
| zbr | you cannot get the binary from one kernel and use it with another one. | 12:04 |
| chandankumar | ok | 12:04 |
| zbr | it was possible by now I would have published a manylinux bynary for that library, me or someone from ansible team. | 12:04 |
| zbr | i kinda envy debian distros now, they don't have this problem. | 12:05 |
| ykarel | panda|rover, flag is enable_centos_cr_repo and it enables cr repo in job | 12:05 |
| ykarel | https://github.com/openstack/tripleo-quickstart/blob/6133bf14f3330d923759e8ae4fdfc12820f90a00/config/release/tripleo-ci/CentOS-7/promotion-testing-hash-master.yml#L111 | 12:05 |
| *** brault has joined #oooq | 12:06 | |
| chandankumar | ykarel: panda|rover https://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset021-master/138763a/logs/undercloud/etc/yum.repos.d/CentOS-CR.repo.txt.gz | 12:07 |
| chandankumar | cr repo is there | 12:07 |
| ykarel | chandankumar, not ^^, quickstart-cr | 12:07 |
| ykarel | it enables | 12:07 |
| panda|rover | ykarel: mmmhhh, I remember we also needed to install CR in the local machine before anything else, to update everything to centos7.7 as first step | 12:09 |
| chandankumar | ok it is also there https://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset021-master/138763a/logs/undercloud/etc/yum.repos.d/quickstart-centos-cr.repo.txt.gz | 12:09 |
| ykarel | panda|rover, local machine? u mean in libvirt based jobs? | 12:10 |
| ykarel | virthost | 12:10 |
| panda|rover | ykarel: no, in zuul | 12:11 |
| panda|rover | ykarel: if you look at the prepare-node tasks, the first thing we need to do is upgrade the base os in the node from 7.6 to 7.7 | 12:11 |
| panda|rover | ykarel: otherwise we are not testing the job fully in centos7.7 | 12:12 |
| chandankumar | ykarel: panda|rover https://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset021-master/138763a/logs/undercloud/home/zuul/repo_setup.log.txt.gz#_2019-09-04_19_26_19 | 12:12 |
| chandankumar | cr is used here | 12:12 |
| panda|rover | ykarel: the first step is simulating that we are getting a node with cento7.7 in it. | 12:12 |
| zbr | ... ouch this upgrade downloaded ~600mb on my machine and took forever to install. | 12:13 |
| ykarel | panda|rover, okk, but repo setup tasks yum update runs iirc, should that take care for upgrading nodes | 12:13 |
| panda|rover | ykarel: what about the jobs that are not using repo setup ? | 12:14 |
| panda|rover | crap | 12:14 |
| ykarel | repo setup runs at all TripleO jobs iiuc | 12:15 |
| panda|rover | ykarel: not the image builds | 12:15 |
| panda|rover | we need to update tripleo-repos for those | 12:15 |
| ykarel | and container builds also | 12:15 |
| ykarel | yes right | 12:15 |
| panda|rover | and I'd like to find the line that updates everything in repo setup | 12:16 |
| panda|rover | and it may not be neough | 12:16 |
| panda|rover | updateing everything after we started quickstart is skipping some testing under centos7.7 | 12:16 |
| zbr | sshnaidm|ruck: panda|rover : see if you can comment on https://review.opendev.org/#/c/677971/6 -- i will be trying to persuade infra to display more in the header (so we know for sure which python version is used by a job) | 12:17 |
| panda|rover | I remember we wanted to be sure that *EVERY* step was run on centos7.7 | 12:17 |
| panda|rover | well 7.6 at the time | 12:17 |
| panda|rover | to avoid surprises | 12:17 |
| chandankumar | zbr: in RHEl7.6 libselinux comes from base | 12:24 |
| chandankumar | sorry centos | 12:25 |
| chandankumar | weshay: time for 1:! | 12:31 |
| chandankumar | 1:1 | 12:31 |
| sshnaidm|ruck | panda|rover, we test cr repos in 021 periodic job | 12:39 |
| weshay | chandankumar hey | 12:42 |
| weshay | sorry I'm late | 12:42 |
| chandankumar | weshay: no problem | 12:42 |
| weshay | you want to chat now or later? | 12:42 |
| zbr | chandankumar: we are talking about python-libselinux here, not libselinux itself which obviously comes from base. | 12:42 |
| chandankumar | weshay: yes | 12:42 |
| *** ratailor has quit IRC | 12:42 | |
| chandankumar | weshay: now would be ebtter | 12:44 |
| chandankumar | *better | 12:44 |
| weshay | joining | 12:44 |
| chandankumar | weshay: https://wiki.centos.org/About/Building_8 | 12:52 |
| *** Goneri has joined #oooq | 12:55 | |
| chandankumar | weshay: https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/rdoinfo-jobs.yaml | 12:55 |
| rfolco | scrum time - https://bluejeans.com/5878458097 | 13:00 |
| *** jpena|lunch is now known as jpena | 13:01 | |
| rfolco | chandankumar, zbr | 13:01 |
| rfolco | zbr, scrum time | 13:02 |
| panda|rover | sshnaidm|ruck: so we are testing CR only partially and only on a single job ? | 13:14 |
| sshnaidm|ruck | panda|rover, what does mean "partially"? | 13:17 |
| panda|rover | chandankumar: can you shoot the loud people in you room ? | 13:17 |
| panda|rover | chandankumar: with a water gun. | 13:17 |
| *** brault has quit IRC | 13:18 | |
| chandankumar | panda|rover: good idea, need to buy it today. | 13:18 |
| sshnaidm|ruck | http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset021-master/381908f/logs/undercloud/etc/yum.repos.d/quickstart-centos-cr.repo.txt.gz | 13:18 |
| sshnaidm|ruck | panda|rover, ^^ | 13:18 |
| chandankumar | sshnaidm|ruck: panda|rover weshay https://review.opendev.org/#/c/673481/ needs +2 and +w | 13:21 |
| panda|rover | sshnaidm|ruck: the testing for CR twofold: 1) we need to be sure we are able to deploy with 7.7 packages, and that's covered in the fs021 job. 2) we need to test that the workflow doesn't have any surprises in 7.7. Partially means that repo-setup is updating everything 7.7 only after quickstart runs. If 7.7 is breaking quickstart for some reason, we'll not detect it there | 13:22 |
| panda|rover | as well as build-* jobs. | 13:22 |
| sshnaidm|ruck | panda|rover, yeah, that's right | 13:25 |
| *** surpatil has quit IRC | 13:25 | |
| panda|rover | sshnaidm|ruck: ok, I'm working to increase the coverage to 2 too, reusing what we had for 7.6 | 13:27 |
| weshay | sshnaidm|ruck should we recheck? https://review.rdoproject.org/r/#/c/22086/ | 13:47 |
| sshnaidm|ruck | weshay, no, new jobs running and passing now | 13:48 |
| *** brault has joined #oooq | 13:50 | |
| ykarel | panda|rover, any chance u looked at master ovb failures | 13:53 |
| ykarel | it would need attention from nova/ironic guys | 13:54 |
| zbr | just got the answer i was expecting: https://bugs.centos.org/view.php?id=16389#c35071 --- mainly "not my problem" | 13:54 |
| chandankumar | zbr: i have gone through this experience | 13:55 |
| panda|rover | ykarel: I think sshnaidm|ruck was looking at it. It he's not I can try. | 13:55 |
| zbr | almost every time we face a centos issue, we get something like that, is frustrating and draining, we need to make our voice heard | 13:55 |
| zbr | i do not think that shutting-up is the right approach here. | 13:56 |
| ykarel | panda|rover, okk it's good then | 13:56 |
| sshnaidm|ruck | ykarel, talk to me | 13:56 |
| ykarel | :) | 13:56 |
| ykarel | sshnaidm|ruck, u saw master ovb failures? | 13:56 |
| sshnaidm|ruck | ykarel, ya, all of them because of different issues, two of no-valid-host | 13:57 |
| ykarel | sshnaidm|ruck, no valid host seems real issue caused by nova patch | 13:57 |
| sshnaidm|ruck | ykarel, which patch? I wanted to see them failing again today to be sure | 13:58 |
| ykarel | sshnaidm|ruck, it failed multiple times i think | 13:58 |
| ykarel | 2 runs till now | 13:58 |
| *** brault has quit IRC | 13:58 | |
| ykarel | sshnaidm|ruck, so nova is switching to openstacksdk | 13:58 |
| ykarel | sshnaidm|ruck, so basically https://review.opendev.org/#/c/656027/ and it;s related patcehs | 13:59 |
| ykarel | i commented there with logs | 14:00 |
| ykarel | and error, | 14:00 |
| ykarel | other ironic related patches can also have issue, i didn't checked in detail, | 14:03 |
| *** Vorrtex has joined #oooq | 14:07 | |
| sshnaidm|ruck | ykarel, I see, will create a bug and gonna check last jobs | 14:07 |
| sshnaidm|ruck | ykarel, which logs you see errors in ? | 14:07 |
| ykarel | sshnaidm|ruck, Thanks | 14:07 |
| ykarel | sshnaidm|ruck, nova-compute | 14:08 |
| *** brault has joined #oooq | 14:08 | |
| sshnaidm|ruck | ykarel, this one BadRequestException: BadRequestException: 400: Client Error for url: http://192.168.24.3:6385/v1/nodes?fields=instance_id&associated=True, Field(s) "instance_id" are not valid | 14:09 |
| ykarel | sshnaidm|ruck, yes | 14:09 |
| sshnaidm|ruck | ykarel, iirc ironic changed recently from uuid to id (or opposite), isn't it related in some way? | 14:10 |
| sshnaidm|ruck | dtantsur, ^^ | 14:10 |
| ykarel | may be, but dtantsur would know more | 14:11 |
| *** soniya29 has quit IRC | 14:15 | |
| sshnaidm|ruck | ykarel, hmm.. I see also errors like "InstanceNotFound: Instance 2a9c9e29-1cdd-4701-ac09-60bcad1c291d could not be found." | 14:15 |
| sshnaidm|ruck | ykarel, with these problems in rdo cloud you never know if it's a bug or it doesn't work again.. | 14:15 |
| bogdando | o/ what is the right place to push a patch for https://github.com/rdo-infra/ansible-role-tripleo-ci-reproducer ? | 14:17 |
| bogdando | I suppose that is a mirror only? | 14:17 |
| sshnaidm|ruck | bogdando, in rdoproject | 14:17 |
| bogdando | sshnaidm|ruck: thanks | 14:18 |
| bogdando | sshnaidm|ruck: got link?.. :D | 14:18 |
| sshnaidm|ruck | bogdando, https://review.rdoproject.org/r/#/q/project:rdo-infra/ansible-role-tripleo-ci-reproducer | 14:19 |
| sshnaidm|ruck | bogdando, https://review.rdoproject.org/r/#/admin/projects/rdo-infra/ansible-role-tripleo-ci-reproducer | 14:19 |
| *** sabedevops has joined #oooq | 14:20 | |
| sshnaidm|ruck | bogdando, feel free to ping me for review | 14:20 |
| *** sabedevops has left #oooq | 14:20 | |
| ykarel | sshnaidm|ruck, yes but it's not always rdo cloud issue :) | 14:20 |
| *** sabedevops has joined #oooq | 14:20 | |
| *** sabedevops has quit IRC | 14:21 | |
| bogdando | sshnaidm|ruck, ykarel: https://review.rdoproject.org/r/#/c/22104/ | 14:24 |
| sshnaidm|ruck | jpena, do you know maybe why experimental jobs are not queued for this patch? https://review.opendev.org/#/c/656027/ I'm posting "check rdo experimental" according to pipeline config | 14:25 |
| bogdando | w/o that log dir, it seems the reproducer fails ansible in zuul executor but the job reports succeeded | 14:25 |
| sshnaidm|ruck | bogdando, hmm.. workspace/logs should be created inside a job | 14:26 |
| bogdando | mhm | 14:26 |
| sshnaidm|ruck | bogdando, it's part of toci_quickstart.sh | 14:26 |
| zbr | sshnaidm|ruck: can you please help me with https://review.opendev.org/#/c/674092/ -- do I miss something? -- a second opinnion would be useful before trying to push on infra. | 14:26 |
| bogdando | well, may be that's just me doing it wrong then | 14:26 |
| sshnaidm|ruck | bogdando, maybe job failed before it's created | 14:27 |
| chandankumar | sshnaidm|ruck: panda|rover weshay https://review.opendev.org/#/c/673481/ needs +2 and +w | 14:27 |
| bogdando | sshnaidm|ruck, ykarel: one more https://review.rdoproject.org/r/#/c/22105/ | 14:28 |
| bogdando | I'm trying basically to trigger an upstream 007 multinode for a patch | 14:28 |
| bogdando | w/o that change zuul can't trace down all the parents | 14:28 |
| sshnaidm|ruck | zbr, not sure I understand purpose of this patch.. | 14:29 |
| jpena | sshnaidm|ruck: checking | 14:29 |
| sshnaidm|ruck | chandankumar, well, firstly it needs review :D | 14:29 |
| zbr | sshnaidm|ruck: we need to fix that one as is blocking rdo changes, like https://review.rdoproject.org/r/#/c/21486/ | 14:29 |
| zbr | mainly zuul roles upstream are *not* tested with rhel, many of them not even with centos. so there are bugs we encounter when we try to use them in rdo. | 14:30 |
| sshnaidm|ruck | zbr, I don't think we should install manually libselinux though, shouldn't it be in the image? | 14:31 |
| zbr | sshnaidm|ruck: another issue caused by lack of proper testing of ansible roles | 14:31 |
| bogdando | sshnaidm|ruck, ykarel: https://review.rdoproject.org/r/22106 and https://review.rdoproject.org/r/#/c/22107/ a few more :) | 14:31 |
| bogdando | not sure if really needed but... | 14:32 |
| sshnaidm|ruck | zbr, and ssh build key role seems like not really place to do it | 14:32 |
| ykarel | ack | 14:32 |
| zbr | sshnaidm|ruck: i have zero control over who is building images, we have multiple zuul instances,... cannot just hope people are dreaming about what needs to be installed or not. | 14:32 |
| sshnaidm|ruck | bogdando, cool, I'll look at it later | 14:32 |
| sshnaidm|ruck | bogdando, did you get it working locally? | 14:32 |
| zbr | each role should take care of its own requirements. | 14:33 |
| bogdando | sshnaidm|ruck: to the stage I can see a non idle CPU on my nodepool nodes :D | 14:33 |
| zbr | for example libselinux is not really neeeded for 100% of ansible usages | 14:33 |
| jpena | sshnaidm|ruck: I don't see any experimental job defined for upstream's openstack-nova in review.rdo | 14:33 |
| sshnaidm|ruck | zbr, what do you mean? we can add/remove elements in DIB afaik | 14:33 |
| bogdando | ans some ansible logs failing a little bit later and later in zuul executor | 14:33 |
| bogdando | not that I've managed to start 007 job tho | 14:33 |
| zbr | alternative is to add dependency on a bootstrapping role. but still, it should be part of the role. | 14:33 |
| sshnaidm|ruck | jpena, I think I have them here: https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/zuul.d/projects.yaml#L119-L123 | 14:34 |
| jpena | sshnaidm|ruck: hm, true. I misread | 14:34 |
| * jpena needs caffeine | 14:34 | |
| zbr | sshnaidm|ruck: images are build in multiple places, by multiple people. | 14:34 |
| sshnaidm|ruck | jpena, and job itself: https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/project-templates.yaml#L170-L174 | 14:34 |
| bogdando | and one more https://review.rdoproject.org/r/#/c/22108/ , prolly nice to have if using custom gerrit SSH keys != id_rsa for nodes ?.. | 14:34 |
| sshnaidm|ruck | zbr, I don't think so | 14:34 |
| bogdando | that's all yet on my contribution plate | 14:35 |
| sshnaidm|ruck | zbr, if we are talking about CI and OS images there | 14:35 |
| sshnaidm|ruck | zbr, not containers! | 14:35 |
| zbr | while I agree that we should do this in the image, i also think that we should also install it inside the role. | 14:35 |
| zbr | configuration management 101: bring the system to the desired state. | 14:35 |
| sshnaidm|ruck | bogdando, if it reports tripleo-ci tenant is not ready after a bootstrap, just need to wait about 10-15 mins, then just recheck in local gerrit | 14:36 |
| bogdando | sshnaidm|ruck: no, the tenant is fine, and the job gets queued | 14:36 |
| bogdando | just not running well :D | 14:36 |
| sshnaidm|ruck | bogdando, ah, ok | 14:37 |
| sshnaidm|ruck | bogdando, logs from job? or from executor | 14:37 |
| bogdando | will post some snippets | 14:37 |
| bogdando | as now I'm at the most distant point of failure | 14:37 |
| sshnaidm|ruck | zbr, then maybe a new role? ssh build key should build key, libselinux there is completely off-topic | 14:38 |
| zbr | i am trying to explain people that almost any ansible role should take care of its own dependencies, that is why it has the "dependencies" implemented. they are not random piecec of code that are supposed to run only when some magic stuff is happening, like when you already installed libselinux. If you need it to run you have two options: a) install it b) add a dependency on a role that does that already. | 14:39 |
| sshnaidm|ruck | jpena, maybe it's related to low priority? Although I thought it still should be queued there, just not running | 14:39 |
| zbr | otherwise: you have broken role. | 14:39 |
| *** ykarel is now known as ykarel|afk | 14:40 | |
| sshnaidm|ruck | zbr, libselinux is not dependency of any role, it's ansible dependency | 14:40 |
| jpena | sshnaidm|ruck: I think the issue is that the patch is already merged | 14:40 |
| zbr | ansible knows to skip including roles multiple times, so there is no performance impact if you include the same bootstrapping role 100 times. | 14:40 |
| jpena | yep, that's it | 14:40 |
| sshnaidm|ruck | jpena, oh, it doesn't work with merged? :( | 14:40 |
| jpena | there's "require: opendev.org: open: True" in the pipeline definition | 14:40 |
| zbr | no is not ansible dependency, is a soft-dependency. | 14:40 |
| bogdando | sshnaidm|ruck: http://paste.openstack.org/show/SROCnzcsBNccmvfeKUJA/ | 14:41 |
| sshnaidm|ruck | jpena, I see.. that's sad | 14:41 |
| zbr | also is a soft-dependency for the managed role, even worse. | 14:41 |
| bogdando | searched in executor by failed=1|FATAL|FAIL|UNREA|Ansible command | 14:41 |
| zbr | s/role/host | 14:41 |
| sshnaidm|ruck | zbr, not sure you can run anything without this package on selinux enabled systems | 14:41 |
| sshnaidm|ruck | zbr, it's very annoying ansible dep :) | 14:42 |
| zbr | sshnaidm|ruck: yes you can run a LOT! mainly the module that needs is the most is "template" module. | 14:42 |
| bogdando | the most interesting the job reports: | 14:43 |
| bogdando | Build succeeded. | 14:43 |
| bogdando | 14:43 | |
| bogdando | - tripleo-ci-centos-7-scenario007-multinode-oooq-container-dlrn-hash-tag http://localhost:8000/01/1001/1/check/tripleo-ci-centos-7-scenario007-multinode-oooq-container-dlrn-hash-tag/92fbf72/ : RETRY_LIMIT in 2m 24s (non-voting) | 14:43 |
| zbr | without it you may get ok. in fact you can use ansible even without having python on the target host, "raw" does work ok. I used it several times to boostrap python itself, | 14:43 |
| bogdando | and executor always logs Ansible complete, result RESULT_NORMAL code 2 | 14:43 |
| zbr | sshnaidm|ruck: but I agree, is an annoying soft-dependency, probably the most important after python itself. | 14:44 |
| bogdando | I want to nail it down, feel too much excited about zuul after reading https://model-engineers.com/en/company/references/success-stories/volvo-cars/ | 14:44 |
| bogdando | also having a local clone of OS infra for dev purposes looks nice :) | 14:45 |
| sshnaidm|ruck | bogdando, I see "changed=4 unreachable=0 failed=1" but don't see failed task | 14:46 |
| sshnaidm|ruck | bogdando, seems like something wrong with keys, maybe | 14:46 |
| bogdando | which line is it? | 14:46 |
| *** udesale has quit IRC | 14:46 | |
| sshnaidm|ruck | 184 | 14:47 |
| sshnaidm|ruck | bogdando, failed on 'secondary' host | 14:47 |
| *** udesale has joined #oooq | 14:47 | |
| sshnaidm|ruck | bogdando, are you trying multinode? | 14:47 |
| bogdando | sshnaidm|ruck: yes | 14:47 |
| bogdando | , 007 | 14:47 |
| sshnaidm|ruck | bogdando, I see.. I don't think I tried multinode recently, so might be surprises | 14:48 |
| sshnaidm|ruck | bogdando, on libvirt or rdo cloud tenant? | 14:48 |
| bogdando | libvirt | 14:48 |
| bogdando | there is also Add port to bridge on switch node failing (seems taking empty nodepool and switch _ips) | 14:49 |
| sshnaidm|ruck | bogdando, yeah, multinode is not tested well there | 14:49 |
| bogdando | wanted a real overcloud :) | 14:50 |
| bogdando | sshnaidm|ruck: http://paste.openstack.org/show/ghY2RKzHUKEYQmnNGvRO/ | 14:52 |
| sshnaidm|ruck | bogdando, then ovb | 14:52 |
| bogdando | sigh | 14:52 |
| bogdando | ovb runs docker yet | 14:53 |
| bogdando | wanted podman | 14:53 |
| bogdando | but anyway, I'll try | 14:53 |
| sshnaidm|ruck | bogdando, ovb rhel8 runs podman :) | 14:54 |
| bogdando | sshnaidm|ruck: perchance, any write-ups to start it quick and simple? | 14:54 |
| sshnaidm|ruck | bogdando, to start what? | 14:54 |
| bogdando | ovb in rdo cloud | 14:55 |
| bogdando | something with more specific instructions than https://github.com/rdo-infra/ansible-role-tripleo-ci-reproducer/blob/master/README.md | 14:55 |
| sshnaidm|ruck | bogdando, yes, just need to share images with you, need your tenant id | 14:55 |
| bogdando | sshnaidm|ruck: thanks, will post | 14:55 |
| sshnaidm|ruck | bogdando, for me it's much easier to run on rdo cloud, libvirt sucks | 14:55 |
| *** aakarsh has joined #oooq | 14:58 | |
| *** bogdando has quit IRC | 15:09 | |
| *** tosky has quit IRC | 15:15 | |
| weshay | zbr https://review.opendev.org/#/c/668851/ merged | 15:16 |
| weshay | zbr so .. scenario03 should work w/ rsyslog now | 15:17 |
| weshay | wait.. I'm getting my scenarios mixed up | 15:17 |
| weshay | zbr sorry.. https://review.opendev.org/#/c/668851/ is just scen001 / 002 | 15:20 |
| *** jfrancoa has quit IRC | 15:20 | |
| weshay | which are working now :) | 15:20 |
| zbr | yep, i knew about that one. i like changes that are removing stuff | 15:21 |
| *** ykarel|afk is now known as ykarel|away | 15:25 | |
| * marios almost out | 15:25 | |
| weshay | sshnaidm|ruck panda|rover I've udpated the cix board https://trello.com/b/j4IcIomh/production-chain-escalation | 15:26 |
| *** ccamacho has quit IRC | 15:26 | |
| sshnaidm|ruck | weshay, ack | 15:26 |
| weshay | scen003 / scen004 are on the board under "waiting on external deps" Please don't let folks close these tickets until they are resolved | 15:27 |
| weshay | they tend to like to just close shit | 15:27 |
| weshay | sshnaidm|ruck did the timeserver change fix vexx btw? | 15:27 |
| *** marios is now known as marios|out | 15:31 | |
| *** ykarel|away has quit IRC | 15:35 | |
| *** holser has quit IRC | 15:36 | |
| *** brault has quit IRC | 15:40 | |
| sshnaidm|ruck | weshay, it fixed time sync issues | 15:40 |
| sshnaidm|ruck | weshay, not it times out on containers start, I think it's too slow because registry is far away.. | 15:41 |
| sshnaidm|ruck | s/mot/now | 15:41 |
| weshay | sshnaidm|ruck ok.. thanks for the update.. I'll update the prod-chain council card | 15:41 |
| *** saneax has quit IRC | 15:48 | |
| *** marios|out has quit IRC | 15:50 | |
| panda|rover | upstream check queue is really clogged today ... :( | 15:51 |
| sshnaidm|ruck | weshay, panda|rover stein started promotion | 15:52 |
| *** ykarel|away has joined #oooq | 15:53 | |
| sshnaidm|ruck | weshay, panda|rover queens promoted.. | 15:53 |
| weshay | sshnaidm|ruck woot.. keeping us out of trouble ++ | 15:53 |
| weshay | saw that | 15:53 |
| sshnaidm|ruck | weshay, panda|rover rocky is failing on 020, rerunning it again | 15:54 |
| weshay | sshnaidm|ruck how's master? | 15:54 |
| weshay | rocky < master | 15:54 |
| sshnaidm|ruck | weshay, panda|rover and master fails most likely because of https://bugs.launchpad.net/tripleo/+bug/1842933 | 15:54 |
| openstack | Launchpad bug 1842933 in tripleo "OVB: Nova compute fails with error BadRequestException: 400: Client Error for url: http://192.168.24.3:6385/v1/nodes?fields=instance_id&associated=True, Field(s) "instance_id" are not valid" [Critical,Triaged] | 15:54 |
| sshnaidm|ruck | because rdo cloud is not stable it's difficult to say if we have a bug or not | 15:55 |
| weshay | sshnaidm|ruck aye.. thanks | 15:57 |
| sshnaidm|ruck | panda|rover, I'm out for today, just keep etherpad updated, I'll look in Sunday | 16:00 |
| *** sshnaidm|ruck is now known as sshnaidm|afk | 16:00 | |
| *** altlogbot_1 has quit IRC | 16:01 | |
| ykarel|away | sshnaidm|afk, panda|rover i noticed tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master-vexxhost running in periodic pipeline, is it intentionally running? | 16:01 |
| *** altlogbot_2 has joined #oooq | 16:01 | |
| ykarel|away | i mean why not running periodic version of it | 16:02 |
| *** irclogbot_1 has quit IRC | 16:02 | |
| ykarel|away | periodic release file | 16:02 |
| *** irclogbot_0 has joined #oooq | 16:03 | |
| panda|rover | ykarel|away: maybe that's the periodic version, just missing a name | 16:03 |
| ykarel|away | panda|rover, it's running with master release file | 16:04 |
| ykarel|away | QUICKSTART_RELEASE=master | 16:04 |
| ykarel|away | periodic have QUICKSTART_RELEASE=promotion-testing-hash-master | 16:04 |
| panda|rover | ykarel|away: I don't know the status there, usually we start with the check job as POC, running them peruiodically, then make the periodi version | 16:05 |
| panda|rover | when they start to get stable | 16:05 |
| ykarel|away | panda|rover, ack, but afair -vexxhost jobs are running from many days in check too | 16:05 |
| sshnaidm|afk | ykarel|away, it's not for promotion, just for periodic runs | 16:05 |
| ykarel|away | sshnaidm|afk, okk but good to run with same release files | 16:06 |
| sshnaidm|afk | ykarel|away, running in check for simplicity | 16:06 |
| ykarel|away | sshnaidm|afk, if it runs with same hash that would be help in ruling out the issue is due to rdo cloud | 16:06 |
| sshnaidm|afk | ykarel|away, fair enough | 16:07 |
| *** irclogbot_0 has quit IRC | 16:07 | |
| sshnaidm|afk | ykarel|away, will change it then | 16:07 |
| *** irclogbot_0 has joined #oooq | 16:07 | |
| ykarel|away | sshnaidm|afk, ack | 16:07 |
| chandankumar | zbr: sshnaidm|afk openstack-ansible-deploy-aio_metal-debian-stable (2. attempt) | 16:18 |
| chandankumar | if a job failed, rechecked again, it will show attempts | 16:18 |
| *** kopecmartin is now known as kopecmartin|off | 16:18 | |
| chandankumar | weshay: we also need to start preparing for train release | 16:21 |
| weshay | chandankumar not quite yet | 16:21 |
| weshay | but yes | 16:21 |
| chandankumar | one more ping for review https://review.opendev.org/#/c/673481/ | 16:33 |
| chandankumar | see ya tomorrow | 16:33 |
| *** chandankumar is now known as raukadah | 16:33 | |
| zbr | weshay: can you sf https://review.opendev.org/#/c/679618/ ? of so thanks. | 16:47 |
| weshay | done | 16:49 |
| *** jaosorior has quit IRC | 16:51 | |
| *** tesseract has quit IRC | 17:06 | |
| *** jpena is now known as jpena|off | 17:07 | |
| *** udesale has quit IRC | 17:18 | |
| *** jaosorior has joined #oooq | 17:22 | |
| *** brault has joined #oooq | 18:05 | |
| *** brault has quit IRC | 18:09 | |
| dtantsur | sshnaidm|afk, ykarel|away, talk to nova folks, they've been refactoring how nova talks to ironic | 18:27 |
| dtantsur | although this one may be an openstacksdk bug | 18:27 |
| *** dtantsur is now known as dtantsur|afk | 18:36 | |
| *** ykarel|away has quit IRC | 18:56 | |
| *** Goneri has quit IRC | 20:44 | |
| *** aakarsh has quit IRC | 20:45 | |
| *** dsneddon has quit IRC | 20:46 | |
| *** Vorrtex has quit IRC | 20:53 | |
| -openstackstatus- NOTICE: Gerrit is being restarted to pick up configuration changes. Should be quick. Sorry for the interruption. | 21:12 | |
| *** dsneddon has joined #oooq | 21:17 | |
| *** dsneddon has quit IRC | 22:37 | |
| *** dsneddon has joined #oooq | 23:05 | |
| *** dsneddon has quit IRC | 23:10 | |
| *** dsneddon has joined #oooq | 23:40 | |
| *** dsneddon has quit IRC | 23:46 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!