| *** holser has quit IRC | 00:02 | |
| rlandy | weshay: you still around? | 00:15 |
|---|---|---|
| rlandy | 2020-02-10 19:00:41.292125 | primary | The `libvirt` module is not importable. Check the requirements. | 00:15 |
| rlandy | env setup on rhel | 00:15 |
| *** rfolco has quit IRC | 01:19 | |
| rlandy | nvm - - got by that commented on review | 02:00 |
| *** rlandy has quit IRC | 02:09 | |
| *** dsneddon_ has quit IRC | 02:52 | |
| *** dsneddon_ has joined #oooq | 03:20 | |
| *** dsneddon_ has quit IRC | 03:36 | |
| *** dsneddon_ has joined #oooq | 04:07 | |
| *** dsneddon_ has quit IRC | 04:12 | |
| *** udesale has joined #oooq | 04:19 | |
| *** ykarel|away is now known as ykarel | 04:20 | |
| *** skramaja has joined #oooq | 04:45 | |
| *** dsneddon_ has joined #oooq | 04:48 | |
| *** dsneddon_ has quit IRC | 04:53 | |
| *** dsneddon_ has joined #oooq | 05:20 | |
| *** dsneddon_ has quit IRC | 05:25 | |
| *** raukadah is now known as chkumar|rover | 05:28 | |
| *** saneax has joined #oooq | 05:54 | |
| *** jmasud has joined #oooq | 05:56 | |
| *** dsneddon_ has joined #oooq | 05:59 | |
| *** sanjayu_ has joined #oooq | 06:01 | |
| *** dsneddon_ has quit IRC | 06:03 | |
| *** saneax has quit IRC | 06:03 | |
| *** chem has quit IRC | 06:14 | |
| *** ratailor has joined #oooq | 06:21 | |
| *** marios has joined #oooq | 06:25 | |
| *** dsneddon_ has joined #oooq | 06:31 | |
| *** dsneddon_ has quit IRC | 06:36 | |
| *** ccamacho has quit IRC | 06:50 | |
| *** jfrancoa has joined #oooq | 06:53 | |
| *** jtomasek has joined #oooq | 07:01 | |
| *** dsneddon_ has joined #oooq | 07:09 | |
| chkumar|rover | zbr, morning | 07:10 |
| chkumar|rover | zbr, around? | 07:10 |
| zbr | o/ | 07:10 |
| chkumar|rover | zbr, I trying to reproduced locally | 07:11 |
| chkumar|rover | but it is not coming on centos 7 | 07:11 |
| chkumar|rover | ensure_text comes from six | 07:11 |
| chkumar|rover | but still not working | 07:11 |
| zbr | i suspect virtualenv 20 released yesterday is causing it. | 07:11 |
| chkumar|rover | zbr, yes, that is | 07:11 |
| zbr | if you upgrade six from pip, it will work. | 07:11 |
| chkumar|rover | let me try that | 07:12 |
| chkumar|rover | zbr, which piece of code is updaitng this pip:20.0.2 virtualenv:20.0.1 setuptools:44.0.0 ? | 07:13 |
| zbr | which version of six do we have on centos-7 ? | 07:13 |
| chkumar|rover | 1.14.0 | 07:14 |
| *** dsneddon_ has quit IRC | 07:14 | |
| zbr | and where do we install newer virtualenv? | 07:14 |
| zbr | because 20 is clearly not from rpm. | 07:15 |
| chkumar|rover | https://bugs.launchpad.net/tripleo/+bug/1862724 | 07:15 |
| openstack | Launchpad bug 1862724 in tripleo "ERROR:root:ImportError: cannot import name ensure_text" [Critical,In progress] - Assigned to chandan kumar (chkumar246) | 07:15 |
| chkumar|rover | check last comment | 07:15 |
| marios | sshnaidm|afk: can you check that when you get a chance you previously -1 it but it looks to be idle for few days now i just re-added +2 https://review.opendev.org/#/c/704979/6 | 07:15 |
| zbr | so somewhere inside install-deps | 07:15 |
| zbr | you need to extract that line so we can file a bug against virtualenv | 07:16 |
| zbr | the log section looks weird, I see installing python-virtualenv-15.1.0-2.el7.noarch but using 20.0.2 | 07:16 |
| zbr | the workaround for us should be add a six>=x.y.z near the line where we install virtualenv via pip. | 07:17 |
| zbr | and only once we point us the bug, i suspect they missed to mention min version for six. | 07:18 |
| zbr | https://github.com/pypa/virtualenv/issues/1551 | 07:19 |
| chkumar|rover | zbr, just adding pip install upgrade six fixes it let me updatet he patch | 07:22 |
| chkumar|rover | sorry not fixed it | 07:22 |
| chkumar|rover | Defaulting to user installation because normal site-packages is not writeable new message is also interesitn | 07:23 |
| zbr | you need to upgrade system six | 07:23 |
| zbr | chkumar|rover: do you remember what I said during our meetings about python2.7, for more than 3 months. | 07:25 |
| zbr | this is the start of an endless row. | 07:25 |
| zbr | setuptools, pip, virtualenv, tox, six,... all will break on centos-7 and we will not be able to install correct dependencies without breaking system. | 07:26 |
| zbr | in this case gabor made a mistake of ignoring feedback and ignoring non-wheel deployments, but it will happen again. | 07:27 |
| zbr | even if we pin everything to known good, we may soon face other problems where we need to use newer features. | 07:28 |
| zbr | if I would blame someone, it would be our rpm packages which they never update packages like six, setuptools, unless there is a storm. | 07:29 |
| zbr | and when this happens it takes weeks for each of them to be upgraded. | 07:29 |
| zbr | ouch, but i see the same on centos-8.... | 07:31 |
| chkumar|rover | zbr, I hope this https://review.opendev.org/#/c/707062/6/install-deps.sh will upgrade six | 07:32 |
| zbr | if you sudo, you can upgrade six and I doubt it will break. six is very stable package afaik. | 07:33 |
| zbr | chkumar|rover: bindep is broken, that is no longer oooq issue, we need to switch to infra. see https://dashboard.zuul.ansible.com/t/ansible/build/1e148fb6d2684932a0620d6b23e5909d/console | 07:35 |
| zbr | probably we need to add a <20.0 to it, until the dust settles. | 07:36 |
| chkumar|rover | zbr, but still the question is what is updating virtualenv related packages in install-deps? | 07:37 |
| zbr | virtualenv itself downlods newer setuptools/pip if you do not mention download=False | 07:39 |
| zbr | but the problem in this case is to find place where we install virtualenv from pip and put a limit | 07:40 |
| zbr | probably there are LOTs of places. | 07:40 |
| *** dsneddon_ has joined #oooq | 07:41 | |
| *** ykarel is now known as ykarel|lunch | 07:45 | |
| *** dsneddon_ has quit IRC | 07:46 | |
| *** udesale has quit IRC | 07:46 | |
| *** udesale has joined #oooq | 07:47 | |
| *** sanjayu_ has quit IRC | 07:58 | |
| zbr | only interesting stuff since last night, https://review.opendev.org/#/c/706871/ | 07:59 |
| *** jmasud has quit IRC | 08:11 | |
| *** jmasud has joined #oooq | 08:13 | |
| chkumar|rover | zbr, finally upgrading system six | 08:14 |
| chkumar|rover | and let's see how it goes | 08:14 |
| chkumar|rover | zbr, bindep task is performed earlier | 08:14 |
| zbr | chkumar|rover: read infra | 08:14 |
| zbr | chkumar|rover: i being told that we need to add `--seeder=pip` to virtuaenv command, but be careful, that option works only with 20+ | 08:17 |
| zbr | they forced virtualenv to be >20 upstream. | 08:17 |
| zbr | so we have a fix to centos-8 that will totally break centos-7. | 08:17 |
| chkumar|rover | zbr, done | 08:25 |
| chkumar|rover | zbr, let's see how ti goes | 08:25 |
| *** tosky has joined #oooq | 08:27 | |
| chkumar|rover | zbr, marios heading to wework, will back in 45 mins | 08:28 |
| *** tesseract has joined #oooq | 08:29 | |
| *** bogdando has joined #oooq | 08:42 | |
| *** ccamacho has joined #oooq | 08:46 | |
| *** dsneddon_ has joined #oooq | 08:46 | |
| *** ykarel|lunch is now known as ykarel | 08:50 | |
| *** dsneddon_ has quit IRC | 08:51 | |
| *** jpena|off is now known as jpena | 08:54 | |
| *** ccamacho has quit IRC | 09:01 | |
| *** amoralej|off is now known as amoralej | 09:03 | |
| chkumar|rover | zbr, seeder does not worked | 09:07 |
| *** sanjayu_ has joined #oooq | 09:08 | |
| chkumar|rover | zbr, is it seed or seeder? | 09:10 |
| chkumar|rover | https://virtualenv.pypa.io/en/stable/cli_interface.html | 09:10 |
| chkumar|rover | --seeder | 09:10 |
| zbr | yeah | 09:10 |
| chkumar|rover | which one? | 09:10 |
| chkumar|rover | --seeder does not worked | 09:11 |
| chkumar|rover | https://review.opendev.org/#/c/707062/9/install-deps.sh | 09:11 |
| *** dtantsur|afk is now known as dtantsur | 09:12 | |
| chkumar|rover | virtualenv --seed=pip .venv | 09:12 |
| chkumar|rover | zbr, so both seed and seeder works | 09:16 |
| chkumar|rover | locally | 09:17 |
| *** dsneddon_ has joined #oooq | 09:20 | |
| zbr | maybe we decide to drop taiga this sprint, it drives me crazy, imposible to convert between issue, story, task, epic, taiga is like Vogon poetry! | 09:23 |
| *** dsneddon_ has quit IRC | 09:25 | |
| *** sshnaidm|afk is now known as sshnaidm | 09:25 | |
| zbr | marios: chkumar|rover: please look again at https://review.opendev.org/#/c/706237/ | 09:29 |
| zbr | increasing test coverage, really needed for other changes. | 09:30 |
| *** dsneddon_ has joined #oooq | 09:30 | |
| marios | will do zbr | 09:31 |
| zbr | marios: i hope you dont want a story for this. | 09:31 |
| chkumar|rover | zbr, I tried all hooks and crooks but not working | 09:40 |
| chkumar|rover | zbr, please have a look https://review.opendev.org/#/c/707062/ | 09:40 |
| sshnaidm | chkumar|rover, can I help with py2 stuff? | 09:42 |
| chkumar|rover | sshnaidm, yes sure | 09:42 |
| chkumar|rover | sshnaidm, we tried to updating six, using seed | 09:42 |
| chkumar|rover | but it does not work | 09:42 |
| sshnaidm | chkumar|rover, what is the problem? | 09:43 |
| chkumar|rover | sshnaidm, https://bugs.launchpad.net/tripleo/+bug/1862724 | 09:43 |
| openstack | Launchpad bug 1862724 in tripleo "ERROR:root:ImportError: cannot import name ensure_text" [Critical,In progress] - Assigned to chandan kumar (chkumar246) | 09:43 |
| *** derekh has joined #oooq | 09:44 | |
| sshnaidm | chkumar|rover, can we limit venv version? | 09:45 |
| chkumar|rover | sshnaidm, yes, but where we can do that? | 09:45 |
| chkumar|rover | sshnaidm, it is getting updating somehow | 09:45 |
| chkumar|rover | *updating | 09:46 |
| sshnaidm | chkumar|rover, in the start of job, it's preinstalled in the image, so we need to remove it as a package and install from pip | 09:46 |
| chkumar|rover | sshnaidm, ok updating the patch | 09:48 |
| *** chem has joined #oooq | 09:51 | |
| chkumar|rover | sshnaidm, https://review.opendev.org/#/c/707062/13/install-deps.sh | 09:56 |
| chkumar|rover | please have a look | 09:56 |
| sshnaidm | chkumar|rover, ack, let's see.. | 09:59 |
| chkumar|rover | sshnaidm, boom it worked | 10:04 |
| sshnaidm | chkumar|rover++ | 10:04 |
| chkumar|rover | sshnaidm, https://review.rdoproject.org/zuul/stream/c946f00b13984db299cf11dd28913fec?logfile=console.log | 10:04 |
| chkumar|rover | zbr++ | 10:05 |
| chkumar|rover | zbr, so seed is not gona work, we need to downgrade virtualenv | 10:05 |
| sshnaidm | chkumar|rover, I think you can remove -2 from the patch | 10:05 |
| chkumar|rover | sshnaidm, yes | 10:05 |
| chkumar|rover | sshnaidm, doe | 10:06 |
| chkumar|rover | done | 10:06 |
| sshnaidm | chkumar|rover, what's next? | 10:06 |
| chkumar|rover | sshnaidm, we can merge this | 10:06 |
| chkumar|rover | sshnaidm, there is one more bug | 10:07 |
| chkumar|rover | sshnaidm, https://bugs.launchpad.net/tripleo/+bug/1862674 | 10:07 |
| openstack | Launchpad bug 1862674 in tripleo "build-containers jobs failing on "no module named http"" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 10:07 |
| chkumar|rover | https://review.opendev.org/#/c/707054/ | 10:07 |
| chkumar|rover | let me add the depends on here | 10:07 |
| chkumar|rover | sshnaidm, it broke other places | 10:09 |
| chkumar|rover | [build-test-packages : Install and update pip] | 10:09 |
| chkumar|rover | https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/roles/build-test-packages/tasks/main.yml#L94 | 10:10 |
| sshnaidm | chkumar|rover, well, I hope it will use the installed venv | 10:12 |
| sshnaidm | chkumar|rover, we don't update virtualenv in jobs, right? | 10:12 |
| chkumar|rover | sshnaidm, nope | 10:12 |
| sshnaidm | great | 10:12 |
| sshnaidm | chkumar|rover, so the problem is with new ironic client? | 10:13 |
| chkumar|rover | sshnaidm, nono, now the new virtualenv broke ansible pip module i think | 10:14 |
| chkumar|rover | logs coming up | 10:14 |
| chkumar|rover | sshnaidm, https://logserver.rdoproject.org/62/707062/13/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens-branch/c946f00/job-output.txt | 10:14 |
| chkumar|rover | sshnaidm, Could not get output from /usr/bin/virtualenv --help: Traceback (most recent call last): | 10:15 |
| chkumar|rover | AttributeError: 'module' object has no attribute 'main' | 10:15 |
| sshnaidm | wow | 10:15 |
| sshnaidm | chkumar|rover, this is from latest run? | 10:16 |
| sshnaidm | chkumar|rover, ah, I see | 10:16 |
| chkumar|rover | sshnaidm, yes | 10:17 |
| *** holser has joined #oooq | 10:20 | |
| chkumar|rover | sshnaidm, if we look at logs, we remove python-virtualenv | 10:20 |
| chkumar|rover | and it got again installed | 10:20 |
| sshnaidm | chkumar|rover, it's installed in some /usr/local or something | 10:21 |
| sshnaidm | chkumar|rover, but path still points to /usr/bin/virtualenv | 10:21 |
| sshnaidm | I hit this problem before, trying to recall how it's solved | 10:21 |
| sshnaidm | 2020-02-11 10:04:10.403124 | primary | warning: file /usr/bin/virtualenv-2.7: remove failed: No such file or directory | 10:23 |
| sshnaidm | 2020-02-11 10:04:10.403179 | primary | warning: file /usr/bin/virtualenv-2: remove failed: No such file or directory | 10:23 |
| sshnaidm | but doesn't remove /usr/bin/virtualenv | 10:23 |
| chkumar|rover | sshnaidm, other thing https://github.com/openstack/tripleo-quickstart/blob/master/quickstart.sh#L113 called twice in toci, as first we call install deps then bootstrap | 10:24 |
| chkumar|rover | what about doing this? | 10:25 |
| chkumar|rover | remove python-virtualenv from pkg then from pip also and then install it via pip? | 10:25 |
| chkumar|rover | let me try that and see how it goes | 10:26 |
| chkumar|rover | virtualenv is coming from bindep.txt file | 10:27 |
| sshnaidm | chkumar|rover, I don't think it installs it from bindep, lemme check in logs | 10:27 |
| chkumar|rover | 2020-02-11 10:04:16.668806 | primary | install_package_deps_via_bindep | 10:28 |
| chkumar|rover | 2020-02-11 10:04:17.559732 | primary | python-virtualenv | 10:28 |
| sshnaidm | chkumar|rover, damn.. then need to do this again after this line | 10:29 |
| sshnaidm | like twice | 10:30 |
| ykarel | iirc virtualenv used to preinstall in nodepool images, need to check if it's still the case | 10:30 |
| sshnaidm | ykarel, it is | 10:30 |
| sshnaidm | now we try to remove it anyhow :) | 10:30 |
| ykarel | sshnaidm, ack | 10:30 |
| sshnaidm | ykarel, can't we just install py3 as default in c7 images when building them? | 10:31 |
| sshnaidm | and forget about py2 forever.. | 10:31 |
| chkumar|rover | sshnaidm, I think we can do that, but issues python3-selinux it is available or not | 10:32 |
| ykarel | sshnaidm, u mean install python3 binary in centos7? | 10:32 |
| ykarel | or install other python3 deps in centos7? | 10:32 |
| ykarel | for centos7 python3 stack is minimal | 10:32 |
| ykarel | so should not support all use cases | 10:33 |
| *** dsneddon_ has quit IRC | 10:33 | |
| sshnaidm | ykarel, I think it should cover what we have right now for py2 there in preinstall | 10:33 |
| ykarel | if it's just python3 install it can be done outside of images | 10:34 |
| zbr | until we get a newer six in OS, we have to install newer version manually. | 10:34 |
| sshnaidm | ykarel, you mean we can't run a tripleo job with py3 as a default in centos7? | 10:34 |
| zbr | other approaches may prove much harder to implement | 10:34 |
| ykarel | sshnaidm, not sure about deployment, but for container build i tried that | 10:35 |
| ykarel | like https://review.opendev.org/#/c/699555/ | 10:35 |
| ykarel | iirc running ansible had issues on python3 | 10:36 |
| ykarel | and deployment rely hugely on it | 10:36 |
| sshnaidm | well, I still think we need to reconsider this | 10:37 |
| sshnaidm | otherwise every day we'll see new surprises | 10:37 |
| ykarel | sshnaidm, so the current issues are seen in non master branches as well? | 10:38 |
| ykarel | if just master, then c7 will go soon | 10:38 |
| ykarel | if others, then yes need to think about it | 10:38 |
| sshnaidm | ykarel, it's CI code, so everywhere | 10:38 |
| ykarel | sshnaidm, ack | 10:40 |
| sshnaidm | chkumar|rover, seems working | 10:44 |
| chkumar|rover | sshnaidm, yes https://review.opendev.org/#/c/707062/ | 10:44 |
| chkumar|rover | brb | 10:49 |
| *** dsneddon_ has joined #oooq | 11:02 | |
| *** dsneddon_ has quit IRC | 11:07 | |
| zbr | chkumar|rover: i think we should not do anything, everything is falling appart and we cannot do anything until infra is fixing it. | 11:10 |
| zbr | they made a decision yesterday which I think it was rushed, but now we need to wait for them to wake up. | 11:11 |
| chkumar|rover | sshnaidm, few jobs behaving weired https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_8cb/707062/14/check/tripleo-ci-centos-7-containers-multinode-rocky/8cb5f32/job-output.txt | 11:12 |
| chkumar|rover | it failed at that place | 11:13 |
| sshnaidm | chkumar|rover, yeah, only for rocky and queens.. | 11:13 |
| chkumar|rover | sshnaidm, not sure what to do there | 11:14 |
| sshnaidm | chkumar|rover, I see it still installs virtualenv with bindep | 11:16 |
| chkumar|rover | sshnaidm, where in build test packages role? | 11:17 |
| sshnaidm | chkumar|rover, no, it doesn't.. Removed: | 11:19 |
| sshnaidm | 2020-02-11 10:43:03.143303 | primary | python-virtualenv.noarch 0:15.1.0-2.el7 | 11:19 |
| chkumar|rover | sshnaidm, for few jobs it is causing problem https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/build-test-packages/tasks/main.yml#L44 | 11:19 |
| sshnaidm | chkumar|rover, maybe race condition, where are depending on release in this case? | 11:19 |
| chkumar|rover | need to check | 11:20 |
| sshnaidm | Successfully installed appdirs-1.4.3 configparser-4.0.2 contextlib2-0.6.0.post1 filelock-3.0.12 importlib-metadata-1.5.0 importlib-resources-1.0.2 pathlib2-2.3.5 scandir-1.10.0 six-1.14.0 typing-3.7.4.1 virtualenv-20.0.1 zipp-1.1.0 | 11:21 |
| sshnaidm | chkumar|rover, is 20.0.1 ok? | 11:21 |
| chkumar|rover | sshnaidm, not ok | 11:21 |
| chkumar|rover | sshnaidm, we need lower than that | 11:21 |
| chkumar|rover | sshnaidm, https://review.opendev.org/#/c/707062/14/install-deps.sh | 11:21 |
| chkumar|rover | "virtualenv<20.0.1" | 11:21 |
| sshnaidm | we need -x in this script, nothing clear what's running.. | 11:22 |
| sshnaidm | chkumar|rover, interesting: ollecting virtualenv<20.0.1 | 11:23 |
| sshnaidm | 2020-02-11 10:43:03.777146 | primary | Downloading http://mirror.gra1.ovh.openstack.org/pypifiles/packages/62/89/72ab7d18e001a23b25eb75ce71bbf9e2a3c142e2dad2e4e2b88012e4ca25/virtualenv-20.0.0-py2.py3-none-any.whl (4.6 MB) | 11:23 |
| sshnaidm | so why the hell it installs 20.0.1 | 11:24 |
| chkumar|rover | sshnaidm, may be because -I option | 11:25 |
| chkumar|rover | sshnaidm, -I, --ignore-installed Ignore the installed packages (reinstalling instead). | 11:25 |
| sshnaidm | chkumar|rover, why to use it? | 11:25 |
| sshnaidm | chkumar|rover, let's remove this option | 11:25 |
| chkumar|rover | yup | 11:26 |
| zbr | chkumar|rover: you will mess the entire systems if you start to allow pip to upgrade system packages. | 11:26 |
| sshnaidm | still seems like race condition somewhere.. | 11:26 |
| zbr | i was told for years that we should only use rpms.... | 11:27 |
| zbr | sudo + pip = yolo | 11:28 |
| chkumar|rover | sshnaidm, zbr I will check with jpena to ship pip and virtualenv in a seperate repo | 11:28 |
| sshnaidm | chkumar|rover, anyway it's preinstalled | 11:28 |
| sshnaidm | chkumar|rover, let's try without -I and see.. | 11:28 |
| zbr | if we bundle a newer six there, we should be fine | 11:28 |
| chkumar|rover | sshnaidm, done | 11:29 |
| *** dsneddon_ has joined #oooq | 11:39 | |
| chkumar|rover | sshnaidm, I think we need to remove virtualenv and then install | 11:43 |
| *** dsneddon_ has quit IRC | 11:44 | |
| sshnaidm | Found existing installation: virtualenv 20.0.1 | 11:44 |
| sshnaidm | chkumar|rover, so remove with pip too? | 11:44 |
| chkumar|rover | sshnaidm, nope | 11:45 |
| sshnaidm | where is this 20.0.1 coming from.. | 11:45 |
| zbr | sshnaidm: nodepool images. | 11:45 |
| sshnaidm | zbr, nodepool images has rpm, not pip installed | 11:45 |
| sshnaidm | it installs python-virtualenv package in nodepool | 11:46 |
| chkumar|rover | sshnaidm, first removing virtualenv then installing | 11:46 |
| sshnaidm | chkumar|rover, pip, yum - ? | 11:46 |
| chkumar|rover | let'see how it goes this time | 11:46 |
| chkumar|rover | pip is coming from rpm | 11:47 |
| sshnaidm | chkumar|rover, ok, let's see | 11:47 |
| sshnaidm | chkumar|rover, what is version of venv we have in centos7 repo? | 11:48 |
| sshnaidm | chkumar|rover, if it's 15.0.1, so should be fine for us.. | 11:49 |
| zbr | in fact any version <20 is ok. | 11:50 |
| zbr | try to do something like: https://review.opendev.org/#/c/707109/1/tools/fixup_stuff.sh | 11:50 |
| zbr | but afaik, infra cased this bump virtualenv. | 11:51 |
| chkumar|rover | sshnaidm, yes | 11:52 |
| zbr | btw, we should raise a ticket to bump six on rhel 7+8, we will need it. | 11:55 |
| *** rfolco has joined #oooq | 12:02 | |
| *** amoralej is now known as amoralej|lunch | 12:04 | |
| marios | chkumar|rover: have you seen that one yet Exception raised: No module named http https://logserver.rdoproject.org/07/24807/12/check/tripleo-ci-promotion-staging/9e1bc7b/logs/tripleo-container-image.prepare.log | 12:04 |
| marios | chkumar|rover: perhaps is one of the known 'py27' issues | 12:05 |
| marios | chkumar|rover: (yeah that one https://bugs.launchpad.net/tripleo/+bug/1862674 ) | 12:08 |
| openstack | Launchpad bug 1862674 in tripleo "build-containers jobs failing on "no module named http"" [Critical,In progress] - Assigned to chandan kumar (chkumar246) | 12:08 |
| *** udesale_ has joined #oooq | 12:11 | |
| *** jmasud has quit IRC | 12:11 | |
| *** udesale has quit IRC | 12:13 | |
| sshnaidm | Successfully installed virtualenv-20.0.0 | 12:14 |
| sshnaidm | 2020-02-11 11:53:18.067129 | primary | ERROR:root:ImportError: cannot import name ensure_text | 12:14 |
| *** jmasud has joined #oooq | 12:14 | |
| sshnaidm | chkumar|rover, so maybe <20.0.0 ? | 12:14 |
| chkumar|rover | sshnaidm, yes doing that | 12:15 |
| *** jaosorior has joined #oooq | 12:15 | |
| weshay | chkumar|rover, 0/ | 12:15 |
| weshay | arxcruz, xferred ownership to you re: beaker.. you can add the idrac info to beaker now | 12:15 |
| arxcruz | weshay: ack | 12:16 |
| chkumar|rover | weshay, \o | 12:16 |
| weshay | arxcruz, you'll want to test by rebooting it via beaker | 12:16 |
| arxcruz | weshay: ok | 12:16 |
| weshay | chkumar|rover, so.. https://review.opendev.org/#/c/707062/ eh | 12:16 |
| chkumar|rover | weshay, yes, donot know what is the perfect solution | 12:17 |
| weshay | chkumar|rover, sshnaidm /me wonders if we just go w/ python3 on centos for tq now | 12:17 |
| arxcruz | python3 >= 3.6 everywhere | 12:18 |
| weshay | arxcruz, well.. we just got that ability on centos-7 | 12:18 |
| *** dsneddon_ has joined #oooq | 12:18 | |
| weshay | chkumar|rover, you want to chat? | 12:18 |
| chkumar|rover | weshay, yes | 12:18 |
| chkumar|rover | weshay, bring sshnaidm and zbr also | 12:19 |
| weshay | https://meet.google.com/xrt-cjvp-uqb?authuser=1 chkumar|rover sshnaidm zbr | 12:19 |
| *** dsneddon_ has quit IRC | 12:23 | |
| *** dtantsur is now known as dtantsur|brb | 12:27 | |
| *** jaosorior has quit IRC | 12:32 | |
| *** dpawlik has joined #oooq | 12:34 | |
| *** ykarel is now known as ykarel|afk | 12:34 | |
| dpawlik | chkumar|rover, could you check if you see test file https://images-vexxhost.rdoproject.org:4433/rhel/ | 12:35 |
| dpawlik | on one of the nodes, that are allowed in security group (38.145.32.0/22 ) | 12:36 |
| *** jpena is now known as jpena|lunch | 12:38 | |
| *** sanjayu__ has joined #oooq | 12:39 | |
| rfolco | arxcruz, weshay whats the secret behind python3.6 errors ? https://review.rdoproject.org/r/#/c/24762/ | 12:39 |
| arxcruz | rfolco: 2020-02-11 12:27:02.600335 | rdo-fedora-28 | ERROR: InvocationError for command /usr/bin/python2 -m virtualenv --no-download --python /usr/bin/python3.6 py36 (exited with code 1) | 12:41 |
| sshnaidm | rfolco, arxcruz this is a problem with new virtualenv | 12:41 |
| *** sanjayu_ has quit IRC | 12:41 | |
| weshay | sshnaidm, chkumar|rover zbr https://hackmd.io/Javdw507QbqweXybJmYBWg | 12:42 |
| dpawlik | weshay, epic site xD | 12:46 |
| zbr | https://review.opendev.org/#/c/707064/1 | 12:47 |
| *** sanjayu_ has joined #oooq | 12:51 | |
| *** rlandy has joined #oooq | 12:51 | |
| arxcruz | weshay: do you have any doc how to configure idrac on beaker ? | 12:51 |
| *** dsneddon_ has joined #oooq | 12:51 | |
| *** sanjayu__ has quit IRC | 12:53 | |
| chkumar|rover | sshnaidm, this time all are in good shape | 12:53 |
| *** dsneddon_ has quit IRC | 12:56 | |
| chkumar|rover | rlandy, gate is broken, recheck not gona help | 12:59 |
| rlandy | chkumar|rover: I am not interested in getting the patch through - depends on is not reviewed yet - I want to see fs020 run again | 13:02 |
| chkumar|rover | rlandy, sorry, all our jobs are broken due to new venv drop | 13:05 |
| zbr | i found another source of problems, on nodepool they still install multiple versions of python (and virtualenv for each). | 13:08 |
| zbr | if you are using ansible to create the virtualenv you have no option to enforce use of "pythonX -m virtualenv", as it does not allow you that. | 13:08 |
| zbr | this may explain why chkumar|rover failed to fix the issue upgrading six, probably he upgraded only one of them. | 13:09 |
| chkumar|rover | dpawlik, will take a look in some time | 13:10 |
| * chkumar|rover headed home, see ya in an hour | 13:11 | |
| *** ratailor has quit IRC | 13:11 | |
| arxcruz | rfolco: sshnaidm https://review.opendev.org/#/c/707133/1/tools/fixup_stuff.sh is how they are trying to fix in devstack | 13:12 |
| sshnaidm | arxcruz, we do almost the same | 13:14 |
| rfolco | sshnaidm, can you please paste our fix ? | 13:14 |
| sshnaidm | https://review.opendev.org/#/c/707062/\ | 13:14 |
| weshay | arxcruz, yup | 13:15 |
| weshay | arxcruz, look at some of our other machines.. | 13:15 |
| weshay | ipmilman, mgmt addr, user, password | 13:15 |
| weshay | is what you need.. | 13:15 |
| rfolco | sshnaidm, thx | 13:15 |
| weshay | then test w/ a reboot | 13:15 |
| arxcruz | weshay: ack | 13:16 |
| weshay | sshnaidm, so the DF is struggling w/ the heat project | 13:16 |
| weshay | sshnaidm, https://review.rdoproject.org/r/#/c/24879/ | 13:16 |
| weshay | wdyt ^ and does it have capacity to keep monitoring $new things | 13:17 |
| * weshay was thinking about adding a page for heat, check/gate blah blah | 13:17 | |
| sshnaidm | weshay, well, maybe | 13:18 |
| *** amoralej|lunch is now known as amoralej | 13:18 | |
| sshnaidm | weshay, not sure we can help here in unblocking heat queue | 13:18 |
| sshnaidm | weshay, but fine to monitor it somewhere in "Non-Tripleo dashboard" | 13:19 |
| sshnaidm | weshay, but I don't see anything in heat queue right now.. does it exist? | 13:20 |
| weshay | sshnaidm, ya.. was just hoping to provide visisibility | 13:21 |
| weshay | sshnaidm, /me looks at zuul config for heat | 13:21 |
| weshay | sshnaidm, no.. /me abandons | 13:23 |
| weshay | it must run in the integration pipeline | 13:23 |
| *** derekh has quit IRC | 13:27 | |
| rfolco | community mtg starting at https://meet.google.com/bqx-xwht-wky?authuser=1 | 13:30 |
| rfolco | agenda https://hackmd.io/IhMCTNMBSF6xtqiEd9Z0Kw?both | 13:30 |
| *** marios is now known as marios|call | 13:31 | |
| migi | tristanC: hey could you tell me why line 33 is needed ? https://softwarefactory-project.io/cgit/config/tree/containers/centos-8/Dockerfile#n33 | 13:31 |
| *** dsneddon_ has joined #oooq | 13:31 | |
| *** jpena|lunch is now known as jpena | 13:32 | |
| *** dsneddon_ has quit IRC | 13:36 | |
| *** dsneddon_ has joined #oooq | 13:38 | |
| rlandy | migi: ah - you are building centos8 .. we're building osp on rhel | 13:38 |
| migi | rlandy: nope, I need rhel8 base | 13:39 |
| rlandy | migi: have you started? | 13:39 |
| migi | rlandy: I have env up and running with some base podman | 13:39 |
| migi | rlandy: ok let's touch after this mtg I will show you what I have | 13:40 |
| rlandy | migi: ok - so we can use podman, kolla and the rel eng dockerfiles to build | 13:40 |
| migi | rlandy: I think we are talking about two use-cases, but let's sync in 20min | 13:40 |
| rlandy | migi: ack | 13:41 |
| migi | rlandy: re: https://review.opendev.org/#/c/682049/ | 13:41 |
| migi | rlandy: to get this in downstream CI | 13:41 |
| dpawlik | chkumar|rover, weshay: there was one thing that I didn;t mention during the meeting: for rcn content, there will be dedicated user: rcn-uploader | 13:47 |
| dpawlik | uploader will be for just for public images, rcn-uploader just for rcn stuff | 13:48 |
| *** dtantsur|brb is now known as dtantsur | 13:49 | |
| chkumar|rover | arxcruz, marios|call rlandy https://review.opendev.org/#/c/707147/1 please have a look, when free | 13:55 |
| *** derekh has joined #oooq | 14:01 | |
| *** ykarel|afk is now known as ykarel | 14:05 | |
| marios|call | panda: when you get a chance wdyt... i added ppc_manifests to the stage-info.yaml as i needed some way to signal to the promotion staging python tests and that is the config they use https://review.rdoproject.org/r/#/c/24807/10..12/ci-scripts/container-push/roles/containers-promote/tasks/check_set_ppc_manifests.yml | 14:06 |
| marios|call | https://review.rdoproject.org/r/#/c/24807/12/ci-scripts/dlrnapi_promoter/tests/staging-setup/promoter_test.py | 14:06 |
| chkumar|rover | sshnaidm, I think i broke rhel8 jobs | 14:10 |
| chkumar|rover | sshnaidm, I will fix in the new review, let's not block the upstream jobs? | 14:10 |
| sshnaidm | chkumar|rover, link? | 14:10 |
| sshnaidm | chkumar|rover, yeah, rhel8 is not blocking gate, we can fix later | 14:10 |
| chkumar|rover | sshnaidm, https://logserver.rdoproject.org/54/707054/3/openstack-check/tripleo-ci-rhel-8-scenario001-standalone-rdo/b5f3998/job-output.txt | 14:11 |
| chkumar|rover | sshnaidm, Cannot uninstall requirement virtualenv, not installed | 14:11 |
| *** Goneri has joined #oooq | 14:12 | |
| sshnaidm | chkumar|rover, I think in a new patch you can skip removing virtualenv with yum/dnf, and to use "|| true" with pip | 14:12 |
| sshnaidm | chkumar|rover, if no installed, no need to reinstall, and we'll continue to use RPM venv | 14:13 |
| chkumar|rover | sshnaidm, yes | 14:13 |
| chkumar|rover | too much break and fix for the day | 14:13 |
| chkumar|rover | weshay, if the zuul jobs gives +1, we can ask infra to force merge if ok? | 14:15 |
| chkumar|rover | sshnaidm, rdo fs01 check got stuck at image prepare | 14:18 |
| chkumar|rover | https://review.rdoproject.org/zuul/stream/6b6c650c51e94a178805e4b92c6ec531?logfile=console.log | 14:18 |
| chkumar|rover | jpena, Hello | 14:19 |
| jpena | chkumar|rover: o/ | 14:19 |
| sshnaidm | chkumar|rover, image prepare might be not related | 14:19 |
| chkumar|rover | jpena, Can we create a seperate repo as dep-test repo which contains latest version of pip, vritualenv setuptool, pbr tox | 14:19 |
| chkumar|rover | ? | 14:19 |
| chkumar|rover | which we can consume in ci | 14:19 |
| chkumar|rover | and avoid hacky way to install these items from master? | 14:20 |
| chkumar|rover | jpena, reason to avoid the current pip and venv mess we are dealing currently | 14:20 |
| jpena | chkumar|rover: hm, we don't even rebuild most of those | 14:21 |
| chkumar|rover | it might be useful for infra also not to bake up images with those stuff | 14:21 |
| chkumar|rover | jpena, how about building and serving them? | 14:22 |
| jpena | chkumar|rover: but... do you want to have them updated too? I see the idea, but it's a slippery slope if we decide to maintain it ourselves | 14:22 |
| chkumar|rover | jpena, yes, | 14:23 |
| chkumar|rover | jpena, may be pulling from fedora rawhide and rebuilding them | 14:23 |
| chkumar|rover | what we have done for py2 to py3 migration just a thought | 14:23 |
| chkumar|rover | jpena, it would be useful for c8 | 14:24 |
| jpena | chkumar|rover: that was a real pain to do, that's why I would prefer to fix it in a different way | 14:24 |
| chkumar|rover | jpena, what is the other way around? | 14:25 |
| jpena | chkumar|rover: for the current issue, I see two short term fixes. Either tox releases a new version with an updated six requirement (which is compatible with virtualenv), or we upgrade six in the image right after installing tox | 14:26 |
| chkumar|rover | jpena, https://hackmd.io/Javdw507QbqweXybJmYBWg plan is here | 14:27 |
| jpena | the long term fix is getting pip to properly resolve dependencies | 14:27 |
| chkumar|rover | jpena, sorry tox is not used in tripleo jobs so we downgraded both in toci | 14:27 |
| chkumar|rover | jpena, I was talking from tripleo ci job presepective | 14:28 |
| chkumar|rover | jpena, but for pip, we need to gate that? | 14:28 |
| chkumar|rover | it keeps breaking | 14:28 |
| chkumar|rover | pip-20.0.0 issue few days back | 14:28 |
| *** marios|call is now known as marios | 14:29 | |
| migi | rlandy: so could we have really quick chat ? bluejeans.com/mpryc | 14:29 |
| chkumar|rover | jpena, we are trying to dump down our thoughts as preventive action here https://hackmd.io/Javdw507QbqweXybJmYBWg | 14:30 |
| migi | rlandy: I need to go in few | 14:30 |
| rlandy | joining | 14:30 |
| jpena | chkumar|rover: that's another option, but the updated versions may come from the image itself. We could go all the way and cap the pip/virtualenv/tox versions when building our images, but I'm not sure it will be accepted | 14:30 |
| chkumar|rover | jpena, we tried to upgrading six, using seed, removing packages all does not worked | 14:31 |
| marios | rfolco: i can't see virtualenv version in https://github.com/rdo-infra/ci-config/blob/master/test-requirements.txt or requirements | 14:33 |
| chkumar|rover | weshay, marios sshnaidm do we also need to think about dlrn venv used in promoter? | 14:34 |
| marios | rfolco: perhaps we can add into the commands https://github.com/rdo-infra/ci-config/blob/master/tox.ini#L39 | 14:35 |
| marios | rfolco: do somethign similar to what chkumar|rover does in https://review.opendev.org/#/c/707062/18/install-deps.sh | 14:35 |
| *** TrevorV has joined #oooq | 14:35 | |
| chkumar|rover | zbr, please take over the patch https://review.opendev.org/#/c/707062/, first priority is to clear the gates then work on improvements | 14:36 |
| weshay | panda, you have a few min? | 14:36 |
| chkumar|rover | sshnaidm, weshay any thoughts here https://review.opendev.org/#/c/707062/ | 14:37 |
| weshay | chkumar|rover, zbr sshnaidm I would say patch on top of this... | 14:38 |
| weshay | however anyone wants to fix it.. | 14:38 |
| weshay | if it's turning jobs green that is the priority | 14:38 |
| zbr | weshay: i am checking with infra re virtualenv. | 14:38 |
| weshay | zbr, thank you | 14:38 |
| zbr | there are two approaches: pin-down virtualenv or bump-both virtualenv and six. | 14:39 |
| sshnaidm | chkumar|rover, maybe, but I hope when we'll get to promotion the venv package will be fixed :) | 14:39 |
| zbr | but first one should not touch six, at all. | 14:39 |
| marios | rfolco: there? zbr me and folco discussing where to fix that for ci-config... looking at https://github.com/rdo-infra/ci-config/blob/master/tox.ini#L39 | 14:39 |
| marios | zbr: ideas? can we fix it in the tox.ini pinning virtualenv? | 14:40 |
| zbr | afaik, no. | 14:41 |
| zbr | tox needs a working virtualenv to function, this needs to happen before tox is called. | 14:41 |
| marios | zbr: i got the feeling it needed to be one level up (i.e. everything inside that tox.ini is happening inside the virtualenv already) | 14:42 |
| zbr | marios: most of time, yes. | 14:42 |
| zbr | marios: let me try somethign with ci-config, 5mins... | 14:43 |
| marios | zbr: ok thanks i am really not sure where we'd add that virtualenv pin there (example py36 blocked on https://bugs.launchpad.net/tripleo/+bug/1862724 https://logserver.rdoproject.org/07/24807/11/check/tox-py36/1acbef2/job-output.txt | 14:43 |
| openstack | Launchpad bug 1862724 in tripleo "ERROR:root:ImportError: cannot import name ensure_text" [Critical,In progress] - Assigned to chandan kumar (chkumar246) | 14:43 |
| marios | i think rfolco also has a blocked patch on that | 14:43 |
| rfolco | https://review.rdoproject.org/r/#/c/24762/ | 14:46 |
| rfolco | I'd appreciate any help to fix ci-config marios zbr | 14:46 |
| rfolco | thx | 14:46 |
| chkumar|rover | rfolco, marios https://review.rdoproject.org/r/24898 might be helpful related to tox | 14:47 |
| marios | thanks chkumar|rover | 14:49 |
| chkumar|rover | see ya people tomorrow, have a nice day ahead | 14:50 |
| chkumar|rover | I will be not available in evening on thursday and friday | 14:50 |
| *** chkumar|rover is now known as raukadah | 14:50 | |
| *** ykarel is now known as ykarel|away | 14:51 | |
| weshay | anyone know how I fix this https://0e6b8aeca433e85b429b-46fd243db6dc394538bd0555f339eba5.ssl.cf1.rackcdn.com/707054/3/check/requirements-check/4f7505e/ | 14:55 |
| *** skramaja has quit IRC | 14:56 | |
| rfolco | marios, https://review.rdoproject.org/r/#/c/24898/ is abandoned | 14:57 |
| rfolco | waiting for zbr to come with a solution from his top hat | 14:57 |
| zbr | rfolco: watch https://review.rdoproject.org/r/#/c/24899/ -- initial test was positive, now running with more jobs. | 14:58 |
| zbr | i think that many of you are not aware of the magic tools/test-setup.sh file. | 14:59 |
| marios | rfolco: ack looks like it was abandoned just now and yeah zbr is digging there | 14:59 |
| zbr | which is run by zuul automatically. | 14:59 |
| zbr | that is a very good place to prepare a tox job. | 15:00 |
| marios | zbr: no i was not aware of that test-setup.sh ... it is called by what? | 15:00 |
| marios | 16:59 <@zbr> which is run by zuul automatically. | 15:00 |
| marios | i see... | 15:00 |
| zbr | bindep itself or something near, runs all the time. | 15:00 |
| marios | zbr: is it documented in zuul config /me goes to look | 15:00 |
| rlandy | raukadah: what should tempest_cidr be for standalone? | 15:06 |
| marios | zbr: there is a role for it looks like | 15:09 |
| *** sanjayu_ has quit IRC | 15:09 | |
| raukadah | rlandy, https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/config/environments/ovb-common.yml#L5 | 15:09 |
| zbr | marios: i see that py36 is still using f28 node. | 15:10 |
| marios | zbr: https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/test-setup | 15:10 |
| raukadah | it comes from undercloud network cidr | 15:10 |
| rlandy | raukadah: standalone as well? | 15:10 |
| raukadah | rlandy, it is for standalone | 15:10 |
| rlandy | yes | 15:10 |
| rlandy | ovb-common.yml <- OVB | 15:10 |
| raukadah | rlandy, for ovb it is 10 dot something http://codesearch.openstack.org/?q=tempest_cidr&i=nope&files=&repos= | 15:11 |
| raukadah | http://codesearch.openstack.org/?q=tempest_cidr&i=nope&files=&repos= | 15:11 |
| raukadah | tempest_cidr: '10.0.0.0/24' | 15:11 |
| rlandy | I'm looking at standalone - so I think it's right | 15:11 |
| raukadah | sshnaidm, can you take a look at thsi https://review.opendev.org/#/c/707054/ | 15:18 |
| raukadah | i tried but tox -e requirements complains | 15:18 |
| raukadah | rlandy, are you hitting this https://bugs.launchpad.net/tripleo/+bug/1860810? | 15:20 |
| openstack | Launchpad bug 1860810 in tripleo "os_ansible_tempest fails w/ create_network() got an unexpected keyword argument 'dns_domain'" [High,Triaged] | 15:20 |
| rlandy | raukadah: possibly | 15:21 |
| raukadah | rlandy, needed new version of openstackclient | 15:22 |
| rlandy | raukadah: how come this is not failing in upstream? | 15:23 |
| raukadah | rlandy, no idea | 15:23 |
| raukadah | rlandy, stein is working fine | 15:23 |
| raukadah | rlandy, sorry train is working fine | 15:24 |
| rlandy | raukadah: so stein is working | 15:24 |
| rlandy | osp 16 - which would be train is failing | 15:24 |
| raukadah | rlandy, for stein, we moved back to validate-tempest | 15:24 |
| rlandy | checking version of openstackclient | 15:24 |
| rlandy | python3-openstackclient.noarch 4.0.0-0.20191025160014.aa64eb6.el8ost | 15:25 |
| rlandy | raukadah: ^^ | 15:25 |
| rlandy | that is what we have in 16 | 15:25 |
| raukadah | rlandy, checking in train rhel8 job | 15:26 |
| rlandy | hard to tell the exact error as it just bails out ... | 15:28 |
| rlandy | 2020-02-10 20:15:20.103110 | primary | TASK [os_tempest : Ensure private network exists] ****************************** | 15:28 |
| rlandy | 2020-02-10 20:15:20.126624 | primary | Monday 10 February 2020 20:15:20 -0500 (0:00:00.096) 0:53:33.932 ******* | 15:28 |
| rlandy | 2020-02-11 03:02:29.811912 | RUN END RESULT_TIMED_OUT: [untrusted : opendev.org/openstack/tripleo-ci/playbooks/tripleo-ci/run-v3.yaml@master] | 15:28 |
| dpawlik | chkumar|rover, weshay: Sorry for that, but now I confirm what name is correct: rcn or rcm. The new user is rcm-uploader, not rcn-uploader | 15:30 |
| rlandy | raukadah: looks like the fix merged 4 hours ago | 15:31 |
| rlandy | trying run again | 15:31 |
| raukadah | rlandy, blame dpawlik ^^ | 15:32 |
| weshay | dpawlik, fak if I know.. I didn't create it :) | 15:33 |
| rlandy | raukadah: why would I blame dpawlik on this one? | 15:34 |
| raukadah | rlandy, ah sorry, forgive dpawlik he is migrating our rcm server to new host | 15:35 |
| zbr | rfolco: this is how pabelanger fix it on ZAC https://github.com/ansible-network/windmill-config/pull/599/files | 15:35 |
| dpawlik | :) | 15:38 |
| * dpawlik <I was starting to read 2hours of conversation> | 15:39 | |
| rfolco | zbr, cool | 15:40 |
| rlandy | ansible.noarch 2.8.8-1.el8ae @rhosp-rhel-8.1-ansible | 15:43 |
| rlandy | raukadah: ^^ that should be 2.9, right? | 15:43 |
| raukadah | rlandy, in rhel8, we ship ansibel-2.8 | 15:43 |
| raukadah | in tripleo job we ship 2.9 | 15:44 |
| rlandy | https://github.com/ansible/ansible/pull/66952 | 15:44 |
| rlandy | ^^ went to 2.9 | 15:44 |
| raukadah | rlandy, let me find another card | 15:44 |
| raukadah | rlandy, https://trello.com/c/OHyUirhO/1325-cixlp1862330tripleociproa-rhel-8-jobs-for-train-and-master-got-broken-with-aborting-target-uses-selinux-but-python-bindings-libs | 15:45 |
| rlandy | "The jobs are back in normal mode. we need ansible 2.9 for RHEL-8 jobs in RHUi" | 15:46 |
| rlandy | I agree with thaht | 15:47 |
| raukadah | rlandy, yes above ansible fix, will just highlight about version incompatibilty but we need new osclient which is not possible currently | 15:47 |
| rlandy | we need it in rhos-release as well | 15:47 |
| rlandy | raukadah: so osp-16 is doomed to fail :( | 15:47 |
| * rlandy is unhappy | 15:47 | |
| raukadah | rlandy, talk to weshay he will share his feeling :-) | 15:48 |
| * rlandy asks rel del | 15:48 | |
| *** udesale_ has quit IRC | 15:52 | |
| *** zbr_ has joined #oooq | 15:53 | |
| *** zbr_ has quit IRC | 15:53 | |
| *** udesale_ has joined #oooq | 15:53 | |
| weshay | zbr, hey.. can you assist w/ global requirements w/ me for a minute? | 15:54 |
| weshay | need infra support | 15:54 |
| zbr | asf? | 15:54 |
| raukadah | weshay, can we just use sed in the containers build role itself? | 15:55 |
| raukadah | to particular version | 15:55 |
| raukadah | it's py27 so | 15:55 |
| rlandy | raukadah: weshay: so rel del can updread to ansible 2.9 but it will require a respin. does this justify? see conversation on #rhos-delivery | 15:57 |
| rlandy | upgrade | 15:57 |
| raukadah | rlandy, ah cool, what about osclient? | 15:57 |
| rlandy | if we can get ansible upgraded elsewhere, will save up the respin | 15:57 |
| weshay | zbr, sure.. and thank you in advance | 15:57 |
| rlandy | let me check that version | 15:57 |
| raukadah | rlandy, can we get a bug report also for the same, may be encounter soon in tripleo side | 15:58 |
| weshay | zbr, https://review.opendev.org/#/c/707054/ | 15:58 |
| rlandy | raukadah: I never hot the selinix problem though | 15:58 |
| raukadah | rlandy, as we used ansible python interpreter so | 15:59 |
| rlandy | http://git.app.eng.bos.redhat.com/git/openstack/tripleo-ci-internal-jobs.git/tree/zuul.d/standalone-jobs.yaml#n35 | 15:59 |
| rlandy | apparently so did we | 15:59 |
| rlandy | maybe osp is different? | 16:00 |
| raukadah | rlandy, yes it is different than we test in rhel8 tripleo rhui | 16:00 |
| raukadah | rdo on rhel8 ~= osp | 16:01 |
| rlandy | raukadah: this is like playing those games where they give you two drawings that look the same and you have to spend hours finding the 3 differences | 16:01 |
| raukadah | weshay, ^^ you can help rlandy on this :-) | 16:01 |
| rlandy | but os_tempest requirements | 16:01 |
| rlandy | weshay is too busy | 16:01 |
| weshay | ya.. unblocking container-builds | 16:01 |
| rlandy | will raise this later | 16:01 |
| weshay | need a few | 16:01 |
| raukadah | rlandy, we will see the magic when downstream will use os_tempest | 16:02 |
| rlandy | raukadah: let's talk tomorrow when you are off rucking ... and see what to do here | 16:02 |
| raukadah | hehe | 16:02 |
| rlandy | osp/rdo/rhui/rhos-release | 16:02 |
| raukadah | rlandy, sure | 16:03 |
| *** Trevor_V has joined #oooq | 16:03 | |
| raukadah | rlandy, I will be not around on thursday and friday in your's morning | 16:03 |
| rlandy | weshay (just for your amusement) ... why is this deploy different from all other deploys? because in all other deploy we run with ansible 2.9 but in this deploy 2.8 ... in all other builds ... | 16:04 |
| panda | weshay: raukadah ok to merge https://review.rdoproject.org/r/24676 ? | 16:04 |
| rlandy | raukadah: this has been broken for weeks - no rush | 16:04 |
| raukadah | panda, rebase please | 16:04 |
| raukadah | arxcruz, by the way Bahubali part 3 is coming soon | 16:05 |
| panda | raukadah: mmmhhh ... | 16:06 |
| *** udesale_ has quit IRC | 16:06 | |
| *** bogdando has quit IRC | 16:06 | |
| *** udesale_ has joined #oooq | 16:07 | |
| *** TrevorV has quit IRC | 16:07 | |
| panda | raukadah: rebased the next in line https://review.rdoproject.org/r/24675 | 16:08 |
| raukadah | panda, thanks! | 16:08 |
| *** rf0lc0 has joined #oooq | 16:11 | |
| *** rfolco has quit IRC | 16:13 | |
| *** udesale_ has quit IRC | 16:16 | |
| weshay | zbr, https://opendev.org/openstack/requirements/src/branch/master/global-requirements.txt#L188-L189 | 16:26 |
| *** dtantsur is now known as dtantsur|afk | 16:34 | |
| * marios packing up | 16:40 | |
| raukadah | weshay, want to move stuff to hackmd for next sprint cc rr marios ? | 16:41 |
| raukadah | tomorrow is a good time i think? | 16:41 |
| marios | raukadah: i'll start catching up on that tomorrow | 16:42 |
| marios | raukadah: i think we'll likely have some call(s) too ;) | 16:42 |
| raukadah | marios, if want to start early, I will be happy to sync | 16:42 |
| marios | raukadah: 18:40 * marios packing up | 16:43 |
| raukadah | marios, aye sir, have a nice night ahead :-) | 16:43 |
| weshay | raukadah, still trying fix requirements | 16:43 |
| marios | u 2 raukadah | 16:43 |
| *** marios is now known as marios|out | 16:45 | |
| *** jpena is now known as jpena|brb | 16:46 | |
| raukadah | marios|out, :-) | 16:47 |
| rlandy | rf0lc0: where did the OSP containers cards go on sprint21 board? | 16:48 |
| rf0lc0 | rlandy, weshay pushed it to sprint 23 | 16:48 |
| rlandy | whatever | 16:48 |
| rlandy | I understand they skipped spint22 | 16:48 |
| rlandy | but hwy move them off sprint21? | 16:48 |
| rlandy | why | 16:49 |
| rf0lc0 | rlandy, you are right, we should have moved it at the end of sprint | 16:49 |
| rf0lc0 | rlandy, let me move it back | 16:49 |
| rlandy | rf0lc0: don't worry | 16:49 |
| rlandy | it doesn't matter | 16:49 |
| rf0lc0 | rlandy, we have to add it back, doing it now | 16:50 |
| rf0lc0 | rlandy, moved it back, sorry for the mistake. | 16:51 |
| rlandy | rf0lc0: thank you | 16:52 |
| rlandy | just making one small update | 16:53 |
| raukadah | rlandy, rf0lc0 c8 comes to priority so I think cc weshay | 16:55 |
| rlandy | raukadah: yep - I understand why the task is not in sprint 22 | 16:55 |
| rlandy | but I couldn't find the card to update from this current sprint | 16:55 |
| rlandy | it's no big deal - I just wanted to make a minor update to the card | 16:56 |
| rf0lc0 | rlandy, next sprint is 22, so centos8 is our focus, we'll push the osp work to sprint 23. | 16:56 |
| *** marios|out has quit IRC | 16:56 | |
| rlandy | rf0lc0: yep - no problem ... I just have a meta issue to raise wrt our use of rel del's files | 16:57 |
| rlandy | that's all - adding that to the card for review - then you have remove them | 16:57 |
| rf0lc0 | rlandy, welcome to bring concerns and we may revisit priorities furing planning mtg on thu | 16:57 |
| rf0lc0 | the idea was to move the card at the end of the sprint, now now. I apologize | 16:58 |
| raukadah | rlandy, sorry for the trouble, we can include in sprint 22 based on tasks available | 16:58 |
| weshay | zbr, https://review.opendev.org/#/c/707062/ | 16:58 |
| rlandy | raukadah: rf0lc0: again - no issue on not working on this next sprint ... I just could find the card to update for Thursday review. I'm done now with the card | 16:59 |
| * rlandy moves back to tls | 17:00 | |
| zbr | FYI, if you see a molecule failure, pin it to last know good for that specific job. For the moment we should replace any >= on its versioning with pins. | 17:08 |
| *** jmasud has quit IRC | 17:13 | |
| *** jmasud has joined #oooq | 17:15 | |
| raukadah | zbr, thank you :-) | 17:15 |
| rf0lc0 | zbr, panic: reflect: indirection through nil pointer to embedded struct [recovered] | 17:17 |
| rf0lc0 | panic: reflect: indirection through nil pointer to embedded struct | 17:17 |
| rf0lc0 | zbr, https://review.rdoproject.org/r/#/c/24899/ | 17:17 |
| rf0lc0 | perhaps change that condition to OR ? | 17:18 |
| rf0lc0 | and ansible_distribution_major_version == '7' | 17:18 |
| rf0lc0 | did you mean centos7 should install python3-libselinux ? | 17:18 |
| zbr | rf0lc0: read again. | 17:23 |
| *** ccamacho has joined #oooq | 17:24 | |
| *** jpena|brb is now known as jpena | 17:25 | |
| raukadah | times up now | 17:25 |
| raukadah | rlandy, zbr sshnaidm weshay thanks for all the help | 17:26 |
| raukadah | panda, sorry, will act on patches tomorrow feeling small panda sleepy | 17:26 |
| *** kopecmartin is now known as kopecmartin|afk | 17:27 | |
| panda | raukadah: no worries, there's a broken job right now ... | 17:27 |
| zbr | there is a much better way to avoid virtualenv issues on centos-8, avoiding it: https://review.opendev.org/#/c/707215/2 | 17:28 |
| zbr | python3 comes with venv, which is compatible. | 17:28 |
| raukadah | zbr, sorry did not get time to work on libvirt molecule will tackle next week together | 17:29 |
| zbr | raukadah: yeah, that is the last think i want to hear now. clearly not enough time/emergy to approach that this week. | 17:30 |
| zbr | we can look next week, hopefully with less issues. | 17:30 |
| raukadah | zbr, it is blocking one of the panda;s work | 17:30 |
| rf0lc0 | zbr, I am still lost, https://review.rdoproject.org/r/#/c/24899/ still fails and I can't fix it | 17:31 |
| *** ccamacho has quit IRC | 17:42 | |
| *** derekh has quit IRC | 17:58 | |
| *** jpena is now known as jpena|off | 18:36 | |
| zbr | rf0lc0: my impression is that f28 image is broken, at least tox. | 18:38 |
| zbr | got tired trying to fix it. | 18:38 |
| rf0lc0 | zbr, only linters now | 18:39 |
| rf0lc0 | https://review.rdoproject.org/r/#/c/24903/ | 18:39 |
| rf0lc0 | and my patch would go green after linters get fixed in the other review ... | 18:39 |
| rf0lc0 | https://review.rdoproject.org/r/#/c/24762/ | 18:39 |
| *** holser has quit IRC | 18:42 | |
| zbr | rf0lc0: that line does not seem right https://review.rdoproject.org/r/#/c/24903/1/tools/test-setup.sh | 18:44 |
| zbr | not sure how you managed to make pass. | 18:44 |
| zbr | and i find out why, we have bad filters on that job. | 18:45 |
| zbr | when touching tox.ini it needs to run all affected jobs. | 18:45 |
| zbr | LOL https://pypi.org/project/virtualenv/#history | 18:47 |
| rf0lc0 | meh | 18:50 |
| rf0lc0 | you ruined my happiness | 18:50 |
| *** amoralej is now known as amoralej|off | 18:51 | |
| *** jtomasek has quit IRC | 19:48 | |
| *** Goneri has quit IRC | 19:53 | |
| weshay | zbr, https://review.opendev.org/#/c/707062/ still works :) | 19:56 |
| weshay | thanks | 19:56 |
| weshay | zbr, and I think we got tripleo-common past the requirements check job \0/ | 19:57 |
| *** owalsh has quit IRC | 20:30 | |
| *** jfrancoa has quit IRC | 20:36 | |
| *** owalsh has joined #oooq | 20:44 | |
| rlandy | weshay: do you need help with something ... just working on ipa now | 20:58 |
| weshay | rlandy, think we're just waiting on patches, but thanks | 20:59 |
| *** Goneri has joined #oooq | 21:13 | |
| rlandy | weshay: if you're not super busy then I'll ask the rhel question from before ... | 21:22 |
| rlandy | rel del can move to ansible 2.9 for osp-16 but it would require a respin. is it worth it? | 21:23 |
| weshay | rlandy, what did you find? | 21:27 |
| weshay | I saw you chatting about that | 21:27 |
| weshay | but not clear on the details | 21:27 |
| * rlandy copies | 21:29 | |
| rlandy | weshay: http://pastebin.test.redhat.com/835174 | 21:30 |
| weshay | rlandy, hrm.. so.. the min version of openstacksdk needs to be bumped for os_ansible tempest to work properly | 21:35 |
| weshay | probably would be a blocker if there was one way to run tempest | 21:35 |
| weshay | rlandy, I would say make it a high priority bug but not a blocker.. | 21:35 |
| rlandy | weshay: so how come this is not an issue on rhel train standalone? | 21:36 |
| rlandy | basically this bug ... https://bugs.launchpad.net/tripleo/+bug/1860810 | 21:37 |
| openstack | Launchpad bug 1860810 in tripleo "os_ansible_tempest fails w/ create_network() got an unexpected keyword argument 'dns_domain'" [High,Triaged] | 21:37 |
| *** Trevor_V has quit IRC | 21:37 | |
| rlandy | if we run validate_tempest it may be fine | 21:37 |
| weshay | rlandy, we DO need a prod-chain blocker because that will block the component pipeline | 21:41 |
| weshay | rlandy, 62 gate failures in 24 hours :) | 21:43 |
| rlandy | weshay: your best day on ruck/rover yet!! | 21:43 |
| weshay | it may be a record.. don't know.. we should track it | 21:44 |
| rlandy | it says 59 now | 21:45 |
| rlandy | so you should screenshot your fine achievement | 21:46 |
| weshay | ya | 21:47 |
| *** dsneddon_ has quit IRC | 21:48 | |
| rlandy | reading back - if we are honest, there is no component pipeline now | 21:51 |
| weshay | rlandy, this is about to pass check https://review.opendev.org/#/c/707062 | 21:53 |
| weshay | rlandy, ya.. well the pieces should still work | 21:54 |
| rlandy | weshay: k - you wnat me to w+? | 21:54 |
| weshay | rlandy, cix is just a signal.. | 21:54 |
| weshay | rlandy, +2.. I can wf | 21:54 |
| weshay | or you can | 21:54 |
| weshay | don't care | 21:54 |
| rlandy | done | 21:55 |
| rlandy | weshay: if standalone 16 passes using validate tempest rather than os_tempest, we should be able to keep the component pipeline working in theory | 21:57 |
| weshay | rlandy, ya.. we just want it on the radar for a future build | 21:58 |
| weshay | we'll be focused on centos-8 upstream for a bit | 21:58 |
| rlandy | weshay: ack - understandable | 21:58 |
| *** dsneddon has joined #oooq | 21:59 | |
| weshay | rlandy, if that fails in the gate again.. I'm going to shoot myself | 22:04 |
| rlandy | well then it better pass gate, no? | 22:05 |
| * rlandy check gate progress | 22:05 | |
| rlandy | maybe we can RCA on how not to get caught in this mess again | 22:07 |
| rlandy | there goes gate | 22:07 |
| rlandy | weshay: don't look - I'll you when it's all over | 22:07 |
| weshay | da.. f is this? http://cougar11.scl.lab.tlv.redhat.com/phase1-16_director-rhel-8.1-virthost-1cont_1comp_1ceph-ipv4-geneve-ceph/76/undercloud-0.tar.gz?undercloud-0/home/stack/overcloud_install.log | 22:10 |
| rlandy | 16 just had a respin yesterday | 22:11 |
| weshay | rlandy, just respun again today | 22:23 |
| rlandy | yep - I see that | 22:24 |
| *** jmasud has quit IRC | 23:49 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!