| rdogerrit | rdo-trunk created rdoinfo master: Bump rdoinfo rocky tags to latest upper-constraints https://review.rdoproject.org/r/24906 | 00:03 |
|---|---|---|
| rdogerrit | rdo-trunk created rdoinfo master: Bump rdoinfo train tags to latest upper-constraints https://review.rdoproject.org/r/24907 | 00:05 |
| rdogerrit | rdo-trunk created rdoinfo master: Promote CBS tags update for rocky-release https://review.rdoproject.org/r/24908 | 00:05 |
| rdogerrit | rdo-trunk created rdoinfo master: Promote CBS tags update for stein-release https://review.rdoproject.org/r/24909 | 00:08 |
| *** rcernin has joined #rdo | 00:08 | |
| rdogerrit | rdo-trunk proposed rdoinfo master: Bump rdoinfo ussuri-uc tags to latest upper-constraints https://review.rdoproject.org/r/24796 | 00:09 |
| *** artom has quit IRC | 00:10 | |
| rdogerrit | rdo-trunk created rdoinfo master: Promote CBS tags update for train-release https://review.rdoproject.org/r/24910 | 00:11 |
| *** slaweq has joined #rdo | 00:11 | |
| *** artom has joined #rdo | 00:14 | |
| *** artom has quit IRC | 00:15 | |
| *** artom has joined #rdo | 00:15 | |
| *** slaweq has quit IRC | 00:16 | |
| *** tosky has quit IRC | 00:38 | |
| *** rf0lc0 has quit IRC | 00:50 | |
| *** rcernin has quit IRC | 00:55 | |
| *** rcernin has joined #rdo | 00:55 | |
| *** rlandy has quit IRC | 01:17 | |
| *** stevebaker has quit IRC | 01:24 | |
| *** stevebaker has joined #rdo | 01:26 | |
| *** avivgt has quit IRC | 01:30 | |
| rdogerrit | Emilien Macchi created rdoinfo master: Bump puppet-kmod to 2.4.0 https://review.rdoproject.org/r/24911 | 01:47 |
| EmilienM | jpena|off: ^ any thoughts on | 01:49 |
| EmilienM | ? | 01:49 |
| EmilienM | see BZ | 01:49 |
| *** mrunge has quit IRC | 02:53 | |
| *** rcernin has quit IRC | 03:06 | |
| *** rcernin has joined #rdo | 03:23 | |
| *** udesale has joined #rdo | 04:05 | |
| *** ramishra has joined #rdo | 04:09 | |
| *** rcernin has quit IRC | 04:16 | |
| *** rcernin has joined #rdo | 04:16 | |
| *** ramishra has quit IRC | 04:32 | |
| *** artom has quit IRC | 04:35 | |
| *** ramishra has joined #rdo | 04:46 | |
| *** ykarel|away is now known as ykarel | 04:58 | |
| *** rcernin is now known as rcernin|lunch | 05:01 | |
| *** skramaja has joined #rdo | 05:18 | |
| *** mmethot_ has joined #rdo | 05:35 | |
| *** mmethot has quit IRC | 05:37 | |
| *** marios has joined #rdo | 05:59 | |
| *** bandini has joined #rdo | 06:00 | |
| *** dsneddon has quit IRC | 06:02 | |
| *** mrunge has joined #rdo | 06:15 | |
| *** aufi has joined #rdo | 06:26 | |
| *** lmiccini has joined #rdo | 06:36 | |
| rdogerrit | Yatin Karel created rdoinfo master: Pin heat-tempest-plugin to good hash https://review.rdoproject.org/r/24912 | 06:36 |
| *** jfrancoa has joined #rdo | 06:51 | |
| *** jtomasek has joined #rdo | 06:51 | |
| *** dsneddon has joined #rdo | 07:06 | |
| *** slaweq has joined #rdo | 07:08 | |
| *** dsneddon has quit IRC | 07:11 | |
| *** slaweq has quit IRC | 07:15 | |
| *** rpittau|afk is now known as rpittau | 07:21 | |
| *** yoctozepto has quit IRC | 07:28 | |
| *** raukadah is now known as chkumar|rover | 07:28 | |
| *** yoctozepto has joined #rdo | 07:29 | |
| *** stevebaker has quit IRC | 07:31 | |
| *** holser has joined #rdo | 07:31 | |
| *** yoctozepto has quit IRC | 07:33 | |
| *** iurygregory has joined #rdo | 07:35 | |
| *** tmazur has joined #rdo | 07:40 | |
| *** matbu has quit IRC | 07:43 | |
| *** kopecmartin|afk is now known as kopecmartin | 07:44 | |
| *** matbu has joined #rdo | 07:46 | |
| *** slaweq has joined #rdo | 07:51 | |
| *** maciejjozefczyk has joined #rdo | 07:54 | |
| *** ykarel is now known as ykarel|lunch | 07:55 | |
| *** slaweq has quit IRC | 07:57 | |
| *** slaweq has joined #rdo | 07:59 | |
| *** yoctozepto has joined #rdo | 07:59 | |
| *** jcapitao has joined #rdo | 08:06 | |
| *** xek_ has joined #rdo | 08:09 | |
| *** tkajinam has quit IRC | 08:10 | |
| *** tosky has joined #rdo | 08:22 | |
| *** amoralej|off is now known as amoralej | 08:26 | |
| *** ccamacho has joined #rdo | 08:27 | |
| *** apevec has joined #rdo | 08:31 | |
| *** jlibosva has joined #rdo | 08:32 | |
| *** ccamposr has joined #rdo | 08:35 | |
| rdogerrit | User marios proposed rdo-infra/ci-config master: Add ppc_manifests check and switch to manifest_push to exclude ppc https://review.rdoproject.org/r/24807 | 08:35 |
| *** ralonsoh has joined #rdo | 08:38 | |
| *** dtantsur|afk is now known as dtantsur | 08:45 | |
| *** migarcia|away is now known as migarcia | 08:48 | |
| zbr | anyone around? i think that our rdo images may have broken virtualenv on them | 08:48 |
| *** jpena|off is now known as jpena | 08:50 | |
| amoralej | zbr, where? | 08:53 |
| amoralej | we fixed an issue with venv and centos7 yesterday | 08:53 |
| amoralej | well, not we, but upstream fix in nodepool elements | 08:53 |
| amoralej | ykarel|lunch, fyi https://review.opendev.org/#/c/707338 | 08:53 |
| amoralej | gfidente|afk, ^ | 08:54 |
| *** paramite has joined #rdo | 08:55 | |
| zbr | amoralej: can you check version of virtualenv on rdo-centos-8 image? even better if you can teach me how to do these myself. | 08:57 |
| *** jpich has joined #rdo | 08:57 | |
| *** gfidente|afk is now known as gfidente | 08:58 | |
| zbr | the upstream ensure-tox from upstream, which we use had a pretty bad support for our platforms and it will take days to fix. | 08:58 |
| rdogerrit | User marios proposed rdo-infra/ci-config master: Add ppc_manifests check and switch to manifest_push to exclude ppc https://review.rdoproject.org/r/24807 | 08:58 |
| zbr | mainly we need to be sure we do keep 'virtualenv<20', as this is breaking us. something is installing newer version on the system. | 08:59 |
| amoralej | zbr, in https://logserver.rdoproject.org/96/24796/8/check/rdoinfo-DLRN-check-centos8/8d9e0e9/job-output.txt | 08:59 |
| amoralej | i see | 08:59 |
| amoralej | 2020-02-12 00:14:14.938513 | centos8 | "msg": "Could not get output from /usr/local/bin/virtualenv --help: ERROR:root:ImportError: cannot import name 'ensure_text'\n" | 08:59 |
| amoralej | that's the issue you are seeing? | 08:59 |
| amoralej | in TASK [bindep : install bindep into temporary venv] | 09:00 |
| tosky | amoralej: I've noticed that as well and I was going to ask | 09:00 |
| tosky | whether it is a (still) known issue | 09:00 |
| zbr | yep, mainly newer virtualenv needs six>1.12 but we have much older ones o all ⅞ platforms | 09:00 |
| tosky | (namely https://review.rdoproject.org/r/#/c/24290/) | 09:00 |
| jpena | I think https://review.opendev.org/707199 has broken us | 09:00 |
| jpena | I'm checking the generated centos8 image from nodepool, and it's failing on that element | 09:01 |
| jpena | " venv: error: unrecognized arguments: --seeder=pip" | 09:01 |
| amoralej | jpena, that was supposed to fix it | 09:01 |
| amoralej | in centos7 | 09:01 |
| zbr | getting new rpm for six would be ideal, but i am pessimistic about ability to deliver it today | 09:01 |
| amoralej | well, in fact it fixed centos7 | 09:01 |
| jpena | I'm checking it on centos8, we have virtualenv 15.1 from package, and it does not recognize that option | 09:01 |
| amoralej | ouh | 09:01 |
| jpena | zbr: python-six is part of Base OS, that's an inconvenience | 09:02 |
| amoralej | so, in images in SF we install it from package instead of source | 09:02 |
| zbr | amoralej: seeder=pip resolves OTHER issues with 20.x versions, | 09:02 |
| amoralej | yeap | 09:02 |
| zbr | FYI, six has no external deps and is VERY safe to upgrade. | 09:02 |
| jpena | amoralej: yes. IIRC, installing from source was causing other problems. We could try changing that, though, and see what happens | 09:02 |
| zbr | probably is the most stable python package, for a reason. | 09:02 |
| jpena | zbr: sure, I'm just saying it's a bit out of our control | 09:03 |
| amoralej | jpena, for consistency with upstream, moving to source would be good | 09:03 |
| amoralej | unless there are other issues | 09:03 |
| jpena | I'll give it a try, let's see | 09:04 |
| zbr | i am afraid that there are lots of doors a newer virtualenv could step-in. | 09:04 |
| zbr | i am afraid that installing for source may cause some problems, unless we first install rpm and after this we install from source. | 09:04 |
| zbr | lets try six from source first, i think that is likely we will see a newer virtualenv soon that fixes it, they already got 3 releases this week. | 09:06 |
| *** dsneddon has joined #rdo | 09:07 | |
| jpena | zbr: we're trying six from source already (see https://softwarefactory-project.io/r/17510) | 09:07 |
| jpena | the problem is that, since we have virtualenv 15.1 from package, using virtualenv --seeder=pip does not work, and breaks the image creation | 09:08 |
| zbr | (after this) can't we trigger a job running image build when we modify the elements? so we can see the outcome directly on the change proposing it? | 09:10 |
| jpena | oh, https://opendev.org/openstack/diskimage-builder/commit/18215274d895b4b6104c5d977c00b623568646b2 | 09:11 |
| jpena | that commit deprecates source install for venv on centos8 | 09:11 |
| amoralej | yeah | 09:11 |
| amoralej | jpena, i just found, it's failing upstream too | 09:11 |
| jpena | sigh | 09:12 |
| amoralej | https://410c0463c0087166dadd-c3ec680ab787b0ae7fbdeb20a072dd12.ssl.cf2.rackcdn.com/707338/1/check/kolla-build-centos-binary/ff2d1d1/job-output.txt | 09:12 |
| amoralej | 2020-02-12 09:02:03.279841 | TASK [Run tox] | 09:12 |
| amoralej | 2020-02-12 09:02:04.646116 | primary | build-centos-binary create: /home/zuul/src/opendev.org/openstack/kolla/.tox/build-centos-binary | 09:12 |
| amoralej | 2020-02-12 09:02:04.813881 | primary | ERROR: invocation failed (exit code 1), logfile: /home/zuul/src/opendev.org/openstack/kolla/.tox/build-centos-binary/log/build-centos-binary-0.log | 09:12 |
| amoralej | 2020-02-12 09:02:04.814218 | primary | ================================== log start =================================== | 09:12 |
| amoralej | 2020-02-12 09:02:04.814262 | primary | ERROR:root:ImportError: cannot import name 'ensure_text' | 09:12 |
| jpena | the only fix I can think of is to revert https://review.opendev.org/707199 | 09:12 |
| jpena | any other ideas? | 09:12 |
| amoralej | that will break centos7 again | 09:12 |
| amoralej | revert should come with virtualenv pin | 09:12 |
| amoralej | < 20 | 09:12 |
| rdogerrit | User marios created rdo-infra/ci-config master: Pin ironicclient under 4.0.0 for py27 ci-promotion-staging job https://review.rdoproject.org/r/24913 | 09:12 |
| marios | zbr: panda: ^ for promotion-staging-job chkumar|rover ^ | 09:13 |
| *** dsneddon has quit IRC | 09:13 | |
| zbr | i am inclined to believe that pin is safer for us. bad part is that upstream they preffered the opposite, but they do not have our baggage. | 09:14 |
| chkumar|rover | marios, does the tox venv breakage got fixed in ci-config? | 09:14 |
| marios | chkumar|rover: yeah i don't know what fixed it though i left a comment on zbr patch about that | 09:14 |
| zbr | chkumar|rover: no. that is what we are talking about, and is not only one repo, is a generalized issue. | 09:14 |
| marios | chkumar|rover: you mean py36 right? (you can see green there https://review.rdoproject.org/r/#/c/24807/ ) | 09:14 |
| chkumar|rover | marios, tox molecule job failing on ppc patch | 09:14 |
| marios | chkumar|rover: is pre-existing *(non voting job) | 09:15 |
| chkumar|rover | marios, yes, ok | 09:15 |
| rdogerrit | Sorin Sbarnea (zbr) proposed rdo-infra/ci-config master: WIP: Prevent virtualenv failure due to older six https://review.rdoproject.org/r/24899 | 09:15 |
| rdogerrit | User marios proposed rdo-infra/ci-config master: Add ppc_manifests check and switch to manifest_push to exclude ppc https://review.rdoproject.org/r/24807 | 09:17 |
| zbr | fixing the virtualenv is not so easy as we have LOTs of images to fix. | 09:17 |
| zbr | and because failure is in pre, is a PITA to wait for it to run 3 times before being able to see the failure logs. | 09:18 |
| arxcruz | hey guys, i'm trying to generate my http password on https://review.rdoproject.org/r/#/settings/http-password and it's not working... is it broken, or disabled ? | 09:18 |
| *** iurygregory has quit IRC | 09:21 | |
| rdogerrit | Sorin Sbarnea (zbr) proposed rdo-infra/ci-config master: WIP: Prevent virtualenv failure due to older six https://review.rdoproject.org/r/24899 | 09:23 |
| jpena | arxcruz: IIRC that password generation from gerrit did not work, since we're using GitHub SSO | 09:26 |
| jpena | what do you need it for? | 09:26 |
| arxcruz | jpena: gertty | 09:26 |
| amoralej | jpena, it seems there is a new version of virtualenv 20.0.2 which fixed it | 09:27 |
| amoralej | and seeds defaults is different | 09:27 |
| amoralej | so we may safely revert that | 09:27 |
| amoralej | yeap, i just checked it | 09:28 |
| rdogerrit | Sorin Sbarnea (zbr) proposed rdo-infra/ci-config master: WIP: Prevent virtualenv failure due to older six https://review.rdoproject.org/r/24899 | 09:30 |
| panda | arxcruz: use https://review.rdoproject.org/sf/user_settings.html | 09:34 |
| arxcruz | panda: the api key? | 09:34 |
| panda | arxcruz: yes, the http password setting is overridden by that, since gerrit is embedded in sf | 09:35 |
| rdogerrit | User marios proposed rdo-infra/ci-config master: Pin ironicclient under 4.0.0 for py27 ci-promotion-staging job https://review.rdoproject.org/r/24913 | 09:37 |
| rdogerrit | User marios proposed rdo-infra/ci-config master: Add ppc_manifests check and switch to manifest_push to exclude ppc https://review.rdoproject.org/r/24807 | 09:37 |
| *** iurygregory has joined #rdo | 09:39 | |
| amoralej | ykarel|lunch, fyi https://review.opendev.org/#/c/707346/ if you have a chance to double check it works in centos7 | 09:40 |
| amoralej | zbr, ^ | 09:40 |
| *** ykarel|lunch is now known as ykarel | 09:41 | |
| ykarel | amoralej, ack hmm i doubted that change where source packages is not used | 09:41 |
| amoralej | broke centos8 | 09:42 |
| ykarel | amoralej, last night i checked 20.0.2 was working fine | 09:42 |
| amoralej | good | 09:43 |
| *** dsneddon has joined #rdo | 09:45 | |
| *** rakhmerov has joined #rdo | 09:47 | |
| *** alexmcleod has joined #rdo | 09:51 | |
| zbr | ykarel: newer virtualenv works fine *if* you have six>1.12, which is currently possible with a "supported centos" | 09:54 |
| ykarel | zbr, yes upgraded six is needed | 09:54 |
| ykarel | for newer virtualenv | 09:55 |
| zbr | and that is needed everywhere, on any image, is a PITA | 09:56 |
| zbr | amoralej: ykarel please help me narrow down what is wrong with "python3 -m tox --version" on rdo-fedora-28 | 10:00 |
| zbr | https://logserver.rdoproject.org/99/24899/12/check/tox-py36/f5f88e0/job-output.txt | 10:00 |
| zbr | i tried to enforce a reinstall of tox, tested locally in isolated containers (same os) works file, | 10:00 |
| arxcruz | panda: thanks | 10:00 |
| zbr | but on zuul: /usr/bin/python3: No module named tox.__main__; 'tox' is a package and cannot be directly executed | 10:01 |
| zbr | this kind of error appears if you have an ancient version of tox (like 1.4.x) installed somewhere in python path. | 10:01 |
| zbr | ahh... I think I know, probably there is one in current user folder. | 10:02 |
| ykarel | zbr, hmm possible it's installed, good to check from what path it's coming | 10:03 |
| *** skramaja has quit IRC | 10:03 | |
| rdogerrit | Sorin Sbarnea (zbr) proposed rdo-infra/ci-config master: WIP: Prevent virtualenv failure due to older six https://review.rdoproject.org/r/24899 | 10:04 |
| zbr | sadldy it does not report path, but i will find-out in minutes | 10:05 |
| ykarel | zbr, import tox, print(tox.__file__)? | 10:06 |
| zbr | and most of this mess comes from the fact that ensure-tox made some questionable decisions, and was not tested on centos. we need to put pressure to merge that change upstream ASAP, so we have jobs on that role | 10:06 |
| zbr | i will do that if my hack does not work. | 10:07 |
| ykarel | python3 -c 'import tox; print(tox.__file__)' | 10:07 |
| *** derekh has joined #rdo | 10:13 | |
| rdogerrit | Merged rdo-infra/ci-config master: Changed log server address and user https://review.rdoproject.org/r/24811 | 10:20 |
| rdogerrit | Sorin Sbarnea (zbr) proposed rdo-infra/ci-config master: WIP: Prevent virtualenv failure due to older six https://review.rdoproject.org/r/24899 | 10:21 |
| ykarel | jpena, amoralej any idea how to run ansible under zuul executor with verbose mode like with -vvvvv | 10:22 |
| *** rcernin|lunch has quit IRC | 10:22 | |
| ykarel | actually i am seeing a failure https://9c3f698558f8c14a5231-c36d4b8e22685faa8d8d3b6482fd29b5.ssl.cf2.rackcdn.com/707330/1/check/tripleo-build-containers-centos-7-rocky/683e070/job-output.txt | 10:23 |
| ykarel | and the error there looks not actual error to me | 10:24 |
| ykarel | so i was thinking of running with verbose mode to have more info | 10:25 |
| ykarel | me looking zuul code, but doesn't look there is some direct way | 10:27 |
| jpena | ykarel: I'm not aware of a way to do that. You can get some more information from the zuul executor logs, but you need to have access to it | 10:31 |
| ykarel | jpena, ack so that's upstream job, so i can try running in software factory and see if i get more info | 10:32 |
| jpena | ykarel: yep, we could help with that | 10:32 |
| rdogerrit | Yatin Karel created rdo-jobs master: [DNM] debug rocky u-c install venv https://review.rdoproject.org/r/24914 | 10:35 |
| ykarel | jpena, trying with ^^ | 10:35 |
| rdogerrit | Chandan Kumar created config master: Added use_image_server var to use new vexxhost image server https://review.rdoproject.org/r/24915 | 10:36 |
| *** rcernin|lunch has joined #rdo | 10:36 | |
| rdogerrit | Merged rdo-infra/rdo-infra-playbooks master: Changed ssh keys related to logs.rdoproject.org https://review.rdoproject.org/r/24810 | 10:37 |
| *** sshnaidm_ has joined #rdo | 10:49 | |
| *** sshnaidm has quit IRC | 10:49 | |
| *** sshnaidm_ is now known as sshnaidm | 10:49 | |
| *** matbu has quit IRC | 11:04 | |
| rdogerrit | User marios proposed rdo-infra/ci-config master: Add ppc_manifests check and switch to manifest_push to exclude ppc https://review.rdoproject.org/r/24807 | 11:05 |
| *** udesale has quit IRC | 11:06 | |
| *** morazi has quit IRC | 11:10 | |
| rdogerrit | Chandan Kumar proposed config master: Added use_image_server var to use new vexxhost image server https://review.rdoproject.org/r/24915 | 11:15 |
| *** jcapitao is now known as jcapitao_afk | 11:20 | |
| rdogerrit | Chandan Kumar proposed config master: Added use_image_server var to use new vexxhost image server https://review.rdoproject.org/r/24915 | 11:29 |
| rdogerrit | User marios proposed rdo-infra/ci-config master: Add ppc_manifests check and switch to manifest_push to exclude ppc https://review.rdoproject.org/r/24807 | 11:31 |
| zbr | amoralej: ykarel : what fo you think about failure at https://logserver.rdoproject.org/99/24899/14/check/tox-py36/cc28f1a/job-output.txt | 11:31 |
| zbr | makes no sense to me, pip failing doing a "freeze" because lack of sudo.... | 11:32 |
| zbr | afaik freeze was listing installed packages. | 11:33 |
| *** rpittau is now known as rpittau|bbl | 11:33 | |
| amoralej | zbr where that test-setup.sh is coming from? | 11:35 |
| amoralej | my understanding was that we don't want to install virtualenv from pip but use the packaged one | 11:35 |
| zbr | from that change: https://review.rdoproject.org/r/#/c/24899/ | 11:35 |
| zbr | so, we should not touch virtualenv using pip. | 11:36 |
| rdogerrit | Chandan Kumar proposed rdo-infra/ci-config master: Modularize Promoter: creating execution path split https://review.rdoproject.org/r/24676 | 11:36 |
| zbr | i will also use pip list instead of freeze, i was not aware that freeze may need sudo. | 11:36 |
| zbr | in my ignorance freeze was like list but with better formatting. | 11:37 |
| rdogerrit | Sorin Sbarnea (zbr) proposed rdo-infra/ci-config master: WIP: Prevent virtualenv failure due to older six https://review.rdoproject.org/r/24899 | 11:38 |
| rdogerrit | Sorin Sbarnea (zbr) proposed rdo-infra/ci-config master: Switch nodesets to centos-8 https://review.rdoproject.org/r/24905 | 11:41 |
| zbr | amoralej: jpena: can we please get more rdo-centos-8 min-nodes on rdo? I see too often waiting for them. | 11:46 |
| *** skramaja has joined #rdo | 11:46 | |
| *** raildo has joined #rdo | 11:47 | |
| zbr | in fact this is rasign another question: cloud-centos-8 vs rdo-centos-8 vs upstream-centos-8 ... which one to use for tox jobs. | 11:47 |
| zbr | the nodeset file does not have any line of documentation regarding when to use or avoid a specific nodeset. | 11:48 |
| jpena | zbr: https://softwarefactory-project.io/r/17524 | 11:48 |
| jpena | about the different nodes, we're planning a cleanup | 11:49 |
| zbr | jpena: thanks. a cleanup would very useful, even few comments mentioning which one are a safe bet would be of great use. | 11:50 |
| zbr | i am trying to get rid of the fedora-28 node use now. | 11:50 |
| rdogerrit | Merged rdo-infra/ci-config master: Pin ironicclient under 4.0.0 for py27 ci-promotion-staging job https://review.rdoproject.org/r/24913 | 11:51 |
| *** marios is now known as marios|food | 11:52 | |
| rdogerrit | Sorin Sbarnea (zbr) proposed rdo-infra/ci-config master: WIP: Prevent virtualenv failure due to older six https://review.rdoproject.org/r/24899 | 11:54 |
| rdogerrit | Sorin Sbarnea (zbr) proposed rdo-infra/ci-config master: WIP: Prevent virtualenv failure due to older six https://review.rdoproject.org/r/24899 | 11:55 |
| *** dsneddon has quit IRC | 11:57 | |
| zbr | amoralej: i think that the rdo-centos-8 image is broken because it fails in bindep | 12:00 |
| zbr | "Could not get output from /usr/local/bin/virtualenv --help: ERROR:root:ImportError: cannot import name 'ensure_text'\n" | 12:01 |
| *** jaosorior has joined #rdo | 12:01 | |
| amoralej | zbr, i think that's the issue with virtualenv that https://review.opendev.org/#/c/707346/ may fix | 12:01 |
| amoralej | although, let me double check | 12:02 |
| zbr | i have to defer, you will still have a broken virtualenv executable on disk | 12:02 |
| zbr | you may only trick bindep to avoid the failure, but not fixing the broken virtualenv. | 12:02 |
| zbr | I think that "/usr/local/bin/virtualenv --help" should pass, at least if the file exists. | 12:03 |
| zbr | afaik on new os, is called virtualenv-3 - the one ^ being the one from python2, which may or may not be installed. | 12:04 |
| *** rfolco has joined #rdo | 12:10 | |
| *** sshnaidm has quit IRC | 12:20 | |
| *** marios|food is now known as marios | 12:22 | |
| rdogerrit | Chandan Kumar created rdo-jobs master: [DNM] standalone venv rhel8 testing https://review.rdoproject.org/r/24917 | 12:23 |
| *** jpich has quit IRC | 12:24 | |
| *** jpich has joined #rdo | 12:25 | |
| *** rfolco has quit IRC | 12:26 | |
| *** rfolco has joined #rdo | 12:26 | |
| *** sshnaidm has joined #rdo | 12:27 | |
| *** sshnaidm has quit IRC | 12:31 | |
| *** udesale has joined #rdo | 12:34 | |
| *** morazi has joined #rdo | 12:39 | |
| *** sshnaidm has joined #rdo | 12:41 | |
| amoralej | zbr, i see what's the problem with those | 12:48 |
| amoralej | we have two virtualenv installations | 12:48 |
| *** jpena is now known as jpena|lunch | 12:49 | |
| amoralej | rpm and pip from tox | 12:49 |
| amoralej | the tox one has wrong version | 12:49 |
| amoralej | 20.0.1 | 12:49 |
| amoralej | and latest attempts to create new images which would bring 20.0.2 are failing | 12:49 |
| amoralej | jpena|lunch, ^ | 12:49 |
| zbr | amoralej: jpena|lunch : please fix the problem, https://review.rdoproject.org/r/#/c/24916/ should show GREEN on all. | 12:50 |
| zbr | that is an empty tox | 12:50 |
| zbr | passes only on two nodesets. | 12:50 |
| rdogerrit | User marios proposed rdo-infra/ci-config master: Add ppc_manifests check and switch to manifest_push to exclude ppc https://review.rdoproject.org/r/24807 | 12:50 |
| amoralej | ah, the failure in recreating 2020-02-12 11:58:17.498 | venv: error: unrecognized arguments: --seeder=pip | 12:50 |
| *** ekultails has joined #rdo | 12:51 | |
| amoralej | so https://review.opendev.org/#/c/707346/ should help | 12:51 |
| amoralej | by recreating a new image with 20.0.2 | 12:51 |
| amoralej | jpena|lunch, any way we could apply https://review.opendev.org/#/c/707346/ in sf nodepool? | 12:52 |
| rdogerrit | Tristan de Cacqueray created rdo-jobs master: podman: disable podman-package-rhel-8 job https://review.rdoproject.org/r/24918 | 12:53 |
| *** nweinber has joined #rdo | 12:54 | |
| *** amoralej is now known as amoralej|lunch | 12:56 | |
| *** rlandy has joined #rdo | 12:57 | |
| *** marios is now known as marios|call | 12:59 | |
| *** sshnaidm has quit IRC | 13:02 | |
| *** rpittau|bbl is now known as rpittau | 13:05 | |
| *** leanderthal has joined #rdo | 13:06 | |
| *** jcapitao_afk is now known as jcapitao | 13:10 | |
| *** ykarel is now known as ykarel|afk | 13:14 | |
| *** TrevorV has joined #rdo | 13:16 | |
| *** marios|call is now known as marios | 13:16 | |
| *** sshnaidm has joined #rdo | 13:17 | |
| *** sshnaidm has quit IRC | 13:18 | |
| *** sshnaidm has joined #rdo | 13:18 | |
| zbr | amoralej|lunch: jpena|lunch: please check https://review.rdoproject.org/r/#/c/24916/ | 13:21 |
| zbr | i also created https://bugs.launchpad.net/tripleo/+bug/1862941 to track this tox issue on rdo nodes. | 13:25 |
| openstack | Launchpad bug 1862941 in tripleo "rdo tox jobs broken due to cannot import name ensure_text" [Critical,Triaged] | 13:25 |
| zbr | in fact is not tox issue, is more of virtualenv issue. | 13:25 |
| *** sshnaidm has quit IRC | 13:28 | |
| *** sshnaidm has joined #rdo | 13:29 | |
| *** jpich has quit IRC | 13:29 | |
| *** jpich has joined #rdo | 13:29 | |
| *** marios is now known as marios|call | 13:30 | |
| *** derekh has quit IRC | 13:30 | |
| *** jpena|lunch is now known as jpena | 13:33 | |
| *** rh-jelabarre has joined #rdo | 13:39 | |
| *** jpich has quit IRC | 13:42 | |
| *** jpich has joined #rdo | 13:42 | |
| *** rcernin|lunch has quit IRC | 13:50 | |
| *** dsneddon has joined #rdo | 13:54 | |
| *** eharney has quit IRC | 13:55 | |
| *** amoralej|lunch is now known as amoralej | 13:58 | |
| *** dsneddon has quit IRC | 13:58 | |
| jcapitao | it's meeting time | 14:00 |
| jcapitao | #startmeeting RDO meeting - 2020-02-12 | 14:00 |
| openstack | Meeting started Wed Feb 12 14:00:51 2020 UTC and is due to finish in 60 minutes. The chair is jcapitao. Information about MeetBot at http://wiki.debian.org/MeetBot. | 14:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 14:00 |
| *** openstack changes topic to " (Meeting topic: RDO meeting - 2020-02-12)" | 14:00 | |
| openstack | The meeting name has been set to 'rdo_meeting___2020_02_12' | 14:00 |
| jcapitao | #topic roll call | 14:01 |
| *** openstack changes topic to "roll call (Meeting topic: RDO meeting - 2020-02-12)" | 14:01 | |
| amoralej | o/ | 14:01 |
| baha | o/ | 14:01 |
| jcapitao | #chair amoralej baha | 14:01 |
| openstack | Current chairs: amoralej baha jcapitao | 14:01 |
| jcapitao | please add any last topics to https://etherpad.openstack.org/p/RDO-Meeting | 14:01 |
| *** lbragstad has quit IRC | 14:02 | |
| mjturek | o/ | 14:02 |
| jcapitao | #chair mjturek | 14:02 |
| openstack | Current chairs: amoralej baha jcapitao mjturek | 14:02 |
| leanderthal | o/ | 14:03 |
| jcapitao | #chair leanderthal | 14:03 |
| openstack | Current chairs: amoralej baha jcapitao leanderthal mjturek | 14:03 |
| jcapitao | ok let's start with first topic | 14:05 |
| jcapitao | #topic openstack-train repositories for altarch | 14:05 |
| *** openstack changes topic to "openstack-train repositories for altarch (Meeting topic: RDO meeting - 2020-02-12)" | 14:05 | |
| *** derekh has joined #rdo | 14:05 | |
| jcapitao | #link http://mirror.centos.org/altarch/7/cloud/ppc64le/ | 14:06 |
| *** ykarel|afk is now known as ykarel | 14:06 | |
| ykarel | o/ | 14:06 |
| jcapitao | #info Currently, the openstack-train repository is not available for altarch, despite the centos-release package being available to 'enable' it | 14:06 |
| jcapitao | #chair ykarel | 14:06 |
| openstack | Current chairs: amoralej baha jcapitao leanderthal mjturek ykarel | 14:06 |
| baha | Right! So this one's pretty quick - we've been looking into image builds for RDO on ppc, and along the way, noticed that no altarch platforms seem to have the openstack-train repository enabled. We're wondering if anyone knows of an ETA. | 14:07 |
| rdogerrit | Ronelle Landy proposed rdo-jobs master: DNM: Testing standalone-multinode-ipa https://review.rdoproject.org/r/24821 | 14:07 |
| amoralej | baha, it's reported in https://bugs.centos.org/view.php?id=16590 | 14:07 |
| amoralej | unfortunately not fixed yet | 14:08 |
| amoralej | i tried to get kb attention in tha past with no luch | 14:08 |
| mjturek | amoralej: would it be helpful for us to reach out to him? | 14:08 |
| baha | Thank you for the bug number! | 14:09 |
| amoralej | mjturek, yes, i think it may help | 14:09 |
| baha | Alright, we'll do that then | 14:09 |
| mjturek | will do then! | 14:09 |
| baha | Thanks amoralej! | 14:09 |
| jcapitao | #info bug tracked on CentOS bug tracker | 14:10 |
| jcapitao | #link https://bugs.centos.org/view.php?id=16590 | 14:10 |
| jcapitao | good | 14:10 |
| jcapitao | next topic | 14:10 |
| jcapitao | #topic RDO registry problems for ppc64le | 14:10 |
| *** openstack changes topic to "RDO registry problems for ppc64le (Meeting topic: RDO meeting - 2020-02-12)" | 14:10 | |
| jcapitao | #info Currently, the centos-base image is broken in the RDO registry for ppc64le | 14:11 |
| mjturek | yeah jpena has been helping here | 14:11 |
| baha | This one got broken up into a bunch of lines on the etherpad, but the gist is that we've been talking to jpena about it and it looks like one of the blobs may be corrupted or just not on the RDO registry filesystem | 14:12 |
| baha | We're wondering if there's a way to delete the container so we can re-upload, or if anyone else has ideas | 14:12 |
| mjturek | jpena: baha and I were discussing yesterday, is the theory that it as simple as deleting the problematic container? or does more need to be purged? ^ | 14:13 |
| jpena | mjturek: I saw multiple containers use the problematic blob, so one option is to remove them all and try again | 14:14 |
| mjturek | jpena will you be able to take that action or do we need to involve others? | 14:15 |
| baha | Our job is pretty stable at this point so we should be fine to do that if it's what's needed, FWIW | 14:15 |
| jpena | well, I'm not familiar with the operation. I could try, but if there's anyone with more knowledge... | 14:16 |
| jpena | I guess I'll give it a go | 14:18 |
| jpena | :) | 14:18 |
| baha | Thank you jpena =) | 14:18 |
| mjturek | thanks jpena if it doesn't go well, we'll start thinking of who else could help with it | 14:19 |
| ykarel | btw is it known what made it corrupted? | 14:19 |
| ykarel | or is it corrupted from beginning | 14:19 |
| mjturek | ykarel no idea | 14:19 |
| ykarel | ohhk, if it was good in beginning, and corrupted recently, good to find what caused that, so it can be avoided | 14:20 |
| jcapitao | #action jpena to remove containers using the problematic blob | 14:20 |
| mjturek | agreed! | 14:20 |
| *** tesseract has quit IRC | 14:20 | |
| jcapitao | ykarel: +1 | 14:20 |
| baha | We were having a lot of timeout issues with uploads in the beginning so my money would be on something going wrong there, but we will see | 14:21 |
| mjturek | baha: could be! those have been resolved luckily | 14:22 |
| *** jdandrea has joined #rdo | 14:23 | |
| jcapitao | jpena: i could give a try if you're out of time | 14:24 |
| *** mcornea has joined #rdo | 14:28 | |
| jcapitao | let's move on | 14:29 |
| jcapitao | #topic Chair for next meeting | 14:29 |
| *** openstack changes topic to "Chair for next meeting (Meeting topic: RDO meeting - 2020-02-12)" | 14:29 | |
| amoralej | i can take it | 14:29 |
| jcapitao | thank you amoralej | 14:29 |
| jcapitao | #action amoralej to chair next meeting | 14:30 |
| jcapitao | #topic open floor | 14:30 |
| *** openstack changes topic to "open floor (Meeting topic: RDO meeting - 2020-02-12)" | 14:30 | |
| jcapitao | now, it's time to bring other topics you want to discuss | 14:31 |
| leanderthal | next week is docs days | 14:31 |
| leanderthal | i'd LOVE it if you could take a minute to swipe at the documentation | 14:31 |
| *** jaosorior has quit IRC | 14:32 | |
| leanderthal | #link http://rdoproject.org/events/docdays/ | 14:32 |
| rdogerrit | Ronelle Landy proposed rdo-jobs master: DNM: Testing standalone-multinode-ipa https://review.rdoproject.org/r/24821 | 14:32 |
| *** jaosorior has joined #rdo | 14:32 | |
| *** marios|call is now known as marios | 14:33 | |
| jcapitao | #link https://lists.rdoproject.org/pipermail/newsletter/2020-February/000068.html | 14:34 |
| jcapitao | I will check it out for sure | 14:35 |
| ykarel | yeah good, let's have some cleanup | 14:36 |
| rdogerrit | Joel Capitao proposed config master: Add job to reqcheck upstream project https://review.rdoproject.org/r/24399 | 14:36 |
| leanderthal | thx so much jcapitao and ykarel !! ! | 14:36 |
| *** eharney has joined #rdo | 14:41 | |
| jcapitao | nothing else to add ? | 14:42 |
| amoralej | not from my side | 14:43 |
| jcapitao | ok, let's get those 15 min back | 14:43 |
| jcapitao | #endmeeting | 14:44 |
| *** openstack changes topic to "RDO: An OpenStack Distribution for CentOS/RHEL | Community meeting is held here every Wednesday at 15:00UTC | Please read our IRC community guidelines: https://www.rdoproject.org/community/irc-etiquette/ | Logged at http://eavesdrop.openstack.org/irclogs/%23rdo/" | 14:44 | |
| openstack | Meeting ended Wed Feb 12 14:44:08 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 14:44 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/rdo_meeting___2020_02_12/2020/rdo_meeting___2020_02_12.2020-02-12-14.00.html | 14:44 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/rdo_meeting___2020_02_12/2020/rdo_meeting___2020_02_12.2020-02-12-14.00.txt | 14:44 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/rdo_meeting___2020_02_12/2020/rdo_meeting___2020_02_12.2020-02-12-14.00.log.html | 14:44 |
| ykarel | Thanks jcapitao | 14:44 |
| leanderthal | thx jcapitao | 14:46 |
| amoralej | thx jcapitao | 14:47 |
| amoralej | jpena, https://review.opendev.org/#/c/707346/ is merged, it will be automatically picken on next image build? | 14:51 |
| jpena | amoralej: we have built an image with the patch cherry-picked, let me finish my current meeting and I'll check if it has been uploaded everywhere | 14:52 |
| amoralej | ack, great | 14:53 |
| amoralej | thanks jpena | 14:53 |
| ykarel | amoralej, is in new image, six >= 1.12 will be installed? | 14:53 |
| ykarel | as that's needed for new virtualenv | 14:53 |
| amoralej | ykarel, it will be in the tox installed with pip | 14:54 |
| amoralej | let me check | 14:54 |
| *** jaosorior has quit IRC | 14:54 | |
| *** avivgt has joined #rdo | 14:55 | |
| amoralej | ykarel, https://softwarefactory-project.io/nodepool-log/upstream-centos-8-0000006899.log | 14:56 |
| amoralej | 2020-02-12 14:06:54.376 | Successfully installed appdirs-1.4.3 distlib-0.3.0 filelock-3.0.12 importlib-metadata-1.5.0 importlib-resources-1.0.2 packaging-20.1 pluggy-0.13.1 py-1.8.1 pyparsing-2.4.6 toml-0.10.0 tox-3.14.3 virtualenv-20.0.2 zipp-2.2.0 | 14:56 |
| amoralej | not sure | 14:56 |
| ykarel | amoralej, so there i see export DIB_INSTALLTYPE_pip_and_virtualenv=package | 14:57 |
| amoralej | ykarel, it's tricky | 14:57 |
| amoralej | virtualenv is installed from pkg there | 14:58 |
| amoralej | but | 14:58 |
| amoralej | there is other element installing tox with pip | 14:58 |
| amoralej | and there | 14:58 |
| ykarel | :( | 14:58 |
| amoralej | virtualenv is installed as dep | 14:58 |
| amoralej | we have two installations | 14:58 |
| ykarel | okk, i see six-1.14.0 | 14:58 |
| ykarel | and i also see Package python3-six.noarch 0:1.11.0-8.el8 will be installed | 14:59 |
| ykarel | so now it depends what python virtualenv is using | 14:59 |
| amoralej | i think both should work now | 15:00 |
| ykarel | amoralej, with 20.0.2 virtualenv? | 15:00 |
| ykarel | that do not work with six < 1.12 | 15:00 |
| amoralej | so, we have 20.0.2 wnad six 1.14 in /usr/local | 15:02 |
| *** weshay is now known as weshay|ruck | 15:02 | |
| amoralej | that's my understanding | 15:02 |
| *** chkumar|rover is now known as raukadah | 15:02 | |
| ykarel | ohkk so six and virtualenv are installed both package and source | 15:03 |
| amoralej | yes | 15:03 |
| amoralej | that's what i understand, but need to test | 15:03 |
| ykarel | amoralej, so /my point is if these are mixed, we will see issues | 15:04 |
| ykarel | else don't | 15:04 |
| amoralej | there are different paths | 15:05 |
| amoralej | i hope they can live together | 15:05 |
| ykarel | yes they ideally live together | 15:06 |
| ykarel | amoralej, but looks like the installation is done with root user, so may have issues | 15:07 |
| ykarel | /usr/libexec/platform-python -m pip install tox | 15:07 |
| ykarel | can u confirm if ^^ is not with root | 15:07 |
| amoralej | that's root | 15:07 |
| ykarel | ohhk then possibly it will corrupt | 15:07 |
| *** jmlowe has joined #rdo | 15:08 | |
| amoralej | ykarel, old six is in /usr/share and new in /usr/local | 15:09 |
| amoralej | it may work | 15:09 |
| ykarel | amoralej, ack let's see | 15:10 |
| *** artom has joined #rdo | 15:11 | |
| zbr | you need to be bit more creative to avoid installation of virtualenv when installing tox, like install it frist from rpm. | 15:12 |
| jpena | so... the centos-8 image is now ok -> https://review.rdoproject.org/r/#/c/24916/ | 15:12 |
| jpena | I'm checking what's going on with the rest | 15:12 |
| zbr | i do have path for tox itself which will help, once released: https://github.com/tox-dev/tox/pull/1519/files | 15:13 |
| zbr | in fact tox is not the culprit here, only the main victim. | 15:13 |
| jpena | zbr: hehe, I proposed that same thing in https://github.com/tox-dev/tox/issues/1516 | 15:13 |
| zbr | jpena: i guess I know gabor better than you ;) | 15:14 |
| jpena | for sure! | 15:14 |
| zbr | the issue is that he conditioned the merge on passing CI, and we have an unrelated problme on https://dev.azure.com/toxdev/tox/_build/results?buildId=2316&view=logs&j=6cd38c3c-3a17-5dab-f848-d6174e69d150&t=713b09b6-8a9e-54ee-2ee7-0b2cab92bc66 | 15:15 |
| jpena | mjturek: btw, I think I have managed to remove the blob by removing all ppc64le images from tripleomaster. Can you try uploading one image and downloading it? | 15:16 |
| jpena | baha: ^ | 15:16 |
| *** avivgt has quit IRC | 15:16 | |
| *** dtantsur is now known as dtantsur|brb | 15:18 | |
| mjturek | thanks jpena will do | 15:18 |
| mjturek | let me see what e've got | 15:18 |
| baha | I think we should be able to upload the known-good one we've got locally | 15:18 |
| rdogerrit | User marios proposed rdo-infra/ci-config master: Add ppc_manifests check and switch to manifest_push to exclude ppc https://review.rdoproject.org/r/24807 | 15:20 |
| *** leanderthal has quit IRC | 15:20 | |
| artom | raukadah, morning :) Did you get a chance to go over https://review.rdoproject.org/r/#/c/24902/ ? | 15:21 |
| *** jmlowe has quit IRC | 15:21 | |
| amoralej | jpena, ykarel yeah, confirmed it's working now, i rechecked https://review.rdoproject.org/r/#/c/24796/ | 15:22 |
| ykarel | amoralej, cool, so then we are good | 15:22 |
| *** jtomasek has quit IRC | 15:22 | |
| rdogerrit | Artom Lifshitz proposed rdoinfo master: Add whitebox-tempest-plugin project https://review.rdoproject.org/r/24902 | 15:24 |
| *** cp- has quit IRC | 15:25 | |
| jpena | so... the cloud-centos-8 image has the same issue when installing tox from PyPi (we have six from python3-six, so it doesn't upgrade to a newer version) | 15:26 |
| *** cp- has joined #rdo | 15:27 | |
| ykarel | jpena, and that image is being used? | 15:27 |
| jpena | ykarel: I guess not much | 15:27 |
| jpena | I have some ideas on how to fix it, but let's wait until we see a failing job | 15:27 |
| mjturek | jpena trying to push now | 15:28 |
| ykarel | jpena, ack | 15:28 |
| mjturek | jpena: not positive it worked http://paste.openstack.org/show/789471/ | 15:30 |
| mjturek | but trying a pull next | 15:30 |
| *** rpittau is now known as rpittau|afk | 15:31 | |
| ykarel | amoralej, can u check https://review.rdoproject.org/r/#/c/24912/ | 15:32 |
| ykarel | i proposed pin for heat-tempest-plugin | 15:32 |
| ykarel | to avoid deps update for centos7 ussuri | 15:32 |
| amoralej | i'm not sure if we should put more effort in pinning to get consistent in centos7, tbh | 15:33 |
| *** jmlowe has joined #rdo | 15:34 | |
| rdogerrit | User marios proposed rdo-infra/ci-config master: Add ppc_manifests check and switch to manifest_push to exclude ppc https://review.rdoproject.org/r/24807 | 15:34 |
| ykarel | amoralej, so keep centos7 at non consistent state until we clear it? | 15:35 |
| ykarel | or u mean update deps instead of pinning | 15:35 |
| amoralej | i mean, letting centos7 die and focusing on getting centos8 ready and promoting | 15:36 |
| amoralej | i'm not sure how far we are of having centos8 jobs | 15:36 |
| ykarel | amoralej, ack okk for me, if consumers are fine with it | 15:36 |
| ykarel | like tripleo | 15:36 |
| amoralej | yeah, tripleo is the main one | 15:36 |
| *** jpich has quit IRC | 15:37 | |
| amoralej | but i'm not sure if it makes much sense to keep promoting on pinned | 15:37 |
| amoralej | we can merge this, np | 15:37 |
| amoralej | ans see if we can keep it alive a bit more | 15:37 |
| *** jpich has joined #rdo | 15:37 | |
| ykarel | hmm once c8 is ready or nearly ready we can drop c7 consistent | 15:38 |
| amoralej | btw, not that i recall the issue with octavia | 15:38 |
| ykarel | but currently on c8 container job is up, that too stuck at venv thing | 15:38 |
| ykarel | and other py2 issues | 15:38 |
| ykarel | amoralej, what octavia issue? | 15:39 |
| ykarel | openssl-static? | 15:39 |
| ykarel | or something else | 15:39 |
| amoralej | yeah, that one | 15:39 |
| ykarel | okk | 15:39 |
| ykarel | amoralej, so what u planning? update openssl-static or something else? | 15:39 |
| amoralej | to be consistent again we may pin octavia-tests-tempest to the last successful | 15:40 |
| amoralej | until we can find out | 15:40 |
| ykarel | amoralej, okk will include that with heat-tempest-plugin | 15:43 |
| mjturek | jpena: same error | 15:44 |
| jpena | (┛ಠ_ಠ)┛彡┻━┻ | 15:44 |
| mjturek | hahahahaha | 15:45 |
| mjturek | http://paste.openstack.org/show/789474/ | 15:45 |
| mjturek | see anything in the logs? | 15:46 |
| rdogerrit | Sorin Sbarnea (zbr) proposed rdo-infra/ci-config master: Switch nodesets to centos-8 https://review.rdoproject.org/r/24905 | 15:46 |
| *** ykarel is now known as ykarel|afk | 15:49 | |
| *** ysandeep is now known as ysandeep|away | 15:50 | |
| jpena | mjturek: hm, I see this | 15:50 |
| jpena | "PUT /v2/tripleomaster/centos-binary-base/manifests/f47571d32d7ced8d5a04c68f446eff0d01fbf7d4_b5a8f808_ppc64le_mjturek_test HTTP/1.1" 500 | 15:50 |
| jpena | so it's giving an error 500 for the PUT operation (upload) | 15:50 |
| mjturek | interesting | 15:50 |
| mjturek | geeze there's no more detail? | 15:51 |
| jpena | http://paste.openstack.org/show/789475/ is the full log from the operations related to your image | 15:51 |
| artom | raukadah, thanks for the review - addressed your feedback at https://review.rdoproject.org/r/#/c/24902/ | 15:53 |
| *** jpich has quit IRC | 15:54 | |
| *** dsneddon has joined #rdo | 15:55 | |
| *** Trevor_V has joined #rdo | 15:55 | |
| *** jpich has joined #rdo | 15:55 | |
| rdogerrit | Sorin Sbarnea (zbr) created rdo-jobs master: WIP: document nodesets https://review.rdoproject.org/r/24919 | 15:57 |
| mjturek | jpena: out of curiosity can you share what version of the registry you are using? | 15:58 |
| tristanC | zbr: could you have a look at https://review.rdoproject.org/r/#/c/24918/ , it seems like rdoproject.org/github-check is always red on libpod ... | 15:58 |
| zbr | jpena: amoralej please comment on ^ as I am not sure I put the right comments onn nodes. | 15:58 |
| jpena | mjturek: sure, that's the embedded registry from openshift 3.7 (we're migrating to 3.11 in a new VM) | 15:58 |
| *** skramaja has quit IRC | 15:59 | |
| *** TrevorV has quit IRC | 15:59 | |
| amoralej | zbr, you mean in https://review.rdoproject.org/r/#/c/24918 ? | 16:00 |
| mjturek | jpena: cool any idea when the upgrade will be finished? | 16:00 |
| *** dsneddon has quit IRC | 16:00 | |
| zbr | amoralej: nope, https://review.rdoproject.org/r/24919 | 16:01 |
| *** udesale has quit IRC | 16:02 | |
| zbr | you do know much better which one are recommended or deprecated, so i rely on you. documenting there is enough to set the right direction. | 16:02 |
| amoralej | zbr, the idea of having separated rdo-centos-8 was to do segmentation of nodes | 16:02 |
| *** udesale has joined #rdo | 16:02 | |
| *** avivgt has joined #rdo | 16:02 | |
| amoralej | some jobs were very long running ones | 16:02 |
| *** tesseract has joined #rdo | 16:02 | |
| jpena | mjturek: I think you could start testing with the new registry already. nhicher: do you have info about registry-vexxhost? | 16:02 |
| amoralej | and separating rdo-centos-8 helped to avoid long queues for short jobs | 16:03 |
| amoralej | but if we prefer to have a single nodeset, ok | 16:03 |
| mjturek | jpena: That would be nice | 16:03 |
| mjturek | just to be sure | 16:03 |
| zbr | we need to make this clear, so people would not which centos-8 one to pick if they want it fox a short tox job vs a deployment. | 16:03 |
| zbr | i would say short is likely 10min but clearly <1h, long >1h | 16:04 |
| *** maciejjozefczyk has quit IRC | 16:06 | |
| nhicher | jpena: what info do you need ? I think we have to sync last tag, not sure it was done since zoli has left the team | 16:07 |
| jpena | nhicher: mostly if it's ready for mjturek to use, and what credentials should they use | 16:07 |
| tristanC | zbr: could we please merge https://review.rdoproject.org/r/#/c/24918/ , it seems like all https://github.com/containers/libpod/pulls are red because of this. | 16:08 |
| zbr | raukadah: ^ is about podman | 16:09 |
| zbr | afaik is ok on my side, while someone is working on a fix. | 16:09 |
| zbr | glad our work was able to indicate issues. | 16:10 |
| zbr | weshay|ruck: ^ what do you think about https://review.rdoproject.org/r/#/c/24918/ ? | 16:10 |
| weshay|ruck | yikes | 16:12 |
| weshay|ruck | raukadah, ^ | 16:12 |
| weshay|ruck | zbr, that's what we test with | 16:12 |
| weshay|ruck | I guess nv is fine to resolve.. but that is a critical packaging bug | 16:12 |
| zbr | maybe we should not do it w/o including a link to LP bug. | 16:13 |
| weshay|ruck | that was my comment in the review | 16:13 |
| *** jmlowe has quit IRC | 16:13 | |
| raukadah | weshay|ruck, tristanC sorry what was the bug? | 16:15 |
| tristanC | raukadah: it seems like rhel8 build job is failing all the time, and iiuc, this results in all the libpod PR to have the red cross | 16:15 |
| raukadah | weshay|ruck, tristanC if something is broken it needs to be fixed, otherwise we will get issues in downstream | 16:16 |
| raukadah | tristanC, can we analayze the issue? which causing red | 16:16 |
| *** Son_Goku has joined #rdo | 16:16 | |
| tristanC | raukadah: i agree, and i don't know what needs to be fixed, but zuul is currently giving false-positive | 16:17 |
| raukadah | lp would be better with cix, and I will look into that tomorrow, as a temp measure we can make nv | 16:17 |
| raukadah | weshay|ruck, ^6 what do you say? | 16:17 |
| tristanC | raukadah: the issue is https://review.rdoproject.org/zuul/build/f655534b4a694c79adfb04210177168c/log/job-output.txt#262 | 16:18 |
| rdogerrit | Rafael Folco proposed rdo-infra/ci-config master: convert get-hash bash task into a reusable role https://review.rdoproject.org/r/24762 | 16:18 |
| tristanC | Unable to find a match: device-mapper-devel glibc-static go-compilers-golang-compiler gpgme-devel libassuan-devel | 16:18 |
| tristanC | raukadah: i think we should merge the voting:false change now, and revert the change once the job is green again | 16:18 |
| raukadah | ah someone changes something in buildrpm,sh | 16:18 |
| raukadah | may be raberpi fix | 16:19 |
| raukadah | need to check | 16:19 |
| weshay|ruck | tristanC, just needs a lp :) | 16:19 |
| raukadah | weshay|ruck, thanks for pinging me | 16:20 |
| raukadah | I will get that fixed tomorrow | 16:20 |
| *** dsneddon has joined #rdo | 16:20 | |
| *** Son_Goku has quit IRC | 16:25 | |
| rdogerrit | Sorin Sbarnea (zbr) proposed rdo-infra/ci-config master: WIP: Switch nodesets to centos-8 https://review.rdoproject.org/r/24905 | 16:33 |
| *** marios is now known as marios|out | 16:45 | |
| mjturek | nhicher: think it's something we could try? | 16:46 |
| *** ramishra has quit IRC | 16:49 | |
| *** udesale has quit IRC | 16:51 | |
| rdogerrit | Javier Peña created rdo-infra/ci-config master: Fix URL to logs in logs.html https://review.rdoproject.org/r/24920 | 16:51 |
| *** udesale has joined #rdo | 16:52 | |
| nhicher | jpena, mjturek: registry is trunk.registry-vexxhost.rdoproject.org, the secret is define in zuul.d/secrets.yaml, name is rdo_vexxhost_registry | 16:52 |
| rdogerrit | wes hayutin created rdo-infra/ci-config master: update dashboard for us22 https://review.rdoproject.org/r/24921 | 16:53 |
| *** avivgt has quit IRC | 16:55 | |
| *** avivgt has joined #rdo | 16:56 | |
| nhicher | jpena, mjturek: for user it should be the same than the registry in rdocloud | 16:56 |
| *** marios|out has quit IRC | 16:57 | |
| jpena | nhicher: could you decrypt the secret and send it privately to mjturek? With that, he could check pushing a single image there, and see what happens | 16:57 |
| *** iurygregory has quit IRC | 16:58 | |
| nhicher | jpena: I think data should be in the instance zoli used to do the sync, I'm checking if I can get what is needed from there | 17:00 |
| *** lmiccini has quit IRC | 17:01 | |
| *** udesale has quit IRC | 17:08 | |
| *** aufi has quit IRC | 17:09 | |
| rdogerrit | Rafael Folco proposed config master: Switch to the new get_hash role https://review.rdoproject.org/r/24847 | 17:11 |
| *** igordc has joined #rdo | 17:17 | |
| *** migarcia is now known as migarcia|away | 17:21 | |
| *** holser has quit IRC | 17:26 | |
| *** tesseract has quit IRC | 17:32 | |
| *** jpich has quit IRC | 17:36 | |
| *** jlibosva has quit IRC | 17:37 | |
| *** iurygregory has joined #rdo | 17:40 | |
| rfolco | jpena, zbr do we have a fix for rdo venv issues ? | 17:42 |
| *** xek__ has joined #rdo | 17:42 | |
| jpena | rfolco: for the centos 8 images, yes | 17:42 |
| jpena | it should be fixed now | 17:42 |
| jpena | are you finding trouble elsewhere? | 17:42 |
| rfolco | https://review.rdoproject.org/r/#/c/24762/ my patch still failing, let me check if the error is different | 17:43 |
| rfolco | oh now its molecule... something has changed there | 17:43 |
| rfolco | jpena, ignore me for now | 17:44 |
| rfolco | jpena, thx | 17:44 |
| jpena | ack | 17:44 |
| *** xek_ has quit IRC | 17:44 | |
| jpena | rfolco: the molecule error looks similar to one we had a few days ago. Try pinning to molecule <=3.0a5 | 17:45 |
| rfolco | jpena, molecule>=3.0a6 in tox.ini I see | 17:46 |
| *** joelinux__ has quit IRC | 17:56 | |
| rdogerrit | Rafael Folco proposed rdo-infra/ci-config master: convert get-hash bash task into a reusable role https://review.rdoproject.org/r/24762 | 17:57 |
| zbr | rfolco: replace conditon with == on molecule | 17:58 |
| zbr | anywhere you spot an error, first this to do to put a == | 17:58 |
| rdogerrit | Merged rdo-infra/ci-config master: update dashboard for us22 https://review.rdoproject.org/r/24921 | 17:58 |
| rfolco | zbr, is it ok to be in the same patch or prefer to be a separated one to ease tracking this single change ? | 17:59 |
| rfolco | https://review.rdoproject.org/r/#/c/24762/20/tox.ini | 17:59 |
| zbr | rfolco: likely same. | 18:00 |
| *** derekh has quit IRC | 18:01 | |
| zbr | rfolco: why using a delegated approach? and why testinfra when I do not see any testinfra files? | 18:03 |
| zbr | asking because i do not know about get_hash role requirements, is there something preventing it from being tested with containers? | 18:04 |
| rfolco | zbr, trying to remember why | 18:05 |
| rfolco | one sec | 18:05 |
| zbr | i see lots of weir stuff inside molecule.yml but i suppose someone added them with a reason. | 18:05 |
| zbr | many options configured there prevent running the scenario on a dev machine, i will ask panda tomorrow. now i am too tired | 18:07 |
| *** igordc has quit IRC | 18:08 | |
| rfolco | zbr, the reason we started w/ delegated was that it was impossible to run docker on docker for some of the promoter tests | 18:08 |
| rfolco | zbr, for this case I don't need it. Is this a problem ? please review it tomorrow if you are tired | 18:09 |
| rfolco | thanks zbr | 18:09 |
| zbr | rfolco: ok, next time you edit molecule file mention this near the "driver: delegated" line. | 18:09 |
| zbr | nothing more. | 18:09 |
| zbr | that is the perfect use-case for delgated. | 18:09 |
| rfolco | zbr, for get-hash I don't need it to be delegated I mean | 18:09 |
| rfolco | zbr, is this a problem? | 18:10 |
| zbr | it is for testability, when delegated it will not be possible to test locally. | 18:10 |
| rfolco | ok, let me switch to normal driver | 18:10 |
| zbr | which means: prefer containers unless DoD reasons. | 18:10 |
| *** jpena is now known as jpena|off | 18:11 | |
| *** jcapitao has quit IRC | 18:11 | |
| zbr | i in a month or so, we will have generic container driver, one that uses either podman or docker, i only need few days to write it. | 18:11 |
| *** eharney has quit IRC | 18:18 | |
| *** gfidente is now known as gfidente|afk | 18:23 | |
| *** igordc has joined #rdo | 18:23 | |
| rdogerrit | Yatin Karel proposed rdoinfo master: Pin heat-tempest-plugin to good hash https://review.rdoproject.org/r/24912 | 18:26 |
| ykarel|afk | amoralej, added octavia-tempest-plugin with ^^ | 18:27 |
| amoralej | ack | 18:27 |
| amoralej | ykarel|afk, i've built a couple of new packages required for rpm-packaging | 18:27 |
| amoralej | python-confluent-kafka-0.11.6-8.el8 and librdkafka-1.3.0-2.el8 | 18:27 |
| *** chem has quit IRC | 18:28 | |
| amoralej | and i'm updating python-six | 18:28 |
| ykarel|afk | amoralej, okk, and added to deps repo too? | 18:28 |
| amoralej | yep | 18:28 |
| ykarel|afk | amoralej, to what version? six | 18:28 |
| ykarel|afk | 1.14? | 18:28 |
| amoralej | yes | 18:28 |
| ykarel|afk | okk | 18:28 |
| rdogerrit | Daniel Pawlik proposed rdo-infra/ci-config master: Fix URL to logs in logs.html https://review.rdoproject.org/r/24920 | 18:34 |
| *** amoralej is now known as amoralej|off | 18:36 | |
| *** social has quit IRC | 18:37 | |
| *** ykarel|afk is now known as ykarel|away | 18:39 | |
| *** jmlowe has joined #rdo | 18:40 | |
| *** tmazur has quit IRC | 18:43 | |
| *** stevebaker has joined #rdo | 18:46 | |
| *** tosky has quit IRC | 18:47 | |
| rdogerrit | Rafael Folco proposed rdo-infra/ci-config master: convert get-hash bash task into a reusable role https://review.rdoproject.org/r/24762 | 18:49 |
| rdogerrit | Gabriele Cerami proposed rdo-infra/ci-config master: Modularize Promoter: add config class https://review.rdoproject.org/r/24675 | 18:59 |
| rdogerrit | Gabriele Cerami proposed rdo-infra/ci-config master: Modularize Promoter: creating execution path split https://review.rdoproject.org/r/24676 | 18:59 |
| rdogerrit | Gabriele Cerami proposed rdo-infra/ci-config master: Modularize Promoter: replace legacy promoter function https://review.rdoproject.org/r/24677 | 18:59 |
| rdogerrit | Gabriele Cerami proposed rdo-infra/ci-config master: Modularize Promoter: replace promote_all_links legacy function https://review.rdoproject.org/r/24678 | 18:59 |
| rdogerrit | Gabriele Cerami proposed rdo-infra/ci-config master: Modularize Promoter: replace setup_logging https://review.rdoproject.org/r/24684 | 18:59 |
| rdogerrit | Gabriele Cerami proposed rdo-infra/ci-config master: Modularize Promoter: replace fetch_hashes https://review.rdoproject.org/r/24687 | 18:59 |
| rdogerrit | Gabriele Cerami proposed rdo-infra/ci-config master: Modularize Promoter: replace fetch_jobs https://review.rdoproject.org/r/24699 | 18:59 |
| rdogerrit | Gabriele Cerami proposed rdo-infra/ci-config master: Modularize Promoter: replace get_latest_hashes https://review.rdoproject.org/r/24694 | 18:59 |
| rdogerrit | Gabriele Cerami proposed rdo-infra/ci-config master: Modularize Promoter: replace tag_qcow_images https://review.rdoproject.org/r/24714 | 18:59 |
| rdogerrit | Gabriele Cerami proposed rdo-infra/ci-config master: Modularize Promoter: replace check named hashes function https://review.rdoproject.org/r/24715 | 18:59 |
| rdogerrit | Gabriele Cerami proposed rdo-infra/ci-config master: Modularize Promoter: replace tag_containers https://review.rdoproject.org/r/24716 | 18:59 |
| rdogerrit | Gabriele Cerami proposed rdo-infra/ci-config master: Modularize Promoter: replace promote_link https://review.rdoproject.org/r/24721 | 18:59 |
| rdogerrit | Gabriele Cerami proposed rdo-infra/ci-config master: Compoment pipeline: Implement aggregate hashes for promoter workflow https://review.rdoproject.org/r/24774 | 18:59 |
| rdogerrit | Gabriele Cerami proposed rdo-infra/ci-config master: Adapt staging environment to DlrnHash and add component pipeline scenario https://review.rdoproject.org/r/24782 | 18:59 |
| rdogerrit | Gabriele Cerami proposed rdo-infra/ci-config master: [WIP] Adapt staging environment for the component pipeline https://review.rdoproject.org/r/24818 | 18:59 |
| panda | sorry ^ | 18:59 |
| *** ralonsoh has quit IRC | 19:04 | |
| *** alexmcleod has quit IRC | 19:07 | |
| *** eharney has joined #rdo | 19:17 | |
| mjturek | nhicher jpena|off: http://paste.openstack.org/show/789483/ | 19:33 |
| mjturek | seeing this with the vexxhost registry ^ | 19:33 |
| mjturek | see anything on your end? | 19:33 |
| nhicher | mjturek: yes: /dev/vdc 1.0T 1.0T 20K 100% /openshift_volumes | 19:38 |
| mjturek | oh my... did I do that? | 19:38 |
| nhicher | mjturek: probably not =) | 19:39 |
| mjturek | phew... I mean... of course I didn't | 19:39 |
| nhicher | mjturek: I will investigate, we don't have anything in place to prune data for now for it was only for testing | 19:39 |
| mjturek | right right | 19:39 |
| mjturek | nhicher: let me know if/when you think I can try another push | 19:40 |
| nhicher | mjturek: we found the issue, the user which should prune the registry doesn't have right to prune images | 19:46 |
| mjturek | cool! | 19:47 |
| *** morazi has quit IRC | 19:48 | |
| rdogerrit | Chandan Kumar created rdo-jobs master: [DNM] full tempest image saving https://review.rdoproject.org/r/24922 | 19:50 |
| *** stevebaker has quit IRC | 19:50 | |
| *** dtantsur|brb is now known as dtantsur|afk | 19:51 | |
| *** stevebaker has joined #rdo | 19:51 | |
| *** igordc has quit IRC | 19:58 | |
| rdogerrit | Tristan de Cacqueray created rdo-infra/rdo-infra-playbooks master: registry-image-pruning: use the correct cluster hostname https://review.rdoproject.org/r/24923 | 20:02 |
| nhicher | mjturek: pruning in progress | 20:08 |
| *** jmlowe has quit IRC | 20:08 | |
| mjturek | great! let me know when you think it's safe to try a push nhicher | 20:09 |
| nhicher | mjturek: it should be ok, we have now 15G available | 20:11 |
| nhicher | mjturek: and pruning is in progress, so feel free to do your testing | 20:12 |
| mjturek | oh great thanks nhicher! | 20:12 |
| nhicher | mjturek: you're welcome | 20:13 |
| mjturek | heyyyy nhicher pushing and pulling to this registry works! | 20:22 |
| mjturek | rfolco ^ | 20:22 |
| mjturek | baha ^ | 20:22 |
| baha | \o/ | 20:22 |
| rfolco | mjturek, what was the issue? | 20:23 |
| mjturek | we don't know what the issue is rfolco | 20:23 |
| mjturek | but the old registry doesn't work | 20:23 |
| mjturek | the new one they are migrating to does | 20:23 |
| rfolco | nice | 20:23 |
| mjturek | 3.9 -> 3.11 | 20:23 |
| mjturek | sorry 3.7 -> 3.11 | 20:23 |
| mjturek | nhicher any idea when rdoregistry will be using this? | 20:25 |
| nhicher | mjturek: you can't push on the old registry ? | 20:47 |
| *** ChanServ has quit IRC | 20:50 | |
| mjturek | nhicher: we can push to the old registry but cannot pull from it | 20:52 |
| nhicher | mjturek: I'm not familiar with the old registry, it's better to wait for jpena|off if it's ok for you | 20:54 |
| mjturek | sure nhicher - but the intent is to switch to the registry on vexxhost eventually, right? | 20:56 |
| nhicher | mjturek: yes probably in 2 weeks | 20:56 |
| mjturek | ahhh okay thank you nhicher | 21:01 |
| nhicher | mjturek: but we can try to fix your pull issue tomorrow with jpena|off | 21:02 |
| mjturek | fair enough! | 21:02 |
| nhicher | mjturek: 3.7 and 3.11 are openshift versions right ? | 21:04 |
| baha | That would be nice! | 21:04 |
| nhicher | mjturek: if yes, no plan to update old registry to 3.11 | 21:04 |
| rdogerrit | Merged rdo-infra/rdo-infra-playbooks master: registry-image-pruning: use the correct cluster hostname https://review.rdoproject.org/r/24923 | 21:04 |
| baha | nhicher: It does indeed refer to openshift version, do you mean that centos 7 jobs won't be using 3.11 when you spin up the new environment? | 21:09 |
| nhicher | baha: all jobs will use the new registry | 21:10 |
| baha | OK! Thank you | 21:11 |
| mjturek | awesome | 21:11 |
| *** artom has quit IRC | 21:12 | |
| *** ChanServ has joined #rdo | 21:13 | |
| *** orwell.freenode.net sets mode: +o ChanServ | 21:13 | |
| *** holser has joined #rdo | 21:16 | |
| rdogerrit | Rafael Folco proposed rdo-infra/ci-config master: convert get-hash bash task into a reusable role https://review.rdoproject.org/r/24762 | 21:17 |
| *** rfolco has quit IRC | 21:18 | |
| *** rcernin has joined #rdo | 21:18 | |
| *** jmlowe has joined #rdo | 21:27 | |
| *** jfrancoa has quit IRC | 21:37 | |
| *** morazi has joined #rdo | 21:40 | |
| *** holser has quit IRC | 21:50 | |
| *** nweinber has quit IRC | 21:54 | |
| *** mcornea has quit IRC | 21:56 | |
| *** raildo_ has joined #rdo | 22:05 | |
| *** raildo has quit IRC | 22:08 | |
| *** sshnaidm has quit IRC | 22:17 | |
| *** sshnaidm has joined #rdo | 22:18 | |
| *** slaweq has quit IRC | 22:26 | |
| *** jmlowe has quit IRC | 22:40 | |
| *** jmlowe has joined #rdo | 22:43 | |
| *** jmlowe has quit IRC | 22:53 | |
| *** tkajinam has joined #rdo | 22:54 | |
| *** jmlowe has joined #rdo | 22:55 | |
| *** Trevor_V has quit IRC | 23:04 | |
| *** rh-jelabarre has quit IRC | 23:04 | |
| *** xek__ has quit IRC | 23:08 | |
| *** jdandrea has quit IRC | 23:09 | |
| *** apevec has quit IRC | 23:10 | |
| *** jmlowe has quit IRC | 23:12 | |
| *** ccamposr has quit IRC | 23:24 | |
| *** rlandy is now known as rlandy|bbl | 23:26 | |
| *** xek has joined #rdo | 23:37 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!