Tuesday, 2022-11-15

opendevreviewRabi Mishra proposed openstack/tripleo-ansible master: Fix compatibility with Ansible 2.14  https://review.opendev.org/c/openstack/tripleo-ansible/+/86439202:53
opendevreviewBrendan Shephard proposed openstack/tripleo-docs master: Update docs for CentOS9  https://review.opendev.org/c/openstack/tripleo-docs/+/86319302:54
opendevreviewTakashi Kajinami proposed openstack/tripleo-ansible master: Fix compatibility with Ansible 2.14  https://review.opendev.org/c/openstack/tripleo-ansible/+/86439203:10
opendevreviewRabi Mishra proposed openstack/tripleo-ansible master: WIP Drop ansible-core pin from requirements  https://review.opendev.org/c/openstack/tripleo-ansible/+/86446803:46
opendevreviewTakashi Kajinami proposed openstack/tripleo-docs master: Update docs for CentOS9  https://review.opendev.org/c/openstack/tripleo-docs/+/86319303:49
opendevreviewTakashi Kajinami proposed openstack/python-tripleoclient master: overcloud image build: Use CentOS Stream 9 by default  https://review.opendev.org/c/openstack/python-tripleoclient/+/86446903:55
opendevreviewTakashi Kajinami proposed openstack/tripleo-common master: Remove image yamls for CentOS 8/RHEL 8  https://review.opendev.org/c/openstack/tripleo-common/+/86449103:58
opendevreviewRabi Mishra proposed openstack/tripleo-ansible master: Fix compatibility with Ansible 2.14  https://review.opendev.org/c/openstack/tripleo-ansible/+/86439204:33
opendevreviewRabi Mishra proposed openstack/tripleo-ansible master: WIP Drop ansible-core pin from requirements  https://review.opendev.org/c/openstack/tripleo-ansible/+/86446804:33
*** ysandeep|out is now known as ysandeep05:57
*** soniya29|rover is now known as soniya29|rover|afk05:59
*** amoralej|off is now known as amoralej07:02
tkajinamhmm. seems c-p job still gets stuck https://review.opendev.org/c/openstack/tripleo-ansible/+/86439207:07
ramishratkajinam: it passed in a dependant job though https://zuul.opendev.org/t/openstack/stream/4ba9f6503c3745258b6d032d0f37459a?logfile=console.log07:11
ramishrawaiting for the logs07:11
tkajinamwired07:11
tkajinamramishra, I suspect infra issue like slow network thing but will check logs once it times out07:12
tkajinamwe probably want to lower down timeout from current 3 hours. It's not quite normal that c-p job takes more than 2 hours.07:12
tkajinamhmm... ok. seems we need to add time for subsequent jobs so we can reduce it.07:14
tkajinamwe can't *&07:14
*** soniya29|rover|afk is now known as soniya29|rover07:46
*** yadnesh|away is now known as yadnesh07:48
mariosramishra: tkajinam: what do you folks think of pinning ansible then and fixing this without the pressure of the gate blocker07:51
mariosramishra: tkajinam: and then we unpin it07:51
mariosreading back... do you think the latest might be OK but it times out because of infra issue ?07:52
mariosakahat|ruck: fyi ^^ 07:52
tkajinammarios, if we can try pinning ansible then it's better we check it concurrently07:53
tkajinamthe latest patch fixes the failure in deployment jobs but still c-p jobs time out consistently07:55
*** slaweq_ is now known as slaweq07:57
opendevreviewMarios Andreou proposed openstack/tripleo-quickstart master: Pin ansible-core until we can fix compatibility with 2.14  https://review.opendev.org/c/openstack/tripleo-quickstart/+/86449808:03
mariostkajinam: ramishra: akahat|ruck: ^^ i think that should work ?08:03
mariosits coming from appstream repo http://mirror.stream.centos.org/9-stream/AppStream/source/tree/Packages/08:04
mariosthere is 2.13 available there so it should use that by excluding 2.14.*08:04
tkajinammarios, I have no clear clue but let's see how it works08:05
mariostkajinam: but it is green for the periodics with tripleo-ansible/+/864392 there .. https://review.rdoproject.org/r/c/testproject/+/41465/27#message-56f5923d98d640025e12c4c9f439bf848c98192608:05
tkajinammarios, the latest problem is that, container image build hungs while generating buildah files. I don't know the actual cause now. We have no errors, but all I see is that the output stops at some point.08:06
tkajinamwired08:06
tkajinamdo we use the same nodeset for periodic jobs ?08:06
mariostkajinam: ack but root is something to do with newer ansible i mean we didn't have this before yesterday right so... either way it would be better to resolve it without pressure. pin ansible for a few days is not a big problem imo08:07
tkajinamyeah08:07
mariostkajinam: no different environment/nodesets so it could be something different in the dependencies/package versions etc there08:07
mariostkajinam: i mean periodic/vexx or ibmcloud and gate is openstack infra08:07
tkajinammarios, the only tricky part is that we can't use testproject to test content provider job but we are probably able to use just upstream gerrit .08:07
tkajinamah, ok08:08
mariostkajinam: yeah just run a dnm tripleo-heat-templates or whatever and run the content provider08:08
tkajinamyeah, I agree08:08
tkajinamI have to leave in 15 minutes in today but will look into it tonight or early tomorrow.08:08
mariostkajinam: i think if you really wanted to you could run content provider in rdo/testproject but no need just run dnm08:08
tkajinams/in today/today/08:08
tkajinammarios, ok. makes sense08:09
mariostkajinam: k we will go with the pin i think for now then agree?08:09
tkajinammarios, I have no objections to pinning, and I think it's the best solution (if it works) now since the gate has been blocked for 2 days08:09
mariostkajinam: thanks08:09
mariostkajinam: yes agree because of the gate blocker and since it is not a big cost to pin for a few days08:10
opendevreviewJiri Podivin proposed openstack/tripleo-ansible master: Adding modules for retrieval of overcloud role info  https://review.opendev.org/c/openstack/tripleo-ansible/+/86215008:12
tkajinamfortunately all fixes we have identified so far should be compatible with ansible 2.13 so we can merge these before unpin08:17
opendevreviewMarios Andreou proposed openstack/tripleo-heat-templates master: DNM testing tripleo-quickstart/+/864498  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/86449908:17
*** jpena|off is now known as jpena08:21
opendevreviewMarios Andreou proposed openstack/tripleo-heat-templates stable/zed: DNM testing tripleo-quickstart/+/864498  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/86447408:21
opendevreviewMarios Andreou proposed openstack/tripleo-heat-templates stable/wallaby: DNM testing tripleo-quickstart/+/864498  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/86450008:24
*** soniya29|rover is now known as soniya29|rover|lunch08:25
mariostkajinam: fyi https://bugs.launchpad.net/tripleo/+bug/1996482/comments/508:33
*** ysandeep is now known as ysandeep|lunch08:45
*** soniya29|rover|lunch is now known as soniya29|rover09:22
*** akahat|ruck is now known as akahat|ruck|afk09:53
Tengumarios: is there an issue with the content provider for 9?09:56
opendevreviewFernando Díaz Bravo proposed openstack/tripleo-ansible master: bnr - Make the galera desync retries and delay configurable  https://review.opendev.org/c/openstack/tripleo-ansible/+/86196609:57
mariosTengu: yeah where are you seeing it09:59
mariosTengu: probably related to ansible 2.1409:59
Tengumarios: well, among places: https://review.opendev.org/c/openstack/tripleo-ansible/+/86439210:01
Tenguthe very patch that should fix the compat for 2.1410:01
mariosTengu: hoping it is resolved by pinning ansible - info there https://bugs.launchpad.net/tripleo/+bug/1996482/comments/510:07
Tenguyeah, read it already :)10:07
Tengulemme know if there are votes to provide.10:08
mariosTengu: https://zuul.openstack.org/status#864498 10:08
mariosTengu: looks like it is paused in all 3 branches thre so it means the containers were built already 10:08
Tengucool10:09
Tengufinger crossed.10:09
Tengubrb, need some good hot tea.10:09
*** ysandeep|lunch is now known as ysandeep10:20
mariostkajinam: Tengu: chandankumar: ysandeep: votes please when you have time https://review.opendev.org/c/openstack/tripleo-quickstart/+/864498/1#message-75d3fc345e113701842fe7980911d9575c1d3759 10:20
Tengumarios: ah, it's also hitting wallaby?10:21
Tengumarios: wondering what will happen downstream then... ?10:21
ysandeepmarios, looking10:22
Tengumarios: you have some extra-extra-spaces in the LP URI in your comment.10:22
Tenguvoted, seems OK, but the typo is annoying :).10:23
opendevreviewVeronika Fisarova proposed openstack/validations-libs master: Run validations with parameters from a file  https://review.opendev.org/c/openstack/validations-libs/+/86429110:26
mariosTengu: https://review.opendev.org/c/openstack/tripleo-quickstart/+/864498/1#message-8e80c30277f429a57612287adede268c44243f4010:30
mariosTengu: yeah sorry about that10:30
mariosTengu: honestly as i wrote i would update it without thought 10:30
mariosTengu: but ... reasons i wrote there10:30
Tengu^^10:30
mariosTengu: and we'll be reverting it ASAP 10:30
ysandeepmarios, will +w as soon as ci reports, +2ed already10:30
Tengumarios: as said - won't block on that lame typo, and you get my +2 anyway.10:30
mariosthnks10:30
mariosTengu: thanks for reviewing it properly ;)10:31
Tengu^^10:31
mariosthank you ysandeep 10:38
Tengumarios: yeah, so, no, content-provider are pretty dead.10:48
Tenguwondering why. TIMED_OUT.10:49
Tengure-re-re-checking, but...10:49
mariosTengu: where?10:51
mariosit was green on the pin patch? 10:51
marioshttps://review.opendev.org/c/openstack/tripleo-quickstart/+/864498/1#message-dd3e70a06e54d4f079b7e795345d266c9554ddde10:51
marios12:49 < Tengu> re-re-re-checking, but...10:53
marios?10:53
marioswhich patch 10:53
marioshttps://zuul.openstack.org/status#864498 10:53
mariosdont see timeout there or i miss something10:53
Tengumarios: https://review.opendev.org/c/openstack/tripleo-operator-ansible/+/864409 - https://zuul.opendev.org/t/openstack/build/9fbeef18bf7d422da11266ff0f4d9af210:57
Tengumarios: and https://review.opendev.org/c/openstack/tripleo-ansible/+/864392 - https://zuul.opendev.org/t/openstack/build/c6bade93696749d6839445d28ccde412 + https://zuul.opendev.org/t/openstack/build/0cba732d7ddf4e1ab408d2c9a40b1b9e10:57
Tenguso we'll face that anyway if it's linked to 2.14 - but with your pinning, we'll get some oxygen to debug this properly.10:57
Tengu2022-11-15 10:38:49.500310 | primary |   "msg": "The task includes an option with an undefined variable. The error was: 'ensure_pip_virtualenv_command' is undefined\n\nThe error appears to be in '/var/lib/zuul/builds/9fbeef18bf7d422da11266ff0f4d9af2/untrusted/project_2/opendev.org/openstack/tripleo-ci/roles/build-containers/tasks/build-report.yaml': line 77, column 7, but may\nbe elsewhere in the file 10:58
Tengudepending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n    - name: Install pytest\n      ^ here\n"10:59
Tengu"woops".10:59
Tenguah, wait - "ignoring errors"10:59
Tenguok - so the actual image build is hitting a timeout.10:59
TenguTASK [tripleo.operator.tripleo_container_image_build : Run tripleo container image build] [ASYNC POLL++] -> RUN END RESULT_TIMED_OUT: [untrusted : opendev.org/openstack/tripleo-ci/playbooks/tripleo-ci/run-v3.yaml@master]11:00
Tenguit seems to be linked to "Attempting python interpreter discovery" task that is coming over and over and over and over again11:02
Tenguhttps://e500f5844c497d7c1455-bb0af7d0ed113130252cfd767637324e.ssl.cf2.rackcdn.com/864409/2/check/tripleo-ci-centos-9-content-provider/9fbeef1/logs/undercloud/home/zuul/container_image_build.log11:02
Tenguguess we'll have to play.11:03
Tenguand we may even need both patches (tripleo-ansible + tripleo-operator-ansible) to be in, actually.11:03
Tenguso pinning ansible is more than probably the right thing to do11:04
Tenguanyway - lunch.11:04
*** pojadhav is now known as pojadhav|dr_appt11:09
*** amoralej is now known as amoralej|off11:29
*** akahat|ruck|afk is now known as akahat|ruck11:32
opendevreviewchandan kumar proposed openstack/tripleo-ansible master: Add missing vars for tripleo-standalone-vars  https://review.opendev.org/c/openstack/tripleo-ansible/+/86393511:41
mariosTengu: ack sorry was in calls but yeah, the content provider thing showed up later but also seems to be part of the ansible.214 and it will need to be fixed before we unpin12:10
opendevreviewArx Cruz proposed openstack/ansible-role-collect-logs master: Fix ansible sanity and linters jobs  https://review.opendev.org/c/openstack/ansible-role-collect-logs/+/86439912:10
Tengumarios: yeah. guess testproject will be put to good use :)12:10
Tengumarios: basically, it looks like it's "attempt python intepreter discovery" for every single container to build...12:11
Tenguhmmm ok. so that job is apparently using tripleo_ansible/playbooks/cli-generate-containerfile.yaml12:16
Tenguo_O12:18
* Tengu doesn't understand how the role can generate those logs12:19
Tengumarios: I'll open a new LP for those content provider jobs, so that we can get a dedicated thread/analysis. OK for you? Or do you prefer all under the same issue (https://bugs.launchpad.net/tripleo/+bug/1996482) ?12:24
Tenguuho. wait. chandankumar it seems the TCIB parameters are passed in some weird order: --work-dir /home/zuul/container-builds >/home/zuul/container_image_build.log 2>&1 --tcib-extras tcib_release=9 --tcib-extras tcib_python_version=3.912:26
Tengui.e. the --tcid-extras are passed after the redirection to the log... not sure it's working as you'd expect.12:27
chandankumarTengu: https://opendev.org/openstack/tripleo-operator-ansible/src/branch/master/roles/tripleo_container_image_build/tasks/main.yml#L2612:28
Tenguchandankumar: yeah, saw it, pushing a patch12:28
chandankumaryes, it should be passed earlier12:29
opendevreviewCedric Jeanneret proposed openstack/tripleo-operator-ansible master: Correct parameter ordering  https://review.opendev.org/c/openstack/tripleo-operator-ansible/+/86451912:29
Tenguchandankumar: -^^12:29
Tengu"fun".12:29
Tengu1y ago12:30
TenguXD12:30
Tengu"fun".12:30
mariosTengu: sure new bug sounds good12:33
mariosthanks12:33
mariosakahat|ruck: ^^^ 12:33
Tengumarios: OK - on it12:33
akahat|ruckack12:34
Tenguakahat|ruck. marios https://bugs.launchpad.net/tripleo/+bug/199661212:40
akahat|ruckTengu, thank you!! 12:40
Tengumarios, akahat|ruck btw - I'm pretty sure we'd be able to avoid this kind of shitstorm...12:40
TenguI'm pretty sure we got "deprecation warning" all over the place, but we don't have anything to check them, do we?12:41
Tengu~> would it make sens to get something like a dry-run (ansible-playbook -c) on the generated ansible things (config-download) that would gather the deprecation warning, for instance?12:41
Tenguiirc we have such a dry-run job downstream; pretty sure having one upstream (even non-voting) with, maybe, proper rules to parse the ansible deprecation warning and expose them in a dedicated log would really help.12:42
Tenguwdyt?12:42
mariosthanks Tengu https://bugs.launchpad.net/tripleo/+bug/1996612/comments/1 12:42
Tengumarios: ah, yeah, should have mentioned the workaround. up to you for the tagging of the issue.12:43
mariosTengu: ack so is clear it isn't blocking alarma alarma12:44
mariosakahat|ruck: i added to hackmd12:44
Tenguwondering how I can test that container build thing locally...12:45
Tenguwill start one of my builders and do some local testing. That should help.12:45
Tengubrb12:45
opendevreviewBogdan Dobrelya proposed openstack/tripleo-heat-templates stable/wallaby: Use python to template cell urls  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/86447513:10
opendevreviewchandan kumar proposed openstack/tripleo-ansible master: Add missing vars for tripleo-standalone-vars  https://review.opendev.org/c/openstack/tripleo-ansible/+/86393513:20
*** pojadhav|dr_appt is now known as pojadhav13:25
opendevreviewMerged openstack/tripleo-quickstart master: Pin ansible-core until we can fix compatibility with 2.14  https://review.opendev.org/c/openstack/tripleo-quickstart/+/86449813:55
*** dasm|off is now known as dasm13:58
*** amoralej|off is now known as amoralej13:58
akahat|ruckrlandy, marios soniya29|rover ^^ patch is merged.. triggering periodic lines13:59
rlandyakahat|ruck: thanks13:59
soniya29|roverakahat|ruck, ack13:59
Tengumarios, akahat|ruck https://bugs.launchpad.net/tripleo/+bug/1996612/comments/2  "fun" I said.14:00
Tenguchandankumar: -^   if you want to follow as well.14:00
* tkajinam is catching up with the latest status14:02
rlandydasm: TC/UA/PM sync14:03
tkajinammarios, glad that pinning ansible fixed the issue, though it brings up a different mystery.14:03
Tengutkajinam: the content-provider is relying on some weird things, and the patches you and I are pushing don't hit that specific thing I guess.14:04
Tengutkajinam: so pinning is needed in order to get things in, and we'll rely on testproject in order to check the revert of the pinning, once what we think fixes are in gerrit14:04
tkajinamTengu, yeah14:10
tkajinamTengu, we probably end up testing image build locally to find out what is causing the whole stuck14:10
Tengutkajinam: I have a local test.14:11
Tenguand... it's fun :)14:11
Tengureally14:11
tkajinamTengu, nice !14:11
Tenguit's just stuck.14:11
TenguI mean.... NOTHING is doing anything apparently14:11
tkajinamhmm14:11
Tenguwant to jump in?14:11
Tengutkajinam: if so, I'll need your ssh pub key14:12
tkajinamTengu, https://github.com/kajinamit.keys14:12
Tengu2s, adding it to the right places.14:14
*** ysandeep is now known as ysandeep|dinner14:41
*** pojadhav is now known as pojadhav|dinner14:42
*** soniya29|rover is now known as soniya29|rover|out14:47
opendevreviewRabi Mishra proposed openstack/tripleo-ansible master: WIP Drop ansible-core pin from requirements  https://review.opendev.org/c/openstack/tripleo-ansible/+/86446814:49
*** yadnesh is now known as yadnesh|away14:58
*** ysandeep|dinner is now known as ysandeep15:00
opendevreviewJiri Podivin proposed openstack/tripleo-validations master: Refactoring default-node-count validation to use openstack-collection modules  https://review.opendev.org/c/openstack/tripleo-validations/+/86442115:03
opendevreviewJiri Podivin proposed openstack/tripleo-validations stable/wallaby: Refactoring node-health validation  https://review.opendev.org/c/openstack/tripleo-validations/+/86455515:05
opendevreviewJiri Podivin proposed openstack/tripleo-validations stable/zed: Add sysctl validation for undercloud preflight checks  https://review.opendev.org/c/openstack/tripleo-validations/+/86455715:08
opendevreviewJiri Podivin proposed openstack/tripleo-validations stable/zed: Removing undercloud-tokenflush validation and related role  https://review.opendev.org/c/openstack/tripleo-validations/+/86455815:08
opendevreviewJiri Podivin proposed openstack/tripleo-validations stable/zed: Refactoring ironic-boot-configuration validation to use openstack collection modules  https://review.opendev.org/c/openstack/tripleo-validations/+/86455915:09
opendevreviewJiri Podivin proposed openstack/tripleo-validations stable/wallaby: Removing undercloud-tokenflush validation and related role  https://review.opendev.org/c/openstack/tripleo-validations/+/86456015:11
opendevreviewJiri Podivin proposed openstack/tripleo-validations stable/wallaby: Refactoring ironic-boot-configuration validation to use openstack collection modules  https://review.opendev.org/c/openstack/tripleo-validations/+/86456115:12
jpodivinbackporting ... backporting never changes.15:13
opendevreviewJiri Podivin proposed openstack/tripleo-validations stable/wallaby: Add sysctl validation for undercloud preflight checks  https://review.opendev.org/c/openstack/tripleo-validations/+/86456215:14
opendevreviewTakashi Kajinami proposed openstack/tripleo-ansible master: Fix compatibility with Ansible 2.14  https://review.opendev.org/c/openstack/tripleo-ansible/+/86439215:24
*** pojadhav|dinner is now known as pojadhav15:31
*** akahat|ruck is now known as akahat|ruck|dinner15:31
opendevreviewMerged openstack/puppet-tripleo stable/train: [Train ONLY] remove if statement from local ca check Current change will not work reliably. Removing this if statement will just cause the script to extract and update the CA trust on each renewal  https://review.opendev.org/c/openstack/puppet-tripleo/+/86418615:33
*** ysandeep is now known as ysandeep|out16:02
opendevreviewBogdan Dobrelya proposed openstack/tripleo-common stable/wallaby: Support ansible inventory merging  https://review.opendev.org/c/openstack/tripleo-common/+/86189016:16
*** marios is now known as marios|out16:28
bogdandochandankumar: hi. do you recall any recent changes in CI around python3-openstacksdk python3-cryptography in centos 9 stream?17:13
bogdandoI'm fixing my not much recent molecule runner zuul node...17:14
bogdandoand it seems that some magic is needed around installing those17:14
bogdandopytester was complaining about broken appstream, so I ran it like pytest ../../../tests/test_molecule.py --ansible-args='-e test_deps_setup_tripleo=false -e test_deps_extra_packages=[]' (no packages install dances)17:16
bogdandothen I had to install galaxy from https://opendev.org/openstack/tripleo-quickstart/raw/branch/master/ansible-collection-requirements.yml  :)17:16
bogdandothen I had to update some paths in ~/src/opendev.org/openstack/tripleo-ansible/ansible-test-env.rc to make it seeing it17:16
bogdandoand now it wants openstacksdk, which cannot locate python3-cryptography in appstream repos17:17
bogdandosigh17:17
* bogdando doesn't want to give up and redeploy that zuul node from more recent molecule jobs :)17:17
bogdandoakahat|ruck|dinner: ^^ perchance17:18
bogdandoI think I fixed that tripleo-centos-appstream breakage (by doing sudo tripleo-repos -b master tripleo-ci-testing), will try to run molecule with defaults17:21
bogdandonope, it wants ansible collections and openstacksdk again17:22
bogdandohttps://review.opendev.org/q/I646cd49b614a9f494e77f9893521d3c070900ed5 looks related17:24
bogdandosoo.. the "workaround" was sudo pip install openstacksdk==0.61.0 :)17:40
bogdandoit's fine for my zuul node, I don't care17:40
bogdandobackground, was trying to dig out produced keystone endpoints IDs from tripleo_keystone_resources17:41
*** jpena is now known as jpena|off17:42
*** akahat|ruck|dinner is now known as akahat|ruck18:31
*** amoralej is now known as amoralej|off19:52
*** rcastillo|rover is now known as rcastillo21:55
*** dasm is now known as dasm|off23:02

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!