Wednesday, 2021-09-01

rdogerritrdo-trunk proposed rdoinfo master: Bump rdoinfo xena-uc tags to latest upper-constraints  https://review.rdoproject.org/r/c/rdoinfo/+/3517600:12
rdogerritrdo-trunk created rdoinfo master: Promote CBS tags update for ussuri-8-testing  https://review.rdoproject.org/r/c/rdoinfo/+/3522300:14
rdogerritrdo-trunk proposed rdoinfo master: Promote CBS tags update for wallaby-8s-testing  https://review.rdoproject.org/r/c/rdoinfo/+/3518800:16
rdogerritrdo-trunk created rdoinfo master: Promote CBS tags update for ussuri-8s-testing  https://review.rdoproject.org/r/c/rdoinfo/+/3522400:17
*** ysandeep|away is now known as ysandeep00:59
rdogerritMerged rdoinfo master: Promote CBS tags update for ussuri-8-testing  https://review.rdoproject.org/r/c/rdoinfo/+/3522305:51
rdogerritMerged rdoinfo master: Promote CBS tags update for wallaby-8s-testing  https://review.rdoproject.org/r/c/rdoinfo/+/3518805:53
rdogerritMerged rdoinfo master: Promote CBS tags update for ussuri-8s-testing  https://review.rdoproject.org/r/c/rdoinfo/+/3522405:53
rdogerritMerged openstack/barbican-distgit ussuri-rdo: openstack-barbican-10.1.0-1  https://review.rdoproject.org/r/c/openstack/barbican-distgit/+/3522205:57
rdogerritMerged openstack/heat-distgit ussuri-rdo: openstack-heat-14.2.0-1  https://review.rdoproject.org/r/c/openstack/heat-distgit/+/3520606:00
*** ykarel is now known as ykarel|away06:07
*** jpena|off is now known as jpena07:03
*** amoralej|off is now known as amoralej07:05
rdogerritChandan Kumar created config master: Use tripleo_podman role for EL8 distros  https://review.rdoproject.org/r/c/config/+/3522507:17
rdogerritMerged config master: Use tripleo_podman role for EL8 distros  https://review.rdoproject.org/r/c/config/+/3522507:24
*** arxcruz is now known as arxcruz|training08:16
*** ysandeep is now known as ysandeep|lunch08:26
rdogerritJavier Peña created openstack/networking-bgpvpn-distgit rpm-master: Remove the networking-bagpipe requirement  https://review.rdoproject.org/r/c/openstack/networking-bgpvpn-distgit/+/3522608:36
rdogerritrdo-trunk created openstack/keystone-distgit ussuri-rdo: openstack-keystone-17.0.1-1  https://review.rdoproject.org/r/c/openstack/keystone-distgit/+/3522708:51
rdogerritrdo-trunk created openstack/swift-distgit ussuri-rdo: openstack-swift-2.25.2-1  https://review.rdoproject.org/r/c/openstack/swift-distgit/+/3522808:51
rdogerritMerged openstack/os-brick-distgit rpm-master: Requirements sync for Xena  https://review.rdoproject.org/r/c/openstack/os-brick-distgit/+/3514908:56
rdogerritMerged openstack/ironic-lib-distgit rpm-master: Requirements sync for Xena  https://review.rdoproject.org/r/c/openstack/ironic-lib-distgit/+/3514808:57
*** ysandeep|lunch is now known as ysandeep09:00
opendevreviewAlfredo Moralejo proposed x/packstack master: DNM test with testing repo added  https://review.opendev.org/c/x/packstack/+/75061409:04
chandankumarjpena: Tengu sshnaidm|afk amoralej here comes the c9 container images https://registry.rdoproject.org:8443/oapi/v1/namespaces/tripleomastercentos9/imagestreamtags/09:07
Tenguchandankumar: wooohoooo09:07
sshnaidm|afk\o/09:08
Tenguso we "just" need to get tripleo-repos09:08
*** sshnaidm|afk is now known as sshnaidm09:08
chandankumarjpena: the current node centos stream 9 image has selinux enforcing mode can we change it to permissive09:08
amoralejchandankumar, nice!09:08
sshnaidmchandankumar+++09:08
Tengunooooo keep it enforcing please :)09:08
* Tengu hides09:08
chandankumarTengu: in pre playbook https://opendev.org/openstack/tripleo-ansible/src/branch/master/tripleo_ansible/roles/tripleo_podman/tasks/tripleo_podman_install.yml#L46 hangs and fails the task09:09
chandankumarI am not sure it is related to selinux settype09:09
Tenguchandankumar: no reason... ? there's no setype set?09:10
chandankumarTengu: for current Job I have skipped that part https://review.rdoproject.org/r/c/config/+/3522509:10
chandankumardue to no_log we donot know anything09:10
chandankumarI tried to run the playbook locally on the node09:10
chandankumarit works fine09:10
Tenguin permissive?09:11
chandankumarin enforcing mode by holding the node09:11
Tenguah, ok. sooo.... not SELinux related?09:11
chandankumarTengu: not sure09:11
chandankumarsomething is cooking there which hangs that task09:11
Tenguwell, the audit.log isn't affected by the nolog set in ansible09:12
Tenguchandankumar: THAT task line 46? I don't see how pushing a *file* can do anything09:12
Tenguespecially hanging09:12
sshnaidmchandankumar, is there a short tag? I see only eec16ab614f9b19a00a463a1f55e8d22_x86_6409:12
chandankumarTengu: https://logserver.rdoproject.org/53/18953/97/check/tripleo-build-containers-stream9-development/85d2afd/job-output.txt09:13
chandankumarTengu:  look for 'Update default network configuration if possible'09:14
chandankumarsshnaidm: let me check the logs09:15
chandankumarsshnaidm: https://logserver.rdoproject.org/53/18953/99/check/tripleo-build-containers-stream9-development/a0596b3/job-output.txt09:15
Tenguchandankumar: I... don't see it as failed?09:15
Tenguchandankumar: I'm more seeing dlrnapi failing on a 401 (UNAUTHORIZED) than anything?09:16
sshnaidmchandankumar, yep, maybe worth to tag them as "current-tripleo" for now09:17
chandankumarTengu: https://paste.opendev.org/raw/808494/09:17
chandankumarTengu: dlrnapi will be fixed09:17
chandankumarsshnaidm: will fix that09:17
Tenguchandankumar: ah, so the "Write containers registries.conf" never hit - is that it? isn't it due to the failed dlrnapi thing?09:18
chandankumarTengu: yes, it never hit09:18
chandankumarit is not related to dlrnapi thing09:19
Tenguchandankumar: I don't see what could prevent it from execute successfuly. it's just a lame "copy", nothing fancy, not even anything reloading a serice or whatsover..09:20
Tenguit doesn't even affect actual network connectivity09:20
chandankumarI am also confused09:20
chandankumarwhat is happening there09:20
Tenguand with the lack of setype or anything selinux related......... I don't see how "permissive" would help09:20
Tenguchandankumar: ah, we don't have any system logs?09:21
Tenguah, YES. we do09:21
Tenguhttps://logserver.rdoproject.org/53/18953/99/check/tripleo-build-containers-stream9-development/a0596b3/logs/system/audit/09:21
chandankumarhttps://logserver.rdoproject.org/53/18953/97/check/tripleo-build-containers-stream9-development/85d2afd/logs/system/09:21
Tengucome here, you nasty audit.log09:21
* chandankumar will be back in an hour09:22
Tenguchandankumar: it's permissive.09:22
Tenguhttps://paste.opendev.org/show/808496/09:22
Tengu"permissive=1"09:22
Tenguso definitively NOT selinux related.09:22
Tengueven though those denied writes will hit OSP sooner or later imho.09:23
Tenguthough... dracut. not on us.09:23
Tenguchandankumar: hmm... I still don't see anything on the system logs that might explain the issue...09:47
TenguI even see some steps for /home/zuul-worker/workspace creation and rsync09:47
Tengubut I think this is... yeah, log gathering.09:48
chandankumarTengu: Any more logs we can grab to look into that?09:55
Tenguchandankumar: tried to get all what I can, I don't see anything..... would it be possible to drop the nolog? iirc, we can set "nolog: false" at the task level.09:57
Tenguchandankumar: i.e. WIP tripleo-ansible change in order to activate logs on the tripleo_podman_install tasks09:58
Tenguthat would already help09:58
Tenguand since it's a block, we should be able to set it at the block level, as the "become"09:58
Tenguchandankumar: or, if you have a hold node with the failed thing, maybe I can access?10:00
chandankumarTengu: I can get the node hold10:01
Tenguchandankumar: you'll need my ssh key I guess then10:01
chandankumarneed to make one change and push the containers10:01
Tenguchandankumar: ok! here's my key:  ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHUnwjB20UKmsSed9X73eGNV5AOEFccQ3NYrRW776pEk10:01
chandankumarthen I will get the node hold10:01
Tenguchandankumar: though I'll be out for lunch in ~10 minutes, back in ~1h, then jumping from call to call until-errr... 3pm CET10:01
Tenguchandankumar: so the window is small, but we might get a hit anyway.10:02
chandankumarTengu: will get before that10:02
Tengugood10:02
apevec[m]mrunge: in this first set of OOO CS9 containers we skip collectd, do you have maybe Copr CS9 builds until CBS is CS9-ready?10:03
mrungeapevec[m]: I didn't had the time to look at that10:04
mrungeapevec[m]: we may want to build collectd together with the rest of RDO?10:04
apevec[m]amoralej: ^ wdyt?10:04
apevec[m]in your Copr?10:05
amoralejchecking10:05
amoralejwe don't have copr yet, i have a mock server10:05
amoralejbut i could make a try10:05
chandankumarsshnaidm: I think we can pull the container via aggregate hash as a tag10:06
chandankumarsshnaidm: Not sure about tagging with current-tripleo as it does not exists on https://trunk.rdoproject.org/centos9-master-preview/10:06
sshnaidmchandankumar, ack10:07
sshnaidmjust would be easier for various scripts..10:07
apevec[m]chandankumar: jpena could we symlink current-tripleo manually?10:08
chandankumarapevec[m]: sounds good10:08
apevec[m]with the hash you have built containers, which one was it?10:08
chandankumarapevec[m]: consistent10:08
apevec[m]that one might have moved?10:08
sshnaidmapevec[m], chandankumar eec16ab614f9b19a00a463a1f55e8d22_x86_6410:08
apevec[m]jpena: ^ let's tag this as current-tripleo10:09
jpenawith components, it's a bit more complicated than creating a simple symlink, but let me give it a try10:09
chandankumarsshnaidm: openstack-aodh-api:eec16ab614f9b19a00a463a1f55e8d22 -> https://trunk.rdoproject.org/centos9-master-preview/consistent/delorean.repo.md510:10
chandankumarand we have this also openstack-aodh-api:eec16ab614f9b19a00a463a1f55e8d22_x86_6410:10
chandankumaras metadata10:10
chandankumarhttps://registry.rdoproject.org:8443/oapi/v1/namespaces/tripleomastercentos9/imagestreamtags/10:10
Tenguchandankumar: going to lunch, back in ~45-60 minutes10:11
chandankumarcheck rawe data10:11
chandankumarTengu: enjoy :-)10:11
amoralejmrunge, collectd-5.12.0-8 is fine?10:18
amoralejrebuild from fedora10:18
rdogerritChandan Kumar created config master: Revert "Use tripleo_podman role for EL8 distros"  https://review.rdoproject.org/r/c/config/+/3515410:19
jpenachandankumar, apevec[m]: I've created the current-tripleo path for centos9-master-preview using the API. I couldn't do it directly on trunk.rdo because we're using a local sqlite3 DB on the builder, but I was able to do it on trunk-centos8 and then rsync10:28
Tenguchandankumar: back10:44
*** pojadhav is now known as pojadhav|brb10:46
rdogerritMerged config master: Revert "Use tripleo_podman role for EL8 distros"  https://review.rdoproject.org/r/c/config/+/3515410:49
chandankumarjpena: can you hold the node for this review https://review.rdoproject.org/r/c/testproject/+/18953 and add pub key ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHUnwjB20UKmsSed9X73eGNV5AOEFccQ3NYrRW776pEk and https://github.com/raukadah.keys10:51
jpenachandankumar: I'm in a meeting now, I'll do it as soon as I finish10:51
chandankumarjpena: sure10:51
Tenguah, that node's for me :)10:52
*** jcapitao is now known as jcapitao_lunch11:03
jpenachandankumar: so... the job ended up successfully11:05
jpenano hold then?11:05
Tenguo_O11:05
chandankumarjpena: I have some changes in config rpeo11:06
chandankumarjpena: it will fail after recheck11:06
jpenaah ok11:06
jpenaso let me set the autohold before rechecking11:06
jpenachandankumar, Tengu: done, you can ssh zuul-worker@38.102.83.22011:12
Tengujpena: thanks!11:13
Tenguchandankumar: installing tmux11:13
Tenguchandankumar: tmux installed, there's a session.11:13
chandankumarjpena: thanks!11:15
chandankumarTengu: issue reproduced as per console log11:15
Tenguchandankumar: care to link that log?11:15
chandankumarTengu: https://review.rdoproject.org/zuul/stream/387a5165b42f4cd998e39da368403c96?logfile=console.log11:16
Tenguchandankumar: well..... it shoulnd't hit that task at all: apparently, there isn't that "cni-podman0" interface.11:17
Tenguerr.... did the node just go away?11:18
chandankumarjpena: I think due to reattempt of job run, node vanishes11:19
Tenguchandankumar, jpena I lost connection to the node o_O11:19
chandankumarjpena: we need to hold again11:19
jpenaah, so it's a retry?11:19
chandankumarjpena: yes11:19
jpenadon't worry, let's wait until the last retry fails and I'll add the ssh keys back11:19
chandankumarTengu: https://opendev.org/openstack/tripleo-ansible/src/branch/master/tripleo_ansible/roles/tripleo_podman/tasks/tripleo_podman_install.yml#L5411:20
chandankumar"cni-podman0" not in ansible_facts["interfaces"]'11:20
chandankumardue to not in, it is going to run11:20
Tenguchandankumar: what I said..11:22
chandankumarsshnaidm: trying out standalone here https://review.rdoproject.org/r/c/testproject/+/3522911:22
Tenguso it should be a "skip" at this point.11:22
sshnaidmchandankumar, ack, but it requires a few hacks to work11:23
chandankumarsshnaidm: yup11:28
jpenaneed to go for lunch, I'll add the ssh keys when I'm back11:30
*** jpena is now known as jpena|lunch11:30
*** rlandy is now known as rlandy|rover11:38
mrungeamoralej: no. there is no amqp1 support in fedora, iirc11:39
mrungeamoralej: taking that back, amqp1 is there11:39
mrungehoweve, fedora and centos subpackage names differ here and there; may require subtle changes11:40
*** pojadhav|brb is now known as pojadhav11:43
rdogerritChandan Kumar created config master: Move no_log to task level  https://review.rdoproject.org/r/c/config/+/3523011:53
chandankumarbandini: hello, 11:57
chandankumarbandini: c9 tripleo containers are pushed on registry11:57
chandankumarbandini: https://review.rdoproject.org/zuul/build/00b27b17ce534b73b0ce6cf038fd5a2211:58
bandinichandankumar: I saw, thanks! I'll play with that in the next days11:58
rdogerritMartin Kopec proposed rdoinfo master: Bump tempest to 28.0.0 in Wallaby  https://review.rdoproject.org/r/c/rdoinfo/+/3491711:59
*** jcapitao_lunch is now known as jcapitao12:01
opendevreviewAlfredo Moralejo proposed x/packstack master: Move puppet-rsync to main branch  https://review.opendev.org/c/x/packstack/+/80691912:02
rdogerritChandan Kumar proposed config master: Move no_log to task level  https://review.rdoproject.org/r/c/config/+/3523012:02
opendevreviewAlfredo Moralejo proposed x/packstack master: DNM test with testing repo added  https://review.opendev.org/c/x/packstack/+/75061412:03
amoralejmrunge, i'll try to rebuild the cbs srpm in cs812:14
*** amoralej is now known as amoralej|lunch12:14
mrungethank you amoralej|lunch 12:15
*** jpena|lunch is now known as jpena12:30
jpenachandankumar, Tengu: I've added the ssh keys to the VM after the last retry, ssh  zuul-worker@38.102.83.8412:32
Tengujpena: after the pgm call :)12:34
*** amoralej|lunch is now known as amoralej13:08
slaweqrlandy|rover: hi13:14
slaweqon Monday CIX call You told me that You will write some comment in https://trello.com/c/ulHXnS4Y13:14
slaweqbut I don't see it anywhere :)13:14
rlandy|roverslaweq: yes - sorry - will update13:14
slaweqcan You help me to check how we can maybe force restart of the ovs in the CI nodes only13:15
slaweqso we don't have to do it in production13:15
slaweqand that would solve this issue13:15
slaweqI think it's the best way to fix it13:15
rdogerritAlfredo Moralejo created rdoinfo master: Switch puppet-rsync to main branch  https://review.rdoproject.org/r/c/rdoinfo/+/3523113:17
Tenguchandankumar: I'm back. Sorry for the delay, "some" blocker things to solve.13:20
chandankumarTengu: no problem13:20
Tenguchandankumar: are you on the host already?13:21
chandankumarnope, was in meetings13:21
Tenguchandankumar: ok - lemme install tmux13:21
Tenguthat will make sharing easier. Do you want a meet?13:21
Tenguah. in fact.... ansible is launched from another host?13:24
chandankumarTengu: ansible runs from the zuul executor13:24
Tenguchandankumar: ok - so that node has, basically, no ansible role installed.13:25
chandankumarTengu: you can find the roles there13:25
chandankumarmeet?13:25
Tenguchandankumar: yep13:25
Tenguhttps://meet.google.com/uqb-hdnw-aqr13:25
rdogerritMerged config master: Move no_log to task level  https://review.rdoproject.org/r/c/config/+/3523013:26
Tengujpena: node can be released, we'll work with -^^  mostly13:31
jpenaTengu: ack13:31
Tengujpena: it's a bit messy... the kind of issue I love for a Wednesday :)13:33
rlandy|roverslaweq: sorry about that  - will work on that today13:55
slaweqrlandy|rover: sure, np14:00
amoralejmeeting time14:01
amoralejyou can add your topics to https://etherpad.opendev.org/p/RDO-Meeting14:01
amoralejremember this week is video meeting in https://meet.google.com/uzo-tfkt-top 14:02
amoralej#startmeeting RDO meeting - 2021-09-0114:02
opendevmeetMeeting started Wed Sep  1 14:02:16 2021 UTC and is due to finish in 60 minutes.  The chair is amoralej. Information about MeetBot at http://wiki.debian.org/MeetBot.14:02
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.14:02
opendevmeetThe meeting name has been set to 'rdo_meeting___2021_09_01'14:02
jcapitaoo/ joining14:02
opendevreviewMerged x/packstack master: Move puppet-rsync to main branch  https://review.opendev.org/c/x/packstack/+/80691914:05
chandankumarTengu++14:11
amoralej#chair jcapitao spotz 14:11
opendevmeetCurrent chairs: amoralej jcapitao spotz14:11
amoralej#topic CentOS8 EoL question from the ML14:12
amoralej#link https://lists.rdoproject.org/pipermail/users/2021-August/001058.html14:12
Tenguchandankumar: apparently known: https://review.opendev.org/c/openstack/tripleo-ansible/+/80642814:14
amoralej#action jcapitao to do a basic test of migrationg a packstack deployment all-in-one from CL8 to CS814:17
amoralej#info based on the results of the tests we will write some recomendations in www.rdoproject.org14:17
amoralej#topic preparation for Xena GA14:18
amoralej#link https://trello.com/c/ofx2cKjc/760-xena-release-preparation14:18
amoralej#info started doing reqchecks for clients and libraries14:20
amoralej#topic centos9 update14:24
amoralej#info devstack can be executed manually on https://review.rdoproject.org/etherpad/p/c9-devstack14:24
amoralej#info cs9 cloud images available in https://composes.stream.centos.org/production/latest-CentOS-Stream/compose/BaseOS/x86_64/images/14:26
rlandy|roverslaweq: these were the original comments on the bug that was marked as duplicate: https://bugs.launchpad.net/tripleo/+bug/1941802/comments/014:30
rdogerritChandan Kumar created config master: Revert "Move no_log to task level"  https://review.rdoproject.org/r/c/config/+/3515514:34
amoralej#action jcapitao to chair next week14:35
amoralej#endmeeting14:36
opendevmeetMeeting ended Wed Sep  1 14:36:39 2021 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)14:36
opendevmeetMinutes:        https://meetings.opendev.org/meetings/rdo_meeting___2021_09_01/2021/rdo_meeting___2021_09_01.2021-09-01-14.02.html14:36
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/rdo_meeting___2021_09_01/2021/rdo_meeting___2021_09_01.2021-09-01-14.02.txt14:36
opendevmeetLog:            https://meetings.opendev.org/meetings/rdo_meeting___2021_09_01/2021/rdo_meeting___2021_09_01.2021-09-01-14.02.log.html14:36
slaweqrlandy|rover: so yes, I can confirm what You mentioned in that bug15:03
slaweqthe issue is similar15:03
rlandy|roverslaweq: ack - I am putting in a patch to restart openvswitch after update15:04
slaweqand the main problem is that in case of that failed job, openvswitch which is installed is 2.13 (that's correct version) but openvswitch which is actually running is 2.12 (which is not good)15:04
rlandy|roverand testing ussuri update with that15:04
slaweqrlandy|rover++ thx a lot15:04
rlandy|roverslaweq: sorry about the delay - lots of churn on the CI team atm15:05
slaweqrlandy|rover: no problem at all :)15:05
slaweqthx for help with that15:05
rdogerritMerged config master: Revert "Move no_log to task level"  https://review.rdoproject.org/r/c/config/+/3515515:13
*** chem is now known as Guest598915:16
*** jpena is now known as jpena|off16:01
*** marios is now known as marios|out16:04
*** ysandeep is now known as ysandeep|out16:15
*** amoralej is now known as amoralej|off16:15
rdogerritrdo-trunk created openstack/cinder-distgit wallaby-rdo: openstack-cinder-18.1.0-1  https://review.rdoproject.org/r/c/openstack/cinder-distgit/+/3523218:26
rdogerritrdo-trunk created rdoinfo master: CBS tags sync with uc for xena-candidate  https://review.rdoproject.org/r/c/rdoinfo/+/3523318:37
rdogerritrdo-trunk created openstack/octavia-distgit rpm-master: openstack-octavia: failed to build 5dd7ad9a  https://review.rdoproject.org/r/c/openstack/octavia-distgit/+/3523419:28
*** rlandy|rover is now known as rlandy|rover|mtg20:29
*** stevebaker_ is now known as stevebaker21:19
*** rlandy|rover|mtg is now known as rlandy|rover21:19
*** rlandy|rover is now known as rlandy|rover|bbl22:23
*** dviroel|ruck is now known as dviroel|out22:43

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!