| zbr | marios: it seems that there are two official/default locations for modules configured module search path = ['~/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules'] | 06:40 |
|---|---|---|
| zbr | first one is user specific, so we probably do not want to use it when installing as root or other ansible users will fail to find that module. | 06:41 |
| zbr | now i need to see how pbr is doing its dark magic with combinations of root/non-root, venv/non-venv/--user, editable/non-editable | 06:42 |
| marios | zbr: also /usr/local | 06:44 |
| marios | when pip | 06:44 |
| marios | zbr: that is the one being discussed on your review | 06:45 |
| zbr | i checked ansible default config on fedora 34, reports only those two. | 06:45 |
| marios | zbr: i just commented that's what i'm adding in my review | 06:45 |
| marios | zbr: in fact | 06:45 |
| marios | zbr: not just pip | 06:45 |
| marios | zbr: on my local install which i did with sudo python setup.py install i got it in /usr/local/share/ansible/plugins/modules/tripleo_get_hash.py | 06:46 |
| marios | zbr: fedora 34 | 06:46 |
| zbr | to just make the matter even more interesting, pip reports a warning during install pointing to this ticket: https://github.com/pypa/pip/issues/9617 | 06:47 |
| zbr | read the ticket well, it also applies to us, not only debian | 06:49 |
| marios | zbr: k will check in bit | 06:49 |
| zbr | yep, is bit long, still digesting it | 06:49 |
| *** amoralej|off is now known as amoralej | 06:51 | |
| -opendevstatus- NOTICE: Due to a configuration error unfortunately the Zuul queue was lost. Please recheck any in-flight changes | 06:55 | |
| zbr | my take on this is: we should stop installing datafiles at all, atempting to install ansible modules using pip guarantees future pain. Probably the only acceptable workaround for this would be to provide a CLI command that installs the modules, something that we would run after installing the python package. | 06:55 |
| zbr | current behavior relies on deprecated distutils which is getting removed,... read https://github.com/pypa/pip/issues/9617 | 06:56 |
| zbr | also points to a SO post | 06:56 |
| zbr | sorry https://github.com/pypa/pip/issues/9999 | 06:57 |
| zbr | that one short. | 06:57 |
| zbr | shortly, ansible default module imports are incompatible with pip installs, as pip will put still in /usr/local/ for root and ~/.local/ for non root, none of them is used by ansible. | 07:01 |
| zbr | in that case, i am inclined to provide a CLI `tripleo-repos --install-ansible-modules` that installs the modules to a location recognized by ansible, avoiding the need to perform manual actions. | 07:02 |
| zbr | for collection that is not even needed, in fact we could just install the collection. | 07:04 |
| zbr | WDYT about avoiding that bit and just going for the collection approach? | 07:05 |
| zbr | marios: considering that there is no direction action to perform on triple-repos, i am inclining to +W the change, anything against? | 07:37 |
| marios | zbr: well it breaks my changes for starters ;) | 07:44 |
| marios | zbr: we said they're independent but obv they are not https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/793331/14/playbooks/tasks/install-tripleo-get-hash.yml#7 | 07:45 |
| marios | zbr: i will have to rework mine and hope it works on collections otherwise i will hunt you down | 07:45 |
| marios | :D | 07:45 |
| zbr | marios: not really breaking you change, in its current form module gets installed into location expected by your change. | 07:50 |
| zbr | ahh, now I see it, you only need to change the chdir line. | 07:51 |
| marios | zbr: k anyway am chasing an issue so i am not blocking your change i already +1 there | 07:51 |
| marios | zbr: i will update the installation after your change merges | 07:51 |
| zbr | okey! | 07:51 |
| marios | zbr: if it doesn't work i will order a drone strike on you from the dark web | 07:52 |
| zbr | if you manage to launch it from a dingy you may be succesfull, apparently with that weather we got a record number of people coming from france | 07:53 |
| marios | zbr: no i always prefer from geostationary orbit, i mean you pay a little extra but imho is worth it | 07:54 |
| zbr | you and bezos | 07:54 |
| pojadhav | sshnaidm, hi | 08:17 |
| sshnaidm | pojadhav, o/ | 08:18 |
| pojadhav | sshnaidm, do you have access to production cockpit server? | 08:19 |
| sshnaidm | pojadhav, yes | 08:19 |
| pojadhav | actually from last 2-3 days upstream gate failing jobs panel is not working from cockpit page | 08:19 |
| pojadhav | http://dashboard-ci.tripleo.org/d/Z4vLSmOGk/cockpit?orgId=1 | 08:20 |
| sshnaidm | pojadhav, do you see failing jobs that are not there? | 08:21 |
| pojadhav | sshnaidm, i cant see anything there (not a single job in that panel..) | 08:22 |
| sshnaidm | pojadhav, I can see a few, but this seems outdated.. | 08:23 |
| sshnaidm | pojadhav, will look, seems like it's stuck on something | 08:23 |
| pojadhav | sshnaidm, now i can see some jobs those are skipped | 08:23 |
| pojadhav | sshnaidm, thank you ! | 08:24 |
| sshnaidm | pojadhav, https://paste.opendev.org/show/807616/ | 08:27 |
| slaweq | chkumar|rover: hi | 08:28 |
| chkumar|rover | slaweq: hello | 08:28 |
| slaweq | chkumar|rover: regarding https://trello.com/c/Ak6LJF2H - can we maybe blacklist that failing test for now? I will need to investigate it more and it will take some time for me for sure | 08:28 |
| chkumar|rover | slaweq: checking | 08:29 |
| chkumar|rover | slaweq: https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/801290 already done | 08:32 |
| slaweq | chkumar|rover: thx a lot | 08:32 |
| sshnaidm | pojadhav, we need to publish logs of telegraf somewhere, maybe a good pet project | 08:32 |
| sshnaidm | so we'll know w/o going to host what's the problem | 08:33 |
| *** ykarel is now known as ykarel|lunch | 08:33 | |
| pojadhav | sshnaidm, yeah.. you are right | 08:33 |
| pojadhav | sshnaidm, will try to work around it for logs.. | 08:33 |
| sshnaidm | cool | 08:34 |
| pojadhav | chkumar|rover, https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/34615 will fix issue about cockpit | 08:34 |
| pojadhav | but once this patch merges, sshnaidm i think cockpit service needs to restart right..? | 08:35 |
| sshnaidm | pojadhav, actually it should rebuild and restart by itself, let's see if it work | 08:35 |
| sshnaidm | take around 5-10 mins after a merge | 08:35 |
| pojadhav | sshnaidm, sure | 08:35 |
| sshnaidm | is not, I'll restart | 08:35 |
| pojadhav | thanks :) | 08:36 |
| sshnaidm | now it's interesting how it worked before.. | 08:36 |
| pojadhav | yeah :D | 08:38 |
| sshnaidm | this line wasn't touched by years | 08:40 |
| sshnaidm | oh, I see, this condition never evaluated before: "if build['end_time'] is None " | 08:41 |
| sshnaidm | so it always was broken there, now for some reason build['end_time'] is None | 08:42 |
| chkumar|rover | sshnaidm: can we also remove skipped jobs shown in cockpit in upstream queue failure? | 08:42 |
| sshnaidm | yeah, of course | 08:43 |
| * pojadhav stepping out 1-2 hrs | 08:43 | |
| slaweq | hi ooo admins, can someone put node on which CI job https://review.rdoproject.org/zuul/stream/901f5d29cd904c2abdeff517428dc58d?logfile=console.log is running on hold for me? I need to debug one kvm related issue there | 09:00 |
| slaweq | it's related to one cix card | 09:00 |
| slaweq | thx in advance | 09:00 |
| slaweq | chkumar|rover: ^^ maybe You can do that for me? | 09:00 |
| chkumar|rover | ykarel|lunch: ^^ can you hold above job? | 09:01 |
| chkumar|rover | slaweq: can you pass the review link? | 09:01 |
| slaweq | chkumar|rover: sure https://review.rdoproject.org/r/c/testproject/+/32157 | 09:01 |
| chkumar|rover | and pub key | 09:01 |
| slaweq | and job is periodic-tripleo-ci-centos-8-scenario010-kvm-standalone-master | 09:01 |
| slaweq | chkumar|rover: my pub key https://paste.opendev.org/show/807617/ | 09:02 |
| chkumar|rover | ykarel|lunch: ^^ | 09:02 |
| soniya | arxcruz, kopecmartin, chkumar|rover, do you have anything to discuss for the today's tempest meeting? | 09:29 |
| arxcruz | not really | 09:29 |
| soniya | kopecmartin, ^^ | 09:31 |
| *** ykarel|lunch is now known as ykarel | 09:46 | |
| ykarel | chkumar|rover, doing | 09:46 |
| ykarel | slaweq, ssh zuul@38.102.83.65 | 09:50 |
| slaweq | ykarel: chkumar|rover thx a lot | 09:51 |
| slaweq | I will ping You again once I will finish there | 09:51 |
| ykarel | ack | 09:51 |
| frenzy_friday | Do we have the retrospective today? | 09:55 |
| sshnaidm | frenzy_friday, I think depends on rlandy | 10:04 |
| sshnaidm | pojadhav, seems like with current fix we can overload the db, not sure it was always the situation, but now it creates a db record for every script run | 10:07 |
| sshnaidm | pojadhav, we need to rethink how we do it, maybe better to include it in sprint | 10:08 |
| sshnaidm | otherwise influx db will crash completely | 10:08 |
| marios | votes please when you have time criteria upgrade jobs https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/34598/1#message-d76b2bb6d2347c0a4c8bf2c154fdef056f5ee8a5 | 10:16 |
| slaweq | ykarel: ok, thx a lot. I'm done with that node | 10:29 |
| slaweq | You can clean it if needed | 10:29 |
| ykarel | slaweq, deleting | 10:34 |
| ykarel | done | 10:34 |
| slaweq | ykarel++ thx once again | 10:59 |
| sshnaidm | chkumar|rover, marios pojadhav please review to prevent influx to blow up: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/34616 | 11:21 |
| marios | ack added to list sshnaidm | 11:21 |
| soniya | arxcruz, kopecmartin, chkumar|rover, I am cancelling the today's meeting due to lack of agenda. | 11:44 |
| frenzy_friday | Hey rlandy , do we have the retrospective today? | 11:45 |
| rlandy | frenzy_friday: I don't have one on my calendar | 11:46 |
| rlandy | let me check | 11:46 |
| rlandy | we should | 11:46 |
| arxcruz | ok | 11:46 |
| rlandy | frenzy_friday: booking one from 12:30 to 1:30 UTC | 11:48 |
| frenzy_friday | rlandy, thanks. Here is the rtro board https://miro.com/app/board/o9J_l6ZW-vk=/ | 11:49 |
| rlandy | chkumar|rover: hey | 11:53 |
| chkumar|rover | rlandy: hello | 11:53 |
| rlandy | chkumar|rover: bhagyashris is out? | 11:53 |
| chkumar|rover | rlandy: yes she is sick | 11:53 |
| rlandy | zbr: sshnaidm: pojadhav: akahat: frenzy_friday: soniya: chkumar|rover: dviroel: marios: arxcruz: sorry for the late noticed | 11:56 |
| rlandy | booked retro in 35 mins from now | 11:56 |
| rlandy | pls see meeting invite | 11:56 |
| marios | rlandy: ack | 11:56 |
| rlandy | retro board is attached | 11:56 |
| rlandy | chkumar|rover: how is life in ruck/rover land? | 11:56 |
| chkumar|rover | rlandy: there is not much disaster this time, we have multiple frequent promotion on all branches | 11:57 |
| rlandy | good | 11:57 |
| chkumar|rover | update jobs failed multiple times pulling old hash | 11:57 |
| chkumar|rover | leading to develop mirror sync script https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/34441 | 11:58 |
| rlandy | chkumar|rover: did you see comment on failing security jobs re: old containers on quay? | 11:58 |
| rlandy | https://bugs.launchpad.net/tripleo/+bug/1936776/comments/6 "quay.io/tripleomaster containers are three days old" | 11:58 |
| chkumar|rover | rlandy: looking | 12:00 |
| chkumar|rover | joining program call | 12:00 |
| rlandy | chkumar|rover: thanks - /home/zuul/validations' does not exist. - that still an issue? https://trello.com/c/qYpG6ueT/2025-cixlp1936218tripleociproamaster-comp-pipeline-network-selected-log-directory-home-zuul-validations-does-not-exist-attempting-to | 12:10 |
| chkumar|rover | rlandy: we need a master promotion in order to fix above job https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-standalone-network-master-validation | 12:12 |
| rlandy | ok | 12:12 |
| chkumar|rover | which is blocked on https://bugs.launchpad.net/tripleo/+bug/1936776/ ipa job failure | 12:12 |
| rlandy | chkumar|rover++ for handling queues alone | 12:13 |
| rlandy | chkumar|rover: k - was a fix proposed there | 12:13 |
| chkumar|rover | rlandy: I think we can remove this https://bugs.launchpad.net/tripleo/+bug/1934994 bug from program call | 12:13 |
| rlandy | ok | 12:14 |
| * rlandy checks if it passed | 12:14 | |
| chkumar|rover | it seems to be an infra issue | 12:14 |
| chkumar|rover | https://code.engineering.redhat.com/gerrit/c/testproject/+/256435 | 12:14 |
| rlandy | chkumar|rover: go ahead | 12:14 |
| chkumar|rover | rlandy: please take a look this issue https://bugs.launchpad.net/tripleo/+bug/1936287/comments/6 | 12:16 |
| chkumar|rover | overcloud-full issue on ussuri also | 12:17 |
| *** rlandy is now known as rlandy|ruck | 12:17 | |
| rlandy|ruck | chkumar|rover: k - I'll look at if we should remove ussuri from operators | 12:28 |
| rlandy|ruck | chkumar|rover: sshnaidm: zbr: https://meet.google.com/cmb-czwr-zah?authuser=0 retro is going on now | 12:31 |
| frenzy_friday | https://miro.com/app/board/o9J_l6ZW-vk=/ | 12:32 |
| *** amoralej is now known as amoralej|lunch | 12:41 | |
| pojadhav | marios, chkumar|rover, sshnaidm, rlandy|ruck zbr : please review - https://review.opendev.org/q/topic:%2522wallaby-upgrade-jobs%2522+(status:open+OR+status:merged)+owner:+pojadhav | 12:51 |
| marios | pojadhav: ack adding to reviews queue | 12:54 |
| pojadhav | marios, thanks! | 12:55 |
| chkumar|rover | https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/34441 | 13:03 |
| chkumar|rover | marios: soniya you are the next ruck rover | 13:08 |
| chkumar|rover | marios: soniya will do the handoff tomorrow | 13:08 |
| soniya | chkumar|rover, okay, no problem :) | 13:08 |
| rlandy|ruck | chkumar|rover: ^^ will join that meeting if it's not too early | 13:09 |
| rlandy|ruck | chkumar|rover: so you think https://bugs.launchpad.net/tripleo/+bug/1934994/comments/7 is not related to the bug? | 13:10 |
| chkumar|rover | rlandy|ruck: will try to keep it before tomorrow's planning if marios and soniya ok ? | 13:10 |
| rlandy|ruck | watching your rerun | 13:10 |
| soniya | chkumar|rover, I am okay | 13:10 |
| rlandy|ruck | ok | 13:10 |
| *** pojadhav is now known as pojadhav|afk | 13:10 | |
| rlandy|ruck | will catch up with marios and soniya afterwards | 13:10 |
| rlandy|ruck | since will be covering us time | 13:10 |
| chkumar|rover | rlandy|ruck: because the issue vanished in component line without any fixes | 13:10 |
| rlandy|ruck | chkumar|rover: so we're just closing that bug? | 13:11 |
| chkumar|rover | rlandy|ruck: yes, but after rhos-17 run | 13:11 |
| rlandy|ruck | chkumar|rover: ack - thanks for rechecking | 13:11 |
| rlandy|ruck | chkumar|rover: wrt https://bugs.launchpad.net/tripleo/+bug/1936287/comments/6 - I have a second patch from Rabi's suggestion | 13:20 |
| rlandy|ruck | whcih wasn't merged yet | 13:20 |
| rlandy|ruck | will try that one with ussuri | 13:20 |
| rlandy|ruck | maybe that will deal with the issue | 13:20 |
| chkumar|rover | rlandy|ruck: sure thanks! | 13:21 |
| chkumar|rover | rlandy|ruck: https://sf.hosted.upshift.rdu2.redhat.com/logs/35/256435/1/check/periodic-tripleo-ci-rhel-8-scenario010-standalone-network-rhos-17/f20f23d/logs/undercloud/var/log/tempest/tempest_run.log | 13:31 |
| chkumar|rover | rlandy|ruck: this time errors are different | 13:31 |
| chkumar|rover | rlandy|ruck: Load Balancer 9f999370-1a1c-455b-acdd-118ba8cea812 is immutable and cannot be updated | 13:31 |
| rlandy|ruck | so it's the same failing test with a different problem | 13:31 |
| chkumar|rover | rlandy|ruck: https://bugs.launchpad.net/tripleo/+bug/1901996 | 13:32 |
| chkumar|rover | similar bug with above issue | 13:32 |
| rlandy|ruck | ages ago | 13:33 |
| rlandy|ruck | will look into it again later | 13:33 |
| *** amoralej|lunch is now known as amoralej | 13:33 | |
| *** rlandy|ruck is now known as rlandy|ruck|mtg | 13:33 | |
| marios | chkumar|rover: bhagyashris: we can sync tomorrow morning on ruck|rover | 13:34 |
| *** rlandy is now known as rlandy|ruck | 13:35 | |
| marios | rlandy|ruck: chkumar|rover: i will join cix call just to listen to issues - one of you will cover yes? | 14:05 |
| rlandy|ruck | marios: ack - will be there | 14:05 |
| marios | rlandy|ruck: chkumar|rover: me and chkumar|rover and bhagyashris and soniya can catch up tomorrow our morning | 14:05 |
| marios | and i will catchup witn yhou tomorrow rlandy|ruck | 14:05 |
| marios | rlandy|ruck: i'll check google chat if you want to leav anything urgent there | 14:05 |
| rlandy|ruck | marios: thanks | 14:06 |
| rlandy|ruck | bhagyashris? is she also helping with ruck/rover? | 14:06 |
| rlandy|ruck | oh - she was last time's | 14:07 |
| marios | rlandy|ruck: i thought she was rover? or ruck this sprint? | 14:07 |
| rlandy|ruck | got it :) | 14:07 |
| chkumar|rover | marios: invite sent | 14:10 |
| marios | chkumar|rover: thanks sounds good | 14:10 |
| zbr | marios: basically alex wanted to revert due to the use of data_files trick for copying ansible modules, i am glad I am on the same page with him, a dirty hack to avoid and we know how. | 14:17 |
| marios | rlandy|ruck: chkumar|rover: cix call | 14:31 |
| rlandy|ruck | zbr; marios: dviroel: arxcruz: ^^ let's pick this up in planning | 14:32 |
| dviroel | +! | 14:35 |
| dviroel | +1 | 14:35 |
| arxcruz | ok | 14:36 |
| chkumar|rover | marios: almost all the issues on CIX board. | 14:42 |
| chkumar|rover | changing my nick now | 14:42 |
| *** chkumar|rover is now known as chandankumar | 14:42 | |
| chandankumar | rlandy|ruck: bhagyashris thanks for helping this sprint. :-) | 14:43 |
| rlandy|ruck | chandankumar: sure - anything I need to follow up on this afternoon? | 14:43 |
| chandankumar | rlandy|ruck: overcloud-full ussuri issue | 14:44 |
| rlandy|ruck | chandankumar: k - train - centos7 hasn't promoted in two months | 14:44 |
| chandankumar | rlandy|ruck: Ipa patch testing | 14:44 |
| rlandy|ruck | are we just leaving that? | 14:44 |
| rlandy|ruck | ack on ipa | 14:44 |
| chandankumar | rlandy|ruck: yes no promotion for train | 14:44 |
| chandankumar | rlandy|ruck: need to check upstream where we have any c7 train voting jobs | 14:45 |
| rlandy|ruck | chandankumar: ok - so just ussuri that really needs to promote | 14:45 |
| chandankumar | and then work on that | 14:45 |
| chandankumar | rlandy|ruck: yes | 14:45 |
| rlandy|ruck | chandankumar: ^^? meaning | 14:45 |
| rlandy|ruck | get rid of c7 voting jobs? | 14:45 |
| chandankumar | let me check the status | 14:45 |
| chandankumar | rlandy|ruck: we have one content provider and containers multinode job on c7 train running upstream | 14:48 |
| chandankumar | https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-content-provider&job_name=tripleo-ci-centos-7-containers-multinode | 14:48 |
| chandankumar | both looks green | 14:48 |
| rlandy|ruck | k - let's add that to the ruck-rover tasks | 14:49 |
| chandankumar | if we have enough promotion for c8, we can check c7 train promotions | 14:49 |
| chandankumar | rlandy|ruck: marios logging out now, see ya tomorrow. | 14:53 |
| rlandy|ruck | chandankumar: have a good night | 14:55 |
| marios | thanks chandankumar enjoy your evening | 15:03 |
| rlandy|ruck | pojadhav|afk: sshnaidm: thanks for fixing cockpit gate tracking | 15:03 |
| *** ykarel is now known as ykarel|away | 15:07 | |
| rlandy|ruck | frenzy_friday: hi - want to meet now? have an open slot | 15:20 |
| frenzy_friday | rlandy|ruck, sorry, just checked. Do you have time now? | 15:52 |
| rlandy|ruck | frenzy_friday: np - let's keep the regular time | 15:52 |
| rlandy|ruck | just rekicking failed jobs | 15:52 |
| frenzy_friday | rlandy|ruck, ok, thanks | 15:52 |
| *** beagles is now known as eagles | 16:05 | |
| *** amoralej is now known as amoralej|off | 16:22 | |
| *** njohnston_ is now known as njohnston | 16:27 | |
| *** marios is now known as marios|out | 16:31 | |
| frenzy_friday | rlandy|ruck, os-tripleo-ci user needs to sign https://review.opendev.org/settings/new-agreement before it can be used. Shall I sign it? | 16:46 |
| rlandy|ruck | frenzy_friday: just the top one | 16:47 |
| rlandy|ruck | I guess so | 16:47 |
| rlandy|ruck | frenzy_friday: select ICLA | 16:48 |
| rlandy|ruck | should not be a problem | 16:48 |
| frenzy_friday | ok | 16:49 |
| rlandy|ruck | frenzy_friday: is anything else needed? | 16:49 |
| frenzy_friday | no I think. me checks | 16:49 |
| rlandy|ruck | sshnaidm: ^^ any concerns? | 16:49 |
| rlandy|ruck | it is used in reproducer | 16:49 |
| rlandy|ruck | per infar doc | 16:49 |
| rlandy|ruck | infra | 16:49 |
| sshnaidm | rlandy|ruck, no, go ahead | 16:51 |
| rlandy|ruck | thanks | 16:51 |
| rlandy|ruck | frenzy_friday: ^^ all good | 16:51 |
| rlandy|ruck | lunch - brb | 17:02 |
| frenzy_friday | rlandy|ruck, auth in make build is working now. All keys in the vm now belong to os-tripleo-ci . Now I am getting toomanyrequests: You have reached your pull rate limit. Could you pls add the RH account there when you get some time? It is on the same tmux | 17:06 |
| frenzy_friday | I tried the docker login from our infra doc already | 17:09 |
| rlandy|ruck | frenzy_friday: we should try get back to vexx | 17:10 |
| rlandy|ruck | will look at it | 17:10 |
| frenzy_friday | rlandy|ruck, ok, I'll create a vm on vexx and try there | 17:10 |
| frenzy_friday | ok, it works now, I was using the wrong creds | 17:12 |
| rlandy|ruck | frenzy_friday: on the internal vm? | 17:20 |
| *** rlandy|ruck is now known as rlandy|ruck|bbl | 22:36 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!