| * dasm => offline | 01:10 | |
| *** dasm is now known as dasm|off | 01:10 | |
| *** soniya is now known as soniya29|rover | 05:41 | |
| marios | o/ | 05:42 |
|---|---|---|
| marios | arxcruz|ruck: soniya29|rover: bhagyashris: Tengu: rlandy_: just sent calendar invite for ruck|rover handoff half hour before scrum | 05:51 |
| soniya29|rover | marios, ack | 05:55 |
| bhagyashris | ack thanks | 05:55 |
| marios | :( no more irc | 05:58 |
| *** ysandeep is now known as ysandeep|afk | 06:13 | |
| Tengu | Marios perfect! | 06:36 |
| Tengu | Jm1[m] I doubt, while patch is in it's not applied AFAIK, there's an issue on the infra side | 06:36 |
| Tengu | I'll be online in a few, just finished my rower workout... | 06:37 |
| *** ysandeep|afk is now known as ysandeep | 07:35 | |
| *** amoralej|off is now known as amoralej | 07:36 | |
| Tengu | aannd hello there :) | 07:50 |
| jm1 | Tengu: o/ :D What do you mean with patch is not applied? You mean its merged but not rolled out to opendev's infra nodes? | 07:51 |
| Tengu | jm1: exactly | 07:52 |
| jm1 | Tengu: ack, then.. i will just wait :D | 07:52 |
| Tengu | 2s, lemme find that job again. | 07:53 |
| Tengu | jm1: ah! apparently it IS applied now: https://zuul.opendev.org/t/openstack/builds?job_name=infra-prod-service-mirror&project=opendev/system-config | 07:53 |
| Tengu | last run was successful. | 07:53 |
| Tengu | soooo.... yeah. you may try again today. | 07:53 |
| jm1 | Tengu: oh that is great, jobs are running again :D | 07:59 |
| Tengu | jm1: lemme know if you encounter new issue, but this time..... It *really* should work. | 08:02 |
| Tengu | jm1: I've also updated the test_mirror.py in order to download the file from galaxy (via the proxy) and ensure it's an archive. | 08:03 |
| Tengu | so I expect it to work properly now. | 08:03 |
| Tengu | oh, great, and my patch against the tripleo-modify-image is really working. | 08:03 |
| Tengu | GOOD | 08:04 |
| *** jpena|off is now known as jpena | 08:05 | |
| Tengu | jm1: is it working fine with the proxy now? | 09:26 |
| marios | soniya29|rover: arxcruz|ruck: let me know if you need something | 09:26 |
| Tengu | (please, tell me yes, please :)) | 09:26 |
| Tengu | marios: btw - when shall I switch my nick and add |rover? after the passing call? | 09:27 |
| arxcruz|ruck | upstream is stable, some jobs fail, i'm reruning and it's working, we will have promotions soon today | 09:27 |
| arxcruz|ruck | so all good on my side | 09:27 |
| arxcruz|ruck | just need to monitor the internal job trigger that we move from f35 to f36 yesterday | 09:27 |
| arxcruz|ruck | but did not run yet | 09:27 |
| marios | Tengu: tomorrow :) | 09:31 |
| Tengu | ok :) | 09:31 |
| marios | careful what you wish for | 09:31 |
| marios | Tengu: shift runs friday to thursday so pickup tomorrow morning | 09:31 |
| Tengu | sounds good. | 09:32 |
| soniya29|rover | marios, sure | 09:43 |
| * soniya29|rover needs a break | 09:44 | |
| *** soniya29|rover is now known as soniya29|rover|brb | 09:44 | |
| jm1 | Tengu: galaxy mirroring is still not working properly :( https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_122/867202/4/check/ansible-collections-openstack-functional-devstack/12223d2/job-output.txt | 09:50 |
| jm1 | Tengu: maybe i am holding it wrong? | 09:50 |
| jm1 | Tengu: https://review.opendev.org/c/openstack/ansible-collections-openstack/+/867202 | 09:50 |
| jm1 | Tengu: interesting is that its always working for ansible 2.9 but no other ansible version such as 2.12 or newer | 09:51 |
| Tengu | lemme check the patch. | 09:51 |
| Tengu | hmm, I checked with local proxy running in a container + ansible-galaxy... | 09:53 |
| Tengu | jm1: so basically, I ran the following in order to test things up: ansible-galaxy collection download -vvvvvvv -s http://localhost:8080/galaxy -p ./operator tripleo.operator | 09:54 |
| Tengu | jm1: ansible-galaxy [core 2.14.0] | 09:54 |
| Tengu | and it was working just fine. Would be interesting to add some debug to that thing, maybe? | 09:54 |
| Tengu | oh, wait, the /galaxy was from the previous form, | 09:55 |
| Tengu | anyway - your change looks correct. lemme do some test here, namely: ansible-galaxy collection download -vvvvvvv -s https://mirror.iad3.inmotion.opendev.org:4448 -p ./operator tripleo.operator | 09:56 |
| Tengu | jm1: and... it works. | 09:56 |
| Tengu | jm1: https://paste.openstack.org/show/bjzK0nI1AY41PcRdSbxK/ | 09:57 |
| Tengu | it just works fine | 09:57 |
| Tengu | not sure why it's failing in CI then :/. | 09:57 |
| Tengu | but here, with the 2.14.0, it just works | 09:57 |
| Tengu | jm1: (and you can also see all the redirects, tests and things done against the proxy - it's really using it, without some weird bypass) | 10:00 |
| jm1 | Tengu: ack, i have rekicked my jobs with -vvvvvvvvvvvvvvv so hopefully we get some more insights | 10:01 |
| Tengu | jm1: sure - lemme know. | 10:18 |
| Tengu | jm1: btw, if you want, you can pass env variables the same way you passed the galaxy_server: ANSIBLE_DEBUG=true ANSIBLE_VERBOSITY=4 | 10:27 |
| Tengu | (seems the max is 4 - meaning -vvvv) | 10:27 |
| Tengu | ok, #lunch - back in ~1h | 10:28 |
| *** ysandeep is now known as ysandeep|afk | 11:00 | |
| marios | back in ~30mins | 11:00 |
| *** dviroel|afk is now known as dviroel | 11:13 | |
| *** rlandy|out is now known as rlandy | 11:14 | |
| rlandy | arxcruz|ruck: all ok with upstream? | 11:16 |
| rlandy | soniya29|rover|brb: marios: hi - looks like we are still missing jenkins reporting on 17.1? | 11:17 |
| rlandy | passed yesterday | 11:18 |
| rlandy | am I missing something> | 11:18 |
| rlandy | reported on a diff hash? | 11:19 |
| rlandy | k - guess that should kick again shortly | 11:24 |
| rlandy | bhagyashris: are you doing ok with standalone? | 11:24 |
| rlandy | Tengu: you doing ok with 9.2 stuff? need help there? | 11:25 |
| *** soniya29|rover|brb is now known as soniya29|rover | 11:25 | |
| arxcruz|ruck | rlandy: only missing train | 11:31 |
| arxcruz|ruck | all other branches were promoted today | 11:31 |
| Tengu | rlandy: so far so good I'd say - my patch corrected the tripleo-modify-image and the job went green, I'm adding some more jobs to see how it goes. | 11:31 |
| Tengu | (in testproject, I mean) | 11:31 |
| Tengu | ah, one is red. let's see. | 11:31 |
| Tengu | oh. of course it's red... my edition removed the Depends-On tripleo-image-modify change. | 11:32 |
| Tengu | #faceplam | 11:32 |
| marios | rlandy: o/ i was a bit hands off today - asked soniya29|rover to ping me if help needed. i did not dig into that issue | 11:35 |
| rlandy | Tengu: ok - so you're going in rr this week | 11:36 |
| Tengu | rlandy: it seems so, yep. | 11:36 |
| rlandy | we can chat on friday how much is left on this task | 11:36 |
| Tengu | yep | 11:36 |
| soniya29|rover | rlandy, seems last build passed - https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/job/pipeline_integration-pcci-17.1_dlrn-rhel-9.1-virthost-3cont_2comp_3ceph-ipv4-geneve-ceph/lastBuild/ | 11:36 |
| rlandy | soniya29|rover: it reported on a diff hash | 11:36 |
| rlandy | the line should kick again at the top of the hour | 11:36 |
| bhagyashris | rlandy, pull issue resolved facing different issue now | 11:37 |
| rlandy | chandankumar: marios: I updated https://review.rdoproject.org/r/c/rdo-jobs/+/46661 Move wallaby line to run on ibm cloud | 11:37 |
| rlandy | bhagyashris: want to share? | 11:37 |
| Tengu | jm1: hmmm, the issue seems weird, for the galaxy proxy: Server https://mirror-int.iad.rax.opendev.org:4448/api has not signed ansible.utils:2.0.0 | 11:38 |
| Tengu | lemme try to locally install that same ansible.utils collection. | 11:38 |
| bhagyashris | rlandy, https://sf.hosted.upshift.rdu2.redhat.com/logs/51/436651/33/check/periodic-tripleo-ci-rhel-9-standalone-rhos-18/94559ed/logs/undercloud/home/zuul/standalone_deploy.log | 11:39 |
| Tengu | fun.... jm1 so, here, for instance, running ansible-galaxy collection install -vvvvvvv -s https://mirror.iad3.inmotion.opendev.org:4448 -p ./ansible ansible.posix indeed shows some occurrences of the "has no signed ansible.posix:<VERSION>", but in the end, it does install: ansible.posix:1.4.0 was installed successfully | 11:40 |
| Tengu | wondering if it has anything to do with the requirement... lemme check its content. | 11:41 |
| Tengu | o_O | 11:43 |
| rlandy | bhagyashris: looking | 11:43 |
| Tengu | wow.... what the... | 11:43 |
| bhagyashris | rlandy, https://sf.hosted.upshift.rdu2.redhat.com/logs/51/436651/33/check/periodic-tripleo-ci-rhel-9-standalone-rhos-18/94559ed/logs/undercloud/var/log/containers/stdouts/container-puppet-cinder.log | 11:45 |
| marios | what about the standalones/other jobs rlandy | 11:46 |
| rlandy | bhagyashris: may be a real issue of our missing a version | 11:47 |
| marios | are they already defaulting to ibm ? rlandy re https://review.rdoproject.org/r/c/rdo-jobs/+/46661 Move wallaby line to | 11:47 |
| rlandy | marios: I think they should inherit the nodeset as using a higher levl yaml tag | 11:47 |
| rlandy | no just the dependencies - also nodeste | 11:47 |
| rlandy | because of where the tag references | 11:47 |
| rlandy | I think | 11:47 |
| marios | rlandy: soniya29|rover: added comment to bz and cix for https://trello.com/c/dbnWFcBB#comment-63c9066bc0280d0153435d62 fyi - know where it comes from/where to dig but i had to switch and will revisit. maybe we remove from criteria for a day/2 | 11:48 |
| Tengu | jm1: ok, the "has not signed <collection:version>" thing is also showing up when hitting the official galaxy server. | 11:48 |
| rlandy | marios: I don't think it is in criteria | 11:48 |
| rlandy | line would have promoted if jenkins would have reported | 11:48 |
| rlandy | we should confirm that | 11:48 |
| rlandy | at least that is what rr tool said | 11:48 |
| marios | rlandy: yes i thought it is/should be... at least it is there for 8 line | 11:48 |
| rlandy | bhagyashris: can you link your job def? | 11:49 |
| rlandy | are you overriding the version to 18? | 11:49 |
| marios | rlandy: erm where is the config http://10.0.110.143/config/ | 11:49 |
| Tengu | jm1: apparently the outputs we get with curl -L https://galaxy.ansible.com/api/v2/collections/community/general/versions/\?page_size\=100 and curl -L https://mirror.iad3.inmotion.opendev.org:4448/api/v2/collections/community/general/versions/\?page_size\=100 are the same - save for the hostname, of course. | 11:50 |
| bhagyashris | rlandy, https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/438357/3/zuul.d/rhos18-rhel9-jobs.yaml#68 one thing i point out is "rhos_release_args" value should be ceph-5.2 -r 9.1 | 11:50 |
| Tengu | I really don't get it. And it seems to fail on community.general, NOT ansible.utils or any others. | 11:50 |
| rlandy | bhagyashris: pls change that to "ceph-5.3 -r 9.1" | 11:53 |
| Tengu | bhagyashris: you probably want to rebase against the top of the branch - I saw many changes related to the ceph version lately. | 11:53 |
| bhagyashris | rlandy, ok. but for container builds we are using 5.2 -r 9.1 | 11:53 |
| rlandy | bhagyashris: ysandeep|afk changed that to 5.3 in 17.1 | 11:55 |
| rlandy | so I think 18 should get the same | 11:55 |
| bhagyashris | https://git.app.eng.bos.redhat.com/git/openstack/tripleo-ci-internal-jobs.git/tree/zuul.d/rhos18-rhel9-jobs.yaml#n68 | 11:55 |
| rlandy | bhagyashris: can be changed afterwards | 11:55 |
| rlandy | should not have caused the current error | 11:55 |
| bhagyashris | ok i will change that in the same patch and will test both the container build and standalone | 11:55 |
| Tengu | oh. 18. ok. nevermind. | 11:55 |
| * Tengu too deep into 17.1 | 11:56 | |
| rlandy | bhagyashris: takashi would be the best person to help debug puppet errors if we have the right repos | 11:56 |
| bhagyashris | rlandy, ok will check with him thanks | 11:57 |
| rlandy | rhos-release-ceph-5.3.repo2023-01-18 13:39 472 | 11:57 |
| rlandy | rhos-release-rhel-9.1.repo2023-01-18 13:39 4.0K | 11:57 |
| rlandy | 5,3 is in there somewhere | 11:58 |
| rlandy | bhagyashris; pls check with him if we are missing something in 18 | 11:58 |
| rlandy | bhagyashris: looks like you have the right repos | 11:58 |
| rlandy | https://sf.hosted.upshift.rdu2.redhat.com/logs/51/436651/33/check/periodic-tripleo-ci-rhel-9-standalone-rhos-18/94559ed/logs/undercloud/etc/yum.repos.d/delorean.repo | 11:58 |
| bhagyashris | yeah | 12:00 |
| *** jgilaber is now known as jgilaber|lunch | 12:01 | |
| Tengu | rlandy: mtg? | 12:01 |
| rlandy | chandankumar: need 5 pls | 12:31 |
| chandankumar | rlandy: yes | 12:31 |
| chandankumar | ok | 12:33 |
| chandankumar | rlandy: https://github.com/openstack-k8s-operators/edpm-ansible | 12:35 |
| *** ysandeep|afk is now known as ysandeep | 12:48 | |
| Tengu | jm1: so yeah... I really, really don't understand why it's failing on the general.community, but not on the others. | 12:52 |
| Tengu | I mean... why is it working for ansible.utils, tripleo.operators, but not that general.community thingy?! | 12:52 |
| Tengu | oh.... no... they... jm1 I'm wondering if the key used for the cache (such as the hostname and path) aren't *too long* for the cache engine they have. | 12:56 |
| Tengu | especially the hostname. | 12:56 |
| Tengu | ok. doesn't look like that kind of issue - using the IP address (hence, shorter key) doesn't change the behavior: still failing. | 12:58 |
| Tengu | (and, btw, I'm starting to understand why galaxy servers are always crashed........) | 13:05 |
| *** amoralej is now known as amoralej|lunch | 13:05 | |
| Tengu | if it really, really queries for the last 100 versions of an object and, then, queries the details for every single version............ I mean, what's the point here? | 13:06 |
| *** dasm|off is now known as dasm | 13:08 | |
| dasm | o/ | 13:08 |
| rlandy | pojadhav: frenzy_friday: dasm: marios: chandankumar: do we need a tc/ua/pm planning meeting next week? | 13:09 |
| rlandy | pojadhav: frenzy_friday: dasm; also - we should a have a retro next week , right? | 13:10 |
| rlandy | I see the main planning meeting in my calendar but no retro | 13:10 |
| rlandy | pls add Tengu: jgilaber|lunch and adrian to those meetings | 13:11 |
| pojadhav | rlandy, there is retro on 25th | 13:11 |
| pojadhav | i can see it on my calender | 13:11 |
| rlandy | pojadhav: ok - I see it | 13:11 |
| Tengu | I don't see it, I think | 13:12 |
| rlandy | hmmm .. clash with next gen CI meeting | 13:12 |
| pojadhav | rlandy, one more thing, on 26th india having public holiday, so india folks will be out that day and we have planning meeting same day | 13:12 |
| rlandy | ugh - getting calendar straight is a job unto itself | 13:12 |
| Tengu | pojadhav: I don't have the retro planing | 13:12 |
| pojadhav | Tengu, I will add you | 13:12 |
| Tengu | thanks :) | 13:12 |
| Tengu | rlandy: if only all the meetings were using a common timezone, like, for instance, UTC... | 13:12 |
| * Tengu ducks and hides | 13:13 | |
| jm1 | Tengu: omg maybe we should try to pin the collection version as a workaround. these queries are crazy | 13:13 |
| rlandy | pojadhav; let's discuaa at scrum today | 13:13 |
| pojadhav | rlandy, yep | 13:13 |
| rlandy | we could move retro to friday | 13:13 |
| rlandy | and have planning on monday if needed | 13:13 |
| rlandy | or friday? | 13:13 |
| Tengu | jm1: yeah, that would help a bit. | 13:13 |
| jm1 | Tengu: oh wait, it is already pinned | 13:13 |
| rlandy | Tengu: they should | 13:13 |
| Tengu | jm1: I don't understand why galaxy client does all of that sh**. | 13:13 |
| rlandy | ie: use UTC | 13:13 |
| Tengu | jm1: they are? | 13:13 |
| Tengu | jm1: an, the community.general. OK. not the ansible.posix nor the ansible.utils: https://opendev.org/openstack/ansible-collections-openstack/src/branch/master/ci/requirements.yml | 13:14 |
| jm1 | Tengu: only community.general is pinged. will pin the others too | 13:14 |
| jm1 | *pinned | 13:14 |
| Tengu | lemme re-run with the pin - didn't... hey | 13:15 |
| Tengu | using my localhost:8080 proxy, it's *working* | 13:15 |
| Tengu | I mean.... | 13:15 |
| Tengu | rm -rf ansible ~/.ansible/galaxy_cache/api.json ; ansible-galaxy collection install -vvvv -s http://localhost:8080 -p ./ansible community.general:4.8.8 ->> community.general:4.8.8 was installed successfully | 13:15 |
| *** jgilaber|lunch is now known as jgilaber | 13:16 | |
| Tengu | jm1: I really suspect galaxy server isn't able to answer all of the running queries. IMHO we should get in touch with the devs in order to, maybe, limit the index to 10 versions max, and check WHY it's looping on all of them without reason. | 13:16 |
| Tengu | if we install only one collection, I don't see why it wouldn't just install the latest available. | 13:16 |
| Tengu | rlandy: one thing with UTC: there's no DST with UTC, meaning the meeting time will change for countries with DST - case for a fair part of the USA, all of EMEA. iirc, India doesn't have DST. | 13:18 |
| * Tengu already played with UTC for DF checkpoint back then | 13:18 | |
| rlandy | Tengu: ye - that happens to us | 13:18 |
| Tengu | it was a fun hunt for a good time | 13:18 |
| rlandy | every half year | 13:18 |
| rlandy | we have to readjust | 13:18 |
| Tengu | yeah, same for me. | 13:18 |
| * Tengu hates DST | 13:18 | |
| * rlandy too | 13:18 | |
| rlandy | we ever had it in south africa | 13:19 |
| rlandy | worked just fine | 13:19 |
| rlandy | ok - it was a small country | 13:19 |
| Tengu | some countries are blessed. | 13:19 |
| Tengu | Europe was supposed to get rid of it. | 13:19 |
| Tengu | but thanks to covid (and not Ukraine situation), the decision has been postponed. | 13:19 |
| Tengu | pfrrtt. | 13:19 |
| Tengu | *and now Ukraine situation | 13:19 |
| rlandy | same in usa | 13:19 |
| rlandy | comes up for debate all the time | 13:20 |
| Tengu | I know some states in the USA don't have DST | 13:20 |
| rlandy | no resolution yet | 13:20 |
| rlandy | so we continue the fun | 13:20 |
| dasm | Tengu: Arizona for sure. No idea about other places | 13:20 |
| rlandy | my favorite is taking a flight the day they change | 13:20 |
| rlandy | and seeing how many people miss the plane | 13:20 |
| dasm | rlandy: my favorite was when friend booked a flight on 06/05 but arrived to an airport 05/06 (European vs. American date notation) | 13:21 |
| dasm | that was painful | 13:21 |
| marios | rlandy: o/ sure re tc/ua sync | 13:22 |
| jm1 | Tengu: maybe these issues are related to 'Server *** has not signed ***' messages? | 13:22 |
| Tengu | jm1: maybe... note that -vvvv doesn't provide as much data... | 13:22 |
| Tengu | so 4 isn't the max. | 13:23 |
| Tengu | jm1: omg... in fact.... even the 100 isn't enough - it wants to get ALL the content, I also see ?page=2&page_size=100 X( | 13:24 |
| Tengu | I mean... yeah, we really want to pin versions. | 13:24 |
| Tengu | hmmmmm. fun.... using localhost:8080, it does work. | 13:24 |
| Tengu | is there a diff with my local env and what I pushed now?! | 13:25 |
| rlandy | pojadhav: frenzy_friday: ok - pls schedule us a pre-planning meeting | 13:25 |
| rlandy | thank you | 13:25 |
| marios | rlandy: bhagyashris: arxcruz|ruck: o/ rr handoff | 13:30 |
| rlandy | arxcruz|ruck: ^^ | 13:31 |
| arxcruz|ruck | joining | 13:32 |
| * frenzy_friday is back, reading | 13:40 | |
| pojadhav | bhagyashris, chandankumar : scrum | 14:01 |
| *** amoralej|lunch is now known as amoralej | 14:05 | |
| arxcruz|ruck | dasm: not sure if it helps: https://www.redhat.com/sysadmin/podman-docker-compose | 14:11 |
| dasm | Tengu: I'm testing this patch: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/46644 | 14:11 |
| dasm | arxcruz|ruck: it doesn't | 14:11 |
| Tengu | dasm: I'll check after the call | 14:12 |
| dasm | Tengu: "Pull containers" step is failing on: https://paste.opendev.org/show/bXZK7jC3VBYK09I7DTkU/ | 14:12 |
| dasm | arxcruz|ruck: i'm able to run both docker-compose and podman, but the issue is with ansible docker-compose module. | 14:12 |
| arxcruz|ruck | ahhh | 14:12 |
| arxcruz|ruck | ping zbr | 14:12 |
| Tengu | yup, indeed. | 14:13 |
| Tengu | zbr is probably the right target | 14:13 |
| dasm | Tengu: arxcruz|ruck I'm running "pull" step but it behave like it runs "up" | 14:13 |
| dasm | which channel? | 14:13 |
| arxcruz|ruck | ansible | 14:13 |
| arxcruz|ruck | dasm: he's red hatter, let me send his info for you | 14:13 |
| dasm | ansible on libera? | 14:13 |
| dasm | it's very quiet channel :) | 14:14 |
| rlandy | Tengu: pls remind marios or arxcruz|ruck or me to explain alternative criteria to you | 14:14 |
| rlandy | jobs that run in downstream | 14:14 |
| rlandy | from upstream criteria | 14:14 |
| Tengu | rlandy: we may check that tomorrow during our 1:1? | 14:14 |
| rlandy | Tengu: yeah - or in your morning meeting with marios | 14:15 |
| Tengu | rlandy: maybe we should get some gdoc in order to get an agenda. | 14:15 |
| Tengu | right, there's also that one. | 14:15 |
| Tengu | rlandy: care to add a "note doc" to our meeting? gcal should create it on your behalf and share it with me directly, while showing it in the meet directly as well :) | 14:16 |
| Tengu | I'm unable to do that, since it's not "my" event :) | 14:16 |
| arxcruz|ruck | Tengu: rlandy marios soniya29|rover fs20 for train passes, we will have a promotion soon, so, tomorrow, you will be up to date with promotions :) | 14:27 |
| arxcruz|ruck | Tengu: https://www.youtube.com/watch?v=79DijItQXMM | 14:27 |
| Tengu | \o/ | 14:28 |
| soniya29|rover | arxcruz|ruck, that's a great news | 14:28 |
| Tengu | jm1: fun part: even if you do pin the version, the CLI will *still* require all of the available versions..... though it won't loop onto them all, apparently | 14:29 |
| jm1 | Tengu: with version pinning it still fails with the same error ("... has not signed...") but much faster https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_27e/867202/6/check/ansible-collections-openstack-functional-devstack/27ed31d/job-output.txt | 14:34 |
| arxcruz|ruck | brb | 14:34 |
| Tengu | jm1: wondering if there isn't some bug in the CLI. | 14:36 |
| Tengu | jm1: it really seems located at the community.general collection. | 14:36 |
| Tengu | and I don't understand why it would work for the other collections and not that one.... nor why it's working when not using the proxy. | 14:37 |
| Tengu | and... well. localhost:8080 is OK - though I don't see differences in the vhost setup :/. | 14:37 |
| Tengu | need to get a break, I'll re-check the httpd config one more time to be sure. | 14:37 |
| jm1 | Tengu: maybe its worth to skip any ansible galaxy install and use github instead | 14:39 |
| jm1 | Tengu: before you sink too much time into that | 14:39 |
| * pojadhav cooking.. afk | 14:41 | |
| rlandy | dasm: marios: went through all the zed upgrades reviews on revuew list - only one left w/o a w+ | 14:44 |
| reviewbot | Do you want me to add your patch to the Review list? Please type something like add to review list <your_patch> so that I can understand. Thanks. | 14:44 |
| rlandy | THT | 14:44 |
| rlandy | there is an open comment | 14:44 |
| Tengu | jm1: I was thinking about using "type: git" as well. | 14:44 |
| Tengu | though it's.... | 14:44 |
| Tengu | annoying. | 14:44 |
| rlandy | chandankumar; could you take one more look at https://review.rdoproject.org/r/c/rdo-jobs/+/46661 pls | 14:44 |
| Tengu | jm1: maybe for the content that fails with the proxy? | 14:45 |
| rlandy | Tengu: would appreciate your vote on https://review.rdoproject.org/r/c/config/+/46628 - on case you have concerns here | 14:45 |
| rlandy | thank you | 14:45 |
| Tengu | 2s | 14:46 |
| jm1 | Tengu: i will replace ansible galaxy with github completely and then monitor whether this improves our ci | 14:46 |
| dasm | rlandy: nice. thanks | 14:46 |
| Tengu | jm1: it's sad, but may indeed be better. though.... damn, it's unbelievable. | 14:46 |
| rlandy | Tengu: bhagyashris: I am putting in the vexxhost resource request today ... if you notice jobs are waiting a long time to schedule because of lack of resources, pls let me know | 14:47 |
| dasm | rlandy: marios removed unnecessary line on THT | 14:47 |
| rlandy | I am hoping we can handle the lesser node | 14:47 |
| rlandy | node allocation | 14:47 |
| Tengu | rlandy: 'k. we'll see. Not nice to do that for my first rr shift though ;) | 14:48 |
| rlandy | Tengu: sorry - no choice | 14:48 |
| jm1 | Tengu: its particular sad because you invested a lot of time in that mirroring already and it works for ansible 2.9. only because of some undocumented verification code without helpful error messages we have to drop ansible galaxy now :/ | 14:48 |
| Tengu | rlandy: regarding your review - I don't really know what are the actual implication of it. less jobs, but what may slip in if they don't run. | 14:48 |
| Tengu | jm1: maybe I can go to the ansible team and try to get some more data, but I have the feeling they won't care... probably stating something like "just ramp up a local ansible-galaxy and that's it". | 14:49 |
| jm1 | Tengu: probably 😬 | 14:51 |
| rlandy | Tengu: anything OVB would pick up but we are short of resources and those jobs are non-voting so many people ignore them anyways | 14:51 |
| rlandy | I am leaving some check testing | 14:51 |
| rlandy | but not much | 14:52 |
| Tengu | rlandy: oh, non-voting. let's kill them then, yeah | 14:52 |
| Tengu | rlandy: maybe mentioning the fact they were non-voting in the commit message may help. | 14:52 |
| rlandy | Tengu: lesser of two evils - need to give dviroel somewhere to run his prow tests | 14:52 |
| rlandy | Tengu: sure | 14:53 |
| Tengu | rlandy: sure - please just point to the "removing non-voting" and you'll get my vote :) | 14:53 |
| rlandy | k | 14:53 |
| marios | dasm: thx revoted | 15:01 |
| dasm | sweet! | 15:01 |
| dasm | rlandy: can you +2+W https://review.opendev.org/c/openstack/tripleo-heat-templates/+/865052 ? | 15:01 |
| Tengu | dasm: done | 15:05 |
| dasm | sweet! ZED Upgrades done | 15:05 |
| rlandy | dasm: looks like Tengu beat me to it :) | 15:12 |
| Tengu | ^^ | 15:12 |
| dasm | \o/ | 15:13 |
| dasm | \( ﾟヮﾟ)/ | 15:13 |
| *** ysandeep is now known as ysandeep|dinner | 15:18 | |
| Tengu | jm1: ok, so.... on the vhost side, I don't see any difference with my local env and what I've pushed. | 15:25 |
| Tengu | it's 1:1 the same. | 15:25 |
| Tengu | oh.... errrr. | 15:25 |
| Tengu | wait. | 15:25 |
| Tengu | of course. | 15:25 |
| Tengu | if I open the same file twice | 15:25 |
| Tengu | #facepalm | 15:26 |
| Tengu | ok. still. there's ONE difference: a comment... and my local env doesn't have the TLS thingy to "substitute" the host. | 15:27 |
| Tengu | lemme add that one back. | 15:28 |
| *** dasm is now known as Guest1846 | 15:29 | |
| Tengu | jm1: if it works locally (as I expect), it means there's something else in the global proxy config... | 15:29 |
| Tengu | yeah... | 15:29 |
| Tengu | that's probably it. | 15:29 |
| Tengu | hmm. and it's not a TLS thing, since upstream mirror/proxy is failing with and without TLS. | 15:30 |
| *** Guest1846 is now known as dasm | 15:30 | |
| rlandy | bhagyashris: wrt downstream - trying to rerun failing jobs atm - pls check with attila in your morning if jenkins is still failing | 15:31 |
| *** ysandeep|dinner is now known as ysandeep | 15:32 | |
| marios | bhagyashris: Tengu: sent recurring invite for morning sync please check inbox and let me know if time is OK for you | 15:36 |
| Tengu | marios: all good, thanks! | 15:38 |
| bhagyashris | rlandy, ack will check tomorrow | 15:57 |
| bhagyashris | marios, ack thanks | 15:57 |
| bhagyashris | rlandy, thanks | 15:57 |
| *** dviroel is now known as dviroel|lunch | 16:01 | |
| Tengu | jm1: I get a workaround! woohooo | 16:03 |
| Tengu | jm1: you can export this env: GALAXY_CACHE_DIR=/dev/null | 16:03 |
| Tengu | that will basically bypass the ansible-galaxy generated cache on-disk. It shouldn't be that a big deal. | 16:04 |
| Tengu | jm1: I suspect the pagination is playing some nasty game somewhere. | 16:05 |
| Tengu | this seems to match: when the version index is over one page, it fails (community.general has 108 entries, while the CLI asks 100 results per page) - while ansible.utils and ansible.posix seem to have a lower amount, and no pagination. | 16:05 |
| Tengu | now, I suspect something on the proxies is pushing something weird - but I really have no idea. If you want, you can check the default cache location, ~/.ansible/galaxy_cache/api.json - and check the content when trying to install the community.general through the proxy, and any of the other collection (ansible.posix for instance). | 16:07 |
| Tengu | there's indeed a missing key in the generated cache file (so a JSON). | 16:07 |
| rlandy | bhagyashris: pls join #openstack-ppci | 16:07 |
| rlandy | pcci | 16:07 |
| ysandeep | sidenote: As irc going away, I think we should move #openstack-pcci to slack as well. | 16:09 |
| Tengu | jm1: it's not a matter of "cache is serving the same content twice and therefore creates some loop". | 16:09 |
| Tengu | ysandeep: if it's a downstream channel, indeed. | 16:09 |
| ysandeep | Tengu, yes it is a downstream channel | 16:10 |
| Tengu | ysandeep: yeah, I have it in my list :). | 16:10 |
| ysandeep | Creating slack channel | 16:10 |
| chandankumar | see ya people! | 16:12 |
| jm1 | Tengu: You found a bug in Ansible Galaxy cache mechanism? Nice! Jobs are rerunning with our env var :D | 16:23 |
| Tengu | jm1: :) | 16:23 |
| ysandeep | folks o/ I have created #openstack-pcci channel on slack as well for internal discussion | 16:24 |
| marios | so sad | 16:28 |
| marios | really is the end of an era | 16:28 |
| marios | very likely last i'll ever use irc again | 16:28 |
| marios | so rlandy does this mean we should create a public oooq | 16:29 |
| Tengu | I still have my private things on IRC :) | 16:29 |
| Tengu | communities and such | 16:29 |
| marios | can we ? if not we need to be able to | 16:29 |
| ysandeep | marios, personally I started liking slack more than irc, dviroel|lunch and me regularly use it in our 1-1 conv.. very handy to keep/track conversations. | 16:29 |
| marios | ysandeep: it seems nice i really like the threading that is a game changer | 16:29 |
| marios | ysandeep: just also sad about irc | 16:30 |
| Tengu | and right now, I have a TheLounge instance running at home, allowing me to get a nice IRC in the browser... meaning I can have everything in the browser now. | 16:30 |
| marios | 18:28 < marios> really is the end of an era | 16:30 |
| jgilaber | marios: I thought this channel is already public? | 16:30 |
| marios | i mean on slack jgilaber | 16:30 |
| marios | do we have one there? | 16:30 |
| ysandeep | we will still use irc for oooq, just another window | 16:30 |
| Tengu | marios: I'd refrain from moving community chans on slack. | 16:30 |
| Tengu | especially since the slack instance we're using now is apparently internal only | 16:30 |
| Tengu | thus not public. | 16:30 |
| marios | that is not cool but ... k two windows to monitor... | 16:30 |
| jgilaber | I think Tengu is right, slack is internal | 16:31 |
| marios | yes Tengu this is what i was geting at ^ | 16:31 |
| Tengu | marios: it's 2 tabs here. well, 3 with that horrible gchat. | 16:31 |
| marios | yah i have gchat in a pinned tab but mostly ignore it | 16:31 |
| jgilaber | Do you have irc in your browser? That would be useful for me | 16:31 |
| marios | no i use irssi | 16:31 |
| marios | and browser for slack :/ | 16:31 |
| marios | so not cool | 16:31 |
| jgilaber | yeah same | 16:31 |
| pojadhav | Tengu, have you got retro invite now ?? | 16:32 |
| pojadhav | jm1, sorry.. missed to remove you from retro invite.. now I did it. | 16:32 |
| Tengu | pojadhav: I do \o/ | 16:32 |
| pojadhav | Tengu, great | 16:33 |
| Tengu | marios: I used irssi until yesterday :) | 16:33 |
| Tengu | marios: I must say.... TheLounge is a really cool thing. bouncer+webIRC | 16:33 |
| Tengu | supporting multiple networks, logging and all the needed features. | 16:33 |
| jm1 | pojadhav: thanks :) | 16:36 |
| *** ysandeep is now known as ysandeep|out | 16:38 | |
| ysandeep|out | reviewbot: please add in reviewlist: https://review.opendev.org/c/openstack/tripleo-ci/+/871085 | 16:41 |
| reviewbot | I have added your review to the Review list | 16:41 |
| * ysandeep|out out see everyone tomorrow | 16:42 | |
| marios | o/ /me too in few | 16:44 |
| Tengu | jm1: so you're all good with using the infra proxies for galaxy? | 16:51 |
| *** dviroel|lunch is now known as dviroel | 16:57 | |
| *** marios is now known as marios|out | 16:59 | |
| Tengu | ok, time to get offline. | 17:01 |
| Tengu | See you tomorrow team! | 17:01 |
| dasm | Tengu: o/ | 17:03 |
| Tengu | jm1: btw: https://review.opendev.org/c/openstack/tripleo-ansible/+/866031 I think it's cleaner.... | 17:07 |
| Tengu | and allows to actually push multiple servers, just in case one is failing. | 17:08 |
| *** jpena is now known as jpena|off | 17:30 | |
| *** amoralej is now known as amoralej|off | 17:47 | |
| jm1 | Tengu: interesting that ANSIBLE_GALAXY_CACHE_DIR works for you .... because i get errors such as "Creating Galaxy API response cache file at '/dev/null/api.json'... NotADirectoryError: [Errno 20] Not a directory: b'/dev/null/api.json'" ?!? | 19:53 |
| Tengu | jm1: check on #opendev - we found the root cause. | 19:53 |
| jm1 | Tengu++ | 19:54 |
| Tengu | jm1: that was a fscking complicated hunt :) | 20:00 |
| Tengu | jm1: if the env var doesn't work, you may want to pass the --no-cache. but imho, you should check what I've proposed against tripleo-ansible. | 20:00 |
| Tengu | jm1: https://review.opendev.org/c/openstack/tripleo-ansible/+/866031/22/zuul.d/playbooks/pre.yml | 20:01 |
| Tengu | here, I set the configuration at ansible level, via its config file. This allows to set all of the needed configurations | 20:02 |
| Tengu | thouuggh.... it seems the cache_dir is indeed something bad. | 20:02 |
| Tengu | apparently we can't configure ansible to NOT use the cache. | 20:03 |
| Tengu | jm1: I'm pretty sure this is why it was working on 2.9 and not on newer - the cache thingy was introduced post-2.9 | 20:03 |
| jm1 | Tengu: I am fine with the galaxy mirror env var if opendev's mirroring finally works :D | 20:13 |
| jm1 | Tengu: not much activity to except in future anyway | 20:14 |
| Tengu | jm1: we're discussing the issue - but it's not THAT simple :) | 20:14 |
| Tengu | jm1: apparently the zuul job gets a name while the mirror knows itself as another one, and with ansible-galaxy using FQDN for everything because it's not able to just use relative URI in the answers.... | 20:15 |
| Tengu | plus mod_sbstitute not supporting httpd env vars, preventing a nice and neat replacement... | 20:15 |
| jm1 | Tengu: sad :( | 20:17 |
| jm1 | Tengu: 10 patches being checked atm. my bet for tomorrow is that 25% will have failed :D | 20:19 |
| Tengu | yeah - it's a corner case, and galaxy is pretty annoying (and dumb) in its answers. | 20:19 |
| Tengu | probably :( | 20:19 |
| jm1 | Tengu: not a big issue, will simply recheck them. its the last big chunk of patches anyway | 20:19 |
| Tengu | jm1: so, the safest is to actually modify ansible-galaxy calls to pass "--no-cache" | 20:19 |
| jm1 | Tengu: does ansible 2.9 support --no-cache? | 20:19 |
| Tengu | jm1: i.e. ansible-galaxy collection install --no-cache | 20:19 |
| Tengu | hmmm | 20:20 |
| Tengu | good question :/ | 20:20 |
| jm1 | Tengu: i will check tomorrow | 20:20 |
| jm1 | Tengu: for today i am out | 20:20 |
| jm1 | Tengu: its also late for you | 20:20 |
| jm1 | Tengu: have a good night! | 20:20 |
| *** dviroel is now known as dviroel|out | 21:05 | |
| *** arxcruz|ruck is now known as arxcruz | 21:50 | |
| * dasm => offline | 23:13 | |
| dasm | o/ | 23:13 |
| *** dasm is now known as dasm|off | 23:13 | |
| *** rlandy is now known as rlandy|out | 23:29 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!