| *** dpawlik5 has quit IRC | 00:10 | |
| *** dpawlik7 has joined #zuul | 00:11 | |
| *** hamalq has quit IRC | 00:28 | |
| *** dpawlik7 has quit IRC | 00:57 | |
| *** dpawlik5 has joined #zuul | 00:57 | |
| *** dpawlik5 has quit IRC | 01:23 | |
| *** dpawlik2 has joined #zuul | 01:23 | |
| *** dpawlik2 has quit IRC | 01:33 | |
| *** dpawlik3 has joined #zuul | 01:36 | |
| *** evrardjp has quit IRC | 02:33 | |
| *** evrardjp has joined #zuul | 02:33 | |
| *** dpawlik9 has joined #zuul | 02:35 | |
| *** dpawlik3 has quit IRC | 02:36 | |
| *** bhavikdbavishi has joined #zuul | 03:30 | |
| *** bhavikdbavishi1 has joined #zuul | 03:33 | |
| *** bhavikdbavishi has quit IRC | 03:35 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 03:35 | |
| *** ykarel has joined #zuul | 03:51 | |
| *** dpawlik9 has quit IRC | 04:35 | |
| *** dpawlik6 has joined #zuul | 04:35 | |
| *** bhavikdbavishi has quit IRC | 04:52 | |
| *** dpawlik7 has joined #zuul | 05:34 | |
| *** dpawlik6 has quit IRC | 05:35 | |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Execute builds via ZooKeeper https://review.opendev.org/c/zuul/zuul/+/770902 | 06:10 |
|---|---|---|
| *** saneax has joined #zuul | 06:14 | |
| *** sanjayu_ has joined #zuul | 06:22 | |
| *** saneax has quit IRC | 06:24 | |
| *** jcapitao has joined #zuul | 06:34 | |
| *** dpawlik5 has joined #zuul | 06:42 | |
| *** dpawlik7 has quit IRC | 06:42 | |
| *** dpawlik5 has quit IRC | 06:52 | |
| *** dpawlik07 has joined #zuul | 06:55 | |
| *** dpawlik07 has quit IRC | 06:57 | |
| *** sshnaidm is now known as sshnaidm|afk | 06:58 | |
| *** dpawlik5 has joined #zuul | 06:58 | |
| *** dpawlik5 has quit IRC | 07:12 | |
| *** dpawlik5 has joined #zuul | 07:13 | |
| *** dpawlik5 is now known as dpawlik | 07:14 | |
| *** sanjayu_ has quit IRC | 07:23 | |
| *** sanjayu_ has joined #zuul | 07:24 | |
| *** dpawlik has quit IRC | 07:28 | |
| *** dpawlik6 has joined #zuul | 07:28 | |
| *** avass has quit IRC | 07:29 | |
| *** avass has joined #zuul | 07:30 | |
| *** avass has quit IRC | 07:32 | |
| *** avass has joined #zuul | 07:34 | |
| *** tosky has joined #zuul | 07:38 | |
| *** dpawlik6 has quit IRC | 07:43 | |
| *** jpena|off is now known as jpena | 07:51 | |
| avass | Looks like tox-py27 isn't working for zuul-jobs because of a new openstacksdk version: https://zuul.opendev.org/t/zuul/build/1d2b154d0d2f4c40b8235f482179e98b | 08:25 |
| avass | oh checking test-requirements py<=3.5 should use an older version so maybe that's because of a new tox version? | 08:45 |
| *** ykarel is now known as ykarel|lunch | 08:50 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Use openstacksdk 0.45.0 for python2.7 https://review.opendev.org/c/zuul/zuul-jobs/+/784894 | 08:52 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Use openstacksdk 0.45.0 for python2.7 https://review.opendev.org/c/zuul/zuul-jobs/+/784894 | 08:54 |
| openstackgerrit | Merged zuul/zuul-jobs master: ensure-podman: Use official podman repos for ubuntu https://review.opendev.org/c/zuul/zuul-jobs/+/765177 | 09:17 |
| *** holser has joined #zuul | 09:30 | |
| openstackgerrit | Sorin Sbârnea proposed zuul/zuul master: Document local testing https://review.opendev.org/c/zuul/zuul/+/766460 | 09:41 |
| *** jcapitao_ has joined #zuul | 09:49 | |
| *** jcapitao has quit IRC | 09:52 | |
| *** jcapitao_ has quit IRC | 09:57 | |
| *** sshnaidm|afk has quit IRC | 10:09 | |
| *** ykarel|lunch is now known as ykarel | 10:11 | |
| *** sshnaidm has joined #zuul | 10:18 | |
| *** fdegir4 is now known as fdegir | 10:22 | |
| *** ykarel_ has joined #zuul | 10:32 | |
| *** ykarel has quit IRC | 10:34 | |
| *** ykarel_ is now known as ykarel | 11:13 | |
| *** jpena is now known as jpena|lunch | 11:31 | |
| *** rlandy has joined #zuul | 11:50 | |
| *** bhavikdbavishi has joined #zuul | 12:11 | |
| pabelanger | avass: thanks for merging podman change | 12:24 |
| *** jcapitao has joined #zuul | 12:25 | |
| *** jpena|lunch is now known as jpena | 12:27 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Document algorithm var for remove-build-sshkey https://review.opendev.org/c/zuul/zuul-jobs/+/783988 | 12:50 |
| *** ykarel_ has joined #zuul | 12:53 | |
| *** ykarel has quit IRC | 12:56 | |
| *** ykarel_ is now known as ykarel | 13:36 | |
| *** bhavikdbavishi has quit IRC | 13:36 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Update operator-framwork to v1.4.2 https://review.opendev.org/c/zuul/zuul-operator/+/784457 | 13:40 |
| openstackgerrit | Merged zuul/zuul-jobs master: ensure-zookeeper: add use_tmpfs parameter https://review.opendev.org/c/zuul/zuul-jobs/+/783982 | 13:48 |
| *** avass has quit IRC | 13:48 | |
| *** avass has joined #zuul | 13:50 | |
| avass | corvus: this is ready to merge right? https://review.opendev.org/c/zuul/zuul-jobs/+/782004 | 13:50 |
| corvus | avass: yep! | 13:50 |
| avass | zuul-jobs test-requirements.txt has been a bit flaky today. sometimes tox-py27 gets different versions of openstacksdk (version >0.45.0 doesn't support python2.7) so hopefully that'll merge | 13:52 |
| zbr | avass: py27 are failing everywhere for various reasons since last week. Most of my patches that had a py27 job failed. | 13:54 |
| zbr | also zuul-quick-start fails to start too: Status code was -1 and not [200]: Connection failure: [Errno 104] Connection reset by peer | 13:54 |
| zbr | https://zuul.opendev.org/t/zuul/builds?job_name=zuul-quick-start&project=zuul/zuul | 13:54 |
| corvus | avass: that's a little confusing -- sdk 46 was released in april; we've been pinned to <48 which was released in june | 13:55 |
| avass | corvus: I compared two jobs that succeeded vs failed and the succeeding one got 0.45.0 and anything >0.45.0 says it's only compatible with >=python3.5 | 13:57 |
| avass | I could be missing something | 13:57 |
| avass | for example: https://zuul.opendev.org/t/zuul/build/4ff471fa35b24d149f76c2cae1fae9bf/log/tox/py27-1.log#13 | 13:58 |
| avass | vs: https://zuul.opendev.org/t/zuul/build/c28e3d1672d04dbdad7924ebba93fe83/log/tox/py27-1.log#13 | 13:59 |
| avass | and it errors on 0.47.0 not being compatible with python2.7 | 13:59 |
| avass | it's pulling the package from two different mirrors however | 14:00 |
| corvus | avass: i agree, sdk >45 requires py3. perhaps some other package has further constrained sdk to <46 for the past 9 months and that constraint was just removed. | 14:02 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Update operator-framwork to v1.4.2 https://review.opendev.org/c/zuul/zuul-operator/+/784457 | 14:09 |
| avass | I couldn't figure out how to constrain the package version with a python_version range however. I'd expect it to be something like this but pip doesn't like it: https://review.opendev.org/c/zuul/zuul-jobs/+/784894 | 14:11 |
| corvus | fungi, clarkb, mordred: ^? | 14:13 |
| mordred | looking | 14:14 |
| mordred | I don't know - that looks reasonable to me - maybe clarkb or fungi know | 14:17 |
| mordred | could also just go for simplicity and pin to <0.46 for python_version <= 3.5 | 14:18 |
| avass | that fails with "Double requirement given: ..." instead | 14:19 |
| avass | because of the <3 as well | 14:19 |
| mordred | corvus, avass : see fungi's comment in #opendev ... this could actually be a mirror issue | 14:19 |
| mordred | (the "why did this start failing all of a sudden") | 14:19 |
| avass | clarkb: I checked with daniel and the problem we saw was actually a problem where if you stacked two changes on top of eachother and the second change deletes a playbook along with the job using it zuul still tries to run that job and errors because it can't find the playbook. I'm not sure if that's related to your issue | 14:25 |
| avass | I think we had something similar a while ago related to branches but I can't remember what the situation looked like then but it gave a similar error from what I remember. | 14:27 |
| fungi | corvus: mordred: to be clear, i haven't looked into it yet. the symptoms sound similar to when fastly has directed clients to the bandersnatch mirror pypi uses as a fallback for when warehouse isn't reachable, and someone said there was a pypi outage over the weekend | 14:27 |
| fungi | we've also seen some fastly endpoints continue directing clients to the fallback instead of warehouse, so it ends up depending on where in the world you are as to which you get | 14:28 |
| avass | fungi: yeah I should have added a "could" to my message | 14:28 |
| mordred | "yay" | 14:28 |
| fungi | also the pypi admins were saying they were going to replace that bandersnatch-built fallback with something which would provide python-requires metadata in the indices, but i have no idea if they ever got around to that | 14:29 |
| corvus | fungi: yeah, seems very likely it's somehow related; though it's also true that we don't have a py 2.x cap, and so must have somehow been getting that some other way. but maybe whatever other way that is is still in effect and when the mirror is normal again we won't need avass's patch? | 14:29 |
| corvus | (and, moreover, that avass's patch may not be working (or would not work in this case) because of the mirror issue?) | 14:30 |
| corvus | so i guess we can't really evaluate that patch until the mirror situation stabilizes | 14:31 |
| avass | This is a bit harder to debug however: https://zuul.opendev.org/t/zuul/build/add02ab2cb5247ad9a9caa8a8d1c0ae0 | 14:40 |
| corvus | avass: post failure, no logs :( | 14:41 |
| corvus | hopefully that's just a one-off failure? | 14:41 |
| avass | I guessing it is | 14:41 |
| clarkb | avass: in this case it is running a job with a second repo included but that second repo doesn't have a change to update to. We expect it to update to latest $branch but that doesn't seem to happen because the code revision for lastest $branch is laraedy there due to a fast forward emrge | 14:47 |
| clarkb | avass: mordred I don't know why that metadata on the requirement doesn't work | 14:48 |
| fungi | right, has to be fast-forwardable, so that both the previous and new branch states are implicitly present | 14:48 |
| clarkb | maybe >=3 needs to be before <= 3.6 and it is order dependnet? | 14:48 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Update operator-framwork to v1.4.2 https://review.opendev.org/c/zuul/zuul-operator/+/784457 | 14:49 |
| avass | tried that and pip just doesn't seem to like multiple version constraints for python_version, "...;python_version<'3.6;python_version>='3'" doesn't work either :/ | 14:50 |
| clarkb | https://www.python.org/dev/peps/pep-0508/ has the full grammar | 14:52 |
| mordred | avass: oh - wait ... | 14:52 |
| clarkb | I think you can use and | 14:52 |
| mordred | yeah | 14:52 |
| clarkb | python_version>=3 and python_version<3.6 or similar | 14:52 |
| mordred | clarkb beat me to it | 14:53 |
| avass | that works | 14:55 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Use openstacksdk 0.45.0 for python2.7 https://review.opendev.org/c/zuul/zuul-jobs/+/784894 | 14:55 |
| *** y2kenny has joined #zuul | 14:58 | |
| *** bhavikdbavishi has joined #zuul | 15:37 | |
| avass | there could be problems with log uploads: https://zuul.opendev.org/t/zuul/buildset/da4c8da2559c4840ac877d4b9519064e | 15:37 |
| corvus | avass: yeah, fungi and clarkb are looking at it in #opendev | 15:40 |
| fungi | should hopefully be resolved in a few minutes | 15:41 |
| fungi | looks like one of our storage donors is returning errors when we try to authenticate in one region | 15:41 |
| -openstackstatus- NOTICE: POST_FAILURE results between 14:00 and 15:50 UTC can be safely rechecked, and were due to authentication problems in one of our storage donor regions | 15:55 | |
| *** ykarel has quit IRC | 16:11 | |
| *** hamalq has joined #zuul | 16:13 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Update operator-framwork to v1.4.2 https://review.opendev.org/c/zuul/zuul-operator/+/784457 | 16:25 |
| clarkb | more zuul users https://superuser.openstack.org/articles/ntt-and-kddi-challenges-for-sustainable-infrastructure-transformation/ | 16:31 |
| fungi | a synopsis at https://zuul-ci.org/users.html would be good | 16:38 |
| *** jpena is now known as jpena|off | 16:58 | |
| *** jcapitao has quit IRC | 17:00 | |
| *** jcapitao has joined #zuul | 17:02 | |
| *** jcapitao has quit IRC | 17:05 | |
| y2kenny | what do you guys use for get gerrit event feed into IRC? | 17:05 |
| clarkb | y2kenny: https://opendev.org/opendev/gerritbot there are opendevorg/gerritbot docker images too iirc | 17:06 |
| y2kenny | clarkb: thanks. I will check it out | 17:06 |
| clarkb | it listens to ssh stream events and then based on a yaml config decides which events to send to irc channels | 17:07 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Remove command args override and update operator-framework to v1.4.2 https://review.opendev.org/c/zuul/zuul-operator/+/784181 | 17:25 |
| *** bhavikdbavishi1 has joined #zuul | 17:25 | |
| tristanC | avass: corvus: fyi, if i understand correctly, the kubernetes network issue i mentioned previously is happening with the old operator-framework version used by the zuul-operator. I squahed the sdk bump with the args override fix in 784181 which should now pass the ci | 17:27 |
| *** bhavikdbavishi has quit IRC | 17:27 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 17:27 | |
| *** sanjayu_ has quit IRC | 17:35 | |
| *** saneax has joined #zuul | 17:50 | |
| *** saneax has quit IRC | 17:55 | |
| *** y2kenny has quit IRC | 17:59 | |
| fungi | now that we're well past the holiday weekend, i can see increased cache memory occupied on opendev's scheduler, but active memory use remains steady and is not increasing: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=64792&rra_id=all | 18:35 |
| fungi | semaphore leak fix seems to be holding up | 18:36 |
| *** bhavikdbavishi has quit IRC | 18:42 | |
| *** bhavikdbavishi has joined #zuul | 18:43 | |
| *** bhavikdbavishi has quit IRC | 18:53 | |
| *** y2kenny has joined #zuul | 18:56 | |
| openstackgerrit | Jeremy Stanley proposed zuul/zuul-website master: User story from KDDI, NEC, and NTT https://review.opendev.org/c/zuul/zuul-website/+/785033 | 19:04 |
| y2kenny | corvus: For https://review.opendev.org/c/zuul/zuul/+/783986, are there anything else needed before it can go in? | 19:05 |
| avass | tobiash: I've been able to hack the docker daemon a bit to allow a "--image" flag that it should pull on startup. It doesn't look like it should be too hard to use the docker packages to create a cli to do that without the daemon so I'll continue looking at that whenever I have time :) | 19:10 |
| avass | tristanC: nice | 19:15 |
| *** hamalq has quit IRC | 19:44 | |
| *** hamalq has joined #zuul | 19:45 | |
| *** josefwells has joined #zuul | 19:47 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-operator master: WIP: Use kopf operator framework https://review.opendev.org/c/zuul/zuul-operator/+/785039 | 20:02 |
| corvus | tristanC: +3; also ^ | 20:07 |
| corvus | i'm sure that's not going to pass tests on the first try :) | 20:12 |
| tristanC | corvus: great, that looks good to me. but you removed the tests | 20:13 |
| corvus | tristanC: oh then maybe they will pass! ;) | 20:13 |
| corvus | sorry that was not intentional | 20:13 |
| corvus | i really thought i kept them, but i think i got turned around in a rebase or something; i'll fix | 20:14 |
| tristanC | corvus: if i remember correctly, one of the operatorsdk's benefit is that it manages resource ownership, how does your implementation sets the owner references? | 20:19 |
| corvus | tristanC: kopf's "adopt()" method does that; in the utils file, when we send all the objects to k8s, we adopt them first, so everything created by the operator is owned by the original zuul crd | 20:20 |
| corvus | er zuul cr not crd :) | 20:20 |
| corvus | https://review.opendev.org/c/zuul/zuul-operator/+/785039/1/zuul_operator/utils.py#45 | 20:21 |
| tristanC | corvus: i see, thanks | 20:24 |
| openstackgerrit | Merged zuul/zuul-operator master: Remove command args override and update operator-framework to v1.4.2 https://review.opendev.org/c/zuul/zuul-operator/+/784181 | 20:35 |
| openstackgerrit | James E. Blair proposed zuul/zuul-operator master: WIP: Use kopf operator framework https://review.opendev.org/c/zuul/zuul-operator/+/785039 | 20:46 |
| openstackgerrit | James E. Blair proposed zuul/zuul-operator master: Bump API version to v1alpha2 https://review.opendev.org/c/zuul/zuul-operator/+/785047 | 20:59 |
| corvus | i think i might continue work with followup changes and squash them, just to avoid having the initial change get bigger right now | 20:59 |
| corvus | mnaser: https://gerrit.googlesource.com/zuul/ops/+/refs/heads/master/k8s has tls zk; you may be able to crib from that to prepare for the upgrade. and remember 3.19.x supports tls zk already so you can do that first. | 21:34 |
| corvus | clarkb, fungi: before you page out all the knowledge about semaphores we obtained last week -- can you take a look at https://review.opendev.org/784523 and see if you think that would have helped? | 21:41 |
| *** avass has quit IRC | 21:44 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Add a checkpoint release note https://review.opendev.org/c/zuul/zuul/+/785054 | 21:58 |
| corvus | i think we're about to wrap up the "event processing in zk" phase of the sos work; i'd like to go ahead and merge those outstanding changes, restart opendev on them, and release a checkpoint (presumably 4.2.0) so we have something to fall back on for the next phase (as we did with 4.1.0 recently) | 21:59 |
| clarkb | corvus: looking | 22:10 |
| corvus | tristanC, mordred: it sort of looks like the job failed due to pxc taking too long -- https://zuul.opendev.org/t/zuul/build/ea03794c933a41c880c51792d7371f8e/log/docker/k8s_operator_zuul-operator-f74b88756-x2mzl_default_f80b73a4-08c9-4f3f-81df-57b47edfc0d9_0.txt | 22:12 |
| corvus | we only give it 4 minutes; i think we'll just need to bump the timeout, as booting a 3-node pxc cluster can take quite a while | 22:13 |
| clarkb | corvus: from the commit message it makes sense. I'll dig into the code too though and leave a proper review | 22:13 |
| clarkb | corvus: note we should coordinate with openstack and airship relaeses this week if we restart opendev | 22:13 |
| clarkb | as they are both trying to get big release artifacts out the door | 22:14 |
| corvus | clarkb: thanks for looking; and noted; i think we can restart either wednesday or friday and try to minimize disruption there. | 22:15 |
| fungi | corvus: on a leak-ish note, not sure if you saw my comments over the weekend but we're continuing to see stuck node requests where launchers take the lock and never report completion back to the scheduler. i'm at a loss as to how to debug further... thread dump maybe? | 22:16 |
| corvus | fungi: yeah, that sounds promising | 22:17 |
| fungi | i'll try to grab one before the next time i restart it | 22:18 |
| fungi | thanks | 22:18 |
| openstackgerrit | James E. Blair proposed zuul/zuul-operator master: WIP: Use kopf operator framework https://review.opendev.org/c/zuul/zuul-operator/+/785039 | 22:19 |
| *** rlandy is now known as rlandy|bbl | 22:20 | |
| *** holser has quit IRC | 22:23 | |
| tristanC | corvus: there seems to be a connection error for cert manager: https://0c87aaef96fbe9fbbe9d-b9bca1a4764ecffb2935d8726176eb4d.ssl.cf2.rackcdn.com/785039/2/check/zuul-operator-functional-k8s/ea03794/docker/k8s_operator_zuul-operator-f74b88756-x2mzl_default_f80b73a4-08c9-4f3f-81df-57b47edfc0d9_0.txt | 22:28 |
| *** y2kenny has quit IRC | 22:29 | |
| clarkb | corvus: I'm trying to convince myself that union is the right operation at https://review.opendev.org/c/zuul/zuul/+/784523/1/zuul/zk/semaphore.py line 178. Should it be intersection? | 22:39 |
| clarkb | intersection will give you the holders that are still holding from the first set without jobs that have stopped or started which the comments in the method imply is what you want? | 22:39 |
| corvus | tristanC: oh yeah, i think that's a transient error; afaict there's no way to determine that it's ready except to try and retry :( there's a cert-manager bug report about it and they say the same. i tried to minimize that, but it's timing dependent; we may be able to just sleep a couple more seconds after it looks like it's ready and reduce the chances we hit it further. but you can see the retry in | 22:43 |
| corvus | action there -- the next log line is a few seconds later and it starts the handler all agoin | 22:43 |
| clarkb | left that question in a review too if it is easier to look at it in that context | 22:43 |
| corvus | clarkb: looking now | 22:43 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Periodically clean up leaked semaphores https://review.opendev.org/c/zuul/zuul/+/784523 | 22:47 |
| corvus | clarkb: yep, i got that wrong, thanks :) | 22:47 |
| clarkb | I knew I took group theory for a reason | 22:50 |
| fungi | i had logic in two separate departments at school and still missed that | 22:53 |
| corvus | me going to csc310: https://1.bp.blogspot.com/-AZNV9I1nBF4/UB6SvKtIR6I/AAAAAAAAAWY/fk8_oWh9JCI/s640/midvale+school+for+the+gifted.jpg | 22:55 |
| clarkb | it was interseting to get the same math subjects from two departments. In CS land it was all practical application of those topics to things relevant to CS. Parsing, algorithm analysis etc. In the math department it was write all the proofs | 23:01 |
| fungi | yeah, i took logic in both math and philosophy departments just to compare them | 23:02 |
| fungi | it was interesting. a lot of it was the same, just using different notation | 23:02 |
| clarkb | and in physics its was "this math is really difficult and I don't want you to have to take differential equiations so we're going to spend a week teaching you just enough to figure out capacitance" | 23:03 |
| fungi | i recall introduction to astrophysics, the professor only cared that your answers were the right order of magnitude | 23:07 |
| corvus | fungi: i have it on good authority that "space is big" | 23:09 |
| corvus | read that in a guidebook somewhere | 23:09 |
| openstackgerrit | James E. Blair proposed zuul/zuul-operator master: WIP: Use kopf operator framework https://review.opendev.org/c/zuul/zuul-operator/+/785039 | 23:11 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
