| *** jamesmcarthur has joined #zuul | 00:09 | |
| *** michael-beaver has quit IRC | 00:10 | |
| *** ianychoi has quit IRC | 00:12 | |
| *** rlandy has quit IRC | 00:13 | |
| *** ianychoi has joined #zuul | 00:15 | |
| *** jamesmcarthur has quit IRC | 00:24 | |
| *** ianychoi has quit IRC | 00:32 | |
| *** jamesmcarthur has joined #zuul | 00:35 | |
| *** ianychoi has joined #zuul | 00:48 | |
| *** jamesmcarthur has quit IRC | 00:49 | |
| *** jamesmcarthur has joined #zuul | 00:51 | |
| *** jamesmcarthur has quit IRC | 00:54 | |
| *** jamesmcarthur has joined #zuul | 00:55 | |
| *** jamesmcarthur has quit IRC | 01:07 | |
| *** igordc has quit IRC | 02:01 | |
| *** jamesmcarthur has joined #zuul | 02:18 | |
| *** jamesmcarthur has quit IRC | 02:56 | |
| *** jamesmcarthur has joined #zuul | 02:58 | |
| *** bhavikdbavishi has joined #zuul | 03:10 | |
| *** bhavikdbavishi1 has joined #zuul | 03:15 | |
| *** bhavikdbavishi has quit IRC | 03:17 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 03:17 | |
| daniel2 | fungi: I added privileged, but the container itself is built without any sudo ability. | 03:44 |
|---|---|---|
| *** jamesmcarthur has quit IRC | 03:45 | |
| *** jamesmcarthur has joined #zuul | 03:47 | |
| *** jamesmcarthur has quit IRC | 03:57 | |
| *** jamesmcarthur has joined #zuul | 03:57 | |
| daniel2 | Beat the system. Injected a new sudoers file. | 03:58 |
| *** jamesmcarthur has quit IRC | 04:02 | |
| *** jamesmcarthur has joined #zuul | 04:33 | |
| *** jamesmcarthur has quit IRC | 04:40 | |
| *** saneax has joined #zuul | 04:46 | |
| *** spsurya has joined #zuul | 05:05 | |
| *** jamesmcarthur has joined #zuul | 05:17 | |
| *** jamesmcarthur has quit IRC | 05:23 | |
| *** jamesmcarthur has joined #zuul | 06:19 | |
| *** ianychoi has quit IRC | 06:20 | |
| *** ianychoi has joined #zuul | 06:23 | |
| *** ianychoi has quit IRC | 06:29 | |
| *** jamesmcarthur has quit IRC | 06:30 | |
| *** ianychoi has joined #zuul | 06:31 | |
| *** saneax has quit IRC | 06:34 | |
| *** jamesmcarthur has joined #zuul | 06:49 | |
| *** jamesmcarthur has quit IRC | 06:58 | |
| *** jamesmcarthur has joined #zuul | 07:01 | |
| *** tosky has joined #zuul | 07:16 | |
| *** hashar has joined #zuul | 07:24 | |
| *** pcaruana has joined #zuul | 07:28 | |
| *** ianychoi has quit IRC | 07:45 | |
| *** jangutter has joined #zuul | 07:46 | |
| *** jpena|off is now known as jpena | 07:49 | |
| *** avass has joined #zuul | 08:07 | |
| *** gtema has joined #zuul | 08:11 | |
| *** jamesmcarthur has quit IRC | 08:17 | |
| *** panda|off is now known as panda | 08:20 | |
| *** recheck_ has quit IRC | 08:22 | |
| *** recheck has joined #zuul | 08:22 | |
| *** recheck has quit IRC | 08:25 | |
| *** recheck has joined #zuul | 08:25 | |
| *** recheck has quit IRC | 08:27 | |
| *** recheck has joined #zuul | 08:27 | |
| openstackgerrit | Gabor Lekeny proposed zuul/zuul master: Fix github webhook authentication https://review.opendev.org/686355 | 08:34 |
| openstackgerrit | Gabor Lekeny proposed zuul/zuul master: Fix github webhook authentication https://review.opendev.org/686355 | 08:38 |
| mordred | tristanC: comment left on https://review.opendev.org/#/c/686249 | 08:58 |
| mordred | tristanC: oh - I see you discussed that in scrollback. might be worth a comment in the tox.ini about it anyway | 08:59 |
| mordred | also - I'd LOVE to add type annotations to openstacksdk - have been waiting until the "drop 2.7" moment | 08:59 |
| mordred | which, AIUI - is actually now, since train is released | 09:00 |
| mordred | tristanC: so if you get bored and want to make an initial patch to openstacksdk adding enough annotations for zuul-registry to be happy - it will provide happiness | 09:02 |
| mordred | tobiash: I'd love to figure out how to make a test that would remind us if we change statsd settings to also update the exporter mapping in https://review.opendev.org/#/c/660473/ - but I have no good ideas for that ATM | 09:05 |
| openstackgerrit | Merged zuul/zuul-registry master: Add tox configuration and fixe flake8 errors https://review.opendev.org/686230 | 09:08 |
| tobiash | mordred: I've no idea either | 09:11 |
| tobiash | zuul-maint: I'll be ooo for several weeks | 09:13 |
| *** jamesmcarthur has joined #zuul | 09:14 | |
| mordred | tobiash: vacation? | 09:15 |
| tobiash | Bike accident :( | 09:15 |
| mordred | ugh. :( | 09:15 |
| mordred | I had a minor one of those 1.5 years ago and only hurt my wrist but it still sucked - hope all gets better! | 09:16 |
| tobiash | Finger broken, at least four weeks single handed... | 09:17 |
| *** jamesmcarthur has quit IRC | 09:18 | |
| mordred | *ugh* that sounds horrible | 09:21 |
| openstackgerrit | Merged zuul/nodepool master: Add statsd_exporter mapping https://review.opendev.org/660473 | 10:14 |
| *** jamesmcarthur has joined #zuul | 10:15 | |
| *** jamesmcarthur has quit IRC | 10:19 | |
| *** mgoddard has quit IRC | 10:23 | |
| *** mgoddard has joined #zuul | 10:26 | |
| *** saneax has joined #zuul | 10:45 | |
| *** jpena is now known as jpena|lunch | 11:00 | |
| openstackgerrit | Gabor Lekeny proposed zuul/zuul master: Fix github webhook authentication https://review.opendev.org/686355 | 11:10 |
| *** bhavikdbavishi has quit IRC | 11:11 | |
| *** jamesmcarthur has joined #zuul | 11:16 | |
| *** jamesmcarthur has quit IRC | 11:20 | |
| *** jamesmcarthur has joined #zuul | 11:26 | |
| *** jamesmcarthur has quit IRC | 11:35 | |
| *** jamesmcarthur has joined #zuul | 11:37 | |
| fungi | tobiash: sorry to hear that, wishing your finger a speedy recovery | 11:39 |
| *** tosky_ has joined #zuul | 11:44 | |
| *** tosky has quit IRC | 11:46 | |
| *** jamesmcarthur has quit IRC | 11:51 | |
| *** jamesmcarthur has joined #zuul | 11:52 | |
| *** saneax has quit IRC | 11:54 | |
| *** jamesmcarthur has quit IRC | 11:59 | |
| *** tosky_ is now known as tosky | 12:01 | |
| *** jamesmcarthur has joined #zuul | 12:07 | |
| *** jpena|lunch is now known as jpena | 12:12 | |
| *** jamesmcarthur has quit IRC | 12:19 | |
| *** jamesmcarthur has joined #zuul | 12:20 | |
| *** nhicher has quit IRC | 12:20 | |
| fungi | anybody happen to know of any zuul users who might be attending the open infra summit and/or project teams gathering in shanghai next month? | 12:26 |
| *** nhicher has joined #zuul | 12:27 | |
| mhu | Hello, there are a few patches still open before the backend side of authz/n in zuul is completed, could they get some reviews please? | 12:30 |
| mhu | https://review.opendev.org/#/c/641099/ and https://review.opendev.org/642408 and https://review.opendev.org/684790 | 12:30 |
| *** jamesmcarthur has quit IRC | 12:33 | |
| *** rlandy has joined #zuul | 12:34 | |
| mordred | fungi: I take it I don't count? | 12:34 |
| mordred | mhu: looking | 12:34 |
| fungi | mordred: sure you count! the questing is, are you going? ;) | 12:35 |
| mhu | fungi: I'll represent software factory users (RDO, OSP downstream CI among others) FWIW | 12:36 |
| openstackgerrit | Monty Taylor proposed zuul/zuul master: Authorization rules: support YAML nested dictionaries https://review.opendev.org/684790 | 12:38 |
| fungi | mhu: awesome! thanks | 12:39 |
| mordred | fungi: yes! | 12:40 |
| mordred | mhu: all lgtm - I rebased the third, it was on top of a previous patchset | 12:40 |
| mhu | thanks! | 12:40 |
| * Shrews would love to get https://review.opendev.org/681857 merged today for an eventual builder restart | 12:47 | |
| *** jamesmcarthur has joined #zuul | 12:52 | |
| mordred | you're a builder restart | 12:52 |
| mordred | SpamapS: done. (I was actually in the middle of looking at that already) | 12:53 |
| mordred | gah. I meant Shrews | 12:55 |
| Shrews | mordred: SpamapS and I are actually the same person | 12:56 |
| tristanC | mordred: nice, i can have a look for openstacksdk then. I think it's a worthy effort, types are super powerful imo. At least for documentation purpose :) | 12:58 |
| mordred | tristanC: I agree. I have, unfortunately, found that we do something in openstacksdk that's going to make it almost impossible to add type hints | 12:59 |
| mordred | tristanC: I'm pondering ways to change it - because I also think it's ultimately a runtime cost that could be a CI or build-time cost instead ... but I've only had one coffee today, so I'm not sure I'm fully smart enough | 13:00 |
| mordred | Shrews: I've thought that for many years | 13:00 |
| Shrews | one of us is the evil, alternate persona. i won't say which | 13:01 |
| mordred | Shrews: only one? or maybe both are? | 13:02 |
| SpamapS | Shrews: your side of the shared consciousness gets much better bbq than my side tho. | 13:05 |
| tristanC | mordred: yeah, some python constructs are difficult to type without inducing some runtime cost | 13:06 |
| mordred | tristanC: the biggest issue we have is with openstack._meta.connection.ConnectionMeta - which is a metaclass that adds a descriptor for each service and alias in os-service-types | 13:07 |
| mordred | mapping it to the proxy object if one exists, or to the generic if it doesn't | 13:08 |
| * SpamapS quite enjoys mypy-ing code... if that's something that needs doing | 13:08 | |
| mordred | tristanC: when you add a py.typed file to openstacksdk - running mypy in zuul-registry gets you: | 13:08 |
| tristanC | SpamapS: we are discussing https://review.opendev.org/#/c/686249/3 | 13:08 |
| mordred | zuul_registry/swift.py:62: error: "Connection" has no attribute "object_store" | 13:09 |
| mordred | tristanC: because there is no object_store attribute on Connection until the metaclass runs | 13:09 |
| mordred | now - I'm not really sure why mypy doesn't run metaclasses - since those are ultimately part of the compile step - but such is life | 13:10 |
| tristanC | mordred: how did you typed the connection object? | 13:11 |
| mordred | tristanC: remote: https://review.opendev.org/686397 WIP Start supporting type info | 13:11 |
| mordred | tristanC: so what I'm thinking is ditching the metaclass (since it also makes the code harder to understand) - just actually putting entries for each service in the file - and making a utility script that uses os-service-types we could use to generate the list / update it when things change | 13:14 |
| SpamapS | IIRC you need to use a protocol for that | 13:16 |
| mordred | SpamapS: you said words that I don't understand | 13:16 |
| SpamapS | https://mypy.readthedocs.io/en/latest/protocols.html | 13:16 |
| SpamapS | https://mypy.readthedocs.io/en/latest/protocols.html#simple-user-defined-protocols specifically | 13:17 |
| tristanC | Could it be that the ConnectionMeta object is considered "arbitrary metaclass code" which is not supported by mypy according to https://mypy.readthedocs.io/en/latest/metaclasses.html#gotchas-and-limitations-of-metaclass-support ? | 13:17 |
| SpamapS | So you will want something like SupportsObjectStore(Protocol) | 13:18 |
| mordred | SpamapS: no - that's a different issue - the problem here is that there is no attribute that exists to attach a type to in the first pace | 13:18 |
| mordred | tristanC: yes, I thnk that's more likely the case | 13:18 |
| SpamapS | mordred: IIRC that's why you use protocols, because the protocol will say what type the attribute will have if/when it exists. | 13:18 |
| mordred | yeah. there's just no attribute | 13:19 |
| mordred | we could actually assign a real type to it - there's just no handle to whcih to attach either a real type or a protocol | 13:19 |
| mordred | this is why I think the metaclass in this case is not enough of a win to justify keeping | 13:20 |
| SpamapS | The things that are expected to have the attribute, will be the ones you attach the protocol to. | 13:20 |
| SpamapS | But yeah, if it's just as easy as inheriting some base class that defines the attribute instead of metaclass-fancy-making ... inheritance seems the simpler way forward. | 13:21 |
| mordred | SpamapS: yah - but ... I don't know what the attribute is without running the metaclass code to generate the list of attributes | 13:21 |
| mordred | exactly | 13:21 |
| SpamapS | Yeah the point of typing is to make the code easier to read and understand.. and from my experience, metaclasses do the opposite. ;) | 13:21 |
| mordred | SpamapS: these are currently a dynamic list of attribute names based on the list of known service types | 13:21 |
| tristanC | mordred: how is the attribute object_store set in the end? | 13:22 |
| mordred | the metaclass attaches it to the dct namespace dictionary of the Connection class | 13:23 |
| SpamapS | Yeah seems like just having a non-functional attribute there and then making sure to replace any hasattr's with .foo is not None would result in better readability. | 13:24 |
| mordred | it's adding a ServiceDescriptor object, which will ultimately return an openstack.object_store.v1._proxy.Proxy - so if we staticly define the attributes, we should be able to mark them with the correct final proxy classes | 13:24 |
| mordred | SpamapS: yah | 13:24 |
| mordred | well - to be fair - the attributes are always there | 13:24 |
| SpamapS | Then your type is Optional[Proxy] | 13:24 |
| mordred | it will *always* be a Proxy | 13:25 |
| SpamapS | oh so yeah.. ditch that meta business | 13:25 |
| mordred | it's a very non-dynamic dynamic metaclass | 13:25 |
| tristanC | it seems like that meta business could be taken care of by a one time script | 13:26 |
| mordred | yup | 13:26 |
| mordred | I honesty think it will be an improvement all the way around | 13:26 |
| fungi | is there a short writeup somewhere on the benefits of typed objects with python? improved readability? helps to catch latent bugs? performance improvements? | 13:32 |
| tristanC | mordred: in the meantime we can fake an object_store attribute for the Connection type, then we need to annotate the get_endpoint and upload_object. The rest of zuul_registry.swift use the more low level session.get|delete which will also needs to be annotated | 13:33 |
| pabelanger | job names are unique across tenants too, right? eg: I cannot have 2 jobs named foo IIRC. Basically, I'd like tenantB:foo to parent to tenantA:foo | 13:35 |
| tristanC | fungi: https://realpython.com/python-type-checking/ has a nice Pros/Cons and some useful examples too | 13:35 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-registry master: WIP Consume typing from openstacksdk and keystoneauth https://review.opendev.org/686403 | 13:35 |
| mordred | tristanC: yeah. I've got some stub patches up now ^^ | 13:36 |
| fungi | tristanC: thanks, the description at https://pypi.org/project/mypy/ was a little light on detail | 13:36 |
| tristanC | fungi: and well, https://en.wikipedia.org/wiki/Type_theory#Practical_impact has some useful content too :-) | 13:38 |
| fungi | nice, thanks again | 13:38 |
| fungi | after so much time working in strictly-typed languages i found the flexible/dynamic typing in python refreshing, but i understand it's not without its down-sides | 13:40 |
| *** jamesmcarthur has quit IRC | 13:40 | |
| mordred | pabelanger: tenants are completely isolated - so you should be able to have a foo jobs in both tenants. however, tenants are completely isolated, so tenantB:foo would not be able to see tenantA:foo to be able to use it as a parent | 13:42 |
| mordred | pabelanger: you can, of course, have the repo that defines foo loaded in both tenantA and tenantB | 13:42 |
| tristanC | fungi: most of the time mypy figure out dynamic type, and adding type annotation to function definition is really handy to understand the code. It's kind of the :arg: and :return: doc string content, but checked | 13:55 |
| *** gtema has quit IRC | 13:55 | |
| *** jamesmcarthur has joined #zuul | 14:14 | |
| *** leifmadsen has left #zuul | 14:22 | |
| fungi | yeah, pretty cool | 14:23 |
| *** hashar has quit IRC | 14:24 | |
| fungi | i just especially enjoy the luxury of not having to do things like recast an int as a float before performing mathematical operations, or recast a byte array as a boolean for conditional checks | 14:24 |
| fungi | the automatic type coercion in python is really pleasant | 14:25 |
| Shrews | until it's not | 14:31 |
| fungi | it's a balancing act | 14:34 |
| *** rfolco is now known as rfolco|bbl | 14:35 | |
| *** jamesmcarthur has quit IRC | 14:35 | |
| fungi | i do appreciate the idea of being able to declare strict typing in places where it might be important and allow loose typing in places where that's more convenient | 14:36 |
| Shrews | for me, it's a directly proportional relationship between how complex a program is and how much I prefer static typing. | 14:38 |
| fungi | i can definitely see that | 14:39 |
| *** jamesmcarthur has joined #zuul | 14:40 | |
| *** jamesmcarthur has quit IRC | 14:41 | |
| *** jamesmcarthur has joined #zuul | 14:44 | |
| *** bhavikdbavishi has joined #zuul | 14:50 | |
| *** jamesmcarthur has quit IRC | 14:57 | |
| *** jpena is now known as jpena|off | 14:59 | |
| *** jamesmcarthur has joined #zuul | 15:00 | |
| *** bhavikdbavishi1 has joined #zuul | 15:18 | |
| *** bhavikdbavishi has quit IRC | 15:20 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 15:20 | |
| *** jamesmcarthur has quit IRC | 15:21 | |
| openstackgerrit | Nicolas Hicher proposed zuul/zuul-jobs master: DNM: debug console issue when iptables-save is called https://review.opendev.org/686424 | 15:29 |
| *** jamesmcarthur has joined #zuul | 15:30 | |
| *** mattw4 has joined #zuul | 15:31 | |
| *** michael-beaver has joined #zuul | 15:33 | |
| tristanC | mordred: i'll push a couple of changes shortly on your py.typed ones to remove the type ignore from zuul_registry.swift module | 15:42 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-registry master: Add type annotations https://review.opendev.org/686249 | 15:59 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-registry master: WIP: use openstacksdk wip annotation https://review.opendev.org/686434 | 15:59 |
| *** rfolco|bbl is now known as rfolco | 16:00 | |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: WIP - Gitlab - Basic handling of merge_requests event https://review.opendev.org/685990 | 16:07 |
| SpamapS | fungi: for those wild and crazy kids who use editors with type-ahead assist, static typing actually is a bigger win, because they type variable1. and their editor often knows the methods/attributes and can help them out. | 16:21 |
| SpamapS | So type-annotating just with the basic python types is actually a good way to facilitate teaching. | 16:22 |
| fungi | oh, neat, i hadn't considered that | 16:22 |
| SpamapS | Also I imagine a well-annotated python code-base eases the pressure toward 100% unit test coverage. | 16:24 |
| tosky | I'd say that could be a slippery slope :) | 16:25 |
| SpamapS | People just like measuring stuff. | 16:26 |
| SpamapS | If you have mypy fully stubbed out including your deps, you can probably rest on 85% unit test coverage and focus on higher value functional tests. | 16:27 |
| tosky | functional tests do not replace the need for complete unit tests; it's a pyramid | 16:28 |
| SpamapS | The problem is, there's a dogma of unit test coverage that, IMO, wastes a lot of time on busy work making sure things are covered instead of focusing on your external interfaces which are where most things break. | 16:29 |
| SpamapS | A unit test covering some code is fine and all, but it starts to be a vanity metric when you're mocking/faking the world. | 16:29 |
| tosky | it all depends on how your code is structured and where the external interfaces are (there may be internal subcomponents with almost-stable interfaces) | 16:30 |
| SpamapS | And the big thing unit test suites are good for, refactoring code, is faster and better served by static typing. | 16:30 |
| tosky | for sure I wasn't suggesting to mock the world, which is as bad | 16:30 |
| SpamapS | "Are you, mocking me?" -- 3rd-party API | 16:31 |
| tosky | uhm, I think I'm not going to totally buy your definition of what unit tests suites are good too | 16:32 |
| *** chandankumar is now known as raukadah | 16:34 | |
| *** jpena|off is now known as jpena | 16:41 | |
| *** igordc has joined #zuul | 16:46 | |
| *** tosky has quit IRC | 17:14 | |
| *** jamesmcarthur has quit IRC | 17:18 | |
| *** jamesmcarthur has joined #zuul | 17:19 | |
| tristanC | SpamapS: I find unittest less and less useful once you have type checks, and unittest may become a real pita when refactoring | 17:20 |
| *** openstackgerrit has quit IRC | 17:21 | |
| corvus | i do a happy dance whenever i have a refactoring commit that's fully tested with no changes to tests :) | 17:30 |
| fungi | that was the impetus for finally adding tests of my mud engine... i wanted to rip out and replace the entire data management layer and needed some way to catch regressions. it's a relief to be able to get confirmation that behaviors you care about in a system of reasonable complexity remain unchanged through major backend uplifts | 17:32 |
| *** openstackgerrit has joined #zuul | 17:34 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-registry master: Add streaming download support https://review.opendev.org/686449 | 17:34 |
| openstackgerrit | James E. Blair proposed zuul/zuul-registry master: Update test script to match playbook https://review.opendev.org/686450 | 17:34 |
| Shrews | corvus: let me know if my comment on 686449 is too verbose :) | 17:36 |
| corvus | Shrews: derp, thanks. i think your comment is spot on. | 17:37 |
| openstackgerrit | James E. Blair proposed zuul/zuul-registry master: Add streaming download support https://review.opendev.org/686449 | 17:38 |
| openstackgerrit | James E. Blair proposed zuul/zuul-registry master: Update test script to match playbook https://review.opendev.org/686450 | 17:38 |
| corvus | i was going to have a stream= argument to get_blob, but then i got to fighting python on generators and decided i had better things to do and just made another method | 17:40 |
| Shrews | i like the separate method better anyway. do one thing and do it well | 17:42 |
| corvus | yeah, it was an easy sell for me | 17:43 |
| Shrews | "Streaming output and also using sessions requires a good understanding of how session locks work." corvus, i'm going to assume you have that understanding and will tell me to not worry about it | 17:49 |
| corvus | Shrews: would you like me to say that? i will say anything for cookies | 17:50 |
| * Shrews hands a whole box of cookies to corvus | 17:50 | |
| corvus | Shrews: it's fine! | 17:50 |
| Shrews | lgtm then :) | 17:51 |
| corvus | Shrews: i don't believe we're using the sessions it's talking about -- i think we would need to set "tools.sessions.on = True" for that | 17:51 |
| *** jamesmcarthur has quit IRC | 17:51 | |
| *** jamesmcarthur has joined #zuul | 17:51 | |
| corvus | (so we're doing auth without sessions, which is totally a thing and easy to do with http basic auth which is what we're doing) | 17:51 |
| *** rfolco has quit IRC | 17:52 | |
| *** jamesmcarthur has quit IRC | 17:52 | |
| Shrews | If my nodepool change gets one more random failure, I'm literally going to flip my desk over in frustration | 17:57 |
| Shrews | that's 3 today, for those counting along with me | 17:58 |
| fungi | failures on image upload jobs? | 17:59 |
| Shrews | fungi: no. totally unrelated stuff | 17:59 |
| Shrews | 1 waiting for zuul, 1 waiting for ssh, and 1 waiting for k8s | 18:00 |
| corvus | Shrews: should we start looking into those? | 18:01 |
| Shrews | corvus: possibly? i've seen them all before, i just seem to be catching them all in my one change | 18:02 |
| Shrews | i commented each failure in https://review.opendev.org/681857 | 18:03 |
| *** jpena is now known as jpena|off | 18:05 | |
| paladox | cookies = biscuit? | 18:07 |
| *** spsurya has quit IRC | 18:07 | |
| corvus | yes i will also accept biscuits and digestives :) | 18:08 |
| paladox | lol | 18:09 |
| * paladox aware that cookies in american english = biscuts in the British lang :) | 18:09 | |
| paladox | cookies in britain are a actually a type of biscuit! | 18:10 |
| openstackgerrit | Tristan Cacqueray proposed zuul/nodepool master: wip: add a standalone launcher object https://review.opendev.org/639632 | 18:10 |
| paladox | wait... they have digestives in the US? | 18:11 |
| * paladox thought that was a british thing :P | 18:11 | |
| corvus | Shrews: that zuul error is weird -- it was waiting for mysql to come up, but according to the logs, mysql was up about 20 seconds after we started waiting for it... | 18:11 |
| fungi | paladox: they have digestives in the usa, you just need to find a grocer who stocks a few shelves of brit comestibles | 18:12 |
| corvus | paladox: my local store carries mcvitie's | 18:12 |
| paladox | heh | 18:12 |
| fungi | i think i can even find them at the local chemist | 18:12 |
| Shrews | corvus: is there something about that wait that might not recognize the service starting AFTER the wait? i didn't look that closely, tbh | 18:12 |
| paladox | mcvitie's are nice! | 18:13 |
| corvus | Shrews: it's the bash loop that checks to see if it can open /dev/tcp/mysql/3306 | 18:13 |
| corvus | i'm worried that it might be some obscure docker thing, but i really have no idea | 18:14 |
| corvus | this is on limestone; let's file that away | 18:15 |
| paladox | fungi do pharmacies sell food in the US? | 18:16 |
| corvus | they sell stuff you can eat but probably nothing you should | 18:16 |
| paladox | heh | 18:18 |
| * paladox knows of one pharmacie that is in sainsburys :P | 18:18 | |
| paladox | *pharmacies | 18:18 |
| fungi | yeah, they tend to be general convenience shops here | 18:18 |
| Shrews | corvus: that wait loop looks like it gave up at 2019-10-03T13:13:19 and mysql said it was ready at 2019-10-03 13:11:37. maybe it just needs more time? | 18:19 |
| Shrews | corvus: oh, nm. that's 2 minutes, not seconds. derp | 18:19 |
| Shrews | i thought we just might be cutting it close, but that's certainly enough time | 18:20 |
| fungi | so a pharmacy counter and first aid type supplies and more recently small clinics, but also often a miniature grocery (mostly shelf-stable/canned goods but sometimes also refrigerated/frozen, usually nothing fresh), greeting cards and gifts, photo studio, whatever | 18:20 |
| Shrews | fungi: we had an old timey pharmacy in Boone that used to grill burgers. it was the best | 18:21 |
| fungi | paladox: i do have vague memories of finding at least snacks and beverages in a boots in the uk though | 18:22 |
| paladox | oh, yes! | 18:22 |
| paladox | they do snacks | 18:22 |
| paladox | they once did a drive through boots here :P | 18:22 |
| paladox | that closed down last year and mcdonalds opened up | 18:22 |
| fungi | Shrews: yup, there's one in "downtown" pittsboro too if you're nostalgic for the experience | 18:23 |
| fungi | and feel like dealing with chapel hill traffic that is | 18:23 |
| Shrews | meh, it's hard to recapture the poor-college-student-desperate-for-a-between-class-cheap-burger experience | 18:24 |
| fungi | not sure i want to relive that, i can still remember it well enough on my own ;) | 18:25 |
| *** bhavikdbavishi has quit IRC | 18:25 | |
| corvus | Shrews: https://www.nytimes.com/2019/10/03/travel/what-to-do-36-hours-in-boone-north-carolina.html | 18:26 |
| paladox | London is scattered with mini shops (at least tesco's/sainsburys) | 18:26 |
| Shrews | ah, nostalgia | 18:30 |
| corvus | i remember it fondly | 18:30 |
| Shrews | corvus: i keep looking for a home just off that very street in the top pic. | 18:31 |
| Shrews | pricey and rare | 18:31 |
| corvus | maybe more so now that the nyt just told everyone about it | 18:31 |
| Shrews | they probably only published it b/c ASU beat UNC a couple of weeks ago. the hype will die down | 18:32 |
| *** recheck has quit IRC | 18:34 | |
| *** recheck has joined #zuul | 18:34 | |
| *** recheck has quit IRC | 18:37 | |
| *** recheck has joined #zuul | 18:37 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Add OpenAPI description for enqueue, dequeue, autohold https://review.opendev.org/674257 | 18:39 |
| *** jamesmcarthur has joined #zuul | 18:47 | |
| Shrews | mhu: left a question for you on 641099 | 18:49 |
| *** rfolco has joined #zuul | 18:57 | |
| * Shrews flips table | 19:05 | |
| Shrews | Ok, I'm declaring nodepool jobs unstable. I've hit the 4th "random" error on https://review.opendev.org/681857 this time on the openshift job | 19:07 |
| Shrews | tristanC: any chance you can check out the k8s and openshift failures there? ^^ maybe there is something weird with our nodesets | 19:08 |
| Shrews | or a provider | 19:09 |
| * Shrews checks provider on the failures | 19:09 | |
| tristanC | Shrews: hum let me check, it seems like it always happen on fortnebula | 19:09 |
| fungi | error: dial tcp 192.168.48.144:8443: connect: this number is no longer in service - please check the number and try again | 19:09 |
| * fungi wonders where the "dial" command comes in there | 19:10 | |
| *** hashar has joined #zuul | 19:10 | |
| Shrews | fungi: of the last 4 failures, 2 were fortnebula, 1 rax, 1 limestone | 19:12 |
| tristanC | Shrews: the two in rax and limestone seems unrelated (it failed at openshift deploy stage) probably before we landed the firewall relax fix | 19:13 |
| *** rlandy is now known as rlandy|brb | 19:13 | |
| Shrews | tristanC: where is that fw fix? | 19:13 |
| tristanC | Shrews: i'm actually unable to find a success for that job in fortnebula | 19:13 |
| tristanC | Shrews: https://review.opendev.org/672785 | 19:14 |
| *** zbr|ruck is now known as zbr | 19:14 | |
| Shrews | tristanC: that landed back in august. these failures are today. i don't see the relation | 19:15 |
| Shrews | tristanC: but if we can show it never succeeds in fortnebula, that's a great clue | 19:16 |
| tristanC | Shrews: I was looking at https://zuul.opendev.org/t/zuul/builds?job_name=nodepool-functional-openshift&result=FAILURE , where the only one that are not fortnebula date from august | 19:16 |
| tristanC | is there something special with fortnebula instance network? | 19:17 |
| Shrews | tristanC: i'm not sure. might want to ask donnyd in #openstack-infra | 19:17 |
| Shrews | does openshift require something special? | 19:18 |
| tristanC | it seems like the failure is caused by being unable to access the instance ip... | 19:19 |
| tristanC | the private_ipv4 | 19:20 |
| fungi | that provider has global ipv6 addressing and rfc 1918 "private" ipv4 addresses connected to the internet through layer 4 pat/overload nat | 19:21 |
| tristanC | more specifically, it's failing with "network is unreachable" | 19:21 |
| fungi | we collect debug info about the network configuration of each node and publish it alongside the jobs other logs, if that helps | 19:22 |
| tristanC | fungi: thanks | 19:23 |
| Shrews | i've yet to find a successful run of the openshift job in FN | 19:23 |
| tristanC | thus it seems like the nodepool host doesn't have ipv4 while the cluster does have one: | 19:23 |
| fungi | should have addresses of all initial interfaces as well as the initial routing table | 19:24 |
| tristanC | https://zuul.opendev.org/t/zuul/build/dc0765e1b481464e8d9823f3d5f7e4f0/log/zuul-info/zuul-info.launcher.txt#26 | 19:24 |
| tristanC | https://zuul.opendev.org/t/zuul/build/dc0765e1b481464e8d9823f3d5f7e4f0/log/zuul-info/zuul-info.cluster.txt#28 | 19:24 |
| logan- | i think the FN nodes run a lower mtu | 19:24 |
| fungi | yeah, but what's weird is that the "launcher" node has no ipv4 address at all on its interface | 19:25 |
| fungi | oh! the "launcher" node is fedora-29 while the "cluster" node is centos-7 | 19:27 |
| fungi | i wonder if fedora-29 is having trouble setting the ipv4 address on ens3 | 19:28 |
| fungi | not sure if those addresses are coming via dhcp or through configdrive/glean in fn | 19:29 |
| Shrews | it's the centos-7 node that gets the "network is unreachable" error though | 19:30 |
| tristanC | Shrews: i see the task is failing from the launcher node | 19:31 |
| Shrews | tristanC: so launcher cannot contact the cluster, right? | 19:31 |
| Shrews | via the v4 addr | 19:32 |
| tristanC | that's what i understand, seems like because the launcher (running fedora) doesn't get an ipv4 address | 19:32 |
| Shrews | well, it gets a private ipv4 address | 19:33 |
| Shrews | but not a public one | 19:33 |
| fungi | not that i can see | 19:34 |
| fungi | https://zuul.opendev.org/t/zuul/build/dc0765e1b481464e8d9823f3d5f7e4f0/log/zuul-info/zuul-info.launcher.txt#26-36 | 19:34 |
| tristanC | Shrews: i probably get one, but it's not showing up on the guest interface: https://zuul.opendev.org/t/zuul/build/dc0765e1b481464e8d9823f3d5f7e4f0/log/zuul-info/zuul-info.launcher.txt#26 | 19:34 |
| tristanC | it* | 19:34 |
| fungi | no ipv4 address at all, not even private, and no v4 routing table, not even a default route entry | 19:34 |
| Shrews | ansible inventory sees a private one | 19:35 |
| Shrews | https://openstack.fortnebula.com:13808/v1/AUTH_e8fd161dc34c421a979a9e6421f823e9/zuul_opendev_logs_dc0/681857/10/check/nodepool-functional-openshift/dc0765e/zuul-info/inventory.yaml | 19:35 |
| fungi | right, that's what openstack assigned | 19:35 |
| fungi | but for some reason it seems the node's operating system didn't configure it at boot | 19:35 |
| Shrews | So, is this something donnyd has to look into for us? Or are we configuring nodes poorly? | 19:36 |
| Shrews | s/configuring/building/ | 19:36 |
| * tristanC looking at glean cmd.py module | 19:36 | |
| fungi | Shrews: i'm guessing it's a problem with glean or our current f29 images in opendev | 19:37 |
| Shrews | *nod* | 19:37 |
| fungi | since the centos-7 node was able to configure its v4 address just fine | 19:37 |
| fungi | maybe if we collected /var/log/syslog or /var/log/messages (wherever those platforms log system messages by default) we could compare their interface configuration logging | 19:40 |
| Shrews | mmm, i think we do | 19:41 |
| fungi | for whatever reason we don't seem to have collected them on that run | 19:41 |
| Shrews | yeah, guess we don't then | 19:44 |
| clarkb | fungi: its probably the network manager race again | 19:45 |
| clarkb | if thr kernel gets an RA before NM starts NM ignores the interfave | 19:45 |
| fungi | is there really no way to tell nm to manage ipv4 addresses but not ipv6 addresses on an interface? ifupdown lets me set them independently, for example | 19:47 |
| tristanC | not sure if that matters, but those statements seem incorrect: https://opendev.org/opendev/glean/src/branch/master/glean/cmd.py#L132 and L125 | 19:50 |
| tristanC | at least on my local fedoras, there is Type=Ethernet for both static and dhcp interfaces | 19:51 |
| clarkb | fungi: as far as I can tell from thr many year old debian bug nope | 19:58 |
| clarkb | the issue aiui is NM doesnt want to touch an interface it thinks something else is managing | 19:58 |
| pabelanger | tristanC: I don't think TYPE is needed for Fedora, but if set will be used. Looks like will fallback to using DEVICE to setup DEVICETYPE in network-scripts | 20:00 |
| pabelanger | would need to check centos-7 / centos-8 | 20:00 |
| pabelanger | but yah, looks to simplify some code | 20:00 |
| openstackgerrit | Paul Belanger proposed zuul/zuul master: WIP: Support Ansible 2.9 https://review.opendev.org/674854 | 20:03 |
| pabelanger | zuul-maint: with ansible 2.9 around the corner, another few weeks I'd like to point humans to topic:multi-ansible-wip. That drops ansible 2.5 support, marks ansible 2.6 as deprecated and defaults zuul to use ansible 2.8! Also, adds ansible 2.9 support! Which would be awesome if we could also tag a release of zuul the day that comes out!! | 20:05 |
| pabelanger | that is all :) | 20:05 |
| corvus | mhu: have you spent much time with keycloak? | 20:11 |
| *** pcaruana has quit IRC | 20:18 | |
| *** jamesmcarthur has quit IRC | 20:21 | |
| *** rlandy|brb is now known as rlandy | 20:38 | |
| *** hashar has quit IRC | 20:42 | |
| *** tosky has joined #zuul | 20:50 | |
| corvus | tristanC: if you have a minute to review https://review.opendev.org/686449 -- we haven't merged the change to put it into production yet for opendev, so it might be nice to get that in first | 21:24 |
| tristanC | corvus: i +3 it | 21:27 |
| openstackgerrit | Merged zuul/zuul-registry master: Add streaming download support https://review.opendev.org/686449 | 21:41 |
| ianw | Shrews: I agree, seeing similar in dib gate | 21:44 |
| openstackgerrit | James E. Blair proposed zuul/zuul-registry master: Fix merge error in streaming support https://review.opendev.org/686505 | 21:47 |
| corvus | fungi, tristanC: ^ whoops. | 21:47 |
| corvus | we should add some assertions in the tests about the contents of the local image cache | 21:48 |
| fungi | yeah, that would have caught it | 21:49 |
| corvus | here's the output from the image list that shows the problem: https://zuul.opendev.org/t/zuul/build/a0c946ca9bb94fc0a85bb43c0a5606a2/console#2/1/17/ubuntu-bionic | 21:49 |
| corvus | you can see registry:2 in there | 21:50 |
| ianw | do we know why when i do say "wget https://openstack.fortnebula.com:13808/v1/AUTH_e8fd161dc34c421a979a9e6421f823e9/zuul_opendev_logs_0cf/684460/10/check/dib-nodepool-functional-openstack-fedora-29-src/0cf5dfc/nodepool/nodepool-launcher.log" i receive a compressed file? | 21:50 |
| ianw | curl --compressed i guess is the trick | 21:52 |
| pabelanger | yah | 21:52 |
| pabelanger | I believe that is because in swift, it is actually compressed | 21:53 |
| openstackgerrit | Merged zuul/zuul-registry master: Update test script to match playbook https://review.opendev.org/686450 | 21:58 |
| ianw | 2019-10-03 02:40:36,203 DEBUG nodepool.NodeLauncher: [node: 0000000000] [ 591.719783] systemd[1]: Starting Network Manager... | 21:58 |
| ianw | i feel like the inner vm's are booting too slowly? that's almost 10 minutes to get to that point | 21:58 |
| ianw | 2019-10-03 02:28:27,937 DEBUG nodepool.NodeLauncher: [node: 0000000000] [ 539.215851] systemd[1]: Started Grow root partition. | 22:00 |
| ianw | other job seemed to stop there | 22:00 |
| ianw | and yeah ... suspiciously close to -> playbooks/nodepool-functional-openstack/templates/nodepool.yaml.j2: boot-timeout: 600 | 22:01 |
| *** mattw4 has quit IRC | 22:05 | |
| *** mattw4 has joined #zuul | 22:05 | |
| tristanC | corvus: i'm rebasing the mypy patch on top of zuul-registry master (without the last fix) and it detected: zuul_registry/storage.py:126: error: "Storage" has no attribute "_path_from_digest" | 22:18 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-registry master: Add type annotations https://review.opendev.org/686249 | 22:23 |
| *** rlandy is now known as rlandy|bbl | 22:31 | |
| corvus | tristanC: are we running flake8? that sounds like something it would have caught too | 22:31 |
| tristanC | corvus: iiuc flake8 doesn't check if attribute exists | 22:33 |
| tristanC | e.g. this pass flake8 checks: http://paste.openstack.org/show/781062/ | 22:33 |
| tristanC | and yes zuul-registry is running flake8 with the same config as the one from zuul | 22:34 |
| *** panda is now known as panda|off | 22:35 | |
| corvus | tristanC: cool, so the compiling/ast steps that mypy does are more substantial than flake8 | 22:36 |
| *** mattw4 has quit IRC | 23:01 | |
| *** mattw4 has joined #zuul | 23:02 | |
| *** armstrongs has joined #zuul | 23:05 | |
| *** jlk has quit IRC | 23:07 | |
| *** armstrongs has quit IRC | 23:15 | |
| *** mhu has quit IRC | 23:20 | |
| *** jamesmcarthur has joined #zuul | 23:23 | |
| *** tosky has quit IRC | 23:23 | |
| *** jamesmcarthur has quit IRC | 23:24 | |
| *** avass has quit IRC | 23:30 | |
| *** igordc has quit IRC | 23:36 | |
| *** michael-beaver has quit IRC | 23:47 | |
| *** mattw4 has quit IRC | 23:49 | |
| pabelanger | looks like zuul_stream and python3 might not be happy, I'm seeing various [Zuul] Log Stream did not terminate in logs | 23:49 |
| pabelanger | https://object-storage-ca-ymq-1.vexxhost.net/v1/a0b4156a37f9453eb4ec7db5422272df/ansible_14/63114/8b0ba58a10cdf98b765fba04a4fd1fcd4b8e0512/third-party-check/ansible-test-network-integration-vyos-python38/acf5e7f/job-output.html#l488 | 23:49 |
| pabelanger | looks like maybe threads not shutting down cleanly | 23:49 |
| pabelanger | too late today to dig more into it | 23:54 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!