| jlk | Is there a test assert to make sure a traceback didn't happen? | 00:45 |
|---|---|---|
| jlk | oooooh | 01:02 |
| jlk | jeblair: I think we weren't quite right with pipeline driver specific requirements. I've set up a test where multiple drivers share the same pipeline each with a driver specific requirement, and the github requirement is disqualifying the gerrit change. | 01:03 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Handle change related reqs on push like events [WIP] https://review.openstack.org/469297 | 01:11 |
| jlk | jeblair: https://review.openstack.org/469297 manages to tickle this. You'd have to look at the test output because it might "pass", but there is definitely a gerrit change that is being evaluated against the github requirements. | 01:12 |
| dmsimard | mordred, pabelanger: re -- floating ip issue, this is what I'm seeing: http://paste.openstack.org/raw/611038/ | 01:51 |
| tristanC | dmsimard: mordred: pabelanger: that is with shade-1.13.2 and nodepool commit fb8bda3 | 04:17 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Decode the console log stream https://review.openstack.org/469181 | 06:05 |
| *** hashar has joined #zuul | 07:06 | |
| *** colettecello has joined #zuul | 08:56 | |
| *** mmedvede_ has joined #zuul | 09:00 | |
| *** dmellado_ has joined #zuul | 09:00 | |
| *** mmedvede has quit IRC | 09:01 | |
| *** dmellado has quit IRC | 09:01 | |
| *** gothicmindfood has quit IRC | 09:01 | |
| *** mmedvede_ is now known as mmedvede | 09:01 | |
| *** dmellado_ is now known as dmellado | 09:46 | |
| *** jkilpatr has quit IRC | 10:47 | |
| *** hashar has quit IRC | 10:47 | |
| *** jkilpatr has joined #zuul | 11:00 | |
| *** hashar has joined #zuul | 11:25 | |
| *** hashar has quit IRC | 11:30 | |
| *** hashar has joined #zuul | 11:30 | |
| mordred | tristanC: oh - so that's a very old shade and also a pretty old nodepool | 11:56 |
| mordred | tristanC: I don't know the exact cause (looking through logs) but I know there are a lot of bugfixes around floating ips since 1.13.2 | 11:57 |
| mordred | tristanC: fwiw, nodepool tag 0.2.0 should be "safe" to run if you want nodepool from before any of the zuulv3 work (that's the tag we made right before v3 work started) - and shade is safe to run with 0.2.0 at least to version 1.20.0 - 1.21 introduced a regression for 0.2.0 that I'm working on fixing - but also just added a gate job for shade to make sure that we don't accidentally break the old version of | 11:59 |
| mordred | nodepool | 11:59 |
| mordred | tristanC: in general - in newer shade, shade stops trusting the nova network cache, because it gets out ofsync, and it talks directly to neutron apis to figure out state of a server's network things | 12:00 |
| mordred | dmsimard: ^^ | 12:00 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Add config option for executor process user https://review.openstack.org/460671 | 12:19 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Add a finger protocol log streamer https://review.openstack.org/456721 | 12:19 |
| dmsimard | mordred: that's helpful, we'll look into it. Thanks. | 12:23 |
| pabelanger | dmsimard: mordred tristanC: I've been using latest shade (via latest nodepool) with rdo cloud recently. I haven't had issues | 12:28 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Add config option for executor process user https://review.openstack.org/460671 | 13:13 |
| *** dkranz has joined #zuul | 13:20 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Fix log streamer for py35 https://review.openstack.org/469480 | 13:26 |
| dmsimard | mordred: any considerations for the shade version if we're transitioning from nodepool snapshots to nodepool-builder/dib images ? 1.20.0 should be cool ? | 13:28 |
| dmsimard | pabelanger: if you know ^ | 13:30 |
| mordred | dmsimard: yes. shade version 1.20.0 should definitely work | 13:30 |
| dmsimard | mordred: ack, trying that -- thanks. | 13:30 |
| mordred | dmsimard: honestly, _latest_ shade _should_ always work - but I missed something the other day so 1.21 currently has a bug for nodepool 0.2.0 and before | 13:30 |
| mordred | (which is why I'm adding a gate job - I _want_ the answer to be "latest shade is always safe") | 13:31 |
| dmsimard | mordred: you missed something ? lies | 13:31 |
| mordred | dmsimard: IKR??? | 13:31 |
| pabelanger | dmsimard: also, the nodepool boot-from-volume patch needs a newer shade release too, so maybe just jump to latest? | 13:31 |
| mordred | dmsimard: it's also a very obviously broken thing that the gate job I'm adding would immediately catch :) | 13:31 |
| mordred | latest shade will work with any nodepool newer than 0.2.0 | 13:32 |
| mordred | fwiw | 13:32 |
| dmsimard | pabelanger: I'm going on PTO tomorrow for a week :P just trying to see if I can do enough to stabilize the situation before I head out | 13:33 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Fix log streamer for py35 https://review.openstack.org/469480 | 13:39 |
| Shrews | ok, that fully works in both py3 and py2 now ^^^^ | 13:39 |
| Shrews | had to do all of the sends/recvs | 13:40 |
| Shrews | pabelanger: is there a known issue with our zuulv3 jobs? | 13:41 |
| pabelanger | Shrews: it has been reported that tox jobs randomly fails | 13:42 |
| pabelanger | I think it is because we are not copying the git repos properly | 13:43 |
| pabelanger | but still need to debug that | 13:43 |
| Shrews | k, just checking | 13:43 |
| pabelanger | Shrews: what are you seeing? | 13:43 |
| pabelanger | I was going to auto hold a node | 13:44 |
| Shrews | random fails, nothing informative in logs | 13:44 |
| pabelanger | but wasn't sure if that was implemented | 13:44 |
| Shrews | https://review.openstack.org/460671 for example | 13:44 |
| mordred | Shrews: https://review.openstack.org/#/c/460671 has 2 comments you might othewise miss since it also has 2 +2s | 13:44 |
| pabelanger | Shrews: Ya, that's the issue | 13:44 |
| pabelanger | I'll look into it now | 13:44 |
| mordred | Shrews: also - bikeshed: I +3d https://review.openstack.org/#/c/456685 - but maybe we should add a prefix? like {tmpdir}/zuul-builds/{uuid} - just so that /tmp itself doens't get swamped if someone it looking for something else? | 13:46 |
| Shrews | mordred: nod. not sure how valuable making the port configurable is since 'finger' command doesn't accept a port??? but we can certainly do that | 13:46 |
| mordred | Shrews: oh - nevermind | 13:46 |
| mordred | Shrews: yes- I don't think the port configurable is valuable - but maybe pabelanger has a use-case in mind? | 13:47 |
| Shrews | mordred: jobroot_dir config | 13:47 |
| mordred | Shrews: yup. I suck | 13:47 |
| Shrews | mordred: though, we might need to make the port configurable just for test scenarios | 13:47 |
| mordred | Shrews: fair enough - maybe also someone winds up ina world where they need to use non-standard ports for $reason | 13:48 |
| pabelanger | Was mostly wanting config for ipaddress, but possible people want non-root port for finger? (don't even know is that is possible) | 13:48 |
| mordred | Shrews: ok - so - transfer my bikeshed question to jobroot_dir ... but maybe I'll just put up a patch and we can discuss there | 13:48 |
| Shrews | yeah. maybe you just want to use the websocket server and not 'finger' itself | 13:49 |
| mordred | yah | 13:49 |
| mordred | also - still not sure the ACTUAL finger client will do the right thing here WRT buffering either ... | 13:49 |
| Shrews | mordred: it does! i just tested it through the py27/py35 changes | 13:49 |
| Shrews | amaze-balls | 13:50 |
| Shrews | i could modify the log file on the fly and watch finger do its thing correctly | 13:50 |
| mordred | Shrews: wow | 13:51 |
| mordred | Shrews: that's super awesome | 13:51 |
| pabelanger | Shrews: jeblair: my guess for 460471 failure is remote.origin.url is missing in the repo, but don't know why | 13:52 |
| pabelanger | let me see if I can get more logging going for our rsync | 13:52 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Allow for using build UUID as temp job dir name https://review.openstack.org/456685 | 13:54 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Allow for specifying root job directory https://review.openstack.org/456691 | 13:55 |
| Shrews | w00t | 13:55 |
| Shrews | SpamapS: as an IBMer, would like your signoff on 456721 before that gets +A'd | 13:56 |
| pabelanger | Oh, executor needs root now | 14:01 |
| pabelanger | interesting | 14:01 |
| Shrews | pabelanger: yeah | 14:03 |
| Shrews | privileged port and all | 14:03 |
| Shrews | pabelanger: but it drops privileges shortly after startup | 14:03 |
| pabelanger | Ya cool | 14:04 |
| pabelanger | wonder if we could setup CAP_NET_BIND_SERVICE capability, acording to google, that's seems to be the hipster way for 2.6.24 kernels + | 14:04 |
| pabelanger | but, I'll update my systemd configuration | 14:05 |
| pabelanger | Wow, even systemd does capabilities now | 14:10 |
| *** hashar has quit IRC | 14:19 | |
| clarkb | Shrews: mordred one reason to possibly use configurable port is if you are behind a proxy maybe?? | 14:25 |
| mordred | clarkb: indeed. although, fwiw, the finger command line client does not accept a port option | 14:26 |
| *** colettecello has quit IRC | 14:26 | |
| clarkb | right but client would hit proxy | 14:26 |
| clarkb | whatever is behind that can be ona rbitrary port | 14:27 |
| *** gothicmindfood has joined #zuul | 14:27 | |
| clarkb | (I dont know tbat anyone needs to run that way though) | 14:27 |
| mordred | ah - nod. yes | 14:28 |
| jeblair | yeah, also, once the whole system is in place, we might consider that it's only important for the *main* zuul log streamer to be on the finger port, since it multiplexes out the the executor finger servers. they could be on alternate ports, and then we could drop the root requirement from the executors. | 14:43 |
| SpamapS | Shrews: ACK, I'll review that. | 14:46 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Bail with an error on a non-existent jobroot_dir https://review.openstack.org/469524 | 14:53 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Make default job dir /tmp/zuul-jobs https://review.openstack.org/469525 | 14:53 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Rename jobroot_dir to jobdir_root https://review.openstack.org/469526 | 14:53 |
| mordred | jeblair, Shrews: ^^ bikeshed patch series | 14:53 |
| mordred | (those bits stood out to me reviewing Shrews series but it seemed easier to just write follow up patches than to suggest edits) | 14:54 |
| SpamapS | Shrews: might be a while though, flying/landing/etc. | 14:54 |
| jeblair | mordred: we already bikeshedded on the last one | 14:54 |
| jeblair | mordred: the disconnect is intentional so that it matches other variables in the config file | 14:55 |
| jeblair | mordred: (in other words, we chose to make things nicer for users and confusing for zuul developers) | 14:55 |
| jeblair | mordred: if you want to fix the disconnect, then i think we need to change the other variables in the config file | 14:56 |
| mordred | jeblair: kk | 14:57 |
| mordred | jeblair: I'm not sure the disconnect is worth changing the other things | 14:57 |
| jeblair | mordred: me neither, though, i'm not sure it's not not worth it. :) | 14:58 |
| pabelanger | jeblair: I like that idea of main streamer with root, others on different ports. But at this point, I don't want to bikeshed too much, until we see finger in action! | 14:58 |
| mordred | (also, that's why that's the last patch in the stack, it was the least-interesting and one I felt least strongly about) | 14:58 |
| pabelanger | excited to use finger command again | 14:58 |
| jeblair | pabelanger: yeah, we can evolve to that | 14:58 |
| pabelanger | ++ | 14:58 |
| Shrews | pabelanger: "Zuul. Making old things new again" | 14:59 |
| Shrews | telnet, finger. need to find a way to bring back gopher | 14:59 |
| jeblair | mordred: if it's not too confusing, "job_dir" in the config file might work....? | 15:00 |
| jeblair | Shrews: you know, gopher is great for interactive browsing of structured data. i hear that's a big thing on the modern interweb.... | 15:01 |
| jeblair | (we just might have some structured data.... gopher status page?) | 15:01 |
| Shrews | :) | 15:03 |
| *** isaacb has joined #zuul | 15:04 | |
| pabelanger | clarkb: jeblair so, for zuulv3.o.o, are we okay for using virtuelenv in production for zuul. | 15:21 |
| pabelanger | follow up question, do we want to do python3 too? | 15:21 |
| jeblair | pabelanger: i'd rather not use a virtualenv. why? | 15:22 |
| jeblair | pabelanger: yes on py3, i think. | 15:22 |
| pabelanger | mostly curious, since we have the option to do so with ansible | 15:23 |
| jeblair | we have the option with puppet too, but we chose not to. | 15:24 |
| tobiash | how do you deploy without venv in a clean way then? | 15:24 |
| jeblair | tobiash: just global pip installs. it's a single-purpose server, so there's no need for containment | 15:25 |
| tobiash | I had bad experiences with global pip installs in my early python days | 15:26 |
| *** hashar has joined #zuul | 15:26 | |
| tobiash | since then I never use pip for global installs | 15:26 |
| jeblair | we've had bad experiences with both. :) | 15:26 |
| tobiash | once the system python is messed up it's often hard work to get it working again ;) | 15:27 |
| jeblair | most of the issues involve pip itself; once that's sorted out, it's usually not a big deal. | 15:28 |
| jeblair | (we spent about 3 years learning how to install pip) | 15:29 |
| tobiash | :) | 15:29 |
| tobiash | if you choose py3 you should land 469181 and 468392 before | 15:30 |
| tobiash | they were needed to get it running in my py3 setup | 15:30 |
| clarkb | also constraints fix a whole set of problems for the single purpose install (if we end up having trouble we could ship a constraints file, though it hasn't be necessary yet for nodepool) | 15:31 |
| jeblair | tobiash: thanks, +2 | 15:31 |
| tobiash | :) | 15:32 |
| tobiash | how do you use pip then for global installs? | 15:35 |
| tobiash | our docker images also can be seen as single purpose machine so I'd like to learn how you are doing this | 15:36 |
| tobiash | if venv also causes issues maybe the docker images also can be improved then | 15:37 |
| jeblair | tobiash: this is how we install pip: http://git.openstack.org/cgit/openstack-infra/system-config/tree/install_puppet.sh#n243 | 15:38 |
| pabelanger | so it looks like we are missing pip3 from production servers, so installing into a virtualenv -p python35 would be easy mode | 15:39 |
| pabelanger | I'll look into updating puppet-pip | 15:39 |
| clarkb | pabelanger: we shouldn't need pip3 | 15:39 |
| tobiash | ah, ok the get-pip script | 15:39 |
| pabelanger | clarkb: okay, how do you have pip setup python3? | 15:39 |
| clarkb | either python3 -m pip or pip install --whatever-arg-is-for-python-install-dir | 15:40 |
| clarkb | trygin to find the arg now | 15:40 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Decode ssh output of gerrit connection https://review.openstack.org/468392 | 15:40 |
| clarkb | (its a lot like virtualenv, it runs under any python and you can specify which python to use and by default it uses python it runs under | 15:41 |
| jeblair | SpamapS, mordred: want to +3 https://review.openstack.org/469181 ? | 15:42 |
| clarkb | looks like -m pip might be the prefered method but also python3 $(which pip) | 15:42 |
| pabelanger | clarkb: ya, pip install --args would be good | 15:43 |
| jeblair | do we have a py2 on xenial (ie, it's not py3 by default?) | 15:44 |
| pabelanger | ya, pip defaulted to python2 when we installed it | 15:44 |
| pabelanger | pip 9.0.1 from /usr/local/lib/python2.7/dist-packages (python 2.7) | 15:45 |
| clarkb | its py3 by default meaning python3 is the default version of python, But that doesn't means that `python` and `pip` run under python3 by default | 15:45 |
| clarkb | because python should always be python2 and python3 should be used for python3 according ot python | 15:45 |
| pabelanger | do we need to update puppet-pip to be more specific? http://git.openstack.org/cgit/openstack-infra/puppet-pip/tree/manifests/init.pp#n16 | 15:45 |
| clarkb | pabelanger: we may want ot also install it for python3 just for completeness | 15:46 |
| pabelanger | ya, I know we do that for DIB today | 15:47 |
| pabelanger | http://git.openstack.org/cgit/openstack/diskimage-builder/tree/diskimage_builder/elements/pip-and-virtualenv/install.d/pip-and-virtualenv-source-install/04-install-pip#n104 | 15:49 |
| pabelanger | could we just build a ubuntu-xenial DIB image and use that for production servers? | 15:50 |
| pabelanger | guess I should take this to #openstack-infra | 15:50 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Decode the console log stream https://review.openstack.org/469181 | 16:01 |
| jeblair | fungi, SpamapS: can I have your thoughts on mordred's change https://review.openstack.org/469525 ? | 16:05 |
| fungi | lookin' | 16:07 |
| jeblair | neat, the failures in 468556 show up with the version of git on xenial, not trusty. so i can reproduce and fix on my xenial machine. | 16:07 |
| SpamapS | jeblair: I'm at CoreOS fest today.. will be a bit out of pocket for reviews. | 16:46 |
| *** jkilpatr has quit IRC | 16:47 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Enable test_multi_branch cloner/executor test https://review.openstack.org/468557 | 16:57 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Actually check out the branch in the jobdir https://review.openstack.org/468556 | 16:57 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Verify executor branch checkout https://review.openstack.org/468564 | 16:57 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Update the merger recent dict when saving the repo state https://review.openstack.org/469595 | 16:57 |
| jlk | jeblair: my initial thought is that on modern systems, /tmp is per-user instantiated, so running zuul as the zuul user, only the zuul user will see zuul things in /tmp/. Other users have their own temp. So stuffing it lower seems even less of a concern. | 16:59 |
| jeblair | SpamapS: thanks, enjoy the festivities! i won't charge for reviews this week so you won't be out of pocket on my account. ;) | 17:00 |
| jeblair | jlk: i feel like you're living in the future :) i don't think xenial is doing that (yet)... | 17:02 |
| pabelanger | why not make it /var/lib/zuul/tmp ? | 17:03 |
| SpamapS | jlk: eh? | 17:03 |
| *** jkilpatr has joined #zuul | 17:04 | |
| SpamapS | /tmp is per-user in containers.. is that what you're calling modern systems? | 17:04 |
| jlk | no, I guess Fedora was well ahead of the curve | 17:04 |
| SpamapS | what does Fedora do? | 17:04 |
| jeblair | fedora definitely lives in the future | 17:04 |
| jlk | https://fedoraproject.org/wiki/Features/tmp-on-tmpfs | 17:04 |
| jlk | hrm, that was part of it, let me find the one where it's per-user | 17:05 |
| pabelanger | Ya, DIB doesn work with tmp-on-tmpfs today. Wrong mount options by default | 17:05 |
| *** isaacb has quit IRC | 17:05 | |
| jlk | https://fedoraproject.org/wiki/Changes/Polyinstantiated_tmp_by_Default | 17:06 |
| dmsimard | mordred, pabelanger: re: floating IP issue -- we might've found something but not sure if it's related or a coincidence yet | 17:18 |
| dmsimard | Some VMs could not successfully be deleted, they had to be forcefully deleted by admins and nodepool kept looping on trying to delete them, i.e 2017-05-31 10:30:10,593 WARNING nodepool.NodePool: Deleting leaked instance bare-centos-7-rdo-cloud-beta-154211 (8727e0ab-ae59-468c-928c-16ee9fb4f96d) in rdo-cloud-beta for node id: 154211 | 17:18 |
| dmsimard | We stopped nodepool, cleaned everything (because they were VMs not being deleted due to floating ip issue..) including the two problematic VMs and we just started nodepool again. I'll monitor to see if we're still seeing the problem. | 17:19 |
| dmsimard | That said, we'll still work on updating shade regardless | 17:19 |
| pabelanger | why are admins manually deleting VMs, and not using nodepool delete command? | 17:20 |
| pabelanger | you should only have nodepool user accessing the project in openstack | 17:20 |
| pabelanger | dmsimard: did you upgrade shade? | 17:22 |
| dmsimard | pabelanger: I think you didn't fully read what I wrote | 17:22 |
| dmsimard | pabelanger: there were two VMs stuck in deleting, nodepool was not able to delete them | 17:22 |
| dmsimard | and nodepool kept trying to delete them | 17:23 |
| pabelanger | right, upgrading shade should have fixed that | 17:23 |
| pabelanger | but, nodepool would need to be stopped / started | 17:23 |
| pabelanger | dmsimard: I understood what you said, I am just not a fan of running manual commands behind the back of nodepool. A fair bit of work has gone into making nodepool not leak things | 17:24 |
| pabelanger | for a while we to manually clean up leaked FIPs by hand | 17:25 |
| dmsimard | pabelanger: they were stuck in deleting, what could nodepool have done ? | 17:25 |
| pabelanger | that is what I am trying to see, did you upgrade shade? | 17:25 |
| SpamapS | jlk: oh wow, so Fedora makes a little FS namespace for you and bind mounts in your own /tmp. Neat. | 17:26 |
| dmsimard | No, not yet, I don't have the luxury of being able to do it easily just like that | 17:26 |
| pabelanger | if so, then you'd likely have to stop /start nodepool again, which it then would have deleted things | 17:26 |
| dmsimard | tristanC: already has a patch up to upgrade shade to 1.20 | 17:26 |
| pabelanger | k | 17:26 |
| dmsimard | pabelanger: I'm not sure you follow, even when I tried to manually delete the VM it would not delete | 17:27 |
| dmsimard | so I don't see how nodepool could have deleted it | 17:27 |
| pabelanger | right, you need a new version of shade is what I am saying | 17:27 |
| dmsimard | apparently something must have happened on the cloud or the hypervisors or something | 17:27 |
| jlk | SpamapS: they used to, I'm not clear on if that still happens. | 17:29 |
| jlk | SpamapS: and it was an option in RHEL6 | 17:29 |
| SpamapS | There's a little passive aggression in that wiki page | 17:30 |
| SpamapS | "Secondly people don't really understand its benefits." | 17:31 |
| jlk | oh there's a ton of passive aggression from the security folks at Red Hat. | 17:35 |
| jlk | particularly the ones out in Brno. | 17:35 |
| jeblair | SpamapS: "No change is user experience, its is completely transparent to the user." might not be a complete and thorough analysis of the impacts... | 17:39 |
| SpamapS | "Just wear this mask. Oxygen will flow into you. The mask is completely transparent." | 17:41 |
| jeblair | mordred: i fixed up the issues at the end of my 'merger do the right thing' stack; if you want to resume reviews there at https://review.openstack.org/469595 and children | 17:44 |
| SpamapS | oh good I want to help that stack get in. I think the last couple of disabled tests might depend on that | 17:57 |
| jeblair | SpamapS: cool. i still plan on enabling the rest of the cloner tests (that stack ends with 2 enabled) | 17:59 |
| jeblair | sorry i've forgotten: is it possible to use bubblewrap on trusty? i see our packport ppa only has xenial. | 18:01 |
| SpamapS | It should work fine setuid | 18:02 |
| SpamapS | (which is how we use it on xenial) | 18:02 |
| SpamapS | it does not have any exotic dependencies though glibc might have changed enough that you have to rebuild it for trusty | 18:02 |
| jeblair | pabelanger, mordred: given ^ should we push a trusty bubblewrap build to our ppa? | 18:03 |
| SpamapS | apt install devscripts ; backportpackage bubblewrap | 18:03 |
| SpamapS | Or that :) | 18:03 |
| SpamapS | well actually .. backportpackage -u ppa:... bubblewrap | 18:03 |
| pabelanger | I think we could do that directly via launchpad ui | 18:03 |
| SpamapS | the launchpad UI just lets you copy binaries | 18:04 |
| pabelanger | ah | 18:04 |
| SpamapS | builds need a source upload | 18:04 |
| jeblair | SpamapS: "backportpackage -u ppa:openstack-ci-core/bubblewrap bubblewrap" ? | 18:04 |
| jeblair | (for the local option) | 18:04 |
| SpamapS | jeblair: actually you also need -d trusty | 18:04 |
| jeblair | SpamapS: if i'm on trusty? | 18:04 |
| SpamapS | oh maybe not | 18:04 |
| SpamapS | I forget if it defaults to the release you're running on | 18:04 |
| jeblair | sounds like it probably wouldn't hurt | 18:05 |
| pabelanger | https://launchpad.net/~openstack-ci-core/+archive/ubuntu/bubblewrap/+copy-packages seems to imply it will rebuild with a different distro | 18:05 |
| jeblair | this is a two-pronged effort: one, getting it installed on my trusty box so i can hack on it easier; and two, having it be in the ppa may be useful for us (since zuulv3-dev is still trusty) and potentially others, for a brief while yet. but not much longer, so probably neither of these things are worth *too much* effort. mostly didn't want us to have to stop running zuulv3-dev if we land bwrap changes. | 18:06 |
| jeblair | Please check bubblewrap 0.1.8-1~ubuntu14.04.1~ppa1 in file:///tmp/backportpackage-7f0Ot7 carefully! | 18:09 |
| jeblair | Do you want to upload the package to ppa:openstack-ci-core/bubblewrap [Y|n]? | 18:09 |
| jeblair | i dunno, do i? :) | 18:10 |
| pabelanger | sure | 18:10 |
| jeblair | done. i guess i'll check back in a while and see if it builds. | 18:12 |
| mordred | SpamapS: re your -1 on https://review.openstack.org/#/c/467611/ ... the next commit in the stack adds the tests | 18:14 |
| mordred | SpamapS: although I definitely agree we should not land patch 1 until patch 2 has passed tests and proven itself | 18:14 |
| jeblair | wfm | 18:15 |
| pabelanger | Oh, I know why we didn't do trusty for bubblewrap | 18:15 |
| pabelanger | the packaging for it was too new | 18:15 |
| pabelanger | it required debhelper >=10 | 18:16 |
| pabelanger | I forgot that I tried building the trusty package locally | 18:16 |
| jeblair | okay, so i guess we shouldn't land bwrap until we're running on xenial | 18:17 |
| jeblair | which i think means that we should start running on xenial immediately. | 18:17 |
| pabelanger | ya, I am working on pip3 support now for puppet-pip, otherwise python2 works fine | 18:17 |
| jeblair | pabelanger: feel free to ping me with any puppet patches and i'll review them with a high priority | 18:18 |
| pabelanger | k | 18:18 |
| SpamapS | mordred: Oh, was that patch added later? I thought I looked for later patches. | 18:50 |
| *** nibalizer has left #zuul | 19:13 | |
| mordred | SpamapS: shouldn't have been - but I do crazy tihngs frequnetly | 19:18 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add a finger protocol log streamer https://review.openstack.org/456721 | 19:23 |
| Shrews | \o/ | 19:24 |
| pabelanger | nice job | 19:31 |
| * mordred hands Shrews a pie | 19:33 | |
| *** jkilpatr has quit IRC | 19:41 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add config option for executor process user https://review.openstack.org/460671 | 19:43 |
| Shrews | so, is there a way to run a zuul test only in the py35 job and skip it in py27? | 19:48 |
| clarkb | Shrews: you can use a test skip, but if its a compile issue you have to have a test list that excludes that file iirc | 19:49 |
| jeblair | Shrews: SpamapS did the opposite recently for the py3 work with a test skip; should be some examples in branch tip now | 19:49 |
| Shrews | jeblair: ah, see that. thx | 19:51 |
| Shrews | any non-jeblair person care to +A https://review.openstack.org/469480 for me? | 19:52 |
| *** jkilpatr has joined #zuul | 19:54 | |
| *** jkilpatr has quit IRC | 19:54 | |
| *** jkilpatr has joined #zuul | 19:54 | |
| clarkb | Shrews: not direclty related to your change but shouldn't the request uuid be a bit more sanitized than it is in there? | 19:55 |
| Shrews | clarkb: how so? | 19:55 |
| clarkb | Shrews: we don't check the length of the input, just that its a subset of alnum | 19:56 |
| clarkb | also uuids can include -'s in their text reprs right? | 19:56 |
| Shrews | i suppose that's a question for jeblair. i'm not sure how zuul generates them | 19:57 |
| jeblair | ours do not have - | 19:59 |
| Shrews | SpamapS: could this failure have something to do with the fd issues you were having? http://logs.openstack.org/80/469480/2/gate/gate-zuul-python27-ubuntu-xenial/fb28b87/testr_results.html.gz | 20:11 |
| Shrews | i didn't follow the convo closely | 20:11 |
| Shrews | i just ran that test 30x locally with no fails. ugh | 20:15 |
| Shrews | jeblair: oh wow. kazoo read locks merged. neat | 20:28 |
| jeblair | Shrews: \o/ now i need to remember how that was going to fix the delete thing | 20:31 |
| SpamapS | Shrews: that is very much like what we were seeing yes. | 20:41 |
| Shrews | i've run it 150x locally and cannot reproduce it | 20:41 |
| Shrews | is it a test interaction thing? b/c i'm running it alone | 20:41 |
| SpamapS | We only saw it when running in parallel. | 20:42 |
| SpamapS | Though I hypothesized it was more to do with sample size. | 20:42 |
| SpamapS | since running testr run --analyze-isolation did not produce a result | 20:42 |
| jeblair | SpamapS, pabelanger: aha! i stacked 461881 on top of my patch series to get the debug log changes and finally see the error: "cannot import name paths" | 20:45 |
| jeblair | so i assume there's a problem with 468208 | 20:45 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Fix log streamer for py35 https://review.openstack.org/469480 | 20:58 |
| *** hashar has quit IRC | 21:16 | |
| *** harlowja has quit IRC | 21:21 | |
| *** dkranz has quit IRC | 21:36 | |
| jlk | jeblair: did you see my comment yesterday about driver specific requires? | 21:52 |
| jeblair | jlk: that they're broken? yes :) i haven't looked more into that, have you? | 21:53 |
| jlk | no, I was just checking that branch out again to see what code was supposed to be there to limit filters to the source of the change | 21:53 |
| *** harlowja has joined #zuul | 22:06 | |
| jlk | jeblair: stepping through here, in manager/__init__.py, addChange(), there is a self.changeish_filters, which seems to have both the github and the gerrit filters in it | 22:12 |
| jlk | and the change it's passing to those, it doesn't necessarily have any immediate indication of which driver it came from | 22:12 |
| jlk | and there's nothing to say "only look at THIS type of filter for changes from that driver" | 22:12 |
| jeblair | jlk: ah, so the filters probably need to check that internally | 22:13 |
| jeblair | (if they care) | 22:13 |
| jlk | hrm. | 22:14 |
| jeblair | should be able to figure it out from change.project | 22:14 |
| jlk | the logic is to see if the matches() function returns true | 22:14 |
| jlk | what you're saying, is that the github matches() function should just return true if ti's a non-github change? | 22:14 |
| jeblair | jlk: heh, i was going to say return false, but i see the problem with that. :) | 22:15 |
| jeblair | jlk: so maybe the pipeline manager needs to do the check | 22:15 |
| jlk | yeah I was thinking it should avoid running match() on mis-matched events | 22:16 |
| jeblair | jlk: so either that, or we can have the filter return true for changes not from their source. | 22:17 |
| jlk | the other problem, is that the filter itself doesn't seem to have an easy indication of what driver it belongs to | 22:18 |
| jlk | ['__class__', '__delattr__', '__dict__', '__doc__', '__format__', '__getattribute__', '__hash__', '__init__', '__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_match_review_required_review', '_required_reviews', '_tidy_reviews', 'current_patchset', 'matches', 'matchesRequiredReviews', 'matchesRequiredStatuses', 'matchesReviews', 'open', 'required_reviews', 'required_s | 22:18 |
| jlk | tatuses', 'statuses'] | 22:18 |
| jlk | I suppose we could add an attribute | 22:19 |
| jeblair | jlk: maybe 'return true' is the easiest/best path for now? | 22:20 |
| jlk | might be. Doesn't sit right with me from a first blush, but... | 22:21 |
| *** jkilpatr has quit IRC | 22:23 | |
| jeblair | jlk: hrm, as i think about it further, we're actually matching 'source' (which is a connection, not just a driver). so i think we need to add that attribute to the filter regardless. | 22:24 |
| jlk | oh, the filters are canonical_hostname specific? | 22:24 |
| jlk | rather than driver specific? | 22:24 |
| jeblair | jlk: yep | 22:26 |
| jeblair | so you can actually apply different github requirements to changes from different github connections | 22:26 |
| jlk | yeah so maybe we should set a canonical_hostname for the filter, and match that up to the canonical_hostname of the change | 22:26 |
| jeblair | jlk: yes, though i would use connection name for this, since most other similar code does the same | 22:26 |
| jlk | okay | 22:26 |
| jeblair | (they're usually 1:1, but not always) | 22:27 |
| patrickeast | hey, maybe dumb question, is 2.5.2 the "best" zuul version to be using with the older jenkins-y style ci's? Or is there some other recommended one? | 22:29 |
| * patrickeast is finally upgrading this 3rd party ci from 2.1.0 | 22:30 | |
| jlk | patrickeast: 2.5 makes use of Ansible as the executor rather than Jenkins | 22:38 |
| jlk | so if you're still jenkins bound, perhaps you should avoid 2.5? | 22:38 |
| patrickeast | ah yea | 22:38 |
| patrickeast | i wasn't sure if there were tagged releases that dropped it or not | 22:38 |
| *** jkilpatr has joined #zuul | 22:38 | |
| patrickeast | alright, guess its time to go digging through git history | 22:39 |
| jeblair | patrickeast, jlk: latest 2.5 is okay | 23:09 |
| jlk | oh oops :( | 23:09 |
| jeblair | so yeah, 2.5.2 is "best" for that :) | 23:10 |
| jeblair | we didn't drop any jenkins support, only added secret ansible support. :) | 23:10 |
| patrickeast | jeblair: thanks for confirming | 23:10 |
| patrickeast | yea i was looking at it and the gearman launcher still needed to be fully integrated | 23:10 |
| patrickeast | s/needed/seemed/ | 23:11 |
| jeblair | patrickeast: basically just ignore "zuul-launcher". that's the ansible stuff. it should be easy to ignore: we didn't document it. :) | 23:11 |
| patrickeast | haha | 23:11 |
| patrickeast | jeblair: perfect | 23:11 |
| patrickeast | jeblair: so far it looks like all my old configs "just work" | 23:12 |
| jlk | jeblair: I think I have something working to do the filtering. Getting it fully fleshed out with logging now. | 23:12 |
| jlk | jeblair: in tests, is there a way to assert that a traceback didn't happen? this test is "passing" because the job never enqueued, because a traceback happened, not because the filter(s) rejected it | 23:12 |
| patrickeast | i don't suppose nodepool has db migration scripts does it? | 23:18 |
| patrickeast | maybe should be pester in -infra | 23:18 |
| patrickeast | s/be/go/ | 23:18 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Handle change related reqs on push like events [WIP] https://review.openstack.org/469297 | 23:28 |
| jlk | jeblair: I think https://review.openstack.org/469297 does the right thing now, as far as skipping mismatched connection filters. Ready for feedback, but I think we need to add logging for the actual rejections we're doing inside the connection filter. Hence the WIP still. Please do review and let me know if this is the right stuffs. | 23:30 |
| jlk | oh that's a weird failure case | 23:33 |
| jlk | ugh, the zuul driver creates a connection that is a list, not an actual connection object. | 23:39 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!