| mordred | jlk: that's quite yuck | 00:55 |
|---|---|---|
| *** jamielennox is now known as jamielennox|away | 02:07 | |
| *** jamielennox|away is now known as jamielennox | 02:21 | |
| *** isaacb has joined #zuul | 06:25 | |
| *** hashar has joined #zuul | 07:37 | |
| openstackgerrit | Dirk Mueller proposed openstack-infra/nodepool master: Add default logging file argument https://review.openstack.org/446324 | 07:49 |
| *** bhavik1 has joined #zuul | 08:22 | |
| openstackgerrit | Javier Peña proposed openstack-infra/zuul master: Find fallback branch in zuul-cloner https://review.openstack.org/442370 | 08:37 |
| openstackgerrit | Javier Peña proposed openstack-infra/zuul master: Find fallback branch in zuul-cloner https://review.openstack.org/442370 | 09:15 |
| *** bhavik1 has quit IRC | 10:47 | |
| *** hashar has quit IRC | 11:06 | |
| *** hashar has joined #zuul | 11:07 | |
| *** isaacb_ has joined #zuul | 12:07 | |
| *** isaacb has quit IRC | 12:08 | |
| openstackgerrit | Javier Peña proposed openstack-infra/zuul master: Find fallback branch in zuul-cloner https://review.openstack.org/442370 | 12:33 |
| *** pabelanger has quit IRC | 12:49 | |
| *** pabelanger has joined #zuul | 12:49 | |
| *** isaacb_ has quit IRC | 13:04 | |
| *** isaacb_ has joined #zuul | 13:05 | |
| *** isaacb_ has quit IRC | 13:13 | |
| pabelanger | morning | 14:56 |
| pabelanger | Shrews: jeblair: going to start the process to restart nodepool | 14:56 |
| pabelanger | then move onto zuulv3-dev | 14:56 |
| pabelanger | nl01.o.o == nodepool in this context | 14:56 |
| Shrews | ack | 14:58 |
| pabelanger | okay, restarted using nodepool-launcher binary now | 14:59 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Add trigger developer doc https://review.openstack.org/453695 | 15:01 |
| pabelanger | okay, starting upgrade of zuulv3-dev | 15:10 |
| pabelanger | zuul restarted | 15:15 |
| pabelanger | but going to stop because of an exception in the logs | 15:15 |
| pabelanger | http://paste.openstack.org/show/605520/ | 15:16 |
| pabelanger | jeblair: you might be interested ^ | 15:16 |
| jeblair | pabelanger: it looks like the "NoneType" it refers to is the project name, since the repr method of TriggerEvent includes the project name: <TriggerEvent ref-replicated None> | 15:19 |
| jeblair | (in other words, that was a ref-replicated event for project "None") | 15:19 |
| jeblair | pabelanger: btw, those are not fatal errors, and it's not in a loop. you could actually keep it running. | 15:19 |
| pabelanger | okay, I can start it back up then | 15:20 |
| pabelanger | I'll see what changed to cause that | 15:20 |
| pabelanger | I also noticed we generated keys for secrets! | 15:21 |
| pabelanger | however, they are world readable :( | 15:21 |
| jeblair | maybe the change to "support" ref-replicated... | 15:21 |
| pabelanger | I'll get a patch to fix that too | 15:21 |
| jeblair | pabelanger: cool. they should be in a non-world readable directory, but yes, we should also fix that. :) | 15:21 |
| pabelanger | jeblair: thanks, I'll look there first | 15:21 |
| jeblair | pabelanger: it looks like we just don't set the project attribute because the event is neither a change nor a ref-update. i think we just need better handling of that, though, to be honest, i think we should ignore it and not bother even equeueing it. | 15:24 |
| jeblair | replication-complete is a more useful event i think. | 15:24 |
| jeblair | (we don't have any support for actually doing anything with it anyway | 15:25 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Clarify TriggerInterface documentation https://review.openstack.org/453713 | 15:31 |
| *** hashar has quit IRC | 15:31 | |
| pabelanger | so, it looks like we pulled down ansible 2.2.2 on the upgrade | 15:33 |
| pabelanger | now seeing some SSH errors | 15:33 |
| *** hashar has joined #zuul | 15:33 | |
| pabelanger | not sure if ansible side or nodepool | 15:33 |
| *** hashar has quit IRC | 15:34 | |
| pabelanger | ssh works on images | 15:35 |
| pabelanger | looking into ansible | 15:35 |
| pabelanger | sudo error on zuulv3-dev will be me | 15:38 |
| pabelanger | k, ~/.ssh/known_hosts for the zuul users was populated with old host keys | 15:50 |
| pabelanger | not sure why, but I have purged them and going to see what could have added them | 15:51 |
| clarkb | don't we tell the zuul user to not populate keys at all though? | 15:51 |
| clarkb | since they are single use nodes with host keys that get deleted? | 15:51 |
| pabelanger | ya, we have ansible use job specific known_hosts file | 15:51 |
| openstackgerrit | Javier Peña proposed openstack-infra/zuul master: Find fallback branch in zuul-cloner https://review.openstack.org/442370 | 15:52 |
| pabelanger | Hmm, that might not have been it | 15:56 |
| pabelanger | Hmm | 16:01 |
| pabelanger | it looks like we might not be generating the known_host file properly | 16:02 |
| pabelanger | http://paste.openstack.org/show/605525/ is the current value | 16:02 |
| pabelanger | I don't believe that is valid, I think we need to also include the IP address | 16:03 |
| pabelanger | http://paste.openstack.org/show/605526/ | 16:03 |
| pabelanger | from ssh-keyscan | 16:03 |
| jeblair | yeah, that looks like it's missing a bit :) | 16:10 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool feature/zuulv3: Include IP address in SSH host key https://review.openstack.org/453734 | 16:14 |
| pabelanger | http://paste.openstack.org/show/605530/ ^ what is generated now | 16:14 |
| jeblair | pabelanger: we already have the ip address in zuul, why don't we assemble it there? | 16:16 |
| pabelanger | jeblair: I was going to do that but what if incorrectly mapped ipv4 to an ipv6 hostkey? | 16:17 |
| pabelanger | or, does hostkey work regardless of ipv4 / ipv6 | 16:17 |
| jeblair | pabelanger: i believe they are typically the some (though, theoretically, a sysadmin could run two different ssh deamons on the two different network stacks). | 16:18 |
| pabelanger | okay, I don't have a preference. I thought it would be safer to build it on the nodepool side, against the IP we scanned, since we had the indo | 16:19 |
| pabelanger | info* | 16:19 |
| clarkb | you'll sometimes find that if you ssh via v6 and accept a key then ssh via v4 ssh will warn you but say the host keys match so its probably ok and carry on | 16:19 |
| jeblair | pabelanger: i think we have to weigh it against the likelihood of a nodepool consumer accessing the node over a different address than the one nodepool used. i believe that is more likely than the case of a node having two different host keys. | 16:19 |
| pabelanger | good point | 16:21 |
| *** robcresswell has joined #zuul | 16:23 | |
| robcresswell | Hello hello. I've been tasked with some research into creating a CI system that mirrors openstack-infra's; Is Zuul v3 considered usable enough / documented enough for people to get started with yet, or is it still very much in a science project stage? | 16:25 |
| pabelanger | jeblair: Hmm, how would I map the known_hosts data for multiple hosts? | 16:25 |
| mordred | hi robcresswell! I think it's somewhere in the middle of those two - it's not science project (we're running it on zuul changes) but it's also not production ready and still needs some features and may still make breaking file format changes before it is ready for production ... | 16:27 |
| pabelanger | robcresswell: we are not recommending zuulv3 for production just yet. We are still working towards an stable API | 16:27 |
| mordred | robcresswell: so the questions would be what is your timeframe and what is your interest in / tolerance for looking at something in an early-alpha kind of place | 16:27 |
| robcresswell | mordred, pabelanger: It's just a research task yet; we're a way off putting it in prod. I'm comfortable working off unstable APIs, just try to gauge progress | 16:28 |
| robcresswell | mordred: Is there much in the way of docs for the current progress? | 16:29 |
| robcresswell | current state, even | 16:29 |
| mordred | awesome. if it's a research task, I'd definitely not spend much effort on zuul v2. also, we haven't spent much time documenting v3 extensively yet | 16:29 |
| pabelanger | Our zuulv3 status ML posts would be a good summary I think: http://lists.openstack.org/pipermail/openstack-dev/2017-March/113148.html | 16:30 |
| mordred | (you can see jobs reported from "zuul" on changes to zuul: https://review.openstack.org/#/c/453347/) | 16:30 |
| jeblair | yes, those periodic status emails are good | 16:30 |
| mordred | also, https://storyboard.openstack.org/#!/board/41 is the workboard tracking work | 16:31 |
| jeblair | pabelanger: the host key should be an attribute of the node | 16:34 |
| robcresswell | pabelanger, mordred: This is promising, I'll spend some time reading. Thanks for the links. Due to the nature of the work, I'll probably look to v3. | 16:34 |
| * robcresswell apologises in advance for stupid questions | 16:34 | |
| pabelanger | jeblair: yes, that make sense. | 16:34 |
| pabelanger | need more coffee | 16:34 |
| mordred | robcresswell: we love stupid questions! | 16:35 |
| jeblair | robcresswell: no worries. oh, there's two more things that may help you put the pieces together in lieu of our actually having updated the documentation yet. let me get links | 16:36 |
| jeblair | robcresswell: the zuul v3 spec which mostly accurately describes what it's all about and how it's different from v2: http://specs.openstack.org/openstack-infra/infra-specs/specs/zuulv3.html | 16:37 |
| jeblair | robcresswell: and this email i sent to the team before our hackathon at the PTG, which focuses on how we set up our bootstrap production instance: http://lists.openstack.org/pipermail/openstack-infra/2017-February/005131.html | 16:38 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Add host IP address to host_keys https://review.openstack.org/453765 | 16:45 |
| robcresswell | Thanks for all the help jeblair :) | 16:46 |
| clarkb | Shrews: I'm starting to write up nodepool operations docs. And noticing that we still have dib-image-list and dib-image-delete commands but no image-upload. Does it make sense to keep the dib-* commands if all you can do with them is list and delete? | 16:55 |
| robcresswell | To elaborate on use case; I'm at Cisco, we have a bunch of plugins as well as other internal projects. We're looking to improve our CI situation, and the logical thinking was to mirror upstream setups; makes debugging/documentation/other help easier. So the overall investigation task is to look at current tooling, see where it's headed, and what the best way | 16:56 |
| robcresswell | to utilise it is. That might be configuring components ourselves, or operating at a higher level and using something like puppet-openstackci (?) to set things up. | 16:56 |
| robcresswell | I'm woefully ignorant on how it all works right now, so it'll be a few days of poking things with sticks and reading code. | 16:56 |
| pabelanger | clarkb: FWIW: I find image-delete complicate to use, too many inputs. So, I just end up using dib-image-delete, which gets all the things | 16:56 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Use ssh for git-upload-pack https://review.openstack.org/436802 | 16:56 |
| pabelanger | complicated* | 16:57 |
| clarkb | pabelanger: so if you dib-image-delete it doesn't just remove it from disk? | 16:57 |
| clarkb | (thats what its supposed to do iirc) | 16:57 |
| pabelanger | right, it will first delete the DIB from clouds, then disk | 16:58 |
| clarkb | ugh | 16:58 |
| clarkb | (I don't think thats the right behavior) | 16:58 |
| pabelanger | you cannot delete a DIB if the image is uploaded | 16:58 |
| pabelanger | currently | 16:58 |
| clarkb | right but thats why that command exists in the first place :) | 16:58 |
| clarkb | you don't necessarily need ti local on disk if its happy in a cloud somewhere | 16:58 |
| pabelanger | ya, if you have limited local HDD space, will be an issues | 17:00 |
| pabelanger | but, image-delete likely needs a global UUID because the amount of flags needed today, is complex IMO | 17:01 |
| *** jamielennox has quit IRC | 17:04 | |
| jeblair | robcresswell: cool, thanks -- yeah, puppet-openstackci is the best way to get a v2 zuul set up, and it's what we recommend for openstack third-party ci operators. | 17:04 |
| jeblair | clarkb: i don't think i've ever heard the use case for dib-image-delete to be to save hd space. | 17:05 |
| clarkb | jeblair: in old nodepool all it did was remove the image from disk, potentially allowing you to rebuild then upload a new image if disk constrained | 17:06 |
| clarkb | but all the dib-* commands only affected "local" resources, nothing in clouds | 17:07 |
| *** jamielennox has joined #zuul | 17:07 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Store project name in ref-replicated events https://review.openstack.org/453796 | 17:17 |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool feature/zuulv3: Start adding operational docs to zuulv3 https://review.openstack.org/453797 | 17:18 |
| jeblair | clarkb: the hierarchical structure we used for images in zk makes that more difficult. it also makes things like 'image-delete' more difficult for the same reason. i'm willing to chalk that up to a learning experience (we did say we wanted to take baby steps with zk and learn from it -- and indeed, everything else we've done in zk is much more flat) and am willing to change it to split it back out into separate tables. that's not going to be ... | 17:21 |
| jeblair | ... trivial, especially backwards compat. but it's doable. maybe as a lower priority. | 17:21 |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool feature/zuulv3: Start adding operational docs to zuulv3 https://review.openstack.org/453797 | 17:22 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove source from pipelines https://review.openstack.org/453362 | 17:23 |
| clarkb | jeblair: I think my only real concern with it now is it feels very redundant since we already have image-delete and image-list if we aren't going to treat the disk images as separate artifacts | 17:24 |
| clarkb | if we think the dib image details should be abstracted away thats fine but we shouldn't have dib-* specific commands inthat case | 17:24 |
| jeblair | clarkb: the naming there is awkward; dib-image-* is "built image file on disk" and image-* is "uploaded image in a cloud". so they are different. | 17:25 |
| clarkb | yup thats why I find the current behavior of dib-image-delete to not be what we want | 17:25 |
| clarkb | we've sort of mashes the two sets together in a weird way for users | 17:25 |
| jeblair | clarkb: i accept that's not what you want. but i don't understand how we've mashed them together. they still seem very separate to me. there is just now a one-way dependency (you can not have an image uploaded that you don't also have a built copy of) | 17:26 |
| Shrews | clarkb: iirc, we cannot just delete the local dib build b/c we can't delete the zookeeper znode w/o first knowing we are the "owner" of the dib (find it on disk). and we can't delete the znode w/o first deleting the uploads. so it's a weird symbiotic relationship that we have now, that may not be ideal | 17:27 |
| clarkb | jeblair: previously dib-* meant only managing resources that are local | 17:28 |
| clarkb | jeblair: but now it means both local and cloud remote | 17:28 |
| clarkb | thats what I mean about mashing together | 17:28 |
| jeblair | clarkb: that is correct. i do still think they are different commands that serve distinct purposes. | 17:28 |
| clarkb | basically it was safe to dib-image-delete without affecting anything in the cloud | 17:28 |
| jeblair | clarkb: yes | 17:28 |
| jeblair | that has changed. | 17:28 |
| clarkb | I'm not sure why we need a separate command from image-delete then | 17:29 |
| pabelanger | is that a result of using ZK, or because we wanted to not leak images in clouds? | 17:29 |
| mordred | so - image-delete would delete from the cloud but not disk, so the image on disk would get re-uploaded right? | 17:29 |
| jeblair | clarkb: image-delete removes a single uploaded image from one cloud. dib-image-delete removes all of them. | 17:29 |
| mordred | ah | 17:29 |
| jeblair | mordred: correct (unless you paused that cloud/image) | 17:29 |
| clarkb | jeblair: wouldn't a --all-clouds ro similar flag be a better way to express that? | 17:29 |
| clarkb | I just don't read `dib-image-delete` as "really go delete everything everywhere` | 17:30 |
| jeblair | clarkb: well, it also deletes the image file. | 17:30 |
| clarkb | and I don't think our existing users will either | 17:30 |
| clarkb | (I've updated the docs to express taht in my change fwiw) | 17:30 |
| clarkb | (so at least the docs are correct now) | 17:30 |
| jeblair | clarkb: this is where documentation helps. we can also change the command name if there's a better one. | 17:30 |
| mordred | yah - I agree the command name does not imply to me (without reading the docs) that the command will delete the local and remote images, but I grok the need for the two commands | 17:31 |
| mordred | I'd vote for figuring out a new command name, if we can | 17:31 |
| * mordred has no better name suggestions yet though | 17:32 | |
| clarkb | perhaps image-purge | 17:33 |
| Shrews | clarkb: also, rbergeron and myself had a discussion about new doc things, so pinging her so that she is aware of your new operation doc | 17:33 |
| clarkb | rbergeron: ping ^ I pushed https://review.openstack.org/453797 | 17:33 |
| mordred | image-purge would imply to me that a bunch of things are going to get deleted | 17:33 |
| clarkb | mordred: aiui thats exactly what happens | 17:33 |
| clarkb | its completely removed from local disk and all the clouds | 17:33 |
| pabelanger | and, unless paused, will trigger a rebuild and reupload | 17:34 |
| jeblair | i think the key thing is that there are two concepts here: a dib-image-file, and an uploaded-image-in-a-cloud. inside the nodepool source code, we now call those 'image' and 'upload', because of exactly this confusion. i think good command names would help communicate that distinction to the user as well so they understand it. | 17:35 |
| mordred | what if we broke a BUNCH of backwards compat and called it "image-delete" and "upload-delete" | 17:36 |
| jeblair | so image-purge is a pretty good descriptor, however, it confuses the two concepts rather than elucidating them to the user. if we call dib-image-delete image-purge, we back ourselves into a corner with dib-image-list. | 17:36 |
| mordred | or image-delete and image-upload-delete | 17:36 |
| jeblair | mordred: i like image-upload-delete. it's more typing, but it's less awkward than 'upload-delete' which can be read as two verbs. :) | 17:36 |
| mordred | yah | 17:37 |
| jeblair | mordred: two contradictory verbs, at that :) | 17:37 |
| clarkb | and then have image-upload-list and image-list | 17:37 |
| clarkb | that would work | 17:37 |
| mordred | whereas image-upload-delete is "delete an image-upload" | 17:37 |
| rbergeron | clarkb: oh excellentay awesome, thanks for the heads up (i will take a peek. it surely is more riveting than, say, the all day phone call i am in) | 17:37 |
| rbergeron | (sigh) | 17:37 |
| jeblair | or we can even completely englishify it as 'delete-image-upload' | 17:37 |
| mordred | jeblair: now you're making crazy noises | 17:38 |
| jeblair | (i have no idea why we make all our command names backwards in openstackland) | 17:38 |
| mordred | (but I'm fan of verb-object myself) | 17:38 |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool feature/zuulv3: Fix internal doc refs to renamed section https://review.openstack.org/453802 | 17:38 |
| clarkb | rbergeron: also ^ | 17:38 |
| clarkb | jeblair: its so that you have to scroll aroundin your shell more to modify other objects | 17:39 |
| mordred | jeblair: I believe it's due to how we create comamnds via nested namespaces | 17:39 |
| clarkb | mordred: yes thats the real reason | 17:39 |
| mordred | so "openstack server create" can look up the server plugin to find the actoins it has and then find the create action on that | 17:39 |
| clarkb | its an easy way to allow server and network to register their own "delete" commands | 17:39 |
| mordred | whereas openstack create server would have to parse and undersatnd the whole thing | 17:39 |
| mordred | yah | 17:39 |
| jeblair | mordred: ah, that's helpful. but i think we still had 'nova image-list' a long time ago? | 17:39 |
| mordred | well, that was written by termie or something, so all bets are off | 17:40 |
| jeblair | point | 17:40 |
| jeblair | anyway, sounds like 'delete-image-upload' may be a winner? | 17:40 |
| clarkb | rbergeron: I also have https://review.openstack.org/452964 for cleaning up zuuls doc warnings | 17:40 |
| clarkb | rbergeron: those two cleanup changes are probably good to get in before a ton of docs work happens as it will help us catch silly bugs (like the images refs in nodepool not existing) | 17:41 |
| mordred | jeblair: ++ | 17:41 |
| jeblair | (i guess the agreement might change if you do that; 'list-images' would probably be preferable to 'list-image'. maybe we can make both work. :) | 17:41 |
| jeblair | (whereas 'image-list' is less impacted by grammar conventions) | 17:42 |
| * jeblair registers for a grammar convention | 17:42 | |
| jeblair | clarkb: thanks, made some corrections/suggestions | 17:51 |
| jeblair | clarkb, mordred: can one of you +3 https://review.openstack.org/453765 ? | 17:58 |
| jeblair | pabelanger, clarkb, mordred: can you review https://review.openstack.org/453796 ? | 17:59 |
| jeblair | both of those are things we should get in asap to restart zuul and make it less spammy. | 17:59 |
| pabelanger | +2 | 18:00 |
| pabelanger | I need to run an errand to the bank | 18:00 |
| pabelanger | back shortly | 18:00 |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool feature/zuulv3: Fix internal doc refs to renamed section https://review.openstack.org/453802 | 18:07 |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool feature/zuulv3: Start adding operational docs to zuulv3 https://review.openstack.org/453797 | 18:07 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add host IP address to host_keys https://review.openstack.org/453765 | 18:10 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove source from pipelines https://review.openstack.org/453362 | 18:24 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove source from pipelines (1/2) https://review.openstack.org/453362 | 18:30 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove source from pipelines (2/2) https://review.openstack.org/453821 | 18:30 |
| jeblair | i split up that commit so my eyes bleed less | 18:30 |
| SpamapS | yeah wow | 18:32 |
| *** harlowja has quit IRC | 18:33 | |
| *** hashar has joined #zuul | 18:37 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Store project name in ref-replicated events https://review.openstack.org/453796 | 18:45 |
| SpamapS | jeblair: my my, you've been busy. I really am enjoying reviewing the multi-source stack. | 19:00 |
| * Shrews currently attempting to grok that stack | 19:14 | |
| jlk | oh that's lovely | 19:19 |
| jlk | Run full tox, two tests fail. Run just the failing test, it passes. | 19:19 |
| jlk | ¯\_(ツ)_/¯ | 19:19 |
| jeblair | jlk: are they the "dequeue" tests? if so, they're probably just running long (they are ridiculously large tests) | 19:20 |
| jlk | no, it was a test_playbook I think. | 19:20 |
| jlk | tests.unit.test_v3.TestAnsible.test_playbook | 19:20 |
| jeblair | jlk: oh. that is also long (it waits 10 seconds for a job to time out), but not quite as busy. | 19:21 |
| jlk | ah. Is there a way to extend the timeouts? | 19:21 |
| jlk | I think there was, I just forget what it was | 19:21 |
| jeblair | jlk: yeah, theres an env var... OS_TEST_TIMEOUT i think? | 19:21 |
| jeblair | yep | 19:22 |
| jlk | thanks | 19:23 |
| SpamapS | jlk: I've been running with OS_TEST_TIMEOUT=9999 | 19:31 |
| SpamapS | still get random fails though | 19:31 |
| SpamapS | so dunno if there's more to it. | 19:31 |
| jlk | yeah I'm getting disappointingly inconsistent results. | 19:31 |
| SpamapS | I wonder if somewhere deep in zuul something else is setting up alarms | 19:31 |
| jlk | I narrowed all my tests down to just the github ones, one test fails. Just run that one test, it passes. | 19:32 |
| jlk | makes me wonder if there is something weird with parallelization | 19:32 |
| Shrews | jeblair: perhaps this will be obvious deeper in the stack, but is there a reason to have both the Source and Connection classes store canonical_hostname (https://review.openstack.org/451110)? There is already a connection between those two classes... | 19:32 |
| SpamapS | jlk: I've taken to running just the ones I care about locally, and (ab)using zuul to run the full suite. | 19:32 |
| SpamapS | jlk: testr does run with as many CPUs as you have by default.. but I've had equally bad results by running with concurrency of 2 | 19:33 |
| jlk | I'm about to do the same. | 19:33 |
| jeblair | Shrews: yes -- though it may not be a good one. :) basically, it's an API decision -- a connection may not necessarily have a canonical_hostname (connections include gerrit and github, but also smtp servers, (future: irc servers, mqtt servers), etc). but a source (as a thing where git repos live) must have one. so i put it in the source API. now, the gerrit connection does have one because it supports the source api, and that's where the ... | 19:37 |
| jeblair | ... actual configuration happens, but that's essentially private to that connection class. | 19:37 |
| Shrews | ok | 19:41 |
| jlk | Apologies for the incoming. | 19:49 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: support github pull reqeust labels https://review.openstack.org/444511 | 19:49 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Allow github trigger to match on branches/refs https://review.openstack.org/445625 | 19:49 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Set filter according to PR/Change in URL https://review.openstack.org/446782 | 19:49 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Add support for requiring github pr head status https://review.openstack.org/449390 | 19:49 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Support for dependent pipelines with github https://review.openstack.org/445292 | 19:49 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Add basic Github Zuul Reporter. https://review.openstack.org/443323 | 19:49 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Configurable SSH access to GitHub https://review.openstack.org/444034 | 19:49 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Add 'push' and 'tag' github webhook events. https://review.openstack.org/443947 | 19:49 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Add 'pr-comment' github webhook event https://review.openstack.org/443959 | 19:49 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Support for github commit status https://review.openstack.org/444060 | 19:49 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Better merge message for GitHub pull reqeusts https://review.openstack.org/445644 | 19:49 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Query changes only via relevant sources https://review.openstack.org/448257 | 19:49 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Encapsulate determining the event purpose https://review.openstack.org/445242 | 19:49 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Allow using webapp from connections https://review.openstack.org/439831 | 19:49 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Merge pull requests from github reporter https://review.openstack.org/444463 | 19:49 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Add support for github enterprise https://review.openstack.org/449258 | 19:49 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Support GitHub PR webhooks https://review.openstack.org/439834 | 19:49 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: GitHub file matching support https://review.openstack.org/446113 | 19:49 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Add trigger capability on github pr review https://review.openstack.org/449365 | 19:49 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Log GitHub API rate limit https://review.openstack.org/446150 | 19:49 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Adds github triggering from status updates https://review.openstack.org/453844 | 19:49 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Implement pipeline requirement on github reviews https://review.openstack.org/453845 | 19:49 |
| Shrews | wow | 19:50 |
| Shrews | that's going to exercise nodepool-launcher quite nicely | 19:50 |
| jlk | it's a rebase! | 19:50 |
| SpamapS | all zig for patches | 19:54 |
| dkranz | pabelanger: I have zookeeper running and was going to try to run nodepool with your playbooks. But I was not sure how I specify my particular config options. Any hints? | 19:55 |
| *** harlowja has joined #zuul | 19:58 | |
| pabelanger | dkranz: http://git.openstack.org/cgit/openstack-infra/nodepool/tree/devstack/plugin.sh?h=feature/zuulv3#n195 is the nodepool.yaml file we are testing in the gate | 19:58 |
| pabelanger | that will have zookeeper settings to use | 19:59 |
| dkranz | pabelanger: ok, cool. How do I tell the playbook to use my file? | 19:59 |
| pabelanger | dkranz: did you create a playbook? | 19:59 |
| dkranz | pabelanger: no, I was using your ansible playbook. does it just do the install? | 20:00 |
| pabelanger | dkranz: if you are using ansible-role-nodepool, you can set the nodepool_file_nodepool_yaml_src variable to a copy of a local nodepool.yaml file. Then when we rerun ansible, the role will use your local copy | 20:01 |
| pabelanger | and place it on the remote node | 20:01 |
| dkranz | pabelanger: thanks, that is what I was looking for | 20:01 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul feature/zuulv3: Add ability to set an insecure prefix/suffix https://review.openstack.org/453851 | 20:05 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul feature/zuulv3: Create wrapper script for executor https://review.openstack.org/453852 | 20:05 |
| * mordred hands dkranz a pie | 20:05 | |
| * SpamapS has begun playing a bit | 20:05 | |
| mordred | SpamapS: neat! | 20:09 |
| *** jamielennox is now known as jamielennox|away | 20:12 | |
| SpamapS | https://bugs.launchpad.net/bugs/1680227 <-- backport request for bubblewrap to Xenial | 20:14 |
| openstack | Launchpad bug 1680227 in Yakkety Backports "Please backport bubblewrap 0.1.7-1 (universe) from zesty" [Undecided,New] | 20:14 |
| SpamapS | doh | 20:16 |
| SpamapS | silly script | 20:17 |
| Shrews | pabelanger: if we bring more projects into the zuulv3 voting realm, maybe we should up the max-servers & min-ready counts in the nodepool config at that time. pretty happy with how it's been performing lately | 20:18 |
| Shrews | at least min-ready, rather. max-servers is probably fine | 20:19 |
| pabelanger | Shrews: ya, I think we can get nodepool next. min-ready should be okay, but max I think we have 10 servers in each region ATM we can use | 20:19 |
| pabelanger | but ya, we can bump if needed | 20:19 |
| SpamapS | hrm | 20:32 |
| SpamapS | mordred: so .. when I run faillocal in bwrap.. I get this: | 20:44 |
| SpamapS | /home/clint/tmp/foo.txt:2017-04-05 13:41:41,505 zuul.AnsibleJob DEBUG Ansible output: ERROR! Invalid callback for stdout specified: zuul_stream | 20:44 |
| SpamapS | mordred: but when I run it without brwap, I get this: | 20:45 |
| SpamapS | /home/clint/tmp/foo_unwrapped.txt:2017-04-05 13:42:56,934 zuul.AnsibleJob DEBUG Ansible output: ERROR! Unexpected Exception: No module named 'zuul' | 20:45 |
| SpamapS | so now I'm really confused | 20:45 |
| mordred | SpamapS: SO ... | 20:47 |
| mordred | SpamapS: we copy zuul's ansible modules to a location (I think in /var) on zuul startup | 20:47 |
| mordred | SpamapS: so you may need to also bind-mount that locatoin | 20:47 |
| SpamapS | oh that's definitely it | 20:48 |
| mordred | I'm not sure about the second thing | 20:48 |
| SpamapS | I have a nice fresh empty /var | 20:48 |
| mordred | SpamapS: /var/lib/zuul/ansible/ | 20:48 |
| SpamapS | state_dir is the variable tho | 20:49 |
| mordred | SpamapS: we do that, incidentally, because pip install -U zuul deletes the files before writing new ones out | 20:49 |
| SpamapS | so need to pass that in as a thing to ro-bind | 20:49 |
| mordred | ++ | 20:49 |
| SpamapS | really enjoying working with bubblewrap | 20:49 |
| SpamapS | if we can layer seccomp and SELinux/AppArmor on it, I'm going to be able to sleep I think. | 20:50 |
| mordred | woot! | 20:50 |
| jlk | Nice! only 5 of that series failed tests. | 20:53 |
| mordred | jlk: woot! | 20:57 |
| jlk | at least two are transient and should clear up with recheck. | 20:57 |
| jlk | so only a couple to focus on. | 20:57 |
| SpamapS | we need to look closer into the transient fails we're all seeing | 20:58 |
| SpamapS | it's going to eat us alive | 20:58 |
| Shrews | jeblair: +0 with comments on https://review.openstack.org/451597 | 21:01 |
| Shrews | fixing in a later review would be fine rather than rebase the entire stack for that | 21:05 |
| Shrews | especially since we don't even generate docs with those docstrings, atm | 21:11 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul feature/zuulv3: Add support for bwrap https://review.openstack.org/453851 | 21:11 |
| * SpamapS jumps the spec gun a little | 21:11 | |
| SpamapS | ^^ this works btw. | 21:11 |
| mordred | SpamapS: WOOT | 21:11 |
| jeblair | Shrews: ah thanks; i guess we have to add the tenant class | 21:11 |
| jeblair | SpamapS: \o/ | 21:11 |
| SpamapS | You just need bubblewrap backported. | 21:11 |
| SpamapS | Which is why it's likely to fail its tests since I added bubblewrap to the bindep | 21:12 |
| Shrews | jeblair: it's in datamodel.rst, but we don't use ":members:" with it. not sure if that's intentional | 21:12 |
| SpamapS | But what you get is a super limited system that shoudl at least provide a nice onion layer for an attacker to escape through. | 21:12 |
| jeblair | Shrews: probably that was just the edge of the api at that time; probably makes sense to add it, since i think these methods will be useful to drivers. | 21:13 |
| SpamapS | It would be nice if we could figure out how to make ssh and ansible not require /dev/null | 21:13 |
| jeblair | speaking of spec -- we should all give the spec a once over and make sure it's added to next tuesday's infra team agenda. | 21:13 |
| Shrews | jeblair: also, i *think* the class itself must be documented before member docs will appear | 21:13 |
| jeblair | Shrews: i think you're right | 21:13 |
| Shrews | jeblair: i can clean that up (and my nit comments) with a review. gives my fingers something to do | 21:14 |
| jeblair | Shrews: ++ | 21:14 |
| jeblair | SpamapS: the spec says "security pants". that tells me it's going to be good. | 21:15 |
| mordred | Shrews: don't you have the backporting super powers? | 21:18 |
| Shrews | tab fail | 21:18 |
| mordred | gah | 21:18 |
| mordred | SpamapS: don't you have the backporting super powers? | 21:18 |
| jeblair | SpamapS: jhesketh left a q on the spec about pros/cons of seccomp | 21:19 |
| Shrews | pabelanger: hrm, RETRY_LIMIT failure still showing up for me: https://review.openstack.org/453713 | 21:20 |
| jeblair | Shrews, pabelanger: did we restart zuul since the fixes landed? | 21:21 |
| Shrews | oh, maybe we need a zuul restart | 21:21 |
| Shrews | yeah, that | 21:21 |
| jlk | oh boy | 21:21 |
| jeblair | Shrews: want to go ahead and do that? should be gtg now. | 21:21 |
| jlk | So I just did all those rebases, but if this change goes in to add canonical host names, I'm going to have to rebase it all again. | 21:21 |
| jlk | oh well. | 21:21 |
| Shrews | jeblair: sure. is it zl01 ? | 21:22 |
| jeblair | Shrews: zuulv3-dev | 21:22 |
| Shrews | heh. glad i asked | 21:22 |
| jeblair | :) | 21:22 |
| jeblair | jlk: yes, when i was writing it, i was thinking that it will have some impact on the github stack. but i tried to keep it minimal. but also, from our chat last week, i think it's essential for progress on it. so hopefully worth it. | 21:23 |
| Shrews | jeblair: do both scheduler and executor need restarting? | 21:23 |
| jlk | yeah it's necessary work. I am not upset | 21:23 |
| Shrews | and is there an order? | 21:23 |
| jeblair | Shrews: both (one bug in each) | 21:24 |
| Shrews | ok. restarted | 21:25 |
| jeblair | Shrews: and order shouldn't generally matter. | 21:25 |
| Shrews | jlk: lemme test my change before you try your stack :) | 21:25 |
| jlk | I'm limited to one rebase a day, it's fine :) | 21:25 |
| jeblair | haha | 21:26 |
| jlk | jeblair: question, something came up while we were thinking about limiting github API calls. With this new model you've done, is there a way that when an event comes in, and Zuul constructs the event object, some details of that event are carried through? For example, when an event is built up, we have to construct a PullRequest object, which takes API calls. That is so we can get data and feed Event attributes. Then the event goes off and is scheduled, then | 21:28 |
| jlk | picked up later by the Source I think. At that point, we're constructing a new PullRequest object from the event details, repeating many of the same API calls. | 21:28 |
| jeblair | SpamapS: security spec looks great to me. jhesketh has 2 comments. | 21:28 |
| Shrews | hrm, zuul still no happy | 21:28 |
| jlk | We're hoping at some point to be able to persist a data object from event throughout, so that we don't call the same APIs a bunch. | 21:28 |
| jlk | not sure if this change has anything to do with that. | 21:29 |
| jlk | ( jamielennox|away can explain this better ) | 21:29 |
| Shrews | jeblair: pabelanger: hrm, /opt/zuul is out of date by days. is it not puppeted on zuulv3-dev? | 21:29 |
| jeblair | Shrews: oh, right, i think pupet is disabled. | 21:30 |
| jeblair | Shrews: so you'll want to do a manual pull there, then 'sudo pip install .' then restart again. | 21:30 |
| Shrews | k | 21:30 |
| mordred | jlk: maybe some sort of caching layer perhaps? | 21:30 |
| jeblair | jlk: the gerrit driver maintains a cache of change objects, so that when we later need to getChange, it returns a cached object. basically for the same reason (it performs many expensive api calls at event time) | 21:30 |
| jlk | mordred: we're doing a caching layer, yes. | 21:30 |
| * mordred thinks jeblair said better words | 21:31 | |
| jeblair | jlk: the sequence is: connection gets event; connection updates internally cached Change; connection creates TriggerEvent (which does not directly reference Change but describes it); connection submits TriggerEvent to scheduler; scheduler pops TriggerEvent; scheduler asks source for Change; source asks connection for Change; connection returns cached Change object. | 21:33 |
| jlk | ]I see | 21:33 |
| jeblair | (we could consider attaching the Change object to the TriggerEvent directly, however, it turns out to be really useful to have the cache once you start doing dependencies between changes, so i think we need it anyway.) | 21:35 |
| jeblair | fungi, clarkb, mordred: i think it's worth giving the zuulv3 security spec at https://review.openstack.org/444495 a once-over this week so that we can put it on the infra agenda next week. | 21:40 |
| Shrews | zuul is happy again | 21:41 |
| pabelanger | ya, we have puppet disabled still | 21:41 |
| pabelanger | woot | 21:41 |
| *** hashar has quit IRC | 21:43 | |
| fungi | awesome. i already had it starred, will make sure to go through now | 21:49 |
| *** harlowja has quit IRC | 22:05 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/nodepool master: Add mqtt module to enable pushing MQTT messages https://review.openstack.org/453915 | 22:12 |
| *** jamielennox|away is now known as jamielennox | 22:13 | |
| SpamapS | mordred: no I don't have backports access, and I relinquished my SRU privs too. Can still upload to dev. | 22:30 |
| jlk | ugh this is frustrating. I can't get a full suite of tests to pass. | 22:51 |
| jhesketh | Morning | 23:01 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/nodepool master: Add mqtt module to enable pushing MQTT messages https://review.openstack.org/453915 | 23:12 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/nodepool master: Enabled sending MQTT messages for node events https://review.openstack.org/453922 | 23:12 |
| *** harlowja has joined #zuul | 23:25 | |
| SpamapS | Hrm, I wonder if we'll have to do some tap-dancing to get a backports package into bindep. | 23:28 |
| jeblair | SpamapS: i think the -backports repos are standard on our nodes | 23:31 |
| SpamapS | jeblair: right, I'm just not sure if packages that don't exist in the regular repo are pulled without a backports/ prefix (which is how you get newer packages that have been updated via backports) | 23:33 |
| SpamapS | my apt brain is a bit fuzzy | 23:33 |
| jeblair | huh, i thought they just showed up as candidates with later versions. | 23:34 |
| SpamapS | they don't, the repo metadata has a setting to prevent that | 23:34 |
| SpamapS | http://us.archive.ubuntu.com/ubuntu/dists/xenial-backports/Release has "NotAutomatic: yes" (amazing work there wordsmith apt engineers) | 23:35 |
| jeblair | SpamapS: oh. then i now understand the problem you describe. i'm now also curious as to the answer. :) | 23:36 |
| SpamapS | I'm looking for candidate | 23:36 |
| jeblair | DisableInhibitNonAutomaticInstallation: true | 23:37 |
| jeblair | would be so much clearer | 23:38 |
| SpamapS | oh hooray, it works | 23:38 |
| SpamapS | jeblair: so it looks like NotAutomatic just forces the apt source to 100 priority as if you apt pinned it that way. | 23:39 |
| SpamapS | so if the package does not exist in the main archive, you'll get the backported one. | 23:39 |
| jeblair | nice | 23:40 |
| SpamapS | backports team looks a little overtaxed these days | 23:40 |
| SpamapS | https://bugs.launchpad.net/xenial-backports | 23:41 |
| SpamapS | 43 waiting | 23:41 |
| SpamapS | well, 40.. a few are support reqs | 23:41 |
| *** jamielennox is now known as jamielennox|away | 23:42 | |
| *** jamielennox|away is now known as jamielennox | 23:45 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!