| *** rlandy has quit IRC | 00:51 | |
| *** jesusaur has quit IRC | 01:22 | |
| *** jesusaur has joined #zuul | 01:24 | |
| *** smyers has quit IRC | 03:08 | |
| dmsimard | I wonder if we need to copy the literal inventory file or if we can use https://image.slidesharecdn.com/moretipsntricks-161011152406/95/more-tips-n-tricks-11-638.jpg?cb=1476199481 | 03:09 |
|---|---|---|
| dmsimard | (slide from https://www.slideshare.net/mobile/bcoca/more-tips-n-tricks) | 03:10 |
| openstackgerrit | Merged openstack-infra/zuul-base-jobs master: Add libre2 to bindep https://review.openstack.org/566731 | 05:02 |
| openstackgerrit | Merged openstack-infra/zuul-base-jobs master: Add logo to docs https://review.openstack.org/566412 | 05:02 |
| *** jesusaur has quit IRC | 05:14 | |
| *** elyezer has quit IRC | 05:17 | |
| *** jesusaur has joined #zuul | 05:19 | |
| *** elyezer has joined #zuul | 05:20 | |
| *** pcaruana has joined #zuul | 06:01 | |
| *** jesusaur has quit IRC | 06:45 | |
| *** jesusaur has joined #zuul | 06:46 | |
| *** gtema has joined #zuul | 06:49 | |
| *** zhuli has left #zuul | 06:57 | |
| *** jpena|off is now known as jpena | 07:52 | |
| *** zigo_ has quit IRC | 08:40 | |
| *** ssbarnea_ has joined #zuul | 08:41 | |
| *** zigo has joined #zuul | 08:45 | |
| *** gtema has quit IRC | 09:01 | |
| *** sshnaidm|rover is now known as sshnaidm|lnch | 09:34 | |
| *** yolanda_ has joined #zuul | 10:40 | |
| *** yolanda has quit IRC | 10:44 | |
| *** jpena is now known as jpena|lunch | 11:46 | |
| *** sshnaidm|lnch is now known as sshnaidm|rover | 11:46 | |
| *** gtema has joined #zuul | 12:00 | |
| openstackgerrit | Benedikt Löffler proposed openstack-infra/zuul master: Add role information to task in zuul_json callback https://review.openstack.org/559998 | 12:13 |
| *** rlandy has joined #zuul | 12:32 | |
| *** openstackgerrit has quit IRC | 12:49 | |
| *** jpena|lunch is now known as jpena | 12:57 | |
| *** dkranz has joined #zuul | 13:12 | |
| *** openstackgerrit has joined #zuul | 13:24 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Clean up configuration file loading https://review.openstack.org/566886 | 13:24 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Clean up configuration file loading https://review.openstack.org/566886 | 13:38 |
| corvus | pabelanger: there's a question on https://storyboard.openstack.org/#!/story/2000791 | 13:41 |
| *** ssbarnea_ has quit IRC | 13:42 | |
| *** swest has quit IRC | 14:09 | |
| *** smyers has joined #zuul | 14:38 | |
| *** acozine1 has joined #zuul | 15:17 | |
| mrhillsman | if i modify nodepool.yaml do i need to restart builder and launcher for the change(s) to get picked up? | 15:29 |
| clarkb | mrhillsman: shouldn't need to. They reread their configs as they run | 15:29 |
| mrhillsman | ty sir | 15:30 |
| clarkb | mrhillsman: if you find a case where it isn't working that is likely a bug | 15:30 |
| pabelanger | corvus: thanks, I've been meaning to start a thread on ML about it. | 15:37 |
| pabelanger | I'll reply to storyboard | 15:37 |
| gtema | pabelanger: I'm here, who raised the question | 15:37 |
| pabelanger | gtema: ah. So, if you haven't looked at, I would recommend looking at: http://git.openstack.org/cgit/openstack/windmill While it is lacking major documentation (hoping to address that shortly) it should give you a working deployment for zuul / nodepool. There are also roles for nodepool / zuul / dib / etc | 15:39 |
| gtema | pabelanger: ok, great | 15:41 |
| pabelanger | gtema: I'm doing a workshop at opendevconf in a few weeks, plan is to help users with a deployment. I'm hoping to take the notes / instructions from that and build out documentation from it | 15:43 |
| gtema | just a short question, why have you frozen 2.3<=ansible<2.4. Any particular reason for that? | 15:44 |
| pabelanger | no reason, I just bumped it to 2.4 then 2.5 came out | 15:45 |
| gtema | pabelanger: ok, so I can play around with fixing thigs then | 15:45 |
| pabelanger | yah, part of the ML post I was going to create on zuul-discuss was questions around with version of ansible we should support, and distro. Today it is ansible 2.4 with ubuntu-bionic, ubuntu-xenial and fedora-27 (soon 28) | 15:47 |
| gtema | pabelanger: it is exactly one thing I have found (and see you have addressed in windmill), that zuul in venv uses global ansible (in my case 2.5) and fails | 15:47 |
| pabelanger | we have the ability to test with debian-stretch and opensuse, but haven't added support | 15:47 |
| pabelanger | gtema: yah, should be able to fix that with setting up PATH to virtualenv | 15:47 |
| pabelanger | for zuul-executor | 15:48 |
| gtema | yes, but also to fix things from ansible, which are failing | 15:48 |
| gtema | pabelanger: will start working on this | 15:48 |
| pabelanger | sure | 15:49 |
| gtema | pabelanger: in f28 it is ansible 2.5, that's why I faced it | 15:51 |
| pabelanger | gtema: yah, we setup ansible via pip today, so it is capped. But do plan on moving to fedora-28, we just added images in openstack-infra last week | 15:52 |
| clarkb | Shrews: is https://review.openstack.org/#/c/566886/ pre work for multi label support? | 16:08 |
| clarkb | oh the topic indicates that yes probably | 16:08 |
| Shrews | clarkb: yep | 16:08 |
| clarkb | I will try to make time to take a look while yesterday's conversation is still fresh | 16:09 |
| Shrews | Plus it's just been bugging me | 16:09 |
| *** harlowja has joined #zuul | 16:17 | |
| jlk | Hey folks! this is one of the new features I hinted at while in Dublin. I'm going to be exploring it a bit more in depth and determining if we can take advantage of it with Zuul: https://developer.github.com/changes/2018-05-07-new-checks-api-public-beta/ | 16:43 |
| jlk | It provides for a richer feedback of what has been ran and the results and such, which may be a solution for our desire to indicate multiple tests having ran, some with pass, some with fail. I believe SpamapS was interested in such things. | 16:44 |
| gundalow | jlk: Was reading about that yesterday, looks very cool | 16:44 |
| mordred | jlk: neat | 16:45 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool master: Update logging format for devstack jobs https://review.openstack.org/566942 | 16:45 |
| jlk | this is kind of cool, there is now an API end point to trigger tests for a given PR/hash | 16:47 |
| jlk | so we COULD from Zuul trigger pipelines in other PRs, by way of the GitHub API | 16:47 |
| tobiash | hrm, just updated to ansible 2.5 and most of our shell tasks seem to fail | 16:48 |
| *** EmilienM is now known as EmilienM_PTO | 16:49 | |
| tobiash | at least shell with args: executable | 16:49 |
| *** ssbarnea_ has joined #zuul | 16:49 | |
| clarkb | tobiash: tristanC mentioned that command tasks fail with that but shell should still accept it iirc | 16:49 |
| tobiash | these are shell tasks, not command | 16:50 |
| jlk | OH cool, we can set "queued" too, as soon as we get the webhook, which is a different state than "in_progress", which can solve the issue where we wanted to have a clickable URL when tests are running for live logs, but couldn't PRODUCE that URL until the tests actually start, after the queueing. | 16:50 |
| tobiash | so doing rollback now and analysis tomorrow | 16:50 |
| clarkb | tobiash: huh executable was how you set the shell path iirc. Seems odd that that would regress | 16:51 |
| tobiash | yes, we use things like pipefail which need bash | 16:52 |
| clarkb | ya we use bash in some places too iirc | 16:53 |
| pabelanger | The executable parameter is removed since version 2.4. If you have a need for this parameter, use the shell module instead. | 16:54 |
| pabelanger | http://docs.ansible.com/ansible/latest/modules/command_module.html | 16:54 |
| pabelanger | so, we need to use shell, not command | 16:54 |
| pabelanger | I know OSA had some issue | 16:55 |
| pabelanger | maybe a regression in 2.5 with shell? | 16:55 |
| tobiash | pabelanger: I'm using the shell module | 16:55 |
| pabelanger | yah, I wonder if something in 2.5 maybe broken with 2.4 removal | 16:55 |
| Shrews | well that doesn't sound good | 16:56 |
| tobiash | seems like every job we have using executable is broken with 2.5 | 16:56 |
| pabelanger | tobiash: what is the error? I can try 2.5 here this afternoon | 16:59 |
| corvus | jlk: cool! i noticed the checks thing yesterday too :) | 16:59 |
| tobiash | I just get "MODULE FAILURE" | 16:59 |
| pabelanger | kk | 16:59 |
| tobiash | and didn't have a quick chance to get any more inf | 16:59 |
| tobiash | info | 16:59 |
| jlk | This looks neat, but it's going to take a bunch of code. | 16:59 |
| jlk | and introduces another state thing to worry about. There can now be a "check run object", which is created when an app (Zuul) is asked to run checks for a PR. It's that object we create via the API and then update with details as a check run progresses, so queued -> in_progress -> completed, and posting various details along the way. We'll have to keep POSTing to that specific check run object ID, which is the state thing to track. | 17:01 |
| corvus | jlk: ok. we can keep that in memory for now; managing that will probably get easier when we move zuul state into zookeeper later. here's a question -- if we abort a run and restart it, do we create a new check run object, or reuse an existing one? | 17:02 |
| gtema | tobiash: one more change for 2.5 would be to get strip_internal_keys from ansible.vars.clean in zuul_json.py (there is relocation history in 2.3, 2.4 and now 2.5) | 17:03 |
| jlk | but what we gain is the ability to post back "output" objects, that have a title, summary, text, annotations, and images. We summary/text support markdown, so rich text feedback for the run. Annotations allow us to point to specific lines of code in the PR with feedback from a given check (linter pointing to the specific line problem, etc..). Images could be interesting to do flow charts of logic things (like zuul pipelines). | 17:04 |
| jlk | corvus: that requires some careful thought. Aborting a run and re-starting it I _believe_ should be a new run object, and we would set the old run object to a state of 'completed', with a conclusion of 'cancelled' | 17:04 |
| Shrews | gtema: that's already taken care of in https://review.openstack.org/562668 | 17:05 |
| tobiash | Shrews: was faster... | 17:05 |
| gtema | Shrews: great | 17:06 |
| gundalow | jlk: Have you see how we use ansible-test & ansibullbot to give user feedback: https://github.com/ansible/ansible/pull/39846#issuecomment-387306637 | 17:06 |
| corvus | jlk: okay, that has some implications for the reporting api, which currently doesn't report interim results at all. we *might* want to think about just creating one check run object, and using it for multiple interim buildsets just to keep things simple. | 17:06 |
| gtema | Shrews: there is also some strange thing with output: I do often get "Log Stream did not terminate" | 17:07 |
| gtema | don't know so far what is causing that | 17:07 |
| jlk | gundalow: I've followed some of it over time. | 17:07 |
| jlk | gundalow: but that's a good example of the type of data that can go into a check run output object | 17:07 |
| openstackgerrit | Merged openstack-infra/nodepool master: Remove use of six https://review.openstack.org/566159 | 17:08 |
| jlk | and in fact could point to the specific lines, so that when looking at the PR diff you'd see the box saying "this test failed on this line" | 17:08 |
| *** jpena is now known as jpena|off | 17:08 | |
| gundalow | jlk: Yup, similar patterns would be cool with new GH functionality. We (Ansible) are waiting to hear back from Shippable as to what their plans for support are | 17:08 |
| tobiash | gtema: you need to run the zuul_stream module at the begin of the build | 17:08 |
| clarkb | jlk: ya thats something we've talked about adding for gerrit too | 17:08 |
| clarkb | (gerrit api supports it just haven't implemented in zuul) | 17:08 |
| jlk | corvus: that could work, if it's all internal to Zuul (the re-runs). However if a user manually requests a re-run (like the existing 'recheck' comment), we'd want to produce a new check run | 17:08 |
| corvus | jlk: yep | 17:08 |
| gundalow | I've built a new image for use with Zuul. What's the process to add that into Nodepool? I know the final step is to update the `name:` in https://softwarefactory-project.io/r/gitweb?p=config.git;a=blob;f=nodepool/ansible-network.yaml#l11 Just not sure how to get the image from my laptop into that | 17:10 |
| gtema | tobiash: ok, has it changed? I have switched zuul installation to a new server and now have this issue | 17:10 |
| jlk | Looks like a user/app can request the whole check suite run (which would be every app installed), or a specific check_run (specific app). | 17:10 |
| tobiash | gtema: wait, it's named zuul_console | 17:10 |
| tobiash | gtema: http://git.openstack.org/cgit/openstack-infra/zuul-jobs/tree/roles/start-zuul-console/tasks/main.yaml | 17:11 |
| jlk | all of this is only for apps. Is apps on GHE yet? | 17:11 |
| clarkb | gtema: you'll upload the image to your rdo-cloud using openstackclient (assuming it is openstack) then udpate lines 11 and 15 of your linked ansible-network.yaml | 17:11 |
| clarkb | er gundalow ^ sorry gtema | 17:11 |
| jlk | (I should probably know this already) | 17:11 |
| tobiash | jlk: apps is on ghe since 2.12 | 17:12 |
| tobiash | ;) | 17:12 |
| jlk | woo! | 17:12 |
| tobiash | we're using it since end of december | 17:12 |
| corvus | gundalow: it's really hard to read uuids in config files; i recommend you do this instead: http://paste.openstack.org/show/720603/ | 17:14 |
| corvus | gundalow: that lets you update the uuid in just one place | 17:14 |
| gundalow | corvus: oh, nice. Will do. Thanks for the suggestion | 17:15 |
| corvus | gundalow: (you can also use "image-name" instead of "image-id" and then have no uuids) | 17:15 |
| jlk | corvus: mapping it in my head... GitHub will generate a new webhook event, CheckSuiteEvent when a PR is opened/modified, or if a user/app requests a CheckSuite run. That event goes out to all apps that are set up correctly. Zuul would get that event and create a check_run object, which creates a CheckRunEvent. We should ALSO consume that CheckRunEvent and use it to start the actual work, passing along that check run ID as part of the event data and added to | 17:18 |
| jlk | the zuul change object data. We may get more CheckRunEvents for 'updated' or 'rerequested'. That rerequested one would cause us to start the run over, creating a NEW check run. | 17:18 |
| jlk | oi, that's a lot to wrap a head around. | 17:18 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Test shell module https://review.openstack.org/566951 | 17:18 |
| tobiash | let's see if that reproduces ^ | 17:18 |
| corvus | jlk: if we get a rerequest while it's running, we're likely to just ignore it. | 17:19 |
| jlk | hrm. | 17:19 |
| corvus | jlk: i mean, we can act on it, but it will probably just map to a pipeline-enqueue action, and if it's already there, that's a noop | 17:20 |
| gundalow | clarkb: corvus So looking at the last row on https://ansible.softwarefactory-project.io/zuul/nodes.html that makes me think I need to update ansible-network image on the `rdo-cloud` provider. Which I don't think I have access to. | 17:20 |
| jlk | not so sure on that. If the user is requesting that we re-start the check run, it could be because the check depends on an external resource update that they've performed since the tests started. | 17:20 |
| corvus | gundalow: we may need to find a softwarefactory admin to help: dmsimard, pabelanger, tristanC? | 17:21 |
| *** gtema has quit IRC | 17:22 | |
| corvus | jlk: yeah. it's a choice. so far, we've chosen the other way in zuul -- tests run to completion unless something that zuul knows about changes. we *could* implement the other thing, but if we do so, we should do it with a flag, because there are good reasons we chose this way so far (among them, discouraging recheck bashing) | 17:24 |
| jlk | nod | 17:24 |
| corvus | (if we choose, it shouldn't be hard to implement) | 17:24 |
| *** acozine1 has quit IRC | 17:26 | |
| jlk | Seems reasonable. | 17:30 |
| jlk | Do we have a way for a user to flat out cancel a test run? | 17:30 |
| jlk | or does it just always go to some conclusion? | 17:30 |
| tobiash | jlk: closing the pr | 17:30 |
| tobiash | but there is currently no way for the user to cancel non-change triggered jobs | 17:31 |
| *** harlowja has quit IRC | 17:31 | |
| corvus | zuul has an intentionally minimalist user interface. we don't have lots of buttons for individual actions because it's designed to be completely automated for the general use case. | 17:32 |
| jlk | correct. This would'nt be a "UI" kind of thing, but a response to stimuli from the code source (eg GitHub) | 17:33 |
| jlk | I'll note that there is no GitHub way to say "cancel tests and stop", just "rerun tests" | 17:33 |
| corvus | good choice, i think :) | 17:34 |
| pabelanger | gundalow: corvus: IIRC tristanC is only admin. #softwarefactory should also have the ability | 17:40 |
| pabelanger | but, would be better if nodepool-builder handle image build / publishing, I know you are using another build tool today | 17:42 |
| gundalow | pabelanger: Ah, OK. Not sure how often we will be doing newer builds, though we will have 7+ platforms | 17:44 |
| gundalow | Automate all the things | 17:45 |
| *** jlvillal is now known as jlvacation | 17:51 | |
| *** ssbarnea_ has quit IRC | 17:59 | |
| *** acozine1 has joined #zuul | 18:00 | |
| *** myoung|ruck is now known as myoung|ruck|biab | 18:07 | |
| tobiash | pabelanger, clarkb: that reproduced the broken shell task: http://logs.openstack.org/51/566951/1/check/zuul-tox-remote/9dd5c47/testr_results.html.gz | 18:08 |
| tobiash | but I was mistaken, it's not because of the executable argument but also affects normal shell tasks | 18:09 |
| tobiash | http://logs.openstack.org/51/566951/1/check/zuul-tox-remote/9dd5c47/job-output.txt.gz#_2018-05-08_18_05_03_072742 | 18:09 |
| pabelanger | permission issue? | 18:10 |
| pabelanger | maybe it is our embedded command module that is the issue | 18:10 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Force driver provider configs to define pool attr https://review.openstack.org/566963 | 18:10 |
| tobiash | looks like a permission issue | 18:11 |
| tobiash | now that I look into the json log I see the same error in my logs too | 18:11 |
| clarkb | tobiash: [Errno 13] Permission denied: '/tmp/console-None.log' is that the error? | 18:11 |
| tobiash | clarkb: yes | 18:11 |
| clarkb | I'm guessing some value is now None that was an actual uuid or similar and now we don't find that file because its not there? | 18:12 |
| pabelanger | seems possible | 18:12 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Force driver provider configs to define pool attr https://review.openstack.org/566963 | 18:14 |
| tobiash | pabelanger, clarkb: looks like it is constructed here: http://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/ansible/library/command.py#n148 | 18:14 |
| clarkb | tobiash: http://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/ansible/library/command.py#n377 is what sets the log_uuid | 18:16 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Force driver provider configs to define pool attr https://review.openstack.org/566963 | 18:16 |
| Shrews | i will eventually get the commit message correct | 18:16 |
| clarkb | and that is set as an ansible module parameter | 18:17 |
| clarkb | tobiash: I'm guessing that the command/shell modules aren't being monkeypatched properly | 18:17 |
| clarkb | and are likely using the built in module instead of zuul which will ignore the log id | 18:17 |
| tobiash | clarkb: I think this probably doesn't match anymore: http://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/ansible/callback/zuul_stream.py#n197 | 18:20 |
| tobiash | that callback injects the zuul_log_id which it now doesn't -> None | 18:21 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Test shell module https://review.openstack.org/566951 | 18:24 |
| *** harlowja has joined #zuul | 18:25 | |
| *** snapiri- has joined #zuul | 18:26 | |
| *** snapiri has quit IRC | 18:28 | |
| *** ssbarnea_ has joined #zuul | 18:38 | |
| *** openstackgerrit has quit IRC | 18:49 | |
| *** openstackgerrit has joined #zuul | 18:52 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Test shell module https://review.openstack.org/566951 | 18:52 |
| tobiash | clarkb, pabelanger: I think that might be the fix ^\ | 18:52 |
| clarkb | tobiash: we should double check there aren't any other modules that were based on the command module that now need to be in that list too | 18:54 |
| tobiash | clarkb: do you know what other modules are based on the command module? | 18:55 |
| tobiash | I thought it were only command and shell? | 18:55 |
| clarkb | tobiash: http://docs.ansible.com/ansible/latest/modules/list_of_commands_modules.html raw and script maybe? | 18:55 |
| tobiash | clarkb: ah, ok, but script is already in the test suite since the secfixes | 18:56 |
| tobiash | clarkb: raw and script are not affected because they don't execute remote python | 19:15 |
| clarkb | tobiash: cool thanks for checking | 19:15 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Test more action modules https://review.openstack.org/567007 | 19:20 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Update to Ansible 2.5 https://review.openstack.org/562668 | 19:20 |
| tobiash | clarkb: the fix worked, splitted test addition and fix into ^ | 19:21 |
| tobiash | Shrews: ^ | 19:24 |
| *** myoung|ruck|biab is now known as myoung|ruck | 19:29 | |
| *** acozine1 has quit IRC | 19:40 | |
| mrhillsman | can x86 machine build arm64 VMs via nodepool | 19:42 |
| pabelanger | I don't believe so, we stood up an arm64 nodepool-builder | 19:43 |
| mrhillsman | ok, thx | 19:43 |
| pabelanger | I mean, maybe you could wrap qemu-arm around chroot? | 19:44 |
| tobiash | I guess you would need dib in qemu in dib ;) | 19:45 |
| mrhillsman | :) | 19:45 |
| tobiash | mrhillsman: but for windows I have a script that behaves like dib and uses a cloud and ansible for image creation | 19:46 |
| tobiash | That can be used inside nodepool | 19:47 |
| pabelanger | yah, nodepool used to do that before, snaphot builds | 19:47 |
| pabelanger | we talked a while back about doing it still, but somebody would need to write the driver for it | 19:48 |
| clarkb | I have heard third party reports of some people doing cross builds for raspis | 20:00 |
| clarkb | but its likely to be significantly simpler to just build on the same arch | 20:00 |
| clarkb | tobiash: debuntu has a chroot mechanism for chrooting arm7 on amd64 or something | 20:01 |
| clarkb | tobiash: I don't quite understand how it all works but supposedly it does | 20:01 |
| mnaser | i was wondering why x86 cant build arm images | 20:10 |
| mnaser | i mean especially if it pretty much just 'unzips' dpkg's? | 20:10 |
| mnaser | especially because it doesn't ever boot a vm or anything | 20:10 |
| mnaser | but i guess chroot is where it goes wrong | 20:10 |
| mrhillsman | simple sounds perfect clarkb :) | 20:13 |
| *** pcaruana has quit IRC | 20:21 | |
| *** dkranz has quit IRC | 20:58 | |
| *** dkranz has joined #zuul | 20:59 | |
| *** dkranz has quit IRC | 21:51 | |
| *** myoung|ruck is now known as myoung|ruck|bbl | 22:23 | |
| *** ssbarnea_ has quit IRC | 23:13 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
