| *** jamesmcarthur has quit IRC | 00:03 | |
| jlk | clarkb: corvus: I also got confirmation that the people working on our slack integration (not sure if it's the public one or internal) do the caching method we've talked about | 00:03 |
|---|---|---|
| jlk | clarkb: corvus: and they'd be interested in the new API too! | 00:04 |
| corvus | jlk: thanks! | 00:04 |
| *** openstackgerrit has joined #zuul | 00:09 | |
| openstackgerrit | Merged openstack-infra/zuul master: web: remove build and job_name filter from the buildset route https://review.openstack.org/636504 | 00:09 |
| *** jamesmcarthur_ has joined #zuul | 00:10 | |
| *** jamesmcarthur_ has quit IRC | 00:12 | |
| *** rlandy is now known as rlandy|bbl | 00:13 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul master: Add Fake Github Review object to test suite https://review.openstack.org/636788 | 00:15 |
| clarkb | corvus: if ^ works you can just squash that down into the change that has the as_dict problem | 00:16 |
| *** jamesmcarthur has joined #zuul | 00:17 | |
| *** jamesmcarthur has quit IRC | 00:19 | |
| *** jamesmcarthur has joined #zuul | 00:20 | |
| *** jamesmcarthur has quit IRC | 00:22 | |
| daniel2 | So I learned that the version of nodepool I have to work with is actually 0.3.0, and I've been given no chance of upgrading it. | 00:27 |
| daniel2 | This feels so counter productive. | 00:27 |
| daniel2 | When it builds the images, it can't actually login because its not actually setuping up the jenkins user. That's using the prepare_node_bare.sh script that was installed | 00:28 |
| *** jamesmcarthur has joined #zuul | 00:31 | |
| *** jamesmcarthur has quit IRC | 00:33 | |
| clarkb | daniel2: you are making snapshot builds then? | 00:38 |
| daniel2 | I believe so, nodepool-builder is building an image, uploading it to openstack and then spinning up an instance | 00:38 |
| clarkb | daniel2: for the initial ssh on a snapshot build nodepool generated a key and set that as metadata on the instance iirc. So you need something like cloud-init to bootstrap that login, then you should be able to run your setup script | 00:38 |
| daniel2 | clarkb: Is that key suppose to be inside of nova? | 00:39 |
| clarkb | daniel2: oh that is the disk image builder workflow in which case disk image builder elements should set up the user | 00:39 |
| daniel2 | thats the case | 00:39 |
| clarkb | in that case your image build needs to configure the user iirc | 00:39 |
| daniel2 | thats done with the setup script? | 00:40 |
| clarkb | with diskimage builder it would be done by your disk image builder elements (one of them). setup script impies to me that you are doing a snapshot not diskimagebuilder build though | 00:41 |
| daniel2 | https://github.com/a10networks-ci/project-config/tree/master/nodepool This is the nodepool config I'm working with. | 00:41 |
| clarkb | daniel2: basically there are two ways nodepool can build an image. The newer more modern method is via local diskimage builder process then uploading into the cloud. The older way is to boot an image already in the cloud, run a setup script or whatever in it, then snapshot that | 00:41 |
| daniel2 | clarkb: no its building an image using nodepool-builder and dick-image-creator | 00:42 |
| daniel2 | disk* | 00:42 |
| clarkb | ok, in that case one of the elements listed at https://github.com/a10networks-ci/project-config/blob/master/nodepool/nodepool.yaml#L44-L53 should create the user and set up the key | 00:43 |
| clarkb | newer nodepool knows how to manage this with metadata and cloud-init/glean, but I don't think that version of nodepool does | 00:43 |
| daniel2 | But it isn't. | 00:43 |
| daniel2 | Thats the problem. | 00:43 |
| clarkb | it isn't creating the user and setting the authorized key? | 00:44 |
| daniel2 | No, it's not. | 00:44 |
| clarkb | ok you'll want to modify an element or add an element to do so then | 00:44 |
| daniel2 | Apparently this used to work 2 years ago, but now I'm questioning how it ever did. | 00:45 |
| clarkb | I think we have a simple example in the nodepool devstack plugin let me dig that up | 00:45 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-preview master: Update gitreview file with correct project name https://review.openstack.org/636791 | 00:46 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-preview master: Add perf testing framework https://review.openstack.org/636792 | 00:46 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-preview master: Reimplement in Rust https://review.openstack.org/636793 | 00:46 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-preview master: Remove C++ version https://review.openstack.org/636794 | 00:46 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-preview master: Use rust:slim base image https://review.openstack.org/636795 | 00:46 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-preview master: Use slice matching for hostname unpacking https://review.openstack.org/636796 | 00:46 |
| clarkb | daniel2: https://git.zuul-ci.org/cgit/nodepool/tree/devstack/plugin.sh#n473 we use the devuser element there (built in to disk image builder) then just below that the DIB_DEV_USER... Var sets the key value | 00:48 |
| *** jamesmcarthur has joined #zuul | 00:49 | |
| mordred | SpamapS: ^^ there's your stuff - and a little more - but I'm not really here | 00:49 |
| daniel2 | clarkb: yeah but like I said, we are using 0.3.0, which is severally outdated and different. | 00:49 |
| clarkb | daniel2: this requirement hasn't changed much unti lrecentl when we added support for the metadata based keys | 00:49 |
| daniel2 | So you're saying I have to create a new element | 00:51 |
| *** jamesmcarthur has quit IRC | 00:51 | |
| clarkb | or use devuser and set the variable like we do in the plugin | 00:53 |
| clarkb | something needs to set the ssh key | 00:53 |
| daniel2 | I dont have the devuser element | 00:53 |
| clarkb | it is built into disk image builder | 00:53 |
| clarkb | https://git.openstack.org/cgit/openstack/diskimage-builder/tree/diskimage_builder/elements/devuser | 00:53 |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul master: Add Fake Github Review object to test suite https://review.openstack.org/636788 | 00:54 |
| *** ruffian_sheep has joined #zuul | 01:03 | |
| ruffian_sheep | Hi,zuul | 01:04 |
| ruffian_sheep | Are you here?SpamapS | 01:05 |
| *** jamesmcarthur has joined #zuul | 01:12 | |
| *** jamesmcarthur has quit IRC | 01:17 | |
| *** jamesmcarthur has joined #zuul | 01:20 | |
| *** jamesmcarthur has quit IRC | 01:24 | |
| SpamapS | mordred: :-D | 01:37 |
| *** bhavikdbavishi has joined #zuul | 01:38 | |
| *** jamesmcarthur has joined #zuul | 01:40 | |
| SpamapS | ruffian_sheep: unfortunately I am just signing off for the day. Might be back later. | 01:41 |
| *** jamesmcarthur has quit IRC | 01:46 | |
| *** jamesmcarthur has joined #zuul | 01:52 | |
| *** jamesmcarthur has quit IRC | 02:00 | |
| *** jamesmcarthur has joined #zuul | 02:15 | |
| *** jamesmcarthur has joined #zuul | 02:16 | |
| *** jamesmcarthur has quit IRC | 02:21 | |
| *** jamesmcarthur has joined #zuul | 02:42 | |
| *** jamesmcarthur has quit IRC | 02:46 | |
| *** jamesmcarthur has joined #zuul | 02:52 | |
| *** ruffian_sheep has quit IRC | 02:55 | |
| *** jamesmcarthur has quit IRC | 03:00 | |
| *** jamesmcarthur has joined #zuul | 03:02 | |
| *** rlandy|bbl is now known as rlandy | 03:10 | |
| *** rlandy has quit IRC | 03:13 | |
| daniel2 | clarkb: thank you for that info, it helped me get past that one issue finally. | 03:31 |
| *** jamesmcarthur has quit IRC | 03:33 | |
| *** jamesmcarthur has joined #zuul | 03:54 | |
| *** jamesmcarthur has quit IRC | 03:59 | |
| *** jamesmcarthur has joined #zuul | 04:04 | |
| *** jamesmcarthur has quit IRC | 04:10 | |
| *** jamesmcarthur has joined #zuul | 04:27 | |
| *** jamesmcarthur has quit IRC | 04:31 | |
| *** bjackman__ has joined #zuul | 04:46 | |
| *** jamesmcarthur has joined #zuul | 04:48 | |
| *** jamesmcarthur has quit IRC | 04:52 | |
| *** jamesmcarthur has joined #zuul | 05:09 | |
| *** jamesmcarthur has quit IRC | 05:13 | |
| *** jamesmcarthur has joined #zuul | 05:30 | |
| *** saneax has joined #zuul | 05:30 | |
| *** jamesmcarthur has quit IRC | 05:35 | |
| *** jamesmcarthur has joined #zuul | 05:51 | |
| *** jamesmcarthur has quit IRC | 05:56 | |
| *** jamesmcarthur has joined #zuul | 06:12 | |
| *** jamesmcarthur has quit IRC | 06:17 | |
| *** snapiri has joined #zuul | 06:29 | |
| *** ruffian_sheep has joined #zuul | 06:30 | |
| ruffian_sheep | SpamapS:Get | 06:31 |
| ruffian_sheep | Can anyone tell me how to solve this problem?zuul-web: zuul-web: error: unrecognized arguments: stop | 06:33 |
| ruffian_sheep | zuul-web: usage: zuul-web [-h] [-c CONFIG] [--version] [-d] | 06:33 |
| ruffian_sheep | I try to restart the zuul-web | 06:33 |
| *** jamesmcarthur has joined #zuul | 06:33 | |
| *** swest has joined #zuul | 06:38 | |
| *** jamesmcarthur has quit IRC | 06:38 | |
| *** saneax has quit IRC | 06:38 | |
| *** quiquell|off is now known as quiquell|rover | 06:47 | |
| *** jamesmcarthur has joined #zuul | 06:54 | |
| *** jamesmcarthur has quit IRC | 07:00 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Implement a Runc driver https://review.openstack.org/535556 | 07:04 |
| *** jamesmcarthur has joined #zuul | 07:05 | |
| *** jamesmcarthur has quit IRC | 07:09 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul-jobs master: Optionally silence git push in mirror-workspace-git-repos https://review.openstack.org/635166 | 07:16 |
| *** bhavikdbavishi has quit IRC | 07:17 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul-jobs master: Optionally silence git in mirror-workspace-git-repos https://review.openstack.org/635166 | 07:21 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul-jobs master: Optionally silence git in mirror-workspace-git-repos https://review.openstack.org/635166 | 07:21 |
| *** jamesmcarthur has joined #zuul | 07:25 | |
| *** jamesmcarthur has quit IRC | 07:30 | |
| *** quiquell|rover is now known as quique|rover|brb | 07:31 | |
| *** bjackman__ has quit IRC | 07:35 | |
| quique|rover|brb | tobiash: good morning | 07:35 |
| quique|rover|brb | tobiash: one vote to go https://review.openstack.org/#/c/633930/ | 07:36 |
| *** bjackman__ has joined #zuul | 07:36 | |
| *** jamesmcarthur has joined #zuul | 07:46 | |
| *** jamesmcarthur has quit IRC | 07:51 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul-jobs master: Optionally silence git in mirror-workspace-git-repos https://review.openstack.org/635166 | 07:52 |
| *** mnasiadka has joined #zuul | 08:03 | |
| tobiash | corvus: not sure if you got an answer regarding that !unsafe tag. quique|rover|brb pointed out in an earlier review comment that this seems to be the correct way in ansible: https://docs.ansible.com/ansible/latest/user_guide/playbooks_advanced_syntax.html#unsafe-or-raw-strings | 08:04 |
| mnasiadka | Hello, does zuul support two changes with cross Depends-On each other (trying to fix failing zuul jobs of two types, don't really want it in one patchset)? Or do I need to include both changes in one Patchset? | 08:06 |
| *** saneax has joined #zuul | 08:06 | |
| tobiash | mnasiadka: no, cyclic dependencies are currently not supported | 08:07 |
| tobiash | (but on the long term roadmap) | 08:07 |
| *** jamesmcarthur has joined #zuul | 08:07 | |
| *** gtema has joined #zuul | 08:07 | |
| mnasiadka | tobiash: so one patchset it is ;) | 08:08 |
| tobiash | quique|rover|brb: +2 from me, but I didn't approve as I saw that corvus had a question about that !unsafe tag | 08:09 |
| *** jamesmcarthur has quit IRC | 08:12 | |
| *** quique|rover|brb is now known as quiquell|rover | 08:15 | |
| quiquell|rover | tobiash: thanks, I will to contact corvus, to see if there are still some issues there | 08:16 |
| ruffian_sheep | Can anyone tell me how to solve this problem?zuul-web: zuul-web: error: unrecognized arguments: stop zuul-web: usage: zuul-web [-h] [-c CONFIG] [--version] [-d] .I try to restart the zuul-web | 08:26 |
| *** jamesmcarthur has joined #zuul | 08:28 | |
| tobiash | ruffian_sheep: it tells you that 'zuul-web stop' won't work | 08:28 |
| tobiash | ruffian_sheep: what are you trying to achieve? | 08:29 |
| ruffian_sheep | I try to build a third party CI for cinder of openstack.Now ,I follow the document:https://zuul-ci.org/docs/zuul/admin/zuul-from-scratch.html. | 08:30 |
| ruffian_sheep | I cannot restart the zuul-web correctly. | 08:30 |
| tobiash | ah you probably found a bug there | 08:31 |
| ruffian_sheep | tobiash:Do you see the document before? | 08:31 |
| ruffian_sheep | I didn't know what's wrong in my steps to do it. | 08:32 |
| *** jamesmcarthur has quit IRC | 08:32 | |
| tobiash | the 'zuul-web stop' way of stopping the zuul components doesn't seem to work with zuul-web yet | 08:34 |
| tobiash | you probably need to kill the process for nwo | 08:34 |
| *** jpena|off is now known as jpena | 08:38 | |
| tobiash | corvus: looks like zuul-web has not command socket yet | 08:38 |
| ruffian_sheep | tobiash:for nwo?what is nwo? | 08:39 |
| tobiash | snwo/now | 08:39 |
| tobiash | I meant 'now' | 08:40 |
| tobiash | sorry for that typo ;) | 08:40 |
| *** electrofelix has joined #zuul | 08:43 | |
| ruffian_sheep | tobiash:I had try to kill it for serveral times.But all of it failed. | 08:43 |
| ruffian_sheep | tobiash:lol | 08:43 |
| tobiash | then remove the stop command for now from the zuul-web.service file, stop the service and manually kill the zuul-web process | 08:44 |
| tobiash | that should hopefully work until we sorted that out | 08:44 |
| tobiash | or reboot ;) | 08:44 |
| ruffian_sheep | tobiash:I have learn som new words last day,including afk from SpamapS and ack from you | 08:44 |
| ruffian_sheep | tobiash:I thought this maybe another new word. | 08:45 |
| ruffian_sheep | tobiash:Sadly,all of the ways you said I had tried in the past | 08:46 |
| *** jamesmcarthur has joined #zuul | 08:48 | |
| tobiash | ruffian_sheep: you could also remove the stop command from the service file and add ' -d' to the zuul-web command line. Then systemd should be able to kill it directly | 08:48 |
| *** jamesmcarthur has quit IRC | 08:53 | |
| *** panda|off is now known as panda | 09:00 | |
| *** jamesmcarthur has joined #zuul | 09:06 | |
| ruffian_sheep | tobiash:Sorry, I am not very clear about this operation. Are you saying to delete this?http://paste.openstack.org/show/745076/ | 09:07 |
| tobiash | ruffian_sheep: yes, remove this line: 'ExecStop=/usr/local/bin/zuul-web stop' | 09:08 |
| tobiash | and change 'ExecStart=/usr/local/bin/zuul-web' to 'ExecStart=/usr/local/bin/zuul-web -d' | 09:09 |
| tobiash | then reboot and restarting zuul-web should probably work | 09:09 |
| *** jamesmcarthur has quit IRC | 09:12 | |
| ruffian_sheep | tobiash:Get | 09:12 |
| ruffian_sheep | tobiash:while wating for the reboot.Can you tell me what's wrong in this case?paste.openstack.org/show/745055/ | 09:13 |
| ruffian_sheep | tobiash:while wating for the reboot.Can you tell me what's wrong in this case?http://paste.openstack.org/show/745055/ (correct ;)) | 09:14 |
| tobiash | looks like it cannot access gerrit | 09:14 |
| ruffian_sheep | tobiash:Is there anything wrong ?http://paste.openstack.org/show/745077/ | 09:24 |
| *** themroc has joined #zuul | 09:25 | |
| tobiash | ruffian_sheep: I don't see a user configured: https://zuul-ci.org/docs/zuul/admin/drivers/gerrit.html#attr-%3Cgerrit%20connection%3E.user | 09:25 |
| tobiash | and I doubt that you can connect to review.openstack.org as user zuul ;) | 09:26 |
| ruffian_sheep | tobiash:Ohh,I just remove it.I have set the user in the connection gerrit | 09:27 |
| tobiash | then you should first check that you can access gerrit manually using that user and key | 09:27 |
| *** jamesmcarthur has joined #zuul | 09:27 | |
| tobiash | I also spotted a different error, you name the project 'test' in your main.yaml | 09:28 |
| tobiash | but that doesn't exist on review.o.o | 09:28 |
| ruffian_sheep | I can see the gerrit event by the cmd ssh -p 29418 myname@review.openstack.org gerrit stream-events | 09:29 |
| tobiash | then it's the non-existing project | 09:29 |
| *** jamesmcarthur has quit IRC | 09:32 | |
| openstackgerrit | Brendan proposed openstack-infra/zuul-jobs master: Use zuul_workspace_root variable for Git workspace prep https://review.openstack.org/636870 | 09:32 |
| openstackgerrit | Brendan proposed openstack-infra/zuul-jobs master: Use zuul_workspace_root variable for Git workspace prep https://review.openstack.org/636870 | 09:34 |
| ruffian_sheep | tobiash:Is the project should exist in the gerrit? | 09:40 |
| tobiash | yes, it must exist, otherwise zuul won't be able to work with the project | 09:40 |
| ruffian_sheep | tobiash:Emmm...I will check it later.Now I cat the messages http://paste.openstack.org/show/745078/.It exist time out | 09:45 |
| ruffian_sheep | tobiash:Do you know the reason? | 09:45 |
| ruffian_sheep | tobiash:http://paste.openstack.org/show/745078/ correct | 09:47 |
| tobiash | ruffian_sheep: the scheduler probably fails because of this missing repo and the other services cannot connect to the scheduler | 09:48 |
| *** jamesmcarthur has joined #zuul | 09:49 | |
| ruffian_sheep | tobiash:What is missing repo?0.0 | 09:52 |
| *** jamesmcarthur has quit IRC | 09:53 | |
| tobiash | as we discussed earlyer you reference a non-existing repo on gerrit which prevents zuul-scheduler from startup. The other zuul services need to connect to zuul-scheduler and thus fail to start too. | 09:54 |
| tobiash | repo == git repository == project on gerrit | 09:54 |
| ruffian_sheep | tobiash:Get | 09:55 |
| ruffian_sheep | tobiash:Let me try to remove the project,just save the name of it. | 09:55 |
| *** AJaeger has quit IRC | 10:01 | |
| ruffian_sheep | tobiash:It's still not working.:http://paste.openstack.org/show/745080/ | 10:02 |
| ruffian_sheep | tobiash:I found that the child process of the executor is not listening on port 4730.Is it right? | 10:03 |
| *** AJaeger has joined #zuul | 10:04 | |
| *** jamesmcarthur has joined #zuul | 10:10 | |
| *** bhavikdbavishi has joined #zuul | 10:12 | |
| *** jamesmcarthur has quit IRC | 10:15 | |
| *** bhavikdbavishi has left #zuul | 10:17 | |
| *** bhavikdbavishi has joined #zuul | 10:17 | |
| *** jamesmcarthur has joined #zuul | 10:31 | |
| *** bhavikdbavishi has quit IRC | 10:34 | |
| *** jamesmcarthur has quit IRC | 10:36 | |
| *** jamesmcarthur has joined #zuul | 10:52 | |
| *** jamesmcarthur has quit IRC | 10:57 | |
| *** jamesmcarthur has joined #zuul | 11:14 | |
| *** jamesmcarthur has quit IRC | 11:18 | |
| *** bjackman__ has quit IRC | 11:34 | |
| *** jamesmcarthur has joined #zuul | 11:35 | |
| *** jamesmcarthur has quit IRC | 11:40 | |
| *** quiquell|rover is now known as quique|rover|r-- | 11:53 | |
| *** jamesmcarthur has joined #zuul | 11:57 | |
| *** jpena is now known as jpena|lunch | 11:58 | |
| *** jamesmcarthur has quit IRC | 12:01 | |
| *** jamesmcarthur has joined #zuul | 12:18 | |
| *** jamesmcarthur has quit IRC | 12:23 | |
| *** bjackman__ has joined #zuul | 12:33 | |
| *** ruffian_sheep has quit IRC | 12:35 | |
| *** jamesmcarthur has joined #zuul | 12:39 | |
| *** jamesmcarthur has quit IRC | 12:44 | |
| *** jamesmcarthur has joined #zuul | 13:01 | |
| *** jamesmcarthur has quit IRC | 13:05 | |
| *** gtema has quit IRC | 13:06 | |
| *** themr0c has joined #zuul | 13:13 | |
| *** themroc has quit IRC | 13:14 | |
| *** jamesmcarthur has joined #zuul | 13:21 | |
| *** jamesmcarthur_ has joined #zuul | 13:22 | |
| *** jpena|lunch is now known as jpena | 13:24 | |
| *** jamesmcarthur_ has quit IRC | 13:29 | |
| *** bjackman__ has quit IRC | 13:39 | |
| *** gtema has joined #zuul | 13:40 | |
| *** rlandy has joined #zuul | 13:41 | |
| *** jamesmcarthur has joined #zuul | 13:49 | |
| *** jamesmcarthur has quit IRC | 13:52 | |
| *** jamesmcarthur has joined #zuul | 13:52 | |
| *** quique|rover|r-- is now known as quiquell|rover | 13:53 | |
| *** jamesmcarthur has quit IRC | 14:01 | |
| *** rfolco is now known as rfolco|off | 14:01 | |
| *** jamesmcarthur has joined #zuul | 14:22 | |
| jkt | tristanC: an update on that zuul-console thing with runc: I'm now seeing quite some non-determinism, these sleeps that I was adding earlier no longer help for some reason | 14:23 |
| jkt | tristanC: one thing which might be relevant is that I've restarted the nodepool launcher many, many times, and the launched zuul-console might have been living for longer. Could that be relevant by any chance? | 14:24 |
| *** jamesmcarthur has quit IRC | 14:26 | |
| tristanC | jkt: zuul-console should be a noop if it's already running, i never seen such issue, one of our host has 80d uptime and the ansible_module_zuul_console.py is healthy | 14:30 |
| tristanC | jkt: it's getting late here, i'll have look tomorrow at executor/zuul-stream/zuul-console interaction to see what could be the culpit | 14:32 |
| tobiash | do we have database (postgres) experts here? | 14:33 |
| tobiash | our build queries that combine tenant, project and pipeline are extremely slow | 14:33 |
| *** gtema has quit IRC | 14:34 | |
| jkt | tristanC: thanks for your help, I really appreciate it | 14:34 |
| *** gcutrini has quit IRC | 14:40 | |
| *** jamesmcarthur has joined #zuul | 14:43 | |
| *** gcutrini has joined #zuul | 14:46 | |
| *** jamesmcarthur has quit IRC | 14:47 | |
| *** nhicher has quit IRC | 14:48 | |
| *** nhicher has joined #zuul | 14:50 | |
| *** electrofelix has quit IRC | 14:53 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Implement a Runc driver https://review.openstack.org/535556 | 15:00 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Implement a Runc driver https://review.openstack.org/535556 | 15:02 |
| tristanC | jkt: perhaps it's because of that missing change: https://review.openstack.org/#/c/535556/19..20/nodepool/driver/runc/playbooks/init.yml | 15:03 |
| *** jamesmcarthur has joined #zuul | 15:04 | |
| tristanC | jkt: there needs to be one and only one zuul-console process running per ip, and it needs to use the same path as the job. | 15:04 |
| tristanC | jkt: thus if you apply the last PS, make sure to kill any ansible_module_zuul_console.py left-over process | 15:04 |
| *** jamesmcarthur has quit IRC | 15:08 | |
| jkt | tristanC: doing that right now, good catch | 15:08 |
| jkt | tristanC: I'll also limit everything to just one runc launcher for now | 15:08 |
| jkt | tristanC: it would be nice if the init/cleanup playbooks killed the console process | 15:09 |
| tristanC | jkt: indeed, though i was hoping zuul console stream would have been refactored by then to not use that zuul-console service ;) | 15:11 |
| tristanC | jkt: in anycase, you should have the task output in the job-output.json file, and i think another solution would be to display that instead in the build page | 15:12 |
| tristanC | jkt: though the console websocket stream is another story, but at least you have the data in the artifact | 15:13 |
| tristanC | jkt: one last thought before i leave, the driver could also be improved to setup a new ip for each container, then each would have their own zuul-console | 15:15 |
| jkt | tristanC: confirming that the job-output.json is complete | 15:16 |
| tristanC | jkt: at the moment, the driver doesn't setup any netns or virtual ip per container | 15:16 |
| jkt | tristanC: one more unrelated point, the console outputs in /tmp are not being removed once the job has finished | 15:16 |
| jkt | it's handy for debugging, though :) | 15:17 |
| jkt | tristanC: that would work for me just fine (assuming it can generate a random MAC and use SLAAC) | 15:17 |
| jkt | I am not that familiar with runc, but I'll take a look | 15:17 |
| tristanC | jkt: runc doesn't do network setup iirc, but we could create the necessary namespace before and make the runc context share the new netns | 15:19 |
| *** saneax has quit IRC | 15:24 | |
| *** jamesmcarthur has joined #zuul | 15:24 | |
| *** jamesmcarthur has quit IRC | 15:29 | |
| *** gtema has joined #zuul | 15:34 | |
| *** jamesmcarthur has joined #zuul | 15:45 | |
| *** jamesmcarthur has quit IRC | 15:50 | |
| *** jamesmcarthur has joined #zuul | 15:52 | |
| *** jamesmcarthur has quit IRC | 15:54 | |
| *** jamesmcarthur has joined #zuul | 15:54 | |
| *** gtema has quit IRC | 15:58 | |
| *** gtema has joined #zuul | 16:00 | |
| clarkb | corvus: https://review.openstack.org/#/c/636764/1 passes tests. I think the only thing missing (and this is relatively minor) is the removal of empty projects from the cache. Do we want to try to get that in first or should we take this as is and see if it helps? | 16:10 |
| clarkb | corvus: for the reuse PR object change we keep hitting deeper errors now that the fake is removed. http://logs.openstack.org/88/636788/2/check/tox-py35/6f4c2b3/job-output.txt.gz#_2019-02-14_01_13_52_952926 I'll see if that is a simple fix like the review as_dict one | 16:16 |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul master: Add Fake Github Review object to test suite https://review.openstack.org/636788 | 16:24 |
| clarkb | I think that was an actual bug with the change ^ should confirm | 16:24 |
| *** gcutrini has quit IRC | 16:28 | |
| *** gcutrini has joined #zuul | 16:31 | |
| *** gcutrini has quit IRC | 16:40 | |
| *** gcutrini has joined #zuul | 16:44 | |
| *** gcutrini has quit IRC | 16:54 | |
| *** quiquell|rover is now known as quiquell|off | 16:57 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: URLTrigger driver time based - artifact change jobs triggering driver https://review.openstack.org/635567 | 17:00 |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: URLTrigger driver time based - artifact change jobs triggering driver https://review.openstack.org/635567 | 17:02 |
| *** bjackman__ has joined #zuul | 17:04 | |
| *** ianychoi has joined #zuul | 17:06 | |
| *** gtema has quit IRC | 17:16 | |
| *** themr0c has quit IRC | 17:30 | |
| *** bjackman__ has quit IRC | 17:47 | |
| corvus | quiquell|off, tobiash: yes, i saw that the commit message says that !unsafe is correct. but raw/endraw is also correct. i wanted to confirm why !unsafe is more correct than raw/endraw. | 17:52 |
| corvus | quiquell|off, tobiash: by deduction from various clues left around, i have concluded that the reason quiquell|off changed from raw/endraw to !unsafe is because raw/endraw can not be easily used to escape a string that itself has "endraw" in it. | 17:53 |
| corvus | quiquell|off, tobiash: is that correct? | 17:53 |
| tobiash | corvus: with rae/endraw you could have problems if you have that endraw tag in the string if you don't additionally escape it | 17:53 |
| tobiash | so I think that's why ansible has this special tag | 17:54 |
| tobiash | corvus: yes, this is correct | 17:55 |
| clarkb | corvus: https://review.openstack.org/#/c/636788/ does pass tests now if you want to squash that into the parent | 17:58 |
| corvus | tobiash, quiquell|off: 633930 +3 -- i did leave a comment on it -- we might want to do that in a followup. | 18:00 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Re-use the github PR object when fetching reviews https://review.openstack.org/636705 | 18:03 |
| corvus | clarkb: done thanks! | 18:03 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Add comment about extra issues request https://review.openstack.org/636706 | 18:03 |
| *** jpena is now known as jpena|off | 18:08 | |
| tobiash | corvus: interesting, does this automatically hook itself into the yaml processing? | 18:09 |
| corvus | tobiash: yes, we use that in configloader | 18:10 |
| tobiash | ah I found and understood the magic :) | 18:16 |
| clarkb | corvus: thoughts on how to move forward with https://review.openstack.org/#/c/636764/1 ? | 18:20 |
| corvus | clarkb: let's ask tobiash to look at it, and if we all think that's the way to go, figure out how we want to do expiration so it doesn't grow without bound | 18:21 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Make UnsafeTag self registering https://review.openstack.org/637023 | 18:21 |
| tobiash | corvus, clarkb: looking | 18:22 |
| corvus | tobiash: that change is designed to help reduce the event backlog we noticed 2 days ago | 18:23 |
| tobiash | corvus: yeah, that was a looong backscroll to read ;) | 18:23 |
| tobiash | corvus: using the change cache that we already have is no option? | 18:24 |
| clarkb | the existing cache caches different data | 18:24 |
| clarkb | it caches the post mangled data only for changes in pipelines | 18:25 |
| corvus | tobiash: could be, it's not indexed the right way though, it's pr->sha, we need sha->pr | 18:25 |
| tobiash | ah ok | 18:25 |
| clarkb | its also only for PRs being tested aiui | 18:25 |
| corvus | and what clarkb says is theoretically true, though, i think at the moment we don't expire it very well. but still, we shouldn't violate it's documented behavior :) | 18:25 |
| clarkb | (which is good because it keeps it small and active but bad when many other PRs could have status updates) | 18:25 |
| tobiash | do we expire the change cache at all? | 18:26 |
| clarkb | tobiash: my read of it was that the intent was to expire contents when they are no longer in a pipeline | 18:27 |
| tobiash | I think I remember an issue with maintain cache? | 18:27 |
| corvus | https://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/scheduler.py#n873 | 18:27 |
| clarkb | fwiw https://docs.python.org/3/library/functools.html#functools.lru_cache might be another option here | 18:27 |
| clarkb | apply that to getPRbySha ? | 18:27 |
| clarkb | it won't warm up as quickly though | 18:27 |
| tobiash | ah yeah, so we still don't expire changes | 18:27 |
| corvus | clarkb: we need external invalidation though -- when a pr's sha updates, we need it removed from the cache. | 18:28 |
| clarkb | ah | 18:28 |
| clarkb | right a PR can have updatign shas | 18:28 |
| jlk | I know y'all aren't GitHub fans, but this is a feature I'm really excited about: https://github.blog/2019-02-14-introducing-draft-pull-requests/ | 18:28 |
| clarkb | jlk: I like that that makes it clear the drafts aren't intend to be secret | 18:29 |
| jlk | yeah | 18:29 |
| clarkb | (this was Gerrit's problem with drafts they were this weird semi secret state that created lots of confusion) | 18:29 |
| jlk | it delays triggering CI stuff too, and required reviews | 18:29 |
| jlk | until the submitter is ready | 18:29 |
| corvus | clarkb: well, gerrit added real WIP a while ago | 18:29 |
| corvus | we just haven't upgraded | 18:29 |
| clarkb | ya | 18:29 |
| tobiash | jlk: cool, and, I know you won't tell me, but when will this be available in ghe? ;) | 18:30 |
| clarkb | corvus: though i think draft still exists? | 18:30 |
| SpamapS | jlk: hah cool. I have a label called 'do-not-merge' that prevents things from entering gate. We can retire that ;) | 18:30 |
| jlk | yup! | 18:30 |
| clarkb | tobiash: corvus if we don't invalidate the other cache maybe its ok to not really expire things from this cache? | 18:30 |
| corvus | clarkb: it gets cleared periodically due to reconfigurations | 18:31 |
| jlk | wait I'm not sure it'll delay CI trigger. I could be wrong on that | 18:31 |
| clarkb | corvus: ah | 18:31 |
| openstackgerrit | Merged openstack-infra/zuul master: Mark as unsafe commit message at inventory https://review.openstack.org/633930 | 18:32 |
| clarkb | corvus: tobiash what if we did a naive thing and set the per project cache size limit to say 4k entries | 18:33 |
| * clarkb does some napkin math for how big that would be for openstack assuming its gerrit was a github | 18:33 | |
| corvus | clarkb: when we hit the limit, what do we expire? | 18:33 |
| tobiash | I guess we can at least remove the sha's of merged prs? | 18:33 |
| corvus | do we want to make it an lru? if so, i think the limit can be much smaller | 18:34 |
| corvus | tobiash: we do that | 18:34 |
| clarkb | ~200MB napkin math for that size of catch. Ya I was thinking lru | 18:34 |
| corvus | tobiash, clarkb: maybe removing closed prs makes it okay to have no limit? | 18:34 |
| tobiash | probably | 18:34 |
| * corvus imagines clarkb reeling in a 200MB fish | 18:34 | |
| clarkb | corvus: oh thats a good point, it will only be open PRs that have seen activity while zuul is running | 18:35 |
| tobiash | the changes we don't clean are probably larger ;) | 18:35 |
| clarkb | considering that ansible is ~1.7k open changes thats smaller than my guestimated cache math above and there is only one of those projects currently. I think if tobiash thinks it is safe for his zuul it is probably safe for all zuuls | 18:37 |
| corvus | at least safe enough for the forseeable future, and easy enough to improve later if needed. | 18:37 |
| tobiash | lol | 18:37 |
| jlk | fwiw, having not read the backlog, I think it would be safe to expire a cache entry for any PR that moves to closed, and update the cache when a push event or a reopen event happens (which would change the head | 18:39 |
| clarkb | jlk: I think the existing code already does that? basically it says if the state is open then add to cache otherwise remove it | 18:41 |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: URLTrigger driver time based - artifact change jobs triggering driver https://review.openstack.org/635567 | 18:42 |
| tobiash | clarkb, corvus: I think that cache implementation is good as is for now | 18:42 |
| clarkb | so ya this may be ready as is | 18:42 |
| corvus | ok, i will verify it's being tested and cleanup | 18:42 |
| tobiash | and my napkin math tells estimates a worst case of around 100mb for us | 18:42 |
| tobiash | typing is hard... | 18:42 |
| tobiash | I think a cleanup of the change cache is probably more important in the medium term | 18:43 |
| corvus | tests.unit.test_github_requirements.TestGithubRequirements.test_trigger_on_status tests this code path already. | 18:44 |
| jlk | okay just got confirmation. PR webhooks are growing a 'draft' status, as is the GET on pull requests via the REST API. | 18:44 |
| jlk | so, zuul will have to decide how it wants to react to draft PRs | 18:44 |
| jlk | ..... which could be a project by project decision (lots o work) or a blanket decision. | 18:45 |
| corvus | if we follow what we do with gerrit in openstack, we'd treat them as 'open' | 18:45 |
| jlk | nod | 18:45 |
| jlk | they're unable to be merged though, so couldn't go to gate. | 18:45 |
| corvus | oh, i guess these could be pipeline requirements | 18:45 |
| corvus | so probably people can decide already :) | 18:45 |
| tobiash | yes | 18:46 |
| tobiash | jlk: is it possible to switch a pr back to draft? | 18:47 |
| jlk | I don't know | 18:47 |
| corvus | i think we have a couple of 'open' literals that we'll need to change to ['open', 'draft'] | 18:47 |
| *** sshnaidm is now known as sshnaidm|off | 18:47 | |
| jlk | corvus: so.. | 18:47 |
| tobiash | and does the hub tool support it? | 18:47 |
| tobiash | I almost never create a pr via the ui | 18:47 |
| tobiash | ;) | 18:47 |
| jlk | corvus: I think the PR state will be `open` even if it's draft. The draft status would be a new key in the json | 18:47 |
| jlk | tobiash: there's an open PR to add it to hub | 18:47 |
| corvus | jlk: ok, i like that | 18:47 |
| tobiash | ah cool | 18:47 |
| jlk | corvus: adding an additional PR state would be pretty disruptive | 18:48 |
| jlk | https://github.com/github/hub/issues/2038 | 18:48 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Cache github PR shas https://review.openstack.org/636764 | 18:51 |
| corvus | tobiash, clarkb: ^ i'm satisfied that's tested by existing tests. we might want to add more explicit unit testing though. i could go either way. | 18:52 |
| clarkb | I'm happy with it being exercised by existing tests | 18:52 |
| tobiash | corvus: posted an idea ^ | 18:56 |
| tobiash | what do you think? | 18:56 |
| clarkb | tobiash: ya I left a similar note on my original change | 18:57 |
| clarkb | I think that would be a good improvement but not necessary to start seeing if this helps | 18:57 |
| corvus | tobiash: oh, wow, how did i miss that | 19:00 |
| corvus | tobiash: i think that's a swell idea -- i think it's simple enough we can get it in now | 19:00 |
| corvus | clarkb: i think it will make a big difference for us and we should include it in the first round of testing | 19:00 |
| clarkb | ok I'm happy to implement it too | 19:01 |
| tobiash | :) | 19:01 |
| clarkb | I had just left it out of my original change too because it potentially introduced more complexity, but I definitely thin kit is something we want | 19:01 |
| corvus | i have to afk for 30m now, so if you have time to write that... :) | 19:01 |
| * clarkb takes a look | 19:03 | |
| *** jamesmcarthur has quit IRC | 19:03 | |
| clarkb | tobiash: corvus I think there may be a small bug in the existing code too. in getPullBySha we use 'project' as the project_name but I think we really want owner/project? | 19:09 |
| clarkb | oh wait I see it nevermind | 19:09 |
| clarkb | that chagne ahppens after the existing code uses it | 19:09 |
| jlk | re draft PRs, https://github.com/sigmavirus24/github3.py/issues/926 | 19:10 |
| jlk | gottta get it in github3.py before seeing it in zuul | 19:10 |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul master: Cache github PR shas https://review.openstack.org/636764 | 19:13 |
| clarkb | tobiash: corvus ^ something like that maybe | 19:13 |
| *** jamesmcarthur has joined #zuul | 19:18 | |
| *** jamesmcarthur has quit IRC | 19:21 | |
| tobiash | clarkb: despite some naming confusion between project_name and project in getPullBySha lgtm | 19:22 |
| clarkb | tobiash: ya that is what I was reasoning about before in IRC. we rename project to mean the project suffix in that method | 19:23 |
| tobiash | clarkb: I think we should clean that up (maybe as a followup) | 19:23 |
| tobiash | the function argument should be project_name as functions that take 'project' mostly take a project object in zuul | 19:24 |
| clarkb | that would read a lot better I think | 19:24 |
| tobiash | so changing this will make it clear and avoid confusion about that | 19:25 |
| *** jamesmcarthur has joined #zuul | 19:30 | |
| corvus | clarkb, tobiash: +3 | 19:37 |
| *** jamesmcarthur has quit IRC | 19:41 | |
| *** jamesmcarthur has joined #zuul | 19:41 | |
| *** rlandy is now known as rlandy|afk | 19:47 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-preview master: Update gitreview file with correct project name https://review.openstack.org/636791 | 19:54 |
| corvus | i'm going to self-approve that as being trivial and necessary | 19:55 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-preview master: Update gitreview file with correct project name https://review.openstack.org/636791 | 19:55 |
| openstackgerrit | Merged openstack-infra/zuul-preview master: Update gitreview file with correct project name https://review.openstack.org/636791 | 19:56 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-preview master: WIP: test docker registry https://review.openstack.org/637037 | 20:02 |
| *** jamesmcarthur has quit IRC | 20:19 | |
| *** jamesmcarthur has joined #zuul | 20:21 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-preview master: WIP: test docker registry https://review.openstack.org/637037 | 20:23 |
| *** jamesmcarthur has quit IRC | 20:24 | |
| corvus | infra-root: \o/ http://38.108.68.64/ is running 1.6.3 -- our automatic upgrade driven by git and ansible worked! | 20:24 |
| corvus | that was changes https://review.openstack.org/629942 and https://review.openstack.org/635016 | 20:25 |
| *** jamesmcarthur has joined #zuul | 20:25 | |
| corvus | now, on to 1.7.1 | 20:26 |
| fungi | ooh! | 20:27 |
| clarkb | yay | 20:33 |
| corvus | oh, sorry that was the wrong channel | 20:47 |
| corvus | tobiash, fungi, quiquell|off: i'll note that by using !unsafe, we have made the ansible inventory file slightly more difficult to load in some yaml parsers. if we find that problematic we could consider base64 encoding the commit message. | 20:52 |
| corvus | (ansible has built-in b64decode jinja filters, so that's still easy to work with) | 20:53 |
| fungi | that seems like a reasonable alternative | 20:53 |
| tobiash | corvus: do you have a use case in mind where this is problematic? | 20:54 |
| corvus | tobiash: anything that involves post-processing inventory files from jobs (maybe someone wants to analyze runs). also perhaps javascript on the build page. | 20:56 |
| corvus | nothing concrete | 20:57 |
| tobiash | hrm, I think we should decide this soon as the commit message is pretty new atm | 20:58 |
| tobiash | Changing this later could break many jobs | 20:58 |
| corvus | yep | 20:59 |
| tobiash | So better safe than sorry and change to base64? | 20:59 |
| corvus | i'd like more input | 20:59 |
| tobiash | Sure | 21:00 |
| corvus | maybe worth asking on the mailing list? | 21:00 |
| tobiash | I think so | 21:00 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Enable logging on registry/push/pull jobs https://review.openstack.org/637049 | 21:01 |
| corvus | clarkb, tobiash: ^ those roles, unsurprisingly, didn't work. i'd like to enable logging there, even though it may expose the credential for our intermediate registry. i'll just change the password after we figure out the problem. | 21:02 |
| clarkb | ok | 21:04 |
| *** jamesmcarthur has quit IRC | 21:19 | |
| *** jamesmcarthur has joined #zuul | 21:27 | |
| *** calebb has joined #zuul | 21:27 | |
| *** jamesmcarthur has quit IRC | 21:31 | |
| pabelanger | Hmm | 21:33 |
| pabelanger | Exception: Project key directory /var/lib/zuul/keys must be mode 0700; current mode is 770 | 21:33 |
| pabelanger | any reason we cannot support 0770? | 21:33 |
| -openstackstatus- NOTICE: Jobs are failing due to ssh host key mismatches caused by duplicate IPs in a test cloud region. We are disabling the region and will let you know when jobs can be rechecked. | 21:33 | |
| clarkb | pabelanger: those keys are pretty important | 21:33 |
| pabelanger | clarkb: yah, agreed. I have a backup user, that is part of zuul group, which I was hoping could read them. | 21:34 |
| pabelanger | currently /var/lib/zuul is zuul:zuul | 21:34 |
| pabelanger | and backup user is backup:zuul | 21:34 |
| pabelanger | but, zuul won't start now | 21:35 |
| fungi | pabelanger: i think 0770 ought to be fine with appropriate precautions. we could also consider setting a safe umask for the keys | 21:36 |
| fungi | also you likely need setgid for that scenario? | 21:37 |
| clarkb | ya and maybe documentation on the dangers | 21:37 |
| fungi | but main concern is if you chgrp the files to, say, www-data and that tree is somehow accessible from a misconfigured apache docroot... that sort of thing | 21:38 |
| pabelanger | yah | 21:39 |
| fungi | also possible that paramiko may freak out on us even if zuul decides its okay? | 21:42 |
| *** jamesmcarthur has joined #zuul | 22:00 | |
| *** jamesmcarthur has quit IRC | 22:08 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Enable logging on registry/push/pull jobs https://review.openstack.org/637049 | 22:08 |
| *** rlandy|afk is now known as rlandy | 22:09 | |
| *** jamesmcarthur has joined #zuul | 22:14 | |
| -openstackstatus- NOTICE: The test cloud region using duplicate IPs has been removed from nodepool. Jobs can be rechecked now. | 22:15 | |
| *** jamesmcarthur has quit IRC | 22:18 | |
| *** jamesmcarthur has joined #zuul | 22:18 | |
| openstackgerrit | Merged openstack-infra/zuul master: Cache github PR shas https://review.openstack.org/636764 | 22:42 |
| fungi | i have a few spare minutes before i have to jump on a conference call. what was the current state of the github pr cache changes? happy to review some | 22:46 |
| clarkb | fungi: line above yours, change just merged | 22:47 |
| clarkb | fungi: now up to us to install it and restart scheduler with it | 22:47 |
| fungi | oh, hah | 22:47 |
| fungi | i should have looked closer. today has been a blur | 22:47 |
| *** jamesmcarthur has quit IRC | 22:51 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Fix pull-from-intermediate-registry artifacts error https://review.openstack.org/637072 | 23:01 |
| *** jamesmcarthur has joined #zuul | 23:01 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Fix undefined attrs in registry push/pull roles https://review.openstack.org/637072 | 23:03 |
| corvus | clarkb, fungi: ^ can you +3 that pls? | 23:04 |
| *** jamesmcarthur has quit IRC | 23:05 | |
| *** jamesmcarthur has joined #zuul | 23:06 | |
| *** cognifloyd has joined #zuul | 23:10 | |
| cognifloyd | Has anyone worked on a Bitbucket driver for zuul? I couldn't find anything in storyboard about it. | 23:10 |
| cognifloyd | Generic git can trigger, but it doesn't have the reporter + requirements integrations that github or gerrit has. | 23:11 |
| openstackgerrit | Jan Kundrát proposed openstack-infra/nodepool master: Implement a Runc driver https://review.openstack.org/535556 | 23:12 |
| clarkb | cognifloyd: there has been interest expressed, but I don't think anyone has gotten as far as code | 23:14 |
| *** rlandy is now known as rlandy|bbl | 23:44 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!