| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: DNM: add ansible_network_os to vars https://review.openstack.org/652424 | 00:25 |
|---|---|---|
| *** jamesmcarthur has joined #zuul | 02:19 | |
| *** jamesmcarthur has quit IRC | 02:26 | |
| *** jamesmcarthur has joined #zuul | 02:26 | |
| *** jamesmcarthur has quit IRC | 02:32 | |
| *** jamesmcarthur has joined #zuul | 02:46 | |
| *** jamesmcarthur has quit IRC | 02:49 | |
| *** jamesmcarthur has joined #zuul | 02:50 | |
| *** jamesmcarthur has quit IRC | 02:54 | |
| *** bhavikdbavishi has joined #zuul | 03:25 | |
| *** bhavikdbavishi1 has joined #zuul | 03:28 | |
| *** bhavikdbavishi has quit IRC | 03:30 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 03:30 | |
| *** jamesmcarthur has joined #zuul | 03:31 | |
| *** raukadah is now known as chandankumar | 03:53 | |
| *** bhavikdbavishi1 has joined #zuul | 04:48 | |
| *** bhavikdbavishi has quit IRC | 04:49 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 04:49 | |
| *** bhavikdbavishi1 has joined #zuul | 04:53 | |
| *** bhavikdbavishi has quit IRC | 04:54 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 04:54 | |
| *** chandankumar has left #zuul | 05:21 | |
| *** chandankumar has joined #zuul | 05:21 | |
| *** quiquell|off is now known as quiquell|rover | 05:46 | |
| *** bjackman has joined #zuul | 05:55 | |
| *** pcaruana has joined #zuul | 06:19 | |
| *** toabctl has joined #zuul | 06:32 | |
| *** gtema has joined #zuul | 07:02 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Add --check-config option to zuul scheduler https://review.openstack.org/542160 | 07:18 |
| *** hashar has joined #zuul | 08:12 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Add support for smart reconfigurations https://review.openstack.org/652114 | 08:12 |
| *** shanemcd has quit IRC | 08:22 | |
| *** shanemcd has joined #zuul | 08:23 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Add --check-config option to zuul scheduler https://review.openstack.org/542160 | 08:26 |
| *** jangutter has joined #zuul | 08:39 | |
| *** electrofelix has joined #zuul | 09:05 | |
| *** gtema has quit IRC | 09:46 | |
| *** zbr has joined #zuul | 09:47 | |
| *** zbr__ has quit IRC | 09:50 | |
| *** panda has joined #zuul | 09:51 | |
| *** bhavikdbavishi has quit IRC | 09:59 | |
| bjackman | How can I add known host keys to Zuul for a Git connection? | 10:10 |
| bjackman | I can always just manually add them in the Zuul scheduler container but I'd rather do it in a way that persists.. | 10:12 |
| *** gtema has joined #zuul | 10:30 | |
| electrofelix | bjackman: seems like you'd want to mount in from a volume? | 10:50 |
| *** quiquell|rover is now known as quique|rover|eat | 11:06 | |
| *** bhavikdbavishi has joined #zuul | 11:10 | |
| *** mhu|off has quit IRC | 11:12 | |
| *** weshay_pto has quit IRC | 11:12 | |
| *** mhu has joined #zuul | 11:13 | |
| *** weshay_pto has joined #zuul | 11:13 | |
| *** panda is now known as panda|lunch | 11:23 | |
| *** gtema has quit IRC | 11:35 | |
| *** quique|rover|eat is now known as quiquell|rover | 11:49 | |
| *** rlandy has joined #zuul | 12:03 | |
| *** rlandy is now known as rlandy|ruck | 12:03 | |
| *** panda|lunch is now known as panda | 12:08 | |
| *** jamesmcarthur has joined #zuul | 12:24 | |
| *** rlandy|ruck is now known as rlandy|ruck|mtg | 12:33 | |
| *** gtema has joined #zuul | 12:34 | |
| *** jamesmcarthur has quit IRC | 12:35 | |
| bjackman | electrofelix, yeah I suppose that's what I'll need to do | 12:37 |
| bjackman | I was hoping there was some Zuul config I was missing that could do it fo rme | 12:37 |
| bjackman | I was hoping there was some Zuul config I was missing that could do it for me | 12:37 |
| *** jamesmcarthur has joined #zuul | 12:46 | |
| electrofelix | bjackman: we have added some stuff to containers to retrieve the keys on start up, it's not really best practice though | 12:46 |
| *** rfolco has joined #zuul | 12:47 | |
| bjackman | electrofelix, right | 12:48 |
| electrofelix | by we I mean another operator of zuul ;-) | 12:48 |
| *** quiquell|rover has quit IRC | 12:55 | |
| *** quiquell has joined #zuul | 12:56 | |
| *** bhavikdbavishi has quit IRC | 13:03 | |
| *** rlandy|ruck|mtg is now known as rlandy|ruck | 13:05 | |
| *** bjackman has quit IRC | 13:05 | |
| *** bjackman has joined #zuul | 13:06 | |
| *** jamesmcarthur has quit IRC | 13:12 | |
| bjackman | Caught my nodepool instance in its apparent deadlock again - this time I dumped the request-list and node-list from the nodepool webapp: https://paste.gnome.org/pyzdqk4gq | 13:13 |
| bjackman | It has a pending request and 4 "ready" nodes but for some reason isn't servicing that pending request | 13:13 |
| bjackman | Any ideas why that might be? | 13:14 |
| sshnaidm | how can I trigger a job in zuul if new container is uploaded to docker.io for example? | 13:32 |
| *** jamesmcarthur has joined #zuul | 13:35 | |
| *** jamesmcarthur_ has joined #zuul | 13:46 | |
| *** jamesmcarthur has quit IRC | 13:49 | |
| Shrews | bjackman: the nodepool logs should say what's going on | 14:04 |
| pabelanger | sshnaidm: you can add a job dependency after the upload to docker job runs | 14:05 |
| *** rlandy|ruck is now known as rlandy|ruck|mtg | 14:05 | |
| pabelanger | or are you asking about triggering another job in a different pipeline? | 14:05 |
| *** gtema has quit IRC | 14:16 | |
| *** gtema has joined #zuul | 14:18 | |
| *** gtema has quit IRC | 14:19 | |
| *** hashar has quit IRC | 14:21 | |
| *** bjackman has quit IRC | 14:37 | |
| mordred | pabelanger: I believe sshnaidm is looking for "I want a zuul trigger plugin that triggers on someone uploading a new image to dockerhub" | 14:40 |
| sshnaidm | mordred, yeah, exactly | 14:40 |
| mordred | pabelanger: so that one could, for instance, do a rebuild of zuul images if the underlying python base image got an update | 14:41 |
| sshnaidm | pabelanger, seems like I need two jobs, one will monitor and second will run checks | 14:41 |
| mordred | sshnaidm: at the moment there isn't a way to do this other than using a periodic pipeline - or something that watches dockerhub and submits patches to a git repo | 14:42 |
| pabelanger | was just going to say that | 14:42 |
| sshnaidm | mordred, I see | 14:42 |
| mordred | it's a type of use case that's come up a few times before, but so far nobody has been able to fully describe what such a trigger would look like in the wider context of the system - so I wouldn't be surprised if we grew that ability, but I would be surprised if we grew that capability in the short-term, simply because there are some deeper design questions that come in to play | 14:43 |
| *** quiquell is now known as quiquell|off | 14:46 | |
| *** bjackman has joined #zuul | 14:48 | |
| bjackman | Shrews, was about to say that the nodepool logs seem to suggest the request was satsified - but when I went to grab something to paste I noticed that earlier on in the logs (I guess when the request first came in) is a kazoo.exceptions.ConnectionLoss | 14:54 |
| bjackman | Will keep digging | 14:54 |
| bjackman | https://paste.gnome.org/p2tpzkkmz | 14:55 |
| *** rlandy|ruck|mtg is now known as rlandy|ruck | 14:56 | |
| Shrews | bjackman: that's ok, we expect the zk connection to become temporarily unavailable at times because "networks" and try to recover correctly, which it seemed it did. I see that the request was actually satisfied ("Fulfilled node request 200-0000002869") and the handler removed, so that request should NOT be pending anymore at that point. That's VERY confusing to me. | 15:02 |
| Shrews | bjackman: if you give the --detail option to 'nodepool list', does it show that node 0000002811 is assigned to that request? | 15:03 |
| Shrews | bjackman: the way it works is, a node is assigned to a request, but it's up to the zuul-scheduler to then change that node's status from READY to IN-USE | 15:04 |
| Shrews | bjackman: so you may need to dig into zuul logs as well for that request | 15:04 |
| bjackman | Shrews, sorry I had to restart the launcher as this is all happening on prod | 15:05 |
| bjackman | So can't dig any further (zuul-scheduler logs will have wrapped by now, too) | 15:05 |
| Shrews | oh, that's unfortunate. well, now you know what else to look for next time :) | 15:05 |
| bjackman | But this happens pretty often so I will look in more detail next time | 15:05 |
| bjackman | Shrews, indeed, thanks | 15:05 |
| bjackman | Has been happening for a while but each time it happens, I haven't had time to really dig into it | 15:06 |
| bjackman | Only found out about the nodepool web UI today | 15:06 |
| Shrews | bjackman: could very well be bugs in the static driver, but hard to tell from just what you pasted. i thing tobiash uses that driver quite often. not sure he sees such problems | 15:06 |
| bjackman | Shrews, I don't really understand why there are deleted nodes, either | 15:07 |
| bjackman | Is that expected? | 15:07 |
| Shrews | bjackman: you're running the latest versions of nodepool/zuul? | 15:07 |
| bjackman | No zuul is 3.6.2.dev4 9abc6403 | 15:07 |
| Shrews | bjackman: those should be transient. i wouldn't expect those to live very long | 15:07 |
| bjackman | Let me see how to find Nodepool version.. | 15:07 |
| bjackman | Ah - it's just happened again. Will gather everything I can... | 15:07 |
| bjackman | Ah the user uploaded a new patchset so the stalled change got dequeued before I could finish | 15:17 |
| bjackman | Shrews, anything I can do to look into the persistent "deleted" nodes in the meantime?> | 15:18 |
| bjackman | Shrews, anything I can do to look into the persistent "deleted" nodes in the meantime? | 15:18 |
| Shrews | bjackman: pulling the log info for any of those deleted nodes would be very helpful, but looks like they've been around a long time, so you may not have that anymore | 15:21 |
| Shrews | i think i can see a code path where losing the zookeeper connection in the middle of a delete could leave that state hanging around, but would be nice to have logs to confirm that | 15:22 |
| Shrews | may have to come up with a test for that... | 15:23 |
| tobiash | I think also a zk session timeout option could help here (we have that already in zuul) | 15:24 |
| Shrews | tobiash: i think getting an exception from zk.deleteRawNode() that we don't handle would leave a znode in the DELETED state, and that's not one of the standard cleanup states we look for in DeletedNodeWorker._cleanupNodes(). I think I can easily create a test to prove that. | 15:26 |
| tobiash | Shrews: cool | 15:27 |
| Shrews | you'd have to be extremely unlucky with timing for that to happen, but looks like bjackman has seen that more than once :( | 15:29 |
| Shrews | (if that is the problem) | 15:29 |
| tobiash | bjackman: what disks/san are you running zk on? | 15:30 |
| tobiash | bjackman: initially I ran it on ceph which was very unstable | 15:30 |
| bjackman | tobiash, everything is just running via docker-compose on the same physical machine in our office | 15:32 |
| Shrews | ooh | 15:32 |
| Shrews | that's, uh, not recommended | 15:32 |
| Shrews | for production | 15:32 |
| bjackman | Shrews, unfortunately in my industry, top brass tends to reject "cloud solutions" | 15:32 |
| Shrews | you're going to get a lot of zookeeper connection issues | 15:32 |
| bjackman | Shrews, why would that cause connection issues? | 15:33 |
| tobiash | bjackman: the problem is IO, if zookeeper is stalled for more than 10s by default, the clients loose their session | 15:34 |
| Shrews | zookeeper is very i/o intensive, and you'll see it stall a lot. | 15:34 |
| tobiash | and with the session the locks etc | 15:34 |
| Shrews | yeah, that | 15:34 |
| bjackman | Ah i see | 15:34 |
| bjackman | I don't think there's much scope for changing the situation there | 15:35 |
| tobiash | bjackman: you could think about putting it on a tmpfs and regularly copy the snapshots back to a disk by a different process | 15:35 |
| bjackman | We have zillions of huge servers but they all run Red Had Linux 1995 | 15:35 |
| bjackman | For HW dev stuff | 15:35 |
| bjackman | Only a couple of things modern enough for containers | 15:36 |
| Shrews | given that architecture, i'm not surprised you're hitting this delete issue so much then. i can try and make that better for that case, but be aware you may hit more such odd cases we haven't quite ironed out yet | 15:38 |
| Shrews | (not that i'm aware of any, but they're probably there) | 15:38 |
| bjackman | Shrews, tobiash so do you think the issue would go away if zk had its own host? | 15:44 |
| tobiash | bjackman: possibly (if on ssd) | 15:44 |
| pabelanger | +1 for moving zk to own host | 15:44 |
| bjackman | It seems like this issue is eventually going to come back at some scale or another though no? | 15:46 |
| bjackman | In my case the max number of parallel jobs in the entire system is 8 | 15:46 |
| bjackman | The machine has 32 cores (granted, only one storage volume) | 15:48 |
| pabelanger | Running mergers / executors also need a lot of disk IO | 15:49 |
| pabelanger | so, after splitting off zookeeper, I could see you also wanting to split them off | 15:49 |
| bjackman | Yeah I can see that that's going to pound the disks | 15:49 |
| bjackman | But there must be a way to make the impact of the IO-thrashing that the system is slow, instead of that it gets stuck | 15:51 |
| pabelanger | given that the executor runs a merger, and you are all-in-one, you might be able to stop zuul-merger process and just rely on zuul-executor. that could save a little io | 15:51 |
| bjackman | pabelanger, I have a feeling I already have that situation - my setup is based on the Getting Started Guide in the Zuul repo | 15:52 |
| bjackman | I don't have a separate service for the merger | 15:52 |
| Shrews | bjackman: Yeah, obviously getting stuck is not what we want. We want to fix such bugs causing that. But running everything in such a constrained environment, you're more likely to hit such bugs... which would be helpful for finding them, but not so convenient for you, I'm afraid. :( | 16:11 |
| Shrews | i'm curious as to which piece is getting "stuck" for you, though. feels like more zuul-side based on the logs you presented earlier | 16:12 |
| Shrews | the i/o contention could certainly make things "seem" stuck | 16:13 |
| *** hashar has joined #zuul | 16:13 | |
| SpamapS | Don't forget that Zookeeper is *extremely* sensitive to IO load. | 16:14 |
| SpamapS | You won't see anything in top, but Zookeeper will feel almost like it's dead if it goes into IO wait for even 5% of the time. | 16:14 |
| Shrews | SpamapS: yep, first thing we noted. he's certainly seeing that | 16:14 |
| SpamapS | k, I'm reading backwards | 16:16 |
| SpamapS | if it's just a toy, tmpfs is a real win there. | 16:16 |
| SpamapS | mordred:also regarding your "I want to trigger when something happens elsewhere"... a webhook trigger driver would be amazing. | 16:17 |
| clarkb | SpamapS: fwiw the disk on the registry filled up again after I freed ~700MB | 16:18 |
| SpamapS | clarkb: :( | 16:18 |
| SpamapS | clarkb:what's going on, too many transient images? | 16:18 |
| clarkb | so we'll have to figure that out today. Probably by nuking the registry contents and starting over. Then make plans for garbage collection and tag deletion periodically | 16:18 |
| clarkb | SpamapS: ya its filling the 40GB device with our zuul and nodepool and gitea builds | 16:19 |
| clarkb | we aren't running garbage collection which won't do much anyway since you have to untag things for that to work, but running by hand hits an error | 16:19 |
| clarkb | I can't even start a container now to try and run it by hand though because that requires making changes on / | 16:19 |
| SpamapS | Ah yeah, with our AWS ECR repos we set some policies to delete tags based on patterns. Works well. we keep 25 change_XXX's, 25 newrev_XXX's, all pipeline_XXX's, and never delete anything until it is at least 7 days old. | 16:20 |
| clarkb | SpamapS: ya I think we'll want similar. But also we need to get the garbage collector working which I think means destroying the registry (unless someone wants to go one by one identifying errosr and manually deleting the bad sha blobs) | 16:21 |
| SpamapS | clarkb:why not build a new registry? | 16:26 |
| * SpamapS prefers to cloud hard | 16:26 | |
| clarkb | its the same difference really. Do you rm -rf /var/lib/registry or openstack server delete | 16:26 |
| SpamapS | but anyway, just tell me when we can recheck patches so they can land. ;) | 16:26 |
| SpamapS | clarkb:me personally, I always server delete. :) | 16:26 |
| clarkb | the biggest downside to ^ is you don't learn as much about the failure modes | 16:26 |
| SpamapS | actually I server stop, server create.. get working.. then delete. | 16:27 |
| SpamapS | You don't want to learn about failure modes, you want to automate yourself into a place where you avoid them. :) | 16:27 |
| clarkb | you want both | 16:27 |
| SpamapS | meh.. not realy. | 16:27 |
| SpamapS | you might :) | 16:27 |
| clarkb | you can't avoid the failure modes if you don't understand them | 16:28 |
| SpamapS | And I always archive root volumes for the RCA | 16:28 |
| clarkb | sure you can rebuild and start over but then you've got a 10 minute outage | 16:28 |
| SpamapS | Don't you have an outage now? | 16:28 |
| clarkb | yes, but if we can udnerstand the failuer and prevent it then no more outages | 16:28 |
| clarkb | is my point | 16:28 |
| clarkb | for example it may be that the disk is super undersized | 16:28 |
| SpamapS | I think my point is that I prefer to decouple the learning from the recovering. :) | 16:28 |
| clarkb | that is something that you need to feed back into the rebuild | 16:29 |
| clarkb | oh sure | 16:29 |
| SpamapS | But also I think if you build a new one you might see it fill up fast again.. seems like you have more than a GC problem.. you have a dangling reference problem. | 16:29 |
| SpamapS | clarkb:anyway, don't let me gum up your process with opinions. If I can help, let me know. And if I can recheck, also let me know. :) | 16:30 |
| clarkb | I took a quick look on the weekend my apologoes if I didn't fix it quickly enough for your preferences :P | 16:30 |
| clarkb | the other issue with GC is you cannot run it whne the registry is online | 16:31 |
| clarkb | which severely limits our ability to GC I think | 16:31 |
| SpamapS | wow, that seems.. wow. | 16:32 |
| SpamapS | good job docker | 16:32 |
| *** bhavikdbavishi has joined #zuul | 16:32 | |
| SpamapS | My preference is for everything ot be 5 nines and in the minutes of downtime a year I prefer to be fed grapes. | 16:33 |
| SpamapS | ;-) | 16:33 |
| SpamapS | sorry if I push.. just peeking in. | 16:33 |
| *** bhavikdbavishi1 has joined #zuul | 16:35 | |
| clarkb | ah ok re GC happening when server is offline it doesn't always require that, only when you don't use the api to mark tags for deletion | 16:36 |
| clarkb | so we should use the api for that I guess | 16:36 |
| clarkb | (odd that people wouldn't use the api) | 16:36 |
| *** bhavikdbavishi has quit IRC | 16:36 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 16:36 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: WIP - Pagure driver - https://pagure.io/pagure/ https://review.openstack.org/604404 | 16:41 |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul master: Fix tox.ini cover target install command https://review.openstack.org/652727 | 16:45 |
| corvus | SpamapS: there's a WIP url trigger change | 16:50 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Fix loss of ZK conn during node delete https://review.openstack.org/652729 | 16:51 |
| Shrews | tobiash: bjackman: I think the test in that ^^ demonstrates the orphaned DELETED node problem | 16:51 |
| SpamapS | corvus:cool. I have a need to run some jobs whenever an EC2 autoscaling group changes membership.. and was just thinking "hm, how can I do this in Zuul?" | 16:52 |
| tobiash | zuul-maint: I'd love a second review on https://review.openstack.org/634597 and parent which should make job canceling more robust and handle failed paused jobs better | 17:06 |
| *** gtema has joined #zuul | 17:08 | |
| *** gtema has quit IRC | 17:10 | |
| mordred | tobiash: done | 17:22 |
| tobiash | mordred: \o/ | 17:22 |
| clarkb | tobiash: mordred note that cannot merge right now due to the broken docker registry (I think) | 17:23 |
| clarkb | I will hopefully have it up and running shortly | 17:23 |
| tobiash | that will be an interesting race, gate vs clarkb :) | 17:24 |
| clarkb | my money is on the gate. I'm taking a snapshot really quickly and those never seem to be a predictable cost | 17:24 |
| *** hashar has quit IRC | 17:37 | |
| electrofelix | experimenting with zuulv3 where's the best place to configure common env variables such as proxies that need to be set for jobs? As we might have some hardware behind a proxy and some in AWS is it better done on a per node basis rather than job basis? | 18:03 |
| *** jamesmcarthur_ has quit IRC | 18:04 | |
| corvus | electrofelix: to set an env var, you'll have to have ansible do that -- so either your task, role, or play will need to set the env variable -- like this: https://docs.ansible.com/ansible/latest/user_guide/playbooks_environment.html | 18:16 |
| corvus | electrofelix: to deal with region-specific information (like a specific proxy for this region) | 18:17 |
| corvus | electrofelix: you can use information that nodepool provides to zuul to construct the contents of the variable | 18:17 |
| corvus | electrofelix: let me dig up links for how we set our region-local mirrors in opendev's zuul | 18:18 |
| corvus | electrofelix: this is the contents of our site-variables.yaml file: https://opendev.org/openstack-infra/project-config/src/branch/master/zuul/site-variables.yaml | 18:23 |
| corvus | electrofelix: see how the zuul_site_mirror_fqdn variable is constructed | 18:23 |
| corvus | electrofelix: the executors include that file in every job that's run (via https://zuul-ci.org/docs/zuul/admin/components.html#attr-executor.variables ) | 18:24 |
| corvus | electrofelix: so all those variables are always automatically set for ansible | 18:24 |
| corvus | electrofelix: you could do something that, and then use that to set the http_proxy environment variables in ansible | 18:25 |
| corvus | electrofelix: or you can use it in roles like this (which we use to configure mirrors/proxies/etc): https://zuul-ci.org/docs/zuul-jobs/general-roles.html?highlight=configure%20mirrors#role-configure-mirrors | 18:26 |
| SpamapS | There's also site variables for shared values. | 18:29 |
| *** bhavikdbavishi has quit IRC | 19:01 | |
| *** jamesmcarthur has joined #zuul | 19:01 | |
| clarkb | SpamapS: registry should be working again now | 19:16 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Support fail-fast in project pipelines https://review.openstack.org/652764 | 19:32 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Support fail-fast in project pipelines https://review.openstack.org/652764 | 19:39 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Support fail-fast in project pipelines https://review.openstack.org/652764 | 19:53 |
| clarkb | no responses to either my bug or PR against docker for the ipv6 issue | 19:59 |
| clarkb | :/ | 19:59 |
| fungi | anybody else have trouble with react-scripts build? i can't figure out how to get it to provide more detail: http://paste.openstack.org/show/749324/ | 20:00 |
| fungi | is the "EEXIST: file already exists" error coming from it, or something else? | 20:00 |
| fungi | or are those fsevents compatibility checks relevant? | 20:01 |
| fungi | i'm gonna take a break to find some food, but will continue poking at it when i return | 20:01 |
| clarkb | fungi: did you start with a clean repo (eg git clean -xfd)? | 20:02 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Fix for orphaned DELETED nodes https://review.openstack.org/652729 | 20:02 |
| fungi | yep, and this is master branch tip of zuul with no changes | 20:02 |
| fungi | oh! | 20:03 |
| fungi | `git clean -dfx` is skippnig web/build | 20:03 |
| fungi | it says "Removing web/node_modules/" | 20:03 |
| fungi | but it leaves web/build intact | 20:03 |
| fungi | why? | 20:03 |
| fungi | aha, web/build is a symlink to ../zuul/web/static/ | 20:04 |
| fungi | maybe git clean won't remove links? | 20:04 |
| fungi | oh, nope, that web/build symlink is actually tracked in git, that's why it doesn't remove it | 20:05 |
| fungi | so i'm guessing the EEXIST in the output isn't relevant | 20:06 |
| fungi | since that path should exist | 20:06 |
| corvus | mordred: ^ | 20:06 |
| fungi | anyway, back shortly | 20:06 |
| mordred | so many words | 20:07 |
| fungi | mordred: the paste is me trying to run `tox -e py37` after installing everything bindep wants | 20:08 |
| * fungi is *really* heading out now, before christine gnaws my leg off | 20:08 | |
| mordred | *weird* - looking | 20:08 |
| clarkb | is it possible nodeenv doesn't build node if you have node installed locally? and if so do you have node installed locally? | 20:10 |
| mordred | I mean - it shouldn't matter too much - that paste shows node v8 being installed | 20:11 |
| mordred | fungi: incidentally - the fs-events thing is sadly unavoidable but also is always there. it's a warning from a dependency | 20:12 |
| SpamapS | clarkb:thank you | 20:18 |
| SpamapS | mordred:it's not *always* there.. just on all production-ready platforms (Linux) ;) | 20:19 |
| mordred | SpamapS: like I said - it's always there :) | 20:19 |
| SpamapS | I'm always there too. | 20:19 |
| * SpamapS creeps | 20:19 | |
| * mordred cannot hide from the watchful eyes of SpamapS | 20:19 | |
| clarkb | fwiw I have py37 locally too and running tox -epy37 after a git clean -xfd worked. WHich makes me wonder about platform stuff | 20:21 |
| clarkb | I have nodejs v8.11.1 installed in the virtualenv too | 20:22 |
| *** jamesmcarthur has quit IRC | 20:37 | |
| *** pcaruana has quit IRC | 20:38 | |
| mordred | clarkb: I just ran the same stuff with py37 - but i have packages installed so I didn't get things instlaled into nodeenv - and it worked | 20:38 |
| *** jamesmcarthur has joined #zuul | 20:39 | |
| clarkb | is it possible fungi's link is broken? | 20:41 |
| *** sshnaidm has quit IRC | 20:41 | |
| clarkb | I know that I've run into weirdness with the .keep file locally | 20:41 |
| clarkb | (git keeps thinking I've deleted it for some reason | 20:41 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool master: Gather host keys for connection-type network_cli https://review.openstack.org/652778 | 20:42 |
| pabelanger | would love a review of ^ | 20:43 |
| pabelanger | helps gather SSH host keys when using conneciton-type network_cli | 20:43 |
| pabelanger | Shrews: ^if you don't mind adding to your review queue | 20:43 |
| pabelanger | so far, testing with vyos and zuul is working | 20:49 |
| *** jamesmcarthur has quit IRC | 20:56 | |
| *** sshnaidm has joined #zuul | 20:57 | |
| fungi | okay, back and seeing what we've got | 21:00 |
| fungi | so... i wonder whether `react-scripts build` has a way to... like... actually say *why* it "failed to compile" | 21:02 |
| fungi | i tried adding --verbose to its invocation in web/package.json, to no avail | 21:03 |
| fungi | even though it's mentioned in the output, https://yarnpkg.com/en/docs/cli/run is surprisingly unhelpful here | 21:03 |
| clarkb | fungi: adding --verbose to the command in the json file would've been what I tried. The docs don't seem to document flags | 21:08 |
| clarkb | might be able to find the source and check /me looks | 21:08 |
| clarkb | https://github.com/facebook/create-react-app/blob/master/packages/react-scripts/bin/react-scripts.js | 21:09 |
| fungi | it's likely just my fault for trying this on debian | 21:09 |
| clarkb | https://github.com/facebook/create-react-app/blob/master/packages/react-scripts/scripts/build.js | 21:09 |
| fungi | woo, yeah, not a lot of options there | 21:10 |
| clarkb | fungi: https://github.com/facebook/create-react-app/blob/master/packages/react-scripts/scripts/build.js#L57-L60 | 21:10 |
| clarkb | it could be that it is failing there and that is where your mkdir error comes from? | 21:11 |
| fungi | that function *is* remarkably devoid of user feedback, so maybe | 21:12 |
| pabelanger | tobiash: 2.8.0b1 of ansible was just cut, I'm going to poke into your 2.8 patch for zuul and see why logging is failing | 21:15 |
| clarkb | fungi https://github.com/facebook/create-react-app/blob/master/packages/react-scripts/scripts/build.js#L126 | 21:15 |
| clarkb | fungi: I read that as it says "failed then the error youv'e got there about mkdir is in fact the error that caused it to fail" | 21:15 |
| openstackgerrit | Merged openstack-infra/nodepool master: Implement max-servers for AWS driver https://review.openstack.org/649474 | 21:17 |
| fungi | okay, which leads me to wonder why it's concerned about that directory existing when i run it, but not for others, given the symlink and directory it targets are both tracked in the git repo | 21:20 |
| *** jamesmcarthur has joined #zuul | 21:27 | |
| openstackgerrit | Merged openstack-infra/zuul master: encrypt: Fix SSL error when using file pubkey https://review.openstack.org/650589 | 21:29 |
| clarkb | also mkdir should be fine creating a dir that already exists | 21:34 |
| fungi | well, mkdir -p is | 21:38 |
| fungi | mkdir will exit nonzero | 21:38 |
| clarkb | could it be a permissions issue? | 21:38 |
| clarkb | where it doesn't see the dir so it tries to recreate it? | 21:38 |
| fungi | seems unlikely. everything in this repo is owner by my user, same one which is executing toc | 21:39 |
| fungi | tox | 21:39 |
| fungi | but now i've tried deleting that symlink and tox hangs indefinitely on develop-inst | 21:39 |
| fungi | looks like the leafmost process is node running web/node_modules/react-scripts/scripts/build.js | 21:40 |
| fungi | nevermind, not hung, just slow. now it's building an ansible wheel | 21:42 |
| fungi | or so it appears from the process list | 21:42 |
| clarkb | it should isntall all the ansible versions for testing | 21:42 |
| clarkb | so that is potentially good news | 21:42 |
| fungi | likely courtesy of today's ansible release | 21:42 |
| fungi | releaseS | 21:42 |
| fungi | well lookie there, it's running tests now | 21:46 |
| fungi | but only after i delete that symlink | 21:46 |
| fungi | leading me to wonder *why* nobody else is encountering this issue. it's a symlink shipped in the git repo, so it should be present for everyone | 21:46 |
| fungi | gotta be something about my machine | 21:47 |
| clarkb | yes the symlink is present for me | 21:47 |
| fungi | once this tox run eventually completes, i'll repeat with and without the symlink just to be sure that's the deciding factor | 21:50 |
| *** jamesmcarthur has quit IRC | 21:52 | |
| openstackgerrit | Merged openstack-infra/zuul master: Centralize job canceling https://review.openstack.org/640609 | 21:52 |
| *** jamesmcarthur has joined #zuul | 21:55 | |
| *** jamesmcarthur has joined #zuul | 22:00 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: upgrade react and react-scripts to ^2.0.0 https://review.openstack.org/631902 | 22:07 |
| clarkb | fungi: ^ heh I wonder if that is realted | 22:08 |
| fungi | familiar words | 22:09 |
| fungi | maybe tristanC is on my wavelength again | 22:09 |
| tristanC | fungi: that *might* help, assuming react-script>2 has more fix/improvement than the curent (old) version zuul is using | 22:11 |
| fungi | yeah, still no clue why on debian/unstable react-scripts build seems to care that the web/build symlink exists | 22:12 |
| openstackgerrit | Merged openstack-infra/zuul master: Reset dependent jobs when aborting paused job https://review.openstack.org/634597 | 22:22 |
| *** jamesmcarthur has quit IRC | 22:25 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul master: Fix dynamic loading of trusted layouts https://review.openstack.org/652787 | 22:26 |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul master: Config errors should not affect config-projects https://review.openstack.org/652788 | 22:26 |
| clarkb | pabelanger corvus mordred ^ fyi | 22:40 |
| corvus | +3 | 22:41 |
| *** mattw4 has joined #zuul | 22:41 | |
| tristanC | clarkb: shouldn't the storyboard story be public now? | 22:47 |
| clarkb | tristanC: probably | 22:47 |
| clarkb | sorry I'm juggling a release note change and an email right now. mabye corvus or fungi can make it public? | 22:48 |
| corvus | can do | 22:48 |
| corvus | done | 22:48 |
| corvus | tristanC: ^ | 22:48 |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul master: Add release note for broken trusted config loading fix https://review.openstack.org/652793 | 22:52 |
| clarkb | corvus: fungi tristanC mordred pabelanger ^ | 22:52 |
| clarkb | any idea if this will be a 3.8 or 3.7.2 release? | 22:54 |
| corvus | clarkb: i think it should be 3.8.0 due to the artifacts change | 22:57 |
| corvus | https://zuul-ci.org/docs/zuul/releasenotes.html#in-development | 22:57 |
| clarkb | thanks | 22:57 |
| corvus | clarkb: so you can link to https://zuul-ci.org/docs/zuul/releasenotes.html#relnotes-3-8-0 | 22:58 |
| pabelanger | catching up | 22:58 |
| clarkb | https://etherpad.openstack.org/p/tBpqrLtzsP is my draft email; | 22:59 |
| pabelanger | +3 on 652793 | 22:59 |
| corvus | clarkb: i think that covers it, other than striking "-ci" from "zuul-ci" :) | 23:00 |
| clarkb | done | 23:00 |
| pabelanger | clarkb: lgtm | 23:02 |
| clarkb | k I think we are just waiting for things to merge then | 23:02 |
| * clarkb finds some tea. it is quite cold today for some reason | 23:03 | |
| pabelanger | Hmm | 23:04 |
| pabelanger | we might be having an outage in rax | 23:04 |
| corvus | good timing | 23:04 |
| pabelanger | moving to #openstack-infra | 23:05 |
| corvus | Shrews, tobiash: based on http://cacti.openstack.org/cacti/graph.php?action=zoom&local_graph_id=63831&rra_id=4&view_type=&graph_start=1522317252&graph_end=1555370436 it looks like we may have introduced a memory leak in nodepool-launcher late last year; we should dig into that soon | 23:21 |
| corvus | my first thought is the zk tree caching stuff... | 23:22 |
| *** bjackman has quit IRC | 23:38 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!