| *** ianychoi has joined #zuul | 00:01 | |
| openstackgerrit | Merged openstack-infra/zuul master: Handle github exceptions better when trying to get a PR https://review.openstack.org/651609 | 00:09 |
|---|---|---|
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/nodepool master: Implement max-servers for AWS driver https://review.openstack.org/649474 | 01:12 |
| SpamapS | woooot.. testing quota management in AWS and it's working | 01:24 |
| pabelanger | yay | 01:25 |
| SpamapS | yeah, still pretty basic, but better than NODE_FAILURE bombs | 01:29 |
| SpamapS | next I kind of want to implement spot instances... | 01:29 |
| SpamapS | seems like the perfect match. | 01:29 |
| SpamapS | and I really think it would be interesting if one could implement a budget limiter instead of max-servers. | 01:29 |
| SpamapS | (Amazon has a pricing API now) | 01:30 |
| *** bhavikdbavishi has joined #zuul | 01:47 | |
| pabelanger | yah, that would be neat | 02:20 |
| *** rlandy|ruck|bbl is now known as rlandy|ruck | 02:42 | |
| *** rlandy|ruck has quit IRC | 02:49 | |
| *** irclogbot_0 has quit IRC | 03:01 | |
| *** irclogbot_1 has joined #zuul | 03:03 | |
| *** bhavikdbavishi has quit IRC | 03:03 | |
| *** raukadah is now known as chandankumar | 03:25 | |
| *** bhavikdbavishi has joined #zuul | 03:59 | |
| *** bjackman_ has joined #zuul | 04:26 | |
| *** quiquell has joined #zuul | 05:50 | |
| *** quiquell is now known as quiquell|rover | 05:50 | |
| *** pcaruana has joined #zuul | 06:41 | |
| *** quiquell|rover is now known as quique|rover|brb | 06:44 | |
| *** gtema has joined #zuul | 06:52 | |
| *** mnaser has quit IRC | 07:06 | |
| *** gundalow has quit IRC | 07:06 | |
| *** gundalow has joined #zuul | 07:06 | |
| *** mnaser has joined #zuul | 07:06 | |
| *** quique|rover|brb is now known as quiquell|rover | 07:08 | |
| openstackgerrit | Brendan proposed openstack-infra/zuul master: gerrit: Add support for 'oldValue' comment-added field https://review.openstack.org/649900 | 07:12 |
| *** jpena|off is now known as jpena | 07:47 | |
| *** mhu is now known as mhu|off | 07:50 | |
| *** electrofelix has joined #zuul | 08:39 | |
| *** bhavikdbavishi1 has joined #zuul | 09:07 | |
| *** bhavikdbavishi has quit IRC | 09:09 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 09:09 | |
| *** bhavikdbavishi has quit IRC | 09:43 | |
| *** bjackman__ has joined #zuul | 09:52 | |
| *** bjackman_ has quit IRC | 09:56 | |
| *** hashar has joined #zuul | 10:02 | |
| *** pcaruana has quit IRC | 10:31 | |
| *** bhavikdbavishi has joined #zuul | 11:05 | |
| *** jpena is now known as jpena|lunch | 11:27 | |
| *** pcaruana has joined #zuul | 11:36 | |
| *** gtema has quit IRC | 11:51 | |
| *** EmilienM is now known as EvilienM | 12:05 | |
| *** rlandy has joined #zuul | 12:16 | |
| *** rlandy is now known as rlandy|ruck | 12:16 | |
| *** jpena|lunch is now known as jpena | 12:28 | |
| zbr | hi! i am here and I confirm that Firefox/UK IPv6/MacOS is me. Now the question is what could have caused that issue? | 12:28 |
| pabelanger | Shrews: clarkb: ^from yesterday, thanks helping zbr | 12:28 |
| zbr | I am using the greasemonkey script from https://github.com/openstack/coats/blob/master/coats/openstack_gerrit_zuul_status.user.js to get stats. | 12:28 |
| zbr | but this script was not changed in like 6months, maybe a new bug? | 12:29 |
| mordred | zbr: weird. I'm looking at the script right now - it looks like it's supposed to only fetch once per 2000 ms | 12:35 |
| *** gtema has joined #zuul | 12:36 | |
| zbr | clearly it came from the browser. i do have lots of tabs but afaik, it does this only on active tab. | 12:36 |
| mordred | yeah. I wonder if maybe there was a browser update with a bug - or maybe the browser just went sideways | 12:37 |
| zbr | i guess this was the only time when this was observed, right? | 12:38 |
| *** jamesmcarthur has joined #zuul | 12:43 | |
| Shrews | I think FF updated on my Mac yesterday so that’s probably a good guess mordred | 12:44 |
| mordred | zbr: well - we just happened to be looking at logs trying to figure out why something else was slow | 12:44 |
| *** fdegir has quit IRC | 12:45 | |
| *** timburke has quit IRC | 12:46 | |
| *** fdegir has joined #zuul | 12:46 | |
| *** timburke has joined #zuul | 12:48 | |
| zbr | Shrews: I can confirm that around that time firefox did an upgrade on my computer. | 12:50 |
| *** bjackman__ has quit IRC | 12:56 | |
| *** altlogbot_0 has joined #zuul | 13:05 | |
| Shrews | mordred: doesn't look like that solved your zuul-preview issue though :( | 13:11 |
| mordred | Shrews: yeah. :( | 13:12 |
| *** bjackman__ has joined #zuul | 13:14 | |
| *** bjackman__ has quit IRC | 13:18 | |
| *** altlogbot_0 has quit IRC | 13:32 | |
| *** altlogbot_3 has joined #zuul | 13:34 | |
| *** quiquell|rover is now known as quiquell|off | 13:37 | |
| *** bhavikdbavishi has quit IRC | 13:38 | |
| *** altlogbot_3 has quit IRC | 13:38 | |
| *** altlogbot_2 has joined #zuul | 13:40 | |
| *** bjackman has joined #zuul | 13:57 | |
| *** altlogbot_2 has quit IRC | 14:00 | |
| *** bjackman has quit IRC | 14:05 | |
| *** jamesmcarthur has quit IRC | 14:16 | |
| *** jamesmcarthur has joined #zuul | 14:17 | |
| *** altlogbot_0 has joined #zuul | 14:27 | |
| *** altlogbot_0 has quit IRC | 14:29 | |
| *** altlogbot_3 has joined #zuul | 14:31 | |
| *** altlogbot_3 has quit IRC | 14:33 | |
| *** altlogbot_2 has joined #zuul | 14:49 | |
| *** altlogbot_2 has quit IRC | 14:51 | |
| *** altlogbot_3 has joined #zuul | 14:54 | |
| *** altlogbot_3 has quit IRC | 14:55 | |
| *** altlogbot_2 has joined #zuul | 15:00 | |
| dmsimard | mordred: (forking from #openstack-infra) one of the things I've been wondering is if it would be possible to have nodepool issue "rebuild" commands (in the openstack driver) instead of delete/create every time | 15:06 |
| dmsimard | I would like to think rebuild should be supported on all clouds but I'm not sure | 15:06 |
| mordred | dmsimard: we've had requests from cloud providers in the past for that - as well as suggestions from cloud providers that they'd prefer we didnt' do rebuild | 15:07 |
| dmsimard | curious if it would make a performance impact | 15:07 |
| mordred | dmsimard: the tricky part comes in the nodepool logic side - because now we have to be able to decide whether to rebuild a node when we're returning it | 15:07 |
| dmsimard | ah, I guess | 15:08 |
| mordred | since when we're _not_ at capacity, we clearly want to delete so that we're shrinking usage | 15:08 |
| mordred | which is to say- I think it could certainly be potentially interesting - but the investigation in to it might be ... hard :) | 15:08 |
| *** goncalo has left #zuul | 15:08 | |
| clarkb | ya maybe it is a only when paused behavior | 15:13 |
| clarkb | paused state implies provider needs to provision new nodes | 15:13 |
| clarkb | not sure how big an impact that will have though | 15:13 |
| clarkb | you have to rewrite the image over the top whoch isnt that different from writing image to new 'disk' | 15:14 |
| clarkb | really you save on network provisioning | 15:14 |
| *** hashar has quit IRC | 15:22 | |
| *** electrofelix has quit IRC | 15:23 | |
| *** jamesmcarthur has quit IRC | 15:27 | |
| *** chandankumar is now known as raukadah | 15:36 | |
| SpamapS | I still believe strongly that rebuild is an abstraction leak from the cloud provider's perspective. It's easier on a small cloud, but it's absolutely the wrong thing at scale. | 15:43 |
| SpamapS | You are glad to pay the cost of delete/create when it means you get workloads where you told your scheduler you want them. | 15:44 |
| mordred | yeah. I think most of the cloud-provider push for use of rebuild had to do with the not-very-efficient custom networking stuff at hp public cloud :) | 15:44 |
| mordred | "please stop killing us by requesting new network ports!!!" :) | 15:45 |
| *** gtema has quit IRC | 15:52 | |
| *** gtema has joined #zuul | 15:53 | |
| SpamapS | how much of nodepool is just "because HP cloud"? ;-) | 15:54 |
| pabelanger | rcarrillocruz still shared crazy hacks he came up with on grozer team | 15:55 |
| SpamapS | BTW, is there any preference for pools in a provider by order? I'm playing with EC2 spot instance pricing and was thinking if there is some kind of priority... you can have like 'first try at 10% of regular price, then 20%, then 30%' ... | 15:55 |
| clarkb | SpamapS: some of it is because rackspace | 15:57 |
| clarkb | more recently it grew some beacuse ovh | 15:57 |
| clarkb | SpamapS: each pool has a separate handler and each handler independently scans the request list | 15:57 |
| clarkb | SpamapS: so no I don't think there is an prioritization between them | 15:58 |
| *** jamesmcarthur has joined #zuul | 15:58 | |
| SpamapS | might be doable with a simple microsecond sleep setting | 15:59 |
| SpamapS | just make the expensive ones sleep a tiny bit so they don't pick things up first | 15:59 |
| SpamapS | fill up the cheap one, then it will pause and the others will grab first. | 15:59 |
| manjeets | corvus, clarkb question regarding zuul ? we had setup ci using new docker-compose it run jobs on project that has zuul.d dir in it | 16:03 |
| manjeets | but doesn't work for the project that has .zuul.yaml | 16:03 |
| manjeets | never proceeds after this line https://github.com/openstack-infra/zuul/blob/master/zuul/executor/server.py#L2763 | 16:04 |
| manjeets | merger shows succes on fetching the new change and I don't see error in the log | 16:04 |
| *** jangutter has quit IRC | 16:04 | |
| dmsimard | SpamapS: I've wanted to do something like that before. Basically a private cloud and a public cloud. Only "burst" in the public cloud if the private cloud is down or is at capacity. | 16:04 |
| manjeets | executor is just stuck | 16:04 |
| *** jamesmcarthur has quit IRC | 16:05 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: WIP - Pagure driver - https://pagure.io/pagure/ https://review.openstack.org/604404 | 16:08 |
| *** rlandy|ruck is now known as rlandy|ruck|mtg | 16:12 | |
| SpamapS | dmsimard:yeah, and on some clouds different regions cost different prices | 16:13 |
| Shrews | SpamapS: one thing i'd like to do in nodepool 4.0 is design in a pluggable request distribution algorithm, so you could do custom things like round-robin, provider preference, first-come-first-server, etc | 16:26 |
| Shrews | or whatever your fancy of the day happens to be | 16:27 |
| Shrews | but hard to do that with the current design | 16:27 |
| clarkb | ya I think that wants the noderequest dispatcher idea I've thrown out before | 16:27 |
| mordred | yeah. it's well optimized currently for "I have a bunch of clouds that are mostly identical to me" | 16:28 |
| clarkb | where nodepool doens't let handlers directly fulfill requests and instead asks pools for resources and hands them out as appropriate to jobs | 16:28 |
| mordred | but not as much for "I havea list of cloud sin preference" or "I have one and only one cloud" | 16:28 |
| clarkb | I want that for simple fifo'ing so that users don't see randomized job start times | 16:28 |
| clarkb | but could do ther things | 16:28 |
| *** jamesmcarthur has joined #zuul | 16:28 | |
| *** jpena has quit IRC | 16:29 | |
| *** rlandy|ruck|mtg is now known as rlandy|ruck | 16:33 | |
| *** jamesmcarthur has quit IRC | 16:44 | |
| *** jamesmcarthur has joined #zuul | 16:51 | |
| *** jamesmcarthur has quit IRC | 17:05 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: WIP: Add support for smart reconfigurations https://review.openstack.org/652114 | 17:13 |
| *** gtema has quit IRC | 17:21 | |
| *** jamesmcarthur has joined #zuul | 17:29 | |
| *** jamesmcarthur has quit IRC | 17:51 | |
| *** irclogbot_1 has quit IRC | 18:08 | |
| *** irclogbot_0 has joined #zuul | 18:12 | |
| pabelanger | Hmm | 18:12 |
| pabelanger | http://paste.openstack.org/show/749256/ | 18:12 |
| pabelanger | so that just prevented zuul from doing a full-reconfigure | 18:12 |
| pabelanger | tobiash: seen ^ before? | 18:13 |
| pabelanger | http://paste.openstack.org/show/749257/ is from scheduler | 18:13 |
| pabelanger | github app has correct permissions | 18:15 |
| SpamapS | Shrews:that does sound cool | 18:21 |
| pabelanger | tobiash: looks like if a cat job fails, we don't retry | 18:24 |
| pabelanger | scheduler side | 18:24 |
| *** jamesmcarthur has joined #zuul | 18:50 | |
| mordred | pabelanger: why would the cat job fail? | 18:53 |
| mordred | pabelanger: and on the first one - with it being a 502 it sounds like a github api hiccup potentially | 18:53 |
| pabelanger | mordred: yah, think api flaked out on github side | 18:54 |
| mordred | so maybe that's another place where we need to add a retry | 18:54 |
| pabelanger | Yah | 18:56 |
| tobiash | pabelanger: correct, there is not retry yet, but I think it makes sense | 19:32 |
| *** jamesmcarthur has quit IRC | 19:42 | |
| *** pcaruana has quit IRC | 20:26 | |
| *** zbr has quit IRC | 20:37 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-preview master: Use splice instead of erase/push_front https://review.openstack.org/652148 | 20:45 |
| *** jamesmcarthur has joined #zuul | 21:19 | |
| clarkb | mordred: ^ commented on that. Its been forever since I did real C++ but I read the docs and have decided its safe and think there is a minor simplifcation that could be done | 21:22 |
| *** manjeets_ has joined #zuul | 21:22 | |
| clarkb | so I +2'd | 21:22 |
| *** jamesmcarthur has quit IRC | 21:23 | |
| mordred | clarkb: so - there's a segfault happening that I think we're still looking for an ultimate reproduction of | 21:24 |
| clarkb | ah yes the segfault that resulted in spamaps deciding rust is better :) | 21:25 |
| mordred | clarkb: which might, amazingly enough, have something to do with the make_pair differences between c++03 and c++11 | 21:25 |
| mordred | clarkb: :) | 21:25 |
| mordred | clarkb: I can't reproduce it on my machine, but Shrews can on his | 21:25 |
| clarkb | back when I wrote C++ I discovered that std iostreams were implemented differently on solaris and linux such that on linux if you reused a stream object on a new fd after an error it reset the flags and on solaris it didn't | 21:27 |
| clarkb | tl;dr I totally believe it | 21:27 |
| *** rlandy|ruck has quit IRC | 21:49 | |
| pabelanger | Umm, how the heck was this possible, on a config-project? https://github.com/ansible-network/zuul-config/pull/190 | 22:02 |
| pabelanger | I changed the jobs to be noop, and zuul speclatively ran the change | 22:02 |
| pabelanger | okay, there is something really weird going on here | 22:05 |
| pabelanger | https://github.com/ansible-network/zuul-config/pull/191 | 22:06 |
| pabelanger | mordred: clarkb: SpamapS: do you happen to know how that is possible?^ | 22:06 |
| mordred | pabelanger: it shouldn't be - are you sure that repo is a config project? | 22:06 |
| pabelanger | https://ansible-network.softwarefactory-project.io/zuul/projects | 22:07 |
| pabelanger | ansible-network/zuul-config is config | 22:07 |
| clarkb | fwiw noop is magical | 22:07 |
| clarkb | it is processed entirely in the scheduler iirc | 22:07 |
| pabelanger | but, isn't that a large issue. | 22:07 |
| clarkb | its possible that happens before any trusted vs untrusted checks | 22:07 |
| clarkb | if it happens only for noop jobs then probably not since a noop job is not really a job just a magical success stauts | 22:07 |
| pabelanger | like, I can just change all config project to noop and bypass things? | 22:08 |
| clarkb | if you approve them which it looks like you did (via gate lable?) | 22:08 |
| pabelanger | I did, but they ran in check first | 22:08 |
| pabelanger | before I approved | 22:08 |
| clarkb | I would look at the noop handling in the scheduler to check if my theory holds weight | 22:09 |
| pabelanger | even https://github.com/ansible-network/zuul-config/pull/191, the revert is now running jobs | 22:09 |
| pabelanger | before I even approved anything | 22:09 |
| mordred | yeah - that seems really weird | 22:09 |
| pabelanger | let me test on zuul.ansible.com | 22:10 |
| pabelanger | okay, ansible/project-config doesn't allow me to do that | 22:13 |
| pabelanger | so, I'll have to ask SF folks to look | 22:13 |
| pabelanger | they are still on 3.6.1, and ansible.z.o is 3.7.1 | 22:13 |
| *** EvilienM is now known as EmilienM | 22:13 | |
| clarkb | pabelanger: they also carry an assortment of patches iirc | 22:14 |
| pabelanger | yes | 22:14 |
| *** manjeets_ has quit IRC | 22:15 | |
| * SpamapS doesn't really know | 22:43 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!