| corvus | clarkb: was just waiting for a second +2; done | 00:01 |
|---|---|---|
| clarkb | iurygregory: sorry that was meant for ianw | 00:02 |
| clarkb | corvus: cool was just checking as I thought it looekd fine | 00:02 |
| *** tosky has quit IRC | 00:17 | |
| openstackgerrit | Merged zuul/zuul master: Gerrit: Add SSH review tag https://review.opendev.org/c/zuul/zuul/+/771733 | 00:19 |
| iurygregory | clarkb, no worries =) | 00:36 |
| openstackgerrit | Merged zuul/zuul master: Gerrit: Add gerrit version check for HTTP review tag https://review.opendev.org/c/zuul/zuul/+/775037 | 01:03 |
| *** ianychoi has joined #zuul | 02:09 | |
| *** jamesmcarthur has quit IRC | 04:12 | |
| *** jamesmcarthur has joined #zuul | 04:13 | |
| *** jamesmcarthur has quit IRC | 04:18 | |
| *** jamesmcarthur has joined #zuul | 04:37 | |
| *** ykarel|away has joined #zuul | 04:40 | |
| *** ykarel|away is now known as ykarel | 04:41 | |
| *** jamesmcarthur has quit IRC | 05:23 | |
| *** jamesmcarthur has joined #zuul | 05:23 | |
| *** jamesmcarthur has quit IRC | 05:28 | |
| *** jamesmcarthur has joined #zuul | 05:28 | |
| *** jfoufas1 has joined #zuul | 05:31 | |
| *** evrardjp has quit IRC | 05:33 | |
| *** evrardjp has joined #zuul | 05:33 | |
| *** jamesmcarthur has quit IRC | 06:11 | |
| *** jamesmcarthur has joined #zuul | 06:11 | |
| *** jamesmcarthur has quit IRC | 06:16 | |
| *** vishalmanchanda has joined #zuul | 06:22 | |
| *** jamesmcarthur has joined #zuul | 06:42 | |
| *** reiterative has quit IRC | 06:49 | |
| *** reiterative has joined #zuul | 06:49 | |
| *** jcapitao has joined #zuul | 07:22 | |
| *** jamesmcarthur has quit IRC | 07:29 | |
| openstackgerrit | Merged zuul/zuul master: Format multi-line log entries https://review.opendev.org/c/zuul/zuul/+/772602 | 07:30 |
| *** jamesmcarthur has joined #zuul | 07:43 | |
| *** ykarel has quit IRC | 07:44 | |
| *** ykarel has joined #zuul | 07:46 | |
| *** jamesmcarthur has quit IRC | 07:52 | |
| *** hashar has joined #zuul | 07:52 | |
| *** rpittau|afk is now known as rpittau | 07:53 | |
| *** jamesmcarthur has joined #zuul | 08:08 | |
| *** jpena|off is now known as jpena | 08:29 | |
| *** jamesmcarthur has quit IRC | 08:30 | |
| *** ianychoi has quit IRC | 08:33 | |
| *** tosky has joined #zuul | 08:36 | |
| *** jamesmcarthur has joined #zuul | 08:44 | |
| *** ianychoi has joined #zuul | 08:48 | |
| *** jamesmcarthur has quit IRC | 09:03 | |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Merge ZooKeeper connection and client classes https://review.opendev.org/c/zuul/zuul/+/771442 | 09:16 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Connect merger to Zookeeper https://review.opendev.org/c/zuul/zuul/+/716221 | 09:16 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Connect fingergw to Zookeeper https://review.opendev.org/c/zuul/zuul/+/716875 | 09:17 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Connect executor to Zookeeper https://review.opendev.org/c/zuul/zuul/+/716262 | 09:17 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Improve typings in context of 744416 https://review.opendev.org/c/zuul/zuul/+/753578 | 09:17 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Make Zookeeper mandatory for Scheduler https://review.opendev.org/c/zuul/zuul/+/756716 | 09:17 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Make ConnectionRegistry mandatory for Scheduler https://review.opendev.org/c/zuul/zuul/+/757095 | 09:18 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Improve typings in context of 756716 and 757095 https://review.opendev.org/c/zuul/zuul/+/757148 | 09:18 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Instantiate executor client, merger, nodepool and app within Scheduler https://review.opendev.org/c/zuul/zuul/+/757149 | 09:18 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Improve typings in context of 756304 https://review.opendev.org/c/zuul/zuul/+/757097 | 09:18 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Component Registry in ZooKeeper https://review.opendev.org/c/zuul/zuul/+/759187 | 09:18 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Move management and result events to model https://review.opendev.org/c/zuul/zuul/+/761163 | 09:18 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Allow (de-)serialization of management events https://review.opendev.org/c/zuul/zuul/+/761164 | 09:19 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Allow (de-)serialization of result events https://review.opendev.org/c/zuul/zuul/+/761165 | 09:19 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Add and fix fields in driver trigger event models https://review.opendev.org/c/zuul/zuul/+/761166 | 09:19 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Allow (de-)serialization of trigger events https://review.opendev.org/c/zuul/zuul/+/761167 | 09:19 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Interface to get a driver's trigger event class https://review.opendev.org/c/zuul/zuul/+/761168 | 09:19 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Clear list of Zookeeper connections after tests https://review.opendev.org/c/zuul/zuul/+/761169 | 09:19 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Increase default test wait timeout to 120s https://review.opendev.org/c/zuul/zuul/+/763754 | 09:19 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Implementation of Zookeeper backed event queues https://review.opendev.org/c/zuul/zuul/+/761170 | 09:20 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Implementation of Zookeeper event watcher https://review.opendev.org/c/zuul/zuul/+/761171 | 09:20 |
| *** jamesmcarthur has joined #zuul | 09:20 | |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Switch to Zookeeper backed trigger event queues https://review.opendev.org/c/zuul/zuul/+/761172 | 09:20 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Switch to Zookeeper backed management event queues https://review.opendev.org/c/zuul/zuul/+/761738 | 09:20 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Driver event ingestion https://review.opendev.org/c/zuul/zuul/+/717299 | 09:20 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Use logical timestamp to detect outdated changes https://review.opendev.org/c/zuul/zuul/+/763755 | 09:21 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Make buildset mandatory on build https://review.opendev.org/c/zuul/zuul/+/770900 | 09:21 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Implement ZooKeeper builds API https://review.opendev.org/c/zuul/zuul/+/770901 | 09:21 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Switch to ZooKeeper backed job execution and result events https://review.opendev.org/c/zuul/zuul/+/770902 | 09:21 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Refactor pipeline processing in run handler https://review.opendev.org/c/zuul/zuul/+/771452 | 09:21 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Dequeue superceded items via management event https://review.opendev.org/c/zuul/zuul/+/771453 | 09:21 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Text stream API for sharded Zookeeper data https://review.opendev.org/c/zuul/zuul/+/771454 | 09:21 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Cache unparsed config files in Zookeeper https://review.opendev.org/c/zuul/zuul/+/771455 | 09:21 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Store tenants in unparsed abide as dict https://review.opendev.org/c/zuul/zuul/+/771456 | 09:21 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Refactor config/tenant (re-)loading https://review.opendev.org/c/zuul/zuul/+/771457 | 09:22 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Tenant read/write lock in Zookeeper https://review.opendev.org/c/zuul/zuul/+/771458 | 09:22 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Lock pipelines during processing https://review.opendev.org/c/zuul/zuul/+/771459 | 09:22 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Lock global event queues during processing https://review.opendev.org/c/zuul/zuul/+/771460 | 09:22 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Store tenant layout state in Zookeeper https://review.opendev.org/c/zuul/zuul/+/771461 | 09:22 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Configure unique command socket path per scheduler https://review.opendev.org/c/zuul/zuul/+/771462 | 09:22 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Support cross scheduler config loading https://review.opendev.org/c/zuul/zuul/+/771463 | 09:22 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Add UUID for queue items https://review.opendev.org/c/zuul/zuul/+/772512 | 09:22 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Store semaphore state in Zookeeper https://review.opendev.org/c/zuul/zuul/+/772513 | 09:22 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Fix test_gerrit.TestPolling.test_config_update https://review.opendev.org/c/zuul/zuul/+/773023 | 09:23 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Move serialization helper methods to ZooKeeperBase class https://review.opendev.org/c/zuul/zuul/+/773024 | 09:23 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Implement ZooKeeper backed merge jobs https://review.opendev.org/c/zuul/zuul/+/773025 | 09:23 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Switch to ZooKeeper backed merge jobs https://review.opendev.org/c/zuul/zuul/+/773026 | 09:23 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Collect statsd information from ZooKeeper rather than gearman https://review.opendev.org/c/zuul/zuul/+/773027 | 09:23 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Remove remaining gearman parts from merger https://review.opendev.org/c/zuul/zuul/+/773028 | 09:23 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Provide zk_client to merger client rather than the whole scheduler https://review.opendev.org/c/zuul/zuul/+/773029 | 09:23 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Remove history from RecordingMergeClient https://review.opendev.org/c/zuul/zuul/+/773030 | 09:23 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Align ZooKeeper builds and merger API https://review.opendev.org/c/zuul/zuul/+/773031 | 09:24 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Make scheduler optional in Nodepool https://review.opendev.org/c/zuul/zuul/+/774605 | 09:24 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Directly return the nodeset in nodepool without the scheduler https://review.opendev.org/c/zuul/zuul/+/774606 | 09:24 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Remove scheduler from Nodepool completely https://review.opendev.org/c/zuul/zuul/+/774607 | 09:24 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Make NodeSet fully serializable https://review.opendev.org/c/zuul/zuul/+/774608 | 09:24 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Calculate build start and end time on executor server https://review.opendev.org/c/zuul/zuul/+/774609 | 09:24 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Lock/unlock nodes on executor server https://review.opendev.org/c/zuul/zuul/+/774610 | 09:24 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: DNM: Reduce number of jobs for SOS development https://review.opendev.org/c/zuul/zuul/+/775081 | 09:24 |
| *** andy-ladjadj has joined #zuul | 09:28 | |
| *** jamesmcarthur has quit IRC | 09:30 | |
| *** andy-ladjadj has quit IRC | 09:31 | |
| *** andy-ladjadj has joined #zuul | 09:33 | |
| *** ykarel is now known as ykarel|lunch | 09:33 | |
| *** Pilou has quit IRC | 09:38 | |
| *** hashar has quit IRC | 10:07 | |
| *** hashar has joined #zuul | 10:08 | |
| *** nils has joined #zuul | 10:16 | |
| *** ykarel|lunch is now known as ykarel | 10:36 | |
| *** jamesmcarthur has joined #zuul | 11:28 | |
| *** jamesmcarthur has quit IRC | 11:35 | |
| tobiash | corvus: I'm having a hard time getting nodepool up again after a cloud image since it gets immediately stuck with deleting 1500 nodes at the same time with one thread each | 11:47 |
| tobiash | I think we need to think about how we can reduce the thread count there | 11:48 |
| tobiash | s/cloud image/cloud outage/ | 11:52 |
| *** jcapitao has quit IRC | 12:02 | |
| tobiash | we might hit a limit in regarding nodes per provider/tenant maybe | 12:16 |
| *** andy-ladjadj has quit IRC | 12:21 | |
| *** rlandy has joined #zuul | 12:29 | |
| *** jpena is now known as jpena|lunch | 12:33 | |
| *** mgagne has quit IRC | 12:52 | |
| *** mgagne has joined #zuul | 12:53 | |
| avass | tobiash: you don't happen to have a workaround to get the kubernetes python library to work behind a proxy? I found this: https://github.com/kubernetes-client/python/issues/333#issuecomment-398087826 | 13:07 |
| tobiash | avass: I never tried that | 13:07 |
| avass | but that would require a patch to nodepool and it's a bit hackish | 13:08 |
| avass | okay | 13:08 |
| avass | I'll see if we can get the endpoint to come to us instead then :) | 13:08 |
| *** jpena|lunch is now known as jpena | 13:26 | |
| *** jamesmcarthur has joined #zuul | 13:32 | |
| *** jamesmcarthur has quit IRC | 13:39 | |
| tobiash | avass: last resort would be tunneling through socat | 13:49 |
| *** vishalmanchanda has quit IRC | 14:14 | |
| *** ikhan has joined #zuul | 14:20 | |
| *** ykarel_ has joined #zuul | 14:26 | |
| openstackgerrit | Adam Richter proposed zuul/zuul master: test https://review.opendev.org/c/zuul/zuul/+/775108 | 14:28 |
| *** ykarel has quit IRC | 14:29 | |
| openstackgerrit | Ron Izraeli proposed zuul/zuul master: Linkify BuildOutput Show URLs in message as links https://review.opendev.org/c/zuul/zuul/+/775109 | 14:29 |
| openstackgerrit | Ron Izraeli proposed zuul/zuul master: Linkify BuildOutput Show URLs in message as links https://review.opendev.org/c/zuul/zuul/+/775109 | 14:31 |
| *** nils has quit IRC | 14:33 | |
| *** nils has joined #zuul | 14:41 | |
| *** ikhan has quit IRC | 14:43 | |
| *** ykarel_ has quit IRC | 14:49 | |
| openstackgerrit | Ron Izraeli proposed zuul/zuul master: Linkify BuildOutput Show URLs in message as links https://review.opendev.org/c/zuul/zuul/+/775109 | 14:50 |
| *** hashar is now known as hasharAway | 14:51 | |
| *** andy-ladjadj has joined #zuul | 14:56 | |
| *** saneax has quit IRC | 15:11 | |
| *** ikhan has joined #zuul | 15:16 | |
| *** ikhan has quit IRC | 15:20 | |
| *** hasharAway is now known as hashar | 15:22 | |
| *** jamesmcarthur has joined #zuul | 15:36 | |
| *** andy-ladjadj has quit IRC | 15:36 | |
| *** jamesmcarthur has quit IRC | 15:42 | |
| openstackgerrit | Adam Richter proposed zuul/zuul master: Linkify BuildOutput Show URLs in message as links https://review.opendev.org/c/zuul/zuul/+/775109 | 15:43 |
| -openstackstatus- NOTICE: Recent POST_FAILURE results from Zuul for builds started prior to 15:47 UTC were due to network connectivity issues reaching one of our log storage providers, and can be safely rechecked | 15:52 | |
| *** jfoufas1 has quit IRC | 16:09 | |
| *** jamesmcarthur has joined #zuul | 16:45 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-storage-proxy master: Use opendev base docker image and add jobs https://review.opendev.org/c/zuul/zuul-storage-proxy/+/774998 | 17:12 |
| corvus | mordred: ^ added gcc to bindep; builds locally now, thanks :) | 17:12 |
| mordred | corvus: \o/ | 17:12 |
| *** jamesmcarthur has quit IRC | 17:20 | |
| *** jamesmcarthur has joined #zuul | 17:20 | |
| corvus | tobiash: oh, looking an the storage proxy -- it looks like you handle multiple clouds by trying all of them. that's an interesting approach! that would probably scale up to, say <10 clouds reasonably well i think. seen any issues with that? | 17:22 |
| tobiash | we use that with 3 clouds atm | 17:23 |
| corvus | tobiash: does it have to get a keystone auth token for each get request to each cloud? | 17:23 |
| corvus | or does it keep them around since the connections are long-lived in the cloudcache? | 17:24 |
| corvus | i'm guessing the latter, so the first request would get X auth tokens, but thereafter marginal cost is small | 17:24 |
| tobiash | corvus: it holds active sessions for each cloud in the background | 17:25 |
| corvus | k, that's what i thought | 17:26 |
| mordred | I like that approach | 17:26 |
| corvus | yeah, i was looking into whether we need to send over extra info about what cloud to use, but i think this is good for now | 17:27 |
| mordred | yeah - the openstack Connection object will cache the tokens - and will re-auth as needed if it times out | 17:27 |
| *** rlandy is now known as rlandy|biab | 17:30 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-storage-proxy master: Remove container prefix and tenant https://review.opendev.org/c/zuul/zuul-storage-proxy/+/775243 | 17:32 |
| corvus | tobiash: I have another question which i have phrased in the form of a patch ^ can you let me know if i'm missing something that required the tenant/prefix stuff? | 17:33 |
| tobiash | corvus: does openstacksdk implicitly use the first part of the path as container? | 17:41 |
| tobiash | corvus: cloud information maybe could be added later optionally like two modes (try all clouds or try cloud that's encoded in the path) | 17:43 |
| *** rpittau is now known as rpittau|afk | 17:43 | |
| corvus | tobiash: well, the existing proxy code explicitly adds the container back into the path, so i think so. | 17:43 |
| corvus | (line 80 in handle_get) | 17:43 |
| tobiash | I think both options have their advantages, the try all clouds mode could enable replication between clouds and silent fallback, the encoded cloud name would improve access times with many cluds | 17:43 |
| tobiash | corvus: ah now I get it, the current code adds the container prefix to the container name like 'opendev-prod-' so that doesn't need to be in the user visible url | 17:45 |
| tobiash | I'd like to retain this behavior | 17:46 |
| *** jpena is now known as jpena|off | 17:47 | |
| corvus | tobiash: okay, i think i see the intent. i'll propose a new thing. | 17:47 |
| clarkb | the reason for the container name mangling like that is some object stores can't handle duplicate container names | 17:49 |
| corvus | clarkb: sure, but zuul knows what container it uploaded to and can tell the proxy in the url | 17:49 |
| corvus | the reason for the container name mangling is tobiash wants to hide the full container name from users | 17:50 |
| *** jamesmcarthur has quit IRC | 17:53 | |
| *** jamesmcarthur has joined #zuul | 17:54 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-storage-proxy master: Make container name prefix more generic https://review.opendev.org/c/zuul/zuul-storage-proxy/+/775243 | 17:55 |
| corvus | tobiash: ^ how about that? | 17:55 |
| clarkb | corvus: ya I just awnted to point out that the data isn't really important, Its largely there to make s3 and ceph happy | 17:55 |
| tobiash | corvus: lgtm, I guess while you're at it we could switch the log to info, the warning was just a crude hack to not having to customize the log config ;) | 17:57 |
| openstackgerrit | James E. Blair proposed zuul/zuul-storage-proxy master: Log at info level https://review.opendev.org/c/zuul/zuul-storage-proxy/+/775247 | 17:59 |
| tobiash | thanks :) | 17:59 |
| corvus | np | 17:59 |
| corvus | tobiash: do you have any thoughts about gunicorn vs uwsgi? | 18:00 |
| *** jamesmcarthur has quit IRC | 18:00 | |
| tobiash | I don't know anymore, I think gunicorn was easier or just the first thing I found | 18:00 |
| tobiash | but at least we had to tune the parameters to make it performant for prod | 18:01 |
| corvus | ok. my expectation is that uwsgi will perform better; i'd like to validate that, and if so, i'll propose a switch. | 18:01 |
| tobiash | ++ | 18:01 |
| tobiash | at that time I took the first thing I could get to work because of ENOTIME | 18:02 |
| tobiash | hence also the missing unit tests | 18:02 |
| openstackgerrit | James E. Blair proposed zuul/zuul-storage-proxy master: Just use CLOUD_NAMES as env variable https://review.opendev.org/c/zuul/zuul-storage-proxy/+/775248 | 18:08 |
| corvus | tobiash: i'm guessing that's the code path you're using anyway, so that would be no change for you ^. | 18:09 |
| tobiash | you can guess what was first and needed to stay compatible during a version bump :) | 18:09 |
| corvus | heh :) | 18:09 |
| *** jamesmcarthur has joined #zuul | 18:17 | |
| *** rlandy|biab is now known as rlandy | 18:37 | |
| tobiash | corvus: I think for nodepool I'll need a load testing tool that generates a lot of node requests without a zuul, would that something that's useful having in nodepool/tools? | 18:37 |
| Shrews | tobiash: i had a similar tool for that. guess i didn't include it in tools | 18:39 |
| * Shrews checks other machine to see if it still exists | 18:39 | |
| tobiash | Shrews: that would be awesome :) | 18:40 |
| mordred | corvus: fwiw - I believe we make a uwsgi base image in the opendev image jobs | 18:41 |
| corvus | tobiash: yes; note that you can generate a lot of requests by running a launcher with a high min-ready count. | 18:41 |
| Shrews | tobiash: ugh, nope. seems it's gone. sorry | 18:41 |
| tobiash | corvus: I'm thinking about something that puts loads of node requests into zk and keeps the nodes for some time and releases them again | 18:42 |
| tobiash | to get fast node turnaround times and have a high instance creation rate | 18:42 |
| tobiash | I think we might approach a limit atm at ~1000 per provider | 18:43 |
| clarkb | tobiash: there are also limits to how many you can do per process (this is why we split things up into multiple launchers | 18:45 |
| tobiash | clarkb: I've already split into one launcher per provider | 18:45 |
| mordred | corvus: yup - we even use it for lodgeit: https://opendev.org/opendev/lodgeit/src/branch/master/Dockerfile | 18:46 |
| tobiash | so to scale more I think I'd need to tune nodepool or split one provider into two smaller ones in different tenants | 18:46 |
| tobiash | if I find a way to tune it I'd prefer that | 18:46 |
| corvus | tobiash: right, but you may only need to write a node request deletor | 18:46 |
| corvus | er | 18:47 |
| corvus | rather a node deleter | 18:47 |
| corvus | use nodepool to drive the request node, then just delete them fast. | 18:47 |
| tobiash | corvus: yes, that's the plan | 18:47 |
| tobiash | not sure if I want to lock/use/unlock the nodes as well to simulate some jobs | 18:47 |
| corvus | right, i was just saying you could let min-ready drive the request side of things and maybe same some time | 18:48 |
| tobiash | maybe keeping each node for some minutes makes the test more realistic | 18:48 |
| corvus | i was assuming you wanted to optimize the delete path, so you wouldn't need to handle locks. basically, min-ready plus mass-delete gets you a quick way to exercise that case | 18:49 |
| corvus | but obviously if you want a more sophisticated simulation, then doing the whole request workflow makes sense | 18:49 |
| corvus | anyway, just a thought that could save a bit of time | 18:49 |
| tobiash | yeah, thx | 18:49 |
| tobiash | I'm not sure if it's just the delete path actually | 18:50 |
| tobiash | corvus: I'm not sure if it's just the delete path actua | 18:50 |
| tobiash | oops | 18:50 |
| tobiash | http://paste.openstack.org/show/802581/ | 18:50 |
| tobiash | that's a profile dump from today | 18:50 |
| clarkb | tobiash: gotcha so that limit likely is also for the launcher as a whole too. A full steam ahead launcher with one provider hits that limit essentailly | 18:51 |
| tobiash | it looks like it is using mostly the optimized list_servers approach already but many threads calling list_servers makes it computationally expensive when approaching hindreds of threads | 18:51 |
| corvus | tobiash: at large numbers of threads in python, context switches overshadow code | 18:52 |
| corvus | 1k threads is the effective limit we've seen in nodepool | 18:52 |
| tobiash | so I'm thinking it might be worth exploring having one deleter and one creator thread that periodically calls list_servers and updates the according states in one go | 18:52 |
| corvus | tobiash: can you elaborate on that? | 18:53 |
| tobiash | currently we have one thread per instance creation/deletion which all call list_servers (which is cached but still is expensive to filter and process when doing with too many threads). I think it might be much more efficient to have a creator and a deleter thread per provider that does the state handling of creating and deleting nodes. | 18:55 |
| tobiash | this way it could call get_servers, update all nodes in the creation/deletion list and cycle back to get_servers | 18:56 |
| tobiash | that way we would avoid the thread contention | 18:56 |
| corvus | so a launcher would only have 2 threads? | 18:56 |
| tobiash | per provide | 18:56 |
| tobiash | I think for the deletion use case this would be easy since that's mostly fire and forget | 18:56 |
| corvus | that's a major redesign, due to the fact that creating and deleting are messy asynchronous operations on multiple resources | 18:56 |
| tobiash | for the creation part that might be more complicated | 18:57 |
| corvus | we looked at doing that for the v3 rewrite, but felt it would be too complex | 18:57 |
| tobiash | I think I'd like to try that at least for deletions since I'm convinced that has some benefit | 18:57 |
| tobiash | it can well be that this is too complicated for the creations | 18:58 |
| corvus | tobiash: i agree this might work for deletion | 19:00 |
| corvus | i mean, obviously it would work; i mean, i agree deletion may be simple enough that implementation would not be too complex to understand :) | 19:01 |
| tobiash | yes, call delete, put instance/node id into a list and let a thread periodically get the server list and mark the deletion as finished when the instance is gone | 19:01 |
| tobiash | creation is much more complex | 19:02 |
| tobiash | but maybe deletion is the bigger problem anyway since that can occur in masses e.g. during a gate reset | 19:02 |
| corvus | and it may even be possible for creation, but i think we'd have to have a state machine engine, or async. and my experience with async is that it's impossible to debug, so i'd vote state machine if we did that. | 19:02 |
| corvus | tobiash: yes. and every delete thread you get rid of is a create thread you can have | 19:03 |
| tobiash | but I think I'll first establish a ground truth against a test cloud and see if that improves nodepool's performance | 19:03 |
| corvus | sounds good | 19:03 |
| tobiash | yeah, state machine is likely easier | 19:03 |
| clarkb | another option is multiprocessing, though i don't necessarily think that is better (particularly if you want thousands of processes) | 19:03 |
| corvus | clarkb: yeah, we could lose some of the benefits of caching cloud responses (list servers) if we do that | 19:04 |
| tobiash | clarkb: multiprocessing would hammer the cloud more since the server list gets quite expensive with 1000 nodes in a tenant | 19:04 |
| mordred | yah. although I've actually had thoughts for quite a while about external caching for those reponses | 19:04 |
| mordred | so - if that does wind up being a road that needs to be explored, I don't think it's impossible to have a shared external cache for the list results | 19:05 |
| corvus | yeah, with external caching multiprocessing is viable | 19:05 |
| corvus | but we spend most of our time waiting on responses to api requests, so the GIL isn't actually that much of an issue. it's more the context switch to each of them which then do a little bit of work to see if they're ready to move on. | 19:05 |
| tobiash | that's then plan b for instance creation optimization :) | 19:06 |
| corvus | heh, uwsgi is more strict about the wsgi protocol than gunicorn | 19:09 |
| tobiash | maybe that's why got gunicorn running first :D | 19:12 |
| *** sshnaidm is now known as sshnaidm|afk | 19:19 | |
| corvus | gunicorn and uwsgi are pretty competitive with big log files; with lots of smaller requests, uwsgi is faster. i'm downloading a 15k host-info.yaml file using ab with 100x concurrency and 10 workers; gunicorn handles 650/sec, uwsgi 1100/sec. | 19:23 |
| tobiash | impressive | 19:23 |
| corvus | (i switched the proxy to just use the requests library to talk to an http server on my local network, so we're missing the whole keystone auth part from this benchmark, but i thought that was preferable to making it a test of network bandwidth) | 19:25 |
| tobiash | ++ | 19:25 |
| corvus | i'll try out this opendev uwsgi container image... | 19:26 |
| fungi | testing appreciated. i don't think we're actually consuming it in opendev yet since we haven't redeployed our lodgeit from the containers yet | 19:38 |
| fungi | though i think noonedeadpunk may be using it already | 19:39 |
| *** jamesmcarthur has quit IRC | 19:52 | |
| *** jamesmcarthur has joined #zuul | 19:53 | |
| *** jamesmcarthur has quit IRC | 19:57 | |
| mordred | fungi: yeah - vexxhost is also using the uwsgi-base image for a few things | 19:58 |
| mordred | https://opendev.org/vexxhost/atmosphere/src/branch/master/Dockerfile for instance | 19:58 |
| openstackgerrit | Jeremy Stanley proposed zuul/zuul-storage-proxy master: Use opendev base docker image and add jobs https://review.opendev.org/c/zuul/zuul-storage-proxy/+/774998 | 19:58 |
| fungi | mordred: ^ as requested | 19:58 |
| mordred | thank you sir! | 19:58 |
| fungi | thank you as well, i just copied and pasted for the most part | 19:59 |
| mordred | best way to write code | 19:59 |
| *** jamesmcarthur has joined #zuul | 20:04 | |
| fungi | everything i know about solid development practices i learned by copying and pasting random bits of php i found under the sofa cushions ;) | 20:15 |
| *** hashar has quit IRC | 21:18 | |
| *** jamesmcarthur has quit IRC | 21:36 | |
| *** jamesmcarthur has joined #zuul | 21:36 | |
| *** jamesmcarthur has quit IRC | 21:42 | |
| *** jamesmcarthur has joined #zuul | 21:57 | |
| *** gmann is now known as gmann_afk | 22:10 | |
| *** PrinzElvis has quit IRC | 22:41 | |
| *** iamweswilson has quit IRC | 22:41 | |
| *** iamweswilson has joined #zuul | 22:41 | |
| *** PrinzElvis has joined #zuul | 22:41 | |
| *** jamesmcarthur has quit IRC | 22:45 | |
| *** jamesmcarthur has joined #zuul | 22:45 | |
| *** jamesmcarthur has quit IRC | 22:49 | |
| *** jamesmcarthur has joined #zuul | 22:49 | |
| *** nils has quit IRC | 22:58 | |
| *** rlandy has quit IRC | 23:05 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-storage-proxy master: Make container name prefix more generic https://review.opendev.org/c/zuul/zuul-storage-proxy/+/775243 | 23:08 |
| openstackgerrit | James E. Blair proposed zuul/zuul-storage-proxy master: Log at info level https://review.opendev.org/c/zuul/zuul-storage-proxy/+/775247 | 23:08 |
| openstackgerrit | James E. Blair proposed zuul/zuul-storage-proxy master: Just use CLOUD_NAMES as env variable https://review.opendev.org/c/zuul/zuul-storage-proxy/+/775248 | 23:08 |
| openstackgerrit | James E. Blair proposed zuul/zuul-storage-proxy master: Switch to uwsgi https://review.opendev.org/c/zuul/zuul-storage-proxy/+/775294 | 23:08 |
| corvus | mordred, tobiash: ^ there we go, that should do it i think | 23:09 |
| *** gmann_afk is now known as gmann | 23:11 | |
| mordred | corvus: that's a neat stack! | 23:20 |
| openstackgerrit | James E. Blair proposed zuul/zuul-storage-proxy master: Switch to uwsgi https://review.opendev.org/c/zuul/zuul-storage-proxy/+/775294 | 23:27 |
| corvus | mordred: yeah, i like how the uwsgi stuff plugs in. ^ one char typo :/ | 23:28 |
| mordred | corvus: \o/ | 23:30 |
| corvus | mordred: i feel like we have tobiash's approval in spirit for the uwsgi switch; so i think the whole stack has enough approvals to merge... | 23:32 |
| corvus | mordred: but can you remove your +w from 775243? | 23:32 |
| mordred | yup! | 23:32 |
| corvus | mordred: i'd like to merge the first 2 changes alone, then verify the docker upload before merging the rest | 23:33 |
| mordred | ++ good plan | 23:33 |
| corvus | cool, done :) | 23:33 |
| openstackgerrit | Merged zuul/zuul-storage-proxy master: Initial import of zuul storage proxy https://review.opendev.org/c/zuul/zuul-storage-proxy/+/774942 | 23:40 |
| *** jamesmcarthur has quit IRC | 23:41 | |
| *** jamesmcarthur has joined #zuul | 23:42 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-storage-proxy master: Use opendev base docker image and add jobs https://review.opendev.org/c/zuul/zuul-storage-proxy/+/774998 | 23:53 |
| corvus | fungi: ^ oops too much copy pasta | 23:54 |
| corvus | i'm going to just re-approve that; it's an obvious copy-pasta fix | 23:54 |
| openstackgerrit | James E. Blair proposed zuul/zuul-storage-proxy master: Make container name prefix more generic https://review.opendev.org/c/zuul/zuul-storage-proxy/+/775243 | 23:55 |
| openstackgerrit | James E. Blair proposed zuul/zuul-storage-proxy master: Log at info level https://review.opendev.org/c/zuul/zuul-storage-proxy/+/775247 | 23:55 |
| openstackgerrit | James E. Blair proposed zuul/zuul-storage-proxy master: Just use CLOUD_NAMES as env variable https://review.opendev.org/c/zuul/zuul-storage-proxy/+/775248 | 23:55 |
| openstackgerrit | James E. Blair proposed zuul/zuul-storage-proxy master: Switch to uwsgi https://review.opendev.org/c/zuul/zuul-storage-proxy/+/775294 | 23:55 |
| *** jamesmcarthur has quit IRC | 23:58 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!