| *** gyee has quit IRC | 00:03 | |
| *** k_mouza has joined #openstack-glance | 01:00 | |
| *** k_mouza has quit IRC | 01:05 | |
| *** k_mouza has joined #openstack-glance | 02:07 | |
| *** k_mouza has quit IRC | 02:11 | |
| *** mvkr has quit IRC | 02:17 | |
| *** baojg has quit IRC | 02:17 | |
| *** baojg has joined #openstack-glance | 02:18 | |
| *** rosmaita has left #openstack-glance | 02:29 | |
| *** mvkr has joined #openstack-glance | 02:30 | |
| openstackgerrit | Cyril Roelandt proposed openstack/glance-specs master: Spec for Glance cache API https://review.opendev.org/665258 | 03:01 |
|---|---|---|
| *** k_mouza has joined #openstack-glance | 03:08 | |
| *** k_mouza has quit IRC | 03:13 | |
| *** rcernin has quit IRC | 03:28 | |
| *** rcernin has joined #openstack-glance | 03:43 | |
| *** rcernin has quit IRC | 03:47 | |
| *** rcernin has joined #openstack-glance | 03:49 | |
| *** k_mouza has joined #openstack-glance | 03:49 | |
| *** k_mouza has quit IRC | 03:53 | |
| *** udesale has joined #openstack-glance | 04:43 | |
| *** eandersson has quit IRC | 05:11 | |
| *** eandersson has joined #openstack-glance | 05:12 | |
| *** m75abrams has joined #openstack-glance | 05:35 | |
| *** nikparasyr has joined #openstack-glance | 06:08 | |
| openstackgerrit | Abhishek Kekane proposed openstack/glance-specs master: Introspect import plugin to calculate virtual size of image https://review.opendev.org/741121 | 06:51 |
| *** ralonsoh has joined #openstack-glance | 06:57 | |
| *** amoralej|off is now known as amoralej | 07:17 | |
| openstackgerrit | Abhishek Kekane proposed openstack/glance master: Fix broken glance-cache-manage utility https://review.opendev.org/742115 | 07:19 |
| *** brinzhang_ has joined #openstack-glance | 07:34 | |
| brinzhang_ | hi, I was update my devstack, and unstack.sh && stack.sh again, but it has an 503 error when curl -g -k --noproxy '*' -s -o /dev/null -w '%{http_code}' http://192.168.222.12/image, and it cannot start g-api service | 07:34 |
| brinzhang_ | more details in http://paste.openstack.org/show/796236/ | 07:34 |
| brinzhang_ | Is there any important changes in glance recently? | 07:35 |
| brinzhang_ | about devstack | 07:35 |
| *** jmlowe has quit IRC | 08:06 | |
| *** jmlowe has joined #openstack-glance | 08:08 | |
| *** k_mouza has joined #openstack-glance | 08:31 | |
| *** rcernin has quit IRC | 08:56 | |
| *** k_mouza has quit IRC | 09:05 | |
| openstackgerrit | Merged openstack/glance stable/train: Fix reloading under PY3 https://review.opendev.org/713720 | 09:10 |
| *** k_mouza has joined #openstack-glance | 09:13 | |
| * abhishekk will be back at 1400 UTC | 09:30 | |
| *** k_mouza has quit IRC | 09:43 | |
| *** k_mouza has joined #openstack-glance | 09:44 | |
| *** k_mouza has quit IRC | 09:46 | |
| *** tristan888 has joined #openstack-glance | 10:06 | |
| *** k_mouza has joined #openstack-glance | 10:07 | |
| *** udesale_ has joined #openstack-glance | 11:03 | |
| *** udesale has quit IRC | 11:06 | |
| *** bhagyashris is now known as bhagyashris|afk | 11:54 | |
| *** tristan888 has quit IRC | 12:07 | |
| *** rosmaita has joined #openstack-glance | 12:09 | |
| *** bhagyashris|afk is now known as bhagyashris | 12:22 | |
| *** k_mouza has quit IRC | 12:23 | |
| *** amoralej is now known as amoralej|lunch | 12:25 | |
| *** k_mouza has joined #openstack-glance | 12:42 | |
| *** baojg has quit IRC | 12:53 | |
| *** baojg has joined #openstack-glance | 12:54 | |
| *** amoralej|lunch is now known as amoralej | 13:26 | |
| dansmith | brinzhang_: enable_service tls-proxy | 13:30 |
| dansmith | brinzhang_: that will make your devstack deploy the way CI does it | 13:30 |
| *** baojg has quit IRC | 13:35 | |
| *** baojg has joined #openstack-glance | 13:37 | |
| abhishekk | rosmaita, smcginnis, jokke glance weekly meeting in 5 mins at #openstack-meeting | 13:58 |
| rosmaita | abhishekk: ty | 13:58 |
| smcginnis | Conflict as usual for me, but will try to lurk and catch up as I can. | 13:59 |
| rosmaita | same here | 13:59 |
| abhishekk | ack | 13:59 |
| *** m75abrams has quit IRC | 14:04 | |
| *** alistarle has joined #openstack-glance | 14:08 | |
| *** alistarle has quit IRC | 14:11 | |
| *** alistarle has joined #openstack-glance | 14:11 | |
| *** alistarle has quit IRC | 14:17 | |
| gmann | abhishekk: regarding this https://review.opendev.org/#/c/742546/6/tempest/api/image/v2/test_images.py@160 | 14:24 |
| gmann | abhishekk: is this os_glance_importing_to_stores added to property in API also ? | 14:25 |
| abhishekk | gmann, in meeting atm | 14:25 |
| *** alistarle has joined #openstack-glance | 14:29 | |
| *** alistarle has quit IRC | 14:31 | |
| *** k_mouza has quit IRC | 14:32 | |
| *** k_mouza has joined #openstack-glance | 14:32 | |
| *** alistarle has joined #openstack-glance | 14:35 | |
| *** alistarle has quit IRC | 14:42 | |
| *** alistarle has joined #openstack-glance | 14:44 | |
| abhishekk | gmann, it will be available in get response if that what you are asking | 14:55 |
| gmann | abhishekk: and if inject_image_metadata is enabled right? | 14:57 |
| abhishekk | gmann, it has nothing to do with inject_image_metadata | 14:57 |
| gmann | abhishekk: but i cannot see that in GET response always | 14:58 |
| abhishekk | gmann, it will be added only after import call is initialized | 14:58 |
| gmann | abhishekk: i see. I am justa dding sleep for testing and later i can add wait for this property. | 14:59 |
| abhishekk | gmann, sounds good | 14:59 |
| abhishekk | let me know If you need anything | 15:00 |
| * abhishekk need to attend another meeting | 15:00 | |
| dansmith | rosmaita: are you coming to the meeting? | 15:03 |
| rosmaita | omw | 15:03 |
| dansmith | sweet | 15:04 |
| *** k_mouza has quit IRC | 15:04 | |
| *** alistarle has quit IRC | 15:05 | |
| dansmith | jokke: ? | 15:06 |
| dansmith | rosmaita: sorry I really didn't mean to jump on you like that | 15:31 |
| dansmith | that was my bad | 15:31 |
| rosmaita | np, as long as you smile when you say that | 15:32 |
| abhishekk | rosmaita, also we are team here :D | 15:34 |
| *** udesale_ has quit IRC | 15:42 | |
| *** baojg has quit IRC | 15:48 | |
| *** baojg has joined #openstack-glance | 15:49 | |
| * abhishekk will be back in hour | 15:58 | |
| *** k_mouza has joined #openstack-glance | 16:01 | |
| dansmith | rosmaita: here's the current wsgi threading stack, fwiw: https://review.opendev.org/#/q/status:open+project:openstack/glance+branch:master+topic:async-native-threads | 16:06 |
| *** brinzhang0 has joined #openstack-glance | 16:07 | |
| dansmith | I actually got it working with two lines of code change locally, but that first patch is the proper abstraction | 16:07 |
| rosmaita | nice | 16:07 |
| *** brinzhang_ has quit IRC | 16:10 | |
| *** nikparasyr has left #openstack-glance | 16:10 | |
| *** k_mouza has quit IRC | 16:13 | |
| *** gyee has joined #openstack-glance | 16:17 | |
| *** amoralej is now known as amoralej|off | 16:25 | |
| openstackgerrit | Erno Kuvaja proposed openstack/glance stable/train: Return the mod_proxy_uwsgi paragraph https://review.opendev.org/742704 | 16:31 |
| abhishekk | dansmith, around? | 16:50 |
| dansmith | abhishekk: yep | 16:50 |
| abhishekk | we are planning to have 1 meeting on Monday to discuss race condition issue | 16:50 |
| abhishekk | what will be your suitable time for the same? | 16:51 |
| dansmith | hang on, let me do the timezone math | 16:51 |
| dansmith | abhishekk: Monday I can do 1700UTC (i.e. eight minutes from now), or any time after that | 16:52 |
| dansmith | tuesday I could do earlier | 16:52 |
| dansmith | actually, wait, | 16:52 |
| dansmith | I could skip something and do 1400-1600 as well | 16:52 |
| dansmith | just not 1600-1700 | 16:53 |
| abhishekk | Cool, I will schedule it for 1400 to 1500 | 16:53 |
| abhishekk | thank you | 16:53 |
| dansmith | ack | 16:54 |
| abhishekk | thank you, I will send the mail on ML shortly | 16:55 |
| dansmith | cool, thanks, definitely looking forward to getting that resolved | 16:57 |
| abhishekk | ++ | 16:57 |
| dansmith | abhishekk: can you explain the scrubber to me real quick to save me having to read/search? | 17:00 |
| abhishekk | scrubber is based on delaying image deletion | 17:00 |
| abhishekk | there is one config option in glance which marks image as soft deleted when enabled | 17:01 |
| abhishekk | and later scrubber can delete those images | 17:01 |
| dansmith | gotcha, and where does the scrubber run? separate daemon run by the operator? | 17:02 |
| abhishekk | separate daemon run by the operator | 17:02 |
| dansmith | okay so I don't need to make that code work without eventlet, it sounds like | 17:02 |
| abhishekk | I think so | 17:03 |
| dansmith | cool | 17:05 |
| openstackgerrit | Dan Smith proposed openstack/glance master: Make glance-api able to do async tasks in WSGI mode https://review.opendev.org/742065 | 17:10 |
| openstackgerrit | Dan Smith proposed openstack/glance master: Make image conversion use a proper python interpreter for prlimit https://review.opendev.org/742314 | 17:10 |
| openstackgerrit | Dan Smith proposed openstack/glance master: Make wsgi_app support graceful shutdown https://review.opendev.org/742493 | 17:10 |
| dansmith | abhishekk: ^ added a knob for native thread pool size, which we can default=1 for the backport | 17:10 |
| abhishekk | dansmith, ack | 17:10 |
| abhishekk | dansmith, I think we should also report a bug for the same so if decided we can backport it as well | 17:11 |
| dansmith | abhishekk: already did: https://bugs.launchpad.net/glance/+bug/1888713 | 17:11 |
| openstack | Launchpad bug 1888713 in Glance "Async tasks, image import not supported in pure-WSGI mode" [Undecided,In progress] | 17:11 |
| abhishekk | cool | 17:12 |
| dansmith | the final patch in there presumptively Closes-Bug that bug, because I don't know of any other wsgi shortcomings at this point | 17:12 |
| abhishekk | there is one, its not shortcoming but missing as we haven't added it for uwsgi | 17:13 |
| abhishekk | periodic job to prefetch image into cache | 17:13 |
| dansmith | okay, but that's a testing thing, not something the bug covers I think | 17:13 |
| dansmith | does prefetch use an async task? | 17:14 |
| abhishekk | Nope | 17:14 |
| dansmith | okay the only prefetch I know of is with the manage command, which shouldn't have anything to do with how the api is deployed.. can you point me at something? | 17:15 |
| dansmith | oh, does the manage command call the api? I didn't think it did, but I see the cache doc makes references to urls | 17:16 |
| abhishekk | glance-cache-prefetcher was a similar utility like scrubber | 17:16 |
| abhishekk | but it has dependency on registry so we moved it under api and now it runs like periodic job | 17:17 |
| dansmith | ah, so you request via API and then that daemon does the thing in the background, separate from the API itself? | 17:17 |
| abhishekk | earlier it was like that | 17:18 |
| abhishekk | now its part of API service | 17:18 |
| abhishekk | it just looks in cache db whether there is any image available for queing and then downloads that image in to cache via Api | 17:19 |
| dansmith | synchronously? | 17:20 |
| abhishekk | it just spawns a thread and run it as a deamon | 17:23 |
| jokke | dansmith: the glance-cache-manage lands through the API to the local sqlite db that is used for caching and when the service runs, it just periodically spawns thread that will take care of the fetching | 17:23 |
| jokke | dansmith: and that thread is not part of the wsgi app | 17:23 |
| abhishekk | https://github.com/openstack/glance/blob/master/glance/common/wsgi.py#L453 | 17:24 |
| jokke | oh it is :P | 17:24 |
| dansmith | ah, okay I see | 17:24 |
| jokke | it might actually work without any issues if it's spawned from there | 17:24 |
| abhishekk | :D it is not part of wsgi_app | 17:24 |
| dansmith | jokke: we don't call any of that code | 17:24 |
| jokke | ohh | 17:24 |
| dansmith | from pure wsgi | 17:24 |
| jokke | so yeah, in that case it's broken | 17:24 |
| dansmith | so, just to be clear, the user can still run the standalone daemon-y thing right? | 17:24 |
| dansmith | because uwsgi will start a bunch of worker processes, each of which shouldn't be spawning their own prefetcher | 17:25 |
| dansmith | if we did that, we'd just need to make sure only one was running at a time, but that's more complex than it needs to be, | 17:26 |
| dansmith | and probably not what people want anyway | 17:26 |
| jokke | nope, I think that's removed as like abhishek mentioned it relied on glance-registry that has been removed | 17:26 |
| dansmith | it's still in the tree | 17:26 |
| dansmith | if it depends on g-reg it should probably be deleted :) | 17:26 |
| jokke | so even if the entry point still exists either it just blows up or is noop | 17:26 |
| jokke | yeah | 17:27 |
| abhishekk | dansmith, that is there but it is broken | 17:27 |
| dansmith | :/ | 17:27 |
| dansmith | well, we can just do the naive thing and make it a daemon off the wsgi worker, but.. that's not ideal.. how hard can it be to just make that work with the local sqlite db? | 17:27 |
| abhishekk | I think it was broken since queens or rocky | 17:27 |
| dansmith | mnaser: presumably you're not using this image prefetcher thing right? | 17:28 |
| abhishekk | dansmith, I will try to run this tomorrow on uwsgi setup I have | 17:29 |
| dansmith | abhishekk: run the standalone command you mean? | 17:29 |
| abhishekk | yes | 17:29 |
| dansmith | okay | 17:30 |
| abhishekk | I will run glance-cache-prefetcher and will note down the issues | 17:30 |
| dansmith | I would normally say that we need not backport fixes to that since it was never working and it's just background stuff, | 17:30 |
| dansmith | but if we're installing something into bin/ that can't possibly work.... I think backporting fixes if they're not too crazy is probably prudent | 17:30 |
| jokke | dansmith: abhishekk: you should have bj invitation, I let abhishekk to announce it to wider public but I knew you two are going to be there | 17:31 |
| dansmith | yup, got it | 17:31 |
| abhishekk | jokke, thanks | 17:31 |
| jokke | dansmith: I doubt vexxhost is using caching g-api as they are running with ceph where nova has direct access | 17:32 |
| dansmith | I'm not sure they're all ceph everywhere, but yeah, makes sense | 17:33 |
| jokke | dansmith: the caching is really beneficial on big environments when you stream the data through glance | 17:33 |
| jokke | makes massive difference FE with swift | 17:33 |
| dansmith | sure I understand the goal | 17:33 |
| dansmith | glance-cache-prefetcher fails to import on startup for me | 17:35 |
| dansmith | which doesn't even make sense, AFICT | 17:35 |
| jokke | abhishekk: you have been poking the cache code lately ore than I have. How about the invalidation and purging code. Is that happening also outside of the wsgi app/our middleware or should that work? | 17:35 |
| dansmith | so, another question, | 17:36 |
| abhishekk | jokke, as far as I know pruner is also broken | 17:36 |
| dansmith | I thought you guys didn't have any jobs configuring devstack in WSGI_MODE=!uwsgi, but since the default was uwsgi until recently, surely any job that tests that has to be configuring devstack that way | 17:37 |
| dansmith | or did I misunderstand what periodic job you're talking about? | 17:37 |
| jokke | dansmith: there is no tempest testing for that as the API has existed only for couple of releases, until that it was 100% node local tooling | 17:40 |
| dansmith | jokke: okay abhishekk mentioned some periodic test, which I thought was a zuul periodic job that needed updating | 17:40 |
| dansmith | maybe the periodic | 17:40 |
| dansmith | just referred to the periodicness of the cache stuff in general? | 17:41 |
| jokke | dansmith: like mentioned 90% of the features implemented past 3-4 years have only unit/functional/integration coverage within repo | 17:41 |
| dansmith | yeah, understand, just was confused by what abhishekk was saying | 17:41 |
| jokke | trying to find that reference | 17:42 |
| abhishekk | dansmith, sorry for the confusion | 17:42 |
| dansmith | jokke: this, and earlier: [10:13:10] <abhishekk>periodic job to prefetch image into cache | 17:42 |
| dansmith | to me, "periodic job" relating to testing, which was the context, would be a zuul periodic job | 17:42 |
| dansmith | so I'm sure it was just my misunderstanding | 17:42 |
| jokke | yes, not zuul job, that is reference to the periodically spun thread within g-api | 17:43 |
| dansmith | gotcha | 17:43 |
| abhishekk | ^^ | 17:43 |
| abhishekk | I thought nova calls such functions as periodic jobs | 17:43 |
| dansmith | it calls them periodic tasks | 17:43 |
| dansmith | but since the context was "what else do we need to test" I just assumed you meant you had a job running somewhere that should be cloned to test wsgi mode as well | 17:44 |
| jokke | ahh ... we try to avoid mixing with the taskflow controlled stuff so there is the big defference when we talk about jobs and tasks | 17:44 |
| dansmith | yep, understandable, clearly my mistake :) | 17:44 |
| jokke | np | 17:44 |
| jokke | good that you asked and not just assume in silence | 17:45 |
| dansmith | abhishekk: I can definitely write up the change to make it spawn these as threads in each worker, and just use a lockfile to make sure only one runs at a time, but I *think* it'd be better to make the standalone daemon work for that case, if it's not too hard | 17:46 |
| dansmith | it would be nice to know what someone like mnaser, who deploys as wsgi now, would want if he _did_ need the prefetcher | 17:46 |
| dansmith | and we can also do both if you just prefer 100% parity | 17:46 |
| dansmith | but regardless, I think installing things into bin/ that don't run is pretty bad :) | 17:47 |
| abhishekk | dansmith, I will try to find out issues tomorrow about the same | 17:47 |
| dansmith | okay | 17:47 |
| abhishekk | I guess I will be able to make it work with minimal changes | 17:47 |
| jokke | abhishekk: did we have any locking ofr the prefetcher or are they just "coordinating" in the sqlite level to not race. We discussed the possible race when you implemented it | 17:47 |
| dansmith | jokke: from a quick glance it looked like we'd only spawn one of those per server... are you thinking each child might have one? | 17:48 |
| abhishekk | jokke, they just coordinate on sql level to not to race | 17:48 |
| jokke | I thought that's the thing yeah, cause we were worried next periodic picking up same veery slow prefetch and you adressed that so it won't be a problem if we have multiples of them running, I just can't remember how | 17:49 |
| dansmith | is that coordination per-image or per-host? | 17:50 |
| dansmith | because if you have multiples, you wouldn't want two images being cached in parallel I would think (or 16 in parallel for that matter) | 17:50 |
| jokke | Currently anything caching related is within the scope of the host | 17:50 |
| dansmith | okay | 17:51 |
| dansmith | so only one thread per host could possibly be making progress anyway? | 17:51 |
| jokke | so the sqlite db is on the local disk of that host and all the actions impact only the caching on that host | 17:51 |
| dansmith | oh right, no I get that, | 17:52 |
| dansmith | I'm wondering what the locking domain is for exclusion, | 17:52 |
| dansmith | whether one thread says in the db "yo, I am going to cache this image", | 17:52 |
| jokke | I think multiple can make progress, so if there is 3 images in the prefetch Q and one process fetching it will go through them in serial matter, but if another process get spun up it will just pick the next one that has not been processed yet | 17:52 |
| dansmith | or one thread says "I will cache all these requests" | 17:52 |
| dansmith | right, okay, that's what we don't want I think | 17:53 |
| dansmith | if you're running 32 wsgi workers for your 32 cores, you do *not* want 32 images being cached in parallel :) | 17:53 |
| jokke | Might be the case indeed ;) | 17:53 |
| jokke | so how it works currently is kind of autobalancing. If there is too much to do, we will spin up new thread every Xmin until the precache Q is empty | 17:54 |
| dansmith | so just to be clear, you and abhishekk said that locking was done at the sql level, but that just means to make sure two threads don't compete for the *same* image, not that they won't compete for the whole queue, is that right? | 17:54 |
| jokke | so it will gradually ramp up the parallelism ;D | 17:54 |
| jokke | dansmith: correct | 17:54 |
| dansmith | okay that seems like a storm waiting to happen to me | 17:54 |
| jokke | And the threads will just die away once the Q is empty | 17:55 |
| dansmith | after you've totally DDoS'd the disk and network though :) | 17:56 |
| jokke | but yeah, If you spin 48 of them up at once, that queue will get processed very quickly :P | 17:56 |
| jokke | And the admin might get surprised on load an traffic | 17:56 |
| dansmith | ah, the import problem on the standalone cacher is a circular dependency | 17:59 |
| jokke | dansmith: right ... and that's the same thing with number of workers. As soon as you're streaming the images through glance-api you want to be careful how many workers you let go wild. Thus we changed the default worker count few cycles ago as these 48 core servers with 96 threads got that amount of workers spun up. And 96 glance workers running in one box will bring pretty much any modern | 18:00 |
| jokke | hardware on it's knees in busy cloud | 18:00 |
| abhishekk | smcginnis, could you please have a quick look here, https://review.opendev.org/742115 | 18:00 |
| abhishekk | I guess I missed lots of discussion, scrolling back | 18:00 |
| dansmith | jokke: yeah for sure, but generally for a prefetch type thing, you want that to be a slow burn in the background and not overwhelm the machine with caching activity.. if you have a lot of *actual* work to do, then you should spend all the machine's resources trying to keep up, but not so for caching | 18:01 |
| jokke | dansmith: exactly | 18:01 |
| jokke | so yes, you don't want every worker spinning up those threads | 18:01 |
| dansmith | so... shouldn't there be a tunable knob limiting how wide the prefetcher will scale, even for the eventlet case? | 18:02 |
| dansmith | because right now, as you said, every Xmins it will get worse | 18:02 |
| dansmith | like the opposite of exponential backoff :) | 18:02 |
| dansmith | abhishekk: that same import thing occurs in the cache prefetcher cmd | 18:03 |
| jokke | dansmith: IIRC we decided to take the approach to do it if it starts breaking things instead of cluttering our already mile long config file | 18:03 |
| jokke | dansmith: It probably wouldn't hurt to have even hardcoded pool that limits it, but that wouldn't help th uwsgi case and like said no complains as of yet | 18:04 |
| dansmith | yeah, understand, it'll just be one of those saturday escalations where the machines are melting down .. I assume not many people have really tried it, but maybe I'm wrong | 18:05 |
| smcginnis | abhishekk: I'm not seeing the relationship between those test changes and the import change yet. Looking closer, but is there a quick explanation? | 18:05 |
| abhishekk | dansmith, just found out glance-cache-manage itself is broken with uwsgi | 18:05 |
| jokke | I think prefetching (as it's manual admin task) is extremely rare anyways and might happen in a cases where new golden image is introduced to the environment. So quite likely no-one even from the deployments that uses the caching is using the precaching | 18:05 |
| dansmith | abhishekk: we should have some, you know, tests for that :D | 18:05 |
| *** ralonsoh has quit IRC | 18:06 | |
| dansmith | jokke: yep | 18:06 |
| abhishekk | smcginnis, there were not tests at all to test the glance-cache-manage tool | 18:06 |
| smcginnis | abhishekk: OK, so removing settings from those functional tests because they were not actually needed? | 18:06 |
| smcginnis | needed/used | 18:06 |
| jokke | dansmith: that said, the edge usecases have had lots of requests towards that. So we indeed might start seeing more usge for it | 18:07 |
| abhishekk | smcginnis, yes | 18:07 |
| smcginnis | abhishekk: Got it, that makes sense then. | 18:07 |
| abhishekk | smcginnis, ack | 18:07 |
| dansmith | jokke: right and edge cases are likely to have smaller WAN pipes and smaller controllers with slower disks... :) | 18:07 |
| jokke | dansmith: also very limited number of images to be actually predeployed. Yet I think you are right, we might get very valid request to do something about it :P | 18:08 |
| jokke | dansmith: you gotta prioritize things and pick your fights like they say | 18:08 |
| dansmith | jokke: depends on the edge cloud.. if it's one of these "tesla rents space on the verizon cloud for self-driving-car workloads" then there are looooots of vendor images | 18:09 |
| jokke | tru | 18:09 |
| dansmith | and they probably pay for pre-caching so handoff happens as the cars move around | 18:09 |
| dansmith | anyway, I shan't get into that rabbit hole, | 18:09 |
| jokke | and in that case we get lots of visibility and that escalation might bring couple of contributors who has heavy interest to make sure it doesn't melt :P | 18:09 |
| dansmith | but I think it's a legit reason not to replicate the exact same thing for the wsgi case.. | 18:10 |
| jokke | I've been saying few years now, one of the biggest problems Glance has community wise is that it's way too solid and just works. If we had massive escalations weekly basis due to world burning, I'm pretty sure we had more hands and eyes working on this code | 18:11 |
| jokke | Yet the people left are like Brian Abhishek and myself who's nature just haven't allowed us to slip there | 18:12 |
| jokke | smcginnis: is great keeping us on our toes when ever we happen to need him to have eyes on something | 18:13 |
| abhishekk | and lately dansmith :D | 18:13 |
| abhishekk | dansmith, now there is catch with the glance-cache-manage with uwsgi | 18:14 |
| abhishekk | glance-cache-manage tries to contact glance on localhost:9292 (defualt) port | 18:14 |
| abhishekk | in case of uwsgi the port is different | 18:15 |
| abhishekk | I found it from glance-uwsgi.ini (http-socket = 127.0.0.1:60999) | 18:15 |
| abhishekk | SO if you run glance-cache-manage like | 18:16 |
| abhishekk | glance-cache-manage --port 60999 queue-image | 18:16 |
| abhishekk | it will work | 18:16 |
| *** brinzhang_ has joined #openstack-glance | 18:16 | |
| dansmith | abhishekk: but you can also do http://localhost/image | 18:16 |
| abhishekk | dansmith, that needs changes in code :D | 18:17 |
| dansmith | WAT | 18:17 |
| dansmith | you can't point it at a url? | 18:17 |
| jokke | dansmith: using uri is not a thing there | 18:17 |
| abhishekk | nope | 18:17 |
| jokke | you may present it host and port | 18:17 |
| dansmith | so all deployers *have* to run on port 9292? | 18:17 |
| abhishekk | we are using a local client which contacts glance using host and port | 18:17 |
| dansmith | ah, host and port, okay | 18:17 |
| abhishekk | no you can spescify --host and --port | 18:18 |
| dansmith | taking a uri should definitely be a goal I think | 18:18 |
| abhishekk | Now when you run glance-cache-prefetcher after that it says "No sql_connection parameter is established" | 18:18 |
| abhishekk | So I will continue to spend some time on it tomorrow | 18:19 |
| *** brinzhang0 has quit IRC | 18:19 | |
| dansmith | abhishekk: cool, sounds workable though | 18:20 |
| dansmith | at least, so far | 18:20 |
| abhishekk | yes | 18:20 |
| dansmith | abhishekk: so seriously, | 18:21 |
| dansmith | what's the plan to get this tested for real? should we poke this from the tempest side? should we do a post-run playbook like I do elsewhere so we're really using the glance binaries? | 18:22 |
| jokke | dansmith: abhishekk: I think the thing to learn from this is. Because so many components in the Glance toolchain has no tests outside of our unit/functional/integration which all runs the api on eventlet, just the service stating on uwsgi and tempest passing really does not tell anything about the health of that deployment | 18:22 |
| abhishekk | dansmith, prefetcher is using eventlet | 18:22 |
| dansmith | abhishekk: maybe more realistic functional tests are appropriate for things like the import issue | 18:22 |
| jokke | dansmith: the later probably. IIUC tempest is very clearly biased to API only | 18:22 |
| abhishekk | dansmith, yes | 18:23 |
| dansmith | jokke: sure, but tempest can queue, and then check status to see that it's working | 18:23 |
| dansmith | devstack should be starting the prefetcher if standalone, and configuring it if bundled with the API so tempest can poke it | 18:23 |
| jokke | dansmith: yeah, that definitely should not hurt | 18:24 |
| abhishekk | dansmith, right | 18:24 |
| dansmith | jokke: minimal tests in tempest means that just starting and running a nova boot does not prove the health, indeed, but I think there is *plenty* of room to expand the scope of integration testing so massively narrow the window that I see right now | 18:25 |
| dansmith | maybe I'll redeploy my devstack with eventlet so I can see about a test there | 18:26 |
| jokke | dansmith: ohh yeah | 18:26 |
| dansmith | we need to at least have devstack configure the prefetcher on in standalone mode | 18:26 |
| dansmith | abhishekk: is that patch you linked to smcginnis above the only thing I need to make it work in eventlet mode right now? | 18:27 |
| abhishekk | yes | 18:27 |
| jokke | I think it was more to point out towards the general tone we had for example today. When people argue "we run it under wsgi in the gate so sure it works", no we run it under wsgi in scope of tempest, nothing else is tested under uwsgi | 18:27 |
| jokke | +3 'u's | 18:28 |
| abhishekk | I am dropping out, almost midnight here | 18:28 |
| abhishekk | dansmith, WIP for policy doc changes | 18:29 |
| abhishekk | will be submitting it by tomorrow EOD | 18:29 |
| jokke | gn abhishekk talk to you tomorrow | 18:29 |
| abhishekk | jokke, good night | 18:29 |
| abhishekk | o/~ | 18:29 |
| mnaser | dansmith: no prefetch usage here bc we use ceph and nova does cow images | 18:31 |
| mnaser | so our glance is really a directory mostly | 18:31 |
| mnaser | and an upload mechanism | 18:31 |
| dansmith | mnaser: gotcha.. let's say you were to use this, | 18:32 |
| dansmith | would you want your API workers to double-duty as background image cachers, with some limit of how many will run at once, or would you rather run a image prefetcher daemon so you can limit the resources for that separately? | 18:33 |
| jokke | mnaser: that's what I thought ... and in that scope I truly believe that it works just fine. And is actually one of the rare cases where I'd argue it makes sense at any level having the overhead of httpd and <nameyourwsgistackhere> | 18:33 |
| mnaser | dansmith: one hundred percent a seperate daemon | 18:34 |
| mnaser | cause id want api to be as stateless as possible | 18:35 |
| dansmith | mnaser: ack, thought so :) | 18:35 |
| jokke | mnaser: even if that means that you need to have eventlet deployed to run that daemon? | 18:35 |
| mnaser | yep, all 'agents' are using eventlet in openstack now anyways | 18:35 |
| mnaser | like nova-compute and neutron-openvswitch-agent | 18:35 |
| mnaser | etc | 18:35 |
| dansmith | jokke: I'm happy to make the daemon run with real threads if it needs it, it's just that it needs to run standalone by itself I think | 18:36 |
| jokke | fair enough | 18:36 |
| dansmith | "needs to be ABLE to run standalone" I mean | 18:36 |
| jokke | dansmith: I think you don't as how it's tied to eventlet already, my point was do we need to decouple it or not | 18:36 |
| jokke | so effectively if you need prefetcher for caching, you just run one g-api process in the host that listens only localhost and you're done for now | 18:37 |
| dansmith | okay wait, maybe I'm missing something, | 18:38 |
| mnaser | does the prefetcher actually host an api? | 18:38 |
| dansmith | the cmd/cache_prefetcher.py is supposed to run standalone and not require an API righgt? | 18:39 |
| dansmith | you can still use the wsgi api to load up the cache, | 18:39 |
| dansmith | but the actual prefetching would happen in that daemon, fed from the DB, no need to run an eventlet API server right? | 18:39 |
| jokke | the prefetcher needs all the db connections and the image modelling iirc. so easiest to have it as daemon is to run g-api with one worker and just let no-one talk to it's endpoint :P | 18:39 |
| dansmith | mnaser: today they're one and the same, but.. | 18:39 |
| dansmith | jokke: except you'd have to have another API exposed, even if localhost which is super uncool | 18:40 |
| jokke | 1 firewall rule :P | 18:40 |
| dansmith | the prefetcher looking at the db, even sqlite, should be fine | 18:40 |
| dansmith | ever been through a security audit? :) | 18:40 |
| jokke | I'm not trying to go over the fence where it's at it's lowest, I'm using the damn gate :D | 18:40 |
| jokke | so yes, probably entry point for it as separate command without loading all the paste crap and actually exposing the api endpoint is fairly doable if someone wants to spend time and do it | 18:42 |
| dansmith | well, like I say, either someone needs to do that, or remove the binary that we're installing which indicates it should work, but is broken now | 18:42 |
| jokke | dansmith: ah, you're referring to that ... either way, I have patch ready to throw towards review.opendev.org for the later. But if you think someone has cycles for it and it helps the uwsgi cause, I'm all for reviewing such patch | 18:43 |
| dansmith | I definitely don't think we should be removing it, I'm just lamenting that it's known broken and still there.. but yes, let's fix it for uwsgi please | 18:45 |
| dansmith | sounds like abhi will at least start it but I will do it if need be | 18:45 |
| jokke | dansmith: btw we have quite a bit of technical debt like that in the repo, where we have fixed something since reg removal for example and the broken/unused bits are still lingering around waiting ofr cleanup | 18:51 |
| jokke | Majority of those things _should_ be documented correctly, just bits of dead code around | 18:53 |
| jokke | There was for very long time the "MVP" approach preached in Glance community where it was encouraged to do minimal impact changes to achive the initial step and leave the housekeeping for follow ups, that later just unfortunately never happened in almost every such case. | 18:56 |
| jokke | And we've still fighting to get out of that attitude towards more complete "If you do something, do it well so no-one else needs to touch it" | 18:57 |
| *** k_mouza has joined #openstack-glance | 19:09 | |
| jokke | kk, I'm off ... until tomorrow o/ | 19:13 |
| *** brinzhang0 has joined #openstack-glance | 19:17 | |
| *** brinzhang_ has quit IRC | 19:20 | |
| *** k_mouza has quit IRC | 19:22 | |
| *** jmlowe has quit IRC | 19:33 | |
| *** jmlowe has joined #openstack-glance | 19:34 | |
| *** brinzhang_ has joined #openstack-glance | 19:37 | |
| *** brinzhang0 has quit IRC | 19:41 | |
| *** jmlowe has quit IRC | 19:57 | |
| *** jmlowe has joined #openstack-glance | 20:00 | |
| openstackgerrit | Merged openstack/glance master: Fix broken glance-cache-manage utility https://review.opendev.org/742115 | 20:03 |
| openstackgerrit | Abhishek Kekane proposed openstack/glance stable/ussuri: Fix broken glance-cache-manage utility https://review.opendev.org/742742 | 20:35 |
| *** jmlowe has quit IRC | 20:57 | |
| *** jmlowe has joined #openstack-glance | 21:00 | |
| *** jmlowe has quit IRC | 21:56 | |
| *** jmlowe has joined #openstack-glance | 22:00 | |
| dansmith | smcginnis: another softball for ya: https://review.opendev.org/#/c/742309/ | 22:24 |
| *** k_mouza has joined #openstack-glance | 22:30 | |
| *** k_mouza has quit IRC | 22:34 | |
| *** jmlowe has quit IRC | 22:58 | |
| *** jmlowe has joined #openstack-glance | 23:00 | |
| *** jmlowe has quit IRC | 23:56 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!