| dansmith | abhishekk: for tomorrow, I think your test_reload() test is racy and some child processes get confused which causes it to hit "no new log file created" and when that happens, the cleanup in the base test also waits forever on the api server child, which is the cause of that TIMED_OUT functional job | 00:00 |
|---|---|---|
| dansmith | maybe we can discuss in the morning because I'm having a hard time understanding how it's working at all actually... | 00:00 |
| *** gyee has quit IRC | 02:11 | |
| *** rcernin has quit IRC | 02:37 | |
| *** rcernin has joined #openstack-glance | 02:48 | |
| *** irclogbot_2 has quit IRC | 03:05 | |
| *** irclogbot_2 has joined #openstack-glance | 03:07 | |
| *** Liang__ has joined #openstack-glance | 03:28 | |
| *** Liang__ has quit IRC | 03:40 | |
| *** Liang__ has joined #openstack-glance | 03:41 | |
| *** ratailor has joined #openstack-glance | 03:46 | |
| *** rcernin has quit IRC | 03:54 | |
| *** Liang__ has quit IRC | 04:07 | |
| *** Liang__ has joined #openstack-glance | 04:09 | |
| *** rcernin has joined #openstack-glance | 04:21 | |
| *** evrardjp has quit IRC | 04:33 | |
| *** evrardjp has joined #openstack-glance | 04:33 | |
| *** ratailor has quit IRC | 04:43 | |
| *** ratailor has joined #openstack-glance | 04:43 | |
| *** m75abrams has joined #openstack-glance | 04:56 | |
| *** udesale has joined #openstack-glance | 05:29 | |
| *** belmoreira has joined #openstack-glance | 05:56 | |
| abhishekk | dansmith, ack | 05:56 |
| *** belmoreira has quit IRC | 05:58 | |
| *** belmoreira has joined #openstack-glance | 06:14 | |
| *** nikparasyr has joined #openstack-glance | 06:31 | |
| *** udesale has quit IRC | 06:51 | |
| *** udesale has joined #openstack-glance | 06:52 | |
| *** priteau has joined #openstack-glance | 08:26 | |
| *** priteau has quit IRC | 08:42 | |
| *** priteau has joined #openstack-glance | 08:58 | |
| *** rcernin has quit IRC | 09:11 | |
| *** rcernin has joined #openstack-glance | 09:33 | |
| *** tkajinam has quit IRC | 09:36 | |
| * abhishekk be back on 1400 UTC | 09:48 | |
| *** udesale_ has joined #openstack-glance | 09:49 | |
| *** udesale has quit IRC | 09:52 | |
| *** rcernin has quit IRC | 09:56 | |
| *** priteau has quit IRC | 10:11 | |
| *** Liang__ has quit IRC | 10:31 | |
| *** k_mouza has joined #openstack-glance | 11:29 | |
| *** k_mouza has quit IRC | 11:30 | |
| *** rcernin has joined #openstack-glance | 11:31 | |
| *** k_mouza has joined #openstack-glance | 11:33 | |
| *** ratailor has quit IRC | 12:03 | |
| *** k_mouza has quit IRC | 12:11 | |
| *** Luzi has joined #openstack-glance | 12:55 | |
| *** Luzi has quit IRC | 13:00 | |
| openstackgerrit | Merged openstack/glance master: Add image_delete_property_atomic() helper https://review.opendev.org/743596 | 13:10 |
| abhishekk | dansmith, hey | 13:45 |
| dansmith | hey | 13:45 |
| abhishekk | I saw your ping early morning | 13:45 |
| dansmith | so, I have a bunch of data and have tried a bunch of things, but I have yet to actually come up with a plan | 13:45 |
| abhishekk | ok | 13:45 |
| dansmith | every time I see this manifest, | 13:45 |
| dansmith | it is in functional cleanup() waiting for the api worker master process to exit, | 13:46 |
| dansmith | but that child shows in the log no intent on shutting down, like it has not received the signal to exit | 13:46 |
| dansmith | rarely if I kill it, I find that your test was about to report failure for one of the SIGHUP operations, | 13:47 |
| abhishekk | ok, so why not it happens consistently? | 13:47 |
| dansmith | but often it reports success and it was just the functional stuff that was hanging, but only ever on your test | 13:47 |
| dansmith | well, the first thing is, I don't see where the functional stuff *ever* tries to stop the children | 13:48 |
| dansmith | so I must be missing something | 13:48 |
| abhishekk | hmm | 13:48 |
| abhishekk | that should be in actual wsgi module | 13:49 |
| dansmith | no, I mean the functional base class has self.process_pid of the api worker it starteed, | 13:50 |
| dansmith | and it looks like when stop() is called it will just waitpid() on that worker, | 13:50 |
| dansmith | I don't see where stop_kill ever gets set True for the regular case | 13:50 |
| abhishekk | right | 13:53 |
| dansmith | so when would we expect the api child to stop? | 13:53 |
| abhishekk | so in actual scenario, it should stop as soon as the task that child is processing is completed | 13:55 |
| abhishekk | somewhere in wait_for_fork method | 13:56 |
| dansmith | when we start cmd.api, it will run until it's told to stop | 13:56 |
| dansmith | if we waitpid() on it, we won't return until that pid decides to exist, which is usually because we have sent it a signal | 13:57 |
| dansmith | but I don't see where we ever do that (stop_kill=False by default, AFAICT) | 13:57 |
| abhishekk | yes, it is false | 13:57 |
| dansmith | so we won't send the master a signal when we stop(), | 13:57 |
| dansmith | so stop() will just hang until the process decides to exit.. what triggers it to know when to exit? | 13:58 |
| abhishekk | nothing atm | 13:58 |
| dansmith | so why do we ever not hang forever there? | 13:59 |
| dansmith | that's what I'm trying to understand | 13:59 |
| dansmith | for every functional test that starts a worker, and then calls stop() in cleanup, why do we *ever* not hang? | 13:59 |
| *** smcginnis has quit IRC | 14:01 | |
| abhishekk | yeah, that's what I am not able to understand as well | 14:01 |
| dansmith | okay | 14:02 |
| dansmith | so, the other abnormality I see, | 14:02 |
| dansmith | is that when this happens, the tail of the log file of the worker will show the two worker PIDs, and those pids are alive under the master, | 14:03 |
| dansmith | but when stop() runs, I've modified it to recursively walk the children of our master process, it finds two other children with (likely) older PIDs to kill | 14:04 |
| *** smcginnis has joined #openstack-glance | 14:05 | |
| dansmith | so I'm worried that the wsgi server's handling of recycling the children might be leaky and there are a couple of zombies left around from a previous HUP, | 14:05 |
| dansmith | and _that_ is why your reload test hangs sometimes when others don't, | 14:05 |
| dansmith | because the master process doesn't exit when asked because of those older children hanging around | 14:06 |
| dansmith | but I'm still working to try to gather more data on that | 14:06 |
| abhishekk | so this you are talking about actual sighup or functional test stuff? | 14:06 |
| dansmith | the actual sighup handling in the wsgi server, only just triggered by your functional test | 14:06 |
| dansmith | but causing the hang we see | 14:07 |
| dansmith | anyway, I want to file a bug for this to track this stuff.. there's not one already open for it I assume right? | 14:07 |
| dansmith | I'm not sure how long the functional tests have been hitting TIMED_OUT, | 14:07 |
| dansmith | but it's as long as I've been around, so a couple of months | 14:07 |
| *** rcernin has quit IRC | 14:08 | |
| abhishekk | there is no bug at the moment for timeout, but this functional timeout was since this cycle as the test was disabled earlier | 14:08 |
| dansmith | oh was it? why? | 14:10 |
| abhishekk | there was some issue with py3 earlier | 14:11 |
| *** m75abrams has quit IRC | 14:11 | |
| dansmith | okay | 14:13 |
| dansmith | https://bugs.launchpad.net/glance/+bug/1891190 | 14:14 |
| openstack | Launchpad bug 1891190 in Glance "test_reload() functional test causes hang and jobs TIMED_OUT" [Undecided,New] | 14:14 |
| abhishekk | also when this test was added, there was no reload method in functional/__init__.py | 14:15 |
| abhishekk | it was added later during when someone added windows support for glance | 14:15 |
| dansmith | okay | 14:16 |
| dansmith | not sure how that works (on the Posix one) either | 14:16 |
| dansmith | as part of my local testing, I've added this to the functional base: | 14:16 |
| dansmith | + self.addCleanup(self.cleanup, force=True) | 14:16 |
| dansmith | which tells cleanup to set stop_kill=True before calling stop() | 14:17 |
| abhishekk | ok | 14:17 |
| dansmith | clearly I'm still missing where this happens normally, but that is more explicit anyway | 14:17 |
| dansmith | there's really no reason we should ever call waitpid() if we are unsure if the signal has been sent | 14:17 |
| dansmith | so unless we find it... :) | 14:17 |
| abhishekk | yeah, will have a neat look as well | 14:18 |
| *** rcernin has joined #openstack-glance | 14:18 | |
| abhishekk | dansmith, IMO we should skip it again until we found the way to get it worked | 14:21 |
| dansmith | I'm a bit conflicted, because skipping tests that are flaky is a good way to keep them flaky and keep them from getting attention | 14:22 |
| dansmith | but...it's failing a lot, so idk | 14:22 |
| abhishekk | this will really cause problem for us during the milestone release when gate is heavily loaded and waiting time will be much higher | 14:23 |
| dansmith | yup | 14:23 |
| dansmith | AFAICT, that other spurious test fail has been squashed, but let me re-query for it and see if it has shown up at all | 14:23 |
| abhishekk | ack | 14:24 |
| *** rcernin has quit IRC | 14:24 | |
| *** rcernin has joined #openstack-glance | 14:27 | |
| *** rcernin has quit IRC | 14:31 | |
| dansmith | abhishekk: I see some failures since the fix landed for the copy revert lifecycle, but *only* on periodic tests, which seems weird | 14:39 |
| abhishekk | looking | 14:40 |
| abhishekk | I can see two failures as well | 14:42 |
| dansmith | in trying to track down this test_reload thing, I have run functional like thousands of times and have not hit the copy revert lifecycle thing ... | 14:43 |
| abhishekk | lets wait for couple of run of periodic cycles | 14:45 |
| dansmith | it seems massively better, if nothing else | 14:46 |
| abhishekk | yes | 14:46 |
| abhishekk | smcginnis, jokke, rosmaita please look when you have time, https://review.opendev.org/745407 | 14:53 |
| rosmaita | ack | 14:53 |
| smcginnis | Done | 14:58 |
| rosmaita | why don't those files show up in https://b1bdaec33567f033c580-b85c91a90386bf84f78798fbdb65380c.ssl.cf1.rackcdn.com/745407/2/check/tempest-full-py3/0c373be/controller/logs/etc/glance/ ? is it something about how devstack installs glance_store? | 14:59 |
| *** brtknr has left #openstack-glance | 15:00 | |
| abhishekk | smcginnis, thank you | 15:01 |
| abhishekk | rosmaita, yes it is | 15:04 |
| dansmith | rosmaita: I expect something is going on there because we install it from git and the uninstall it right after | 15:04 |
| rosmaita | ok, i didn't want to test locally, so was looking for a shortcut | 15:04 |
| *** nikparasyr has left #openstack-glance | 15:04 | |
| abhishekk | I also didn't tested locally, but verified it by cloning glance_store and running setup.py install | 15:05 |
| abhishekk | it is copying those files in /etc/glance | 15:06 |
| rosmaita | abhishekk: i didn't doubt you for a minute! | 15:20 |
| abhishekk | rosmaita, no, I am not saying that, I am saying that after this change I have also not check by installing devstack that it is copying files to /etc/glance | 15:21 |
| rosmaita | gotcha ... there may need to be some work on the devstack side if that tempest-full-py3 run is typical of a devstack install | 15:23 |
| abhishekk | ack | 15:24 |
| *** belmoreira has quit IRC | 15:25 | |
| * abhishekk going for dinner break | 15:33 | |
| *** rcernin has joined #openstack-glance | 15:40 | |
| *** rcernin has quit IRC | 15:44 | |
| * abhishekk back from dinner | 16:11 | |
| dansmith | well, I have had my "find and kill everything on stop" change running for two hours now without a failure, which is as long as I think I've seen it go | 16:15 |
| dansmith | so I still don't have a cause, and suspect something is actually not right with the wsgi thing, but this seems like legit care to have in the test base anyway | 16:16 |
| dansmith | we're clearly leaking processes: | 16:18 |
| dansmith | % pgrep -f cmd.api | wc -l | 16:18 |
| dansmith | 36 | 16:18 |
| dansmith | those leaked processes are re-parented to init, so they must be orphaned by a master that lost track of them i think | 16:20 |
| abhishekk | sounds like processes are never killed in tests as pointed by you | 16:27 |
| dansmith | it's more than that | 16:29 |
| dansmith | because I'm killing the whole process tree under our api master process | 16:29 |
| dansmith | and afaict, we only leak processes when we run test_reload(), | 16:30 |
| dansmith | which I suspect means we're losing track of the workers we're starting in the sighup | 16:30 |
| abhishekk | may be in teardown we should see how much workers are there in pre_pid and post_pid dict and kill them in all? | 16:31 |
| openstackgerrit | Merged openstack/glance master: Heartbeat the actual work of the task https://review.opendev.org/743426 | 16:32 |
| dansmith | I'm effectively doing that in cleanup() and not finding any extras to kill, yet ps shows an ever-growing number of cmd.api workers running | 16:32 |
| dansmith | after stopping the tests: | 16:33 |
| dansmith | % pgrep -f cmd.api | wc -l | 16:33 |
| dansmith | 31 | 16:33 |
| dansmith | the | 16:33 |
| openstackgerrit | Merged openstack/glance master: Update task message during import https://review.opendev.org/743427 | 16:34 |
| *** udesale_ has quit IRC | 16:34 | |
| abhishekk | strange, even after stopping the test workers are increasing | 16:35 |
| dansmith | yeah | 16:36 |
| dansmith | well, not increasing after stop, | 16:36 |
| dansmith | just never going away | 16:36 |
| dansmith | because they're re-parented to init because their master has died | 16:36 |
| abhishekk | ok | 16:36 |
| openstackgerrit | Merged openstack/glance_store master: Copy data files to glance upon installation https://review.opendev.org/745407 | 16:37 |
| openstackgerrit | Dan Smith proposed openstack/glance master: Increase aggression in functional base API worker management https://review.opendev.org/745708 | 16:41 |
| dansmith | abhishekk: ^ this is my test change, but am going to write another reload check to see if I can catch the leak | 16:41 |
| abhishekk | dansmith, ack, looking | 16:42 |
| dansmith | abhishekk: why are you shelling out to call /bin/kill by the way? | 16:42 |
| dansmith | because that creates children and isn't necessary to send signals of course | 16:42 |
| abhishekk | dansmith, could you please point out the location, it was written may be 4 years before | 16:43 |
| dansmith | abhishekk: like this: https://github.com/openstack/glance/blob/master/glance/tests/functional/test_reload.py#L140-L141 | 16:44 |
| abhishekk | dansmith, at that time I thought this was easy way to do it | 16:46 |
| dansmith | okay, maybe I'll change it to use signal directly and see if that changes behavior at all | 16:46 |
| *** zzzeek has quit IRC | 16:47 | |
| abhishekk | ack | 16:47 |
| *** k_mouza has joined #openstack-glance | 17:04 | |
| *** k_mouza has quit IRC | 17:09 | |
| dansmith | ugh, that patch failed the copy revert test :/ | 17:15 |
| abhishekk | :( | 17:18 |
| abhishekk | rosmaita, you were right | 17:21 |
| abhishekk | when devstack installs glance_store (which I tricked as we don't have new version of glance_store released) it copies those data files in /usr/local/etc/glance/ path and not to /etc/glance | 17:22 |
| abhishekk | dansmith, ^^ | 17:23 |
| dansmith | expected by default I think, | 17:23 |
| dansmith | not sure why nova and glance behave differently in devstack | 17:23 |
| dansmith | gmann: can you help? | 17:23 |
| abhishekk | then I guess we need to add that path in here https://github.com/openstack/glance_store/blob/master/etc/glance/rootwrap.conf#L7 | 17:25 |
| abhishekk | otherwise image will never be created if cinder backend is used in devstack | 17:25 |
| dansmith | abhishekk: is glance_store being installed from git or pypi? | 17:25 |
| abhishekk | i have verified that os-brick also copies data files to /usr/local/etc/os-brick | 17:26 |
| abhishekk | dansmith, from git | 17:26 |
| dansmith | abhishekk: are you installing with setup_develop? | 17:26 |
| abhishekk | dansmith, by me you mean devstack, then yes | 17:27 |
| dansmith | okay just looking at differences | 17:27 |
| abhishekk | and if in data_files if I begin with /etc/glance then it copies those files in /etc/glance | 17:30 |
| dansmith | right, but you see that other projects don't do that because it breaks portable install | 17:30 |
| abhishekk | yes, right | 17:32 |
| dansmith | abhishekk: looks like in the nova case at least, inc/rootwrap is copying the files to /etc/nova | 17:40 |
| dansmith | we do the same for paste.ini, | 17:40 |
| dansmith | and we just write the config directly | 17:40 |
| dansmith | so I think that's the magic | 17:40 |
| dansmith | https://zuul.opendev.org/t/openstack/build/f13715b6d142467bb1342fc001c939b7/log/controller/logs/devstacklog.txt#7070 | 17:40 |
| dansmith | the thing we call is: configure_rootwrap nova | 17:41 |
| dansmith | so maybe configure_rootwrap glance_store ? | 17:41 |
| abhishekk | ohh | 17:41 |
| abhishekk | imo it is different behavior for 3rd party libraries | 17:44 |
| dansmith | why? | 17:45 |
| abhishekk | as I said os-brick also copies data files to /usr/local/etc/os-brick | 17:45 |
| dansmith | nova and glance do too right? | 17:45 |
| abhishekk | may be gmann will help us | 17:45 |
| abhishekk | nova, cinder, glance do copy to /etc/* respective folder | 17:46 |
| dansmith | no, devstack does that | 17:47 |
| dansmith | the package itself should copy them, to /usr/local also | 17:47 |
| dansmith | if lib/glance is installing glance_store, then it should do the same for the glance_store configs | 17:48 |
| dansmith | (or configure things to look at /usr/local if possible) | 17:49 |
| abhishekk | even if I manually clone glance_store now and run setup.py install it copies those data files to /usr/local/glance directory | 17:52 |
| dansmith | right | 17:52 |
| dansmith | which devstack can copy to /etc/glance (or wherever it's needed) right? | 17:52 |
| abhishekk | i think it should, but it doesn't | 17:53 |
| dansmith | you have to _make_ it do that :) | 17:53 |
| abhishekk | :D | 17:54 |
| abhishekk | glance_store is using setup_dev_lib for installing | 17:56 |
| dansmith | right, same for nova | 18:00 |
| dansmith | then we run this: https://github.com/openstack/devstack/blob/master/lib/nova#L238 | 18:00 |
| dansmith | which copies things to /etc/nova/rootwrap | 18:00 |
| abhishekk | looking | 18:02 |
| abhishekk | may be I need call same function in devstack cinder support patch | 18:04 |
| dansmith | abhishekk: [10:41:07] <dansmith>so maybe configure_rootwrap glance_store ? | 18:04 |
| dansmith | :) | 18:04 |
| abhishekk | ohh, I missed that :D | 18:04 |
| abhishekk | Ahh, I disconnected 5-10 minutes because of VPN issue in between :D | 18:05 |
| gmann | abhishekk: dansmith yeah you need to write the copy things in devstack to whatever location you want to /etc/<service> | 18:07 |
| abhishekk | dansmith, gmann last doubt | 18:08 |
| gmann | you can do in job also | 18:08 |
| abhishekk | glance_store is library | 18:08 |
| gmann | or devstack | 18:08 |
| dansmith | ack, I was thinking devstack should be doing the setup.py install with --prefix , but yeah looks like devstack copies manually | 18:08 |
| gmann | yeah | 18:08 |
| abhishekk | so if I avoid cloning of glance_store and allowed pip to install it then configure_rootwrap glance_store will not found glance_store repo in my local env | 18:08 |
| dansmith | abhishekk: yeah, so maybe don't use configure_rootwrap and copy from /usr/local/etc/glance_store, | 18:09 |
| dansmith | or make configure_rootwrap_from_installed() or something | 18:09 |
| abhishekk | yeah | 18:09 |
| abhishekk | will do it :D | 18:10 |
| dansmith | abhishekk: how does it work for os_brick? maybe you can include the glance_store rootwrap stuff from /usr/local | 18:12 |
| dansmith | smcginnis: rosmaita ^ | 18:12 |
| abhishekk | dansmith, I haven't found os-brick rootwrap reference in cinder.conf | 18:13 |
| dansmith | yeah that's what I'm wondering | 18:13 |
| dansmith | anyway, copying will make it included in the log dump, which is good too | 18:13 |
| abhishekk | +1 I will copy it | 18:14 |
| smcginnis | cinder and nova have had to maintain duplicate rootwrap.conf files. We run nearly everything through privsep now, so no longer need to sync those changes. | 18:14 |
| smcginnis | It caused a big problem with needing to lock step changes between projects. | 18:14 |
| dansmith | smcginnis: | 18:16 |
| dansmith | smcginnis: "we" being "brick" in this case? | 18:16 |
| dansmith | meaning, brick uses privsep? | 18:16 |
| smcginnis | Cinder and os-brick. | 18:16 |
| smcginnis | https://opendev.org/openstack/os-brick/src/branch/master/os_brick/privileged/rootwrap.py#L22 | 18:17 |
| dansmith | okay so maybe the bit of devstack that copied the brick rootwrap config into place has been removed to avoid the sync? | 18:18 |
| smcginnis | Maybe? | 18:19 |
| dansmith | or are you saying that nova and cinder included the things brick needed in their own config? | 18:19 |
| smcginnis | That's how it was always done. And probably is still the case for most existing things. | 18:19 |
| dansmith | oh wow okay | 18:19 |
| jokke | iirc that's how we used to do it (I think it was still at the time when the cinder store was there but not really usable). I just recall some discussion around "those files should not be shipped with the service but with the lib" | 18:22 |
| *** k_mouza has joined #openstack-glance | 18:23 | |
| * dansmith nods | 18:30 | |
| abhishekk | there is another catch, devstack doesn't copy any of the data file to glance_store to anywhere :/ | 18:38 |
| abhishekk | it copies those file to /usr/local/etc/glance only if I use setup.py install | 18:38 |
| dansmith | abhishekk: you don't mean devstack, I assume, you mean that when you install with pip you don't see the data files? | 18:41 |
| dansmith | I think pip puts those files in the egg, IIRC | 18:42 |
| abhishekk | didn't found those in the eg | 18:43 |
| abhishekk | didn't found those in the egg | 18:43 |
| dansmith | because it's being installed with -e maybe | 18:46 |
| abhishekk | may be I should move back to https://review.opendev.org/#/c/743800/1/lib/glance@47 | 18:47 |
| abhishekk | and then line #148 | 18:48 |
| dansmith | I think that is the wrong move.. agree gmann ? | 18:48 |
| dansmith | glance forcing LIBS_FROM_GIT+=glance_store seems like a -2 to me | 18:48 |
| abhishekk | what is the use of this variable then? | 18:49 |
| dansmith | it should be controlled by the job, not a devstack module | 18:50 |
| dansmith | and, you should always be testing glance against the released version of glance_store, | 18:50 |
| dansmith | unless you're testing a depends-on | 18:50 |
| abhishekk | hmm | 18:50 |
| dansmith | and if you override it, you will break the ability to control that, AFAIK | 18:51 |
| dansmith | abhishekk: have you released glance_store yet? I would expect that pip installing from pypi will do a full install and not a -e one, such that the files will be installed, | 18:51 |
| dansmith | like you see from os-brick | 18:51 |
| abhishekk | I haven't released glance_store yet | 18:52 |
| dansmith | if those files aren't installed when installed from pip normally, then nobody can use glance_store from pip, so we might as well delete it from pypi :) | 18:52 |
| abhishekk | I can verify it by removing os-brick and reinstalling it again using pip | 18:54 |
| jokke | abhishekk: you won't be seeing those data_files coming from pip before the glance_store with that setup.cfg change has been merged | 18:54 |
| abhishekk | jokke, that change is merged | 18:54 |
| dansmith | it merged didn't it? | 18:54 |
| jokke | s/merged/released/ | 18:54 |
| abhishekk | ack | 18:54 |
| jokke | realized my error just when I hit enter ;) | 18:54 |
| dansmith | oh abhishekk are you installing from pypi? I thought you were installing from local | 18:54 |
| abhishekk | was disconnected in between | 18:59 |
| abhishekk | I missed conversation from <dansmith> oh abhishekk are you installing from pypi? I thought you were installing from local | 19:01 |
| dansmith | that's all I said | 19:01 |
| abhishekk | ok | 19:01 |
| dansmith | it seems like jokke thinks you're installing glance_store from pypi expecting the data_files change to be there, even though you haven't released it | 19:01 |
| dansmith | which obviously won't work, | 19:01 |
| dansmith | but I was assuming you're installing from the local git tree with pip, like devstack does | 19:02 |
| dansmith | if devstack does an install from git of a package, and uses -e, I don't think that it will install the data files, | 19:03 |
| dansmith | but if you install a released package from pypi it should | 19:03 |
| dansmith | I imagine devstack only does the from-git -e install if it has it in /opt/stack, | 19:03 |
| dansmith | so you should be able to either grab it from /usr/local if installed from pip release, or from /opt/stack if installed from pip local | 19:04 |
| abhishekk | dansmith, neither of it happening | 19:10 |
| abhishekk | when I try to installing from local git tree it runs command like; | 19:10 |
| abhishekk | pip_install -e '/opt/stack/glance_store[cinder]' | 19:11 |
| abhishekk | setup_develop /opt/stack/glance_store cinder | 19:12 |
| abhishekk | setup_develop /opt/stack/glance_store cinder | 19:12 |
| abhishekk | _setup_package_with_constraints_edit /opt/stack/glance_store -e cinder | 19:12 |
| dansmith | right, like I said, -e won't install the data_files | 19:12 |
| dansmith | because it's installing "in place" anyway | 19:12 |
| dansmith | so in that case, you'd get them from /opt/stack/glance_store | 19:13 |
| dansmith | if you release and install from pypi it *should* install them to /usr/local | 19:13 |
| dansmith | like you see from os_brick | 19:13 |
| abhishekk | dansmith, ack | 19:15 |
| abhishekk | so I need to tweak the code like that | 19:15 |
| dansmith | the devstack code, yes | 19:17 |
| dansmith | if glance_store is in LIBS_FROM_GIT, then configure_rootwrap should work, if not, then copy from /usr/local | 19:17 |
| *** k_mouza has quit IRC | 19:20 | |
| *** k_mouza has joined #openstack-glance | 19:26 | |
| *** k_mouza has joined #openstack-glance | 19:27 | |
| *** k_mouza has quit IRC | 19:32 | |
| abhishekk | dansmith, Configure cinder store for glance https://review.opendev.org/743800 | 19:40 |
| abhishekk | dansmith, if glance_store is in LIBS_FROM_GIT, then configure_rootwrap will not work as it accepts project_name as only param and then it will copy files from etc/glance_store to /etc/glance_store not /etc/glance | 19:41 |
| *** rcernin has joined #openstack-glance | 19:41 | |
| dansmith | abhishekk: ack | 19:43 |
| abhishekk | dansmith, +1 for comments, easy way :D | 19:45 |
| dansmith | :) | 19:45 |
| *** rcernin has quit IRC | 19:45 | |
| dansmith | I think we're highlighting that glance_store having rootwrap rules that have to be installed into /etc/glance's directory is a little confusing and weird | 19:46 |
| dansmith | it's definitely an unusual situation I think | 19:47 |
| abhishekk | yes | 19:48 |
| openstackgerrit | Abhishek Kekane proposed openstack/glance master: PoC for lazy loading cinder store https://review.opendev.org/742373 | 19:53 |
| * abhishekk signing out for the day | 19:57 | |
| jokke | dansmith: I don't think they do ... one can put the rules where ever they want and point the rootwrap.conf to right path | 19:58 |
| dansmith | jokke: that's what I was asking above.. if we can configure glance to look at both the glance rootwrap.d and the glance_store rootwrap.d that seems much more ideal than having to meld them | 19:59 |
| *** gyee has joined #openstack-glance | 19:59 | |
| jokke | dansmith: there is no glance rootwrap ... it's coming from glance store | 20:00 |
| jokke | https://github.com/openstack/glance_store/blob/master/etc/glance/rootwrap.conf#L7 these are just the default values where the rules will be looked at but that line can have any path where you want to put them | 20:01 |
| dansmith | oh I thought we still needed it for some of the file stuff, but if that's all really in glance_store then.. much easier | 20:01 |
| jokke | yeah, literally only thing using it is the cinder driver in glance_store ... we have no need for it otherwise | 20:01 |
| dansmith | I'm surprised we wouldn't use brick for all the cinder root things we need to do anyway... | 20:02 |
| jokke | Don't even get me started, but we can't as brick doesn't support most of the cinder backends ;) | 20:03 |
| jokke | s/backends/backend protocols/ | 20:03 |
| dansmith | um, okay :) | 20:04 |
| jokke | There is bowl of special sphagetti sauce in the cinder driver that in perfect world should be all handled by brick | 20:05 |
| gmann | dansmith: abhishekk : note, all lib/projects defined as 'required_projects' in zuul job are by default added to LIBS_FROM_GIT by devstack. | 20:10 |
| gmann | if you want to test the released version then we need to make sure glance_store is not in 'required_projects' | 20:11 |
| dansmith | gmann: yep, ack | 20:11 |
| gmann | which is case now in our jobs. | 20:11 |
| dansmith | gmann: abhishekk was making lib/glance force glance_store into LIBS_FROM_GIT all the time, which I think is -2 right? | 20:11 |
| gmann | yeah, we should not hard code and let job decide. | 20:12 |
| dansmith | ++ thought so, just making sure :) | 20:12 |
| gmann | sorry for being late responsive. i am doing downstream things today :) | 20:13 |
| dansmith | gmann: np, thanks for following up :) | 20:14 |
| *** openstackgerrit has quit IRC | 20:52 | |
| -openstackstatus- NOTICE: The openstackgerrit IRC bot (gerritbot) will be offline for a short period while we redeploy it on a new server | 20:52 | |
| *** zzzeek has joined #openstack-glance | 21:38 | |
| *** gregwork has joined #openstack-glance | 22:45 | |
| *** rcernin has joined #openstack-glance | 22:45 | |
| *** tkajinam has joined #openstack-glance | 22:54 | |
| *** openstackgerrit has joined #openstack-glance | 23:49 | |
| openstackgerrit | Dan Smith proposed openstack/glance master: Make wait_for_fork() more robust against infinite deadlock https://review.opendev.org/745763 | 23:49 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!