| *** shu-mutow has joined #openstack-sdks | 00:12 | |
| *** markvoelker has quit IRC | 00:19 | |
| *** markvoelker has joined #openstack-sdks | 00:47 | |
| *** JudeC has quit IRC | 00:48 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/os-service-types master: Updated from OpenStack Service Type Authority https://review.openstack.org/585062 | 00:50 |
|---|---|---|
| openstackgerrit | Merged openstack/python-openstackclient master: Do not require port argument when updating floating IP https://review.openstack.org/575057 | 00:53 |
| *** chenyb4 has joined #openstack-sdks | 01:02 | |
| *** slaweq has joined #openstack-sdks | 01:11 | |
| *** slaweq has quit IRC | 01:15 | |
| *** edmondsw has joined #openstack-sdks | 01:27 | |
| *** edmondsw has quit IRC | 01:31 | |
| *** mhen has quit IRC | 01:40 | |
| *** mhen has joined #openstack-sdks | 01:41 | |
| *** slaweq has joined #openstack-sdks | 02:11 | |
| *** slaweq has quit IRC | 02:16 | |
| *** dave-mccowan has quit IRC | 02:31 | |
| *** samP has joined #openstack-sdks | 02:57 | |
| *** samP has quit IRC | 02:59 | |
| *** samP has joined #openstack-sdks | 03:05 | |
| *** samP has quit IRC | 03:10 | |
| *** samP has joined #openstack-sdks | 03:13 | |
| *** samP has quit IRC | 03:15 | |
| *** edmondsw has joined #openstack-sdks | 03:15 | |
| *** edmondsw has quit IRC | 03:19 | |
| *** samP has joined #openstack-sdks | 03:19 | |
| *** samP has quit IRC | 03:26 | |
| *** samP has joined #openstack-sdks | 03:39 | |
| *** samP has quit IRC | 03:39 | |
| *** samP has joined #openstack-sdks | 03:46 | |
| *** samP has quit IRC | 03:48 | |
| *** samP has joined #openstack-sdks | 03:55 | |
| *** samP has quit IRC | 03:55 | |
| *** samP has joined #openstack-sdks | 03:57 | |
| *** samP has quit IRC | 03:58 | |
| *** samP has joined #openstack-sdks | 04:03 | |
| *** samP has quit IRC | 04:05 | |
| *** samP has joined #openstack-sdks | 04:09 | |
| *** samP has quit IRC | 04:10 | |
| *** slaweq has joined #openstack-sdks | 04:11 | |
| *** JudeC has joined #openstack-sdks | 04:15 | |
| *** slaweq has quit IRC | 04:16 | |
| *** samP has joined #openstack-sdks | 04:16 | |
| *** samP has quit IRC | 04:16 | |
| *** samP has joined #openstack-sdks | 04:22 | |
| *** samP has quit IRC | 04:23 | |
| *** samP has joined #openstack-sdks | 04:28 | |
| *** samP has quit IRC | 04:28 | |
| *** samP has joined #openstack-sdks | 04:29 | |
| *** samP has quit IRC | 04:29 | |
| *** samP has joined #openstack-sdks | 04:38 | |
| *** samP has quit IRC | 04:39 | |
| *** e0ne has joined #openstack-sdks | 04:40 | |
| *** e0ne has quit IRC | 04:40 | |
| *** samP has joined #openstack-sdks | 04:44 | |
| *** samP has quit IRC | 04:45 | |
| *** honza has quit IRC | 04:47 | |
| *** honza has joined #openstack-sdks | 04:48 | |
| *** tellesnobrega has quit IRC | 04:49 | |
| *** honza is now known as Guest56850 | 04:49 | |
| *** tellesnobrega has joined #openstack-sdks | 04:49 | |
| *** samP has joined #openstack-sdks | 04:52 | |
| *** e0ne has joined #openstack-sdks | 04:53 | |
| *** samP has quit IRC | 04:53 | |
| *** e0ne has quit IRC | 04:55 | |
| *** samP has joined #openstack-sdks | 04:57 | |
| *** samP has quit IRC | 04:58 | |
| *** samP has joined #openstack-sdks | 05:03 | |
| *** edmondsw has joined #openstack-sdks | 05:03 | |
| *** samP has quit IRC | 05:04 | |
| *** edmondsw has quit IRC | 05:08 | |
| *** JudeC has quit IRC | 05:09 | |
| *** samP has joined #openstack-sdks | 05:10 | |
| *** samP has quit IRC | 05:10 | |
| *** slaweq has joined #openstack-sdks | 05:11 | |
| *** samP has joined #openstack-sdks | 05:12 | |
| *** samP has quit IRC | 05:12 | |
| *** samP has joined #openstack-sdks | 05:14 | |
| *** slaweq has quit IRC | 05:15 | |
| *** samP has quit IRC | 05:15 | |
| *** Luzi has joined #openstack-sdks | 05:45 | |
| openstackgerrit | Josephine Seifert proposed openstack/python-openstackclient master: Don't sent disk_over_commit if nova api > 2.24 https://review.openstack.org/582334 | 05:54 |
| openstackgerrit | Josephine Seifert proposed openstack/python-openstackclient master: [WIP] osc-included image signing (using openstacksdk) https://review.openstack.org/580086 | 05:55 |
| *** slaweq has joined #openstack-sdks | 06:11 | |
| *** slaweq has quit IRC | 06:16 | |
| *** slaweq has joined #openstack-sdks | 07:11 | |
| *** JudeC has joined #openstack-sdks | 07:12 | |
| *** slaweq has quit IRC | 07:12 | |
| *** slaweq has joined #openstack-sdks | 07:16 | |
| *** thanhnb has joined #openstack-sdks | 07:24 | |
| thanhnb | hello | 07:26 |
| thanhnb | sorry about my english. | 07:26 |
| thanhnb | i want help about "Connection" in openstack sdk. | 07:27 |
| thanhnb | i has read in docs. It said "Using an existing authenticated keystoneauth1.session.Session, such as might exist inside of an OpenStack service operational context" | 07:27 |
| thanhnb | How can i use "keystoneauth1.session.Session" in this situation. | 07:28 |
| *** markvoelker has quit IRC | 07:29 | |
| cmurphy | thanhnb: you can create a keystoneauth session following these docs https://docs.openstack.org/keystoneauth/latest/using-sessions.html | 07:33 |
| thanhnb | i has created it | 07:34 |
| thanhnb | but it seem not work | 07:34 |
| thanhnb | http://paste.openstack.org/show/726497/ | 07:36 |
| thanhnb | this is how i do it | 07:36 |
| thanhnb | am i right? | 07:36 |
| cmurphy | thanhnb: it seems okay to me, what problem are you having? | 07:37 |
| *** gildub has joined #openstack-sdks | 07:38 | |
| thanhnb | http://paste.openstack.org/show/726498/ | 07:39 |
| thanhnb | this is bug | 07:39 |
| thanhnb | http://paste.openstack.org/show/726499/ this is code when i has bug | 07:40 |
| *** kimamisa has joined #openstack-sdks | 07:42 | |
| thanhnb | i think the session is ok, but the connection doesn't create | 07:42 |
| cmurphy | thanhnb: what version of openstacksdk and keystoneauth do you have? | 07:46 |
| cmurphy | I tried your code and it works for me so it might be a bug in one of the libraries | 07:47 |
| thanhnb | keystoneauth1==3.10.0 and openstacksdk==0.17.0 | 07:49 |
| thanhnb | what is your version of openstacksdk and keystoneauth1? | 07:50 |
| cmurphy | thanhnb: same versions | 07:53 |
| cmurphy | hmm :/ | 07:53 |
| thanhnb | :( | 07:53 |
| thanhnb | let me try another venv. :( | 07:56 |
| thanhnb | cmurphy- so anyway, thanks you :D | 07:58 |
| cmurphy | thanhnb: no problem but I'm still not sure what's going on :) | 08:02 |
| thanhnb | i has created another venv | 08:02 |
| thanhnb | but it still error @@ | 08:03 |
| thanhnb | same with this. | 08:03 |
| thanhnb | can you send me your code @@ | 08:03 |
| cmurphy | thanhnb: i'm using your same code just with the password and user id changed | 08:04 |
| thanhnb | :( | 08:04 |
| cmurphy | actually now i've tried it on a different system and it's giving me a completely different error, "Auth plugin requires parameters which were not given: auth_url" so i'm even more confused :) | 08:05 |
| thanhnb | i has openstack in centos 7 | 08:06 |
| thanhnb | and run code in ubuntu18 @@ | 08:06 |
| thanhnb | i has same bug "not given: auth_url" when i dont declared "auth_url" | 08:07 |
| *** jpich has joined #openstack-sdks | 08:07 | |
| thanhnb | my openstack is queen | 08:08 |
| thanhnb | @@ | 08:09 |
| *** Luzi has quit IRC | 08:11 | |
| thanhnb | i have tried it in different node, but it still error | 08:13 |
| *** gkadam has joined #openstack-sdks | 08:13 | |
| cmurphy | thanhnb: aha i reproduced your problem | 08:15 |
| *** e0ne has joined #openstack-sdks | 08:15 | |
| thanhnb | wow :D | 08:15 |
| cmurphy | when I first tried it I had credentials set in OS_* environment variables | 08:15 |
| cmurphy | if i unset OS_PASSWORD then the problem occurs | 08:16 |
| cmurphy | based on that doc i think that's probably not what's intended, want to file a bug? | 08:17 |
| thanhnb | so i need "OS_PASSWORD" in my environment? | 08:17 |
| *** shu-mutow has quit IRC | 08:18 | |
| thanhnb | what is "want to file a bug" , i dont understand @@ | 08:18 |
| thanhnb | i just newbie in openstack | 08:18 |
| *** JudeC has quit IRC | 08:19 | |
| cmurphy | thanhnb: i mean to say, could you report this bug in the bug tracker? I believe you can do that here https://storyboard.openstack.org/#!/project/972 | 08:20 |
| *** gildub has quit IRC | 08:20 | |
| thanhnb | yes, i think i can do this. | 08:20 |
| thanhnb | :) | 08:20 |
| cmurphy | :) | 08:21 |
| cmurphy | cc mordred ^ | 08:21 |
| *** Luzi has joined #openstack-sdks | 08:26 | |
| *** ralonsoh has joined #openstack-sdks | 08:31 | |
| *** dtantsur|afk is now known as dtantsur | 08:33 | |
| *** cdent has joined #openstack-sdks | 08:49 | |
| *** dmellado has quit IRC | 09:03 | |
| *** markvoelker has joined #openstack-sdks | 09:33 | |
| *** stephenfin has quit IRC | 09:47 | |
| *** stephenfin has joined #openstack-sdks | 09:49 | |
| *** markvoelker has quit IRC | 10:04 | |
| *** thanhnb has quit IRC | 10:14 | |
| *** dmellado has joined #openstack-sdks | 10:24 | |
| *** e0ne has quit IRC | 10:27 | |
| *** chenyb4 has quit IRC | 10:31 | |
| *** e0ne has joined #openstack-sdks | 10:32 | |
| *** edmondsw has joined #openstack-sdks | 10:35 | |
| *** edmondsw has quit IRC | 10:40 | |
| *** e0ne has quit IRC | 10:40 | |
| *** cdent has quit IRC | 10:55 | |
| *** markvoelker has joined #openstack-sdks | 11:01 | |
| *** markvoelker has quit IRC | 11:34 | |
| *** cdent has joined #openstack-sdks | 11:53 | |
| *** edmondsw has joined #openstack-sdks | 11:57 | |
| *** e0ne has joined #openstack-sdks | 12:07 | |
| *** kimamisa has quit IRC | 12:12 | |
| *** markvoelker has joined #openstack-sdks | 12:12 | |
| mordred | cmurphy: reading | 12:15 |
| mordred | oh poo. thanhnb is gone | 12:16 |
| mordred | cmurphy: fwiw, I think this is a doc bug - it is implying that the above is something that people should do | 12:20 |
| mordred | the whole "Using an existing authenticated keystoneauth1.session.Session, such as might..." is really meant for inside of nova or something else where keystone_middleware has already provided a Session - I don't expect it to be a pattern people use in any other context | 12:25 |
| *** mriedem has joined #openstack-sdks | 12:28 | |
| *** fabian_ has joined #openstack-sdks | 12:45 | |
| *** chenyb4 has joined #openstack-sdks | 12:51 | |
| *** cdent has quit IRC | 12:55 | |
| *** r-mibu has joined #openstack-sdks | 13:04 | |
| cmurphy | mordred: but in theory shouldn't that work? | 13:07 |
| mordred | yes - in theory it should work - so I'm sure there is a bug in there somewhere | 13:07 |
| mordred | but the first bug is that doing it that way is the hardest possible way to use openstacksdk, so if someone is trying to get started and that's the way they're trying to get started, the docs are leading them astray | 13:08 |
| mordred | (I'm especially curious as to why you having OS_PASSWORD set in the env had any impact at all due to the way session is saved and reused in that workflow) | 13:09 |
| cmurphy | fair enough, it led me astray too | 13:09 |
| mordred | like - that session has an Auth plugin already, so openstacksdk should not be doing _anything_ with auth information | 13:09 |
| cmurphy | exactly | 13:09 |
| cmurphy | that is what was surprising | 13:09 |
| mordred | ++ | 13:10 |
| *** dave-mccowan has joined #openstack-sdks | 13:11 | |
| *** dave-mcc_ has joined #openstack-sdks | 13:20 | |
| openstackgerrit | Eric Fried proposed openstack/os-service-types master: Switch to stestr https://review.openstack.org/585349 | 13:21 |
| openstackgerrit | Eric Fried proposed openstack/os-service-types master: Updated from OpenStack Service Type Authority https://review.openstack.org/585062 | 13:22 |
| *** dave-mccowan has quit IRC | 13:22 | |
| *** mriedem is now known as mriedem_away | 13:24 | |
| *** cdent has joined #openstack-sdks | 13:40 | |
| openstackgerrit | Eric Fried proposed openstack/os-service-types master: Switch to stestr https://review.openstack.org/585349 | 13:44 |
| *** Luzi has quit IRC | 13:53 | |
| *** mriedem_away is now known as mriedem | 14:11 | |
| *** fabian_ has quit IRC | 14:13 | |
| *** mriedem1 has joined #openstack-sdks | 14:18 | |
| *** chenyb4 has quit IRC | 14:18 | |
| openstackgerrit | Eric Fried proposed openstack/os-service-types master: Switch to stestr https://review.openstack.org/585349 | 14:19 |
| *** mriedem has quit IRC | 14:21 | |
| *** mriedem1 is now known as mriedem | 14:24 | |
| *** chenyb4 has joined #openstack-sdks | 14:30 | |
| *** ralonsoh_ has joined #openstack-sdks | 14:31 | |
| *** ralonsoh has quit IRC | 14:33 | |
| *** samP has joined #openstack-sdks | 14:38 | |
| *** ralonsoh has joined #openstack-sdks | 14:57 | |
| *** purplerbot has quit IRC | 14:57 | |
| *** purplerbot has joined #openstack-sdks | 14:57 | |
| *** ralonsoh_ has quit IRC | 15:01 | |
| *** purplerbot has quit IRC | 15:08 | |
| *** purplerbot has joined #openstack-sdks | 15:08 | |
| openstackgerrit | Artom Lifshitz proposed openstack/python-openstackclient master: Don't sent disk_over_commit if nova api > 2.24 https://review.openstack.org/582334 | 15:09 |
| *** chenyb4 has quit IRC | 15:16 | |
| openstackgerrit | Monty Taylor proposed openstack/os-service-types master: Updated from OpenStack Service Type Authority https://review.openstack.org/585062 | 15:17 |
| samueldmq | heya | 15:44 |
| samueldmq | how many clouds have we validated shade (or the upper abstraction layer) against? | 15:45 |
| samueldmq | mordred: ^ | 15:45 |
| mordred | samueldmq: many :) | 15:45 |
| samueldmq | mordred: nice, I'd like to do some formal validation and report results | 15:46 |
| mordred | samueldmq: all of the openstack public clouds- and many of the private (I don't have a count) | 15:46 |
| samueldmq | I was thinking about all those in the passport program to start | 15:46 |
| samueldmq | oh that's awesome ... | 15:46 |
| mordred | samueldmq: sweet. you may also want to talk to melvinhillsman - the openlab folks have been working on setting up jobs to run openstacksdk tests against actual public cloud accounts | 15:46 |
| rcarrillocruz | yah, openlab, that | 15:47 |
| * rcarrillocruz goes back to his cave | 15:47 | |
| *** samP has quit IRC | 15:47 | |
| samueldmq | rcarrillocruz: o/ | 15:47 |
| rcarrillocruz | o/ | 15:47 |
| samueldmq | mordred: awesome, I met him at last summit, will talk to him | 15:47 |
| openstackgerrit | Monty Taylor proposed openstack/os-service-types master: Allow passing in service types with _ in them https://review.openstack.org/585410 | 15:48 |
| openstackgerrit | Monty Taylor proposed openstack/os-service-types master: Add flag for returning unofficial types https://review.openstack.org/585411 | 15:48 |
| mordred | samueldmq: woot. | 15:48 |
| samueldmq | mordred: where is shade code in the new -sdk? | 15:49 |
| mordred | samueldmq: although - for your case, it's also possible that doing the formal validation by hand would be fine - since that's different from making sure to run validation on every patch | 15:49 |
| mordred | samueldmq: openstack/cloud | 15:49 |
| samueldmq | mordred: ++ exactly... just want to validate the tool works | 15:50 |
| samueldmq | also, I don't need/want to be exhaustive or too complex | 15:50 |
| mordred | samueldmq: and it's all attached to the Connection object - so as a user if you do "conn - openstack.connect(cloud='vexxhost')" - you can use that conn object same as the OpenStackCloud object from shade | 15:50 |
| *** samP has joined #openstack-sdks | 15:50 | |
| mordred | samueldmq: ++ | 15:50 |
| samueldmq | validating a core set of functionalities should be fine (whatever be a core set: vm managemnt?) | 15:50 |
| mordred | samueldmq: https://review.openstack.org/#/c/582459/ is a patch you might want to look at | 15:50 |
| mordred | samueldmq: it's a patch to allow running our ansible tests against a public cloud that is not devstack | 15:51 |
| mordred | Shrews: ^^ speaking of - you might want to check out that patch too | 15:51 |
| samueldmq | mordred: awesome | 15:51 |
| samueldmq | do all the tests normally pass? | 15:51 |
| samueldmq | I wouldn't be surprised otherwise... there may be some services that do not exist in some cases, for ex | 15:52 |
| mordred | samueldmq: well - our ansible tests have a pretty small surface area | 15:52 |
| mordred | I'd love to be able to run our functional tests against a cloud - and hopefully we'll also get that working at some point | 15:53 |
| samueldmq | mordred: how do you see that? | 15:53 |
| mordred | samueldmq: and I think it's ok for the test suite to skip some tests if they don't have a service | 15:53 |
| samueldmq | reimplement with ansible | 15:53 |
| samueldmq | or wrap the existing tests somehow to make it owrk | 15:53 |
| mordred | nah - it's mostly just about reworking base fixtures | 15:53 |
| mordred | in some clouds we may not have admin, for instance, so making a new project and testing keystone admin functions should all be skipped | 15:54 |
| samueldmq | to allow oyu input the cloud to talk against | 15:54 |
| mordred | but we could still test that booting a vm works | 15:54 |
| samueldmq | when running tests right | 15:54 |
| *** samP has quit IRC | 15:54 | |
| mordred | the tests already will skip if the service doesn't exist - then we have a devstack-specific test that responds to env vars and fails if an envvar asserts a service should exist but the shade tests don't find it | 15:55 |
| mordred | that way we can make sure we don't accidentally fail open | 15:55 |
| mordred | i the gate | 15:55 |
| mordred | so we could probably expand on that somehow | 15:55 |
| Shrews | mordred: lgtm | 15:56 |
| mordred | to allow manual running against public clouds to be more flexible without sacrificing gate stuff | 15:56 |
| mordred | Shrews: woot | 15:56 |
| samueldmq | mordred: how are those vars set up then? | 15:56 |
| samueldmq | I assume that cn be automatic by looking up the service catalog | 15:56 |
| samueldmq | but would require some work | 15:56 |
| mordred | well - "has_service('network')" uses the service catalog | 15:56 |
| mordred | but the env var is defined in the test job | 15:56 |
| mordred | so the test job (which sets up the services in the first place) also sets vars that say "I believe I set up magnum" | 15:57 |
| mordred | then the shade test says "the test config expects magnum, do I see it in the catalog" | 15:57 |
| samueldmq | ah makes sense | 15:57 |
| samueldmq | but none of that apply to manual run | 15:57 |
| mordred | exactly | 15:57 |
| samueldmq | also, runnig against real public clouds wouldn't require any setup | 15:58 |
| mordred | that's the theory :) | 15:58 |
| samueldmq | openstack/ocnfig that is | 15:58 |
| mordred | in practice, I think there are still a good amount of code in fixtures that assumes we're running against a devstack | 15:58 |
| mordred | and could be make more smarter, we just haven't had the time yet | 15:58 |
| Shrews | yeah | 15:58 |
| samueldmq | "whatever that can happen will happen" someone said | 15:58 |
| mordred | ++ | 15:58 |
| samueldmq | and then shade exists :-) | 15:58 |
| samueldmq | kk I have some info to mull over | 15:59 |
| *** samP has joined #openstack-sdks | 15:59 | |
| samueldmq | I'm happy I got an old laptop running linux again, was bored using windows all the time for company stuff | 16:00 |
| Shrews | somewhere in shade i had a script that allowed us to run those tests against another (non-devstack) cloud | 16:00 |
| samueldmq | which means I am expecting to start having fun around here | 16:00 |
| mordred | samueldmq: yay! | 16:00 |
| samueldmq | Shrews: that'd be a good start | 16:00 |
| mordred | Shrews: yeah- I remember that ... | 16:00 |
| samueldmq | \o/ | 16:00 |
| Shrews | oh, even tox support for that | 16:00 |
| Shrews | tox -e ansible -- -c cloudX [TAG ...] | 16:00 |
| Shrews | how clever i was | 16:01 |
| *** jpich has quit IRC | 16:01 | |
| samueldmq | and that looks up on the regular /opt/openstack/clouds.yaml? | 16:01 |
| Shrews | yeah | 16:01 |
| samueldmq | or is there a special thing that was setup for devstack | 16:01 |
| samueldmq | kk | 16:01 |
| Shrews | and the tags were to selectively choose which things you wanted to test (in case the cloud didn't support something) | 16:02 |
| samueldmq | gotcha | 16:02 |
| Shrews | looks like we carried that over to sdk, but i haven't tested it there | 16:03 |
| samueldmq | I'll take a look to get familiar | 16:03 |
| samueldmq | and see what works and what doesn't | 16:03 |
| Shrews | \o/ | 16:04 |
| samueldmq | :D | 16:04 |
| *** e0ne has quit IRC | 16:10 | |
| *** dtantsur is now known as dtantsur|afk | 16:39 | |
| *** JudeC has joined #openstack-sdks | 16:51 | |
| *** ralonsoh has quit IRC | 17:02 | |
| *** slaweq has quit IRC | 17:16 | |
| *** e0ne has joined #openstack-sdks | 17:30 | |
| openstackgerrit | Merged openstack/openstacksdk master: Run ansible tests against specific public cloud https://review.openstack.org/582459 | 17:38 |
| *** r-mibu has quit IRC | 17:51 | |
| *** timburke_ is now known as timburke | 18:14 | |
| *** e0ne has quit IRC | 18:14 | |
| *** slaweq has joined #openstack-sdks | 18:15 | |
| *** mriedem1 has joined #openstack-sdks | 18:16 | |
| *** mriedem has quit IRC | 18:16 | |
| *** gkadam has quit IRC | 18:17 | |
| *** bobh has joined #openstack-sdks | 18:19 | |
| openstackgerrit | Merged openstack/keystoneauth master: Add ability to filter version data by service-type https://review.openstack.org/585029 | 18:22 |
| *** mriedem1 is now known as mriedem | 18:27 | |
| mordred | Shrews, dtroyer, dhellmann: feel like reviewing some os-service-types patches? (https://review.openstack.org/#/q/project:openstack/os-service-types+status:open) | 18:37 |
| Shrews | i don't even know what that repo is (though i can infer from the name) lol | 18:51 |
| mordred | Shrews: :) | 18:54 |
| *** bobh has quit IRC | 18:58 | |
| *** alex_xu has quit IRC | 19:03 | |
| *** alex_xu has joined #openstack-sdks | 19:07 | |
| corvus | mordred: i'm having trouble using the new data= parameter to create_object: | 19:32 |
| corvus | File "/home/corvus/git/openstack-infra/zuul-jobs/.tox/py35/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 7525, in create_object | 19:32 |
| corvus | file_size = len(data) | 19:32 |
| corvus | TypeError: object of type 'DeflateFilter' has no len() | 19:32 |
| corvus | mordred: the session.put code i was using before handled data being an iterator | 19:32 |
| mordred | corvus: ah! hrm. SO ... | 19:33 |
| corvus | er, cloud.object_store.put which i guess is the proxy put. which, i think, is basically going to call session.put. you get the idea. | 19:34 |
| mordred | corvus: that len() is there in service of figuring out whether the data to be uploaded needs to be uploaded as multiple segments | 19:34 |
| corvus | mordred: i believe httplib handles that automatically | 19:34 |
| mordred | well, not for swift it doesn't | 19:35 |
| corvus | (briefly, if it is an iterator with no len, it gets uploaded in multiple segments) | 19:35 |
| corvus | sorry, chunked encoding | 19:35 |
| corvus | are you talking about something else? | 19:35 |
| mordred | yah - this is about uploading to multiple swift objects if the object size is > max_objet_size | 19:36 |
| mordred | and then creating either a DLO or SLO object out of that | 19:36 |
| mordred | (dynamic large object or static large object) | 19:36 |
| corvus | mordred: that's something you have to decide on before you upload? | 19:36 |
| mordred | yup- it impacts how you upload things - and also causes sdk to spawn up a set of worker threads to handle it if it's needed | 19:37 |
| corvus | that sounds fundamentally incompatible with streaming compression | 19:38 |
| mordred | well ... | 19:39 |
| mordred | I believe each segment upload itself could be uploaded with streaming compression | 19:40 |
| mordred | we should ask notmyname about it though ... | 19:40 |
| mordred | corvus: the simple approach could be skip the length/segment step if data is passed as a parameter | 19:41 |
| mordred | corvus: fwiw, vexxhost has 'max_file_size': 5368709120 | 19:42 |
| corvus | mordred: just assume the object isn't too large? | 19:42 |
| mordred | yeah. I mean, the 'standard' max_file_size is what vexxhost has -which is 5G | 19:42 |
| mordred | corvus: but - the way these work is one of two ways (each resulting in a set of discreet objects being uploaded) | 19:43 |
| mordred | in both cases the data is uploaded in a set of 'segment' objects. in static large objects there is then a manifest object that lists the segment objects and the order in which they should be served | 19:44 |
| mordred | in dynamic the segment objects follow a naming scheme with an empty manifest object (so you don't have to keep track of the list and then upload the manifest object) | 19:45 |
| mordred | each of the segment objects is a completely independent swift object, so I'd expect uploading to them with deflate would work | 19:45 |
| mordred | oh - then when you fetch the objects, as a client, you just fetch the manifest object, and swift knows that what you want to do is stream the concatenation of the segment objects | 19:46 |
| mordred | it's the download that I'm not sure what happens | 19:47 |
| corvus | the download? | 19:47 |
| mordred | the fetch | 19:47 |
| corvus | i'm confused, i thought you just described what happens in that case | 19:48 |
| mordred | yes - in the standard case - I do not know how it interacts with streaming compression | 19:48 |
| mordred | largely becaues I've never tested it - I'd imagine it would DTRT | 19:49 |
| mordred | but don't actually know | 19:49 |
| corvus | oh i see what you're saying | 19:49 |
| mordred | corvus: out of curiosity - what is the iterator that has the data in it? | 19:50 |
| mordred | oh. the deflatefilter | 19:51 |
| mordred | *duh* it's in the traceback | 19:51 |
| corvus | yeah. it's a thing i wrote that compresses 16k at a time and produces an unknown amount of data each chunk | 19:52 |
| mordred | corvus: well - here's another question then ... | 19:55 |
| mordred | corvus: does max_file_size apply to the compresesd or uncompresed value | 19:57 |
| corvus | excellent question! it's all very vague | 19:57 |
| corvus | i'm assuming, for the moment, that swift doesn't attempt to do any decoding. so probably it applies to the compressed value. | 19:58 |
| corvus | (i'm assuming it is stored in the way i send it) | 19:58 |
| corvus | i have little factual basis for making that assumption | 19:59 |
| mordred | yeah | 20:00 |
| mordred | corvus: for now, if you make a len method on your object that just returns 0 | 20:00 |
| mordred | if should get you past your issue | 20:00 |
| mordred | file_size is only used a little later to determine if file_size < max_segment_size ... | 20:01 |
| mordred | so if you return 0, the code pass data on through directly | 20:01 |
| openstackgerrit | Merged openstack/os-service-types master: Switch to stestr https://review.openstack.org/585349 | 20:02 |
| openstackgerrit | Merged openstack/os-service-types master: Updated from OpenStack Service Type Authority https://review.openstack.org/585062 | 20:02 |
| openstackgerrit | Monty Taylor proposed openstack/openstacksdk master: Update create_object to handled chunked data https://review.openstack.org/585532 | 20:03 |
| mordred | corvus: ^^ also, I think that should at least do a workaround | 20:03 |
| corvus | mordred: though does it get passed through to the underlying session method? i need that not to have a len method, or it'll avoid chunked uploads. | 20:03 |
| corvus | mordred: i'll try your patch out in a little bit | 20:04 |
| mordred | corvus: ah. | 20:05 |
| mordred | corvus: oh for the love of ... | 20:05 |
| mordred | corvus: we don't use file_size in the data path at all | 20:06 |
| openstackgerrit | Monty Taylor proposed openstack/openstacksdk master: Update create_object to handled chunked data https://review.openstack.org/585532 | 20:07 |
| mordred | corvus: ^^ that should fix it more appropriately | 20:08 |
| corvus | mordred: ack. i'll give it a spin when i finish dealing with trailing slashes :) | 20:09 |
| mordred | corvus: trailing slashes are the worst | 20:09 |
| mordred | corvus: they're almost as terrible as INCORRECT WHITESPACE | 20:09 |
| *** notmyname has joined #openstack-sdks | 20:09 | |
| notmyname | mordred: hello | 20:09 |
| mordred | yay it's notmyname ! | 20:09 |
| mordred | notmyname: we have several questions - I will try to ask them in some semblance of order | 20:10 |
| mordred | notmyname: first of all, if you are uploading a large object and you want to upload compressed with a deflate header - does the compressed or uncompressed size count towards max_file_size? | 20:11 |
| corvus | specifically, "content-encoding: deflate" is what's happening here | 20:12 |
| mordred | yeah. corvus is likely to say smarter words than me | 20:12 |
| notmyname | that is likely to be dependent on something between the client and swift itself | 20:14 |
| notmyname | there's not anything in swift that will accept compressed data and store in uncompressed | 20:15 |
| notmyname | however, if you've got some caching thing (CDN or otherwise) that understands those headers, that work can be done there | 20:15 |
| notmyname | swift will happily store the content-encoding header, if you send it, and return it on a read request. | 20:16 |
| notmyname | lol, rackspace took my name off the author byline ;-) https://blog.rackspace.com/cloud-files-cdn-compresses-at-the-edge | 20:17 |
| mordred | ok. so - what about SLO/DLO objects and concatenation? if the segments are uploaded compressed and swift doesnt' natively do any uncompression, I'm guessing that could get weird for the read? | 20:17 |
| mordred | notmyname: haha | 20:18 |
| mordred | and by 'get weird' I mean 'not work" | 20:18 |
| notmyname | let me try something... | 20:19 |
| timburke | i'd expect you'd want to open the large object, stream it through a compressor, and break out segments from that compressed stream. that way when you go to download the large object, you'll get a singular large, compressed stream | 20:21 |
| timburke | breaking the large object into segments first then compressing is unlikely to end well | 20:23 |
| notmyname | oh hi timburke | 20:23 |
| corvus | makes sense | 20:23 |
| corvus | unfortunately, we have to know ahead of time if we're going to upload a large object or a normal one yeah? | 20:23 |
| notmyname | yeah, what he said. "breaking the large object into segments first then compressing is unlikely to end well". and I just confirmed I wasn't forgetting something about this against a dev box | 20:24 |
| mordred | then I guess you could put content-encoding: deflate header on the manifest object - and a browser would theoretically dtrt? | 20:25 |
| notmyname | corvus: you can use an SLO even if the total object size is much less than a single "normal" object limit | 20:25 |
| notmyname | mordred: ya | 20:25 |
| mordred | notmyname: oh. well that's certainly an interesting thought ... | 20:26 |
| mordred | notmyname, timburke: the overall problem we're trying to solve is what to do with the intersection of openstacksdk transparently creating large objects for you and a user of openstacksdk wanting to pass in an interable that is a compressed stream | 20:26 |
| mordred | it seems like one way to deal with it might be to just always create a SLO if someone passes in an interable instead of a bytes or a filename | 20:27 |
| timburke | corvus: depends on how many api requests you're willing to make :-) one option would be to always upload as a large object (like notmyname said) or upload one segment's worth to the base name, then do a server-side copy to the segment location once you realize you need a large object | 20:27 |
| mordred | ooh. that second one sounds reasonable too | 20:27 |
| corvus | if it's not crazy to create a SLO when not strictly necessary, maybe we could make the decision based on the size of the uncompressed data. so if it's > max size, go ahead and SLO even if it's not strictly necessary.... we'd still only do it for "big" files :) | 20:27 |
| corvus | or that second one. :) | 20:27 |
| notmyname | is this for log files? | 20:28 |
| notmyname | for the zuul jobs? | 20:28 |
| mordred | notmyname: yup | 20:28 |
| corvus | notmyname: for starters (so unlikely to hit it) but container/machine images probably aren't far behind. | 20:28 |
| timburke | if you've got enough memory, you could buffer the first MB or so, if it all fits do it as a normal object; otherwise fall back to SLO | 20:29 |
| notmyname | then in that case, I'd optimize for simpler client write path instead of optimal read latency | 20:29 |
| notmyname | since these will be frequently written and rarely read | 20:29 |
| notmyname | timburke has the right idea | 20:29 |
| notmyname | .read(1024*1024) on the input, if you get the full MB, then do a SLO. if not, write a normal object | 20:30 |
| corvus | memory is actually an issue; we could end up attempting a lot (hundreds? many many hundreds?) of these simultaneously on a 8g vm | 20:31 |
| timburke | no upload pooling? i feel like you'd probably be able to saturate your i/o with tens of workers rather than hundreds... but maybe this is getting into the need to have a simple client | 20:36 |
| timburke | server-side copy (or always SLO, all the time) may work out best | 20:37 |
| notmyname | corvus: mordred: so the general answer is that swift will store the bytestream you send it and also headers that may have some definition for clients (eg content-encoding). SLOs aren't special in that the segments are simply slices of the resulting range. swift doesn't do any interpretation of the contents of objects | 20:38 |
| mordred | timburke: yah - server-side copy or always SLO both sound like good general options | 20:39 |
| mordred | there's definitely a balancing act we're trying to do here with wanting SDK to DTRT and yet also providing enough knobs so that we can do the zuul log upload thing efficiently | 20:39 |
| mordred | notmyname, timburke: thanks both of you - this has been super helpful | 20:41 |
| corvus | ++ | 20:42 |
| timburke | fwiw, swiftclient opts for the buffering thing when uploading from stdin -- i think we go up to 16MB (or something like that?) then start uploading 16MB segments. since its stdin, there's only one upload per-process, so we don't feel too bad about the memory | 20:43 |
| *** d0ugal has quit IRC | 20:44 | |
| mordred | timburke: yah - the fun part of this story is that once sdk switches to "oh, you wanted an SLO" - it does so with a pool of threads (similar to swiftuploader in swiftclient) | 20:49 |
| mordred | of course, actually ... now that I think about it - that won't work for iterators that don't have seek anyway | 20:49 |
| mordred | since it does it in parallel for files by opening multiple handles and seeking on them ... so to support SLO from an input stream we'll need to reengineer what we're doing anyway | 20:50 |
| mordred | corvus: ^^ | 20:50 |
| corvus | this may be a limited use case. we won't want to use the deflatefilter for, say, already compressed images. i've only got it set up to engage for text/ types with no encoding right now. | 20:51 |
| mordred | corvus: ah - cool. | 20:51 |
| mordred | corvus: I'm almost starting to feel like we should add your compressiong streaming code into sdk itself so that we can put it further down the stack | 20:52 |
| mordred | corvus: like, put it around the file reads after the seek in the SLO segment uploads | 20:52 |
| mordred | and make an option to create_object "compress=False" or something like that (just thinking out loud) | 20:53 |
| corvus | mordred: i think it may be pretty domain-specific; i don't think it's appropriate for everything | 20:53 |
| mordred | good point. oh - and also that would be the wrong place anyway | 20:53 |
| notmyname | eg https://github.com/openstack/swift/blob/master/swift/common/internal_client.py#L54 ? | 20:53 |
| corvus | (i actually anticipate some period of us tweaking when this gets used) | 20:53 |
| openstackgerrit | Merged openstack/os-service-types master: Allow passing in service types with _ in them https://review.openstack.org/585410 | 20:54 |
| corvus | notmyname: why in the world isn't that in the standard library? :) | 20:54 |
| corvus | notmyname: i wrote one of those too. i may improve it now :) | 20:55 |
| mordred | corvus: it seems like the 'don't run len on data' patch from above should be the only thing you'd need for the easier case - and that we can probably wait until later to deal with SLO and streamed input | 20:58 |
| corvus | mordred: your patch works. though using that method as opposed to the proxy results in a HEAD request to the container before each PUT | 21:13 |
| mordred | corvus: yeah. we should really cache that container status | 21:19 |
| corvus | mordred: that's probably going to be an extra 1500 requests for, say, a devstack job, yeah? | 21:20 |
| corvus | what's it for? | 21:20 |
| mordred | corvus: create_object will create the container for you if it doesn't exist | 21:20 |
| mordred | head is checking container existence | 21:20 |
| corvus | mordred: i want to set some things on the container when it's created; i assumed it'd be best for me to handle that explicitly before doing the upload | 21:22 |
| mordred | corvus: yes - it's totally best for you to do that | 21:22 |
| mordred | corvus: is this multiple executions or a single program with a single long-lived session? | 21:23 |
| corvus | mordred: a single program with 30 threads uploading in parallel | 21:23 |
| mordred | corvus: hrm. there is a container cache already - it seems like it should only make one HEAD | 21:24 |
| corvus | mordred: my test isn't big enough to re-use a thread. maybe they're all racing to get the first head | 21:24 |
| mordred | probably so. we could make the container cache more threadsafe though | 21:24 |
| corvus | mordred: i'm not yet performing my existence check -- would that prime the cache? | 21:25 |
| mordred | similar to how we do for servers | 21:25 |
| mordred | yes | 21:25 |
| mordred | get_container | 21:25 |
| mordred | will do it | 21:25 |
| corvus | cool, then we may not need to do anything to resolve this. i'll plumb that code in now. | 21:25 |
| mordred | cool | 21:25 |
| corvus | mordred: yep that took care of it. sorry for the false alarm :) | 21:28 |
| *** gildub has joined #openstack-sdks | 21:28 | |
| *** slaweq has quit IRC | 21:30 | |
| mordred | yay! | 21:32 |
| *** edmondsw has quit IRC | 21:34 | |
| *** gildub has quit IRC | 21:37 | |
| *** notmyname has left #openstack-sdks | 21:43 | |
| *** JudeC_ has joined #openstack-sdks | 21:44 | |
| *** JudeC has quit IRC | 21:44 | |
| openstackgerrit | Merged openstack/python-openstackclient master: Fix error with image show when image name is None https://review.openstack.org/529464 | 21:45 |
| *** cdent has quit IRC | 21:45 | |
| *** cdent has joined #openstack-sdks | 22:06 | |
| *** slaweq has joined #openstack-sdks | 22:11 | |
| *** cdent has quit IRC | 22:14 | |
| *** slaweq has quit IRC | 22:16 | |
| openstackgerrit | Dean Troyer proposed openstack/python-openstackclient master: Support --community in openstack image list https://review.openstack.org/565152 | 22:48 |
| *** slaweq has joined #openstack-sdks | 22:50 | |
| *** slaweq has quit IRC | 22:54 | |
| mordred | dtroyer: https://review.openstack.org/#/c/582334 <-- I left a novel of a comment there - I'm guessing we should probably just land it in that form since it's definitely an error it's fixing and leave making it more better to next cycle ... | 23:10 |
| mordred | dtroyer: but whatever you think | 23:11 |
| *** slaweq has joined #openstack-sdks | 23:11 | |
| *** slaweq has quit IRC | 23:16 | |
| corvus | mordred: you said i could pass a dict to the connection constructor... how's that go again? | 23:17 |
| corvus | just openstack.connect(**args) ? | 23:18 |
| corvus | hrm, that gets me: keystoneauth1.exceptions.auth_plugins.MissingRequiredOptions: Auth plugin requires parameters which were not given: auth_url | 23:18 |
| dtroyer | mordred: agreed, there are a number of microversion-affected changes to compute outstanding, some merged already, all of them are likely in this situation, this gets things working to some extent, we'll clean it up when we SDK-enable things and that may influence the order we do that work in... | 23:19 |
| corvus | mordred: oh apparently that's because profile is ignored, so i need to add auth_url in there | 23:26 |
| corvus | mordred: reading https://docs.openstack.org/openstacksdk/latest/user/transition_from_profile.html it's not clear how to accomplish what i want (construct a connection from a dict, using profile=vexxhost and not specifying auth_url) | 23:28 |
| *** gildub has joined #openstack-sdks | 23:50 | |
| *** yolanda_ has joined #openstack-sdks | 23:56 | |
| *** yolanda has quit IRC | 23:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!