| *** dkalleg has quit IRC | 00:20 | |
| *** JRobinson__ has quit IRC | 00:39 | |
| *** JRobinson__ has joined #openstack-ansible | 00:43 | |
| *** abitha has quit IRC | 01:40 | |
| *** daneyon has quit IRC | 03:29 | |
| *** logan2 has quit IRC | 03:35 | |
| *** gtt116 has joined #openstack-ansible | 04:13 | |
| gtt116 | yo | 04:13 |
|---|---|---|
| *** meshok0 has joined #openstack-ansible | 04:27 | |
| *** sdake_ has joined #openstack-ansible | 04:56 | |
| *** sdake has quit IRC | 04:59 | |
| *** javeriak has joined #openstack-ansible | 05:04 | |
| *** meshok0 has quit IRC | 05:21 | |
| *** JRobinson__ is now known as JRobinson__afk | 05:35 | |
| *** shausy has joined #openstack-ansible | 05:36 | |
| *** abitha has joined #openstack-ansible | 05:50 | |
| *** JRobinson__afk is now known as JRobinson__ | 05:53 | |
| *** radek__ has joined #openstack-ansible | 06:10 | |
| *** abitha has quit IRC | 06:20 | |
| *** metral_zzz is now known as metral | 06:35 | |
| *** sdake has joined #openstack-ansible | 06:56 | |
| *** sdake has quit IRC | 06:57 | |
| *** sdake has joined #openstack-ansible | 06:57 | |
| *** sdake_ has quit IRC | 07:00 | |
| *** abitha has joined #openstack-ansible | 07:20 | |
| *** abitha has quit IRC | 07:25 | |
| *** meshok0 has joined #openstack-ansible | 07:26 | |
| *** subscope has joined #openstack-ansible | 07:34 | |
| *** JRobinson__ has quit IRC | 07:43 | |
| *** sdake has quit IRC | 08:06 | |
| *** sura8257 has quit IRC | 08:18 | |
| *** vdo has joined #openstack-ansible | 08:34 | |
| *** subscope has quit IRC | 08:54 | |
| *** javeriak has quit IRC | 09:06 | |
| *** abitha has joined #openstack-ansible | 09:10 | |
| *** abitha has quit IRC | 09:14 | |
| *** sura8257_ has joined #openstack-ansible | 09:49 | |
| *** gtt116_ has joined #openstack-ansible | 10:18 | |
| *** gtt116 has quit IRC | 10:21 | |
| svg | A good day to all | 10:39 |
| *** sura8257_ has quit IRC | 10:50 | |
| *** abitha has joined #openstack-ansible | 10:59 | |
| *** jaypipes has joined #openstack-ansible | 11:00 | |
| *** abitha has quit IRC | 11:04 | |
| odyssey4me | o/ svg how was your w/end | 11:44 |
| svg | o/ odyssey4me | 11:49 |
| svg | calm | 11:49 |
| svg | didn't do anythin, achievement unlocked | 11:49 |
| svg | needed some rest from openstack network troubleshooting | 11:50 |
| *** sdake has joined #openstack-ansible | 11:54 | |
| *** sdake has quit IRC | 11:55 | |
| *** sdake has joined #openstack-ansible | 11:55 | |
| evrardjp | hello everyone | 12:02 |
| evrardjp | Good afternoon for some, morning for the others | 12:03 |
| svg | A good day to you too evrardjp | 12:03 |
| svg | an upate on our network troubleshooting debacle | 12:07 |
| svg | it seems things are falling down once router namespaces are created | 12:08 |
| svg | possibly somewhere between 100-200 on the whole stack, but hard to pin an exact number here | 12:09 |
| svg | i/o with containers also seem to interfere with the loopback adapter | 12:10 |
| *** KLevenstein has joined #openstack-ansible | 12:11 | |
| svg | when things go wrong, we can't ping lo anymor | 12:12 |
| *** KLevenstein has quit IRC | 12:54 | |
| openstackgerrit | Kevin Carter proposed stackforge/os-ansible-deployment: Updates the container build process https://review.openstack.org/191215 | 12:54 |
| *** KLevenstein has joined #openstack-ansible | 12:55 | |
| *** KLevenstein has quit IRC | 12:55 | |
| *** KLevenstein has joined #openstack-ansible | 12:56 | |
| openstackgerrit | Kevin Carter proposed stackforge/os-ansible-deployment: Updates the container build process https://review.openstack.org/191517 | 12:57 |
| *** KLevenstein has quit IRC | 13:06 | |
| *** KLevenstein has joined #openstack-ansible | 13:07 | |
| *** KLevenstein has quit IRC | 13:16 | |
| *** gtt116__ has joined #openstack-ansible | 13:30 | |
| *** gtt116_ has quit IRC | 13:33 | |
| *** ccrouch has joined #openstack-ansible | 13:37 | |
| *** Mudpuppy has joined #openstack-ansible | 13:51 | |
| *** alextricity has joined #openstack-ansible | 13:54 | |
| *** alextricity has quit IRC | 13:56 | |
| *** shausy has quit IRC | 13:56 | |
| cloudnull | Morning. | 14:02 |
| svg | Good afternoon :) | 14:07 |
| *** stevemar has joined #openstack-ansible | 14:12 | |
| *** jwagner_away is now known as jwagner | 14:16 | |
| *** openstackgerrit has quit IRC | 14:24 | |
| *** openstackgerrit has joined #openstack-ansible | 14:24 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 14:28 | |
| *** KLevenstein has joined #openstack-ansible | 14:34 | |
| *** galstrom_zzz is now known as galstrom | 14:38 | |
| sigmavirus24 | Hi svg | 14:46 |
| svg | hi sigmavirus24 | 14:46 |
| sigmavirus24 | Did you figure out what was happening with your networking woes? | 14:47 |
| cloudnull | we some folks have some time, these items need to be reviewed/worked on https://review.openstack.org/#/q/starredby:cloudnull+status:open,n,z | 14:48 |
| svg | sigmavirus24: can you read ^^ (about 2,5 h ago) -> latest news | 14:49 |
| sigmavirus24 | svg: I was hoping there was newer news :D | 14:50 |
| cloudnull | svg: are your envs creating 200 + routers or just network namespaces ? | 14:51 |
| cloudnull | cc ^ Apsu | 14:51 |
| Apsu | Hallo | 14:52 |
| cloudnull | Apsu: we'rent there some kernel tunings that we had done for envs that were/are using LOTS of neutron namespaces ? | 14:53 |
| cloudnull | i thought we had something , but im precaffiniated and in training. | 14:54 |
| Apsu | Mmm, there's some things | 14:54 |
| svg | when tat happns, certain things hit the fan, and metal hosts etc start to become unresponsive network wise | 14:55 |
| svg | (bit of delay, traveling on train) | 14:55 |
| svg | cloudnull: we have a base unit of deploy (heat) with two hosts, with each two nics connected to two networks, each network has its router connected to an external network | 14:56 |
| svg | only the external network is part of another tenant | 14:57 |
| Apsu | svg: What are the names of these namespaces and where are you seeing them? | 14:57 |
| Apsu | Are they all qrouter-* and qdhcp-*? | 14:58 |
| svg | we typicaly test that in batches of 505, and that often starts to fail wsomewehere between 100 and two hundred | 14:58 |
| svg | Apsu: yes, seen on the neutron-agents containers | 14:59 |
| Apsu | Oh, so wait, you're specifically creating 100s of routers and/or networks, and it falls over between 100-200 | 15:00 |
| svg | yes | 15:00 |
| Apsu | Not that you've got a router and a couple networks and you're randomly getting hundreds of namespaces, lol | 15:00 |
| Apsu | ok | 15:00 |
| svg | (not sure where it failes exactly, but something like that) | 15:01 |
| Apsu | What kind of load averages are you seeing? | 15:01 |
| Apsu | I mean, that's a LOT of routers/networks | 15:01 |
| Apsu | And Neutron is very inefficient at handling that many, fyi | 15:01 |
| Apsu | That's 10x the scale most people operate at | 15:01 |
| *** KLevenstein__ has joined #openstack-ansible | 15:04 | |
| svg | load average is almost nothing, mostly less than one on a 48 core box | 15:04 |
| *** KLevenstein has quit IRC | 15:05 | |
| *** KLevenstein__ is now known as KLevenstein | 15:05 | |
| svg | the base design idea we have is to encapsulate every app in its own network, with a "public" network and a management network for vm access | 15:05 |
| svg | the latter being silly imho | 15:06 |
| svg | then they want to do green/blue deployment with such a base 'bubble' | 15:07 |
| svg | they decided to move from vmware away to SDN to be able to grow beyond the 4096 vlan limiit... | 15:09 |
| Apsu | lol | 15:09 |
| Apsu | What is this I don't even :) | 15:09 |
| Apsu | I see the VLAN limit as a hint. Mayyyybe you should be doing this differently, type thing | 15:10 |
| Apsu | Unless you're reselling datacenter space | 15:10 |
| odyssey4me | cloudnull it would appear that the apt cache updating has vastly improved success :) nice catch on the bug, and the patch to ansible too! | 15:11 |
| cloudnull | hopfully that patch https://github.com/ansible/ansible-modules-core/pull/1517 goes in for 1.9.2 | 15:12 |
| svg | for the record, there were also some minor dns and network issues we straightened out, which barfed at us at deploy time | 15:12 |
| svg | but ok, sao basically we're doing it wrong :) | 15:12 |
| svg | to many networks | 15:13 |
| openstackgerrit | Darren Birkett proposed stackforge/os-ansible-deployment: Set permissions on user_secrets.yml to 0600 https://review.openstack.org/191851 | 15:13 |
| Apsu | svg: We've seen scaling issues in Neutron before -- with and without containers -- when getting into the hundreds of networks range | 15:14 |
| Apsu | Mostly around slightly older kernel versions | 15:14 |
| Apsu | But also the fact Neutron's architecture/services scale poorly | 15:15 |
| Apsu | Some of the agents ship around the full port lists very often, for instance | 15:15 |
| Apsu | Which gets large quickly and isn't a cheap (enough) operation to query | 15:15 |
| *** sdake_ has joined #openstack-ansible | 15:23 | |
| palendae | cloudnull: Playing with your container build patches in an AIO | 15:24 |
| palendae | Code looks good, just want to see it before I vote | 15:24 |
| svg | Apsu does this also explain why other containers and hosts get in trouble to communicate withc each other? | 15:24 |
| palendae | see it in action | 15:24 |
| *** sdake has quit IRC | 15:27 | |
| Apsu | svg: Possibly. That might be something else though | 15:30 |
| Apsu | svg: Maybe LXC has some limitations around high netns counts. My guess is probably not, probably something else going on | 15:31 |
| vincent_vdk | Apsu: I'm wondering how larger hosting providers solve this problem | 15:32 |
| vincent_vdk | like OVH, Rackspace etc.. | 15:33 |
| cloudnull | Vincent_vdk I'm my experience providers at very large scale start leaning on sdn providers like plumgrid, etc. | 15:35 |
| Apsu | I'm not sure how much I can say about Rackspace's public cloud scaling architecture, but... | 15:35 |
| vincent_vdk | :) | 15:36 |
| Apsu | I can tell you that there's a lot of custom OVS plumbing, involving pushing flows directly in without letting the standard python bits do the calculations and port shipping | 15:36 |
| Apsu | Because it didn't scale :) | 15:36 |
| palendae | If only there was an open source project where those fixes could have been pushed... >.> | 15:37 |
| Apsu | That'd be sweet. We should start one | 15:37 |
| palendae | nova-network | 15:37 |
| *** nosleep77 has joined #openstack-ansible | 15:37 | |
| Apsu | Nice name idea. Let's propose it | 15:37 |
| palendae | It's networks for nova, you see | 15:37 |
| Apsu | Right right, I follow | 15:37 |
| cloudnull | vincent_vdk: http://www.slideshare.net/andyhky/rackspace-hypervisor-networking-show-tell < andy hill network guru from rax public cloud. | 15:38 |
| cloudnull | a little old but worth a watch https://www.openstack.org/summit/openstack-summit-atlanta-2014/session-videos/presentation/neutron-at-scale | 15:39 |
| Apsu | tl;dr, broooooken | 15:39 |
| cloudnull | ^ truth | 15:39 |
| vincent_vdk | thanks for the pointers | 15:40 |
| Apsu | Honestly, going from the tested deployment size of devstack (> 1 node isn't common in the CI chain) to a production public cloud is a cosmic scale shift | 15:41 |
| Apsu | Takes entirely different methods to deal with | 15:41 |
| vincent_vdk | we are now also looking at HP Helion, but they also use ovs underneath | 15:41 |
| Apsu | OVS itself isn't really the challenge, per se. It's moreso the machinery around what talks to it and how it deals with scale | 15:43 |
| Apsu | Neutron's mechanism for shipping around port information from L2 providers is poor with the stock OVS/LinuxBridge plugins | 15:44 |
| Apsu | Essentially every agent loop iteration there's a full port list query, ship and comparison with the database, to converge changes if necessary | 15:45 |
| palendae | Sounds like the problem is big networks | 15:45 |
| Apsu | To align each agent machine's state | 15:45 |
| Apsu | Because l2 agents are essential converging state machines | 15:45 |
| openstackgerrit | Ian Cordasco proposed stackforge/os-ansible-deployment: Add support for deploying Keystone with Fernet https://review.openstack.org/189998 | 15:46 |
| Apsu | palendae: Yep, pretty much | 15:47 |
| Apsu | Neutron's fairly dumb with respect to that for stock plugins | 15:48 |
| cloudnull | It's no secret that our pub cloud uses ovs , but they have a custom sdn plugin that makes all the magic possible. | 15:48 |
| cloudnull | But its worth mentioning that they employ a dedicated ops team and support staff to ensure all of that is happy. Ovs is a support and upgrade nightmare , IMHO. | 15:49 |
| palendae | I'd imagine any deployer running at that scale would employ a dedicated team for the network | 15:50 |
| palendae | Clouds seem to still be very much pets for the ones operating them | 15:50 |
| vincent_vdk | all sounds very promising | 15:52 |
| vincent_vdk | :) | 15:52 |
| *** jaypipes has quit IRC | 16:03 | |
| *** gus has joined #openstack-ansible | 16:04 | |
| *** jaypipes has joined #openstack-ansible | 16:05 | |
| *** sdake has joined #openstack-ansible | 16:11 | |
| *** sdake_ has quit IRC | 16:14 | |
| *** daneyon has joined #openstack-ansible | 16:26 | |
| *** daneyon has quit IRC | 16:26 | |
| *** daneyon has joined #openstack-ansible | 16:27 | |
| *** abitha has joined #openstack-ansible | 16:34 | |
| *** meshok0 has quit IRC | 16:50 | |
| svg | okm thanks apsu cloudnull for your insights, I'm sending our team back to the drawing board... | 16:50 |
| Apsu | svg: Good luck! If there's some specific failure with containers and namespaces I can help take a look at, let me know | 16:51 |
| svg | Thanks. Hopelfulle the next issues will be clearer, if any. | 16:52 |
| *** Mudpuppy has quit IRC | 16:54 | |
| *** javeriak has joined #openstack-ansible | 16:54 | |
| *** dkalleg has joined #openstack-ansible | 17:01 | |
| *** jwagner is now known as jwagner_away | 17:02 | |
| sigmavirus24 | So are we okay with having rsync installed on all of the keystone containers just to synchronize stuff for fernet? | 17:14 |
| *** gtt116__ has quit IRC | 17:17 | |
| *** gtt116__ has joined #openstack-ansible | 17:18 | |
| stevelle | sigmavirus24: would something like this work for you? http://paste.openstack.org/show/294397/ | 17:44 |
| stevelle | I wasn't clear on whether there was actually an issue with the shared secret ever hitting the deploy host | 17:45 |
| sigmavirus24 | stevelle: it may, but synchronize would remove the necessity for that complexity so long as we can all agree rsync inside of the keystone containers is okay | 17:45 |
| sigmavirus24 | stevelle: no one's objected to fetch/copy'ing other than we then have to add a local_action to clean it all up | 17:46 |
| sigmavirus24 | That said, we're still (poorly) reimplementing synchronize just to avoid rsync which seems silly to me | 17:46 |
| stevelle | I find this easy enough to understand | 17:46 |
| stevelle | but I don't feel strongly either way | 17:46 |
| stevelle | I suppose you already have rsa keys on all the hosts so you can rsync. | 17:47 |
| sigmavirus24 | Believe so | 17:48 |
| stevelle | whereas that is exactly what I was setting up with the above play | 17:48 |
| sigmavirus24 | be back shortly after a quick lunch intermission | 17:52 |
| stevelle | I suppose I can't' think of a reason sync won't work. The only hesitation there is that we are adding a package that the os_install_keystone.yml doesn't include already but that's in a var already. | 17:52 |
| *** javeriak has quit IRC | 17:56 | |
| *** javeriak has joined #openstack-ansible | 17:57 | |
| *** Mudpuppy has joined #openstack-ansible | 18:04 | |
| *** galstrom is now known as galstrom_zzz | 18:06 | |
| *** stevemar2 has joined #openstack-ansible | 18:09 | |
| *** stevemar has quit IRC | 18:09 | |
| *** meshok0 has joined #openstack-ansible | 18:14 | |
| openstackgerrit | Steve Lewis proposed stackforge/os-ansible-deployment: Configure DB addresses for each service https://review.openstack.org/190271 | 18:19 |
| stevelle | cores: ^ could use a refresh of workflow when you have time | 18:20 |
| *** jwagner_away is now known as jwagner | 18:25 | |
| *** jmccrory has quit IRC | 18:26 | |
| *** sdake_ has joined #openstack-ansible | 18:29 | |
| *** jmccrory has joined #openstack-ansible | 18:31 | |
| openstackgerrit | Steve Lewis proposed stackforge/os-ansible-deployment: Allow Horizon to access multiple regions https://review.openstack.org/190302 | 18:33 |
| *** sdake has quit IRC | 18:34 | |
| palendae | stevelle: Done | 18:37 |
| *** nosleep77 has left #openstack-ansible | 18:43 | |
| *** stevemar2 is now known as stevemar | 18:51 | |
| *** meshok01 has joined #openstack-ansible | 19:05 | |
| *** meshok0 has quit IRC | 19:05 | |
| openstackgerrit | Ian Cordasco proposed stackforge/os-ansible-deployment: Create Junit XML Report from tempest run https://review.openstack.org/191103 | 19:07 |
| openstackgerrit | Steve Lewis proposed stackforge/os-ansible-deployment: Move haproxy vars to defaults https://review.openstack.org/190721 | 19:10 |
| openstackgerrit | Steve Lewis proposed stackforge/os-ansible-deployment: Updated to the latest stable rabbitmq release https://review.openstack.org/191514 | 19:24 |
| stevelle | cores: another workflow refresh needed on https://review.openstack.org/#/c/190220/ | 19:26 |
| openstackgerrit | Steve Lewis proposed stackforge/os-ansible-deployment: Split environment file into component parts https://review.openstack.org/190220 | 19:26 |
| openstackgerrit | Merged stackforge/os-ansible-deployment: Updates the container build process https://review.openstack.org/191215 | 19:38 |
| sigmavirus24 | Apsu: are you certain that we set up the containers such that one can ssh from one container to another? | 19:38 |
| Apsu | sigmavirus24: No I'm not certain. I was actually thinking host to host | 19:49 |
| Apsu | I suspect containers don't have privkeys on them, just authorized_keys | 19:50 |
| sigmavirus24 | Apsu: yeah that's what it looks like :/ | 19:50 |
| sigmavirus24 | And host to containers | 19:50 |
| Apsu | yeah | 19:50 |
| sigmavirus24 | Yeah that's what it looks like but I can't get past the keystone role at this point in time | 19:50 |
| sigmavirus24 | so I'm runnign a setup-everything without my changes using synchronize | 19:50 |
| *** javeriak has quit IRC | 19:50 | |
| *** sdake has joined #openstack-ansible | 19:54 | |
| *** javeriak has joined #openstack-ansible | 19:58 | |
| *** sdake_ has quit IRC | 19:58 | |
| *** daneyon_ has joined #openstack-ansible | 19:59 | |
| *** sdake_ has joined #openstack-ansible | 20:01 | |
| *** daneyon has quit IRC | 20:02 | |
| *** sdake has quit IRC | 20:05 | |
| *** javeriak has quit IRC | 20:06 | |
| *** javeriak has joined #openstack-ansible | 20:08 | |
| *** javeriak has quit IRC | 20:19 | |
| svg | Apsu: we cleaned all deploys in the stack, got left with about 5 networks and 2 routers - stack works but feels sluggish, asking ro a net list or router list, from cli or horizon, takes several seconds | 20:35 |
| Apsu | svg: Odd. Load averages on controllers? | 20:36 |
| svg | all < 0.48 | 20:37 |
| svg | how can I check whick backend component doing a neutron net/router-list takes about 10s | 20:43 |
| svg | nu, lijkt me een goed plan om al die netwerken eens te vergeten, en te testen op de rest | 20:44 |
| *** sdake has joined #openstack-ansible | 20:44 | |
| svg | als we daarmee een deelproblem weglaten, wordt een ander probleem miscchien meer evident | 20:45 |
| svg | oops | 20:45 |
| svg | sorry, wrond window :) | 20:45 |
| Apsu | svg: Well, --debug is a god first start. | 20:45 |
| svg | :) | 20:46 |
| Apsu | Er, good, lulz | 20:46 |
| *** KLevenstein has quit IRC | 20:46 | |
| *** sdake_ has quit IRC | 20:48 | |
| *** KLevenstein has joined #openstack-ansible | 20:52 | |
| *** Mudpuppy has quit IRC | 20:54 | |
| openstackgerrit | Kevin Carter proposed stackforge/os-ansible-deployment: Added apt update tasks to everything using apt https://review.openstack.org/191528 | 20:55 |
| openstackgerrit | Kevin Carter proposed stackforge/os-ansible-deployment: Add read/write_affinity settings for Swift https://review.openstack.org/191023 | 20:55 |
| openstackgerrit | Kevin Carter proposed stackforge/os-ansible-deployment: Allow protocol to be set per endpoint-type https://review.openstack.org/190736 | 20:56 |
| openstackgerrit | Kevin Carter proposed stackforge/os-ansible-deployment: Set default pass in aio scripts to be random https://review.openstack.org/190266 | 20:56 |
| openstackgerrit | Kevin Carter proposed stackforge/os-ansible-deployment: Configurable lxc cache and rootfs path https://review.openstack.org/190048 | 20:56 |
| openstackgerrit | Kevin Carter proposed stackforge/os-ansible-deployment: Ceph/RBD support https://review.openstack.org/181957 | 20:56 |
| openstackgerrit | Kevin Carter proposed stackforge/os-ansible-deployment: Updated MariaDB to the new release version https://review.openstack.org/178259 | 20:56 |
| *** jwagner is now known as jwagner_away | 20:58 | |
| svg | cloudnull: you made an update to the ceph review; how am I supposed to pull in that subpatch back into my tree? | 21:01 |
| cloudnull | You should just be able to rebase the patch off of current master. | 21:01 |
| cloudnull | # git fetch --all | 21:02 |
| cloudnull | # git rebase -i origin/master | 21:02 |
| svg | that won't pull in the small change you made | 21:02 |
| *** Mudpuppy has joined #openstack-ansible | 21:03 | |
| svg | https://review.openstack.org/#/c/181957/19..21/playbooks/roles/lxc_container_create/tasks/container_create.yml,cm | 21:03 |
| *** stevemar2 has joined #openstack-ansible | 21:03 | |
| *** stevemar has quit IRC | 21:04 | |
| svg | cloudnull: ^^ | 21:05 |
| cloudnull | Yes? | 21:06 |
| svg | rebasing won't pull in the small change you made | 21:06 |
| cloudnull | If you rebase your local branch with whats now in master it should fix it right up. | 21:07 |
| svg | how is that possible, as you added a change that is not yet in master | 21:08 |
| svg | ok, sorry, that one is in master | 21:09 |
| svg | urgh | 21:09 |
| svg | time to go to sleep, sorry | 21:09 |
| cloudnull | No worries. | 21:09 |
| *** Mudpuppy has quit IRC | 21:11 | |
| *** javeriak has joined #openstack-ansible | 21:13 | |
| openstackgerrit | Kevin Carter proposed stackforge/os-ansible-deployment: changed container bind mounts to use abspath https://review.openstack.org/188558 | 21:18 |
| *** yaya has joined #openstack-ansible | 21:20 | |
| *** meshok01 has quit IRC | 21:20 | |
| openstackgerrit | Ian Cordasco proposed stackforge/os-ansible-deployment: Add support for deploying Keystone with Fernet https://review.openstack.org/189998 | 21:22 |
| *** yaya has left #openstack-ansible | 21:34 | |
| openstackgerrit | Ian Cordasco proposed stackforge/os-ansible-deployment: Add support for deploying Keystone with Fernet https://review.openstack.org/189998 | 21:38 |
| *** Mudpuppy has joined #openstack-ansible | 21:39 | |
| *** Mudpuppy has quit IRC | 21:39 | |
| *** Mudpuppy has joined #openstack-ansible | 21:39 | |
| openstackgerrit | Ian Cordasco proposed stackforge/os-ansible-deployment: Add support for deploying Keystone with Fernet https://review.openstack.org/189998 | 21:43 |
| *** stevemar2 is now known as stevemar | 21:43 | |
| openstackgerrit | Ian Cordasco proposed stackforge/os-ansible-deployment: Add support for deploying Keystone with Fernet https://review.openstack.org/189998 | 21:50 |
| openstackgerrit | Merged stackforge/os-ansible-deployment: Split environment file into component parts https://review.openstack.org/190220 | 21:52 |
| *** KLevenstein has quit IRC | 21:56 | |
| *** Mudpuppy has quit IRC | 22:02 | |
| openstackgerrit | Ian Cordasco proposed stackforge/os-ansible-deployment: Add support for deploying Keystone with Fernet https://review.openstack.org/189998 | 22:05 |
| sigmavirus24 | So in running ^ on an AIO, I'm seeing http://paste.openstack.org/show/IGhYKNSqqnmIiYONF0NO/ and I'm not quite sure why | 22:20 |
| *** stevemar has quit IRC | 22:21 | |
| *** dkalleg has quit IRC | 22:37 | |
| *** dkalleg has joined #openstack-ansible | 22:39 | |
| sigmavirus24 | dolphm: http://paste.openstack.org/show/JQVaFpRp21BpskWxWwcR/ | 22:40 |
| openstackgerrit | Merged stackforge/os-ansible-deployment: Updates the container build process https://review.openstack.org/191517 | 22:40 |
| sigmavirus24 | dolphm: so that HMAC_CTX error is coming from a 400 in Keystone it seems: http://paste.openstack.org/show/294535/ | 22:42 |
| sigmavirus24 | Doesn't seem there's much detail coming back from the cinder client though | 22:42 |
| sigmavirus24 | going to start going through keystone logs | 22:43 |
| *** sura8257 has joined #openstack-ansible | 22:45 | |
| sigmavirus24 | dolphm: http://paste.openstack.org/show/294537/ is out of the apache logs | 22:45 |
| dolphm | sigmavirus24: looking.. | 22:45 |
| dolphm | sigmavirus24: oh that's fun | 22:46 |
| sigmavirus24 | Yeahhh, right? | 22:46 |
| dolphm | sigmavirus24: wonder if something is butchering the fernet token? | 22:46 |
| sigmavirus24 | Hm | 22:46 |
| * sigmavirus24 wonders if rsync is butchering the token | 22:46 | |
| sigmavirus24 | nope | 22:48 |
| sigmavirus24 | both containers have the exact same content | 22:48 |
| sigmavirus24 | for both keys | 22:48 |
| sigmavirus24 | oh that's another thing | 22:48 |
| dolphm | sigmavirus24: https://github.com/pyca/cryptography/issues/1776 | 22:48 |
| sigmavirus24 | max_keys is set to 3 but it only generates two | 22:48 |
| dolphm | sigmavirus24: so it's under the configured maximum? working as intended! | 22:49 |
| * sigmavirus24 wasn't sure if maximum was meant to be inclusive or not | 22:49 | |
| dolphm | sigmavirus24: no matter how many times you do keystone manage fernet-rotate now, it'll keep the number of keys under the maximum | 22:49 |
| sigmavirus24 | cool | 22:49 |
| dolphm | sigmavirus24: your first rotate will give you 3 keys | 22:50 |
| dolphm | and it'll stay at that number | 22:50 |
| sigmavirus24 | dolphm: so the version of cryptography deployed is 0.9.1 | 22:50 |
| sigmavirus24 | which according to github is in 0.9.1 | 22:51 |
| sigmavirus24 | so another issue it seems? | 22:51 |
| dolphm | sigmavirus24: based on the conversation in that bug, i wouldn't be surprised if it was the same issue somewhere else | 22:52 |
| sigmavirus24 | so if you look at https://github.com/pyca/cryptography/commit/fbd7d7f2adb5dfb66175e5a2f77e4e0c4a640107 the problem is with how the library is loaded and I'm not sure how that could have regressed | 22:53 |
| sigmavirus24 | thanks dolphm ;) | 22:54 |
| sigmavirus24 | I was going to ping them when I had a better idea | 22:54 |
| *** JRobinson__ has joined #openstack-ansible | 22:54 | |
| *** dkalleg has quit IRC | 22:54 | |
| dolphm | sigmavirus24: turn the number of apache threads down to 1? | 22:54 |
| sigmavirus24 | I haven't yet | 22:55 |
| dolphm | sigmavirus24: when we benchmarked keystone w/ apache, we didn't get much benefit out of threading at all in either mpm_event or mpm_worker | 22:55 |
| dolphm | we also weren't memory constrained at all, so more processes was a simpler path forward | 22:56 |
| * dolphm afk | 23:00 | |
| *** dkalleg has joined #openstack-ansible | 23:22 | |
| openstackgerrit | Miguel Grinberg proposed stackforge/os-ansible-deployment: Use default service_region var for all roles https://review.openstack.org/189664 | 23:38 |
| openstackgerrit | Miguel Grinberg proposed stackforge/os-ansible-deployment: Support an externally hosted keystone https://review.openstack.org/192015 | 23:38 |
| *** daneyon_ has quit IRC | 23:43 | |
| *** javeriak has quit IRC | 23:58 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!