| openstackgerrit | Ian Cordasco proposed stackforge/os-ansible-deployment: Keystone Federation Service Provider Configuration https://review.openstack.org/194395 | 00:07 |
|---|---|---|
| *** abitha has quit IRC | 00:34 | |
| *** tomjoseka has quit IRC | 00:37 | |
| *** alop has quit IRC | 00:44 | |
| *** alop has joined #openstack-ansible | 00:44 | |
| *** jmckind has quit IRC | 00:45 | |
| *** pilgrimstack1 has joined #openstack-ansible | 01:09 | |
| *** pilgrimstack has quit IRC | 01:11 | |
| *** alop has quit IRC | 02:00 | |
| openstackgerrit | Ian Cordasco proposed stackforge/os-ansible-deployment: Fix the Juno -> Kilo upgrade script https://review.openstack.org/207687 | 02:06 |
| *** bapalm has joined #openstack-ansible | 02:09 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 02:12 | |
| *** bapalm has quit IRC | 02:19 | |
| *** markvoelker has joined #openstack-ansible | 02:19 | |
| *** markvoelker_ has joined #openstack-ansible | 02:21 | |
| *** markvoelker has quit IRC | 02:23 | |
| *** openstackgerrit has quit IRC | 02:31 | |
| *** openstackgerrit has joined #openstack-ansible | 02:31 | |
| openstackgerrit | Major Hayden proposed stackforge/os-ansible-deployment: Small readme fix https://review.openstack.org/207700 | 03:02 |
| *** markvoelker_ has quit IRC | 03:23 | |
| *** markvoelker_ has joined #openstack-ansible | 03:26 | |
| *** sdake_ has joined #openstack-ansible | 03:42 | |
| *** sdake has quit IRC | 03:43 | |
| *** sdake_ has quit IRC | 03:45 | |
| *** sdake has joined #openstack-ansible | 03:45 | |
| *** sdake_ has joined #openstack-ansible | 03:51 | |
| *** sdake has quit IRC | 03:52 | |
| *** sdake_ has quit IRC | 04:21 | |
| *** sdake has joined #openstack-ansible | 04:21 | |
| openstackgerrit | Merged stackforge/os-ansible-deployment: remove conntrackd package https://review.openstack.org/207441 | 04:23 |
| openstackgerrit | Nolan Brubaker proposed stackforge/os-ansible-deployment-specs: Add spec for the Liberty cycle upgrade path https://review.openstack.org/207713 | 04:28 |
| *** sdake has quit IRC | 04:28 | |
| *** sdake_ has joined #openstack-ansible | 04:28 | |
| *** sdake has joined #openstack-ansible | 04:31 | |
| *** sdake_ has quit IRC | 04:31 | |
| *** sdake_ has joined #openstack-ansible | 04:35 | |
| *** sdake has quit IRC | 04:36 | |
| *** sdake_ has quit IRC | 04:41 | |
| *** sdake has joined #openstack-ansible | 04:42 | |
| *** sdake has quit IRC | 04:44 | |
| *** sdake has joined #openstack-ansible | 04:45 | |
| *** sdake has quit IRC | 04:47 | |
| *** sdake has joined #openstack-ansible | 04:47 | |
| *** fawadkhaliq has joined #openstack-ansible | 04:55 | |
| *** davidself has quit IRC | 05:14 | |
| *** misc_ is now known as misc | 05:24 | |
| *** ig0r_ has joined #openstack-ansible | 05:25 | |
| *** sdake has quit IRC | 05:35 | |
| *** yapeng has joined #openstack-ansible | 05:38 | |
| *** annashen_ has joined #openstack-ansible | 05:39 | |
| *** sdake has joined #openstack-ansible | 05:41 | |
| *** markvoelker_ has quit IRC | 05:48 | |
| *** shausy has joined #openstack-ansible | 06:06 | |
| *** evrardjp_ is now known as evrardjp | 06:07 | |
| *** javeriak has joined #openstack-ansible | 06:07 | |
| *** ig0r__ has joined #openstack-ansible | 06:11 | |
| *** ig0r_ has quit IRC | 06:12 | |
| *** Mudpuppy has quit IRC | 06:13 | |
| *** annashen_ has quit IRC | 06:15 | |
| *** javeriak_ has joined #openstack-ansible | 06:24 | |
| *** javeriak has quit IRC | 06:27 | |
| *** yapeng has quit IRC | 06:31 | |
| *** shausy has quit IRC | 06:38 | |
| *** shausy has joined #openstack-ansible | 06:40 | |
| *** shausy has quit IRC | 06:43 | |
| *** shausy has joined #openstack-ansible | 06:44 | |
| *** markvoelker has joined #openstack-ansible | 06:48 | |
| *** markvoelker has quit IRC | 06:53 | |
| odyssey4me | morning all | 07:50 |
| odyssey4me | meteorfox The definitions of the roles are up to you as the spec creator - you may wish to define them in your spec to be clear if you see them as different roles, although I think Dployer typically would speak to the deployment experience whereas Operator would typically speak to the experience of keeping it running and operating it after the deployment is done. | 07:53 |
| odyssey4me | mattt would you mind quickly reviewing some backports? | 07:54 |
| openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: Add sample Keystone Federation SP configuration for ADFS https://review.openstack.org/203736 | 07:56 |
| mattt | odyssey4me: yeah sure, got a list ? | 07:58 |
| mattt | (or urgent ones) | 07:58 |
| mattt | or n/m, let me just scan gerrit :) | 07:58 |
| odyssey4me | mattt https://review.openstack.org/#/q/starredby:%22Jesse+Pretorius%22+project:stackforge/os-ansible-deployment+branch:kilo,n,z | 07:59 |
| mattt | odyssey4me: what's going on w/ https://review.openstack.org/#/c/207105/ ? | 08:06 |
| mattt | needs a tickle? (recheck) | 08:07 |
| odyssey4me | mattt we have an issue with document change gate check at the moment - they never end up in the check queue | 08:07 |
| odyssey4me | so don't worry about those - if you've voted then I'll figure out the gate issue later | 08:07 |
| mattt | i haven't a clue what's going on in https://review.openstack.org/#/c/207386/, i'm just going to have to assume this is all good to go | 08:09 |
| odyssey4me | mattt yep, it's a backport - don't +w it, but add your vote | 08:10 |
| mattt | k done | 08:10 |
| odyssey4me | thx mattt | 08:10 |
| evrardjp | good morning everyone | 08:21 |
| *** javeriak_ has quit IRC | 08:21 | |
| openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: Enable Horizon to consume a Keystone v3 API endpoint https://review.openstack.org/206575 | 08:25 |
| mattt | howdy evrardjp | 08:25 |
| odyssey4me | morning! | 08:26 |
| svg | morning al | 08:27 |
| mattt | mroning svg | 08:27 |
| svg | latest update a couple of days ago from cloudnull for ceph patch release was milestone: 11.1.0 → 11.1.1 | 08:28 |
| svg | Not sure why that is, given the semantaic versioning, I'd expect 11.1.0? | 08:29 |
| svg | see https://bugs.launchpad.net/openstack-ansible/+bug/1455238 | 08:29 |
| openstack | Launchpad bug 1455238 in openstack-ansible trunk "Ceph Support" [Wishlist,In progress] - Assigned to Serge van Ginderachter (svg) | 08:29 |
| *** javeriak has joined #openstack-ansible | 08:32 | |
| svg | git-harry: you here? | 08:36 |
| mattt | svg: i think ceph will be going into 11.2 | 08:36 |
| mattt | (the ceph client stuff that is) | 08:36 |
| svg | How exactly does this gets decided? | 08:37 |
| svg | mattt: do you perhaps understand git-harry 's latest comment (on line 52) in https://review.openstack.org/#/c/181957/32 ? | 08:39 |
| mattt | svg: the if statement one? | 08:39 |
| mattt | his comment makes sense, but i had to look at it 2x also :P | 08:40 |
| svg | I see how given that show_image_direct_url has beome a var, the if is not needed any more, but I don't understand what he meant by 'The var is false if glance_default_store != 'rbd' and true if it is.' | 08:40 |
| odyssey4me | svg we don't yet have the 11.2.0 milestone setup and 11.1.0 releases today and the patch isn't ready yet so it'll have to move to the next release | 08:40 |
| mattt | svg: well glance_show_image_direct_url: "{{ glance_default_store == 'rbd' }}" in defaults/main.yml | 08:40 |
| mattt | svg: so glance_show_image_direct_url will always be true/false, meaning you can always set show_image_direct_url and therefore the condition around it is not needed | 08:41 |
| mattt | svg: i don't handle release management but as i understand it stuff that has been backported will be queued up for the next release | 08:41 |
| svg | ok, my coin fell, to translate a ducth saying :) | 08:42 |
| mattt | could be me but i think that is lost in translation :) | 08:43 |
| odyssey4me | yeah, it needs to get into master before we can consider the milestone to attach it to | 08:43 |
| git-harry | svg: hello | 08:43 |
| git-harry | mattt: the penny dropped | 08:43 |
| svg | hi harry, problem solved now | 08:43 |
| mattt | git-harry: WHY YOU SO SMART | 08:43 |
| openstackgerrit | Serge van Ginderachter proposed stackforge/os-ansible-deployment: Add Ceph/RBD support to playbooks https://review.openstack.org/181957 | 08:44 |
| mattt | if this gates i'm smashing the merge button | 08:46 |
| git-harry | haha | 08:46 |
| *** javeriak_ has joined #openstack-ansible | 08:47 | |
| *** javeriak has quit IRC | 08:48 | |
| *** markvoelker has joined #openstack-ansible | 08:49 | |
| *** markvoelker has quit IRC | 08:54 | |
| *** shausy has quit IRC | 09:06 | |
| *** shausy has joined #openstack-ansible | 09:07 | |
| openstackgerrit | Merged stackforge/os-ansible-deployment: Remove {{ from "with_items" and "when" statements https://review.openstack.org/207175 | 09:09 |
| openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: Increase ssh delay after flush net cache tasks https://review.openstack.org/207793 | 09:26 |
| mattt | git-harry svg: found one more small issue preventing the handler from working | 09:49 |
| git-harry | mattt: I may be looking at the same thing | 09:50 |
| svg | sorry for that | 09:50 |
| git-harry | mattt: is that where the host is in multiple groups? | 09:50 |
| mattt | git-harry: just updated w/ my comments, should explain it | 09:51 |
| mattt | svg: no worries, we're 99% there :) | 09:51 |
| mattt | git-harry: the issue was largely that we define service to be an array, and point it to an array | 09:52 |
| mattt | so service ends up being service: [[service1,2,3]] | 09:52 |
| mattt | so we loop over service and hand off an array to the handler, which pukes | 09:52 |
| mattt | but actually i'm just testing this, and it's still broken :P | 09:53 |
| mattt | so svg perhaps hold off before you make any changes :P | 09:53 |
| *** javeriak_ has quit IRC | 09:54 | |
| svg | I should ask the guy that wrote the subelement plugin | 09:56 |
| mattt | svg: is that yours? | 09:56 |
| odyssey4me | lol | 09:57 |
| odyssey4me | as I recall, yes :p | 09:57 |
| svg | mattt: so it seems lists dont get flattened here (they do in the base with_items) | 10:02 |
| *** javeriak has joined #openstack-ansible | 10:07 | |
| mattt | svg: still looking, but i made the change i suggested in a dummy playbook which works, but not when i run os-glance-install.yml | 10:09 |
| mattt | svg: fatal: [infra1-578153_glance_container-28178217] => the key service should point to a list, got '{{ cinder_service_names }}' | 10:09 |
| git-harry | mattt: that var isn't defined | 10:12 |
| git-harry | because you're running it on the glance hosts and the var is coming from the openstack role | 10:12 |
| mattt | AH | 10:13 |
| mattt | that makes sense | 10:13 |
| git-harry | That's causing me an issue because cinder-volume and nova-compute are on the same host | 10:13 |
| mattt | service: '{{ nova_service_names|default() }}' ? | 10:19 |
| mattt | probably not the right approach either tbh | 10:20 |
| git-harry | you could probably use 'is defined' | 10:20 |
| *** gparaskevas has joined #openstack-ansible | 10:27 | |
| gparaskevas | hello everyone | 10:27 |
| mattt | gparaskevas: hey there | 10:28 |
| mattt | gparaskevas: sorry we missed your swift question yesterday | 10:28 |
| gparaskevas | hello mat no problem at all :) | 10:29 |
| gparaskevas | i am still facing the swift issue | 10:30 |
| gparaskevas | and i dont get it really | 10:30 |
| odyssey4me | gparaskevas are you wanting to deploy swift? | 10:30 |
| odyssey4me | and do you have swift hosts defined in your openstack_user_config | 10:31 |
| gparaskevas | i only deploy it for cinder backend and evaluating purpose overall | 10:31 |
| gparaskevas | let me check user_config once again | 10:31 |
| gparaskevas | as far as i can understand i ony define storag hosts in openstack_user_config right? | 10:33 |
| gparaskevas | then i have a swift.yml in conf.d folder | 10:33 |
| mattt | gparaskevas: what's your swift.yml look like and can you remind us what the error is you're hitting ? | 10:35 |
| gparaskevas | sure give me a sec | 10:35 |
| gparaskevas | http://paste.openstack.org/show/406574/->error | 10:37 |
| gparaskevas | http://paste.openstack.org/show/406574/ ->error | 10:37 |
| gparaskevas | http://paste.openstack.org/show/406575/ ->swift.yml | 10:38 |
| gparaskevas | the weird thing is that the config is the same i have already used and successfully deployed the stack in the past | 10:38 |
| mattt | gparaskevas: hmm, wonder if something on our end has changed | 10:39 |
| gparaskevas | i dont really knnow | 10:39 |
| gparaskevas | but i did clone the latest OSAD | 10:39 |
| mattt | gparaskevas: https://github.com/stackforge/os-ansible-deployment/blob/master/etc/openstack_deploy/conf.d/swift.yml.aio#L30 | 10:40 |
| mattt | you probably need to set something similar | 10:40 |
| gparaskevas | looks like! | 10:40 |
| gparaskevas | let me check | 10:40 |
| mattt | gparaskevas: what version of os-ansible-deployment are you using btw? | 10:44 |
| mattt | swift_vars according to the docs is optional | 10:44 |
| gparaskevas | i have cloned master and also openstack release is set to master | 10:46 |
| mattt | gparaskevas: sounds like a bug, but i'm not well versed w/ swift so i could be wrong | 10:47 |
| mattt | try setting it in your conf.d file and see if that helps | 10:47 |
| gparaskevas | ok so it worked now | 10:48 |
| gparaskevas | i did define container vars: and swift_vars and i worked | 10:48 |
| gparaskevas | http://paste.openstack.org/show/406577/ -> this is what i did | 10:49 |
| gparaskevas | so maybe it optinal and just needs to be empty but defined | 10:49 |
| gparaskevas | ? | 10:49 |
| mattt | gparaskevas: smells like a bug to me :) | 10:50 |
| mattt | gparaskevas: we can ping odyssey4me when he gets back, he's more familiar w/ swift than i am | 10:51 |
| mattt | i don't want to feed you the wrong information, so perhaps it's best to actually set zone, region, etc. under swift_vars | 10:51 |
| gparaskevas | yes i dont know eiither and i am not ver familiar with sswift | 10:52 |
| gparaskevas | so as you propose! | 10:52 |
| mattt | gparaskevas: can you stick around or swing by in an hour or two so we can ask him? | 10:52 |
| gparaskevas | yeah sure | 10:52 |
| mattt | great, talk to you in a bit! | 10:53 |
| odyssey4me | gparaskevas so I guess you mean glance backend, not cinder backend? | 11:08 |
| *** ig0r__ has quit IRC | 11:15 | |
| gparaskevas | yes glance i am sorry | 11:18 |
| gparaskevas | and the problem was resolved by adding swift_vars and container_vars in swift.yml | 11:19 |
| gparaskevas | let me paste | 11:19 |
| *** ig0r_ has joined #openstack-ansible | 11:19 | |
| gparaskevas | http://paste.openstack.org/show/406577/ | 11:19 |
| gparaskevas | i happened to keep my swift.yml from a past installation which did not contain the swift_vars definition and that seems to be the problem | 11:20 |
| gparaskevas | so i added swift_vars and then it worked | 11:20 |
| gparaskevas | is swift_vars optional or should i always include it? | 11:21 |
| mattt | gparaskevas: it sounds like a bug to me, setting it to an empty value is pointless from a deployer's perspective | 11:27 |
| mattt | gparaskevas: https://github.com/stackforge/os-ansible-deployment/blob/master/playbooks/roles/os_swift/tasks/swift_storage_hosts_setup.yml#L28-L32 | 11:28 |
| mattt | in another part of the code we actually force-set it incase it doesn't exist | 11:29 |
| mattt | so it sounds like a bug, i'll create an issue for this | 11:29 |
| gparaskevas | great | 11:29 |
| gparaskevas | is there anything i can do? | 11:29 |
| mattt | gparaskevas: continue to be awesome | 11:30 |
| mattt | gparaskevas: but seriously once i create it you can have a poke around | 11:31 |
| mattt | gparaskevas: otherwise we'll have one of our swift guys see if it's a valid issue or not | 11:31 |
| odyssey4me | that swift_vars error has popped up once in a while but been resolved, or so I thought | 11:32 |
| mattt | odyssey4me: we had a similar issue with one of the swift proxy vars, i know andymccr fixed that | 11:33 |
| odyssey4me | ah, that's what I'm thinking of | 11:34 |
| odyssey4me | well, you do need to designate a zone for each storage host - so I think you actually need some vars otherwise it won't work | 11:35 |
| odyssey4me | oh, odd - you're designating swift_vars for your compute host? | 11:35 |
| odyssey4me | gparaskevas what does your openstack_user_config look like? | 11:36 |
| javeriak | hey guys, is there a haproxy-install.yml equivalent playbook in the kilo branch? like we use in juno | 11:38 |
| javeriak | nevermind, i see it :) | 11:38 |
| odyssey4me | javeriak noddy badge to you :p | 11:39 |
| odyssey4me | gparaskevas are you intentionally putting your compute host as a swift object storage host? is it just for testing because you only have one host? | 11:40 |
| odyssey4me | if so, model your config on this: https://github.com/stackforge/os-ansible-deployment/blob/master/etc/openstack_deploy/conf.d/swift.yml.aio and make sure that you have nothing conflicting in openstack_user_config | 11:40 |
| evrardjp | I can't force me to NOT see there is a space missing on the last 2 lines of your paste gparaskevas | 11:43 |
| odyssey4me | lol evrardjp | 11:47 |
| gparaskevas | lol | 11:51 |
| gparaskevas | laughed really hard let me check and add it | 11:53 |
| odyssey4me | gparaskevas that's unlikely to be the issue though - you should not have to add swift_vars to your compute node | 11:54 |
| odyssey4me | assuming that you have storage hosts specified somewhere? | 11:54 |
| gparaskevas | its compute and swift | 11:54 |
| openstackgerrit | Serge van Ginderachter proposed stackforge/os-ansible-deployment: Add Ceph/RBD support to playbooks https://review.openstack.org/181957 | 11:55 |
| gparaskevas | the name is compute1 | 11:55 |
| odyssey4me | ok, then check the AIO examples for reference - that's how we do gate testing so that defines a fairly minimal set of vars which need to be set | 11:55 |
| openstackgerrit | Jean-Philippe Evrard proposed stackforge/os-ansible-deployment: Documentation of the ChangeID Id5a74db2399166af2d6ac289b71ebb0de04f5679 about static routes https://review.openstack.org/207849 | 11:55 |
| gparaskevas | its all for evaluation this is no prodaction and its not AIO | 11:55 |
| gparaskevas | its actually 4 vms | 11:56 |
| gparaskevas | thanks again very much! | 11:56 |
| odyssey4me | gparaskevas sure, but use it as a reference for a very minimal setup - that is a base set of required variables | 11:57 |
| odyssey4me | check the structure carefully too | 11:57 |
| gparaskevas | ok | 11:57 |
| evrardjp | so, I have a few hours to test federation support | 12:04 |
| odyssey4me | evrardjp haha | 12:04 |
| odyssey4me | installing it is simple enough, understanding how it works and consuming it is another matter | 12:05 |
| evrardjp | I don't start from 0, I already have a test federation not linked to openstack :p | 12:05 |
| odyssey4me | we've made the install dead simple | 12:05 |
| odyssey4me | ok, do you have an existing IDP then? | 12:05 |
| evrardjp | yup | 12:05 |
| evrardjp | lots of them :D | 12:05 |
| evrardjp | I don't know anything about the federation IN openstack though | 12:06 |
| evrardjp | like, where are the metadata file, the generation timers, how to switch from standard keystone to a federated SP in keystone... | 12:07 |
| evrardjp | I'll check the commit | 12:07 |
| evrardjp | and the blueprints | 12:07 |
| odyssey4me | evrardjp that's no issue - with federation in keystone the built-in keystone is still an option for auth | 12:07 |
| odyssey4me | all we do is add another optional auth layer on top of keystone | 12:08 |
| evrardjp | with shibd? | 12:08 |
| odyssey4me | if you want to setup a Keystone SP for an IDP, all we support right now is the use of saml2 tokens | 12:08 |
| evrardjp | mod_mellon? | 12:08 |
| evrardjp | ok | 12:08 |
| evrardjp | seems fair | 12:08 |
| odyssey4me | all we've instrumented so far is the Shibboleth2 SP configuration, ie mod_shibd | 12:09 |
| odyssey4me | also key to note is that if you want WebSSO, Horizon/Keystone currently only support one IDP for WebSSO | 12:09 |
| odyssey4me | you can have as many IDP's as you like for CLI auth though | 12:09 |
| odyssey4me | and another thing - most of the CLI tooling is not yet in kilo, but is being prepped up and should be available around liberty | 12:10 |
| odyssey4me | we've provided a convenience script that uses curl to do the missing pieces | 12:10 |
| evrardjp | I'm not sure to understand what you meant by "only support one IdP for websso" "you can have as many IDP's as your like for CLI auth" | 12:12 |
| evrardjp | If I have a DS, can't I bring my IdP to auth on the openstack keystone SP? | 12:12 |
| evrardjp | IdPs (plural form) | 12:12 |
| evrardjp | on horizon | 12:12 |
| odyssey4me | evrardjp ah, if you have a DS then yes you can handle multiple IDP's for SSO | 12:13 |
| evrardjp | if it's only for CLI, that's gonna cause me a pain, as I have multiple IdPs that should be able to auth on horizon using SAML2 | 12:13 |
| odyssey4me | that's not something we baked into the testing, so we may need a patch set to enable that nicely :) that'll be an opportunity for you to add another patch! | 12:13 |
| evrardjp | :D | 12:14 |
| evrardjp | good to know what has been done and what hasn't | 12:14 |
| odyssey4me | our docs team is busy taking our raw notes and compiling decent docs - we haven't added any to the repo just yet | 12:14 |
| evrardjp | but it changes my priorities | 12:14 |
| odyssey4me | but yeah, if you want to configure keystone as an SP then both myself and miguelgrinberg know the drill well | 12:15 |
| odyssey4me | evrardjp I take it that this is yours? https://review.openstack.org/207849 | 12:15 |
| odyssey4me | I've added a note around the commit message - I can tweak it for you to show you unless you want to take a stab yourself quickly? | 12:15 |
| evrardjp | yeah, my first one... I've seen the warning about size | 12:16 |
| *** subscope has quit IRC | 12:16 | |
| evrardjp | Will correct it | 12:16 |
| odyssey4me | evrardjp thank you for adding! I take it that your legal dept approved contributing? | 12:18 |
| evrardjp | more or less | 12:18 |
| evrardjp | We went to an agreement | 12:18 |
| evrardjp | that's all you have to know | 12:18 |
| evrardjp | ;) | 12:18 |
| odyssey4me | excellent :) | 12:18 |
| evrardjp | for docimpact, what should I mention? Kinda noob on this | 12:19 |
| evrardjp | DocImpact: ?? | 12:19 |
| odyssey4me | just have a line above the change id that has the word 'DocImpact' | 12:19 |
| openstackgerrit | Jean-Philippe Evrard proposed stackforge/os-ansible-deployment: Documentation of the static-route feature https://review.openstack.org/207849 | 12:20 |
| odyssey4me | it will generate a bug for the project which we assign to the docs team and they will review the change and perhaps tweak stuff or whatever | 12:20 |
| odyssey4me | evrardjp perfect :) | 12:20 |
| evrardjp | np It's now part of my job too to have a good doc ;) | 12:21 |
| odyssey4me | evrardjp so now whenever you have some time to kill you can look at the patch submissions and comment/vote on them: https://review.openstack.org/#/q/project:stackforge/os-ansible-deployment+status:open,n,z | 12:27 |
| odyssey4me | :) | 12:27 |
| mattt | svg: you there | 12:27 |
| evrardjp | you know time is the most valuable resource in the world right? ;D | 12:27 |
| odyssey4me | evrardjp time spent reviewing now is less time spent finding bugs later | 12:28 |
| odyssey4me | ;) | 12:28 |
| svg | o/ | 12:28 |
| evrardjp | true | 12:28 |
| mattt | svg: going to make a small change, to add cinder_ceph_client to inventory/group_vars/hosts.yml | 12:28 |
| mattt | it looks like it's not able to pick this var up ceph_client role in os-cinder-install.yml in the pre_tasks | 12:29 |
| mattt | so defaulting it in group_vars also will be ok, like we do w/ nova_ceph_client | 12:29 |
| svg | ok | 12:30 |
| mattt | svg: also think the logic in https://review.openstack.org/#/c/181957/33..34/playbooks/roles/ceph_client/tasks/ceph_all.yml,cm isn't right | 12:30 |
| mattt | svg: wait, n/m .... | 12:30 |
| *** tlian has joined #openstack-ansible | 12:31 | |
| * svg waits | 12:31 | |
| mattt | svg: why would any of those vars be iterable? | 12:31 |
| svg | b/c they are lists of services | 12:31 |
| *** subscope has joined #openstack-ansible | 12:31 | |
| *** javeriak has quit IRC | 12:31 | |
| mattt | svg: wrong variable name? | 12:32 |
| mattt | svg: glance_ceph_client: glance | 12:32 |
| mattt | is the default | 12:32 |
| svg | wtf | 12:32 |
| * svg slaps himself with a fail whale | 12:32 | |
| mattt | svg: WAIT | 12:33 |
| mattt | svg: i was actually on your wave length there, until i realised the variable was wrong | 12:33 |
| mattt | svg: what we could do in ceph_components is set service to service: '{{ glance_service_names|default([]) }}' | 12:34 |
| mattt | svg: or we were thinking of redefining the service list for nova/glance/cinder in ceph_common, so we can limit what actually gets restarted | 12:35 |
| svg | there's a bigger risk actually here | 12:36 |
| svg | I tried re-using those service lists from other roles | 12:36 |
| svg | and those vars are known b/c those other roles where run and parsed earlier | 12:36 |
| svg | but *if* those roles are not ran (like in running onlky ceph tags), I'm actually not sure the vars will still be known | 12:37 |
| svg | so that empty default might kick in at bad times | 12:37 |
| svg | I think it might be better to redefine a list of services hardcoded in the ceph role, and then we can limit it to only the needed services | 12:38 |
| svg | so +1 to your last thought | 12:38 |
| mattt | git-harry: get to it | 12:41 |
| mattt | :) | 12:41 |
| svg | "Make it so" ;) | 12:42 |
| *** bapalm has joined #openstack-ansible | 12:42 | |
| openstackgerrit | Matt Thompson proposed stackforge/os-ansible-deployment: Add Ceph/RBD support to playbooks https://review.openstack.org/181957 | 12:47 |
| gparaskevas | mattt can you help me with this http://paste.openstack.org/show/406589/ | 12:53 |
| *** b3rnard0 is now known as b3rnard0_away | 12:56 | |
| mattt | gparaskevas: oh dear | 12:56 |
| mattt | gparaskevas: give me a sec, i've seen this before | 12:57 |
| gparaskevas | sure | 12:58 |
| openstackgerrit | Serge van Ginderachter proposed stackforge/os-ansible-deployment: Add Ceph/RBD support to playbooks https://review.openstack.org/181957 | 12:58 |
| svg | mattt git-harry pushed my fix on top of mattt's ^^ | 12:59 |
| odyssey4me | gparaskevas ah, one of the neutron wheels is between things again | 13:01 |
| odyssey4me | which branch/tag are you using? | 13:01 |
| gparaskevas | master | 13:02 |
| odyssey4me | ok, so gparaskevas master is tracking liberty so this sort of thing will happen often because the openstack code is constantly changing | 13:02 |
| odyssey4me | it'd be better to build your test environment from the kilo branch as the upstream kilo has less changes. | 13:03 |
| gparaskevas | odyssey4me: ok seems fair | 13:03 |
| odyssey4me | essentially someone's submitted a patch upstream while your repo was figuring out dependencies... the upstream patch merged and changed the db migration steps for neutron, effectively making your set of wheels pretty useless | 13:04 |
| odyssey4me | you can try and run repo-build again and see if the upstream things is solved, but if you want a smoother ride then rather stick with a more stable branch :) | 13:04 |
| gparaskevas | odyssey4me: ok then, i will give a try to kil branch (i guess by changing /inventory/group_vars/all.yml ?) | 13:05 |
| gparaskevas | odyssey4me: kilo* | 13:06 |
| odyssey4me | gparaskevas nope, switch to the kilo branch of os-ansible-deployment | 13:06 |
| gparaskevas | ok | 13:06 |
| odyssey4me | there are loads of upstream dependencies which we set per branch | 13:07 |
| odyssey4me | kilo and master are relatively close in ansible code, so it's not likely that your openstack_deploy config settings should need a change | 13:07 |
| mattt | odyssey4me: yeah that all sounds familiar, i know we've hit this in the past | 13:08 |
| gparaskevas | odyssey4me: sounds logical. rebuilding my repo and giving a last try. i will thne go on with kilo branch of osad. | 13:10 |
| *** javeriak has joined #openstack-ansible | 13:10 | |
| *** TheIntern has joined #openstack-ansible | 13:23 | |
| *** b3rnard0_away is now known as b3rnard0 | 13:23 | |
| *** markvoelker has joined #openstack-ansible | 13:26 | |
| mattt | svg git-harry so agreed that we'll just redefine the service lists in ceph_client ? | 13:27 |
| *** jmckind has joined #openstack-ansible | 13:27 | |
| svg | +1 | 13:27 |
| mattt | svg git-harry : i'll give it a new variable name to not conflict with what already exists | 13:27 |
| git-harry | mattt: yeah, I was just doing it | 13:27 |
| git-harry | but if you're doing it I'll stop | 13:28 |
| mattt | git-harry: haven't started so i'll let you continue on | 13:28 |
| *** KLevenstein has joined #openstack-ansible | 13:29 | |
| *** javeriak has quit IRC | 13:30 | |
| openstackgerrit | git-harry proposed stackforge/os-ansible-deployment: Add Ceph/RBD support to playbooks https://review.openstack.org/181957 | 13:30 |
| *** markvoelker_ has joined #openstack-ansible | 13:30 | |
| *** markvoelker has quit IRC | 13:31 | |
| *** jmckind has quit IRC | 13:36 | |
| *** jmckind has joined #openstack-ansible | 13:37 | |
| mattt | git-harry: not sure i'd have taken that approach | 13:37 |
| mattt | git-harry: if you have that var in ceph_client and os_glance (for example), which wins? | 13:37 |
| mattt | maybe ceph_cinder_service_names instead? | 13:38 |
| mattt | git-harry: i guess what i'm trying to say is you don't want your var in ceph_client to mess with stuff happening in os_glance | 13:38 |
| mattt | but that's because i dont' know how ansible actually works | 13:38 |
| Sam-I-Am | odyssey4me: is there an rpc equivalent of 207849? | 13:41 |
| jwitko | Hey All, When I want to add something to a deployments nova.conf (force_config_drive=true for example), Which nova container do I add it to? I must add it on all 3 correct? | 13:46 |
| alextricity | jwitko: It would be better to add it to the ansible template and run the nova playbooks, no? | 13:47 |
| jwitko | alextricity, I am trying to keep the osad repo and my changes separate. I have my own ansible repo for applying my customizations | 13:48 |
| alextricity | jwitko: You would have to add it to all nova containers then | 13:48 |
| *** Mudpuppy has joined #openstack-ansible | 13:48 | |
| openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: Properly parse and quote arguments in upgrade script https://review.openstack.org/207687 | 13:48 |
| odyssey4me | jwitko force_config_drive is already set | 13:48 |
| *** Mudpuppy has quit IRC | 13:49 | |
| *** Mudpuppy has joined #openstack-ansible | 13:49 | |
| git-harry | mattt: I don't think they will affect each other but I don't know for sure | 13:49 |
| jwitko | hah, so it is | 13:49 |
| odyssey4me | and if a setting is not available already in the project, then it's best to register a bug to request that it be added, so that it becomes part of the project instead of a customisation that will hurt you when you try to upgrade | 13:49 |
| git-harry | mattt: I'll change it to be safe | 13:50 |
| odyssey4me | then you can work out how to add it through ansible and submit a patch for it :) | 13:50 |
| openstackgerrit | Matthew Kassawara proposed stackforge/os-ansible-deployment: Documentation of the static-route feature https://review.openstack.org/207849 | 13:51 |
| *** Mudpuppy has quit IRC | 13:52 | |
| *** Mudpuppy has joined #openstack-ansible | 13:52 | |
| openstackgerrit | git-harry proposed stackforge/os-ansible-deployment: Add Ceph/RBD support to playbooks https://review.openstack.org/181957 | 13:54 |
| odyssey4me | heh, Sam-I-Am it might have been more polite to ask evrardjp if you could take over his patch before you did it | 13:54 |
| odyssey4me | good edit though :p | 13:54 |
| jwitko | odyssey4me, alrighty. so I also had to set ‘force_gateway_on_subnet=false’, is that one accurate for a bug report? | 13:54 |
| evrardjp | odyssey4me: politeness... so overrated ;) | 13:55 |
| odyssey4me | jwitko yep, for sure | 13:55 |
| jwitko | cool, I’ll submit one. | 13:55 |
| odyssey4me | jwitko cool :) | 13:55 |
| *** yapeng has joined #openstack-ansible | 13:56 | |
| *** spotz_zzz is now known as spotz | 14:01 | |
| mattt | git-harry svg : you guys happy? :) | 14:03 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 14:03 | |
| * svg does his happy dance | 14:04 | |
| * mattt tosses svg a non alcoholic beverage | 14:05 | |
| * svg looks a bit sad to mattt | 14:05 | |
| git-harry | mattt: I'll let you know once I've retested everything from scratch | 14:07 |
| mattt | i probably ought to do the same :-/ | 14:12 |
| gparaskevas | stupid question :is there a specific reason that HAproxy stats arent included? | 14:14 |
| palendae | gparaskevas: Depends on which haproxy you're talking about, but haproxy as an LB for the whole stack isn't really considered production in OSAD | 14:16 |
| palendae | There's also HAproxy solely for keystone with federation, but I'm less familiar with that | 14:16 |
| gparaskevas | i know but even for the testing is quit handy to see the stats :) i told it was stupid | 14:17 |
| gparaskevas | :P | 14:17 |
| palendae | That's probably true | 14:19 |
| palendae | What stats aren't included where? | 14:19 |
| gparaskevas | haproxy can listen o specific port and present stats about backends and frontends | 14:20 |
| *** bapalm_ has joined #openstack-ansible | 14:23 | |
| odyssey4me | gparaskevas palendae haproxy is haproxy - nothing seperate for keystone/federation | 14:24 |
| odyssey4me | all we did was add haproxy ssl offloading support to haproxy | 14:24 |
| palendae | odyssey4me: So it uses the same AIO haproxy? | 14:24 |
| palendae | Ok | 14:24 |
| palendae | For some reason I thought there was independent haproxy load balancing just for keystone now | 14:24 |
| odyssey4me | gparaskevas and no, no reason other than what palendae said - we do already have a wish list bug to add that capability I think | 14:25 |
| odyssey4me | but I might be mixing it up with rabbitmq | 14:25 |
| odyssey4me | palendae haha, no way - there's enough complexity as it is | 14:25 |
| gparaskevas | if there is a bug about it i can fix that | 14:25 |
| odyssey4me | gparaskevas if you can't find a bug, register one - and if you can submit a patch for it, go for it! | 14:26 |
| gparaskevas | ok thanks | 14:26 |
| palendae | https://bugs.launchpad.net/openstack-ansible/+bug/1446432 ? | 14:26 |
| openstack | Launchpad bug 1446432 in openstack-ansible "Enable HAProxy stats Web UI" [Wishlist,Confirmed] | 14:26 |
| *** bapalm has quit IRC | 14:26 | |
| odyssey4me | there we go :) | 14:26 |
| odyssey4me | the patch should be pretty straightforward, so if you work on it then assign the bug to yourself and go ahead | 14:27 |
| gparaskevas | hahah boy that escalated quicly | 14:27 |
| *** andymccr has quit IRC | 14:28 | |
| palendae | open source: "Why isn't this a thing?" "Cause nobody wrote it, here's the bug" :) | 14:29 |
| odyssey4me | gparaskevas you're welcome :) | 14:30 |
| *** yaya has joined #openstack-ansible | 14:33 | |
| *** TheIntern has quit IRC | 14:34 | |
| *** markvoelker_ has quit IRC | 14:40 | |
| *** fawadkhaliq has quit IRC | 14:41 | |
| *** TheIntern has joined #openstack-ansible | 14:42 | |
| sigmavirus24 | palendae: sometimes | 14:45 |
| sigmavirus24 | palendae: other times open source is like "Why isn't this a thing? WHY DIDN'T YOU WRITE THIS YET? I CAN'T WRITE THAT! THAT'S NOT MY JOB" (all in the issue report) | 14:45 |
| palendae | Yeah, there's that too | 14:46 |
| *** klindgren has quit IRC | 14:49 | |
| mattt | git-harrys svg : that job hasn't even started gating yet :( | 14:50 |
| palendae | Betting zuul is still backed way up | 14:50 |
| openstackgerrit | Ian Cordasco proposed stackforge/os-ansible-deployment: Use playbook status to report upgrade failure https://review.openstack.org/207936 | 14:52 |
| openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: Add galera_address to all playbooks installing galera_client https://review.openstack.org/207939 | 14:53 |
| sigmavirus24 | palendae: last night it was down to 600 items in the gate from 6k | 14:53 |
| sigmavirus24 | so I started approving things | 14:53 |
| sigmavirus24 | because yolo | 14:53 |
| palendae | Down an order of magnitude, sounds good to me | 14:54 |
| *** fawadkhaliq has joined #openstack-ansible | 14:54 | |
| sigmavirus24 | Yep | 14:54 |
| sigmavirus24 | It's better | 14:54 |
| sigmavirus24 | but still not good probably | 14:54 |
| openstackgerrit | Merged stackforge/os-ansible-deployment-specs: Limit the distribution of .my.cnf https://review.openstack.org/203754 | 14:54 |
| *** fawadkhaliq has quit IRC | 14:59 | |
| gparaskevas | odyssey4me: i have created a patch based on master branch | 15:00 |
| gparaskevas | odyssey4me: this is rgaridng haproxy stats. how can i submit this for review? | 15:00 |
| odyssey4me | gparaskevas great! have you ever submitted any patches using gerrit before? | 15:01 |
| gparaskevas | odyssey4me: firsttimer here | 15:01 |
| odyssey4me | gparaskevas so it's a bit complicated, but described here: https://github.com/stackforge/os-ansible-deployment/blob/master/CONTRIBUTING.rst | 15:01 |
| odyssey4me | and in links from there as we use the same approach as openstack | 15:02 |
| odyssey4me | see how far you get and we can assist you when you're stuck | 15:02 |
| gparaskevas | odyssey4me: thats ok ! thanks againn very much i will come back for either help or with a submited patch! | 15:03 |
| odyssey4me | gparaskevas :) we look forward to it | 15:03 |
| gparaskevas | odyssey4me: thanks again bye! :) | 15:03 |
| mattt | gparaskevas: i'm heading out shortly, but i can also help you through this next week if you need | 15:04 |
| gparaskevas | mattt: ok i will ping you then if i need any help thnka you! :) | 15:05 |
| mattt | gparaskevas: no worries, have a good weekend | 15:05 |
| *** fawadkhaliq has joined #openstack-ansible | 15:06 | |
| *** subscope has quit IRC | 15:07 | |
| *** gparaskevas has quit IRC | 15:09 | |
| *** andymccr has joined #openstack-ansible | 15:25 | |
| *** yapeng has quit IRC | 15:30 | |
| *** daneyon has joined #openstack-ansible | 15:36 | |
| *** KLevenstein has quit IRC | 15:36 | |
| openstackgerrit | Ian Cordasco proposed stackforge/os-ansible-deployment: Keystone Federation Service Provider Configuration https://review.openstack.org/194395 | 15:37 |
| openstackgerrit | Ian Cordasco proposed stackforge/os-ansible-deployment: Add sample Keystone Federation SP configuration for ADFS https://review.openstack.org/203736 | 15:38 |
| sigmavirus24 | Also zuul is <200 jobs right now | 15:38 |
| sigmavirus24 | so | 15:38 |
| openstackgerrit | Tom Cameron proposed stackforge/os-ansible-deployment: Add new roles to create a router container and custom routes https://review.openstack.org/203683 | 15:47 |
| *** shausy has quit IRC | 15:47 | |
| palendae | odyssey4me, sigmavirus24: Yeah, my mariadb restart is taking ~10 minutes to happen | 15:48 |
| sigmavirus24 | we're not upgrading mariadb as part of this are we? | 15:48 |
| palendae | odyssey4me: fwiw, when I tried last night, I also tried a manual restart of it from within the container, and it took as long | 15:48 |
| sigmavirus24 | that didn't merge into kilo iirc | 15:48 |
| palendae | Not that I know of | 15:48 |
| palendae | The spec is only for liberty anyway | 15:48 |
| sigmavirus24 | okay | 15:48 |
| palendae | Also the ordering should no longer affect this | 15:48 |
| sigmavirus24 | right | 15:48 |
| sigmavirus24 | I think ordering was a red herring of sorts | 15:49 |
| palendae | Yeah | 15:49 |
| palendae | Though I'm still curious why we're not matching the order of greenfield | 15:49 |
| sigmavirus24 | Yeah that's a very valid question IMO | 15:49 |
| palendae | There may be a good reason for it, but I wasn't able to glean it from what I saw there | 15:50 |
| odyssey4me | palendae sigmavirus24 because cloudnull_afk ? | 15:53 |
| palendae | odyssey4me: More curious if there was a concious reason or if it was an oversight | 15:53 |
| palendae | Either way, if it's not material to how upgrades work now, I'm fine with not touching it | 15:54 |
| odyssey4me | palendae no idea - it probably has reasons which are in Kevin's head :p | 15:54 |
| palendae | Sure | 15:54 |
| odyssey4me | I could try and reverse engineer the reasoning, but I think we're beyond that right now. | 15:54 |
| palendae | And now we're learning what we need to do without his head here :) | 15:54 |
| palendae | Yeah, sadly I don't think now's the time | 15:54 |
| palendae | For liberty's upgrades, definitely | 15:54 |
| *** jwagner_away is now known as jwagner | 15:56 | |
| palendae | There we go, mariadb finally restarted | 15:58 |
| *** markvoelker has joined #openstack-ansible | 15:59 | |
| odyssey4me | palendae d34dh0r53 mattt sigmavirus24 cloudnull_afk andymccr hughsaunders we need to make a call on whether we release 11.1.0 today, or shift the release date to monday to give more time to get the last reviews done | 16:00 |
| sigmavirus24 | palendae: fun | 16:00 |
| sigmavirus24 | I'm +1 on giving us more time | 16:00 |
| odyssey4me | IMO we have too many reviews still in flight, many of which are key deliverables | 16:00 |
| sigmavirus24 | == ^ | 16:00 |
| d34dh0r53 | I'm also +1 on more time | 16:01 |
| stevelle | +1 for Monday | 16:01 |
| odyssey4me | essentially if they slip this release, then they slip to 11.2.0 which hasn't got a date set yet | 16:01 |
| palendae | odyssey4me: I am agreed; releasing 11.1.0 right now will mean we tag on broken commits | 16:01 |
| *** markvoelker has quit IRC | 16:02 | |
| odyssey4me | considering the time, we won't get a vote from mattt and cloudnull_afk andymccr and hughsaunders are on leave, so we have agreement | 16:02 |
| odyssey4me | I'll pop an email to the mailing list for reference. | 16:03 |
| palendae | Thanks | 16:03 |
| sigmavirus24 | odyssey4me: see that I replied to your questions about the upgrade process | 16:05 |
| sigmavirus24 | I hope that clarifies things | 16:05 |
| odyssey4me | sigmavirus24 will check, thanks | 16:05 |
| sigmavirus24 | http://lists.openstack.org/pipermail/openstack-dev/2015-July/071099.html for everyone's reference | 16:05 |
| *** yaya has quit IRC | 16:07 | |
| palendae | odyssey4me, sigmavirus24: And my install (head Juno -> head Kilo + patch) is now failing on the same packages | 16:08 |
| sigmavirus24 | on apt-transport-https? | 16:08 |
| palendae | No | 16:08 |
| odyssey4me | palendae it's not packages - it's that the repo server isn't working due to haproxy dropping | 16:09 |
| sigmavirus24 | my juno AIOs are still building | 16:09 |
| palendae | 2nd comment on https://review.openstack.org/#/c/207687/2 | 16:09 |
| odyssey4me | (I guess) | 16:09 |
| palendae | odyssey4me: Yeah, sorry - this is the symptom | 16:09 |
| palendae | However, it's odd that only a handful aren't being found | 16:09 |
| sigmavirus24 | hm | 16:09 |
| odyssey4me | palendae I think it's because those are new packages - the othesrs are still there and don't need downloading | 16:10 |
| palendae | Thus far it's ldappool and oslo.middleware, will probably have to wait for the rest to time out | 16:10 |
| palendae | Ah, good point | 16:10 |
| palendae | Yeah, you're right | 16:10 |
| palendae | The ones that succeed are all 'ok', not 'changed' | 16:10 |
| sigmavirus24 | so setup-infrastructure.yml is failing to clone the repo appropriately? | 16:10 |
| odyssey4me | sigmavirus24 nope, that works | 16:10 |
| palendae | The repo's fine | 16:10 |
| sigmavirus24 | oh it's just hap? | 16:10 |
| palendae | haproxy gets messed up during the upgrade | 16:11 |
| odyssey4me | it's that haproxy won't restart because there's a port conflict | 16:11 |
| palendae | Can't restart it | 16:11 |
| palendae | Can't go through it | 16:11 |
| *** pilgrimstack1 has quit IRC | 16:11 | |
| palendae | It is getting updated for the new containers, because I see the repo_all conf present | 16:11 |
| sigmavirus24 | how is there a port conflict | 16:12 |
| sigmavirus24 | is it not stopped properly? | 16:12 |
| palendae | I wonder if the old containers that we delete aren't being cleaned up | 16:12 |
| odyssey4me | now if only haproxy gave us a log file to work with :/ | 16:12 |
| sigmavirus24 | Or if things are holding connections to that IP+port and we need to stop all services on upgrade or something stupid | 16:12 |
| odyssey4me | sigmavirus24 it's definitely stopped, and the host has no conflicting port that I can find | 16:12 |
| odyssey4me | it's very odd | 16:12 |
| sigmavirus24 | odyssey4me: lsof -i | grep 8181 | 16:13 |
| sigmavirus24 | look for any connections still open to that port | 16:13 |
| palendae | sigmavirus24: That returns nothign for me | 16:13 |
| palendae | We delete: rsyslog_all, nova_api_ec2, nova_spice_console. I don't see haproxy entries for any of those on my AIO | 16:13 |
| sigmavirus24 | yeah on my new Juno AIO I'm not seeing anything listening on that port | 16:14 |
| sigmavirus24 | what about lsof -i | grep haproxy | 16:15 |
| palendae | Nope | 16:15 |
| palendae | [ALERT] 211/161452 (16835) : Starting frontend keystone_service-front: cannot bind socket is the error | 16:16 |
| palendae | I notice both keystone_service-front and keystone_service-back reference 5000, but I don't think that's it | 16:16 |
| sigmavirus24 | fun stuff | 16:17 |
| palendae | lsof -i | grep 5000 shows nothing... | 16:17 |
| sigmavirus24 | so nothing is listening on the port | 16:17 |
| palendae | Hm | 16:17 |
| sigmavirus24 | Does grepping for haproxy show anything? | 16:17 |
| palendae | No | 16:18 |
| sigmavirus24 | d'oh yeah | 16:18 |
| sigmavirus24 | haproxy's not running | 16:18 |
| palendae | keystone_service-front is using 192.168.1.1 | 16:18 |
| odyssey4me | palendae sigmavirus24 narrowed it down, it's the keystone service that's failing | 16:18 |
| palendae | Other services use * | 16:18 |
| Sam-I-Am | palendae: stevelle saw something similar a few nights ago | 16:18 |
| Sam-I-Am | it was because the ip address for the lb was not a valid ip on the host | 16:18 |
| palendae | odyssey4me: Yeah...is it keystone itself or is it the haproxy config file? | 16:18 |
| palendae | Bam | 16:19 |
| palendae | odyssey4me: setting keystone_service-front to use * instead of 192.168.1.1 let haproxy start for me | 16:19 |
| stevelle | good catch | 16:19 |
| odyssey4me | ok, yeah - so the haproxy play sets the service to the particular addresses now - as the public needs to listen on the public address and the internal on the internal | 16:19 |
| sigmavirus24 | So using https://github.com/sigmavirus24/os-ansible-deployment/tree/kilo-upgrades failed on the container-interfaces temporary playbook too | 16:19 |
| Sam-I-Am | namely *_lb_vip_address | 16:20 |
| odyssey4me | so this means that the issue is purely haproxy specific | 16:20 |
| odyssey4me | yep Sam-I-Am | 16:20 |
| *** KLevenstein has joined #openstack-ansible | 16:20 | |
| palendae | odyssey4me: yeah, though hard to give timely feedback on the original bug when we hit it >.< | 16:20 |
| odyssey4me | so the haproxy play uses the external address for keystone_service and creates keystone_internal if the external and internal are different | 16:20 |
| palendae | And confirmed - os_keystone is installing the failed packages now | 16:21 |
| odyssey4me | this is to allow ssl binding for the public address but not the internal one | 16:21 |
| palendae | Then it failed on creating the DB for the service, hah | 16:21 |
| odyssey4me | let me check out the play and see where the issue is being caused as I know this well | 16:21 |
| palendae | sigmavirus24: You had a fix for the keystone DB not being found? | 16:21 |
| odyssey4me | I think this is all related | 16:22 |
| sigmavirus24 | yep | 16:22 |
| odyssey4me | either the external or internal address is being overidden or is wrong | 16:22 |
| sigmavirus24 | so Apsu | 16:22 |
| Apsu | yus | 16:23 |
| palendae | mysql's not even running on the galera container | 16:23 |
| odyssey4me | /etc/openstack_deploy/openstack_user_config.yml: internal_lb_vip_address: 172.29.236.100 | 16:23 |
| odyssey4me | /etc/openstack_deploy/openstack_user_config.yml: external_lb_vip_address: 192.168.1.1 | 16:23 |
| palendae | Gonna try a manual start and see what happens | 16:23 |
| sigmavirus24 | remember how we were splitting the file name being passed into run_lock? well that's not going to work super well because https://github.com/sigmavirus24/os-ansible-deployment/blob/kilo-upgrades/scripts/run-upgrade.sh#L622 | 16:23 |
| palendae | :| | 16:24 |
| odyssey4me | hahaha | 16:24 |
| Apsu | sigmavirus24: Take out the spaces. | 16:24 |
| Apsu | .yml||true | 16:24 |
| Apsu | :P | 16:24 |
| Apsu | I MAKE GUD FEEX | 16:24 |
| palendae | https://youtu.be/ZnHmskwqCCQ | 16:24 |
| sigmavirus24 | Will that actually do the right thing for basename? | 16:24 |
| odyssey4me | good plan, do another patch set in the existing patch | 16:24 |
| odyssey4me | no idea, best to test | 16:25 |
| sigmavirus24 | holy poop it will | 16:25 |
| *** annashen has joined #openstack-ansible | 16:25 | |
| palendae | sigmavirus24: That's...scary | 16:25 |
| sigmavirus24 | palendae: it's because we don't quote that argument to basename | 16:25 |
| sigmavirus24 | oh wait | 16:25 |
| sigmavirus24 | haha | 16:25 |
| sigmavirus24 | that'll do horrible things to that call | 16:25 |
| sigmavirus24 | it will become `basename /tmp/fix_container_interfaces.yml||true .yml` | 16:25 |
| palendae | It's fine | 16:26 |
| odyssey4me | blast - the external address is set by default for juno's AIO: etc/rpc_deploy/rpc_user_config.yml: external_lb_vip_address: 192.168.1.1 | 16:26 |
| sigmavirus24 | so `basename /tmp/fix_container_interfaces.yml` | 16:26 |
| palendae | Everything's fine | 16:26 |
| sigmavirus24 | || true | 16:26 |
| palendae | odyssey4me: Aha! | 16:26 |
| sigmavirus24 | || true .yml | 16:26 |
| palendae | So we need haproxy upgrade steps >., | 16:26 |
| *** sdake has quit IRC | 16:26 | |
| Apsu | sigmavirus24: Yeah, that's not going to work. | 16:26 |
| Apsu | In fact, that whole thing is kind of weirdly built | 16:26 |
| sigmavirus24 | Apsu: agreed | 16:26 |
| odyssey4me | palendae so my feeling is pretty simple - for a real upgrade the deployer should check that the lb addresses are correctly set - even if they match it's ok | 16:27 |
| Apsu | Looking at it all | 16:27 |
| sigmavirus24 | hence using an associative array in a better way | 16:27 |
| palendae | Whole thing assumed we're passing in a file name | 16:27 |
| palendae | Nothing but a filename | 16:27 |
| sigmavirus24 | palendae: and because it wasn't using the right exit status to check, it all succeeded | 16:27 |
| odyssey4me | for an AIO test we need to correct that, but that's only for a gate test - so no need for a patch | 16:27 |
| palendae | odyssey4me: So you're thinking doc note? | 16:27 |
| sigmavirus24 | which is why I'm using that branch | 16:27 |
| odyssey4me | palendae yes, most definitely | 16:27 |
| palendae | odyssey4me: I think there *is* a need for a patch, but maybe not now - we need to have run-upgrade.sh at least gating for success/failure | 16:28 |
| odyssey4me | we can't do an automated upgrade step that happily edits that value | 16:28 |
| odyssey4me | palendae yes, but that'll be in a gate check script which is more specific | 16:28 |
| palendae | Sure | 16:28 |
| sigmavirus24 | right | 16:28 |
| palendae | Ok, so it's future work for a gating check | 16:28 |
| odyssey4me | either that or we need to add a patch to juno to more closely align with how the aio is bootstrapped in kilo | 16:28 |
| palendae | ...which we'd need to replicate when doing upgrade work :) | 16:28 |
| odyssey4me | which may not be a bad idea | 16:28 |
| palendae | Probably not | 16:29 |
| palendae | So is all this related to mariadb's slow start times? | 16:29 |
| palendae | Because even manually, it's been 8 minuts now | 16:29 |
| odyssey4me | let me log a bug for that - I'll work on it as I had some trouble setting up the juno AIO earlier today and would like to see it work more similarly to kilo as the workflow is better | 16:29 |
| palendae | Agreed | 16:30 |
| palendae | odyssey4me: So that'll be 10.1.12, no? | 16:31 |
| odyssey4me | palendae yep | 16:32 |
| palendae | Ok | 16:32 |
| *** alop has joined #openstack-ansible | 16:33 | |
| odyssey4me | palendae https://bugs.launchpad.net/openstack-ansible/+bug/1480390 | 16:34 |
| openstack | Launchpad bug 1480390 in openstack-ansible juno "Align juno's gate-check/bootstrapping more closely to kilo/master" [Medium,Confirmed] - Assigned to Jesse Pretorius (jesse-pretorius) | 16:34 |
| *** sdake has joined #openstack-ansible | 16:37 | |
| sigmavirus24 | So fun story, associative array keys in bash don't fit the use-case well :/ | 16:37 |
| palendae | Rewrite it | 16:38 |
| palendae | Make this thing a playbook | 16:38 |
| palendae | (only half joking) | 16:38 |
| *** annashen_ has joined #openstack-ansible | 16:47 | |
| sigmavirus24 | ah, I wsan't properly declaring my array as an associative array | 16:47 |
| sigmavirus24 | fun | 16:47 |
| sigmavirus24 | well it's deterministic but not based on insert order | 16:48 |
| sigmavirus24 | ugh | 16:48 |
| palendae | 20 minutes now, galera still not restarted | 16:49 |
| sigmavirus24 | no one said upgrades were fast :( | 16:50 |
| palendae | Er, mariadb | 16:50 |
| *** annashen has quit IRC | 16:50 | |
| palendae | That was just me manually trying to start the service | 16:50 |
| * sigmavirus24 knows what you mean | 16:50 | |
| sigmavirus24 | palendae: is your haproxy working? | 16:50 |
| palendae | Yes | 16:50 |
| sigmavirus24 | k | 16:51 |
| palendae | Can hit repo container through it | 16:51 |
| palendae | Gonna try restarting the container | 16:52 |
| *** yaya has joined #openstack-ansible | 16:53 | |
| *** TheIntern has quit IRC | 17:00 | |
| palendae | So sigmavirus24 are you still working on syntax stuff with run-upgrade.sh? | 17:01 |
| sigmavirus24 | about to push a revision | 17:01 |
| sigmavirus24 | so yes | 17:01 |
| palendae | I'm trying to track see what's causing this mariadb service to not start | 17:01 |
| palendae | Ok, cool | 17:01 |
| openstackgerrit | Ian Cordasco proposed stackforge/os-ansible-deployment: Properly parse and quote arguments in upgrade script https://review.openstack.org/207687 | 17:01 |
| sigmavirus24 | palendae: ^ | 17:02 |
| palendae | Thanks | 17:02 |
| palendae | Will probably just throw out this one | 17:02 |
| palendae | Have to manually fix the haproxy thing too | 17:02 |
| *** markvoelker has joined #openstack-ansible | 17:03 | |
| sigmavirus24 | oh yeah, how does oen fix the haproxy bits? | 17:03 |
| sigmavirus24 | can I fix that before kicking off this next test? | 17:04 |
| palendae | Good question, not sure I caught it from odyssey4me | 17:04 |
| odyssey4me | sigmavirus24 palendae hang a sec, let me make you a shortcut fix for it :) | 17:05 |
| palendae | Ok | 17:05 |
| palendae | If I have to manually type something, it's fine | 17:05 |
| sigmavirus24 | == palendae | 17:05 |
| sigmavirus24 | just don't want that to mess with this next run | 17:05 |
| palendae | Right | 17:06 |
| odyssey4me | export PUBLIC_ADDRESS=$(ip -o -4 addr show dev eth0 | awk -F '[ /]+' '/global/ {print $4}'); sed -i "s/external_lb_vip_address:.*/external_lb_vip_address: ${PUBLIC_ADDRESS}/" /etc/openstack_deploy/openstack_user_config.yml | 17:06 |
| odyssey4me | essentially grab the public address and replace 192.168.1.1 in openstack_user_config / rpc_user_config | 17:08 |
| palendae | Yep | 17:08 |
| odyssey4me | in fact, probably best to do that after juno is deployed and before you start the upgrade | 17:09 |
| palendae | Right | 17:09 |
| palendae | That's where I was going to | 17:09 |
| palendae | I consider that an upgrade step | 17:09 |
| palendae | Configuration prior to | 17:09 |
| odyssey4me | yeah, so obviously just replace openstack_deploy/openstack_user_Config with rpc... | 17:10 |
| odyssey4me | palendae fyi when I looked into mariadb not starting right it came down to one container being broken - this is the hazard of AIO's so I wouldn't waste too much time on it | 17:11 |
| palendae | Yeah, torn down | 17:11 |
| sigmavirus24 | https://etherpad.openstack.org/p/osad-juno-kilo-upgrades fwiw | 17:15 |
| sigmavirus24 | I'm using this to track things now that we've got extra steps | 17:15 |
| palendae | Ok | 17:16 |
| palendae | sigmavirus24: So is your github tree different than the gerrit patch now? | 17:18 |
| sigmavirus24 | palendae: no it just has two patches on it | 17:18 |
| sigmavirus24 | it has the one to properly fail in the script | 17:18 |
| sigmavirus24 | to hopefully help us catch further problems | 17:18 |
| sigmavirus24 | and it has the fixes to the syntax | 17:18 |
| palendae | Ok | 17:19 |
| sigmavirus24 | sigh | 17:20 |
| sigmavirus24 | that doesn't work | 17:20 |
| *** fawadkhaliq has quit IRC | 17:23 | |
| *** fawadkhaliq has joined #openstack-ansible | 17:23 | |
| *** markvoelker has quit IRC | 17:26 | |
| *** yaya has quit IRC | 17:35 | |
| odyssey4me | palendae sigmavirus24 FYI 10.1.11 is tagged and the wheels are rebuilding - if you're building a juno box right now it might croak on the repo sync | 17:35 |
| palendae | It's about there, so we'll see | 17:35 |
| sigmavirus24 | hah | 17:36 |
| sigmavirus24 | well that's nice | 17:36 |
| sigmavirus24 | looks like Gerrit bot isn't working anymore | 17:36 |
| sigmavirus24 | palendae: also, head's up, my kilo-upgrades branch updated to include the latest revision of the thing to find the right filename to get the basename from | 17:37 |
| *** sdake has quit IRC | 17:39 | |
| *** sdake has joined #openstack-ansible | 17:40 | |
| *** sdake has quit IRC | 17:44 | |
| *** sdake has joined #openstack-ansible | 17:45 | |
| openstackgerrit | Merged stackforge/os-ansible-deployment: Keystone Federation Identity Provider Configuration https://review.openstack.org/207386 | 17:46 |
| *** jwagner is now known as jwagner_lunch | 17:46 | |
| *** yaya has joined #openstack-ansible | 17:49 | |
| odyssey4me | sigmavirus24 are the fixes in a specific branch? | 17:54 |
| sigmavirus24 | sorry just repushed with the latest version | 17:54 |
| odyssey4me | sigmavirus24 palendae mind voting on two backports quickly? https://review.openstack.org/207189 https://review.openstack.org/207174 | 17:57 |
| *** yaya has quit IRC | 17:59 | |
| openstackgerrit | Ian Cordasco proposed stackforge/os-ansible-deployment: Fix repo section in example config file https://review.openstack.org/207189 | 17:59 |
| sigmavirus24 | sorry palm hit the mousepad on that one | 17:59 |
| sigmavirus24 | accidentally rebased | 18:00 |
| sigmavirus24 | but mattt's +2 stuck trhough | 18:00 |
| odyssey4me | sigmavirus24 yeah, a rebase doesn't destroy votes, only workflow | 18:02 |
| sigmavirus24 | yep | 18:02 |
| sigmavirus24 | Didn't mean to introduce any noise though is all | 18:03 |
| openstackgerrit | Merged stackforge/os-ansible-deployment: Enable Horizon to consume a Keystone v3 API endpoint https://review.openstack.org/206575 | 18:04 |
| openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: Enable Horizon to consume a Keystone v3 API endpoint https://review.openstack.org/208012 | 18:12 |
| *** TheIntern has joined #openstack-ansible | 18:20 | |
| openstackgerrit | Christopher H. Laco proposed stackforge/os-ansible-deployment: Updated juno for new dev work - 07.31.2015 https://review.openstack.org/208013 | 18:21 |
| *** markvoelker has joined #openstack-ansible | 18:28 | |
| *** markvoelker has quit IRC | 18:37 | |
| *** fawadkhaliq has quit IRC | 18:40 | |
| *** sdake has quit IRC | 18:42 | |
| *** sdake has joined #openstack-ansible | 18:44 | |
| palendae | sigmavirus24: So what should I be testing; your github tree or the one in gerrit? | 18:57 |
| palendae | I'm inclined to go gerrit since that's the only I'll be voting on | 18:58 |
| *** KLevenstein has quit IRC | 18:59 | |
| *** fawadkhaliq has joined #openstack-ansible | 19:05 | |
| openstackgerrit | Merged stackforge/os-ansible-deployment: Cleanup unused functions in swift_rings.py https://review.openstack.org/207174 | 19:06 |
| sigmavirus24 | palendae: github has both patches in one convenient branch | 19:07 |
| sigmavirus24 | they should both be up-to-date too | 19:07 |
| odyssey4me | sigmavirus24 which is the other patch? | 19:07 |
| palendae | Oh, well I grabbed the gerrit one | 19:07 |
| palendae | Yeah... | 19:07 |
| palendae | Do we have 2 dependent ones? | 19:07 |
| sigmavirus24 | no | 19:07 |
| sigmavirus24 | one is the script that checks the exit status of the playbook (not echo) | 19:07 |
| sigmavirus24 | the other is the one that fixes the syntax | 19:07 |
| sigmavirus24 | Also https://review.openstack.org/206790 will help us with one of our bugs | 19:08 |
| *** fawadkhaliq has quit IRC | 19:09 | |
| *** yaya has joined #openstack-ansible | 19:12 | |
| sigmavirus24 | Apsu: can you look at my approach in https://review.openstack.org/#/c/207687/1 to find the yml file? | 19:24 |
| Apsu | sigmavirus24: Sure, sec | 19:25 |
| sigmavirus24 | er, that should just be https://review.openstack.org/207687 | 19:26 |
| palendae | Really curious why mariadb is taking forever to restart with kilo | 19:26 |
| *** ig0r_ has quit IRC | 19:26 | |
| sigmavirus24 | the /1 will probably send you to the first patchset | 19:26 |
| Apsu | sigmavirus24: I looked at the last one. Have you tested the == *.yml? I'm concerned that's going to glob the CWD. You probably want a regex comparison | 19:27 |
| sigmavirus24 | Apsu: i did test that and it appears to work | 19:27 |
| sigmavirus24 | I was concerned about the same thing though | 19:27 |
| Apsu | Might be working because [[ is a builtin. | 19:28 |
| Apsu | But a regex would probably still be better... hrm | 19:28 |
| palendae | sigmavirus24: Are you seeing the slow mariadb restarts too? | 19:31 |
| sigmavirus24 | I can't even actually finish building an AIO | 19:31 |
| sigmavirus24 | This one has been stuck here for a while | 19:31 |
| palendae | Building with Juno? | 19:32 |
| sigmavirus24 | or maybe it hasn't | 19:32 |
| sigmavirus24 | Yep | 19:32 |
| palendae | My Juno part's been done | 19:32 |
| palendae | Dunno if I just got lucky with the wheel rebuild though | 19:32 |
| *** openstack has joined #openstack-ansible | 19:35 | |
| *** sdake has quit IRC | 19:38 | |
| sigmavirus24 | One is stuck on updating apt-cache | 19:38 |
| sigmavirus24 | so screw it | 19:38 |
| palendae | That took a while too | 19:38 |
| sigmavirus24 | rebuild them both | 19:38 |
| sigmavirus24 | or just one | 19:38 |
| sigmavirus24 | the other just started moving again | 19:38 |
| palendae | It took a few minutes for mine to get going | 19:38 |
| palendae | Unsure why all this is so much slower with this patch | 19:38 |
| *** sdake has joined #openstack-ansible | 19:39 | |
| palendae | sigmavirus24: Yeah, still failing to connect to mysql | 19:54 |
| sigmavirus24 | is haproxy up? | 19:54 |
| palendae | Yes | 19:55 |
| sigmavirus24 | and I bet it's not even running, yes? | 19:55 |
| palendae | mysql? nope | 19:56 |
| * sigmavirus24 would love to know why that's failing to come back | 19:56 | |
| palendae | Running setup-infrastructure fails, to | 19:57 |
| palendae | too* | 19:57 |
| *** jmckind has quit IRC | 19:57 | |
| *** jmckind has joined #openstack-ansible | 19:58 | |
| *** jmckind has quit IRC | 19:58 | |
| *** jmckind has joined #openstack-ansible | 19:59 | |
| *** sdake_ has joined #openstack-ansible | 20:01 | |
| *** sdake has quit IRC | 20:05 | |
| palendae | sigmavirus24: I *think* setup-infrastructure is getting skipped | 20:05 |
| sigmavirus24 | still? | 20:05 |
| sigmavirus24 | =( | 20:06 |
| sigmavirus24 | I'd love to know why these AIOs on performance boxes are taking so damn long | 20:07 |
| palendae | Yeah | 20:07 |
| palendae | I deleted my mysql container, re-ran lxc-container-create and setup-infrastrucutre | 20:08 |
| palendae | sigmavirus24: Yep, setup-openstack is proceeding past keystone | 20:09 |
| sigmavirus24 | bizarre | 20:10 |
| palendae | Guess I'll spin up another to see if I get the failure again and try the setup-infrastructure step without rebuilding the container | 20:11 |
| sigmavirus24 | So Apsu https://gist.github.com/sigmavirus24/80866bd6843ff4159fea contains only the logic I care about for the purpose of fixing this upgrade script problem. If you grep for 'FOUND' in the output, it is picking up the appropriate bits afaict | 20:24 |
| sigmavirus24 | So that == *.yml is working it seems | 20:24 |
| Apsu | sigmavirus24: Ok then. | 20:25 |
| Apsu | I'm good with it for this purpose | 20:25 |
| sigmavirus24 | yeah | 20:25 |
| sigmavirus24 | That was also "confirmation" that setup-infrastructure.yml should be getting called by the script for palendae | 20:26 |
| sigmavirus24 | while I totally believe it isn't | 20:26 |
| palendae | Sure - I'm building another AIO too | 20:27 |
| *** TheIntern has quit IRC | 20:27 | |
| sigmavirus24 | I'm kind of mad at bash | 20:28 |
| sigmavirus24 | also wondering if it's so inappropriate to just document the steps for an upgrade at this point instead of relying on this shell script | 20:29 |
| palendae | rpc-openstack calls the script | 20:29 |
| palendae | But I hear you | 20:29 |
| * sigmavirus24 sighs | 20:30 | |
| Apsu | There's ways to make this much better in bash, just have to rewrite it a bit | 20:31 |
| *** bapalm_ has quit IRC | 20:38 | |
| palendae | sigmavirus24: Now my memcached and galera containers are missing pip packages | 20:41 |
| *** sdake_ is now known as sdake | 20:46 | |
| *** KLevenstein has joined #openstack-ansible | 20:46 | |
| sigmavirus24 | Apsu: so https://gist.github.com/sigmavirus24/aa3934974d8412c35cde happens even with that for-loop checking for .yml | 20:53 |
| palendae | Hrm, my galera and memcached containers on new build aren't getting packages for Juno | 20:54 |
| Apsu | sigmavirus24: Sec | 20:58 |
| *** phalmos has joined #openstack-ansible | 20:59 | |
| Apsu | sigmavirus24: Link me the full script right now? | 20:59 |
| sigmavirus24 | Apsu: https://github.com/sigmavirus24/os-ansible-deployment/blob/kilo-upgrades/scripts/run-upgrade.sh#L667 | 20:59 |
| sigmavirus24 | Er https://github.com/sigmavirus24/os-ansible-deployment/blob/kilo-upgrades/scripts/run-upgrade.sh | 20:59 |
| sigmavirus24 | sans the anchor to a specific line | 20:59 |
| Apsu | Yeah, line 79 | 21:00 |
| sigmavirus24 | we just changed a thing with mariadb right and backported it to kilo/juno right d34dh0r53? | 21:00 |
| Apsu | You're quoting $2 | 21:00 |
| Apsu | Including the || true. | 21:00 |
| sigmavirus24 | oh god damnit | 21:01 |
| Apsu | Having fun yet? | 21:01 |
| sigmavirus24 | that was from before this all started I think | 21:01 |
| sigmavirus24 | whatever | 21:02 |
| openstackgerrit | Ian Cordasco proposed stackforge/os-ansible-deployment: Properly parse and quote arguments in upgrade script https://review.openstack.org/207687 | 21:02 |
| *** phalmos has quit IRC | 21:02 | |
| d34dh0r53 | sigmavirus24: we changed the sst method to xtrabackup-v2, AFAIK that's the only think we've changed in Galera so far | 21:03 |
| sigmavirus24 | d34dh0r53: on which branches? kilo, master, anywhere else? | 21:03 |
| *** phalmos has joined #openstack-ansible | 21:03 | |
| d34dh0r53 | juno, kilo and I think it was ported to master in lieu of the Galera v10 patch | 21:04 |
| sigmavirus24 | so | 21:04 |
| sigmavirus24 | could that be causing problems on upgrade to kilo? | 21:04 |
| sigmavirus24 | I mean, I'd hope not from juno->kilo | 21:05 |
| d34dh0r53 | what's the error? | 21:05 |
| sigmavirus24 | if it's in both, I would hope it wouldn't cause problems | 21:05 |
| sigmavirus24 | d34dh0r53: mariadb just takes forever for it to restart | 21:05 |
| sigmavirus24 | upwards of 10 and 20 minutes | 21:05 |
| palendae | I haven't found anything in the logs to indicate a problem | 21:05 |
| d34dh0r53 | on just 1 container? | 21:05 |
| palendae | Yeah, AIOs | 21:06 |
| *** fawadkhaliq has joined #openstack-ansible | 21:06 | |
| d34dh0r53 | hmm, well, I don't think it's related to the sst thing, that caused the restart to fail quickly | 21:06 |
| palendae | Well that's good, heh | 21:07 |
| d34dh0r53 | but I think we may have something happening with galera data corruption during upgrades, as this sort of thing seemed to happen to QE | 21:07 |
| d34dh0r53 | but their logs indicated that there was a problem | 21:07 |
| d34dh0r53 | mysql was segfaulting | 21:08 |
| palendae | ...oh | 21:08 |
| * sigmavirus24 waves to phalmos | 21:08 | |
| phalmos | Good day sir | 21:08 |
| d34dh0r53 | this seems somewhat coincidental | 21:08 |
| palendae | Indeed | 21:09 |
| sigmavirus24 | d34dh0r53: coincidental but repeatable | 21:09 |
| sigmavirus24 | but could just be that we have 1 galera container | 21:09 |
| d34dh0r53 | true | 21:09 |
| palendae | sigmavirus24: Have you spun up a new juno install yet? | 21:10 |
| palendae | Using head of juno I'm not getting packages | 21:10 |
| sigmavirus24 | I've 2 in progress | 21:10 |
| sigmavirus24 | one I just kicked off | 21:10 |
| sigmavirus24 | Another that's been running slowly | 21:10 |
| *** fawadkhaliq has quit IRC | 21:11 | |
| Apsu | "When in doubt, segfault it out" - Galera | 21:14 |
| sigmavirus24 | Go and brush your core dumps off | 21:15 |
| palendae | sigmavirus24: pip says https://mirror.rackspace.com/rackspaceprivatecloud/python_packages/10.1.11/python_memcached-1.56-py2.py3-none-any.whl is 404 | 21:16 |
| palendae | Hm | 21:16 |
| palendae | https://mirror.rackspace.com/rackspaceprivatecloud/python_packages/10.1.11/python_memcached-1.56-py2.py3-none-any.whl | 21:16 |
| palendae | So the one I coped first is there, the 2nd isn't | 21:16 |
| sigmavirus24 | palendae: getting the same thing | 21:16 |
| palendae | I copied the second from my terminal | 21:17 |
| palendae | The first was a link from my browser | 21:17 |
| palendae | Oh wait, yeah | 21:17 |
| palendae | It's not there | 21:17 |
| sigmavirus24 | I bet this is a mirror issue | 21:18 |
| *** KLevenstein has quit IRC | 21:19 | |
| palendae | yeah | 21:19 |
| sigmavirus24 | as in | 21:19 |
| sigmavirus24 | it hasn't synchronized yet | 21:19 |
| sigmavirus24 | WOOOOOOOO | 21:19 |
| palendae | =\ | 21:19 |
| palendae | yeah, it's that one package that I'm seeing failures for | 21:19 |
| palendae | yay | 21:19 |
| *** Mudpuppy_ has joined #openstack-ansible | 21:22 | |
| *** tlian has quit IRC | 21:23 | |
| *** Mudpuppy has quit IRC | 21:23 | |
| openstackgerrit | Ian Cordasco proposed stackforge/os-ansible-deployment-specs: Added spec to remove upstream repo dependency https://review.openstack.org/203706 | 21:24 |
| openstackgerrit | Ian Cordasco proposed stackforge/os-ansible-deployment-specs: Added spec to remove upstream repo dependency https://review.openstack.org/203706 | 21:25 |
| *** Mudpuppy_ is now known as Mudpuppy | 21:27 | |
| *** yaya has quit IRC | 21:30 | |
| *** bapalm_ has joined #openstack-ansible | 21:31 | |
| *** hiddentoken has joined #openstack-ansible | 21:38 | |
| *** hiddentoken has quit IRC | 21:39 | |
| *** hiddentoken has joined #openstack-ansible | 21:40 | |
| *** jmckind has quit IRC | 21:42 | |
| *** sdake has quit IRC | 21:54 | |
| *** sdake has joined #openstack-ansible | 21:54 | |
| *** bapalm_ has quit IRC | 21:55 | |
| *** sdake has quit IRC | 22:02 | |
| sigmavirus24 | I have a juno AIO and I'm running the upgrade script | 22:08 |
| palendae | Picking up Juno | 22:09 |
| sigmavirus24 | yeah so Apsu's catch was good but still doesn't fix that | 22:13 |
| sigmavirus24 | $2 still makes openstack-ansible think it's openstack-ansible "/tmp/playbook.yml || true" apparently | 22:13 |
| palendae | Huh, stil can't get python memcached | 22:13 |
| Apsu | Must be quoted on the way in | 22:13 |
| stevelle | $2, I thought this was free! | 22:14 |
| Apsu | You'll need to expand it | 22:14 |
| sigmavirus24 | Apsu: *$2? | 22:14 |
| sigmavirus24 | =P | 22:14 |
| palendae | """"""$2""""""" | 22:14 |
| palendae | " | 22:14 |
| Apsu | lolz | 22:15 |
| Apsu | sigmavirus24: So, a quick and dirty way to do it when you know for sure the string contents is simple to eval it. | 22:21 |
| Apsu | If you need to run a command with your parameter as the arg, you'd do: eval "command $2", or similar | 22:21 |
| Apsu | Then bash will parse the string contents as though it were literal | 22:21 |
| sigmavirus24 | ugh so so so ugly | 22:23 |
| Apsu | Mmhmm. Whole thing needs rewritten | 22:24 |
| sigmavirus24 | Is there another way/ | 22:24 |
| sigmavirus24 | Also, I really don't want to hop into #bash | 22:24 |
| *** sdake has joined #openstack-ansible | 22:25 | |
| Apsu | sigmavirus24: Don't have to. Just a moment | 22:25 |
| * sigmavirus24 hung out in there for a summer | 22:26 | |
| sigmavirus24 | It wasn't exactly pleasant | 22:26 |
| Apsu | lol | 22:27 |
| sigmavirus24 | it was fun how many people tried to crash the bot | 22:29 |
| sigmavirus24 | So many tried the fork bomb and that of course doesn't work | 22:29 |
| Apsu | mmhmm | 22:31 |
| sigmavirus24 | Apsu: idea | 22:32 |
| sigmavirus24 | arr=($2); command ${arr[@]} | 22:32 |
| Apsu | Eh | 22:33 |
| Apsu | Won't work if $2 is a string itself :) | 22:34 |
| sigmavirus24 | nope | 22:34 |
| sigmavirus24 | don't work | 22:34 |
| *** mnestheu1 has joined #openstack-ansible | 22:34 | |
| *** mnestheu1 is now known as scarlisle | 22:34 | |
| sigmavirus24 | Apsu: https://gist.github.com/sigmavirus24/71bb720aff5e47a7c6a1 | 22:34 |
| Apsu | Honestly what you're trying to do is convert a string into a commandline, and that's what eval does. It can be used safely if you're not pumping user-supplied strings into it... | 22:35 |
| sigmavirus24 | yeah | 22:36 |
| sigmavirus24 | okay | 22:36 |
| scarlisle | Hello all. I've tried searching for how to update rpc_user_config.yml with an additional vlan range for an existing vlan network, but have so far been unsuccessful. What's the proper syntax for that? | 22:37 |
| Apsu | scarlisle: Using Juno? | 22:38 |
| *** spotz is now known as spotz_zzz | 22:38 | |
| scarlisle | Apsu yessir | 22:38 |
| Apsu | Yeah, we've got a bug about that. The ml2_conf template is broken, doesn't iterate over the vlan range list | 22:38 |
| openstackgerrit | Ian Cordasco proposed stackforge/os-ansible-deployment: Properly parse and quote arguments in upgrade script https://review.openstack.org/207687 | 22:40 |
| scarlisle | I see | 22:41 |
| sigmavirus24 | Apsu: feel free to review ^ | 22:42 |
| palendae | Apsu: I thought that was fixed in kilo? | 22:42 |
| sigmavirus24 | palendae: scarlisle is on Juno though | 22:42 |
| sigmavirus24 | and we can't just tell them to upgrade to kilo now can we =P | 22:43 |
| Apsu | palendae: Indeed it is | 22:43 |
| Apsu | scarlisle: Upgrade to kilo | 22:43 |
| Apsu | sigmavirus24: JUST THAT EASY | 22:43 |
| palendae | Nope, not right now | 22:43 |
| scarlisle | lol | 22:44 |
| Apsu | sigmavirus24: Reviewed. | 22:45 |
| scarlisle | "Upgrade brah, its super simple." <confirm-solved> | 22:45 |
| Apsu | ^ | 22:45 |
| sigmavirus24 | scarlisle: just don't upgrade with scripts/run-upgrade.sh ... yet | 22:46 |
| sigmavirus24 | unless you want to test a patch for us | 22:46 |
| sigmavirus24 | :D | 22:46 |
| sigmavirus24 | (which at least two of us are presently attempting to test | 22:46 |
| scarlisle | hehe | 22:46 |
| sigmavirus24 | but first juno AIOs need to build | 22:46 |
| sigmavirus24 | And you know how fast juno is to build =P | 22:46 |
| * sigmavirus24 apologizes for being too snarky | 22:46 | |
| Apsu | It's fiiiiine. It's all fiiiine | 22:47 |
| sigmavirus24 | Apsu: need some "date +%s" in there | 22:47 |
| sigmavirus24 | like repeated 200 times =P | 22:47 |
| * sigmavirus24 goes to eat dinner | 22:47 | |
| sigmavirus24 | it won't be a snickers | 22:47 |
| Apsu | lol | 22:47 |
| *** Mudpuppy has quit IRC | 22:49 | |
| palendae | sigmavirus24: STILL not getting python_memcached | 22:50 |
| scarlisle | Apsu do you have that bug link handy, by chance? | 22:50 |
| scarlisle | tried searching for it...maybe I'm searching for the wrong things | 22:50 |
| Apsu | sec | 22:51 |
| Apsu | scarlisle: https://bugs.launchpad.net/openstack-ansible/juno/+bug/1475436 | 22:54 |
| openstack | Launchpad bug 1475436 in openstack-ansible juno "VLAN range issue in ml2_conf.ini" [Wishlist,Triaged] | 22:54 |
| Apsu | Looks like there's a workaround that'd probably function, commented near the bottom | 22:54 |
| *** annashen_ has quit IRC | 22:54 | |
| *** annashen has joined #openstack-ansible | 22:55 | |
| scarlisle | gotcha | 22:58 |
| scarlisle | well, I appreciate the help, gents | 22:58 |
| scarlisle | have good evening! | 22:59 |
| Apsu | You too | 23:00 |
| *** scarlisle has quit IRC | 23:00 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 23:26 | |
| *** daneyon has quit IRC | 23:30 | |
| *** daneyon has joined #openstack-ansible | 23:31 | |
| *** phalmos has quit IRC | 23:42 | |
| *** daneyon has quit IRC | 23:46 | |
| *** alop has quit IRC | 23:52 | |
| *** sdake has quit IRC | 23:57 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!