Tuesday, 2015-03-17

*** mahito has joined #openstack-ansible00:04
daneyon_I added swift to my v10 deployment, but it doesn;t appear that Keystone is adding the required swift user/service/endpoint, etc.. I used this playbook: https://github.com/stackforge/os-ansible-deployment/blob/juno/etc/rpc_deploy/conf.d/swift.yml00:12
daneyon_^ In looking at the keystone-add-all-services.yml, I don't see Swift info.00:13
*** vmtrooper has joined #openstack-ansible00:28
*** vmtrooper has quit IRC00:33
*** stevemar has joined #openstack-ansible00:43
*** jmccrory has joined #openstack-ansible00:50
openstackgerritSteve Lewis proposed stackforge/os-ansible-deployment: Enable fatal_deprecations configuration per OpenStack service  https://review.openstack.org/16491100:51
openstackgerritSteve Lewis proposed stackforge/os-ansible-deployment: Enable fatal_deprecations configuration per OpenStack service  https://review.openstack.org/16491100:53
*** jmccrory has quit IRC00:55
cloudnulldaneyon_ in v10 adding swift requires a swift config and rerun all of the swift plays. the keystone service entry is added here: https://github.com/stackforge/os-ansible-deployment/blob/juno/rpc_deployment/playbooks/openstack/swift-proxy.yml#L3100:57
cloudnullmake sure you have something similar to https://github.com/stackforge/os-ansible-deployment/blob/juno/etc/rpc_deploy/conf.d/swift.yml.example00:57
cloudnullthe absolute minimum config required can be seen here: https://github.com/stackforge/os-ansible-deployment/blob/juno/etc/rpc_deploy/conf.d/swift.yml00:58
*** britthouser has quit IRC01:04
*** britthouser has joined #openstack-ansible01:07
openstackgerritMerged stackforge/os-ansible-deployment: Simplify and improve bootstrap/gate/run scripts  https://review.openstack.org/16383701:49
*** MentalRay_ has joined #openstack-ansible02:00
*** vmtrooper has joined #openstack-ansible02:17
Sam-I-AmMentalRay_: hey02:17
*** galstrom_zzz is now known as galstrom02:18
MentalRay_hey :)02:18
Sam-I-AmMentalRay_: didnt know you hung in here02:18
MentalRay_first visit :p02:19
Sam-I-AmMentalRay_: welcome!02:19
MentalRay_thanks thanks02:19
MentalRay_still working?02:19
Sam-I-Amnot... really.02:20
Sam-I-Ammaybe helping someone in #openstack02:20
Sam-I-Amprobably need to catch up on some networking docs work02:20
Sam-I-Amspent the day in training, so i'm really behind on stuff02:20
MentalRay_hehehe ok02:22
*** vmtrooper has quit IRC02:22
*** sigmavirus24_awa is now known as sigmavirus2402:24
*** sdake__ has quit IRC02:32
openstackgerritKevin Carter proposed stackforge/os-ansible-deployment-specs: Initial commit for OS-Ansible-Deployment specs  https://review.openstack.org/16493302:56
cloudnullPlease review that as soon as possible. this is what commit to get our specs repo going. ^02:57
sigmavirus24cloudnull: why do we have testr in there?03:03
sigmavirus24what tests will be run?03:03
cloudnullit runs a build test against the specs03:04
cloudnullit makes sure all of the sections are present, and that the spec is line wrapped at 7903:04
cloudnulland that the spec has been created in a directory that has an index03:04
sigmavirus24hm03:06
sigmavirus24yeah I just happened upon the actual test files03:06
sigmavirus24interesting03:06
cloudnullthe setup was copied from other spec repos. i simply tweaked it for our use case.03:06
sigmavirus24:thumbsup:03:07
openstackgerritSteve Lewis proposed stackforge/os-ansible-deployment: Enable fatal_deprecations configuration per OpenStack service  https://review.openstack.org/16491103:08
sigmavirus24stevelle: so we're just going to break the gate today? That's what we're going to do?03:10
stevelleconfirmed.03:10
stevellebecause apparently I was hungry and distracted03:10
sigmavirus24stevelle: http://weknowmemes.com/wp-content/uploads/2012/06/red-do-you-think-im-smart.jpg03:10
cloudnullhahahah03:11
stevelleAt least I found why I failed the gate.  :thumbsup: to the improved ansible logging03:11
cloudnullhttp://docs-draft.openstack.org/33/164933/1/check/gate-os-ansible-deployment-specs-docs/7875a01//doc/build/html/specs/template.html03:13
* sigmavirus24 points finger at other guy03:13
cloudnulldraft doc, all pretty like03:13
sigmavirus24thank you cloudnull03:13
sigmavirus24I was going to pull that down03:13
openstackgerritKevin Carter proposed stackforge/os-ansible-deployment-specs: Initial commit for OS-Ansible-Deployment specs  https://review.openstack.org/16493303:14
cloudnulli just updated the template to show an example.03:14
sigmavirus24damnit03:14
* sigmavirus24 shakes fist03:15
*** galstrom is now known as galstrom_zzz03:17
*** markvoelker has quit IRC03:29
*** vmtrooper has joined #openstack-ansible03:49
*** britthouser has quit IRC04:02
*** stevemar has quit IRC04:22
*** stevemar has joined #openstack-ansible04:23
*** sigmavirus24 is now known as sigmavirus24_awa04:24
openstackgerritMerged stackforge/os-ansible-deployment: Revise gate script library to report correctly  https://review.openstack.org/16391404:32
*** vmtrooper has quit IRC04:37
*** MentalRay_ has quit IRC04:47
*** ishant|2 has joined #openstack-ansible05:53
*** ishant|2 has quit IRC05:54
*** ishant has joined #openstack-ansible05:54
openstackgerritMerged stackforge/os-ansible-deployment: Remove 'holland' package from the wheel repository  https://review.openstack.org/16482206:11
*** vmtrooper has joined #openstack-ansible06:26
*** vmtrooper has quit IRC06:31
*** sdake has joined #openstack-ansible06:44
*** stevemar has quit IRC06:44
*** sdake has quit IRC06:48
*** markvoelker has joined #openstack-ansible07:04
*** markvoelker has quit IRC07:08
*** MentalRay_ has joined #openstack-ansible07:54
*** MentalRay_ has quit IRC07:54
*** markvoelker has joined #openstack-ansible08:05
*** markvoelker has quit IRC08:10
*** vmtrooper has joined #openstack-ansible08:15
*** vmtrooper has quit IRC08:21
*** mahito has quit IRC08:35
*** markvoelker has joined #openstack-ansible09:06
*** markvoelker has quit IRC09:11
*** vmtrooper has joined #openstack-ansible10:04
*** markvoelker has joined #openstack-ansible10:07
*** vmtrooper has quit IRC10:09
*** markvoelker has quit IRC10:12
*** markvoelker has joined #openstack-ansible11:08
*** markvoelker has quit IRC11:14
*** britthouser has joined #openstack-ansible11:24
*** markvoelker has joined #openstack-ansible11:38
*** galstrom_zzz is now known as galstrom11:51
*** vmtrooper has joined #openstack-ansible11:53
*** vmtrooper has quit IRC11:58
*** KLevenstein has joined #openstack-ansible12:03
*** stevemar has joined #openstack-ansible12:05
*** jaypipes has joined #openstack-ansible12:20
*** galstrom is now known as galstrom_zzz13:07
*** sdake has joined #openstack-ansible13:25
*** sdake__ has joined #openstack-ansible13:27
*** sdake has quit IRC13:31
openstackgerritMerged stackforge/os-ansible-deployment: Ensure return code passes through output trimming  https://review.openstack.org/16448013:41
*** vmtrooper has joined #openstack-ansible13:42
*** vmtrooper has quit IRC13:47
*** sdake has joined #openstack-ansible13:48
*** sdake__ has quit IRC13:51
*** Mudpuppy has joined #openstack-ansible14:00
*** Mudpuppy has quit IRC14:01
*** Mudpuppy has joined #openstack-ansible14:02
openstackgerritJesse Pretorius proposed stackforge/os-ansible-deployment: Adjust method for trimming ansible output debug information  https://review.openstack.org/16380914:05
*** sigmavirus24_awa is now known as sigmavirus2414:07
openstackgerritJesse Pretorius proposed stackforge/os-ansible-deployment: Ensure return code passes through output trimming  https://review.openstack.org/16508514:09
*** ishant has quit IRC14:15
*** galstrom_zzz is now known as galstrom14:56
*** jaypipes has quit IRC15:06
*** erikmwilson is now known as Guest7875715:06
*** erikmwilson has joined #openstack-ansible15:06
*** erikmwilson_ has joined #openstack-ansible15:06
palendaeodyssey4me, cloudnull: Looks like the checksum changes didn't take care of it: http://logs.openstack.org/93/161893/1/check/os-ansible-deployment-dsvm-check-commit/4241128/console.html#_2015-03-17_15_05_18_95915:07
*** erikmwilson has quit IRC15:08
*** erikmwilson has joined #openstack-ansible15:08
palendaeApsu as well ^15:09
palendaeThat's the latest check for https://review.openstack.org/#/c/161893/15:09
Apsulooking15:10
palendaeApsu: I was trying to add the checksum rules to the utility container to see if the intermittent failures went away15:10
palendaeEvidently it doesn't15:11
palendaeI would hope it's not a resource thing15:11
odyssey4mepalendae Apsu yeah, it could be a resource thing - but it could also be something along the lines of many other things - the db not committing in time (the AIO has a 3 node DB cluster), neutron spinning on something (as it does), an IP conflict of some sort?15:13
palendaeYeah..15:13
odyssey4meit may be worth digging into the logs we now collect to see if there're any other bits that show what's going on15:13
palendaeLots of corner cases to chase down. I think it's going to take doing a checkout on an AIO and adding --debug to the runner in addition to log spelunking15:14
ApsuLooks that way15:14
odyssey4methe trouble is the intermittency15:14
palendaeYep15:14
palendaeExactly15:14
palendaeWhich was insanely hard to catch earlier, too, but that was because the --serial change hadn't landed15:15
odyssey4mean option is also to just modify your existing patch into a WIP test patch, and add --debug to the TEMPEST_OPTS/TESTR_OPTS (whichever is appropriate)15:15
palendaeI think it is WIP15:15
palendaeOr do you mean a patch against testing15:15
odyssey4mebut yes, if you can get a fail on a cloudserver AIO you'll have a better tool to diagnose properly - using openstack-infra to figure this out is harder15:16
palendaeYeah15:16
d34dh0r53palendae: IIRC the gate scripts do an iptables -F which may be wiping out your rules15:16
hughsaundersdaneyon_: cinder_service_port is defaulted to 8776 in group_vars/cinder_all, have you overridden that anywhere?15:17
palendaeodyssey4me: The --debug option for tester would be to drop a pdb in to the code and look in the middle of the test15:17
d34dh0r53not 100% sure of the ordering there15:17
palendaed34dh0r53: My current patch is after that15:17
d34dh0r53ok15:17
d34dh0r53just making sure of the obvious15:17
odyssey4mepalendae ah, cloud server test then :)15:17
palendaeodyssey4me: Yep15:17
palendaeYeah, my intention was to inspect the environment the scenario sets up15:18
odyssey4mepalendae Apsu rackertom if you guys could try to urgently figure out the root cause and find a solution for consistant results it'd be appreciated... the network tests are a serious problem in our gating right now15:21
Apsuodyssey4me: Aye15:22
palendaeI'll help where I can, split my brain between class and this15:23
ApsuYeah, I'm going to setup a box15:23
cloudnullcan we get some core reviewers on: https://review.openstack.org/#/c/164714/15:24
cloudnulland this one too: https://review.openstack.org/#/c/164933/15:28
Sam-I-Amthere are network problems?15:28
palendaeSam-I-Am: Yeah, some tempest network tests are getting timeouts15:28
cloudnullrandomly .15:28
Sam-I-Amthe best kind15:29
cloudnullit cycles between, no ping, server fault, and success.15:29
palendaeOh, server faults still?15:29
palendaeCrap15:29
rackertomCool! Do we have any of the setup logs anywhere15:29
palendaerackertom: https://review.openstack.org/#/c/161893/, logs for latest failure at http://logs.openstack.org/93/161893/1/check/os-ansible-deployment-dsvm-check-commit/4241128/15:30
Sam-I-Amwonder if its slight timing issues15:30
*** Bjoern__ has joined #openstack-ansible15:30
palendaeSam-I-Am: A good guess. This is all on AIOs15:30
palendaeBut haven't proven it yet15:30
*** Bjoern__ is now known as BjoernT15:31
palendaeWe certainly had timing issues with it prior to getting all the tests serialized15:31
*** vmtrooper has joined #openstack-ansible15:31
cloudnullpalendae: http://logs.openstack.org/09/163809/7/check/os-ansible-deployment-dsvm-check-commit/11000bf/console.html#_2015-03-17_15_12_44_71015:31
Sam-I-Am172.29.248.9 is a fip?15:31
cloudnullSam-I-Am yes, that should be a float from a flat net15:31
palendaecloudnull: Oof - that's befor ethe test even15:32
Sam-I-Amdid anyone try adding 1 or 2 sheps here?15:32
ApsuSam-I-Am: It's tempest that's failing in its looping tests. So no we haven't modified tempest yet15:32
Sam-I-Amhavent looked at the code, but sometimes i've seen fips take longer to actually get assigned15:32
ApsuI'm digging into it15:32
*** vmtrooper has quit IRC15:35
cloudnullpalendae and it doesnt look like a resource issue either:  http://logs.openstack.org/09/163809/7/check/os-ansible-deployment-dsvm-check-commit/11000bf/console.html#_2015-03-17_15_12_44_71815:36
ApsuYeah, didn't appear to be when we were running it manually and it was still failing15:37
ApsuSo I'm going to try to debug the tests15:37
ApsuOr recreate them manually15:37
ApsuSo I can investigate15:37
daneyon_hughsaunders: I have not overridden the cinder port15:40
rackertomAm I reading that right? It's trying to get .9 running, fails the test, deletes it, and tries again?15:40
hughsaundersdaneyon_: could you put -debug: var=internal_vip_address -debug: var=cinder_service_port before the cinder api content check, and check that the ip and port match your service catalog?15:42
daneyon_Will Swift work using the default swift1/2/3.img instead of actual drives? I am getting the following error: proxy-server: ERROR Insufficient Storage 192.168.222.42:6002/swift3.img (txn: tx18697e8e53ff45b1a9370-005507acdb) (client_ip: 192.168.223.195)15:43
Sam-I-Amdaneyon_: is this a test env?15:44
daneyon_Sam-I-Am: yes15:44
palendaerackertom: Correct - Neutron is provisioning all of this stuff, doing the test, then tearing it all down for the next test15:46
palendaeSo it gets torn down on success or failure15:46
rackertomSo if it's a timing problem then it's only made worse by bringing down what was just set up.15:46
palendaeFor our debugging purposes, yeah.15:47
Sam-I-Amdaneyon_: it probably works enough for tempest15:47
Sam-I-Amso, whatever test it runs15:47
Sam-I-Ami'd imagine those loopback mounts are small15:47
palendaeWhen we had tempest tests running in parallel, I think tests were finishing at different times and tearing stuff down in the middle of other tests15:48
palendaeSorry, Neutron's not provisioning this...Tempest is15:48
palendaeTempest is exercising openstack15:48
daneyon_Sam-I-Am: Surprising it would pass Tempest.15:50
Sam-I-Amdaneyon_: what were you trying to do?15:50
daneyon_Sam-I-Am: I was just trying to do a swift stat... show that Swift works. It would be nice to store 1 glance image. I'm currently hacking around it by copying my 1 image across my 3 controller nodes file store backend... just a cirros image15:51
cloudnullandymccr ^ ping15:52
Sam-I-Amyeah, swift stat should work15:52
andymccrreading up15:53
Sam-I-Ami cant find where those loopbacks are created, so i dont know their size15:53
andymccrdaneyon_: if you want to test the swift plays using .img files it should work fine as long as they are mounted in the correct location15:54
andymccrthe swift plays dont actually do any of the mounting/formatting etc15:54
daneyon_ya, I don;t see any info re: a loopback in fstab/mtab15:56
cloudnullbug triage meeting in this channel in a couple of minutes:  cloudnull, mattt, andymccr, d34dh0r53, hughsaunders, b3rnard0, palendae, Sam-I-Am, odyssey4me, serverascode, rromans, mancdaz, dolphm, _shaps_, BjoernT, claco15:57
cloudnulldaneyon_ our AIO scripts do all that for us when were testing. if this is a deployment on gear you'd have to do that outside of ansible.15:58
cloudnulland then rerun the swift plays.15:58
daneyon_cloudnull: OK15:59
daneyon_From what I could tell, it didn't seem that the loopback gets setup, but I wanted to double check with the group15:59
b3rnard0hello16:00
cloudnullfor sure.16:00
rromans\o16:01
b3rnard0bug triage notes are going to be here: https://etherpad.openstack.org/p/openstack_ansible_bug_triage.2015-03-17-16.0016:01
andymccrdaneyon_: we probably need better docs in the repo to explain the swift side better16:01
cloudnullfirst bug from launchpad: https://bugs.launchpad.net/openstack-ansible/+bug/143087216:02
openstackLaunchpad bug 1430872 in openstack-ansible "Add token dhcp_domain to dhcp_agent.ini template" [Undecided,New]16:02
cloudnullthis is an issue that we can address in master.16:03
cloudnulldo we want to address this in Juno?16:03
BjoernTyes juno16:03
cloudnullIMO Icehouse is no.16:03
d34dh0r530/16:03
BjoernTwe have active deployments16:03
BjoernTno icehouse16:03
odyssey4meseems reasonable and fairly simple to do16:04
cloudnullyup its a simple variable add to the dhcp.ini16:04
cloudnullwho wants it16:04
odyssey4meI'm happy to do it.16:05
cloudnullok.16:05
odyssey4meI think we're missing a lot of people in this triage today.16:05
palendaeopenstack: We are16:05
palendaeEr16:05
palendaeYeah, a bunch of people here not paying attention16:05
palendaeI'd be willing to do this too, but it'll have to wait from me16:05
cloudnullha.16:05
cloudnullmiguelgrinberg, alextricity : https://bugs.launchpad.net/openstack-ansible/+bug/1432742 <- what say you ?16:06
openstackLaunchpad bug 1432742 in openstack-ansible "SoftwareDeployment Heat resource can not poll internal heat API" [Undecided,New] - Assigned to Miguel Alejandro Cantu (miguel-cantu)16:06
cloudnullto change the vip address or not to change? this is the question.16:06
odyssey4mehmm, there's already a patch for this I think: https://review.openstack.org/16478516:06
Sam-I-Ampalendae: neutron consumes 100% brain cpu16:07
miguelgrinbergI think alextricity is off this week, not sure if he'll be online to comment16:07
cloudnulli saw him yesterday .16:07
d34dh0r53he's out the rest of the week16:07
cloudnullok.16:07
miguelgrinbergcloudnull: I'd like to understand his setup, I never needed to change the metadata server16:07
cloudnullok. we should also note that this is now a default that can be changed per the user configuration as needed. so its no longer hard coded.16:08
cloudnullmaybe we leave it as is, and note that he can set heat_metadata_server_url: "{{ heat_service_proto }}://{{ external_lb_vip_address }}:{{ heat_cfn_serv16:09
cloudnullice_port }}" in his user_variables.yml16:09
cloudnullthoughts?16:09
odyssey4mecloudnull that's for master - right?16:09
cloudnullyup16:10
odyssey4meat this point in time it would seem that we don't have enough information to work from16:10
cloudnullso then im going to mark the bug as incomplete.16:10
odyssey4meso add a comment about that for master16:10
odyssey4mebut miguelgrinberg's questions need answering16:10
miguelgrinbergcloudnull: incomplete is fine, we can resume the discussion then he's back16:11
cloudnullyesir .16:11
cloudnulldone.16:13
cloudnullhttps://bugs.launchpad.net/openstack-ansible/+bug/143267016:13
openstackLaunchpad bug 1432670 in openstack-ansible "ldappool required for keystone AD backend" [Undecided,New]16:13
cloudnullthat issue is inprogress hughsaunders is that correct?16:13
odyssey4meyup: https://review.openstack.org/16471516:13
hughsaunderscloudnull: yeah16:13
cloudnullare we backporting to Juno ?16:14
hughsaundersI think that would be sensible16:14
hughsaunderstag added16:14
BjoernTI like Juno, because we have an active deployment on juno using this16:15
cloudnulldone.16:15
cloudnullhttps://bugs.launchpad.net/openstack-ansible/+bug/143243816:15
openstackLaunchpad bug 1432438 in openstack-ansible "Update README notes to reflect the upstream ansible-lxc module merge" [Low,New]16:15
palendaecloudnull: That could probably be pushed way out, maybe even wishlisted16:15
palendaeFor when it actually lands in an ansible release16:15
cloudnullpalendae do you think you can bang out that readme update?16:15
cloudnullok.16:16
daneyon_andymccr: Might not be a bad idea to do something like this: https://github.com/stackforge/puppet-swift/blob/master/manifests/storage/loopback.pp16:16
palendaeSure; I can say it's merged but not released, too16:16
palendaeEither way16:16
cloudnullup to you palendae .16:16
palendaeI could get that out pretty quickly today though16:16
palendaeYeah, I'll do it16:16
cloudnulldone.16:17
daneyon_andymccr: The puppet-swift loopback setup works well for test environments16:17
cloudnullnext: https://bugs.launchpad.net/openstack-ansible/+bug/143287316:17
openstackLaunchpad bug 1432873 in openstack-ansible "Add FDB bridge entry fails if old entry not removed" [Undecided,New]16:17
hughsaundersdaneyon_: the AIO scripts do a similar thing, good for testing: https://github.com/stackforge/os-ansible-deployment/blob/master/scripts/bootstrap-aio.sh#L184-L19916:17
BjoernTjust added the reference to openstack-ansible16:18
odyssey4mecloudnull that looks like an upstream bug - not sure if we can do much about that within os-ansible-deployment?16:18
daneyon_hughsaunders:  thx16:18
BjoernTThis issue is new and caused by neutron l2pop plugin16:18
ApsuIt is an upstream bug.16:19
ApsuThey're not handling the exit code appropriately to make the add call idempotent, or clearing old ones first.16:19
BjoernTodyssey4me: Yes you are right. We did this on other bugs to have a reference. I can remove it if needed16:19
odyssey4mefor this particular kind of bug, unless there is a workaround, I really don't think os-ansible-deployment can do anything - this should be handled in upstream and hopefully will make it into Kilo/Liberty16:20
*** Mudpuppy has quit IRC16:21
Apsuyep16:21
odyssey4me I doubt this would be eligible for a stable backport16:21
*** Mudpuppy has joined #openstack-ansible16:21
*** Mudpuppy has quit IRC16:21
odyssey4meSo I'd vote this as invalid for openstack-ansible.16:21
cloudnullodyssey4me i would suggest that we look into running updated 2014.2.2 with the stable juno release for our bits, it may not be "fixed" but there was some work that was backported for juno that pertained to ml2 + l2 pop.16:22
cloudnullour juno is running 2014.2.116:22
odyssey4mecloudnull I would support that, but that's a minor release update and not specifically to solve this patch16:22
cloudnullfor sure. we'd need to do to 10.1.016:23
cloudnullsorry 10.2.x16:23
odyssey4meit may improve the behaviour though, so I'd say that perhaps it could be a 'related' bug16:23
cloudnullthat sounds sensible to me. other thoughts?16:24
d34dh0r53+116:25
*** jaypipes has joined #openstack-ansible16:27
cloudnulldone.16:27
cloudnullhttps://bugs.launchpad.net/openstack-ansible/+bug/143226916:28
openstackLaunchpad bug 1432269 in openstack-ansible "Adding a new Cinder storage node fails api service check" [Undecided,New]16:28
cloudnullthis was an issue that daneyon_  reported.16:28
cloudnullhughsaunders have you had any luck reproducing this issue?16:28
hughsaunderscloudnull: no, I have tested extensively and can't get the check to fail, even with the exact response tat daneyon_ is getting :(16:29
daneyon_cloudnull: I will be rebuilding my env tomorrow and I will see if i run into the same error16:29
cloudnullh...16:29
hughsaundersdaneyon_: thanks for helping with the debug, will be interesting to see if you hit it again16:29
cloudnullBjoernT have you seen anything like this with your deployments ?16:29
daneyon_hughsaunders: will do16:29
BjoernTNo i did not see this behavior in our environments16:30
cloudnullok, so ill leave that issue open for the time being.16:31
cloudnullnext: https://bugs.launchpad.net/openstack-ansible/+bug/143132416:31
openstackLaunchpad bug 1431324 in openstack-ansible "gate: change ansible to run in virtual environment" [Undecided,New]16:31
*** sigmavirus24 is now known as sigmavirus24_awa16:31
cloudnullthis seems like a sensible change and something that the bootstrap script should be able to take care of.16:32
cloudnullthoughts?16:32
odyssey4mecloudnull the ansible bootstrap or the aio bootstrap?16:32
cloudnullbootstrap-ansible.sh16:32
cloudnulli mean that we can do that as a rule and not as the exception .16:33
odyssey4meI would think that perhaps we could make it optional16:33
cloudnullor maybe make a tunable ?16:33
cloudnulldefault sys install, tunable venv?16:33
odyssey4meif deployers have a dedicated deployment server, then they should be able to deploy without the virtualenv16:33
odyssey4meyep, I think that'd be best16:34
odyssey4methe gate-check-commit can use the tunable16:34
cloudnullsounds good to me.16:34
d34dh0r53+1 to the tunable16:34
cloudnullbp to juno ?16:34
odyssey4meI think that'd be wise. This issue was actually discovered when doing a multi-node juno set of gate tests.16:35
cloudnullok.16:35
odyssey4meideally we should have this before we tag 10.1.316:35
cloudnulltypie typie make it go16:36
cloudnulland we're done here. with regard to new issues.16:37
cloudnulldo we have anything thats open that people want to lobby for ?16:37
BjoernTlet me check16:37
BjoernThttps://bugs.launchpad.net/openstack-ansible/+bug/1416880 probably should be moved into rpcextras?16:38
openstackLaunchpad bug 1416880 in openstack-ansible juno "Iteration over ansible device for disk_utilisation maas check include sr0 device" [Medium,New]16:38
cloudnullyesir16:38
odyssey4me+116:38
BjoernTok.16:38
BjoernThttps://bugs.launchpad.net/openstack-ansible/+bug/141126016:38
openstackLaunchpad bug 1411260 in openstack-ansible trunk "Redirect neutron-ha-tool logs to a file" [Wishlist,New]16:38
b3rnard0thanks BjoernT16:38
BjoernTid like this one added for 10.1.3 as well and 9.0.7 ?16:39
odyssey4meI disagree with doing this the way it's been suggested.16:39
odyssey4meThe tool logs to syslog already, which logs to a file.16:39
cloudnull+116:39
BjoernT-116:39
BjoernTit's nightmare to debug16:40
odyssey4meWe've used the tool verbatim from the source and would rather not be changing it.16:40
cloudnull+216:40
cloudnulllol16:40
hughsaundersmy comment on that bug is out of date16:40
BjoernTlolm the syslog is infected with dhcp messages16:40
odyssey4mein that case perhaps it'd be better to implement some sort of syslog/dnsmasq configuration to manage that.16:41
BjoernTeither way is fine16:41
odyssey4mealternatively, make the att neutron-ha-tool log to a specific syslog facility so that it can be diverted16:42
odyssey4mebut, quite honestly, I don't see why grep -v isn't easy :p16:42
odyssey4meI would vote for icehouse - no fix16:43
cloudnullhttps://github.com/stackforge/os-ansible-deployment/blob/juno/rpc_deployment/roles/neutron_l3_ha/templates/neutron_ha_tool.sh.j2#L34 - so we can change that to a file of sorts.16:43
BjoernTnot on millions of lines the logfiles growing fast16:43
cloudnullbut im with odyssey4me : grep -v dhcp -e neutron /var/log/message ... done.16:44
odyssey4mecloudnull yeah the tool itself is relatively easy to change to use python debugging and thus configurable files.... but is it worth the effort?16:44
BjoernTcloudnull: right. Exposing it to a file makes it available to /openstack/log/<container>16:45
BjoernTsomething we use everyday16:45
BjoernTalso we have more diskspace available than /var/log/syslog16:45
odyssey4mean alternative is to log it to local0-9 and configurs syslog to push that to /openstack/log/whatever16:46
odyssey4medon't those logs get pushed into elasticsearch?16:47
BjoernTyes they do. But elasticsearch is not working still16:48
odyssey4mewell, the simplest and the least moving parts would be to make the tool output to stdout and make the cron job redirect that output to a file16:49
BjoernT+116:49
BjoernTand don't forget logrotate16:49
BjoernTWho's working on https://bugs.launchpad.net/openstack-ansible/+bug/1428833, if we are done with neutron-ha-tool logfile16:50
openstackLaunchpad bug 1428833 in openstack-ansible trunk "Add novnc console support in favor of spice" [High,Triaged] - Assigned to Andy McCrae (andrew-mccrae)16:50
BjoernT?16:50
odyssey4meBjoernT andymccr has logged a bug upstream for it16:51
odyssey4meif spice won't do it, we'll try and get it into horizon16:51
BjoernThmm, as I stated there are many other ossie and web sockify 0.6 still creates zombies16:51
BjoernTI meant issues16:51
cloudnullBjoernT: thats a no go in juno16:52
BjoernTwhy that ?16:52
cloudnullbecause its a large change to the stack16:52
cloudnullwe can see about doing it in trunk but theres not a lot of interest.16:52
BjoernTit still is supported right ? I don't think it is large when we add additional containers and flip the config setting and adding a different F5 pool16:53
odyssey4meuh, what're we talking about here?16:54
BjoernTThat said, RedHat for instance moved back to novnc. We are talking about spice vs novnc16:54
cloudnulladding novnc instead of spice16:54
odyssey4meah, I'm happy to do this for master, and it may backport to juno but will have to only make it in for 10.2.0 - if that16:55
cloudnullif that16:55
odyssey4meit's a feature change, so requires a feature release according to the rules16:55
cloudnull+116:55
odyssey4me(the tagging rules in the release process)16:55
odyssey4methat said, I see that Andy did volunteer to do it - the priority just hasn't been there yet16:56
BjoernTYou can call it what you want we have a broken console right now in RPC>9 and I don't see upstream can fix it fast enough.16:56
odyssey4meit's already marked high for master, so ideally it'll make it into 11.0.016:56
BjoernTPersonally I don't care which console, if we can fix it enough16:56
BjoernTfast enough16:57
odyssey4meBjoernT so as we discussed last round, if it really is that bad for RPC then lobby for it within RPC16:57
BjoernTlol, that issue will affect all users16:58
BjoernTso what's next to discuss?16:59
odyssey4meagreed, but os-ansible-deployment has yet to find it as a priority as the resource allocations have been focused elsewhere by its stakeholders16:59
b3rnard0looks like we are out of time17:00
b3rnard0notes are here: https://etherpad.openstack.org/p/openstack_ansible_bug_triage.2015-03-17-16.0017:00
BjoernTok17:01
cloudnullok so we're done here.17:04
cloudnulltime for the foods17:04
cloudnullthanks everyone :)17:04
*** vmtrooper has joined #openstack-ansible17:20
*** vmtrooper has quit IRC17:25
*** sigmavirus24_awa is now known as sigmavirus2417:28
*** Mudpuppy has joined #openstack-ansible17:37
*** Mudpuppy has quit IRC17:39
*** Mudpuppy has joined #openstack-ansible17:40
palendaeApsu: Any progress on the network stuff?17:40
*** Mudpuppy has quit IRC17:40
*** Mudpuppy has joined #openstack-ansible17:41
openstackgerritNolan Brubaker proposed stackforge/os-ansible-deployment: Update README to reflect state of the lxc module  https://review.openstack.org/16516918:08
*** sdake__ has joined #openstack-ansible18:14
cloudnullcore peoples please review https://review.openstack.org/#/c/164714/  and https://review.openstack.org/#/c/164933/18:16
*** sdake has quit IRC18:18
openstackgerritJesse Pretorius proposed stackforge/os-ansible-deployment: Rename test sets and include more api tests  https://review.openstack.org/16517318:19
cloudnullandymccr hughsaunders odyssey4me d34dh0r53 mattt git-harry meow please > https://review.openstack.org/#/c/164714/ and https://review.openstack.org/#/c/164933/ <- i'd like to see our current blueprints become specs starting today if at all possible.18:21
* odyssey4me is looking18:22
hughsaundersupdated gerrit watched projects to include the specs repo18:30
hughsaundersshould help not missing specs reviews18:31
odyssey4mestill work to be done in that review IMO18:31
d34dh0r53merged the rsyslog-client review, that looks very good to me18:32
hughsaundersodyssey4me: yeah, I removed my +2 after seeing your comments18:32
openstackgerritKevin Carter proposed stackforge/os-ansible-deployment-specs: Initial commit for OS-Ansible-Deployment specs  https://review.openstack.org/16493318:39
cloudnullresponded to comments inline and fixed the couple of nits18:39
odyssey4mealso needs work in https://review.openstack.org/16471418:44
*** sdake has joined #openstack-ansible18:45
*** sdake__ has quit IRC18:48
*** galstrom is now known as galstrom_zzz18:49
openstackgerritKevin Carter proposed stackforge/os-ansible-deployment: Adds rsyslog client role and enables it in all plays  https://review.openstack.org/16471418:52
cloudnullreplied to inline comments and fixed nits ^18:52
odyssey4mecloudnull see published reply in patch set 2 for the rsyslog client18:54
cloudnullsay what ?18:55
odyssey4mesorry - patch set 2 of specs template change18:55
cloudnullodyssey4me: "'Make sure to document how gate testing will be impacted, indicating any changes which will need to be made to scripts, configurations, the AIO bootstrapping, etc. All changes made must be compatible with the AIO gate-check-commit and with a multi-node external CI gate check." ?18:57
odyssey4meyup18:57
cloudnullwhat if its a change in tempest only?18:57
cloudnullIE "It is assumed that if the change is OpenStack related it should be accompanied by some kind of a test in Tempest"18:57
odyssey4meif it's just tempest, then the gate check commit scripts are not impacted and that should be noted.18:58
cloudnullso wouldn't that be covered with "Please discuss how the change will be tested"18:58
odyssey4meessentially I'd like an explicit consideration of how an AIO and the gate checks will be impacted by whatever change.18:58
odyssey4meI think that's only one part - the other part is how gate checks will be impacted.18:59
cloudnullit seems a bit too specific, imo18:59
d34dh0r53odyssey4me: cloudnull: after reading through the comments in os_aggregate_storage.js on patch set 5 of rsyslog-client I'm thinking that we should set rotate and maxage to configurable vars and default them to 14 and 60 respectively.18:59
odyssey4meif a bunch of tempest tests are added, will the gate check actually run those extra tests and does it have the required resources to do so?18:59
cloudnulli would assume so as it would be a change to our tempest plays/runner script.19:00
cloudnullbut we're not testing everything in the aio as is19:01
cloudnullso it may be a smoke test ?19:01
cloudnullor something else ?19:01
odyssey4mecloudnull that's the point - if someone is adding something, gate testing must be deliberately thought through and a note added about how the test are impacted - if it's not being tested in the commit test, then is it tested elsewhere or is it going to be an orphan of some sort19:02
cloudnulldo we provide language for all of the potential scenarios ?19:02
cloudnullbecause thats the road we're going down .19:02
odyssey4meI'm not asking for specialised language, just deliberate mention of gate test impacts19:02
cloudnullok.19:02
odyssey4med34dh0r53 I would think so, but if it's never used then it's just another thing to manage and maintain. So perhaps leaving it as-is for now is ok and it can be made configurable later if that is expressed as a need.19:04
odyssey4meI'm happy with it as-is.19:05
d34dh0r53odyssey4me: yeah, that's fair.  I had that consideration as well.  Kind of a do it now or do it later question.19:05
d34dh0r53odyssey4me: ok, I'm good with it then19:06
*** vmtrooper has joined #openstack-ansible19:08
openstackgerritJesse Pretorius proposed stackforge/os-ansible-deployment: Rename test sets and include more api tests  https://review.openstack.org/16517319:12
openstackgerritKevin Carter proposed stackforge/os-ansible-deployment-specs: Initial commit for OS-Ansible-Deployment specs  https://review.openstack.org/16493319:13
*** vmtrooper has quit IRC19:13
openstackgerritJesse Pretorius proposed stackforge/os-ansible-deployment: Ensure return code passes through output trimming  https://review.openstack.org/16508519:14
openstackgerritKevin Carter proposed stackforge/os-ansible-deployment-specs: Initial commit for OS-Ansible-Deployment specs  https://review.openstack.org/16493319:14
odyssey4me<---- happy with changes cloudnull19:16
cloudnullhahaha19:16
openstackgerritKevin Carter proposed stackforge/os-ansible-deployment-specs: Initial commit for OS-Ansible-Deployment specs  https://review.openstack.org/16493319:19
cloudnulli wasn't, it failed tox, for formatting. so i fixed it ^19:19
cloudnullline wrap over 79 ...19:19
*** KLevenstein_ has joined #openstack-ansible19:20
odyssey4mecloudnull you are a bad man :p19:20
*** KLevenstein has quit IRC19:20
*** KLevenstein_ is now known as KLevenstein19:20
*** KLevenstein has quit IRC19:38
*** KLevenstein has joined #openstack-ansible19:40
*** sdake__ has joined #openstack-ansible20:21
*** sdake has quit IRC20:25
*** sigmavirus24 is now known as sigmavirus24_awa20:44
*** Mudpuppy has quit IRC20:44
*** sdake has joined #openstack-ansible20:45
cloudnullthis has now passed , cores please go review it https://review.openstack.org/#/c/164933/20:46
openstackgerritMerged stackforge/os-ansible-deployment-specs: Initial commit for OS-Ansible-Deployment specs  https://review.openstack.org/16493320:48
*** sdake__ has quit IRC20:49
*** Mudpuppy has joined #openstack-ansible20:56
*** vmtrooper has joined #openstack-ansible20:58
d34dh0r53Is this: https://blueprints.launchpad.net/nova/+spec/io-ops-weight something we'd like to look at for kilofication of nova?21:02
*** vmtrooper has quit IRC21:02
*** sigmavirus24_awa is now known as sigmavirus2421:03
BjoernTd34dh0r53 possibly, customers would like that. Aren't there any libvirt QOS feature supported yet ? Something more interesting ;-)21:04
d34dh0r53:) not in the blueprints that I'm looking at, but I haven't been through all of them yet21:05
d34dh0r53cloudnull: ping21:11
cloudnullpong21:11
d34dh0r53cloudnull: are we creating kilo/approved/<bp-name> for specs?21:11
cloudnulljust kilo/<bp-name>21:12
d34dh0r53👍21:12
cloudnulland if its the first one make sure you add it to an index21:12
d34dh0r53??21:12
sigmavirus24d34dh0r53: eg. index.rst21:12
d34dh0r53ahh21:13
d34dh0r53does index.rst go in specs/ or specs/kilo/?21:16
cloudnullyea we have specs/kilo/*21:22
cloudnullin the index21:23
cloudnullor rather we should have21:23
openstackgerritMerged stackforge/os-ansible-deployment: Add ldappool to keystone packages  https://review.openstack.org/16471521:24
*** KLevenstein has quit IRC21:28
palendaeDoes anyone know if we can make jobs skip gating if they're to specific files? Like, https://review.openstack.org/#/c/165169/ is failing on the gate tests21:28
palendaeIdeally - changes that include only READM.rst could skip a full gate commit check21:28
sigmavirus24palendae: don't think so21:29
ApsuYou get a gate! And You get a gate! Everybody gets a gate!21:30
palendaeHm21:30
*** KLevenstein has joined #openstack-ansible21:31
sigmavirus24Apsu: you get a jenkins-job-builder and you get an out of date project-config21:31
Apsuyey!21:31
palendaeI'm assuming we get the changeset in the gate job, wonder if we could check for that21:31
ApsuThere really should be a nightly job that runs tox to generate new sample configs21:31
sigmavirus24palendae: git show HEAD?21:31
ApsuOr a pre-merge hook21:31
sigmavirus24Apsu: new jenkins job that fails when it's out of date21:32
cloudnullpalendae we're waiting on an upstream commit from infra which will no-op rst and docs21:32
sigmavirus24propose that and watch how angry everyone gets21:32
Apsulolz21:32
palendaecloudnull: Ok, I thought I saw that, but wasn't sure on the status21:32
sigmavirus24Apsu: you'll make TC with that moxie21:32
palendaecloudnull: Cool, that's basically what I wantd, thanks21:32
Apsusigmavirus24: I'd be down with Moxie Marlinspike on TC21:33
* sigmavirus24 too21:33
palendaecloudnull: Ah, found it. https://review.openstack.org/#/c/164271/ I added a +121:36
*** Mudpuppy_ has joined #openstack-ansible21:41
*** Mudpuppy has quit IRC21:44
*** erikmwilson is now known as Guest8460421:59
*** erikmwilson has joined #openstack-ansible21:59
*** sigmavirus24 is now known as sigmavirus24_awa22:14
openstackgerritSteve Lewis proposed stackforge/os-ansible-deployment: Enable fatal_deprecations configuration per OpenStack service  https://review.openstack.org/16491122:15
*** Mudpuppy has joined #openstack-ansible22:20
*** Mudpuppy_ has quit IRC22:23
*** Mudpuppy has quit IRC22:24
*** erikmwilson has quit IRC22:25
daneyon_Even if I deploy multiple haproxy load-balancers, keepalived is not part of the code for HA between the haproxy nodes... correct?22:37
daneyon_Also, any reason why haproxy is not containerized?22:39
matttdaneyon_: there is no keepalived iirc22:40
daneyon_mattt: ya, i don't see it anywhere. I just wanted to make sure22:40
*** erikmwilson_ is now known as erikmwilson22:43
matttdaneyon_: i can't give you a conclusive reason behind us not putting it in a container, but most of our testing is done on cloud instances etc. and we use the routable IP from the instance itself for the haproxy VIP22:43
*** KLevenstein has quit IRC22:44
*** vmtrooper has joined #openstack-ansible22:46
*** vmtrooper has quit IRC22:51
*** stevemar has quit IRC22:52
daneyon_mattt: Thx for the feedback22:56
*** daneyon has joined #openstack-ansible23:18
*** daneyon_ has quit IRC23:21

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!