Tuesday, 2015-03-10

*** sdake_ has joined #openstack-ansible00:00
*** sdake__ has joined #openstack-ansible00:10
*** jmccrory has quit IRC00:11
*** sdake_ has quit IRC00:14
*** vmtrooper has joined #openstack-ansible01:30
*** vmtrooper has quit IRC01:35
*** markvoelker has joined #openstack-ansible02:51
*** daneyon has joined #openstack-ansible03:03
*** vmtrooper has joined #openstack-ansible03:19
*** sdake__ has quit IRC03:23
*** vmtrooper has quit IRC03:24
*** sdake_ has joined #openstack-ansible03:47
*** markvoelker has quit IRC03:53
*** markvoelker has joined #openstack-ansible03:53
*** markvoelker has quit IRC03:57
*** markvoelker has joined #openstack-ansible04:24
*** markvoelker has quit IRC04:41
sdake_quick development question04:55
sdake_in playbooks/roles/os_heat/defaults/main.yml04:55
sdake_is this the file that creates the heat container?04:56
*** vmtrooper has joined #openstack-ansible05:08
*** vmtrooper has quit IRC05:12
*** stevemar has quit IRC05:31
*** markvoelker has joined #openstack-ansible05:38
*** vmtrooper has joined #openstack-ansible06:56
*** vmtrooper has quit IRC07:01
*** openstackgerrit has quit IRC07:35
*** openstackgerrit has joined #openstack-ansible07:36
*** vmtrooper has joined #openstack-ansible08:45
openstackgerritMatt Thompson proposed stackforge/os-ansible-deployment: Set backend_alg for swift_proxy to source  https://review.openstack.org/16291408:46
*** vmtrooper has quit IRC08:51
openstackgerritMatt Thompson proposed stackforge/os-ansible-deployment: Backport 94ba3bc to juno  https://review.openstack.org/16291709:07
openstackgerritMatt Thompson proposed stackforge/os-ansible-deployment: Backport 51abd0b to juno branch  https://review.openstack.org/16292209:17
*** markvoelker has quit IRC09:22
*** sdake_ has quit IRC10:09
*** sdake_ has joined #openstack-ansible10:14
*** vmtrooper has joined #openstack-ansible10:34
*** vmtrooper has quit IRC10:39
*** sdake_ has quit IRC10:51
*** markvoelker has joined #openstack-ansible10:53
openstackgerritMatt Thompson proposed stackforge/os-ansible-deployment: Set backend_alg for swift_proxy to source  https://review.openstack.org/16296711:23
mattt3rd time lucky11:23
cloudnullMorning.11:48
cloudnullSdake_ the containers are created in the lxc_container_create role.11:49
odyssey4mecloudnull: in preparation for an addition to the contributing guidelines, we've got an etherpad to put down some points of view: https://etherpad.openstack.org/p/os-ansible-deployment-backports11:54
odyssey4methe idea is to be a bit more explicit about how backports should be done to settle confusion around cherry picks, mostly11:55
cloudnullNice. I'll give that a look.11:55
*** vmtrooper has joined #openstack-ansible12:23
*** vmtrooper has quit IRC12:28
*** markvoelker has quit IRC12:38
*** markvoelker has joined #openstack-ansible12:39
*** markvoelker has quit IRC12:43
openstackgerritHugh Saunders proposed stackforge/os-ansible-deployment: Revert "Trim ansible's output to console.log"  https://review.openstack.org/16298412:47
*** stevemar has joined #openstack-ansible13:06
odyssey4mehaha - I've solved the dict merge issue for the tunable blueprint13:09
odyssey4mecheck it out: https://gist.github.com/odyssey4me/bb1ae8d6f8b6c00a730a13:09
*** markvoelker has joined #openstack-ansible13:13
openstackgerritgit-harry proposed stackforge/os-ansible-deployment: WIP: Testing hash_behaviour  https://review.openstack.org/16298913:17
*** stevemar has quit IRC13:18
*** stevemar has joined #openstack-ansible13:19
odyssey4megit-harry ^13:20
*** sigmavirus24_awa is now known as sigmavirus2413:25
*** sdake_ has joined #openstack-ansible13:49
sdake_morning13:51
sdake_if i were to add a conditional variable for installing from source vs installing from packages, do folks have a variable name preference13:51
openstackgerritHugh Saunders proposed stackforge/os-ansible-deployment: Restore tempest install branch to pre-refactor SHA  https://review.openstack.org/16300613:52
odyssey4mesdake_ not particularly - I would guess that it'd be discussed properly within a review submitted13:53
sdake_odyssey4me thanks i'll make something up then13:53
odyssey4mesomething like openstack_install_method13:53
odyssey4meor considering that each role is separate, perhaps glance_install_method and others for each role13:54
odyssey4meie namespaced to the role13:54
sdake_the problem with namespaced to the role is it would have to be configured in a bunch of palces right?13:56
matttsdake_: would there be a case where you'd want one service installed via packages and the rest via source, or vice-versa?13:58
sdake_matt I cannot think of any13:58
sdake_either you will want all source or all packages13:58
sdake_the all packages use case comes from "I want support from RHT, so I must use their packages"13:58
sdake_or ubuntu in same scenario13:59
*** Mudpuppy has joined #openstack-ansible14:00
odyssey4mesdake_ sure, it does mean managing it per role - but the user can then override each role individually14:00
sdake_odyssey4me so this is probably stuff that hsould be hashed in a review so folks dont lose cotext :)14:01
odyssey4methe alternative is to use one variable - something like openstack_deploy_method - and use that same variable across all roles, but provide a default per role14:01
sdake_but thanks for the initial suggestion14:01
odyssey4meyup, agreed :)14:01
*** alextricity has quit IRC14:03
odyssey4menote that for 'debug' or 'verbose' we kind-of do that14:03
odyssey4mefor each role, there are defaults14:03
odyssey4mebut we also set a value in https://github.com/stackforge/os-ansible-deployment/blob/master/playbooks/inventory/group_vars/all.yml which the inventory overrides14:04
odyssey4me(I think)14:04
*** vmtrooper has joined #openstack-ansible14:12
sdake_odyssey4me is there an easy way to do an all in one install with only 3 NICs?14:14
*** alextricity has joined #openstack-ansible14:14
odyssey4methe AOI config we have is designed for only 1 NIC14:14
sdake_nice14:14
odyssey4mebut you can adjust it by simply adjusting your interfaces configuration file14:15
sdake_i've read most of hte code base and I didn't come across a doc that describes launching aio14:15
sdake_but admittidly i read quickly14:15
odyssey4mescripts/bootstrap-aio.sh does the AIO host setup14:15
sdake_just to get a feel for structure14:15
sdake_cool thanks!14:15
odyssey4mehttps://github.com/stackforge/os-ansible-deployment/blob/master/development-stack.rst14:16
odyssey4meit's entirely plausible that we may have some inaccurate bits there - so please do submit patches for any fixes you find are necessary14:17
*** vmtrooper has quit IRC14:17
*** markvoelker has quit IRC14:37
*** markvoelker has joined #openstack-ansible14:37
openstackgerritHugh Saunders proposed stackforge/os-ansible-deployment: Add oslo.context to tempest packages  https://review.openstack.org/16302514:40
*** markvoelker has quit IRC14:42
palendaeandymccr: On https://review.openstack.org/#/c/163006/, what are some of the ramifications of pinning the SHA that are unexpected?14:42
andymccrpalendae: the reason for the SHA change was to go back to requirements that were added (oslo.context) but i think its not a direct requirement and is required by another package that is already required in the SHA14:43
andymccrso it wont actually change anything there.14:43
palendaeI've been changing the SHA when doing neutron tempest tests to make sure I actually get tests that exercise the code14:43
palendaefwiw, when I changed it in my environments, I set tempest_git_install_branch in user_variables.yml and got that SHA14:44
andymccrwe should probably set it to something. so it might be worth still making but im not sure it will solve the problem we have now14:45
palendaeGotcha14:46
palendaeI do know on master, it did not have the same scenario tests as the SHA14:46
*** markvoelker has joined #openstack-ansible14:49
*** jaypipes has quit IRC14:53
*** BjoernT has joined #openstack-ansible14:58
BjoernTMorning14:58
palendaeMorning BjoernT14:59
b3rnard0morning14:59
*** stevemar2 has joined #openstack-ansible15:01
*** stevemar has quit IRC15:01
BjoernTlets do this15:01
b3rnard0welcome to another edition of the bug triage15:01
b3rnard0notes will be here: https://etherpad.openstack.org/p/openstack_ansible_bug_triage.2015-03-10-16.0015:02
d34dh0r53mornings15:02
b3rnard0d34dh0r53 and odyssey4me will help with this meeting as cloudnull is at the philly conference15:02
b3rnard0okay, who has a bug they would like to discuss15:03
odyssey4meoh dear, time zone confusion... I was expecting it to be in another hour :)15:04
d34dh0r53my phone says it should be at 1115:04
d34dh0r53which is in an hour15:04
odyssey4meanyway - morning everyone15:04
b3rnard0oh damn you are right. it's set to utc15:05
d34dh0r53yeah15:05
b3rnard0BjoernT, can you attend at 11?15:05
BjoernTunfortunately no, I mail you if I have urgent bugs. From my side it's worth talking about spice and windows/linux issues15:06
d34dh0r53we can hold the main meeting now and I can run another at 11 for UTC15:07
b3rnard0BjoernT: links please15:08
odyssey4methis is the triage list: https://bugs.launchpad.net/openstack-ansible/+bugs?search=Search&field.status=New&orderby=importance&start=015:09
odyssey4mestarting with https://bugs.launchpad.net/openstack-ansible/+bug/143007115:10
openstackLaunchpad bug 1430071 in openstack-ansible "Swift MaaS Replication Checks Failing - Traceback - KeyError: 'replication_type'" [Undecided,New]15:10
odyssey4meandymccr any comment here?15:10
odyssey4melooks like a maas bug to me - needs to be moved to the maas/rpc repo?15:11
andymccryeh it would have to be moved out of os-ansible-deployment15:11
d34dh0r53+115:11
andymccron the legitimacy or not of the bug, im not sure.15:11
odyssey4meright: https://bugs.launchpad.net/openstack-ansible/+bug/142994715:12
openstackLaunchpad bug 1429947 in openstack-ansible "openstack_interface.cfg.example has incorrect configuration for br-vlan" [Undecided,New]15:12
palendaeodyssey4me: I believe that's valid. Apsu, thoughts? ^15:13
odyssey4meyup, this is valid - although low priority I would argue as the .example is effectively documentation15:13
odyssey4meany volunteers to get it fixed quickly? this is low hanging fruit15:13
BjoernTb3rnard0: https://bugs.launchpad.net/openstack-ansible/+bug/142883315:13
openstackLaunchpad bug 1428833 in openstack-ansible "Add novnc console support in favor of spice" [Undecided,New]15:13
d34dh0r53right, that could also be tagged low-hanging-fruit15:13
palendaeodyssey4me: I can grab it15:13
odyssey4megreat, thanks palendae15:13
palendaeAssigned15:14
odyssey4meBjoernT let's finish the new bug triage before we go into anything else (if we have time)15:14
BjoernTsure15:14
BjoernTit was you fyi15:14
odyssey4menext: https://bugs.launchpad.net/openstack-ansible/+bug/142920115:14
openstackLaunchpad bug 1429201 in openstack-ansible "Cinder.conf Default Availability Zone Config Name Incorrect" [Undecided,New]15:14
BjoernTthats a bug yes15:15
odyssey4mefor icehouse, juno and master?15:16
BjoernTjuno and higher15:16
stevellelow hanging fruit, also15:16
odyssey4memaster is fine: https://github.com/stackforge/os-ansible-deployment/blob/master/playbooks/roles/os_cinder/templates/cinder.conf.j2#L5915:16
odyssey4meany volunteers to fix this for juno then?15:17
b3rnard0we are no longer updating icehouse, correct (except for super critical issues)? so any bugs will have to go into juno and higher15:17
odyssey4meany thoughts on severity here?15:18
bgmccollumbasically if you dont specify an AZ, the volume creation fails15:18
bgmccollumannoying in dashboard since "any" is the default selection15:18
BjoernTicehouse does not have az cinder capabilities in ansbile so Juno is ok15:18
odyssey4meso this is actually critical then, or high at least15:18
BjoernT+115:19
palendaeYeah, I agree15:19
odyssey4mevounteers?15:19
stevellex15:19
odyssey4mealright stevelle, allocated15:20
odyssey4menext one: https://bugs.launchpad.net/openstack-ansible/+bug/142883315:20
openstackLaunchpad bug 1428833 in openstack-ansible "Add novnc console support in favor of spice" [Undecided,New]15:20
BjoernTYeah that's a tough one, I guess we need some devs support. I don't think the zombie issues in websockify will fix the horrendous windows experience15:21
odyssey4mewell, this is essentially a wishlist item - but we can prioritise15:22
BjoernTI will test windows 2k12 instances with the latest qxl driver soon, redhat updated the driver if I saw correctly15:22
openstackgerritNolan Brubaker proposed stackforge/os-ansible-deployment: Fix the br-vlan mode in the example interfaces  https://review.openstack.org/16304415:22
BjoernTThis issue is actually pretty serious for customers15:22
odyssey4meall the issues mentioned really need upstream fixes, and andymccr has been working on that15:22
BjoernTUpstream has most of the issues NOT fixed15:23
BjoernTjust websockify15:23
odyssey4mea suitable workaround for the project is to implement novnc - to at least provide options to workaround15:23
odyssey4methis would be a new role and new features15:23
d34dh0r53yeah, I think we probably need to allow the option of novnc15:23
BjoernTright, I still advocate to spend few days looking into the issues before giving up spice15:23
odyssey4meBjoernT yes, the fixes are needed upstream... that's the point15:24
odyssey4meany volunteers to do this for master first? to backport this to juno will need to be a subsequent decision15:24
openstackgerritNolan Brubaker proposed stackforge/os-ansible-deployment: Fix the br-vlan mode in the example interfaces  https://review.openstack.org/16304815:24
bgmccollumcan spice and novnc be run at the same time?15:26
odyssey4mebgmccollum perhaps - they'd have to be deployed in seperate containers15:26
BjoernTno they cant15:26
BjoernTthey are mutually exclusive configure inside the nova.conf, we can run the containers still at the same time15:27
alextricitypulled from the 10.1.2 tag yesterday and just noticed the /etc/rpc_deploy/rpc_user_config.yml file is built for an AIO. whoops15:27
odyssey4meandymccr you volunteering to setup a novnc role?15:27
andymccrodyssey4me:  sure15:27
odyssey4mealextricity yes, that's by design - the .example file is the documentation15:28
odyssey4meright, next: https://bugs.launchpad.net/openstack-ansible/+bug/142863915:29
openstackLaunchpad bug 1428639 in openstack-ansible "Nova requires python-libguestfs in Kilo" [Undecided,New]15:29
palendaeI think any 'in Kilo' bugs should be associated with their appropriate blueprint15:29
andymccr+115:29
odyssey4mesure, when the blueprint has been registered15:29
odyssey4meimportance?15:29
odyssey4memedium for now?15:30
palendaeYeah15:30
d34dh0r53medium or high, not sure if this blows up every time15:30
odyssey4meit's still a month until the first kilo rc upstream15:30
BjoernTwithout that I would guess you can;t spin up instances15:30
palendaeRight, but it's in Kilo15:30
*** sdake__ has joined #openstack-ansible15:30
BjoernTyes15:30
odyssey4meso for the project right now, I'd say medium - it can be reprioritised later15:30
palendaeSo it's not affecting anyone right now15:30
d34dh0r53+115:30
odyssey4mehttps://bugs.launchpad.net/openstack-ansible/+bug/142846915:31
openstackLaunchpad bug 1428469 in openstack-ansible "Neutron rootwarp(s) need to be updated for Kilo" [Undecided,New]15:31
palendae(hopefullly)15:31
odyssey4meagain, same - medium15:31
odyssey4mehappy?15:31
BjoernTyes15:31
d34dh0r53yep15:32
odyssey4meright: https://bugs.launchpad.net/openstack-ansible/+bug/142846615:32
openstackLaunchpad bug 1428466 in openstack-ansible "Upgrade from 9.0.6 to 10.1.2 - nova-manage.log permissions" [Undecided,New]15:32
odyssey4meinteresting - has anyone else seen this?15:32
palendaeI have not15:33
palendaeBut I haven't done many upgrades, either15:33
odyssey4mecan we have a volunteer to triage this?15:33
d34dh0r53I'll take a look at it15:33
odyssey4meany suggestions for importance?15:33
d34dh0r53med for now15:33
*** sdake_ has quit IRC15:34
bgmccollumive seen it twice now15:34
odyssey4methanks d34dh0r5315:34
bgmccollumout of two upgrades15:34
palendae100% for bgmccollum >.<15:34
BjoernTI have not seen this but if we alter playbooks in that way we expect permission than we should be checking those prior15:34
odyssey4meok, marking as confirmed for now until d34dh0r53 can triage15:34
odyssey4menext: https://bugs.launchpad.net/openstack-ansible/+bug/142844515:34
openstackLaunchpad bug 1428445 in openstack-ansible "Neutron needs plugin references removed" [Undecided,New]15:34
odyssey4meah, kilo - moving on15:35
b3rnard0+115:35
odyssey4meright, next non-kilo one: https://bugs.launchpad.net/openstack-ansible/+bug/142810615:35
openstackLaunchpad bug 1428106 in openstack-ansible "Old version of requests being installed post commit 8e6dbd01c9924ebecffdc04f74e4f6d042610ed8" [Undecided,New]15:35
*** jwagner_away is now known as jwagner15:37
odyssey4memattt any comments on priority?15:38
d34dh0r53can this even be fixed pre-kilo?15:39
odyssey4meit looks pretty much like a decision was reached - how should we set the status? triaged?15:39
matttd34dh0r53: not according to cloudnull15:39
d34dh0r53yeah, I would say triaged15:39
odyssey4memedium importance?15:39
matttodyssey4me: that sounds ok to me15:39
d34dh0r53yeah, me too15:39
matttbecause juno/icehouse are unaffected15:39
odyssey4memattt shall I allocate to you?15:39
matttodyssey4me: yeah sure15:39
odyssey4mehttps://bugs.launchpad.net/openstack-ansible/+bug/142843515:40
openstackLaunchpad bug 1428435 in openstack-ansible "Set the default ansible logging to log to a file" [Undecided,New]15:40
odyssey4meisn't this already done?15:40
BjoernTyeah wishlist. Don't even know if this is possible15:40
BjoernTthere was actually a 2nd gub from myself15:41
odyssey4meah - this is done for the AIO only right now15:41
odyssey4mecurrently the AIO has it implemented for the gate checks: https://github.com/stackforge/os-ansible-deployment/blob/master/scripts/bootstrap-aio.sh#L45-L4915:42
BjoernTI would be good to have a standard log over all ansible playboook executions15:42
odyssey4meimportance?15:42
BjoernTlow ?15:43
odyssey4meyeah, I would agree either low or wishlist15:43
d34dh0r53let's go low15:44
odyssey4menext: https://bugs.launchpad.net/openstack-ansible/+bug/142784615:45
openstackLaunchpad bug 1427846 in openstack-ansible "Upgrade 9.0.6 to 10.1.2 - Horizon Static Content Permission Denied" [Undecided,New]15:45
palendaeSounds similar to the other 9.06 -> 10.1.2 bug15:45
BjoernTsame issue as the nova manage bug15:45
BjoernTyes15:45
d34dh0r53yeah, you can give this one to me as well15:45
odyssey4mepriority?15:46
d34dh0r53triaged, medium15:46
odyssey4medone, next: https://bugs.launchpad.net/openstack-ansible/+bug/142789315:46
openstackLaunchpad bug 1427893 in openstack-ansible trunk "Please increate the network neighbour L2 tables " [Undecided,New]15:46
*** nosleep77 has joined #openstack-ansible15:47
odyssey4meI'm guessing that should be 'increase' ?15:47
odyssey4melooks sensible to me for larger deployments15:47
Apsuodyssey4me: Looking.15:47
odyssey4meBjoernT thoughts on priority?15:47
BjoernTlow15:49
BjoernTsorry, my display stalled15:49
d34dh0r53BjoernT: was this in response to a specific issue with a deployment?15:49
BjoernTor wishlist15:50
Apsud34dh0r53: That's my question15:50
BjoernTno not yet15:50
odyssey4mewishlist then?15:50
ApsuI'm reading Mirantis' blog post on the issue that's linked in the bug and I see several things wrong.15:50
BjoernTyes15:50
Apsuodyssey4me: No, I'm not even sure it's reasonable yet or applies to us.15:50
odyssey4meok, Apsu - can you dig into it a bit more15:50
ApsuI'm also not sure it's the solution to their problem.15:50
ApsuYep.15:50
odyssey4meI'll set it as a wishlist item, but leave it on new15:50
ApsuAlrighty.15:51
BjoernTsetting higher net.ipv4.neigh.default.gc_thresh1 reduces arp traffic imho15:51
BjoernTapart from dhcp15:51
odyssey4menext: https://bugs.launchpad.net/openstack-ansible/+bug/142701115:51
openstackLaunchpad bug 1427011 in openstack-ansible "Upgrade from 9.0.6 to 10.1.2 - rabbitmq issue" [Undecided,New]15:51
odyssey4methis problem has a suitable workaround, and the code is actually in icehouse15:51
ApsuBjoernT: Yes it does, but you would need to first identify ARP traffic as an issue of significance. We can talk about it more after meeting.15:52
BjoernTI had only one customer dealing with this issue.15:52
BjoernTapsu: sure15:52
BjoernTI did downgrade manually rabbitmq to 3.4.3  on this customer and pinned it so we won't update and then downgrade again in a newer release15:52
odyssey4methis is essentially something that needs fixing in docs15:53
BjoernTyes15:53
odyssey4meit should be a known issue for anything < 9.0.715:53
BjoernTand 10.1.115:53
BjoernTeverything prior to pinning15:53
odyssey4mehigh importance?15:54
odyssey4meyup - I've added a comment to that effect15:54
BjoernTI'm just falling into this issue with a new deployment. funny15:54
BjoernTWe should add this soon15:54
*** sdake__ has quit IRC15:54
BjoernTto the docs15:54
*** sdake_ has joined #openstack-ansible15:55
odyssey4mewell, os-ansible-deployment has no docs :p15:55
BjoernTyepp I can move it15:56
odyssey4meta15:57
odyssey4mehttps://bugs.launchpad.net/openstack-ansible/+bug/142626015:57
openstackLaunchpad bug 1426260 in openstack-ansible "tempest.scenario.test_minimum_basic.TestMinimumBasicScenario setups fails" [Undecided,New]15:57
matttdon't we run that in the aio gate ?15:58
odyssey4mesigmavirus24 cloudnull git-harry any comment?15:59
d34dh0r53I just commented on that bug15:59
odyssey4memattt yeah, this looks like a gate aio tempest fail15:59
odyssey4metransient though - definitely not consistant15:59
*** markvoelker has quit IRC15:59
*** claco has joined #openstack-ansible15:59
d34dh0r53I think it's a parallel issue15:59
matttk15:59
odyssey4mesigmavirus24 may be right that it relates to the edge case where the port is somehow used16:00
*** markvoelker has joined #openstack-ansible16:00
sigmavirus24I forget what made me think that though16:00
git-harryI didn't look at it in any detail when I logged it but I noticed it had happened several times. I'd say it was a low priority16:00
sigmavirus24Perhaps it was a moment of clarity in the haze of cloud16:00
cloudnullThat's the issue I was trying to resolve by setting aio tempest gating to serial.16:00
sigmavirus24I haven't seen it in a whiel though16:00
odyssey4mewe have a low priority vote - any takers?16:00
d34dh0r53cloudnull: did that help?16:01
sigmavirus24odyssey4me: seconded16:01
cloudnullIt seemed to.16:01
*** vmtrooper has joined #openstack-ansible16:01
odyssey4meany volunteers to try and recreate it after the serial gating tempest patch has landed?16:01
cloudnullI noticed 503 when all ram was consumed and instance was swapping heavily.16:01
*** stevemar2 is now known as stevemar16:01
odyssey4meneed more info on the bug perhaps?16:02
d34dh0r53odyssey4me: we should link the serial patch to this bug16:02
palendaeI've seen that happen, too16:02
palendaeThe 503s16:02
palendaeand the swapping16:02
odyssey4meseen in the gate, or in repeated run testing - it is different16:02
palendaerepeated run testing16:02
odyssey4methis was specifically in the gate16:02
palendaeAh16:02
palendaeThen I have no16:02
palendaenot*16:02
odyssey4merepeated run testing is different because tempest doesn't clean up16:03
* palendae shakes fist16:03
odyssey4meso eventually, especially on an AIO, the system runs out of resources and dies16:03
odyssey4mealright - leaving it as confirmed... someone can volunteer to pick it up at some point16:03
cloudnullThe issue is resolved in master we should bp to branches for gating. If its not already.16:04
Apsuodyssey4me: What resources? And what doesn't it clean up? I see it clean networks and instances, for sure. Which has been an issue in something palendae and I are working on, because it cleans everything we'd like to troubleshoot on error16:04
cloudnullAs for aio dying because tempest idk think we care from a deployment standpoint.16:04
* cloudnull going back to not paying attention. 16:05
*** markvoelker has quit IRC16:05
odyssey4menext: https://bugs.launchpad.net/openstack-ansible/+bug/142625416:05
openstackLaunchpad bug 1426254 in openstack-ansible "Race creating /openstack/log" [Undecided,New]16:05
BjoernTthat's an old issue, I only came across onece16:05
palendaeApsu: I think it's the testr repo, but not sure why that'd chew up RAM16:05
BjoernTonce16:05
*** erikmwilson has joined #openstack-ansible16:05
*** vmtrooper has quit IRC16:06
odyssey4meyeah BjoernT I thought this was resolved ages back16:06
odyssey4megit-harry did you see this recently?16:06
b3rnard0hey odyssey4me, are we going to continue for another hour?16:06
BjoernTit might  be. Didn't see it for few month now16:06
odyssey4mewhoops - out of time, I see now16:06
git-harryI think there was a related bug16:06
odyssey4me4-5 bugs left to quickly go through I guess16:07
git-harryI think hughsaunders fixed the other one but that was prior to the megapatch16:07
d34dh0r53I've seen some other people join, after the remaining bugs we can open it up to see if any of the new folks have any specific bug related issues16:08
odyssey4merelated: https://bugs.launchpad.net/openstack-ansible/+bug/140712116:08
openstackLaunchpad bug 1407121 in openstack-ansible trunk "/openstack directory creation race" [Medium,Fix committed] - Assigned to Hugh Saunders (hughsaunders)16:08
hughsaundershaha16:08
hughsaundersvoluntold16:08
hughsaundersoh, thats the old one..16:09
odyssey4meok, I'll add a link to the old one - but this one needs some sort of triage16:09
odyssey4meany volunteers?16:09
Apsuhughsaunders: Your analysis is likely still correct, though. https://github.com/stackforge/os-ansible-deployment/blob/master/playbooks/roles/lxc_container_create/tasks/container_create.yml is targeted at containers but the directory creation is delegated to the host16:10
ApsuSo presumably the parallel delegates don't serialize just because they're delegated to the same host16:10
ApsuHence, race16:10
ApsuPerhaps a better way to do it is replace the file: module usage with mkdir -p, which won't fail on existing path.16:10
ApsuI don't like it but it looks like the file module assumes some things about state that are time sensitive. Most likely it checks for directory existence and if it's not there tries to make it. If it gets created inbetween those steps, boom16:11
git-harryodyssey4me: sounds like Apsu wants this one16:11
hughsaundersor fail_when: never16:12
Apsuhughsaunders: There's that too.16:12
Apsugit-harry: hughsaunders made a suggestion last, so it's all his16:12
Apsu:)16:12
odyssey4meyeah, hughsaunders has suggested the more ansible-like way16:12
odyssey4meso - any volunteers to work on this?16:12
odyssey4mecomments on importance?16:12
hughsaunderscan assign me16:12
odyssey4menext: https://bugs.launchpad.net/openstack-ansible/+bug/142619116:13
openstackLaunchpad bug 1426191 in openstack-ansible trunk "make keystone ensure endpoints command stateful" [Undecided,New]16:13
odyssey4melooks like a wishlist item to me for now16:14
d34dh0r53+116:15
odyssey4menext: https://bugs.launchpad.net/openstack-ansible/+bug/142595316:15
openstackLaunchpad bug 1425953 in openstack-ansible "Unknown var external_vip_address referenced in os-horizon" [Undecided,New]16:15
*** jmccrory has joined #openstack-ansible16:15
b3rnard0i have to head out for my next meeting but bug notes are here: https://etherpad.openstack.org/p/openstack_ansible_bug_triage.2015-03-10-16.0016:16
odyssey4meis this master only git-harry ?16:17
git-harryodyssey4me: it should be16:17
odyssey4mealright, importance suggestions?16:18
BjoernThttps://bugs.launchpad.net/openstack-ansible/+bug/1427011 can be closed16:20
openstackLaunchpad bug 1427011 in openstack-ansible "Upgrade from 9.0.6 to 10.1.2 - rabbitmq issue" [High,Triaged]16:20
BjoernTmoved to docs-issues16:20
Apsuodyssey4me: Thinking somewhat high, as a broken variable reference will halt a deploy16:20
odyssey4mealright, done - needs confirmation and a volunteer, but that doesn't have to happen now16:21
stevellex16:21
git-harry:q16:21
odyssey4mestevelle thanks16:21
odyssey4melast one: https://bugs.launchpad.net/openstack-ansible/+bug/142378316:22
openstackLaunchpad bug 1423783 in openstack-ansible "Horizon is not generating CSS correctly" [Undecided,New]16:22
odyssey4meextensive digging has been done into this, without finding a root cause16:22
BjoernTinterestingly the statements I put in always fixes that16:23
bgmccollumInitial deployments will be OK, but over time, as resources are recompressed, their names diverge16:23
bgmccollumspecifying the HASHING_METHOD to content ensures consistent resource names16:24
odyssey4mebgmccollum oh? are you able to dig a little more into it?16:24
odyssey4mebgmccollum the hashing method is already set16:24
bgmccollumleave a multi-controller install running for a while..it just falls over16:24
bgmccollumodyssey4me: https://github.com/stackforge/os-ansible-deployment/blob/juno/rpc_deployment/roles/horizon_common/templates/local_settings.py16:25
bgmccollumi dont see hasing method for juno16:25
odyssey4mebgmccollum the hashing method is a default in horizon16:26
odyssey4mehttps://github.com/openstack/horizon/blob/stable/juno/openstack_dashboard/settings.py#L21216:26
odyssey4mehttps://github.com/openstack/horizon/blob/stable/icehouse/openstack_dashboard/settings.py#L14516:26
odyssey4meplease see the previous investigation: https://bugs.launchpad.net/openstack-ansible/+bug/140289016:26
openstackLaunchpad bug 1423783 in openstack-ansible "duplicate for #1402890 Horizon is not generating CSS correctly" [Undecided,New]16:27
odyssey4meany my last comment here: https://bugs.launchpad.net/openstack-ansible/+bug/142378316:27
openstackLaunchpad bug 1423783 in openstack-ansible "Horizon is not generating CSS correctly" [Undecided,New]16:27
odyssey4meright now, unless we have a dev to volunteer to try and validate this we're going to have to mark it as needing more info16:27
odyssey4meevery investigation thus far has led us back to the same point - everything is set as it should be and it should just work16:28
ApsuI can try to validate it.16:28
odyssey4meand we have nothing else to go on without more detailed triage16:28
ApsuAlready looking at the code16:28
*** daneyon_ has joined #openstack-ansible16:29
odyssey4meApsu ok, I'll allocate to you - just look carefully through all the work that's already been done on it please.16:29
Apsuodyssey4me: Yep, read both bugs fully16:29
odyssey4meI'll mark is as low for now, so we don't go around this bush again - it can be re-classified at a later date.16:30
odyssey4mealright, that's all bugs classified16:30
matttApsu: i can look at that in a future sprint, but i'm tied up w/ work in this sprint16:31
odyssey4methe floor is open to any questions/comments16:31
Apsumattt: np16:31
odyssey4mesorry about rushing through all those - there were rather a lot to go through :)16:31
*** daneyon has quit IRC16:31
odyssey4mealright, if there's nothing more then I think we can say we're done16:33
Apsuodyssey4me: Say it. Say it!16:34
odyssey4melol16:35
odyssey4mewe're done?16:35
ApsuThere it is16:35
ApsuMore conviction next time. Less question marks ;P16:35
odyssey4me#jobdone #meeting_over #hastagssuck16:35
Apsululz16:36
d34dh0r53lol, thanks everyone16:36
odyssey4methanks for keeping the notes after b3rnard0 left d34dh0r5316:37
odyssey4menow, where was I16:38
d34dh0r53sure thing, final notes are here: https://etherpad.openstack.org/p/openstack_ansible_bug_triage.2015-03-10-16.0016:38
odyssey4meI see you're still tweaking them :p16:38
d34dh0r53:) done now16:39
odyssey4memattt the nightly external CI heat multinode tests - does it just run the smoke tests?16:39
odyssey4meor more than that16:39
odyssey4meand do the smoke tests include neutron tests?16:40
*** jaypipes has joined #openstack-ansible16:40
matttodyssey4me: yeah smoke tests do create networks, subnets, routers, etc.16:43
matttodyssey4me: the heat multi node stuff does a smoke test, the original multi node job ran a small subset of api tests only16:43
odyssey4meawesome, thanks16:43
matttnp16:47
openstackgerritAndy McCrae proposed stackforge/os-ansible-deployment: Dont look for used_ips on containerless hosts  https://review.openstack.org/16308316:47
*** jaypipes has quit IRC17:07
*** markvoelker has joined #openstack-ansible17:09
*** markvoelker has quit IRC17:19
*** markvoelker has joined #openstack-ansible17:19
*** sdake_ has quit IRC17:21
*** markvoelker_ has joined #openstack-ansible17:21
*** markvoelker has quit IRC17:24
hughsaunderscloudnull: if I add similarly formatted vars for oslo.context to https://github.com/stackforge/os-ansible-deployment/blob/master/playbooks/vars/repo_packages/openstack_other.yml will the package get built?17:26
*** stevemar has quit IRC17:32
palendaeodyssey4me: Are you around?17:42
*** claco has left #openstack-ansible17:43
andymccrpalendae: he headed home17:44
palendaeandymccr: Thanks17:46
*** vmtrooper has joined #openstack-ansible17:50
*** vmtrooper has quit IRC17:54
alextricityAnsible question: When calling ansible-playbooks, how does ansible load the dynamic inventory?18:01
alextricityIn other words, when and how is dynamic_inventory.py invoked?18:02
*** jaypipes has joined #openstack-ansible18:04
palendaealextricity: Ansible will look for executables in the inventory/ dir and run them, iirc18:06
palendaeAlso I think you can specify in the ansible.cfg file18:07
alextricitypalendae. Yes it's in there. I wasn't sure why it was configured to the inventory/ directory and not the dynamic_inventory.py script. I didn't know that ansible looks for executables in said directory18:08
alextricityThank you18:08
palendaealextricity: Yeah, it's not well documented in ansible itself, sadly18:08
palendaeI think if there's multiple in there, it'll do it in directory listing order (so whatever ls would produce)18:09
alextricitypalendae: haha. Yeah after spending like 30 min trying to find that info in the docs, I just decided to ask here18:09
Apsualextricity: http://docs.ansible.com/intro_dynamic_inventory.html#using-multiple-inventory-sources https://github.com/ansible/ansible/blob/devel/lib/ansible/inventory/dir.py#L5318:19
alextricityApsu, thank you18:20
ApsuEssentially it loops over the directory entries, and recursively through subdirectories that aren't host_vars/group_vars/vars_plugins18:20
ApsuIf it's an executable, it runs it as a dynamic inventory script18:20
ApsuIf it's not a directory and not an executable, it parses it as a static inventory file18:20
ApsuIt merges the results, same as using multiple static inventory files.18:21
bgmccollumApsu, odyssey4me: I have an environment with diverged static resources18:24
Apsubgmccollum: Oo18:24
bgmccollumhttp://paste.openstack.org/show/191369/18:24
Apsubgmccollum: Now, those resources get created on-demand, from what I understand so far18:27
ApsuThe 6c6 files are the same. 8ed appears unique on infra3, and obviously there's lots of uniques on infra118:27
Apsu8ed doesn't appear to have the same size as any of the others18:27
Apsu18c and 8f4 on infra1 are very close in size. Possible they're duplicates18:28
ApsuCan you md5sum those two?18:28
*** sdake_ has joined #openstack-ansible18:28
bgmccollumyeah18:28
bgmccollumhttp://paste.openstack.org/show/191370/18:29
ApsuOh, hah18:29
ApsuThe names Are the md5sums. The prefixes at least.18:29
ApsuSo that tells us one useful thing.18:30
ApsuWith the hash/content method (which appear identical in the django compressor source), we get md5 = filename18:30
ApsuWhich means that if something is missing, it's not because of name divergence.18:30
ApsuIt must be due to a failure to generate or process the resource.18:30
Apsubgmccollum: Can you identify where the unique files on infra1 are referenced in the html, so you can curl/browse those pages on infra2/3 and see if they get generated?18:32
Apsubgmccollum: Or did you already encounter an error18:32
bgmccollumYeah, I'm seeing 404 is browser console18:33
bgmccollumadmin/hypervisors is trying to load 18c03018c8f6.css18:33
bgmccollumsame with project/volumes - completely unstyled18:34
ApsuInteresting.18:35
bgmccollumdo you want access to this env?18:35
ApsuI'd take it, sure. My next step would to be to find out exactly what creates the files and when/how it gets run18:36
bgmccollumApsu: emailed18:38
cloudnullHughsaunders yes. If we add it to the vars it'll get built.18:39
*** stevemar has joined #openstack-ansible18:41
cloudnullIn master. For the other branches we'll need to go build it.18:44
*** markvoelker_ has quit IRC19:01
*** prometheanfire is now known as blueness-evil-tw19:01
*** blueness-evil-tw is now known as prometheanfire19:02
*** jmccrory has quit IRC19:03
*** jaypipes is now known as jay-afk19:09
*** jay-afk is now known as jaypipes-afk19:10
palendaeandymccr, hughsaunders: If either of you are here - is tempest broken across the board now due to the oslo.context issue? Seeing it even if I pin to master now19:10
hughsaunderspalendae: yep19:11
palendaeSon of a19:11
palendaeOk19:11
hughsaundersneeds oslo.context 0.2.019:12
palendaeEven that older SHA does19:12
hughsaundersyeah :(19:13
hughsaundersthe dep is through tempest-lib19:13
palendaeThat certainly puts a damper on getting the neutron tests sorted19:13
hughsaunderswhich is not capped in the old sha19:13
d34dh0r53can I get looks at: https://review.openstack.org/#/c/162967/19:13
palendaeAnd it doesn't look like we pin oslo.contect anywhere19:13
d34dh0r53and https://review.openstack.org/#/c/163048/19:13
hughsaundersi ran out of time looking at how to get 0.2.0 built19:13
palendaehughsaunders: Is there a bug for this yet?19:18
hughsaundersyeah should be linked to my reviews19:19
*** jmccrory has joined #openstack-ansible19:27
*** vmtrooper has joined #openstack-ansible19:39
*** vmtrooper has quit IRC19:44
Apsubgmccollum: Got an example of an unstyled area for easy testing, btw?19:48
*** daneyon has joined #openstack-ansible19:50
*** daneyon_ has quit IRC19:53
*** britthouser has joined #openstack-ansible20:15
*** ccrouch has joined #openstack-ansible20:27
*** britthouser has quit IRC20:40
*** jwagner is now known as jwagner_away20:42
*** jwagner_away is now known as jwagner20:44
openstackgerritEvan Callicoat proposed stackforge/os-ansible-deployment: Enable offline compression for resource files  https://review.openstack.org/16318321:08
BjoernTapsu: I just have a new 10.1.2 with failed css , who wants to debug it  before I add my config again?21:11
ApsuBjoernT: I don't, the issue is definitely missing OFFLINE_COMPRESS :)21:12
ApsuThe default is False, and I can see how it's broken without it enabled.21:12
ApsuBjoernT: That's what ^ that commit is for. Just amending it to target it at the right bug since we have dupes21:12
openstackgerritEvan Callicoat proposed stackforge/os-ansible-deployment: Enable offline compression for resource files  https://review.openstack.org/16318321:14
BjoernTnice, I try that21:14
BjoernTlol I get the permission issue now:21:15
BjoernTTASK: [horizon_common | Collect and compress static files] ********************21:15
BjoernTfailed: [667164-infra01_horizon_container-8e9851a1] => (item=horizon-manage.py collectstatic --noinput) => {"changed": true, "cmd": ["horizon-manage.py", "collectstatic", "--noinput"], "delta": "0:00:00.338413", "end": "2015-03-10 21:15:06.758614", "item": "horizon-manage.py collectstatic --noinput", "rc": 1, "start": "2015-03-10 21:15:06.420201"}21:15
BjoernTstderr: Traceback (most recent call last):21:15
BjoernTthat has not yet been reported?21:16
ApsuNo idea, I just got voluntold to help figure this one out today :)21:16
BjoernTok, it is reported https://bugs.launchpad.net/openstack-ansible/+bug/142784621:18
openstackLaunchpad bug 1427846 in openstack-ansible "Upgrade 9.0.6 to 10.1.2 - Horizon Static Content Permission Denied" [Medium,Triaged] - Assigned to David Wilde (david-wilde-rackspace)21:18
openstackgerritNolan Brubaker proposed stackforge/os-ansible-deployment: Fix tempest dependency versions  https://review.openstack.org/16319421:25
*** stevemar has quit IRC21:27
*** stevemar has joined #openstack-ansible21:27
*** vmtrooper has joined #openstack-ansible21:28
sigmavirus24BjoernT: I suspect that role isn't running as the right user21:29
sigmavirus24I know mattt had sent a PR updating a bunch of roles to run as specific users but I don't remember if that was pre-stackforge or post21:30
BjoernTI think the role was changed to run as www-data ? Saw a old commit somewhere. The old role ran as root21:30
BjoernTthat's why it bails out21:30
BjoernTwe should have a removal of the old files prior to the compress21:30
*** vmtrooper has quit IRC21:33
matttsigmavirus24: that was in stackforge i believe21:36
matttsadly we never tested upgrades thoroughly :(21:36
*** sdake_ has quit IRC21:37
*** jwagner is now known as jwagner_away21:39
*** sdake_ has joined #openstack-ansible21:40
*** Mudpuppy_ has joined #openstack-ansible21:42
*** Mudpuppy has quit IRC21:45
*** Mudpuppy_ has quit IRC21:47
*** jaypipes-afk is now known as JAYPIPES21:59
*** JAYPIPES is now known as jaypipes21:59
*** sigmavirus24 is now known as sigmavirus24_awa22:00
*** sdake__ has joined #openstack-ansible22:33
*** sdake_ has quit IRC22:37
*** nosleep77 has quit IRC22:50
daneyonI am able to deploy v9 w/o any errors. I am at the step of verifying my deployment and I am unable to find openrc in the utility container. I try to run find but i get permission denied.23:05
daneyonnm.. i found openrc in the keystone container23:10
*** sdake__ has quit IRC23:11
*** vmtrooper has joined #openstack-ansible23:16
*** vmtrooper has quit IRC23:21
*** sdake_ has joined #openstack-ansible23:39
*** sdake_ has quit IRC23:44
*** britthouser has joined #openstack-ansible23:45
*** jmccrory has quit IRC23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!