| *** markvoelker has joined #openstack-ansible | 00:02 | |
| *** sdake has quit IRC | 00:22 | |
| miguelgrinberg | cloudnull: quick question when you have a moment. I updated the git tag for horizon in vars/repo_packages/openstack_services.yml and then launched the repo-server play, expecting it will add a wheel for the new version I requested. But it does not seem to work that way. What do I need to do to add a new version to the repo-server? | 01:01 |
|---|---|---|
| palendae | miguelgrinberg: I think I had to use repo-install | 01:09 |
| miguelgrinberg | palendae: oh, okay, let me try that | 01:09 |
| palendae | Oh, http://git.openstack.org/cgit/stackforge/os-ansible-deployment/tree/playbooks/repo-build.yml | 01:09 |
| palendae | repo-build does the packages | 01:10 |
| miguelgrinberg | yeah, seems repo-build is the one, thanks | 01:11 |
| *** appprod0 has joined #openstack-ansible | 02:11 | |
| *** appprod0 has left #openstack-ansible | 02:11 | |
| cloudnull | Miguelgrinberg did you make it go? | 02:17 |
| *** jsm has quit IRC | 02:26 | |
| *** hj-hp has quit IRC | 02:41 | |
| miguelgrinberg | cloudnull: yes, it worked, but have a bunch of other dependencies that I need to bring. I think I'm going to wait until your patch lands, you've done all the work already. | 02:50 |
| palendae | Should probably have all the people doing individual services reviewing the configs | 02:52 |
| *** jsm has joined #openstack-ansible | 02:58 | |
| cloudnull | Yea the kilo patch is the basic kilo functionality once that's we'll need to go through the individual services and make sure they are prod ready. | 03:10 |
| cloudnull | That'll take a bit | 03:11 |
| cloudnull | Also there will need to be some discovery because several APIs are now deprecated in kilo. Which will effect our upgrade process. | 03:11 |
| cloudnull | Palendae assuming the minimal kilo patch goes through all of the configs are going to need further review and a lot of testing. | 03:13 |
| palendae | Yeah | 03:14 |
| *** galstrom_zzz is now known as galstrom | 03:14 | |
| *** galstrom is now known as galstrom_zzz | 03:33 | |
| *** sdake has joined #openstack-ansible | 03:36 | |
| *** sdake has quit IRC | 03:42 | |
| *** sdake has joined #openstack-ansible | 03:44 | |
| *** sdake has quit IRC | 03:55 | |
| *** ishant has joined #openstack-ansible | 04:48 | |
| *** stevemar has quit IRC | 07:58 | |
| *** markvoelker has quit IRC | 10:26 | |
| openstackgerrit | Andy McCrae proposed stackforge/os-ansible-deployment-specs: Adds swift_standalone for Juno spec https://review.openstack.org/166854 | 11:05 |
| openstackgerrit | Andy McCrae proposed stackforge/os-ansible-deployment: Allow swift stand alone to be installed for Juno https://review.openstack.org/167170 | 11:09 |
| *** markvoelker has joined #openstack-ansible | 11:26 | |
| *** markvoelker has quit IRC | 11:31 | |
| openstackgerrit | Andy McCrae proposed stackforge/os-ansible-deployment: Fix dedicated replication network logic https://review.openstack.org/167177 | 11:33 |
| openstackgerrit | Darren Birkett proposed stackforge/os-ansible-deployment: Add support for passing api version to glance module https://review.openstack.org/167188 | 11:58 |
| *** markvoelker has joined #openstack-ansible | 11:58 | |
| *** KLevenstein has joined #openstack-ansible | 12:05 | |
| *** jaypipes has quit IRC | 12:06 | |
| *** jaypipes has joined #openstack-ansible | 12:07 | |
| *** galstrom_zzz is now known as galstrom | 12:09 | |
| openstackgerrit | Darren Birkett proposed stackforge/os-ansible-deployment: Add support for passing api version to glance module https://review.openstack.org/167188 | 12:14 |
| openstackgerrit | Darren Birkett proposed stackforge/os-ansible-deployment: Add support for passing api version to glance module https://review.openstack.org/167188 | 12:16 |
| *** ishant has quit IRC | 12:36 | |
| openstackgerrit | Andy McCrae proposed stackforge/os-ansible-deployment: Fix dedicated replication network logic https://review.openstack.org/167198 | 12:45 |
| *** galstrom is now known as galstrom_zzz | 12:56 | |
| *** gregdek has joined #openstack-ansible | 13:01 | |
| *** dstanek has joined #openstack-ansible | 13:33 | |
| *** Mudpuppy has joined #openstack-ansible | 14:07 | |
| *** sdake has joined #openstack-ansible | 14:10 | |
| cloudnull | mattt: ping | 14:10 |
| cloudnull | it seems that im replying to your inline comments as your writing them | 14:10 |
| cloudnull | :) | 14:10 |
| mattt | cloudnull: oops :P | 14:10 |
| mattt | cloudnull: sorry for the barrage of comments | 14:10 |
| cloudnull | no, on the contrary, bring them on | 14:11 |
| mattt | cloudnull: i guess the main concern mancdaz and i had was the glance v1 deprecation | 14:11 |
| mattt | is that absolutely necessary in this commit? | 14:11 |
| cloudnull | it is. because of `fatal_deprecations`. | 14:11 |
| cloudnull | glance / cinder v1 have been deprecated and the services will not start because of it | 14:12 |
| mancdaz | cloudnull deprecated !+ unsupported | 14:12 |
| mattt | cloudnull: hmm, when i updated my checkout to re-enable v1 glance restarted fine | 14:12 |
| mancdaz | != | 14:12 |
| openstack | mancdaz: Error: "=" is not a valid command. | 14:12 |
| mattt | cloudnull: the same applied to cinder, everything continued to work afterwards | 14:12 |
| mancdaz | ok deprecated doesn't mean you HAVE to remove now | 14:12 |
| mancdaz | we have committed in 11.0.0 to as little end user impact as possible | 14:13 |
| cloudnull | mancdaz it does when we have fatal_deprecations on the gate job | 14:13 |
| mancdaz | the only reason we're being forced down this route is because we arbitrarily turned on fatel deps | 14:13 |
| cloudnull | right. | 14:13 |
| mancdaz | so let's turn that off and reduce the scope of this change | 14:13 |
| git-harry | Yeah, I'm not convinced by using fatal deprecations | 14:13 |
| mattt | i have fatal deprecations on and didn't run into those issues | 14:13 |
| *** KLevenstein has quit IRC | 14:14 | |
| mattt | (i built using the AIO script) | 14:14 |
| git-harry | mattt: you won't unless you run the appropriate bit of code | 14:14 |
| cloudnull | ^ | 14:14 |
| mattt | ok but i hit v1 api afterwards | 14:15 |
| mattt | and it returned a response | 14:15 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 14:15 | |
| mancdaz | no matter - we can't just drop glance api v1 support in this release | 14:15 |
| mancdaz | same for cinder | 14:15 |
| cloudnull | well we can in the project, its configurable for that reason. | 14:15 |
| cloudnull | but if we remove the toggle for the gate job then we can ship with cinder/glance v1. | 14:16 |
| cloudnull | you'll also notice that ec2 and s3 api's have been removed, due to deprecations. | 14:17 |
| mattt | i haven't gotten that far yet :P | 14:17 |
| cloudnull | but its not like those worked anyway | 14:17 |
| mattt | (got sidelined trying to get those scenario tests to pass) | 14:17 |
| cloudnull | mattt the scenario tests will not pass at this point. | 14:18 |
| mattt | cloudnull: why is that? | 14:18 |
| mattt | cloudnull: i'm running into networking issues on my AIO, so i couldn't properly test them | 14:18 |
| mattt | cloudnull: i also commented on bumping tempest sha, i'm wondering if we're a bit premature on that one | 14:19 |
| cloudnull | mainly because of cinder interacting with quota options that cause it to fault. and a couple of the neutron tests do not pass because the expected output is incorrect. | 14:19 |
| cloudnull | mattt i replied on that comment. | 14:19 |
| mattt | cloudnull: wonder if the quota issues go away if we go back to v1? | 14:20 |
| cloudnull | the old sha is no longer compatible because it has dependencies that are below what kilo requires. | 14:20 |
| mattt | cloudnull: ok that makes sense | 14:20 |
| cloudnull | as for cinder and the quota issues. its only because of fatal deprecations. | 14:20 |
| *** git-harry has quit IRC | 14:21 | |
| mattt | cloudnull: yep | 14:21 |
| cloudnull | and cinder seems to have a bug at this stage. | 14:21 |
| mancdaz | this is also outside of the scope of the initial spec | 14:21 |
| *** git-harry has joined #openstack-ansible | 14:21 | |
| mancdaz | "No changes to the current testing and or gating framework will be made. The minimum viable Kilo deployment will be required to pass the same gate tests as are required by our production systems." | 14:21 |
| cloudnull | mancdaz its certainly in scope, if we adhere to what we're presently set to gate on. | 14:21 |
| cloudnull | mancdaz its passing the same multi-node commit tests instead of the scenario tests. | 14:22 |
| mancdaz | cloudnull these are not the tests we are looking for | 14:22 |
| * mancdaz waves hand | 14:22 | |
| cloudnull | haha | 14:22 |
| mattt | oh no you di'int | 14:22 |
| cloudnull | but it wont pass. | 14:22 |
| cloudnull | and we cant use the older tempest sha that we know works | 14:23 |
| cloudnull | so we iterate or we stagnate. | 14:23 |
| cloudnull | im for removing the fatal deprecations. | 14:23 |
| mancdaz | yes | 14:24 |
| cloudnull | and setting v1 cinder glance to enabled by defaultl. | 14:24 |
| *** hughsaunders has quit IRC | 14:24 | |
| mancdaz | we are being forced down an unwanted path by this | 14:24 |
| mancdaz | yes! | 14:24 |
| cloudnull | but if we reimplement it, this is whats coming :) | 14:24 |
| mancdaz | sure thing - but that's in a more controlled manner with proper informed decisions around what's coming | 14:24 |
| mattt | cloudnull: regarding glance v2 | 14:25 |
| cloudnull | that said, we still wont be able to pass the scenario test. | 14:25 |
| *** hughsaunders has joined #openstack-ansible | 14:25 | |
| mattt | 'nova image-list' didn't even work for me | 14:25 |
| mattt | how do you pass it a version? | 14:25 |
| *** claco has joined #openstack-ansible | 14:25 | |
| mancdaz | cloudnull I'd love to be able to pass the scenario tests, and tbh they will need to pass for the release, so assuming we go back to v1 glance/cinder, is it not possible to get those tests passing then? | 14:26 |
| mancdaz | as in, what would still be broken that would not allow those tests to pass? | 14:26 |
| cloudnull | neutron i would suspect | 14:27 |
| cloudnull | the tests themselves pass, be a few of them are returning a 200 when tempest expects a 202 | 14:27 |
| mancdaz | cloudnull upstream must be passing with the current b3 code though? | 14:28 |
| cloudnull | i would suspect they aren't running the same tests. | 14:28 |
| mancdaz | so mostly we think some neutron tests might not pass - perhaps it's worth spending a bit of time (after going back to v1 apis) trying to get this stuff going. At least we can get a clear answer about what upstream is doing and why we might absolutely need to change the tests that we have in the gate for the short term | 14:30 |
| mancdaz | changing tests because the ones we want to pass are failing, seems such wrong | 14:30 |
| *** git-harry has quit IRC | 14:31 | |
| cloudnull | it does seem wrong, but we're also testing on a mid release version of tempest that is likely broken in some fun ways. | 14:31 |
| andymccr | so wrong that it has angered git-harry to a state where he had to leave! | 14:31 |
| cloudnull | and i think we should get kilo to be operational and then we can re-introduce the tests that the system should be passing. | 14:32 |
| cloudnull | while everyone is in a reviewing mood. please go review https://review.openstack.org/#/c/166095/ and https://review.openstack.org/#/c/166574/ | 14:37 |
| *** git-harry has joined #openstack-ansible | 14:38 | |
| cloudnull | with https://review.openstack.org/#/c/166095/ of particular interest because it seems that with that change neutron gating is passing more often than not. | 14:38 |
| mattt | i'm reviewed out | 14:38 |
| palendae | -1s all around | 14:40 |
| *** KLevenstein has joined #openstack-ansible | 14:40 | |
| mattt | cloudnull: i'm still on the fence w/ that rpc/api_workers change, i've been running those settings in the heat multi node and still get sporadic neutron failures | 14:40 |
| Apsu | mattt: It definitely still fails with just rpc_workers = 0. | 14:40 |
| cloudnull | sporadic seems to be better than roughly 50% of the time. | 14:40 |
| *** sdake__ has joined #openstack-ansible | 14:41 | |
| palendae | Sounds like it's true that this doesn't fix it completely, but it does succeed more than it did | 14:41 |
| Apsu | I just +1'd it because even though it's not the final answer, it's still the correct change to make. | 14:41 |
| Apsu | Working on the bigger swap change now, because that's another failure reduction. | 14:42 |
| mattt | haha | 14:42 |
| mattt | ok i can get on board w/ that | 14:42 |
| palendae | I think it's got more impact on AIOs vs the multinode tests from what you're saying mattt | 14:43 |
| palendae | Is that accurate? | 14:43 |
| openstackgerrit | Kevin Carter proposed stackforge/os-ansible-deployment: Removes the fatal_deprecations from the gate job https://review.openstack.org/167257 | 14:43 |
| *** sdake has quit IRC | 14:44 | |
| openstackgerrit | Kevin Carter proposed stackforge/os-ansible-deployment: Removes the fatal_deprecations from the gate job https://review.openstack.org/167257 | 14:46 |
| mattt | palendae: that sounds right | 14:47 |
| *** git-harry has quit IRC | 14:47 | |
| *** git-harry has joined #openstack-ansible | 14:47 | |
| palendae | cloudnull: Did you just rebase that? | 14:47 |
| palendae | Has the workers change in bootstrap-aio.sh | 14:47 |
| cloudnull | i did for the deps | 14:48 |
| sigmavirus24 | Anyone going to look at https://review.openstack.org/#/c/166574/ ? | 14:48 |
| *** stevemar has joined #openstack-ansible | 14:48 | |
| sigmavirus24 | o/ stevemar | 14:49 |
| *** git-harry has quit IRC | 14:50 | |
| *** git-harry has joined #openstack-ansible | 14:50 | |
| stevemar | sigmavirus24, o/ | 14:50 |
| Apsu | palendae: mattt: cloudnull: Digging into the actual root cause, with debug = True. Time to drink from the firehose | 14:51 |
| mattt | cloudnull: didn't want to just revert the actual commit for the fatal deprecations stuff? | 14:52 |
| palendae | Drink deep | 14:52 |
| mattt | cloudnull: i think we leave some cruft behind by not reverting | 14:53 |
| cloudnull | the vars that were added to the roles are valuable. | 14:53 |
| cloudnull | we can turn it on in testing locally using ansible . | 14:53 |
| mattt | cloudnull: ok | 14:53 |
| cloudnull | but removing it from the gate is what we're after. | 14:53 |
| mattt | cloudnull: will we ever turn them back on in gate? if so maybe we jsut want to comment where it writes to user vars | 14:55 |
| cloudnull | we could do that. | 14:55 |
| mattt | cloudnull: because we leave a bunch of exports on those vars in bootstrap-aio.sh and gate-check-commit.sh which go nowhere right now | 14:56 |
| mattt | cloudnull: https://github.com/stackforge/os-ansible-deployment/blob/master/scripts/gate-check-commit.sh#L44-L51 and https://github.com/stackforge/os-ansible-deployment/blob/master/scripts/bootstrap-aio.sh#L32-L39 | 14:56 |
| openstackgerrit | Kevin Carter proposed stackforge/os-ansible-deployment: Removes the fatal_deprecations from the gate job https://review.openstack.org/167257 | 14:59 |
| cloudnull | mattt that removes the exports to enable | 14:59 |
| cloudnull | but leaves the scaffolding in place. | 15:00 |
| *** git-harry has quit IRC | 15:00 | |
| cloudnull | git-harry has rage quit once again. | 15:00 |
| Sam-I-Am | lol | 15:00 |
| *** git-harry has joined #openstack-ansible | 15:01 | |
| cloudnull | and he returns! | 15:01 |
| mattt | cloudnull: +2d | 15:02 |
| cloudnull | ty sir | 15:02 |
| Sam-I-Am | meetingses? | 15:04 |
| palendae | Sam-I-Am: Still have an hour | 15:05 |
| *** jaypipes has quit IRC | 15:05 | |
| openstackgerrit | Kevin Carter proposed stackforge/os-ansible-deployment: Set the neutron default workers https://review.openstack.org/166095 | 15:07 |
| Sam-I-Am | oh, utc :/ | 15:07 |
| openstackgerrit | Kevin Carter proposed stackforge/os-ansible-deployment: Adds new provider_networks module and reformats ml2_conf.ini https://review.openstack.org/166574 | 15:07 |
| *** hj-hp has joined #openstack-ansible | 15:07 | |
| openstackgerrit | Kevin Carter proposed stackforge/os-ansible-deployment: Removes the fatal_deprecations from the gate job https://review.openstack.org/167257 | 15:07 |
| palendae | I just correctedmy entry | 15:07 |
| cloudnull | "UTC – Coordinated Universal Time" not this DST/CST/SOME SHIT time we have here. | 15:08 |
| palendae | CDT* | 15:08 |
| palendae | It's noon to 1 for me | 15:08 |
| palendae | THANKS A LOT CLOUDNULL | 15:08 |
| cloudnull | yw? | 15:09 |
| openstackgerrit | Kevin Carter proposed stackforge/os-ansible-deployment: Updated repository for minimum viable kilo install https://review.openstack.org/166986 | 15:14 |
| openstackgerrit | Kevin Carter proposed stackforge/os-ansible-deployment: Updated repository for minimum viable kilo install https://review.openstack.org/166986 | 15:16 |
| cloudnull | mattt mancdaz palendae ^ kilo commit with the deps for the removal of the fatal_deprecations. | 15:17 |
| mancdaz | cloudnull thankyou | 15:18 |
| cloudnull | no thank you for reviewing all the things | 15:18 |
| mancdaz | no thank you for making things to review | 15:18 |
| cloudnull | so this commit is a go: "https://review.openstack.org/#/c/167257/4" which depends on "https://review.openstack.org/#/c/166574" which itself depends on "https://review.openstack.org/#/c/166095/4" | 15:19 |
| palendae | wooo wooo all aboard the dependency train | 15:19 |
| cloudnull | so if we're all happy with "https://review.openstack.org/#/c/167257/4" can we get core reviewers to do the needfuls on the other two. | 15:19 |
| mattt | cloudnull: can you revert that stuff in tempest resources? | 15:20 |
| mancdaz | cloudnull my comments were not addressed in the last of those | 15:20 |
| mancdaz | it's a question really | 15:20 |
| mancdaz | on https://review.openstack.org/#/c/166095/3 | 15:21 |
| mattt | cloudnull: also tempest.conf.j2 wasn't reverted to continue to work w/ the older apis | 15:21 |
| mancdaz | patchset 3 | 15:21 |
| mancdaz | oh wait | 15:21 |
| cloudnull | mancdaz: i replied inline https://review.openstack.org/#/c/166095/3/playbooks/roles/os_neutron/templates/neutron.conf.j2 | 15:21 |
| mancdaz | you did | 15:21 |
| cloudnull | and the commit message was change | 15:21 |
| cloudnull | *changed. | 15:21 |
| cloudnull | mattt let me address that in the kilo patch. | 15:22 |
| mattt | cloudnull: k thanks | 15:23 |
| *** sdake has joined #openstack-ansible | 15:25 | |
| openstackgerrit | Kevin Carter proposed stackforge/os-ansible-deployment: Updated repository for minimum viable kilo install https://review.openstack.org/166986 | 15:26 |
| cloudnull | mattt ^ that should do it. but let me know if im still being an idiot | 15:26 |
| cloudnull | :) | 15:26 |
| mattt | cloudnull: there are no idiots here sir | 15:27 |
| *** sdake__ has quit IRC | 15:29 | |
| mattt | cloudnull: https://review.openstack.org/#/c/166986/6/playbooks/roles/os_tempest/tasks/tempest_resources.yml | 15:33 |
| mattt | cloudnull: all that can be reverted right? | 15:33 |
| *** jwagner_away is now known as jwagner | 15:34 | |
| *** galstrom_zzz is now known as galstrom | 15:39 | |
| *** nosleep77 has joined #openstack-ansible | 15:43 | |
| *** hj-hp has quit IRC | 15:51 | |
| cloudnull | mattt we can revert that. though i think its still worth while to fix our glance module to support v2 | 15:54 |
| mattt | cloudnull: yeah i think mancdaz was working on an independent review to do that | 15:56 |
| *** rromans has quit IRC | 15:57 | |
| *** sdake has quit IRC | 15:58 | |
| *** Bjoern__ has joined #openstack-ansible | 16:01 | |
| *** rrrobbb has joined #openstack-ansible | 16:01 | |
| cloudnull | bug triage time. | 16:01 |
| cloudnull | cloudnull, mattt, andymccr, d34dh0r53, hughsaunders, b3rnard0, palendae, Sam-I-Am, odyssey4me, serverascode, rromans, mancdaz, dolphm, _shaps_, BjoernT, claco, echiu, dstanek - triage time | 16:02 |
| palendae | Hi | 16:02 |
| hughsaunders | hey | 16:02 |
| b3rnard0 | Notes are here: https://etherpad.openstack.org/p/openstack_ansible_bug_triage.2015-03-24-16.00 | 16:02 |
| mattt | \o | 16:02 |
| sigmavirus24 | o/ | 16:02 |
| Sam-I-Am | hi | 16:02 |
| Sam-I-Am | b3rnard0: time to fire it up, captain launchpad | 16:03 |
| b3rnard0 | np Admiral Diagram | 16:04 |
| Sam-I-Am | heh | 16:04 |
| Sam-I-Am | so i guess we have no bugs? | 16:05 |
| cloudnull | so lets get started. | 16:05 |
| cloudnull | first up https://bugs.launchpad.net/openstack-ansible/+bug/1435597 | 16:05 |
| openstack | Launchpad bug 1435597 in openstack-ansible "scripts/bootstrap-ansible.sh creates openstack-ansible without support for unprivileged users" [Undecided,New] | 16:05 |
| cloudnull | i think dolphm patched that already. | 16:06 |
| dstanek | hi | 16:06 |
| cloudnull | hi dstanek | 16:06 |
| cloudnull | yup. - fixed. https://review.openstack.org/#/c/167010/ <- needs gating love | 16:06 |
| andymccr | dang gate! | 16:07 |
| cloudnull | so next: https://bugs.launchpad.net/openstack-ansible/+bug/1435472 | 16:08 |
| openstack | Launchpad bug 1435472 in openstack-ansible "Create VIP for rsyslog logging destination" [Undecided,New] | 16:08 |
| cloudnull | Bjoern__ you around ? | 16:08 |
| *** Bjoern__ is now known as BjoernT | 16:08 | |
| BjoernT | yes | 16:08 |
| BjoernT | thats a wishlist | 16:09 |
| BjoernT | any question to the context? | 16:09 |
| cloudnull | so that would only effect Juno. | 16:09 |
| Apsu | None from me. Seems sensible. | 16:09 |
| BjoernT | alright next bug i guess | 16:10 |
| cloudnull | in master rsyslog at the service level is shipping to an rsyslog server for which will give you access to the raw logs on a single host. | 16:10 |
| cloudnull | or multiple if you scale it out that way | 16:10 |
| *** IRTermite has joined #openstack-ansible | 16:11 | |
| BjoernT | right, once you have VIP or support multiple destionations | 16:11 |
| cloudnull | yea in master we have multi destinations. | 16:11 |
| cloudnull | next: https://bugs.launchpad.net/openstack-ansible/+bug/1435350 | 16:12 |
| openstack | Launchpad bug 1435350 in openstack-ansible "We need a swift_standalone play for Juno" [Undecided,New] - Assigned to Andy McCrae (andrew-mccrae) | 16:12 |
| * cloudnull hands mic to andymccr | 16:12 | |
| andymccr | pr is in | 16:12 |
| andymccr | but basically, for swift standalone to work nicely in juno we need a new play - its really not impactful in that you dont have to run that play if you dont want | 16:12 |
| andymccr | mainly because keystone sets up ALL services at the start which will break if you dont specify the other services and their vars. | 16:13 |
| cloudnull | ok. so i tagged the bug for juno / invalid in master. | 16:13 |
| andymccr | yeh not valid for master | 16:14 |
| andymccr | https://review.openstack.org/#/c/167170/ | 16:14 |
| cloudnull | next: https://bugs.launchpad.net/openstack-ansible/+bug/1434597 | 16:14 |
| openstack | Launchpad bug 1434597 in openstack-ansible "nova-compute logs are not in the logging node" [Undecided,Confirmed] | 16:14 |
| cloudnull | miguelgrinberg: needs more datas. | 16:14 |
| cloudnull | next: https://bugs.launchpad.net/openstack-ansible/+bug/1434315 | 16:15 |
| openstack | Launchpad bug 1434315 in openstack-ansible "SYMLINK_DIR used in scripts/bootstrap_aio.sh but not defined" [Undecided,New] | 16:15 |
| cloudnull | palendae: ^ | 16:15 |
| palendae | Yeah - I wastrying the boostrap_aio.sh script without the gate check commit, and it required SYMLINK_DIR, which wasn't defined inside bootstrap_aio | 16:16 |
| palendae | I'm not sure exactly where we should fix that, but it seems fairly simple | 16:16 |
| cloudnull | imo we should fix that in the `boostrap_aio.sh ` script if thats where its being required. | 16:17 |
| Sam-I-Am | i found this issue too | 16:17 |
| palendae | Ok | 16:17 |
| palendae | I think it's as simple as moving the definition over | 16:17 |
| Sam-I-Am | typey typey | 16:18 |
| palendae | I can take it | 16:18 |
| cloudnull | assigned and triaged. | 16:18 |
| palendae | kk | 16:19 |
| cloudnull | next: https://bugs.launchpad.net/openstack-ansible/+bug/1434233 | 16:19 |
| openstack | Launchpad bug 1434233 in openstack-ansible "Non-existent variable in os_glance role" [Undecided,New] - Assigned to Matt Thompson (mattt416) | 16:19 |
| mattt | cloudnull: so we uncovered this one while playing w/ stacktach | 16:20 |
| cloudnull | mattt looks like we should just remove that var | 16:20 |
| mattt | cloudnull: yeah, some of the variable names have changed, it may get cleaned up as we move to kilo and scrutinise all the configs | 16:20 |
| cloudnull | its not being used. | 16:20 |
| cloudnull | sounds good. | 16:21 |
| cloudnull | ill target it to kilo. | 16:21 |
| mattt | cloudnull: but i also think we have a bigger issue on hand in that we cannot consistently turn on notifications across projects in o-a-d | 16:21 |
| mattt | cloudnull: but i will raise a spec/blueprint for that one later | 16:21 |
| cloudnull | yea, i was talking to alextricity about that very thing just today. | 16:21 |
| cloudnull | mattt can you and alextricity sync up on that ? | 16:22 |
| mattt | cloudnull: certainly | 16:22 |
| cloudnull | next: https://bugs.launchpad.net/openstack-ansible/+bug/1433619 | 16:22 |
| openstack | Launchpad bug 1433619 in openstack-ansible "Please layout the swift-drive-audit configuration" [Undecided,New] | 16:22 |
| *** sdake has joined #openstack-ansible | 16:22 | |
| cloudnull | BjoernT andymccr ^ | 16:23 |
| BjoernT | yup another wishlist | 16:23 |
| cloudnull | andymccr is that something that we can target at Kilo? | 16:23 |
| BjoernT | I like actually earlier, a juno release | 16:23 |
| andymccr | i mean we can, i dont think it provides much - its just a trawl through kern.log for disk errors | 16:23 |
| andymccr | we should disk errors far more reliably via the dell monitoring stuff | 16:24 |
| *** sdake__ has joined #openstack-ansible | 16:24 | |
| BjoernT | andymccr: Yes dell is installed here but we have cdc installs too | 16:25 |
| BjoernT | and the os usuallr reports io errors earlier than dell tool | 16:25 |
| andymccr | BjoernT: its fine to add it although regex on loop devices seems weird, and drive-audit is not (imo) a great way to detect failures. | 16:26 |
| BjoernT | in many cases it will be the only way, especially if you do disk passthrough | 16:27 |
| *** sdake has quit IRC | 16:27 | |
| andymccr | and if the regex is wrong nobody will know or care and it just wont say anything | 16:28 |
| andymccr | but sure adding it is easy | 16:28 |
| BjoernT | alright we take this offline, I don't see your concerns | 16:28 |
| cloudnull | ok. so next. https://bugs.launchpad.net/openstack-ansible/+bug/1433291 | 16:30 |
| openstack | Launchpad bug 1433291 in openstack-ansible "rpc_deploy/conf.d/swift.yml file should be inert by default" [Undecided,New] | 16:30 |
| cloudnull | revdr around ? | 16:30 |
| BjoernT | he is but not here | 16:30 |
| andymccr | i thought that used to be the case (swift.yml commented out) and it should be | 16:30 |
| andymccr | so makes sense. | 16:30 |
| andymccr | or make the default in the etc dir in the repo like swift.yml.sample or something so it isn't automatically read in. | 16:31 |
| BjoernT | swift.yml is not commented out | 16:31 |
| BjoernT | if you do a cp from /opt/os-ansible-deployment/etc/rpc_deploy/conf.d you'll copy that aio configuration accidentially | 16:32 |
| andymccr | BjoernT: "used to be the case" e.g. not the case now | 16:32 |
| BjoernT | I think that's his concern | 16:32 |
| BjoernT | the issues is, you get inventory errors running normal playbooks | 16:33 |
| andymccr | hmm it was added by odyssey4me for the aio gate i believe. so i think that would have an impact there | 16:34 |
| Sam-I-Am | i thought all the stock configs do aio now | 16:34 |
| cloudnull | so we should look at tagging the configs used by the AIO gate jobs such that they are not impacting when someone blindly copies the configs into place. | 16:34 |
| Sam-I-Am | and i thought that only applied to v11 | 16:34 |
| andymccr | cloudnull: yeh | 16:34 |
| Sam-I-Am | but maybe it does v10 too? | 16:35 |
| andymccr | cloudnull: perhaps the aio could copy it from like .sample or .aio | 16:35 |
| cloudnull | yea that sounds reasonable. | 16:35 |
| andymccr | that way blind copy wont error and it will still work for gate | 16:35 |
| andymccr | and as a sample | 16:35 |
| Sam-I-Am | ideally there should be no config files you can just blindly copy | 16:35 |
| Sam-I-Am | they should all be .sample or .aio or something | 16:35 |
| Sam-I-Am | (unless they dont require editing, of course) | 16:35 |
| Sam-I-Am | i clear out crowds... | 16:39 |
| cloudnull | ok so lets update that issue and work through getting the example configs tagged with aio / sample or something. | 16:39 |
| Sam-I-Am | makes sense | 16:40 |
| *** jaypipes_ has joined #openstack-ansible | 16:41 | |
| cloudnull | next: https://bugs.launchpad.net/openstack-ansible/+bug/1433240 | 16:41 |
| openstack | Launchpad bug 1433240 in openstack-ansible "Move cinder-volumes onto bare metal host" [Undecided,New] | 16:42 |
| openstackgerrit | Nolan Brubaker proposed stackforge/os-ansible-deployment: Move SYMLINK_DIR declaration to bootstrap-aio.sh https://review.openstack.org/167316 | 16:42 |
| openstackgerrit | Nolan Brubaker proposed stackforge/os-ansible-deployment: Move SYMLINK_DIR declaration to bootstrap-aio.sh https://review.openstack.org/167316 | 16:42 |
| cloudnull | im kind of in agreement that cinder in a container-volume in a container is kind of like swift storage nodes / nova compute in a container. | 16:42 |
| cloudnull | its cool, it has its uses, but the service just lends itself to being on the physical machine. | 16:43 |
| claco | my only concern is upgrades | 16:43 |
| Sam-I-Am | any hope for containerizing nova again? | 16:43 |
| cloudnull | not until a kernel bug is fixe.d | 16:43 |
| claco | that seems like a 10 -> 11, or 11-> 12 thing, but not great for minor/revision releases | 16:43 |
| cloudnull | agreeded. | 16:44 |
| Sam-I-Am | cloudnull: i thought Apsu was working on a workaround with veth magic or something. | 16:44 |
| cloudnull | if we decided to make that change it would not effect prior branches. | 16:44 |
| BjoernT | Why would be containerize single use boxes ? | 16:44 |
| claco | oh, yeah...because of the inventory setting | 16:44 |
| Apsu | Sam-I-Am: That won't work, sadly | 16:44 |
| *** sdake has joined #openstack-ansible | 16:44 | |
| Sam-I-Am | BjoernT: does cinder volume always run on a separate host? | 16:44 |
| Apsu | The iSCSI kernel module is where the problem seems to lie | 16:45 |
| Sam-I-Am | Apsu: patchy patchy make the goes | 16:45 |
| BjoernT | normally yes, but not in NFS case | 16:45 |
| Apsu | Needs to be patched to support generic netlink | 16:45 |
| Apsu | s/patched/rewritten/ | 16:45 |
| Sam-I-Am | rewritety rewritety | 16:45 |
| BjoernT | we still see issues at tgt inside containers and don't know where they are coming from | 16:45 |
| cloudnull | so can we spike on this? | 16:46 |
| cloudnull | and does anyone want to take owner ship of it ? | 16:46 |
| Sam-I-Am | cloudnull: the tgt issue, or un-containerizing cinder? | 16:48 |
| Sam-I-Am | the latter seems like is_metal = True, goes. | 16:48 |
| *** sdake__ has quit IRC | 16:49 | |
| cloudnull | Sam-I-Am: yes. | 16:49 |
| cloudnull | that is true. but what does an upgrade look like? | 16:49 |
| cloudnull | and changing from in container to out of container will cause volume downtime. | 16:50 |
| andymccr | it would be a good idea to have documented issues we're solving for on this though.otherwise i have no way of telling if its better or the same | 16:50 |
| cloudnull | so spiking on that would be really helpful to understanding all of the impacts. | 16:50 |
| cloudnull | also what andymccr said. | 16:50 |
| cloudnull | so who wants it? is it important to us? | 16:51 |
| andymccr | in principle i agree with the volumes on physical host but we're already in containers so there should be some benefit to moving to physical host, at which point we need to be able to tell/measure whether its improved. | 16:51 |
| Apsu | I'd like to know what the symptoms are with tgt | 16:51 |
| Apsu | So we can adequately analyze them and figure out what we need to do from there | 16:51 |
| andymccr | Apsu: +1 | 16:51 |
| cloudnull | Apsu you volunteering ? | 16:52 |
| andymccr | voluntolding | 16:52 |
| Apsu | cloudnull: Why the hell not, you know I love me the cinder | 16:52 |
| *** sdake__ has joined #openstack-ansible | 16:52 | |
| Sam-I-Am | Apsu: is it better than neutron? | 16:52 |
| Apsu | He says as he reads the source code of brctl... | 16:52 |
| Sam-I-Am | Apsu: masochist | 16:52 |
| palendae | Good thing brctl has unit tests >.> | 16:52 |
| palendae | <.< | 16:52 |
| Apsu | ... yeah... | 16:53 |
| Sam-I-Am | Apsu: this wouldnt be a problem with OVS | 16:53 |
| Apsu | Good thing it's been updated in the past 9 years. Oh wait | 16:53 |
| claco | focus | 16:53 |
| Sam-I-Am | isnt there a newer command to replace brctl? | 16:53 |
| Sam-I-Am | bridge2 | 16:53 |
| Apsu | cloudnull: More to triage? | 16:53 |
| claco | I have a log for the fire: https://bugs.launchpad.net/openstack-ansible/+bug/1399382 | 16:54 |
| openstack | Launchpad bug 1399382 in openstack-ansible "F5 Pool Monitoring in Kilo" [High,New] - Assigned to Evan Callicoat (apsu-2) | 16:54 |
| claco | like should be moved to extras | 16:54 |
| cloudnull | yup. | 16:54 |
| Apsu | +1 | 16:55 |
| claco | * your card | 16:55 |
| claco | ^ do the thing | 16:55 |
| cloudnull | we have one more issue: https://bugs.launchpad.net/openstack-ansible/+bug/1432269 | 16:55 |
| openstack | Launchpad bug 1432269 in openstack-ansible "Adding a new Cinder storage node fails api service check" [Undecided,New] | 16:55 |
| *** sdake has quit IRC | 16:56 | |
| Apsu | I don't know how to move bugs. Unless you mean close this bug and reopen on other project. If there is an LP project for extras. | 16:56 |
| cloudnull | daneyon you around ? | 16:56 |
| cloudnull | were we ever able to nail this down | 16:56 |
| cloudnull | hughsaunders you we're working on that too right? | 16:56 |
| claco | Apsu: move to github extras repo issues, link them, close the lp | 16:56 |
| Apsu | claco: roger | 16:56 |
| hughsaunders | cloudnull: I did look at it, I couldn't reproduce | 16:57 |
| hughsaunders | daneyon mentioned that he was going to rebuild his lab and let us know if he comes across the same error again | 16:57 |
| daneyon | cloudnull here | 16:57 |
| cloudnull | hey there :) | 16:58 |
| daneyon | hughsaunders I have not been able to rebuild mt lab. | 16:58 |
| daneyon | hughsaunders After writing-up osad, I've been busy cutting the M3 release for Kolla | 16:58 |
| hughsaunders | daneyon: no problem, just means the bug is probably incomplete as we haven't been able to reproduce | 16:59 |
| cloudnull | so i've marked the issue as incomplete. hughsaunders can you update the issue with some bits noting what you tested and that you cant reproduce? | 16:59 |
| hughsaunders | cloudnull: yep | 16:59 |
| cloudnull | no worries daneyon. thanks for all the testing that you did though. much appreciated. | 16:59 |
| cloudnull | and thanks for the mention on twitter :) | 17:00 |
| b3rnard0 | wow, did we just finish on time | 17:00 |
| cloudnull | so we're done here. | 17:00 |
| Sam-I-Am | b3rnard0: you did an amazing job with time, sir | 17:00 |
| cloudnull | are there any other items that people want to lobby for ? | 17:00 |
| b3rnard0 | https://etherpad.openstack.org/p/openstack_ansible_bug_triage.2015-03-24-16.00 | 17:00 |
| BjoernT | no | 17:00 |
| cloudnull | great. thanks everyone ! | 17:01 |
| andymccr | great job! | 17:02 |
| palendae | https://review.openstack.org/#/c/167316/ reviews please | 17:02 |
| Apsu | Sam-I-Am: Strongly thinking about trying to remove all use of brctl in neutron and replace with ip/bridge. Apparently the brctl backend (libbridge) uses ioctls that were deprecated pre kernel 2.5 | 17:03 |
| alextricity | Question: In the openstack_environment.yml file, what defines the os-infra_containers group? | 17:03 |
| Apsu | And it uses the result of that as part of figuring out its exit code as successful or not... | 17:03 |
| Sam-I-Am | Apsu: have you talked to matt about that? | 17:04 |
| Apsu | Sam-I-Am: mattt or ... | 17:04 |
| Sam-I-Am | yay for screen scraping | 17:04 |
| Sam-I-Am | Apsu: uhh... matt that wanders over to our area sometimes? | 17:04 |
| Sam-I-Am | long hair? | 17:04 |
| Apsu | oh, thode | 17:04 |
| Sam-I-Am | yeah, thats it | 17:04 |
| claco | yeah. neutron contrib | 17:04 |
| cloudnull | also review https://review.openstack.org/#/c/166095 <- please, now. :) | 17:06 |
| Sam-I-Am | claco: yeah, until he gets fed up (which i imagine is soon) | 17:06 |
| Sam-I-Am | cloudnull: reviewerated | 17:07 |
| *** jwagner is now known as jwagner_away | 17:07 | |
| cloudnull | also mattt mancdaz it looks like glance v1 is not passing even the api testing "http://logs.openstack.org/86/166986/6/check/os-ansible-deployment-dsvm-check-commit/d2cc18d/console.html#_2015-03-24_16_20_51_944" | 17:08 |
| *** jmccrory has joined #openstack-ansible | 17:10 | |
| mancdaz | cloudnull mattt gone, I'm just about to head out. We were both going to re-test and review that patch in the morning if it hadn't already been done by everyone else today | 17:14 |
| cloudnull | ok. | 17:14 |
| cloudnull | im kicking it too. se if i can determine why it hates glance. | 17:14 |
| openstackgerrit | Justin Shepherd proposed stackforge/os-ansible-deployment: Fixing defcore test list generation https://review.openstack.org/167329 | 17:17 |
| galstrom | to all: I am sorry ^ | 17:17 |
| claco | ha | 17:17 |
| andymccr | -2 | 17:17 |
| andymccr | youre welcome | 17:17 |
| galstrom | but the defcore list will now actually run the right set of tests | 17:17 |
| andymccr | what part of -2 was unclear? | 17:17 |
| galstrom | andymccr: did you mean +2 for amazing bash-ness | 17:18 |
| galstrom | cause that shit is awesome | 17:18 |
| mancdaz | galstrom defcore don't care | 17:19 |
| mancdaz | scenario is where the cool kids hang out | 17:19 |
| galstrom | :) | 17:19 |
| galstrom | this can make us OG defcore | 17:20 |
| Apsu | galstrom: Wow, that's kind of awful. | 17:20 |
| galstrom | defcore before there was a defcore | 17:20 |
| mancdaz | we're not going to use the defcore tests anywhere in the very short term | 17:20 |
| galstrom | Apsu: right.. i am actually a little proud of that horribleness | 17:20 |
| mancdaz | galstrom still doesn't beat that ruby shit you did in chef | 17:20 |
| galstrom | mancdaz: agreed.. but this way.. if I run an install.. i can just run that script with the defcore argument.. | 17:20 |
| galstrom | mancdaz: did you see that is still in there | 17:21 |
| mancdaz | galstrom haha no | 17:21 |
| mancdaz | I don't look back - only forwards | 17:21 |
| mancdaz | that's our matra right? | 17:21 |
| mancdaz | *mantra | 17:21 |
| mancdaz | cloudnull I blame sigmavirus24 | 17:22 |
| galstrom | mancdaz: just for you https://github.com/stackforge/cookbook-openstack-compute/blob/master/templates/default/api-paste.ini.erb#L100-L103 | 17:22 |
| mancdaz | he broked glance so bad | 17:22 |
| mancdaz | MY EYES | 17:22 |
| cloudnull | he's core. which means it is his fault. | 17:22 |
| galstrom | you are welcome | 17:22 |
| mancdaz | actually it's not quite as bad as I remember | 17:23 |
| mancdaz | but it's still hideous | 17:23 |
| Apsu | galstrom: I'm going to need you to fill out an Ishikawa diagram and answer 5 "Why?"s for your crimes. | 17:24 |
| Apsu | Actually, 5 might not be enough | 17:24 |
| galstrom | yeah.. 5 is probably a bit low | 17:24 |
| galstrom | mancdaz: s/hideous/perfectly clear/ | 17:25 |
| Apsu | Sam-I-Am: Neat: https://bugzilla.redhat.com/show_bug.cgi?id=1160815#c1 verifies my suspicion that setting the forward_delay is only relevant to stp!=0, and LB agent sets stp=0. Also this indicates the kernel wouldn't accept forward_delay=0 anyway. | 17:31 |
| openstack | bugzilla.redhat.com bug 1160815 in NetworkManager "DELAY=0 in bridge configuration results in default (30 second) listening/learning delay" [Unspecified,Closed: currentrelease] - Assigned to lrintel | 17:31 |
| Apsu | So... going to try the bridge stoofs with ip link and ignore fd. | 17:31 |
| Sam-I-Am | hmmmmm | 17:41 |
| Sam-I-Am | interesting | 17:41 |
| *** jwagner_away is now known as jwagner | 17:43 | |
| sigmavirus24 | mancdaz: I broked glance, eh? | 17:50 |
| rackertom | claco: I think in general the issues have all been perfect storm situations. But that's usually when you want things to be least frustrating the most. | 18:04 |
| *** jmccrory has quit IRC | 18:09 | |
| *** jmccrory has joined #openstack-ansible | 18:09 | |
| *** Mudpuppy_ has joined #openstack-ansible | 18:19 | |
| *** Mudpuppy has quit IRC | 18:21 | |
| *** jwagner is now known as jwagner_away | 18:50 | |
| *** KLevenstein has quit IRC | 18:57 | |
| *** KLevenstein has joined #openstack-ansible | 19:06 | |
| *** rromans has joined #openstack-ansible | 19:14 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 19:17 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 19:17 | |
| *** rrrobbb has quit IRC | 19:17 | |
| *** Mudpuppy_ has quit IRC | 19:18 | |
| *** Mudpuppy has joined #openstack-ansible | 19:18 | |
| *** KLevenstein has quit IRC | 19:55 | |
| *** KLevenstein has joined #openstack-ansible | 20:22 | |
| dstanek | cloudnull: what was that tempest error you were saying to watch out for? | 20:24 |
| *** hj-hp has joined #openstack-ansible | 20:24 | |
| dstanek | right now i'm getting an ssh timeout | 20:25 |
| palendae | yeah, that one's been pretty prevalent | 20:25 |
| dstanek | ok, if it's not just me i'll move on | 20:26 |
| palendae | Nope, lots of us | 20:27 |
| openstackgerrit | Kevin Carter proposed stackforge/os-ansible-deployment: Updated repository for minimum viable kilo install https://review.openstack.org/166986 | 20:29 |
| *** jmccrory1 has joined #openstack-ansible | 20:34 | |
| *** jmccrory has quit IRC | 20:34 | |
| *** jmccrory has joined #openstack-ansible | 20:39 | |
| *** jmccrory1 has quit IRC | 20:42 | |
| dstanek | if i run ./scripts/teardown.sh should i expect to be able to run ./scripts/gate-check-commit.sh again on the same node? | 20:59 |
| *** openstackgerrit has quit IRC | 21:07 | |
| *** openstackgerrit has joined #openstack-ansible | 21:07 | |
| cloudnull | dstanek yes it should. | 21:10 |
| dstanek | cloudnull: i got a crazy hard drive error - if i can reproduce i'll file a bug | 21:10 |
| *** Mudpuppy_ has joined #openstack-ansible | 21:11 | |
| cloudnull | whats the error ? | 21:12 |
| *** Mudpuppy has quit IRC | 21:13 | |
| dstanek | cloudnull: gone now :-( | 21:14 |
| cloudnull | yw? | 21:14 |
| cloudnull | :) | 21:14 |
| dstanek | cloudnull: no i just close that tmux window. so i have no way to know | 21:15 |
| cloudnull | ah. | 21:15 |
| dstanek | new machine is being provisioned now so i'll know shortly | 21:17 |
| *** DustD1_ has left #openstack-ansible | 21:33 | |
| mattt | anyone know what these memcache failures are? | 21:34 |
| mattt | is it limited in scope to HP instances? | 21:34 |
| *** jmccrory has quit IRC | 21:34 | |
| cloudnull | what failures mattt ? | 21:36 |
| mattt | cloudnull: http://logs.openstack.org/09/167009/1/gate/os-ansible-deployment-dsvm-check-commit/cd57ee9/console.html | 21:37 |
| mattt | i've seen a ton of these in the past 12 hours | 21:37 |
| mattt | cloudnull: more specifically http://logs.openstack.org/09/167009/1/gate/os-ansible-deployment-dsvm-check-commit/cd57ee9/console.html#_2015-03-24_21_32_35_108 | 21:37 |
| cloudnull | looks like [ 2015-03-24 21:32:35,057 p=30505 u=root | failed: [aio1_memcached_container-9084a693] => (item=memcached) => {"attempts": 5, "failed": true, "item": "memcached"} ] | 21:39 |
| cloudnull | which seems to indicate that apt repo is dead or down. | 21:39 |
| mattt | we don't pin memcached do we ? | 21:40 |
| mattt | always seems to fail there | 21:40 |
| cloudnull | no its going back to the apt repo at archive.ubuntu or mirror.rackspace. | 21:41 |
| cloudnull | but it does seem to fail a lot on the hp instances. | 21:42 |
| mattt | cloudnull: that's what i'm noticing | 21:44 |
| mattt | i've built a bunch of RS instances today and have not hit that once | 21:44 |
| *** sc has quit IRC | 21:50 | |
| Apsu | mirror.rackspace.com is a much better mirror than archive | 21:50 |
| Apsu | Well, much less loaded | 21:50 |
| *** sc has joined #openstack-ansible | 21:52 | |
| *** jmccrory has joined #openstack-ansible | 21:53 | |
| *** jmccrory has quit IRC | 21:53 | |
| *** daneyon_ has joined #openstack-ansible | 21:56 | |
| *** daneyon has quit IRC | 21:59 | |
| *** dstanek has quit IRC | 22:01 | |
| *** hj-hp has quit IRC | 22:06 | |
| *** hj-hp has joined #openstack-ansible | 22:07 | |
| mattt | i'm not even sure neutron is getting set up properly on this kilo patch | 22:11 |
| mattt | cloudnull: did you have any luck booting instances and assigning floats etc? | 22:11 |
| *** dstanek has joined #openstack-ansible | 22:14 | |
| *** Mudpuppy_ has quit IRC | 22:26 | |
| *** KLevenstein has quit IRC | 22:37 | |
| *** BjoernT has quit IRC | 22:59 | |
| *** markvoelker has quit IRC | 23:06 | |
| *** sdake__ has quit IRC | 23:10 | |
| *** hj-hp has quit IRC | 23:12 | |
| *** galstrom is now known as galstrom_zzz | 23:12 | |
| *** dstanek has quit IRC | 23:17 | |
| *** dstanek has joined #openstack-ansible | 23:18 | |
| dstanek | cloudnull: i'm getting this after running gate-check-commit.sh && teardown.sh && gate-check-commit.sh - http://paste.openstack.org/raw/196245/ | 23:20 |
| openstackgerrit | Kevin Carter proposed stackforge/os-ansible-deployment: Updated repository for minimum viable kilo install https://review.openstack.org/166986 | 23:24 |
| cloudnull | dstanek: that issue is when a partion table is re-written and the host is having issues informing the kernel of the change. | 23:26 |
| cloudnull | i've seen that a few times when the lxc containers on the /var/lib/lxc are on a partition that is not empty. | 23:27 |
| *** jmccrory has joined #openstack-ansible | 23:27 | |
| dstanek | cloudnull: how can i recover? | 23:30 |
| dstanek | i could try a reboot, but that's harder to automate | 23:30 |
| cloudnull | when the host goes into that state a reboot is just about all you can do. | 23:30 |
| cloudnull | are you automating the teardown / rebuild ? | 23:30 |
| cloudnull | are you looking to try and automate the redeployment of keystone ? | 23:31 |
| dstanek | not at this point. but i was doing similar things with devstack | 23:31 |
| cloudnull | like build destroy, build again ? | 23:31 |
| dstanek | i have some script that configure devstack, run tests, teardown and start a new configuration | 23:32 |
| dstanek | similar to what the gate does, except that i am on the same node | 23:32 |
| cloudnull | rax pub cloud image ? | 23:33 |
| cloudnull | if so, which one ? | 23:33 |
| dstanek | yeah. for the devstack tests i use a 4g performance instance running 14.04 | 23:33 |
| *** stevemar has quit IRC | 23:34 | |
| cloudnull | i've had really good success with the "Ubuntu 14.04 LTS (Trusty Tahr) (PVHVM)" image and the flavor Performance1-8/General1-8 | 23:34 |
| dstanek | that's what i'm using right now for this node | 23:34 |
| *** stevemar has joined #openstack-ansible | 23:35 | |
| cloudnull | so we can attack it another way. | 23:36 |
| cloudnull | when you build and then test, you can run the lxc-container-destory.yml plau | 23:36 |
| cloudnull | which will destroy all of the containers and their resulting binds/bits | 23:36 |
| cloudnull | and then rerun the setup-everything.yml play | 23:37 |
| cloudnull | to rebuild. | 23:37 |
| dstanek | i'll give that a try | 23:37 |
| dstanek | none of this is really necessary just yet. trying to get a feel for what i can do | 23:37 |
| cloudnull | doing it that way would be the fastest way to rebuild all the things. as the partitioning and base system packages would already be resolved. | 23:38 |
| cloudnull | workflow might be -- build: openstack-ansible haproxy-install.yml setup-everything.yml, destroy: openstack-ansible lxc-container-destroy.yml, rebuild: openstack-ansible setup-everything.yml | 23:40 |
| cloudnull | which is essentially what the scripts are running for you. except the teardown one, which is doing a lot more to nuke everything from orbit. | 23:41 |
| dstanek | yeah, that really does nuke everything | 23:45 |
| cloudnull | it tries its best, but may be a bit to aggressive. I made it a script instead of a base play because i didn't want someone accidentally executing it destroying parts of their cloud by accident. | 23:46 |
| *** markvoelker has joined #openstack-ansible | 23:47 | |
| dstanek | even after the reboot i can't get it recreated, but i haven't look into it much yet | 23:49 |
| *** markvoelker has quit IRC | 23:52 | |
| dstanek | ok, looking better. small bug in one of the scripts | 23:53 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!