| *** sdake_ has quit IRC | 00:00 | |
| *** sacharya has quit IRC | 00:04 | |
| *** mahito has joined #openstack-ansible | 00:22 | |
| *** bilal has joined #openstack-ansible | 00:25 | |
| bilal | any workaround? | 00:26 |
|---|---|---|
| bilal | hey.. TASK: [container_restart | Test Container Networking] always gives errors and times out | 00:26 |
| *** bilal has quit IRC | 00:35 | |
| *** bilal has joined #openstack-ansible | 00:35 | |
| bilal | what is elasticksearch container | 00:36 |
| cloudnull | o/ bilal | 00:37 |
| cloudnull | what would you like to know ? | 00:37 |
| bilal | there is this elasticsearch_container that gets created on the logger host. And when this container gets restarted in the task TASK: [container_restart | Restart containers] during rackspace10 install; it always gives error | 00:39 |
| cloudnull | whats the error ? | 00:40 |
| bilal | {"err": "Failed to change the state of the container.", "failed": true, "item": "logging_elasticsearch_container-e3833834", "rc": 2} | 00:40 |
| bilal | msg: State failed to change for container [ logging_elasticsearch_container-e3833834 ] -- [ running ] != [ stopped ] | 00:42 |
| cloudnull | it seems that ansible is not able to stop the container on the logging host. | 00:43 |
| cloudnull | if you login to the logging host | 00:43 |
| cloudnull | can you do an "lxc-stop -n logging_elasticsearch_container-e3833834" | 00:43 |
| cloudnull | and if you do, does the container stop? which you can tell with "lxc-ls -f" | 00:44 |
| bilal | the container is already in STOPPED state | 00:45 |
| bilal | and lxc-start is giving this error: | 00:46 |
| bilal | lxc-start: conf.c: instantiate_veth: 2978 failed to attach 'vethJOVYLX' to the bridge 'lxcbr0': No such device | 00:46 |
| cloudnull | if you do an ip a l breth0 | 00:46 |
| cloudnull | does that show you an lxc bridge device ? | 00:47 |
| cloudnull | sorry | 00:47 |
| cloudnull | ip a l lxcbr0 | 00:47 |
| bilal | no.. Device "lxcbr0" does not exist. | 00:47 |
| bilal | how do you create lxcbr0 manually? | 00:48 |
| cloudnull | are you on 10.1.2 ? | 00:49 |
| cloudnull | either way you can run "lxc-system-manage system-force-rebuild" on the logging host | 00:50 |
| cloudnull | That will rebuild everything needed to make the lxc system run and it should start the containers | 00:50 |
| bilal | it did.. thanks a lot. Really appreciate the help | 00:52 |
| cloudnull | anytime . | 00:54 |
| *** sacharya has joined #openstack-ansible | 01:34 | |
| bilal | in the TASK: [container_restart | Restart containers] while running host_setup playbook for rackspace 10; 90% of the conatiners on all hosts gets timedout with this error: msg: Timeout when waiting for 10.xx.xx.196:22 | 01:43 |
| bilal | failed: [bilalrsinfra-XS23-TY3] => (item=bilalrsinfra-XS23-TY3_nova_api_os_compute_container-32b91bc2) => {"elapsed": 180, "failed": true, "item": "bilalrsinfra-XS23-TY3_nova_api_os_compute_container-32b91bc2"} | 01:44 |
| bilal | and the playbook ends with: FATAL: all hosts have already failed -- aborting | 01:48 |
| bilal | any help? | 01:48 |
| cloudnull | sounds like an ip conflict, ie the management network is either not ready or unavailable on the host. | 01:57 |
| cloudnull | can you ssh to the containers on one of the failing hosts ? | 01:57 |
| bilal | To the host, yes i can ssh from deploy machine. But to that container inside host i cannot even ping from deploy host | 02:03 |
| *** sacharya has quit IRC | 02:03 | |
| cloudnull | so from the deploy host you can not ping or ssh to the containers ? | 02:05 |
| bilal | no i cant | 02:05 |
| cloudnull | that seems like the host network is not setup or that there is some type of filtering on the hosts that is making networking impossible. | 02:06 |
| bilal | okay. let me check | 02:11 |
| *** stevemar has joined #openstack-ansible | 02:12 | |
| *** mahito_ has joined #openstack-ansible | 02:15 | |
| *** mahito has quit IRC | 02:17 | |
| openstackgerrit | Merged stackforge/os-ansible-deployment: Flake8 update - inventory-manage.py https://review.openstack.org/171102 | 02:31 |
| *** sacharya has joined #openstack-ansible | 02:33 | |
| *** mahito has joined #openstack-ansible | 02:40 | |
| *** mahito_ has quit IRC | 02:43 | |
| *** mahito has quit IRC | 02:54 | |
| *** mahito has joined #openstack-ansible | 03:19 | |
| openstackgerrit | Kevin Carter proposed stackforge/os-ansible-deployment: Updated the repo scripts https://review.openstack.org/171777 | 03:38 |
| *** mahito has quit IRC | 03:58 | |
| *** mahito has joined #openstack-ansible | 04:03 | |
| openstackgerrit | Sudarshan Acharya proposed stackforge/os-ansible-deployment: Managing policy file with default file and user variables. https://review.openstack.org/168104 | 04:22 |
| *** sacharya has quit IRC | 04:26 | |
| *** raginbajin has quit IRC | 04:32 | |
| *** javeriak has quit IRC | 04:33 | |
| *** bilal has quit IRC | 04:34 | |
| *** raginbajin has joined #openstack-ansible | 04:35 | |
| *** sacharya has joined #openstack-ansible | 04:35 | |
| *** mahito has quit IRC | 04:39 | |
| *** mahito has joined #openstack-ansible | 04:40 | |
| *** stevemar has quit IRC | 04:57 | |
| *** sacharya has quit IRC | 05:24 | |
| *** mahito has quit IRC | 05:32 | |
| *** mahito has joined #openstack-ansible | 05:33 | |
| *** nosleep77 has quit IRC | 06:00 | |
| *** mahito has quit IRC | 06:10 | |
| *** mahito has joined #openstack-ansible | 06:17 | |
| *** mahito has quit IRC | 06:22 | |
| *** mahito has joined #openstack-ansible | 06:26 | |
| *** sdake_ has joined #openstack-ansible | 06:32 | |
| *** javeriak has joined #openstack-ansible | 06:51 | |
| *** sdake_ has quit IRC | 07:05 | |
| *** mahito has quit IRC | 07:16 | |
| *** sdake has joined #openstack-ansible | 07:18 | |
| *** sdake_ has joined #openstack-ansible | 07:25 | |
| *** sdake has quit IRC | 07:26 | |
| *** sdake_ has quit IRC | 07:30 | |
| *** sdake has joined #openstack-ansible | 07:52 | |
| odyssey4me | wow, we suddenly have several tempest failures in the gate... everything was operating just fine yesterday - I wonder what changed... perhaps another dependency change in the mix? or... I suppose we're tracking master so it could be any number of patches upstream | 08:23 |
| *** javeriak has quit IRC | 09:00 | |
| odyssey4me | andymccr hughsaunders I'm guessing that we need an environment change to place the ELK containers on a particular host? | 09:30 |
| andymccr | odyssey4me: yeh | 09:31 |
| odyssey4me | as such, it would seem that https://review.openstack.org/170558 is necessary, right? | 09:31 |
| andymccr | ive done that | 09:31 |
| andymccr | to make it easiest yeh that'd be required | 09:31 |
| andymccr | otherwise you'd have to manually edit the env file which is more admin/complicated | 09:32 |
| odyssey4me | yeah, without that patch we'd have to ask a deployer to edit the environment file - whereas with the patch we can simply copy a file over | 09:32 |
| andymccr | But I used the logging to test that patch, so here is a logging.yml to pop inside env.d http://pastie.org/10082059 | 09:32 |
| andymccr | tempted to do a separate patch to remove the md5sum stuff cos that becomes even less useful when using env.d | 09:35 |
| odyssey4me | andymccr go for it | 09:35 |
| andymccr | odyssey4me: can you think of another way to solve that issue? (Issue being - if we do an upgrade from say 9-10 or 11-12 or w/e and the default environment file/user_vars have changed we won't know) | 09:35 |
| andymccr | so you will run the playbooks and it will just fail out until you have added all the vars in. | 09:36 |
| andymccr | even though we update the sample user_variables file. | 09:36 |
| odyssey4me | andymccr well, in theory that's technical debt to be handled by the upgrade script - surely? | 09:36 |
| andymccr | odyssey4me: not really | 09:36 |
| andymccr | unless we are having an upgrade script between each version all the time | 09:36 |
| andymccr | example, the service_profiler_hmac_key vars that were added as part of kilofication | 09:37 |
| andymccr | if you just upgrade from before kilofication to after you will fail out unless you update your vars file, but you don't really have any notifier to update your vars file | 09:37 |
| odyssey4me | andymccr good question, for which I have no answer currently - but my thinking is that we need an upgrade script to handle transitions between variable renames and other upgrading issues for any upgrades | 09:38 |
| odyssey4me | but I can see how that would get messy quickly | 09:38 |
| odyssey4me | the method thus far appears to be to do the work in isolation first and to consider the upgrade problem while doing so, but not to be hung up on it | 09:38 |
| andymccr | odyssey4me: yeh so the initial md5sum never really solved it, but in theory it would prompt you to check your user_vars/env files to ensure they are updated | 09:38 |
| andymccr | in practice it didnt do that at all, and really didnt do anything | 09:39 |
| *** sdake_ has joined #openstack-ansible | 09:46 | |
| openstackgerrit | Andy McCrae proposed stackforge/os-ansible-deployment: Remove md5sum check for environment vs user_config https://review.openstack.org/171987 | 09:49 |
| *** sdake_ has quit IRC | 09:49 | |
| *** sdake has quit IRC | 09:50 | |
| andymccr | hmm ok so cloudnull put a comment on the initial bug - maybe my understanding of the point of the md5sum is wrong. can abandon if necessary but i'm still not sure it solves a problem that anybody has , since your environment will never have changed unless you change it :) | 09:52 |
| openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: Adjust AIO swap size to 1.5x RAM https://review.openstack.org/171992 | 10:15 |
| andymccr | we currently dont have a way to overwrite the size of a container for 1 specific container? | 10:39 |
| odyssey4me | andymccr dunno, don't we? | 11:00 |
| odyssey4me | I just encountered another issue relating to the wheel repo - how do we implement additional wheels to be built for extras? | 11:01 |
| odyssey4me | (without adjusting osad) | 11:01 |
| andymccr | odyssey4me: we don't but im patching it now :D (re: the fs_size) | 11:03 |
| andymccr | the repo stuff im not sure honestly | 11:04 |
| andymccr | i dont think we do or have a good mechanism | 11:04 |
| andymccr | because for some of the extras that got moved out i needed to do things like "--isolation" for installing pip packages to avoid the repo :) (which is not ideal) | 11:04 |
| odyssey4me | well, I know that it's easy enough to add something to the appropriate file, then rebuild the repo... but that's hardly ideal and definitely breaks the 'frozen repo' model | 11:06 |
| andymccr | yeh | 11:06 |
| andymccr | quite! | 11:06 |
| odyssey4me | we have to implement the extras in such a way that it's more integrated, not an after-thought | 11:06 |
| odyssey4me | I had some discussions yesterday for doing that, but want to get this done before going there. | 11:07 |
| andymccr | agree, but i think the only way to do that is to treat all things the same - e.g. have an over-arching repo that then pulls in all roles (e.g. any 3rd party roles, and os-a-d roles and extra roles) | 11:07 |
| andymccr | that way they are all the same, and that one over-arching repo can store your env/vars/playbooks for everything | 11:07 |
| andymccr | but it raises questions about a community version of that over-arching repo and that'd have to differ so its not quite that simple | 11:08 |
| odyssey4me | but that's not possible now with all roles in a single repo, so we'll have to fudge it with scripts | 11:08 |
| odyssey4me | ansible-galaxy's command-line can do the imports, but only if the roles are in their own repo | 11:08 |
| andymccr | so thats the problem, the "design" works except that how its setup now doesnt match the design | 11:09 |
| andymccr | e.g. all the roles should be separate entirely and all be pulled in | 11:09 |
| andymccr | but as it stands we have basically all the roles already so any 3rd party/extras roles are "after thoughts" | 11:09 |
| odyssey4me | so we have the option of doing our extras roles in separate repositories, and having the extras repo simply housing the plays - then the plays are run from the osad repo root just like all the osad standard stuff | 11:10 |
| odyssey4me | that way we don't have to carry a copy of the inventory module, or any plugins/libraries | 11:10 |
| andymccr | sure but the plays themselves would have to be carried in the os-a-d repo when they are not os-a-d plays | 11:11 |
| andymccr | as would the vars/env files | 11:11 |
| odyssey4me | it would change the workflow somewhat, but it would be a lot more integrated and deploy more cohesively | 11:11 |
| odyssey4me | andymccr nope the plays can be external entirely, they're just executed from inside the osad/playbooks directory | 11:12 |
| andymccr | sure but how are you pulling those in then | 11:12 |
| odyssey4me | eg: from /opt/os-ansible-deploy/playbooks you execute: openstack-ansible /opt/rpc-extras/playbooks/blah.yml | 11:12 |
| andymccr | so you are cloning that repo anyway? | 11:13 |
| odyssey4me | the roles in the ansible bootstrap (part of setup) would be imported into the ansible namespace (so it doesn't dirty the osad git tree) - ie they will be located in /etc/ansible/roles/ | 11:13 |
| openstackgerrit | Merged stackforge/os-ansible-deployment: Update tempest config for current master https://review.openstack.org/171142 | 11:14 |
| odyssey4me | andymccr yeah, the repo cloning is inevitable unless we package the plays - but note that the repo contents are simply the plays and any other convenience scripts... no roles | 11:14 |
| andymccr | i understand how the roles get pulled in, but then you are cloning a repo to run a play from within the os-a-d tree (the play exists elsewhere) to then run plays you have pulled in. you could argue thats less integrated | 11:14 |
| odyssey4me | it's the only way I can see it working with the way the repositories are structured now | 11:15 |
| andymccr | id argue you would have a my-common, which pulls in the individual os-a-d roles and any addition 3rd party/extra roles and inside that repo all the plays i need exist. | 11:15 |
| odyssey4me | the better way is to split the roles into repositories, for sure, but that adds much more complexity | 11:15 |
| andymccr | is the point not that the way its structured now means there is no ideal solution, so the way it is now is fine since its potato/potato (which doesnt work over irc) | 11:16 |
| *** subscope has quit IRC | 11:17 | |
| odyssey4me | yeah, but now I'm having to build osad - then re-run various osad plays in order to build containers and configure them once I have the extras setup | 11:17 |
| andymccr | not really? i mean you have to copy over the env files etc | 11:17 |
| andymccr | if you've done that you dont have to run anything extra | 11:17 |
| andymccr | which is the same as if you pull in the roles - you would still need to copy the env files | 11:17 |
| andymccr | if you havent done that then sure you will. | 11:17 |
| odyssey4me | yeah, I suppose that's fair - it is possible to do most of the setup initially if you're aware of how to do it all | 11:18 |
| odyssey4me | it's clunky and sucks though :( | 11:18 |
| andymccr | but that doesnt change if you pull in the role | 11:18 |
| andymccr | because the bits that define what gets setup still need to be done, so if you dont know to do that you will pull in the roles fine and they wont work | 11:18 |
| andymccr | that step of "copy over vars/env" is unavoidable and would still sit in rpc-extras so that step is exactly the same | 11:19 |
| odyssey4me | yeah, agreed - the only difference with the suggested change of workflow is that the extras repo doesn't need to carry libraries/inventory | 11:19 |
| andymccr | yeh basically, but trade off is that you then run a play from inside the os-a-d dir space which is quite clunky also | 11:19 |
| *** subscope has joined #openstack-ansible | 11:20 | |
| odyssey4me | blast, with the current implementation you can't do role dependancies | 11:27 |
| odyssey4me | andymccr any thoughts on how I get pip installed into the container using the pip_install role (ideally) in the current setup? | 11:27 |
| odyssey4me | I don't want to use the pip_lockdown - but even if I did the same problem would apply. | 11:28 |
| andymccr | i feel like i must've done this lemme look | 11:29 |
| *** stevemar has joined #openstack-ansible | 11:43 | |
| openstackgerrit | Andy McCrae proposed stackforge/os-ansible-deployment: Allow lxc_container settings per container type https://review.openstack.org/172019 | 11:53 |
| openstackgerrit | Andy McCrae proposed stackforge/os-ansible-deployment: Set the container_fs_size to 12G for Glance https://review.openstack.org/172020 | 11:55 |
| *** stevemar has quit IRC | 12:02 | |
| odyssey4me | andymccr did you figure out your method? | 12:25 |
| odyssey4me | (for pip installs) | 12:25 |
| odyssey4me | as far as I can see, pip_install as a role is depended on by pip_lockdown, which is depended on by each service that uses pip packages | 12:26 |
| andymccr | hmm | 12:26 |
| andymccr | so pip was already installed because im not creating anything new | 12:26 |
| andymccr | so i didnt solve that problem :/ | 12:26 |
| odyssey4me | yeah, in this case for the ELK containers there is no pip because nothing installs it - you were working on hosts which have it installed already | 12:26 |
| andymccr | yeh | 12:27 |
| andymccr | hmm | 12:27 |
| andymccr | so the problem is that now we want to install pip packages that arent in the repo? | 12:27 |
| odyssey4me | if we were running the plays from inside os-ansible-deployment/playbooks then it would have access to the roles... but we aren't | 12:27 |
| andymccr | so if we run the pip-install role against those hosts then its locked down | 12:27 |
| odyssey4me | nope, the pip_install role doesn't lock it down - the problem is that the extras playbooks don't have access to the pip_install role | 12:28 |
| odyssey4me | so either I have to install pip in the elasticsearch role using whatever method I choose, or I need to find a way to depend on the pip_install role | 12:28 |
| andymccr | but the nova role doesnt call the pip_install role for example | 12:30 |
| andymccr | or theyre using a dependency on pip_lockdown | 12:31 |
| odyssey4me | yes, it depends on pip_lockdown, which depends on pip_install | 12:31 |
| andymccr | thats problematic then | 12:32 |
| odyssey4me | I'll symlink it for now, but this provides a pretty hard reason to change how we're implementing the extras. | 12:34 |
| odyssey4me | A way, of course, would be to extract the pip_install into its own role and pull it into the ansible namespace using ansible-galaxy. | 12:34 |
| andymccr | i dont think its just extras that needs to change the implementation | 12:34 |
| andymccr | can we not include roles based on a location? | 12:35 |
| andymccr | instead of symlinking | 12:35 |
| odyssey4me | yeah, agreed - the point being is that now that we're faced with actually trying to do this it's turning out to be more complex than originally thought | 12:35 |
| odyssey4me | unfortunately ansible-galaxy can't import from a file location | 12:35 |
| odyssey4me | http://docs.ansible.com/galaxy.html#advanced-control-over-role-requirements-files | 12:36 |
| odyssey4me | you can source from ansible galaxy, git, bitbucket, http but not file it seems | 12:36 |
| andymccr | http://docs.ansible.com/intro_configuration.html#roles-path | 12:37 |
| andymccr | could adjust the roles_path in ansible.cfg for the extras | 12:37 |
| odyssey4me | ooh, there's an option - let me try that | 12:38 |
| odyssey4me | nice one :) | 12:38 |
| andymccr | if only we could do that for library also | 12:38 |
| odyssey4me | http://docs.ansible.com/intro_configuration.html#library | 12:38 |
| andymccr | oh you can i think. maybe | 12:38 |
| odyssey4me | hahaha | 12:38 |
| andymccr | and inventory... | 12:39 |
| odyssey4me | and http://docs.ansible.com/intro_configuration.html#inventory | 12:39 |
| andymccr | haha | 12:39 |
| odyssey4me | great minds... | 12:39 |
| andymccr | well that'd solve a lot of that fudgery. still not perfect, but at least it resolves the majority | 12:39 |
| odyssey4me | yes, let me try them out and see - this may actually simplify a lot of things | 12:40 |
| odyssey4me | it'll still require a prep script to be run to set the directory, but that's small change | 12:40 |
| odyssey4me | or perhaps just an assumption about where the os-ansible-deployment is located and a documented warning | 12:40 |
| andymccr | well there is a default i think | 12:42 |
| andymccr | its just that i dont ever deploy it in the default :D | 12:42 |
| andymccr | so if you do it should work without setting that | 12:42 |
| odyssey4me | ooh, it works | 12:46 |
| odyssey4me | so you can set the roles path and it still checks the <current directory>/roles | 12:47 |
| odyssey4me | but with this we can include the os-ansible-deployment/playbooks/roles/ too | 12:47 |
| odyssey4me | and it appears that the dynamic inventory works too | 12:47 |
| andymccr | awesome | 12:48 |
| andymccr | MERGE IT! | 12:48 |
| odyssey4me | yeah, I'll prep a patch for that in particular and include a README edit | 12:49 |
| cloudnull | Morning. | 13:13 |
| *** Mudpuppy has joined #openstack-ansible | 13:19 | |
| *** Mudpuppy has quit IRC | 13:25 | |
| *** Mudpuppy has joined #openstack-ansible | 13:25 | |
| odyssey4me | o/ cloudnull | 13:26 |
| odyssey4me | has anyone had any luck ferreting out why the gate's broken today? | 13:35 |
| odyssey4me | for master, at the very least | 13:35 |
| cloudnull | i just started looking at that | 13:41 |
| *** stevemar has joined #openstack-ansible | 13:46 | |
| andymccr | odyssey4me: i have not yet had a chance to. | 13:46 |
| *** sacharya has joined #openstack-ansible | 13:49 | |
| *** ccrouch has joined #openstack-ansible | 13:52 | |
| *** openstackgerrit has quit IRC | 13:53 | |
| *** openstackgerrit has joined #openstack-ansible | 13:53 | |
| *** markvoelker has quit IRC | 13:58 | |
| *** markvoelker has joined #openstack-ansible | 13:59 | |
| openstackgerrit | Matthew Kassawara proposed stackforge/os-ansible-deployment: Remove deprecated use_namespaces option https://review.openstack.org/172068 | 14:05 |
| mattt | lots of 503 from keystone in gate errors | 14:06 |
| andymccr | mattt: yeh but its weirdly only when tempest seems to install/setup ? | 14:06 |
| *** sdake has joined #openstack-ansible | 14:10 | |
| mattt | andymccr: http://logs.openstack.org/36/171036/4/gate/os-ansible-deployment-dsvm-check-commit/f0aa7a7/logs/aio1_nova_api_os_compute_container-fa731224/nova-api-os-compute.log | 14:14 |
| Sam-I-Am | mattt: this happened to me | 14:18 |
| *** KLevenstein has joined #openstack-ansible | 14:23 | |
| mattt | Sam-I-Am: on your aio? | 14:23 |
| mattt | Sam-I-Am: do you still have it up? | 14:23 |
| Sam-I-Am | mattt: yeah, and no... i tore it down for a rebuild | 14:31 |
| Sam-I-Am | thinking it was transient | 14:31 |
| odyssey4me | I think it's likely that something upstream has changed and is affecting us. | 14:32 |
| * Sam-I-Am shockedface | 14:32 | |
| Sam-I-Am | things breaking basic functionality after feature freeze? | 14:32 |
| Sam-I-Am | like the horizon refactor earlier this week | 14:33 |
| odyssey4me | Sam-I-Am typically it ends up being a dependent module, like one of the oslo goodies or something even further afield like requests | 14:33 |
| mattt | building an AIO now | 14:34 |
| odyssey4me | and hpcloud-b4's consistant apt install failures aren't helping | 14:34 |
| Sam-I-Am | ugh @ dumb | 14:39 |
| Sam-I-Am | https://bugs.launchpad.net/ubuntu/+source/nova/+bug/1439280 | 14:39 |
| openstack | Launchpad bug 1439280 in nova (Ubuntu) "Libvirt CPU affinity error" [Undecided,Incomplete] | 14:39 |
| Sam-I-Am | the last post before my reply... obviously didn't read the bug "testing using qemu" | 14:39 |
| Sam-I-Am | it probably works fine with kvm/nesting, but this is not the issue at hand | 14:40 |
| Sam-I-Am | and it only affects the ubuntu packages, not source, so its really an ubuntu problem | 14:40 |
| openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: [WIP] Diagnostic information for gate checks https://review.openstack.org/172096 | 14:47 |
| *** erikmwilson is now known as Guest81531 | 14:52 | |
| *** erikmwilson has joined #openstack-ansible | 14:52 | |
| *** erikmwilson_ has joined #openstack-ansible | 14:52 | |
| odyssey4me | cloudnull so no issues on larger cloud server builds? do you think this is a resourcing issue? | 15:04 |
| cloudnull | it looks that way | 15:04 |
| cloudnull | im trying again on p1.8 | 15:04 |
| odyssey4me | odd, because yesterday all was fine and dandy | 15:04 |
| cloudnull | and will try to narrow it down. | 15:05 |
| odyssey4me | the birds were tweeting in the clouds and stuff :p | 15:05 |
| cloudnull | also if we can get https://review.openstack.org/#/c/170952/ that eliminates the ec2 container | 15:06 |
| cloudnull | not that one container makes all the difference, but it might make some . | 15:06 |
| odyssey4me | cloudnull agreed, but I've been rechecking all day with no success | 15:06 |
| stevelle | communitay | 15:06 |
| Sam-I-Am | i'm testing with affinity = 1 | 15:07 |
| Sam-I-Am | which should reduce some things | 15:07 |
| odyssey4me | heh, ok - only one merge has succeeded since this morning | 15:07 |
| Sam-I-Am | kind of seems like galera might be ooming | 15:07 |
| Sam-I-Am | because everything starts 500ing when keystone falls over, and it seems to be having db problems | 15:07 |
| mattt | Sam-I-Am: that was my guess | 15:10 |
| Sam-I-Am | we are doing a LOT more things than typical devstack gating | 15:12 |
| odyssey4me | Sam-I-Am 'cos we're awesome like that :p | 15:15 |
| odyssey4me | is our reduced cache size for galera still taking effect? | 15:17 |
| odyssey4me | is it perhaps taking too much effect? | 15:18 |
| mattt | yeah gcache.size = 50M is still being set | 15:21 |
| mattt | my node just went mental while running tempest playbook | 15:25 |
| mattt | [ 3494.507150] Killed process 14410 (mysqld) total-vm:5564496kB, anon-rss:514892kB, file-rss:0kB | 15:26 |
| Sam-I-Am | mattt: standard aio with affinity > 1 ? | 15:28 |
| mattt | Sam-I-Am: yeah i just ran the gate-check-commit.sh script to boot the node | 15:30 |
| Sam-I-Am | i'm still running through my aff=1 | 15:31 |
| Sam-I-Am | so far there's a lot more resources available | 15:31 |
| Sam-I-Am | like... over a gig of ram free | 15:31 |
| *** Bjoern__ has joined #openstack-ansible | 15:32 | |
| *** jwagner_away is now known as jwagner | 15:37 | |
| *** jwagner is now known as jwagner_away | 15:38 | |
| Sam-I-Am | mattt: tempest passed | 15:47 |
| Sam-I-Am | mattt: some swap used, but almost a gig of ram available | 15:47 |
| mattt | Sam-I-Am: my aio has 1 GB of swap | 15:49 |
| mattt | Sam-I-Am: i'm currently running tempest (after having to rerun the tempest playbook) | 15:49 |
| *** Bjoern__ is now known as BjoernT | 15:49 | |
| mattt | Sam-I-Am: all swap used, ~ 100 MB of memory available | 15:49 |
| Sam-I-Am | eeooo | 15:53 |
| cloudnull | meeting in 5: cloudnull, mattt, andymccr, d34dh0r53, hughsaunders, b3rnard0, palendae, Sam-I-Am, odyssey4me, serverascode, rromans, mancdaz, dolphm, _shaps_, BjoernT, claco, echiu, dstanek | 15:54 |
| mattt | Sam-I-Am: yeah, my node is effectively unresponsive | 15:55 |
| *** BjoernT2 has joined #openstack-ansible | 15:58 | |
| *** jwagner_away is now known as jwagner | 16:00 | |
| *** BjoernT2 has quit IRC | 16:02 | |
| *** BjoernT2 has joined #openstack-ansible | 16:03 | |
| openstackgerrit | Merged stackforge/os-ansible-deployment: Correct a syntax error in some hosts: patterns https://review.openstack.org/169506 | 16:20 |
| odyssey4me | \o/ | 16:20 |
| * svg would like to know what people here were smoking whenthey designed the flow of variable definitions | 16:23 | |
| BjoernT | svg: +1 | 16:27 |
| odyssey4me | svg what is smoked by some, while interesting, is beyond the scope of discussion here :p | 16:28 |
| odyssey4me | heh, what're you referring to | 16:28 |
| Sam-I-Am | variable definitions? smoke? | 16:28 |
| odyssey4me | but first, you had best be talking about the master branch and not the previous branches... the older branches have a lot of organically grown mess :) | 16:29 |
| Sam-I-Am | odyssey4me: dont we prefer organic artisinal code over mass-produced gmo code? | 16:30 |
| odyssey4me | svg in the major re-working done in the master branch, we hope that we have done things in a more sane fashion... if not, then patches welcome :D | 16:31 |
| odyssey4me | Sam-I-Am no comment :p | 16:32 |
| cloudnull | svg community meeting in #openstack-meeting-4 if your interested. | 16:32 |
| *** mnestheu1 has joined #openstack-ansible | 16:32 | |
| andymccr | odyssey4me: im not sure we're on the same page re that bug? what regression is it? it seems to me like an upstream bug in the lineinfile module that miguelgrinberg already fixed | 17:00 |
| andymccr | or not fixed but addressed | 17:00 |
| BjoernT | andmccr: As i said in the bug, the regex is not good anyway | 17:01 |
| andymccr | im not sure what the correct resolution would be for fixing exisitng installs except to upgrade ansible | 17:01 |
| andymccr | BjoernT: yeh i agree its not amazing regex | 17:01 |
| BjoernT | it is actually not a regex | 17:01 |
| d34dh0r53 | just recheck/reverified all of the failed jobs | 17:01 |
| andymccr | well it just does a string search so its kinda like .*MaxSessions.* | 17:01 |
| odyssey4me | andymccr all I'm trying to highlight is that we've hit this before, not that we solved it in the right way | 17:02 |
| odyssey4me | as I recall we did two things - one in the plays and one in the gate bash script | 17:02 |
| andymccr | odyssey4me: the upstream lineinfile bug? oh ok. i mean sure we can work around it in the meantime | 17:03 |
| odyssey4me | anyway, glad we're hitting it again and hopefully we can find a better solution this round | 17:03 |
| andymccr | odyssey4me: surely the solution is ensure a version fo ansible > than the one with that bug in it? | 17:03 |
| odyssey4me | andymccr is it resolved in a specific version? | 17:04 |
| andymccr | https://github.com/ansible/ansible-modules-core/issues/736 | 17:04 |
| odyssey4me | sorry - I haven't looked into miguel's PR in detail | 17:04 |
| andymccr | not sure what the specific version is | 17:05 |
| andymccr | but > 1.6.10 :D | 17:05 |
| andymccr | also since our gate isnt seeing that issue im wondering if we're using a newer version for juno | 17:05 |
| odyssey4me | ah, then we have the awesome option of simply using a later version for icehouse/juno - master's already on a newer version | 17:05 |
| odyssey4me | the gate doesn't see it because we fiudge it | 17:05 |
| odyssey4me | https://github.com/stackforge/os-ansible-deployment/blob/juno/scripts/bootstrap-aio.sh#L85-L88 | 17:06 |
| odyssey4me | unless I'm completely confused about the issue - I did read it rather quickly | 17:08 |
| *** jwagner is now known as jwagner_away | 17:13 | |
| odyssey4me | I've got to run. We'll catch up tomorrow! | 17:13 |
| andymccr | can't wait odyssey4me! | 17:13 |
| *** openstack has quit IRC | 17:13 | |
| *** openstack has joined #openstack-ansible | 17:15 | |
| *** BjoernT2 has quit IRC | 17:20 | |
| openstackgerrit | Matthew Kassawara proposed stackforge/os-ansible-deployment: Update keystone middleware in nova for Kilo https://review.openstack.org/172153 | 17:27 |
| *** sdake_ has joined #openstack-ansible | 18:05 | |
| *** sdake has quit IRC | 18:09 | |
| *** javeriak has joined #openstack-ansible | 18:17 | |
| *** yaya has joined #openstack-ansible | 18:22 | |
| *** javeriak has quit IRC | 18:22 | |
| *** sdake has joined #openstack-ansible | 18:25 | |
| *** sdake_ has quit IRC | 18:29 | |
| openstackgerrit | Merged stackforge/os-ansible-deployment: Nova Kilofication Work https://review.openstack.org/170952 | 18:43 |
| d34dh0r53 | nova kilofication finally merged, that eliminates a container which I hope will take some of the memory pressure off of the gates | 18:45 |
| b3rnard0 | d34dh0r53: cool | 18:46 |
| *** javeriak has joined #openstack-ansible | 19:02 | |
| *** javeriak_ has joined #openstack-ansible | 19:07 | |
| *** javeriak has quit IRC | 19:08 | |
| *** erikmwilson has quit IRC | 19:11 | |
| *** javeriak has joined #openstack-ansible | 19:12 | |
| *** javeriak_ has quit IRC | 19:13 | |
| *** javeriak_ has joined #openstack-ansible | 19:14 | |
| *** javeriak has quit IRC | 19:16 | |
| *** javeriak has joined #openstack-ansible | 19:31 | |
| *** javeriak_ has quit IRC | 19:35 | |
| *** erikmwilson_ is now known as erikmwilson | 19:42 | |
| *** yaya has quit IRC | 19:49 | |
| *** jwagner_away is now known as jwagner | 20:01 | |
| openstackgerrit | Steve Lewis proposed stackforge/os-ansible-deployment: Genericize how we update SSL settings for Apache https://review.openstack.org/171838 | 20:05 |
| stevelle | I'm having major issues with irc today. Hoping I have them resolved now but lost some history. Did any progress happen around the broken gating on master? | 20:13 |
| BjoernT | who want's to debug #1440784, have one frozen spice console | 20:28 |
| *** stevelle has quit IRC | 20:34 | |
| *** stevelle has joined #openstack-ansible | 20:37 | |
| *** stevelle_ has joined #openstack-ansible | 20:38 | |
| *** stevelle has quit IRC | 20:39 | |
| *** stevelle_ is now known as stevelle | 20:52 | |
| *** sdake has quit IRC | 20:53 | |
| *** stevemar has quit IRC | 20:58 | |
| *** jwagner is now known as jwagner_away | 20:59 | |
| *** jwagner_away is now known as jwagner | 21:05 | |
| *** mancdaz has quit IRC | 21:35 | |
| *** mancdaz has joined #openstack-ansible | 21:38 | |
| openstackgerrit | Merged stackforge/os-ansible-deployment: Flake8 update - swift_rings.py https://review.openstack.org/171038 | 21:40 |
| *** BjoernT has quit IRC | 21:53 | |
| *** jwagner is now known as jwagner_away | 21:57 | |
| *** KLevenstein has quit IRC | 22:00 | |
| openstackgerrit | Steve Lewis proposed stackforge/os-ansible-deployment: Genericize how we update SSL settings for Apache https://review.openstack.org/171838 | 22:06 |
| *** sdake has joined #openstack-ansible | 22:07 | |
| *** mnestheu1 has quit IRC | 22:09 | |
| *** sdake_ has joined #openstack-ansible | 22:10 | |
| *** sdake has quit IRC | 22:14 | |
| openstackgerrit | Steve Lewis proposed stackforge/os-ansible-deployment: Genericize how we update SSL settings for Apache https://review.openstack.org/171838 | 22:34 |
| *** sdake_ has quit IRC | 22:47 | |
| *** JRobinson__ has joined #openstack-ansible | 22:48 | |
| *** sacharya has quit IRC | 23:10 | |
| *** britthouser has joined #openstack-ansible | 23:14 | |
| *** britthou_ has quit IRC | 23:17 | |
| *** britthouser has quit IRC | 23:21 | |
| *** britthouser has joined #openstack-ansible | 23:22 | |
| *** britthou_ has joined #openstack-ansible | 23:25 | |
| *** britthouser has quit IRC | 23:27 | |
| *** britthou_ has quit IRC | 23:27 | |
| *** britthouser has joined #openstack-ansible | 23:27 | |
| *** britthouser has quit IRC | 23:49 | |
| *** Mudpuppy has quit IRC | 23:51 | |
| *** britthouser has joined #openstack-ansible | 23:52 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!