openstackgerrit | Kevin Carter proposed stackforge/os-ansible-deployment: Updated for 10.1.3 tag https://review.openstack.org/172243 | 00:07 |
---|---|---|
*** stevelle has quit IRC | 01:45 | |
*** stevelle has joined #openstack-ansible | 01:46 | |
*** javeriak has quit IRC | 01:58 | |
openstackgerrit | Kevin Carter proposed stackforge/os-ansible-deployment: Updated galera cache size when boostrapping AIO https://review.openstack.org/172285 | 02:24 |
*** sdake has joined #openstack-ansible | 02:26 | |
*** sdake_ has joined #openstack-ansible | 02:54 | |
*** sdake has quit IRC | 02:57 | |
cloudnull | Sam-I-Am odyssey4me mattt mancdaz have a gander at - https://review.openstack.org/172285 - seems that tuning galera down for use in the AIO make the gate job complete faster/successfully. | 03:26 |
cloudnull | will recheck a couple of times just for the sake of testing but the results [ http://logs.openstack.org/85/172285/1/check/os-ansible-deployment-dsvm-check-commit/ac397c2/logs/instance-info/host_info_1428633111.log ] look good. | 03:27 |
*** JRobinson__ is now known as JRobinson__afk | 03:35 | |
*** markvoelker has quit IRC | 03:35 | |
*** sacharya has joined #openstack-ansible | 03:43 | |
*** markvoelker has joined #openstack-ansible | 04:06 | |
*** stevemar has joined #openstack-ansible | 04:07 | |
*** markvoelker has quit IRC | 04:12 | |
*** sdake_ has quit IRC | 04:27 | |
*** sacharya has quit IRC | 04:34 | |
*** javeriak has joined #openstack-ansible | 04:51 | |
*** javeriak has quit IRC | 04:53 | |
*** JRobinson__afk is now known as JRobinson__ | 05:11 | |
*** sdake has joined #openstack-ansible | 05:14 | |
*** sdake has quit IRC | 05:14 | |
*** sdake has joined #openstack-ansible | 05:14 | |
*** sdake_ has joined #openstack-ansible | 05:25 | |
*** sdake has quit IRC | 05:29 | |
*** stevemar has quit IRC | 05:32 | |
*** sdake_ has quit IRC | 05:55 | |
*** markvoelker has joined #openstack-ansible | 06:09 | |
*** markvoelker has quit IRC | 06:14 | |
*** ishant has joined #openstack-ansible | 06:25 | |
*** ishant has quit IRC | 06:34 | |
*** bilal has joined #openstack-ansible | 06:38 | |
*** ishant has joined #openstack-ansible | 06:40 | |
bilal | During rackspace 10 deployment when i run host-setup playbook; some of the containers are failing because they are stale entries from my old configuration. how can i remove them? | 06:41 |
bilal | how do i clear the cache when i run the host-setup playbook? | 06:44 |
*** ishant has quit IRC | 06:44 | |
*** ishant has joined #openstack-ansible | 06:44 | |
*** markvoelker has joined #openstack-ansible | 07:10 | |
*** javeriak has joined #openstack-ansible | 07:10 | |
*** markvoelker has quit IRC | 07:15 | |
mattt | bilal: you effectively want to do a complete re-install? is that correct? | 07:16 |
bilal | yes matt | 07:17 |
mattt | bilal: the ansible inventory script will write a few files to /etc/rpc_deploy, you'll want to clean those up | 07:18 |
bilal | okay.. and delete all the previous containers as well? | 07:19 |
mattt | bilal: i think the files in question are rpc_hostnames_ips.yml and rpc_inventory.json, but just compare to what is there and what was initially copied over (https://github.com/stackforge/os-ansible-deployment/tree/juno/etc/rpc_deploy) | 07:20 |
mattt | bilal: removing containers should do it for the most part, keep in mind that some services are not containerised though (nova-compute being one example) | 07:21 |
openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: Adjust AIO swap size to half of RAM, up to 8GB https://review.openstack.org/171992 | 07:23 |
bilal | okay. let me try that. Thanks mattt :D | 07:26 |
mattt | bilal: yeah just be careful that you don't delete those files on an operational cluster | 07:27 |
mattt | bilal: give us a shout if you have any other questions | 07:28 |
openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: Adjust AIO swap size to 4G/8G, depending on RAM amount https://review.openstack.org/171992 | 07:28 |
openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: [WIP] Diagnostic information for gate checks https://review.openstack.org/172096 | 07:30 |
bilal | one more question: i have no entry for aio1_swift_proxy_container-96f0ffc3 in my rpc_user_config.yml.. but the host-setup playbook always try to ssh into it and gives error | 07:33 |
odyssey4me | bilal that'll be inside your inventory somewhere, so take a look in /etc/rpc_deploy with grep -r and you should find it | 07:34 |
bilal | odyssey4me: ya it is in rpc_inventory.json file.. if i remove this entry, it wont cause any issue right? | 07:38 |
odyssey4me | that file is dynamically generated whenever you run ansible | 07:38 |
odyssey4me | if you're looking to surgically remove a host, I'm not entirely sure what the right way to do it is | 07:39 |
odyssey4me | are you working in a dev/test environment? is there a reason you're looking to surgically remove a particular host? | 07:39 |
odyssey4me | (or container) | 07:40 |
odyssey4me | mattt how does one remove one container reference in the inventory? is it a matter of just removing the host/container from (rpc|openstack)_hostnames_ips.yml ? | 07:41 |
bilal | odyssey4me: ya its a dev setup.. i dont want a seperate storage node. that is why removing its entries | 07:45 |
mattt | bilal: you can't just remove it tho | 07:52 |
mattt | because it will get re-added any time you run ansible | 07:52 |
mattt | bilal: do you have your swift.yml configured correctly? https://github.com/stackforge/os-ansible-deployment/blob/master/etc/openstack_deploy/conf.d/swift.yml#L17-L19 | 07:54 |
mattt | bilal: sounds like you've not configured it to put that service onto an existing host? | 07:54 |
mattt | odyssey4me: i would imagine so, i've not hacked up any of those files yet so can't say conclusively :) | 07:56 |
bilal | mattt: ya i havent configured it. I think ill just add the existing infra host ip in here? | 07:57 |
mattt | bilal: yep, and if you don't need swift you could comment out that file and just make sure you don't run any swift playbooks | 07:58 |
*** markvoelker has joined #openstack-ansible | 08:11 | |
*** markvoelker has quit IRC | 08:16 | |
*** daneyon has quit IRC | 08:26 | |
*** javeriak has quit IRC | 08:43 | |
*** javeriak has joined #openstack-ansible | 08:49 | |
*** javeriak has quit IRC | 08:50 | |
bilal | galera_container in two of my infra hosts are not changing state. which is causing infrastructure-setup playbook to error out. | 08:52 |
bilal | logs: TASK: [container_common | Restart container] ********************************** failed: [infra22_galera_container-525eeede] => {"err": "Failed to change the state of the container.", "failed": true, "rc": 2} msg: State failed to change for container [ infra22_galera_container-525eeede ] -- [ running ] != [ stopped ] failed: [infra333_galera_container-a0a71283] => {"err": "Failed to change the state of the container.", | 08:52 |
*** zz_avozza is now known as avozza | 08:55 | |
mattt | bilal: hmmmm | 09:10 |
*** markvoelker has joined #openstack-ansible | 09:12 | |
*** bilal has quit IRC | 09:15 | |
*** markvoelker has quit IRC | 09:17 | |
*** jreypo has joined #openstack-ansible | 09:19 | |
*** bilal has joined #openstack-ansible | 09:30 | |
bilal | mattt: sorry i lost the connection. Any help or suggestion from your side? | 09:31 |
mattt | bilal: sorry, not familiar w/ that error :( | 09:32 |
bilal | no worries :) let me see if i can find something | 09:35 |
odyssey4me | bilal it's just informing you that it was unable to switch the container between states - from stopped to started or started to stopped | 09:36 |
odyssey4me | check if the container is running, try stopping and starting is manually, etc | 09:36 |
odyssey4me | I assume that you are building on Ubuntu Trusty? | 09:36 |
bilal | stopping and starting manually is also giving error. yes its ubuntu trusty. if i destroy the container, will i have to run host-setup again to bring it back? or i can just re-run infrastructure-setup? | 09:38 |
mattt | bilal: you can have a poke under /var/log/lxc and see if anything is in the container's log file | 09:43 |
mattt | bilal: but yes you would have to re-run the host setup otherwise deploying the infrastructure bits will fail | 09:43 |
bilal | here is what container's log file is saying:: lxc-start 1428655266.167 ERROR lxc_conf - conf.c:mount_entry:2049 - No such file or directory - failed to mount '/openstack/infra22_galera_container-525eeede' on '/usr/lib/x86_64-linux-gnu/lxc/var/lib/mysql' lxc-start 1428655266.167 ERROR lxc_conf - conf.c:lxc_setup:4163 - failed to setup the mount entries for 'infra22_galera_container-525eeede' lxc-start 14286 | 09:45 |
odyssey4me | bilal if you've shifted architecture between plays you can expect a lot of brokenness... the system is designed to be implemented in a pretty specific way and deviating from that is something of an advanced topic... it may be easier to start from scratch again using your changed configs now that you're more aware of what you're wanting to achieve | 09:48 |
*** jreypo has quit IRC | 09:52 | |
svg | good morning all | 09:54 |
svg | I have a question (working on ceph support for v10) | 09:55 |
odyssey4me | odd - it appears that some of the openstack-infra instances have their nameserver set to 127.0.0.1, any thoughts on what may be doing the DNS resolution as a replacement? | 09:55 |
odyssey4me | o/ svg | 09:55 |
svg | a ceph rbd backend gets defined on cincer-volume - the storage_hosts in the environment | 09:56 |
odyssey4me | sure | 09:56 |
svg | now, once there are rbd backends defined on cinder, nova needs to get some configuration too, with info/vars coming from those cinder containers | 09:57 |
svg | I'm wondering what the best option would be to check if cinder has rbd support, then do the nova config om compute hosts | 09:58 |
svg | given those vars are not available to the compute hosts | 09:58 |
odyssey4me | we use standard openstack code, so if the driver is in-tree (which rbd is, afaik) then you have support for it | 09:58 |
svg | yes, but there is no need to do that configuration, if no cinder_volume hosts has any rbd backend | 09:59 |
odyssey4me | you'd have to setup rbd clients on every compute host and the appropriate ceph.conf to allow access (as far as I remember, operating from memory here and memory is from using this over 6 months ago) | 09:59 |
svg | and the needed vars to configure nova+compute are only available to cinder hosts | 09:59 |
svg | question is not on how to do it openstack wise, but how to implement this in oad ansible | 10:00 |
odyssey4me | well, you'd have to hack at it really... we will be looking at that, but only later this year | 10:01 |
mancdaz | svg you only want to drop in config if using rbd, else skip that part of the config? | 10:01 |
svg | given how cinder backends are definied in the rpc_user... file, those settings only exist on cinder hosts afaics | 10:01 |
svg | mancdaz: yes | 10:01 |
odyssey4me | I would expect you'd have to add some plays for any software deployments you'd need, you'd need to also edit the various configurations that are dropped | 10:01 |
odyssey4me | ah, if it's just the cinder conf you're concerned with then we already have multi-backend support so you'd just need to define the settings as a backend | 10:02 |
mancdaz | right | 10:02 |
odyssey4me | something like this: http://docs.rackspace.com/rpc/api/v10/bk-rpc-installation/content/sec_configure-cinder.html | 10:03 |
mancdaz | https://github.com/stackforge/os-ansible-deployment/blob/juno/rpc_deployment/roles/cinder_common/templates/cinder.conf#L52-L59 | 10:03 |
odyssey4me | but instead of the netapp backend settings, use the rbd backend settings from the openstack admin guide | 10:03 |
svg | yes yes :) I'm aware of that and got that covered | 10:04 |
svg | hold on | 10:04 |
*** avozza is now known as zz_avozza | 10:08 | |
svg | ok, so my point is, the variabale `cinder_backends` is only available for cinder_volumes_containers | 10:09 |
svg | but I need information from cinder, to configure nova | 10:09 |
svg | what would be the appropriate way to make them available to nova | 10:10 |
*** markvoelker has joined #openstack-ansible | 10:13 | |
*** markvoelker has quit IRC | 10:17 | |
svg | odyssey4me: mancdaz ^^ | 10:18 |
odyssey4me | svg I'm stuck in the middle of other work right now and can't fairly commit time to figure that out. Sorry - it's not something I can reply to off the top of my head. | 10:18 |
svg | ok, no prob | 10:19 |
openstackgerrit | Matt Thompson proposed stackforge/os-ansible-deployment: Ensure OpenStack commands are run as correct user https://review.openstack.org/172368 | 10:30 |
mancdaz | svg you could add a block to the nova.conf to parse the cinder_backends from the user_vars, and pull out the info you need | 10:47 |
mancdaz | https://github.com/stackforge/os-ansible-deployment/blob/juno/rpc_deployment/roles/cinder_common/templates/cinder.conf#L48 | 10:47 |
svg | mancdaz: but cinder_backends is only defined on cinder_hosts, not compute_hosts | 10:49 |
mancdaz | and probably add the cinder_backends info to the compute host definition in rpc_user_config | 10:49 |
svg | hm, so that would be duplicating the same info | 10:49 |
mattt | svg: vars not available to compute hosts? | 10:58 |
mattt | svg: cinder_backends should be available to everything no? | 10:59 |
svg | mattt: you can verify that in the rpc_inventory.json, cinder_backend is only defined on cinder hosts | 10:59 |
mattt | svg: ah i see, hmmm | 11:01 |
mattt | svg: you can lookup vars by group tho | 11:01 |
svg | by group? didn't think so, only by host | 11:02 |
mattt | svg: my ansible isn't the greatest, gimme a sec while i look at some stuff | 11:02 |
mattt | svg: i thought there was a cleaner way, but i did find this in our roles: hostvars[groups['heat_api'][0]]['stack_user_domain_id'] | 11:04 |
svg | yes, but that won't work, as possibly ceph is not defined as a backend on all cinder hosts :) | 11:05 |
svg | I'm working on a loop structure now, to find that out. | 11:05 |
mattt | k :) | 11:07 |
*** markvoelker has joined #openstack-ansible | 11:13 | |
*** markvoelker has quit IRC | 11:18 | |
*** mnestheu1 has joined #openstack-ansible | 12:02 | |
*** markvoelker has joined #openstack-ansible | 12:11 | |
*** ishant has quit IRC | 12:26 | |
*** sdake has joined #openstack-ansible | 12:31 | |
*** sdake_ has joined #openstack-ansible | 12:32 | |
*** sdake has quit IRC | 12:36 | |
*** markvoelker has quit IRC | 12:43 | |
*** markvoelker has joined #openstack-ansible | 12:44 | |
*** sdake_ has quit IRC | 12:47 | |
*** zz_avozza is now known as avozza | 13:04 | |
cloudnull | Morning | 13:22 |
odyssey4me | morning cloudnull | 13:22 |
cloudnull | How his it? | 13:22 |
cloudnull | *how's it | 13:23 |
odyssey4me | not too bad - the galera cache change patch has behaved, so it's in the merge queue now | 13:24 |
odyssey4me | once that's done we can rebase all the other patches that are approved already | 13:24 |
*** avozza is now known as zz_avozza | 13:25 | |
cloudnull | Nice. | 13:27 |
cloudnull | With that , In testing , my p1.8 hosts were using next to no swap. So that should improve the overall gating experience. | 13:28 |
odyssey4me | it figures that the database would be a big RAM consumer | 13:29 |
cloudnull | Yea and with the stock settings it could use 12+gbs of ram | 13:30 |
odyssey4me | it'll make https://review.openstack.org/171992 a little unnecessary, but perhaps that's a good way to head off other issues | 13:30 |
cloudnull | I think that should go in anyway. | 13:30 |
cloudnull | Itll hedge all bets. | 13:31 |
odyssey4me | with https://review.openstack.org/172096 I've been trying to figure out what's causing hpcloud-b4 to always fail on the container apt-get updates | 13:31 |
odyssey4me | but the builds haven't hit hpcloud-b4 yet.... always other regions | 13:31 |
odyssey4me | I did find that hpcloud appears to have some other DNS setup going as the stock image seems to have DNS set to 127.0.0.1 | 13:32 |
odyssey4me | when the containers get built, if they're using the host resolv.conf that'll obviously cause failures... but it doesn't tell us why hpcloud-b4 is the only one with issues | 13:33 |
odyssey4me | I've started thinking that perhaps instead of replacing resolv.conf as we do in the AIO bootstrap, perhaps we should append the google DNS to whatever's there? | 13:33 |
openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: Add AIO diagnostics and change DNS configuration https://review.openstack.org/172096 | 13:45 |
openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: Add AIO diagnostics and change DNS configuration https://review.openstack.org/172096 | 13:47 |
openstackgerrit | Matthew Kassawara proposed stackforge/os-ansible-deployment: Remove deprecated use_namespaces option https://review.openstack.org/172068 | 13:52 |
cloudnull | odyssey4me that sounds sensible. | 13:55 |
cloudnull | adding the nameserver will be a better solution overall and be less invasive. | 13:56 |
*** JRobinson__ has quit IRC | 13:58 | |
*** KLevenstein has joined #openstack-ansible | 14:02 | |
openstackgerrit | Merged stackforge/os-ansible-deployment: Updated galera cache size when boostrapping AIO https://review.openstack.org/172285 | 14:02 |
*** Mudpuppy has joined #openstack-ansible | 14:24 | |
*** jwagner_away is now known as jwagner | 14:33 | |
*** sdake has joined #openstack-ansible | 14:46 | |
*** sdake_ has joined #openstack-ansible | 14:49 | |
*** sdake has quit IRC | 14:50 | |
*** Mudpuppy has quit IRC | 14:52 | |
*** Mudpuppy has joined #openstack-ansible | 14:56 | |
*** sdake has joined #openstack-ansible | 14:58 | |
cloudnull | cores please review https://review.openstack.org/#/c/172243/ | 15:01 |
*** sdake_ has quit IRC | 15:02 | |
*** sacharya has joined #openstack-ansible | 15:33 | |
openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: Remove deprecated use_namespaces option https://review.openstack.org/172068 | 15:38 |
openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: Update Keystone config and policy for Kilo https://review.openstack.org/167475 | 15:38 |
openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: Harden Keystone's Apache config https://review.openstack.org/168523 | 15:38 |
openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: Genericize how we update SSL settings for Apache https://review.openstack.org/171838 | 15:38 |
openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: Allow lxc_container settings per container type https://review.openstack.org/172019 | 15:38 |
openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: Set the container_fs_size to 12G for Glance https://review.openstack.org/172020 | 15:38 |
*** ccrouch has left #openstack-ansible | 15:41 | |
*** ccrouch has joined #openstack-ansible | 15:42 | |
cloudnull | go odyssey4me go =) | 16:04 |
openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: Adjust AIO swap size to 4G/8G, depending on RAM amount https://review.openstack.org/171992 | 16:06 |
odyssey4me | alright, I think everything that was queued up has now been rebased on the galera patch and is back into the queue | 16:07 |
odyssey4me | if anything was +w'd, I've ensured that the +w is back | 16:07 |
odyssey4me | we're hogging the openstack-infra queue :) | 16:08 |
cloudnull | hopefully all the things gate now :) | 16:11 |
odyssey4me | yep, I'll leave you to it - the fire's going, so it's almost time to get the meat on the grill :) | 16:12 |
odyssey4me | have a great weekend! | 16:12 |
cloudnull | you have a great weekend too ! | 16:15 |
cloudnull | thanks for everything you do brother ! | 16:15 |
*** sdake_ has joined #openstack-ansible | 16:16 | |
*** zz_avozza is now known as avozza | 16:17 | |
*** sdake has quit IRC | 16:20 | |
openstackgerrit | Andy McCrae proposed stackforge/os-ansible-deployment: Remove md5sum check for environment vs user_config https://review.openstack.org/171987 | 16:23 |
*** daneyon has joined #openstack-ansible | 16:24 | |
*** daneyon_ has joined #openstack-ansible | 16:25 | |
*** daneyon has quit IRC | 16:29 | |
openstackgerrit | Kevin Carter proposed stackforge/os-ansible-deployment: Updated gate scripts for better consistency https://review.openstack.org/172505 | 16:50 |
*** jwagner is now known as jwagner_away | 17:04 | |
*** KLevenstein has quit IRC | 17:14 | |
cloudnull | cores can we get some reviews on https://review.openstack.org/#/c/171992/ | 17:33 |
openstackgerrit | Merged stackforge/os-ansible-deployment: Flake8 update - openstack-wheel-builder.py https://review.openstack.org/171041 | 17:37 |
stevelle | How do we feel about locking down the upstream repos to rc1 as is becomes available per project? | 17:46 |
openstackgerrit | Merged stackforge/os-ansible-deployment: Update Keystone config and policy for Kilo https://review.openstack.org/167475 | 17:48 |
*** javeriak has joined #openstack-ansible | 17:50 | |
openstackgerrit | Merged stackforge/os-ansible-deployment: Harden Keystone's Apache config https://review.openstack.org/168523 | 17:51 |
openstackgerrit | Merged stackforge/os-ansible-deployment: Flake8 update - dynamic_inventory.py https://review.openstack.org/171036 | 17:51 |
*** javeriak has quit IRC | 18:02 | |
*** bilal has quit IRC | 18:07 | |
*** KLevenstein has joined #openstack-ansible | 18:07 | |
openstackgerrit | Merged stackforge/os-ansible-deployment: Genericize how we update SSL settings for Apache https://review.openstack.org/171838 | 18:13 |
*** bilal has joined #openstack-ansible | 18:19 | |
*** daneyon has joined #openstack-ansible | 18:24 | |
*** javeriak has joined #openstack-ansible | 18:25 | |
*** daneyon_ has quit IRC | 18:26 | |
bilal | i have two of each container in my infra hosts. e.g. infra22_galera_container-35b6e44d and infra22_galera_container-525eeede and these are also in infra3 and infra1. is it normal? | 18:48 |
bilal | this is rackspace 10 setup | 18:48 |
*** sacharya has quit IRC | 19:01 | |
*** Mudpuppy_ has joined #openstack-ansible | 19:06 | |
*** Mudpuppy has quit IRC | 19:09 | |
*** Mudpuppy_ has quit IRC | 19:10 | |
openstackgerrit | Merged stackforge/os-ansible-deployment: Updated for 10.1.3 tag https://review.openstack.org/172243 | 19:22 |
mattt | bilal: that is not normal | 20:14 |
*** javeriak has quit IRC | 20:24 | |
mattt | bilal: did you figure it out? | 20:44 |
*** mnestheu1 has quit IRC | 21:20 | |
*** britthouser has quit IRC | 21:23 | |
*** KLevenstein has quit IRC | 21:31 | |
*** stevemar has joined #openstack-ansible | 21:48 | |
*** javeriak has joined #openstack-ansible | 22:04 | |
*** stevemar has quit IRC | 22:07 | |
bilal | hey mattt: ya i redid from scratch now.. deleted all containers, cleared the cache, everything going smooth now :D | 22:26 |
*** markvoelker has quit IRC | 22:40 | |
*** avozza is now known as zz_avozza | 23:09 | |
*** zz_avozza is now known as avozza | 23:10 | |
*** avozza is now known as zz_avozza | 23:29 | |
*** Mudpuppy has joined #openstack-ansible | 23:33 | |
*** markvoelker has joined #openstack-ansible | 23:39 | |
*** zz_avozza is now known as avozza | 23:41 | |
*** Mudpuppy has quit IRC | 23:50 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!