Friday, 2015-04-10

openstackgerritKevin Carter proposed stackforge/os-ansible-deployment: Updated for 10.1.3 tag  https://review.openstack.org/17224300:07
*** stevelle has quit IRC01:45
*** stevelle has joined #openstack-ansible01:46
*** javeriak has quit IRC01:58
openstackgerritKevin Carter proposed stackforge/os-ansible-deployment: Updated galera cache size when boostrapping AIO  https://review.openstack.org/17228502:24
*** sdake has joined #openstack-ansible02:26
*** sdake_ has joined #openstack-ansible02:54
*** sdake has quit IRC02:57
cloudnullSam-I-Am odyssey4me mattt mancdaz have a gander at - https://review.openstack.org/172285 - seems that tuning galera down for use in the AIO make the gate job complete faster/successfully.03:26
cloudnullwill recheck a couple of times just for the sake of testing but the results [ http://logs.openstack.org/85/172285/1/check/os-ansible-deployment-dsvm-check-commit/ac397c2/logs/instance-info/host_info_1428633111.log ] look good.03:27
*** JRobinson__ is now known as JRobinson__afk03:35
*** markvoelker has quit IRC03:35
*** sacharya has joined #openstack-ansible03:43
*** markvoelker has joined #openstack-ansible04:06
*** stevemar has joined #openstack-ansible04:07
*** markvoelker has quit IRC04:12
*** sdake_ has quit IRC04:27
*** sacharya has quit IRC04:34
*** javeriak has joined #openstack-ansible04:51
*** javeriak has quit IRC04:53
*** JRobinson__afk is now known as JRobinson__05:11
*** sdake has joined #openstack-ansible05:14
*** sdake has quit IRC05:14
*** sdake has joined #openstack-ansible05:14
*** sdake_ has joined #openstack-ansible05:25
*** sdake has quit IRC05:29
*** stevemar has quit IRC05:32
*** sdake_ has quit IRC05:55
*** markvoelker has joined #openstack-ansible06:09
*** markvoelker has quit IRC06:14
*** ishant has joined #openstack-ansible06:25
*** ishant has quit IRC06:34
*** bilal has joined #openstack-ansible06:38
*** ishant has joined #openstack-ansible06:40
bilalDuring rackspace 10 deployment when i run host-setup playbook; some of the containers are failing because they are stale entries from my old configuration. how can i remove them?06:41
bilalhow do i clear the cache when i run the host-setup playbook?06:44
*** ishant has quit IRC06:44
*** ishant has joined #openstack-ansible06:44
*** markvoelker has joined #openstack-ansible07:10
*** javeriak has joined #openstack-ansible07:10
*** markvoelker has quit IRC07:15
matttbilal: you effectively want to do a complete re-install?  is that correct?07:16
bilalyes matt07:17
matttbilal: the ansible inventory script will write a few files to /etc/rpc_deploy, you'll want to clean those up07:18
bilalokay.. and delete all the previous containers as well?07:19
matttbilal: i think the files in question are rpc_hostnames_ips.yml and rpc_inventory.json, but just compare to what is there and what was initially copied over (https://github.com/stackforge/os-ansible-deployment/tree/juno/etc/rpc_deploy)07:20
matttbilal: removing containers should do it for the most part, keep in mind that some services are not containerised though (nova-compute being one example)07:21
openstackgerritJesse Pretorius proposed stackforge/os-ansible-deployment: Adjust AIO swap size to half of RAM, up to 8GB  https://review.openstack.org/17199207:23
bilalokay. let me try that. Thanks mattt :D07:26
matttbilal: yeah just be careful that you don't delete those files on an operational cluster07:27
matttbilal: give us a shout if you have any other questions07:28
openstackgerritJesse Pretorius proposed stackforge/os-ansible-deployment: Adjust AIO swap size to 4G/8G, depending on RAM amount  https://review.openstack.org/17199207:28
openstackgerritJesse Pretorius proposed stackforge/os-ansible-deployment: [WIP] Diagnostic information for gate checks  https://review.openstack.org/17209607:30
bilalone more question:  i have no entry for aio1_swift_proxy_container-96f0ffc3 in my rpc_user_config.yml.. but the host-setup playbook always try to ssh into it and gives error07:33
odyssey4mebilal that'll be inside your inventory somewhere, so take a look in /etc/rpc_deploy with grep -r and you should find it07:34
bilalodyssey4me: ya it is in rpc_inventory.json file.. if i remove this entry, it wont cause any issue right?07:38
odyssey4methat file is dynamically generated whenever you run ansible07:38
odyssey4meif you're looking to surgically remove a host, I'm not entirely sure what the right way to do it is07:39
odyssey4meare you working in a dev/test environment? is there a reason you're looking to surgically remove a particular host?07:39
odyssey4me(or container)07:40
odyssey4memattt how does one remove one container reference in the inventory? is it a matter of just removing the host/container from (rpc|openstack)_hostnames_ips.yml ?07:41
bilalodyssey4me: ya its a dev setup.. i dont want a seperate storage node. that is why removing its entries07:45
matttbilal: you can't just remove it tho07:52
matttbecause it will get re-added any time you run ansible07:52
matttbilal: do you have your swift.yml configured correctly?  https://github.com/stackforge/os-ansible-deployment/blob/master/etc/openstack_deploy/conf.d/swift.yml#L17-L1907:54
matttbilal: sounds like you've not configured it to put that service onto an existing host?07:54
matttodyssey4me: i would imagine so, i've not hacked up any of those files yet so can't say conclusively :)07:56
bilalmattt: ya i havent configured it.  I think ill just add the existing infra host ip in here?07:57
matttbilal: yep, and if you don't need swift you could comment out that file and just make sure you don't run any swift playbooks07:58
*** markvoelker has joined #openstack-ansible08:11
*** markvoelker has quit IRC08:16
*** daneyon has quit IRC08:26
*** javeriak has quit IRC08:43
*** javeriak has joined #openstack-ansible08:49
*** javeriak has quit IRC08:50
bilalgalera_container in two of my infra hosts are not changing state. which is causing infrastructure-setup playbook to error out.08:52
bilallogs: TASK: [container_common | Restart container] **********************************  failed: [infra22_galera_container-525eeede] => {"err": "Failed to change the state of the container.", "failed": true, "rc": 2} msg: State failed to change for container [ infra22_galera_container-525eeede ] -- [ running ] != [ stopped ] failed: [infra333_galera_container-a0a71283] => {"err": "Failed to change the state of the container.",08:52
*** zz_avozza is now known as avozza08:55
matttbilal: hmmmm09:10
*** markvoelker has joined #openstack-ansible09:12
*** bilal has quit IRC09:15
*** markvoelker has quit IRC09:17
*** jreypo has joined #openstack-ansible09:19
*** bilal has joined #openstack-ansible09:30
bilalmattt: sorry i lost the connection. Any help or suggestion from your side?09:31
matttbilal: sorry, not familiar w/ that error :(09:32
bilalno worries :) let me see if i can find something09:35
odyssey4mebilal it's just informing you that it was unable to switch the container between states - from stopped to started or started to stopped09:36
odyssey4mecheck if the container is running, try stopping and starting is manually, etc09:36
odyssey4meI assume that you are building on Ubuntu Trusty?09:36
bilalstopping and starting manually is also giving error. yes its ubuntu trusty. if i destroy the container, will i have to run host-setup again to bring it back? or i can just re-run infrastructure-setup?09:38
matttbilal: you can have a poke under /var/log/lxc and see if anything is in the container's log file09:43
matttbilal: but yes you would have to re-run the host setup otherwise deploying the infrastructure bits will fail09:43
bilalhere is what container's log file is saying::  lxc-start 1428655266.167 ERROR    lxc_conf - conf.c:mount_entry:2049 - No such file or directory - failed to mount '/openstack/infra22_galera_container-525eeede' on '/usr/lib/x86_64-linux-gnu/lxc/var/lib/mysql'       lxc-start 1428655266.167 ERROR    lxc_conf - conf.c:lxc_setup:4163 - failed to setup the mount entries for 'infra22_galera_container-525eeede'       lxc-start 1428609:45
odyssey4mebilal if you've shifted architecture between plays you can expect a lot of brokenness... the system is designed to be implemented in a pretty specific way and deviating from that is something of an advanced topic... it may be easier to start from scratch again using your changed configs now that you're more aware of what you're wanting to achieve09:48
*** jreypo has quit IRC09:52
svggood morning all09:54
svgI have a question (working on ceph support for v10)09:55
odyssey4meodd - it appears that some of the openstack-infra instances have their nameserver set to 127.0.0.1, any thoughts on what may be doing the DNS resolution as a replacement?09:55
odyssey4meo/ svg09:55
svga ceph rbd backend gets defined on cincer-volume - the storage_hosts in the environment09:56
odyssey4mesure09:56
svgnow, once there are rbd backends defined on cinder, nova needs to get some configuration too, with info/vars coming from those cinder containers09:57
svgI'm wondering what the best option would be to check if cinder has rbd support, then do the nova config om compute hosts09:58
svggiven those vars are not available to the compute hosts09:58
odyssey4mewe use standard openstack code, so if the driver is in-tree (which rbd is, afaik) then you have support for it09:58
svgyes, but there is no need to do that configuration, if no cinder_volume hosts has any rbd backend09:59
odyssey4meyou'd have to setup rbd clients on every compute host and the appropriate ceph.conf to allow access (as far as I remember, operating from memory here and memory is from using this over 6 months ago)09:59
svgand the needed vars to configure nova+compute are only available to cinder hosts09:59
svgquestion is not on how to do it openstack wise, but how to implement this in oad ansible10:00
odyssey4mewell, you'd have to hack at it really... we will be looking at that, but only later this year10:01
mancdazsvg you only want to drop in config if using rbd, else skip that part of the config?10:01
svggiven how cinder backends are definied in the rpc_user... file, those settings only exist on cinder hosts afaics10:01
svgmancdaz: yes10:01
odyssey4meI would expect you'd have to add some plays for any software deployments you'd need, you'd need to also edit the various configurations that are dropped10:01
odyssey4meah, if it's just the cinder conf you're concerned with then we already have multi-backend support so you'd just need to define the settings as a backend10:02
mancdazright10:02
odyssey4mesomething like this: http://docs.rackspace.com/rpc/api/v10/bk-rpc-installation/content/sec_configure-cinder.html10:03
mancdazhttps://github.com/stackforge/os-ansible-deployment/blob/juno/rpc_deployment/roles/cinder_common/templates/cinder.conf#L52-L5910:03
odyssey4mebut instead of the netapp backend settings, use the rbd backend settings from the openstack admin guide10:03
svgyes yes :) I'm aware of that and got that covered10:04
svghold on10:04
*** avozza is now known as zz_avozza10:08
svgok, so my point is, the variabale `cinder_backends` is only available for cinder_volumes_containers10:09
svgbut I need information from cinder, to configure nova10:09
svgwhat would be the appropriate way to make them available to nova10:10
*** markvoelker has joined #openstack-ansible10:13
*** markvoelker has quit IRC10:17
svgodyssey4me: mancdaz ^^10:18
odyssey4mesvg I'm stuck in the middle of other work right now and can't fairly commit time to figure that out. Sorry - it's not something I can reply to off the top of my head.10:18
svgok, no prob10:19
openstackgerritMatt Thompson proposed stackforge/os-ansible-deployment: Ensure OpenStack commands are run as correct user  https://review.openstack.org/17236810:30
mancdazsvg you could add a block to the nova.conf to parse the cinder_backends from the user_vars, and pull out the info you need10:47
mancdazhttps://github.com/stackforge/os-ansible-deployment/blob/juno/rpc_deployment/roles/cinder_common/templates/cinder.conf#L4810:47
svgmancdaz: but cinder_backends is only defined on cinder_hosts, not compute_hosts10:49
mancdazand probably add the cinder_backends info to the compute host definition in rpc_user_config10:49
svghm, so that would be duplicating the same info10:49
matttsvg: vars not available to compute hosts?10:58
matttsvg: cinder_backends should be available to everything no?10:59
svgmattt: you can verify that in the rpc_inventory.json, cinder_backend is only defined on cinder hosts10:59
matttsvg: ah i see, hmmm11:01
matttsvg: you can lookup vars by group tho11:01
svgby group? didn't think so, only by host11:02
matttsvg: my ansible isn't the greatest, gimme a sec while i look at some stuff11:02
matttsvg: i thought there was a cleaner way, but i did find this in our roles: hostvars[groups['heat_api'][0]]['stack_user_domain_id']11:04
svgyes, but that won't work, as possibly ceph is not defined as a backend on all cinder hosts :)11:05
svgI'm working on a loop structure now, to find that out.11:05
matttk :)11:07
*** markvoelker has joined #openstack-ansible11:13
*** markvoelker has quit IRC11:18
*** mnestheu1 has joined #openstack-ansible12:02
*** markvoelker has joined #openstack-ansible12:11
*** ishant has quit IRC12:26
*** sdake has joined #openstack-ansible12:31
*** sdake_ has joined #openstack-ansible12:32
*** sdake has quit IRC12:36
*** markvoelker has quit IRC12:43
*** markvoelker has joined #openstack-ansible12:44
*** sdake_ has quit IRC12:47
*** zz_avozza is now known as avozza13:04
cloudnullMorning13:22
odyssey4memorning cloudnull13:22
cloudnullHow his it?13:22
cloudnull*how's it13:23
odyssey4menot too bad - the galera cache change patch has behaved, so it's in the merge queue now13:24
odyssey4meonce that's done we can rebase all the other patches that are approved already13:24
*** avozza is now known as zz_avozza13:25
cloudnullNice.13:27
cloudnullWith that , In testing , my p1.8 hosts were using next to no swap. So that should improve the overall gating experience.13:28
odyssey4meit figures that the database would be a big RAM consumer13:29
cloudnullYea and with the stock settings it could use 12+gbs of ram13:30
odyssey4meit'll make https://review.openstack.org/171992 a little unnecessary, but perhaps that's a good way to head off other issues13:30
cloudnullI think that should go in anyway.13:30
cloudnullItll hedge all bets.13:31
odyssey4mewith https://review.openstack.org/172096 I've been trying to figure out what's causing hpcloud-b4 to always fail on the container apt-get updates13:31
odyssey4mebut the builds haven't hit hpcloud-b4 yet.... always other regions13:31
odyssey4meI did find that hpcloud appears to have some other DNS setup going as the stock image seems to have DNS set to 127.0.0.113:32
odyssey4mewhen the containers get built, if they're using the host resolv.conf that'll obviously cause failures... but it doesn't tell us why hpcloud-b4 is the only one with issues13:33
odyssey4meI've started thinking that perhaps instead of replacing resolv.conf as we do in the AIO bootstrap, perhaps we should append the google DNS to whatever's there?13:33
openstackgerritJesse Pretorius proposed stackforge/os-ansible-deployment: Add AIO diagnostics and change DNS configuration  https://review.openstack.org/17209613:45
openstackgerritJesse Pretorius proposed stackforge/os-ansible-deployment: Add AIO diagnostics and change DNS configuration  https://review.openstack.org/17209613:47
openstackgerritMatthew Kassawara proposed stackforge/os-ansible-deployment: Remove deprecated use_namespaces option  https://review.openstack.org/17206813:52
cloudnullodyssey4me that sounds sensible.13:55
cloudnulladding the nameserver will be a better solution overall and be less invasive.13:56
*** JRobinson__ has quit IRC13:58
*** KLevenstein has joined #openstack-ansible14:02
openstackgerritMerged stackforge/os-ansible-deployment: Updated galera cache size when boostrapping AIO  https://review.openstack.org/17228514:02
*** Mudpuppy has joined #openstack-ansible14:24
*** jwagner_away is now known as jwagner14:33
*** sdake has joined #openstack-ansible14:46
*** sdake_ has joined #openstack-ansible14:49
*** sdake has quit IRC14:50
*** Mudpuppy has quit IRC14:52
*** Mudpuppy has joined #openstack-ansible14:56
*** sdake has joined #openstack-ansible14:58
cloudnullcores please review https://review.openstack.org/#/c/172243/15:01
*** sdake_ has quit IRC15:02
*** sacharya has joined #openstack-ansible15:33
openstackgerritJesse Pretorius proposed stackforge/os-ansible-deployment: Remove deprecated use_namespaces option  https://review.openstack.org/17206815:38
openstackgerritJesse Pretorius proposed stackforge/os-ansible-deployment: Update Keystone config and policy for Kilo  https://review.openstack.org/16747515:38
openstackgerritJesse Pretorius proposed stackforge/os-ansible-deployment: Harden Keystone's Apache config  https://review.openstack.org/16852315:38
openstackgerritJesse Pretorius proposed stackforge/os-ansible-deployment: Genericize how we update SSL settings for Apache  https://review.openstack.org/17183815:38
openstackgerritJesse Pretorius proposed stackforge/os-ansible-deployment: Allow lxc_container settings per container type  https://review.openstack.org/17201915:38
openstackgerritJesse Pretorius proposed stackforge/os-ansible-deployment: Set the container_fs_size to 12G for Glance  https://review.openstack.org/17202015:38
*** ccrouch has left #openstack-ansible15:41
*** ccrouch has joined #openstack-ansible15:42
cloudnullgo odyssey4me  go =)16:04
openstackgerritJesse Pretorius proposed stackforge/os-ansible-deployment: Adjust AIO swap size to 4G/8G, depending on RAM amount  https://review.openstack.org/17199216:06
odyssey4mealright, I think everything that was queued up has now been rebased on the galera patch and is back into the queue16:07
odyssey4meif anything was +w'd, I've ensured that the +w is back16:07
odyssey4mewe're hogging the openstack-infra queue :)16:08
cloudnullhopefully all the things gate now :)16:11
odyssey4meyep, I'll leave you to it - the fire's going, so it's almost time to get the meat on the grill :)16:12
odyssey4mehave a great weekend!16:12
cloudnullyou have a great weekend too !16:15
cloudnullthanks for everything you do brother !16:15
*** sdake_ has joined #openstack-ansible16:16
*** zz_avozza is now known as avozza16:17
*** sdake has quit IRC16:20
openstackgerritAndy McCrae proposed stackforge/os-ansible-deployment: Remove md5sum check for environment vs user_config  https://review.openstack.org/17198716:23
*** daneyon has joined #openstack-ansible16:24
*** daneyon_ has joined #openstack-ansible16:25
*** daneyon has quit IRC16:29
openstackgerritKevin Carter proposed stackforge/os-ansible-deployment: Updated gate scripts for better consistency  https://review.openstack.org/17250516:50
*** jwagner is now known as jwagner_away17:04
*** KLevenstein has quit IRC17:14
cloudnullcores can we get some reviews on https://review.openstack.org/#/c/171992/17:33
openstackgerritMerged stackforge/os-ansible-deployment: Flake8 update - openstack-wheel-builder.py  https://review.openstack.org/17104117:37
stevelleHow do we feel about locking down the upstream repos to rc1 as is becomes available per project?17:46
openstackgerritMerged stackforge/os-ansible-deployment: Update Keystone config and policy for Kilo  https://review.openstack.org/16747517:48
*** javeriak has joined #openstack-ansible17:50
openstackgerritMerged stackforge/os-ansible-deployment: Harden Keystone's Apache config  https://review.openstack.org/16852317:51
openstackgerritMerged stackforge/os-ansible-deployment: Flake8 update - dynamic_inventory.py  https://review.openstack.org/17103617:51
*** javeriak has quit IRC18:02
*** bilal has quit IRC18:07
*** KLevenstein has joined #openstack-ansible18:07
openstackgerritMerged stackforge/os-ansible-deployment: Genericize how we update SSL settings for Apache  https://review.openstack.org/17183818:13
*** bilal has joined #openstack-ansible18:19
*** daneyon has joined #openstack-ansible18:24
*** javeriak has joined #openstack-ansible18:25
*** daneyon_ has quit IRC18:26
bilali have two of each container in my infra hosts. e.g. infra22_galera_container-35b6e44d and infra22_galera_container-525eeede and these are also in infra3 and infra1. is it normal?18:48
bilalthis is rackspace 10 setup18:48
*** sacharya has quit IRC19:01
*** Mudpuppy_ has joined #openstack-ansible19:06
*** Mudpuppy has quit IRC19:09
*** Mudpuppy_ has quit IRC19:10
openstackgerritMerged stackforge/os-ansible-deployment: Updated for 10.1.3 tag  https://review.openstack.org/17224319:22
matttbilal: that is not normal20:14
*** javeriak has quit IRC20:24
matttbilal: did you figure it out?20:44
*** mnestheu1 has quit IRC21:20
*** britthouser has quit IRC21:23
*** KLevenstein has quit IRC21:31
*** stevemar has joined #openstack-ansible21:48
*** javeriak has joined #openstack-ansible22:04
*** stevemar has quit IRC22:07
bilalhey mattt: ya i redid from scratch now.. deleted all containers, cleared the cache, everything going smooth now :D22:26
*** markvoelker has quit IRC22:40
*** avozza is now known as zz_avozza23:09
*** zz_avozza is now known as avozza23:10
*** avozza is now known as zz_avozza23:29
*** Mudpuppy has joined #openstack-ansible23:33
*** markvoelker has joined #openstack-ansible23:39
*** zz_avozza is now known as avozza23:41
*** Mudpuppy has quit IRC23:50

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!