| *** erictian has joined #openstack-chef | 00:02 | |
| *** erictian_ has joined #openstack-chef | 00:02 | |
| *** erictian has quit IRC | 00:02 | |
| *** erictian has joined #openstack-chef | 00:03 | |
| *** erictian_ has quit IRC | 00:03 | |
| *** erictian_ has joined #openstack-chef | 00:04 | |
| *** erictian has quit IRC | 00:07 | |
| *** erictian_ has quit IRC | 00:09 | |
| *** erictian has joined #openstack-chef | 00:34 | |
| *** otter768 has joined #openstack-chef | 01:12 | |
| *** otter768 has quit IRC | 01:17 | |
| xmltok | yeah its weird, it wasnt a problem before either | 01:48 |
|---|---|---|
| *** erictian has quit IRC | 02:22 | |
| *** erictian has joined #openstack-chef | 02:42 | |
| *** kobtea has joined #openstack-chef | 03:13 | |
| *** otter768 has joined #openstack-chef | 03:13 | |
| *** otter768 has quit IRC | 03:18 | |
| *** kobtea has quit IRC | 03:18 | |
| *** galstrom_zzz is now known as galstrom | 04:18 | |
| *** otter768 has joined #openstack-chef | 04:26 | |
| *** galstrom is now known as galstrom_zzz | 04:27 | |
| *** erictian has quit IRC | 04:29 | |
| *** erictian has joined #openstack-chef | 04:29 | |
| *** erictian has quit IRC | 04:34 | |
| *** GonZo2K has joined #openstack-chef | 04:34 | |
| *** galstrom_zzz is now known as galstrom | 04:48 | |
| *** erictian has joined #openstack-chef | 04:57 | |
| *** erictian_ has joined #openstack-chef | 05:15 | |
| *** erictian has quit IRC | 05:15 | |
| *** GonZo2K has quit IRC | 05:40 | |
| *** GonZo2K has joined #openstack-chef | 05:41 | |
| *** galstrom is now known as galstrom_zzz | 05:50 | |
| *** GonZo2K has quit IRC | 06:41 | |
| *** opscode-logger has quit IRC | 07:00 | |
| *** opscode-logger has joined #openstack-chef | 07:00 | |
| *** os-chef-bot has quit IRC | 07:12 | |
| *** os-chef-bot has joined #openstack-chef | 07:13 | |
| *** otter768 has quit IRC | 07:31 | |
| *** kobtea has joined #openstack-chef | 08:40 | |
| *** erictian_ has quit IRC | 08:42 | |
| *** erictian has joined #openstack-chef | 08:43 | |
| *** kobtea has quit IRC | 08:44 | |
| *** erictian has quit IRC | 08:47 | |
| *** otter768 has joined #openstack-chef | 09:32 | |
| *** otter768 has quit IRC | 09:36 | |
| *** erictian has joined #openstack-chef | 09:43 | |
| *** erictian has quit IRC | 09:48 | |
| *** erictian has joined #openstack-chef | 10:44 | |
| *** erictian has quit IRC | 10:53 | |
| *** otter768 has joined #openstack-chef | 11:33 | |
| *** otter768 has quit IRC | 11:37 | |
| *** erictian has joined #openstack-chef | 11:48 | |
| *** erictian has quit IRC | 11:54 | |
| *** otter768 has joined #openstack-chef | 13:34 | |
| *** erictian has joined #openstack-chef | 13:37 | |
| *** otter768 has quit IRC | 13:38 | |
| *** erictian has quit IRC | 13:42 | |
| *** GonZo2K has joined #openstack-chef | 14:04 | |
| *** erictian has joined #openstack-chef | 15:26 | |
| *** erictian has quit IRC | 15:31 | |
| *** otter768 has joined #openstack-chef | 15:35 | |
| *** otter768 has quit IRC | 15:40 | |
| xmltok | it was only the metadata that needed to be redefined, the others were just kicked off from delayed restarts. i can get around the problem if i unwind the template in my wrapper and define it again | 16:01 |
| j^2 | Nice! Any chance you can push these up to us? | 16:02 |
| *** galstrom_zzz is now known as galstrom | 16:04 | |
| *** erictian has joined #openstack-chef | 16:19 | |
| *** erictian has quit IRC | 16:30 | |
| *** erictian has joined #openstack-chef | 16:30 | |
| *** erictian has quit IRC | 16:35 | |
| *** ehaselwanter has joined #openstack-chef | 16:36 | |
| *** erictian has joined #openstack-chef | 16:44 | |
| *** dotplus has quit IRC | 17:03 | |
| *** dotplus has joined #openstack-chef | 17:05 | |
| xmltok | yeah i still need to work out my account | 17:12 |
| xmltok | also we should really add some neutron-netns-cleanup stuff into the network cookbook, i think i just finally managed to make that thing work | 17:13 |
| xmltok | my metadata would stop working because the netns would be polluted with old ips and routes | 17:13 |
| xmltok | a simple cron job of neutron-netns-cleanup --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/dhcp_agent.ini is probably all that is necessary, but we may want to just run it with chef. i dont konw how often it should be run, seems like pretty often | 17:14 |
| *** galstrom is now known as galstrom_zzz | 17:14 | |
| xmltok | i still can't save contact information on review.openstack.org | 17:23 |
| j^2 | xmltok: is it the same error? | 17:26 |
| xmltok | ah my email address in launchpad wasnt the one with the openstack foundation | 17:26 |
| xmltok | To ssh://kraig@review.openstack.org:29418/stackforge/cookbook-openstack-network | 17:26 |
| xmltok | ! [remote rejected] master -> master (prohibited by Gerrit) | 17:26 |
| xmltok | im getting closer | 17:26 |
| j^2 | :D | 17:27 |
| *** erictian has quit IRC | 17:28 | |
| *** erictian has joined #openstack-chef | 17:29 | |
| xmltok | ok, bug is filed | 17:32 |
| *** erictian has quit IRC | 17:33 | |
| j^2 | D | 17:33 |
| j^2 | :D | 17:33 |
| j^2 | link? | 17:33 |
| j^2 | i’m gonna get os-chef-bot to start posting the bugs when people file em | 17:34 |
| j^2 | but it seems it’s been more challanging then i’d like | 17:34 |
| *** galstrom_zzz is now known as galstrom | 17:34 | |
| openstackgerrit | Kraig Amador proposed stackforge/cookbook-openstack-network: Install the metadata agent before attempting to restart it Move the metadata_agent.ini template definition to after the package installation so that the immediate restart notification will not error. https://review.openstack.org/137971 | 17:35 |
| *** otter768 has joined #openstack-chef | 17:36 | |
| xmltok | dude, i did it | 17:36 |
| xmltok | https://review.openstack.org/137971 | 17:36 |
| xmltok | months in the making.. | 17:36 |
| j^2 | yay! | 17:38 |
| j^2 | now you have no excuses :P | 17:38 |
| xmltok | its true | 17:40 |
| xmltok | https://bugs.launchpad.net/openstack-chef/+bug/1397702 :) | 17:40 |
| *** otter768 has quit IRC | 17:40 | |
| j^2 | :D | 17:41 |
| j^2 | sooo | 17:41 |
| j^2 | neutron-netns-cleanup | 17:41 |
| j^2 | what does that actually do? | 17:41 |
| xmltok | it rolls through the netns space on the box and deletes interfaces that are no longer used by dhcp/metadata agents | 17:43 |
| *** ehaselwanter has quit IRC | 17:43 | |
| xmltok | if you do 'ip netns list' youll see all your namespaces, if you do 'ip netns exec <namespace> ifconfig' you'll see the interfaces | 17:43 |
| xmltok | over time, you may end up with a bunch of dead ips in that list | 17:43 |
| xmltok | there is routes in there too, so you can end up with VMs trying to connect to the metadata agent on an IP that is no longer being used | 17:44 |
| j^2 | oh how | 17:45 |
| j^2 | wow i mean | 17:45 |
| j^2 | yeah that seems like it should be done pretty soon here | 17:45 |
| xmltok | yeah big deal, been a huge pain in the ass for me for a while, i finally figured out wtf was going on with that this morning | 17:45 |
| j^2 | can you take ownership of this? or do you want to hand it off to some one else? | 17:46 |
| xmltok | my only concern are the parameters, i think those are consistent with OVS installations though? | 17:46 |
| xmltok | i also dont know if we want to shove that into cron or what, i think cron is probably right | 17:47 |
| xmltok | treat it like the keystone cleanup | 17:47 |
| j^2 | yeah there are sain defaults with it irrc | 17:47 |
| j^2 | iirc | 17:47 |
| xmltok | # ls -la /etc/cron.d/neutron-dhcp-agent-netns-cleanup | 17:48 |
| xmltok | -rw-r--r-- 1 root root 319 Jun 16 01:03 /etc/cron.d/neutron-dhcp-agent-netns-cleanup | 17:48 |
| xmltok | hmmm | 17:48 |
| xmltok | thats weird | 17:48 |
| xmltok | there is a neutron-l3-agent-netns-cleanup one too, we just need one for ml2 i guess | 17:48 |
| xmltok | those come from the ubuntu packages, maybe this should be solved in the metadata package | 17:49 |
| xmltok | i cant find anything that runs the keystone cleanup either | 17:50 |
| xmltok | yeah, we have our own token_flush and nova-manage db archive_deleted_rows cron scripts | 17:51 |
| xmltok | it looks like there may be a lot of cron stuff missing | 17:51 |
| j^2 | ouch that’s a problem | 17:52 |
| xmltok | 0 3 * * 1,4 root source ~/openrc && /usr/bin/keystone-manage token_flush && /usr/bin/nova-manage db archive_deleted_rows --max_rows $MAX_ROWS | 17:52 |
| xmltok | i got this | 17:52 |
| j^2 | maybe open another bug report to track what you think we’re missing? | 17:52 |
| j^2 | :) | 17:52 |
| xmltok | done | 17:55 |
| j^2 | :D | 17:55 |
| xmltok | this is going pretty good, im up to the dashboard now but running into apache LockFile problems from being on the old cookbook | 17:58 |
| xmltok | im not really sure how that is working in my current environment | 17:58 |
| xmltok | i guess the 9.1.0 version of the cookbook was ok to work with apache 2.0.0 | 18:00 |
| j^2 | yeah i think we’re gonna have to bump everything to 2.0.0 for apache but that requires a massive rewrite of our specs | 18:00 |
| xmltok | https://github.com/stackforge/cookbook-openstack-dashboard/commit/5878de34d9a70f8fed2147c3d61e434bb5e64388 | 18:00 |
| xmltok | ahh | 18:01 |
| j^2 | i thought jklare fixed that…. | 18:01 |
| j^2 | hmm | 18:01 |
| j^2 | yeah the < is supposed to be the other way > | 18:02 |
| j^2 | the alligator was supposed to eat the bigger one ;) | 18:02 |
| xmltok | hehe | 18:03 |
| xmltok | open a bug report :) | 18:03 |
| j^2 | ha! | 18:06 |
| j^2 | yaeh i think he has…. | 18:06 |
| j^2 | or was going to | 18:06 |
| j^2 | aw man, this is gross | 18:12 |
| j^2 | man i can’t wait till jenkins is doing all this shit i’m doing | 18:21 |
| *** erictian has joined #openstack-chef | 18:29 | |
| *** galstrom is now known as galstrom_zzz | 18:29 | |
| *** galstrom_zzz is now known as galstrom | 18:36 | |
| xmltok | are you running all of the tests manually? | 18:36 |
| *** erictian has quit IRC | 18:38 | |
| xmltok | woohoo, changing to <= made my dashboard work, almost | 18:43 |
| j^2 | which tests? | 18:44 |
| j^2 | xmltok: nice! | 18:44 |
| xmltok | whatever you want jenkins to do | 18:45 |
| *** galstrom is now known as galstrom_zzz | 18:48 | |
| xmltok | i think even if we get that netns cleanup script running its not going to clean up correctly, im not having it work unless i do a --force which deletes the interface entirely | 19:18 |
| j^2 | xmltok: ah, well the testing framework will be run via jenkins eventually | 19:24 |
| j^2 | but i have to do it by hand till we have a “baseline" | 19:25 |
| j^2 | then i’ll make jenkins build/run it | 19:25 |
| j^2 | xmltok: that seems dangerous | 19:25 |
| xmltok | yeah force sucks | 19:26 |
| xmltok | you need to restart the services after you do it to make it recreate everything | 19:26 |
| j^2 | ewww | 19:26 |
| j^2 | yeah as a cleanup that seems heavy handed | 19:26 |
| j^2 | “nuke everything and restart everything! oh and lets do this randomly so my tables don’t get to big" | 19:27 |
| j^2 | prod envs are gonna love that one | 19:27 |
| xmltok | it may be my bad | 19:28 |
| xmltok | default['openstack']['network']['dhcp']['dhcp_delete_namespaces'] = 'True' | 19:28 |
| xmltok | default['openstack']['network']['l3']['router_delete_namespaces'] = 'True' | 19:28 |
| xmltok | they were false | 19:28 |
| j^2 | ah | 19:30 |
| alanmeadows1 | Without force, the netns cleanup script really is only helpful for cleaning up *empty* namespaces that happen to be left dangling that can be removed | 19:33 |
| alanmeadows1 | which is safe in production | 19:33 |
| xmltok | it doesnt fix it, there is all kinds of bugs opened around this stuff too, i give up | 19:34 |
| xmltok | force deleted all of my namespaces, many of them were active | 19:34 |
| alanmeadows1 | Yes, rather invasive force is :) | 19:35 |
| alanmeadows1 | Also if you're really looking for a "start from square one" you probably need to rope in ovs-cleanup too which is equally destructive | 19:35 |
| alanmeadows1 | to purge all the openvswitch artifacts | 19:35 |
| *** otter768 has joined #openstack-chef | 19:36 | |
| alanmeadows1 | (like netns cleanup force requiring a l3 agen restart, ovs-cleanup would require a plugin restart) | 19:36 |
| *** otter768 has quit IRC | 19:41 | |
| *** erictian has joined #openstack-chef | 19:48 | |
| openstackgerrit | JJ Asghar proposed stackforge/cookbook-openstack-network: Centos 6.5 and up has and old version of dnsmasq https://review.openstack.org/137973 | 19:49 |
| j^2 | well that was ^^^^ three days of work…. | 19:50 |
| xmltok | i hate old rhel versions, 7.0 is barely tolerable | 19:51 |
| xmltok | there must be millions of lost man hours due to their old versions of everything | 19:51 |
| j^2 | yep | 19:52 |
| j^2 | and i think we’re not offically supporting centos7 till the stable/juno cut? I vaguely remember that some place | 19:52 |
| j^2 | oh jeebus, we don’t have descs in the Rakefile | 19:54 |
| j^2 | what the hell man | 19:54 |
| j^2 | yak shave yak shave yak shave | 19:55 |
| *** dmowrer_ has joined #openstack-chef | 19:55 | |
| *** dmowrer has joined #openstack-chef | 19:57 | |
| *** dmowrer_ has quit IRC | 19:57 | |
| j^2 | ahhh!!! | 19:57 |
| j^2 | http://i.imgur.com/zHohELu.png | 20:03 |
| xmltok | haha | 20:07 |
| openstackgerrit | JJ Asghar proposed stackforge/cookbook-openstack-network: Centos 6.5 and up has and old version of dnsmasq https://review.openstack.org/137973 | 20:11 |
| openstackgerrit | JJ Asghar proposed stackforge/cookbook-openstack-network: This adds descriptions to the rake tasks https://review.openstack.org/137975 | 20:16 |
| *** erictian has quit IRC | 20:21 | |
| *** erictian has joined #openstack-chef | 20:22 | |
| *** erictian has quit IRC | 20:30 | |
| *** erictian has joined #openstack-chef | 20:31 | |
| *** erictian has quit IRC | 20:37 | |
| *** dmowrer has quit IRC | 20:56 | |
| *** dmowrer has joined #openstack-chef | 21:15 | |
| *** ehaselwanter has joined #openstack-chef | 21:33 | |
| *** otter768 has joined #openstack-chef | 21:37 | |
| *** otter768 has quit IRC | 21:42 | |
| *** ehaselwanter has quit IRC | 22:49 | |
| *** otter768 has joined #openstack-chef | 23:38 | |
| *** otter768 has quit IRC | 23:43 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
