| *** spotz_zzz is now known as spotz | 00:00 | |
| *** thorst has quit IRC | 00:02 | |
| openstackgerrit | Neill Cox proposed openstack/openstack-ansible-ironic: Add tests for the ironic REST API https://review.openstack.org/298654 | 00:24 |
|---|---|---|
| *** thorst has joined #openstack-ansible | 00:25 | |
| *** johnmilton has joined #openstack-ansible | 00:25 | |
| *** schwicht has joined #openstack-ansible | 00:29 | |
| *** asettle has quit IRC | 00:32 | |
| *** schwicht_ has joined #openstack-ansible | 00:34 | |
| *** schwicht has quit IRC | 00:35 | |
| *** busterswt has quit IRC | 00:44 | |
| *** thorst has quit IRC | 00:45 | |
| *** thorst has joined #openstack-ansible | 00:46 | |
| *** keedya has quit IRC | 00:46 | |
| *** saneax is now known as saneax_AFK | 00:50 | |
| *** busterswt has joined #openstack-ansible | 00:52 | |
| *** thorst has quit IRC | 00:54 | |
| *** iceyao has joined #openstack-ansible | 01:03 | |
| *** rramos has quit IRC | 01:09 | |
| *** iceyao_ has joined #openstack-ansible | 01:09 | |
| *** iceyao has quit IRC | 01:13 | |
| *** iceyao_ has quit IRC | 01:13 | |
| *** iceyao has joined #openstack-ansible | 01:13 | |
| *** iceyao has quit IRC | 01:18 | |
| *** iceyao has joined #openstack-ansible | 01:18 | |
| *** asettle has joined #openstack-ansible | 01:22 | |
| *** keedya has joined #openstack-ansible | 01:24 | |
| *** asettle has quit IRC | 01:28 | |
| *** sdake has quit IRC | 01:28 | |
| *** sdake has joined #openstack-ansible | 01:32 | |
| *** busterswt has quit IRC | 01:35 | |
| *** weezS has joined #openstack-ansible | 01:39 | |
| *** coolj has quit IRC | 01:43 | |
| *** coolj has joined #openstack-ansible | 01:43 | |
| *** thorst has joined #openstack-ansible | 01:51 | |
| *** busterswt has joined #openstack-ansible | 01:54 | |
| *** thorst has quit IRC | 01:59 | |
| *** weezS has quit IRC | 02:11 | |
| *** sdake_ has joined #openstack-ansible | 02:20 | |
| *** sdake has quit IRC | 02:22 | |
| *** asettle has joined #openstack-ansible | 02:29 | |
| *** deadnull has quit IRC | 02:35 | |
| *** retreved has quit IRC | 02:46 | |
| *** spotz is now known as spotz_zzz | 02:54 | |
| *** thorst has joined #openstack-ansible | 02:56 | |
| *** openstackstatus has quit IRC | 03:01 | |
| *** thorst has quit IRC | 03:04 | |
| *** sdake_ has quit IRC | 03:06 | |
| *** sdake has joined #openstack-ansible | 03:07 | |
| *** iceyao has quit IRC | 03:09 | |
| *** iceyao has joined #openstack-ansible | 03:09 | |
| *** sdake has quit IRC | 03:17 | |
| *** sdake has joined #openstack-ansible | 03:18 | |
| *** v1k0d3n has joined #openstack-ansible | 03:19 | |
| *** joker_ has quit IRC | 03:20 | |
| *** joker_ has joined #openstack-ansible | 03:20 | |
| *** prometheanfire has quit IRC | 03:22 | |
| *** prometheanfire has joined #openstack-ansible | 03:23 | |
| *** iceyao has quit IRC | 03:23 | |
| *** iceyao has joined #openstack-ansible | 03:23 | |
| *** jmccrory has quit IRC | 03:26 | |
| *** jmccrory has joined #openstack-ansible | 03:26 | |
| *** sdake has quit IRC | 03:28 | |
| *** iceyao has quit IRC | 03:31 | |
| *** iceyao has joined #openstack-ansible | 03:32 | |
| *** keedya has quit IRC | 03:47 | |
| *** schwicht_ has quit IRC | 03:48 | |
| *** v1k0d3n has quit IRC | 03:51 | |
| *** sekrit has quit IRC | 03:51 | |
| *** johnmilton has quit IRC | 03:54 | |
| *** thorst has joined #openstack-ansible | 04:01 | |
| *** rramos has joined #openstack-ansible | 04:02 | |
| *** busterswt has quit IRC | 04:02 | |
| *** sekrit has joined #openstack-ansible | 04:05 | |
| *** thorst has quit IRC | 04:09 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_tempest: Fix broken installation task https://review.openstack.org/301533 | 04:11 |
| *** asettle has quit IRC | 04:17 | |
| *** asettle has joined #openstack-ansible | 04:18 | |
| *** rramos has quit IRC | 04:24 | |
| *** javeriak has joined #openstack-ansible | 04:25 | |
| *** persia has quit IRC | 04:31 | |
| *** persia has joined #openstack-ansible | 04:33 | |
| *** rramos has joined #openstack-ansible | 04:37 | |
| *** jayc has joined #openstack-ansible | 04:44 | |
| *** shausy has joined #openstack-ansible | 04:55 | |
| *** admin0 has joined #openstack-ansible | 04:56 | |
| *** nhadie has joined #openstack-ansible | 04:58 | |
| *** rramos has quit IRC | 05:01 | |
| *** asettle has quit IRC | 05:02 | |
| *** asettle has joined #openstack-ansible | 05:03 | |
| *** nhadzter has joined #openstack-ansible | 05:03 | |
| *** nhadie has quit IRC | 05:06 | |
| *** thorst has joined #openstack-ansible | 05:06 | |
| *** admin0 has quit IRC | 05:08 | |
| *** nhadzter has quit IRC | 05:09 | |
| *** javeriak has quit IRC | 05:12 | |
| *** thorst has quit IRC | 05:13 | |
| *** nhadzter has joined #openstack-ansible | 05:27 | |
| *** shausy has quit IRC | 05:35 | |
| *** shausy has joined #openstack-ansible | 05:36 | |
| *** iceyao has quit IRC | 05:38 | |
| *** iceyao has joined #openstack-ansible | 05:38 | |
| *** iceyao has quit IRC | 05:39 | |
| *** iceyao has joined #openstack-ansible | 05:39 | |
| *** joker_ has quit IRC | 05:39 | |
| *** javeriak has joined #openstack-ansible | 05:39 | |
| *** iceyao has quit IRC | 05:40 | |
| *** iceyao_ has joined #openstack-ansible | 05:40 | |
| *** admin0 has joined #openstack-ansible | 05:43 | |
| *** zhangjn has quit IRC | 05:45 | |
| *** zhangjn has joined #openstack-ansible | 05:46 | |
| openstackgerrit | Sashi Dahal proposed openstack/openstack-ansible: ceph configuration for glance and nova https://review.openstack.org/301922 | 05:46 |
| *** admin0 has quit IRC | 05:51 | |
| *** saneax_AFK is now known as saneax | 05:57 | |
| *** nhadzter has quit IRC | 06:09 | |
| *** mikelk has joined #openstack-ansible | 06:10 | |
| *** thorst has joined #openstack-ansible | 06:11 | |
| *** thorst has quit IRC | 06:18 | |
| *** asettle has quit IRC | 06:20 | |
| *** pcaruana has joined #openstack-ansible | 06:26 | |
| *** asettle has joined #openstack-ansible | 06:33 | |
| *** asettle has quit IRC | 06:37 | |
| *** unlaudable has joined #openstack-ansible | 06:44 | |
| *** unlaudable has quit IRC | 06:44 | |
| openstackgerrit | Matt Thompson proposed openstack/openstack-ansible-os_nova: [WIP] Use tempest for testing https://review.openstack.org/301566 | 06:46 |
| *** javeriak has quit IRC | 06:56 | |
| *** neilus has joined #openstack-ansible | 06:57 | |
| *** neilus has quit IRC | 07:01 | |
| *** nhadzter has joined #openstack-ansible | 07:04 | |
| *** admin0 has joined #openstack-ansible | 07:10 | |
| *** admin0 has quit IRC | 07:11 | |
| *** javeriak has joined #openstack-ansible | 07:14 | |
| *** thorst has joined #openstack-ansible | 07:16 | |
| *** admin0 has joined #openstack-ansible | 07:19 | |
| *** neilus has joined #openstack-ansible | 07:19 | |
| *** thorst has quit IRC | 07:24 | |
| *** neilus has quit IRC | 07:25 | |
| *** jayc has quit IRC | 07:36 | |
| *** Oku_OS-away is now known as Oku_OS | 07:38 | |
| *** asettle has joined #openstack-ansible | 07:42 | |
| *** jamielennox is now known as jamielennox|away | 07:45 | |
| *** asettle has quit IRC | 07:45 | |
| *** wangqwsh has joined #openstack-ansible | 07:56 | |
| admin0 | morning all | 07:57 |
| *** neilus has joined #openstack-ansible | 08:13 | |
| *** admin0 has quit IRC | 08:17 | |
| *** admin0 has joined #openstack-ansible | 08:19 | |
| *** thorst has joined #openstack-ansible | 08:21 | |
| wangqwsh | morning | 08:26 |
| *** openstackstatus has joined #openstack-ansible | 08:28 | |
| *** ChanServ sets mode: +v openstackstatus | 08:28 | |
| *** thorst has quit IRC | 08:30 | |
| -openstackstatus- NOTICE: jobs depending on npm are now working again | 08:31 | |
| *** iceyao_ has quit IRC | 08:49 | |
| *** iceyao has joined #openstack-ansible | 08:50 | |
| *** iceyao has quit IRC | 08:51 | |
| *** iceyao has joined #openstack-ansible | 08:51 | |
| *** iceyao has quit IRC | 08:53 | |
| *** iceyao has joined #openstack-ansible | 08:53 | |
| *** iceyao has quit IRC | 08:58 | |
| *** iceyao_ has joined #openstack-ansible | 08:59 | |
| *** javeriak has quit IRC | 09:05 | |
| *** javeriak has joined #openstack-ansible | 09:22 | |
| *** shausy has quit IRC | 09:24 | |
| *** nhadzter has quit IRC | 09:24 | |
| odyssey4me | o/ | 09:25 |
| *** iceyao has joined #openstack-ansible | 09:26 | |
| odyssey4me | coolj yes, you can theme horizon using the dict horizon_custom_uploads where you supply the source/dest files: https://github.com/openstack/openstack-ansible-os_horizon/blob/master/tasks/horizon_post_install.yml#L51-L57 | 09:27 |
| *** iceyao_ has quit IRC | 09:27 | |
| *** thorst has joined #openstack-ansible | 09:27 | |
| odyssey4me | we should add some sort of sample to the defaults, and add some documentation in http://docs.openstack.org/developer/openstack-ansible/install-guide/configure-horizon.html for that <--- mhayden :) | 09:28 |
| *** deadnull has joined #openstack-ansible | 09:29 | |
| *** ChanServ changes topic to "Mitaka Release Critical Bugs: https://goo.gl/ipD9BL || Launchpad: https://launchpad.net/openstack-ansible || Weekly Meetings: https://wiki.openstack.org/wiki/Meetings/openstack-ansible || Review Dashboard: https://goo.gl/tTmdgs" | 09:30 | |
| deadnull | my initial setup failed, and my galera cluster wont come up, i tried running openstack-ansible galera-install.yml --tags galera-bootstrap but setup-infra is still failing, is my best/oply option to destroy containers and start over? | 09:30 |
| *** ChanServ changes topic to "Launchpad: https://launchpad.net/openstack-ansible || Weekly Meetings: https://wiki.openstack.org/wiki/Meetings/openstack-ansible || Review Dashboard: https://goo.gl/tTmdgs" | 09:30 | |
| odyssey4me | deadnull why's it failing? | 09:31 |
| deadnull | mysql is not running on one of the containers | 09:31 |
| odyssey4me | yeah, that shouldn't be an issue - it needs one running node to be able to work | 09:32 |
| odyssey4me | have you tried working through http://docs.openstack.org/developer/openstack-ansible/install-guide/ops-galera-recovery.html ? | 09:32 |
| deadnull | odyssey4me its failing on this task '[galera_server | Check for cluster state failure]' | 09:32 |
| deadnull | i tried the first step, i will run through that now, thanks | 09:34 |
| *** thorst has quit IRC | 09:34 | |
| odyssey4me | deadnull assuming this is a fresh install, try executing openstack-ansible -e 'galera_ignore_cluster_state=true' galera-install.yml | 09:34 |
| deadnull | yes, perfectly clean | 09:35 |
| deadnull | the original failure on this host was due to lxc running out of space | 09:35 |
| odyssey4me | that effectively ignores the cluster state and resets the cluster, which is not advised in production | 09:35 |
| deadnull | i have run the setup-openstack playbooks, but i dont have horizon, which caused me to backtrack and uncover the ctrl01 ran out of space and galera was not running | 09:36 |
| odyssey4me | hmm, that's odd - if horizon wasn't setup properly then it would seem that perhaps the right group name was used in openstack_user_config.yml | 09:36 |
| deadnull | i have horizon containers, just nothing is running on br-mgmt :443 | 09:37 |
| deadnull | so then i re-ran setup-openstack | 09:37 |
| odyssey4me | do you have the group 'os-infra_hosts' in openstack_user_config? | 09:38 |
| deadnull | yeah the CLI is working perfectly | 09:38 |
| deadnull | cinder is working, keystone is replying | 09:38 |
| odyssey4me | it's strange that horizon's tasks were skipped though | 09:38 |
| deadnull | i think its an odd situation with running out of space causing an issue, just odd the failures didnt show until the 2nd run | 09:39 |
| odyssey4me | very | 09:39 |
| deadnull | im going to run the ignore cluster state, then kick off setup-infra, then setup-openstack | 09:39 |
| deadnull | i'll be around all day, hopefully will get this squared away this AM | 09:39 |
| odyssey4me | one of the things we'd like to get included in this cycle is a better pre-flight check to determine whether everything that needs to be in place, is in place, before we touch anything | 09:39 |
| deadnull | thank you for the help so far.. I am documenting all the steps i am running across. basically writing a prereq file for my team | 09:39 |
| odyssey4me | admin0 with regards to what to consider 'stable', working with OpenStack is a little different to vendor provided software, so the rules applied needs to be a little different | 09:41 |
| odyssey4me | deadnull sure thing, happy to help :) | 09:41 |
| deadnull | yeah, that would be great (preflight check), i have been using ansible extensively in our environment outside of openstack, happy to help out. | 09:41 |
| deadnull | still ramping up on OSA now, so it might be a little bit ;-) | 09:42 |
| odyssey4me | admin0 OpenStack is developed and released in 6 month cycles, whereas vendors typically release major versions every 2-3 years depending on the type of software | 09:42 |
| odyssey4me | deadnull any contributions are welcome: bug reporting, bug triage, patches for docs or code - it's all good | 09:43 |
| odyssey4me | also helping people in the channel is also very welcome | 09:43 |
| odyssey4me | or the ML | 09:43 |
| openstackgerrit | git-harry proposed openstack/openstack-ansible: DEFCORE: Updated tempest config and resources https://review.openstack.org/302626 | 09:43 |
| deadnull | that play failed with a dpkg error on mariadb... should i blow away the container on my ctrl01 | 09:44 |
| odyssey4me | admin0 so the best advice for anyone adopting OpenStack is to start testing the new version immediately after release, to help report and resolve bugs to improve stabilisation with the intent of having a fully tested and stable implementation/upgrade path by the first milestone of the next cycle, then ideally to actually implement that into production by the second milestone | 09:45 |
| odyssey4me | deadnull yeah, sometimes it is easier to just blow away the containers - the data stays behind on the host so if you really don't care about the data at this point then clear out the /openstack/<container-name> directories on the host | 09:46 |
| odyssey4me | there's a playbook to destroy the containers, just use '--limit galera_all' to ensure that it only removes the ones you want to :) | 09:47 |
| odyssey4me | that playbook won't delete the folders on the host though | 09:47 |
| *** dsariel has joined #openstack-ansible | 09:48 | |
| *** jiteka has quit IRC | 09:51 | |
| *** javeriak has quit IRC | 09:53 | |
| hughsaunders | odyssey4me: https://goo.gl/03qZPi | 09:55 |
| *** iceyao has quit IRC | 09:55 | |
| deadnull | can i use the same -l galera_all when running setup_hosts speed that step up? | 09:55 |
| *** iceyao_ has joined #openstack-ansible | 09:55 | |
| *** iceyao_ has quit IRC | 09:56 | |
| *** iceyao has joined #openstack-ansible | 09:56 | |
| deadnull | openstack-ansible lxc-containers-create.yml --limit galera_all | 09:59 |
| deadnull | seems like it should do it | 09:59 |
| deadnull | kicking off setup-infra now, :crosses_fingers: | 10:01 |
| *** iceyao has quit IRC | 10:04 | |
| *** iceyao has joined #openstack-ansible | 10:04 | |
| *** iceyao has quit IRC | 10:06 | |
| *** iceyao has joined #openstack-ansible | 10:07 | |
| *** karimb has joined #openstack-ansible | 10:13 | |
| *** jiteka has joined #openstack-ansible | 10:22 | |
| deadnull | odyssey4me seems like I cant win today... rabbitmq container failed to cluster on that ctrl01 host | 10:24 |
| odyssey4me | deadnull yep | 10:24 |
| odyssey4me | deadnull do you have time sync setup on all the hosts? | 10:25 |
| *** iceyao has quit IRC | 10:25 | |
| *** iceyao has joined #openstack-ansible | 10:25 | |
| *** karimb has quit IRC | 10:26 | |
| deadnull | odyssey4me yes, 5 statrum 1 GPS clocks on every node, seems to be perfect | 10:26 |
| deadnull | in log: reporting "already member of cluster" | 10:27 |
| odyssey4me | hmm, ok - odd that it's having issues - perhaps the disk space has been a bugs issue all along | 10:27 |
| odyssey4me | is your root partition particularly small? | 10:28 |
| deadnull | root for lxc or root for the host | 10:28 |
| deadnull | 917G 16G 856G 2% /var/lib/lxc | 10:28 |
| deadnull | 28G 7.3G 19G 28% / | 10:28 |
| deadnull | seems odd that ctrl01 is having issues and nothing else is, of course the sysadmin who setup this host didnt use lvm :-\ | 10:29 |
| *** McMurlock1 has joined #openstack-ansible | 10:29 | |
| odyssey4me | ah, that's fun | 10:29 |
| openstackgerrit | Matt Thompson proposed openstack/openstack-ansible-os_nova: [WIP] Use tempest for testing https://review.openstack.org/301566 | 10:29 |
| odyssey4me | maybe the sysadmin should setup cobbler to automate the host setup and borrow some of the configs done in this gist: https://gist.github.com/cloudnull/f71e3078f9a0018017c3 | 10:30 |
| *** pjm6 has joined #openstack-ansible | 10:30 | |
| pjm6 | morning all :) | 10:30 |
| *** thorst has joined #openstack-ansible | 10:32 | |
| deadnull | do you think the 30GB / partition is causing the issues, or should i blow everything away not just galera and start over | 10:32 |
| *** pjm6 has quit IRC | 10:35 | |
| *** pjm6 has joined #openstack-ansible | 10:36 | |
| *** karimb has joined #openstack-ansible | 10:38 | |
| odyssey4me | deadnull well, perhaps - there are bind mounts from the containers to the hosts for logs and critical data (like the database contents) | 10:38 |
| odyssey4me | deadnull so you could perhaps seperate /openstack into its own partition too | 10:39 |
| deadnull | that is very do-able | 10:39 |
| *** thorst has quit IRC | 10:39 | |
| deadnull | i have another 1TB drive sitting in there, this is far from long-term production | 10:39 |
| deadnull | trying to avoid a complete OS re-install, but i might have to | 10:39 |
| deadnull | i'll start with /openstack relocation | 10:39 |
| odyssey4me | I don't think it'll need very much space, but you will want room to grow | 10:39 |
| deadnull | yeah the docs do state at least 100GB per controller host | 10:40 |
| deadnull | 30GB < 100GB :0 | 10:40 |
| deadnull | :) | 10:40 |
| odyssey4me | yep :) | 10:40 |
| odyssey4me | if your lxc vg is on a RAID disk, then perhaps carve out a section of the lxc vg for the /openstack partition | 10:41 |
| odyssey4me | the containers themselves will mostly only use 5G each (there are one or two exceptions), so not all that much is needed for the lxc vg - you have almost 100GB there | 10:42 |
| odyssey4me | maybe take 30GB of that for the /openstack partition and see how that goes | 10:42 |
| deadnull | yeah i'll give it a shot, once I stop the containers, and move the data, start with setup-hosts again I presume, then move to setup-infra > setup-openstack | 10:43 |
| odyssey4me | yep | 10:44 |
| *** pjm6 has quit IRC | 10:46 | |
| *** karimb has quit IRC | 10:53 | |
| *** karimb has joined #openstack-ansible | 10:54 | |
| *** pjm6 has joined #openstack-ansible | 10:57 | |
| *** karimb has quit IRC | 11:01 | |
| deadnull | odyssey4me thanks again, headed into the office, been a long day already... I will be back shortly (hopefully with good news) | 11:02 |
| deadnull | cheers | 11:03 |
| *** deadnull has quit IRC | 11:03 | |
| *** McMurlock1 has quit IRC | 11:03 | |
| *** johnmilton has joined #openstack-ansible | 11:08 | |
| Bofu2U | morn | 11:10 |
| *** thorst has joined #openstack-ansible | 11:11 | |
| winggundamth | if I using Ceph. Cinder Volumes service can deploy on 3 of infra host right? | 11:11 |
| *** thorst_ has joined #openstack-ansible | 11:11 | |
| *** pjm6 has quit IRC | 11:11 | |
| *** thorst has quit IRC | 11:15 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_tempest: Remove volume-feature-enabled/snapshot override https://review.openstack.org/302400 | 11:17 |
| *** iceyao has quit IRC | 11:18 | |
| *** iceyao has joined #openstack-ansible | 11:18 | |
| *** iceyao has quit IRC | 11:19 | |
| *** iceyao has joined #openstack-ansible | 11:19 | |
| *** deadnull has joined #openstack-ansible | 11:19 | |
| deadnull | aaaand I'm back | 11:20 |
| Bofu2U | deadnull: I've been awaiting your return anxiously. | 11:20 |
| deadnull | Bofu2U :-) | 11:20 |
| Bofu2U | ;) | 11:20 |
| *** admin0 has quit IRC | 11:26 | |
| *** john_bar has quit IRC | 11:29 | |
| *** admin0 has joined #openstack-ansible | 11:31 | |
| *** pjm6 has joined #openstack-ansible | 11:36 | |
| *** pjm6 has quit IRC | 11:40 | |
| openstackgerrit | Matt Thompson proposed openstack/openstack-ansible: Add tempest_log_dir variable https://review.openstack.org/302692 | 11:50 |
| *** Oku_OS is now known as Oku_OS-away | 11:52 | |
| *** keedya has joined #openstack-ansible | 11:55 | |
| *** weshay has joined #openstack-ansible | 11:56 | |
| *** javeriak has joined #openstack-ansible | 11:58 | |
| deadnull | ok, so setup-hosts went clean after the /openstack migration to a larger volume | 11:58 |
| deadnull | now kicking off setup-infrastructure | 11:58 |
| openstackgerrit | Matt Thompson proposed openstack/openstack-ansible-os_tempest: Updates for tempest in functional tests https://review.openstack.org/302697 | 11:58 |
| openstackgerrit | Matt Thompson proposed openstack/openstack-ansible-os_nova: [WIP] Use tempest for testing https://review.openstack.org/301566 | 12:00 |
| mhayden | mornin' | 12:06 |
| mhayden | odyssey4me: i need to get a look at that doc blueprint that slid my way | 12:06 |
| *** jhesketh has quit IRC | 12:07 | |
| *** jhesketh has joined #openstack-ansible | 12:07 | |
| *** admin0 has quit IRC | 12:16 | |
| deadnull | odyssey4me no dice, still failed on the same step with rabbitmq after relocating to a larger drive | 12:17 |
| *** javeriak has quit IRC | 12:17 | |
| deadnull | i guess my next step is to blow away all the containers, and then start from setup-hosts on a clean install? | 12:17 |
| odyssey4me | deadnull yeah, that's odd - perhaps a full teardown would be a good idea | 12:18 |
| odyssey4me | the initial work to get the hosts and networking done right is usually the most painful | 12:18 |
| deadnull | teardown using lxc-containers-destroy.yml or full taredown as in bare metal | 12:18 |
| odyssey4me | try just destroying the containers, and clearing out the directories that contain the logs/data in /openstack on each host | 12:19 |
| deadnull | sounds good, I will start there... im working on an ansible playbook to bring them up from metal, basically similar to the cobbler config you sent earlier | 12:20 |
| *** woodard has joined #openstack-ansible | 12:23 | |
| deadnull | ok logs clean and /openstack is clean, now going to give this a show starting with setup-hosts.yml | 12:24 |
| *** woodard has quit IRC | 12:24 | |
| *** woodard has joined #openstack-ansible | 12:25 | |
| *** admin0 has joined #openstack-ansible | 12:26 | |
| *** Oku_OS-away is now known as Oku_OS | 12:38 | |
| *** joker_ has joined #openstack-ansible | 12:42 | |
| *** admin0 has quit IRC | 12:43 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 12:47 | |
| *** iceyao has quit IRC | 12:48 | |
| *** v1k0d3n has joined #openstack-ansible | 12:53 | |
| *** automagically has joined #openstack-ansible | 12:55 | |
| *** keedya has quit IRC | 12:59 | |
| *** keedya has joined #openstack-ansible | 13:00 | |
| openstackgerrit | Merged openstack/openstack-ansible: DEFCORE: Updated tempest config and resources https://review.openstack.org/302626 | 13:04 |
| *** jayc has joined #openstack-ansible | 13:07 | |
| *** eric_lopez has quit IRC | 13:10 | |
| automagically | morning | 13:10 |
| deadnull | automagically morning! | 13:10 |
| *** eric_lopez has joined #openstack-ansible | 13:11 | |
| *** jayc has quit IRC | 13:13 | |
| *** jayc has joined #openstack-ansible | 13:13 | |
| *** javeriak has joined #openstack-ansible | 13:14 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_tempest: Allocation pool for public subnet https://review.openstack.org/300481 | 13:18 |
| *** jayc is now known as _55_ | 13:19 | |
| *** psilvad has joined #openstack-ansible | 13:20 | |
| cloudnull | morning | 13:22 |
| cloudnull | sysadmin: https://github.com/cloudnull/osa-multi-node-aio -- thats the updated version of the gist posted by odyssey4me earlier today. | 13:23 |
| javeriak | morning cloudnull, glad to have caught you | 13:25 |
| cloudnull | ohai :) | 13:25 |
| javeriak | having some problems with my horizon | 13:25 |
| cloudnull | whaats going on ? | 13:25 |
| javeriak | it wont load after the login page | 13:25 |
| javeriak | ive got tons of things deployed on it, >500 VMs etc, so might it be scale/load issue? | 13:26 |
| cloudnull | mitaka or liberty ? | 13:26 |
| javeriak | liberty | 13:27 |
| javeriak | 12.0.0 to be precise | 13:27 |
| cloudnull | does it work if you go to the URL https://$YOURIPHERE/identity/ | 13:27 |
| javeriak | 12.0.6** i mean | 13:27 |
| cloudnull | the default landing page takes you to project which makes a miserable sql query and that may be taking a very long time to load | 13:28 |
| *** psilvad_ has joined #openstack-ansible | 13:29 | |
| cloudnull | when hundreds of VMs exist under a single tenant that page load time will take forever. | 13:29 |
| cloudnull | idk if thats the issue | 13:29 |
| cloudnull | but its something i know happens | 13:29 |
| javeriak | hmm im trying the identity page, but thats taking forever too | 13:29 |
| cloudnull | the default landing page is typically https://$YOUIPHERE/project/ | 13:30 |
| cloudnull | is it in a loading state? | 13:30 |
| javeriak | aha the identity page worked! | 13:30 |
| cloudnull | cool | 13:30 |
| cloudnull | you should be able to navigate around. | 13:30 |
| cloudnull | just dont go to the overview page "/project" | 13:31 |
| javeriak | thanks as usual :) | 13:31 |
| javeriak | yea its mega slow, but seems to be working | 13:31 |
| cloudnull | i wonder if there's something we can do to make it faster. | 13:32 |
| openstackgerrit | Merged openstack/openstack-ansible-plugins: Add allocation pool support to neutron module https://review.openstack.org/300421 | 13:32 |
| cloudnull | neillc <- has worked on horizon in the past, maybe there's some black magic we can consume | 13:32 |
| javeriak | hmm yea that would help, this is on that osic cluster btw, and ive got of things deployed on it now, so the network page wont load either now | 13:32 |
| *** b3rnard0_away is now known as b3rnard0 | 13:33 | |
| javeriak | tons** of things deployed on it | 13:33 |
| cloudnull | nice! | 13:33 |
| cloudnull | is the cli performance good with that many things deployed ? | 13:34 |
| openstackgerrit | Merged openstack/openstack-ansible-openstack_hosts: Add a check for hosts particular kernel modules https://review.openstack.org/291797 | 13:34 |
| cloudnull | javeriak: also is access to the OSIC cluster something that you might be able to share with a few folks to see if we can poke at things for the purpose of optimizing? | 13:35 |
| javeriak | cloudnull yea ali is fine, thats what we have been using mostly to test with. yea we have it for another week, when would you guys want to look? | 13:37 |
| javeriak | cli** | 13:38 |
| *** KLevenstein has joined #openstack-ansible | 13:38 | |
| cloudnull | sure that would be awesome | 13:38 |
| cloudnull | there may be tweaks we can make to make things generally faster for horizon users. | 13:38 |
| cloudnull | its completely possible that horizon just sucks but maybe we can make it suck less. | 13:39 |
| deadnull | odyssey4me so good news so far, hosts and infra went through clean after blowing away the containers. | 13:39 |
| deadnull | it was def. just having a hard time realizing it was already joined into rabbitmq cluster, so it was timing out joining | 13:40 |
| cloudnull | bah. rabbit... | 13:40 |
| deadnull | my thoughts exactly... | 13:42 |
| deadnull | zmq has its own issues also | 13:42 |
| *** pjm6 has joined #openstack-ansible | 13:42 | |
| cloudnull | I've not given ZMQ a go in a while | 13:44 |
| cloudnull | in the icehouse and juno timeframe a few of us tried it . | 13:44 |
| cloudnull | sadly the oslo drivers were garbage and we went back to rabbit | 13:44 |
| cloudnull | IDK if theyve gotten better. | 13:44 |
| *** mgoddard_ has joined #openstack-ansible | 13:44 | |
| *** briancubed has joined #openstack-ansible | 13:46 | |
| *** mgoddard has quit IRC | 13:48 | |
| *** Oku_OS is now known as Oku_OS-away | 13:48 | |
| cloudnull | deadnull: have you messed with ZMQ recently? | 13:50 |
| deadnull | cloudnull I have used ZMQ but not with oslo, going to take a peek after i get this deployment online | 13:51 |
| cloudnull | cool | 13:51 |
| openstackgerrit | Merged openstack/openstack-ansible-os_heat: Fix heat trustee configuration https://review.openstack.org/302433 | 13:51 |
| cloudnull | id love to know how it goes. | 13:51 |
| deadnull | absolutely | 13:51 |
| cloudnull | if we can decentralize messaging that would be awesome | 13:51 |
| deadnull | i'll get some docs together, agreed.. that would be great | 13:51 |
| deadnull | might be mitaka timeframe | 13:52 |
| cloudnull | if i can help let me know. | 13:52 |
| javeriak | also im having some problems with rabbitmq keys on liberty, i took out the latest liberty tag (this is another setup btw); and playbooks fail with "Memcache key not found"; apparantly the rabbitmq.key file disappears from one of the infra container | 13:52 |
| deadnull | absolutely, will do | 13:52 |
| *** jthorne has joined #openstack-ansible | 13:53 | |
| briancubed | Hey, folks. I need some triage advice. Doing a 5-node virtual deployment using some new integration work from Nuage Networks. We have this working with AIO, so now we're trying multinode. | 13:53 |
| cloudnull | javeriak: thats likely an issue with rabbitmq not being able to sync ssl keys to all of the nodes in the cluster. | 13:53 |
| briancubed | I put a log snippet here: http://pastebin.com/GGFWyD67 | 13:53 |
| cloudnull | if you rerun the whole play does it fail: openstack-ansible rabbitmq-install.yml | 13:53 |
| *** iceyao has joined #openstack-ansible | 13:54 | |
| javeriak | cloudnull, yea it fails at the same 'Distribute self signed ssl' task everytime | 13:54 |
| cloudnull | briancubed: seems from your deploy node can not ssh to the containers on the other node. | 13:55 |
| cloudnull | are they one the same network ? | 13:55 |
| odyssey4me | javeriak I think we fixed that up for Mitaka - we use slurp instead of cat to pick up the keys and it's much more reliable | 13:55 |
| odyssey4me | we also found that we were putting the keys down badly | 13:55 |
| cloudnull | odyssey4me: maybe we need to backport that fix to liberty ? | 13:55 |
| briancubed | cloudnull: yes. a number of similar operations, waiting for ssh, succeed | 13:55 |
| briancubed | yes, they are on the same network, that is | 13:56 |
| cloudnull | do they all fail on the same host? | 13:56 |
| javeriak | odyssey4me there was a similar problem before as well with the nova and keystone keys i remember, i used to fix it by combing the distribute and other playbook because the key wasnt registering across | 13:56 |
| *** ametts has joined #openstack-ansible | 13:56 | |
| briancubed | the snippet I pasted was from a re-run of the setup_hosts playbook | 13:56 |
| odyssey4me | javeriak it was https://github.com/openstack/openstack-ansible-rabbitmq_server/commit/bc411c677a4aef36817d990f8849d8d8d8befea3 and https://github.com/openstack/openstack-ansible-rabbitmq_server/commit/c9773b9d9c85dbec0422839829a9dedbd07991d0 that fixed it up | 13:56 |
| odyssey4me | cloudnull we've had very few reports of the problem from liberty deployers | 13:57 |
| odyssey4me | javeriak you may find that you just need to clear the fact cache | 13:57 |
| *** busterswt has joined #openstack-ansible | 13:57 | |
| odyssey4me | that often helps | 13:57 |
| javeriak | odyssey4me okay i'll try clearing the fact cahce first and then use those fixes | 13:57 |
| javeriak | thanks guys, let me see if that helps | 13:57 |
| odyssey4me | javeriak if those fixes work, then please propose a combined patch into liberty if that code's not already there | 13:58 |
| openstackgerrit | Matt Thompson proposed openstack/openstack-ansible-security: [WIP] Doc updates https://review.openstack.org/302776 | 13:58 |
| javeriak | odyssey4me sure will do | 13:58 |
| briancubed | i searched online and saw a post where odyssey4me suggested increasing the ssh timeout using ssh_delay in user_variables.yml. I set it to several values, as high as 20 (assuming minutes), but it doesn't seem to have any effect. | 13:58 |
| cloudnull | looks like liberty has this one already https://github.com/openstack/openstack-ansible-rabbitmq_server/commit/bc411c677a4aef36817d990f8849d8d8d8befea3 | 13:59 |
| *** admin0 has joined #openstack-ansible | 14:00 | |
| cloudnull | and it could use https://github.com/openstack/openstack-ansible-rabbitmq_server/commit/c9773b9d9c85dbec0422839829a9dedbd07991d0 to resolve the memcached issue, assuming its worth backporting at this point. | 14:00 |
| cloudnull | briancubed: is the ip address conflicting with something else on your network ? | 14:00 |
| *** Oku_OS-away is now known as Oku_OS | 14:00 | |
| cloudnull | the failed ones that is. | 14:01 |
| cloudnull | if you login to the host where the container is is running, can you attach to it, and from within it can you ping other devices on your network ? | 14:01 |
| briancubed | cloudnull: I don't think so. These nodes are VMs. I have them connected via a private network. Just to check, I tried pinging the problematic addresses when the playbook wasn't running. No response. | 14:02 |
| cloudnull | are they vms w/ KVM ? | 14:03 |
| briancubed | KVM, yes | 14:03 |
| cloudnull | or something like a public cloud provider ? | 14:03 |
| cloudnull | i know KVM works i test using the following repo https://github.com/cloudnull/osa-multi-node-aio | 14:03 |
| briancubed | KVM running on a bare-metal server under Ubuntu | 14:04 |
| *** mgoddard has joined #openstack-ansible | 14:04 | |
| *** mgoddard_ has quit IRC | 14:04 | |
| javeriak | odyssey4me the code in the rabbitmq_ssl_key_distribute.yml file is quite different between liberty and mitaka; https://github.com/openstack/openstack-ansible/blob/liberty/playbooks/roles/rabbitmq_server/tasks/rabbitmq_ssl_key_distribute.yml & https://github.com/openstack/openstack-ansible-rabbitmq_server/blob/c9773b9d9c85dbec0422839829a9dedbd07991d0/tasks/rabbitmq_ssl_key_distribute.yml | 14:04 |
| briancubed | Im sure it works. I'm trying to figure out what I've done incorrectly. | 14:04 |
| cloudnull | briancubed: hum... | 14:04 |
| cloudnull | bonded nics in the vms? | 14:04 |
| cloudnull | maybe a STP issue? | 14:05 |
| *** cloudtrainme has joined #openstack-ansible | 14:05 | |
| cloudnull | check the kernel log | 14:05 |
| cloudnull | if sg offloading is on it may be causing a stack trace w/ KVM | 14:06 |
| *** automagically has quit IRC | 14:06 | |
| briancubed | i'm not using bonded nics. I setup bridges to eth0, eth1 | 14:06 |
| cloudnull | this is the host interface config I use | 14:06 |
| cloudnull | https://github.com/cloudnull/osa-multi-node-aio/blob/master/templates/kvm-bonded-bridges.cfg | 14:06 |
| briancubed | kernel log on the deployment host? It doubles as the 1st infra host. | 14:06 |
| cloudnull | yes | 14:07 |
| briancubed | okay. i'll have a look. Time for a meeting... Be back later... | 14:07 |
| *** sdake has joined #openstack-ansible | 14:07 | |
| cloudnull | this is the sg offload option that needs to be there | 14:07 |
| briancubed | thank you! | 14:07 |
| cloudnull | https://github.com/cloudnull/osa-multi-node-aio/blob/master/templates/kvm-bonded-bridges.cfg#L75 | 14:07 |
| cloudnull | anytime best of luck briancubed | 14:07 |
| cloudnull | have a good meeting :) | 14:07 |
| javeriak | cloudnull the code in the rabbitmq_ssl_key_distribute.yml file is quite different between liberty and mitaka ... my paste above ^ | 14:08 |
| javeriak | not sure how to apply the mitaka fixes herein liberty | 14:08 |
| cloudnull | javeriak: looking | 14:08 |
| *** Brew has joined #openstack-ansible | 14:12 | |
| *** sdake has quit IRC | 14:12 | |
| *** automagically has joined #openstack-ansible | 14:14 | |
| admin0 | odyssey4me: what does link referenced in any content mean ? | 14:16 |
| odyssey4me | you provide a link, but no content references the link | 14:17 |
| admin0 | isn’t it the same as the other 3 links above that just give more info/examples on the subject ? | 14:17 |
| *** ChrisBenson has joined #openstack-ansible | 14:17 | |
| cloudnull | javeriak: http://cdn.pasteraw.com/lc2zwqot88trbcqns07gmlvm6jkb07g that should do it | 14:17 |
| odyssey4me | the other links are referenced in content | 14:17 |
| admin0 | oh .. so i should say below the example code block . hey if you need a working example, go to this link and mention the link again ? | 14:18 |
| admin0 | sorry .. new to all this :D | 14:18 |
| javeriak | thanks cloudnull | 14:19 |
| *** rromans_ is now known as rromans | 14:19 | |
| cloudnull | give that a go and let me know. | 14:19 |
| javeriak | yes trying now | 14:19 |
| cloudnull | if it works for you, would you mind putting together a backport PR for that ? | 14:19 |
| odyssey4me | admin0 see how line 10 has the content, and line 16 provides the link for the content | 14:20 |
| odyssey4me | admin0 also look at the gate test result for docs to see how the final document is rendered | 14:20 |
| *** thorst_ has quit IRC | 14:21 | |
| admin0 | i have no clue what a gate is :D | 14:21 |
| *** sdake has joined #openstack-ansible | 14:21 | |
| admin0 | how do I see it ? | 14:22 |
| odyssey4me | admin0 look at https://review.openstack.org/301922 - see the jenkins check result, it has a link - that link for the docs job shows the resulting rendered document | 14:22 |
| cloudnull | admin0: "the gate" is the common term used for the jenkins test environment | 14:22 |
| odyssey4me | admin0 if you execute 'tox -e docs' on your computer (after installing tox with pip) then you'll get a rendered set of documents on your workstation that you can browse | 14:22 |
| *** thorst_ has joined #openstack-ansible | 14:22 | |
| cloudnull | http://docs.openstack.org/infra/manual/developers.html#automated-testing | 14:23 |
| admin0 | now I know how to check the gate \o/ | 14:23 |
| palendae | Worth noting: I don't know that tox -e docs works for all the repos; openstack-ansible definitely | 14:23 |
| *** KLevenstein has quit IRC | 14:24 | |
| admin0 | i want to first contribute to the openstack-ansible before expanding :) | 14:24 |
| admin0 | but i know how to check gate now :) | 14:24 |
| cloudnull | automagically: ping | 14:24 |
| automagically | Yes? | 14:24 |
| cloudnull | is this something we should be pushing to other roles https://review.openstack.org/#/c/291908/ ? | 14:24 |
| cloudnull | seems sensible to me | 14:24 |
| *** Mudpuppy has joined #openstack-ansible | 14:25 | |
| *** Mudpuppy has quit IRC | 14:25 | |
| cloudnull | but w/ the POC tag in front I wanted to raise the question | 14:25 |
| automagically | cloudnull: I think so. A little up front validation by each role that its preconditions are satisfied should help discover issues more quickly | 14:25 |
| cloudnull | +1 | 14:25 |
| *** Mudpuppy has joined #openstack-ansible | 14:25 | |
| cloudnull | I like it, odyssey4me? | 14:26 |
| automagically | I think I got enough positive feedback on it and no negative feedback at this point, so I think we are ready to move forward | 14:26 |
| *** thorst__ has joined #openstack-ansible | 14:26 | |
| admin0 | apart from documentation, where can I help on the testing side ? on new deployments or upgrades ? | 14:27 |
| automagically | I need to amend that patchset before further reviews though, I think I just spotted an issue with it | 14:27 |
| cloudnull | I'd almost think that the individual role secrets should be checked as well as the playbook secrets, something like rabbitmq / mariadb user creates, etc.. | 14:27 |
| *** thorst_ has quit IRC | 14:27 | |
| cloudnull | ok | 14:27 |
| cloudnull | was just working through my review q so i figured I'd ping you. | 14:28 |
| *** spotz_zzz is now known as spotz | 14:28 | |
| openstackgerrit | Travis Truman (automagically) proposed openstack/openstack-ansible-os_ceilometer: Fail fast when required secrets are not present https://review.openstack.org/291908 | 14:29 |
| *** javeriak has quit IRC | 14:30 | |
| *** automagically has left #openstack-ansible | 14:30 | |
| *** automagically has joined #openstack-ansible | 14:31 | |
| cloudnull | automagically: i have a nit sorry. : ) | 14:34 |
| *** javeriak has joined #openstack-ansible | 14:34 | |
| *** pjm6 has quit IRC | 14:35 | |
| *** thorst__ has quit IRC | 14:37 | |
| *** spotz is now known as spotz_zzz | 14:38 | |
| javeriak | cloudnull your patch worked, I'll push a PR into liberty in a bit | 14:40 |
| cloudnull | +1 | 14:40 |
| javeriak | also i just got some more feedback from the team working with the osic cluster, and openstack calls are going through very slowly | 14:41 |
| *** spotz_zzz is now known as spotz | 14:41 | |
| *** thorst_ has joined #openstack-ansible | 14:41 | |
| automagically | cloudnull: Agreed with your nit, new patchset coming | 14:41 |
| admin0 | odyssey4me: | 14:42 |
| admin0 | oops | 14:42 |
| cloudnull | automagically: i shouldve raised that before but i wasnt thinking, and then i was, and then i saw a squirrel, and now here we are. :) | 14:43 |
| *** psilvad_ has quit IRC | 14:44 | |
| *** pjm6 has joined #openstack-ansible | 14:44 | |
| openstackgerrit | Travis Truman (automagically) proposed openstack/openstack-ansible-os_ceilometer: Fail fast when required secrets are not present https://review.openstack.org/291908 | 14:45 |
| openstackgerrit | Merged openstack/openstack-ansible: Specify allocation pool for public subnet https://review.openstack.org/300489 | 14:45 |
| *** psilvad_ has joined #openstack-ansible | 14:47 | |
| *** ametts has quit IRC | 14:48 | |
| *** psilvad_ has quit IRC | 14:48 | |
| *** ametts has joined #openstack-ansible | 14:49 | |
| *** javeriak_ has joined #openstack-ansible | 14:49 | |
| *** saneax is now known as saneax_AFK | 14:50 | |
| *** javeriak has quit IRC | 14:50 | |
| admin0 | dunno what happened but now I get: http://pastebin.com/BnV6Uaz1 .. anyone can help me with this git thing :D ? | 14:52 |
| busterswt | @palendae https://bugs.launchpad.net/openstack-ansible/+bug/1563448 | 14:53 |
| openstack | Launchpad bug 1563448 in openstack-ansible "Change to linuxbridge agent defaults affects upgrades" [Medium,Triaged] - Assigned to admin0 (shashi-eu) | 14:53 |
| palendae | busterswt: Ahhh, ok, I think I misunderstood your previous comment | 14:54 |
| openstackgerrit | Javeria Khan proposed openstack/openstack-ansible: Fixes for SSL key / cert distribution for rabbitmq https://review.openstack.org/302824 | 14:54 |
| busterswt | palendae sorry, i probably didn't phrase it well. | 14:54 |
| *** thorst__ has joined #openstack-ansible | 14:54 | |
| admin0 | busterswt: i took that up, but l2pop by default is not recommended .. so it is to be documented and left to the person upgrading | 14:55 |
| *** psilvad_ has joined #openstack-ansible | 14:55 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 14:55 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 14:55 | |
| odyssey4me | cloudnull automagically re https://review.openstack.org/291908 yes, I think it's a good pattern - my only other suggestion was to have an 'extras' folder in the role with a 'user_secrets_rolename.yml' file in it. Then the README can simply include that file, and when the role is included in the integrated build it can grab the file (potentially). Something like https://github.com/openstack/openstack-ansible-os_designate/blob/master/extras | 14:55 |
| odyssey4me | /user_secrets_designate.yml | 14:55 |
| busterswt | admin0 thanks. Are we saying it's not recommended based on the comments in https://review.openstack.org/#/c/252100/? Or has there been other discussion outside of that? | 14:55 |
| busterswt | admin0 I'm not disagreeing, but multicast probably introduces other operational issues. | 14:56 |
| *** _55_ has quit IRC | 14:56 | |
| admin0 | that, and I also had a chat on the neutron channel where it was suggested not to enable that | 14:56 |
| * busterswt wishes that would have been the message four releases ago. | 14:57 | |
| *** ametts has quit IRC | 14:57 | |
| cloudnull | busterswt: that was the recommendation from the neutron team due to additional complexity. | 14:57 |
| *** javeriak has joined #openstack-ansible | 14:57 | |
| busterswt | cloudnull for sure. we see it every day :) | 14:57 |
| cloudnull | hahaha | 14:57 |
| *** cristicalin has joined #openstack-ansible | 14:58 | |
| *** thorst_ has quit IRC | 14:58 | |
| automagically | Interesting idea odyssey4me | 14:58 |
| *** ametts has joined #openstack-ansible | 14:58 | |
| cloudnull | but if users are upgrading from earlier releases, where it wasn't an option to have it disabled, and or deployers think it should always be on, its easy enough to enable. | 14:58 |
| automagically | Will dig into that a bit and see what I can do | 14:58 |
| *** cristicalin has quit IRC | 14:58 | |
| *** cristicalin has joined #openstack-ansible | 14:59 | |
| *** javeriak_ has quit IRC | 14:59 | |
| odyssey4me | busterswt so when we upgrade the agent, the agent restarts - so I take it that the restart then doesn't do what's needed? | 14:59 |
| busterswt | cloudnull yeah i'm not opposed to turning it off by default, but i think there needs to be more documentation/discussion around the pros/cons of either approach, and how to revert | 14:59 |
| *** mgoddard_ has joined #openstack-ansible | 15:00 | |
| odyssey4me | busterswt cloudnull a big part of the feedback from the neutron team was that l2pop is very buggy and should be avoided | 15:00 |
| *** thorst__ is now known as thorst________ | 15:00 | |
| odyssey4me | it seems that somehow we managed to scrape by with it on in previous releases | 15:00 |
| busterswt | odyssey4me that's correct. The vxlan interfaces are not 'refreshed' as part of an agent restart, so there's technically no downtime during a restart of the agent. But what happens as a result is that ARP proxy stays ON, and the bridge flooding entries aren't updated. So with l2pop off, the ARP table isn't populated, ARP traffic gets dropped due to proxy, and no updates to the multicast group | 15:01 |
| busterswt | odyssey4me l2pop has been on by default since v9 and yes, it's not been without issues. some within Neutron control, others outside of it. | 15:01 |
| odyssey4me | busterswt perhaps that relates to https://bugs.launchpad.net/openstack-ansible/+bug/1523031 ? | 15:01 |
| openstack | Launchpad bug 1523031 in openstack-ansible trunk "Neighbor table entry for router missing with Linux bridge + L3HA + L2 population" [High,Confirmed] | 15:01 |
| openstackgerrit | Sashi Dahal proposed openstack/openstack-ansible: ceph configuration for glance and nova https://review.openstack.org/301922 | 15:02 |
| deadnull | so the setup-openstack ran clean after a re-do, everything look s good except neutron, returning a 503, going to dive into it now, but glance is happy and I have an image uploaded! | 15:02 |
| deadnull | cc odyssey4me cloudnull ^^ | 15:02 |
| busterswt | odyssey4me yeah that's one. Also allowed-address-pairs issues addressed by https://bugs.launchpad.net/neutron/+bug/1551805 | 15:02 |
| openstack | Launchpad bug 1551805 in openstack-manuals " add arp_responder flag to linuxbridge agent" [Undecided,New] - Assigned to Anseela M M (anseela-m00) | 15:02 |
| *** mgoddard has quit IRC | 15:03 | |
| odyssey4me | busterswt that's a docimpact bug, I expect you mean https://bugs.launchpad.net/neutron/+bug/1445089 ? | 15:04 |
| openstack | Launchpad bug 1445089 in neutron "allowed-address-pairs broken with l2pop/arp responder and LinuxBridge/VXLAN" [Undecided,Fix released] - Assigned to Mark McClain (markmcclain) | 15:04 |
| *** pjm6_ has joined #openstack-ansible | 15:05 | |
| *** briancubed has quit IRC | 15:05 | |
| odyssey4me | hmm, it looks like that's resolve for Mitaka and Liberty | 15:05 |
| cloudnull | RE:L2POP -- not to be that guy, but it would’ve been good for us to bring up concerns like this 4 months ago. though I think a doc fix is in order for liberty | 15:07 |
| cloudnull | odyssey4me: that issue is resolved in mitaka | 15:07 |
| odyssey4me | yeah, perhaps a release note is in order | 15:07 |
| *** thorst________ is now known as thorst_ | 15:08 | |
| cloudnull | and this https://review.openstack.org/#/c/288050/ review fixes the arp responder problem in liberty | 15:08 |
| cloudnull | which was merged | 15:08 |
| *** pjm6 has quit IRC | 15:08 | |
| cloudnull | so when we rev forward that'll be fixed for us too. | 15:08 |
| busterswt | odyssey4me yes, that's the bug. | 15:08 |
| odyssey4me | we'll need someone to put together a doc entry and a release note pointing to it - and the doc entry should go through the reasoning, etc and recommend using the default, but explain how to enable l2pop if that's preferred | 15:08 |
| odyssey4me | I don't have the expertise to figure that one out - not sure if busterswt is happy to do it? | 15:09 |
| admin0 | odyssey4me: is it just mentining in the docs .. hey if you have l2pop enabled, set up this variblle to true, else it will break ? | 15:09 |
| odyssey4me | admin0 right now there is not entry, which is why we're discussing that there should be | 15:10 |
| cloudnull | i think something added in reno would be good | 15:11 |
| admin0 | whats a reno :D ? | 15:11 |
| cloudnull | release notes | 15:11 |
| admin0 | :D | 15:11 |
| deadnull | cloudnull figured out the vlan adapter needs more than 1 VLAN to unpack ;-) | 15:11 |
| cloudnull | ah, yes it needs a range, even if that range is 1 -- 1:1 | 15:12 |
| deadnull | gotcha | 15:12 |
| odyssey4me | admin0 http://docs.openstack.org/releasenotes/openstack-ansible/mitaka.html is generated from https://github.com/openstack/openstack-ansible/tree/stable/mitaka/releasenotes/notes | 15:12 |
| deadnull | what is the best process to change the config and re-deploy neutron, just setup-openstack.yml --limit neutron? | 15:12 |
| odyssey4me | deadnull it'd be neutron_all | 15:13 |
| deadnull | odyssey4me gotcha, makes sense, thanks! | 15:13 |
| odyssey4me | but you can also just run os-neutron-install.yml | 15:13 |
| deadnull | of course I can :) | 15:14 |
| deadnull | makes sense, learning the lay of the playbooks still | 15:14 |
| *** javeriak has quit IRC | 15:15 | |
| *** pjm6 has joined #openstack-ansible | 15:15 | |
| *** wangqwsh has quit IRC | 15:17 | |
| *** javeriak has joined #openstack-ansible | 15:18 | |
| busterswt | odyssey4me happy to update the docs at some point | 15:18 |
| palendae | cloudnull: I agree with you, but no one was working on upgrades 4 months ago | 15:18 |
| *** pjm6_ has quit IRC | 15:19 | |
| cloudnull | deadnull: i use openstack-ansible os-neutron-install.yml | 15:19 |
| cloudnull | if its just a reconfiguration then you can use a tag | 15:19 |
| cloudnull | --tags neutron-config | 15:19 |
| odyssey4me | busterswt that would be great - it'll take a bit of discussion with the neutron guys to get the details of why not to use l2pop and what circumstances it'd ok to use it so that the documentation can inform appropriately | 15:20 |
| busterswt | cloudnull odyssey4me that arp responder bug addresses the allowed-address-pairs issue we had, but has the side benefit of addressing slow arp programming by the L2 agents | 15:20 |
| odyssey4me | to get the right information will need someone who understands how it all works, otherwise I'd happily help out, but I'm dumb with this stuff :/ | 15:21 |
| busterswt | but it's unknown as to how the arp traffic across the overlay affects network performance, so something we need to look at | 15:21 |
| cloudnull | palendae: true but we we're all here when that conversation was happening and no concerns were raised then. Not saying that its bad to rehash it but we could better participation from downstream as things are developed which will help us catch these types of things | 15:21 |
| busterswt | no worries | 15:21 |
| cloudnull | **we could use better | 15:21 |
| odyssey4me | cloudnull my hope is that we can work on Mitaka->Newton upgrades between Newton-3 and Newton-RC1 in this cycle to try and catch it all up so that we don't end up in this sort of situation | 15:22 |
| cloudnull | ++ | 15:22 |
| odyssey4me | ideally we should be providing feedback to the service dev teams before a release, not after | 15:22 |
| cloudnull | also we have reno and are forcing folks to use it so that will help too | 15:23 |
| cloudnull | in early liberty it was not implemented or forced | 15:23 |
| odyssey4me | yup, in this cycle we've upped the requirement for docs and release notes where warranted | 15:23 |
| *** weezS has joined #openstack-ansible | 15:24 | |
| odyssey4me | even the restrospective reno compiles by mhayden have been useful to some - many thanks to mhayden for putting those together! | 15:24 |
| mhayden | wheee :) | 15:24 |
| cloudnull | ++ | 15:24 |
| deadnull | cloudnull odyssey4me looking much better, i have networking service online and happy... going to grab a quick bite, but the instance didnt launch. no valid host was found... | 15:25 |
| cloudnull | did you configure a flat network by chance? | 15:26 |
| cloudnull | in the openstack_user_config file? | 15:26 |
| deadnull | no flat network | 15:27 |
| deadnull | i dont have any native vlans on br-vlan | 15:27 |
| cloudnull | something like so in the config https://github.com/openstack/openstack-ansible/blob/master/etc/openstack_deploy/openstack_user_config.yml.aio#L42-L50 ? | 15:27 |
| *** cloudtrainme has quit IRC | 15:28 | |
| deadnull | cloudnull flat is not required, correct? or did I read that wrong | 15:28 |
| cloudnull | it is not. | 15:28 |
| deadnull | ok | 15:28 |
| cloudnull | im asking because a lot of folks have tripped over that | 15:28 |
| cloudnull | but sounds like your good | 15:28 |
| deadnull | gotcha | 15:29 |
| *** cristicalin has quit IRC | 15:29 | |
| deadnull | would i need to re-run any playbooks on nova | 15:29 |
| cloudnull | no. | 15:29 |
| cloudnull | not for a network reconfiguration | 15:29 |
| cloudnull | using the same driver | 15:29 |
| cloudnull | check the scheduler/nova services | 15:29 |
| cloudnull | nova service-list | 15:30 |
| cloudnull | also make sure all of the neutron agents are happy | 15:30 |
| cloudnull | neutron agent-list | 15:30 |
| deadnull | ok hopping in my utility container | 15:30 |
| cloudnull | go eat :) | 15:30 |
| cloudnull | openstack will be broken when you get back :) | 15:30 |
| cloudnull | referencing "going to grab a quick bite" -- :D | 15:30 |
| odyssey4me | maybe he needs the openstack-fixerater :p | 15:31 |
| cloudnull | the fixerator can help | 15:31 |
| cloudnull | https://gist.github.com/cloudnull/9361461 | 15:31 |
| *** cloudtrainme has joined #openstack-ansible | 15:31 | |
| cloudnull | also if you need to know how it works use "asd" | 15:31 |
| cloudnull | :p | 15:31 |
| cloudnull | -- https://pypi.python.org/pypi/AdvancedSearchDiscovery | 15:32 |
| * cloudnull yup im a helper | 15:32 | |
| deadnull | lol @ fixerator | 15:32 |
| deadnull | like asd, been using silversearcher on my log container | 15:33 |
| deadnull | nova and neutron are happy | 15:33 |
| deadnull | bbs | 15:33 |
| *** deadnull is now known as rovalent_afk | 15:33 | |
| cloudnull | deadnull: asd isnt that useful. put it in a venv and try it out. | 15:33 |
| mhayden | odyssey4me / cloudnull: i'm totally okay with a new branch on the security role for https://review.openstack.org/#/c/302385/ | 15:37 |
| *** galstrom_zzz is now known as galstrom | 15:37 | |
| odyssey4me | mhayden so the SHA would still have to be managed anyway with a branch - I'll have to check with the release team whether we can create a liberty branch at this stage | 15:38 |
| cloudnull | odyssey4me: +1 for a branch | 15:40 |
| cloudnull | w/ it we can backport as needed | 15:40 |
| cloudnull | and manage it outside of the OSA repo | 15:40 |
| cloudnull | IMO it will stay more stable that way | 15:40 |
| mhayden | odyssey4me: thanks for asking about that | 15:41 |
| *** mikelk has quit IRC | 15:43 | |
| *** afranc has joined #openstack-ansible | 15:44 | |
| cloudnull | javeriak: ping | 15:44 |
| cloudnull | regarding performance https://review.openstack.org/#/c/301288 | 15:44 |
| cloudnull | that patch in liberty will make a fairly big impact | 15:45 |
| *** javeriak has quit IRC | 15:45 | |
| cloudnull | we are running that now in the OSIC public cloud w/ ~375 compute nodes and haven't seen performance issues like you we're describing. not that, that change alone will fix all the things, but its something worth considering | 15:46 |
| mhayden | we should rewrite ansible in erlang | 15:47 |
| cloudnull | um no. . . | 15:47 |
| cloudnull | :) | 15:47 |
| evrardjp | for security, branch + tag imo | 15:47 |
| cloudnull | erlang is the devil | 15:47 |
| evrardjp | cloudnull erlang is 1970s | 15:48 |
| evrardjp | not the devil | 15:48 |
| mhayden | evrardjp: branch *and* tag? | 15:48 |
| cloudnull | 1970s == devil | 15:48 |
| *** gregfaust has joined #openstack-ansible | 15:48 | |
| *** rovalent_afk is now known as _deadnull | 15:48 | |
| cloudnull | cores / reviewers -- these changes could use some love -- https://review.openstack.org/#/q/project:%255Eopenstack/openstack-ansible.*+branch:liberty+status:open,n,z | 15:48 |
| evrardjp | mhayden yup, branching for an easy differentiation, tags for an easy versioning of update inside the branch | 15:49 |
| evrardjp | but that's only my personal opinion | 15:49 |
| *** admin0 has quit IRC | 15:49 | |
| mhayden | are we tagging with the other IRR's? maybe i missed that | 15:49 |
| evrardjp | we do | 15:50 |
| evrardjp | depending on which ones | 15:51 |
| evrardjp | but keepalived does | 15:51 |
| evrardjp | however keepalived doesn't need the branches, so it's quite easier | 15:51 |
| evrardjp | the advantage of tagging is an easy update using ansible-galaxy | 15:51 |
| evrardjp | I'm not sure it's really needed for everything cloned directly on openstack git repos instead of galaxy | 15:52 |
| *** pcaruana has quit IRC | 15:53 | |
| *** jayc has joined #openstack-ansible | 15:54 | |
| *** shanec has joined #openstack-ansible | 15:57 | |
| *** eil397 has joined #openstack-ansible | 15:59 | |
| *** jmccrory_ has joined #openstack-ansible | 15:59 | |
| *** shanec_ has joined #openstack-ansible | 16:00 | |
| odyssey4me | meeting in #openstack-meeting-4 cloudnull, mattt, andymccr, d34dh0r53, hughsaunders, b3rnard0, palendae, Sam-I-Am, odyssey4me, serverascode, rromans, erikmwilson, mancdaz, _shaps_, BjoernT, claco, echiu, dstanek, jwagner, ayoung, prometheanfire, evrardjp, arbrandes, mhayden, scarlisle, luckyinva, ntt, javeriak, automagically, spotz, vdo, jmccrory, alextricity25, jasondotstar, KLevenstein, admin0, michaelgugino, ametts, v1k0d3n, | 16:01 |
| odyssey4me | severion, bgmccollum | 16:01 |
| *** ametts has quit IRC | 16:01 | |
| *** javeriak has joined #openstack-ansible | 16:02 | |
| *** shanec has quit IRC | 16:02 | |
| javeriak | cloudnull sorry was away, cool so this should help with the openstack commands in general? since everything seems to be going thru slowly. guess i only need to run the rabbitmq playbooks after applying this? | 16:03 |
| *** automagically has quit IRC | 16:03 | |
| *** automagically has joined #openstack-ansible | 16:04 | |
| *** mgoddard_ has quit IRC | 16:06 | |
| *** jayc_ has joined #openstack-ansible | 16:07 | |
| *** jayc has quit IRC | 16:07 | |
| *** mgoddard has joined #openstack-ansible | 16:08 | |
| *** _deadnull is now known as rovalent_afk | 16:12 | |
| stevelle | o/ evrardjp, wb | 16:14 |
| evrardjp | stevelle thanks ;) | 16:15 |
| rovalent_afk | lunch acquired , full stomach clear head | 16:15 |
| pjm6 | anyone who had installed and runned tempest, had glance not working in uploading files? | 16:16 |
| pjm6 | delete / upload | 16:16 |
| *** ametts has joined #openstack-ansible | 16:17 | |
| cloudnull | javeriak: yes | 16:17 |
| cloudnull | itll restart the app and apply the changes | 16:17 |
| cloudnull | pjm6: no. not that i can recall. | 16:18 |
| pjm6 | hey cloudnull | 16:18 |
| pjm6 | thanks | 16:18 |
| pjm6 | for some reason it stops working the delete / update | 16:18 |
| pjm6 | but i can create vms with the image | 16:18 |
| pjm6 | if I try delete I receive: UnknownScheme: Unknown scheme 'file' found in URI | 16:18 |
| javeriak | cloudnull would it help it horizon too? it still seems sluggish to me... | 16:18 |
| pjm6 | in upload: Error in store configuration. Adding images to store is disabled. | 16:19 |
| cloudnull | javeriak: itll help services respond faster via messaging | 16:19 |
| cloudnull | so maybe ? | 16:19 |
| javeriak | alright, will let the testing continue and see what we get | 16:19 |
| javeriak | thanks for the link | 16:19 |
| rovalent_afk | cloudnull odyssey4me so odd, i re-ran os-nova-install.yml, and instance is running (from ISO off of glance) no block yet | 16:21 |
| *** jayc_ is now known as jayc | 16:25 | |
| *** Bjoern_ has joined #openstack-ansible | 16:28 | |
| *** daneyon_ has quit IRC | 16:28 | |
| *** daneyon has joined #openstack-ansible | 16:29 | |
| *** Bjoern_ is now known as BjoernT | 16:30 | |
| cloudnull | rovalent_afk: good to go then? :) | 16:36 |
| *** KLevenstein has joined #openstack-ansible | 16:37 | |
| *** Oku_OS is now known as Oku_OS-away | 16:38 | |
| *** Oku_OS-away is now known as Oku_OS | 16:39 | |
| *** iceyao has quit IRC | 16:39 | |
| *** eil397 has quit IRC | 16:42 | |
| *** eil397 has joined #openstack-ansible | 16:42 | |
| *** javeriak_ has joined #openstack-ansible | 16:44 | |
| *** pjm6 has quit IRC | 16:45 | |
| *** javeriak has quit IRC | 16:45 | |
| *** galstrom is now known as galstrom_zzz | 16:46 | |
| javeriak_ | cloudnull its still slow, im getting ~35sec for a neuton net-list; while the same thing on a local setup of mine takes 0.9s | 16:47 |
| javeriak_ | the cli is still useable though, just slow; horizon still comes and goes | 16:47 |
| cloudnull | bummer | 16:48 |
| cloudnull | well it was worth a try | 16:48 |
| *** Oku_OS is now known as Oku_OS-away | 16:48 | |
| javeriak_ | cloudnull yea its ok; do let me know if you guys think of anything else to help performance for such large installs | 16:51 |
| evrardjp | haproxy ? | 16:51 |
| *** rovalent_afk is now known as deadnull | 16:51 | |
| evrardjp | issues with reverse dns lookups, I don't know... I'm stabbing in the dark without logs :p | 16:52 |
| evrardjp | maybe you should analyse where the performance gets worse thanks to your load balancer | 16:53 |
| cloudnull | evrardjp: that may be something to improve db performance | 16:53 |
| javeriak_ | im using simple haproxy installed on the deploy node | 16:54 |
| evrardjp | I don't know the issue, I'm barely jumped on a topic I don't know, just trying to help | 16:54 |
| *** admin0 has joined #openstack-ansible | 16:54 | |
| javeriak_ | hmm i wonder if a proper dedicated LB would help hee | 16:54 |
| evrardjp | my point was: as all api calls go through the reverse proxy, you could theoretically deduce metrics there | 16:55 |
| cloudnull | hap should be plenty performant, though i have seen reverse DNS slowing things down in the past | 16:55 |
| evrardjp | haproxy could have its issues sometimes, depending on the config, but I've never seen it slow | 16:56 |
| evrardjp | I only saw configuration issues | 16:56 |
| *** cloudtrainme has quit IRC | 16:57 | |
| evrardjp | if you think its the fault of haproxy, try disabling in haproxy the backend in excess | 16:57 |
| evrardjp | temporarily | 16:57 |
| evrardjp | if it's faster, then it is haproxy, else you have to look elsewhere | 16:58 |
| evrardjp | but you could also do that by checking at haproxy metrics | 16:58 |
| cloudnull | javeriak_: you can try setting -- mysql> SET GLOBAL skip_name_resolve=1 | 16:58 |
| cloudnull | and see if that helps | 16:58 |
| *** weezS has quit IRC | 16:59 | |
| *** admin0 has quit IRC | 17:00 | |
| *** admin0 has joined #openstack-ansible | 17:00 | |
| javeriak_ | cloudnull alright | 17:03 |
| *** dsariel is now known as dsariel|afk | 17:04 | |
| openstackgerrit | Merged openstack/openstack-ansible: Ensure the OpenStack gate has access to the logs https://review.openstack.org/301282 | 17:04 |
| admin0 | i think finally satisfied everyone via this latest update to the ceph docs :D | 17:05 |
| *** ametts has quit IRC | 17:07 | |
| *** pcaruana has joined #openstack-ansible | 17:08 | |
| *** michaelgugino has joined #openstack-ansible | 17:08 | |
| *** pjm6 has joined #openstack-ansible | 17:08 | |
| *** dsariel|afk has quit IRC | 17:09 | |
| odyssey4me | jmccrory automagically d34dh0r53 stevelle a review of https://review.openstack.org/300505 would be appreciated to support https://review.openstack.org/296525 | 17:11 |
| * automagically looking | 17:11 | |
| odyssey4me | I'm out for the night - have a great day everyone. | 17:12 |
| automagically | Later odyssey4me | 17:12 |
| cloudnull | Also -- Congratulations everyone -- http://www.openstack.org/software/mitaka -- its been a lot of hard work and I'm prod to say I've worked with the lot of you on getting our Mitaka release out the door. | 17:13 |
| cloudnull | odyssey4me: cheers | 17:13 |
| *** ametts has joined #openstack-ansible | 17:20 | |
| *** pjm6 has quit IRC | 17:21 | |
| *** woodard has quit IRC | 17:24 | |
| *** pjm6 has joined #openstack-ansible | 17:24 | |
| *** nhadzter has joined #openstack-ansible | 17:28 | |
| openstackgerrit | Merged openstack/openstack-ansible-plugins: Add logic to NOT build "proprietary" packages https://review.openstack.org/300505 | 17:29 |
| openstackgerrit | Merged openstack/openstack-ansible: Ensure the OpenStack gate has access to the logs https://review.openstack.org/301283 | 17:38 |
| admin0 | should i test new mitaka first, or start with liberty -> mitaka ? | 17:39 |
| pjm6 | just now that i was starting to begin understand and have liberty working xD | 17:39 |
| pjm6 | anyway congratulations guys for the good work :D | 17:39 |
| *** weezS has joined #openstack-ansible | 17:48 | |
| cloudnull | admin0: i'd start with liberty | 17:49 |
| admin0 | liberty -> mitaka | 17:49 |
| cloudnull | mitaka is out , but many of the projects will be stabalizing over the next few weeks . | 17:49 |
| deadnull | rabbit, you are going to be the death of me today | 17:49 |
| cloudnull | pjm6 too | 17:50 |
| deadnull | still hitting the no valid host, not enough hosts available... | 17:50 |
| jmccrory | deadnull: when trying to boot an instance? | 17:53 |
| deadnull | jmccrory , just trying to launch an instance, i added both a qcow2 (ubuntu) image and an iso (alpine) to glance | 17:53 |
| deadnull | block is failing still but im less concerned about that | 17:53 |
| *** phiche has joined #openstack-ansible | 17:54 | |
| jmccrory | make sure your compute services are active with `nova service-list`, and nova-scheduler and nova-compute logs should have more info on why hypervisors are either being filtered out or failing to schedule | 17:55 |
| jmccrory | also check `neutron agent-list` | 17:56 |
| deadnull | jmccrory neutron agent-list and nova service-list are great | 17:56 |
| deadnull | i'll check logs again | 17:56 |
| admin0 | deadnull: first check the libvirt log in the compute node .. that usually tells you the exact reason | 17:56 |
| admin0 | like neutron not responding, network issue, cannot mount filesystem etc | 17:57 |
| michaelgugino | speaking of neutron plugin stuff... Hoping to get some more input on https://review.openstack.org/#/c/298765/ and the associated change https://review.openstack.org/#/c/300605/ | 17:57 |
| mongo2 | and as a last resort run a grep -ir with the machine UUID against the rsyslog dir. | 17:57 |
| admin0 | deadnull are you using spreading ? | 17:57 |
| deadnull | admin0 spreading? :) | 17:57 |
| admin0 | deadnull: https://www.openstackfaq.com/openstack-stacking-vs-spreading/ | 17:58 |
| admin0 | you might be hitting this :D | 17:58 |
| deadnull | nice doc! | 17:58 |
| admin0 | i have a 20 lines SQL query to predit this :D | 17:58 |
| deadnull | i have 1 nova compute node | 17:58 |
| *** phiche has quit IRC | 17:58 | |
| admin0 | but its too complex here | 17:58 |
| admin0 | jsut 1 node :D | 17:58 |
| admin0 | ok | 17:58 |
| deadnull | libvirtd.log has "virNetlinkEventCallback:340 : nl_recv returned with error: No buffer space available" | 17:59 |
| deadnull | im going ot start a new instance | 17:59 |
| admin0 | :D | 17:59 |
| admin0 | always check the libvirt first | 17:59 |
| admin0 | saves you time | 17:59 |
| *** mgariepy has quit IRC | 17:59 | |
| deadnull | good tip | 17:59 |
| deadnull | i was using silversearcher for the UUID on syslog container | 17:59 |
| deadnull | and i thought that was great :) | 17:59 |
| *** mgariepy has joined #openstack-ansible | 18:00 | |
| admin0 | i have thousands of vms on hundreds of compute nodes .. and no horizon .. and public cloud .. so issues keep popping up here and there | 18:00 |
| admin0 | and i think years of debugging experince = check libvirt first | 18:00 |
| admin0 | :D | 18:00 |
| deadnull | :) | 18:00 |
| *** cloudtrainme has joined #openstack-ansible | 18:01 | |
| deadnull | so the ISO instance launched from the image just fine. | 18:01 |
| deadnull | trying the qcow2 img now | 18:02 |
| admin0 | deadnull: whats the kernel on the compiute node ? | 18:02 |
| mongo2 | michaelgugino: I dont' have the time to test the proposed changes there but why not just use the plugin name? E.G. OVSNeutronPluginV2, this would allow people whom need to extend it to support vendor plugins to do so without mapping. | 18:02 |
| deadnull | 4.2.0-35-generic | 18:02 |
| admin0 | ok | 18:02 |
| deadnull | 3.13 was the min right | 18:02 |
| *** phiche has joined #openstack-ansible | 18:03 | |
| mongo2 | deadnull: are you using ixgbe network cards? | 18:03 |
| admin0 | deadnull: google will show that many have errors that you see . you need to check what you have and the solution | 18:04 |
| deadnull | mongo2 yes | 18:05 |
| deadnull | twinax running from all my nodes to a Cisco Cat 6k, right now for the POC single link to the upstream switch. Moving to bonds when we get to prod | 18:06 |
| michaelgugino | mongo2: the plugin name is ml2. OVS is a driver of the ml2 plugin. I don't think people are using OVSNeutronPluginV2 any more, it's not referenced in any docs. | 18:06 |
| michaelgugino | Does anyone have a list of the summit sessions for OSA? I looked at the schedule, only seen the 101 session | 18:10 |
| mongo2 | michaelgugino: Yes I only have my older hand installed stack there...one that actually suffered from teh pain of the change to ml_2. I'll try to deploy a stack, if I do i'll give feedback. We need it as linux bridge is pretty slow. | 18:10 |
| michaelgugino | mongo2: I'm more looking for code review more so than functional. Specifically if the cores like the method of implementation. | 18:11 |
| *** jayc has quit IRC | 18:12 | |
| mongo2 | deadnull: if you are on ubuntu try upgrading to linux-image-generic-lts-xenial and/or try adding "pre-up ethtool -G $IFACE rx 4078 tx 4078" to the interfaces. I am not at the office but there was some crazy thing with multiqueue and the igxbe driver IIRC. | 18:13 |
| admin0 | container_vars: for say X containers, where should it be alighed ? | 18:13 |
| admin0 | same line as hosts declaration ? | 18:13 |
| mongo2 | michaelgugino: yes, meatspace limitation, I have to do functional to do a valid code review. | 18:13 |
| cloudnull | michaelgugino: https://www.openstack.org/summit/austin-2016/summit-schedule/#day=2016-04-27&summit_types=2&tags=3542 | 18:16 |
| michaelgugino | sweet, thanks | 18:17 |
| *** javeriak_ has quit IRC | 18:17 | |
| michaelgugino | so, we're primarily getting together on wednesday then? | 18:17 |
| *** admin0 has quit IRC | 18:17 | |
| cloudnull | it seems that way | 18:18 |
| *** jmccrory_ has quit IRC | 18:18 | |
| cloudnull | i think we're wed-fri | 18:19 |
| *** v1k0d3n has quit IRC | 18:19 | |
| *** jayc has joined #openstack-ansible | 18:24 | |
| *** sdake_ has joined #openstack-ansible | 18:26 | |
| *** sdake has quit IRC | 18:26 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_neutron: Nuage neutron plugin ansible changes https://review.openstack.org/296525 | 18:26 |
| *** automagically has quit IRC | 18:28 | |
| *** BjoernT has quit IRC | 18:35 | |
| *** phiche1 has joined #openstack-ansible | 18:35 | |
| *** phiche has quit IRC | 18:38 | |
| *** admin0 has joined #openstack-ansible | 18:40 | |
| *** automagically has joined #openstack-ansible | 18:46 | |
| *** rstarmer has joined #openstack-ansible | 18:53 | |
| *** woodard has joined #openstack-ansible | 18:59 | |
| *** admin0 has quit IRC | 19:06 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 19:17 | |
| *** deadnull is now known as _deadnull | 19:26 | |
| *** clickboom has joined #openstack-ansible | 19:31 | |
| *** clickboom is now known as aabradsh | 19:35 | |
| *** aabradsh has quit IRC | 19:35 | |
| *** cloudtrainme has quit IRC | 19:37 | |
| *** LanceHaig has joined #openstack-ansible | 19:40 | |
| *** karimb has joined #openstack-ansible | 19:42 | |
| *** clickboom has joined #openstack-ansible | 19:43 | |
| *** _deadnull is now known as deadnull | 19:47 | |
| *** karimb has quit IRC | 19:47 | |
| *** michaelgugino has quit IRC | 19:54 | |
| *** McMurlock1 has joined #openstack-ansible | 19:57 | |
| *** ametts has quit IRC | 20:01 | |
| *** cloudtrainme has joined #openstack-ansible | 20:08 | |
| *** galstrom_zzz is now known as galstrom | 20:13 | |
| *** ametts has joined #openstack-ansible | 20:17 | |
| *** pjm6 has quit IRC | 20:23 | |
| *** sdake_ is now known as sdake | 20:23 | |
| *** jthorne_ has joined #openstack-ansible | 20:31 | |
| *** jthorne has quit IRC | 20:31 | |
| *** johnmilton has quit IRC | 20:32 | |
| *** woodard has quit IRC | 20:32 | |
| *** michaelgugino has joined #openstack-ansible | 20:33 | |
| openstackgerrit | Jimmy McCrory proposed openstack/openstack-ansible-galera_server: Remove dependency on python2_lxc git source https://review.openstack.org/303063 | 20:35 |
| *** ametts has quit IRC | 20:41 | |
| *** admin0 has joined #openstack-ansible | 20:56 | |
| mrda | Morning OSA | 21:00 |
| *** pcaruana has quit IRC | 21:01 | |
| michaelgugino | howdy | 21:01 |
| mrda | o/ | 21:02 |
| *** weshay has quit IRC | 21:03 | |
| *** psilvad_ has quit IRC | 21:04 | |
| *** deadnull has quit IRC | 21:05 | |
| *** eil397 has quit IRC | 21:06 | |
| *** clickboo_ has joined #openstack-ansible | 21:06 | |
| *** automagically has quit IRC | 21:08 | |
| *** Nepoc has quit IRC | 21:09 | |
| *** clickboom has quit IRC | 21:10 | |
| *** Nepoc has joined #openstack-ansible | 21:10 | |
| *** jayc has quit IRC | 21:11 | |
| *** thorst_ has quit IRC | 21:12 | |
| *** rstarmer has quit IRC | 21:13 | |
| *** clickboo_ has quit IRC | 21:15 | |
| *** thorst_ has joined #openstack-ansible | 21:18 | |
| *** Mudpuppy has quit IRC | 21:18 | |
| *** KLevenstein_ has joined #openstack-ansible | 21:18 | |
| *** Mudpuppy has joined #openstack-ansible | 21:19 | |
| *** McMurlock1 has quit IRC | 21:19 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ironic: Change the default thread pool size https://review.openstack.org/303079 | 21:19 |
| cloudnull | morning mrda | 21:19 |
| *** KLevenstein has quit IRC | 21:21 | |
| *** KLevenstein_ is now known as KLevenstein | 21:21 | |
| *** thorst__ has joined #openstack-ansible | 21:21 | |
| *** eil397 has joined #openstack-ansible | 21:21 | |
| *** thorst_ has quit IRC | 21:22 | |
| *** Mudpuppy has quit IRC | 21:23 | |
| *** woodard has joined #openstack-ansible | 21:24 | |
| *** jayc has joined #openstack-ansible | 21:25 | |
| *** thorst__ has quit IRC | 21:25 | |
| openstackgerrit | Julian Montez proposed openstack/openstack-ansible-os_swift: Cast region key's value as an integer https://review.openstack.org/303087 | 21:29 |
| *** spotz is now known as spotz_zzz | 21:30 | |
| *** weezS has quit IRC | 21:35 | |
| *** b3rnard0 is now known as b3rnard0_away | 21:35 | |
| *** galstrom is now known as galstrom_zzz | 21:36 | |
| *** rstarmer has joined #openstack-ansible | 21:38 | |
| *** woodard has quit IRC | 21:38 | |
| *** cloudtrainme has quit IRC | 21:41 | |
| *** thorst_ has joined #openstack-ansible | 21:44 | |
| *** weshay has joined #openstack-ansible | 21:45 | |
| *** thorst_ has quit IRC | 21:48 | |
| openstackgerrit | Jimmy McCrory proposed openstack/openstack-ansible-galera_server: Reorganize test playbooks https://review.openstack.org/303090 | 21:50 |
| *** jthorne_ has quit IRC | 21:51 | |
| *** phiche has joined #openstack-ansible | 21:54 | |
| *** phiche1 has quit IRC | 21:55 | |
| stevelle | I'm seeing haproxy stopped during run-playbooks, not sure why. | 21:57 |
| *** admin0 has quit IRC | 21:59 | |
| cloudnull | stevelle: hap is getting stopped ? | 22:04 |
| cloudnull | maybe port 5000 issue? | 22:04 |
| stevelle | cloudnull: stopped or dying | 22:04 |
| stevelle | service haproxy status shows it stopped, a start fixes and I can resume but it happened a couple times for me | 22:04 |
| stevelle | no clues in log | 22:05 |
| cloudnull | well thats not good | 22:05 |
| *** phiche has quit IRC | 22:07 | |
| *** phiche has joined #openstack-ansible | 22:08 | |
| *** phiche has quit IRC | 22:14 | |
| *** busterswt has quit IRC | 22:15 | |
| *** nhadzter has quit IRC | 22:19 | |
| *** winggundamth has quit IRC | 22:21 | |
| *** nhadzter has joined #openstack-ansible | 22:22 | |
| cloudnull | stevelle: i've seen that happen in the past with hap restarting and trying to bind port 5000 which is being used by "something else" for a moment | 22:24 |
| cloudnull | but not where it whole sale dies | 22:24 |
| * cloudnull looking some more | 22:24 | |
| stevelle | If we see it pop up more, I'd worry. | 22:25 |
| openstackgerrit | Merged openstack/openstack-ansible: Ensure cleanup-rabbitmq-vhost is re-runnable https://review.openstack.org/302377 | 22:25 |
| *** winggundamth has joined #openstack-ansible | 22:30 | |
| *** michaelgugino has quit IRC | 22:30 | |
| *** gregfaust has quit IRC | 22:31 | |
| *** cloudtrainme has joined #openstack-ansible | 22:31 | |
| *** cloudtrainme has quit IRC | 22:32 | |
| *** cloudtrainme has joined #openstack-ansible | 22:32 | |
| openstackgerrit | Jimmy McCrory proposed openstack/openstack-ansible-galera_server: Reorganize test playbooks https://review.openstack.org/303090 | 22:33 |
| *** Nepoc has quit IRC | 22:34 | |
| *** Nepoc has joined #openstack-ansible | 22:35 | |
| *** cloudtrainme has quit IRC | 22:37 | |
| *** Brew has quit IRC | 22:40 | |
| openstackgerrit | Jimmy McCrory proposed openstack/openstack-ansible-galera_server: Reorganize test playbooks https://review.openstack.org/303090 | 22:43 |
| *** sdake has quit IRC | 22:54 | |
| *** sdake has joined #openstack-ansible | 22:58 | |
| *** cloader89 has joined #openstack-ansible | 23:00 | |
| openstackgerrit | Michael Carden proposed openstack/openstack-ansible-ironic: Add tests for the ironic CLI https://review.openstack.org/303104 | 23:05 |
| *** rstarmer has quit IRC | 23:06 | |
| openstackgerrit | Jimmy McCrory proposed openstack/openstack-ansible-galera_server: Reorganize test playbooks https://review.openstack.org/303090 | 23:07 |
| neillc | cloudnull: horizon performance is a reall problem, especially at scale. it's a major focus of work for horizon devs. answer is things like the new swift ui r1chardj0n3s has been working on and perhaps serachlight | 23:10 |
| cloudnull | thakns neillc javeria -- not here now -- was working with it and seeing issues w/ liberty | 23:12 |
| *** sdake has quit IRC | 23:12 | |
| cloudnull | any tips I be able to pass on regarding making it better? | 23:12 |
| *** KLevenstein has quit IRC | 23:12 | |
| cloudnull | thanks for getting back to me btw neillc | 23:12 |
| neillc | np | 23:13 |
| neillc | sadly I don't have any great tips. richard is probably the person to ask, but not sure if he will either | 23:13 |
| neillc | many of the problems are about the apis we consume, combined with old style full page loads on every refresh | 23:14 |
| *** busterswt has joined #openstack-ansible | 23:14 | |
| stevelle | does horizon use the python-* clients or it's own rest apis generally? | 23:20 |
| openstackgerrit | Jimmy McCrory proposed openstack/openstack-ansible-galera_server: Reorganize test playbooks https://review.openstack.org/303090 | 23:28 |
| *** markvoelker has quit IRC | 23:29 | |
| *** jamielennox|away is now known as jamielennox | 23:36 | |
| *** eil397 has left #openstack-ansible | 23:46 | |
| *** busterswt has quit IRC | 23:52 | |
| *** markvoelker has joined #openstack-ansible | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!