| *** DanyC has quit IRC | 00:00 | |
| *** cshen has joined #openstack-ansible | 00:05 | |
| *** cshen has quit IRC | 00:10 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Add upgrade task options https://review.openstack.org/575496 | 00:31 |
|---|---|---|
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Add retries to all tasks with external deps https://review.openstack.org/575573 | 00:31 |
| *** tux_ has joined #openstack-ansible | 00:35 | |
| *** fatdragon has joined #openstack-ansible | 00:44 | |
| *** fatdragon has quit IRC | 00:44 | |
| *** tasker has joined #openstack-ansible | 00:50 | |
| tasker | https://docs.openstack.org/openstack-ansible-os_cinder/pike/configure-cinder.html << -- is there a list somewhere of all the available options for `cinder_backends`? I'm hoping to set GoodnessFunction for a volume type. | 00:52 |
| *** tasker has quit IRC | 00:59 | |
| *** jeferobles has joined #openstack-ansible | 01:02 | |
| *** gyee has quit IRC | 01:09 | |
| *** TxGirlGeek has joined #openstack-ansible | 01:40 | |
| *** TxGirlGeek has quit IRC | 01:45 | |
| *** pmannidi has joined #openstack-ansible | 01:46 | |
| *** markvoelker has joined #openstack-ansible | 01:47 | |
| *** JamesZhu has quit IRC | 01:54 | |
| *** JamesZhu has joined #openstack-ansible | 01:54 | |
| *** dxiri has joined #openstack-ansible | 01:55 | |
| *** DanyC has joined #openstack-ansible | 01:57 | |
| *** dxiri has quit IRC | 02:00 | |
| *** DanyC has quit IRC | 02:03 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_keystone master: Install upstream NGINX repository https://review.openstack.org/575273 | 02:03 |
| *** mriedem has quit IRC | 02:07 | |
| *** cjloader has joined #openstack-ansible | 02:08 | |
| *** dxiri has joined #openstack-ansible | 02:20 | |
| *** nurdie has quit IRC | 02:20 | |
| cloudnull | tasker the cinder _backend variable is a generator using the multi-backend type | 02:20 |
| cloudnull | so you can pass in any and all options | 02:21 |
| *** markvoelker has quit IRC | 02:21 | |
| *** dxiri has quit IRC | 02:25 | |
| *** nurdie has joined #openstack-ansible | 02:25 | |
| *** poopcat has quit IRC | 02:41 | |
| *** poopcat has joined #openstack-ansible | 02:45 | |
| openstackgerrit | Merged openstack/openstack-ansible-ops master: Add upgrade task options https://review.openstack.org/575496 | 02:56 |
| *** jeferobles has quit IRC | 02:57 | |
| *** cjloader has quit IRC | 02:59 | |
| *** DanyC has joined #openstack-ansible | 02:59 | |
| *** cjloader has joined #openstack-ansible | 02:59 | |
| *** DanyC has quit IRC | 03:03 | |
| *** cjloader has quit IRC | 03:04 | |
| *** dxiri has joined #openstack-ansible | 03:08 | |
| *** dxiri has quit IRC | 03:13 | |
| *** dxiri has joined #openstack-ansible | 03:17 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Update metricbeat to monitor new services https://review.openstack.org/575592 | 03:18 |
| *** markvoelker has joined #openstack-ansible | 03:18 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Update packetbeat to enable new captures https://review.openstack.org/575598 | 03:26 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Update auditbeat for new inclusions https://review.openstack.org/575601 | 03:33 |
| *** jeferobles has joined #openstack-ansible | 03:34 | |
| *** nurdie has quit IRC | 03:42 | |
| *** nurdie has joined #openstack-ansible | 03:42 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Update heartbeat to make better use of variables https://review.openstack.org/575602 | 03:42 |
| *** nurdie has quit IRC | 03:47 | |
| *** nurdie has joined #openstack-ansible | 03:50 | |
| *** markvoelker has quit IRC | 03:51 | |
| tux_ | Folks i am setting up networking using systemd-networkd https://major.io/2017/04/13/openstack-ansible-on-centos-7-with-systemd-networkd/ | 03:57 |
| tux_ | In this example Major used 4 different NIC, in my case i have single bon0 and i am trying to fit in major setup but it didn't work | 03:58 |
| tux_ | How do i add multiple bridge interface on single bond0? | 03:58 |
| tux_ | or i need to create VLAN on bond0 and map each vlan to associated bridge? | 03:59 |
| *** TxGirlGeek has joined #openstack-ansible | 03:59 | |
| *** poopcat has quit IRC | 04:10 | |
| tux_ | i think i figured out | 04:12 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Update filebeat https://review.openstack.org/575608 | 04:14 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Breakup site into more consumable meta playbooks https://review.openstack.org/575609 | 04:19 |
| cloudnull | tux_ youll have to create various vlans, or use an integration bridge with veth pairs | 04:20 |
| cloudnull | I personally would use a bond to vlan tagged interfaces | 04:20 |
| tux_ | I pick VLANs | 04:20 |
| cloudnull | ++ | 04:20 |
| tux_ | Very interesting, even my switch port isn't configured for TRUNK still i can ping host after setting up VLAN with systemd-networkd | 04:21 |
| prometheanfire | bond -> bridge -> break-out -> vlan-bridge-> veth | 04:22 |
| prometheanfire | then host and whatever you connect it to can access the vlan | 04:22 |
| prometheanfire | both vlat and tagged | 04:22 |
| prometheanfire | can attach a veth to the first bridge to get the trunked vlans | 04:23 |
| *** hasseniu_ has quit IRC | 04:23 | |
| *** cjloader has joined #openstack-ansible | 04:31 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Disable sniffing for elasticsearch output https://review.openstack.org/575614 | 04:33 |
| *** jeferobles has quit IRC | 04:34 | |
| *** cjloader has quit IRC | 04:36 | |
| tux_ | rebooting system to see what is going on | 04:36 |
| tux_ | there are very poor documentation regarding systemd-networkd :( | 04:37 |
| *** jeferobles has joined #openstack-ansible | 04:38 | |
| prometheanfire | The official docs worked for me | 04:43 |
| *** markvoelker has joined #openstack-ansible | 04:48 | |
| *** TxGirlGeek has quit IRC | 04:50 | |
| openstackgerrit | Merged openstack/openstack-ansible-ops master: Update metricbeat to monitor new services https://review.openstack.org/575592 | 04:58 |
| *** jeferobles has quit IRC | 05:00 | |
| openstackgerrit | Merged openstack/openstack-ansible-ops master: Add retries to all tasks with external deps https://review.openstack.org/575573 | 05:09 |
| *** tux_ has quit IRC | 05:11 | |
| *** nicolasbock has joined #openstack-ansible | 05:14 | |
| *** dxiri has quit IRC | 05:15 | |
| openstackgerrit | Merged openstack/openstack-ansible-ops master: Update packetbeat to enable new captures https://review.openstack.org/575598 | 05:17 |
| cloudnull | tux_ what are you seeing ? | 05:17 |
| *** nicolasbock has quit IRC | 05:18 | |
| *** pcaruana has quit IRC | 05:18 | |
| *** pmannidi_ has joined #openstack-ansible | 05:20 | |
| *** pmannidi_ has quit IRC | 05:20 | |
| *** markvoelker has quit IRC | 05:21 | |
| *** jeferobles has joined #openstack-ansible | 05:31 | |
| *** exodusftw has quit IRC | 05:32 | |
| *** exodusftw has joined #openstack-ansible | 05:37 | |
| openstackgerrit | Merged openstack/openstack-ansible-ops master: Disable sniffing for elasticsearch output https://review.openstack.org/575614 | 05:41 |
| openstackgerrit | Merged openstack/openstack-ansible-ops master: Update auditbeat for new inclusions https://review.openstack.org/575601 | 05:45 |
| *** jeferobles has quit IRC | 05:47 | |
| *** jeferobles has joined #openstack-ansible | 05:49 | |
| *** DanyC has joined #openstack-ansible | 05:53 | |
| *** jeferobles has quit IRC | 05:53 | |
| *** jeferobles has joined #openstack-ansible | 05:56 | |
| *** jeferobles has quit IRC | 06:03 | |
| *** jeferobles has joined #openstack-ansible | 06:09 | |
| *** DanyC has quit IRC | 06:11 | |
| *** dmsimard has quit IRC | 06:13 | |
| *** mma has quit IRC | 06:16 | |
| *** cshen has joined #openstack-ansible | 06:23 | |
| *** jeferobles has quit IRC | 06:25 | |
| *** jeferobles has joined #openstack-ansible | 06:25 | |
| *** dmsimard has joined #openstack-ansible | 06:27 | |
| *** cjloader has joined #openstack-ansible | 06:27 | |
| *** jeferobles has quit IRC | 06:28 | |
| *** jeferobles has joined #openstack-ansible | 06:31 | |
| *** maybebuggy has joined #openstack-ansible | 06:31 | |
| *** cjloader has quit IRC | 06:32 | |
| openstackgerrit | ShangXiao proposed openstack/openstack-ansible-os_keystone master: Add release notes link to README https://review.openstack.org/575625 | 06:34 |
| *** jeferobles has quit IRC | 06:34 | |
| *** gkadam has joined #openstack-ansible | 06:38 | |
| *** jeferobles has joined #openstack-ansible | 06:40 | |
| openstackgerrit | ShangXiao proposed openstack/openstack-ansible-os_glance master: Add release notes link to README https://review.openstack.org/575626 | 06:41 |
| *** Bhujay has joined #openstack-ansible | 06:43 | |
| *** pcaruana has joined #openstack-ansible | 06:44 | |
| *** harryhuang_ has joined #openstack-ansible | 06:46 | |
| openstackgerrit | ShangXiao proposed openstack/openstack-ansible-os_blazar master: Add release notes link to README https://review.openstack.org/575627 | 06:48 |
| *** epalper has joined #openstack-ansible | 06:50 | |
| openstackgerrit | Merged openstack/openstack-ansible-ops master: Update filebeat https://review.openstack.org/575608 | 06:54 |
| *** zul has quit IRC | 06:56 | |
| *** zul has joined #openstack-ansible | 06:56 | |
| *** jeferobles has quit IRC | 07:00 | |
| *** pmannidi has quit IRC | 07:03 | |
| *** sep_ has quit IRC | 07:06 | |
| *** sep_ has joined #openstack-ansible | 07:06 | |
| *** mma has joined #openstack-ansible | 07:07 | |
| openstackgerrit | Merged openstack/openstack-ansible-ops master: Update heartbeat to make better use of variables https://review.openstack.org/575602 | 07:07 |
| *** mbuil has joined #openstack-ansible | 07:09 | |
| *** fghaas has joined #openstack-ansible | 07:09 | |
| *** hassenius has joined #openstack-ansible | 07:11 | |
| *** markvoelker has joined #openstack-ansible | 07:18 | |
| openstackgerrit | ShangXiao proposed openstack/openstack-ansible-os_masakari master: Update README.rst information https://review.openstack.org/575662 | 07:21 |
| *** poul_ has joined #openstack-ansible | 07:22 | |
| *** mma has quit IRC | 07:26 | |
| *** sxc731 has joined #openstack-ansible | 07:26 | |
| openstackgerrit | ShangXiao proposed openstack/openstack-ansible-os_designate master: Add release notes link to README https://review.openstack.org/575666 | 07:28 |
| *** tosky has joined #openstack-ansible | 07:28 | |
| *** vnogin has joined #openstack-ansible | 07:30 | |
| openstackgerrit | ShangXiao proposed openstack/openstack-ansible-os_monasca master: Add release notes link to README https://review.openstack.org/575667 | 07:35 |
| *** radeks has joined #openstack-ansible | 07:38 | |
| *** vurtz has joined #openstack-ansible | 07:41 | |
| *** Bhujay has quit IRC | 07:46 | |
| *** fghaas has left #openstack-ansible | 07:46 | |
| *** markvoelker has quit IRC | 07:50 | |
| openstackgerrit | ShangXiao proposed openstack/openstack-ansible-os_magnum master: Add release notes link to README https://review.openstack.org/575674 | 07:51 |
| *** jbadiapa_ is now known as jbadiapa | 07:54 | |
| openstackgerrit | Merged openstack/openstack-ansible-openstack_hosts master: Correct defaults https://review.openstack.org/574766 | 07:57 |
| openstackgerrit | ShangXiao proposed openstack/openstack-ansible-memcached_server master: Update links information in README.rst https://review.openstack.org/575678 | 07:58 |
| *** rmart04 has joined #openstack-ansible | 07:59 | |
| *** Bhujay has joined #openstack-ansible | 08:02 | |
| openstackgerrit | Merged openstack/openstack-ansible-tests master: Add yum.repos.d to log collection https://review.openstack.org/574722 | 08:03 |
| *** DanyC has joined #openstack-ansible | 08:11 | |
| *** Bhujay has joined #openstack-ansible | 08:14 | |
| *** radeks has quit IRC | 08:16 | |
| *** pvradu has joined #openstack-ansible | 08:20 | |
| *** threestrands has quit IRC | 08:20 | |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-os_nova master: Add support for using distribution packages for OpenStack services https://review.openstack.org/567530 | 08:24 |
| *** cshen has quit IRC | 08:26 | |
| *** nurdie has quit IRC | 08:34 | |
| *** markvoelker has joined #openstack-ansible | 08:48 | |
| *** cshen has joined #openstack-ansible | 08:53 | |
| *** pvradu has quit IRC | 09:00 | |
| *** pvradu has joined #openstack-ansible | 09:01 | |
| *** electrofelix has joined #openstack-ansible | 09:03 | |
| openstackgerrit | Merged openstack/openstack-ansible-openstack_hosts master: Add release notes link to README https://review.openstack.org/575049 | 09:09 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-os_neutron master: Do not pip install keystoneclient on the host https://review.openstack.org/575701 | 09:19 |
| *** markvoelker has quit IRC | 09:22 | |
| hwoarang | mnaser: do you know what else would be great? if you can put a more recent virtualenv on RDO. They have python-pip but not virtualenv. That would be awesome! having virtualenv from pip is a pain since it conflicts with distro stuff | 09:24 |
| *** cjloader has joined #openstack-ansible | 09:27 | |
| openstackgerrit | Merged openstack/openstack-ansible-lxc_hosts master: making lxc image prestage timeout configurable https://review.openstack.org/571405 | 09:29 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-lxc_container_create master: SUSE: Add support for openSUSE Leap 15 https://review.openstack.org/574717 | 09:31 |
| *** cjloader has quit IRC | 09:32 | |
| evrardjp | hwoarang: good idea | 09:32 |
| evrardjp | it would simplify things a lot, not having through software collections too | 09:32 |
| * hwoarang trusts that mnaser will make it happen and life will be great from now on | 09:33 | |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible master: Do not install tmux https://review.openstack.org/575706 | 09:39 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-tests master: get-ansible-role-requirements: Use refspec when outside of OpenStack CI https://review.openstack.org/575708 | 09:46 |
| *** cshen has quit IRC | 09:47 | |
| *** cshen has joined #openstack-ansible | 09:57 | |
| openstackgerrit | Merged openstack/openstack-ansible stable/pike: Update the output for "openstack floating" command https://review.openstack.org/574942 | 10:01 |
| *** cshen has quit IRC | 10:05 | |
| mbuil | guys, what to do if you request more info to solve a bug but nobody replies? https://bugs.launchpad.net/openstack-ansible/+bug/1775315?comments=all I'd send an email to the guy reporting the bug but he did not register any mail | 10:13 |
| openstack | Launchpad bug 1775315 in openstack-ansible "ODL : ERROR networking_odl.common.odl_features ConnectionError: HTTPConnectionPool" [Undecided,Incomplete] - Assigned to Manuel Buil (mbuil) | 10:13 |
| *** mma has joined #openstack-ansible | 10:13 | |
| *** cristicalin has joined #openstack-ansible | 10:17 | |
| *** vnogin has quit IRC | 10:18 | |
| *** markvoelker has joined #openstack-ansible | 10:18 | |
| *** cristicalin has quit IRC | 10:25 | |
| hwoarang | mbuil: just give it sometime | 10:33 |
| hwoarang | leave it as incomplete for now | 10:33 |
| *** cronix has joined #openstack-ansible | 10:37 | |
| *** sxc731 has quit IRC | 10:41 | |
| mbuil | hwoarang: ok! | 10:45 |
| *** markvoelker has quit IRC | 10:51 | |
| *** pfb291 has joined #openstack-ansible | 10:55 | |
| *** JamesZhu has quit IRC | 11:00 | |
| *** JamesZhu has joined #openstack-ansible | 11:01 | |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-os_nova master: Add support for using distribution packages for OpenStack services https://review.openstack.org/567530 | 11:03 |
| *** yolanda has quit IRC | 11:03 | |
| *** yolanda has joined #openstack-ansible | 11:06 | |
| *** cshen has joined #openstack-ansible | 11:06 | |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible master: Allow a bypass of operating system https://review.openstack.org/575715 | 11:11 |
| mnaser | hwoarang: i can try and see what i can do about it :) | 11:26 |
| * hwoarang is looking forward to it | 11:27 | |
| mnaser | "Install packages required to build neutron python package" we should add retries tot hat | 11:29 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-tests master: Remove all non-voting jobs https://review.openstack.org/575303 | 11:33 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-tests master: Add retries to install packages required for Neutron https://review.openstack.org/575718 | 11:33 |
| mnaser | ^ there we go, that will take care of a random failure i just saw on os_neutron | 11:34 |
| *** vurtz has quit IRC | 11:35 | |
| *** rodolof has joined #openstack-ansible | 11:38 | |
| *** mma has quit IRC | 11:41 | |
| *** markvoelker has joined #openstack-ansible | 11:48 | |
| *** sxc731 has joined #openstack-ansible | 11:48 | |
| *** vurtz has joined #openstack-ansible | 11:49 | |
| *** markvoelker has quit IRC | 12:03 | |
| *** markvoelker has joined #openstack-ansible | 12:03 | |
| *** mma has joined #openstack-ansible | 12:05 | |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_nova master: Switch non-voting jobs to use Zuul stanza https://review.openstack.org/575286 | 12:31 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_nova master: Drop usage of nova_spice_distro_packages in the role https://review.openstack.org/575723 | 12:31 |
| *** cristicalin has joined #openstack-ansible | 12:31 | |
| mnaser | hwoarang: the `nova_spice_distro_packages` might affect your current packaging work so i'd like you to comment on that | 12:32 |
| evrardjp | cmart: sorry I wasn't | 12:32 |
| hwoarang | i will try to look at it later on | 12:32 |
| mnaser | cool, no rush :) | 12:32 |
| *** cristicalin has quit IRC | 12:33 | |
| evrardjp | mnaser: probably worth adding a lint test | 12:33 |
| mnaser | evrardjp: that would do what exactly? | 12:33 |
| evrardjp | check if package installation has always retries? | 12:34 |
| mnaser | ooh that's an interesting one | 12:34 |
| evrardjp | I can walk you through it | 12:34 |
| evrardjp | if you're interested | 12:35 |
| evrardjp | you'll break stuff though. | 12:35 |
| evrardjp | : ) | 12:35 |
| evrardjp | but it's for a better good. | 12:35 |
| mgariepy | morning everyone | 12:37 |
| *** chhagarw has joined #openstack-ansible | 12:37 | |
| *** chhavi__ has joined #openstack-ansible | 12:38 | |
| mnaser | evrardjp: wrote it, just testing locally and ill push | 12:46 |
| mnaser | yes it'll break stuff but it'll be quick easy fixes and help us improve stability | 12:46 |
| *** pfb291 has quit IRC | 12:50 | |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-tests master: Remove all non-voting jobs https://review.openstack.org/575303 | 12:50 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-tests master: Add retry install packages linter https://review.openstack.org/575727 | 12:50 |
| evrardjp | mnaser: tbh I haven't seen much of those issues. | 12:52 |
| *** pfb291 has joined #openstack-ansible | 12:52 | |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-os_nova master: Add support for using distribution packages for OpenStack services https://review.openstack.org/567530 | 12:54 |
| mnaser | evrardjp: doesnt hurt | 12:54 |
| mnaser | oh nice | 12:55 |
| mnaser | the gate is moving smoothly today | 12:55 |
| mnaser | ok looks like implied role tests are failing | 12:56 |
| mnaser | ok we need to bump our tempest sha | 12:58 |
| mnaser | https://github.com/openstack/tempest/commit/7d2b636a30057ed8db8cfd4fe2248f509b3570f1#diff-a147b612e7ea0662bce254e8fbbf0e92 | 12:58 |
| ansmith | evrardjp: hi, having issue with fetching patchset contents for ansible-role-qdrouterd | 12:58 |
| ansmith | evrardjp: e.g. git review -d 575505 | 12:59 |
| mnaser | nvm, this was fixed in https://review.openstack.org/#/c/574149/ .. 10 minutes after my change T_T | 12:59 |
| evrardjp | ansmith: if you're using git-review it's normal | 12:59 |
| evrardjp | ahah | 12:59 |
| evrardjp | ansmith: yeah | 12:59 |
| evrardjp | I fixed it in my patch | 12:59 |
| evrardjp | when we'll have that fixed, it will work | 13:00 |
| evrardjp | ansmith: let's see what the gates gives | 13:01 |
| evrardjp | after that, you can just git pull and everything will work great again | 13:01 |
| ansmith | evrardjp: need to add the doc dir to make sphinx happy next... | 13:02 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-tests master: Add retry install packages linter https://review.openstack.org/575727 | 13:08 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-tests master: Remove all non-voting jobs https://review.openstack.org/575303 | 13:09 |
| *** dave-mccowan has joined #openstack-ansible | 13:11 | |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-lxc_hosts master: SUSE: Add support for openSUSE Leap 15 https://review.openstack.org/573176 | 13:14 |
| *** mma has quit IRC | 13:15 | |
| *** dave-mcc_ has joined #openstack-ansible | 13:15 | |
| *** dave-mccowan has quit IRC | 13:16 | |
| mnaser | so who can help me debug the ceph no space on device issue | 13:21 |
| mnaser | i have a vm right now that was held to me by infra | 13:21 |
| mnaser | not sure what next steps are | 13:21 |
| hwoarang | your best bet is prob cloudnull | 13:21 |
| *** sxc731 has quit IRC | 13:25 | |
| *** tux_ has joined #openstack-ansible | 13:27 | |
| cloudnull | I'd be happy to pop on to the machine. | 13:28 |
| mnaser | cloudnull: pub keys? | 13:28 |
| *** blinkiz has quit IRC | 13:28 | |
| cloudnull | https://github.com/cloudnull.keys | 13:29 |
| mnaser | cloudnull: root@146.20.110.218 | 13:29 |
| mnaser | cloudnull: only thing i did was try to run `/opt/venv-build-script.sh /var/www/repo/os-releases/18.0.0.0b3/ubuntu-16.04-x86_64/venv-build-options-os_rally.txt` again manually and it built fine | 13:30 |
| *** vurtz has quit IRC | 13:31 | |
| cloudnull | Im in and started a tmux session | 13:31 |
| *** cmart has joined #openstack-ansible | 13:32 | |
| *** throwsb1 has joined #openstack-ansible | 13:33 | |
| *** vurtz has joined #openstack-ansible | 13:38 | |
| *** dave-mcc_ has quit IRC | 13:39 | |
| cloudnull | mnaser it's not exhibiting the same issue. | 13:40 |
| cloudnull | the metadata is free and clear, the balance looks good, free space is fine | 13:40 |
| mnaser | cloudnull: do you wanna re-run the aio and see if it fails again? | 13:44 |
| openstackgerrit | Merged openstack/openstack-ansible-os_keystone master: Switch non-voting jobs to use Zuul stanza https://review.openstack.org/575292 | 13:47 |
| *** lbragstad is now known as elbragstad | 13:48 | |
| cloudnull | sorry watching world cup | 13:50 |
| cloudnull | uruguay just had a spectacular goal | 13:50 |
| * cloudnull back to cloud | 13:50 | |
| hwoarang | ;p | 13:50 |
| cloudnull | mnaser sure I'll start running the aio again from the start and see if we can monitor for the failure | 13:51 |
| hwoarang | cloudnull: could i pick your brain for a sec? | 13:51 |
| mgariepy | mnaser, mind if I edit this patch to test mod_proxy_uwsgi in openstack-ansible-uw_apache-centos-7-nv ? https://review.openstack.org/#/c/575309 | 13:51 |
| *** TxGirlGeek has joined #openstack-ansible | 13:51 | |
| cloudnull | sure whats up ? | 13:54 |
| hwoarang | cloudnull: i am having some troubles with lxc and leap15 | 13:55 |
| hwoarang | i was hoping you could suggest something based on that log http://logs.openstack.org/17/574717/2/check/openstack-ansible-dir-opensuse-150/e8b2880/logs/host/lxc/lxc-container1.log.txt.gz | 13:55 |
| hwoarang | (apparmor is off) | 13:55 |
| hwoarang | i think the prob is this line lxc_conf - conf.c:lxc_setup_rootfs:1254 - Failed to mount rootfs "/var/lib/lxc/container1/rootfs" onto "/usr/lib64/lxc/rootfs" with options "(null)". | 13:56 |
| *** tux_ has quit IRC | 13:58 | |
| mnaser | mgariepy: yes go for it :D | 13:59 |
| openstackgerrit | Marc Gariépy (mgariepy) proposed openstack/openstack-ansible-os_keystone master: [DNM] test staging uwsgi pkgs from rdo https://review.openstack.org/575309 | 14:01 |
| cloudnull | hogepodge I've never seen that error before ? | 14:04 |
| cloudnull | opps. | 14:04 |
| cloudnull | hwoarang | 14:04 |
| hwoarang | hmm | 14:04 |
| hwoarang | ok then | 14:04 |
| cloudnull | are we consistently seeing that in the leap15 job ? | 14:04 |
| hwoarang | yep | 14:05 |
| hwoarang | for all backends | 14:05 |
| hwoarang | easily reproducible on vagrant too | 14:05 |
| * cloudnull goes to add leap15 to my local cloud to test. | 14:05 | |
| cloudnull | if you reproduce it in vegrant, can you get any of the containrs started? | 14:05 |
| hwoarang | nope | 14:06 |
| hwoarang | same version (2.0.9) works on 42.3. so it must be a config fallout somewhere. | 14:06 |
| *** rodolof has quit IRC | 14:08 | |
| *** rodolof has joined #openstack-ansible | 14:09 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible master: Removed conditional in test setup https://review.openstack.org/575748 | 14:12 |
| cloudnull | hwoarang is the underlying libs for lxc the same | 14:12 |
| cloudnull | maybe its pulling in some lxc3 parser? | 14:13 |
| hwoarang | nope all lxc2 | 14:15 |
| *** dave-mccowan has joined #openstack-ansible | 14:15 | |
| cloudnull | any known patches leap15 is pulling into lxc2 ? | 14:17 |
| cloudnull | also can we set a couple mount options in config ? | 14:18 |
| openstackgerrit | Merged openstack/openstack-ansible-os_designate master: Add release notes link to README https://review.openstack.org/575666 | 14:18 |
| openstackgerrit | Merged openstack/openstack-ansible-os_monasca master: Add release notes link to README https://review.openstack.org/575667 | 14:19 |
| openstackgerrit | Merged openstack/openstack-ansible-os_masakari master: Update README.rst information https://review.openstack.org/575662 | 14:22 |
| *** rodolof has quit IRC | 14:24 | |
| *** cjloader has joined #openstack-ansible | 14:28 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Add GeoIP capabilities into logstash https://review.openstack.org/575755 | 14:29 |
| *** sxc731 has joined #openstack-ansible | 14:30 | |
| *** tux_ has joined #openstack-ansible | 14:31 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-ops master: MNAIO: Add access to libvirt v4 via UCA Queens https://review.openstack.org/575757 | 14:31 |
| *** tux__ has joined #openstack-ansible | 14:32 | |
| *** nurdie has joined #openstack-ansible | 14:32 | |
| *** cjloader has quit IRC | 14:32 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-ops master: MNAIO: Add access to libvirt v4 via UCA Queens https://review.openstack.org/575757 | 14:33 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-ops master: MNAIO: Add access to libvirt v4 via UCA Queens https://review.openstack.org/575757 | 14:34 |
| *** tux_ has quit IRC | 14:35 | |
| *** rpittau has quit IRC | 14:36 | |
| *** cjloader has joined #openstack-ansible | 14:36 | |
| *** throwsb1 has quit IRC | 14:42 | |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-os_nova master: Add support for using distribution packages for OpenStack services https://review.openstack.org/567530 | 14:42 |
| openstackgerrit | Merged openstack/openstack-ansible-memcached_server master: Update links information in README.rst https://review.openstack.org/575678 | 14:42 |
| openstackgerrit | Merged openstack/openstack-ansible-os_magnum master: Add release notes link to README https://review.openstack.org/575674 | 14:42 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-tests master: Add retry install packages linter https://review.openstack.org/575727 | 14:49 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-tests master: Remove all non-voting jobs https://review.openstack.org/575303 | 14:49 |
| *** sxc731 has quit IRC | 14:50 | |
| *** cshen has quit IRC | 14:50 | |
| *** throwsb1 has joined #openstack-ansible | 14:50 | |
| cloudnull | mnaser im still tinkering with that node | 14:52 |
| cloudnull | however I cant get it to have the issue again | 14:52 |
| mnaser | cloudnull: thats so weird, ugh | 14:54 |
| cloudnull | giving it a go again | 14:54 |
| *** cjloader_ has joined #openstack-ansible | 14:58 | |
| *** cjloader has quit IRC | 14:58 | |
| *** chhavi__ has quit IRC | 14:59 | |
| *** cjloader_ has quit IRC | 14:59 | |
| *** cjloader has joined #openstack-ansible | 14:59 | |
| throwsb1 | Hey, just checking if anyone else is seeing haproxy galera health check issues. If I have the healthcheck enabled, haproxy is getting a connection error and backend is shutdown. | 15:00 |
| *** pcaruana has quit IRC | 15:01 | |
| throwsb1 | If I disable, I can connect through external vip, This is a a fresh install of queens with latest and fresh ubuntu 16.04.4 install and patch. So far nothing is standing out. | 15:02 |
| *** canori01 has joined #openstack-ansible | 15:05 | |
| *** sxc731 has joined #openstack-ansible | 15:07 | |
| cloudnull | throwsb1 if you curl from the same node as the load balancer to the galera nodes on port 9200 does it return ok? | 15:08 |
| throwsb1 | let me chech | 15:09 |
| cloudnull | there is a whitelist option for the health check, which limits what hosts can connect to the galera node. | 15:10 |
| *** rmart04 has quit IRC | 15:10 | |
| cloudnull | if you cant connect to the galera node on port 9200 I suspect the whitelist might be too limited ? | 15:10 |
| throwsb1 | looks like that might be the case | 15:14 |
| throwsb1 | although I am running an other queens env with 1 or 2 minor revisions back with out issues | 15:14 |
| throwsb1 | is it modifying just the galera whithelist? | 15:15 |
| throwsb1 | cloudnull: I am guessing you meant running curl <IP>:3306 to get something back from galera | 15:20 |
| throwsb1 | to fe vip, connection refused, internal vip I get curl empty reply, curl to galera container I get http response | 15:20 |
| *** mbuil has quit IRC | 15:21 | |
| cloudnull | its `curl IP:9200` | 15:22 |
| cloudnull | that should return something to the effect that the cluster in sync, etc | 15:22 |
| *** jeferobles has joined #openstack-ansible | 15:23 | |
| throwsb1 | I get nothing nor is anything listening on 9200 | 15:23 |
| *** pvradu has quit IRC | 15:25 | |
| *** cjloader has quit IRC | 15:26 | |
| *** cjloader has joined #openstack-ansible | 15:26 | |
| cloudnull | on the galera node the xinetd process should be running and responding on 9200 | 15:27 |
| *** cjloader has quit IRC | 15:28 | |
| *** sxc731 has quit IRC | 15:29 | |
| *** cjloader has joined #openstack-ansible | 15:29 | |
| jamesdenton | >> only_from = 172.29.237.62 172.29.236.100 127.0.0.1 | 15:30 |
| jamesdenton | perhaps the host is using something other than .100. I manually configured my IPs | 15:30 |
| *** dxiri has joined #openstack-ansible | 15:30 | |
| jamesdenton | that's in /etc/xinetd.d/mysqlchk | 15:31 |
| throwsb1 | cloudnull: yup I just got it. :) on env not working I am curling from infra to ip of galera and getting conenction reset from peer. container does not have curl | 15:31 |
| throwsb1 | on working cluster env, did the same test and nothing comes back. meaning looks like it connected | 15:31 |
| cloudnull | we can make that whitelist more permissive. | 15:32 |
| cloudnull | also that whitelist can be overridden | 15:32 |
| openstackgerrit | David Wilde proposed openstack/openstack-ansible-ops master: Increase VM SSH timeouts https://review.openstack.org/575772 | 15:35 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-ops master: Use file instead of command for disk image clean-up https://review.openstack.org/575777 | 15:44 |
| throwsb1 | cloudnull: whitelist worked. is there a way to make it perm if I redeploy? | 15:44 |
| throwsb1 | jamesdenton: not sure if this will help, but in my env, my mgmt and vlan are on the same network. so I just added the vlan to the whitelist. | 15:46 |
| jamesdenton | the whitelist in the mysqlchk file? | 15:46 |
| throwsb1 | yes | 15:46 |
| jamesdenton | cool | 15:46 |
| throwsb1 | then restarted xinetd | 15:47 |
| throwsb1 | hoping there is a way to make perm | 15:47 |
| cloudnull | throwsb1 https://github.com/openstack/openstack-ansible-galera_server/blob/master/defaults/main.yml#L64-L74 | 15:47 |
| cloudnull | you can set this in one of your user variable files | 15:48 |
| *** armaan has joined #openstack-ansible | 15:48 | |
| throwsb1 | sweet, thanks!!! | 15:48 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-ops master: MNAIO: Use file instead of command for disk image clean-up https://review.openstack.org/575777 | 15:48 |
| *** gyee has joined #openstack-ansible | 15:52 | |
| *** radeks has joined #openstack-ansible | 15:53 | |
| tux__ | Folks, how does i tell systemd-networkd to send DHCP_HOSTNAME to my dhcp server? | 15:54 |
| tux__ | I have multiple VLAN and i want to set two different hostname to register on DDNS | 15:55 |
| tux__ | i have two domain vlan10.example.com and vlan20.example.com so i want both my Linux nic send relevant hostname to dhcp server to register their hostname | 15:56 |
| *** sxc731 has joined #openstack-ansible | 15:59 | |
| tux__ | hmm i think SendHostname= | 15:59 |
| *** cjloader_ has joined #openstack-ansible | 16:01 | |
| *** cjloader has quit IRC | 16:01 | |
| *** cjloader_ has quit IRC | 16:01 | |
| *** cjloader has joined #openstack-ansible | 16:02 | |
| *** dave-mccowan has quit IRC | 16:06 | |
| openstackgerrit | Merged openstack/openstack-ansible master: Do not install tmux https://review.openstack.org/575706 | 16:06 |
| *** jeferobles has quit IRC | 16:17 | |
| CobHead | Anyone had any success upgrading an Ocata Openstack installation with Openstack Ansible? | 16:17 |
| CobHead | Will it break in new and exciting ways? | 16:18 |
| *** nattanon has joined #openstack-ansible | 16:19 | |
| *** sxc731 has quit IRC | 16:19 | |
| cloudnull | CobHead i've no ocata in prod, I cant really comment, however I think mhayden did an ocata to pike upgrade? | 16:20 |
| CobHead | Hm, I would love to believe that the upgrade process is painless, but I very much doubt so | 16:20 |
| *** Bhujay has quit IRC | 16:20 | |
| *** cjloader has quit IRC | 16:21 | |
| *** cjloader has joined #openstack-ansible | 16:22 | |
| nattanon | cloudnull: Do you ever met that console on horizon do everything twice just like I'm type 'a' the output on console is 'aa'. | 16:22 |
| cloudnull | nattanon I've not had that issue. | 16:22 |
| cloudnull | I'm using novnc | 16:22 |
| *** armaan has quit IRC | 16:23 | |
| CobHead | Yeah, I've had that happen, but that can happen in any traditional console. | 16:23 |
| *** epalper has quit IRC | 16:23 | |
| CobHead | You could try restarting the nova console service. | 16:23 |
| CobHead | You use spice? | 16:23 |
| nattanon | CobHead: Unfortunatly, I'm try all the thing all service related like nova-consoleauth, nova-novncproxy, nova-compute. | 16:26 |
| *** cjloader has quit IRC | 16:26 | |
| nattanon | CobHead: I'm using novnc. | 16:26 |
| CobHead | Is there a lot of latency between your network and the Openstack installation? | 16:28 |
| CobHead | If the ack was not retrieved within x, it will probably resend the packet again, and in the meantime while waiting for the second ack, it will get the first ack. | 16:29 |
| *** vurtz has quit IRC | 16:31 | |
| *** maybebuggy has quit IRC | 16:33 | |
| nattanon | CobHead: latency is too low. More info i got that novnc open 2 socket per console if i clear all the cache it will be normal for a while then the second connection is established everything will be twice. | 16:34 |
| CobHead | Hmm. I can't say for sure as I don't use novnc in my env. | 16:35 |
| CobHead | I use Spice, but I seldom use the console via Horizon | 16:35 |
| *** vurtz has joined #openstack-ansible | 16:37 | |
| nattanon | CobHead: Thank for your trying. Three day still confused. | 16:39 |
| CobHead | I'll keep an eye out in my Openstack travels for the same issue | 16:40 |
| *** poul_ has quit IRC | 16:41 | |
| cloudnull | mnaser IDK about the full fs issue. I have another pr which might help. | 16:41 |
| nattanon | cloudnull: Did your novnc baseurl in nova.conf use /vnc_lite.html or /vnc_auto.html | 16:42 |
| cloudnull | though i can reproduce it on that node. | 16:42 |
| cloudnull | mnaser I have it all still running in tmux on that node if you want to poke around | 16:42 |
| *** gkadam has quit IRC | 16:43 | |
| *** hamzy_ has joined #openstack-ansible | 16:48 | |
| *** hamzy has quit IRC | 16:51 | |
| *** cjloader has joined #openstack-ansible | 16:52 | |
| openstackgerrit | Merged openstack/openstack-ansible-ops master: MNAIO: Use file instead of command for disk image clean-up https://review.openstack.org/575777 | 17:00 |
| *** jra has quit IRC | 17:02 | |
| *** jra has joined #openstack-ansible | 17:02 | |
| *** yiorgos_272 has quit IRC | 17:11 | |
| *** tux__ has quit IRC | 17:15 | |
| *** tux_ has joined #openstack-ansible | 17:15 | |
| *** epalper has joined #openstack-ansible | 17:16 | |
| jrosser | cloudnull: if you want to try a working horizon profiler you need https://review.openstack.org/#/c/575793/, https://review.openstack.org/#/c/575791/ and https://github.com/openstack/osprofiler/commit/ab01236a6918fb803aac854199425a7aef70f27b | 17:21 |
| jrosser | some nice detective work from stuartgr1 there | 17:21 |
| *** radeks has quit IRC | 17:21 | |
| tux_ | cloudnull: Question, is it possible i create BOND + VLAN and doesn't create bridge interface ? | 17:26 |
| tux_ | in systemd-networkd i can tell it create vlan interface name br-mgmt | 17:27 |
| tux_ | do you think it will create any issue? | 17:27 |
| tux_ | anybody out there who can answer my question? | 17:28 |
| *** dave-mccowan has joined #openstack-ansible | 17:33 | |
| openstackgerrit | Merged openstack/openstack-ansible-ops master: MNAIO: Add access to libvirt v4 via UCA Queens https://review.openstack.org/575757 | 17:33 |
| *** chhagarw has quit IRC | 17:33 | |
| tux_ | openstackgerrit: any thought ? | 17:34 |
| tux_ | i am kind of stuck here | 17:34 |
| tux_ | want to know should i create bridge interface of just keep my vlan name to br-mgmt, br-vxlan etc.. | 17:34 |
| canori01 | You'll need the bridges as they are needed by the containers | 17:37 |
| *** poul has joined #openstack-ansible | 17:37 | |
| *** epalper has quit IRC | 17:37 | |
| tux_ | I understand openstack-ansible read interface name... but what if i create multiple VLAN interface and name them br-mgmt, br-vxlan etc... why they have to be bridge? | 17:38 |
| tux_ | Do you following what i am trying to explain? | 17:39 |
| tux_ | canori01: ? | 17:39 |
| canori01 | They have to be bridges so that you can give the lxc's layer2 to your physical network. Bonds don't have that feature | 17:40 |
| canori01 | You can assign your vlan tagged bond to the bridge though | 17:41 |
| *** gkadam has joined #openstack-ansible | 17:44 | |
| tux_ | That make sense lxc need bridge interface to reach out.. but that is just for br-mgmt not br-vxlan, be-storage etc.. | 17:45 |
| cloudnull | Tux_ I'm afk but yes you can create vlan tagged interfaces from the bond without a bridge. | 17:45 |
| tux_ | cloudnull: perfect!! so i don't need to create bridge for br-mgmt too right? | 17:46 |
| cloudnull | Something like bon0.xxx | 17:46 |
| cloudnull | Which would then be plugged into a bridge or ip'd | 17:46 |
| *** sgrasley has quit IRC | 17:46 | |
| tux_ | I am going to set my vlan.X name to br-mgmt | 17:46 |
| *** sgrasley has joined #openstack-ansible | 17:46 | |
| tux_ | without setting up bridge | 17:46 |
| cloudnull | Yes. I'd create the vlan tagged interfaces and then add it to the bridge. | 17:47 |
| cloudnull | If needed. | 17:47 |
| tux_ | if i don't want to add in bridge does that will create any issue? | 17:47 |
| cloudnull | You'll need br-mgmt for osa , but how you get there is up to you. | 17:47 |
| tux_ | OSA only care about br-XXXX name which is hard coded in playbook | 17:48 |
| cloudnull | Yes. Those bridges are defined in the openstack_use_config | 17:48 |
| cloudnull | https://review.openstack.org/#/c/553636/ | 17:49 |
| tux_ | I think i got it, i was just trying to understand by we need bridge if i just create bunch of vlan interface and name it br-XXXX | 17:49 |
| cloudnull | That's a good example on creating all the networks using networkd and dummy interfaces | 17:50 |
| cloudnull | Containers will plugin to the various bridges. | 17:50 |
| cloudnull | Using veth pairs. | 17:51 |
| tux_ | let me understand config | 17:52 |
| tux_ | what is the use of br-vlan-veth here? | 17:59 |
| tux_ | why there are veth ? | 17:59 |
| tux_ | i am planning to setup network without veth, so i will map my number of vlan with br-xxx nic | 18:02 |
| ansmith | cloudnull, mnaser, evrardjp: if you have a moment, could you swing by https://review.openstack.org/#/c/575537 and https://review.openstack.org/#/c/575505? | 18:04 |
| ansmith | should be a good base to focus on the functional tests | 18:05 |
| *** electrofelix has quit IRC | 18:05 | |
| *** nattanon has quit IRC | 18:17 | |
| *** gkadam_ has joined #openstack-ansible | 18:33 | |
| *** gkadam has quit IRC | 18:37 | |
| *** poul_ has joined #openstack-ansible | 18:40 | |
| *** poul has quit IRC | 18:41 | |
| *** poul_ has quit IRC | 18:46 | |
| ivve | not really related to osa but anyone ever experienced keepalived totally broked out dumping "Netlink: filter function error" all the time? tcpdumped the vrrp traffic seems fine but keeps going into this state on all controllers | 18:48 |
| ivve | borked* | 18:48 |
| *** radeks has joined #openstack-ansible | 18:52 | |
| *** kaiokmo has quit IRC | 18:55 | |
| cloudnull | Tux_ the containers will automatically plug into the bridges as they come up and that is using a veth pair. | 18:57 |
| tux_ | In my lab machine i didn't do anything fancy like veth etc.. and everything works fine and my openstack is up and running.. that is why i surprised | 18:58 |
| cloudnull | ivve ive had random issues with keepalived in the past. they've always been random transient issues , so nothing that has resulted in it being totally broked | 18:58 |
| ivve | :D | 18:59 |
| tux_ | Do you mind sending list of file in your /etc/systemd/network dir? | 18:59 |
| mnaser | cloudnull: i havent dug into that, been busy today but ill try to see if i can see something, but it completed a full run i guess? | 18:59 |
| tux_ | cat * /etc/systemd/network/* | 18:59 |
| cloudnull | tux_ nothing for you to do with the veth-pairs. its an automatic process | 18:59 |
| *** radeks_ has joined #openstack-ansible | 18:59 | |
| tux_ | oh!!! | 18:59 |
| *** poul has joined #openstack-ansible | 18:59 | |
| cloudnull | the containers will plugin into the bridges as needed | 18:59 |
| tux_ | that is why... i am confused | 19:00 |
| ivve | its really wierd but im not sure routing br-mgmt & tenant networks is such a good idea | 19:00 |
| tux_ | now i am good | 19:00 |
| ivve | this feels like a network issue of some kind | 19:00 |
| ivve | keepalived vrrp traffic seems to be passing between hosts np | 19:00 |
| tux_ | when i saw your github link in networkd playbook bunch of time veth so i was wonder why did we using veth in networkd? | 19:00 |
| ivve | but thats basically the only thing thats broken | 19:00 |
| jrosser | tux_: it is used to hook up eth12 for example | 19:01 |
| ivve | and im ofc suspecting network since tenant IP's are not all reachable... its totally random but ~30% of ip's that are either flips or assigned to instances aren't reachable but can be seen in the qrouter namespace | 19:01 |
| *** radeks has quit IRC | 19:01 | |
| cloudnull | ivve something broken in path ? | 19:02 |
| cloudnull | I assume neutron agents are in containers | 19:02 |
| ivve | also linklocals between instances & metadata services gets no route to host, SOMETIMES! | 19:02 |
| ivve | yes | 19:02 |
| jrosser | tux_: in some cases neutron *requires* an interface, not a bridge, so a new interface eth12 is created for that purpose and wired down to the bridge with a veth | 19:02 |
| cloudnull | on the host run lxc-veth-check | 19:02 |
| *** poul has quit IRC | 19:03 | |
| cloudnull | if someone ifup/down a bridge it could create a situation like that | 19:03 |
| ivve | 2.0.8 | 19:03 |
| *** poul has joined #openstack-ansible | 19:03 | |
| cloudnull | if it didn't kickout a mess of notifications then there's nothing wrong there/ | 19:03 |
| cloudnull | brctl show $bridge_name | 19:03 |
| cloudnull | does that should the ethernet device in the list? | 19:04 |
| tux_ | jrosser: let me tell you what i am doing.. I have 2 NIC and i created bond0 (active-backup), on top of bond0 i am creating all br-xxx interface | 19:04 |
| *** hassenius has quit IRC | 19:04 | |
| ivve | cloudnull: it does show the bond0 that its supposed to be connected with as well as the container veths | 19:05 |
| cloudnull | thats good, so then nothing likely busted from the host to the container. | 19:05 |
| cloudnull | in the container the links are up and functional ? | 19:06 |
| ivve | yea the environment works perfectly a part from some instances not getting route to metadata service every now and then | 19:06 |
| ivve | and ofc all keepalived are broken in the sense that they all report Keepalived_healthcheckers[345712]: Netlink: filter function error, Keepalived_vrrp[345713]: Netlink: filter function error | 19:07 |
| cloudnull | welp then the fun begins tracking down what transient problem is causing that to happen :) | 19:07 |
| ivve | all containers and services seem to work | 19:07 |
| ivve | the only thing that really stands out is keepalived | 19:07 |
| ivve | haproxy works, rabbit ok | 19:08 |
| cloudnull | are you seeing kernel stacktraces in dmesg or other suspicious traffic drops elsewhere? | 19:08 |
| ivve | no stack traces | 19:08 |
| cloudnull | MtU is all >= 1500 | 19:08 |
| ivve | yes | 19:08 |
| ivve | just the stuff from keepalived | 19:08 |
| cloudnull | error rate on the host interface normal ? | 19:08 |
| ivve | TX packets:92767500 errors:0 dropped:12 overruns:0 carrier:0 | 19:09 |
| ivve | RX packets:88402919 errors:0 dropped:0 overruns:0 frame:0 | 19:09 |
| cloudnull | that looks good to me :) | 19:09 |
| cloudnull | maybe its a contention issue? | 19:09 |
| cloudnull | just too much IO | 19:09 |
| cloudnull | ? | 19:09 |
| ivve | me & a guy are working our assses off on this env and can't just figure it out | 19:09 |
| ivve | nope its a fresh deploy | 19:10 |
| ivve | more or less | 19:10 |
| ivve | some test instances going on | 19:10 |
| cloudnull | some upstream filtering taking place? | 19:10 |
| ivve | clarify? | 19:10 |
| cloudnull | something misconfigured on one of the switches? | 19:11 |
| ivve | thats my bet but network guy says no :P | 19:11 |
| ivve | and switches are too advanced for me, arista | 19:12 |
| ivve | but i basically asked about multicast traffic, there isn't much going on afaik more than few vlans & vrf | 19:13 |
| *** brad[] has joined #openstack-ansible | 19:13 | |
| ivve | switches are stacked.. (mlag) | 19:13 |
| ivve | lacp on the bonds | 19:13 |
| ivve | but nothing exceptional other than that | 19:14 |
| cloudnull | hashing policy on the host being defined ? | 19:14 |
| ivve | yes | 19:14 |
| ivve | bond-xmit-hash-policy layer2 | 19:14 |
| ivve | fast | 19:14 |
| jrosser | igmp querier? | 19:14 |
| ivve | pass | 19:15 |
| ivve | :D | 19:15 |
| jrosser | ivve: this is keepalived on the haproxy nodes? | 19:16 |
| ivve | ah got it jrosser | 19:16 |
| ivve | im not sure it is | 19:16 |
| ivve | but yes the keepalived is on haproxy nodes on bare yes | 19:17 |
| jrosser | right so i run something practically identical but with cisco | 19:17 |
| jrosser | and thats ok | 19:17 |
| ivve | we used fallback for booting and used maas, that was a nightmare to get going | 19:17 |
| ivve | so these switches scares me :D apart from the price ofc | 19:18 |
| jrosser | 2 switches, mode4 vpc, and i pxeboot over the vpc too | 19:18 |
| jrosser | which was a total nightmare | 19:18 |
| ivve | can relate | 19:18 |
| ivve | but thats working now, learned a lot about that | 19:18 |
| jrosser | however i've not had a problem with keepalived/haproxy | 19:20 |
| ivve | hmm what else, using two bonds one for br-mgmt & br-storage and other bond for vxlan & vlan | 19:20 |
| jrosser | buy you mentioned instances getting to metadata service, isnt that something else entirely? | 19:20 |
| ivve | im uploading a new image to make sure now | 19:20 |
| ivve | its nothing with that | 19:20 |
| ivve | it was a qcow with ceph which should be raw | 19:20 |
| ivve | so im fixing this then try to deploy 10x again, usually 1-3 fails basically they don't connect with link-local and fail networking | 19:21 |
| ivve | ill get proper logs pastebin | 19:21 |
| jrosser | do you have more than just the two switches? | 19:22 |
| ivve | also, if i restart keepalived, its all fine if no instances are deployed. deploy instances & keepalived goes to the bin | 19:22 |
| ivve | however the VIP stays, so all services continue to function | 19:22 |
| ivve | well yes, customers networks are connected and routed | 19:23 |
| jrosser | and any vxlan? | 19:23 |
| ivve | i asked for vrrps anyways but not vxlans, im guessing id collision | 19:24 |
| ivve | since vrrp id can | 19:24 |
| jrosser | the reason i ask there is the common factor between vrrp and vxlan is multicast | 19:24 |
| ivve | mm well just the vxlan from os | 19:24 |
| jrosser | but that makes all sorts of multicast | 19:25 |
| ivve | it does | 19:25 |
| ivve | but its just the software br-vxlan | 19:26 |
| ivve | you're saying that could be it? | 19:26 |
| jrosser | well i don't know - but if starting instances breaks your vrrp then i'd certainly be looking at the multicast config as thats a common factor | 19:27 |
| jrosser | all broadcast/unknown/mcast traffic from inside each vxlan vni gets bumped onto a multicast group on the overlay network | 19:28 |
| ivve | tcpdump -n -e -i br-vxlan vrrp quiet like a grave | 19:28 |
| ivve | but both are using the same one... 224.0.0.18 | 19:28 |
| ivve | that could be an issue, i realise that | 19:29 |
| tux_ | I had same issue when i was building OSA on VMware for testing and found vSwitch was blocking multicast etc because of security, as soon as i ACCEPT all traffic it fixed all short of issue | 19:30 |
| tux_ | Make sure inside your physical network nothing else going on | 19:31 |
| ivve | i can see multicast traffic from node1 -> other ctrl nodes | 19:31 |
| ivve | on the group | 19:32 |
| ivve | that led me to believe that multicast was allowed but also asked and got a yes | 19:32 |
| ivve | wouldn't that be proof enough? | 19:32 |
| tux_ | are you using physical servers? | 19:34 |
| jrosser | i may have gone a little further, as i have my vxlan all in its own vrf and thats the only one that does multicast routing | 19:34 |
| ivve | but lets say that all these vlans are routed | 19:35 |
| jrosser | the vrrp for keepalived all happens in one l2 domain so no multicast ever needs enter/leave that vlan | 19:35 |
| ivve | would that cause issues? | 19:35 |
| jrosser | it shouldnt do, as the vxlan tunnels should be using a different multicast address to vrrp | 19:37 |
| ivve | jrosser: i was under the impression thats the deal with br-vxlan as well as it is in its own vlan | 19:37 |
| jrosser | i just think it's very suspicious that its all ok until you start an instance. just my 2p :) | 19:37 |
| ivve | ethertype IPv4 (0x0800), length 54: 169.254.192.4 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 50, authtype simple, intvl 2s, length | 19:37 |
| ivve | from the vxlan | 19:37 |
| ivve | rtype IPv4 (0x0800), length 54: 10.22.4.40 > 224.0.0.18: VRRPv2, Advertisement, vrid 10, prio 150, authtype simple, intvl 1s, length 20 | 19:38 |
| ivve | from the br-mgmt | 19:38 |
| jrosser | oh wait | 19:38 |
| ivve | and thats the link-local? :D | 19:38 |
| jrosser | do you see both of those on br-mgmt? | 19:38 |
| ivve | no separate | 19:39 |
| jrosser | oh good, as that's what i was wondering | 19:39 |
| *** hassenius has joined #openstack-ansible | 19:39 | |
| ivve | they are even on separate interface.. well bond anyway | 19:39 |
| ivve | separate vlan, but both vlans are routed | 19:40 |
| ivve | but multicast should stay in there | 19:40 |
| ivve | afaik | 19:40 |
| ivve | unless some switch magic does something or vrf | 19:40 |
| *** hassenius has quit IRC | 19:44 | |
| *** radeks_ has quit IRC | 19:45 | |
| ivve | this is wierd | 19:49 |
| ivve | the instance can reach out | 19:49 |
| ivve | say, google | 19:49 |
| ivve | but can't icmp the instance | 19:49 |
| ivve | and its 1 of 10 identical | 19:49 |
| ivve | :D | 19:49 |
| ivve | so the same sec group is applied | 19:49 |
| ivve | def networking issue | 19:50 |
| *** sxc731 has joined #openstack-ansible | 19:56 | |
| *** poul has quit IRC | 19:57 | |
| ivve | jees so strange, thanks a bunch for input guys. ill continue troubleshooting but this last thing is just mindbending | 20:03 |
| ivve | i deployed an instance, it is able to reach anything outside the provider network... but other networks routed to the provider network can't reach that specific machine.. but it can reach others from the very same tenant | 20:04 |
| *** poul has joined #openstack-ansible | 20:04 | |
| *** dxiri has quit IRC | 20:05 | |
| *** dxiri has joined #openstack-ansible | 20:08 | |
| ivve | the next instance deploy can reach the controller node and outside no problem | 20:08 |
| *** poul has quit IRC | 20:09 | |
| ivve | uhm | 20:19 |
| ivve | instances located on the controller nodes seem to be the issue | 20:19 |
| ivve | can't be migrated off controller nodes to compute nodes | 20:19 |
| *** lkoranda has quit IRC | 20:22 | |
| *** lkoranda has joined #openstack-ansible | 20:30 | |
| *** cmart has quit IRC | 20:33 | |
| *** guilhermesp has quit IRC | 20:50 | |
| *** guilhermesp has joined #openstack-ansible | 21:00 | |
| *** poul has joined #openstack-ansible | 21:05 | |
| *** gkadam_ has quit IRC | 21:05 | |
| openstackgerrit | Merged openstack/openstack-ansible-ops master: Increase VM SSH timeouts https://review.openstack.org/575772 | 21:09 |
| *** throwsb1 has quit IRC | 21:11 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible master: Set the BTRFS the mixed flag when creating the btrfs volume https://review.openstack.org/575850 | 21:13 |
| cloudnull | mnaser ^ | 21:14 |
| cloudnull | last attempt, unless someone else has ideas on how to solve this intermittent issue that only happens on the rax cloud... | 21:15 |
| cloudnull | otherwise we should circle up and get https://review.openstack.org/#/c/575033/ in | 21:15 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible master: Set the BTRFS mixed flag when creating the BTRFS volumes https://review.openstack.org/575850 | 21:16 |
| *** nurdie has quit IRC | 21:18 | |
| *** sxc731 has quit IRC | 21:18 | |
| *** nurdie has joined #openstack-ansible | 21:19 | |
| *** nurdie has quit IRC | 21:23 | |
| *** poul has quit IRC | 21:35 | |
| *** poul has joined #openstack-ansible | 21:44 | |
| *** poul has quit IRC | 21:48 | |
| *** tux_ has quit IRC | 21:55 | |
| *** poul has joined #openstack-ansible | 22:02 | |
| *** poul has quit IRC | 22:07 | |
| *** poul has joined #openstack-ansible | 22:12 | |
| *** cshen has joined #openstack-ansible | 22:15 | |
| *** poul has quit IRC | 22:17 | |
| *** hassenius has joined #openstack-ansible | 22:17 | |
| *** cshen has quit IRC | 22:19 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Break out setup and always tag facts https://review.openstack.org/575864 | 22:20 |
| *** hassenius has quit IRC | 22:21 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Break out setup and always tag facts https://review.openstack.org/575864 | 22:24 |
| *** poul has joined #openstack-ansible | 22:29 | |
| *** poul has quit IRC | 22:34 | |
| *** poul has joined #openstack-ansible | 22:47 | |
| openstackgerrit | Merged openstack/openstack-ansible-ops master: Breakup site into more consumable meta playbooks https://review.openstack.org/575609 | 22:48 |
| *** dave-mccowan has quit IRC | 22:48 | |
| openstackgerrit | Merged openstack/openstack-ansible-ops master: Add GeoIP capabilities into logstash https://review.openstack.org/575755 | 22:50 |
| *** poul has quit IRC | 22:51 | |
| *** poul has joined #openstack-ansible | 22:56 | |
| *** poul has quit IRC | 23:01 | |
| *** tux_ has joined #openstack-ansible | 23:08 | |
| *** tux_ has quit IRC | 23:12 | |
| *** poul has joined #openstack-ansible | 23:14 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Correct community beats index https://review.openstack.org/575873 | 23:15 |
| *** hamzy_ has quit IRC | 23:15 | |
| openstackgerrit | Merged openstack/openstack-ansible-ops master: Break out setup and always tag facts https://review.openstack.org/575864 | 23:18 |
| *** poul has quit IRC | 23:18 | |
| *** gyee has quit IRC | 23:24 | |
| *** hamzy has joined #openstack-ansible | 23:25 | |
| *** poul has joined #openstack-ansible | 23:32 | |
| *** poul has quit IRC | 23:36 | |
| *** hassenius has joined #openstack-ansible | 23:38 | |
| *** tosky has quit IRC | 23:41 | |
| *** hassenius has quit IRC | 23:42 | |
| *** poul has joined #openstack-ansible | 23:50 | |
| *** TxGirlGeek has quit IRC | 23:54 | |
| *** poul has quit IRC | 23:54 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!