| logan- | o/ | 00:01 |
|---|---|---|
| cloudnull | hows it ? | 00:01 |
| openstackgerrit | Merged openstack/openstack-ansible-os_rally stable/queens: Remove galera_client meta-dependency https://review.openstack.org/603768 | 00:02 |
| logan- | meh. fighting with ocata. | 00:03 |
| openstackgerrit | Merged openstack/openstack-ansible stable/queens: Move ara to scripts-library https://review.openstack.org/603697 | 00:04 |
| cloudnull | that does sound pretty awesome ! :) | 00:06 |
| *** tosky has quit IRC | 00:09 | |
| openstackgerrit | Merged openstack/openstack-ansible-ops master: Convert the curator action file into multiple files https://review.openstack.org/603849 | 00:18 |
| openstackgerrit | Merged openstack/openstack-ansible-os_ironic stable/rocky: Remove galera_client meta-dependency https://review.openstack.org/603793 | 00:28 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Update monitoring index for replicas https://review.openstack.org/603937 | 00:31 |
| odyssey4me | meh, insomnia :/ | 00:32 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Update monitoring index for replicas https://review.openstack.org/603937 | 00:32 |
| openstackgerrit | Merged openstack/openstack-ansible stable/rocky: Add nspawn to experimental jobs https://review.openstack.org/603493 | 00:33 |
| *** ullbeking has quit IRC | 00:37 | |
| *** ullbeking has joined #openstack-ansible | 00:38 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Update monitoring index for replicas https://review.openstack.org/603937 | 00:42 |
| *** gyee has quit IRC | 00:42 | |
| *** dxiri has quit IRC | 00:43 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_ironic stable/pike: Fix OpenSSL import in test config https://review.openstack.org/603939 | 00:51 |
| openstackgerrit | Merged openstack/openstack-ansible stable/rocky: Avoid using loop_var in endpoint manage https://review.openstack.org/603547 | 00:57 |
| *** FrankZhang has quit IRC | 01:00 | |
| *** ebbex has quit IRC | 01:00 | |
| *** cloudnull has quit IRC | 01:00 | |
| openstackgerrit | Merged openstack/openstack-ansible master: How to configure Telemetry for radosgw https://review.openstack.org/586063 | 01:01 |
| *** Guest58757 has joined #openstack-ansible | 01:03 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_keystone master: SUSE: Add support for openSUSE Leap 15 https://review.openstack.org/603114 | 01:21 |
| openstackgerrit | Merged openstack/openstack-ansible-os_octavia stable/pike: Remove galera_client meta-dependency https://review.openstack.org/603789 | 01:24 |
| *** mmercer has quit IRC | 01:27 | |
| *** mschuppert has quit IRC | 01:29 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_cinder stable/rocky: Add missing dep for SUSE distro installs https://review.openstack.org/603941 | 01:29 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_cinder stable/rocky: Clean up role tests https://review.openstack.org/603868 | 01:29 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_cinder stable/rocky: Remove galera_client meta-dependency https://review.openstack.org/603714 | 01:30 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_cinder stable/rocky: Remove python-keystoneclient package install https://review.openstack.org/603409 | 01:30 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_nova stable/pike: import zuul job settings from project-config https://review.openstack.org/595062 | 01:40 |
| guilhermesp | cloudnull: hey what you think https://review.openstack.org/603906 ? | 01:44 |
| *** cshen has joined #openstack-ansible | 01:46 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_cinder stable/ocata: Remove tempest testing https://review.openstack.org/603943 | 01:47 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_cinder stable/ocata: Clean up role tests https://review.openstack.org/603876 | 01:47 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_cinder stable/ocata: import zuul job settings from project-config https://review.openstack.org/594743 | 01:47 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_cinder stable/ocata: Updated from OpenStack Ansible Tests https://review.openstack.org/602302 | 01:47 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_neutron stable/pike: import zuul job settings from project-config https://review.openstack.org/595061 | 01:50 |
| *** cshen has quit IRC | 01:51 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_neutron master: switch documentation job to new PTI https://review.openstack.org/600630 | 01:57 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_octavia master: switch documentation job to new PTI https://review.openstack.org/600631 | 01:57 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_octavia master: switch documentation job to new PTI https://review.openstack.org/594685 | 01:57 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_neutron master: switch documentation job to new PTI https://review.openstack.org/594681 | 01:58 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_designate stable/queens: Pin tempest plugin https://review.openstack.org/603946 | 02:04 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_designate stable/queens: Pin tempest plugin https://review.openstack.org/603947 | 02:04 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_designate stable/pike: Pin tempest plugin https://review.openstack.org/603948 | 02:05 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_designate stable/pike: Pin tempest plugin https://review.openstack.org/603948 | 02:06 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_designate stable/ocata: Remove tempest testing https://review.openstack.org/603949 | 02:08 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_designate stable/ocata: Updated from OpenStack Ansible Tests https://review.openstack.org/602304 | 02:09 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_designate stable/ocata: import zuul job settings from project-config https://review.openstack.org/594745 | 02:09 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-nspawn_hosts master: fix tox python3 overrides https://review.openstack.org/584828 | 02:14 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_sahara stable/ocata: Remove tempest testing https://review.openstack.org/603950 | 02:17 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_sahara stable/ocata: import zuul job settings from project-config https://review.openstack.org/594759 | 02:17 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-repo_build stable/ocata: import zuul job settings from project-config https://review.openstack.org/595026 | 02:19 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_almanach stable/ocata: Do not log passwords https://review.openstack.org/603951 | 02:20 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_almanach stable/pike: Do not log passwords https://review.openstack.org/603952 | 02:21 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_almanach stable/pike: import zuul job settings from project-config https://review.openstack.org/595044 | 02:22 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_heat stable/ocata: import zuul job settings from project-config https://review.openstack.org/594749 | 02:22 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_glance stable/ocata: import zuul job settings from project-config https://review.openstack.org/594746 | 02:23 |
| *** FrankZhang has joined #openstack-ansible | 02:23 | |
| openstackgerrit | Merged openstack/ansible-hardening master: import zuul job settings from project-config https://review.openstack.org/594190 | 02:40 |
| openstackgerrit | Merged openstack/ansible-role-systemd_service stable/rocky: import zuul job settings from project-config https://review.openstack.org/595140 | 02:48 |
| openstackgerrit | Merged openstack/ansible-hardening master: switch documentation job to new PTI https://review.openstack.org/594191 | 02:51 |
| openstackgerrit | Merged openstack/ansible-hardening stable/ocata: import zuul job settings from project-config https://review.openstack.org/594726 | 02:51 |
| openstackgerrit | Merged openstack/ansible-hardening stable/pike: import zuul job settings from project-config https://review.openstack.org/595031 | 02:51 |
| openstackgerrit | Merged openstack/ansible-role-systemd_service master: import zuul job settings from project-config https://review.openstack.org/594200 | 02:54 |
| openstackgerrit | Merged openstack/ansible-role-systemd_networkd master: import zuul job settings from project-config https://review.openstack.org/594198 | 02:54 |
| openstackgerrit | Merged openstack/ansible-hardening stable/queens: import zuul job settings from project-config https://review.openstack.org/595083 | 02:56 |
| openstackgerrit | Merged openstack/ansible-config_template master: import zuul job settings from project-config https://review.openstack.org/594188 | 02:56 |
| openstackgerrit | Merged openstack/ansible-config_template master: switch documentation job to new PTI https://review.openstack.org/594189 | 02:56 |
| openstackgerrit | Merged openstack/ansible-role-systemd_networkd master: switch documentation job to new PTI https://review.openstack.org/594199 | 02:56 |
| openstackgerrit | Merged openstack/ansible-role-systemd_networkd stable/rocky: import zuul job settings from project-config https://review.openstack.org/595139 | 02:56 |
| openstackgerrit | Merged openstack/ansible-role-systemd_mount stable/rocky: import zuul job settings from project-config https://review.openstack.org/595138 | 02:56 |
| openstackgerrit | Merged openstack/ansible-role-systemd_mount master: import zuul job settings from project-config https://review.openstack.org/594196 | 02:56 |
| openstackgerrit | Merged openstack/ansible-role-systemd_mount master: switch documentation job to new PTI https://review.openstack.org/594197 | 02:56 |
| openstackgerrit | Merged openstack/ansible-role-systemd_service master: switch documentation job to new PTI https://review.openstack.org/594201 | 02:59 |
| openstackgerrit | Merged openstack/ansible-hardening stable/rocky: import zuul job settings from project-config https://review.openstack.org/595135 | 03:04 |
| *** kmlau777 has joined #openstack-ansible | 03:15 | |
| *** radeks has joined #openstack-ansible | 03:19 | |
| *** radeks has quit IRC | 03:27 | |
| *** radeks has joined #openstack-ansible | 03:41 | |
| *** cshen has joined #openstack-ansible | 03:48 | |
| *** udesale has joined #openstack-ansible | 03:53 | |
| *** cshen has quit IRC | 03:53 | |
| *** radeks has quit IRC | 04:11 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_heat stable/rocky: Remove galera_client meta-dependency https://review.openstack.org/603734 | 04:19 |
| openstackgerrit | Merged openstack/openstack-ansible-os_cloudkitty stable/rocky: Remove galera_client meta-dependency https://review.openstack.org/603761 | 04:32 |
| *** cshen has joined #openstack-ansible | 04:42 | |
| *** cshen has quit IRC | 04:46 | |
| *** cshen has joined #openstack-ansible | 04:47 | |
| *** olivierb has joined #openstack-ansible | 04:48 | |
| *** olivierb has quit IRC | 04:49 | |
| *** cshen has quit IRC | 04:51 | |
| *** cshen has joined #openstack-ansible | 05:06 | |
| *** cshen has quit IRC | 05:12 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_octavia stable/rocky: Remove galera_client meta-dependency https://review.openstack.org/603786 | 05:15 |
| Guest58757 | guilhermesp added a couple suggestions but nice pr | 05:32 |
| *** Guest58757 is now known as cloudnull | 05:32 | |
| cloudnull | ^ looks like xnc was mad at my username | 05:33 |
| cloudnull | **znc | 05:33 |
| *** shyamb has joined #openstack-ansible | 05:34 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Update monitoring index for replicas https://review.openstack.org/603937 | 05:36 |
| openstackgerrit | Merged openstack/openstack-ansible-os_magnum stable/rocky: Remove mysql-python https://review.openstack.org/603492 | 05:38 |
| *** shyamb has quit IRC | 05:39 | |
| *** shyamb has joined #openstack-ansible | 05:48 | |
| *** chkumar|off is now known as chkumar|ruck | 05:53 | |
| openstackgerrit | Merged openstack/openstack-ansible-repo_build stable/ocata: import zuul job settings from project-config https://review.openstack.org/595026 | 05:55 |
| *** shyamb has quit IRC | 06:03 | |
| *** shyamb has joined #openstack-ansible | 06:03 | |
| openstackgerrit | Merged openstack/openstack-ansible-nspawn_hosts master: fix tox python3 overrides https://review.openstack.org/584828 | 06:26 |
| *** cshen has joined #openstack-ansible | 06:28 | |
| *** DanyC has joined #openstack-ansible | 06:41 | |
| *** gkadam has joined #openstack-ansible | 06:46 | |
| *** fghaas has joined #openstack-ansible | 06:47 | |
| *** shyamb has quit IRC | 06:55 | |
| *** DanyC has quit IRC | 07:03 | |
| *** DanyC has joined #openstack-ansible | 07:03 | |
| *** hamzaachi has joined #openstack-ansible | 07:07 | |
| *** DanyC has quit IRC | 07:08 | |
| *** shyamb has joined #openstack-ansible | 07:08 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_glance stable/ocata: import zuul job settings from project-config https://review.openstack.org/594746 | 07:10 |
| openstackgerrit | Merged openstack/openstack-ansible-os_tempest stable/rocky: Update branch to point to Rocky https://review.openstack.org/599810 | 07:12 |
| *** olivierb has joined #openstack-ansible | 07:21 | |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-os_neutron master: Testing neutron jobs https://review.openstack.org/528692 | 07:39 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-os_nova master: [DNM] - Testing openSUSE nova role https://review.openstack.org/495767 | 07:40 |
| openstackgerrit | Merged openstack/openstack-ansible-tests stable/rocky: Pin ARA to the current version https://review.openstack.org/603798 | 07:40 |
| *** shyamb has quit IRC | 07:41 | |
| *** shyamb has joined #openstack-ansible | 07:41 | |
| arxcruz | hey guys, can someone help me with a patch? | 07:43 |
| arxcruz | i'm trying to enable stackviz on os_tempest but i'm getting this error: | 07:43 |
| arxcruz | http://logs.openstack.org/00/603100/15/check/openstack-ansible-functional-ubuntu-xenial/83dd3c9/job-output.txt.gz#_2018-09-20_04_46_36_661483 | 07:43 |
| arxcruz | Could not find or access '/openstack/venvs/stackviz/share/stackviz-html | 07:43 |
| arxcruz | problem is, the venv is being created at TASK [os_tempest : Install stackviz] i already debug the output, and it's being installed, but it seems it's disapearing from /openstack/venvs/stackviz | 07:44 |
| arxcruz | patch is https://review.openstack.org/#/c/603100/ | 07:44 |
| *** jean-philippe has joined #openstack-ansible | 07:48 | |
| *** jean-philippe is now known as evrardjp | 07:49 | |
| *** shyamb has quit IRC | 07:59 | |
| *** ebbex has joined #openstack-ansible | 08:06 | |
| *** DanyC has joined #openstack-ansible | 08:14 | |
| *** tosky has joined #openstack-ansible | 08:21 | |
| *** DanyC has joined #openstack-ansible | 08:24 | |
| *** shyamb has joined #openstack-ansible | 08:34 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_heat stable/ocata: import zuul job settings from project-config https://review.openstack.org/594749 | 08:37 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-ops master: Add option to allow a deployer to manually set master nodes https://review.openstack.org/584462 | 08:49 |
| *** Bhujay has joined #openstack-ansible | 09:00 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_ironic stable/pike: Fix OpenSSL import in test config https://review.openstack.org/603939 | 09:04 |
| *** ChanServ changes topic to "Launchpad: https://launchpad.net/openstack-ansible || Weekly Meetings: https://wiki.openstack.org/wiki/Meetings/openstack-ansible || Review Dashboard: http://bit.ly/2Nrt3xV" | 09:32 | |
| odyssey4me | morning folks - I've updated the dashboard short link to be more complete as I've noticed some are not seeing all reviews | 09:33 |
| *** hamzaachi has quit IRC | 09:35 | |
| odyssey4me | oops, a little too much in that one - updating | 09:35 |
| *** ChanServ changes topic to "Launchpad: https://launchpad.net/openstack-ansible || Weekly Meetings: https://wiki.openstack.org/wiki/Meetings/openstack-ansible || Review Dashboard: http://bit.ly/2podPLH" | 09:37 | |
| *** fghaas has quit IRC | 09:39 | |
| *** ChanServ changes topic to "Launchpad: https://launchpad.net/openstack-ansible || Weekly Meetings: https://wiki.openstack.org/wiki/Meetings/openstack-ansible || Review Dashboard: http://bit.ly/2xA1eZC" | 09:40 | |
| odyssey4me | found a few more strays, that should be a little cleaner | 09:41 |
| odyssey4me | I think perhaps I should give zuul a chance to catch up today. :) | 09:44 |
| *** cshen has quit IRC | 09:45 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/rocky: Revert "Add lxc3 compatibility" https://review.openstack.org/603602 | 09:51 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_congress master: Define cache_timeout variable https://review.openstack.org/604027 | 09:55 |
| ebbex | q | 09:57 |
| *** shyamb has quit IRC | 09:59 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_barbican stable/rocky: Remove galera_client meta-dependency https://review.openstack.org/603728 | 10:01 |
| *** pcaruana has joined #openstack-ansible | 10:04 | |
| *** shyamb has joined #openstack-ansible | 10:17 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_neutron stable/pike: Clean up role tests https://review.openstack.org/603862 | 10:19 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_neutron stable/pike: import zuul job settings from project-config https://review.openstack.org/595061 | 10:19 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_neutron stable/pike: Clean up role tests https://review.openstack.org/603862 | 10:21 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_neutron stable/pike: import zuul job settings from project-config https://review.openstack.org/595061 | 10:21 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_neutron stable/ocata: Clean up role tests https://review.openstack.org/603865 | 10:23 |
| odyssey4me | jrosser hwoarang mind reviewing https://review.openstack.org/603941 ? | 10:25 |
| jrosser | odyssey4me: done | 10:26 |
| *** cshen has joined #openstack-ansible | 10:27 | |
| *** ppetit has joined #openstack-ansible | 10:33 | |
| Bhujay | hi everybody , any idea why the ironic node stops during botting process at initramfs prompt with this errors: http://paste.openstack.org/show/730403/ , | 10:34 |
| Bhujay | the image was made using disk-image builder | 10:34 |
| hwoarang | odyssey4me: thanks. I informed the opensuse cloud team for a potential fix | 10:36 |
| hwoarang | nicolasbock: ^ :) | 10:36 |
| jrosser | is this a know brokenness? there are tons and tons of these http://logs.openstack.org/45/603745/1/check/openstack-ansible-functional-opensuse-423/a4e6977/job-output.txt.gz#_2018-09-19_23_42_38_329624 | 10:37 |
| *** fghaas has joined #openstack-ansible | 10:37 | |
| Bhujay | cloudnul, andymccr , prometheanfire ^ | 10:37 |
| *** evrardjp has quit IRC | 10:37 | |
| *** evrardjp has joined #openstack-ansible | 10:38 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-ops master: MNAIO: Disable metering services by default https://review.openstack.org/604034 | 10:38 |
| nicolasbock | hwoarang: Do I understand you correctly that cursive is a runtime dependency of Cinder? | 10:40 |
| hwoarang | jrosser: saw that but i dont know what it means. i think it's fixed because i ran two jobs this morning and they worked | 10:40 |
| hwoarang | nicolasbock: i think so. odyssey4me is that right? | 10:40 |
| arxcruz | odyssey4me: hey, can you please help me on https://review.openstack.org/#/c/603100/ ? not sure why the venv isn't there, i did some debug, and pip is installing properly... | 10:41 |
| odyssey4me | hwoarang: it was added as a requirement for rocky, yes: https://github.com/openstack/cinder/commit/5ba82a1d697305e558b2848cd1aa69b1f2f97756 | 10:41 |
| odyssey4me | arxcruz: sure, lemme take a peek | 10:42 |
| arxcruz | odyssey4me: it should have a /openstack/venvs/stackviz there, but there isn't, in fact should have the tempest venv as well, not sure what i'm doing wrong | 10:42 |
| odyssey4me | arxcruz: reviewed with some suggestions and I think the solution to the actual problem :) | 10:54 |
| arxcruz | odyssey4me: thanks! | 10:56 |
| odyssey4me | arxcruz: oh, and the reason the tempest venv isn't there is because we like confusing everyone by overriding it :/ https://github.com/openstack/openstack-ansible-tests/blob/697616662e76031fb4b46a7bda363fbf880b7d99/test-vars.yml#L357 | 10:56 |
| odyssey4me | That's a legacy thing which I hope to eradicate this cycle. | 10:56 |
| *** udesale has quit IRC | 10:58 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_nova stable/rocky: Clean up role tests https://review.openstack.org/601870 | 10:59 |
| *** fghaas has quit IRC | 10:59 | |
| *** joabdearaujo has quit IRC | 11:01 | |
| *** ansmith has quit IRC | 11:05 | |
| *** shyamb has quit IRC | 11:12 | |
| *** shyamb has joined #openstack-ansible | 11:12 | |
| *** pcaruana has quit IRC | 11:15 | |
| *** fghaas has joined #openstack-ansible | 11:15 | |
| *** shyamb has quit IRC | 11:16 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_nova stable/pike: import zuul job settings from project-config https://review.openstack.org/595062 | 11:18 |
| *** pcaruana has joined #openstack-ansible | 11:20 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-ops master: MNAIO: Ensure consistent defaults https://review.openstack.org/604059 | 11:21 |
| *** kmlau777 has quit IRC | 11:27 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_cinder stable/rocky: Add missing dep for SUSE distro installs https://review.openstack.org/603941 | 11:30 |
| *** pcaruana has quit IRC | 11:32 | |
| devx | morning all | 11:37 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-ops master: MNAIO: Correct README regarding DATA_DISK_DEVICE https://review.openstack.org/604064 | 11:39 |
| *** pcaruana has joined #openstack-ansible | 11:39 | |
| odyssey4me | o/ devx how're things? | 11:39 |
| devx | good, trying to get back to the flow of things and then some | 11:40 |
| *** fghaas has quit IRC | 11:40 | |
| *** pcaruana has quit IRC | 11:50 | |
| openstackgerrit | Arx Cruz proposed openstack/openstack-ansible-os_tempest master: WIP - Enable stackviz support https://review.openstack.org/603100 | 11:55 |
| *** shyamb has joined #openstack-ansible | 11:57 | |
| *** priteau has joined #openstack-ansible | 12:03 | |
| *** ppetit has quit IRC | 12:04 | |
| *** aedc has joined #openstack-ansible | 12:14 | |
| guilhermesp | thanks cloudnull ! I'm going to take a look at your comments :D | 12:14 |
| *** aedc is now known as aedc_ | 12:15 | |
| *** aedc_ is now known as aedc | 12:15 | |
| *** ppetit has joined #openstack-ansible | 12:20 | |
| *** canori01 has quit IRC | 12:27 | |
| *** canori01 has joined #openstack-ansible | 12:31 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_keystone stable/rocky: Remove galera_client meta-dependency https://review.openstack.org/603777 | 12:33 |
| openstackgerrit | Arx Cruz proposed openstack/openstack-ansible-os_tempest master: WIP - Enable stackviz support https://review.openstack.org/603100 | 12:34 |
| mgariepy | morning everyone | 12:41 |
| *** radeks has joined #openstack-ansible | 12:42 | |
| *** Bhujay has quit IRC | 12:43 | |
| *** Bhujay has joined #openstack-ansible | 12:44 | |
| *** ansmith has joined #openstack-ansible | 12:44 | |
| *** Bhujay has quit IRC | 12:45 | |
| *** Bhujay has joined #openstack-ansible | 12:45 | |
| *** gkadam has quit IRC | 12:46 | |
| *** gkadam has joined #openstack-ansible | 12:48 | |
| *** gkadam has quit IRC | 12:48 | |
| hwoarang | morning mgariepy | 12:50 |
| arxcruz | who was the guy on ptg that told it would help me with suse packaging? | 12:50 |
| hwoarang | odyssey4me: iirc we've seen this prob in the past http://paste.opensuse.org/84934074 but this manifests again in Leap 15 local builds. Do you happen to remember how we solved it? | 12:51 |
| hwoarang | arxcruz: there are a few suse people here. just tell us what you are looking for | 12:51 |
| arxcruz | hwoarang: https://review.openstack.org/#/c/598143/ | 12:51 |
| arxcruz | it's failing on suse ci | 12:51 |
| hwoarang | ok maybe nicolasbock can help. i am not familiar with this particular suse ci instance | 12:52 |
| openstackgerrit | Taseer Ahmed proposed openstack/openstack-ansible-os_blazar master: Move MQ vhost/user creation into role https://review.openstack.org/604078 | 13:00 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-os_nova master: SUSE: Add support for openSUSE Leap 15 https://review.openstack.org/604080 | 13:01 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-os_glance master: SUSE: Add support for openSUSE Leap 15 https://review.openstack.org/603133 | 13:02 |
| hwoarang | cores: would anyone like to review https://review.openstack.org/#/c/603139/ so I can make progress with the roles? no tempest no party again :) | 13:03 |
| *** shyamb has quit IRC | 13:07 | |
| *** shyamb has joined #openstack-ansible | 13:07 | |
| mgariepy | hwoarang, done | 13:12 |
| mgariepy | can i trade it with ? https://review.openstack.org/#/c/595240/ | 13:13 |
| *** dmsimard has quit IRC | 13:15 | |
| *** spatel has joined #openstack-ansible | 13:15 | |
| hwoarang | mgariepy: done. this is good business :p | 13:15 |
| mgariepy | haha ;D | 13:15 |
| mgariepy | hwoarang, https://review.openstack.org/#/c/601623/1 << the bug has been updated today, shoud you abandon or we still need to merge it ? | 13:19 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-os_keystone master: SUSE: Switch distro jobs as voting https://review.openstack.org/604085 | 13:19 |
| hwoarang | mgariepy: the package will take a couple of weeks before it lands in Leap. it has to go through internal QA, make it available in the enterprize product and then sync back to Leap | 13:20 |
| mgariepy | ok no problem | 13:20 |
| hwoarang | i can revert the patch in a couple of weeks time once it lands | 13:20 |
| mgariepy | perfect thanks i wasn't sure how fast things were moving :) | 13:21 |
| *** spatel has quit IRC | 13:21 | |
| *** shyamb has quit IRC | 13:23 | |
| *** udesale has joined #openstack-ansible | 13:27 | |
| hwoarang | well you know the enterprise machine has its own pace :D | 13:32 |
| *** goldenfri has joined #openstack-ansible | 13:33 | |
| mgariepy | seems faster than university speed ;p haha | 13:34 |
| *** aedc_ has joined #openstack-ansible | 13:35 | |
| hwoarang | haha yeah that's true for sure :) | 13:36 |
| *** aedc has quit IRC | 13:38 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_congress master: Define cache_timeout variable https://review.openstack.org/604027 | 13:43 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_congress stable/rocky: Define cache_timeout variable https://review.openstack.org/604092 | 13:43 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_congress stable/rocky: Define cache_timeout variable https://review.openstack.org/604092 | 13:43 |
| *** spatel has joined #openstack-ansible | 13:44 | |
| *** strattao has joined #openstack-ansible | 13:44 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_congress stable/queens: Define cache_timeout variable https://review.openstack.org/604093 | 13:45 |
| olivierb | hello everyone | 13:45 |
| olivierb | any current known issue with rabbitmq repos ??? | 13:45 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_congress stable/rocky: Remove galera_client meta-dependency https://review.openstack.org/603791 | 13:45 |
| olivierb | https://dl.bintray.com/rabbitmq/rpm/erlang/19/el/7/repodata/repomd.xml returns forbidden | 13:46 |
| d34dh0r53 | returning forbidden for me as well | 13:49 |
| spatel | cloudnull: are you there? | 13:49 |
| spatel | i need your 5 min | 13:49 |
| spatel | cloudnull: let me know when you here | 13:50 |
| *** radeks has quit IRC | 13:50 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_almanach stable/pike: Do not log passwords https://review.openstack.org/603952 | 13:53 |
| openstackgerrit | Merged openstack/openstack-ansible-os_almanach stable/pike: import zuul job settings from project-config https://review.openstack.org/595044 | 13:53 |
| openstackgerrit | Merged openstack/openstack-ansible-os_tacker stable/pike: Clean up role tests https://review.openstack.org/603885 | 13:56 |
| openstackgerrit | Merged openstack/openstack-ansible-os_designate stable/ocata: Remove tempest testing https://review.openstack.org/603949 | 13:56 |
| *** chkumar|ruck is now known as chkumar|off | 13:56 | |
| mgariepy | logan-, are you using the cinder cluster stuff altready ? | 14:00 |
| spatel | jamesdenton: ^^ | 14:03 |
| jamesdenton | hi spatel | 14:03 |
| openstackgerrit | Guilherme Steinmuller Pimentel proposed openstack/openstack-ansible-ops master: Add variable to define a beat service state https://review.openstack.org/603906 | 14:04 |
| guilhermesp | mnaser: cloudnull ^ | 14:04 |
| * mnaser looks | 14:04 | |
| spatel | jamesdenton: 1 sec my machine got hung | 14:04 |
| arxcruz | nicolasbock: hey, can you take a look at https://review.openstack.org/#/c/598143/ and identify why suse isn't building the rpm ? | 14:06 |
| *** cshen has quit IRC | 14:07 | |
| spatel | jamesdenton: I am configuring SR-IOV and i found this stanza in documentation | 14:07 |
| spatel | http://paste.openstack.org/show/730431/ | 14:07 |
| jamesdenton | right | 14:07 |
| spatel | so i need to add that extra standza in OSA ? | 14:07 |
| spatel | currently i have this one in file http://paste.openstack.org/show/730432/ | 14:07 |
| spatel | i should keep both right? | 14:08 |
| jamesdenton | If you want to use the same provider label for SR-IOV that you would use for non-SRIOV, just add "sriov_host_interfaces: "p1p1,p4p1"" | 14:08 |
| jamesdenton | and the other group_bind (although it may not really be used) | 14:09 |
| jamesdenton | make sure to specify the SR-IOV interfaces for your environment. it may just be one | 14:09 |
| spatel | this is my final config can you verify ? http://paste.openstack.org/show/730434/ | 14:09 |
| *** aevoo has joined #openstack-ansible | 14:10 | |
| spatel | i should keep both br-vlan and SRIOV right? | 14:10 |
| spatel | my SR-IOV interface is sriov_host_interfaces: "eno1" | 14:10 |
| spatel | Trying to understand how br-vlan work with SR-IOV | 14:11 |
| spatel | I want to keep 50% machine SR-IOV and 50% regular | 14:12 |
| *** aedc_ has quit IRC | 14:13 | |
| jamesdenton | so in that output you have one provider label named 'vlan' that will use br-vlan bridge (and whatever interface is in there), you have 'flat' that will use 'eth12' and you have 'physnet1' that will use eno1 for its traffic. That sound right? | 14:14 |
| openstackgerrit | Merged openstack/ansible-hardening master: file_perms: Skip rpm verification step for Leap 15 https://review.openstack.org/601623 | 14:15 |
| jamesdenton | you could also use the same provider label for SR-IOV and non-SRIOV traffic.. just slide line 31 below line 7, and line 34 below line 9. When you create a 'direct' port, that will leverage the sriov agent. when you create a 'normal' port, that will leverage linuxbridge agent | 14:15 |
| spatel | I have two NIC (eno1, eno2) currently i have bond them in bond0 for HA ( and created br-mgmt, br-vlan, br-vxvlan on bond0 using VLAN tag) | 14:16 |
| jamesdenton | sure. a assume br-vlan contains bond0, right? | 14:16 |
| spatel | you said SR-IOV won't work with bonding.. | 14:16 |
| jamesdenton | right. that's right | 14:16 |
| spatel | Yes br-vlan contains bond0 | 14:16 |
| openstackgerrit | Merged openstack/openstack-ansible-os_designate stable/ocata: import zuul job settings from project-config https://review.openstack.org/594745 | 14:16 |
| openstackgerrit | Merged openstack/openstack-ansible-os_designate stable/ocata: Updated from OpenStack Ansible Tests https://review.openstack.org/602304 | 14:16 |
| spatel | jamesdenton: here you go http://paste.openstack.org/show/730436/ | 14:17 |
| jamesdenton | you would have to break that bond. if it's a Mode 1 bond (active/passive) you MIGHT be able to keep the bond in place and use VFs from one or both of the NICs for instances, but i don't know if that'll work or if there are side effects | 14:17 |
| spatel | They are running on bond0 (mode=4) | 14:18 |
| spatel | I thought i will break bond0 and keep eno2 in bond0 and use eno1 for SR-IOV traffic | 14:19 |
| spatel | eno1 will be accessible for customer where they sending lots of UDP packet... | 14:19 |
| spatel | rest of the VLAN traffic will go thought br-vlan (bond0) | 14:20 |
| spatel | through* | 14:20 |
| jamesdenton | thru not-so-bond0? | 14:20 |
| jamesdenton | yeah, it's worth a shot | 14:20 |
| spatel | how SR-IOV VLAN get DHCP IP? | 14:20 |
| spatel | lets try something... is this looks good to you? http://paste.openstack.org/show/730438/ | 14:22 |
| spatel | I kept br-vlan in place and add extra stanza for SR-IOV | 14:22 |
| spatel | jamesdenton: ^^ | 14:23 |
| *** radeks has joined #openstack-ansible | 14:23 | |
| openstackgerrit | Merged openstack/openstack-ansible master: Ensure package cache is updated for nspawn containers https://review.openstack.org/588430 | 14:27 |
| openstackgerrit | Merged openstack/openstack-ansible stable/rocky: Use latest tag for ansible-resolvconf role https://review.openstack.org/603705 | 14:27 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/rocky: Ensure package cache is updated for nspawn containers https://review.openstack.org/604106 | 14:29 |
| openstackgerrit | Andy Smith proposed openstack/openstack-ansible-os_nova master: Update messaging notification configuration https://review.openstack.org/604107 | 14:31 |
| hwoarang | would anyone like to get another +2 in https://review.openstack.org/#/c/599784/4 to make mbuil_ a happy neutron person? | 14:31 |
| mgariepy | done | 14:32 |
| openstackgerrit | Merged openstack/openstack-ansible-os_nova stable/ocata: Clean up role tests https://review.openstack.org/601876 | 14:34 |
| *** cshen has joined #openstack-ansible | 14:35 | |
| *** cshen has quit IRC | 14:40 | |
| *** goldenfri has quit IRC | 14:40 | |
| *** KeithMnemonic has joined #openstack-ansible | 14:40 | |
| *** goldenfri has joined #openstack-ansible | 14:41 | |
| *** spatel has quit IRC | 14:46 | |
| *** spatel has joined #openstack-ansible | 14:50 | |
| spatel | jamesdenton: sorry my wifi kicked me out :( | 14:50 |
| spatel | my client doesn't have auto re-connect so i wasn't in channel | 14:50 |
| openstackgerrit | Merged openstack/openstack-ansible-os_cinder stable/pike: Clean up role tests https://review.openstack.org/603873 | 14:51 |
| *** Bhujay has quit IRC | 14:51 | |
| *** fghaas has joined #openstack-ansible | 14:53 | |
| *** dmsimard has joined #openstack-ansible | 14:54 | |
| *** cshen has joined #openstack-ansible | 14:55 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_cinder stable/queens: Clean up role tests https://review.openstack.org/603869 | 15:00 |
| *** goldenfri has quit IRC | 15:02 | |
| jamesdenton | spatel that ought to work. DHCP will work normally. | 15:03 |
| spatel | can you just just last time verify? http://paste.openstack.org/show/730438/ | 15:04 |
| openstackgerrit | Andy Smith proposed openstack/openstack-ansible-os_neutron master: Update messaging notification configuration https://review.openstack.org/604117 | 15:04 |
| spatel | so i can run playbook and push out changes | 15:04 |
| openstackgerrit | Merged openstack/openstack-ansible master: [DOC] fix some typos and fix some stuff https://review.openstack.org/595240 | 15:05 |
| spatel | if all looks good then i can go ahead | 15:05 |
| jamesdenton | there are some other changes, too, if I'm not mistaken | 15:06 |
| spatel | https://docs.openstack.org/openstack-ansible-os_neutron/latest/configure-network-services.html | 15:06 |
| spatel | as per doc yes... | 15:07 |
| jamesdenton | right | 15:07 |
| jamesdenton | LGTM (no warranty implied) | 15:07 |
| spatel | i have specified container_interface: "eth11" multiple time does that looks ok? | 15:07 |
| *** goldenfri has joined #openstack-ansible | 15:07 | |
| spatel | jamesdenton: i can understand... my openstack is anyway not in use currently because i move all customer to AWS | 15:08 |
| *** olivierb has quit IRC | 15:08 | |
| spatel | do i need to change this or not container_interface: "eth11" ? (because br-vlan already using it..) | 15:09 |
| spatel | or i shouldn't worry about that? | 15:09 |
| jamesdenton | It's really only for the neutron agent container. BUT, because you're using two provider labels Neutron may actually want a unique interface for each. So, for now you might be better off collapsing those into one stanza like i mentioned earlier. The port type (normal vs direct) will dictate whether or not SR-IOV is used | 15:09 |
| jamesdenton | if you change that to say, eth13, you would need to run some container playbooks to create that interface and veth | 15:09 |
| openstackgerrit | Merged openstack/openstack-ansible-os_magnum stable/rocky: Remove galera_client meta-dependency https://review.openstack.org/603706 | 15:10 |
| jamesdenton | i haven't had enough coffee to talk thru that today | 15:10 |
| spatel | hmmm!! | 15:10 |
| spatel | where is cloudnull??? | 15:10 |
| spatel | Wondering no-one implement SR-IOV on OSA ? | 15:11 |
| *** goldenfri has quit IRC | 15:11 | |
| jamesdenton | I'm sure a few have. I have an environment running it. I helped write those patches. | 15:12 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-os_swift master: SUSE: Add support for openSUSE Leap 15 https://review.openstack.org/604122 | 15:12 |
| spatel | is it possible you can get me some sample config for that environment... ? | 15:12 |
| spatel | all i need that stanza to understand how to implement | 15:13 |
| jamesdenton | the guide shows the same config i'm using | 15:13 |
| *** goldenfri has joined #openstack-ansible | 15:13 | |
| spatel | but do you have eth11 or eth13 ? | 15:13 |
| spatel | that is all i want to know | 15:13 |
| openstackgerrit | Merged openstack/openstack-ansible-os_ceilometer stable/rocky: Update branch to point to Rocky https://review.openstack.org/599784 | 15:13 |
| hwoarang | mnaser: given the rabbitmq repo failures, would it make sense to talk to RDO people to package rabbitmq-server as part of RDO? | 15:15 |
| spatel | oh!!! wait... now i understand what you saying... | 15:15 |
| mnaser | hwoarang: rabbitmq-server is already packaged in rdo | 15:15 |
| spatel | just use single stanza for both SRIOV and br-vlan right | 15:15 |
| mnaser | with a tested/integrated versions | 15:15 |
| jamesdenton | well, that is irrelevant. What you need to know is that eth11 is some arbitrary interface inside an LXC container that is one end of a veth pair connected to br-vlan on the network node. In the LXC container (neutron agent container, specifically, eth11 is used for provider/tenant networks using the 'vlan' provider label. | 15:15 |
| hwoarang | hmm can't see it on https://trunk.rdoproject.org/centos7-rocky/3e/5a/3e5a2d227c813937ad7ea23d334f7d23c1f8a2cc_0c712243/ | 15:15 |
| mnaser | so they not only package it, but they test it too as part of all the tooling | 15:15 |
| mnaser | its in the deps | 15:15 |
| mnaser | one sec | 15:15 |
| hwoarang | ok then maybe we should fix the repo to use the distro package instead of the external | 15:16 |
| *** dxiri has joined #openstack-ansible | 15:16 | |
| mnaser | i think they're published by cbs | 15:16 |
| mnaser | im trying to find where they are | 15:17 |
| mnaser | https://trunk.rdoproject.org/centos7-master/deps/latest/x86_64/ | 15:17 |
| mnaser | https://trunk.rdoproject.org/centos7-master/deps/latest/noarch/ | 15:17 |
| mnaser | there it is | 15:17 |
| mnaser | noarch because erlang | 15:17 |
| mnaser | rabbitmq 3.6.5 -- not sure how new/old that is | 15:17 |
| mnaser | with erlang 19.3.6 | 15:17 |
| jamesdenton | spatel yes, the same stanza simply ensures that for the linuxbridge side, vlan maps to eth11 inside the agent container, or br-vlan on a compute, or it maps to "host_bind_override" interface. For SR-IOV agent, the 'vlan' label maps to interfaces listed in "sriov_host_interfaces". that's it | 15:17 |
| openstackgerrit | Merged openstack/openstack-ansible-os_ceilometer stable/rocky: Bug fix: ceilometer_service_setup_host missing https://review.openstack.org/603175 | 15:19 |
| openstackgerrit | Merged openstack/openstack-ansible-os_ceilometer stable/rocky: Add the extra parameter in the tooz dependency https://review.openstack.org/603176 | 15:19 |
| hwoarang | mnaser: ok thank you i will try to switch from external to distro see if htat fixes the job | 15:19 |
| *** fghaas has quit IRC | 15:19 | |
| mnaser | hwoarang: thank you | 15:20 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-rabbitmq_server master: RedHat: Use the distribution package https://review.openstack.org/604124 | 15:20 |
| openstackgerrit | Taseer Ahmed proposed openstack/openstack-ansible-os_blazar master: Move MQ vhost/user creation into role https://review.openstack.org/604078 | 15:20 |
| spatel | jamesdenton: that means all i need to do is just add sriov_host_interfaces: "eno1" in existing stanza right? | 15:20 |
| spatel | check this out http://paste.openstack.org/show/730446/ | 15:20 |
| spatel | is this looks good now? | 15:20 |
| spatel | i will keep "net_name: "vlan"" same | 15:20 |
| spatel | and in group add "- neutron_sriov_nic_agent" | 15:21 |
| jamesdenton | there you go. | 15:21 |
| spatel | I think now my brain started working :) | 15:21 |
| *** cshen has quit IRC | 15:22 | |
| spatel | damn it.. i thought for SR-IOV you have to add new stanza | 15:22 |
| spatel | I wish we can clear that in document that if you already have "br-vlan" then please modify and append following | 15:22 |
| jamesdenton | it can probably be made cleared, sure | 15:23 |
| jamesdenton | *clearer | 15:23 |
| spatel | cool!! | 15:23 |
| spatel | i will open ticket if that would be helpful | 15:24 |
| jamesdenton | It might be better to work your way thru the process and document any inconsistencies or confusion, then we can take that feedback and adjust accordingly? | 15:25 |
| *** dave-mccowan has quit IRC | 15:25 | |
| spatel | I am on my way :) i will try to capture all possible thing which makes me confuse | 15:26 |
| jamesdenton | right on. | 15:26 |
| spatel | In order to attach SR-IOV interface, we have to create neutron port for instance right? | 15:26 |
| spatel | this is related nova boot command | 15:27 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Add playbook to automatically refresh fields in kibana https://review.openstack.org/603517 | 15:27 |
| spatel | Horizon doesn't have that support if i am not wrong | 15:27 |
| *** fghaas has joined #openstack-ansible | 15:28 | |
| jamesdenton | i believe you need to create the port first as a 'direct' port then attach. i think you can attach with nova boot. i don't know horizon support offhand | 15:29 |
| *** ThiagoCMC has joined #openstack-ansible | 15:29 | |
| *** dave-mccowan has joined #openstack-ansible | 15:29 | |
| spatel | Ok | 15:29 |
| spatel | let me first finish my playbook stuff and then i will play with that | 15:29 |
| spatel | jamesdenton: is it possible if my 50% machine configured for SR-IOV and other non-SR-IOV? how to tell OSA to make it work? | 15:32 |
| spatel | i meant how to isolate my compute resource? | 15:33 |
| jamesdenton | you would use AZs or aggregates, or something. | 15:33 |
| spatel | hmm! | 15:34 |
| spatel | Ok let me see | 15:34 |
| *** ThiagoCMC has quit IRC | 15:35 | |
| *** dxiri has quit IRC | 15:35 | |
| openstackgerrit | Andy Smith proposed openstack/openstack-ansible-os_cinder master: Update messaging notification configuration https://review.openstack.org/604131 | 15:39 |
| logan- | mgariepy: which clustering? | 15:41 |
| spatel | jamesdenton: do i need to do any special configuration on my physical eno1 interface? | 15:41 |
| jamesdenton | you need to make sure VFs are enabled | 15:41 |
| spatel | that are enabled | 15:41 |
| jamesdenton | then i think thats it. | 15:42 |
| spatel | k | 15:42 |
| mgariepy | logan-, you can set cluster=whatever in cinder.conf and it should add this to the volumes in the db | 15:42 |
| logan- | o rly | 15:42 |
| mgariepy | https://blueprints.launchpad.net/cinder/+spec/cinder-volume-active-active-support | 15:42 |
| logan- | so that prevents having to migrate volumes between cinder volume services when one goes down? | 15:42 |
| mgariepy | there is a bunch of patches already merged but a lot of other stuff not quite there yet.. | 15:43 |
| logan- | i think my stuffs using the hack where you set a consistent 'host' | 15:43 |
| logan- | they don't recommend doing it, but it has never caused me any problems, and with it we never have to migrate volumes when a volume container goes down | 15:44 |
| mgariepy | yeah that's probably works-ish. but it's not protecting you for doing stuff on 2 server at the same time. | 15:44 |
| logan- | which stuff do you mean | 15:46 |
| *** fghaas has quit IRC | 15:47 | |
| *** hamzaachi has joined #openstack-ansible | 15:47 | |
| openstackgerrit | Merged openstack/openstack-ansible-ops master: Update monitoring index for replicas https://review.openstack.org/603937 | 15:47 |
| mgariepy | maybe you can have race condition that make your volume attached by 2 vms at the same time, not quite sure. | 15:47 |
| openstackgerrit | Cam J. Loader proposed openstack/openstack-ansible-specs master: create Designate back end provider spec https://review.openstack.org/556586 | 15:49 |
| *** dariko has joined #openstack-ansible | 15:50 | |
| mgariepy | 2018-09-20 15:49:11.748 4734 ERROR cinder.volume.manager [req-d021a469-30a8-4a2d-a5fc-6e95308334ae - - - - -] Active-Active configuration is not currently supported by driver cinder.volume.drivers.rbd.RBDDriver. | 15:50 |
| mgariepy | 2018-09-20 15:49:11.749 4734 ERROR cinder.cmd.volume [req-d021a469-30a8-4a2d-a5fc-6e95308334ae - - - - -] Volume service controller01-cinder-volumes-container-a02ed322@volumes_hdd failed to start.: VolumeDriverException: Volume driver reported an error: Active-Active configuration is not currently supported by driver cinder.volume.drivers.rbd.RBDDriver. | 15:50 |
| mgariepy | 2018-09-20 15:49:11.749 4734 ERROR cinder.cmd.volume Traceback (most recent call last): | 15:50 |
| mgariepy | 2018-09-20 15:49:11.749 4734 ERROR cinder.cmd.volume File "/openstack/venvs/cinder-17.0.8/lib/python2.7/site-packages/cinder/cmd/volume.py", line 101, in _launch_service | 15:50 |
| mgariepy | 2018-09-20 15:49:11.749 4734 ERROR cinder.cmd.volume cluster=cluster) | 15:50 |
| mgariepy | 2018-09-20 15:49:11.749 4734 ERROR cinder.cmd.volume File "/openstack/venvs/cinder-17.0.8/lib/python2.7/site-packages/cinder/service.py", line 403, in create | 15:50 |
| mgariepy | 2018-09-20 15:49:11.749 4734 ERROR cinder.cmd.volume cluster=cluster) | 15:50 |
| mgariepy | 2018-09-20 15:49:11.749 4734 ERROR cinder.cmd.volume File "/openstack/venvs/cinder-17.0.8/lib/python2.7/site-packages/cinder/service.py", line 156, in __init__ | 15:51 |
| mgariepy | 2018-09-20 15:49:11.749 4734 ERROR cinder.cmd.volume *args, **kwargs) | 15:51 |
| mgariepy | 2018-09-20 15:49:11.749 4734 ERROR cinder.cmd.volume File "/openstack/venvs/cinder-17.0.8/lib/python2.7/site-packages/cinder/volume/manager.py", line 276, in __init__ | 15:51 |
| mgariepy | 2018-09-20 15:49:11.749 4734 ERROR cinder.cmd.volume raise exception.VolumeDriverException(message=msg) | 15:51 |
| mgariepy | 2018-09-20 15:49:11.749 4734 ERROR cinder.cmd.volume VolumeDriverException: Volume driver reported an error: Active-Active configuration is not currently supported by driver cinder.volume.drivers.rbd.RBDDriver. | 15:51 |
| *** fghaas has joined #openstack-ansible | 15:51 | |
| mgariepy | opps. sorry about that. tough I had only 1 line.. | 15:51 |
| *** dariko has quit IRC | 15:52 | |
| mgariepy | seems like cinder-volume don't like the cluster config right now. | 15:53 |
| *** gyee has joined #openstack-ansible | 15:58 | |
| logan- | ah yeah i guess each backend driver has to add support | 16:01 |
| openstackgerrit | Alex Redinger proposed openstack/openstack-ansible-os_keystone stable/ocata: Increase retries for Ensure service tenant play https://review.openstack.org/604144 | 16:02 |
| *** fghaas has quit IRC | 16:02 | |
| mgariepy | i was hoping to fix the multiple cache images needed for each image but well.. it's not like it's going to take a lot more space anyway. | 16:03 |
| *** vnogin has joined #openstack-ansible | 16:04 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Add more optionality when customizing node roles https://review.openstack.org/603526 | 16:04 |
| logan- | yeah | 16:05 |
| *** vnogin has quit IRC | 16:09 | |
| openstackgerrit | Alex Redinger proposed openstack/openstack-ansible-os_keystone stable/pike: Increase retries for Ensure service tenant play https://review.openstack.org/604147 | 16:10 |
| openstackgerrit | Arx Cruz proposed openstack/openstack-ansible-os_tempest master: WIP - Enable stackviz support https://review.openstack.org/603100 | 16:11 |
| *** aedc_ has joined #openstack-ansible | 16:14 | |
| odyssey4me | mnaser guilhermesp bommer, we're getting loads of [Errno 14] HTTPS Error 403 - Forbidden from https://dl.bintray.com/rabbitmq/ - we may have to implement another reverse proxy :/ | 16:15 |
| *** aevoo has quit IRC | 16:17 | |
| spatel | jamesdenton: ^^ | 16:27 |
| -spatel- 03:01.0 Ethernet controller [0200]: Broadcom Limited NetXtreme II BCM57810 10 Gigabit Ethernet Virtual Function [14e4:16af] | 16:27 | |
| -spatel- 03:01.1 Ethernet controller [0200]: Broadcom Limited NetXtreme II BCM57810 10 Gigabit Ethernet Virtual Function [14e4:16af] | 16:27 | |
| openstackgerrit | Guilherme Steinmuller Pimentel proposed openstack/openstack-ansible-ops master: Add variable to define a beat service state https://review.openstack.org/603906 | 16:27 |
| spatel | based on lspci -nn | 16:27 |
| spatel | nova_pci_passthrough_whitelist: '{"address":"14e4:*.*", "physical_network":"physnet1"}' | 16:28 |
| spatel | I should use "14e4:16af" right? | 16:28 |
| jamesdenton | that should be vendor_id | 16:28 |
| spatel | is that looking good what i pasted ? | 16:28 |
| jamesdenton | address would be 0000:03:01.*, for example. vendor_id would be 14e4 | 16:29 |
| spatel | Okay!! product_id=0000:03:01 and vendor_id=14e4 | 16:30 |
| spatel | Got it | 16:30 |
| jamesdenton | no no, pci address=0000:03:01. product is is 16af | 16:32 |
| spatel | oh!! | 16:33 |
| spatel | Got it what you saying... | 16:34 |
| *** hamzaachi has quit IRC | 16:35 | |
| spatel | jamesdenton: is this looks good http://paste.openstack.org/show/730455/ | 16:38 |
| spatel | going to run playbook now | 16:38 |
| *** dariko has joined #openstack-ansible | 16:39 | |
| jamesdenton | you don't need them all | 16:40 |
| jamesdenton | you only need one | 16:40 |
| jamesdenton | nova_pci_passthrough_whitelist: '{ "physical_network":"vlan", "devname":"eno1" }' | 16:40 |
| spatel | hmm!! really? | 16:40 |
| *** Bhujay has joined #openstack-ansible | 16:40 | |
| jamesdenton | yep | 16:40 |
| spatel | how it will understand which NIC is enabled for VF? | 16:40 |
| jamesdenton | you told it. eno1 | 16:41 |
| spatel | oh!! :) | 16:41 |
| jamesdenton | :D | 16:41 |
| *** dariko has quit IRC | 16:41 | |
| -spatel- neutron_plugin_types: | 16:41 | |
| -spatel- - ml2.sriov | 16:41 | |
| spatel | it should go here user_variables.yml | 16:41 |
| spatel | right? | 16:41 |
| *** Bhujay has quit IRC | 16:41 | |
| jamesdenton | yes | 16:41 |
| spatel | cool! going to remove PCI lines | 16:41 |
| *** Bhujay has joined #openstack-ansible | 16:42 | |
| spatel | only keeping one... we need to change that document then :) | 16:42 |
| *** radez has quit IRC | 16:42 | |
| jamesdenton | "Possible options which can be configured:" | 16:42 |
| jamesdenton | "It is recommended to use whitelisting by either the Linux device name (devname attribute) or by the PCI vendor and product id combination (vendor_id and product_id attributes)" | 16:43 |
| *** radez has joined #openstack-ansible | 16:43 | |
| spatel | I kept only one line "nova_pci_passthrough_whitelist: '{ "physical_network":"vlan", "devname":"eno1" }'" | 16:44 |
| spatel | running playbook now | 16:44 |
| jamesdenton | k | 16:44 |
| jamesdenton | Is this Pike? Queens? | 16:45 |
| jamesdenton | Rocky? :D | 16:45 |
| spatel | Queens | 16:45 |
| jamesdenton | k | 16:45 |
| spatel | this playbook will run on all nodes right openstack-ansible os-nova-install.yml ? | 16:45 |
| jamesdenton | yes | 16:45 |
| spatel | is it possible i can just test on single box? | 16:45 |
| spatel | i have 80 compute nodes... | 16:45 |
| spatel | just to safe side.. if possible to run on single node and then i go nuke | 16:46 |
| jamesdenton | hmm. you might be able to limit it to network/controllers and the one compute | 16:46 |
| spatel | openstack-ansible os-neutron-install.yml this one will run on all node also | 16:46 |
| jamesdenton | yep. so try the limit, too | 16:47 |
| spatel | I have to run them on infra* nodes and then i can limit on compute | 16:47 |
| spatel | let me try | 16:47 |
| spatel | How about this ? | 16:49 |
| spatel | openstack-ansible os-neutron-install.yml -l compute-infra_hosts | 16:49 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-rabbitmq_server master: RedHat: Use the distribution package https://review.openstack.org/604124 | 16:50 |
| openstackgerrit | Merged openstack/openstack-ansible-os_designate stable/pike: Pin tempest plugin https://review.openstack.org/603948 | 16:52 |
| *** udesale has quit IRC | 16:52 | |
| *** DanyC has quit IRC | 16:59 | |
| *** ThiagoCMC has joined #openstack-ansible | 17:06 | |
| *** gkadam has joined #openstack-ansible | 17:14 | |
| *** ppetit has quit IRC | 17:33 | |
| cloudnull | afternoons :) | 17:34 |
| *** Bhujay has quit IRC | 17:34 | |
| spatel | cloudnull: very late :) | 17:34 |
| spatel | since morning people finding you | 17:34 |
| cloudnull | its 5 o'clock somewhere | 17:34 |
| spatel | haha | 17:34 |
| cloudnull | hows it ? | 17:34 |
| spatel | i was just kidding.. this morning i was looking for you but jamesdenton help me out | 17:35 |
| *** aevoo has joined #openstack-ansible | 17:37 | |
| cloudnull | dr. jamesdenton is a beautiful human | 17:37 |
| spatel | Indeed | 17:39 |
| jamesdenton | *blush* | 17:39 |
| jamesdenton | lunch time.. bbiab | 17:39 |
| * cloudnull reading back | 17:39 | |
| *** aedc_ has quit IRC | 17:39 | |
| cloudnull | looks like you're doing SRIOV things | 17:39 |
| spatel | cloudnull: yes | 17:39 |
| cloudnull | hows that going ? | 17:39 |
| spatel | going to run playbook now | 17:40 |
| spatel | openstack-ansible os-neutron-install.yml -l compute-infra_hosts is this looking good? | 17:40 |
| spatel | i want to run only on infra* nodes | 17:40 |
| jamesdenton | specifically, the neutron_server containers really | 17:40 |
| spatel | -l neutron_server i should do this then | 17:40 |
| spatel | but sriov agent should be running on physical infra node right? | 17:41 |
| jamesdenton | no, it only runs on the computes | 17:41 |
| spatel | oh!!! | 17:41 |
| jamesdenton | and SR-IOV capable computes at that | 17:41 |
| spatel | then i think i should limit with -l neutron_server | 17:41 |
| spatel | doing it now | 17:42 |
| ThiagoCMC | Guys, is it possible to have 1 single OSA Ansible Server to deploy many independent OpenStack deployments (i.e., many Regions)? Or is it better/recommended to have 1 OSA Ansible per Region? | 17:42 |
| cloudnull | ThiagoCMC I generally use 1 deployment node per region | 17:42 |
| ThiagoCMC | cloudnull, thanks! =) | 17:43 |
| cloudnull | that said I know folks are doing one to many | 17:43 |
| ThiagoCMC | Oh, I might try that! | 17:43 |
| cloudnull | recently we've been using the loggng host as the deployment host | 17:43 |
| cloudnull | so not a dedicated deployment node | 17:43 |
| ThiagoCMC | Interesting! I'm also thinking about Ansible Tower... | 17:44 |
| cloudnull | ++ that might be a good option too | 17:44 |
| *** dave-mccowan has quit IRC | 17:44 | |
| cloudnull | last time i used ansible tower was in version 1.7 and it didnt work well due to our use of complex variables. I'd be really curious if its better today | 17:45 |
| cloudnull | if you give it a go i'd love to know your thoughts | 17:45 |
| ThiagoCMC | I see, because of the "Ansible Wrapper" and OSA uses? | 17:46 |
| *** dave-mccowan has joined #openstack-ansible | 17:46 | |
| cloudnull | no it was the jinja | 17:47 |
| spatel | jamesdenton: with SR-IOV do i need to disable security group ? | 17:47 |
| spatel | enable_security_group | 17:47 |
| ThiagoCMC | Hmm... Ok! | 17:47 |
| cloudnull | ThiagoCMC like so https://github.com/openstack/openstack-ansible-os_nova/blob/master/vars/main.yml#L20 | 17:47 |
| cloudnull | things like that used to crash tower | 17:47 |
| ThiagoCMC | Ouch! | 17:47 |
| cloudnull | IDK if it still does | 17:47 |
| ThiagoCMC | I'll try it in a month or two! I'll let you know. | 17:48 |
| ThiagoCMC | =) | 17:48 |
| cloudnull | I'd hope not . OSA isn't the only one doing that kinda of thing | 17:48 |
| cloudnull | I've just not circled back to try again | 17:48 |
| guilhermesp | odyssey4me: still getting loads of [Errno 14] HTTPS Error 403 - Forbidden? | 17:48 |
| cloudnull | I had decent luck with https://github.com/ansible-semaphore/semaphore - but the UI was just a little too basic for what I hoping | 17:49 |
| logan- | ThiagoCMC: another idea is throwaway nodes for deployment | 17:50 |
| logan- | you don't need a persistent deployment host necessarily | 17:50 |
| logan- | ThiagoCMC: a very simple way to do that is by storing your /etc/openstack_deploy in git as we do here https://github.com/openstack/limestone-ci-cloud/tree/master/openstack_deploy and then using a simple script to run the deployment https://github.com/openstack/limestone-ci-cloud/blob/master/scripts/run_deploy.sh | 17:54 |
| *** cshen has joined #openstack-ansible | 17:54 | |
| ThiagoCMC | logan-, nice!!! | 17:56 |
| jrosser | ++on that. My deployment node is disposable too | 18:00 |
| logan- | only catch is you'll have to manually update your openstack_inventory.json into the repo when you add or remove hosts/containers, especially containers, otherwise they'll just keep getting re-generated. so teh process is generally, bootstrap a deployment container with an updated openstack_user_config.yml, run openstack-ansible/inventory/dynamic_inventory.py so the openstack_inventory.json is generated, and then commit the new one to the git | 18:00 |
| logan- | tree | 18:00 |
| ThiagoCMC | logan-, could I have, for example, /etc/openstack_deploy_region1, ...region2, ...region3 and use this run_deploy.sh ? Like, 1 subdirectory under /etc for each region? | 18:01 |
| ThiagoCMC | Hmm... I'll definitely take a look into this! | 18:02 |
| logan- | sort of. you could do multiple openstack_deploy subdirs in the git tree and then add some sort of selection to the script when you bootstrap the node. like if you used a docker container for manual deployments you could pass an envvar that would tell run_deploy which inventory to copy | 18:04 |
| logan- | osa uses /etc/openstack_deploy exclusively, so the easiest way is to work out a solution where you only have one region/cloud per deployment node, and then set it up so you can select the cloud you're trying to deploy when you bootstrap the node. either by using a different git repo, different inventory dirs, etc | 18:06 |
| jrosser | I have multiple deploys in the same git repo in different dirs, clone it to /opt then symlink the required one into /etc | 18:07 |
| odyssey4me | guilhermesp: I think so. There're a bunch of them. I've not followed up regarding why we're suddenly getting 403's. | 18:08 |
| spatel | jamesdenton: i have created SR-IOV port and attach to instance, and instance is up and running.. how do i know its really attached to VF ? | 18:09 |
| ThiagoCMC | logan-, thank you! I have a much better understanding about what to do now. ^_^ | 18:11 |
| spatel | jamesdenton: it created two interface in my machine... even i gave single --nic option http://paste.openstack.org/show/730466/ | 18:12 |
| spatel | I can see ens5 mac is matching with eno1 | 18:12 |
| spatel | but ens5 doesn't get any IP | 18:13 |
| logan- | np ThiagoCMC | 18:14 |
| spatel | oh! wait that did magic "dhclient ens5" | 18:14 |
| spatel | but still i am surprise why i got two NIC when i didn't specify two nic | 18:15 |
| spatel | This is the command i use to create instance | 18:15 |
| -spatel- [root@ostack-osa ~(keystone_admin)]# nova boot --flavor pp.2xlarge --image "CentOS-7-1611-x86_64" --nic port-id=db6e1d7c-a4c8-4fc6-ac6d-030ca8418f2f --nic net-id=a34c47ab-0e29-47c2-ade6-1c5c5fa6ed04 --security-groups allow_all_traffic bar-mq-15vcpu --availability-zone numa:ostack-compute-65 | 18:15 | |
| spatel | jamesdenton: let me know when you back from your lunch :) | 18:16 |
| jamesdenton | spatel what do you get with "neutron agent-list --agent_type 'NIC Switch agent'", just to confirm | 18:16 |
| jamesdenton | you did specify two NICs - you specified a port directly for one, and a network for the other. you passed --nic twice | 18:16 |
| spatel | http://paste.openstack.org/show/730467/ | 18:17 |
| jamesdenton | If your intention was to specify one network via the port, then just pass the port. The network is inferred (via port creation) | 18:17 |
| jamesdenton | looks promising | 18:17 |
| spatel | oh!!! so much excitement :) | 18:17 |
| spatel | let me delete and re-create :) | 18:17 |
| guilhermesp | ping evrardjp | 18:18 |
| jamesdenton | so, if you do an "ip link show' - i think - on the compute, one of the VFs should have the MAC address of the neutron port | 18:18 |
| spatel | nova boot --flavor pp.2xlarge --image "CentOS-7-1611-x86_64" --nic port-id=db6e1d7c-a4c8-4fc6-ac6d-030ca8418f2f --nic net-id=a34c47ab-0e29-47c2-ade6-1c5c5fa6ed04 --security-groups allow_all_traffic bar-mq-15vcpu --availability-zone numa:ostack-compute-65 | 18:19 |
| spatel | is this looks good to you? | 18:19 |
| jamesdenton | hmm, i don't think you will be able to leverage security groups | 18:20 |
| *** mandeer has joined #openstack-ansible | 18:20 | |
| jamesdenton | so it may just be ignored/not applied | 18:20 |
| spatel | let me remove that | 18:20 |
| spatel | building... | 18:20 |
| jamesdenton | well, it will apply default. you will want to update the port with --no-security-groups/--disable-port-security | 18:20 |
| spatel | what if i want two NIC on instance? and i want only nic1 attach to SR-IOV and nic2 regular one ? | 18:21 |
| mandeer | hey guys, I was wondering if i could get a little guidance on network setup | 18:21 |
| mandeer | i was in the other day, and someone said it's fine to put br-mgmt on the native vlan | 18:21 |
| mandeer | but what implication does that have for br-vlan? | 18:22 |
| jamesdenton | and in addition, you will need to make sure the VM has the VF driver. i assume that's bnx2x? maybe? | 18:22 |
| mandeer | is it correct that br-vlan is what containers use to create connections to the network, both tagged and native? | 18:22 |
| spatel | jamesdenton: it create two interface on my instance again... hmmm! that is very odd | 18:22 |
| jamesdenton | mandeer if you're using the same interfaces it will be problematic | 18:22 |
| mandeer | ok. so br-vlan on native/ | 18:23 |
| jamesdenton | spatel you passed --nic twice again | 18:23 |
| spatel | oh!!!! i thought i have to pass both port-id and --nic so port ID will attach to nic.. | 18:23 |
| spatel | damn it... | 18:23 |
| spatel | i am wrong | 18:23 |
| spatel | let me fix it... | 18:23 |
| jamesdenton | mandeer the untagged (physical) interface in br-vlan allows neutron to tag traffic accordingly. br-mgmt would need to have a tagged interface off that physical interface (i.e. not native) | 18:23 |
| mandeer | jamesdenton: i ordered some cheap gigabit usb3 adapters so i can do more sane 2-nic setup in my lab | 18:23 |
| jamesdenton | spatel yep, just need the port. | 18:24 |
| spatel | fixing it | 18:24 |
| jamesdenton | mandeer i used to use USB connectors. works well | 18:24 |
| mandeer | what would the recommended sane starter setup be for two nics? in terms of traffic | 18:25 |
| mandeer | you want to keep br-vlan and br-vxlan separate from storage, right? | 18:26 |
| jamesdenton | let neutron have one (for br-vlan) and run everything else on the other | 18:26 |
| mandeer | ok | 18:26 |
| jamesdenton | for a lab config it doesn't really matter | 18:26 |
| jamesdenton | you can also use the same nic for vxlan/vlan and the other for storage/container/mgmt | 18:26 |
| mandeer | ok, sweet | 18:27 |
| mandeer | thank you, that helps | 18:27 |
| jamesdenton | sure. good luck to ya | 18:27 |
| spatel | jamesdenton: EPIC win!!!!! it works http://paste.openstack.org/show/730468/ | 18:28 |
| *** Leo_m has joined #openstack-ansible | 18:28 | |
| spatel | it attached to VF | 18:28 |
| jamesdenton | well i'll be damned | 18:28 |
| jamesdenton | you get an IP? | 18:28 |
| spatel | not automatically i have to do "dhclient ens4" | 18:29 |
| spatel | how to make it auto ? | 18:29 |
| jamesdenton | Well, that depends on the image i guess. I guess the default interface it's configured for is not ens4 | 18:29 |
| jamesdenton | this a cent image? | 18:29 |
| spatel | but when i create instance my interface name is getting change like sometime ens4 sometime ens5 etc... | 18:30 |
| spatel | I am using CentOS7 image | 18:30 |
| jamesdenton | mandeer here is a single interface config if it's helpful: https://gist.github.com/busterswt/01e706086c6bc6108e66e06fc992d1c0 | 18:30 |
| jamesdenton | spatel what interface is listed in /etc/sysconfig/network-scripts? | 18:31 |
| spatel | currently /etc/sysconfig/network-scripts/ifcfg-eth0 & /etc/sysconfig/network-scripts/ifcfg-eth1 | 18:31 |
| spatel | eth0 and eth1 | 18:31 |
| *** tosky has quit IRC | 18:32 | |
| jamesdenton | yeah, so i'm not sure how to ensure consistency unless you pass a kernel parameter to disable those names: net.ifnames=0 | 18:32 |
| jamesdenton | when you did a dhclient you got an IP? | 18:32 |
| spatel | yes | 18:33 |
| spatel | my machine is live in network | 18:33 |
| spatel | i can ssh in | 18:33 |
| spatel | don't you think i need to create ifcfg-ensX file and set BOOTPROTO=DHCP ? | 18:33 |
| mandeer | jamesdenton: yeah i've been working off the examples provided in the openstack-ansible project | 18:33 |
| mandeer | i just got confused about the networks, since there's a lot of info to assimilate | 18:34 |
| mandeer | i'm also working on centos7 | 18:34 |
| jamesdenton | spatel in this case, sure. and maybe update your image if you think ens4 will be the interface moving forward | 18:35 |
| mandeer | if i get a working config that matches the .test.example files, would it be worthwhile to add the files in a pull request? | 18:35 |
| jamesdenton | mandeer gotcha. Yeah that config will look a little different, but same thing applies. | 18:35 |
| spatel | let me build multiple instance and see if my interface name is getting change | 18:35 |
| mandeer | seeing those debian based network configs makes me a little jealous to be honest | 18:36 |
| jamesdenton | spatel will be waiting for your UDP results | 18:36 |
| spatel | i am about to run | 18:36 |
| jamesdenton | mandeer i'm told https://netplan.io is the future | 18:37 |
| jamesdenton | spatel No problem. Nice work on that and hope the results pay off | 18:38 |
| *** hamzaachi has joined #openstack-ansible | 18:38 | |
| mandeer | that's exciting. yaml is a blessing | 18:38 |
| spatel | yes... i have big hope... currently fixing my stupid DNS issue... not able to run yum | 18:39 |
| jamesdenton | update your subnet and rerun dhclient maybe | 18:40 |
| spatel | i think i found issue... let me fix it.. | 18:42 |
| *** sep has quit IRC | 18:46 | |
| *** radeks has quit IRC | 18:46 | |
| *** radeks has joined #openstack-ansible | 18:46 | |
| spatel | I have Cisco nexus vPC base bonding so i have to do something on switch side to break that bonding | 18:48 |
| spatel | i have break that in OS but vPC doesn't like it | 18:48 |
| jamesdenton | yeah, that could be a problem. | 18:49 |
| *** ThiagoCMC has quit IRC | 18:52 | |
| openstackgerrit | Guilherme Steinmuller Pimentel proposed openstack/openstack-ansible-lxc_hosts master: Drop SELinux support for CentOS 7 https://review.openstack.org/604190 | 18:54 |
| spatel | jamesdenton: i have to remove all my bond0 config and make it eno1 and eno2 because vPC doesn't working that way... | 18:56 |
| spatel | fixing it now.. | 18:56 |
| jamesdenton | k | 18:56 |
| jamesdenton | you tried keeping the bond and using the VF off eno1? | 18:57 |
| jamesdenton | oh yeah, i see that in your pasteit | 18:57 |
| jamesdenton | good to know it won't work | 18:57 |
| spatel | yes | 18:57 |
| spatel | i thought i will use eno1 for sr-iov and other eno2 for other VLAN traffic | 18:58 |
| spatel | so i kept bond0 attach with eno2 | 18:58 |
| *** lkoranda has quit IRC | 19:00 | |
| *** ThiagoCMC has joined #openstack-ansible | 19:03 | |
| spatel | jamesdenton: i fixed all those issue | 19:05 |
| spatel | running packet test | 19:05 |
| spatel | which interface i should check for drop ? | 19:06 |
| jamesdenton | hmm. you will probably not be able to see it. | 19:06 |
| jamesdenton | do you have two VMs now with SR-IOV interfaces? | 19:06 |
| spatel | vf 2 MAC fa:16:3e:e3:07:97, vlan 200, tx rate 10000 (Mbps), max_tx_rate 10000Mbps, spoof checking on, link-state auto | 19:06 |
| jamesdenton | yeah but you won't be able to capture ob that | 19:06 |
| jamesdenton | it's invisible to the OS, effectively | 19:07 |
| spatel | just single VM currently running on sr-iov | 19:07 |
| jamesdenton | can you spin up another on the same node? | 19:07 |
| spatel | i can but how that will solve problem? | 19:07 |
| spatel | let me create one more instance | 19:07 |
| jamesdenton | i was messing around with another utility, nuttcp, which showed UDP drops in the output. could help here | 19:08 |
| spatel | hmm!! | 19:08 |
| spatel | what i should do after second instance ? | 19:09 |
| *** lkoranda has joined #openstack-ansible | 19:09 | |
| spatel | jamesdenton: don't you think after attach VF all TX/RX drop should be visible inside OS? | 19:10 |
| jamesdenton | download nuttcp, maybe with yum. Then on 1 VM run "nuttcp -S" and on the other, try "nuttcp -T 30 -i -u -R 6G <IP Addr>". That will send 6Gbps UDP traffic. You would see something similar to this: http://paste.openstack.org/show/730473/ | 19:10 |
| spatel | on ens5 interface i can see all physical nic attributes so its directly attached to physical.. | 19:11 |
| spatel | doing it | 19:11 |
| jamesdenton | spatel No. The only way i know to get per-VF statistics would be to use macvtap interface w/ VF. But that causes drop in performance. | 19:12 |
| spatel | hmm! | 19:12 |
| spatel | let me try you nuttcp but it won't give us UDP result but anyway | 19:12 |
| jamesdenton | Some drivers do reveal that info, i believe, i just don't know what to use. | 19:13 |
| jamesdenton | That syntax i gave should use UDP with the '-u' flag | 19:13 |
| spatel | ok | 19:13 |
| *** lkoranda has quit IRC | 19:13 | |
| spatel | jamesdenton: how to automate this port-creating stuff... ??/ | 19:14 |
| jamesdenton | for i in... | 19:14 |
| jamesdenton | lol | 19:14 |
| spatel | :( | 19:15 |
| jamesdenton | i dunno, what do you mean? | 19:15 |
| spatel | currently if i want to use sr-iov i have to first create port | 19:15 |
| spatel | how to automate that so when i say attach to sr-iov subnet auto create neutron port | 19:16 |
| jamesdenton | you may just need to script that out for your own workflow. | 19:16 |
| spatel | hmm!! i think horizon doesn't have that feature too | 19:16 |
| spatel | anyway no big-deal | 19:16 |
| jamesdenton | It may be there? https://bugs.launchpad.net/horizon/+bug/1402959 | 19:17 |
| openstack | Launchpad bug 1402959 in OpenStack Dashboard (Horizon) "Support Launching an instance with a port with vnic_type=direct" [Medium,Fix released] - Assigned to Akihiro Motoki (amotoki) | 19:17 |
| openstackgerrit | Merged openstack/openstack-ansible stable/rocky: Revert "Add lxc3 compatibility" https://review.openstack.org/603602 | 19:18 |
| spatel | hmmm | 19:18 |
| *** lkoranda has joined #openstack-ansible | 19:20 | |
| *** skiedude has joined #openstack-ansible | 19:21 | |
| spatel | nuttcp running... | 19:22 |
| spatel | lots of drop | 19:22 |
| *** radeks has quit IRC | 19:22 | |
| spatel | sending you result | 19:22 |
| spatel | here you go http://paste.openstack.org/show/730476/ | 19:23 |
| skiedude | Before I go through the process of making a bug is anyone aware of any major config changes for cinder/glance between queens 17.0.8 and Rocky? When i run the upgrade the cinder api/cinder storege/glance containers all lose connectivity, can't ping/ssh to them via ip, can still connect via lxc-attach | 19:23 |
| openstackgerrit | Merged openstack/openstack-ansible stable/rocky: Pin ARA to the current version https://review.openstack.org/603797 | 19:23 |
| spatel | jamesdenton: ^^ | 19:23 |
| jamesdenton | spatel indeed. | 19:24 |
| jamesdenton | might be worth comparing to two normal VMs (non SR-IOV) for reference | 19:24 |
| spatel | let me try that hold on | 19:24 |
| jamesdenton | also - inside that VM can you run 'ethtool -i ens4'? | 19:25 |
| spatel | http://paste.openstack.org/show/730478/ | 19:26 |
| spatel | its real physical nic | 19:26 |
| jamesdenton | yeah just checking | 19:26 |
| jamesdenton | what are specs of those VMs? beefy? | 19:27 |
| spatel | 15 vCPU 14G memory | 19:27 |
| *** vnogin has joined #openstack-ansible | 19:28 | |
| spatel | jamesdenton: if i use "-R 1G" option then i am getting zero packet loss | 19:29 |
| spatel | what is -R ? | 19:29 |
| jamesdenton | the rate limiter | 19:30 |
| spatel | are you getting same result in your network ? | 19:30 |
| spatel | packet drop ? | 19:30 |
| spatel | with sr-iov | 19:30 |
| jamesdenton | Well yes and no. I haven't tested SR-IOV for this, i did the testing on DPDK I could push just < 6G with 0% packet loss with DPDK and 4vCPU w/ affinity | 19:31 |
| spatel | hmmm!!! | 19:32 |
| jamesdenton | i have a node with the same NIC, but i don't know how easily i can bring it in | 19:32 |
| spatel | that is interesting | 19:32 |
| jamesdenton | that testing pegged a single core - it's not threaded | 19:32 |
| *** ThiagoCMC has quit IRC | 19:33 | |
| spatel | i forgot option to check drop using ethtool | 19:34 |
| spatel | do you remember ? | 19:34 |
| jamesdenton | was it ethtool or ifconfig? | 19:34 |
| spatel | ethtool | 19:34 |
| spatel | ethtool -S eno5 or something else | 19:35 |
| jamesdenton | -S is stats, so maybe | 19:35 |
| spatel | looking | 19:35 |
| jamesdenton | the physical function, though, won't reveal any stats related to the virtual function AFAIK | 19:36 |
| jamesdenton | you might run it inside the VM to see | 19:36 |
| spatel | jamesdenton: zero packet drop :) | 19:37 |
| spatel | on nuttcp :) | 19:37 |
| jamesdenton | with 1G? | 19:37 |
| spatel | when i run from other node i can easily pump 6G | 19:37 |
| spatel | sending you result | 19:37 |
| jamesdenton | k | 19:37 |
| spatel | I have setup foo VM on other compute node and running test from there http://paste.openstack.org/show/730480/ | 19:38 |
| spatel | now running with -R 7G | 19:39 |
| spatel | -R 7G (drop/pkt 0.00987 %loss) | 19:39 |
| spatel | so its matching with your DPDK result | 19:39 |
| jamesdenton | hmm. well if it were really 6G it would be reflected in the middle column | 19:39 |
| jamesdenton | check mine out - 5998-5999Mbps | 19:40 |
| *** ThiagoCMC has joined #openstack-ansible | 19:41 | |
| spatel | hmm | 19:41 |
| spatel | if you check my result i got 0.00287 %loss in -R 6G | 19:41 |
| spatel | let me run iperf and see | 19:42 |
| jamesdenton | right, but it doesn't look like it's pumping out 6G. | 19:42 |
| jamesdenton | k | 19:42 |
| spatel | -R 5G ( drop/pkt 0.00 %loss ) | 19:42 |
| spatel | so 5Gbps is our sweet spot.. | 19:43 |
| spatel | running iperf | 19:43 |
| jamesdenton | I don't know - your results seem to be more on par with 1G. | 19:43 |
| jamesdenton | if you do iPerf, make sure you call out bandwidth with -b | 19:44 |
| spatel | this is odd | 19:44 |
| spatel | iperf - [ 3] 0.0-10.0 sec 1.26 GBytes 1.08 Gbits/sec | 19:44 |
| *** hamzaachi has quit IRC | 19:44 | |
| spatel | -b 1000m is this you talking about? | 19:45 |
| jamesdenton | yes | 19:45 |
| jamesdenton | or -b 1G | 19:45 |
| spatel | 1.16 GBytes 1.00 Gbits/sec | 19:46 |
| spatel | this is very low | 19:46 |
| jamesdenton | try 5G | 19:46 |
| spatel | k | 19:46 |
| jamesdenton | it's artificial limit | 19:46 |
| spatel | 840 MBytes 705 Mbits/sec | 19:46 |
| spatel | worst | 19:46 |
| spatel | something is not right | 19:47 |
| spatel | let me run between non-sriov vm | 19:47 |
| jamesdenton | please do | 19:47 |
| spatel | 840 MBytes 705 Mbits/sec ( -b 5000m ) | 19:49 |
| spatel | same result | 19:49 |
| spatel | this is very odd | 19:49 |
| spatel | i have 10G network i should be able to send 5G atleast | 19:49 |
| jamesdenton | well you can - you've demonstrated that with TCP | 19:50 |
| spatel | iperf also using TCP | 19:50 |
| *** hamzaachi has joined #openstack-ansible | 19:50 | |
| spatel | i was expecting same result with both tool | 19:50 |
| jamesdenton | you're testing udp in this case, though, right? | 19:50 |
| spatel | no TCP | 19:51 |
| jamesdenton | what is your command? | 19:51 |
| spatel | i am only focusing TCP at present | 19:51 |
| spatel | iperf -c 10.10.10.10 -b 5000m | 19:51 |
| spatel | on server side iperf -s | 19:51 |
| jamesdenton | oh, i think -b is ignored for tcp. requires -u for udp | 19:52 |
| spatel | oh!! | 19:52 |
| spatel | crap!! | 19:52 |
| spatel | let me try again | 19:52 |
| spatel | without -b (0.0-10.0 sec 1.22 GBytes 1.04 Gbits/sec) | 19:53 |
| jamesdenton | same for non-SRIOV vms? | 19:53 |
| spatel | currently i am running on non-SR-IOV | 19:53 |
| spatel | even in SR-IOV i am getting same resilt | 19:53 |
| spatel | result | 19:53 |
| jamesdenton | hmm, yesterday i thought you were getting near 10G iperf w/ tcp? | 19:54 |
| spatel | yes..not 10G but 5G | 19:55 |
| spatel | let me see what is wrong here | 19:55 |
| spatel | let me run HOST to HOST and then VM to VM | 19:55 |
| jamesdenton | k | 19:55 |
| spatel | WTF>>> host to host (0.0-10.0 sec 1.84 GBytes 1.58 Gbits/sec) | 20:00 |
| spatel | sorry for my language i was typing WTH :( | 20:00 |
| jamesdenton | the horror | 20:00 |
| spatel | do you think this is because of Linuxbridge ? | 20:01 |
| spatel | I am using linux bridge | 20:01 |
| jamesdenton | this is just host to host right? are the IPs even configured on bridges? | 20:01 |
| spatel | running between computer node on br-mgmt | 20:02 |
| spatel | every compute node has br-mgmt IP on same LAN | 20:02 |
| spatel | so running between them | 20:02 |
| jamesdenton | ok - that's your true mgmt interface for the servers? | 20:02 |
| spatel | yes. | 20:02 |
| spatel | I have created bond of two eno1+eno2 (10G nic) | 20:03 |
| *** mandeer has quit IRC | 20:03 | |
| spatel | and on-top of bond0 i have created bond0.VLAN and ontop of VLAN i have create br-mgmt, br-vlan etc.. | 20:03 |
| spatel | bond0-->bond0.VLAN--->br-mgmt | 20:04 |
| spatel | i think this is something related to linuxbridge | 20:04 |
| spatel | let me find two physical machine without bonding.. and run test | 20:05 |
| spatel | hold on | 20:05 |
| jamesdenton | sounds reasonable.k | 20:05 |
| spatel | same result... (1.09 GBytes 939 Mbits/sec) | 20:06 |
| spatel | hmmm | 20:06 |
| jamesdenton | one sec | 20:06 |
| spatel | something is not right.. let me check my switch | 20:07 |
| jamesdenton | k | 20:07 |
| spatel | this is interesting... https://forums.freenas.org/index.php?threads/10gb-nics-and-iperf-is-showing-1gb-speeds.41218/ | 20:08 |
| spatel | reading. | 20:08 |
| spatel | can you run if you have 10G network... just curious | 20:10 |
| jamesdenton | yeah, i just did and got: [ 3] 0.0-10.0 sec 11.0 GBytes 9.41 Gbits/sec | 20:10 |
| jamesdenton | between two computes on br-mgmt | 20:10 |
| *** gkadam has quit IRC | 20:10 | |
| spatel | can you send me command ? | 20:11 |
| spatel | and kernel version | 20:11 |
| spatel | i am using CentOS 7.5 and it has 3.10 version of kernel | 20:11 |
| jamesdenton | iperf -s; iperf -c x.x.x.x; 4.15.0-33-generic (ubuntu); ixgbe 5.1.0-k | 20:12 |
| jrosser | does ethtool show the nic is actually running at 10g? | 20:12 |
| *** priteau has quit IRC | 20:13 | |
| jamesdenton | for me it does. spatel - 'ethtool eno1` | 20:13 |
| *** vnogin has quit IRC | 20:13 | |
| spatel | you have intel so that is beast | 20:15 |
| spatel | I have Qlogic | 20:15 |
| jamesdenton | it can't suck that much though | 20:15 |
| spatel | do you think it could be kernel issue? | 20:15 |
| jamesdenton | come to ubuntu and all of your problems will be solved | 20:15 |
| jamesdenton | :D | 20:15 |
| spatel | lets do one thing | 20:16 |
| spatel | let me upgrade kernel 4.x | 20:16 |
| spatel | and give it a try | 20:16 |
| jamesdenton | what version are you on? | 20:16 |
| spatel | 3.10.0-514.el7.x86_64 | 20:16 |
| spatel | 3.10 isn't that old also | 20:16 |
| jamesdenton | that kernel is ~2 yrs old it looks like. i can't decipher where rhel kernels are compared to upstream | 20:17 |
| spatel | jamesdenton: do you have MTU 9000 ? | 20:17 |
| jamesdenton | not in this test | 20:17 |
| spatel | ok because i am also not using jumbo frame | 20:18 |
| jamesdenton | what is your 'modinfo bnx2x' | 20:18 |
| spatel | http://paste.openstack.org/show/730482/ | 20:18 |
| jrosser | spatel: have you checked that the phy is running at 10g with ethtool? | 20:19 |
| spatel | oh!!!! | 20:20 |
| spatel | you are right | 20:20 |
| jrosser | just to check really dumb problems like accidentially using 1g optic or a DAC cable that it doesnt understand | 20:20 |
| spatel | http://paste.openstack.org/show/730483/ | 20:20 |
| spatel | check this http://paste.openstack.org/show/730483/ | 20:20 |
| spatel | 1000baseT/Full | 20:20 |
| spatel | damn it.. hold on... | 20:20 |
| spatel | i think i grab wrong box | 20:21 |
| spatel | they have 1G nic | 20:21 |
| spatel | damn it | 20:21 |
| jrosser | :) | 20:21 |
| jamesdenton | jrosser is the real MVP | 20:21 |
| spatel | today is bad day for me :( | 20:22 |
| spatel | finding 10G machine hold on | 20:22 |
| spatel | jamesdenton: this is interesting.... | 20:25 |
| spatel | i am running iperf on Ceph node and got this result with br-mgmt (0.0-10.0 sec 10.3 GBytes 8.88 Gbits/sec) | 20:26 |
| jamesdenton | ok thats promising | 20:26 |
| spatel | so something is wrong in compute node.. | 20:26 |
| jamesdenton | just the one compute? or both? | 20:26 |
| spatel | let me check other compute node.. | 20:26 |
| spatel | checking other compute node | 20:27 |
| *** ansmith has quit IRC | 20:29 | |
| spatel | other compute node [ 4] 0.0-10.0 sec 8.99 GBytes 7.72 Gbits/sec | 20:30 |
| spatel | now going back to my BAD compute node to see | 20:30 |
| jamesdenton | k | 20:30 |
| spatel | i am going to reboot my compute node... something is not right.. | 20:37 |
| spatel | all other compute node running fine but SR-IOV one not | 20:37 |
| jamesdenton | was that the compute connected at 1G? | 20:39 |
| spatel | no that is on 10G i check with ethtool | 20:40 |
| jamesdenton | and iperf from that node to ceph node or other 10G looked OK? | 20:41 |
| spatel | after reboot :) | 20:43 |
| spatel | [ 4] 0.0-10.0 sec 5.12 GBytes 4.39 Gbits/sec | 20:43 |
| spatel | wondering why i am not getting 10g ? | 20:44 |
| spatel | do you think because of iptables? | 20:44 |
| jamesdenton | where was that iperf performed? | 20:44 |
| spatel | yes iperf | 20:45 |
| spatel | let me fire up instance and run nuttcp | 20:45 |
| cjloader | is there an easy way to update all roles SHA? | 20:48 |
| *** ThiagoCMC has quit IRC | 20:48 | |
| cjloader | seems like some SHAs were not pointing to stable/rocky up until a few days ago | 20:49 |
| openstackgerrit | Taseer Ahmed proposed openstack/openstack-ansible-os_blazar master: Move MQ vhost/user creation into role https://review.openstack.org/604078 | 20:56 |
| cloudnull | cjloader if the ansible role requirements file has the values you need/want you can rerun bootstrap-ansible.sh | 20:59 |
| guilhermesp | any core around to push that up https://review.openstack.org/#/c/603906/ ? | 21:00 |
| cloudnull | done | 21:00 |
| guilhermesp | cloudnull: \m/ | 21:00 |
| *** goldenfri has quit IRC | 21:02 | |
| admin0 | whom from this channel will be in the summit ? | 21:03 |
| mnaser | <-- | 21:04 |
| jamesdenton | <-- | 21:04 |
| guilhermesp | o/ | 21:04 |
| cloudnull | prometheanfire odyssey4me ? | 21:04 |
| * cloudnull no | 21:04 | |
| mnaser | cloudnull: no?! ;( | 21:04 |
| guilhermesp | ??? cloudnull | 21:05 |
| cloudnull | most likely not. :'( | 21:05 |
| prometheanfire | wat | 21:05 |
| prometheanfire | I am, yes | 21:05 |
| mnaser | i mean if its time constraints | 21:05 |
| mnaser | id understand | 21:05 |
| mnaser | if others, i can try to talk to the foundation | 21:05 |
| cjloader | i'm not | 21:06 |
| cloudnull | I thought about applying for travel assistance. we'll see | 21:06 |
| mnaser | cloudnull: i think the deadline is up but we can send some emails | 21:07 |
| cloudnull | thanks! I appreciate it, though don't go out of your way. | 21:08 |
| cloudnull | yup missed it, deadline was aug 30 | 21:09 |
| cloudnull | admin0 you going? | 21:09 |
| admin0 | yep | 21:09 |
| cloudnull | cool! | 21:09 |
| cloudnull | spatel how about you, headed to the summit? | 21:12 |
| spatel | berlin ? | 21:12 |
| cloudnull | yup | 21:12 |
| spatel | this year isn't not possible :) | 21:13 |
| cloudnull | then Denver, in the spring :) | 21:13 |
| spatel | may be next year when my openstack will be valuable for my company | 21:13 |
| spatel | how much for entry fee ? | 21:13 |
| spatel | Denver | 21:14 |
| cloudnull | its usually ~$700 | 21:15 |
| spatel | ok | 21:15 |
| spatel | i will ask my company if they ready for that | 21:16 |
| cloudnull | https://www.openstack.org/summit/berlin-2018/faq/ - from berlin, i suspect denver will be similar | 21:16 |
| cloudnull | though the conference is longer. | 21:16 |
| spatel | are you going berlin? | 21:16 |
| mnaser | cloudnull: do we need journalbeat if we're just logging files? | 21:16 |
| mnaser | as in like just the nova service log files | 21:17 |
| mnaser | i'm trying to figure out what more does journalbeat bring in | 21:17 |
| cloudnull | nope. it will ship all of the journals into ELK but its not required. | 21:17 |
| mnaser | cloudnull: in an nspawn first world, we would just run journalbeat i assume | 21:18 |
| openstackgerrit | Merged openstack/openstack-ansible-ops master: Add variable to define a beat service state https://review.openstack.org/603906 | 21:18 |
| *** mmercer has joined #openstack-ansible | 21:19 | |
| *** ansmith has joined #openstack-ansible | 21:19 | |
| cloudnull | journalbeat and filebeat are both useful no matter. | 21:20 |
| cloudnull | some services use the journal some create a file | 21:20 |
| cloudnull | by running both you capture all the things | 21:20 |
| cloudnull | we're running both by default | 21:20 |
| mnaser | ok cool | 21:20 |
| mnaser | so looks like we'll skip on {audit,heart,metric,packet}beat in order to get some manageable amount of data | 21:21 |
| mnaser | for now | 21:21 |
| cloudnull | ++ | 21:23 |
| cloudnull | metric/packetbeat generates a lot | 21:23 |
| spatel | cloudnull: are you going berlin ? | 21:23 |
| spatel | curious | 21:23 |
| cloudnull | heartbeat is byfar the lighest | 21:23 |
| cloudnull | spatel nope | 21:23 |
| cloudnull | not at this point | 21:23 |
| spatel | ok | 21:24 |
| admin0 | do we have a working logging/monitoring now ? like kind of automatically setup like the rest of the containers ? | 21:24 |
| admin0 | not loggin .. i meant graphing | 21:24 |
| admin0 | metrics and graphs | 21:24 |
| cloudnull | admin0 we've been working on the ELK tooling | 21:25 |
| cloudnull | which has some graphs though we could do with a lot more | 21:25 |
| admin0 | does it work out of the box . like enable a variable and access it via some http:// url/port combo ? | 21:28 |
| spatel | jamesdenton: what is your setting here "ethtool -g eno1" ? | 21:28 |
| spatel | RX/TX ring buffer? | 21:28 |
| cloudnull | admin0 it does https://github.com/openstack/openstack-ansible-ops/tree/master/elk_metrics_6x | 21:33 |
| cloudnull | you do have to add the setup to the deployment but it does work out of the box | 21:34 |
| *** cshen has quit IRC | 21:34 | |
| admin0 | \o/ | 21:35 |
| admin0 | i have 6 OSA platforms now to run this on | 21:35 |
| admin0 | 4 are production .. 2 internal | 21:35 |
| admin0 | all OSA | 21:35 |
| admin0 | still a lot of manual steps though .. | 21:35 |
| admin0 | prone to copy/paste/drift error . | 21:36 |
| guilhermesp | you could either use grafana or kibana dashboards admin0 | 21:36 |
| admin0 | 99% of us will just copy/paste the whole setuff to get it over with .. so would be better if its all there like in elk.var.sample and all we need to do is rename it | 21:36 |
| admin0 | which one is better/recommended ? | 21:37 |
| guilhermesp | but kibana dashboards are a bit outdated, needs to be fixed | 21:37 |
| *** radeks has joined #openstack-ansible | 21:38 | |
| admin0 | does it also allow search of logs :D ? | 21:38 |
| admin0 | i will give this a try | 21:39 |
| guilhermesp | yeah filebeat is one of the beats responsible to ship logs... I used to use a lot in my openstack dashboards | 21:39 |
| guilhermesp | and the kibana custom dashboards playbook works for older versions of kibana | 21:39 |
| guilhermesp | not that one in master, I need to fix this | 21:40 |
| jamesdenton | spatel i am out for the night. currently set to 512/512 | 21:41 |
| jamesdenton | good luck and ttyl | 21:41 |
| admin0 | how taxing is this to the disks ? | 21:41 |
| jrosser | admin0: i have the kibana containers on my infra nodes, they are quite lightweight | 21:42 |
| admin0 | graphite is for metrics, kibana is for the logs search etc ..is it one vs the other ? | 21:42 |
| spatel | jamesdenton: Okay!! thanks | 21:42 |
| jrosser | i have 3 nodes dedicated entirely to to elasticsearch data nodes | 21:42 |
| *** mandeer has joined #openstack-ansible | 21:43 | |
| jrosser | and it uses them pretty fully | 21:43 |
| spatel | jamesdenton: good night and thanks for the help with SR-IOV :) | 21:43 |
| spatel | it was wonderful working with you :) | 21:43 |
| guilhermesp | well depends on which beats are you using... packetbeat and metric demand a lot o storage. That's why we submitted this PR that was just merged https://review.openstack.org/#/c/603906/ admin0 | 21:43 |
| guilhermesp | but you have curator to define your data retention | 21:43 |
| admin0 | i am more the copy/paste and cross-fingers guy .. so whatever that document copy/paste default will install :) | 21:43 |
| openstackgerrit | Arx Cruz proposed openstack/openstack-ansible-os_tempest master: WIP - Enable stackviz support https://review.openstack.org/603100 | 21:44 |
| *** mandeer has quit IRC | 21:44 | |
| admin0 | guilhermesp, any idea on the size it generates .. like say 3x controllers, 3 compute for example | 21:47 |
| admin0 | to try to guess if there is any baseline on the storage size it might take | 21:47 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-rabbitmq_server master: RedHat: Use the distribution package https://review.openstack.org/604124 | 21:47 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-rabbitmq_server master: RedHat: Use the distribution package https://review.openstack.org/604124 | 21:49 |
| cloudnull | admin0 the elk tooling will adjust the retention periods based on the amount of storage you provide to the instances running it | 21:50 |
| guilhermesp | admin0: well I used to have one deployment like this, 3xinfra nodes and around 25xcompute nodes, also 6xstorage nodes. I've never pain attention of the average data generated. But I used to have a 500GB disk in my elk node | 21:51 |
| guilhermesp | my data retention was about 30 days I think | 21:51 |
| admin0 | currently there is one rsyslog container which has all the logs .. it is the same logs that also goes to the elasticsearch right ? | 21:51 |
| guilhermesp | not sure if it is admin0 I think all logs are collected by the beats and send directly to elk stack | 21:53 |
| *** skiedude has quit IRC | 21:54 | |
| admin0 | " To install these dependencies throughout the cluster without having to invoke a repo-build run the following adhoc Ansible command can by used." -- is this run just on the infra nodes/controllers ? | 21:55 |
| admin0 | or to the whole cluster ? | 21:55 |
| jrosser | admin0: first question is do you want to install osprofiler | 21:56 |
| jrosser | you can skip all of that if you just want monitoring/metrics | 21:57 |
| admin0 | i want monitoring/metrics .. but i am also looking if there is an easy way to provide the logs search feature using elasticsearch/kibana .. so that when something does not work, getting to the root cause is easy | 21:58 |
| admin0 | osprofiler does sound tempting, but not sure if its actually required | 21:59 |
| jrosser | you can always add it later | 21:59 |
| admin0 | ok | 21:59 |
| jrosser | kibana lets you do log search/query easily | 21:59 |
| admin0 | so for the deployment without OSProfiler, i only need to follow the Elasticsearch, APM, Kibana endpoint configuration copy/paste and skip to the "Once the overrides are in-place the openstack-ansible playbooks will need to be rerun. " section ... | 22:01 |
| *** strattao has quit IRC | 22:02 | |
| jrosser | no - thats still in the optional section about osprofiler | 22:03 |
| jrosser | there is not need to run any OSA playbooks other than haproxy | 22:04 |
| * jrosser out for today | 22:05 | |
| cloudnull | have a great night jrosser | 22:05 |
| admin0 | thanks | 22:05 |
| jrosser | oh btw any idea on aa_profile before someone else has a go ^^ ? | 22:05 |
| spatel | cloudnull: is it ok to upgrade kernel on compute node ? | 22:06 |
| admin0 | so haproxy role is going to check for these config overrides and call the appropriate module/install directly | 22:06 |
| cloudnull | spatel yes, typically | 22:06 |
| spatel | 3.x to 4.x ? | 22:06 |
| cloudnull | should be fine | 22:06 |
| spatel | doing it | 22:07 |
| spatel | doing Kernel 4.17 | 22:07 |
| cloudnull | we've seen issue before where a module was needed in an earlier kernel thats no longer needed. | 22:07 |
| cloudnull | but in general it wokrs just fine | 22:07 |
| spatel | lets figure out :) | 22:07 |
| cloudnull | :D | 22:08 |
| cloudnull | does cent now have 4.x ? | 22:08 |
| spatel | stock centos 7.5 has 3.10.x | 22:08 |
| spatel | i want to test 4.x to see if it boost performance | 22:09 |
| cloudnull | going custom? | 22:09 |
| spatel | no downloading 4.x RPM | 22:09 |
| spatel | https://www.mysterydata.com/how-to-install-or-upgrade-to-kernel-4-16-in-centos-7-cwp7-and-vestacp/ | 22:10 |
| cloudnull | sweet! | 22:10 |
| *** radeks has quit IRC | 22:10 | |
| cloudnull | if that works that should provide a decent performance bump, however specter/meltdown might now be a factor if they were not before. | 22:11 |
| *** radeks has joined #openstack-ansible | 22:11 | |
| * cloudnull does not know if you have those patches in 3.10.x | 22:11 | |
| spatel | i don't think we have any patch | 22:12 |
| spatel | we mostly don't care about security ;) | 22:12 |
| mnaser | i think cloudnull means that centos might backport these patches | 22:12 |
| openstackgerrit | Merged openstack/openstack-ansible-os_gnocchi stable/rocky: Clean up role tests https://review.openstack.org/601856 | 22:12 |
| cloudnull | ++ | 22:13 |
| spatel | rebooting node finger cross | 22:13 |
| cloudnull | you can disable most of the specter/meltdown patches if they do cause serious performance issues | 22:13 |
| spatel | how do disable them? | 22:14 |
| spatel | first thing how to check they are there? | 22:14 |
| spatel | googling... | 22:15 |
| spatel | cloudnull: all looking good after kernel upgrade :) | 22:18 |
| spatel | let me fire up VM | 22:18 |
| cloudnull | spatel cat /sys/devices/system/cpu/vulnerabilities/* | 22:19 |
| -spatel- [root@sriov-vm ~]# cat /sys/devices/system/cpu/vulnerabilities/* | 22:19 | |
| -spatel- cat: /sys/devices/system/cpu/vulnerabilities/*: No such file or directory | 22:19 | |
| cloudnull | the command tail -n 2 /sys/devices/system/cpu/vulnerabilities/* is easier to read | 22:20 |
| cloudnull | if it shows anything as "Vulnerable then its not patched | 22:20 |
| cloudnull | spatel is that on the host? | 22:21 |
| spatel | yes its compute node | 22:21 |
| -spatel- [root@sriov-vm ~]# tail -n 2 /sys/devices/system/cpu/vulnerabilities/* | 22:21 | |
| -spatel- tail: cannot open ‘/sys/devices/system/cpu/vulnerabilities/*’ for reading: No such file or directory | 22:21 | |
| cloudnull | and "/sys/devices/system/cpu/vulnerabilities" does not exist ? | 22:21 |
| spatel | sorry wrong | 22:21 |
| cloudnull | you can also `cat /proc/cpuinfo` to see the various "bugs" that impact your proc | 22:22 |
| spatel | here you go http://paste.openstack.org/show/730487/ | 22:22 |
| spatel | look like its enabled | 22:22 |
| cloudnull | yup its all patched up | 22:22 |
| spatel | how do i disable it? | 22:22 |
| * cloudnull does not recommend this from a security point of view | 22:23 | |
| cloudnull | GRUB_CMDLINE_LINUX="spectre_v2=off nopti pti=off" | 22:23 |
| cloudnull | will disable most of them | 22:23 |
| spatel | doing it.. | 22:23 |
| cloudnull | :D | 22:23 |
| spatel | security not important for my company | 22:23 |
| spatel | we need performance | 22:23 |
| spatel | rebooting node | 22:24 |
| *** hamzaachi has quit IRC | 22:25 | |
| cloudnull | might as well add "l1tf=off" too | 22:25 |
| cloudnull | sorry ad to look that one up | 22:25 |
| spatel | will add that too | 22:25 |
| spatel | should i do that in VM also? | 22:25 |
| spatel | or just compute enough | 22:25 |
| cloudnull | your vms will have the patches too | 22:25 |
| spatel | ok | 22:26 |
| cloudnull | disabling on the host will make the host more performant but inside teh vms you will need to do the same | 22:26 |
| cloudnull | or use an unpatched kernel | 22:26 |
| spatel | ok will do | 22:27 |
| *** radeks_ has joined #openstack-ansible | 22:28 | |
| *** radeks has quit IRC | 22:31 | |
| cloudnull | spatel, so this looks like all the options to disable all the things on your host GRUB_CMDLINE_LINUX="spectre_v2=off nopti pti=off nospec_store_bypass_disable spec_store_bypass_disable=off l1tf=off" | 22:33 |
| spatel | oh! | 22:33 |
| spatel | let me do that | 22:33 |
| spatel | cloudnull: why i am still seeing them tail -n 2 /sys/devices/system/cpu/vulnerabilities/* | 22:36 |
| cloudnull | did you update grub ? | 22:36 |
| cloudnull | after making the changes? | 22:36 |
| cloudnull | something like `grub2-mkconfig -o /boot/grub2/grub.cfg` | 22:37 |
| cloudnull | ubuntu has an update-grub2 command, idk if cent has the same | 22:37 |
| spatel | yes i did grub2-makeconfig | 22:37 |
| spatel | here is the resutl http://paste.openstack.org/show/730488/ | 22:37 |
| *** dmsimard has quit IRC | 22:37 | |
| spatel | it is saying "Vulnerable" means it is disable right? | 22:37 |
| cloudnull | yes | 22:38 |
| cloudnull | spectre_v1 will always show mitigated it has no performance impact | 22:38 |
| spatel | phew!! | 22:38 |
| cloudnull | l1tf should be disabled too if you have l1tf=off | 22:39 |
| spatel | i did that one | 22:39 |
| cloudnull | https://www.suse.com/support/kb/doc/?id=7023077 | 22:40 |
| * cloudnull trying it on one sever | 22:41 | |
| spatel | hmmm! i can see better performance with nuttcp | 22:42 |
| *** cshen has joined #openstack-ansible | 22:42 | |
| spatel | if you check my previous mesg, when i was talking to jamesdenton i was getting drop with nuttcp | 22:42 |
| spatel | now i am seeing 0.00 %loss | 22:43 |
| spatel | i have to send this result to jamesdenton | 22:43 |
| cloudnull | odd I'm not seeing the l1tf option work for me either | 22:43 |
| cloudnull | on ole | 22:43 |
| cloudnull | 4.18.5-1-default | 22:44 |
| spatel | why you guys using OLE? | 22:44 |
| cloudnull | not on ole | 22:44 |
| cloudnull | just added those lines to a nuc i have | 22:45 |
| cloudnull | http://paste.openstack.org/show/730489/ | 22:45 |
| cloudnull | seems to have disabled most of the patches | 22:45 |
| *** cshen has quit IRC | 22:47 | |
| spatel | same result i have | 22:47 |
| spatel | i am going to enable that patch and going to run test again | 22:48 |
| spatel | if that patch was culprit then i am happy | 22:48 |
| *** spsurya has quit IRC | 22:48 | |
| *** xdfil has quit IRC | 22:48 | |
| *** xdfil has joined #openstack-ansible | 22:50 | |
| spatel | after that patch i am seeing significant performance improvement | 22:50 |
| *** spsurya has joined #openstack-ansible | 22:50 | |
| *** Leo_m has quit IRC | 22:52 | |
| spatel | cloudnull: gotta go.. | 22:52 |
| spatel | i will catch you tomorrow of if you available in night | 22:52 |
| *** sawblade6 has joined #openstack-ansible | 22:52 | |
| spatel | cloudnull: Thanks for the help with kernel stuff that is really good one.. :) | 22:53 |
| *** spatel has quit IRC | 22:54 | |
| cloudnull | take care | 22:54 |
| cloudnull | ttyl | 22:54 |
| *** sawblade6 has quit IRC | 22:57 | |
| *** radeks_ has quit IRC | 23:20 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_swift stable/rocky: Clean up role tests https://review.openstack.org/603895 | 23:28 |
| *** andreykurilin has quit IRC | 23:34 | |
| *** andreykurilin has joined #openstack-ansible | 23:35 | |
| *** gyee has quit IRC | 23:41 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!