Monday, 2020-11-02

*** spatel has joined #openstack-ansible00:28
*** spatel has quit IRC00:30
*** cshen has joined #openstack-ansible00:34
*** cshen has quit IRC00:38
*** tosky has quit IRC00:57
*** rfolco has quit IRC01:10
dmsimardthis query is cool too: http://paste.openstack.org/raw/799589/02:31
dmsimard^ takes the 10000 longest tasks, gets metrics about their actions and sorts the result by total duration02:32
*** cshen has joined #openstack-ansible02:34
*** cshen has quit IRC02:38
*** d34dh0r53 has joined #openstack-ansible04:33
*** cshen has joined #openstack-ansible04:34
*** cshen has quit IRC04:39
*** cshen has joined #openstack-ansible04:59
*** cshen has quit IRC05:04
*** pto has joined #openstack-ansible05:25
*** pto has joined #openstack-ansible05:27
*** evrardjp has quit IRC05:33
*** evrardjp has joined #openstack-ansible05:33
*** miloa has joined #openstack-ansible05:52
*** sshnaidm|off is now known as sshnaidm|rover06:49
*** cshen has joined #openstack-ansible07:00
*** cshen has quit IRC07:04
noonedeadpunkdmsimard: it really is. unfortunate part is that we need this in CI, however I think we can save output to file and store it07:13
*** pto has quit IRC07:20
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible master: Disable manila 2 vms tempest scenario  https://review.opendev.org/76077107:27
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_manila master: Add Ubuntu Focal CI jobs  https://review.opendev.org/75465307:27
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_neutron master: Fix OVN functional test  https://review.opendev.org/76018807:29
*** pto has joined #openstack-ansible07:30
*** pto has quit IRC07:40
*** pto has joined #openstack-ansible07:44
*** pto has quit IRC07:54
*** pcaruana has joined #openstack-ansible07:54
*** pto has joined #openstack-ansible07:55
*** cshen has joined #openstack-ansible07:57
*** pto has quit IRC08:08
*** pto_ has joined #openstack-ansible08:09
*** pto_ has quit IRC08:09
*** pto has joined #openstack-ansible08:10
*** pto_ has joined #openstack-ansible08:11
*** pto_ has quit IRC08:11
*** pto_ has joined #openstack-ansible08:12
*** pto has quit IRC08:14
*** andrewbonney has joined #openstack-ansible08:30
*** kukacz_ has joined #openstack-ansible08:37
*** tosky has joined #openstack-ansible08:45
*** kukacz13 has joined #openstack-ansible08:48
*** kleini has joined #openstack-ansible08:52
*** kukacz13 has left #openstack-ansible08:58
*** pto_ has quit IRC08:58
*** pto has joined #openstack-ansible09:01
*** fanfi10 has joined #openstack-ansible09:12
*** fanfi10 has left #openstack-ansible09:13
*** fanfi has joined #openstack-ansible09:17
openstackgerritMerged openstack/openstack-ansible-os_cloudkitty master: Reduce number of processes on small systems  https://review.opendev.org/76061709:21
*** rpittau|afk is now known as rpittau09:22
*** gshippey has joined #openstack-ansible09:36
*** yolanda has joined #openstack-ansible09:36
openstackgerritMerged openstack/openstack-ansible-rabbitmq_server master: Bump rabbitmq version  https://review.opendev.org/75966410:03
*** pto has quit IRC10:13
openstackgerritMerged openstack/ansible-role-systemd_networkd master: Fix linter errors and update tests  https://review.opendev.org/75623410:19
openstackgerritMerged openstack/ansible-role-systemd_networkd master: Updated from OpenStack Ansible Tests  https://review.opendev.org/75419510:19
openstackgerritMerged openstack/openstack-ansible-os_glance master: Do not symlink ceph libraries for distro path  https://review.opendev.org/76058110:19
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/ansible-role-systemd_networkd master: Add GPG Key for EPEL8 Repo  https://review.opendev.org/75914510:22
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/ansible-role-systemd_networkd master: Add GPG Key for EPEL8 Repo  https://review.opendev.org/75914510:22
openstackgerritMerged openstack/openstack-ansible-os_cinder master: Do not symlink ceph libraries for distro path  https://review.opendev.org/76058210:24
*** yasemind has quit IRC10:26
*** yasemind has joined #openstack-ansible10:26
*** fanfi has quit IRC10:27
openstackgerritMerged openstack/openstack-ansible-os_ironic master: Reduce number of processes on small systems  https://review.opendev.org/76063210:32
openstackgerritMerged openstack/openstack-ansible-os_mistral master: Reduce number of processes on small systems  https://review.opendev.org/76063610:33
openstackgerritMerged openstack/openstack-ansible-galera_server master: Do not remove mysql-common during upgrade  https://review.opendev.org/76054110:36
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_cinder stable/train: Do not symlink ceph libraries for distro path  https://review.opendev.org/76081310:37
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_cinder stable/ussuri: Do not symlink ceph libraries for distro path  https://review.opendev.org/76081410:37
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_cinder stable/stein: Do not symlink ceph libraries for distro path  https://review.opendev.org/76081510:38
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_glance stable/ussuri: Do not symlink ceph libraries for distro path  https://review.opendev.org/76081610:38
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_glance stable/stein: Do not symlink ceph libraries for distro path  https://review.opendev.org/76081810:38
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_glance stable/train: Do not symlink ceph libraries for distro path  https://review.opendev.org/76082010:39
*** odyssey4me|PTO is now known as odyssey4me10:43
openstackgerritMerged openstack/openstack-ansible-os_gnocchi master: Reduce number of processes on small systems  https://review.opendev.org/76062410:54
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_ironic master: Define condition for the first play host one time  https://review.opendev.org/75634011:04
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_ironic master: Update Inspector listener address and network  https://review.opendev.org/76066011:04
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible stable/ussuri: Bump SHAs for stable/ussuri  https://review.opendev.org/76073111:21
*** rfolco has joined #openstack-ansible11:52
*** mgariepy has quit IRC12:11
*** ianychoi has joined #openstack-ansible12:24
*** kukacz_ has quit IRC12:35
dmsimardnoonedeadpunk: yeah you could run the CLI command at the end of a job and save it somewhere I guess12:56
dmsimardit hasn't landed in master yet, still experimenting a bit12:56
dmsimardthere is value in searching across multiple playbooks from multiple jobs though :)12:57
*** mgariepy has joined #openstack-ansible13:02
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible master: Fix upgrade jobs for bind-to-mgmt  https://review.opendev.org/75846113:23
*** rh-jelabarre has joined #openstack-ansible13:23
openstackgerritMerged openstack/openstack-ansible-os_tempest master: Migrate to content provider jobs/templates  https://review.opendev.org/75928713:29
*** dmsimard has quit IRC13:51
*** dmsimard has joined #openstack-ansible13:51
*** dave-mccowan has quit IRC14:00
*** cshen has quit IRC14:01
*** dmsimard has quit IRC14:03
*** dmsimard has joined #openstack-ansible14:03
*** spatel has joined #openstack-ansible14:13
evrardjphey, how is OSA dealing with the ansible service bug? https://github.com/ansible/ansible/pull/7233714:19
*** CeeMac has joined #openstack-ansible14:23
noonedeadpunkevrardjp: I think we don't now14:25
evrardjpyeah you probably don't have the bpf problem :)14:25
evrardjpthat's good news I guess14:25
noonedeadpunkI've herad about the issue but we kind of did nothing until it touched us (which didn't happen yet)14:27
noonedeadpunkand I think we don't have 5.8 kernel anywhere by default)14:33
*** jamesdenton has joined #openstack-ansible14:34
spateljamesdenton: i successfully deployed OVS in lab with OSA (i build br-vlan in linuxbridge manually, but i think i should let OVS to handle br-vlan creation)14:44
spatelnow i am going to add new compute node with OVS+DPDK support, lets see how it goes.14:44
jamesdentoncongrats! ovs can create br-vlan for you as an OVS bridge. but you can't have an ovs and lxb bridge named br-vlan simultaneously14:44
spateli did little mistake where i build br-vlan in linuxbridge and then attached it with br-provider (but anyway next time i will let OVS to drive that process)14:45
mgariepyjamesdenton, did you ever run the l3-agent with ovs in container?14:46
spatelno14:46
jamesdentonyou only really need br-provider and an interface in it. br-vlan (as a linuxbridge) would not be necessary14:46
spateli don't need l3-agent in my setup14:46
jamesdentonmgariepy i think we did a while back, but have been running neutron agents on baremetal now for a while.14:46
mgariepyok14:47
spatelin document i got little confused between br-provider and br-vlan14:47
jamesdentonunderstandable14:47
*** cshen has joined #openstack-ansible14:47
spatelin my setup i don't need br-vlan right?14:48
jamesdentonmore than likely not.14:49
spatelok14:49
spatellet me configured dpdk using OSA :) real fun start from here14:50
spatelI am following this guide hope this is right one - https://docs.openstack.org/openstack-ansible-os_neutron/latest/app-openvswitch-dpdk.html14:50
jamesdentonthat is the right one.14:50
jamesdentonkeep us posted14:51
spatelOh yes, i am only confused about how to deal with bonding but lets see..14:51
jamesdentonyeah, i wouldn't complicate your life with that now14:51
spatelk14:52
jamesdentoni recall wanting to do it with ansible, and tweaked the module, but the PR got lost in review hell: https://github.com/ansible/ansible/pull/5417714:52
jamesdentoni'll see if i can find the time to redo it in the new repo14:53
spatelthank you, so do you want me to try out with bonding or without bonding?14:54
jamesdentoni would try it without, for now.14:54
spatelwhat would be easy for first run?14:54
spatelok14:54
spatelgot it14:54
jamesdentoni wrote that bonding module so that eventually OSA could build the bond for you. i'll revisit that14:55
spatelyou are awesome! lets do it14:56
ThiagoCMCMorning guys! Sorry for double posting it... Is this: https://docs.openstack.org/openstack-ansible/ussuri/user/messaging/messaging.html <- supported already? I would like to minimize RabbitMQ usage in my new deployment.15:05
ThiagoCMCBTW James, too bad and Ansible pull 54177 wasn't implemented! I'm also using Ansible to configure OVS and I miss that functionality. It's especially hard to build OVS+DPDK bonds and bridges and doable...15:06
*** macz_ has joined #openstack-ansible15:09
dmsimardnoonedeadpunk: aggregating by path is kinda neat too: http://paste.openstack.org/raw/799607/15:16
spatelThiagoCMC: how do you running OVS+DPDK ? (without bonding?)15:28
ThiagoCMCI was running OVS+DPDK in a Lab with just QEMU (Libvirt), no OpenStack was involved...15:28
ThiagoCMCIt works great! But it consumes a lot of CPU / Energy15:29
spatelwhat do you mean with lot of CPU / Energy ?15:29
ThiagoCMCThe DPDK proccess uses about 200~400% of CPU all the time. I was using a Supermicro server, with 2 x 10G NIC cards (so, 2 CPU cores for each NIC + extra for multi queues)15:30
*** rfolco has quit IRC15:41
*** recyclehero has quit IRC15:48
spatelhmm, mostly we give dedicated CPU core for DPDK so it shouldn't eat other core resources15:52
ThiagoCMCYep, it doesn't... But still consumes electricity, even if no packets are passing15:53
spatelvery strange, i am also building dpdk so lets see15:53
ThiagoCMCCool! I was about to test this: https://blog.mellanox.com/2016/12/three-ways-asap2-beats-dpdk-for-cloud-and-nfv/ - but my priorities changed....15:55
kleiniI am not using DPDK but OVS for bonds and bridges at least on the compute and network nodes and it works really great. Debugging is a lot easier as the whole networking takes place in OVS.15:58
*** recyclehero has joined #openstack-ansible16:00
ThiagoCMCkleini, just be careful with OVN!  :-P16:00
kleiniAs I don't have an idea, what OVN is, I will stick to Open vSwitch. This satisfies all our needs.16:03
openstackgerritMerged openstack/openstack-ansible-os_nova master: Install qemu from buster-backports  https://review.opendev.org/76056716:06
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible master: Do not warn about incorrect group naming  https://review.opendev.org/76095216:10
ThiagoCMCOVN is an OpenvSwitch "feature set", you don't need Neutron-*-agents when using it, concentrating things even more on OpenvSwitch itself!16:11
ThiagoCMChttp://www.openvswitch.org//support/slides/OVN-Vancouver.pdf16:11
*** fanfi_ has joined #openstack-ansible16:13
mgariepyThiagoCMC, do you run it ?16:13
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible master: Fix upgrade jobs for bind-to-mgmt  https://review.opendev.org/75846116:15
ThiagoCMCI tried but then, I faces this: https://bugs.launchpad.net/neutron/+bug/1689880 - and stopped.16:17
openstackLaunchpad bug 1689880 in neutron "[OVN] The "neutron_sync_mode = repair" option breaks the whole cloud!" [Undecided,New]16:17
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible master: Fix upgrade jobs for bind-to-mgmt  https://review.opendev.org/75846116:17
ThiagoCMCYou have to be an OpenvSwitch expert to go OVS+OVN16:17
ThiagoCMCIf it fails, you're in trouble lol16:17
spatelkleini: are you using bonding inside OVS ?16:19
*** fanfi has joined #openstack-ansible16:21
*** mgariepy has quit IRC16:22
jamesdentonThiagoCMC i am actively reworking that PR into the new openvswitch collection module. we'll see how it goes16:23
*** yasemind20 has joined #openstack-ansible16:28
ThiagoCMCjamesdenton, nice!!!! I'm definitely willing to give it another shot!16:28
*** rfolco has joined #openstack-ansible16:30
*** yasemind20 has quit IRC16:34
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible master: Move nspawn to unmaintained state  https://review.opendev.org/76095416:36
spateljamesdenton: is it mandatory to have hugepagesz=1G  or i can set to hugepagesz=2M ?16:36
jamesdenton"it depends"16:38
jamesdentoni believe 2M should be OK16:39
spatelcool16:40
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible master: Improve repo server haproxy healthcheck to verify repo synchronisation  https://review.opendev.org/66047716:56
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-repo_server master: Use infra jobs for testing  https://review.opendev.org/76096016:59
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_manila master: Cleanup after repo_build and pip_install retirement  https://review.opendev.org/72765917:03
*** _kukacz has joined #openstack-ansible17:03
noonedeadpunkfolks, vote on https://review.opendev.org/#/c/755973/ is really needed17:06
*** rpittau is now known as rpittau|afk17:08
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible master: Added Openstack Adjutant role deployment  https://review.opendev.org/75631017:13
*** fanfi_ has quit IRC17:16
*** tosky has quit IRC17:28
spateljamesdenton: in this section http://paste.openstack.org/show/799611/17:31
spatelovs_dpdk_* all these variables for compute only or it should go out to neutron server also?17:32
*** mgariepy has joined #openstack-ansible17:33
fanfifolks, could somebody help me please. After fresh installation of OpenStack-Ansible, compute node is not able to join. In nova-compute log I see following error: Failed to discover available identity versions when contacting http://IP:5000/v3/. Attempting to parse version from URL. And ERROR nova17:36
fanfikeystoneauth1.exceptions.connection.UnknownConnectionError: Unexpected exception for http://IP:5000/v3/auth/tokens: Failed to parse: http://IP:5000/v3/auth/tokens17:36
spatelfanfi: it would be good if you post full logs trace here - http://paste.openstack.org/17:38
fanfispatel http://paste.openstack.org/show/799616/17:42
*** mmethot has quit IRC17:45
*** mmethot_ has joined #openstack-ansible17:45
spatelare you sure keystone is health?17:45
spatelor may be some python3-urllib3 conflict17:47
fanfispatel yes I am getting answer if I use curl17:47
spateli would say check logs for any exception, is this new compute node are you trying to add?17:48
fanfispatel this is myt first compute node ...it's fresh installation17:49
fanfispatel looks like manual installation python3-urllib3 fixed the issue ! thank you so much!17:54
jamesdentonspatel those overrides assume a homogenous environment, meaning same cards, same slots, same cpus, etc. i don't remember if dpdk is installed on the network hosts or not17:56
spateli have same hardware in my cloud including NIC17:57
*** sshnaidm|rover is now known as sshnaidm|afk17:57
spateljamesdenton: trying to understand ovs_dpdk_lcore_mask17:58
jamesdentoni need to make sure that's still required. but what it is/was is a binary mask of the core you want to reserve for host operating system functions18:06
jamesdentonthe docs have a non-formatted table18:06
spatelits mask for vm cores or its mask for DPDK core?18:08
jamesdentonhttp://paste.openstack.org/show/799620/18:08
jamesdentonlcore mask is OS, and pmd mask is for DPDK, the remaining cores (not defined) for the VMs18:09
spatelwhat is (dot) in that table?18:10
spatelThis is what i have for my compute - http://paste.openstack.org/show/799621/18:11
jamesdentonthe dot represents cores that were unset. just not enough room to draw a table that big18:12
jrosserfanfi: have you used openstack-absible to install nova/keystone/.... from source code or distribution packages? your paths are all /usr/lib/python3.6/site-packages/nova/... which looks unusual18:13
spateljamesdenton: i wish there is a script to do that18:15
jamesdentoneventually, there could be. ultimately, it would be nice to just define the cores in a list and let ansible figure it out18:16
*** andrewbonney has quit IRC18:16
*** fanfi has quit IRC18:20
*** miloa has quit IRC18:39
*** d34dh0r53 has quit IRC18:45
*** pcaruana has quit IRC19:11
*** elduderino80 has joined #openstack-ansible19:21
_kukaczjrosser: fanfi seems to be AFK. I know he did use OSA but I'm not aware of what installation source he picked19:21
openstackgerritMerged openstack/openstack-ansible-os_nova master: Do not symlink ceph libraries for distro path  https://review.opendev.org/76058319:24
*** kukacz_ has joined #openstack-ansible19:25
*** _kukacz has quit IRC19:29
ThiagoCMCDoes OSA with Ceph-Ansible (from OSA Playbooks) support erasure code Ceph pools?19:50
*** d34dh0r53 has joined #openstack-ansible19:52
*** cshen has quit IRC19:59
*** cshen has joined #openstack-ansible20:04
*** pcaruana has joined #openstack-ansible20:06
*** persia has quit IRC20:08
*** cshen has quit IRC20:09
*** persia has joined #openstack-ansible20:09
*** cshen has joined #openstack-ansible20:19
*** pcaruana has quit IRC20:19
*** tosky has joined #openstack-ansible20:27
*** MickyMan77 has quit IRC20:30
*** MickyMan77 has joined #openstack-ansible20:30
openstackgerritRafael Folco proposed openstack/openstack-ansible-os_tempest stable/ussuri: Switch tripleo jobs to content provider  https://review.opendev.org/76101920:36
openstackgerritRafael Folco proposed openstack/openstack-ansible-os_tempest stable/train: Switch tripleo jobs to content provider  https://review.opendev.org/76102120:39
*** farda has joined #openstack-ansible20:46
*** rfolco has quit IRC20:46
*** rfolco has joined #openstack-ansible20:51
jamesdentonjrosser could use your wisdom here: https://bugs.launchpad.net/openstack-ansible/+bug/190258520:55
openstackLaunchpad bug 1902585 in openstack-ansible "uwsgi socket backlog exceeded" [Undecided,New]20:55
*** rfolco has quit IRC20:56
*** farda has quit IRC20:59
*** elduderino80 has quit IRC21:00
*** rfolco has joined #openstack-ansible21:06
*** elduderino80 has joined #openstack-ansible21:49
*** rfolco has quit IRC21:59
*** fanfi has joined #openstack-ansible22:13
fanfijrosser19:13:42fanfi: have you used openstack-absible to install nova/keystone/.... from source code or distribution packages? your paths are all /usr/lib/python3.6/site-packages/nova/... which looks unusual22:14
fanfisorry for delay answer. i used as install method - distro22:15
*** cshen has quit IRC22:20
*** cshen has joined #openstack-ansible22:20
*** spatel has quit IRC22:45
*** jamesdenton has quit IRC23:31
*** kukacz_ has quit IRC23:34
*** cshen has quit IRC23:39
*** CeeMac has quit IRC23:42

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!