Monday, 2020-06-22

*** spotz has joined #openstack-ansible00:25
*** markvoelker has joined #openstack-ansible01:25
*** markvoelker has quit IRC01:29
*** cshen has joined #openstack-ansible01:52
*** cshen has quit IRC01:57
*** cshen has joined #openstack-ansible02:50
*** cshen has quit IRC02:55
*** cp- has quit IRC02:56
*** cp- has joined #openstack-ansible03:02
*** markvoelker has joined #openstack-ansible03:26
*** markvoelker has quit IRC03:30
*** markvoelker has joined #openstack-ansible04:14
*** markvoelker has quit IRC04:19
*** evrardjp has quit IRC04:33
*** evrardjp has joined #openstack-ansible04:33
*** udesale has joined #openstack-ansible05:33
*** miloa has joined #openstack-ansible05:55
*** markvoelker has joined #openstack-ansible06:15
*** markvoelker has quit IRC06:20
*** rpittau|afk is now known as rpittau06:21
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-plugins master: Do not use paramkio transport for delegated tasks  https://review.opendev.org/73721906:51
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-plugins master: Remove debug output from linear strategy plugin  https://review.opendev.org/73722006:51
*** sep has quit IRC06:53
*** sep has joined #openstack-ansible06:53
*** markvoelker has joined #openstack-ansible06:54
openstackgerritJonathan Rosser proposed openstack/openstack-ansible master: Increase default ansible forks from 5 to 20  https://review.opendev.org/73722106:56
*** pcaruana has joined #openstack-ansible06:56
*** markvoelker has quit IRC06:59
*** cshen has joined #openstack-ansible07:02
*** stingrayza has joined #openstack-ansible07:23
*** also_stingrayza has quit IRC07:25
*** tosky has joined #openstack-ansible07:29
*** sshnaidm|off is now known as sshnaidm|ruck07:29
*** janno_ has joined #openstack-ansible07:31
*** janno has quit IRC07:35
openstackgerritJonathan Rosser proposed openstack/openstack-ansible master: Increase default ansible forks from 5 to 20  https://review.opendev.org/73722107:50
*** raukadah is now known as chandankumar07:58
*** janno_ has quit IRC08:05
*** janno has joined #openstack-ansible08:07
admin0morning08:14
jrossernoonedeadpunk: we should take a look at the linear strategy08:15
jrossernoonedeadpunk: https://review.opendev.org/#/c/737220/1/strategy/linear.py08:15
jrosserthis is hugely expensive code08:15
jrosseri broke the tests with that so i've misunderstood what it's doing a bit08:16
*** markvoelker has joined #openstack-ansible08:24
*** markvoelker has quit IRC08:29
*** janno has quit IRC09:04
*** janno has joined #openstack-ansible09:04
noonedeadpunkSo it seems that we use it to "optimize" conditionals and delegations, but wondering if it do right jobs nowadays, considering that ansible moved forward a lot comparing to days when this has been written...09:05
noonedeadpunkand I guess what we see now, is that even skipping takes a lot of time, so tasks with failed conditions are still queued then?09:06
*** aedc has joined #openstack-ansible09:22
*** janno has quit IRC09:25
*** janno has joined #openstack-ansible09:25
admin0i am trying osa-integrated ceph ..  and stuck at ceph-mon : waiting for the monitor(s) to form the quorum...09:40
admin0what is the way to fix it ?09:40
admin0let me gist my config09:44
admin0https://gist.github.com/a1git/5590f44c198b68c190cc5ea6a9514090  -- my config .. stuck on ceph-mon : waiting for the monitor(s) to form the quorum...10:15
*** rpittau is now known as rpittau|bbl10:15
*** markvoelker has joined #openstack-ansible10:25
*** markvoelker has quit IRC10:30
openstackgerritMerged openstack/openstack-ansible-os_placement master: Cleanup ansible_python_interpreter  https://review.opendev.org/73626811:23
jrossernoonedeadpunk: sorry got in meetings11:27
jrosserbut yes i've been looking at whats taking time in our linear strategy11:27
jrosserthe conditionals optimisation doesnt really seem to save much, measurably11:27
jrosserthe use of paramiko for delegated tasks costs us extra time11:28
noonedeadpunkyeah, eventually I was wondering a week ago or so if we can disable it, as have a thought that this extends our connection time11:29
jrosserbut really the part dealing with physical_host_addrs is costing us double time11:29
noonedeadpunkbut, when I left only sshkey out of ssh auth methods - got exactly the same time for running a single task11:29
jrosseri used my existing AIO on centos-8 and am just re-running lxc-container-create playbook with everything already deployed11:29
jrossertrying to get a feel for the amount of overhead rather than actually time taken with 'changed' tasks11:30
noonedeadpunkI think it's fair method - was thinking about doing the same11:31
jrosserpretty amazing here https://review.opendev.org/#/c/737221/11:31
jrosserbuster is only 5 mins slower on LXC than metal11:31
jrosserbut could be wildly different nodes, so not really very scientific11:31
jrosseri might start taking more stuff out that we don't use anywhere11:33
noonedeadpunkhm, and you mean that argument about max connections for ssh is not valid anymore?11:34
jrosserthe bug there seems to be about what the defaults are, and boostrap_host incresases them to 100 already11:34
noonedeadpunkbut bootstrap host is for CI only?11:35
noonedeadpunkLike it won't adjust settings for compute hosts, let's say11:35
jrosserthats right, the default config is still whatever the host comes with11:37
jrosserlooking at the original but it seems this affects hosts with many containers11:37
jrosserso specifically infra i think, rather than compute11:38
noonedeadpunkyeah, but doesn't that mean we can get regrassion on non-aio deployments?11:40
noonedeadpunk*regression11:41
jrosserpossibly - we could look as adjusting the sshd config, or adding it to the deployer pre-requisites11:50
jrosserin the linked bug though they were running into issues at forks=4011:51
jrosserwhich is quite a large number11:51
*** rh-jelabarre has joined #openstack-ansible11:51
jrosseri think we should do *something* though, because just addinf -f=20 takes 20% off the runtime of my lxc-container-create test11:51
*** miloa has quit IRC11:56
openstackgerritMerged openstack/openstack-ansible stable/stein: ceph: run handlers after common  https://review.opendev.org/73699811:56
openstackgerritHervĂ© Beraud proposed openstack/openstack-ansible master: Prefer to use the name of mariadb rather mysql  https://review.opendev.org/73726311:58
openstackgerritHervĂ© Beraud proposed openstack/openstack-ansible master: Rename mysql to mariadb for registered var  https://review.opendev.org/73726312:00
*** miloa has joined #openstack-ansible12:03
openstackgerritMerged openstack/openstack-ansible-os_tempest master: Remove the neutron-fwaas since it retired  https://review.opendev.org/73609912:04
*** sep has quit IRC12:17
*** jbadiapa has joined #openstack-ansible12:19
*** rpittau|bbl is now known as rpittau12:22
*** udesale_ has joined #openstack-ansible12:25
*** markvoelker has joined #openstack-ansible12:26
*** udesale has quit IRC12:27
noonedeadpunkjrosser: can you look through that cherry-pick? https://review.opendev.org/#/c/731601/12:30
*** sep has joined #openstack-ansible12:30
*** markvoelker has quit IRC12:31
noonedeadpunkbtw I agree that we proably should increase number of workers12:31
noonedeadpunkmaybe we can do that as a part of openstack_hosts or smth...12:32
*** markvoelker has joined #openstack-ansible12:34
*** markvoelker has quit IRC12:39
openstackgerritMerged openstack/openstack-ansible-lxc_hosts master: copy the actual keyring  https://review.opendev.org/71888312:44
*** aedc_ has joined #openstack-ansible12:57
*** aedc has quit IRC12:59
jrossernoonedeadpunk: i think the swift stuff has been fixed properly upstream now13:04
jrosserhttps://review.opendev.org/#/c/737071/13:04
noonedeadpunkoh, so we can drop these nasty workarounds...13:04
noonedeadpunkhowever, we still need them for train13:05
*** cshen has quit IRC13:11
*** vakuznet has joined #openstack-ansible13:12
vakuznetafter 20.1.2 upgrade i'm getting "2020-06-22 07:24:41.838 201 ERROR octavia.network.drivers.neutron.base   File "/openstack/venvs/octavia-20.1.2/lib/python2.7/site-packages/neutronclient/client.py", line 107, in _cs_request13:14
vakuznet                                                                                    2020-06-22 07:24:41.838 201 ERROR octavia.network.drivers.neutron.base     raise exceptions.SslCertificateValidationError(reason=e)"13:15
vakuznetcert is self-singned generated by osa13:15
openstackgerritMerged openstack/openstack-ansible-os_octavia master: Cleanup ansible_python_interpreter  https://review.opendev.org/73626113:19
openstackgerritMerged openstack/openstack-ansible-os_ironic master: Cleanup ansible_python_interpreter  https://review.opendev.org/73626213:24
openstackgerritMerged openstack/openstack-ansible-os_swift master: Cleanup ansible_python_interpreter  https://review.opendev.org/73626613:28
jrossernoonedeadpunk: that swift patch is for train i think13:28
noonedeadpunkyep, it is13:29
noonedeadpunkbut we need to sort of revert it for master and ussuri?13:29
jrosserideally yes becasue we added quite a lot of stuff for the workaround i think13:29
jrosserunless there was also something else that needed two config files13:29
noonedeadpunkiirc it was the only reason...13:30
*** d34dh0r53 has joined #openstack-ansible13:34
admin0i need some pointers on how to fix this ceph-mon : waiting for the monitor(s) to form the quorum...13:39
*** cshen has joined #openstack-ansible13:41
*** cshen has quit IRC13:48
openstackgerritMerged openstack/openstack-ansible-ceph_client master: Cleanup ansible_python_interpreter  https://review.opendev.org/73626713:52
*** cshen has joined #openstack-ansible13:52
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-plugins master: Do not use paramkio transport for delegated tasks  https://review.opendev.org/73721913:55
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-plugins master: Remove physical_host_addrs parameter from ssh plugin  https://review.opendev.org/73722013:55
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-plugins master: Remove chroot connection from ssh plugin  https://review.opendev.org/73727313:55
*** d34dh0r53 has quit IRC14:01
*** mmethot has quit IRC14:02
*** mmethot has joined #openstack-ansible14:02
*** markvoelker has joined #openstack-ansible14:07
*** markvoelker has quit IRC14:12
openstackgerritMatthew Thode proposed openstack/openstack-ansible-lxc_hosts stable/ussuri: copy the actual keyring  https://review.opendev.org/73728014:28
openstackgerritMatthew Thode proposed openstack/openstack-ansible-lxc_hosts stable/train: copy the actual keyring  https://review.opendev.org/73162614:29
*** spatel has joined #openstack-ansible14:40
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-openstack_hosts master: Enable PowerTools repository for distro installs  https://review.opendev.org/73728415:00
*** miloa has quit IRC15:12
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/ansible-role-systemd_networkd master: Add CentOS 8 support  https://review.opendev.org/73712215:13
*** cshen has quit IRC15:17
*** spatel has quit IRC15:19
prometheanfirefunc tests still failing on train?15:24
prometheanfirehttps://review.opendev.org/731626 seems consistant15:24
*** markvoelker has joined #openstack-ansible15:32
*** markvoelker has quit IRC15:36
*** cshen has joined #openstack-ansible15:44
*** cshen has quit IRC15:50
*** gyee has joined #openstack-ansible15:55
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible master: [WIP] Centos-8 support  https://review.opendev.org/68962916:01
*** rpittau is now known as rpittau|afk16:01
*** sshnaidm|ruck is now known as sshnaidm|afk16:13
*** udesale_ has quit IRC16:21
*** markvoelker has joined #openstack-ansible16:24
*** markvoelker has quit IRC16:29
*** spatel has joined #openstack-ansible16:30
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-plugins master: Remove chroot connection from ssh plugin  https://review.opendev.org/73727316:33
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-plugins master: Remove physical_host_addrs parameter from ssh plugin  https://review.opendev.org/73722016:33
*** markvoelker has joined #openstack-ansible16:44
*** markvoelker has quit IRC16:48
*** cshen has joined #openstack-ansible16:50
*** cshen has quit IRC16:55
*** cshen has joined #openstack-ansible17:02
*** cshen has quit IRC17:06
*** markvoelker has joined #openstack-ansible17:07
*** markvoelker has quit IRC17:12
openstackgerritGeorgina Shippey proposed openstack/openstack-ansible-plugins master: Identity providers can be created with specifed domain  https://review.opendev.org/73565417:13
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-plugins master: Remove physical_host_addrs parameter from ssh plugin  https://review.opendev.org/73722017:22
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-openstack_hosts master: Fix CentOS functional test  https://review.opendev.org/73733117:25
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-openstack_hosts master: Enable PowerTools repository for distro installs  https://review.opendev.org/73728417:28
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-openstack_hosts master: Fix CentOS functional test  https://review.opendev.org/73733117:31
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-openstack_hosts master: Fix CentOS functional test  https://review.opendev.org/73733117:31
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-openstack_hosts master: Enable PowerTools repository for distro installs  https://review.opendev.org/73728417:32
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-plugins master: Remove chroot connection from ssh plugin  https://review.opendev.org/73727317:40
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-plugins master: Remove physical_host_addrs parameter from ssh plugin  https://review.opendev.org/73722017:40
jrossermnaser: check out the runtime on https://review.opendev.org/#/c/737221/ openstack-ansible-deploy-aio_lxc-debian-buster job18:00
jrosserguess which cloud that ran on?18:00
mnaserjrosser: you're the second person to tell me today, someone else thought their job actually didn't run at all ;)18:01
jrosseri thought my patch had a good effect18:01
jrosserturns out bigger iron :)18:01
mnaserjrosser: it's not out yet but amd epyc gen 2 systems with raid-0 local storage18:01
jrosserit's only 5mins slower than the equivalent metal job18:02
*** aedc_ has quit IRC18:03
*** aedc has joined #openstack-ansible18:06
jrossermnaser: you might be interested in this too https://review.opendev.org/#/c/737220/18:06
jrossereven though it fails one test it takes 50% off the re-runtime of playbooks/lxc-container-create.yml18:07
jrosseri.e for already deployed and no changes18:07
*** cshen has joined #openstack-ansible18:12
openstackgerritMatthew Thode proposed openstack/openstack-ansible-lxc_hosts master: move listchanges removal to where it's needed  https://review.opendev.org/73162818:13
openstackgerritMatthew Thode proposed openstack/openstack-ansible-lxc_hosts stable/ussuri: move listchanges removal to where it's needed  https://review.opendev.org/73734618:16
jrosserprometheanfire: usually we merge / cherry-pick.....18:17
prometheanfireI did cherry-pick (if you are talking about https://review.opendev.org/737346 )18:18
mnaserjrosser: that's massive, its one of those "ansible sucks at copying things over"18:18
prometheanfirehmm, not sure why the commit message didn't update18:18
prometheanfiremaybe it's because I needed to add -X to --continue (due to conflict)18:19
jrossermnaser: in my AIO just running it was WOAH! as the text flew past18:19
mnaserjrosser: more proof that ripping out the openstack_services.yml which gets included in every single playbook/task will have a huge speed-up18:20
jrosseri went through the whole strategy/linear and took all the various bits in/out and that one hurts massively18:20
mnaserhow many vars do we add to every single task with that alone...18:20
jrosserthe test that fails is covering this case https://review.opendev.org/#/c/550665/18:20
jrosserwhich i didnt even know was a thing.......18:21
jrosser"When delegating to a host outside of the inventory..." <- what is that?18:22
prometheanfireya, stripped it there too18:22
openstackgerritMatthew Thode proposed openstack/openstack-ansible-lxc_hosts stable/ussuri: move listchanges removal to where it's needed  https://review.opendev.org/73734618:25
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-plugins master: Remove physical_host_addrs parameter from ssh plugin  https://review.opendev.org/73722018:35
mnaserjrosser: sounds like a weird... thing.18:50
mnaserid rip it out.18:50
*** jeh has joined #openstack-ansible18:51
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-plugins master: Remove physical_host_addrs parameter from ssh plugin  https://review.opendev.org/73722019:01
*** cshen has quit IRC19:05
*** cshen has joined #openstack-ansible19:10
*** jeh has quit IRC19:12
*** cshen has quit IRC19:15
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-plugins master: Remove physical_host_addrs parameter from ssh plugin  https://review.opendev.org/73722019:25
*** aedc_ has joined #openstack-ansible19:25
*** markvoelker has joined #openstack-ansible19:25
*** aedc has quit IRC19:27
*** markvoelker has quit IRC19:30
*** spatel has quit IRC19:53
*** pcaruana has quit IRC20:42
*** cshen has joined #openstack-ansible20:52
*** markvoelker has joined #openstack-ansible20:55
*** cshen has quit IRC20:57
openstackgerritVadim Kuznetsov proposed openstack/openstack-ansible-os_neutron master: VLAN tag support for ovs port  https://review.opendev.org/73737520:58
*** markvoelker has quit IRC20:59
*** cshen has joined #openstack-ansible21:01
*** jbadiapa has quit IRC21:05
*** cshen has quit IRC21:05
*** spatel has joined #openstack-ansible21:13
*** spatel has quit IRC21:36
*** spatel has joined #openstack-ansible21:42
*** spatel has quit IRC21:46
*** spatel has joined #openstack-ansible21:52
*** pmacdonnell has quit IRC21:55
*** pmacdonnell has joined #openstack-ansible21:55
*** spatel has quit IRC22:10
*** spatel has joined #openstack-ansible22:28
*** spatel has quit IRC22:31
*** markvoelker has joined #openstack-ansible22:56
*** tosky has quit IRC22:59
*** markvoelker has quit IRC23:00
*** cshen has joined #openstack-ansible23:01
*** cshen has quit IRC23:06
*** cshen has joined #openstack-ansible23:41
*** cshen has quit IRC23:46

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!