| *** dpawlik has joined #openstack-kolla | 00:53 | |
| *** dpawlik has quit IRC | 00:58 | |
| *** negronjl has quit IRC | 01:04 | |
| *** negronjl has joined #openstack-kolla | 01:06 | |
| *** igordc has quit IRC | 01:25 | |
| *** guozijn has joined #openstack-kolla | 01:58 | |
| *** dpawlik has joined #openstack-kolla | 02:54 | |
| *** dpawlik has quit IRC | 02:58 | |
| *** trungnv has joined #openstack-kolla | 03:13 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Refactor NSS database var https://review.opendev.org/689083 | 03:38 |
|---|---|---|
| *** goldyfruit has quit IRC | 03:46 | |
| *** guozijn has quit IRC | 04:33 | |
| *** igordc has joined #openstack-kolla | 04:35 | |
| *** igordc has quit IRC | 04:35 | |
| *** igordc has joined #openstack-kolla | 04:35 | |
| *** Luzi has joined #openstack-kolla | 04:50 | |
| *** guozijn has joined #openstack-kolla | 04:55 | |
| *** trungnv has quit IRC | 05:00 | |
| *** skramaja has joined #openstack-kolla | 05:03 | |
| *** igordc has quit IRC | 05:05 | |
| *** pbing19 has joined #openstack-kolla | 05:07 | |
| *** hrw has quit IRC | 05:07 | |
| *** hrw has joined #openstack-kolla | 05:09 | |
| *** dpawlik has joined #openstack-kolla | 05:15 | |
| *** rohit02 has joined #openstack-kolla | 05:43 | |
| rohit02 | does kolla stein supports rabbitmq and internal endpoints over ssl | 05:44 |
| openstackgerrit | Merged openstack/kolla-ansible master: Neutron: add support to use legacy iptables https://review.opendev.org/685967 | 05:53 |
| openstackgerrit | Merged openstack/kolla master: Tidy up release notes for Train https://review.opendev.org/688968 | 05:53 |
| yoctozepto | rohit02: internal endpoints tls is train | 05:57 |
| yoctozepto | morning channel | 05:59 |
| *** cah_link has joined #openstack-kolla | 06:00 | |
| *** pbing19 has quit IRC | 06:05 | |
| *** pbing19 has joined #openstack-kolla | 06:18 | |
| *** dpawlik has quit IRC | 06:20 | |
| *** dpawlik has joined #openstack-kolla | 06:27 | |
| guozijn | morning | 06:39 |
| *** trungnv has joined #openstack-kolla | 06:39 | |
| guozijn | I have a question, why delete the namespace before the neutron l3 agent starts? | 06:42 |
| openstackgerrit | Merged openstack/kolla-ansible stable/stein: Zun: disable image validation https://review.opendev.org/686943 | 06:42 |
| guozijn | This will affect overlay network, which is very dangerous in production environment! | 06:42 |
| guozijn | In our own strategy, we got rid of the wrapper. | 06:45 |
| *** hjensas has joined #openstack-kolla | 06:58 | |
| *** tonythomas has joined #openstack-kolla | 06:58 | |
| yoctozepto | guozijn: what do you have in mind? | 07:00 |
| guozijn | Can we remove this wrapper? | 07:01 |
| guozijn | Use l3 agent directly. | 07:02 |
| yoctozepto | what wrapper? | 07:03 |
| *** gfidente has joined #openstack-kolla | 07:04 | |
| guozijn | neutron-l3-agent-wrapper.sh.j2 in role neutron | 07:04 |
| Blinkiz | Hello. I have question about the support_matrix. https://docs.openstack.org/kolla/latest/support_matrix.html | 07:05 |
| Blinkiz | I do not fully understand what the difference is between binary and source here. Take for example cinder. It says that the source is "T" (tested) but the binary is "C" (community) maintained. When I install cinder with Kolla-ansible, what am I using, binary or source? | 07:06 |
| *** pbing19 has quit IRC | 07:10 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Fix OpenSSL template https://review.opendev.org/689086 | 07:14 |
| yoctozepto | guozijn: ah, you mean this | 07:14 |
| yoctozepto | Blinkiz: you make the choice :D | 07:14 |
| yoctozepto | you either select binary (i.e. provided by distro) or source (provided by us upstream) | 07:15 |
| Blinkiz | yoctozepto, There is more "source" which is "T" then binary. Am guessing source is better.. | 07:15 |
| Blinkiz | aha, yeah. now I remember. | 07:15 |
| yoctozepto | binary is just a name for distro-provided | 07:15 |
| Blinkiz | I did that in the beginning, right :) | 07:15 |
| yoctozepto | we can't really be too much responsible for that | 07:15 |
| yoctozepto | as we don't run distros | 07:15 |
| yoctozepto | guozijn: so you are running neutron l3 agent in ha mode | 07:16 |
| yoctozepto | this happens on l3 agent restart and gets rid of previous l3 namespaces which are likely wrong | 07:17 |
| yoctozepto | this is not the only point that keeps the overlay network running - it only affects the router | 07:17 |
| guozijn | Yes, you're right. But it is unreasonable to delete all the namespaces. | 07:20 |
| Blinkiz | So support matrix is 34 "T" for centos and 30 "T" for ubuntu. Alright.. | 07:20 |
| Blinkiz | barbican, bifrost, redis and tacker is community supported on Ubuntu but supported on Centos. | 07:22 |
| yoctozepto | Blinkiz: yup | 07:22 |
| yoctozepto | there should be mistral in the same state I think | 07:23 |
| Blinkiz | oh, mistral also.. | 07:23 |
| *** pcaruana has joined #openstack-kolla | 07:26 | |
| *** pbing19 has joined #openstack-kolla | 07:27 | |
| yoctozepto | guozijn: I don't know what to recommend you really, the previous namespaces are likely wrong and assumed to have been kept up by other neutron l3 agents | 07:34 |
| yoctozepto | are you referring perhaps to a situation where all l3 agents are restarted in about the same time? | 07:34 |
| *** stingrayza has joined #openstack-kolla | 07:42 | |
| guozijn | If the restart is just the active one | 07:49 |
| guozijn | The vrrp protocol will switch the primary to another one… | 07:51 |
| guozijn | Another effect is that the agent will start slower. | 07:53 |
| guozijn | I'm going to rethink it, thank you for your help. :) | 07:54 |
| *** pbing19 has quit IRC | 07:57 | |
| *** rpittau|afk is now known as rpittau | 08:01 | |
| Blinkiz | Is there any problem with running Ubuntu 18.04 on the baremetal server and then using "kolla_base_distro: centos"? Should work or will work? Or do not do that? :) | 08:02 |
| yoctozepto | guozijn: thanks for making me think as well - if you feel this is important you may raise it as a bug to further discussion | 08:03 |
| mnasiadka | Blinkiz: in theory it should work, but openvswitch for example loads a kernel module ;) | 08:03 |
| yoctozepto | Blinkiz: no forced problem but it's not tested | 08:03 |
| Blinkiz | okay, thanks! | 08:04 |
| yoctozepto | so yeah, things like libvirt+qemu+kvm or ovs may start behaving erratic | 08:04 |
| yoctozepto | due to hidden assumptions | 08:04 |
| yoctozepto | not met in kernel | 08:04 |
| yoctozepto | and modules | 08:04 |
| stingrayza | morningo/ | 08:07 |
| stingrayza | s/go/g o/ | 08:07 |
| *** sri_ has joined #openstack-kolla | 08:12 | |
| Blinkiz | I have created a bug about the choice about kolla_base_distro. Maybe it will be changed in the future :) | 08:14 |
| *** k_mouza has joined #openstack-kolla | 08:17 | |
| *** dougsz has joined #openstack-kolla | 08:18 | |
| hrw | Blinkiz: link? | 08:18 |
| Blinkiz | https://bugs.launchpad.net/kolla-ansible/+bug/1848665 | 08:18 |
| openstack | Launchpad bug 1848665 in kolla-ansible "Kolla Linux distribution choice should be same as the host" [Undecided,New] | 08:18 |
| mnasiadka | yoctozepto: so what are we doing with this bloody fetch_ceph_keys.py? | 08:19 |
| yoctozepto | mnasiadka: deprecating :-) | 08:27 |
| mnasiadka | yoctozepto: yeah, in U - but first we need to fix it :) | 08:27 |
| yoctozepto | mnasiadka: removal in U :D | 08:27 |
| mnasiadka | yoctozepto: currently I see we have a problem with the upgrade... | 08:27 |
| yoctozepto | yeah, it's a painful path it seems | 08:27 |
| mnasiadka | yoctozepto: well, it's already deprecated, I meant removal | 08:27 |
| yoctozepto | so Train blocker would be ceph | 08:27 |
| yoctozepto | incredible xD | 08:27 |
| mnasiadka | well, python3 on Ubuntu - not ceph :D | 08:28 |
| yoctozepto | but our ceph support's technical debt | 08:28 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: [nova][train] Remove deprecated RetryFilter https://review.opendev.org/689337 | 08:31 |
| Blinkiz | I sometimes find myself questioning if Kolla is CI testing some feature or not. Am thinking about running linuxbridge instead of openvswitch. Is this tested by your CI? | 08:31 |
| yoctozepto | Blinkiz: nope | 08:32 |
| yoctozepto | but it's still very likely to work | 08:32 |
| mnasiadka | yoctozepto: are you going to return that debt today? :D | 08:32 |
| Blinkiz | yoctozepto, Thank you for the answer :) | 08:32 |
| yoctozepto | mnasiadka: I did not take the loan | 08:33 |
| mnasiadka | yoctozepto: me neither, but I guess it's a Swiss francs loan - we need to repay it, and then remove it :D | 08:33 |
| yoctozepto | mnasiadka: tbh, I feel pretty bad today, sore throat, feeling cold | 08:34 |
| mnasiadka | yoctozepto: any ideas how to fix the upgrade problem? | 08:34 |
| yoctozepto | mnasiadka: stop fetching the keys | 08:35 |
| yoctozepto | it is not needed | 08:35 |
| *** rgogunskiy has joined #openstack-kolla | 08:35 | |
| *** priteau has joined #openstack-kolla | 08:36 | |
| mnasiadka | ok, let me look into it after my mental breakdown with ovs | 08:40 |
| *** rohit02 has quit IRC | 08:41 | |
| *** hamzy_ has joined #openstack-kolla | 08:41 | |
| *** rohit02 has joined #openstack-kolla | 08:41 | |
| *** hamzy has quit IRC | 08:43 | |
| *** guozijn has quit IRC | 08:44 | |
| *** guozijn has joined #openstack-kolla | 08:46 | |
| *** dpawlik has quit IRC | 09:47 | |
| *** rohit02 has quit IRC | 09:54 | |
| *** rohit02 has joined #openstack-kolla | 09:55 | |
| hrw | heh... 48GB for /var/lib/docker was not enough. have to add remounting /var/lib/docker after each test | 09:55 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: Use versioned python binary with fetch ceph keyrings https://review.opendev.org/688392 | 10:00 |
| *** dpawlik has joined #openstack-kolla | 10:09 | |
| *** priteau has quit IRC | 10:13 | |
| Blinkiz | Hmm. I can not get mariadb to start after a kolla-ansible .. deploy command. https://paste.ubuntu.com/p/SjFkPr9b3Z/ | 10:13 |
| Blinkiz | mariadb_recovery does not work either.. Hmm | 10:13 |
| *** cally725 has quit IRC | 10:21 | |
| openstackgerrit | Doug Szumski proposed openstack/kolla-ansible master: Add unit tests for Nova Cells filters https://review.opendev.org/689413 | 10:24 |
| yoctozepto | Blinkiz: InnoDB: A downgrade from MariaDB 10.2.2 or later is not supported. | 10:25 |
| dougsz | yocotozepto: ^ your unit tests Sir | 10:25 |
| yoctozepto | you changed containers without clearing data | 10:25 |
| yoctozepto | thanks dougsz | 10:25 |
| dougsz | yw | 10:26 |
| yoctozepto | lgtm | 10:28 |
| dougsz | cheers, good to have for that nasty regex stuff | 10:30 |
| yoctozepto | indeed, is json format at least planned for this stuff? | 10:31 |
| yoctozepto | it's kind of lame to have to parse tables | 10:31 |
| *** guozijn has quit IRC | 10:34 | |
| *** sri_ has quit IRC | 10:36 | |
| *** sri_ has joined #openstack-kolla | 11:04 | |
| *** dpawlik has quit IRC | 11:09 | |
| *** kplant has joined #openstack-kolla | 11:12 | |
| *** trungnv has quit IRC | 11:30 | |
| *** dpawlik has joined #openstack-kolla | 11:56 | |
| dougsz | yoctozepto: Indeed, not supported for that command, we should raise it in Nova, can't be too much work | 12:15 |
| yoctozepto | dougsz: could you handle that? | 12:16 |
| yoctozepto | maybe ussuri could be cleaner then (and less error-prone) | 12:16 |
| dougsz | yeah, there are some other things which would be nice as well | 12:16 |
| yoctozepto | though this looks like an easy win | 12:17 |
| yoctozepto | changing output formatting | 12:17 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible stable/stein: Do not install ntp package on host when not used https://review.opendev.org/689435 | 12:20 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible stable/rocky: Do not install ntp package on host when not used https://review.opendev.org/689436 | 12:20 |
| openstackgerrit | Keith Plant proposed openstack/kolla master: Bump Openstack Exporter version to 0.6.0 https://review.opendev.org/689437 | 12:21 |
| *** larsks has quit IRC | 12:25 | |
| *** vmixor has joined #openstack-kolla | 12:26 | |
| *** rgogunskiy has quit IRC | 12:29 | |
| *** larsks has joined #openstack-kolla | 12:32 | |
| *** Luzi has quit IRC | 12:46 | |
| *** jdwidari has joined #openstack-kolla | 12:56 | |
| *** larsks has quit IRC | 13:00 | |
| *** larsks has joined #openstack-kolla | 13:01 | |
| kplant | any plans to upgrade to rabbitmq 3.8 in train? | 13:03 |
| openstackgerrit | Doug Szumski proposed openstack/kolla-ansible master: Add unit tests for Nova Cells filters https://review.opendev.org/689413 | 13:15 |
| *** rgogunskiy has joined #openstack-kolla | 13:16 | |
| *** skramaja has quit IRC | 13:19 | |
| *** goldyfruit has joined #openstack-kolla | 13:28 | |
| hrw | rather not | 13:30 |
| hrw | kplant: note that distros we use still have 3.7 | 13:31 |
| kplant | fair point | 13:31 |
| *** cah_link has quit IRC | 13:31 | |
| kplant | i'm looking at the builtin promtheus and grafana support in 3.8 | 13:32 |
| kplant | saves having to use a separate exporter | 13:32 |
| *** gfidente has quit IRC | 13:37 | |
| *** larsks has quit IRC | 13:44 | |
| *** larsks has joined #openstack-kolla | 13:46 | |
| *** dave-mccowan has joined #openstack-kolla | 13:46 | |
| *** gfidente has joined #openstack-kolla | 13:49 | |
| *** pcaruana has quit IRC | 13:53 | |
| *** ab-a has quit IRC | 13:53 | |
| mnasiadka | hrw: well, RDO in Train uses 3.6 | 13:54 |
| *** ab-a has joined #openstack-kolla | 13:54 | |
| kplant | i can play around with this: https://github.com/deadtrickster/prometheus_rabbitmq_exporter | 13:55 |
| kplant | can be used as a 3.7 plugin instead of a separate exporter | 13:55 |
| *** dave-mccowan has quit IRC | 13:56 | |
| *** munimeha1 has joined #openstack-kolla | 13:57 | |
| *** iclon__ has quit IRC | 14:11 | |
| *** dpawlik has quit IRC | 14:18 | |
| *** Blinkiz has quit IRC | 14:21 | |
| *** rohit02 has quit IRC | 14:21 | |
| *** larsks has quit IRC | 14:29 | |
| *** larsks has joined #openstack-kolla | 14:30 | |
| *** gfidente has quit IRC | 14:32 | |
| mnasiadka | kplant: well, I think somebody wanted to add this to Kolla's Docker collection, but we don't really want to support something, that comes from a ,,deadtrickster'' account on GitHub I think :D | 14:35 |
| *** gfidente has joined #openstack-kolla | 14:36 | |
| *** sri_ has quit IRC | 14:39 | |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI https://review.opendev.org/676376 | 14:44 |
| *** amoralej has joined #openstack-kolla | 14:49 | |
| amoralej | hi | 14:50 |
| amoralej | you have plans to do a RC release for train soon? | 14:50 |
| amoralej | we are preparing the last missing packages for RDO Train | 14:50 |
| amoralej | i know it's trailing-cycle and you still have time but having a RC would help... | 14:51 |
| *** rgogunskiy has quit IRC | 14:55 | |
| yoctozepto | amoralej: we will be cutting the stable branch next week | 15:01 |
| yoctozepto | you could use that | 15:01 |
| yoctozepto | RC might follow, might not | 15:01 |
| *** FlorianFa has quit IRC | 15:01 | |
| amoralej | ok | 15:01 |
| amoralej | usually branches are cut with rc1 | 15:01 |
| amoralej | you will not do it that way? | 15:01 |
| amoralej | i'm fine with creating package from commit snapshot | 15:02 |
| amoralej | but a tag is preferred | 15:02 |
| yoctozepto | amoralej: I don't remember if it's set in stone behavior | 15:02 |
| yoctozepto | you are packaging kolla? :-) | 15:03 |
| amoralej | not sure for trailing-cycle model | 15:03 |
| amoralej | yes we do | 15:03 |
| amoralej | in RDO | 15:03 |
| yoctozepto | so you are packaging the packaging of packaging | 15:03 |
| yoctozepto | as we build binary from RDO | 15:03 |
| yoctozepto | did not know RDO packaged kolla | 15:03 |
| yoctozepto | always something to learn | 15:04 |
| amoralej | yoctozepto, yes we package it so users can create the containers | 15:05 |
| amoralej | using RDO distro packages | 15:05 |
| yoctozepto | obviously :-) | 15:07 |
| yoctozepto | amoralej: do you also package kolla ansible? | 15:07 |
| amoralej | no | 15:07 |
| yoctozepto | I see | 15:08 |
| amoralej | if you are interested in what do we include in rdo there is a repo https://github.com/redhat-openstack/rdoinfo | 15:09 |
| amoralej | and a tool rdopkg | 15:09 |
| amoralej | in train we have https://github.com/redhat-openstack/rdoinfo/blob/master/tags/train.yml | 15:09 |
| amoralej | that's the full list of openstack packages we include | 15:09 |
| yoctozepto | ah, puppet, the good old times | 15:10 |
| amoralej | :) | 15:10 |
| billp | has anyone seen high cpu usage in the rabbit containers for the beam.smp process and if so what were fixes? | 15:10 |
| yoctozepto | billp: multicore cpu? | 15:10 |
| yoctozepto | rabbitmq has some ugly scheduling which uselessly takes cpu time | 15:11 |
| yoctozepto | display cpu times per thread and they should be low | 15:11 |
| billp | yeah, I was just reading up on runtime tuning. I also read where docker does not limit cpu but I did not think about rabbitmq limiting itself | 15:18 |
| yoctozepto | it's just doing some funky erlang stuff | 15:18 |
| yoctozepto | I really wish we could replace rabbitmq with something else | 15:19 |
| noxoid | arent kafka images already built by kolla? | 15:21 |
| noxoid | theres also qdrouterd it looks like which can use the amqp driver | 15:22 |
| noxoid | but yea im looking at switching to a different message broker. the only unexplained issues ive had with openstack are related to rabbitmq falling over | 15:23 |
| yoctozepto | noxoid: is openstack tested with anything else? | 15:25 |
| openstackgerrit | Gowrishankar proposed openstack/kolla master: Provision building netcontrold container https://review.opendev.org/689090 | 15:26 |
| yoctozepto | (than rabbitmq) | 15:26 |
| noxoid | nothing more than rabbitmq as far as i know | 15:26 |
| noxoid | :( | 15:26 |
| yoctozepto | well, we can always be the pioneers | 15:28 |
| noxoid | true. and the other drivers are still CI tested by oslo messaging https://github.com/openstack/oslo.messaging/blob/master/.zuul.yaml | 15:31 |
| *** igordc has joined #openstack-kolla | 15:32 | |
| yoctozepto | amqp1 is for qdrouterd? | 15:38 |
| noxoid | should be yes | 15:39 |
| noxoid | apache qpid | 15:39 |
| yoctozepto | then yes | 15:42 |
| yoctozepto | wonder if putting it first means something ;-) | 15:43 |
| noxoid | well they also put py27 first lol | 15:43 |
| yoctozepto | noxoid: that is expected | 15:44 |
| yoctozepto | you usually count in growing order | 15:44 |
| noxoid | fair | 15:45 |
| yoctozepto | but they might have also applied it alphabetically | 15:45 |
| yoctozepto | amqp, kafka, rabbit | 15:45 |
| yoctozepto | so who knows | 15:45 |
| yoctozepto | important is that it is tested | 15:45 |
| yoctozepto | kafka sounds like an overkill to met | 15:45 |
| yoctozepto | to me* | 15:45 |
| yoctozepto | also considering it is designed for longer term queues | 15:46 |
| yoctozepto | openstack's are to be emptied soon | 15:46 |
| noxoid | very true. i would hope that since amqp is a protocol spec that any software can implement, that it would be just as tested/supported as rabbit which has its own driver because it uses an older version of the amqp spec | 15:46 |
| yoctozepto | apache qpid written in C sounds better for my debugging than rabbitmq written in erlang | 15:46 |
| yoctozepto | also qpid is much lighter weight | 15:47 |
| yoctozepto | I guess it might be just right | 15:47 |
| noxoid | yep | 15:47 |
| yoctozepto | it seems RDO packages qpid as well (looks at amoralej) | 15:47 |
| noxoid | i looked at seeing what it would take switching over to qpid from rabbit using k-a and it didnt seem like that much work | 15:47 |
| noxoid | k-a shoves the transport url into a single var which could be overridden | 15:47 |
| yoctozepto | I don't think you can switch a running installation though ;p | 15:47 |
| amoralej | yoctozepto, yes, we included qpid but default is rabbit | 15:48 |
| noxoid | eh rabbit has caused enough random outages for me lol | 15:48 |
| noxoid | i may get fed up one day and switch on a whim | 15:48 |
| yoctozepto | amoralej: thanks for insights | 15:50 |
| yoctozepto | noxoid: well, my cluster is still in usefulness testing, rabbitmq is pretty stable but I am thinking long term as well | 15:50 |
| amoralej | glad to answer your questions if you need something | 15:50 |
| yoctozepto | :-) | 15:51 |
| noxoid | it would definitely be nice as an alternative. it could also just be i have an issue elsewhere in my network causing issues and im incorrectly blaming rabbitmq | 15:51 |
| noxoid | but the aforementioned cpu usage is a pain too | 15:51 |
| dougsz | https://review.opendev.org/#/c/688432/ | 15:52 |
| dougsz | https://review.opendev.org/689413 | 15:52 |
| dougsz | ^ A couple of cells patches if anyone has a minute :) | 15:52 |
| yoctozepto | noxoid: https://bugs.launchpad.net/kolla-ansible/+bug/1847727 if you like bug-themed soap operas in multiple acts | 15:52 |
| openstack | Launchpad bug 1847727 in kolla-ansible "CI: failing on CRITICAL may be still too strong" [Medium,Triaged] | 15:52 |
| yoctozepto | mnasiadka: ^^^ (dougsz) | 15:53 |
| noxoid | ooo | 15:53 |
| dougsz | yargh | 15:53 |
| yoctozepto | dougsz: glad you found and fixed the issue with unit tests | 15:58 |
| yoctozepto | hrw: https://zuul.opendev.org/t/openstack/build/5c64183ea72d46778efc6abc6c9d09e9 behaved oddly, please check | 15:58 |
| dougsz | yoctozepto: myself to blame! | 15:59 |
| yoctozepto | dougsz: me too, I could have found that but I was too programmed to use context as the first argument and did not see the problem :-) | 16:01 |
| yoctozepto | either way, great we are churning | 16:01 |
| dougsz | indeed, thanks for taking a look :) | 16:01 |
| yoctozepto | :-) | 16:02 |
| *** rpittau is now known as rpittau|afk | 16:04 | |
| hrw | ED25519 host key for 104.130.207.161 has changed and you have requested strict checking. | 16:07 |
| hrw | Host key verification failed. | 16:07 |
| yoctozepto | noxoid: now with that rabbitmq on the plate I wonder whether the random issues are not from rabbitmq :-) | 16:09 |
| noxoid | possible | 16:10 |
| noxoid | that bug report has similar symptoms to what i saw | 16:12 |
| noxoid | keepalived bugging out would explain my errors | 16:15 |
| noxoid | AMQP server on 10.101.0.3:5672 is unreachable: Server unexpectedly closed connection. Trying again in 1 seconds.: IOError: Server unexpectedly closed connection | 16:15 |
| noxoid | 2019-10-14 14:17:03.314 64 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: ConnectionForced: Too many heartbeats missed | 16:15 |
| noxoid | but i need to look into it more at another time | 16:15 |
| noxoid | thanks for the link though | 16:16 |
| yoctozepto | nah, checked the logs | 16:16 |
| yoctozepto | they are pure | 16:16 |
| yoctozepto | I've already crossed out mariadb | 16:16 |
| yoctozepto | but forgot about rabbitmq | 16:16 |
| yoctozepto | now crossed it out too | 16:16 |
| yoctozepto | so we are left with sheer slowness (haproxy timeout) + keepalived bug | 16:17 |
| yoctozepto | and then upgrade taking down one service is obviously likely to cause timeouts | 16:17 |
| yoctozepto | should not apply to deploy/reconfigure with no changes | 16:17 |
| yoctozepto | but it does = action load | 16:18 |
| yoctozepto | mnasiadka: both ceph's CR+2 | 16:32 |
| yoctozepto | I had some serious mind block in the kolla case | 16:33 |
| yoctozepto | explanation is in comment | 16:33 |
| yoctozepto | hrw: https://review.opendev.org/688392 - and its dependency please | 16:33 |
| yoctozepto | so that we can again look at those other random failures on ubuntu ceph | 16:34 |
| yoctozepto | and not permabroken | 16:34 |
| hrw | yoctozepto: LGTM | 16:35 |
| yoctozepto | hrw: ;D | 16:36 |
| yoctozepto | mnasiadka: so it will land | 16:36 |
| yoctozepto | really minimal | 16:36 |
| yoctozepto | 2 problems in 3 lines changed | 16:36 |
| mnasiadka | Awesome | 16:37 |
| yoctozepto | I expect the same efficiency in the future :D | 16:37 |
| *** igordc has quit IRC | 16:40 | |
| yoctozepto | ah, you only missed to link it to the bug | 16:42 |
| yoctozepto | it's at the gates so I'll mark it manually | 16:42 |
| *** iclon has joined #openstack-kolla | 16:44 | |
| yoctozepto | done | 16:44 |
| *** k_mouza_ has joined #openstack-kolla | 16:44 | |
| mnasiadka | Thanks | 16:46 |
| openstackgerrit | Pierre Riteau proposed openstack/kolla-ansible stable/stein: Fixes Monasca log transformer UTC offset exception https://review.opendev.org/689544 | 16:47 |
| openstackgerrit | Pierre Riteau proposed openstack/kolla-ansible stable/rocky: Fixes Monasca log transformer UTC offset exception https://review.opendev.org/689545 | 16:48 |
| *** k_mouza has quit IRC | 16:48 | |
| * hrw off | 16:48 | |
| *** k_mouza_ has quit IRC | 16:48 | |
| *** iclon has quit IRC | 16:51 | |
| *** iclon has joined #openstack-kolla | 16:51 | |
| *** iclon has quit IRC | 16:53 | |
| *** dougsz has quit IRC | 16:58 | |
| openstackgerrit | Will Szumski proposed openstack/kolla-ansible master: Limit open file descriptors for Neutron agent containers https://review.opendev.org/689548 | 16:58 |
| *** iclon has joined #openstack-kolla | 17:00 | |
| *** iclon has quit IRC | 17:11 | |
| kplant | mnasiadka: totally agree | 17:43 |
| kplant | would prefer the builtin plugin in 3.8 :] | 17:43 |
| *** amoralej is now known as amoralej|off | 17:44 | |
| mnasiadka | kplant: well, we moved to upstream mariadb, we can move to upstream rmq in U if it makes sense, propose it on some meeting :) or override it locally and build your own images | 17:56 |
| kplant | yeah i think that's the right path. hopefully for the former prevails | 17:58 |
| kplant | i think operators really should be looking at rabbitmq | 17:58 |
| kplant | not just me | 17:59 |
| *** k_mouza has joined #openstack-kolla | 18:13 | |
| openstackgerrit | Merged openstack/kolla master: Fix fetch_ceph_keys to run in python3 https://review.opendev.org/688636 | 18:14 |
| *** igordc has joined #openstack-kolla | 18:17 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Use versioned python binary with fetch ceph keyrings https://review.opendev.org/688392 | 18:21 |
| openstackgerrit | Merged openstack/kayobe stable/rocky: Set provisioning and cleaning networks as non-shared https://review.opendev.org/686361 | 18:26 |
| *** negronjl has quit IRC | 18:27 | |
| *** negronjl has joined #openstack-kolla | 18:30 | |
| *** k_mouza has quit IRC | 18:33 | |
| *** k_mouza has joined #openstack-kolla | 18:35 | |
| *** k_mouza has quit IRC | 18:38 | |
| *** kplant has quit IRC | 19:50 | |
| *** vmixor has quit IRC | 19:51 | |
| *** mixor has joined #openstack-kolla | 19:51 | |
| *** igordc has quit IRC | 19:54 | |
| *** igordc has joined #openstack-kolla | 20:23 | |
| *** jcmdln has quit IRC | 20:31 | |
| *** k_mouza has joined #openstack-kolla | 20:38 | |
| *** k_mouza has quit IRC | 20:42 | |
| *** mixor has quit IRC | 20:58 | |
| *** goldyfruit has quit IRC | 21:10 | |
| *** jbadiapa has quit IRC | 21:50 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!