| *** nurdie has joined #openstack-ansible | 00:32 | |
| *** macz_ has joined #openstack-ansible | 00:37 | |
| *** macz_ has quit IRC | 00:42 | |
| *** gyee has quit IRC | 00:44 | |
| *** MickyMan77 has joined #openstack-ansible | 01:03 | |
| *** MickyMan77 has quit IRC | 01:11 | |
| *** cshen has joined #openstack-ansible | 01:16 | |
| *** cshen has quit IRC | 01:21 | |
| *** Muran has joined #openstack-ansible | 01:26 | |
| *** Muran has quit IRC | 01:30 | |
| *** MickyMan77 has joined #openstack-ansible | 01:44 | |
| *** recyclehero has quit IRC | 01:45 | |
| *** MickyMan77 has quit IRC | 01:53 | |
| *** jawad_axd has joined #openstack-ansible | 01:54 | |
| *** jawad_axd has quit IRC | 01:58 | |
| *** jawad_axd has joined #openstack-ansible | 02:15 | |
| *** cshen has joined #openstack-ansible | 02:17 | |
| *** jawad_axd has quit IRC | 02:20 | |
| *** cshen has quit IRC | 02:21 | |
| *** MickyMan77 has joined #openstack-ansible | 02:23 | |
| *** MickyMan77 has quit IRC | 02:32 | |
| *** ChiTo has quit IRC | 02:34 | |
| *** jawad_axd has joined #openstack-ansible | 02:35 | |
| *** jawad_axd has quit IRC | 02:40 | |
| *** jawad_axd has joined #openstack-ansible | 02:56 | |
| *** jawad_axd has quit IRC | 03:01 | |
| *** MickyMan77 has joined #openstack-ansible | 03:07 | |
| *** jawad_axd has joined #openstack-ansible | 03:17 | |
| *** jawad_axd has quit IRC | 03:21 | |
| *** jawad_axd has joined #openstack-ansible | 03:38 | |
| *** jawad_axd has quit IRC | 03:42 | |
| *** d34dh0r53 has quit IRC | 03:44 | |
| *** jawad_axd has joined #openstack-ansible | 03:58 | |
| *** jawad_axd has quit IRC | 04:03 | |
| *** evrardjp has quit IRC | 04:34 | |
| *** evrardjp has joined #openstack-ansible | 04:34 | |
| *** nurdie has quit IRC | 04:38 | |
| *** gillesMo has quit IRC | 04:43 | |
| *** recyclehero has joined #openstack-ansible | 04:57 | |
| *** jawad_axd has joined #openstack-ansible | 05:00 | |
| *** jawad_axd has quit IRC | 05:05 | |
| *** gillesMo has joined #openstack-ansible | 05:07 | |
| *** MickyMan77 has quit IRC | 05:16 | |
| *** nurdie has joined #openstack-ansible | 05:39 | |
| *** nurdie has quit IRC | 05:43 | |
| *** macz_ has joined #openstack-ansible | 06:02 | |
| *** macz_ has quit IRC | 06:07 | |
| *** yolanda has quit IRC | 06:14 | |
| *** rpittau|afk is now known as rpittau | 06:21 | |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-lxc_hosts master: Use lxc_image_cache_server_mirrors as image source https://review.opendev.org/759310 | 06:23 |
|---|---|---|
| *** jawad_axd has joined #openstack-ansible | 06:40 | |
| *** Muran has joined #openstack-ansible | 06:50 | |
| *** Muran has quit IRC | 06:53 | |
| *** Muran has joined #openstack-ansible | 06:54 | |
| noonedeadpunk | recyclehero: this means that allocation ratio the way you've configured is not working properly... as here's what I see in my sandbox horizon https://imgur.com/a/rHKnpcL | 06:55 |
| noonedeadpunk | so horizon should not be point of verification if allocation ratios are set correctly or not - that's what I'm saying | 06:56 |
| noonedeadpunk | well, it's really interesting why it get reverted back | 06:59 |
| noonedeadpunk | I think that it worth asking in #openstack-nova - maybe they will suggest something regarding config | 07:00 |
| noonedeadpunk | and morning everyone:) | 07:06 |
| *** andrewbonney has joined #openstack-ansible | 07:13 | |
| *** cshen has joined #openstack-ansible | 07:27 | |
| *** tosky has joined #openstack-ansible | 07:36 | |
| *** Muran has quit IRC | 07:39 | |
| *** Muran has joined #openstack-ansible | 07:40 | |
| *** Muran has quit IRC | 07:41 | |
| *** Muran has joined #openstack-ansible | 07:41 | |
| *** shyamb has joined #openstack-ansible | 07:44 | |
| *** Muran has quit IRC | 07:49 | |
| *** Muran has joined #openstack-ansible | 07:50 | |
| *** shyamb has quit IRC | 07:53 | |
| *** shyamb has joined #openstack-ansible | 07:54 | |
| *** Muran has quit IRC | 07:54 | |
| CeeMac | morning | 07:56 |
| *** shyamb has quit IRC | 07:57 | |
| CeeMac | i think there has been ongoing 'discussions' between horizon and nova people over the years on the fact the CPU graphs in Horizon don't reflect the commit ratio of CPUs. Its something to do with the APIs Horizon uses to pull the resource information that speaks to a component that doesn't have any visibility of the over commit values, so can only report true CPU/core count | 07:57 |
| CeeMac | i looked into it a while back but never reached a solid conclusion on how best to check the 'actual' used/free CPUs based on the over commit | 07:58 |
| CeeMac | recyclehero: ^ | 07:58 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible master: Use parallel git clone https://review.opendev.org/588372 | 07:59 |
| *** jawad_axd has quit IRC | 08:37 | |
| *** jawad_axd has joined #openstack-ansible | 08:38 | |
| noonedeadpunk | wondering what does `WARNING urllib3.connectionpool: Connection pool is full, discarding connection: localhost` mean during ansible runtime... | 08:44 |
| noonedeadpunk | ie https://zuul.openstack.org/stream/6db0a6695ce243eea881113370cb16f1?logfile=console.log | 08:44 |
| noonedeadpunk | oh, can this be ara? | 08:45 |
| noonedeadpunk | dmsimard ? | 08:46 |
| noonedeadpunk | output from console http://paste.openstack.org/show/799307/ | 08:47 |
| noonedeadpunk | and play pretty stuck that way | 08:48 |
| noonedeadpunk | I saw that in for other tasks in random places as well | 08:51 |
| noonedeadpunk | oh, well, https://2e774ffe9e481649075b-3c2a18acb5109e625907972e3aa6a592.ssl.cf1.rackcdn.com/759229/4/check/openstack-ansible-lxc-btrfs-ubuntu-bionic/496af20/job-output.txt | 08:52 |
| *** cshen_ has joined #openstack-ansible | 08:56 | |
| *** cshen has quit IRC | 08:59 | |
| *** jawad_ax_ has joined #openstack-ansible | 09:13 | |
| *** jawad_axd has quit IRC | 09:15 | |
| *** Muran has joined #openstack-ansible | 09:38 | |
| *** odyssey4me has joined #openstack-ansible | 09:40 | |
| *** hamzaachi has joined #openstack-ansible | 09:42 | |
| openstackgerrit | Arx Cruz proposed openstack/openstack-ansible-os_tempest master: Re-adding redhat-7.yml distro var https://review.opendev.org/758823 | 09:42 |
| *** sep has quit IRC | 09:43 | |
| openstackgerrit | Arx Cruz proposed openstack/openstack-ansible-os_tempest master: Re-adding redhat-7.yml distro var https://review.opendev.org/758823 | 09:43 |
| *** Muran has quit IRC | 09:45 | |
| *** Muran has joined #openstack-ansible | 09:48 | |
| *** Muran has quit IRC | 09:53 | |
| *** sep has joined #openstack-ansible | 10:03 | |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_barbican master: Clean up barbican.conf https://review.opendev.org/759084 | 10:06 |
| *** sep has quit IRC | 10:07 | |
| *** sep has joined #openstack-ansible | 10:08 | |
| *** Muran has joined #openstack-ansible | 10:08 | |
| *** pto has joined #openstack-ansible | 10:10 | |
| *** Muran has quit IRC | 10:20 | |
| *** pto has quit IRC | 10:22 | |
| *** pto has joined #openstack-ansible | 10:22 | |
| *** ioni has quit IRC | 10:24 | |
| *** masterpe has quit IRC | 10:24 | |
| *** fridtjof[m] has quit IRC | 10:24 | |
| recyclehero | morning guys, it was a long night! | 10:32 |
| recyclehero | thanks noonedeadpunk CeeMac | 10:32 |
| recyclehero | I will head on to nova | 10:32 |
| noonedeadpunk | jrosser: sorry, I know you are away today, but need to ask about https://review.opendev.org/#/c/759229/ vs https://review.opendev.org/#/c/759310/ as we need to unblock gates and do point releases... I feel like 759310 is more reliable tbh, despite it's not base but minimal image. And we should move debian to it as well probably, but there's circular dependency with lxc_containers_create so planing to do that for master only | 10:33 |
| noonedeadpunk | the only thing that rerally concerns me is centos execution time | 10:33 |
| noonedeadpunk | not sure if it's because of the image though | 10:34 |
| *** ioni has joined #openstack-ansible | 10:34 | |
| *** pto has quit IRC | 10:37 | |
| *** pto has joined #openstack-ansible | 10:38 | |
| jrosser | noonedeadpunk: hi, just only on my phone for a while, but the runtime is quite concerning | 10:46 |
| jrosser | even just the infra job takes very very long | 10:47 |
| noonedeadpunk | I think we might have issue with ara as well, but have no idea what exactly influence it so badly | 10:47 |
| jrosser | we can always pin it back just to take out a variable | 10:48 |
| noonedeadpunk | not really... as to do that we need one of these patches to merge:) | 10:49 |
| noonedeadpunk | I think I will try in single sandbox to get container on new and old image to see the difference | 10:50 |
| *** yolanda has joined #openstack-ansible | 10:57 | |
| *** masterpe has joined #openstack-ansible | 10:59 | |
| *** fridtjof[m] has joined #openstack-ansible | 10:59 | |
| *** pto has quit IRC | 10:59 | |
| noonedeadpunk | what is interesting - functional job is not outstanding. and in console issues seemed to happen only when interacting with containers | 11:06 |
| *** shyamb has joined #openstack-ansible | 11:08 | |
| *** shyam89 has joined #openstack-ansible | 11:08 | |
| *** shyam89 has quit IRC | 11:08 | |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_manila master: Switch CI job for centos to centos-8 https://review.opendev.org/754652 | 11:15 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_manila stable/ussuri: Add centos-8 support https://review.opendev.org/759396 | 11:16 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_manila master: Add Ubuntu Focal CI jobs https://review.opendev.org/754653 | 11:18 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_manila master: Add image download for manila https://review.opendev.org/704972 | 11:19 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_manila master: Add image download for manila https://review.opendev.org/704972 | 11:21 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_manila master: Updated from OpenStack Ansible Tests https://review.opendev.org/754172 | 11:22 |
| *** pto has joined #openstack-ansible | 11:24 | |
| dmsimard | noonedeadpunk: the connection pool warning comes from ara | 11:27 |
| dmsimard | it stays stuck ? | 11:27 |
| noonedeadpunk | yep | 11:27 |
| dmsimard | damn, https://github.com/ansible-community/ara/blob/master/ara/plugins/callback/ara_default.py#L180 | 11:28 |
| dmsimard | i didn't seem to be able to reproduce the issue with 10, I guess it needs to be reduced | 11:29 |
| noonedeadpunk | well, we probably have more ansible threads now then 10 | 11:29 |
| dmsimard | number of ansible forks shouldn't matter | 11:30 |
| noonedeadpunk | or exactly 10 (not sure hwat has merged) | 11:30 |
| dmsimard | I need to send kids at school and then I'll figure a fix, sorry about that | 11:31 |
| *** gshippey has joined #openstack-ansible | 11:31 | |
| dmsimard | if it's super problematic, please pin to 1.5.1 temporarily | 11:31 |
| dmsimard | i was too optimistic :/ | 11:32 |
| noonedeadpunk | I wish we could lol | 11:32 |
| dmsimard | what do you mean ? | 11:33 |
| noonedeadpunk | ubuntu has dropped their base images several days ago and gates are broken. so to fix them and pin ara we need too unblock gates first which is hard because of ara | 11:34 |
| noonedeadpunk | but yeah, we will try:) | 11:34 |
| dmsimard | oh no | 11:34 |
| noonedeadpunk | anyway thanks for looking into that | 11:34 |
| pto | Has something been changed? I am doing a fresh install of 21.0.1 on ubuntu focal. The TASK [lxc_hosts : Unpack base image] fail with fatal: [os_infra1]: FAILED! => {"changed": false, "msg": "Source '/tmp/ubuntu-base-20.04-base-amd64.tar.gz' does not exist"} | 11:44 |
| *** shyam89 has joined #openstack-ansible | 11:44 | |
| noonedeadpunk | pto: yeah, ubuntu dropped images.... | 11:44 |
| *** shyam89 has quit IRC | 11:44 | |
| pto | noonedeadpunk: oh... How to get around it? | 11:44 |
| noonedeadpunk | try setting lxc_hosts_container_image_download_legacy: true | 11:44 |
| noonedeadpunk | we will make release that fixes it soon | 11:45 |
| pto | Should i go for 21.0.1 tag or the latest ussuri/master | 11:45 |
| noonedeadpunk | as for now it's not even merged... | 11:47 |
| pto | Generally speaking, which is most "stable" | 11:47 |
| noonedeadpunk | so enabling lxc_hosts_container_image_download_legacy or defining valid url with lxc_hosts_container_image_url | 11:47 |
| *** shyamb has quit IRC | 11:47 | |
| noonedeadpunk | are the best options at the moment tbh | 11:48 |
| noonedeadpunk | we have several patches which covers that but none of them merged yet | 11:48 |
| noonedeadpunk | so all current releases are broken at the moment in the same way | 11:49 |
| pto | Thanks for clarifying this. | 11:52 |
| *** rfolco has joined #openstack-ansible | 11:53 | |
| *** pto has quit IRC | 11:53 | |
| *** pto has joined #openstack-ansible | 11:53 | |
| *** rh-jelabarre has joined #openstack-ansible | 11:59 | |
| *** pto_ has joined #openstack-ansible | 12:04 | |
| *** pto has quit IRC | 12:05 | |
| *** NewJorg has quit IRC | 12:10 | |
| *** nurdie has joined #openstack-ansible | 12:16 | |
| *** NewJorg has joined #openstack-ansible | 12:16 | |
| kleini | for me applying patch https://review.opendev.org/759229 to /etc/ansible/roles/lxc_hosts solves the problem for 21.1.0 on Ubuntu 18.04 very well. but this gets overwritten with every bootstrapping of Ansible | 12:22 |
| noonedeadpunk | we have also https://review.opendev.org/#/c/759310 as second option, but there's issue with centos image at the moment | 12:31 |
| *** nurdie has quit IRC | 12:37 | |
| dmsimard | noonedeadpunk: hey o/ I reproduced the "oddly getting stuck" and I can't reproduce it anymore by lowering the amount of threads so we'll go with that for now | 12:52 |
| dmsimard | only happens with the default offline client, doesn't occur with the http client :/ | 12:52 |
| noonedeadpunk | we're lucky... | 12:54 |
| dmsimard | hmmm, nope, I managed to reproduce the stuck issue with less threads too | 13:00 |
| dmsimard | doesn't happen all the time | 13:00 |
| *** pto_ has quit IRC | 13:05 | |
| *** pto has joined #openstack-ansible | 13:05 | |
| noonedeadpunk | it totally does not! | 13:09 |
| noonedeadpunk | like 1 ou of 6 jobs get stuck | 13:09 |
| *** jawad_ax_ has quit IRC | 13:10 | |
| *** jawad_axd has joined #openstack-ansible | 13:11 | |
| *** jawad_axd has quit IRC | 13:12 | |
| *** jawad_axd has joined #openstack-ansible | 13:12 | |
| *** jawad_axd has quit IRC | 13:13 | |
| *** ianychoi_ has joined #openstack-ansible | 13:13 | |
| *** jawad_axd has joined #openstack-ansible | 13:13 | |
| *** ianychoi has quit IRC | 13:16 | |
| dmsimard | can reproduce the issue with offline client and >=2 threads but not one thread | 13:16 |
| dmsimard | http client can use >= 4 threads without any problems :( | 13:17 |
| noonedeadpunk | hm, pretty weird... | 13:18 |
| *** jeh has quit IRC | 13:28 | |
| *** akahat has quit IRC | 13:29 | |
| *** nurdie has joined #openstack-ansible | 13:29 | |
| *** akahat has joined #openstack-ansible | 13:31 | |
| *** nurdie has quit IRC | 13:38 | |
| *** gshippey has quit IRC | 13:49 | |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible stable/ussuri: Fix manila tempest jobs https://review.opendev.org/759426 | 13:53 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_manila stable/ussuri: Add centos-8 support https://review.opendev.org/759396 | 13:54 |
| noonedeadpunk | jrosser: so, in my vm old centos image took 13m for lxc-containers-create.yml and new 23m o_O | 13:56 |
| noonedeadpunk | have no idea what this can be... | 13:56 |
| noonedeadpunk | some selinux or what... I'd say ssh config if we were using ssh for connection... | 13:58 |
| noonedeadpunk | will try to create 2 empty containers and compare them.... | 13:59 |
| *** d34dh0r53 has joined #openstack-ansible | 14:00 | |
| *** jawad_axd has quit IRC | 14:00 | |
| *** rpittau is now known as rpittau|afk | 14:06 | |
| *** cshen_ has quit IRC | 14:16 | |
| *** jawad_axd has joined #openstack-ansible | 14:20 | |
| noonedeadpunk | well, new image is +300mb in size... | 14:38 |
| noonedeadpunk | no difference in case of regular facts gathering task... | 14:42 |
| *** nurdie has joined #openstack-ansible | 14:46 | |
| dmsimard | noonedeadpunk: I have a patch up to fix the locking https://review.opendev.org/#/c/759439/ | 14:47 |
| dmsimard | will merge it and then I'd like to test OSA against master before releasing it | 14:47 |
| dmsimard | I couldn't reproduce locally but you never know | 14:48 |
| dmsimard | "it worked in devstack", "it works on my machine", etc. :p | 14:48 |
| jrosser | noonedeadpunk: thats wierd with the new image being slow | 14:48 |
| jrosser | wonder if somehow the same thing that always made centos-7 slow, we just never found what | 14:49 |
| noonedeadpunk | might be... | 14:49 |
| noonedeadpunk | was also thinking about that | 14:49 |
| *** yann-kaelig has joined #openstack-ansible | 14:52 | |
| *** spatel has joined #openstack-ansible | 14:58 | |
| jrosser | i figure we should fix the release note for this https://review.opendev.org/#/c/758029/2/templates/nova.conf.j2 | 14:59 |
| jrosser | or even revert the whole patch, not sure really as it was possible all along with 0.0 values :( | 14:59 |
| noonedeadpunk | well it's not merged yet | 15:02 |
| noonedeadpunk | put -W | 15:03 |
| noonedeadpunk | I thought about that we probably don't need it as well. but it actually doesn't make things worse | 15:03 |
| noonedeadpunk | that's what sean-k-mooney said, and referenced to https://github.com/openstack/nova/blob/master/nova/conf/compute.py#L420-L453 | 15:04 |
| *** nurdie has quit IRC | 15:05 | |
| *** nurdie has joined #openstack-ansible | 15:06 | |
| noonedeadpunk | well, what 's worse, that removing variable after first deployment won't have any effect... | 15:07 |
| *** nurdie has quit IRC | 15:10 | |
| *** macz_ has joined #openstack-ansible | 15:10 | |
| *** nurdie has joined #openstack-ansible | 15:12 | |
| noonedeadpunk | so probably we should just comment these variables out by default and check if they are defined... or even just leave that for overrides | 15:16 |
| noonedeadpunk | jrosser: or should I release -W? | 15:16 |
| jrosser | well the commit message and reno text is kind of wrong | 15:17 |
| noonedeadpunk | what worries me - `Once set to a non-default value, it is not possible to “unset” the config to get back to the default behavior` | 15:17 |
| jrosser | wheres that from? | 15:18 |
| noonedeadpunk | https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.cpu_allocation_ratio | 15:18 |
| jrosser | oh yes | 15:18 |
| * jrosser reads again | 15:18 | |
| noonedeadpunk | same for ram_allocation_ratio and disk_allocation_ratio¶ | 15:18 |
| noonedeadpunk | it's in note | 15:18 |
| noonedeadpunk | well, we can discuss that on monday then | 15:19 |
| jrosser | urg "the default behaviour" like what does that actually mean | 15:20 |
| noonedeadpunk | I think it populates inventory and then overrides set with api may not work properly? | 15:21 |
| noonedeadpunk | as like initial_cpu_allocation_ratio read only during node discovery and population and futher change of the value will have no effect | 15:21 |
| noonedeadpunk | Sean was trying ot explain this today :) | 15:22 |
| jrosser | i saw | 15:22 |
| jrosser | kind of complicated logic | 15:23 |
| jrosser | complicated / unexpected | 15:23 |
| noonedeadpunk | it is... but now numa pinned vms can live migrate, so probably was worth it | 15:23 |
| jrosser | so i guess the question is if there is a need to change the current behaviour or defaults | 15:28 |
| jrosser | setting defaults of 0.0 and overrides will make the scheme via placement work, but only if you do that first time | 15:29 |
| *** gyee has joined #openstack-ansible | 15:48 | |
| openstackgerrit | David Moreau Simard proposed openstack/openstack-ansible master: DNM: Test OSA with ara on master https://review.opendev.org/759464 | 15:53 |
| dmsimard | ^ let's see if anything in there gets stuck | 15:56 |
| noonedeadpunk | jrosser: yeh | 15:56 |
| noonedeadpunk | and you should now what you're doing before dpeloying osa | 15:56 |
| noonedeadpunk | which is almost never the case:) | 15:57 |
| jrosser | perhaps really we could drop the patch and do a nice explanation in os_nova role docs | 15:57 |
| noonedeadpunk | returning to centos - this really simpla task took 2 mins - whaaat https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d55/759310/7/check/openstack-ansible-deploy-infra_lxc-centos-8/d552d15/logs/ara-report/results/1613.html | 16:01 |
| noonedeadpunk | just trying to somehow reproduce issue and try to find the way to reproduce | 16:02 |
| noonedeadpunk | but nothing close in sandbox | 16:05 |
| noonedeadpunk | wow. new image has somehow jumbo-frame o_O | 16:06 |
| noonedeadpunk | http://paste.openstack.org/show/799331/ | 16:06 |
| noonedeadpunk | feels like $reason | 16:06 |
| jrosser | thats interesting | 16:07 |
| noonedeadpunk | and from old one on the same host has 1500 | 16:08 |
| jrosser | i have seen other jobs in one of our repos failing functional tests with mtu | 16:08 |
| jrosser | sorry cant remember which | 16:08 |
| jrosser | but iirc that was also centos | 16:08 |
| noonedeadpunk | I'm wondering why it decided to set it to 9000... Maybe NetworkManager..... | 16:10 |
| *** MickyMan77 has joined #openstack-ansible | 16:11 | |
| jrosser | i wonder if there is conflict between what networkd and nm are doing | 16:11 |
| *** _mmethot_ has quit IRC | 16:12 | |
| jrosser | after all we do insert networkd into the image ourselves | 16:12 |
| noonedeadpunk | well, container restart did reset it to 1500 | 16:19 |
| noonedeadpunk | another restart and.... http://paste.openstack.org/show/799333/ lol | 16:19 |
| *** MickyMan77 has quit IRC | 16:20 | |
| guilhermesp | noonedeadpunk: i think that deserves a backport https://review.opendev.org/#/c/751767/ | 16:23 |
| guilhermesp | at least till train | 16:23 |
| noonedeadpunk | well, have nothing against it | 16:24 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_magnum stable/ussuri: Add deployment of keystone_auth_default_policy https://review.opendev.org/759471 | 16:25 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_magnum stable/train: Add deployment of keystone_auth_default_policy https://review.opendev.org/759472 | 16:25 |
| guilhermesp | appreciated noonedeadpunk | 16:26 |
| noonedeadpunk | sure, np | 16:30 |
| noonedeadpunk | well, it's not nm, as once I dropped it, eth0 stopped starting until systemd-networkd manual restart | 16:31 |
| noonedeadpunk | so something is fighting, but seems to be not nm | 16:32 |
| *** MickyMan77 has joined #openstack-ansible | 16:44 | |
| *** recyclehero has quit IRC | 16:50 | |
| *** gundalow_ has joined #openstack-ansible | 16:50 | |
| *** persia_ has joined #openstack-ansible | 16:51 | |
| *** jawad_axd has quit IRC | 16:52 | |
| *** yann-kaelig has quit IRC | 16:56 | |
| *** persia has quit IRC | 16:57 | |
| *** gundalow has quit IRC | 16:57 | |
| *** gundalow_ is now known as gundalow | 16:57 | |
| noonedeadpunk | oh, well, other container also can randomly get 9000 mtu | 17:05 |
| noonedeadpunk | despite what is set in lxc interface config | 17:06 |
| *** andrewbonney has quit IRC | 17:06 | |
| noonedeadpunk | oh, well, lxcbr0 has also 9000 mtu | 17:07 |
| *** recyclehero has joined #openstack-ansible | 17:10 | |
| noonedeadpunk | I can recall some race condition when we should wait networkd for some reason.... | 17:16 |
| noonedeadpunk | ah it was https://review.opendev.org/#/c/734089/ | 17:17 |
| *** nurdie has quit IRC | 17:19 | |
| *** MickyMan77 has quit IRC | 17:19 | |
| *** mmercer has joined #openstack-ansible | 17:21 | |
| noonedeadpunk | ok, so it's us who set mtu to 9000 https://opendev.org/openstack/openstack-ansible/src/branch/master/tests/roles/bootstrap-host/tasks/prepare_networking.yml#L101 | 17:21 |
| noonedeadpunk | except not sure about how lxcbr0 gets 9000 | 17:22 |
| nsmeds | There are more specific channels for this - however I've had better luck with y'all :P has anyone ever seen `resolvconf-pull-resolved.service: Failed with result 'start-limit-hit'.` repeated in their logs anytime they create/delete a network? | 17:23 |
| nsmeds | Running Ubuntu 18.04 with Train, deployed with openstack-ansible. | 17:23 |
| nsmeds | There doesn't appear to be any issues with the networks - but I'm trying to figure out _why_ this is being restarted so many times it hits a rate limit. | 17:23 |
| noonedeadpunk | never noticed | 17:23 |
| noonedeadpunk | wondering why on ubuntu aio I don;t see mtu 9000 | 17:29 |
| *** cshen has joined #openstack-ansible | 17:29 | |
| *** renich has joined #openstack-ansible | 17:31 | |
| *** cshen has quit IRC | 17:34 | |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible master: Set container MTU equal to dummy interfaces https://review.opendev.org/759498 | 17:43 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-lxc_hosts master: Use lxc_image_cache_server_mirrors as image source https://review.opendev.org/759310 | 17:44 |
| jrosser | noonedeadpunk: i am thinking that lxcbr0 ever being mtu 9000 is wrong | 17:50 |
| jrosser | eth0->lxcbr0 is the default route and so should be mtu 1500 always | 17:50 |
| noonedeadpunk | well, it's src-nated so it probably can be 9000? | 17:52 |
| noonedeadpunk | (not sure) | 17:52 |
| jrosser | nat will just rewrite some if the ip headers | 17:53 |
| jrosser | the packet needs to be small enough to leave the host and maybe out to the internet | 17:53 |
| *** cshen has joined #openstack-ansible | 17:54 | |
| noonedeadpunk | I kind of out of ideas why in the world this centos 8 image executes simple tasks like template for 2 mins. and it feels like networking thing I guess... except there's no networking as connection plugin just attaches with lxc command? | 17:54 |
| noonedeadpunk | and removal of NM makes things even worse for some reason | 17:54 |
| noonedeadpunk | so interface doesn't get ip addr | 17:55 |
| jrosser | does it reproduce locally? | 17:55 |
| jrosser | 2 mins for template, i mean | 17:55 |
| noonedeadpunk | no, if it's single task | 17:55 |
| noonedeadpunk | but like lxc-containers-create took 23m instead of 13m for old image | 17:56 |
| noonedeadpunk | and the same for setup-infrastructure | 17:56 |
| noonedeadpunk | so it reproduced locally, but now idea what triggers it... | 17:56 |
| noonedeadpunk | maybe should try to remove ara | 17:56 |
| noonedeadpunk | just to be super sure | 17:57 |
| *** nurdie has joined #openstack-ansible | 18:06 | |
| *** cshen has quit IRC | 18:10 | |
| *** cshen has joined #openstack-ansible | 18:11 | |
| dmsimard | fwiw my test with the fix in master hasn't locked down https://review.opendev.org/#/c/759464/ | 18:17 |
| dmsimard | i'll do a recheck just to be sure | 18:17 |
| *** elduderino80 has joined #openstack-ansible | 18:29 | |
| *** cshen has quit IRC | 18:40 | |
| *** nurdie has quit IRC | 18:49 | |
| *** nurdie has joined #openstack-ansible | 18:55 | |
| *** MickyMan77 has joined #openstack-ansible | 18:56 | |
| *** cshen has joined #openstack-ansible | 19:22 | |
| *** nurdie has quit IRC | 19:27 | |
| *** MickyMan77 has quit IRC | 19:31 | |
| *** nurdie has joined #openstack-ansible | 19:32 | |
| *** cloudnull has quit IRC | 19:47 | |
| *** cloudnull has joined #openstack-ansible | 19:47 | |
| *** klamath_atx has quit IRC | 19:48 | |
| *** cloudnull has quit IRC | 19:49 | |
| *** klamath_atx has joined #openstack-ansible | 19:56 | |
| *** cloudnull has joined #openstack-ansible | 19:58 | |
| *** cloudnull has quit IRC | 20:04 | |
| *** cloudnull has joined #openstack-ansible | 20:05 | |
| *** klamath_atx has quit IRC | 20:05 | |
| *** MickyMan77 has joined #openstack-ansible | 20:07 | |
| *** klamath_atx has joined #openstack-ansible | 20:15 | |
| *** klamath_atx has quit IRC | 20:22 | |
| *** MickyMan77 has quit IRC | 20:27 | |
| *** klamath_atx has joined #openstack-ansible | 20:33 | |
| *** sc has quit IRC | 20:36 | |
| *** sc has joined #openstack-ansible | 20:36 | |
| *** klamath_atx has quit IRC | 20:42 | |
| *** vesper has joined #openstack-ansible | 20:44 | |
| *** mgariepy has quit IRC | 20:50 | |
| *** prometheanfire has quit IRC | 20:50 | |
| *** vesper11 has quit IRC | 20:50 | |
| *** prometheanfire has joined #openstack-ansible | 20:50 | |
| *** nurdie has quit IRC | 20:52 | |
| *** rf0lc0 has joined #openstack-ansible | 20:52 | |
| *** nurdie has joined #openstack-ansible | 20:52 | |
| *** rfolco has quit IRC | 20:53 | |
| *** nurdie has quit IRC | 20:53 | |
| *** rf0lc0 has quit IRC | 20:56 | |
| *** mgariepy has joined #openstack-ansible | 21:00 | |
| *** cshen has quit IRC | 21:13 | |
| *** klamath_atx has joined #openstack-ansible | 21:24 | |
| *** cshen has joined #openstack-ansible | 21:26 | |
| *** cshen has quit IRC | 21:30 | |
| *** klamath_atx has quit IRC | 21:37 | |
| *** macz_ has quit IRC | 21:40 | |
| *** spatel has quit IRC | 21:40 | |
| *** elduderino80 has quit IRC | 22:12 | |
| dmsimard | second round of jobs came back without locking either and I also can no longer reproduce locally so I'll tag a release with the fix in | 22:16 |
| *** klamath_atx has joined #openstack-ansible | 22:29 | |
| *** NewJorg has quit IRC | 22:31 | |
| *** NewJorg has joined #openstack-ansible | 22:33 | |
| dmsimard | ara 1.5.3 is tagged and it'll be on pypi/mirrors soon enough, I've created this issue for the random lock up: https://github.com/ansible-community/ara/issues/183 | 22:49 |
| dmsimard | thanks for pointing it out <3 | 22:49 |
| *** macz_ has joined #openstack-ansible | 23:03 | |
| *** macz_ has quit IRC | 23:07 | |
| *** cshen has joined #openstack-ansible | 23:09 | |
| *** cshen has quit IRC | 23:14 | |
| *** klamath_atx has quit IRC | 23:16 | |
| *** MickyMan77 has joined #openstack-ansible | 23:27 | |
| *** cshen has joined #openstack-ansible | 23:35 | |
| *** owalsh_ has joined #openstack-ansible | 23:40 | |
| *** cshen has quit IRC | 23:40 | |
| *** owalsh has quit IRC | 23:43 | |
| *** owalsh has joined #openstack-ansible | 23:45 | |
| *** tosky has quit IRC | 23:46 | |
| *** owalsh_ has quit IRC | 23:48 | |
| *** klamath_atx has joined #openstack-ansible | 23:51 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!