| *** diurnalist has quit IRC | 00:21 | |
| *** cah_link has quit IRC | 00:44 | |
| *** diurnalist has joined #openstack-kolla | 00:46 | |
| *** diurnalist has quit IRC | 01:11 | |
| *** diurnalist has joined #openstack-kolla | 01:14 | |
| *** zhanglong has joined #openstack-kolla | 01:17 | |
| *** diurnalist has quit IRC | 01:36 | |
| *** zhanglong has quit IRC | 02:26 | |
| *** zhanglong has joined #openstack-kolla | 02:28 | |
| *** ktibi has joined #openstack-kolla | 03:05 | |
| *** ktibi has quit IRC | 03:09 | |
| *** skramaja has joined #openstack-kolla | 03:50 | |
| *** JangwonLee_ has joined #openstack-kolla | 04:52 | |
| *** JangwonLee has quit IRC | 04:54 | |
| *** evrardjp has quit IRC | 05:34 | |
| *** evrardjp has joined #openstack-kolla | 05:35 | |
| *** sri_ has joined #openstack-kolla | 05:49 | |
| *** zhanglong has quit IRC | 05:55 | |
| *** zhanglong has joined #openstack-kolla | 05:57 | |
| openstackgerrit | Dincer Celik proposed openstack/kolla-ansible stable/train: Fixes gnocchi-api script name for Ubuntu/Debian https://review.opendev.org/710186 | 06:00 |
|---|---|---|
| *** abdysn has joined #openstack-kolla | 06:03 | |
| *** kozhukalov has joined #openstack-kolla | 06:07 | |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: OVN Support https://review.opendev.org/696841 | 06:42 |
| *** dannins has joined #openstack-kolla | 07:10 | |
| *** jbadiapa has joined #openstack-kolla | 07:19 | |
| *** dougsz has joined #openstack-kolla | 07:38 | |
| *** dougsz has quit IRC | 07:44 | |
| yoctozepto | morning | 07:59 |
| yoctozepto | what a silent day here | 07:59 |
| openstackgerrit | Piotr Rabiega proposed openstack/kolla master: Add collectd-pcie-errors package for PCIe Errors (C7 only) https://review.opendev.org/710029 | 08:05 |
| mnasiadka | well | 08:07 |
| *** bengates has joined #openstack-kolla | 08:07 | |
| *** bengates has quit IRC | 08:07 | |
| *** bengates has joined #openstack-kolla | 08:08 | |
| kevinz | hi there | 08:16 |
| yoctozepto | hi there as well | 08:17 |
| mnasiadka | yoctozepto: willing to +2+w this https://review.opendev.org/#/c/709204/4 - so we can go forward with most of the C8 backports on Train? | 08:45 |
| patchbot | patch 709204 - kolla (stable/train) - CentOS 8: Use upstream Ceph/master - 4 patch sets | 08:45 |
| yoctozepto | osmanlicilegi: could you elaborate on that /run/netns issue? when does it fail? seemingly not from start | 08:46 |
| yoctozepto | mnasiadka: ok | 08:46 |
| *** stingrayza has quit IRC | 08:53 | |
| mnasiadka | yoctozepto: there's a bug raised I think by kevinz (if kevinz == kevin-zhao) | 08:56 |
| yoctozepto | mnasiadka: yeah, but I'm not following :-) | 08:56 |
| kevinz | yes it is mine | 08:56 |
| kevinz | I met the issue for some times. but not 100% reproduced..it is werid | 08:57 |
| kevinz | weird | 08:57 |
| mnasiadka | yoctozepto: I have a weird idea to add/remove multiple subnets in the CI (and multiple routers) and see if that happens | 09:04 |
| yoctozepto | nah, I see the issue now | 09:08 |
| yoctozepto | it needs same testing as the other bug with neutron | 09:08 |
| yoctozepto | lemme find it | 09:08 |
| mnasiadka | meaning? | 09:08 |
| *** rpittau|afk is now known as rpittau | 09:09 | |
| mnasiadka | anyway, if it can't open netns - mounting /run/netns should be the answer | 09:09 |
| yoctozepto | https://bugs.launchpad.net/kolla-ansible/+bug/1863982 | 09:09 |
| openstack | Launchpad bug 1863982 in kolla-ansible "Upgrade from Rocky to Stein, router namespace disappear" [Undecided,Confirmed] - Assigned to Radosław Piliszek (yoctozepto) | 09:09 |
| mnasiadka | and if osmanlicilegi tested and it works for him (and no bindmount for /run/netns throws errors) | 09:09 |
| yoctozepto | mnasiadka: yeah, it does something with the problem | 09:09 |
| yoctozepto | as for testing | 09:09 |
| yoctozepto | it is most likely affected by restarts as it is when container data is lost | 09:10 |
| mnasiadka | exit code 4 | 09:10 |
| mnasiadka | lovely | 09:10 |
| yoctozepto | still testing a bit | 09:10 |
| mnasiadka | if neutron would print the command in the debug, it would be even more lovely | 09:10 |
| yoctozepto | where 4? | 09:10 |
| *** ktibi has joined #openstack-kolla | 09:15 | |
| mnasiadka | in the bug you pasted | 09:15 |
| yoctozepto | ah, was not aware | 09:16 |
| yoctozepto | though it looks like device exists but not its namespace | 09:17 |
| yoctozepto | so they are kinda related too | 09:17 |
| mnasiadka | wonder if kevinz hit the problem, that in stein we did not mount /run at this point (because the backport already merged) | 09:18 |
| yoctozepto | mnasiadka: me too | 09:18 |
| kevinz | Launchpad bug 1863982 in kolla-ansible "Upgrade from Rocky to Stein, router namespace disappear" [Undecided,Confirmed] - Assigned to Radosław Piliszek (yoctozepto) | 09:19 |
| openstack | Launchpad bug 1863982 in kolla-ansible "Upgrade from Rocky to Stein, router namespace disappear" [Undecided,Confirmed] https://launchpad.net/bugs/1863982 - Assigned to Radosław Piliszek (yoctozepto) | 09:19 |
| * osmanlicilegi is back | 09:19 | |
| yoctozepto | nah | 09:19 |
| yoctozepto | it was day before that | 09:19 |
| mnasiadka | so /run/netns might not help at all | 09:19 |
| kevinz | this one, I can get netns in l3_agent, and dhcp-agent eventually | 09:19 |
| mnasiadka | :D | 09:19 |
| yoctozepto | yeah, indeed | 09:19 |
| kevinz | But can't get the netns at hosts | 09:19 |
| mnasiadka | well, it may help a bit | 09:19 |
| yoctozepto | I was about testing, both seemingly are triggered by service restart | 09:20 |
| mnasiadka | well, adding /run/netns back will not hurt, and at least helps osmanlicilegi's case | 09:20 |
| yoctozepto | in CI we only set up router after it's all stable | 09:20 |
| mnasiadka | (so probably hurts everybody's case right now) | 09:20 |
| yoctozepto | agreed | 09:20 |
| kevinz | and restart l3_agent will easy help | 09:20 |
| osmanlicilegi | let me check logs if there's any error since yesterday | 09:21 |
| yoctozepto | still doing some fake testing localy, will report back and either +2 or -1 | 09:21 |
| kevinz | I've a question, after deploy a train cluster | 09:21 |
| kevinz | And the ns will just appear in * agent container? | 09:21 |
| *** k_mouza has joined #openstack-kolla | 09:21 | |
| yoctozepto | mnasiadka: we did not release the buggy version, did we? | 09:22 |
| mnasiadka | yoctozepto: no, no release since then | 09:22 |
| yoctozepto | mnasiadka: if we did not, then remove the reno | 09:22 |
| yoctozepto | mnasiadka: but adapt previous one | 09:22 |
| *** tonythomas has joined #openstack-kolla | 09:25 | |
| yoctozepto | osmanlicilegi: you did not recreate the container, only restarted it, right? | 09:26 |
| mnasiadka | yoctozepto: yeah, will do | 09:27 |
| mnasiadka | yoctozepto, mgoddard: should we accept Kolla C7 changes at this point (like this one - https://review.opendev.org/#/c/710029/)? For me it totally makes no sense, because sooner or later we will drop C7 from master. | 09:28 |
| patchbot | patch 710029 - kolla - Add collectd-pcie-errors package for PCIe Errors (... - 2 patch sets | 09:28 |
| yoctozepto | mnasiadka: only for bp | 09:29 |
| mnasiadka | yoctozepto: well, it's adding a collectd plugin package to c7, but there's no c8 part - so what's the point? :D | 09:29 |
| yoctozepto | mnasiadka: ask if they want it in bp, otherwise none | 09:30 |
| mnasiadka | makes sense | 09:30 |
| mnasiadka | yoctozepto: around CI - I see aarch one is totally network-wise unreliable... I stopped looking why it fails ;) | 09:31 |
| openstackgerrit | Merged openstack/kayobe master: CI: set previous_release to train https://review.opendev.org/709145 | 09:32 |
| mnasiadka | "stdout": "Error during database migration: Migration cannot continue until all volumes have been migrated to the `__DEFAULT__` volume type. Please run `cinder-manage db online_data_migrations`. There are still untyped volumes unmigrated.\n", | 09:33 |
| yoctozepto | mnasiadka: I never did :-) | 09:33 |
| mnasiadka | that is interesting in the upgrade jobs | 09:33 |
| yoctozepto | oh my | 09:33 |
| yoctozepto | is it breaking? random? | 09:33 |
| mnasiadka | yeah, random :) | 09:34 |
| mnasiadka | one job failed out of 4 upgrade jobs | 09:34 |
| yoctozepto | wonder where is the race there | 09:35 |
| yoctozepto | which part fails? | 09:35 |
| mnasiadka | running bootstrap | 09:37 |
| mnasiadka | https://e7aa0df1ac641e2ee8fd-b023e6398111e1b55f35dc3d44640709.ssl.cf1.rackcdn.com/710186/1/check/kolla-ansible-ubuntu-source-upgrade-ceph/6eb6b69/primary/logs/ansible/upgrade | 09:38 |
| *** gfidente|afk is now known as gfidente | 09:39 | |
| mnasiadka | I don't think we're doing online data migration at any time | 09:39 |
| mnasiadka | lol, we are | 09:40 |
| mnasiadka | but after db sync | 09:40 |
| mnasiadka | https://github.com/openstack/kolla/blob/master/docker/cinder/cinder-api/extend_start.sh | 09:40 |
| yoctozepto | well, one can't do online migrations if schema is not ready? | 09:42 |
| yoctozepto | weird | 09:43 |
| yoctozepto | add that to CI errors and we'll see to it later | 09:43 |
| yoctozepto | (much later) | 09:43 |
| mnasiadka | yoctozepto: and can't do db sync if volumes are untyped :D | 09:45 |
| yoctozepto | fwiw, https://bugs.launchpad.net/neutron/+bug/1812364 xD | 09:46 |
| openstack | Launchpad bug 1812364 in neutron "Error "OSError: [Errno 22] failed to open netns" in l3-agent logs" [High,Confirmed] - Assigned to Miguel Lavalle (minsel) | 09:46 |
| yoctozepto | looks like ubuntu being bad at netns in general | 09:47 |
| *** cah_link has joined #openstack-kolla | 09:48 | |
| *** Nirtal has joined #openstack-kolla | 09:49 | |
| mnasiadka | awesome | 09:50 |
| yoctozepto | I did some testing on centos, now proceeding on ubuntu | 09:51 |
| mnasiadka | yoctozepto: I assume centos works? | 09:57 |
| mnasiadka | yoctozepto: hmm, maybe Ubuntu and AppArmor does something nasty from time to time? | 09:57 |
| yoctozepto | mnasiadka: I am doing dry testing with ip netns to see how I can make it break | 09:58 |
| yoctozepto | mnasiadka: in different scenarios | 09:59 |
| *** dmellado has quit IRC | 09:59 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/train: CentOS 8: Add deploy jobs in CI https://review.opendev.org/709536 | 10:10 |
| osmanlicilegi | mnasiadka, yoctozepto: I rolled back and tried from scratch, netns errors appeared imemdiately. in 5 minutes, all namespaces disappeared and I lost connectivity to vms. adding /run/netns fixes problem immediately. | 10:23 |
| mnasiadka | in 5 minutes might be the key | 10:24 |
| mnasiadka | :0 | 10:24 |
| mnasiadka | :) | 10:24 |
| yoctozepto | :O | 10:24 |
| mnasiadka | we don't leave it stranding for 5 minutes | 10:24 |
| mnasiadka | (in CI) | 10:24 |
| osmanlicilegi | right | 10:25 |
| yoctozepto | yeah, we don't | 10:26 |
| yoctozepto | osmanlicilegi: but is it that the containers are started without mount | 10:26 |
| yoctozepto | osmanlicilegi: routers are properly created | 10:27 |
| osmanlicilegi | there's one point I couldn't understand. the containers cannot manage netns after restarted so how previously creates namespaces disappear? | 10:27 |
| yoctozepto | osmanlicilegi: and after 5 minuters poof? | 10:27 |
| yoctozepto | well, as for my dry testing | 10:29 |
| yoctozepto | centos and ubuntu behave much the same, error messages have been modified slightly | 10:29 |
| yoctozepto | scenario 1: w/o mount | 10:29 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: Custom haproxy script for monitoring galera https://review.opendev.org/710213 | 10:29 |
| yoctozepto | if you create a netns in container | 10:30 |
| yoctozepto | and then restart it | 10:30 |
| yoctozepto | then netns is lost but you have its reference in filesystem | 10:30 |
| yoctozepto | which causes RTNETLINK errors about invalid reference | 10:31 |
| yoctozepto | because seemignly netns appears in /run/netns by means of touch and mount -t proc | 10:31 |
| yoctozepto | and those magic proc mounts will be lost | 10:31 |
| yoctozepto | still, /run/netns is not going anywhere | 10:32 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/train: DNM: Testing cloudkitty in train https://review.opendev.org/710215 | 10:32 |
| yoctozepto | if you recreate the container, then it's obviously starting from clean plate then, so only by means of neutron-server may it know previous namespace names | 10:32 |
| yoctozepto | scenario 2: w/ mount | 10:32 |
| yoctozepto | it must be shared, that's true | 10:33 |
| yoctozepto | otherwise it ends up like in scenario 1 but also affects recreations | 10:33 |
| yoctozepto | as for what breaks ubuntu in 5 minutes - no idea | 10:34 |
| osmanlicilegi | hmm you're right imho | 10:36 |
| cosmicsound | good day | 10:36 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla master: WIP: Add mysql client to haproxy image for Galera state checking https://review.opendev.org/710217 | 10:38 |
| mgoddard | morning all | 10:38 |
| yoctozepto | that said, lemme break my semiproduction wrt those mounts | 10:39 |
| *** cah_link has quit IRC | 10:43 | |
| *** stingrayza has joined #openstack-kolla | 10:43 | |
| openstackgerrit | Michal Nasiadka proposed openstack/kayobe master: Add framework to deploy user-defined containers on seed https://review.opendev.org/709689 | 10:45 |
| cosmicsound | is this guide valid? https://docs.openstack.org/kolla-ansible/train/reference/containers/kuryr-guide.html | 10:45 |
| cosmicsound | asking because there is a kolla.conf with this: | 10:46 |
| cosmicsound | [Service] | 10:46 |
| cosmicsound | ExecStart= | 10:46 |
| cosmicsound | # ExecStart commandline copied from 'docker-ce' package. Same on CentOS/Debian/Ubuntu systems. | 10:46 |
| cosmicsound | ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock -H tcp://172.22.0.1:2375 | 10:46 |
| *** riuzen has joined #openstack-kolla | 10:46 | |
| cosmicsound | and the docs mention this: | 10:46 |
| cosmicsound | ExecStart= -H tcp://172.16.1.13:2375 -H unix:///var/run/docker.sock --cluster-store=etcd://172.16.1.13:2379 --cluster-advertise=172.16.1.13:2375 | 10:46 |
| cosmicsound | no more need for dockerd? or fd:// --containerd tags? | 10:46 |
| mnasiadka | yoctozepto: probably after 5 minutes it's trying to find if netns' are created, and fails miserably for some reason | 10:47 |
| yoctozepto | mnasiadka: well, they are still after 5 minutes :D | 10:48 |
| yoctozepto | still there* | 10:48 |
| mnasiadka | yoctozepto: create routers, restart container, are they still there? :) | 10:48 |
| yoctozepto | mnasiadka: no, they aren't; I meant w/o restart | 10:49 |
| yoctozepto | osmanlicilegi: you do service restart *after* router creation? | 10:49 |
| mnasiadka | oh boy, 709203 failed on collecting logs and cancelled the whole pack of C8 changes | 10:50 |
| mnasiadka | mgoddard: ^^ | 10:50 |
| yoctozepto | meh | 10:50 |
| osmanlicilegi | yoctozepto: no svc restart | 10:50 |
| yoctozepto | mnasiadka: ^ | 10:51 |
| mnasiadka | yoctozepto: changing bind mounts when reconfigure must restart containers, come on | 10:51 |
| yoctozepto | mnasiadka: yeah, so we are talking existing routers? | 10:53 |
| yoctozepto | osmanlicilegi: ^ | 10:53 |
| osmanlicilegi | yoctozepto: what you exactly mean by servie restart? you mean docker? | 10:53 |
| osmanlicilegi | k-a only restarts containers you know | 10:53 |
| mgoddard | mnasiadka: :( | 10:53 |
| yoctozepto | osmanlicilegi: yeah, I meant that | 10:54 |
| yoctozepto | I was asking whether your scenario is that routers existed before | 10:54 |
| yoctozepto | reconfigure happened | 10:54 |
| osmanlicilegi | I haven't had any svc or server reboot. you are right that my existing namespaces disappeared after restarting containers. I think I'm gonna open a case to canonical as this shouldn't happen if I think correctly. | 10:55 |
| *** shyamb has joined #openstack-kolla | 10:56 | |
| osmanlicilegi | in any situation I need /run/netns back :] | 10:56 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla master: WIP: Add mysql client to haproxy image for Galera state checking https://review.opendev.org/710217 | 10:57 |
| mnasiadka | yoctozepto: but I think we do init-runonce before upgrade, and then run the upgrade - so this should show some problems? | 10:58 |
| hrw | mgoddard: https://paste.centos.org/view/afe5d480 fixes mistral/mistral-dashboard situation ;d | 11:00 |
| *** bengates has quit IRC | 11:04 | |
| *** riuzen has quit IRC | 11:05 | |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: version-check: handle more situations https://review.opendev.org/710229 | 11:06 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: do not use unversioned Python binary https://review.opendev.org/710231 | 11:10 |
| *** rpittau is now known as rpittau|bbl | 11:12 | |
| openstackgerrit | Merged openstack/kolla stable/stein: Bump stein versions https://review.opendev.org/710068 | 11:16 |
| *** kozhukalov has quit IRC | 11:17 | |
| *** bengates has joined #openstack-kolla | 11:20 | |
| *** kozhukalov has joined #openstack-kolla | 11:21 | |
| *** ktibi has quit IRC | 11:22 | |
| yoctozepto | I reproduced | 11:23 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: CentOS 8: Add overcloud CI job https://review.opendev.org/709787 | 11:23 |
| yoctozepto | osmanlicilegi, mnasiadka, kevinz: I reproduced osmanlicilegi case | 11:23 |
| yoctozepto | the scenario is you have some routers already | 11:23 |
| yoctozepto | reconfigure w/o the mounts | 11:23 |
| yoctozepto | everything is fine and dandy | 11:23 |
| yoctozepto | (even after 5 minutes, sorry to spoil it!) | 11:24 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: Make Kayobe code compatible with Python 3 https://review.opendev.org/635586 | 11:24 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: Bump Ansible version range to 2.8 - 2.9 https://review.opendev.org/708922 | 11:24 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: CentOS 8: Disable ntpd, enable chrony container https://review.opendev.org/709785 | 11:24 |
| yoctozepto | yet if you restart it NOW | 11:24 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: CentOS 8: Bump MichaelRigart.interfaces to 1.4.0 https://review.opendev.org/709786 | 11:24 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: CentOS 8: Add overcloud CI job https://review.opendev.org/709787 | 11:24 |
| yoctozepto | (.e.g l3 agent, these are nicert to break) | 11:24 |
| yoctozepto | (as dhcp has own quirks) | 11:24 |
| yoctozepto | it breaks in the same way as osmanlicilegi pasted there | 11:24 |
| *** bengates has quit IRC | 11:26 | |
| *** bengates has joined #openstack-kolla | 11:27 | |
| osmanlicilegi | yoctozepto: namespaces still exists? | 11:31 |
| yoctozepto | osmanlicilegi: they most likely still live on host, I commented on: https://bugs.launchpad.net/kolla-ansible/+bug/1864856 | 11:32 |
| openstack | Launchpad bug 1864856 in kolla-ansible ussuri "dhcp-agent failed to open netns" [High,In progress] - Assigned to Michal Nasiadka (mnasiadka) | 11:32 |
| *** kozhukalov has quit IRC | 11:35 | |
| yoctozepto | there is a workaround | 11:35 |
| yoctozepto | clearn /run/netns inside the container and restart it | 11:35 |
| *** kozhukalov has joined #openstack-kolla | 11:35 | |
| yoctozepto | mnasiadka, osmanlicilegi, kevinz: btw, metadata agent does not require netns access so is irrelevant there | 11:41 |
| *** kozhukalov has quit IRC | 11:43 | |
| yoctozepto | otoh, ovn metadata does | 11:46 |
| *** ivve has joined #openstack-kolla | 11:53 | |
| openstackgerrit | Alfredo Moralejo proposed openstack/kolla master: DNM Test new deps based on final CBS builds https://review.opendev.org/710240 | 12:03 |
| *** kplant has joined #openstack-kolla | 12:07 | |
| osmanlicilegi | yoctozepto, better than sherlock holmes. | 12:13 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla master: WIP: Add mysql client to haproxy image for Galera state checking https://review.opendev.org/710217 | 12:17 |
| mnasiadka | yoctozepto: ovn does, it spawns processes in netns | 12:17 |
| mnasiadka | yoctozepto: the normal metadata agent doesn't? so how does that work? :D | 12:17 |
| yoctozepto | mnasiadka: l3&dhcp orchestrate data | 12:17 |
| mnasiadka | ah ok | 12:18 |
| yoctozepto | mnasiadka: the purpose of metadata is to listen on socket | 12:18 |
| mnasiadka | no l3 & dhcp in OVN | 12:18 |
| yoctozepto | yeah | 12:18 |
| yoctozepto | so they had to move that part | 12:18 |
| yoctozepto | ovn is different design | 12:18 |
| yoctozepto | has* | 12:18 |
| mnasiadka | ok, makes sense - we could do /run/netns in metadata based on neutron_plugin_agent in the OVN change - once /run/netns merges | 12:20 |
| mnasiadka | so we don't need /run/netns in metadata now | 12:20 |
| mnasiadka | let me update change | 12:20 |
| *** bengates has quit IRC | 12:20 | |
| yoctozepto | mnasiadka: yeah, also one more thing, now testing whether it could be separate docker volume | 12:21 |
| mnasiadka | it could, but if we share with the host - you can do ip netns exec from the host | 12:22 |
| mnasiadka | which is helpful | 12:22 |
| yoctozepto | more isolation vs less | 12:22 |
| * osmanlicilegi brb | 12:23 | |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: Add /run/netns bindmount to Neutron containers https://review.opendev.org/710051 | 12:25 |
| mnasiadka | mgoddard: any idea which path should we take? more isolation vs less? :D | 12:26 |
| yoctozepto | mnasiadka: reno to fix | 12:27 |
| yoctozepto | mnasiadka: rather add | 12:27 |
| mnasiadka | well, you wanted me to adapt the old one? | 12:27 |
| yoctozepto | mnasiadka: exactly | 12:28 |
| yoctozepto | mnasiadka: ah, sorry, gerrit playing tricks on me | 12:28 |
| openstackgerrit | Chason Chan proposed openstack/kolla-ansible master: [Docs] Pin kolla-anisble to the same version of quickstart guide https://review.opendev.org/707896 | 12:28 |
| mnasiadka | yoctozepto: How many bugs should I reference in the reno? both, old one or new one? :D | 12:29 |
| yoctozepto | mnasiadka: it's fine now | 12:29 |
| mnasiadka | me no touch | 12:29 |
| yoctozepto | mnasiadka: just see if it renders nicely | 12:29 |
| mnasiadka | ah, we wanted to run doc8 against reno | 12:29 |
| mgoddard | the question is whether to put /run/netns in a docker volume vs bind mounted to /run/netns? | 12:29 |
| mnasiadka | mgoddard: yup | 12:30 |
| mnasiadka | mgoddard: sometimes it's nice to be able to do ip netns exec from the host level | 12:30 |
| mgoddard | I would say bind mount is less weird | 12:30 |
| mgoddard | +1 | 12:30 |
| mgoddard | and people expect to be able to do it | 12:30 |
| mgoddard | also it changes one less thing in a fairly tricky level | 12:31 |
| mgoddard | could be knock-on effects of not being shared | 12:31 |
| *** ktibi has joined #openstack-kolla | 12:32 | |
| *** cz3 is now known as papiez | 12:33 | |
| *** papiez is now known as cz3 | 12:33 | |
| *** shyamb has quit IRC | 12:33 | |
| *** rlljorge has joined #openstack-kolla | 12:34 | |
| yoctozepto | moreover, you can't use volume mount | 12:35 |
| yoctozepto | because they don't support submounts :-) | 12:35 |
| yoctozepto | problem solved, go with direct bind | 12:36 |
| yoctozepto | also, /run/netns needs to be on tmpfs because it must be cleared with kernel state | 12:36 |
| yoctozepto | otherwise reboots would break | 12:36 |
| yoctozepto | so the default place is best | 12:36 |
| yoctozepto | eot | 12:36 |
| yoctozepto | ;D | 12:36 |
| *** kozhukalov has joined #openstack-kolla | 12:39 | |
| *** shyamb has joined #openstack-kolla | 12:40 | |
| *** cah_link has joined #openstack-kolla | 12:47 | |
| mnasiadka | yoctozepto: well, we are generating reno, but I would expect build-releasenotes job would also lint them a bit better | 12:49 |
| yoctozepto | mnasiadka: I got answer on ml - no way | 12:51 |
| yoctozepto | mnasiadka: and no plans (no sorry variant) | 12:52 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Add doc8 linting of release notes https://review.opendev.org/710244 | 12:55 |
| mnasiadka | let's see - better than nothing | 12:55 |
| *** shyamb has quit IRC | 12:56 | |
| *** negronjl has quit IRC | 12:56 | |
| openstackgerrit | Michal Nasiadka proposed openstack/kayobe master: Add framework to deploy user-defined containers on seed https://review.opendev.org/709689 | 12:57 |
| *** negronjl has joined #openstack-kolla | 12:59 | |
| *** sean-k-mooney has joined #openstack-kolla | 13:00 | |
| *** ktibi has quit IRC | 13:00 | |
| mnasiadka | what's happening with Zuul today... everything is so slooooow... | 13:01 |
| *** ktibi has joined #openstack-kolla | 13:03 | |
| *** dmellado has joined #openstack-kolla | 13:05 | |
| *** shyamb has joined #openstack-kolla | 13:05 | |
| yoctozepto | mnasiadka: check if tripleo is in the queues | 13:08 |
| mnasiadka | yoctozepto: tripleo is always in the queue | 13:09 |
| yoctozepto | mnasiadka: then you have your answer ;-) | 13:09 |
| yoctozepto | mnasiadka: https://review.opendev.org/709829 and https://review.opendev.org/709830 | 13:12 |
| patchbot | patch 709829 - kolla-ansible - Fix client TLS in neutron-metadata-agent - 1 patch set | 13:12 |
| patchbot | patch 709830 - kolla-ansible - Fix neutron-metadata-agent to use provided CA for ... - 1 patch set | 13:12 |
| yoctozepto | mnasiadka: tyvm | 13:13 |
| mnasiadka | yw | 13:13 |
| *** shyamb has quit IRC | 13:16 | |
| cosmicsound | yoctozepto , this | 13:17 |
| cosmicsound | TASK [haproxy : Checking if kolla_internal_vip_address and kolla_external_vip_address are not pingable from any node] *********************************************** | 13:17 |
| cosmicsound | ok: [compute-1] => (item={'address': '172.22.0.22', 'command': 'ping'}) | 13:17 |
| cosmicsound | ok: [compute-1] => (item={'address': '151.80.247.164', 'command': 'ping'}) | 13:17 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla master: WIP: Add mysql client to haproxy image for Galera state checking https://review.opendev.org/710217 | 13:17 |
| cosmicsound | lets see how this works | 13:18 |
| cosmicsound | fatal: [compute-1]: FAILED! => {"changed": false, "cmd": ["ip", "-o", "addr", "show", "dev", "eno1"], "delta": "0:00:00.001558", "end": "2020-02-27 13:17:31.084314", "failed_when_result": true, "rc": 0, "start": "2020-02-27 13:17:31.082756", "stderr": "", "stderr_lines": [], "stdout": "2: eno1 inet 151.80.247.160/29 brd 151.80.247.167 scope | 13:18 |
| cosmicsound | global eno1\\ valid_lft forever preferred_lft forever\n2: eno1 inet 51.91.153.141/24 brd 51.91.153.255 scope global dynamic eno1\\ valid_lft 85775sec preferred_lft 85775sec\n2: eno1 inet6 fe80::d250:99ff:fed6:c83a/64 scope link \\ valid_lft forever preferred_lft forever", "stdout_lines": ["2: eno1 inet 151.80.247.160/29 | 13:18 |
| cosmicsound | brd 151.80.247.167 scope global eno1\\ valid_lft forever preferred_lft forever", "2: eno1 inet 51.91.153.141/24 brd 51.91.153.255 scope global dynamic eno1\\ valid_lft 85775sec preferred_lft 85775sec", "2: eno1 inet6 fe80::d250:99ff:fed6:c83a/64 scope link \\ valid_lft forever preferred_lft forever"]} | 13:18 |
| yoctozepto | 'tis is madness | 13:19 |
| cosmicsound | hmm the vip must be pingable from all nodes | 13:19 |
| yoctozepto | this is* | 13:19 |
| cosmicsound | i think i know whats wrong | 13:19 |
| mnasiadka | yoctozepto: https://review.opendev.org/#/c/710213/1 - do we disable haproxy user somewhere? or what are you referring to? | 13:19 |
| patchbot | patch 710213 - kolla-ansible - WIP: Custom haproxy script for monitoring galera - 1 patch set | 13:19 |
| yoctozepto | mnasiadka: yeah, we do for the original check | 13:19 |
| yoctozepto | mnasiadka: in mariadb itself | 13:19 |
| yoctozepto | mnasiadka: it is mostly ineffective as elaborated elsewhere | 13:20 |
| yoctozepto | mnasiadka: but I also don't think yours is a huge improvement (sorry!) | 13:20 |
| yoctozepto | mnasiadka: we probably need to replace haproxy by itself | 13:20 |
| mnasiadka | yoctozepto: anything that will disable backend on real terms is better - mysqlchk in haproxy is lame | 13:21 |
| yoctozepto | mnasiadka: still, I think yours could be better than current | 13:21 |
| yoctozepto | mnasiadka: agreed | 13:21 |
| mnasiadka | yoctozepto: and adding proxysql is rather big work | 13:21 |
| yoctozepto | mnasiadka: also agreed | 13:21 |
| mnasiadka | yoctozepto: so if tripleo can survive with clusterchk script, we maybe can survive with that | 13:22 |
| *** sm806 has quit IRC | 13:22 | |
| yoctozepto | mnasiadka: they use pacemaker, they are making pace | 13:22 |
| yoctozepto | mnasiadka: we use keepalived, we want to keep it alive | 13:22 |
| yoctozepto | :D | 13:22 |
| mnasiadka | yoctozepto: but still they rely on clusterchk in xinetd and doing galera checks that way | 13:23 |
| yoctozepto | mnasiadka: yeah, xinetd is lovely | 13:23 |
| yoctozepto | mnasiadka: what does osa do? | 13:23 |
| mnasiadka | 4th attempt of multinode on OVN change... it's getting better and better | 13:23 |
| mnasiadka | yoctozepto: they do lxc, I don't know if I want to know :) | 13:23 |
| *** rpittau|bbl is now known as rpittau | 13:24 | |
| kplant | hey yankcrime - did you ever get the openstack cli working with openid here: https://www.stackhpc.com/tag/keycloak.html ? | 13:26 |
| yankcrime | kplant: no, never finished that bit off sadly - ping johnthetubaguy when he's around, he might have done some more with it since | 13:27 |
| kplant | appreciatei t | 13:28 |
| yoctozepto | mnasiadka: xD | 13:28 |
| yoctozepto | mnasiadka: but they also do plain host | 13:28 |
| mnasiadka | yoctozepto: and systemd-nspawn | 13:29 |
| yoctozepto | mnasiadka: really? | 13:29 |
| mnasiadka | yoctozepto: really! | 13:29 |
| yoctozepto | mnasiadka: I thought they were the last bastion of "deploy on metal" | 13:30 |
| mnasiadka | 6h 50m | 13:32 |
| mnasiadka | OVN change in check queue | 13:32 |
| mnasiadka | lol | 13:32 |
| mnasiadka | yoctozepto: seems we're linting reno: https://zuul.opendev.org/t/openstack/build/cebe1c0738f7451284cbf47dd1b18c77 | 13:33 |
| yoctozepto | mnasiadka: oh my, that's long! | 13:33 |
| yoctozepto | mnasiadka: it's contextless but still better than nothing :-) | 13:34 |
| mnasiadka | yoctozepto: at least we're checking for line length :D | 13:35 |
| *** sm806 has joined #openstack-kolla | 13:37 | |
| yoctozepto | mnasiadka: heck yeah | 13:39 |
| *** bengates has joined #openstack-kolla | 13:39 | |
| *** bengates has quit IRC | 13:40 | |
| *** bengates has joined #openstack-kolla | 13:40 | |
| yoctozepto | mnasiadka: fwiw, today's party with neutron made me read enough neutron code to discover a security bug :O | 13:40 |
| mnasiadka | yoctozepto: lol, raise it :) | 13:42 |
| mnasiadka | fixing those lines... ugh | 13:43 |
| yoctozepto | mnasiadka: writing it up now | 13:43 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Add doc8 linting of release notes https://review.opendev.org/710244 | 14:01 |
| mnasiadka | yoctozepto: fixed all too long lines ^^ | 14:02 |
| *** zhanglong has quit IRC | 14:05 | |
| yoctozepto | mnasiadka: Thanks for reporting a bug, the Neutron team loves you! | 14:10 |
| yoctozepto | mnasiadka: I hope they will still :P | 14:11 |
| cosmicsound | is this normal that they have no domain? https://mdb.uhlhost.net/uploads/58def43858473468/image.png | 14:14 |
| yoctozepto | I guess? | 14:17 |
| mnasiadka | cosmicsound: why not? | 14:20 |
| *** rlljorge has quit IRC | 14:23 | |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: WIP: CI: Add overcloud host configure job https://review.opendev.org/710277 | 14:26 |
| *** sri_ has quit IRC | 14:27 | |
| *** abdysn has quit IRC | 14:54 | |
| cosmicsound | mnasiadka , i have issues deploying magnum k8s cluster | 15:16 |
| cosmicsound | and mostly related to heat | 15:16 |
| *** ktibi has quit IRC | 15:21 | |
| *** diurnalist has joined #openstack-kolla | 15:29 | |
| mnasiadka | cosmicsound: paste your logs somewhere, I can take a look | 15:30 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: CentOS 8: Bump MichaelRigart.interfaces to 1.4.0 https://review.opendev.org/709786 | 15:39 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: Bump Ansible version range to 2.8 - 2.9 https://review.opendev.org/708922 | 15:39 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: CentOS 8: Disable ntpd, enable chrony container https://review.opendev.org/709785 | 15:39 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: CentOS 8: Add overcloud CI job https://review.opendev.org/709787 | 15:39 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: WIP: CI: Add overcloud host configure job https://review.opendev.org/710277 | 15:39 |
| *** ktibi has joined #openstack-kolla | 15:46 | |
| *** noxoid has quit IRC | 15:47 | |
| *** noxoid has joined #openstack-kolla | 15:47 | |
| *** noxoid has quit IRC | 15:50 | |
| *** bengates has quit IRC | 15:53 | |
| cosmicsound | mnasiadka , http://paste.openstack.org/show/790064/ this is last cloud-init log from master | 15:56 |
| *** priteau has joined #openstack-kolla | 16:00 | |
| *** bengates has joined #openstack-kolla | 16:07 | |
| *** factor has quit IRC | 16:08 | |
| *** factor has joined #openstack-kolla | 16:08 | |
| *** factor has quit IRC | 16:11 | |
| *** factor has joined #openstack-kolla | 16:13 | |
| *** factor has quit IRC | 16:22 | |
| mnasiadka | cosmicsound: and that's where it stops? | 16:22 |
| *** factor has joined #openstack-kolla | 16:22 | |
| *** factor has quit IRC | 16:23 | |
| *** dosaboy has quit IRC | 16:45 | |
| *** bengates has quit IRC | 16:48 | |
| cosmicsound | mnasiadka , yes | 16:57 |
| cosmicsound | it will eventually time out | 16:57 |
| cosmicsound | made a clean redeployment ow | 16:58 |
| cosmicsound | will try again and give feeds | 16:58 |
| *** dosaboy has joined #openstack-kolla | 17:01 | |
| *** diurnalist has quit IRC | 17:09 | |
| openstackgerrit | Pierre Riteau proposed openstack/kolla stable/stein: Bump stein versions https://review.opendev.org/710318 | 17:10 |
| *** skramaja has quit IRC | 17:13 | |
| *** diurnalist has joined #openstack-kolla | 17:14 | |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: WIP: CI: Add overcloud host configure job https://review.opendev.org/710277 | 17:16 |
| *** shyamb has joined #openstack-kolla | 17:17 | |
| *** shyamb has quit IRC | 17:17 | |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: WIP: CI: Add overcloud host configure job https://review.opendev.org/710277 | 17:23 |
| *** rpittau is now known as rpittau|afk | 17:23 | |
| *** ktibi has quit IRC | 17:27 | |
| cosmicsound | based on this: https://releases.openstack.org/teams/magnum.html#magnum-ui | 17:28 |
| cosmicsound | how do i know what version of release runs in my containers? | 17:28 |
| cosmicsound | is there a way to get all versions centralised | 17:28 |
| cosmicsound | i deploy from source, i would expect to have latest releases? | 17:31 |
| mnasiadka | cosmicsound: yes, if you deploy from source images - those should give you latest release | 17:34 |
| *** evrardjp has quit IRC | 17:35 | |
| *** evrardjp has joined #openstack-kolla | 17:35 | |
| cosmicsound | thanks mnasiadka | 17:38 |
| cosmicsound | and is there still a way to see the versions running? | 17:38 |
| cosmicsound | i tried openstack service list --long its not giving versions out | 17:38 |
| umbSublime | cosmicsound, not sure if this is what you mean, but there is `openstack versions show` (It will show API, and microversions) | 17:39 |
| *** igordc has joined #openstack-kolla | 17:39 | |
| umbSublime | Not related to release though, only API info | 17:39 |
| *** priteau has quit IRC | 17:40 | |
| cosmicsound | i tried also versions show | 17:43 |
| cosmicsound | umbSublime , for example the current train latest magnum release its 9.2.0-4 according to https://docs.openstack.org/releasenotes/magnum/train.html# | 17:44 |
| umbSublime | Are you talking about distribution package versions ? Or python module versions ? | 17:44 |
| cosmicsound | i was wondering if there is a way to get that release version from anywhere in cli | 17:44 |
| cosmicsound | yes distribution release of the package/module | 17:44 |
| umbSublime | Ohh, I'm pretty sure all binaries have a --version. I've never ran magnum, but something like `magnum --version` on the host running the service | 17:46 |
| umbSublime | s/binaries/services | 17:46 |
| cosmicsound | its not really what one expects | 17:47 |
| cosmicsound | if i run that i get the CLI version | 17:47 |
| cosmicsound | not the os runing | 17:47 |
| cosmicsound | or the os running is on 9.x and i get 2.x | 17:47 |
| cosmicsound | magnum --version | 17:48 |
| cosmicsound | 2.17.0 | 17:48 |
| umbSublime | Wouldn't that be the version of the client ? | 17:49 |
| mnasiadka | cosmicsound: pip freeze | grep magnum | 17:49 |
| mnasiadka | And that’s client version what you pasted | 17:49 |
| cosmicsound | yes i knnow | 17:50 |
| cosmicsound | magnum-conductor --version | 17:50 |
| cosmicsound | 9.2.0 | 17:50 |
| cosmicsound | this is the magnum realistic version i would say | 17:50 |
| cosmicsound | and in that case the latest release 9.2.0-4 is not in kolla yet? | 17:51 |
| cosmicsound | or should i try to manual tag/ | 17:51 |
| *** k_mouza has quit IRC | 17:51 | |
| cosmicsound | mnasiadka , should i try magnum_tag=9.2.0-4 in globals.yml | 17:52 |
| cosmicsound | and pull reconfigure? | 17:52 |
| mnasiadka | cosmicsound: that tag is for docker tag, and if you are downloading images from docker hub - it won’t work this way | 17:55 |
| *** igordc has quit IRC | 18:01 | |
| *** gfidente is now known as gfidente|afk | 18:04 | |
| *** igordc has joined #openstack-kolla | 18:04 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla stable/train: CentOS 8: Install python in the base image (Train only) https://review.opendev.org/710326 | 18:07 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: WIP: CI: Add overcloud host configure job https://review.opendev.org/710277 | 18:07 |
| cosmicsound | in globals.yml is the warning: # NOTE: This variable has been deprecated and will be removed in the U cycle. | 18:18 |
| cosmicsound | its this about cadf? | 18:18 |
| *** k_mouza has joined #openstack-kolla | 18:21 | |
| *** jimcrowleyibm has joined #openstack-kolla | 18:22 | |
| *** kplant has quit IRC | 18:24 | |
| *** tonythomas has quit IRC | 18:25 | |
| *** k_mouza has quit IRC | 18:25 | |
| *** sean-k-mooney has quit IRC | 19:05 | |
| *** kplant has joined #openstack-kolla | 19:06 | |
| -openstackstatus- NOTICE: Memory pressure on zuul.opendev.org is causing connection timeouts resulting in POST_FAILURE and RETRY_LIMIT results for some jobs since around 06:00 UTC today; we will be restarting the scheduler shortly to relieve the problem, and will follow up with another notice once running changes are reenqueued. | 19:11 | |
| -openstackstatus- NOTICE: The scheduler for zuul.opendev.org has been restarted; any changes which were in queues at the time of the restart have been reenqueued automatically, but any changes whose jobs failed with a RETRY_LIMIT, POST_FAILURE or NODE_FAILURE build result in the past 14 hours should be manually rechecked for fresh results | 19:45 | |
| mnasiadka | yoctozepto: about those renos - can you just fix them? I don’t want to look at them even second time :) | 20:00 |
| yoctozepto | mnasiadka: k, tomorrow (honestly me neither, but I still have a pending round to fix current ones) | 20:01 |
| yoctozepto | mnasiadka: ^^ could you btw recheck changes notice mentions? | 20:04 |
| *** diurnalist has quit IRC | 20:16 | |
| *** diurnalist has joined #openstack-kolla | 20:26 | |
| *** kozhukalov has quit IRC | 20:27 | |
| *** kozhukalov has joined #openstack-kolla | 20:28 | |
| *** kplant has quit IRC | 20:46 | |
| openstackgerrit | Merged openstack/kolla-ansible stable/train: Use more permissive regex to remove the offending 127.0.1.1 https://review.opendev.org/710126 | 20:51 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: Add /run/netns bindmount to Neutron containers https://review.opendev.org/710051 | 20:54 |
| *** kozhukalov has quit IRC | 21:08 | |
| openstackgerrit | Merged openstack/kolla-ansible stable/stein: Use more permissive regex to remove the offending 127.0.1.1 https://review.opendev.org/710127 | 21:26 |
| openstackgerrit | Merged openstack/kolla-ansible master: CI: Add addressing on external network https://review.opendev.org/709362 | 21:26 |
| *** ktibi has joined #openstack-kolla | 21:27 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Fix client TLS in neutron-metadata-agent https://review.opendev.org/709829 | 21:30 |
| openstackgerrit | Merged openstack/kolla-ansible master: Fix neutron-metadata-agent to use provided CA for Nova metadata https://review.opendev.org/709830 | 21:30 |
| *** dougsz has joined #openstack-kolla | 21:31 | |
| *** ktibi has quit IRC | 21:32 | |
| *** kozhukalov has joined #openstack-kolla | 21:40 | |
| *** negronjl has quit IRC | 21:46 | |
| *** negronjl has joined #openstack-kolla | 21:51 | |
| *** negronjl has quit IRC | 22:00 | |
| *** negronjl has joined #openstack-kolla | 22:02 | |
| *** jimcrowleyibm is now known as jimcrowleyibm[aw | 22:10 | |
| openstackgerrit | Merged openstack/kolla master: kibana: enable for non-x86 on Debian/Ubuntu https://review.opendev.org/707787 | 22:31 |
| *** dougsz has quit IRC | 22:44 | |
| *** negronjl has quit IRC | 22:48 | |
| *** negronjl has joined #openstack-kolla | 22:49 | |
| cosmicsound | anyone seen this with registry? | 22:51 |
| cosmicsound | Error response from daemon: Get https://registry.uhlhost.net:7777/v2/: error parsing HTTP 429 response body: invalid character 'R' looking for beginning of value: "Retry later\n" | 22:51 |
| cosmicsound | when i login from local machine works fine | 22:51 |
| cosmicsound | when i try to login from server i get that invalid char R | 22:51 |
| cosmicsound | ... | 22:52 |
| *** ivve has quit IRC | 22:57 | |
| cosmicsound | got it solved | 23:15 |
| cosmicsound | it was just server protection | 23:15 |
| *** kozhukalov has quit IRC | 23:35 | |
| *** cah_link1 has joined #openstack-kolla | 23:49 | |
| *** cah_link has quit IRC | 23:51 | |
| *** cah_link1 is now known as cah_link | 23:51 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
