Thursday, 2020-02-27

*** diurnalist has quit IRC00:21
*** cah_link has quit IRC00:44
*** diurnalist has joined #openstack-kolla00:46
*** diurnalist has quit IRC01:11
*** diurnalist has joined #openstack-kolla01:14
*** zhanglong has joined #openstack-kolla01:17
*** diurnalist has quit IRC01:36
*** zhanglong has quit IRC02:26
*** zhanglong has joined #openstack-kolla02:28
*** ktibi has joined #openstack-kolla03:05
*** ktibi has quit IRC03:09
*** skramaja has joined #openstack-kolla03:50
*** JangwonLee_ has joined #openstack-kolla04:52
*** JangwonLee has quit IRC04:54
*** evrardjp has quit IRC05:34
*** evrardjp has joined #openstack-kolla05:35
*** sri_ has joined #openstack-kolla05:49
*** zhanglong has quit IRC05:55
*** zhanglong has joined #openstack-kolla05:57
openstackgerritDincer Celik proposed openstack/kolla-ansible stable/train: Fixes gnocchi-api script name for Ubuntu/Debian  https://review.opendev.org/71018606:00
*** abdysn has joined #openstack-kolla06:03
*** kozhukalov has joined #openstack-kolla06:07
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: OVN Support  https://review.opendev.org/69684106:42
*** dannins has joined #openstack-kolla07:10
*** jbadiapa has joined #openstack-kolla07:19
*** dougsz has joined #openstack-kolla07:38
*** dougsz has quit IRC07:44
yoctozeptomorning07:59
yoctozeptowhat a silent day here07:59
openstackgerritPiotr Rabiega proposed openstack/kolla master: Add collectd-pcie-errors package for PCIe Errors (C7 only)  https://review.opendev.org/71002908:05
mnasiadkawell08:07
*** bengates has joined #openstack-kolla08:07
*** bengates has quit IRC08:07
*** bengates has joined #openstack-kolla08:08
kevinzhi there08:16
yoctozeptohi there as well08:17
mnasiadkayoctozepto: willing to +2+w this https://review.opendev.org/#/c/709204/4 - so we can go forward with most of the C8 backports on Train?08:45
patchbotpatch 709204 - kolla (stable/train) - CentOS 8: Use upstream Ceph/master - 4 patch sets08:45
yoctozeptoosmanlicilegi: could you elaborate on that /run/netns issue? when does it fail? seemingly not from start08:46
yoctozeptomnasiadka: ok08:46
*** stingrayza has quit IRC08:53
mnasiadkayoctozepto: there's a bug raised I think by kevinz (if kevinz == kevin-zhao)08:56
yoctozeptomnasiadka: yeah, but I'm not following :-)08:56
kevinzyes it is mine08:56
kevinzI met the issue for some times. but not 100% reproduced..it is werid08:57
kevinzweird08:57
mnasiadkayoctozepto: I have a weird idea to add/remove multiple subnets in the CI (and multiple routers) and see if that happens09:04
yoctozeptonah, I see the issue now09:08
yoctozeptoit needs same testing as the other bug with neutron09:08
yoctozeptolemme find it09:08
mnasiadkameaning?09:08
*** rpittau|afk is now known as rpittau09:09
mnasiadkaanyway, if it can't open netns - mounting /run/netns should be the answer09:09
yoctozeptohttps://bugs.launchpad.net/kolla-ansible/+bug/186398209:09
openstackLaunchpad bug 1863982 in kolla-ansible "Upgrade from Rocky to Stein, router namespace disappear" [Undecided,Confirmed] - Assigned to Radosław Piliszek (yoctozepto)09:09
mnasiadkaand if osmanlicilegi tested and it works for him (and no bindmount for /run/netns throws errors)09:09
yoctozeptomnasiadka: yeah, it does something with the problem09:09
yoctozeptoas for testing09:09
yoctozeptoit is most likely affected by restarts as it is when container data is lost09:10
mnasiadkaexit code 409:10
mnasiadkalovely09:10
yoctozeptostill testing a bit09:10
mnasiadkaif neutron would print the command in the debug, it would be even more lovely09:10
yoctozeptowhere 4?09:10
*** ktibi has joined #openstack-kolla09:15
mnasiadkain the bug you pasted09:15
yoctozeptoah, was not aware09:16
yoctozeptothough it looks like device exists but not its namespace09:17
yoctozeptoso they are kinda related too09:17
mnasiadkawonder if kevinz hit the problem, that in stein we did not mount /run at this point (because the backport already merged)09:18
yoctozeptomnasiadka: me too09:18
kevinzLaunchpad bug 1863982 in kolla-ansible "Upgrade from Rocky to Stein, router namespace disappear" [Undecided,Confirmed] - Assigned to Radosław Piliszek (yoctozepto)09:19
openstackLaunchpad bug 1863982 in kolla-ansible "Upgrade from Rocky to Stein, router namespace disappear" [Undecided,Confirmed] https://launchpad.net/bugs/1863982 - Assigned to Radosław Piliszek (yoctozepto)09:19
* osmanlicilegi is back09:19
yoctozeptonah09:19
yoctozeptoit was day before that09:19
mnasiadkaso /run/netns might not help at all09:19
kevinzthis one, I can get netns in l3_agent, and dhcp-agent eventually09:19
mnasiadka:D09:19
yoctozeptoyeah, indeed09:19
kevinzBut can't get the netns at hosts09:19
mnasiadkawell, it may help a bit09:19
yoctozeptoI was about testing, both seemingly are triggered by service restart09:20
mnasiadkawell, adding /run/netns back will not hurt, and at least helps osmanlicilegi's case09:20
yoctozeptoin CI we only set up router after it's all stable09:20
mnasiadka(so probably hurts everybody's case right now)09:20
yoctozeptoagreed09:20
kevinzand restart l3_agent will easy help09:20
osmanlicilegilet me check logs if there's any error since yesterday09:21
yoctozeptostill doing some fake testing localy, will report back and either +2 or -109:21
kevinzI've a question, after deploy a train cluster09:21
kevinzAnd the ns will just appear in * agent container?09:21
*** k_mouza has joined #openstack-kolla09:21
yoctozeptomnasiadka: we did not release the buggy version, did we?09:22
mnasiadkayoctozepto: no, no release since then09:22
yoctozeptomnasiadka: if we did not, then remove the reno09:22
yoctozeptomnasiadka: but adapt previous one09:22
*** tonythomas has joined #openstack-kolla09:25
yoctozeptoosmanlicilegi: you did not recreate the container, only restarted it, right?09:26
mnasiadkayoctozepto: yeah, will do09:27
mnasiadkayoctozepto, mgoddard: should we accept Kolla C7 changes at this point (like this one - https://review.opendev.org/#/c/710029/)? For me it totally makes no sense, because sooner or later we will drop C7 from master.09:28
patchbotpatch 710029 - kolla - Add collectd-pcie-errors package for PCIe Errors (... - 2 patch sets09:28
yoctozeptomnasiadka: only for bp09:29
mnasiadkayoctozepto: well, it's adding a collectd plugin package to c7, but there's no c8 part - so what's the point? :D09:29
yoctozeptomnasiadka: ask if they want it in bp, otherwise none09:30
mnasiadkamakes sense09:30
mnasiadkayoctozepto: around CI - I see aarch one is totally network-wise unreliable... I stopped looking why it fails ;)09:31
openstackgerritMerged openstack/kayobe master: CI: set previous_release to train  https://review.opendev.org/70914509:32
mnasiadka"stdout": "Error during database migration: Migration cannot continue until all volumes have been migrated to the `__DEFAULT__` volume type. Please run `cinder-manage db online_data_migrations`. There are still untyped volumes unmigrated.\n",09:33
yoctozeptomnasiadka: I never did :-)09:33
mnasiadkathat is interesting in the upgrade jobs09:33
yoctozeptooh my09:33
yoctozeptois it breaking? random?09:33
mnasiadkayeah, random :)09:34
mnasiadkaone job failed out of 4 upgrade jobs09:34
yoctozeptowonder where is the race there09:35
yoctozeptowhich part fails?09:35
mnasiadkarunning bootstrap09:37
mnasiadkahttps://e7aa0df1ac641e2ee8fd-b023e6398111e1b55f35dc3d44640709.ssl.cf1.rackcdn.com/710186/1/check/kolla-ansible-ubuntu-source-upgrade-ceph/6eb6b69/primary/logs/ansible/upgrade09:38
*** gfidente|afk is now known as gfidente09:39
mnasiadkaI don't think we're doing online data migration at any time09:39
mnasiadkalol, we are09:40
mnasiadkabut after db sync09:40
mnasiadkahttps://github.com/openstack/kolla/blob/master/docker/cinder/cinder-api/extend_start.sh09:40
yoctozeptowell, one can't do online migrations if schema is not ready?09:42
yoctozeptoweird09:43
yoctozeptoadd that to CI errors and we'll see to it later09:43
yoctozepto(much later)09:43
mnasiadkayoctozepto: and can't do db sync if volumes are untyped :D09:45
yoctozeptofwiw, https://bugs.launchpad.net/neutron/+bug/1812364 xD09:46
openstackLaunchpad bug 1812364 in neutron "Error "OSError: [Errno 22] failed to open netns" in l3-agent logs" [High,Confirmed] - Assigned to Miguel Lavalle (minsel)09:46
yoctozeptolooks like ubuntu being bad at netns in general09:47
*** cah_link has joined #openstack-kolla09:48
*** Nirtal has joined #openstack-kolla09:49
mnasiadkaawesome09:50
yoctozeptoI did some testing on centos, now proceeding on ubuntu09:51
mnasiadkayoctozepto: I assume centos works?09:57
mnasiadkayoctozepto: hmm, maybe Ubuntu and AppArmor does something nasty from time to time?09:57
yoctozeptomnasiadka: I am doing dry testing with ip netns to see how I can make it break09:58
yoctozeptomnasiadka: in different scenarios09:59
*** dmellado has quit IRC09:59
openstackgerritMark Goddard proposed openstack/kolla-ansible stable/train: CentOS 8: Add deploy jobs in CI  https://review.opendev.org/70953610:10
osmanlicilegimnasiadka, yoctozepto: I rolled back and tried from scratch, netns errors appeared imemdiately. in 5 minutes, all namespaces disappeared and I lost connectivity to vms. adding /run/netns fixes problem immediately.10:23
mnasiadkain 5 minutes might be the key10:24
mnasiadka:010:24
mnasiadka:)10:24
yoctozepto:O10:24
mnasiadkawe don't leave it stranding for 5 minutes10:24
mnasiadka(in CI)10:24
osmanlicilegiright10:25
yoctozeptoyeah, we don't10:26
yoctozeptoosmanlicilegi: but is it that the containers are started without mount10:26
yoctozeptoosmanlicilegi: routers are properly created10:27
osmanlicilegithere's one point I couldn't understand. the containers cannot manage netns after restarted so how previously creates namespaces disappear?10:27
yoctozeptoosmanlicilegi: and after 5 minuters poof?10:27
yoctozeptowell, as for my dry testing10:29
yoctozeptocentos and ubuntu behave much the same, error messages have been modified slightly10:29
yoctozeptoscenario 1: w/o mount10:29
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Custom haproxy script for monitoring galera  https://review.opendev.org/71021310:29
yoctozeptoif you create a netns in container10:30
yoctozeptoand then restart it10:30
yoctozeptothen netns is lost but you have its reference in filesystem10:30
yoctozeptowhich causes RTNETLINK errors about invalid reference10:31
yoctozeptobecause seemignly netns appears in /run/netns by means of touch and mount -t proc10:31
yoctozeptoand those magic proc mounts will be lost10:31
yoctozeptostill, /run/netns is not going anywhere10:32
openstackgerritMark Goddard proposed openstack/kolla-ansible stable/train: DNM: Testing cloudkitty in train  https://review.opendev.org/71021510:32
yoctozeptoif you recreate the container, then it's obviously starting from clean plate then, so only by means of neutron-server may it know previous namespace names10:32
yoctozeptoscenario 2: w/ mount10:32
yoctozeptoit must be shared, that's true10:33
yoctozeptootherwise it ends up like in scenario 1 but also affects recreations10:33
yoctozeptoas for what breaks ubuntu in 5 minutes - no idea10:34
osmanlicilegihmm you're right imho10:36
cosmicsoundgood day10:36
openstackgerritMichal Nasiadka proposed openstack/kolla master: WIP: Add mysql client to haproxy image for Galera state checking  https://review.opendev.org/71021710:38
mgoddardmorning all10:38
yoctozeptothat said, lemme break my semiproduction wrt those mounts10:39
*** cah_link has quit IRC10:43
*** stingrayza has joined #openstack-kolla10:43
openstackgerritMichal Nasiadka proposed openstack/kayobe master: Add framework to deploy user-defined containers on seed  https://review.opendev.org/70968910:45
cosmicsoundis this guide valid? https://docs.openstack.org/kolla-ansible/train/reference/containers/kuryr-guide.html10:45
cosmicsoundasking because there is a kolla.conf with this:10:46
cosmicsound[Service]10:46
cosmicsoundExecStart=10:46
cosmicsound# ExecStart commandline copied from 'docker-ce' package. Same on CentOS/Debian/Ubuntu systems.10:46
cosmicsoundExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock -H tcp://172.22.0.1:237510:46
*** riuzen has joined #openstack-kolla10:46
cosmicsoundand the docs mention this:10:46
cosmicsoundExecStart= -H tcp://172.16.1.13:2375 -H unix:///var/run/docker.sock --cluster-store=etcd://172.16.1.13:2379 --cluster-advertise=172.16.1.13:237510:46
cosmicsoundno more need for dockerd? or fd:// --containerd tags?10:46
mnasiadkayoctozepto: probably after 5 minutes it's trying to find if netns' are created, and fails miserably for some reason10:47
yoctozeptomnasiadka: well, they are still after 5 minutes :D10:48
yoctozeptostill there*10:48
mnasiadkayoctozepto: create routers, restart container, are they still there? :)10:48
yoctozeptomnasiadka: no, they aren't; I meant w/o restart10:49
yoctozeptoosmanlicilegi: you do service restart *after* router creation?10:49
mnasiadkaoh boy, 709203 failed on collecting logs and cancelled the whole pack of C8 changes10:50
mnasiadkamgoddard: ^^10:50
yoctozeptomeh10:50
osmanlicilegiyoctozepto: no svc restart10:50
yoctozeptomnasiadka: ^10:51
mnasiadkayoctozepto: changing bind mounts when reconfigure must restart containers, come on10:51
yoctozeptomnasiadka: yeah, so we are talking existing routers?10:53
yoctozeptoosmanlicilegi: ^10:53
osmanlicilegiyoctozepto: what you exactly mean by servie restart? you mean docker?10:53
osmanlicilegik-a only restarts containers you know10:53
mgoddardmnasiadka: :(10:53
yoctozeptoosmanlicilegi: yeah, I meant that10:54
yoctozeptoI was asking whether your scenario is that routers existed before10:54
yoctozeptoreconfigure happened10:54
osmanlicilegiI haven't had any svc or server reboot. you are right that my existing namespaces disappeared after restarting containers. I think I'm gonna open a case to canonical as this shouldn't happen if I think correctly.10:55
*** shyamb has joined #openstack-kolla10:56
osmanlicilegiin any situation I need /run/netns back :]10:56
openstackgerritMichal Nasiadka proposed openstack/kolla master: WIP: Add mysql client to haproxy image for Galera state checking  https://review.opendev.org/71021710:57
mnasiadkayoctozepto: but I think we do init-runonce before upgrade, and then run the upgrade - so this should show some problems?10:58
hrwmgoddard: https://paste.centos.org/view/afe5d480 fixes mistral/mistral-dashboard situation ;d11:00
*** bengates has quit IRC11:04
*** riuzen has quit IRC11:05
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: version-check: handle more situations  https://review.opendev.org/71022911:06
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: do not use unversioned Python binary  https://review.opendev.org/71023111:10
*** rpittau is now known as rpittau|bbl11:12
openstackgerritMerged openstack/kolla stable/stein: Bump stein versions  https://review.opendev.org/71006811:16
*** kozhukalov has quit IRC11:17
*** bengates has joined #openstack-kolla11:20
*** kozhukalov has joined #openstack-kolla11:21
*** ktibi has quit IRC11:22
yoctozeptoI reproduced11:23
openstackgerritMark Goddard proposed openstack/kayobe master: CentOS 8: Add overcloud CI job  https://review.opendev.org/70978711:23
yoctozeptoosmanlicilegi, mnasiadka, kevinz: I reproduced osmanlicilegi case11:23
yoctozeptothe scenario is you have some routers already11:23
yoctozeptoreconfigure w/o the mounts11:23
yoctozeptoeverything is fine and dandy11:23
yoctozepto(even after 5 minutes, sorry to spoil it!)11:24
openstackgerritMark Goddard proposed openstack/kayobe master: Make Kayobe code compatible with Python 3  https://review.opendev.org/63558611:24
openstackgerritMark Goddard proposed openstack/kayobe master: Bump Ansible version range to 2.8 - 2.9  https://review.opendev.org/70892211:24
openstackgerritMark Goddard proposed openstack/kayobe master: CentOS 8: Disable ntpd, enable chrony container  https://review.opendev.org/70978511:24
yoctozeptoyet if you restart it NOW11:24
openstackgerritMark Goddard proposed openstack/kayobe master: CentOS 8: Bump MichaelRigart.interfaces to 1.4.0  https://review.opendev.org/70978611:24
openstackgerritMark Goddard proposed openstack/kayobe master: CentOS 8: Add overcloud CI job  https://review.opendev.org/70978711:24
yoctozepto(.e.g l3 agent, these are nicert to break)11:24
yoctozepto(as dhcp has own quirks)11:24
yoctozeptoit breaks in the same way as osmanlicilegi pasted there11:24
*** bengates has quit IRC11:26
*** bengates has joined #openstack-kolla11:27
osmanlicilegiyoctozepto: namespaces still exists?11:31
yoctozeptoosmanlicilegi: they most likely still live on host, I commented on: https://bugs.launchpad.net/kolla-ansible/+bug/186485611:32
openstackLaunchpad bug 1864856 in kolla-ansible ussuri "dhcp-agent failed to open netns" [High,In progress] - Assigned to Michal Nasiadka (mnasiadka)11:32
*** kozhukalov has quit IRC11:35
yoctozeptothere is a workaround11:35
yoctozeptoclearn /run/netns inside the container and restart it11:35
*** kozhukalov has joined #openstack-kolla11:35
yoctozeptomnasiadka, osmanlicilegi, kevinz: btw, metadata agent does not require netns access so is irrelevant there11:41
*** kozhukalov has quit IRC11:43
yoctozeptootoh, ovn metadata does11:46
*** ivve has joined #openstack-kolla11:53
openstackgerritAlfredo Moralejo proposed openstack/kolla master: DNM Test new deps based on final CBS builds  https://review.opendev.org/71024012:03
*** kplant has joined #openstack-kolla12:07
osmanlicilegiyoctozepto, better than sherlock holmes.12:13
openstackgerritMichal Nasiadka proposed openstack/kolla master: WIP: Add mysql client to haproxy image for Galera state checking  https://review.opendev.org/71021712:17
mnasiadkayoctozepto: ovn does, it spawns processes in netns12:17
mnasiadkayoctozepto: the normal metadata agent doesn't? so how does that work? :D12:17
yoctozeptomnasiadka: l3&dhcp orchestrate data12:17
mnasiadkaah ok12:18
yoctozeptomnasiadka: the purpose of metadata is to listen on socket12:18
mnasiadkano l3 & dhcp in OVN12:18
yoctozeptoyeah12:18
yoctozeptoso they had to move that part12:18
yoctozeptoovn is different design12:18
yoctozeptohas*12:18
mnasiadkaok, makes sense - we could do /run/netns in metadata based on neutron_plugin_agent in the OVN change - once /run/netns merges12:20
mnasiadkaso we don't need /run/netns in metadata now12:20
mnasiadkalet me update change12:20
*** bengates has quit IRC12:20
yoctozeptomnasiadka: yeah, also one more thing, now testing whether it could be separate docker volume12:21
mnasiadkait could, but if we share with the host - you can do ip netns exec from the host12:22
mnasiadkawhich is helpful12:22
yoctozeptomore isolation vs less12:22
* osmanlicilegi brb12:23
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: Add /run/netns bindmount to Neutron containers  https://review.opendev.org/71005112:25
mnasiadkamgoddard: any idea which path should we take? more isolation vs less? :D12:26
yoctozeptomnasiadka: reno to fix12:27
yoctozeptomnasiadka: rather add12:27
mnasiadkawell, you wanted me to adapt the old one?12:27
yoctozeptomnasiadka: exactly12:28
yoctozeptomnasiadka: ah, sorry, gerrit playing tricks on me12:28
openstackgerritChason Chan proposed openstack/kolla-ansible master: [Docs] Pin kolla-anisble to the same version of quickstart guide  https://review.opendev.org/70789612:28
mnasiadkayoctozepto: How many bugs should I reference in the reno? both, old one or new one? :D12:29
yoctozeptomnasiadka: it's fine now12:29
mnasiadkame no touch12:29
yoctozeptomnasiadka: just see if it renders nicely12:29
mnasiadkaah, we wanted to run doc8 against reno12:29
mgoddardthe question is whether to put /run/netns in a docker volume vs bind mounted to /run/netns?12:29
mnasiadkamgoddard: yup12:30
mnasiadkamgoddard: sometimes it's nice to be able to do ip netns exec from the host level12:30
mgoddardI would say bind mount is less weird12:30
mgoddard+112:30
mgoddardand people expect to be able to do it12:30
mgoddard also it changes one less thing in a fairly tricky level12:31
mgoddardcould be knock-on effects of not being shared12:31
*** ktibi has joined #openstack-kolla12:32
*** cz3 is now known as papiez12:33
*** papiez is now known as cz312:33
*** shyamb has quit IRC12:33
*** rlljorge has joined #openstack-kolla12:34
yoctozeptomoreover, you can't use volume mount12:35
yoctozeptobecause they don't support submounts :-)12:35
yoctozeptoproblem solved, go with direct bind12:36
yoctozeptoalso, /run/netns needs to be on tmpfs because it must be cleared with kernel state12:36
yoctozeptootherwise reboots would break12:36
yoctozeptoso the default place is best12:36
yoctozeptoeot12:36
yoctozepto;D12:36
*** kozhukalov has joined #openstack-kolla12:39
*** shyamb has joined #openstack-kolla12:40
*** cah_link has joined #openstack-kolla12:47
mnasiadkayoctozepto: well, we are generating reno, but I would expect build-releasenotes job would also lint them a bit better12:49
yoctozeptomnasiadka: I got answer on ml - no way12:51
yoctozeptomnasiadka: and no plans (no sorry variant)12:52
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: CI: Add doc8 linting of release notes  https://review.opendev.org/71024412:55
mnasiadkalet's see - better than nothing12:55
*** shyamb has quit IRC12:56
*** negronjl has quit IRC12:56
openstackgerritMichal Nasiadka proposed openstack/kayobe master: Add framework to deploy user-defined containers on seed  https://review.opendev.org/70968912:57
*** negronjl has joined #openstack-kolla12:59
*** sean-k-mooney has joined #openstack-kolla13:00
*** ktibi has quit IRC13:00
mnasiadkawhat's happening with Zuul today... everything is so slooooow...13:01
*** ktibi has joined #openstack-kolla13:03
*** dmellado has joined #openstack-kolla13:05
*** shyamb has joined #openstack-kolla13:05
yoctozeptomnasiadka: check if tripleo is in the queues13:08
mnasiadkayoctozepto: tripleo is always in the queue13:09
yoctozeptomnasiadka: then you have your answer ;-)13:09
yoctozeptomnasiadka: https://review.opendev.org/709829 and https://review.opendev.org/70983013:12
patchbotpatch 709829 - kolla-ansible - Fix client TLS in neutron-metadata-agent - 1 patch set13:12
patchbotpatch 709830 - kolla-ansible - Fix neutron-metadata-agent to use provided CA for ... - 1 patch set13:12
yoctozeptomnasiadka: tyvm13:13
mnasiadkayw13:13
*** shyamb has quit IRC13:16
cosmicsoundyoctozepto , this13:17
cosmicsoundTASK [haproxy : Checking if kolla_internal_vip_address and kolla_external_vip_address are not pingable from any node] ***********************************************13:17
cosmicsoundok: [compute-1] => (item={'address': '172.22.0.22', 'command': 'ping'})13:17
cosmicsoundok: [compute-1] => (item={'address': '151.80.247.164', 'command': 'ping'})13:17
openstackgerritMichal Nasiadka proposed openstack/kolla master: WIP: Add mysql client to haproxy image for Galera state checking  https://review.opendev.org/71021713:17
cosmicsoundlets see how this works13:18
cosmicsoundfatal: [compute-1]: FAILED! => {"changed": false, "cmd": ["ip", "-o", "addr", "show", "dev", "eno1"], "delta": "0:00:00.001558", "end": "2020-02-27 13:17:31.084314", "failed_when_result": true, "rc": 0, "start": "2020-02-27 13:17:31.082756", "stderr": "", "stderr_lines": [], "stdout": "2: eno1    inet 151.80.247.160/29 brd 151.80.247.167 scope13:18
cosmicsoundglobal eno1\\       valid_lft forever preferred_lft forever\n2: eno1    inet 51.91.153.141/24 brd 51.91.153.255 scope global dynamic eno1\\       valid_lft 85775sec preferred_lft 85775sec\n2: eno1    inet6 fe80::d250:99ff:fed6:c83a/64 scope link \\       valid_lft forever preferred_lft forever", "stdout_lines": ["2: eno1    inet 151.80.247.160/2913:18
cosmicsoundbrd 151.80.247.167 scope global eno1\\       valid_lft forever preferred_lft forever", "2: eno1    inet 51.91.153.141/24 brd 51.91.153.255 scope global dynamic eno1\\       valid_lft 85775sec preferred_lft 85775sec", "2: eno1    inet6 fe80::d250:99ff:fed6:c83a/64 scope link \\       valid_lft forever preferred_lft forever"]}13:18
yoctozepto'tis is madness13:19
cosmicsoundhmm the vip must be pingable from all nodes13:19
yoctozeptothis is*13:19
cosmicsoundi think i know whats wrong13:19
mnasiadkayoctozepto: https://review.opendev.org/#/c/710213/1 - do we disable haproxy user somewhere? or what are you referring to?13:19
patchbotpatch 710213 - kolla-ansible - WIP: Custom haproxy script for monitoring galera - 1 patch set13:19
yoctozeptomnasiadka: yeah, we do for the original check13:19
yoctozeptomnasiadka: in mariadb itself13:19
yoctozeptomnasiadka: it is mostly ineffective as elaborated elsewhere13:20
yoctozeptomnasiadka: but I also don't think yours is a huge improvement (sorry!)13:20
yoctozeptomnasiadka: we probably need to replace haproxy by itself13:20
mnasiadkayoctozepto: anything that will disable backend on real terms is better - mysqlchk in haproxy is lame13:21
yoctozeptomnasiadka: still, I think yours could be better than current13:21
yoctozeptomnasiadka: agreed13:21
mnasiadkayoctozepto: and adding proxysql is rather big work13:21
yoctozeptomnasiadka: also agreed13:21
mnasiadkayoctozepto: so if tripleo can survive with clusterchk script, we maybe can survive with that13:22
*** sm806 has quit IRC13:22
yoctozeptomnasiadka: they use pacemaker, they are making pace13:22
yoctozeptomnasiadka: we use keepalived, we want to keep it alive13:22
yoctozepto:D13:22
mnasiadkayoctozepto: but still they rely on clusterchk in xinetd and doing galera checks that way13:23
yoctozeptomnasiadka: yeah, xinetd is lovely13:23
yoctozeptomnasiadka: what does osa do?13:23
mnasiadka4th attempt of multinode on OVN change... it's getting better and better13:23
mnasiadkayoctozepto: they do lxc, I don't know if I want to know :)13:23
*** rpittau|bbl is now known as rpittau13:24
kplanthey yankcrime - did you ever get the openstack cli working with openid here: https://www.stackhpc.com/tag/keycloak.html ?13:26
yankcrimekplant: no, never finished that bit off sadly - ping johnthetubaguy when he's around, he might have done some more with it since13:27
kplantappreciatei t13:28
yoctozeptomnasiadka: xD13:28
yoctozeptomnasiadka: but they also do plain host13:28
mnasiadkayoctozepto: and systemd-nspawn13:29
yoctozeptomnasiadka: really?13:29
mnasiadkayoctozepto: really!13:29
yoctozeptomnasiadka: I thought they were the last bastion of "deploy on metal"13:30
mnasiadka6h 50m13:32
mnasiadkaOVN change in check queue13:32
mnasiadkalol13:32
mnasiadkayoctozepto: seems we're linting reno: https://zuul.opendev.org/t/openstack/build/cebe1c0738f7451284cbf47dd1b18c7713:33
yoctozeptomnasiadka: oh my, that's long!13:33
yoctozeptomnasiadka: it's contextless but still better than nothing :-)13:34
mnasiadkayoctozepto: at least we're checking for line length :D13:35
*** sm806 has joined #openstack-kolla13:37
yoctozeptomnasiadka: heck yeah13:39
*** bengates has joined #openstack-kolla13:39
*** bengates has quit IRC13:40
*** bengates has joined #openstack-kolla13:40
yoctozeptomnasiadka: fwiw, today's party with neutron made me read enough neutron code to discover a security bug :O13:40
mnasiadkayoctozepto: lol, raise it :)13:42
mnasiadkafixing those lines... ugh13:43
yoctozeptomnasiadka: writing it up now13:43
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: CI: Add doc8 linting of release notes  https://review.opendev.org/71024414:01
mnasiadkayoctozepto: fixed all too long lines ^^14:02
*** zhanglong has quit IRC14:05
yoctozeptomnasiadka: Thanks for reporting a bug, the Neutron team loves you!14:10
yoctozeptomnasiadka: I hope they will still :P14:11
cosmicsoundis this normal that they have no domain?  https://mdb.uhlhost.net/uploads/58def43858473468/image.png14:14
yoctozeptoI guess?14:17
mnasiadkacosmicsound: why not?14:20
*** rlljorge has quit IRC14:23
openstackgerritMark Goddard proposed openstack/kayobe master: WIP: CI: Add overcloud host configure job  https://review.opendev.org/71027714:26
*** sri_ has quit IRC14:27
*** abdysn has quit IRC14:54
cosmicsoundmnasiadka , i have issues deploying magnum k8s cluster15:16
cosmicsoundand mostly related to heat15:16
*** ktibi has quit IRC15:21
*** diurnalist has joined #openstack-kolla15:29
mnasiadkacosmicsound: paste your logs somewhere, I can take a look15:30
openstackgerritMark Goddard proposed openstack/kayobe master: CentOS 8: Bump MichaelRigart.interfaces to 1.4.0  https://review.opendev.org/70978615:39
openstackgerritMark Goddard proposed openstack/kayobe master: Bump Ansible version range to 2.8 - 2.9  https://review.opendev.org/70892215:39
openstackgerritMark Goddard proposed openstack/kayobe master: CentOS 8: Disable ntpd, enable chrony container  https://review.opendev.org/70978515:39
openstackgerritMark Goddard proposed openstack/kayobe master: CentOS 8: Add overcloud CI job  https://review.opendev.org/70978715:39
openstackgerritMark Goddard proposed openstack/kayobe master: WIP: CI: Add overcloud host configure job  https://review.opendev.org/71027715:39
*** ktibi has joined #openstack-kolla15:46
*** noxoid has quit IRC15:47
*** noxoid has joined #openstack-kolla15:47
*** noxoid has quit IRC15:50
*** bengates has quit IRC15:53
cosmicsoundmnasiadka , http://paste.openstack.org/show/790064/ this is last cloud-init log from master15:56
*** priteau has joined #openstack-kolla16:00
*** bengates has joined #openstack-kolla16:07
*** factor has quit IRC16:08
*** factor has joined #openstack-kolla16:08
*** factor has quit IRC16:11
*** factor has joined #openstack-kolla16:13
*** factor has quit IRC16:22
mnasiadkacosmicsound: and that's where it stops?16:22
*** factor has joined #openstack-kolla16:22
*** factor has quit IRC16:23
*** dosaboy has quit IRC16:45
*** bengates has quit IRC16:48
cosmicsoundmnasiadka , yes16:57
cosmicsoundit will eventually time out16:57
cosmicsoundmade a clean redeployment ow16:58
cosmicsoundwill try again and give feeds16:58
*** dosaboy has joined #openstack-kolla17:01
*** diurnalist has quit IRC17:09
openstackgerritPierre Riteau proposed openstack/kolla stable/stein: Bump stein versions  https://review.opendev.org/71031817:10
*** skramaja has quit IRC17:13
*** diurnalist has joined #openstack-kolla17:14
openstackgerritMark Goddard proposed openstack/kayobe master: WIP: CI: Add overcloud host configure job  https://review.opendev.org/71027717:16
*** shyamb has joined #openstack-kolla17:17
*** shyamb has quit IRC17:17
openstackgerritMark Goddard proposed openstack/kayobe master: WIP: CI: Add overcloud host configure job  https://review.opendev.org/71027717:23
*** rpittau is now known as rpittau|afk17:23
*** ktibi has quit IRC17:27
cosmicsoundbased on this: https://releases.openstack.org/teams/magnum.html#magnum-ui17:28
cosmicsoundhow do i know what version of release runs in my containers?17:28
cosmicsoundis there a way to get all versions centralised17:28
cosmicsoundi deploy from source, i would expect to have latest releases?17:31
mnasiadkacosmicsound: yes, if you deploy from source images - those should give you latest release17:34
*** evrardjp has quit IRC17:35
*** evrardjp has joined #openstack-kolla17:35
cosmicsoundthanks mnasiadka17:38
cosmicsoundand is there still a way to see the versions running?17:38
cosmicsoundi tried openstack service list --long its not giving versions out17:38
umbSublimecosmicsound, not sure if this is what you mean, but there is `openstack versions show` (It will show API, and microversions)17:39
*** igordc has joined #openstack-kolla17:39
umbSublimeNot related to release though, only API info17:39
*** priteau has quit IRC17:40
cosmicsoundi tried also versions show17:43
cosmicsoundumbSublime , for example the current train latest magnum release its 9.2.0-4 according to https://docs.openstack.org/releasenotes/magnum/train.html#17:44
umbSublimeAre you talking about distribution package versions ? Or python module versions ?17:44
cosmicsoundi was wondering if there is a way to get that release version from anywhere in cli17:44
cosmicsoundyes distribution release of the package/module17:44
umbSublimeOhh, I'm pretty sure all binaries have a --version. I've never ran magnum, but something like `magnum --version` on the host running the service17:46
umbSublimes/binaries/services17:46
cosmicsoundits not really what one expects17:47
cosmicsoundif i run that i get the CLI version17:47
cosmicsoundnot the os runing17:47
cosmicsoundor the os running is on 9.x and i get 2.x17:47
cosmicsoundmagnum --version17:48
cosmicsound2.17.017:48
umbSublimeWouldn't that be the version of the client ?17:49
mnasiadkacosmicsound: pip freeze | grep magnum17:49
mnasiadkaAnd that’s client version what you pasted17:49
cosmicsoundyes i knnow17:50
cosmicsoundmagnum-conductor --version17:50
cosmicsound9.2.017:50
cosmicsoundthis is the magnum realistic version i would say17:50
cosmicsoundand in that case the latest release 9.2.0-4 is not in kolla yet?17:51
cosmicsoundor should i try to manual tag/17:51
*** k_mouza has quit IRC17:51
cosmicsoundmnasiadka , should i try magnum_tag=9.2.0-4 in globals.yml17:52
cosmicsoundand pull reconfigure?17:52
mnasiadkacosmicsound: that tag is for docker tag, and if you are downloading images from docker hub - it won’t work this way17:55
*** igordc has quit IRC18:01
*** gfidente is now known as gfidente|afk18:04
*** igordc has joined #openstack-kolla18:04
openstackgerritMark Goddard proposed openstack/kolla stable/train: CentOS 8: Install python in the base image (Train only)  https://review.opendev.org/71032618:07
openstackgerritMark Goddard proposed openstack/kayobe master: WIP: CI: Add overcloud host configure job  https://review.opendev.org/71027718:07
cosmicsoundin globals.yml is the warning: # NOTE: This variable has been deprecated and will be removed in the U cycle.18:18
cosmicsoundits this about cadf?18:18
*** k_mouza has joined #openstack-kolla18:21
*** jimcrowleyibm has joined #openstack-kolla18:22
*** kplant has quit IRC18:24
*** tonythomas has quit IRC18:25
*** k_mouza has quit IRC18:25
*** sean-k-mooney has quit IRC19:05
*** kplant has joined #openstack-kolla19:06
-openstackstatus- NOTICE: Memory pressure on zuul.opendev.org is causing connection timeouts resulting in POST_FAILURE and RETRY_LIMIT results for some jobs since around 06:00 UTC today; we will be restarting the scheduler shortly to relieve the problem, and will follow up with another notice once running changes are reenqueued.19:11
-openstackstatus- NOTICE: The scheduler for zuul.opendev.org has been restarted; any changes which were in queues at the time of the restart have been reenqueued automatically, but any changes whose jobs failed with a RETRY_LIMIT, POST_FAILURE or NODE_FAILURE build result in the past 14 hours should be manually rechecked for fresh results19:45
mnasiadkayoctozepto: about those renos - can you just fix them? I don’t want to look at them even second time :)20:00
yoctozeptomnasiadka: k, tomorrow (honestly me neither, but I still have a pending round to fix current ones)20:01
yoctozeptomnasiadka: ^^ could you btw recheck changes notice mentions?20:04
*** diurnalist has quit IRC20:16
*** diurnalist has joined #openstack-kolla20:26
*** kozhukalov has quit IRC20:27
*** kozhukalov has joined #openstack-kolla20:28
*** kplant has quit IRC20:46
openstackgerritMerged openstack/kolla-ansible stable/train: Use more permissive regex to remove the offending 127.0.1.1  https://review.opendev.org/71012620:51
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: Add /run/netns bindmount to Neutron containers  https://review.opendev.org/71005120:54
*** kozhukalov has quit IRC21:08
openstackgerritMerged openstack/kolla-ansible stable/stein: Use more permissive regex to remove the offending 127.0.1.1  https://review.opendev.org/71012721:26
openstackgerritMerged openstack/kolla-ansible master: CI: Add addressing on external network  https://review.opendev.org/70936221:26
*** ktibi has joined #openstack-kolla21:27
openstackgerritMerged openstack/kolla-ansible master: Fix client TLS in neutron-metadata-agent  https://review.opendev.org/70982921:30
openstackgerritMerged openstack/kolla-ansible master: Fix neutron-metadata-agent to use provided CA for Nova metadata  https://review.opendev.org/70983021:30
*** dougsz has joined #openstack-kolla21:31
*** ktibi has quit IRC21:32
*** kozhukalov has joined #openstack-kolla21:40
*** negronjl has quit IRC21:46
*** negronjl has joined #openstack-kolla21:51
*** negronjl has quit IRC22:00
*** negronjl has joined #openstack-kolla22:02
*** jimcrowleyibm is now known as jimcrowleyibm[aw22:10
openstackgerritMerged openstack/kolla master: kibana: enable for non-x86 on Debian/Ubuntu  https://review.opendev.org/70778722:31
*** dougsz has quit IRC22:44
*** negronjl has quit IRC22:48
*** negronjl has joined #openstack-kolla22:49
cosmicsoundanyone seen this with registry?22:51
cosmicsoundError response from daemon: Get https://registry.uhlhost.net:7777/v2/: error parsing HTTP 429 response body: invalid character 'R' looking for beginning of value: "Retry later\n"22:51
cosmicsoundwhen i login from local machine works fine22:51
cosmicsoundwhen i try to login from server i get that invalid char R22:51
cosmicsound...22:52
*** ivve has quit IRC22:57
cosmicsoundgot it solved23:15
cosmicsoundit was just server protection23:15
*** kozhukalov has quit IRC23:35
*** cah_link1 has joined #openstack-kolla23:49
*** cah_link has quit IRC23:51
*** cah_link1 is now known as cah_link23:51

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!