Thursday, 2020-03-19

« Wednesday, 2020-03-18 Index Friday, 2020-03-20 »
*** jimcrowleyibm is now known as jimcrowleyibm[aw00:08
*** jimcrowleyibm[aw has quit IRC00:08
*** diurnalist has quit IRC00:12
*** k_mouza has joined #openstack-kolla00:16
*** k_mouza has quit IRC00:20
*** sean-k-mooney2 has joined #openstack-kolla00:26
*** sean-k-mooney1 has quit IRC00:27
*** sean-k-mooney2 has quit IRC00:53
*** sean-k-mooney2 has joined #openstack-kolla00:55
*** sean-k-mooney2 has quit IRC01:08
*** vishalmanchanda has joined #openstack-kolla01:33
*** sorin-mihai has quit IRC01:38
*** sorin-mihai has joined #openstack-kolla01:38
*** mgoddard has quit IRC01:49
*** mgoddard has joined #openstack-kolla01:57
*** diurnalist has joined #openstack-kolla02:09
*** diurnalist has quit IRC02:14
*** zhanglong has joined #openstack-kolla02:28
openstackgerritJames Kirsch proposed openstack/kolla-ansible master: Add support for encrypting backend HAProxy traffic  https://review.opendev.org/71200502:28
*** k_mouza has joined #openstack-kolla03:03
*** k_mouza has quit IRC03:08
*** diurnalist has joined #openstack-kolla03:42
*** skramaja has joined #openstack-kolla04:47
*** evrardjp has quit IRC05:36
*** evrardjp has joined #openstack-kolla05:36
*** riuzen has quit IRC05:55
*** zhanglong has quit IRC05:57
*** diurnalist has quit IRC05:57
*** zhanglong has joined #openstack-kolla05:57
*** diurnalist has joined #openstack-kolla06:11
*** abdysn has joined #openstack-kolla06:11
*** riuzen has joined #openstack-kolla06:15
*** diurnalist has quit IRC06:16
riuzenI wanna ask about ironic, could I use same network for cleaning network+provisioning network?06:17
*** skramaja_ has joined #openstack-kolla06:51
*** skramaja has quit IRC06:51
*** k_mouza has joined #openstack-kolla07:04
*** k_mouza has quit IRC07:08
*** cah_link has joined #openstack-kolla07:12
*** gfidente|afk has quit IRC07:31
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible stable/train: DNM: Debug Ceph permission weirdness on reconfigure  https://review.opendev.org/71369107:35
mnasiadkariuzen: better to ask on #openstack-ironic07:36
*** riuzen has quit IRC07:36
openstackgerritGrzegorz Sikorski proposed openstack/kolla-ansible stable/rocky: rabbitmq: update cluster version on reconfigure  https://review.opendev.org/71362807:37
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible stable/train: DNM: Debug Ceph permission weirdness on reconfigure  https://review.opendev.org/71369107:39
*** JangwonLee has quit IRC07:58
*** JangwonLee has joined #openstack-kolla07:59
*** bengates has joined #openstack-kolla08:00
*** klindgren has quit IRC08:01
*** klindgren has joined #openstack-kolla08:01
*** threestrands has quit IRC08:01
openstackgerritGrzegorz Sikorski proposed openstack/kolla-ansible stable/rocky: rabbitmq: update cluster version on reconfigure  https://review.opendev.org/71362808:08
*** rwellum_ has quit IRC08:08
*** portdirect has quit IRC08:08
*** kevinz has quit IRC08:08
*** crindi has quit IRC08:09
*** mwhahaha has quit IRC08:09
*** portdirect has joined #openstack-kolla08:10
*** crindi has joined #openstack-kolla08:10
*** mwhahaha has joined #openstack-kolla08:10
*** rwellum_ has joined #openstack-kolla08:10
*** kevinz has joined #openstack-kolla08:11
*** rpittau|afk is now known as rpittau08:11
*** chrizl has quit IRC08:20
*** zhanglong has quit IRC08:22
*** zhanglong has joined #openstack-kolla08:27
*** klindgren has quit IRC08:31
*** klindgren has joined #openstack-kolla08:31
*** jbadiapa has joined #openstack-kolla08:33
yoctozeptomnasiadka: https://etherpad.openstack.org/p/kolla-ci-errors ceph clients quality -> sigsegv08:34
*** dougsz has joined #openstack-kolla08:38
*** klindgren has quit IRC08:38
*** klindgren has joined #openstack-kolla08:39
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible stable/stein: Ironic: fix documentation  https://review.opendev.org/71379408:39
hrwmgoddard: what do you think about publishing aarch64 images as :tag-aarch64?08:41
hrwI would like to add debian-source-upgrade job08:42
hrwfirst x86-64 then also aarch6408:42
*** riuzen has joined #openstack-kolla08:44
openstackgerritMarcin Juszkiewicz proposed openstack/kolla-ansible master: CI: add Debian/source upgrade job  https://review.opendev.org/71379508:46
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: CI: Enable fluentd in MariaDB jobs  https://review.opendev.org/71366309:01
yoctozeptohrw: ++ for debian upgrade09:02
openstackgerritMarcin Juszkiewicz proposed openstack/kolla-ansible master: CI: add Debian/source upgrade job  https://review.opendev.org/71379509:03
hrwyoctozepto: thx for comments09:03
hrw-ENOTENOUGHCOFFEEYET09:03
*** yoctozepto has quit IRC09:07
*** klindgren has quit IRC09:10
*** yoctozepto has joined #openstack-kolla09:10
*** klindgren has joined #openstack-kolla09:11
*** skramaja_ has quit IRC09:12
*** diurnalist has joined #openstack-kolla09:12
*** skramaja has joined #openstack-kolla09:14
mnasiadkayoctozepto: but did the glance image upload fail on that job, or what's the issue? ?:)09:16
yoctozeptomnasiadka: glance api container died09:16
*** yoctozepto0 has joined #openstack-kolla09:16
mnasiadkano backtrace no nothing?09:16
yoctozeptomnasiadka: sigsegv code, nothing else09:17
yoctozeptomnasiadka: 1 of 3 died09:17
yoctozeptojust like that :-)09:18
yoctozeptoit was at loading rbd store09:18
yoctozeptoand sigsegv comes from bad c, not python09:18
*** diurnalist has quit IRC09:18
mnasiadkabut still, that glance debug is lousy09:18
yoctozeptoso I blame ceph client libs09:18
mnasiadkayoctozepto: does it happen everytime?09:18
yoctozeptomnasiadka: no, happened only once so far09:19
yoctozeptomnasiadka: treat it as curiosity, not something to fix asap :-)09:19
mnasiadkawe have too many in ,,curiosity'' category :D09:19
yoctozeptomnasiadka: indeed, whole kolla-ci-errors is just that :D09:19
cosmicsoundgood day09:26
*** yoctozepto9 has joined #openstack-kolla09:27
*** yoctozepto0 has quit IRC09:27
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible stable/train: ceph: Add become to gathering OSD IDs on reconfigure  https://review.opendev.org/71369109:29
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible stable/train: ceph: Add become to gathering OSD IDs on reconfigure  https://review.opendev.org/71369109:30
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible stable/stein: ceph: Add become to gathering OSD IDs on reconfigure  https://review.opendev.org/71380509:30
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible stable/rocky: ceph: Add become to gathering OSD IDs on reconfigure  https://review.opendev.org/71380609:30
mnasiadkayoctozepto: easy as that ^^09:30
openstackgerritMark Goddard proposed openstack/kolla-ansible stable/stein: Ironic: fix documentation  https://review.opendev.org/71379409:31
yoctozeptomnasiadka: dafu09:31
mgoddardmorning09:32
mnasiadkayoctozepto: I guess they fixed a bug in ceph with wrong permissions :D09:32
mnasiadkamorning mgoddard09:33
*** riuzen has quit IRC09:34
yoctozeptomorning mgoddard09:34
mgoddardhow are we all today?09:34
yoctozeptomnasiadka: lol, we might want a reno for that as it surely broke folks...09:34
mnasiadkayoctozepto: ah right, let me add09:35
mnasiadkamgoddard: I would be better, if we merged a change to use :master tag back, instead of :master-centos809:35
*** ktibi has joined #openstack-kolla09:35
mnasiadkaand I guess we should remove images with :master-centos8 tag from Docker Hub :D09:35
yoctozeptowhich reminds me I am to add a tool to create those reno templates09:35
yoctozeptomnasiadka: ++09:35
yoctozeptomnasiadka: add that to centos8 board09:36
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible stable/stein: Ironic: fix documentation  https://review.opendev.org/71379409:37
*** zhanglong has quit IRC09:42
hrwmgoddard: fine.09:42
hrwI managed to achieve my plan - 3TB of data before vacations ;d09:43
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible stable/train: ceph: Add become to gathering OSD IDs on reconfigure  https://review.opendev.org/71369109:46
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible stable/train: ceph: Add become to gathering OSD IDs on reconfigure  https://review.opendev.org/71369109:46
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible stable/stein: ceph: Add become to gathering OSD IDs on reconfigure  https://review.opendev.org/71380509:46
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible stable/rocky: ceph: Add become to gathering OSD IDs on reconfigure  https://review.opendev.org/71380609:46
mnasiadkahrw: planning vacations in this crazy chaos is an achievement :)09:47
*** Fl1nt has joined #openstack-kolla09:52
*** riuzen has joined #openstack-kolla09:53
Fl1ntGood morning everyone! How is your lockdown going today?09:54
cosmicsoundif anyone can give me some hints what would be config around such network: http://paste.openstack.org/show/3HYGn7Ol6krhGrttk2a4/09:54
cosmicsoundit seems something is wrong in my current setup, maybe someone can see another way out of this09:55
yoctozeptoFl1nt: good, how's yours?09:56
hrwmnasiadka: D:09:56
cosmicsoundlocked down with good music to work best ;)09:56
mnasiadkahrw: although I have some planned vacations in June :)09:56
hrwmnasiadka: took almost 2 years to collect 3rd TB. 4th one should be quicker09:57
hrw;D09:57
cosmicsoundany neutron/network hints on my config are welcomed :)09:57
openstackgerritMerged openstack/kayobe master: Docs: fix libvirt connection URI for tenks  https://review.opendev.org/70766709:58
*** vishalmanchanda has quit IRC10:03
Fl1nt@yoctozepto, Very good thanks, a bright sunny one fortunatly ^^10:03
Fl1nt@cosmicsound, hell yeah ^^10:04
Fl1nt@cosmicsound, what do you want to know about that conf?10:04
cosmicsoundFl1nt , well it seems no matter what I do my magnum does not works, i troubleshooted a while and i guess something is wrong on my network10:05
cosmicsoundsince eno2 can route only vrack/local vlans, and eno1 is public10:06
yoctozeptoFl1nt: glad to hear that10:06
Fl1ntyour neutron external interface should be eno1 not eno2 as it is your eno1 nic that hold your public IP10:06
yoctozeptoFl1nt: here not too much, but the air is nice and clean :-)10:06
cosmicsoundmy cofig has eno2 as neutron interface, i guess thats the main issue, just that if I use eno1 as neutron that will kill my current public network ip assigned on eno110:07
Fl1nt@yoctozepto, As long as you're safe, it's already a good starting point :D10:07
yoctozeptoFl1nt, cosmicsound: neutron external interface will have its ip address eaten up by neutron external bridge10:07
yoctozeptoFl1nt: indeed!10:07
Fl1nt@cosmicsound, you could use teamed nic and set your interface as vnic of that team.10:07
Fl1ntvnic or vlan interface tho10:08
Fl1ntdepends on your needs10:08
cosmicsoundFl1nt , i want to be able to assing later on other FO ip ranges i got10:16
cosmicsoundand ofcourse to have working network on my magnum cluster10:16
mnasiadkamgoddard, yoctozepto, hrw: are we thinking about merging Docker healthchecks in k-a - or do we want to postpone it to V?10:17
cosmicsoundyou mean to make a bond of eno1 and eno2 and use that as a neutron external, i got it right?10:17
yoctozeptomnasiadka: we are always thinking10:18
mnasiadkayoctozepto: good that you are, I'm not always thinking :)10:18
hrwmnasiadka: fine for U10:18
mnasiadkawell, I was referring to this change: https://review.opendev.org/#/c/676389/10:19
patchbotpatch 676389 - kolla-ansible - Use Docker healthchecks for core services - 49 patch sets10:19
mnasiadkano reviews - I did a recheck today and it still works10:19
Fl1nt@cosmicsound, nope, team eno1/eno2 add required vnic/vlans on it and use those vnic/vlans as neutron_external or network_interface, that would let you keep the existing.10:19
hrwon it10:19
Fl1ntI personally assign network_interface on my team0 but that's because I've a trunk that let me tag packets from neutron.10:20
yoctozeptomnasiadka: but you did not ask yourself :-)10:20
*** vishalmanchanda has joined #openstack-kolla10:22
hrwmnasiadka: keystone_fernet_healthcheck is missing?10:23
hrwmnasiadka: you define one for keystone, for keystone_ssh but not for keyston_fernet10:23
mnasiadkahrw: well, I focused on these that were easy to check - fernet container doesn't really run anything but cron :)10:24
hrw    healthcheck: "{{ keystone_fernet_healthcheck | default({}) }}"10:24
mnasiadkayeah, I can remove it, I ran a script to edit all - just comment :)10:27
hrwmnasiadka: similar to neutron/defaults/main.yml - you use many .*_healthcheck vars which are not defined in this file10:28
hrwneutron_infoblox_ipam_agent_healthcheck for example10:28
mnasiadkaok, I'll limit the vars to those, that are really set and ask for reviews once again :)10:29
hrwthx10:30
hrwmnasiadka: I like the idea of that patch and know that it is not small so can be tough to spot everything10:30
mnasiadkayeah, and I haven't really touched it for a month or so :)10:31
*** yoctozepto has quit IRC10:37
*** yoctozepto9 is now known as yoctozepto10:37
cosmicsoundFl1nt , will give a try10:37
cosmicsoundthanks for suggestion10:37
*** k_mouza has joined #openstack-kolla10:38
Fl1ntsure be my guess :D10:38
hrwmnasiadka: everyone has such ones ;d10:38
*** riuzen has quit IRC10:51
*** ivve has joined #openstack-kolla10:55
hrwhttps://review.opendev.org/#/c/713795/ anyone?10:59
patchbotpatch 713795 - kolla-ansible - CI: add Debian/source upgrade job - 2 patch sets10:59
hrwpassed zuul, needs second core10:59
mgoddard+A11:01
openstackgerritMerged openstack/kayobe master: Fix Kayobe overcloud introspection data save  https://review.opendev.org/70916311:03
openstackgerritMark Goddard proposed openstack/kayobe stable/train: Fix Kayobe overcloud introspection data save  https://review.opendev.org/71383811:04
hrwcoolio11:04
*** riuzen has joined #openstack-kolla11:06
hrwmgoddard: what do you think about publishing aarch64 images as :tag-aarch64? would like to add aarch64 upgrade job but need published images for it11:06
mgoddardhrw: it's fine by me11:07
openstackgerritMerged openstack/kayobe master: CI: Redirect stderr to ansible logs  https://review.opendev.org/70765711:07
mgoddardwould need to be published for train & master11:07
hrwyes11:07
openstackgerritMark Goddard proposed openstack/kayobe stable/stein: Fix Kayobe overcloud introspection data save  https://review.opendev.org/71384111:07
hrwmgoddard: will look a bit like revert of 'centos8 as master' patch ;D11:09
openstackgerritMerged openstack/kayobe master: Junos switch: update ncclient to 0.6.7+  https://review.opendev.org/70779611:12
Fl1ntquestion, any reason why we keep using ini inventory all over kolla/kayobe and pretty much any ansible playbooks out there? We use the YAML model in here, we do that as it's a more portable format for any other side tools that use the inventory.11:14
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: CI: publish AArch64 Debian/source images  https://review.opendev.org/71384711:15
mgoddardhrw: how reliable is ARM now? Enough to do daily publishing?11:16
hrwmgoddard: should be11:19
hrwmgoddard: if not then I will shout at kevinz11:19
mgoddarddoes that help?11:19
hrwworks for me11:20
hrw;d11:20
mgoddardnice, will try some time :)11:20
yoctozeptoFl1nt: due to random ordering of yaml dicts11:20
* yoctozepto slaps hrw around a bit with a large trout11:20
yoctozepto^ this also works11:20
hrwyoctozepto: I do wonder where you found that trout. Auchan does not sell any fish this morning.11:22
openstackgerritMarcin Juszkiewicz proposed openstack/kolla stable/train: CI: publish AArch64 Debian/source images  https://review.opendev.org/71385111:22
hrwpatch was not cherrypickable anyway11:23
*** klindgren has quit IRC11:23
openstackgerritMark Goddard proposed openstack/kayobe master: CentOS 8: Support DNF  https://review.opendev.org/71104411:23
openstackgerritMark Goddard proposed openstack/kayobe master: CI: Add overcloud host configure job  https://review.opendev.org/71027711:23
*** klindgren has joined #openstack-kolla11:24
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: CI: Add instance console log to test-core-openstack.sh  https://review.opendev.org/71385311:25
*** riuzen has quit IRC11:26
openstackgerritMerged openstack/kolla master: Revert "mark bifrost as unbuildable"  https://review.opendev.org/71350611:27
openstackgerritMerged openstack/kolla-ansible master: CI: add Debian/source upgrade job  https://review.opendev.org/71379511:27
*** rpittau is now known as rpittau|bbl11:30
openstackgerritMarcin Juszkiewicz proposed openstack/kolla-ansible master: CI: add Debian source upgrade job on AArch64  https://review.opendev.org/71385411:42
hrwmgoddard: running local builds to collect build times for https://review.opendev.org/71367011:46
patchbotpatch 713670 - kolla - CI: do not clean package cache if not publishing - 1 patch set11:46
hrwmgoddard: so far from experience most gaining one is centos811:48
openstackgerritMerged openstack/kayobe stable/stein: Skip resmo.ntp role if ntp_service_enabled is false  https://review.opendev.org/71166711:59
openstackgerritMerged openstack/kayobe stable/rocky: Skip resmo.ntp role if ntp_service_enabled is false  https://review.opendev.org/71166811:59
openstackgerritMerged openstack/kayobe stable/train: Skip resmo.ntp role if ntp_service_enabled is false  https://review.opendev.org/71166612:02
openstackgerritMerged openstack/kayobe stable/train: Stop gzipping logs in get-logs.sh  https://review.opendev.org/71244212:03
openstackgerritMerged openstack/kayobe stable/rocky: Stop gzipping logs in get-logs.sh  https://review.opendev.org/71244412:03
openstackgerritMerged openstack/kayobe stable/stein: Stop gzipping logs in get-logs.sh  https://review.opendev.org/71244312:06
openstackgerritMerged openstack/kayobe stable/train: CI: Make provision-net external  https://review.opendev.org/70891512:06
*** k_mouza has quit IRC12:06
*** k_mouza has joined #openstack-kolla12:07
Fl1nt@hrw, wait, you're french?12:20
hrwFl1nt: non, d'où vient cette idée?12:22
hrw(powered by google translate as I ignore existance of French as much as possible)12:22
Fl1ntAuchan :D12:23
*** cah_link1 has joined #openstack-kolla12:24
*** cah_link has quit IRC12:26
*** cah_link1 is now known as cah_link12:26
hrwFl1nt: in 500 meters area I have: Auchan (.fr), Lidl (.de), Biedronka (.pt) and some small groceries12:26
Fl1nthum... Definitely not in France so as we don't get Biedronka12:27
Fl1ntnevermind ^^12:27
*** hjensas has quit IRC12:33
*** hjensas has joined #openstack-kolla12:35
cosmicsoundosmanlicilegi , can you share a glimpse of your netplan for this one? https://bugs.launchpad.net/kolla-ansible/+bug/1851745/comments/312:35
openstackLaunchpad bug 1851745 in kolla-ansible "Neutron_external_interface issue need help" [Undecided,Invalid]12:35
kevinzmgoddard: hrw: feel free to try image publish on arm64 :D12:38
hrw;d12:42
*** k_mouza has quit IRC12:42
openstackgerritMerged openstack/kolla-ansible stable/rocky: ceph: Add become to gathering OSD IDs on reconfigure  https://review.opendev.org/71380612:49
*** gfidente has joined #openstack-kolla12:50
*** k_mouza has joined #openstack-kolla12:52
*** sean-k-mooney2 has joined #openstack-kolla12:53
*** sean-k-mooney2 is now known as sean-k-mooney12:53
*** scottsol has joined #openstack-kolla12:53
openstackgerritMerged openstack/kolla-ansible master: CentOS 8: Use master image tag by default  https://review.opendev.org/71327212:55
openstackgerritMerged openstack/kolla-ansible stable/train: ceph: Add become to gathering OSD IDs on reconfigure  https://review.opendev.org/71369112:55
*** rpittau|bbl is now known as rpittau12:59
osmanlicilegicosmicsound: http://paste.openstack.org/show/mtCwR5SwloBZ34QJjn6o/13:04
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: OVN Support  https://review.opendev.org/69684113:07
osmanlicilegicosmicsound: btw sorry for missing your reply. I'm very busy for weeks...13:08
yoctozeptoFl1nt: "biedronka" is Polish for "ladybug"13:12
Fl1ntok, but I think he was talking about the portugese retail ^^13:14
Fl1ntIs all stackHPC in poland ?13:14
yoctozeptoFl1nt: yes, it's Portuguese, Polish trade is dominated by foreign companies13:15
yoctozeptoFl1nt: StackHPC no13:15
yoctozeptoFl1nt: neither me nor hrw are StackHPC13:16
yoctozeptoFl1nt: mgoddard and mnasiadka are StackHPC13:16
yoctozeptoFl1nt: and only me, hrw and mnasiadka Polish13:16
Fl1ntoh ok ^^ thought you were pretty much all SHPC ^^13:16
hrwFl1nt: it would be bad for business13:17
yoctozepto;o13:18
Fl1nthow that? Is there still some Oracle people in here ?13:18
mnasiadkayoctozepto: https://review.opendev.org/#/c/713853/ - care to review? :)13:20
patchbotpatch 713853 - kolla-ansible - CI: Add instance console log to test-core-openstac... - 1 patch set13:20
yoctozeptodespite Greek origins of project's name, we insulted old gods and they turned away from us13:20
yoctozeptoFl1nt: ^13:21
yoctozeptomnasiadka: very well, I had that in todos13:22
Fl1ntBecause you didn't sacrified enough ^^ They felt screwed, then look at what you did!13:22
yoctozeptowonder if i did not make a bug about it13:22
yoctozeptonope, none seen13:22
yoctozeptomnasiadka: commented13:23
mnasiadkayoctozepto: commented back :D13:27
openstackgerritGrzegorz Sikorski proposed openstack/kolla-ansible stable/ocata: rabbitmq: update cluster version on reconfigure  https://review.opendev.org/71387813:33
openstackgerritGrzegorz Sikorski proposed openstack/kolla-ansible stable/ocata: rabbitmq: update cluster version on reconfigure  https://review.opendev.org/71387813:36
yoctozeptomnasiadka: ++++13:40
yoctozeptomnasiadka: https://etherpad.openstack.org/p/kolla-ci-errors line 7313:43
*** abdysn has quit IRC13:43
mnasiadkayoctozepto: yay!13:43
mnasiadkayoctozepto: that connectivity failure is only on Ubuntu? interesting13:48
yoctozeptomnasiadka: yeah, another example now13:49
yoctozeptomnasiadka: fluentd + yours mariadb ci mod as effect failed on that thing13:49
yoctozeptomnasiadka: so repeatable13:49
openstackgerritMerged openstack/kayobe master: Make docker registry volume configurable  https://review.opendev.org/70860713:50
yoctozeptomnasiadka: well, now I had more tools (timestamping and debug in init-runonce)13:53
yoctozeptomnasiadka: so we know it is from linking router with tenant subnet13:53
yoctozeptoit must have resulted in a bad link that even libvirt felt13:54
yoctozeptowhatever that means13:54
mnasiadkayoctozepto: it badly resembles me this bug: https://bugs.launchpad.net/neutron/+bug/1865891 :)13:54
openstackLaunchpad bug 1865891 in neutron "Race condition during removal of subnet from the router and removal of subnet" [Medium,In progress] - Assigned to Rodolfo Alonso (rodolfo-alonso-hernandez)13:54
yoctozeptomnasiadka: but we are doing no delet13:55
*** k_mouza has quit IRC13:55
yoctozeptomnasiadka: you have found that real fast13:55
cosmicsoundthanks osmanlicilegi!13:55
yoctozeptomnasiadka: share your tricks :-)13:55
*** k_mouza has joined #openstack-kolla13:57
yoctozeptoall logs fine here13:57
yoctozeptowe'll see the console log some time and then we know if instance got address13:58
yoctozeptoif it did, then l3 to blame and we are on good path to report13:58
mnasiadkayoctozepto: no tricks, just being on proper channels ;)14:00
*** jimcrowleyibm has joined #openstack-kolla14:04
*** jimcrowleyibm is now known as jimcrowleyibm[aw14:04
*** jimcrowleyibm has joined #openstack-kolla14:05
yoctozeptomnasiadka: sure :-)14:07
yoctozeptohrw: commented all aarch64 patches14:07
mnasiadkahmm, those cinder failures in upgrade jobs are interesting14:08
openstackgerritMerged openstack/kolla-ansible stable/stein: ceph: Add become to gathering OSD IDs on reconfigure  https://review.opendev.org/71380514:08
mnasiadkayoctozepto: https://zuul.opendev.org/t/openstack/build/f15cc15edc7b481595aa49d00aee9128/log/primary/logs/ansible/upgrade#6787114:09
yoctozeptomnasiadka: "some"14:09
yoctozeptomnasiadka: where them logs at? :D14:11
mnasiadkayoctozepto: https://zuul.opendev.org/t/openstack/build/f15cc15edc7b481595aa49d00aee9128/log/primary/logs/kolla/cinder/cinder-api.txt14:13
mnasiadkayoctozepto: it doesn't look good14:13
mnasiadka:)14:13
yoctozeptomnasiadka: well, that explains the precision of the error message14:14
mnasiadkayeah14:14
mnasiadkaon other nodes cinder-api log is empty14:14
mnasiadkahow the hell it's possible that it works at all14:14
mnasiadkacinder-volume also looks interesting14:15
mnasiadkahttps://zuul.opendev.org/t/openstack/build/f15cc15edc7b481595aa49d00aee9128/log/primary/logs/kolla/cinder/cinder-volume.txt#109814:15
yoctozeptomnasiadka: well, cinder-api is for internal stuff, cinder-wsgi is the "api"14:16
yoctozeptodafu that volume14:17
yoctozeptosomething went wrong14:18
yoctozeptomnasiadka: is ubuntu the same?14:18
mnasiadkayoctozepto: yup14:19
openstackgerritMark Goddard proposed openstack/kolla stable/train: CentOS 8: Build bifrost image  https://review.opendev.org/71350914:20
openstackgerritMark Goddard proposed openstack/kolla stable/train: CentOS 8: Build bifrost image  https://review.opendev.org/71350914:21
*** diurnalist has joined #openstack-kolla14:23
mnasiadkayoctozepto: why there's no cinder-wsgi logs on secondary1/2? :)14:23
mnasiadka*there are14:23
yoctozeptomnasiadka: sh... here we go again ;D14:24
mnasiadkaThere is NO need to do data online migration since Rocky, Stein and Train.14:24
mnasiadkainteresting14:24
yoctozeptohmm14:24
mnasiadkahttps://bugs.launchpad.net/cinder/+bug/1806156/comments/5014:24
openstackLaunchpad bug 1806156 in Cinder "shared_targets_online_data_migration fails when cinder-volume service not running" [Undecided,Confirmed]14:24
yoctozeptoah, so maybe we hit a race condition14:24
yoctozeptomnasiadka: please propose to drop14:25
yoctozeptomnasiadka: and create our bug for that14:25
yoctozeptomnasiadka: makes sense14:25
mnasiadkahttps://review.opendev.org/#/c/672466/14:25
patchbotpatch 672466 - cinder - Fix online data migrations (MERGED) - 3 patch sets14:25
mnasiadkaseems they fixed it in master14:25
yoctozeptomnasiadka: well, how did not we see this earlier anyway?14:25
yoctozeptomnasiadka: anyways, this reminds of this nova race we had with discovery14:26
mnasiadkayoctozepto: god knows, maybe we try to ignore some errors :D14:26
yoctozeptomnasiadka: ended up disabling autodiscovery ;-)14:26
yoctozeptomnasiadka: the issue remains that only one api seems ever active14:26
yoctozeptomnasiadka: maybe haproxy is configured to not do roundrobin with cinder?14:26
mnasiadka#cinder-needs-some-love14:27
mnasiadkauhh14:27
*** dking_desktop has quit IRC14:28
yoctozeptomnasiadka: nah, only horizon14:29
yoctozeptomnasiadka: then it looks like logs are missing14:29
*** TrevorV has joined #openstack-kolla14:30
*** diurnalist has quit IRC14:32
*** diurnalist has joined #openstack-kolla14:41
mnasiadkayoctozepto: then I decide to call it a day, and a have a day off tomorrow14:46
mnasiadkalet me know if cinder decides to fix itself :D14:46
openstackgerritMerged openstack/kolla-ansible master: CI: Add instance console log to test-core-openstack.sh  https://review.opendev.org/71385314:50
yoctozeptomnasiadka: not joining today's call?14:52
mnasiadkayoctozepto: I'll try, but not promise14:52
openstackgerritMerged openstack/kayobe stable/train: Fix Kayobe overcloud introspection data save  https://review.opendev.org/71383814:53
yoctozeptohrw: joining?15:04
mgoddardhttps://meet.google.com/tfm-aiey-tff15:04
*** ricolin has quit IRC15:08
*** skramaja has quit IRC15:08
hrwwill do15:11
*** ricolin has joined #openstack-kolla15:13
mnasiadkahrw: slyszymy Cie, ale Ty nas chyba nie ;)15:14
hrwexactly15:16
hrwwelcome in XXI century of video conferencing software15:16
hrwhave fun, I skip15:16
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: CI: publish AArch64 Debian/source images  https://review.opendev.org/71384715:28
hrwedited commit message15:28
*** Trevor_V has joined #openstack-kolla15:37
*** TrevorV has quit IRC15:40
*** vishalmanchanda has quit IRC15:43
*** cah_link1 has joined #openstack-kolla15:44
*** cah_link has quit IRC15:44
*** cah_link1 is now known as cah_link15:44
yoctozeptowhat a productive meeting, we must repeat that from time to time15:46
mgoddard+115:46
openstackgerritMark Goddard proposed openstack/kolla stable/train: CentOS 8: Build bifrost image  https://review.opendev.org/71350915:48
yoctozeptocould use a different agenda though, tls might get boring15:49
hrwand zoom ;)15:49
hrwor bluejeans15:50
yoctozeptohrw: well, at least I learnt what was so funny about our introduction in last ptg15:51
hrw;)15:54
*** cah_link1 has joined #openstack-kolla15:54
hrwI have to buy a new mic15:54
*** cah_link has quit IRC15:55
*** cah_link1 is now known as cah_link15:55
*** dking_desktop has joined #openstack-kolla15:57
Fl1nt@hrw, because you dropped it too hard last time? xD15:58
yoctozeptoFl1nt: no, that was bass15:59
hrwFl1nt: it is Logitech Webcam Pro 9000. the only Pro in it is name.16:00
Fl1nthuhuhu16:01
* Fl1nt going back to the salty mine ^^16:01
*** cah_link has quit IRC16:01
*** cah_link has joined #openstack-kolla16:02
hrwFl1nt: typical VGA camera with overboosted specs on packaging16:02
Fl1ntyeah, it's been a while since I don't get any ^^ Just used to my laptop one :D16:03
*** k_mouza has quit IRC16:07
*** k_mouza has joined #openstack-kolla16:08
hrwFl1nt: I have two laptops on my desk, 3 (broken) under and use my desktop ;D16:10
*** cah_link has quit IRC16:12
*** cah_link1 has joined #openstack-kolla16:13
openstackgerritMarcin Juszkiewicz proposed openstack/kolla-ansible master: CI: add Debian source upgrade job on AArch64  https://review.opendev.org/71385416:13
hrwyoctozepto: thx16:13
openstackgerritMarcin Juszkiewicz proposed openstack/kolla stable/train: CI: publish AArch64 Debian/source images  https://review.opendev.org/71385116:14
hrwyoctozepto: and again ;D16:14
yoctozeptohrw: k-a recommented :-)16:15
yoctozeptok fine16:15
*** cah_link1 has quit IRC16:17
hrwnoticed16:17
*** k_mouza has quit IRC16:18
openstackgerritMarcin Juszkiewicz proposed openstack/kolla-ansible master: CI: add Debian source upgrade job on AArch64  https://review.opendev.org/71385416:19
hrwdone16:19
*** k_mouza has joined #openstack-kolla16:19
yoctozeptook16:21
*** JDoed has joined #openstack-kolla16:27
mnasiadkayoctozepto: got time to look into: https://zuul.opendev.org/t/openstack/build/2f8b2a4692d74962878fc11135c6e6cb/log/primary/logs/ansible/test-core-openstack ?16:33
mnasiadkayoctozepto: seems like cirros image has dns resolution enabled in sshd?16:34
*** bengates has quit IRC16:34
yoctozeptomnasiadka: hmm, more like mtu issue?16:36
mnasiadkamight also be16:36
yoctozeptodebug1: expecting SSH2_MSG_KEX_ECDH_REPLY16:36
yoctozeptomnasiadka: could apply gawk to that to have timestamps16:37
yoctozeptomnasiadka: and do ssh -vv16:37
mnasiadkalooking at init-core - networks mtu is correct - 1450 on external and 1400 on internal geneve16:38
mnasiadkaor not16:39
mnasiadkageneve with ipv4 is like 58 bytes iirc16:39
Fl1ntAnyone using geneve? Curious to know about pitfalls and why did you choosed it over vxlan.16:40
yoctozeptomnasiadka: who computed that mtu?16:41
mnasiadkayoctozepto: I guess neutron, we don't supply the value16:41
yoctozeptomnasiadka: thought so, might be they need a bug report :-)16:41
mnasiadkaFl1nt: OVN chose, we did not :)16:41
Fl1ntoh ok, but isn't OVN supposed to be deprecated?16:42
Fl1ntlike only used by RHOSP ?16:42
yoctozeptoFl1nt: lol, no, it's the future16:43
mnasiadkaFl1nt: no, OVN is one of three in-tree drivers in Neutron now16:43
mnasiadkaFl1nt: ml2/ovs will likely be deprecated in longer future16:43
mnasiadkayoctozepto: well, we set global_physnet_mtu properly, so wtf16:43
Fl1ntoh ok, notice that, I like ovs/ml2 it's flexible and works fine ^^16:43
* hrw off16:45
mnasiadkayoctozepto: naah, seems 50 is ok - it's the default16:45
yoctozeptomnasiadka: 50 is when no options are used16:46
mnasiadkayoctozepto: it could be even 3816:46
mnasiadkaseems devstack ovn uses 38 and it works16:46
mnasiadkaso it's rather something else16:46
yoctozeptomnasiadka: ok16:46
mnasiadkayoctozepto: https://github.com/openstack/networking-ovn/blob/eda5d7f80d877601170631c5f5485370ea701f42/devstack/lib/networking-ovn#L7116:47
yoctozeptomnasiadka: Defaults to 38 bytes plus the IP version16:48
yoctozepto# overhead (20 bytes for IPv4 (default) or 40 bytes for IPv616:48
yoctozeptoso it's 58 for ipv416:48
mnasiadkalet me set it in ml2_conf, and let's see if it helps16:49
yoctozeptomnasiadka: https://opendev.org/openstack/neutron-lib/src/branch/master/neutron_lib/constants.py#L325 are they fscking kidding me16:49
yoctozeptomnasiadka: it will16:49
yoctozeptolemme bugreport dear neutron16:50
*** k_mouza has quit IRC16:51
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: OVN Support  https://review.opendev.org/69684116:52
mnasiadkaok then, let it roll16:52
*** k_mouza has joined #openstack-kolla16:52
mnasiadkayoctozepto: at least now we're testing networking properly, so we can catch this :)16:53
yoctozeptomnasiadka: :-)16:53
mnasiadkaok, enough for today16:54
mnasiadkaI'll be off tomorrow, my time to take care of daughter16:54
yoctozeptomnasiadka: https://bugs.launchpad.net/neutron/+bug/186813716:56
openstackLaunchpad bug 1868137 in neutron "default geneve overhead is set not enough" [Undecided,New]16:56
Fl1nthum... OVN is really interesting, I especially like the transparent DVR/HA and OVSDB vs RMQ features (for neutron related things)16:56
yoctozeptomnasiadka: just do the launchpad dance and you are free16:56
yoctozeptomnasiadka: have fun with daughter, family is very important16:56
mnasiadkadanced with launchpad16:56
mnasiadkayoctozepto: have a good weekend :)16:56
Fl1nt@mnasiadka, keep it safe buddy16:57
yoctozeptomnasiadka: and that it affects you! :P16:57
yoctozeptomnasiadka: thanks, same to you16:57
*** JDoed has left #openstack-kolla16:57
Fl1ntHow is the OVN support with Train ?16:58
mnasiadkayoctozepto: done!16:58
mnasiadkaFl1nt: in kolla-ansible? Nonexistent16:58
Fl1ntyep, was my question ^^16:59
yoctozeptomnasiadka: TYVM!16:59
Fl1ntok, is it planned for U or V release?16:59
yoctozeptoFl1nt: Ussuri16:59
Fl1ntok, cool, thx, will definitely give it a try.16:59
Fl1ntthe distributed nature of it being the most interesting part on my use-case ^^17:00
yoctozeptoFl1nt: yup, it looks really promising17:04
yoctozeptoFl1nt: it takes a lot of work off neutron17:04
mgoddardtime for a quick review mnasiadka? https://review.opendev.org/#/c/702706/17:05
patchbotpatch 702706 - kolla - CentOS 8: Enable hacluster and masakari images - 5 patch sets17:05
Fl1ntyep, and that will for sure improve neutron stability for the best, but it add up architectural complexity that some will for sure hate ^^17:06
openstackgerritMerged openstack/kolla-ansible master: Fix deployment of fluentd without any enabled OpenStack services  https://review.opendev.org/71370417:07
mnasiadkamgoddard: done, looks good17:07
mgoddardthanks17:07
yoctozeptoFl1nt: I wouldn't be so sure about that stability :D17:08
openstackgerritMerged openstack/kolla-ansible master: CI: Enable fluentd in MariaDB jobs  https://review.opendev.org/71366317:19
*** scottsol has quit IRC17:27
*** pbing19 has joined #openstack-kolla17:30
*** evrardjp has quit IRC17:36
*** evrardjp has joined #openstack-kolla17:36
*** k_mouza has quit IRC17:43
*** k_mouza has joined #openstack-kolla17:50
*** hrw has quit IRC17:52
openstackgerritMark Goddard proposed openstack/kolla stable/rocky: Bump glance for Rocky  https://review.opendev.org/71395217:54
yoctozeptomgoddard: what about hrw's aarch64 publishers?17:58
*** ktibi has quit IRC17:58
yoctozeptomgoddard: if you see nothing bad about them, then maybe let's merge before tomorrow17:58
mgoddardyoctozepto: if they're reliable then I'm happy17:58
yoctozeptomgoddard: we will only know tomorrow17:58
yoctozeptomgoddard: exactly why asking17:58
yoctozeptomgoddard: in the worst case we skip one day of images...17:59
mgoddardapproved17:59
*** dougsz has quit IRC18:02
yoctozeptomgoddard: and train?18:03
mgoddardI guess18:04
*** hrw has joined #openstack-kolla18:04
yoctozeptomgoddard: I mean https://review.opendev.org/71385118:05
patchbotpatch 713851 - kolla (stable/train) - CI: publish AArch64 Debian/source images - 2 patch sets18:05
*** rpittau is now known as rpittau|afk18:06
openstackgerritMark Goddard proposed openstack/kolla stable/train: CI: publish AArch64 Debian/source images  https://review.opendev.org/71385118:07
hrwmgoddard: thanks18:22
mgoddardhrw: np18:22
* hrw off18:23
yoctozepto+A18:25
*** k_mouza has quit IRC18:29
*** k_mouza has joined #openstack-kolla18:32
openstackgerritMerged openstack/kayobe master: CentOS 8: Support DNF  https://review.opendev.org/71104418:34
yoctozeptomnasiadka: ovn green for c8 woop!18:53
openstackgerritMerged openstack/kolla master: CentOS 8: Enable hacluster and masakari images  https://review.opendev.org/70270618:55
openstackgerritMerged openstack/kolla-ansible master: CI: CentOS 8: Enable Masakari job and periodics  https://review.opendev.org/71289718:55
mnasiadkayoctozepto: oh my god it works!19:00
*** pbing19 has quit IRC19:02
*** pbing19 has joined #openstack-kolla19:03
*** k_mouza has quit IRC19:07
openstackgerritMerged openstack/kolla master: CI: publish AArch64 Debian/source images  https://review.opendev.org/71384719:29
r3ap3rI have a quick question, does kolla-ansible setup ovmf for uefi based guests?19:35
r3ap3rto support I mean ^19:35
*** Trevor_V has quit IRC19:41
openstackgerritPedro Henrique Pereira Martins proposed openstack/kolla-ansible master: Add suport to OpenID Connect Authentication flow  https://review.opendev.org/69543219:48
*** TrevorV has joined #openstack-kolla20:12
openstackgerritMerged openstack/kayobe stable/stein: Fix Kayobe overcloud introspection data save  https://review.opendev.org/71384120:16
*** TrevorV has quit IRC20:18
*** gfidente is now known as gfidente|afk20:33
*** jimcrowleyibm is now known as jimcrowleyibm[aw20:51
*** ivve has quit IRC21:08
*** iniazi has joined #openstack-kolla21:24
openstackgerritJames Kirsch proposed openstack/kolla-ansible master: Add support for encrypting backend HAProxy traffic to Keystone service  https://review.opendev.org/71398621:25
*** iniazi_ has joined #openstack-kolla21:29
*** iniazi has quit IRC21:30
openstackgerritMerged openstack/kolla stable/train: CI: publish AArch64 Debian/source images  https://review.opendev.org/71385121:38
*** sorin-mihai has quit IRC21:47
*** ktibi has joined #openstack-kolla21:54
*** gfidente|afk has quit IRC21:58
*** ktibi has quit IRC21:58
*** k_mouza has joined #openstack-kolla22:09
*** k_mouza has quit IRC22:14
*** Fl1nt has quit IRC22:25
*** jbadiapa has quit IRC22:50
*** diurnalist has quit IRC23:12
*** k_mouza has joined #openstack-kolla23:25
*** k_mouza has quit IRC23:29
*** iniazi_ is now known as iniazi23:30
*** vedup19 has joined #openstack-kolla23:57
« Wednesday, 2020-03-18 Index Friday, 2020-03-20 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!