Friday, 2020-02-21

openstackgerritJason Anderson proposed openstack/kolla-ansible master: [haproxy] Support interfaces with '-' chars  https://review.opendev.org/66332500:09
*** vishwanathj has joined #openstack-kolla00:10
*** vishwanathj has quit IRC00:14
*** zhanglong has joined #openstack-kolla00:27
*** AjiArya has joined #openstack-kolla00:51
*** igordc has quit IRC00:58
*** portdirect has quit IRC01:12
*** stackedsax has quit IRC01:13
*** stackedsax has joined #openstack-kolla01:13
*** portdirect has joined #openstack-kolla01:15
*** AjiArya has quit IRC01:36
*** mcnugit has joined #openstack-kolla02:09
*** vishwanathj has joined #openstack-kolla02:10
*** vishwanathj has quit IRC02:15
*** factor has joined #openstack-kolla02:25
*** AjiArya has joined #openstack-kolla02:34
AjiAryaHello guys02:39
AjiAryaWhy is this error always happen on the first try of run kolla-ansible deploy http://paste.openstack.org/show/789825/02:39
AjiAryaIt'll be ok on the second try deploy02:40
AjiAryaI'm using CentOS 7 source with everything on latest version02:41
*** jcmdln has quit IRC03:14
*** zhanglong has quit IRC04:27
*** zhanglong has joined #openstack-kolla04:29
*** vishwanathj has joined #openstack-kolla04:48
*** vishwanathj has quit IRC04:53
*** zhanglong has quit IRC05:21
*** mcnugit has quit IRC05:22
*** evrardjp has quit IRC05:34
*** evrardjp has joined #openstack-kolla05:34
emccormickFYI mgoddard When building Stein containers I'm getting ceph-common-14.2.1-0.el7.x86_6405:41
emccormickThat's Nautilus rather than the expected Luminous, and a really old release as well05:42
emccormickHey AjiArya Did you solve your problem from last night?05:42
emccormickShould be able to set your virt type to qemu for that one05:43
emccormickAjiArya Also, I don't have an answer for the other question. Might be that it's taking the ceph-mgr prometheus module too long to initialize05:45
emccormickdoesn't seem critical anyway since the second run passes.05:46
*** AjiArya has quit IRC06:07
osmanlicilegiyoctozepto: "cilek" means strawberry in Turkish and "osmanli cilegi" is a kind of strawberry which is grown only at my hometown and I like it more than honey :] https://tourismguidance.beun.edu.tr/ottoman-strawberry.html you folks, if you like jam and wanna try it, just let me know. one day I will retire myself from IT and be a farmer for ottoman strawberry :]06:09
osmanlicilegiAjiArya: not clear what's happening. you should try kolla-ansible deploy -vvvv to get more details about the error.06:12
*** ricolin has quit IRC06:15
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: OVN Support  https://review.opendev.org/69684106:22
*** AjiArya has joined #openstack-kolla06:26
*** numans has joined #openstack-kolla06:29
*** vishwanathj has joined #openstack-kolla06:49
AjiAryaemccormick: I'm still rebuilding my machine system, I'll inform you as soon as possible06:52
AjiAryaemccormick: Alright I'll ignore that ceph mgr failure06:52
*** vishwanathj has quit IRC06:53
*** cah_link has joined #openstack-kolla07:05
openstackgerritMerged openstack/kolla-ansible master: CI: Add linuxbridge jobs  https://review.opendev.org/70831807:09
openstackgerritMerged openstack/kolla-ansible master: CI: Fine tune Galera gmcast.peer_timeout to 15 seconds  https://review.opendev.org/70781807:09
*** ricolin has joined #openstack-kolla07:21
mnasiadkamorning07:35
mnasiadkayoctozepto: I think linaro has some good network problems - https://zuul.opendev.org/t/openstack/build/0803ba5e74fa4580b8854103a68a166a/log/primary/logs/build/000_FAILED_openstack-base.log#586 (aarch64 pipeline)07:35
yoctozeptomnasiadka: maybe because of ping07:36
yoctozeptomorning folks07:36
yoctozeptoosmanlicilegi: ah, thanks for explanations; in the meantime you might enjoy playing stardew valley ;D07:36
*** ivve has quit IRC07:39
*** zhanglong has joined #openstack-kolla07:46
mnasiadkayoctozepto: I think you need to rebase this https://review.opendev.org/#/c/708497/07:46
patchbotpatch 708497 - kolla-ansible - CI: Replace dummy interface (fake for neutron exte... - 2 patch sets07:46
yoctozeptomnasiadka: k, later07:47
yoctozeptoAjiArya: it's probably due to a bug, note ceph deployment in k-a is deprecated now07:47
yoctozeptoAjiArya: probably mgr was not yet up07:48
yoctozeptodking_desktop: right indeed07:48
yoctozeptoemccormick: interesting, hmm, things must have changed in repos I believe07:48
yoctozeptomnasiadka: want to triage that?07:49
mnasiadkaemccormick: is it centos or ubuntu, that is getting Luminous ceph-common?07:49
*** rpittau|afk is now known as rpittau07:50
*** zhanglong has quit IRC07:50
*** zhanglong has joined #openstack-kolla07:53
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: OVN Support  https://review.opendev.org/69684107:57
*** bengates has joined #openstack-kolla07:59
yoctozeptoemccormick, mnasiadka: oh, folks, it's nautilus in stein centos https://ethercalc.openstack.org/kolla-infra-service-matrix08:08
mnasiadkaWell, Nautilus is newer than Luminous, right?08:09
mnasiadkaSo I don't know what's the issue still :)08:09
mnasiadkabut yeah, CentOS Storage SIG is not the best in updates08:11
mnasiadka14.2.1 vs 14.2.7 is a bit of a pita08:11
yoctozeptomnasiadka: so we are running PT15S now, huh08:13
yoctozeptomnasiadka: so no more timeouts? :D08:13
mnasiadkayoctozepto: time will tell08:13
*** Tony31 has joined #openstack-kolla08:14
*** vishwanathj has joined #openstack-kolla08:19
*** JangwonLee has joined #openstack-kolla08:23
*** vishwanathj has quit IRC08:24
*** JangwonLee_ has quit IRC08:24
*** AjiArya has quit IRC08:29
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: Use Docker healthchecks for core services  https://review.opendev.org/67638908:34
mnasiadkaosmanlicilegi: rebased ^^08:35
*** factor has quit IRC08:36
*** factor has joined #openstack-kolla08:36
osmanlicilegimnasiadka: thx!08:44
*** ivve has joined #openstack-kolla08:44
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: Stop using deprecated stores and default_store in glance  https://review.opendev.org/70811408:48
*** zhanglong has quit IRC08:50
yoctozeptomnasiadka: I have read glance sources for that ^08:52
mnasiadkayoctozepto: lol, good job08:53
yoctozeptomnasiadka: so it really is better now08:53
*** zhanglong has joined #openstack-kolla08:53
mnasiadkayoctozepto: I took the mission of analyzing all errors/warnings about deprecation and fixing them, glance is/was first on the list.08:53
yoctozeptomnasiadka: it's pro work, +208:54
yoctozeptomnasiadka: we learn a few things along the path08:54
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: CI: Replace dummy interface (fake for neutron external) with VXLAN  https://review.opendev.org/70849708:56
yoctozeptomnasiadka: get me that w+108:57
Tony31Good morning guys08:58
Tony31Just want to say thanks for your help this week with kayobe/kolla08:58
*** tonythomas has joined #openstack-kolla08:58
Tony31I tested deleting openstack components (like compute node or controller node) and then re-deploying. Aside from a couple of minor problems that were easily fixed, everything works great.08:59
yoctozeptoTony31: glad to hear that :-)09:00
Tony31thank you to yoctozepto and mnasiadka09:00
yoctozeptoyw09:00
Tony31👍🏻09:00
*** Tony31 has left #openstack-kolla09:01
openstackgerritMerged openstack/kolla stable/train: CentOS 8: work around mod_ssl certificate issue  https://review.opendev.org/70241909:03
*** Tony31 has joined #openstack-kolla09:08
*** AjiArya has joined #openstack-kolla09:09
Tony31When I try and launch an instance on centos 7 physical compute node, I get this error `invalid argument: could not find capabilities for domaintype=kvm` But when I try and verify this from the physical machine I see kvm supported: `# lsmod | grep -i kvm09:10
Tony3113503  1 kvm`09:10
openstackgerritMerged openstack/kolla stable/train: CentOS 8: Add zuul build and publish jobs  https://review.opendev.org/70242009:12
Tony31in the nova_compute and nova_libvirt container I get the same output :)09:12
*** dougsz has joined #openstack-kolla09:13
Tony31error pastebin from the compute host https://pastebin.com/hHrk1Q53 - is this a bug?09:13
cosmicsoundGood day09:13
*** k_mouza has joined #openstack-kolla09:15
yoctozeptoTony31: odd, they are two different things09:21
yoctozeptoTony31: that's not enough to have kvm09:21
Tony31thanks for the tip. Will keep digging :)09:21
yoctozeptoTony31: it needs to have cpu-specific module loaded09:21
yoctozeptoe.g. kvm_intel (or intel_kvm, always forget)09:21
yoctozeptosometimes checking dmesg helps09:22
yoctozeptoalso bios config09:22
cosmicsoundWhere is best to find release schedule for train?09:22
yoctozeptocosmicsound: train is released already09:23
yoctozeptocosmicsound: https://releases.openstack.org/09:23
yoctozeptoand add up to 2 months for kolla09:23
*** Tony3151 has joined #openstack-kolla09:23
yoctozeptoso we really released in December afair09:24
Tony3151got disconnected09:24
yoctozeptoTony3151:09:24
yoctozepto10:21:05 <yoctozepto> Tony31: odd, they are two different things09:24
yoctozepto10:21:30 <yoctozepto> Tony31: that's not enough to have kvm09:24
yoctozepto10:21:30 <Tony31> thanks for the tip. Will keep digging :)09:24
yoctozepto10:21:38 <yoctozepto> Tony31: it needs to have cpu-specific module loaded09:24
yoctozepto10:21:53 <yoctozepto> e.g. kvm_intel (or intel_kvm, always forget)09:24
yoctozepto10:22:11 <yoctozepto> sometimes checking dmesg helps09:24
yoctozepto10:22:19 <yoctozepto> also bios config09:24
yoctozeptoand you need to work on that network connection of yours09:24
Tony3151connection is great I have 100M fibre. It's Australia's link to the rest of the world that's the problem :)09:25
*** Tony31 has quit IRC09:27
cosmicsoundTony3151 , I would recommend you look into thelounge.09:27
cosmicsoundIf you want I can provide you a connection from Europe.09:27
cosmicsoundTo be stable :)09:27
cosmicsoundyoctozepto , i was thinking there are more releases on same version09:28
cosmicsoundsuch as 9.0 9.0.1 9.0.209:28
cosmicsoundYet thats kolla versioning, not Train09:28
AjiAryaosmanlicilegi: this is the verbose result http://paste.openstack.org/show/789840/09:29
yoctozeptocosmicsound: https://releases.openstack.org/teams/kolla.html#team-train-kolla and https://docs.openstack.org/releasenotes/kolla/train.html09:29
yoctozeptoAjiArya: as I said, it's probably due to ceph mgr not being ready yet09:30
AjiAryaJust send what osmanlicilegi asked09:32
yoctozeptoAjiArya: just repeating because I did not know whether you read that :-)09:33
AjiAryaSorry09:33
AjiAryaI've read those09:33
mgoddardmorning09:34
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: CI: Remove useless pre task  https://review.opendev.org/70907109:38
Tony3151mgoddard morning :)09:59
Tony3151yoctozepto - I have fixed the issue now but I am not sure about it. The fix, apparently was to enable `intel_iommu` in the cmdline and boot... Here's a virt-host-validate from before I done the reboot. Although, to install virt-host-validate I had to install `libvirt-client` and its dependencies which could have fixed the problem. To install the10:01
Tony3151host, I used the iso for centos 7 and done the basic minimal install. I had a similar problem to this when I used the cloud-image and used the kayobe all in one. But the diffrerence there was nested virt. Here there are no nests :)10:01
Tony3151eventually found the tab: https://pastebin.com/p54tvgSb10:02
yoctozeptoTony3151: odd, iommu should not be required10:02
Tony3151I think i was missing some libvirt package...10:02
yoctozeptoTony3151: yeah, it's just a warn10:02
yoctozeptoTony3151: you should not have been missing any10:02
yoctozeptoTony3151: the libvirt and qemu are containerized too10:02
Tony3151Odd one...10:03
yoctozeptoTony3151: was it centos 7.710:03
yoctozepto?10:03
Tony3151hmm I wonder if all I needed was a reboot10:03
Tony3151Indeed centos 7 and updated before running host configure / deploy10:03
yoctozeptoTony3151: I'll take a note to spin up such machine and see for myself10:04
yoctozeptoTony3151: so you did minimal install10:05
yoctozeptoTony3151: from iso10:05
yoctozeptoTony3151: then updated10:05
yoctozeptoTony3151: then no reboot?10:05
Tony3151yes, the iso is the 1903 version. Then once I was in, I removed wpa_supplicant, then ran `yum update -y` and disabled selinux and rebooted (so the host configure did not need to wait for host reboot). Then I done a deploy10:06
Tony3151But, actually I dont think the host got rebooted after host configure/deploy - this could be the reason. I found before that libvirt needs a reboot (I couldnt find the service)10:06
mnasiadkamgoddard, yoctozepto: https://review.opendev.org/#/c/696841/96 - addressed I think all comments and it still works...10:09
patchbotpatch 696841 - kolla-ansible - OVN Support - 96 patch sets10:09
*** zhanglong has quit IRC10:10
Tony3151yoctozepto - thanks again for the help. Have a good weekend all10:13
*** Tony3151 has left #openstack-kolla10:13
mnasiadkamgoddard: another thing - are you fine with that change? https://review.opendev.org/#/c/707800/2 - if yes, I'll populate it in a similar way to other Kolla family repos10:19
patchbotpatch 707800 - kolla - [community goal]: Add contributor and PTL guide - 2 patch sets10:19
*** vishwanathj has joined #openstack-kolla10:19
openstackgerritMark Goddard proposed openstack/kolla-ansible master: WIP: CI: Add monasca scenario  https://review.opendev.org/64989310:23
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Fix invalid JSON body in Elasticsearch API requests  https://review.opendev.org/70909010:23
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Fix kibana deployment with openstack_cacert unset  https://review.opendev.org/70909110:23
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Fix monasca deployment due to monasca_log_dir  https://review.opendev.org/70909210:23
*** vishwanathj has quit IRC10:24
mgoddardmnasiadka: will take a look10:26
*** jbadiapa has joined #openstack-kolla10:31
openstackgerritMichal Nasiadka proposed openstack/kolla master: WIP: Fix Debian ping issue with Neutron  https://review.opendev.org/70910210:58
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Check for supported OS distributions in prechecks  https://review.opendev.org/69944311:06
*** happyhemant has joined #openstack-kolla11:08
openstackgerritMark Goddard proposed openstack/kayobe master: CI: Test SSH connectivity to deployed instances  https://review.opendev.org/70760411:19
openstackgerritMark Goddard proposed openstack/kolla-ansible master: WIP: CI: Add monasca scenario  https://review.opendev.org/64989311:22
yoctozeptomnasiadka: not issue with ping? care to elaborate? :D11:32
mnasiadkayoctozepto: it's not ping's fault, that neutron is running "ping sysctl -w some_var=1"11:33
mnasiadka:D11:33
yoctozeptomnasiadka: lol, that's the command? gosh11:33
yoctozeptotoo new python issue?11:33
mnasiadkahave no clue, a new candidate for bug at neutron side11:34
yoctozeptomnasiadka: go ahead, full support from me11:35
*** vishwanathj has joined #openstack-kolla11:40
*** rpittau is now known as rpittau|bbl11:43
*** AjiArya has quit IRC11:44
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible stable/stein: Change /run bind mount for neutron/openvswitch  https://review.opendev.org/70911511:46
*** zhanglong has joined #openstack-kolla11:49
*** dougsz has quit IRC12:07
openstackgerritMark Goddard proposed openstack/kolla-ansible master: CI: sync inventory with multinode  https://review.opendev.org/70912112:12
*** JangwonLee has quit IRC12:13
*** JangwonLee has joined #openstack-kolla12:14
*** ricolin has quit IRC12:14
yoctozeptomgoddard: some questions from yesterday for you:12:16
yoctozepto19:39:52 <yoctozepto> mgoddard, mnasiadka: am I right that kayobe-config(-dev) is just config included in kayobe but cut off at release points?12:16
yoctozepto19:41:59 <yoctozepto> mgoddard, mnasiadka: and is -dev variant simply base filled with aio defaults?12:16
mgoddardsorry, missed those12:17
mgoddardthat's basically correct12:18
*** zhanglong has quit IRC12:19
*** zhanglong has joined #openstack-kolla12:21
yoctozeptomgoddard: that "basically" worries me every time12:34
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: [baremetal] Drop useless reboot task  https://review.opendev.org/70912712:43
*** zhanglong has quit IRC12:48
*** zhanglong has joined #openstack-kolla12:49
*** sluna has quit IRC12:53
*** sluna has joined #openstack-kolla12:53
*** zhanglong has quit IRC12:54
mnasiadkamgoddard, yoctozepto: maybe instead syncing inventory in tests/templates once in a while - we could make a custom part of inventory that we template out in CI and the other part would be standard aio/multinode - we would just put it in one directory for Ansible to consume?12:55
*** rpittau|bbl is now known as rpittau13:00
*** vishwanathj has quit IRC13:00
*** vishwanathj has joined #openstack-kolla13:01
*** vishwanathj has quit IRC13:06
*** Luzi has joined #openstack-kolla13:08
*** dougsz has joined #openstack-kolla13:08
*** negronjl has quit IRC13:10
*** negronjl has joined #openstack-kolla13:12
yoctozeptomnasiadka: +113:12
yoctozeptomnasiadka: better reliability13:12
mgoddardI was going to suggest we just make it a requirement to update the test inventory any time the inventory changes13:18
mgoddardit's not just a static inventory, it's a template13:19
mgoddardwe have things like ceph transition13:19
todinI am trying to get ovs-dpdk working, but somehow I can't bind the NIC device, in a non kolla deployed container it's working. Just for test, how could I add those docker args "--security-opt seccomp=unconfined --cap-add=ALL" to the ovsdpdk_vswitchd container?13:37
openstackgerritMark Goddard proposed openstack/kayobe master: CentOS 8: Use ansible_playbook_python for localhost dependencies  https://review.opendev.org/70892113:47
openstackgerritMark Goddard proposed openstack/kayobe master: Make Kayobe code compatible with Python 3  https://review.opendev.org/63558613:47
openstackgerritMark Goddard proposed openstack/kayobe master: Bump Ansible version range to 2.8 - 2.9  https://review.opendev.org/70892213:47
openstackgerritMark Goddard proposed openstack/kayobe master: WIP: CentOS 8  https://review.opendev.org/70769013:47
openstackgerritMark Goddard proposed openstack/kayobe master: CI: set previous_release to train  https://review.opendev.org/70914513:55
openstackgerritMark Goddard proposed openstack/kayobe master: CI: Test SSH connectivity to deployed instances  https://review.opendev.org/70760413:55
*** rpittau is now known as rpittau|afk13:57
*** bjolo has quit IRC14:11
emccormickyoctozepto,mnasiadk I swear I looked at that ethercalc yesterday and it said Centos Stein was Luminous. I musta had a brain cramp ;)14:12
emccormickIf one was going to add the download.ceph.com repo instead, where would be the best place to do that? Hopefully I can squash the Centos one that RDO is forcing14:13
yoctozeptomgoddard: well, we should simulate how users would use our software so mnasiadka has a point there14:13
emccormickI can't be putting 14.2.1 on there14:13
yoctozeptoemccormick: the bad thing is that it requires epel14:13
emccormickhmm14:14
emccormickwell, while I'm about adding repos it's not that hard. Are you concerned it might blow up versions of other things?14:14
*** Luzi has quit IRC14:18
*** sluna has quit IRC14:18
yoctozeptowe removed it on purpose, it does indeed tends to affect things14:23
yoctozeptobetter attack real problem14:23
yoctozeptomake centos storage sig release14:23
yoctozeptoor join their ranks and help14:23
emccormickHah14:27
emccormickThat's not going to help me like...tonight14:27
emccormickbut yes I will see if I can figure out how to ping them14:27
*** dking_desktop has left #openstack-kolla14:37
*** TrevorV has joined #openstack-kolla14:49
openstackgerritMark Goddard proposed openstack/kayobe master: Update documentation for release tasks  https://review.opendev.org/70915415:03
*** ivve has quit IRC15:12
openstackgerritMark Goddard proposed openstack/kayobe master: CentOS 8: Use ansible_playbook_python for localhost dependencies  https://review.opendev.org/70892115:39
openstackgerritMark Goddard proposed openstack/kayobe master: Make Kayobe code compatible with Python 3  https://review.opendev.org/63558615:39
openstackgerritMark Goddard proposed openstack/kayobe master: Bump Ansible version range to 2.8 - 2.9  https://review.opendev.org/70892215:39
openstackgerritMark Goddard proposed openstack/kayobe master: WIP: CentOS 8  https://review.opendev.org/70769015:39
*** lile has joined #openstack-kolla15:49
yoctozeptoemccormick: well, it's not always the case that those newer versions are considerably better than current - sometimes they even break stuff; best read releasenotes; and note openstack does not use all ceph features, especially many manageability ones so they being patched is highly irrelevant ;-)15:53
yoctozeptoemccormick: that said storage sig needs  help for sure15:53
openstackgerritDoug Szumski proposed openstack/kayobe master: Fix Kayobe overcloud introspection data save  https://review.opendev.org/70916315:54
mnasiadkaemccormick: we've been there - I mean using the download.ceph.com repo - but Storage SIG got better and updated the packages - and convinced us to go back16:05
mnasiadkaemccormick: but we could put that topic on next Kolla meeting agenda - because from my perspective the best idea would be to use upstream ceph.com packages on both Ubuntu and CentOS - so we would have the same versions on both distros.16:06
emccormickmnasladka I agree re: using upstream Ceph and having version consistency across all Kolla deployments would be nice. That Matrix shouldn't need to exist except to say which version of Ceph is in each release of Kolla16:11
*** cah_link has quit IRC16:15
yoctozeptomnasiadka: convince ceph to build required packages based on centos, w/o epel16:19
yoctozeptomnasiadka: otherwise storage sig at least provides better patching of these auxiliary things16:19
yoctozeptomnasiadka: this really is dumb in case of ceph, since it's really the same company behind both sides...16:20
yoctozeptoemccormick: ^16:20
mnasiadkayeah, the main problem behind CentOS Storage SIG is one guy doing the packaging and testing semi-manually16:21
yoctozeptomnasiadka: can't we organize this in opendev?16:22
yoctozeptomnasiadka: this really is *not* that hard16:22
mnasiadkayoctozepto: well, it's centos - we shouldn't really touch that, packaging of deps is not really openstack problem :)16:23
yoctozeptomnasiadka: well, sure; just that ceph is major component in many deploys ;-)16:23
mnasiadkaI'll speak with Giulio Fidente next week, let's see if he needs help in automating those bloody updates or something16:23
mnasiadkasecond thing is - there has been so many Nautilus releases that were broken16:24
mnasiadkathat maybe he was waiting for stabilizing :)16:24
mnasiadkahttps://ethercalc.openstack.org/kolla-ceph-versions16:24
openstackgerritMark Goddard proposed openstack/kolla-ansible master: DNM: Testing cloudkitty in master  https://review.opendev.org/68155516:24
mnasiadkaubuntu is a bit better on keeping up16:24
mnasiadkamgoddard: got a minute for https://review.opendev.org/#/c/709115/ ?16:25
patchbotpatch 709115 - kolla-ansible (stable/stein) - Change /run bind mount for neutron/openvswitch - 1 patch set16:25
mnasiadkathen it's only rocky that is left16:25
mgoddardmnasiadka: was just looking at it16:28
mgoddardand it's off16:28
yoctozeptomnasiadka: and at breaking things16:30
yoctozeptomnasiadka: so I see a correlation there16:30
*** bengates has quit IRC16:33
openstackgerritMark Goddard proposed openstack/kolla-ansible master: DNM: Testing cloudkitty in master  https://review.opendev.org/68155516:37
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Bifrost: Stop stopping RabbitMQ  https://review.opendev.org/70918916:42
openstackgerritMark Goddard proposed openstack/kayobe master: CI: set previous_release to train  https://review.opendev.org/70914516:42
openstackgerritMark Goddard proposed openstack/kayobe master: CentOS 8: Use ansible_playbook_python for localhost dependencies  https://review.opendev.org/70892117:02
openstackgerritMark Goddard proposed openstack/kayobe master: Make Kayobe code compatible with Python 3  https://review.opendev.org/63558617:02
openstackgerritMark Goddard proposed openstack/kayobe master: Bump Ansible version range to 2.8 - 2.9  https://review.opendev.org/70892217:02
openstackgerritMark Goddard proposed openstack/kayobe master: WIP: CentOS 8  https://review.opendev.org/70769017:02
openstackgerritMark Goddard proposed openstack/kayobe master: WIP: CentOS 8  https://review.opendev.org/70769017:09
openstackgerritMerged openstack/kolla-ansible master: CI: Replace dummy interface (fake for neutron external) with VXLAN  https://review.opendev.org/70849717:19
openstackgerritMerged openstack/kolla-ansible stable/stein: Change /run bind mount for neutron/openvswitch  https://review.opendev.org/70911517:27
*** evrardjp has quit IRC17:34
*** evrardjp has joined #openstack-kolla17:35
*** Trevor_V has joined #openstack-kolla17:39
*** TrevorV has quit IRC17:42
*** k_mouza has quit IRC17:43
openstackgerritMark Goddard proposed openstack/kolla-ansible stable/train: Configure Cinder to use lioadm on CentOS/RHEL 8  https://review.opendev.org/70241717:43
*** dougsz has quit IRC17:52
openstackgerritMark Goddard proposed openstack/kolla stable/train: CentOS 8: Don't force tag in build jobs  https://review.opendev.org/70245217:58
openstackgerritMark Goddard proposed openstack/kolla stable/train: CentOS 8: base image fixes for RabbitMQ, fluentd & InfluxDB  https://review.opendev.org/70920117:58
openstackgerritMark Goddard proposed openstack/kolla stable/train: CentOS 8: Update packages in images  https://review.opendev.org/70920217:58
openstackgerritMark Goddard proposed openstack/kolla stable/train: CentOS 8: Remove shellinabox from ironic-conductor  https://review.opendev.org/70920317:58
openstackgerritMark Goddard proposed openstack/kolla stable/train: CentOS 8: Use upstream Ceph/master  https://review.opendev.org/70920417:58
openstackgerritMark Goddard proposed openstack/kolla stable/train: CentOS 8: Allow SSH access to keystone_ssh and nova_ssh  https://review.opendev.org/70920517:58
openstackgerritMark Goddard proposed openstack/kolla stable/train: CentOS 8: disable repo_gpgcheck for grafana image  https://review.opendev.org/70920617:58
openstackgerritMark Goddard proposed openstack/kolla stable/train: CentOS 8: Relax ironic iPXE EFI bootloader location  https://review.opendev.org/70920717:58
mgoddardCentos 8 train go!17:58
*** k_mouza has joined #openstack-kolla18:08
*** tonythomas has quit IRC18:11
*** aleccoder has quit IRC18:12
*** klippo has quit IRC18:12
*** jovial[m] has quit IRC18:12
*** k_mouza has quit IRC18:12
*** k_mouza has joined #openstack-kolla18:15
openstackgerritPierre Riteau proposed openstack/kayobe master: Add support for configuring Cumulus switches with NCLU  https://review.opendev.org/63694118:15
*** primeministerp has joined #openstack-kolla18:16
*** k_mouza has quit IRC18:17
noxoid_^ that actually reminds me. does anyone here use the cumulus ml2 driver for any release newer than queens? it looks like the neutron plugin hasnt been updated in a long time18:27
*** openstackstatus has quit IRC18:33
*** klippo has joined #openstack-kolla19:02
*** ne0h|nihilist has joined #openstack-kolla19:03
*** igordc has joined #openstack-kolla19:10
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible stable/rocky: Change /run bind mount for neutron/openvswitch  https://review.opendev.org/70922519:15
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible stable/rocky: Change /run bind mount for neutron/openvswitch  https://review.opendev.org/70922519:15
*** aleccoder has joined #openstack-kolla19:16
*** negronjl has quit IRC19:19
*** negronjl has joined #openstack-kolla19:22
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible stable/rocky: Change /run bind mount for neutron/openvswitch  https://review.opendev.org/70759819:22
mnasiadkaugh, backporting is not always clean :)19:22
*** primeministerp has quit IRC19:26
osmanlicilegimnasiadka: dont give up :]19:29
yoctozeptonoxoid_: how does it work? (curious)19:34
noxoid_neutron hardware ml2 drivers?19:35
noxoid_they're added as a mechanism driver to ml2_conf.ini and allow neutron to make changes on the switch (usually vlan related) via an API19:38
noxoid_if you're asking about the cumulus driver specifically, i do not know lol. my company is looking into purchasing a lot of cumulus switches but their sales people seem to have exaggerated how much they support the neutron plugin19:39
noxoid_https://github.com/CumulusNetworks/networking-cumulus19:40
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: Construct service configuration urls using kolla_internal_fqdn  https://review.opendev.org/70662719:43
osmanlicileginoxoid_: we have tens of cumulus running on msn2410 and I an clearly say forget the ml2_cumulus driver. from my point of view, the question is why choose cumulus instead of onyx?19:47
noxoid_havent heard of them19:49
yoctozeptonoxoid_: yeah, I mean what and when openstack configures on these switches19:52
yoctozeptonoxoid_: so this is to wire up vlans for physical network deployments?19:52
yoctozepto(provider network)19:52
noxoid_yoctozepto, yes. its particularly helpful for ironic where it can automatically change the vlan on a bare metal servers port so it can communicate with the rest of the tenant network19:53
yoctozeptonoxoid_: I see, otherwise one has to run with all vlans trunked which is not top notch security so to speak19:54
noxoid_yep19:54
noxoid_theres also layer3 drivers floating around that allow the switches to act as VTEPs19:55
yoctozeptonoxoid_: I'm blessed with dumb switches so I'm rather into overlaying and stuff19:55
noxoid_hehe im in a similar boat. i have arista switches for openstack but havent had the time to look into how the arista drivers can help improve my existing deployment19:56
osmanlicileginoxoid_: I think the most common hardware and support for cumulus is provided by mellanox. if you are planning to go with mellanox, you should ask them about onyx.19:56
noxoid_osmanlicilegi, thanks. we're heavily leaning towards a cumulus express switch at the moment which is actually edgecore hardware19:57
*** iniazi has joined #openstack-kolla19:58
yoctozeptoegonzalez: https://bugs.launchpad.net/mistral/+bug/1864238 fyi19:59
openstackLaunchpad bug 1864238 in kolla-ansible "kolla-ansible CI - NFV jobs fail on mistral test" [Medium,Triaged]19:59
yoctozeptonoxoid_: I have worse: netgear "smart managed" switches19:59
yoctozeptowhich are actually dumb ones with lame http interface19:59
noxoid_my condolences :P19:59
yoctozeptonoxoid_: thanks20:00
noxoid_the struggle is real20:00
osmanlicileginoxoid_: I don't mean cumulus is bad. I just have doubts about whether it's mature or not. we have onyx switches without problems. we have cumulus switches with problems :)20:00
noxoid_thats fair. we definitely prefer arista but budgets are sadly a thing20:00
osmanlicilegiyoctozepto: you should think about hacking and installing ovs on your switches :P20:02
yoctozeptoosmanlicilegi: if you know of any instructions, especially those that would work on a running system and not break the interswicth link then I'm all ears20:05
yoctozeptowas not cumulus oss?20:05
yoctozeptoso theoretically you can tweak it however you want20:06
yoctozeptoand you stay in oss world20:06
yoctozeptonoxoid_, osmanlicilegi: ^20:09
*** negronjl has quit IRC20:09
noxoid_yea cumulus is based on debian20:09
yoctozeptoso why would taking care of that driver be too bad20:10
yoctozeptoas in one can even set up a pretty decent CI with it20:10
yoctozeptow/o rela hardware20:10
yoctozeptoreal*20:10
noxoid_it probably wouldnt in the big picture, but its always good to look at community involvement when choosing OSS components20:10
*** negronjl has joined #openstack-kolla20:11
yoctozeptomgoddard, mnasiadka: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_4e5/706627/4/check/build-openstack-releasenotes/4e580d2/docs/unreleased.html need to think about linting of renos, they got really ugly as we missed some links were misformatted20:14
yoctozeptonoxoid_: yeah, I'm just thinking loud because it's a bit undermining that such project does not integrate well with openstack20:15
mnasiadkayoctozepto: shouldn’t doc8 do the trick?20:15
yoctozeptomnasiadka: no idea where it gets applied, will see20:16
mnasiadkayoctozepto: in docs, but it runs only on doc/*20:16
yoctozeptomnasiadka: yeah, but renos are not yet docified when it runs at all20:17
mnasiadkaWell, it’s still some sort of sphinxdoc, maybe there is some normal option to lint.20:18
yoctozeptomnasiadka: I'll ask on ml20:19
*** TrevorV has joined #openstack-kolla20:23
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: Remove useless neutron-linuxbridge-agent env vars  https://review.opendev.org/70865820:24
*** Trevor_V has quit IRC20:27
iniazion which node types do I run the cinder-volume & cinder-backup (the [storage] inventory)?  on controller nodes?  or all compute nodes?  or just on the ceph servers? all ceph servers?20:39
iniazii'm using external ceph for k-a deployment20:40
*** ne0h|nihilist has quit IRC21:06
*** reddhatt has joined #openstack-kolla21:09
reddhattHi all!  We are using kolla ansible, branch master. We are getting a problem when adding new servers to the cluster. It downloads newer image versions that are not compatible with the images in controllers.21:11
*** lile has quit IRC21:11
reddhattWhat is the best approach to deal with this issue?21:12
noxoid_reddhatt, the documentation recommends to build your own kolla images to prevent the images on docker hub being different from what you have deployed. it is also not recommended to use the master branch unless you are doing development work21:14
noxoid_iniazi, you only need to run one cinder-volume/cinder-backup if you're using ceph but i run them on each of my controllers21:15
*** primeministerp has joined #openstack-kolla21:16
*** happyhemant has quit IRC21:17
yoctozeptoiniazi: in ceph case you can likely put those on controllers since it's ceph doing the job21:18
yoctozeptomore than one for failover21:18
yoctozeptoreddhatt: I sincerely hope you are not using master in production21:19
yoctozeptoas for your case, I would set up local registry, push images known to work now from hosts to it, and use it in kolla-ansible to continue21:20
yoctozeptothat said, for production please use stable branches and build own images to avoid such issues21:21
iniazinoxoid_ & yoctozepto; thanks, i'll run it on my three controllers then21:22
iniazialso, for python venv's i setup venv (for python/kolla-ansible) on the services hosts not just deployment host... or is venv only need on the deployment host?  because on the service/openstack hosts, I couldn't figure out a way to configure that, other than add to each inventory item:  ansible_python_interpreter=/home/USER/venv/kolla/bin/python21:24
iniazii mean configure that in the kolla-ansible multi-node/globals.yml, etc.21:25
*** bjolo has joined #openstack-kolla21:37
reddhattnoxoid_ and yoctozepto: Thank you very much!21:38
noxoid_iniazi, im not sure if its the "correct" way to do it but i put the following in my inventory:21:38
noxoid_http://paste.openstack.org/show/789881/21:38
*** reddhatt has quit IRC21:39
*** TrevorV has quit IRC21:47
iniazicool, let me try that.22:03
*** primeministerp has quit IRC22:03
iniazinoxoid: i think that worked.  i commented out my per-node interpreter and ran ping via ansible, and got  "discovered_interpreter_python": "/usr/bin/python", and then I stuck the [all:vars] in and that disappeared mean it didn't have to autodiscover the interpreter...22:07
*** primeministerp has joined #openstack-kolla22:09
*** primeministerp has quit IRC22:11
*** primeministerp has joined #openstack-kolla22:13
*** primeministerp has quit IRC22:13
*** primeministerp has joined #openstack-kolla22:14
primeministerphey guys, is it possible to build centos 8 containers yet?22:15
*** dosaboy has quit IRC22:25
*** primeministerp has quit IRC22:31
emccormicknoxoid_ just use networking-ansible or networking-generic on the cumulus switches w/ Ironic22:33
emccormickYou'll want to be configuring your cumulus switches with Ansible, so it's sensible to have Ironic do it that way as well22:35
noxoid_was not aware of this, thank you22:35
noxoid_yea we're familiar with ansible just didnt realize there was a generic hardware ml2 interface. i figured it was only vendor-specific22:35
emccormickI love my cumulus switches, fyi but yeah I wouldn't put too much stock in the plugin22:35
emccormickkeep it simple22:35
emccormickFrom what I can see, networking-ansible has been in the base Kolla neutron base image since Queens so you should be good to go to test it out22:38
emccormickThere's networking-generic too which would just be you making bash scripts I guess. Probably too generic ;)22:39
noxoid_excellent, i appreciate this22:39
noxoid_yea i may need to resort to networking-generic since apparently it supports the brocade fastirons that litter our datacenters22:39
noxoid_but this gives me hope22:39
emccormickIt technically supports anything ;)22:40
noxoid_yea :P22:40
emccormickBut it leaves all the work up to you. I mean really pretty much all of it22:40
noxoid_we had almost resigned ourselves to writing a custom ml2 plugin22:40
noxoid_so almost anything is an improvement over that22:41
bjolotroubleshooting ovs and stumbled upon something.22:50
bjolois this a bug?22:51
bjolohttp://paste.openstack.org/show/789885/22:51
bjoloseems like the ovs containers dont mount /var/run/openvswitch22:51
noxoid_https://opendev.org/openstack/kolla-ansible/commit/c4ad080d9a0a79caadac39ecc1a2891219741797 was merged recently that might be related22:53
noxoid_dunno how recent your deployment is or what branch its on22:53
bjolorocky built in december22:53
bjolocentos7 source on centos7 host22:55
bjolowe have a weird issue with ovs vxlan not getting ARPs and vms not getting dhcp. packets just vanishing...22:57
bjolohence me playing with the ovs commands and found this22:57
emccormickI feel like I just saw mnasladka having a cage match with that backport today23:00
emccormickmay not have landed in Rocky?23:00
bjolook23:02
*** primeministerp has joined #openstack-kolla23:06
primeministerpIs it possible to deploy centos7 containers onto a centos8 host w/ kolla-ansible by forcing the openstack_tag or openstack_tag_suffix?23:07
*** dave-mccowan has quit IRC23:11
bjoloactually i guess the issue is the .pid file not found. my containers have /run mounted but the .pid file does not exist23:16
*** dave-mccowan has joined #openstack-kolla23:30
bjoloprimeministerp: centos7 on centos8 should not be a problem in general right? its a container. just like ubuntu container on centos23:38
bjolowhy do you think you have to force anything?23:39
primeministerpbjolo: because when I tried it detects centos 8 and adds a -centos8 to the container image tag23:45
primeministerpbjolo: I don't really see how to stop it23:46
primeministerpbjolo: are there docs for doing so?23:47
primeministerpbjolo: also it's not a matter of it can't, I just need to be quick.  I'm wondering if there's a path of least resistance23:47
primeministerpbjolo: for what i'm doing i'm stuck on centos 8 on the host23:48
*** jbadiapa has quit IRC23:48
*** dosaboy has joined #openstack-kolla23:51
*** primeministerp has quit IRC23:57
*** primeministerp has joined #openstack-kolla23:58

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!