| openstackgerrit | Jason Anderson proposed openstack/kolla-ansible master: [haproxy] Support interfaces with '-' chars https://review.opendev.org/663325 | 00:09 |
|---|---|---|
| *** vishwanathj has joined #openstack-kolla | 00:10 | |
| *** vishwanathj has quit IRC | 00:14 | |
| *** zhanglong has joined #openstack-kolla | 00:27 | |
| *** AjiArya has joined #openstack-kolla | 00:51 | |
| *** igordc has quit IRC | 00:58 | |
| *** portdirect has quit IRC | 01:12 | |
| *** stackedsax has quit IRC | 01:13 | |
| *** stackedsax has joined #openstack-kolla | 01:13 | |
| *** portdirect has joined #openstack-kolla | 01:15 | |
| *** AjiArya has quit IRC | 01:36 | |
| *** mcnugit has joined #openstack-kolla | 02:09 | |
| *** vishwanathj has joined #openstack-kolla | 02:10 | |
| *** vishwanathj has quit IRC | 02:15 | |
| *** factor has joined #openstack-kolla | 02:25 | |
| *** AjiArya has joined #openstack-kolla | 02:34 | |
| AjiArya | Hello guys | 02:39 |
| AjiArya | Why is this error always happen on the first try of run kolla-ansible deploy http://paste.openstack.org/show/789825/ | 02:39 |
| AjiArya | It'll be ok on the second try deploy | 02:40 |
| AjiArya | I'm using CentOS 7 source with everything on latest version | 02:41 |
| *** jcmdln has quit IRC | 03:14 | |
| *** zhanglong has quit IRC | 04:27 | |
| *** zhanglong has joined #openstack-kolla | 04:29 | |
| *** vishwanathj has joined #openstack-kolla | 04:48 | |
| *** vishwanathj has quit IRC | 04:53 | |
| *** zhanglong has quit IRC | 05:21 | |
| *** mcnugit has quit IRC | 05:22 | |
| *** evrardjp has quit IRC | 05:34 | |
| *** evrardjp has joined #openstack-kolla | 05:34 | |
| emccormick | FYI mgoddard When building Stein containers I'm getting ceph-common-14.2.1-0.el7.x86_64 | 05:41 |
| emccormick | That's Nautilus rather than the expected Luminous, and a really old release as well | 05:42 |
| emccormick | Hey AjiArya Did you solve your problem from last night? | 05:42 |
| emccormick | Should be able to set your virt type to qemu for that one | 05:43 |
| emccormick | AjiArya Also, I don't have an answer for the other question. Might be that it's taking the ceph-mgr prometheus module too long to initialize | 05:45 |
| emccormick | doesn't seem critical anyway since the second run passes. | 05:46 |
| *** AjiArya has quit IRC | 06:07 | |
| osmanlicilegi | yoctozepto: "cilek" means strawberry in Turkish and "osmanli cilegi" is a kind of strawberry which is grown only at my hometown and I like it more than honey :] https://tourismguidance.beun.edu.tr/ottoman-strawberry.html you folks, if you like jam and wanna try it, just let me know. one day I will retire myself from IT and be a farmer for ottoman strawberry :] | 06:09 |
| osmanlicilegi | AjiArya: not clear what's happening. you should try kolla-ansible deploy -vvvv to get more details about the error. | 06:12 |
| *** ricolin has quit IRC | 06:15 | |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: OVN Support https://review.opendev.org/696841 | 06:22 |
| *** AjiArya has joined #openstack-kolla | 06:26 | |
| *** numans has joined #openstack-kolla | 06:29 | |
| *** vishwanathj has joined #openstack-kolla | 06:49 | |
| AjiArya | emccormick: I'm still rebuilding my machine system, I'll inform you as soon as possible | 06:52 |
| AjiArya | emccormick: Alright I'll ignore that ceph mgr failure | 06:52 |
| *** vishwanathj has quit IRC | 06:53 | |
| *** cah_link has joined #openstack-kolla | 07:05 | |
| openstackgerrit | Merged openstack/kolla-ansible master: CI: Add linuxbridge jobs https://review.opendev.org/708318 | 07:09 |
| openstackgerrit | Merged openstack/kolla-ansible master: CI: Fine tune Galera gmcast.peer_timeout to 15 seconds https://review.opendev.org/707818 | 07:09 |
| *** ricolin has joined #openstack-kolla | 07:21 | |
| mnasiadka | morning | 07:35 |
| mnasiadka | yoctozepto: I think linaro has some good network problems - https://zuul.opendev.org/t/openstack/build/0803ba5e74fa4580b8854103a68a166a/log/primary/logs/build/000_FAILED_openstack-base.log#586 (aarch64 pipeline) | 07:35 |
| yoctozepto | mnasiadka: maybe because of ping | 07:36 |
| yoctozepto | morning folks | 07:36 |
| yoctozepto | osmanlicilegi: ah, thanks for explanations; in the meantime you might enjoy playing stardew valley ;D | 07:36 |
| *** ivve has quit IRC | 07:39 | |
| *** zhanglong has joined #openstack-kolla | 07:46 | |
| mnasiadka | yoctozepto: I think you need to rebase this https://review.opendev.org/#/c/708497/ | 07:46 |
| patchbot | patch 708497 - kolla-ansible - CI: Replace dummy interface (fake for neutron exte... - 2 patch sets | 07:46 |
| yoctozepto | mnasiadka: k, later | 07:47 |
| yoctozepto | AjiArya: it's probably due to a bug, note ceph deployment in k-a is deprecated now | 07:47 |
| yoctozepto | AjiArya: probably mgr was not yet up | 07:48 |
| yoctozepto | dking_desktop: right indeed | 07:48 |
| yoctozepto | emccormick: interesting, hmm, things must have changed in repos I believe | 07:48 |
| yoctozepto | mnasiadka: want to triage that? | 07:49 |
| mnasiadka | emccormick: is it centos or ubuntu, that is getting Luminous ceph-common? | 07:49 |
| *** rpittau|afk is now known as rpittau | 07:50 | |
| *** zhanglong has quit IRC | 07:50 | |
| *** zhanglong has joined #openstack-kolla | 07:53 | |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: OVN Support https://review.opendev.org/696841 | 07:57 |
| *** bengates has joined #openstack-kolla | 07:59 | |
| yoctozepto | emccormick, mnasiadka: oh, folks, it's nautilus in stein centos https://ethercalc.openstack.org/kolla-infra-service-matrix | 08:08 |
| mnasiadka | Well, Nautilus is newer than Luminous, right? | 08:09 |
| mnasiadka | So I don't know what's the issue still :) | 08:09 |
| mnasiadka | but yeah, CentOS Storage SIG is not the best in updates | 08:11 |
| mnasiadka | 14.2.1 vs 14.2.7 is a bit of a pita | 08:11 |
| yoctozepto | mnasiadka: so we are running PT15S now, huh | 08:13 |
| yoctozepto | mnasiadka: so no more timeouts? :D | 08:13 |
| mnasiadka | yoctozepto: time will tell | 08:13 |
| *** Tony31 has joined #openstack-kolla | 08:14 | |
| *** vishwanathj has joined #openstack-kolla | 08:19 | |
| *** JangwonLee has joined #openstack-kolla | 08:23 | |
| *** vishwanathj has quit IRC | 08:24 | |
| *** JangwonLee_ has quit IRC | 08:24 | |
| *** AjiArya has quit IRC | 08:29 | |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: Use Docker healthchecks for core services https://review.opendev.org/676389 | 08:34 |
| mnasiadka | osmanlicilegi: rebased ^^ | 08:35 |
| *** factor has quit IRC | 08:36 | |
| *** factor has joined #openstack-kolla | 08:36 | |
| osmanlicilegi | mnasiadka: thx! | 08:44 |
| *** ivve has joined #openstack-kolla | 08:44 | |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: Stop using deprecated stores and default_store in glance https://review.opendev.org/708114 | 08:48 |
| *** zhanglong has quit IRC | 08:50 | |
| yoctozepto | mnasiadka: I have read glance sources for that ^ | 08:52 |
| mnasiadka | yoctozepto: lol, good job | 08:53 |
| yoctozepto | mnasiadka: so it really is better now | 08:53 |
| *** zhanglong has joined #openstack-kolla | 08:53 | |
| mnasiadka | yoctozepto: I took the mission of analyzing all errors/warnings about deprecation and fixing them, glance is/was first on the list. | 08:53 |
| yoctozepto | mnasiadka: it's pro work, +2 | 08:54 |
| yoctozepto | mnasiadka: we learn a few things along the path | 08:54 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: CI: Replace dummy interface (fake for neutron external) with VXLAN https://review.opendev.org/708497 | 08:56 |
| yoctozepto | mnasiadka: get me that w+1 | 08:57 |
| Tony31 | Good morning guys | 08:58 |
| Tony31 | Just want to say thanks for your help this week with kayobe/kolla | 08:58 |
| *** tonythomas has joined #openstack-kolla | 08:58 | |
| Tony31 | I tested deleting openstack components (like compute node or controller node) and then re-deploying. Aside from a couple of minor problems that were easily fixed, everything works great. | 08:59 |
| yoctozepto | Tony31: glad to hear that :-) | 09:00 |
| Tony31 | thank you to yoctozepto and mnasiadka | 09:00 |
| yoctozepto | yw | 09:00 |
| Tony31 | 👍🏻 | 09:00 |
| *** Tony31 has left #openstack-kolla | 09:01 | |
| openstackgerrit | Merged openstack/kolla stable/train: CentOS 8: work around mod_ssl certificate issue https://review.opendev.org/702419 | 09:03 |
| *** Tony31 has joined #openstack-kolla | 09:08 | |
| *** AjiArya has joined #openstack-kolla | 09:09 | |
| Tony31 | When I try and launch an instance on centos 7 physical compute node, I get this error `invalid argument: could not find capabilities for domaintype=kvm` But when I try and verify this from the physical machine I see kvm supported: `# lsmod | grep -i kvm | 09:10 |
| Tony31 | 13503 1 kvm` | 09:10 |
| openstackgerrit | Merged openstack/kolla stable/train: CentOS 8: Add zuul build and publish jobs https://review.opendev.org/702420 | 09:12 |
| Tony31 | in the nova_compute and nova_libvirt container I get the same output :) | 09:12 |
| *** dougsz has joined #openstack-kolla | 09:13 | |
| Tony31 | error pastebin from the compute host https://pastebin.com/hHrk1Q53 - is this a bug? | 09:13 |
| cosmicsound | Good day | 09:13 |
| *** k_mouza has joined #openstack-kolla | 09:15 | |
| yoctozepto | Tony31: odd, they are two different things | 09:21 |
| yoctozepto | Tony31: that's not enough to have kvm | 09:21 |
| Tony31 | thanks for the tip. Will keep digging :) | 09:21 |
| yoctozepto | Tony31: it needs to have cpu-specific module loaded | 09:21 |
| yoctozepto | e.g. kvm_intel (or intel_kvm, always forget) | 09:21 |
| yoctozepto | sometimes checking dmesg helps | 09:22 |
| yoctozepto | also bios config | 09:22 |
| cosmicsound | Where is best to find release schedule for train? | 09:22 |
| yoctozepto | cosmicsound: train is released already | 09:23 |
| yoctozepto | cosmicsound: https://releases.openstack.org/ | 09:23 |
| yoctozepto | and add up to 2 months for kolla | 09:23 |
| *** Tony3151 has joined #openstack-kolla | 09:23 | |
| yoctozepto | so we really released in December afair | 09:24 |
| Tony3151 | got disconnected | 09:24 |
| yoctozepto | Tony3151: | 09:24 |
| yoctozepto | 10:21:05 <yoctozepto> Tony31: odd, they are two different things | 09:24 |
| yoctozepto | 10:21:30 <yoctozepto> Tony31: that's not enough to have kvm | 09:24 |
| yoctozepto | 10:21:30 <Tony31> thanks for the tip. Will keep digging :) | 09:24 |
| yoctozepto | 10:21:38 <yoctozepto> Tony31: it needs to have cpu-specific module loaded | 09:24 |
| yoctozepto | 10:21:53 <yoctozepto> e.g. kvm_intel (or intel_kvm, always forget) | 09:24 |
| yoctozepto | 10:22:11 <yoctozepto> sometimes checking dmesg helps | 09:24 |
| yoctozepto | 10:22:19 <yoctozepto> also bios config | 09:24 |
| yoctozepto | and you need to work on that network connection of yours | 09:24 |
| Tony3151 | connection is great I have 100M fibre. It's Australia's link to the rest of the world that's the problem :) | 09:25 |
| *** Tony31 has quit IRC | 09:27 | |
| cosmicsound | Tony3151 , I would recommend you look into thelounge. | 09:27 |
| cosmicsound | If you want I can provide you a connection from Europe. | 09:27 |
| cosmicsound | To be stable :) | 09:27 |
| cosmicsound | yoctozepto , i was thinking there are more releases on same version | 09:28 |
| cosmicsound | such as 9.0 9.0.1 9.0.2 | 09:28 |
| cosmicsound | Yet thats kolla versioning, not Train | 09:28 |
| AjiArya | osmanlicilegi: this is the verbose result http://paste.openstack.org/show/789840/ | 09:29 |
| yoctozepto | cosmicsound: https://releases.openstack.org/teams/kolla.html#team-train-kolla and https://docs.openstack.org/releasenotes/kolla/train.html | 09:29 |
| yoctozepto | AjiArya: as I said, it's probably due to ceph mgr not being ready yet | 09:30 |
| AjiArya | Just send what osmanlicilegi asked | 09:32 |
| yoctozepto | AjiArya: just repeating because I did not know whether you read that :-) | 09:33 |
| AjiArya | Sorry | 09:33 |
| AjiArya | I've read those | 09:33 |
| mgoddard | morning | 09:34 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: CI: Remove useless pre task https://review.opendev.org/709071 | 09:38 |
| Tony3151 | mgoddard morning :) | 09:59 |
| Tony3151 | yoctozepto - I have fixed the issue now but I am not sure about it. The fix, apparently was to enable `intel_iommu` in the cmdline and boot... Here's a virt-host-validate from before I done the reboot. Although, to install virt-host-validate I had to install `libvirt-client` and its dependencies which could have fixed the problem. To install the | 10:01 |
| Tony3151 | host, I used the iso for centos 7 and done the basic minimal install. I had a similar problem to this when I used the cloud-image and used the kayobe all in one. But the diffrerence there was nested virt. Here there are no nests :) | 10:01 |
| Tony3151 | eventually found the tab: https://pastebin.com/p54tvgSb | 10:02 |
| yoctozepto | Tony3151: odd, iommu should not be required | 10:02 |
| Tony3151 | I think i was missing some libvirt package... | 10:02 |
| yoctozepto | Tony3151: yeah, it's just a warn | 10:02 |
| yoctozepto | Tony3151: you should not have been missing any | 10:02 |
| yoctozepto | Tony3151: the libvirt and qemu are containerized too | 10:02 |
| Tony3151 | Odd one... | 10:03 |
| yoctozepto | Tony3151: was it centos 7.7 | 10:03 |
| yoctozepto | ? | 10:03 |
| Tony3151 | hmm I wonder if all I needed was a reboot | 10:03 |
| Tony3151 | Indeed centos 7 and updated before running host configure / deploy | 10:03 |
| yoctozepto | Tony3151: I'll take a note to spin up such machine and see for myself | 10:04 |
| yoctozepto | Tony3151: so you did minimal install | 10:05 |
| yoctozepto | Tony3151: from iso | 10:05 |
| yoctozepto | Tony3151: then updated | 10:05 |
| yoctozepto | Tony3151: then no reboot? | 10:05 |
| Tony3151 | yes, the iso is the 1903 version. Then once I was in, I removed wpa_supplicant, then ran `yum update -y` and disabled selinux and rebooted (so the host configure did not need to wait for host reboot). Then I done a deploy | 10:06 |
| Tony3151 | But, actually I dont think the host got rebooted after host configure/deploy - this could be the reason. I found before that libvirt needs a reboot (I couldnt find the service) | 10:06 |
| mnasiadka | mgoddard, yoctozepto: https://review.opendev.org/#/c/696841/96 - addressed I think all comments and it still works... | 10:09 |
| patchbot | patch 696841 - kolla-ansible - OVN Support - 96 patch sets | 10:09 |
| *** zhanglong has quit IRC | 10:10 | |
| Tony3151 | yoctozepto - thanks again for the help. Have a good weekend all | 10:13 |
| *** Tony3151 has left #openstack-kolla | 10:13 | |
| mnasiadka | mgoddard: another thing - are you fine with that change? https://review.opendev.org/#/c/707800/2 - if yes, I'll populate it in a similar way to other Kolla family repos | 10:19 |
| patchbot | patch 707800 - kolla - [community goal]: Add contributor and PTL guide - 2 patch sets | 10:19 |
| *** vishwanathj has joined #openstack-kolla | 10:19 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: WIP: CI: Add monasca scenario https://review.opendev.org/649893 | 10:23 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Fix invalid JSON body in Elasticsearch API requests https://review.opendev.org/709090 | 10:23 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Fix kibana deployment with openstack_cacert unset https://review.opendev.org/709091 | 10:23 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Fix monasca deployment due to monasca_log_dir https://review.opendev.org/709092 | 10:23 |
| *** vishwanathj has quit IRC | 10:24 | |
| mgoddard | mnasiadka: will take a look | 10:26 |
| *** jbadiapa has joined #openstack-kolla | 10:31 | |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla master: WIP: Fix Debian ping issue with Neutron https://review.opendev.org/709102 | 10:58 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Check for supported OS distributions in prechecks https://review.opendev.org/699443 | 11:06 |
| *** happyhemant has joined #openstack-kolla | 11:08 | |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: CI: Test SSH connectivity to deployed instances https://review.opendev.org/707604 | 11:19 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: WIP: CI: Add monasca scenario https://review.opendev.org/649893 | 11:22 |
| yoctozepto | mnasiadka: not issue with ping? care to elaborate? :D | 11:32 |
| mnasiadka | yoctozepto: it's not ping's fault, that neutron is running "ping sysctl -w some_var=1" | 11:33 |
| mnasiadka | :D | 11:33 |
| yoctozepto | mnasiadka: lol, that's the command? gosh | 11:33 |
| yoctozepto | too new python issue? | 11:33 |
| mnasiadka | have no clue, a new candidate for bug at neutron side | 11:34 |
| yoctozepto | mnasiadka: go ahead, full support from me | 11:35 |
| *** vishwanathj has joined #openstack-kolla | 11:40 | |
| *** rpittau is now known as rpittau|bbl | 11:43 | |
| *** AjiArya has quit IRC | 11:44 | |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible stable/stein: Change /run bind mount for neutron/openvswitch https://review.opendev.org/709115 | 11:46 |
| *** zhanglong has joined #openstack-kolla | 11:49 | |
| *** dougsz has quit IRC | 12:07 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: CI: sync inventory with multinode https://review.opendev.org/709121 | 12:12 |
| *** JangwonLee has quit IRC | 12:13 | |
| *** JangwonLee has joined #openstack-kolla | 12:14 | |
| *** ricolin has quit IRC | 12:14 | |
| yoctozepto | mgoddard: some questions from yesterday for you: | 12:16 |
| yoctozepto | 19:39:52 <yoctozepto> mgoddard, mnasiadka: am I right that kayobe-config(-dev) is just config included in kayobe but cut off at release points? | 12:16 |
| yoctozepto | 19:41:59 <yoctozepto> mgoddard, mnasiadka: and is -dev variant simply base filled with aio defaults? | 12:16 |
| mgoddard | sorry, missed those | 12:17 |
| mgoddard | that's basically correct | 12:18 |
| *** zhanglong has quit IRC | 12:19 | |
| *** zhanglong has joined #openstack-kolla | 12:21 | |
| yoctozepto | mgoddard: that "basically" worries me every time | 12:34 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: [baremetal] Drop useless reboot task https://review.opendev.org/709127 | 12:43 |
| *** zhanglong has quit IRC | 12:48 | |
| *** zhanglong has joined #openstack-kolla | 12:49 | |
| *** sluna has quit IRC | 12:53 | |
| *** sluna has joined #openstack-kolla | 12:53 | |
| *** zhanglong has quit IRC | 12:54 | |
| mnasiadka | mgoddard, yoctozepto: maybe instead syncing inventory in tests/templates once in a while - we could make a custom part of inventory that we template out in CI and the other part would be standard aio/multinode - we would just put it in one directory for Ansible to consume? | 12:55 |
| *** rpittau|bbl is now known as rpittau | 13:00 | |
| *** vishwanathj has quit IRC | 13:00 | |
| *** vishwanathj has joined #openstack-kolla | 13:01 | |
| *** vishwanathj has quit IRC | 13:06 | |
| *** Luzi has joined #openstack-kolla | 13:08 | |
| *** dougsz has joined #openstack-kolla | 13:08 | |
| *** negronjl has quit IRC | 13:10 | |
| *** negronjl has joined #openstack-kolla | 13:12 | |
| yoctozepto | mnasiadka: +1 | 13:12 |
| yoctozepto | mnasiadka: better reliability | 13:12 |
| mgoddard | I was going to suggest we just make it a requirement to update the test inventory any time the inventory changes | 13:18 |
| mgoddard | it's not just a static inventory, it's a template | 13:19 |
| mgoddard | we have things like ceph transition | 13:19 |
| todin | I am trying to get ovs-dpdk working, but somehow I can't bind the NIC device, in a non kolla deployed container it's working. Just for test, how could I add those docker args "--security-opt seccomp=unconfined --cap-add=ALL" to the ovsdpdk_vswitchd container? | 13:37 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: CentOS 8: Use ansible_playbook_python for localhost dependencies https://review.opendev.org/708921 | 13:47 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: Make Kayobe code compatible with Python 3 https://review.opendev.org/635586 | 13:47 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: Bump Ansible version range to 2.8 - 2.9 https://review.opendev.org/708922 | 13:47 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: WIP: CentOS 8 https://review.opendev.org/707690 | 13:47 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: CI: set previous_release to train https://review.opendev.org/709145 | 13:55 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: CI: Test SSH connectivity to deployed instances https://review.opendev.org/707604 | 13:55 |
| *** rpittau is now known as rpittau|afk | 13:57 | |
| *** bjolo has quit IRC | 14:11 | |
| emccormick | yoctozepto,mnasiadk I swear I looked at that ethercalc yesterday and it said Centos Stein was Luminous. I musta had a brain cramp ;) | 14:12 |
| emccormick | If one was going to add the download.ceph.com repo instead, where would be the best place to do that? Hopefully I can squash the Centos one that RDO is forcing | 14:13 |
| yoctozepto | mgoddard: well, we should simulate how users would use our software so mnasiadka has a point there | 14:13 |
| emccormick | I can't be putting 14.2.1 on there | 14:13 |
| yoctozepto | emccormick: the bad thing is that it requires epel | 14:13 |
| emccormick | hmm | 14:14 |
| emccormick | well, while I'm about adding repos it's not that hard. Are you concerned it might blow up versions of other things? | 14:14 |
| *** Luzi has quit IRC | 14:18 | |
| *** sluna has quit IRC | 14:18 | |
| yoctozepto | we removed it on purpose, it does indeed tends to affect things | 14:23 |
| yoctozepto | better attack real problem | 14:23 |
| yoctozepto | make centos storage sig release | 14:23 |
| yoctozepto | or join their ranks and help | 14:23 |
| emccormick | Hah | 14:27 |
| emccormick | That's not going to help me like...tonight | 14:27 |
| emccormick | but yes I will see if I can figure out how to ping them | 14:27 |
| *** dking_desktop has left #openstack-kolla | 14:37 | |
| *** TrevorV has joined #openstack-kolla | 14:49 | |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: Update documentation for release tasks https://review.opendev.org/709154 | 15:03 |
| *** ivve has quit IRC | 15:12 | |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: CentOS 8: Use ansible_playbook_python for localhost dependencies https://review.opendev.org/708921 | 15:39 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: Make Kayobe code compatible with Python 3 https://review.opendev.org/635586 | 15:39 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: Bump Ansible version range to 2.8 - 2.9 https://review.opendev.org/708922 | 15:39 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: WIP: CentOS 8 https://review.opendev.org/707690 | 15:39 |
| *** lile has joined #openstack-kolla | 15:49 | |
| yoctozepto | emccormick: well, it's not always the case that those newer versions are considerably better than current - sometimes they even break stuff; best read releasenotes; and note openstack does not use all ceph features, especially many manageability ones so they being patched is highly irrelevant ;-) | 15:53 |
| yoctozepto | emccormick: that said storage sig needs help for sure | 15:53 |
| openstackgerrit | Doug Szumski proposed openstack/kayobe master: Fix Kayobe overcloud introspection data save https://review.opendev.org/709163 | 15:54 |
| mnasiadka | emccormick: we've been there - I mean using the download.ceph.com repo - but Storage SIG got better and updated the packages - and convinced us to go back | 16:05 |
| mnasiadka | emccormick: but we could put that topic on next Kolla meeting agenda - because from my perspective the best idea would be to use upstream ceph.com packages on both Ubuntu and CentOS - so we would have the same versions on both distros. | 16:06 |
| emccormick | mnasladka I agree re: using upstream Ceph and having version consistency across all Kolla deployments would be nice. That Matrix shouldn't need to exist except to say which version of Ceph is in each release of Kolla | 16:11 |
| *** cah_link has quit IRC | 16:15 | |
| yoctozepto | mnasiadka: convince ceph to build required packages based on centos, w/o epel | 16:19 |
| yoctozepto | mnasiadka: otherwise storage sig at least provides better patching of these auxiliary things | 16:19 |
| yoctozepto | mnasiadka: this really is dumb in case of ceph, since it's really the same company behind both sides... | 16:20 |
| yoctozepto | emccormick: ^ | 16:20 |
| mnasiadka | yeah, the main problem behind CentOS Storage SIG is one guy doing the packaging and testing semi-manually | 16:21 |
| yoctozepto | mnasiadka: can't we organize this in opendev? | 16:22 |
| yoctozepto | mnasiadka: this really is *not* that hard | 16:22 |
| mnasiadka | yoctozepto: well, it's centos - we shouldn't really touch that, packaging of deps is not really openstack problem :) | 16:23 |
| yoctozepto | mnasiadka: well, sure; just that ceph is major component in many deploys ;-) | 16:23 |
| mnasiadka | I'll speak with Giulio Fidente next week, let's see if he needs help in automating those bloody updates or something | 16:23 |
| mnasiadka | second thing is - there has been so many Nautilus releases that were broken | 16:24 |
| mnasiadka | that maybe he was waiting for stabilizing :) | 16:24 |
| mnasiadka | https://ethercalc.openstack.org/kolla-ceph-versions | 16:24 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: DNM: Testing cloudkitty in master https://review.opendev.org/681555 | 16:24 |
| mnasiadka | ubuntu is a bit better on keeping up | 16:24 |
| mnasiadka | mgoddard: got a minute for https://review.opendev.org/#/c/709115/ ? | 16:25 |
| patchbot | patch 709115 - kolla-ansible (stable/stein) - Change /run bind mount for neutron/openvswitch - 1 patch set | 16:25 |
| mnasiadka | then it's only rocky that is left | 16:25 |
| mgoddard | mnasiadka: was just looking at it | 16:28 |
| mgoddard | and it's off | 16:28 |
| yoctozepto | mnasiadka: and at breaking things | 16:30 |
| yoctozepto | mnasiadka: so I see a correlation there | 16:30 |
| *** bengates has quit IRC | 16:33 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: DNM: Testing cloudkitty in master https://review.opendev.org/681555 | 16:37 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Bifrost: Stop stopping RabbitMQ https://review.opendev.org/709189 | 16:42 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: CI: set previous_release to train https://review.opendev.org/709145 | 16:42 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: CentOS 8: Use ansible_playbook_python for localhost dependencies https://review.opendev.org/708921 | 17:02 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: Make Kayobe code compatible with Python 3 https://review.opendev.org/635586 | 17:02 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: Bump Ansible version range to 2.8 - 2.9 https://review.opendev.org/708922 | 17:02 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: WIP: CentOS 8 https://review.opendev.org/707690 | 17:02 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: WIP: CentOS 8 https://review.opendev.org/707690 | 17:09 |
| openstackgerrit | Merged openstack/kolla-ansible master: CI: Replace dummy interface (fake for neutron external) with VXLAN https://review.opendev.org/708497 | 17:19 |
| openstackgerrit | Merged openstack/kolla-ansible stable/stein: Change /run bind mount for neutron/openvswitch https://review.opendev.org/709115 | 17:27 |
| *** evrardjp has quit IRC | 17:34 | |
| *** evrardjp has joined #openstack-kolla | 17:35 | |
| *** Trevor_V has joined #openstack-kolla | 17:39 | |
| *** TrevorV has quit IRC | 17:42 | |
| *** k_mouza has quit IRC | 17:43 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/train: Configure Cinder to use lioadm on CentOS/RHEL 8 https://review.opendev.org/702417 | 17:43 |
| *** dougsz has quit IRC | 17:52 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla stable/train: CentOS 8: Don't force tag in build jobs https://review.opendev.org/702452 | 17:58 |
| openstackgerrit | Mark Goddard proposed openstack/kolla stable/train: CentOS 8: base image fixes for RabbitMQ, fluentd & InfluxDB https://review.opendev.org/709201 | 17:58 |
| openstackgerrit | Mark Goddard proposed openstack/kolla stable/train: CentOS 8: Update packages in images https://review.opendev.org/709202 | 17:58 |
| openstackgerrit | Mark Goddard proposed openstack/kolla stable/train: CentOS 8: Remove shellinabox from ironic-conductor https://review.opendev.org/709203 | 17:58 |
| openstackgerrit | Mark Goddard proposed openstack/kolla stable/train: CentOS 8: Use upstream Ceph/master https://review.opendev.org/709204 | 17:58 |
| openstackgerrit | Mark Goddard proposed openstack/kolla stable/train: CentOS 8: Allow SSH access to keystone_ssh and nova_ssh https://review.opendev.org/709205 | 17:58 |
| openstackgerrit | Mark Goddard proposed openstack/kolla stable/train: CentOS 8: disable repo_gpgcheck for grafana image https://review.opendev.org/709206 | 17:58 |
| openstackgerrit | Mark Goddard proposed openstack/kolla stable/train: CentOS 8: Relax ironic iPXE EFI bootloader location https://review.opendev.org/709207 | 17:58 |
| mgoddard | Centos 8 train go! | 17:58 |
| *** k_mouza has joined #openstack-kolla | 18:08 | |
| *** tonythomas has quit IRC | 18:11 | |
| *** aleccoder has quit IRC | 18:12 | |
| *** klippo has quit IRC | 18:12 | |
| *** jovial[m] has quit IRC | 18:12 | |
| *** k_mouza has quit IRC | 18:12 | |
| *** k_mouza has joined #openstack-kolla | 18:15 | |
| openstackgerrit | Pierre Riteau proposed openstack/kayobe master: Add support for configuring Cumulus switches with NCLU https://review.opendev.org/636941 | 18:15 |
| *** primeministerp has joined #openstack-kolla | 18:16 | |
| *** k_mouza has quit IRC | 18:17 | |
| noxoid_ | ^ that actually reminds me. does anyone here use the cumulus ml2 driver for any release newer than queens? it looks like the neutron plugin hasnt been updated in a long time | 18:27 |
| *** openstackstatus has quit IRC | 18:33 | |
| *** klippo has joined #openstack-kolla | 19:02 | |
| *** ne0h|nihilist has joined #openstack-kolla | 19:03 | |
| *** igordc has joined #openstack-kolla | 19:10 | |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible stable/rocky: Change /run bind mount for neutron/openvswitch https://review.opendev.org/709225 | 19:15 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible stable/rocky: Change /run bind mount for neutron/openvswitch https://review.opendev.org/709225 | 19:15 |
| *** aleccoder has joined #openstack-kolla | 19:16 | |
| *** negronjl has quit IRC | 19:19 | |
| *** negronjl has joined #openstack-kolla | 19:22 | |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible stable/rocky: Change /run bind mount for neutron/openvswitch https://review.opendev.org/707598 | 19:22 |
| mnasiadka | ugh, backporting is not always clean :) | 19:22 |
| *** primeministerp has quit IRC | 19:26 | |
| osmanlicilegi | mnasiadka: dont give up :] | 19:29 |
| yoctozepto | noxoid_: how does it work? (curious) | 19:34 |
| noxoid_ | neutron hardware ml2 drivers? | 19:35 |
| noxoid_ | they're added as a mechanism driver to ml2_conf.ini and allow neutron to make changes on the switch (usually vlan related) via an API | 19:38 |
| noxoid_ | if you're asking about the cumulus driver specifically, i do not know lol. my company is looking into purchasing a lot of cumulus switches but their sales people seem to have exaggerated how much they support the neutron plugin | 19:39 |
| noxoid_ | https://github.com/CumulusNetworks/networking-cumulus | 19:40 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: Construct service configuration urls using kolla_internal_fqdn https://review.opendev.org/706627 | 19:43 |
| osmanlicilegi | noxoid_: we have tens of cumulus running on msn2410 and I an clearly say forget the ml2_cumulus driver. from my point of view, the question is why choose cumulus instead of onyx? | 19:47 |
| noxoid_ | havent heard of them | 19:49 |
| yoctozepto | noxoid_: yeah, I mean what and when openstack configures on these switches | 19:52 |
| yoctozepto | noxoid_: so this is to wire up vlans for physical network deployments? | 19:52 |
| yoctozepto | (provider network) | 19:52 |
| noxoid_ | yoctozepto, yes. its particularly helpful for ironic where it can automatically change the vlan on a bare metal servers port so it can communicate with the rest of the tenant network | 19:53 |
| yoctozepto | noxoid_: I see, otherwise one has to run with all vlans trunked which is not top notch security so to speak | 19:54 |
| noxoid_ | yep | 19:54 |
| noxoid_ | theres also layer3 drivers floating around that allow the switches to act as VTEPs | 19:55 |
| yoctozepto | noxoid_: I'm blessed with dumb switches so I'm rather into overlaying and stuff | 19:55 |
| noxoid_ | hehe im in a similar boat. i have arista switches for openstack but havent had the time to look into how the arista drivers can help improve my existing deployment | 19:56 |
| osmanlicilegi | noxoid_: I think the most common hardware and support for cumulus is provided by mellanox. if you are planning to go with mellanox, you should ask them about onyx. | 19:56 |
| noxoid_ | osmanlicilegi, thanks. we're heavily leaning towards a cumulus express switch at the moment which is actually edgecore hardware | 19:57 |
| *** iniazi has joined #openstack-kolla | 19:58 | |
| yoctozepto | egonzalez: https://bugs.launchpad.net/mistral/+bug/1864238 fyi | 19:59 |
| openstack | Launchpad bug 1864238 in kolla-ansible "kolla-ansible CI - NFV jobs fail on mistral test" [Medium,Triaged] | 19:59 |
| yoctozepto | noxoid_: I have worse: netgear "smart managed" switches | 19:59 |
| yoctozepto | which are actually dumb ones with lame http interface | 19:59 |
| noxoid_ | my condolences :P | 19:59 |
| yoctozepto | noxoid_: thanks | 20:00 |
| noxoid_ | the struggle is real | 20:00 |
| osmanlicilegi | noxoid_: I don't mean cumulus is bad. I just have doubts about whether it's mature or not. we have onyx switches without problems. we have cumulus switches with problems :) | 20:00 |
| noxoid_ | thats fair. we definitely prefer arista but budgets are sadly a thing | 20:00 |
| osmanlicilegi | yoctozepto: you should think about hacking and installing ovs on your switches :P | 20:02 |
| yoctozepto | osmanlicilegi: if you know of any instructions, especially those that would work on a running system and not break the interswicth link then I'm all ears | 20:05 |
| yoctozepto | was not cumulus oss? | 20:05 |
| yoctozepto | so theoretically you can tweak it however you want | 20:06 |
| yoctozepto | and you stay in oss world | 20:06 |
| yoctozepto | noxoid_, osmanlicilegi: ^ | 20:09 |
| *** negronjl has quit IRC | 20:09 | |
| noxoid_ | yea cumulus is based on debian | 20:09 |
| yoctozepto | so why would taking care of that driver be too bad | 20:10 |
| yoctozepto | as in one can even set up a pretty decent CI with it | 20:10 |
| yoctozepto | w/o rela hardware | 20:10 |
| yoctozepto | real* | 20:10 |
| noxoid_ | it probably wouldnt in the big picture, but its always good to look at community involvement when choosing OSS components | 20:10 |
| *** negronjl has joined #openstack-kolla | 20:11 | |
| yoctozepto | mgoddard, mnasiadka: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_4e5/706627/4/check/build-openstack-releasenotes/4e580d2/docs/unreleased.html need to think about linting of renos, they got really ugly as we missed some links were misformatted | 20:14 |
| yoctozepto | noxoid_: yeah, I'm just thinking loud because it's a bit undermining that such project does not integrate well with openstack | 20:15 |
| mnasiadka | yoctozepto: shouldn’t doc8 do the trick? | 20:15 |
| yoctozepto | mnasiadka: no idea where it gets applied, will see | 20:16 |
| mnasiadka | yoctozepto: in docs, but it runs only on doc/* | 20:16 |
| yoctozepto | mnasiadka: yeah, but renos are not yet docified when it runs at all | 20:17 |
| mnasiadka | Well, it’s still some sort of sphinxdoc, maybe there is some normal option to lint. | 20:18 |
| yoctozepto | mnasiadka: I'll ask on ml | 20:19 |
| *** TrevorV has joined #openstack-kolla | 20:23 | |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: Remove useless neutron-linuxbridge-agent env vars https://review.opendev.org/708658 | 20:24 |
| *** Trevor_V has quit IRC | 20:27 | |
| iniazi | on which node types do I run the cinder-volume & cinder-backup (the [storage] inventory)? on controller nodes? or all compute nodes? or just on the ceph servers? all ceph servers? | 20:39 |
| iniazi | i'm using external ceph for k-a deployment | 20:40 |
| *** ne0h|nihilist has quit IRC | 21:06 | |
| *** reddhatt has joined #openstack-kolla | 21:09 | |
| reddhatt | Hi all! We are using kolla ansible, branch master. We are getting a problem when adding new servers to the cluster. It downloads newer image versions that are not compatible with the images in controllers. | 21:11 |
| *** lile has quit IRC | 21:11 | |
| reddhatt | What is the best approach to deal with this issue? | 21:12 |
| noxoid_ | reddhatt, the documentation recommends to build your own kolla images to prevent the images on docker hub being different from what you have deployed. it is also not recommended to use the master branch unless you are doing development work | 21:14 |
| noxoid_ | iniazi, you only need to run one cinder-volume/cinder-backup if you're using ceph but i run them on each of my controllers | 21:15 |
| *** primeministerp has joined #openstack-kolla | 21:16 | |
| *** happyhemant has quit IRC | 21:17 | |
| yoctozepto | iniazi: in ceph case you can likely put those on controllers since it's ceph doing the job | 21:18 |
| yoctozepto | more than one for failover | 21:18 |
| yoctozepto | reddhatt: I sincerely hope you are not using master in production | 21:19 |
| yoctozepto | as for your case, I would set up local registry, push images known to work now from hosts to it, and use it in kolla-ansible to continue | 21:20 |
| yoctozepto | that said, for production please use stable branches and build own images to avoid such issues | 21:21 |
| iniazi | noxoid_ & yoctozepto; thanks, i'll run it on my three controllers then | 21:22 |
| iniazi | also, for python venv's i setup venv (for python/kolla-ansible) on the services hosts not just deployment host... or is venv only need on the deployment host? because on the service/openstack hosts, I couldn't figure out a way to configure that, other than add to each inventory item: ansible_python_interpreter=/home/USER/venv/kolla/bin/python | 21:24 |
| iniazi | i mean configure that in the kolla-ansible multi-node/globals.yml, etc. | 21:25 |
| *** bjolo has joined #openstack-kolla | 21:37 | |
| reddhatt | noxoid_ and yoctozepto: Thank you very much! | 21:38 |
| noxoid_ | iniazi, im not sure if its the "correct" way to do it but i put the following in my inventory: | 21:38 |
| noxoid_ | http://paste.openstack.org/show/789881/ | 21:38 |
| *** reddhatt has quit IRC | 21:39 | |
| *** TrevorV has quit IRC | 21:47 | |
| iniazi | cool, let me try that. | 22:03 |
| *** primeministerp has quit IRC | 22:03 | |
| iniazi | noxoid: i think that worked. i commented out my per-node interpreter and ran ping via ansible, and got "discovered_interpreter_python": "/usr/bin/python", and then I stuck the [all:vars] in and that disappeared mean it didn't have to autodiscover the interpreter... | 22:07 |
| *** primeministerp has joined #openstack-kolla | 22:09 | |
| *** primeministerp has quit IRC | 22:11 | |
| *** primeministerp has joined #openstack-kolla | 22:13 | |
| *** primeministerp has quit IRC | 22:13 | |
| *** primeministerp has joined #openstack-kolla | 22:14 | |
| primeministerp | hey guys, is it possible to build centos 8 containers yet? | 22:15 |
| *** dosaboy has quit IRC | 22:25 | |
| *** primeministerp has quit IRC | 22:31 | |
| emccormick | noxoid_ just use networking-ansible or networking-generic on the cumulus switches w/ Ironic | 22:33 |
| emccormick | You'll want to be configuring your cumulus switches with Ansible, so it's sensible to have Ironic do it that way as well | 22:35 |
| noxoid_ | was not aware of this, thank you | 22:35 |
| noxoid_ | yea we're familiar with ansible just didnt realize there was a generic hardware ml2 interface. i figured it was only vendor-specific | 22:35 |
| emccormick | I love my cumulus switches, fyi but yeah I wouldn't put too much stock in the plugin | 22:35 |
| emccormick | keep it simple | 22:35 |
| emccormick | From what I can see, networking-ansible has been in the base Kolla neutron base image since Queens so you should be good to go to test it out | 22:38 |
| emccormick | There's networking-generic too which would just be you making bash scripts I guess. Probably too generic ;) | 22:39 |
| noxoid_ | excellent, i appreciate this | 22:39 |
| noxoid_ | yea i may need to resort to networking-generic since apparently it supports the brocade fastirons that litter our datacenters | 22:39 |
| noxoid_ | but this gives me hope | 22:39 |
| emccormick | It technically supports anything ;) | 22:40 |
| noxoid_ | yea :P | 22:40 |
| emccormick | But it leaves all the work up to you. I mean really pretty much all of it | 22:40 |
| noxoid_ | we had almost resigned ourselves to writing a custom ml2 plugin | 22:40 |
| noxoid_ | so almost anything is an improvement over that | 22:41 |
| bjolo | troubleshooting ovs and stumbled upon something. | 22:50 |
| bjolo | is this a bug? | 22:51 |
| bjolo | http://paste.openstack.org/show/789885/ | 22:51 |
| bjolo | seems like the ovs containers dont mount /var/run/openvswitch | 22:51 |
| noxoid_ | https://opendev.org/openstack/kolla-ansible/commit/c4ad080d9a0a79caadac39ecc1a2891219741797 was merged recently that might be related | 22:53 |
| noxoid_ | dunno how recent your deployment is or what branch its on | 22:53 |
| bjolo | rocky built in december | 22:53 |
| bjolo | centos7 source on centos7 host | 22:55 |
| bjolo | we have a weird issue with ovs vxlan not getting ARPs and vms not getting dhcp. packets just vanishing... | 22:57 |
| bjolo | hence me playing with the ovs commands and found this | 22:57 |
| emccormick | I feel like I just saw mnasladka having a cage match with that backport today | 23:00 |
| emccormick | may not have landed in Rocky? | 23:00 |
| bjolo | ok | 23:02 |
| *** primeministerp has joined #openstack-kolla | 23:06 | |
| primeministerp | Is it possible to deploy centos7 containers onto a centos8 host w/ kolla-ansible by forcing the openstack_tag or openstack_tag_suffix? | 23:07 |
| *** dave-mccowan has quit IRC | 23:11 | |
| bjolo | actually i guess the issue is the .pid file not found. my containers have /run mounted but the .pid file does not exist | 23:16 |
| *** dave-mccowan has joined #openstack-kolla | 23:30 | |
| bjolo | primeministerp: centos7 on centos8 should not be a problem in general right? its a container. just like ubuntu container on centos | 23:38 |
| bjolo | why do you think you have to force anything? | 23:39 |
| primeministerp | bjolo: because when I tried it detects centos 8 and adds a -centos8 to the container image tag | 23:45 |
| primeministerp | bjolo: I don't really see how to stop it | 23:46 |
| primeministerp | bjolo: are there docs for doing so? | 23:47 |
| primeministerp | bjolo: also it's not a matter of it can't, I just need to be quick. I'm wondering if there's a path of least resistance | 23:47 |
| primeministerp | bjolo: for what i'm doing i'm stuck on centos 8 on the host | 23:48 |
| *** jbadiapa has quit IRC | 23:48 | |
| *** dosaboy has joined #openstack-kolla | 23:51 | |
| *** primeministerp has quit IRC | 23:57 | |
| *** primeministerp has joined #openstack-kolla | 23:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!