| *** ivve has quit IRC | 00:33 | |
| *** msalo has joined #openstack-kolla | 01:33 | |
| *** goldyfruit_ has quit IRC | 01:53 | |
| *** msalo has quit IRC | 02:08 | |
| *** zhanglong has joined #openstack-kolla | 02:24 | |
| *** msalo has joined #openstack-kolla | 02:47 | |
| *** msalo has quit IRC | 02:58 | |
| *** msalo has joined #openstack-kolla | 02:59 | |
| *** schwicht has joined #openstack-kolla | 03:11 | |
| *** msalo has quit IRC | 03:12 | |
| *** msalo has joined #openstack-kolla | 03:45 | |
| *** rohit02 has joined #openstack-kolla | 03:47 | |
| *** msalo has quit IRC | 04:24 | |
| *** msalo has joined #openstack-kolla | 04:52 | |
| *** skramaja has joined #openstack-kolla | 04:58 | |
| *** msalo has quit IRC | 05:26 | |
| *** msalo has joined #openstack-kolla | 05:35 | |
| openstackgerrit | YU CHENGDE proposed openstack/kolla-ansible master: [Animbus]fix: Extend the counts of subproject by Kolla-Ansible https://review.opendev.org/694714 | 05:38 |
|---|---|---|
| *** msalo has quit IRC | 05:42 | |
| *** sri_ has joined #openstack-kolla | 05:46 | |
| *** msalo has joined #openstack-kolla | 06:03 | |
| *** msalo has quit IRC | 06:08 | |
| *** Luzi has joined #openstack-kolla | 06:11 | |
| *** msalo has joined #openstack-kolla | 06:22 | |
| *** msalo has quit IRC | 06:26 | |
| *** msalo has joined #openstack-kolla | 06:27 | |
| *** msalo has quit IRC | 06:39 | |
| osmanlicilegi | morning | 06:40 |
| *** numans_ has joined #openstack-kolla | 06:44 | |
| sri_ | morning , quick question, is there any way we can enable rbd mirring in kolla-ansible | 06:46 |
| osmanlicilegi | sri_: rbd mirrors are not supported with k-a, you have to configure them manually. also, k-a ceph support is deprecated so you should plan to build an external cluster with tools liks ceph-ansible if you are building a long term cluster. | 06:57 |
| osmanlicilegi | sri_: https://docs.openstack.org/kolla-ansible/latest/reference/storage/external-ceph-guide.html | 06:58 |
| sri_ | osmanlicilegi, understood, thanks :) | 06:58 |
| *** msalo has joined #openstack-kolla | 07:18 | |
| yoctozepto | morning | 07:21 |
| yoctozepto | bline: are you trying deploying centos on ubuntu? because it looks like there is py2.7 in the image so it must be centos... | 07:22 |
| *** rpittau|afk is now known as rpittau | 07:28 | |
| *** dpawlik has joined #openstack-kolla | 07:34 | |
| *** jschlueter has quit IRC | 07:37 | |
| *** jbadiapa has joined #openstack-kolla | 07:38 | |
| *** jschlueter has joined #openstack-kolla | 07:38 | |
| *** vrdel has joined #openstack-kolla | 07:42 | |
| *** igordc has joined #openstack-kolla | 07:58 | |
| *** msalo has quit IRC | 08:01 | |
| *** igordc has quit IRC | 08:02 | |
| *** luksky has joined #openstack-kolla | 08:08 | |
| *** msalo has joined #openstack-kolla | 08:36 | |
| mnasiadka | morning | 08:38 |
| *** ivve has joined #openstack-kolla | 08:46 | |
| *** gfidente has joined #openstack-kolla | 08:49 | |
| mgoddard | bline: hi. There are various bugs reported about ubuntu+KVM. Perhaps it's one of those | 08:57 |
| mgoddard | bline: https://bugs.launchpad.net/kolla-ansible/+bug/1787392 | 08:58 |
| openstack | Launchpad bug 1787392 in kolla-ansible "Permissions of /dev/kvm are altered by Ubuntu 16.04 aarch64 udev rules and nova_libvirt/nova_compute can't access the kvm module" [Undecided,New] | 08:58 |
| mgoddard | bline: and a really old one: https://bugs.launchpad.net/kolla/+bug/1681461 | 08:58 |
| openstack | Launchpad bug 1681461 in kolla "Instance fails to boot: Could not access KVM kernel module: Permission denied" [High,Confirmed] - Assigned to Chason Chan (chen-xing) | 08:58 |
| *** dswebb has joined #openstack-kolla | 09:05 | |
| *** dougsz has joined #openstack-kolla | 09:09 | |
| *** hamzaachi has joined #openstack-kolla | 09:11 | |
| *** msalo has quit IRC | 09:11 | |
| *** msalo has joined #openstack-kolla | 09:14 | |
| hrw | morning | 09:27 |
| cp | hrw: night :) | 09:45 |
| *** jovial[m] has joined #openstack-kolla | 09:45 | |
| hrw | ~ugt | 09:45 |
| hrw | cp: Universal Greeting Time is the convention of greeting people in online chat with "good morning" regardless of the local time of day of either person. | 09:46 |
| cp | hrw: I see :) | 09:46 |
| hrw | http://www.total-knowledge.com/~ilya/mips/ugt.html for more | 09:46 |
| cp | I stand corrected :) | 09:47 |
| yoctozepto | That is no hard rule :D | 09:49 |
| cp | mgoddard: if you could look back in the logs I asked a question regarding the "A universe from nothing" | 09:50 |
| cp | Probably some stupid mistake on my part, but I got stuck half way | 09:50 |
| *** hamzaachi has quit IRC | 09:53 | |
| mgoddard | hi cp, I'll take a look | 09:55 |
| *** rohit02 has quit IRC | 09:55 | |
| *** rohit02 has joined #openstack-kolla | 09:56 | |
| mgoddard | cp: are you using centos? | 09:57 |
| mgoddard | cp: looks like some kvm issue | 09:58 |
| mgoddard | cp: is kvm enabled in the bios? | 09:58 |
| mgoddard | lsmod | egrep 'kvm' | 09:58 |
| mgoddard | and are you running on bare metal or a VM? | 09:58 |
| *** jcmdln has quit IRC | 10:02 | |
| *** jovial[m] has quit IRC | 10:06 | |
| *** lennyb has quit IRC | 10:06 | |
| *** zhanglong has quit IRC | 10:06 | |
| *** pcaruana has joined #openstack-kolla | 10:10 | |
| cp | I am running Centos 7 | 10:10 |
| cp | KVM in the BIOS ? | 10:10 |
| cp | kvm_intel 241664 0 | 10:11 |
| cp | kvm 647168 1 kvm_intel | 10:11 |
| cp | irqbypass 16384 1 kvm | 10:11 |
| cp | I think we're all clear on this level | 10:11 |
| mgoddard | cp: dmesg | grep kvm | 10:11 |
| cp | ok wrong server | 10:12 |
| cp | let me check again | 10:12 |
| *** owalsh has quit IRC | 10:12 | |
| cp | kvm_amd 2177260 3 | 10:12 |
| cp | kvm 632424 1 kvm_amd | 10:12 |
| cp | irqbypass 13503 3 kvm | 10:12 |
| cp | there you go | 10:12 |
| mgoddard | ooh AMD | 10:12 |
| mgoddard | I wonder if that fools us | 10:12 |
| cp | [ 9.818393] kvm: Nested Paging enabled | 10:13 |
| mgoddard | cp: here's the role: https://github.com/stackhpc/ansible-role-libvirt-vm | 10:13 |
| cp | I wonder what will happen when I will do this on my aarch64 machines | 10:14 |
| mgoddard | cp: I doubt you will deploy the seed VM on ARM | 10:16 |
| mgoddard | cp: but if you want to run openstack on them I'm sure you will have some fun | 10:16 |
| mgoddard | hrw is our local ARM expert | 10:16 |
| *** jschlueter has quit IRC | 10:17 | |
| mgoddard | cp: maybe try looking at 'virsh capabilities' | 10:17 |
| cp | mgoddard: why ? | 10:17 |
| cp | Why would I have fun ? | 10:18 |
| mgoddard | it's an x86 world | 10:18 |
| *** hamzaachi has joined #openstack-kolla | 10:18 | |
| mgoddard | it should be possible | 10:18 |
| cp | mgoddard: that is the plan | 10:18 |
| mgoddard | we have support for building arm kolla images | 10:18 |
| *** jschlueter has joined #openstack-kolla | 10:19 | |
| *** owalsh has joined #openstack-kolla | 10:22 | |
| sri_ | hi all, did anyone configured karbor ? my questions is how do we setup Site-A and Site-B replication in karbor. I thought both cluster will talk to same bank for restoring on other site is that right ? or do i need to setup karbor-database replication between two sites ! i know this questions not related to kolla-ansible but incase if anyone shed some light on this... | 10:25 |
| cp | cp: I doubt you will deploy the seed VM on ARM <--- why not ? | 10:45 |
| mgoddard | cp: ok, maybe you will :) | 10:50 |
| mgoddard | I don't think it will work without some code changes though | 10:51 |
| cp | ok, which part of the code do you will require changes ? | 10:51 |
| cp | I'd like to know | 10:51 |
| cp | I guess I can find out myself, but heads up would be appreciated | 10:52 |
| mgoddard | cp: it's hard to say. It (kayobe) hasn't been designed with ARM in mind | 10:54 |
| mgoddard | cp: which hosts do you want to use ARM for? All of them | 10:55 |
| mgoddard | ? | 10:55 |
| cp | it could be all of them yes | 10:58 |
| mgoddard | cp: first of all, you will need to ensure all necessary packages are available for ARM. Hopefully we're not using anything too exotic | 11:00 |
| mgoddard | cp: Bare metal provisioning will be different for ARM | 11:00 |
| mgoddard | cp: what's the use case for the cloud? | 11:01 |
| cp | mgoddard: so the vm seems to be created this time after I rebooted the bare metal machine | 11:03 |
| *** cah_link has joined #openstack-kolla | 11:03 | |
| cp | but this time ssh has such a huge lag it ansible times out | 11:04 |
| *** rohit02 has quit IRC | 11:06 | |
| *** luksky has quit IRC | 11:06 | |
| *** rohit02 has joined #openstack-kolla | 11:07 | |
| *** dougsz has quit IRC | 11:08 | |
| mgoddard | cp: can you ssh in manually? | 11:10 |
| cp | mgoddard: yes | 11:10 |
| mgoddard | to the VM? | 11:10 |
| cp | mgoddard: yes | 11:11 |
| mgoddard | could be a DNS issue | 11:11 |
| mgoddard | that can lead to SSH lag | 11:11 |
| cp | I understand that but why ? | 11:11 |
| cp | The deploy_seed is flaky ? | 11:13 |
| cp | I see things like : | 11:14 |
| cp | # resolves to IP address of API interface for RabbitMQ. This seems to be | 11:14 |
| cp | # required since https://review.openstack.org/#/c/584427 was merged. | 11:14 |
| patchbot | https://review.opendev.org/#/c/584427/ - kolla-ansible - Update rabbitmq to use new conf & clustering (MERGED) - 17 patch sets | 11:14 |
| cp | echo "Workaround: upgrading the controller host" | 11:14 |
| cp | run_kayobe overcloud host upgrade | 11:14 |
| cp | mgoddard: "bare metal provisionning will be different for ARM" , why do you assume it will ? | 11:15 |
| cp | mgoddard: what will make it different ? | 11:16 |
| cp | are you talking about hardware differences ? | 11:16 |
| mgoddard | cp: PXE does not work on ARM. You need to use grub | 11:18 |
| cp | iPXE works on ARM | 11:18 |
| mgoddard | cp: does it? | 11:19 |
| mgoddard | ok | 11:19 |
| cp | Anything else ? | 11:19 |
| cp | I have a UEFI bootloader | 11:19 |
| cp | SBSA compliant | 11:19 |
| mgoddard | well give it a try then :) | 11:20 |
| cp | I have a BMC too that can do IPMI 2.0 and RedFish | 11:20 |
| cp | I will , I just wanted to know if I had forgotten something | 11:20 |
| cp | since you have lot more experience with OpenStack / Kayobe than I currently do | 11:21 |
| hrw | mgoddard: ha ha ha (pxe comment) | 11:21 |
| *** luksky has joined #openstack-kolla | 11:22 | |
| mgoddard | hrw: that was my understanding, guess I'm wrong | 11:22 |
| hrw | mgoddard: proper AArch64 servers are as boring as x86-64 ones. | 11:22 |
| mgoddard | I was going on what I've seen in the ironic docs | 11:22 |
| cp | hrw: not as boring | 11:23 |
| cp | hrw: a lot more open | 11:23 |
| hrw | cp: unpack, put in rack, connect cables, run install - as with x86-64 | 11:23 |
| hrw | that kind of boring | 11:23 |
| cp | ah yes | 11:23 |
| hrw | no 'you need to treat it special, use separate images per server' crap | 11:23 |
| * hrw off for a bit | 11:24 | |
| cp | anyway at the moment I am using my AMD machine | 11:24 |
| cp | and that already does not work | 11:25 |
| mgoddard | https://docs.openstack.org/ironic/latest/install/configure-pxe.html#pxe-multi-architecture-setup | 11:25 |
| cp | yes I had read that | 11:26 |
| cp | it mentions aarch64 | 11:26 |
| cp | among others | 11:26 |
| mnasiadka | mgoddard: if you would have some time today to look at: https://review.opendev.org/#/c/694523/ would be nice. After that we'll remove ONOS and start simplifying Neutron configs - as a base before pushing OVN code to review | 11:31 |
| patchbot | patch 694523 - kolla-ansible - Remove OpenDaylight role - 3 patch sets | 11:31 |
| mnasiadka | s/we'll remove/I'll propose removing/ | 11:34 |
| mnasiadka | ;) | 11:34 |
| *** dougsz has joined #openstack-kolla | 11:47 | |
| *** born2bake has joined #openstack-kolla | 11:51 | |
| mgoddard | mnasiadka: done | 11:52 |
| mgoddard | mnasiadka: reno addition could be a follow up if you like | 11:52 |
| mnasiadka | mgoddard: naah, you're right - let me fix it ;) | 11:53 |
| born2bake | Hi guys, I am wondering if someone had same issue. We have 2 controllers and when we reboot both servers, mariadb cluster is down after reboot. Then I need to run mariadb_recovery twice and after that all services are working fine. Is there any way to eliminate mariadb_recovery usage after reboot? | 11:54 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: Remove OpenDaylight role https://review.opendev.org/694523 | 11:54 |
| mnasiadka | mgoddard: should be better now ^^ :) | 11:55 |
| dougsz | born2bake: 3 nodes, and reboot one at a time | 11:55 |
| born2bake | 3 nodes minimum for mariadb cluster? | 11:55 |
| mnasiadka | 3 nodes minimum for HA in kolla-ansible - it's written in the docs I think | 11:55 |
| yoctozepto | mnasiadka: whitespace in releasenote :< | 11:56 |
| mnasiadka | geez | 11:57 |
| mnasiadka | it's too early | 11:57 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: Remove OpenDaylight role https://review.opendev.org/694523 | 11:57 |
| mnasiadka | yoctozepto: happy? :) | 11:57 |
| yoctozepto | mnasiadka: yea | 11:57 |
| *** dpawlik has quit IRC | 11:58 | |
| yoctozepto | let mgoddard accept it and we are done with odl | 11:58 |
| *** dpawlik has joined #openstack-kolla | 12:01 | |
| *** zhanglong has joined #openstack-kolla | 12:04 | |
| cp | mgoddard: ok so something had modified the /etc/nsswitch.conf | 12:08 |
| cp | and put hosts: file dns myhostname in there | 12:08 |
| yoctozepto | mgoddard, mnasiadka: https://bugs.launchpad.net/kolla-ansible/+bug/1852940 - what's our stance on such mixing? | 12:09 |
| openstack | Launchpad bug 1852940 in kolla-ansible "Instance failed to spawn: libvirtError: invalid argument: could not find capabilities for domaintype=kvm" [Undecided,New] | 12:09 |
| yoctozepto | invalidate bug? | 12:09 |
| cp | same in the VM | 12:10 |
| mgoddard | cp: hmm | 12:10 |
| hrw | mgoddard: https://review.opendev.org/#/c/694099/ waits for you as well. block ussuri development | 12:10 |
| patchbot | patch 694099 - kolla - base: generate en_US.UTF-8 locale data for Debian/... - 6 patch sets | 12:10 |
| mgoddard | hrw: I'll look today | 12:11 |
| hrw | mgoddard: thanks | 12:11 |
| mgoddard | cp: are you using ubuntu or centos on the seed hypervisor? | 12:12 |
| mgoddard | yoctozepto: I don't think we explicitly say we don't support it, and it sounds like it has worked previously | 12:14 |
| cp | using centos 7 on host and vm | 12:14 |
| mgoddard | but it's not something we're going to test upstream or care about too much | 12:14 |
| cp | host = bare metal vm = seed | 12:14 |
| *** born2bake has quit IRC | 12:15 | |
| mgoddard | cp: do you have working DNS in the VM? | 12:17 |
| cp | Ok so in the documentation there is no mention I have to do any of this | 12:17 |
| cp | mgoddard: some instructions are missing >? | 12:18 |
| mgoddard | any of what? | 12:18 |
| cp | mgoddard: touching anythying concerning the VM | 12:18 |
| mgoddard | the instructions have been tested on a few x86 machines | 12:18 |
| cp | I'd like to know why I had `myhostname` in the /etc/nsswitch.conf to start with | 12:19 |
| cp | it is obviously kayobe scripts that did that | 12:20 |
| cp | since it was in both the host and the VM | 12:20 |
| cp | when I fixed that , the timeout went away ( of course ) | 12:20 |
| cp | now I have a new error | 12:21 |
| cp | fatal: [seed]: FAILED! => {"changed": false, "msg": "Failure talking to yum: Cannot find a valid baseurl for repo: base/7/x86_64"} | 12:21 |
| cp | to retry, use: --limit @/home/stack/kayobe-venv/share/kayobe/ansible/kayobe-target-venv.retry | 12:21 |
| mgoddard | I just checked one of my machines, I have myhostname in nsswitch and it doesn't cause a problem | 12:21 |
| *** kplant has joined #openstack-kolla | 12:23 | |
| mgoddard | cp: fresh CentOS 7.6 cloud image also has it | 12:23 |
| mgoddard | so ask centos | 12:24 |
| cp | right I just checked that | 12:24 |
| cp | but anyway not having it | 12:24 |
| mgoddard | it has never been a problem for us | 12:24 |
| cp | Ok then I do not know | 12:24 |
| mgoddard | I imagine there is something else going on | 12:24 |
| cp | Probably | 12:25 |
| *** schwicht has quit IRC | 12:26 | |
| yoctozepto | myhostname is in vanilla centos install | 12:30 |
| cp | yeah | 12:30 |
| cp | but somehow it causes a problem at login | 12:31 |
| cp | slowing login | 12:31 |
| cp | I am not familiar with centos | 12:31 |
| yoctozepto | it should just respond with own ip address when asked about own hostname | 12:31 |
| mgoddard | again, is DNS working? | 12:32 |
| mgoddard | cp: ^ | 12:32 |
| cp | mgoddard: in the VM ? | 12:34 |
| mgoddard | cp: yeah, or the hoost | 12:35 |
| cp | host yes | 12:35 |
| cp | seed vm --> DNS does not work | 12:37 |
| kplant | what's the output of: getent ahosts $(hostname) | 12:38 |
| kplant | on the vm | 12:38 |
| cp | probably the instructions if rerunned break something ? | 12:38 |
| cp | [centos@seed ~]$ getent ahosts $(hostname) | 12:38 |
| cp | fe80::5054:ff:fe52:bd97 STREAM seed | 12:38 |
| cp | fe80::5054:ff:fe52:bd97 DGRAM | 12:38 |
| cp | fe80::5054:ff:fe52:bd97 RAW | 12:38 |
| cp | 192.168.33.5 STREAM | 12:38 |
| cp | 192.168.33.5 DGRAM | 12:38 |
| cp | 192.168.33.5 RAW | 12:38 |
| mgoddard | cp: maybe your configure-local-networking.sh script failed? | 12:39 |
| mgoddard | there should be NAT configured on the host | 12:39 |
| cp | should I rerun it ? | 12:39 |
| kplant | i don't see how nss-myhostname would be a problem here, seems like it's grabbing from /etc/hostname just fine | 12:40 |
| cp | net.ipv4.conf.all.forwarding = 1 | 12:41 |
| cp | RTNETLINK answers: File exists | 12:41 |
| *** luksky has quit IRC | 12:47 | |
| *** kplant has quit IRC | 12:50 | |
| *** kplant has joined #openstack-kolla | 13:02 | |
| *** zhanglong has quit IRC | 13:02 | |
| *** zhanglong has joined #openstack-kolla | 13:07 | |
| *** lennyb has joined #openstack-kolla | 13:09 | |
| *** zhanglong has quit IRC | 13:13 | |
| yoctozepto | mgoddard, mnasiadka: https://launchpad.net/kolla-ansible/+milestone/9.0.0 | 13:22 |
| yoctozepto | oh so many pending ;-( | 13:22 |
| mnasiadka | uh | 13:24 |
| mnasiadka | yoctozepto: https://bugs.launchpad.net/kolla-ansible/+bug/1846820 - I haven't seen that in the CI for some time | 13:25 |
| openstack | Launchpad bug 1846820 in kolla-ansible "nova-conductor may crash during deploy due to haproxy-keystone 504" [Medium,Triaged] | 13:25 |
| yoctozepto | I saw it | 13:25 |
| yoctozepto | but it's not mandatory to pin it to the milestone | 13:25 |
| yoctozepto | most likely there is something in that CI | 13:25 |
| yoctozepto | the issue is general | 13:26 |
| mnasiadka | so that means we could even set it to low | 13:26 |
| yoctozepto | we lose vip addr from time to time | 13:26 |
| yoctozepto | haproxy timeouts | 13:26 |
| yoctozepto | etc. | 13:26 |
| mnasiadka | I saw a couple of read timeout in kolla_docker | 13:26 |
| mnasiadka | if we're on timeouts :D | 13:26 |
| yoctozepto | yeah, there is something in general, never had these issues locally on moderate nodes | 13:26 |
| yoctozepto | so there must be some aggressive cpu time sharing or something... | 13:26 |
| yoctozepto | I will low it and unpin | 13:27 |
| yoctozepto | no need to spend time now | 13:27 |
| *** zhanglong has joined #openstack-kolla | 13:29 | |
| mnasiadka | yoctozepto: so if we count out that, and all starting with CI: - then it's not a lot left ;) | 13:31 |
| yoctozepto | mnasiadka: yeah, that's what keeps me uplifted | 13:31 |
| *** luksky has joined #openstack-kolla | 13:39 | |
| *** KeithMnemonic has joined #openstack-kolla | 13:40 | |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: CI: Reconfigure deployment after checking health https://review.opendev.org/694771 | 13:41 |
| yoctozepto | hopefully not b0rken ;D | 13:41 |
| hrw | yoctozepto: 'Stop merging broken code'? :D | 13:47 |
| yoctozepto | hrw: ;D | 13:47 |
| hrw | to those not on beginning of vPTG call - I gave yoctozepto stickers with this phrase | 13:48 |
| kplant | oof | 13:48 |
| kplant | that's rough | 13:48 |
| hrw | he also got Kolla logo stickers | 13:49 |
| *** yan0s has joined #openstack-kolla | 13:52 | |
| *** zhanglong has quit IRC | 13:55 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/rocky: Collect rabbitmq logs by fluentd https://review.opendev.org/694275 | 13:56 |
| *** zhanglong has joined #openstack-kolla | 13:57 | |
| yoctozepto | kplant: nah, I like the stickers, I think I showed them to camera when mnasiadka was speaking ;D | 13:58 |
| yoctozepto | AND THAT WAS ROUGH | 13:58 |
| hrw | we did | 13:58 |
| kplant | i would like to place an order please | 13:59 |
| kplant | 1,000 stickers | 13:59 |
| kplant | the NPCs here could use the advice | 13:59 |
| mgoddard | or for anyone who disklikes CI: "Stop slowing down the merge of my amazing code" | 14:02 |
| yoctozepto | ah yeah, hrw was holding the merging sticker | 14:02 |
| yoctozepto | and I was holding the koala | 14:02 |
| yoctozepto | mgoddard: sure, because it is always amazing | 14:02 |
| mgoddard | yoctozepto: thank you | 14:03 |
| yoctozepto | mgoddard: np | 14:03 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: DNM: CI: Break etcd in zun https://review.opendev.org/694778 | 14:07 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: CI: Add ansible-lint to tox https://review.opendev.org/694779 | 14:08 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: DNM: CI: Break etcd in zun https://review.opendev.org/694778 | 14:09 |
| yoctozepto | mnasiadka: did you run ansible-lint? | 14:11 |
| mnasiadka | yoctozepto: yeah, looks awful :) | 14:11 |
| yoctozepto | or just slapped it in there so that CI can sweat a bit? ;D | 14:11 |
| yoctozepto | yeah, I was afraid | 14:11 |
| mnasiadka | at least we know how many local_action's do we have in the code :) | 14:12 |
| *** rohit02 has quit IRC | 14:12 | |
| yoctozepto | always look at the bright side of life *whistles* | 14:12 |
| *** lennyb has quit IRC | 14:14 | |
| yoctozepto | mnasiadka: I think we lost at least bashate in ci now | 14:14 |
| mnasiadka | nope, we did not - https://review.opendev.org/#/c/694779/1/zuul.d/project.yaml - the change here should do tox -e linters | 14:15 |
| patchbot | patch 694779 - kolla-ansible - WIP: CI: Add ansible-lint to tox - 1 patch set | 14:15 |
| mnasiadka | yoctozepto: see here: http://zuul.openstack.org/stream/59a4a00f84bb420c84bcd04ed1fac3eb?logfile=console.log | 14:16 |
| mnasiadka | it's running | 14:16 |
| yoctozepto | mnasiadka: which job is that? | 14:17 |
| mnasiadka | openstack-tox-linters | 14:17 |
| mnasiadka | yoctozepto: enabling ansible-role-jobs in project.yml gives us this: https://github.com/openstack/openstack-zuul-jobs/blob/7fce37a849ece42e3a5fb7b0943b4583e9dbfedc/zuul.d/project-templates.yaml#L1575 | 14:18 |
| yoctozepto | mhm, tox-linters, nice | 14:18 |
| mnasiadka | ok, I'll need to fix other jobs (like pep8 which now misses doc/requirements.txt), but let's wait for final zuul results ;) | 14:20 |
| *** schwicht has joined #openstack-kolla | 14:26 | |
| tumble | as far as I understand https://docs.openstack.org/security-guide/networking/architecture.html I need a neutron l3 agent on every compute node if I want to make it accessible from an external network. Is that true? Because only network nodes get this agent in kolla-ansible and I'm still wondering if every compute node must be a network node then. :/ | 14:30 |
| mnasiadka | tumble: you linked to a picture, where l3 agent is not on the compute node | 14:33 |
| *** zhanglong has quit IRC | 14:34 | |
| yoctozepto | You do not need l3 agent on computes | 14:37 |
| yoctozepto | Unless you want to run neutron dvr | 14:37 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: CI: Add ansible-lint to tox https://review.opendev.org/694779 | 14:38 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: CI: Add ansible-lint to tox https://review.opendev.org/694779 | 14:38 |
| *** goldyfruit has joined #openstack-kolla | 14:40 | |
| openstackgerrit | Viktor Michalek proposed openstack/kolla-ansible master: Vitrage: typo in precheck task https://review.opendev.org/694788 | 14:41 |
| hrw | mnasiadka: +1 so far | 14:42 |
| mnasiadka | hrw: sure, it will be a long way tox fix all things that ansible-lint complains about ;) | 14:43 |
| mgoddard | mnasiadka: are there any genuine issues it complains about? | 14:43 |
| mnasiadka | mgoddard: not really, in most cases it's just local_action vs delegate_to, command:/shell: that don't do changed_when, and so on | 14:44 |
| *** sri_ has quit IRC | 14:44 | |
| mgoddard | mnasiadka: that was my experience | 14:44 |
| mnasiadka | mgoddard: there is one that probably we could skip - it checks for galaxy_info in meta | 14:44 |
| *** dougsz has quit IRC | 14:45 | |
| hrw | mnasiadka: and skip '160 chars reached'? | 14:50 |
| mnasiadka | hrw: yeah, that's actually my bad in the common role :) | 14:50 |
| *** goldyfruit_ has joined #openstack-kolla | 14:51 | |
| hrw | mnasiadka: 2>/dev/null? | 14:52 |
| *** goldyfruit has quit IRC | 14:53 | |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: CI: Add ansible-lint to tox https://review.opendev.org/694779 | 14:53 |
| mnasiadka | hrw: I think it only happens once, so it's easy to fix it in one place - but we can of course skip it if we prefer - we can talk about it on the meeting on Wed | 14:54 |
| hrw | mnasiadka: commented and will add one more comment in a moment | 14:57 |
| mnasiadka | sure, more the merrier | 14:57 |
| hrw | done | 14:58 |
| *** Luzi has quit IRC | 14:59 | |
| *** dougsz has joined #openstack-kolla | 15:01 | |
| *** vmixor has joined #openstack-kolla | 15:04 | |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: CI: Add ansible-lint to tox https://review.opendev.org/694779 | 15:05 |
| mnasiadka | hrw: implemented ;) | 15:05 |
| hrw | mnasiadka: s/'/" left | 15:06 |
| mnasiadka | ugh | 15:06 |
| hrw | mnasiadka: I dropped whole echo line locally | 15:06 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: CI: Add ansible-lint to tox https://review.opendev.org/694779 | 15:07 |
| mnasiadka | why not, if we feel that we need it - we can add it later in some sort | 15:07 |
| mnasiadka | dropped | 15:07 |
| yoctozepto | Drop, drop | 15:10 |
| hrw | (venv3) 16:05 (258s) hrw@puchatek:kolla-ansible$ ./tools/ansible-lint.sh | 15:11 |
| hrw | /home/hrw/devel/linaro/kolla-ansible/ansible/roles/baremetal/tasks/post-install.yml:168: [E503] Tasks that run when changed should likely be handlers | 15:11 |
| hrw | easy output | 15:11 |
| *** dpawlik has quit IRC | 15:14 | |
| hrw | mnasiadka: does https://paste.centos.org/view/3406e4cc look good? | 15:18 |
| mnasiadka | yup | 15:18 |
| mnasiadka | maybe the docker and pip version should be a variable to easily change it/override it? | 15:19 |
| hrw | mnasiadka: took docker from requirements of kolla-ansible and pip 19.3 is iirc first one to complain about py2 | 15:20 |
| mnasiadka | ok, makes sense | 15:20 |
| hrw | mnasiadka: if patch makes sense then I will write proper commit message and send to gerrit | 15:21 |
| tumble | does this ring a bell to anyone? "neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent RemoteError: Remote error: InvalidTargetVersion Invalid target version 1.5" It seems to be the source of the network issues on one of my compute nodes | 15:23 |
| mnasiadka | hrw: it does, but it's tip of the iceberg looking at the ansible-lint output - do we want to do a patch series with depends-on, or how do we want to manage it? | 15:27 |
| mnasiadka | for sure we should start with cleaning up common, because it's getting linted with each role due to dependency in meta | 15:29 |
| openstackgerrit | Merged openstack/kolla-ansible master: Remove OpenDaylight role https://review.opendev.org/694523 | 15:30 |
| openstackgerrit | Merged openstack/kolla-ansible master: Attempt to pull image before stopping and removing container https://review.opendev.org/694278 | 15:39 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: Remove Neutron integration with ONOS https://review.opendev.org/694796 | 15:39 |
| yoctozepto | Oh, a gift :D | 15:41 |
| hrw | mnasiadka: have to start with something. does not have to be in any patch series imho | 15:43 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla-ansible master: Some ansible-lint fixes https://review.opendev.org/694797 | 15:45 |
| *** luksky has quit IRC | 15:59 | |
| openstackgerrit | Dincer Celik proposed openstack/kolla master: WIP: Upgrade RabbitMQ to 3.8 https://review.opendev.org/694799 | 15:59 |
| *** hamzy has quit IRC | 16:01 | |
| openstackgerrit | Dincer Celik proposed openstack/kolla master: WIP: Upgrade RabbitMQ to 3.8 https://review.opendev.org/694799 | 16:04 |
| *** hamzaachi has quit IRC | 16:12 | |
| *** hamzaachi has joined #openstack-kolla | 16:14 | |
| *** hamzaachi_ has joined #openstack-kolla | 16:30 | |
| *** hamzaachi has quit IRC | 16:33 | |
| openstackgerrit | Merged openstack/kolla-ansible stable/rocky: Collect rabbitmq logs by fluentd https://review.opendev.org/694275 | 16:34 |
| *** skramaja has quit IRC | 16:43 | |
| *** hamzy has joined #openstack-kolla | 16:51 | |
| *** hamzaachi_ has quit IRC | 16:59 | |
| *** yan0s has quit IRC | 17:02 | |
| *** luksky has joined #openstack-kolla | 17:02 | |
| *** rpittau is now known as rpittau|afk | 17:18 | |
| yoctozepto | ubuntu building is b0rken | 17:19 |
| bline | yoctozepto: It was centos images; I'm giving the ubuntu images a try right now | 17:19 |
| yoctozepto | due some libssl conflict between it and its dev package | 17:19 |
| yoctozepto | due to* | 17:19 |
| yoctozepto | bline: yeah, I read the bug report later :-) | 17:20 |
| mnasiadka | yoctozepto: it's a mirror issue, or ubuntu broke something? :) | 17:20 |
| yoctozepto | looks like mirror issue, checking upstream | 17:21 |
| *** hamzy has quit IRC | 17:21 | |
| yoctozepto | upstream looks fine eye-verified | 17:21 |
| *** hamzy has joined #openstack-kolla | 17:22 | |
| *** gfidente is now known as gfidente|afk | 17:25 | |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible stable/train: Attempt to pull image before stopping and removing container https://review.opendev.org/694812 | 17:26 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible stable/stein: Attempt to pull image before stopping and removing container https://review.opendev.org/694813 | 17:26 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible stable/rocky: Attempt to pull image before stopping and removing container https://review.opendev.org/694814 | 17:26 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: Fix local_action and line_length>160 in common role https://review.opendev.org/694815 | 17:27 |
| yoctozepto | mnasiadka https://review.opendev.org/694666 | 17:27 |
| patchbot | patch 694666 - kolla-ansible - Use internal API for heat -> keystone communication - 2 patch sets | 17:27 |
| mnasiadka | yoctozepto: rolling | 17:27 |
| yoctozepto | danke schon | 17:28 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: Fix local_action and line_length>160 in common role https://review.opendev.org/694815 | 17:32 |
| yoctozepto | testing new irc client - can someone mention me? | 17:36 |
| mnasiadka | yoctozepto: | 17:37 |
| *** Chaserjim has joined #openstack-kolla | 17:37 | |
| yoctozepto | Thanks mnasiadka, works like a charm | 17:38 |
| bline | ubuntu images fail in a different way: Instance failed to spawn: libvirt.libvirtError: internal error: process exited while connecting to monitor: Could not access KVM kernel module: Permission denied | 17:51 |
| bline | going to dig into that more | 17:51 |
| yoctozepto | bline: you can share new details in the bug report | 17:55 |
| bline | sure, will do that | 17:56 |
| yoctozepto | thanks, it keeps it organized better | 17:57 |
| bline | right now I'm going to try and change the udev rules for /dev/kvm | 17:57 |
| openstackgerrit | Merged openstack/kolla-ansible master: Vitrage: typo in precheck task https://review.opendev.org/694788 | 17:58 |
| yoctozepto | ok, I cleaned up the issues for train milestone | 18:01 |
| yoctozepto | 2 open issues remain | 18:01 |
| yoctozepto | one ironic: https://bugs.launchpad.net/kolla-ansible/+bug/1847054 mgoddard | one masakari: https://bugs.launchpad.net/kolla-ansible/+bug/1847050 goldyfruit_ | 18:01 |
| openstack | Launchpad bug 1847054 in kolla-ansible "CI: nova-compute-ironic reports errors in the ironic scenario" [Medium,Triaged] | 18:01 |
| openstack | Launchpad bug 1847050 in kolla-ansible "CI: masakari-instancemonitor reports an error" [Medium,Triaged] | 18:01 |
| *** hamzaachi_ has joined #openstack-kolla | 18:02 | |
| yoctozepto | the others are in reviews now | 18:02 |
| bline | anyone know what group /dev/kvm should belong to for nova_libvirtd to access it? | 18:05 |
| *** hamzaachi_ has quit IRC | 18:09 | |
| *** hamzaachi_ has joined #openstack-kolla | 18:10 | |
| yoctozepto | well, the defaults should be fine; start with 777 to see if the problem is indeed with permissions | 18:10 |
| kplant | [root@chrnc-dev-os-compute-01 ~]# ls -lah /dev/kvm | 18:11 |
| kplant | crw-rw----. 1 root 42427 10, 232 Nov 13 18:53 /dev/kvm | 18:11 |
| yoctozepto | yeah, 42427 looks like the right group for kolla qemu | 18:12 |
| yoctozepto | so it acted upon properly hmm | 18:13 |
| bline | thanks yoctozepto, it was 660, I changed it to 666 and it's working | 18:13 |
| yoctozepto | wait, wait | 18:13 |
| yoctozepto | did you disable the host's libvirt? | 18:13 |
| bline | yes | 18:13 |
| yoctozepto | hmm | 18:13 |
| yoctozepto | then by all means it should work with previous permissions | 18:14 |
| bline | so now I just need to figure out what group permissions I need, it was root:kvm, I tried root:libvirt-qemu with no success | 18:14 |
| yoctozepto | could be they changed something recently in ubuntu/docker that broke the permission flow | 18:14 |
| yoctozepto | nah, the number there was fine | 18:15 |
| yoctozepto | can't get better than that | 18:15 |
| yoctozepto | need some ubuntu user to try that out | 18:15 |
| yoctozepto | hrw: ^ ubuntu client | 18:15 |
| yoctozepto | serve them well :D | 18:15 |
| bline | I'll just live with 666 for now; will update the ticket | 18:15 |
| *** hamzaachi_ has quit IRC | 18:16 | |
| yoctozepto | yeah, thanks, not much that comes to mind | 18:16 |
| *** hamzaachi_ has joined #openstack-kolla | 18:16 | |
| bline | these servers are all on an internal network so it should be safe | 18:16 |
| yoctozepto | you could check what user/group the libvirt container process started with | 18:16 |
| yoctozepto | yeah, it is locally either way | 18:16 |
| yoctozepto | so only other local services can hijack libvirt now | 18:17 |
| bline | ticket updated | 18:21 |
| *** msalo has quit IRC | 18:24 | |
| yoctozepto | mgoddard, mnasiadka: https://bugs.launchpad.net/kolla-ansible/+bug/1853041 this was unexpected | 18:34 |
| openstack | Launchpad bug 1853041 in kolla-ansible "enabling ironic reconfigures physnet1 as vlan and all as flat" [Undecided,Triaged] | 18:34 |
| *** hjensas has quit IRC | 18:36 | |
| bline | ugg, I just noticed there is no magnum | 18:38 |
| yoctozepto | no magnum? | 18:38 |
| bline | atleast not listed in horizon | 18:38 |
| yoctozepto | ah | 18:38 |
| bline | wonder if that's why I moved to centos images before lol | 18:39 |
| bline | deployment log shows it installing the magnum images.. | 18:41 |
| openstackgerrit | Viktor Michalek proposed openstack/kolla master: Designate: pool-manager has been removed https://review.opendev.org/694617 | 18:41 |
| kplant | do you see magnum running on the control node(s) | 18:44 |
| bline | yes, both api and conductor | 18:44 |
| *** igordc has joined #openstack-kolla | 18:46 | |
| bline | both control nodes show them running | 18:46 |
| kplant | does the cli work? | 18:47 |
| kplant | something like 'openstack coe cluster list' | 18:47 |
| openstackgerrit | Viktor Michalek proposed openstack/kolla-ansible stable/train: Vitrage: typo in precheck task https://review.opendev.org/694830 | 18:50 |
| bline | no coe from openstack cli | 18:51 |
| *** gouthamr_ is now known as gouthamr | 18:51 | |
| bline | no errors in magnum log file, just a couple of warnings | 18:52 |
| bline | WARNING keystonemiddleware.auth_token [-] AuthToken middleware is set with keystone_authtoken.service_token_roles_required set to False. This is backwards compatible but deprecated behaviour. Please set this to True. | 18:53 |
| bline | and WARNING oslo_log.versionutils [req-8dfe17b3-e5a9-4567-a481-d9eaee2fdf80 - - - - -] Deprecated: _send_cluster_metrics() is deprecated as of Rocky and may be removed in Train. It will not be superseded. | 18:53 |
| *** msalo has joined #openstack-kolla | 19:00 | |
| *** hjensas has joined #openstack-kolla | 19:11 | |
| openstackgerrit | Dincer Celik proposed openstack/kolla master: WIP: Upgrade RabbitMQ to 3.8 https://review.opendev.org/694799 | 19:18 |
| *** Chaserjim has quit IRC | 19:19 | |
| *** Chaserjim has joined #openstack-kolla | 19:20 | |
| *** dpawlik has joined #openstack-kolla | 19:26 | |
| *** cah_link has quit IRC | 19:39 | |
| kplant | how about openstack endpoint list --service magnum | 19:59 |
| yoctozepto | bline, you need to install cli package to have coe commands | 19:59 |
| kplant | ^ | 19:59 |
| yoctozepto | Not sure about its name though | 19:59 |
| yoctozepto | Those warnings are irrelevant | 20:00 |
| kplant | python-magumclient | 20:00 |
| kplant | in pypi | 20:00 |
| yoctozepto | Yeah | 20:00 |
| kplant | python-magnumclient * | 20:00 |
| yoctozepto | Makes sense | 20:00 |
| yoctozepto | Yeah, I read that right xD | 20:00 |
| bline | openstack endpoint list --service magnum; lists 3 endpoints. python-magnumclient is installed; openstack coe --help: Unknown command [u'coe'] | 20:20 |
| kplant | i am confuse | 20:20 |
| kplant | what release of openstack is this? | 20:20 |
| kplant | what about 'magnum cluster-list' | 20:21 |
| *** dpawlik has quit IRC | 20:22 | |
| bline | ImportError: No module named magnumclient.shell | 20:22 |
| bline | I installed openstack cli via pip2 install python-openstackclient | 20:23 |
| bline | I couldn't get it working with python3 | 20:23 |
| kplant | and you installed python-magnumclient with pip2? | 20:23 |
| bline | the openstack images are ubuntu/train | 20:24 |
| bline | yeah | 20:24 |
| kplant | oh | 20:24 |
| kplant | i don't think train is stable yet? for k-a | 20:24 |
| bline | hmm | 20:24 |
| bline | I only moved to train because I couldn't ever get magnum working with stein | 20:24 |
| *** hamzaachi_ has quit IRC | 20:26 | |
| *** dpawlik has joined #openstack-kolla | 20:27 | |
| * bline gives stein another try | 20:36 | |
| *** kplant has quit IRC | 20:38 | |
| *** dpawlik has quit IRC | 20:41 | |
| *** dpawlik has joined #openstack-kolla | 20:44 | |
| yoctozepto | Train is stable | 20:48 |
| yoctozepto | Just not released | 20:48 |
| stingrayza | anybody by chance at kubecon (san diego) ? I'd love to say hi... | 20:48 |
| yoctozepto | You need magnum client for the py version used by openstack client | 20:49 |
| *** schwicht has quit IRC | 20:50 | |
| *** goldyfruit_ has quit IRC | 21:02 | |
| *** dougsz has quit IRC | 21:05 | |
| *** goldyfruit has joined #openstack-kolla | 21:17 | |
| *** dpawlik has quit IRC | 21:29 | |
| *** schwicht has joined #openstack-kolla | 21:31 | |
| *** dpawlik has joined #openstack-kolla | 21:43 | |
| *** schwicht has quit IRC | 21:48 | |
| *** hamzaachi has joined #openstack-kolla | 21:52 | |
| *** hamzaachi has quit IRC | 21:53 | |
| *** hamzaachi has joined #openstack-kolla | 21:54 | |
| *** hamzaachi has quit IRC | 21:55 | |
| *** hamzaachi has joined #openstack-kolla | 21:55 | |
| *** hamzaachi_ has joined #openstack-kolla | 22:06 | |
| *** hamzaachi has quit IRC | 22:07 | |
| *** msalo has quit IRC | 22:08 | |
| *** dpawlik has quit IRC | 22:08 | |
| *** schwicht has joined #openstack-kolla | 22:24 | |
| *** pcaruana has quit IRC | 22:26 | |
| *** schwicht has quit IRC | 22:26 | |
| *** hamzaachi_ has quit IRC | 22:38 | |
| *** msalo has joined #openstack-kolla | 22:46 | |
| *** msalo has quit IRC | 22:51 | |
| *** luksky has quit IRC | 23:02 | |
| *** ivve has quit IRC | 23:26 | |
| *** vrdel has quit IRC | 23:34 | |
| *** goldyfruit has quit IRC | 23:36 | |
| *** zhanglong has joined #openstack-kolla | 23:47 | |
| *** gfidente|afk has quit IRC | 23:50 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!