| *** hjensas has quit IRC | 00:15 | |
| *** hjensas has joined #openstack-kolla | 00:17 | |
| *** factor has joined #openstack-kolla | 00:20 | |
| *** cdearborn has quit IRC | 00:24 | |
| *** boilami has joined #openstack-kolla | 00:45 | |
| *** dmsimard1 has joined #openstack-kolla | 00:50 | |
| *** dmsimard has quit IRC | 00:51 | |
| *** dmsimard1 is now known as dmsimard | 00:52 | |
| *** zhanglong has joined #openstack-kolla | 01:20 | |
| *** zhanglong has quit IRC | 01:22 | |
| *** zhanglong has joined #openstack-kolla | 01:27 | |
| *** oncall-pokemon has joined #openstack-kolla | 01:28 | |
| *** oncall-pokemon has quit IRC | 01:32 | |
| *** oncall-pokemon has joined #openstack-kolla | 01:33 | |
| *** boilami has quit IRC | 02:05 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible master: Manage nova scheduler workers count https://review.opendev.org/721198 | 02:17 |
|---|---|---|
| *** jcmdln has quit IRC | 02:49 | |
| *** factor has quit IRC | 02:49 | |
| *** hongbin has joined #openstack-kolla | 03:00 | |
| *** factor has joined #openstack-kolla | 03:15 | |
| *** k_mouza has joined #openstack-kolla | 04:17 | |
| *** srin has joined #openstack-kolla | 04:19 | |
| *** k_mouza has quit IRC | 04:21 | |
| *** ricolin has joined #openstack-kolla | 04:31 | |
| *** evrardjp has quit IRC | 04:35 | |
| *** evrardjp has joined #openstack-kolla | 04:35 | |
| *** hongbin has quit IRC | 04:35 | |
| *** skramaja has joined #openstack-kolla | 04:43 | |
| *** ykarel|away is now known as ykarel | 04:45 | |
| *** zhanglong has quit IRC | 04:48 | |
| *** ricolin has quit IRC | 04:52 | |
| *** zhanglong has joined #openstack-kolla | 04:52 | |
| *** ricolin has joined #openstack-kolla | 04:56 | |
| *** cah_link has joined #openstack-kolla | 04:57 | |
| *** ricolin_ has joined #openstack-kolla | 05:06 | |
| *** ricolin has quit IRC | 05:09 | |
| *** vishalmanchanda has joined #openstack-kolla | 05:09 | |
| *** cah_link has quit IRC | 05:15 | |
| *** ykarel is now known as ykarel|afk | 05:16 | |
| srin | hi, I have some questions | 05:19 |
| srin | I installed Openstack via Kolla-ansible and I can't ssh/ping to any Openstack network | 05:21 |
| srin | external network is 172.24.202.0/24 and management network is 10.0.0.0/24 | 05:23 |
| srin | https://pastebin.com/QKaBA5Wc | 05:25 |
| srin | this is my multinode setup | 05:25 |
| srin | https://pastebin.com/psjw1ZmM | 05:27 |
| srin | and this is my /etc/kolla/globals.yml setup | 05:27 |
| srin | baremetals os are centos 7 | 05:30 |
| *** ykarel|afk is now known as ykarel | 05:33 | |
| sri_ | morning | 05:34 |
| sri_ | srin: hi, where are trying to reach these network's from ? and I see you have three nic's which nic has the default gateway ? | 05:37 |
| srin | from controller0(172.24.202.11, 172.24.202.250, 10.0.0.11, 10.0.0.250) and my pc(172.24.200.xxx) | 05:42 |
| srin | controller0's default gateway is em1 | 05:42 |
| srin | is > has | 05:43 |
| srin | em1 has default gateway | 05:43 |
| srin | em2 and em3 hasn't gateway | 05:43 |
| sri_ | srin: Ok, can you reach the em1 network ? | 05:45 |
| sri_ | I mea | 05:45 |
| srin | yes, I always ssh'ing to 172.24.202.11 | 05:45 |
| sri_ | Ok that's looks good. | 05:47 |
| srin | https://pastebin.com/gmq0XvdT | 05:47 |
| srin | this is my interfaces setup | 05:47 |
| *** cah_link has joined #openstack-kolla | 05:48 | |
| sri_ | when you ping the 10.0.0.x network from your desktop, can you check any packets reaching the em2 nic | 05:52 |
| srin | no i cant | 05:52 |
| srin | management network has dedicated switch | 05:52 |
| srin | and has no internet | 05:53 |
| srin | It just connected to servers via em2 interfaces | 05:54 |
| srin | controller0, compute0~2's em2 interfaces | 05:54 |
| srin | em3 is only for controller0 | 05:56 |
| srin | and it is connected to external network's switch | 05:56 |
| sri_ | srin, hmm, I think it something has do network configuration on your switch side, I am not a networking guy. :( | 05:57 |
| srin | yes I think too.. and also I am not good at networking things | 05:57 |
| srin | thank you anyways :) | 05:58 |
| srin | I think my settings are not weird but I don't know why I can't connect or ping openstack network | 05:59 |
| sri_ | hmm, let wait if someone can help you to fix your issue. | 06:00 |
| srin | oh | 06:01 |
| srin | thank you for your kindness | 06:01 |
| sri_ | np :) | 06:05 |
| yoctozepto | morning | 06:16 |
| *** gfidente|afk is now known as gfidente | 06:46 | |
| osmanlicilegi | morning | 06:59 |
| *** rpittau|afk is now known as rpittau | 07:03 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Fix monasca deployment due to monasca_log_dir https://review.opendev.org/709092 | 07:05 |
| openstackgerrit | Merged openstack/kolla-ansible stable/train: [horizon] Move 'balance' HAProxy keyword https://review.opendev.org/720516 | 07:05 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: kolla-toolbox: use only binary packages https://review.opendev.org/721252 | 07:27 |
| mnasiadka | morning | 07:31 |
| *** bengates has joined #openstack-kolla | 07:32 | |
| *** bengates has quit IRC | 07:32 | |
| *** bengates has joined #openstack-kolla | 07:33 | |
| openstackgerrit | Merged openstack/kolla-ansible stable/stein: [horizon] Move 'balance' HAProxy keyword https://review.opendev.org/720517 | 07:35 |
| openstackgerrit | Merged openstack/kolla-ansible stable/train: [haproxy-config] Fix missing servers in split cfg https://review.opendev.org/720472 | 07:35 |
| openstackgerrit | Merged openstack/kolla-ansible stable/stein: [haproxy-config] Fix missing servers in split cfg https://review.opendev.org/720473 | 07:35 |
| *** ricolin_ has quit IRC | 07:47 | |
| *** ricolin has joined #openstack-kolla | 07:47 | |
| *** threestrands has quit IRC | 07:55 | |
| *** saneax has joined #openstack-kolla | 07:59 | |
| *** JangwonLee_ has joined #openstack-kolla | 08:00 | |
| *** lennyb has joined #openstack-kolla | 08:04 | |
| *** KeithMnemonic has quit IRC | 08:05 | |
| *** e0ne has joined #openstack-kolla | 08:08 | |
| *** born2bake has joined #openstack-kolla | 08:27 | |
| mgoddard | morning | 08:37 |
| *** skramaja has quit IRC | 08:43 | |
| *** priteau has joined #openstack-kolla | 08:45 | |
| *** skramaja has joined #openstack-kolla | 08:48 | |
| cosmicsound | morning mgoddard | 08:49 |
| hrw | lo | 08:50 |
| srin | hmm | 08:51 |
| srin | I reinstalled few times and still cannot reachable | 08:52 |
| *** JangwonLee__ has joined #openstack-kolla | 08:56 | |
| hrw | srin: you can reach controller, right? | 08:58 |
| hrw | srin: no plans to use floating ips in controller/pc network? | 08:59 |
| mgoddard | yoctozepto: replied to your proposal :) | 08:59 |
| *** JangwonLee_ has quit IRC | 09:00 | |
| *** k_mouza has joined #openstack-kolla | 09:02 | |
| *** e0ne has quit IRC | 09:03 | |
| *** e0ne has joined #openstack-kolla | 09:03 | |
| srin | hrw: I owned 172.24.202.0/24 subnet from my company network manager | 09:04 |
| srin | and also I created some floating ips and attach to vms | 09:05 |
| srin | added security group and allow all ports and icmp | 09:06 |
| srin | in vms, can ping and ssh each vms | 09:06 |
| srin | but it can't ping router's external interface(172.24.202.xxx) | 09:07 |
| srin | and controller0 pc too | 09:07 |
| *** zhanglong has quit IRC | 09:07 | |
| srin | 172.24.202.11, 172.24.202.250, 10.0.0.11, 10.0.0.250 | 09:08 |
| *** vishalmanchanda has quit IRC | 09:09 | |
| * hrw out of ideas | 09:09 | |
| srin | same | 09:12 |
| srin | :( | 09:12 |
| *** ykarel is now known as ykarel|lunch | 09:14 | |
| openstackgerrit | ya.wang proposed openstack/kolla-ansible master: Fix cyborg api failed to load api-paste.ini file https://review.opendev.org/721556 | 09:15 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Manage nova scheduler workers count https://review.opendev.org/721198 | 09:15 |
| hrw | mgoddard: we should consider disabling epel in c8/train | 09:16 |
| mgoddard | hrw: why? | 09:16 |
| hrw | mgoddard: suggestion from rdo team | 09:17 |
| *** happyhemant has joined #openstack-kolla | 09:17 | |
| hrw | mgoddard: and c8/train is more or less copy of c8/ussuri where we disable epel | 09:17 |
| mgoddard | hrw: we had to do some changes to disable epel | 09:17 |
| hrw | mgoddard: I will do a test local build without it to check | 09:18 |
| mgoddard | hrw: go for it. We could maybe get away with it for centos 8 since it's new, but only if changes required are small | 09:19 |
| hrw | mgoddard: enable_extra_repos() is ussuri only ;) | 09:20 |
| mgoddard | hrw: yep | 09:20 |
| yoctozepto | mgoddard: thanks, I hope only good stuff in there (e.g. mail full of praise) | 09:25 |
| mgoddard | yoctozepto: you may be partially disappointed :p | 09:25 |
| yoctozepto | mgoddard: you break my heart | 09:35 |
| mgoddard | yoctozepto: I can only apologise | 09:36 |
| hrw | mgoddard: epel is needed for erlang for rabbitmq. centos has rmq 3.8 in repo | 09:36 |
| hrw | will check with c8 erlang | 09:36 |
| mgoddard | yoctozepto: my response might have been different if we were starting this project from scratch | 09:37 |
| mgoddard | hrw: we pull erlang from bintray | 09:37 |
| mgoddard | or maybe not on trina | 09:37 |
| mgoddard | train | 09:37 |
| hrw | mgoddard: not in train | 09:38 |
| mgoddard | ok | 09:38 |
| hrw | c8 has erlang 22 with rmq 3.8 | 09:39 |
| hrw | we can use it to install 3.7 from bintray | 09:40 |
| *** k_mouza has quit IRC | 09:41 | |
| yoctozepto | mgoddard: I skimmed over it and it's not as bad as you described it; I am bad at externalizing my brain contents, hence this "feature" feature looks shady; will amend | 09:48 |
| yoctozepto | (and reply) | 09:48 |
| *** e0ne has quit IRC | 09:59 | |
| *** e0ne has joined #openstack-kolla | 10:01 | |
| *** tonythomas has joined #openstack-kolla | 10:09 | |
| *** jbadiapa has joined #openstack-kolla | 10:12 | |
| hrw | any idea why mariadb image wants 'pv' for centos but not for Deb/ubu? | 10:12 |
| hrw | anyway removed | 10:13 |
| *** xinliang has joined #openstack-kolla | 10:18 | |
| *** e0ne has quit IRC | 10:18 | |
| *** e0ne has joined #openstack-kolla | 10:19 | |
| *** ykarel|lunch is now known as ykarel | 10:20 | |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla stable/train: [WIP] switch to official CentOS 8 repositories https://review.opendev.org/721329 | 10:24 |
| hrw | all details in commit message | 10:24 |
| *** rpittau is now known as rpittau|bbl | 10:31 | |
| hrw | fsck | 10:39 |
| hrw | 9 images needs fix because of dpdk package in centos... | 10:39 |
| hrw | or at least one of them | 10:40 |
| hrw | but can be ignored as long as user does not have Mellanox cards ;d | 10:42 |
| hrw | otherwise +31MB | 10:43 |
| *** e0ne_ has joined #openstack-kolla | 10:45 | |
| *** e0ne has quit IRC | 10:45 | |
| *** jopdorp has joined #openstack-kolla | 10:46 | |
| *** xinliang has quit IRC | 10:48 | |
| *** Tengu has quit IRC | 10:48 | |
| *** Tengu has joined #openstack-kolla | 10:50 | |
| *** ricolin has quit IRC | 11:00 | |
| *** ricolin has joined #openstack-kolla | 11:00 | |
| hrw | can we deprecate/remove bifrost? | 11:01 |
| hrw | or at least move to unbuildable or sth | 11:01 |
| mgoddard | hrw: you know the answer to this | 11:02 |
| hrw | mgoddard: "patches welcome" of course | 11:06 |
| mgoddard | hrw: :p | 11:06 |
| mgoddard | we use it in kayobe. patches welcome to stop (and replace with alternative and provide migration path) | 11:07 |
| hrw | mgoddard: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_b02/721329/2/check/kolla-build-centos8-source/b0209db/ | 11:07 |
| hrw | mgoddard: I consider it SUCCESS as only bifrost failed | 11:07 |
| mgoddard | hrw: zuul considers it FAILURE :) | 11:08 |
| hrw | mgoddard: patch is WIP so this FAILURE is SUCCESS :D | 11:08 |
| hrw | especially when bifrost fails randomly | 11:09 |
| mgoddard | hrw: it's just missing the debootstrap package | 11:09 |
| hrw | mgoddard: which is not in the dockerfile.j2 so should shut itself and go away quickly | 11:10 |
| hrw | it's abomination | 11:11 |
| hrw | whatever it does after being created should be done in k-a imho | 11:11 |
| hrw | as now it is bastard image which failures are something to just ignore as they are not our fault | 11:13 |
| mgoddard | hrw: I'm not saying I like it, but we depend on it. | 11:14 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: WIP: Switch default docker storage driver to overlay2 https://review.opendev.org/658815 | 11:18 |
| *** Fl1nt has joined #openstack-kolla | 11:25 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Avoid multiple haproxy restarts after reconfiguration https://review.opendev.org/708385 | 11:26 |
| Fl1nt | Hi o/ | 11:27 |
| Fl1nt | How are you all today? | 11:27 |
| hrw | sleepy | 11:30 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/train: Fix fernet bootstrap and key distribution - follow up https://review.opendev.org/721582 | 11:31 |
| Fl1nt | yeah, me too, the weather isn't helping a lot tbh :D | 11:31 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/train: CentOS 8: CI: Test CentOS 7 to 8 migration https://review.opendev.org/711601 | 11:32 |
| *** seco has joined #openstack-kolla | 11:35 | |
| mnasiadka | mgoddard, yoctozepto: got some spare cycles this week to look at: https://review.opendev.org/#/q/topic:mariadb-clustercheck+(status:open+OR+status:merged) ? | 11:37 |
| *** cah_link has quit IRC | 11:39 | |
| *** cah_link has joined #openstack-kolla | 11:40 | |
| *** k_mouza has joined #openstack-kolla | 11:42 | |
| *** priteau has quit IRC | 11:44 | |
| Fl1nt | hum... I'm using a ceph-external cluster on rocky k-a (7.2.1), that cluster have images/volumes/volumes-backup and vms volume as erasure-code pool with ec_overwrite allowed and a each ec pool get a regular replicated pool attached | 11:45 |
| Fl1nt | those attached replicated pool are the cinder rbd pool within the configuration | 11:45 |
| Fl1nt | on my ceph.conf file | 11:45 |
| *** k_mouza has quit IRC | 11:46 | |
| Fl1nt | I've a [client.cinder] section with: rbd default data pool = ec_pool_name | 11:46 |
| Fl1nt | I've also create a client with mon 'allow r' and osd 'allow rw pool=volumes, allow rw pool=volumes-metadata' | 11:47 |
| Fl1nt | I've followed that doc: https://docs.openstack.org/kolla-ansible/rocky/reference/external-ceph-guide.html | 11:48 |
| *** e0ne_ has quit IRC | 11:48 | |
| *** e0ne has joined #openstack-kolla | 11:48 | |
| Fl1nt | found out that: backend_host=rbd:volumes | 11:48 |
| Fl1nt | should not be in cinder-volume.conf | 11:48 |
| Fl1nt | and fixed it | 11:48 |
| *** e0ne has quit IRC | 11:49 | |
| Fl1nt | however, I'm still getting cinder-volume rbd.py driver (actually client.connect() from rados package) which complain about not being to connect to the ceph cluster | 11:49 |
| Fl1nt | with an errno 13 Permission Denied | 11:50 |
| *** e0ne has joined #openstack-kolla | 11:50 | |
| Fl1nt | which usually arrive when ceph client isn't able to find the appropriate key for a user. | 11:50 |
| Fl1nt | the ceph cluster is a nautilus 14 release. | 11:50 |
| Fl1nt | it's working well as other non-openstack machine correctly connect to. | 11:51 |
| Fl1nt | my ceph.client.cinder.keyring file are available on /etc/ceph/ folder with appropriate ownership and access right (0700 cinder:cinder for keyring and 0600 cinder:cinder for ceph.conf). | 11:53 |
| Fl1nt | my cluster_network and public_network are reachable from within the cinder container | 11:55 |
| *** JangwonLee_ has joined #openstack-kolla | 12:02 | |
| *** JangwonLee has joined #openstack-kolla | 12:04 | |
| yoctozepto | mgoddard, hrw: goal for victoria to coordinate effort of dropping bifrost? | 12:05 |
| yoctozepto | mnasiadka: oh, it's green; so what was wrong? | 12:06 |
| hrw | yoctozepto: move of everything under ^USER to k-a probably fix it | 12:06 |
| *** JangwonLee__ has quit IRC | 12:06 | |
| *** bersace has quit IRC | 12:07 | |
| *** JangwonLee_ has quit IRC | 12:07 | |
| mnasiadka | yoctozepto: I missed environment: when starting clustercheck container in mariadb recovery :) | 12:07 |
| *** bersace has joined #openstack-kolla | 12:07 | |
| *** e0ne_ has joined #openstack-kolla | 12:10 | |
| *** e0ne has quit IRC | 12:10 | |
| *** dougsz has joined #openstack-kolla | 12:14 | |
| *** JangwonLee_ has joined #openstack-kolla | 12:14 | |
| *** k_mouza has joined #openstack-kolla | 12:16 | |
| *** JangwonLee__ has joined #openstack-kolla | 12:17 | |
| *** JangwonLee has quit IRC | 12:18 | |
| *** JangwonLee_ has quit IRC | 12:20 | |
| *** JangwonLee_ has joined #openstack-kolla | 12:21 | |
| *** JangwonLee__ has quit IRC | 12:25 | |
| *** rpittau|bbl is now known as rpittau | 12:31 | |
| yoctozepto | hrw: we have all the ironic components nicely packaged, just use them, mgoddard knows what to do I think, just not first prio, am I right? ;p | 12:34 |
| yoctozepto | mnasiadka: ouch, happens | 12:34 |
| mgoddard | yoctozepto: depends on available resource on our side. It's not a small task | 12:55 |
| mgoddard | hrw: only fixed from your perspective, it just shifts the problems to k-a | 12:58 |
| hrw | mgoddard: so all build jobs work and only bifrost k-a scenario shows that it is broken or not | 12:59 |
| mgoddard | hrw: bifrost installs packages and clones sources. we want them in the image | 12:59 |
| *** jbadiapa has quit IRC | 13:00 | |
| hrw | mgoddard: can it do that in really repeatable way? as now it fails randomly | 13:00 |
| mgoddard | hrw: it's as repeatable as any ansible | 13:00 |
| mgoddard | when you say it fails randomly, you mean fail then pass, or it breaks then we fix it? | 13:01 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/train: Fix fernet bootstrap and key distribution - follow up https://review.opendev.org/721582 | 13:05 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Fix fernet bootstrap and key distribution - follow up https://review.opendev.org/707080 | 13:05 |
| hrw | ok, let end the discussion | 13:08 |
| *** ykarel is now known as ykarel|afk | 13:08 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Fix fernet bootstrap and key distribution - follow up https://review.opendev.org/707080 | 13:08 |
| *** bengates_ has joined #openstack-kolla | 13:15 | |
| *** bengates has quit IRC | 13:15 | |
| *** zhanglong has joined #openstack-kolla | 13:16 | |
| hrw | yoctozepto: let py2 rot! https://review.opendev.org/#/c/713283/ | 13:21 |
| patchbot | patch 713283 - kolla-ansible - Remove support for Python 2 on the host - 6 patch sets | 13:21 |
| *** jimcrowleyibm has joined #openstack-kolla | 13:22 | |
| *** zhanglong has quit IRC | 13:28 | |
| *** zhanglong has joined #openstack-kolla | 13:28 | |
| *** gfidente has quit IRC | 13:28 | |
| *** e0ne has joined #openstack-kolla | 13:41 | |
| *** e0ne_ has quit IRC | 13:41 | |
| *** KeithMnemonic has joined #openstack-kolla | 13:44 | |
| *** gfidente has joined #openstack-kolla | 13:46 | |
| *** bersace has quit IRC | 13:52 | |
| *** bersace has joined #openstack-kolla | 13:52 | |
| *** dougsz has quit IRC | 13:59 | |
| *** jbadiapa has joined #openstack-kolla | 13:59 | |
| *** TrevorV has joined #openstack-kolla | 14:15 | |
| *** dougsz has joined #openstack-kolla | 14:16 | |
| *** zhanglong has quit IRC | 14:18 | |
| *** ykarel|afk is now known as ykarel | 14:22 | |
| *** ricolin has quit IRC | 14:23 | |
| *** ricolin has joined #openstack-kolla | 14:23 | |
| *** tonythomas has quit IRC | 14:28 | |
| *** srin has quit IRC | 14:30 | |
| *** srin has joined #openstack-kolla | 14:31 | |
| *** cdearborn has joined #openstack-kolla | 14:33 | |
| dking | For Kayobe, does localhost not pick up networks.yml variables? Or could there be a reason why internal_net_name, which is set in networks.yml showing as empty in venvs/kayobe/share/kayobe/ansible/kolla-ansible.yml ? | 14:40 |
| mgoddard | dking: those variables are global, applied to every host. | 14:41 |
| mgoddard | dking: if you add -v, kayobe will show the commands it uses. Check you are using the correct configuration paths | 14:41 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Add docs and release note for CentOS 8 https://review.opendev.org/721632 | 14:43 |
| dking | Oh, wait. Now I see that it's the net_vip_address filter. | 14:44 |
| *** dmellado has quit IRC | 14:49 | |
| *** skramaja has quit IRC | 14:55 | |
| sean-k-mooney | mgoddard: that centos bug looks like it might be in oslo_config | 15:02 |
| sean-k-mooney | its complaing that the set of config options that it is iterating over is changing | 15:02 |
| sean-k-mooney | im not really sure how that can happen. | 15:03 |
| sean-k-mooney | its failing here https://github.com/openstack/oslo.config/blame/579a323a43a7f6fdacc3766a1b4b994ceffc6771/oslo_config/cfg.py#L2500-L2506 | 15:05 |
| dking | Okay, so I see now. It's needing a *_vip_address. Is that set directly somewhere? I didn't seem to need to do that when deploying the seed host. Also, I don't see it set explicitly in the example at https://docs.openstack.org/kayobe/latest/configuration/network.html#complete-example | 15:08 |
| mgoddard | sean-k-mooney: just got a reply, suggests this is probably related: https://github.com/openstack/oslo.config/commit/e3e2ba55eeeb86a9bc0624bb2592e46583e839e7 | 15:08 |
| sean-k-mooney | mgoddard: its https://github.com/openstack/oslo.config/commit/e3e2ba55eeeb86a9bc0624bb2592e46583e839e7#diff-768b817a50237989cacd1a8064b4a8af | 15:08 |
| sean-k-mooney | hehe yep | 15:09 |
| mgoddard | sean-k-mooney: although I think the call stack looks different | 15:09 |
| mgoddard | comes from _warn_immutability | 15:10 |
| sean-k-mooney | yes although any modificaiotn to self._opts could cause the same issue | 15:13 |
| Fl1nt | ok, so, I've tried to use the ceph admin key, it works, what's bugging me is that I've even give cinder/glance and nova client key caps as mon 'allow *' osd 'allow *' mgr 'allow *' which should work whatever. | 15:15 |
| sean-k-mooney | its hard to see how a concurrent mutation would happen in this case | 15:17 |
| *** dmellado has joined #openstack-kolla | 15:17 | |
| Fl1nt | does the glance/nova/cinder user have to be client.<service> or just <service> ? my keyring are named ceph.client.<service>.keyring and within ceph I've client.<service> when I do a ceph -n admin --keyring /etc/ceph/ceph.client.admin.keyring status isn't working but if I use ceph -n client.admin it works. | 15:17 |
| sean-k-mooney | mgoddard: technically in python3 .values() is not a new list its an interator over a view | 15:18 |
| mgoddard | sean-k-mooney: right, but I'm not seeing where a modification takes place. Must be buried somewhere | 15:18 |
| sean-k-mooney | so we might need to make https://github.com/openstack/oslo.config/blame/579a323a43a7f6fdacc3766a1b4b994ceffc6771/oslo_config/cfg.py#L2502 for info in list(...): | 15:18 |
| sean-k-mooney | i cant see how it would take place in this code path unless we yeild exectuion via eventlet or there is a second thread | 15:19 |
| mgoddard | sean-k-mooney: both source and binary images are using oslo.config 6.11.2 | 15:20 |
| sean-k-mooney | is this only happeing in the binary version | 15:20 |
| mgoddard | yes | 15:20 |
| sean-k-mooney | are they using python 2 or 3 | 15:20 |
| mgoddard | py3 on centos8 | 15:21 |
| sean-k-mooney | actully so there are two train centos images right 1 is centos 7 and the other is centos8 | 15:21 |
| dswebb | hi all, I know I've seen it somewhere but my googlefoo is failing me, do you know the link for the matrix which maps kolla versions to ovs versions? | 15:21 |
| mgoddard | sean-k-mooney: train supports both c7 and c8 | 15:23 |
| mgoddard | in both source and binary flavours | 15:23 |
| sean-k-mooney | mgoddard: yep and are you seeing this on c7 binary too | 15:24 |
| mgoddard | sean-k-mooney: nope :) | 15:24 |
| mgoddard | only c8/binary/train | 15:24 |
| sean-k-mooney | im wondering is the a change in behavior in python 3.7 vs 3.6 | 15:24 |
| sean-k-mooney | *2.7 | 15:24 |
| mgoddard | works on master | 15:24 |
| sean-k-mooney | although it still does not explain why it works on source | 15:24 |
| sean-k-mooney | and ya master | 15:25 |
| sean-k-mooney | mgoddard: im still wonding if something like https://github.com/openstack/oslo.config/commit/7ad146c48791bebee12fbdcdb1ac2427237343c3 is needed | 15:26 |
| mgoddard | sean-k-mooney: I expect it's something like that | 15:27 |
| *** seco has quit IRC | 15:27 | |
| mgoddard | sean-k-mooney: both source and binary installing nova 20.1.1. RDO has 20.1.1-2 | 15:28 |
| sean-k-mooney | ya ok i was hoping to see something obvious looking at the code but i dont but i think its likely a subtel bug in oslo config. | 15:28 |
| mgoddard | sean-k-mooney: I'll update the nova bug | 15:29 |
| sean-k-mooney | i just dont know why it would only show up in this edgecase | 15:29 |
| sean-k-mooney | mgoddard: did you use kibana to check if tha shows up in any other jobs? | 15:29 |
| mgoddard | sean-k-mooney: I haven't yet | 15:29 |
| sean-k-mooney | it would be intersting to see if any of the devstack or other jobs hit this. | 15:30 |
| sean-k-mooney | i might just present more in the kolla job for some reason | 15:30 |
| *** dmellado has quit IRC | 15:35 | |
| *** dmellado has joined #openstack-kolla | 15:37 | |
| dking | mgoddard: I have a few quick questions for today if you have a moment. First, if I want to deploy baremetal servers, they aren't standalone servers, but part of an overcloud installed by Kayboe? | 15:37 |
| *** cah_link has quit IRC | 15:39 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/train: CentOS 8: CI: Test CentOS 7 to 8 migration https://review.opendev.org/711601 | 15:40 |
| *** ykarel is now known as ykarel|away | 15:40 | |
| mgoddard | dking: when you say you want to deploy baremetal servers, are they for control/hypervisors or to provide bare metal to users? | 15:41 |
| sean-k-mooney | mgoddard: is that testign migration of the host os or vms | 15:42 |
| mgoddard | sean-k-mooney: host OS | 15:42 |
| sean-k-mooney | mgoddard: there are limited case that we expect to work for the vms | 15:42 |
| dking | Well, I was just tossing around the idea of just deploying baremetal servers and then handling the OpenStack deploy on them using our own tools. | 15:42 |
| sean-k-mooney | ok | 15:42 |
| mgoddard | sean-k-mooney: I was unable to get live migration working from c7 to c8. Any pointers would be appreciated :) | 15:43 |
| mgoddard | sean-k-mooney: it's trying to test this: https://opendev.org/openstack/kolla-ansible/src/branch/master/specs/centos8-migration.rst | 15:43 |
| sean-k-mooney | gotch ya. | 15:44 |
| mgoddard | dking: when you say deploying openstack using your own tools, are these tools you have already built? | 15:44 |
| dking | Since we'll potentially have very many overclouds with different configurations, possibly hundreds or more. But it seems that in terms of Kayobe, the baremetals are part of a specific overcloud, anyway. | 15:44 |
| dking | Possibly. We have some tools that might work, but not entirely. I'm just trying to think out the best options. | 15:45 |
| mgoddard | dking: sounds like you might want to run the control plane in VMs if you have lots of small clouds? | 15:45 |
| mgoddard | dking: presumably you would at least want to run compute on bare metal | 15:46 |
| dking | We're wanting the controllers on bare metal as well, since we want each of the overclouds to be physically independent of each other. | 15:48 |
| mgoddard | dking: ok, so what is the problem with baremetals being part of a specific overcloud? | 15:48 |
| *** ricolin has quit IRC | 15:49 | |
| dking | The idea I was tossing around was just having Kayobe setup a bunch of independent baremetal servers, and then having something we build do the job of actually placing OpenStack on them. But I don't like the idea of having the overhead of a separate OpenStack instance just for that. | 15:51 |
| dking | Besides, I don't want to re-invent the wheel. So, I might stick with just having Kayobe install the individual overclouds. | 15:51 |
| mgoddard | dking: I would definitely recommend against building your own OpenStack deployment tool | 15:53 |
| mgoddard | dking: tbh, kolla ansible is the magic part. Kayobe mostly adds bare metal provisioning | 15:53 |
| dking | Yeah. I think that we're pretty solid with Kolla-ansible at this piont. The baremetal provisioning is what I'm looking at now. I'm just trying to get an idea of how things will handle at scale. | 15:55 |
| kplant | sorry if this has been answered but, can kayobe assume ownership of clouds already provisioned by k-a? | 15:55 |
| dking | For instance, one problem with TripleO was that you can only run one deploy or update at a time, and that would take about 30 minutes. | 15:55 |
| kplant | tripleo is scary | 15:56 |
| dking | Yeah. I didn't like all the Heat stuff at all. There were so many different moving parts to investigate. One very big advantage of Kayobe so far has been that I'm only really dealing with Ansible, rather than Ansible, Heat, Puppet, etc. | 15:56 |
| kplant | tripleo kind of seems impossible to fix once it fails | 15:57 |
| mgoddard | kplant: in theory, yes. It's been on my todo list to run through and document it. Mostly it's a case of dropping your /etc/kolla/ config into kayobe-config/etc/kayobe/kolla/ | 16:01 |
| mgoddard | at least for the openstack config | 16:01 |
| mgoddard | dking: the clouds should be independent. The commands are all modular, and provisioning is (intentionally) very decoupled from host configuration, and that from openstack deployment | 16:02 |
| mgoddard | dking: at the provisioning level, it's just operating on individual nodes in ironic. There's no 'stack' tying them together | 16:03 |
| mgoddard | dking: then beyond provisioning, each cloud should be entirely independent - there's no real tie back to the seed | 16:03 |
| dking | That sounds good, then. | 16:04 |
| mgoddard | unless you run a docker registry or something on it, but that's a deploy time thing | 16:04 |
| mgoddard | dking: I think the multiple envs per config thing will be key for you once you hit 10+ clouds, but you should be able to roll without for a good while | 16:05 |
| dking | That should be fine for initial testing. | 16:06 |
| Fl1nt | ok, I'm lost | 16:06 |
| Fl1nt | my cinder issue is a permission denied issue when using a <service_name> as the username. | 16:06 |
| Fl1nt | but then it's a connectivity issue when using client.<service_name> as the username. | 16:07 |
| Fl1nt | right from inside my cinder_volume container | 16:07 |
| Fl1nt | if I use ceph -n client.<service> --keyring /etc/ceph/ceph.client.<service_name>.keyring status | 16:08 |
| Fl1nt | it work... | 16:08 |
| Fl1nt | but if I use ceph -n <service> blabla I get the errno 13 Permission denied issue... | 16:08 |
| Fl1nt | WTF | 16:08 |
| Fl1nt | does you people have specific [client.<service_name] section containing your keyring = <path> option within your ceph.conf ? | 16:09 |
| openstackgerrit | Merged openstack/kayobe stable/train: Fix ntp and chrony in mixed CentOS 7 and 8 environments https://review.opendev.org/720731 | 16:13 |
| openstackgerrit | Merged openstack/kayobe stable/train: Prevent openrc files from using wrong OS_CACERT value https://review.opendev.org/720733 | 16:13 |
| openstackgerrit | Merged openstack/kayobe master: Docs: fix reference to deprecated external_net_name https://review.opendev.org/720734 | 16:13 |
| *** bengates_ has quit IRC | 16:17 | |
| openstackgerrit | James Kirsch proposed openstack/kolla master: Add stunnel container for Glance service https://review.opendev.org/720850 | 16:20 |
| *** kklimonda has quit IRC | 16:28 | |
| *** kklimonda has joined #openstack-kolla | 16:30 | |
| *** evrardjp has quit IRC | 16:35 | |
| *** evrardjp has joined #openstack-kolla | 16:35 | |
| *** e0ne has quit IRC | 16:36 | |
| *** e0ne_ has joined #openstack-kolla | 16:36 | |
| *** kklimonda has quit IRC | 16:38 | |
| *** happyhemant has quit IRC | 16:38 | |
| *** rpittau is now known as rpittau|afk | 16:39 | |
| *** kklimonda has joined #openstack-kolla | 16:39 | |
| *** stackedsax has quit IRC | 16:40 | |
| *** happyhemant has joined #openstack-kolla | 16:40 | |
| *** egonzalez has quit IRC | 16:41 | |
| *** klindgren has quit IRC | 16:41 | |
| *** klindgren has joined #openstack-kolla | 16:41 | |
| *** niedbalski has quit IRC | 16:42 | |
| *** TheJulia has quit IRC | 16:42 | |
| *** r3ap3r has quit IRC | 16:42 | |
| *** egonzalez has joined #openstack-kolla | 16:43 | |
| *** stackedsax has joined #openstack-kolla | 16:43 | |
| *** r3ap3r has joined #openstack-kolla | 16:43 | |
| *** TheJulia has joined #openstack-kolla | 16:44 | |
| *** niedbalski has joined #openstack-kolla | 16:44 | |
| Fl1nt | \o/ FOUND OUT \o/ | 16:52 |
| Fl1nt | Joy and hapiness :D | 16:52 |
| Fl1nt | So, the root cause is still to identify, but I don't know why, my keyring checksum are different on host than on repository... | 16:53 |
| Fl1nt | thanks to ceph documentation of the librados, I've found out why I had such weird results. | 16:53 |
| Fl1nt | once using the file with the appropriate checksum it all works like a charm. | 16:54 |
| *** seco has joined #openstack-kolla | 16:55 | |
| *** seco has quit IRC | 17:00 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Remove support for Python 2 on the host https://review.opendev.org/713283 | 17:01 |
| *** jbadiapa has quit IRC | 17:02 | |
| *** dougsz has quit IRC | 17:03 | |
| *** spiette_ is now known as spiette | 17:06 | |
| mrunge | hrw, hi, collectd-sensubility and python-sqlalchemy-collectd built on c8, they should hit the mirrors shortly | 17:08 |
| *** e0ne_ has quit IRC | 17:19 | |
| *** k_mouza has quit IRC | 17:22 | |
| hrw | mrunge: thanks! | 17:35 |
| Fl1nt | So, just for anyone in the futur stumbling on this same case, becareful of your keyring newline, our keyring files were created within a IDE not having newline enforced on each file and so that created the behavior as no newline seems to upset ceph | 17:38 |
| Fl1nt | CEPH being written mostly using C++/C it's not that surprising, but yet, that's not really something your expect to create such cryptic error and behavior. | 17:38 |
| Fl1nt | end of story, end of day for me, see you on tomorrow dear inhabitant of the chan ;-) | 17:39 |
| *** Fl1nt has quit IRC | 17:39 | |
| hrw | mrunge: landed | 17:42 |
| *** hamzy__ has quit IRC | 17:59 | |
| *** gfidente is now known as gfidente|afk | 18:00 | |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: Improve CI ansible plays to pass ansible-lint https://review.opendev.org/720664 | 18:05 |
| *** jopdorp has quit IRC | 18:25 | |
| *** gmann is now known as gmann_lunch | 18:31 | |
| *** saneax has quit IRC | 18:38 | |
| cosmicsound | http://paste.openstack.org/show/792500/ got these weird defunct/zombies @ cloudkitty | 18:46 |
| *** saneax has joined #openstack-kolla | 18:49 | |
| *** gmann_lunch is now known as gmann | 18:51 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Fix that cyborg conductor failed to communicate with placement https://review.opendev.org/721158 | 18:57 |
| sean-k-mooney | so shellinabox and socat which were removed as part of disabling eple have not been replaced https://github.com/openstack/kolla/commit/1fe8012ce2aec549430212d90ffd728889d6e4aa | 19:05 |
| sean-k-mooney | actully wong link | 19:06 |
| sean-k-mooney | https://github.com/openstack/kolla/commit/397651ec4b5b93a55d0b8b8f69dfb37dd85cfc8d | 19:06 |
| sean-k-mooney | any reason not to provide socat instead so that an ironic terminal can be used on centos 8 | 19:10 |
| sean-k-mooney | given that kolla now support the nova serial console proxy i think if you use ipmi-socat as the console interface it should work in horizon https://docs.openstack.org/ironic/latest/admin/console.html#node-serial-console | 19:12 |
| *** cah_link has joined #openstack-kolla | 19:13 | |
| sean-k-mooney | oh its in the base iamge https://github.com/openstack/kolla/commit/293a1d9f8c32ed89084368acae079d9c57d8f30f#diff-1c12f06ccd6feb71a22c6c66b4ab40a7 i wonder if that is still true | 19:13 |
| sean-k-mooney | yes it is ok ipmi-socat is disabeld by default but it should work never mind | 19:14 |
| *** bline has quit IRC | 19:17 | |
| *** bline has joined #openstack-kolla | 19:18 | |
| *** k_mouza has joined #openstack-kolla | 19:23 | |
| *** k_mouza has quit IRC | 19:28 | |
| *** e0ne has joined #openstack-kolla | 19:36 | |
| openstackgerrit | Merged openstack/kolla-ansible stable/train: Fix service_mapped_to_host filter for common services https://review.opendev.org/720669 | 19:36 |
| openstackgerrit | Merged openstack/kolla-ansible master: Manage nova scheduler workers count https://review.opendev.org/721198 | 19:36 |
| *** dougsz has joined #openstack-kolla | 19:53 | |
| *** hjensas has quit IRC | 19:59 | |
| *** hjensas has joined #openstack-kolla | 19:59 | |
| *** hjensas has quit IRC | 20:02 | |
| *** hjensas has joined #openstack-kolla | 20:05 | |
| *** seco has joined #openstack-kolla | 20:08 | |
| *** TrevorV has quit IRC | 20:12 | |
| *** seco has quit IRC | 20:13 | |
| *** e0ne has quit IRC | 20:14 | |
| *** dougsz has quit IRC | 20:35 | |
| *** e0ne has joined #openstack-kolla | 20:37 | |
| *** cah_link1 has joined #openstack-kolla | 20:45 | |
| *** cah_link has quit IRC | 20:47 | |
| *** cah_link1 is now known as cah_link | 20:47 | |
| openstackgerrit | James Kirsch proposed openstack/kolla-ansible master: Add support for encrypting glance/heat api https://review.opendev.org/707131 | 20:48 |
| openstackgerrit | Merged openstack/kolla-ansible master: Improve CI ansible plays to pass ansible-lint https://review.opendev.org/720664 | 21:15 |
| *** saneax has quit IRC | 21:24 | |
| *** k_mouza has joined #openstack-kolla | 21:25 | |
| *** k_mouza has quit IRC | 21:29 | |
| *** sean-k-mooney has quit IRC | 21:40 | |
| *** sean-k-mooney has joined #openstack-kolla | 21:43 | |
| *** e0ne has quit IRC | 21:56 | |
| *** dcapone2004 has joined #openstack-kolla | 22:01 | |
| dcapone2004 | does kolla-ansible support Centos 8? | 22:01 |
| *** jimcrowleyibm has quit IRC | 22:14 | |
| *** cah_link has quit IRC | 22:14 | |
| sean-k-mooney | kind of. its a new feature in ussuri release and some of the support has been backported to train | 22:17 |
| *** brault has quit IRC | 22:22 | |
| dcapone2004 | hmmm....having some trouble with train | 22:56 |
| dcapone2004 | on the most basic step … do you need to be using the git repo for the backports? | 22:56 |
| *** calbers has quit IRC | 23:04 | |
| *** calbers has joined #openstack-kolla | 23:04 | |
| *** bersace has quit IRC | 23:04 | |
| *** calbers has quit IRC | 23:05 | |
| *** calbers has joined #openstack-kolla | 23:15 | |
| *** calbers has quit IRC | 23:39 | |
| *** calbers has joined #openstack-kolla | 23:41 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!