| *** ChanServ changes topic to "IRC meetings on Wednesdays @ 15:00 UTC - agenda @ https://goo.gl/OXB0DL | Whiteboard: https://bit.ly/2MM7mWF | IRC channel is *LOGGED* @ http://goo.gl/3mzZ7b" | 00:37 | |
| -openstackstatus- NOTICE: The Gerrit service at review.opendev.org is back up and running; for outage details see analysis here: http://lists.opendev.org/pipermail/service-announce/2020-October/000011.html | 00:37 | |
| *** wuchunyang has joined #openstack-kolla | 01:12 | |
| *** LinPeiWen has quit IRC | 02:01 | |
| *** LinPeiWen has joined #openstack-kolla | 02:30 | |
| *** bsanjeewa has joined #openstack-kolla | 02:48 | |
| *** skramaja has joined #openstack-kolla | 03:26 | |
| *** wuchunyang has quit IRC | 04:00 | |
| *** bsanjeewa has quit IRC | 04:29 | |
| *** evrardjp has quit IRC | 04:33 | |
| *** evrardjp has joined #openstack-kolla | 04:33 | |
| *** abdysn has joined #openstack-kolla | 05:23 | |
| *** LinPeiWen has quit IRC | 06:02 | |
| *** cah_link has joined #openstack-kolla | 06:05 | |
| *** cah_link has quit IRC | 06:05 | |
| *** wuchunyang has joined #openstack-kolla | 06:07 | |
| *** cah_link has joined #openstack-kolla | 06:13 | |
| *** bsanjeewa has joined #openstack-kolla | 06:17 | |
| *** cah_link has quit IRC | 06:20 | |
| *** cah_link has joined #openstack-kolla | 06:27 | |
| *** LinPeiWen has joined #openstack-kolla | 06:29 | |
| mnasiadka | morning | 06:34 |
|---|---|---|
| *** dciabrin has joined #openstack-kolla | 06:35 | |
| *** bsanjeewa has quit IRC | 06:35 | |
| *** rpittau|afk is now known as rpittau | 06:42 | |
| *** vishalmanchanda has joined #openstack-kolla | 06:47 | |
| *** ebbex has joined #openstack-kolla | 06:49 | |
| *** bsanjeewa has joined #openstack-kolla | 06:58 | |
| *** kemopq has joined #openstack-kolla | 07:00 | |
| *** jbadiapa has joined #openstack-kolla | 07:02 | |
| *** bengates has joined #openstack-kolla | 07:18 | |
| yoctozepto | morning | 07:25 |
| yoctozepto | mgoddard, mnasiadka, hrw: it's likely only a formality but we should review the changes merged to our projects in the affected period: | 07:26 |
| yoctozepto | https://static.opendev.org/project/opendev.org/gerrit-diffs/openstack/kayobe.git/ | 07:26 |
| yoctozepto | https://static.opendev.org/project/opendev.org/gerrit-diffs/openstack/kolla-ansible.git/ | 07:26 |
| yoctozepto | https://static.opendev.org/project/opendev.org/gerrit-diffs/openstack/kolla.git/ | 07:27 |
| mnasiadka | I can look into it after 1pm | 07:27 |
| yoctozepto | ok | 07:27 |
| yoctozepto | I will definitely review k-a changes today | 07:28 |
| yoctozepto | not touching kayobe for obvious reasons | 07:28 |
| mnasiadka | should we create something for tracking, generate a csv and ,,tick'' what has been done? | 07:28 |
| yoctozepto | there ain't that much to track but please proceed if you find it helpful | 07:28 |
| mnasiadka | well, for kayobe it doesn't make sense | 07:28 |
| mnasiadka | haven't looked for the numbers in k and k-a | 07:29 |
| *** openstackgerrit has quit IRC | 07:38 | |
| *** Fl1nt has joined #openstack-kolla | 07:39 | |
| yoctozepto | slightly biggur | 07:39 |
| *** e0ne has joined #openstack-kolla | 07:41 | |
| *** dougsz has joined #openstack-kolla | 07:50 | |
| mnasiadka | yoctozepto: only slightly :) | 07:51 |
| mnasiadka | yoctozepto: Pierre looked through kayobe, and it looks fine - so one thing off the radar. | 07:52 |
| yoctozepto | ++ | 07:53 |
| *** priteau has joined #openstack-kolla | 07:54 | |
| *** dougsz has quit IRC | 07:55 | |
| *** openstackgerrit has joined #openstack-kolla | 07:57 | |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: build: unbreak JSON serializer on failed builds https://review.opendev.org/758957 | 07:57 |
| *** shyamb has joined #openstack-kolla | 08:02 | |
| shyamb | Hi | 08:02 |
| mgoddard | morning | 08:11 |
| mgoddard | kayobe diffs look fine | 08:11 |
| hrw | kolla stable/ussuri looks good | 08:14 |
| hrw | kolla master too | 08:16 |
| *** k_mouza has joined #openstack-kolla | 08:16 | |
| openstackgerrit | Mark Goddard proposed openstack/kayobe stable/train: docs: Add link to Kolla Ansible CentOS 8 page https://review.opendev.org/758958 | 08:16 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: docs: Add link back to Train CentOS 8 migration page https://review.opendev.org/758960 | 08:21 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: docs: Add link back to Train CentOS 8 migration page https://review.opendev.org/758962 | 08:24 |
| *** SvenKieske has joined #openstack-kolla | 08:31 | |
| SvenKieske | Morning :) It seems I could use some help with https://review.opendev.org/#/c/758576/ , there are Gatingfailures, e.g.: https://zuul.opendev.org/t/openstack/build/b10b7b96db334cbb849c46a80004cff2/console which seem unrelated to me? but I may be wrong. | 08:33 |
| patchbot | patch 758576 - kolla-ansible - re-add rabbitmq config for interfaces - 4 patch sets | 08:33 |
| mgoddard | yoctozepto, mnasiadka, hrw: I've skimmed all of the gerrit patches, and can't see anything amiss | 08:45 |
| priteau | I checked kayobe, didn't see anything wrong | 08:57 |
| *** gfidente has joined #openstack-kolla | 09:19 | |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: docs: Add link back to Train CentOS 8 migration page https://review.opendev.org/758960 | 09:22 |
| openstackgerrit | Mark Goddard proposed openstack/kolla master: Switch to Victoria stable branches https://review.opendev.org/755339 | 09:22 |
| mnasiadka | mgoddard: so let's assume we're good | 09:25 |
| *** devfaz has quit IRC | 09:38 | |
| *** abdysn has quit IRC | 09:45 | |
| *** shyam89 has joined #openstack-kolla | 09:53 | |
| *** shyamb has quit IRC | 09:56 | |
| *** dardelean has joined #openstack-kolla | 09:58 | |
| *** wuchunyang has quit IRC | 10:07 | |
| *** bsanjeewa has quit IRC | 10:08 | |
| *** shyam89 has quit IRC | 10:13 | |
| *** shyamb has joined #openstack-kolla | 10:13 | |
| *** k_mouza has quit IRC | 10:16 | |
| *** k_mouza has joined #openstack-kolla | 10:19 | |
| *** abdysn has joined #openstack-kolla | 10:27 | |
| yoctozepto | mgoddard, mnasiadka, hrw: I reviewed k-a and it looks ok | 10:31 |
| yoctozepto | all branches | 10:31 |
| *** bsanjeewa has joined #openstack-kolla | 10:32 | |
| *** k_mouza_ has joined #openstack-kolla | 10:33 | |
| *** k_mouza has quit IRC | 10:37 | |
| *** priteau has quit IRC | 10:39 | |
| yoctozepto | escurator fails to install: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_a33/758957/1/check/kolla-build-centos8-source/a332fbf/kolla/build/000_FAILED_elasticsearch-curator.log | 10:42 |
| *** k_mouza has joined #openstack-kolla | 10:46 | |
| mnasiadka | I've seen that error before | 10:46 |
| mnasiadka | yoctozepto: https://github.com/elastic/curator/issues/1496 - sounds familiar | 10:47 |
| *** k_mouza_ has quit IRC | 10:49 | |
| yoctozepto | hmm, but on centos8 it should not be py38 | 10:49 |
| yoctozepto | ah, I see, wheels | 10:50 |
| yoctozepto | indeed it tries compiling pyyaml | 10:50 |
| yoctozepto | so it's py version irrelevant; someone just got wrong assumption | 10:51 |
| hrw | or we need to install libyaml-dev | 10:52 |
| yoctozepto | or both indeed | 10:52 |
| hrw | INFO:kolla.common.utils.elasticsearch-curator: ModuleNotFoundError: No module named 'cx_Freeze' | 10:53 |
| hrw | anyway | 10:53 |
| openstackgerrit | Merged openstack/kolla-ansible master: docs: Add link back to Train CentOS 8 migration page https://review.opendev.org/758962 | 10:55 |
| *** cah_link has quit IRC | 10:58 | |
| *** cah_link has joined #openstack-kolla | 11:00 | |
| *** abdysn has quit IRC | 11:02 | |
| *** bsanjeewa_ has joined #openstack-kolla | 11:06 | |
| *** wuchunyang has joined #openstack-kolla | 11:06 | |
| *** bsanjeewa has quit IRC | 11:09 | |
| yoctozepto | wonder why it broke all of a sudden... | 11:14 |
| yoctozepto | should we perhaps pin curator's version? | 11:15 |
| yoctozepto | otherwise it gets latest everywhere | 11:15 |
| *** priteau has joined #openstack-kolla | 11:15 | |
| yoctozepto | sad it's not properly packages | 11:15 |
| yoctozepto | packaged* | 11:15 |
| hrw | deprecate? | 11:17 |
| mnasiadka | yoctozepto: hrw’s proposal also fixes cron issue :D | 11:21 |
| hrw | curator imports from cx_freeze without even depending on it | 11:23 |
| hrw | wtf? | 11:23 |
| hrw | and cx_freeze requires gcc | 11:25 |
| *** priteau has quit IRC | 11:26 | |
| mnasiadka | yoctozepto: are we sure there are no binary packages? rpms? | 11:26 |
| *** bsanjeewa_ has quit IRC | 11:26 | |
| yoctozepto | mnasiadka: no, me not | 11:28 |
| hrw | added comment to curator issue | 11:37 |
| hrw | long comment | 11:37 |
| *** SvenKieske has quit IRC | 11:39 | |
| yoctozepto | libyaml is not found or a compiler error: forcing --without-libyaml | 11:50 |
| yoctozepto | did you try just installing it? | 11:50 |
| *** jonaspaulo has joined #openstack-kolla | 11:50 | |
| hrw | then I would need to install compiler and then I could just 'pip3 install cx_freeze' | 11:54 |
| *** priteau has joined #openstack-kolla | 12:01 | |
| *** wuchunyang has quit IRC | 12:01 | |
| *** shyamb has quit IRC | 12:03 | |
| *** abdysn has joined #openstack-kolla | 12:17 | |
| *** k_mouza has quit IRC | 12:27 | |
| *** thaulow has joined #openstack-kolla | 12:34 | |
| thaulow | Hello everyone! | 12:38 |
| thaulow | Fast question, I tried to upgrade from Train to Ussuri, and then I got this error: | 12:39 |
| thaulow | not valid: Valid values are [ussuri, train, 9.2, 15.0, 14.0, 13.0, 12.2, 12.1, 12.0, 11.1, 11.0, 10.1, 10.0], but found 'stein' | 12:39 |
| thaulow | I changed the config file and tried to redeploy, but then the value went back to "stein" again... | 12:39 |
| thaulow | Any idea how this can be changed permanently? :/ | 12:40 |
| *** k_mouza has joined #openstack-kolla | 12:41 | |
| mnasiadka | how can the value go back by itself? :) | 12:42 |
| thaulow | Good question... I edited the file from Stein to Ussuri, and even checked once again that it was in fact changed. Then I run the upgrade command, and then the file is back to Stein again... | 12:42 |
| thaulow | I have tried to look on the server for other refferences to Stein, and also through the Kolla Ansible project if there is any scripts that is defaulting, and overwriting the config file | 12:43 |
| thaulow | Also tried removing the line completely, (comes back again), and I tried to add a config file to ansible, but it did not work either | 12:43 |
| yoctozepto | thaulow, mnasiadka: which place is that? | 12:51 |
| mnasiadka | thaulow: yes, it would be good to point which file are you editing, and if you are using Kayobe, or only Kolla-Ansible :) | 12:52 |
| openstackgerrit | Pierre Riteau proposed openstack/kayobe master: [DNM] Debug kayobe-tox-ansible job https://review.opendev.org/759036 | 12:53 |
| thaulow | Changed all the files refferencing stein in /etc/kolla/config/ | 12:54 |
| thaulow | ironic oslo_config.cfg.ConfigFileValueError: Value for option pin_release_version from LocationInfo(location=<Locations.user: (4, True)>, detail='/etc/ironic/ironic.conf') isnot valid: Valid values are [ussuri, train, 9.2, 15.0, 14.0, 13.0, 12.2, 12.1, 12.0, 11.1, 11.0, 10.1, 10.0], but found 'stein'2020-10-20 22:47:07.945 6 ERROR ironic | 12:54 |
| thaulow | Only using Kolla-Ansible | 12:55 |
| *** k_mouza_ has joined #openstack-kolla | 12:55 | |
| *** k_mouza has quit IRC | 12:56 | |
| *** devfaz has joined #openstack-kolla | 13:01 | |
| *** k_mouza_ has quit IRC | 13:07 | |
| yoctozepto | wtos is that | 13:10 |
| yoctozepto | mgoddard, mnasiadka: some ironic weirdness for ya ^ | 13:11 |
| *** cah_link has quit IRC | 13:11 | |
| *** cah_link has joined #openstack-kolla | 13:12 | |
| *** k_mouza has joined #openstack-kolla | 13:26 | |
| priteau | thaulow: can you grep for `openstack_previous_release_name` in your global variables? | 13:33 |
| priteau | and in the kolla-ansible install you're using | 13:39 |
| *** abdysn has quit IRC | 13:39 | |
| openstackgerrit | Albert Braden proposed openstack/kolla master: mariadb: Fixes incremental backup failure when full not created today https://review.opendev.org/758486 | 13:40 |
| *** skramaja has quit IRC | 13:40 | |
| *** TrevorV has joined #openstack-kolla | 13:42 | |
| thaulow | Priteau, the variable isnt set in the global file | 13:45 |
| thaulow | Latest Kolla Ansible install | 13:45 |
| priteau | What about in your kolla-ansible installation, ansible/group_vars/all.yml | 13:45 |
| *** k_mouza has quit IRC | 13:54 | |
| *** k_mouza has joined #openstack-kolla | 13:54 | |
| thaulow | hmm that might actually be the issue... | 13:54 |
| thaulow | root@OsController1:~/kolla-ansible-10.1.0/ansible/group_vars# grep 'openstack_previous_release_name' all.yml openstack_previous_release_name: "train" | 13:54 |
| thaulow | Priteau | 13:55 |
| priteau | That looks fine | 13:56 |
| priteau | You're 100% sure this is the kolla-ansible version in use? | 13:56 |
| priteau | Because "stein" coming back when you deploy would imply you're using kolla-ansible train | 13:57 |
| yoctozepto | is it only ironic affected this weirdly? | 14:07 |
| yoctozepto | btw, kollars, apimon session live now: https://openinfra2020.netlify.app/a/event/24723 | 14:20 |
| *** thaulow has quit IRC | 14:21 | |
| *** wuchunyang has joined #openstack-kolla | 14:23 | |
| priteau | yoctozepto: only ironic is using this variable | 14:36 |
| yoctozepto | priteau: ack | 14:37 |
| *** wuchunyang has quit IRC | 14:38 | |
| *** wuchunyang has joined #openstack-kolla | 14:48 | |
| antonym | are there any reference kolla multinode inventory examples using cells? i haven't really been able to dig up a fully working example | 14:57 |
| *** wuchunyang has quit IRC | 14:59 | |
| priteau | mgoddard: I was just checking your WIP fernet update, for a 3-node cluster and default rotation values, the cron output looks much better. I see lots of tests are failing, I can update them if you like. | 14:59 |
| mgoddard | priteau: I only updated a few tests. I agree it looks better | 15:00 |
| yoctozepto | antonym: yes, the one used by our CI scenario; it's autogenerated so best take an example run: | 15:00 |
| yoctozepto | https://zuul.opendev.org/t/openstack/builds?job_name=kolla-ansible-centos8-source-cells | 15:00 |
| antonym | cool, i'll check there, thanks! | 15:00 |
| mgoddard | priteau: what I have not convinced myself about yet is whether it fixes all case | 15:00 |
| mgoddard | priteau: we currently do some rounding of the rotation interval to days/hours/minutes, depending on its size | 15:01 |
| yoctozepto | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_8bc/758952/1/check/kolla-ansible-centos8-source-cells/8bc2e9f/primary/logs/kolla_configs/inventory | 15:01 |
| yoctozepto | antonym: ^ | 15:01 |
| yoctozepto | 3 single-node cells | 15:01 |
| antonym | awesome, that should help | 15:01 |
| yoctozepto | and corresponding globals: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_8bc/758952/1/check/kolla-ansible-centos8-source-cells/8bc2e9f/primary/logs/kolla_configs/globals.yml | 15:02 |
| yoctozepto | you can play around browsing the dir structure at this address | 15:02 |
| *** wuchunyang has joined #openstack-kolla | 15:02 | |
| yoctozepto | please suggest doc improvements if it helped you and docs did not :-) | 15:02 |
| mgoddard | priteau: I suppose we need to be sure that rounding will not cause problems, or remove the rounding | 15:02 |
| mgoddard | priteau: possibly we need to define a min/max interval, and ensure we are within limits | 15:03 |
| antonym | yeah, once i get something working in the lab, i'll propose a few doc changes | 15:03 |
| priteau | The issue is that fernet_key_rotation_interval is defined in seconds, so can be within 59 seconds of the days/hours/minutes rounding? | 15:04 |
| mgoddard | antonym: have you seen the docs? https://docs.openstack.org/kolla-ansible/latest/reference/compute/nova-cells-guide.html | 15:04 |
| antonym | yeah, i used those but the inventory listed there seems to be a subset of the main inventory, so i wasn't sure if only that was needed or if i needed to add the cell nodes in other locations | 15:05 |
| antonym | when i ran it, the cells were missing things like the kolla-toolbox, etc | 15:05 |
| antonym | also server-bootstrap only seemed to ignore the cell nodes too | 15:05 |
| mgoddard | priteau: it's more that if the interval is > 1 day, we round it to whole days. Same for hours | 15:06 |
| priteau | oh I see | 15:06 |
| mgoddard | priteau: hmm, maybe I misremembered that | 15:06 |
| mgoddard | actually looks like it might be ok | 15:07 |
| mgoddard | so maybe the fix is good | 15:07 |
| mgoddard | would be nice to add more validation to the tests, to check the min/max rotation interval | 15:08 |
| mgoddard | I suppose if fernet_token_allow_expired_window >= fernet_token_expiry, we should always be able to make it fit | 15:09 |
| mnasiadka | mgoddard, yoctozepto: wdyt about removing using two Ceph keys from nova-cell role, and just use one (cinder) - as it's in the Ceph docs? | 15:10 |
| mgoddard | otherwise there may be some awkward remainder that we cannot use | 15:10 |
| *** Fl1nt has quit IRC | 15:11 | |
| mgoddard | mnasiadka: is there ever a use case for them to be different? | 15:11 |
| *** Fl1nt has joined #openstack-kolla | 15:11 | |
| mnasiadka | mgoddard: there was with kolla-ceph, because we generated two (god knows why) | 15:11 |
| Fl1nt | Hi everyone! | 15:11 |
| mnasiadka | mgoddard: for example if you have a different user for nova root disks and a different one for cinder volumes | 15:12 |
| mnasiadka | mgoddard: we could just make it configurable, if you need just one, or you need multiple | 15:12 |
| Fl1nt | mgoddard, the yoctozepto's change that you abandonned: https://review.opendev.org/758057 was exactly why I initially did the whole work about being able to detect python-2.x and python-3.x on this change. | 15:13 |
| patchbot | patch 758057 - kolla (stable/train) - Improve pip install process for offline deployment. (ABANDONED) - 1 patch set | 15:13 |
| mgoddard | Fl1nt: if you can make it work, feel free | 15:14 |
| mnasiadka | mgoddard: but I'd say our current approach is not really good, we enforce users to upload two keys that are identical :) | 15:14 |
| Fl1nt | if you look at the history of this change, I initially did the appropriate patch to make it work on both PY2.x and PY3.x and then decided to abbandon the PY2.x and PY3.x compatibility patch because of wu.chunyang comment of sept 18 4:40AM | 15:15 |
| priteau | mgoddard: the cron generator doesn't care about token expiry / allow expired settings right? we just need to make sure it creates a cronjob at the requested interval across N nodes? | 15:15 |
| mgoddard | Fl1nt: right, we don't need py2 in ussuri+ | 15:16 |
| mgoddard | Fl1nt: if you want to add py2 support to the train patch, go for it | 15:16 |
| Fl1nt | yep, so if we want to got it on train I can do the patch. | 15:16 |
| Fl1nt | ok cool | 15:17 |
| mgoddard | priteau: right. however, it doesn't really honour the interval passed to it, since it has to fit the rotations into a week | 15:17 |
| mgoddard | priteau: so while it doesn't care what they represent, it could be passed a min & max interval | 15:18 |
| mgoddard | currently we are just passing max (but treating is as a min, which was the source of the bug) | 15:18 |
| mnasiadka | well, rotation should be somewhere between min (fernet_token_expiry) and max (fernet_token_expiry + expired_window) | 15:20 |
| mnasiadka | closer to min the better I guess, the expired_window usage should be a last resort if e.g. a controller fails | 15:20 |
| yoctozepto | mgoddard, mnasiadka: we can certainly have nova with rbd and no cinder | 15:24 |
| yoctozepto | but it does not smell production-grade :-) | 15:25 |
| Fl1nt | mgoddard, regarding cloudkitty, I can confirm that it needs a default metrics.yml when dealing with prometheus collector, I'll add that to my CK patch btw. | 15:25 |
| Fl1nt | works like a charm now ^^ | 15:25 |
| mnasiadka | yoctozepto: we often have cinder with rbd and nova just attaching volumes from ceph, but using local filesystem for root/ephemeral disks | 15:26 |
| yoctozepto | mgoddard, mnasiadka: render unto Cinder the things that are Cinder's | 15:26 |
| mnasiadka | yoctozepto: but it still doesn't make sense to default to a config, that is not the recommended approach in ceph docs | 15:26 |
| yoctozepto | mnasiadka: yeah and libvirt then needs the cinder key, not nova | 15:26 |
| yoctozepto | mnasiadka: please propose your approach, I can consider it | 15:27 |
| yoctozepto | mnasiadka: now my head's elsewhere | 15:27 |
| mnasiadka | yeah, will do in W | 15:27 |
| yoctozepto | mnasiadka: ok | 15:27 |
| yoctozepto | I hope to find some time for upstream tomorrow | 15:29 |
| *** dmsimard1 has joined #openstack-kolla | 15:30 | |
| Fl1nt | mnasiadka, yoctozepto this issue can be solved if we create a openstack keyring having access to volumes/backups/vms rbd pool | 15:30 |
| Fl1nt | we can't use nova with nova as libvirt call cinder whatever. | 15:31 |
| Fl1nt | no | 15:31 |
| mnasiadka | Fl1nt: it's not an issue of what you can create on ceph side, but the amount of keys K-A external ceph functionality requires from a user today. | 15:31 |
| Fl1nt | only one | 15:31 |
| mnasiadka | not really | 15:31 |
| *** dmsimard has quit IRC | 15:32 | |
| *** dmsimard1 is now known as dmsimard | 15:32 | |
| mnasiadka | in theory two, you can set them to the same name, so we copy them twice, under two different names into libvirt config... | 15:32 |
| Fl1nt | not really because we have all these tasks doing dumb things, but we could actually decide to get only one key | 15:32 |
| Fl1nt | I only have two personnally and one of our staging cluster using only 1 'openstack' | 15:33 |
| Fl1nt | Do we want one unique key knowing that whatever happen, nova will require the cinder one. | 15:33 |
| mgoddard | priteau: reading https://docs.openstack.org/keystone/latest/admin/fernet-token-faq.html again, perhaps there is no min. We just set max_active_keys to compensate. The only limit is expiry | 15:36 |
| mnasiadka | mgoddard: we could just rotate once a year and get over with it :D | 15:37 |
| yoctozepto | xD | 15:38 |
| yoctozepto | let users rotate themselves | 15:38 |
| priteau | Wasn't this the plot in Lost? | 15:38 |
| mgoddard | what I don't understand - the rotation changes merged in june 2019. How are we only just seeing issues? | 15:38 |
| yoctozepto | xDDDD | 15:38 |
| yoctozepto | magic | 15:38 |
| mnasiadka | mgoddard: so, the only issue is the check I introduced in keystone-startup.sh that checks if key is stale | 15:39 |
| mgoddard | my theory - the defaults lead to the stale keys existing on a Sunday | 15:39 |
| mnasiadka | yes, but nobody noticed this before we introduced the check in keystone-startup.sh | 15:39 |
| mgoddard | but why not? | 15:40 |
| mnasiadka | therefore - nobody uses OpenStack on Sunday | 15:40 |
| mgoddard | seems not | 15:40 |
| mnasiadka | or - the expiry and expire window stuff in Keystone is broken and doesn't check for anything :) | 15:40 |
| mnasiadka | I would vote for the Sunday thing still ;) | 15:40 |
| mgoddard | mnasiadka: did you get anywhere using a short expiry in CI? | 15:41 |
| mnasiadka | mgoddard: it gets rotated (also on Ubuntu after the patch), the only problem is in the upgrade jobs - where we lower the expiry time, but that's other issue | 15:42 |
| mnasiadka | https://review.opendev.org/#/c/758434/ | 15:42 |
| patchbot | patch 758434 - kolla-ansible - [DNM]: Keystone fernet rotation testing - 4 patch sets | 15:42 |
| *** k_mouza has quit IRC | 15:43 | |
| mgoddard | mnasiadka: did you verify that it fails without the change in config.yml? | 15:44 |
| mnasiadka | actually I didn't notice centos8 singlenode job fails on reconfigure | 15:45 |
| openstackgerrit | Merged openstack/kolla-ansible stable/train: baremetal: Install iptables for Docker if enabled https://review.opendev.org/758071 | 15:45 |
| mnasiadka | patchset 3 is without change in config.yml | 15:45 |
| *** jbadiapa is now known as jbadiapa|away | 15:46 | |
| priteau | mgoddard: because few people actually use long lived tokens? | 15:48 |
| mgoddard | priteau: but isn't this about the key expiry? | 15:48 |
| priteau | You're right, I guess this could be an issue if we were rotating too frequently | 15:52 |
| *** cah_link has quit IRC | 15:53 | |
| *** cah_link has joined #openstack-kolla | 15:54 | |
| priteau | If I understand correctly, the current script can end up under rotating. We haven't seen it as an issue (expect with the new stale check) because it's only a low security issue? We could keep the same fernet key forever and OpenStack would still work, right? | 15:55 |
| *** bengates has quit IRC | 15:57 | |
| mgoddard | I think you are right priteau | 16:04 |
| *** rpittau is now known as rpittau|afk | 16:04 | |
| mgoddard | in that case, does the keystone startup check make sense? | 16:05 |
| mgoddard | well, it makes sense to check if there is a token | 16:05 |
| mgoddard | s/token/key/ | 16:05 |
| mgoddard | but should we fail if the key is older than the rotation interval? | 16:06 |
| mgoddard | and maybe my proposed patch is wrong | 16:08 |
| mgoddard | should it rather be the max_active_keys we adjust, based on the largest rotation interval? | 16:08 |
| mnasiadka | well, max_active_keys allows you to have multiple active tokens, in case you rotate more often than the key_expiry? | 16:11 |
| priteau | Yeah, maybe we should go with a fixed rotation period, like in the keystone example, and adjust the number of keys accordingly | 16:13 |
| mnasiadka | For example, your token is valid for 24 hours and we want to rotate keys every six hours. We will need to make sure tokens that were created at 08:00 AM on Monday are still valid at 07:00 AM on Tuesday, assuming they were not prematurely revoked. To accomplish this, we will want to make sure we set max_active_keys=6 in our keystone configuration file. This will allow us to hold all keys that might still be | 16:13 |
| mnasiadka | required to validate a previous token, but keeps the key repository limited to only the keys that are needed. | 16:13 |
| mnasiadka | that's from the FAQ | 16:13 |
| *** k_mouza has joined #openstack-kolla | 16:15 | |
| yoctozepto | Victor shows kolla-ansible tricks: https://openinfra2020.netlify.app/a/event/24657 | 16:15 |
| openstackgerrit | Albert Braden proposed openstack/kolla master: mariadb: Fixes incremental backup failure when full not created today https://review.opendev.org/758486 | 16:18 |
| *** k_mouza has quit IRC | 16:19 | |
| *** TrevorV has quit IRC | 16:20 | |
| *** k_mouza has joined #openstack-kolla | 16:26 | |
| *** TrevorV has joined #openstack-kolla | 16:26 | |
| *** k_mouza has quit IRC | 16:28 | |
| *** TrevorV has quit IRC | 16:39 | |
| *** wuchunyang has quit IRC | 16:41 | |
| *** TrevorV has joined #openstack-kolla | 16:45 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: WIP: Fix keystone cron generator https://review.opendev.org/758797 | 16:49 |
| mgoddard | priteau: ^ found an issue with the >1day intervals. Even though it was more frequent, it was still lumpy. It now gets the hours right. Possibly other calcs are wrong though | 16:51 |
| priteau | Thanks Mark | 16:51 |
| *** gfidente is now known as gfidente|afk | 16:51 | |
| *** cah_link has quit IRC | 16:53 | |
| *** Trevor_V has joined #openstack-kolla | 16:54 | |
| *** TrevorV has quit IRC | 16:55 | |
| *** dardelean has quit IRC | 17:08 | |
| ebbex | so i'm following the kayobe/train documentation, i've created a venv (kayobe), upgraded pip, installed kayobe 7.1.0, copied kayobe-config/stable/train to /etc/kayobe, updated inventory with [seed-hypervisor] and [seed]. I run `kayobe seed hypervisor host configure` and "ERROR! the role 'singleplatform-eng.users' was not found in ..." | 17:10 |
| ebbex | in ~/venvs/kayobe/share/kayobe/ansible/roles/ there's a bunch of roles, but no 'singleplatform-eng', did i miss something? | 17:11 |
| *** dardelean has joined #openstack-kolla | 17:12 | |
| yoctozepto | mgoddard: it could be that I asked it already and forgot but is there a place where bikolla progress is tracked? an etherpad for example? | 17:15 |
| *** Fl1nt has quit IRC | 17:19 | |
| *** e0ne has quit IRC | 17:33 | |
| *** Luzi has joined #openstack-kolla | 17:35 | |
| *** williampiv has quit IRC | 17:43 | |
| *** priteau has quit IRC | 18:01 | |
| *** k_mouza has joined #openstack-kolla | 18:04 | |
| *** k_mouza has quit IRC | 18:08 | |
| *** Luzi has quit IRC | 18:09 | |
| mgoddard | hi ebbex | 18:09 |
| mgoddard | you nee to run kayobe control host bootstrap | 18:10 |
| mgoddard | https://docs.openstack.org/kayobe/latest/deployment.html#ansible-control-host | 18:10 |
| mgoddard | yoctozepto: https://storyboard.openstack.org/#!/story/2004293 | 18:11 |
| yoctozepto | mgoddard: ok; in other words, there has not been much progress on it | 18:14 |
| yoctozepto | mgoddard: what is lacking? could you drop a few words on that story? | 18:14 |
| yoctozepto | (and I want them to be non-random words!) | 18:14 |
| mgoddard | yoctozepto: zero progress | 18:15 |
| mgoddard | yoctozepto: it's a big job, mainly due to migrating existing systems | 18:15 |
| mgoddard | yoctozepto: also all of the user config would have to change, so it will be disruptive | 18:17 |
| ebbex | mgoddard: i ended up running ansible-galaxy install -r ..requirements.yml, but this seems much better, now the roles go into the venv, cheers! | 18:19 |
| yoctozepto | mgoddard: why all the config? | 18:23 |
| mgoddard | yoctozepto: bifrost.yml -> second globals.yml | 18:23 |
| yoctozepto | mgoddard: hmm, indeed | 18:23 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: [DNM] Check centos-8-stream https://review.opendev.org/759121 | 18:25 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: [DNM] Check centos-8-stream https://review.opendev.org/759121 | 18:31 |
| *** primeministerp has joined #openstack-kolla | 18:33 | |
| *** primeministerp has quit IRC | 18:39 | |
| *** primeministerp has joined #openstack-kolla | 18:39 | |
| ebbex | is seed-hypervisor centos/redhat only? | 18:43 |
| *** primeministerp has quit IRC | 18:44 | |
| *** primeministerp has joined #openstack-kolla | 18:44 | |
| yoctozepto | ebbex: yes | 19:12 |
| yoctozepto | ebbex: cento | 19:12 |
| yoctozepto | ebbex: for ubuntu we welcome help :-) | 19:12 |
| *** jbadiapa|away has quit IRC | 19:46 | |
| *** bsanjeewa has joined #openstack-kolla | 19:50 | |
| *** dciabrin_ has joined #openstack-kolla | 19:51 | |
| *** dciabrin has quit IRC | 19:54 | |
| *** k_mouza has joined #openstack-kolla | 19:55 | |
| *** vishalmanchanda has quit IRC | 19:57 | |
| *** muhaha has joined #openstack-kolla | 19:57 | |
| yoctozepto | mnasiadka: OVN unhappy: 2020-10-21 19:20:21 clean_up CreateNetwork: Error while executing command: BadRequestException: 400, Invalid input for operation: Requested MTU is too big, maximum is 1392. | 19:58 |
| *** k_mouza has quit IRC | 20:00 | |
| *** williampiv has joined #openstack-kolla | 20:00 | |
| *** priteau has joined #openstack-kolla | 20:02 | |
| *** priteau has left #openstack-kolla | 20:03 | |
| *** priteau has joined #openstack-kolla | 20:05 | |
| *** stingrayza has quit IRC | 20:09 | |
| *** k_mouza has joined #openstack-kolla | 20:26 | |
| *** jonaspaulo has quit IRC | 20:28 | |
| *** k_mouza has quit IRC | 20:31 | |
| *** stingrayza has joined #openstack-kolla | 20:33 | |
| *** bsanjeewa has quit IRC | 20:48 | |
| *** e0ne has joined #openstack-kolla | 21:05 | |
| *** e0ne has quit IRC | 21:11 | |
| *** e0ne has joined #openstack-kolla | 21:12 | |
| *** bsanjeewa has joined #openstack-kolla | 21:21 | |
| *** kemopq has quit IRC | 21:21 | |
| *** muhaha has quit IRC | 21:30 | |
| *** bsanjeewa has quit IRC | 21:31 | |
| *** bsanjeewa has joined #openstack-kolla | 21:31 | |
| *** bsanjeewa_ has joined #openstack-kolla | 21:32 | |
| *** bsanjeewa has quit IRC | 21:36 | |
| *** gfidente|afk has quit IRC | 22:20 | |
| *** e0ne has quit IRC | 22:33 | |
| *** k_mouza has joined #openstack-kolla | 22:35 | |
| *** k_mouza has quit IRC | 22:39 | |
| *** bsanjeewa_ has quit IRC | 22:43 | |
| *** Trevor_V has quit IRC | 22:55 | |
| *** dardelean has quit IRC | 23:59 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!