Friday, 2020-10-16

*** k_mouza has joined #openstack-kolla00:15
*** k_mouza has quit IRC00:20
*** k_mouza has joined #openstack-kolla00:30
*** k_mouza has quit IRC00:35
*** k_mouza has joined #openstack-kolla00:39
*** k_mouza has quit IRC00:43
*** k_mouza has joined #openstack-kolla01:02
*** k_mouza has quit IRC01:06
*** k_mouza has joined #openstack-kolla01:08
*** k_mouza has quit IRC01:13
*** JamesBenson has quit IRC02:22
*** JamesBenson has joined #openstack-kolla02:22
*** openstackgerrit has quit IRC03:57
*** zzzeek has quit IRC04:06
*** zzzeek has joined #openstack-kolla04:23
*** zzzeek has quit IRC04:25
*** zzzeek has joined #openstack-kolla04:27
*** also_stingrayza has joined #openstack-kolla04:27
*** stingrayza has quit IRC04:30
*** evrardjp has quit IRC04:33
*** evrardjp has joined #openstack-kolla04:33
*** dave-mccowan has quit IRC04:36
*** JamesBenson has quit IRC04:42
*** JamesBenson has joined #openstack-kolla04:43
*** JamesBenson has quit IRC04:48
*** nikparasyr has joined #openstack-kolla04:55
*** zzzeek has quit IRC05:09
*** jbadiapa has joined #openstack-kolla05:13
*** zzzeek has joined #openstack-kolla05:16
*** openstackgerrit has joined #openstack-kolla05:17
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible master: Use a better process name for httpd subprocess  https://review.opendev.org/75851905:17
*** JamesBenson has joined #openstack-kolla05:24
*** JamesBenson has quit IRC05:29
yoctozeptomorning05:57
mnasiadkamorning05:58
*** k_mouza has joined #openstack-kolla05:59
mnasiadkamgoddard: what is interesting, we never changed the rotation script or the crontab file in keystone, so wondering from where does it come from06:00
mnasiadkamgoddard: or maybe we just need to drop the check for a stale token, or just run the rotation if we discover a stale token06:01
mnasiadkayoctozepto: ^^06:01
mnasiadkaif it's stale - it means the cron job didn't rotate it at all...06:01
yoctozeptoimpossible06:01
yoctozeptoI mean, I believe I was getting it rotated06:02
yoctozeptowill check it later06:02
*** k_mouza has quit IRC06:04
mnasiadkahttps://review.opendev.org/#/c/758434/06:09
patchbotpatch 758434 - kolla-ansible - [DNM]: Keystone fernet rotation testing - 3 patch sets06:09
mnasiadkathis change shows it's getting rotated on multinode jobs06:09
mnasiadka(ignore the upgrade jobs fails - if you change token expiry timings during upgrade, the container will be stuck restarting claiming the tokens are stale)06:10
*** cah_link has joined #openstack-kolla06:13
*** iniazi has joined #openstack-kolla06:23
*** jbalciunas has quit IRC06:38
*** jbalciunas has joined #openstack-kolla06:39
*** vishalmanchanda has joined #openstack-kolla06:41
*** jbalciunas has quit IRC06:43
*** mchlumsky has quit IRC06:50
*** mchlumsky has joined #openstack-kolla06:51
*** bengates has joined #openstack-kolla07:02
*** maharg102 has joined #openstack-kolla07:11
*** maharg101 has quit IRC07:12
*** jbalciunas has joined #openstack-kolla07:19
yoctozeptoyeah, I didn't mean to pick on upgrades07:36
mnasiadkabut it doesn't mean we shouldn't try to fix it :)07:37
mnasiadkabut first let's focus on why people are coming with stale tokens07:37
*** dougsz has joined #openstack-kolla07:38
*** devfaz has quit IRC07:50
*** devfaz has joined #openstack-kolla07:50
*** devfaz has quit IRC07:51
*** devfaz has joined #openstack-kolla07:52
*** wuchunyang has joined #openstack-kolla07:55
*** Tengu has quit IRC08:00
yoctozeptoall my Train deployments churning happily08:08
yoctozeptoI might not have used the latest k+k-a though08:08
yoctozeptomnasiadka: do you think it could be Ussuri+?08:09
yoctozeptodo we have reports of failures on Train?08:09
mnasiadkayoctozepto: kplant has on centos7+train I think08:11
mnasiadkabut without logging of the crontab script, we're a bit blind08:11
yoctozeptoduh, true08:11
mnasiadkanot speaking about the fact crond only sends logs via syslog08:11
yoctozeptokplant: could you confirm + give versions of kolla (used to build images) and k-a (used to deploy)08:12
yoctozeptomnasiadka, kplant: and was it standard 3-controller deployment?08:12
mnasiadkayoctozepto: other stupid question, what is the difference between plugins and additions in Kolla? I can't seem to find any difference in the code :)08:19
yoctozeptomnasiadka: I think I asked this question once; there might be none :-)08:20
yoctozeptomgoddard might know08:20
*** Tengu has joined #openstack-kolla08:22
mnasiadkayoctozepto: I see somebody was lazy, instead of adding a check if plugins directory contains setup.cfg or something similar - he created additions - https://github.com/openstack/kolla/commit/9b1e519267f022ceb3f920976591a8f8233addc408:24
yoctozeptomgoddard, mnasiadka: wdyt about approaching https://bugs.launchpad.net/kolla-ansible/+bug/1837551 by using KOLLA_SKIP for rp_filter in Victoria (with a relevant upgrade note anyhow) and setting as wontfix in earlier?08:24
openstackLaunchpad bug 1837551 in kolla-ansible victoria "rp_filter not set properly" [Medium,Triaged]08:24
yoctozeptomnasiadka: eh! :D08:24
mnasiadkayoctozepto: life :)08:25
mnasiadkayoctozepto: trivialfix - https://review.opendev.org/#/c/757076/08:26
patchbotpatch 757076 - kolla-ansible - Fix fernet cron path on Ubuntu/Debian - 3 patch sets08:26
openstackgerritMerged openstack/kolla-ansible stable/ussuri: baremetal: Install iptables for Docker if enabled  https://review.opendev.org/75807008:28
yoctozeptomnasiadka: trivial but should not you mark some bug as closed? and write a reno? It's pretty important08:30
yoctozeptomnasiadka: so, basically, ubuntu users have been missing rotations for at least 2 years? nice08:30
yoctozeptoosmanlicilegi: ^ check this out08:30
mnasiadkayoctozepto: basically it seems yes08:35
hrwelo08:35
mnasiadkaand our fernet tokens (token lifetime + expire window) is 3 days08:35
mnasiadkaand crontab on standard values rotates the keys every ~4 days (midnight on Wed and Sun)08:36
*** Tengu has quit IRC08:36
yoctozeptomnasiadka: well, that's problematic, but still better if rotations DO take place xD08:39
mnasiadkahaha, right08:39
*** k_mouza has joined #openstack-kolla08:41
*** Tengu has joined #openstack-kolla08:43
* osmanlicilegi is back08:49
*** bengates has quit IRC08:50
*** bengates has joined #openstack-kolla08:51
osmanlicilegiyoctozepto: I will. I'm still trying to catch up what I have missed for the last few weeks :]08:52
*** k_mouza has quit IRC08:55
SvenKieskeyoctozepto: regarding the failing fernet token rotation: yes, we experienced this on train.. brought our containers in a restart loop08:58
SvenKieskethis is a 3 controlnodes deployment on ubuntu08:59
SvenKieskeglad it is fixed now08:59
*** cah_link1 has joined #openstack-kolla09:05
*** brinzhang_ has quit IRC09:05
*** jbadiapa has quit IRC09:05
*** cah_link has quit IRC09:05
*** cah_link1 is now known as cah_link09:05
*** brinzhang_ has joined #openstack-kolla09:05
*** jbadiapa has joined #openstack-kolla09:06
yoctozeptoSvenKieske: but Train AND Ubuntu, no?09:12
openstackgerritBartosz Bezak proposed openstack/kolla master: rally: add rally-openstack in source - follow up  https://review.opendev.org/75854009:17
*** kevko has joined #openstack-kolla09:19
*** JamesBenson has joined #openstack-kolla09:26
*** k_mouza has joined #openstack-kolla09:26
*** k_mouza has quit IRC09:29
*** k_mouza has joined #openstack-kolla09:29
*** JamesBenson has quit IRC09:30
*** brinzhang_ has quit IRC09:48
*** jonaspaulo has joined #openstack-kolla09:48
SvenKieskeyoctozepto: yes10:07
SvenKieskewe had no key rotations since april (when we last deployed the containers), pretty unfortunate.. lucky it is still a development environment10:08
SvenKieskecan someone maybe shed some light on my question regarding rabbitmq interface configuration from yesterday? http://lists.openstack.org/pipermail/openstack-discuss/2020-October/018014.html I'm also willing to provide patches, if needed, we already signed the corp. cla10:09
*** wuchunyang has quit IRC10:13
mnasiadkaSvenKieske: that was long time ago, either those options did not work with new rabbitmq, or Paul didn't think those options were relevant - if you'd like to raise a change to add that support back - we'd be happy to help10:40
*** Tengu has quit IRC10:41
openstackgerritPierre Riteau proposed openstack/kayobe stable/stein: CI: Fix kayobe-tox-molecule job  https://review.opendev.org/75855010:44
openstackgerritPierre Riteau proposed openstack/kayobe stable/stein: Performance: skip LVM configuration if no groups configured  https://review.opendev.org/75807510:45
*** Tengu has joined #openstack-kolla10:53
*** JamesBenson has joined #openstack-kolla11:00
*** wuchunyang has joined #openstack-kolla11:07
*** JamesBenson has quit IRC11:10
*** JamesBenson has joined #openstack-kolla11:10
openstackgerritMichal Nasiadka proposed openstack/kolla master: WIP: use upper constraints in openstack-base  https://review.opendev.org/75855311:11
*** shyamb has joined #openstack-kolla11:15
yoctozeptoSvenKieske: +2 for adding it back; I firewalled it away for myself but makes sense to bind it more tightly11:18
*** shyamb has quit IRC11:46
SvenKieskeyoctozepto: mnasiadka: okay will prepare this, my colleague has already some experience with contributing patches so I guess I can figure out the correct workflow for myself, will ask if I got any questions left.12:02
kplantyoctozepto, mnasiadka: centos7+train (stable/train from git)12:05
kplantthree controller, two network, three monitor, N compute12:05
yoctozeptoand you are getting permanently stale tokens?12:05
kplantnot sure tbh, the deployment is only a few days old12:06
openstackgerritMichal Nasiadka proposed openstack/kolla master: version-check: Use independent release data as default  https://review.opendev.org/75855612:10
mnasiadkakplant: just to be sure, your crontab runs rotation script on sundays and wednesdays on midnight? and no crontab entry on the third host?12:11
*** jbalciunas has quit IRC12:11
kplanti believe that's correct12:12
kplantlet me recheck12:12
*** jbalciunas has joined #openstack-kolla12:12
openstackgerritMichal Nasiadka proposed openstack/kolla master: version-check: Use independent release data as default  https://review.opendev.org/75855612:12
kplantyeah: 0 - sunday, 3 - wednesday, nothing on 3rd control -- you got it12:14
mnasiadkakplant: given that keystone fernet token should be valid 3 days, that gives a bit of a time window for them to be invalid...12:15
kplantsure does12:16
kplantnot sure why they were invalid yesterday though12:17
kplantthursday should have been perfect12:17
*** jbalciunas has quit IRC12:18
mnasiadkaSo, fernet token expiry is 1 day + we allow for 2 days extra time (in keystone config)12:18
mnasiadkaI’ll look into the script that generates crontabs and see what we can do12:18
mnasiadkaWe should rather rotate them every day and leave those 2 days for ,,contingency’’12:20
*** wuchunyang has quit IRC12:31
kplantdid anything change within stable/train ?12:31
kplanti've never experienced this is any of my other deployments12:31
ozzzomy change failed zuul: https://review.opendev.org/#/c/758486/12:33
patchbotpatch 758486 - kolla - Closes bug 1897948 - 1 patch set12:33
openstackbug 1897948 in kolla-ansible "Incorrect setting for --incremental-history-name in kolla/docker/mariadb/backup.sh" [Wishlist,Triaged] https://launchpad.net/bugs/189794812:33
ozzzoThe error messaages are pretty clear and I see what needs to be done to the code, but I'm a bit foggy on the procedure for submitting an updated review12:33
ozzzodo I just do git commit and git review again, or is there a different procedure?12:33
ozzzoI think I found it; it looks like I need to do: git commit -a --amend12:38
ozzzotrying that now12:38
openstackgerritMerged openstack/kolla stable/train: Checks for heat_user_domain explicitly  https://review.opendev.org/75805912:47
openstackgerritMerged openstack/kolla stable/stein: Checks for heat_user_domain explicitly  https://review.opendev.org/75806012:47
openstackgerritMerged openstack/kolla stable/stein: Bump versions for Stein  https://review.opendev.org/75756612:47
openstackgerritMerged openstack/kayobe stable/stein: CI: Fix kayobe-tox-molecule job  https://review.opendev.org/75855012:49
openstackgerritAlbert Braden proposed openstack/kolla master: Closes bug 1897948  https://review.opendev.org/75848612:52
openstackbug 1897948 in kolla-ansible "Incorrect setting for --incremental-history-name in kolla/docker/mariadb/backup.sh" [Wishlist,Triaged] https://launchpad.net/bugs/189794812:52
*** jbalciunas has joined #openstack-kolla13:07
openstackgerritMerged openstack/kolla stable/train: Bump versions for Train  https://review.opendev.org/75804713:12
openstackgerritMerged openstack/kolla stable/ussuri: Add sysfsutils to nova-compute ubuntu source  https://review.opendev.org/75805113:12
openstackgerritMerged openstack/kolla stable/train: Add sysfsutils to nova-compute ubuntu source  https://review.opendev.org/75805213:12
openstackgerritMerged openstack/kolla stable/stein: Add sysfsutils to nova-compute ubuntu source  https://review.opendev.org/75805313:12
mnasiadkakplant: this is part of fernet improvements we backported all the way to train13:15
mnasiadkakplant: probably the thing with checking if keystone fernet token is stale on keystone startup is causing problems, but then we shouldn't have such old tokens :)13:27
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: [DNM]: Keystone fernet rotation testing  https://review.opendev.org/75843413:35
*** priteau has joined #openstack-kolla13:39
*** jbalciunas has quit IRC14:17
*** dave-mccowan has joined #openstack-kolla14:20
*** vishalmanchanda has quit IRC14:20
*** jbalciunas has joined #openstack-kolla14:23
openstackgerritSven Kieske proposed openstack/kolla-ansible master: re-add rabbitmq config for interfaces  https://review.opendev.org/75857614:29
*** hrw has quit IRC14:32
SvenKieskemy first proposal, so I hope everything fits, you can ping me today about any needed changes, till about 19:00 CEST, or on monday again.14:50
mnasiadkayoctozepto: ^^ we want to treat it as a bug, or new functionality? :)14:51
SvenKieskewell, yeah, I was asking that myself. If you need a blueprint for that, or something like that, just give me heads up, will have to read up on how to create those, though.14:56
*** cah_link has quit IRC15:00
mnasiadkaSvenKieske: naah, no blueprints, just I'm thinking if we want to backport that to stable branches, or not - but I guess it would be nice to get this functionality back in stable branches.15:00
*** jbalciunas has quit IRC15:02
*** nikparasyr has left #openstack-kolla15:03
openstackgerritMerged openstack/kayobe stable/stein: Performance: skip LVM configuration if no groups configured  https://review.opendev.org/75807515:03
mnasiadkaSvenKieske: please raise a bug that this is missing without any deprecation notice, and add "Closes-Bug: #bug_id" in the commit message - you'll also need a release note (please read about reno tool usage somewhere in openstack docs).15:04
SvenKieskeokay, will do15:07
openstackgerritSven Kieske proposed openstack/kolla-ansible master: re-add rabbitmq config for interfaces  https://review.opendev.org/75857615:18
SvenKieskeChange is updated :)15:19
*** jbalciunas has joined #openstack-kolla15:20
*** e0ne has quit IRC15:33
*** bengates has quit IRC15:38
SvenKieskeI'll check for the release notes bits now15:48
*** cah_link has joined #openstack-kolla15:52
openstackgerritDoug Szumski proposed openstack/kolla-ansible stable/ussuri: Use become when copying custom Fluentd config  https://review.opendev.org/75858815:56
SvenKieskemnasiadka: okay I did as you ask, beside the doc change, that one will take some time to figure out.. your docs on how to handle docs are huge! will have to read up on how this is supposed to work.16:00
SvenKieskemnasiadka: yes, I agree this should be backported, in fact, we still use train release (need to upgrade soonish[TM]) and would rather use upstreamed patches than our own.16:03
*** cah_link has quit IRC16:04
*** cah_link has joined #openstack-kolla16:05
*** cah_link has quit IRC16:12
*** k_mouza has quit IRC16:24
*** k_mouza has joined #openstack-kolla16:31
*** dougsz has quit IRC16:31
SvenKieskemnasiadka: should I add the documentation to the same Changeset as the code? or maybe anyone else can answer this also?16:33
*** k_mouza has quit IRC16:35
*** k_mouza has joined #openstack-kolla16:36
*** k_mouza has quit IRC16:36
yoctozeptomnasiadka: fine to be a bugfix I guess; it broke at some point heh16:44
*** k_mouza has joined #openstack-kolla16:44
yoctozeptoSvenKieske: yes, best would be the same16:44
*** jbalciunas has quit IRC16:44
*** k_mouza has quit IRC16:49
openstackgerritSven Kieske proposed openstack/kolla-ansible master: re-add rabbitmq config for interfaces  https://review.opendev.org/75857616:49
SvenKieskedone16:49
*** k_mouza has joined #openstack-kolla16:52
*** k_mouza has quit IRC16:56
*** hrw has joined #openstack-kolla17:00
*** maharg102 has quit IRC17:01
*** k_mouza has joined #openstack-kolla17:26
*** k_mouza has quit IRC17:31
*** k_mouza has joined #openstack-kolla17:37
*** k_mouza has quit IRC17:42
*** k_mouza has joined #openstack-kolla17:46
*** k_mouza has quit IRC17:51
*** k_mouza has joined #openstack-kolla17:55
*** k_mouza has quit IRC17:59
*** k_mouza has joined #openstack-kolla18:07
*** k_mouza has quit IRC18:11
*** k_mouza has joined #openstack-kolla18:15
*** dougsz has joined #openstack-kolla18:16
*** jonaspaulo has quit IRC18:19
*** k_mouza has quit IRC18:20
*** k_mouza has joined #openstack-kolla18:20
*** cah_link has joined #openstack-kolla18:22
*** k_mouza has quit IRC18:25
*** k_mouza has joined #openstack-kolla18:31
*** k_mouza has quit IRC18:35
*** dougsz has quit IRC18:57
*** maharg101 has joined #openstack-kolla18:58
*** jbadiapa has quit IRC19:03
*** maharg101 has quit IRC19:04
openstackgerritMerged openstack/kolla stable/ussuri: Checks for heat_user_domain explicitly  https://review.opendev.org/75805819:34
*** priteau has quit IRC19:40
*** zzzeek has quit IRC19:43
*** zzzeek has joined #openstack-kolla19:45
openstackgerritMerged openstack/kolla-ansible stable/ussuri: Performance: use a single config file for fluentd  https://review.opendev.org/75781320:06
openstackgerritMerged openstack/kolla-ansible stable/train: Performance: use a single config file for fluentd  https://review.opendev.org/75781620:06
openstackgerritMerged openstack/kolla-ansible stable/stein: Performance: use a single config file for fluentd  https://review.opendev.org/75781720:06
openstackgerritMerged openstack/kolla-ansible master: Update release note for httpd keep alive  https://review.opendev.org/74651220:06
*** kevko has quit IRC20:25
*** kevko has joined #openstack-kolla20:29
*** dswebb has quit IRC21:24
*** rgogunskiy has quit IRC21:36
*** dave-mccowan has quit IRC22:00
*** dave-mccowan has joined #openstack-kolla22:04
*** stingrayza has joined #openstack-kolla22:34
*** also_stingrayza has quit IRC22:37
*** dave-mccowan has quit IRC23:12
*** dave-mccowan has joined #openstack-kolla23:31
*** dave-mccowan has quit IRC23:55

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!