| *** k_mouza has joined #openstack-kolla | 00:15 | |
| *** k_mouza has quit IRC | 00:20 | |
| *** k_mouza has joined #openstack-kolla | 00:30 | |
| *** k_mouza has quit IRC | 00:35 | |
| *** k_mouza has joined #openstack-kolla | 00:39 | |
| *** k_mouza has quit IRC | 00:43 | |
| *** k_mouza has joined #openstack-kolla | 01:02 | |
| *** k_mouza has quit IRC | 01:06 | |
| *** k_mouza has joined #openstack-kolla | 01:08 | |
| *** k_mouza has quit IRC | 01:13 | |
| *** JamesBenson has quit IRC | 02:22 | |
| *** JamesBenson has joined #openstack-kolla | 02:22 | |
| *** openstackgerrit has quit IRC | 03:57 | |
| *** zzzeek has quit IRC | 04:06 | |
| *** zzzeek has joined #openstack-kolla | 04:23 | |
| *** zzzeek has quit IRC | 04:25 | |
| *** zzzeek has joined #openstack-kolla | 04:27 | |
| *** also_stingrayza has joined #openstack-kolla | 04:27 | |
| *** stingrayza has quit IRC | 04:30 | |
| *** evrardjp has quit IRC | 04:33 | |
| *** evrardjp has joined #openstack-kolla | 04:33 | |
| *** dave-mccowan has quit IRC | 04:36 | |
| *** JamesBenson has quit IRC | 04:42 | |
| *** JamesBenson has joined #openstack-kolla | 04:43 | |
| *** JamesBenson has quit IRC | 04:48 | |
| *** nikparasyr has joined #openstack-kolla | 04:55 | |
| *** zzzeek has quit IRC | 05:09 | |
| *** jbadiapa has joined #openstack-kolla | 05:13 | |
| *** zzzeek has joined #openstack-kolla | 05:16 | |
| *** openstackgerrit has joined #openstack-kolla | 05:17 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible master: Use a better process name for httpd subprocess https://review.opendev.org/758519 | 05:17 |
|---|---|---|
| *** JamesBenson has joined #openstack-kolla | 05:24 | |
| *** JamesBenson has quit IRC | 05:29 | |
| yoctozepto | morning | 05:57 |
| mnasiadka | morning | 05:58 |
| *** k_mouza has joined #openstack-kolla | 05:59 | |
| mnasiadka | mgoddard: what is interesting, we never changed the rotation script or the crontab file in keystone, so wondering from where does it come from | 06:00 |
| mnasiadka | mgoddard: or maybe we just need to drop the check for a stale token, or just run the rotation if we discover a stale token | 06:01 |
| mnasiadka | yoctozepto: ^^ | 06:01 |
| mnasiadka | if it's stale - it means the cron job didn't rotate it at all... | 06:01 |
| yoctozepto | impossible | 06:01 |
| yoctozepto | I mean, I believe I was getting it rotated | 06:02 |
| yoctozepto | will check it later | 06:02 |
| *** k_mouza has quit IRC | 06:04 | |
| mnasiadka | https://review.opendev.org/#/c/758434/ | 06:09 |
| patchbot | patch 758434 - kolla-ansible - [DNM]: Keystone fernet rotation testing - 3 patch sets | 06:09 |
| mnasiadka | this change shows it's getting rotated on multinode jobs | 06:09 |
| mnasiadka | (ignore the upgrade jobs fails - if you change token expiry timings during upgrade, the container will be stuck restarting claiming the tokens are stale) | 06:10 |
| *** cah_link has joined #openstack-kolla | 06:13 | |
| *** iniazi has joined #openstack-kolla | 06:23 | |
| *** jbalciunas has quit IRC | 06:38 | |
| *** jbalciunas has joined #openstack-kolla | 06:39 | |
| *** vishalmanchanda has joined #openstack-kolla | 06:41 | |
| *** jbalciunas has quit IRC | 06:43 | |
| *** mchlumsky has quit IRC | 06:50 | |
| *** mchlumsky has joined #openstack-kolla | 06:51 | |
| *** bengates has joined #openstack-kolla | 07:02 | |
| *** maharg102 has joined #openstack-kolla | 07:11 | |
| *** maharg101 has quit IRC | 07:12 | |
| *** jbalciunas has joined #openstack-kolla | 07:19 | |
| yoctozepto | yeah, I didn't mean to pick on upgrades | 07:36 |
| mnasiadka | but it doesn't mean we shouldn't try to fix it :) | 07:37 |
| mnasiadka | but first let's focus on why people are coming with stale tokens | 07:37 |
| *** dougsz has joined #openstack-kolla | 07:38 | |
| *** devfaz has quit IRC | 07:50 | |
| *** devfaz has joined #openstack-kolla | 07:50 | |
| *** devfaz has quit IRC | 07:51 | |
| *** devfaz has joined #openstack-kolla | 07:52 | |
| *** wuchunyang has joined #openstack-kolla | 07:55 | |
| *** Tengu has quit IRC | 08:00 | |
| yoctozepto | all my Train deployments churning happily | 08:08 |
| yoctozepto | I might not have used the latest k+k-a though | 08:08 |
| yoctozepto | mnasiadka: do you think it could be Ussuri+? | 08:09 |
| yoctozepto | do we have reports of failures on Train? | 08:09 |
| mnasiadka | yoctozepto: kplant has on centos7+train I think | 08:11 |
| mnasiadka | but without logging of the crontab script, we're a bit blind | 08:11 |
| yoctozepto | duh, true | 08:11 |
| mnasiadka | not speaking about the fact crond only sends logs via syslog | 08:11 |
| yoctozepto | kplant: could you confirm + give versions of kolla (used to build images) and k-a (used to deploy) | 08:12 |
| yoctozepto | mnasiadka, kplant: and was it standard 3-controller deployment? | 08:12 |
| mnasiadka | yoctozepto: other stupid question, what is the difference between plugins and additions in Kolla? I can't seem to find any difference in the code :) | 08:19 |
| yoctozepto | mnasiadka: I think I asked this question once; there might be none :-) | 08:20 |
| yoctozepto | mgoddard might know | 08:20 |
| *** Tengu has joined #openstack-kolla | 08:22 | |
| mnasiadka | yoctozepto: I see somebody was lazy, instead of adding a check if plugins directory contains setup.cfg or something similar - he created additions - https://github.com/openstack/kolla/commit/9b1e519267f022ceb3f920976591a8f8233addc4 | 08:24 |
| yoctozepto | mgoddard, mnasiadka: wdyt about approaching https://bugs.launchpad.net/kolla-ansible/+bug/1837551 by using KOLLA_SKIP for rp_filter in Victoria (with a relevant upgrade note anyhow) and setting as wontfix in earlier? | 08:24 |
| openstack | Launchpad bug 1837551 in kolla-ansible victoria "rp_filter not set properly" [Medium,Triaged] | 08:24 |
| yoctozepto | mnasiadka: eh! :D | 08:24 |
| mnasiadka | yoctozepto: life :) | 08:25 |
| mnasiadka | yoctozepto: trivialfix - https://review.opendev.org/#/c/757076/ | 08:26 |
| patchbot | patch 757076 - kolla-ansible - Fix fernet cron path on Ubuntu/Debian - 3 patch sets | 08:26 |
| openstackgerrit | Merged openstack/kolla-ansible stable/ussuri: baremetal: Install iptables for Docker if enabled https://review.opendev.org/758070 | 08:28 |
| yoctozepto | mnasiadka: trivial but should not you mark some bug as closed? and write a reno? It's pretty important | 08:30 |
| yoctozepto | mnasiadka: so, basically, ubuntu users have been missing rotations for at least 2 years? nice | 08:30 |
| yoctozepto | osmanlicilegi: ^ check this out | 08:30 |
| mnasiadka | yoctozepto: basically it seems yes | 08:35 |
| hrw | elo | 08:35 |
| mnasiadka | and our fernet tokens (token lifetime + expire window) is 3 days | 08:35 |
| mnasiadka | and crontab on standard values rotates the keys every ~4 days (midnight on Wed and Sun) | 08:36 |
| *** Tengu has quit IRC | 08:36 | |
| yoctozepto | mnasiadka: well, that's problematic, but still better if rotations DO take place xD | 08:39 |
| mnasiadka | haha, right | 08:39 |
| *** k_mouza has joined #openstack-kolla | 08:41 | |
| *** Tengu has joined #openstack-kolla | 08:43 | |
| * osmanlicilegi is back | 08:49 | |
| *** bengates has quit IRC | 08:50 | |
| *** bengates has joined #openstack-kolla | 08:51 | |
| osmanlicilegi | yoctozepto: I will. I'm still trying to catch up what I have missed for the last few weeks :] | 08:52 |
| *** k_mouza has quit IRC | 08:55 | |
| SvenKieske | yoctozepto: regarding the failing fernet token rotation: yes, we experienced this on train.. brought our containers in a restart loop | 08:58 |
| SvenKieske | this is a 3 controlnodes deployment on ubuntu | 08:59 |
| SvenKieske | glad it is fixed now | 08:59 |
| *** cah_link1 has joined #openstack-kolla | 09:05 | |
| *** brinzhang_ has quit IRC | 09:05 | |
| *** jbadiapa has quit IRC | 09:05 | |
| *** cah_link has quit IRC | 09:05 | |
| *** cah_link1 is now known as cah_link | 09:05 | |
| *** brinzhang_ has joined #openstack-kolla | 09:05 | |
| *** jbadiapa has joined #openstack-kolla | 09:06 | |
| yoctozepto | SvenKieske: but Train AND Ubuntu, no? | 09:12 |
| openstackgerrit | Bartosz Bezak proposed openstack/kolla master: rally: add rally-openstack in source - follow up https://review.opendev.org/758540 | 09:17 |
| *** kevko has joined #openstack-kolla | 09:19 | |
| *** JamesBenson has joined #openstack-kolla | 09:26 | |
| *** k_mouza has joined #openstack-kolla | 09:26 | |
| *** k_mouza has quit IRC | 09:29 | |
| *** k_mouza has joined #openstack-kolla | 09:29 | |
| *** JamesBenson has quit IRC | 09:30 | |
| *** brinzhang_ has quit IRC | 09:48 | |
| *** jonaspaulo has joined #openstack-kolla | 09:48 | |
| SvenKieske | yoctozepto: yes | 10:07 |
| SvenKieske | we had no key rotations since april (when we last deployed the containers), pretty unfortunate.. lucky it is still a development environment | 10:08 |
| SvenKieske | can someone maybe shed some light on my question regarding rabbitmq interface configuration from yesterday? http://lists.openstack.org/pipermail/openstack-discuss/2020-October/018014.html I'm also willing to provide patches, if needed, we already signed the corp. cla | 10:09 |
| *** wuchunyang has quit IRC | 10:13 | |
| mnasiadka | SvenKieske: that was long time ago, either those options did not work with new rabbitmq, or Paul didn't think those options were relevant - if you'd like to raise a change to add that support back - we'd be happy to help | 10:40 |
| *** Tengu has quit IRC | 10:41 | |
| openstackgerrit | Pierre Riteau proposed openstack/kayobe stable/stein: CI: Fix kayobe-tox-molecule job https://review.opendev.org/758550 | 10:44 |
| openstackgerrit | Pierre Riteau proposed openstack/kayobe stable/stein: Performance: skip LVM configuration if no groups configured https://review.opendev.org/758075 | 10:45 |
| *** Tengu has joined #openstack-kolla | 10:53 | |
| *** JamesBenson has joined #openstack-kolla | 11:00 | |
| *** wuchunyang has joined #openstack-kolla | 11:07 | |
| *** JamesBenson has quit IRC | 11:10 | |
| *** JamesBenson has joined #openstack-kolla | 11:10 | |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla master: WIP: use upper constraints in openstack-base https://review.opendev.org/758553 | 11:11 |
| *** shyamb has joined #openstack-kolla | 11:15 | |
| yoctozepto | SvenKieske: +2 for adding it back; I firewalled it away for myself but makes sense to bind it more tightly | 11:18 |
| *** shyamb has quit IRC | 11:46 | |
| SvenKieske | yoctozepto: mnasiadka: okay will prepare this, my colleague has already some experience with contributing patches so I guess I can figure out the correct workflow for myself, will ask if I got any questions left. | 12:02 |
| kplant | yoctozepto, mnasiadka: centos7+train (stable/train from git) | 12:05 |
| kplant | three controller, two network, three monitor, N compute | 12:05 |
| yoctozepto | and you are getting permanently stale tokens? | 12:05 |
| kplant | not sure tbh, the deployment is only a few days old | 12:06 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla master: version-check: Use independent release data as default https://review.opendev.org/758556 | 12:10 |
| mnasiadka | kplant: just to be sure, your crontab runs rotation script on sundays and wednesdays on midnight? and no crontab entry on the third host? | 12:11 |
| *** jbalciunas has quit IRC | 12:11 | |
| kplant | i believe that's correct | 12:12 |
| kplant | let me recheck | 12:12 |
| *** jbalciunas has joined #openstack-kolla | 12:12 | |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla master: version-check: Use independent release data as default https://review.opendev.org/758556 | 12:12 |
| kplant | yeah: 0 - sunday, 3 - wednesday, nothing on 3rd control -- you got it | 12:14 |
| mnasiadka | kplant: given that keystone fernet token should be valid 3 days, that gives a bit of a time window for them to be invalid... | 12:15 |
| kplant | sure does | 12:16 |
| kplant | not sure why they were invalid yesterday though | 12:17 |
| kplant | thursday should have been perfect | 12:17 |
| *** jbalciunas has quit IRC | 12:18 | |
| mnasiadka | So, fernet token expiry is 1 day + we allow for 2 days extra time (in keystone config) | 12:18 |
| mnasiadka | I’ll look into the script that generates crontabs and see what we can do | 12:18 |
| mnasiadka | We should rather rotate them every day and leave those 2 days for ,,contingency’’ | 12:20 |
| *** wuchunyang has quit IRC | 12:31 | |
| kplant | did anything change within stable/train ? | 12:31 |
| kplant | i've never experienced this is any of my other deployments | 12:31 |
| ozzzo | my change failed zuul: https://review.opendev.org/#/c/758486/ | 12:33 |
| patchbot | patch 758486 - kolla - Closes bug 1897948 - 1 patch set | 12:33 |
| openstack | bug 1897948 in kolla-ansible "Incorrect setting for --incremental-history-name in kolla/docker/mariadb/backup.sh" [Wishlist,Triaged] https://launchpad.net/bugs/1897948 | 12:33 |
| ozzzo | The error messaages are pretty clear and I see what needs to be done to the code, but I'm a bit foggy on the procedure for submitting an updated review | 12:33 |
| ozzzo | do I just do git commit and git review again, or is there a different procedure? | 12:33 |
| ozzzo | I think I found it; it looks like I need to do: git commit -a --amend | 12:38 |
| ozzzo | trying that now | 12:38 |
| openstackgerrit | Merged openstack/kolla stable/train: Checks for heat_user_domain explicitly https://review.opendev.org/758059 | 12:47 |
| openstackgerrit | Merged openstack/kolla stable/stein: Checks for heat_user_domain explicitly https://review.opendev.org/758060 | 12:47 |
| openstackgerrit | Merged openstack/kolla stable/stein: Bump versions for Stein https://review.opendev.org/757566 | 12:47 |
| openstackgerrit | Merged openstack/kayobe stable/stein: CI: Fix kayobe-tox-molecule job https://review.opendev.org/758550 | 12:49 |
| openstackgerrit | Albert Braden proposed openstack/kolla master: Closes bug 1897948 https://review.opendev.org/758486 | 12:52 |
| openstack | bug 1897948 in kolla-ansible "Incorrect setting for --incremental-history-name in kolla/docker/mariadb/backup.sh" [Wishlist,Triaged] https://launchpad.net/bugs/1897948 | 12:52 |
| *** jbalciunas has joined #openstack-kolla | 13:07 | |
| openstackgerrit | Merged openstack/kolla stable/train: Bump versions for Train https://review.opendev.org/758047 | 13:12 |
| openstackgerrit | Merged openstack/kolla stable/ussuri: Add sysfsutils to nova-compute ubuntu source https://review.opendev.org/758051 | 13:12 |
| openstackgerrit | Merged openstack/kolla stable/train: Add sysfsutils to nova-compute ubuntu source https://review.opendev.org/758052 | 13:12 |
| openstackgerrit | Merged openstack/kolla stable/stein: Add sysfsutils to nova-compute ubuntu source https://review.opendev.org/758053 | 13:12 |
| mnasiadka | kplant: this is part of fernet improvements we backported all the way to train | 13:15 |
| mnasiadka | kplant: probably the thing with checking if keystone fernet token is stale on keystone startup is causing problems, but then we shouldn't have such old tokens :) | 13:27 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: [DNM]: Keystone fernet rotation testing https://review.opendev.org/758434 | 13:35 |
| *** priteau has joined #openstack-kolla | 13:39 | |
| *** jbalciunas has quit IRC | 14:17 | |
| *** dave-mccowan has joined #openstack-kolla | 14:20 | |
| *** vishalmanchanda has quit IRC | 14:20 | |
| *** jbalciunas has joined #openstack-kolla | 14:23 | |
| openstackgerrit | Sven Kieske proposed openstack/kolla-ansible master: re-add rabbitmq config for interfaces https://review.opendev.org/758576 | 14:29 |
| *** hrw has quit IRC | 14:32 | |
| SvenKieske | my first proposal, so I hope everything fits, you can ping me today about any needed changes, till about 19:00 CEST, or on monday again. | 14:50 |
| mnasiadka | yoctozepto: ^^ we want to treat it as a bug, or new functionality? :) | 14:51 |
| SvenKieske | well, yeah, I was asking that myself. If you need a blueprint for that, or something like that, just give me heads up, will have to read up on how to create those, though. | 14:56 |
| *** cah_link has quit IRC | 15:00 | |
| mnasiadka | SvenKieske: naah, no blueprints, just I'm thinking if we want to backport that to stable branches, or not - but I guess it would be nice to get this functionality back in stable branches. | 15:00 |
| *** jbalciunas has quit IRC | 15:02 | |
| *** nikparasyr has left #openstack-kolla | 15:03 | |
| openstackgerrit | Merged openstack/kayobe stable/stein: Performance: skip LVM configuration if no groups configured https://review.opendev.org/758075 | 15:03 |
| mnasiadka | SvenKieske: please raise a bug that this is missing without any deprecation notice, and add "Closes-Bug: #bug_id" in the commit message - you'll also need a release note (please read about reno tool usage somewhere in openstack docs). | 15:04 |
| SvenKieske | okay, will do | 15:07 |
| openstackgerrit | Sven Kieske proposed openstack/kolla-ansible master: re-add rabbitmq config for interfaces https://review.opendev.org/758576 | 15:18 |
| SvenKieske | Change is updated :) | 15:19 |
| *** jbalciunas has joined #openstack-kolla | 15:20 | |
| *** e0ne has quit IRC | 15:33 | |
| *** bengates has quit IRC | 15:38 | |
| SvenKieske | I'll check for the release notes bits now | 15:48 |
| *** cah_link has joined #openstack-kolla | 15:52 | |
| openstackgerrit | Doug Szumski proposed openstack/kolla-ansible stable/ussuri: Use become when copying custom Fluentd config https://review.opendev.org/758588 | 15:56 |
| SvenKieske | mnasiadka: okay I did as you ask, beside the doc change, that one will take some time to figure out.. your docs on how to handle docs are huge! will have to read up on how this is supposed to work. | 16:00 |
| SvenKieske | mnasiadka: yes, I agree this should be backported, in fact, we still use train release (need to upgrade soonish[TM]) and would rather use upstreamed patches than our own. | 16:03 |
| *** cah_link has quit IRC | 16:04 | |
| *** cah_link has joined #openstack-kolla | 16:05 | |
| *** cah_link has quit IRC | 16:12 | |
| *** k_mouza has quit IRC | 16:24 | |
| *** k_mouza has joined #openstack-kolla | 16:31 | |
| *** dougsz has quit IRC | 16:31 | |
| SvenKieske | mnasiadka: should I add the documentation to the same Changeset as the code? or maybe anyone else can answer this also? | 16:33 |
| *** k_mouza has quit IRC | 16:35 | |
| *** k_mouza has joined #openstack-kolla | 16:36 | |
| *** k_mouza has quit IRC | 16:36 | |
| yoctozepto | mnasiadka: fine to be a bugfix I guess; it broke at some point heh | 16:44 |
| *** k_mouza has joined #openstack-kolla | 16:44 | |
| yoctozepto | SvenKieske: yes, best would be the same | 16:44 |
| *** jbalciunas has quit IRC | 16:44 | |
| *** k_mouza has quit IRC | 16:49 | |
| openstackgerrit | Sven Kieske proposed openstack/kolla-ansible master: re-add rabbitmq config for interfaces https://review.opendev.org/758576 | 16:49 |
| SvenKieske | done | 16:49 |
| *** k_mouza has joined #openstack-kolla | 16:52 | |
| *** k_mouza has quit IRC | 16:56 | |
| *** hrw has joined #openstack-kolla | 17:00 | |
| *** maharg102 has quit IRC | 17:01 | |
| *** k_mouza has joined #openstack-kolla | 17:26 | |
| *** k_mouza has quit IRC | 17:31 | |
| *** k_mouza has joined #openstack-kolla | 17:37 | |
| *** k_mouza has quit IRC | 17:42 | |
| *** k_mouza has joined #openstack-kolla | 17:46 | |
| *** k_mouza has quit IRC | 17:51 | |
| *** k_mouza has joined #openstack-kolla | 17:55 | |
| *** k_mouza has quit IRC | 17:59 | |
| *** k_mouza has joined #openstack-kolla | 18:07 | |
| *** k_mouza has quit IRC | 18:11 | |
| *** k_mouza has joined #openstack-kolla | 18:15 | |
| *** dougsz has joined #openstack-kolla | 18:16 | |
| *** jonaspaulo has quit IRC | 18:19 | |
| *** k_mouza has quit IRC | 18:20 | |
| *** k_mouza has joined #openstack-kolla | 18:20 | |
| *** cah_link has joined #openstack-kolla | 18:22 | |
| *** k_mouza has quit IRC | 18:25 | |
| *** k_mouza has joined #openstack-kolla | 18:31 | |
| *** k_mouza has quit IRC | 18:35 | |
| *** dougsz has quit IRC | 18:57 | |
| *** maharg101 has joined #openstack-kolla | 18:58 | |
| *** jbadiapa has quit IRC | 19:03 | |
| *** maharg101 has quit IRC | 19:04 | |
| openstackgerrit | Merged openstack/kolla stable/ussuri: Checks for heat_user_domain explicitly https://review.opendev.org/758058 | 19:34 |
| *** priteau has quit IRC | 19:40 | |
| *** zzzeek has quit IRC | 19:43 | |
| *** zzzeek has joined #openstack-kolla | 19:45 | |
| openstackgerrit | Merged openstack/kolla-ansible stable/ussuri: Performance: use a single config file for fluentd https://review.opendev.org/757813 | 20:06 |
| openstackgerrit | Merged openstack/kolla-ansible stable/train: Performance: use a single config file for fluentd https://review.opendev.org/757816 | 20:06 |
| openstackgerrit | Merged openstack/kolla-ansible stable/stein: Performance: use a single config file for fluentd https://review.opendev.org/757817 | 20:06 |
| openstackgerrit | Merged openstack/kolla-ansible master: Update release note for httpd keep alive https://review.opendev.org/746512 | 20:06 |
| *** kevko has quit IRC | 20:25 | |
| *** kevko has joined #openstack-kolla | 20:29 | |
| *** dswebb has quit IRC | 21:24 | |
| *** rgogunskiy has quit IRC | 21:36 | |
| *** dave-mccowan has quit IRC | 22:00 | |
| *** dave-mccowan has joined #openstack-kolla | 22:04 | |
| *** stingrayza has joined #openstack-kolla | 22:34 | |
| *** also_stingrayza has quit IRC | 22:37 | |
| *** dave-mccowan has quit IRC | 23:12 | |
| *** dave-mccowan has joined #openstack-kolla | 23:31 | |
| *** dave-mccowan has quit IRC | 23:55 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!