| opendevreview | Michal Nasiadka proposed openstack/kolla master: Fix aarch64 builds https://review.opendev.org/c/openstack/kolla/+/909915 | 06:00 |
|---|---|---|
| mnasiadka | morning | 07:05 |
| mnasiadka | jovial: that was probably rackspace mirror issue? | 07:05 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: Fix glance not running after upstream cache changes https://review.opendev.org/c/openstack/kolla-ansible/+/912030 | 07:07 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Fix creation of ovs bridges https://review.opendev.org/c/openstack/kolla-ansible/+/911591 | 07:07 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: Fix glance not running after upstream cache changes https://review.opendev.org/c/openstack/kolla-ansible/+/912030 | 08:04 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: Remove setuptools pin https://review.opendev.org/c/openstack/kolla/+/901828 | 08:07 |
| kevko | morning | 08:15 |
| mnasiadka | kevko: updated your patch, Kolla's documented philosophy says we don't want to own defaults unless it's required - and it seems to pass without setting cache driver | 08:19 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Fix creation of ovs bridges https://review.opendev.org/c/openstack/kolla-ansible/+/911591 | 08:20 |
| kevko | mnasiadka: okay, no problem ...it was hell to debug :D ... i had a feeling that it's needed to be set for now ... I wrote it to commit message | 08:23 |
| mnasiadka | yeah, updated the commit message as well | 08:23 |
| kevko | mnasiadka: but as I see you've amended also commit message | 08:23 |
| kevko | mnasiadka: thanks .. I am glad that I can ask you guys for reviews because CI will work again :D :D | 08:25 |
| kevko | mnasiadka: and push new ones | 08:25 |
| mnasiadka | yeah, tooz patch landed, but the release and u-c bump is not there yet | 08:26 |
| mnasiadka | so it would make sense to merge the pin today as well | 08:26 |
| kevko | mnasiadka: btw, I found that glance-api configuration file (for paste_deploy group) don't work as it's described in a describtion above ... | 08:27 |
| kevko | mnasiadka: there is described that if you change config_file to something different (paste ini config) ... as example "foo.ini" ....on the end it will be /etc/glance/foo.ini ... | 08:28 |
| mnasiadka | lol | 08:29 |
| kevko | mnasiadka: but if you change it ...it is just not working :D | 08:29 |
| kevko | mnasiadka: https://github.com/openstack/glance/blob/519ce0b94dcc6dad229fbe08d8d21ccdefb6c99e/glance/common/config.py#L682-L705 | 08:30 |
| kevko | mnasiadka: aaaand ...another review ..do you think it's wrong here ? https://review.opendev.org/c/openstack/glance/+/911955 | 08:31 |
| kevko | mnasiadka: I think authot wrote that function wrong ...i fixed it and unit is not working now ... but I really hate unit fixing :D | 08:32 |
| mnasiadka | lol | 08:32 |
| mnasiadka | well, then the unit test should be fixed first, to demonstrate that it's not working properly ;-) | 08:32 |
| kevko | mnasiadka: i know ... but i am not good in it :D | 08:33 |
| kevko | mnasiadka: but check that function ....original is returning always True from function can_migrate_to_central_db ...but it makes sense only if cache is turned on ... | 08:34 |
| kevko | mnasiadka: By the way, Kolla worked with this fix in Glance, it was actually the inspiration for me to try to fix Kolla without changing the code in Glance, but this was the initial problem... (cache_dir was None in glance-api and /var/lib/.... in glance-cachce ....so it started failing ..... but initial problem was that it actually jumped into | 08:37 |
| kevko | code for cache ...) | 08:37 |
| kevko | so kolla proposed patch is kind a workaround ...but following also docs ...so ... | 08:38 |
| opendevreview | Roman Krček proposed openstack/kolla-ansible master: Split ipv4 and ipv6 systemctl config https://review.opendev.org/c/openstack/kolla-ansible/+/912091 | 08:52 |
| kevko | mnasiadka: e\\ndocker.errors.APIError: 500 Server Error for http+docker://localhost/v1.44/images/create?tag=master-ubuntu-jammy&fromImage=mirror-int.iad.rax.opendev.org%3A4447%2Fopenstack.kolla%2Ffluentd: Internal Server Error (\"Get \"https://mirror-int.iad.rax.opendev.org:4447/v2/\": context deadline exceeded (Client.Timeout exceeded while | 08:58 |
| kevko | awaiting headers)\")\\n'" | 08:58 |
| kevko | aaaaaaaa | 08:58 |
| mnasiadka | kevko: yep, reported on opendev | 08:58 |
| kevko | mnasiadka: how fast they work ? :D | 08:58 |
| mnasiadka | have no clue, in this timezone it's probably only frickler ;-) | 08:59 |
| kevko | Or is it just temporary? | 08:59 |
| mnasiadka | We've seen that in other jobs, and some Kayobe jobs yesterday | 08:59 |
| kevko | So I will have to play in my sandbox.. :) | 09:01 |
| kevko | mnasiadka: what about that tooz ? do we have review for that already somewhere ? | 09:02 |
| mnasiadka | https://review.opendev.org/c/openstack/kolla/+/911703 | 09:02 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: openstack-base: Pin tooz to 5.0.0 https://review.opendev.org/c/openstack/kolla/+/911703 | 09:02 |
| mnasiadka | added depends-on on glance patch | 09:03 |
| kevko | mnasiadka: thanks | 09:11 |
| kevko | mnasiadka: wondering ..that from a code it should be enough to just define chaching options in glance-api.conf | 09:12 |
| mnasiadka | we could make it better once CI is working I guess :) | 09:14 |
| mnasiadka | so let's merge this - maybe 7 rechecks will be enough ;-) | 09:14 |
| kevko | i am ok with it | 09:15 |
| kevko | mnasiadka: I mean ... not only in glance-api ...just drop jinja conditional inside the glance-api.conf | 09:16 |
| opendevreview | Roman Krček proposed openstack/kolla-ansible master: Split ipv4 and ipv6 systemctl config https://review.opendev.org/c/openstack/kolla-ansible/+/905831 | 09:17 |
| kevko | mnasiadka: yep, working | 09:23 |
| kevko | mnasiadka: let me edit last time | 09:24 |
| mnasiadka | kevko: sure, the current run is going to fail anyway :( | 09:24 |
| kevko | mnasiadka: I know :) | 09:25 |
| kevko | mnasiadka: it's possible that the fix in glance will land earlier :D https://review.opendev.org/c/openstack/glance/+/911955 | 09:37 |
| opendevreview | Jake Hutchinson proposed openstack/kayobe master: Register baremetal compute nodes in Ironic https://review.opendev.org/c/openstack/kayobe/+/909671 | 10:29 |
| SvenKieske | mhm, I'm not sure yet, but could someone verify that they have the keystone role "service" created in their envs? kevko? mnasiadka? maybe I found a bug - rather community members, but I had now two instances where the service role was missing | 10:33 |
| SvenKieske | I'm currently setting up a fresh env to test myself, but it still takes some time | 10:34 |
| SvenKieske | I have a hunch https://review.opendev.org/c/openstack/kolla-ansible/+/909002 is buggy, I downloaded the logs and I found: "ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service." | 10:35 |
| SvenKieske | but I still have to go through all the logs to verify | 10:35 |
| opendevreview | scoopex proposed openstack/kolla-ansible master: rabbitmq: configuration improvements https://review.opendev.org/c/openstack/kolla-ansible/+/900528 | 10:41 |
| opendevreview | Jake Hutchinson proposed openstack/kayobe master: Register baremetal compute nodes in Ironic https://review.opendev.org/c/openstack/kayobe/+/909671 | 10:47 |
| opendevreview | Jake Hutchinson proposed openstack/kayobe master: Register baremetal compute nodes in Ironic https://review.opendev.org/c/openstack/kayobe/+/909671 | 10:47 |
| opendevreview | Jake Hutchinson proposed openstack/kayobe master: Register baremetal compute nodes in Ironic https://review.opendev.org/c/openstack/kayobe/+/909671 | 10:51 |
| kevko | SvenKieske: I will check in a while ... Smoking 🚬 now | 11:48 |
| kevko | SvenKieske: I have service role in my master test stack | 11:52 |
| kevko | (kolla-ansible)[root]# openstack role assignment list --names | egrep '\ cinder|\ nova' | 12:02 |
| kevko | | admin | cinder@Default | | service@Default | | | False | | 12:02 |
| kevko | | service | cinder@Default | | service@Default | | | False | | 12:02 |
| kevko | | admin | nova@Default | | service@Default | | | False | | 12:02 |
| kevko | | service | nova@Default | | service@Default | | | False | | 12:02 |
| kevko | SvenKieske: but I have also the same error log in keystone log | 12:02 |
| kevko | https://paste.openstack.org/show/bV2ZJ4dMupTwRCO8iZkA/ | 12:12 |
| SvenKieske | thats the role assignment | 12:15 |
| kevko | openstack role show service << this produce error | 12:15 |
| SvenKieske | query for the role itself instead | 12:15 |
| SvenKieske | yeah I suspected as much | 12:15 |
| kevko | SvenKieske: but api normally replied | 12:16 |
| kevko | https://paste.openstack.org/show/bvUMxl2aTtApAgVhZx5e/ | 12:16 |
| SvenKieske | kevko, what I currently suspect is this comment: https://review.opendev.org/c/openstack/keystone/+/863420/comment/655c953c_92da126b/ | 12:16 |
| SvenKieske | "If you specify any project name for bootstrap command then there is no need to create service role for that project, it requires only admin, member and reader roles. " | 12:16 |
| SvenKieske | and I'm not sure if we run bootstrap command with the right incantation to actually create service role | 12:17 |
| SvenKieske | but I'm also not sure I understand that comment correctly. | 12:17 |
| SvenKieske | what I currently think - might be wrong - is you need to call keystone-bootstrap without args to bootstrap basic stuff? but we always call it with args? not sure how NULL args are handled. but this might be very well wrong or incomplete, still digging through code and logs | 12:20 |
| kevko | Before the introduction of the service role, a service had to be granted the admin role in order to have elevated privileges, which gave a service powers way beyond what was necessary. With the service role in place, we can now allow all service-to-service APIs to default to the service role only. For example, a policy that requires service can be | 12:23 |
| kevko | expressed as: | 12:23 |
| SvenKieske | it might also just be that we don't run keystone-bootstrap during upgrades? that's still on the list (because all envs where this was reported where upgrades) | 12:23 |
| SvenKieske | nvm, at least it seems we do run the role on upgrade, if it's not bugged somehow: https://review.opendev.org/c/openstack/kolla-ansible/+/909085/4/ansible/roles/keystone/tasks/upgrade.yml | 12:25 |
| kevko | SvenKieske: i am 80 percent sure this is bug in keystone | 12:25 |
| SvenKieske | xD | 12:25 |
| kevko | SvenKieske: ah, maybe not :) | 12:26 |
| SvenKieske | mhm, maybe we don't run keystone-bootstrap command early enough and thus the errors are produced? but they still show up later, don't they? even after the service role is created? weird | 12:26 |
| kevko | SvenKieske: this is returning None https://github.com/openstack/keystone/blob/2ac039b717669bf9744f72161e82bdac46dbfacf/keystone/assignment/role_backends/sql.py#L63 | 12:27 |
| kevko | SvenKieske: what about this ? | 12:32 |
| kevko | Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration.\x1b[00m | 12:32 |
| SvenKieske | mhm, interesting | 12:39 |
| SvenKieske | where would we have deprecated policy rules? did you find this in keystone log? | 12:40 |
| SvenKieske | ah I have these too | 12:40 |
| kevko | The problem is that when you ask for the role "service"... it prepopulates "service" in the code, and it falls into functions that expect IDs. | 12:46 |
| opendevreview | Jake Hutchinson proposed openstack/kayobe master: Register baremetal compute nodes in Ironic https://review.opendev.org/c/openstack/kayobe/+/909671 | 12:52 |
| *** carloss_ is now known as carloss | 13:15 | |
| kevko | SvenKieske: do you know btw why designate-manage pool update from time to time returns -1 ? | 13:37 |
| SvenKieske | no idea, really :D because it's designate? | 13:38 |
| kevko | SvenKieske: designate working very nice .. | 13:47 |
| kevko | SvenKieske: i think maybe because some type of timeout maybe | 13:47 |
| SvenKieske | yeah sorry. agreed it works, but the design is imho showing it's age, but there's nothing better to replace it just yet I guess. | 13:48 |
| kevko | SvenKieske: what is bad on design ? | 13:49 |
| SvenKieske | e.g. it can't take advantage of more efficient DNS update algorithms. it only supports AXFR, it's thus not suitable for large DNS deployments imho (maybe that was also never intended). | 13:51 |
| SvenKieske | it would be nice e.g. if it could hook up into powerdns api, but that would require a redesign, I actually looked into doing that until I realized that everything is build around the AXFR assumption and then gave up. | 13:52 |
| SvenKieske | there's nothing work with relying on AXFR if you only have, I don't know 1000 domains or some such? so it's viable for small-/mid-sized setup I'd say. | 13:54 |
| SvenKieske | s/work/wrong/ | 13:54 |
| mnasiadka | kevko: so what’s up with glance? Do we need your patch? | 14:05 |
| kevko | mnasiadka: for now yes ..because glance itself is broken ... then no ... | 14:08 |
| kevko | mnasiadka: we can merge it ...and then we can revert after glance will be merged | 14:09 |
| kevko | mnasiadka: there is a problem that if there is no option for cache_dir ...it's none ..and in a code it's handled in bad way | 14:09 |
| kevko | mnasiadka: check my glance patch | 14:09 |
| dougszu | SvenKieske: If it's any help, I'm running a March 1st checkout and I see the service role in `openstack role list` | 14:24 |
| mnasiadka | kevko: i see extremely stable glance ci ;) rechecked the k-a patch | 14:24 |
| opendevreview | Mark Goddard proposed openstack/kolla-ansible master: Avoid unnecessary secondary fact gathering failure https://review.opendev.org/c/openstack/kolla-ansible/+/912263 | 14:36 |
| frickler | mnasiadka: jovial: what about release highlights? they would be due today and I haven't seen any (but I was also out today so maybe missed something) | 14:51 |
| kevko | mnasiadka: https://review.opendev.org/c/openstack/glance/+/911955 merging :) | 15:09 |
| kevko | mnasiadka: after glance will be merged ... k-a patch will not be needed | 15:10 |
| kevko | mnasiadka: but i don't know how to trigger fresh new build glance images ...depends-on to some kolla patch with dummy RUN echo 'rebuild' or what ? | 15:11 |
| SvenKieske | dougszu: thanks for the data point; might be this is only triggered in some upgrade scenarios, but I'm still not sure how, because we also run the service register playbook on upgrade | 15:26 |
| kevko | SvenKieske: as I said ...it's bug in keystone | 15:29 |
| SvenKieske | kevko: did you already find something? | 15:32 |
| SvenKieske | I'm currently writing up how it's supposed to work and document the status quo bascially | 15:33 |
| SvenKieske | so if I can include any pointer in the bug report that would be helpful to continue the research :) | 15:33 |
| SvenKieske | kevko: do you mean this? https://github.com/openstack/keystone/blob/2ac039b717669bf9744f72161e82bdac46dbfacf/keystone/assignment/role_backends/sql.py#L63 | 15:33 |
| SvenKieske | ah right, you mentioned it above, that if you trigger this function with "service" there is no "role_id" and it returns the exception | 15:35 |
| kevko | SvenKieske: yeah - I've already told you .. | 15:37 |
| mnasiadka | frickler: let me chase bbezak | 15:38 |
| kevko | SvenKieske: simply said ... in a code all functions accept role_id ..but in whole chain of calling functions there is a call with function(role_name) | 15:39 |
| mnasiadka | kevko: they will get built tomorrow morning | 15:39 |
| kevko | SvenKieske: it's not only service role ...just check your log for keystone and in second terminal execute openstack role show foo ...and you will see the same error | 15:39 |
| mnasiadka | It doesn't make any sense to trigger build and publish at this point of day ;) | 15:39 |
| kevko | mnasiadka: no problem ... but i can create my own to just verify and then abandon no ? | 15:40 |
| mnasiadka | we can recheck the tooz pin as well | 15:42 |
| SvenKieske | kevko: yeah it's also in keystone catalog: https://github.com/openstack/keystone/blob/2ac039b717669bf9744f72161e82bdac46dbfacf/keystone/catalog/core.py#L150 | 15:56 |
| mnasiadka | kevko: so basically everything will be fixed on Monday it seems ;-) | 16:07 |
| SvenKieske | kevko: my current downstream writeup, will do a proper keystone bugreport once I understood this mess: https://github.com/osism/issues/issues/907 | 16:15 |
| mnasiadka | basically I think somebody should fix keystone logging, all those tracebacks are not helping - and GET to non-existent domain/project/role should be some silent error, at most a debug log message I guess | 16:16 |
| mnasiadka | frickler: https://review.opendev.org/c/openstack/releases/+/912282 | 16:46 |
| mnasiadka | (for kolla and k-a) | 16:46 |
| Lockesmith | Is there a guide anywhere on how to correctly use tls certs generated externally via certbot? | 22:54 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!