| *** sshnaidm is now known as sshnaidm|off | 02:57 | |
| *** raukadah is now known as chandankumar | 06:12 | |
| jrosser | something very weird with these db pooling patches, the db sync commands fail | 08:18 |
|---|---|---|
| noonedeadpunk | yeah, saw that, haven't debugged yet | 08:20 |
| noonedeadpunk | it feels that it tries to do more operations then allowed by pool or smth | 08:20 |
| * noonedeadpunk in meetings queue | 08:21 | |
| noonedeadpunk | queue is wrong word but I hope you got context | 08:21 |
| noonedeadpunk | row probably would be more correct one | 08:21 |
| jrosser | https://zuul.opendev.org/t/openstack/build/84edd8ff25434c28ba30659541ff2542/log/logs/host/syslog.txt#16314 | 08:22 |
| jrosser | there's a stack trace at least | 08:23 |
| noonedeadpunk | um.... | 08:24 |
| noonedeadpunk | `ERROR glance pool.logger.debug("Error on connect(): %s", e)` | 08:25 |
| noonedeadpunk | so AttributeError is probably result of pooling issue | 08:26 |
| noonedeadpunk | damiandabrowski[m]: you would be interested ^ | 08:26 |
| damiandabrowski[m] | :( I'll have a look | 09:34 |
| noonedeadpunk | I'm about to test out thing in aio | 09:38 |
| noonedeadpunk | stack trace in readable way https://paste.opendev.org/show/811425/ | 09:42 |
| damiandabrowski[m] | thanks! | 09:43 |
| noonedeadpunk | oh, i know, lol | 09:45 |
| noonedeadpunk | that is stupid :) | 09:45 |
| noonedeadpunk | https://zuul.opendev.org/t/openstack/build/84edd8ff25434c28ba30659541ff2542/log/logs/etc/host/glance/glance-manage.conf.txt | 09:45 |
| noonedeadpunk | `max_overflow = 50` is on connection string | 09:45 |
| noonedeadpunk | damiandabrowski[m]: ^ | 09:45 |
| noonedeadpunk | so it's just tempalting issue | 09:46 |
| damiandabrowski[m] | omg :D so probably i need to add one extra blank line after connection string, right? https://paste.openstack.org/raw/811427/ | 09:48 |
| damiandabrowski[m] | but can You explain me why it's needed? :D | 09:48 |
| noonedeadpunk | um, not sure, it might be config_template bug actually.... | 09:50 |
| noonedeadpunk | or probably it's somesthing related to whitespace control https://jinja.palletsprojects.com/en/3.0.x/templates/#whitespace-control | 09:51 |
| noonedeadpunk | yes, it's whitespace control | 09:53 |
| noonedeadpunk | so if add `+` to last endif it works as expected | 09:53 |
| noonedeadpunk | ie https://paste.opendev.org/show/811428/ | 09:53 |
| noonedeadpunk | (see {% endif +%} ) | 09:54 |
| noonedeadpunk | (I never fully understood whole whitespace control reasoning) | 09:57 |
| damiandabrowski[m] | great, thanks! | 09:59 |
| jrosser | i've noticed this before, that for a bunch of our templates the blank lines in the .j2 files are super important | 10:02 |
| damiandabrowski[m] | yeah, i didn't know about it, but now it kind of makes sense | 10:03 |
| jrosser | otherwise that kind of joining one line to the next happens, but its really unclear what it is makes that happen | 10:03 |
| damiandabrowski[m] | and that's why we can use conditionals like `{% if nova_ceilometer_enabled %}` without creating an extra blank line | 10:03 |
| jrosser | as in lots of places we have consecutive lines just fine | 10:03 |
| damiandabrowski[m] | i think it happens when {% %} takes a whole line, so statements like proxyclient_address from here, work fine: https://paste.openstack.org/show/811430/ | 10:05 |
| jrosser | manybe theres no implicit newline after a final %} | 10:06 |
| damiandabrowski[m] | i think i just wrote something stupid | 10:11 |
| damiandabrowski[m] | but yeah, probably using nested conditionals in one line is a problem for some reason | 10:11 |
| opendevreview | Merged openstack/openstack-ansible-os_nova master: Don't fail when nova_console_type is disabled https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/820246 | 10:11 |
| noonedeadpunk | well since adding `+` before last `%}` fixes that which is what exactly decribed in whitespace control | 10:21 |
| noonedeadpunk | `you can manually disable the trim_blocks behavior by putting a plus sign (+) at the end of a block` | 10:22 |
| opendevreview | Merged openstack/ansible-role-uwsgi master: Refactor definition of lock path https://review.opendev.org/c/openstack/ansible-role-uwsgi/+/820208 | 10:22 |
| damiandabrowski[m] | that's right, i'm fixing it right now | 10:23 |
| noonedeadpunk | and by default `a single trailing newline is stripped if present` but yeah, I guess things go wrong when we use that as oneliner... So next line jsut get's stripped | 10:24 |
| opendevreview | Merged openstack/openstack-ansible-os_trove master: Database connection pooling improvements https://review.opendev.org/c/openstack/openstack-ansible-os_trove/+/820243 | 10:24 |
| opendevreview | Merged openstack/openstack-ansible-os_ironic master: Database connection pooling improvements https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/820224 | 10:36 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible-os_aodh master: Database connection pooling improvements https://review.opendev.org/c/openstack/openstack-ansible-os_aodh/+/820226 | 10:39 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible-os_barbican master: Database connection pooling improvements https://review.opendev.org/c/openstack/openstack-ansible-os_barbican/+/820227 | 10:40 |
| noonedeadpunk | damiandabrowski[m]: btw it shouldn't be an issue in defaults though - it only raises while template renderring | 10:40 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible-os_blazar master: Database connection pooling improvements https://review.opendev.org/c/openstack/openstack-ansible-os_blazar/+/820228 | 10:40 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible-os_cinder master: Database connection pooling improvements https://review.opendev.org/c/openstack/openstack-ansible-os_cinder/+/820229 | 10:40 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible-os_cloudkitty master: Database connection pooling improvements https://review.opendev.org/c/openstack/openstack-ansible-os_cloudkitty/+/820230 | 10:40 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible-os_designate master: Database connection pooling improvements https://review.opendev.org/c/openstack/openstack-ansible-os_designate/+/820231 | 10:40 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible-os_glance master: Database connection pooling improvements https://review.opendev.org/c/openstack/openstack-ansible-os_glance/+/820232 | 10:40 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible-os_heat master: Database connection pooling improvements https://review.opendev.org/c/openstack/openstack-ansible-os_heat/+/820233 | 10:41 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible-os_magnum master: Database connection pooling improvements https://review.opendev.org/c/openstack/openstack-ansible-os_magnum/+/820234 | 10:41 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible-os_manila master: Database connection pooling improvements https://review.opendev.org/c/openstack/openstack-ansible-os_manila/+/820235 | 10:41 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible-os_masakari master: Database connection pooling improvements https://review.opendev.org/c/openstack/openstack-ansible-os_masakari/+/820236 | 10:41 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible-os_mistral master: Database connection pooling improvements https://review.opendev.org/c/openstack/openstack-ansible-os_mistral/+/820237 | 10:41 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible-os_murano master: Database connection pooling improvements https://review.opendev.org/c/openstack/openstack-ansible-os_murano/+/820238 | 10:41 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible-os_neutron master: Database connection pooling improvements https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/820223 | 10:41 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible-os_octavia master: Database connection pooling improvements https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/820220 | 10:41 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible-os_placement master: Database connection pooling improvements https://review.opendev.org/c/openstack/openstack-ansible-os_placement/+/820239 | 10:41 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible-os_sahara master: Database connection pooling improvements https://review.opendev.org/c/openstack/openstack-ansible-os_sahara/+/820240 | 10:41 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible-os_senlin master: Database connection pooling improvements https://review.opendev.org/c/openstack/openstack-ansible-os_senlin/+/820241 | 10:41 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible-os_tacker master: Database connection pooling improvements https://review.opendev.org/c/openstack/openstack-ansible-os_tacker/+/820242 | 10:41 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible-os_zun master: Database connection pooling improvements https://review.opendev.org/c/openstack/openstack-ansible-os_zun/+/820222 | 10:41 |
| damiandabrowski[m] | noonedeadpunk: hmm, but aodh which has a connection string defined in defaults also failed, let me have a look | 10:45 |
| opendevreview | James Gibson proposed openstack/openstack-ansible master: Add documentation of security improvements made to Openstack Ansible https://review.opendev.org/c/openstack/openstack-ansible/+/820370 | 10:46 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Define manila_osapi_share_workers for CI https://review.opendev.org/c/openstack/openstack-ansible/+/820010 | 10:48 |
| noonedeadpunk | damiandabrowski[m]: it failed with some upgrade check, si likely to be unrelated | 10:48 |
| damiandabrowski[m] | ahhh i see, let me revert it then | 10:49 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible-os_aodh master: Database connection pooling improvements https://review.opendev.org/c/openstack/openstack-ansible-os_aodh/+/820226 | 10:50 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible-os_cloudkitty master: Database connection pooling improvements https://review.opendev.org/c/openstack/openstack-ansible-os_cloudkitty/+/820230 | 10:51 |
| damiandabrowski[m] | noonedeadpunk: is there anything left we really want to push before X release? | 11:04 |
| jrosser | damiandabrowski[m]: there are a bunch of role default var names that you've changed and these really should have a release note | 11:09 |
| jawad-axd | Hi all here, I have runnign OSA env. (wallaby, ubuntu 20.04) with external ceph cluster. I am running manila, cinder already. Want to install/integrate object storage where I can use horizon to create containers and objects. I am not sure how it will do it, since it should be just integration of keystone and rados gateway. How should I define it in openstack_user-config.yaml and user_variables.yaml? | 11:09 |
| jrosser | anyone upgrading needs to know that to update their overrides | 11:09 |
| jawad-axd | @jrosser Any comment on that? Thanks | 11:10 |
| jrosser | jawad-axd: you already run radosgw in the external ceph? | 11:11 |
| jawad-axd | yes | 11:12 |
| jrosser | ok | 11:12 |
| jrosser | so, just like when you integrate external ceph mon with OSA you can also integrate external rgw | 11:12 |
| opendevreview | Merged openstack/openstack-ansible-ceph_client master: Update ceph clients release to pacific https://review.opendev.org/c/openstack/openstack-ansible-ceph_client/+/820009 | 11:13 |
| jrosser | jawad-axd: see here https://github.com/openstack/openstack-ansible/blob/master/playbooks/ceph-rgw-install.yml#L17 | 11:13 |
| noonedeadpunk | jrosser: I think you meant to more explicit rather then just https://review.opendev.org/c/openstack/openstack-ansible/+/819424/6/releasenotes/notes/db-pooling-7c42f3aed39d5fc9.yaml ? | 11:13 |
| jrosser | if OSA has deployed the radosgw there will be a host group called {{ rgw_group_name }} | 11:14 |
| jrosser | but see that it is also looking at the variable ceph_rgws | 11:14 |
| jrosser | that is where you can give a list of external radosgw, just like you would have done for your mon | 11:14 |
| noonedeadpunk | I'm not sure if we should have reno for _each_ service, but might be indeed we need more explicit version of waht exactly has changed | 11:16 |
| damiandabrowski[m] | noonedeadpunk: i think it's about changes like this: https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/820225/1/defaults/main.yml | 11:16 |
| damiandabrowski[m] | for consistency, i've renamed keystone_database_max_pool_size -> keystone_db_max_pool_size as an example | 11:16 |
| jrosser | jawad-axd: to make the horizon dashboard work you must set the RGW up to serve swift API, even if you actually want S3 for users | 11:16 |
| noonedeadpunk | oh, it's bad) | 11:17 |
| damiandabrowski[m] | i can find all repos when i made this changes and write releasenote for them | 11:17 |
| jrosser | it's quite a big deal to change role defaults | 11:17 |
| noonedeadpunk | at least we should cover that with reno and keep compatability for at least one release | 11:17 |
| noonedeadpunk | yeah, agree | 11:17 |
| jrosser | damiandabrowski[m]: defaults/main.yml is kind of the 'published API' for the roles and theres an expectation of reasonable stability there | 11:18 |
| jrosser | those vars end up in deployers user_variables and group/host_vars to provide the customisations that they want | 11:19 |
| jrosser | so we have to be careful when changing / renaming things as it can break a lot of stuff | 11:19 |
| opendevreview | Merged openstack/openstack-ansible master: Update ceph-ansible release to pacific https://review.opendev.org/c/openstack/openstack-ansible/+/820008 | 11:20 |
| jawad-axd | @jrosser Thanks. I am gonna try it now. | 11:21 |
| damiandabrowski[m] | thanks jrosser let me check how many repos are affected | 11:21 |
| jrosser | jawad-axd: there will be a few steps - set up the keystone integration with osa playbooks, there will be stuff needed in the radosgw config files for keystone | 11:22 |
| jrosser | you need connectivity between the rgw and haproxy, and also the service catalog entries for object storage need to be present and point to the radosgw endpoint/vip | 11:23 |
| jrosser | i guess my point is that this is not a automatic setup, it is expected you'll need to understand/address a bunch of things to make it work | 11:24 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server master: Allow to provide policy state https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/819917 | 11:27 |
| damiandabrowski[m] | jrosser: noonedeadpunk i have changed variable names only for keystone. So if it's only a single repo, do we still need to keep compatibility for one release or just write deprecation info in releasenotes? | 11:28 |
| noonedeadpunk | if we are to follow best practises you know the answer. esp considering that keystone is core service | 11:29 |
| jawad-axd | @jrosser I am aware of that, its not plain automatic setup. Have seen few articles on it. | 11:31 |
| noonedeadpunk | but at least we should cover with release note | 11:32 |
| damiandabrowski[m] | thanks, i'll fix it later today | 11:37 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Do not fail when nova console is disabled https://review.opendev.org/c/openstack/openstack-ansible/+/820192 | 11:40 |
| *** arxcruz|rover is now known as arxcruz | 12:43 | |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_octavia master: Database connection pooling improvements https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/820220 | 13:47 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_zun master: Database connection pooling improvements https://review.opendev.org/c/openstack/openstack-ansible-os_zun/+/820222 | 13:47 |
| noonedeadpunk | we need to merge https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/820203/1 to fix octavia | 13:57 |
| noonedeadpunk | this will also bring pile of patches btw... https://review.opendev.org/c/openstack/openstack-ansible-tests/+/819915 | 13:59 |
| spatel | noonedeadpunk or jrosser - could you take a look at this log and tell me what is wrong here? - https://paste.opendev.org/show/811436/ | 14:10 |
| jrosser | i would guess you have type/error/formatting problem in provider_networks in openstack_user_config.yml | 14:12 |
| jrosser | *typo hah | 14:12 |
| spatel | hmm let me see | 14:14 |
| spatel | all other playbooks working fine | 14:14 |
| jrosser | they likley would, neutron is where provider_networks needs to be dealt with | 14:16 |
| spatel | jrosser can you see any typo here - https://paste.opendev.org/show/811438/ | 14:30 |
| spatel | all looks good to me | 14:30 |
| jamesdenton | spatel you're missing a container_interface on the br-provider network | 14:52 |
| jamesdenton | it's the logic currently requires it but it may not really be necessary | 14:53 |
| spatel | hmm | 14:53 |
| jamesdenton | just set container_interface: eth11 and you should be fine. I will not be used | 14:54 |
| spatel | can you give me snippet ? how did it work in first time? | 14:54 |
| spatel | oh in - container_bridge: br-provider section right? | 14:54 |
| jamesdenton | well actually, because northd is in a container it could be used, but it's essentially trying to connect a veth to the br-provider bridge in that container | 14:55 |
| jamesdenton | yes, that section | 14:55 |
| spatel | re-running playbook and look like your nailed it :) | 14:55 |
| jamesdenton | i'm not using lxc for neutron anymore so i can't compare | 14:56 |
| spatel | that works! | 14:56 |
| jamesdenton | Cool cool. Just FYI, i ran into an issue from Wallaby->Xena (master) yesterday w/ OVN - "Could not retrieve schema from tcp:10.20.0.30:6641,tcp:10.20.0.22:6641,tcp:10.20.0.23:6641". Seems northd isn't starting, and the logs show this: https://paste.opendev.org/show/811439/ | 14:57 |
| jamesdenton | I'm not sure why it's expecting SSL now | 14:57 |
| jamesdenton | i have to step away, still looking into that one | 14:58 |
| spatel | i have seen that error | 14:58 |
| spatel | make sure you have this config in place - https://paste.opendev.org/show/811440/ | 14:59 |
| spatel | do you have 3 node infra or single node? | 14:59 |
| spatel | BRB - meeting time | 15:00 |
| jamesdenton | spatel that's the thing, i never configured SSL for OVN! | 15:11 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Define manila_osapi_share_workers for CI https://review.opendev.org/c/openstack/openstack-ansible/+/820010 | 15:28 |
| *** chandankumar is now known as raukadah | 15:29 | |
| opendevreview | Merged openstack/openstack-ansible-os_senlin master: Database connection pooling improvements https://review.opendev.org/c/openstack/openstack-ansible-os_senlin/+/820241 | 15:34 |
| opendevreview | Merged openstack/openstack-ansible-os_placement master: Database connection pooling improvements https://review.opendev.org/c/openstack/openstack-ansible-os_placement/+/820239 | 15:38 |
| spatel | jamesdenton ah! very odd, then may be by default ovn using SSL. if you don't care about SSL then just switch to tcp | 15:38 |
| spatel | but very odd that you are seeing that ssl issue | 15:39 |
| opendevreview | Merged openstack/openstack-ansible-os_glance master: Database connection pooling improvements https://review.opendev.org/c/openstack/openstack-ansible-os_glance/+/820232 | 15:41 |
| opendevreview | Merged openstack/openstack-ansible-os_designate master: Database connection pooling improvements https://review.opendev.org/c/openstack/openstack-ansible-os_designate/+/820231 | 15:43 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible-os_keystone master: Database connection pooling improvements https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/820225 | 15:43 |
| opendevreview | Merged openstack/openstack-ansible-os_tacker master: Database connection pooling improvements https://review.opendev.org/c/openstack/openstack-ansible-os_tacker/+/820242 | 15:43 |
| damiandabrowski[m] | I have fixed db-pooling for keystone, please take a look when You have a moment: https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/820225/ | 15:45 |
| damiandabrowski[m] | I also wonder how can we ensure that we won't forget to remove these variables in next release? :D | 15:45 |
| noonedeadpunk | um, I'm not 100% sure but I'd say there's an issue | 15:45 |
| noonedeadpunk | usually we leave # TODO(nickname): Do $stuff in $release | 15:46 |
| opendevreview | Merged openstack/openstack-ansible-os_heat master: Database connection pooling improvements https://review.opendev.org/c/openstack/openstack-ansible-os_heat/+/820233 | 15:46 |
| opendevreview | Merged openstack/openstack-ansible-os_barbican master: Database connection pooling improvements https://review.opendev.org/c/openstack/openstack-ansible-os_barbican/+/820227 | 15:47 |
| noonedeadpunk | damiandabrowski[m]: but if you tested and it works that way as well - it's okey | 15:48 |
| opendevreview | Merged openstack/openstack-ansible-os_blazar master: Database connection pooling improvements https://review.opendev.org/c/openstack/openstack-ansible-os_blazar/+/820228 | 15:49 |
| opendevreview | Merged openstack/openstack-ansible-os_masakari master: Database connection pooling improvements https://review.opendev.org/c/openstack/openstack-ansible-os_masakari/+/820236 | 15:49 |
| noonedeadpunk | damiandabrowski[m]: there's also an issue with neutron https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/820223 | 15:49 |
| damiandabrowski[m] | ouh...thanks | 15:51 |
| opendevreview | Merged openstack/openstack-ansible-os_cloudkitty master: Database connection pooling improvements https://review.opendev.org/c/openstack/openstack-ansible-os_cloudkitty/+/820230 | 15:51 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible-os_keystone master: Database connection pooling improvements https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/820225 | 15:53 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible-os_keystone master: Database connection pooling improvements https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/820225 | 15:54 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible-os_neutron master: Database connection pooling improvements https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/820223 | 15:55 |
| opendevreview | Merged openstack/openstack-ansible-os_sahara master: Database connection pooling improvements https://review.opendev.org/c/openstack/openstack-ansible-os_sahara/+/820240 | 15:55 |
| opendevreview | Merged openstack/openstack-ansible-os_aodh master: Database connection pooling improvements https://review.opendev.org/c/openstack/openstack-ansible-os_aodh/+/820226 | 16:04 |
| opendevreview | Merged openstack/openstack-ansible-os_magnum master: Database connection pooling improvements https://review.opendev.org/c/openstack/openstack-ansible-os_magnum/+/820234 | 16:07 |
| noonedeadpunk | damiandabrowski[m]: and, you renamed neutron_db_pool_size to neutron_db_max_pool_size | 16:12 |
| noonedeadpunk | which I'd say we can avoid doing at all.... | 16:13 |
| spatel | jrosser i am playing with ovn pki implementation and i am able to create/install certs but i am seeing its using same CA/cert/key file to copy on each compute nodes, what we need to do to create each compute has own cert? | 16:23 |
| spatel | what is best option copy same cert to all compute or do you recommend different cert for each compute? | 16:24 |
| spatel | i think option one is easy and simple | 16:25 |
| spatel | i am using rabbitMQ task/main.yml example to implement that with ovn | 16:25 |
| jrosser | if you look at os_nova you can see in the role defaults that the cert contents depends on a bunch of host variables https://github.com/openstack/openstack-ansible-os_nova/blob/master/defaults/main.yml#L569-L581 | 16:25 |
| jrosser | and then when the certs are installed they are referencing the appropriately named certs on the deploy host https://github.com/openstack/openstack-ansible-os_nova/blob/master/defaults/main.yml#L590-L600 | 16:27 |
| spatel | hmm let me check work flow from os_nova example. | 16:28 |
| jrosser | for the rabbitmq role it also creates uniqe certs per host https://github.com/openstack/openstack-ansible-rabbitmq_server/blob/master/defaults/main.yml#L153-L158 | 16:29 |
| spatel | This is what i did in os_neutron/default/main.yml - https://paste.opendev.org/show/811449/ | 16:29 |
| jrosser | don't be confused by the rabbit role also creating a CA - you wont need to do that | 16:29 |
| jrosser | the rabbit role has it in the defaults because that role could be used outside OSA so needs to stand alone | 16:29 |
| jrosser | thats not the case for os_neutron | 16:29 |
| spatel | I have noticed RabbitMQ has own CA, so you are saying we shouldn't do that with OVN correct? | 16:31 |
| jrosser | correct | 16:31 |
| jrosser | strip it back to look more like what is done for nova | 16:31 |
| jrosser | rabbit / galera / haproxy roles can create their own CA as they have a life outside OSA | 16:32 |
| spatel | make sense | 16:32 |
| jrosser | hopefully this is not too complex | 16:32 |
| jrosser | everything is driven by those data structures in defaults/main.yml | 16:32 |
| spatel | i am just wrapping my head around this structure. where is the default OSA CA files located? - https://paste.opendev.org/show/811450/ | 16:34 |
| jrosser | those are the certs | 16:35 |
| spatel | where is the CA file? we don't keep them on deployment node? | 16:35 |
| jrosser | look in something like /etc/openstack_deploy/pki/roots/..... | 16:35 |
| spatel | ah!! got it | 16:36 |
| jrosser | so just like when you define the certs in a role its a list | 16:36 |
| spatel | sorry, i didn't pay attention to that | 16:36 |
| jrosser | same for the roots so there can be as many as we need | 16:36 |
| jrosser | and the "signed_by" in a cert says which root has generated it | 16:36 |
| spatel | when i will take example from os_nova then my all cert will get singed by ExampleCorpRoot correct? | 16:36 |
| jrosser | this https://github.com/openstack/openstack-ansible-os_nova/blob/master/defaults/main.yml#L574 | 16:37 |
| jrosser | which goes to https://github.com/openstack/openstack-ansible-os_nova/blob/master/defaults/main.yml#L564 | 16:38 |
| jrosser | in turn goes to https://opendev.org/openstack/openstack-ansible/src/branch/master/inventory/group_vars/all/ssl.yml#L25 | 16:38 |
| spatel | look like i can pretty much copy paste these stanza in neutron playbook and that should work after adjusting dir path etc | 16:38 |
| jrosser | yes exactly, thats the idea | 16:39 |
| jrosser | the stuff from defaults/main.yml should be kind of portable across roles with some search/replace | 16:39 |
| spatel | sweet! let me give it a shot | 16:39 |
| jrosser | nova got a bit complicated because libvirt/qemu/vnc all wanted slightly different things | 16:40 |
| jrosser | so hopfully OVN is a bit more consistent and you can keep it simple | 16:40 |
| spatel | why don't we create standard place for all cert like /etc/pki/ so that way we don't need to know which directory it will go | 16:40 |
| jrosser | hmm? | 16:40 |
| spatel | for example /etc/pki/libvirt | 16:41 |
| jrosser | well some things have locations they expect to find stuff | 16:41 |
| spatel | if we dump all certs in /etc/pki/ then we don't need to adjust PKI code for each role | 16:42 |
| jrosser | right, but if (for example) libvirt looks somewhere else, it needs to be all adjustable | 16:42 |
| spatel | let me first try your code out and see.. i may be saying lots of word which doesn't make sense | 16:43 |
| jrosser | if OVN lets you specify a path to the files, thats cool | 16:43 |
| jrosser | not everything does and you have to put it exactly where the program wants it | 16:43 |
| spatel | +1 | 16:43 |
| spatel | ovn is flexible about path | 16:44 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible-os_neutron master: Database connection pooling improvements https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/820223 | 16:45 |
| jrosser | ok so in that case you can use a standard location just like we do for rabbit https://github.com/openstack/openstack-ansible-rabbitmq_server/blob/master/defaults/main.yml#L105 | 16:45 |
| spatel | ok | 16:46 |
| noonedeadpunk | jrosser: any idea how we can avoid oom killing qemu for manila CI? | 16:54 |
| noonedeadpunk | that only happens for centos though | 16:54 |
| noonedeadpunk | I'm about to adjust oom score tbh | 16:54 |
| jrosser | manila is a ceph deployment? | 16:54 |
| noonedeadpunk | not sure what would be killed instead though | 16:54 |
| noonedeadpunk | it is | 16:54 |
| noonedeadpunk | we can probably avoid rgw somehow? | 16:55 |
| jrosser | there is quite possibly a load of ram chewed up there | 16:55 |
| jrosser | for manila test certainly yes | 16:55 |
| jrosser | and we can make sure that for CI we just make one OSD | 16:55 |
| jrosser | like no replicatin | 16:55 |
| noonedeadpunk | according to output of https://zuul.opendev.org/t/openstack/build/fa6f6db2d57a422ba3c89aea4f07359f/log/logs/openstack/instance-info/ps_20-55-33.log.txt most consumers are rabbit, galera and neutron. but ofc ceph is also big enough | 16:56 |
| noonedeadpunk | hm, indeed, osd are ram consuming. | 16:56 |
| noonedeadpunk | would need to check how to do that and leave cluster healthy... | 16:57 |
| noonedeadpunk | but thanks for idea - that was helpful! | 16:57 |
| spatel | add swap is only solution to stop oom temporally | 16:57 |
| noonedeadpunk | I was thinking about that. but it would make jobs really slow... and I guess even with swapiness 0 it begin to be used before manila even start... | 16:58 |
| noonedeadpunk | eventually jobs were passing before :( | 16:58 |
| jrosser | there is also a ram target for OSD | 16:59 |
| jrosser | which ceph-ansible defaults to (host ram - safety margin) / num osd | 16:59 |
| jrosser | and if you set the is_hci flag for a host it makes a very very different assumption, that most of the host ram is required for other things, rather than OSDs | 17:00 |
| jrosser | we were looking at this very recently as we have a set of mon and osd co-located in one deployment and it was doing OOM | 17:00 |
| jrosser | noonedeadpunk: https://github.com/ceph/ceph-ansible/blob/master/group_vars/all.yml.sample#L339-L342 | 17:02 |
| noonedeadpunk | mon and osd on same host not greatest idea from my experience | 17:03 |
| noonedeadpunk | it's even better to have mons in containers | 17:03 |
| jrosser | yes in this case it's a LXD mon on a host which also has OSD | 17:03 |
| noonedeadpunk | as during upgrade, when you need to upgrade monitors first - you will get ceph-common and ceph-osd upgraded as well... | 17:03 |
| jrosser | but still the config for the OSD gives it all the ram | 17:04 |
| noonedeadpunk | yeah | 17:04 |
| jrosser | another gotcha was adding more OSD to an existing storage node | 17:04 |
| jrosser | there is a ceph-ansible bug where it doesnt restart the existing OSD with the new memory target | 17:04 |
| noonedeadpunk | ah... | 17:04 |
| jrosser | so you also end up OOM when the extra disks go in | 17:04 |
| noonedeadpunk | wel, our stor teram for reasons I don't understand drift away from ceph-ansible.... | 17:05 |
| opendevreview | James Gibson proposed openstack/openstack-ansible master: Add documentation of security improvements made to Openstack Ansible https://review.opendev.org/c/openstack/openstack-ansible/+/820370 | 17:18 |
| opendevreview | Merged openstack/openstack-ansible-os_octavia master: Refactor definition of lock path https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/820203 | 17:30 |
| opendevreview | Merged openstack/openstack-ansible-os_octavia master: Use config_template as a collection https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/819867 | 17:30 |
| evrardjp | hello mgoddard | 18:02 |
| spatel | jrosser what is servercert.pem vs clientcert.pem ? | 19:15 |
| jrosser | where? | 19:15 |
| spatel | https://github.com/openstack/openstack-ansible-os_nova/blob/master/defaults/main.yml#L591 | 19:15 |
| spatel | https://github.com/openstack/openstack-ansible-os_nova/blob/master/defaults/main.yml#L604 | 19:15 |
| jrosser | for nova live migration it needs "mutual tls" | 19:16 |
| jrosser | so there is a cert on the server side | 19:16 |
| jrosser | and also there is a cert presented "as a client" to another live migration host | 19:16 |
| spatel | in ovn case i only need 3 file CA / certs / key | 19:16 |
| jrosser | so the validation is bidirectional | 19:16 |
| jrosser | well it depends | 19:17 |
| jrosser | i don't know if OVN wants (or is able) to validate clients | 19:17 |
| jrosser | theres two things possible with TLS, secure the traffic | 19:17 |
| jrosser | and the optionally you can validate the authenticity of the client | 19:18 |
| spatel | Here is the flow of ovn - https://satishdotpatel.github.io/ovn-ssl-setup-with-openstack/ | 19:18 |
| spatel | I am generating compute cert and then sign it and copy it to compute machine. | 19:19 |
| jrosser | well, according to this https://bugzilla.redhat.com/show_bug.cgi?id=1601926 it is possible to use mutual tls for ovn | 19:19 |
| jrosser | you are generating the compute server cert | 19:19 |
| jrosser | i think? | 19:21 |
| spatel | I created CA authority and then i generate certificate for each components.. like ovn-central and ovn-controller etc.. | 19:21 |
| spatel | then i copy those certs/key to compute node and then my compute node start using SSL to talk to ovn-central component | 19:22 |
| jrosser | the trouble is that the diagram doesnt show which is a client and which is a server | 19:22 |
| jrosser | one end must initiate the connection? | 19:23 |
| spatel | ovn-central is server component | 19:23 |
| spatel | ovn-controller which run on compute connect to ovn-central (so assuming computes are always client and ovn-central is server which is located on infra node) | 19:24 |
| spatel | ovn-central never initiate first connection to compute node or never make connection. only compute make first connection. | 19:26 |
| spatel | I think in OVN clustering we may need mutual connection because each cluster node talk to member vise verse. | 19:27 |
| jrosser | no thats not what it means | 19:27 |
| jrosser | server = listening service | 19:27 |
| jrosser | client = thing that connects to server | 19:27 |
| jrosser | so server certificate, is there basically to secure the traffic to/from the client | 19:28 |
| jrosser | client certificate lets the server validate that it is a legitimate client | 19:28 |
| spatel | Reading - https://www.cloudflare.com/learning/access-management/what-is-mutual-tls/ | 19:28 |
| jrosser | for normal web browsing like https there is almost never a client cert | 19:29 |
| spatel | oh! now i am following you | 19:29 |
| jrosser | so thats kind of whats not clear with the OVN stuff, which is intended to be server side cert, and which to be client | 19:30 |
| jrosser | we have some mTLS stuff here to secure confluence/jira etc, and the users need a specially issued personal client cert in their browsers | 19:31 |
| jrosser | it lets the server side validate that the user is legitimate and allowed | 19:32 |
| jrosser | so for this OVN stuff thats why it's important to be clear what the purpose of the cert is (and comment as such, you'll see that in os_nova) | 19:33 |
| spatel | I got your point, this way it will provide 100% security both way.. | 19:33 |
| jrosser | yes thats right | 19:33 |
| jrosser | it means that nothing can connect to the ovn-central unless it's another ovn component | 19:33 |
| jrosser | well, actually only something that has a certificate which can validate against the provided CA | 19:34 |
| jrosser | you can examine the certificates from your example with `openssl x509 -in certificate.crt -text -noout` | 19:35 |
| jrosser | then see here https://github.com/openstack/openstack-ansible-os_nova/blob/master/defaults/main.yml#L579-L581 | 19:35 |
| jrosser | the purpose of the cert is encoded inside it, for os_nova for simplicity we create one dual purpose server & client cert | 19:36 |
| spatel | I can understand in NOVA case we need mTLS so two compute node trust each other. but in OVN case i don't think we have that scenario. | 19:36 |
| jrosser | it looks like that is what is being set up in your ovn blog? | 19:37 |
| spatel | in OVN case all we want to do is secure communication between ovn-central and ovn-controller | 19:37 |
| spatel | Yes, that is what i did | 19:38 |
| jrosser | otherwise what is the certificate for on the ovn-controller, if it's only a client | 19:38 |
| spatel | hmm! Let me first finish this playbook and then see how it goes.. look like i have more question then answer. i would like jamesdenton also take a look and give some input if i am confused in process. | 19:41 |
| spatel | for now i am keeping things as it's to just see what goes where. | 19:42 |
| jrosser | see the -p -c & -C flags here https://www.ovn.org/support/dist-docs/ovn-controller.8.html | 19:44 |
| jrosser | this is just normal mTLS stuff by the look of it | 19:44 |
| spatel | hmm.. yes seems like.. | 19:45 |
| spatel | because i am installing certs on each components that means its kinda using mTLS | 19:46 |
| jrosser | yes, so with the PKI role you need to get the flags in the certs right | 19:46 |
| jrosser | thats why it might be useful to inspect the ones that the `ovs-pki` tool made for you | 19:46 |
| spatel | ovs-pki is just toolchain to create certs etc easy way. | 19:47 |
| spatel | i don't think it does anything magical for ovn | 19:48 |
| spatel | jrosser - i have added this snippet in os_neutron/default/main.yml - https://paste.opendev.org/show/811455/ | 20:08 |
| spatel | when i run os_neutron playbook somehow these code didn't trigger | 20:09 |
| jrosser | do you add the pki role? | 20:10 |
| jrosser | and also it seems that this needs to treat the controller and central nodes differently | 20:11 |
| opendevreview | Merged openstack/openstack-ansible-os_murano master: Database connection pooling improvements https://review.opendev.org/c/openstack/openstack-ansible-os_murano/+/820238 | 20:12 |
| spatel | do you add the pki role? where i should add that? | 20:12 |
| jrosser | see here in os_nova https://github.com/openstack/openstack-ansible-os_nova/blob/master/tasks/main.yml#L122-L154 | 20:14 |
| jrosser | also see how it is used twice with different data for libvirt and vnc | 20:14 |
| jrosser | this is what i mean by you may have different requirements for the controller vs. central | 20:15 |
| spatel | This is what i was looking - https://github.com/openstack/openstack-ansible-os_nova/blob/master/tasks/main.yml#L122-L154 | 20:16 |
| jrosser | yes | 20:17 |
| spatel | let me add that and see.. yes i will create two different task to handle controller vs central | 20:17 |
| opendevreview | Merged openstack/openstack-ansible-os_mistral master: Database connection pooling improvements https://review.opendev.org/c/openstack/openstack-ansible-os_mistral/+/820237 | 20:25 |
| spatel | any idea - https://paste.opendev.org/show/811456/ | 20:29 |
| spatel | jrosser ^ | 20:32 |
| jrosser | thats wierd, no i have no idea | 20:36 |
| spatel | in pki/default/main.yml has pki_certificates: [] | 20:38 |
| spatel | but default should work | 20:38 |
| jrosser | right, but for nova it's overidden https://github.com/openstack/openstack-ansible-os_nova/blob/master/tasks/main.yml#L132 | 20:39 |
| spatel | so where we define nova_pki_compute_certificates ? | 20:40 |
| spatel | i am not seeing it anywhere | 20:40 |
| spatel | I am searching grep "nova_pki_compute_certificates" * -r and found nothing | 20:41 |
| jrosser | https://github.com/openstack/openstack-ansible-os_nova/blob/master/defaults/main.yml#L569-L581 | 20:42 |
| spatel | hmm | 20:42 |
| jrosser | i'm just giving you examples from nova as thats merged and works | 20:42 |
| spatel | yes i can see it now | 20:43 |
| spatel | i do have that in os_neutron also | 20:43 |
| spatel | look like need a break :) i will debug it in night with fresh beer | 20:47 |
| spatel | jrosser Thank you for helping me out, let me keep debug and see.. | 20:51 |
| jrosser | no problem | 20:51 |
| spatel | Have a good weekend | 20:51 |
| jrosser | you too | 20:52 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!