| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-tests master: Bump Ansible version for tests repo https://review.opendev.org/c/openstack/openstack-ansible-tests/+/921434 | 08:09 |
|---|---|---|
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-tests master: Remove legacy centos-8-stream job definition https://review.opendev.org/c/openstack/openstack-ansible-tests/+/922408 | 08:09 |
| jrosser | noonedeadpunk: do you have an idea about how we can adjust openstack_resources to use the cirros images already on the CI nodes? | 08:27 |
| jrosser | they are already in /opt/cache/files https://opendev.org/openstack/project-config/src/branch/master/nodepool/elements/cache-devstack/source-repository-images | 08:28 |
| jrosser | we need to make image_download_path point to that https://github.com/openstack/openstack-ansible-plugins/blob/master/roles/openstack_resources/tasks/image_upload.yml#L45 | 08:30 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Delete glusterd data along with container data https://review.opendev.org/c/openstack/openstack-ansible/+/924910 | 08:36 |
| noonedeadpunk | I think there was a way to point to a file rather then url | 08:43 |
| noonedeadpunk | so you can supply `filename` instead of `url` field for the image | 08:44 |
| noonedeadpunk | https://opendev.org/openstack/openstack-ansible-plugins/src/branch/master/roles/openstack_resources/tasks/image_upload.yml#L89 | 08:44 |
| noonedeadpunk | but then - I'd guess the file should be somehow on utility host.... | 08:45 |
| noonedeadpunk | and then there's nothing in the role to transfer the image | 08:46 |
| noonedeadpunk | but if we bind mount the folder to utility container :D | 08:47 |
| jrosser | iirc /openstack is mounted verywhere | 09:03 |
| jrosser | so we could symlink that one the host (i think this is also how we deal with installing services from the zuul repos with file:///) | 09:03 |
| jrosser | what we want to happen is the uri module to see that the file is already there in the download location and not get it again | 09:04 |
| jrosser | but for outside CI we want it to download when the cache is not present | 09:04 |
| jrosser | btw this is exactly the same issue i have with the magnum stuff were it downloads the huge images over and over because of the use of a tmpdir | 09:05 |
| noonedeadpunk | I think how we're skipping image download here - is fetching all images and selecting by checksum | 09:11 |
| noonedeadpunk | like if place this as first task: https://opendev.org/openstack/openstack-ansible-plugins/src/branch/master/roles/openstack_resources/tasks/image.yml#L24-L34 | 09:13 |
| noonedeadpunk | but yeah, then you kinda rely on checksum or some metadata in uploaded images | 09:13 |
| noonedeadpunk | so I was thinking in this direction rather then making tmp dir a permanent one | 09:16 |
| jrosser | sure but directly before that its doing this https://opendev.org/openstack/openstack-ansible-plugins/src/branch/master/roles/openstack_resources/tasks/image_upload.yml#L31-L36 | 09:16 |
| * noonedeadpunk still struggling to understand wtf with uwsgi | 09:16 | |
| jrosser | so every single time we get a different tmpdir and it downloads the image regardless? | 09:16 |
| jrosser | oh i see - you mean if you already know the checksum in glance then you could use that to check if downloading is needed | 09:17 |
| noonedeadpunk | right now yes. but not if we'd images | rejectattr('checksum', 'in', available_images | map(attribute='checksum')) | 09:18 |
| noonedeadpunk | or smth like that | 09:18 |
| jrosser | thats kind of ok until wanting to do format conversion | 09:18 |
| noonedeadpunk | or by some metadata/property/tag we set (or can set) to the image to filter for | 09:18 |
| jrosser | the ansible we have for this knows how to turn qcow to raw before putting into glance becasue of ceph | 09:19 |
| noonedeadpunk | we build images with dib in raw right away here on the contrary | 09:19 |
| noonedeadpunk | so why I'm a bit scared about not tmpdir - is that it might get huge really fast | 09:20 |
| jrosser | yes it could | 09:20 |
| noonedeadpunk | esp once you start dealing with win server + mssql or smth like that.... | 09:23 |
| noonedeadpunk | so, what I did, was packing images, and then having 2 checksums - for packed and resulting image | 09:23 |
| noonedeadpunk | then you can verify checksum right on download while still reject already existing ones from glance.. | 09:24 |
| noonedeadpunk | I'd guess that might work with conversion as well? | 09:24 |
| noonedeadpunk | like add `url_checksum` property? | 09:24 |
| noonedeadpunk | but dunno if conversion to raw will result in same sha each time or not... | 09:26 |
| noonedeadpunk | we could also add some optional image_build_timestamp option and compare that instead of checksum | 09:26 |
| jrosser | seems there are lots of use cases | 09:27 |
| jrosser | we dont have the engineering effort to build our own images | 09:27 |
| jrosser | so it's really a case of doing the best automation we can for uploading a list of upstream ubuntu/debian/fedora/whatever | 09:28 |
| noonedeadpunk | and proceed with download only if timestamp is more modern then uploaded | 09:28 |
| noonedeadpunk | but isn't it harder to detect url for download upstream image then just run dib with pre-defined flags? | 09:28 |
| noonedeadpunk | anyway | 09:28 |
| jrosser | yeah i mean if the upstream image is broken in some way it's not my problem | 09:29 |
| jrosser | but if the one i build is broken that it becomes my problem | 09:29 |
| noonedeadpunk | well... it would be mine regardless who broke the image | 09:29 |
| jrosser | :) | 09:29 |
| noonedeadpunk | as end-users don't care where they're coming from | 09:29 |
| noonedeadpunk | but that they work | 09:30 |
| noonedeadpunk | so yeah, we run a tempest job with the image to ensure image is working, and if it does - it's promoted as artifact | 09:31 |
| noonedeadpunk | so storing same set of images twice - would love to avoid that frankly speaking... | 09:31 |
| noonedeadpunk | but yeah, worth putting some cycles into that for sure | 09:53 |
| noonedeadpunk | as current state is just annoying | 09:53 |
| noonedeadpunk | crap, really don't get this uwsgi stuff... get pcre on 2 out of 3 keystone containers in infra lxc | 10:28 |
| noonedeadpunk | the first one does not get it | 10:28 |
| noonedeadpunk | the one that does not get it is the first which being ran against | 10:28 |
| noonedeadpunk | so likely some ordering issue... | 10:29 |
| jrosser | does it build the wheel first time? | 10:29 |
| noonedeadpunk | it does | 10:30 |
| noonedeadpunk | and then it apparently uses the wheel... | 10:30 |
| jrosser | but actually what is there is some upstream thing rather than our wheel? | 10:30 |
| noonedeadpunk | as in python_venv_build log on containers I see wheel being donwloaded | 10:31 |
| noonedeadpunk | and seems it also kinda depends on presence of the libpcre not only on repo container, but also where it runs | 10:33 |
| jrosser | if it was dynamically linked, then totally | 10:34 |
| jrosser | the -dev package would bring in the headers to be able to compile against the library | 10:35 |
| noonedeadpunk | but then it's kinda inconsistent with how it works with ssl... | 10:35 |
| jrosser | but the lib itself would be also needed in the target | 10:35 |
| jrosser | ldd would tell you | 10:35 |
| noonedeadpunk | ah.... | 10:36 |
| noonedeadpunk | fair | 10:36 |
| noonedeadpunk | and libssl is jsut default present everywhere I'd guess | 10:36 |
| gokhan | hello folks, how can we change deployment node? is it enough to move /opt/openstack-ansible /etc/openstack_deploy/ /etc/ansible and run bootstrap-ansible.sh script? | 10:38 |
| jrosser | iirc this is why there are separate package lists in the pyhton_venv_build role for build host and target host | 10:38 |
| noonedeadpunk | just what makes things worse, is that uwsgi uses completely own setup.py thing | 10:38 |
| noonedeadpunk | some profiling, etc.. | 10:39 |
| jrosser | yes, i have memory of fixing an upstream bug in uwsgi and it was 8-O | 10:39 |
| noonedeadpunk | gokhan: it's enough to move /etc/openstack_deploy and run bootstrap | 10:39 |
| gokhan | noonedeadpunk, ok thanks :) | 10:40 |
| noonedeadpunk | /opt/openstack-ansible - is just a clone, if you don't have any changes in there (which ideally you shouldn't) - just clone and checkout to the same version | 10:41 |
| noonedeadpunk | so the only folder that _really_ matter on deploy host is openstack_deploy one - rest should be just brough from git and be stateless | 10:42 |
| noonedeadpunk | jrosser: yeah, you're completely right about that | 10:43 |
| noonedeadpunk | keystone is not smallest/fastest playbook to retry :( | 10:47 |
| gokhan | noonedeadpunk, yes I don't have any customization on /opt/openstacxk-ansible. this week I have upgraded one of my env from victoria-em to antelope seamlessly :) | 10:47 |
| noonedeadpunk | ah, that sounds really nice! | 10:47 |
| noonedeadpunk | quite some changes we made since victoria | 10:48 |
| gokhan | yes there are lots of changes from victoria to antelope. now all of my deployments are in antelope and next week ı will start to try upgrading from antelope to caracal in test environment. | 10:52 |
| jrosser | i'm going to take a look at whats happening with the ceph job, feels broken | 11:07 |
| noonedeadpunk | crap, really don't understand what's going on.... | 11:16 |
| noonedeadpunk | it's somehow each time results slightly differently.... | 11:17 |
| jrosser | indeed | 11:17 |
| jrosser | ^ i mean for ceph | 11:17 |
| jrosser | same for uwsgi? | 11:17 |
| noonedeadpunk | yeah, I was talking about uwsgi :D | 11:18 |
| noonedeadpunk | like it would be quite nice to somehow override the `required_pcre` during build: https://github.com/unbit/uwsgi/blob/master/uwsgiconfig.py#L1102 | 11:19 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-role-uwsgi master: DNM: Install pcre on destination hosts https://review.opendev.org/c/openstack/ansible-role-uwsgi/+/924922 | 11:42 |
| noonedeadpunk | it kind of feels we're doing smth wrong with storage of wheels | 11:47 |
| noonedeadpunk | so uwsgi wheel is built totally with pcre support: https://paste.openstack.org/show/b67LDrMEOIq9ktejTYjA/ | 11:49 |
| jrosser | for ceph this is not reproducible locally https://paste.opendev.org/show/bivSnUwkyC0MlpsUFq1s/ | 11:50 |
| noonedeadpunk | it somehow feels like smth I saw... | 11:52 |
| jrosser | yeah i saw a whole bunch like this failing | 11:53 |
| noonedeadpunk | and locally you ran also metal, not lxc? | 11:53 |
| jrosser | there was discussion of new setuptools vs. packaging module too for u-c | 11:53 |
| jrosser | but your uwsgi think | 11:53 |
| jrosser | is the wheel name the same as the one you would get anyway from pypi? | 11:53 |
| jrosser | name / version | 11:54 |
| noonedeadpunk | yeah, I think it is | 11:54 |
| jrosser | i suspect the trouble lies there | 11:54 |
| jrosser | with perhaps relative precedence of our local wheels and the upstream ones | 11:54 |
| noonedeadpunk | I feel it might also have smth to do with tags... | 11:55 |
| noonedeadpunk | as this time I even get `--find-links", "http://localhost/os-releases/29.1.0.dev63/ubuntu-22.04-x86_64/wheels"` | 11:55 |
| noonedeadpunk | and connection failure as a result | 11:56 |
| noonedeadpunk | huh | 11:57 |
| noonedeadpunk | this does not get included apparently https://opendev.org/openstack/openstack-ansible/src/branch/master/playbooks/os-keystone-install.yml#L29-L33 | 11:58 |
| noonedeadpunk | * https://opendev.org/openstack/openstack-ansible/src/branch/master/playbooks/os-keystone-install.yml#L73-L77 | 11:58 |
| noonedeadpunk | So we'd need to do smth like that I believe: https://review.opendev.org/c/openstack/openstack-ansible/+/924307 | 11:59 |
| jrosser | for any of our normal "install from source" openstack things we get a version like `magnum==18.0.1.dev4` which is going to win over anything in pypi very obviously | 11:59 |
| noonedeadpunk | we should always win over pypi... | 12:00 |
| noonedeadpunk | I _think_ so at least | 12:01 |
| noonedeadpunk | What I'm kind of annoyed about, that seems we're wasting plenty of time inspecting all venv_default_pip_install_args and our repo container for each single package | 12:02 |
| noonedeadpunk | *inspecting pypi | 12:03 |
| jrosser | i think it's "install this thing across all the indexes you have" | 12:06 |
| noonedeadpunk | https://pastebin.com/raw/cfKYdUtL | 12:07 |
| noonedeadpunk | so after all it does fetch the correct one | 12:08 |
| noonedeadpunk | ok, so I think I was fighting with tags after all... | 12:09 |
| noonedeadpunk | And I was using them heavily to speed things up a bit... | 12:12 |
| noonedeadpunk | pf | 12:24 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_keystone master: Do not install uWSGI as part of Keystone https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/924928 | 12:28 |
| noonedeadpunk | this ^ | 12:28 |
| noonedeadpunk | so role was finding already existing wheel for uwsgi that was built not with wsgi role, but just by keystone, and it was used down the line.... | 12:29 |
| noonedeadpunk | /o\ | 12:29 |
| gokhan | how can we open debug mode of services which running behind uwsgi ? | 12:34 |
| gokhan | ok I find it disable-logging=false in uwsgi | 12:41 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-role-uwsgi master: DNM: Install pcre on destination hosts https://review.opendev.org/c/openstack/ansible-role-uwsgi/+/924922 | 13:13 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Properly apply `always` tag to install_defaults https://review.opendev.org/c/openstack/openstack-ansible/+/924938 | 13:59 |
| noonedeadpunk | any guess wtf is that? https://opendev.org/openstack/openstack-ansible-os_keystone/src/branch/master/defaults/main.yml#L252-L260 | 14:18 |
| jrosser | this all looks legacy from when keystone role deployed nginx | 14:30 |
| jrosser | and making it serve on localhost for then serving onward by the webserver? | 14:31 |
| noonedeadpunk | but it's kind of used for apache: https://opendev.org/openstack/openstack-ansible-os_keystone/src/branch/master/templates/keystone-httpd.conf.j2#L145 | 14:31 |
| noonedeadpunk | for some reason | 14:31 |
| noonedeadpunk | and like... why just not to use 5000 | 14:32 |
| jrosser | becasue apache listens there? | 14:32 |
| noonedeadpunk | ah | 14:33 |
| jrosser | the rest of that apache config is handling federation / websso / whatever also on port 5000 | 14:33 |
| jrosser | and everything not that is passed to the keystone service | 14:33 |
| jrosser | likley could have two backends in haproxy and have some routing rules to decouple those | 14:34 |
| noonedeadpunk | and we have a logic in uwsgi role that does not allow not to listen on http | 14:40 |
| noonedeadpunk | https://opendev.org/openstack/ansible-role-uwsgi/src/branch/master/templates/uwsgi.ini.j2#L16-L21 | 14:40 |
| noonedeadpunk | so that's why we need to have both sock and http, while http does not make sense at all | 14:40 |
| noonedeadpunk | yeah, finally, uwsgi seems happy: https://zuul.opendev.org/t/openstack/build/8eb0d0a0ed46414bb3fb0309a49d1b0d/log/logs/host/keystone-wsgi-public.service.journal-14-33-16.log.txt#12-14 | 14:41 |
| jrosser | thats just allowing one or more bind addresses? i think i misunderstand | 14:46 |
| noonedeadpunk | yeah, but it doesn't allow none adresses and have a unix socket used instead | 14:47 |
| jrosser | aaah | 14:47 |
| noonedeadpunk | so for keystone the only thing that;s used with apache is this: https://zuul.opendev.org/t/openstack/build/8eb0d0a0ed46414bb3fb0309a49d1b0d/log/logs/etc/host/uwsgi/keystone-wsgi-public.ini.txt#30 | 14:48 |
| noonedeadpunk | but not L6 | 14:48 |
| noonedeadpunk | L6 is just there as we can't avoid having it I guess | 14:48 |
| jrosser | i wonder if you can set it to an empty list | 14:48 |
| noonedeadpunk | oh, you can | 14:49 |
| noonedeadpunk | I was thinking only about empty value, not about empty list | 14:49 |
| noonedeadpunk | thanks! | 14:49 |
| jrosser | is iterable, but no items | 14:49 |
| noonedeadpunk | yep, true | 14:49 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_keystone master: Remove excessive bindings for uWSGI https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/924945 | 14:56 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_keystone master: Remove excessive bindings for uWSGI https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/924945 | 15:00 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-tests stable/2024.1: Bump Ansible version for tests repo https://review.opendev.org/c/openstack/openstack-ansible-tests/+/921435 | 15:16 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Properly apply `always` tag to install_defaults https://review.opendev.org/c/openstack/openstack-ansible/+/924938 | 15:31 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-role-uwsgi master: Install pcre on destination hosts https://review.opendev.org/c/openstack/ansible-role-uwsgi/+/924922 | 15:34 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!