| *** chandankumar is now known as chkumar|ruck | 04:47 | |
| *** chkumar|ruck is now known as chandankumar | 04:51 | |
| *** ysandeep|out is now known as ysandeep | 05:49 | |
| *** ysandeep is now known as ysandeep|lunch | 07:29 | |
| jrosser | morning | 07:39 |
|---|---|---|
| *** arxcruz|out is now known as arxcruz | 08:11 | |
| johnnyd | Hey there, I'm currently facing a problem with the OS-ansible inventory. We have some inconsistencies in it. More precisely, the services cinder-volumes is declared in 3 lxc-containers (this is normal, we have 3 controllers) but also on 2 of the 3 controllers (so it's also installed in bare metal). If we delete the inventory and regenerate it, this disappear but obviously we generate something incorrect because IP and hostnames of every lxc | 08:22 |
| johnnyd | So, the question is, how to clean this properly except removing it by hand ? Thanks :) | 08:22 |
| damiandabrowski[m] | hey jrosser! | 08:28 |
| damiandabrowski[m] | johnnyd: I'm not 100% sure, but i think there's no good way to remove host from the inventory. IIRC the `scripts/inventory-manage.py` will remove it only from openstack_inventory.json, but removed host will still exist in openstack_hostnames_ips.yml | 08:29 |
| johnnyd | typically, this is normal: https://pastebin.com/raw/pxZhcQ4t | 08:32 |
| johnnyd | and this is not: https://pastebin.com/raw/ACcaqpsY | 08:33 |
| johnnyd | I don't really want to remove the os-controller-3 declaration because we need it for HAproxy, just remove the cinder_backends part and storage address | 08:35 |
| noonedeadpunk | mornings | 08:42 |
| jrosser | johnnyd: keep /etc/openstack_deploy under git then you can use a combination of inventory_manage.py and some grep/hand editing with the ability to roll back if it goes wrong | 08:48 |
| noonedeadpunk | just use ansible-vault for encrypting user_secrets.yml :) | 08:51 |
| johnnyd | yup, we already have our setup with git :) | 08:52 |
| johnnyd | that was our intuition, we'll edit that by hand | 08:53 |
| jrosser | cinder volume service is optionally on metal nodes for iscsi, as that doesnt work in containers | 08:53 |
| johnnyd | actually, we are using a Ceph backend | 08:54 |
| jrosser | noonedeadpunk: this is a wierd error https://paste.opendev.org/show/bIcy9AiNGEjvlqbCKh6K/ | 08:59 |
| jrosser | i was sanity checking the gluster stuff by trying to mount from the repo container into the glance container, just to make sure all the systemd_mount stuff is robust | 08:59 |
| jrosser | and it seems that systemd_mount calls systemd_service, which has conditionals on vars which only exist in systemd_mount..... | 09:00 |
| noonedeadpunk | sounds like systemd_mounts defined wierdly? | 09:14 |
| jrosser | yes - i'm looking at that now | 09:15 |
| jrosser | pretty hard to debug | 09:15 |
| noonedeadpunk | maybe ansible_hostname is undefined? | 09:15 |
| noonedeadpunk | like stale facts or smth? | 09:15 |
| noonedeadpunk | ah, but it fails in glance? | 09:19 |
| jrosser | with that little test playbook, yes | 09:20 |
| jrosser | just using glance as an empty container which is not a repo server | 09:20 |
| jrosser | to test that the glusterfs client works with just systemd_mount | 09:21 |
| jrosser | oh i see why | 09:26 |
| jrosser | i have https://paste.opendev.org/show/bkis6o2S20aA9hx7xOPz/ in /etc/ansible/roles/systemd_mount/tasks/systemd_install.yml | 09:27 |
| jrosser | so thats a block: with a when:, which calls a role that runs a handler | 09:27 |
| jrosser | and the handler gets the when: condition applied as part of import_role | 09:28 |
| noonedeadpunk | brrrrr | 09:36 |
| noonedeadpunk | Then at this level variable shouldn't be as `systemd_mount_types` at all, since it's getting defined only inside role? | 09:37 |
| jrosser | i think thats right - i can't think of a clean way to do this except set_fact something for the when: in a previous task | 09:37 |
| jrosser | as systemd_mount role vars won't be available inside systemd_service | 09:38 |
| jrosser | huh well set_fact: `_configure_glusterfs: "{{ 'glusterfs' in systemd_mount_types }}"` makes it work straight away | 09:41 |
| jrosser | seems that the root cause is lazy vs. immediate evaluation and there are lots of ansible issues related to that incl https://github.com/ansible/ansible/issues/10374 | 09:48 |
| noonedeadpunk | well.... yeah... | 09:50 |
| *** ysandeep|lunch is now known as ysandeep | 09:50 | |
| jrosser | its easy to forget that block: and include_<> with when: will apply the when condition everywhere | 09:50 |
| jrosser | usually it's OK but for including roles inside roles its tricky | 09:51 |
| opendevreview | Jonathan Rosser proposed openstack/ansible-role-systemd_mount master: Add support for glusterfs mounts https://review.opendev.org/c/openstack/ansible-role-systemd_mount/+/837750 | 10:28 |
| SiavashSardari | morning | 10:43 |
| SiavashSardari | would someone please help me with https://zuul.opendev.org/t/openstack/build/b1b5b45fa02c4bf180a68f2da11bb127/ | 10:43 |
| noonedeadpunk | folks, can we please review backports to stable branches? https://review.opendev.org/q/parentproject:openstack/openstack-ansible+branch:%255Estable/.*+status:open+,25 | 10:45 |
| noonedeadpunk | SiavashSardari: huh, that's interesting.... | 10:47 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Add ability to define bridge type for containers https://review.opendev.org/c/openstack/openstack-ansible/+/837742 | 10:47 |
| SiavashSardari | @noonedeadpunk I checked the parent commit. it was after the CI issue got merged. have I missed something? | 10:50 |
| noonedeadpunk | Well it's not related to commit itself, it's more likely some packaging issue or some braking change has landed... | 10:51 |
| SiavashSardari | well thanks | 10:55 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Cleanup distro vars https://review.opendev.org/c/openstack/openstack-ansible/+/837845 | 10:56 |
| noonedeadpunk | SiavashSardari: I will try to check what's wrong if it will fail again | 10:57 |
| jrosser | [Errno 2] No such file or directory: b'ceph-volume'" suggests some upstream change there | 11:04 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-lxc_hosts master: Ensure systemd-udev is present in centos containers https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/837865 | 11:07 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-repo_server master: Add facility to store repo contents on a remote mount https://review.opendev.org/c/openstack/openstack-ansible-repo_server/+/837587 | 11:10 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-repo_server master: Clean up legacy lsycnd, rsync and ssh key config https://review.opendev.org/c/openstack/openstack-ansible-repo_server/+/837859 | 11:11 |
| opendevreview | Merged openstack/ansible-role-python_venv_build stable/stein: Remove tripleo-ci-centos-7-standalone-os-tempest from gate layout https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/838170 | 11:19 |
| *** dviroel|out is now known as dviroel | 11:31 | |
| *** ysandeep is now known as ysandeep|afk | 11:41 | |
| SiavashSardari | there is an issue regarding the default_bind_mount_logs variable in osa, the default value is true, which cause the lxc config file to be like https://paste.opendev.org/show/bvUwH0C2pxAYg2M7ejt4/. | 12:04 |
| SiavashSardari | the issue is that there is two mount entry for /var/log and mostly it doesn't create an issue till first restart. after a restart there would be a race condition between which dir gets mounted first. | 12:04 |
| opendevreview | Siavash Sardari proposed openstack/openstack-ansible master: Disable the default_bind_mount_logs https://review.opendev.org/c/openstack/openstack-ansible/+/838501 | 12:05 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-repo_server master: Remove all code for lsync, rsync and ssh https://review.opendev.org/c/openstack/openstack-ansible-repo_server/+/837588 | 12:06 |
| SiavashSardari | I believe removing the extra mount entry requires restarting the container if the /var/log entry has won the race. | 12:07 |
| jrosser | SiavashSardari: i think i remember something like this before | 12:08 |
| jrosser | there were some patches for a similar problem | 12:08 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-plugins master: Add role for creating simple glusterfs servers https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/837582 | 12:15 |
| SiavashSardari | @jrosser I will search the lauchpad. I'd appreciate if you could help me find the patches you mentioned | 12:18 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Use glusterfs to synchronise repo server contents https://review.opendev.org/c/openstack/openstack-ansible/+/837589 | 12:19 |
| opendevreview | Merged openstack/openstack-ansible-rabbitmq_server stable/xena: Verify if hosts file already managed with OSA https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/836166 | 12:21 |
| jrosser | SiavashSardari: is this related to your issue? https://review.opendev.org/c/openstack/openstack-ansible/+/752059 | 12:22 |
| SiavashSardari | this is related | 12:39 |
| opendevreview | Merged openstack/openstack-ansible-os_cinder stable/xena: Add the ability to disable send_actions option in cinder-volume-usage-audit service. https://review.opendev.org/c/openstack/openstack-ansible-os_cinder/+/838210 | 12:40 |
| SiavashSardari | @jrosser but the I believe the patch you mentioned does not solve the issue. there is always one of the mount entries work | 12:42 |
| SiavashSardari | let me restart one container and send you the log. | 12:42 |
| opendevreview | Merged openstack/openstack-ansible-os_cinder stable/wallaby: Add the ability to disable send_actions option in cinder-volume-usage-audit service. https://review.opendev.org/c/openstack/openstack-ansible-os_cinder/+/838211 | 12:43 |
| SiavashSardari | https://paste.opendev.org/show/boifQ4Qg3WhVRtztTuwK/ | 12:46 |
| noonedeadpunk | well, that worked for me at least, as I see mounts ordered properly now as well as journals from containers be preset on metal hosts | 12:51 |
| noonedeadpunk | but more interesting would be to see your lxc config | 12:51 |
| SiavashSardari | @noonedeadpunk please take a look at this https://paste.opendev.org/show/bVbQK7AMkWnvNPao4zDD/ | 12:53 |
| *** ysandeep|afk is now known as ysandeep | 12:53 | |
| SiavashSardari | this is my lxc config https://paste.opendev.org/show/b6YjEnL0Ue6MMkkDulsu/ | 12:54 |
| noonedeadpunk | are you sure you have patch mentioned? | 12:56 |
| SiavashSardari | yep i do have that | 12:59 |
| *** spotz__ is now known as spotz | 13:12 | |
| opendevreview | Merged openstack/openstack-ansible stable/ussuri: Mark OSA repository as safe in git.config in CI https://review.opendev.org/c/openstack/openstack-ansible/+/838021 | 13:23 |
| noonedeadpunk | Thinking about that, I'm not sure that that patch would re-order things correctly. So it's likely doing right things only on freshly spawned containers | 13:30 |
| noonedeadpunk | As once line is present, task will do nothing I guess | 13:30 |
| *** spotz is now known as spotz__ | 13:54 | |
| *** spotz__ is now known as spotz | 13:54 | |
| SiavashSardari | @noonedeadpuck exactly. actually now that I'm looking at things more closely, having /var/log mounted on host is usefull in case of some services like rabbit. I'm not sure but I think the root cause is that /var/log is the parent of /var/log/journal and mounting one would cause problem for mounting the other one | 14:22 |
| SiavashSardari | let me try something on my dev. I will keep you posted | 14:23 |
| *** dviroel is now known as dviroel|afk | 14:24 | |
| noonedeadpunk | SiavashSardari: it's just ordering issue in the config | 14:30 |
| noonedeadpunk | if you would deploy from scratch you would have /var/log mounted and /var/log/journal inisde it which prefectly makes sense | 14:31 |
| SiavashSardari | yep you're right. i tried to regenerate the issue with my hypothesis but no luck yet | 14:35 |
| noonedeadpunk | I should have wroted a release note for that commit to handle upgrade path... or well, abscence of it | 14:36 |
| noonedeadpunk | but if you do OS upgrade, you will have container re-generated and issue fixed :p | 14:36 |
| SiavashSardari | do we support ubuntu 22 yet? :D | 14:37 |
| noonedeadpunk | nope... | 14:37 |
| noonedeadpunk | it hasn't even released yet :) | 14:37 |
| noonedeadpunk | does it? | 14:38 |
| noonedeadpunk | and don't re-ask in 2 days ! :p | 14:38 |
| SiavashSardari | no but maybe we can start with nightly builds :)) | 14:38 |
| *** dviroel|afk is now known as dviroel|mtg | 14:39 | |
| SiavashSardari | Eureka Eureka finally I could regenerate the issue | 14:42 |
| SiavashSardari | I believe the order of the mount point is wrong | 14:42 |
| SiavashSardari | the /var/log should be before /var/log/journal. | 14:43 |
| SiavashSardari | at least my test shows that. | 14:43 |
| *** ysandeep- is now known as ysandeep|out | 14:45 | |
| SiavashSardari | mounting parent dir before the child scenario https://paste.opendev.org/show/bbeVnxOeEisyFzizsVVr/ | 14:47 |
| SiavashSardari | mounting the child dir before parent scenario https://paste.opendev.org/show/bbeVnxOeEisyFzizsVVr/ | 14:49 |
| noonedeadpunk | `the /var/log should be before /var/log/journal.` yes and that excatly what happens for me on all envs... | 14:49 |
| noonedeadpunk | and that actually what changes patch we're talking about | 14:50 |
| noonedeadpunk | BUT, if container already exist with wrong ordering, it won't fix it | 14:50 |
| *** dviroel|mtg is now known as dviroel | 14:54 | |
| SiavashSardari | oh I get it now it has the wrong order from the beginning. | 14:54 |
| SiavashSardari | @noonedeadpunk ci job failed for https://review.opendev.org/c/openstack/openstack-ansible/+/837742 again. | 14:57 |
| spatel | Ubuntu 22 - Release date is April 21, 2022 | 14:59 |
| spatel | noonedeadpunk i am interested in Ubuntu 22 so hope we can get it setup on CI-CD asap after release :) | 15:00 |
| noonedeadpunk | #startmeeting openstack_ansible_meeting | 15:00 |
| opendevmeet | Meeting started Tue Apr 19 15:00:38 2022 UTC and is due to finish in 60 minutes. The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:00 |
| opendevmeet | The meeting name has been set to 'openstack_ansible_meeting' | 15:00 |
| noonedeadpunk | #topic rollcall | 15:00 |
| spatel | \o/ | 15:00 |
| damiandabrowski[m] | hey! | 15:01 |
| noonedeadpunk | o/ | 15:01 |
| mgariepy | hey ! o/ | 15:01 |
| jrosser | hello | 15:01 |
| noonedeadpunk | #topic office hours | 15:02 |
| damiandabrowski[m] | and thanks for nominating me! (i just saw the message on ML) | 15:02 |
| noonedeadpunk | so. seems we have issue with ceph-ansible version... Will need to take a look at the actuall thing first time tomorrow in the morning.... | 15:03 |
| noonedeadpunk | but that sounds really weird, I havn't heard a thing that they're replacing cinder-volume with anything else... | 15:04 |
| noonedeadpunk | *ceph-volume | 15:04 |
| damiandabrowski[m] | FYI, i'm not sure if it's related, but I've hit this error when deploying AIO+ceph today https://paste.openstack.org/show/bRMsII6P2s1YyjugFuWk/ | 15:04 |
| damiandabrowski[m] | but haven't looked much into this yet | 15:05 |
| noonedeadpunk | that's interesting actually. | 15:05 |
| noonedeadpunk | I can recall we were having apt pinning for ceph somehow.... | 15:05 |
| jrosser | we should have apt pins to make sure that the ceph version comes from where we expect | 15:05 |
| jrosser | which iirc on ubuntu should always be download.ceph.com | 15:06 |
| noonedeadpunk | but I think we did that only for ceph-client role? | 15:06 |
| jrosser | could be, yes | 15:06 |
| *** dviroel is now known as dviroel|lunch | 15:06 | |
| jrosser | https://github.com/openstack/openstack-ansible-ceph_client/blob/master/defaults/main.yml#L43 | 15:08 |
| noonedeadpunk | huh and we don't really include role anywhere | 15:08 |
| noonedeadpunk | unless we rely on that? https://opendev.org/openstack/openstack-ansible-ceph_client/src/branch/master/meta/main.yml#L40-L45 | 15:09 |
| jrosser | yes it comes from meta | 15:09 |
| noonedeadpunk | but, um.... | 15:09 |
| noonedeadpunk | but anyway failure is during ceph-ansible runtime | 15:09 |
| noonedeadpunk | (in CI at least) | 15:09 |
| noonedeadpunk | Also I spent time previous week for unplanned activity of fixing run of roles in check mode | 15:10 |
| noonedeadpunk | was interesting how much doable is that... | 15:10 |
| noonedeadpunk | hosts and infra seems quite fine, openstack services are tricky | 15:11 |
| noonedeadpunk | I also thought it would be easy to have CI job but then realized that to run in check mode it has plenty of dependencies in terms of missing services and certificates. | 15:11 |
| noonedeadpunk | Still doable but appeared a bit harder then expected | 15:12 |
| noonedeadpunk | Will likely continue that after more important things are done:) | 15:12 |
| noonedeadpunk | BTW we kind of need to make last release of V and move it to EM | 15:12 |
| noonedeadpunk | jrosser: do we need wip here? https://review.opendev.org/c/openstack/openstack-ansible-tests/+/836335 | 15:14 |
| jrosser | no - i think it was just to try to get things merged in order down the stable branches | 15:15 |
| jrosser | this has all been a quite big mess with not really any cherry-picks that work | 15:15 |
| jrosser | so i had to patch each branch pretty much individually | 15:15 |
| jrosser | and really the patch we need is https://review.opendev.org/c/openstack/openstack-ansible-tests/+/837368 | 15:16 |
| jrosser | there is a bunch of broken things due to that ^ | 15:16 |
| jrosser | the gluster patches are nearly ready for proper review | 15:18 |
| jrosser | i have to make a hack^W fix for rocky | 15:19 |
| noonedeadpunk | But hopefully NeilHanlon would ping for some packaging changes one day? | 15:20 |
| jrosser | hopefully yes | 15:20 |
| noonedeadpunk | ok great. And then I can pick CentOS 9 deployment on top of gluster :) | 15:21 |
| NeilHanlon | Yes, i am working on fixing up those release packages for, e.g., gluster | 15:22 |
| jrosser | i just need to decide if we lineinfile/regex the repo file or vendor a copy of it and just copy: it into place | 15:22 |
| noonedeadpunk | great) | 15:22 |
| jrosser | do you think we should be more specific about pinning the ceph repo on ceph hosts? | 15:24 |
| jrosser | if we pin the client to ceph.com but not the server, we still have potential conflicts between ubuntu / uca / ceph.com repos | 15:25 |
| noonedeadpunk | jrosser: Um, can you point to that Rocky hack as I can't find it quickly :) | 15:26 |
| jrosser | no i still try to write it :) | 15:26 |
| noonedeadpunk | ah, ok then :) | 15:26 |
| jrosser | but i believe we have to adjust mirrorlist=http://mirrorlist.centos.org/?release=$stream&arch=$basearch&repo=ResilientStorage&infra=$infra | 15:27 |
| noonedeadpunk | regarding ceph pinning - that kind of make sense... Despite that should be part of ceph-ansible to be fair... | 15:27 |
| jrosser | so that always release=8-stream | 15:27 |
| damiandabrowski[m] | i need to spend some time trying to figure out why i get this downgrade error, then I'll have a better view on how do we want to pin ceph repo. Planning to look into this during the evening | 15:29 |
| noonedeadpunk | ceph-ansible already has requirements.yml in place, so technically we can push them a change to handle pinning and add another dependency on our roles :) | 15:29 |
| jrosser | NeilHanlon: do you have a bug link i can put in my patch for glusterfs? | 15:29 |
| noonedeadpunk | don't think they're gonna like it though... | 15:31 |
| noonedeadpunk | I kind of more and more thinking that integration of ceph-ansible should be more for CI purposes only. | 15:32 |
| noonedeadpunk | while it's good to have playbooks for that and some sort of integration, I'm not really sure if anybody should do that in production :) | 15:33 |
| noonedeadpunk | (at least as is) | 15:33 |
| damiandabrowski[m] | I'm going to use it in production in a few weeks, i can share my thoughts then :D | 15:34 |
| noonedeadpunk | the questions come when you're trying to upgrade stuff :) | 15:34 |
| noonedeadpunk | as we pin specific ceph-ansible version that is tighten to ceph releases quite hardly | 15:35 |
| SiavashSardari | i have it in production. (he types with shaking hands :D ) | 15:35 |
| noonedeadpunk | as then when you upgrade osa you should either override ceph-ansible or do upgrade ceph at same time kind of. | 15:36 |
| mgariepy | i had it with osa for exactly 1 deploy then i move the ceph-ansible on the side right after | 15:36 |
| SiavashSardari | we are thinking about replacing it with ceph-adm, not starting the procedure though | 15:37 |
| jrosser | ceph-ansible is fine if you use it on its own | 15:37 |
| jrosser | i wouldnt copy the approach from OSA CI into a production deployment | 15:37 |
| noonedeadpunk | nah, we're not talking about ceph-adm, but more about at what level we should provide ceph deployment | 15:38 |
| noonedeadpunk | and if we explicit enough about how to use that properly | 15:38 |
| SiavashSardari | tbh I like the idea of having one repo for both but at the end of the day we are using them kinda separately and will override the role-requirement if needed | 15:41 |
| damiandabrowski[m] | i just think that ceph integration is a huge advantage of openstack-ansible project(especially for a newcomer who is is comparing different deployment tools and trying to choose one) | 15:41 |
| damiandabrowski[m] | so maybe we can list our main issues with it and think what we can do about them? | 15:42 |
| jrosser | ceph-ansible sometimes has very specific requirements for the ansible version (though this is less important with modern ansible) | 15:42 |
| jrosser | and versions of ceph-ansible only support very specific releases of ceph | 15:42 |
| mgariepy | specific major rel | 15:43 |
| jrosser | so you ended up in very difficult places at upgrade time when the openstack, ansible and ceph (and maybe also OS) versions all had to change simultaneously | 15:43 |
| jrosser | and thats a total nightmare for upgrades | 15:43 |
| damiandabrowski[m] | but You don't have to run `ceph-install.yml` in the middle of openstack upgrade, right? :D | 15:44 |
| noonedeadpunk | no, if you're not running /run-upgrade.sh :p | 15:45 |
| noonedeadpunk | (which what newcommers will do) | 15:45 |
| jrosser | and we test exactly "zero" of that approach :) | 15:45 |
| noonedeadpunk | and you will get newer clients anyway unless explicitly set in user_variables | 15:45 |
| noonedeadpunk | I'm not talking about dropping support, but likely more being more explicit in documentation about possible caveats by doing that | 15:46 |
| damiandabrowski[m] | so maybe what we need to do is to show a warning message when running run-upgrade.sh with ceph enabled? | 15:46 |
| damiandabrowski[m] | ah yeah, i was just going to say that | 15:46 |
| damiandabrowski[m] | that another option is put some information in docs | 15:47 |
| jrosser | well - back to where we started there is really only benefit in pinning to ceph.com | 15:47 |
| jrosser | the UCA packages didnt used to have debug symbols and that was a problem for us when things were broken | 15:47 |
| damiandabrowski[m] | saying that OSA upgrades may be tricky when it manages ceph as well | 15:47 |
| noonedeadpunk | I can recall some early on bionic distro provided ceph version was treated as newer one comparing to one that was coming from ceph.com | 15:48 |
| jrosser | yes, i remember having to fix that | 15:48 |
| noonedeadpunk | I guess that's when we added pinning :) | 15:49 |
| jrosser | i think it was the local patch that ubuntu applied made the version number sematically greater | 15:49 |
| noonedeadpunk | yup | 15:49 |
| noonedeadpunk | while in fact it wasn't and dependencies were broken | 15:49 |
| noonedeadpunk | anyway - let's see what the current issues are. | 15:50 |
| jrosser | damiandabrowski[m]: you might want to look at this as it addresses similar errors to those you were seeing https://github.com/openstack/openstack-ansible-os_cinder/blob/9f2bf29db8ef921cfad7857dcb7652436d0d887b/tasks/main.yml#L183-L209 | 15:51 |
| noonedeadpunk | ofc we can add pinning in playbook alone | 15:51 |
| damiandabrowski[m] | thanks, I'll have a look | 15:51 |
| jrosser | i had to split the list of packages into those to install before, and after the ceph_client role is run | 15:52 |
| damiandabrowski[m] | btw. we already managed to merge few of my tempest patches, but many of them still remain unreviewed. | 15:53 |
| damiandabrowski[m] | it would be awesome to close this topic soon ;) https://review.opendev.org/q/topic:tempest-damian-2021-12 | 15:53 |
| mgariepy | i'll take a look | 15:55 |
| noonedeadpunk | damiandabrowski[m]: aren't most of them need rebase? | 15:56 |
| noonedeadpunk | see tons of `Indirect ancestor` in gerrit | 15:57 |
| noonedeadpunk | likely they won't merge cleanly but not sure | 15:57 |
| damiandabrowski[m] | that's right, but the plan was to get initial reviews -> apply Your suggestions -> rebase -> merge | 15:58 |
| damiandabrowski[m] | by doing it like this, we'll avoid running too much unnecessary zuul jobs | 15:58 |
| damiandabrowski[m] | but I can rebase them now if You want :D | 15:58 |
| damiandabrowski[m] | but main drawback of this plan is that You'll need to leave Your reviews twice on these patches(before and after rebase) | 16:00 |
| noonedeadpunk | well, rebase in gerrit doesn't remove Code-Review label | 16:01 |
| noonedeadpunk | it would remove jsut Workflow | 16:01 |
| noonedeadpunk | but ok, indeed:) | 16:01 |
| damiandabrowski[m] | ahhh okay | 16:01 |
| noonedeadpunk | #endmeeting | 16:01 |
| opendevmeet | Meeting ended Tue Apr 19 16:01:40 2022 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:01 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/openstack_ansible_meeting/2022/openstack_ansible_meeting.2022-04-19-15.00.html | 16:01 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/openstack_ansible_meeting/2022/openstack_ansible_meeting.2022-04-19-15.00.txt | 16:01 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/openstack_ansible_meeting/2022/openstack_ansible_meeting.2022-04-19-15.00.log.html | 16:01 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-plugins master: Add role for creating simple glusterfs servers https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/837582 | 16:04 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-plugins master: Add role for creating simple glusterfs servers https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/837582 | 16:05 |
| *** dviroel|lunch is now known as dviroel | 16:14 | |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-plugins master: Add role for creating simple glusterfs servers https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/837582 | 16:56 |
| jrosser | hrrm the ceph job is breaking things in the gate | 17:15 |
| mgariepy | hmm let me have a look | 17:18 |
| mgariepy | i'll biuld an aio to check it out. | 17:18 |
| jrosser | its failing on msg": "[Errno 2] No such file or directory: b'ceph-volume'" | 17:20 |
| mgariepy | yep i saw that part but it will be easier to see what is going on in a vm | 17:20 |
| jrosser | if we were using the ubuntu package that should be in ceph-osd https://packages.ubuntu.com/focal/amd64/ceph-osd/filelist | 17:20 |
| jrosser | and in that CI jobs we get the package from ceph-osd/focal-updates,now 17.1.0-0ubuntu3~cloud0 amd64 [installed,automatic] | 17:22 |
| jrosser | also i see upgrade jobs failing on git clone for ansible-sshd fatal: reference is not a tree: 59a3cb190f7913e32b044dc2733cfe7e46c4c99a | 17:25 |
| damiandabrowski[m] | FYI I have found out why I'm hitting librados2/librbd1 downgrade issue on my AIO | 17:33 |
| damiandabrowski[m] | lxc_hosts installs lxc-templates package, which has some dependency to librados2/librbd1, so these 2 packages are firstly installed from UCA repo | 17:34 |
| jrosser | oh wow | 17:34 |
| damiandabrowski[m] | and that's why they need to be downgraded afterwards | 17:34 |
| damiandabrowski[m] | now I'm trying to figure out the exact dependency tree and how to solve it properly | 17:34 |
| jrosser | this definatly means we should pin to ceph.com | 17:34 |
| jrosser | really early, like in openstack_hostts | 17:34 |
| damiandabrowski[m] | +1 | 17:35 |
| jrosser | we already run the apt_package_pinning role there https://github.com/openstack/openstack-ansible-openstack_hosts/blob/master/meta/main.yml#L38-L41 | 17:36 |
| damiandabrowski[m] | great, so it should be fairly easy to implement | 17:41 |
| jrosser | yeah, hopefully just needs a var adding in the right place into inventory/group_vars/.... | 17:42 |
| damiandabrowski[m] | if I understand it correctly, the dependency chain looks like this: lxc-templates->cloud-image-utils->qemu-utils->qemu-block-extra->librbd1/librados2 | 17:43 |
| damiandabrowski[m] | I need to leave for ~3h, can continue later if needed | 17:43 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-plugins master: Add role for creating simple glusterfs servers https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/837582 | 17:46 |
| mgariepy | ceph-volume is splitted in uca. https://paste.openstack.org/show/bLMvIdro1Wp0eMiWh0rd/ | 19:38 |
| mgariepy | lxc host role does pull the librados2 and librbd1 pkg which then breaks the install of ceph via ceph community :/ | 20:30 |
| mgariepy | not sure what is the best way to fix this, appliing the pin before without the repo will have a net 0 effect. | 20:31 |
| mgariepy | https://zuul.opendev.org/t/openstack/build/7ee28ca3d39e472fab4c55798b4dec8b/log/logs/etc/host/apt/preferences.d/ceph_community_pin.pref.txt | 20:31 |
| mgariepy | so i guess we need the ceph repo before running lxc-container-host playbook. | 20:33 |
| damiandabrowski[m] | ouh, You're right | 20:37 |
| mgariepy | we migt need to run : https://github.com/openstack/openstack-ansible-ceph_client/blob/master/tasks/ceph_preinstall_apt.yml and https://github.com/openstack/openstack-ansible/blob/master/playbooks/common-tasks/ceph-server.yml | 20:37 |
| mgariepy | to apply the repo + the pin | 20:37 |
| damiandabrowski[m] | and I assume this repo is added by ceph-ansible(/etc/ansible/roles/ceph-ansible/roles/ceph-common/tasks/installs/install_debian_packages.yml) | 20:37 |
| mgariepy | the repo can be added by both roles | 20:38 |
| mgariepy | either ceph_client or ceph-ansible | 20:38 |
| mgariepy | all repos and pins should be applied to a server prior installing any software :D | 20:40 |
| mgariepy | haha | 20:40 |
| *** dviroel is now known as dviroel|out | 20:41 | |
| damiandabrowski[m] | hmm, why do You think we also need to run https://github.com/openstack/openstack-ansible/blob/master/playbooks/common-tasks/ceph-server.yml early in setup-hosts? | 20:45 |
| damiandabrowski[m] | I may be missing something but apt pinning can be done directly via ceph_client role | 20:46 |
| damiandabrowski[m] | so I'm just looking for another reason to run ceph-server.yml | 20:46 |
| damiandabrowski[m] | i only wonder where exactly do we want to run ceph_client role(probably we need to implement 'preinstall' tag as well to avoid installing ceph packages and only add repo/pins?). | 21:48 |
| damiandabrowski[m] | Only on lxc_hosts as 'lxc-templates' package triggers ceph-related packages(librbd1,librados2) to be installed which breaks things or maybe on all baremetal nodes? | 21:49 |
| damiandabrowski[m] | the other question is when do we want to run it? only when OSA manages ceph, so `groups['ceph_all'] | length > 0`? | 21:50 |
| damiandabrowski[m] | ehh, it's way more complicated than i thought :D | 21:50 |
| * damiandabrowski[m] heading to bed, good night | 21:50 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!