| *** ysandeep|out is now known as ysandeep|rover | 03:24 | |
| *** ysandeep|rover is now known as ysandeep|afk | 04:24 | |
| *** ysandeep|afk is now known as ysandeep|rover | 04:45 | |
| noonedeadpunk | jrosser: let's just hardcode https://opendev.org/openstack/openstack-ansible-tests/src/branch/master/test-vars.yml#L21 for now (not relying on env var) and will revert once branch | 06:36 |
|---|---|---|
| jrosser | ok | 06:37 |
| noonedeadpunk | ah, but I see now how it's different | 06:37 |
| jrosser | we don't have an easy variable that we can use to set the branch of "all services" in the os_<blah> roles | 06:38 |
| jrosser | like a openstack_git_install_branch here would be just what we need https://github.com/openstack/openstack-ansible-os_keystone/blob/master/defaults/main.yml#L41 | 06:39 |
| jrosser | but thats for another time, this is kind of a big mess | 06:39 |
| jrosser | and unfortunatley i have made a mistake with the gluster/rockylinux stuff and we're broken | 06:39 |
| noonedeadpunk | but we set them all | 06:40 |
| jrosser | not in the functional tests do we? | 06:40 |
| jrosser | oh sorry you are right, we do | 06:40 |
| noonedeadpunk | we do, yeah. But it's based on TESTING_BRANCH at the first place https://opendev.org/openstack/openstack-ansible-tests/src/branch/master/test-vars.yml#L147 | 06:40 |
| noonedeadpunk | now I'm thinking, that we should check if TESTING_BRANCH is already set | 06:41 |
| jrosser | we should make SERVICES_BRANCH or something for that | 06:41 |
| jrosser | anyway, this is pretty important to get fixed whichever way https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/842999 | 06:42 |
| jrosser | though of course theres no actual test coverage there with that patch | 06:42 |
| jrosser | which is how this got broken in the first place :( | 06:42 |
| noonedeadpunk | give me several mins, will make some patch to show what I mean :) | 06:43 |
| jrosser | sure | 06:43 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-tests master: Provide opportunity to override TESTING_BRANCH https://review.opendev.org/c/openstack/openstack-ansible-tests/+/843108 | 06:58 |
| noonedeadpunk | jrosser: not sure if it will work though :) ^ | 06:58 |
| noonedeadpunk | at least it was the idea | 07:00 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Allow lxc container backing store to be defined as part of the scenario https://review.opendev.org/c/openstack/openstack-ansible/+/843011 | 07:06 |
| jrosser | noonedeadpunk: i think it will break on u-c still being master | 07:09 |
| jrosser | from the cached zuul repo | 07:09 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Add zuul jobs and template for testing different LXC backing stores https://review.opendev.org/c/openstack/openstack-ansible/+/843027 | 07:13 |
| noonedeadpunk | ah yes, as we're using local zuul one... | 07:15 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-tests master: Provide opportunity to override TESTING_BRANCH https://review.opendev.org/c/openstack/openstack-ansible-tests/+/843108 | 07:18 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-tests master: Provide opportunity to override TESTING_BRANCH https://review.opendev.org/c/openstack/openstack-ansible-tests/+/843108 | 07:32 |
| *** ysandeep|rover is now known as ysandeep|rover|lunch | 08:02 | |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-tests master: Provide opportunity to override TESTING_BRANCH https://review.opendev.org/c/openstack/openstack-ansible-tests/+/843108 | 08:17 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Allow lxc container backing store to be defined as part of the scenario https://review.opendev.org/c/openstack/openstack-ansible/+/843011 | 08:18 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Add zuul jobs and template for testing different LXC backing stores https://review.opendev.org/c/openstack/openstack-ansible/+/843027 | 08:19 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-tests master: Provide opportunity to override TESTING_BRANCH https://review.opendev.org/c/openstack/openstack-ansible-tests/+/843108 | 08:19 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-tests master: Provide opportunity to override TESTING_BRANCH https://review.opendev.org/c/openstack/openstack-ansible-tests/+/843108 | 08:25 |
| noonedeadpunk | ok, I dunno... environment passed to the run_test.sh obviosly not picked up on runtime... | 08:25 |
| noonedeadpunk | or test-ansible-env-prep.sh even not executed during test... | 08:25 |
| noonedeadpunk | So centos stil fails... | 08:26 |
| jrosser | i was confused by that | 08:33 |
| jrosser | i had a patch yesterday which added set -x to test-ansible-env-prep.sh becasue it was really confusing what was happening | 08:34 |
| *** ysandeep|rover|lunch is now known as ysandeep|rover | 08:38 | |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-tests master: Provide opportunity to override TESTING_BRANCH https://review.opendev.org/c/openstack/openstack-ansible-tests/+/843108 | 08:48 |
| * noonedeadpunk trying to make "projects update" presentation | 08:51 | |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-ops master: Update beat templates from 7.11 reference https://review.opendev.org/c/openstack/openstack-ansible-ops/+/778849 | 10:20 |
| *** dviroel|out is now known as dviroel | 11:21 | |
| noonedeadpunk | so Rocky now fails with gluster installation: Failed to download metadata for repo 'centos-gluster8': Cannot prepare internal mirrorlist: No URLs in mirrorlist | 11:35 |
| mbahn | Hi, we're planning to add multiple federated service providers. Both openidc: one for keycloak (don't mistake that with keystone) authentication and one for another external openidc provider) but I just saw that only a single trusted IdP in keystone_sp.trusted_idp_list is supported. Why is that so? Is it because of the keystone httpd config? | 11:37 |
| noonedeadpunk | I think only jrosser / andrewbonney can answer that :) | 11:43 |
| jrosser | noonedeadpunk: i have a patch for glusterfs | 12:26 |
| noonedeadpunk | oh? | 12:26 |
| jrosser | https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/842999 | 12:26 |
| noonedeadpunk | ah, lol, indeed. | 12:26 |
| jrosser | but of course that fails because of test repo stuff | 12:28 |
| noonedeadpunk | yeah-yeah... | 12:30 |
| noonedeadpunk | I really wonder why testing_branch is not being seen inside run_tests.sh | 12:38 |
| noonedeadpunk | I kind of have a guss why.... | 12:39 |
| noonedeadpunk | And your approach was right with adding new variable | 12:41 |
| noonedeadpunk | Anyway can be fixed with follow-up | 12:41 |
| jrosser | mbahn: it is true that trusted_idp_list in the keystone ansible role, so you can potentially define several. However, there are several places in the current code where the list is assumed to have only one entry. You can see those here https://codesearch.opendev.org/?q=trusted_idp_list.0.&i=nope&literal=nope&files=&excludeFiles=&repos= | 12:42 |
| jrosser | mbahn: if you want to use more two OIDC IdP then it would probably be only small work to iterate over those lists where there is currently use of the '.0' element | 12:43 |
| jrosser | that would be a really nice thing to contribute to OSA if you were able to | 12:44 |
| jrosser | my team have a keystone<>keycloak OIDC integration already so can maybe give you some pointers if needed | 12:44 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-tests master: Add SERVICES_BRANCH variable for better control https://review.opendev.org/c/openstack/openstack-ansible-tests/+/843140 | 12:51 |
| opendevreview | Merged openstack/openstack-ansible-tests master: Provide opportunity to override TESTING_BRANCH https://review.opendev.org/c/openstack/openstack-ansible-tests/+/843108 | 13:12 |
| mbahn | @jrosser: okay thanks for the info :) I'll have a look at it and see if I can contribute. Just btw, I already have one OIDC integration running and working (EGI Check-In if you're interested) and tried to add the keycloak integration for our own institute | 13:18 |
| mbahn | Just to get the info right: if I add the second IdP in the user_variables.yml it would be translated as '.1' in Ansible? | 13:19 |
| jrosser | mbahn: ok cool, we had really quite good success building an openstack-ansible all-in-one test setup for working on this stuff | 13:19 |
| jrosser | mbahn: well, yes it would be .1, but the better thing to do would be to replace the explicit indexing into the list with a 'for' loop over all the list elements, however many there might be | 13:20 |
| jrosser | this whole section here would be a for loop https://github.com/openstack/openstack-ansible-os_keystone/blob/master/templates/keystone-httpd.conf.j2#L48-L101 | 13:22 |
| mbahn | cool thanks. Let's see if I can manage to do something in the next days | 13:22 |
| jrosser | i don't know if this would also need seperate data for each idp https://github.com/openstack/openstack-ansible-os_keystone/blob/master/tasks/keystone_federation_sp_idp_setup.yml | 13:24 |
| jrosser | oh, well thats already iterating over the trusted_idp_list here https://github.com/openstack/openstack-ansible-os_keystone/blob/master/tasks/main_keystone_federation_sp_idp_setup.yml#L2 | 13:25 |
| mgariepy | mariadb 10.6.8 is avalaible now :D | 13:27 |
| * jrosser wonders what new bugs there will be | 13:28 | |
| noonedeadpunk | and 10.8.3 :D | 13:28 |
| jrosser | i would rather they make centos-9 packages | 13:28 |
| noonedeadpunk | it's not LTS though | 13:28 |
| mgariepy | well it does fix a couple cve.. | 13:28 |
| noonedeadpunk | doh. and no ubuntu 22.04? Rly? | 13:29 |
| mbahn | well, good thing that I already worked with the keystone-httpd.conf.j2 (see last commit) :D understanding what ansible does with j2 files is easier now | 13:29 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Allow lxc container backing store to be defined as part of the scenario https://review.opendev.org/c/openstack/openstack-ansible/+/843011 | 13:37 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Add zuul jobs and template for testing different LXC backing stores https://review.opendev.org/c/openstack/openstack-ansible/+/843027 | 13:38 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Add zuul jobs and template for testing different LXC backing stores https://review.opendev.org/c/openstack/openstack-ansible/+/843027 | 13:38 |
| jrosser | i wonder why SERVICES_BRANCH ends up as master still | 13:47 |
| opendevreview | Merged openstack/openstack-ansible stable/xena: Allow fast SSH cipher for upgrade jobs https://review.opendev.org/c/openstack/openstack-ansible/+/829258 | 13:58 |
| noonedeadpunk | hm | 14:00 |
| noonedeadpunk | run_tests.sh doesn't pass it? | 14:01 |
| jrosser | maybe we should print the values of these as they are passed in as well for debugging | 14:15 |
| jrosser | before assigning a default | 14:16 |
| jrosser | hopefully this is going to fix the rocky/glusterfs trouble https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/842999 | 14:25 |
| jrosser | rocky jobs can be seen passing with that patch here https://review.opendev.org/c/openstack/openstack-ansible/+/843011 | 14:26 |
| noonedeadpunk | nice | 14:38 |
| jrosser | oh did you see i think now lxc_hosts/container_create don't need openstack-ansible-tests any more | 14:42 |
| noonedeadpunk | I saw you pushed several changed to move that to integrated and loved that :) | 14:53 |
| opendevreview | Merged openstack/openstack-ansible-plugins master: Fix installation of glusterfs repo on rocky linux https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/842999 | 14:57 |
| noonedeadpunk | #startmeeting openstack_ansible_meeting | 15:00 |
| opendevmeet | Meeting started Tue May 24 15:00:24 2022 UTC and is due to finish in 60 minutes. The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:00 |
| opendevmeet | The meeting name has been set to 'openstack_ansible_meeting' | 15:00 |
| noonedeadpunk | #topic roll call | 15:00 |
| noonedeadpunk | o/ | 15:00 |
| damiandabrowski[m] | hey! | 15:00 |
| mgariepy | half there o/ | 15:01 |
| jrosser | o/ hello | 15:02 |
| noonedeadpunk | #topic office hours | 15:05 |
| noonedeadpunk | I think we're super close to merging repo, centos and ubuntu | 15:07 |
| noonedeadpunk | I suggest creating milestone release after that | 15:08 |
| noonedeadpunk | and would be awesome to make it this week actually... maybe next... | 15:08 |
| jrosser | repo server gluster stuff just has a few cleanup patches to merge which i think are now ok | 15:09 |
| jrosser | centos is very close, just the pam_env thing to verify and merge | 15:09 |
| damiandabrowski[m] | jrosser: have You managed to solve this issue with mounting cephfs after container restart or You need some help? | 15:09 |
| jrosser | hmm? | 15:10 |
| jrosser | i've not been looking at cephfs | 15:10 |
| damiandabrowski[m] | glusterfs* sorry ;) | 15:10 |
| jrosser | oooh - do we still have trouble there | 15:10 |
| jrosser | oh it was a race condition with systemd wasnt it | 15:11 |
| damiandabrowski[m] | yeah, exactly | 15:11 |
| jrosser | tbh i have not spent any time on that | 15:11 |
| jrosser | also i don't really have any great idea after reading the systemd bug report | 15:11 |
| damiandabrowski[m] | i think there might be some workaround with putting systemd-mount behind systemd service | 15:13 |
| damiandabrowski[m] | I'll try to spend some time on this during this week but can't promise anything :/ | 15:13 |
| noonedeadpunk | Ok, next to that I've started creating presentation for project update | 15:15 |
| noonedeadpunk | I think deadline for it is end of the week for it to be shown/promoted during summit | 15:16 |
| jrosser | something is also wrong with repo server upgrades here https://zuul.opendev.org/t/openstack/build/7e33eaf8cfb5445c92bcbe38647b792d/log/job-output.txt#19556-19558 | 15:16 |
| jrosser | i've not seen that before | 15:16 |
| *** ysandeep|rover is now known as ysandeep|out | 15:17 | |
| noonedeadpunk | unarchive issue? interesting... | 15:17 |
| noonedeadpunk | Here's the link for presentation https://docs.google.com/presentation/d/1INXXvAY9MuwASYW4eXLye7YvFaosv5K-7XS7jX6AlpI/edit?usp=sharing | 15:17 |
| noonedeadpunk | feedback is welcome. And also if somebody wants to participate in writing down sound or add some slides - let me know:) | 15:17 |
| damiandabrowski[m] | good job so far! | 15:19 |
| jrosser | i wonder if the unarchive starts before the mount is made completely | 15:19 |
| NeilHanlon | slides look great so far (: | 15:20 |
| noonedeadpunk | jrosser: that would be weird given it's systemd managing mount | 15:20 |
| jrosser | yeah, though i don't know if it's asynchronous | 15:21 |
| noonedeadpunk | I'd say ansible module should wait until service/mount started | 15:21 |
| noonedeadpunk | But it look like some permissions issue tbh... | 15:22 |
| noonedeadpunk | `Cannot change ownership to uid 998, gid 33` | 15:22 |
| jrosser | oh now thats interesting actually | 15:23 |
| jrosser | i don't think that we ensure that the repo / nginx uid and gid are consistent between the repo servers | 15:23 |
| jrosser | that could go wrong | 15:23 |
| jrosser | though in this upgrade case it should be only one of then makes the archive and unpacks it into the glusterfs | 15:24 |
| jrosser | but i had been thinking we need to do something about the uid/gid anyway | 15:25 |
| noonedeadpunk | oh yes, actually that's good point | 15:25 |
| noonedeadpunk | but are you sure it's about same server? | 15:25 |
| noonedeadpunk | as it's infra job, which means there're 3 of them? | 15:26 |
| jrosser | thats right | 15:26 |
| noonedeadpunk | Maybe we archived on one, and unpackking on another one? But unlikely... | 15:26 |
| jrosser | hah well look at the failing tasks - its on two different ones | 15:27 |
| noonedeadpunk | we actually need to run_once this https://opendev.org/openstack/openstack-ansible-repo_server/src/branch/master/tasks/repo_pre_install.yml#L80-L88 as well :) | 15:28 |
| noonedeadpunk | But good that we didn't | 15:28 |
| jrosser | something very odd is happening there | 15:28 |
| jrosser | it's running a bunch of tasks against only two of the 3 | 15:29 |
| noonedeadpunk | I think we have serial: 1, 100% | 15:29 |
| noonedeadpunk | so "as designed"? | 15:29 |
| noonedeadpunk | https://opendev.org/openstack/openstack-ansible/src/branch/master/playbooks/repo-install.yml#L25 | 15:29 |
| jrosser | aah right yes | 15:30 |
| jrosser | so we need is_first_play_host type of thing like the other roles | 15:31 |
| jrosser | i can look at that | 15:31 |
| jrosser | it could be at the moment we are lucky that uid/gid are assigned with the same algorithm in 3 identical containers | 15:32 |
| jrosser | thats not going to be the case for long lived deployments or cross-OS in the future | 15:33 |
| noonedeadpunk | yes, totally | 15:34 |
| noonedeadpunk | I haven't actually looked much in Octavia PKI since I pushed path. I hope to get to it tomorrow | 15:35 |
| noonedeadpunk | As it;s smth we need to land sooner better | 15:35 |
| jrosser | we could look at bugs? i think a bunch have been addressed | 15:39 |
| noonedeadpunk | #topic bug triage | 15:41 |
| noonedeadpunk | yeah. I can't recall anyting new, but we closed a lot lately | 15:41 |
| noonedeadpunk | #link https://bugs.launchpad.net/openstack-ansible/+bug/1973762 | 15:42 |
| noonedeadpunk | jrosser: I think you was working on this one? | 15:42 |
| jrosser | i didnt make a patch, but basically we template out a file that is already provided by the distro | 15:43 |
| jrosser | then i think our changes to make it ext4 get set back to btrfs | 15:43 |
| noonedeadpunk | uh | 15:44 |
| jrosser | i thought we could make a systemd drop-in to override the fs type= which would then be persistent | 15:44 |
| jrosser | but i didnt try it yet | 15:44 |
| noonedeadpunk | So we basically should be using systemd overrides for that? | 15:44 |
| jrosser | theres some vauge comment in the original file about it being a temporary placeholder until some later version of systemd | 15:45 |
| noonedeadpunk | I think https://opendev.org/openstack/openstack-ansible-lxc_hosts/commit/337ddf87802c7597b7c51a071eb9829a4b87defa is related | 15:45 |
| jrosser | so it might be that we need to template it out, and also to make the drop in as well | 15:45 |
| noonedeadpunk | But I'm not sure it makes things better or worse | 15:45 |
| jrosser | well, i see this on my Xena deploys today | 15:46 |
| noonedeadpunk | Commit is for master only. And I think it makes things even worse | 15:46 |
| jrosser | do they both make the same unit though? | 15:47 |
| jrosser | how is it worse | 15:47 |
| noonedeadpunk | It creates another mount? for same endpoints | 15:47 |
| noonedeadpunk | And kind of not ensuring that existing one is stopped/disabled | 15:48 |
| noonedeadpunk | I can look at that | 15:49 |
| noonedeadpunk | And smth backportable should be made as well | 15:49 |
| noonedeadpunk | for stable branches | 15:49 |
| jrosser | i have an AIO here and i think that use of systemd_service overwrites the original unit | 15:49 |
| jrosser | but iirc the old code does that too | 15:50 |
| jrosser | anyway | 15:50 |
| noonedeadpunk | Huh, I thought we're placing our units only under /etc/system | 15:50 |
| noonedeadpunk | oh, well, if they have same names at the end - it's just an override then | 15:51 |
| noonedeadpunk | https://opendev.org/openstack/ansible-role-systemd_mount/src/branch/master/tasks/systemd_mounts.yml#L56 | 15:51 |
| jrosser | right, in my AIO i have /etc/systemd/system/var-lib-machines.mount | 15:52 |
| jrosser | and its our one with ext4 | 15:52 |
| noonedeadpunk | Then I guess it should be fine :) | 15:54 |
| noonedeadpunk | the only part left - backport smth to stable branches | 15:54 |
| noonedeadpunk | I'm thinking to just put simple override of fs type to /etc/systemd/system and be done with it | 15:55 |
| jrosser | we should look back though eavesdrop to what spatel shared | 15:55 |
| spatel | Reading thread | 15:57 |
| jrosser | this is what we tread on https://packages.ubuntu.com/search?suite=focal&arch=any&mode=exactfilename&searchon=contents&keywords=var-lib-machines.mount | 15:58 |
| *** dviroel is now known as dviroel|lunch | 15:58 | |
| spatel | Got it :) | 15:58 |
| noonedeadpunk | I think it's quite clear what the issue is... | 16:00 |
| noonedeadpunk | #endmeeting | 16:00 |
| opendevmeet | Meeting ended Tue May 24 16:00:21 2022 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:00 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/openstack_ansible_meeting/2022/openstack_ansible_meeting.2022-05-24-15.00.html | 16:00 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/openstack_ansible_meeting/2022/openstack_ansible_meeting.2022-05-24-15.00.txt | 16:00 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/openstack_ansible_meeting/2022/openstack_ansible_meeting.2022-05-24-15.00.log.html | 16:00 |
| jrosser | i am a little surprised that `systemctl status var-lib-machines.mount` doesnt show the path to the unit file on my Xena deploy | 16:01 |
| jrosser | but the title is `Virtual Machine and Container Storage (Compatibility)` | 16:02 |
| noonedeadpunk | oh, yes, it's weird indeed... | 16:04 |
| noonedeadpunk | I have some feeling, that overrides does not work for mounts | 16:07 |
| noonedeadpunk | or well, it fully overrides whole unit rather then just specific options | 16:08 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-lxc_hosts stable/xena: Do not mess up with distro-provided machines mount https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/843169 | 16:12 |
| noonedeadpunk | I guess it can be as simple as that ^ | 16:17 |
| noonedeadpunk | spatel: would be great if you could check that... | 16:18 |
| spatel | +1 looks good so far | 16:20 |
| spatel | osa provide file will override this one correct? /lib/systemd/system/var-lib-machines.mount | 16:21 |
| noonedeadpunk | Nope, it would place it now to `/etc/systemd/system/var-lib-machines.mount` the one in /lib is provided by systemd-containerd | 16:24 |
| noonedeadpunk | So not to mess up with it (as it will be overriden on package update) we will place ours to /etc/ | 16:24 |
| noonedeadpunk | which should be a valid override of it | 16:25 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-ceph_client stable/victoria: Disable setuptools auto discovery https://review.opendev.org/c/openstack/openstack-ansible-ceph_client/+/842949 | 16:30 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-ceph_client stable/victoria: Use global package_state https://review.opendev.org/c/openstack/openstack-ansible-ceph_client/+/842408 | 16:31 |
| *** dviroel|lunch is now known as dviroel | 16:58 | |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-tests stable/wallaby: Fix rich version for repos that don't have test-requirements https://review.opendev.org/c/openstack/openstack-ansible-tests/+/843175 | 17:11 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-ceph_client stable/wallaby: Use global package_state https://review.opendev.org/c/openstack/openstack-ansible-ceph_client/+/842407 | 17:12 |
| opendevreview | Merged openstack/ansible-role-python_venv_build stable/train: Remove tripleo-ci-centos-7-standalone-os-tempest from gate layout https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/838169 | 17:12 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Allow lxc container backing store to be defined as part of the scenario https://review.opendev.org/c/openstack/openstack-ansible/+/843011 | 17:15 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Add zuul jobs and template for testing different LXC backing stores https://review.opendev.org/c/openstack/openstack-ansible/+/843027 | 17:16 |
| spatel | noonedeadpunk +1 | 18:36 |
| opendevreview | Merged openstack/openstack-ansible-os_keystone stable/wallaby: add oauth support https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/832788 | 19:39 |
| spatel | noonedeadpunk what is wrong here - https://paste.opendev.org/show/bNm4iaj7UgMyVoip8WDD/ | 19:52 |
| spatel | since i upgraded to 23.3.0 i started getting this error | 19:52 |
| spatel | some kind of conflict | 19:53 |
| opendevreview | Merged openstack/openstack-ansible-os_octavia stable/wallaby: Fix condition for deleting old amp images https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/842936 | 19:55 |
| NeilHanlon | looks like it wants to downgrade librados2 and librdb1 | 20:02 |
| NeilHanlon | spatel 👆 | 20:03 |
| NeilHanlon | seems like maybe a broken dep or something weird in the ubuntu packages, if I'm reading it correct 🤔 | 20:04 |
| jrosser | spatel: can you show `apt policy ceph-common` for that node? | 20:22 |
| spatel | sorry i was in meeting, i am back now.. let me show you | 20:26 |
| spatel | jrosser https://paste.opendev.org/show/bdYKdTELLejqADWyHhwz/ | 20:27 |
| jrosser | spatel: there is an explanation here i think in the commit message https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/838657 | 20:32 |
| jrosser | there should be an apt log on your hosts to show you what was installed, and what was upgraded/downgraded | 20:34 |
| jrosser | you could also look into this as well https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/838855/3 | 20:36 |
| jrosser | and add `APT::Install-Recommends "false";` to your hosts apt config | 20:36 |
| spatel | hmm | 20:37 |
| jrosser | those patches are all to master as we saw the Quincy release of ceph from UCA interfere with expecting to get Octopus from ceph.com | 20:37 |
| jrosser | you may have a similar but slightly different situation on a stable branch | 20:37 |
| jrosser | a combination of looking in the apt log, and what `apt policy` shows you should let you work out whats going on | 20:38 |
| spatel | let me first try APT::Install-Recommends "false" | 20:38 |
| jrosser | well, it's a little late now | 20:39 |
| spatel | why? | 20:39 |
| jrosser | is that a compute node or a controller? | 20:39 |
| spatel | compute node | 20:39 |
| jrosser | interesting | 20:40 |
| jrosser | so those patches i showed you fixed a thing that happened when LXC got installed, it has a 'recommended' dependancy of something which in turn required librbd | 20:40 |
| spatel | In this cloud we have 180 compute nodes life was good and last week i have upgraded to 23.3.0 and today trying to add 30 more compute nodes but encounter this error | 20:40 |
| jrosser | so a very indirect dependancy | 20:40 |
| jrosser | and we figured out that was due to installing the lxc-templates package really early in the deployment | 20:41 |
| jrosser | long long before the ceph_client role had made config that only packages from download.ceph.com were to be used | 20:42 |
| jrosser | so $something thats being installed on your compute node is pulling in librbd | 20:42 |
| jrosser | and then later along comes ceph_client and sets up the repo priorities properly, and the previously installed librbd now needs to be downgraded to match the new apt policy | 20:43 |
| spatel | APT::Install-Recommends "false" didn't help anyway | 20:43 |
| opendevreview | Merged openstack/openstack-ansible-os_neutron master: Support dns_domain_keywords extension https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/842102 | 20:44 |
| jrosser | if a too new librbd has already been installed, then yes, it won;t make any difference | 20:44 |
| jrosser | which branch (name?) too late for me to parse release numbers :) | 20:45 |
| spatel | how about if i remove that apt_pin ? | 20:45 |
| spatel | wallaby / 23.3.0 | 20:45 |
| jrosser | remove? | 20:45 |
| spatel | librbd1 | 20:45 |
| spatel | This one currently installed | 20:46 |
| spatel | ii librbd1 16.2.7-0ubuntu0.21.04.1~cloud2 amd64 RADOS block device client library | 20:46 |
| spatel | OSA trying to install 15.2.x | 20:46 |
| jrosser | it would be very helpful to know why that got installed | 20:46 |
| jrosser | which is where looking in the apt log will help you | 20:47 |
| spatel | Let me check apt logs | 20:47 |
| jrosser | btw unless you find a specific thing changed in 23.3.0 it is unlikley to be something we changed in openstack-ansible | 20:48 |
| jrosser | it is much more likley to be some external factors like new pacakges in UCA which we don't have an appropriate apt pin for early enough in the deployment | 20:49 |
| spatel | jrosser this is how it got installed by OSA - https://paste.opendev.org/show/bmbFY9ezTRIgtgpB7rAt/ | 20:52 |
| spatel | look like qemu-* packages install that lib | 20:54 |
| jrosser | right | 20:57 |
| jrosser | so i think this is a bug | 20:57 |
| jrosser | which has probably been there for a long time | 20:58 |
| jrosser | would be interesting to compare with the apt log on one of your previous hosts | 20:58 |
| spatel | i can check that also wait.. | 20:58 |
| jrosser | what happens is that in our CI job for ceph the ceph OSD is setup before nova, so the apt pins are placed on the *same node* as what will be the compute node, early in the deploy | 20:59 |
| jrosser | and then when it comes time to install librbd for qemu-*, it's already present from the ceph-osd setup, and meets the apt-policy becasue the pins were already placed | 21:00 |
| jrosser | i have the apt log from CI here and we can see it has installed librbd to the correct version before qemu https://zuul.opendev.org/t/openstack/build/f7abcc2628b34f779a9e55f7467c2eed/log/logs/host/apt/history.log.txt#56-60 | 21:02 |
| jrosser | oh wait, thats also an upgrade..... `Upgrade: librbd1:amd64 (15.2.16-0ubuntu0.20.04.1, 16.2.7-0ubuntu0.21.04.1~cloud2)` | 21:03 |
| jrosser | hmmm | 21:03 |
| spatel | This is my old box and that package comes from difference place - https://paste.opendev.org/show/ba8RbMsY53RTPr1vyPAn/ | 21:05 |
| jrosser | can you show what apt-policy says there? | 21:06 |
| spatel | https://paste.opendev.org/show/b6808PTwTBSOLV1u1yyq/ | 21:09 |
| spatel | This is old box | 21:09 |
| spatel | https://paste.opendev.org/show/bdYKdTELLejqADWyHhwz/ | 21:10 |
| spatel | This is new box | 21:10 |
| *** dviroel is now known as dviroel|out | 21:11 | |
| spatel | here is the librbd comparison - https://paste.opendev.org/show/bmdFvO44YmuX4yn41Y4r/ | 21:12 |
| spatel | I have to leave now but i will keep poking and try to report bug if its real issue. | 21:12 |
| jrosser | it is odd that those packages have 21.04 in the name | 21:12 |
| jrosser | i have started a bug, i'll drop the link here for you | 21:13 |
| spatel | Thank you! | 21:13 |
| spatel | I may go offline and won't see link but anyway will followup with you tomorrow :) | 21:13 |
| spatel | thank you for helping out | 21:13 |
| jrosser | https://bugs.launchpad.net/openstack-ansible/+bug/1975643 | 21:16 |
| jrosser | i really do wonder about the 21.04 in the name, thats just not feeling right | 21:16 |
| spatel | +1 | 21:17 |
| NeilHanlon | 👀https://www.reddit.com/r/Python/comments/uwhzkj/i_think_the_ctx_package_on_pypi_has_been_hacked/ | 22:49 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!