| opendevreview | James Kirsch proposed openstack/kolla-ansible master: Support for keystone scoped authorization https://review.opendev.org/c/openstack/kolla-ansible/+/692179 | 02:31 |
|---|---|---|
| mnasiadka | mgoddard: forgot about them, will do it today :) | 04:50 |
| yoctozepto | morning | 06:00 |
| yoctozepto | priteau: oh, oh, that explains why train started failing on python-libvirt | 06:02 |
| yoctozepto | and we have a report about the same on victoria | 06:02 |
| yoctozepto | though in CI i have only seen it in train | 06:02 |
| yoctozepto | argh | 06:02 |
| yoctozepto | centos 8 is a joke now | 06:03 |
| *** rpittau|afk is now known as rpittau | 07:13 | |
| priteau | morning | 07:19 |
| Fl1nt | Hi everyone! :D | 07:23 |
| *** hrw is now known as Guest2211 | 07:31 | |
| parallax | Morning | 07:31 |
| *** Guest2211 is now known as hrw | 07:32 | |
| priteau | yoctozepto: I was looking at the build error in train. virDomainAuthorizedSSHKeysGet was added in libvirt-python v6.10.0 | 07:36 |
| Fl1nt | priteau, you get build errors on train ? | 07:36 |
| priteau | CI does | 07:36 |
| Fl1nt | Interestingly weird, I did a build on yesterday evening and don't get it. | 07:37 |
| Fl1nt | which distribution? | 07:37 |
| Fl1nt | DEB Base I suppose? | 07:37 |
| priteau | CentOS 8 | 07:42 |
| priteau | Only affects masakari-monitors | 07:42 |
| opendevreview | Pierre Riteau proposed openstack/kolla stable/train: Fix build of masakari-monitors image https://review.opendev.org/c/openstack/kolla/+/796387 | 07:43 |
| Fl1nt | Aaaaaaah ok, I can see why :D | 07:43 |
| Fl1nt | Nice catch tho ^^ | 07:43 |
| Fl1nt | by the way, this versionning drift of a key component bring me back to a question that I had from a meeting recently. | 07:45 |
| Fl1nt | How do we track dependencies versionning on kolla? From my POV we do it through the CI that will error (like this issue) on regular build, but with stream bringing daily changes, will we be able to do it that way? | 07:46 |
| yoctozepto | priteau: I wonder if we shouldn't actually pin libvirt itself | 07:47 |
| mgoddard | morning | 07:48 |
| priteau | yoctozepto: yes, maybe. I mostly wanted to check if it fixes it | 07:48 |
| yoctozepto | priteau: sure, thanks for working on that; also - any idea why only train fails in CI and newer ones only for users? | 07:50 |
| priteau | No idea | 07:51 |
| priteau | Upper constraints are different obviously | 07:51 |
| Fl1nt | mgoddard, let say I want to make a bug report the umbrella for few smaller patch set, how do I do that? | 08:02 |
| mgoddard | Fl1nt: Related-Bug: #123456 | 08:03 |
| Fl1nt | aaaah perfect ! thx | 08:03 |
| Fl1nt | Do I only need this Related-Bug or do I still need to put Change-Id and Bug: #1234 ? | 08:03 |
| priteau | Change-Id is mandatory for Gerrit | 08:10 |
| priteau | git-review will add it for you if you remove it | 08:10 |
| Fl1nt | ok, noticed, thanks ^^ | 08:10 |
| priteau | If you remove the Change-Id, and git-review adds another one back, you will create a duplicate change in Gerrit | 08:11 |
| Fl1nt | Additional though, we use apt within our Dockerfile for DEB based distribs, I do prefer this notation, BUT it seems apt doesn't have stabilized CLI APIs shouldn't we rather use apt-get alternatives in order to avoid any weird behaviors because of that? | 08:11 |
| Guest529 | priteau: https://bugs.launchpad.net/bugs/1931817 | 08:11 |
| yoctozepto | priteau: yeah, they are different, but should not be different user vs CI | 08:13 |
| priteau | I agree | 08:13 |
| priteau | I have a victoria dev environment running actually, can give it a try | 08:13 |
| parallax | https://review.opendev.org/ down? | 08:16 |
| parallax | no, just slow | 08:16 |
| Fl1nt | So, I'm working on improving our images build when dealing with an offline environment. The first patch will be for dumb-init based images; We currently pin the version used to 1.2.2 from github. | 08:17 |
| Fl1nt | I use the CentOS/Ubuntu provided binary versions from more than a year on a fairly large platform now | 08:18 |
| Fl1nt | and on COS8 I'm even more advanced than the pinned version | 08:18 |
| Fl1nt | without any issue/error or outage | 08:18 |
| Fl1nt | in order to support multiple scenarii such as offline/online/ sources based/binary based I've two options | 08:19 |
| Fl1nt | either I conditionally do it on the dumb-init installation block | 08:19 |
| Fl1nt | OR | 08:19 |
| Fl1nt | I can condition the dumb-init block just for online+source based or offline+source based scenarii | 08:20 |
| Fl1nt | What do you prefer folks? | 08:20 |
| mgoddard | Fl1nt: if every distro has a sufficient version packaged we could use that | 08:28 |
| Fl1nt | I know that CentOS and Debian/Ubuntu does have 1.2+ releases, do we support any other distrib? | 08:32 |
| Fl1nt | Debian stable (buster) use 1.2.2-1.1 | 08:32 |
| Fl1nt | Ubuntu stable (21.04) use 1.2.5-1 | 08:32 |
| Fl1nt | CentOS stable (8/8Stream) use 1.2.6 | 08:33 |
| opendevreview | Mark Goddard proposed openstack/kolla master: Pin td-agent to 4.0.* to fix missing logs https://review.opendev.org/c/openstack/kolla/+/795086 | 08:33 |
| Fl1nt | 1.2.2.6 sorry | 08:33 |
| yoctozepto | we usue ubuntu focal (20.04), not 21.04 | 08:34 |
| yoctozepto | and now debian bullseye too | 08:34 |
| yoctozepto | use* | 08:34 |
| yoctozepto | anyhow, they all look good enough | 08:34 |
| Fl1nt | ok, let's go then | 08:35 |
| opendevreview | Merged openstack/kolla master: [docs] Fix Debian release name https://review.opendev.org/c/openstack/kolla/+/796275 | 08:35 |
| opendevreview | Radosław Piliszek proposed openstack/kolla master: Pin td-agent to 4.0.* to fix missing logs https://review.opendev.org/c/openstack/kolla/+/795086 | 08:36 |
| Fl1nt | last question, do you prefer an additional conditional test based on source for the dumb_init_installation block and then by default for rdo/binary install_type I use the packages or everything dealed on dumb_init block? | 08:36 |
| yoctozepto | just install the packages and we are good | 08:37 |
| mgoddard | +1 | 08:37 |
| mnasiadka | morning | 08:37 |
| yoctozepto | it's an old workaround | 08:37 |
| Fl1nt | ok ok | 08:38 |
| Fl1nt | perfect | 08:38 |
| opendevreview | Radosław Piliszek proposed openstack/kolla stable/wallaby: [docs] Fix Debian release name https://review.opendev.org/c/openstack/kolla/+/796228 | 08:40 |
| opendevreview | Mark Goddard proposed openstack/kolla-ansible master: Reno follow up for docker_disable_ip_forward https://review.opendev.org/c/openstack/kolla-ansible/+/796406 | 08:50 |
| hub_kalle | Hi, I'm running into a wierd issue when trying to deploy kayobe: on "baremetal node list" shows my nodes, everything looks fine (7 nodes, 3 controllers, 4 compute) . I do "kayobe overcloud provision" and everything looks perfect until it fails when waiting for the node to become active again | 08:52 |
| hub_kalle | it appears all the comute node are hung up in "wait for callback" | 08:53 |
| hub_kalle | but it looks like the provisioning worked | 08:53 |
| hub_kalle | all the node are accessible via ssh | 08:53 |
| hub_kalle | and they are provisioned with ubuntu 20.04 | 08:54 |
| hub_kalle | (i'm running kayobe wallaby/stable) | 08:54 |
| hub_kalle | now I'm trying to figure out how to debug this | 08:55 |
| hub_kalle | the bifrost deploy log only tells me that it waited for callback to long | 08:55 |
| opendevreview | Mark Goddard proposed openstack/kolla-ansible master: Drop /sys/fs/cgroup mounts https://review.opendev.org/c/openstack/kolla-ansible/+/793785 | 08:55 |
| hub_kalle | Question: should I take this up with someone at ironic or do you guys have an idea? | 08:56 |
| priteau | hub_kalle: are you sure your nodes were not provisioned in an earlier attempt? | 09:00 |
| jingvar | Hi! Which OS should I use for a-multiverse | 09:00 |
| mgoddard | jingvar: on a-multiverse-from-nothing-master, you can use CentOS Stream/Linux 8 or Ubuntu 20.04 | 09:01 |
| hub_kalle | yes I'm watching the whole procedure via console | 09:04 |
| hub_kalle | and I had that exact problem before when those machines where always starting into a Centos 8 | 09:05 |
| jingvar | mgoddard: thanks, what about CentOS Stream and OMVF? is it fixed, or I have to downgrade the package? | 09:06 |
| hub_kalle | Question: i'm running "kayobe overcloud deprovision" in between attempts. | 09:06 |
| priteau | jingvar: The issue is fixed thanks to the updated libvirt package in AppStream | 09:06 |
| hub_kalle | Does this means it's possible I'm now just seeing the result of an earlier failed attempt? | 09:07 |
| jingvar | good news | 09:08 |
| opendevreview | Merged openstack/kolla stable/wallaby: [docs] Fix Debian release name https://review.opendev.org/c/openstack/kolla/+/796228 | 09:09 |
| mgoddard | hub_kalle: it sounds like the machines are not PXE booting the IPA ramdisk, but booting into an existing OS | 09:18 |
| mgoddard | hub_kalle: are they configured to PXE boot on the correct NIC? You should be able to watch the boot on a machine to see what it's doing | 09:18 |
| hub_kalle | pretty sure that's not the problem but i'm going to run the process again (I configures them to boot via pxe on all nics and I did auto discovery on them which only works if pxe works) | 09:25 |
| mgoddard | hub_kalle: you can also use tcpdump on dhcp and pxe ports to be sure | 09:30 |
| hub_kalle | mgoddard: that will be my next step | 09:31 |
| Fl1nt | mgoddard, where can I put default vars value for kolla? not k-a. | 09:39 |
| opendevreview | Mark Goddard proposed openstack/kolla-ansible master: WIP: Log fluentd events in TSV format https://review.opendev.org/c/openstack/kolla-ansible/+/795512 | 09:45 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: Switch OPENSTACK_RELEASE back to master https://review.opendev.org/c/openstack/kolla/+/796415 | 09:56 |
| hub_kalle | mgoddard: looks like you're right :( (even though I have no clue why, on the provisioning step these node skip PXE) - sorry for the dumb question then | 10:00 |
| jingvar | kayobe seed hypervisor host configure - where should I configure username ? Failed to connect to the host via ssh: centos@192.168.33.4 - my user is cloud-user , controller_bootstrap_user: cloud-user does'nt work | 10:10 |
| jingvar | for a-multiverse-from-nothing-master | 10:11 |
| jingvar | ansible/kayobe-ansible-user.yml >> ansible_user: "{{ bootstrap_user }}" | 10:14 |
| priteau | jingvar: seed_hypervisor_bootstrap_user | 10:17 |
| jingvar | thanks, i've found it | 10:18 |
| opendevreview | Maksim Malchuk proposed openstack/kayobe master: Set defaults for openstack_cacert https://review.opendev.org/c/openstack/kayobe/+/793703 | 10:22 |
| opendevreview | Verification of a change to openstack/kolla-ansible failed: Disable docker's ip-forward when iptables disabled https://review.opendev.org/c/openstack/kolla-ansible/+/796223 | 10:23 |
| opendevreview | Maksim Malchuk proposed openstack/kayobe master: TLS certificates management sync with Kolla-Ansible https://review.opendev.org/c/openstack/kayobe/+/793697 | 10:24 |
| opendevreview | Marcin Juszkiewicz proposed openstack/kolla master: drop leftovers of RHEL support https://review.opendev.org/c/openstack/kolla/+/785569 | 10:30 |
| hrw | morning | 10:31 |
| sean-k-mooney | o/ | 10:39 |
| sean-k-mooney | any ceph experts around at the moment? | 10:40 |
| sean-k-mooney | has anyone tried to get kolla-ansible working with cephadm recently | 10:40 |
| sean-k-mooney | im having trouble getting the serivces to be able to auth to ceph | 10:41 |
| sean-k-mooney | my admin keyring works fine for me but i cant get the client keyrings to work for the services | 10:41 |
| sean-k-mooney | do we know if there is a writeup on hits | 10:41 |
| hrw | ssh: connect to host review.openstack.org port 29418: Connection timed out | 10:41 |
| sean-k-mooney | *this | 10:41 |
| hrw | anyone has it? | 10:41 |
| sean-k-mooney | hrw: you should use review.opendev.org | 10:41 |
| sean-k-mooney | hrw: the redirect kind of works for http but ssh does not | 10:42 |
| mnasiadka | sean-k-mooney: we have a cephadm based CI in stable/wallaby | 10:42 |
| sean-k-mooney | at least i have had no look usign the old domain with git-review | 10:42 |
| mnasiadka | hrw: I have the same, it sometimes works ;) | 10:43 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: Update previous_release to Victoria https://review.opendev.org/c/openstack/kolla-ansible/+/796422 | 10:43 |
| hrw | sean-k-mooney: time to check .git/config then | 10:43 |
| sean-k-mooney | hrw: or your /etc/hosts | 10:43 |
| sean-k-mooney | i used to have review.openstack.org hard coded for a time when we had dns issues in the past | 10:44 |
| hrw | sean-k-mooney: it was .git/config | 10:44 |
| sean-k-mooney | ack | 10:44 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: Switch OPENSTACK_RELEASE back to master https://review.opendev.org/c/openstack/kolla/+/796415 | 10:44 |
| hrw | my /etc/hosts contains only one 10.*.*.* entry | 10:44 |
| mnasiadka | hrw: but review.opendev.org has the same problem for me, so I don't know if it will help much :) | 10:46 |
| sean-k-mooney | mnasiadka: i might see what ye are doing different | 10:46 |
| sean-k-mooney | mnasiadka: Failed to configure store correctly: Store rbd could not be configured correctly. Reason: Error in store configuration: [errno 13] RADOS permission denied (error connecting to the cluster) Disabling add method. | 10:46 |
| sean-k-mooney | that is what im seeign in glance | 10:47 |
| hrw | mnasiadka: ;D | 10:47 |
| mnasiadka | sean-k-mooney: we are doing this in CI - https://github.com/openstack/kolla-ansible/blob/3675b442c9f3e77a2a3b776c75c03fc3ce853c73/tests/run.yml#L308 + role https://github.com/openstack/kolla-ansible/blob/master/roles/cephadm/tasks/main.yml - nothing fancy | 10:48 |
| sean-k-mooney | same issue in cinder | 10:48 |
| sean-k-mooney | the reall issue is 2021-06-14T06:52:55.468+0000 7f827ffff700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2,1]\n[errno 13] RADOS permission denied (error connecting to the cluster)\n which shows up in nova | 10:48 |
| mnasiadka | sean-k-mooney: ah, that's probably due to security patch in ceph | 10:49 |
| sean-k-mooney | mnasiadka: do you think the service might be trying to use v1 | 10:49 |
| mnasiadka | sean-k-mooney: https://docs.ceph.com/en/latest/security/CVE-2021-20288/#cve-2021-20288 | 10:49 |
| sean-k-mooney | instead of v2 | 10:49 |
| mnasiadka | sean-k-mooney: let me guess, you're using a ubuntu deployment of Kolla? | 10:50 |
| sean-k-mooney | yes | 10:50 |
| sean-k-mooney | ubuntu source | 10:50 |
| sean-k-mooney | with ceph form i guess upstream | 10:50 |
| sean-k-mooney | i assume cephadm pulls the images directly form ceph.com? | 10:51 |
| sean-k-mooney | well docker but not distro ceph | 10:51 |
| mnasiadka | so Ubuntu hasn't updated the client ceph packages to a version that is mention in the CVE, but your server is patched - so you need to set auth_allow_insecure_global_id_reclaim to True | 10:51 |
| sean-k-mooney | ah ok | 10:51 |
| sean-k-mooney | im glad i asked because i was not finding any info on this with google | 10:52 |
| sean-k-mooney | so as admin in need to do something like this | 10:53 |
| sean-k-mooney | ceph config set mon auth_allow_insecure_global_id_reclaim true | 10:53 |
| sean-k-mooney | form the cephadm shell | 10:53 |
| sean-k-mooney | or just set it in my ceph.conf | 10:53 |
| mnasiadka | sean-k-mooney: yes, ceph config is enough | 10:54 |
| sean-k-mooney | mnasiadka: would it make sense to rebuild the kolla images locally with the ceph repo enabeld to get a new ceph client in the kolla containers? or just wait for ubunut to update | 10:54 |
| mnasiadka | sean-k-mooney: just like we do in CI - https://github.com/openstack/kolla-ansible/blob/3675b442c9f3e77a2a3b776c75c03fc3ce853c73/roles/cephadm/tasks/main.yml#L89 | 10:55 |
| mnasiadka | sean-k-mooney: I don't think UCA has updated the ceph packages yet | 10:55 |
| mnasiadka | so a rebuild won't help | 10:55 |
| sean-k-mooney | mnasiadka++ | 10:55 |
| sean-k-mooney | mnasiadka: well i was going to enable the package form ceph.com if i rebuilt | 10:55 |
| mnasiadka | sean-k-mooney: ah, that one would help for sure ;) | 10:56 |
| opendevreview | Mark Goddard proposed openstack/kolla-ansible master: WIP: Log fluentd events in TSV format https://review.opendev.org/c/openstack/kolla-ansible/+/795512 | 11:03 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: Update previous_release to Wallaby https://review.opendev.org/c/openstack/kolla-ansible/+/796422 | 11:29 |
| opendevreview | Merged openstack/kolla-ansible master: Drop /sys/fs/cgroup mounts https://review.opendev.org/c/openstack/kolla-ansible/+/793785 | 11:38 |
| opendevreview | Mark Goddard proposed openstack/kolla-ansible master: WIP: Log fluentd events in TSV format https://review.opendev.org/c/openstack/kolla-ansible/+/795512 | 11:38 |
| hub_kalle | mgoddard: I debugged my problem (somehow some of my node didn't do PXE) a little more, turns out for some reason my PXE Settings get overwritten at some point: I set the Boot Sequence to PXE over Hard Drive and at sometime between restarts it resets to Hard Drive over PXE | 11:39 |
| hub_kalle | Question: is there any chance this is a kayobe issue (If yes I would take some time to figure out what happens and if neccessary write a bug report) | 11:39 |
| hub_kalle | (it is quite likely this is just a dell idrac issue since all of the affected machines are from one batch of very new machines) | 11:40 |
| hub_kalle | I'm just not sure weather kayobe telling idrac to boot via PXE or to boot via Hard Drive could be the problem | 11:41 |
| opendevreview | Radosław Piliszek proposed openstack/kolla-ansible stable/wallaby: Drop /sys/fs/cgroup mounts https://review.opendev.org/c/openstack/kolla-ansible/+/796231 | 11:43 |
| opendevreview | Radosław Piliszek proposed openstack/kolla-ansible stable/wallaby: Drop /sys/fs/cgroup mounts https://review.opendev.org/c/openstack/kolla-ansible/+/796231 | 11:43 |
| mgoddard | hub_kalle: ironic will tell the nodes to boot from disk once provisioning is complete, but will PXE boot during provisioning | 11:46 |
| priteau | hub_kalle: make sure you have the latest firmware and idrac versions | 12:21 |
| priteau | also are you using bios or uefi boot? | 12:22 |
| hub_kalle | I'm using bios | 12:24 |
| hub_kalle | priteau: any recommendation because I'm only using Bios because I always used it (which isn't a good reason for anything) | 12:25 |
| priteau | if you were using UEFI I would tell you to change the boot_mode in ironic, but that's not required in your case since bios is still the default | 12:25 |
| sean-k-mooney | even today using bios tend to be a better bet for older hardware | 12:25 |
| sean-k-mooney | i think there is an ironic settign for local boot | 12:27 |
| hrw | sean-k-mooney: s/older/x86(-64)/ in general probably | 12:28 |
| sean-k-mooney | ironic node-update <node-uuid> add properties/capabilities="boot_option:local" | 12:28 |
| sean-k-mooney | is ^ being set anywhere | 12:28 |
| hub_kalle | btw firmware and idrac version should be preety up to date. But it's possible they're not the latest (maybe half a year old) | 12:28 |
| sean-k-mooney | https://docs.openstack.org/ironic/pike/install/include/local-boot-partition-images.html | 12:29 |
| sean-k-mooney | hub_kalle: you do not want local boot right? | 12:29 |
| hub_kalle | no I wanted to provision via kayobe so I needed PXE boot and for some reason my Boot Sequence on the new Servers seems to reset at some point between restarts. | 12:31 |
| hub_kalle | It's not a big problem, now that I know it exists. But it's annoying because now I need to observe the Deployment Process to see weather PXE worked or not ... | 12:32 |
| hub_kalle | but before debugging this I will update all firmware and iDRAC and see if the problem still persists | 12:32 |
| hub_kalle | I just wanted to know weather or this is possibly a kayobe issue because then I would put in some time debugging it thouroughly | 12:33 |
| sean-k-mooney | hub_kalle: while you might loose some feature if you continue have issues with idrac you could try the geneirc ipmi driver too | 12:33 |
| hub_kalle | good idea | 12:34 |
| hub_kalle | even though I'm not convinced this is a kayobe/ironic issue at all | 12:35 |
| hub_kalle | wouldn't be the first time Dell iDRAC is broken | 12:35 |
| hub_kalle | a by the way, I didn't update iDRAC because, a colleague of mine just bricked a server a few weeks ago by updating iDRAC ... | 12:36 |
| sean-k-mooney | oh lovely | 12:37 |
| sean-k-mooney | that is always fun | 12:37 |
| hub_kalle | ... when we called DELL about it the technician just said: "yeah, that version is broken" | 12:37 |
| sean-k-mooney | ya updating the bmc is like updating the bios | 12:37 |
| hub_kalle | ... the version that is recommended on their webpage :) | 12:37 |
| sean-k-mooney | you only do it when you have no other option | 12:37 |
| hub_kalle | exactly | 12:37 |
| priteau | sean-k-mooney: if hub_kalle is using the default configuration of kayobe, he's most likely using 1) whole disk images 2) local boot (which doesn't prevent provisioning via pxe boot) 3) ipmi driver | 12:44 |
| priteau | Does the internal TLS feature support encrypting database traffic too? | 12:49 |
| hub_kalle | priteau and sean-k-mooney: that is indeed what i do right now - all my settings in ironic where done by the bifrost auto discovery feature and so far i didn't try any iDRAC Settings in kayobe. wierdly enough the auto discovery had no PXE issue | 12:52 |
| priteau | hub_kalle: you can try the following: log into the bifrost_deploy container, change /etc/ironic/ironic.conf to set debug = True, restart ironic-conductor, tail the logs to check the ipmitool commands being run while provisioning | 12:53 |
| priteau | Then you can run the same commands manually to see what happens | 12:53 |
| hub_kalle | cool thanks i will do that | 12:53 |
| priteau | And of course check the node's console for any messages | 12:53 |
| priteau | Could be that it tries to pxe boot then times out and falls back to disk? | 12:54 |
| hub_kalle | doesnÄ | 12:54 |
| hub_kalle | didn't look like it | 12:54 |
| hub_kalle | the real mystery still is: the boot sequence set in the bios changes by itself | 12:55 |
| hub_kalle | it changes from PXE->Hard Drive to Hard Drive->PXE at some point that I haven't figured out yet | 12:56 |
| opendevreview | Merged openstack/kolla-ansible stable/wallaby: Disable docker's ip-forward when iptables disabled https://review.opendev.org/c/openstack/kolla-ansible/+/796223 | 12:57 |
| hub_kalle | but i will turn on debug in bifrost, if there really is something wrong in kayobe/ironic (which I doubt) it should be visible there | 12:57 |
| priteau | yoctozepto: mgoddard: latest news from c8s, upload of new advanced-virt package (which should bring libvirt 7.4.0) "should happen before wednesday cob us time .. we normally do one stream compose per week (minimally)" | 12:58 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: Switch OPENSTACK_RELEASE back to master https://review.opendev.org/c/openstack/kolla/+/796415 | 13:07 |
| opendevreview | Gaël THEROND proposed openstack/kolla master: Fix missing templating block for kolla-toolbox. https://review.opendev.org/c/openstack/kolla/+/796279 | 13:15 |
| pomac | Hey, we've run in to a bit of a issue when upgrading one of our couds - it is missing vxlan_sys_4789 on all sides - so all vxlan traffic is being dropped. On *all* nodes including L3 nodes... | 13:22 |
| pomac | openvswitch with vxlan on viktoria - any clues of how to add it back - it seems like it has to be a manual step now | 13:22 |
| opendevreview | Mark Goddard proposed openstack/kolla-ansible master: WIP: Log fluentd events in TSV format https://review.opendev.org/c/openstack/kolla-ansible/+/795512 | 13:23 |
| Fl1nt | pomac, vxlan_sys_4789 is supposed to be there, why do you think it's no longer "automated" ? | 13:26 |
| pomac | Fl1nt: well, 92 nodes doesn't agree with you | 13:27 |
| pomac | Fl1nt: as in, i know it's supposed to be there - got that long in the debugging - but it's not | 13:27 |
| Fl1nt | I'm not telling you it's not, I'm rather asking if you think of it because you made an upgrade and it vanished or did you saw something on k-a | 13:28 |
| pomac | Fl1nt: upgrade and full deploy - neither has it :( | 13:28 |
| Fl1nt | weird, looking at neutron. | 13:29 |
| Fl1nt | upgrade or rolling_upgrade | 13:29 |
| pomac | Fl1nt: What i do see is that br-tun has no interfaces and all connection attempts ends up with ICMP noshuch-port | 13:29 |
| pomac | rolling is default, right - so thats what happened - but we have done full deploy on reinstalled nodes as well | 13:30 |
| Fl1nt | right | 13:30 |
| Fl1nt | ok ok, OVS or Linux Bridge based? | 13:31 |
| pomac | OVS | 13:31 |
| mnasiadka | neutron-openvswitch-agent should add it | 13:31 |
| Fl1nt | exactly, except if the DB is corrupted/empty. | 13:32 |
| pomac | humm... Nothing like that from ovsdb-server that i can see | 13:33 |
| pomac | once we got this far (started with suspecting switches and all kinds of things) we also wondered why it hadn't loaded the vxlan kernel module - we're running basically the same config in two other kolla-ansible victoria installs but.... | 13:34 |
| pomac | anything you'd suggest - things are basically down anyway | 13:36 |
| Fl1nt | hold on, which distro are you using pomac | 13:43 |
| pomac | btw, python3-neutronclient 7.2.1 | 13:43 |
| pomac | uhm... no | 13:45 |
| Fl1nt | looks like you've got a config drift, either on your kolla config or within your images build. | 13:46 |
| Fl1nt | do you use an overrided openvswitch-agent.ini file? | 13:47 |
| pomac | Nope, we used to have override ml2 config | 13:47 |
| Fl1nt | may well be a hint, like, the bridge_mappings using improper physnet addr etc | 13:48 |
| pomac | humm... will look at that, but it should be the same as before | 13:49 |
| opendevreview | Gaël THEROND proposed openstack/kolla master: Improve offline build scenario. * Use distribution based dumb-init package. * Add conditional test for source based build. https://review.opendev.org/c/openstack/kolla/+/796480 | 13:55 |
| Fl1nt | How should I name the variable use to specify kolla where to find out packages that are not provided by distribution providers? "tarballs_internal_repo_root_url" ? | 14:02 |
| yoctozepto | pomac: which distro is it? | 14:02 |
| hrw | Fl1nt: you mean packages from out-of-distro/external repos? | 14:09 |
| hrw | Fl1nt: which names you already have? | 14:09 |
| hrw | bbiab | 14:11 |
| yoctozepto | mnasiadka: https://review.opendev.org/c/openstack/kolla-ansible/+/796406 | 14:12 |
| yoctozepto | thx | 14:14 |
| yoctozepto | priteau: I believe you can lift w-1 on https://review.opendev.org/c/openstack/kolla/+/795490 | 14:16 |
| priteau | why? it's still not fixed | 14:18 |
| yoctozepto | priteau: it passes in CI though | 14:18 |
| yoctozepto | so must got fixed | 14:18 |
| priteau | what | 14:18 |
| yoctozepto | look at the results | 14:18 |
| priteau | libvirt-daemon 7.0.0-14.1.el8 | 14:19 |
| Fl1nt | hrw, I mean, currently, our internal mirrors are all hosted under one URL umbrella: https://repository.domain.tld/<distro>|<brand>/ and we use two vars exemple: | 14:19 |
| priteau | Maybe it has a cherry-pick of the fixed merged in 7.40 | 14:19 |
| priteau | 7.4.0 | 14:19 |
| Fl1nt | distribution_internal_repo_root_url: https://repository.domain.tld/ and then tarballs_internal_repo_root_url: https://repository.domain.tld/tarbals/ | 14:20 |
| Fl1nt | we splitted it on two vars because some may well have one server only serving mirrors of distribution packages | 14:21 |
| priteau | yoctozepto: commit message doesn't match at all what's really happening in kolla (it made sense for kayobe), I will need to update it | 14:21 |
| Fl1nt | and another one such as: tarballs.domain.tld | 14:21 |
| yoctozepto | priteau: I stopped following what shappens in centos | 14:21 |
| yoctozepto | priteau: ah, right, the commit message is wrong | 14:22 |
| yoctozepto | all right, you go fix it and cherry pick again | 14:22 |
| yoctozepto | then we approve :-) | 14:22 |
| priteau | the short version is that things should change again by tomorrow so could wait until then to make sure it's all fine | 14:22 |
| priteau | because we will move from libvirt 7.0.0 (from c8 AV) to libvirt 7.4.0 (from c8s AV) | 14:23 |
| yoctozepto | oh boy | 14:25 |
| * yoctozepto 's head's spinning | 14:25 | |
| Fl1nt | RH and Openstack are kinda going crazy with all those tightned release schedules, I mean, enterprise can't just move on with a migration every 3/6 months, it's just too much. | 14:27 |
| Fl1nt | I mean sure, automation. | 14:27 |
| Fl1nt | in a perfect world | 14:27 |
| Fl1nt | but in reality, oh men, there are a loooooot of company relying on shitty OS implementation, specifics and weird constraints. | 14:28 |
| opendevreview | Merged openstack/kolla master: Pin td-agent to 4.0.* to fix missing logs https://review.opendev.org/c/openstack/kolla/+/795086 | 14:28 |
| Fl1nt | If at least you would be guarantee to be able to to a N+3/4 upgrade even not rolling but one by one. | 14:29 |
| Fl1nt | but you're not. | 14:29 |
| Fl1nt | Good luck people still being on Rocky or Stein to upgrade to Wallaby peacefully ^^ | 14:29 |
| Fl1nt | </ rant> | 14:29 |
| opendevreview | Radosław Piliszek proposed openstack/kolla stable/wallaby: Pin td-agent to 4.0.* to fix missing logs https://review.opendev.org/c/openstack/kolla/+/796438 | 14:29 |
| *** rpittau is now known as rpittau|afk | 14:29 | |
| yoctozepto | Fl1nt: if you get RHOSP, then you get fast forward upgrade support goodies | 14:32 |
| opendevreview | Merged openstack/kolla-ansible master: Reno follow up for docker_disable_ip_forward https://review.opendev.org/c/openstack/kolla-ansible/+/796406 | 14:32 |
| opendevreview | Radosław Piliszek proposed openstack/kolla stable/victoria: Pin td-agent to 4.0.* to fix missing logs https://review.opendev.org/c/openstack/kolla/+/796439 | 14:32 |
| Fl1nt | yoctozepto, What do you mean? | 14:33 |
| yoctozepto | Fl1nt: I meant you were ranting about RH, OpenStack, and upgrades | 14:34 |
| yoctozepto | answered your rant | 14:34 |
| Fl1nt | From memories, RHOSP is OoO based, which is... kinda messy and using puppet everywhere so, I'll stick with kolla :D | 14:34 |
| mnasiadka | yoctozepto: You haven't really ran FFU ever, right? :D | 14:34 |
| yoctozepto | but that's what RH supports, it answers your rant :P | 14:34 |
| yoctozepto | mnasiadka: yeah, I have not; I gave up at the initial deploy :-) | 14:35 |
| Fl1nt | I currently have the chance to work on a brand new platform, so I implemented automation from the beginning, but gosh, even with that it can be daunting sometimes. | 14:35 |
| Fl1nt | what really annoy me with all those new stuff is: One of the fundamental of Linux is KISS and well separated concerns, but every single piece of software now a day require its own "director" node but seems like all sysadmins want to host them on the same host for whatever reason I can't stick with. | 14:37 |
| yoctozepto | I guess no need to persuade us | 14:37 |
| Fl1nt | Oh yeah, sure, was just outing my thoughts as I'm just a bit annoyed by some reading of the industry ^^ | 14:38 |
| Fl1nt | Anyway, I need a break :D | 14:38 |
| Fl1nt | see ya all on tomorrow ! | 14:38 |
| opendevreview | Radosław Piliszek proposed openstack/kolla-ansible stable/wallaby: Reno follow up for docker_disable_ip_forward https://review.opendev.org/c/openstack/kolla-ansible/+/796440 | 14:38 |
| yoctozepto | bye Fl1nt | 14:38 |
| Fl1nt | bye yoctozepto o/ | 14:39 |
| yoctozepto | o/ | 14:39 |
| yoctozepto | priteau: I cleaned up the whiteboard - no kayobe nor kolla ansible blocker for wallaby | 14:40 |
| yoctozepto | only waiting for kolla unpin | 14:40 |
| yoctozepto | can you confirm it for kayobe? | 14:41 |
| mnasiadka | yoctozepto: it's something I probably won't forget until retirement :) | 14:42 |
| yoctozepto | lol | 14:42 |
| priteau | yoctozepto: I looked at the source of the libvirt 7.0.0 package and really could not understand why it doesn't have the error since it doesn't have the fix for unknown qemu feature | 14:43 |
| priteau | But actually, if you look at the libvirt logs, the error is still there: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_c46/795490/1/check/kolla-ansible-centos8s-source/c46c30b/primary/logs/kolla/libvirt/libvirtd.txt | 14:44 |
| yoctozepto | priteau: perhaps they lowered the actual criticality | 14:45 |
| yoctozepto | inbetween | 14:45 |
| yoctozepto | the releases | 14:45 |
| priteau | maybe it's not a fatal error anymore, but with 7.4.0 it shouldn't produce this message at all | 14:47 |
| yoctozepto | that would be better | 14:47 |
| yoctozepto | ok, master does not seem to have any blockers | 14:48 |
| priteau | yoctozepto: I am leaving my W-1 for now because we don't want to release a day before libvirt changes again, with the associated potential breakage | 14:58 |
| yoctozepto | priteau: agreed, that makes sense | 15:04 |
| opendevreview | Merged openstack/kolla-ansible stable/wallaby: Reno follow up for docker_disable_ip_forward https://review.opendev.org/c/openstack/kolla-ansible/+/796440 | 15:19 |
| * sean-k-mooney personally wishes rhosp had become kolla-ansible based in osp 13 instead of ooo but ya. | 15:30 | |
| sean-k-mooney | oh the amd-sev-es issue | 15:31 |
| sean-k-mooney | i think we have a downstream bug for that somewhere | 15:31 |
| sean-k-mooney | it was breakign the rdo/downstream ci | 15:31 |
| sean-k-mooney | i suggested pinning the cpu model but i dont know if that fixed it or not | 15:32 |
| sean-k-mooney | so useing cpu_mode=custom and cpu_model=nehelem or simlar | 15:32 |
| sean-k-mooney | yoctozepto: are ye still hitting that | 15:33 |
| sean-k-mooney | ah its been fixed in libvirt good https://github.com/libvirt/libvirt/commit/61d95a1073833ec4323c1ef28e71e913c55aa7b9 | 15:35 |
| opendevreview | Maksim Malchuk proposed openstack/kayobe master: TLS certificates management sync with Kolla-Ansible https://review.opendev.org/c/openstack/kayobe/+/793697 | 15:49 |
| priteau | sean-k-mooney: we are on top of it ;-) | 15:55 |
| jingvar | How can I stop -- while it is in state "wait call-back". (HTTP 400) | 15:58 |
| jingvar | a node can stay in a wait state for a long time or even never time out - funny | 16:01 |
| priteau | jingvar: https://bugs.launchpad.net/ironic/+bug/1655385 | 16:21 |
| *** ricolin_ is now known as ricolin | 16:26 | |
| jingvar | priteau : i can't catch the idea | 16:35 |
| priteau | Comment #8 gives instructions to force the node out of wait call-back state | 16:35 |
| jingvar | <priteau> : but I don't have "ironic" command in the container | 16:41 |
| priteau | this is an old comment. you will have to translate it to `openstack baremetal` commands | 16:41 |
| jingvar | I tried but can't find these options, maybe I'm tired | 16:43 |
| opendevreview | Merged openstack/kolla-ansible stable/wallaby: Drop /sys/fs/cgroup mounts https://review.opendev.org/c/openstack/kolla-ansible/+/796231 | 16:52 |
| opendevreview | Merged openstack/kolla stable/victoria: Pin td-agent to 4.0.* to fix missing logs https://review.opendev.org/c/openstack/kolla/+/796439 | 17:14 |
| *** gfidente is now known as gfidente|afk | 17:14 | |
| opendevreview | Merged openstack/kolla stable/wallaby: Pin td-agent to 4.0.* to fix missing logs https://review.opendev.org/c/openstack/kolla/+/796438 | 17:23 |
| pomac | yoctozepto: centos 8 - built from source though - we're trying different versions | 17:28 |
| pomac | yoctozepto: I was afk - and the containers were built from source =) | 17:31 |
| *** ricolin_ is now known as ricolin | 17:32 | |
| *** gilou_ is now known as Gilou | 17:49 | |
| pomac | is there a way to manually set up and add vxlan_ssy_4789 | 18:01 |
| pomac | sys even | 18:01 |
| pomac | turns out that turning off l2_population causes neutron to create vxlan_sys_4789 | 18:13 |
| pomac | According to the neutron code no interfaces should be created when it's starting - can't find any exception | 18:27 |
| pomac | but then, i'm tired | 18:27 |
| opendevreview | Gaël THEROND proposed openstack/kolla master: Improve offline build scenario. https://review.opendev.org/c/openstack/kolla/+/796480 | 18:31 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: Update previous_release to Wallaby https://review.opendev.org/c/openstack/kolla-ansible/+/796422 | 18:59 |
| ozzzo | where can I find a good document on openstack routers? | 19:43 |
| opendevreview | Piotr Parczewski proposed openstack/kolla-ansible master: Reduce container metrics cardinality https://review.opendev.org/c/openstack/kolla-ansible/+/783887 | 19:44 |
| ozzzo | I read this but it doesn't explain much: https://docs.openstack.org/python-openstackclient/train/cli/command-objects/router.html | 19:44 |
| parallax | ozzzo: here OVN related document: https://docs.openstack.org/neutron/latest/admin/ovn/routing.html | 19:54 |
| ozzzo | ty parallax | 19:55 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!