| hrw | simondodsley: I would say that this is a matter of host os configuration | 05:45 |
|---|---|---|
| opendevreview | Maksim Malchuk proposed openstack/kolla-ansible master: Add forgotten release note for 892427 https://review.opendev.org/c/openstack/kolla-ansible/+/893708 | 08:03 |
| mmalchuk | hrw anyway nvme-cli is installed for cineder-volume in I84c7633765749a5e9ced983505ef290c32154ccc | 08:09 |
| mmalchuk | good morninng Kolla! o/ | 08:10 |
| opendevreview | Maksim Malchuk proposed openstack/kolla-ansible master: Add forgotten release note for 892427 https://review.opendev.org/c/openstack/kolla-ansible/+/893708 | 08:14 |
| kevko | morning | 08:18 |
| kevko | where is docker/podman installed in tests ? | 08:24 |
| mmalchuk | what you are talking about? what is tests? | 08:33 |
| *** ralonsoh_ is now known as ralonsoh | 08:41 | |
| hrw | mmalchuk: because it needs it | 08:45 |
| kevko | mmalchuk in ci | 08:51 |
| mmalchuk | kevko maybe in infra jobs? | 08:52 |
| kevko | because i wanted to debug podman ..but then i found that if you have container_engine in whole patchset ..it's also building via podman :D | 08:52 |
| kevko | but podman support is not merged yet | 08:52 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: [DNM] Just test something https://review.opendev.org/c/openstack/kolla-ansible/+/893609 | 08:57 |
| bbezak | kevko, frickler: Hi, got a sec to review simple change? https://review.opendev.org/c/openstack/kolla-ansible/+/893453 | 09:17 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: ovn: Improve clustering https://review.opendev.org/c/openstack/kolla-ansible/+/868929 | 09:18 |
| hrw | bbezak: done | 09:26 |
| mmalchuk | please review and merge: https://review.opendev.org/c/openstack/kolla-ansible/+/893708 | 09:29 |
| bbezak | cheers hrw! | 09:30 |
| SvenKieske | does anybody in here now where the git source for the "download-logs.sh" file on the zuul artifacts tab is hosted? I'd like to rewrite it :) | 09:35 |
| SvenKieske | found it: https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/local-log-download/templates/download-logs.sh.j2 | 09:40 |
| jangutter | SvenKieske: that one's so old that all bugs are features now! | 09:48 |
| SvenKieske | I guess we could easily save some working hours by rewriting it. I don't want to know how many devs are waiting for those sequential downloads, also this mix of bash and python in the same file is rather disturbing :D | 09:50 |
| SvenKieske | under the precondition that the infrastructure can handle the parallel download of these logs..oO | 09:50 |
| jangutter | SvenKieske: sequential downloads have the benefit of not DoS-ing the storage providers. Agreed that pythonic bash is next-level weird. Also, won't run on the default bash that macos is doomed with... | 09:53 |
| * SvenKieske needs more caffeine before conducting reviews.. | 10:04 | |
| * jangutter needs more caffeine before being allowed to access the Internet. | 10:07 | |
| * mmalchuk simple need more caffeine | 10:12 | |
| opendevreview | Verification of a change to openstack/kolla-ansible stable/zed failed: Added precheck for OpenSearch migration https://review.opendev.org/c/openstack/kolla-ansible/+/893453 | 10:33 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: [DNM] Just test something https://review.opendev.org/c/openstack/kolla-ansible/+/893609 | 10:34 |
| opendevreview | Jan Gutter proposed openstack/kolla-ansible master: WIP debug kolla-ansible-ubuntu-upgrade-cephadm https://review.opendev.org/c/openstack/kolla-ansible/+/890822 | 10:43 |
| * hrw needs to power on espresso machine | 10:52 | |
| jangutter | hrw: if your espresso machine is keeping you awake, https://review.opendev.org/c/openstack/kolla-ansible/+/891097 can put you right back to sleep! | 10:54 |
| hrw | jangutter: first would need to know what ara is | 10:57 |
| jangutter | hrw: example of an html-rendered upgrade run: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_a7a/890822/9/check/kolla-ansible-ubuntu-upgrade-cephadm/a7acbb4/primary/ara-report/ara-html/ | 10:58 |
| jangutter | hrw: and apologies, ARA is one of those recursive acronym thingies. ( the above link was from the zuul build here: https://zuul.opendev.org/t/openstack/build/a7acbb4d8a8d4d5184f726fc9ef58f39/logs ) | 10:59 |
| mmalchuk | hrw ara reports like mind blown | 11:00 |
| hrw | I see | 11:01 |
| jangutter | the user interface is decidedly non-obvious, but the amount of tracing it enables is extensive. | 11:04 |
| hrw | +1 and comments | 11:04 |
| jangutter | hrw, thanks! | 11:04 |
| hrw | +1 as 'I like it but [here come comments]" | 11:04 |
| opendevreview | Merged openstack/kolla-ansible stable/2023.1: debian: Add Bookworm Host OS support https://review.opendev.org/c/openstack/kolla-ansible/+/892744 | 11:11 |
| opendevreview | Merged openstack/kolla-ansible master: Add forgotten release note for 892427 https://review.opendev.org/c/openstack/kolla-ansible/+/893708 | 11:11 |
| mmalchuk | anyone can restore from abandoned state https://review.opendev.org/q/Ief8dca4e27197c9576e215cbd960da75f6fdc20c ? | 11:29 |
| mmalchuk | Erik Olof Gunnar Andersson is here? | 11:29 |
| frickler | mmalchuk: nope, eandersson in #openstack-dns | 11:34 |
| mmalchuk | thanks | 11:35 |
| mmalchuk | will send pm him | 11:36 |
| opendevreview | Dr. Jens Harbott proposed openstack/kolla master: Use latest tagged version from gnocchi again https://review.opendev.org/c/openstack/kolla/+/893722 | 11:37 |
| frickler | mmalchuk: fyi ^^ | 11:38 |
| mmalchuk | frickler too quick) | 11:38 |
| frickler | why? | 11:38 |
| mmalchuk | ok will abandon my backports | 11:39 |
| mmalchuk | because 4.6 needed in all stable branches, there is no fix old setuptools in 4.4 and 4.5 | 11:39 |
| mmalchuk | https://review.opendev.org/q/I3ca4e10508c26b752412789502ceb917ecb4dbeb | 11:40 |
| mmalchuk | abandoned | 11:45 |
| mmalchuk | folks, please merge trivial change: https://review.opendev.org/q/Id8bf642a57b54ecafe0ee6981e044b1aba872ef7 | 11:48 |
| frickler | mmalchuk: iiuc there's 4.4.4 and 4.5.1 with the fix, too, I guess we want to use that for stable branches? | 11:48 |
| mmalchuk | frickler they tagged before the fix afaik | 11:49 |
| mmalchuk | we need b52f7414d595e8d765fde50992f2776a77b98de6 | 11:50 |
| mmalchuk | lets see | 11:50 |
| frickler | https://github.com/gnocchixyz/gnocchi/releases/tag/4.5.1 | 11:50 |
| frickler | if that doesn't work, we can ping tobias to add what's still missing | 11:51 |
| mmalchuk | b52f7414d595e8d765fde50992f2776a77b98de6 exists only in master not in stable/4.4 or stable/4.6 | 11:53 |
| mmalchuk | b52f7414d595e8d765fde50992f2776a77b98de6 exists only in master not in stable/4.4 or stable/4.5 | 11:53 |
| mmalchuk | frickler https://paste.openstack.org/show/821475/ | 11:54 |
| mmalchuk | ops... youre right | 11:57 |
| mmalchuk | the hash were changed | 11:57 |
| mmalchuk | we need 4.5.1 and 4.4.4 | 11:57 |
| mmalchuk | anywey backports not needed, we need new changes in each stable branch | 11:58 |
| mmalchuk | wil restore and change backports | 12:01 |
| opendevreview | Maksim Malchuk proposed openstack/kolla stable/2023.1: Revert "Disable build of Gnocchi containers" https://review.opendev.org/c/openstack/kolla/+/890538 | 12:04 |
| opendevreview | Maksim Malchuk proposed openstack/kolla stable/2023.1: Revert "Disable build of Gnocchi containers" https://review.opendev.org/c/openstack/kolla/+/890538 | 12:04 |
| opendevreview | Maksim Malchuk proposed openstack/kolla stable/zed: Revert "Disable build of Gnocchi containers" https://review.opendev.org/c/openstack/kolla/+/890539 | 12:07 |
| opendevreview | Maksim Malchuk proposed openstack/kolla stable/yoga: Revert "Disable build of Gnocchi containers" https://review.opendev.org/c/openstack/kolla/+/890540 | 12:10 |
| opendevreview | Maksim Malchuk proposed openstack/kolla stable/xena: Revert "Disable build of Gnocchi containers" https://review.opendev.org/c/openstack/kolla/+/890541 | 12:11 |
| mmalchuk | done | 12:13 |
| mmalchuk | frickler thanks | 12:13 |
| opendevreview | Verification of a change to openstack/kolla-ansible stable/zed failed: Added precheck for OpenSearch migration https://review.opendev.org/c/openstack/kolla-ansible/+/893453 | 12:14 |
| opendevreview | Maksim Malchuk proposed openstack/kolla-ansible stable/2023.1: Fix designate-producers not running properly https://review.opendev.org/c/openstack/kolla-ansible/+/893398 | 12:16 |
| opendevreview | Michal Arbet proposed openstack/kolla master: [DNM] Test kolla-toolbox permissions https://review.opendev.org/c/openstack/kolla/+/893605 | 12:18 |
| opendevreview | Maksim Malchuk proposed openstack/kolla-ansible stable/2023.1: Fix designate-producers not running properly https://review.opendev.org/c/openstack/kolla-ansible/+/893398 | 12:18 |
| opendevreview | Maksim Malchuk proposed openstack/kolla-ansible stable/zed: Fix designate-producers not running properly https://review.opendev.org/c/openstack/kolla-ansible/+/893399 | 12:18 |
| opendevreview | Maksim Malchuk proposed openstack/kolla-ansible stable/yoga: Fix designate-producers not running properly https://review.opendev.org/c/openstack/kolla-ansible/+/893400 | 12:20 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: [DNM] Just test something https://review.opendev.org/c/openstack/kolla-ansible/+/893609 | 12:20 |
| opendevreview | Maksim Malchuk proposed openstack/kolla-ansible stable/xena: Fix designate-producers not running properly https://review.opendev.org/c/openstack/kolla-ansible/+/893441 | 12:21 |
| opendevreview | Michal Arbet proposed openstack/kolla master: [DNM] Test kolla-toolbox permissions https://review.opendev.org/c/openstack/kolla/+/893605 | 13:15 |
| SvenKieske | mnasiadka: is the newest non-voting rocky8-ovn failure in https://review.opendev.org/c/openstack/kolla-ansible/+/868929 worthwhile to investigate? at first glance it seems like an unrelated(?) issue during ssh connection. | 13:45 |
| mnasiadka | Failed to download metadata for repo 'baseos': repomd.xml parser error: Element <repomd> was not found - Bad repomd file | 13:46 |
| mnasiadka | seems download.docker.com has some problem with rpm repo | 13:46 |
| mnasiadka | SvenKieske: ^^ | 13:46 |
| SvenKieske | isn't that an error because of the previous ssh error? but I don't know | 13:46 |
| mnasiadka | well, that's for appstream | 13:46 |
| mnasiadka | interesting | 13:46 |
| mnasiadka | NeilHanlon: is there any known problem with rocky linux mirrors? :) | 13:48 |
| frickler | I think I heard something over in the osa channel | 13:49 |
| frickler | 11:50 < jrosser> hmm also a notification in our internal slack that rocky repos are a bit broken | 13:50 |
| frickler | why do they do that to me? I've been promoting rocky as stable alternative to centos alpha stream and now this | 13:52 |
| SvenKieske | :D | 13:52 |
| SvenKieske | we still live in the dark age of information technology, most of the stuff is broken, most of the time. | 13:53 |
| kevko | just use debian :) | 13:56 |
| mmalchuk | lol | 13:57 |
| mmalchuk | packages.grafana.com also broken | 13:57 |
| kevko | russian attack :D | 13:58 |
| NeilHanlon | mnasiadka: yeah have been having infra issues overnight :\ | 13:58 |
| mmalchuk | russian submarine atack))) | 13:58 |
| NeilHanlon | I (think) things are stable now | 13:58 |
| kevko | our banks in cz are affected by russian attack second day several days :/ | 13:58 |
| NeilHanlon | :( | 13:58 |
| NeilHanlon | that's shitty | 13:58 |
| kevko | yeah :( | 13:59 |
| NeilHanlon | frickler: i'll have you know I did it on purpose just to spite you /s | 13:59 |
| SvenKieske | has rocky linux not some kind of status page? | 14:00 |
| mmalchuk | no matter who is attacker | 14:00 |
| SvenKieske | I find virtually no information about this stuff online | 14:00 |
| SvenKieske | I remember volunteering sysadmin/devops work for rocky when they where new on the scene. not a single reply via their community channels | 14:00 |
| mmalchuk | fake news) | 14:00 |
| NeilHanlon | weird that our status page is absent from search engines, though | 14:01 |
| NeilHanlon | https://status.rockylinux.org/ | 14:01 |
| mnasiadka | it says all green :) | 14:01 |
| NeilHanlon | it is, now! :P | 14:01 |
| SvenKieske | the last incident on that page was 5.9.2023 ... is that run by some PR department? | 14:01 |
| NeilHanlon | ... | 14:02 |
| NeilHanlon | 5.9.2023 is today | 14:02 |
| SvenKieske | ah wait, 5.9. is today | 14:02 |
| SvenKieske | where is my caffeine again? | 14:02 |
| SvenKieske | sheesh, so basically everything was down? | 14:02 |
| SvenKieske | looks like flapping to me though: https://status.rockylinux.org/787820740#logs | 14:03 |
| NeilHanlon | Yeah, our haproxy tier was having a time | 14:03 |
| SvenKieske | hopefully not k-a related xD | 14:04 |
| SvenKieske | NeilHanlon: re: finding the status page via google: it might help if rockylinux.org/robots.txt wouldn't 404 | 14:07 |
| SvenKieske | googline "statuspage rockylinux" has no relevant results on the first page for me, also not via "status rockylinux" | 14:07 |
| * SvenKieske needs really to get some caffeine now.. | 14:08 | |
| NeilHanlon | I've given up trying to appease search engines, honestly. | 14:09 |
| SvenKieske | you have set "meta name='robots' content='noindex'" so it's no surprise a search engine won't find it | 14:11 |
| SvenKieske | But I also hate webdev with a passion, but occasionally I need to debug this crap. | 14:11 |
| NeilHanlon | yeah, i guess that'd be something uptimerobot needs to address/fix | 14:11 |
| SvenKieske | no idea who or what _that_ is supposed to be, but sounds about right :D | 14:12 |
| NeilHanlon | that statuspage is hosted by https://uptimerobot.com/ | 14:12 |
| SvenKieske | ah, a saas for uptime sites | 14:12 |
| NeilHanlon | it'll be gone by EOY, so i'm not overly worried | 14:13 |
| SvenKieske | here's hoping they are not running rocky linux, because you might have a circular dependency then :D | 14:13 |
| SvenKieske | wasn't that amazon who did host their statuspage on aws? :D | 14:13 |
| SvenKieske | ah they (uptimerobot) are running on cloudflare | 14:14 |
| mmalchuk | amazon behind the cloudflare afaik | 14:16 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: [DNM] Just test something https://review.opendev.org/c/openstack/kolla-ansible/+/893609 | 14:43 |
| opendevreview | Verification of a change to openstack/kolla-ansible stable/zed failed: Added precheck for OpenSearch migration https://review.opendev.org/c/openstack/kolla-ansible/+/893453 | 14:50 |
| SvenKieske | github is also being rockstable today: https://www.githubstatus.com/incidents/smdz34v7j8q0 | 16:33 |
| Karni | May you please this bandwith tests: https://i.imgur.com/SmMXan9.png | 16:51 |
| Karni | Do you have any suggestion for our bandwith fluctuation over different servers? | 16:51 |
| Karni | Note: in our setup, controlers and network nodes are in virtual machines hosted on a ESXi cluster. | 16:52 |
| Karni | Test Design: VM1 <---> ASAv Cisco Firewall <---> VM2 | 16:52 |
| Karni | In column7&8 there are to number per cell. The upper number is the bit rate in Mb/s for the traffic that goes from VM1 to VM2 buth through the ASAv that is the gateway for VMs. | 16:52 |
| Karni | In that picture, the two last column are the result in MultiQueue-enabled and MultiQueu-disabled servers. | 16:53 |
| Karni | Column4: The compute hosting ASAv as the monitoring tool to cature bit rates. | 16:54 |
| SvenKieske | Karni: well I'm not sure your setup has anything to do with kolla-ansible? Maybe I'm misunderstanding you? this room is not about esxi|vmware | 16:54 |
| Karni | Column5: The compute number where VM1 is located | 16:54 |
| Karni | Column6: the other VM | 16:54 |
| Karni | the other compute* | 16:54 |
| Karni | It's an openstack deployed by Kolla; I don't know any other active community channel. | 16:55 |
| SvenKieske | if this is an openstack setup the first things to know would be which openstack release this is, and which network components of openstack are in use: ovn, ovs, dvr, floating ips, LBs|octavia? | 16:56 |
| SvenKieske | if this is a kolla setup you are in the correct room :) but it's maybe getting a little late for most people contributing here (end of workday in europe) :) | 16:56 |
| Karni | I mean, we have deployed the controllers on virtual machines. Can it be a bottleneck for our tenant network issue? The network bandwith vary per compute from 100MB/s to 5000Mb/s! | 16:56 |
| SvenKieske | Karni: without knowing more details that _could_ be a problem, if your control plane also runs centralized network nodes? | 16:57 |
| Karni | Someone told if we enable the multiqueue feature on images, it may help. that picture is the result for mq-enabled and no-mq servers. | 16:58 |
| Karni | network nodes are on separe nodes. | 16:58 |
| Karni | I'm also going to enable DPDK on servers to see if it can help | 17:00 |
| SvenKieske | what is the physical limit of these networks? have you performed iperf tests between each physical node to test if the physical network really can perform as fast you need to? | 17:01 |
| SvenKieske | it's a very basic question, but many people have firmware issues or settings issues in their network and never tested if the 100gbit switch really works the way it's advertised :) | 17:02 |
| Karni | On all physical computes nodes, 10G-Nic cards are installed | 17:05 |
| Karni | Yes, there's no issue between physical servers. They can reach to ~10Gb/s. | 17:10 |
| Karni | > if the 100gbit switch really works the way it's advertised: I will ask the datacenter team to be sure. Thanks for the hint | 17:12 |
| SvenKieske | np; I'm also afk for now, maybe back later again. | 17:14 |
| opendevreview | Maksim Malchuk proposed openstack/kolla-ansible stable/xena: CI: set centos8s and ubuntu upgrade jobs non-voting https://review.opendev.org/c/openstack/kolla-ansible/+/893796 | 17:44 |
| mmalchuk | mnasiadka ubuntu upgrade job failed due to old constrains in wallaby, since it eol lets fix this on CI https://review.opendev.org/c/openstack/kolla-ansible/+/893796 | 17:46 |
| mmalchuk | https://zuul.opendev.org/t/openstack/build/e227d7dd485a41219d824c7a82654f52 | 17:48 |
| mmalchuk | an example | 17:48 |
| opendevreview | Merged openstack/kolla-ansible master: Prevent libvirtd reload when only generating config https://review.opendev.org/c/openstack/kolla-ansible/+/876991 | 18:21 |
| opendevreview | Maksim Malchuk proposed openstack/kolla-ansible stable/xena: Use better default bind address for ironic-tftp https://review.opendev.org/c/openstack/kolla-ansible/+/893422 | 19:43 |
| spatel | mnasiadka Around? | 19:45 |
| spatel | I have question related Octivia deployment in production. I have dedicated vlan for lb-mgmt. As per this doc all I need just specify octavia_network_interface: bond0.100 | 19:46 |
| spatel | https://docs.openstack.org/kolla-ansible/latest/reference/networking/octavia.html | 19:46 |
| spatel | does kolla-ansible going to attach my bond0.100 interface to ovs ? | 19:47 |
| spatel | or I have to do by hand? | 19:47 |
| opendevreview | Verification of a change to openstack/kolla-ansible stable/zed failed: Added precheck for OpenSearch migration https://review.opendev.org/c/openstack/kolla-ansible/+/893453 | 22:05 |
| opendevreview | Merged openstack/kolla-ansible stable/zed: Added precheck for OpenSearch migration https://review.opendev.org/c/openstack/kolla-ansible/+/893453 | 23:22 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
