| mnasiadka | dmsimard[m]: nice, do you have any idea why ara plugin makes our CI runs at least 10 minutes slower?:) | 05:36 |
|---|---|---|
| opendevreview | Merged openstack/kolla-ansible master: senlin: add missing cafile parameter https://review.opendev.org/c/openstack/kolla-ansible/+/894383 | 06:41 |
| opendevreview | Merged openstack/kolla-ansible stable/2023.1: Fixes task name in notify module to the actual task name https://review.opendev.org/c/openstack/kolla-ansible/+/893206 | 06:42 |
| opendevreview | Merged openstack/kolla-ansible stable/zed: Fixes task name in notify module to the actual task name https://review.opendev.org/c/openstack/kolla-ansible/+/893207 | 06:46 |
| opendevreview | Sven Kieske proposed openstack/kolla master: remove unused crmsh EL7 repo https://review.opendev.org/c/openstack/kolla/+/894290 | 07:19 |
| SvenKieske | cores: do you think the podman change should also change current docs, bascially s/docker/{docker,podman}/g https://review.opendev.org/c/openstack/kolla-ansible/+/799229 ? | 07:55 |
| opendevreview | Bartosz Bezak proposed openstack/kolla-ansible master: CI: add q35 hardware machine type to tests https://review.opendev.org/c/openstack/kolla-ansible/+/894199 | 08:29 |
| opendevreview | Bartosz Bezak proposed openstack/kolla-ansible master: Configure Nova libvirt.num_pcie_ports to 16 by default https://review.opendev.org/c/openstack/kolla-ansible/+/768301 | 08:29 |
| frickler | SvenKieske: I think a dedicated patch for the docs might be better. we can then also discuss whether s/docker/container engine/ might be better depending on context | 08:41 |
| SvenKieske | yeah, thought about that as well | 08:47 |
| opendevreview | Sven Kieske proposed openstack/kolla-ansible stable/2023.1: senlin: add missing cafile parameter https://review.opendev.org/c/openstack/kolla-ansible/+/894432 | 10:36 |
| dmsimard[m] | mnasiadka: there is a performance overhead to recording with ara, can't do much about that: https://ara.readthedocs.io/en/latest/troubleshooting.html#improving-playbook-recording-performance | 11:55 |
| dmsimard[m] | the idea is that the overhead should be small enough to make it worth it | 11:56 |
| dmsimard[m] | that overhead is much smaller than it once was and we'd like to reduce it further but there will always be a performance hit because you're asking ansible to record things instead of doing nothing | 11:57 |
| dmsimard[m] | hope that makes sense | 11:57 |
| dmsimard[m] | if you're curious, there's also a couple benchmarks on the blog to measure that performance hit and what options are best | 11:59 |
| dmsimard[m] | for example, running an ara server lets you use the http API client which lets you multi-thread the callback which speeds things up significantly | 12:00 |
| mnasiadka | dmsimard[m]: well, basically the overhead for writing to the db is probably something we can allow for (unless it makes our jobs time out), but the html rendering is a pain - because we would need to upload all those files into OpenDev CI storage | 12:07 |
| mnasiadka | jangutter: ^^ | 12:07 |
| dmsimard[m] | oh, the html generation is documented to be slow and inefficient | 12:09 |
| dmsimard[m] | there is no changing that, it's just how it is | 12:09 |
| jangutter | mnasiadka, dmsimard[m]: agreed - html rendering is really only useful if you need to explicitly pass around links to a run, which is why it's really only useful in extreme situations. | 12:10 |
| dmsimard[m] | once upon a time, exactly because html generation was slow and inefficient, we came up with a middleware to use sqlite databases instead of html: https://superuser.openinfra.dev/articles/scaling-ara-ansible/ | 12:10 |
| dmsimard[m] | this worked back when logs were hosted on logs.openstack.org but it stopped working when openstack-infra migrated to swift for logs | 12:11 |
| dmsimard[m] | this backend still exists and it still works, you just need somewhere to host the server and the sqlite files: https://ara.readthedocs.io/en/latest/distributed-sqlite-backend.html | 12:12 |
| jangutter | on our side, we enable ARA by default in our CI to a backend ara server. Our scale is much smaller, but you do need the occasional db maintenance. logging to sqlite doesn't have this drawback... | 12:14 |
| jangutter | ... but we're happy to run the ARA server (the cost of hosting it is trivial compared to our perceived advantages) | 12:15 |
| dmsimard[m] | yeah, for usage with large scale CI the approach with standalone sqlite databases works best, it's just one small file to upload and no html generation takes place | 12:16 |
| dmsimard[m] | the special database backend expects sqlite files to be on disk, though, so uploading to swift breaks that | 12:18 |
| jangutter | My opinion: ARA html generation is extremely useful for that once-in-a-blue-moon case where you have to show folks a step by step walkthrough of an ansible run. | 12:18 |
| dmsimard[m] | yeah, it is niche but there is a reason for it to exist :) | 12:19 |
| jangutter | ARA sqlite logging - I'm marginally in favour of using sqlite by default, even with the performance hit. This performance hit is reasonably low for the node count in the CI, ... but ... if there's no _persistent_ use of it, it's hard to justify even the minimal overhead. | 12:20 |
| dmsimard[m] | there's some numbers on sqlite vs mysql and postgresql performance here: https://ara.recordsansible.org/blog/2023/02/26/benchmarking-ara-with-different-databases-and-python3.11-for-fun-and-science/ | 12:22 |
| dmsimard[m] | tl;dr sqlite is good enough, though mysql and postgre allow for multi-threading which reduces the recording overhead | 12:23 |
| jangutter | yeah, on the CI scale I think it's probably just into double-digits (with the other non-ara bits that take up time in CI) | 12:23 |
| dmsimard[m] | having mysql wouldn't make html generation and upload faster, though, and then the database isn't portable | 12:24 |
| jangutter | So, to summarize: sqlite is probably still the best option for the scale, and the only question that remains is whether it should be on by default or not? | 12:27 |
| dmsimard[m] | something like that | 12:28 |
| dmsimard[m] | openstack-ansible has similar issues with html generation | 12:28 |
| dmsimard[m] | if openinfra could host a server somewhere with the distributed sqlite backend where you'd upload your sqlite files, that'd truly be best so you wouldnt need to generate html | 12:29 |
| dmsimard[m] | but I think that boat has long sailed | 12:29 |
| dmsimard[m] | I could help with that but I don't have the funding for a server like that 😅 | 12:30 |
| jangutter | yeah, we use S3 ourselves, so like swift, its not a logserver. | 12:31 |
| jangutter | One of the reasons I'm okay with adding it in as a manually triggered event is because it's been broken for so long. If it were essential, folks would have complained. | 12:34 |
| dmsimard[m] | the approach with a comment in the commit message is clever | 12:35 |
| opendevreview | Jake Hutchinson proposed openstack/kolla-ansible master: Ironic parameter rework and default NTP server https://review.opendev.org/c/openstack/kolla-ansible/+/893031 | 12:38 |
| opendevreview | Christian Berendt proposed openstack/kolla-ansible master: ceilometer: handle custom event_pipeline.yaml as template https://review.opendev.org/c/openstack/kolla-ansible/+/894182 | 12:38 |
| opendevreview | Bartosz Bezak proposed openstack/kolla-ansible master: Configure Nova libvirt.num_pcie_ports to 16 by default https://review.opendev.org/c/openstack/kolla-ansible/+/768301 | 12:56 |
| SvenKieske | could I get some reviews for this backport, please? https://review.opendev.org/c/openstack/kolla-ansible/+/894432 | 13:25 |
| opendevreview | Juan Pablo Suazo proposed openstack/kolla-ansible master: Deploy Glance with S3 backend support https://review.opendev.org/c/openstack/kolla-ansible/+/844614 | 13:47 |
| mnasiadka | SvenKieske: do you plan to do other backports than only this one? | 13:51 |
| SvenKieske | mnasiadka: my boss only asked about this one, but I could maybe backport this further - if there are no roadblocks in the way :) | 13:58 |
| mnasiadka | SvenKieske: it would make sense to make backports all the way to yoga - it should have a bug number attached, but I guess we can live with that | 13:59 |
| SvenKieske | it has relnotes at least, so it won't go unnoticed, will create backports | 14:01 |
| opendevreview | Sven Kieske proposed openstack/kolla-ansible stable/zed: senlin: add missing cafile parameter https://review.opendev.org/c/openstack/kolla-ansible/+/894436 | 14:02 |
| SvenKieske | mhm, yoga has merge conflicts | 14:02 |
| opendevreview | Sven Kieske proposed openstack/kolla-ansible stable/yoga: senlin: add missing cafile parameter https://review.opendev.org/c/openstack/kolla-ansible/+/894437 | 14:06 |
| opendevreview | Sven Kieske proposed openstack/kolla-ansible stable/yoga: senlin: add missing cafile parameter https://review.opendev.org/c/openstack/kolla-ansible/+/894437 | 14:15 |
| opendevreview | Bartosz Bezak proposed openstack/kolla-ansible master: WIP: Run Tempest in test-core-openstack https://review.opendev.org/c/openstack/kolla-ansible/+/878826 | 14:20 |
| opendevreview | Juan Pablo Suazo proposed openstack/kolla-ansible master: Deploy Glance and Cinder Backup with S3 backend support https://review.opendev.org/c/openstack/kolla-ansible/+/844614 | 14:55 |
| opendevreview | Juan Pablo Suazo proposed openstack/kolla-ansible master: Deploy Glance and Cinder Backup with S3 backend support https://review.opendev.org/c/openstack/kolla-ansible/+/844614 | 15:01 |
| opendevreview | Merged openstack/kolla-ansible master: Revert "CI: retry smoke tests and instance creation" https://review.opendev.org/c/openstack/kolla-ansible/+/893550 | 15:14 |
| opendevreview | Merged openstack/kolla-ansible master: CI: add block support to validate-all-file.py https://review.opendev.org/c/openstack/kolla-ansible/+/894155 | 15:14 |
| opendevreview | Merged openstack/kolla-ansible master: cleanup: fix cleanup containers https://review.opendev.org/c/openstack/kolla-ansible/+/892330 | 15:22 |
| opendevreview | Merged openstack/kolla-ansible master: CI: add q35 hardware machine type to tests https://review.opendev.org/c/openstack/kolla-ansible/+/894199 | 16:15 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible stable/2023.1: CI: add block support to validate-all-file.py https://review.opendev.org/c/openstack/kolla-ansible/+/894439 | 16:28 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible stable/zed: CI: add block support to validate-all-file.py https://review.opendev.org/c/openstack/kolla-ansible/+/894440 | 16:28 |
| greatgatsby_ | Hello. Our host aggregates seem to get out of sync with our provider aggregates. There seems to be a `nova-manage placement sync_aggregates` command, but we're confused how they're getting out of sync in the first place. Any suggestions of what could be the cause? | 17:10 |
| opendevreview | Juan Pablo Suazo proposed openstack/kolla-ansible master: Deploy Glance and Cinder Backup with S3 backend support https://review.opendev.org/c/openstack/kolla-ansible/+/844614 | 17:28 |
| opendevreview | Bartosz Bezak proposed openstack/kolla-ansible stable/2023.1: CI: add q35 hardware machine type to tests https://review.opendev.org/c/openstack/kolla-ansible/+/894581 | 17:36 |
| opendevreview | Bartosz Bezak proposed openstack/kolla-ansible stable/zed: CI: add q35 hardware machine type to tests https://review.opendev.org/c/openstack/kolla-ansible/+/894582 | 17:36 |
| opendevreview | Bartosz Bezak proposed openstack/kolla-ansible stable/yoga: CI: add q35 hardware machine type to tests https://review.opendev.org/c/openstack/kolla-ansible/+/894583 | 17:37 |
| mnasiadka | greatgatsby_: that’s rather a question to raise on #openstack-nova | 18:36 |
| greatgatsby_ | mnasiadka: thanks | 18:40 |
| mmalchuk | folks, please +W last changes: | 18:41 |
| mmalchuk | https://review.opendev.org/q/I3ca4e10508c26b752412789502ceb917ecb4dbeb | 18:41 |
| mmalchuk | https://review.opendev.org/q/I0401bfc03cd31d72c5a2ae0a111889d5c29a8aa2 | 18:41 |
| mmalchuk | https://review.opendev.org/q/I169e51c0f5c691518ada1837990b5bdd2a3d1481 | 18:41 |
| mmalchuk | https://review.opendev.org/q/Ief8dca4e27197c9576e215cbd960da75f6fdc20c | 18:41 |
| mmalchuk | it strange to see some of them merged randomly | 18:42 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Add support of podman deployment https://review.opendev.org/c/openstack/kolla-ansible/+/799229 | 19:08 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: [DNM] Try to fix aarch64 kolla_logs with relabel:shared https://review.opendev.org/c/openstack/kolla-ansible/+/894572 | 19:29 |
| opendevreview | Jan Gutter proposed openstack/kolla-ansible master: debian: Remove kolla-ansible-debian-zun job https://review.opendev.org/c/openstack/kolla-ansible/+/894573 | 19:37 |
| opendevreview | Merged openstack/kolla master: remove unused crmsh EL7 repo https://review.opendev.org/c/openstack/kolla/+/894290 | 19:43 |
| opendevreview | Juan Pablo Suazo proposed openstack/kolla-ansible master: Deploy Glance and Cinder Backup with S3 backend support https://review.opendev.org/c/openstack/kolla-ansible/+/844614 | 19:53 |
| opendevreview | Christian Berendt proposed openstack/kolla-ansible master: ceilometer: process custom event_pipeline.yaml with merge_yaml https://review.opendev.org/c/openstack/kolla-ansible/+/894182 | 20:01 |
| opendevreview | Merged openstack/kolla master: build: distutils.StrictVersion() is deprecated https://review.opendev.org/c/openstack/kolla/+/886741 | 20:04 |
| opendevreview | Jan Gutter proposed openstack/kolla-ansible master: debian: Remove kolla-ansible-debian-zun job https://review.opendev.org/c/openstack/kolla-ansible/+/894573 | 20:13 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: [DNM] Try to fix aarch64 kolla_logs with relabel:shared https://review.opendev.org/c/openstack/kolla-ansible/+/894572 | 20:32 |
| opendevreview | Jan Gutter proposed openstack/kolla-ansible stable/2023.1: debian: Remove kolla-ansible-debian-zun job https://review.opendev.org/c/openstack/kolla-ansible/+/894593 | 20:32 |
| opendevreview | Jan Gutter proposed openstack/kolla-ansible stable/2023.1: debian: fix the zun job name https://review.opendev.org/c/openstack/kolla-ansible/+/894578 | 20:44 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: [DNM] Try to fix aarch64 kolla_logs with relabel:shared https://review.opendev.org/c/openstack/kolla-ansible/+/894572 | 21:16 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: [DNM] Try to fix aarch64 kolla_logs with relabel:shared https://review.opendev.org/c/openstack/kolla-ansible/+/894572 | 21:54 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: [DNM] Try to fix aarch64 kolla_logs with relabel:shared https://review.opendev.org/c/openstack/kolla-ansible/+/894572 | 22:22 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!