| mnasiadka | yoctozepto: re collections - we would need to move to FQCNs and install required collections on the node running kolla-ansible command - first is more or less easy, second would need an additional command to install from requirements.yml file I guess | 06:30 |
|---|---|---|
| mnasiadka | mgoddard: seems ansible-core 2.11 doesn't like kolla_toolbox module approach - https://538ea43bba4ab74baa0f-8156621505f6355ffb6896f9b15e0b7a.ssl.cf1.rackcdn.com/796758/7/check/kolla-ansible-centos8s-source/fa8182f/primary/logs/ansible/deploy https://github.com/ansible/ansible/blob/9691eb5f2054b2c39a33b3ecb2bdb60c167b590b/lib/ansible/parsing/mod_args.py#L343 | 06:40 |
| yoctozepto | duh and duh | 06:54 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: haproxy: Use haproxy 2.2.x https://review.opendev.org/c/openstack/kolla/+/804513 | 06:57 |
| *** rpittau|afk is now known as rpittau | 07:07 | |
| *** amoralej|off is now known as amoralej | 07:08 | |
| yoctozepto | mnasiadka's mass approval in the gates ;D | 07:46 |
| mnasiadka | yoctozepto: yeah well, I convinced myself to review some stuff ;) | 07:48 |
| *** mgoddard- is now known as mgoddard | 07:58 | |
| mnasiadka | yoctozepto: https://review.opendev.org/c/openstack/kolla-ansible/+/785229 - here's one that needs your review | 07:59 |
| yoctozepto | ok, checking out, looks short | 08:00 |
| yoctozepto | mnasiadka: any reason we call it integration now and not exporter? | 08:02 |
| mnasiadka | yoctozepto: have no clue, we're just scraping fluentd ;) | 08:02 |
| yoctozepto | mnasiadka: ok, I think I understand ;-) | 08:07 |
| yoctozepto | approved with some nits | 08:07 |
| mgoddard | yoctozepto: I wanted s/integration/metrics/, but hey :) | 08:22 |
| mgoddard | yoctozepto: exporter is external to the service being monitored | 08:22 |
| mnasiadka | ,,integrated exporter'' :D | 08:25 |
| opendevreview | Merged openstack/kolla-ansible stable/ussuri: Fix regression in image pull action v2 https://review.opendev.org/c/openstack/kolla-ansible/+/805602 | 08:26 |
| opendevreview | Merged openstack/kolla-ansible stable/ussuri: Add ability to retry image pulling https://review.opendev.org/c/openstack/kolla-ansible/+/805567 | 08:26 |
| ignaziocassano | yoctozepto : Hi, unfortunately my managers are not agree to switch on debian so I cannot test it | 08:33 |
| yoctozepto | ignaziocassano: I see | 08:35 |
| yoctozepto | mgoddard: yeah, that's what I figured finally :-) | 08:35 |
| ignaziocassano | yoctozepto : do you know if I can get help on any irc for the issue on nova libvirt on ubuntu ? | 08:38 |
| yoctozepto | no idea | 08:39 |
| yoctozepto | they would probably send you back to us | 08:39 |
| opendevreview | Merged openstack/kolla-ansible stable/victoria: Add ability to retry image pulling https://review.opendev.org/c/openstack/kolla-ansible/+/805566 | 08:39 |
| opendevreview | Merged openstack/kolla-ansible stable/wallaby: Add ability to retry image pulling https://review.opendev.org/c/openstack/kolla-ansible/+/805565 | 08:39 |
| opendevreview | Merged openstack/kolla-ansible stable/ussuri: Set changed_when to false for group_by tasks https://review.opendev.org/c/openstack/kolla-ansible/+/805133 | 08:39 |
| opendevreview | Merged openstack/kolla-ansible stable/victoria: Set changed_when to false for group_by tasks https://review.opendev.org/c/openstack/kolla-ansible/+/805132 | 08:39 |
| opendevreview | Merged openstack/kolla-ansible stable/wallaby: Use Docker healthchecks for rabbitmq services https://review.opendev.org/c/openstack/kolla-ansible/+/804436 | 08:39 |
| opendevreview | Merged openstack/kolla-ansible stable/wallaby: Add suppress_type_name to stop warnings with ES7 https://review.opendev.org/c/openstack/kolla-ansible/+/804157 | 08:39 |
| opendevreview | Merged openstack/kolla-ansible stable/ussuri: Check config when checking the containers https://review.opendev.org/c/openstack/kolla-ansible/+/803673 | 08:39 |
| opendevreview | Merged openstack/kolla-ansible stable/victoria: Check config when checking the containers https://review.opendev.org/c/openstack/kolla-ansible/+/803672 | 08:39 |
| opendevreview | Merged openstack/kolla-ansible stable/wallaby: Check config when checking the containers https://review.opendev.org/c/openstack/kolla-ansible/+/803671 | 08:40 |
| opendevreview | Merged openstack/kolla-ansible stable/wallaby: [CI] Slim down Masakari job https://review.opendev.org/c/openstack/kolla-ansible/+/802870 | 08:40 |
| ignaziocassano | yoctozepto : So the bug will reimain in unassigned state or do you think someone will investigate ? | 08:40 |
| opendevreview | Verification of a change to openstack/kolla-ansible master failed: Replace auth_uri with www_authenticate_uri https://review.opendev.org/c/openstack/kolla-ansible/+/797625 | 08:40 |
| yoctozepto | ignaziocassano: I am still investigating... to some extent | 08:42 |
| opendevreview | Radosław Piliszek proposed openstack/kolla-ansible master: [CI] Test instance health after upgrade https://review.opendev.org/c/openstack/kolla-ansible/+/806331 | 08:44 |
| yoctozepto | ignaziocassano: ^ for example like that | 08:46 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: haproxy: Use haproxy 2.2.x https://review.opendev.org/c/openstack/kolla/+/804513 | 08:52 |
| opendevreview | Merged openstack/kolla-ansible master: Use api-paste.ini from /etc/neutron https://review.opendev.org/c/openstack/kolla-ansible/+/804288 | 08:55 |
| opendevreview | Merged openstack/kolla-ansible master: Fix Masakari in multi-region deploys https://review.opendev.org/c/openstack/kolla-ansible/+/805451 | 08:55 |
| opendevreview | Verification of a change to openstack/kolla-ansible master failed: Update Manila deploy steps for Wallaby https://review.opendev.org/c/openstack/kolla-ansible/+/802743 | 08:55 |
| opendevreview | Merged openstack/kolla-ansible master: Use Docker healthchecks for keystone-fernet container https://review.opendev.org/c/openstack/kolla-ansible/+/804546 | 09:02 |
| opendevreview | Merged openstack/kolla-ansible master: Use Docker healthchecks for memcached services https://review.opendev.org/c/openstack/kolla-ansible/+/783779 | 09:02 |
| opendevreview | Merged openstack/kolla-ansible master: Use Docker healthchecks for nova-spicehtml5proxy service https://review.opendev.org/c/openstack/kolla-ansible/+/804391 | 09:04 |
| opendevreview | Verification of a change to openstack/kolla-ansible master failed: Use Docker healthchecks for memcached service https://review.opendev.org/c/openstack/kolla-ansible/+/804560 | 09:11 |
| opendevreview | Merged openstack/kolla-ansible master: [CI] Test Swift upgrades https://review.opendev.org/c/openstack/kolla-ansible/+/722798 | 09:31 |
| opendevreview | Radosław Piliszek proposed openstack/kolla-ansible master: [CI] Test instance health after upgrade https://review.opendev.org/c/openstack/kolla-ansible/+/806331 | 09:33 |
| opendevreview | Merged openstack/kolla-ansible master: Support monitoring Fluentd with Prometheus https://review.opendev.org/c/openstack/kolla-ansible/+/785229 | 09:34 |
| opendevreview | Radosław Piliszek proposed openstack/kolla-ansible stable/wallaby: Fix Masakari in multi-region deploys https://review.opendev.org/c/openstack/kolla-ansible/+/806222 | 09:37 |
| opendevreview | Radosław Piliszek proposed openstack/kolla-ansible stable/victoria: Fix Masakari in multi-region deploys https://review.opendev.org/c/openstack/kolla-ansible/+/806223 | 09:37 |
| yoctozepto | mgoddard, mnasiadka: oldie https://review.opendev.org/c/openstack/kolla-ansible/+/804522/1 | 09:39 |
| mgoddard | yoctozepto: ask again on tuesday :D | 09:40 |
| yoctozepto | mgoddard: short patch | 09:40 |
| yoctozepto | mgoddard: is approving disabled till Tuesday? :D | 09:40 |
| mgoddard | yoctozepto: half day, half day of work to do | 09:42 |
| yoctozepto | mgoddard: ok, I will find a twice smaller patch then | 09:44 |
| yoctozepto | mnasiadka: wow, you have approved memcached healthchecks twice today ;d | 09:45 |
| mnasiadka | yoctozepto: that's how it ends when you don't review anything for weeks ;) | 10:13 |
| yoctozepto | ignaziocassano: https://bugs.launchpad.net/kolla-ansible/+bug/1941706 triaged, reproducible in CI (cc mgoddard, mnasiadka - it really seems ubuntu folks just have not reported this issue yet :O ) | 10:45 |
| yoctozepto | debian passes, waiting for centos | 10:46 |
| ignaziocassano | yoctozepto: thanks | 10:50 |
| mnasiadka | yoctozepto: why am I not surprised... | 11:49 |
| yoctozepto | mnasiadka: centos failed too | 12:04 |
| opendevreview | Radosław Piliszek proposed openstack/kolla-ansible master: [CI] Test instance health after upgrade https://review.opendev.org/c/openstack/kolla-ansible/+/806331 | 12:40 |
| blmt | I have a curiosity regarding kolla updates (so minor versions not releases) | 12:42 |
| yoctozepto | please share your curiosity | 12:43 |
| blmt | it seems whenever we attempt one all of our VM root disk goes read only. I assume this a side effect of using NFS as a backing storage | 12:43 |
| yoctozepto | hmm, possible; it certainly does not happen with ceph rbd | 12:43 |
| blmt | netapp driver to be more precise but I don't think it matters in this specific case | 12:43 |
| yoctozepto | the more we know, the better | 12:44 |
| blmt | running victoria with a netapp on tap system mounted via NFS on Ubuntu 20.04 | 12:45 |
| blmt | everything works nicely except for this update issue. | 12:45 |
| yoctozepto | ok, so the vm instances work fine except for their disks going read only | 12:46 |
| opendevreview | Merged openstack/kolla-ansible master: Replace auth_uri with www_authenticate_uri https://review.opendev.org/c/openstack/kolla-ansible/+/797625 | 12:47 |
| blmt | correct | 12:49 |
| mazzy | hey guys | 12:49 |
| mazzy | during the deployment of monasca I get the error that kolla/ubuntu-binary-monasca-api:wallaby image is not available | 12:49 |
| mazzy | Error response from daemon: pull access denied for kolla/ubuntu-binary-monasca-api, repository does not exist or may require 'docker login': denied: requested access to the resource is denied | 12:50 |
| yoctozepto | mazzy: yes, it's not packaged | 12:50 |
| mazzy | aaaah got it | 12:50 |
| mazzy | so i need to build it for myself | 12:50 |
| mazzy | may I ask why is not packaged? | 12:50 |
| yoctozepto | mazzy: I mean it's not packaged for ubuntu; "binary" means to use distribution packages | 12:51 |
| yoctozepto | the answer would be probably that Canonical does not care about Monasca ;-) | 12:51 |
| mazzy | I see | 12:51 |
| yoctozepto | you can use the "source" variant to get monasca | 12:51 |
| yoctozepto | you can switch it for this one project | 12:51 |
| mazzy | makes sense | 12:51 |
| yoctozepto | monasca_install_type: source | 12:52 |
| yoctozepto | in globals | 12:52 |
| mazzy | thanks | 12:53 |
| yoctozepto | mazzy: yw | 12:55 |
| yoctozepto | blmt: https://bugs.launchpad.net/kolla-ansible/+bug/1783978 does this behaviour happen as well? | 12:55 |
| yoctozepto | I'm not sure if we have enough nfs users to report bugs often enough | 12:55 |
| *** amoralej is now known as amoralej|lunch | 13:01 | |
| blmt | it seems so | 13:01 |
| blmt | just restarted the container and the mount count went from 1024 -> 2048 | 13:02 |
| blmt | and I seriously doubt we have so many VM in there.. checking | 13:05 |
| blmt | well at least for the time being we can report bugs on that | 13:07 |
| blmt | hopefully we'll be switching to RBD soon | 13:07 |
| blmt | a bit off-topic yoctozepto: any experience with EC on RBD? | 13:08 |
| mazzy | I was wondering if there is any interest in supporting any other OS. I wanted to try to have Flatcar running Kolla deployment. | 13:17 |
| mazzy | at the end Kolla is fully containerized in Docker which is already a big thing. Do you find any big issues in supporting an OS like Flatcar? | 13:18 |
| *** rpittau is now known as rpittau|afk | 13:42 | |
| yoctozepto | blmt: no; last time I checked it was discouraged | 13:54 |
| yoctozepto | mazzy: no idea; you mean as host, right? we still need some distro inside the containers though | 13:55 |
| mazzy | yes i mean as host. yeah I agree to have distro inside the containers | 13:55 |
| mazzy | i would need to give a try and see if we can have flatcar running kolla | 13:56 |
| yoctozepto | blmt: coudl you try the workaround by mouza8 that's posted there? | 13:57 |
| yoctozepto | mazzy: that would be doable as long as ansible supports it | 13:57 |
| yoctozepto | also do note there might be some assumptions in kolla ansible that host_distro == image_distro | 13:57 |
| mazzy | yeah indeed i wanted to ask right this. | 13:58 |
| yoctozepto | but we could probably improve this if it's not an overkill to do | 13:58 |
| mazzy | i believe there are switches around that select a specific ansible path accortding the distro var in globals | 13:58 |
| yoctozepto | yeah, that is one thing | 13:58 |
| mazzy | do you have anything else in mind? | 13:59 |
| yoctozepto | but the other is the general assumption I wrote above that will now be invalidated | 13:59 |
| yoctozepto | no, I think it has the same root - checking host distro for image internals | 13:59 |
| yoctozepto | or really vice versa | 13:59 |
| yoctozepto | tbh, just give it a try and let us know :-) | 13:59 |
| mazzy | sure I will | 14:01 |
| opendevreview | Piotr Parczewski proposed openstack/kolla master: Fix Elasticsearch Curator https://review.opendev.org/c/openstack/kolla/+/805990 | 14:02 |
| *** amoralej|lunch is now known as amoralej | 14:08 | |
| blmt | yoctozepto I'll look into the patch | 14:12 |
| opendevreview | Piotr Parczewski proposed openstack/kolla master: Fix Elasticsearch Curator https://review.opendev.org/c/openstack/kolla/+/805990 | 14:51 |
| mazzy | ping yoctozepto so the first block that I met running kolla is that it cannot compile netifaces because ofc course flatcar lacks of the building tools | 14:59 |
| yoctozepto | mazzy: compile netifaces on target? | 15:00 |
| mazzy | yeah I was trying that but I've realized that it is no sense to even force the target to be the deployer | 15:01 |
| yoctozepto | :-) | 15:01 |
| mazzy | so was just a try | 15:02 |
| mazzy | very surprised the bootstrap target command went green w/o issues... | 15:47 |
| mazzy | let's try the next | 15:47 |
| mazzy | and after a tiny change also the prechecks went in successfully | 16:10 |
| mazzy | one issue I get during the deploy that i've solved was the following error: | 16:48 |
| mazzy | ``` | 16:48 |
| mazzy | AnsibleError: An unhandled exception occurred while templating '{{ [ansible_facts.processor_vcpus, 5]|min }}'. Error was a <class 'AttributeError'>, original message: module 'builtins' has no attribute 'get'" | 16:48 |
| yoctozepto | it did not discover the amount of cpus? | 16:53 |
| mazzy | i can see in the facts the processor_vcpus var | 16:54 |
| mazzy | so i do not think is that | 16:54 |
| yoctozepto | ok | 16:54 |
| yoctozepto | perhaps the python version on target is somehow unsupported properly | 16:55 |
| mazzy | i get from the fact this | 16:55 |
| mazzy | "ansible_processor_vcpus": 8, | 16:55 |
| yoctozepto | yeah, this is correct; the error message is quite weird | 16:55 |
| mazzy | yeah I suspected as well the verrsion on the target | 16:55 |
| yoctozepto | as if python failed in internals | 16:55 |
| mazzy | which is the latest pypy | 16:55 |
| yoctozepto | pypy? | 16:55 |
| mazzy | yes | 16:56 |
| yoctozepto | use some cpython; pypy might have issues | 16:56 |
| yoctozepto | not sure if ansible tests with pypy... | 16:56 |
| mazzy | yeah i thought as well | 16:56 |
| mazzy | but the good thing is that i had onl ythat issue | 16:56 |
| mazzy | i'm almost done to put up an entire fleet of controlplane machines | 16:57 |
| mazzy | and no errors | 16:57 |
| mazzy | of course I would need to tests and so on... but it looks promising | 16:57 |
| yoctozepto | mhm | 16:57 |
| *** amoralej is now known as amoralej|off | 17:11 | |
| mazzy | i've run some tests and the installation works perfectly | 17:51 |
| mazzy | on Flatcar latest version | 17:51 |
| yoctozepto | great | 17:52 |
| mazzy | would be ok to wire a PR? | 17:52 |
| yoctozepto | yeah, but it's called a "change" in gerrit's terminology | 17:53 |
| yoctozepto | https://docs.openstack.org/kolla-ansible/latest/contributor/contributing.html | 17:54 |
| mazzy | should we put it here? https://review.opendev.org/q/status:open+project:openstack/kolla-ansible | 17:54 |
| yoctozepto | yeah | 17:54 |
| mazzy | i havne't seen under tests if you run tests against different distros | 17:56 |
| mazzy | it's not clear to me where if you have e2e for the different distros | 18:10 |
| mazzy | so in my case the change that I would like to add how will be tested? do you have an env where to spin tests? | 18:11 |
| yoctozepto | yeah, we use zuul as our CI; we have tests per each main distro | 18:12 |
| mazzy | could you point me there? | 18:12 |
| yoctozepto | adding flatcar would need adding its images to opendev first though | 18:12 |
| mazzy | i haven't seen them | 18:12 |
| yoctozepto | yeah, check out .zuul.d/ | 18:12 |
| mazzy | maybe i'm looking in the wrong place | 18:12 |
| mazzy | ooh got it | 18:12 |
| yoctozepto | for flatcar nodeset/images, best ask in #opendev (another channel on this network) | 18:13 |
| yoctozepto | the folks there should be able to help you | 18:13 |
| yoctozepto | for now, I am signing off | 18:13 |
| yoctozepto | good luck | 18:13 |
| mazzy | cool gotcha | 18:13 |
| mazzy | thanks | 18:13 |
| * yoctozepto off | 18:13 | |
| opendevreview | Merged openstack/kolla master: Remove Prometheus v1 image https://review.opendev.org/c/openstack/kolla/+/803636 | 18:52 |
| opendevreview | Verification of a change to openstack/kolla-ansible master failed: CI: Don't generate certificates before upgrade https://review.opendev.org/c/openstack/kolla-ansible/+/792793 | 19:23 |
| opendevreview | Merged openstack/kolla-ansible master: CI: Reduce neutron RPC service workers to 1 https://review.opendev.org/c/openstack/kolla-ansible/+/777077 | 19:28 |
| opendevreview | James Kirsch proposed openstack/kolla-ansible master: Support for keystone scoped authorization https://review.opendev.org/c/openstack/kolla-ansible/+/692179 | 19:40 |
| opendevreview | James Kirsch proposed openstack/kolla-ansible master: Support for keystone scoped authorization https://review.opendev.org/c/openstack/kolla-ansible/+/692179 | 20:25 |
| opendevreview | James Kirsch proposed openstack/kolla-ansible master: Support for keystone scoped authorization https://review.opendev.org/c/openstack/kolla-ansible/+/692179 | 21:01 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!