| opendevreview | Boxiang Zhu proposed openstack/kolla master: Add skyline Dockerfile https://review.opendev.org/c/openstack/kolla/+/810796 | 02:04 |
|---|---|---|
| opendevreview | Boxiang Zhu proposed openstack/kolla-ansible master: Add skyline role https://review.opendev.org/c/openstack/kolla-ansible/+/810566 | 02:07 |
| opendevreview | Merged openstack/kayobe stable/victoria: Fix IPA builder version typo https://review.opendev.org/c/openstack/kayobe/+/812745 | 02:11 |
| opendevreview | Merged openstack/kayobe stable/wallaby: Add support for apt proxy setting https://review.opendev.org/c/openstack/kayobe/+/812698 | 02:11 |
| *** amoralej|off is now known as amoralej | 06:45 | |
| mnasiadka | yoctozepto: interesting, based on the log the cluster is synced, but the variable is still Initialized | 07:42 |
| opendevreview | Mark Goddard proposed openstack/kayobe master: CI: add Infra VM jobs https://review.opendev.org/c/openstack/kayobe/+/813048 | 07:47 |
| OpenstackThunde | Hello Team, I enabled Central Logging and Monasca .... And I ran the deployment but for some reason I cant see the logging on the Horizon dashboard.. Any idea where I could be going wrong ? | 08:04 |
| opendevreview | Mark Goddard proposed openstack/kayobe master: Use merge_configs and merge_yaml to generate Kolla configs https://review.opendev.org/c/openstack/kayobe/+/804068 | 08:08 |
| jeeva | moved from deploying ussuri to wallaby | 08:16 |
| jeeva | but getting these errors now: https://pastebin.com/PhHd4rEA | 08:16 |
| jeeva | testing it with a pull to a node limited | 08:16 |
| jeeva | any ideas ? | 08:17 |
| OpenstackThunde | @opendevreview .... Thanks for the doc but I haven't used Kayobe before .... Is there any clearer guideline I use ? | 08:18 |
| mgoddard | jeeva: use wallaby tag rather than 12.1.0 | 08:19 |
| opendevreview | Mark Goddard proposed openstack/kayobe master: CI: add host configure tests for firewalld https://review.opendev.org/c/openstack/kayobe/+/812299 | 08:20 |
| jeeva | @mgoddard, thanks will try | 08:24 |
| opendevreview | Pierre Riteau proposed openstack/kolla stable/train: Require pip version compatible with Python 2 https://review.opendev.org/c/openstack/kolla/+/813144 | 08:24 |
| opendevreview | Pierre Riteau proposed openstack/kolla stable/train: Require pip version compatible with Python 2 https://review.opendev.org/c/openstack/kolla/+/813144 | 08:24 |
| OpenstackThunde | I am actually running it on Wallby | 08:25 |
| OpenstackThunde | Wallaby | 08:25 |
| jeeva | mgoddard, there is no image with a name of wallaby though | 08:32 |
| mgoddard | oh, you're using your own registry | 08:33 |
| jeeva | openstack_tag: "wallaby" | 08:33 |
| mgoddard | ah | 08:33 |
| jeeva | i'm under the impression kolla will pull it down to our local registry | 08:34 |
| mgoddard | possibly we need to be more clear about the namespace - I think kolla builds with a namespace of kolla, but kolla-ansible uses openstack.kolla | 08:34 |
| mgoddard | jeeva: try setting docker_namespace to kolla | 08:34 |
| mgoddard | yoctozepto, mnasiadka: one to sort out before release ^ | 08:34 |
| jeeva | not following the "try setting docker_namespace to kolla" | 08:35 |
| yoctozepto | jeeva: in kolla ansible config, in globals.yml | 08:54 |
| yoctozepto | docker_namespace: kolla | 08:54 |
| yoctozepto | mgoddard, mnasiadka: good point | 08:54 |
| yoctozepto | mnasiadka: yeah, the log says everything's fine | 08:54 |
| yoctozepto | I am worried there is a bug in MariaDB, arar :< | 08:54 |
| mnasiadka | yoctozepto: maybe that variable is not populated - try dumping all wsrep variables and check the others | 09:00 |
| mnasiadka | mgoddard: there's a patch in Kayobe to follow kolla-ansible, so either we change it in kolla - or update docs | 09:00 |
| mgoddard | probably we should change the default to be consistent | 09:02 |
| mnasiadka | it won't hurt, agree | 09:03 |
| yoctozepto | mnasiadka: yeah, good idea | 09:03 |
| yoctozepto | (about wsrep var dump) | 09:04 |
| opendevreview | Pierre Riteau proposed openstack/kolla stable/train: Use local u-c file to avoid SSL verification issues https://review.opendev.org/c/openstack/kolla/+/813153 | 09:16 |
| jinyuanliu | https://review.opendev.org/c/openstack/kolla/+/793795 | 09:21 |
| jinyuanliu | Please review it for me, if any core has time | 09:22 |
| jinyuanliu | thx | 09:22 |
| opendevreview | Pierre Riteau proposed openstack/kolla stable/train: Use local u-c file to avoid SSL verification issues https://review.opendev.org/c/openstack/kolla/+/813153 | 09:24 |
| opendevreview | Radosław Piliszek proposed openstack/kolla-ansible master: [DNM] Test MariaDB https://review.opendev.org/c/openstack/kolla-ansible/+/812944 | 09:37 |
| opendevreview | Pierre Riteau proposed openstack/kolla master: Add prometheus-msteams image https://review.opendev.org/c/openstack/kolla/+/812678 | 09:59 |
| priteau | Kayobe master is broken | 10:01 |
| priteau | master CI | 10:01 |
| priteau | nova.exception.TooOldComputeService: Current Nova version does not support computes older than Xena but the minimum compute service level in your system is 56 and the oldest supported service level is 57. | 10:01 |
| jeeva | 10.102.17.2%3A5000%2Fopenstack.kolla%2Fubuntu-source-fluentd: Not Found (\"manifest for 10.102.17.2:5000/openstack.kolla/ubuntu-source-fluentd:12.1.0 not found: manifest unknown: manifest unknown\")\\n'"} | 10:10 |
| jeeva | i dont see where it gets the "/openstack.kolla" part from | 10:10 |
| jeeva | sudo docker pull 10.102.17.2:5000/kolla/ubuntu-source-fluentd:12.1.0 works fine when i do it manually | 10:10 |
| jeeva | but deploy adds a "/openstack.kolla/..." in the registry | 10:11 |
| priteau | docker_namespace: "openstack.kolla" | 10:11 |
| mgoddard | priteau: I guess we need to merge the kolla xena release patches as the world has moved on | 10:11 |
| jeeva | #docker_namespace: "kolla" | 10:11 |
| jeeva | its that | 10:11 |
| jeeva | in my globals.yml | 10:11 |
| mgoddard | yoctozepto: time for https://review.opendev.org/q/topic:%22kolla-xena-release%22+(status:open%20OR%20status:merged) ? | 10:12 |
| priteau | mgoddard: Oh, is kolla master still tracking master? | 10:12 |
| mgoddard | yes | 10:12 |
| mgoddard | I guess this is why we have a release schedule... | 10:13 |
| priteau | OK, I thought that was merged | 10:13 |
| jeeva | what should i change the namespace to if the default is "kolla", it must look like this 10.102.17.2:5000/openstack.kolla/ubuntu-source-fluentd:12.1.0 | 10:14 |
| priteau | jeeva: are you using kolla-ansible master branch? | 10:15 |
| jeeva | i changed it to "kolla" | 10:16 |
| jeeva | it went further | 10:16 |
| jeeva | now back to the insecure registry | 10:16 |
| priteau | jeeva: I may be mistaken but it sounds like you're using master branch (which I would avoid now while the release work is still in progress) instead of wallaby | 10:17 |
| priteau | mgoddard: Not sure if it's worth reverting kayobe to use docker hub for CI, by the time the patch will be checked and merged it will be near the end of the afternoon. But if we can merge the kolla patches today, Kayobe CI would be back to green on Monday? | 10:21 |
| mgoddard | priteau: probably best to wait | 10:22 |
| opendevreview | Pierre Riteau proposed openstack/kolla stable/train: CI: Fix multiple issues for stable/train https://review.opendev.org/c/openstack/kolla/+/813082 | 10:36 |
| opendevreview | Michal Arbet proposed openstack/kolla master: Make cron's logrotate schedule configurable https://review.opendev.org/c/openstack/kolla/+/813008 | 11:05 |
| opendevreview | Pierre Riteau proposed openstack/kolla-ansible stable/train: CI: Remove ara integration https://review.opendev.org/c/openstack/kolla-ansible/+/813171 | 11:27 |
| kevko | anyone to advice ? | 11:33 |
| kevko | re-scheduled: invalid argument: could not find capabilities for domaintype=kvm \n | 11:33 |
| jeeva | {"changed": false, "msg": "MariaDB cluster is not synced. Please wait for WSREP sync before proceeding."} -- how long is this WSREP sync ? | 11:33 |
| kevko | victoria debian binary -> control plane upgraded to wallaby | 11:33 |
| kevko | jeeva: how fast are your machines :D | 11:33 |
| jeeva | very fast | 11:33 |
| kevko | deploy one ? | 11:34 |
| jeeva | testbed, DELL R630's | 11:34 |
| kevko | hmm | 11:34 |
| kevko | it should be quite fast | 11:34 |
| jeeva | 48 cores, 328 GB ram per controller | 11:34 |
| jeeva | 384* | 11:34 |
| kevko | version of mariadb ? | 11:34 |
| ignaziocassano | hello, I've just installed openstack kolla wallaby and when I create and empty volume on netapp ontap it works. It fails ig I create a volume from images. cinder-volume logs: https://paste.openstack.org/show/809875/ | 11:35 |
| jeeva | kevko, 10.102.17.2:5000/kolla/ubuntu-source-mariadb-server:12.1.0 | 11:35 |
| kevko | i meant mariadb inside container | 11:36 |
| jeeva | udo docker inspect mariadb | grep 'version' | 11:37 |
| jeeva | "kolla_version": "12.1.0", | 11:37 |
| jeeva | only version related item i see in the container | 11:38 |
| jeeva | WARNING: IPv4 forwarding is disabled. Networking will not work. | 11:43 |
| jeeva | seems maybe because of this | 11:43 |
| kevko | yeah ..probably | 11:47 |
| kevko | yoctozepto: (nova-libvirt)[root@compute1 /]# ls -la /dev/kvm | 11:47 |
| kevko | crw-rw---- 1 root systemd-resolve 10, 232 Oct 8 06:30 /dev/kvm | 11:47 |
| kevko | this is weird right | 11:47 |
| kevko | ? | 11:47 |
| jeeva | doesn't seem like the case, added ipv4 to the sysctl and restarted networking | 11:49 |
| jeeva | "msg": "MariaDB cluster is not synced. Please wait for WSREP sync before proceeding. | 11:50 |
| frickler | iiuc this is what yoctozepto was looking at with https://review.opendev.org/c/openstack/kolla-ansible/+/812944 | 11:52 |
| frickler | doesn't seem to be 100% reproducible though | 11:52 |
| yoctozepto | frickler: yeah, this seems related | 12:02 |
| yoctozepto | I mean, something weird with wsrep after the version bump | 12:02 |
| yoctozepto | and it fails too often in CI | 12:02 |
| yoctozepto | but it's on master here, not wallaby | 12:08 |
| yoctozepto | wallaby is not affected | 12:09 |
| * yoctozepto just read the backlog | 12:09 | |
| ignaziocassano | hello, I've just installed openstack kolla wallaby and when I create and empty volume on netapp ontap it works. It fails ig I create a volume from images. cinder-volume logs: https://paste.openstack.org/show/809875/ . Something is changed on cinder ? | 12:14 |
| opendevreview | Mark Goddard proposed openstack/kayobe master: CI: add host configure tests for firewalld https://review.opendev.org/c/openstack/kayobe/+/812299 | 12:20 |
| yoctozepto | ignaziocassano: is nova able to boot from those images fine? i.e. if you boot from image without creating a new volume | 12:25 |
| ignaziocassano | yoctozepto: II am going to test it. | 12:27 |
| ignaziocassano_ | yoctozepto: yes, ephemeral works | 12:37 |
| kevko | looks like debian-binary-libvirt victoria is broken somehow :/ | 12:41 |
| yoctozepto | kevko: it does not differ from source one and it works for me :-) | 12:42 |
| yoctozepto | what fails for you? | 12:42 |
| yoctozepto | (well, it should not differ at least :D _ | 12:42 |
| kevko | yoctozepto: https://paste.opendev.org/show/809880/ | 12:44 |
| kevko | i've replaced nova-libvirt container with centos-source one and it started to work | 12:44 |
| kevko | what's weird is this | 12:44 |
| kevko | yoctozepto: (nova-libvirt)[root@compute1 /]# ls -la /dev/kvm | 12:44 |
| kevko | <kevko> crw-rw---- 1 root systemd-resolve 10, 232 Oct 8 06:30 /dev/kvm | 12:44 |
| kevko | yoctozepto: wallaby test stack --> | 12:45 |
| kevko | (nova-libvirt)[root@compute0 /]# ls -la /dev/kvm | 12:45 |
| kevko | crw-rw---- 1 root qemu 10, 232 Oct 5 15:15 /dev/kvm | 12:45 |
| yoctozepto | weird indeed | 12:45 |
| kevko | yeah | 12:46 |
| kevko | let me exec into container and check snippets regarding user creation in a package | 12:48 |
| opendevreview | Pierre Riteau proposed openstack/kolla stable/train: Fix multiple issues for stable/train https://review.opendev.org/c/openstack/kolla/+/813082 | 12:49 |
| *** amoralej is now known as amoralej|lunch | 12:52 | |
| opendevreview | Pierre Riteau proposed openstack/kolla-ansible stable/train: CI: Remove ara integration https://review.opendev.org/c/openstack/kolla-ansible/+/813171 | 13:03 |
| opendevreview | Uwe Grawert proposed openstack/kolla-ansible master: Grafana: Run priviliged when copying home dashboard file https://review.opendev.org/c/openstack/kolla-ansible/+/813185 | 13:07 |
| *** amoralej|lunch is now known as amoralej | 13:17 | |
| opendevreview | Verification of a change to openstack/kolla master failed: Update versions for Xena https://review.opendev.org/c/openstack/kolla/+/812729 | 13:22 |
| priteau | Kolla CI is hitting the same upgrade issue as Kayobein https://review.opendev.org/c/openstack/kolla/+/812729 | 13:46 |
| priteau | Will be fixed by https://review.opendev.org/c/openstack/kolla/+/812726? | 13:47 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Add api_workers for each service to defaults https://review.opendev.org/c/openstack/kolla-ansible/+/813193 | 13:47 |
| yoctozepto | mnasiadka, frickler: https://54e727e34e3178aca82f-3ad27eaf65bc806a39bd9f42516bdf66.ssl.cf2.rackcdn.com/812944/2/check/kolla-ansible-ubuntu-source-mariadb/205153c/primary/logs/ansible/deploy all wsrep vars tell the same - the node failed to join | 13:52 |
| yoctozepto | logs do not agree | 13:53 |
| yoctozepto | ah no | 13:55 |
| yoctozepto | they do agree, only need to check the right host ofc | 13:55 |
| yoctozepto | "2021-10-08 13:07:35 2 [Note] WSREP: Server status change connected -> disconnected" | 13:55 |
| yoctozepto | 2021-10-08 13:07:35 2 [Warning] WSREP: 0x564490c95b08 down context(s) not set | 13:55 |
| yoctozepto | 2021-10-08 13:07:35 2 [Warning] WSREP: Sending JOIN failed: -107 (Transport endpoint is not connected). Will retry in new primary component. | 13:55 |
| yoctozepto | 2021-10-08 13:07:35 0 [ERROR] WSREP: IST didn't contain all write sets, expected last: 3 last received: 2 | 13:56 |
| yoctozepto | 2021-10-08 13:07:35 7 [ERROR] WSREP: Receiving IST failed, node restart required: IST receiver reported failure: 71 (Protocol error) | 13:56 |
| yoctozepto | at /home/buildbot/buildbot/build/galera/src/replicator_smm.hpp:pop_front():315. Null event. | 13:56 |
| frickler | yoctozepto: https://54e727e34e3178aca82f-3ad27eaf65bc806a39bd9f42516bdf66.ssl.cf2.rackcdn.com/812944/2/check/kolla-ansible-ubuntu-source-mariadb/205153c/primary/logs/kolla/mariadb/mariadb.txt | 14:01 |
| frickler | seems primary and sec2 in the end get together, just sec1 doesn't | 14:01 |
| frickler | do you want me to hold a job in order to look at things in place? | 14:02 |
| opendevreview | Verification of a change to openstack/kolla master failed: release: Use Xena sources by default https://review.opendev.org/c/openstack/kolla/+/812726 | 14:15 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Add api_workers for each service to defaults https://review.opendev.org/c/openstack/kolla-ansible/+/813193 | 14:20 |
| yoctozepto | frickler: yeah, we could; but I still don't know what to look at; one of the 3 simply fails | 14:32 |
| frickler | yoctozepto: not sure either, the plan is to just look around until something looks suspicous | 14:36 |
| mnasiadka | yoctozepto: well at least we could check if re-bootstrapping the node helps | 14:44 |
| opendevreview | Pierre Riteau proposed openstack/kolla-ansible master: Stop creating unused cron/logrotate directory https://review.opendev.org/c/openstack/kolla-ansible/+/813199 | 14:49 |
| yoctozepto | mnasiadka, frickler: yeah, "node restart required" suggests it may help to restart the service | 14:56 |
| yoctozepto | seems to be a known issue https://jira.mariadb.org/browse/MDEV-22796 | 14:57 |
| yoctozepto | I wonder why it uses ist on the first run | 14:58 |
| yoctozepto | maybe we could make it use SST | 14:58 |
| mnasiadka | https://jira.mariadb.org/browse/MDEV-22797 | 15:00 |
| mnasiadka | this one looks fantastic | 15:00 |
| yoctozepto | 2021-10-08 13:07:35 2 [Note] WSREP: Installed new state from SST: 9b6e72e4-2838-11ec-8354-e6cfb7bf188b:3 | 15:01 |
| yoctozepto | lol | 15:02 |
| yoctozepto | should we recompile mariadb | 15:02 |
| yoctozepto | :P | 15:02 |
| yoctozepto | well, at least galera | 15:02 |
| yoctozepto | I wonder why there is 0 response from devs | 15:02 |
| yoctozepto | yeah, let's first try if restart helps | 15:08 |
| yoctozepto | so please hold the nodeset, frickler | 15:08 |
| yoctozepto | at least we will have more logs and more behaviour checks to report | 15:08 |
| yoctozepto | I will also spin up 10.6 testing | 15:09 |
| opendevreview | Radosław Piliszek proposed openstack/kolla-ansible master: [DNM] Test MariaDB 10.5 https://review.opendev.org/c/openstack/kolla-ansible/+/812944 | 15:13 |
| opendevreview | Radosław Piliszek proposed openstack/kolla-ansible master: [DNM] Test MariaDB 10.5 https://review.opendev.org/c/openstack/kolla-ansible/+/812944 | 15:13 |
| opendevreview | Radosław Piliszek proposed openstack/kolla master: [DNM] MariaDB 10.6 https://review.opendev.org/c/openstack/kolla/+/811015 | 15:13 |
| opendevreview | Radosław Piliszek proposed openstack/kolla-ansible master: [DNM] Test MariaDB 10.6 https://review.opendev.org/c/openstack/kolla-ansible/+/813202 | 15:14 |
| opendevreview | Verification of a change to openstack/kolla master failed: Use RDO's Xena Delorean packages https://review.opendev.org/c/openstack/kolla/+/812730 | 15:25 |
| opendevreview | Pierre Riteau proposed openstack/kolla stable/train: Fix multiple issues for stable/train https://review.opendev.org/c/openstack/kolla/+/813082 | 15:33 |
| yoctozepto | 2021-10-08 14:57:01.271106 nova.exception.TooOldComputeService: Current Nova version does not support computes older than Xena but the minimum compute service level in your system is 56 and the oldest supported service level is 57. | 15:51 |
| yoctozepto | something went wrong | 15:51 |
| opendevreview | Mark Goddard proposed openstack/kayobe master: DNM: infra VMs: use wait_for https://review.opendev.org/c/openstack/kayobe/+/813212 | 15:52 |
| yoctozepto | it did not template out the use of built images | 15:52 |
| yoctozepto | hmm, no, it did | 15:53 |
| yoctozepto | what the | 15:53 |
| mgoddard | yoctozepto: same issue as kayobe - need to merge kolla switch to xena sources | 15:53 |
| opendevreview | Mark Goddard proposed openstack/kayobe master: CI: add Infra VM jobs https://review.opendev.org/c/openstack/kayobe/+/813048 | 15:53 |
| yoctozepto | "stderr": "Error: operation wait on node rabbit@primary timed out. Timeout value used: 10000", | 15:54 |
| yoctozepto | yeah, that one had only random failures | 15:54 |
| yoctozepto | fingers crossed for gating | 16:03 |
| yoctozepto | 10.6 has the same issue | 16:27 |
| yoctozepto | so it's 10.4+ | 16:27 |
| yoctozepto | duh | 16:27 |
| yoctozepto | revert bump to 10.5 for now and discourage the usage of debian's mariadb image? ;d | 16:28 |
| yoctozepto | though I'd prefer we make the issue more visible in upstream as it's something that was introduced with new galera | 16:29 |
| *** amoralej is now known as amoralej|off | 16:29 | |
| yoctozepto | soon ubuntu will force a newer version on us anyway (like debian did) | 16:29 |
| * yoctozepto off | 16:33 | |
| opendevreview | Merged openstack/kolla master: release: Use Xena sources by default https://review.opendev.org/c/openstack/kolla/+/812726 | 17:27 |
| ozzzo_work | where do people talk about designate? #openstack-designate is pretty deserted | 17:49 |
| johnsom | ozzzo_work: #openstack-dns | 18:06 |
| ozzzo_work | ok ty johnsom | 18:34 |
| opendevreview | Maksim Malchuk proposed openstack/kolla-ansible master: Update dnsmasq logging https://review.opendev.org/c/openstack/kolla-ansible/+/813240 | 18:44 |
| opendevreview | Uwe Grawert proposed openstack/kolla-ansible master: Grafana: Run priviliged when copying home dashboard file https://review.opendev.org/c/openstack/kolla-ansible/+/813185 | 18:55 |
| opendevreview | Pierre Riteau proposed openstack/kolla stable/train: Fix multiple issues for stable/train https://review.opendev.org/c/openstack/kolla/+/813082 | 20:15 |
| opendevreview | Merged openstack/kolla master: Use RDO's Xena Delorean packages https://review.opendev.org/c/openstack/kolla/+/812730 | 20:23 |
| opendevreview | Maksim Malchuk proposed openstack/kolla-ansible master: Update dnsmasq logging https://review.opendev.org/c/openstack/kolla-ansible/+/813240 | 20:24 |
| opendevreview | Pierre Riteau proposed openstack/kolla master: Fix GitHub tarball URL for monasca-thresh https://review.opendev.org/c/openstack/kolla/+/813252 | 20:36 |
| opendevreview | Pierre Riteau proposed openstack/kolla master: Fix GitHub tarball URL for monasca-thresh https://review.opendev.org/c/openstack/kolla/+/813252 | 20:38 |
| opendevreview | Maksim Malchuk proposed openstack/kolla-ansible master: Update dnsmasq logging https://review.opendev.org/c/openstack/kolla-ansible/+/813240 | 20:43 |
| opendevreview | Maksim Malchuk proposed openstack/kayobe master: Use net_mask filter instead of ansible's ipaddr https://review.opendev.org/c/openstack/kayobe/+/813265 | 22:47 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!