Friday, 2021-10-08

opendevreviewBoxiang Zhu proposed openstack/kolla master: Add skyline Dockerfile  https://review.opendev.org/c/openstack/kolla/+/81079602:04
opendevreviewBoxiang Zhu proposed openstack/kolla-ansible master: Add skyline role  https://review.opendev.org/c/openstack/kolla-ansible/+/81056602:07
opendevreviewMerged openstack/kayobe stable/victoria: Fix IPA builder version typo  https://review.opendev.org/c/openstack/kayobe/+/81274502:11
opendevreviewMerged openstack/kayobe stable/wallaby: Add support for apt proxy setting  https://review.opendev.org/c/openstack/kayobe/+/81269802:11
*** amoralej|off is now known as amoralej06:45
mnasiadkayoctozepto: interesting, based on the log the cluster is synced, but the variable is still Initialized07:42
opendevreviewMark Goddard proposed openstack/kayobe master: CI: add Infra VM jobs  https://review.opendev.org/c/openstack/kayobe/+/81304807:47
OpenstackThundeHello Team, I enabled Central Logging and Monasca .... And I ran the deployment but for some reason I cant see the logging on the Horizon dashboard.. Any idea where I could be going wrong ?08:04
opendevreviewMark Goddard proposed openstack/kayobe master: Use merge_configs and merge_yaml to generate Kolla configs  https://review.opendev.org/c/openstack/kayobe/+/80406808:08
jeevamoved from deploying ussuri to wallaby08:16
jeevabut getting these errors now: https://pastebin.com/PhHd4rEA08:16
jeevatesting it with a pull to a node limited08:16
jeevaany ideas ?08:17
OpenstackThunde@opendevreview .... Thanks for the doc but I haven't used Kayobe before .... Is there any clearer guideline I use ?08:18
mgoddardjeeva: use wallaby tag rather than 12.1.008:19
opendevreviewMark Goddard proposed openstack/kayobe master: CI: add host configure tests for firewalld  https://review.opendev.org/c/openstack/kayobe/+/81229908:20
jeeva@mgoddard, thanks will try08:24
opendevreviewPierre Riteau proposed openstack/kolla stable/train: Require pip version compatible with Python 2  https://review.opendev.org/c/openstack/kolla/+/81314408:24
opendevreviewPierre Riteau proposed openstack/kolla stable/train: Require pip version compatible with Python 2  https://review.opendev.org/c/openstack/kolla/+/81314408:24
OpenstackThundeI am actually running it on Wallby 08:25
OpenstackThundeWallaby08:25
jeevamgoddard, there is no image with a name of wallaby though08:32
mgoddardoh, you're using your own registry08:33
jeevaopenstack_tag: "wallaby"08:33
mgoddardah08:33
jeevai'm under the impression kolla will pull it down to our local registry08:34
mgoddardpossibly we need to be more clear about the namespace - I think kolla builds with a namespace of kolla, but kolla-ansible uses openstack.kolla08:34
mgoddardjeeva: try setting docker_namespace to kolla08:34
mgoddardyoctozepto, mnasiadka: one to sort out before release ^08:34
jeevanot following the "try setting docker_namespace to kolla"08:35
yoctozeptojeeva: in kolla ansible config, in globals.yml08:54
yoctozeptodocker_namespace: kolla08:54
yoctozeptomgoddard, mnasiadka: good point08:54
yoctozeptomnasiadka: yeah, the log says everything's fine08:54
yoctozeptoI am worried there is a bug in MariaDB, arar :<08:54
mnasiadkayoctozepto: maybe that variable is not populated - try dumping all wsrep variables and check the others09:00
mnasiadkamgoddard: there's a patch in Kayobe to follow kolla-ansible, so either we change it in kolla - or update docs09:00
mgoddardprobably we should change the default to be consistent09:02
mnasiadkait won't hurt, agree09:03
yoctozeptomnasiadka: yeah, good idea09:03
yoctozepto(about wsrep var dump)09:04
opendevreviewPierre Riteau proposed openstack/kolla stable/train: Use local u-c file to avoid SSL verification issues  https://review.opendev.org/c/openstack/kolla/+/81315309:16
jinyuanliuhttps://review.opendev.org/c/openstack/kolla/+/79379509:21
jinyuanliuPlease review it for me, if any core has time09:22
jinyuanliuthx09:22
opendevreviewPierre Riteau proposed openstack/kolla stable/train: Use local u-c file to avoid SSL verification issues  https://review.opendev.org/c/openstack/kolla/+/81315309:24
opendevreviewRadosław Piliszek proposed openstack/kolla-ansible master: [DNM] Test MariaDB  https://review.opendev.org/c/openstack/kolla-ansible/+/81294409:37
opendevreviewPierre Riteau proposed openstack/kolla master: Add prometheus-msteams image  https://review.opendev.org/c/openstack/kolla/+/81267809:59
priteauKayobe master is broken10:01
priteaumaster CI10:01
priteaunova.exception.TooOldComputeService: Current Nova version does not support computes older than Xena but the minimum compute service level in your system is 56 and the oldest supported service level is 57.10:01
jeeva10.102.17.2%3A5000%2Fopenstack.kolla%2Fubuntu-source-fluentd: Not Found (\"manifest for 10.102.17.2:5000/openstack.kolla/ubuntu-source-fluentd:12.1.0 not found: manifest unknown: manifest unknown\")\\n'"}10:10
jeevai dont see where it gets the "/openstack.kolla" part from10:10
jeevasudo docker pull 10.102.17.2:5000/kolla/ubuntu-source-fluentd:12.1.0 works fine when i do it manually10:10
jeevabut deploy adds a "/openstack.kolla/..." in the registry10:11
priteaudocker_namespace: "openstack.kolla"10:11
mgoddardpriteau: I guess we need to merge the kolla xena release patches as the world has moved on10:11
jeeva#docker_namespace: "kolla"10:11
jeevaits that10:11
jeevain my globals.yml10:11
mgoddardyoctozepto: time for https://review.opendev.org/q/topic:%22kolla-xena-release%22+(status:open%20OR%20status:merged) ?10:12
priteaumgoddard: Oh, is kolla master still tracking master?10:12
mgoddardyes10:12
mgoddardI guess this is why we have a release schedule...10:13
priteauOK, I thought that was merged10:13
jeevawhat should i change the namespace to if the default is "kolla", it must look like this 10.102.17.2:5000/openstack.kolla/ubuntu-source-fluentd:12.1.010:14
priteaujeeva: are you using kolla-ansible master branch?10:15
jeevai changed it to "kolla"10:16
jeevait went further10:16
jeevanow back to the insecure registry10:16
priteaujeeva: I may be mistaken but it sounds like you're using master branch (which I would avoid now while the release work is still in progress) instead of wallaby10:17
priteaumgoddard: Not sure if it's worth reverting kayobe to use docker hub for CI, by the time the patch will be checked and merged it will be near the end of the afternoon. But if we can merge the kolla patches today, Kayobe CI would be back to green on Monday?10:21
mgoddardpriteau: probably best to wait10:22
opendevreviewPierre Riteau proposed openstack/kolla stable/train: CI: Fix multiple issues for stable/train  https://review.opendev.org/c/openstack/kolla/+/81308210:36
opendevreviewMichal Arbet proposed openstack/kolla master: Make cron's logrotate schedule configurable  https://review.opendev.org/c/openstack/kolla/+/81300811:05
opendevreviewPierre Riteau proposed openstack/kolla-ansible stable/train: CI: Remove ara integration  https://review.opendev.org/c/openstack/kolla-ansible/+/81317111:27
kevkoanyone to advice ? 11:33
kevkore-scheduled: invalid argument: could not find capabilities for domaintype=kvm \n11:33
jeeva{"changed": false, "msg": "MariaDB cluster is not synced. Please wait for WSREP sync before proceeding."} -- how long is this WSREP sync ?11:33
kevkovictoria debian binary -> control plane upgraded to wallaby11:33
kevkojeeva: how fast are your machines :D 11:33
jeevavery fast11:33
kevkodeploy one ? 11:34
jeevatestbed, DELL R630's11:34
kevkohmm11:34
kevkoit should be quite fast 11:34
jeeva48 cores, 328 GB ram per controller11:34
jeeva384*11:34
kevkoversion of mariadb ? 11:34
ignaziocassanohello, I've just installed openstack kolla wallaby and when I create and empty volume on netapp ontap it works. It fails ig I create a volume from images. cinder-volume logs: https://paste.openstack.org/show/809875/11:35
jeevakevko, 10.102.17.2:5000/kolla/ubuntu-source-mariadb-server:12.1.011:35
kevkoi meant mariadb inside container 11:36
jeevaudo docker inspect mariadb | grep 'version'11:37
jeeva                "kolla_version": "12.1.0",11:37
jeevaonly version related item i see in the container11:38
jeevaWARNING: IPv4 forwarding is disabled. Networking will not work.11:43
jeevaseems maybe because of this11:43
kevkoyeah ..probably 11:47
kevkoyoctozepto: (nova-libvirt)[root@compute1 /]# ls -la /dev/kvm 11:47
kevkocrw-rw---- 1 root systemd-resolve 10, 232 Oct  8 06:30 /dev/kvm11:47
kevkothis is weird right  11:47
kevko? 11:47
jeevadoesn't seem like the case, added ipv4 to the sysctl and restarted networking11:49
jeeva"msg": "MariaDB cluster is not synced. Please wait for WSREP sync before proceeding.11:50
frickleriiuc this is what yoctozepto was looking at with https://review.opendev.org/c/openstack/kolla-ansible/+/81294411:52
fricklerdoesn't seem to be 100% reproducible though11:52
yoctozeptofrickler: yeah, this seems related12:02
yoctozeptoI mean, something weird with wsrep after the version bump12:02
yoctozeptoand it fails too often in CI12:02
yoctozeptobut it's on master here, not wallaby12:08
yoctozeptowallaby is not affected12:09
* yoctozepto just read the backlog12:09
ignaziocassanohello, I've just installed openstack kolla wallaby and when I create and empty volume on netapp ontap it works. It fails ig I create a volume from images. cinder-volume logs: https://paste.openstack.org/show/809875/ . Something is changed on cinder ?12:14
opendevreviewMark Goddard proposed openstack/kayobe master: CI: add host configure tests for firewalld  https://review.opendev.org/c/openstack/kayobe/+/81229912:20
yoctozeptoignaziocassano: is nova able to boot from those images fine? i.e. if you boot from image without creating a new volume12:25
ignaziocassanoyoctozepto: II am going to test it.12:27
ignaziocassano_yoctozepto: yes, ephemeral works12:37
kevkolooks like debian-binary-libvirt victoria is broken somehow :/12:41
yoctozeptokevko: it does not differ from source one and it works for me :-)12:42
yoctozeptowhat fails for you?12:42
yoctozepto(well, it should not differ at least :D _12:42
kevkoyoctozepto: https://paste.opendev.org/show/809880/12:44
kevkoi've replaced nova-libvirt container with centos-source one and it started to work 12:44
kevkowhat's weird is this 12:44
kevkoyoctozepto: (nova-libvirt)[root@compute1 /]# ls -la /dev/kvm 12:44
kevko<kevko> crw-rw---- 1 root systemd-resolve 10, 232 Oct  8 06:30 /dev/kvm12:44
kevkoyoctozepto: wallaby test stack --> 12:45
kevko(nova-libvirt)[root@compute0 /]# ls -la /dev/kvm 12:45
kevkocrw-rw---- 1 root qemu 10, 232 Oct  5 15:15 /dev/kvm12:45
yoctozeptoweird indeed12:45
kevkoyeah 12:46
kevkolet me exec into container and check snippets regarding user creation in a package12:48
opendevreviewPierre Riteau proposed openstack/kolla stable/train: Fix multiple issues for stable/train  https://review.opendev.org/c/openstack/kolla/+/81308212:49
*** amoralej is now known as amoralej|lunch12:52
opendevreviewPierre Riteau proposed openstack/kolla-ansible stable/train: CI: Remove ara integration  https://review.opendev.org/c/openstack/kolla-ansible/+/81317113:03
opendevreviewUwe Grawert proposed openstack/kolla-ansible master: Grafana: Run priviliged when copying home dashboard file  https://review.opendev.org/c/openstack/kolla-ansible/+/81318513:07
*** amoralej|lunch is now known as amoralej13:17
opendevreviewVerification of a change to openstack/kolla master failed: Update versions for Xena  https://review.opendev.org/c/openstack/kolla/+/81272913:22
priteauKolla CI is hitting the same upgrade issue as Kayobein https://review.opendev.org/c/openstack/kolla/+/81272913:46
priteauWill be fixed by https://review.opendev.org/c/openstack/kolla/+/812726?13:47
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Add api_workers for each service to defaults  https://review.opendev.org/c/openstack/kolla-ansible/+/81319313:47
yoctozeptomnasiadka, frickler: https://54e727e34e3178aca82f-3ad27eaf65bc806a39bd9f42516bdf66.ssl.cf2.rackcdn.com/812944/2/check/kolla-ansible-ubuntu-source-mariadb/205153c/primary/logs/ansible/deploy all wsrep vars tell the same - the node failed to join13:52
yoctozeptologs do not agree13:53
yoctozeptoah no13:55
yoctozeptothey do agree, only need to check the right host ofc13:55
yoctozepto"2021-10-08 13:07:35 2 [Note] WSREP: Server status change connected -> disconnected"13:55
yoctozepto2021-10-08 13:07:35 2 [Warning] WSREP: 0x564490c95b08 down context(s) not set13:55
yoctozepto2021-10-08 13:07:35 2 [Warning] WSREP: Sending JOIN failed: -107 (Transport endpoint is not connected). Will retry in new primary component.13:55
yoctozepto2021-10-08 13:07:35 0 [ERROR] WSREP: IST didn't contain all write sets, expected last: 3 last received: 213:56
yoctozepto2021-10-08 13:07:35 7 [ERROR] WSREP: Receiving IST failed, node restart required: IST receiver reported failure: 71 (Protocol error)13:56
yoctozepto at /home/buildbot/buildbot/build/galera/src/replicator_smm.hpp:pop_front():315. Null event.13:56
frickleryoctozepto: https://54e727e34e3178aca82f-3ad27eaf65bc806a39bd9f42516bdf66.ssl.cf2.rackcdn.com/812944/2/check/kolla-ansible-ubuntu-source-mariadb/205153c/primary/logs/kolla/mariadb/mariadb.txt14:01
fricklerseems primary and sec2 in the end get together, just sec1 doesn't14:01
fricklerdo you want me to hold a job in order to look at things in place?14:02
opendevreviewVerification of a change to openstack/kolla master failed: release: Use Xena sources by default  https://review.opendev.org/c/openstack/kolla/+/81272614:15
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Add api_workers for each service to defaults  https://review.opendev.org/c/openstack/kolla-ansible/+/81319314:20
yoctozeptofrickler: yeah, we could; but I still don't know what to look at; one of the 3 simply fails14:32
frickleryoctozepto: not sure either, the plan is to just look around until something looks suspicous14:36
mnasiadkayoctozepto: well at least we could check if re-bootstrapping the node helps14:44
opendevreviewPierre Riteau proposed openstack/kolla-ansible master: Stop creating unused cron/logrotate directory  https://review.opendev.org/c/openstack/kolla-ansible/+/81319914:49
yoctozeptomnasiadka, frickler: yeah, "node restart required" suggests it may help to restart the service14:56
yoctozeptoseems to be a known issue https://jira.mariadb.org/browse/MDEV-2279614:57
yoctozeptoI wonder why it uses ist on the first run14:58
yoctozeptomaybe we could make it use SST14:58
mnasiadkahttps://jira.mariadb.org/browse/MDEV-2279715:00
mnasiadkathis one looks fantastic15:00
yoctozepto2021-10-08 13:07:35 2 [Note] WSREP: Installed new state from SST: 9b6e72e4-2838-11ec-8354-e6cfb7bf188b:315:01
yoctozeptolol15:02
yoctozeptoshould we recompile mariadb15:02
yoctozepto:P15:02
yoctozeptowell, at least galera15:02
yoctozeptoI wonder why there is 0 response from devs15:02
yoctozeptoyeah, let's first try if restart helps15:08
yoctozeptoso please hold the nodeset, frickler15:08
yoctozeptoat least we will have more logs and more behaviour checks to report15:08
yoctozeptoI will also spin up 10.6 testing15:09
opendevreviewRadosław Piliszek proposed openstack/kolla-ansible master: [DNM] Test MariaDB 10.5  https://review.opendev.org/c/openstack/kolla-ansible/+/81294415:13
opendevreviewRadosław Piliszek proposed openstack/kolla-ansible master: [DNM] Test MariaDB 10.5  https://review.opendev.org/c/openstack/kolla-ansible/+/81294415:13
opendevreviewRadosław Piliszek proposed openstack/kolla master: [DNM] MariaDB 10.6  https://review.opendev.org/c/openstack/kolla/+/81101515:13
opendevreviewRadosław Piliszek proposed openstack/kolla-ansible master: [DNM] Test MariaDB 10.6  https://review.opendev.org/c/openstack/kolla-ansible/+/81320215:14
opendevreviewVerification of a change to openstack/kolla master failed: Use RDO's Xena Delorean packages  https://review.opendev.org/c/openstack/kolla/+/81273015:25
opendevreviewPierre Riteau proposed openstack/kolla stable/train: Fix multiple issues for stable/train  https://review.opendev.org/c/openstack/kolla/+/81308215:33
yoctozepto2021-10-08 14:57:01.271106 nova.exception.TooOldComputeService: Current Nova version does not support computes older than Xena but the minimum compute service level in your system is 56 and the oldest supported service level is 57.15:51
yoctozeptosomething went wrong15:51
opendevreviewMark Goddard proposed openstack/kayobe master: DNM: infra VMs: use wait_for  https://review.opendev.org/c/openstack/kayobe/+/81321215:52
yoctozeptoit did not template out the use of built images15:52
yoctozeptohmm, no, it did15:53
yoctozeptowhat the15:53
mgoddardyoctozepto: same issue as kayobe - need to merge kolla switch to xena sources15:53
opendevreviewMark Goddard proposed openstack/kayobe master: CI: add Infra VM jobs  https://review.opendev.org/c/openstack/kayobe/+/81304815:53
yoctozepto    "stderr": "Error: operation wait on node rabbit@primary timed out. Timeout value used: 10000",15:54
yoctozeptoyeah, that one had only random failures15:54
yoctozeptofingers crossed for gating16:03
yoctozepto10.6 has the same issue16:27
yoctozeptoso it's 10.4+16:27
yoctozeptoduh16:27
yoctozeptorevert bump to 10.5 for now and discourage the usage of debian's mariadb image? ;d16:28
yoctozeptothough I'd prefer we make the issue more visible in upstream as it's something that was introduced with new galera16:29
*** amoralej is now known as amoralej|off16:29
yoctozeptosoon ubuntu will force a newer version on us anyway (like debian did)16:29
* yoctozepto off16:33
opendevreviewMerged openstack/kolla master: release: Use Xena sources by default  https://review.opendev.org/c/openstack/kolla/+/81272617:27
ozzzo_workwhere do people talk about designate? #openstack-designate is pretty deserted17:49
johnsomozzzo_work: #openstack-dns18:06
ozzzo_workok ty johnsom 18:34
opendevreviewMaksim Malchuk proposed openstack/kolla-ansible master: Update dnsmasq logging  https://review.opendev.org/c/openstack/kolla-ansible/+/81324018:44
opendevreviewUwe Grawert proposed openstack/kolla-ansible master: Grafana: Run priviliged when copying home dashboard file  https://review.opendev.org/c/openstack/kolla-ansible/+/81318518:55
opendevreviewPierre Riteau proposed openstack/kolla stable/train: Fix multiple issues for stable/train  https://review.opendev.org/c/openstack/kolla/+/81308220:15
opendevreviewMerged openstack/kolla master: Use RDO's Xena Delorean packages  https://review.opendev.org/c/openstack/kolla/+/81273020:23
opendevreviewMaksim Malchuk proposed openstack/kolla-ansible master: Update dnsmasq logging  https://review.opendev.org/c/openstack/kolla-ansible/+/81324020:24
opendevreviewPierre Riteau proposed openstack/kolla master: Fix GitHub tarball URL for monasca-thresh  https://review.opendev.org/c/openstack/kolla/+/81325220:36
opendevreviewPierre Riteau proposed openstack/kolla master: Fix GitHub tarball URL for monasca-thresh  https://review.opendev.org/c/openstack/kolla/+/81325220:38
opendevreviewMaksim Malchuk proposed openstack/kolla-ansible master: Update dnsmasq logging  https://review.opendev.org/c/openstack/kolla-ansible/+/81324020:43
opendevreviewMaksim Malchuk proposed openstack/kayobe master: Use net_mask filter instead of ansible's ipaddr  https://review.opendev.org/c/openstack/kayobe/+/81326522:47

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!