Thursday, 2022-07-21

opendevreviewMichal Nasiadka proposed openstack/kayobe master: ovn: add network group to neutron-ovn-metadata-agent  https://review.opendev.org/c/openstack/kayobe/+/85060806:18
opendevreviewKonstantin Yarovoy proposed openstack/kolla master: Refactor image module  https://review.opendev.org/c/openstack/kolla/+/82871607:31
opendevreviewKonstantin Yarovoy proposed openstack/kolla master: Refactor image module  https://review.opendev.org/c/openstack/kolla/+/82871607:54
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: WIP: clustercheck: move from xinetd to socat  https://review.opendev.org/c/openstack/kolla-ansible/+/85061108:29
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: WIP: clustercheck: move from xinetd to socat  https://review.opendev.org/c/openstack/kolla-ansible/+/85061108:42
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: WIP: clustercheck: move from xinetd to socat  https://review.opendev.org/c/openstack/kolla-ansible/+/85061108:42
EugenMayerHello. Trying to upgrade with the xena series. Updated to the latest xena kolla/kolla-ansible, but when i run deploy it fails with the sasl password https://gist.github.com/EugenMayer/9fa2521af059933fac0a39ff71314ee5 - is this a known issue?08:45
EugenMayers/with/within08:46
mnasiadkaEugenMayer: have you merged the passwords.yml changes? like here: https://docs.openstack.org/kolla-ansible/latest/user/operating-kolla.html#preparation-the-real-deal08:47
EugenMayerhmm i was looking on https://docs.openstack.org/kolla-ansible/latest/user/operating-kolla.html#upgrade-procedure .. i seem to not been aware sorry. checking your docs08:48
EugenMayermnasiadka that was surely it, i did not follow those docs. Will do better next time, thank you for the hint!08:54
mnasiadkaEugenMayer: np09:01
EugenMayermnasiadka seems like the upgrade went fine at first sight, the ansible task completed without issues. now on all computes nova seem to have issues to start the docker container: Restarting (0) 6 seconds ago             nova_compute09:07
EugenMayeram i usually forced to restart the controllers/computes after a in-series upgrade=09:08
mnasiadkano, it should work09:09
mnasiadkacheck docker logs, nova-compute logs09:09
EugenMayerdocker logs looked non problematic, checking the nova logs now09:10
EugenMayer2022-07-21 11:10:56.775 7 INFO nova.virt.libvirt.driver [-] Connection event '0' reason 'Failed to connect to libvirt: authentication failed: authentication failed'09:11
EugenMayerhttps://gist.github.com/EugenMayer/bad693ff3a4a989b54abbce641e7c0d109:12
mnasiadkayeah, that's related to sasl for sure09:17
mnasiadkayou can check libvirt logs for anything meaningful09:17
EugenMayermnasiadka where do i find those?09:19
EugenMayerfound those09:20
EugenMayer2022-07-21 09:19:23.586+0000: 6539: error : virNetSASLSessionServerStep:594 : authentication failed: Failed to start SASL negotiation: -20 (SASL(-13): user not found: unable to canonify user and get auxprops)09:20
EugenMayer2022-07-21 09:19:23.586+0000: 6539: error : remoteDispatchAuthSaslStep:3991 : authentication failed: authentication failed09:20
EugenMayermnasiadka seems to be related to what was already required for the password merge, the new sasl password. Seems like this has been introduced but maybe the user has not been created on 'upgrade'09:27
EugenMayermaybe i would have needed to add something new to the globals too09:29
EugenMayermnasiadka i guess i'am confronted with https://docs.openstack.org/releasenotes/kolla-ansible/xena.html#upgrade-notes09:36
EugenMayerwhile i generated the password and uses kolla upgrade, i assume (and docker ps tells the same story) that all containers have been replaced using the latest xena variant, including the libvirt one09:37
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: WIP: clustercheck: move from xinetd to socat  https://review.opendev.org/c/openstack/kolla-ansible/+/85061109:38
EugenMayeri merged the multinode inventory changes too mnasiadka, but i cannot get rid of the issue with the sasl password10:08
mnasiadkamgoddard: any idea what might be wrong? ^^10:34
EugenMayermnasiadka looking at the ansible playbooks, i ran saslpasswd2 -c -p -a libvirt on the nova_libvirt container with nova/the sasl password10:39
EugenMayeri double checked that my nova auth.conf is correct, which it is10:39
opendevreviewMerged openstack/kolla-ansible master: Add [taskflow] section for masakari.conf.j2  https://review.opendev.org/c/openstack/kolla-ansible/+/85029111:47
opendevreviewIvan Halomi proposed openstack/kolla-ansible master: Add systemd container control  https://review.opendev.org/c/openstack/kolla-ansible/+/81672411:57
*** amoralej is now known as amoralej|lunch12:20
opendevreviewMichal Arbet proposed openstack/kolla master: Update ProxySQL to version 2.4  https://review.opendev.org/c/openstack/kolla/+/85063612:28
opendevreviewSergei Raiskii proposed openstack/kolla-ansible master: Allow cinder-backup to be configured to use S3 backend.  https://review.opendev.org/c/openstack/kolla-ansible/+/84971112:38
kevkohmm, kolla build is probably broken because of some pip package incompatibiltiy :/13:05
hrw?13:07
hrwwhich build13:08
kevkohrw: https://review.opendev.org/c/openstack/kolla/+/850636 << if you check the logs ...there is no problem with proxysql ..but with pip dependencies13:11
kevko hrw: for example -> https://zuul.opendev.org/t/openstack/build/af9881ee80ad4b808626fa7d8f361841/log/kolla/build/000_FAILED_openstack-base.log <-13:11
hrwINFO:kolla.common.utils.openstack-base:[91mERROR: Could not install packages due to an OSError: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))13:12
kevkohmm, what is that :/13:12
hrwINFO:kolla.common.utils.openstack-base:1mWARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))': /pypi/simple/grpcio/13:12
hrwfailed to connect to pypi mirror13:12
kevkohmmm...it looks like intermittent error 13:13
hrwrecheck?13:13
kevkoyeah13:13
kevkounfortunatelly RemoteDisconnected('Remote end closed connection without response') will be never fixed :( 13:14
opendevreviewMerged openstack/kolla-ansible master: make "external_labels" optional on prometheus.yml  https://review.opendev.org/c/openstack/kolla-ansible/+/85044613:26
opendevreviewVerification of a change to openstack/kolla master failed: Install etcd binary from github  https://review.opendev.org/c/openstack/kolla/+/84736813:26
*** amoralej|lunch is now known as amoralej13:28
kevkohrw: still same :/14:05
opendevreviewMerged openstack/kolla-ansible master: init-runonce: Migrate to ECDSA keys  https://review.opendev.org/c/openstack/kolla-ansible/+/84718714:11
opendevreviewMerged openstack/ansible-collection-kolla master: packages: add tzdata to ubuntu package list  https://review.opendev.org/c/openstack/ansible-collection-kolla/+/85033614:11
opendevreviewMerged openstack/kolla-ansible master: Add support for Ubuntu Jammy Jellyfish (22.04 LTS)  https://review.opendev.org/c/openstack/kolla-ansible/+/84607014:12
jingvarI have to use custom config.json for some services , is there a way except downstream the kolla-ansible code (templates)?14:13
opendevreviewMichal Arbet proposed openstack/kolla master: Run proxysql with proxysql user  https://review.opendev.org/c/openstack/kolla/+/85065614:14
opendevreviewMichal Arbet proposed openstack/kolla master: Run proxysql with proxysql user  https://review.opendev.org/c/openstack/kolla/+/85065614:15
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Add proxysql support for database  https://review.opendev.org/c/openstack/kolla-ansible/+/77021514:21
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Edit services roles to support database sharding  https://review.opendev.org/c/openstack/kolla-ansible/+/77021614:21
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: [CI] Test ProxySQL with shards in the nova cells scenario  https://review.opendev.org/c/openstack/kolla-ansible/+/77062114:21
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: [DNM] Trigger cells job  https://review.opendev.org/c/openstack/kolla-ansible/+/83891614:21
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Use Docker healthchecks for mariadb-server service  https://review.opendev.org/c/openstack/kolla-ansible/+/80561614:21
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Add proxysql support for database  https://review.opendev.org/c/openstack/kolla-ansible/+/77021514:29
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Edit services roles to support database sharding  https://review.opendev.org/c/openstack/kolla-ansible/+/77021614:29
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: [CI] Test ProxySQL with shards in the nova cells scenario  https://review.opendev.org/c/openstack/kolla-ansible/+/77062114:29
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: [DNM] Trigger cells job  https://review.opendev.org/c/openstack/kolla-ansible/+/83891614:29
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Use Docker healthchecks for mariadb-server service  https://review.opendev.org/c/openstack/kolla-ansible/+/80561614:29
kevkohrw: hmm, I am able to build it normally on localhost ..what is different in CI ? it's failing every time 15:01
hrwkevko: http://mirror.bhs1.ovh.opendev.org:8080/pypi/simple/grpcio/ - check which version failed?15:05
hrwhm. fetching works15:06
kevkohrw: it looks like it's not problem with grpcio only15:11
kevkohttps://paste.opendev.org/show/bskwQbwP8VLVJchVXknW/15:11
hrwhm. asked on #opendev?15:12
yoctozeptohrw, kevko: #opendev is debugging the issue15:12
hrwyoctozepto: thanks15:12
yoctozeptoit affects all projects15:12
yoctozeptosomething about pypi mirrors15:12
kevkorrr15:12
kevkoswitch to binary images :P 15:12
kevkohaha15:13
kevkoyoctozepto: could you check quickly proxysql change i've fixed last two comments ? 15:13
kevkoyoctozepto: few lines 15:13
yoctozeptokevko: k-a?15:16
hrwkevko: what are binary images?15:18
kevkoyoctozepto: yeah 15:19
kevkohrw: just joking ...kolla binary images ..15:19
hrwkevko: I tend to put sarcasm far too often than people are used to15:20
kevkohrw: haha, I'll try to get used to it15:22
kevkoyoctozepto, hrw: it looks like there is more context here -> https://zuul.opendev.org/t/openstack/build/8b8b388d3e2d484b9c2215144b50d505/log/kolla/build/openstack-base.log15:32
kevkohmm ..no, it's the same 15:33
*** amoralej is now known as amoralej|off15:57
yoctozeptokevko: why have you moved the start script to k-a?16:17
yoctozeptothere is no templating happening16:17
yoctozeptodo you have any plans for that? why complicate an already complicated thing? :D16:18
kevkoyoctozepto: well, it's better to modify config.json ..not script inside ..and if i didn't use bash ..i was not able to log to file ..16:24
kevkoyoctozepto: on the other side ..if you think it was better before ..i can revert it :P 16:25
yoctozeptokevko: I was just curious why it's done like this; I don't see an issue with going this way but it's, so to speak, unorthodox :-)16:25
kevkoyoctozepto: it was also frickler comment few patchset before ..so he will be happy :P 16:26
yoctozeptokevko: was it now?16:26
yoctozeptolet me see16:26
yoctozeptoyou have added frickler as a reviewer16:27
yoctozeptohe has not commented on it at all16:27
kevkoreally ? i remember :D 16:27
yoctozeptokevko: maybe you think about it so much that frickler has appeared in your dreams :-)16:30
yoctozeptoanyway, to help you forget about proxysql, I have given you a CR+216:30
yoctozeptoplease use it wisely16:30
kevkoi saw :D 16:30
kevkoi am going to reply to your comments in patchsets in stack 16:30
yoctozeptogood16:33
yoctozeptolet's get this over with16:33
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Edit services roles to support database sharding  https://review.opendev.org/c/openstack/kolla-ansible/+/77021617:01
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: [CI] Test ProxySQL with shards in the nova cells scenario  https://review.opendev.org/c/openstack/kolla-ansible/+/77062117:01
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: [DNM] Trigger cells job  https://review.opendev.org/c/openstack/kolla-ansible/+/83891617:01
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Use Docker healthchecks for mariadb-server service  https://review.opendev.org/c/openstack/kolla-ansible/+/80561617:01
opendevreviewKyle Dean proposed openstack/kayobe master: Add support for firewalld on Ubuntu  https://review.opendev.org/c/openstack/kayobe/+/85039017:08
opendevreviewKyle Dean proposed openstack/kayobe master: Add support for firewalld on Ubuntu  https://review.opendev.org/c/openstack/kayobe/+/85039017:08
opendevreviewJuan Pablo Suazo proposed openstack/kolla-ansible master: Allows for multiple Ceph Conf files  https://review.opendev.org/c/openstack/kolla-ansible/+/84802920:12
opendevreviewJuan Pablo Suazo proposed openstack/kolla-ansible master: Allows for multiple Ceph Conf files  https://review.opendev.org/c/openstack/kolla-ansible/+/84802920:18
xiaolonghello23:43

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!