Tuesday, 2021-10-19

*** mazzy509 is now known as mazzy5001:23
*** vishalmanchanda_ is now known as vishalmanchanda03:51
opendevreviewMerged openstack/kayobe master: Add reno about chrony removal  https://review.opendev.org/c/openstack/kayobe/+/81437405:36
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: mariadb: Remove wsrep-notify.sh  https://review.opendev.org/c/openstack/kolla-ansible/+/81439305:53
eugenmayerGood morning06:09
eugenmayerI have a question, does multinode refer to multiple controller nodes (so controller ha) or multible nodes in general, like 1 controller + 5 computes?06:10
eugenmayer(compared to all-in-one)06:10
jingvar3 controllers06:44
jingvarha is optional06:45
mnasiadkaeugenmayer: multiple nodes06:51
mnasiadkaeugenmayer: if you're asking about inventory file06:51
mnasiadkamgoddard, priteau: are we ready to branch kayobe?06:52
opendevreviewMerged openstack/kayobe master: Add missing apt playbook for infrastructure VMs  https://review.opendev.org/c/openstack/kayobe/+/81402807:15
mnasiadkayoctozepto: I guess we need to post rc2 after fixing mariadb?07:18
*** amoralej|off is now known as amoralej07:36
opendevreviewMerged openstack/kayobe stable/victoria: Remove CentOS 7 image resolv.conf workaround  https://review.opendev.org/c/openstack/kayobe/+/81393407:48
yoctozeptomnasiadka: yes, I guess let's wait for the other mariadb fixes (hopefully nothing else is broken), then rc2 and release at more or less the same time (like, give it 2-3 days to sit)07:50
yoctozeptothat was my idea :-)07:50
eugenmayercan someone explain why `Checking if kolla_internal_vip_address and kolla_external_vip_address are not pingable from any nod` is a needed check?07:52
eugenmayerwhy is the controller not allowed to be pinging this IP?07:52
eugenmayeri mean, when i start to boostrap my cluster, i set kolla_internal_vip_address to the IP of the interface of the controller of the management network. Is this the misunderstanding? Do i have to pick a ip of that network, but a yet not assigned one?07:56
mgoddardeugenmayer: the VIP address is managed by keepalived, and should not be present initially08:25
mgoddardeugenmayer: you'll normally have an api interface address at the beginning of a deploy, then afterwards keepalived will add a VIP (or two) on one of the hosts08:26
eugenmayerunderstood. One questions, do you run kolla-ansible from the controller node or rather from an admin device (consumer device) or even rather from a other server, only there to run all the deployments? it seems to me kolla-ansible has no reason to be on the controller node itself, right?08:28
opendevreviewMerged openstack/kayobe stable/ussuri: Remove CentOS 7 image resolv.conf workaround  https://review.opendev.org/c/openstack/kayobe/+/81393508:36
opendevreviewSeena Fallah proposed openstack/kolla-ansible master: mariadb: use add_host to include inactive hosts in shard grouping  https://review.opendev.org/c/openstack/kolla-ansible/+/81427608:38
mgoddardeugenmayer: right - usually there is a separate deployment node08:41
eugenmayeris see, that makes a lot of sense08:43
opendevreviewMark Goddard proposed openstack/kolla-ansible master: WIP: Support custom services  https://review.opendev.org/c/openstack/kolla-ansible/+/81453008:52
opendevreviewMark Goddard proposed openstack/kolla-ansible master: WIP: Support custom services  https://review.opendev.org/c/openstack/kolla-ansible/+/81453009:05
opendevreviewPierre Riteau proposed openstack/kayobe master: Add support for configuring proxy settings  https://review.opendev.org/c/openstack/kayobe/+/81404609:05
opendevreviewMark Goddard proposed openstack/kolla-ansible master: docs: Improve info about neutron external interface  https://review.opendev.org/c/openstack/kolla-ansible/+/81396509:08
opendevreviewMerged openstack/kolla-ansible stable/xena: [mariadb] Start new nodes serially  https://review.opendev.org/c/openstack/kolla-ansible/+/81426909:35
opendevreviewMerged openstack/kolla-ansible stable/wallaby: [mariadb] Start new nodes serially  https://review.opendev.org/c/openstack/kolla-ansible/+/81445009:35
opendevreviewMerged openstack/kolla-ansible stable/wallaby: Correctly create the dhcp_agent.ini and l3_agent.ini  https://review.opendev.org/c/openstack/kolla-ansible/+/81356009:35
opendevreviewMerged openstack/kolla-ansible stable/victoria: Correctly create the dhcp_agent.ini and l3_agent.ini  https://review.opendev.org/c/openstack/kolla-ansible/+/81356109:35
opendevreviewMerged openstack/kolla-ansible stable/ussuri: Correctly create the dhcp_agent.ini and l3_agent.ini  https://review.opendev.org/c/openstack/kolla-ansible/+/81356209:35
opendevreviewMerged openstack/kolla-ansible stable/wallaby: Add support for Ironic inspection through DHCP-relay  https://review.opendev.org/c/openstack/kolla-ansible/+/81389009:35
opendevreviewMerged openstack/kolla-ansible stable/victoria: Add support for Ironic inspection through DHCP-relay  https://review.opendev.org/c/openstack/kolla-ansible/+/81389109:35
opendevreviewMerged openstack/kolla-ansible stable/ussuri: Add support for Ironic inspection through DHCP-relay  https://review.opendev.org/c/openstack/kolla-ansible/+/81389209:35
opendevreviewMark Goddard proposed openstack/kolla-ansible master: WIP: Support custom services  https://review.opendev.org/c/openstack/kolla-ansible/+/81453009:37
blmtgood morning all09:40
blmtanyone using OVN in production? we are hitting some sort of random packet loss09:41
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: mariadb: Remove wsrep-notify.sh  https://review.opendev.org/c/openstack/kolla-ansible/+/81439309:45
mnasiadkablmt: random?09:46
opendevreviewPierre Riteau proposed openstack/kayobe master: Allow Docker to use insecure registry when deployed by Kayobe  https://review.opendev.org/c/openstack/kayobe/+/81216809:51
opendevreviewPierre Riteau proposed openstack/kayobe master: Prevent Bifrost from using firewalld  https://review.opendev.org/c/openstack/kayobe/+/81109709:55
blmtmnasiadka it started a week ago and we are seeing packet drop in a random fashion: we can loose anything between 1-2% to 30-40%09:56
blmtwe are using OVN-DVR and the issue seems to be limited to VM with floating IPs09:57
blmtwe are using OVN-DVR and the issue seems to be limited to VMs with floating IPs09:57
mnasiadkaso the traffic between VMs (without touching floating ip) works properly?09:57
blmtyes09:57
mnasiadkaso then it seems to be a problem on the gateway09:58
mnasiadkaeither physical interface or some OVS/OVN flows misconfiguration09:58
mnasiadkayou can try resyncing the OVN NB DB with Neutron09:59
mnasiadkaRun 'neutron-ovn-db-sync-util --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --ovn-neutron_sync_mode repair' in neutron_server container10:00
blmttried that one yesterday but unless a global ovs cleanup/restart is needed we did not see any changes10:01
blmtweirdly enough the situation improved when we tried to stop the ovn/ovs services on all but 1 controller node10:02
blmtthat would lead to "some" flow misconfiguration that affects in some misterious way the FIP which should be residing on the compute nodes anyhow10:03
mnasiadkaare you sure you have distributed_floating_ip enabled in ml2.conf?10:03
blmtyep10:03
blmtand it all worked like a charm as well10:04
blmtuntil last week10:05
mnasiadkathat's very interesting ;)10:06
mnasiadkaand those VMs are on a Geneve type network, not VLAN?10:06
blmtall geneve10:06
opendevreviewMark Goddard proposed openstack/kolla-ansible master: WIP: Support custom services  https://review.opendev.org/c/openstack/kolla-ansible/+/81453010:08
mnasiadkablmt: then sorry, have no idea - maybe try on #openstack-neutron10:10
blmtthanks for trying anyhow :)10:11
eugenmayerare there any kolla specific docs on how to configure cinder (with lvm backend?) - or aree the cinder docs all i need. E.g. how to set the vgname in kolla?10:31
eugenmayerok found them, sorry https://docs.openstack.org/kolla-ansible/queens/reference/cinder-guide.html10:42
mnasiadkawell, I hope you're not deploying queens10:43
eugenmayerxena is my targeet - why the question?10:45
mnasiadkabecause you pasted link to queens release docs10:49
eugenmayeroh - google :)10:51
eugenmayeranyway to always go to latest .. replacing queens with latest does not work10:52
mnasiadkahttps://docs.openstack.org/kolla-ansible/latest/reference/storage/cinder-guide.html10:57
opendevreviewRadosław Piliszek proposed openstack/kolla-ansible master: [DNM] dummy  https://review.opendev.org/c/openstack/kolla-ansible/+/81455110:57
opendevreviewRadosław Piliszek proposed openstack/kolla-ansible master: [DNM] dummy 2  https://review.opendev.org/c/openstack/kolla-ansible/+/81455110:58
opendevreviewRadosław Piliszek proposed openstack/kolla-ansible master: [DNM] dummy 2  https://review.opendev.org/c/openstack/kolla-ansible/+/81455110:58
opendevreviewRadosław Piliszek proposed openstack/kolla-ansible stable/xena: [DNM] dummy 2  https://review.opendev.org/c/openstack/kolla-ansible/+/81446610:59
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: [DNM]: Test AIO without haproxy  https://review.opendev.org/c/openstack/kolla-ansible/+/81455311:03
*** amoralej is now known as amoralej|lunch11:05
eugenmayermnasiadka, thank you sir!11:22
mgoddardeugenmayer: google usually gets it right if you include the release name in your search11:23
eugenmayer"xena kolla-ansible cinder" gives me rocky first, so i guess not this time but i understand and will include the release name the next time of course11:25
opendevreviewMerged openstack/kayobe master: Set proxy option in early dnf invocation  https://review.opendev.org/c/openstack/kayobe/+/80224611:30
opendevreviewMark Goddard proposed openstack/kolla-ansible master: WIP: Support custom services  https://review.opendev.org/c/openstack/kolla-ansible/+/81453011:31
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: [DNM]: Test AIO without haproxy  https://review.opendev.org/c/openstack/kolla-ansible/+/81455311:31
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: mariadb: Remove wsrep-notify.sh  https://review.opendev.org/c/openstack/kolla-ansible/+/81439312:08
mflynnTrying to run kolla-build to build the images for train on centos 8 and seeing errors on several of the containers. Wonder if anyone has seen these.12:21
mflynnINFO:kolla.common.utils.openstack-base:  ERROR: Command errored out with exit status 1:12:21
mflynnINFO:kolla.common.utils.openstack-base:   command: /var/lib/kolla/venv/bin/python3.6 -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-zkmtzdbp/anyjson_0519f1b9fd034721aa3555dba16a629b/setup.py'"'"'; __file__='"'"'/tmp/pip-install-zkmtzdbp/anyjson_0519f1b9fd034721aa3555dba16a629b/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from 12:21
mflynnsetuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-90y77p4r12:21
mflynnINFO:kolla.common.utils.openstack-base:       cwd: /tmp/pip-install-zkmtzdbp/anyjson_0519f1b9fd034721aa3555dba16a629b/12:21
mflynnINFO:kolla.common.utils.openstack-base:  Complete output (1 lines):12:21
mflynnINFO:kolla.common.utils.openstack-base:  error in anyjson setup command: use_2to3 is invalid.12:21
mflynnINFO:kolla.common.utils.openstack-base:  ----------------------------------------12:21
mflynnINFO:kolla.common.utils.openstack-base:12:21
mflynnINFO:kolla.common.utils.openstack-base:WARNING: Discarding https://files.pythonhosted.org/packages/c3/4d/d4089e1a3dd25b46bebdb55a992b0797cff657b4477bc32ce28038fdecbc/anyjson-0.3.3.tar.gz#sha256=37812d863c9ad3e35c0734c42e0bf0320ce8c3bed82cd20ad54cb34d158157ba (from https://pypi.org/simple/anyjson/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.12:21
mflynnINFO:kolla.common.utils.openstack-base:12:21
mflynnINFO:kolla.common.utils.openstack-base:ERROR: Cannot install anyjson because these package versions have conflicting dependencies.12:21
mflynnINFO:kolla.common.utils.openstack-base:12:21
mflynnINFO:kolla.common.utils.openstack-base:The conflict is caused by:12:21
mflynnINFO:kolla.common.utils.openstack-base:    The user requested anyjson12:21
mflynnINFO:kolla.common.utils.openstack-base:    The user requested (constraint) anyjson===0.3.312:22
mflynnINFO:kolla.common.utils.openstack-base:To fix this you could try to:12:22
mflynnINFO:kolla.common.utils.openstack-base:1. loosen the range of package versions you've specified12:22
mflynnINFO:kolla.common.utils.openstack-base:2. remove package versions to allow pip attempt to solve the dependency conflict12:22
mflynnINFO:kolla.common.utils.openstack-base:ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies12:22
mflynnINFO:kolla.common.utils.openstack-base:12:22
mnasiadkamflynn: please, paste such things into paste.openstack.org12:29
mnasiadkaand we're aware Train is broken, it's just nobody has fixed it (since it's Train, EM, and so on)12:30
kplantare the images still floating around on docker hub for train-centos8 usable?12:32
mnasiadkaI guess they're from last known successful build, so should work12:32
kplantwould you instead recommend going from train/centos7 -> ussuri/centos8 ?12:33
kplantskipping train/centos812:34
opendevreviewMerged openstack/kayobe master: Ubuntu systemd-networkd: VLAN ifname heuristics  https://review.opendev.org/c/openstack/kayobe/+/81227112:34
kplantor is that even possible12:35
mnasiadkakplant: I don't think so, kolla-ansible prechecks probably won't allow you to do it12:43
mnasiadkabut I might be wrong ;)12:44
mnasiadkaand moving to an OS that will be EOL end of this year - well, this is up to you :)12:44
kplantwe don't really have a choice do we?12:45
kplanti didn't see any stream tags for centos 812:45
kplanttrain*12:45
mnasiadkastream is available from Victoria12:45
kplanti get that but we're on train/centos-7 now12:46
mnasiadkawell then you need to reprovision to centos 8, maybe you can survive by going directly to stream on the host OS, using already built Train images with CentOS Linux 8 and then forward to V/W - but I don't think anybody I know tested that approach12:48
mnasiadkaOr if you need to reprovision, look into Debian/Ubuntu ;)12:48
*** amoralej|lunch is now known as amoralej12:53
kplantor rocky!12:55
hrwtrain/c7 -> train/c8 -> ussuri/c8 -> victoria/cs8 -> wallaby/cs8 -> xena/cs8 -> yoga/cs[89]?13:00
mnasiadkaPTG starting now ;)13:00
mnasiadkakplant: no Rocky - yet13:00
hrwor just s/c8/cs8/ on train13:01
mnasiadkahrw: that sounds like a workflow to get to crazy land13:01
hrwofficial way is 'release by release', right?13:01
mnasiadkayes, but obviously you shouldn't stop on Train :)13:02
mnasiadkamgoddard, yoctozepto coming?13:02
hrwother way is setup new cluster on w/cs8, connect t/c7 compute/storage nodes to it and migrate control/storage/compute from t/c7 to w/cs813:02
kplanthrw: that's the plan, our problem is train/c8 fails to build13:05
hrwso use last published ones?13:05
* hrw -> PTG zoom13:05
kplanttrying to copy them in presently, also working around this awesome 200 pull limit :)13:06
kplantthough nobody's fault in here13:06
opendevreviewPierre Riteau proposed openstack/kayobe master: Add missing hook support for environment-create  https://review.opendev.org/c/openstack/kayobe/+/81457613:06
opendevreviewMark Goddard proposed openstack/kolla-ansible master: WIP: Support custom services  https://review.opendev.org/c/openstack/kolla-ansible/+/81453013:07
kevkoyoctozepto, mnasiadka: i'm limited with time and also has a vacation .. could I join now for couple of minutes ? 14:06
yoctozeptokevko: you can join always14:09
opendevreviewMerged openstack/kayobe master: Allow Docker to use insecure registry when deployed by Kayobe  https://review.opendev.org/c/openstack/kayobe/+/81216814:10
opendevreviewMerged openstack/kayobe master: Prevent Bifrost from using firewalld  https://review.opendev.org/c/openstack/kayobe/+/81109714:43
eugenmayerafter deploying with kolla, the controller suddenly cannot be reached via the management network.14:46
mgoddardeugenmayer: maybe the network interface got plugged into an OVS bridge? you might need to use a separate one for SSH/API14:49
eugenmayerhmm, not sure - i already have a dedicated interface for management14:58
eugenmayerthis is the one i also deploy via - so you mean i need an extra interface for the deployment itself?14:58
eugenmayermgoddard, ^^15:04
eugenmayerso in the end i need 3 networks: vm network, management network, deploy network?15:04
jingvargive me a second15:04
jingvaradmin_oc_net_name: mgmt 15:06
jingvarmgmt_vlan: 32215:06
eugenmayernot sure what that is jingvar - globals.yml?>15:06
jingvarnetworks .yaml15:06
jingvarthere few layers of networks15:06
eugenmayerwhere do i put this?15:07
eugenmayeris that file in the /opt/kolla folder i installed kollab-ansible in?15:07
jingvarnetworks have logical role and phishical ip assigment and finaly interface mapping15:08
jingvarkayobe-config/etc/kayobe15:08
eugenmayerand the fix above creates a vlan for the mgmnt network - or what is it for?15:08
jingvarquestion is what is your network schema15:10
jingvarhttps://github.com/jingvar/a-universe-from-nothing/commit/fe9c5d51f632b2063fee12b82d9d68d5901efcee15:10
eugenmayercurrently i have 172.30.0.0/24 for the vm network, 10.0.0.0/24 for the management network. All boxes (deploy,controller, compute1,compute2) are parts of all those networks15:10
jingvaran example for separate networks15:10
eugenmayerjingvar, https://github.com/EugenMayer/openstack-lab/blob/master/Vagrantfile#L415:11
eugenmayerhttps://github.com/EugenMayer/openstack-lab/blob/master/deploy/3_configure_kolla.sh#L2315:11
eugenmayerhttps://github.com/EugenMayer/openstack-lab/blob/master/deploy/3_configure_kolla.sh#L3115:11
eugenmayeroh, i got it15:12
eugenmayernet vm network is enp0s9 - so i have to put the neutron_intereface on enp0s9 ?15:12
eugenmayerenp0s8 is the mngmnt interface15:12
eugenmayerthe neutron_external_interface and the kolla_external_vip_interface must be different interfaces ,r ight15:14
eugenmayerapi_interface should be enp0s8 too (mngmnt)15:15
eugenmayeractually i'am not sure anymore if network_interfadce should not be equal to the mngmnt interface, sounds right to me15:15
opendevreviewMark Goddard proposed openstack/kolla-ansible master: WIP: Support custom services  https://review.opendev.org/c/openstack/kolla-ansible/+/81453015:15
eugenmayerjust not sure about neutron_external_interface - should this map to the provider interface or the one i share for the vms?15:15
eugenmayertrying to understand https://docs.openstack.org/kolla-ansible/latest/admin/production-architecture-guide.html#network-configuration but not sure i do15:16
jingvarooo, I 'm not shure that is good doc15:18
jingvarkolla/inventory/overcloud/host_vars/cmp01:neutron_external_interface: "bond0"15:24
jingvarkayobe/inventory/group_vars/compute/network-interfaces:external_interface: bond015:26
eugenmayerbond?15:26
jingvarin my case yes, I have several compute node  types15:26
jingvarkolla/inventory/overcloud/host_vars/cmp01-ucs:neutron_external_interface: "eth2"15:27
jingvarkayobe/inventory/group_vars/compute_ucs/network-interfaces:external_interface: eth215:28
eugenmayerthe question is, what is eth2 semantically, your management interface or the shared bridget/interface for all compyte nodes (shared vm network later on)15:28
jingvareth2 is dedictated interface for neutron - where it can bring flat, vlan etc for external traffic15:30
jingvarI m not shure that on this interface can be other networks15:31
jingvarin my perspective your configuration is all networks (mgmt,provision etc) on first interface and second one for neutron15:33
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: mariadb: Remove wsrep-notify.sh  https://review.opendev.org/c/openstack/kolla-ansible/+/81439315:42
eugenmayerjingvar, thank you -i assume similar15:54
eugenmayerok it deployed somewhat, but i have a load of 50, guess it has hogged the memory, lets see15:57
em_can someone shortly explain, if all-in-one regards to "controller and compute on one node" or rather that the controller will not be HA but only be 1 single controller, still the compute nodes are seperate nodes anyway16:23
mnasiadkaem_: the first16:26
em_thank you!16:32
opendevreviewRadosław Piliszek proposed openstack/kolla-ansible master: [DNM] Test Debian all  https://review.opendev.org/c/openstack/kolla-ansible/+/81461016:41
eugenmayerafter deploying with colla, i have a massive load on the controller. Seems the erlang based beam is the reason - any hints on what is happening?16:41
eugenmayercompute nodes are idle, deploy also. Just the controller runs hot since about 30 minutes16:42
eugenmayerit has 4 cores of an AMD Ryzen 7 5800X 8-Core and 4GB of ram (test setup) - seems more then enough16:42
eugenmayertrying to givee it 5 cores with 10GB of ram, but that is all somewhat unexpected for something that is basically blank (no vms, no nothing)16:46
jingvarMem:          503Gi       9.4Gi       486Gi       101Mi       8.0Gi       491Gi16:47
jingvar[root@ctl02 ~]# free -h16:48
jingvarit is fresh deployment 16:48
jingvarbut with additional services16:48
eugenmayerinteresting, so 10GB is about right?16:48
jingvaryep16:48
eugenmayerlets see16:49
eugenmayerThank you for the hint16:49
jingvarwhen you have problem with memory comes OOM kill some containers , docker start them again ... loop , high cpu 16:50
jingvarAs I remember - "a multiverse from nothing" deploy controllers with 8G RAM16:51
eugenmayerwell it is a little unusual to run a local desktop lab with 18GB of ram usagee, but well, at least i have 64GB ram here in my desktop, so should be fine16:53
jingvarin the beggining16:54
-opendevstatus- NOTICE: Both Gerrit and Zuul services are being restarted briefly for minor updates, and should return to service momentarily; all previously running builds will be reenqueued once Zuul is fully started again17:00
eugenmayerIt worked this time. Great!17:05
eugenmayerwhere do i find the initial credentials when deploying wit kolla?17:12
jingvardo you mean .rc file?17:25
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: mariadb: use add_host to include inactive hosts in shard grouping  https://review.opendev.org/c/openstack/kolla-ansible/+/81427617:34
opendevreviewMark Goddard proposed openstack/kolla-ansible master: WIP: Support custom services  https://review.opendev.org/c/openstack/kolla-ansible/+/81453019:56
opendevreviewSeena Fallah proposed openstack/kolla-ansible master: mariadb: use add_host to include inactive hosts in shard grouping  https://review.opendev.org/c/openstack/kolla-ansible/+/81427622:15
opendevreviewSeena Fallah proposed openstack/kolla-ansible master: mariadb: use add_host to include inactive hosts in shard grouping  https://review.opendev.org/c/openstack/kolla-ansible/+/81427622:28

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!