Friday, 2020-02-07

*** Tengu has quit IRC00:04
*** Tengu has joined #openstack-kolla00:06
*** k_mouza has quit IRC00:41
*** k_mouza has joined #openstack-kolla00:43
*** jbadiapa has quit IRC01:37
*** dking_desktop has joined #openstack-kolla01:57
*** k_mouza has quit IRC02:11
*** jcmdln has joined #openstack-kolla02:26
*** xinliang has quit IRC02:55
*** xinliang has joined #openstack-kolla02:58
*** cah_link has joined #openstack-kolla03:23
*** xinliang has quit IRC03:33
*** cah_link1 has joined #openstack-kolla03:39
*** cah_link has quit IRC03:40
*** xinliang has joined #openstack-kolla03:41
*** cah_link1 has quit IRC03:43
*** cah_link has joined #openstack-kolla03:43
*** cah_link has quit IRC03:47
*** cah_link has joined #openstack-kolla03:48
*** cah_link1 has joined #openstack-kolla04:23
*** cah_link has quit IRC04:24
*** cah_link1 is now known as cah_link04:24
*** cah_link1 has joined #openstack-kolla04:27
*** cah_link has quit IRC04:29
*** cah_link1 is now known as cah_link04:29
*** skramaja has joined #openstack-kolla04:38
*** Tony31 has joined #openstack-kolla04:42
*** cah_link has quit IRC04:47
Tony31Hi is anyone here yet?05:08
*** evrardjp has quit IRC05:34
*** evrardjp has joined #openstack-kolla05:34
*** shyamb has joined #openstack-kolla05:54
*** numans has quit IRC06:02
*** skramaja has quit IRC06:04
yoctozeptoTony31: that was very early, we only really start opening our eyes at that time :-)06:31
Tony31no excuse ;)06:31
Tony31is it possible to use `host_vars` to set a variable for a host? My problem is that, the controller VM is not PXEboot, so it does not have a PXE interface and the command `kayobe overcloud host configure` fails because it validates this interface. So I wanted to set this for the controller host only: `provision_oc_net_name: inside` but I tried tons06:33
Tony31of things and none of them resolved.06:33
Tony31About 30 mins ago I changed this setting in network-allocation.yml :/06:33
*** shyamb has quit IRC07:02
*** shyamb has joined #openstack-kolla07:03
*** jbadiapa has joined #openstack-kolla07:05
*** ricolin has quit IRC07:11
*** shyamb has quit IRC07:22
*** cah_link has joined #openstack-kolla07:22
*** shyamb has joined #openstack-kolla07:35
hrwmorning07:41
hrwyoctozepto: infra passed basic CI :D07:41
*** bengates has joined #openstack-kolla07:49
*** rpittau|afk is now known as rpittau08:10
yoctozeptohrw: yay08:14
yoctozeptowhat was wrong then?08:15
hrwdockerfile rewriting08:15
yoctozeptohmm, oddly it somehow worked locally08:15
hrwyep08:15
hrwk-a change will be tricky08:15
hrwand then merging all after c8 work08:15
yoctozeptoTony31: need to wait for mnasiadka and/or mgoddard, question too specific to kayobe08:16
mnasiadkamorning08:16
*** carli has joined #openstack-kolla08:17
Tony31good morning :)08:17
Tony31TBH I don't like just waking up and having things demanded of me, so apologies for that08:17
*** gfidente has joined #openstack-kolla08:18
mnasiadkaTony31: no problem, you can override the variables on host_vars/group_vars level - so it should work08:19
Tony31wouldnt work :(08:20
mnasiadkaTony31: under inventory/host_vars/host_name/network.yml just put the overrides08:20
Tony31I was using host_vars/hostname.yml as well as host_vars/compute0 and host_vars/compute0.yml and I also copied that to the kolla directory.... but having read your update now I think I Was doing it wrong08:20
Tony31I'll try it - although I am halfway through deploying at the moment and it's failing waiting for mariadb08:21
Tony31it shows an error that its waiting to connect to hostname-int.domain:3306 but hostname-int.domain is not dns resolvabe and there are no /etc/host entries for it. I just put in a host entry - how is this part supposed to work?08:22
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: WIP: Create 'infra' type of images  https://review.opendev.org/69875308:22
hrwmoved aarch64 job to ubuntu08:22
*** tonythomas has joined #openstack-kolla08:23
*** negronjl has quit IRC08:23
Tony31the /etc/hosts didnt resolve that issue08:24
Tony31this host will not be reachable from the ansible host... `TASK [mariadb : Waiting for MariaDB service to be ready through VIP] **`08:25
*** negronjl has joined #openstack-kolla08:27
openstackgerritRadosław Piliszek proposed openstack/kolla master: Add collectd-pmu package for Intel PMU (C7 only)  https://review.opendev.org/70456008:27
*** shyamb has quit IRC08:27
Tony31another problem I am seeing is the controller is not getting configured on the storage network. for tripleo, when an instance is built, the controller logs into the iscsi target and installs the image into it. Then the iscsi is moved to the host so the instance can boot from it. Which kayobe `network` is this for? I guess I have this listed08:28
Tony31incorrectly in the networks section, so it's using another network instead of my storage net.08:28
Tony31mnasiadka `inventory/host_vars/hostname/network.yml` is also not working08:30
Tony31my hostname is juc-kcont1-prd for the controller08:30
mnasiadkaso it's inventory/host_vars/juc-kcont1-prd/network.yml?08:31
Tony31wait, now there's only one error instead of two08:31
mnasiadkaand are you sure your controller in inventory/overcloud is juc-kcont1-prd?08:31
Tony31so it looks like it's half working - i'll check spelling08:31
mnasiadkanot by ip or something?08:31
openstackgerritRadosław Piliszek proposed openstack/kolla master: Add collectd-pmu package for Intel PMU (C7 only)  https://review.opendev.org/70456008:31
Tony31The error is `failed: [juc-kcont1-prd] (item={u'var_name': u'kolla_inspector_dnsmasq_interface', u'description': u'Bare metal introspection network interface name', u'required': True}) => {"ansible_loop_var": "item", "changed": false, "item": {"description": "Bare metal introspection network interface name", "required": true, "var_name":08:32
Tony31"kolla_inspector_dnsmasq_interface"}, "msg": "The Kolla Ansible variable kolla_inspector_dnsmasq_interface (Bare metal introspection network interface name) is invalid. Value: \"<undefined>\".\n"}`08:32
Tony31do you know which network does this relate to in `networks.yml` ? I thought this was `inspection_net_name:`08:32
Tony31mnasiadka I am not using the seed baremetal inventory, so that file was empty... is it wrong?08:33
mnasiadkaTony31: well, you have no overcloud nodes to deploy openstack to? :)08:36
Tony31:)08:36
Tony31understood - I put in the hostname08:36
*** ivve has joined #openstack-kolla08:37
mnasiadkaTony31: one moment, so you had this node in inventory/hosts, right? under controllers group?08:39
mnasiadkaTony31: or the seed/seed-hypervisor group?08:39
Tony31inventory/hosts as a controller08:39
yoctozeptomnasiadka, hrw: https://review.opendev.org/70456008:40
patchbotpatch 704560 - kolla - Add collectd-pmu package for Intel PMU (C7 only) - 5 patch sets08:40
Tony31in the controllers group08:40
mnasiadkaok, the seed/seed-hypervisor groups are empty?08:40
Tony31no, [seed] has the seed in there. but i manage the VMs (seed / compute / controller / ansible) manually in terms of installing the OS and setting up SSH etc08:41
Tony31seed hypervisor is empty08:41
mnasiadkayoctozepto: once zuul passes I'll push it :)08:41
mnasiadkaTony31: well, if you don't need to install OS on anything, I suppose you don't need the seed08:41
mnasiadkabut I might be wrong, I'm not so knowledgable in kayobe as Mark is08:42
Tony31i wanted to use the seed to build out the real physical compute systems later08:42
Tony31seed seems to be working ok08:42
mnasiadkaah ok08:42
mnasiadkaTony31: so, it would be good when sharing errors to include the task/play name it fails on - it's easier for us to help you :)08:43
Tony31i can't get passed this error though `failed: [juc-kcont1-prd] (item={u'var_name': u'kolla_inspector_dnsmasq_interface', u'description': u'Bare metal introspection network interface name', u'required': True}) => {"ansible_loop_var": "item", "changed": false, "item": {"description": "Bare metal introspection network interface name", "required":08:43
Tony31true, "var_name": "kolla_inspector_dnsmasq_interface"}, "msg": "The Kolla Ansible variable kolla_inspector_dnsmasq_interface (Bare metal introspection network interface name) is invalid. Value: \"<undefined>\".\n"}`08:43
Tony31`TASK [Validate overcloud host Kolla Ansible network configuration] **`08:43
Tony31I created that host_vars file and I have :08:43
Tony31`provision_oc_net_name: inside08:43
Tony31initially I had an error about the provision network - I dont get that error anymore08:44
Tony31I'm not sure if the `inspection_net_name:` is the right one to be using... maybe it is not since I am getting the error but I can't link that error to a network08:44
mnasiadkaTony31: do you have kolla_enable_ironic set to True?08:44
Tony31where would it be set?08:45
Tony31i can check08:45
mnasiadkain kolla.yml08:45
Tony31no - the only thing enabled in the kolla.yml file is the ansible bug thing08:45
Tony31`workaround_ansible_issue_8743: yes`08:46
mnasiadkaanyway - just set kolla_inspector_dnsmasq_interface variable to some interface08:51
mnasiadkayoctozepto: we still haven't fixed fluentplugin-elasticsearch on Ubuntu/rocky?08:54
yoctozeptomnasiadka: yeah, probably08:54
*** k_mouza has joined #openstack-kolla09:00
*** k_mouza has quit IRC09:05
*** dougsz has joined #openstack-kolla09:06
Tony31where's `kolla_inspector_dnsmasq_interface` ? is there a kayobe variable for this ?09:17
Tony31I searched the kayobe directory and these are the results:09:17
mgoddardmorning09:17
mgoddardevening Tony31 :)09:17
hrwhttps://review.opendev.org/#/c/706428/1 - brexit everywhere...09:17
patchbotpatch 706428 - opendev/system-config - Replace nb03.openstack.org - 1 patch set09:17
Tony31[cv-user@juc-kach1-prd kayobe]$ grep -rn -e "kolla_inspector_dnsmasq_interface"09:18
Tony31"kolla_inspector_dnsmasq_interface"09:18
Tony31venvs/kayobe/share/kayobe/ansible/roles/kolla-ansible/tests/test-extras.yml:27:        kolla_inspector_dnsmasq_interface: "eth9"09:18
Tony31all results are in the `venv`09:18
Tony31was expecting a `kayobe-config`09:19
hrwyoctozepto: +209:19
mgoddardTony31: bonus 'feature' of kayobe is that it enables ironic by default. we should probably change that09:21
mgoddardTony31: try setting `kolla_enable_ironic: no` in your etc/kayobe/kolla.yml09:22
*** shyamb has joined #openstack-kolla09:22
*** shyamb has quit IRC09:24
*** shyamb has joined #openstack-kolla09:25
openstackgerritNoboru Iwamatsu proposed openstack/kolla-ansible master: Adapt to Octavia Certificate Configuration Guide.  https://review.opendev.org/70620309:28
Tony31cheers - so now ironic is turned off, do I need to manually install all my hosts? Or can I pxe some?09:31
*** k_mouza has joined #openstack-kolla09:32
Tony31hrw - under sea cables are always breaking... I'm always getting issues going from WA to Sri Lanka. Sometimes, the only path is through USA and across Europe.09:35
Tony31Capacity is a common issue also these days. Not enough of it09:35
hrwTony31: cables...09:36
Tony31even if there are mutliple under sea cables, we cant expect to use them. It depends on who owns the cables and whom they are leased to / or which ISPs are paying "rent" to use them.09:37
hrwTony31: I remember times when it was easier to cross a street with data than send it over network09:37
hrwTony31: as data had to pass around country09:37
Tony31Today thats mostly true for cloud migrations :)  When there are TB of data for example. I'm lucky as I've only been doing this about 15 years.09:38
hrw:)09:38
hrwTony31: that's why Amazon still has truck-as-a-service for migrating data to the cloud09:39
Tony31:)09:40
Tony31I need to speak with you guys about a bunch of things:09:40
Tony311. the ` kayobe overcloud host configure` is now passing with ironic = off.09:40
Tony312. can I no longer use ironic now?09:41
Tony31(no real issue if not) I just need to plan it out09:41
Tony31when trying to deploy the overcloud, the compute node is not being touched.. not sure why. May be it's set up after the controller is deployed09:41
Tony314. The controller is not getting the storage network configured into it and I am not sure why... I need the storage network to be able to build instances.09:42
hrwyou lost point 3.09:42
Tony313. when trying to deploy the overcloud, the compute node is not being touched.. not sure why. May be it's set up after the controller is deployed :)09:42
Tony31for point 3, in inventory/overcloud when I was trying to use baremetal it had put both controller and compute node under `[controllers]` I removed both of these when I deleted the 2 x baremetal nodes from the seed09:44
Tony31for point 4, which network is used for building instances from glance?09:44
Tony315. `kayobe overcloud service deploy` fails because it's waiting for mariadb and fails with error about connecting to the host:3306. How is DNS managed here? Do I need to create the dns entry or host entry manually?09:46
mgoddard1. good09:47
mgoddard2. no ironic for baremetal instances, but yes for deploying controllers/hypervisors via the seed09:48
mgoddard3. sounds like the inventory is not setup correctly. Compute nodes needs to be in the compute group09:48
Tony31I thought ironic was used for deploying hypervisors?09:48
mgoddardit is09:48
mgoddardbut that is running in the seed09:49
Tony31sorry for my confusion - I've just turned it off in kolla.yml09:49
mgoddardkolla_enable_ironic allows you to deploy another ironic on the controllers which can be used to deploy baremetal compute nodes (not hypervisors, users get bare metal)09:49
Tony31ok. thanks for that - I still am not quite sure but it will probably make sense when I see it later09:50
Tony31:)09:50
Tony31about the inventory, i have this: `[baremetal-compute]# Add baremetal compute nodes here if required.juc-kcomp1-prd`09:50
Tony31the baremetal-compute is the only "compute" section09:50
Tony31is it correct?09:50
Tony31deployment now fails with `TASK [nova-cell : Checking available compute nodes in inventory] *************************************************************************************************************************09:51
Tony31disabled.\n"}09:51
Tony31`¯\_(ツ)_/¯`09:51
mgoddardah, no09:52
mgoddardjust [compute]09:52
Tony31ah - it's not there. i'll add it09:52
hrwhm. we have not merged yet erlang/rabbitmq changes, right?09:52
mgoddard[baremetal-compute] is for baremetal compute nodes, not hypervisors09:53
mgoddardTony31: have you seen https://docs.google.com/presentation/d/1L6KHoqcQi835Idix2xO_XzlgSn7nUk5DTr71A7dZ5vk ?09:53
mgoddardin particular, https://docs.google.com/presentation/d/1L6KHoqcQi835Idix2xO_XzlgSn7nUk5DTr71A7dZ5vk/edit#slide=id.g347582824b_0_45309:53
hrwok, have to create train patch from https://review.opendev.org/695356 one...09:54
patchbotpatch 695356 - kolla - rabbitmq: update to latest 3.7.x (+ erlang 22.x) (ABANDONED) - 16 patch sets09:54
Tony31mgoddard I hadnt seen that...09:54
Tony31My understanding was "compute" was a hypervisor for the instances09:54
mgoddardyes09:55
mgoddardwhich is what you want09:55
Tony31yes09:55
Tony31but the ACH doesnt deploy it via seed?09:56
Tony31the controller deploys it?09:56
Tony31wait09:56
Tony31are you saying `baremetal-compute` is for deploying instances from openstack, not as VMs but onto the entire bare-metal ?09:56
mgoddardyes09:57
Tony31ah I understand now09:57
Tony31thank you09:57
mgoddardnp09:57
yoctozeptohrw: train patch?09:59
hrwyoctozepto: it will land on master first but I need it in train10:01
yoctozeptohrw: mgoddard will block you10:01
hrwyoctozepto: Linaro Developer Cloud migrates to Train so we need patched libvirt so we need to add repo10:01
yoctozeptolibvirt? you mentioned erlang+rmq10:01
hrwyoctozepto: once patch is in gerrit I have infra to using them on CI :D10:02
hrwyoctozepto: repo info taken from that patch ;d10:02
yoctozeptohrw: ah10:03
*** Tony312 has joined #openstack-kolla10:03
Tony312i got kicked out and i cant log in because my session hasnt timed out yet10:03
*** Tony31 has quit IRC10:03
hrwyoctozepto: https://paste.centos.org/view/f907f0ed10:04
Tony312cant hang around right now - will try and come back online later but if i cant then I'll send an email. Thanks again guys for all your help.10:05
*** Tony312 has quit IRC10:10
*** devfaz has quit IRC10:10
mnasiadkayoctozepto: https://review.opendev.org/#/c/706055 +w or not +w? :)10:10
patchbotpatch 706055 - kolla-ansible - CI: Replace cinder-lvm scenario with zun scenario - 1 patch set10:10
mnasiadkait's a bit blocking a nice queue of centos8 changes10:11
yoctozeptomnasiadka: +w but merge me the kolla one too10:11
yoctozeptomnasiadka: https://review.opendev.org/70605910:11
patchbotpatch 706059 - kolla - CI: Replace cinder-lvm scenario with zun scenario - 7 patch sets10:11
*** theZorro266 has quit IRC10:15
mnasiadkayoctozepto: the overlapping subnet thing - I suppose you'll follow up?10:17
yoctozeptomnasiadka: I have no idea why it popped there10:17
yoctozeptomnasiadka: it's on ansible side anyway10:18
yoctozeptomnasiadka: looks random (it worked in previous run above)10:18
yoctozeptomnasiadka: so just merge10:19
mnasiadkamgoddard: ^^ - keen on +2+w?10:21
mgoddardkolla needs to go first10:22
yoctozeptomgoddard: yeah, you are asked about kolla10:23
mgoddarddone10:24
yoctozeptomgoddard: tyvm10:24
mgoddardyvw10:24
*** jovial[m] has quit IRC10:27
*** aleccoder has quit IRC10:27
yoctozeptomgoddard, mnasiadka, hrw: though I notice kolla is not gating on k-a jobs in this very change10:32
yoctozeptoarar10:32
mgoddardit should gate on aio source jobs10:32
yoctozeptomgoddard: it does not, probably another zuul bug eh10:33
mgoddardyoctozepto: '\..' listed in irrelevant files10:34
mgoddard.zuul/10:34
yoctozeptomgoddard: ah, true10:34
mgoddardmaybe we should rename it to zuul.d10:37
mgoddardwould be more consistent10:37
mgoddardand less obtuse10:37
*** shyamb has quit IRC10:38
mnasiadkaso just go for it, simple change :)10:40
hrwmeh. infra images will be untestable in k-a on aarch64 ;(10:40
hrw2h needed to just build and we do not publish them10:40
hrwmnasiadka: and pile of merge conflicts on gerrit ;D10:41
mnasiadkahrw: good pile is not bad :)10:41
mnasiadkahrw: so why don't you publish them from Linaro CI? :D10:41
hrwmnasiadka: I do not think that you want to check which tag is latest?10:42
hrwmnasiadka: linaro/debian-source-base:ussuri-235, linaro/debian-source-base:train-ldc-2, linaro/debian-source-base:rocky-lcd-1210:42
hrwmnasiadka: linaro/debian-source-base:ussuri-235, linaro/debian-source-base:train-ldc-2, linaro/debian-source-base:rocky-ldc-1210:42
mnasiadkahrw: I was rather thinking of publishing them on Docker Hub with our normal approach?10:43
hrwmnasiadka: will you share docker credentials?10:43
hrwI would not10:43
hrwmnasiadka: also we do weekly builds not daily10:44
hrwand if they fail to build I complain upstream and send patches. if they fail to publish I just do not care10:44
mnasiadkathis is not the best process I guess :)10:45
*** devfaz has joined #openstack-kolla10:45
*** priteau has joined #openstack-kolla10:46
yoctozeptore BadRequestException: 400: Client Error for url: http://192.0.2.10:9696/v2.0/subnets, Invalid input for operation: Requested subnet with cidr: 10.0.0.0/24 for network: e7c1bd4a-a9bc-40bf-beaa-8a27f6ead1b0 overlaps with another subnet.10:47
yoctozeptoit's an effect of "Lost connection to MySQL server during query" in neutron-server10:47
yoctozeptoit seemingly "forgot" it did the insert10:47
hrwmnasiadka: fine for us10:48
*** admin0 has quit IRC10:49
priteauHi. I am looking into a problem with CloudKitty (Stein) returning only 10,000 dataframes entries when using InfluxDB as the storage backend.10:51
priteauThis appears to be directly linked to the InfluxDB configuration set by Kolla Ansible: https://opendev.org/openstack/kolla-ansible/src/branch/master/ansible/roles/influxdb/templates/influxdb.conf.j2#L4510:51
priteauUpstream InfluxDB has changed the default to 0 (unlimited) a while ago: https://docs.influxdata.com/influxdb/v1.7/administration/config/#max-row-limit-010:51
priteauAnyone know the potential impact of changing this setting in Kolla Ansible? dougsz?10:52
mnasiadkahrw: around dockerhub creds - zuul has those10:54
hrwmnasiadka: I am aware10:54
priteauOur default setting of 10000 rows hasn't changed since the initial import of the influxdb role in 201610:56
dougsz priteau, I have no objection to matching the upstream configuration.10:56
priteauAny potential impact on Monasca?10:57
dougszWe have generally been tuning infludb config via an override10:57
dougszI think query-timeout is probably most important - which we leave untouched in KA by default10:58
hrwhttps://paste.centos.org/view/a4dfef72 is quite ugly for master ;(10:58
* hrw off11:00
*** shyamb has joined #openstack-kolla11:00
dougsz priteau: I guess this is Cloud Kitty doing some mega query rather than in chunks?11:01
priteauThere is code to paginate but it doesn't appear to be used in this version11:04
*** shyam89 has joined #openstack-kolla11:05
*** mleone87 has joined #openstack-kolla11:06
*** shyamb has quit IRC11:06
mnasiadkapriteau: looking at the docs I think we can safely remove that setting - and then it will take the default (0 in newer influxdb and 10000 in previous versions)11:06
yoctozepto+1 for aligning with upstream11:08
mnasiadkaand it's in the http backend options, so I guess it doesn't change anything in the database11:11
*** aleccoder has joined #openstack-kolla11:12
priteauIndeed I think it's only for queries, I was just wondering if it could break some InfluxDB clients by making queries return massive amounts of data.11:12
priteauIt fixes the issue for CloudKitty at least.11:13
*** rpittau is now known as rpittau|bbl11:18
*** shyamb has joined #openstack-kolla11:25
*** shyam89 has quit IRC11:28
openstackgerritMerged openstack/kolla master: Add collectd-pmu package for Intel PMU (C7 only)  https://review.opendev.org/70456011:41
*** shyam89 has joined #openstack-kolla11:50
*** shyamb has quit IRC11:53
*** xinliang has quit IRC11:54
*** kplant has joined #openstack-kolla12:02
*** shyamb has joined #openstack-kolla12:09
*** shyam89 has quit IRC12:11
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: CI: Debug init-runonce  https://review.opendev.org/70648212:18
yoctozeptomgoddard, mnasiadka: making init-runonce less of a black box ^12:19
mnasiadkayoctozepto: maybe I'm a bit idealistic - but wouldn't it be better to rewrite it to Ansible?12:20
yoctozeptomnasiadka: would be, and to ditch the openstack python client in favor of ansible modules yeah12:21
mgoddardthere was a patch to do that12:21
yoctozeptolet's find it12:21
yoctozeptohttps://review.opendev.org/52307212:22
patchbotpatch 523072 - kolla-ansible - ansiblize the init-runonce script - 10 patch sets12:22
yoctozepto"Otherwise.. if it isn't broken..."12:23
mnasiadkawell, shade is not needed anymore12:23
yoctozeptomnasiadka: well, it's openstacksdk now12:24
mnasiadkawell, we all know it's broken-ish12:25
mgoddardtbh, ansible openstack modules are more of a black box than the openstack CLI12:34
mgoddarderrors are rarely helpful12:34
mgoddardand I'm not sure we should give the impression this is for production use12:34
mgoddardyoctozepto, mnasiadka: ^12:34
yoctozeptomgoddard: for now let's just add debugging caps to the script and use them12:35
mnasiadkawell, I wouldn't even give the impression for users using init-runonce script - especially the beginning ones :)12:35
mnasiadkaI would prefer them to use some doc to setup basic image/network structures12:35
mnasiadkaand move init-runonce under tests/12:36
mgoddardwe could do that12:36
mgoddardalthough temp symlink would be nice since kayobe uses it12:36
mnasiadkayeah, like a normal deprecation process12:36
mnasiadkagenerally we have a lot of things under tools/ that give the impression that a user could use it12:38
*** k_mouza has quit IRC12:39
*** k_mouza_ has joined #openstack-kolla12:39
yoctozeptomnasiadka: yeah, like kolla-ansible12:41
yoctozeptowho would use that12:41
mnasiadkaI wasn't really mentioning it12:41
mnasiadkabut setup-compute-libvirt.yaml?12:41
yoctozeptomnasiadka: setup_gate :-)12:41
mnasiadkayeah, I'm a newbie and will run ./setup_gate.sh12:42
mnasiadkaI hope it will deploy a gateway12:42
mnasiadka:D12:42
yoctozeptovirt-guest-shutdown.target12:42
yoctozeptotroll directory12:42
mnasiadkaI'll analyse what is there, move most of the stuff to contrib/ (because I feel it belongs there)12:42
yoctozeptomnasiadka: or even in the "recycle bin"12:44
mnasiadkacontrib/trash12:44
mnasiadka:)12:44
*** ktibi has joined #openstack-kolla12:52
mnasiadkayoctozepto, mgoddard: do we use init-vpn anywhere? :)13:07
*** shyamb has quit IRC13:09
*** rpittau|bbl is now known as rpittau13:14
*** hongbin has joined #openstack-kolla13:15
*** Luzi has joined #openstack-kolla13:16
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: Tidy up tools/ subdirectory  https://review.opendev.org/70649613:17
openstackgerritMerged openstack/kolla master: CI: Replace cinder-lvm scenario with zun scenario  https://review.opendev.org/70605913:21
*** mleone87 has quit IRC13:31
mgoddardmnasiadka: not that I'm aware of13:34
*** mleone87 has joined #openstack-kolla13:36
mgoddardmnasiadka: can you remove W-1 from https://review.opendev.org/#/c/706055/113:36
patchbotpatch 706055 - kolla-ansible - CI: Replace cinder-lvm scenario with zun scenario - 1 patch set13:36
mnasiadkamgoddard: done13:37
mgoddardthanks13:37
yoctozeptomnasiadka: vpn? whaaa? what does it do?13:40
mnasiadkayoctozepto: init vpnaas it seems13:42
*** ktibi has quit IRC13:43
*** ktibi has joined #openstack-kolla13:45
*** hongbin has quit IRC13:54
yoctozeptomnasiadka: verrry interesting13:59
*** yan0s has joined #openstack-kolla14:00
openstackgerritMerged openstack/kolla master: Add ssh clients to Ironic Conductor container  https://review.opendev.org/70614914:02
mnasiadkayoctozepto: not really, given vpnaas is dead-ish14:05
*** jovial[m] has joined #openstack-kolla14:11
yoctozeptomnasiadka: oh, well14:15
*** pbing19 has joined #openstack-kolla14:18
*** rlljorge has joined #openstack-kolla14:28
rlljorgeHello, I am trying enable fwaas but I docker neutron_server not start with error Error loading class by alias: NoMatches: No 'neutron.service_plugins' driver found, looking for 'firewall' .... the full log in http://paste.openstack.org/show/789292/14:29
*** ktibi has quit IRC14:39
yoctozeptorlljorge: which release of kolla-ansible against which release of kolla?14:40
yoctozeptolooks like wrong version used14:40
rlljorgeyoctozepto kolla-ansible 8.0.014:42
yoctozeptorlljorge: why 8.0.0? and not 8.1.0? https://pypi.org/project/kolla-ansible/#history14:42
yoctozeptoI assume you run against stein?14:43
rlljorgeyoctozepto is Stein release14:43
rlljorgeO will upgrade to 8.1.0 now14:43
*** Luzi has quit IRC14:45
yoctozeptomgoddard, mnasiadka: keepalived is the current ci job rapist; it looks like we have a high chance of failing if keepalived ever loses its vip because services do not seem to like mariadb connections breaking14:52
*** ktibi has joined #openstack-kolla14:57
mgoddardlanguage yoctozepto15:03
*** rpittau is now known as rpittau|brb15:04
yoctozeptomgoddard: where? ;o15:04
mgoddardnot conducive to an inclusive community15:05
mgoddardany idea why we are seeing this now and not previously?15:06
*** pbing19 has quit IRC15:07
yoctozeptomgoddard: had to lookup the smart "c" word15:07
mgoddardsorry15:07
yoctozeptomgoddard: no idea15:07
yoctozeptoit happened now and then though15:07
yoctozeptoso it's not something new15:07
*** TrevorV has joined #openstack-kolla15:08
yoctozeptobut having fixed mariadb and keystone we are left with keepalived :-)15:08
mgoddardwhy would keepalived lose it's vip? upgrade?15:08
yoctozeptomgoddard: nah, haproxy check returning 1 or even better - signal 1515:09
mnasiadkayoctozepto: can we for now introduce an option to disable this bloody hcheck script and investigate what is happenning with ghis fine piece of artwork code in the background?15:09
yoctozeptoon upgrade we restart services anyway15:09
mnasiadkaIs bloody an inclusive wording?15:09
yoctozeptomnasiadka: I would rate it the same as the "r" word15:10
yoctozeptomnasiadka: usually safe but not quite :-)15:10
*** ktibi has quit IRC15:10
mnasiadkaWe need an inclusive rating bot15:10
yoctozeptoyeah, that would clear up things; surely our language standards in Poland are much more relaxed...15:11
yoctozeptomnasiadka: you propose to disable just in CI? how's it going to help?15:11
mnasiadkayoctozepto: I think mgoddard would be very freightened if we adapted to the style 80% of Polish people speak :)15:12
mnasiadkayoctozepto: what is the alternative? Add logging to the script and check what is happening?15:12
yoctozeptomnasiadka: for sure, thankfully we are in those 20%, aren't we? ;D15:12
mgoddardbloody is fine IMO15:13
yoctozeptomnasiadka: it connects to haproxy via unix socket15:13
mgoddardbut I am no expert15:13
yoctozeptonot much going on there15:13
mnasiadkaNot mentioning that this script is a bit lame15:13
yoctozeptowell, it would move vip if you killed haproxy15:13
mnasiadkaMaybe this unix socket is the problem?15:13
yoctozeptoyeah, could be haproxy lame with handling it15:13
mnasiadkaNow it’s moving the vip without haproxy being dead15:14
yoctozeptoyeah, indeed15:14
mnasiadkaJust unresponsive15:14
mnasiadkaIn a way15:14
yoctozeptoyeah, possibly15:14
yoctozeptoat least via socket15:14
yoctozeptowonder why sometimes signal 15 and sometimes return 115:14
yoctozeptosignal 15 is for timeout15:15
yoctozeptobut you should be getting 1 for timeout anyways15:15
yoctozeptoeh15:15
mnasiadkaWeird15:15
*** ivve has quit IRC15:15
mnasiadkaSecond thing how can we improve oslo.db connection handling? Are there any tunables?15:15
yoctozeptoyeah, it looks like that socket is not functioning the way one imagined15:16
yoctozeptono idea, I guess normally you would just repeat the command15:16
yoctozeptothough in the case of neutron discussed just today15:16
yoctozeptoit created the subnet15:16
yoctozeptofailed to notify client15:17
yoctozeptoclient retried15:17
yoctozeptoand faild due to existing already15:17
yoctozeptoreal drama15:17
mnasiadkaRaise a bug in neutron :)15:19
*** rpittau|brb is now known as rpittau15:20
yoctozeptomnasiadka: or oslo or openstack client or all at once15:23
yoctozeptomnasiadka: the thing is I don't have logs of client15:24
yoctozeptoonly server's15:24
yoctozeptomgoddard, mnasiadka: https://review.opendev.org/706055 w+1 ?15:27
patchbotpatch 706055 - kolla-ansible - CI: Replace cinder-lvm scenario with zun scenario - 1 patch set15:27
mgoddarddone15:29
yoctozeptomgoddard: danke schön!15:31
*** vmixor has joined #openstack-kolla15:32
mgoddardyoctozepto: can we discuss https://review.opendev.org/#/c/699888/ ?15:33
patchbotpatch 699888 - kolla-ansible - Copy CA into containers. - 23 patch sets15:33
mgoddard"need to ping mgoddard on that"15:33
yoctozeptomgoddard: yeah, go ahead :-)15:33
mgoddardyou pinged?15:33
yoctozeptooh, sorry, I was writing a reply but I can't see even a draft15:33
yoctozeptomgoddard: let me write here then eh15:34
mgoddardgiven we have a few patches chained with +2a, wdyt about merge and refactor?15:34
yoctozeptothe code in patch is generally ok but a bit incostinent15:34
yoctozeptoinconsistent*15:34
mgoddardhow so?15:35
mgoddardordering of conditions?15:35
mgoddardhmm, also some missing group check15:36
yoctozeptoyeah15:36
yoctozeptoand comments for docs15:36
yoctozeptoso it needs a followup15:36
yoctozeptoi'm ok with +2 as it's no backport material15:36
yoctozeptobut we need a followup to keep quality15:36
mgoddardI'm sure James will follow up if asked15:37
yoctozeptomgoddard: ok then, this applies also to releasenote :D15:37
yoctozeptono need to write about issue with py2.7.9 in ussuri...15:37
*** igordc has joined #openstack-kolla15:37
mgoddardtrue15:40
yoctozeptomgoddard: so, what do we do?15:40
mgoddardI don't mind. We can modify the note in a follow up15:41
openstackgerritPierre Riteau proposed openstack/kolla-ansible master: Use InfluxDB default [http]/max-row-limit setting  https://review.opendev.org/70654715:42
yoctozeptomgoddard: done then15:43
yoctozeptomgoddard: your turn15:43
mgoddardyoctozepto: oh yeah, there was this one: https://review.opendev.org/#/c/700788/15:48
patchbotpatch 700788 - kolla-ansible - Use kolla_toolbox to execute REST methods - 18 patch sets15:48
mgoddarddo we really want to make it a fix?15:48
yoctozeptomgoddard: you might be right it's really not worth the effort15:49
generalfuzzhi - I will update note and refactor copying certs into role in https://review.opendev.org/#/c/664516/15:52
patchbotpatch 664516 - kolla-ansible - Add support for encrypting backend HAProxy traffic - 9 patch sets15:52
*** kplant has quit IRC15:56
*** kplant has joined #openstack-kolla16:01
yoctozeptogeneralfuzz: thanks :-)16:06
openstackgerritViktor Michalek proposed openstack/kolla-ansible master: Support for custom API paste files  https://review.opendev.org/69505416:08
*** ktibi has joined #openstack-kolla16:26
*** pbing19 has joined #openstack-kolla16:28
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: CI: Debug init-runonce  https://review.opendev.org/70648216:34
*** bengates has quit IRC16:39
*** yan0s has quit IRC16:43
*** carli has quit IRC16:43
openstackgerritRadosław Piliszek proposed openstack/kolla stable/train: CI: Replace cinder-lvm scenario with zun scenario  https://review.opendev.org/70656316:45
*** ktibi has quit IRC16:46
*** ktibi has joined #openstack-kolla16:53
openstackgerritMerged openstack/kolla-ansible master: CI: Replace cinder-lvm scenario with zun scenario  https://review.opendev.org/70605516:54
*** hongbin has joined #openstack-kolla16:55
*** hongbin has quit IRC16:59
*** rpittau is now known as rpittau|afk17:02
*** noxoid_ is now known as noxoid17:06
mgoddardmnasiadka: I thought of another improvement to ext ceph user experience. Prechecks for the expected configs (conf.conf, keyrings)17:13
mgoddardmnasiadka: will be especially useful following change of defaults for keys17:13
mnasiadkamgoddard: good, that’s on my list :)17:13
mgoddardmnasiadka: also just spotted a few hard coded ceph.client.gnocchi.keyring17:15
mnasiadkamgoddard: ouch, will look for them on Monday - if you can raise a bug I would be grateful :)17:16
*** ktibi has quit IRC17:17
yoctozeptomgoddard, mnasiadka: are you aware whether sahara works in kolla?17:17
yoctozepto(or even in general?)17:17
mgoddardmnasiadka: https://bugs.launchpad.net/kolla-ansible/+bug/186239317:18
openstackLaunchpad bug 1862393 in kolla-ansible ussuri "Some hard coded ceph.client.gnocchi.keyring left" [Undecided,New]17:18
mgoddardyoctozepto: I spent some time with it a few years ago, but not recently17:18
yoctozeptomgoddard: w/ kolla?17:18
mgoddardyes17:18
yoctozeptomgoddard: thanks, raises spirits :-)17:19
mgoddardhaven't seen many patches since then though17:19
mgoddardmaybe it still works17:19
yoctozeptowill see17:19
yoctozeptomnasiadka, mgoddard: are we backport ceph-ansible to train?17:20
mgoddardyou mean the CI?17:20
yoctozeptoasking because backporting in kolla would be quite easy before I do mine with zun17:20
yoctozeptobut k-a is a bit of pain17:20
yoctozeptomgoddard: CI17:20
mgoddardprobably a bit overkill to do two four ceph jobs17:20
yoctozeptowe ain't doing ceph-ansible port for train :-)17:21
yoctozeptomnasiadka: you agree? I do17:21
yoctozeptothen I backport zun and make backporting c-a painful17:21
yoctozeptolooks like mnasiadka is on the move already17:21
yoctozeptoI see nobody promoted sahara away from 3rd tier17:22
mnasiadkaLess work for me - I’m happier :)17:22
yoctozeptomnasiadka: all right, then agreed17:22
openstackgerritMerged openstack/kolla-ansible master: CentOS 8: Add deploy jobs in CI  https://review.opendev.org/69354417:24
openstackgerritMerged openstack/kolla-ansible master: Copy CA into containers.  https://review.opendev.org/69988817:25
*** oncall-pokemon has joined #openstack-kolla17:25
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: CI: Enable etcd in ceph and cinder-lvm scenarios  https://review.opendev.org/69748517:25
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible stable/train: CI: Test Zun with Cinder LVM backend (iSCSI)  https://review.opendev.org/70658717:29
yoctozeptohad to backport this first ^17:29
yoctozeptotrain CI did not include cinder lvm for zun17:29
yoctozeptowell, let's make it consistent17:29
oncall-pokemonwith the kolla prometheus. what would cause the keys `openstack_nova_total_vms` to not be populated?17:33
openstackgerritMerged openstack/kolla-ansible master: Use kolla_toolbox to execute REST methods  https://review.opendev.org/70078817:33
openstackgerritMerged openstack/kolla-ansible master: Generate self signed TLS certificates  https://review.opendev.org/70132317:33
*** evrardjp has quit IRC17:34
*** evrardjp has joined #openstack-kolla17:34
*** mleone87 has quit IRC17:35
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible stable/train: CI: Test Zun with Cinder LVM backend (iSCSI)  https://review.opendev.org/70658717:35
*** ivve has joined #openstack-kolla17:38
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible stable/train: CI: Replace cinder-lvm scenario with zun scenario  https://review.opendev.org/70659117:39
*** k_mouza_ has quit IRC17:44
mgoddardyoctozepto: did you see this thread: http://lists.openstack.org/pipermail/openstack-discuss/2020-January/012071.html17:44
mgoddardreferences OSA galera config: https://opendev.org/openstack/openstack-ansible/src/branch/master/inventory/group_vars/haproxy/haproxy.yml#L48-L4917:44
mgoddardthought you might be interested as current galera tormentor17:45
yoctozeptomgoddard: thanks, I think I saw that but then forgot, might come in handy indeed17:46
yoctozeptomgoddard: osa also uses other means of liveness discovery - our script is limited and might allow a catching up instance to serve requests (or rather rejections :-) )17:50
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: Docs: show other versions  https://review.opendev.org/70659517:55
*** priteau has quit IRC17:56
*** dougsz has quit IRC18:02
*** rlljorge has quit IRC18:09
*** k_mouza has joined #openstack-kolla18:14
*** igordc has quit IRC18:18
*** k_mouza has quit IRC18:19
yoctozeptomgoddard, mnasiadka: for lolz: k-a master ci b0rken with c8 failing on bootstrap not able to find python2... TASK [Gather facts] ************************************************************18:32
yoctozeptomgoddard, mnasiadka: oddly zuul ran jobs from definitions on master but used old files...18:35
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: CI: Debug init-runonce  https://review.opendev.org/70648218:35
openstackgerritMerged openstack/kolla master: CentOS 8: Add deploy jobs in CI  https://review.opendev.org/70595218:36
*** igordc has joined #openstack-kolla18:49
*** k_mouza has joined #openstack-kolla18:50
*** k_mouza has quit IRC18:51
openstackgerritViktor Michalek proposed openstack/kolla-ansible master: Support for custom API paste files  https://review.opendev.org/69505418:58
openstackgerritViktor Michalek proposed openstack/kolla-ansible master: Support for custom API paste files  https://review.opendev.org/69505419:02
*** gfidente is now known as gfidente|afk19:11
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: Tidy up tools/ subdirectory  https://review.opendev.org/70649619:16
dking_desktopThis is probably better suited for the neutron channel, but would anybody know how I should configure neutron_external_interface if I'm using my bond (bond0) for all my VLANs, but I also need to be able to connect to the controller to receive untagged requests?19:19
*** igordc has quit IRC19:19
openstackgerritViktor Michalek proposed openstack/kolla-ansible master: Adds support of CADF notifying for other APIs next to keystone.  https://review.opendev.org/67457919:20
openstackgerritMerged openstack/kayobe stable/rocky: Fix multiple CI failures  https://review.opendev.org/70477419:26
openstackgerritMerged openstack/kayobe stable/stein: Use internal API endpoints in overcloud API interaction  https://review.opendev.org/70462019:26
openstackgerritMerged openstack/kayobe stable/stein: Stop running kolla-target-venv in seed hypervisor host upgrade  https://review.opendev.org/70461419:26
noxoidcurious, has anyone deployed standard ironic (not kayobe) in a multi-region environment? it looks like the k-a playbooks for ironic are heavily dependent on enable_keystone being set to yes which is not the case in multi-region environments19:35
openstackgerritMerged openstack/kayobe stable/stein: Use mariabackup for database backups  https://review.opendev.org/70462419:37
openstackgerritMerged openstack/kayobe stable/stein: Don't list tasks for config dump  https://review.opendev.org/70460919:37
openstackgerritMerged openstack/kayobe stable/stein: Allow Kayobe to wipe virtualized disks  https://review.opendev.org/70461219:37
noxoidah yea its been reported https://bugs.launchpad.net/kolla-ansible/+bug/180921619:37
openstackLaunchpad bug 1809216 in kolla-ansible "Ironic in other regions do not register endpoints" [Medium,Triaged]19:37
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: Remove kolla-ceph  https://review.opendev.org/70430919:39
openstackgerritViktor Michalek proposed openstack/kolla-ansible master: Adds support of CADF notifying for other APIs next to keystone.  https://review.opendev.org/67457919:41
*** pbing19 has quit IRC19:57
*** cah_link has quit IRC20:08
openstackgerritViktor Michalek proposed openstack/kolla-ansible master: Adds support of CADF notifying for other APIs next to keystone.  https://review.opendev.org/67457920:09
openstackgerritMerged openstack/kayobe stable/stein: Fix kayobe overcloud provision without seed in limit  https://review.opendev.org/70461620:10
openstackgerritMerged openstack/kayobe stable/stein: Fix use of IPA images in Bifrost without a checksum  https://review.opendev.org/70461820:10
openstackgerritZhuo Zhen proposed openstack/kolla-ansible master: Configurable logrotate rotation interval and count  https://review.opendev.org/70661620:12
openstackgerritViktor Michalek proposed openstack/kolla-ansible master: Added support for custom logging.conf  https://review.opendev.org/67981220:17
*** kplant has quit IRC20:42
*** tonythomas has quit IRC20:44
*** primeministerp has joined #openstack-kolla20:46
*** primeministerp has joined #openstack-kolla20:47
*** primeministerp has joined #openstack-kolla20:48
*** primeministerp has quit IRC20:48
*** primeministerp has joined #openstack-kolla20:48
*** primeministerp has quit IRC20:49
*** primeministerp has joined #openstack-kolla20:50
*** k_mouza has joined #openstack-kolla20:52
*** k_mouza has quit IRC20:56
openstackgerritJames Kirsch proposed openstack/kolla-ansible master: Service configuration urls should be constructed using kolla_internal_fqdn  https://review.opendev.org/70662721:43
openstackgerritPierre Riteau proposed openstack/kayobe stable/rocky: Stop running kolla-target-venv in seed hypervisor host upgrade  https://review.opendev.org/70461521:51
openstackgerritPierre Riteau proposed openstack/kayobe stable/rocky: Allow Kayobe to wipe virtualized disks  https://review.opendev.org/70461321:51
openstackgerritPierre Riteau proposed openstack/kayobe stable/rocky: Fix kayobe overcloud provision without seed in limit  https://review.opendev.org/70461721:52
openstackgerritPierre Riteau proposed openstack/kayobe stable/rocky: Fix use of IPA images in Bifrost without a checksum  https://review.opendev.org/70461921:52
openstackgerritPierre Riteau proposed openstack/kayobe stable/rocky: Use internal API endpoints in overcloud API interaction  https://review.opendev.org/70462121:52
openstackgerritPierre Riteau proposed openstack/kayobe stable/rocky: Don't list tasks for config dump  https://review.opendev.org/70461121:52
*** jbadiapa has quit IRC21:55
*** primeministerp has quit IRC22:06
*** vmixor has quit IRC22:12
*** TrevorV has quit IRC22:57
*** oncall-pokemon has quit IRC23:15

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!