Thursday, 2020-02-06

*** k_mouza has joined #openstack-kolla00:01
*** k_mouza has quit IRC00:05
*** igordc has quit IRC00:08
*** noxoid_ has joined #openstack-kolla00:14
*** dmellado_ has joined #openstack-kolla00:16
*** dcapone2004 has quit IRC00:17
*** noxoid has quit IRC00:21
*** sm806 has quit IRC00:21
*** mgkwill has quit IRC00:21
*** dmellado has quit IRC00:21
*** dmellado_ is now known as dmellado00:22
*** sm806 has joined #openstack-kolla00:22
*** chrizl has joined #openstack-kolla00:23
*** egonzalez has quit IRC00:25
*** cz3 has quit IRC00:25
*** mnasiadka has quit IRC00:25
*** mwhahaha has quit IRC00:25
*** r3ap3r has quit IRC00:25
*** ab-a has quit IRC00:25
*** logan- has quit IRC00:25
*** mnasiadka has joined #openstack-kolla00:25
*** cz3 has joined #openstack-kolla00:25
*** r3ap3r has joined #openstack-kolla00:25
*** mwhahaha has joined #openstack-kolla00:25
*** egonzalez has joined #openstack-kolla00:25
*** rm_work has quit IRC00:27
*** logan- has joined #openstack-kolla00:27
*** openstackstatus has quit IRC00:28
*** Jeffrey4l has quit IRC00:29
*** rm_work has joined #openstack-kolla00:30
*** ab-a has joined #openstack-kolla00:30
*** Jeffrey4l has joined #openstack-kolla00:31
*** skramaja has quit IRC01:11
*** dking_desktop has left #openstack-kolla01:34
*** zhurong has joined #openstack-kolla03:03
openstackgerritXinliang Liu proposed openstack/kolla-ansible stable/rocky: Use become for Cinder LVM precheck  https://review.opendev.org/70617203:48
*** k_mouza has joined #openstack-kolla04:00
*** k_mouza has quit IRC04:05
*** skramaja has joined #openstack-kolla04:20
*** zhurong has quit IRC04:41
*** zhurong has joined #openstack-kolla05:07
*** Tony31 has joined #openstack-kolla05:18
Tony31good morning05:18
*** shyamb has joined #openstack-kolla05:31
*** evrardjp has quit IRC05:33
*** evrardjp has joined #openstack-kolla05:34
*** sorin-mihai has joined #openstack-kolla05:48
*** sean-k-mooney has quit IRC05:53
*** sean-k-mooney has joined #openstack-kolla05:55
*** dking_desktop has joined #openstack-kolla06:08
*** Tony3136 has joined #openstack-kolla06:15
Tony3136.06:16
*** shyamb has quit IRC06:17
*** shyamb has joined #openstack-kolla06:17
*** Tony31 has quit IRC06:18
*** Tony3136 is now known as Tony3106:18
*** shyamb has quit IRC06:25
*** shyamb has joined #openstack-kolla06:27
Tony31hi guys06:38
Tony31having some trouble running `kayobe overcloud provision` . Seems to be getting stuck trying to connect to the baremetal VMs.06:41
*** zhurong has quit IRC06:42
Tony31I was expecting the VMs to pxeboot and come up with the IPs listed in network-allocation.yml but instead it comes up with with an IP in the inspection range06:43
Tony31any idea what I could be doing wrong here?06:43
*** shyamb has quit IRC07:00
*** sean-k-mooney has quit IRC07:01
*** sean-k-mooney has joined #openstack-kolla07:03
osmanlicilegimorning07:04
Tony31hi osman07:05
*** shyamb has joined #openstack-kolla07:18
*** hjensas has joined #openstack-kolla07:24
Tony31could anyone help with an error that I get when running `kayobe overcloud provision` ?07:39
Tony31pretty please07:39
*** shyamb has quit IRC07:40
yoctozeptomgoddard, mnasiadka: kayobe user-to-be asking for your assistance07:42
yoctozeptoTony31: sorry, neither me nor osmanlicilegi are kayobe folks :<07:42
Tony31Thanks for the response anyway :) :)07:42
Tony31I must leave the office today before 5pm to fix my gf's internet but i am planning to come back online today from home so I can try and catch up with the `Kayobe` guys. I think I am really close now :)07:43
yoctozeptoTony31: those gfs always break their internets eh :D07:46
Tony31:)  actually, both hers and mine went down last night but hers never came back. We both use the Australia "NBN" with different ISPs. the NBN is a government network initiative to provide high-speed internet (because previously it was rubbish ADSL >6mb/s everywhere). Mine use fibre to my place but hers uses VDSL. The xDSL all use cheap equipment and07:49
Tony31it's really bad. OFten these problems happen and they have to "rebuild the port" which I think means delete the config and put it back in again 🙈07:49
Tony31sorry I mean <6mb/s07:49
Tony31her isp told her to configure the modem... it's been working for a month or two and I definitely configured it 🙈07:50
Tony31yoctozepto is there a `kolla` equivalent of `kayobe overcloud provision` ?07:51
Tony31I was looking at this page to find if there was https://docs.openstack.org/kolla-ansible/latest/reference/bare-metal/ironic-guide.html but I didnt piece together anything there, but could be my lack of knowledge of it07:52
yoctozeptoTony31: hah, bad isps in there07:52
Tony31I mean, do you know if there is a `kolla` equivalent of `kayobe overcloud provision` ?07:53
yoctozeptoTony31: kolla-ansible works on pre-provisioned servers07:53
mnasiadkamorning07:53
Tony31ah07:53
yoctozeptoso you have to install the OS yourself07:53
Tony31good morning07:53
yoctozeptosetup ssh access07:53
Tony31that could be an alternative for me07:53
yoctozeptoconfigure networking per docs07:53
yoctozeptoand then can deploy07:53
Tony31because at the moment I am using VMs to try this out and I am pxebooting them but there is no IPMI or BMC. So I'm running the command and then powering up the VMs manually and crossing fingers. It did work ok for tripleO deployment a few years ago07:54
yoctozeptowell, personally I had to abandon kayobe for the time being due to my fujitsu servers not liking ironic access; I still have ironic in plans but well, time is time :-)07:54
yoctozeptooh man, what an overkill07:55
Tony31i had so many issues with the baremetal part of tripleO, the manual power up was the easy bit :)  Then they updated the RPM between the working week and the weekend and so I hit a few bugs which almost caused an abort of the deployment07:55
Tony31mnasiadka hi - do you have any capacity to try and help me with `kayobe overcloud provision ` please?07:56
mnasiadkaTony31: first tell me what is the problem, then I can tell you if I can help :)07:57
openstackgerritPiotr Rabiega proposed openstack/kolla master: Add collectd-intel_pmu plugin  https://review.opendev.org/70456007:57
Tony31OK :)07:57
Tony31when I run that command, it fails at the `TASK [Ensure the ironic node is available]`07:58
Tony31I have 2 VMs added as baremetal07:58
Tony31introspection was performed and I named the nodes as controller0 and compute007:58
Tony31when I power up the VMs they PXE boot and get an IP in the introspection range. I was expecting them to get an ip within the allocation pool07:59
Tony31this could be pointing to the problem but I havent been able to figure it out yet07:59
yoctozeptoand to answer "08:51:51 <Tony31> yoctozepto is there a `kolla` equivalent of `kayobe overcloud provision` ?" -> in kolla-ansible it's just deploy :-)08:00
*** dciabrin has joined #openstack-kolla08:00
yoctozeptoand now I go silent and you try to debug kayobe, good luck08:00
Tony31`# openstack baremetal node list08:00
Tony31+--------------------------------------+-------------+---------------+-------------+--------------------+-------------+08:00
Tony31False       |08:00
mnasiadkaTony31: in the sense of having 2 VMs added as baremetal - do you use universe from nothing approach?08:03
Tony31yes I did do that08:03
Tony31but this time I am designing a production deployment08:03
mnasiadkaand Ironic first needs to introspect a node, before it can be set to available state and provisioned08:03
Tony31on control host VM and seed VM08:03
Tony31I can try the introspection again08:04
mnasiadkathe question is if the introspection fails (and why)?08:04
Tony31just starting it now :)  thanks for your help08:05
*** dciabrin has quit IRC08:09
*** cah_link has joined #openstack-kolla08:10
Tony31I get this error `dev/null 2>&1 && sleep 0'", "controller0 | FAILED | rc=1 >>", "The requested action \"inspect\" can not be performed on node \"5fb8a19d-60a9-4359-8d8b-a9ecc5cb22bc\" while it is in state \"enroll\". (HTTP 400)non-zero return code"]}`08:11
Tony31the provisioning state is currently `enroll`08:11
Tony31do you know what state it needs to be in for `introspection`?08:12
openstackgerritEddie Yen proposed openstack/kolla-ansible stable/rocky: Remove /etc/hosts entries pointing hostname to localhost and prevent cloud-init to manage /etc/hosts  https://review.opendev.org/70619208:13
*** dciabrin has joined #openstack-kolla08:13
*** k_mouza has joined #openstack-kolla08:15
Tony31should the provisioning state be `manageable` ?08:16
Tony31I could try manually resetting and re-trying08:16
*** k_mouza has quit IRC08:19
Tony31do you know which process should be making the nodes moved from `enroll` to `manage` ? because I could try running that again08:21
*** tonythomas has joined #openstack-kolla08:24
Tony31from `openstack baremetal node show` I see `last_error             | Failed to validate power driver interface for node 6d45927a-a8e8-46c7-8f31-b0ccb1992400. Error: Missing the following IPMI credentials in node's driver_info: ['ipmi_address'].               |08:28
Tony31could this be a reason? Can I use a fakePXE interface for the VMs ?08:29
yoctozeptoTony31: well, you most likely *have to* :-)08:36
yoctozeptobut not sure about fakepxe08:36
Tony31I'm trying :)08:36
yoctozeptoI think it's called differently for power08:36
Tony31i set a dummy pxe username/password for ipmi and now the error is that there is no ip address08:39
Tony31`last_error             | Failed to validate power driver interface for node 6d45927a-a8e8-46c7-8f31-b0ccb1992400. Error: Missing the following IPMI credentials in node's driver_info: ['ipmi_address'].               |08:39
yoctozeptoTony31: yea, cause vms have no bmc08:39
yoctozeptoand you did not configure it08:39
Tony31cant find this variable at all08:40
yoctozeptoit won't help to fake it08:40
yoctozeptoneed to change the driver for power08:40
Tony31cant find that either, or a list of drivers08:40
yoctozeptoah, ok08:40
Tony31going to have to go in a sec but i'll be back on later08:40
yoctozeptomgoddard, mnasiadka: so we pinpointed that to Tony31 using vms08:41
yoctozeptonow the question is how to handle this in kayobe08:41
Tony31I'd like to use VMs for controller and at least one compute... but then later on, bring in my 2 x Cisco servers in a chassis which can use ipmi no problem08:41
Tony31catch you later thanks for your help :)08:41
mnasiadkaTony31: just thinking - do you really follow the guide on universe-from-nothing and use tenks?08:45
*** Tony31 has quit IRC08:46
*** carli has joined #openstack-kolla08:49
mnasiadkayoctozepto: do you have any idea why the upgrade jobs fail here? https://review.opendev.org/#/c/704309/08:54
patchbotpatch 704309 - kolla-ansible - WIP: Remove kolla-ceph - 6 patch sets08:54
mnasiadkaseems like it uses inventory from later version, but code from earlier :)08:54
*** jaicaa has quit IRC08:55
mnasiadkaand probably it does, since we template out the inventory from tests/templates08:55
yoctozeptomnasiadka: yeah, exactly08:56
yoctozeptoinventory has to be compatible with both versions08:56
mnasiadkaoh boy, I'll need to add another if in the inventory template08:56
yoctozeptomnasiadka: well, it would be easier if ansible assumed that to be empty and only generated warning08:57
*** jaicaa has joined #openstack-kolla08:58
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Remove kolla-ceph  https://review.opendev.org/70430909:01
*** k_mouza has joined #openstack-kolla09:04
mnasiadkayoctozepto: life can't be easy :)09:07
mgoddardmorning09:10
*** dougsz has joined #openstack-kolla09:11
*** dciabrin has quit IRC09:18
*** dciabrin has joined #openstack-kolla09:18
*** shyamb has joined #openstack-kolla09:22
*** brtknr has quit IRC09:40
openstackgerritMark Goddard proposed openstack/kayobe master: WIP: Use python3 for local kolla-ansible execution  https://review.opendev.org/70500009:45
openstackgerritNoboru Iwamatsu proposed openstack/kolla-ansible master: Adapt to Octavia Certificate Configuration Guide.  https://review.opendev.org/70620309:51
*** ivve has joined #openstack-kolla09:58
*** ktibi has joined #openstack-kolla09:59
*** dciabrin has quit IRC10:02
ktibiHi guys, I have a strange behavior on nova. I don't know why, but nova create vms on the same host. When the host is full, nova change. By example, I created 20vms and all VM are on the same host. but I have 4 other hosts with 2 instances.10:02
ktibiI have this behavior only on plateform deploy by kolla :/10:03
ktibiany suggestions ?10:03
*** bjolo has joined #openstack-kolla10:05
*** gfidente|afk is now known as gfidente10:06
openstackgerritMerged openstack/kolla-ansible stable/rocky: Use become for Cinder LVM precheck  https://review.opendev.org/70617210:08
*** dciabrin has joined #openstack-kolla10:11
yoctozeptoktibi: odd, which release?10:22
yoctozeptomaybe it had some bug in placement/nova which causes this10:23
ktibiqueens10:23
ktibirocky*10:23
*** shyamb has quit IRC10:32
*** rpittau|afk is now known as rpittau10:45
*** ricolin_ has joined #openstack-kolla10:47
openstackgerritRadosÅ‚aw Piliszek proposed openstack/kolla master: CI: Replace cinder-lvm scenario with zun scenario  https://review.opendev.org/70605910:49
yoctozeptomgoddard, mnasiadka: https://review.opendev.org/706059 let's merge before it gets another conflict...10:50
patchbotpatch 706059 - kolla - CI: Replace cinder-lvm scenario with zun scenario - 2 patch sets10:50
openstackgerritMark Goddard proposed openstack/kolla master: CentOS 8: Enable hacluster images  https://review.opendev.org/70270610:51
yoctozeptoktibi: hmm, /me running stein; would you mind posting about this to ml? might be it was a known problem10:51
*** dciabrin has quit IRC10:53
*** dciabrin has joined #openstack-kolla10:54
yoctozeptomgoddard: sorry for bother but it would be best if you rebased c8 jobs on top of respective zun's lvm patch10:59
*** shyamb has joined #openstack-kolla10:59
mgoddardyoctozepto: depends if we are backporting zun/lvm to train11:00
mgoddardif not, will be cleaner for me to merge first11:00
yoctozeptomgoddard: I planned to since it's the very same story11:00
mgoddardyoctozepto: k11:00
yoctozeptomgoddard: needless cycles better spared11:00
mgoddardyoctozepto: yours or mine? :p11:01
yoctozeptomgoddard: ci's11:01
yoctozeptomgoddard: I believe ours are very necessary :D11:01
mgoddardah that's annoying11:02
mgoddardIf I build bifrost but it is marked unbuildable, it will build base and openstack-base before telling me it couldn't build bifrost-base11:03
yoctozeptololz11:03
yoctozeptoyeah, kolla awaiting rewrite11:03
mnasiadkait should be pretty easy to just throw an error when the supplied images are unbuildable11:04
mgoddardnaturally, it's fiddly11:04
mnasiadkaok, after forging the inventory template - the upgrade jobs work11:05
mgoddardwe want to apply filters to parents11:05
openstackgerritMerged openstack/kolla-ansible stable/rocky: Remove /etc/hosts entries pointing hostname to localhost and prevent cloud-init to manage /etc/hosts  https://review.opendev.org/70619211:05
*** Tony31 has joined #openstack-kolla11:13
Tony31hi again11:13
Tony31mnasiadka I did follow the guide but in the end, I didnt need to make any variable changes. Initially I was trying to configure the networking because I understood incorrectly that it would be bridged to the LAN. But instead, if I didnt make any changes everything just worked except for some problems with an ansible module in the venv and missing11:15
Tony31packages11:15
Tony31but as I wont use tenks in prod, is it relevant?11:15
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: Remove kolla-ceph  https://review.opendev.org/70430911:15
Tony31since you mentioned tenks - i'll go and take another look at it11:15
yoctozeptomgoddard: kayobe user-to-be ^11:16
Tony31I still have the vm that has the all in one. Openstack runs pretty quick on it tbh11:16
mgoddardhi Tony3111:16
mgoddardtenks is just used for testing11:17
mgoddardwe use it in CI to test ironic11:17
mgoddardand also in a universe from nothing to create baremetal VMs for controllers and computes11:17
Tony31Hi Mark11:18
mgoddardis there something we can put in the README to make it clearer that there's no bridge required?11:18
Tony31maybe I can use it as a virtual BMC but I dont want to run VMs inside VMs for the controller / compute11:18
yoctozeptomgoddard: Tony31 has a problem with kayobe bifrosty part11:18
mgoddard+1, it's good for testing but not production11:18
yoctozeptowhere it requires some power interface to vms11:18
Tony31yes - I was going to ask if I could make a pull request and suggest some udpates to it?11:18
mgoddardof course11:18
yoctozeptoTony31: absolutely11:18
yoctozeptowe love good doc updates :-)11:19
Tony31my background is networking so I'm probably looking at it from a bad angle11:19
mgoddardthat would be much appreciated11:19
mgoddardwhere were the missing packages btw?11:19
Tony31ok - will do it after getting this up and running I still want to find out how to do a bunch of things like add a 2nd storage network to the compute/controller, add cinder driver, domain auth., horizon updates etc11:20
Tony31missing packages I got by installing `yum install qemu-kvm libvirt libvirt-python libguestfs-tools virt-install qemu-img virt-manager libvirt-client virt-viewer bridge-utils` so I am not sure which package(s) fixed the issue. I also needed to reboot afterwards11:22
Tony31then I found that there's an openstack module that is getting deprecated and I needed to source the kolla venv and run `pip install os-client-config`11:23
Tony31sorry11:23
Tony31not kolla11:23
Tony31tenks venv11:23
Tony31is there a way I can tell kayobe to ignore the ipmi stuff? I'll power on the VM myself... I know it goes against automation but I'm not sure how else it would work, with control and compute in a VM on oVirt11:25
mgoddardstrange - we install os-client-config in the tenks venv11:25
Tony31cue twilight zone music :)11:26
mgoddardTony31: so let me see if I understand. You are trying to set up a production environment, and want to create some VMs for your controllers and use those11:26
Tony31well this is how I have it running with tripleo pike at the moment: single controller/ network VM. 2 x physical compute nodes which are Cisco hardware. Plus an undercloud director thats shut down now.11:27
mgoddardok11:27
mgoddarddo you have enough VM automation to create VMs with a CentOS image and an IP?11:27
yoctozepto(not that one needs too much automation for a single controller)11:28
Tony31I'm thinking to deploy another controller VM but also a compute VM so that i can manage a DR failover (I'll failover the hypervisor and VMs) because of the absence of knowledge on how to achieve the same with openstack. I have 2 x physical compute nodes in the DR and I could use one for oVirt and the other for openstack. But at the moment both are11:28
Tony31in ovirt11:28
mgoddardif so I would suggest skipping the bifrost part for the controller VM - no need to pretend it's baremetal11:28
yoctozepto^ +111:29
Tony31... I'd like to make the controller HA but I havent gone down that road yet - I Wanted to ask you guys could I achieve it later on? I think I can, just have not searched for the docs11:29
mgoddardjust create a CentOS VM and add it to your inventory11:29
mgoddardyes, kolla supports HA11:29
Tony31ok thanks Mark11:29
yoctozeptoTony31: it's easy then, kolla-ansible does HA for you11:29
Tony31That was on my mind today11:29
yoctozeptojust slap more controllers :D11:29
mgoddardetc/kayobe/inventory/hosts:11:29
Tony31:)11:29
yoctozeptointo the inventory11:29
yoctozeptomgoddard: you have commands in kayobe that proxy to k-a deploy, no?11:30
Tony31How do I get rid of the 2 fake baremetal things in the seed / bifrost container?11:30
mnasiadkayoctozepto: your zun change just failed on ubuntu/debian :D11:30
mgoddardsimilar to this: https://github.com/openstack/kayobe-config-dev/blob/master/etc/kayobe/inventory/hosts11:30
mgoddardadd a host to the controllers group11:30
mgoddarddefine it's IP in etc/kayobe/network-allocation.yml11:30
yoctozeptomnasiadka: on debian, on grafana11:30
Tony31yes got it :)  thanks - will treat it like the seed11:30
yoctozeptomnasiadka: kolla ci goes red11:30
yoctozeptoagain11:31
mnasiadkauhh11:31
Tony31my seed is a VM I installed myself on the ovirt host using cloud-init11:31
mgoddardopenstack baremetal node delete <node>11:31
mnasiadkayoctozepto: what's wrong with grafana on debian? :)11:31
mgoddardto remove from bifrost11:31
Tony31ah great - so just typical openstack commands then :)11:31
yoctozeptomnasiadka: everything11:31
Tony31thanks a lot :)11:31
mgoddardyep11:31
mgoddardyou need to run them from inside the bifrost container though11:31
yoctozeptomnasiadka: INFO:kolla.common.utils.grafana:E: Failed to fetch https://packages.grafana.com/oss/deb/dists/stable/main/binary-amd64/Packages.bz2  File has unexpected size (14821 != 14971). Mirror sync in progress? [IP: 151.101.198.217 443]11:31
Tony31I guess I Went down the wrong path because I was trying to apply lessons learned from tripleo11:31
mgoddardbut it sounds like you've done that already11:31
Tony31yep :)  was looking around there today11:32
mgoddardwe're a lot more flexible than tripleo :)11:32
mgoddardbring your own VMs - not a problem11:32
Tony31when I deployed tripleo it kept placing the controller on the compute node. it was sheer luck I got it deployed how I Wanted it.11:32
mgoddardlol11:32
yoctozeptomgoddard: and we have much less code to break :-)11:32
mgoddardit's a bit of a jelly11:32
Tony31i tagged and named and everything, but it just picked a random host for the placement11:32
Tony31:)  kayobe is much better11:33
yoctozepto:D11:33
mgoddardwoop11:33
yoctozeptowait till you see bikolla11:33
mgoddardI would suggest going bare metal for computes in production though11:33
yoctozepto^ +111:33
mgoddarddepending on what you need for DR11:33
Tony31another question i had - but it's probably in the docs so i can find it another time. if i make changes to the deployment and want to execute those, is there a process guide to follow?11:34
Tony31i have 2 x bare metal at the moment.. the issue I am trying to avoid is the different networks in DR compared to prod11:34
mgoddardkayobe overcloud service reconfigure11:34
mgoddardthat will regenerate config and containers if necessary11:34
yoctozeptomnasiadka: thankfully, debian is non-voting11:34
Tony31the physical machine in DR will have VLANs in different subnets. Additionally the storage is replicated there but the IP's and IQNs are different11:34
mgoddardhttps://docs.openstack.org/kayobe/latest/administration/overcloud.html#reconfiguring-containerised-services11:35
Tony31thanks mark - saving :)11:35
Tony31also when i was configuring the networks, it asks for gateways and vlans - well in most cases I dont need to tag vlans because the baremetal chassis takes care of that, and so does the hypervisor (ovirt). I did have an issue with the seed because it had 2 x default gateways and initially it broke networking11:36
Tony31how do you take care of that?11:36
Tony31I say "asks" what I mean is, in the yml file it has commented out fields... I just filled them all out11:37
Tony31:)11:37
*** rlljorge has joined #openstack-kolla11:38
yoctozeptoTony31: these are to properly configure baremetals11:40
yoctozeptowhen it *would* configure interfaces witg tags11:40
Tony31ah ok :)11:40
Tony31I planned to try deploying and see if it worked and go from there11:40
openstackgerritMark Goddard proposed openstack/kolla master: CentOS 8: Add deploy jobs in CI  https://review.opendev.org/70595211:40
Tony31thanks for the info11:40
yoctozeptoyw11:41
mgoddardTony31: you don't need all the fields for every network. In general there should only be one network with a gateway11:41
Tony31yoctozepto btw I managed to fix that internet in under 60 seconds. no joke. All modem lights were green. So I traced to `1.1.1.1` and it worked. nslookup was going to an `fe80:` ipv6 address.. so i turned off ipv6 and everything worked. . I think the outage last night was them deploying ipv6. I've yet to check though11:42
Tony31Mark, again I think I am using past tripleo experience and causing a problem for myself... I need a default gateway for the nodes.. for all of them (except the controller) it's the management network.11:42
yoctozeptoTony31: oh, you are one of those baddies who disable ipv611:43
Tony31the controller, however, has a default gateway out via the external API network so the horizone dashboard can be loaded. I'll need to check again11:43
Tony31yoctozepto only when it's not set up or not set up properly :)11:43
Tony31im an ipv6 forum gold member11:44
openstackgerritMark Goddard proposed openstack/kolla-ansible master: CentOS 8: Add deploy jobs in CI  https://review.opendev.org/69354411:44
Tony31lol11:44
rlljorgeMorning ! I have a bug with train release and I try deploy stein. Stein is compatible with ansible 2.9.2 and kolla-ansible 9.1.0.dev160 ?11:44
yoctozeptoTony31: nice!11:45
mgoddardTony31: there are two approaches. You could use ansible group_vars to set different gateways for different groups (needs to be removed from networks.yml which has precedence)11:45
yoctozeptorlljorge: that kolla ansible is for master/ussuri11:45
Tony31ok thank you - makes perfect sense.11:45
mgoddardor you can use policy-based routing on the external network to avoid asymmetric routing11:45
yoctozeptorlljorge: stein is 8.*11:46
Tony31i like your first option - i had not thought of it, but now you said it, it does make sense11:46
mgoddardhttps://docs.openstack.org/kayobe/latest/configuration/network.html#advanced-policy-based-routing11:46
rlljorgeyoctozepto thanks11:46
Tony31so just hypothetically thinking... I could deploy openstack using kayobe, then manually create a VM on openstack. Then deploy/reconfigure openstack with kayobe and make that VM a controller node :D11:48
yoctozeptoTony31: yeah, though that would be trying to go worse than tripleo :-)11:49
yoctozepto(in the sense of complicating life)11:49
Tony31:)11:50
*** brtknr has joined #openstack-kolla11:54
Tony31thanks again guys - im off to get some dinner. have a good one :)11:59
*** shyamb has quit IRC12:11
*** pcaruana has quit IRC12:13
*** Tony31 has quit IRC12:15
yoctozeptomgoddard: it really does not seem like I need that dance for mariadb12:19
yoctozeptomgoddard: mind you it still runs recovery when necessary12:20
yoctozepto(xtra->maria)12:20
*** kplant has joined #openstack-kolla12:23
rlljorgefatal: [compute02]: FAILED! => {"changed": false, "msg": "Unable to start service docker: Job for docker.service failed because the control process exited with error code. See \"systemctl status docker.service\" and \"journalctl -xe\" for details.\n"}12:34
hrwhttps://marcin.juszkiewicz.com.pl/fosdem/ - covers all years with videos (2015-2020)12:34
rlljorgewhen I try execute bootstrap-servers12:34
*** shyamb has joined #openstack-kolla12:34
yoctozeptorlljorge: need to see that journalctl12:34
rlljorgeyoctozepto http://paste.openstack.org/show/789220/12:36
yoctozeptorlljorge: would be nice to see above Feb 06 09:35:36 control01.sianet.com.br systemd[1]: docker.service failed.12:37
rlljorgeyoctozepto: http://paste.openstack.org/show/789221/12:37
rlljorgeI Think the problem is here12:37
yoctozeptorlljorge: indeed - did you previously bootstrap it with the latest kolla-ansible?12:40
rlljorgeyoctozepto is12:40
yoctozeptolooks like bootstrapping it again with an older one will kill it (which is not that unexpected)12:40
yoctozeptofor your case just delete the json file12:41
yoctozeptoon all bootstrapped host12:41
yoctozeptothen rerun the playbook12:41
rlljorgeyoctozepto perfect ! Works12:41
yoctozepto:-)12:41
*** cah_link has quit IRC12:42
rlljorgeyoctozepto: tkank you12:42
yoctozeptorlljorge: you're welcome12:43
*** shyamb has quit IRC12:45
*** rpittau is now known as rpittau|bbl12:52
*** jopdorp has joined #openstack-kolla12:57
jopdorphi!12:57
yoctozeptohi jopdorp12:57
jopdorpany hints about how to add my custom theme to my kolla ansible deployment?12:57
jopdorpI'm currently working with both an all-in-one deployment and a multinode12:57
yoctozeptojopdorp: customize the horizon image, build it, push it to registry and configure k-a to use it12:58
jopdorpwhich registry?12:58
yoctozeptojopdorp: docker registry, probably one you want to deploy internally12:58
jopdorpright12:59
jopdorpokay, thanks!12:59
yoctozeptominimal: https://docs.docker.com/registry/12:59
yoctozeptobut I use harbor12:59
yoctozeptohttps://goharbor.io/12:59
jopdorpthanks!13:00
kplant^ as do i13:00
*** dciabrin has quit IRC13:00
*** dciabrin has joined #openstack-kolla13:01
kplantyoctozepto: have you found a better way to delete a tag (and all images associated with that tag) than having to write code to do so?13:01
kplantit seems docker specifically left that ability out of non-docker enterprise >_<13:01
*** ricolin_ has quit IRC13:02
jopdorpwill this fix be merged to train?13:03
jopdorphttps://review.opendev.org/#/c/705949/213:03
patchbotpatch 705949 - kolla-ansible (stable/train) - Fix keystone fernet bootstrap - 2 patch sets13:03
yoctozeptokplant: you mean like prune?13:03
yoctozeptojopdorp: yeah, it needs a little more fixing on master still13:03
kplantyoctozepto: in the registry, not on the local docker hosts13:03
jopdorpokay, I think I encountered a bug that will be fixed by it13:04
yoctozeptokplant: ah, you mean the basic docker registry? I thought it was painful to manage so moved to harbor13:04
yoctozeptokplant: it's doable from harbor ui afair13:04
kplantyeah, but i have the same probably with harbor13:05
kplanti see no way to mass delete images :-|13:05
yoctozeptojopdorp: care to share?13:05
yoctozeptokplant: ah, mass...13:05
kplantyeah, like delete all images with tag "stein-dev"13:05
hrworaclelinux fails in stein...13:06
hrwdo we care?13:06
yoctozeptohrw: badly?13:07
yoctozeptohrw: ah, tripleo13:08
yoctozeptohrw: easy to fix, disable building13:08
yoctozeptohrw: only tripleo wants it13:08
jopdorpi can't find the exact error i was facing right now, sorry!13:09
yoctozeptojopdorp: no problem, the fix is needed anyways13:10
yoctozeptothe good thing is code/logic quality grows with every cycle :-)13:10
rlljorgeme again LOL ! Another erro13:12
rlljorgefatal: [control03]: FAILED! => {"changed": true, "msg": "'Traceback (most recent call last):\\n  File \"/tmp/ansible_ymVEod/ansible_module_kolla_docker.py\", line 983, in main\\n    result = bool(getattr(dw, module.params.get(\\'action\\'))())\\n  File \"/tmp/ansible_ymVEod/ansible_module_kolla_docker.py\", line 731, in13:12
rlljorgerecreate_or_restart_container\\n    self.start_container()\\n  File \"/tmp/ansible_ymVEod/ansible_module_kolla_docker.py\", line 744, in start_container\\n    self.pull_image()\\n  File \"/tmp/ansible_ymVEod/ansible_module_kolla_docker.py\", line 556, in pull_image\\n    repository=image, tag=tag, stream=True\\n  File13:12
rlljorge\"/usr/lib/python2.7/site-packages/docker/api/image.py\", line 415, in pull\\n    self._raise_for_status(response)\\n  File \"/usr/lib/python2.7/site-packages/docker/api/client.py\", line 263, in _raise_for_status\\n    raise create_api_error_from_http_exception(e)\\n  File \"/usr/lib/python2.7/site-packages/docker/errors.py\", line 31, in13:12
rlljorgecreate_api_error_from_http_exception\\n    raise cls(e, response=response, explanation=explanation)\\nNotFound: 404 Client Error: Not Found (\"manifest for kolla/centos-binary-fluentd:8.0.0 not found: manifest unknown: manifest unknown\")\\n'"}13:12
jopdorpyoctozepto: do you have any docs about how to use a docker image from my own registry?13:14
rlljorgejopdorp from docker13:14
rlljorgeops13:14
rlljorgemy fault sorry13:14
jopdorpsetting up my own registry won't be a problem13:16
jopdorpbut how to I tell kolla to use it for the horizon image?13:16
kplantwhat you should do is build them all into your local registry13:17
kplanthttps://docs.openstack.org/kolla/latest/admin/image-building.html13:17
kplantyou could technically only put horizon in there, change the registry and pull/deploy with "--tags horizon"13:17
kplantbut i don't think that's a great idea13:17
kplantunless you're only testnig13:17
kplantyoctozepto: i think harbor has a custom api call, you can throw a DELETE at /repositories/{repo_name}/tags/{tag}13:18
*** kklimonda has joined #openstack-kolla13:18
kplantwhy didn't they put that in the UI >:(13:18
*** skramaja has quit IRC13:18
*** gmann has quit IRC13:23
*** vkmc has quit IRC13:24
jopdorpkplant: so how would I tell kolla to use my own registry?13:26
*** irclogbot_0 has quit IRC13:26
jopdorpnever mind, found it at: https://docs.openstack.org/kolla/latest/admin/image-building.html13:26
openstackgerritMark Goddard proposed openstack/kolla-ansible master: CI: Use auto-detected python interpreter except on CentOS 7  https://review.opendev.org/70626513:27
*** cz3 has quit IRC13:28
*** irclogbot_0 has joined #openstack-kolla13:29
*** generalfuzz has quit IRC13:30
*** rpittau|bbl is now known as rpittau13:30
hrwyoctozepto: I look at https://review.opendev.org/#/c/634019 and have that feeling that fluentd Dockerfile needs to be rewritten from scratch13:32
patchbotpatch 634019 - kolla - Do not gem install for binary type - 6 patch sets13:32
hrwgave -113:34
*** dciabrin has quit IRC13:55
*** jopdorp has quit IRC13:55
*** dciabrin has joined #openstack-kolla13:57
*** dciabrin has quit IRC13:57
*** cz3 has joined #openstack-kolla14:00
rlljorgeHi there ! I have 2 noob questions ;-)  1) Can I rollback/downgrade from train to stein for example ? 2) Can I change Horizon logo using kolla ?14:01
*** generalfuzz has joined #openstack-kolla14:02
hrwrlljorge: 2. create own theme and add it into image via template overrides14:03
*** kplant has quit IRC14:04
rlljorgehrw Do you know if exists any documentation or example by this ?14:04
*** vkmc has joined #openstack-kolla14:06
*** kplant has joined #openstack-kolla14:06
hrwrlljorge: moment14:07
hrwrlljorge: https://git.linaro.org/leg/sdi/kolla/ldc-overlay.git/commit/?id=2c4abf12eb37a9636e315cfc0dc3b7221d17f34214:08
hrwrlljorge: this override adds Linaro's theme to horizon14:08
rlljorgehrw Great ! I will try .... thank you !14:09
yoctozeptohrw: +1 to your -114:10
yoctozeptorlljorge: downgrades are not possible14:11
hrwrlljorge: if you want your theme to be default then the easiest way is to have just one theme installed ;D14:11
yoctozeptothis is irrespective of deployment method14:11
openstackgerritMerged openstack/kolla-ansible stable/rocky: Fix qemu loading of ceph.conf (permission error)  https://review.opendev.org/70607314:37
openstackgerritMerged openstack/kolla-ansible stable/stein: Fix qemu loading of ceph.conf (permission error)  https://review.opendev.org/70607214:37
openstackgerritMerged openstack/kolla stable/stein: nova-libvirt: add UEFI packages to support UEFI instances  https://review.opendev.org/70606114:41
rlljorgeThe bug https://bugs.launchpad.net/cinder/+bug/1860739 not happens in stein release .... In a new deployment what the best choise in your opinion , go head with stein release and upgrade in future or wait for fix in train release ? backup/restore is important for my customers.14:43
openstackLaunchpad bug 1860739 in OpenStack Compute (nova) "Cinder backup failed for restore volume snapshot (for instance snapshot)" [Undecided,Confirmed]14:43
rlljorgeyoctozepto: your opinion is very importante ....14:46
rlljorgekplant your opinion is very important14:46
mnasiadkayoctozepto, hrw, mgoddard: https://review.opendev.org/#/c/704309/14:48
patchbotpatch 704309 - kolla-ansible - Remove kolla-ceph - 8 patch sets14:48
hrwmnasiadka: 'Kolla-Ansible Ceph deployment mechanism has been deprecated in Train [1].' - had to read it three times to notice kolla-ansible not ceph-ansible ;D14:49
mnasiadkahaha14:49
*** rphillips has quit IRC14:49
*** rphillips has joined #openstack-kolla14:50
hrwmnasiadka: and looks like I need to read external ceph docs first14:51
mnasiadkahrw: happy reading14:51
hrwmnasiadka: as all those cinder_backend_ceph like vars gets now set to 'no'14:51
hrwwhile it feels like '{{ external_ceph }}' would be better fit there14:52
hrwor sth14:52
hrwmnasiadka: the more I look at it the less I understand ;D14:57
*** vmixor has joined #openstack-kolla15:04
hrwto setup ceph cluster I need 3 machines for osd, right? mon runs on each, mgr on any?15:04
mnasiadkathink of mon and mgr as the control plane - usually 3 nodes15:05
mnasiadkaosd is where the data resides15:05
hrwso 3 trash pc + 1TB hdd to each15:06
*** hrw has quit IRC15:15
*** hrw has joined #openstack-kolla15:15
*** TrevorV has joined #openstack-kolla15:30
*** spiette has quit IRC15:32
openstackgerritAdeberg proposed openstack/kolla-ansible master: Swift: remove meta field from rsync command  https://review.opendev.org/70630315:32
*** spiette has joined #openstack-kolla15:34
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: WIP: Create 'infra' type of images  https://review.opendev.org/69875316:00
hrwjust rebased to see CI failures16:00
mgoddardyoctozepto: you wanted to ping be about a role for CA certs? https://review.opendev.org/#/c/699888/23/ansible/roles/zun/tasks/config.yml@4216:09
patchbotpatch 699888 - kolla-ansible - Copy CA into containers. - 23 patch sets16:09
*** jopdorp has joined #openstack-kolla16:20
jopdorphi!16:20
jopdorphow would we go about configuring ldap as the backend for keystone?16:20
jopdorpi found this doc16:21
jopdorphttps://docs.openstack.org/kolla-ansible/rocky/reference/keystone-guide.html16:21
jopdorpit hints at using keystone_token_provider somehow16:21
noxoid_ /etc/kolla/config/keystone/domains/keystone.internal.conf16:22
noxoid_assuming you wanted the domain "internal" to be backed by ldap16:22
noxoid_you can then put your ldap config in that file as defined by keystone documentation16:22
jopdorpcould send a link to that documentation?16:23
jopdorpi found this16:23
jopdorphttps://docs.openstack.org/kolla-ansible/latest/admin/advanced-configuration.html16:23
noxoid_https://docs.openstack.org/keystone/stein/admin/configuration.html#integrate-identity-with-ldap16:23
jopdorpright16:24
jopdorpit's a keystone override config right?16:24
jopdorphow could i derive this file /etc/kolla/config/keystone/domains/keystone.internal.conf16:24
jopdorp?16:24
jopdorpI feel like that is the key to making many configuration changes16:24
noxoid_http://paste.openstack.org/show/789232/16:26
noxoid_thats what we use to integrate with our freeipa service. not all settings may be correct but it minimally works16:26
jopdorpthat helps, thank you!16:26
jopdorpstill, how did you find that it had to be in this file: /etc/kolla/config/keystone/domains/keystone.internal.conf16:27
jopdorp?16:27
noxoid_by reading the kolla-ansible source code16:27
jopdorpright16:27
jopdorpokay16:27
hrwok. failed as expected16:40
*** rpittau is now known as rpittau|afk16:40
*** jopdorp has quit IRC16:44
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: WIP: Create 'infra' type of images  https://review.opendev.org/69875316:45
hrwsome debugs and dropped many CI jobs16:46
*** k_mouza has quit IRC16:49
*** jovial[m] has joined #openstack-kolla16:51
*** weshay|ruck is now known as weshay16:52
jovial[m]How can we stop old config files persisting in the fluentd container?16:52
jovial[m]I was thinking we could just copy the folders and not *.conf in config_files - thoughts?16:54
jovial[m]that should nuke the old files on restart, right?16:54
jovial[m]any downsides to that?16:55
*** carli has quit IRC17:00
mgoddardyoctozepto: so much for those saved CI cycles :p17:03
mnasiadkamgoddard: you talk about it like it's a princess worth saving :)17:04
mgoddardand I am the frog17:04
mgoddardjovial[m]: I'm not sure about the behaviour of directory copies in that script17:05
mgoddardbut nuking implies some extra work17:05
jovial[m]merge: merges the source directory into the target directory instead of replacing it. Boolean, defaults to false.17:06
mgoddardmaybe we need an option like rsync's --delete17:06
*** ktibi has quit IRC17:06
mgoddardanyway, heading of17:06
openstackgerritWill Szumski proposed openstack/kolla-ansible master: Delete stale fluent config on restart  https://review.opendev.org/70632717:15
*** ivve has quit IRC17:22
openstackgerritRadosÅ‚aw Piliszek proposed openstack/kolla master: CI: Replace cinder-lvm scenario with zun scenario  https://review.opendev.org/70605917:25
openstackgerritMerged openstack/kolla-ansible stable/train: Fix multiple issues with MariaDB handling  https://review.opendev.org/70541417:26
*** k_mouza has joined #openstack-kolla17:26
*** k_mouza has quit IRC17:30
openstackgerritRadosÅ‚aw Piliszek proposed openstack/kolla master: CI: Replace cinder-lvm scenario with zun scenario  https://review.opendev.org/70605917:32
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: WIP: Create 'infra' type of images  https://review.opendev.org/69875317:33
*** evrardjp has quit IRC17:33
hrwlet's see will it pass17:34
*** evrardjp has joined #openstack-kolla17:34
*** k_mouza has joined #openstack-kolla17:37
openstackgerritRadosÅ‚aw Piliszek proposed openstack/kolla master: CI: Replace cinder-lvm scenario with zun scenario  https://review.opendev.org/70605917:40
openstackgerritRadosÅ‚aw Piliszek proposed openstack/kolla master: CI: Replace cinder-lvm scenario with zun scenario  https://review.opendev.org/70605917:42
hrwinfra build progresses ;D17:43
*** dougsz has quit IRC17:54
*** jonaspaulo has quit IRC17:59
hrwINFO:kolla.common.utils.ceph-base:Successfully tagged kolla/debian-infra-ceph-base:9.1.017:59
hrwhttps://3358148dd61163c40ee5-fed9d885f48c89b65645af8cb1093bc8.ssl.cf5.rackcdn.com/698753/7/check/kolla-build-debian-source/a2823f5/kolla/build/ceph-base.log17:59
hrwYAY!18:00
hrwhttps://3358148dd61163c40ee5-fed9d885f48c89b65645af8cb1093bc8.ssl.cf5.rackcdn.com/698753/7/check/kolla-build-debian-source/a2823f5/kolla/docker-info.txt shows source AND infra images built18:01
hrwnow starts the problem...18:01
hrwCI needs to build binary/source/infra at one machine18:02
*** dcapone2004 has joined #openstack-kolla18:32
dcapone2004Does kolla-ansible support deployment using the neutron "trunk" plugin?18:32
dcapone2004I have attempted to modify the neutron.conf file in /etc/kolla/config to append trunk to the list of service-plugins and ran a reconfigure, but the neutron.conf generated and placed in the containers does not include the trunk service plugin18:33
*** spiette has quit IRC18:40
*** spiette has joined #openstack-kolla18:45
yoctozeptodcapone2004: hmm, it looks missing from default service_plugins18:47
yoctozeptodcapone2004: the ugly solution is to write down all service_plugins that are in use atm18:47
yoctozeptoand add trunk in an override18:47
yoctozeptothen reconfigure18:47
*** ivve has joined #openstack-kolla18:47
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: WIP: Create 'infra' type of images  https://review.opendev.org/69875318:50
hrwthis build will be tricky18:51
dcapone2004when you say in an override, you mean in the /etc/kolla/config/neutron/neutron.conf file?18:56
dcapone2004because that is basically what I did, but it doesn't seem to pick up the added service_plugin18:57
dcapone2004[DEFAULT]18:58
dcapone2004service_plugins=firewall_v2,vpnaas,trunk18:58
dcapone2004is what is in the /etc/kolla/config/neutron/neutron.conf file18:58
dcapone2004and inside the neutron-server container after reconfigure, I get...19:08
dcapone2004docker exec -it 5adc50321697 cat /etc/neutron/neutron.conf | grep plugin19:08
dcapone2004core_plugin = ml219:08
dcapone2004service_plugins = firewall_v2,router,vpnaas19:08
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: WIP: Create 'infra' type of images  https://review.opendev.org/69875319:10
yoctozeptodcapone2004: it's  /etc/kolla/config/neutron.conf19:13
yoctozeptouse directory for per-service overrides19:13
dcapone2004ok...I thought neutron would be considered a "service"....what would be considered a service for purposes of overrides so I can update my docs19:14
*** k_mouza has quit IRC19:19
*** k_mouza has joined #openstack-kolla19:25
*** k_mouza has quit IRC19:30
dcapone2004nice...that worked, thanks...just wish things were a bit more clear on what is a service and what isn't to know the correct place for overrides19:41
openstackgerritRadosÅ‚aw Piliszek proposed openstack/kolla master: CI: Replace cinder-lvm scenario with zun scenario  https://review.opendev.org/70605919:42
*** rlljorge has quit IRC19:51
*** jonaspaulo has joined #openstack-kolla19:51
openstackgerritRadosÅ‚aw Piliszek proposed openstack/kolla master: CentOS 8: Add deploy jobs in CI  https://review.opendev.org/70595219:53
yoctozeptodcapone2004: well, the docs mention both locations afair19:54
*** openstackstatus has joined #openstack-kolla19:56
*** ChanServ sets mode: +v openstackstatus19:56
dcapone2004yes, I thought it meant that it should work in either location, not necessarily that only one of the locations would work and to try all19:57
openstackgerritWill Szumski proposed openstack/kolla-ansible master: Delete stale fluent config on restart  https://review.opendev.org/70632720:00
*** tonythomas has quit IRC20:07
yoctozeptodcapone2004: nah, dirs are for specific services like neutron-server for the api server20:10
*** kplant has quit IRC20:11
*** gfidente is now known as gfidente|afk20:17
*** jonaspaulo has quit IRC20:24
openstackgerritRafael Folco proposed openstack/kolla master: [WIP] centos8: remove rabbitmq pinning  https://review.opendev.org/70364020:32
openstackgerritRafael Folco proposed openstack/kolla master: [WIP] centos8: remove rabbitmq pinning  https://review.opendev.org/70364020:33
dking_desktopCould somebody point me in the right direction as to what is going on when I see the following error in the ironic-inspector log: NotFound: Can not transition from state 'starting' on event 'start' (no defined transition)20:46
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: WIP: Create 'infra' type of images  https://review.opendev.org/69875320:47
*** irclogbot_0 has quit IRC20:53
*** irclogbot_3 has joined #openstack-kolla20:54
openstackgerritAntony Messerli proposed openstack/kolla master: Add ssh clients to Ironic Conductor container  https://review.opendev.org/70614921:02
*** k_mouza has joined #openstack-kolla21:11
*** k_mouza has quit IRC21:16
*** TrevorV has quit IRC21:16
*** dcapone2004 has quit IRC21:17
*** gfidente|afk has quit IRC21:22
yoctozeptodking_desktop: best try on #openstack-ironic21:40
dking_desktopyoctozepto: Thank you.21:45
*** vmixor has quit IRC22:08
*** dking_desktop has quit IRC22:09
*** negronjl has quit IRC22:12
*** negronjl has joined #openstack-kolla22:17
*** ivve has quit IRC22:31
*** k_mouza has joined #openstack-kolla22:49
openstackgerritAdeberg proposed openstack/kolla-ansible master: Swift: remove meta field from rsync command  https://review.opendev.org/70630322:58

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!