Monday, 2020-02-03

*** xinliang has joined #openstack-kolla00:26
*** k_mouza has joined #openstack-kolla00:57
*** k_mouza has quit IRC01:05
*** k_mouza has joined #openstack-kolla01:15
*** k_mouza has quit IRC01:19
*** k_mouza has joined #openstack-kolla01:56
*** k_mouza has quit IRC02:00
*** xinliang has quit IRC02:36
*** xinliang has joined #openstack-kolla03:07
*** Tony31 has joined #openstack-kolla03:26
*** Tony31 has quit IRC03:33
*** xinliang has quit IRC03:53
openstackgerritMerged openstack/kolla-ansible stable/train: Fix qemu loading of ceph.conf (permission error)  https://review.opendev.org/70532904:13
*** mixor has joined #openstack-kolla04:31
*** negronjl has quit IRC04:42
*** negronjl has joined #openstack-kolla04:43
openstackgerritDincer Celik proposed openstack/kolla master: WIP: Upgrade Elasticsearch and Kibana to 6.x  https://review.opendev.org/64774805:07
*** hongbin has joined #openstack-kolla05:23
*** mixor has quit IRC05:32
*** evrardjp has quit IRC05:33
*** evrardjp has joined #openstack-kolla05:34
*** icarusfactor has joined #openstack-kolla05:36
*** factor has quit IRC05:38
*** skramaja has joined #openstack-kolla05:49
*** icarusfactor has quit IRC06:02
*** hongbin has quit IRC06:16
*** cah_link has joined #openstack-kolla06:18
*** ricolin has quit IRC06:24
osmanlicilegimorning06:33
cosmicsoundGood day osmanlicilegi06:35
osmanlicilegicosmicsound: thx, you too :]06:40
*** Luzi has joined #openstack-kolla06:48
*** Tony31 has joined #openstack-kolla06:58
Tony31hi guys - could anyone help me get my head around the "all in one" kayobe approach?06:59
Tony31My problems are around the networking - what ever I try doesnt seem to work, or rather gets past the problem but then I find another06:59
Tony31my latest problem seems to be that the aio hypervisor/host can't route to the seed VM's IP because the host has real network and bridge network in the same subnet07:00
Tony31For example, my host VM has IP 192.168.1.100. So I have `seed_hv_ip=192.168.1.100`. If I then use 192.168.1.0/24 for the network allocations, this passes all the validations but I'm unable to access the seed VM once it's deployed07:04
*** mixor has joined #openstack-kolla07:08
*** numans has joined #openstack-kolla07:20
*** cah_link has quit IRC07:30
*** mixor has quit IRC07:33
*** jbadiapa has joined #openstack-kolla07:39
*** pcaruana has joined #openstack-kolla07:43
*** gfidente has joined #openstack-kolla07:52
yoctozeptoTony31: yeah, they seem to have to be different, could be prechecks are not tight enough there @ mgoddard, mnasiadka07:57
Tony31hi Yoctozepto - i started again and did not change any config files this time... I have managed to progress beyond the last problem I had. I am not sure if this will be functional (yet) but I'll see :)  (learning all the way...)07:58
Tony31thanks for your reply :)07:58
yoctozeptoTony31: yeah, getting openstack done properly is a great sysops adventure :-)07:59
Tony31:)  I felt like Alice in Wonderland last time :)08:00
yoctozeptohaha08:03
mnasiadkayoctozepto: it's trial by fire, not adventure :)08:03
yoctozeptomnasiadka: shhh, don't scare users :-)08:04
*** rpittau|afk is now known as rpittau08:05
*** tonythomas has joined #openstack-kolla08:07
*** dswebb has quit IRC08:08
*** xinliang has joined #openstack-kolla08:09
*** luksky has joined #openstack-kolla08:09
*** xinliang has quit IRC08:34
*** xinliang has joined #openstack-kolla08:41
mgoddardmorning08:46
mgoddardI'm out today and tomorrow08:46
yoctozeptoas a matter of fact, I am out tomorrow :-) (local conf)08:51
*** dougsz has joined #openstack-kolla08:53
yoctozeptomgoddard: if you have just one moment, I would be glad to get your blessing on mariadb https://review.opendev.org/70450708:55
patchbotpatch 704507 - kolla-ansible - Followup on MariaDB handling fixes - 3 patch sets08:55
mgoddardyoctozepto: let's see how train wifi goes08:56
yoctozeptomgoddard: crossing fingers08:56
Tony31kayobe guys - I have a couple of machines I built for machine learning, with multiple nvidia GPUs etc. .  I am reading up the stackHPC website about ironic and managing hardware. I dont have the right nvidia GPUs to be able to virtualise them (example, run the bare metal as a host and run VMs on top, with pci pass-through). But I wanted to ask -08:58
Tony31could I manage this hardware from openstack and allow people to deploy and access the real hardware, then destroy when theyr'e done?08:58
Tony31basically, use the real hardware and gpu's like a VM?08:59
Tony31I guess I'd need to make a custom image with the drivers08:59
mgoddardhi Tony31. Are you sure you can't use PCI passthrough? No sharing, just pass through the entire GPU to a VM09:01
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: DNM: keepalived testing  https://review.opendev.org/69092009:01
mgoddardotherwise, you could use ironic to make the bare metal available to users. In that case it's treated like a black box - the user can do (almost) what they like with the machine09:02
yoctozeptoTony31: like mgoddard says, pci passtrough requires only vt-d in case of intel to work, no nvidia specific stuff nor sr-iov09:02
yoctozepto(mentioning because some assume otherwise)09:03
yoctozeptomgoddard: thx, I'll be backporting this to stable branches, at least train is nice as it affects upgrade jobs stability09:04
Tony31I have 3 x nvidia titan XP in each system - I was told that they wont work for virtualisation o_0 - but it would be better using them in Openstack I think anyway09:04
yoctozeptoTony31: it depends on platform/mobo, not gpu09:05
mgoddardyoctozepto: ok. Let's try to land the original patch and follow up together09:05
yoctozeptoany pci device can go passthrough09:05
yoctozeptomgoddard: yes, indeed09:05
Tony31the system can be virtualised. I'll need to have a play with it - thanks for the info :)09:05
Tony31it's dual xeon, tons of RAM, trying to find info on the motherboard09:07
yoctozeptoTony31: look up vt-d, dmar and iommu in google09:07
yoctozeptoit generally needs to be enabled in bios setup and then linux kernel09:07
Tony31thank you for that info09:09
*** bersace has quit IRC09:09
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible upgrade jobs  https://review.opendev.org/70498409:12
Tony31ok - i managed to get the seed deployed in the all-in-one.. the "network" is hidden behind the VM. Heading off now and will deploy the control plane tomorrow. Thanks for the help again today :)09:13
Tony31have a good one09:13
*** k_mouza has joined #openstack-kolla09:15
*** xinliang has quit IRC09:16
*** xinliang has joined #openstack-kolla09:17
*** Tony31 has quit IRC09:18
*** dougsz has quit IRC09:18
*** dougsz has joined #openstack-kolla09:21
*** bersace has joined #openstack-kolla09:30
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: DNM: haproxy timeout - raise value for server  https://review.opendev.org/69070209:36
*** xinliang has quit IRC09:41
*** Tengu has quit IRC09:46
*** hjensas has joined #openstack-kolla09:47
*** Tengu has joined #openstack-kolla09:48
*** Tengu has quit IRC09:54
*** Tengu has joined #openstack-kolla09:55
openstackgerritDincer Celik proposed openstack/kolla master: WIP: Upgrade Elasticsearch and Kibana to 6.x  https://review.opendev.org/64774810:01
*** Tengu has quit IRC10:03
*** skramaja has quit IRC10:03
*** Tengu has joined #openstack-kolla10:03
openstackgerritMerged openstack/kolla-ansible master: Followup on MariaDB handling fixes  https://review.opendev.org/70450710:06
*** Tengu has quit IRC10:08
*** Tengu has joined #openstack-kolla10:13
*** jopdorp has joined #openstack-kolla10:14
*** Tengu_ has joined #openstack-kolla10:14
*** dougsz has quit IRC10:19
*** dougsz has joined #openstack-kolla10:27
*** jopdorp has quit IRC10:41
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible upgrade jobs  https://review.opendev.org/70498410:47
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: Ceph-Ansible upgrade jobs  https://review.opendev.org/70498410:47
*** jbadiapa has quit IRC10:49
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible stable/train: Fix multiple issues with MariaDB handling  https://review.opendev.org/70541410:55
*** reph has joined #openstack-kolla11:08
rephhi, deploy using kolla 9.0.1 yields "pull access denied for kolla/ubuntu-binary-cloudkitty-api, repository does not exist or may require \\'docker login\\"11:09
rephkolla/ubuntu-binary-cloudkitty-api is in fact not available on dockerhub11:11
*** reph has quit IRC11:21
*** reph has joined #openstack-kolla11:23
mnasiadkareph: cloudkitty ubuntu binary is marked as unbuildable11:27
mnasiadkatry ubuntu-source11:27
rephmnasiadka thanks, didn't stumble upon that piece of information :/11:39
rephcloudkitty will also fail when using source, unfortunately, at least that's what it did on friday11:41
rephit said "cloudkitty "AttributeError: 'NoneType' object has no attribute 'auth_url'""11:42
*** rpittau is now known as rpittau|bbl11:42
rephin cloudkitty/tenant_fetcher/keystone.py somewhere around line 50, wheree it tries to read from self.auth.auth_url11:44
mnasiadkareph: then it might be something wrong with configuration done by k-a11:50
mnasiadkareph: never used that, so can't tell11:50
*** luksky has quit IRC11:52
yoctozeptoreph: https://docs.openstack.org/kolla/train/support_matrix.html11:55
yoctozeptothe other issue might be a bug in k-a11:55
yoctozeptoplease report11:56
yoctozeptolink: https://bugs.launchpad.net/kolla-ansible11:56
*** jbadiapa has joined #openstack-kolla12:03
*** luksky has joined #openstack-kolla12:17
*** cah_link has joined #openstack-kolla12:19
*** kplant has joined #openstack-kolla12:21
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: Fix keystone fernet bootstrap  https://review.opendev.org/69871012:41
mnasiadkayoctozepto: if you have a moment - I would be grateful for looking into this: https://review.opendev.org/#/c/70498412:51
patchbotpatch 704984 - kolla-ansible - Ceph-Ansible upgrade jobs - 12 patch sets12:51
*** hjensas has quit IRC13:16
*** hjensas has joined #openstack-kolla13:27
*** theZorro266 has joined #openstack-kolla13:30
*** rpittau|bbl is now known as rpittau13:34
theZorro266Hello, couldn't find a solution with some searches: I have been following https://docs.openstack.org/kolla-ansible/train/reference/storage/external-ceph-guide.html after using ceph-ansible as I understand ceph deployment by kolla-ansible is deprecated. I am running into the error "MissingRequiredSource: /var/lib/kolla/config_files/ceph.* file is not found" although the files exist in13:40
theZorro266"/etc/kolla/config/glance/". Is this a user error? The guide tells me nothing about permissions, so I assumed it should be fine with the same permissions as "/etc/kolla/glance-api/".13:40
kplantwhat services are you using ceph for?13:43
kplantand did you include both ceph.conf and the ceph keyrings?13:43
theZorro266I would like to use ceph for glance, cinder, nova, gnocchi and manila. I am generating keys for all of these services and the keys are in the appropiate directories as written in the linked guide13:44
*** rlljorge has joined #openstack-kolla13:44
theZorro266And yeah, both ceph.client.glance.keyring and ceph.conf are in /etc/kolla/config/glance/13:44
theZorro266but not in /etc/kolla/glance-api/13:45
yoctozeptomnasiadka: I will13:45
theZorro266and I understood that they should be merged into /etc/kolla/glance-api/13:45
kplantthis is what mine looks like: https://pastebin.com/hSDusB7v13:46
theZorro266this is what it looks like over here: https://pastebin.com/iyghGxnM13:49
theZorro266Which version of kolla-ansible are you using? I see things have changed regarding glance external ceph setup between 9.0.1 and master13:51
kplantstable/stein and stable/train13:51
kplantso13:52
kplantyour permissions are 066013:52
kplantwhat user are you running ansible as?13:52
kplantif it's not root or in the root group it won't read them13:53
theZorro266I see. I run ansible as a user, of course13:53
theZorro266That is what I was asking above. I assumed /etc/kolla/glance-api/ should have the same permissions as in /etc/kolla/config/glance13:54
kplantif you o+r on ceph.* you should be good13:55
theZorro266Yeah, not forgetting the directories, thank you! :) I'll try and see if that works13:57
*** reph has quit IRC14:07
theZorro266It seems to work now, thanks again kplant :) This seems related to the "with_fileglob" in roles/glance/tasks/external_ceph.yml, because the list of files is obviously gathered as the user running ansible. This would mean only the folders in /etc/kolla/config should be world-readable/listable and not the keyrings/configs. Shouldn't this be in the docs? I could make a PR, but I'd need a hint14:14
theZorro266where to insert it to.14:14
theZorro266strike that. It copies the files, so it needs read access to the files as well14:15
kplant^14:15
theZorro266but still, it would be better to have the files user-owned by the user running Ansible in that case14:16
kplanti think that's in the quickstart14:16
theZorro266thats only the /etc/kolla directory14:17
theZorro266and all files within are being created and owned by root:root with zero permissions for world14:18
theZorro266that's where it misled me14:18
*** Luzi has quit IRC14:22
theZorro266the external ceph guide is the first time I came into contact with the /etc/kolla/config directory, so any other place with that info should at least be linked from there in my understanding14:23
*** TrevorV has joined #openstack-kolla14:33
*** dougsz has quit IRC14:33
mnasiadkatheZorro266: it's in the docs - https://docs.openstack.org/kolla-ansible/latest/admin/advanced-configuration.html#openstack-service-configuration-in-kolla - just read it :)14:42
rlljorgeHI folks, some ideia ?14:50
rlljorgefatal: [network02]: FAILED! => {14:50
rlljorgeexact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Set fluentd facts\n  ^ here\n"14:50
theZorro266mnasiadka: thanks :) What do you mean with "in the docs" though? When I search for "/etc/kolla/config" on my search engine (not the documentation's search), only then I will find that page14:50
theZorro266My point is that the user should be made aware of ownership of directories and files in /etc/kolla/config as it is different from the rest of the directories and files in /etc/kolla14:52
*** dougsz has joined #openstack-kolla14:52
mnasiadkatheZorro266: how different?14:57
mnasiadkarlljorge: ansible version? kolla_ansible package version?14:58
*** reph has joined #openstack-kolla14:59
*** jbadiapa has quit IRC15:02
theZorro266mnasiadka: well, kolla-ansible creates /etc/kolla/glance-api with root:root and 0770, now in /etc/kolla/config/glance my user running kolla-ansible must have at least read access15:02
theZorro266I am not saying the later does not make sense, it just didn't strike my mind soon enough15:04
rlljorgemnasiadka: ansible 2.9.2 stable/stein tag 8.1.0 , I deploy train release with success because a bug in cinder/nova I will train stein release15:04
rlljorgedeployed*15:05
*** vmixor has joined #openstack-kolla15:08
admin0ok guys .. i want to test/document a multinode kolla deployment .. i am using osa(openstack ansible) all the way, but now want to give kolla a try .. i need a bit of help with the mappings15:17
admin0i want to move database ( mysql) and rabbitmq to their own hosts  ( not the controllers) .. how do I do this ?15:17
kplantpretty sure there's already host groups in the example inventory files for that15:19
admin0test rig:15:19
admin03x controllers -- c1 c2 c3 ; 3x data servers - d1 d2 d3 ( for mysql and rabbitmq)  ; 3x nova computes - n1 n2 n3 (kvm based) ; 3x lxd computes - x1 x2 x3 ( for use with lxd) ; 3x ceph osd nodes - s1 s2 s3  ;3x swift nodes - os1 os2 os3 ;  2x swift proxy node - op1  and op215:19
kplantyep, both mariadb and rabbitmq already have groups defined15:19
admin0awesome15:19
admin0https://docs.openstack.org/kolla-ansible/latest/user/multinode.html  -- does not say much15:20
kplant"For more advanced roles, the operator can edit which services will be associated in with each group. Keep in mind that some services have to be grouped together and changing these around can break your deployment"15:21
kplanti think that's the closest to what you're looking for15:21
kplantbut if you take a look at multinode they're already there15:21
*** mloza has joined #openstack-kolla15:25
mlozaIs there plan to backport this to stein https://review.opendev.org/#/c/686369/ ?15:26
patchbotpatch 686369 - kolla-ansible - Allow passing arguments to RabbitMQ server (MERGED) - 5 patch sets15:26
kplant^ that is a pretty nice one to fix the cpu usage15:35
kplantmnasiadka yoctozepto mgoddard what do you guys think about that?15:48
kplanthttps://bugs.launchpad.net/kolla-ansible/+bug/1846467 impacts stein and that patch gives ops the ability to correct it15:48
openstackLaunchpad bug 1846467 in kolla-ansible train "RabbitMQ container - high CPU load on multicore systems" [Medium,Fix released]15:48
noxoidyea my rabbitmq container is using 120-190% cpu mostly idle on a stein deployment15:58
kplantmine does the same, on stein15:59
admin0kplant, just to reconfirm .. if i want separate api and storage but on the same interface, i need to say for example create 2 vlans, create 2 bridges with br_ (underscore) name and then pass it like api_interface: br_api right ?16:01
kplantthey don't have to be bridges16:03
kplantbut yes16:03
*** jbadiapa has joined #openstack-kolla16:03
kplantcan be a raw interface or bond also16:03
admin0ok .. so ovs is used all the way and not linuxbridge16:04
kplantem0.11,em0.12 bond0.11,bond0.12 etc.16:04
kplantdepends, you can configure k-a to use linubridge and not ovs16:04
kplantheading to lunch, i'll be around after16:05
admin0where do I map some hosts to be using kvm and some using lxd16:05
*** luksky has quit IRC16:32
*** rpittau is now known as rpittau|afk16:33
oncall-pokemonWith kolla, in the inventory file you can setup multiple servers under [mariadb]. What does it mean if you add two or more? will replication start or something?16:52
kplantadmin0: you can override variables per host just as you do with ansible16:59
kplantin the inventory or host_vars16:59
rlljorgeoncall-pokemon: I think is need 3 os more because is a galera cluster17:00
openstackgerritDoug Szumski proposed openstack/kolla-ansible master: Extract useful information from HAProxy logs  https://review.opendev.org/70548417:02
dougsz^ For people who enjoy reading regex's17:02
oncall-pokemonrlljorge: not sure what that means. mind expanding a bit. I'm a bit of a noobie17:06
openstackgerritDoug Szumski proposed openstack/kolla-ansible master: Extract useful information from HAProxy logs  https://review.opendev.org/70548417:09
yankcrimeoncall-pokemon: yeah, if you specify more than one node it'll create a cluster for you17:12
yankcrimedata is replicated across all members of the cluster17:12
yankcrime(it's mariadb galera if you want to read more)17:12
oncall-pokemonawesome thanks17:18
oncall-pokemonCan I add more to the list without destruction to the one on the controller now?17:18
oncall-pokemonwe already have some data17:18
rlljorgeoncall-pokemon: Yes17:19
rlljorgeIf the answer is 1 node, then you need 3 nodes: when 1 node fails, the two remaining nodes have quorum.17:20
kplanti think the answer is growing from a 1 node mariadb to a 3 node galera17:20
kplantquestion**17:20
kplanti don't know if that's supported17:20
admin0kplant, is something like graylog supported ? to make a central logging server ?17:27
kplantkibana is17:28
kplantthere's fluentd, collectd, .... i think there's more17:28
admin0kolla_internal_tls_cert . is this a pem with the crt, ca cert and keyfile all inside those block ?17:28
admin0i have say domain.crt and domain.key .. what goes where17:29
kplanti just feed it a pem with the entire chain and key17:29
admin0ok17:33
*** evrardjp has quit IRC17:33
*** evrardjp has joined #openstack-kolla17:34
*** k_mouza has quit IRC17:39
*** jbadiapa has quit IRC17:40
dougszadmin0: there is Fluent + Kibana + ES (enable_central_logging) or Monasca for centralised logging17:52
*** gfidente is now known as gfidente|afk17:59
*** dougsz has quit IRC18:03
*** k_mouza has joined #openstack-kolla18:11
*** k_mouza has quit IRC18:15
*** k_mouza has joined #openstack-kolla18:15
*** tonythomas has quit IRC18:19
*** weshay is now known as weshay|ruck18:30
*** reph has quit IRC18:34
*** luksky has joined #openstack-kolla18:37
*** cah_link has quit IRC18:40
rlljorgeHello @kplant in the last week we talk about the bug https://bugs.launchpad.net/kolla-ansible/+bug/1860739 Do you can share your kolla_base_distro with me ?18:41
openstackLaunchpad bug 1860739 in OpenStack Compute (nova) "Cinder backup failed for restore volume snapshot (for instance snapshot)" [Undecided,Confirmed]18:41
yoctozeptooncall-pokemon, yankcrime, rlljorge, kplant: just to clarify - scaling mariadb galera cluster up *is* supported18:46
yoctozeptoscaling down is kinda too, it just involves a manual step of stopping extra mariadb when cluster is healthy with all the other member18:47
yoctozeptos alive18:47
yoctozepto(premature enter)18:47
rlljorgeyoctozepto nice ! thanks18:48
*** rlljorge has quit IRC19:36
yoctozeptomnasiadka: is that keystone fix ready to merge or you really expected a failure?19:50
mnasiadkayoctozepto: well, it seems to work - was really just sarcastic :)19:53
mnasiadkayoctozepto: I tested it also on scale out19:54
yoctozeptomnasiadka: very well!19:54
mnasiadkaWhich brings an observation - wouldn’t we want to have a periodic scale out job?19:54
mnasiadkaIt is probably something our users sometimes do... or not19:55
*** gfidente|afk is now known as gfidente19:56
*** jonaspaulo has quit IRC20:02
*** irclogbot_1 has quit IRC20:05
*** irclogbot_1 has joined #openstack-kolla20:06
yoctozeptomnasiadka: makes sense, add it to scenarios20:17
yoctozepto1->3 scale obviously :-)20:17
*** jonaspaulo has joined #openstack-kolla20:17
mnasiadkayoctozepto: yeah, 3 to 5 is rather an overkill :)20:20
yoctozeptomnasiadka: well, it tests if an existing multi does not get killed20:24
yoctozeptowe *might* want to have it running20:24
yoctozeptobut daily is an overkill20:24
yoctozeptoespecially if you overdo the enabled services20:24
mnasiadkayoctozepto: yeah, maybe weekly makes sense - 1->3->5 - it will probably fail multiple times :) I can start with 1->3 and lets see how that works20:27
yoctozeptomnasiadka: it must not fail, we fixed a lot20:28
yoctozeptomnasiadka: only haproxy/keepalived remains20:28
*** gfidente has quit IRC20:28
yoctozeptomnasiadka: I restored my machinery to play with it20:28
yoctozeptomnasiadka: hoping to get it fixed too20:28
yoctozeptomnasiadka: then we are golden like them kings 8-)20:29
mnasiadkayoctozepto: it would be good to understand why it fails, and how to make it better20:29
yoctozeptomnasiadka: that's the goal20:29
mnasiadkayoctozepto: if we are in keepalived dept - would be good to document a procedure of maintenance failover :)20:29
* yoctozepto not listening20:30
mnasiadkaHaha20:30
*** k_mouza has quit IRC20:34
*** kplant has quit IRC20:58
*** bjolo has quit IRC21:47
*** vmixor has quit IRC22:11
*** dciabrin has quit IRC22:38
*** dciabrin has joined #openstack-kolla22:43
*** TrevorV has quit IRC22:53

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!