Monday, 2020-06-29

*** cshen has joined #openstack-ansible00:15
*** cshen has quit IRC00:20
*** markvoelker has joined #openstack-ansible01:14
*** markvoelker has quit IRC01:46
*** markvoelker has joined #openstack-ansible01:57
*** cshen has joined #openstack-ansible02:16
*** markvoelker has quit IRC02:16
*** cshen has quit IRC02:24
*** cshen has joined #openstack-ansible02:30
*** cshen has quit IRC02:34
*** johanssone has quit IRC02:43
*** johanssone has joined #openstack-ansible02:45
*** cshen has joined #openstack-ansible03:44
*** cshen has quit IRC03:49
*** markvoelker has joined #openstack-ansible04:17
*** markvoelker has quit IRC04:22
*** udesale has joined #openstack-ansible04:28
*** evrardjp has quit IRC04:33
*** evrardjp has joined #openstack-ansible04:33
*** miloa has joined #openstack-ansible04:54
*** cshen has joined #openstack-ansible05:45
*** cshen has quit IRC05:49
*** jawad_axd has joined #openstack-ansible05:59
*** jawad_axd has quit IRC06:03
*** pcaruana has joined #openstack-ansible06:08
*** viks____ has joined #openstack-ansible06:42
*** shyamb has joined #openstack-ansible06:49
*** rpittau|afk is now known as rpittau06:51
*** jbadiapa has joined #openstack-ansible07:05
*** luksky has joined #openstack-ansible07:05
*** shyamb has quit IRC07:11
CeeMacmorning07:15
*** cshen has joined #openstack-ansible07:15
*** sshnaidm is now known as sshnaidm|ruck07:24
*** shyamb has joined #openstack-ansible07:32
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-lxc_hosts master: Do not install python2 or its dev package on Ubuntu Bionic hosts  https://review.opendev.org/73797907:47
*** shyam89 has joined #openstack-ansible07:54
openstackgerritJonathan Rosser proposed openstack/openstack-ansible master: Drop support for openSUSE  https://review.opendev.org/72554107:58
*** shyamb has quit IRC07:58
*** shyamb has joined #openstack-ansible07:58
*** shyam89 has quit IRC08:02
*** jawad_axd has joined #openstack-ansible08:11
*** jawad_axd has quit IRC08:20
*** jawad_axd has joined #openstack-ansible08:21
*** shyam89 has joined #openstack-ansible08:22
*** shyamb has quit IRC08:23
*** jawad_ax_ has joined #openstack-ansible08:30
*** jawad_axd has quit IRC08:35
*** jawad_ax_ has quit IRC08:48
*** jawad_axd has joined #openstack-ansible08:49
*** markvoelker has joined #openstack-ansible08:51
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-os_neutron master: Update calico to new repository location.  https://review.opendev.org/73108008:55
*** markvoelker has quit IRC08:56
*** shyam89 has quit IRC09:00
*** Berthe01 has joined #openstack-ansible09:00
*** shyam89 has joined #openstack-ansible09:00
*** also_stingrayza is now known as stingrayza09:09
*** markvoelker has joined #openstack-ansible09:24
*** ioni has quit IRC09:25
*** stingrayza has quit IRC09:25
*** stingrayza has joined #openstack-ansible09:29
*** markvoelker has quit IRC09:29
*** shyam89 has quit IRC09:30
*** shyamb has joined #openstack-ansible09:37
*** ioni has joined #openstack-ansible09:42
*** jawad_axd has quit IRC09:45
*** jawad_axd has joined #openstack-ansible09:49
*** shyamb has quit IRC10:00
*** shyamb has joined #openstack-ansible10:00
*** jawad_axd has quit IRC10:06
*** shyamb has quit IRC10:18
*** rpittau is now known as rpittau|bbl10:21
*** shyamb has joined #openstack-ansible10:21
*** tosky has joined #openstack-ansible10:57
admin0is it possible to de-couple an OSA+ceph to just osa while the ceph is being managed further by ceph-ansible ?11:00
*** shyamb has quit IRC11:00
*** udesale has quit IRC11:02
*** udesale has joined #openstack-ansible11:02
*** cshen has quit IRC11:02
*** cshen has joined #openstack-ansible11:03
*** cshen has quit IRC11:09
*** cshen has joined #openstack-ansible11:10
*** shyamb has joined #openstack-ansible11:11
*** strattao has joined #openstack-ansible11:14
watersjadmin0, yes can specify using an existing ceph cluster as your storage11:16
admin0i have one cluster done with osa+ceph .. checking if i want ceph to be managed further by ceph-ansible, how do I de-couple it ?11:17
*** stingrayza has quit IRC11:20
ionii'm actually scared to update ceph11:21
*** cshen_ has joined #openstack-ansible11:24
*** cshen has quit IRC11:26
*** stingrayza has joined #openstack-ansible11:26
openstackgerritGeorgina Shippey proposed openstack/openstack-ansible-os_keystone master: Identity providers can be created with specified domain  https://review.opendev.org/73674111:38
openstackgerritGeorgina Shippey proposed openstack/openstack-ansible-os_keystone stable/train: Identity providers can be created with specified domain  https://review.opendev.org/73820211:39
openstackgerritGeorgina Shippey proposed openstack/openstack-ansible-os_keystone stable/ussuri: Identity providers can be created with specified domain  https://review.opendev.org/73820111:40
jrosseradmin0: imho the biggest issue you have is the OSA dynamic inventory11:44
jrosserin order to stop OSA deploying things into the already existing ceph mon / whatever else containers you need to remove those from the inventory11:44
jrosserbut then who/what is responsible for those containers11:44
*** pcaruana has quit IRC11:50
admin0i will drop the idea till ceph+osa works .. when you guys ( the developers ) think they will drop ceph , then maybe there will be a migration plan forward12:00
*** rpittau|bbl is now known as rpittau12:00
jrosseradmin0: i'm not sure i follow? ceph+osa does work?12:02
*** pcaruana has joined #openstack-ansible12:02
*** rh-jelabarre has joined #openstack-ansible12:03
*** markvoelker has joined #openstack-ansible12:05
*** markvoelker has quit IRC12:06
admin0it works fine12:08
admin0just thinking of future ( taking a hint from kolla-ansible project, which dropped kolla+ceph from ussuri release)12:09
ioniwhy ?12:10
sepadmin0, https://blueprints.launchpad.net/kolla-ansible/+spec/remove-ceph  ; kolla does not need to deploy ceph since there are so many other tools to do it now.12:11
jrosserisn't that the removal of a custom deployment of ceph rather than leveraging an external tool like ceph-ansible, as is done in the OSA case?12:12
jrosserpersonally i think that the OSA calling ceph-ansible is a nice get-you-going hook, but you have to completely learn ceph-ansible anyway in order to put the right data in user_variables.yml12:15
jrosserand then once you dig into the interdependencies of ansible versions, ceph-ansible versions and the version of ceph chosen by OSA for each release it can quickly become more desirable from an operator POV to deploy openstack and ceph independently12:17
*** udesale_ has joined #openstack-ansible12:20
*** udesale has quit IRC12:22
*** vakuznet has quit IRC12:25
kleiniespecially the upgrade of Ceph Mimic to Nautilus along with OSA Stein to Train did not work for me in the past. Maybe this works today as it was very obvious, that a special ceph-ansible upgrade script needs to be run12:29
Berthe01Hi there, Could some please confirm me the statements I have to add in my user_config file in order to install functionnal masakari services (api, engine and monitors) on my OpnStk hosts ?12:30
ionikleini: https://bugs.launchpad.net/openstack-ansible/+bug/1860772 this?12:30
openstackLaunchpad bug 1860772 in openstack-ansible "Ceph OSD restart failure while upgrading from Stein to Train" [Undecided,New]12:30
Berthe01masakari-infra_hosts: seems to install only API + Engine12:30
Berthe01masakari-monitor_containers: seems to be what i need, does it looks ok for you ?12:31
kleiniyes, exactly, I even commented on that bug :-D12:31
ionikleini: seems fixed12:34
kleinithen I have to test that again12:34
ionikleini: test it and let me know :) i'm still on stein due to that12:41
*** shyam89 has joined #openstack-ansible12:43
kleiniioni, wrote down in my story to inform you once I tested it again.12:43
*** shyamb has quit IRC12:44
*** shyam89 has quit IRC12:47
*** mgariepy has joined #openstack-ansible12:51
*** mgariepy has quit IRC12:55
*** mgariepy has joined #openstack-ansible12:55
openstackgerritKevin Carter (cloudnull) proposed openstack/ansible-config_template master: Update config_template to use the builtin _get_remote_user function  https://review.opendev.org/73805813:06
mgariepygood morning everyone13:08
jrossero/ hello13:11
mgariepyjrosser, do you have a routed env with osa?13:24
jrossermgariepy: that can mean lots of things :) which bit?13:27
mgariepythe control plane13:27
jrossersort of13:28
mgariepyin order to migrage fip accross multiple rack, if each rack are routed i guess i'll have to have a /20~ish or something to move the IP.13:28
jrosserthe control plane is in it's own L2 subnets13:28
jrosserbut the computes are split into several others13:28
jrosserso rather than split "one controller per rack" sort of model I kept the control plane logically as one L2 domain13:29
mgariepyok13:29
jrosserbecause that lets me do vpc/mlag on the bonds to a pair of TOR13:30
admin0hey guys .. what was the link to upgrade the OS also from rocky to move to stein13:30
jrosserand keep the control plane wholly on on that pair of TOR13:30
jrosserwith then L3 uplinks to spine, and then L3 on to computes/storage and so on13:30
jrosserand i guess for FIP my network nodes are also in that same L2 "controlpane" bit13:31
jrosserbecasue of vrrp13:32
mgariepyadmin0, https://etherpad.opendev.org/p/osa-rocky-bionic-upgrade and13:32
mgariepyhttps://review.opendev.org/#/c/71080313:33
*** ioni has quit IRC13:46
*** spatel has joined #openstack-ansible13:47
*** namrata has joined #openstack-ansible14:02
*** dave-mccowan has quit IRC14:07
*** ioni has joined #openstack-ansible14:08
*** dave-mccowan has joined #openstack-ansible14:09
*** cshen_ has quit IRC14:28
namrataHi, I am on stable/train branch and when I install Nova, my installation fails at  nova-status upgrade check with the error Placement Api endpoint not foundHere is the error http://paste.openstack.org/show/795323/14:31
ioninamrata: new installation or upgrade?14:32
namrataNew installation14:33
*** mgariepy has quit IRC14:33
ioninamrata: the idea is that in train, placement api is separate, you need to declare placement-infra_hosts14:33
ionire-run setup-hosts and install placement with os-placement-install.yml14:34
ionibefore installing nova14:34
namrataioni thanks! I got that will install the same. Thanks again14:35
ioninamrata: you're welcome14:39
*** mgariepy has joined #openstack-ansible14:47
*** Berthe01 has quit IRC14:56
*** gyee has joined #openstack-ansible15:03
*** Berthe01 has joined #openstack-ansible15:12
*** also_stingrayza has joined #openstack-ansible15:36
*** stingrayza has quit IRC15:37
*** mjwales has joined #openstack-ansible16:05
mjwalesHi, we are part way through a xenial -> bionic upgrade and are having issues migrating VMs to "new" bionic compute nodes16:05
mjwalesThe error on the compute host is "Live Migration failure: operation failed: guest CPU doesn't match specification: missing features: vmx: libvirtError: operation failed: guest CPU doesn't match specification: missing features: vmx" but the CPU arch is the same for the host we are trying to migrate the VM too16:06
mjwalesCurrent compute host: Intel(R) Xeon(R) CPU E5-2683 v4 @ 2.10GHz Host migrating to: Intel(R) Xeon(R) CPU E5-2683 v4 @ 2.10GHz16:07
*** Brace has joined #openstack-ansible16:07
*** namrata has quit IRC16:08
jamesdentonis VT-x enabled in BIOS?16:09
mjwalesWill check now. The output of /proc/cpuinfo matches on both hosts though16:12
*** miloa has quit IRC16:13
*** idlemind has joined #openstack-ansible16:13
mjwalesjamesdenton: yes enabled on the host we are migrating to16:16
kleininova-compute logs in detail on every compute node the features found by libvirt including CPU flags. you should compare that16:17
mjwalesOn the compute node nova-compute.log contains <feature name='vmx'/>16:19
*** udesale_ has quit IRC16:23
jamesdentonnoonedeadpunk Any chance we could get a new release for stein and train for os_neutron role? recent upstream commit in both stable branches that fixes ugly OVS bug16:35
ionijamesdenton: i wonder what OVS bug, care to share the bug report?16:37
jamesdentonsure - https://launchpad.net/bugs/186482216:38
openstackLaunchpad bug 1864822 in neutron "Openvswitch Agent - Connexion openvswitch DB Broken" [Medium,Fix released] - Assigned to Slawek Kaplonski (slaweq)16:38
mgariepymjwales, which tag are you using ?16:40
mjwalesrocky-em16:40
mjwales18.1.20 vends are being deployed16:40
mjwales*venvs16:40
ionijamesdenton: i have other problem with ovs, https://bugs.launchpad.net/ubuntu/+source/openvswitch/+bug/1839592 and from time to time i have to restart ovs16:42
openstackLaunchpad bug 1839592 in glibc (Ubuntu) "Open vSwitch (Version 2.9.2) goes into deadlocked state" [High,Confirmed]16:42
mgariepymjwales, i guess an offline migration would work ?16:43
jamesdentonioni we have a similar issue where the commands lock up, but hadn't really dug into it. we just have a cron job at the moment that restarts ovs and agent16:43
ionijamesdenton: yes, if an instance is rebooted, it won't get any access to the network16:44
ionijamesdenton: if i log into compute node and run ps aux, I can see a lot of crons being stuck and then i know is into a deadlock16:44
*** cshen has joined #openstack-ansible16:45
ionididn't had time to test to see if is alright to get the new glibc package from ubuntu20 on ubuntu1816:45
ionibut that is mostly a hack16:45
mgariepymjwales, if you are running ephemeral disk for your vm you need to make sure you are ''blockpull'' the eph disk on the xenial node: line 86 of https://etherpad.opendev.org/p/osa-rocky-bionic-upgrade16:45
mjwalesmgariepy: no ephemeral disk we are using ceph for shared storage to16:45
mjwalesas a test we moved a VM from the "new" host compute-1 to compute-5 then back to compute-1 which worked fine. However moving a VM that has been running on compute-5 for a while to compute-1 fails with the Libvirt error16:47
mgariepyany pkg update on these ?16:48
jamesdentonioni that bug i mentioned would usually affect us on the restart of ovs agent, but sometimes out of nowhere.16:48
ionimjwales: offline migration is out of question?16:48
ionijamesdenton: alright, i'll read the comments to see if is related to my problem, maybe it is :)16:48
mjwalesIdeally we were trying to go without downtime for running VMs16:49
jamesdentoni don't think it will fix the deadlocks, but at least help the agent recover properly in some cases. good luck16:49
mjwalesmgariepy: do you mean pkg updates on the vms or compute hosts? compute-1 has been updated to bionic and compute-5 is running xenial16:49
*** cshen has quit IRC16:50
mgariepyon the compute hosts.16:50
mjwalesYes...from 16.04 to 18.0416:50
mgariepybetween the first test and the failed one.16:51
mjwalesNope16:51
mgariepyso a vm spawned on bionic can be moved back and forth but not the other way around .. that's ... interesting.16:52
mjwalesmgariepy: both were created on xenial16:53
mjwalesThere doesn't seem to be any logic behind it16:53
mgariepysame flavor ?16:54
mgariepysame base image ?16:54
mjwalesDifferent16:54
mgariepycan you compare the vm xml of both vms ?16:55
mjwalesWill give it a shot now16:56
mgariepydid you reboot the compute host between the 2 tests?16:58
mjwalesNo, we didnt16:59
mjwalesXML dumps show that one vm has vmx in the xml and one does not hence why it moved17:00
mjwalesIs it worth giving the host a reboot?17:00
mgariepynot sure.17:01
mgariepyother vm have the vmx in the xml ?17:01
mjwalesI'll give it a shot on one of the other updated nodes and see if that fixes it17:01
mjwalesAll the other vms on the host have vmx17:03
mgariepysame flavor ? same extra-specS?17:03
mjwalesIt looks like a reboot of the upgraded nodes fixes the issue. Tested on one compute host, trying a second now17:09
mgariepydid you do a dist-upgrade or a fresh install ?17:09
mjwalesFresh install17:10
jrosserjamesdenton: releases are nearly good to go, some votes and rechecks needed https://review.opendev.org/#/q/topic:bump_osa+(status:open+OR+status:merged)17:15
jrosserwould be worth checking that pulls in the SHA you need for the ovs bug17:15
mjwalesmgariepy: a reboot did the trick17:17
mjwalesnot sure if the process we are following is at fault though17:18
mjwalesbasic process we are following is below so if you can spot anything wrong please shoud17:19
mjwalesdisable compute nodemigrate vms offclean install 18.04Run pre reqsopenstack-ansible setup-hosts.yml --limit localhost,NEW_HOST_NAMEopenstack-ansible ceph-install.ymlansible nova_all -m setup -a 'filter=ansible_local gather_subset="!all"'openstack-ansible setup-openstack.yml --limit localhost,NEW_HOST_NAME17:19
mgariepymaybe something is configured somewhere but not loaded on 18.04 for rocky.17:21
mjwalesMany hours later it seems that adding a precautionary reboot after each upgrade is well worthwhile :P17:23
mjwalesOnce we've upgraded the cluster we will make our upgrade process available. We merged things from multiple source to get something that worked for us (fingers crossed)17:23
*** idlemind has quit IRC17:24
mgariepyyou can also check what is the isseu ,maybe something is loaded into kvm and libvirt was started before or something.17:25
mgariepyfun time with upgrades17:26
*** idlemind has joined #openstack-ansible17:28
mgariepymaybe the kvm nested was not enabled when the kvm module was loaded.17:29
*** mjwales has left #openstack-ansible17:29
*** idlemind has quit IRC17:48
spateljrosser: did we made any progress on centos8 or still dealing with some road-blockers ?18:13
spatelsorry i was little busy on other projects last week18:14
*** cshen has joined #openstack-ansible18:45
*** cshen has quit IRC18:50
*** masterpe has joined #openstack-ansible18:57
masterpeCan I configure Designate together with powerDNS?18:58
*** idlemind has joined #openstack-ansible19:07
*** sshnaidm|ruck is now known as sshnaidm|afk19:12
spatelmasterpe: you have to install powerDNS yourself i don't think OSA provide that19:14
spatelmasterpe: curious why you want to do that?19:15
masterpeI do not want to install powerDNS via OSA, but I'm trying to figure out how you get desigante working together with powerdns19:17
*** skelly has joined #openstack-ansible19:31
ionimasterpe: yes, works pretty good19:45
ionimasterpe: but there is a catch, since designate-mdsns is inside the container, you have to port forward inside its port19:46
ionisince all the zones added into designate are masters and powerdns is slave19:47
ioniaxfr must work19:47
ionidepending of your infrastructure, you may need the port forward19:47
noonedeadpunkjamesdenton: yes, sure20:09
noonedeadpunkat least for stein...20:10
noonedeadpunkfor train we need to get https://review.opendev.org/#/c/738357/ passing20:10
noonedeadpunk(I hope neutron has correct sha in that bump)20:10
noonedeadpunkand we fail with swift... https://48a12864234aa2205a2e-b0513d7a5caaf512c62df58e84959553.ssl.cf2.rackcdn.com/738357/1/check/openstack-ansible-deploy-aio_ceph-ubuntu-bionic/062c824/logs/openstack/aio1_utility_container-42b04f06/utility/stestr_results.html20:12
noonedeadpunkeventually with ceph rgw I guess...20:13
noonedeadpunkBut not really sure if swift tempest test is appropriate now for rgw...20:13
noonedeadpunkbtw your vote on stein stuff may help https://review.opendev.org/#/c/738359/ )20:16
noonedeadpunkoh, by the way, did we merge everything we wanted regaridng focal?20:16
noonedeadpunkShould I place bump for ussuri for another RC?20:16
noonedeadpunkI guess we need to fix swift tempest plugin version for train as well or smth...20:20
spatelFolks, do you like my new dashboard for power usage of Racks - https://imgur.com/a/O4l0RTj20:20
jamesdentonnoonedeadpunk these are upstream Neutron commits, so it looks like we're tying to the stable/stein and stable/train branches, so this may be moot20:35
noonedeadpunk but were they backported to S and T? or fix is only for master?20:37
noonedeadpunkI'm just not sure about what exact patch we're talking..20:38
jamesdentonyes, they were backported.20:43
jamesdentonhttps://review.opendev.org/#/c/731296/ and https://review.opendev.org/#/c/731295/20:44
*** cshen has joined #openstack-ansible20:45
*** cshen has quit IRC20:50
*** sshnaidm|afk has quit IRC20:51
*** sshnaidm|afk has joined #openstack-ansible20:53
*** markvoelker has joined #openstack-ansible21:25
*** markvoelker has quit IRC21:30
*** spatel has quit IRC21:39
*** markvoelker has joined #openstack-ansible21:58
*** markvoelker has quit IRC21:58
*** luksky has quit IRC22:05
admin0is there a table in the db which stores ram and cpu of hypervisors ?22:21
admin0like its usage etc22:21
admin0nvm, found it22:30
*** cshen has joined #openstack-ansible22:45
*** tosky has quit IRC22:46
*** cshen has quit IRC22:50
*** markvoelker has joined #openstack-ansible23:05
*** markvoelker has quit IRC23:20
*** markvoelker has joined #openstack-ansible23:57

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!