Monday, 2020-01-27

*** DanyC has joined #openstack-ansible00:41
*** DanyC has quit IRC00:46
*** cshen has joined #openstack-ansible01:01
*** cshen has quit IRC01:06
*** DanyC has joined #openstack-ansible01:20
*** DanyC has quit IRC01:26
*** openstackgerrit has quit IRC02:04
*** shyamb has joined #openstack-ansible02:18
*** hwoarang has quit IRC02:27
*** hwoarang has joined #openstack-ansible02:27
*** hwoarang has quit IRC02:37
*** hwoarang has joined #openstack-ansible02:39
*** shyam89 has joined #openstack-ansible03:05
*** shyamb has quit IRC03:07
*** shyamb has joined #openstack-ansible03:11
*** shyam89 has quit IRC03:12
*** shyam89 has joined #openstack-ansible03:13
*** shyamb has quit IRC03:16
*** DanyC has joined #openstack-ansible03:21
*** DanyC has quit IRC03:26
*** DanyC has joined #openstack-ansible03:29
*** DanyC has quit IRC03:35
*** jhesketh has quit IRC03:56
*** DanyC has joined #openstack-ansible04:07
*** shyam89 has quit IRC04:12
*** DanyC has quit IRC04:12
*** udesale has joined #openstack-ansible04:25
*** shyamb has joined #openstack-ansible04:27
*** raukadah is now known as chkumar|rover04:45
*** shyamb has quit IRC05:43
*** shyamb has joined #openstack-ansible05:46
*** DanyC has joined #openstack-ansible06:08
*** DanyC has quit IRC06:13
*** DanyC has joined #openstack-ansible06:41
*** DanyC has quit IRC06:46
*** shyamb has quit IRC07:03
*** hwoarang has quit IRC07:03
*** hwoarang has joined #openstack-ansible07:04
*** shyamb has joined #openstack-ansible07:04
*** elenalindq has joined #openstack-ansible07:26
*** luksky has joined #openstack-ansible07:29
*** openstackgerrit has joined #openstack-ansible07:55
openstackgerritMerged openstack/openstack-ansible master: Imported Translations from Zanata  https://review.opendev.org/70395007:55
*** tosky has joined #openstack-ansible07:56
*** cshen has joined #openstack-ansible08:06
elenalindqGood morning/day/evening, folks ! A question about manage.py that replaces inventory-manage.py in Train.  On a Train OpenStack deployment (upgraded from Stein) running manage.py from  osa_toolkit directory with "python manage.py " errors with "ImportError: No module named osa_toolkit" .  Am I missing something or is this broken for you as well?08:14
jrosserelenalindq: I’m not sure anything has changed there, you probably want inventory_manage.py from the scripts/ directory?08:16
*** cjloader has quit IRC08:16
*** prometheanfire has quit IRC08:16
*** prometheanfire has joined #openstack-ansible08:17
elenalindqjrosser:  inventory_manage.py fails and in the script I read "# NOTE(nrb/palendae): The contents of this file were moved # to manage.py in order to facilitate importing of the python code# This file remains for backwards compatibility "08:17
*** rpittau|afk is now known as rpittau08:18
jrosserelenalindq: hmm, you said first that running manage.py fails, but also that inventory_manage.py fails as well?08:20
elenalindqYes.08:20
elenalindqinventory-manage.py from scripts fails with "TypeError: '<' not supported between instances of 'str' and 'NoneType'08:21
elenalindq"08:21
jrosserI don’t believe there has been a deliberate change of behaviour here for the train release08:21
*** shyamb has quit IRC08:21
jrosserelenalindq: I have a train deployment so can take a look at this08:22
elenalindqjrosser Thank you very much!08:22
*** pcaruana has joined #openstack-ansible08:25
jrosserthe “functional” parts of inventory_manage.py were moved out into the osa_toolkit library 3 years ago08:25
jrosserimho this looks like a refactoring rather than an attempt to move the script08:26
*** shyamb has joined #openstack-ansible08:26
*** jbadiapa has joined #openstack-ansible08:27
*** ebbex has joined #openstack-ansible08:35
*** DanyC has joined #openstack-ansible08:49
*** DanyC has quit IRC08:50
*** DanyC has joined #openstack-ansible08:59
*** DanyC has quit IRC09:00
*** vesper11 has quit IRC09:01
*** DanyC has joined #openstack-ansible09:01
jrosserelenalindq: it seems to work here http://paste.openstack.org/show/788816/09:02
elenalindqThank you for checking jrosser.09:03
jrosserelenalindq: is this a new deployment?09:04
elenalindqI deployed Stein and then upgrade to Train.09:04
elenalindqthe inventory script did work in Stein, but it does not in Train so I assumed it's because it is deprecated.09:04
*** vesper11 has joined #openstack-ansible09:05
elenalindqOr that is what I understood from the note in the script09:05
jrosseri think that means that the code has been refactored to allow the functions in manage.py to be used in other scripts09:05
jrosserosa_toolkit is a library really09:05
jrosserone difference is that for train the /opt/ansible-runtime venv with be python3 rather than python2, which may expose an inventory related problem differently on Train than Stein09:07
jrosserfrom the error "TypeError: '<' not supported between instances of 'str' and 'NoneType' I would guess that there is some empty variable which evaulates to None09:09
*** shyamb has quit IRC09:19
arxcruzjrosser: https://review.opendev.org/690377 it's an automated test, but seems to be broken, can i just upload a new patchset to fix it ?09:19
jrosserarxcruz: see the comment on line 7 of that patch09:23
jrosserif you make the necessary change in openstack-ansible-tests it'll get proposed to all the places that need it09:23
arxcruzjrosser: :O i didn't even care to look the comments09:29
arxcruzthanks09:29
jrossernoonedeadpunk: would be good to talk about the os_tempest fixes when you're around, i'm not sure what the right approach is for ubuntu distro installs09:34
*** shyamb has joined #openstack-ansible09:54
jrosserelenalindq: if you get really stuck you can make a test setup with a VM and openstack-ansible and copy the /etc/openstack_deploy over then poke into whats happening with the inventory10:05
jrosseri'd expect there is an empty group or an undefined host resulting in that "none"10:06
*** shyamb has quit IRC10:06
elenalindqjrosser:  Thank you so much for your suggestion, will try.10:11
jrosserthere is also an inventory history in /etc/openstack_deploy/backup_openstack_inventory.tar which you may be able to bisect something thats broken this10:12
*** sshnaidm is now known as sshnaidm|afk10:27
*** aedc has quit IRC10:56
*** aedc has joined #openstack-ansible10:57
*** udesale has quit IRC10:58
*** cshen has quit IRC11:06
*** cshen has joined #openstack-ansible11:15
*** rpittau is now known as rpittau|bbl11:21
*** sshnaidm|afk is now known as sshnaidm11:34
*** maharg101 has joined #openstack-ansible11:44
maharg101when using openvswitch, do the controller hosts need ovs bridges, or just the compute nodes ?11:45
*** cshen has quit IRC11:47
maharg101in a containerised install that is11:47
*** luksky has quit IRC11:55
jrossermaharg101: if the controller is running the neutron L3 functions then i'd expect so12:01
jrosser(i'm kinda presuming that as its linuxbridge here)12:02
*** cshen has joined #openstack-ansible12:09
noonedeadpunkjrosser: o/ sure, can talk12:12
jrossernoonedeadpunk: o/ hello12:12
jrosserthis is where i am a bit stuck https://review.opendev.org/#/c/703979/12:13
jrosserthe ubuntu distro job fails becasue the distro supplied python packages (via apt) do not satisfy the constraints for tempest and it's dependancies12:14
jrosserwe've fixed the issue with neutron-lib but now it blows up on keystone-tempest-plugin like this https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_663/703979/2/check/openstack-ansible-deploy-aio_distro_metal-ubuntu-bionic/663c03c/logs/ara-report/result/54f77f42-ab74-49b3-b37b-5be69f72d637/12:16
*** cshen has quit IRC12:17
*** cshen has joined #openstack-ansible12:20
openstackgerritMerged openstack/openstack-ansible-os_neutron stable/stein: Update vpnaas rootwrap filters  https://review.opendev.org/70231312:25
*** cshen has quit IRC12:25
*** cshen has joined #openstack-ansible12:25
*** rh-jelabarre has joined #openstack-ansible12:28
noonedeadpunkjrosser: oh, yeah, I saw that....12:30
noonedeadpunkI was going to create a sandbox for closer look12:31
jrosseri've fixed stein with direct patches in a similar manner and I think thats been OK becasue the u-c hadnt got ahead of the distro python versions12:31
jrosserso this is a wider issue for distro installs when UCA doesnt seem to supply up to date python packages to satisfy the constraints12:32
jrossernoonedeadpunk: if you could take a look that would be great - i'm wondering if we should talk to someone from the ubuntu/canonical side to see what they expect to happen12:34
noonedeadpunkbecause afaik we were installing keystone-tempest-plugin not as a distro package, but clone and install from source for the system python (instead of the venv)12:34
noonedeadpunkbut yes, what you say makes sense...12:35
jrosserthis is odd12:35
jrosser":stderr: ERROR: nova 20.0.1 has requirement SQLAlchemy>=1.2.19, but you'll have sqlalchemy 1.2.18 which is incompatible."12:35
jrosseri wonder why a requirement from nova is in play there12:35
noonedeadpunkyeah, completely odd.....12:36
*** DanyC has quit IRC12:36
jrosserthe thing with source vs. distro in this case, if there was an apt package for keystone-temepst-plugin then i'd assume that someone would have to package the required versions of the python modules that needed12:38
noonedeadpunkyes, but for ubuntu distro vs source is kinda install for system python vs install in venv12:39
noonedeadpunkWhich shouldn't make difference, except we're trying to install for py2....12:40
jrosseroh, are we?12:41
*** cshen has quit IRC12:42
noonedeadpunknot sure. We're omiting venv, but do not provide executable isntead, so we're leaving ansible to decide I guess12:42
noonedeadpunkoh, no, we provide virtualenv12:43
noonedeadpunk* executable12:43
*** ansmith_ has quit IRC12:43
jrosseryes, thats one of the things i needed to fix12:43
*** DanyC has joined #openstack-ansible12:44
jrossernoonedeadpunk: we should maybe try ussuri UCA https://github.com/openstack/openstack-ansible-openstack_hosts/blob/master/vars/ubuntu-18.04.yml#L9912:48
jrosserwhich seems to be existing to some extent http://ubuntu-cloud.archive.canonical.com/ubuntu/dists/bionic-updates/ussuri/12:49
noonedeadpunkoh, indeed12:49
noonedeadpunkbut still not really answers why we can't install pip pacgaes of version we need12:49
jrosserit doesnt, and i also didnt find the versions of the python packages we seem to need in ussuri uca either12:50
jrosseri think that the pip packages cant be installed because pip can't uninstall the pyyaml that came from distro package12:51
jrossersame for 'wrapt'12:52
*** cshen has joined #openstack-ansible12:53
noonedeadpunkthat is weird since like you can update system python packages which are isntalled with distro package via pip... Like I did that tons of times..12:54
noonedeadpunkBut probably these packages are not in requirements12:54
noonedeadpunkso pip is not asked to update them12:54
jrosserERROR: Cannot uninstall 'wrapt'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.12:54
*** rpittau|bbl is now known as rpittau12:55
jrosseri think there is a flag you can set on pip to make it just carry on regardless12:55
jrosserand then you get the new files mixed up with the ones it couldnt remove12:55
noonedeadpunkbut yes - we need to switch uca to ussuri - that's great call12:57
noonedeadpunkthen it will probably have more fresh istro python packages which will satisfy with requirements12:58
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-openstack_hosts master: Switch to Ussuri UCA repo  https://review.opendev.org/70431712:59
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-os_tempest master: Use contraints for tempest plugins  https://review.opendev.org/70397913:00
jrosser^ lets try it and see13:01
noonedeadpunk++13:01
*** aedc has quit IRC13:13
*** aedc has joined #openstack-ansible13:14
*** aedc has quit IRC13:17
*** aedc has joined #openstack-ansible13:21
*** ansmith_ has joined #openstack-ansible13:37
openstackgerritMerged openstack/openstack-ansible-ceph_client stable/train: Use print function  https://review.opendev.org/70311413:43
*** aedc has quit IRC13:52
*** aedc has joined #openstack-ansible13:57
*** udesale has joined #openstack-ansible14:03
*** luksky has joined #openstack-ansible14:16
*** lkoranda has joined #openstack-ansible14:28
openstackgerritMerged openstack/openstack-ansible master: Allow ceph metal CI deployments  https://review.opendev.org/70285314:32
jamesdentonAny chance we could spin a new stable/stein release soon? There's a bug that's been fixed in upstream stable/stein neutron that addresses issues with multiple external provider networks.14:55
*** cjloader has joined #openstack-ansible15:08
noonedeadpunkjamesdenton: where it has been fixed? like it was osa or neutron one?15:31
jamesdentonneutron neutron15:32
*** openstacking_123 has joined #openstack-ansible15:32
noonedeadpunkmaybe you have some sha handy? :p15:33
jamesdentonone sec15:33
openstacking_123Hope all is well. I was wonder in anyone had a best practice for swapping out Ubuntu 16 infra nodes with Ubuntu18 nodes pre upgrade ?15:33
noonedeadpunkactually there's a bump https://review.opendev.org/#/c/704251/1/playbooks/defaults/repo_packages/openstack_services.yml  just want to verify if it's included15:33
jamesdentonnoonedeadpunk that's the one15:34
noonedeadpunkopenstacking_123: yep, I did such upgrade on rocky - there's an etherpad with notes on the process which helped me https://etherpad.openstack.org/p/osa-rocky-bionic-upgrade15:35
jamesdentonyou're on top of it. thank you!15:35
openstacking_123noonedeadpunk fantastic thank you!15:36
noonedeadpunkjamesdenton: oh, ok. So then I'll probably need to break the rules a bit and release based on that sha without waiting 2 weeks for it's testing. What do you think mnaser?15:37
openstacking_123noonedeadpunk Whe you say 'do the host upgrade' are you referring to the Ubuntu 16 to 18 in place upgrade?15:38
mnasernoonedeadpunk, jamesdenton:  i think we could proceed.  we have enough ci to trust the commits and unless something crazy happens15:39
mnaserwe can simply make another patch release15:39
jamesdenton*thumbs up*15:39
mnaserhmm15:45
mnaseri see an issue in our upgrading tooling with placement extraction15:45
mnaserwe don't have a way to handle the servie running in two sides so when you run haproxy playbook15:46
mnaseryou pretty much dont have placement until you deploy placement15:46
mnasersee https://github.com/openstack/openstack-ansible/blob/master/inventory/group_vars/haproxy/haproxy.yml#L161-L17215:47
*** theintern_ has joined #openstack-ansible15:50
jrosserwe need to revert this out of stein before releaseing please https://github.com/openstack/openstack-ansible-os_tempest/blob/stable/stein/defaults/main.yml#L4815:54
* jrosser py3 tempest bonfire fallout15:54
jrosserjamesdenton: ^15:54
jamesdentonlooking15:54
jamesdentonthe whole commit?15:55
jrosserjust a mo let me do it15:57
jamesdentonkk15:57
jrosserthe state of the bonfire is here https://ethercalc.openstack.org/y2ekxxq1dd1t15:57
mnaserjrosser: do we need to backport rabbitmq 3.8.2 update too?16:01
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-os_tempest stable/stein: Revert u-c url to value from openstack_services.yml  https://review.opendev.org/70434316:01
jrosserjamesdenton: i left a note on here about what needs doing for stein https://review.opendev.org/#/c/704251/116:02
jrossermnaser: well it was a bit speculative really to bump this early rather than find out very late if it worked or not16:03
jamesdentonperfect16:03
mnaserjrosser: right but i was wondering if you had some magical rabbitmq stuff that got fixed by going to 3.8.2 :p16:03
jrosseri wish :)16:03
mnaserso if that had benefits of going back to stable/trai16:03
mnaserhah16:03
jrosserby all means have a go16:03
jrosserbut i still think its 50/50 mess between rabbit and something in oslo that doesnt reconnect16:04
jrossermnaser: speaking of rabbitmq i was a bit uncertain about this one https://review.opendev.org/#/c/703577/16:05
jrossercalling out that specifically for centos without explaining why looked wierd16:05
*** theintern_ has quit IRC16:07
*** lkoranda has quit IRC16:08
*** theintern_ has joined #openstack-ansible16:19
*** udesale has quit IRC16:19
*** gyee has joined #openstack-ansible16:29
openstackgerritErik Berg proposed openstack/openstack-ansible stable/rocky: avoid watchdog 0.10.0 and possible future releases  https://review.opendev.org/70435216:29
*** cshen has quit IRC16:34
*** maharg101 has quit IRC16:35
*** maharg101 has joined #openstack-ansible16:35
*** theintern_ has quit IRC16:37
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible stable/rocky: Bump SHAs for stable/rocky  https://review.opendev.org/70425416:45
*** theintern_ has joined #openstack-ansible16:49
*** chkumar|rover is now known as raukadah16:54
*** hwoarang has quit IRC17:08
*** hwoarang has joined #openstack-ansible17:09
*** gregwork has joined #openstack-ansible17:15
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-os_tempest stable/stein: Adapt tempestconf job to use integrated repo  https://review.opendev.org/70436817:21
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-os_tempest stable/stein: Revert u-c url to value from openstack_services.yml  https://review.opendev.org/70434317:21
*** theintern_ has quit IRC17:23
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-os_tempest stable/train: Use correct pip executable for mixed distro/source installs  https://review.opendev.org/70436917:25
*** luksky has quit IRC17:27
openstackgerritJonathan Rosser proposed openstack/openstack-ansible stable/train: Install appropriate tempest releases on stable branches  https://review.opendev.org/70357217:31
*** rpittau is now known as rpittau|afk18:18
*** DanyC has quit IRC18:20
jrossernoonedeadpunk: what do you think about “fixing up” the distro python with ansible/pip for the failing bionic distro job? 1) remove troublesome python module apt packages 2) replace using pip + u-c18:22
jrosserI’m not sure how much we want to be messing with the distro python because there are lots of ways this might go wrong18:22
jrosserparticularly a later installed package with a depends on the troublesome module18:24
jrosseralternatively, we just give up with the mixed install and tempest is always from source on Ubuntu18:24
noonedeadpunkI like the last option more tbh18:26
noonedeadpunklike I still don't fully ubderstood what goes wrong with mixed install (had no time for checking this) but since ubuntu do not pack tempest plugins in packages - I think they suggest doing source install all the time...18:32
*** sshnaidm is now known as sshnaidm|afk18:35
jrossernoonedeadpunk: ok I will look if there is a clean way to switch to always source installs on ubuntu18:40
*** openstacking_123 has quit IRC18:42
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible master: [WIP] Add manila tempest testing  https://review.opendev.org/67465318:48
*** mgariepy has quit IRC18:52
*** mgariepy has joined #openstack-ansible18:57
*** ansmith has joined #openstack-ansible19:05
jrosserraukadah: do you know what’s going on here? https://zuul.opendev.org/t/openstack/build/8c5077b592874ab292b86d3f65c62f5d19:06
jrosserI’ll recheck in case it’s transient19:06
*** ansmith_ has quit IRC19:06
*** gregwork has quit IRC19:25
*** KeithMnemonic has joined #openstack-ansible19:26
*** admin0 has quit IRC19:44
*** admin0 has joined #openstack-ansible19:46
*** ansmith_ has joined #openstack-ansible19:46
*** luksky has joined #openstack-ansible19:47
*** ansmith has quit IRC19:48
*** kplant has joined #openstack-ansible20:10
*** mrda has quit IRC20:21
*** elenalindq has quit IRC20:25
*** jbadiapa has quit IRC20:25
*** openstackstatus has joined #openstack-ansible20:27
*** ChanServ sets mode: +v openstackstatus20:27
*** gyee has quit IRC20:27
*** gyee has joined #openstack-ansible20:28
*** mrda has joined #openstack-ansible20:28
*** DanyC has joined #openstack-ansible20:30
jrossercould i get some reviews here https://review.opendev.org/#/c/704369/20:32
*** kplant has quit IRC20:37
jrosserraukadah: i think the error i posted before is this https://github.com/ansible/ansible/issues/6449520:38
jrosseri notice that job runs under ansible 2.9.4 which we have no coverage for at all on the OSA side20:41
*** lbragstad has joined #openstack-ansible20:41
*** elenalindq has joined #openstack-ansible20:41
jrosserthere is an outstanding bug/PR for the ansible rabbitmq policy module thats not merged so we are stuck on 2.8.x until that is fixed20:42
*** lbragsta_ has joined #openstack-ansible20:47
*** lbragstad has quit IRC20:50
*** lbragsta_ has quit IRC20:50
*** mgariepy has quit IRC20:55
*** ansmith_ has quit IRC21:42
*** elenalindq has quit IRC21:56
*** aedc has quit IRC22:17
*** aedc has joined #openstack-ansible22:19
*** ahosam has joined #openstack-ansible22:21
*** rh-jelabarre has quit IRC22:42
*** ahosam has quit IRC22:49
*** ahosam has joined #openstack-ansible22:49
*** tosky has quit IRC23:45

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!