Thursday, 2020-05-21

*** dpaclt has quit IRC00:17
*** watersj has quit IRC01:33
*** schwicht has quit IRC01:48
*** schwicht has joined #openstack-ansible01:51
*** spatel has joined #openstack-ansible01:56
*** tinwood has quit IRC02:12
*** KeithMnemonic has quit IRC02:14
*** tinwood has joined #openstack-ansible02:14
*** djhankb has quit IRC02:20
*** stingrayza has joined #openstack-ansible02:34
*** also_stingrayza has quit IRC02:38
*** djhankb has joined #openstack-ansible02:47
*** redrobot has quit IRC02:49
*** nurdie has joined #openstack-ansible02:57
*** itandops has quit IRC03:08
*** rh-jelabarre has quit IRC04:07
*** evrardjp has quit IRC04:33
*** evrardjp has joined #openstack-ansible04:33
*** spatel has quit IRC04:54
*** cshen has joined #openstack-ansible05:00
*** cshen has quit IRC05:05
*** nurdie has quit IRC05:20
*** udesale has joined #openstack-ansible05:29
*** raukadah is now known as chandankumar05:32
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible master: [WIP] Make upgrade jobs get role relevant versions  https://review.opendev.org/71084905:42
*** cshen has joined #openstack-ansible05:45
*** schwicht has quit IRC06:06
openstackgerritJonathan Rosser proposed openstack/ansible-role-python_venv_build stable/stein: Verify version number for virtualenv>=20.0.0  https://review.opendev.org/72984406:13
*** udesale has quit IRC06:31
*** udesale has joined #openstack-ansible06:33
*** udesale has quit IRC06:38
*** udesale has joined #openstack-ansible06:38
*** dpaclt has joined #openstack-ansible07:07
*** threestrands has quit IRC07:08
dpacltHi All , Just wanted to reconfirm whether to define storage_hosts as controller/infrastructure_hosts if my backend is Ceph .07:13
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-tests master: Add Ubuntu Focal non voting check jobs  https://review.opendev.org/72985407:14
jannodpaclt: do you want to deploy the ceph backend through openstack-ansible or does the cluster already exist?07:15
dpacltI want to deploy ceph backend through openstack-ansible ,dont have another cluster07:15
jannowe deployed our ceph cluster with openstack-ansible we have groups ceph-mon_hosts and ceph-osd_hosts07:16
jannowhere ceph-mon_hosts are the infrastructure hosts and ceph-osd_hosts are the actual storage hosts07:17
dpacltWe have defined ceph-mon and ceph -osd hosts  as you said ...07:17
dpaclt.Can see storage hosts defined as  storage_hosts: *infrastructure_hosts in https://docs.openstack.org/openstack-ansible/latest/user/ceph/full-deploy.html07:18
jannoah, yes, you are right. we have storage_hosts defined as well07:19
jannostorage_hosts: *infrastructure_hosts07:20
*** NewJorg has quit IRC07:21
dpacltoh ok thanks for the confirmation.07:21
dpacltI was intending to use native ceph as backend for manila ... so manila_data_hosts and manila_infra_hosts also should be *infrastructure_hosts right ?07:22
*** NewJorg has joined #openstack-ansible07:23
dpacltSo was wondering whether manila_data_hosts is infrastructure_hosts or ceph -osd hosts ?07:30
*** tosky has joined #openstack-ansible07:31
noonedeadpunkI think yes... But had only metal deployment on hands so it didn't matter in my case...07:34
noonedeadpunkjrosser: ok, seems that we need to disable ci anyway to merge https://review.opendev.org/#/c/728434/307:34
jrosserdpaclt: storage_hosts may be referring to API like cinder, not just the actual storage itself07:49
jrossernoonedeadpunk: o/ morning, let me look at the galera thing07:49
dpacltjrosser ohh ok then it can be infrastructure node i guess07:49
noonedeadpunkYeah, I think it all depends if driver_handles_share_servers is true or false07:50
dpacltmine was   driver_handles_share_servers is false07:51
jrossernoonedeadpunk: so yes this is circular isn't it? becasue the galera_server role uses the integrated repo in it's tests, which itself needs updates for the changes to galera_server07:53
noonedeadpunkyeah. and we're failing because of utility07:54
jrosserand theres no point adding depends-on becasue it just makes it worse07:54
noonedeadpunkSo thinking about just dropping lxc jobs for now07:54
*** cshen has quit IRC07:55
jrosserand the metal jobs succeed becasue galera_server has previously run against the host which does what utility would have needed07:55
jrosseris that right?07:56
noonedeadpunkYEah, I think that's what happen07:57
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible master: Use galera server role to install galera client  https://review.opendev.org/72843407:57
jrosserok so i had a few more things we should check, the patch to galera_server needs checking that everything (that we care about) from the galera_client role defaults can still be overriden07:58
jrosserthat the vars from galera_server that actually apply to both maybe we change the name to make them galera_<foo> rather than galera_server_<foo>07:58
jrosserany role tests from client to move across, (the server ones look pretty good though)07:59
jrosserrelease note / any docs update07:59
jrosserthat first one about checking anything we accidentally drop from galera_client would be great to get second eyes on08:00
*** cshen has joined #openstack-ansible08:02
dpacltjrosser should we define manila_shares under  group bind  in br-storage08:06
jrosserdpaclt: i dont know - i've never deployed manila myself08:08
jrosserdoes the documentation for the manila ansible role help?08:09
dpacltTo be frank no.. i didnt get a proper documentation of manila08:10
jrosserit's also quite complex as irrc there are at least two different ways you can use it08:13
jrossernoonedeadpunk: can you review this as we have a few things still to merge into os_neutron that are now unblocked https://review.opendev.org/#/c/728804/208:14
* jrosser removes vote from own patch :/08:15
noonedeadpunkMaybe you can vote as it's jamesdenton who patched the last time?08:16
noonedeadpunkI feel a bit uncomfortable is such cases tbh08:17
jrosseri'll get gshippey to take a look08:17
*** mgariepy has quit IRC08:20
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-openstack_hosts master: Fix _package_repos for ubuntu focal  https://review.opendev.org/72986708:22
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-tests master: Add Ubuntu Focal non voting check jobs  https://review.opendev.org/72985408:23
*** NewJorg has quit IRC08:23
jrossernoonedeadpunk: i was also thinking about what to do for depends-on patches to the ansible openstack collection, and it's kinda ugly08:24
jrosserbecause if we set up a job to test a review in the collection with a depends-on, we are saying it's ok to merge an OSA patch when the collection patch merges08:25
jrosserand you would then expect that OSA works outside CI, maybe requiring a fix in the collection to have landed08:26
jrosserbut thats not the case if we consume the collection from the published version in ansible galaxy, we'd be always on a released version08:27
noonedeadpunkyeah, I see what you mean. Like not being possible to install collaction just from git is smth I really miss08:30
*** NewJorg has joined #openstack-ansible08:30
jrosseri might look at using the collection from git, not using ansible-galaxy to install08:31
jrosserthen treat it the same as one of our roles08:31
*** aedc has joined #openstack-ansible08:31
*** aedc has quit IRC08:32
jrosserthats not documented becasue the assumption is you use galaxy but the file tree it drops down is extremely similar to the original repo08:32
noonedeadpunkI was trying to look isntalling from git but kinda didn't find any docs allowing that to do08:43
noonedeadpunkbut yes, using git would kinda resolve she issue and we can kinda just bump to sha and update it with release script08:44
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-lxc_hosts master: Use btrfs-progs instead of btrfs-tools  https://review.opendev.org/72987608:45
*** cshen has quit IRC08:46
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-tests master: Add Ubuntu Focal non voting check jobs  https://review.opendev.org/72985408:46
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible master: Return lxc jobs  https://review.opendev.org/72987808:50
*** mgariepy has joined #openstack-ansible09:04
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-lxc_hosts master: Do not install python2 packages on systems that should be python3 only  https://review.opendev.org/72988309:11
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-tests master: Add Ubuntu Focal non voting check jobs  https://review.opendev.org/72985409:12
*** cshen has joined #openstack-ansible09:14
*** cshen has quit IRC09:19
openstackgerritDuncan Martin Walker proposed openstack/openstack-ansible-ops master: Ensure that the beat setup role runs when elk_beat_setup=true  https://review.opendev.org/72965409:20
noonedeadpunkjrosser: actually I think we can really use just galaxy version. It's anyway more flexible than we do have now09:20
noonedeadpunkJust didn't get about one sed here https://review.opendev.org/#/c/718431/309:21
jrosseryeah, looking at that it can be simplified09:27
jrosseri think i was probably thinking about what happens if you have a deploy host managing more than one deployment, which isnt something i ever do09:28
*** gshippey has joined #openstack-ansible09:30
noonedeadpunkbut like we have deloyment host for several deployments, but they do share osa version09:31
noonedeadpunkOtherwise all we also need different venvs with different ansible versions wich is not the case at the moment09:32
noonedeadpunkand since you have the same ansible version, I think it's ok to have same collections version as well...09:32
*** itandops has joined #openstack-ansible09:33
*** arxcruz is now known as arxcruz|off09:33
noonedeadpunkbtw yes, manila failed for centos7 ceph deployment09:35
noonedeadpunkbecause of that pyaml thing09:36
noonedeadpunktbh, I'd drop centos 7 at that moment if only we had centos 8 implemented09:36
jrossernoonedeadpunk: yes centos7 has been on my mind also to drop now09:53
jrosserbut i worry that if we make big patches for that, backporting centos-8 support is going to be very bad09:54
noonedeadpunkWe can actually maybe ignore octopus issue for centos-7 then?09:54
jrosserfor V we have no choice but to drop centos-7 i think09:54
jrosserit's just a question of exactly when we do it09:55
noonedeadpunkOh, you're right, for U we may keep it09:55
jrosserfor U we still seem to have it working by some miracle09:55
jrosseronly distro jobs are going to be really difficult for U, like the magnum issue09:55
noonedeadpunkdespite packages are for T09:56
jrosseri guess from a python pov we may just be building more from source during the wheel build09:56
jrosserfor the source based installs i'm not sure that U distro packages are an issue at all09:57
noonedeadpunkyeah, for source based they do not matter I think09:57
jrosserthe thing that will ultimately break centos-7 completely is a not-new-enough libvirt, something really fundamental09:58
jrossersomething like that09:58
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible master: Update ceph release to octopus  https://review.opendev.org/72933110:14
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_neutron master: Cleanup after repo_build and pip_install retirement  https://review.opendev.org/72740110:24
noonedeadpunkjrosser: so, should be create branches out of https://review.opendev.org/#/c/729258/ or we want smth to bump here?10:27
*** mgariepy has quit IRC10:30
*** dpaclt has quit IRC10:34
*** itandops has quit IRC10:34
openstackgerritMerged openstack/openstack-ansible-os_neutron master: networking-ovn is now integrated into neutron  https://review.opendev.org/72880410:51
openstackgerritMerged openstack/ansible-role-uwsgi master: Do not use bare variables in ansible conditionals  https://review.opendev.org/72725810:51
noonedeadpunkchandankumar: hey! have a question regarding rdo repos. Trying to add https://trunk.rdoproject.org/centos8-master/current-passed-ci/  as repo (used common one for centos 710:54
noonedeadpunkbut get "Failed to download metadata for repo 'rdo-current-passed-ci'"10:54
chandankumarnoonedeadpunk, is it coming everywhere? It might a network glitch11:01
chandankumarcan you recheck if not then I will ask jpena to take a look11:02
noonedeadpunkchandankumar: I think it's probably the way we configure it but not sure11:02
noonedeadpunkAs repo looks like having some new fromat11:02
noonedeadpunkwhile we stil define it in /etc/yum.repos.d/rdo.repo11:02
noonedeadpunkAlong with https://trunk.rdoproject.org/centos8-master/deps/latest/ wich works11:03
noonedeadpunkAs eventually link opens and available, but I think that dnf expects to see there metadata which is not just there11:03
chandankumarnoonedeadpunk, can you join #rdo channel11:05
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-tests master: Add Ubuntu Focal non voting check jobs  https://review.opendev.org/72985411:12
noonedeadpunkjrosser: do you know where VERSION_ID comes from?11:14
jrosserwhere is that?11:14
noonedeadpunkhttps://review.opendev.org/#/c/728259/14/test-log-collect.sh11:14
noonedeadpunkand actually throughout our scripts11:14
noonedeadpunkBut the thing is that all of them run in zuul run11:14
jrosseroh yes11:14
jrosserit's in the scripts library i think11:15
noonedeadpunkWHile test-log-collect is in post11:15
noonedeadpunkwhich means it's out of the context11:15
noonedeadpunk(probably)11:15
noonedeadpunkJust really dont see it's definition anywhere...11:16
noonedeadpunkLike start thinging it may come from zuul or whatever11:16
jrosseroh hmm the tests repo11:17
jrosseruh-oh11:18
noonedeadpunkoh11:18
noonedeadpunkit's just . /etc/os-release lol11:19
jrosserVERSION_ID comes from tox i think11:20
noonedeadpunkno, just sourcing /etc/os-release11:20
jrosserdo you have a link to that?11:21
jrosseroh found it11:21
noonedeadpunksure11:21
noonedeadpunkhttps://opendev.org/openstack/openstack-ansible-tests/src/branch/master/test-ansible-functional.sh#L9111:21
noonedeadpunklike too easy lol11:21
jrosserhttps://zuul.opendev.org/t/openstack/build/d8af419677454d7b949e95bcb4916b18/log/job-output.txt#1429-143811:22
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-tests master: Deploy modern ara for py3 deployments  https://review.opendev.org/72825911:22
jrosserso we need that in the log collect script11:22
noonedeadpunkyeah, already patched11:22
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-lxc_hosts master: Add legacy config key mappings for lxc version 4  https://review.opendev.org/72991911:24
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-tests master: Add Ubuntu Focal non voting check jobs  https://review.opendev.org/72985411:25
jrosserit will be good to drop centos -> drop lxc2 -> big simplification of lxc roles11:25
*** watersj has joined #openstack-ansible11:26
watersjwhat do you suggest when deploying, checkout specific branch stable/[version] or run from head?11:27
jrosserwatersj: on the stable branches we make tagged released about every two weeks, those are your most stable points and provide well defined places to do minor version upgrades between11:28
jrosserstable/<branch> is the place where patches land before the next two-weekly tag on that branch so you might use that to pull in a backported bugfix faster than we have released a tag11:29
jrossermaster is where the development for the next (Victoria) release cycle happens, so you would use that to contribute code or run a bleeding-edge lab, but not for production11:29
watersjjrosser, ty just wanted make sure11:31
jrosserno worries, as usual the answer is "it depends" :)11:31
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-openstack_hosts master: CentOS 8 support  https://review.opendev.org/70945511:34
*** mgariepy has joined #openstack-ansible11:55
Bracemgariepy: 210 vms, 850 networks, 440 routers (sorry for the lag in responding)12:00
mgariepyBrace, no worries.12:04
mgariepyare all of them backed by ceph volumes ?12:04
mgariepyor you have ephemeral stuff as well?12:05
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-tests master: Add Ubuntu Focal non voting check jobs  https://review.opendev.org/72985412:07
*** dpaclt has joined #openstack-ansible12:08
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-lxc_hosts master: Add legacy config key mappings for lxc version 4  https://review.opendev.org/72991912:12
*** schwicht has joined #openstack-ansible12:12
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-lxc_hosts master: Add vars files and container prep script for ubuntu focal  https://review.opendev.org/72993712:15
*** rh-jelabarre has joined #openstack-ansible12:16
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-tests master: Add Ubuntu Focal non voting check jobs  https://review.opendev.org/72985412:16
*** cshen has joined #openstack-ansible12:33
noonedeadpunkjrosser: https://review.opendev.org/#/c/728259/ ara works12:37
noonedeadpunkI think we can maerge it now12:38
*** cshen has quit IRC12:38
noonedeadpunkthen we will unblock functional tests12:42
Braceit's all ceph12:47
Bracemgariepy: I don't think we have any ephemeral stuff12:54
mgariepyok12:55
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-galera_server master: Combine galera client role into galera_server  https://review.opendev.org/72590312:55
jrossernoonedeadpunk: ara looks good, approved12:55
openstackgerritamolkahat proposed openstack/openstack-ansible-os_tempest master: Added tempest ironic resources setup.  https://review.opendev.org/72070513:00
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-tests master: Add Ubuntu Focal non voting check jobs  https://review.opendev.org/72985413:01
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-tests master: Add Ubuntu Focal non voting check jobs  https://review.opendev.org/72985413:02
openstackgerritMerged openstack/openstack-ansible-os_sahara master: Update docstheme for style  https://review.opendev.org/72970613:04
*** rh-jelabarre has quit IRC13:11
jrossernoonedeadpunk: there is another os_neutron patch from mnaser here https://review.opendev.org/#/c/708665/ - is that one important?13:12
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_neutron master: config: add region_name to nova  https://review.opendev.org/70866513:13
noonedeadpunk dunno but jsut fixed this nit13:14
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-os_neutron stable/stein: Add extra rootwrap filter  https://review.opendev.org/70477813:14
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-os_nova stable/stein: pre-install: use path and drop dest  https://review.opendev.org/70907413:15
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-os_mistral stable/stein: Add meta to mistral  https://review.opendev.org/69698113:16
dpacltHi all ,I am getting (32, 'EPIPE') error while uploading glance image,back end is ceph..any suggestions13:22
dmsimardnoonedeadpunk: it works! no wait, it failed :p https://a5625bfb1c655cabd60a-b71c90f67122d6f7b38b8442bc3a114a.ssl.cf5.rackcdn.com/728259/15/check/openstack-ansible-functional-centos-8/b78f492/logs/ara-report/result/50.html13:22
noonedeadpunkbut it's expected failure :p13:23
*** cshen has joined #openstack-ansible13:24
dmsimardI left a comment in the patch13:26
dmsimardI left another, actually13:27
dmsimardnoonedeadpunk: no need to block the patch, can be improved in another patchset :p13:29
noonedeadpunkok, won't13:29
noonedeadpunkbut really need a follow-up one13:29
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-tests master: Add Ubuntu Focal non voting check jobs  https://review.opendev.org/72985413:32
*** rh-jelabarre has joined #openstack-ansible13:35
*** spatel has joined #openstack-ansible13:37
*** spatel has quit IRC13:42
jrossernoonedeadpunk: this is sad https://github.com/python-poetry/poetry/issues/136313:55
jrosseri'm currently not able to run a pure python 3 bionic role test becasue of that13:55
jrosseror else i'm missing something important13:56
* noonedeadpunk hears about that libbrary first time13:56
noonedeadpunkbut bug looks pretty related yes13:56
noonedeadpunkhttps://github.com/python-poetry/poetry/issues/1363#issuecomment-555913200 that is sad13:57
jrosserlike this http://paste.openstack.org/show/793842/13:57
jrossernot sure how to raise a bug for that to be released14:05
jrosserbecause it's all focussed on the apt package which contains that python module14:05
jrosserrather than the python itself14:06
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-lxc_hosts master: Add vars files and container prep script for ubuntu focal  https://review.opendev.org/72993714:11
noonedeadpunkjrosser: so and isntallation of python3-apt do not help? Or it's already installed?14:12
jrossertrouble is this is running in a tox venv14:12
jrosserso the python3-apt package is installed, but outside the venv14:12
noonedeadpunkcan think only about symlinking....14:13
jrosserhmm yes i was just thinking the same14:13
noonedeadpunkTHis is suuuuuper old https://answers.launchpad.net/ubuntu/+source/python-apt/+question/28552214:13
jrosseri'm trying to debug something else so this is just a distraction for now - but i think is shows we still use python2 a lot14:13
noonedeadpunkwith the same tox problem lol14:13
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-tests master: Simlify ara setup  https://review.opendev.org/72997514:15
noonedeadpunkwondering if selinux installation will work14:16
jrossertypo there "Simplify"14:18
noonedeadpunkyeah. I also thinking about setting some fixed versions of libselinux14:18
noonedeadpunkin case it will work at all14:18
noonedeadpunkHope you got what I tried to say with so awfull grammar :p14:19
*** udesale_ has joined #openstack-ansible14:33
*** udesale has quit IRC14:36
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-tests master: Deploy modern ara for py3 deployments  https://review.opendev.org/72825914:55
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-tests master: Deploy modern ara for py3 deployments  https://review.opendev.org/72825914:57
*** gyee has joined #openstack-ansible15:04
*** dirk has quit IRC15:38
*** mgariepy has quit IRC15:47
*** admin0 has joined #openstack-ansible16:01
*** NewJorg has quit IRC16:09
*** NewJorg has joined #openstack-ansible16:10
*** dpaclt82 has joined #openstack-ansible16:10
*** dpaclt has quit IRC16:11
*** mgariepy has joined #openstack-ansible16:23
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-galera_server master: Bump MariaDB version to 10.4  https://review.opendev.org/71783016:37
*** cshen has quit IRC16:45
*** udesale_ has quit IRC16:47
gshippey@noonedeadpunk are you happy for me to +1 workflow on this https://review.opendev.org/#/c/728434/ ??17:00
noonedeadpunkgshippey: yeah I think it's good topic and we defenitely need this patch to merge to resolve dependency17:02
noonedeadpunkgshippey: but if you have some thoughs on it it would be really great to have such input17:02
*** mgariepy has quit IRC17:05
*** mgariepy has joined #openstack-ansible17:05
gshippeyI think that ones good to go, going to do a double check of the combination patch!17:07
*** cshen has joined #openstack-ansible17:14
*** cshen has quit IRC17:19
*** jawad_axd has quit IRC17:30
openstackgerritMerged openstack/openstack-ansible-lxc_hosts master: ensure 20listchanges is not in debian container  https://review.opendev.org/71888217:30
*** jawad_axd has joined #openstack-ansible17:31
openstackgerritMerged openstack/openstack-ansible-os_neutron master: Updated from OpenStack Ansible Tests  https://review.opendev.org/72732218:14
watersjgetting this during my deployment when setup-openstack, TASK [python_venv_build : Install python packages into the venv] "stdout: Ignoring argparse: markers 'python_version == \"2.7\"' don't match your environment, full stacktrace is here https://pastebin.com/JXGpvAVD18:26
watersjubuntu 18.04, stable/train18:26
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-tests master: Deploy modern ara for py3 deployments  https://review.opendev.org/72825918:35
openstackgerritMerged openstack/openstack-ansible master: Use galera server role to install galera client  https://review.opendev.org/72843418:38
jrosserwatersj: I don’t think that is the actual error, probably need to look in the more specific log file generated by the venv build18:51
*** jawad_axd has quit IRC18:52
watersjreference to missing pkg-config in keystone container. but i not overriding or specifying python18:52
openstackgerritMerged openstack/ansible-role-python_venv_build stable/stein: Verify version number for virtualenv>=20.0.0  https://review.opendev.org/72984418:55
openstackgerritMerged openstack/openstack-ansible-os_neutron master: config: add region_name to nova  https://review.opendev.org/70866518:59
*** mathlin_ has quit IRC19:00
*** jamesdenton has quit IRC19:08
watersjCannot find libsystemd or libsystemd-journal. then it fails and retries 5 times19:13
watersjwithin keystone container19:13
*** jamesdenton has joined #openstack-ansible19:13
*** AJaeger has left #openstack-ansible19:13
jrosserwatersj: which operating system?19:14
watersjubuntu 18.0419:15
watersjshould it not be python319:15
watersj?19:15
jrosseri believe that those messages are just the pip resolver discounting candidate installs becasue they are for the wrong python version19:17
jrosseryour issue is that systemd_python cannot be built, and the error suggests that the C library headers for libsystemd and libsystemd-journal cannot be found19:18
jrosserlibsystemd-dev should be installed due to this https://github.com/openstack/openstack-ansible-os_keystone/blob/master/vars/debian.yml#L3219:18
watersjappears to be ii  libsystemd0:amd64 237-3ubuntu10.4019:19
jrosserthat is the library itself19:19
watersjmaybe I need to go to specific version of stable/train?19:20
jrosserthe development headers are always in -dev packages which must also be present to compile the python bindings for those libraries19:20
jrosserhmm19:21
jrosserdo you have repo_server containers?19:23
watersji do19:24
*** mathlin has joined #openstack-ansible19:26
watersjlooking at deploy server os_keystone/vars/debian.yml:  - libsystemd-dev19:26
watersjI reverted my test environment to a clean state and am trying to redeploy19:27
watersjpreviously keystone was working, so only thing change is possible later checkout of stable/train19:28
jrosserit should not be trying to build wheels in the keystone container19:29
jrosserthat should be delegated to the repo container and happen earlier in the keystone deployment19:29
jrosseryou can see in your log it downloads SQAlechemy and systemd-python as .tgz, rather than get the wheels from the repo server19:30
jrosseri do not think there is good value in switching about branches or tags of stable/train to "fix" this19:30
*** mathlin_ has joined #openstack-ansible19:31
jrosserthe variable i pointed you to with libsystemd-dev in it is a list of packages which must be installed into the repo server for the wheel build19:31
jrossernot into the keystone container19:32
jrosseri would really recommend building an all-in-one in a VM which you can use as a reference, because that is what is used to validate the tagged releases that are made19:32
*** mathlin has quit IRC19:33
watersjok19:33
jrosseryou can also pick any of our CI jobs to look at19:33
jrossersee the link in the IRC topic https://bit.ly/2SAcGAn19:33
jrosserhere are all the patches to the train branch https://review.opendev.org/#/q/project:openstack/openstack-ansible+branch:stable/train19:34
*** cshen has joined #openstack-ansible19:34
jrosseryou will be able to find the most recent release we made, which would be this https://review.opendev.org/#/c/728696/19:35
jrosserif you click one of the jobs (lxc, bionic would be representative) https://zuul.opendev.org/t/openstack/build/ec172dc2c4f9402aa88917a7ccb6f34719:35
jrosserunder the logs tab you can explore all the log files from the deployment19:36
watersjlet me scale it back to AIO, then do multi-node19:37
watersjjrosser, ty19:37
jrosserfor example, we can look at the keystone python venv build log https://zuul.opendev.org/t/openstack/build/ec172dc2c4f9402aa88917a7ccb6f347/log/logs/openstack/aio1_keystone_container-3c7b8ebe/python_venv_build.log.txt#6629-665819:37
jrosserand interestingly you can see there that the openstack CI python wheel server is providing the built wheel there rather than it being made on the OSA repo server19:38
jrosserso there could well be a bug there that is only manifesting itself outside the CI environment19:38
jrosseri have to go now but hopefully that is some going pointers for digging into what is broken19:39
*** jamesdenton has quit IRC19:40
*** jamesdenton has joined #openstack-ansible19:43
*** yolanda has quit IRC19:43
*** arkan has joined #openstack-ansible20:04
arkanHi guys my name is arkan, I'm new to openstack20:05
arkanI want to install it to my 2 rack servers20:06
arkanI'm using openstack-ansible20:06
arkanI have some errors in this stage openstack-ansible setup-hosts.yml20:06
arkancan anyone help me please?20:07
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible master: Return lxc jobs  https://review.opendev.org/72987820:07
noonedeadpunkpretty early for errors. what are they?20:07
arkanI think I've done a wrong network configuration20:08
arkanI have 2 rack servers, each one has 4 NICs (1GB), I want to use one server for Controller and the other for Compute20:10
arkanI'm using one NIC from each server, and each NIC has sub interfaces for different vlans20:11
arkanthose NICs (2 in total) are connected to my router20:11
noonedeadpunkjrosser: can you kindly check https://review.opendev.org/#/c/728259/18 one more time?20:12
noonedeadpunkI've implemented dmsimard's advices here all in all20:13
jrossernoonedeadpunk: sure20:13
noonedeadpunknot sure hwta this ara-mange error for centos 8 is :(20:13
jrossernoonedeadpunk: could you read back with watersj issue just now - I’m slightly worried about the repo server cleanup patches20:14
noonedeadpunkarkan: ok, sounds ok20:14
arkanyes20:14
dmsimardnoonedeadpunk: where is the error ?20:15
arkanwhere you want me to paste the errors?20:16
arkanI'm not aware of the https://review.opendev.org, this is the first time when I'm seeing it20:16
noonedeadpunkdmsimard: https://zuul.opendev.org/t/openstack/build/6ef4ac7de9514c2bb61415b560be924d/log/job-output.txt#245820:17
noonedeadpunkarkan: you can paste here http://paste.openstack.org/ or wherever conmfortable20:17
arkangreat20:18
arkanone moment, because the process is still running and I've used openstack-ansible setup-hosts.yml -vvvv20:18
noonedeadpunkjrosser: hm, you mean this one? https://opendev.org/openstack/openstack-ansible-repo_server/commit/53a8add55b480166ce5ded7df4e196f70d613c1e20:19
arkan@noonedeadpunk I will past my netplan there first and the config until the process finishes20:20
arkan@noondeadpunk http://paste.openstack.org/show/793859/20:23
noonedeadpunkoh, I faound where this error comes from I guess20:24
arkan@nooneeadpunk http://paste.openstack.org/show/793860/20:25
arkan@noonedeadpunk when the logs was running in my screen, I saw it was connectivity, it could not reach the internet20:26
noonedeadpunkI think you shouldn't have gateway for br-mgmt20:26
noonedeadpunkor not having for eno120:27
arkan@noonedeadpunk http://paste.openstack.org/show/793861/20:27
arkanmy eno1 is the only interface that is connected to the internet20:28
arkanthrough the server ip: 192.168.50.1020:28
noonedeadpunkbut I see gateway configured for br-mgmt as well20:29
noonedeadpunkso you might have 2 or wrong default route20:29
arkanyes, that is true20:30
arkanI will show you my router also20:30
arkanone moment20:30
arkanhttps://drive.google.com/drive/folders/1t71q1YlAz5JVohVFuWIzLNQ6j4GgqoL2?usp=sharing20:32
arkanI will put my config files there20:34
noonedeadpunkdmsimard: so we probably just not divide subunit report now? https://opendev.org/openstack/openstack-ansible-tests/src/branch/master/test-log-collect.sh#L26220:39
arkan@noonedeadpunk I've created openstack folder with the 3 files 1.user_variables 2.openstack_user_config.yml 3.01-netcfg.yaml20:39
noonedeadpunkok, so the question is - can you ping world, like pypi.org from hosts?20:40
arkanyes20:40
noonedeadpunkok, cool. Then the most insteresting part is error:)20:41
arkanyes, I'm waiting for the process to finish, it took a lot of time20:41
arkanand It's still running20:41
noonedeadpunkok, cool20:41
*** also_stingrayza has joined #openstack-ansible20:41
noonedeadpunkit just pretty late here....20:41
arkanhere in Romania it's 23:4220:42
arkanand I'm eagerly want to see Horizon20:42
arkanI spent a few days trying to make this work, and I'm still continuing this road :))20:43
noonedeadpunkSo we're in the same timezone lol20:43
arkanwow nice20:43
*** stingrayza has quit IRC20:45
dmsimardnoonedeadpunk: subunit does not exist in 1.x (yet)20:46
dmsimardwouldn't be terribly complicated to re-implement using the API, it's this part: https://github.com/ansible-community/ara/blob/acf7769f00de59c2d388409af62ccaa030b7bfc5/ara/cli/generate.py#L176-L29120:48
dmsimardI'd like to keep subunit out of reqs.txt though20:49
dmsimardif it's something you need I can create an issue about it20:49
noonedeadpunkhard to say how much we need it tbh, we just always had it, so seems that we need to update our script to omit it for new ara20:51
dmsimardit may have been used for openstack-health at some point ?20:51
dmsimardI don't recall the details20:51
dmsimardsubunit is arcane magic to me20:52
noonedeadpunksame for me :p20:52
dmsimardI remember needing to go through subunit's unit/functional/integration test to understand how the damn thing worked20:52
noonedeadpunkyeah, it's totally for openstack-health20:54
noonedeadpunkBut I think we used a grafana for a while...20:54
*** cshen has quit IRC20:54
noonedeadpunkbut anyway, yeah20:55
jrossernoonedeadpunk: do we merge it? :)20:55
noonedeadpunkgreat question lol20:55
* jrosser only got half an eye on this subunit business20:55
noonedeadpunkso it really neded for filling in data here http://status.openstack.org/openstack-health/#/20:56
*** arkan has quit IRC20:56
* noonedeadpunk start thinking if we should use new ara for centos 8 only20:57
*** arkan has joined #openstack-ansible21:04
arkan@noonedeadpunk sorry I was disconnected21:04
arkan@noonedeadpunk I've put the errors in the files21:05
arkanhttps://drive.google.com/drive/folders/1t71q1YlAz5JVohVFuWIzLNQ6j4GgqoL2?usp=sharing21:05
arkanI've tried to paste it on paste.openstack.org but it does not work now21:06
arkanso I put the errors in the file in my drive21:06
arkannoonedeadpunk: it seems that it can not connect from inside the container21:11
arkan: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/p/python-apt/python-apt-common_1.6.5ubuntu0.2_all.deb  Temporary failure resolving \'us.archive.ubuntu.com\'21:11
noonedeadpunkarkan: ok, so network from host works.21:12
noonedeadpunkbut seems the issue is in lxc containers21:12
arkannoonedeadpunk: I think the network config. from inside the container has the problem21:12
arkanbut from the host it can connect to the internet21:12
noonedeadpunkok, so containers should connect to the internet through lxcbr0 by doing src-nat21:13
noonedeadpunkand they tend to set nameserver to the ip which is lxcbr21:14
noonedeadpunkthis bridge is configured automatically21:14
noonedeadpunkBut I still have doubts regarding having 2 default routes.21:14
noonedeadpunkwhat is the output of ip r on controller?21:15
noonedeadpunk`ip r`21:15
*** gshippey has quit IRC21:16
arkannoonedeadpunk: so how can I modify my netplan config to make it work?21:17
arkanshould I remove the gateway from br-mgmt21:17
arkanshould I remove the gateway from br-mgmt?21:17
arkaneno1 has the server ip (vlan 50 in router untagged)21:18
noonedeadpunkI'd say yes21:19
noonedeadpunkbut not sure that issue is really exactly in it21:19
arkanbr-mgmt: 172.29.236.10/22 (uses vlan10 tagged through the router)21:19
noonedeadpunkgenerally you shouldn't have 2 default rautes in any time21:19
noonedeadpunk*routes21:20
noonedeadpunkyou can check current routes with `ip r` command21:20
noonedeadpunkso i'd use gateway for eno1 and not frm br-mgmt21:20
arkanroot@controller1:~# ip route21:20
arkandefault via 192.168.50.1 dev eno1 proto static21:20
arkandefault via 172.29.236.1 dev br-mgmt proto static21:20
arkan10.0.3.0/24 dev lxcbr0 proto kernel scope link src 10.0.3.1 linkdown21:20
arkan172.29.236.0/22 dev br-mgmt proto kernel scope link src 172.29.236.1021:20
arkan172.29.240.0/22 dev br-vxlan proto kernel scope link src 172.29.240.1021:20
arkan192.168.50.0/24 dev eno1 proto kernel scope link src 192.168.50.1021:20
arkan192.168.50.0/24 dev br-flat proto kernel scope link src 192.168.50.10021:20
noonedeadpunkhuh, so traffic should go through eno1 which is correct21:22
noonedeadpunkcan you ping router or world from inside container?21:22
noonedeadpunkyou can access container with `lxc-attach -n <container_name>`21:22
arkanroot@controller1:~# lxc list21:23
arkan+------+-------+------+------+------+-----------+21:23
arkan| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |21:23
arkan+------+-------+------+------+------+-----------+21:23
arkanthere is no container running21:23
noonedeadpunklxc-ls -a21:23
noonedeadpunklxc-ls --active21:24
arkanthe command is not working21:24
arkanyes21:24
arkannow it listed the containers21:24
arkanno21:25
arkanroot@controller1:~# lxc-attach -n controller1_rabbit_mq_container-0349abaa21:25
arkanroot@controller1-rabbit-mq-container-0349abaa:~#21:25
arkanroot@controller1-rabbit-mq-container-0349abaa:~#21:25
arkanroot@controller1-rabbit-mq-container-0349abaa:~#21:25
arkanroot@controller1-rabbit-mq-container-0349abaa:~# ping google.com21:25
arkanping: google.com: Temporary failure in name resolution21:25
arkanroot@controller1-rabbit-mq-container-0349abaa:~#21:25
arkan21:25
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-tests master: Deploy modern ara for CentOS 8 deployments  https://review.opendev.org/72825921:25
arkanroot@controller1-rabbit-mq-container-0349abaa:~# ip r21:25
arkan172.29.236.0/22 dev eth1 proto kernel scope link src 172.29.239.22821:25
noonedeadpunkso it's missing default route?21:26
noonedeadpunkthat's weird21:26
noonedeadpunkwhat about `ip a`?21:26
arkanroot@controller1-rabbit-mq-container-0349abaa:~# ip link21:26
arkan1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 100021:26
arkan    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:0021:26
arkan149: eth0@if150: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 100021:26
arkan    link/ether 00:16:3e:f1:0a:7b brd ff:ff:ff:ff:ff:ff link-netnsid 021:26
arkan151: eth1@if152: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 100021:26
arkan    link/ether 00:16:3e:ab:22:48 brd ff:ff:ff:ff:ff:ff link-netnsid 021:26
arkanroot@controller1-rabbit-mq-container-0349abaa:~# ip a21:27
arkan1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 100021:27
arkan    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:0021:27
arkan    inet 127.0.0.1/8 scope host lo21:27
arkan       valid_lft forever preferred_lft forever21:27
arkan    inet6 ::1/128 scope host21:27
arkan       valid_lft forever preferred_lft forever21:27
arkan149: eth0@if150: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 100021:27
arkan    link/ether 00:16:3e:f1:0a:7b brd ff:ff:ff:ff:ff:ff link-netnsid 021:27
arkan    inet6 fe80::216:3eff:fef1:a7b/64 scope link21:27
arkan       valid_lft forever preferred_lft forever21:27
arkan151: eth1@if152: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 100021:27
arkan    link/ether 00:16:3e:ab:22:48 brd ff:ff:ff:ff:ff:ff link-netnsid 021:27
arkan    inet 172.29.239.228/22 brd 172.29.239.255 scope global eth121:27
arkan       valid_lft forever preferred_lft forever21:27
arkan    inet6 fe80::216:3eff:feab:2248/64 scope link21:27
arkan       valid_lft forever preferred_lft forever21:27
arkanroot@controller1-rabbit-mq-container-0349abaa:~#21:27
noonedeadpunkoh my, please use some pastebin :)21:27
noonedeadpunkok, so eth0 don't have an ip adress for some reason while should21:28
*** cshen has joined #openstack-ansible21:28
arkanhttps://pastebin.com/P9JEKzMu21:29
arkanyes, I don't know why21:29
noonedeadpunkyeah I got it - so problem is that eth0 don't have ip address21:29
arkanI was thinking that something wrong with my network config21:29
noonedeadpunkwhat bridges do you have on host?21:30
noonedeadpunkYeah21:30
noonedeadpunkBut I don't think it's smth with osa config21:30
arkanI think in my netplan21:30
arkanfrom what I've read in the doc21:30
arkanbr-vlan does not need an ip address21:31
arkanbut the problem that my router can not be configured without an ip address for that port21:31
arkanand my 2 servers are connected to the router21:31
arkanport 1 on router ---> for server 1 (controller) and port 2 on router --->  for (compute)21:32
noonedeadpunkso I'd try removing second defult route and re-create containers21:32
arkanwhat is the new netplan config that I should use?21:33
arkanyou can download make change to the file from my drive21:33
*** cshen has quit IRC21:33
noonedeadpunkyou can drop containers like this openstack-ansible /opt/openstack-ansible/playbooks/lxc-containers-destroy.yml --limit rabbitmq_all21:34
arkanok21:34
noonedeadpunkabd create openstack-ansible /opt/openstack-ansible/playbooks/lxc-containers-create.yml --limit rabbitmq_all,lxc_hosts21:34
noonedeadpunkjust drop gateway from br=mgmt21:34
arkanI will drop the gateway first from br-mgmt21:35
arkanI dropped the gateway from both controller and compute21:41
arkanand I ran the above code21:41
arkanI've received an error21:42
arkanhttps://pastebin.com/ybVmnkgL21:43
arkanand from inside the container21:49
arkanhttps://pastebin.com/r2EmiusH21:49
arkannoonedeadpunk: but how to reach the internet if it's tagged inside the route?21:52
arkanI can not see default gateway in the route21:53
arkanonly untagged vlan (50) can get to the internet21:54
jrosserarkan: bear in mind it’s late :) but you are missing eth0 having an ip from inside your container21:55
jrossereth0 will be assigned an address in the 10.x range by dnsmasq running on the host21:56
arkanyes, I will try to figure it out, till tomorrow, thanks a lot @noonedeadpunk @jrosser for your help21:57
arkangood night21:57
jrossereth0 should be the default route in the container, and the host NATs eth0 on behalf of each container and the traffic goes out of the hosts default route21:57
arkanok, I will check about it21:59
arkanthanks a lot21:59
*** arkan has quit IRC22:02
*** rh-jelabarre has quit IRC22:09
*** itandops has joined #openstack-ansible22:24
*** itandops has quit IRC22:52
*** itandops has joined #openstack-ansible22:53
*** dpaclt82 has quit IRC23:14
*** NewJorg has quit IRC23:29
*** gyee has quit IRC23:46

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!