Thursday, 2018-11-22

openstackgerritArx Cruz proposed openstack/openstack-ansible-os_tempest master: Add centos-7 job with support to python-tempestconf  https://review.openstack.org/61902100:31
*** cshen has quit IRC00:34
*** dcdamien has quit IRC00:35
*** tosky has quit IRC01:07
*** jamesdenton has quit IRC01:12
*** cshen has joined #openstack-ansible01:30
*** cshen has quit IRC01:35
*** thuydang has joined #openstack-ansible02:16
*** thuydang has quit IRC02:34
*** thuydang has joined #openstack-ansible02:38
*** mmercer has quit IRC02:43
*** mgariepy has quit IRC03:24
*** mmercer has joined #openstack-ansible03:25
*** mgariepy has joined #openstack-ansible03:35
*** udesale has joined #openstack-ansible04:15
*** thuydang has quit IRC04:20
*** ivve has joined #openstack-ansible04:39
*** czunker has joined #openstack-ansible05:08
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: reduce container count on cluster test  https://review.openstack.org/61943605:19
*** chandankumar has joined #openstack-ansible05:33
*** chandankumar is now known as chkumar|ruck05:33
*** ahosam has joined #openstack-ansible06:31
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Update elk_6x for 6.5.x  https://review.openstack.org/61944806:51
*** ahosam has quit IRC06:57
*** radeks_ has joined #openstack-ansible07:10
*** cshen has joined #openstack-ansible07:16
*** pcaruana has joined #openstack-ansible07:22
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Update elk_6x for 6.5.x  https://review.openstack.org/61944807:24
cshensomething wrong with zuul? Now I got more TIMEOUT after commented 'recheck'. https://review.openstack.org/#/c/58124807:25
jonheropensuse jobs seem to timeout yesterday, not sure if it's the same today or not07:30
cshenLet me do 'recheck' one more time.07:33
jonherI already did I think :)07:33
jrosserQuite often the upstream packages get broken, perhaps more often for suse than anything else07:36
jrosserAnd that is one reason for timeouts, it’s worth digging into the log and finding out why if it is persistent07:37
jrossercloudnull: seems my day looks all elk shaped now :)07:38
cshenjonher thanks, saw that07:40
*** hamzaachi has joined #openstack-ansible07:42
*** rpittau has joined #openstack-ansible07:44
*** eumel8 has joined #openstack-ansible07:51
*** gkadam has joined #openstack-ansible08:15
*** jbadiapa has quit IRC08:16
*** shardy has joined #openstack-ansible08:26
*** jbadiapa has joined #openstack-ansible08:32
*** jbadiapa has quit IRC08:37
*** mma has joined #openstack-ansible08:42
openstackgerritMarcus Murwall proposed openstack/openstack-ansible-os_keystone master: Adds support for changing port for keystone galera database.  https://review.openstack.org/61947408:44
*** hamzaachi_ has joined #openstack-ansible08:46
*** hamzaachi has quit IRC08:48
*** tosky has joined #openstack-ansible08:49
*** chhagarw has joined #openstack-ansible08:55
*** trident has quit IRC08:57
*** trident has joined #openstack-ansible08:59
*** dcdamien has joined #openstack-ansible09:03
*** jastack has joined #openstack-ansible09:04
*** jastack has quit IRC09:06
*** jstack__ has joined #openstack-ansible09:06
*** jstack__ has quit IRC09:12
*** jawad_ax_ has joined #openstack-ansible09:12
*** DanyC has joined #openstack-ansible09:17
*** jbadiapa has joined #openstack-ansible09:19
*** mrhillsman is now known as openlab09:19
*** openlab is now known as mrhillsman09:20
*** jawad_ax_ has quit IRC09:27
openstackgerritArx Cruz proposed openstack/openstack-ansible-os_tempest master: Add centos-7 job with support to python-tempestconf  https://review.openstack.org/61902109:27
arxcruzevrardjp: ^09:28
*** jawad_axd has joined #openstack-ansible09:28
*** hamzaachi_ has quit IRC09:32
*** hamzaachi_ has joined #openstack-ansible09:32
*** jawad_axd has quit IRC09:36
*** Emine has joined #openstack-ansible09:41
*** jawad_axd has joined #openstack-ansible09:42
*** hamzaachi_ has quit IRC09:46
*** rgogunskiy has quit IRC09:50
*** rgogunskiy has joined #openstack-ansible09:51
*** electrofelix has joined #openstack-ansible09:52
*** hamzaachi has joined #openstack-ansible09:55
*** rgogunskiy has quit IRC09:56
odyssey4meyeah, it looks like we're hitting timeouts reaching the mariadb  repository or installing from it09:58
jonhercshen: looks like galera install takes +1,5h for centos and suse for some reason09:58
*** rgogunskiy has joined #openstack-ansible10:04
openstackgerritMerged openstack/ansible-role-systemd_service master: fix tox python3 overrides  https://review.openstack.org/60680810:05
evrardjparxcruz: good job : )10:05
arxcruz:D10:06
evrardjparxcruz: I don't know if I can edit your patches, so sorry for the back and forth :p10:06
*** rgogunskiy has quit IRC10:09
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: WIP: Enable passing nspawn jobs  https://review.openstack.org/60843310:10
openstackgerritMerged openstack/ansible-role-systemd_networkd master: fix tox python3 overrides  https://review.openstack.org/60676310:10
*** rgogunskiy has joined #openstack-ansible10:18
*** rgogunskiy has quit IRC10:18
*** rgogunskiy has joined #openstack-ansible10:18
arxcruzevrardjp: no problem, it's good to learn more10:18
evrardjpCI will be very calm today, it's time to hack things around10:23
*** ianychoi has quit IRC10:25
nowster"Failed to import test module: keystone_tempest_plugin.tests.api.identity.v3.test_identity_providers" due to "ImportError: libxslt.so.1: cannot open shared object file: No such file or directory". Has someone seen this?10:27
evrardjpI am curious why these tests are loaded, but let's say okay...10:31
evrardjpnowster: we probably removed a package in the recent changes for cleanup10:31
odyssey4menowster where are you seeing that?10:31
nowstertempest tests10:32
odyssey4meyes, but which patch/role?10:32
nowsterDoing the -vvv run now.10:32
nowsterit's master, with jrosser's changes10:32
odyssey4menowster you're going to need to be more specific if you want help10:33
odyssey4mehowever, that's showing that there's a missing libxml package on the host in question10:33
nowsterRight.10:33
nowsterI'm doing an AIO install for testing purposes on arm64.10:34
odyssey4meat runtime, this package will be needed: https://github.com/openstack/openstack-ansible-os_cinder/blob/master/vars/ubuntu.yml#L2710:34
jrosserodyssey4me: i had some other issues yesterday with tempest plugins missing dependancies, i need to dig a bit more and might need some help with that10:34
odyssey4meat wheel build time this package will be needed: https://github.com/openstack/openstack-ansible-os_cinder/blob/master/vars/ubuntu.yml#L5010:35
odyssey4meat least I think those are the two10:35
odyssey4meah no, it was in lance where we worked this out: dev: https://github.com/openstack/openstack-ansible-os_glance/blob/master/vars/ubuntu.yml#L2810:36
odyssey4meruntime: https://github.com/openstack/openstack-ansible-os_glance/blob/master/vars/ubuntu.yml#L2210:36
*** gary_perkins has joined #openstack-ansible10:36
odyssey4meI suspect many of the openstack services will require those two packages.10:36
jrosserodyssey4me: what do you think of this? http://logs.openstack.org/78/613078/17/experimental/openstack-ansible-integrated-deploy-aio/49a21cb/job-output.txt.gz#_2018-11-21_15_31_00_97244610:40
jrosserit seems that a dependancy of one of the tempest plugins is not available from the repo server?10:40
odyssey4mejrosser 'Failed building wheel for dulwich'10:41
nowster"unable to execute 'x86_64-linux-gnu-gcc': No such file or directory" means you've not got a compiler.10:41
jrosseryes it's becasue there is no build chain on the utiity container10:41
jrosserwhich is reasonable10:41
odyssey4meis this integrated build?10:42
odyssey4meyes, it seems it is - that's odd10:42
jrosseris it becasue the repo server isnt aware of which tempest plugins *might* be needed, as that gets decided later10:44
odyssey4mebut of course, the tempest plugin installs are isolated and don't use the repo server I don't think10:44
*** hamzaachi has quit IRC10:45
jrosserwhats the best plan here - should we require a toolchain on the utility container because of this?10:45
odyssey4methe best plan, I think, is to change this mechanism entirely so that it doesn't install like this... it's something I started doing before berlin10:46
odyssey4meif we need to stop some bleeding now, then I'd say let's put the toolchain down by adding the requisite runtime packages in https://github.com/openstack/openstack-ansible-os_tempest/blob/master/tasks/tempest_install_source.yml#L3710:47
*** admin0 has joined #openstack-ansible10:47
admin0how do i enable magnum and babrican on an AIO ?10:48
jrosserwell - the only thing it's holding up is merging the strip-down-the-lxc_hosts packages patch10:48
admin0\o10:48
jrosserso, if theres a fix for the way the tempest plugin requirements are built we could wait for that10:48
odyssey4meadmin0 the same way you would any other services10:49
admin0just specifying the mangum-infra_host in the config is enough ?10:49
odyssey4meadmin0 https://docs.openstack.org/openstack-ansible/rocky/user/aio/quickstart.html see the sentence 'To add OpenStack Services over and above...'10:50
admin0conf :)10:50
admin0got it10:50
openstackgerritMerged openstack/openstack-ansible-os_neutron master: fix tox python3 overrides  https://review.openstack.org/60681211:07
openstackgerritMerged openstack/openstack-ansible-os_swift master: remove loop control when running include_role  https://review.openstack.org/60781111:12
*** udesale has quit IRC11:15
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: [docs] Clean up the AIO user story  https://review.openstack.org/61952211:20
*** gkadam has quit IRC11:24
mnaserodyssey4me: i think somehow creating a router is killing our nodes..11:27
mnaserand we cant gather logs..11:27
admin0there are no logs at all ?11:28
mnaseror it looks like after the first tempest flavor comes in it dies, it died the same exact way before11:28
*** gkadam has joined #openstack-ansible11:28
mnasernope, because the vm becomes unpingable at that point11:28
odyssey4memnaser oh, that's nice :/11:28
mnasernotice http://logs.openstack.org/33/608433/9/check/openstack-ansible-deploy-aio_nspawn-centos-7/b51661e/job-output.txt.gz11:28
mnaserand11:28
mnaserhttp://logs.openstack.org/33/608433/9/check/openstack-ansible-deploy-aio_nspawn-centos-7/36278e5/11:28
mnasersame failure, same place, different providers11:28
mnaserrax vs inap11:29
odyssey4memnaser oh you're referring to nspawn - right, that was an issue that cloudnull was trying to work out IIRC11:29
mnaseroooh i see11:29
mnaseryes i am referring to nspawn11:29
mnaserdo you know if he made any progress anywhere?11:29
odyssey4mehe said that was the last issue to work out for the gate at the PTG - something something flat networks and tempest tests11:29
mnaseryeah i remember that11:29
mnaserbut as i recall it was just tests failing *during* tempest runs, not nodes combusting11:30
odyssey4meI dunno - nice to see that it's getting that far though.11:30
*** ianychoi has joined #openstack-ansible11:31
mnaseryeah its exciting, i have had to make one change (os_glance)11:31
odyssey4meIt seems a bit odd that this is failing though, given that the routers are not in containers.11:31
mnaseryeah, well, my "theory" that it's routers breaking it is just a theory11:31
mnaseri dont think adding flavors might be affecting it11:31
mnaserbut probably that till the router goes up is enough time till it goes poof11:32
mnasermight just have to spin one locally i guess11:32
mnaserand the routers live in netns.. so they shouldn't affect connectivity..11:33
odyssey4mewe're getting heavy timeout failures for the integrated build though, and it seems to related to fetching the galera packages11:33
odyssey4meI thought we'd sorted out infra mirrors for those?11:33
odyssey4mewell, an infra reverse proxy11:33
mnasermaybe i added the reverse proxy but not didnt point at it?11:33
mnaseri dont remember a lot of the details11:33
odyssey4mecheck this out: http://logs.openstack.org/18/619218/2/check/openstack-ansible-deploy-aio_metal-opensuse-423/f03aa25/job-output.txt.gz#_2018-11-22_00_08_04_86277011:34
odyssey4metwo hours, stuck11:34
mnaserchanged: [aio1] => (item={u'description': u'MariaDB Repo', u'name': u'MariaDB', u'baseurl': u'http://downloads.mariadb.com/MariaDB/mariadb-10.2.17/yum/opensuse42-amd64'})11:35
mnasercertainly not using mirrors there11:35
mnaserlet me see11:35
odyssey4meone hour: http://logs.openstack.org/18/619218/2/check/openstack-ansible-deploy-aio_lxc-opensuse-150/1460c9c/job-output.txt.gz#_2018-11-22_00_42_08_12067411:35
mnasermaybe opensuse isn't consuming it11:35
odyssey4meI suspect we only have an ubuntu mirror. It'd be ideal if we could have a reverse proxy instead, I think.11:35
mnaserhttps://github.com/openstack-infra/system-config/commit/40cbd743154abf37acc7bb0921115db344b3648f#diff-adaea27db1d872629323905d8000330911:36
mnasernope, i added a mirror for them all11:36
mnaserin b4 this is a case of integrated repo not having the mirror and the o-a-t having them11:36
odyssey4meah, a reverse proxy - not mirror11:36
mnaserstill, based on the task, its not consuming it at all11:36
mnaserits going to mariadb directly, if you look one task above11:36
odyssey4meyep, ok - do we know what the right URL is for that?11:36
mnaserone sec11:37
*** aedc has quit IRC11:37
odyssey4meaha NODEPOOL_MARIADB_PROXY11:38
mnaserhttp://downloads.mariadb.com/MariaDB/mariadb-10.2.17/yum/opensuse42-amd64/ => http://mirror.sjc1.vexxhost.openstack.org:8080/MariaDB/mariadb-10.2.17/yum/opensuse42-amd64/11:38
odyssey4mehttp://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/roles/mirror-info/templates/mirror_info.sh.j2#n7411:38
odyssey4meok, I'll work up a patch for it now.11:39
mnaseri guess i did one half of it11:39
mnasersorry11:39
odyssey4meno worries - it takes a village :)11:39
mnaserodyssey4me: nope i did part of it11:39
mnaserecho "galera_repo_host=${NODEPOOL_MIRROR_HOST}:8080"11:39
mnaserin oat11:39
mnaseri bet its just integrated..11:39
mnaserodyssey4me: yep, its just missing in integrated repo, ill let you push it up :)11:40
mnaseryay more stable and faster jobs11:41
odyssey4methanks for the pointer - that saves me some time :)11:41
mnaserodyssey4me: you get to make the giant commit that wipes most of openstack-ansible-tests :D11:41
mnaser(soon)11:41
openstackgerritMohammed Naser proposed openstack/openstack-ansible master: rsyslog: drop container  https://review.openstack.org/61952611:42
odyssey4memnaser yeah, I've got the right patches up to do the ground work - once those are passing I'll pick a role as a tester  and try it out11:43
mnaser++11:43
mnaserim running centos7 + nspawn locally in vagrant11:45
odyssey4mehmm, it's not clear to me how the tests repo changes for the mariadb proxy are working at all11:48
odyssey4mehttp://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/roles/mirror-info/templates/mirror_info.sh.j2#n74 has the suffix 'mariadb', but the roles use 'MariaDB'11:48
mnaserodyssey4me: because it does like /mariadb/MariaDB/11:49
mnaserwait11:49
guilhermespmorning o/11:49
mnaseryou're right11:49
mnaserhttp://mirror.sjc1.vexxhost.openstack.org:8080/MariaDB/mariadb-10.2.17/11:49
mnaserthat's the functining url11:49
odyssey4meyep11:49
mnaserohhh11:49
openstackgerritchenge proposed openstack/openstack-ansible-ops master: Correct spelling mistakes  https://review.openstack.org/61953011:49
mnaserbecause i dont use that variable odyssey4me11:50
mnaserecho "galera_repo_host=${NODEPOOL_MIRROR_HOST}:8080"11:50
mnaserso it swaps out that part and ~just works~11:50
mnaserprobably should fix that in system-config then11:50
odyssey4meoh, weird - so the reverse proxy is accepting 'MariaDB' then, even though it's configured for 'mariadb'?11:51
odyssey4meor the reverse proxy is actually configured for 'MariaDB' and the env var there is wrong11:51
mnaserodyssey4me: configure_mirror is just a "helper", the real config is probably configured with /MariaDB?11:51
mnaserhttps://github.com/openstack-infra/system-config/blob/ae24d65d78e5fe3208dde2e58215824f92f724dc/modules/openstack_project/templates/mirror.vhost.erb#L258-L26111:52
mnaseri remember now, it didn't work properly at first, and we had to change it to upper case with help on infra11:52
mnaserbut i guess we never fixed that part11:52
odyssey4meaha, ok - I'll push a patch up to correct the env var too then11:52
mnaserthank you11:52
odyssey4meok, that all makes more sense now11:52
mnaserthe latest nspawn build seems to have managed to create flavors but died after11:53
mnaserrunning limestone this time so it *has* to be something in our stuff, running in vagrant now should be useful/helpful11:53
openstackgerritchenge proposed openstack/openstack-ansible-plugins master: Correct spelling mistakes  https://review.openstack.org/61953111:54
openstackgerritMaoyangLiu proposed openstack/openstack-ansible-os_horizon master: fix the url from http to https  https://review.openstack.org/61953211:54
openstackgerritchenge proposed openstack/openstack-ansible-lxc_hosts master: Correct spelling mistakes  https://review.openstack.org/61953411:57
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: Add MariaDB infrastructure mirrors  https://review.openstack.org/61953511:58
odyssey4memnaser evrardjp ^ that should resolve the most recent timeouts seen most especially on suse, but this will benefit all builds11:58
odyssey4methanks mnaser for doing all the leg work there11:58
openstackgerritchenge proposed openstack/openstack-ansible-lxc_container_create master: Correct spelling mistakes  https://review.openstack.org/61953612:00
guilhermespI'm curious, why some tests are passing if I didn't even tested the role? https://review.openstack.org/#/c/618820/12:06
chkumar|ruckjrosser, I didnot get a change to look at barbican tempest plugin failure12:08
czunkerHi12:09
czunkerThanks to everyone involved in openstack-ansible. Last night we finished our upgrade from Pike to Queens with openstack-ansible.12:09
chkumar|ruckjrosser, it is something related to this https://github.com/openstack/barbican-tempest-plugin/blob/master/barbican_tempest_plugin/clients.py#L22 one12:09
czunkerWe had some problems migrating the neutron agents from lxc to baremetal but got the best support one can think of. Thank you.12:09
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: Remove unnecessary octavia scenario AIO bootstrap  https://review.openstack.org/61921812:10
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: Ensure AIO container_tech/install_method vars are namespaced  https://review.openstack.org/61923312:10
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: Implement AIO changes to allow test usage by roles  https://review.openstack.org/55179112:11
odyssey4mecores, I'd appreciate some eyes on https://review.openstack.org/#/c/619522/ - it cleans up the AIO user guide a bit and makes it easier to link to specific parts12:13
jrosserchkumar|ruck: interesting - it was a failure i'd not seen before.....12:14
odyssey4methe resulting guide makes much more sense IMO: http://logs.openstack.org/22/619522/1/check/openstack-tox-docs/61c7551/html/user/aio/quickstart.html12:14
chkumar|ruckjrosser, you can locally reproduce it12:14
chkumar|ruckjrosser, git clone tempest and barbican-tempest-plugin12:14
chkumar|ruckinstall it in a venv12:14
chkumar|ruckthen tempest init workspace12:14
chkumar|ruckcd workspace12:14
chkumar|rucktempest run -l12:15
mnaserman running this vagrant thing really took a toll on my local machine lol12:15
*** hamzaachi has joined #openstack-ansible12:15
chkumar|ruckduring test directly it will fail12:15
chkumar|ruckI have seen this previously with ironic and barbican tempest plugin12:15
jrosserchkumar|ruck: oh right, so it's actually totally decoupled from any OSA stuff12:15
chkumar|ruckit is something related to dynamic credentials migration12:16
chkumar|ruckjrosser, If that patch is urgent, i will take a look tomorrow morning12:16
odyssey4meafk for a bit12:17
jrosserchkumar|ruck: the patch isn't urgent, but i guess the wider implication is that the tempest role changes might now have broken the barbican role tests12:20
chkumar|ruckjrosser, yes might be12:20
guilhermespcores, would be good to already merge this https://review.openstack.org/#/c/618820/ to not become a giant patch?12:26
mnaserguilhermesp: i think it's better to have one patch we merge that actually works, that way, at any point, the repo is functional12:27
mnaserbut i'll leave others to chime in too12:27
mnaseralso -- how do we feel about adding https://lists.centos.org/pipermail/centos-announce/2015-December/021585.html to centos .. having 1.9 means we can do shallow clones in centos 7 which would speed things up a loooot12:27
guilhermespk mnaser thanks. Sounds reasonable anyways12:27
jrossermnaser: "[WARNING]: Ignoring depth argument. Shallow clones are only available for HEAD, branches, tags or in combination with refspec." we might want to investigate that12:29
mnaserso i guess its borked for both ubuntu and centos12:29
mnaserfor different reasons12:30
mnaser:p12:30
mnasermaybe we need to add a refspec but that's above my git-fu12:30
evrardjpmnaser: I laughed at the youtube video.12:33
chkumar|ruckjrosser, I will take a look12:33
mnaser:D12:33
evrardjpmnaser: you're not on holiday today?12:34
chkumar|ruckjrosser, does openstack-ansible gives some of kind of reproducer to setup the same job locally on libvirt?12:34
mnaserCanadian thanksgiving is second Monday of October :)12:34
mnaserI am on a small holiday because running aio using vagrant on my laptop has destroyed it :p12:34
guilhermesphahahhahaha12:34
jrosserchkumar|ruck: i think you can run the role test locally by cloning the repo and doing "./run_tests.sh functional"12:35
chkumar|ruckjrosser, sure12:36
nowsterWow! That's the first time I've had scripts/bootstrap-aio.sh fail.12:36
evrardjpguilhermesp: reviewed12:39
nowsterTASK [systemd_service : Create TEMP run dir] ***********************************12:39
nowsterfatal: [localhost]: FAILED! => {"msg": "'dict object' has no attribute 'interface'"}12:39
guilhermespawesome thanks evrardjp !12:39
evrardjpguilhermesp: I gave you extra homework, don't be so happy :p12:39
evrardjpnowster: mmmm , no facts in cache?12:39
evrardjpnowster: which branch, master?12:40
guilhermespwell, at this stage, I'm really happy accepting homework related to this role evrardjp :)12:40
openstackgerritchen jiao proposed openstack/openstack-ansible-rabbitmq_server master: spelling error  https://review.openstack.org/61955612:40
nowstermaster12:40
nowstertotally clean Ubuntu 18.0412:40
nowsteramd6412:40
nowsterNow, that worked yesterday! ;-)12:41
*** cshen_ has joined #openstack-ansible12:42
evrardjphttps://github.com/openstack/openstack-ansible/commits/master has nothing since yesterday12:43
evrardjpso it's in the environmenet12:43
evrardjpmaybe cleanup your fact cache?12:44
nowsterIt's a totally fresh machine.12:44
nowsterFirst run.12:44
evrardjplet's see what changed in roles then12:44
*** cshen has quit IRC12:45
guilhermespevrardjp: answered your comments :)12:47
*** czunker has quit IRC13:00
openstackgerritLuis Filipe Domingues proposed openstack/openstack-ansible-os_cinder master: Add possibility to have private cinder volume types  https://review.openstack.org/61926413:02
openstackgerritGuilherme  Steinmuller Pimentel proposed openstack/openstack-ansible-os_placement master: [WIP] Create base files to install placement  https://review.openstack.org/61882013:08
evrardjpnowster: I have a new machine I am trying it13:11
evrardjplet's find this bad boy together!13:12
evrardjpok got no issue on my side13:12
evrardjpnowster: could you paste a full log somewhere?13:13
nowsterI'm reverting the most recent merge in systemd_service... no change.13:15
nowsterevrardjp: full log of how much?13:15
nowsterscripts/bootstrap-aio.sh ?13:16
evrardjpnowster: of the full bootstrap-ansible + bootstrap aio?13:16
nowsterok. From scratch, or just re-run them?13:16
nowsterevrardjp: ^^13:16
evrardjpjust re-run them both, and paste on pastebin or something13:16
nowsterOK.13:17
nowsterDoing that now.13:17
nowsterevrardjp: http://www.nowster.me.uk/work/openstack/bootstrap-ansible.log.txt  and  http://www.nowster.me.uk/work/openstack/bootstrap-aio.log.txt13:25
openstackgerritLuis Filipe Domingues proposed openstack/openstack-ansible-os_cinder master: Add possibility to have private cinder volume types  https://review.openstack.org/61926413:27
evrardjpnowster: do you have any override files in your /etc/openstack_deploy/user_* ?13:29
evrardjpnowster: http://paste.openstack.org/show/735937/ here is the result of my run13:30
nowsternothing13:33
nowsterTotally clean check out of master13:34
*** udesale has joined #openstack-ansible13:44
jrossernowster: did you have any issues with the repo build on arm?13:44
jrosserlike this http://paste.openstack.org/show/735939/13:45
jrosseroh wait, thats my fault :/13:49
*** pcaruana has quit IRC13:50
nowsterjrosser: the one I mentioned earlier13:55
nowsterwith the missing xslt library13:55
jrosserrighto - that was only once you got to tempest though?13:55
nowsteryes13:55
nowsterI've been consistently failing at the tempest steps.13:56
jrosserhopefully i should get to the same point today13:56
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible-nspawn_hosts master: Change ssh variable in Ubuntu and remove masks  https://review.openstack.org/61933914:00
*** chkumar|ruck has quit IRC14:02
mnaserok so14:02
mnaseri think part of what's going on is that.. we no longer have bridges?14:02
mnaserso in nspawn, because of macvlan stuff, we don't have br-vlan anymore so neutron-ovs cant attach to it?14:02
mnaseror neutron-whatever-agent14:02
mnaserneutron.agent.linux.utils [req-e71f875b-0f1a-4b8d-8578-2025aa9f747f - - - - -] Exit code: 1; Stdin: ; Stdout: ; Stderr: device eth0 is already a member of a bridge; can't enslave it to bridge brq6a0c2821-92.14:03
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-repo_server master: Enable git automatic thread count detection  https://review.openstack.org/61958014:04
mnaserand im guessing thats because container_bridge: "{{ ansible_default_ipv4['alias'] }}" instead of container_bridge: "br-vlan"14:04
mnaserhttps://github.com/openstack/openstack-ansible/blob/master/tests/roles/bootstrap-host/tasks/main.yml#L132-L13714:08
mnaserspecifically we don't create bridges anymore14:08
*** pcaruana has joined #openstack-ansible14:25
*** emine__ has joined #openstack-ansible14:30
*** Emine has quit IRC14:30
openstackgerritMohammed Naser proposed openstack/openstack-ansible master: WIP: Enable passing nspawn jobs  https://review.openstack.org/60843314:34
mnaserwell that was a lot of work.  lets see how this works14:34
openstackgerritLuis Filipe Domingues proposed openstack/openstack-ansible-os_cinder master: Add possibility to have private cinder volume types  https://review.openstack.org/61926414:39
openstackgerritJonathan Rosser proposed openstack/openstack-ansible master: [DNM] Test arm64  https://review.openstack.org/61830514:39
openstackgerritGuilherme  Steinmuller Pimentel proposed openstack/openstack-ansible-os_placement master: [WIP] Create base files to install placement  https://review.openstack.org/61882014:40
openstackgerritLuis Filipe Domingues proposed openstack/openstack-ansible-os_cinder master: Add possibility to have private cinder volume types  https://review.openstack.org/61926414:46
*** jawad_axd has quit IRC14:47
*** jawad_axd has joined #openstack-ansible14:48
*** jawad_ax_ has joined #openstack-ansible14:50
openstackgerritMohammed Naser proposed openstack/openstack-ansible master: WIP: Enable passing nspawn jobs  https://review.openstack.org/60843314:52
*** jawad_axd has quit IRC14:53
*** jawad_ax_ has quit IRC14:55
*** emine__ has quit IRC15:02
*** hamzaachi has quit IRC15:02
nowstermnaser: trying your changes15:13
mnasernowster: which one?15:20
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: [docs] Clean up the AIO user story  https://review.openstack.org/61952215:23
nowstermnaser: 60843315:23
nowsterstill getting that "interface" problem15:25
mnasernowster: paste?15:25
mnaserhttp://zuul.openstack.org/stream/70a09d0dd0bb4040b6a93945bccd2f07?logfile=console.log gate seems to be ok15:25
nowstermnaser: as before15:26
nowsterevrardjp: http://www.nowster.me.uk/work/openstack/bootstrap-ansible.log.txt  and  http://www.nowster.me.uk/work/openstack/bootstrap-aio.log.txt15:26
mnaseroh um15:26
mnasermy fix doesnt actually touch that15:27
nowsterSorry!15:27
mnaserthis is only for nspawn deployments15:27
mnaserlooks like you have a normal lxc one15:27
nowsterThought it did.15:27
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: [docs] Clean up the AIO user story  https://review.openstack.org/61952215:27
mnaserif it's creating those devices, then it's not nspawn15:27
mnaserbr-<etc>15:27
admin0i am eager to test when its working .. currently have 4 clusters in a hybrid  lb and ovs mode15:28
admin0looking forward to move to nspawn - ovs15:28
mnaser:)15:31
*** gkadam has quit IRC15:51
evrardjpadmin0: I guess the beginning will be nspawn lxb :p15:52
*** cshen_ has quit IRC15:52
admin0should be a good test ..16:00
mgariepyanyone knows if there is an implementation client of neutron port_forwarding api ? https://review.openstack.org/#/q/topic:bp/port_forwarding16:01
mnasermgariepy: https://review.openstack.org/#/c/616710/ kinda.16:12
mgariepyok16:16
mgariepynice to see that they just code the server side ready for the release. ..16:17
openstackgerritMerged openstack/openstack-ansible-os_gnocchi stable/rocky: coordination_url should be placed in DEFAULT section  https://review.openstack.org/61698816:25
*** lbragstad has joined #openstack-ansible16:28
*** cshen has joined #openstack-ansible16:29
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/queens: Disable nested virt  https://review.openstack.org/61961016:34
*** Emine has joined #openstack-ansible16:35
*** dcdamien has quit IRC16:36
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/pike: Fix ansible-sshd version  https://review.openstack.org/61899516:36
*** lbragstad has quit IRC16:39
eumel8evrardjp: still around?16:46
evrardjpI am16:46
evrardjpfor about 14 minutes16:46
evrardjphow can I help?16:46
evrardjplet's encrypt?16:46
eumel8yes16:46
eumel8what you mean with yaml notation?16:47
*** cshen has quit IRC16:47
evrardjpjust replacing the foor=bar inline with foo: bar16:47
evrardjpwith the proper indentation16:47
evrardjpit's easier to read16:47
evrardjpbut that's no big deal16:47
eumel8ah, got it, thx16:48
*** udesale has quit IRC16:48
eumel8and regarding haproxy_ssl_fqdn_public you think it's an ip-address, right?16:48
odyssey4meif any cores are around to peek at https://review.openstack.org/619522 I'd appreciate it16:51
odyssey4meit's a docs change to make the AIO documentation make a bit more sense16:52
openstackgerritFrank Kloeker proposed openstack/openstack-ansible-haproxy_server master: Add feature Letsencrypt SSL certification  https://review.openstack.org/58677416:53
eumel8too late ;)16:54
noonedeadpunkfolks, what is the right way to get some custom package to be installed on compute/management nodes? I need some extra package for monitoring, but it seems, that they all try to get installed from repo container (due to pip.conf)16:54
noonedeadpunkI mean python package:)16:54
*** ThiagoCMC has quit IRC16:59
*** cshen has joined #openstack-ansible17:03
odyssey4menoonedeadpunk you can use your own playbook, and just use the '--isolated' pip install flag to make it ignore pip.conf17:03
odyssey4mehopefully soon I'll be able to get rid of pip.conf so this is no longer an issue17:03
noonedeadpunkodyssey4me: great tip, thanks!17:04
*** shardy has quit IRC17:05
noonedeadpunkI think it will become possible once all roles will move to python_venv_build role, am I right?17:05
odyssey4menoonedeadpunk yeah, that'll make it much, much easier to do more customised things per role17:07
*** cshen has quit IRC17:07
odyssey4meAll the roles are already there, but I need to get rid of repo_build from the process so that each role independently builds.17:08
odyssey4meI've just got to make the time to get one role's implementation done - the rest can just follow the pattern.17:09
odyssey4meI'm gonna try and work on that tomorrow.17:09
mnaserodyssey4me: i think part of things breaking was the fact that network configs were so divergant17:11
mnaseri added work converging them into the nspawn patch and its improved a lot of things i think17:12
odyssey4memnaser ah yeah, changing up the networking to be much simpler was something worked on later by cloudnull17:12
mnaserit was confusing to debug and a lot of things were mismatching being nspawn specific17:12
mnaseri think the most recent issue left now is the fact that before we used tos etup this fake eth12/eth13 for flat/vlan networks17:12
mnaserwhich no longer is there17:13
*** DanyC has quit IRC17:13
*** cshen has joined #openstack-ansible17:13
mnaserInterface eth12 for physical network flat does not exist. Agent terminated!17:13
mnaseri wonder if you can put a macvlan interface in a bridge17:13
noonedeadpunkThat sounds really amazing. So do you need any help with this?17:13
*** DanyC has joined #openstack-ansible17:13
odyssey4meah ok - well, if we can move away from all these fake interfaces I think that'll be a move towards a better world17:13
odyssey4memnaser even if the nspawn config diverges - I think that's ok - as long as it's moving towards something simpler and more understandable17:14
mnaserodyssey4me: well it was a lot of things being duplicated, right now i moved it to a single file with jinja if's when things are different17:15
mnaserwhich imho is a lot easier17:15
*** vnogin has joined #openstack-ansible17:16
odyssey4menoonedeadpunk if you're asking me if I need help with the python build simplification, then yes - once we have one role worked out, then I'll need help replicating the pattern across other roles and working out any issues that come out from it... because every time I push up a patch to change this, the house of cards falls apart17:16
*** vnogin has quit IRC17:16
odyssey4memnaser hah! I actually had started getting that done in another patch... so I'm glad you're doing it17:17
mnaserright now it's failing with this mapping17:17
mnaserInterface mappings: {'flat': 'eth12', 'vlan': 'eth0'}17:17
mnaserfor flat, i can put it on mv-flat which can solve it17:17
mnaserbut for the vlan, i guess i'll have to try to find a way to use the extra networks or whatever :X17:17
*** DanyC has quit IRC17:17
admin0noonedeadpunk, what monitoring are you using ?17:18
*** cshen has quit IRC17:18
*** mma has quit IRC17:18
mnaserdoes anyone know off the top of their mind if we use vlan networks in aio..17:18
noonedeadpunkodyssey4me: Yeah, I see. So you may ping me, and I may help in replicating change across roles to get this routine out of you, so that you have more time for another great tasks:)17:19
mnaseroo wait, hmm17:19
mnasernow i know what happens17:19
odyssey4memnaser It doesn't really matter if we do or don't. We can set new terms - whatever the terms you're better able to maintain.17:19
noonedeadpunkadmin0: currently we're running zabbix. Actually I don't know whether it's the best solution, but I get used to it17:19
mnaserneutron creates a bridge and gets eth0 as the interface mapping17:19
mnaserand then when it creates that network/routers/whatever17:20
mnaserit creates a bridge with eth0 on it17:20
mnaserwhich probably was killing connectivity17:20
* mnaser ponders17:20
odyssey4meah, that makes sense17:20
odyssey4meIt does perhaps make better sense to give neutron a fake bridge to work with, rather than eth0.17:20
mnaseri guess thats why eth12/eth13 are useful17:21
mnaserbecause we can build a seperate network there17:21
odyssey4mewell, if we name it better then it'll be more understandable17:21
mnaseryeah, we need to create fake devices regardless17:21
jrosseri think it has to be an interface17:21
mnaseri'm just kinda thinking that we should actually build an entire intenral network not touching eth0 .. ever17:22
odyssey4meperhaps something we can do is just describe how it's setup in the AIO user story, so that people can understand17:22
mnaserlike create a fake interface and build everything on top of that, and just literally host the public vip on eth0 and nothing more17:23
odyssey4methat's kinda how the AIO works now, doesn't it?17:23
mnaserwell not in nspawn because it piggybacks on eth017:23
mnaserall the macvlans are created on eth017:23
odyssey4meok, I think the LXC setup is like that17:24
mnaseryep, indeed it looks like it17:24
mnaserso i should do some nspawn-y cleanups17:24
odyssey4mebut ok - lesson learned, the nspawn changes was trying to simplify it more - but clearly we need the fake device to hang off for gating purposes17:24
*** chhagarw has quit IRC17:25
odyssey4meif we can document why we did it this way, this time, then we won't waste time in the future trying to do it another way again17:25
mnaseryeah, we can do all that, but off 1 single fake device17:25
mnaserimho i would aim to replicate lxc just to make it work first17:26
mnaserand then we can simplify17:26
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-plugins master: Add group_vars to py_pkgs lookup  https://review.openstack.org/46629517:32
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-plugins master: Add group_vars/all path to py_pkgs lookup  https://review.openstack.org/46629517:34
*** Emine has quit IRC17:34
*** mma has joined #openstack-ansible17:35
*** DanyC has joined #openstack-ansible17:36
*** mma has quit IRC17:38
*** DanyC has quit IRC17:41
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: Move git SHA pins to group_vars  https://review.openstack.org/61962917:45
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-plugins master: Add group_vars/all path to py_pkgs lookup  https://review.openstack.org/46629517:45
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: Move git SHA pins to group_vars  https://review.openstack.org/61962917:46
openstackgerritMohammed Naser proposed openstack/openstack-ansible master: WIP: Enable passing nspawn jobs  https://review.openstack.org/60843317:47
mnaserthis patch is slowly getting bigger and bigger but i'm getting more comfortable with nspawn17:47
odyssey4memnaser once you have something that works you can break the patch down into smaller parts that are easier to review17:47
mnaserodyssey4me: yeah i think initially i thought it was going to be relatively simple but it has proven otherwise17:48
mnaserbut indeed ill be slowly splitting them up once i got a fully working thing17:48
odyssey4meI've found I need to do that a lot lately - get a working WIP, then figure out how to break it down into more digestable and more easiluy iterable parts.17:48
mnasersome stuff i can already split up now anyawys17:49
mnaserlike https://review.openstack.org/#/c/608433/13/playbooks/common-tasks/os-nspawn-container-setup.yml17:49
odyssey4mesure, but I also find that sometimes you end up uhaving to go back, because you find something new17:49
odyssey4mebut yeah, if something is a sure shot you may as well get a fresh pr pushed up to get it in already for anyone trying it out17:50
mnaseryeah i'll probably split if there are unrelated things that clearly could use a fix17:51
odyssey4meI'm out for the night - cheers folks.17:51
mnasertake care odyssey4me  -- thanks for the help today17:51
*** mmercer has quit IRC17:54
*** cshen has joined #openstack-ansible17:56
*** hamzaachi has joined #openstack-ansible17:58
*** cshen has quit IRC18:01
*** vnogin has joined #openstack-ansible18:04
*** jamesdenton has joined #openstack-ansible18:04
*** jawad_axd has joined #openstack-ansible18:32
*** jawad_axd has quit IRC18:36
*** frippe75 has joined #openstack-ansible18:40
*** mma has joined #openstack-ansible18:43
*** rpittau has quit IRC18:46
*** mma has quit IRC18:48
*** cshen has joined #openstack-ansible18:48
*** electrofelix has quit IRC18:53
*** speedmann has quit IRC18:54
openstackgerritJonathan Rosser proposed openstack/openstack-ansible master: Refactor ceph server playbooks  https://review.openstack.org/61114619:03
*** vnogin has quit IRC19:04
*** Emine has joined #openstack-ansible19:08
*** Emine has quit IRC19:48
*** rmart04 has joined #openstack-ansible19:53
*** dcdamien has joined #openstack-ansible19:57
openstackgerritMohammed Naser proposed openstack/openstack-ansible master: WIP: Enable passing nspawn jobs  https://review.openstack.org/60843320:02
openstackgerritMohammed Naser proposed openstack/openstack-ansible master: nspawn: correct task names  https://review.openstack.org/61963620:02
*** rmart04 has left #openstack-ansible20:16
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-rabbitmq_server master: Add support for arm64 on ubuntu  https://review.openstack.org/61963720:20
openstackgerritMarc Gariépy (mgariepy) proposed openstack/openstack-ansible-os_neutron master: Add port_forwarding to neutron configuration.  https://review.openstack.org/61964020:40
*** hamzaachi has quit IRC20:46
*** cshen has quit IRC21:09
*** cshen has joined #openstack-ansible21:11
*** ivve has quit IRC21:12
*** dcdamien has quit IRC21:45
*** lbragstad has joined #openstack-ansible21:54
*** cshen has quit IRC22:25
*** ahosam has joined #openstack-ansible22:47
*** ahosam has quit IRC22:57
*** jawad_axd has joined #openstack-ansible23:03
*** jawad_axd has quit IRC23:08
dmsimardthe OSA docs are good23:18
dmsimardgreat work :)23:18
*** lbragstad has quit IRC23:23
*** tosky has quit IRC23:31

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!