Thursday, 2019-06-27

*** openstack has joined #openstack-ansible13:14
*** ChanServ sets mode: +o openstack13:14
jrosserit's another sitation where we don't want to end up with a split pattern across the roles13:14
evrardjpI dropped the hammer13:16
evrardjpThis is indeed not a direction I would like this project to go to.13:17
evrardjpI would be fine with having a rhel file, and changing the order the same way everywhere.13:17
*** schwicht has joined #openstack-ansible13:20
*** schwicht has quit IRC13:20
*** schwicht has joined #openstack-ansible13:21
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-os_ironic master: Add integrated coverage  https://review.opendev.org/66008313:22
mnaseri don't get what the big issue is with that change13:25
mnasera -2 means you are hard blocking that patch13:25
noonedeadpunkjrosser: Is there a way to copy CA certificates inside containers (I'm regarding https://review.opendev.org/#/c/641445/)? I feel like there should be separate patch for copying isnide containers?13:26
mnaserwe did all sorts of workarounds to get opensuse 42 to work only to eventually drop it13:26
openstackgerritMerged openstack/openstack-ansible-os_tempest master: Updated from OpenStack Ansible Tests  https://review.opendev.org/66787313:26
mnaseris the issue here that we're "doing things for rhel 8 specifically"13:27
jrossernoonedeadpunk: it does that already, becasue openstack_hosts also targets the contaimers13:28
mnaserand the layered stuff is neat too tbh because it takes a common base and puts things on top of it13:28
mnaserit allows us to maintain a common base + overrides per specific os13:28
evrardjpmnaser: I am not against thel 8, I am all for it.13:29
mnaseri actually like the layering approach13:29
evrardjpthe layering is a bad idea as it's hard to track13:29
mnaserand given that it's taken a month to land a single patch for this, i think that reflects poorly from us to throw a -2 after working 2 months to work on it.13:29
mnaserit'13:29
mnaserit'll just be easier for them to just maintain their own role and call it a day13:29
evrardjpI am opposed as this being a snowflake where ppl can do what they want13:30
evrardjpI am fine with it if we define it's the new standard13:30
mnaseri like this approach.  i think it's useful for a few scenarios (and it's not as confusing as you mention imho).  i don't think it's fair for me to tell someone 'if you want to change this, go and fix 30 other roles'13:31
evrardjpso you think it's the way to go for the project to have all the roles as snowflakes?13:31
mnaserthey already are13:31
mnaserwe're slowly but surely trying to add more automation like the mq_setup and db_setup to un-snowflake them13:31
evrardjpthis doesn't explain WHY we need this,  why we need the ordering change13:31
evrardjpthis looks by far more complex than just adding a var file13:32
mnaser"Variables are loaded based on granularity, the more specific ones overriding the more generic ones. This allowing us to declare overrides in a minimal number of places."13:32
evrardjpit also now silently allow breakages13:32
mnaseri.e. redhat.yml contains tempest_distro_packages and redhat-8.yml doesn't13:32
mnaserwhich means less work when redhat-9 comes if tempest_distro_packages stays the same for example13:32
mnaserim not sure how it silently allows breakages if we're testing the entire set of supported operating systems13:33
evrardjpI don't see why you can't have that with redhat13:33
evrardjp.yml13:33
evrardjpthat's what we always did13:33
mnaserif we're testing rh7/rh8/ubuntu16/ubuntu1813:33
openstackgerritMerged openstack/ansible-role-python_venv_build master: Updated from OpenStack Ansible Tests  https://review.opendev.org/66788613:33
noonedeadpunkI don't think that it's good idea to devide for redhat and redhat-813:33
evrardjpand if some specific package was different, we used a conditional inside the var file13:33
noonedeadpunkAs it's not explicit13:33
evrardjpanyway13:33
evrardjplet's discuss it first13:34
evrardjpbefore defining a new standard13:34
openstackgerritMerged openstack/openstack-ansible-os_zun master: db_setup: refactor database setup to a common file  https://review.opendev.org/66707413:34
mnaseri think throwing a -2 after a few people have been working on a patch for 1 month without us doing enough reviews reflects poorly of us.  we either speak early instead of having someone iterate on things for a month before telling them you're wrong13:34
mnaseranyways.13:34
mnaseri don't think this is fair so i'm actually going to change this up to work for rhel-8 based on that change13:35
mnaserand i hope we can land that, then we can discuss the change13:35
martinkennellyHi all, is DPDK integration into neutron available in OSA? I see this patch https://review.opendev.org/#/c/590810 but its not merged.13:35
mnaserand i hope we can review it quickly13:35
evrardjpI can remove my vote if you feel it's better for the project.13:35
mnaseri just think it's pretty crappy when someone is trying to land something for a month and then they're told "nope" after 11 patchsets13:35
evrardjpI haven't seen this in the ML or in the meetings13:36
mnaserwe should either speak earlier, or accept that we'd "stayed silent" for too long and if its not terrible .. its ok13:36
jrosserso the reason i asked evrardjp to look at this was we had a wierd blow-up this week to to vars inclusion13:37
evrardjpjrosser: this will just get amplified13:37
jrosserwhere if role A includes role B, and B tries to include vars/foo.yml, that can actually come from A/vars/foo.yml13:38
evrardjpwe'll now have no failures, it's gonna be silently loading wrong vars.13:38
evrardjpthat will be amazing.13:38
noonedeadpunk+1 with that13:38
evrardjpBut hey, let's not make a wrong message to ppl13:38
mnaserdoes anyone wanna volunteer helping the tripleo team in landing this by making it something we 'accept'13:39
evrardjpI proposed solutions in my review!13:39
evrardjpI don't -2 without reasons!13:39
evrardjpI think we need to discuss this all together, and understand the risks13:39
jrosserso a very practical example here is that redhat-8.yml will get included from the parent role if it doesnt exist in an included role13:40
evrardjpif we do this, we can kiss goodbye the assert for checking the vars are properly loaded13:40
evrardjpoh wait, maybe we can13:41
noonedeadpunkAnyway I'm not sure that such implicit things are great....13:42
jrossermartinkennelly: you are right the DPDK stuff is not merged, jamesdenton may be able to advise on that13:42
openstackgerritMerged openstack/openstack-ansible-os_trove master: Updated from OpenStack Ansible Tests  https://review.opendev.org/65804213:42
evrardjpalso there is a way different than this patchset (which I proposed as an alternative), to do the same behaviour13:42
noonedeadpunkIt also may result in decreasing redability13:42
martinkennellyThanks jrosser!13:42
evrardjpmake the loop a var, and from that, make our lookup first found the default13:42
jrossermartinkennelly: as i remember it is exceedinly difficult to automate this and correcly deal with NUMA setup13:42
evrardjpppl can then adapt to what they want13:43
*** dave-mccowan has quit IRC13:43
openstackgerritMohammed Naser proposed openstack/openstack-ansible-os_tempest master: Update the list of python3 packages  https://review.opendev.org/66199413:43
jamesdentonhi martinkennelly - those patches need to be rebased, at a minimum, and probably restructured. Unfortunately, I'm short on time.13:44
martinkennellyjrosser: I see. Ill have a read through the discussions surrounding this now. Thanks.13:44
martinkennellyjamesdenton: thanks for the update!13:45
mnaserevrardjp: jrosser noonedeadpunk please check 66199413:45
*** pcaruana has quit IRC13:46
jrossermartinkennelly: perhaps take a look at this https://www.openstack.org/videos/summits/denver-2019/pushing-packets-how-do-the-ml2-drivers-stack-up13:46
*** pcaruana has joined #openstack-ansible13:47
martinkennellyjrosser: Cheeers, will take a look:)13:47
*** CeeMac has quit IRC13:58
evrardjpmnaser: better to isolate that into smaller chunks. Now we can tackle the package installation and the purpose of this patch.14:00
*** ohwhyosa has joined #openstack-ansible14:01
*** spatel has joined #openstack-ansible14:01
guilhermespmornings14:02
ohwhyosamorning!14:02
guilhermespthanks for the eyes on db-refactor stuff! we are almost there, just a few things to fix and we have our db stuff refactored!14:02
spatelmorning folks14:03
spateli need help here - https://bugs.launchpad.net/openstack-ansible/+bug/183439414:04
openstackLaunchpad bug 1834394 in openstack-ansible "os_octavia : Setup the service project quota endpoint for block-storage service in RegionOne region not found" [Undecided,In progress] - Assigned to Jonathan Rosser (jrosser)14:04
ohwhyosahey guys, I tried with an openstack-ansible stein stable with pretty much everything default except for novnc instead of spice, and the console still throws an invalid token and does not work14:04
ohwhyosahttps://pastebin.com/qq5GkxSJ14:04
ohwhyosa^--> That error14:04
jrosserspatel: see this https://review.opendev.org/#/c/664551/14:05
openstackgerritMerged openstack/openstack-ansible-os_trove master: db_setup: refactor database setup to a common file  https://review.opendev.org/66574114:05
jrosserspatel: i put tons of effort currently into fixing the octavia CI job, there are lots of fixes need merging14:05
spateljrosser: I did that but if you look at bug its something else going on14:05
openstackgerritMerged openstack/openstack-ansible-os_rally master: db_setup: refactor database setup to a common file  https://review.opendev.org/66573714:05
spatelos_quota module looking for block-storage which i don't have14:06
ohwhyosahi jrosser! forgot to tell you, thanks for the review on gerrit!14:06
guilhermespjrosser: do you have a topic for those fixed so we can be reviewing ?14:07
guilhermespfixes**14:07
jrosserspatel: i don't see that in an AIO here?14:07
jrosserguilhermesp: let me just do that14:08
spateljrosser: I am not building AIO, i am building it on production 3 node controller. i did testing in AIO and it works because AIO build Cinder.14:08
jrosserspatel: ok - and do you have cinder in you prod deployment?14:09
spatelNO14:10
spatelI don't use cinder.14:10
spateljrosser: i found this one during googling.. https://github.com/ansible/ansible/issues/4124014:14
spatelThey are saying this bug has been fixed but look like not.14:14
spatelI did compare patch files and look like i do have all these patches in place - https://github.com/ansible/ansible/pull/45775/files14:15
jrosserspatel: out of interest how is your storage working?14:15
spatelI have ceph storage for nova so my vms use for disk, i don't need block-storage.14:16
openstackgerritMerged openstack/openstack-ansible-os_placement master: db_setup: refactor database setup to a common file  https://review.opendev.org/66573314:16
openstackgerritMerged openstack/openstack-ansible stable/stein: Revert "Set CentOS 7 distro job to non-voting"  https://review.opendev.org/66519214:16
jrosserwe'd normally be using ceph rbd via cinder - sounds like you have a configuration i'm not familiar with14:17
spatelmy current cloud running without cinder, and ceph storage for my vms disk, last 1 year my cloud is running like that14:20
mgariepyyou are running ephemeral vms on ceph.14:20
spatelCinder is just to add extra piece of storage and i don't use it because we don't need disk, all we need CPU / memory for out business14:21
spatelmgariepy: Yes we can say that14:21
mgariepycinder is for more ""persistant"" disk.14:21
spatelYes cinder is for presistant14:21
spatelwe don't care about data truly speaking.. we are streaming company and just stream data using CPU and memory14:22
spatelthat is why i didn't deploy cinder to reduce complexity and make it more simple.14:23
jrosserguilhermesp: i'm just running an aio_metal_octavia here to see where all the RAM gets chewed up14:23
guilhermespnice jrosser I haven't had time to test it locally, but I have added a depends-on here https://review.opendev.org/#/c/664888/ but seems to be breaking on debian too14:25
jrosserguilhermesp: ah we don't wind down the octavia threads/processes on any octavia tests it seems14:25
jrosserlet me patch that14:25
guilhermespcool jrosser14:25
jrosserwas the same for placement14:25
guilhermespthat one right ? https://review.opendev.org/#/c/667898/14:27
guilhermespmaybe this patech is in a wrong topic?14:28
openstackgerritMerged openstack/openstack-ansible-os_placement master: Fix aio_distro_metal jobs for openSUSE  https://review.opendev.org/66629814:32
jrosserguilhermesp: yes thats it, well i just put in the same topic everything i have on the go for reducing the AIO memory footprint14:33
guilhermespfair enough jrosser14:33
guilhermespi did my review on those that are +114:33
guilhermespreviews*14:33
* guilhermesp typing crazy always 14:34
jrosserthankyou - i've still not found a combination of all of those that make octavia pass yet so theres still some more fiddling to do14:34
guilhermespfeels like we are in the right path to make it work14:34
openstackgerritJonathan Rosser proposed openstack/openstack-ansible master: Restrict the number of octavia processes and threads in AIO  https://review.opendev.org/66794114:37
*** vnogin has joined #openstack-ansible14:37
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-os_nova master: Install netcat on compute hosts  https://review.opendev.org/66675914:59
*** ivve has quit IRC15:00
*** CeeMac has joined #openstack-ansible15:01
*** hamzy has quit IRC15:13
jrosseroh wow introducing placement service and not capping the processes/threads cost us 1.1GB of ram15:14
*** pcaruana has quit IRC15:14
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-os_ironic master: Use ironic credentials when connecting to other APIs.  https://review.opendev.org/66795715:14
evrardjpjrosser: if there is ram, let's use it! :p15:16
jrosseryeah, till the tests fail :)15:16
evrardjp:D15:16
mnaserevrardjp: you're nitpicking now.15:16
mnaserall the change does is make things work for rhel 8 at this point15:16
*** cshen has quit IRC15:17
mnaserand the rescue doesnt hurt anyone except make it useful because clearly15:17
mnaserthere is a note that says15:17
evrardjpDid I miss something? It's possible... I am just human15:17
mnaser"workaround for cryptic ansible dnf error: MarkingError: no package matched which may hide the packages names."15:17
mnaserwhich makes it impossible to know which ones were attempted to be installed.15:17
openstackgerritMerged openstack/openstack-ansible-openstack_hosts master: Install user supplied CA certificates into system trust store  https://review.opendev.org/64144515:20
openstackgerritMohammed Naser proposed openstack/openstack-ansible-os_tempest master: Update the list of python3 packages  https://review.opendev.org/66199415:22
openstackgerritMohammed Naser proposed openstack/openstack-ansible-os_tempest master: Update the list of python3 packages  https://review.opendev.org/66199415:23
openstackgerritMohammed Naser proposed openstack/openstack-ansible-os_tempest master: Update the list of python3 packages  https://review.opendev.org/66199415:23
jrosserand a further 1.4G saved by turning down the octavia threads/processes15:23
openstackgerritMohammed Naser proposed openstack/openstack-ansible-os_tempest master: Update the list of python3 packages  https://review.opendev.org/66199415:24
mnaser^ https://review.opendev.org/#/c/661994/1615:24
mnasercan we please land this now?15:24
*** udesale has joined #openstack-ansible15:25
*** kopecmartin is now known as kopecmartin|off15:28
*** ohwhyosa has quit IRC15:28
jrosserright this https://review.opendev.org/#/c/667898/ and this https://review.opendev.org/#/c/667941/ together knock 2.5G off the memory footprint on octavia tests - that should hopefully unwedge that role15:29
mnaser jrosser thanks for your efforts15:30
jrossermnaser: no worries - whilst i have a test env up i'll see if theres anything else chewing up a bunch of ram15:30
mnaserjrosser: this is probably a good thing to look into generalizing across roles15:31
mnaserjrosser: i wonder if defaults/workers.yml could be a thing we sync across roles15:31
jrossercould be - right now there is a huge user_variables.yml.j2 that this all ends up in15:31
mnaseryeah, and then we can have some sort of default in the integrated15:31
mnaserso all roles can use the same 'base' setting15:31
mnasercause right now we have to override it in every single one which is a bit..... eh15:32
jrosserthough the settings we have in the role defaults are for prod rather than CI15:32
jrosserwe could reverse that15:32
mnaseryea but even then, we don't even have them in a way where you can configure it everywhere at once15:32
mnaseri.e. you have to configure # of workers for every single role15:32
*** luksky has quit IRC15:34
*** chandankumar is now known as raukadah15:34
*** hamzy has joined #openstack-ansible15:40
spatelreboot15:41
spatelsorry wrong window15:41
*** vnogin has quit IRC15:45
evrardjpmnaser: I wasn't sure if you had a planning for OSA in shanghai, so I copy pasted the last convos/todo into that: https://etherpad.openstack.org/p/osa-shanghai-summit-planning15:51
noonedeadpunkevrardjp: do you have plans to visit it?15:57
*** vnogin has joined #openstack-ansible16:15
openstackgerritMerged openstack/openstack-ansible-galera_server master: Replace opensuse 42.3 functional testing with opensuse-15  https://review.opendev.org/66768316:16
*** vnogin has quit IRC16:20
*** luksky has joined #openstack-ansible16:30
openstackgerritCraig McIntyre proposed openstack/openstack-ansible-os_glance master: Add missing pre-reqs for glance cinder store  https://review.opendev.org/66770416:38
guilhermespjrosser: maybe sahara is also running into the same issue as octavia... https://review.opendev.org/#/q/project:openstack/openstack-ansible-os_sahara16:46
jrosserYou can see it easily in th nova compute log, failure to allocate memory16:48
*** gkadam has quit IRC16:51
*** udesale has quit IRC16:54
*** hamzaachi has joined #openstack-ansible17:02
*** kaiokmo has quit IRC17:04
*** hamzy has quit IRC17:07
*** kaiokmo has joined #openstack-ansible17:07
*** hamzy has joined #openstack-ansible17:12
*** martinkennelly has quit IRC17:13
*** schwicht has quit IRC17:18
*** noonedeadpunk has quit IRC17:18
*** weifan has joined #openstack-ansible17:22
*** rholloway has joined #openstack-ansible17:29
*** rholloway has quit IRC17:34
*** hamzaachi has quit IRC17:49
*** schwicht has joined #openstack-ansible17:53
*** electrofelix has quit IRC17:53
*** pcaruana has joined #openstack-ansible17:54
openstackgerritDirk Mueller proposed openstack/openstack-ansible-os_tempest master: Switch to openSUSE Leap 15.1  https://review.opendev.org/66730218:04
openstackgerritDirk Mueller proposed openstack/openstack-ansible-os_neutron master: Switch to openSUSE Leap 15.1  https://review.opendev.org/66730318:04
*** hamzy has quit IRC18:24
*** hamzy has joined #openstack-ansible18:25
guilhermespyep jrosser http://logs.openstack.org/38/665738/1/check/openstack-ansible-deploy-aio_metal_sahara-debian-stable/69a0c6a/logs/openstack/aio1-nova/nova-conductor.log.txt.gz#_2019-06-22_06_01_24_11218:30
*** hamzy has quit IRC18:30
mgariepyjrosser, endpoint group filter aren't really great for now ;)18:30
mgariepylots of broken bits here and there18:31
guilhermespso jrosser I assume we'd do the same here for sahara https://review.opendev.org/#/c/667898/1/tests/roles/bootstrap-host/templates/user_variables.aio.yml.j218:35
*** weifan has quit IRC18:38
spatelAny idea - http://paste.openstack.org/show/753522/18:41
spatelI did try to upgrade module but same error  - pip install --upgrade decorate18:41
openstackgerritGuilherme  Steinmuller Pimentel proposed openstack/openstack-ansible master: Restrict the number of sahara processes and threads in AIO  https://review.opendev.org/66801918:44
openstackgerritGuilherme  Steinmuller Pimentel proposed openstack/openstack-ansible-os_sahara master: db_setup: refactor database setup to a common file  https://review.opendev.org/66573818:45
openstackgerritGuilherme  Steinmuller Pimentel proposed openstack/openstack-ansible-os_octavia master: db_setup: refactor database setup to a common file  https://review.opendev.org/66488818:47
*** hamzaachi has joined #openstack-ansible18:47
spatelcrap openstack document need to update, i did pip install --ignore-installed python-openstackclient  and that works18:47
*** weifan has joined #openstack-ansible19:18
*** hamzaachi has quit IRC19:20
*** weifan has quit IRC19:31
*** weifan has joined #openstack-ansible19:31
*** raub has joined #openstack-ansible19:31
*** hamzaachi has joined #openstack-ansible19:35
*** spatel has quit IRC19:37
*** kplant has quit IRC19:39
*** weifan has quit IRC19:51
*** weifan has joined #openstack-ansible19:52
*** weifan has quit IRC19:56
*** weifan has joined #openstack-ansible19:59
*** ivve has joined #openstack-ansible20:12
*** weifan has quit IRC20:16
*** ansmith_ has quit IRC20:19
*** weifan has joined #openstack-ansible20:20
*** weifan has quit IRC20:27
*** weifan has joined #openstack-ansible20:30
openstackgerritCraig McIntyre proposed openstack/openstack-ansible-os_glance master: Add missing pre-reqs for glance cinder store  https://review.opendev.org/66770420:32
jrosserhrm [OBS:Cloud:OpenStack:Master|http://mirror.iad.rax.opendev.org/opensuse/repositories/Cloud:/OpenStack:/Master/openSUSE_Leap_15.0] Valid metadata not found at specified URL20:32
openstackgerritMerged openstack/openstack-ansible-os_tempest master: Update the list of python3 packages  https://review.opendev.org/66199420:37
*** weifan has quit IRC20:45
*** weifan has joined #openstack-ansible20:47
*** weifan has quit IRC20:48
*** weifan has joined #openstack-ansible20:48
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-os_octavia master: Ensure that admin endpoint is used to configure octavia quota  https://review.opendev.org/66455120:51
*** weifan has quit IRC20:53
*** weifan has joined #openstack-ansible20:54
*** hamzaachi has quit IRC20:58
*** pcaruana has quit IRC21:05
*** ansmith_ has joined #openstack-ansible21:06
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-os_octavia master: Use the admin endpoint to upload glance images  https://review.opendev.org/66779121:09
openstackgerritDirk Mueller proposed openstack/openstack-ansible-os_heat master: Switch to openSUSE Leap 15.1  https://review.opendev.org/66805521:23
openstackgerritDirk Mueller proposed openstack/openstack-ansible-os_cinder master: Switch to openSUSE Leap 15.1  https://review.opendev.org/66805621:26
openstackgerritDirk Mueller proposed openstack/openstack-ansible-os_keystone master: Switch to openSUSE Leap 15.1  https://review.opendev.org/66805721:28
openstackgerritDirk Mueller proposed openstack/openstack-ansible-os_glance master: Switch to openSUSE Leap 15.1  https://review.opendev.org/66805821:30
Nick_Ahow would I modify lxc config with openstack-ansible? running lxd computes with nova-lxd21:56
*** schwicht has quit IRC22:29
*** weifan has quit IRC22:53
openstackgerritMerged openstack/openstack-ansible-os_zun master: Replace git.openstack.org URLs with opendev.org URLs  https://review.opendev.org/65727022:54
*** kplant has joined #openstack-ansible22:59
*** tosky has quit IRC23:00
*** luksky has quit IRC23:00
*** spatel has joined #openstack-ansible23:13
*** spatel has quit IRC23:14
openstackgerritMerged openstack/openstack-ansible-os_ironic master: Use ironic credentials when connecting to other APIs.  https://review.opendev.org/66795723:32
*** kplant has quit IRC23:41
openstackgerritMerged openstack/openstack-ansible master: Add aio_metal and aio_distro_metal openSUSE 15 jobs  https://review.opendev.org/66605823:43

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!