| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Add Redhat to the ELK deployment capabilities https://review.openstack.org/591467 | 00:00 |
|---|---|---|
| *** dxiri has quit IRC | 00:29 | |
| *** vollman has quit IRC | 00:42 | |
| *** nicolasbock has quit IRC | 00:45 | |
| *** vnogin has joined #openstack-ansible | 00:45 | |
| *** flaviosr has quit IRC | 00:57 | |
| *** sep has quit IRC | 00:57 | |
| *** mhayden has quit IRC | 00:57 | |
| *** mubix has quit IRC | 00:57 | |
| *** mpjetta has quit IRC | 00:57 | |
| *** v1k0d3n has quit IRC | 00:57 | |
| *** storshoo has quit IRC | 00:57 | |
| *** nicolasbock has joined #openstack-ansible | 00:58 | |
| *** flaviosr has joined #openstack-ansible | 01:01 | |
| *** sep has joined #openstack-ansible | 01:04 | |
| *** mhayden has joined #openstack-ansible | 01:04 | |
| *** mubix has joined #openstack-ansible | 01:04 | |
| *** v1k0d3n has joined #openstack-ansible | 01:04 | |
| *** storshoo has joined #openstack-ansible | 01:04 | |
| *** sep has quit IRC | 01:04 | |
| *** openstackgerrit has quit IRC | 01:06 | |
| *** dxiri has joined #openstack-ansible | 01:52 | |
| *** dxiri has quit IRC | 01:57 | |
| logan- | whats the method to examine of the system journals in the gate logs | 02:00 |
| logan- | anytime i download one and use journalctl -D I get "Journal file test/system.journal is truncated, ignoring file." | 02:01 |
| *** vnogin has quit IRC | 02:10 | |
| cloudnull | logan- I think that's because of the gzip | 02:35 |
| cloudnull | many of the services should have the journals dumped | 02:35 |
| cloudnull | in a log file that ends in .journal.log | 02:35 |
| logan- | thanks | 02:37 |
| logan- | it is not gzipped though: 'file system.journal': system.journal: Journal file, online, compressed | 02:37 |
| logan- | if i rename to .gz and try to extract it gives gzip: system.journal.gz: not in gzip format | 02:38 |
| *** Bhujay has joined #openstack-ansible | 03:04 | |
| cloudnull | hum that is particular | 03:23 |
| cloudnull | I wonder if the journals are corrupt because we just wholesale upload it | 03:23 |
| cloudnull | would could export/dump the journals at the end of the job ? | 03:24 |
| *** Bhujay has quit IRC | 03:40 | |
| *** udesale has joined #openstack-ansible | 03:53 | |
| *** mmalchuk has quit IRC | 03:58 | |
| *** poopcat has quit IRC | 04:04 | |
| *** KeithMnemonic has quit IRC | 04:08 | |
| *** dave-mccowan has quit IRC | 04:12 | |
| *** Bhujay has joined #openstack-ansible | 04:21 | |
| *** gyee has quit IRC | 05:05 | |
| *** gregoryo has joined #openstack-ansible | 05:15 | |
| *** shyamb has joined #openstack-ansible | 05:51 | |
| jrosser | cloudnull: ive also totally failed to download/wxamine journals like that | 05:53 |
| *** shyamb has quit IRC | 05:57 | |
| *** mmalchuk has joined #openstack-ansible | 05:58 | |
| *** shyamb has joined #openstack-ansible | 06:03 | |
| *** odyssey4me has quit IRC | 06:14 | |
| *** odyssey4me has joined #openstack-ansible | 06:14 | |
| *** noonedeadpunk has quit IRC | 06:15 | |
| *** DanyC has joined #openstack-ansible | 06:27 | |
| evrardjp | good morning | 06:29 |
| *** DanyC_ has joined #openstack-ansible | 06:30 | |
| hwoarang | good morning | 06:32 |
| *** DanyC has quit IRC | 06:33 | |
| *** DanyC_ has quit IRC | 06:37 | |
| *** pcaruana has joined #openstack-ansible | 06:44 | |
| jrosser | morning | 06:44 |
| *** shyamb has quit IRC | 06:48 | |
| *** shyamb has joined #openstack-ansible | 06:53 | |
| *** cshen has joined #openstack-ansible | 06:53 | |
| *** fghaas has joined #openstack-ansible | 06:56 | |
| *** madorn has quit IRC | 06:57 | |
| *** madorn has joined #openstack-ansible | 06:58 | |
| *** openstackgerrit has joined #openstack-ansible | 07:12 | |
| openstackgerrit | wutao proposed openstack/openstack-ansible-os_sahara master: Drop un-used packages from role https://review.openstack.org/591569 | 07:12 |
| *** hamzaachi has joined #openstack-ansible | 07:24 | |
| *** shyamb has quit IRC | 07:27 | |
| *** shyamb has joined #openstack-ansible | 07:27 | |
| *** olivierbourdon38 has joined #openstack-ansible | 07:39 | |
| *** shyamb has quit IRC | 07:48 | |
| evrardjp | cloudnull: did you get the chance to look at the node? | 07:55 |
| evrardjp | your node* | 07:55 |
| *** DanyC has joined #openstack-ansible | 08:07 | |
| *** DanyC has quit IRC | 08:09 | |
| *** Bhujay has quit IRC | 08:10 | |
| *** DanyC has joined #openstack-ansible | 08:11 | |
| *** openstackstatus has quit IRC | 08:12 | |
| *** DanyC has quit IRC | 08:16 | |
| *** phasespace has joined #openstack-ansible | 08:24 | |
| *** DanyC has joined #openstack-ansible | 08:25 | |
| *** shyamb has joined #openstack-ansible | 08:28 | |
| *** Bhujay has joined #openstack-ansible | 08:34 | |
| openstackgerrit | Taseer Ahmed proposed openstack/openstack-ansible-os_blazar master: This patch ensures that the functional tests for Blazar pass. https://review.openstack.org/571388 | 08:40 |
| *** noonedeadpunk has joined #openstack-ansible | 08:49 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-lxc_hosts master: Allow user-defined extra distro packages https://review.openstack.org/591412 | 08:55 |
| openstackgerrit | Dmitriy Rabotjagov (noonedeadpunk) proposed openstack/openstack-ansible-lxc_container_create stable/queens: Implements possibility to use physical networks in containers. https://review.openstack.org/591590 | 08:58 |
| *** vnogin has joined #openstack-ansible | 09:02 | |
| *** electrofelix has joined #openstack-ansible | 09:04 | |
| openstackgerrit | Dmitriy Rabotjagov (noonedeadpunk) proposed openstack/openstack-ansible-os_horizon master: Implements custom theme distribution https://review.openstack.org/586318 | 09:11 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-lxc_hosts master: Allow user-defined extra distro packages https://review.openstack.org/591412 | 09:12 |
| *** gkadam has joined #openstack-ansible | 09:18 | |
| *** gregoryo has quit IRC | 09:19 | |
| *** shyamb has quit IRC | 09:24 | |
| hwoarang | evrardjp: odyssey4me we are seeing a failure in opnfv and i'd like another fresh pair of eyes to look at it if possible | 09:25 |
| hwoarang | we are almost on master, and we have nova failing at this point all the time | 09:26 |
| hwoarang | TASK [os_nova : Perform a cell_v2 discover] ******************************************************************************************************************************************** | 09:26 |
| hwoarang | fatal: [compute00]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname controller00: Name or service not known\r\n", "unreachable": true} | 09:26 |
| hwoarang | notice that we are executing on compute00 but ssh complains about controller00 | 09:26 |
| hwoarang | any clue on what could be wrong or where to look? | 09:26 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-lxc_hosts master: Allow user-defined extra distro packages https://review.openstack.org/591412 | 09:29 |
| hwoarang | actually on the previous task I see | 09:29 |
| hwoarang | TASK [os_nova : Set the delegated task facts] ****************************************************************************************************************************************** | 09:29 |
| hwoarang | ok: [compute00 -> 172.29.239.54] | 09:29 |
| odyssey4me | hwoarang: I've been seeing that too like here: http://logs.openstack.org/59/589259/4/check/openstack-ansible-functional-ubuntu-xenial/6e06008/job-output.txt.gz#_2018-08-13_11_33_38_567763 | 09:29 |
| odyssey4me | from https://review.openstack.org/#/c/589259/ | 09:30 |
| hwoarang | so looks like we are delegating from compute00 to controller00 and that does not work with hostname. not sure why it's not using its ip to connect | 09:30 |
| hwoarang | let me check that log | 09:30 |
| hwoarang | aha | 09:30 |
| jrosser | i've been fiddling with similar | 09:30 |
| jrosser | using ad-hoc debug command to get hostvars and ansible_hostname / ansible_fqdn has been useful | 09:31 |
| jrosser | then running ad-hoc setup module and see if anything changes/fixes itself | 09:31 |
| odyssey4me | I'm off today, so won't be in. Will catch up tomorrow to see if anyone's found anything useful. | 09:32 |
| evrardjp | odyssey4me: I am off tomorrow | 09:33 |
| evrardjp | just fyi | 09:33 |
| *** markvoelker has joined #openstack-ansible | 09:33 | |
| *** shyamb has joined #openstack-ansible | 09:34 | |
| *** openstackstatus has joined #openstack-ansible | 09:40 | |
| *** ChanServ sets mode: +v openstackstatus | 09:40 | |
| hwoarang | jrosser: do you think that this could be a problem with our connection plugin? | 09:49 |
| hwoarang | the old workaround for me is to add entries in /etc/hosts in the deployment host | 09:49 |
| hwoarang | *only | 09:50 |
| jrosser | i'm not sure - the connection plugin does tell you pretty much exactly what it is doing if you run with -vvvv | 09:51 |
| hwoarang | yeah it's using the hostname instead of ip | 09:51 |
| jrosser | this should all end up in the /etc/hosts of all the hosts shouldnt it? | 09:53 |
| jrosser | is your deployment host separate? | 09:53 |
| hwoarang | yes | 09:53 |
| jrosser | hmm right thats how my lab works too | 09:54 |
| jrosser | but i have proper DNS for all the hardware which matches up with the OSA inventory | 09:54 |
| hwoarang | tricky | 09:54 |
| jrosser | hwoarang: in fact i totally rely on it connecting via hostname, as ssh config makes the deploy host connect to the lab through a bastion based on the lab having its own subdomain | 10:04 |
| jrosser | but your issue seems to be between target nodes rather than deploy->target? | 10:05 |
| *** markvoelker has quit IRC | 10:07 | |
| hwoarang | looking into it | 10:10 |
| *** shyamb has quit IRC | 10:14 | |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible master: [WIP] Replace tempest with healthchecks https://review.openstack.org/587544 | 10:14 |
| jrosser | evrardjp: quick question - have i put these vars in the right place, doesnt seem to work https://review.openstack.org/#/c/591412/5/tests/test.yml | 10:31 |
| evrardjp | jrosser: where does lxc_hosts_extra_distro_packages come from? | 10:33 |
| jrosser | just here https://review.openstack.org/#/c/591412/5/defaults/main.yml | 10:34 |
| evrardjp | interesting | 10:35 |
| evrardjp | let me check were it fails | 10:35 |
| jrosser | i was looking through the lxc cache prep script output and its like those stay empty | 10:36 |
| noonedeadpunk | evrardjp: Hi, could you please take a look at https://review.openstack.org/#/c/586318/ - adding new fact seems not to have a lot of sense, as I need to iterate over a dict in both cases. And as each theme should be placed in its own idrectory - with_glob is not the best option. Of course, we may document to have a leading folder in archive... But I think I'be managed to make it more readable | 10:36 |
| evrardjp | well the assert succeeds though | 10:36 |
| jrosser | yes it does :( i wanted to be double sure and see it actually install the package in the log | 10:36 |
| jrosser | and of course the irony of all of this is that the test that fails is the ony one i've found that actually tries to do the install http://logs.openstack.org/12/591412/5/check/openstack-ansible-functional-centos-7/943a228/logs/host/lxc-cache-prep-commands.log.txt.gz | 10:38 |
| evrardjp | jrosser: maybe worth checking if the logs are well gathered at the end of the play | 10:39 |
| evrardjp | as they seem to be gathered at some point in time | 10:39 |
| evrardjp | wait | 10:40 |
| evrardjp | there is something weird | 10:40 |
| evrardjp | the tasks don't appear in ara | 10:40 |
| jrosser | this is odd - look at the timestamps here http://logs.openstack.org/12/591412/5/check/openstack-ansible-functional-ubuntu-bionic/25bece0/logs/ | 10:42 |
| evrardjp | that's the play that runs | 10:42 |
| evrardjp | http://logs.openstack.org/12/591412/5/check/openstack-ansible-lxc-dir-ubuntu-xenial/a493cad/logs/ara-report/file/db3c6371-14cc-4990-b9ed-d33065c6c83c/ | 10:42 |
| jrosser | would'nt we expect those all to be fairly current | 10:42 |
| jrosser | it's like a mixture of old/new runs | 10:42 |
| evrardjp | I was looking at this | 10:42 |
| *** cshen has quit IRC | 10:42 | |
| evrardjp | that happens sometimes | 10:43 |
| evrardjp | I would suggest to wait a little and recheck | 10:43 |
| jrosser | spooky :) | 10:43 |
| jrosser | that could get very confusing | 10:43 |
| evrardjp | that's not the first time I see this | 10:43 |
| evrardjp | it's generally nothing to worry about | 10:43 |
| evrardjp | but my concern right now is a precedence issue | 10:43 |
| evrardjp | and lxc_hosts is running lxc_container_create tests.yml | 10:43 |
| evrardjp | that would be pretty bad | 10:44 |
| jrosser | is there a better place i can put these vars? | 10:44 |
| evrardjp | that's fine | 10:44 |
| jrosser | ok | 10:44 |
| evrardjp | you can in test-vars | 10:44 |
| evrardjp | but I think it's fine like this | 10:44 |
| evrardjp | I still want to move to goss | 10:45 |
| *** shyamb has joined #openstack-ansible | 10:45 | |
| evrardjp | when this will be done it will be easier | 10:45 |
| jrosser | anyway thx for the advice, i'll recheck it a bit later | 10:45 |
| jrosser | if this one looks ok it can be a pattern for adding extra packages elsewhere | 10:46 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible master: [WIP] Replace tempest with healthchecks https://review.openstack.org/587544 | 10:50 |
| evrardjp | jrosser: I think it's better in the task to have flexibility | 10:51 |
| evrardjp | defaults are not even forced to exist | 10:51 |
| *** sep has joined #openstack-ansible | 10:53 | |
| jrosser | evrardjp: would you like more changes - from yesterdays discussion the defaults were there to document the entry point | 10:53 |
| evrardjp | jrosser: that's better to have defaults for documentation | 10:54 |
| evrardjp | I am fine with that right now -- what I meant was this pattern is good | 10:54 |
| evrardjp | even for undocumented features | 10:54 |
| jrosser | ah ok :) no problem | 10:54 |
| jrosser | hmm looks like htop is a poor choice of test package -> EPEL | 10:57 |
| evrardjp | haha | 10:59 |
| evrardjp | sl ? | 10:59 |
| *** udesale has quit IRC | 10:59 | |
| evrardjp | that's probably a poor test package too | 10:59 |
| jrosser | sysstat maybe | 10:59 |
| evrardjp | but that's fun | 10:59 |
| evrardjp | cowsay? | 10:59 |
| evrardjp | sysstat have various names due to packaging | 11:00 |
| jrosser | it's pretty hard to find trivial packages that are on all distros | 11:00 |
| evrardjp | who spoke yesterday about a ceph issue with no such file or directory | 11:02 |
| evrardjp | hwoarang: ? | 11:02 |
| jrosser | yes and i've seen that too | 11:02 |
| evrardjp | yeah | 11:02 |
| evrardjp | well My patch is blocked due to this, so I have to work on it | 11:02 |
| jrosser | see this https://github.com/ceph/ceph-ansible/commit/f422efb1d6b56ce56a7d39a21736a471e4ed357c | 11:02 |
| *** markvoelker has joined #openstack-ansible | 11:04 | |
| evrardjp | yeah I am rolling back to an old commit in stable/rocky | 11:05 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible stable/rocky: Temporarily use tempest master for all builds https://review.openstack.org/591357 | 11:09 |
| evrardjp | that should unblock rocky creation | 11:09 |
| evrardjp | let me freeze master to previous sha, until a fix is found | 11:09 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible master: Temporarily use tempest master for all builds https://review.openstack.org/590778 | 11:12 |
| *** cshen has joined #openstack-ansible | 11:13 | |
| *** shyamb has quit IRC | 11:31 | |
| *** shyamb has joined #openstack-ansible | 11:32 | |
| noonedeadpunk | evrardjp: is this patch ok for you now? https://review.openstack.org/#/c/586318/ Please take a look on it, once you have time to spare | 11:33 |
| *** markvoelker has quit IRC | 11:37 | |
| jrosser | evrardjp: there may be a pattern to the vanishing vars in https://review.openstack.org/#/c/591412/ - openstack-ansible-functional-* seems to see them openstack-ansible-lxc-* seems to miss them | 11:38 |
| evrardjp | noonedeadpunk: sorry I was busy -- will do when I can | 11:39 |
| jrosser | i suspect this might be down to how the vars are overridden for specialising those -lxc-* tests? | 11:39 |
| evrardjp | jrosser: I would like to finish what I am up to, then I can help you | 11:39 |
| evrardjp | sorry for the delay | 11:39 |
| jrosser | don't worry - i'm about to go for lunch and have a pile of paperwork to deal with | 11:40 |
| jrosser | any time is cool | 11:40 |
| *** ssbarnea has quit IRC | 11:44 | |
| *** nicolasbock has quit IRC | 11:49 | |
| hwoarang | evrardjp: sorry i was out for lunch | 11:50 |
| hwoarang | jrosser: evrardjp https://github.com/ceph/ceph-ansible/pull/3015 | 11:50 |
| *** ssbarnea has joined #openstack-ansible | 11:52 | |
| evrardjp | the problem is that docker_exec_cmd is empty? | 11:52 |
| evrardjp | and not testing ceph command? | 11:52 |
| hwoarang | the problem is the ceph command | 11:53 |
| evrardjp | well anyway... I see | 11:53 |
| hwoarang | doesn't the commit message explain it? if not, i need to improve it | 11:53 |
| evrardjp | it's good | 11:55 |
| evrardjp | I am just eating at the same time, so I didn't want to clicky click | 11:55 |
| evrardjp | but now I am clicky clicky anyway | 11:55 |
| *** savvas has joined #openstack-ansible | 11:55 | |
| evrardjp | wait... bad idea. | 11:55 |
| evrardjp | Food time! | 11:55 |
| *** kaiokmo has quit IRC | 11:56 | |
| *** shyamb has quit IRC | 12:02 | |
| *** shyamb has joined #openstack-ansible | 12:06 | |
| *** hamzaachi has quit IRC | 12:07 | |
| mnaser | hi everyone | 12:25 |
| noonedeadpunk | hi mnaser | 12:28 |
| *** udesale has joined #openstack-ansible | 12:33 | |
| *** shyamb has quit IRC | 12:34 | |
| *** shyamb has joined #openstack-ansible | 12:34 | |
| *** shyamb has quit IRC | 12:40 | |
| *** dave-mccowan has joined #openstack-ansible | 12:43 | |
| mnaser | how are you noonedeadpunk ? | 12:44 |
| *** hamzaachi has joined #openstack-ansible | 12:45 | |
| noonedeadpunk | mnaser not bad, actually, thanks) probably you may review cherry pick from master, if you have time of course? https://review.openstack.org/#/c/591335/ and https://review.openstack.org/#/c/591590/ | 12:47 |
| mnaser | sure :D | 12:48 |
| noonedeadpunk | mnaser great, thanks) | 12:49 |
| *** asettle has joined #openstack-ansible | 12:54 | |
| *** Bhujay has quit IRC | 13:07 | |
| *** Bhujay has joined #openstack-ansible | 13:07 | |
| *** kaiokmo has joined #openstack-ansible | 13:16 | |
| *** hvvcben has joined #openstack-ansible | 13:30 | |
| evrardjp | may I get some votes on https://review.openstack.org/#/c/590778/ and https://review.openstack.org/#/c/591357/ to unblock gates for rocky and master? | 13:30 |
| mnaser | evrardjp: done | 13:34 |
| evrardjp | jrosser: could you have a look at my pebkac here: https://review.openstack.org/#/c/587544/ ? | 13:34 |
| *** hvvcben has quit IRC | 13:35 | |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible-os_tempest master: Run smoke and dashboard tests by default https://review.openstack.org/587617 | 13:35 |
| * jrosser looks | 13:35 | |
| evrardjp | thanks mnaser and jrosser for the votes | 13:36 |
| evrardjp | jrosser: what did you say about https://review.openstack.org/#/c/591412/5 and patterns? | 13:37 |
| evrardjp | you mean the jobs are different | 13:37 |
| evrardjp | and? | 13:37 |
| jrosser | well theres a few other places to do this | 13:37 |
| jrosser | nspawn_hosts and maybe openstack_hosts | 13:37 |
| jrosser | so i wanted to just get it all tidy on one first before doing the others | 13:38 |
| evrardjp | ahah ofc | 13:38 |
| evrardjp | wait a moment | 13:38 |
| evrardjp | so right what gets tested with your test.yml change is only applied on lxc_hosts | 13:38 |
| evrardjp | as the lxc-* jobs are applying the lxc-container-create test jobs | 13:39 |
| jrosser | ok - this isnt very obvious :) | 13:39 |
| *** Bhujay has quit IRC | 13:40 | |
| evrardjp | yeah | 13:40 |
| evrardjp | maybe we should use "cross" for all our cross repo testing | 13:40 |
| jrosser | ok - so now i know that so long as there is enough coverage in the -functional-* jobs it should be ok | 13:41 |
| evrardjp | yup | 13:41 |
| jrosser | righto, so i'll just try to find a package that installs everywhere and it should pass, not sure why centos is nv there as it was OK earlier | 13:42 |
| evrardjp | lshw | 13:42 |
| evrardjp | I checked it's installable everywhere | 13:43 |
| jrosser | oh also NODE_FAILURE for the opensuse-150 on the last go :) | 13:43 |
| evrardjp | that's infra issue | 13:43 |
| noonedeadpunk | evrardjp: do you mean, that instead of introducing new variable, I may specify extra key inside _horizon_available_themes? | 13:44 |
| noonedeadpunk | * horizon_custom_themes | 13:45 |
| openstackgerrit | Dmitriy Rabotjagov (noonedeadpunk) proposed openstack/openstack-ansible-os_horizon master: Implements custom theme distribution https://review.openstack.org/586318 | 13:46 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-lxc_hosts master: Allow user-defined extra distro packages https://review.openstack.org/591412 | 13:46 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-plugins master: strategy: linear: Obtain IPs from all physical hosts https://review.openstack.org/591672 | 13:46 |
| hwoarang | jrosser:evrardjp i have a fix for the connection plugin issue^^ | 13:46 |
| hwoarang | could you have a look please? | 13:47 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-plugins master: strategy: linear: Obtain IPs from all physical hosts https://review.openstack.org/591672 | 13:48 |
| *** KeithMnemonic has joined #openstack-ansible | 13:48 | |
| openstackgerrit | Dmitriy Rabotjagov (noonedeadpunk) proposed openstack/openstack-ansible-os_horizon master: Implements custom theme distribution https://review.openstack.org/586318 | 13:49 |
| evrardjp | jrosser: for the NODE_FAILURE, infra has fixed it, and a recheck should do the trick | 13:51 |
| evrardjp | noonedeadpunk: I am not sure that's what I meant... :) ahah me and my english! | 13:51 |
| jrosser | hwoarang: is it possible to construct a test for that case? i wonder if it is possible to fiddle the inventory to make it look like there are two hosts, perhaps using the different IP on the bridges? | 13:52 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-plugins master: strategy: linear: Obtain IPs from all physical hosts https://review.openstack.org/591672 | 13:55 |
| evrardjp | hwoarang: good catch | 13:55 |
| evrardjp | jrosser: I think we should have unit tests on the linear strategy | 13:56 |
| hwoarang | jrosser: not sure i get that. the code simply iterates over the list of physical_hosts. it's using that as a key to the physical_host_addrs dictionary | 13:56 |
| noonedeadpunk | evrardjp: I understand that my solution is pretty complicated and, probably, not the easiest thing to read. It might be done much simplier by synchronize module, but it establishes separate connection for rsync, which is not really good as for me. Also, I've just realized, that adding another key to horizon_custom_themes dict, which will specify full path to archive, may do the trick. | 13:56 |
| evrardjp | hwoarang: yeah I think if we want to improve reliability of this, we should introduce a better code structure for the strategy, and include unit tests | 13:56 |
| jrosser | hwoarang: i think that i meant that the current tests probably don't try to excercise delegating between hosts | 13:57 |
| jrosser | becasue there is only one | 13:57 |
| jrosser | but it may be possible to make it look like more than one | 13:57 |
| evrardjp | noonedeadpunk: well my concern is that we now have 2 mechanisms to upload files arbitrarily on the horizon servers | 13:57 |
| evrardjp | and it's not listed in the commit message how they are different | 13:58 |
| evrardjp | I think it's worth pointing that you're using this because first method is not good enough for you -- that's what matters | 13:58 |
| noonedeadpunk | evrardjp: oh, ok, now I got it. | 13:58 |
| evrardjp | noonedeadpunk: after that we can think of another commit to cleanup what was not working fine | 14:00 |
| openstackgerrit | Andy Smith proposed openstack/openstack-ansible master: [WIP] Add qdrouterd role for messaging backend deployment https://review.openstack.org/570261 | 14:00 |
| jrosser | hwoarang: ok right so this exists https://github.com/openstack/openstack-ansible-plugins/blob/master/tests/test-connection-plugin.yml#L75-L85 | 14:01 |
| jrosser | i probably need to understand better what failed and why that didnt catch it | 14:01 |
| *** mjwales has joined #openstack-ansible | 14:01 | |
| openstackgerrit | Merged openstack/openstack-ansible-repo_server stable/rocky: Create virtualenv with '--never-download' https://review.openstack.org/591041 | 14:01 |
| mjwales | Hey guys, I'm having issues getting provider networks to work in a newly deployed cloud | 14:02 |
| noonedeadpunk | evrardjp: the idela solution would be to write some hook around synchronize module, but now I don't feel like I can write such things in a good manner... | 14:02 |
| mjwales | On one of the compute nodes the neutron log contains "2018-08-14 15:02:38.455 116102 INFO neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent [-] Interface mappings: {'flat': 'eth12', 'physnet1': 'eth20', 'vlan': 'br-vlan'} | 14:03 |
| mjwales | 2018-08-14 15:02:38.455 116102 INFO neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent [-] Bridge mappings: {} | 14:03 |
| mjwales | 2018-08-14 15:02:38.465 116102 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent [-] Interface eth12 for physical network flat does not exist. Agent terminated!" | 14:03 |
| evrardjp | noonedeadpunk: fixing the synchronize or making it an action should do | 14:03 |
| jrosser | mjwales: search for eth12 here https://docs.openstack.org/project-deploy-guide/openstack-ansible/newton/app-config-test.html | 14:03 |
| evrardjp | but anyway the archive works | 14:04 |
| evrardjp | copy recursive would work | 14:04 |
| Tahvok | mjwales: does the interface exist? | 14:04 |
| jrosser | mjwales: see if that makes sense / matches how you have set your host up | 14:04 |
| mjwales | Ah I've missed that in the setup! Will add that on all the compute nodes now. Thanks. | 14:05 |
| jrosser | network nodes too iirc | 14:06 |
| jrosser | mjwales: the key thing here is that neutron needs an actual interface, rather than a bridge | 14:06 |
| jrosser | so eth12 is a bogus interface wired to br-vlan with a veth pair | 14:07 |
| mjwales | jrosser: I've got 3 infra nodes and 7 compute nodes. Neutron is running in a container on the infra nodes so does it still need to be added on them? | 14:07 |
| jrosser | neutron agent is in the container | 14:07 |
| jrosser | but the actual network namespaces etc will be on the metal | 14:07 |
| noonedeadpunk | evrardjp: I don't think that horizon theme will include hundreds of files. But still recursive copy has a limitation on number of files. So yep, it needs to be discussed... | 14:08 |
| mjwales | Okay, this is quite a learning curve. Will add the eth12 interface on all the nodes now | 14:08 |
| evrardjp | mjwales: the latest documentation for user stories is in: https://docs.openstack.org/openstack-ansible/latest/user/index.html . If you want a real deployment follow the deploy guide. | 14:08 |
| jrosser | evrardjp: google foiled me a gain - would be nice to search-engine-optimise the latest docs :) | 14:09 |
| *** olivierbourdon38 has quit IRC | 14:09 | |
| evrardjp | jrosser: yeah. | 14:09 |
| evrardjp | that search sucks | 14:09 |
| mjwales | I followed the ceph production guide for this deployment. Need to make sure I refer back to the example config for all the comments etc | 14:10 |
| evrardjp | noonedeadpunk: to be honest, I kinda prefer the archive | 14:10 |
| evrardjp | ppl have to prepare them the way they want | 14:10 |
| evrardjp | we just say: please put x there. | 14:10 |
| evrardjp | jrosser: did you find my pebkac? | 14:11 |
| evrardjp | I will destroy my node and recreate I have messed up so many things | 14:11 |
| jrosser | evrardjp: not yet keep getting distracted let me look again, i just finished an aio so will compare | 14:12 |
| jrosser | in fact i could stick that patch on and try it | 14:12 |
| evrardjp | oh wait, if you have an aio you could compare | 14:12 |
| evrardjp | that's even better | 14:12 |
| evrardjp | having something where tempest ran is cool | 14:12 |
| jrosser | yes it's literally just finished this moment - xenial/master | 14:12 |
| evrardjp | cool I have none like this available right now | 14:13 |
| evrardjp | I am not familiar with the master way of networking (kinda have trouble with systemd-networkd way of configuring thigs) | 14:13 |
| openstackgerrit | Dmitriy Rabotjagov (noonedeadpunk) proposed openstack/openstack-ansible-os_horizon master: Implements custom theme distribution https://review.openstack.org/586318 | 14:14 |
| evrardjp | so right now I am sure it's on the network creation I must have a booboo | 14:14 |
| evrardjp | openstack network creation | 14:14 |
| jrosser | evrardjp: see this re networkd https://review.openstack.org/#/c/590424/ | 14:14 |
| jrosser | argh back in a moment | 14:14 |
| evrardjp | noonedeadpunk: nice! | 14:15 |
| *** cshen has quit IRC | 14:15 | |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-plugins master: strategy: linear: Obtain IPs from all physical hosts https://review.openstack.org/591672 | 14:16 |
| *** fghaas has quit IRC | 14:17 | |
| jrosser | evrardjp: so before i do anything heres the tempest results from my fresh aio http://jrosser.woaf.net/stestr_results.html | 14:22 |
| evrardjp | I feel silly noonedeadpunk -- sorry for bouncing you around | 14:22 |
| *** phasespace has quit IRC | 14:23 | |
| dw | does anyone know which component is setting up auto idle-logout? it's rather frustrating | 14:23 |
| dw | im logged in via ssh if it makes any difference | 14:24 |
| logan- | dw: ansible-hardening does this. | 14:24 |
| mnaser | dw: ansible-hardening | 14:24 |
| logan- | set apply_security_hardening: false | 14:24 |
| openstackgerrit | Merged openstack/openstack-ansible-os_ceilometer stable/rocky: Drop un-used packages from role https://review.openstack.org/591044 | 14:26 |
| *** fghaas has joined #openstack-ansible | 14:28 | |
| jrosser | evrardjp: my test of you healthcheck patch fails the same way | 14:29 |
| evrardjp | dw: welcome : ) | 14:30 |
| evrardjp | logan-: that's rather radical | 14:31 |
| evrardjp | folks, we shouldn't merge things in stable/rocky | 14:31 |
| evrardjp | not yet | 14:31 |
| evrardjp | as they are not testable in the integrated gates | 14:32 |
| evrardjp | (should be fixed soon though!) | 14:32 |
| evrardjp | dw you're giving openstack-ansible a try to see how fast it goes with mitogen? | 14:33 |
| dw | others are, i'm just playing whack-the-bug ;) | 14:33 |
| jrosser | dw: o/ | 14:33 |
| * jrosser meetings | 14:33 | |
| jrosser | evrardjp: its the same no route to host - i got meetings but can look more at it in a while | 14:33 |
| evrardjp | dw: we've got many things that are custom, due to the size and history of the project | 14:34 |
| evrardjp | jrosser: yeah thanks it's probably something silly | 14:34 |
| dw | patched strategies, connection plugins.. it's all good :) | 14:34 |
| evrardjp | dw: inorite | 14:34 |
| evrardjp | :D | 14:34 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-plugins master: strategy: linear: Obtain IPs from all physical hosts https://review.openstack.org/591672 | 14:35 |
| evrardjp | some have... let's say legacy cruft in it. We should rewrite them. But everybody is busy with other things | 14:35 |
| evrardjp | dw: how are you testing things right now? | 14:38 |
| evrardjp | Because you'll hit the networking model issues etc. | 14:38 |
| dw | evrardjp: a single quite abused ubuntu 16.04 machine, but this is fine for now, i have more than plenty to work through already | 14:39 |
| *** olivierbourdon38 has joined #openstack-ansible | 14:39 | |
| evrardjp | dw: it's gonna be defaced with an openstack deploy | 14:40 |
| evrardjp | if you have initial requirements it should be fine | 14:40 |
| evrardjp | (https://docs.openstack.org/openstack-ansible/latest/user/aio/quickstart.html) | 14:41 |
| *** olivierbourdon38 has quit IRC | 14:43 | |
| mjwales | Struggling to get my head around provider_networks and all the bridges. I've got three infra nodes that have a bridge br-ex-public for access to my public floating IP pool. None of the compute nodes have access to this network directly. I added a network to provider_networks of type flat and group_binds of neutron_linuxbridge_agent. Am I at all along the right lines? A bit confused by the container and host_bind_override setti | 14:44 |
| mjwales | ngs. | 14:44 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-plugins master: strategy: linear: Obtain IPs from all physical hosts https://review.openstack.org/591672 | 14:45 |
| logan- | hwoarang: if our current method of using the localhost_alt physical_host is unsufficient to reproduce the issue maybe it is time to switch to using a multinode nodeset in that test | 14:49 |
| logan- | nothing stopping us from having multiple separate hosts, it just hasnt been a need to reproduce past issues yet | 14:50 |
| evrardjp | mjwales: IIRC you'd need to say "point to that network" so host_bind_overrides would need to have a nic on the neutron network node (not a bridge) that can be used for passing the br-ex-public things | 14:51 |
| evrardjp | mjwales: oh wait, I think you can pass bridges now | 14:51 |
| evrardjp | have a try | 14:52 |
| *** Bhujay has joined #openstack-ansible | 14:52 | |
| evrardjp | logan-: I'd welcome multi-node testing :) | 14:52 |
| mjwales | evrardjp: my current deployment is using the bridge with errors such as: Interface eth20 for physical network physnet1 does not exist. The mapping is {'flat': 'eth12', 'physnet1': 'eth20', 'vlan': 'br-vlan'}. So I am assuming from this I need to create eth20 on the hosts similar to eth12 on the br-vlan? | 14:54 |
| evrardjp | mjwales: I don't have enough information about your openstack_user_config | 14:54 |
| evrardjp | mjwales: did you read the documentation and user guides? | 14:54 |
| *** mmalchuk has quit IRC | 14:55 | |
| hwoarang | logan-: i agree but i have never done multinode setup in zuul before so it's not something i can do right now :/ | 14:55 |
| mjwales | evrardjp: yep multiple times. I just think I am missing something or making it more complicated than it is. Happy to stick my openstack_user_config on pastebin? | 14:55 |
| hwoarang | i am hoping that 'check experimental' will be enough to give a green light right now | 14:55 |
| evrardjp | mjwales: yeah | 14:55 |
| evrardjp | mjwales: there are comments in the user guide for the test scenario | 14:56 |
| evrardjp | it's easy to miss | 14:56 |
| evrardjp | but it's there | 14:56 |
| evrardjp | mjwales: we have changed the model recently to have neutron agents on bare metal now though | 14:56 |
| evrardjp | so there are a few changes that aren't in the docs (yet!) | 14:56 |
| evrardjp | any help is welcomed | 14:57 |
| mjwales | evrardjp: https://pastebin.com/BZSBt22g. I'm deploying Queens v17.0.7 if that is any help | 14:58 |
| evrardjp | yup | 14:59 |
| evrardjp | mjwales: L47 you have a host_bind_override of eth12 | 15:00 |
| evrardjp | do you have eth12 on your machine? | 15:00 |
| evrardjp | if not you can wire it | 15:00 |
| mjwales | yep, ive added that top the br-vlan ifcg file | 15:00 |
| evrardjp | add this eth12 to your bridge br-ex-public | 15:00 |
| evrardjp | and you're done | 15:00 |
| evrardjp | remove the network br-ex-public | 15:01 |
| evrardjp | our openstack_user_config is now more than ever confusing :p | 15:02 |
| mjwales | I'm so confused...how does that work? | 15:02 |
| evrardjp | well keep in mind neutron agents used to be containerized | 15:02 |
| evrardjp | so what did this do: | 15:02 |
| evrardjp | you added a physical interface on the container, container_interface, named eth12. It was a veth (container_type), plugged into the host in the container_bridge (br-vlan) | 15:03 |
| evrardjp | however neutron could get things passed like that easily, and we had to implement a hack | 15:04 |
| evrardjp | to say use "eth12" (host_bind_override) on the host | 15:04 |
| evrardjp | as the configuration for neutron physical net (net_name) | 15:04 |
| evrardjp | nowadays, things are simpler, as this would not apply anymore if I am not mistaken | 15:05 |
| evrardjp | mjwales: so remove the first br-vlan that's flat | 15:05 |
| evrardjp | or remove the last one | 15:06 |
| evrardjp | and use the proper host_bind_override to pass to neutron | 15:06 |
| *** cshen has joined #openstack-ansible | 15:06 | |
| mjwales | Okay, think I am finally getting this. | 15:06 |
| evrardjp | don't forget the net_name | 15:07 |
| mjwales | I'll go off and have a tinker with the setup following your instructions above. Thanks for all the help :) | 15:08 |
| evrardjp | mjwales: keep in mind of something | 15:08 |
| evrardjp | mjwales: inventory is not something to mess up with, when it's configured | 15:08 |
| evrardjp | so deploy only when you're ready | 15:09 |
| mjwales | This is my first openstack-ansible deployment after moving from Fuel. I'll get everything running and then redeploy the entire cloud to ensure it is clean. | 15:10 |
| logan- | hwoarang: any way you could share connection debugging (ansible with -vvvvv) for that failed task? | 15:10 |
| hwoarang | i dont have that anymore i am afraid. but it's nothing special really | 15:12 |
| hwoarang | the dictionary only contains the host->ip mapping for the physical_host entry. but if you delegate to a container in another host, the mapping for that host is missing so ansible tries to ssh to $hostname for that second host instead of its ip | 15:13 |
| hwoarang | if you dont have DNS or /etc/hosts entry for this second host, then ssh fails due to name resolution probs | 15:13 |
| *** DanyC has quit IRC | 15:14 | |
| *** DanyC has joined #openstack-ansible | 15:15 | |
| logan- | thats what https://github.com/openstack/openstack-ansible-plugins/blob/master/tests/test-connection-plugin.yml#L75-L85 does | 15:17 |
| *** DanyC has quit IRC | 15:19 | |
| logan- | it delegates from https://github.com/openstack/openstack-ansible-plugins/blob/master/tests/inventory#L15-L20 to containers on localhost | 15:19 |
| hwoarang | yeah but it delegates to localhost | 15:19 |
| hwoarang | everybody can ssh to localhost ;p | 15:19 |
| hwoarang | problem is if you ssh to a real host you dont know about | 15:19 |
| hwoarang | so we need a test which delegates from a container on localhost to a container to hostA somewhere else | 15:20 |
| hwoarang | do you understand what i mean? | 15:20 |
| logan- | got it | 15:20 |
| hwoarang | ok | 15:20 |
| hwoarang | i am about to leave now but i can share logs tomorrow. i can run the opnfv tests again overnight | 15:20 |
| logan- | so the reverse direction of this test (where the source is <otherhost> delegating to localhost) | 15:20 |
| hwoarang | yes | 15:21 |
| hwoarang | well no, source can be localhost, destination needs to be somewhere else | 15:21 |
| logan- | err yes | 15:21 |
| logan- | gotcha | 15:21 |
| hwoarang | ok | 15:21 |
| *** cshen has quit IRC | 15:29 | |
| *** mjwales has left #openstack-ansible | 15:30 | |
| FrankZhang | evrardjp: hey man, just wonder the reason why changing ceph-ansible branch to specific SHA here: https://review.openstack.org/#/c/590778/5/ansible-role-requirements.yml | 15:31 |
| evrardjp | FrankZhang: see commit message | 15:31 |
| FrankZhang | cool, saw it when scroll down a bit | 15:33 |
| *** cshen has joined #openstack-ansible | 15:33 | |
| openstackgerrit | Dmitriy Rabotjagov (noonedeadpunk) proposed openstack/openstack-ansible-os_horizon master: Implements custom theme distribution https://review.openstack.org/586318 | 15:34 |
| openstackgerrit | Dmitriy Rabotjagov (noonedeadpunk) proposed openstack/openstack-ansible-os_horizon master: Implements custom theme distribution https://review.openstack.org/586318 | 15:35 |
| openstackgerrit | Dmitriy Rabotjagov (noonedeadpunk) proposed openstack/openstack-ansible-os_horizon master: Implements custom theme distribution https://review.openstack.org/586318 | 15:36 |
| *** savvas has quit IRC | 15:36 | |
| *** fghaas has quit IRC | 15:39 | |
| evrardjp | mnaser: thanks for the work on publishing ceilometer on PyPI :) | 15:48 |
| evrardjp | mnaser: do you know what's not published on PyPI yet? | 15:48 |
| mnaser | evrardjp: im not sure, i know a bunch of other major ones are | 15:49 |
| mnaser | mordred was working on some of that | 15:49 |
| *** hamzaachi has quit IRC | 15:49 | |
| * mnaser is worried about highlighting everyone for the meting | 15:50 | |
| *** vnogin has quit IRC | 15:50 | |
| bgmccollum | if they didn't want to be highlighted, then wouldn't have put their name in the wiki | 15:53 |
| evrardjp | mnaser: don't be, do it! | 15:53 |
| evrardjp | bgmccollum: that's fair | 15:53 |
| * mnaser holds breath | 15:54 | |
| mnaser | cloudnull, DimGR, andymccr, d34dh0r53, hughsaunders, b3rnard0, palendae, odyssey4me, serverascode, rromans, erikmwilson, mancdaz, _shaps_, BjoernT, claco, echiu, dstanek, jwagner, ayoung, prometheanfire, evrardjp, arbrandes, scarlisle, luckyinva, ntt, javeriak, spotz, vdo, jmccrory, alextricity25, jasondotstar, admin0, michaelgugino, ametts, bgmccollum, darrenc, JRobinson__, colinmcnamara, thorst, adreznec, eil397, | 15:54 |
| mnaser | qwang,nishpatwa_, cathrichardson, drifterza, hwoarang, cshen, ullbeking, Tahvok, mnaser, nicolasbock, jrosser | 15:54 |
| mnaser | meeting in 5 minutes! | 15:54 |
| bgmccollum | that wasn't so bad, was it? | 15:54 |
| evrardjp | bgmccollum: I know the next volunteer for running the meeting :D | 15:55 |
| * bgmccollum backs slowly away | 15:55 | |
| evrardjp | bgmccollum: thanks for being so frequently there -- I appreciate it :) | 15:55 |
| spotz | So aparently we're missing a freenode bot in the channel which is why we can ping:) | 15:55 |
| evrardjp | cores, it would be nice if we continue the rotation of meeting chairs | 15:56 |
| evrardjp | or anyone, not just cores :) | 15:56 |
| noonedeadpunk | as meeting in several minutes I have 1 quick question - how to add smth to agenda? I didn't find a link to etherpad on wiki. Or it's closed? I don't need to add anything right now, but still | 15:59 |
| mnaser | you can edit the wiki page noonedeadpunk | 16:00 |
| evrardjp | you need to be logged in to see the wiki edit button | 16:00 |
| *** throwsb1 has joined #openstack-ansible | 16:00 | |
| evrardjp | the wiki link is on the topic of this chan | 16:00 |
| mnaser | #startmeeting openstack_ansible_meeting | 16:01 |
| openstack | Meeting started Tue Aug 14 16:01:23 2018 UTC and is due to finish in 60 minutes. The chair is mnaser. Information about MeetBot at http://wiki.debian.org/MeetBot. | 16:01 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 16:01 |
| openstack | The meeting name has been set to 'openstack_ansible_meeting' | 16:01 |
| noonedeadpunk | evrardjp: yep, but the question was about agenda location | 16:01 |
| noonedeadpunk | nevermind) | 16:01 |
| mnaser | #topic rollcall | 16:01 |
| mnaser | o/ | 16:01 |
| guilhermesp | o/ | 16:02 |
| *** gyee has joined #openstack-ansible | 16:02 | |
| *** pcaruana has quit IRC | 16:02 | |
| bgmccollum | o/ | 16:02 |
| evrardjp | o/ | 16:02 |
| ansmith | o/ | 16:02 |
| mnaser | #topic last week highlights | 16:02 |
| mnaser | i apologize, i missed the last meeting, could anyone update on which are stale items and which were added? | 16:03 |
| mnaser | inside https://wiki.openstack.org/wiki/Meetings/openstack-ansible#Meeting_section_.22Last_week_highlights.22 | 16:03 |
| evrardjp | I haven't changed those sorry | 16:03 |
| mnaser | no worries, they all seem somewhat still relevant | 16:03 |
| evrardjp | let me update them real quick | 16:03 |
| mnaser | jrosser: has put in a whole lot of work on bionic | 16:04 |
| mnaser | still hacking around https://review.openstack.org/#/c/591287/ afaik | 16:04 |
| jrosser | o/ its very close - i have a review for enabling a nv test on the integrated build which gets as far as the usual tempest fail | 16:05 |
| evrardjp | mnaser: updated. | 16:05 |
| mnaser | please help with reviews and any work that jrosser needs help with. | 16:05 |
| mnaser | next: leap15 doesnt seem to have progressed, it would be good to get some updates to know if we're close or not, because we have these always failing jobs | 16:06 |
| mnaser | so we want to be good citizens of infra and avoid having a forever failing job | 16:06 |
| evrardjp | mnaser: there was issues in infra on leap 15, and mariadb started to move this morning | 16:07 |
| mnaser | okay cool, so we have progress so that's awesome | 16:07 |
| evrardjp | so it's slowly progressing | 16:07 |
| mnaser | evrardjp: RC1 published. Branching happened, but we need a freeze in OSA. I am tracking this. | 16:07 |
| guilhermesp | We also have the transient issues with cinder. I didnt take a deep look into it yet. But a intend to do | 16:07 |
| mnaser | sounds good | 16:08 |
| guilhermesp | Im on mobile, so I wont find the patch easily | 16:08 |
| evrardjp | guilhermesp: maybe the fix jrosser is helping. It helped me at least. But we need to track if this happens again. | 16:08 |
| mnaser | i assume the freeze is https://review.openstack.org/#/c/590503/ | 16:08 |
| evrardjp | but that's a good highlight, thanks guilhermesp ! | 16:08 |
| evrardjp | mnaser: yes | 16:09 |
| mnaser | so it looks like it's going through the gates right now | 16:09 |
| *** rpittau has quit IRC | 16:09 | |
| mnaser | and then | 16:09 |
| mnaser | evrardjp: please fill this etherpad https://etherpad.openstack.org/p/osa-stein-ptg | 16:09 |
| evrardjp | mnaser: indeed. The extra part will be to check when tempest is released | 16:09 |
| evrardjp | then we can use tempest from pypi in stable/rocky | 16:09 |
| mnaser | evrardjp: solid! | 16:09 |
| mnaser | if you're attending (or not) the ptg, please add topic discussions. i would love for us to find a way to get folks who cant make it up in a google hangout or something | 16:10 |
| evrardjp | and last, is we fix ceph-ansible, which hwoarang has a patch for | 16:10 |
| evrardjp | mnaser: ++ | 16:10 |
| evrardjp | please add your conversations, questions, or anything there. | 16:10 |
| mnaser | i think it would probably be good to send an email out if anyone isn't around to see that, but ive been trying to push it on irc here and there | 16:10 |
| mnaser | if we don't have any other things, we can jump to bug triage | 16:11 |
| evrardjp | agreed | 16:12 |
| evrardjp | I will send the email | 16:12 |
| mnaser | evrardjp: cool, thank you | 16:12 |
| mnaser | #topic bug triage | 16:12 |
| mnaser | #link https://bugs.launchpad.net/openstack-ansible/+bug/1786292 | 16:12 |
| openstack | Launchpad bug 1786292 in openstack-ansible "Integration of swift api with ceph RGW" [Critical,New] | 16:12 |
| jrosser | i have some work on go-faster which we could talk about at the ned | 16:12 |
| jrosser | *end | 16:12 |
| guilhermesp | I will be afk, read the logs later. Seeya | 16:12 |
| mnaser | sigh | 16:12 |
| mnaser | guilhermesp: take care | 16:12 |
| mnaser | jrosser: please if you can! :) | 16:13 |
| mnaser | that is *not* a critical bug | 16:13 |
| bgmccollum | My reading is a misunderstanding between swift and radosgw / ceph | 16:13 |
| mnaser | bgmccollum: i kinda agree. | 16:13 |
| mnaser | "swift is still not using Ceph" | 16:13 |
| bgmccollum | it never will | 16:13 |
| mnaser | yep | 16:13 |
| bgmccollum | *well, maybe not never* | 16:13 |
| andymccr | he might mean swiftcli - but that is probably because it has 2 endpoints of the same type and may be default picking the first one (swift) | 16:13 |
| prometheanfire | never is a long time | 16:14 |
| mnaser | yeah both ceph and swift are deployed which is never right | 16:14 |
| mnaser | *ceph radosgw and swift | 16:14 |
| jrosser | this is a user story | 16:14 |
| jrosser | becasue there is subtlety here if you want to so S3 and swift at the same time | 16:14 |
| jrosser | *do | 16:14 |
| * jrosser adds to todo list | 16:15 | |
| prometheanfire | I could see using both, but very rarely | 16:15 |
| jrosser | i mean those things served up by ceph/radosgw | 16:15 |
| mnaser | well right now its the swift client probably not sure about which endpoint to use | 16:15 |
| mnaser | so i think the issue here is that a) you cant run both swift and ceph radosgw (at least, not suppoted and tested by OSA) | 16:16 |
| mnaser | and i think that's it considering we don't own ceph-ansible playbooks and they create all the things | 16:16 |
| andymccr | we create the endpoints etc | 16:17 |
| andymccr | but yeah having multiple endpoints of the same type wont work as expected. | 16:17 |
| mnaser | andymccr: oh, well in that case maybe we should ensure: absent for the swift endpoints if radosgw is being deployed? | 16:17 |
| *** cshen has quit IRC | 16:17 | |
| mnaser | it might flip flop if someone wants to deploy swift and ceph, but that's them doing something wrong | 16:17 |
| mnaser | or maybe have the swift playbook die if radosgw is enabled and vice versa | 16:17 |
| andymccr | mnaser, i think we just change the port if swift exists (to 7980) - maybe we shouldn't setup the endpoints? or we should put it in a different region | 16:18 |
| andymccr | that feels like a hack to get it to work though - i feel like if you really want that your use case is quite specific and you should probably dictate how we do that | 16:18 |
| mnaser | andymccr: i agree. | 16:18 |
| andymccr | in this case i think there is confusion for sure though | 16:19 |
| mnaser | so what do we feel like setting this too | 16:19 |
| bgmccollum | i think the intent needs to be understood. if he though swift had to be deloyed to get the Swift API backed by Ceph (radosgw), then maybe a documentation addition to clear up the confusion...? | 16:19 |
| andymccr | ^ yeah - we should ask for more info/feedback | 16:19 |
| mnaser | bgmccollum: could you comment for more info? | 16:20 |
| bgmccollum | sure | 16:20 |
| mnaser | thank you bgmccollum | 16:21 |
| mnaser | bgmccollum: you can put the status to incomplete too | 16:21 |
| mnaser | #link https://bugs.launchpad.net/openstack-ansible/+bug/1785592 | 16:22 |
| openstack | Launchpad bug 1785592 in openstack-ansible "dynamic inventory doesn't handle is_metal: false->true change" [Undecided,New] - Assigned to Kevin Carter (kevin-carter) | 16:22 |
| bgmccollum | yes | 16:22 |
| mnaser | that does seem like a very plausable bug | 16:23 |
| mnaser | i think its because the inventory is not removed | 16:23 |
| chandankumar | evrardjp: hello | 16:23 |
| evrardjp | chandankumar: hello, we are in bug triage meeting, can this wait? | 16:23 |
| chandankumar | evrardjp: sure | 16:23 |
| evrardjp | thanks | 16:24 |
| mnaser | i guess it's already assigned to cloudnull | 16:24 |
| mnaser | i will put this as confirmed because i know its actually an issue | 16:24 |
| mnaser | confirmed/high | 16:25 |
| mnaser | #link https://bugs.launchpad.net/openstack-ansible/+bug/1785386 | 16:26 |
| openstack | Launchpad bug 1785386 in openstack-ansible "Integration of Swift and Manila with Openstack-ansible having Ceph backend?" [Wishlist,New] | 16:26 |
| mnaser | looks like this was taken care of last time by spotz | 16:27 |
| mnaser | status is still new thats probably why it showed up | 16:27 |
| mnaser | so we can do confirmed/wishlist | 16:27 |
| mnaser | seems okay? | 16:28 |
| spotz | Yeah sorry! | 16:28 |
| mnaser | next up | 16:28 |
| mnaser | #link https://bugs.launchpad.net/openstack-ansible/+bug/1785365 | 16:28 |
| openstack | Launchpad bug 1785365 in openstack-ansible "LXC container network slow because of kernel debug mesg" [Undecided,New] | 16:28 |
| *** shyamb has joined #openstack-ansible | 16:29 | |
| evrardjp | thanks spotz | 16:29 |
| evrardjp | (sorry for the lag) | 16:29 |
| mnaser | "Later i found iptables checksum-fill for port 80 rule was causing all issue." | 16:29 |
| mnaser | looks like it's a user issue? | 16:29 |
| evrardjp | I think I fixed that. | 16:29 |
| evrardjp | it's both an user issue and a default thing for an AIO | 16:29 |
| evrardjp | I've pushed something, and it has a release note for users. | 16:30 |
| evrardjp | maybe I used the wrong Fixes-Bug. | 16:30 |
| mnaser | evrardjp: got a link to include as a comment to that bug? | 16:30 |
| spotz | wow that reminds me of something we found and fixed 2 years ago | 16:30 |
| evrardjp | my bad, none: https://review.openstack.org/#/c/589463/ | 16:30 |
| evrardjp | mnaser: sorry for that. | 16:30 |
| evrardjp | yeah will do | 16:31 |
| mnaser | evrardjp: it's all good | 16:31 |
| *** DanyC has joined #openstack-ansible | 16:32 | |
| evrardjp | updated | 16:33 |
| mnaser | perfect thank you | 16:33 |
| mnaser | #link https://bugs.launchpad.net/openstack-ansible/+bug/1784880 | 16:33 |
| openstack | Launchpad bug 1784880 in openstack-ansible "Pike to queen upgrade neutron issue " [Undecided,New] | 16:33 |
| mnaser | i think this is a duplicate of the earlier one | 16:34 |
| mnaser | duplicate of 1785592 i think | 16:34 |
| evrardjp | checking 1785592then | 16:35 |
| evrardjp | not so sure if it's a duplicate | 16:35 |
| evrardjp | but they are all linked to moving to bare metal | 16:35 |
| evrardjp | this work was not fully QA-ed | 16:36 |
| mnaser | its an upgrade | 16:36 |
| mnaser | pike => queens took agents to baremetal | 16:36 |
| evrardjp | yeah | 16:36 |
| mnaser | so i think the same thing happened | 16:36 |
| bgmccollum | sounds similar to me | 16:36 |
| evrardjp | they are similar | 16:36 |
| evrardjp | that's the right term | 16:36 |
| evrardjp | the context is the same, not sure what's the root cause | 16:36 |
| evrardjp | but indeed if the inventory is wrong, it could lead to issues in the upgrade. | 16:37 |
| mnaser | i think its a duplicate imho | 16:37 |
| mnaser | its happening because is_metal changes and our inventory doesnt delete the old containers | 16:38 |
| evrardjp | let's mark it duplicate | 16:38 |
| evrardjp | I am fine with the duplicate. | 16:38 |
| mnaser | done | 16:38 |
| mnaser | #link https://bugs.launchpad.net/openstack-ansible/+bug/1783668 | 16:38 |
| openstack | Launchpad bug 1783668 in openstack-ansible "Playbook openstack-service-setup does not run on MNAIO" [Undecided,New] | 16:38 |
| mnaser | i dunno anything about that bug at all | 16:39 |
| jrosser | idk why that would be different from a regular deploy | 16:40 |
| evrardjp | likewise | 16:40 |
| mnaser | MNAIO is multinode all in one.. the one in -ops repo? | 16:40 |
| bgmccollum | yes | 16:40 |
| evrardjp | oh | 16:40 |
| evrardjp | wait | 16:40 |
| jrosser | the playbooks must have been able to target the utility container during the deploy so thats probably a reasonbly obvious error somewhere | 16:40 |
| jrosser | odyssey4me has been working on the mnaio recently | 16:41 |
| evrardjp | jrosser: well I have asked questions | 16:41 |
| mnaser | i think thats good | 16:41 |
| evrardjp | we'll see | 16:41 |
| mnaser | that comment should get us somewhere from there | 16:41 |
| mnaser | do we wanna mark incomplete in the meantime or? | 16:42 |
| evrardjp | yup | 16:42 |
| mnaser | sounds good | 16:43 |
| mnaser | #link https://bugs.launchpad.net/openstack-ansible/+bug/1783423 | 16:43 |
| openstack | Launchpad bug 1783423 in openstack-ansible "Flush all of the cache in memcached issue" [Undecided,New] | 16:43 |
| mnaser | why dont we just use the variables from inventory? | 16:44 |
| mnaser | https://github.com/openstack/openstack-ansible-memcached_server/blob/master/defaults/main.yml#L48-L49 | 16:45 |
| mnaser | as in those 2 | 16:45 |
| evrardjp | yeah | 16:47 |
| evrardjp | we should | 16:47 |
| evrardjp | but how do we load those? | 16:47 |
| evrardjp | as this is not in the group | 16:47 |
| evrardjp | and probably overriden | 16:47 |
| mnaser | evrardjp: dont we run this playbook with openstack-ansible ? | 16:47 |
| evrardjp | it's in group vars | 16:47 |
| evrardjp | so we can now | 16:47 |
| evrardjp | yeah replacing with variables should be the right decision | 16:48 |
| bgmccollum | https://github.com/openstack/openstack-ansible/blob/master/scripts/run-upgrade.sh#L194 | 16:48 |
| mnaser | great so confirmed/medium and add a comment mentioning we can use that instead of regex? | 16:48 |
| evrardjp | yeah | 16:48 |
| evrardjp | I have to go | 16:48 |
| evrardjp | sorry to skip the end of the meeting. ttyl everyone! | 16:48 |
| mnaser | np | 16:49 |
| mnaser | #link https://bugs.launchpad.net/openstack-ansible/+bug/1782388 | 16:49 |
| openstack | Launchpad bug 1782388 in openstack-ansible "Installing Multipath But Not Enabling In Nova Causes Volume Attachment Failures" [Undecided,New] | 16:49 |
| mnaser | all yours bgmccollum :P | 16:50 |
| bgmccollum | so... | 16:50 |
| bgmccollum | last comment sums up findings... | 16:50 |
| bgmccollum | Here is what I've discovered... | 16:50 |
| bgmccollum | If multipath-tools is installed (which is now the default), then nova.conf *MUST* have `iscsi_use_multipath = True` set in nova.conf, or attachments won't work. If your compute hosts are also volume hosts (meaning, multipath is installed and running), then you *MUST* have `use_multipath_for_image_xfer = True` set in cinder.conf under [lvm], or volume migration won't work. | 16:50 |
| bgmccollum | If the volume and instance are on the same host, then attachments aren't an issue... | 16:51 |
| mnaser | bgmccollum: so is this possibly a nova/cinder issue? | 16:51 |
| bgmccollum | but im no expert, and would really like to see if someone else experiences these same issues | 16:51 |
| bgmccollum | possibly... | 16:52 |
| mnaser | i dont really know much about multipathing and lvm/cinder :( | 16:52 |
| bgmccollum | attachments failing doesn't manifest when the volume and instance are on the same host...which is how the gate is setup... | 16:53 |
| bgmccollum | so, need someone with a real deployment with volume / compute co-located | 16:53 |
| bgmccollum | using lvm/iscsi | 16:53 |
| mnaser | i think we dont use multipath in the gate too.. i think? | 16:54 |
| *** udesale has quit IRC | 16:54 | |
| *** hamzaachi has joined #openstack-ansible | 16:54 | |
| bgmccollum | its installed by default | 16:54 |
| mnaser | i'm not sure i can be much help | 16:54 |
| mnaser | id like to keep the bug as 'new', maybe next meeting we'll have more people | 16:55 |
| bgmccollum | no worries... | 16:55 |
| mnaser | #link https://bugs.launchpad.net/openstack-ansible/+bug/1778586 | 16:55 |
| openstack | Launchpad bug 1778586 in openstack-ansible "aio_lxc fails on openSUSE Leap 42.3: package conflict between gettext and gettext-runtime" [Medium,Incomplete] - Assigned to Jean-Philippe Evrard (jean-philippe-evrard) | 16:55 |
| mnaser | i set that as medium/incomplete as it seems to have already been assigned and worked on | 16:55 |
| mnaser | #topic open discussion | 16:56 |
| mnaser | we have a few minutes :) | 16:56 |
| jrosser | https://etherpad.openstack.org/p/osa-faster | 16:56 |
| jrosser | i've been thinking about how to "go faster", this is my braindump | 16:57 |
| jrosser | some could be nonsense, some worthwhile, and i wanted to show this as it's background to some of my reviews, like the eatmydata stuff | 16:57 |
| mnaser | i like these ideas | 16:58 |
| mnaser | mitogen has a really interesting way of being able to delegate to lxc containers | 16:58 |
| mnaser | so we avoid our whole custom connection plugin magic | 16:58 |
| jrosser | i have an AIO+mitogen which gets as far as keystone | 16:58 |
| mnaser | be fun to hack on that | 16:59 |
| jrosser | which for pretty much just switching it on is excellent | 16:59 |
| jrosser | dw: ^ nice work! | 16:59 |
| mnaser | anyone would like to volunteer for running next weeks meeting? | 16:59 |
| mnaser | i guess not, i'll run next weeks :) | 17:00 |
| mnaser | thanks everyone!! | 17:01 |
| mnaser | #endmeeting | 17:01 |
| openstack | Meeting ended Tue Aug 14 17:01:08 2018 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 17:01 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/openstack_ansible_meeting/2018/openstack_ansible_meeting.2018-08-14-16.01.html | 17:01 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/openstack_ansible_meeting/2018/openstack_ansible_meeting.2018-08-14-16.01.txt | 17:01 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/openstack_ansible_meeting/2018/openstack_ansible_meeting.2018-08-14-16.01.log.html | 17:01 |
| *** shyamb has quit IRC | 17:05 | |
| openstackgerrit | Merged openstack/openstack-ansible-ops master: Add Redhat to the ELK deployment capabilities https://review.openstack.org/591467 | 17:06 |
| dw | mnaser: as we're discovering over on #mitogen, it's not working quite well in combination with delegate_to at present :) i'm digging through this evening to figure out precisely how it's supposed to work | 17:15 |
| *** cshen has joined #openstack-ansible | 17:16 | |
| jamesdenton | If installing OSA on SUSE, is OpenSUSE 42.x the way to go, or is 15 good? | 17:18 |
| *** hamzaachi has quit IRC | 17:20 | |
| *** Bhujay has quit IRC | 17:22 | |
| *** tacoboy has joined #openstack-ansible | 17:25 | |
| *** cshen has quit IRC | 17:25 | |
| openstackgerrit | James Denton proposed openstack/openstack-ansible-os_neutron master: Adds networking-ovn support https://review.openstack.org/584067 | 17:27 |
| cloudnull | jamesdenton 42.3 | 17:31 |
| jamesdenton | thx | 17:31 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Update curator to better metric storage https://review.openstack.org/591798 | 17:32 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Update curator to better metric storage https://review.openstack.org/591798 | 17:34 |
| cjloader | anyone seenn this when deploying master to an mnaio? | 17:36 |
| cjloader | fatal: [infra1]: FAILED! => {"changed": true, "cmd": "python2 /opt/get-pip.py\n $(which pip2 || which pip) install setuptools pycrypto pyyaml", "delta": "0:00:00.011988", "end": "2018-08-14 11:53:27.308375", "msg": "non-zero return code", "rc": 1, "start": "2018-08-14 11:53:27.296387", "stderr": "python2: can't open file '/opt/get-pip.py': [Errno 2] No such file or directory\ninstall: target 'pyyam | 17:37 |
| cjloader | l' is not a directory", "stderr_lines": ["python2: can't open file '/opt/get-pip.py': [Errno 2] No such file or directory", "install: target 'pyyaml' is not a directory"] | 17:37 |
| openstackgerrit | James Denton proposed openstack/openstack-ansible-os_neutron master: Adds networking-ovn support https://review.openstack.org/584067 | 17:37 |
| cjloader | [Get pip2 and pw gen deps] | 17:39 |
| openstackgerrit | Taseer Ahmed proposed openstack/openstack-ansible-os_blazar master: Fix functional tests for Blazar https://review.openstack.org/571388 | 17:41 |
| *** cshen has joined #openstack-ansible | 17:46 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Update curator to better metric storage https://review.openstack.org/591798 | 17:49 |
| cloudnull | cjloader seems like pyyaml is not installed ? | 17:50 |
| *** gkadam has quit IRC | 17:53 | |
| *** cshen has quit IRC | 17:55 | |
| *** poopcat has joined #openstack-ansible | 17:59 | |
| *** hamzaachi has joined #openstack-ansible | 18:02 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Update curator to better metric storage https://review.openstack.org/591798 | 18:03 |
| *** cshen has joined #openstack-ansible | 18:06 | |
| jamesdenton | cloudnull I hear you havin' namespace issues | 18:24 |
| *** electrofelix has quit IRC | 18:26 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-os_neutron master: Ensure that the neutron services are not in a privatetmp https://review.openstack.org/591821 | 18:33 |
| cloudnull | jamesdenton yes I did | 18:33 |
| cloudnull | but that should help that | 18:33 |
| jamesdenton | aye | 18:33 |
| cjloader | perhaps this breaks mnaio? https://github.com/openstack/openstack-ansible-ops/commit/241837fc9f8efc0366612df024e65e031c444269 | 18:41 |
| cjloader | nvm | 18:50 |
| openstackgerrit | Andy Smith proposed openstack/openstack-ansible master: [WIP] Add qdrouterd role for messaging backend deployment https://review.openstack.org/570261 | 18:54 |
| cloudnull | if folks have a momenbt to review https://review.openstack.org/#/q/starredby:cloudnull+status:open,n,z | 18:55 |
| cloudnull | that'd be greatly appreciated. | 18:55 |
| cloudnull | and should help resolve some of the issues we've seen with neutron | 18:55 |
| *** mjwales has joined #openstack-ansible | 18:55 | |
| cloudnull | thanks jamesdenton for helping track that issue down | 18:55 |
| cloudnull | -cc evrardjp jrosser | 18:55 |
| *** mjwales has quit IRC | 18:55 | |
| jamesdenton | cloudnull sure. Sorry I hadn't mentioned it - actually wanted to try and validate before bringing it up just hadn't gotten around to it | 18:56 |
| *** mjwales has joined #openstack-ansible | 18:56 | |
| cloudnull | super helpful. | 18:56 |
| cloudnull | jrosser / evrardjp were running into this today and in testing locally I suspect this is one of the main reasons we've been seeing flakiness in the gate. | 18:57 |
| cloudnull | especially with newer kernels | 18:57 |
| jamesdenton | ahh. Yeah I only noticed it recently - but lots of factors in my environment - 4.15, Rocky and master | 18:58 |
| jamesdenton | heh | 18:58 |
| mjwales | Evening guys, sorry to be a pain but I am still having issues getting networking up and running on a clean deployment. On my three infra nodes I have br-ex-public which provides access to my external network - this is configured in the provider_networks as flat with a net_name of physnet1 and set to group_binds network_hosts. When os-neutron-install.yml runs it adds physnet1:eth20 to the linuxbridge_agent.ini on the compute no | 19:00 |
| mjwales | des which do not have a br-ex-public bridge. | 19:00 |
| jamesdenton | hey mjwales - sorry for the trouble | 19:01 |
| jamesdenton | can you post your openstack_user_config.yml at http://paste.openstack.org? | 19:02 |
| *** hamzaachi has quit IRC | 19:03 | |
| *** hamzaachi has joined #openstack-ansible | 19:04 | |
| mjwales | http://paste.openstack.org/show/728034/ - thanks in advance :) | 19:04 |
| cloudnull | jamesdenton yea the newer kernels allow for more of those private options | 19:06 |
| mjwales | The error in the neutron-linuxbridge-agent.log is : ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent [-] Interface eth20 for physical network physnet1 does not exist. Agent terminated!. But i cannot see how eth20 can exist on the compute nodes as they do not have access to my external network | 19:08 |
| jamesdenton | You're right, it doesn't exist. But the generated config set the mapping to physnet1:eth20 | 19:09 |
| jamesdenton | Couple of things here that can be a little confusing, so bear with me | 19:10 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Update curator to better metric storage https://review.openstack.org/591798 | 19:11 |
| jamesdenton | container_bridge is a linux bridge on the physical host that will connect to the respective lxc container(s). On a compute node, that same value will be used for the Neutron provider mapping by default (e.g. physnet1:br-vlan). The linuxbridge agent will then create subinterfaces off br-vlan for any neutron (vlan) provider network. For VLAN 50, the result would be br-vlan.50 in a brq linux bridge configured by neutron. | 19:14 |
| jamesdenton | If host_bind_override is specified, the value is used *instead* - so it would use the named interface rather than the value for container_bridge. | 19:15 |
| *** fghaas has joined #openstack-ansible | 19:16 | |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_octavia master: Enable event streamer and provisioning status sync for V1 API https://review.openstack.org/591829 | 19:17 |
| jamesdenton | Most environments are good to keep the 'br-vlan' provider mapping, and let neutron create the subinterfaces when you create the provider network. I wouldn't usually bother with a flat network. | 19:17 |
| jamesdenton | container_interface is the interface name *inside* the LXC container and doesn't apply to computes | 19:17 |
| mjwales | I have two external networks that provide public (i.e. internet facing floating IPs) and private (i.e. on campus floating IPs) I couldn't see how to achieve this without creating other provider networks? | 19:18 |
| jamesdenton | https://github.com/openstack/openstack-ansible-plugins/blob/master/library/provider_networks#L206-L214 | 19:18 |
| jamesdenton | Do you have one NIC? | 19:18 |
| jamesdenton | with multiple VLANs for these provider networks? Or are they on different NICs? | 19:19 |
| *** DanyC has quit IRC | 19:19 | |
| *** DanyC has joined #openstack-ansible | 19:19 | |
| *** fghaas has quit IRC | 19:21 | |
| mjwales | Infra nodes have 3x10G NICs and compute have 2x10G NICs. For the infra nodes 2x10G NICs are bonded and provide br-mgmt, br-storage, br-vlan, br-vxlan the remaining NIC is connected to our campus network and has two bridges br-ex (for horizon dashboard access) and currently br-ex-public to try anf get the public floating IPs working. For the compute nodes the 2x10G NICs are bonded and provide br-mgmt, br-storage, br-vlan and br | 19:21 |
| mjwales | -vxlan. | 19:21 |
| mjwales | Does that make sense? | 19:22 |
| mjwales | The two external networks are on different tagged VLANs provided by our campus network team accessible from the NIC on the infra nodes. | 19:23 |
| jamesdenton | Perfect. | 19:23 |
| *** DanyC has quit IRC | 19:23 | |
| jamesdenton | So, usually br-vlan is configured to contain a single interface or bond. On an infra node, the neutron_agents container is connected to br-vlan. In your case, the interface inside that container would be named eth12. The eth10 interface would be connected to br-vlan. When you create the provider network, let's say VLAN 50, neutron would create an interface named eth12.50 and connect it to a linux bridge named | 19:28 |
| jamesdenton | 'brqXXXXXX' along with the qgXXXXX interface it created for the router. This configuration is unique to OSA, in a way, since we introduce this LXC container. The second Neutron provider network you create, VLAN 100, would have a corresponding automatically-created interface named 'eth12.100' that is connected to a different brqYYYYY bridge and a different qg interface for a different router. (routers can only | 19:28 |
| jamesdenton | connect to a single external/floating ip network) | 19:28 |
| jamesdenton | For the tenant network side, it would go one of two ways: 1) you create a vlan project/tenant network (vlan 2000). On the compute, br-vlan would be connected to a bond, and br-vlan.2000 would be created automagically and connected to the corresponding brqZZZZZZ bridge for the project network. If you specify host_bind_override as bond1, then bond1.2000 would be used instead of br-vlan.2000 | 19:29 |
| *** vnogin has joined #openstack-ansible | 19:29 | |
| jamesdenton | Anyway - multiple VLANs would be configured to use the provider label of physnet1 | 19:29 |
| jamesdenton | it's rare when you need more than one "provider" interface or mapping | 19:30 |
| jamesdenton | clear as mud | 19:30 |
| mjwales | Okay, but I don't actually see the VLANs from OpenStack | 19:30 |
| jamesdenton | which vlans | 19:30 |
| jamesdenton | "101:200,301:400"? | 19:30 |
| mjwales | The external networks which provide floating IPs | 19:31 |
| jamesdenton | Well, you have to create those using the API. | 19:31 |
| jamesdenton | and when you create the network, use --provider-physical-network physnet1 | 19:32 |
| jamesdenton | that label maps to eth12 inside neutron agent containers and right now, eth20 on computes | 19:32 |
| jamesdenton | but that would need to be addressed | 19:32 |
| mjwales | So on the compute nodes in my current setup it is complaining about eth20. I resolved the errors on the infra nodes by adding br-ex-pub-veth to the br-ex-public bridge with a peer of eth20 | 19:36 |
| mjwales | The compute nodes don't have a br-ex-public...how would I create the veth eth20? | 19:36 |
| jamesdenton | i think that's unnecessary here. I assume your bond is setup as a trunk, right? And they've given you the VLAN ID's you'll need to use? | 19:37 |
| mjwales | Nope, don't have VLAN IDs. We are moving from a Fuel architecture where both the external networks were configured as flat networks on separate bridges | 19:37 |
| jamesdenton | ahh | 19:37 |
| jamesdenton | but those underlying bridges would have had some kind of vlan-tagged interface in them? | 19:38 |
| jamesdenton | you only get one untagged vlan per physical interface | 19:38 |
| bgmccollum | mjwales, jamesdenton i think the `group_binds` for your `br-ex-public` provider_network config might be wrong... | 19:40 |
| bgmccollum | not targeting the compute hosts | 19:40 |
| bgmccollum | or rather neutron_linuxbridge_agent hosts | 19:40 |
| mjwales | That is the weird thing there is no vlan-tagged interface in the previous configuration. I can literally assign an IP from both the VLANs on the same interface and it magically works | 19:41 |
| jamesdenton | do you happen to have the output if 'ip addr show' and/or 'brctl show' on the Fuel setup? Or if it was ovs, then 'ovs-vsctl show'? | 19:42 |
| jamesdenton | this cat can be skinned a few ways. My way or the wrong way. | 19:43 |
| jamesdenton | (kidding) | 19:43 |
| mjwales | The fuel setup got killed to move to OSA | 19:43 |
| jamesdenton | O_o | 19:43 |
| jamesdenton | heh ok | 19:43 |
| jamesdenton | what is connected to your 'br-vlan' bridge on the infra? what physical interface? | 19:44 |
| mjwales | bond0 | 19:44 |
| jamesdenton | What other interfaces are in there at the moment? | 19:45 |
| mjwales | So on the infra nodes I have bond0 and enp59s0f0. enp59s0f0 is connected to our University network | 19:46 |
| mjwales | On the compute nodes I just have bond0 | 19:47 |
| mjwales | On all the nodes bond0 provides br-mgmt, br-storage, br-vlan and br-vxlan | 19:47 |
| jamesdenton | ok, thats fine. what interface do you have in br-mgmt then? bond0.X? | 19:48 |
| jamesdenton | and do you happen to know what the switchport config looks like | 19:48 |
| mjwales | Not for the enp59s0f0 uplink as that is centrally managed. br-mgmt is on bond0.10 | 19:49 |
| jamesdenton | k. So if your former external networks were both flat, but both using bond0, there had to have been some network magic there. Have you tried removing the new "br-ex-public" provider network and creating a single flat network with one of the subnets? | 19:54 |
| mjwales | I can give it a go. All I've tried so far is the config I pasted. The eth20 errors on the compute nodes is what got me | 19:56 |
| jamesdenton | understandable. That can be resolved by removing host_bind_override: "eth20" on line 77 of that pastebin you sent and rerunning the os-neutron-install playbook | 19:58 |
| jamesdenton | Then it would expect a bridge named 'br-ex-public' | 19:59 |
| mjwales | Yeah that's the problem that bridge cannot exist on the compute nodes as they do not have access to the network | 19:59 |
| jamesdenton | well the bridge can exist, but it doesn't need to be connected to anything. | 19:59 |
| mjwales | Okay so simply create a bridge not linked to anything and neutron will be happy :) | 20:00 |
| jamesdenton | yes - just to get over the hump | 20:00 |
| jamesdenton | there may be a better way to do it, just need a little more info/time | 20:00 |
| bgmccollum | is there a group_binds that only targets the agents container on the infra hosts? ignoring the compute hosts? | 20:00 |
| jamesdenton | can you use 'not' there? i don't know | 20:01 |
| *** fghaas has joined #openstack-ansible | 20:01 | |
| mjwales | bgmcollum: I couldn't find anything that gave me that. The docs said to refer to the files in env.d for more info but on my git clone I only have three files iin env.d | 20:01 |
| bgmccollum | all the various groups confuse me...never learned them all.. | 20:01 |
| bgmccollum | let me inspect the inventory of my deployment... | 20:02 |
| jamesdenton | this is good blog post material. good excuse to figure it all out | 20:02 |
| mjwales | Our deployment is a bit weird as our cloud is used for teaching and research. The researchers generally needs IPs that are accessible from the internet but we want private (campus only) IPs for the students so it is less risky. | 20:03 |
| bgmccollum | `neutron_agents_container` might be the correct group_binds | 20:05 |
| jamesdenton | perhaps. | 20:05 |
| bgmccollum | | neutron_agents_container | qct01_neutron_agents_container-d7034820 | | 20:05 |
| bgmccollum | | | qct03_neutron_agents_container-750547e1 | | 20:05 |
| bgmccollum | | | qct02_neutron_agents_container-df700103 | | 20:05 |
| bgmccollum | yup | 20:05 |
| jamesdenton | mjwales totally doable. looks like bgmccollum has you set. good luck | 20:06 |
| mjwales | I'll give it a go and report back (most likely be tomorrow as it is starting to get a little late and I've been looking at this for 14 hours). Thanks for all the help, it is greatly appreciated :) | 20:06 |
| jrosser | jamesdenton: i struugled hugely with the same issues mjwales is having | 20:07 |
| mjwales | jrosser: please don't say that. my scheduled downtime runs out a week thursday and the cloud needs to be back up and fully functional | 20:07 |
| jrosser | setting up dedicated network nodes with dedicated external interfaces using osa user_config / provider_networks is very hard | 20:08 |
| jrosser | i ended up ignoring all that and just wiring the neutron config in manually in group_vars | 20:08 |
| jrosser | imho the documentation could really do with an example of nicely segregated external networks that only go to network nodes | 20:09 |
| mjwales | As a newcomer to this I have struggled with the documentation around everything. Be happy to help contribute what I have learned | 20:10 |
| jrosser | likewise as ive had to really do my own thing here | 20:11 |
| spotz | mjwales: Once you're back to a happy state I can help get you set up and going on that:) | 20:13 |
| bgmccollum | root@qct01:/opt/openstack-ansible# ./scripts/inventory-manage.py -G | 20:13 |
| bgmccollum | helps to demystify which group contains which hosts / containers | 20:13 |
| bgmccollum | known which is the right groups is the fun part | 20:14 |
| bgmccollum | knowing | 20:14 |
| mjwales | spotz: great was thinking today I'd like to get involved with OSA and adding to the docs seems like a good way to start | 20:14 |
| spotz | mjwales: Getting folks started is my specialty:) | 20:14 |
| jamesdenton | jrosser i can work on some docs later this week / next week. different scenarios and whatnot. mjwales you're welcome to lend a hand, too, or i can contribute to whatever you put together. | 20:17 |
| *** fatdragon has joined #openstack-ansible | 20:17 | |
| jamesdenton | off to cub scouts... good luck and ttyl! | 20:18 |
| jrosser | im happy to share my config - even happier to be shown how i should have done it! | 20:18 |
| jamesdenton | please do! will hit you up | 20:18 |
| jrosser | cool thanks | 20:18 |
| *** cshen has quit IRC | 20:19 | |
| *** cshen has joined #openstack-ansible | 20:20 | |
| *** fghaas has quit IRC | 20:23 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/ansible-role-systemd_service master: Set private sandbox options to false by default https://review.openstack.org/591843 | 20:24 |
| *** hamzaachi has quit IRC | 20:25 | |
| *** throwsb1 has quit IRC | 20:35 | |
| *** hamzaachi has joined #openstack-ansible | 20:44 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Update curator to better metric storage https://review.openstack.org/591798 | 20:45 |
| cloudnull | any cores around want to give https://review.openstack.org/#/c/591843 a push forward | 20:49 |
| spotz | jamesdenton mjwales jrosser - I love team work:) | 20:54 |
| spotz | cloudnull: looking | 20:54 |
| FrankZhang | systemd service role were executed twice here, which eventually cause "ERROR! A worker was found in a dead state" https://github.com/openstack/openstack-ansible-os_neutron/blob/master/tasks/neutron_post_install.yml#L183 | 20:55 |
| *** mjwales has left #openstack-ansible | 21:04 | |
| openstackgerrit | Merged openstack/ansible-role-systemd_service master: Set private sandbox options to false by default https://review.openstack.org/591843 | 21:06 |
| *** mmalchuk has joined #openstack-ansible | 21:17 | |
| cloudnull | :) | 21:25 |
| cloudnull | thanks spotz logan- | 21:25 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible master: [WIP] Replace tempest with healthchecks https://review.openstack.org/587544 | 21:25 |
| spotz | cloudnull: NP | 21:25 |
| *** hamzaachi has quit IRC | 21:27 | |
| *** mmalchuk_ has joined #openstack-ansible | 21:27 | |
| openstackgerrit | Antony Messerli proposed openstack/openstack-ansible-ops master: Allow for Swift Leap Upgrade to be skipped https://review.openstack.org/591053 | 21:27 |
| *** mmalchuk has quit IRC | 21:28 | |
| *** DanyC has joined #openstack-ansible | 21:45 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Update curator to better metric storage https://review.openstack.org/591798 | 22:25 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Update curator to better metric storage https://review.openstack.org/591798 | 22:37 |
| *** vnogin has quit IRC | 22:38 | |
| *** HighWatersDev has joined #openstack-ansible | 22:40 | |
| *** cshen has quit IRC | 22:52 | |
| *** cshen has joined #openstack-ansible | 23:03 | |
| *** cshen has quit IRC | 23:08 | |
| *** DanyC has quit IRC | 23:22 | |
| *** DanyC has joined #openstack-ansible | 23:22 | |
| *** DanyC has quit IRC | 23:27 | |
| *** HighWatersDev has left #openstack-ansible | 23:37 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!